[ { "path": ".codespellrc", "content": "[codespell]\nskip = .git,*.pdf,*.svg,*.tsv,man,formats.jsonl,sample_data,*.cast,golden\n# some ad-hoc or less common word spellings or vars used\n# `F`requency and such\nignore-regex = `[A-Z]`[a-z]+\\b\nignore-words-list = vertexes,wil,ned,parms,datas,sav,inport,miniscule\n" }, { "path": ".devcontainer/Dockerfile", "content": "FROM python:3\n" }, { "path": ".devcontainer/devcontainer.json", "content": "{\n \"name\": \"VisiData\",\n \"dockerFile\": \"Dockerfile\",\n \"extensions\": [\n \"ms-python.python\"\n ],\n \"postCreateCommand\": \".devcontainer/post-create.sh\"\n}\n" }, { "path": ".devcontainer/launch.json", "content": "{\n \"version\": \"0.2.0\",\n \"configurations\": [\n {\n \"name\": \"Python: vd launch\",\n \"type\": \"python\",\n \"request\": \"launch\",\n \"program\": \"${workspaceFolder}/bin/vd\",\n \"args\": [\n \"-f\",\n \"pandas\",\n \"${workspaceFolder}/sample_data/benchmark.csv\"\n ],\n \"console\": \"integratedTerminal\"\n },\n {\n \"name\": \"Python: vd attach\",\n \"type\": \"python\",\n \"request\": \"attach\",\n \"processId\": \"${command:pickProcess}\"\n }\n ]\n}\n" }, { "path": ".devcontainer/post-create.sh", "content": "#!/bin/bash\n\nmake install-dev setup-vscode\n" }, { "path": ".devcontainer/settings.json", "content": "{\n \"python.pythonPath\": \"/usr/local/bin/python\",\n \"python.testing.pytestArgs\": [\n \"visidata\"\n ],\n \"python.testing.unittestEnabled\": false,\n \"python.testing.nosetestsEnabled\": false,\n \"python.testing.pytestEnabled\": true\n}\n" }, { "path": ".gitattributes", "content": "www/asciinema-player.js binary\n" }, { "path": ".github/FUNDING.yml", "content": "github: saulpw\npatreon: saulpw\n" }, { "path": ".github/ISSUE_TEMPLATE/bug_report.md", "content": "---\nname: Bug report\nabout: Create an issue if anything doesn't appear to be working right.\ntitle: ''\nlabels: bug\nassignees: ''\n\n---\n\n**Small description**\n\n**Data to reproduce**\n\n[Please include a small amount of sample data that demonstrates the issue.]\n\n**Steps to reproduce**\n\n[If the issue takes more than a few commands, please attach a [saved commandlog](http://visidata.org/docs/save-restore/).]\n\n**Expected result**\n\n**Actual result with screenshot**\n\n[If there is an error message, please include the full stack trace shown with `Ctrl+E` or `vd --debug`.]\n\n**Configuration**\n\n- Does this issue reproduce without any plugins or configuration (using the `-N` CLI flag)?\n\n- Does this issue reproduce with either the [latest release](https://www.visidata.org/releases/), or with the [develop branch](https://www.visidata.org/install/#update-visidata-from-an-existing-installation)?\n\n**Additional context**\n\n- What platform and version are you using (Linux, MacOS, Windows)?\n\n- Which version of Python?\n\n- Which terminal are you using (for display and input issues)?\n" }, { "path": ".github/ISSUE_TEMPLATE/feature-request.md", "content": "---\nname: Feature request\nabout: Provide feedback on how you want VisiData to grow in the future.\ntitle: '[] '\nlabels: wishlist\nassignees: ''\n\n---\n\nWe want to hear your ideas about where VisiData can go and anything you think could be made smoother.\nWe consider all ideas that are feasible, but please don't be offended if we close it--we generally limit our open issues to bugs and small concrete feature requests that are part of the planned roadmap.\n\nIf you want to propose a command, please include:\n\n- [] what should the helpstring say\n- [] where should this command go in the menu system\n" }, { "path": ".github/ISSUE_TEMPLATE/loader-request.md", "content": "---\nname: Loader request\nabout: Suggest a loader that supports a new data format\ntitle: \"[loader] \"\nlabels: feature request\nassignees: ''\n\n---\n\nWe love to build loaders with people that are familiar with the format and how it is used. Knowing about quirks and variations and usage-in-the-wild is very hepful.\n\nTo that end, a new loader can be created quite quickly with the following:\n\n1) A small sample representative dataset in that format.\n\n2) A basic Python script which queries and dumps the data (possibly using an external package).\n\n" }, { "path": ".github/ISSUE_TEMPLATE/question.md", "content": "---\nname: Question\nabout: Ask a question you have about VisiData.\ntitle: '[question]'\nlabels: question\nassignees: ''\n\n---\n\nIf you think your question would benefit from input from anyone in the community (e.g. 'How do I X in VisiData') consider [opening a Discussion instead](https://github.com/saulpw/visidata/discussions/new/choose).\n\nIf you think it would benefit from a more synchronised back-and-forth with Saul, consider joining us in libera.chat (#visidata) or [discord](https://bluebird.sh/chat).\n\n" }, { "path": ".github/PULL_REQUEST_TEMPLATE.md", "content": "- [ ] If contributing a core loader, [the loader checklist](https://visidata.org/docs/contributing#loader) was referenced.\n- [ ] If registering an external plugin, [the plugin checklist](https://visidata.org/docs/contributing#plugins) was referenced.\n" }, { "path": ".github/workflows/main.yml", "content": "name: visidata-ci-build\non:\n pull_request:\n branches:\n - stable\n - develop\n push:\n branches:\n - stable\n - develop\n\njobs:\n full-tests:\n\n strategy:\n matrix:\n python-version: [\"3.9\", \"3.14\"]\n\n runs-on: ubuntu-slim\n timeout-minutes: 20\n steps:\n - uses: actions/checkout@v4\n\n - name: Set up Python ${{ matrix.python-version }}\n uses: actions/setup-python@v4\n with:\n python-version: ${{ matrix.python-version }}\n cache: 'pip'\n\n - name: Install VisiData with test dependencies\n run: make install-test\n\n - name: Run tests\n run: make test\n\n unit-tests:\n\n strategy:\n matrix:\n python-version: [\"3.10\", \"3.11\", \"3.12\", \"3.13\"]\n\n runs-on: ubuntu-slim\n timeout-minutes: 10\n steps:\n - uses: actions/checkout@v4\n\n - name: Set up Python ${{ matrix.python-version }}\n uses: actions/setup-python@v4\n with:\n python-version: ${{ matrix.python-version }}\n cache: 'pip'\n\n - name: Install VisiData with test dependencies\n run: make install-test\n\n - name: Run unit tests\n run: make test\n" }, { "path": ".github/workflows/vdsql.yml", "content": "name: vdsql-testing\non:\n pull_request:\n branches:\n - stable\n - develop\n push:\n branches:\n - stable\n - develop\n\njobs:\n run-tests:\n\n strategy:\n matrix:\n python-version: [\"3.10\", \"3.14\"]\n\n runs-on: ubuntu-slim\n steps:\n - uses: actions/checkout@v2\n\n - name: Set up Python ${{ matrix.pythonversion }}\n uses: actions/setup-python@v2\n with:\n python-version: ${{ matrix.python-version }}\n\n - name: Install VisiData\n run: make install\n\n - name: Install vdsql\n run: pip3 install visidata/apps/vdsql/\n\n - name: Ensure it starts up\n run: vdsql --version\n\n - name: Run tests\n run: make test-vdsql\n" }, { "path": ".github/workflows/vgit.yml", "content": "name: vgit-testing\non:\n pull_request:\n branches:\n - stable\n - develop\n push:\n branches:\n - stable\n - develop\n\njobs:\n run-tests:\n\n strategy:\n matrix:\n python-version: [\"3.9\", \"3.10\", \"3.11\"]\n\n runs-on: ubuntu-slim\n steps:\n - uses: actions/checkout@v2\n\n - name: Set up Python ${{ matrix.pythonversion }}\n uses: actions/setup-python@v2\n with:\n python-version: ${{ matrix.python-version }}\n\n - name: Install VisiData\n run: make install\n\n - name: Install vgit\n run: pip3 install visidata/apps/vgit/\n\n - name: Ensure it starts up\n run: vgit --version\n\n - name: Create visidatarc\n run: touch ~/.visidatarc\n\n - name: Import vgit\n run: echo \"import visidata.apps.vgit\" >> ~/.visidatarc\n\n - name: Run tests\n run: make test-vgit\n" }, { "path": ".gitignore", "content": "__pycache__\n.tox/\n.idea/\n.vscode/\nvisidata.egg-info/\n*.egg-info/\n*v_env*\n*~\n*.swp\n*.swo\n*dist\n*_build\nbuild/\ntags\n/tests/log/\n# Mac files\n.DS_Store\n.direnv\n.envrc\n.meta\n\n# Generated man pages (run `make man` to build)\nvisidata/man/vd.1\nvisidata/man/visidata.1\nvisidata/man/vd.txt\ndocs/man.md\n" }, { "path": ".gitmodules", "content": "[submodule \"deps/pyxlsb\"]\n\tpath = deps/pyxlsb\n\turl = https://github.com/saulpw/pyxlsb.git\n\n[submodule \"deps/requests-cache\"]\n\tpath = deps/requests-cache\n\turl = https://github.com/saulpw/requests-cache.git\n" }, { "path": ".gitpod.Dockerfile", "content": "FROM python:3\n" }, { "path": ".gitpod.yml", "content": "image:\n file: .gitpod.Dockerfile\n\ntasks:\n - init: pip install -r ./dev/requirements-dev.txt && pip install -e .\n" }, { "path": ".mailmap", "content": "# .mailmap - Consolidate git author identities\n# See: https://git-scm.com/docs/gitmailmap\n#\n# Format: Canonical Name Old Name \n# Lines map alternative identities to canonical form for git shortlog, blame, etc.\n\n# AJ Kerrigan\nAJ Kerrigan \nAJ Kerrigan \n\n# Anja Kefala\nAnja Kefala anjakefala \nAnja Kefala anjakefala \nAnja Kefala Kefala \nAnja Kefala \n\n# David Branner (was misconfigured as \"Your Name\")\nDavid Branner Your Name \n\n# geekscrapy (also committed as molley)\ngeekscrapy molley \ngeekscrapy \n\n# pacien\npacien \n\n# Saul Pwanson\nSaul Pwanson saulpw \nSaul Pwanson \nSaul Pwanson \n" }, { "path": ".theia/settings.json", "content": "{\n \"python.testing.pytestArgs\": [\n \"visidata\"\n ],\n \"python.testing.unittestEnabled\": false,\n \"python.testing.nosetestsEnabled\": false,\n \"python.testing.pytestEnabled\": true\n}\n" }, { "path": "AGENTS.md", "content": "# Repository Guidelines\n\n## Start Here (Progressive Disclosure)\n- Read `CLAUDE.md` first for the primary contributor workflow and architecture context.\n- Use focused deep dives only when needed:\n - `dev/STYLE.md` for coding patterns and API conventions.\n - `dev/GIT.md` for commit and branch practices.\n - `dev/DOCS.md` for documentation syntax and style.\n - `dev/PERFORMANCE.md` for profiling and optimization work.\n- Human gate is required: all AI-assisted code must be reviewed, approved, and tested by a human before merge or PR.\n\n## Project Structure\n- `visidata/`: main package (`features/`, `loaders/`, `apps/`, `themes/`).\n- `tests/`: functional replay tests (`.vd/.vdj/.vdx`) and `tests/golden/` outputs.\n- `visidata/tests/`: Python unit tests.\n- `docs/` and `dev/`: user and developer documentation.\n\n## Build, Test, and Development Commands\n- `python3 -m pip install .`: install local package.\n- `python3 -m pip install \".[test]\"`: install optional test dependencies.\n- `vd --version`: startup sanity check.\n- `pytest -sv visidata/tests/`: run unit tests.\n- `dev/test.sh -j 4`: run functional cmdlog tests and compare `tests/golden/`.\n- `vd . --batch`: quick load smoke test.\n\n## Coding and Command Conventions\n- Follow `dev/STYLE.md` for naming, quoting, decorators, and sheet/column patterns.\n- Use the canonical command pattern shown in `CLAUDE.md`:\n - `BaseSheet.addCommand('', 'command-name', 'code', 'help text')`\n- For class placement decisions and exceptions, defer to `dev/STYLE.md`.\n\n## Testing Guidelines\n- Update `tests/*.vd*` for workflow-visible behavior changes.\n- Update `visidata/tests/` for isolated Python logic.\n- Before PR: run `pytest -sv visidata/tests/` and `dev/test.sh -j 4`.\n\n## Commit & Pull Request Guidelines\n- Base PRs on `develop` (not `stable`).\n- Follow existing short, imperative subjects; optional scope prefixes like `[test]`, `[docs]`, `[vdsql]`.\n- Keep commits focused and include tests/docs for behavior changes.\n- PRs should include problem, approach, risks, and linked issues; add screenshots/gifs and reproducible `.vd` scripts for UI changes.\n- Follow CAA and copyright assignment requirements in `CONTRIBUTING.md`.\n" }, { "path": "CHANGELOG.md", "content": "# VisiData version history\n\n# v3.4 (unreleased)\n\nThanks to @midichef for many bugfixes and improvements.\n\n## New Features\n\n- [commands] add `define-command` and `document` for current row (#655)\n- [repl] add embedded ptipython REPL (#2736)\n- [edit] add `vd.editCellBindings` for user-customizable cell editing keybindings (#2986)\n- [keys] prettykeys overhaul: all bindings now use human-readable key names (#2594)\n- [selection] add `select-to-prev-selected` and `select-to-next-selected` commands\n- [profile] add `--profile` flag for main thread profiling\n- [vimcompat] add vimcompat experiment with go-page-half bindings @daviewales (#2927)\n- [packages] add Python packages sheet\n- [daw] add vdaw: experimental transcript-based audio editor\n- [colors] add conversion functions to/from xterm256\n\n## Improvements\n\n- [column] add `color_readonly` option; better status for non-writable columns (#2936)\n- [column] mark expanded columns and custom columns as writable (#2936)\n- [edit-cell] fail early on readonly column (#2936)\n- [input] add Ctrl+Bksp, Alt+Bksp, Alt+d bindings (#2500)\n- [keys] add Ctrl+Shift+Up/Down and Alt+NumPad prettykeys\n- [cmdlog] expose CommandLogJsonl (#2940)\n- [indexsheet] bind cancel row to zCtrl+C (#2938)\n- [menu] add show-cursor and show-expr (#2848)\n- [dedupe] enable custom sheet name suffixes\n- [incr] use setValuesTyped to allow setting of non-numeric columns\n- [modify] add Progress to setValuesTyped\n- [movement] add go-screen-* commands (#2797)\n- [graph] accept refline input for multiple xcols; better error checking\n- [graph][seaborn] add title and axis labels @meestahp\n- [aggregator] show summary labels in first non-aggregated column if right edge not on screen\n- [types] add numtype, use for avg/median aggregators @pequiste (#2868)\n- [aggregators] copy formatter/displayer col attrs to AggrColumn\n- [aggregators] preserve type returned by mean/avg/median instead of coercing to int @pequiste\n- [icons] show icons with sheets: Dir, FreqTable, Graph\n- [canvas_text] add maxXY from darkdraw, fix g sliders\n- [clipboard api] add get/setClipboardRows and get/setClipboardCols\n- [help] refer to visidata clipboard as \"internal clipboard\" (#2864 #2865)\n- [memory] move open-memos to BaseSheet\n- [mouse api] add vd.enableMouse; ignore if no curses.mousemask (#2913 #2851)\n- [config] respect XDG_CONFIG_HOME and XDG_CACHE_HOME on macOS @maxim-uvarov-ai-assistant\n- [xdg] use user_data_dir for StoredLists like input_history (#2889)\n- [statusbar] make lstatus_max truncation preserve vd markup (#2908)\n- [shell] add options.max_threads for asynccache used by addcol-shell\n- [asynccache/shell] make asynccache/addcol-shell thread-safe and runnable in macros and replay (#2826)\n- [xlsx] add hlink/folHLink to color list (#2948)\n- [syscopy] add cursorFullDisplay for complex objects (#2806)\n\n## Loaders\n\n- [vdsql] add postgres_schema option to show tables from multiple schemas (#2027)\n- [vdsql] show SQL in sidebar (#2200)\n- [vdsql] aliases for .ddb, .sqlite, .db (#2259)\n- [vdsql] bump ibis version (#2410 #2682)\n- [vdsql] fix join compatibility with Ibis >= 9.0 @terencelaurent (#2899)\n- [airtable] update loader to work with latest pyairtable library @mplattner\n- [conll] update loader to work with pyconll v4.0\n- [eml/pcap] enable loaders for mhtml, cap, pcapng, ntar file types\n- [html] for # of header rows, use options.header\n- [mysql] fix connecting with no password @pequiste\n- [http] add default user agent (#2880)\n\n## Bugfixes\n\n- [pandas] fix loader bugs @weichm (#2968)\n- [mainloop] do not exit if rightstatus throws an exception\n- [mainloop] reset numTimeouts when there are unfinishedThreads @iamleot (#2931)\n- [canvas] delete in linear time instead of quadratic\n- [canvas] properly handle labels containing markup\n- [column] fix col width calc for data with markup\n- [column] fix width that formatValue used for list/dict/tuple\n- [loaders] stop truncating markup in fixed-width data\n- [column] stop truncating col names holding markup\n- [input] fix cursor x when editing data with markup\n- [input] fix use of rowidx before it is set\n- [input] fix perf pegging cpu (#2911)\n- [input] allow editing header of readonly column\n- [palette] when no choices match, show warning on Enter\n- [palette] align choices with prompt before Tab is pressed\n- [palette] clarify that TAB precedes ENTER\n- [settings] prevent lockup when keystroke is same as longname\n- [mainloop] for bindkey cmd, show longname in fail msg; catch fail() from execCommand\n- [repeat] use queueCommand instead of replayOne (#2932)\n- [popen] kill leftover processes that have NOT finished\n- [exit] kill subprocesses before os._exit\n- [modify] clear modified status after load to prevent overvigilant quitguard\n- [cmdlog] replay commands on hidden columns (#2849)\n- [cmdlog] make vdj shebang work on macOS @ilyagr (#2870)\n- [status] do not show previous longname after unbound key (#2779)\n- [join] fail on concat of sheets with unequal # of cols\n- [sync] mark finished thread reliably with endTime\n- [freqtbl] disable paste-before and paste-after\n- [path] remove progress in iter (#2323)\n- [path] handle filenames ending in . in Python 3.14 (#2887)\n- [macro] fix vd attempts to use deleted macros @haoyeau (#2893)\n- [clipboard] fix wrong api usage\n- [http] fix default user agent (#2880)\n- [canvas] fix division by zero when col's only value is a large float (#2884)\n- [movement] fix desc of scroll-up/down/left/right amount\n- [fuzzymatch] stringify non-str values\n- [debug] do not avoid cleanup if options.debug\n- [aggr] catch nonexistent aggr in addcol-aggregate\n- [reload] reload_or_replace function should be on BaseSheet\n- [open-row] allow pandas to open row as sheet (#2925)\n- [postgres] unquote url.password in postgres loader @egwynn\n- [theme] remove nonexistent color from light theme\n- [shell] move open-row from incorrect Open toplevel menu\n- [mainloop] fix undo of first user alteration to a replayed cmdlog\n- [curses] fix crash on startup on NetBSD wscons tty consoles @rsmirnov90 (#2851)\n\n## API\n\n- [threads] add asyncsingle_queue decorator\n\n# v3.3 (2025-09-07)\n\n- added options.disp_help_flags; deprecated options.disp_expert\n - these feature flags are supported: cmdpalette hints nometacols guides inputkeys inputfield all\n - see helpstr for more details\n - set option to '' to turn off all help overlays\n\n- fix: include missing files in MANIFEST.in @chenrui333\n\nThanks to @midichef for most of the other bugfixes and improvements.\n\n## Improvements\n\n- [csv] auto-detect CSV format when csv_delimiter is specified @dennisangemi\n- [basesheet-] start new column names fresh for each sheet\n\n- [form-] after confirm, Esc/^C/^Q/q show disconfirm msg\n\n- [cmdpalette-] add instruction to use Esc to cancel\n- [cmdpalette-] keep case of results, allow case-sensitive search\n- [cmdpalette-] scroll choices with PageUp/PageDown\n- [cmdpalette-] hide choices that do not match input\n\n- [save-] suggest 'stdin' as filename base instead of '-'\n- [sheets-] on reload, fetch fresh data for URLs\n- [tar-] add cols for ext and file type desc, use name as keycol\n- performance improvements\n- display improvements\n\n## Bugfixes\n\n- [archive-] fix extracting to a chosen path\n- [archive-] fix error extracting to overwrite existing file\n- [archive-] fix loading zip files from inside archive files\n\n- [cliptext-] fix display of empty markup as several blank lines\n- [cliptext-] restore display translation of unprintable chars\n- [sheets-] remove calcSingleColLayout use of inaccurate visibleRows\n\n- [plugins liveupdate-] fix broken addcol-new\n\n- [curses-] fix crash on startup on NetBSD (and probably other \\*BSD) wscons tty consoles, which do not expose \"mousemask\" in their system curses implementations. @rsmirnov90\n- [input-] fix editText for files on command line with -b -i -p (#2840)\n- [plugin-] fix autoload for Python versions < 3.10\n\n- [macro-] fix corruption when saving macros after deletion #2787\n- [reload-] wait for any previous reload_rows to finish #2808\n- [resize-] fix resize-col-half for new columns with no rows #2795\n- [sheet-] fix reload on sheets without col layout #2790\n- [status] fix error when no help_sidebars\n\n## Cosmetic\n\n- [freqtbl-] stop printing status for selection on source sheet\n- [input-] handle screen resize during inputMultiple(), editCell(), input()\n\n# API\n\n- [threads-] add asyncsingle_queue decorator\n\n# v3.2 (2025-06-15)\n\nThanks to @midichef for many bugfixes and improvements.\n\n## Improvements\n\n* [config] XDG for default visidata_dir #2716 (#2755)\n- [dev] add Python 3.13 test coverage\n\n- [windows] install vd.cmd #2619 @ptyork\n* [windows] Enable Windows clipboard in WSL #1920 @daviewales\n- [windows] update windows-curses version to 2.4.1 #2119\n* [windows] fix mouse support #2676 @ptyork\n\n- [sort] allow z[ and z] to reverse col sort dir or ignore col\n- [sort] replay sort-add/-change ordering from cmdlog input arg\n- [sort] add sort order to ColumnsSheet #2649\n\n* [loaders psv] add simple .psv loader based on Tsv sheet #2727\n* [loaders numpy] support 2d matrices (#2724) @maxfl\n\n### Commands\n\n* [aggr] addcol-aggregate and addcol-rank-sheet\n- [edit] add sysedit-cell command, using external editor\n- [errors] add sysopen-error command to view vd source code in editor\n- [freeze] add setcol-freeze, bind to z' #2260\n- [graph] add zoom-all-y #2751\n- [join] add per-jointype commands #2603\n- [layout] bind g- to hide-uniform-cols #2577 #2735\n- [syscopy] add syscol-colname #2760\n\n### Options\n\n- [draw] add options.color_multiline_bottom #2715\n- [graph] add options.color_graph_refline\n- [disp] Rename options.disp_pixel_random to options.disp_graph_pixel_random @cool-RR\n\n### Tweaks\n\n* [cmdpalette-] fix scoring of space-separated search terms\n* [cmdpalette-] make fuzzy match case-insensitive\n* [docs] Add WindowFunctionGuide and AggregatorSheet guides #2558 @thejud\n* [theme] update light theme #2729\n* [status] remove non-precious sheets from sheetlist #2573\n- [input-] fix word locator for Ctrl+Right motion\n- [input-] allow edit of cell in hidden column #2749\n- [errors-] make ErrorSheet/ErrorsSheet into singletons\n- [threads-] make threads-all show a singleton sheet\n- [threads-] allow repeated toggling of profiling\n- [IndexSheet] move gC and gI to IndexSheet #2603\n- [main-] enable cell editing for interactive batch mode #2639\n- [loaders hdf5] guess types(hdf5), understand unsigned int type @maxfl\n- [loaders archive,sqlite-] guess sqlite/tar/zip filetypes confidently\n- [loaders vds-] save/restore column-specific properties via getstate/setstate #2699\n\n## Bugfixes\n\n- [aggr-] allow undo for aggregate-col/cols\n- [aggr-] cap runtime when formatting memo status\n- [canvas-] stop infinite refresh for graphs with many points\n- [cmdlog-] save prev replay when starting new replay #2531\n- [deprecated-] show warning when using deprecated commands #2215\n- [features-] reload_every: wait for reload before looping\n- [features-] sysedit: modify cell only if editor changes value (#2656)\n* [freqtbl-] fix excessive memory usage for undo of selections #2759\n- [incr-] addcol-incr-step as expected #2769\n- [input-] preserve None cells on external editor quit\n- [join-] fix putValue for merge rows absent in any source sheet\n- [layout-] change resize-cols-max into a toggle #2782\n- [macro-] only record commands that are replayable\n- [mailcap] install dead battery for mailcap in python 3.13 #2576\n- [main-] prevent hang when vd -p - reads from terminal\n- [npy-] fix bug truncating cols wider than window #2783\n- [open-] fix open-file for - in cmdlogs #2582\n- [open-] fix opening a dir with an filetype extension\n- [reload-] have reload-modified check for earlier changes #2551\n- [save-] fix error string\n- [sheets-] record key-col toggle for replay as key-col-on/-off\n- [sort-] fix undo when sheet has a previous ordering\n- [undo-] fix undo editing cells in ValueColumn, ExpandedColumn, ColumnSourceAttr #2765\n- [vdsql-] ensure each thread in vdsql tracks its single connection\n\n## Cosmetic\n\n- [cliptext-] truncate sheet names handling full-width chars\n- [column-] do not rjustify non-numeric values #2750\n- [csv-] warn when guessed option differs from default #2690\n- [dir-] sort by filename, after sorting by modtime\n- [errors-] fix error-recent always showing \"no error\"\n- [fixed loader] use empty str for null_value in fixed width sheet\n- [form-] fix underlining of substrings by FormCanvas\n- [help- sidebar-] prevent sidebar flicker #2630\n- [help] fix duplicate help descriptions #2762\n- [input-] fix editline() for characters having screen width > 1\n- [input-] speed up pasting long strings into line editor\n- [input-] fix off-by-1 when drawing at right edge of screen\n- [main-] format vd_cli exceptions to be caretless\n- [menu-] remove duplicate Help entry #2714\n- [menu] replace hint with motd\n- [reload-] tolerate stale columns in drawcache during allAggregators #2607 #2763\n- [save-] warn if no save destination given #2580\n- [status] use sheet.icon on sheetlist #2772\n- [threads-] catch error trying to start a second profiler\n\n# v3.1 (2024-10-14)\n\n- drop support for Python 3.7 #2231\n- [vdsql] bump ibis dep to v8\n- [test] fix minimum pandas version to 1.5.3\n\n## Improvements\n\n- [sidebar] Ctrl+G to cycle sidebar (including off) in both main and input modes #2202\n- [sidebar help] rightmost statusline ({vd.sidebarStatus}) shows number of help sidebars (clickable to toggle sidebar)\n- [sheet tabs] left statusline sheet tabs (clickable to jump to sheet) #2030\n- [aggregators] immediate async summary of column aggregators on bottom rows #2209\n- [graph] reflines at user-defined x or y #2487\n- [guides] add several guides (thanks to @thejud @reagle @ajkerrigan @anjakefala)\n- [expressions] much cleanup\n - `row[\"colname\"]` to refer to a specific column by name, without needing to be a valid identifier #2539\n - `_row` is underlying row object\n - support curcol in g= and select and search expressions #2524\n - $curcol for addcol-sh\n - globals include most symbols from math by default; also datetime and date both as the vd.date subclass\n - [input] memory variables included as autocomplete keys #2509\n\n## Tweaks\n\n- [history] create visidata_dir (default ~/.visidata/) if not exists to enable input history by default #2298\n- [cli] \"-p -\" replays stdin as a .vdj file\n- [guide] allow front matter in guide .md files; \"sheettype\" metadata to associate with a sheet\n- [cmdpalette] do not fuzzymatch keys that start with _\n- [interface] hidden columns minutely visible past right of sheet; `_` to re-expand #2394\n- [interface] record all commands in macro (even replay=False); add status indicator when recording #2435\n- [interface] clickable cell error symbol\n- [interface] default theme color improvements, including a slight bg glow for cursor column\n- [aggregators] add keymin; handle corner cases #2308\n- many small interface improvements: error messages, clarifications, edge cases, guardrails. (thanks to @midichef for many of these)\n\n## Options\n\n- max_rows to stop loading early #2356\n- [xlsx] add xlsx_color_cells (default: True); set to False to disable xlsx cell colorizers\n- regex_skip is now replayable\n- disp_multiline_focus to only expand cursor row when multiline enabled #2205\n- disp_expert (int); set to 1 to enable advanced interface elements and unhide advanced columns by default; set to 5 (or greater) to disable input palette and help sidebars\n- pass disp_wrap_* to textwrap for multiline display #2506\n- rename note_* to disp_note_* #2381\n- separate color_longname into color_longname_status and color_longname_guide\n\n## Commands\n\n- addcol-histogram to make histogram from any numeric column #2208\n- exec-longname-simple for single-line (non-palette) input\n- go-col-name to go to a column by name\n- [dir] open-row-filetype to open row using the given filetype\n- [graph] plot-source-selected on ColumnsSheet #2424\n- [experimental] sort-selected-asc/desc #2295\n- deprecate view-cell #2381\n- unfurl finishes with column cursor on unfurled column\n\n## API\n\n- [command] addCommand(..., replay=False) to not add to cmdlog\n- [sort] add Sheet.ordering property as list[tuple[Column,bool]] #2295\n- [path] note nonstandard behavior of RepeatFile.read()\n- [input] allow different record kwarg for different fields in inputMultiple\n\n## Loaders\n\n- [msgpack] new loader #2419\n- [grep] new loader for output of grep/ripgrep #2443\n- [csv] display loading/saving progress\n- [csv] handle more dialect parameters from Sniffer\n- [csv tsv] remove default regex_skip of hash lines #2458\n- [eml] save email parts without filenames #2376\n- [fixed] skip separating space in fixed_width loader #2255\n- [json] remove column type inference #2475 #2131\n- [json] warn if some cols/sheets were not saved #2199\n- [markdown] use col_href as link for col\n- [ods] strip empty cells/rows from end #2440\n- [rec] roundtrippable\n- [sas] allow edits to source sheet\n- [save] derived null in text formats should be empty string #2476\n- [save] handle empty delimiter when options.safety_first\n- [text] do not strip lines of whitespace #2328\n- [tsv] allow empty regex_skip\n- [tsv] allow saving with NUL delimiters\n- [tsv] if field or row delimiter is NUL, disable regex skipping\n- [vds] allow sheets with no rows #2342\n- [xml] save XML as utf-8 instead of utf-8-sig #2520\n\n## Bugfixes\n\n- [command] addcol-sh: ensure arguments are properly quoted #2415\n- [selection] make toggle thread-safe for FreqTableSheet #1671\n- [undo] prevent undo of command that created sheet #2244\n- [undo] cmdlog always use rowidx, not rowkey #2246\n- [cliptext] fix onclick url #2466\n- [mouse] fix mouse-disable #2379\n- [sort] do not modify Sheet._ordering #2254\n- [bindings] do not re-prettykeys; call prettykeys for all bindkey() too #2247\n- [terminal] add no-op for ANSI focus-in and focus-in #2247\n- [sort] add _ordering to Sheet class instead of __init__ #2190\n- [transpose] allow to set values #2548\n- [sheet] make 20 tabs clickable #2030\n- [macro] do not record macro inner steps #2531\n- [filetype] use correct case for file extension determination #2263\n- [replay] stop stderr batch progress when interactive #2251\n- [replay] wait for replay to start before stderr progress #2251\n- [replay] fix rare duplication of replay cmds #2392\n- and many more\n\n# v3.0.2 (2024-01-15)\n\n### Fixes and minor improvements\n\n- [cli] add `--guides` to open the guide index\n- [cmdpalette] only first 10 suggestions should have shortcut keys #2242\n- [draw] use default bg on col hdr sep\n- [input] draw help sidebar on top of sheet/updater #2241\n- [graph] sort by x and y columns when diving (PR by @midichef #2226)\n- [guides] improve formatting of command and options in sidebar + guides\n- [replay] stop stderr batch progress when entering interactive mode #2251\n- [shell] clean up DirSheet sidebar\n- [sidebar] grab user's keystroke of sidebar-toggle for helpstring #2250\n- [sort] a better fix for maintaining sort ordering on sheet copies #2190 #2254\n- [tests] add \"test\" extras for installing PyPI packages needed to run tests\n\n# v3.0.1 (2024-01-07)\n\n### Fixes and minor improvements\n\n- [color] add `color_longname` to use instead of `color_keystrokes` #2219\n- [columns] add `displayer` attribute to saved column state\n- [cmdpalette] Enter executes first row, if partially typed out #2219\n- [cmdpalette] add sidebar for longname and aggregator palette #2219\n- [guide] add `show-command-info` to display command info for a keystroke #2228\n- [keys] add `*BtnUp` pretty keys for `BUTTON#_RELEASED` #2219\n- [dup-selected] dup-selected should unselect all rows in copied sheet #2225\n- [expr] fix KeyError crash with invalid inputs in `expr` for Python 3.12 #2179\n- [help] fix columns sheet sidebar\n- [loaders html] ignore parsing exceptions on invalid urls #2227\n- [sheets] limit end separators to rightmost visible column of sheet (PR by @midichef #2237)\n- [sidebar] fix fmtstr for case where source is BaseSheet #2239\n- [sort] maintain ordering on sheet copies #2190\n- [test] update unit tests to use packaged sample.tsv and benchmark.csv #2218\n- [threads] do not try to cancel already finished thread #2235\n- [tests] add `assert-expr` and `assert-expr-row` to evaluate Python expressions, and assert result is truthy\n- [tests] parametrize feature tests (PR by @ajkerrigan #2230)\n\n# v3.0 (2023-12-30)\n\n- [reorg] move independent modules into visidata/{features|experimental|themes}\n- [sidebar] add sidebar #2064\n - #1733 for full description/discussion\n - will contain a stack list of status messages\n - color syntax is [:bold]footext[/]\n - supports `[:code]`, `[:bold]`, `[:italic]`, `[:underline]`, `[:onclick]` (makes a clickable link), `[heading]`\n - [:] at terminal clears; [/] at terminal pops\n - supports basic markdown for colouring\n - toggle with `sidebar-toggle` (bound to `b`)\n - make bottom msg entirely clickable\n - make dedent and header parsing standard\n - add `options.disp_sidebar`\n - make sheet name clickable to open **Sheets Stack**\n - make options clickable\n - add `open-sidebar` (bound to `gb`) to open sidebar in new sheet\n- display sheet and feature help documentation in sidebar\n - added bundles of Guide Sheets\n - open Guide Sheets into a full VisiData sheet with `Enter`\n- added `options.disp_help` (default: 2), an integer and to set novice/expert mode. (#1961)\n When `disp_help` is:\n - -1: status messages aren't shown\n - 0: no help is shown\n - 1: sidebar sheet help is shown\n - 2: help is shown for the input widget on the sidebar (this was condensed and moved from the former input help panel) #2085 #1961\n - put input help on sidebar; `disp_help` from -1 to 3\n - mark options to be excluded if `disp_help` greater than `option.max_help`\n - add contextual help while editing options in **OptionsSheet**\n - `Ctrl+G` cycles input help sidebar\n- [guide] add GuideIndex toc and open-guide-index\n - add guides for macros, selection, errors\n- [filetype] add guesser to sniff filetype from data #130 #1759 #1881 #1880 #1883 #1978\n- add support for Python 3.12 #1934\n- migrate pkg_resources to importlib_resources (PR by @zormit and @anjakefala #1968 #1911)\n- [threads] no new threads while prev cmd still running #1148\n- [batch] add progress every half second to batch mode #1182\n- [replay] implemented a single-thread replay mechanism #1575 #2102\n - much simpler architecturally, but loses the ability to pause/resume, and show 1/N progress (now just shows number of queued commands remaining).\n - replay in general should be more stable and possibly faster.\n - Ctrl+N is now no-op for similar functionality to replay-advance\n - replay-advance command itself is removed\n - removed options.replay_movement and its functionality\n - options.replay_wait (-w on CLI) now works by setting the curses timeout, which will affect the display update frequency for all async commands. previously was hard-coded. try -w 0.001 while loading a big file to see a very rapid update.\n- add options.overwrite='n' mode #1805\n - replace `options.confirm_overwrite` with `options.overwrite`, which can be\n 'n' (no/never), 'y' (yes/always), 'c' (confirm)\n - add vd.confirmOverwrite()\n - inputPath/inputFilename will not suggest the existing value or complete filenames if overwrite not y or c\n - add --ro/--readonly cli flag (opposite of existing -y)\n - add [RO] readonly marker on right status after [M] modified\n- add command palette (PR by @moritz #2059 #247)\n - press Space to exec command by longname and bring up command palette\n - type to match by longname and/or description\n - uses fzf's fuzzymatch algo\n - press Tab/Shift+Tab to cycle through commands #2154\n - Enter to execute highlighted command\n - 0-9 to execute numbered command\n - `options.disp_cmdpal_max` for number of suggestions\n - `options.color_cmdpalette` for base color\n - `chooseAggregators` and `jointype` also use palette #1027 #2195\n - chooseAggregators still allows selection of multiple aggregators\n - `options.color_match` is color for matching chars in palette chooser\n- convert input history to use StoredList #2142\n - rename lastInputs.jsonl to input_history.jsonl #2142\n - remove `options.input_history`\n - will automatically record if `options.visidata_dir` exists\n- convert macros to use StoredList #2142\n - cannot save macros if `options.visidata_dir` does not exist\n - rename `options.visidata_dir`/macros.tsv' to `options.visidata_dir`/macros.jsonl'\n - add vd2to3.vdx script to port from 2.x macros to 3.x macros\n- [aggregators] sum uses start value from type of first value for Python 3.8+ #1996 #1999 #2009\n- [build] add a .desktop for VisiData #1738\n- [choose] add type for join/aggregators history #2075\n- [cli] add `-i` to run interactive mode after batch #1714\n- [clipboard] implement a more universal paste that uses positional columns #1377\n- [columns] add `setcol-precision-more` and `-less` #1609 #1650\n - works on float, floatsi, currency, date columns\n- [commands] allow space-seperated keystrokes #2067\n- [confirm] make yes/no buttons clickable #1740 #2075\n- [cosmetic] standardize `__repr__` for sheets and columns (PR by cool-RR #2091 #1757)\n- [describe] default width=10 for describe columns\n- [dir] set name of '.' to current dir name #1775\n- [dir] set name relative to previously loaded directory #1775\n- [dir] get default save name from sheet name #1775\n- [dir] rename `options.dir_recurse:bool` to `options.dir_depth:int` #1715\n- [display] add `resize-height-input` (`zv`) and `resize-height-max` (`gzv`) #1307\n- [encoding] use `options.save_encoding` for lsv, geojson, texttable savers\n- [encoding] change default options.encoding to utf-8-sig to detect/remove BOM #200 #908 #909 #1711\n- [expand] change default depth of expand-col(s)-depth to 0 (PR by @cool-RR #1809)\n- [expand] change sampling warning to aside #1054 #2066\n- [expand] ignore non-cursor types in mixed-type columns\n- [features] procmgr to view/manage processes, memory/cpu stats\n- [features] ping to traceroute a hostip\n- [features] add `addcol-histogram` #2052\n - display a histogram-like column for any column of ints\n- [features] add `contract-cols-depth` etc (bound to `g)`, `z)`, `gz)` #1695\n- [features] add contract-source-cols bound to ) #1702\n- [features] reload-modified calls `reload_rows`, adding support for tail #1686\n- [features] add `addcol-audiometadata` from vmutagen plugin\n - adds metadata column for audio files\n- [features] to **DirSheet** add `sysopen-mailcap`\n - opens using mailcap entry for current row, guessing mimetype\n- [features] add `addcol-source` to add current column to source sheet #988\n- [features] incorporate dedupe plugin by @jsvine\n - `select-duplicate-rows` selects each row in current sheet that is a duplicate of a prior row\n - `dedupe-rows` pushes a new sheet with only non-duplicate rows of current sheet\n- [features] incorporate normcol (by @jsvine)\n - `normalize-col-names` normalizes the names of all *non-hidden* columns\n- [features] add server to listen for commands on `options.server_addr` and `options.server_port` (if set)\n- [features] add Sheet.knownCols to preconfigure of columns by name #1488\n ```\n Sheet.knownCols.timestamp.type = date\n DirSheet.knownCols.directory.width = 0\n ```\n- [freq] add select-first command\n- [freq] base histogram width on column width #1807\n- [freq] set default disp_histogram to U+25A0 BLACK SQUARE (■)) (PR by @daviewales #1949 #1807)\n - [themes] ascii8 disp_histogram to *\n- [freq] add `open-preview` for split pane of source rows at cursor #1086\n- [graph] colorbrewer palette chooser (thanks @er1kb)\n- [graph] add commands to open external graph with matplotlib #1056\n- [help] remove `Enter` binding for **HelpSheet** to `exec-command`\n- [history] remove cmdlog_histfile\n- [input] change `Ctrl+G` to toggle `options.disp_help`\n- [input] add `Ctrl+N` to insert prettykeys of literal keystroke\n- [input] allow percentage of starting value for input into commands\n- [input] `inputMultiple` saves/remembers dict input\n- [input] `Tab`/`Shift+Tab` move left/right in edit-mode like excel #2169\n- [join] allow selecting of join columns from all columns sheet #1224\n - unbind `&` for **ColumnsSheet** `join-cols`\n- [join] join now works with typed values, not display values #2015\n- [join] improve warning for typing source key columns before joining #2117\n- [join] clarify ensureLoaded status message #2137\n- [keys] add shifted function prettykeys (Ctrl+Shift+F1 etc)\n- [layout] stop errors: hide-col on empty sheet, inputMultiple (PR by @midichef #1963)\n- [linux] change default system clipboard cmd to wl-copy if the user is using wayland (PR by @rj1 #1763)\n- [loaders] add `options.regex_skip` for text formats to allow e.g. comment skipping #1559\n - Added default values for `regex_skip` to existing source sheets like tsv/csv/lsv/json/jsonla.\n - Use --regex-skip='' (or otherwise set the option to '') to disable this behavior.\n- [loaders] add mailbox formats mbox/maildir/mmdf/babyl/mh loader (as supported by Python mailbox stdlib)\n- [loaders] .jrnl format (jrnl.sh) loader+saver\n- [loaders] add reddit API loader\n- [loaders] add matrix API loader\n- [loaders] add orgmode loader\n- [loaders] add scraper\n - table of HTML elements as parsed by `beautifulsoup4`\n- [loaders] add Parquet writer (PR by @mr-majkel #2053 #2044)\n- [loaders] add s3 loader (built by @ajkerrigan)\n - open Amazon S3 paths and objects\n- [loaders] add support for jsonla (JSONL arrays) format (PR by @daviewales #1730) #1726\n- [loaders] add zulip API loader\n- [loaders] add airtable API loader\n- [loaders] add .fec loader by @jsvine\n- [loaders] add f5log loader by @bitwisecook\n- [loaders] add a toml loader (PR by @ajkerrigan #1894 #1580 #1587)\n- [loaders] add .conll loader by @polm\n- [loaders html] display title/aria-label/caption/summary (PR by @midichef #2146)\n- [loaders http] replace requests with urllib #1808 #1704\n- [loaders http] guess filetype based on magic bytes #1760\n- [loaders jsonl] allow slash comments (PR by @geekscrapy #2025)\n- [loaders jsonl] deduce numeric column as float, not int #2131 #1698\n- [loaders png] use 2x2 unicode blocks instead of braille\n- [loaders shell] allow deleting of directories unless `options.safety_first=True` #1965\n- [loaders sqlite] save list/dict as json #1589\n- [loaders sqlite] add `exec-sql` command to input query #1719\n- [loaders xlsx] add cell colorizers from source #1718\n- [loaders xlsx] add `column_letter` to meta columns\n- [loaders xml] ignore comments\n- [macros] allow deleting of macro with commit on **MacrosSheet** #1569\n- [macros] add longname/keystrokes to **MacrosSheet** #1569 #1741\n- [macros] add help sidebar to end macro input #1569 #1741\n- [menu] move Edit>Add-rows to Row>Add\n- [menu] add `go-row-number` to menu #1766\n- [menu] move commit-sheet under File>Save\n- [menu] add resize-cols-input to Columns -> Resize (PR by @njthomas #1887)\n- [modules] include module name in Option/Command sheets\n- [motd] default motd is \"Support VisiData\" instead of blank\n- [mouse] onclick with url launches $BROWSER with url; add `displayer_url` #2031\n- [open] try using options.filetype for path #1710\n - useful for configuring default filetype when reading from stdin\n- [open] add `reopen-last-closed` which reopens the most recently closed sheet (PR by @cool-RR #1813) #1811\n- [open-syspaste] create new table from system clipboard #1680\n- [open-syspaste] enable filetype selection (PR by @daviewales #1717)\n- [options] add `option.json_ensure_ascii` (default: True) (PR by @joaosousa1 #1776) #1772\n - option for non-ASCII characters to be saved to JSON, on False will encode to utf-8\n- [path] change most uses of Path.name to Path.base_stem #2188\n - Path.name is the same as .base_stem for now\n- [plugins] remove external Plugins Sheet; only show installed plugins\n- [profile] dump profile to cwd when profiling enabled\n- [forms] provide help text for color, encoding, encoding errors, and regex #1961\n- [reload] support tail with `reload-modified` #1686\n- [reload] add `reload-rows` which preserves existing columns + cursor position #1655 #1683 #1663\n - [melt] refactor to support `reload-rows` #2101\n- [regex] use inputMultiple to allow changing regex_flags #1925\n- [regex] use `inputRegex` (which has regex help) for all regex commands\n- [rename-col] add `options.rename_cascade` #2088\n - if True, columns renames are cascaded into expressions\n- [replay] has been refactored to be sync, instead of a separate async process #1773 #1714\n- [save] add `options.save_encoding (default: 'utf-8') to differentiate from `options.encoding` when saving a file #1708\n- [save] add saver for STATA files (PR by @raffaem #1563)\n- [save] keep headers in txt as tsv if only one sheet and more than a single column #2173\n- [save] add status when save finished #2157\n- [selected] add `onclick` to \"selection status\" to quickly `dup-selected`\n- [sheet] add `select-equal-selected` (unbound) to select rows with values in current column in already selected rows #1327\n- [sheet] add `clean-names` (unbound) to set options.clean_names on sheet and clean visible column names\n- [sheet] remove left-click for sheets-stack #2030 #1656\n- [sheet] add `open-source` (bound to backtick) to open source sheet\n- [setcol-fake] add `setcol-fake` (unbound) adds a column of Faker generated 'faketypes'\n- [sparkline] add `addcol-sparkline` (unbound): adds a sparkline of all numeric columns\n- [status] downgrade sheet \"finished loading\" to debug\n- [tests] call all test_func(vd) defined in modules during pytest\n- [tests] run all unit tests in CI\n- [tests] add test for loading a directory #1798\n- [themes] add options.theme and visidata/themes directory of additional themes (light, ascii8, asciimono) #1682 #1691\n- [themes] keystrokes/code now with gray bg\n- [types] add `ipaddr` and `ipnet` types`. add `type-ipaddr` and `type-ipnet` commands (unbound) (PR by @ajkerrigan #1946 #1782 #1910)\n - also add `select-supernets` (unbound) which selects rows where the CIDR block value includes the input address space\n- [types] add `type-url` and `open-url` #2031\n- [types] add `type-datetime` #1572 #1380 #397\n- [ui] change menu, status, and other colors to be more visible\n- [ui] add `options.disp_scroll_context` to keep *n* more lines above/below cursor on screen\n- [ui] add version to menu status\n- [undo] options.undo can only be set globally\n- [usd] provide USD(s) function to convert string like '£300' or '205 AUD' to equivalent US$ as float\n- [windows] change default system clipboard command to clip.exe\n- [zip] add `sysopen-row` (`Ctrl+O`) to open file in `$EDITOR` #1708\n\n## experimental features (must be imported manually)\n\n- [diff] got moved to experimental\n- [inplace] optional replacement commands which update the new Column live as you write the expression\n- [livesearch] add `dup-search` and `dup-search-cols` which search for regex forwards, creating a duplicate sheet with matching rows live\n- [mark] mark rows to more easily move cursor to them\n- [noahs] add basic structure for Noah's Tapestry data game\n- [rownum] addcol-rownum and addcol-delta\n- [slide-cells] shift cells in cursor row to the right\n- add loaders gdrive and gsheets\n\n# deprecated\n\n- [dev] deprecate `col.setValueSafe` and `sheet.checkCursorNoExceptions`\n- [regex] deprecate `addcol-subst` and `setcol-subst`\n - `addcol-regex-subst` and `setcol-regex-subst` use inputMultiple, instead of more fragile search/replace one-line input\n- [regex] deprecate `split-col` and `capture-col`\n- [keys] change ScrollWheelUp to ScrollUp etc\n- [multiline] rename `visibility-sheet` to `toggle-multiline`\n- [utils] deprecate `onExit` context manager\n\n## bugfixes\n\n- [aggregators] use statistics.median for more correct median #1914\n- [aggregators] fix cancelling of long-running aggregators #1036\n- [canvas] fix clicks on labels and unplotted canvs (PR by @midichef #1984)\n- [canvas] put a max limit on y-axis label width (PR by @midichef #2177)\n- [chooser] choose only exactly matching strings (PR by @daviewales #1902)\n- [cli] support `options.encoding_errors` for stdin #2047\n- [clipboard] warn when pasting before copying (PR by @midichef #1793)\n- [clipboard] improve error when deleting row on empty sheet (PR by @midichef #2006)\n- [clipboard] save to tempfile, do not confirm\n- [clipboard] save as given filetype\n- [cliptext] fix double-width char display #1918\n- [cliptext] do not crash with x<0 #2138\n- [cliptext] do not crash with miniscule widths #2138\n- [cmdlog] check for empty cursor column when adding a column (PR by @midichef #1783)\n- [cmdlog] ensure record of global options in all cmdlogs\n- [cmdlog] do not log undos for non-loggable commands #1827\n- [colorizers] fix custom colorizers showing in sheet context #1225\n- [columns] speed up getMaxWidth for wide columns (PR by @midichef #1747) #1728\n- [columns] fix allColumnsSheet to ignore non-TableSheet\n- [confirm] remove flicker in alacritty #2040\n- [confirm] commit on sheet without source should always confirm\n- [currency] fix currency_neg option\n- [curses] ignore early keys pressed before curses is initialised if Esc present #1993\n- [cursor] cursorColIndex now returns None if empty #1803\n- [deps] add requests-cache submodule to root visidata #1748\n- [dir] support '..' and resolve dirname relative to CWD #1801\n - if user uses `open-dir-parent` outside of the CWD, switch to absolute paths\n- [display] fix visibility with col.height>1\n- [expr] more informative 'column not modifiable' error message #1764\n- [extensible] do not copy over existing attribute #2190\n- [fill] allow filling with values that are logically false (PR by @midichef #1794)\n- [freq] fix names for openRow #1777\n- [freq] correctly group null/error values for `options.numeric_binning` #1410\n- [graph] fix graph ranges for xmax, ymax < 1 (PR by @midichef #1752) #1673 #1697\n- [graph] fix data on edges being drawn offscreen (PR by @midichef #1850)\n- [graph] fix graph legend drawn too early (PR by @midichef #1980)\n- [graph] fixes to various graphing edge cases (PR by @midichef #1896)\n- [graph] fix top margin location and simplify y-coordinate calculation (PR by @midichef #1915)\n- [graph] labels: add tick symbol, int precision, right margin (PR by @midichef #1931)\n- [graph] fix legend display of full-width characters (PR by @midichef #1958)\n- [graph] widen left margin to hold y-axis labels (PR by @midichef #1998)\n- [graph] update fixPoint() to use inverted-y coordinates (PR by @midichef #2139 #2111)\n- [index] add longname for g< and g> #2011\n- [input] fix Ctrl+T swap on empty string #1684\n- [input] fix Ctrl+V with special keystrokes #1799\n- [input] erase status bar after prompt #1947\n- [input] fix toggle input help #1971 #1994\n- [input] include history for unfocused items #1947\n- [inputsingle-] loop until keystroke (do not timeout)\n- [join] fail if differing number of keycols #1678\n- [join] fix join-merge (PR by @yphillip #1923 #1843)\n- [jsonl] include all columns in first row, even if null\n- [keystrokes] only check duplicate prefixes from allPrefixes #1829\n- [keys] add `Shift+Tab`\n- [layout] fail if `hide-col` on empty sheet\n- [loaders fixed] use maxWidth for saving if larger than column width #1849\n- [loaders fixed] don't truncate wide columns with fixed width saver (PR by @daviewales #1890)\n- [loaders http] add `options.http_ssl_verify` to replace `options.http_req_verify` #1939\n- [loaders http] fix parsing link header (PR by @Midichef #1924 #1898)\n- [loaders html] fix failure from colspan with only td tags (PR by @midichef #2002)\n- [loaders html] prevent error when parsing an empty table (PR by @midichef #2140)\n- [loaders imap] enable imap and fix folder name extraction (PR by @justin2004 #1917)\n- [loaders json] include null columns in first row, in more cases (PR by @midichef #2109)\n- [loaders jsonl] save None as null #2183\n- [loaders mysql] unquote password before sending to client (PR by @dufferzafar #1933)\n- [loaders pandas] handle read methods that produce a list of dataframes (PR by @ajkerrigan #1990 #1986)\n- [loaders parquet] stringify source to handle both URLs and local paths (PR by @ajkerrigan #1913)\n- [loaders parquet] show string value for Parquet `large_string` (PR by @daviewales #2018 #2003)\n- [loaders parquet] fix parquet reading from zip or s3 (PR by @takacsd #2133)\n- [loaders parquet] handle parquet directories (PR by @mr-majkel #2160 #2159)\n- [loaders pivot] fix missing anytype import\n- [loaders postgres] quote schema and table name (PR by @isosphere #2129)\n- [loaders pyobj] similar sheet names for dive-/open- and pyobj- #1988\n- [loaders pyobj] do not skip properties that raise\n- [loaders png] fix `rgb_to_attr` to return str colornum\n- [loaders rec] support %sort; continue loading on exception #2022\n- [loaders sav] use fork of `savReaderWriter` for the sake of Python 3.10+ support #1867\n- [loaders sqlite] prevent creation of ./- file when reading from stdin (PR by @midichef #1945)\n- [loaders sqlite] explicitly fail when file is not on disk\n- [loaders tsv] use options.encoding for reading files\n- [loaders ttf] implement `closePath()` to draw missing lines (PR by @midichef #1979)\n- [loaders vds] save hidden columns also (PR by @pacien #2093 #2089)\n- [loaders vds] fix 'keyerror: exprcolumn' for .vds (PR by @pacien #2036 #2045)\n- [loaders vds] fix .csv to .vds conversion #2037\n- [loaders xlsx] saver now replaces illegal characters instead of aborting #1402\n- [loaders xml] silence FutureWarning from lxml (PR by @midichef #2149)\n- [loaders xml] fix file loading (PR by @midichef #2148)\n- [loaders yaml] support tuples in YAML files (PR by @cool-RR #1824) #1822\n- [loaders zip] fix loading and extracting files locally + remotely #2127\n- [macos] do not bind empty string to anything\n- [macros] add prompt for cancelling macro #1810 #1812\n- [macros] specify a clearer message #1810\n- [macros] append newline to macros.tsv if necessary #1569\n- [macros] save file as longname/keystroke #2084\n- [main] print version string once, not twice (PR by @midichef #1837)\n- [main] remove forced unload before interactive mode #1943\n- [menu] use \"Alt+x\" keybinding instead of \"^[x\"\n- [metasheets] do not use options.encoding for internal sheet saving\n- [misc] remove trailing commas from addCommand (PR by @midichef #1962)\n- [modify] do not call saveSheets on commit\n- [modify] commitMods do not call putValue for changes to added/deleted rows\n - also fix ItemColumn.putValue and AttrColumn.putValue to call parent\n Column.putValue before setting the value on the row\n- [modify] confirm() overwrite on root sheet source path\n- [modify] always set col.defer\n- [modify] do not fail on Column.putValue if no setter\n- [mouse] fix mouse-click on bottom pane\n- [open] silence ResourceWarnings for unclosed files (PR by @midichef #2152)\n- [options] disable adding rows (PR by @midichef #1944)\n- [options] fix option editing\n- [options] fix helpstr display when editing\n- [paste] add new rows to sheet if necessary\n- [path] set name to '.' for givenpath of '.' #1768\n- [path] fix progress bar for compression formats #1175 #1255\n- [path] fix open() using both binary and text mode (PR by @midichef #1955)\n- [path] binary mode does not take newline argument\n- [pyobj] fix pyobj-cell #2001\n- [regex] issue warning when no columns to add #1778\n- [regex] check for regex capture group #1778\n- [rename-col] skip gratuitous rewrites when undoing (PR by @pacien #2092)\n- [replay] turn off confirm dialogs during replay\n- [replay] clearCaches before moving cursor #1773\n- [replay] enable confirm in interactive batch mode (PR by @midichef #1751)\n- [replay] do not push sheet if not already pushed #1681\n- [save] handle saving 0 sheets #1720\n- [save] fix confirm message on commit #2090\n- [scroll-cells] do not error scrolling offscreen column #1908\n- [search] handle no rows and invalid regex (PR by @midichef #2125)\n- [settings] clear cache correctly before set\n- [sheets] fix NameError for mincolidx #1672\n- [sheets] add confirm for `quit-sheet-free` (PR by @midichef #1755)\n- [sheets] make sure addColumn called on all columns\n - not just dynamically-created columns\n - addColumn is needed to set .sheet and .defer, among other things\n- [sheets] fix reload() for tsv sheets with key columns (PR by @midichef #1997)\n- [sheets] fix recursion crash of Python >= 3.8, <3.9.10 (PR by #midichef\n #1722) #1696\n- [sheets] pop columns kwarg so raw list not set via final update() in\n constructor\n- [sheets] fix slowness from adding rows during getMaxWidth (PR by @midichef\n #1982)\n- [sheets] no longer insert column in draw() in debug mode (PR by @midichef\n #1995)\n- [sheets] fix errors on sheets with no columns (PR by @midichef #2124)\n- [sheets] fix splitcell to handle attribute/text pairs (PR by @ajkerrigan #2020)\n- [sheets] recreate sort columns for copied sheet (PR by @midichef #2192 #2190)\n- [shell] fix copying of a directory #1970\n- [status] fix Alt+Shift+Shift+X #1828\n- [status] update right status before exec #996\n- [status] add caller/module to statuses, and print on --debug #2037\n- [status] catch all exceptions during drawing #2174\n- [sort] show sort arrow for sort columns described by name (PR by @midichef #1876)\n- [syscopy] always copy as utf-8\n- [term] allow non-color term like vt102\n- [threads] allow @AsyncThread funcs to have status kwarg\n- [threads] remove non-sheet threads from unfinishedThreads (do not sync on them)\n- [quit-sheet-free] re-entering a subsheet left using quit-sheet-free should reload the subsheet #1679\n- [ui] integrate scrollfix from @geekscrapy #1441\n- [undo] ensure undo is sheet-specific for duped/copied sheets #1780\n- [undo] fix undo for first modification on a sheet-specific HelpSheet #1820\n- [undo] fix the removal of [M] (modified mark) after undo #1800\n- [undo] remove last matching cmdlog row, instead of first (PR by @midichef #2010)\n- [unzip-http] fix recursion error when fetching remote zipfile (PR by @midichef #2116 #2110)\n- [vdx] fix save error\n- [windows] add Alt+ keybindings for powershell #1630\n- [windows] limit windows-curses version to 2.3.0 (PR by @bartbroere #1901 #1841)\n - asottile noticed this was a regression in the last windows-curses release\n in this issue: zephyrproject-rtos/windows-curses#41\n- [windows] update windows-curses version to support Python 3.11 #2062\n- [windows] fix syspaste (PR by @midichef #1921 #1920)\n- [windows] fallback to `scr.getch` if no `scr.get_wch` #192\n - handles `AttributeError: '_curses.curses window' object has no attribute 'get_wch'`\n- [windows] tempfiles must be closed before reopening #2118\n- [zsh] fix zsh completion (PR by @dbaynard #1960 #1959)\n\n## performance\n\n- [startup] delay import of `urllib.request`, `pkg_resources`, and\n `dateutil`\n- [startup] remove `unittest.mock`\n- [draw] `drawcache` make_formatter\n- [test] use lambda instead of Mock for addstr\n- [] `Column.formatValue` inline fmtstr\n- [settings] `@lru_cache` sheet obj in `SettingsMgr._mappings`\n- fastpath getitemdeep\n - fallback to getattrdeep if len(row) < index\n- [sort] move `addProgress` outside of sortkey\n- [parquet] cache large strings and cap at 1MB #2003 #1068\n- [modify] check col/row before isChanged\n- [status] move getStatusSource into features to improve startup perf\n\n\n## api\n\n- [cli] printout gone; use `builtins.print`\n- [color] use `ColorAttr` throughout #2061 #2017\n - separate out fg/bg\n - allow bg and fg to take precedence independently\n - fixes issues with forced bg=black on sidebar for warning, and statusbar for working\n- [guides] add API for getting and adding guides\n - add vd,getGuide and vd.addGuide\n - add Helpers for formatting commands and options\n- [help] add HelpSheet to globals\n- [hint] add hint mechanism to find best `Sheet.hint_function`\n - add hints for types\n- [input] add vd.injectInput and vd.getCommandInput\n- [keys] use prettykeys for allPrefixes #1592\n- [menu] vd.addMenuItems with convenient string syntax\n- [modify] Sheet.commitAddRow and Sheet.commitDeleteRow\n- [modules] vd.importModule, vd.importSubmodules, vd.importExternal, vd.setPersistentOptions\n - add importExternal for most loaders and features #1739 #1765\n- [path] filesize can taken any Path-like\n- [pivot freq] re-add FreqTableSheet and PivotSheet to globals #1731\n- [sheet] add vd.addCommand as alias for BaseSheet.addCommand\n- [tests] add vd.resetVisiData\n- [vdx] runvdx() to execute vdx strings\n- rename vd.draw_sheet to vd.drawSheet\n- change order of parameters for vd.subwindow to (x,y,w,h)\n- change order of args to onMouse to x,y\n- add vd.aside for a silent status message\n- add GlobalsSheetsSheet to globals\n- vd.queueCommand can take input,sheet,row,col kwargs #1681\n- add `@visidata.stored_property` to persist property\n\n\n# v2.11.1 (2023-07-16)\n\n- [tests] fix tests for Python >=3.11\n- [path] update for Python 3.12 API (reported by @QuLogic #1934)\n\n## Improvements and bugfixes\n\n- [chooser] choose only exactly matching strings in chooser (PR by @daviewales #1902)\n- [columns] speed up `getMaxWidth()` for wide columns, and correct some edge cases (PR by @midichef #1747)\n- [freqtbl] Default `disp_histogram` to U+25A0 BLACK SQUARE (■)) (PR by @daviewales #1949)\n- [loaders fixed] do not truncate wide columns with fixed-width saver (PR by @daviewales #1890)\n- add missing import `copy`\n- [graph] fix graph ranges for xmax, ymax < 1 (PR by @midichef #1752)\n- [graph] fix data on edges being drawn offscreen (PR by @midichef #1850)\n- [input] fix `Ctrl+T` swap on empty input (reported by @gfrmin #1684)\n- [inputsingle] loop until keystroke (do not timeout)\n- [fill] allow filling with values that are logically false (PR by @midichef #1794)\n- [macos] do not bind empty string to any keybinding\n- [paste] add new rows to sheet if insufficient rows\n- [path Dirsheet] set name to '.' for givenpath of '.' (reported by @geekscrapy #1768)\n- [path] fix progress for compressed files (reported by @bitwisecook #1255 #1175)\n- [replay] clearCaches before moving cursor (reported by @mokalan #1773)\n- [save] handle saving 0 sheets (reported by @reagle #1266 #1720)\n- [settings] clear cache correctly before set\n- [undo] fix so that undo is Sheet-specific on copied sheets (reported by @geekscrapy #1780)\n- [undo] undoing `zd` now removes `[M]` (modification mark) (reported by @Freed-Wu #1800)\n\n# v2.11 (2023-01-15)\n\n- [ci] drop support for Python 3.6 (related to https://github.com/actions/setup-python/issues/543)\n- [ci] add support for Python 3.11 (#1585)\n\n- [dirsheet] add `open-dir-parent` (bound to backtick)\n- [join] add new \"concat\" jointype to behave similar to \"append\" but keeps first sheet type and columns (requested by @frosencrantz #1598)\n- [zip] add multisave for `.zip` (save each sheet in options.save_filetype format into given .zip file)\n- [sysedit] add `sysedit-selected` (bound to `g Ctrl+O`) (requested by @Delapouite #1596)\n - edit cells in multiple rows in `$EDITOR`\n - only handles cell modifications, not added or deleted rows\n\n\n## Improvements\n\n- [aggregators] add 95 and 99 percentile (p95 and p99)\n- [fill-col] speed up `fill-col` for sheets with many empty cells (PR by @midichef #1657)\n- [loaders hdf5] add support for arrays of scalars (requested by @linwaytin #1602)\n- [graph] fail if no numeric xcols are given\n- [open-cell-file] warn when file or url in cell does not exist (requested by @geekscrapy #1540)\n- [sqlite] add passthrough options (reported by @cwarden #1622)\n- [sqlite] add options.sqlite_onconnect to be executed before running any statement (requested by @cwarden #1622)\n- [xml] add passthrough options for xml_parser; default xml_parser_huge_tree=True (PR by @midichef #1668)\n\n## Bugfixes\n\n- [columns] `dup-sheet` now carries over attributes of columns added by `add-column`\n- [columns] **SettableColumn** should not be deferred (reported by @frosencrantz #1568)\n- [customdate] recognise type-customdate as numeric (requested by @tdussa #1613)\n- [describe] fix custom describe aggregators (reported by @edupont #1574)\n- [dirsheet] fix incorrect filename with multiple extensions (reported by @kunliugithub #1571)\n- [display] show `disp_oddspace` for surrogate escapes (reported by @geekscrapy #1544)\n- [graph] fix div-by-zero with only one y-value (reported by @midichef #1673)\n- [install] ensure setuptools files have appropriate permissions (reported by @icp1994 #1591)\n- [install] update data files in setup.py based on PEP 420 (reported by @Oblomov #1675)\n- [keystrokes] add `kDN` and `kUP` to translation table (reported by @djpohly #1336)\n- [loaders html] fix loading of relative links in html table (reported by @frosencrantz #1599)\n- [loaders xlsx] store `None` as empty string in `save_xlsx` (reported and PR by @dbaynard #1626 #1629)\n- [macros] override CLI parsing options for MacrosSheet (reported by @frosencrantz #1607)\n- [macros] query again for keystroke if used by existing macro (#1658)\n- [macros] do not include `nonLogged` commands in macro (reported by @geekscrapy #1569)\n- [macros] add reload for **MacroSheet** (reported by @geekscrapy #1569)\n- [menu] 2x ESC should exit menu\n- [mouse] fix mouse-clicks on statusbar when splitpane is off (reported by @frosencrantz #1625)\n- [numpy] fix loader\n- [open_txt] fix Exception with `open-config` when no `~/.visidatarc` (reported by @gunchev #1611)\n- [pdb] fix entering of pdb breakpoints for Python 3.9+ (reported by @jasonwirth #1317)\n- [sheets] sort all sheets on global **Sheets Sheet** (reported by @franzhuang #1620)\n- [types] format int/vlen as true int (reported by @xlucn #1674)\n- [unzip-http] fix file extraction (`x`) on remote zip file\n- [unzip-http] handle files smaller than 64K (reported by @frosencrantz #1567)\n- [zsh-completion] fixed (reported by @pigmonkey #1583; PR by @Freed-Wu #1646)\n\n## API\n\n- raise Exception from causes in utils.py (PR by @cool-RR #1633)\n- add `HistogramColumn` to allow overrides (requested by @andycraig #1621)\n- easier external numeric types with `@vd.numericType()` decorator (inspired by @s1291 #1394)\n\n- [frequency table] `dive-rows` renamed to `dive-selected`\n\n# v2.10.2 (2022-10-08)\n\n- add .vdx, a simplified new cmdlog format\n- add `-N`/`--nothing` command to disable loading .visidatarc and plugin addons\n- add `addcol-aggr` to add an aggregator column to the **FreqTable** without needing to\n regenerate it (requested by @geekscrapy #1541)\n\n## Improvements\n\n- [cli] load commandline file arguments from the start (requested by @reagle #1471)\n- [cli] `--config=''` now does not try to load any config\n- [open] rename `zo` `open-cell` command to `open-cell-file`\n- [loaders whl] load python .whl (reported by @frosencrantz #1539)\n\n## Bugfixes\n\n- [cli] fix for empty arg\n- [DirSheet] fix bug where `Enter` no longer opened a file from the **DirSheet** (reported by @frosencrantz #1527)\n- [input paste] fix pasting via a Path via `Ctrl+Y` into input (reported by @frosencrantz #1546)\n- [menu] allow VisiData to run without menu\n- [mouse] catch any curses.getmouse() errors (reported by @geekscrapy #1553)\n- [performance] allow vd to be truly idle (reported by WizzardUU #1532)\n- [plugins_autoload] catch error for environment having invalid package metadata (reported by @jsdealy #1529)\n- [plugins_autoload] catch exception if plugin fails to load\n- [plugins-autoload] fix check for if plugins_autoload is set in args\n- [plugins-autoload] update for importlib-metadata 5.0 API (reported by @jkerhin #1550)\n- [pyobj] undo rename of `open-row`/`open-cell` (were renamed to `open-X-pyobj`) (revert of eff9833e6A)\n- [sheets] ensure IndexSheets are precious, and that **SheetsSheet** is not (reported by @frosencrantz #1547)\n- [unzip-http] extracting a file now checks for overwrite (reported by @frosencrantz #1452)\n- [windows clipboard] fix piping to clip command through stdin (thanks @daviewales for the fix; reported by @pshangov #1431)\n\n## API\n\n- expose `CommandLogBase` (was `_CommandLog`)\n- [options] allow FooSheet.options instead of .class_options\n- add separate non-async `select_row`, `toggle_row`, and `unselect_row` for selection of single rows\n- the before/after decorators now do not fail if api functions they are decorating do not already exist\n\n# v2.10.1 (2022-09-14)\n\n## Improvements\n\n- [docs] document `-d` option (thanks @abitrolly for PR #1515)\n- [freq] disable histogram if `disp_histlen` or `disp_histogram` set to 0 or empty string\n- [guard] add `guard-sheet-off` which unsets `options.quitguard` on current sheet (thanks @hanfried for PR #1517)\n- [menu] add `BUTTON1_CLICKED` (same as `BUTTON1_PRESSED`)\n- [open] add `zo` to open file or url from path in current cell\n\n## Bugfixes\n\n- fix Guix build problems (reported by @ryanprior #1499)\n- add support for sheet names with multiple `.` (periods) in the name (requested by @geekscrapy #1494)\n- [cmdlog] add more portable shebang in vdj\n- [date] fix custom date greater than or equal to comparison\n- [macros] fix `macro-record` (#1513)\n- [macros] refresh `macro-sheet` upon macro addition\n- [macros] ensure macros are set upon startup\n- [plugins] update usd plugin api (thanks @hanfried for PR #1510)\n- [repeat] fix `repeat-` (#1513)\n- [status] reduce priority of active colouring (reported by @geekscrapy #804)\n\n## API\n\n- add `ExpandedColumn` to globals\n- add `Extensible.before` and `Extensible.after`\n - `def foo` decorated with `@VisiData.before` will run it before `vd.foo()`\n - `def foo` decorated with `@VisiData.after` will run it immediately after\n\n\n\n# v2.10 (2022-08-28)\n\n- [plugins] load all entry points in `visidata.plugins` group before config load\n - add entry_points={'visidata.plugins': 'foo=foo'} to plugin load plugin automatically when launching VisiData\n\n- [deps] require `importlib_metadata` >= 3.6\n - following https://github.com/pypa/twine/pull/732\n\n## Improvements\n\n- [draw] redraw only every 100 ms if any keys pending (requested by @ansoncg #1459)\n- [IndexSheet] shown name is only final name component\n- [loaders html] add table of all links on page (requested by @dufferzafar #1424)\n- [loaders html] `open-row` on **LinksSheet** to open url (requested by @dufferzafar #1424)\n- [options] add `options.http_req_*` to send headers/etc to requests (requested by @daviewales #1446)\n- [options pyobj] add `options.fmt_expand_dict` and `options.fmt_expand_list` for formatting expanded\n list and dict column names (requested by @joe-opensrc #1457)\n- [threads-sheet] add `z Ctrl+T` (`threads-sheet`) to open **ThreadsSheet** for current sheet\n- [threads-sheet] add `g Ctrl+C` (`cancel-all`) to **ThreadsSheet**\n- [zsh] add scripts for zsh completion (PR by @Freed-Wu #1455)\n - tutorial: https://visidata.org/docs/shell/\n\n## Bugfixes\n\n- [addcol-] set cursor at added column for `addcol-new`/`addcol-bulk` (reported by @jsvine #1445)\n- [cmdlog] `Ctrl+S` from a **CommandLog** now defaults to `.vdj` (reported by @jsvine #1443)\n- [display] format entire string for undetermined width (reported by and fixed by @jsvine #1442)\n- [formatter] fix len format strings\n- [LastInputsSheet] catch other exceptions during reload\n- [loader npz] fix .npz loader (reported by @Shahin-rmz #1440)\n- [loader geojson] fix plotting and saving geojson files (fixed by @mwayne #1477)\n- [loader geojson] improve feature property manipulation (fixed by @mwayne #1477)\n- [menu] upon menu item keypress, move to item (reported by @reagle #1470)\n- [menu] fix `ALT+` navigation while within menu (reported by @reagle #1470)\n - now requires two `ESC` to exit\n- [open] allow binary files from archives (reported by @frosencrantz #1430)\n- [save] do not save unknown filetype as `save_filetype`\n- [save visidatarc] only save rows on **OptionsSheet** to visidatarc\n- [sheets] fix name reconstruction for files with multiple and no suffixes (#1450)\n- [sheets] do not include empty name parts in sheet name\n- [unzip-http] **FreqTableSheet** `open-row` now loads links (reported by @frosencrantz #1458)\n- [zip] use correct rowdef in extract (reported by @frosencrantz #1430)\n- [zip] do not create directory for extract\n\n## snippets\n\n- add snippets/scrolloff.py which mimics vim's scrollof context lines (requested by @gennaro-tedesco #1441)\n\n## vdplus\n\n- `open-memusage` was moved to vdplus\n\n## API\n\n- add InferColumnsSheet\n - it infers the columns and their types from the rows it gets which are dicts\n - used by json, npy loader\n- add vd.printout and vd.printerr for builtins.print to stdout and stderr\n- add `vd.view()`\n- fix Extensible.init() to work with classes with no `__init__`\n- add `Sheet.sidebar` and `Sheet.sidebar_title` properties\n\n## Deprecated\n\n- remove VisiDataSheet\n- remove vdmenu\n\n# v2.9.1 (2022-07-21)\n\n- [unzip-http] move urllib3 to optional dependencies\n\n# 2.9 (2022-07-20)\n\n- [ux] add confirming modal dialog\n - only \"y\" required to confirm\n- add XDG support (thanks @jck for the PR #1420)\n - `options.config` default is now `\"$XDG_CONFIG_HOME\"/visidata/config.py` if `$XDG_CONFIG_HOME` is set and `config.py` exists. If not, falls back to the standard `$HOME/.visidatarc`.\n - vendor [appdirs.py](https://github.com/ActiveState/appdirs/blob/master/appdirs.py)\n- [cmdlog] support variables in .vdj (requested by @jungle-boogie #1364)\n - in the .vdj, write variables like so: `${variableName}`\n - then on the CLI: `vd -p foo.vdj variableName=bar`\n- [loaders arrow] new Apache Arrow IPC loader/saver (requested by @d-miketa #1369) (requires `pyarrow`)\n - add `.arrow` (file) and .arrows (streaming) formats\n - add more native `parquet` loader via `pyarrow`\n- preliminary \"windowing\" for referencing x rows before and y rows after in an expression (requested by @maxigit #1399, @MMesch #1129, @samuelludwig #1210)\n - press `w` (longname: `addcol-window`) followed by two numbers: the number of rows to aggregate *before* and *after* the current row.\n - there will be a new column, `foo_window`, where each row contains a list of aggregated rows. after that, e.g. `=` `sum(foo_window)` to get a running total for each row\n- add `setcol-format-enum` which takes e.g. `A=apple B=banana` and uses that as a mapping when formatting a column.\n- vendor `https://github.com/saulpw/unzip-http`; allows the downloading of individual files from a .zip file over http without downloading the entire archive (requires `urllib3` package)\n- add `save-source` to save a root sheet directly to its source\n- add `setcol-formatter` to specify formatting function used for Column (default: `generic` formats based on type and fmtstr). Can be `json` or `python` or a custom formatter\n\n\n## Improvements\n\n- [cli] when `-v` or `-h` VisiData now does not read config or do anything else (requested by @geekscrapy #1340)\n- [cmdlog] set `.vdj` to be the default cmdlog format\n- [replay] allow column names to be numbers (reported by @frosencrantz #1309)\n - if wishing to reference a column index, required to be an int in a .vdj cmdlog\n- [cmdlog] when saving cmdlogs, type column indices as integers, and column names as strings\n - If replaying, and *col* is an `int`, the CmdLog will index by position.\n - If *col* is a `str` it will index by column name.\n- [display] preview first n elements of a list/dict cell\n- [regex] add unbound `addcol-` commands\n- [rtl] improvements to right-to-left text display (requested by @dotancohen #1392)\n- [man] have `vd --help` open the .txt manpage by default (requested by @halloleo #1332)\n- [mouse] invert scroll wheel direction (requested by @marcobra #1351)\n- [performance] improvements to plotting of empty canvas, multiline display, and draw-ing functions\n- [plugins] notify when plugin update available (thanks @geekscrapy for PR #1355)\n\n## Bugfixes\n\n- [aggregators] fail on setting an unknown aggregator in **Columns Sheet** (reported by @geekscrapy #1299)\n- [aggregators] handle `delete-cell` case for aggregators column in **Columns Sheet** (reported by @geekscrapy #1299)\n- [aggregators] fix quartile aggregators (reported by @pnfnp #1312)\n- [aggregators] fix copying of aggregators when duplicating a sheet (reported by @frosencrantz #1373)\n- [canvas] do not use \"other\" label when there are exactly 9 columns being plotted (reported by @tdussa #1198)\n- [cli] fix `+:subsheet:col:row:` when `load_lazy` is False\n- [delete-row] clear deleted rows from `selectedRows` (reported by @geekscrapy #1284)\n- [exec-longname] output warning, if longname does not exist\n- [expr] prefer visible columns over hidden columns (reported by @frosencrantz #1360)\n- [freeze-sheet] carry over column attributes for freeze-sheet (reported by @frosencrantz #1373)\n- [import-python] use command-specific history (reported by @frosencrantz #1243)\n- [IndexSheet] fix renaming of sheet names from an IndexSheet (reported by @aborruso #1339)\n- [input] handle history for non-string input values (reported by @frosencrantz #1371)\n- [loaders pandas] fix (`dup-selected`) `\"`of selected rows for **Pandas Sheet** (reported and fixed by @jasonwrith #1315 #1316)\n- [loaders usv] swap delimiters (reported by @frosencrantz #1383)\n- [loaders usv] save delimiter override options (reported by @frosencrantz #1383)\n- [loaders usv] fix saving header with usv row delimiter (reported by @frosencrantz #1383)\n- [loaders xlsx] fix clipboard on XlsxSheets (reported by @jungle-boogie #1348)\n- [macros] fix macro-record keystroke setting (reported by @fatherofinvention #1280)\n- [mouse] stay disabled after input (reported by @holderbp #1401)\n- [mouse] fix for pypy3 (thanks @LaPingvino for PR\n- [quitguard] refinement of quit-sheet protection (reported by @geekscrapy #1037, #1381)\n- [save-selected] get sheet names for saving from selected rows (reported by @aborruso #1339)\n- [shell] strip trailing whitespace in `z;` output (reported by @justin2004 #1370)\n- [tty] fix bug where piping async output into stdin broke visidata keyboard input (reported by @ovikk13 #1347)\n- [undo] fix issue where undoing a reload blanks the current sheet; do not set undos for reload sheet (#1302)\n- [unset-option] fix issue where Exception is raised on the next undo-able command run after `unset-option` (reported by @ajkerrigan #1267)\n- [windows] require `windows-curses` installation on Windows (thanks @ajkerrigan for PR #1407; reported by @schiltz3 #1268, @aagha #1406)\n\n## API\n\n- add Column.formatter (generic, json, python)\n- add SqliteQuerySheet to globals\n- `vd.loadConfigFile()` no longer needs a filename argument, and will use `options.config` by default (#211)\n- use `newline=\"` for csv.writer (thanks @daviewales for PR #1368)\n- make `ItemColumn` a proper class for inheritance\n- add `openJoin` and `openMelt` to allow overriding by plugin sheetsS\n- addColumn takes `*cols` (reported by @pyglot #1414)\n\n## Deprecated\n\n- deprecate old vdmenu system\n - remove `Shift+V` command\n\n# 2.8 (2021-12-15)\n\n## Improvements\n\n- [plugins] include pip stderr in warning\n- [plugins] use returncode to determine if pip install failed, before adding to imports (thanks @geekscrapy for PR #1215)\n- [cmdlog] add sheet creation command to cmdlog (requested by @aborruso #1209)\n- [open] strip whitespace from the beginning and end of inputted filenames\n- [options] `options.input_history` and `options.cmdlog_histfile` can now be an absolute paths (requested by @geekscrapy #1200)\n - relative paths are relative to `options.visidata_dir`\n- [splitwin] automatically switch to pane where sheet is pushed to\n\n## Bugfixes\n\n- [curses] suppress invalid color errors in Python 3.10 (thanks @ajkerrigan for reporting #1227 and for PR #1231)\n - Adapt to [Python 3.10 curses changes](https://docs.python.org/3/whatsnew/3.10.html#curses) which can raise a `ValueError` on invalid color numbers.\n- [curses cosmetic] simplify error message, if curses fails to initialise\n- [loaders json] skip blank lines in json files, instead of stopping at them (thanks @geekscrapy for PR #1216)\n- [loaders jsonl] fix duplicate columns when loading fixed columns sheets in jsonl format (report by @0ceanlight)\n - example of formats with fixed columns is darkdraw's `DrawingSheet`\n- [loaders fixed] fix saver (thanks @geekscrapy for PR #1238)\n- [loaders postgres] fix recognition of postgres loader (reported by @ryanmjacobs #1229)\n- [loaders sqlite] fix the loading of sqlite VIEWs for sqlite version 3.36.0+ (reported by @frosencrantz #1222)\n- [help-commands] now lists commands only for the current sheet (reported by @geekscrapy #1217)\n- [textcanvas] ENTER on canvas should push copied source sheet for points within cursor\n- [pivot freq] use `options.histogram_bins` from source sheet\n- [curses cosmetic] fix issue where if a curses initialisation Exception is called, a second Exception follows\n- [quit-sheet-free] fix bug where quit-sheet-free, when multiple sheets opened in CLI, was not working (reported by @geekscrapy #1236)\n- [options] fix instance where local options sheet was called, instead of global options sheet (thanks @geekscrapy for PR #1241)\n\n## API\n\n- add standard Python `breakpoint()` to drop into the pdb debugger\n- export `run()` to global api\n- add CsvSheet, ZipSheet, TarSheet to global api (thanks @geekscrapy for PR #1235)\n\n# 2.7.1 (2021-11-15)\n\n- Bugfix: fix Enter on helpmenu (reported by @geekscrapy #1196)\n\n# 2.7 (2021-11-14)\n\n## Improvements\n\n- [movement] bind Home/End to go-top/go-bottom (thanks @geekscrapy #1161)\n- [api] add vd.urlcache as alias for urlcache global (thanks @geekscrapy for PR #1164)\n- [plugins] do not continue installation if main package fails pip install (thanks @geekscrapy for PR #1194)\n- [plugins] allow for plugin records without SHA256; warn if absent (thanks @geekscrapy for PR #1183)\n- [load_lazy] do not load subsheets, if `sheet.options.load_lazy` is True (thanks @geekscrapy for PR #1193)\n- [save] confirm when `save_foo` function does not exist and saver fallsback to `options.save_filetype` (reported by @geekscrapy #1180)\n- [save] `options.save_filetype` default now 'tsv'\n- several cosmetic improvements\n\n## Loaders\n\n- [lsv] add `lsv` filetype for simple awk-like records (requested by @fourjay #1179)\n- [ods] add `odf` filetype for Open Document Format spreadsheets\n- [xlsx] add extra columns (`cellobject`, `fontcolor`, `fillcolor`) if `options.xlsx_meta_columns` (default False) (thanks @hoclun-rigsep for PR #1098)\n- [sqlite] allow query/insert (no modify/delete yet) for `WITHOUT ROWID` tables (requested by @stephancb #1111)\n\n## Bugfixes\n\n- [savers compression formats] fix corruption when saving to compression formats (#1159)\n- fix \"ModuleNotFoundError: no module named 'plugins'\" error on startup (#1131 #1152)\n- [windows] fix issue with Enter key on Windows (reported by @hossam-houssien #1154)\n- [draw] fix multiline rows by making height fixed for all rows (reported by @geekscrapy #916)\n- [DirSheet] fix bug where fix key column sheets (e.g. DirSheet, SqliteIndexSheet) keycols were not being saved in batchmode (reported by @geekscrapy #1181)\n- [async] make sure all threads started on sheet are cancelable (reported by @geekscrapy #1136)\n- [AttrDict] fix bug with setting value on nested AttrDict\n- [dup-X-deep] fix error with async_deepcopy (thanks @pstuifzand for fix)\n- [join] fix 'inconsistent-keys' issue when joining between XlsxSheet with typed columns and CsvSheet with untyped columns (reported by @davidwales #1124)\n- [sqlite] handle sqlite column names with spaces (thanks @davidskeck for PR #1157)\n- [sqlite] use `options.encoding` and `options.encoding_errors` for decoding of sqlite db text (reported by @WesleyAC #1156)\n- [xlsx] add handling for EmptyCell instances (thanks @hoclun-rigsep for PR #1121)\n- [xlsx] gate sheet name cleaning on `options.clean_names` (reported by @davidwales #1122)\n- [macos] fix bindings for `Option`+key\n- [random-rows] fix import (reported by @geekscrapy #1162)\n- [save-selected] better default save filename (reported by @geekscrapy #1180)\n- [save] fix bug where saving multiple sheets to a single non-embeddable format did not result in fail (reported by @geekscrapy #1180)\n- [slide] fix Shift slide-down and Shift slide-up with arrow keys (reported by @a-y-u-s-h #1137)\n- [replay] fix replay where `join-sheets` operation hangs (reported by @agjohnson #1141)\n- [undo] no more KeyError when Undoing modifications (reported by @geekscrapy #1133)\n- [unfurl-col] fix unfurl-col on cells containing exceptions (reported by @jsvine #1171)\n\n# 2.6.1 (2021-09-28)\n\n## Bugfixes\n\n- [editor] fix sysopen-row (thanks @frosencrantz #1116)\n- [loaders fixed] fix saver (#1123)\n- [loaders shell] fix copy-files\n- [loaders sqlite] fix import error on exception (thanks @jsvine #1125)\n\n# 2.6 (2021-09-19)\n\n## Major feature\n\n- [menu] new hierarchical menu system\n - `Alt+F`, `Alt+E`, etc to open submenus (`Alt+` underlined capital letter in toplevel menu)\n - `Ctrl+H` to activate Help menu (manpage now at `gCtrl+H`)\n - `q` or `Esc` to exit menu\n - Enter to expand submenu item or execute command\n - or left mouse click to activate and navigate menu\n - only show commands available on current sheet\n - sheet-specific commands highlighted with `options.color_menu_spec`\n - new options:\n - `disp_menu`: display menu if inactive (default True). Can still activate menu with Ctrl+H/Alt+F\n - `disp_menu_keys`: whether to display shortcuts inline (default True)\n - `disp_menu_fmt`: upper right display on menu bar (like `disp_status_fmt`/`disp_rstatus_fmt`)\n - theme colors: `color_menu` `color_menu_active` `color_menu_spec` `color_menu_help`\n - theme chars: `disp_menu_boxchars` `disp_menu_more` `disp_menu_push` `disp_menu_input` `disp_menu_fmt`\n\n## Interface changes\n\n- [expand-col] only expand to one level\n- [slide] remove slide row/col with mouse\n- [macos] add bindings for Option+key to Alt+key\n- [modified] limit use of sheet protection (thanks @geekscrapy #1037)\n- [python] rebind g^X to new import-python command (what exec-python was mostly used for)\n- [npy] add `npy_allow_pickle` option (default False)\n- [join] rename join-sheets on IndexSheet to join-selected; bind both g& and & to join-selected\n- [loaders pandas] add error message for unpickling non-dataframes\n- [join] fail if no key columns on any sheet (thanks @geekscrapy #1061)\n- [loaders xlsx] enable access to cell metadata (thanks @hoclun-rigsep #1088)\n- many performance, progress bar, and UI responsiveness improvements\n\n## Bugfixes\n\n- [cli] issue warning if +sheet-position not found (thanks @geekscrapy #1046)\n- [clipboard] do not copy newline for syscopy-cell (thanks @geekscrapy #1064)\n- [column] detect existing column by row key instead of column name (thanks @geekscrapy #1058)\n- [color] set `color_current_row` to the same precedence as `color_current_column` (thanks @frosenrantz #1100)\n- [command] do not fail/abort on unknown command\n- [draw] Sort indicator on top of More indicator (thanks @geekscrapy #1071)\n- [join] fix multiple extend (thanks @cwarden)\n- [join] allow extended columns to be modified (thanks @cwarden)\n- [join] fix for rowdefs without bool (like pandas)\n- [loaders dirsheet] continue after exception in copyfile\n- [loaders fixed] fix fixed-format saver\n- [loaders fixed] save uses `global options.encoding` (thanks @geekscrapy #1060)\n- [loaders mysql] do not stop loading on first error (thanks @SuRaMoN #1085)\n- [loaders pandas] fix column rename\n- [loaders sqlite] save based on column names, not position\n- [loaders sqlite] allow changing value of cells that were NULL (thanks @mattenklicker #1052)\n- [loaders sqlite] add message on not currently supporting WITHOUT ROWID (thanks @stephancb #1111)\n- [multisave] fix breaking typo\n- [open_txt] load new blank sheet for 0 byte files (thanks @geekscrapy #1047)\n- [save] do not set a default for `options.save_filetype` (thanks @frosencrantz #1072)\n- [split-pane mouse] activate pane on click (thanks @frosencrantz #954)\n- [unfurl] handle unfurling exceptions (close #1053)\n- [quitguard] confirm quit when set on a specific sheet even if not precious or modified\n- [yaml] Fix yaml loader traces on no attribute `_colnames` (thanks @frosencrantz #1104)\n- [visidatarc] catch all visidatarc exceptions upon load\n\n# v2.5 (2021-07-08)\n\n- [social] #visidata has moved off of freenode to libera.chat\n- [deps] required pandas version for the pandas loader has been bumped to at least 1.0.5\n- [caa] new PR submitters required to sign CAA\n\n## Features\n\n- [cli] when no arguments on commandline, open currentDirSheet (previously vdmenu); -f opens empty sheet of that filetype\n- [clipboard] bind `x` family to `cut-*` (thanks @geekscrapy #895)\n- [date] add specialized comparators for `datetime.date` (thanks @aborruso #975)\n - visidata.date now compares to datetime.date (previously raised exception)\n - identical dates compare equal even if intra-day times are different\n - this does not work for incompletely specified visidata.date; e.g.\n `visidata.date(2016, 10, 29, 4, 0, 0) != visidata.date(2016, 10, 29)`\n- [DirSheet] add y/gy to copy file(s) to given directory\n- [loaders vds] save non-jsonable cells as string (thanks @pacien #1011)\n- [loaders zstd] support loading zstd-compressed files (thanks @lxcode #971)\n- [movement] bind `Ctrl+Left/Right` to `go-left`/`right-page` (thanks @davidwales #1002)\n- [options] save to foo.visidatarc from OptionsSheet (thanks @njthomas #958)\n- [sqlite] RENAME and DROP tables from SqliteIndexSheet\n- [unfurl] add `options.unfurl_empty` to include row for empty list/dict (thanks @frosencrantz #898)\n- [quitguard] confirm quit/reload only if sheet modified (references #955, #844, #483; thanks @jvns, @frosencrantz)\n\n## Improvements\n\n- [addRow] advance cursor if row inserted before cursor\n- [archive] add .lzma as alias for .xz\n- [clipboard] gzp pastes None if nothing on clipboard\n- [clipboard] make syspaste async\n- [clipboard] bind `zP` to syspaste-cells and gzP to syspaste-cells-selected (thanks @jvns and frosencrantz #983, #990)\n- [cliptext] better support for combining and variant chars (thanks @lxcode #758 #1034)\n- [colors] reduce color swatch size to remove flashing (thanks @frosencrantz #946)\n- [encoding] specify encoding explicitly for all Path.open_text (thanks @pacien #1016)\n- [error] exceptionCaught(status=False) to add to status history, but not post to status (thanks @frosencrantz #982)\n- [freqtbl] copy fmtstr from source col to aggcol (thanks @geekscrapy #1003)\n- [help] ENTER/exec-command to execute command on undersheet (thanks @geekscrapy #1011)\n- [help] add `all_bindings` hidden column (thanks @frosencrantz #896)\n- [inputs] put reused input at end of lastInputs (thanks @geekscrapy #1033)\n- [loaders json] streamify save to .json\n- [loaders npy] add `npy_allow_pickle` option, default False\n- [loaders tsv] increase bufsize to improve loader performance by 10%\n- [path] all Path.open track Progress via read/filesize (thanks @jspatz #987)\n- [path] add Progress for opening compressed files\n- [path] implement line-seek operations (thanks @pacien #1010)\n- [regex expand] deprecate `options.expand_col_scanrows`; standardize on `options.default_sample_size` (thanks @jsvine)\n- [regex] \"match regex\" to \"capture regex\" (thanks @geekscrapy #1032)\n- [shell] `addcol-shell` pass command to $SHELL (thanks @juston2004 #1023)\n- [shortcut] allow shortcut for jump-sheet to be settable\n- [splitwin] push sheet in empty pane iff splitwin\n- [stdin] use cli --encoding option for piped data (thanks @pacien #1018)\n- [undo] remove undo for reload (replaced with quitguard+confirm)\n- [quit] add Shift+Q/quit-sheet-free to quit and free associated memory (thanks @cwarden)\n\n## Display\n- [canvas] add `options.disp_canvas_charset` to change displayed chars (thanks @albert-ying #963)\n- [canvas] use sheet specific options for draw\n- [disp] format list/dict as [n]/{n} only for anytype\n- [save] iterdispvals(format=True) convert None to empty string\n\n## Bugfixes\n\n- [batch] ensure quitguard is off during batch mode\n- [canvas[ fix error on dive into cursor including y-axis\n- [cli] have an actual error if there is a missing argument for final option\n- [cli] do nothing (no error) if no sources given\n- [clipboard] fix zy/gzp regression (thanks @sfranky #961)\n- [clipboard] syscopy-cell do not include column name\n- [cmdlog] fix bug where customising replayable options in Options Sheet led to issues opening metasheets (thanks @jsvine #952)\n- [cmdlog] fix bug where cmdlog records new sheet name, instead of old sheet name for `rename-sheet` (thanks @aborruso #979)\n- [color] track precedence so colorizers apply over `color_current_row`\n- [color] determine color availability with `init_pair`\n- [color] do not break on nonsense color\n- [column] getitemdeep/setitemdeep get/set dotted item key if exists (thanks @frosencrantz #991)\n- [column] fix bug where hard crash occurs when cursor on cell of SheetsSheet is on cursorDisplay (thanks @frosencrantz #1029)\n- [curses] add default `vd.tstp_signal` for non-cli users\n- [execCommand] warn gracefully if bound command longname does not exist\n- [expr] setValuesFromExpr do not stop processing on exception\n- [join] fix when keys have different names (thanks @aborruso #964)\n- [join] fix for rowdefs without bool (like pandas)\n- [join] fix multiple extend (thanks @cwarden for reporting)\n- [loaders fixed] fix editing in final column for fixed-width load (thanks @mwayne #974)\n- [loaders geojson] do not abort plot if rows have errors\n- [loaders html] add columns even if not in first row\n- [loaders pandas] fix column rename\n- [loaders rec json] fix adding new columns for json and rec loaders (thanks @ajkerrigan #959)\n- [loaders postgresql] add postgresql scheme (fixes #966) (thanks @zormit #967)\n- [loaders sqlite] fix saving deleted cells (thanks @mattenklicker #969)\n- [loaders vds] save SettableColumn as Column (thanks @pacien #1012)\n- [loaders zip] fix extract-selected-to\n- [open] fix regression where opening blank sheets of type tsv, csv, txt, etc was not working\n- [plugins] fix stdout/error from plugins installation message (was in bytes, changed to str)\n- [quit] remove sheets from **Sheets Sheet** upon quit\n- [save-col] fix inputPath error (thanks @savulchik #962)\n- [shell] fix `options.dir_hidden`; also apply to dirs when `dir_recurse`\n- [textsheet] fix reload after `^O` sysopen\n\n## vdplus\n\n- moved clickhouse, vsh, vgit, windows to vdplus\n\n\n# v2.4 (2021-04-11)\n\n- [splitwindow] stabilize sheet stack associations\n - `Shift+Z` pushes 'under sheet' (if any) onto other stack\n - `Shift+Z` does not swap panes anymore\n - `g Tab` swaps panes\n - `options.disp_splitwin_pct` is always not sheet-specific\n\n- [status] show nSelectedRows on rstatus\n\n- [color] remove `options.use_default_colors` (thanks @lxcode #939)\n - `options.color_default` can now have both fg and bg\n - other color options which do not specify fg or bg will use the missing component from `color_default`\n - to use terminal default colors, set `options.color_default=\"\"`\n\n## Bugfixes\n\n- [loaders gzip] fix progress bar when opening gzip (thanks @geekscrapy #925)\n- [loaders http] fix loading files from url without specifying filetype\n- [loaders sqlite] use `TABLE_XINFO` for hidden/virtual columns (thanks @dotcs #945)\n- [loaders sqlite] perf improvement: do not pre-count rows (required full table scan)\n- [loaders vds] save typed values instead of formatted display values (thanks @frosencrantz #885)\n- [loaders xlsx] stringify \"header\" row values for column names (thanks @davidwales #921)\n- [pyobj-show-hidden] grab visibility lvl from sheet specific option (thanks @frosencrantz #947)\n- [splitwin] prevent flickering-on-full-window\n- [splitwin] if top sheet quit, keep bottom sheet in bottom pane\n- [splitwin] full-screen/splitwin close all sheets should be part of the same stack\n\n# v2.3 (2021-04-03)\n\n## Features\n - [colors] allow background colors (thanks @frosencrantz #435)\n - use \"*fg* on *bg*\" e.g. \"212 yellow on 14 red\"\n - \"bg *bg* fg *fg*\" (or reversed)\n - attributes always apply to foreground regardless of position in colorstr\n - as before, only the first valid color in a category (fg/bg) is used; subsequent color names (even unknown ones) are ignored\n - allocate colors on demand, instead of \"all\" 256 colors as fg\n - **Colors Sheet** now only includes colors actually allocated\n - [colors] set `use_default_colors` default to `True` (was `False`)\n - [delete] do not move deleted values to clipboard (thanks @geekscrapy #895)\n - `delete-*` commands are changed to not alter the clipboard\n - the previous `delete-*` commands are renamed to `cut-*` (unbound)\n - this affects: `delete-row`, `delete-selected`, `delete-cell`, `delete-cells`\n - [jump-first] bound `g^^` to cycle through sheets\n - [null] `zd` / `gzd` `delete-cells` set to `options.null_value` instead of `None`\n - [memories] add MemorySheet on `Alt+M`, `Alt+m` adds current cell to sheet (thanks @UrDub and @geekscrapy #912)\n - useful for storing values to reference later\n - both names and values can be edited on MemorySheet\n - [aggregator] `memo-aggregator`(z+; formerly called `show-aggregate`) adds value to memory sheet\n - [clipboard] clipboard stored on memory sheet; zy/zp use vd.memory.clipval;\n - [plugins] allow install from github url to local pip repo\n - [plugins] add darkdraw to plugins.jsonl\n - [png] save image as RGBA\n - [pyobj-expr] `Ctrl+X` within `Ctrl+X` input suspends directly into python REPL\n - [splitwin] now involves two different sheetstacks that build and quit independently (thanks @lamchau #894)\n - [splitwin] allows stickier panes for push/quit\n - [splitwin] splitwin-half (`Z`) swaps panes if already active\n - [splitwin] only re-split (with `zZ`) if sheets are not already split, otherwise adjust split percent\n - [save_filetype] if `save_ext` does not exist, or if `options.save_filetype` is different from default, use `options.save_filetype`\n - [vdplus] auto-import, ignore if not available\n\n## Bugfixes\n - [aggregator] fix typo in deciles description (thanks @cwarden #922)\n - [copy] copying BasicRow (new sheets), now does not error (still blank)\n - [cmdlog] for `open-file` source logging in cmdlog, we want paths to physical files, so if src is a **Sheet** grabs its source\n - [defer] fix pasting in deferred sheets\n - [eval] fix **ExprColumns** on empty rows\n - [help] move signal config earlier in runcycle, to accommodate --help (thanks @frosencrantz #926)\n - [open] create blank sheet of appropriate type when path does not exist\n - [pandas] fix conflict between dropped index and existing column (thanks thomanq #937)\n - [plugins] only check for plugins.jsonl once daily (previously: every start-up)\n - [pivot] fix `openRow`\n - [pivot] fix bug with sheet name\n - [png] fix saving directly from canvas\n - [sort] fix sorting of visidata.Path objects (thanks @frosencrantz #897)\n - [splitwin] fix cursor behaviour on both panes when active\n - cursor movement on inactive panes is blocked\n - [SuspendCurses] workaround for bug in curses.wrapper (thanks @frosencrantz #899)\n - [undo] do not set undo for a `commit-sheet`\n\n## Api\n - [addRows] addRows(rows, index, undo) adds rows at index, sets undo if True\n - set undo to False, if using addRows within an addUndo function\n - [deleteBy] add an undo flag to deleteBy\n - [clipboard] change `cliprows` to be a simple list of rows\n - new **DrawablePane** super-base class\n - [json] rowdef now **AttrDict** for massive convenience\n\n# v2.2.1 (2021-02-07)\n\n## Bugfixes\n - [setcol-fill] use row identity to identify selected rows (thanks @frosencrantz, #884)\n - for jsonl, empty rows are identical ({}), and if ones is selected, previously it would result in all of them being filled.\n - also, fill with most recent *non-null* value\n\n## man\n - add a manpage visidata.1\n - fix typo\n\n# v2.2 (2021-01-30)\n\n## Options\n\n - [cli options] now global by default; use `-n` to set option as sheet-specific instead\n - add `-n`/`--nonglobal` to make subsequent CLI options \"sheet-specific\" (applying only to paths specified directly on the CLI)\n - keep `-g`/`--global` to make subsequent CLI options \"global\" (applying to all sheets by default unless overridden)\n - invert the default: now CLI options are global by default (thus `-g` is a no-op unless preceded by `-n` on the CLI)\n - `-g` no longer acts as a toggle\n\n - [input] add `options.input_history` (thanks @tsibley and @ajkerrigan #468)\n - basename of file to store persistent input history (default of `''` means disabled)\n - caveat: persistent file only read if option given before first input\n\n - [options.fancy_chooser] now disabled by default--use `Ctrl+X` to open from a choose() prompt\n\n## Types\n\n - [types] add `floatlocale` type (thanks @Guiriguanche #863)\n - add commands `type-floatlocale` and `type-floatlocale-selected` (unbound by default)\n - `floatlocale` parses based on `LC_NUMERIC` envvar (must be set before launching)\n - parsing is 20x slower than with standard float column\n - will parse commas as decimals (e.g. '1,1') if LC_NUMERIC is set to a locale like 'en_DK.UTF-8'\n\n## Loaders\n\n - [loaders geojson] add loading and saving support for geojson files (thanks @draco #876)\n - [loaders vds] add loader/saver for custom .vds format (VisiData Sheet) to save column properties and data for multiple sheets in one file\n - [ux] autoload all subsheets by default; set `options.load_lazy` to disable\n - removes a minor friction with unloaded subsheets\n\n - [loaders http] add `options.http_max_next` to limit api pagination (default 0 - no pagination) (thanks @aborruso #830)\n\n## Bugfixes and Adjustments\n\n - [cli] fail properly if path cannot be opened\n - [defer] only mention number of deleted rows, if some were deleted\n - [go-pageup go-pagedown] ensure cursor stays in the same relative positions\n - [loaders mysql] fix mysql loader duplicating tables for each database (thanks @SuRaMoN #868)\n - [loaders mysql] perform asynchronous data fetch for mysql loader (thanks @SuRaMoN #869)\n - [loaders pandas] fix empty subsets for dup-selected and frequency table `open-row` (thanks @ajkerrigan #881 #878)\n - [loaders shp] fix display (thanks @dracos #874)\n - [loaders shp] fix saving to geojson (thanks @dracos #876)\n - [replay] fix replaying of .vd with `set-option`\n - [slide] fix bug when sliding key columns to the left, after sliding them to the right\n - [types] add command `type-floatsi-selected` on **Columns Sheet**\n\n - [expand] errors and nulls can now be expanded with `expand-cols` (thanks @geekscrapy #865)\n\n - [open] openSource now uses **'global'** `options.filetype` instead of sheet-specific as previous\n - to set the filetype for a file locally, set through cli: `vd -f tsv sample.foo`\n - to set in the **CommandLog**, use sheet=\"global\" with longname=\"set-option\"\n\n - [loaders http] raise exception if http status is not 20x (thanks @geekscrapy #848)\n - [loaders shp] support more Shapefile types (thanks @dracos #875)\n\n## API\n - add `create` kwarg to `openSource()`, to create the file if it does not exist already\n - [settings] 'global' is now 'default', and 'override' is 'global'\n - 'default' is the default setting within VisiData\n - 'global' is a user override on that default that applies globally\n - sheet-specific overrides global and default, for the sheet it is specific to\n - options set through visidatarc and cli are 'global' unless otherwise specified\n - [save] grab `save_foo` from **SheetType** first\n - allows overrides of sheet-specific saving\n\n# v2.1.1 (2021-01-03)\n\n - [macros] allow macro interfaces to be longnames (thanks @frosencrantz #787)\n - [save] better default save filename for url sheets (thanks @geekscrapy #824)\n\n## Bugfixes\n - [cmdlog] record column, sheet, and row info for open-cell\n - [cmdlog] catch case of 'override' sheet for set-option\n - [expr-col] `curcol` now works for multiple invocations (thanks @geekscrapy #659)\n - [loaders postgres] account for postgres_schema when rendering Postgres tables (thanks @jdormit for PR #852)\n - [loaders url] fail unknown URL scheme (thanks @geekscrapy for PR #84)\n - [pyobj] fix Pyobj Sheets for lists (thanks @brookskindle #843)\n - [pipe] handle broken pipes gracefully (thanks @robdmc #851)\n - [scroll] fix issue with jagged scrolling down (thanks @uoee #832)\n - [sort] fix bug where total progress in sorting is (100 * # of columns to sort) (thanks @cwarden)\n\n## api\n - format_field formats int(0) and float(0.0) as \"0\" (thanks @geekscrapy for PR #821)\n - add TypedWrapper.__len__ (thanks @geekscrapy)\n\n# v2.1 (2020-12-06)\n\n - [add] add bulk rows and cols leave cursor on first added (like add singles)\n - [add] add colname input to `addcol-new`\n - [aggregators] add mode and stdev to aggregator options (thanks @jsvine for PR #754)\n - [api] add options.unset()\n - [columns] add hidden 'keycol' to **ColumnsSheet** (thanks @geekscrapy for feature request #768)\n - [cli] support running as `python -m visidata` (thanks @abitrolly for PR #785)\n - [cli] add `#!vd -p` as first line of `.vdj` for executable vd script\n - [cli] allow `=` in `.vd` replay parameters\n - [clipboard] clipboard commands now require some selected rows #681\n - [commands] add unset-option command bound to `d` on OptionsSheet #733\n - [config] `--config=''` now ignores visidatarc (thanks @rswgnu for feature request #777)\n - [defer] commit changes, even if no deferred changes\n - [deprecated] add traceback warnings for deprecated calls (thanks @ajkerrigan for PR #724)\n - [display] add sort indication #582\n - [display] show ellipsis on left side with non-zero hoffset (thanks @frosencrantz for feature request #751)\n - [expr] allow column attributes as variables (thanks @frosencrantz for feature request #659)\n - [freq] change `numeric_binning` back to False by default\n - [input] Shift+Arrow within `edit-cell` to move cursor and re-enter edit mode\n - [loaders http] have automatic API pagination (thanks @geekscrapy for feature request #480)\n - [loaders json] improve loading speedup 50% (thanks @lxcode for investigating and pointing this out #765)\n - this makes JSON saving non-deterministic in Python 3.6, as the order of fields output is dependent on the order within the dict\n - (this is the default behaviour for dicts in Python 3.7+)\n - [loaders json] try loading as jsonl before json (inverted)\n - jsonl is a streamable format, so this way it doesn't have to wait for the entire contents to be loaded before failing to parse as json and then trying to parse as jsonl\n - fixes api loading with http so that contents of each response are added as they happen\n - unfurl toplevel lists\n - functionally now jsonl and json are identical\n - [loaders json] try parsing `options.json_indent` as int (thanks @frosencrantz for the bug report #753)\n this means json output can't be indented with a number. this seems like an uncommon use case\n - [loaders json] skip lines starting with `#`\n - [loaders pdf] `options.pdf_tables` to parse tables from pdf with tabular\n - [loaders sqlite] use rowid to update and delete rows\n - note that this will not work with WITHOUT ROWID sqlite tables\n - [loaders xlsx] add active column (thanks @kbd for feature request #726)\n - [loaders zip] add extract-file, extract-selected, extract-file-to, extract-selected-to commands\n - [macros] add improved macro system (thanks @bob-u for feature request #755)\n - `m` (`macro-record`) begins recording macro; `m` prompts for keystroke, and completes recording\n - macro can then be executed every time provided keystroke is used, will override existing keybinding\n - `gm` opens an index of all existing macros, can be directly viewed with `Enter` and then modified with `Ctrl+S`\n - macros will run command on current row, column sheet\n - remove deprecated `z Ctrl+D` older iteration of macro system\n - [regex] use capture names for column names, if available, in `capture-col` (thanks @tsibley for PR #808)\n - allows for pre-determining friendlier column names, saving a renaming step later\n - [save] `g Ctrl+S` is `save-sheets-selected` on **IndexSheet**\n - new command allows some or all sheets on an **IndexSheet** to be saved (and not the sheets on the sheet stack)\n - [saver] add fixed-width saver (uses col.width)\n - [saver sqlite] ensureLoaded when saving sheets to sqlite db\n - [search] `search-next` and `searchr-next` are now bound to n and N (was `next-search` and `search-prev`)\n - [select] differentiate select-equal- and select-exact- (thanks @geekscrapy for feature request #734)\n - previous select-equal- matched type value\n - now select-equal- matches display value\n - add `z,` and `gz,` bindings for select-exact-cell/-row\n - [sheets] sorting on **SheetsSheet** now does not sort **SheetsSheet** itself. (thanks @klartext and @geekscrapy for bug reports #761 #518)\n - [status] use `color_working` for progress indicator (thanks @geekscrapy for feature request #804)\n - [types] add floatsi parser (sponsored feature by @anjakefala #661)\n - floatsi type now parses SI strings (like 2.3M)\n - use `z%` to set column type to floatsi\n\n## Bugfixes\n\n - [api] expose visidata.view (thanks @alekibango for bug report #732)\n - [color] use `color_column_sep` for sep chars (thanks @geekscrapy for bug report)\n - [defer] frozen columns should not be deferred (thanks @frosencrantz for bug report #786)\n - [dir] fix commit-sheet and delete-row on DirSheet\n - [draw] fix display for off-screen cursor with multiline rows\n - [expr] remove duplicate tabbing suggestions (thanks @geekscrapy for bug report #747)\n - [expr] never include computing column (thanks @geekscrapy for bug report #756)\n - only checks for self-reference; 2+ cycles still raises RecursionException\n - caches are now for each cell, instead of for each row\n - [freeze] freeze-sheet with errors should replace with null\n - [loaders frictionless] assume JSON if no format (thanks scls19fr for bug report #803)\n - from https://specs.frictionlessdata.io/data-resource/#data-location):\n - a consumer of resource object MAY assume if no format or mediatype property is provided that the data is JSON and attempt to process it as such.\n\n - [loaders hdf5] misc bugfixes to hdf5 dataset loading (thanks @amotl for PR #728)\n - [loaders jsonl] fix copy-rows\n - [loaders pandas] support loading Python objects directly (thanks @ajkerrigan for PR #816 and scls19fr for bug report #798)\n - [loaders pandas] ensure all column names are strings (thanks @ajkerrigan for PR #816 and scls19fr for bug report #800)\n - [loaders pandas] build frequency table using a copy of the source (thanks @ajkerrigan for PR #816 and scls19fr for bug report #802)\n - [loaders sqlite] fix commit-sheet\n - [loaders sqlite] fix commit deletes\n - [loaders xlsx] only reload Workbook sheets to avoid error (thanks @aborruso for bug report #797)\n - [loaders vdj] fix add-row\n - [man] fix warnings with manpage (thanks @jsvine for the bug report #718)\n - [movement] fix scroll-cells (thanks @jsvine for bug report #762)\n - [numeric binning] perform degenerate binning when number of bins greater than number of values\n - (instead of when greater than width of bins)\n - [numeric binning] if width of bins is 1, fallback to degenerate binning\n - [numeric binning] degenerate binning should resemble non-numeric binning (thanks @setop for bug report #791)\n - [options] fix `confirm_overwrite` in batch mode\n - fix `-y` to set `confirm_overwrite` to False (means, no confirmation necessary for overwrite)\n - make `confirm()` always fail in batch mode\n - make `confirm_overwrite` a sheet-specific option\n - [plugins] only reload **Plugins Sheet** if not already loaded\n - [replay] move to replay context after getting sheet (thanks @rswgnu for bug report #796)\n - [replay] do not push replaying .vd on sheet stack (thanks @rswgnu for bug report #795)\n - [scroll] zj/zk do nothing in single-line mode (thanks @jsvine for suggestion)\n - [shell] empty stdin to avoid hanging process (thanks @frosencrantz for bug report #752)\n - [status] handle missing attributes in `disp_rstatus_fmt` (thanks @geekscrapy for bug report #764)\n - [tabulate] fix savers to save in their own format (thanks @frosencrantz for bug report #723)\n - [typing] fix indefinite hang for typing (thanks @lxcode for issue #794)\n - [windows] add Ctrl+M as alias for Ctrl+J #741 (thanks @bob-u for bug report #741)\n - [windows man] package man/vd.txt as a fallback for when man is not available on os (thanks @bob-u for bug report #745)\n\n## Plugins\n- add conll loader to **PluginsSheet** (thanks @polm)\n- remove livesearch\n- add clickhouse loader\n\n## Commands\n- if `options.some_selected_rows` is True, `setcol-expr`, `setcol-iter`, `setcol-subst`, `setcol-subst`, `setcol-subst-all` will return all rows, if none selected\n\n## API\n- [columns] add Column.visibleWidth\n- [open] additionally search for `open_filetype` within the vd scope\n- [select] rename `someSelectedRows` to `onlySelectedRows`\n- [select] add new `someSelectedRows` and `options.some_selected_rows` (thanks maufdez for feature request #767)\n - if options is True, and no rows are selected, `someSelectedRows` will return all rows\n- [status] allow non-hashable status msgs by deduping based on stringified contents\n- [isNumeric] isNumeric is part of vdobj\n\n# v2.0.1 (2020-10-13)\n\n## Bugfixes\n - Fix printing of motd to status\n\n# v2.0 (2020-10-12)\n\n## Additions and Improvements\n - [aggregators] allow custom aggregators in plugins/visidatarc (thanks @geekscrapy for the feature request #651)\n - [loaders xlsx] automatically clean sheet name when saving; warn if sheet name changes (thanks @geekscrapy for the request #594)\n - [columns] unhide height attribute by default (thanks @frosencrantz for feature request #660)\n - add .vcf (VCard) loader\n - [sqlite] remove name of db from an **SqliteSheet**'s name, only tablename\n - [syspaste] make `syspaste-` replayable and undoable (note that `syspaste-` value will be recorded in **CommandLog**)\n - [savers] many text saver filetypes via tabulate library (thanks @jsvine for original vdtabulate plugin)\n - [calc] ExprColumn no longer cached by default\n - [loaders rec] add new .rec file loader and multisheet saver (recutils)\n - [savers] implemented multisheet saver for both json and jsonl\n - [loaders eml] add new .eml file loader for email files with attachments\n\n## Options\n - add `options.incr_base` (thanks @chocolateboy for the suggestion #647)\n - (former) `options.force_valid_colnames` renamed to `options.clean_names`\n - applies to **Sheets** and **Columns** now (thanks @geekscrapy for the request #594)\n - for --X=Y, do not replace - with _ in Y (thanks @forensicdave for bug report #657)\n - add `options.default_height` for visibility toggle (thanks @frosencrantz for feature request #660)\n - add support for `--` option-ending option on CLI.\n - [input] default now `fancy_chooser` = True\n - when fancy_chooser enabled, aggregators and jointype are chosen with a ChoiceSheet.\n - `s` to select, `Enter` to choose current row, `g Enter` to choose selected rows, `q` to not choose any\n - numeric_binning is now True by default (enables numeric binning on **PivotSheet** and **FreqTable** for numeric columns\n\n## Command changes and additions\n - (former) setcol-range (`gz=`) renamed to `setcol-iter`\n - (former) `addcol-range-step` (`i`) renamed to `addcol-incr-step`\n - (former) `setcol-range` (`gi`) renamed to `setcol-incr`\n - (former) `addcol-range-step` (`zi`) renamed to `addcol-incr-step`\n - (former) `setcol-range-step` (`gzi`) renamed to `setcol-incr-step`\n - add `scroll-cells-*` to scroll display of cells while remaining in a Column; bind to [g]z{hjkl}\n - (former) unbind z{hjkl} from `scroll-col` (thanks @geekscrapy for feature request #662)\n - add `type-floatsi` bound to `z%` (#661)\n - `reload-selected` now reloads all **Sheets** if none selected (thanks @geekscrapy for PR #685)\n - add customdate with fixed fmtstr for parsing (use `z@` and input a fmtstr compatible with strptime (thanks @suntzuisafterU for feature request #677)\n\n## Bugfixes\n - [DirSheet] use changed ext as filetype when loading files (thanks @frosencrantz for bug report #645)\n - [slide] several major improvements to column sliding; key column sliding now works (thanks much to @geekscrapy for bug hunting #640)\n - [open-row] **Sheets Sheet** should be removed from stack upon `open-row` (thanks @cwarden for the bug report)\n - [cli] re-add --version (thanks @mlawren for bug report #674)\n - [open-config] fix `gO` (thanks @geekscrapy for bug report #676)\n - [splitwin] handle swap case for single sheet (thanks @geekscrapy for bug report #679)\n - [loaders xlsx] handle `None` column names for all **Sequence Sheet** loaders (thanks @jsvine for bug report #680)\n - [settings] retrieve from cache for top sheet if obj is None (thanks @aborruso for the bug report #675)\n - [settings] check if option is set on specific sheet before falling back to override\n - [describe] have **DescribeSheet** use source column's sheet's `options.null_value` to calculate its null column (thanks @aborruso for the bug report #675)\n - [undo] ensure that undos for complex commands (like `expand-cols`) are set more frequently (thanks @frosencrantz for the bug report #668)\n - it is still possible to find race conditions if the user presses commands fast enough, however they should happen far less frequently\n - [vlen] fix numeric binning for `vlen()` (thanks @frosencrantz for bug report #690)\n - [pivot] fix pivot case where no aggregator is set\n - [pyobj] fix filtering for **PyobjSheet**\n - [DirSheet] fix sorting for directory column of **DirSheet** (thanks @frosencrantz for bug report #691)\n - [json] fix bug saving cells with nested date values (thanks @ajkerrigan for PR #709)\n - [input] fix Ctrl+W bug when erasing word at beginning of line\n - [plugins] import `.visidata/plugins` by default\n - [pandas] use a safer `reset_index()` to avoid losing data when updating a pandas index (thanks @ajkerrigan for PR #710)\n - [threads] disable `add-row` on **ThreadsSheet** (thanks @geekscrapy for bug report #713)\n\n## deprecated\n - complete removal of `status` and `statuses` from deprecated (thanks @frosencrantz for bug report #621)\n - longnames are now `open-status` and `show-status`\n - remove `cursorColIndex`\n\n## API and Interface\n - `Sheet(*names, **kwargs)` autojoins list of name parts\n - `openSource()`, `aggregator()`, and `aggregators` are now part of vdobj\n - `set_option` is now `setOption`\n - move `isError` to `Column.isError`\n - deprecate `load_pyobj`, now **PyobjSheet**\n - add `.getall('foo_')` which returns all options prefixed with `'foo_'`; deprecated `options('foo_')`\n - `nSelected` is now `nSelectedRows`\n - make `Column.width` property, so setting is same as `Column.setWidth`\n - `evalexpr` is now `evalExpr`\n - `format` is now `formatValue`\n - `SettableColumn.cache` is now `._store`\n - `vdtype()` is now `vd.addType()`\n - add `addColumnAtCursor` (thanks @geekscrapy for bug report #714)\n\n## Plugins\n - update sparkline (thanks @layertwo #696)\n - plugin dependencies now install into `plugins-deps` (former plugin-deps)\n\n## Dev niceties\n - Fully automate dev setup with Gitpod.io (thanks @ajkerrigan for PR #673)\n\n\n# v2.-4 (2020-07-27)\n\n## Additions and Improvements\n - [cmdlog] allow sheet-specific set-option for replay\n - [columns] add default uppercase names for created columns (like VisiCalc)\n - these names are global; no default name is ever reused\n - [cosmetic] a column with a width 1 will now display (thanks @frosencrantz for the bug report #512)\n - [defer] move defermods and vls back into vdcore\n - configure sqlite and DirSheet to use it\n - [dir] allow explicit filetype when loading a directory (thanks @geekscrapy for the bug report #546)\n - [errors] ErrorsSheet on `g Ctrl+E` lists errors, instead of concatenating\n - [expand-cols] account for all visible rows when expanding a column (thanks @ajkerrigan for PR #497)\n - [loaders csv] use `options.safe_error` for cell exceptions on save\n - [loaders http] use file format in path if loader available (thanks @jsvine for PR #576 and bug report #531)\n - if not, fail back to MIME type (prev behaviour)\n - [loaders imap] add loader for imap://\n - [loaders json] handle non-dict rows in json data (thanks @ajkerrigan for PR #541 and @jsvine for bug report #529)\n - [loaders jsonl] show parse errors in every column\n - [loaders MySQL] add support for MySQL loader (thanks @p3k for PR #617)\n - [loaders pandas] upgrade pandas to 1.0.3 (thanks @ajkerrigan for PR #563)\n - [loaders pandas] add auto-loaders for feather, gbq, orc, parquet, pickle, sas, stata (thanks @khughitt for bug report #460)\n - [loaders pdf] add simple pdf loader\n - [loaders postgres] add support for connecting directly to rds (thanks @danielcynerio for PR #536)\n - the url has the following format: `rds://db_user@hostname:port/region/dbname`\n - it assumes that the AWS IAM for the user is configured properly\n - [loaders xls/xlsx] add save_xls and save_xlsx (thanks @geekscrapy for PR #574)\n - [loaders yaml] allow diving into YAML rows (thanks @ajkerrigan for PR #533)\n - [loaders yaml] use the default safe YAML loader (thanks @tsibley for the PR #600 )\n - the full loader is unsafe because serialized files can be constructed which run arbitrary code during their deserialization\n - the safe loader supports a very large subset of YAML and supports the most common uses of YAML\n - [loaders yaml] support files containing multiple documents (thanks @tsibley for PR #601)\n - [options] set visidata_dir and config from `$VD_DIR` and `$VD_CONFIG` (thanks @tsibley for bug report #448)\n - [type fmtstr] thousands separator (thanks @dimonf for bug report #575)\n - default for int/float is string.format for roundtripping accurately in data text files like csv\n - if fmtstr starts with '%', use locale.format_string (with grouping)\n - otherwise, use python string.format\n - currency uses locale, and is grouped.\n - [quitguard] if set on specific sheet, only confirm quit on that sheet (thanks @jsvine for bug report #538)\n - [undo] add undo for `rename-col-x` family, mouse slide, and `reload-sheet` (thanks @jsvine for feature request #528)\n\n## Command changes\n - [add-sheet] renamed to open-new; new sheet always has a single column\n - [config] bind `g Shift+O` back to open-config (#553)\n - [dive] convert many dive- commands to open- (#557)\n - add open-row bound to `ENTER` on Sheet itself\n - add `open-source` unbound on BaseSheet\n - deprecate `dive-*` longname\n - [options] options-global now `Shift+O`; options-sheet now `z Shift+O`\n - [multi-line] have visibility toggle Multi-Line Row on TextSheets (Closes #513)\n - used to toggle `wrap`\n\n## Command additions\n - [canvas] add resize-x/y-input commands to set x/y axis dimensions (thanks @pigmonkey for feature request #403)\n - [errors] add select-error-col and select-error (thanks @pigmonkey for feature request #402)\n - [input] `Ctrl+Y` paste from cell clipboard\n - [input] Ctrl+Left/Right move cursor to prev/next word\n - [iota] add `i` family of commands (iota/increment)\n - (former) setcol-range (`gz=`) renamed to `setcol-iter`\n - `addcol-range-step` (`i`): add column with incremental value\n - `secol-range` (`gi`): set current column for selected rows to incremental values\n - `addcol-range-step` (`zi`): add column with incremental values times given step\n - `setcol-range` (`gzi`): set current column for selected rows to incremental values times given step\n - [mouse] add unbound `mouse-enable` and `mouse-disable` commands\n - [quitguard add unbound `guard-sheet` command to set quitguard on current sheet (thanks jsvine for feature request #538)\n - [unfurl-col] add command, bound to `zM`, which does row-wise expansion of iterables in a column (thanks @frosencrantz for feature request and jsvine for initial code sample #623)\n - thanks @jsvine for name and initial implementation\n\n## Options\n - [cli] custom cli option parsing (thanks @tsibley for the behaviour request #573)\n - `--options` apply as sheet-specific option overrides to the sources following them\n - the last setting for a given option is the cli-given override setting (applies to all cli sources, unless they have the option already set)\n - this allows both\n - `vd -f csv foo.txt`\n - `vd foo.txt -f csv`\n - `--help` opens the manpage\n - `-g` prefix sets option globally for all sheets\n - [cli] add --imports (default \"plugins\") (thanks @tsibley for feature request #448)\n - space-separated list of modules to import into globals before loading .visidatarc\n - plugins can be installed by VisiData without modifying .visidatarc\n - [chooser] experimental `options.fancy_chooser`\n - when fancy_chooser enabled, aggregators and jointype are chosen with a ChoiceSheet.\n - press `ENTER` on any row to choose a single option, or select some rows, and press `ENTER` to choose the selectedrows\n - warning: the mechanism to do this effectively launches another instance of visidata, and so it is possible to get into an embedded state (if you jump around sheets, for example, instead of selecting). 'gq' should still work (thought `CTRL+Q` may need to be pressed several times).\n - [dir] add `-r` alias for `--dir-recurse`\n - [join-cols] add `options.value_joiner` to combine cell values for join-col (thanks @aborruso for feature request #550)\n - [join-cols] add `options.name_joiner` to combine column names for join-col, and sheet names for dive-row (thanks @aborruso for feature request #550)\n - sheet names for join-sheets are still joined with '+' or '&' for the time being\n - [loaders html] add `options.html_title` to exclude the sheetname when saving sheet as html table (thanks @geekscrapy for PR #566)\n - [loaders postgres] add support for custom postgres schema (Thanks @p3k for PR #615)\n - schema defaults to `public` but can be overridden using the `--postgres-schema` flag:\n - `vd --postgres-schema=foo postgres://user:pw@localhost/foobar`\n - [loaders zip] -f filetype now applies to inner files\n - [mouse] add options.mouse_interval to control the max time between press/release for click (ms)\n - set to 0 to disable completely\n - [pyobj] add `options.expand_col_scanrows` to set the number of rows to check when expanding columns (0 = all)\n - [type fmtstr] add fmtstr options for numerical types\n - add options.disp_currency_fmt\n - add options.disp_int_fmt\n - add options.disp_date_fmt\n\n## Plugins\n - [dependencies] install plugin dependencies into vd dir (thanks @tsibley for feature request #448)\n - [diff] diff is now a plugin\n - `--diff` is not available as a cmdline argument anymore\n - [vds3] bumped to 0.4 (@ajkerrigan)\n - [marks] initial release 0.1; marks selected rows with a keystroke; utils for selecting + viewing marked rows (@saulpw)\n - [genericSQL] initial release (1.0); basic loader for MySQL (Oracle, MySQL) (@aswanson)\n - [diff] is now a plugin (@saulpw)\n\n## Bugfixes\n - [cmdlog] fix case where CommandLog `open-` entries would not be replayable\n - [cmdlog] record keystrokes for command\n - [cmdlog] global cmdlog behaviour is now consistent with VisiData v1.5.2 cmdlog\n - [dirsheet] check if directory before grabbing filetype from ext (thanks @frosencrantz for bug report #629)\n - handles case where `.` in directory name\n - [helpsheet] do not include deprecated longnames (thanks @frosencrantz for bug report #621)\n - [input] flush input buffer upon newline in input; prevent pastes with accidental newlines from becoming keystrokes (thanks @geekscrapy for bug report #585)\n - [loaders csv] PEP 479 fix for csv loader (thanks @ajkerrigan for PR #499)\n - This avoids the following error when opening CSV files in Python 3.8: `RuntimeError: generator raised StopIteration`, but maintains the behaviour of gracefully handling malformed CSV files.\n - References:\n - https://www.python.org/dev/peps/pep-0479/#examples-of-breakage\n - https://github.com/python/cpython/pull/6381/files\n - [loaders html] cast to str before writing (thanks @geekscrapy for bug report #501)\n - [loaders html md] preserve formatting of display values when saving\n - [loaders html] fix string formatting issue for the html table name when saving (thanks @geekscrapy for PR #566)\n - [loaders pandas] bugfixes for sort (thanks @ajkerrigan for PR #496)\n - [loaders pandas] fix row deletion + its undo (thanks @ajkerrigan for PR #496)\n - [loaders pandas] improve regex select/unselect logic (thanks @ajkerrigan for PR #496)\n - [loaders pandas] fix row selection/deselection (thanks @ajkerrigan for PR #496)\n - [loaders postgres] load an estimate of row numbers for improved performance (thanks @danielcynerio for PR #549)\n - [loaders postgres] fix expand column to work on a json column in postgres (thanks @danielcynerio for PR #552)\n - [loaders sqlite] save display value if not supported sqlite type (thanks @jtf621 for bug report #570)\n - [loaders xml] correctly copy columns; fix path (#504)\n - [numeric-binning] fix numeric-binning bug with currency type column\n - [dir] fix dup-rows-deep on DirSheet (thanks @geekscrapy for bug report #489)\n - [rstatus] fix rstatus when repeating a command with no keystrokes (Thanks @ajkerrigan for bug report #577)\n - [save-sheets] fix saving multi-sheets as individual files to directory\n - [settings] remove internal option defaults from cmdlog\n - [sheets_all] make opened .vd/.vdj precious\n - [transpose] handle case where columns are numeric (thanks @frosencrantz for bug report #631)\n - [undo] fix undo with duplicate-named sheets (thanks @jsvine for bug report #527)\n - [utils] Fix namedlist bug with column named after VisiData attrs (particularly 'length') (thanks @tsibley for bug report #543)\n \n\n## Infrastructure / API\n - [asyncsingle] ensure that unfinished threads decorated with @asyncsingle do not block upon sync()\n - used so that domotd() and PluginsSheet().reload() do not block replay progression\n - [open-] switch from vd.filetype to open_ext; deprecate vd.filetype\n - [warnings] output Python warnings to status\n\n# v2.-3 (2020-03-09)\n\n## Major changes\n - [cosmetic] change default column separators\n - [json] make json load/save key order same as column order (ensures round-trip #429)\n - [commands.tsv] remove commands.tsv; move helpstr into code\n\n## Major features\n - add Split Window\n - options.disp_splitwin_pct (default: 0) controls height of second sheet on screen\n - add .vdj for cmdlog in jsonl format\n - add plugins/bazaar.jsonl for PluginsSheet in jsonl format\n\n### new commands\n - `splitwin-half` (`Shift+Z`) -- split screen, show sheet under top sheet\n - `splitwin-close` (`g Shift+Z`) -- closes split screen, current sheet full screens\n - `splitwin-swap` (`TAB`) -- swap to other pane\n - `splitwin-input` (`z Shift+Z`) -- queries for height of split window\n - `repeat-last` (unbound) -- run the previous cmd longname with any previous input (thanks #visidata for feature request! #441)\n - `repeat-input` (unbound) -- run the last command longname with empty, queried input (thanks #visidata for feature request! #441)\n - `resize-cols-input` (`gz_`) -- resize all visible columns to given input width\n - thanks @sfranky for feature request #414\n - `save-col-keys` (unbound) -- save current column and key columns\n - fixes #415; thanks @sfranky for feature request\n\n### new options\n - options.disp_float_fmt; default fmtstr to format for float values (default: %0.2f)\n - thanks khughitt for PR! #410\n\n## Additions and Improvements\n - add merge jointype (thanks @sfranky for feature request #405)\n - like \"outer\" join, except combines columns by name and each cell returns the first non-null/non-error value\n - use color_diff to merge join diffs\n - on edit, set values on *all* sheets which have the given row\n - adjust `save-cmdlog` input message for clarity\n - all sheets have a name (thanks @ajkerrigan for helping iron out the kinks with PR #472)\n - add args re-parsing to handle plugin options (helps with #443; thanks tkossak for bug report)\n - vdmenu should only get pushed outside of replay and batch mode\n - move cursor to row/col of undone command (thanks @jsvine for request)\n - move urlcache into async reload (affects PluginsSheet and motd)\n - add 'type' column to `SheetsSheet`\n\n### Command changes\n\n- `HOME`/`END` now bound to `go-leftmost`/`go-rightmost`\n - thanks [@gerard_sanroma](https://twitter.com/gerard_sanroma/status/1222128370567327746) for request\n- `z Ctrl+HOME`/`z Ctrl+END` now bound to `go-top`/`go-bottom`\n- `Ctrl+N` now bound to `replay-advance`\n\n### longname renamings\n- `search-next` (was `next-search`)\n- `search-prev` (was `prev-search`)\n- `jump-prev` (was `prev-sheet`)\n- `go-prev-value` (was `prev-value`)\n- `go-next-value` (was `next-value`)\n- `go-prev-selected` (was `prev-selected`)\n- `go-next-selected` (was `next-selected`)\n- `go-prev-null` (was `prev-null`)\n- `go-next-null` (was `next-null`)\n- `go-right-page` (was `page-right`)\n- `go-left-page` (was `page-left`)\n\n## Plugins\n - add usd plugin\n - provide USD(s) function to convert strings with currencies to equivalent US$ as float\n - uses data from fixer.io\n - add vds3 by @ajkerrigan\n - initial support for browsing S3 paths and read-only access to object\n - add \"provides\" column for plugins (helps with #449; thanks @tsibley for feature request)\n - standardize author in bazaar.jsonl\n - \"Firstname Lastname @githbhandle\"\n\n## Bugfixes\n - [cmdlog] fix issue with `append_tsv_row`, that occurred with `options.cmdlog_histfile` set\n - [replay] fix replaying of rowkeys\n - [replay] fix race condition which required the `--replay-wait` workaround\n - [plugins] ensure that `options.confirm_overwrite` applies to plugin installation\n - [slide] fix slide-leftmost\n - had inconsistent behaviour when a sheet had key columns\n - [slide] use visibleCol variants, such that slide works as expected with hidden cols\n - [options min_memory_mb] disable (set to 0) if \"free\" command not available\n - [core] auto-add raw default column only if options.debug (fixes #424; thanks @frosencrantz for bug report)\n - [cli] fix --config (thanks @osunderdog for bug report! #427)\n - [draw] fix status flickering that occurred with certain terminals (thanks @vapniks for bug report #412)\n - [txt save] save all visibleCols instead of only first one\n - [json] avoid adding columns twice when loading JSON dicts (thanks @ajkerrigan for bug report (#444) and PR (#447)\n - [fixed] fixed error that occurs when there are no headerlines (thanks @frosencrantz for bug report #439)\n - [pcap] update loader with modern api\n - [csv] catch rows with csv.Errors and yield error msg\n - [curses] keypad(1) needs to be set on all newwin (fixes #458)\n - [save-sheets] address two bugs with `g Ctrl+S`\n - [batch api] override editline() in batch mode (addresses #464; thanks @Geoffrey42 for bug report)\n - [replay] better handling of failed confirm (addresses #464; thanks @Geoffrey42 for bug report)\n - [asyncthread] with changed decorators, asyncthread should be the closest decorator to the function\n - if it is not, the act of decorating becomes spawned off, instead of calls to the function being decorated\n - [canvas] update Canvas delete- commands with current API (fixes #334)\n\n## Infrastructure / API\n - rename `Sheet` to `TableSheet`\n - deprecate `Sheet` but keep it around as a synonym probably forever\n - use HTTPS protocol for git submodules (thanks @tombh for PR #419)\n - this allows installation of VisiData in automated environments such as\n Dockerfiles where the git user is not logged into Github\n - unit tests have been migrated to pytest\n - use counter to keep track of frequency of column names\n - for joins, we want un-ambiguous sheets of origin when more than one sheet has a c.name\n - all sheets use addColumn api instead of manually appending columns\n - set terminal height/width via LINES/COLUMNS via curses.use_env (thanks halloleo for feature request #372)\n - update pip command to pull development branch of vsh (thanks @khughitt for PR #457)\n - change longnames *-replay to replay-*\n - rename vd.run() to vd.mainloop()\n - `vd.save_foo(p, *sheets)` throughout\n - standardize on vd.exceptionCaught\n - Sheet.addRows renamed to Sheet.addNewRows\n - option overrides can be done with SubSheet.options\n - options set with Sheet.options\n - extend status() varargs to error/fail/warning\n - add @BaseSheet.command decorator\n - rename tidydata.py to melt.py\n - deprecate globalCommand; use BaseSheet.addCommand\n - remove vd.addCommand\n - deprecate theme(); use option() instead\n - deprecate global bindkey/unbindkey\n - move commands, bindkeys, `_options` globals to vd object\n - DisplayWrapper compares with its value\n - this allows sensible colorizers like `lambda s,c,r,v: v==3`\n - Sheet.addColorizer now apply to single sheet itself (fixes #433; thanks @frosencrantz for bug report)\n - add Sheet.removeColorizer (thanks @frosencrantz for feature request #434)\n\n# v2.-2 (2019-12-03)\n\n## Major changes\n - [cmdlog] every sheet now has its own cmdlog\n - change `Shift+D` to `cmdlog-sheet`, with commands from source sheets recursively\n - `gShift+D` now `cmdlog-all`\n - `zShift+D` `cmdlog-sheet-only`\n - [dirsheet] VisiData's DirSheet is readonly; move write-mode for DirSheet to `vls` (see plugins)\n - [options] `options-global` bound to `gO`and `options-sheet` to `O`\n - `open-config` is now unbound (previously `gO`)\n - [defermods] has been moved to an opt-in plugin\n - [vdmenu] launching `vd` without a source file, opens menu of core sheets\n - press `Enter` to open sheet described in current row\n\n## Major Features\n - [IndexSheet] index into sub-sheets from command line (thanks @aborruso for suggestion #214)\n - currently works for html and hdf5 loaders\n - `+:subsheet:col:row` in cli \n - `subsheet` the topsheet upon load, with cursor located in cell at `row` and `col`\n - `+:subsheet::` to ignore row/col\n - can name toplevel source index if more than one: `+toplevel:subsheet::`\n\n## Additions and improvements\n - [add-rows] now undo-able\n - [aggregators] show-aggregate with quantiles (thanks @wesleyac for feature request #395)\n - [cli] `-P ` on commandline executes command on startup \n - [cmdlog] jump commands are not logged\n - [config] set VisiData height/width via LINES/COLUMNS envvars (thanks @halloleo for suggestion #372)\n - [csv] add `csv_lineterminator` option (default: '\\r\\n') (thanks @dbandstra for bug report #387)\n - retain csv writer default DOS line endings\n - [describe] add `options.describe_aggrs` (thanks @unhammer for suggestion #273)\n - space-separated list of statistics to calculate for numeric columns\n - default to existing 'mean stdev'\n - add this to .visidatarc for e.g. a harmonic mean to be added automatically to the describe sheet:\n ```\n from statistics import harmonic_mean\n options.describe_aggrs += ' harmonic_mean' # note the leading space\n ```\n - [describe] add hidden \"type\" col (thanks aborruso for suggestion #356)\n - [dirsheet] add `open-dir-current` command to open the DirSheet for the current directory\n - [help] add `help-commands-all` on `gz^H` (thanks @frosencrantz for suggestion #393)\n - [help] add `help-search` command (thanks @paulklemm for suggestion #247)\n - opens a commands sheet filtered by the input regex.\n - [loaders] add --header and --skip universal handling to all sheets that inherit from `SequenceSheet` (currently tsv/csv/fixed/xlsx/xls)\n - [menu] if no arguments, open VisiData Main Menu instead of DirSheet\n - [plugins] update PluginsSheet to add sha256 and vdpluginsdeps \n - [plugins] PluginsSheet now loads plugins in `~/.visidata/plugins/__init__.py` instead of in `~/.visidatarc`\n - to use this feature, add `from plugins import *` to `~/.visidatarc`\n - [pyobj] for security reasons, `.py` loader moved out of VisiData core and into snippets\n - Note that the PyObj loader auto-imports `.py` modules upon loading them\n - [ttf] use `Enter` to plot instead of `.`\n\n## Plugins\n - add hello world minimal plugin\n - update viewtsv example (thanks @suhrig for --skip improvement suggestions #347)\n - add vmailcap with `^V` to view by mimetype (thanks @cwarden for suggestion)\n - add basic frictionless loader (thanks @aborruso for suggestion #237)\n - `-f frictionless` with .json either http[s] or local file\n - .zip may not work yet\n - add fdir filetype; opens a DirSheet for a .txt with a list of paths\n - move trackmod and defermod out of VisiData core and into module defermods.py\n - defermods defers saving to source until commit-sheet\n - deleted rows are colored red and visible until commit\n - trackmods tracks changes in source sheet until save-sheet\n - deletes are removed upon delete-row(s)\n - defermods and trackmods are not on by default, `import visidata.defermods` must be added to visidatarc\n - plugin/loader authors: by default, all sheets that inherit from BaseSheet have .defermods=False and .trackmods set to True when defermods is imported\n - create package `vsh`; add to it `vls`, `vping`, `vtop`\n - `vls` contains write-mode for DirSheet\n - add vmutagen for audio tags on DirSheet\n - `Alt+m` to add the mutagen columns on the DirSheet\n - add geocoding using nettoolkit.com API\n - add `addcol-geocode` command to add lat/long columns from location/address column\n - new commands in rownum plugin\n - `addcol-rownum` adds column with original row ordering\n - `addcol-delta` adds column with delta of current column\n - vtask is now a discrete plugin\n\n## Bugfixes\n - [bindkey] move global bindkey after BaseSheet bindkey (thanks @sfranky for bug report #379)\n - [cmdlog] now will check for `confirm-overwrite`\n - [dirsheet] commit/restat/filesize interactions (thanks @Mikee-3000 for bug report #340)\n - [dirsheet] pass filetype to openSource\n - if filetype is not passed, options.filetype would overload file ext\n - [expr] catch recursive expression columns (columns that calculate their cells using themselves) (thanks @chocolateboy for bug report #350)\n - [fixed] various improvements to fixed-width sheet loader (thanks @frosencrantz for thorough bughunting #331)\n - [http] use options.encoding when no encoding is provided by responses headers (thanks @tsibley for the PR #370)\n - [join] joining columns in the ColumnSheet resulted in exception (thanks @frosencrantz for bug report #336)\n - [load] fix replay sync bug (required wait previously)\n - however, look out for `vd *` with lots of big datasets, they will now all load simultaneously\n - [longname] fix getCommand() error reporting\n - [mbtiles] now works again\n - [metasheets] created VisiDataMetaSheet which sets system TsvSheet options\n - now changes in tsv options for source files will not affect HelpSheet, CmdLog or PluginsSheet\n - thanks frosencrantz for bug report #323\n - [options] no error on unset if option not already set\n - [path] filesize of url is 0\n - [path] fix piping bug (vd failed to read stdin) (thanks @ajkerrigan for bug report #354)\n - [plugins] ensure consistent Python exe for plugin installs (thanks @ajkerrigan for fix)\n - [plugins] make plugin removal more predictable (thanks @ajkerrigan for fix)\n - [prev-sheet] would stack trace if more than one sheet loaded and no other sheet visited (thanks @frosencrantz for bug report #342)\n - [regex] will not silently fail if some example rows are not matches\n - [save] convert savers to use itervalues\n - itervalues(format=False) now yields OrderedDict of col -> value\n - value is typed value if format=False, display string if True\n - options.safety_first will convert newlines and tabs to options.tsv_safe_newline and options.tsv_safe_tab (thanks @mesibov for bug report #76)\n - [sheets] colorizer exceptions are now caught\n - [sheets] keycols now keep order they are keyed\n - [sysedit] trim all trailing newlines from external edits (thanks @sfranky for bug report #378)\n - [tsv] column name \"length\" prevented loading (thanks @suhrig for bug report #344)\n - [undo] redo with cmd on first row did not move cursor (thanks @Mikee-3000 for bug report #339)\n - now row/col context are set as strings, even when they are numeric indices\n\n## Infrastructure / API\n - [add-row] create a default newRow for Sheet (thanks @for-coursera for bug report #363)\n - [calc] add INPROGRESS sentinel\n - sentinel that looks like an exception for calcs that have not completed yet\n - [extensible] add new cached_property, which caches until clear_all_caches, which clears all cached_property\n - [Fanout] add Fanout\n - fan out attribute changes to every element in a list; tracks undo for list\n - [lazy_property] newSheet and cmdlog are now lazy_property\n - this enables the overwriting and extending of them by plugins\n - [loaders] add sheet.iterload()\n - will use sheet.source to populate and then yield each row\n - [loaders] vd.filetype(ext, ExtSheet) to register a constructor\n - [loaders] add Sheet.iterrows() to yield row objects \n - grouping use iterrows() for streaming input\n - __iter__() yields LazyComputeRows\n ```\n for row in vd.openSource('foo.csv'):\n print(row.date, row.name)\n ```\n - [IndexSheet] refactor SheetsSheet parent to IndexSheet\n - HtmlTablesSheet now inherits from IndexSheet\n - excel index changed to standard IndexSheet model\n - VisiDataSheet changed into IndexSheet\n - move join-sheets to IndexSheet\n - [options] add unset() to unset options (thanks @khughitt for suggestion #343)\n - [path] consolidate PathFd, UrlPath, and HttpPath into Path\n - [SequenceSheet] refactor tsv, csv, xls(x), fixed_width to inherit from SequenceSheet\n - [sheets] vd.sheetsSheet is sheetstack, vd.allSheetsSheet is sheetpile\n - [sheets] rename LazyMap to LazyChainMap and LazyMapRow to LazyComputeRow\n - [shortcut] BaseSheet.shortcut now property\n - [status] make right status more configurable (thanks @layertwo #375 and khugitt #343 for filing issues)\n - BaseSheet.progressPct now returns string instead of int\n - BaseSheet.rightStatus() now returns string only (not color)\n - by default uses `options.disp_rstatus_fmt`, configured like `disp_status_fmt`\n - progress indicator (% and gerund) moved out of rightStatus and into drawRightStatus\n - [undo] use undofuncs to associate command with its undo\n - [undo/redo] moved to undo.py\n - [vd] add sheet properties for errors and statuses\n - [vd] vd.quit() now takes `*sheets`\n - [vd] rename main() to main_vd()\n\n# v2.-1 (2019-08-18)\n\n## Major changes\n\n - Minimum Python requirement bumped to 3.6\n - Several interface changes (see below)\n\n## Major features\n\n - add Alt/Esc as prefix; Alt+# to go to that sheet\n - Alt+ layer not otherwise used; bind with `^[x` for Alt+X\n - undo/redo\n - [new command] `options.undo` (default: True) to enable infinite linear undo/redo\n - provisionally bound to `Shift+U` and `Shift+R`\n - will undo most recent modification on current sheet\n - `undoEditCells` assumes commands modified only selectedRows\n - multi-line rows\n - toggle by pressing `v` on any sheets with truncated values\n - range binning for numeric columns\n - `options.numeric_binning` (default: False) is the feature flag\n - [feature freqtbl] numeric binning for frequency/pivot table\n - `options.histogram_bins` to set number of bins (0 to choose a reasonable default)\n - (thanks @paulklemm for the issue #244)\n - stdout pipe/redirect\n - `ls|vd|lpr` to interactively select a list of filenames to send to the printer\n - `q`/`gq` to output nothing\n - `Ctrl+Q` to output current sheet (like at end of -b)\n - `vd -o-` to send directly to the terminal (not necessary if already redirected)\n - plugin framework\n - plugins are optional Python modules that extend or modify base VisiData's functionality\n - this release establishes a structure for creating plugins, and provides an interface within VisiData for installing them\n - `open-plugins` opens the **PluginsSheet**\n - to download and install a plugin, move the cursor to its row and press `a` (add)\n - to turn off a plugin, move the cursor to its row and press `d` (delete).\n - for more information see (https://visidata.org/docs/plugins)\n - deferred changes\n - modifications are now highlighted with yellow, until committed to with `^S` (`save-sheet`)\n\n## interface changes\n\n- `setcol-*`, `dive-selected`, `dup-selected-*`, `columns-selected`, `describe-selected` use only selectedRows (do not use all rows if none selected) #265 (thanks @cwarden)\n- `edit-cells` renamed to `setcol-input`\n- `fill-nulls` renamed to `setcol-fill`\n- `paste-cells` renamed to `setcol-clipboard`\n- `dup-cell`/`dup-row` on SheetFreqTable and DescribeSheet renamed to `dive-cell`/`dive-row`\n- `next-page`/`prev-page` renamed to `go-pagedown`/`go-pageup`\n- `save-col` always saves all rows in current column (instead of selectedRows or rows)\n- `copy-*` use only selectedRows, warning if none selected (cmdlog safe)\n- `syscopy-*` use only selectedRows, fail if none selected (not cmdlog safe)\n- all `plot-selected` are now `plot-numerics`; `plot-numerics` uses all rows\n- Shift+S pushes `sheets-stack`; gS pushes `sheets-all`. removed graveyard sheet.\n- `random-rows` is no longer bound to any key by default (was Shift+R).\n- `freq-summary` was `freq-rows`; adds summary for selected rows\n- cmdlog is now based on longname instead of keystrokes\n- cmdlog does not log resize commands\n- exit with error code on error during replay (suggested by @cwarden #240)\n- split `Ctrl+V` (check-version) into `Ctrl+V` (show-version) and `z Ctrl+V` (require-version)\n- `show-expr` now unbound from `z=`\n- add `options.row_delimiter` (default to `\\n`)\n\n\n## plugins\n\n- vfake: anonymizes columns\n- livesearch: filter rows as you search\n- rownum: add column of original row ordering\n- sparkline: add a sparkline column to visualise trends of numeric cells in a row (thanks @layertwo #297)\n\n## Bugfixes\n\n- [addcol-new] addcol-new now works in batch mode (thanks @cwarden for the bug report #251)\n- [canvas] clipstr xname to prevent overlap with 1st element in xaxis\n- [color] column separator color applies to regular rows (thanks @mightymiff for bug report #321)\n- [DirSheet] delete-selected now deletes all of the selected files upon save-sheet (thanks @cwarden for the bug report #261)\n- [display] fix resizing issue with wide chars (thanks @polm for the bug report #279 and for the fix #284 )\n- [freqtbl] unselect-rows now updates source rows (thanks @cwarden for bug report #318)\n- [go-col-regex] nextColRegex sheet is implicit parameter\n- [help] use tab as sep for system sheets (thanks @frosencrantz for bug report #323)\n- [plot] graphing currency values now works\n- [pyobj] SheetDict nested editing (thanks @egwynn for the bug report #245)\n- [txt] TextSheets now save as .txt\n- [yaml] handle sources that do not load as lists (thanks @frosencrantz for bug report #327)\n- [vdtui] make Sheet sortable (related to an issue found by @jsvine #241)\n\n\n## Additions and improvements\n\n- [addcol-new] does not ask for column name\n- [aggr] add `list` aggregator (thanks @chocolateboy #263)\n- [canvas] add legend width to fit max key (thanks @nicwaller for request)\n- [chooseMany] error() on invalid choice #169\n- [command join] add join-sheets-top2 (`&`) / join-sheets-all (`g&`) to Sheet to join top 2/all sheets in sheets-stack\n- [command sort] `sort-*-add` bound to z[] and gz[] to add additional sort columns\n- [command syspaste-cells] add `syspaste-cells` to paste into vd from system clipboard (thanks kovasap for PR #258)\n- [describe] add `sum` (thanks @pigmonkey for suggestion #315)\n- [DirSheet] include folders and hidden files\n- [exec-longname] enable history\n- [freeze-sheet] only freeze visibleCols\n- [html] add links column where hrefs available (suggested by @am-zed #278)\n- [license] remove MIT license from vdtui; all code now licensed under GPL3\n- [loader fixed] provide a way to limit the max number of columns created (thanks @frosencrantz for suggestion #313)\n - added `options.fixed_maxcols` (default: no limit)\n- [loader fixed] loaders override putValue, not setValue (thanks @aborruso for bug report #298)\n- [loader jira] add support for jira filetype, a markdown derivative compatible with Atlassian JIRA (thanks @layertwo #301)\n- [loader Pyobj] `py` filetype to import and explore a python module: `vd -f py curses`\n- [loader pyxlsb] add .xlsb loader (suggested by @woutervdijke #246)\n- [loader ndjson ldjson] add as aliases for jsonl\n- [loader npy] add .npy loader, including type detection\n- [loader npz] add support for .npz index\n- [loader usv] add .usv loader\n- [macros] is now deprecated\n- [motd] domotd is asyncsingle and thus not sync-able\n- [mouse] bind Ctrl+scrollwheel to scroll-left/right; change to move cursor by `options.scroll_incr` (like scroll-up/down)\n- [mouse] slide columns/rows with left-click and drag\n- [openSource] create new blank sheet if file does not exist\n- [option json] add `options.json_sort_keys` (default True) to sort keys when saving to JSON (thanks @chocolateboy for PR #262)\n- [option regex+] `options.default_sample_size` (default 100) to set number of example rows for regex split/capture (now async). use None for all rows. (thanks @aborruso #219)\n- [option vd] `--config` option to specify visidatarc file (suggested by @jsvine #236)\n- [option vdtui] remove `curses_timeout` option (fix to 100ms)\n- [pandas] support multi-line column names (suggested by @jtrakk #223)\n- [pandas] implement sort() for pandas DataFrame (suggested by @migueldvb #257)\n- [pandas] use value_counts() for PandasSheetFreqTable (thanks @azjps for PR #267)\n- [pandas] selection support for PandasSheet (thanks @azjps for PR #267)\n- [pandas] reset index (thanks @danlat #277)\n- [pandas] if the df contains an index column, hide it\n- [pcap] adds saver for .pcap to json (thanks @layertwo for PR #272)\n- [perf] expr columns are now set to cache automatically\n- [perf] drawing performance improvements\n- [perf] minor improvements to cliptext\n- [perf] several minor optimisations to color\n- [precious] describe-sheet is now precious; error-sheet and threads-sheet are not\n- [replay] show comments as status (suggested by @cwarden)\n- [save] make all `save_` callers async\n- [sqlite] add save (CREATE/INSERT only; for wholesale saving, not updates)\n- [sqlite] `Ctrl+S` to commit add/edit/deletes\n- [sqlite] add support for .sqlite3 extension\n- [tar] add support for opening tar files (thanks @layertwo #302)\n- [vdmenu] `Shift+V` opens menu of core sheets\n - press `Enter` to open sheet described in current row\n- [win] several changes made for increased windows-compatibility (thanks @scholer!)\n- [yaml] bump min required version (thanks @frosencrantz for suggestion #326)\n\n\n## API\n- VisiData, BaseSheet, Column inherit from Extensible base class\n - internal modules and plugins can be self-contained\n - `@X.property @X.lazy_property`, `X.init()`, `@X.api`\n- remove Sheet.vd; 'vd' attrs now available in execstr\n- remove hooks\n- add @deprecated(ver) decorator; put deprecations in deprecated.py\n- `vd.sync(*threads)` waits on specific threads (returned by calls to `@asyncthread` functions)\n- add Sheet.num for left status prompt\n- pivot and frequency table have been consolidated for numeric binning\n- add Sheet.nFooterRows property\n- Sheet.column() takes colname instead of regex; add Sheet.colsByName cached property\n- use addRow to rows.append in reload()\n- Selection API is overloadable for subclasses of Sheet whose rows don't have a stable id() (like pandas)\n- use locale.format_string and .currency\n - uses user default locale settings at startup\n - changes fmtstr back to %fmt (from {:fmt})\n- vdtui broken apart into separate modules: editline, textsheet, cliptext, color, column, sheet\n - much code reorganization throughout\n- convert all `vd()` to `vd`\n- remove ArrayColumns, NamedArrayColumns\n- urlcache now takes days=n\n- Sheet.rowid\n- add windowWidth and windowHeight\n - Sheets use their own .scr, in preparation for split-screen\n- add VisiData.finalInit() stage\n - call vd.finalInit() at end of module imports to initialise VisiData.init() members\n - so that e.g. cmdlog is not created until all internal sheet-specific options has been set\n- remove replayableOption() (now replay an argument within option())\n- CursesAttr is now ColorAttr; ColorAttr now a named tuple\n - variables that contain a ColorAttr have been renamed from attr to cattr for readability\n- improvements to scrolling API\n- rename most cases of Sheet*/Column* to *Sheet/*Column\n- use pathlib.Path in visidata.Path\n- remove BaseSheet.loaded; add BaseSheet.rows = UNLOADED\n- vd.push no longer returns sheet\n- add @asyncsingle for asyncthread singleton\n\n## Deps\n- add submodule fork of pyxlsb for VisiData integration\n- add amoffat/sh as submodule for vgit and vsh\n- [postgres] swap for binary version of dep\n\n\n# v1.5.2 (2019-01-12)\n\n## Bugfixes\n- [regex] fix `g*` #239 (thanks to @jsvine for bug hunting)\n- [editline] suspend during editline will resume in editline\n- [editline] `Ctrl+W` on an empty value in editline now works\n\n## Docs\n- [manpage] update the manpage to be more accurate for boolean command line options\n\n# v1.5.1 (2018-12-17)\n\n## Bugfixes\n- [canvas] fix mouse right-click and cursor movement on canvas\n- [idle performance] fix regression\n- [columns] fix editing of \"value\" column on ColumnsSheet\n- [describe] fix colorizer inheritance\n- [csv] always create at least one column\n- [pandas] fix pandas eval (`=`, etc) #208 (thanks to @nickray for suggesting)\n- [pandas] preserve columns types from DataFrame #208 (thanks to @nickray for suggesting)\n- [pandas] remove data autodetect #208 (thanks to @nickray for suggesting)\n\n## Additions and changes\n- [selection] `options.bulk_select_clear` per #229 (thanks to @aborruso for suggesting)\n- [setcol-subst-all] add `gz*` to substitute over all visible cols (thanks to @aborruso for suggesting)\n- [options] Shift+O now global options (was sheet options); `zO` now sheet options; `gO` now opens .visidatarc which can be edited (was global options)\n- [sort] orderBy now asynchronous #207 (thanks to @apnewberry for suggesting)\n- [fill] fill now async; uses previous non-null regardless of selectedness #217 (thanks to @aborruso for suggesting)\n- [pandas] `options.pandas_filetype_*` passed to `pandas.read_` (like `csv_*` to Python `csv` module) # 210 (thanks to @pigmonkey for suggesting)\n- [rename-col-selected] `z^` now renames the current column to contents of selected rows (previously `gz^`); `gz^` now renames all visible columns to contents of selected rows #220 (thanks to @aborruso for suggesting)\n- [vdtui null] show null display note in cells which match `options.null_value` (was only for None) # 213 (thanksto @aborruso for suggesting)\n- [vdtui] visidata.loadConfigFile(\"~/.visidatarc\") for use in REPL #211 (thanks to @apnewberry for suggesting)\n- [progress] include thread name on right status during async\n- [progress] add gerund to display (instead of threadname)\n- [http] user specified filetype overrieds mime type\n - e.g. `vd https://url.com/data -f html`\n- [clipboard] use `options.save_filetype` for default format\n\n\n\n# v1.5 (2018-11-02)\n\n## Bugfixes\n- [clipboard] fix broken `gzY` (syscopy-cells)\n- [cmdlog] always encode .vd files in utf-8, regardless of options.encoding\n- [tsv] major `save_tsv` performance improvement\n- [tsv] make short rows missing entries editable\n- [shp] reset columns on reload\n- [graph] shift rightmost x-axis label to be visible\n- [http] allow CLI urls to have `=` in them\n- [fixed width] truncate cell edits on fixed width sheets\n- [aggregators] ignore unknown aggregators\n- `visidata.view(obj)`: obj no longer required to have a `__name__`\n\n## Additions and changes\n- [save tsv json] errors are saved as `options.safe_error` (default `#ERR`)\n - if empty, error message is saved instead\n- [plugins] `~/.visidata` added to sys.path on startup\n - put plugin in `~/.visidata/vdfoo.py`\n - put `import vdfoo` in `.visidatarc` to activate\n- [aggregators] show-aggregate (`z+`) now aggregates selectedRows\n- [tsv] add unnamed columns if extra cells in rows\n- [diff] now based on display value (more intuitive)\n- [mouse] move to column also\n- [mouse] right-click to rename-col, rename-sheet, or edit-cell\n- [cosmetic] addcol-new (`za`) input new column name on top of new column\n- [cosmetic] include file iteration in progress meter\n- [xls xlsx] use options.header to determine column names\n\n# v1.4 (2018-09-23)\n\n## Bugfixes\n\n- batch mode with no script should use implicit stdin only if no other files given (Closed #182)\n- [pivot] pivot keycolumn copy was yielding strange nulls\n- [join] fix extend join\n- [csv] include first row in file even if `options.header` == 0\n- [sysclip] fix bug where `gzY` did not copy selected rows (Closed #186)\n- [motd] fix bug with disabling `options.motd_url` (Closed #190)\n\n## Additions and changes\n\n- various improvements in performance and in CPU usage (Closed #184, #153)\n- [pyobj] `visidata.view(obj)` and `visidata.view_pandas(df)`\n- [pandas] `-f pandas` loads file with `pandas.read_`\n- [TextSheet] wrap made consistent with new options\n- [date] date minus date now gives float number of days instead of seconds\n- [pcap] add support for reading pcapng (thanks @layertwo!)\n- [setcol] limit `gz=` range parameters to the number of rows selected to be filled (thanks @ssiegel!)\n- [anytype] format anytype with simple str()\n\n# v1.3.1 (2018-08-19)\n\n- [http] add `tab-seperated-values` to content_filetypes mapping\n- [join] add `extend` join type to use keep all rows and retain **SheetType** from first selected sheet\n- `rename-sheet` renames current sheet to input\n- [json] add options.json_indent for pretty-printing\n- [tsv json txt] add options.save_errors (default True) to include errors when saving\n- remove all options.foo_is_null and fix according to 178-nulls.md\n- add `z^C` and `gz^C` to cancel threads in current/selected row(s)\n- [bugfix] `^R` (reload) on a filtered sheet (`\"`) now reloads only the filtered rows\n- [aggregators] fix summation with exceptions\n- [DirSheet] add `gz^R` (reload-rows) to undo modifications on selected rows\n\n# v1.3 (2018-08-11)\n\n- commands overhaul; see `commands.tsv` (command longnames should now be largely stable)\n- add quantile aggregators (q3/q4/q5/q10)\n- add `z;` to add new column from bash *expr*, with `$`columnNames as variables\n- keyboard layout (thanks to @deinspanjer for the inspiration)\n- `O` launches sheet-specific options (see design/169.md); `gO` launches global OptionsSheet\n- options.wrap now defaults to False\n- options.quitguard enables confirmation before quitting last sheet\n- options.safety_first makes loading/saving more robust, at the cost of performance\n - currently only removing NULs from csv input\n- dedup, sort, color status messages by \"priority\" (thanks to @jsvine for suggestion)\n- remove menu system\n- can now edit source values from FreqSheet\n\n- Command changes\n - `^H` is now main command to open the manpage! `z^H` opens a list of all commands for this sheet.\n - `R` (`random-sheet`) pushes sheet instead of selecting (reverting to former behavior)\n - `za` (`addcol-empty`) asks for column name\n - `zd` (`delete-cell`) moves value to clipboard (\"cut\", like other delete commands)\n - add `gI` (`describe-all`) like `gC` (`columns-all`)\n - add `gS` (`sheets-graveyard`)\n - add `g(`, `z(`, `gz(` variants of `(` 'expand-column'\n - add `z|` and `z\\` to un/select by python expr (thanks to @jsvine for suggestion)\n - add `z#` to set type of current column to `len`\n - add `z;` to get the stdout/stderr from running a cmdline program with $colname args\n - `Space` is now bound to exec-longname (was `menu`; `^A` was exec-longname previously)\n\n- Loaders:\n - add pandas adapter\n - add xml loader\n - add pcap loader (thanks to @vbrown608 and @TemperedNetworks)\n - add yaml loader (thanks to @robcarrington, @JKiely, @anjakefala at PyCon Sprints for making this happen)\n - add jsonl saver\n - remove `tsv_safe_char` and split into `tsv_safe_newline` and `tsv_safe_tab`\n\n- initial commit of a task warrior app (vtask)\n\n## minor changes\n- more portable system clipboard handling (thanks @chocolateboy for PR)\n- [json] no more incremental display during loading (need better json parser than stdlib)\n- `date` supports adding a number of days (or `6*hours`, `9*months`, etc)\n- hidden columns are darkened on columns sheet\n- exception rollup\n- dev/commands.tsv table of commands\n- motd default url uses https\n- improve ProfileSheet\n- [DirSheet] editable `mode` (set to octal like `0o0644`)\n\n# v1.2.1 (2018-07-05)\n\n- python 3.7\n - Change `async` decorator to `asyncthread` and rename `async.py` to avoid using Python 3.7 keyword\n\n# v1.2 (2018-04-30)\n\n- macro system\n - `gD` goes to directory browser of `options.visidata_dir` (default to `~/.visidata/`) which contains saved commandlogs and macros\n - `z^S` on CommandLog saves selected rows to macro for given keystroke, saving to `.visidata/macro/command-longname.vd`\n - macro list saved at `.visidata/macros.vd` (keystroke, filename)\n- `options.cmdlog_histfile` for auto-appended (default: empty means disabled)\n- [DirSheet] edits make deferred changes to any field\n - add `directory` and `filetype` columns\n - note: only 256 changes maintained per column (same as column cache size)\n - `^S` saves all deferred changes\n - `z^S` saves changes for the current file only\n - `^R` clears all changes (reload)\n - `z^R` clears changes on the current file only\n - `d`/`gd` marks the current/selected file for deletion\n - if `directory` is edited, on `^S` (save) file is moved (if directory not existing, a new directory is created)\n- [New conda package](https://github.com/conda-forge/visidata-feedstock)\n- add .visidatarc [snippets](https://github.com/saulpw/visidata/tree/stable/snippets) with examples of extra functionality\n- add replayable options [#97](https://github.com/saulpw/visidata/issues/97)\n- `g^S` for multisave to single file (`.html`, `.md` and `.txt` are currently supported) or directory\n- `z^S` to save selected rows of current column only (along with key columns)\n- `T` to transpose rows and columns [#129](https://github.com/saulpw/visidata/issues/129)\n- `^A` to specify a command longname to execute\n- `^O`/`g^O` to open current/selected files in external editor\n- `g^R` on SheetsSheet to reload all [selected] sheets\n- `options.error_is_null` to treat errors as nulls when applicable\n- `g,` fixed to compare by visible column values, not by row objects\n- `gv` to unhide all columns\n- `gM` open melted sheet (unpivot) with key columns retained and *regex* capture groups determining how the non-key columns will be reduced to Variable-Value rows\n- `g*` replace selected row cells in current column with regex transform\n- Shift-Up/Down aliases for mac [#135](https://github.com/saulpw/visidata/issues/135)\n- options.wrap now true by default on TextSheet (`v` to toggle)\n- `save_txt` with single column concatenates all values to single file\n- `+` can add multiple aggregators\n- ^X bugfix: use evalexpr over cursorRow\n- `z`/`gz` `s`/`t`/`u` to select to/from cursorRow\n- `z<` and `z>` to move up/down to next null cell\n- `\"` no longer reselects all rows\n- `sheet-set-diff` command to act like `--diff`\n- math functions (like sin/cos) now at toplevel\n- bugfix: freeze\n- all `csv_` options sent to csv.reader/writer\n- `options.tsv_safe_char` to replace \\t and \\n in tsv files; set to empty for speedup during tsv save\n- loaders and savers\n - support bz2 and xz (LZMA) compression (thanks @jpgrayson)\n - add loaders for:\n - sas7bda (SAS; requires `sas7bdat`)\n - xpt (SAS; requires `xport`)\n - sav (SPSS; requires `savReaderWriter`)\n - dta (Stata; requires `pandas`)\n - .shp can save as .geojson\n - add htm as alias for html filetype\n - json bugfix: fix [#133](https://github.com/saulpw/visidata/issues/133) json loader column deduction\n- [experimental] bin/vsh initial commit\n\n# v1.1 (2018-03-05)\n\n- VisiData will be included in the [next debian repository release](https://tracker.debian.org/pkg/visidata)!\n- remove all install dependencies\n - additional libraries must be installed manually for certain loaders; see requirements.txt\n- experimental hierarchical menu system with SPACE to explore commands\n - use standard movement keys (`hjkl`/`arrows`) to navigate within a command level\n - Use `Enter`/`q` to navigate down/up a command tree\n - abort with `gq` or `^C`\n - existing chooseOne selections (aggregators/joins) still use simple input() for now\n - most longnames changed\n - let me know if anyone is using any longnames at all, and we will stabilize the names\n - if you do end up playing with it, please let me know what did and didn't work for you\n- randomized message/announcement/tip on startup; disable with `options.motd_url = None`\n - cache messages in `$HOME/.visidata/`\n\nCommand additions/changes:\n\n- add `za` and `gza` to add 1/N new blanks column\n- add `(` and `)` commands to expand/collapse list/dict columns (e.g. in nested json)\n- add `Backspace` command to drop sheet like `q` and also scrub its history from the cmdlog\n- [canvas] add `d` and `gd` to delete points from source sheet\n- remove `!@#$%-_` special actions on columns sheet\n- alias Shift+Arrows to `HJKL` (may not work in all environments)\n- alias `ENTER` to modify-edit-cell by default\n- add `Y`/`gY`/`zY` to copy row/selected/cell to system clipboard (with options.clipboard_copy_cmd)\n\n- filename `-` works to specify stdin/stdout (`-b -o -` will dump final sheet to stdout)\n- search/select uses most recent when not given any (as in vim/etc)\n- annotate None with disp_note_none ('∅'); previously was not visually distinguishable from empty string\n\n- save to .md org-mode compatible table format\n- load/view/edit/save png, edit pixels in data form\n- load/view ttf/otf font files\n- [canvas] draw quadratic curves with qcurve([(x,y)...])\n- improvements/bugfixes: pivot, describe, melt, sqlite, shp, html\n\n# v1.0 (2018-01-24)\n\n- date.__sub__ returns timedelta object (was int days)\n- pivot table bugfixes\n- many cosmetic fixes\n- disable default profiling for perf improvements\n- remove .visidatarc support in PWD or XDG; only $HOME/.visidatarc supported now\n- website and docs complete overhaul\n- do not execute .py files\n- apt/brew packages submitted\n\n# v0.99 (2017-12-22)\n\n- tab completion for filename and python expr\n- `v` now 'visibility toggle' (moved from `w`)\n- `^W` to erase a word in the line editor\n- `gC`\n- `--version` (thanks to @jsvine)\n- `options.use_default_colors` (thanks to @wavexx)\n- `median` aggregator\n- .html loads tables (requires lxml)\n - simple http works (requires requests)\n- json save\n- json incremental load\n- [cmdlog] use rowkey if available instead of row number; options.rowkey_prefix\n- [cmdlog] only set row/col when relevant\n- [vdtui] task renamed to thread\n- /howto/dev/loader\n- /design/graphics\n\n# v0.98.1 (2017-12-04)\n\n- [packaging]\n - make non Python standard library loader dependencies optional\n - provide method for full installation via `pip3 install \"visidata[full]\"`\n- [visidata.org](http://visidata.org) change copyright in footer\n- [docs] add csv dialects to manpage (closes issue #88)\n- [bugfix]\n - fix for `^Z` in builtin line editor\n - fixed-width loader needs source kwarg\n\n# v0.98 (2017-11-23)\n\n- [visidata.org](http://visidata.org) revamp\n\n- [canvas] graphs and maps!\n - `.` or `g.` to push a graph or a map from the current sheet (dot=plot)\n - supports .shp and vector .mbtiles\n - mouse left-click/drag to set cursor\n - mouse right-click/drag to scroll canvas\n - scrollwheel to zoom in/out on a canvas\n - `s`/`u` to select/unselect rows at canvas cursor\n - `ENTER` to push source sheet with only rows at canvas cursor\n - 1-9 to toggle display of 'layers' (colors)\n - `_` to zoom out to full width\n - `disp_pixel_random` option chooses pixel attrs at random (weighted), instead of most common\n - `+`/`-` to zoom in/out via keyboard\n\n- Updates to commands\n - Remove ` (backtick) command\n - Remove most zscroll commands (`zs`/`ze`)\n - `zz` moves cursor to center, uncertain about the future of `zt` due to conflict with `t` for toggle\n - `ga` adds N new rows\n - `gz=` sets value for selected/all rows to a Python sequence in this column\n - `z_` sets column width to given value\n - `z-` cuts column width in half\n - `P` is now \"paste before\" (like vim); `R` now pushes a random sample\n - `^Z` now sends SIGSTOP; `^O` \"opens\" the external $EDITOR (from builtin line editor)\n - [ColumnsSheet] Added `~!@#$` commands back, to set type of source columns\n - `w` is becoming a more universal \"visibility toggle\"\n - [TextSheet] `w` toggles wordwrap\n - [canvas] `w` toggles display of the labels\n - [pyobj] `w` toggles hidden properties and methods\n\n- Updates to command line args and options\n - set initial row/col with `+:` (numeric only)\n - `--delimiter`/`-d ` option (separate from `--csv-delimiter`) sets delimiter for tsv filetype\n - `--replay-wait`/`-w` renamed from `--delay`/`-d`\n - `disp_date_fmt` option for date display format string (default is date-only)\n - `zero_is_null`/`empty_is_null`/`none_is_null`/`false_is_null` set which values are considered null (previously was `aggr_null_filter`)\n - `--skiplines` option renamed to `--skip`, and `--headerlines` to `--header`\n\n- Design improvements\n - Add specific rowtype for each sheet (see right status)\n - dates are a kind of numeric type (useful for graphing as the x-axis)\n - `use_default_colors` (at behest of @wavexx)\n - more robust Progress indicator\n - populate DescribeSheet in async thread\n - remove default names for unnamed columns\n - history up/down in edit widget now feels right\n\n- API changes\n - change main Column API to getter(col, row) and setter(col, row, val)\n - move Path and subclasses out of vdtui\n - TextSheet source is any iterable of strings\n - Sheet.filetype provides default save filename extension\n\n\n## 0.97.1 (2017-10-29)\n- Fix postgres lazy import\n- BugFix: issue #83 - `z?` works on OSX\n- BugFix: on SheetsSheets itself now does nothing\n- Move from readthedocs to visidata.org\n\n\n## 0.97 (2017-10-05)\n- Features\n - [replay]\n - move vdplay into vd --play\n - -p --play now replays scripts live\n - --delay interspaces replay by delay seconds\n - --batch to replay without interface\n - --output to save at end of replay\n - --replay-movement=True has --play move the cursor cell-by-cell\n - -y --confirm-overwrite=False\n - replay scripts can be strformatted with field=value\n - add ^U command to pause/resume playback\n - add ^K to cancel replay\n - add Space command to go to next step of replay while paused\n - [global]\n - remap toggle to 't' (was Space)\n - remap ^Y to push sheet of cursorRow\n - 'A' creates new sheet with N empty columns\n - remap 'r' to regex search of row key\n - add zr/zc to go to row/col number\n - F1/z? now launches man page\n - gF1/gz? now launches commands sheet\n - add `f` command to fill empty cells with the content of a non-empty cell up the current column\n - add Del/gDel to set value(s) to None\n - remove TAB/Shift-TAB sheet cycling\n - add z^ command to set current column name to current cell value\n - add gz^ to set current column name to cell values in selected rows\n - add 'z=' to show computed expression over current row\n - z' adds cache to current column (gz' for all columns)\n - `gh` moves cursor to leftmost column (instead of leftmost non-key column)\n - [aggregators]\n - allow multiple aggregators\n - 'g+ adds an aggregator to selected columns on columns sheets\n - sets the exact set of aggregators on the column sheet with 'e'/'ge'\n - 'z+' displays result of aggregation over selected rows for current column on status\n - rework aggregators so multiple aggregators can be set\n - [sheets sheet]\n - '&' on Sheets sheet is now sole join sheet command; jointype is input directly\n - add sheet concat\n - [columns sheet]\n - remap ~!@#$%^ on Columns sheet to behave like they do on other sheets\n - add g prefix to ~!@#$%^ to operate on all 'selected rows' on Columns sheet (thus modifying column parameters on source sheet)\n - [textsheet]\n - add 'w' command on TextSheets to toggle wrap\n - [cmdlog]\n - editlog renamed to cmdlog\n - cmdlog has a new format which minimises recordings of movement commands\n - '^D' now saves cmdlog sheet\n - [pivot]\n - zEnter pushes this cell; Enter pushes whole row\n - [describe]\n - add DescribeSheet with 'I' command for viewing descriptive statistics\n - add zs/zu/zt/zEnter commands to engage with rows on source sheet which are being described in current cell of describe sheet\n - [frequency]\n - 'zF' provides summary aggregation\n - [metasheets]\n - add hidden source column to metasheets\n - ^P view status history\n - [loaders]\n - add 'postgres' schema for simple loader from postgres:// url\n - add gEnter for .zip file mass open\n - add 'fixed' filetype to use fixed column detector\n - [clipboard]\n - remove `B` clipboard sheet\n - rework all d/y/p commands for only one buffer\n - remove g^Z and gp\n - [options]\n - remove -d debug option\n - add --diff to add colorizer against base sheet\n - diffs a pair of tsvs cell-by-cell\n - theme options removed as CLI arguments (still available for .visidatarc or apps)\n - `'` appends frozen column\n - rename and reorder options\n- Community\n - [docs]\n - replace .rst userguide with VisiData [man page](http://visidata.org/man)\n - [visidata.org]\n - update index.html\n - automate creation of tour pages from tours.vd\n - tours will be played and recorded using asciinema\n - then compiled into a .html with mkdemo.py for http://visidata.org/tour\n - upload html version of manpage\n- Internals\n - renamed toplevel command() to globalCommand(); removed Sheet.command(); sheet commands now specified in Sheet.commands list of Command() objects at class level\n - setter API now (sheet,col,row,value)\n - move `visidata/addons/*.py` into toplevel package\n\n## 0.96 (2017-08-21)\n- data can be piped through stdin\n- remap: `N` is now previous match (instead of `p`)\n- `:` now regex split\n- add `bin/viewtsv` example tsv viewer as an example of a small vdtui application\n- add `options.cmd_after_edit` for automove after edit\n- add clipboard functionality\n - `y` yanks row at cursor to clipboard; `gy` copies all selected rows\n - `d` deletes row and move to clipboard; `gd` moves all selected rows\n - `p` now pastes the row most recently added to the clipboard after current row; `gp` pastes all rows from clipboard after current row\n - `Shift-B` opens clipboard sheet\n - `Ctrl+z` now undoes the most recent delete; `gCtrl+z` undoes all deletes\n- Fix cursor row highlighting of identical rows\n\n## v0.95.2\n- move some functionality out of vdtui into separate python files\n- add Ctrl+z command to launch external $EDITOR\n- add ``options.force_valid_names``\n\n## v0.94 (2017-08-02)\n- add options.textwrap for TextSheet\n- add vd.remove(sheet)\n- Sheet.sources now modifiable\n\n## v0.93 (2017-07-30)\n- fix display/feel bugs in editText\n- remove BACKSPACE for editlog undo\n- fix colorizer API\n- add `ctrl-u` command to toggle profiling of main thread\n- fix `C`olumn statistics (`options.col_stats` still disabled by default)\n\n## v0.92 (2017-07-11)\n- `F`requency sheet groups numeric data into intervals\n - added `histogram_bins` and `histogram_even_interval` options\n - added `w` command on the sheet that toggles `histogram_even_interval`\n- change key for 'eval Python expression as new pyobj sheet' from Ctrl+O to Ctrl+X\n\n## v0.91 (2017-06-28)\n- make options automatically typed based on default\n- documentation cleanups\n- remove R command (set filetype on CLI)\n\n## v0.80\n- tour of screenshot.gif\n- regex transform now `*` (';' is still regex split)\n- Make regex search/select to work more like vim\n- Move several non-essential commands out of vd.py\n- change license of vd.py to MIT\n- vdtutor start\n- currency type with `$`; str type moved to `~`; remove type autodetect\n- www/ for landing page\n- move from .md to .rst for documentation\n\n## v0.61 (2017-06-12)\n- colorizers\n- `g[` and `g]` to sort by all key columns\n- `;` and `.` experimental regex commands\n\n## v0.59 (2017-05-31)\n- pivot sheets with `W`\n- undo with `BACKSPACE` and replay with `ga`\n- dev guide and user guide\n- `ge` mass edit\n- freeze with `g'`\n\n## v0.44\n- creating sustainable dev process at RC\n- `z` scrolling prefix\n\n## v0.42\n- async select/unselect\n- aggregator functions on columns\n- .xls\n\n## v0.41 (2017-01-18)\n- asynchronous commands (each in its own thread) with\n - `^T` sheet of long-running commands\n - `^C` cancel\n - `ENTER` to see the final performance profile\n- `P` random population of current sheet\n- headerlines default now 1\n\n## v0.40\n- options settable with command-line arguments (`--encoding=cp437`)\n- input() histories with UP/DOWN (and viewable with `I`)\n- unicode input now works\n- editText clears value on first typing\n- `\"` duplicates sheet with only selected rows; `g\"` duplicates entire sheet verbatim\n\n## v0.38\n- sortable date\n- open_zip comes back\n\n## v0.37\n- `g~` (autodetect all columns)\n- `\"` copies row to immediately following\n- nulls, uniques on columns sheet\n\n## v0.36\n- right column\n- regex subst\n- unreverse [/] sort keys ([ = ascending)\n\n## v0.35 (2016-12-04)\n- reverse [/] sort keys\n- goto `r`ow by number or `c`olumn by name\n\n## v0.33\n- type detection with `~`\n- date type\n- fix outer join\n\n## v0.32\n- expose col.type in column header\n- push value conversion to time of usage/display\n\n## v0.31\n- F1 help sheet\n- ^O directly exposes eval result as sheet\n- custom editText with initial value, ESC that raises VEscape, and readline edit keys\n\n## v0.30 (2016-11-27)\n- make all sheets subclasses of VSheet\n- remove .zip opening and url fetching\n- added options ColumnStats and csv_header\n\n## v0.29\n- pin key columns to left\n- join sheets on exact key match\n- -r/--readonly mode\n\n## v0.28 (2016-11-22)\n- inputs: .csv, .tsv, .json, .hdf5, .xlsx, .zip\n- outputs: .csv, .tsv\n- hjkl cursor movement, t/m/b scroll to position screen cursor\n- skip up/down columns by value\n- row/column reordering and deleting\n- resize column to fix max width of onscreen row\n- filter/search by regex in column\n- sort asc/desc by one column\n- `g`lobal prefix supersizes many commands\n- `e`dit cell contents\n- convert column to int/str/float\n- reload sheet with different format options\n- add new column by Python expression\n- `s`elect/`u`nselect rows, bulk delete with `gd`\n- `F`requency table for current column with histogram\n- `S`heets metasheet to manage/navigate multiple sheets,\n- `C`olumns metasheet\n- `O`ptions sheet to change the style or behavior\n- `E`rror metasheet\n- `g^P` status history sheet\n\n## v0.14 (2016-11-13)\n" }, { "path": "CLAUDE.md", "content": "# VisiData Development Guide\n\nQuick reference for VisiData development. For detailed coding patterns, conventions, and best practices, see **[dev/STYLE.md](dev/STYLE.md)**.\n\n`CLAUDE.md` and `AGENTS.md` are complementary: use this file for primary contributor workflow and architecture context, and use `AGENTS.md` for concise agent-oriented repository guidance.\n\n## Important Note About AI Usage\n\nVisiData (created in 2016) is 99% written by humans and is NOT a vibe-coded AI project. This file is meant to allow AI-assisted development of features and bugfixes. **All code must be reviewed and approved and tested by a human before being merged into the codebase or submitted as a PR.**\n\n## Repository Structure\n\n```\nvisidata/\n├── visidata/ # Main package\n│ ├── *.py # Core modules (sheet.py, column.py, etc.)\n│ ├── features/ # Auto-loaded feature plugins\n│ ├── loaders/ # File format loaders\n│ ├── apps/ # Standalone applications\n│ └── experimental/ # Experimental features (load/install with 'import visidata.experimental.foo')\n├── tests/ # Test files\n├── docs/ # Documentation\n└── dev/ # Development utilities and docs\n```\n\n## Features Directory (`visidata/features/`)\n\n- All `.py` files in this directory are **automatically imported** when VisiData starts\n- Each feature file should be self-contained\n- Features extend VisiData functionality without modifying core files\n\n## Quick Reference\n\n### Core Classes\n- `BaseSheet` - Minimal sheet functionality\n- `Sheet` / `TableSheet` - Sheet with columns and rows (most common)\n- `Column` - Column definition with getter/setter\n\n### Adding Commands\n```python\nBaseSheet.addCommand('', 'command-name', 'code', 'help text')\n```\n\n### Adding to Global Namespace\n```python\nvd.addGlobals(MyClass=MyClass) # Use keyword args, not dict\n```\n\n### Adding Menu Items\n```python\nvd.addMenuItems('''\n Menu > Submenu > Item Name > command-name\n''')\n```\n\n### Example Feature Structure\n```python\nfrom visidata import vd, Sheet, Column\n\n# rowdef: description of what a row represents\nclass MySheet(Sheet):\n rowtype = 'items'\n columns = [\n Column('name', getter=lambda c,r: r.attribute),\n ]\n\n def reload(self):\n self.rows = [...]\n\nBaseSheet.addCommand('', 'my-command', 'code', 'help')\nvd.addGlobals(MySheet=MySheet)\n```\n\n## Development Workflow\n\n1. Add `.py` file to `visidata/features/`\n2. Run `vd` and test interactively\n3. Iterate and refine\n4. Document with docstrings and comments\n\n## Make Targets\n\n- `make test` — run all tests\n- `make help` — list all targets\n\n## Documentation\n\nFor comprehensive development documentation, see the `dev/` directory:\n\n### [dev/STYLE.md](dev/STYLE.md) - Coding Style and Patterns\nUse this when writing code, creating features, or defining sheets and columns.\n- Naming conventions (camelCaps, under_score, etc.)\n- Feature file structure and patterns\n- Sheet and Column class patterns\n- Command and menu integration\n- API decorators\n- Best practices and examples\n\n### [dev/GIT.md](dev/GIT.md) - Version Control Practices\nUse this when making commits or preparing pull requests.\n- Commit message format and conventions\n- Issue tracking in code\n- Branch and merge workflow\n- Patch-safe commit marking\n\n### [dev/DOCS.md](dev/DOCS.md) - Documentation Writing\nUse this when writing user-facing documentation, help text, or in-app guides.\n- VisiData's markdown syntax\n- Display attribute syntax (colors, clickable links)\n- Option and command reference format\n- Technical writing guidelines\n\n### [dev/PERFORMANCE.md](dev/PERFORMANCE.md) - Performance Analysis\nUse this when investigating or optimizing performance issues.\n- Finding reproducible performance issues\n- Profiling techniques and tools\n- Analyzing profiling results\n- Optimization workflow\n\n### [dev/OPTIONS.md](dev/OPTIONS.md) - Options System\nUse this when working with options, adding new options, or understanding how configuration resolves.\n- Resolution chain (instance → class → global → default)\n- How sheets and paths participate in options\n- Setting and reading options at different levels\n\n## Updating Documentation\n\nWhen making **user-facing changes** (new commands, changed behavior, new options, new/changed loaders, UI changes), check [docs/README.md](docs/README.md) to identify which documentation files need to be updated.\n" }, { "path": "CODE_OF_CONDUCT.md", "content": "Please don't be a dick ♥\n" }, { "path": "CONTRIBUTING.md", "content": "# Contributing\n\n## Spread the Word\n\nThe single best way you can contribute, is to share your enthusiasm about VisiData with other people.\nA vibrant community is essential to its sustainable development.\n\nHowever, direct and forceful promotion is probably not the most effective approach for a tool like VisiData.\nPeople generally need to be exposed several times and from several sources before they will try some terminal utility they've never heard of before.\n\nSome people are interested, but are daunted by the installation process or the interface; you can [help them get it installed](/install), and provide a few pointers to get started.\nDon't make it too complicated or overload them with too many features.\nStick to the basics: arrow keys, quit, help, search, sort, freq table.\n\nWe also need people to mention VisiData in their forums and communities that relate to data and terminal programs.\nDon't spam or do a drive-by promotion; these are largely ineffective and will often be received negatively.\nEndorsements have more weight from people who actively post about other relevant topics; we don't want to become the \"VisiData Brigade\".\n\nFinally, if you are on \"Web 2.social\", you can post a [tweet](https://twitter.com/visidata) or a [tutorial]() or a [demo](https://www.youtube.com/watch?v=N1CBDTgGtOU) or [host a workshop](https://www.meetup.com/pt-BR/Journocoders/events/258035880/), or anything else you think might make people interested in exploring the wonderful world of VisiData.\n\n## Support on Patreon\n\nIf VisiData saves you time on a regular basis, and especially if VisiData makes your paid work easier, please contribute to [my Patreon](https://www.patreon.com/saulpw).\n\n## Start a Project Using VisiData\n\nIf you know Python and want to augment it to suit your own workflow, you can create a loader or a plugin. In support of this, I have written [a detailed api guide for VisiData](https://www.visidata.org/docs/api/).\n\nHere are some great examples:\n\n - [jsvine's custom visidata plugins](https://github.com/jsvine/visidata-plugins)\n - [layertwo's pcap loader](https://github.com/saulpw/visidata/blob/develop/visidata/loaders/pcap.py)\n\nWithout fail, these projects lead to discovering bugs and help flesh out the API, which result in design improvements in VisiData.\nImportantly, each issue found this way comes with real world motivations, so it is easy to explain your reasoning behind proposals and core feature requests.\n\n## Feature Requests\n\nVisiData is designed to be extensible, and most feature requests can be implemented as a one line command, or a tiny snippet of code to include in a `.visidatarc`.\n\nIf this would require changes to the VisiData core, and a reasonable design is approved, then the issue can stay open until the core changes have been made.\nOtherwise, in the spirit of Marie Kondo, the issue will be closed without prejudice.\n\nFeature requests with some amount of working Python code are more likely to get attention.\nDesign proposals with concrete use cases are very welcome.\n\n## Writing a well constructed bug report\n\nIf you encounter any bugs or have any problems, please [create an issue on GitHub](https://github.com/saulpw/visidata/issues).\n\nA great bug report will include:\n\n - a stacktrace, if there is an unexpected error; the most recent full stack traces can be viewed with `Ctrl+E` (then saved with `Ctrl+S`)\n - a [.vd](http://visidata.org/docs/save-restore/) and sample dataset that reproduces the issue\n - a .png/.gif (esp. for user interface changes)\n\nSome examples of great bug reports:\n\n - [#350 by @chocolateboy](https://github.com/saulpw/visidata/issues/350)\n - [#340 by @Mikee-3000](https://github.com/saulpw/visidata/issues/340)\n\n\n## Setting Up Git Hooks\n\nVisiData includes git hooks in `dev/hooks/` that run unit tests and cmdlog tests before pushing. To enable them:\n\n```\ngit config core.hooksPath dev/hooks\n```\n\n## Submitting Source Code\n\nCheck out the [Plugin Authors Guide](https://visidata.org/docs/api) for an overview of the API.\nCode in `visidata/features/` or `visidata/loaders/` is generally welcome, as long as it is useful to someone and safe for everyone.\nUpdates or additions to the core code should be proposed via an [Github Issue](https://github.com/saulpw/visidata/issues/new/choose) before submitting a PR.\n\nVisiData has two main branches:\n\n - [stable](https://github.com/saulpw/visidata/tree/stable) has the last known good version of VisiData (what is in pypi/brew/apt).\n - [develop](https://github.com/saulpw/visidata/tree/develop) has the most up-to-date version of VisiData (which will eventually be merged to stable).\n\nAll pull requests should be submitted against `develop`. Submitters will need to e-sign a [Copyright Assignment Agreement (CAA)](https://visidata.org/caa) before a pull request will be accepted.\n\n# Open Source License and Copyright\n\nVisiData is an open-source utility that can be installed and used for free (under the terms of the [GPL3](https://www.gnu.org/licenses/gpl-3.0.en.html)).\n\nThe core VisiData utility and rendering library will always be both free and libre.\n\nAs the copyright holder, Saul Pwanson has the authority to negotiate other license terms.\n\n**By submitting changes to this repository, you acknowledge that you assign copyright to the owner of the repository ([Saul Pwanson ](mailto:vd@saul.pw)).**\n" }, { "path": "Dockerfile.alpine", "content": "FROM python:3.8-alpine\n\nRUN pip install requests python-dateutil wcwidth\n\nRUN mkdir -p /opt/visidata\nWORKDIR /opt/visidata\nCOPY . ./\nRUN sh -c 'yes | pip install -vvv .'\n\nENV TERM=\"xterm-256color\"\nENTRYPOINT bin/vd\n" }, { "path": "Dockerfile.darkdraw.alpine", "content": "FROM visidata\n\nRUN apk add git\nRUN pip install git+https://github.com/devottys/darkdraw.git@master\nRUN sh -c \"echo >>~/.visidatarc import darkdraw\"\n\nENV TERM=\"xterm-256color\"\nENTRYPOINT [\"/opt/visidata/bin/vd\", \"-f\", \"ddw\"]\n" }, { "path": "LICENSE.gpl3", "content": " GNU GENERAL PUBLIC LICENSE\n Version 3, 29 June 2007\n\n Copyright (C) 2007 Free Software Foundation, Inc. \n Everyone is permitted to copy and distribute verbatim copies\n of this license document, but changing it is not allowed.\n\n Preamble\n\n The GNU General Public License is a free, copyleft license for\nsoftware and other kinds of works.\n\n The licenses for most software and other practical works are designed\nto take away your freedom to share and change the works. By contrast,\nthe GNU General Public License is intended to guarantee your freedom to\nshare and change all versions of a program--to make sure it remains free\nsoftware for all its users. We, the Free Software Foundation, use the\nGNU General Public License for most of our software; it applies also to\nany other work released this way by its authors. You can apply it to\nyour programs, too.\n\n When we speak of free software, we are referring to freedom, not\nprice. Our General Public Licenses are designed to make sure that you\nhave the freedom to distribute copies of free software (and charge for\nthem if you wish), that you receive source code or can get it if you\nwant it, that you can change the software or use pieces of it in new\nfree programs, and that you know you can do these things.\n\n To protect your rights, we need to prevent others from denying you\nthese rights or asking you to surrender the rights. Therefore, you have\ncertain responsibilities if you distribute copies of the software, or if\nyou modify it: responsibilities to respect the freedom of others.\n\n For example, if you distribute copies of such a program, whether\ngratis or for a fee, you must pass on to the recipients the same\nfreedoms that you received. You must make sure that they, too, receive\nor can get the source code. And you must show them these terms so they\nknow their rights.\n\n Developers that use the GNU GPL protect your rights with two steps:\n(1) assert copyright on the software, and (2) offer you this License\ngiving you legal permission to copy, distribute and/or modify it.\n\n For the developers' and authors' protection, the GPL clearly explains\nthat there is no warranty for this free software. For both users' and\nauthors' sake, the GPL requires that modified versions be marked as\nchanged, so that their problems will not be attributed erroneously to\nauthors of previous versions.\n\n Some devices are designed to deny users access to install or run\nmodified versions of the software inside them, although the manufacturer\ncan do so. This is fundamentally incompatible with the aim of\nprotecting users' freedom to change the software. The systematic\npattern of such abuse occurs in the area of products for individuals to\nuse, which is precisely where it is most unacceptable. Therefore, we\nhave designed this version of the GPL to prohibit the practice for those\nproducts. If such problems arise substantially in other domains, we\nstand ready to extend this provision to those domains in future versions\nof the GPL, as needed to protect the freedom of users.\n\n Finally, every program is threatened constantly by software patents.\nStates should not allow patents to restrict development and use of\nsoftware on general-purpose computers, but in those that do, we wish to\navoid the special danger that patents applied to a free program could\nmake it effectively proprietary. To prevent this, the GPL assures that\npatents cannot be used to render the program non-free.\n\n The precise terms and conditions for copying, distribution and\nmodification follow.\n\n TERMS AND CONDITIONS\n\n 0. Definitions.\n\n \"This License\" refers to version 3 of the GNU General Public License.\n\n \"Copyright\" also means copyright-like laws that apply to other kinds of\nworks, such as semiconductor masks.\n\n \"The Program\" refers to any copyrightable work licensed under this\nLicense. Each licensee is addressed as \"you\". \"Licensees\" and\n\"recipients\" may be individuals or organizations.\n\n To \"modify\" a work means to copy from or adapt all or part of the work\nin a fashion requiring copyright permission, other than the making of an\nexact copy. The resulting work is called a \"modified version\" of the\nearlier work or a work \"based on\" the earlier work.\n\n A \"covered work\" means either the unmodified Program or a work based\non the Program.\n\n To \"propagate\" a work means to do anything with it that, without\npermission, would make you directly or secondarily liable for\ninfringement under applicable copyright law, except executing it on a\ncomputer or modifying a private copy. Propagation includes copying,\ndistribution (with or without modification), making available to the\npublic, and in some countries other activities as well.\n\n To \"convey\" a work means any kind of propagation that enables other\nparties to make or receive copies. Mere interaction with a user through\na computer network, with no transfer of a copy, is not conveying.\n\n An interactive user interface displays \"Appropriate Legal Notices\"\nto the extent that it includes a convenient and prominently visible\nfeature that (1) displays an appropriate copyright notice, and (2)\ntells the user that there is no warranty for the work (except to the\nextent that warranties are provided), that licensees may convey the\nwork under this License, and how to view a copy of this License. If\nthe interface presents a list of user commands or options, such as a\nmenu, a prominent item in the list meets this criterion.\n\n 1. Source Code.\n\n The \"source code\" for a work means the preferred form of the work\nfor making modifications to it. \"Object code\" means any non-source\nform of a work.\n\n A \"Standard Interface\" means an interface that either is an official\nstandard defined by a recognized standards body, or, in the case of\ninterfaces specified for a particular programming language, one that\nis widely used among developers working in that language.\n\n The \"System Libraries\" of an executable work include anything, other\nthan the work as a whole, that (a) is included in the normal form of\npackaging a Major Component, but which is not part of that Major\nComponent, and (b) serves only to enable use of the work with that\nMajor Component, or to implement a Standard Interface for which an\nimplementation is available to the public in source code form. A\n\"Major Component\", in this context, means a major essential component\n(kernel, window system, and so on) of the specific operating system\n(if any) on which the executable work runs, or a compiler used to\nproduce the work, or an object code interpreter used to run it.\n\n The \"Corresponding Source\" for a work in object code form means all\nthe source code needed to generate, install, and (for an executable\nwork) run the object code and to modify the work, including scripts to\ncontrol those activities. However, it does not include the work's\nSystem Libraries, or general-purpose tools or generally available free\nprograms which are used unmodified in performing those activities but\nwhich are not part of the work. For example, Corresponding Source\nincludes interface definition files associated with source files for\nthe work, and the source code for shared libraries and dynamically\nlinked subprograms that the work is specifically designed to require,\nsuch as by intimate data communication or control flow between those\nsubprograms and other parts of the work.\n\n The Corresponding Source need not include anything that users\ncan regenerate automatically from other parts of the Corresponding\nSource.\n\n The Corresponding Source for a work in source code form is that\nsame work.\n\n 2. Basic Permissions.\n\n All rights granted under this License are granted for the term of\ncopyright on the Program, and are irrevocable provided the stated\nconditions are met. This License explicitly affirms your unlimited\npermission to run the unmodified Program. The output from running a\ncovered work is covered by this License only if the output, given its\ncontent, constitutes a covered work. This License acknowledges your\nrights of fair use or other equivalent, as provided by copyright law.\n\n You may make, run and propagate covered works that you do not\nconvey, without conditions so long as your license otherwise remains\nin force. You may convey covered works to others for the sole purpose\nof having them make modifications exclusively for you, or provide you\nwith facilities for running those works, provided that you comply with\nthe terms of this License in conveying all material for which you do\nnot control copyright. Those thus making or running the covered works\nfor you must do so exclusively on your behalf, under your direction\nand control, on terms that prohibit them from making any copies of\nyour copyrighted material outside their relationship with you.\n\n Conveying under any other circumstances is permitted solely under\nthe conditions stated below. Sublicensing is not allowed; section 10\nmakes it unnecessary.\n\n 3. Protecting Users' Legal Rights From Anti-Circumvention Law.\n\n No covered work shall be deemed part of an effective technological\nmeasure under any applicable law fulfilling obligations under article\n11 of the WIPO copyright treaty adopted on 20 December 1996, or\nsimilar laws prohibiting or restricting circumvention of such\nmeasures.\n\n When you convey a covered work, you waive any legal power to forbid\ncircumvention of technological measures to the extent such circumvention\nis effected by exercising rights under this License with respect to\nthe covered work, and you disclaim any intention to limit operation or\nmodification of the work as a means of enforcing, against the work's\nusers, your or third parties' legal rights to forbid circumvention of\ntechnological measures.\n\n 4. Conveying Verbatim Copies.\n\n You may convey verbatim copies of the Program's source code as you\nreceive it, in any medium, provided that you conspicuously and\nappropriately publish on each copy an appropriate copyright notice;\nkeep intact all notices stating that this License and any\nnon-permissive terms added in accord with section 7 apply to the code;\nkeep intact all notices of the absence of any warranty; and give all\nrecipients a copy of this License along with the Program.\n\n You may charge any price or no price for each copy that you convey,\nand you may offer support or warranty protection for a fee.\n\n 5. Conveying Modified Source Versions.\n\n You may convey a work based on the Program, or the modifications to\nproduce it from the Program, in the form of source code under the\nterms of section 4, provided that you also meet all of these conditions:\n\n a) The work must carry prominent notices stating that you modified\n it, and giving a relevant date.\n\n b) The work must carry prominent notices stating that it is\n released under this License and any conditions added under section\n 7. This requirement modifies the requirement in section 4 to\n \"keep intact all notices\".\n\n c) You must license the entire work, as a whole, under this\n License to anyone who comes into possession of a copy. This\n License will therefore apply, along with any applicable section 7\n additional terms, to the whole of the work, and all its parts,\n regardless of how they are packaged. This License gives no\n permission to license the work in any other way, but it does not\n invalidate such permission if you have separately received it.\n\n d) If the work has interactive user interfaces, each must display\n Appropriate Legal Notices; however, if the Program has interactive\n interfaces that do not display Appropriate Legal Notices, your\n work need not make them do so.\n\n A compilation of a covered work with other separate and independent\nworks, which are not by their nature extensions of the covered work,\nand which are not combined with it such as to form a larger program,\nin or on a volume of a storage or distribution medium, is called an\n\"aggregate\" if the compilation and its resulting copyright are not\nused to limit the access or legal rights of the compilation's users\nbeyond what the individual works permit. Inclusion of a covered work\nin an aggregate does not cause this License to apply to the other\nparts of the aggregate.\n\n 6. Conveying Non-Source Forms.\n\n You may convey a covered work in object code form under the terms\nof sections 4 and 5, provided that you also convey the\nmachine-readable Corresponding Source under the terms of this License,\nin one of these ways:\n\n a) Convey the object code in, or embodied in, a physical product\n (including a physical distribution medium), accompanied by the\n Corresponding Source fixed on a durable physical medium\n customarily used for software interchange.\n\n b) Convey the object code in, or embodied in, a physical product\n (including a physical distribution medium), accompanied by a\n written offer, valid for at least three years and valid for as\n long as you offer spare parts or customer support for that product\n model, to give anyone who possesses the object code either (1) a\n copy of the Corresponding Source for all the software in the\n product that is covered by this License, on a durable physical\n medium customarily used for software interchange, for a price no\n more than your reasonable cost of physically performing this\n conveying of source, or (2) access to copy the\n Corresponding Source from a network server at no charge.\n\n c) Convey individual copies of the object code with a copy of the\n written offer to provide the Corresponding Source. This\n alternative is allowed only occasionally and noncommercially, and\n only if you received the object code with such an offer, in accord\n with subsection 6b.\n\n d) Convey the object code by offering access from a designated\n place (gratis or for a charge), and offer equivalent access to the\n Corresponding Source in the same way through the same place at no\n further charge. You need not require recipients to copy the\n Corresponding Source along with the object code. If the place to\n copy the object code is a network server, the Corresponding Source\n may be on a different server (operated by you or a third party)\n that supports equivalent copying facilities, provided you maintain\n clear directions next to the object code saying where to find the\n Corresponding Source. Regardless of what server hosts the\n Corresponding Source, you remain obligated to ensure that it is\n available for as long as needed to satisfy these requirements.\n\n e) Convey the object code using peer-to-peer transmission, provided\n you inform other peers where the object code and Corresponding\n Source of the work are being offered to the general public at no\n charge under subsection 6d.\n\n A separable portion of the object code, whose source code is excluded\nfrom the Corresponding Source as a System Library, need not be\nincluded in conveying the object code work.\n\n A \"User Product\" is either (1) a \"consumer product\", which means any\ntangible personal property which is normally used for personal, family,\nor household purposes, or (2) anything designed or sold for incorporation\ninto a dwelling. In determining whether a product is a consumer product,\ndoubtful cases shall be resolved in favor of coverage. For a particular\nproduct received by a particular user, \"normally used\" refers to a\ntypical or common use of that class of product, regardless of the status\nof the particular user or of the way in which the particular user\nactually uses, or expects or is expected to use, the product. A product\nis a consumer product regardless of whether the product has substantial\ncommercial, industrial or non-consumer uses, unless such uses represent\nthe only significant mode of use of the product.\n\n \"Installation Information\" for a User Product means any methods,\nprocedures, authorization keys, or other information required to install\nand execute modified versions of a covered work in that User Product from\na modified version of its Corresponding Source. The information must\nsuffice to ensure that the continued functioning of the modified object\ncode is in no case prevented or interfered with solely because\nmodification has been made.\n\n If you convey an object code work under this section in, or with, or\nspecifically for use in, a User Product, and the conveying occurs as\npart of a transaction in which the right of possession and use of the\nUser Product is transferred to the recipient in perpetuity or for a\nfixed term (regardless of how the transaction is characterized), the\nCorresponding Source conveyed under this section must be accompanied\nby the Installation Information. But this requirement does not apply\nif neither you nor any third party retains the ability to install\nmodified object code on the User Product (for example, the work has\nbeen installed in ROM).\n\n The requirement to provide Installation Information does not include a\nrequirement to continue to provide support service, warranty, or updates\nfor a work that has been modified or installed by the recipient, or for\nthe User Product in which it has been modified or installed. Access to a\nnetwork may be denied when the modification itself materially and\nadversely affects the operation of the network or violates the rules and\nprotocols for communication across the network.\n\n Corresponding Source conveyed, and Installation Information provided,\nin accord with this section must be in a format that is publicly\ndocumented (and with an implementation available to the public in\nsource code form), and must require no special password or key for\nunpacking, reading or copying.\n\n 7. Additional Terms.\n\n \"Additional permissions\" are terms that supplement the terms of this\nLicense by making exceptions from one or more of its conditions.\nAdditional permissions that are applicable to the entire Program shall\nbe treated as though they were included in this License, to the extent\nthat they are valid under applicable law. If additional permissions\napply only to part of the Program, that part may be used separately\nunder those permissions, but the entire Program remains governed by\nthis License without regard to the additional permissions.\n\n When you convey a copy of a covered work, you may at your option\nremove any additional permissions from that copy, or from any part of\nit. (Additional permissions may be written to require their own\nremoval in certain cases when you modify the work.) You may place\nadditional permissions on material, added by you to a covered work,\nfor which you have or can give appropriate copyright permission.\n\n Notwithstanding any other provision of this License, for material you\nadd to a covered work, you may (if authorized by the copyright holders of\nthat material) supplement the terms of this License with terms:\n\n a) Disclaiming warranty or limiting liability differently from the\n terms of sections 15 and 16 of this License; or\n\n b) Requiring preservation of specified reasonable legal notices or\n author attributions in that material or in the Appropriate Legal\n Notices displayed by works containing it; or\n\n c) Prohibiting misrepresentation of the origin of that material, or\n requiring that modified versions of such material be marked in\n reasonable ways as different from the original version; or\n\n d) Limiting the use for publicity purposes of names of licensors or\n authors of the material; or\n\n e) Declining to grant rights under trademark law for use of some\n trade names, trademarks, or service marks; or\n\n f) Requiring indemnification of licensors and authors of that\n material by anyone who conveys the material (or modified versions of\n it) with contractual assumptions of liability to the recipient, for\n any liability that these contractual assumptions directly impose on\n those licensors and authors.\n\n All other non-permissive additional terms are considered \"further\nrestrictions\" within the meaning of section 10. If the Program as you\nreceived it, or any part of it, contains a notice stating that it is\ngoverned by this License along with a term that is a further\nrestriction, you may remove that term. If a license document contains\na further restriction but permits relicensing or conveying under this\nLicense, you may add to a covered work material governed by the terms\nof that license document, provided that the further restriction does\nnot survive such relicensing or conveying.\n\n If you add terms to a covered work in accord with this section, you\nmust place, in the relevant source files, a statement of the\nadditional terms that apply to those files, or a notice indicating\nwhere to find the applicable terms.\n\n Additional terms, permissive or non-permissive, may be stated in the\nform of a separately written license, or stated as exceptions;\nthe above requirements apply either way.\n\n 8. Termination.\n\n You may not propagate or modify a covered work except as expressly\nprovided under this License. Any attempt otherwise to propagate or\nmodify it is void, and will automatically terminate your rights under\nthis License (including any patent licenses granted under the third\nparagraph of section 11).\n\n However, if you cease all violation of this License, then your\nlicense from a particular copyright holder is reinstated (a)\nprovisionally, unless and until the copyright holder explicitly and\nfinally terminates your license, and (b) permanently, if the copyright\nholder fails to notify you of the violation by some reasonable means\nprior to 60 days after the cessation.\n\n Moreover, your license from a particular copyright holder is\nreinstated permanently if the copyright holder notifies you of the\nviolation by some reasonable means, this is the first time you have\nreceived notice of violation of this License (for any work) from that\ncopyright holder, and you cure the violation prior to 30 days after\nyour receipt of the notice.\n\n Termination of your rights under this section does not terminate the\nlicenses of parties who have received copies or rights from you under\nthis License. If your rights have been terminated and not permanently\nreinstated, you do not qualify to receive new licenses for the same\nmaterial under section 10.\n\n 9. Acceptance Not Required for Having Copies.\n\n You are not required to accept this License in order to receive or\nrun a copy of the Program. Ancillary propagation of a covered work\noccurring solely as a consequence of using peer-to-peer transmission\nto receive a copy likewise does not require acceptance. However,\nnothing other than this License grants you permission to propagate or\nmodify any covered work. These actions infringe copyright if you do\nnot accept this License. Therefore, by modifying or propagating a\ncovered work, you indicate your acceptance of this License to do so.\n\n 10. Automatic Licensing of Downstream Recipients.\n\n Each time you convey a covered work, the recipient automatically\nreceives a license from the original licensors, to run, modify and\npropagate that work, subject to this License. You are not responsible\nfor enforcing compliance by third parties with this License.\n\n An \"entity transaction\" is a transaction transferring control of an\norganization, or substantially all assets of one, or subdividing an\norganization, or merging organizations. If propagation of a covered\nwork results from an entity transaction, each party to that\ntransaction who receives a copy of the work also receives whatever\nlicenses to the work the party's predecessor in interest had or could\ngive under the previous paragraph, plus a right to possession of the\nCorresponding Source of the work from the predecessor in interest, if\nthe predecessor has it or can get it with reasonable efforts.\n\n You may not impose any further restrictions on the exercise of the\nrights granted or affirmed under this License. For example, you may\nnot impose a license fee, royalty, or other charge for exercise of\nrights granted under this License, and you may not initiate litigation\n(including a cross-claim or counterclaim in a lawsuit) alleging that\nany patent claim is infringed by making, using, selling, offering for\nsale, or importing the Program or any portion of it.\n\n 11. Patents.\n\n A \"contributor\" is a copyright holder who authorizes use under this\nLicense of the Program or a work on which the Program is based. The\nwork thus licensed is called the contributor's \"contributor version\".\n\n A contributor's \"essential patent claims\" are all patent claims\nowned or controlled by the contributor, whether already acquired or\nhereafter acquired, that would be infringed by some manner, permitted\nby this License, of making, using, or selling its contributor version,\nbut do not include claims that would be infringed only as a\nconsequence of further modification of the contributor version. For\npurposes of this definition, \"control\" includes the right to grant\npatent sublicenses in a manner consistent with the requirements of\nthis License.\n\n Each contributor grants you a non-exclusive, worldwide, royalty-free\npatent license under the contributor's essential patent claims, to\nmake, use, sell, offer for sale, import and otherwise run, modify and\npropagate the contents of its contributor version.\n\n In the following three paragraphs, a \"patent license\" is any express\nagreement or commitment, however denominated, not to enforce a patent\n(such as an express permission to practice a patent or covenant not to\nsue for patent infringement). To \"grant\" such a patent license to a\nparty means to make such an agreement or commitment not to enforce a\npatent against the party.\n\n If you convey a covered work, knowingly relying on a patent license,\nand the Corresponding Source of the work is not available for anyone\nto copy, free of charge and under the terms of this License, through a\npublicly available network server or other readily accessible means,\nthen you must either (1) cause the Corresponding Source to be so\navailable, or (2) arrange to deprive yourself of the benefit of the\npatent license for this particular work, or (3) arrange, in a manner\nconsistent with the requirements of this License, to extend the patent\nlicense to downstream recipients. \"Knowingly relying\" means you have\nactual knowledge that, but for the patent license, your conveying the\ncovered work in a country, or your recipient's use of the covered work\nin a country, would infringe one or more identifiable patents in that\ncountry that you have reason to believe are valid.\n\n If, pursuant to or in connection with a single transaction or\narrangement, you convey, or propagate by procuring conveyance of, a\ncovered work, and grant a patent license to some of the parties\nreceiving the covered work authorizing them to use, propagate, modify\nor convey a specific copy of the covered work, then the patent license\nyou grant is automatically extended to all recipients of the covered\nwork and works based on it.\n\n A patent license is \"discriminatory\" if it does not include within\nthe scope of its coverage, prohibits the exercise of, or is\nconditioned on the non-exercise of one or more of the rights that are\nspecifically granted under this License. You may not convey a covered\nwork if you are a party to an arrangement with a third party that is\nin the business of distributing software, under which you make payment\nto the third party based on the extent of your activity of conveying\nthe work, and under which the third party grants, to any of the\nparties who would receive the covered work from you, a discriminatory\npatent license (a) in connection with copies of the covered work\nconveyed by you (or copies made from those copies), or (b) primarily\nfor and in connection with specific products or compilations that\ncontain the covered work, unless you entered into that arrangement,\nor that patent license was granted, prior to 28 March 2007.\n\n Nothing in this License shall be construed as excluding or limiting\nany implied license or other defenses to infringement that may\notherwise be available to you under applicable patent law.\n\n 12. No Surrender of Others' Freedom.\n\n If conditions are imposed on you (whether by court order, agreement or\notherwise) that contradict the conditions of this License, they do not\nexcuse you from the conditions of this License. If you cannot convey a\ncovered work so as to satisfy simultaneously your obligations under this\nLicense and any other pertinent obligations, then as a consequence you may\nnot convey it at all. For example, if you agree to terms that obligate you\nto collect a royalty for further conveying from those to whom you convey\nthe Program, the only way you could satisfy both those terms and this\nLicense would be to refrain entirely from conveying the Program.\n\n 13. Use with the GNU Affero General Public License.\n\n Notwithstanding any other provision of this License, you have\npermission to link or combine any covered work with a work licensed\nunder version 3 of the GNU Affero General Public License into a single\ncombined work, and to convey the resulting work. The terms of this\nLicense will continue to apply to the part which is the covered work,\nbut the special requirements of the GNU Affero General Public License,\nsection 13, concerning interaction through a network will apply to the\ncombination as such.\n\n 14. Revised Versions of this License.\n\n The Free Software Foundation may publish revised and/or new versions of\nthe GNU General Public License from time to time. Such new versions will\nbe similar in spirit to the present version, but may differ in detail to\naddress new problems or concerns.\n\n Each version is given a distinguishing version number. If the\nProgram specifies that a certain numbered version of the GNU General\nPublic License \"or any later version\" applies to it, you have the\noption of following the terms and conditions either of that numbered\nversion or of any later version published by the Free Software\nFoundation. If the Program does not specify a version number of the\nGNU General Public License, you may choose any version ever published\nby the Free Software Foundation.\n\n If the Program specifies that a proxy can decide which future\nversions of the GNU General Public License can be used, that proxy's\npublic statement of acceptance of a version permanently authorizes you\nto choose that version for the Program.\n\n Later license versions may give you additional or different\npermissions. However, no additional obligations are imposed on any\nauthor or copyright holder as a result of your choosing to follow a\nlater version.\n\n 15. Disclaimer of Warranty.\n\n THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY\nAPPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT\nHOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM \"AS IS\" WITHOUT WARRANTY\nOF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,\nTHE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR\nPURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM\nIS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF\nALL NECESSARY SERVICING, REPAIR OR CORRECTION.\n\n 16. Limitation of Liability.\n\n IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING\nWILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS\nTHE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY\nGENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE\nUSE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF\nDATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD\nPARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),\nEVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF\nSUCH DAMAGES.\n\n 17. Interpretation of Sections 15 and 16.\n\n If the disclaimer of warranty and limitation of liability provided\nabove cannot be given local legal effect according to their terms,\nreviewing courts shall apply local law that most closely approximates\nan absolute waiver of all civil liability in connection with the\nProgram, unless a warranty or assumption of liability accompanies a\ncopy of the Program in return for a fee.\n\n END OF TERMS AND CONDITIONS\n\n How to Apply These Terms to Your New Programs\n\n If you develop a new program, and you want it to be of the greatest\npossible use to the public, the best way to achieve this is to make it\nfree software which everyone can redistribute and change under these terms.\n\n To do so, attach the following notices to the program. It is safest\nto attach them to the start of each source file to most effectively\nstate the exclusion of warranty; and each file should have at least\nthe \"copyright\" line and a pointer to where the full notice is found.\n\n \n Copyright (C) \n\n This program is free software: you can redistribute it and/or modify\n it under the terms of the GNU General Public License as published by\n the Free Software Foundation, either version 3 of the License, or\n (at your option) any later version.\n\n This program is distributed in the hope that it will be useful,\n but WITHOUT ANY WARRANTY; without even the implied warranty of\n MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n GNU General Public License for more details.\n\n You should have received a copy of the GNU General Public License\n along with this program. If not, see .\n\nAlso add information on how to contact you by electronic and paper mail.\n\n If the program does terminal interaction, make it output a short\nnotice like this when it starts in an interactive mode:\n\n Copyright (C) \n This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.\n This is free software, and you are welcome to redistribute it\n under certain conditions; type `show c' for details.\n\nThe hypothetical commands `show w' and `show c' should show the appropriate\nparts of the General Public License. Of course, your program's commands\nmight be different; for a GUI interface, you would use an \"about box\".\n\n You should also get your employer (if you work as a programmer) or school,\nif any, to sign a \"copyright disclaimer\" for the program, if necessary.\nFor more information on this, and how to apply and follow the GNU GPL, see\n.\n\n The GNU General Public License does not permit incorporating your program\ninto proprietary programs. If your program is a subroutine library, you\nmay consider it more useful to permit linking proprietary applications with\nthe library. If this is what you want to do, use the GNU Lesser General\nPublic License instead of this License. But first, please read\n.\n" }, { "path": "MANIFEST.in", "content": "include README.md\ninclude LICENSE.gpl3\ninclude CHANGELOG.md\ninclude CONTRIBUTING.md\ninclude CODE_OF_CONDUCT.md\ninclude requirements.txt\ninclude requirements.scm\ninclude visidata/man/vd.1\ninclude visidata/man/vd.txt\ninclude visidata/man/visidata.1\ninclude visidata/ddw/input.ddw\ninclude visidata/ddw/regex.ddw\ninclude visidata/tests/sample.tsv\ninclude visidata/tests/benchmark.csv\ninclude visidata/desktop/visidata.desktop\ninclude visidata/desktop/org.visidata.VisiData.metainfo.xml\ninclude visidata/desktop/icons/48x48/visidata.png\ninclude visidata/desktop/icons/32x32/visidata.png\ninclude visidata/experimental/noahs_tapestry/*.json\ninclude visidata/experimental/noahs_tapestry/*.md\ninclude visidata/experimental/noahs_tapestry/*.ddw\ninclude visidata/experimental/noahs_tapestry/*.sqlite\ninclude visidata/guides/*.md\n" }, { "path": "Makefile", "content": ".PHONY: help \\\n install install-dev install-test install-all \\\n test test-all test-vgit test-vdsql \\\n build man zsh-completion docker \\\n setup-hooks setup-vscode lint \\\n diff-test clean\n\nhelp:\n\t@echo \"Install:\"\n\t@echo \" make install pip install visidata\"\n\t@echo \" make install-dev editable install with dev deps\"\n\t@echo \" make install-test install with test deps\"\n\t@echo \" make install-all install with all optional deps\"\n\t@echo \"\"\n\t@echo \"Test:\"\n\t@echo \" make test run all tests (same as test-all)\"\n\t@echo \"\"\n\t@echo \"Build:\"\n\t@echo \" make man generate man pages (requires soelim, preconv, aha)\"\n\t@echo \" make zsh-completion generate zsh completion script\"\n\t@echo \" make docker build docker images\"\n\t@echo \"\"\n\t@echo \"Setup:\"\n\t@echo \" make setup-hooks configure git to use dev/hooks\"\n\t@echo \" make setup-vscode copy devcontainer configs to .vscode/\"\n\t@echo \"\"\n\t@echo \"Utility:\"\n\t@echo \" make lint run ruff linter\"\n\t@echo \" make diff-test show diffs from last test run\"\n\t@echo \" make clean remove generated files\"\n\ninstall:\n\tpip3 install .\n\ninstall-dev:\n\tpip3 install -r dev/requirements-dev.txt\n\tpip3 install -e .\n\ninstall-test:\n\tpip3 install .\n\tpip3 install \".[test]\"\n\ninstall-all:\n\tpip3 install \".[all]\"\n\ntest: test-all\n\ntest-all:\n\tdev/test-all.sh\n\ntest-vgit:\n\tvd -p visidata/apps/vgit/tests/*.vdx --batch\n\ntest-vdsql:\n\tcd visidata/apps/vdsql && ./test.sh\n\nbuild: man zsh-completion\n\nman:\n\tdev/mkman.sh\n\nzsh-completion:\n\tpython3 dev/zsh-completion.py _visidata\n\ndocker:\n\tdev/build-container\n\n# Setup\n\nsetup-hooks:\n\tgit config core.hooksPath dev/hooks\n\nsetup-vscode:\n\tmkdir -p .vscode\n\tcp .devcontainer/launch.json .vscode/launch.json\n\tcp .devcontainer/settings.json .vscode/settings.json\n\n# Utility\n\nlint:\n\truff check .\n\ndiff-test:\n\tdev/diff-test.sh\n\nclean:\n\trm -f visidata/man/vd.1 visidata/man/visidata.1 visidata/man/vd.txt\n\trm -f docs/man.md\n" }, { "path": "README.md", "content": "# VisiData v3.3\n\n[![Tests](https://github.com/saulpw/visidata/workflows/visidata-ci-build/badge.svg)](https://github.com/saulpw/visidata/actions/workflows/main.yml)\n[![Gitpod ready-to-code](https://img.shields.io/badge/Gitpod-ready--to--code-blue?logo=gitpod)](https://gitpod.io/#https://github.com/saulpw/visidata)\n\n[![discord](https://img.shields.io/discord/880915750007750737?label=discord)](https://visidata.org/chat)\n[![Mastodon](https://img.shields.io/mastodon/follow/110136431814047095?domain=https%3A%2F%2Ffosstodon.org)](https://fosstodon.org/@saulpw)\n\nA terminal interface for exploring and arranging tabular data.\n\n![Frequency table](http://visidata.org/videos/freq-move-row.gif)\n\nVisiData supports tsv, csv, sqlite, json, xlsx (Excel), hdf5, and [many other formats](https://visidata.org/formats).\n\n## Platform requirements\n\n- Linux, OS/X, or Windows (with WSL)\n- Python 3.8+\n- additional Python modules are required for certain formats and sources\n\n## Install\n\nTo install the latest release from PyPi:\n\n pip3 install visidata\n\nTo try VisiData without installing, use [pipx](https://pipx.pypa.io/) or [uv](https://docs.astral.sh/uv/):\n\n pipx run visidata # or: uvx visidata\n\nTo install permanently (adds `vd` to your PATH):\n\n pipx install visidata # or: uv tool install visidata\n\nAdditional Python packages are needed for some formats:\n\n pipx install visidata --preinstall openpyxl --preinstall lxml\n # or: uv tool install visidata --with openpyxl --with lxml\n\nTo add format packages to an existing pipx install:\n\n pipx inject visidata openpyxl lxml\n\nTo install the cutting edge `develop` branch (no warranty expressed or implied):\n\n pip3 install git+https://github.com/saulpw/visidata.git@develop\n\nSee [visidata.org/install](https://visidata.org/install) for detailed instructions for all available platforms and package managers.\n\n### Usage\n\n $ vd \n $ | vd\n\nPress `Ctrl+Q` to quit at any time.\n\nHundreds of other commands and options are also available; see the documentation.\n\n### Documentation\n\n* [VisiData documentation](https://visidata.org/docs)\n* [Plugin Author's Guide and API Reference](https://visidata.org/docs/api)\n* [Quick reference](https://visidata.org/man) (available within `vd` with `Ctrl+H`), which has a list of commands and options.\n* [Intro to VisiData Tutorial](https://jsvine.github.io/intro-to-visidata/) by [Jeremy Singer-Vine](https://www.jsvine.com/)\n\n### Help and Support\n\nIf you have a question, issue, or suggestion regarding VisiData, please [create an issue on Github](https://github.com/saulpw/visidata/issues) or chat with us at #visidata on [irc.libera.chat](https://libera.chat/).\n\nIf you use VisiData regularly, please [support me on Patreon](https://www.patreon.com/saulpw)!\n\n## License\n\nCode in the `stable` branch of this repository, including the main `vd` application, loaders, and plugins, is available for use and redistribution under GPLv3.\n\n## Credits\n\nVisiData is conceived and developed by Saul Pwanson ``.\n\nAnja Kefala `` maintains the documentation and packages for all platforms.\n\nMany thanks to numerous other [contributors](https://visidata.org/credits/), and to those wonderful users who provide feedback, for helping to make VisiData the awesome tool that it is.\n" }, { "path": "bin/filter-doc.py", "content": "#!/usr/bin/python3\n\nimport fileinput\nimport inspect\nimport visidata\n\ntmpl='''\n~~~\n {name}{sign} [ [{sourcepy}:{sourceline}](https://github.com/saulpw/visidata/visidata/blob/develop/visidata/{sourcepy}#{sourceline})]\n~~~\n{doc}\n'''\n\nfor line in fileinput.input():\n if line.startswith(':#'):\n objname, funcname = line[2:].strip().split('.')\n func = getattr(getattr(visidata, objname), funcname)\n src = visidata.Path(inspect.getsourcefile(func)).parts[-1]\n print(tmpl.format(\n name=objname+'.'+func.__name__,\n doc=inspect.getdoc(func) or 'XXX',\n sourcepy=src,\n sourceline=inspect.getsourcelines(func)[1],\n sign=inspect.signature(func)))\n else:\n print(line, end='')\n" }, { "path": "bin/vd", "content": "#!/usr/bin/env python3\n\nimport visidata.main\n\nif __name__ == '__main__':\n visidata.main.vd_cli()\n" }, { "path": "bin/vd2to3.vdx", "content": "#!/usr/bin/env -S vd -p\n# VisiData v3.0dev\nopen-file ~/.visidata/macros.tsv\ncol command\nrename-col binding\ncol filename\nrename-col source\nsave-sheet ~/.visidata/macros.jsonl\nquit-all\n" }, { "path": "bin/viewtsv.py", "content": "#!/usr/bin/env python3\n\n# as plugin: `from viewtsv import open_tsv` in .visidatarc\n# standalone: `viewtsv.py '\n\nfrom visidata import VisiData, Sheet, ColumnItem, asyncthread, options\n\n@VisiData.api\ndef open_tsv(vd, p):\n return MinimalTsvSheet(p.base_stem, source=p)\n\n\nclass MinimalTsvSheet(Sheet):\n rowtype = 'rows'\n\n @asyncthread\n def reload(self):\n self.rows = []\n\n delim = options.delimiter\n header = True\n with open(self.source, encoding=options.encoding) as fp:\n for line in fp:\n line = line[:-1] # strip trailing newline\n\n if header:\n if delim in line:\n header = False\n self.columns = []\n for i, colname in enumerate(line.split()):\n self.addColumn(ColumnItem(colname, i))\n continue\n\n self.addRow(line.split(delim))\n\n\n# a minimal main() for standalone apps\nif __name__ == '__main__':\n import sys\n from visidata import run, Path, vd\n run(*(vd.open_tsv(Path(fn)) for fn in sys.argv[1:]))\n" }, { "path": "dev/DOCS.md", "content": "# VisiData Documentation Style Guide\n\n`docs/man.md` is generated from the manpage source (`visidata/man/vd.inc`) by `dev/mkman.sh`. Edit `vd.inc`, not `man.md`.\n\nVisiData supports basic markdown like # Headings, **bold**, *italics*, `code snippets`, and _underscore_.\n\nVisiData has its own display attribute syntax. For e.g.:\n\n [:onclick ][/] formats into a clickable url that will open in $BROWSER.\n\n [:red on black][/] changes the colour of to be red text on black background. Any color option can be used after :, like [:warning], [:error], [:menu].\n\nVisiData replaces `{vd.options.disp_selected_note}` with `+`, the value that `vd.options.disp_selected_note` is set to.\nReference any option value with `{vd.options.optname}`.\nThis is a great way to ensure that the appropriate option is displayed, even if the user changed the option value.\n\n- Use `{help.commands.longname}` to put the properly formatted string (below) into the GuideSheet.\nIt’s much preferred to change the command helpstring itself, in order to make this pattern work, than to write it out manually.\nIt will look like this:\n\n - `` (``) to .\n\nThe keystroke immediately follows the bullet. Do not say “Press” or “Use” within VisiData docs and helpstrings.\n\n- List relevant options with the following pattern.\n\n - [:onclick options-sheet