[ { "path": ".bettercodehub.yml", "content": "# https://bettercodehub.com\n\n# Search for project directories in the root dir (level 1).\ncomponent_depth: 1\n\nlanguages:\n - name: python\n\n production:\n include:\n # The CLI app lacks extension.\n - /proxmove\n exclude:\n - /setup.py\n" }, { "path": "CHANGES.rst", "content": "Changes\n-------\n\n* **HEAD** - XXXX-XX-XX\n\n* **1.2** - 2022-09-21\n\n Changes:\n\n - Add ZFS snapshot migration for minimal downtime: full sync, poweroff,\n final sync, poweron.\n - Add --wait-before-stop to allow manual actions after initial sync.\n - Add discard=on on ZFS volumes by default.\n - Prefer rsync over scp.\n - Add ssh_cipher setting (for AES speedup) for zfs-to-zfs transfers.\n - Add workaround to fix boot order on 6.1+6.3 mixed clusters\n - Show --average-rate in pv(1) viewer.\n\n Tweaks/fixes:\n\n - Create temp dir automatically, instead of complaining.\n - Set onboot=0 on old-VM after migration so it does not auto-start.\n - Improve various pre-copy checks.\n - Fix confusing double exception, fix various other minor annoyances.\n - Better ssh handling (accept fingerprints, forwarding only when necessary).\n - Better handling of unsupported LXC.\n - Also copy the source pool membership to the destination.\n\n* **1.1** - 2021-06-11\n\n Features/fixes:\n\n - Add basic resume support.\n - Allow ``:port`` in ``ssh=user@host:port`` config.\n - Unmount cdrom media before moving.\n - Fix destination volume naming (use ``vm-NNN-disk-N`` instead of\n ``vm-NNN-virtioN``).\n - Some documentation improvements.\n\n* **1.0** - 2020-01-17\n\n Features/fixes:\n\n - Fix disk I/O resource hog overuse on newer PVE clusters.\n - Fix API connection to newer PVE clusters.\n - Add faster ssh cipher by default.\n - Work around Proxmox API timeout.\n - Improved usability through better logging and prepare checks.\n\n* **0.1.0** - 2018-11-22\n\n Bugs fixed:\n\n - Show that not seeing a VM is probably a PVEAdmin-permissions issue.\n - Don't die if image_size cannot be determined.\n - Place the sample files and docs in a /usr/share/...proxmove subdir.\n\n* **0.0.9** - 2017-03-28\n\n New features:\n\n - Added --no-verify-ssl option.\n\n Bugs fixed:\n\n - Fix str-cast bug with ZFS destination creation.\n - Fix ignoring of non-volume properties like \"scsihw\".\n\n* **0.0.8** - 2017-01-26\n\n New features:\n\n - Partial LXC container move implemented. Not complete.\n\n Bugs fixed:\n\n - Allow ZFS paths to be more than just the a pool name. Instead of\n e.g. ``path=zfs:mc9-8-ssd`` we now also allow\n ``path=zfs:rpool/data/images``. Closes #7.\n\n* **v0.0.7** - 2016-10-07\n\n Bugs fixed:\n\n - Instead of trusting on the \"size=XXG\" which may or may not be\n present in the storage volume config, it reads the QCOW header or\n ZFS volume size directly. Also checks that the values are available\n before attempting a move.\n\n* **v0.0.6** - 2016-09-21\n\n New features:\n\n - Add --bwlimit in Mbit/s to limit bandwidth during transfer. Will use\n the scp(1) -l option or for ZFS use the mbuffer(1) auxiliary. As an\n added bonus mbuffer may improve ZFS send/recv speed through\n buffering. Closes #4.\n - Add --skip-disks option to skip copying of the disks. Use this if\n you want to copy the disks manually. Closes #3.\n - Add --skip-start option to skip autostarting of the VM.\n - Adds optional pv(1) pipe viewer progress bar to get ETA in ZFS\n transfers.\n - Add hidden --debug option for more verbosity.\n - Add hidden --ignore-exists option that allows you to test moves\n between the same cluster by creating an alias (second config).\n\n Bugs fixed:\n\n - Format is not always specified in the properties. If it isn't, use\n the image filename suffix when available.\n - Sometimes old values aren't available in the \"pending\" list. Don't croak.\n Closes #2.\n - Begun refactoring. Testing bettercodehub.com.\n - Also check whether temporary (renamed) VMs exist before starting.\n\n* **v0.0.5** - 2016-08-30\n\n - Added support for ZFS to ZFS disk copy. QCOW2 to ZFS and ZFS to ZFS\n is now tested.\n\n* **v0.0.4** - 2016-08-30\n\n - Major overhaul of configuration format and other changes.\n" }, { "path": "README.rst", "content": "|proxmove|\n==========\n\n*The Proxmox VM migrator: migrates VMs between different Proxmox VE clusters.*\n\nMigrating a virtual machine (VM) on a PVE-cluster from one node to\nanother is implemented in the Proxmox Virtual Environment (PVE). But\nmigrating a VM from one PVE-cluster to another is not.\n\nproxmove helps you move VMs between PVE-clusters with minimal hassle.\n*And if you use ZFS, with minimal downtime too.*\n\n\nExample invocation:\n\n.. code-block:: console\n\n $ proxmove SOURCE_CLUSTER DEST_CLUSTER DEST_NODE DEST_STORAGE VM_NAME1...\n\nBut, to get it to work, you'll need to configure ``~/.proxmoverc``\nfirst. See `Configuration`_.\n\n\nAdditional tips:\n\n- If source and destination filesystems use ZFS, the move is done in two\n stages, by copying an initial snapshot while the source VM is still\n up. Combine with ``--wait-before-stop`` for additional control.\n- Use ``--debug``; it doesn't flood your screen, but provides useful clues\n about what it's doing.\n- If your network bridge is different on the ``DEST_CLUSTER``, use\n ``--skip-start``; that way *proxmove* \"completes\" successfully when\n done with the move. (You'll still need to change the bridge before\n starting the VM obviously.)\n- If *proxmove* detects that a move was in progress, it will\n interactively attempt a resume. For ZFS to ZFS syncs, it will do\n *another* initial resync before shutting down the source VM.\n\n\nFull invocation specification (``--help``):\n\n.. code-block::\n\n usage: proxmove [-c FILENAME] [-n] [--bwlimit MBPS] [--no-verify-ssl]\n [--skip-disks] [--skip-start] [--wait-before-stop]\n [--ssh-ciphers CIPHERS] [--debug] [--ignore-exists]\n [-h] [--version]\n source destination nodeid storage vm [vm ...]\n\n Migrate VMs from one Proxmox cluster to another.\n\n positional arguments:\n source alias of source cluster\n destination alias of destination cluster\n nodeid node on destination cluster\n storage storage on destination node\n vm one or more VMs (guests) to move\n\n optional arguments:\n -c FILENAME, --config FILENAME\n use alternate configuration inifile\n -n, --dry-run stop before doing any writes\n --bwlimit MBPS limit bandwidth in Mbit/s\n --no-verify-ssl skip ssl verification on the api hosts\n --skip-disks do the move, but skip copying of the disks;\n implies --skip-start\n --skip-start do the move, but do not start the new instance\n --wait-before-stop prepare the move, but ask for user\n confirmation before shutting down the old\n instance (useful if you have to move\n networks/IPs)\n --ssh-ciphers CIPHERS\n comma separated list of ssh -c ciphers to\n prefer, (aes128-gcm@openssh.com is supposed to\n be fast if you have aes on your cpu); set to\n \"-\" to use ssh defaults\n\n debug arguments:\n --debug enables extra debug logging\n --ignore-exists continue when target VM already exists; allows\n moving to same cluster\n\n other actions:\n -h, --help show this help message and exit\n --version show program's version number and exit\n\n Cluster aliases and storage locations should be defined in\n ~/.proxmoverc (or see -c option). See the example proxmoverc.sample.\n It requires [pve:CLUSTER_ALIAS] sections for the proxmox \"api\" URL and\n [storage:CLUSTER_ALIAS:STORAGE_NAME] sections with \"ssh\", \"path\" and\n \"temp\" settings.\n\n\nExample run\n-----------\n\nFirst you need to configure ``~/.proxmoverc``; see below.\n\nWhen configured, you can do something like this:\n\n.. code-block:: console\n\n $ proxmove apple-cluster banana-cluster node2 node2-ssd the-vm-to-move\n 12:12:27: Attempt moving apple-cluster => banana-cluster<6669ad2c> (node 'node2'): the-vm-to-move\n 12:12:27: - source VM the-vm-to-move@node1\n 12:12:27: - storage 'ide2': None,media=cdrom (host=, guest=)\n 12:12:27: - storage 'virtio0': sharedsan:565/vm-565-disk-1.qcow2,format=qcow2,iops_rd=4000,iops_wr=500,size=50G (host=37.7GiB, guest=50.0GiB)\n 12:12:27: Creating new VM 'the-vm-to-move' on 'banana-cluster', node 'node2'\n 12:12:27: - created new VM 'the-vm-to-move--CREATING' as UPID:node2:00005977:1F4D78F4:57C55C0B:qmcreate:126:user@pve:; waiting for it to show up\n 12:12:34: - created new VM 'the-vm-to-move--CREATING': the-vm-to-move--CREATING@node2\n 12:12:34: Stopping VM the-vm-to-move@node1\n 12:12:42: - stopped VM the-vm-to-move@node1\n 12:12:42: Ejected (cdrom?) volume 'ide2' (none) added to the-vm-to-move--CREATING@node2\n 12:12:42: Begin copy of 'virtio0' (sharedsan:565/vm-565-disk-1.qcow2,format=qcow2,iops_rd=4000,iops_wr=500,size=50G) to local-ssd\n 12:12:42: scp(1) copy from '/pool0/san/images/565/vm-565-disk-1.qcow2' (on sharedsan) to 'root@node2.banana-cluster.com:/node2-ssd/temp/temp-proxmove/vm-126-virtio0'\n Warning: Permanently added 'node2.banana-cluster.com' (ECDSA) to the list of known hosts.\n vm-565-disk-1.qcow2 100% 50GB 90.5MB/s 09:26\n Connection to san.apple-cluster.com closed.\n 12:22:08: Temp data '/node2-ssd/temp/temp-proxmove/vm-126-virtio0' on local-ssd\n 12:22:08: Writing data from temp '/node2-ssd/temp/temp-proxmove/vm-126-virtio0' to '/dev/zvol/node2-ssd/vm-126-virtio0' (on local-ssd)\n (100.00/100%)\n Connection to node2.banana-cluster.com closed.\n 12:24:25: Removing temp '/node2-ssd/temp/temp-proxmove/vm-126-virtio0' (on local-ssd)\n 12:24:26: Starting VM the-vm-to-move@node2\n 12:24:27: - started VM the-vm-to-move@node2\n 12:24:27: Completed moving apple-cluster => banana-cluster<6669ad2c> (node 'node2'): the-vm-to-move\n\nBefore, ``the-vm-to-move`` was running on ``apple-cluster`` on ``node1``.\n\nAfterwards, ``the-vm-to-move`` is running on ``banana-cluster`` on ``node2``.\nThe ``the-vm-to-move`` on the ``apple-cluster`` has been stopped and renamed to\n``the-vm-to-move--MIGRATED``.\n\n\nConfiguration\n-------------\n\nSet up the ``~/.proxmoverc`` config file. First you need to define which\nclusters you have. For example *apple-cluster* and *banana-cluster*.\n\n.. code-block:: ini\n\n ; Example cluster named \"apple-cluster\" with 3 storage devices, one\n ; shared, and two which exist on a single node only.\n ;\n ; The user requires various permissions found in the PVEVMAdmin role (VM\n ; allocate + audit) and PVEAuditor role (Datastore audit) and PVEPoolAdmin\n ; (to inspect and create pools). And PVESDNAdmin role for the network conf.\n ;\n [pve:apple-cluster]\n api=https://user@pve:PASSWORD@apple-cluster.com:443\n\n ; Example cluster named \"banana-cluster\" with 2 storage devices; both\n ; storage devices exist on the respective nodes only.\n [pve:banana-cluster]\n api=https://user@pve:PASSWORD@banana-cluster.com:443\n\nNext, it needs configuration for the storage devices. They are expected\nto be reachable over SSH; both from the caller and from each other\n(using SSH-agent forwarding).\n\nThe following defines two storage devices for the *apple-cluster*, one shared\nand one local to *node1* only.\n\nIf on *sharedsan*, the images are probably called something like\n``/pool0/san/images/VMID/vm-VMID-disk1.qcow2``, while in Proxmox, they are\nreferred to as ``sharedsan:VMID/vm-VMID-disk1.qcow2``.\n\n.. code-block:: ini\n\n [storage:apple-cluster:sharedsan] ; \"sharedsan\" is available on all nodes\n ssh=root@san.apple-cluster.com\n path=/pool0/san/images\n temp=/pool0/san/private\n\n [storage:apple-cluster:local@node1] ; local disk on node1 only\n ssh=root@node1.apple-cluster.com\n path=/srv/images\n temp=/srv/temp\n\nIf you use ZFS storage on *banana-cluster*, the storage config could look\nlike this. Disk volumes exist on the ZFS filesystem ``node1-ssd/images``\nand ``node2-ssd/images`` on the nodes *node1* and *node2* respectively.\n\nNote that the ``temp=`` path is always a regular path.\n\n.. code-block:: ini\n\n [storage:banana-cluster:node1-ssd@node1]\n ssh=root@node1.banana-cluster.com\n path=zfs:node1-ssd/images\n temp=/node1-ssd/temp\n\n [storage:banana-cluster:node2-ssd@node2]\n ssh=root@node2.banana-cluster.com\n path=zfs:node2-ssd/images\n temp=/node2-ssd/temp\n\nThe config file looks better with indentation. The author suggests this layout:\n\n.. code-block:: ini\n\n [pve:apple-cluster]\n ...\n\n [storage:apple-cluster:sharedsan]\n ...\n [storage:apple-cluster:local@node1]\n ...\n\n [pve:banana-cluster]\n ...\n\n [storage:banana-cluster:node1-ssd@node1]\n ...\n\n\nDebugging\n---------\n\nIf you run into a ``ResourceException``, you may want to patch proxmoxer 1.0.3\nto show the HTTP error reason as well.\n\n.. code-block:: udiff\n\n --- proxmoxer/core.py\t2019-04-04 09:13:16.832961589 +0200\n +++ proxmoxer/core.py\t2019-04-04 09:15:45.434175030 +0200\n @@ -75,8 +75,10 @@ class ProxmoxResource(ProxmoxResourceBas\n logger.debug('Status code: %s, output: %s', resp.status_code, resp.content)\n\n if resp.status_code >= 400:\n - raise ResourceException(\"{0} {1}: {2}\".format(resp.status_code, httplib.responses[resp.status_code],\n - resp.content))\n + raise ResourceException('{0} {1} (\"{2}\"): {3}'.format(\n + resp.status_code, httplib.responses[resp.status_code],\n + resp.reason, # reason = textual status_code\n + resp.content))\n elif 200 <= resp.status_code <= 299:\n return self._store[\"serializer\"].loads(resp)\n\nIt might reveal a bug (or new feature), like::\n\n proxmoxer.core.ResourceException:\n 500 Internal Server Error (\"only root can set 'vmgenid' config\"):\n b'{\"data\":null}'\n\n\nLicense\n-------\n\nproxmove is free software: you can redistribute it and/or modify it under\nthe terms of the GNU General Public License as published by the Free\nSoftware Foundation, version 3 or any later version.\n\n\n.. |proxmove| image:: assets/proxmove_head.png\n :alt: proxmove\n" }, { "path": "TODO.rst", "content": "TODO\n----\n\n* On missing disk (bad config), the zfs send stalls and mbuffer waits for\n infinity.\n\n* Rename \"VM\" to \"Instance\" so \"LXC\" containers don't feel misrepresented.\n\n* Communicate with the storage API to check/configure storage (more easily).\n\n* Create a ``--config`` command to set up a basic configuration. Combine with\n storage API reading.\n\n* Fix so LXC-copying works (this is a bit tricky because of the necessity for\n a temporary image/tarball to install).\n\n* Create a proxmoxer_api example that returns test data, so we can run tests.\n\n* Let it work with pve-zsync -- a daemon that syncs data between nodes:\n https://pve.proxmox.com/wiki/PVE-zsync\n" }, { "path": "artwork/LICENSE.CC.BY-NC-SA.4-0.source", "content": "https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode.txt\n" }, { "path": "artwork/LICENSE.CC.BY-NC-SA.4-0.txt", "content": "Attribution-NonCommercial-ShareAlike 4.0 International\n\n=======================================================================\n\nCreative Commons Corporation (\"Creative Commons\") is not a law firm and\ndoes not provide legal services or legal advice. Distribution of\nCreative Commons public licenses does not create a lawyer-client or\nother relationship. Creative Commons makes its licenses and related\ninformation available on an \"as-is\" basis. Creative Commons gives no\nwarranties regarding its licenses, any material licensed under their\nterms and conditions, or any related information. Creative Commons\ndisclaims all liability for damages resulting from their use to the\nfullest extent possible.\n\nUsing Creative Commons Public Licenses\n\nCreative Commons public licenses provide a standard set of terms and\nconditions that creators and other rights holders may use to share\noriginal works of authorship and other material subject to copyright\nand certain other rights specified in the public license below. The\nfollowing considerations are for informational purposes only, are not\nexhaustive, and do not form part of our licenses.\n\n Considerations for licensors: Our public licenses are\n intended for use by those authorized to give the public\n permission to use material in ways otherwise restricted by\n copyright and certain other rights. Our licenses are\n irrevocable. Licensors should read and understand the terms\n and conditions of the license they choose before applying it.\n Licensors should also secure all rights necessary before\n applying our licenses so that the public can reuse the\n material as expected. Licensors should clearly mark any\n material not subject to the license. This includes other CC-\n licensed material, or material used under an exception or\n limitation to copyright. More considerations for licensors:\n\twiki.creativecommons.org/Considerations_for_licensors\n\n Considerations for the public: By using one of our public\n licenses, a licensor grants the public permission to use the\n licensed material under specified terms and conditions. If\n the licensor's permission is not necessary for any reason--for\n example, because of any applicable exception or limitation to\n copyright--then that use is not regulated by the license. Our\n licenses grant only permissions under copyright and certain\n other rights that a licensor has authority to grant. Use of\n the licensed material may still be restricted for other\n reasons, including because others have copyright or other\n rights in the material. A licensor may make special requests,\n such as asking that all changes be marked or described.\n Although not required by our licenses, you are encouraged to\n respect those requests where reasonable. More_considerations\n for the public:\n\twiki.creativecommons.org/Considerations_for_licensees\n\n=======================================================================\n\nCreative Commons Attribution-NonCommercial-ShareAlike 4.0 International\nPublic License\n\nBy exercising the Licensed Rights (defined below), You accept and agree\nto be bound by the terms and conditions of this Creative Commons\nAttribution-NonCommercial-ShareAlike 4.0 International Public License\n(\"Public License\"). To the extent this Public License may be\ninterpreted as a contract, You are granted the Licensed Rights in\nconsideration of Your acceptance of these terms and conditions, and the\nLicensor grants You such rights in consideration of benefits the\nLicensor receives from making the Licensed Material available under\nthese terms and conditions.\n\n\nSection 1 -- Definitions.\n\n a. Adapted Material means material subject to Copyright and Similar\n Rights that is derived from or based upon the Licensed Material\n and in which the Licensed Material is translated, altered,\n arranged, transformed, or otherwise modified in a manner requiring\n permission under the Copyright and Similar Rights held by the\n Licensor. For purposes of this Public License, where the Licensed\n Material is a musical work, performance, or sound recording,\n Adapted Material is always produced where the Licensed Material is\n synched in timed relation with a moving image.\n\n b. Adapter's License means the license You apply to Your Copyright\n and Similar Rights in Your contributions to Adapted Material in\n accordance with the terms and conditions of this Public License.\n\n c. BY-NC-SA Compatible License means a license listed at\n creativecommons.org/compatiblelicenses, approved by Creative\n Commons as essentially the equivalent of this Public License.\n\n d. Copyright and Similar Rights means copyright and/or similar rights\n closely related to copyright including, without limitation,\n performance, broadcast, sound recording, and Sui Generis Database\n Rights, without regard to how the rights are labeled or\n categorized. For purposes of this Public License, the rights\n specified in Section 2(b)(1)-(2) are not Copyright and Similar\n Rights.\n\n e. Effective Technological Measures means those measures that, in the\n absence of proper authority, may not be circumvented under laws\n fulfilling obligations under Article 11 of the WIPO Copyright\n Treaty adopted on December 20, 1996, and/or similar international\n agreements.\n\n f. Exceptions and Limitations means fair use, fair dealing, and/or\n any other exception or limitation to Copyright and Similar Rights\n that applies to Your use of the Licensed Material.\n\n g. License Elements means the license attributes listed in the name\n of a Creative Commons Public License. The License Elements of this\n Public License are Attribution, NonCommercial, and ShareAlike.\n\n h. Licensed Material means the artistic or literary work, database,\n or other material to which the Licensor applied this Public\n License.\n\n i. Licensed Rights means the rights granted to You subject to the\n terms and conditions of this Public License, which are limited to\n all Copyright and Similar Rights that apply to Your use of the\n Licensed Material and that the Licensor has authority to license.\n\n j. Licensor means the individual(s) or entity(ies) granting rights\n under this Public License.\n\n k. NonCommercial means not primarily intended for or directed towards\n commercial advantage or monetary compensation. For purposes of\n this Public License, the exchange of the Licensed Material for\n other material subject to Copyright and Similar Rights by digital\n file-sharing or similar means is NonCommercial provided there is\n no payment of monetary compensation in connection with the\n exchange.\n\n l. Share means to provide material to the public by any means or\n process that requires permission under the Licensed Rights, such\n as reproduction, public display, public performance, distribution,\n dissemination, communication, or importation, and to make material\n available to the public including in ways that members of the\n public may access the material from a place and at a time\n individually chosen by them.\n\n m. Sui Generis Database Rights means rights other than copyright\n resulting from Directive 96/9/EC of the European Parliament and of\n the Council of 11 March 1996 on the legal protection of databases,\n as amended and/or succeeded, as well as other essentially\n equivalent rights anywhere in the world.\n\n n. You means the individual or entity exercising the Licensed Rights\n under this Public License. Your has a corresponding meaning.\n\n\nSection 2 -- Scope.\n\n a. License grant.\n\n 1. Subject to the terms and conditions of this Public License,\n the Licensor hereby grants You a worldwide, royalty-free,\n non-sublicensable, non-exclusive, irrevocable license to\n exercise the Licensed Rights in the Licensed Material to:\n\n a. reproduce and Share the Licensed Material, in whole or\n in part, for NonCommercial purposes only; and\n\n b. produce, reproduce, and Share Adapted Material for\n NonCommercial purposes only.\n\n 2. Exceptions and Limitations. For the avoidance of doubt, where\n Exceptions and Limitations apply to Your use, this Public\n License does not apply, and You do not need to comply with\n its terms and conditions.\n\n 3. Term. The term of this Public License is specified in Section\n 6(a).\n\n 4. Media and formats; technical modifications allowed. The\n Licensor authorizes You to exercise the Licensed Rights in\n all media and formats whether now known or hereafter created,\n and to make technical modifications necessary to do so. The\n Licensor waives and/or agrees not to assert any right or\n authority to forbid You from making technical modifications\n necessary to exercise the Licensed Rights, including\n technical modifications necessary to circumvent Effective\n Technological Measures. For purposes of this Public License,\n simply making modifications authorized by this Section 2(a)\n (4) never produces Adapted Material.\n\n 5. Downstream recipients.\n\n a. Offer from the Licensor -- Licensed Material. Every\n recipient of the Licensed Material automatically\n receives an offer from the Licensor to exercise the\n Licensed Rights under the terms and conditions of this\n Public License.\n\n b. Additional offer from the Licensor -- Adapted Material.\n Every recipient of Adapted Material from You\n automatically receives an offer from the Licensor to\n exercise the Licensed Rights in the Adapted Material\n under the conditions of the Adapter's License You apply.\n\n c. No downstream restrictions. You may not offer or impose\n any additional or different terms or conditions on, or\n apply any Effective Technological Measures to, the\n Licensed Material if doing so restricts exercise of the\n Licensed Rights by any recipient of the Licensed\n Material.\n\n 6. No endorsement. Nothing in this Public License constitutes or\n may be construed as permission to assert or imply that You\n are, or that Your use of the Licensed Material is, connected\n with, or sponsored, endorsed, or granted official status by,\n the Licensor or others designated to receive attribution as\n provided in Section 3(a)(1)(A)(i).\n\n b. Other rights.\n\n 1. Moral rights, such as the right of integrity, are not\n licensed under this Public License, nor are publicity,\n privacy, and/or other similar personality rights; however, to\n the extent possible, the Licensor waives and/or agrees not to\n assert any such rights held by the Licensor to the limited\n extent necessary to allow You to exercise the Licensed\n Rights, but not otherwise.\n\n 2. Patent and trademark rights are not licensed under this\n Public License.\n\n 3. To the extent possible, the Licensor waives any right to\n collect royalties from You for the exercise of the Licensed\n Rights, whether directly or through a collecting society\n under any voluntary or waivable statutory or compulsory\n licensing scheme. In all other cases the Licensor expressly\n reserves any right to collect such royalties, including when\n the Licensed Material is used other than for NonCommercial\n purposes.\n\n\nSection 3 -- License Conditions.\n\nYour exercise of the Licensed Rights is expressly made subject to the\nfollowing conditions.\n\n a. Attribution.\n\n 1. If You Share the Licensed Material (including in modified\n form), You must:\n\n a. retain the following if it is supplied by the Licensor\n with the Licensed Material:\n\n i. identification of the creator(s) of the Licensed\n Material and any others designated to receive\n attribution, in any reasonable manner requested by\n the Licensor (including by pseudonym if\n designated);\n\n ii. a copyright notice;\n\n iii. a notice that refers to this Public License;\n\n iv. a notice that refers to the disclaimer of\n warranties;\n\n v. a URI or hyperlink to the Licensed Material to the\n extent reasonably practicable;\n\n b. indicate if You modified the Licensed Material and\n retain an indication of any previous modifications; and\n\n c. indicate the Licensed Material is licensed under this\n Public License, and include the text of, or the URI or\n hyperlink to, this Public License.\n\n 2. You may satisfy the conditions in Section 3(a)(1) in any\n reasonable manner based on the medium, means, and context in\n which You Share the Licensed Material. For example, it may be\n reasonable to satisfy the conditions by providing a URI or\n hyperlink to a resource that includes the required\n information.\n 3. If requested by the Licensor, You must remove any of the\n information required by Section 3(a)(1)(A) to the extent\n reasonably practicable.\n\n b. ShareAlike.\n\n In addition to the conditions in Section 3(a), if You Share\n Adapted Material You produce, the following conditions also apply.\n\n 1. The Adapter's License You apply must be a Creative Commons\n license with the same License Elements, this version or\n later, or a BY-NC-SA Compatible License.\n\n 2. You must include the text of, or the URI or hyperlink to, the\n Adapter's License You apply. You may satisfy this condition\n in any reasonable manner based on the medium, means, and\n context in which You Share Adapted Material.\n\n 3. You may not offer or impose any additional or different terms\n or conditions on, or apply any Effective Technological\n Measures to, Adapted Material that restrict exercise of the\n rights granted under the Adapter's License You apply.\n\n\nSection 4 -- Sui Generis Database Rights.\n\nWhere the Licensed Rights include Sui Generis Database Rights that\napply to Your use of the Licensed Material:\n\n a. for the avoidance of doubt, Section 2(a)(1) grants You the right\n to extract, reuse, reproduce, and Share all or a substantial\n portion of the contents of the database for NonCommercial purposes\n only;\n\n b. if You include all or a substantial portion of the database\n contents in a database in which You have Sui Generis Database\n Rights, then the database in which You have Sui Generis Database\n Rights (but not its individual contents) is Adapted Material,\n including for purposes of Section 3(b); and\n\n c. You must comply with the conditions in Section 3(a) if You Share\n all or a substantial portion of the contents of the database.\n\nFor the avoidance of doubt, this Section 4 supplements and does not\nreplace Your obligations under this Public License where the Licensed\nRights include other Copyright and Similar Rights.\n\n\nSection 5 -- Disclaimer of Warranties and Limitation of Liability.\n\n a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE\n EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS\n AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF\n ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,\n IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,\n WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR\n PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,\n ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT\n KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT\n ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.\n\n b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE\n TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,\n NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,\n INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,\n COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR\n USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN\n ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR\n DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR\n IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.\n\n c. The disclaimer of warranties and limitation of liability provided\n above shall be interpreted in a manner that, to the extent\n possible, most closely approximates an absolute disclaimer and\n waiver of all liability.\n\n\nSection 6 -- Term and Termination.\n\n a. This Public License applies for the term of the Copyright and\n Similar Rights licensed here. However, if You fail to comply with\n this Public License, then Your rights under this Public License\n terminate automatically.\n\n b. Where Your right to use the Licensed Material has terminated under\n Section 6(a), it reinstates:\n\n 1. automatically as of the date the violation is cured, provided\n it is cured within 30 days of Your discovery of the\n violation; or\n\n 2. upon express reinstatement by the Licensor.\n\n For the avoidance of doubt, this Section 6(b) does not affect any\n right the Licensor may have to seek remedies for Your violations\n of this Public License.\n\n c. For the avoidance of doubt, the Licensor may also offer the\n Licensed Material under separate terms or conditions or stop\n distributing the Licensed Material at any time; however, doing so\n will not terminate this Public License.\n\n d. Sections 1, 5, 6, 7, and 8 survive termination of this Public\n License.\n\n\nSection 7 -- Other Terms and Conditions.\n\n a. The Licensor shall not be bound by any additional or different\n terms or conditions communicated by You unless expressly agreed.\n\n b. Any arrangements, understandings, or agreements regarding the\n Licensed Material not stated herein are separate from and\n independent of the terms and conditions of this Public License.\n\n\nSection 8 -- Interpretation.\n\n a. For the avoidance of doubt, this Public License does not, and\n shall not be interpreted to, reduce, limit, restrict, or impose\n conditions on any use of the Licensed Material that could lawfully\n be made without permission under this Public License.\n\n b. To the extent possible, if any provision of this Public License is\n deemed unenforceable, it shall be automatically reformed to the\n minimum extent necessary to make it enforceable. If the provision\n cannot be reformed, it shall be severed from this Public License\n without affecting the enforceability of the remaining terms and\n conditions.\n\n c. No term or condition of this Public License will be waived and no\n failure to comply consented to unless expressly agreed to by the\n Licensor.\n\n d. Nothing in this Public License constitutes or may be interpreted\n as a limitation upon, or waiver of, any privileges and immunities\n that apply to the Licensor or You, including from the legal\n processes of any jurisdiction or authority.\n\n=======================================================================\n\nCreative Commons is not a party to its public\nlicenses. Notwithstanding, Creative Commons may elect to apply one of\nits public licenses to material it publishes and in those instances\nwill be considered the “Licensor.” The text of the Creative Commons\npublic licenses is dedicated to the public domain under the CC0 Public\nDomain Dedication. Except for the limited purpose of indicating that\nmaterial is shared under a Creative Commons public license or as\notherwise permitted by the Creative Commons policies published at\ncreativecommons.org/policies, Creative Commons does not authorize the\nuse of the trademark \"Creative Commons\" or any other trademark or logo\nof Creative Commons without its prior written consent including,\nwithout limitation, in connection with any unauthorized modifications\nto any of its public licenses or any other arrangements,\nunderstandings, or agreements concerning use of licensed material. For\nthe avoidance of doubt, this paragraph does not form part of the\npublic licenses.\n\nCreative Commons may be contacted at creativecommons.org.\n\n" }, { "path": "artwork/README.rst", "content": "proxmove artwork\n================\n\nThe artwork -- the proxmove logo, created by Ura Design (@uracreative) --\nis licensed according to the\n*Attribution-NonCommercial-ShareAlike 4.0 International* Creative Commons\nlicense (CC BY-NC-SA 4.0);\nsee `LICENSE.CC.BY-NC-SA.4-0.txt\n`_.\n" }, { "path": "proxmove", "content": "#!/usr/bin/env python3\n# vim: set ts=8 sw=4 sts=4 et ai:\nfrom __future__ import print_function\n\"\"\"\nproxmove: Proxmox Node Migration -- migrate nodes from one proxmox\ncluster to another\n\nThis is proxmove. proxmove is free software: you can redistribute it\nand/or modify it under the terms of the GNU General Public License as\npublished by the Free Software Foundation, version 3 or any later\nversion.\n\"\"\"\nimport argparse\nimport configparser\nimport logging\nimport logging.config\nimport os\nimport random\nimport re\nimport string\nimport subprocess\nimport sys\nimport time\nfrom base64 import b64encode\nfrom collections import OrderedDict, defaultdict\nfrom datetime import datetime\nfrom proxmoxer import ProxmoxAPI, ResourceException\nfrom http.client import InvalidURL\nfrom urllib.parse import urlparse\n\n__author__ = 'Walter Doekes'\n__copyright__ = 'Copyright (C) Walter Doekes, OSSO B.V. 2016-2022'\n__licence__ = 'GPLv3+'\n__version__ = '1.2+dev'\n\n# TODO: see NotImplementedErrors\n# TODO: split up into modules (storage, etc.)\n\nlog = logging.getLogger('proxmove')\n\n# Drop LC_* from environment.\nenviron = os.environ\nfor _key in list(environ.keys()):\n if _key.startswith('LC_'):\n del environ[_key]\n\nSUFFIX_CLONING = '--CLONING' # source, when cloning\nSUFFIX_CREATING = '--CREATING' # dest, while cloning\nSUFFIX_MIGRATED = '--MIGRATED' # source, when done\nPROXMOX_VOLUME_TYPES = (\n 'efidisk', 'ide', 'sata', 'scsi', 'virtio')\nPROXMOX_VOLUME_TYPES_RE = re.compile( # \"scsi0\", but not \"scsihw\"\n r'^({})\\d+$'.format('|'.join(PROXMOX_VOLUME_TYPES)))\nPROXMOVE_VOLUME_NAME_FMT = 'vm-{vmid}-disk-{diskidx}' # dst volume name\n\n\ndef argv2str(argv):\n def esc(word):\n if all(i in (\n 'ABCDEFGHIJKLMNOPQRSTUVWXYz'\n 'abcdefghijklmnopqrstuvwxyz'\n '0123456789_-:.,/@') for i in word):\n return word\n return \"'{}'\".format(\n word.replace('\\\\', '\\\\\\\\').replace(\"'\", \"\\\\'\"))\n\n return ' '.join(esc(i) for i in argv)\n\n\ndef human_size_fmt(num, suffix='B'):\n for unit in ('', 'Ki', 'Mi', 'Gi', 'Ti'):\n if abs(num) < 1024.0:\n return '{:3.1f}{}{}'.format(num, unit, suffix)\n num /= 1024.0\n return '{:.1f}{}{}'.format(num, 'Ti', suffix)\n\n\nclass _HumanSizeScan(object):\n multipliers = {\n 'T': 1024 * 1024 * 1024 * 1024,\n 'G': 1024 * 1024 * 1024,\n 'M': 1024 * 1024,\n 'K': 1024,\n 'B': 1,\n }\n\n def __call__(self, num_str):\n numeric, suffix = self.split(num_str)\n if '.' in numeric:\n numeric = float(numeric)\n else:\n numeric = int(numeric)\n multiplier = self.multipliers.get(suffix[0:1].upper())\n if not multiplier:\n if suffix:\n raise ValueError('unknown suffix found in {!r}'.format(\n num_str))\n multiplier = 1\n return int(numeric * multiplier)\n\n def split(self, num_str):\n num_str = num_str.lstrip()\n num_parts = []\n suffix = ''\n for i, ch in enumerate(num_str):\n if ch in '0123456789.':\n num_parts.append(ch)\n elif ch == ',':\n pass\n else:\n suffix = num_str[i:].strip()\n break\n return ''.join(num_parts), suffix\nhuman_size_scan = _HumanSizeScan() # noqa\n\n\ndef name_from_conf(type_, config):\n \"\"\"\n For 'lxc', the unique name is in 'hostname'. For others it is in 'name'.\n \"\"\"\n if type_ == 'lxc':\n assert 'name' not in config, config\n assert 'hostname' in config, config\n return config['hostname']\n else:\n assert type_ == 'qemu', type_\n\n assert 'hostname' not in config, config\n assert 'name' in config, config\n return config['name']\n\n\nclass ProxmoveError(Exception):\n pass\n\n\nclass PrepareError(ProxmoveError):\n \"\"\"\n Failure during preparation and requirements checking. This is only\n raised if we haven't mutated any data yet.\n \"\"\"\n pass\n\n\nclass ResumablePrepareError(PrepareError):\n \"\"\"\n Failure during preparation. This is resumable if the user wishes so.\n \"\"\"\n pass\n\n\nclass ArgumentParser14191(argparse.ArgumentParser):\n \"\"\"ArgumentParser from argparse that handles out-of-order positional\n arguments.\n\n This is a workaround created by Glenn Linderman in July 2012. You\n can now do this:\n\n parser = ArgumentParser14191()\n parser.add_argument('-f', '--foo')\n parser.add_argument('cmd')\n parser.add_argument('rest', nargs='*')\n # some of these would fail with the regular parser:\n for args, res in (('-f1 cmd 1 2 3', 'ok'),\n ('cmd -f1 1 2 3', 'would_fail'),\n ('cmd 1 -f1 2 3', 'would_fail'),\n ('cmd 1 2 3 -f1', 'ok')):\n try: out = parser.parse_args(args.split())\n except: print 'args', 'failed', res\n # out: Namespace(cmd='cmd', foo='1', rest=['1', '2', '3'])\n\n Bugs: http://bugs.python.org/issue14191\n Files: http://bugs.python.org/file26273/t18a.py\n Changes: renamed to ArgumentParser14191 ** PEP cleaned ** hidden\n ErrorParser inside ArgumentParser14191 ** documented ** used\n new-style classes super calls (Walter Doekes, March 2015)\n \"\"\"\n class ErrorParser(argparse.ArgumentParser):\n def __init__(self, *args, **kwargs):\n self.__errorobj = None\n kwargs['add_help'] = False\n super(ArgumentParser14191.ErrorParser, self).__init__(\n *args, **kwargs)\n\n def error(self, message):\n if self.__errorobj:\n self.__errorobj.error(message)\n else:\n argparse.ArgumentParser.error(self, message)\n\n def seterror(self, errorobj):\n self.__errorobj = errorobj\n\n def __init__(self, *args, **kwargs):\n self.__setup = False\n self.__opt = ArgumentParser14191.ErrorParser(*args, **kwargs)\n super(ArgumentParser14191, self).__init__(*args, **kwargs)\n self.__opt.seterror(self)\n self.__setup = True\n\n def add_argument(self, *args, **kwargs):\n super(ArgumentParser14191, self).add_argument(*args, **kwargs)\n if self.__setup:\n chars = self.prefix_chars\n if args and len(args[0]) and args[0][0] in chars:\n self.__opt.add_argument(*args, **kwargs)\n\n def parse_args(self, args=None, namespace=None):\n ns, remain = self.__opt.parse_known_args(args, namespace)\n ns = super(ArgumentParser14191, self).parse_args(remain, ns)\n return ns\n\n\nclass CalledProcessError2(subprocess.CalledProcessError):\n def __str__(self):\n return '{}\\n\\n(output)\\n{}'.format(\n super().__str__(), self.output.decode('ascii', 'replace'))\n\n\nclass ProxmoxClusters(dict):\n @classmethod\n def from_filename(cls, filename):\n clusters = cls()\n parser = configparser.ConfigParser(\n interpolation=None, inline_comment_prefixes=('#', ';'),\n empty_lines_in_values=False)\n\n try:\n with open(filename) as fp:\n try:\n parser.read_file\n except AttributeError:\n parser.readfp(fp)\n else:\n parser.read_file(fp)\n except FileNotFoundError:\n raise ValueError('cannot access config file: {}'.format(\n filename))\n\n for section in parser.sections():\n if section == configparser.DEFAULTSECT:\n raise ValueError(\n 'unexpected default section: {}'.format(section))\n\n section_split = section.split(':')\n type_ = section_split.pop(0)\n\n # [pve:CLUSTER_ALIAS]\n if type_ == 'pve' and len(section_split) == 1:\n cluster_alias = section_split[0]\n clusters[cluster_alias] = ProxmoxCluster.from_section(\n cluster_alias, parser.items(section))\n # [storage:CLUSTER_ALIAS:STORAGE_NAME[@NODE]]\n elif type_ == 'storage' and len(section_split) == 2:\n cluster_alias = section_split[0]\n if cluster_alias not in clusters:\n raise ValueError(\n 'storage describing unknown cluster {!r}: {}'.format(\n cluster_alias, section))\n clusters[cluster_alias].add_storage(\n section_split[1], parser.items(section))\n # [...]\n else:\n raise ValueError(\n 'unknown section type; use pve/storage: {}'.format(\n section))\n\n return clusters\n\n\nclass ProxmoxPools(object):\n \"\"\"\n Helper to find out which pool a VM is in\n \"\"\"\n def __init__(self, cluster):\n self._cluster = cluster\n self._vm_to_pool = None\n\n def get_poolid(self, vmid):\n \"Get the poolid (name) that the VM is in\"\n self._populate_cache()\n return self._vm_to_pool.get(int(vmid), None)\n\n def ensure_poolid(self, poolid):\n \"Make sure poolid exists\"\n try:\n self._cluster.api.pools.create(poolid=poolid)\n except ResourceException as e:\n if str(e).startswith('403'):\n raise ProxmoveError(\n 'got 403 when trying to set pool info; '\n 'do you have the PVEPoolAdmin permission on /pool?')\n\n def set_poolid(self, vmid, poolid):\n \"Put VM in a poolid, creating the pool if necessary\"\n self.ensure_poolid(poolid)\n # vms is a comma-separated list of vmid strings.\n # (To remove one or more, we'd add delete=1.)\n # (If pool did not exist we'd get: 500, '' does not exist)\n self._cluster.api.pools(poolid).put(vms=str(vmid))\n\n # Update cache with newly set values.\n if self._vm_to_pool is not None:\n self._vm_to_pool[int(vmid)] = poolid\n\n def _populate_cache(self):\n if self._vm_to_pool is not None:\n return\n\n ret = {}\n for pooldict in self._cluster.api.pools.get():\n poolid = pooldict['poolid'] # {'poolid': 'FOOBAR'}\n try:\n poolinfo = self._cluster.api.pools(poolid).get()\n except ResourceException as e:\n if str(e).startswith('403'):\n raise ProxmoveError(\n 'got 403 when trying to get pool info; '\n 'do you have the PVEPoolAdmin permission on /pool?')\n raise\n\n for memberdict in poolinfo['members']:\n # Examples of memberdict:\n # {'name': 'voiphu2', ..., 'id': 'qemu/124', 'type': 'qemu',\n # 'node': 'rs05', 'vmid': 124, 'diskread': 779919072, ...}\n # {'id': 'storage/rs05/local', 'shared': 0,\n # 'type': 'storage', ..., 'node': 'rs05', ...}\n # We're only interested in VMs in the pool.\n if 'vmid' in memberdict:\n vmid = int(memberdict['vmid'])\n assert vmid not in ret, (ret, memberdict)\n ret[vmid] = poolid\n\n self._vm_to_pool = ret\n\n\nclass ProxmoxCluster(object):\n @classmethod\n def from_section(cls, name, section):\n cluster = cls(name)\n\n for key, value in section:\n if key == 'api':\n if cluster.api_url:\n raise ValueError(\n 'duplicate api key in pve section {!r}'.format(\n name))\n cluster.api_url = value\n else:\n raise ValueError(\n 'unknown key {!r} in pve section {!r}'.format(\n key, name))\n\n return cluster\n\n def __init__(self, name):\n self.name = name\n self.repoid = None\n self.api_url = None\n self.api_verify_ssl = True\n self.pools = ProxmoxPools(self)\n self._cache = {}\n self._storages = {}\n self._storages_nodespecific = defaultdict(dict)\n self._vms_by_id = {}\n\n def no_verify_ssl(self):\n \"Call to disable SSL verification.\"\n self.api_verify_ssl = False\n\n def set_bandwidth_limit(self, bandwidth_limit):\n # assume that no storages will be added after calling this\n for storage in self._storages.values():\n storage.set_bandwidth_limit(bandwidth_limit)\n\n def set_ssh_ciphers(self, ssh_ciphers):\n # assume that no storages will be added after calling this\n for storage in self._storages.values():\n storage.set_ssh_ciphers(ssh_ciphers)\n\n def add_storage(self, name, section):\n try:\n name, node = name.split('@', 1)\n except ValueError:\n storage = ProxmoxStorage.from_section(name, section)\n self._storages[name] = storage\n else:\n storage = ProxmoxStorage.from_section(name, section)\n self._storages_nodespecific[node][name] = storage\n\n def get_nodes(self):\n if 'nodes' not in self._cache:\n tmp = self.api.nodes.get()\n log.debug('(api) %r nodes: %s', self.name, tmp)\n nodes = [\n node['node'] for node in tmp\n if node.get('status') == 'online' and node['type'] == 'node']\n nodes.sort()\n self._cache['nodes'] = nodes\n return self._cache['nodes']\n\n def get_storage(self, node, storage):\n ret = None\n if node in self._storages_nodespecific:\n ret = self._storages_nodespecific[node].get(storage)\n if not ret:\n ret = self._storages.get(storage)\n if not ret:\n raise ValueError(\n 'storage {!r} in {!r} cluster missing in config'.format(\n storage, self.name))\n return ret\n\n @property\n def api(self):\n if hasattr(self, '_api'):\n # Use manual timer, because proxmoxer times out and doesn't\n # re-auth after getting a 401. Of course this is a workaround,\n # it would've been better to handle the 401 and re-auth. If the\n # session is lost or has a lower timeout, we'd still fail.\n # See https://github.com/swayf/proxmoxer/issues/26\n api_age = (time.time() - self._api_start)\n if api_age > (30 * 60):\n del self._api\n\n if not hasattr(self, '_api'):\n try:\n res = urlparse(self.api_url)\n except InvalidURL as e:\n raise ValueError(\n 'splitting {!r} api {!r} URL failed: {}'.format(\n self.name, self.api_url, e))\n log.debug('(api) Connecting to %s', res.hostname)\n api = ProxmoxAPI(\n res.hostname, port=res.port, user=res.username,\n password=res.password, verify_ssl=self.api_verify_ssl)\n self._api = api\n self._api_start = time.time()\n\n return self._api\n\n def create_vm(self, config, nodeid, poolid):\n if not nodeid:\n nodeid = self.get_random_node()\n\n if poolid is not None:\n self.pools.ensure_poolid(poolid)\n\n name = config['name']\n suffix = SUFFIX_CREATING\n name_with_suffix = '{}{}'.format(name, suffix)\n\n log.info('Creating new VM {!r} on {!r}, node {!r}'.format(\n name_with_suffix, self.name, nodeid))\n\n # Create disks according to config. Temporarily set ide\\d+ and\n # virtio\\d+ devices to none until we can get them copied over.\n mutable_config = {}\n for key, value in config.items():\n if PROXMOX_VOLUME_TYPES_RE.match(key):\n # Wipe it. We'll add the disks manually later on.\n pass\n else:\n mutable_config[key] = value\n\n # Guess new VMID, set id and name.\n vmid = self.get_free_vmid()\n mutable_config['vmid'] = vmid\n mutable_config['name'] = name_with_suffix\n assert 'hostname' not in mutable_config, mutable_config # lxc??\n\n # Create new VM.\n log.debug('- creating VM: {!r}'.format(mutable_config))\n\n api_node = self.api.nodes(nodeid)\n try:\n vmhash = getattr(api_node, 'qemu').create(**mutable_config)\n except ResourceException:\n log.exception(\n 'Failed to create VM with parameters:\\n\\n'\n ' # {base_url}\\n'\n ' api.nodes(\"{node}/{type}\")'\n '.create(**{config})\\n\\n'\n 'Do you have PVEVMAdmin + PVESDNAdmin perms?\\n\\n'\n .format(\n base_url=self.api._backend.base_url,\n node=nodeid, type='qemu', config=mutable_config))\n raise\n\n # Wait a while to ensure that we get the VM.\n log.info(\n '- created new VM {!r} as {}; waiting for it to show up'.format(\n name_with_suffix, vmhash))\n for i in range(30):\n try:\n self._cache = {} # purge cache, we expect changes\n vm = self.get_vm(name, suffix=suffix)\n except ProxmoxVm.DoesNotExist:\n pass\n else:\n break\n time.sleep(1)\n else:\n raise ProxmoveError('Could not get newly created VM {!r}'.format(\n name_with_suffix))\n\n log.info('- created new VM {!r}: {}'.format(vm.name, vm))\n\n if poolid is not None:\n self.pools.set_poolid(vm.id, poolid)\n log.info('- placed VM {!r} in pool {}'.format(vm.name, poolid))\n\n return vm\n\n def get_vms_dict(self):\n if 'cluster.resources.type=vm' not in self._cache:\n self._cache['cluster.resources.type=vm'] = (\n self.api.cluster.resources.get(type='vm'))\n # log.debug(\n # '(api) %r vms: %s', self.name, sorted(\n # i['name']\n # for i in self._cache['cluster.resources.type=vm']))\n return self._cache['cluster.resources.type=vm']\n\n def get_free_vmid(self):\n \"\"\"\n BEWARE: To get the numbers right, we need to have enough\n permissions to see all VMs. PVEVMAdmin permissions required?\n \"\"\"\n self._cache = {} # purge cache, so we don't get stale VMID lists\n vms = self.get_vms_dict()\n if not vms:\n return 100\n ordered_vms = [vm['vmid'] for vm in vms]\n ordered_vms.sort()\n if (ordered_vms[-1] - ordered_vms[0] + 1) == len(ordered_vms):\n return ordered_vms[-1] + 1\n prev = ordered_vms[0]\n for vmid in ordered_vms[1:]:\n if prev + 1 != vmid:\n return prev + 1\n prev = vmid\n raise NotImplementedError('this cannot happen: {}'.format(\n ordered_vms))\n\n def get_random_node(self):\n nodes = self.get_nodes()\n return random.choice(nodes)\n\n def get_vm(self, name, suffix=''):\n name_with_suffix = name + suffix\n\n # Find exact match, but warn about others:\n candidates = []\n for vm in self._vms_by_id.values():\n if vm.name == name_with_suffix:\n return vm\n if vm.basename == name:\n candidates.append(vm)\n\n # Warn about other VMs found. This might be because you're\n # moving on the same cluster?\n if candidates:\n for vm in candidates:\n log.warning(\n 'Did not find {}, but did find {} when looking for VM...'\n .format(name_with_suffix, vm.name))\n\n # Look it up, or raise DoesNotExist.\n proxmox_vms = self.get_vms_dict()\n res = [vm for vm in proxmox_vms if vm.get('name') == name_with_suffix]\n if len(res) == 0:\n raise ProxmoxVm.DoesNotExist(\n 'VM named {!r} not found in cluster {!r}; '\n 'do you have the PVEVMAdmin role?'.format(\n name, self.name))\n elif len(res) > 1:\n raise ProxmoveError(\n 'VM named {!r} found multiple times in cluster {!r}'.format(\n name, self.name))\n\n # Insert and return.\n vm = ProxmoxVm.from_dict(name, res[0], cluster=self)\n self._vms_by_id[vm.id] = vm\n return vm\n\n def ping(self):\n version = self.api.version.get()\n if not isinstance(version, dict) or 'release' not in version:\n raise ProxmoveError(\n 'cluster {!r} did not return proper version: {!r}'.format(\n self.name, version))\n self.repoid = version['repoid']\n\n def __str__(self):\n if self.repoid:\n return '{}<{}>'.format(self.name, self.repoid)\n return self.name\n\n\nclass ProxmoxStorage(object):\n @classmethod\n def from_section(cls, name, section):\n storages = []\n for model in (ProxmoxStoragePlain, ProxmoxStorageZfs):\n try:\n storage = model.from_section(name, section)\n except PrepareError:\n pass\n else:\n storages.append(storage)\n\n if len(storages) != 1:\n raise PrepareError(\n 'storage section {!r} could not be parsed (requires ssh/path/'\n 'keys)'.format(name))\n\n storage = storages[0]\n storage.set_from_section(section)\n return storage\n\n def __init__(self, name):\n # Set from inifile.\n self.name = name\n self.ssh = None\n self.path = None\n self.temp = None\n\n # Set through command line.\n self.bwlimit_mbps = None\n self.ssh_ciphers = None\n\n def check_prerequisites(self):\n self.check_prerequisite_config()\n self.check_prerequisite_commands()\n self.check_prerequisite_paths()\n\n def check_prerequisite_config(self):\n if not (self.ssh and self.path and self.temp):\n raise PrepareError(\n 'missing one or more of ssh/path/temp in {!r} storage '\n 'section'.format(self.name))\n\n def check_prerequisite_commands(self):\n raise NotImplementedError('subclasses need to implement this')\n\n def check_prerequisite_paths(self):\n assert self.temp and self.temp.startswith('/'), self.temp\n try:\n self.ssh_command(['mkdir', '-p', self.temp])\n except subprocess.CalledProcessError:\n raise PrepareError(\n 'temp dir {!r} on storage {!r} does not exist; '\n 'please create it!'.format(self.temp, self.name))\n\n def get_physical_size(self, image_location):\n \"\"\"\n Get exact size of physical (host-observed) disk.\n \"\"\"\n raise NotImplementedError(\"subclasses need to implement this\")\n\n def get_volume_size(self, image_location):\n \"\"\"\n Get exact size of (guest-observed) volume.\n \"\"\"\n raise NotImplementedError(\"subclasses need to implement this\")\n\n def get_transfer_size(self, image_location):\n \"\"\"\n Get size which we need to transfer. Probably equals physical size.\n \"\"\"\n return self.get_physical_size(image_location)\n\n def test_copy(self, dst_storage, src_format=None):\n \"\"\"\n Check if we can copy from source (this) to destination (other).\n \"\"\"\n if not self.can_copy_to(dst_storage, src_format=src_format):\n raise NotImplementedError(\n 'format conversion from {!r} ({}) to {!r} not implemented'\n .format(\n type(self).__name__, src_format,\n type(dst_storage).__name__))\n\n def can_copy_to(self, dst_storage, src_format=None):\n raise NotImplementedError(\"subclasses need to implement this\")\n\n def can_copy_from(self, src_storage, src_format=None):\n raise NotImplementedError(\"subclasses need to implement this\")\n\n def copy(self, image_size, disk_size, src_location, src_format,\n dst_storage, dst_id, dst_name):\n new_path, new_format = dst_storage.copy_already_done(\n self, dst_name, disk_size, image_size)\n if new_path:\n return new_path, new_format\n\n dst_temp = self.copy_to_temp(src_location, dst_storage, dst_name)\n\n if dst_temp:\n log.info('Temp data {!r} on {}'.format(dst_temp, dst_storage))\n new_path, new_format = dst_storage.copy_from_temp(\n disk_size, dst_temp, src_format, dst_id, dst_name)\n else:\n assert not src_format, src_format\n new_path, new_format = self.copy_direct(\n image_size, src_location, dst_storage, dst_name)\n\n return new_path, new_format\n\n def copy_already_done(self, src_storage, dst_name, disk_size, image_size):\n raise NotImplementedError('subclasses need to implemented this')\n\n def copy_to_temp(self, src_location, dst_storage, dst_name):\n raise NotImplementedError('subclasses need to implemented this')\n\n def copy_from_temp(self, disk_size, src_temp, src_format, dst_id,\n dst_name):\n raise NotImplementedError('subclasses need to implemented this')\n\n def copy_direct(self, image_size, src_location, dst_storage, dst_name):\n raise NotImplementedError('subclasses need to implemented this')\n\n def run_command(self, command, hide_stderr=False, tty=False, via=None):\n if via is not None:\n command_str = argv2str(command)\n return via.ssh_command(\n [command_str], hide_stderr=hide_stderr, tty=tty, via_this=True)\n\n log.debug('(exec) %s', argv2str(command))\n\n kwargs = {}\n if tty:\n kwargs['stdin'] = sys.stdin\n kwargs['stdout'] = sys.stdout\n if hide_stderr:\n kwargs['stderr'] = subprocess.DEVNULL\n else:\n kwargs['stderr'] = sys.stderr\n\n if tty:\n # We don't need to catch KeyboardInterrupt and SystemExit\n # now that we have ssh -t.\n proc = subprocess.Popen(command, **kwargs)\n status = proc.wait()\n if status != 0:\n raise CalledProcessError2(\n returncode=status, cmd=command,\n output='Failure with status {}'.format(status))\n return ''\n\n if hide_stderr:\n kwargs['stderr'] = subprocess.PIPE\n proc = subprocess.Popen(command, stdout=subprocess.PIPE, **kwargs)\n (out, err) = proc.communicate()\n if proc.returncode != 0:\n if err:\n out = out + b'\\n\\n(stderr)\\n' + err\n raise CalledProcessError2(\n returncode=proc.returncode, cmd=command, output=out)\n return out\n\n def has_commands(self, commands):\n \"\"\"\n TODO: We should run one with _all_ commands we might need and then\n cache the result.\n \"\"\"\n fail = False\n try:\n result = self.ssh_command(['which'] + commands)\n except subprocess.CalledProcessError as e:\n fail = True\n result = e.output\n\n result = result.decode('ascii', 'replace').strip()\n found = set([i.rsplit('/', 1)[-1] for i in result.split('\\n')])\n expected = set(commands)\n missing = (expected - found)\n if missing:\n log.debug('Missing commands: %r', missing)\n return False\n if fail:\n return False\n # (double check against excess values..)\n assert not (found - expected), (expected, found)\n return True\n\n def _get_ssh_args(self, key_forwarding=False):\n ssh_command = ['ssh', '-o', 'LogLevel=error']\n\n if key_forwarding:\n # If we're doing key forwarding, we don't care about the\n # ciphers _to_ the this machine, as we expect data\n # transfers to go from this machine to a _next_ machine.\n ssh_command.append('-A')\n elif self.ssh_ciphers not in (None, '', 'default', 'defaults'):\n ssh_command.extend(['-c', self.ssh_ciphers])\n\n if ':' in self.ssh:\n user_host, port = self.ssh.split(':', 1)\n else:\n user_host, port = self.ssh, '22'\n if port != '22':\n ssh_command.extend(['-p', port])\n\n ssh_command.append(user_host)\n return ssh_command\n\n def ssh_command(self, command, hide_stderr=False, tty=False,\n via=None, via_this=False):\n assert isinstance(None, (ProxmoxStorage, type(None))), via\n\n if not hasattr(self, '_is_ssh_checked'):\n # We could auto-add host key using StrictHostKeyChecking=no,\n # but I'm not sure we want to.\n full_command = self._get_ssh_args() + ['/bin/true']\n try:\n self.run_command(full_command)\n except subprocess.CalledProcessError:\n raise PrepareError(\n 'failed to ssh to {!r} for storage {!r}: {}'.format(\n self.ssh, self.name, argv2str(full_command)))\n self._is_ssh_checked = True\n\n extra = []\n if tty and not via:\n extra.append('-t')\n return self.run_command(\n self._get_ssh_args(key_forwarding=via_this) + extra + command,\n hide_stderr=hide_stderr, tty=tty, via=via)\n\n def set_bandwidth_limit(self, bandwidth_limit):\n self.bwlimit_mbps = bandwidth_limit\n\n def set_ssh_ciphers(self, ssh_ciphers):\n self.ssh_ciphers = ssh_ciphers\n\n def set_from_section(self, section):\n for key, value in section:\n if key == 'ssh':\n self.set_ssh(value)\n elif key == 'path':\n self.set_path(value)\n elif key == 'temp':\n self.set_temp(value)\n\n def set_ssh(self, value):\n if self.ssh:\n raise ValueError(\n 'duplicate ssh key in {!r} storage section: {}'.format(\n self.name, value))\n if value.startswith('-'): # TODO: nor spaces or quotes...\n raise ValueError(\n 'ssh value cannot start with an option-dash in {!r} '\n 'storage section: {!r}'.format(self.name, value))\n self.ssh = value\n\n def set_path(self, value):\n if self.path:\n raise ValueError(\n 'duplicate path key in {!r} storage section: {}'.format(\n self.name, value))\n self.path = value\n\n def set_temp(self, value):\n if self.temp:\n raise ValueError(\n 'duplicate temp key in {!r} storage section: {}'.format(\n self.name, value))\n if not value.startswith('/'):\n raise ValueError(\n 'unexpected tokens in temp path in {!r} storage '\n 'section: {}'.format(self.name, value))\n self.temp = value\n\n def __str__(self):\n return self.name\n\n\nclass ProxmoxStoragePlain(ProxmoxStorage):\n has_discard_support = False\n has_snapshot_support = False\n\n @classmethod\n def from_section(cls, name, section):\n paths = [value for key, value in section if key == 'path']\n if len(paths) != 1 or not paths[0].startswith('/'):\n raise PrepareError('not my kind of config')\n\n return cls(name)\n\n def get_physical_size(self, image_location):\n \"\"\"\n Read physical disk size using OS tools (ls).\n\n (The size as observed by the VM host.)\n \"\"\"\n if not hasattr(self, '_get_physical_size'):\n self._get_physical_size = {}\n\n if image_location not in self._get_physical_size:\n path = os.path.join(self.path, image_location)\n # Use ls -l instead of stat because stat %s/%z is not\n # standardized across BSD/GNU.\n try:\n data = self.ssh_command(\n ['ls', '-l', path], hide_stderr=True)\n except subprocess.CalledProcessError:\n log.warning(\n 'Could not get image size from plain storage {!r}'.format(\n path))\n exact_size = None\n else:\n exact_size = int(data.split()[4].decode('ascii', 'replace'))\n self._get_physical_size[image_location] = exact_size\n\n return self._get_physical_size[image_location]\n\n def get_volume_size(self, image_location):\n \"\"\"\n Read header bytes from qcow/qcow2 image to determine volume size.\n\n (The size observed by the VM guest.)\n \"\"\"\n if image_location.endswith('.raw'):\n # Raw image? Then its size is equal to the file size.\n return self.get_physical_size(image_location)\n elif image_location.endswith(('.qcow', '.qcow2')):\n # Extract the filesystem size from the qcow/qcow2 image below.\n pass\n else:\n # No support yet for other formats.\n return None\n\n path = os.path.join(self.path, image_location)\n try:\n data = self.ssh_command(\n ['hd', path, '|', 'head', '-n2'], hide_stderr=True)\n except subprocess.CalledProcessError:\n return None\n\n # Extract qcow2 data. Example:\n # 00000000 51 46 49 fb 00 00 00 03 00 00 00 00 00 00 00 00 |...\n # 00000010 00 00 00 00 00 00 00 10 00 00 00 05 00 00 00 00 |...\n hexdata = (\n b''.join([i[10:58] for i in data.split(b'\\n')])\n .decode('ascii', 'replace').replace(' ', ''))\n try:\n # uint32_t magic; /* 'Q', 'F', 'I' followed by 0xfb. */\n if int(hexdata[0:8], 16) != 0x514649fb:\n raise ValueError('magic', hexdata[0:8])\n # uint32_t version;\n if int(hexdata[8:16], 16) > 3: # up to version 3 is known\n raise ValueError('version', hexdata[8:16])\n # uint64_t backing_file_offset;\n if int(hexdata[16:32], 16) != 0:\n raise ValueError('backing_file_offset', hexdata[16:32])\n # uint32_t backing_file_size;\n if int(hexdata[32:40], 16) != 0:\n raise ValueError('backing_file_size', hexdata[32:40])\n # VERSION1: uint32_t mtime;\n # VERSION2: uint32_t cluster_bits;\n # uint64_t size; /* in bytes */\n size = int(hexdata[48:64], 16)\n except ValueError as e:\n raise ValueError(\n 'bad QCOW{{,2}} header ({} = {}), path: {}, data: {}'.format(\n e.args[0], e.args[1], path, hexdata))\n\n return size\n\n def check_prerequisite_commands(self):\n if not self.has_commands(['ssh', 'rsync']):\n raise PrepareError(\n 'missing required binaries ssh and/or rsync on storage {!r}; '\n 'please install them'.format(self.name))\n\n def copy_already_done(self, src_storage, dst_name, disk_size, image_size):\n log.warning('FIXME: Non-zfs resume not implemented yet') # XXX\n return None, None\n\n def can_copy_to(self, dst_storage, src_format=None):\n # We don't do conversion. Hope the other side does.\n return dst_storage.can_copy_from(self, src_format=src_format)\n\n def can_copy_from(self, src_storage, src_format=None):\n # We don't do conversion and only know about qcow2 and raw.\n return isinstance(src_storage, ProxmoxStoragePlain) and src_format in (\n 'qcow2', 'raw')\n\n def copy_to_temp(self, src_location, dst_storage, dst_name):\n \"\"\"\n Copy image from source to a temp destination; checks pre-existence.\n\n Return tempfile path on destination.\n \"\"\"\n src_path = os.path.join(self.path, src_location)\n dst_temp = os.path.join(dst_storage.temp, 'temp-proxmove', dst_name)\n\n # mkdir temp location\n dst_storage.ssh_command(['mkdir', '-p', os.path.dirname(dst_temp)])\n try:\n # test -f on the destination file. The assumption is that it\n # doesn't exist. If it does, we'll have to resume OR abort.\n dst_storage.ssh_command(['test', '!', '-f', dst_temp])\n except subprocess.CalledProcessError:\n # It exists already. \"Compare\" files and auto-resume of equal.\n self._copy_to_temp_verify_existing(\n src_location, dst_storage, dst_temp)\n else:\n # It doesn't exist. Do copy.\n self._copy_to_temp_exec(src_path, dst_storage, dst_temp)\n\n return dst_temp\n\n def _copy_to_temp_verify_existing(self, src_location, dst_storage,\n dst_temp):\n \"\"\"\n Check equality of src_path and dst_temp and raise error if unequal.\n \"\"\"\n src_size, dst_size = self.get_physical_size(src_location), -1\n assert src_size != dst_size\n try:\n # Use ls -l instead of stat because stat %s/%z is not\n # standardized across BSD/GNU.\n data = dst_storage.ssh_command(\n ['ls', '-l', dst_temp], hide_stderr=True)\n dst_size = int(data.split()[4].decode('ascii', 'replace'))\n except (subprocess.CalledProcessError, UnicodeDecodeError, ValueError):\n pass\n log.debug(\n 'Comparing existing files {!r} (size {}) with '\n '{!r} (size {})'.format(\n src_location, src_size, dst_temp, dst_size))\n if dst_size != src_size:\n raise ProxmoveError(\n 'Temp file {!r} exists on target with different file '\n 'size; please examine (and remove it?)'.format(dst_temp))\n log.info(\n 'File {!r} exists on destination already; resuming because sizes '\n 'are equal ({})'.format(dst_temp, dst_size))\n\n def _copy_to_temp_exec(self, src_path, dst_storage, dst_temp):\n \"\"\"\n Copy src_path over dst_temp on target. Overwrites existing files.\n \"\"\"\n if ':' in dst_storage.ssh:\n user_host, port = dst_storage.ssh.split(':', 1)\n else:\n user_host, port = dst_storage.ssh, '22'\n\n rsync_dest = '{}:{}'.format(user_host, dst_temp)\n log.info('rsync(1) copy from {!r} (on {}) to {!r}'.format(\n src_path, self, rsync_dest))\n\n # Build remote ssh command for rsync. (Prefer rsync(1) over scp(1)\n # because scp has been observed to be slower.)\n rsync_ssh = ['ssh', '-o', 'LogLevel=error', '-o', 'StrictHostKeyChecking=no']\n if port != 22:\n rsync_ssh.extend(['-p', str(port)])\n if self.ssh_ciphers not in (None, '', 'default', 'defaults'):\n rsync_ssh.extend(['-c', self.ssh_ciphers])\n\n # rsync, using ssh+rsync instead of local-rsync, so we can add our\n # beloved options. (Double quoted because this is passed to ssh_command\n # first.)\n rsync_cmd = [\n 'rsync', '--rsh', '\"{}\"'.format(' '.join(rsync_ssh)), '--progress']\n if False:\n rsync_cmd.append('-z') # gzip compression (yuck, prefer lz4)\n\n # Add bandwidth limits in MByte/s\n if self.bwlimit_mbps:\n rsync_cmd.append('--bwlimit={}m'.format(self.bwlimit_mbps / 8.0))\n # Source/destination.\n rsync_cmd.extend([src_path, rsync_dest])\n # Exec.\n self.ssh_command(rsync_cmd, tty=True)\n\n def copy_from_temp(self, disk_size, src_temp, src_format, dst_id,\n dst_name):\n if src_format not in ('qcow2', 'raw'):\n raise NotImplementedError(\n 'format conversion from {!r} to plain not implemented'.format(\n src_format))\n dst_format = src_format\n\n rel_path = os.path.join(\n str(dst_id), '{}.{}'.format(dst_name, dst_format))\n dst_path = os.path.join(self.path, rel_path)\n\n log.info('Moving data from {!r} to {!r}'.format(src_temp, dst_path))\n self.ssh_command(['mkdir', '-p', os.path.dirname(dst_path)])\n self.ssh_command(['mv', src_temp, dst_path])\n\n # In case old_format != new_format, we would need to update\n # properties. So the following must be true for now.\n assert src_format == dst_format, (src_format, dst_format)\n return rel_path, dst_format\n\n def set_path(self, value):\n if not value.startswith('/'):\n raise ValueError(\n 'path should start with / in {!r} storage section: {}'.format(\n self.name, value))\n return super().set_path(value)\n\n\nclass ProxmoxStorageZfs(ProxmoxStorage):\n has_discard_support = True\n has_snapshot_support = True\n\n @classmethod\n def from_section(cls, name, section):\n paths = [value for key, value in section if key == 'path']\n if len(paths) != 1 or not paths[0].startswith('zfs:'):\n raise PrepareError('not my kind of config')\n\n return cls(name)\n\n def check_prerequisite_commands(self):\n if not self.has_commands(['ssh', 'zfs', 'mbuffer', 'rsync']):\n raise PrepareError(\n 'missing one or more required binaries mbuffer, rsync, ssh, '\n 'zfs, on storage {!r}; please install them'.format(self.name))\n\n def get_physical_size(self, image_location):\n \"\"\"\n Get exact size of physical (host-observed) disk (= zvol size).\n \"\"\"\n return self.get_volume_size(image_location)\n\n def get_volume_size(self, image_location):\n \"\"\"\n Get exact size of (guest-observed) volume.\n \"\"\"\n pool = '{}/{}'.format(self.path, image_location)\n data = self.ssh_command(\n ['zfs', 'get', '-Hpo', 'value', 'volsize', pool], hide_stderr=True)\n number = data.decode('ascii', 'replace').strip()\n if not number:\n return None\n\n return int(number)\n\n def get_transfer_size(self, image_location):\n \"\"\"\n Get size which we need to transfer. This is an estimate!\n \"\"\"\n temp_snapname = '{}/{}@temp-{}'.format(\n self.path, image_location, random.random())\n # XXX: When resuming a snapshot, this won't be correct.\n data = self.ssh_command(\n ['zfs', 'snapshot', temp_snapname, '&&',\n 'zfs', 'send', '-Rnv', temp_snapname, '&&',\n 'zfs', 'destroy', temp_snapname], hide_stderr=True)\n parts = data.split()\n if not parts:\n log.warning(\n 'Could not get image size from ZFS snapshot; '\n 'trying volume size instead')\n return self.get_volume_size(image_location)\n\n human_size = parts[-1].decode('ascii', 'replace')\n # Add a bit of size since it's a guesstimate. You'd rather not\n # see the transfer go over 100%.\n return int(human_size_scan(human_size) * 1.02)\n\n def copy_already_done(self, src_storage, dst_name, disk_size, image_size):\n assert self.has_snapshot_support, self\n if src_storage.has_snapshot_support:\n # No need to skip, we can just finalize with a tiny snapshot sync.\n # Plus, we cannot be sure that we have all data, since we may have\n # started migrating while the source VM was still up.\n return None, None\n\n try:\n found_size = self.get_volume_size(dst_name)\n except CalledProcessError2:\n # Does not exist. All good. We'll copy it. (Or resume if possible.)\n return None, None\n if disk_size == found_size:\n return dst_name, None\n return ProxmoveError('Volume {!r} exists.. this is bad'.format(\n dst_name))\n\n def can_copy_to(self, dst_storage, src_format=None):\n # Same storage, we should be good.\n if isinstance(dst_storage, ProxmoxStorageZfs):\n assert src_format is None, src_format\n return True\n\n # We don't know how to send raw images. Simply cat /dev/zvol/...?\n return False\n\n def can_copy_from(self, src_storage, src_format=None):\n # Same storage, we should be good.\n if isinstance(src_storage, ProxmoxStorageZfs):\n assert src_format is None, src_format\n return True\n\n # We can read/convert raw/qcow2 from ProxmoxStoragePlain.\n if isinstance(src_storage, ProxmoxStoragePlain) and src_format in (\n 'qcow2', 'raw'):\n return True\n\n return False\n\n def copy_to_temp(self, src_location, dst_storage, dst_name):\n if isinstance(dst_storage, ProxmoxStorageZfs):\n # ZFS->ZFS copying requires no temp files.\n return None\n else:\n # Probably copy /dev/zvol/... and set src_format='raw'.\n raise NotImplementedError(\n 'ZFS->other copying is not implemented yet')\n\n def copy_from_temp(self, disk_size, src_temp, src_format, dst_id,\n dst_name):\n assert disk_size, disk_size\n dst_zfs = '{}/{}'.format(self.path, dst_name)\n self.ssh_command(['zfs', 'create', '-V', str(disk_size), dst_zfs])\n\n dst_path = os.path.join('/dev/zvol', self.path, dst_name)\n log.info('Writing data from temp {!r} to {!r} (on {})'.format(\n src_temp, dst_path, self))\n\n if src_format is None:\n src_format = 'raw'\n if src_format not in ('qcow2', 'raw'):\n raise NotImplementedError(\n 'format conversion from {!r} not implemented'.format(\n src_format))\n\n self.ssh_command(\n # -n = no create volume\n # -p = progress\n # -t none = cache=none\n # qemu-img provides various cache options:\n # - writethrough = default, O_DSYNC (safest, slowest)\n # - writeback = new default, syncing to host [fast, resource hog]\n # - none = sync to host, O_DIRECT (bypass host page cache)\n # - unsafe = write to host, ignore all sync [fast, resource hog]\n # We would like 'unsafe' to get the best performance.\n # Unfortunately this appears to block the target system I/O too\n # much, causing I/O load spikes on other users of the system.\n # Using 'none' does not have this effect instead, at the expense\n # of a slower import (conversion).\n ['qemu-img', 'convert', '-n', '-p', '-t', 'none',\n '-f', src_format, '-O', 'raw', src_temp, dst_path], tty=True)\n log.info('Removing temp {!r} (on {})'.format(dst_path, self))\n self.ssh_command(['rm', src_temp])\n\n # Return name and format (dst_name is the filesystem name on the\n # ZFS pool known to belong to dst_storage).\n return dst_name, None\n\n def _get_latest_zfs_snapshots(self, zfs_name):\n \"Get ZFS snapshots by latest first\"\n try:\n snapshots = self.ssh_command([\n 'zfs', 'list', '-Honame', '-d1', '-tsnapshot', '-Screation',\n zfs_name], hide_stderr=True)\n except CalledProcessError2:\n # If the destination volume did not exist, we'd end up here.\n snapshots = []\n else:\n # Might be empty if the volume existed but had no snapshots.\n snapshots = snapshots.decode('ascii').strip()\n if snapshots:\n snapshots = [\n snapshot.split('@', 1)[1]\n for snapshot in snapshots.split('\\n')]\n else:\n snapshots = []\n return snapshots\n\n def copy_direct(self, image_size, src_location, dst_storage, dst_name):\n # ...\n if ':' in dst_storage.ssh:\n user_host, port = dst_storage.ssh.split(':', 1)\n else:\n user_host, port = dst_storage.ssh, '22'\n\n src_zfs_snapshot = 'proxmove-{}'.format(\n datetime.now().strftime('%y%m%d-%H%M%S'))\n src_zfs = '{}/{}'.format(self.path, src_location)\n dst_zfs = '{}/{}'.format(dst_storage.path, dst_name)\n log.info('zfs(1) send/recv {} data from {!r} to {!r} (on {})'.format(\n (image_size and human_size_fmt(image_size) or ''),\n src_zfs, dst_zfs, dst_storage))\n\n # Get any snapshots that existed already ordered by newest first.\n src_snapshots = self._get_latest_zfs_snapshots(src_zfs)\n dst_snapshots = dst_storage._get_latest_zfs_snapshots(dst_zfs)\n for try_snapshot in src_snapshots:\n if try_snapshot in dst_snapshots:\n incremental = '-I {}'.format(try_snapshot)\n force_receive = '-F'\n break\n else:\n incremental = force_receive = ''\n\n # Create new snapshot.\n self.ssh_command([\n 'zfs', 'snapshot', '{}@{}'.format(src_zfs, src_zfs_snapshot)])\n\n # mbuffer takes k-, M- or G-bytes\n mbuffer_write_limit = ''\n if self.bwlimit_mbps:\n mbuffer_write_limit = '-R {}k'.format(self.bwlimit_mbps * 128)\n\n # zfs send sends unencrypted/decompressed -- prefer compression\n if (self.has_commands(['lz4']) and\n dst_storage.has_commands(['lz4'])):\n optional_lz4_spipe = 'lz4 -z | '\n optional_lz4_dpipe = 'lz4 -d | '\n else:\n optional_lz4_spipe = optional_lz4_dpipe = ''\n log.warning(\n 'lz4(1) command is not found on the src or dest storage; '\n 'consider installing it to get better transfer rates')\n\n # pv shows a nice progress bar. It's optional.\n if dst_storage.has_commands(['pv']):\n optional_pv_pipe = (\n # --fineta does not exist on all versions..\n # --force is required to make it display anything..\n 'pv --force --eta --average-rate --progress -s {} | '.format(\n image_size))\n else:\n optional_pv_pipe = ''\n log.warning(\n 'pv(1) command is not found on the destination storage; '\n 'consider installing it to get a pretty progress bar')\n\n if self.ssh_ciphers not in (None, '', 'default', 'defaults'):\n ssh_ciphers = '-c {}'.format(self.ssh_ciphers)\n else:\n ssh_ciphers = ''\n\n # Older mbuffer(1) [v20140310-3] on the receiving end\n # may say: \"mbuffer: warning: No controlling terminal\n # and no autoloader command specified.\" This is fixed\n # in newer versions [v20150412-3].\n # pv(1) on dst will show the speed *after* decompression.\n self.ssh_command(\n [\"zfs send -R {incremental} {src_zfs}@{src_zfs_snapshot} | \"\n \"{compress} mbuffer -q -s 128k -m 1G {src_bwlim} | \"\n \"ssh {ssh_ciphers} -o LogLevel=error -o StrictHostKeyChecking=no \"\n \"{dst_userhost} -p {dst_port} \"\n \"'mbuffer {optional_quiet_mbuffer} -s 128k -m 1G | \"\n \" {decompress} {dst_pv}\"\n \" zfs recv {force_receive} {dst_zfs}'\".format(\n incremental=incremental,\n src_zfs=src_zfs,\n src_zfs_snapshot=src_zfs_snapshot,\n src_bwlim=mbuffer_write_limit,\n ssh_ciphers=ssh_ciphers,\n dst_userhost=user_host,\n dst_port=port,\n optional_quiet_mbuffer=(\n '-q' if optional_pv_pipe else ''),\n compress=optional_lz4_spipe,\n decompress=optional_lz4_dpipe,\n dst_pv=optional_pv_pipe,\n force_receive=force_receive,\n dst_zfs=dst_zfs)],\n tty=True,\n via_this=True)\n\n # Return name and format (dst_name is the filesystem name on the\n # ZFS pool known to belong to dst_storage).\n return dst_name, None\n\n def set_path(self, value):\n assert value.startswith('zfs:'), value\n pool_name = value[4:]\n valid_characters = string.ascii_letters + string.digits + '_-/'\n if not pool_name or pool_name.startswith('/') or not all(\n i in valid_characters for i in pool_name):\n raise PrepareError(\n 'invalid characters in zfs pool name found {!r} storage '\n 'section: {}'.format(self.name, value))\n return super().set_path(pool_name)\n\n\nclass ProxmoxVm(object):\n class DoesNotExist(ProxmoveError):\n pass\n\n @classmethod\n def from_dict(cls, name, dict_, cluster):\n vm = cls(\n name, dict_['name'], dict_['node'], dict_['type'], dict_['vmid'],\n dict_['status'], cluster)\n\n # LXC(FIXME): When creating an LXC container, the get_config()\n # may contain a 'digest' only. We need to wait until its fully\n # populated with at least the name. This is a hack, and we\n # definitely should not use AssertionError for it.\n for i in range(10):\n try:\n # Get config and check for pending changes at once.\n vm.get_config()\n except AssertionError:\n time.sleep(1)\n else:\n break\n else:\n raise\n\n return vm\n\n def __init__(self, basename, name, node, type_, id_, status, cluster):\n self.basename = basename # without \"--CREATING\" or similar suffix\n self.name = name\n self.node = node\n self.type = type_ # qemu|lxc|...\n self.id = id_\n self.status = status # \"running\"/\"stopped\"\n # NOTE: api_vm does not get any magic refresh that lazy calling of the\n # cluster.api provides.\n self.api_vm = getattr(cluster.api.nodes(node), type_)(id_)\n self.cluster = cluster\n self.volumes = 0 # used to index/create the new volume names\n self._cache = {}\n\n @property\n def poolid(self):\n return self.cluster.pools.get_poolid(self.id)\n\n def set_translator(self, config_translator):\n self._translator = config_translator\n\n def check_config(self):\n \"\"\"\n Check whether we can move this VM.\n \"\"\"\n if self.type == 'lxc':\n # No pending changes\n return\n\n config = self.get_config()\n\n # Check pending. We expect a list of dictionaries with a 'key'\n # key and 'value' and/or 'pending' keys.\n pending_config = self.api_vm.pending.get() # does not exist for lxc\n pending = []\n for dict_ in pending_config:\n keys = dict_.keys()\n if keys == set(['key', 'value']):\n assert config.get(dict_['key']) == dict_['value']\n else:\n pending.append('{!r}: {!r} => {!r}'.format(\n dict_['key'], dict_.get('value'),\n dict_.get('pending')))\n\n if pending:\n # Contains 'pending' changes. Refuse to continue.\n pending.sort()\n raise PrepareError(\n 'VM {!r} contains pending changes:\\n {}'.format(\n self.name, '\\n '.join(pending)))\n\n def get_config(self):\n \"\"\"\n Get current configuration and store name.\n \"\"\"\n if 'config' not in self._cache:\n next_config = self.api_vm.config.get()\n name = name_from_conf(self.type, next_config)\n assert name == self.name, (name, self.name)\n self._cache['config'] = next_config\n return self._cache['config']\n\n def create_volume(\n self, volume_key, source_volume, storage,\n requires_snapshot_support):\n \"\"\"\n Create volume from source_volume.\n\n If requires_snapshot_support, this is called when the source machine is\n still up.\n \"\"\"\n assert PROXMOX_VOLUME_TYPES_RE.match(volume_key), volume_key\n\n if storage is None:\n # Take properties and set first argument to 'none'.\n # E.g. \"san06:abc.iso,media=cdrom\" => \"none,media=cdrom\"\n parts = source_volume.properties.split(',', 1)\n parts[0] = 'none'\n properties = ','.join(parts)\n self.api_vm.config.put(**{volume_key: properties})\n self._cache = {}\n log.info('Ejected (cdrom?) volume {!r} ({}) added to {}'.format(\n volume_key, properties, self))\n elif not requires_snapshot_support or (\n source_volume.storage.has_snapshot_support and\n storage.has_snapshot_support):\n # Generate the new disk name. This might be completely\n # different from what you had on the source. Here it's by\n # disk order (taken from the source volumes ordered dict).\n new_volume_name = PROXMOVE_VOLUME_NAME_FMT.format(\n vmid=self.id, diskidx=self.volumes)\n\n # We actually have to do copying.\n log.info('Begin copy of {!r} ({}) to {} ({})'.format(\n volume_key, source_volume, storage, new_volume_name))\n new_volume = source_volume.clone(\n new_storage=storage, new_vmid=self.id,\n new_volume_name=new_volume_name)\n new_properties = self._translator.volume_properties(new_volume)\n try:\n self.api_vm.config.put(**{volume_key: new_properties})\n except Exception:\n log.exception(\n 'Issue when adding volume to new VM: {!r}'.format(\n new_properties))\n raise\n\n self.volumes += 1\n else:\n # Also increment volumes. So we'll use the same order of disks.\n self.volumes += 1\n\n def get_volumes(self):\n if 'volumes' not in self._cache:\n volumes = OrderedDict()\n for slot, value in sorted(self.get_config().items()):\n volume = ProxmoxVolume.from_cluster_node_and_config(\n self.cluster, self.node, slot, value)\n if volume:\n volumes[slot] = volume\n self._cache['volumes'] = volumes\n return self._cache['volumes']\n\n def rename(self, new_name):\n if self.type == 'lxc':\n self.api_vm.config.put(hostname=new_name)\n else:\n self.api_vm.config.put(name=new_name)\n\n self.name = new_name\n\n def configure_autoboot(self, boot):\n boot_int = 1 if boot else 0\n self.api_vm.config.put(onboot=boot_int)\n\n def ensure_started(self, timeout=120):\n if self.status == 'running':\n log.debug('Skipping start, already running: {}'.format(self))\n return\n\n log.info('Starting VM {}'.format(self))\n\n self.api_vm.status.start.create()\n for i in range(timeout + 10):\n time.sleep(1)\n status = self.api_vm.status.current.get()\n if status['status'] == 'running':\n self.status = status['status']\n break\n else:\n self.status = status['status']\n raise ProxmoveError(\n 'VM {!r} refuses to start: status = {!r}'.format(\n self.name, self.status))\n\n log.info('- started VM {}'.format(self))\n\n def ensure_stopped(self, timeout=120):\n if self.status == 'stopped':\n log.debug('Skipping stop, already stopped: {}'.format(self))\n return\n\n log.info('Stopping VM {}; will forcibly kill after {} seconds'.format(\n self, timeout + 10))\n\n # forceStop takes a boolean, but proxmoxer won't pass True as\n # 'true', but as 'True' which is not valid JSON.\n self.api_vm.status.shutdown.create(forceStop='1', timeout=timeout)\n for i in range(timeout + 10):\n time.sleep(1)\n status = self.api_vm.status.current.get()\n if status['status'] == 'stopped':\n self.status = status['status']\n break\n else:\n self.status = status['status']\n raise ProxmoveError(\n 'VM {!r} refuses to shut down: status = {!r}'.format(\n self.name, self.status))\n\n log.info('- stopped VM {}'.format(self))\n\n def add_comment(self, comment):\n config = self.get_config()\n if 'description' in config:\n comment = config['description'].rstrip() + '\\n' + comment.strip()\n else:\n comment = comment.strip()\n\n self.api_vm.config.put(description=comment)\n self._cache = {} # drop cache\n\n def __str__(self):\n return '{}@{}<{}/{}/{}>'.format(\n self.name, self.node, self.type, self.id, self.status)\n\n\nclass ProxmoxVolume(object):\n @classmethod\n def from_cluster_node_and_config(self, cluster, node, slot, value):\n \"\"\"\n Return ProxmoxVolume from cluster, node, slot and value, or None\n if this is not a volume type.\n \"\"\"\n if not PROXMOX_VOLUME_TYPES_RE.match(slot):\n return None\n\n # cluster = <...>\n # node = 'pveX'\n # slot = 'ide2', 'virtio0'\n # value = 'local:iso/ubuntu-18.04-amd64.iso,media=cdrom',\n # 'nfs07:113/vm-113-disk-0.qcow2,size=70G'\n # 'none,media=cdrom'\n try:\n storage_location, properties = value.split(',', 1)\n if storage_location == 'none':\n storage, location = None, None\n else:\n storage, location = storage_location.split(':', 1)\n except ValueError:\n raise ProxmoveError('disk slot {!r} parse error: {!r}'.format(\n slot, value))\n\n # Removable disk? Assume it's ejected, so we don't need to look\n # up the storage for it.\n if 'media=cdrom' in properties.split(','):\n storage = location = None\n\n if storage:\n # Might raise exception if not found.\n storage = cluster.get_storage(node, storage)\n\n return ProxmoxVolume(location, properties, storage=storage)\n\n def __init__(self, location, properties, storage=None):\n self.location = location\n self.properties = properties\n self.storage = storage\n\n def is_removable(self):\n parts = self.properties.split(',')\n return self.location is None or 'media=cdrom' in parts\n\n def get_format(self):\n \"\"\"\n Return image file format. Use the format property, or the image\n file extension as fallback.\n \"\"\"\n ret = self.get_property('format')\n\n if isinstance(self.storage, ProxmoxStorageZfs):\n if ret is None:\n pass\n elif ret == 'raw':\n # Could be a stale setting from a previous migration?\n log.debug(\n '- ignoring superfluous format=raw on ZFS filesystem')\n ret = None\n else:\n raise ProxmoveError(\n 'unexpected format={} on ZFS source'.format(ret))\n elif isinstance(self.storage, ProxmoxStoragePlain):\n # For the plain storage, we must have a format. Take the\n # file extension.\n if not ret:\n ret = self.location.rsplit('.', 1)[-1]\n\n return ret\n\n def get_property(self, what):\n search = '{}='.format(what)\n parts = self.properties.split(',')\n for part in parts:\n if part.startswith(search):\n return part[len(search):]\n return None\n\n def get_properties(self, **updates):\n \"\"\"\n Get properties list (abc=def,ghi=jkl) but update properties from\n the updates argument or leave them out if the value is None.\n \"\"\"\n search_list = tuple('{}='.format(i) for i in updates.keys())\n ret = []\n\n parts = self.properties.split(',')\n for part in parts:\n if not part.startswith(search_list):\n ret.append(part)\n\n for key, value in updates.items():\n if value:\n parts.append('{}={}'.format(key, value))\n\n return ','.join(ret)\n\n def get_size(self, observer):\n \"\"\"\n Get size of volume, either 'guest' size (filesystem size,\n visible to the guest OS) or 'host' size (image/blob size,\n observed when transferring data between storage nodes).\n \"\"\"\n # filesystem size vs image/blob size\n assert observer in ('host', 'guest'), observer\n cache_key = '_get_size__{}'.format(observer)\n\n if not hasattr(self, cache_key):\n if self.storage:\n if observer == 'host':\n size = self.storage.get_transfer_size(self.location)\n elif observer == 'guest':\n size = self.storage.get_volume_size(self.location)\n size_property = self.get_property('size') # \",size=64G\"\n if size_property:\n size_property = human_size_scan(size_property)\n assert size == size_property, (size, size_property)\n else:\n raise NotImplementedError(observer)\n setattr(self, cache_key, size)\n else:\n setattr(self, cache_key, None)\n\n return getattr(self, cache_key)\n\n def get_human_size(self, observer):\n size = self.get_size(observer)\n if not size:\n return ''\n return human_size_fmt(size)\n\n def test_clone(self, new_storage):\n \"\"\"\n Do preliminary checks before cloning/copying a volume.\n \"\"\"\n assert isinstance(new_storage, ProxmoxStorage), new_storage\n self.storage.test_copy(new_storage, src_format=self.get_format())\n\n def clone(self, new_storage, new_vmid, new_volume_name):\n \"\"\"\n We expect new_volume_name to be constructed from\n PROXMOVE_VOLUME_NAME_FMT.\n \"\"\"\n assert isinstance(new_storage, ProxmoxStorage), new_storage\n assert str(new_vmid) in new_volume_name, (new_volume_name, new_vmid)\n\n # Create new name and let the storage backend do the copying.\n new_path, new_format = self.storage.copy(\n self.get_size('host'), self.get_size('guest'),\n self.location, self.get_format(),\n new_storage, new_vmid, new_volume_name)\n\n # Log message so we know that any percentages shown really are\n # irrelevant. (pv(1) gets a guesstimate which may be off by a\n # few percent. We don't want the user to think the transfer\n # stopped at 98%.)\n log.info('Volume transferring/conversion 100% (is/was) complete!')\n return ProxmoxVolume(\n new_path, self.get_properties(format=new_format),\n storage=new_storage)\n\n def as_properties(self):\n \"\"\"\n Return something like: \"san06:123/vm-123-disk.qcow2,size=50G,...\"\n \"\"\"\n if self.storage:\n location = '{}:{}'.format(self.storage.name, self.location)\n else:\n assert self.location == 'none', self.location\n location = self.location\n\n if self.properties:\n return '{},{}'.format(location, self.properties)\n return location\n\n def __str__(self):\n if self.storage:\n return '{}:{},{}'.format(\n self.storage.name, self.location, self.properties)\n return '{},{}'.format(self.location, self.properties)\n\n\nclass BaseConfigTranslator(object):\n def config(self, old_config):\n \"\"\"\n Translate VM dict from source to something writable\n\n Drops readonly properties\n \"\"\"\n new_config = {}\n for key, value in old_config.items():\n if key == 'digest':\n # The digest is used to prevent changes if the current\n # config doesn't match the digest. This blocks\n # concurrent updates.\n pass\n elif key == 'meta':\n # Proxmox [67].x may provide a meta key in the listing, but it\n # will not accept any on creation. Example:\n # {'meta': 'creation-qemu=6.1.0,ctime=1643742755'}\n pass\n elif key == 'vmgenid':\n # Let proxmox generate this id. Otherwise we may get a \"only\n # root can set 'vmgenid' config\".\n pass\n elif key == 'smbios1':\n new_config[key] = self._set_smbios1(value)\n else:\n new_config[key] = value\n\n # Overwrite boot with a safe 'cd' default when\n # 'order=virtio0;ide2;net0' is not supported on this 6.1+6.3 cluster\n # mix.\n if True:\n log.debug('- setting boot=cd workaround for 6.1+6.3 mixcluster')\n new_config['boot'] = 'cd'\n\n return new_config\n\n def volume_properties(self, new_volume):\n \"\"\"\n Translate volume properties from source to something writable\n\n Returns new_volume.as_properties()\n \"\"\"\n return new_volume.as_properties()\n\n def _set_smbios1(self, smbios1):\n # Fix up smbios1 for changes between PVE 5.x and PVE 6.x:\n # uuid=12-34-56,manufacturer=blah-123\n # --> uuid=12-34-56,base64=1,manufacturer=YmxhaC0xMjM=\n if ',' in smbios1 and ',base64=1,' not in smbios1:\n parts = smbios1.split(',')\n assert parts[0].startswith('uuid='), smbios1\n out = [parts[0], 'base64=1']\n for part in parts[1:]:\n key, value = part.split('=', 1)\n b64value = b64encode(value.encode('utf-8')).decode('ascii')\n out.append('{}={}'.format(key, b64value))\n smbios1 = ','.join(out)\n log.debug('- setting smbios1 to {!r}'.format(smbios1))\n return smbios1\n\n\nclass TurnLxcIntoQemuConfigTranslator(object):\n def config(self, old_config):\n new_config = super().config(old_config)\n if 'name' not in new_config and 'hostname' in new_config:\n new_config = self._lxc_to_qemu(new_config)\n return new_config\n\n def _lxc_to_qemu(self, old_config):\n new_config = {}\n for key, value in old_config.items():\n if key == 'hostname':\n new_config['name'] = value\n elif key in (\n 'arch', 'cpulimit', 'cpuunits', 'nameserver',\n 'searchdomain', 'rootfs', 'ostemplate', 'ostype', 'swap'):\n pass\n elif key.startswith('net') and key[3:].isdigit():\n new_config[key] = self._lxc_net_to_qemu_net(value)\n else:\n new_config[key] = value\n\n return new_config\n\n def _lxc_net_to_qemu_net(self, net):\n # name=eth0,bridge=vmbr0,gw=10.x.x.x,\\\n # hwaddr=00:C0:FF:EE:BE:EF,ip=10.x.x.x/y,type=veth\n # -> virtio=MAC,bridge=BRIDGE\n ret = []\n hwaddr = ''\n parts = net.split(',')\n for part in parts:\n if part.startswith('bridge='):\n ret.append(part)\n elif part.startswith('hwaddr='):\n hwaddr = part[7:]\n ret.insert(0, 'virtio={}'.format(hwaddr))\n return ','.join(ret)\n\n\nclass AddDiscardOnTranslator(object):\n def volume_properties(self, new_volume):\n \"\"\"\n Add \"discard=on\" flag to properties if possible\n \"\"\"\n props = super().volume_properties(new_volume)\n\n if new_volume.storage and new_volume.storage.has_discard_support:\n values = props.split(',')\n name, rest = values[0], values[1:]\n rest = [\n i for i in rest if i not in ('discard=on', 'discard=off')]\n rest.append('discard=on')\n rest.sort()\n props = ','.join([name] + rest)\n\n return props\n\n\nclass DefaultConfigTranslator(\n AddDiscardOnTranslator,\n TurnLxcIntoQemuConfigTranslator,\n BaseConfigTranslator):\n pass\n\n\nclass VmMover(object):\n MOVE_INITIAL = 1 # new is down, preparing snapshottable copy\n MOVE_FINAL = 2 # old/new are down, finalize last changes\n\n def __init__(self, options):\n self.src_pve = options.source\n self.dst_pve = options.destination\n self.dst_node = options.nodeid\n self.dst_storage = options.storage\n self.vms_requested = options.vm\n\n self.opt_ignore_exists = options.ignore_exists\n self.opt_skip_disks = options.skip_disks\n self.opt_skip_start = (\n # Also don't start of there are no disks:\n options.skip_start or options.skip_disks)\n self.opt_wait_before_stop = options.wait_before_stop\n\n self.vms_to_move = None\n self.storages = None\n\n def prepare(self):\n log.debug('Sanity checks and preparation')\n self.prepare_vm_list()\n self.prepare_vm_config()\n self.prepare_storage_list()\n self.prepare_storage_prerequisites()\n\n def prepare_vm_list(self):\n log.debug('Checking VMs existence on source and destination')\n vms = []\n for vm_name in self.vms_requested:\n src_suffix = ''\n try:\n self.prepare_vm_check_source(vm_name)\n\n except ResumablePrepareError as err1:\n log.warning(err1)\n try:\n self.prepare_vm_check_destination(vm_name)\n except ResumablePrepareError as err2:\n log.warning(err2)\n resume_work = input('Continue [y/N]? ')\n if resume_work.strip().lower() != 'y':\n raise\n src_suffix = SUFFIX_CLONING\n else:\n raise # err1\n\n else:\n try:\n self.prepare_vm_check_destination(vm_name)\n except ResumablePrepareError as err1:\n log.warning(err1)\n log.warning(\n 'Destination is created but source still appears to '\n 'be running. Looks like you are resuming an initial '\n '(pre-shutdown) migration. Please ascertain that '\n 'the new VM data is not old')\n resume_work = input('Continue [y/N]? ')\n if resume_work.strip().lower() != 'y':\n raise\n\n vm = self.src_pve.get_vm(vm_name, suffix=src_suffix)\n vms.append(vm)\n\n self.vms_to_move = vms\n self.vms_requested = None\n\n def prepare_vm_check_source(self, vm_name):\n for suffix in (SUFFIX_CLONING, SUFFIX_MIGRATED):\n try:\n self.src_pve.get_vm(vm_name, suffix=suffix)\n except ProxmoxVm.DoesNotExist:\n pass\n else:\n vm_name_with_suffix = '{}{}'.format(vm_name, suffix)\n err = 'VM {!r} exists on source already'.format(\n vm_name_with_suffix)\n if suffix == SUFFIX_CLONING:\n raise ResumablePrepareError(err)\n raise PrepareError(err)\n\n def prepare_vm_check_destination(self, vm_name):\n for suffix in ('', SUFFIX_CREATING):\n try:\n self.dst_pve.get_vm(vm_name, suffix=suffix)\n except ProxmoxVm.DoesNotExist:\n pass\n else:\n vm_name_with_suffix = '{}{}'.format(vm_name, suffix)\n err = 'VM {!r} exists on destination already'.format(\n vm_name_with_suffix)\n if suffix == SUFFIX_CREATING:\n raise ResumablePrepareError(err)\n if not self.opt_ignore_exists:\n raise PrepareError(err)\n\n def prepare_vm_config(self):\n log.debug('Checking for problematic config in {} VMs to move'.format(\n len(self.vms_to_move)))\n\n for vm in self.vms_to_move:\n # Checks whether the internal config is okay.\n vm.check_config()\n # Check whether volume sizes can be determined. We may need\n # both host and guest size.\n self.prepare_vm_config_volumes(vm)\n\n def prepare_vm_config_volumes(self, vm):\n for volume in vm.get_volumes().values():\n if volume.get_size('host') is None:\n # If we have no size of the host, then we most\n # definately won't have any size of the guest\n pass\n\n elif volume.get_size('guest') is None:\n # This could become a problem: if we don't know the disk\n # size, we can probably not convert from QCOW2 to ZFS\n # safely.\n raise PrepareError(\n 'unknown guest disk-size of volume {} on VM {}'.format(\n volume, vm))\n\n def prepare_storage_list(self):\n storages = set()\n for vm in self.vms_to_move:\n for volume in vm.get_volumes().values():\n if volume.storage:\n storages.add(volume.storage)\n\n # Check destination storage against access via the sources.\n for src_storage in storages:\n # We'll be doing: \" ssh src_storage 'ssh dst_storage' \"\n # Set tty=True so the user gets any host key questions.\n self.dst_storage.ssh_command(\n ['/bin/true'], tty=True, via=src_storage)\n\n storages.add(self.dst_storage)\n\n log.debug('Found {} relevant storages: {}'.format(\n len(storages), ', '.join(str(i) for i in storages)))\n self.storages = storages\n\n def prepare_storage_prerequisites(self):\n log.debug('Checking storage prerequisites')\n for storage in self.storages:\n storage.check_prerequisites()\n\n def run(self, dry_run):\n translator = DefaultConfigTranslator()\n for vm in self.vms_to_move:\n self.move_vm(vm, translator, dry_run)\n\n def move_vm(self, src_vm, translator, dry_run):\n # Config and notes:\n # {'bootdisk': 'virtio0',\n # 'cores': 8,\n # 'digest': '8f8bb34b5544282c71fefc7ed0dbbaa57d148505',\n # # v-- we \"eject\" the cdrom, replacing it with \"none\"\n # 'ide2': 'san06:iso/debian-8.0.0-amd64-netinst.iso,media=cdrom',\n # 'memory': 4096,\n # 'name': 'vm-to-move.example.com',\n # # v-- vmbr138 will work fine as long as cluster is in same location\n # 'net0': 'virtio=00:C0:FF:EE:BE:EF,bridge=vmbr138',\n # # v-- is constant (apparently?)\n # 'ostype': 'l26',\n # 'smbios1': 'uuid=76223226-e334-488c-ae8a-a174690e2803,'\n # 'manufacturer=example,product=vm-to-move',\n # 'sockets': 1,\n # # v-- disk moving is covered in ProxmoxVolume\n # 'virtio0': 'san8:520/vm-520-disk-1.qcow2,format=qcow2,iops_rd=5000,'\n # 'iops_wr=400,size=50G'}\n #\n # Steps:\n # - translate old config to new config\n # - create new config on nodeX on dest (no disks, \"CREATING\" name)\n # - stop old host, rename to \"CLONING\"\n # - move all disks, one by one\n # - rename dest to real name, rename source to \"MIGRATED\", add comment\n log.info('Attempt moving {} => {} (node {!r}): {}'.format(\n self.src_pve, self.dst_pve, self.dst_node, src_vm.name))\n\n log.info('- source VM {}'.format(src_vm))\n for volume_key, volume in src_vm.get_volumes().items():\n # TODO: Check whether volume is accessible? Must be combined\n # with the is_removable() and skip_disks options below.\n log.info('- storage {!r}: {} (host={}, guest={})'.format(\n volume_key, volume, volume.get_human_size('host'),\n volume.get_human_size('guest')))\n # NOTE: Test volume moving first. We don't want\n # NotImplementedError at the end of a long upload.\n self._move_vm_volume(src_vm, None, volume_key, volume, None)\n\n if not dry_run:\n dst_vm = self._start_moving_vm(src_vm, translator)\n self._move_vm_volumes(src_vm, dst_vm, move_mode=self.MOVE_FINAL)\n self._finish_moving_vm(src_vm, dst_vm)\n\n def _start_moving_vm(self, src_vm, translator):\n if src_vm.name == src_vm.basename:\n try:\n # Check for existing target VM. Happens if we're resuming.\n dst_vm = self.dst_pve.get_vm(\n src_vm.basename, suffix=SUFFIX_CREATING)\n except ProxmoxVm.DoesNotExist:\n # Don't create VM in exception handler. Errors would\n # then appear like the DoesNotExist is the root cause.\n dst_vm = None\n if not dst_vm:\n # Pristine VMs: start, by translating config and\n # creating a new VM.\n dst_config = translator.config(src_vm.get_config())\n dst_vm = self.dst_pve.create_vm(\n dst_config, nodeid=self.dst_node,\n poolid=src_vm.poolid)\n dst_vm.set_translator(translator)\n\n # Do initial attempt at moving VM volumes before bringing the\n # source down. This is possible if we have snapshot support.\n self._move_vm_volumes(src_vm, dst_vm, move_mode=self.MOVE_INITIAL)\n # It is important that we reset the volumes count for the\n # MOVE_FINAL run.\n dst_vm.volumes = 0\n\n # If you want to do stuff just before shutting down the old\n # running instance, now is the time:\n if self.opt_wait_before_stop:\n log.info(\n 'Pausing because of --wait-before-stop. Please do the '\n 'needful before resuming with Y')\n resume_work = ''\n while resume_work != 'y':\n resume_work = input('Resume [y]? ').strip().lower()\n\n # Stop old VM and rename.\n if src_vm.type != 'qemu':\n raise NotImplementedError('No LXC move support')\n src_vm.ensure_stopped()\n src_vm.rename(src_vm.basename + SUFFIX_CLONING)\n else:\n # Okay. When name!=basename we've already started.\n dst_vm = self.dst_pve.get_vm(\n src_vm.basename, suffix=SUFFIX_CREATING)\n dst_vm.set_translator(translator)\n assert src_vm.poolid == dst_vm.poolid, (\n src_vm.poolid, dst_vm.poolid)\n\n assert src_vm.name == src_vm.basename + SUFFIX_CLONING, src_vm\n assert dst_vm.name == dst_vm.basename + SUFFIX_CREATING, dst_vm\n return dst_vm\n\n def _move_vm_volumes(self, src_vm, dst_vm, move_mode):\n # LXC(FIXME): Here we should migrate/copy the ROOTFS data\n # for LXC too. Example old config: {\n # 'rootfs': 'pool0-pve2-ssd:subvol-105-disk-1,size=20G',\n # 'searchdomain': '91.xxx.xxx.xx', 'cpuunits': 1024,\n # 'cpulimit': '2', 'swap': 0, 'hostname': 'osso-xxxxxxxxxxxxxx',\n # 'net1': 'bridge=vmbr535,hwaddr=32:xx:xx:xx:xx:xx,'\n # 'ip=10.xx.xx.xx/24,name=eth1,type=veth',\n # 'memory': 2048, 'net0': 'bridge=vmbr0,gw=10.xxx.x.x,'\n # 'hwaddr=66:xx:xx:xx:xx:xx,ip=10.xxx.x.xx/24,'\n # 'name=eth0,type=veth',\n # 'ostype': 'ubuntu', 'nameserver': '91.xxx.xxx.xx 8.8.8.8',\n # 'arch': 'amd64'}\n # Example new config: {\n # 'rootfs': 'mc11-6-local-ssd:subvol-160-disk-1,size=20G',\n # 'searchdomain': '91.xxx.xxx.xx', 'cpuunits': 1024,\n # 'ostemplate':\n # # Note that 'vztmpl' is hardcoded in PVE/Storage/Plugin.pm\n # 'san06:vztmpl/ubuntu-14.04-standard_14.04-1_amd64.tar.gz',\n # 'arch': 'amd64', 'cpulimit': '2',\n # 'hostname': 'osso-swift-mgmt-tcn--CREATING',\n # 'nameserver': '91.xxx.xxx.xx 8.8.8.8', 'memory': 2048,\n # 'net0': 'bridge=vmbr0,gw=10.xxx.x.x,hwaddr=66:xx:xx:xx:xx:xx,'\n # 'ip=10.xxx.x.xx/24,name=eth0,type=veth', 'ostype': 'ubuntu',\n # 'net1': 'bridge=vmbr535,hwaddr=32:xx:xx:xx:xx:xx,'\n # 'ip=10.xx.xx.xx/24,name=eth1,type=veth', 'swap': 0}\n for volume_key, volume in src_vm.get_volumes().items():\n self._move_vm_volume(src_vm, dst_vm, volume_key, volume, move_mode)\n\n def _move_vm_volume(self, src_vm, dst_vm, volume_key, volume, move_mode):\n if volume.is_removable():\n dst_storage = None # eject\n elif self.opt_skip_disks:\n log.debug('Not copying volume {} as requested'.format(\n volume_key))\n dst_storage = None\n else:\n dst_storage = self.dst_storage\n\n if dst_vm:\n # Do it.\n dst_vm.create_volume(\n volume_key, volume, storage=dst_storage,\n requires_snapshot_support=(move_mode == self.MOVE_INITIAL))\n elif dst_storage:\n # Test only.\n assert move_mode is None, move_mode\n volume.test_clone(dst_storage)\n\n def _finish_moving_vm(self, src_vm, dst_vm):\n # Done? Rename both.\n dst_vm.rename(dst_vm.basename)\n src_vm.rename(src_vm.basename + SUFFIX_MIGRATED)\n src_vm.configure_autoboot(False)\n src_vm.add_comment('{} UTC: Migrated to {}'.format(\n datetime.utcnow(), dst_vm))\n\n # Start VM.\n if self.opt_skip_start:\n log.debug('Not starting VM as requested')\n else:\n dst_vm.ensure_started()\n\n log.info('Completed moving {} => {} (node {!r}): {}'.format(\n self.src_pve, self.dst_pve, self.dst_node, dst_vm.name))\n\n\nclass CommandLoader(object):\n def __init__(self):\n self.argparse()\n self.load_config()\n self.process()\n\n def argparse(self):\n self.argparse_create()\n self.argparse_add_optional()\n self.argparse_add_debug()\n self.argparse_add_other()\n self.argparse_add_arguments()\n self.argparse_run()\n\n def argparse_create(self):\n # CURRENT:\n # - move SRCCLUSTER DSTCLUSTER DSTNODE DSTSTORAGE VM..\n # FUTURE:\n # - ssh CLUSTER:STORAGE\n # - nodes CLUSTER (show nodes (and storage?))\n # - storages CLUSTER:NODE (show nodes)\n # - move SRC DST DSTNODE VM..\n # - dumpconfig (based with the storage devices..)\n self.parser = ArgumentParser14191(\n add_help=False, # we'll be adding it to a separate group\n description=(\n 'Migrate VMs from one Proxmox cluster to another.'),\n epilog=(\n 'Cluster aliases and storage locations should be defined '\n 'in ~/.proxmoverc (or see -c option). See the example '\n 'proxmoverc.sample. It requires [pve:CLUSTER_ALIAS] sections '\n 'for the proxmox \"api\" URL and '\n '[storage:CLUSTER_ALIAS:STORAGE_NAME] sections with \"ssh\", '\n '\"path\" and \"temp\" settings.'))\n\n def argparse_add_optional(self):\n grp = self.parser.add_argument_group('optional arguments')\n grp.add_argument(\n '-c', '--config', action='store', metavar='FILENAME',\n default='~/.proxmoverc', help=(\n 'use alternate configuration inifile'))\n grp.add_argument(\n '-n', '--dry-run', action='store_true', help=(\n 'stop before doing any writes'))\n grp.add_argument(\n '--bwlimit', action='store', metavar='MBPS', type=int, help=(\n 'limit bandwidth in Mbit/s'))\n grp.add_argument(\n '--no-verify-ssl', action='store_true', help=(\n 'skip ssl verification on the api hosts'))\n grp.add_argument(\n '--skip-disks', action='store_true', help=(\n 'do the move, but skip copying of the disks; '\n 'implies --skip-start'))\n grp.add_argument(\n '--skip-start', action='store_true', help=(\n 'do the move, but do not start the new instance'))\n grp.add_argument(\n '--wait-before-stop', action='store_true', help=(\n 'prepare the move, but ask for user confirmation before '\n 'shutting down the old instance (useful if you have to move '\n 'networks/IPs)'))\n grp.add_argument(\n '--ssh-ciphers', action='store', metavar='CIPHERS',\n default='aes128-gcm@openssh.com,aes128-cbc', help=(\n 'comma separated list of ssh -c ciphers to prefer, '\n '(aes128-gcm@openssh.com is supposed to be fast if you have '\n 'aes on your cpu); set to \"-\" to use ssh defaults'))\n\n def argparse_add_debug(self):\n grp = self.parser.add_argument_group('debug arguments')\n grp.add_argument(\n '--debug', action='store_true', help=(\n 'enables extra debug logging'))\n grp.add_argument(\n # Ignores the fact that a VM exists already. Can be used to\n # test/do moves between the same cluster\n '--ignore-exists', action='store_true', help=(\n 'continue when target VM already exists; allows moving to '\n 'same cluster'))\n\n def argparse_add_other(self):\n grp = self.parser.add_argument_group('other actions')\n grp.add_argument(\n '-h', '--help', action='help', help=(\n 'show this help message and exit'))\n grp.add_argument(\n '--version', action='version', version=(\n 'proxmove {}'.format(__version__)))\n\n def argparse_add_arguments(self):\n self.parser.add_argument(\n 'source', action='store', help=(\n 'alias of source cluster'))\n self.parser.add_argument(\n 'destination', action='store', help=(\n 'alias of destination cluster'))\n self.parser.add_argument(\n 'nodeid', action='store', help=(\n 'node on destination cluster'))\n self.parser.add_argument(\n 'storage', action='store', help=(\n 'storage on destination node'))\n self.parser.add_argument(\n 'vm', action='store', nargs='+', help=(\n 'one or more VMs (guests) to move'))\n\n def argparse_run(self):\n self.args = self.parser.parse_args()\n\n # Set debug mode as soon as possible.\n if self.args.debug:\n log.setLevel('DEBUG')\n\n def load_config(self):\n log.debug('Parsing config file: {}'.format(self.args.config))\n try:\n self.clusters = ProxmoxClusters.from_filename(\n os.path.expanduser(self.args.config))\n except ValueError as e:\n self.parser.error(e.args[0])\n\n for cluster in self.clusters.values():\n if self.args.no_verify_ssl:\n cluster.no_verify_ssl()\n if self.args.bwlimit:\n cluster.set_bandwidth_limit(self.args.bwlimit)\n if self.args.ssh_ciphers:\n cluster.set_ssh_ciphers(self.args.ssh_ciphers)\n\n def process(self):\n self.process_clusters()\n self.process_nodes()\n\n def process_clusters(self):\n if (not self.args.ignore_exists and\n self.args.source == self.args.destination):\n self.parser.error(\n 'source and destination arguments are the same '\n '(see --ignore-exists if you want to migrate to the '\n 'same cluster)')\n\n self.args.source = self.process_cluster(self.args.source)\n self.args.destination = self.process_cluster(self.args.destination)\n\n def process_cluster(self, cluster_name):\n try:\n cluster = self.clusters[cluster_name]\n except KeyError:\n found_clusters = ', '.join(sorted(self.clusters.keys()))\n self.parser.error(\n 'cluster {!r} is not configured in config '\n '(expected one of: {})'.format(cluster_name, found_clusters))\n\n try:\n cluster.ping()\n except Exception as e:\n self.parser.error(\n 'cluster {!r} is unavailable: {}: {}'.format(\n cluster.name, type(e).__name__, e.args[0]))\n\n return cluster\n\n def process_nodes(self):\n if self.args.nodeid not in self.args.destination.get_nodes():\n self.parser.error(\n 'destination node {!r} not online (got: {})'.format(\n self.args.nodeid,\n repr(self.args.destination.get_nodes())[1:-1]))\n\n try:\n self.args.storage = self.args.destination.get_storage(\n self.args.nodeid, self.args.storage)\n except ValueError:\n self.parser.error(\n 'destination storage {!r} not found in node {!r}'.format(\n self.args.storage, self.args.nodeid))\n\n def get_options(self):\n return self.args\n\n\ndef main():\n # Set logging mode.\n logconfig = {\n 'version': 1,\n 'formatters': {\n 'full': {\n 'format': '%(asctime)-15s: %(levelname)s: %(message)s'}},\n 'handlers': {\n 'console': {\n 'class': 'logging.StreamHandler', 'formatter': 'full'}},\n 'loggers': {\n '': {'handlers': ['console'], 'level': 'WARNING'},\n 'proxmove': {'handlers': ['console'], 'level': 'INFO',\n 'propagate': False}},\n }\n logging.config.dictConfig(logconfig)\n\n # Load up config.\n options = CommandLoader().get_options()\n\n # Create mover and start the move.\n vmmover = VmMover(options)\n try:\n vmmover.prepare()\n except PrepareError as e:\n log.error('%s', e)\n print('aborting', file=sys.stderr)\n sys.exit(1)\n else:\n vmmover.run(options.dry_run)\n\n\nif __name__ == '__main__':\n main()\n" }, { "path": "proxmoverc.sample", "content": "; PVE clusters should to be specified as follows.\n;\n; [pve:CLUSTER_ALIAS]\n; ; The API must be reachable over HTTPS. The USER needs PVEVMAdmin\n; ; powers on '/' in your PVE cluster.\n; api=https://USER@pve:PASSWORD@HOST:PORT\n;\n; The PVE clusters have one or more storage devices that must be\n; reachable over SSH. When moving data between storage devices, the\n; data is first copied to the `temp` location on the destination storage\n; and then converted to the correct format.\n;\n; Every cluster needs at least one storage device. Some may be shared\n; across nodes, others exist on a single node only.\n;\n; If the path is a plain path, file-based images are used. If the path\n; starts with \"zfs:\", it is considered to be a zfs filesystem path.\n;\n; [storage:CLUSTER_ALIAS:DISK_NAME[@NODE_NAME]]\n; ssh=root@HOST\n; path=/path/to/somewhere ; for plain disk (DISK_NAME:VMID/...), or\n; path=zfs:ZFS_FS_NAME ; for ZFS (DISK_NAME:vm-VMID-disk-1)\n\n; Example cluster named \"banana-cluster\" with 3 storage devices, one\n; shared, and two which exist on a single node only.\n[pve:banana-cluster]\napi=https://user@pve:PASSWORD@banana-cluster.com:443\n\n [storage:banana-cluster:sharedsan] ; \"sharedsan\" is available on all nodes\n ssh=root@san.banana-cluster.com\n path=/pool0/san/images\n temp=/pool0/san/private\n\n [storage:banana-cluster:local@node1] ; local disk on node1 only\n ssh=root@node1.banana-cluster.com\n path=/srv/images\n temp=/srv/temp\n\n [storage:banana-cluster:local@node2] ; other local disk on node2 only\n ssh=root@node2.banana-cluster.com\n path=/srv/images\n temp=/srv/temp\n\n; Example cluster named \"the-new-cluster\" with 2 storage devices; both\n; storage devices exist on the respective nodes only.\n[pve:the-new-cluster]\napi=https://user@pve:PASSWORD@the-new-cluster.com:443\n\n [storage:the-new-cluster:node1-ssd@node1]\n ssh=root@node1.the-new-cluster.com\n path=zfs:node1-ssd/images\n temp=/node1-ssd/temp\n\n [storage:the-new-cluster:node2-ssd@node2]\n ssh=root@node2.the-new-cluster.com\n path=zfs:node2-ssd/images\n temp=/node2-ssd/temp\n\n; vim: set syn=cfg:\n" }, { "path": "requirements.txt", "content": "# mkvirtualenv proxmove --system-site-packages --python=`which python3`\nproxmoxer>=0.2.4\n" }, { "path": "setup.py", "content": "#!/usr/bin/env python3\nfrom distutils.core import setup\n\nwith open('proxmove') as fp:\n for line in fp:\n if line.startswith('__version__'):\n version = line.split(\"'\")[1]\n break\n\nlong_description = []\nfor filename in ('README.rst', 'CHANGES.rst', 'TODO.rst'):\n with open(filename) as fp:\n long_description.append(fp.read())\nlong_description = '\\n\\n'.join(long_description)\n\n\nsetup(\n name='proxmove',\n version=version,\n scripts=['proxmove'],\n data_files=[\n ('share/doc/proxmove', [\n 'README.rst', 'CHANGES.rst', 'TODO.rst']),\n ('share/proxmove', [\n 'proxmoverc.sample'])],\n description=(\n 'Migrate virtual machines between different Proxmox VM clusters'),\n long_description=long_description,\n author='Walter Doekes, OSSO B.V.',\n author_email='wjdoekes+proxmove@osso.nl',\n url='https://github.com/ossobv/proxmove',\n license='GPLv3+',\n platforms=['linux'],\n classifiers=[\n 'Development Status :: 4 - Beta',\n 'Intended Audience :: System Administrators',\n ('License :: OSI Approved :: GNU General Public License v3 '\n 'or later (GPLv3+)'),\n 'Operating System :: POSIX :: Linux',\n 'Programming Language :: Python :: 3',\n 'Topic :: System :: Clustering',\n ],\n install_requires=[\n 'proxmoxer>=0.2.4',\n 'requests>=2.9.1',\n ],\n)\n\n# vim: set ts=8 sw=4 sts=4 et ai tw=79:\n" } ]