[ { "path": ".gitignore", "content": ".bundle\n.rvmrc\n*~\n*.pyc\n*.egg\n*.egg-info\n.eggs\n*.pid\n*.pid.lock\n*.gem\n*.rpm\n*.deb\ndocker/*/generated\ndocker/config.yml\n/doc/.site\n/doc/sdk/python/arvados\n/doc/sdk/python/arvados.html\n/doc/sdk/python/index.html\n/doc/sdk/python/search.js\n/doc/sdk/R/arvados\n/doc/sdk/java-v2/javadoc\n*.class\n/sdk/cli/binstubs/\n/sdk/ruby/binstubs/\n/services/api/binstubs/\n/services/login-sync/binstubs/\n/services/api/config/arvados-clients.yml\n/contrib/arvados-bootstrap/build/\n/contrib/arvados-bootstrap/dist/\n/sdk/cwl/build/\n/sdk/cwl/dist/\n/sdk/python/build/\n/sdk/python/dist/\n/services/dockercleaner/build/\n/services/dockercleaner/dist/\n/services/fuse/build/\n/services/fuse/dist/\n/tools/cluster-activity/build/\n/tools/cluster-activity/dist/\n/tools/crunchstat-summary/build/\n/tools/crunchstat-summary/dist/\n/tools/python-metapackage/build/\n/tools/python-metapackage/dist/\n/tools/user-activity/build/\n/tools/user-activity/dist/\n*#*\nvendor/\ntmp/\n.DS_Store/\n.vscode\n.Rproj.user\n*.bak\n*.log\narvados-snakeoil-ca.pem\n.vagrant\n/packages/\n.eslintcache\n" }, { "path": ".licenseignore", "content": "*agpl-3.0.html\n*agpl-3.0.txt\napache-2.0.txt\nAUTHORS\n*/bootstrap.css\n*/bootstrap.js\n*bootstrap-theme.css\n*by-sa-3.0.html\n*by-sa-3.0.txt\n*COPYING\ndoc/fonts/*\ndoc/_includes/_config_default_yml.liquid\ndoc/_includes/_terraform_*_tfvars.liquid\ndoc/user/cwl/federated/*\ndoc/_includes/_federated_cwl.liquid\n*/docker_image\ndocker/jobs/apt.arvados.org*.list\ndocker/jobs/1078ECD7.key\n*/en.bootstrap.yml\n*font-awesome.css\n*.gif\n.gitignore\n*/.gitignore\n*/.gitkeep\n*/.gitstub\n*.gz\n*.gz.report\n*.ico\n*.jpg\n*.svg\n*.odg\n*.json\n*LICENSE*.html\n.licenseignore\n*LICENSE*.txt\n*.lock\n*.log\n*.map\n*.min.css\n*.min.js\n*.png\n*/proc_stat\n*/pytest.ini\n*/README\n*/robots.txt\n*/runit-docker/*\n*/sb-admin.css.scss\n*/script/rails\nsdk/cwl/tests/input/blorp.txt\nsdk/cwl/tests/tool/blub.txt\nsdk/cwl/tests/19109-upload-secondary/*\nsdk/cwl/tests/federation/data/*\nsdk/cwl/tests/fake-keep-mount/fake_collection_dir/.arvados#collection\nsdk/cwl/tests/container_request_9tee4-xvhdp-kk0ja1cl8b2kr1y-arv-mount.txt\nsdk/cwl/tests/container_request_9tee4-xvhdp-kk0ja1cl8b2kr1y-crunchstat.txt\nsdk/go/manifest/testdata/*_manifest\nsdk/java/.classpath\nsdk/java/pom.xml\nsdk/java/.project\nsdk/java/.settings/org.eclipse.jdt.core.prefs\nsdk/java/src/main/resources/log4j.properties\nsdk/pam/examples/shellinabox\nsdk/pam/pam-configs/arvados\nsdk/python/tests/data/*\nservices/api/config/unbound.template\nservices/api/config/config.default.yml\nservices/arv-web/sample-cgi-app/public/.htaccess\nservices/arv-web/sample-cgi-app/public/index.cgi\nservices/keepproxy/pkg-extras/etc/default/keepproxy\n*.tar\ntools/crunchstat-summary/tests/crunchstat_error_messages.txt\ntools/crunchstat-summary/crunchstat_summary/synchronizer.js\ntools/cluster-activity/tests/*.html\ntools/cluster-activity/tests/*.csv\ncontrib/R-sdk/DESCRIPTION\ncontrib/R-sdk/NAMESPACE\ncontrib/R-sdk/.Rbuildignore\ncontrib/R-sdk/ArvadosR.Rproj\n*.Rd\nlib/dispatchcloud/test/sshkey_*\n*.asc\ncontrib/java-sdk-v2/build.gradle\ncontrib/java-sdk-v2/settings.gradle\ncontrib/java-sdk-v2/src/test/resources/*\nsdk/cwl/tests/wf/feddemo\ngo.mod\ngo.sum\ndoc/install/*.xlsx\nsdk/cwl/tests/wf/hello.txt\nsdk/cwl/tests/wf/indir1/hello2.txt\nsdk/cwl/tests/chipseq/data/Genomes/*\nCITATION.cff\nSECURITY.md\nlib/crunchstat/testdata/*\nlib/controller/localdb/testdata/*.pub\nsdk/ruby-google-api-client/*\nservices/api/bin/rails\nservices/api/bin/rake\nservices/api/bin/setup\nservices/api/bin/yarn\nservices/api/storage.yml\nservices/api/test.rb.example\nservices/api/config/boot.rb\nservices/api/config/environment.rb\nservices/api/config/initializers/application_controller_renderer.rb\nservices/api/config/initializers/assets.rb\nservices/api/config/initializers/backtrace_silencers.rb\nservices/api/config/initializers/content_security_policy.rb\nservices/api/config/initializers/cookies_serializer.rb\nservices/api/config/initializers/filter_parameter_logging.rb\nservices/api/config/initializers/mime_types.rb\nservices/api/config/initializers/new_framework_defaults_*.rb\nservices/api/config/initializers/permissions_policy.rb\nservices/api/config/initializers/wrap_parameters.rb\nservices/api/config/locales/en.yml\nservices/api/config.ru\nservices/workbench2/*.d.ts\nservices/workbench2/*.css\nservices/workbench2/*.scss\nservices/workbench2/README.md\nservices/workbench2/public/*\nservices/workbench2/.yarnrc\nservices/workbench2/.npmrc\nservices/workbench2/src/lib/cwl-svg/*\nservices/workbench2/tools/arvados_config.yml\nservices/workbench2/cypress/fixtures/files/5mb.bin\nservices/workbench2/cypress/fixtures/files/15mb.bin\nservices/workbench2/cypress/fixtures/files/cat.png\nservices/workbench2/cypress/fixtures/files/banner.html\nservices/workbench2/cypress/fixtures/files/tooltips.txt\nservices/workbench2/cypress/fixtures/webdav-propfind-outputs.xml\nservices/workbench2/.yarn/releases/*\nservices/workbench2/package.json\nservices/workbench2/yarn.lock\n" }, { "path": "AUTHORS", "content": "# Names should be added to this file with this pattern:\n#\n# For individuals:\n# Name \n#\n# For organizations:\n# Organization \n#\n# See python fnmatch module documentation for more information.\n\nCuroverse, Inc. <*@curoverse.com>\nAdam Savitzky \nColin Nolan \nDavid \nGuillermo Carrasco \nJoshua Randall \nPresident and Fellows of Harvard College <*@harvard.edu>\nThomas Mooney \nChen Chen \nVeritas Genetics, Inc. <*@veritasgenetics.com>\nCurii Corporation <*@curii.com>\nDante Tsang \nCodex Genetics Ltd \nBruno P. Kinoshita \nGeorge Chlipala \n" }, { "path": "CITATION.cff", "content": "cff-version: 1.2.0\nmessage: \"If you use this software, please cite it as below.\"\nauthors:\n- name: \"The Arvados Authors\"\n- family-names: \"Amstutz\"\n given-names: \"Peter\"\n orcid: \"https://orcid.org/0000-0003-3566-7705\"\n- family-names: \"Bértoli\"\n given-names: \"Javier\"\n family-names: \"César\"\n given-names: \"Nico\"\n- family-names: \"Clegg\"\n given-names: \"Tom\"\n orcid: \"https://orcid.org/0000-0001-6751-2930\"\n- family-names: \"Di Pentima\"\n given-names: \"Lucas\"\n orcid: \"https://orcid.org/0000-0002-2807-6854\"\n- family-names: \"Kutyła\"\n given-names: \"Daniel\"\n- family-names: \"Li\"\n given-names: \"Jiayong\"\n- family-names: \"Smith\"\n given-names: \"Stephen\"\n- family-names: \"Vandewege\"\n given-names: \"Ward\"\n orcid: \"https://orcid.org/0000-0002-2527-6949\"\n- family-names: \"Wait Zaranek\"\n given-names: \"Alexander\"\n orcid: \"https://orcid.org/0000-0002-0415-9655\"\n- family-names: \"Wait Zaranek\"\n given-names: \"Sarah\"\n orcid: \"https://orcid.org/0000-0003-4716-9121\"\ntitle: \"Arvados\"\nabstract: \"Arvados is an open source platform for managing, processing, and sharing genomic and other large scientific and biomedical data.\"\ntype: software\nurl: \"https://github.com/arvados/arvados/\"\ndoi: 10.5281/zenodo.6382942\n" }, { "path": "CODE_OF_CONDUCT.md", "content": "Arvados Code of Conduct\n=======================\n\nThe Arvados Project is dedicated to providing a harassment-free experience for\neveryone. We do not tolerate harassment of participants in any form.\n\nThis code of conduct applies to all Arvados Project spaces both online and off:\nGitter chat, Redmine issues, wiki, mailing lists, forums, video chats, and any other\nArvados spaces. Anyone who violates this code of conduct may be sanctioned or\nexpelled from these spaces at the discretion of the Arvados Team.\n\nSome Arvados Project spaces may have additional rules in place, which will be\nmade clearly available to participants. Participants are responsible for\nknowing and abiding by these rules.\n\nHarassment includes, but is not limited to:\n\n - Offensive comments related to gender, gender identity and expression, sexual\norientation, disability, mental illness, neuro(a)typicality, physical\nappearance, body size, age, race, or religion.\n - Unwelcome comments regarding a person’s lifestyle choices and practices,\nincluding those related to food, health, parenting, drugs, and employment.\n - Deliberate misgendering or use of [dead](https://www.quora.com/What-is-deadnaming/answer/Nancy-C-Walker)\nor rejected names.\n - Gratuitous or off-topic sexual images or behaviour in spaces where they’re not\nappropriate.\n - Physical contact and simulated physical contact (eg, textual descriptions like\n“\\*hug\\*” or “\\*backrub\\*”) without consent or after a request to stop.\n - Threats of violence.\n - Incitement of violence towards any individual, including encouraging a person\nto commit suicide or to engage in self-harm.\n - Deliberate intimidation.\n - Stalking or following.\n - Harassing photography or recording, including logging online activity for\nharassment purposes.\n - Sustained disruption of discussion.\n - Unwelcome sexual attention.\n - Pattern of inappropriate social contact, such as requesting/assuming\ninappropriate levels of intimacy with others\n - Continued one-on-one communication after requests to cease.\n - Deliberate “outing” of any aspect of a person’s identity without their consent\nexcept as necessary to protect vulnerable people from intentional abuse.\n - Publication of non-harassing private communication.\n\nThe Arvados Project prioritizes marginalized people’s safety over privileged\npeople’s comfort. The Arvados Leadership Team will not act on complaints regarding:\n\n - ‘Reverse’ -isms, including ‘reverse racism,’ ‘reverse sexism,’ and ‘cisphobia’\n - Reasonable communication of boundaries, such as “leave me alone,” “go away,” or\n“I’m not discussing this with you.”\n - Communicating in a [tone](http://geekfeminism.wikia.com/wiki/Tone_argument)\nyou don’t find congenial\n\nReporting\n---------\n\nIf you are being harassed by a member of the Arvados Project, notice that someone\nelse is being harassed, or have any other concerns, please contact the Arvados\nProject Team at contact@arvados.org. If person who is harassing\nyou is on the team, they will recuse themselves from handling your incident. We\nwill respond as promptly as we can.\n\nThis code of conduct applies to Arvados Project spaces, but if you are being\nharassed by a member of Arvados Project outside our spaces, we still want to\nknow about it. We will take all good-faith reports of harassment by Arvados Project\nmembers, especially the Arvados Team, seriously. This includes harassment\noutside our spaces and harassment that took place at any point in time. The\nabuse team reserves the right to exclude people from the Arvados Project based on\ntheir past behavior, including behavior outside Arvados Project spaces and\nbehavior towards people who are not in the Arvados Project.\n\nIn order to protect volunteers from abuse and burnout, we reserve the right to\nreject any report we believe to have been made in bad faith. Reports intended\nto silence legitimate criticism may be deleted without response.\n\nWe will respect confidentiality requests for the purpose of protecting victims\nof abuse. At our discretion, we may publicly name a person about whom we’ve\nreceived harassment complaints, or privately warn third parties about them, if\nwe believe that doing so will increase the safety of Arvados Project members or\nthe general public. We will not name harassment victims without their\naffirmative consent.\n\nConsequences\n------------\n\nParticipants asked to stop any harassing behavior are expected to comply\nimmediately.\n\nIf a participant engages in harassing behavior, the Arvados Team may\ntake any action they deem appropriate, up to and including expulsion from all\nArvados Project spaces and identification of the participant as a harasser to other\nArvados Project members or the general public.\n\nThis anti-harassment policy is based on the [example policy from the Geek\nFeminism wiki](http://geekfeminism.wikia.com/wiki/Community_anti-harassment/Policy),\ncreated by the Geek Feminism community.\n" }, { "path": "CONTRIBUTING.md", "content": "[comment]: # (Copyright © The Arvados Authors. All rights reserved.)\n[comment]: # ()\n[comment]: # (SPDX-License-Identifier: CC-BY-SA-3.0)\n\n# Contributing to Arvados\n\nArvados is free software, which means it is free for all to use, learn\nfrom, and improve. We encourage contributions from the community that\nimprove Arvados for everyone. Some examples of contributions are bug\nreports, bug fixes, new features, and scripts or documentation that help\nwith using, administering, or installing Arvados. We also love to\nhear about Arvados success stories.\n\n## Reporting Issues\n\nArvados uses [GitHub Issues](https://github.com/arvados/arvados/issues). You can file issues against any Arvados component there. Even if you're not sure which component causes the issue, you can still file problem reports and we'll work with you to address them.\n\n## Contributing Code\n\nThe preferred method for making contributions is through GitHub pull requests. The rest of this guide helps orient you with the code and discusses requirements for all contributions, from the smallest typo fix to entire new components.\n\nIf you're interested in developing a large new feature for Arvados, please file an issue to discuss it with us first. We can give you guidance on how to best organize the work before you start it.\n\n## Setting Up Your Development Environment\n\nThe [Arvados source code is hosted on GitHub](https://github.com/arvados/arvados). Once you clone it, you'll find guides for specific topics under the `doc/development` directory. You'll probably want to [install a development environment](doc/development/Prerequisites.md) and [learn how to run tests](doc/development/RunningTests.md). There are also some component-specific guides.\n\n### Setting Up Git\n\nWe provide Git configuration and hooks to help you follow project conventions.\n\n`doc/development/git.conf` includes a block of Git configuration settings. You can set it up for your checkout by running `git config edit --local`: insert the contents of `doc/development/git.conf`, edit them following the comments, then save and exit.\n\nInstall our `prepare-commit-msg` hook:\n\n```sh\n$ install -b -m 755 doc/development/prepare-commit-msg.sh .git/hooks/prepare-commit-msg\n```\n\n## Prepare a Development Branch\n\nIf you haven't before, fork the Arvados repository using the GitHub \"Fork\" button. If you have, make sure your fork's `main` branch is up-to-date with Arvados'.\n\nThen start a new branch for your development named like `1234-your-work`. The number at the start should match the GitHub issue this request is associated with. Then briefly describe the main change your branch makes.\n\n### Coding Standards\n\nPlease familiarize yourself with our [coding standards](doc/development/CodingStandards.md) for the component(s) you're working on and follow them in your work.\n\n### Sign Off Your Commits\n\nContributions must be signed off. The sign-off is a simple line at the end of each commit message which certifies that you wrote it or otherwise have the right to contribute it under the license listed in the file(s) modified. Make sure each commit message contains the following line with your real name and email (sorry, no pseudonymous or anonymous contributions):\n\n Arvados-DCO-1.1-Signed-off-by: Alex Doe \n\nWhen you add this, you certify the below (from ):\n\n> Developer Certificate of Origin \n> Version 1.1\n>\n> Copyright (C) 2004, 2006 The Linux Foundation and its contributors.\n>\n> Everyone is permitted to copy and distribute verbatim copies of this\n> license document, but changing it is not allowed.\n>\n>\n> Developer's Certificate of Origin 1.1\n>\n> By making a contribution to this project, I certify that:\n>\n> (a) The contribution was created in whole or in part by me and I\n> have the right to submit it under the open source license\n> indicated in the file; or\n>\n> (b) The contribution is based upon previous work that, to the best\n> of my knowledge, is covered under an appropriate open source\n> license and I have the right under that license to submit that\n> work with modifications, whether created in whole or in part\n> by me, under the same open source license (unless I am\n> permitted to submit under a different license), as indicated\n> in the file; or\n>\n> (c) The contribution was provided directly to me by some other\n> person who certified (a), (b) or (c) and I have not modified\n> it.\n>\n> (d) I understand and agree that this project and the contribution\n> are public and that a record of the contribution (including all\n> personal information I submit with it, including my sign-off) is\n> maintained indefinitely and may be redistributed consistent with\n> this project or the open source license(s) involved.\n\n### Add License Headers\n\nThe comments at the top of each file must contain this copyright notice:\n\n> Copyright © The Arvados Authors. All rights reserved.\n\nThey must also contain an `SPDX-License-Identifier` to identify the license of this component.\n\nIn most cases you can copy this header from another file in the component. If you need more guidance, refer to [the COPYING file](COPYING).\n\nIf it is not technically possible to add these comments to a file (for example, because it's a binary test file), you may add its path to the `.licenseignore` file instead.\n\n### Add Your Authorship\n\nIf you are not already listed in [the AUTHORS file](AUTHORS), please add yourself in the branch, following the documented format.\n\n## Create Your Pull Request\n\nOnce you've finished pushing changes to your branch, create a pull request against `arvados:main` with the following checklist filled out:\n\n * All agreed upon points are implemented / addressed. Describe changes from pre-implementation design.\n ** _comments_\n * Anything not implemented (discovered or discussed during work) has a follow-up story.\n ** _comments_\n * Code is tested and passing, both automated and manual, what manual testing was done is described.\n ** _comments_\n * The tested code incorporates recent main branch changes.\n ** _confirm_\n * New or changed UI/UX has gotten feedback from stakeholders.\n ** _comments_\n * Documentation has been updated.\n ** _comments_\n * Behaves appropriately at the intended scale (describe intended scale).\n ** _comments_\n * Considered backwards and forwards compatibility issues between client and server.\n ** _comments_\n * Follows our coding standards, including GUI style guidelines\n ** _comments_\n\n\"Incorporates recent main branch changes\" means that the branch is either based on, or merged, the `main` branch within the last week. The more active development on a component is, the more important it is to be up-to-date with main to avoid surprising test failures post-merge.\n\nUI/UX stands for “User Interface / User Experience”. This includes new or modified GUI elements in Workbench and as well as usability elements of command line tools.\n\nStakeholders typically include the product manager and may include designers, salespeople, customers, and other end users as appropriate. In this process, the assigned developer demos the new feature, makes note of any feedback, and then based on their judgement either: implements the changes, provides a reason why the feedback cannot be acted on, or discusses how to handle the feedback with the product manager and/or assigned reviewer. This feedback is typically obtained in earlier drafts of the pull request before it is submitted for final review.\n\nA member of the core team will review the pull request. They may have questions or comments through the pull request interface. Once all issues have been resolved, your branch will be merged.\n\n## Continuous Integration\n\nContinuous integration is hosted at . Currently, external contributors cannot trigger test runs. Trusted contributors may be given permission to do so.\n\n## Community Chat\n\nYou can chat with other members of the [Arvados community on Gitter](https://gitter.im/arvados/community). Come say hi!\n" }, { "path": "COPYING", "content": "Unless indicated otherwise in the header of the file, the files in this\nrepository are distributed under one of three different licenses: AGPL-3.0,\nApache-2.0 or CC-BY-SA-3.0.\n\nIndividual files contain an SPDX tag that indicates the license for the file.\nThese are the three tags in use:\n\n SPDX-License-Identifier: AGPL-3.0\n SPDX-License-Identifier: Apache-2.0\n SPDX-License-Identifier: CC-BY-SA-3.0\n\nThis enables machine processing of license information based on the SPDX\nLicense Identifiers that are available here: http://spdx.org/licenses/\n\nThe full license text for each license is appended below, and is also available\nin this directory:\n\n AGPL-3.0: agpl-3.0.txt\n Apache-2.0: apache-2.0.txt\n CC-BY-SA-3.0: cc-by-sa-3.0.txt\n\nAs a general rule, code in the sdk/ directory is licensed Apache-2.0,\ndocumentation in the doc/ directory is licensed CC-BY-SA-3.0, and\neverything else is licensed AGPL-3.0.\n\n###############################################################################\n\n Apache License\n Version 2.0, January 2004\n http://www.apache.org/licenses/\n\n TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n 1. Definitions.\n\n \"License\" shall mean the terms and conditions for use, reproduction,\n and distribution as defined by Sections 1 through 9 of this document.\n\n \"Licensor\" shall mean the copyright owner or entity authorized by\n the copyright owner that is granting the License.\n\n \"Legal Entity\" shall mean the union of the acting entity and all\n other entities that control, are controlled by, or are under common\n control with that entity. For the purposes of this definition,\n \"control\" means (i) the power, direct or indirect, to cause the\n direction or management of such entity, whether by contract or\n otherwise, or (ii) ownership of fifty percent (50%) or more of the\n outstanding shares, or (iii) beneficial ownership of such entity.\n\n \"You\" (or \"Your\") shall mean an individual or Legal Entity\n exercising permissions granted by this License.\n\n \"Source\" form shall mean the preferred form for making modifications,\n including but not limited to software source code, documentation\n source, and configuration files.\n\n \"Object\" form shall mean any form resulting from mechanical\n transformation or translation of a Source form, including but\n not limited to compiled object code, generated documentation,\n and conversions to other media types.\n\n \"Work\" shall mean the work of authorship, whether in Source or\n Object form, made available under the License, as indicated by a\n copyright notice that is included in or attached to the work\n (an example is provided in the Appendix below).\n\n \"Derivative Works\" shall mean any work, whether in Source or Object\n form, that is based on (or derived from) the Work and for which the\n editorial revisions, annotations, elaborations, or other modifications\n represent, as a whole, an original work of authorship. For the purposes\n of this License, Derivative Works shall not include works that remain\n separable from, or merely link (or bind by name) to the interfaces of,\n the Work and Derivative Works thereof.\n\n \"Contribution\" shall mean any work of authorship, including\n the original version of the Work and any modifications or additions\n to that Work or Derivative Works thereof, that is intentionally\n submitted to Licensor for inclusion in the Work by the copyright owner\n or by an individual or Legal Entity authorized to submit on behalf of\n the copyright owner. For the purposes of this definition, \"submitted\"\n means any form of electronic, verbal, or written communication sent\n to the Licensor or its representatives, including but not limited to\n communication on electronic mailing lists, source code control systems,\n and issue tracking systems that are managed by, or on behalf of, the\n Licensor for the purpose of discussing and improving the Work, but\n excluding communication that is conspicuously marked or otherwise\n designated in writing by the copyright owner as \"Not a Contribution.\"\n\n \"Contributor\" shall mean Licensor and any individual or Legal Entity\n on behalf of whom a Contribution has been received by Licensor and\n subsequently incorporated within the Work.\n\n 2. Grant of Copyright License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n copyright license to reproduce, prepare Derivative Works of,\n publicly display, publicly perform, sublicense, and distribute the\n Work and such Derivative Works in Source or Object form.\n\n 3. Grant of Patent License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n (except as stated in this section) patent license to make, have made,\n use, offer to sell, sell, import, and otherwise transfer the Work,\n where such license applies only to those patent claims licensable\n by such Contributor that are necessarily infringed by their\n Contribution(s) alone or by combination of their Contribution(s)\n with the Work to which such Contribution(s) was submitted. If You\n institute patent litigation against any entity (including a\n cross-claim or counterclaim in a lawsuit) alleging that the Work\n or a Contribution incorporated within the Work constitutes direct\n or contributory patent infringement, then any patent licenses\n granted to You under this License for that Work shall terminate\n as of the date such litigation is filed.\n\n 4. Redistribution. You may reproduce and distribute copies of the\n Work or Derivative Works thereof in any medium, with or without\n modifications, and in Source or Object form, provided that You\n meet the following conditions:\n\n (a) You must give any other recipients of the Work or\n Derivative Works a copy of this License; and\n\n (b) You must cause any modified files to carry prominent notices\n stating that You changed the files; and\n\n (c) You must retain, in the Source form of any Derivative Works\n that You distribute, all copyright, patent, trademark, and\n attribution notices from the Source form of the Work,\n excluding those notices that do not pertain to any part of\n the Derivative Works; and\n\n (d) If the Work includes a \"NOTICE\" text file as part of its\n distribution, then any Derivative Works that You distribute must\n include a readable copy of the attribution notices contained\n within such NOTICE file, excluding those notices that do not\n pertain to any part of the Derivative Works, in at least one\n of the following places: within a NOTICE text file distributed\n as part of the Derivative Works; within the Source form or\n documentation, if provided along with the Derivative Works; or,\n within a display generated by the Derivative Works, if and\n wherever such third-party notices normally appear. The contents\n of the NOTICE file are for informational purposes only and\n do not modify the License. You may add Your own attribution\n notices within Derivative Works that You distribute, alongside\n or as an addendum to the NOTICE text from the Work, provided\n that such additional attribution notices cannot be construed\n as modifying the License.\n\n You may add Your own copyright statement to Your modifications and\n may provide additional or different license terms and conditions\n for use, reproduction, or distribution of Your modifications, or\n for any such Derivative Works as a whole, provided Your use,\n reproduction, and distribution of the Work otherwise complies with\n the conditions stated in this License.\n\n 5. Submission of Contributions. Unless You explicitly state otherwise,\n any Contribution intentionally submitted for inclusion in the Work\n by You to the Licensor shall be under the terms and conditions of\n this License, without any additional terms or conditions.\n Notwithstanding the above, nothing herein shall supersede or modify\n the terms of any separate license agreement you may have executed\n with Licensor regarding such Contributions.\n\n 6. Trademarks. This License does not grant permission to use the trade\n names, trademarks, service marks, or product names of the Licensor,\n except as required for reasonable and customary use in describing the\n origin of the Work and reproducing the content of the NOTICE file.\n\n 7. Disclaimer of Warranty. Unless required by applicable law or\n agreed to in writing, Licensor provides the Work (and each\n Contributor provides its Contributions) on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n implied, including, without limitation, any warranties or conditions\n of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n PARTICULAR PURPOSE. You are solely responsible for determining the\n appropriateness of using or redistributing the Work and assume any\n risks associated with Your exercise of permissions under this License.\n\n 8. Limitation of Liability. In no event and under no legal theory,\n whether in tort (including negligence), contract, or otherwise,\n unless required by applicable law (such as deliberate and grossly\n negligent acts) or agreed to in writing, shall any Contributor be\n liable to You for damages, including any direct, indirect, special,\n incidental, or consequential damages of any character arising as a\n result of this License or out of the use or inability to use the\n Work (including but not limited to damages for loss of goodwill,\n work stoppage, computer failure or malfunction, or any and all\n other commercial damages or losses), even if such Contributor\n has been advised of the possibility of such damages.\n\n 9. Accepting Warranty or Additional Liability. While redistributing\n the Work or Derivative Works thereof, You may choose to offer,\n and charge a fee for, acceptance of support, warranty, indemnity,\n or other liability obligations and/or rights consistent with this\n License. However, in accepting such obligations, You may act only\n on Your own behalf and on Your sole responsibility, not on behalf\n of any other Contributor, and only if You agree to indemnify,\n defend, and hold each Contributor harmless for any liability\n incurred by, or claims asserted against, such Contributor by reason\n of your accepting any such warranty or additional liability.\n\n END OF TERMS AND CONDITIONS\n\n APPENDIX: How to apply the Apache License to your work.\n\n To apply the Apache License to your work, attach the following\n boilerplate notice, with the fields enclosed by brackets \"[]\"\n replaced with your own identifying information. (Don't include\n the brackets!) The text should be enclosed in the appropriate\n comment syntax for the file format. We also recommend that a\n file or class name and description of purpose be included on the\n same \"printed page\" as the copyright notice for easier\n identification within third-party archives.\n\n Copyright [yyyy] [name of copyright owner]\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n\n###############################################################################\n\n GNU AFFERO GENERAL PUBLIC LICENSE\n Version 3, 19 November 2007\n\n Copyright (C) 2007 Free Software Foundation, Inc. \n Everyone is permitted to copy and distribute verbatim copies\n of this license document, but changing it is not allowed.\n\n Preamble\n\n The GNU Affero General Public License is a free, copyleft license for\nsoftware and other kinds of works, specifically designed to ensure\ncooperation with the community in the case of network server software.\n\n The licenses for most software and other practical works are designed\nto take away your freedom to share and change the works. By contrast,\nour General Public Licenses are intended to guarantee your freedom to\nshare and change all versions of a program--to make sure it remains free\nsoftware for all its users.\n\n When we speak of free software, we are referring to freedom, not\nprice. Our General Public Licenses are designed to make sure that you\nhave the freedom to distribute copies of free software (and charge for\nthem if you wish), that you receive source code or can get it if you\nwant it, that you can change the software or use pieces of it in new\nfree programs, and that you know you can do these things.\n\n Developers that use our General Public Licenses protect your rights\nwith two steps: (1) assert copyright on the software, and (2) offer\nyou this License which gives you legal permission to copy, distribute\nand/or modify the software.\n\n A secondary benefit of defending all users' freedom is that\nimprovements made in alternate versions of the program, if they\nreceive widespread use, become available for other developers to\nincorporate. Many developers of free software are heartened and\nencouraged by the resulting cooperation. However, in the case of\nsoftware used on network servers, this result may fail to come about.\nThe GNU General Public License permits making a modified version and\nletting the public access it on a server without ever releasing its\nsource code to the public.\n\n The GNU Affero General Public License is designed specifically to\nensure that, in such cases, the modified source code becomes available\nto the community. It requires the operator of a network server to\nprovide the source code of the modified version running there to the\nusers of that server. Therefore, public use of a modified version, on\na publicly accessible server, gives the public access to the source\ncode of the modified version.\n\n An older license, called the Affero General Public License and\npublished by Affero, was designed to accomplish similar goals. This is\na different license, not a version of the Affero GPL, but Affero has\nreleased a new version of the Affero GPL which permits relicensing under\nthis license.\n\n The precise terms and conditions for copying, distribution and\nmodification follow.\n\n TERMS AND CONDITIONS\n\n 0. Definitions.\n\n \"This License\" refers to version 3 of the GNU Affero General Public License.\n\n \"Copyright\" also means copyright-like laws that apply to other kinds of\nworks, such as semiconductor masks.\n\n \"The Program\" refers to any copyrightable work licensed under this\nLicense. Each licensee is addressed as \"you\". \"Licensees\" and\n\"recipients\" may be individuals or organizations.\n\n To \"modify\" a work means to copy from or adapt all or part of the work\nin a fashion requiring copyright permission, other than the making of an\nexact copy. The resulting work is called a \"modified version\" of the\nearlier work or a work \"based on\" the earlier work.\n\n A \"covered work\" means either the unmodified Program or a work based\non the Program.\n\n To \"propagate\" a work means to do anything with it that, without\npermission, would make you directly or secondarily liable for\ninfringement under applicable copyright law, except executing it on a\ncomputer or modifying a private copy. Propagation includes copying,\ndistribution (with or without modification), making available to the\npublic, and in some countries other activities as well.\n\n To \"convey\" a work means any kind of propagation that enables other\nparties to make or receive copies. Mere interaction with a user through\na computer network, with no transfer of a copy, is not conveying.\n\n An interactive user interface displays \"Appropriate Legal Notices\"\nto the extent that it includes a convenient and prominently visible\nfeature that (1) displays an appropriate copyright notice, and (2)\ntells the user that there is no warranty for the work (except to the\nextent that warranties are provided), that licensees may convey the\nwork under this License, and how to view a copy of this License. If\nthe interface presents a list of user commands or options, such as a\nmenu, a prominent item in the list meets this criterion.\n\n 1. Source Code.\n\n The \"source code\" for a work means the preferred form of the work\nfor making modifications to it. \"Object code\" means any non-source\nform of a work.\n\n A \"Standard Interface\" means an interface that either is an official\nstandard defined by a recognized standards body, or, in the case of\ninterfaces specified for a particular programming language, one that\nis widely used among developers working in that language.\n\n The \"System Libraries\" of an executable work include anything, other\nthan the work as a whole, that (a) is included in the normal form of\npackaging a Major Component, but which is not part of that Major\nComponent, and (b) serves only to enable use of the work with that\nMajor Component, or to implement a Standard Interface for which an\nimplementation is available to the public in source code form. A\n\"Major Component\", in this context, means a major essential component\n(kernel, window system, and so on) of the specific operating system\n(if any) on which the executable work runs, or a compiler used to\nproduce the work, or an object code interpreter used to run it.\n\n The \"Corresponding Source\" for a work in object code form means all\nthe source code needed to generate, install, and (for an executable\nwork) run the object code and to modify the work, including scripts to\ncontrol those activities. However, it does not include the work's\nSystem Libraries, or general-purpose tools or generally available free\nprograms which are used unmodified in performing those activities but\nwhich are not part of the work. For example, Corresponding Source\nincludes interface definition files associated with source files for\nthe work, and the source code for shared libraries and dynamically\nlinked subprograms that the work is specifically designed to require,\nsuch as by intimate data communication or control flow between those\nsubprograms and other parts of the work.\n\n The Corresponding Source need not include anything that users\ncan regenerate automatically from other parts of the Corresponding\nSource.\n\n The Corresponding Source for a work in source code form is that\nsame work.\n\n 2. Basic Permissions.\n\n All rights granted under this License are granted for the term of\ncopyright on the Program, and are irrevocable provided the stated\nconditions are met. This License explicitly affirms your unlimited\npermission to run the unmodified Program. The output from running a\ncovered work is covered by this License only if the output, given its\ncontent, constitutes a covered work. This License acknowledges your\nrights of fair use or other equivalent, as provided by copyright law.\n\n You may make, run and propagate covered works that you do not\nconvey, without conditions so long as your license otherwise remains\nin force. You may convey covered works to others for the sole purpose\nof having them make modifications exclusively for you, or provide you\nwith facilities for running those works, provided that you comply with\nthe terms of this License in conveying all material for which you do\nnot control copyright. Those thus making or running the covered works\nfor you must do so exclusively on your behalf, under your direction\nand control, on terms that prohibit them from making any copies of\nyour copyrighted material outside their relationship with you.\n\n Conveying under any other circumstances is permitted solely under\nthe conditions stated below. Sublicensing is not allowed; section 10\nmakes it unnecessary.\n\n 3. Protecting Users' Legal Rights From Anti-Circumvention Law.\n\n No covered work shall be deemed part of an effective technological\nmeasure under any applicable law fulfilling obligations under article\n11 of the WIPO copyright treaty adopted on 20 December 1996, or\nsimilar laws prohibiting or restricting circumvention of such\nmeasures.\n\n When you convey a covered work, you waive any legal power to forbid\ncircumvention of technological measures to the extent such circumvention\nis effected by exercising rights under this License with respect to\nthe covered work, and you disclaim any intention to limit operation or\nmodification of the work as a means of enforcing, against the work's\nusers, your or third parties' legal rights to forbid circumvention of\ntechnological measures.\n\n 4. Conveying Verbatim Copies.\n\n You may convey verbatim copies of the Program's source code as you\nreceive it, in any medium, provided that you conspicuously and\nappropriately publish on each copy an appropriate copyright notice;\nkeep intact all notices stating that this License and any\nnon-permissive terms added in accord with section 7 apply to the code;\nkeep intact all notices of the absence of any warranty; and give all\nrecipients a copy of this License along with the Program.\n\n You may charge any price or no price for each copy that you convey,\nand you may offer support or warranty protection for a fee.\n\n 5. Conveying Modified Source Versions.\n\n You may convey a work based on the Program, or the modifications to\nproduce it from the Program, in the form of source code under the\nterms of section 4, provided that you also meet all of these conditions:\n\n a) The work must carry prominent notices stating that you modified\n it, and giving a relevant date.\n\n b) The work must carry prominent notices stating that it is\n released under this License and any conditions added under section\n 7. This requirement modifies the requirement in section 4 to\n \"keep intact all notices\".\n\n c) You must license the entire work, as a whole, under this\n License to anyone who comes into possession of a copy. This\n License will therefore apply, along with any applicable section 7\n additional terms, to the whole of the work, and all its parts,\n regardless of how they are packaged. This License gives no\n permission to license the work in any other way, but it does not\n invalidate such permission if you have separately received it.\n\n d) If the work has interactive user interfaces, each must display\n Appropriate Legal Notices; however, if the Program has interactive\n interfaces that do not display Appropriate Legal Notices, your\n work need not make them do so.\n\n A compilation of a covered work with other separate and independent\nworks, which are not by their nature extensions of the covered work,\nand which are not combined with it such as to form a larger program,\nin or on a volume of a storage or distribution medium, is called an\n\"aggregate\" if the compilation and its resulting copyright are not\nused to limit the access or legal rights of the compilation's users\nbeyond what the individual works permit. Inclusion of a covered work\nin an aggregate does not cause this License to apply to the other\nparts of the aggregate.\n\n 6. Conveying Non-Source Forms.\n\n You may convey a covered work in object code form under the terms\nof sections 4 and 5, provided that you also convey the\nmachine-readable Corresponding Source under the terms of this License,\nin one of these ways:\n\n a) Convey the object code in, or embodied in, a physical product\n (including a physical distribution medium), accompanied by the\n Corresponding Source fixed on a durable physical medium\n customarily used for software interchange.\n\n b) Convey the object code in, or embodied in, a physical product\n (including a physical distribution medium), accompanied by a\n written offer, valid for at least three years and valid for as\n long as you offer spare parts or customer support for that product\n model, to give anyone who possesses the object code either (1) a\n copy of the Corresponding Source for all the software in the\n product that is covered by this License, on a durable physical\n medium customarily used for software interchange, for a price no\n more than your reasonable cost of physically performing this\n conveying of source, or (2) access to copy the\n Corresponding Source from a network server at no charge.\n\n c) Convey individual copies of the object code with a copy of the\n written offer to provide the Corresponding Source. This\n alternative is allowed only occasionally and noncommercially, and\n only if you received the object code with such an offer, in accord\n with subsection 6b.\n\n d) Convey the object code by offering access from a designated\n place (gratis or for a charge), and offer equivalent access to the\n Corresponding Source in the same way through the same place at no\n further charge. You need not require recipients to copy the\n Corresponding Source along with the object code. If the place to\n copy the object code is a network server, the Corresponding Source\n may be on a different server (operated by you or a third party)\n that supports equivalent copying facilities, provided you maintain\n clear directions next to the object code saying where to find the\n Corresponding Source. Regardless of what server hosts the\n Corresponding Source, you remain obligated to ensure that it is\n available for as long as needed to satisfy these requirements.\n\n e) Convey the object code using peer-to-peer transmission, provided\n you inform other peers where the object code and Corresponding\n Source of the work are being offered to the general public at no\n charge under subsection 6d.\n\n A separable portion of the object code, whose source code is excluded\nfrom the Corresponding Source as a System Library, need not be\nincluded in conveying the object code work.\n\n A \"User Product\" is either (1) a \"consumer product\", which means any\ntangible personal property which is normally used for personal, family,\nor household purposes, or (2) anything designed or sold for incorporation\ninto a dwelling. In determining whether a product is a consumer product,\ndoubtful cases shall be resolved in favor of coverage. For a particular\nproduct received by a particular user, \"normally used\" refers to a\ntypical or common use of that class of product, regardless of the status\nof the particular user or of the way in which the particular user\nactually uses, or expects or is expected to use, the product. A product\nis a consumer product regardless of whether the product has substantial\ncommercial, industrial or non-consumer uses, unless such uses represent\nthe only significant mode of use of the product.\n\n \"Installation Information\" for a User Product means any methods,\nprocedures, authorization keys, or other information required to install\nand execute modified versions of a covered work in that User Product from\na modified version of its Corresponding Source. The information must\nsuffice to ensure that the continued functioning of the modified object\ncode is in no case prevented or interfered with solely because\nmodification has been made.\n\n If you convey an object code work under this section in, or with, or\nspecifically for use in, a User Product, and the conveying occurs as\npart of a transaction in which the right of possession and use of the\nUser Product is transferred to the recipient in perpetuity or for a\nfixed term (regardless of how the transaction is characterized), the\nCorresponding Source conveyed under this section must be accompanied\nby the Installation Information. But this requirement does not apply\nif neither you nor any third party retains the ability to install\nmodified object code on the User Product (for example, the work has\nbeen installed in ROM).\n\n The requirement to provide Installation Information does not include a\nrequirement to continue to provide support service, warranty, or updates\nfor a work that has been modified or installed by the recipient, or for\nthe User Product in which it has been modified or installed. Access to a\nnetwork may be denied when the modification itself materially and\nadversely affects the operation of the network or violates the rules and\nprotocols for communication across the network.\n\n Corresponding Source conveyed, and Installation Information provided,\nin accord with this section must be in a format that is publicly\ndocumented (and with an implementation available to the public in\nsource code form), and must require no special password or key for\nunpacking, reading or copying.\n\n 7. Additional Terms.\n\n \"Additional permissions\" are terms that supplement the terms of this\nLicense by making exceptions from one or more of its conditions.\nAdditional permissions that are applicable to the entire Program shall\nbe treated as though they were included in this License, to the extent\nthat they are valid under applicable law. If additional permissions\napply only to part of the Program, that part may be used separately\nunder those permissions, but the entire Program remains governed by\nthis License without regard to the additional permissions.\n\n When you convey a copy of a covered work, you may at your option\nremove any additional permissions from that copy, or from any part of\nit. (Additional permissions may be written to require their own\nremoval in certain cases when you modify the work.) You may place\nadditional permissions on material, added by you to a covered work,\nfor which you have or can give appropriate copyright permission.\n\n Notwithstanding any other provision of this License, for material you\nadd to a covered work, you may (if authorized by the copyright holders of\nthat material) supplement the terms of this License with terms:\n\n a) Disclaiming warranty or limiting liability differently from the\n terms of sections 15 and 16 of this License; or\n\n b) Requiring preservation of specified reasonable legal notices or\n author attributions in that material or in the Appropriate Legal\n Notices displayed by works containing it; or\n\n c) Prohibiting misrepresentation of the origin of that material, or\n requiring that modified versions of such material be marked in\n reasonable ways as different from the original version; or\n\n d) Limiting the use for publicity purposes of names of licensors or\n authors of the material; or\n\n e) Declining to grant rights under trademark law for use of some\n trade names, trademarks, or service marks; or\n\n f) Requiring indemnification of licensors and authors of that\n material by anyone who conveys the material (or modified versions of\n it) with contractual assumptions of liability to the recipient, for\n any liability that these contractual assumptions directly impose on\n those licensors and authors.\n\n All other non-permissive additional terms are considered \"further\nrestrictions\" within the meaning of section 10. If the Program as you\nreceived it, or any part of it, contains a notice stating that it is\ngoverned by this License along with a term that is a further\nrestriction, you may remove that term. If a license document contains\na further restriction but permits relicensing or conveying under this\nLicense, you may add to a covered work material governed by the terms\nof that license document, provided that the further restriction does\nnot survive such relicensing or conveying.\n\n If you add terms to a covered work in accord with this section, you\nmust place, in the relevant source files, a statement of the\nadditional terms that apply to those files, or a notice indicating\nwhere to find the applicable terms.\n\n Additional terms, permissive or non-permissive, may be stated in the\nform of a separately written license, or stated as exceptions;\nthe above requirements apply either way.\n\n 8. Termination.\n\n You may not propagate or modify a covered work except as expressly\nprovided under this License. Any attempt otherwise to propagate or\nmodify it is void, and will automatically terminate your rights under\nthis License (including any patent licenses granted under the third\nparagraph of section 11).\n\n However, if you cease all violation of this License, then your\nlicense from a particular copyright holder is reinstated (a)\nprovisionally, unless and until the copyright holder explicitly and\nfinally terminates your license, and (b) permanently, if the copyright\nholder fails to notify you of the violation by some reasonable means\nprior to 60 days after the cessation.\n\n Moreover, your license from a particular copyright holder is\nreinstated permanently if the copyright holder notifies you of the\nviolation by some reasonable means, this is the first time you have\nreceived notice of violation of this License (for any work) from that\ncopyright holder, and you cure the violation prior to 30 days after\nyour receipt of the notice.\n\n Termination of your rights under this section does not terminate the\nlicenses of parties who have received copies or rights from you under\nthis License. If your rights have been terminated and not permanently\nreinstated, you do not qualify to receive new licenses for the same\nmaterial under section 10.\n\n 9. Acceptance Not Required for Having Copies.\n\n You are not required to accept this License in order to receive or\nrun a copy of the Program. Ancillary propagation of a covered work\noccurring solely as a consequence of using peer-to-peer transmission\nto receive a copy likewise does not require acceptance. However,\nnothing other than this License grants you permission to propagate or\nmodify any covered work. These actions infringe copyright if you do\nnot accept this License. Therefore, by modifying or propagating a\ncovered work, you indicate your acceptance of this License to do so.\n\n 10. Automatic Licensing of Downstream Recipients.\n\n Each time you convey a covered work, the recipient automatically\nreceives a license from the original licensors, to run, modify and\npropagate that work, subject to this License. You are not responsible\nfor enforcing compliance by third parties with this License.\n\n An \"entity transaction\" is a transaction transferring control of an\norganization, or substantially all assets of one, or subdividing an\norganization, or merging organizations. If propagation of a covered\nwork results from an entity transaction, each party to that\ntransaction who receives a copy of the work also receives whatever\nlicenses to the work the party's predecessor in interest had or could\ngive under the previous paragraph, plus a right to possession of the\nCorresponding Source of the work from the predecessor in interest, if\nthe predecessor has it or can get it with reasonable efforts.\n\n You may not impose any further restrictions on the exercise of the\nrights granted or affirmed under this License. For example, you may\nnot impose a license fee, royalty, or other charge for exercise of\nrights granted under this License, and you may not initiate litigation\n(including a cross-claim or counterclaim in a lawsuit) alleging that\nany patent claim is infringed by making, using, selling, offering for\nsale, or importing the Program or any portion of it.\n\n 11. Patents.\n\n A \"contributor\" is a copyright holder who authorizes use under this\nLicense of the Program or a work on which the Program is based. The\nwork thus licensed is called the contributor's \"contributor version\".\n\n A contributor's \"essential patent claims\" are all patent claims\nowned or controlled by the contributor, whether already acquired or\nhereafter acquired, that would be infringed by some manner, permitted\nby this License, of making, using, or selling its contributor version,\nbut do not include claims that would be infringed only as a\nconsequence of further modification of the contributor version. For\npurposes of this definition, \"control\" includes the right to grant\npatent sublicenses in a manner consistent with the requirements of\nthis License.\n\n Each contributor grants you a non-exclusive, worldwide, royalty-free\npatent license under the contributor's essential patent claims, to\nmake, use, sell, offer for sale, import and otherwise run, modify and\npropagate the contents of its contributor version.\n\n In the following three paragraphs, a \"patent license\" is any express\nagreement or commitment, however denominated, not to enforce a patent\n(such as an express permission to practice a patent or covenant not to\nsue for patent infringement). To \"grant\" such a patent license to a\nparty means to make such an agreement or commitment not to enforce a\npatent against the party.\n\n If you convey a covered work, knowingly relying on a patent license,\nand the Corresponding Source of the work is not available for anyone\nto copy, free of charge and under the terms of this License, through a\npublicly available network server or other readily accessible means,\nthen you must either (1) cause the Corresponding Source to be so\navailable, or (2) arrange to deprive yourself of the benefit of the\npatent license for this particular work, or (3) arrange, in a manner\nconsistent with the requirements of this License, to extend the patent\nlicense to downstream recipients. \"Knowingly relying\" means you have\nactual knowledge that, but for the patent license, your conveying the\ncovered work in a country, or your recipient's use of the covered work\nin a country, would infringe one or more identifiable patents in that\ncountry that you have reason to believe are valid.\n\n If, pursuant to or in connection with a single transaction or\narrangement, you convey, or propagate by procuring conveyance of, a\ncovered work, and grant a patent license to some of the parties\nreceiving the covered work authorizing them to use, propagate, modify\nor convey a specific copy of the covered work, then the patent license\nyou grant is automatically extended to all recipients of the covered\nwork and works based on it.\n\n A patent license is \"discriminatory\" if it does not include within\nthe scope of its coverage, prohibits the exercise of, or is\nconditioned on the non-exercise of one or more of the rights that are\nspecifically granted under this License. You may not convey a covered\nwork if you are a party to an arrangement with a third party that is\nin the business of distributing software, under which you make payment\nto the third party based on the extent of your activity of conveying\nthe work, and under which the third party grants, to any of the\nparties who would receive the covered work from you, a discriminatory\npatent license (a) in connection with copies of the covered work\nconveyed by you (or copies made from those copies), or (b) primarily\nfor and in connection with specific products or compilations that\ncontain the covered work, unless you entered into that arrangement,\nor that patent license was granted, prior to 28 March 2007.\n\n Nothing in this License shall be construed as excluding or limiting\nany implied license or other defenses to infringement that may\notherwise be available to you under applicable patent law.\n\n 12. No Surrender of Others' Freedom.\n\n If conditions are imposed on you (whether by court order, agreement or\notherwise) that contradict the conditions of this License, they do not\nexcuse you from the conditions of this License. If you cannot convey a\ncovered work so as to satisfy simultaneously your obligations under this\nLicense and any other pertinent obligations, then as a consequence you may\nnot convey it at all. For example, if you agree to terms that obligate you\nto collect a royalty for further conveying from those to whom you convey\nthe Program, the only way you could satisfy both those terms and this\nLicense would be to refrain entirely from conveying the Program.\n\n 13. Remote Network Interaction; Use with the GNU General Public License.\n\n Notwithstanding any other provision of this License, if you modify the\nProgram, your modified version must prominently offer all users\ninteracting with it remotely through a computer network (if your version\nsupports such interaction) an opportunity to receive the Corresponding\nSource of your version by providing access to the Corresponding Source\nfrom a network server at no charge, through some standard or customary\nmeans of facilitating copying of software. This Corresponding Source\nshall include the Corresponding Source for any work covered by version 3\nof the GNU General Public License that is incorporated pursuant to the\nfollowing paragraph.\n\n Notwithstanding any other provision of this License, you have\npermission to link or combine any covered work with a work licensed\nunder version 3 of the GNU General Public License into a single\ncombined work, and to convey the resulting work. The terms of this\nLicense will continue to apply to the part which is the covered work,\nbut the work with which it is combined will remain governed by version\n3 of the GNU General Public License.\n\n 14. Revised Versions of this License.\n\n The Free Software Foundation may publish revised and/or new versions of\nthe GNU Affero General Public License from time to time. Such new versions\nwill be similar in spirit to the present version, but may differ in detail to\naddress new problems or concerns.\n\n Each version is given a distinguishing version number. If the\nProgram specifies that a certain numbered version of the GNU Affero General\nPublic License \"or any later version\" applies to it, you have the\noption of following the terms and conditions either of that numbered\nversion or of any later version published by the Free Software\nFoundation. If the Program does not specify a version number of the\nGNU Affero General Public License, you may choose any version ever published\nby the Free Software Foundation.\n\n If the Program specifies that a proxy can decide which future\nversions of the GNU Affero General Public License can be used, that proxy's\npublic statement of acceptance of a version permanently authorizes you\nto choose that version for the Program.\n\n Later license versions may give you additional or different\npermissions. However, no additional obligations are imposed on any\nauthor or copyright holder as a result of your choosing to follow a\nlater version.\n\n 15. Disclaimer of Warranty.\n\n THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY\nAPPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT\nHOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM \"AS IS\" WITHOUT WARRANTY\nOF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,\nTHE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR\nPURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM\nIS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF\nALL NECESSARY SERVICING, REPAIR OR CORRECTION.\n\n 16. Limitation of Liability.\n\n IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING\nWILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS\nTHE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY\nGENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE\nUSE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF\nDATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD\nPARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),\nEVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF\nSUCH DAMAGES.\n\n 17. Interpretation of Sections 15 and 16.\n\n If the disclaimer of warranty and limitation of liability provided\nabove cannot be given local legal effect according to their terms,\nreviewing courts shall apply local law that most closely approximates\nan absolute waiver of all civil liability in connection with the\nProgram, unless a warranty or assumption of liability accompanies a\ncopy of the Program in return for a fee.\n\n END OF TERMS AND CONDITIONS\n\n How to Apply These Terms to Your New Programs\n\n If you develop a new program, and you want it to be of the greatest\npossible use to the public, the best way to achieve this is to make it\nfree software which everyone can redistribute and change under these terms.\n\n To do so, attach the following notices to the program. It is safest\nto attach them to the start of each source file to most effectively\nstate the exclusion of warranty; and each file should have at least\nthe \"copyright\" line and a pointer to where the full notice is found.\n\n \n Copyright (C) \n\n This program is free software: you can redistribute it and/or modify\n it under the terms of the GNU Affero General Public License as published by\n the Free Software Foundation, either version 3 of the License, or\n (at your option) any later version.\n\n This program is distributed in the hope that it will be useful,\n but WITHOUT ANY WARRANTY; without even the implied warranty of\n MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n GNU Affero General Public License for more details.\n\n You should have received a copy of the GNU Affero General Public License\n along with this program. If not, see .\n\nAlso add information on how to contact you by electronic and paper mail.\n\n If your software can interact with users remotely through a computer\nnetwork, you should also make sure that it provides a way for users to\nget its source. For example, if your program is a web application, its\ninterface could display a \"Source\" link that leads users to an archive\nof the code. There are many ways you could offer source, and different\nsolutions will be better for different programs; see section 13 for the\nspecific requirements.\n\n You should also get your employer (if you work as a programmer) or school,\nif any, to sign a \"copyright disclaimer\" for the program, if necessary.\nFor more information on this, and how to apply and follow the GNU AGPL, see\n.\n\n###############################################################################\n\nAttribution-ShareAlike 3.0 Unported\n\n CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE LEGAL SERVICES. DISTRIBUTION OF THIS LICENSE DOES NOT CREATE AN ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS INFORMATION ON AN \"AS-IS\" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES REGARDING THE INFORMATION PROVIDED, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM ITS USE.\n\nLicense\n\nTHE WORK (AS DEFINED BELOW) IS PROVIDED UNDER THE TERMS OF THIS CREATIVE COMMONS PUBLIC LICENSE (\"CCPL\" OR \"LICENSE\"). THE WORK IS PROTECTED BY COPYRIGHT AND/OR OTHER APPLICABLE LAW. ANY USE OF THE WORK OTHER THAN AS AUTHORIZED UNDER THIS LICENSE OR COPYRIGHT LAW IS PROHIBITED.\n\nBY EXERCISING ANY RIGHTS TO THE WORK PROVIDED HERE, YOU ACCEPT AND AGREE TO BE BOUND BY THE TERMS OF THIS LICENSE. TO THE EXTENT THIS LICENSE MAY BE CONSIDERED TO BE A CONTRACT, THE LICENSOR GRANTS YOU THE RIGHTS CONTAINED HERE IN CONSIDERATION OF YOUR ACCEPTANCE OF SUCH TERMS AND CONDITIONS.\n\n1. Definitions\n\n \"Adaptation\" means a work based upon the Work, or upon the Work and other pre-existing works, such as a translation, adaptation, derivative work, arrangement of music or other alterations of a literary or artistic work, or phonogram or performance and includes cinematographic adaptations or any other form in which the Work may be recast, transformed, or adapted including in any form recognizably derived from the original, except that a work that constitutes a Collection will not be considered an Adaptation for the purpose of this License. For the avoidance of doubt, where the Work is a musical work, performance or phonogram, the synchronization of the Work in timed-relation with a moving image (\"synching\") will be considered an Adaptation for the purpose of this License.\n \"Collection\" means a collection of literary or artistic works, such as encyclopedias and anthologies, or performances, phonograms or broadcasts, or other works or subject matter other than works listed in Section 1(f) below, which, by reason of the selection and arrangement of their contents, constitute intellectual creations, in which the Work is included in its entirety in unmodified form along with one or more other contributions, each constituting separate and independent works in themselves, which together are assembled into a collective whole. A work that constitutes a Collection will not be considered an Adaptation (as defined below) for the purposes of this License.\n \"Creative Commons Compatible License\" means a license that is listed at https://creativecommons.org/compatiblelicenses that has been approved by Creative Commons as being essentially equivalent to this License, including, at a minimum, because that license: (i) contains terms that have the same purpose, meaning and effect as the License Elements of this License; and, (ii) explicitly permits the relicensing of adaptations of works made available under that license under this License or a Creative Commons jurisdiction license with the same License Elements as this License.\n \"Distribute\" means to make available to the public the original and copies of the Work or Adaptation, as appropriate, through sale or other transfer of ownership.\n \"License Elements\" means the following high-level license attributes as selected by Licensor and indicated in the title of this License: Attribution, ShareAlike.\n \"Licensor\" means the individual, individuals, entity or entities that offer(s) the Work under the terms of this License.\n \"Original Author\" means, in the case of a literary or artistic work, the individual, individuals, entity or entities who created the Work or if no individual or entity can be identified, the publisher; and in addition (i) in the case of a performance the actors, singers, musicians, dancers, and other persons who act, sing, deliver, declaim, play in, interpret or otherwise perform literary or artistic works or expressions of folklore; (ii) in the case of a phonogram the producer being the person or legal entity who first fixes the sounds of a performance or other sounds; and, (iii) in the case of broadcasts, the organization that transmits the broadcast.\n \"Work\" means the literary and/or artistic work offered under the terms of this License including without limitation any production in the literary, scientific and artistic domain, whatever may be the mode or form of its expression including digital form, such as a book, pamphlet and other writing; a lecture, address, sermon or other work of the same nature; a dramatic or dramatico-musical work; a choreographic work or entertainment in dumb show; a musical composition with or without words; a cinematographic work to which are assimilated works expressed by a process analogous to cinematography; a work of drawing, painting, architecture, sculpture, engraving or lithography; a photographic work to which are assimilated works expressed by a process analogous to photography; a work of applied art; an illustration, map, plan, sketch or three-dimensional work relative to geography, topography, architecture or science; a performance; a broadcast; a phonogram; a compilation of data to the extent it is protected as a copyrightable work; or a work performed by a variety or circus performer to the extent it is not otherwise considered a literary or artistic work.\n \"You\" means an individual or entity exercising rights under this License who has not previously violated the terms of this License with respect to the Work, or who has received express permission from the Licensor to exercise rights under this License despite a previous violation.\n \"Publicly Perform\" means to perform public recitations of the Work and to communicate to the public those public recitations, by any means or process, including by wire or wireless means or public digital performances; to make available to the public Works in such a way that members of the public may access these Works from a place and at a place individually chosen by them; to perform the Work to the public by any means or process and the communication to the public of the performances of the Work, including by public digital performance; to broadcast and rebroadcast the Work by any means including signs, sounds or images.\n \"Reproduce\" means to make copies of the Work by any means including without limitation by sound or visual recordings and the right of fixation and reproducing fixations of the Work, including storage of a protected performance or phonogram in digital form or other electronic medium.\n\n2. Fair Dealing Rights. Nothing in this License is intended to reduce, limit, or restrict any uses free from copyright or rights arising from limitations or exceptions that are provided for in connection with the copyright protection under copyright law or other applicable laws.\n\n3. License Grant. Subject to the terms and conditions of this License, Licensor hereby grants You a worldwide, royalty-free, non-exclusive, perpetual (for the duration of the applicable copyright) license to exercise the rights in the Work as stated below:\n\n to Reproduce the Work, to incorporate the Work into one or more Collections, and to Reproduce the Work as incorporated in the Collections;\n to create and Reproduce Adaptations provided that any such Adaptation, including any translation in any medium, takes reasonable steps to clearly label, demarcate or otherwise identify that changes were made to the original Work. For example, a translation could be marked \"The original work was translated from English to Spanish,\" or a modification could indicate \"The original work has been modified.\";\n to Distribute and Publicly Perform the Work including as incorporated in Collections; and,\n to Distribute and Publicly Perform Adaptations.\n\n For the avoidance of doubt:\n Non-waivable Compulsory License Schemes. In those jurisdictions in which the right to collect royalties through any statutory or compulsory licensing scheme cannot be waived, the Licensor reserves the exclusive right to collect such royalties for any exercise by You of the rights granted under this License;\n Waivable Compulsory License Schemes. In those jurisdictions in which the right to collect royalties through any statutory or compulsory licensing scheme can be waived, the Licensor waives the exclusive right to collect such royalties for any exercise by You of the rights granted under this License; and,\n Voluntary License Schemes. The Licensor waives the right to collect royalties, whether individually or, in the event that the Licensor is a member of a collecting society that administers voluntary licensing schemes, via that society, from any exercise by You of the rights granted under this License.\n\nThe above rights may be exercised in all media and formats whether now known or hereafter devised. The above rights include the right to make such modifications as are technically necessary to exercise the rights in other media and formats. Subject to Section 8(f), all rights not expressly granted by Licensor are hereby reserved.\n\n4. Restrictions. The license granted in Section 3 above is expressly made subject to and limited by the following restrictions:\n\n You may Distribute or Publicly Perform the Work only under the terms of this License. You must include a copy of, or the Uniform Resource Identifier (URI) for, this License with every copy of the Work You Distribute or Publicly Perform. You may not offer or impose any terms on the Work that restrict the terms of this License or the ability of the recipient of the Work to exercise the rights granted to that recipient under the terms of the License. You may not sublicense the Work. You must keep intact all notices that refer to this License and to the disclaimer of warranties with every copy of the Work You Distribute or Publicly Perform. When You Distribute or Publicly Perform the Work, You may not impose any effective technological measures on the Work that restrict the ability of a recipient of the Work from You to exercise the rights granted to that recipient under the terms of the License. This Section 4(a) applies to the Work as incorporated in a Collection, but this does not require the Collection apart from the Work itself to be made subject to the terms of this License. If You create a Collection, upon notice from any Licensor You must, to the extent practicable, remove from the Collection any credit as required by Section 4(c), as requested. If You create an Adaptation, upon notice from any Licensor You must, to the extent practicable, remove from the Adaptation any credit as required by Section 4(c), as requested.\n You may Distribute or Publicly Perform an Adaptation only under the terms of: (i) this License; (ii) a later version of this License with the same License Elements as this License; (iii) a Creative Commons jurisdiction license (either this or a later license version) that contains the same License Elements as this License (e.g., Attribution-ShareAlike 3.0 US)); (iv) a Creative Commons Compatible License. If you license the Adaptation under one of the licenses mentioned in (iv), you must comply with the terms of that license. If you license the Adaptation under the terms of any of the licenses mentioned in (i), (ii) or (iii) (the \"Applicable License\"), you must comply with the terms of the Applicable License generally and the following provisions: (I) You must include a copy of, or the URI for, the Applicable License with every copy of each Adaptation You Distribute or Publicly Perform; (II) You may not offer or impose any terms on the Adaptation that restrict the terms of the Applicable License or the ability of the recipient of the Adaptation to exercise the rights granted to that recipient under the terms of the Applicable License; (III) You must keep intact all notices that refer to the Applicable License and to the disclaimer of warranties with every copy of the Work as included in the Adaptation You Distribute or Publicly Perform; (IV) when You Distribute or Publicly Perform the Adaptation, You may not impose any effective technological measures on the Adaptation that restrict the ability of a recipient of the Adaptation from You to exercise the rights granted to that recipient under the terms of the Applicable License. This Section 4(b) applies to the Adaptation as incorporated in a Collection, but this does not require the Collection apart from the Adaptation itself to be made subject to the terms of the Applicable License.\n If You Distribute, or Publicly Perform the Work or any Adaptations or Collections, You must, unless a request has been made pursuant to Section 4(a), keep intact all copyright notices for the Work and provide, reasonable to the medium or means You are utilizing: (i) the name of the Original Author (or pseudonym, if applicable) if supplied, and/or if the Original Author and/or Licensor designate another party or parties (e.g., a sponsor institute, publishing entity, journal) for attribution (\"Attribution Parties\") in Licensor's copyright notice, terms of service or by other reasonable means, the name of such party or parties; (ii) the title of the Work if supplied; (iii) to the extent reasonably practicable, the URI, if any, that Licensor specifies to be associated with the Work, unless such URI does not refer to the copyright notice or licensing information for the Work; and (iv) , consistent with Ssection 3(b), in the case of an Adaptation, a credit identifying the use of the Work in the Adaptation (e.g., \"French translation of the Work by Original Author,\" or \"Screenplay based on original Work by Original Author\"). The credit required by this Section 4(c) may be implemented in any reasonable manner; provided, however, that in the case of a Adaptation or Collection, at a minimum such credit will appear, if a credit for all contributing authors of the Adaptation or Collection appears, then as part of these credits and in a manner at least as prominent as the credits for the other contributing authors. For the avoidance of doubt, You may only use the credit required by this Section for the purpose of attribution in the manner set out above and, by exercising Your rights under this License, You may not implicitly or explicitly assert or imply any connection with, sponsorship or endorsement by the Original Author, Licensor and/or Attribution Parties, as appropriate, of You or Your use of the Work, without the separate, express prior written permission of the Original Author, Licensor and/or Attribution Parties.\n Except as otherwise agreed in writing by the Licensor or as may be otherwise permitted by applicable law, if You Reproduce, Distribute or Publicly Perform the Work either by itself or as part of any Adaptations or Collections, You must not distort, mutilate, modify or take other derogatory action in relation to the Work which would be prejudicial to the Original Author's honor or reputation. Licensor agrees that in those jurisdictions (e.g. Japan), in which any exercise of the right granted in Section 3(b) of this License (the right to make Adaptations) would be deemed to be a distortion, mutilation, modification or other derogatory action prejudicial to the Original Author's honor and reputation, the Licensor will waive or not assert, as appropriate, this Section, to the fullest extent permitted by the applicable national law, to enable You to reasonably exercise Your right under Section 3(b) of this License (right to make Adaptations) but not otherwise.\n\n5. Representations, Warranties and Disclaimer\n\nUNLESS OTHERWISE MUTUALLY AGREED TO BY THE PARTIES IN WRITING, LICENSOR OFFERS THE WORK AS-IS AND MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE WORK, EXPRESS, IMPLIED, STATUTORY OR OTHERWISE, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF TITLE, MERCHANTIBILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, ACCURACY, OR THE PRESENCE OF ABSENCE OF ERRORS, WHETHER OR NOT DISCOVERABLE. SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OF IMPLIED WARRANTIES, SO SUCH EXCLUSION MAY NOT APPLY TO YOU.\n\n6. Limitation on Liability. EXCEPT TO THE EXTENT REQUIRED BY APPLICABLE LAW, IN NO EVENT WILL LICENSOR BE LIABLE TO YOU ON ANY LEGAL THEORY FOR ANY SPECIAL, INCIDENTAL, CONSEQUENTIAL, PUNITIVE OR EXEMPLARY DAMAGES ARISING OUT OF THIS LICENSE OR THE USE OF THE WORK, EVEN IF LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.\n\n7. Termination\n\n This License and the rights granted hereunder will terminate automatically upon any breach by You of the terms of this License. Individuals or entities who have received Adaptations or Collections from You under this License, however, will not have their licenses terminated provided such individuals or entities remain in full compliance with those licenses. Sections 1, 2, 5, 6, 7, and 8 will survive any termination of this License.\n Subject to the above terms and conditions, the license granted here is perpetual (for the duration of the applicable copyright in the Work). Notwithstanding the above, Licensor reserves the right to release the Work under different license terms or to stop distributing the Work at any time; provided, however that any such election will not serve to withdraw this License (or any other license that has been, or is required to be, granted under the terms of this License), and this License will continue in full force and effect unless terminated as stated above.\n\n8. Miscellaneous\n\n Each time You Distribute or Publicly Perform the Work or a Collection, the Licensor offers to the recipient a license to the Work on the same terms and conditions as the license granted to You under this License.\n Each time You Distribute or Publicly Perform an Adaptation, Licensor offers to the recipient a license to the original Work on the same terms and conditions as the license granted to You under this License.\n If any provision of this License is invalid or unenforceable under applicable law, it shall not affect the validity or enforceability of the remainder of the terms of this License, and without further action by the parties to this agreement, such provision shall be reformed to the minimum extent necessary to make such provision valid and enforceable.\n No term or provision of this License shall be deemed waived and no breach consented to unless such waiver or consent shall be in writing and signed by the party to be charged with such waiver or consent.\n This License constitutes the entire agreement between the parties with respect to the Work licensed here. There are no understandings, agreements or representations with respect to the Work not specified here. Licensor shall not be bound by any additional provisions that may appear in any communication from You. This License may not be modified without the mutual written agreement of the Licensor and You.\n The rights granted under, and the subject matter referenced, in this License were drafted utilizing the terminology of the Berne Convention for the Protection of Literary and Artistic Works (as amended on September 28, 1979), the Rome Convention of 1961, the WIPO Copyright Treaty of 1996, the WIPO Performances and Phonograms Treaty of 1996 and the Universal Copyright Convention (as revised on July 24, 1971). These rights and subject matter take effect in the relevant jurisdiction in which the License terms are sought to be enforced according to the corresponding provisions of the implementation of those treaty provisions in the applicable national law. If the standard suite of rights granted under applicable copyright law includes additional rights not granted under this License, such additional rights are deemed to be included in the License; this License is not intended to restrict the license of any rights under applicable law.\n\n Creative Commons Notice\n\n Creative Commons is not a party to this License, and makes no warranty whatsoever in connection with the Work. Creative Commons will not be liable to You or any party on any legal theory for any damages whatsoever, including without limitation any general, special, incidental or consequential damages arising in connection to this license. Notwithstanding the foregoing two (2) sentences, if Creative Commons has expressly identified itself as the Licensor hereunder, it shall have all rights and obligations of Licensor.\n\n Except for the limited purpose of indicating to the public that the Work is licensed under the CCPL, Creative Commons does not authorize the use by either party of the trademark \"Creative Commons\" or any related trademark or logo of Creative Commons without the prior written consent of Creative Commons. Any permitted use will be in compliance with Creative Commons' then-current trademark usage guidelines, as may be published on its website or otherwise made available upon request from time to time. For the avoidance of doubt, this trademark restriction does not form part of the License.\n\n Creative Commons may be contacted at https://creativecommons.org/.\n" }, { "path": "Makefile", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\nexport WORKSPACE?=$(shell pwd)\nhelp:\n\t@echo >&2\n\t@echo >&2 \"There is no default make target here. Did you mean 'make test'?\"\n\t@echo >&2\n\t@echo >&2 \"More info:\"\n\t@echo >&2 \" Installing --> http://doc.arvados.org/install\"\n\t@echo >&2 \" Developing/contributing --> https://github.com/arvados/arvados\"\n\t@echo >&2 \" Project home --> https://arvados.org\"\n\t@echo >&2\n\t@false\ntest:\n\tbuild/run-tests.sh ${TEST_FLAGS}\npackages:\n\tbuild/run-build-packages-all-targets.sh ${PACKAGES_FLAGS}\ntest-packages:\n\tbuild/run-build-packages-all-targets.sh --test-packages ${PACKAGES_FLAGS}\n" }, { "path": "README.md", "content": "[comment]: # (Copyright © The Arvados Authors. All rights reserved.)\n[comment]: # ()\n[comment]: # (SPDX-License-Identifier: CC-BY-SA-3.0)\n\n[![Join the chat at https://gitter.im/arvados/community](https://badges.gitter.im/arvados/community.svg)](https://gitter.im/arvados/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) | [Installing Arvados](https://doc.arvados.org/install/index.html) | [Installing Client SDKs](https://doc.arvados.org/sdk/index.html) | [Report a bug](https://github.com/arvados/arvados/issues/new) | [Development and Contributing](CONTRIBUTING.md)\n\n\n\n[Arvados](https://arvados.org) is an open source platform for\nmanaging, processing, and sharing genomic and other large scientific\nand biomedical data. With Arvados, bioinformaticians run and scale\ncompute-intensive workflows, developers create biomedical\napplications, and IT administrators manage large compute and storage\nresources.\n\nThe key components of Arvados are:\n\n* *Keep*: Keep is the Arvados storage system for managing and storing large\ncollections of files. Keep combines content addressing and a\ndistributed storage architecture resulting in both high reliability\nand high throughput. Every file stored in Keep can be accurately\nverified every time it is retrieved. Keep supports the creation of\ncollections as a flexible way to define data sets without having to\nre-organize or needlessly copy data. Keep works on a wide range of\nunderlying filesystems and object stores.\n\n* *Crunch*: Crunch is the orchestration system for running [Common Workflow Language](https://www.commonwl.org) workflows. It is\ndesigned to maintain data provenance and workflow\nreproducibility. Crunch automatically tracks data inputs and outputs\nthrough Keep and executes workflow processes in Docker containers. In\na cloud environment, Crunch optimizes costs by scaling compute on demand.\n\n* *Workbench*: The Workbench web application allows users to interactively access\nArvados functionality. It is especially helpful for querying and\nbrowsing data, visualizing provenance, and tracking the progress of\nworkflows.\n\n* *Command Line tools*: The command line interface (CLI) provides convenient access to Arvados\nfunctionality in the Arvados platform from the command line.\n\n* *API and SDKs*: Arvados is designed to be integrated with existing infrastructure. All\nthe services in Arvados are accessed through a RESTful API. SDKs are\navailable for Python, Go, R, Perl, Ruby, and Java.\n\n# Documentation\n\nComplete documentation, including the [User Guide](https://doc.arvados.org/user/index.html), [Installation documentation](https://doc.arvados.org/install/index.html), [Administrator documentation](https://doc.arvados.org/admin/index.html) and\n[API documentation](https://doc.arvados.org/api/index.html) is available at http://doc.arvados.org/\n\nIf you wish to build the Arvados documentation from a local git clone, see\n[doc/README.textile](doc/README.textile) for instructions.\n\n# Community\n\n[![Join the chat at https://gitter.im/arvados/community](https://badges.gitter.im/arvados/community.svg)](https://gitter.im/arvados/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)\n\nThe [Arvados community channel](https://gitter.im/arvados/community)\nchannel at [gitter.im](https://gitter.im) is available for live\ndiscussion and support.\n\nThe [Arvados developement channel](https://gitter.im/arvados/development)\nchannel at [gitter.im](https://gitter.im) is used to coordinate development.\n\nThe [Arvados user mailing list](http://lists.arvados.org/mailman/listinfo/arvados)\nis used to announce new versions and other news.\n\nAll participants are expected to abide by the [Arvados Code of Conduct](CODE_OF_CONDUCT.md).\n\n# Reporting bugs\n\n[Report an issue on GitHub](https://github.com/arvados/arvados/issues/new)\n\n# Development and Contributing\n\nSee [CONTRIBUTING](CONTRIBUTING.md) for information about Arvados development and how to contribute to the Arvados project.\n\n# Licensing\n\nArvados is Free Software. See [COPYING](COPYING) for information about the open source licenses used in Arvados.\n" }, { "path": "SECURITY.md", "content": "# Arvados Project Security Policy\n\n## Supported Versions\n\nThe Arvados project will issue security fixes by making point releases\non the current stable release series (X.Y.0, X.Y.1, X.Y.2, etc).\n\nThe most recent stable release version, along with release notes and\nupgrade notes documenting security fixes, can be found at these\nlocations:\n\nhttps://arvados.org/releases/\n\nhttps://doc.arvados.org/admin/upgrading.html\n\nThe Arvados project does not support versions older than the current\nstable release except by special arrangement (contact info@curii.com).\n\nRelease announcements, including notification of security fixes, are\nsent to the Arvados announcement list:\n\nhttps://lists.arvados.org//mailman/listinfo/arvados\n\n## Reporting Security Issues\n\nIf you believe you have found a security vulnerability in any Arvados-owned repository, please report it to us through coordinated disclosure.\n\n**Please do not report security vulnerabilities through public GitHub issues, discussions, or pull requests.**\n\nInstead, please send an email to dev@curii.com.\n\nPlease include as much of the information listed below as you can to help us better understand and resolve the issue:\n\n * The type of issue (e.g., remote code execution, SQL injection, or cross-site scripting)\n * Full paths of source file(s) related to the manifestation of the issue\n * The location of the affected source code (tag/branch/commit or direct URL)\n * Any special configuration required to reproduce the issue\n * Step-by-step instructions to reproduce the issue\n * Proof-of-concept or exploit code (if possible)\n * Impact of the issue, including how an attacker might exploit the issue\n\nThis information will help us triage your report more quickly.\n" }, { "path": "agpl-3.0.txt", "content": " GNU AFFERO GENERAL PUBLIC LICENSE\n Version 3, 19 November 2007\n\n Copyright (C) 2007 Free Software Foundation, Inc. \n Everyone is permitted to copy and distribute verbatim copies\n of this license document, but changing it is not allowed.\n\n Preamble\n\n The GNU Affero General Public License is a free, copyleft license for\nsoftware and other kinds of works, specifically designed to ensure\ncooperation with the community in the case of network server software.\n\n The licenses for most software and other practical works are designed\nto take away your freedom to share and change the works. By contrast,\nour General Public Licenses are intended to guarantee your freedom to\nshare and change all versions of a program--to make sure it remains free\nsoftware for all its users.\n\n When we speak of free software, we are referring to freedom, not\nprice. Our General Public Licenses are designed to make sure that you\nhave the freedom to distribute copies of free software (and charge for\nthem if you wish), that you receive source code or can get it if you\nwant it, that you can change the software or use pieces of it in new\nfree programs, and that you know you can do these things.\n\n Developers that use our General Public Licenses protect your rights\nwith two steps: (1) assert copyright on the software, and (2) offer\nyou this License which gives you legal permission to copy, distribute\nand/or modify the software.\n\n A secondary benefit of defending all users' freedom is that\nimprovements made in alternate versions of the program, if they\nreceive widespread use, become available for other developers to\nincorporate. Many developers of free software are heartened and\nencouraged by the resulting cooperation. However, in the case of\nsoftware used on network servers, this result may fail to come about.\nThe GNU General Public License permits making a modified version and\nletting the public access it on a server without ever releasing its\nsource code to the public.\n\n The GNU Affero General Public License is designed specifically to\nensure that, in such cases, the modified source code becomes available\nto the community. It requires the operator of a network server to\nprovide the source code of the modified version running there to the\nusers of that server. Therefore, public use of a modified version, on\na publicly accessible server, gives the public access to the source\ncode of the modified version.\n\n An older license, called the Affero General Public License and\npublished by Affero, was designed to accomplish similar goals. This is\na different license, not a version of the Affero GPL, but Affero has\nreleased a new version of the Affero GPL which permits relicensing under\nthis license.\n\n The precise terms and conditions for copying, distribution and\nmodification follow.\n\n TERMS AND CONDITIONS\n\n 0. Definitions.\n\n \"This License\" refers to version 3 of the GNU Affero General Public License.\n\n \"Copyright\" also means copyright-like laws that apply to other kinds of\nworks, such as semiconductor masks.\n\n \"The Program\" refers to any copyrightable work licensed under this\nLicense. Each licensee is addressed as \"you\". \"Licensees\" and\n\"recipients\" may be individuals or organizations.\n\n To \"modify\" a work means to copy from or adapt all or part of the work\nin a fashion requiring copyright permission, other than the making of an\nexact copy. The resulting work is called a \"modified version\" of the\nearlier work or a work \"based on\" the earlier work.\n\n A \"covered work\" means either the unmodified Program or a work based\non the Program.\n\n To \"propagate\" a work means to do anything with it that, without\npermission, would make you directly or secondarily liable for\ninfringement under applicable copyright law, except executing it on a\ncomputer or modifying a private copy. Propagation includes copying,\ndistribution (with or without modification), making available to the\npublic, and in some countries other activities as well.\n\n To \"convey\" a work means any kind of propagation that enables other\nparties to make or receive copies. Mere interaction with a user through\na computer network, with no transfer of a copy, is not conveying.\n\n An interactive user interface displays \"Appropriate Legal Notices\"\nto the extent that it includes a convenient and prominently visible\nfeature that (1) displays an appropriate copyright notice, and (2)\ntells the user that there is no warranty for the work (except to the\nextent that warranties are provided), that licensees may convey the\nwork under this License, and how to view a copy of this License. If\nthe interface presents a list of user commands or options, such as a\nmenu, a prominent item in the list meets this criterion.\n\n 1. Source Code.\n\n The \"source code\" for a work means the preferred form of the work\nfor making modifications to it. \"Object code\" means any non-source\nform of a work.\n\n A \"Standard Interface\" means an interface that either is an official\nstandard defined by a recognized standards body, or, in the case of\ninterfaces specified for a particular programming language, one that\nis widely used among developers working in that language.\n\n The \"System Libraries\" of an executable work include anything, other\nthan the work as a whole, that (a) is included in the normal form of\npackaging a Major Component, but which is not part of that Major\nComponent, and (b) serves only to enable use of the work with that\nMajor Component, or to implement a Standard Interface for which an\nimplementation is available to the public in source code form. A\n\"Major Component\", in this context, means a major essential component\n(kernel, window system, and so on) of the specific operating system\n(if any) on which the executable work runs, or a compiler used to\nproduce the work, or an object code interpreter used to run it.\n\n The \"Corresponding Source\" for a work in object code form means all\nthe source code needed to generate, install, and (for an executable\nwork) run the object code and to modify the work, including scripts to\ncontrol those activities. However, it does not include the work's\nSystem Libraries, or general-purpose tools or generally available free\nprograms which are used unmodified in performing those activities but\nwhich are not part of the work. For example, Corresponding Source\nincludes interface definition files associated with source files for\nthe work, and the source code for shared libraries and dynamically\nlinked subprograms that the work is specifically designed to require,\nsuch as by intimate data communication or control flow between those\nsubprograms and other parts of the work.\n\n The Corresponding Source need not include anything that users\ncan regenerate automatically from other parts of the Corresponding\nSource.\n\n The Corresponding Source for a work in source code form is that\nsame work.\n\n 2. Basic Permissions.\n\n All rights granted under this License are granted for the term of\ncopyright on the Program, and are irrevocable provided the stated\nconditions are met. This License explicitly affirms your unlimited\npermission to run the unmodified Program. The output from running a\ncovered work is covered by this License only if the output, given its\ncontent, constitutes a covered work. This License acknowledges your\nrights of fair use or other equivalent, as provided by copyright law.\n\n You may make, run and propagate covered works that you do not\nconvey, without conditions so long as your license otherwise remains\nin force. You may convey covered works to others for the sole purpose\nof having them make modifications exclusively for you, or provide you\nwith facilities for running those works, provided that you comply with\nthe terms of this License in conveying all material for which you do\nnot control copyright. Those thus making or running the covered works\nfor you must do so exclusively on your behalf, under your direction\nand control, on terms that prohibit them from making any copies of\nyour copyrighted material outside their relationship with you.\n\n Conveying under any other circumstances is permitted solely under\nthe conditions stated below. Sublicensing is not allowed; section 10\nmakes it unnecessary.\n\n 3. Protecting Users' Legal Rights From Anti-Circumvention Law.\n\n No covered work shall be deemed part of an effective technological\nmeasure under any applicable law fulfilling obligations under article\n11 of the WIPO copyright treaty adopted on 20 December 1996, or\nsimilar laws prohibiting or restricting circumvention of such\nmeasures.\n\n When you convey a covered work, you waive any legal power to forbid\ncircumvention of technological measures to the extent such circumvention\nis effected by exercising rights under this License with respect to\nthe covered work, and you disclaim any intention to limit operation or\nmodification of the work as a means of enforcing, against the work's\nusers, your or third parties' legal rights to forbid circumvention of\ntechnological measures.\n\n 4. Conveying Verbatim Copies.\n\n You may convey verbatim copies of the Program's source code as you\nreceive it, in any medium, provided that you conspicuously and\nappropriately publish on each copy an appropriate copyright notice;\nkeep intact all notices stating that this License and any\nnon-permissive terms added in accord with section 7 apply to the code;\nkeep intact all notices of the absence of any warranty; and give all\nrecipients a copy of this License along with the Program.\n\n You may charge any price or no price for each copy that you convey,\nand you may offer support or warranty protection for a fee.\n\n 5. Conveying Modified Source Versions.\n\n You may convey a work based on the Program, or the modifications to\nproduce it from the Program, in the form of source code under the\nterms of section 4, provided that you also meet all of these conditions:\n\n a) The work must carry prominent notices stating that you modified\n it, and giving a relevant date.\n\n b) The work must carry prominent notices stating that it is\n released under this License and any conditions added under section\n 7. This requirement modifies the requirement in section 4 to\n \"keep intact all notices\".\n\n c) You must license the entire work, as a whole, under this\n License to anyone who comes into possession of a copy. This\n License will therefore apply, along with any applicable section 7\n additional terms, to the whole of the work, and all its parts,\n regardless of how they are packaged. This License gives no\n permission to license the work in any other way, but it does not\n invalidate such permission if you have separately received it.\n\n d) If the work has interactive user interfaces, each must display\n Appropriate Legal Notices; however, if the Program has interactive\n interfaces that do not display Appropriate Legal Notices, your\n work need not make them do so.\n\n A compilation of a covered work with other separate and independent\nworks, which are not by their nature extensions of the covered work,\nand which are not combined with it such as to form a larger program,\nin or on a volume of a storage or distribution medium, is called an\n\"aggregate\" if the compilation and its resulting copyright are not\nused to limit the access or legal rights of the compilation's users\nbeyond what the individual works permit. Inclusion of a covered work\nin an aggregate does not cause this License to apply to the other\nparts of the aggregate.\n\n 6. Conveying Non-Source Forms.\n\n You may convey a covered work in object code form under the terms\nof sections 4 and 5, provided that you also convey the\nmachine-readable Corresponding Source under the terms of this License,\nin one of these ways:\n\n a) Convey the object code in, or embodied in, a physical product\n (including a physical distribution medium), accompanied by the\n Corresponding Source fixed on a durable physical medium\n customarily used for software interchange.\n\n b) Convey the object code in, or embodied in, a physical product\n (including a physical distribution medium), accompanied by a\n written offer, valid for at least three years and valid for as\n long as you offer spare parts or customer support for that product\n model, to give anyone who possesses the object code either (1) a\n copy of the Corresponding Source for all the software in the\n product that is covered by this License, on a durable physical\n medium customarily used for software interchange, for a price no\n more than your reasonable cost of physically performing this\n conveying of source, or (2) access to copy the\n Corresponding Source from a network server at no charge.\n\n c) Convey individual copies of the object code with a copy of the\n written offer to provide the Corresponding Source. This\n alternative is allowed only occasionally and noncommercially, and\n only if you received the object code with such an offer, in accord\n with subsection 6b.\n\n d) Convey the object code by offering access from a designated\n place (gratis or for a charge), and offer equivalent access to the\n Corresponding Source in the same way through the same place at no\n further charge. You need not require recipients to copy the\n Corresponding Source along with the object code. If the place to\n copy the object code is a network server, the Corresponding Source\n may be on a different server (operated by you or a third party)\n that supports equivalent copying facilities, provided you maintain\n clear directions next to the object code saying where to find the\n Corresponding Source. Regardless of what server hosts the\n Corresponding Source, you remain obligated to ensure that it is\n available for as long as needed to satisfy these requirements.\n\n e) Convey the object code using peer-to-peer transmission, provided\n you inform other peers where the object code and Corresponding\n Source of the work are being offered to the general public at no\n charge under subsection 6d.\n\n A separable portion of the object code, whose source code is excluded\nfrom the Corresponding Source as a System Library, need not be\nincluded in conveying the object code work.\n\n A \"User Product\" is either (1) a \"consumer product\", which means any\ntangible personal property which is normally used for personal, family,\nor household purposes, or (2) anything designed or sold for incorporation\ninto a dwelling. In determining whether a product is a consumer product,\ndoubtful cases shall be resolved in favor of coverage. For a particular\nproduct received by a particular user, \"normally used\" refers to a\ntypical or common use of that class of product, regardless of the status\nof the particular user or of the way in which the particular user\nactually uses, or expects or is expected to use, the product. A product\nis a consumer product regardless of whether the product has substantial\ncommercial, industrial or non-consumer uses, unless such uses represent\nthe only significant mode of use of the product.\n\n \"Installation Information\" for a User Product means any methods,\nprocedures, authorization keys, or other information required to install\nand execute modified versions of a covered work in that User Product from\na modified version of its Corresponding Source. The information must\nsuffice to ensure that the continued functioning of the modified object\ncode is in no case prevented or interfered with solely because\nmodification has been made.\n\n If you convey an object code work under this section in, or with, or\nspecifically for use in, a User Product, and the conveying occurs as\npart of a transaction in which the right of possession and use of the\nUser Product is transferred to the recipient in perpetuity or for a\nfixed term (regardless of how the transaction is characterized), the\nCorresponding Source conveyed under this section must be accompanied\nby the Installation Information. But this requirement does not apply\nif neither you nor any third party retains the ability to install\nmodified object code on the User Product (for example, the work has\nbeen installed in ROM).\n\n The requirement to provide Installation Information does not include a\nrequirement to continue to provide support service, warranty, or updates\nfor a work that has been modified or installed by the recipient, or for\nthe User Product in which it has been modified or installed. Access to a\nnetwork may be denied when the modification itself materially and\nadversely affects the operation of the network or violates the rules and\nprotocols for communication across the network.\n\n Corresponding Source conveyed, and Installation Information provided,\nin accord with this section must be in a format that is publicly\ndocumented (and with an implementation available to the public in\nsource code form), and must require no special password or key for\nunpacking, reading or copying.\n\n 7. Additional Terms.\n\n \"Additional permissions\" are terms that supplement the terms of this\nLicense by making exceptions from one or more of its conditions.\nAdditional permissions that are applicable to the entire Program shall\nbe treated as though they were included in this License, to the extent\nthat they are valid under applicable law. If additional permissions\napply only to part of the Program, that part may be used separately\nunder those permissions, but the entire Program remains governed by\nthis License without regard to the additional permissions.\n\n When you convey a copy of a covered work, you may at your option\nremove any additional permissions from that copy, or from any part of\nit. (Additional permissions may be written to require their own\nremoval in certain cases when you modify the work.) You may place\nadditional permissions on material, added by you to a covered work,\nfor which you have or can give appropriate copyright permission.\n\n Notwithstanding any other provision of this License, for material you\nadd to a covered work, you may (if authorized by the copyright holders of\nthat material) supplement the terms of this License with terms:\n\n a) Disclaiming warranty or limiting liability differently from the\n terms of sections 15 and 16 of this License; or\n\n b) Requiring preservation of specified reasonable legal notices or\n author attributions in that material or in the Appropriate Legal\n Notices displayed by works containing it; or\n\n c) Prohibiting misrepresentation of the origin of that material, or\n requiring that modified versions of such material be marked in\n reasonable ways as different from the original version; or\n\n d) Limiting the use for publicity purposes of names of licensors or\n authors of the material; or\n\n e) Declining to grant rights under trademark law for use of some\n trade names, trademarks, or service marks; or\n\n f) Requiring indemnification of licensors and authors of that\n material by anyone who conveys the material (or modified versions of\n it) with contractual assumptions of liability to the recipient, for\n any liability that these contractual assumptions directly impose on\n those licensors and authors.\n\n All other non-permissive additional terms are considered \"further\nrestrictions\" within the meaning of section 10. If the Program as you\nreceived it, or any part of it, contains a notice stating that it is\ngoverned by this License along with a term that is a further\nrestriction, you may remove that term. If a license document contains\na further restriction but permits relicensing or conveying under this\nLicense, you may add to a covered work material governed by the terms\nof that license document, provided that the further restriction does\nnot survive such relicensing or conveying.\n\n If you add terms to a covered work in accord with this section, you\nmust place, in the relevant source files, a statement of the\nadditional terms that apply to those files, or a notice indicating\nwhere to find the applicable terms.\n\n Additional terms, permissive or non-permissive, may be stated in the\nform of a separately written license, or stated as exceptions;\nthe above requirements apply either way.\n\n 8. Termination.\n\n You may not propagate or modify a covered work except as expressly\nprovided under this License. Any attempt otherwise to propagate or\nmodify it is void, and will automatically terminate your rights under\nthis License (including any patent licenses granted under the third\nparagraph of section 11).\n\n However, if you cease all violation of this License, then your\nlicense from a particular copyright holder is reinstated (a)\nprovisionally, unless and until the copyright holder explicitly and\nfinally terminates your license, and (b) permanently, if the copyright\nholder fails to notify you of the violation by some reasonable means\nprior to 60 days after the cessation.\n\n Moreover, your license from a particular copyright holder is\nreinstated permanently if the copyright holder notifies you of the\nviolation by some reasonable means, this is the first time you have\nreceived notice of violation of this License (for any work) from that\ncopyright holder, and you cure the violation prior to 30 days after\nyour receipt of the notice.\n\n Termination of your rights under this section does not terminate the\nlicenses of parties who have received copies or rights from you under\nthis License. If your rights have been terminated and not permanently\nreinstated, you do not qualify to receive new licenses for the same\nmaterial under section 10.\n\n 9. Acceptance Not Required for Having Copies.\n\n You are not required to accept this License in order to receive or\nrun a copy of the Program. Ancillary propagation of a covered work\noccurring solely as a consequence of using peer-to-peer transmission\nto receive a copy likewise does not require acceptance. However,\nnothing other than this License grants you permission to propagate or\nmodify any covered work. These actions infringe copyright if you do\nnot accept this License. Therefore, by modifying or propagating a\ncovered work, you indicate your acceptance of this License to do so.\n\n 10. Automatic Licensing of Downstream Recipients.\n\n Each time you convey a covered work, the recipient automatically\nreceives a license from the original licensors, to run, modify and\npropagate that work, subject to this License. You are not responsible\nfor enforcing compliance by third parties with this License.\n\n An \"entity transaction\" is a transaction transferring control of an\norganization, or substantially all assets of one, or subdividing an\norganization, or merging organizations. If propagation of a covered\nwork results from an entity transaction, each party to that\ntransaction who receives a copy of the work also receives whatever\nlicenses to the work the party's predecessor in interest had or could\ngive under the previous paragraph, plus a right to possession of the\nCorresponding Source of the work from the predecessor in interest, if\nthe predecessor has it or can get it with reasonable efforts.\n\n You may not impose any further restrictions on the exercise of the\nrights granted or affirmed under this License. For example, you may\nnot impose a license fee, royalty, or other charge for exercise of\nrights granted under this License, and you may not initiate litigation\n(including a cross-claim or counterclaim in a lawsuit) alleging that\nany patent claim is infringed by making, using, selling, offering for\nsale, or importing the Program or any portion of it.\n\n 11. Patents.\n\n A \"contributor\" is a copyright holder who authorizes use under this\nLicense of the Program or a work on which the Program is based. The\nwork thus licensed is called the contributor's \"contributor version\".\n\n A contributor's \"essential patent claims\" are all patent claims\nowned or controlled by the contributor, whether already acquired or\nhereafter acquired, that would be infringed by some manner, permitted\nby this License, of making, using, or selling its contributor version,\nbut do not include claims that would be infringed only as a\nconsequence of further modification of the contributor version. For\npurposes of this definition, \"control\" includes the right to grant\npatent sublicenses in a manner consistent with the requirements of\nthis License.\n\n Each contributor grants you a non-exclusive, worldwide, royalty-free\npatent license under the contributor's essential patent claims, to\nmake, use, sell, offer for sale, import and otherwise run, modify and\npropagate the contents of its contributor version.\n\n In the following three paragraphs, a \"patent license\" is any express\nagreement or commitment, however denominated, not to enforce a patent\n(such as an express permission to practice a patent or covenant not to\nsue for patent infringement). To \"grant\" such a patent license to a\nparty means to make such an agreement or commitment not to enforce a\npatent against the party.\n\n If you convey a covered work, knowingly relying on a patent license,\nand the Corresponding Source of the work is not available for anyone\nto copy, free of charge and under the terms of this License, through a\npublicly available network server or other readily accessible means,\nthen you must either (1) cause the Corresponding Source to be so\navailable, or (2) arrange to deprive yourself of the benefit of the\npatent license for this particular work, or (3) arrange, in a manner\nconsistent with the requirements of this License, to extend the patent\nlicense to downstream recipients. \"Knowingly relying\" means you have\nactual knowledge that, but for the patent license, your conveying the\ncovered work in a country, or your recipient's use of the covered work\nin a country, would infringe one or more identifiable patents in that\ncountry that you have reason to believe are valid.\n\n If, pursuant to or in connection with a single transaction or\narrangement, you convey, or propagate by procuring conveyance of, a\ncovered work, and grant a patent license to some of the parties\nreceiving the covered work authorizing them to use, propagate, modify\nor convey a specific copy of the covered work, then the patent license\nyou grant is automatically extended to all recipients of the covered\nwork and works based on it.\n\n A patent license is \"discriminatory\" if it does not include within\nthe scope of its coverage, prohibits the exercise of, or is\nconditioned on the non-exercise of one or more of the rights that are\nspecifically granted under this License. You may not convey a covered\nwork if you are a party to an arrangement with a third party that is\nin the business of distributing software, under which you make payment\nto the third party based on the extent of your activity of conveying\nthe work, and under which the third party grants, to any of the\nparties who would receive the covered work from you, a discriminatory\npatent license (a) in connection with copies of the covered work\nconveyed by you (or copies made from those copies), or (b) primarily\nfor and in connection with specific products or compilations that\ncontain the covered work, unless you entered into that arrangement,\nor that patent license was granted, prior to 28 March 2007.\n\n Nothing in this License shall be construed as excluding or limiting\nany implied license or other defenses to infringement that may\notherwise be available to you under applicable patent law.\n\n 12. No Surrender of Others' Freedom.\n\n If conditions are imposed on you (whether by court order, agreement or\notherwise) that contradict the conditions of this License, they do not\nexcuse you from the conditions of this License. If you cannot convey a\ncovered work so as to satisfy simultaneously your obligations under this\nLicense and any other pertinent obligations, then as a consequence you may\nnot convey it at all. For example, if you agree to terms that obligate you\nto collect a royalty for further conveying from those to whom you convey\nthe Program, the only way you could satisfy both those terms and this\nLicense would be to refrain entirely from conveying the Program.\n\n 13. Remote Network Interaction; Use with the GNU General Public License.\n\n Notwithstanding any other provision of this License, if you modify the\nProgram, your modified version must prominently offer all users\ninteracting with it remotely through a computer network (if your version\nsupports such interaction) an opportunity to receive the Corresponding\nSource of your version by providing access to the Corresponding Source\nfrom a network server at no charge, through some standard or customary\nmeans of facilitating copying of software. This Corresponding Source\nshall include the Corresponding Source for any work covered by version 3\nof the GNU General Public License that is incorporated pursuant to the\nfollowing paragraph.\n\n Notwithstanding any other provision of this License, you have\npermission to link or combine any covered work with a work licensed\nunder version 3 of the GNU General Public License into a single\ncombined work, and to convey the resulting work. The terms of this\nLicense will continue to apply to the part which is the covered work,\nbut the work with which it is combined will remain governed by version\n3 of the GNU General Public License.\n\n 14. Revised Versions of this License.\n\n The Free Software Foundation may publish revised and/or new versions of\nthe GNU Affero General Public License from time to time. Such new versions\nwill be similar in spirit to the present version, but may differ in detail to\naddress new problems or concerns.\n\n Each version is given a distinguishing version number. If the\nProgram specifies that a certain numbered version of the GNU Affero General\nPublic License \"or any later version\" applies to it, you have the\noption of following the terms and conditions either of that numbered\nversion or of any later version published by the Free Software\nFoundation. If the Program does not specify a version number of the\nGNU Affero General Public License, you may choose any version ever published\nby the Free Software Foundation.\n\n If the Program specifies that a proxy can decide which future\nversions of the GNU Affero General Public License can be used, that proxy's\npublic statement of acceptance of a version permanently authorizes you\nto choose that version for the Program.\n\n Later license versions may give you additional or different\npermissions. However, no additional obligations are imposed on any\nauthor or copyright holder as a result of your choosing to follow a\nlater version.\n\n 15. Disclaimer of Warranty.\n\n THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY\nAPPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT\nHOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM \"AS IS\" WITHOUT WARRANTY\nOF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,\nTHE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR\nPURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM\nIS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF\nALL NECESSARY SERVICING, REPAIR OR CORRECTION.\n\n 16. Limitation of Liability.\n\n IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING\nWILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS\nTHE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY\nGENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE\nUSE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF\nDATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD\nPARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),\nEVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF\nSUCH DAMAGES.\n\n 17. Interpretation of Sections 15 and 16.\n\n If the disclaimer of warranty and limitation of liability provided\nabove cannot be given local legal effect according to their terms,\nreviewing courts shall apply local law that most closely approximates\nan absolute waiver of all civil liability in connection with the\nProgram, unless a warranty or assumption of liability accompanies a\ncopy of the Program in return for a fee.\n\n END OF TERMS AND CONDITIONS\n\n How to Apply These Terms to Your New Programs\n\n If you develop a new program, and you want it to be of the greatest\npossible use to the public, the best way to achieve this is to make it\nfree software which everyone can redistribute and change under these terms.\n\n To do so, attach the following notices to the program. It is safest\nto attach them to the start of each source file to most effectively\nstate the exclusion of warranty; and each file should have at least\nthe \"copyright\" line and a pointer to where the full notice is found.\n\n \n Copyright (C) \n\n This program is free software: you can redistribute it and/or modify\n it under the terms of the GNU Affero General Public License as published by\n the Free Software Foundation, either version 3 of the License, or\n (at your option) any later version.\n\n This program is distributed in the hope that it will be useful,\n but WITHOUT ANY WARRANTY; without even the implied warranty of\n MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n GNU Affero General Public License for more details.\n\n You should have received a copy of the GNU Affero General Public License\n along with this program. If not, see .\n\nAlso add information on how to contact you by electronic and paper mail.\n\n If your software can interact with users remotely through a computer\nnetwork, you should also make sure that it provides a way for users to\nget its source. For example, if your program is a web application, its\ninterface could display a \"Source\" link that leads users to an archive\nof the code. There are many ways you could offer source, and different\nsolutions will be better for different programs; see section 13 for the\nspecific requirements.\n\n You should also get your employer (if you work as a programmer) or school,\nif any, to sign a \"copyright disclaimer\" for the program, if necessary.\nFor more information on this, and how to apply and follow the GNU AGPL, see\n.\n" }, { "path": "apache-2.0.txt", "content": "\n Apache License\n Version 2.0, January 2004\n http://www.apache.org/licenses/\n\n TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n 1. Definitions.\n\n \"License\" shall mean the terms and conditions for use, reproduction,\n and distribution as defined by Sections 1 through 9 of this document.\n\n \"Licensor\" shall mean the copyright owner or entity authorized by\n the copyright owner that is granting the License.\n\n \"Legal Entity\" shall mean the union of the acting entity and all\n other entities that control, are controlled by, or are under common\n control with that entity. For the purposes of this definition,\n \"control\" means (i) the power, direct or indirect, to cause the\n direction or management of such entity, whether by contract or\n otherwise, or (ii) ownership of fifty percent (50%) or more of the\n outstanding shares, or (iii) beneficial ownership of such entity.\n\n \"You\" (or \"Your\") shall mean an individual or Legal Entity\n exercising permissions granted by this License.\n\n \"Source\" form shall mean the preferred form for making modifications,\n including but not limited to software source code, documentation\n source, and configuration files.\n\n \"Object\" form shall mean any form resulting from mechanical\n transformation or translation of a Source form, including but\n not limited to compiled object code, generated documentation,\n and conversions to other media types.\n\n \"Work\" shall mean the work of authorship, whether in Source or\n Object form, made available under the License, as indicated by a\n copyright notice that is included in or attached to the work\n (an example is provided in the Appendix below).\n\n \"Derivative Works\" shall mean any work, whether in Source or Object\n form, that is based on (or derived from) the Work and for which the\n editorial revisions, annotations, elaborations, or other modifications\n represent, as a whole, an original work of authorship. For the purposes\n of this License, Derivative Works shall not include works that remain\n separable from, or merely link (or bind by name) to the interfaces of,\n the Work and Derivative Works thereof.\n\n \"Contribution\" shall mean any work of authorship, including\n the original version of the Work and any modifications or additions\n to that Work or Derivative Works thereof, that is intentionally\n submitted to Licensor for inclusion in the Work by the copyright owner\n or by an individual or Legal Entity authorized to submit on behalf of\n the copyright owner. For the purposes of this definition, \"submitted\"\n means any form of electronic, verbal, or written communication sent\n to the Licensor or its representatives, including but not limited to\n communication on electronic mailing lists, source code control systems,\n and issue tracking systems that are managed by, or on behalf of, the\n Licensor for the purpose of discussing and improving the Work, but\n excluding communication that is conspicuously marked or otherwise\n designated in writing by the copyright owner as \"Not a Contribution.\"\n\n \"Contributor\" shall mean Licensor and any individual or Legal Entity\n on behalf of whom a Contribution has been received by Licensor and\n subsequently incorporated within the Work.\n\n 2. Grant of Copyright License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n copyright license to reproduce, prepare Derivative Works of,\n publicly display, publicly perform, sublicense, and distribute the\n Work and such Derivative Works in Source or Object form.\n\n 3. Grant of Patent License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n (except as stated in this section) patent license to make, have made,\n use, offer to sell, sell, import, and otherwise transfer the Work,\n where such license applies only to those patent claims licensable\n by such Contributor that are necessarily infringed by their\n Contribution(s) alone or by combination of their Contribution(s)\n with the Work to which such Contribution(s) was submitted. If You\n institute patent litigation against any entity (including a\n cross-claim or counterclaim in a lawsuit) alleging that the Work\n or a Contribution incorporated within the Work constitutes direct\n or contributory patent infringement, then any patent licenses\n granted to You under this License for that Work shall terminate\n as of the date such litigation is filed.\n\n 4. Redistribution. You may reproduce and distribute copies of the\n Work or Derivative Works thereof in any medium, with or without\n modifications, and in Source or Object form, provided that You\n meet the following conditions:\n\n (a) You must give any other recipients of the Work or\n Derivative Works a copy of this License; and\n\n (b) You must cause any modified files to carry prominent notices\n stating that You changed the files; and\n\n (c) You must retain, in the Source form of any Derivative Works\n that You distribute, all copyright, patent, trademark, and\n attribution notices from the Source form of the Work,\n excluding those notices that do not pertain to any part of\n the Derivative Works; and\n\n (d) If the Work includes a \"NOTICE\" text file as part of its\n distribution, then any Derivative Works that You distribute must\n include a readable copy of the attribution notices contained\n within such NOTICE file, excluding those notices that do not\n pertain to any part of the Derivative Works, in at least one\n of the following places: within a NOTICE text file distributed\n as part of the Derivative Works; within the Source form or\n documentation, if provided along with the Derivative Works; or,\n within a display generated by the Derivative Works, if and\n wherever such third-party notices normally appear. The contents\n of the NOTICE file are for informational purposes only and\n do not modify the License. You may add Your own attribution\n notices within Derivative Works that You distribute, alongside\n or as an addendum to the NOTICE text from the Work, provided\n that such additional attribution notices cannot be construed\n as modifying the License.\n\n You may add Your own copyright statement to Your modifications and\n may provide additional or different license terms and conditions\n for use, reproduction, or distribution of Your modifications, or\n for any such Derivative Works as a whole, provided Your use,\n reproduction, and distribution of the Work otherwise complies with\n the conditions stated in this License.\n\n 5. Submission of Contributions. Unless You explicitly state otherwise,\n any Contribution intentionally submitted for inclusion in the Work\n by You to the Licensor shall be under the terms and conditions of\n this License, without any additional terms or conditions.\n Notwithstanding the above, nothing herein shall supersede or modify\n the terms of any separate license agreement you may have executed\n with Licensor regarding such Contributions.\n\n 6. Trademarks. This License does not grant permission to use the trade\n names, trademarks, service marks, or product names of the Licensor,\n except as required for reasonable and customary use in describing the\n origin of the Work and reproducing the content of the NOTICE file.\n\n 7. Disclaimer of Warranty. Unless required by applicable law or\n agreed to in writing, Licensor provides the Work (and each\n Contributor provides its Contributions) on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n implied, including, without limitation, any warranties or conditions\n of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n PARTICULAR PURPOSE. You are solely responsible for determining the\n appropriateness of using or redistributing the Work and assume any\n risks associated with Your exercise of permissions under this License.\n\n 8. Limitation of Liability. In no event and under no legal theory,\n whether in tort (including negligence), contract, or otherwise,\n unless required by applicable law (such as deliberate and grossly\n negligent acts) or agreed to in writing, shall any Contributor be\n liable to You for damages, including any direct, indirect, special,\n incidental, or consequential damages of any character arising as a\n result of this License or out of the use or inability to use the\n Work (including but not limited to damages for loss of goodwill,\n work stoppage, computer failure or malfunction, or any and all\n other commercial damages or losses), even if such Contributor\n has been advised of the possibility of such damages.\n\n 9. Accepting Warranty or Additional Liability. While redistributing\n the Work or Derivative Works thereof, You may choose to offer,\n and charge a fee for, acceptance of support, warranty, indemnity,\n or other liability obligations and/or rights consistent with this\n License. However, in accepting such obligations, You may act only\n on Your own behalf and on Your sole responsibility, not on behalf\n of any other Contributor, and only if You agree to indemnify,\n defend, and hold each Contributor harmless for any liability\n incurred by, or claims asserted against, such Contributor by reason\n of your accepting any such warranty or additional liability.\n\n END OF TERMS AND CONDITIONS\n\n APPENDIX: How to apply the Apache License to your work.\n\n To apply the Apache License to your work, attach the following\n boilerplate notice, with the fields enclosed by brackets \"[]\"\n replaced with your own identifying information. (Don't include\n the brackets!) The text should be enclosed in the appropriate\n comment syntax for the file format. We also recommend that a\n file or class name and description of purpose be included on the\n same \"printed page\" as the copyright notice for easier\n identification within third-party archives.\n\n Copyright [yyyy] [name of copyright owner]\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n" }, { "path": "build/README", "content": "Prerequisites\n=============\n\nIn order to build packages, you will need:\n\n* Ansible installed following the instructions in `tools/ansible/README.md`\n* `ansible-galaxy` and `ansible-playbook` in `$PATH` (e.g., by activating\n your Ansible virtualenv, or having symlinks to those commands inside it)\n* Docker installed\n* permission to run Docker commands\n* the `WORKSPACE` environment variable set to the absolute path of an\n Arvados Git work tree\n\nThe Ansible playbook `tools/ansible/install-dev-tools.yml` can install all\nof these prerequisites except the last.\n\nQuickstart\n==========\n\nBuild and test all the packages for a distribution on your architecture by\nrunning:\n\n ./run-build-test-packages-one-target.sh --target DISTRO\n\nThis will build package build and test Docker images for the named target\ndistribution, build all packages in a build container, then test all\npackages in a test container.\n\nLimit the build to a single package by adding the `--only-build\nPACKAGE_NAME` option. This is helpful when a build is mostly in good shape\nand you're tracking down last bugs in one or two packages.\n\nGet more verbose output by adding the `--debug` option.\n\nBy default the script avoids rebuilding or retesting packages that it\ndetects have already been done in past runs. You can force the script to\nrebuild or retest package(s) with the `--force-build` and `--force-test`\noptions, respectively.\n\nRun the script with `--help` for more information about other options.\n\nScripts in this directory\n=========================\n\nrun-tests.sh Run unit and integration test suite.\n\nrun-build-test-packages-one-target.sh Entry point, wraps\n run-build-packages-one-target.sh to\n perform package building and testing\n inside Docker.\n\nrun-build-packages-one-target.sh Build packages for one target inside Docker.\n\nrun-build-packages-all-targets.sh Run run-build-packages-one-target.sh\n for every target.\n\nrun-build-packages.sh Actually build packages. Intended to run\n inside Docker container with proper\n build environment.\n\nrun-build-packages-python-and-ruby.sh Build Python and Ruby packages suitable\n for upload to PyPi and Rubygems.\n\nrun-library.sh A library of functions shared by the\n various scripts in this\n directory.\n\nbuild_docker_image.py Build a Docker image from Arvados\n source components\n\nAdding a new target\n===================\n\nIn order to build packages on a new distribution, you MUST:\n\n* Define containers to build the package build and test Docker images in\n `tools/ansible/files/development-docker-images.yml`.\n* Create `package-testing/test-packages-TARGET.sh`, ideally by making it a\n symlink to `FORMAT-common-test-packages.sh`.\n* Update the package download code near the bottom of `test_package_presence`\n in `run-library.sh` so it can download packages for the new distribution.\n\nOf course, any part of our package build or test infrastructure may need to\nbe updated to accommodate the process for new distributions. If you're\nhaving trouble building lots of packages, consider grepping these build\nscripts for the identifier of the closest working target, and see if you may\nneed to add branches or similar hooks for your target. If you're having\ntrouble building specific packages, consider doing the same for those\npackages' `fpm-info.sh` files.\n" }, { "path": "build/build_docker_image.py", "content": "#!/usr/bin/env python3\n# build_docker_image.py - Build a Docker image with Python source packages\n#\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n#\n# Requires you have requirements.build.txt installed\n\nimport argparse\nimport logging\nimport os\nimport re\nimport runpy\nimport shlex\nimport shutil\nimport subprocess\nimport sys\nimport tempfile\n\nfrom pathlib import Path\n\nlogger = logging.getLogger('build_docker_image')\n_null_loghandler = logging.NullHandler()\nlogger.addHandler(_null_loghandler)\n\ndef _log_cmd(level, msg, *args):\n *args, cmd = args\n if logger.isEnabledFor(level):\n logger.log(level, f'{msg}: %s', *args, ' '.join(shlex.quote(s) for s in cmd))\n\n\ndef _log_and_run(cmd, *, level=logging.DEBUG, check=True, **kwargs):\n _log_cmd(level, \"running command\", cmd)\n return subprocess.run(cmd, check=check, **kwargs)\n\n\nclass OptionError(ValueError):\n pass\n\n\nclass DockerImage:\n _BUILD_ARGS = {}\n _REGISTRY = {}\n\n @classmethod\n def register(cls, subcls):\n cls._REGISTRY[subcls.NAME] = subcls\n pre_name, _, shortname = subcls.NAME.rpartition('/')\n if pre_name == 'arvados':\n cls._REGISTRY[shortname] = subcls\n return subcls\n\n @classmethod\n def build_from_args(cls, args):\n try:\n subcls = cls._REGISTRY[args.docker_image]\n except KeyError:\n raise OptionError(f\"unrecognized Docker image {args.docker_image!r}\") from None\n else:\n return subcls(args)\n\n def __init__(self, args):\n self.extra_args = args.extra_args\n self.workspace = args.workspace\n if args.tag is not None:\n self.tag = args.tag\n elif version := (args.version or self.dev_version()):\n self.tag = f'{self.NAME}:{version}'\n else:\n self.tag = None\n\n def __enter__(self):\n tmpname = self.NAME.replace('/', '-')\n self.context_dir = Path(tempfile.mkdtemp(prefix=f'{tmpname}.'))\n return self\n\n def __exit__(self, exc_type, exc_value, exc_tb):\n shutil.rmtree(self.context_dir, ignore_errors=True)\n del self.context_dir\n\n def build_docker_image(self):\n logger.info(\"building Docker image %s\", self.tag or self.NAME)\n cmd = ['docker', 'image', 'build']\n cmd.extend(\n f'--build-arg={key}={val}'\n for key, val in self._BUILD_ARGS.items()\n )\n cmd.append(f'--file={self.workspace / self.DOCKERFILE_PATH}')\n if self.tag is not None:\n cmd.append(f'--tag={self.tag}')\n cmd.append(str(self.context_dir))\n return _log_and_run(cmd)\n\n def dev_version(self):\n return None\n\n\nclass PythonVenvImage(DockerImage):\n DOCKERFILE_PATH = 'build/docker/python-venv.Dockerfile'\n _EXTRAS = {}\n _TEST_COMMAND = None\n\n def __init__(self, args):\n arv_vars = runpy.run_path(args.workspace / 'sdk/python/arvados_version.py')\n self.arv_pymod = arv_vars['ARVADOS_PYTHON_MODULES'][self._PACKAGE_NAME]\n super().__init__(args)\n\n def dev_version(self):\n return self.arv_pymod.get_version(self.workspace / self.arv_pymod.src_path)\n\n def build_python_wheel(self, src_dir):\n logger.info(\"building Python wheel at %s\", src_dir)\n cmd = [sys.executable, '-m', 'build', '--outdir', str(self.context_dir)]\n return _log_and_run(cmd, cwd=src_dir, umask=0o022)\n\n def build_requirements(self):\n with (self.context_dir / 'requirements.txt').open('w') as requirements_file:\n for whl_path in self.context_dir.glob('*.whl'):\n name, _, _ = whl_path.stem.partition('-')\n try:\n name += f' [{self._EXTRAS[name]}]'\n except KeyError:\n pass\n whl_uri = Path('/usr/local/src', whl_path.name).as_uri()\n print(name, '@', whl_uri, file=requirements_file)\n\n def build_docker_image(self):\n for path in self.extra_args:\n self.build_python_wheel(path)\n for dep in self.arv_pymod.dependencies:\n self.build_python_wheel(self.workspace / dep.src_path)\n self.build_python_wheel(self.workspace / self.arv_pymod.src_path)\n self.build_requirements()\n result = super().build_docker_image()\n if self.tag and self._TEST_COMMAND:\n _log_and_run(\n ['docker', 'run', '--rm', '--tty', self.tag] + self._TEST_COMMAND,\n stdout=subprocess.DEVNULL,\n )\n return result\n\n\n@DockerImage.register\nclass ClusterActivityImage(PythonVenvImage):\n NAME = 'arvados/cluster-activity'\n _BUILD_ARGS = {\n 'APT_PKGLIST': 'libcurl4',\n 'OLD_PKGNAME': 'python3-arvados-cluster-activity',\n }\n _EXTRAS = {\n 'arvados_cluster_activity': 'prometheus',\n }\n _PACKAGE_NAME = 'arvados-cluster-activity'\n _TEST_COMMAND = ['arv-cluster-activity', '--version']\n\n\n@DockerImage.register\nclass JobsImage(PythonVenvImage):\n NAME = 'arvados/jobs'\n _BUILD_ARGS = {\n 'APT_PKGLIST': 'libcurl4 nodejs',\n 'OLD_PKGNAME': 'python3-arvados-cwl-runner',\n }\n _PACKAGE_NAME = 'arvados-cwl-runner'\n _TEST_COMMAND = ['arvados-cwl-runner', '--version']\n\n\nclass Environments:\n @staticmethod\n def production(args):\n if args.version is None:\n raise OptionError(\n \"$ARVADOS_BUILDING_VERSION must be set to build production images\"\n )\n\n @staticmethod\n def development(args):\n return\n\n _ARG_MAP = {\n 'dev': development,\n 'devel': development,\n 'development': development,\n 'prod': production,\n 'production': production,\n }\n\n @classmethod\n def parse_argument(cls, s):\n try:\n return cls._ARG_MAP[s.lower()]\n except KeyError:\n raise ValueError(f\"unrecognized environment {s!r}\")\n\n\nclass UploadActions:\n @staticmethod\n def to_arvados(tag):\n logger.info(\"uploading Docker image %s to Arvados\", tag)\n name, _, version = tag.rpartition(':')\n if name:\n cmd = ['arv-keepdocker', name, version]\n else:\n cmd = ['arv-keepdocker', tag]\n return _log_and_run(cmd)\n\n @staticmethod\n def to_docker_hub(tag):\n logger.info(\"uploading Docker image %s to Docker Hub\", tag)\n cmd = ['docker', 'push', tag]\n for tries_left in range(4, -1, -1):\n try:\n docker_push = _log_and_run(cmd)\n except subprocess.CalledProcessError:\n if tries_left == 0:\n raise\n else:\n break\n return docker_push\n\n _ARG_MAP = {\n 'arv-keepdocker': to_arvados,\n 'arvados': to_arvados,\n 'docker': to_docker_hub,\n 'docker_hub': to_docker_hub,\n 'dockerhub': to_docker_hub,\n 'keepdocker': to_arvados,\n }\n\n @classmethod\n def parse_argument(cls, s):\n try:\n return cls._ARG_MAP[s.lower()]\n except KeyError:\n raise ValueError(f\"unrecognized upload method {s!r}\")\n\n\nclass ArgumentParser(argparse.ArgumentParser):\n def __init__(self):\n super().__init__(\n prog='build_docker_image.py',\n usage='%(prog)s [options ...] IMAGE_NAME [source directory ...]',\n )\n # We put environment variables for the tool in the args so the rest\n # of the program has a single place to access parameters.\n env_workspace = os.environ.get('WORKSPACE')\n if version := os.environ.get('ARVADOS_BUILDING_VERSION'):\n version = re.sub(r'~(dev[0-9])', r'.\\1', version, 1)\n version = re.sub(r'~(a|b|rc)([0-9])', r'\\1\\2', version, 1)\n self.set_defaults(\n version=version,\n workspace=Path(env_workspace) if env_workspace else None,\n )\n\n self.add_argument(\n '--environment',\n type=Environments.parse_argument,\n default=Environments.production,\n help=\"\"\"One of `development` or `production`.\nYour build settings will use defaults and be validated based on this setting.\nDefault is `production` because it's the strictest.\n\"\"\")\n\n self.add_argument(\n '--loglevel',\n type=self._parse_loglevel,\n default=logging.WARNING,\n help=\"\"\"Log level to use, like `debug`, `info`, `warning`, or `error`\n\"\"\")\n\n self.add_argument(\n '--tag', '-t',\n help=\"\"\"Tag for the built Docker image.\nDefault is generated from the image name and build version.\n\"\"\")\n\n self.add_argument(\n '--upload-to',\n type=UploadActions.parse_argument,\n help=\"\"\"After successfully building the Docker image, upload it to\nthis destination. Choices are `arvados` or `docker_hub`. Both require\ncredentials in place to work.\n\"\"\")\n\n self.add_argument(\n 'docker_image',\n metavar='IMAGE_NAME',\n choices=sorted(DockerImage._REGISTRY),\n help=\"\"\"Docker image to build.\nSupported images are: %(choices)s.\n\"\"\")\n\n self.add_argument(\n 'extra_args',\n metavar='SOURCE_DIR',\n type=Path,\n nargs=argparse.ZERO_OR_MORE,\n default=[],\n help=\"\"\"Before building the Docker image, the tool will build a\nPython wheel from each source directory and add it to the Docker build context.\nYou can use this during testing to install specific development versions of\ndependencies.\n\"\"\")\n\n def _parse_loglevel(self, s):\n try:\n return logging.getLevelNamesMapping()[s.upper()]\n except KeyError:\n raise ValueError(f\"unrecognized logging level {s!r}\")\n\n\ndef main(args):\n if not isinstance(args, argparse.Namespace):\n args = ArgumentParser().parse_args(args)\n if args.workspace is None:\n raise OptionError(\"$WORKSPACE must be set to the Arvados source directory\")\n args.environment(args)\n docker_image = DockerImage.build_from_args(args)\n if args.upload_to and not docker_image.tag:\n raise OptionError(\"cannot upload a Docker image without a tag\")\n with docker_image:\n docker_image.build_docker_image()\n if args.upload_to:\n args.upload_to(docker_image.tag)\n return os.EX_OK\n\n\nif __name__ == '__main__':\n argparser = ArgumentParser()\n _args = argparser.parse_args()\n logging.basicConfig(\n format=f'{logger.name}: %(levelname)s: %(message)s',\n level=_args.loglevel,\n )\n try:\n returncode = main(_args)\n except OptionError as err:\n argparser.error(err.args[0])\n returncode = 2\n except subprocess.CalledProcessError as err:\n _log_cmd(\n logging.ERROR,\n \"command failed with exit code %s\",\n err.returncode,\n err.cmd,\n )\n returncode = err.returncode\n exit(returncode)\n" }, { "path": "build/check-copyright-notices", "content": "#!/bin/bash\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\nset -e\n\nfix=false\nwhile [[ \"${@}\" != \"\" ]]\ndo\n arg=${1}\n shift\n case ${arg} in\n --help)\n cat <&2 \"Unrecognized argument '${arg}'. Try $0 --help\"\n exit 2\n ;;\n esac\ndone\n\nfixer() {\n want=\"${want}\" perl -pi~ - \"${1}\" <<'EOF'\nBEGIN { undef $/ }\ns{^((\\#\\!.*?\\n|\\n*---\\n.*?\\n\\.\\.\\.\\n|<\\?xml.*?>\\n)\\n?)?}{${2}$ENV{want}\\n\\n}ms\nEOF\n}\n\nIFS=$'\\n' read -a ignores -r -d $'\\000' <.licenseignore || true\nresult=0\n\ncoproc git ls-files -z ${@} \"\n ;;\n *.liquid)\n fixer=fixer\n cs=$'{% comment %}\\n'\n cc=\"\"\n ce=$'\\n{% endcomment %}'\n grepAfter=3\n grepBefore=1\n ;;\n *.textile)\n fixer=fixer\n cs=\"###. \"\n cc=\"....\"\n ce=\n ;;\n *.css)\n fixer=fixer\n cs=\"/* \"\n cc=\"\"\n ce=\" */\"\n ;;\n *.coffee)\n fixer=fixer\n cs=\"### \"\n cc=\"\"\n ce=\" ###\"\n ;;\n *.go | *.scss | *.java | *.js)\n fixer=fixer\n cc=\"//\"\n ;;\n *.sql)\n fixer=fixer\n cc=\"--\"\n ;;\n *.html | *.svg)\n fixer=fixer\n cs=\"\"\n ;;\n *)\n cc=\"#\"\n hashbang=$(head -n1 ${fnm})\n if [[ ${hashbang} = \"#!/bin/sh\" ]] || [[ ${hashbang} = \"#!/bin/bash\" ]]\n then\n fixer=fixer\n fi\n ;;\n esac\n wantGPL=\"${cs:-${cc}${cc:+ }}Copyright (C) The Arvados Authors. All rights reserved.\n${cc}\n${cc}${cc:+ }SPDX-License-Identifier: AGPL-3.0${ce}\"\n wantApache=\"${cs:-${cc}${cc:+ }}Copyright (C) The Arvados Authors. All rights reserved.\n${cc}\n${cc}${cc:+ }SPDX-License-Identifier: Apache-2.0${ce}\"\n wantBYSA=\"${cs:-${cc}${cc:+ }}Copyright (C) The Arvados Authors. All rights reserved.\n${cc}\n${cc}${cc:+ }SPDX-License-Identifier: CC-BY-SA-3.0${ce}\"\n wantBYSAmd=\"[comment]: # (Copyright © The Arvados Authors. All rights reserved.)\n[comment]: # ()\n[comment]: # (SPDX-License-Identifier: CC-BY-SA-3.0)\"\n found=$(head -n20 \"$fnm\" | egrep -A${grepAfter} -B${grepBefore} 'Copyright.*All rights reserved.' || true)\n case ${fnm} in\n Makefile | build/* | lib/* | tools/* | apps/* | services/* | sdk/cli/bin/crunch-job)\n want=${wantGPL}\n ;;\n crunch_scripts/* | docker/* | sdk/*)\n want=${wantApache}\n ;;\n doc/*)\n want=${wantBYSA}\n ;;\n README.md)\n want=${wantBYSAmd}\n ;;\n *)\n want=\n ;;\n esac\n case \"$found\" in\n \"$wantGPL\")\n ;;\n \"$wantApache\")\n ;;\n \"$wantBYSA\")\n ;;\n \"$wantBYSAmd\")\n ;;\n \"\")\n if [[ -z ${found} ]] && [[ -n ${want} ]] && [[ $fix = true ]] && [[ $fixer != \"\" ]]\n then\n ${fixer} ${fnm}\n else\n echo \"missing copyright notice: $fnm\"\n result=1\n fi\n ;;\n *)\n echo \"nonstandard copyright notice: $fnm '${found}'\"\n result=1\n ;;\n esac\ndone <&${COPROC[0]}\nexit $result\n" }, { "path": "build/create-plot-data-from-log.sh", "content": "#!/bin/bash\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\nbuild=$1\nfile=$2\noutputdir=$3\n\nusage() {\n echo \"./$0 build_number file_to_parse output_dir\"\n echo \"this script will use the build output to generate *csv and *txt\"\n echo \"for jenkins plugin plot https://github.com/jenkinsci/plot-plugin/\"\n}\n\nif [ $# -ne 3 ]\nthen\n usage\n exit 1\nfi\n\nif [ ! -e $file ]\nthen\n usage\n echo \"$file doesn't exist! exiting\"\n exit 2\nfi\nif [ ! -w $outputdir ]\nthen\n usage\n echo \"$outputdir isn't writeable! exiting\"\n exit 3\nfi\n\n#------------------------------\n## MAXLINE is the amount of lines that will read after the pattern\n## is match (the logfile could be hundred thousands lines long).\n## 1000 should be safe enough to capture all the output of the individual test\nMAXLINES=1000\n\n## TODO: check $build and $file make sense\n\nfor test in \\\n test_Create_and_show_large_collection_with_manifest_text_of_20000000 \\\n test_Create,_show,_and_update_description_for_large_collection_with_manifest_text_of_100000 \\\n test_Create_one_large_collection_of_20000000_and_one_small_collection_of_10000_and_combine_them\ndo\n cleaned_test=$(echo $test | tr -d \",.:;/\")\n (zgrep -i -E -A$MAXLINES \"^[A-Za-z0-9]+Test: $test\" $file && echo \"----\") | tail -n +1 | tail --lines=+3|grep -B$MAXLINES -E \"^-*$\" -m1 > $outputdir/$cleaned_test-$build.txt\n result=$?\n if [ $result -eq 0 ]\n then\n echo processing $outputdir/$cleaned_test-$build.txt creating $outputdir/$cleaned_test.csv\n echo $(grep ^Completed $outputdir/$cleaned_test-$build.txt | perl -n -e '/^Completed (.*) in [0-9]+ms.*$/;print \"\".++$line.\"-$1,\";' | perl -p -e 's/,$//g'|tr \" \" \"_\" ) > $outputdir/$cleaned_test.csv\n echo $(grep ^Completed $outputdir/$cleaned_test-$build.txt | perl -n -e '/^Completed.*in ([0-9]+)ms.*$/;print \"$1,\";' | perl -p -e 's/,$//g' ) >> $outputdir/$cleaned_test.csv\n else\n echo \"$test was't found on $file\"\n cleaned_test=$(echo $test | tr -d \",.:;/\")\n > $outputdir/$cleaned_test.csv\n fi\ndone\n" }, { "path": "build/docker/python-venv.Dockerfile", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n#\n# Build this with `build/build_docker_image.py`\n\nFROM debian:bookworm-slim\n\nRUN apt-get update -q \\\n && DEBIAN_FRONTEND=noninteractive apt-get install -qy python3-venv \\\n && python3 -m venv /opt/arvados-py\n\n# The build script sets up our build context with all the Python source\n# packages to install.\nCOPY . /usr/local/src/\n\nRUN /opt/arvados-py/bin/pip install -qq --no-cache-dir --no-input \\\n -r /usr/local/src/requirements.txt\n\n### Stage 2\nFROM debian:bookworm-slim\nMAINTAINER Arvados Package Maintainers \nARG APT_PKGLIST\nARG OLD_PKGNAME=python3-arvados-python-client\n\nRUN apt-get update -q \\\n && DEBIAN_FRONTEND=noninteractive apt-get install -qy python3 $APT_PKGLIST\n\n# The symlinks provide path compatibility with old package-based images.\nRUN adduser --disabled-password --gecos 'Crunch execution user' crunch \\\n && install --directory --owner=crunch --group=crunch --mode=0700 \\\n /keep /tmp/crunch-src /tmp/crunch-job \\\n && ln -s /opt/arvados-py \"/usr/lib/$OLD_PKGNAME\"\n\nUSER crunch\nENV PATH=/opt/arvados-py/bin:/usr/local/bin:/usr/bin:/bin\n\nCOPY --from=0 /opt/arvados-py/ /opt/arvados-py/\n" }, { "path": "build/go-python-package-scripts/postinst", "content": "#!/bin/sh\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\nset -e\n\n# Detect rpm-based systems: the exit code of the following command is zero\n# on rpm-based systems\nif /usr/bin/rpm -q -f /usr/bin/rpm >/dev/null 2>&1; then\n # Red Hat (\"%{...}\" is interpolated at package build time)\n pkg=\"%{name}\"\n pkgtype=rpm\n prefix=\"${RPM_INSTALL_PREFIX}\"\nelse\n # Debian\n script=\"$(basename \"${0}\")\"\n pkg=\"${script%.postinst}\"\n pkgtype=deb\n prefix=/usr\nfi\n\ncase \"${pkgtype}-${1}\" in\n deb-configure | rpm-1)\n dest_dir=\"/lib/systemd/system\"\n if ! [ -d \"${dest_dir}\" ]; then\n exit 0\n fi\n\n # Find the unit file we need to install.\n unit_file=\"${pkg}.service\"\n for dir in \\\n \"${prefix}/share/doc/${pkg}\" \\\n \"${dest_dir}\"; do\n if [ -e \"${dir}/${unit_file}\" ]; then\n src_dir=\"${dir}\"\n break\n fi\n done\n if [ -z \"${src_dir}\" ]; then\n echo >&2 \"WARNING: postinst script did not find ${unit_file} anywhere.\"\n exit 0\n fi\n\n # Install/update the unit file if necessary.\n if [ \"${src_dir}\" != \"${dest_dir}\" ]; then\n cp \"${src_dir}/${unit_file}\" \"${dest_dir}/\" || exit 0\n fi\n\n # Enable service, and make sure systemd re-reads the unit\n # file, in case we changed it.\n if [ -e /run/systemd/system ]; then\n systemctl daemon-reload || true\n eval \"$(systemctl -p UnitFileState show \"${pkg}\")\"\n case \"${UnitFileState}\" in\n disabled)\n # Failing to enable or start the service is not a\n # package error, so don't let errors here\n # propagate up.\n systemctl enable \"${pkg}\" || true\n systemctl start \"${pkg}\" || true\n ;;\n enabled)\n systemctl reload-or-try-restart \"${pkg}\" || true\n ;;\n esac\n fi\n ;;\nesac\n" }, { "path": "build/go-python-package-scripts/prerm", "content": "#!/bin/sh\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\nset -e\n\n# Detect rpm-based systems: the exit code of the following command is zero\n# on rpm-based systems\nif /usr/bin/rpm -q -f /usr/bin/rpm >/dev/null 2>&1; then\n # Red Hat (\"%{...}\" is interpolated at package build time)\n pkg=\"%{name}\"\n pkgtype=rpm\n prefix=\"${RPM_INSTALL_PREFIX}\"\nelse\n # Debian\n script=\"$(basename \"${0}\")\"\n pkg=\"${script%.prerm}\"\n pkgtype=deb\n prefix=/usr\nfi\n\ncase \"${pkgtype}-${1}\" in\n deb-remove | rpm-0)\n if [ -e /run/systemd/system ]; then\n systemctl stop \"${pkg}\" || true\n systemctl disable \"${pkg}\" || true\n fi\n if [ -e \"${prefix}/share/doc/${pkg}/${pkg}.service\" ]; then\n # Unit files from Python packages get installed by\n # postinst so we have to remove them explicitly here.\n rm \"/lib/systemd/system/${pkg}/${pkg}.service\" || true\n fi\n ;;\nesac\n" }, { "path": "build/package-testing/common-test-packages.sh", "content": "#!/bin/sh\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\nset -eu\n\nFAIL=0\n\necho\n\nwhile read so && [ -n \"$so\" ]; do\n if ldd \"$so\" | grep \"not found\" ; then\n echo \"^^^ Missing while scanning $so ^^^\"\n FAIL=1\n fi\ndone < \"$ARV_PACKAGES_DIR/$1.before\"\n\napt-get $DASHQQ_UNLESS_DEBUG --allow-insecure-repositories update\n\napt-get $DASHQQ_UNLESS_DEBUG -y --allow-unauthenticated install \"$1\" >\"$STDOUT_IF_DEBUG\" 2>\"$STDERR_IF_DEBUG\" ||\n install_status=$?\n\ndpkg-query --show > \"$ARV_PACKAGES_DIR/$1.after\"\n\ndiff \"$ARV_PACKAGES_DIR/$1.before\" \"$ARV_PACKAGES_DIR/$1.after\" > \"$ARV_PACKAGES_DIR/$1.diff\" || true\n\nmkdir -p /tmp/opts\ncd /tmp/opts\n\nexport ARV_PACKAGES_DIR=\"/arvados/packages/$target\"\n\nif [[ -f $(ls -t \"$ARV_PACKAGES_DIR/$1\"_*.deb 2>/dev/null | head -n1) ]] ; then\n debpkg=$(ls -t \"$ARV_PACKAGES_DIR/$1\"_*.deb | head -n1)\nelse\n debpkg=$(ls -t \"$ARV_PACKAGES_DIR/processed/$1\"_*.deb | head -n1)\nfi\n\ndpkg-deb -x $debpkg .\n\nif [[ \"$DEBUG\" != \"0\" ]]; then\n find -type f -name '*.so' | while read so; do\n printf \"\\n== Package dependencies for %s ==\\n\" \"$so\"\n # dpkg is not fully aware of merged-/usr systems: ldd may list a library\n # under /lib where dpkg thinks it's under /usr/lib, or vice versa.\n # awk constructs globs that we pass to `dpkg --search` to be flexible\n # about which version we find. This could potentially return multiple\n # results, but doing better probably requires restructuring this whole\n # code to find and report the best match across multiple dpkg queries.\n ldd \"$so\" \\\n | awk 'BEGIN { ORS=\"\\0\" } ($3 ~ /^\\//) {print \"*\" $3}' \\\n | sort --unique --zero-terminated \\\n | xargs -0 --no-run-if-empty dpkg --search \\\n | cut -d: -f1 \\\n | sort --unique\n done\nfi\n\ncase \"${install_status:-0}-$1\" in\n 0-* | 100-arvados-api-server )\n exec /jenkins/package-testing/common-test-packages.sh \"$1\"\n ;;\n *)\n exit \"$install_status\"\n ;;\nesac\n" }, { "path": "build/package-testing/rpm-common-test-packages.sh", "content": "#!/bin/bash\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\nset -eu\n\n# Set up\nDEBUG=${ARVADOS_DEBUG:-0}\nSTDOUT_IF_DEBUG=/dev/null\nSTDERR_IF_DEBUG=/dev/null\nif [[ \"$DEBUG\" != \"0\" ]]; then\n STDOUT_IF_DEBUG=/dev/stdout\n STDERR_IF_DEBUG=/dev/stderr\nfi\n\ntarget=\"$(basename \"$0\" .sh)\"\ntarget=\"${target##*-}\"\n\nmicrodnf --assumeyes clean all\ntouch /var/lib/rpm/*\n\nexport ARV_PACKAGES_DIR=\"/arvados/packages/$target\"\n\nrpm -qa | sort > \"$ARV_PACKAGES_DIR/$1.before\"\nmicrodnf --assumeyes install \"$1\" || install_status=\"$?\"\nrpm -qa | sort > \"$ARV_PACKAGES_DIR/$1.after\"\ndiff \"$ARV_PACKAGES_DIR/$1\".{before,after} >\"$ARV_PACKAGES_DIR/$1.diff\" || true\n\nmkdir -p /tmp/opts\ncd /tmp/opts\n\nrpm2cpio $(ls -t \"$ARV_PACKAGES_DIR/$1\"-*.rpm | head -n1) | cpio -idm 2>/dev/null\n\nif [[ \"$DEBUG\" != \"0\" ]]; then\n find -name '*.so' | while read so; do\n echo -e \"\\n== Packages dependencies for $so ==\"\n ldd \"$so\" \\\n | awk '($3 ~ /^\\//){print $3}' | sort -u | xargs rpm -qf | sort -u\n done\nfi\n\ncase \"${install_status:-0}-$1\" in\n 0-* )\n # Install other packages alongside to test for build id conflicts.\n # This can be removed after we have test-provision-rocky8, #21426.\n microdnf --assumeyes install arvados-client arvados-server python3-arvados-python-client\n ;;\n 1-arvados-api-server )\n ;;\n *)\n exit \"$install_status\"\n ;;\nesac\n\nexec /jenkins/package-testing/common-test-packages.sh \"$1\"\n" }, { "path": "build/package-testing/test-package-arvados-api-server.sh", "content": "#!/bin/sh\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\nset -e\n\nPACKAGE_NAME=arvados-api-server\nAPI_GEMS_LS=\"$(mktemp --tmpdir api-gems-XXXXXX.list)\"\ntrap 'rm -f \"$API_GEMS_LS\"' EXIT INT TERM QUIT\n\ncd \"/var/www/${PACKAGE_NAME%-server}\"\n\ncat_dropins() {\n cat /lib/systemd/system/arvados-railsapi.service.d/*.conf\n}\n\ncheck_gem_dirs() {\n local when=\"$1\"; shift\n env -C shared/vendor_bundle/ruby ls -1 >\"$API_GEMS_LS\"\n local ls_count=\"$(wc -l <\"$API_GEMS_LS\")\"\n if [ \"$ls_count\" = 1 ]; then\n return 0\n fi\n echo \"Package $PACKAGE_NAME FAILED: $ls_count gem directories created after $when:\" >&2\n case \"${ARVADOS_DEBUG:-0}\" in\n 0) cat \"$API_GEMS_LS\" >&2 ;;\n *) env -C shared/vendor_bundle/ruby find -maxdepth 3 -type d -ls >&2 ;;\n esac\n return 11\n}\n\nexpect_grep() {\n local expect_exit=\"$1\"; shift\n local actual_exit=0\n grep \"$@\" >/dev/null || actual_exit=$?\n if [ \"$actual_exit\" -eq \"$expect_exit\" ]; then\n return 0\n fi\n echo \"Package $PACKAGE_NAME FAILED: \\`grep\" \"$@\" \"\\` returned exit code $actual_exit\" >&2\n case \"$actual_exit\" in\n 0) return 1 ;;\n *) return \"$actual_exit\" ;;\n esac\n}\n\n# We intentionally don't hardcode a Bundler path here because other parts of our\n# infrastructure expect `bundle` to be available system-wide after installation.\n# After that infrastructure is fixed, this test can invoke `bundle` the same\n# way the postinst script does.\nenv -C current bundle list >\"$ARV_PACKAGES_DIR/$PACKAGE_NAME.gems\"\ncheck_gem_dirs \"initial install\"\n\ncase \"$TARGET\" in\n debian*|ubuntu*)\n cat_dropins | expect_grep 0 -x SupplementaryGroups=www-data\n ;;\n rocky*)\n cat_dropins | expect_grep 1 \"^SupplementaryGroups=\"\n microdnf --assumeyes install nginx\n microdnf --assumeyes reinstall \"$PACKAGE_NAME\" || test $? -eq 1\n check_gem_dirs \"package reinstall\"\n cat_dropins | expect_grep 0 -x SupplementaryGroups=nginx\n ;;\n *)\n echo \"$0: WARNING: Unknown target '$TARGET'.\" >&2\n ;;\nesac\n" }, { "path": "build/package-testing/test-package-arvados-client.sh", "content": "#!/bin/sh\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\nset -e\n\narvados-client -version >/dev/null\n" }, { "path": "build/package-testing/test-package-arvados-docker-cleaner.sh", "content": "#!/bin/sh\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\nset -e\n\narvados-docker-cleaner -h >/dev/null\n" }, { "path": "build/package-testing/test-package-python3-arvados-cwl-runner.sh", "content": "#!/bin/sh\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\nset -e\n\narvados-cwl-runner --version >/dev/null\n" }, { "path": "build/package-testing/test-package-python3-arvados-python-client.sh", "content": "#!/bin/sh\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\narv-put --version >/dev/null || exit\n\n. /usr/lib/python3-arvados-python-client/bin/activate\npython </dev/null\n" }, { "path": "build/package-testing/test-package-python3-python-arvados-fuse.sh", "content": "#!/bin/sh\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\nset -e\n\narv-mount --version\n" }, { "path": "build/pypkg_info.py", "content": "#!/usr/bin/env python3\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\"\"\"pypkg_info.py - Introspect installed Python packages\n\nThis tool can read metadata about any Python package installed in the current\nenvironment and report it out in various formats. We use this mainly to pass\ninformation through when building distribution packages.\n\"\"\"\n\nimport argparse\nimport enum\nimport importlib.metadata\nimport os\nimport sys\n\nfrom pathlib import PurePath\n\nclass RawFormat:\n def format_metadata(self, key, value):\n return value\n\n def format_path(self, path):\n return str(path)\n\n\nclass FPMFormat(RawFormat):\n PYTHON_METADATA_MAP = {\n 'summary': 'description',\n }\n\n def format_metadata(self, key, value):\n key = key.lower()\n key = self.PYTHON_METADATA_MAP.get(key, key)\n return f'--{key}={value}'\n\n\nclass Formats(enum.Enum):\n RAW = RawFormat\n FPM = FPMFormat\n\n @classmethod\n def from_arg(cls, arg):\n try:\n return cls[arg.upper()]\n except KeyError:\n raise ValueError(f\"unknown format {arg!r}\") from None\n\n\ndef report_binfiles(args):\n bin_names = [\n PurePath('bin', path.name)\n for pkg_name in args.package_names\n for path in importlib.metadata.distribution(pkg_name).files\n if path.parts[-3:-1] == ('..', 'bin')\n ]\n fmt = args.format.value().format_path\n return (fmt(path) for path in bin_names)\n\ndef report_metadata(args):\n dist = importlib.metadata.distribution(args.package_name)\n fmt = args.format.value().format_metadata\n for key in args.metadata_key:\n yield fmt(key, dist.metadata.get(key, ''))\n\ndef unescape_str(arg):\n arg = arg.replace('\\'', '\\\\\\'')\n return eval(f\"'''{arg}'''\", {})\n\ndef parse_arguments(arglist=None):\n parser = argparse.ArgumentParser()\n parser.set_defaults(action=None)\n format_names = ', '.join(fmt.name.lower() for fmt in Formats)\n parser.add_argument(\n '--format', '-f',\n choices=list(Formats),\n default=Formats.RAW,\n type=Formats.from_arg,\n help=f\"Output format. Choices are: {format_names}\",\n )\n parser.add_argument(\n '--delimiter', '-d',\n default='\\n',\n type=unescape_str,\n help=\"Line ending. Python backslash escapes are supported. Default newline.\",\n )\n subparsers = parser.add_subparsers()\n\n binfiles = subparsers.add_parser('binfiles')\n binfiles.set_defaults(action=report_binfiles)\n binfiles.add_argument(\n 'package_names',\n nargs=argparse.ONE_OR_MORE,\n )\n\n metadata = subparsers.add_parser('metadata')\n metadata.set_defaults(action=report_metadata)\n metadata.add_argument(\n 'package_name',\n )\n metadata.add_argument(\n 'metadata_key',\n nargs=argparse.ONE_OR_MORE,\n )\n\n args = parser.parse_args()\n if args.action is None:\n parser.error(\"subcommand is required\")\n return args\n\ndef main(arglist=None):\n args = parse_arguments(arglist)\n try:\n for line in args.action(args):\n print(line, end=args.delimiter)\n except importlib.metadata.PackageNotFoundError as error:\n print(f\"error: package not found: {error.args[0]}\", file=sys.stderr)\n return os.EX_NOTFOUND\n else:\n return os.EX_OK\n\nif __name__ == '__main__':\n exit(main())\n" }, { "path": "build/rails-package-scripts/README.md", "content": "[//]: # Copyright (C) The Arvados Authors. All rights reserved.\n[//]: #\n[//]: # SPDX-License-Identifier: AGPL-3.0\n\nWhen run-build-packages.sh builds a Rails package, it generates the package's pre/post-inst/rm scripts by concatenating `arvados-api-server.sh` to define common variables, then the actual step script. Especially when this infrastructure was shared with the old Rails Workbench, this seemed like the least worst option to share code between these files and packages. More advanced code generation would've been too much trouble to integrate into our build process at this time. Trying to inject portions of files into other files seemed error-prone and likely to introduce bugs to the end result.\n" }, { "path": "build/rails-package-scripts/arvados-api-server.sh", "content": "#!/bin/sh\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\n# This file declares variables common to all scripts for one Rails package.\n\nPACKAGE_NAME=arvados-api-server\nINSTALL_PATH=/var/www/arvados-api\nCONFIG_PATH=/etc/arvados/api\nDOC_URL=\"http://doc.arvados.org/install/install-api-server.html#configure\"\nRELEASE_PATH=$INSTALL_PATH/current\nRELEASE_CONFIG_PATH=$RELEASE_PATH/config\nSHARED_PATH=$INSTALL_PATH/shared\n" }, { "path": "build/rails-package-scripts/postinst.sh", "content": "#!/bin/sh\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\n# This code runs after package variable definitions.\n\nset -e\n\nfor DISTRO_FAMILY in $(. /etc/os-release && echo \"${ID:-} ${ID_LIKE:-}\"); do\n case \"$DISTRO_FAMILY\" in\n debian)\n RESETUP_CMD=\"dpkg-reconfigure $PACKAGE_NAME\"\n break ;;\n rhel)\n RESETUP_CMD=\"dnf reinstall $PACKAGE_NAME\"\n break ;;\n esac\ndone\nif [ -z \"$RESETUP_CMD\" ]; then\n echo \"$PACKAGE_NAME postinst skipped: don't recognize the distribution from /etc/os-release\" >&2\n exit 0\nfi\n# This will be set to a command path after we install the version we need.\nBUNDLE=\n\n# systemd_ctl is just \"systemctl if we booted with systemd, otherwise a noop.\"\n# This makes the package installable in Docker containers, albeit without any\n# service deployment.\nif [ -d /run/systemd/system ]; then\n systemd_ctl() { systemctl \"$@\"; }\nelse\n systemd_ctl() { true; }\nfi\n\nsystemd_quote() {\n if [ $# -ne 1 ]; then\n echo \"error: systemd_quote requires exactly one argument\" >&2\n return 2\n fi\n # See systemd.syntax(7) - Use double quotes with backslash escapes\n echo \"$1\" | sed -re 's/[\\\\\"]/\\\\\\0/g; s/^/\"/; s/$/\"/'\n}\n\nrun_and_report() {\n # Usage: run_and_report ACTION_MSG CMD\n # This is the usual wrapper that prints ACTION_MSG, runs CMD, then writes\n # a message about whether CMD succeeded or failed. Returns the exit code\n # of CMD.\n local action_message=\"$1\"; shift\n local retcode=0\n echo -n \"$action_message...\"\n if \"$@\"; then\n echo \" done.\"\n else\n retcode=$?\n echo \" failed.\"\n fi\n return $retcode\n}\n\nreport_not_ready() {\n local exitcode=\"$1\"; shift\n local reason=\"$1\"; shift\n local doc_url=\"${1:-}\"\n case \"$doc_url\" in\n http://* | https://* ) ;;\n /*) doc_url=\"https://doc.arvados.org${doc_url}\" ;;\n \\#*) doc_url=\"https://doc.arvados.org/install/install-api-server.html${doc_url}\" ;;\n *) doc_url=\"https://doc.arvados.org/install/${doc_url}\" ;;\n esac\n cat >&2 <\n\nAfter you do that, resume $PACKAGE_NAME setup by running:\n $RESETUP_CMD\nEOF\n exit \"${exitcode:-20}\"\n}\n\nsetup_confdirs() {\n local confdir confgrp\n case \"$WWW_OWNER\" in\n \"\") confgrp=root ;;\n *) confgrp=\"$WWW_OWNER\" ;;\n esac\n for confdir in \"$@\"; do\n if [ ! -d \"$confdir\" ]; then\n install -d -g \"$confgrp\" -m 0750 \"$confdir\"\n fi\n done\n}\n\nsetup_conffile() {\n # Usage: setup_conffile CONFFILE_PATH [SOURCE_PATH]\n # Both paths are relative to RELEASE_CONFIG_PATH.\n # This function will try to safely ensure that a symbolic link for\n # the configuration file points from RELEASE_CONFIG_PATH to CONFIG_PATH.\n # If SOURCE_PATH is given, this function will try to install that file as\n # the configuration file in CONFIG_PATH, and return 1 if the file in\n # CONFIG_PATH is unmodified from the source.\n local conffile_relpath=\"$1\"; shift\n local conffile_source=\"$1\"\n local release_conffile=\"$RELEASE_CONFIG_PATH/$conffile_relpath\"\n local etc_conffile=\"$CONFIG_PATH/$(basename \"$conffile_relpath\")\"\n\n # Note that -h can return true and -e will return false simultaneously\n # when the target is a dangling symlink. We're okay with that outcome,\n # so check -h first.\n if [ ! -h \"$release_conffile\" ]; then\n if [ ! -e \"$release_conffile\" ]; then\n ln -s \"$etc_conffile\" \"$release_conffile\"\n # If there's a config file in /var/www identical to the one in /etc,\n # overwrite it with a symlink after porting its permissions.\n elif cmp --quiet \"$release_conffile\" \"$etc_conffile\"; then\n local ownership=\"$(stat -c \"%u:%g\" \"$release_conffile\")\"\n local owning_group=\"${ownership#*:}\"\n if [ 0 != \"$owning_group\" ]; then\n chgrp \"$owning_group\" \"$CONFIG_PATH\" /etc/arvados\n fi\n chown \"$ownership\" \"$etc_conffile\"\n chmod --reference=\"$release_conffile\" \"$etc_conffile\"\n ln --force -s \"$etc_conffile\" \"$release_conffile\"\n fi\n fi\n\n if [ -n \"$conffile_source\" ]; then\n if [ ! -e \"$etc_conffile\" ]; then\n install -g \"$WWW_OWNER\" -m 0640 \\\n \"$RELEASE_CONFIG_PATH/$conffile_source\" \"$etc_conffile\"\n return 1\n # Even if $etc_conffile already existed, it might be unmodified from\n # the source. This is especially likely when a user installs, updates\n # database.yml, then reconfigures before they update application.yml.\n # Use cmp to be sure whether $etc_conffile is modified.\n elif cmp --quiet \"$RELEASE_CONFIG_PATH/$conffile_source\" \"$etc_conffile\"; then\n return 1\n fi\n fi\n}\n\nprepare_database() {\n # Prevent PostgreSQL from trying to page output\n unset PAGER\n DB_MIGRATE_STATUS=`\"$BUNDLE\" exec bin/rake db:migrate:status 2>&1 || true`\n if echo \"$DB_MIGRATE_STATUS\" | grep -qF 'Schema migrations table does not exist yet.'; then\n # The database exists, but the migrations table doesn't.\n run_and_report \"Setting up database\" \"$BUNDLE\" exec bin/rake db:schema:load db:seed\n elif echo \"$DB_MIGRATE_STATUS\" | grep -q '^database: '; then\n run_and_report \"Running db:migrate\" \"$BUNDLE\" exec bin/rake db:migrate db:seed\n elif echo \"$DB_MIGRATE_STATUS\" | grep -q 'database .* does not exist'; then\n run_and_report \"Running db:setup\" \"$BUNDLE\" exec bin/rake db:setup\n else\n # We don't have enough configuration to even check the database.\n return 1\n fi\n}\n\ncase \"$DISTRO_FAMILY\" in\n debian) WWW_OWNER=www-data ;;\n rhel) WWW_OWNER=\"$(id --group --name nginx || true)\" ;;\nesac\n\n# Before we do anything else, make sure some directories and files are in place\nif [ ! -e $SHARED_PATH/log ]; then mkdir -p $SHARED_PATH/log; fi\nif [ ! -e $RELEASE_PATH/tmp ]; then mkdir -p $RELEASE_PATH/tmp; fi\nif [ ! -e $RELEASE_PATH/log ]; then ln -s $SHARED_PATH/log $RELEASE_PATH/log; fi\nif [ ! -e $SHARED_PATH/log/production.log ]; then touch $SHARED_PATH/log/production.log; fi\n\ncd \"$RELEASE_PATH\"\nexport RAILS_ENV=production\n# Bundler behaves inconsistently when gems are available system-wide.\n# Avoid those bugs by starting with a GEM_HOME that *only* contains Bundler.\nexport GEM_HOME=\"$SHARED_PATH/bundler\"\nexport GEM_PATH=\"$GEM_HOME\"\n# We still need to set directory switches because RHEL configures `gem` with\n# built-in options that override the environment variables.\nrun_and_report \"Installing bundler\" gem install \\\n --bindir \"$GEM_HOME/bin\" \\\n --install-dir \"$GEM_HOME\" \\\n --version \"~> 2.5.0\" \\\n bundler\nBUNDLE=\"$GEM_HOME/bin/bundle\"\nrun_and_report \"Running bundle install\" \"$BUNDLE\" install --prefer-local --quiet\nrun_and_report \"Verifying bundle is complete\" \"$BUNDLE\" exec true\n# Some of our infrastructure expects `bundler` to be available system-wide\n# after installing arvados-api-server. Ensure that's the case.\n# TODO: Make the other infrastructure stop doing that, then delete this code.\nfor bcmd in bundle bundler; do\n if ! command -v \"$bcmd\" >/dev/null 2>&1; then\n cat >\"/usr/local/bin/$bcmd\" <&2\n exit 12\nfi\n\"$BUNDLE\" exec \"$passenger-config\" build-native-support\n# `passenger-config install-standalone-runtime` downloads an agent, but at\n# least with Passenger 6.0.23 (late 2024), that version tends to segfault.\n# Compiling our own is safer.\n\"$BUNDLE\" exec \"$passenger-config\" compile-agent --auto --optimize\n\"$BUNDLE\" exec \"$passenger-config\" install-standalone-runtime --auto --brief\n\necho -n \"Creating symlinks to configuration in $CONFIG_PATH ...\"\nsetup_confdirs /etc/arvados \"$CONFIG_PATH\"\nsetup_conffile environments/production.rb environments/production.rb.example \\\n || true\n# Rails 5.2 does not tolerate dangling symlinks in the initializers\n# directory, and this one can still be there, left over from a previous\n# version of the API server package.\nrm -f $RELEASE_PATH/config/initializers/omniauth.rb\necho \"... done.\"\n\necho -n \"Extending systemd unit configuration ...\"\nif [ -z \"$WWW_OWNER\" ]; then\n systemd_group=\"%N\"\nelse\n systemd_group=\"$(systemd_quote \"$WWW_OWNER\")\"\nfi\ninstall -d /lib/systemd/system/arvados-railsapi.service.d\n# The 20 prefix is chosen so most user overrides should come after, which\n# is what most admins will expect, but there's still space to put drop-ins\n# earlier.\ncat >/lib/systemd/system/arvados-railsapi.service.d/20-postinst.conf <= 20.3\n\n# Technically this shouldn't be required: the build process should get the\n# build requirements listed in pyproject.toml. But it's nice to have it\n# cached.\nsetuptools ~= 80.9\n\nwheel\n" }, { "path": "build/requirements.tests.txt", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n#\n# Python requirements for run-tests\n\n### Requirements for run-tests.sh itself\n-r requirements.build.txt\n\n# Required by sdk/python/tests/run_test_server.py\n# which is run directly by run-tests.sh\nPyYAML\n\n# yq is used by run-tests.sh directly and controller tests\nyq ~= 3.4\n\n### Requirements for Python tests generally\n# Required by older, unittest-style Python tests\n# Prefer using pytest.mark.parametrize in new tests\nparameterized\n\n# Our chosen Python testing tool\npytest\n\n### Requirements for individual tests\n# Run by CWL integration tests\ncwltest >= 2.5.20241122133319, < 3.0\n\n# Required to build Python SDK documentation\npdoc ~= 16.0\n\n# Used by controller and keep-web tests\ns3cmd\n" }, { "path": "build/run-build-packages-all-targets.sh", "content": "#!/bin/bash\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\nread -rd \"\\000\" helpmessage <\n Version to build (default:\n \\$ARVADOS_BUILDING_VERSION-\\$ARVADOS_BUILDING_ITERATION or\n 0.1.timestamp.commithash)\n\nWORKSPACE=path Path to the Arvados source tree to build packages from\n\nEOF\n\nif ! [[ -n \"$WORKSPACE\" ]]; then\n echo >&2 \"$helpmessage\"\n echo >&2\n echo >&2 \"Error: WORKSPACE environment variable not set\"\n echo >&2\n exit 1\nfi\n\nif ! [[ -d \"$WORKSPACE\" ]]; then\n echo >&2 \"$helpmessage\"\n echo >&2\n echo >&2 \"Error: $WORKSPACE is not a directory\"\n echo >&2\n exit 1\nfi\n\nset -e\n\nPARSEDOPTS=$(getopt --name \"$0\" --longoptions \\\n help,test-packages,debug,command:,only-test:,build-version: \\\n -- \"\" \"$@\")\nif [ $? -ne 0 ]; then\n exit 1\nfi\n\nCOMMAND=\nDEBUG=\nTEST_PACKAGES=\nONLY_TEST=\n\neval set -- \"$PARSEDOPTS\"\nwhile [ $# -gt 0 ]; do\n case \"$1\" in\n --help)\n echo >&2 \"$helpmessage\"\n echo >&2\n exit 1\n ;;\n --debug)\n DEBUG=\"--debug\"\n ;;\n --command)\n COMMAND=\"$2\"; shift\n ;;\n --test-packages)\n TEST_PACKAGES=\"--test-packages\"\n ;;\n --only-test)\n ONLY_TEST=\"$1 $2\"; shift\n ;;\n --build-version)\n ARVADOS_BUILDING_VERSION=\"$2\"; shift\n ;;\n --)\n if [ $# -gt 1 ]; then\n echo >&2 \"$0: unrecognized argument '$2'. Try: $0 --help\"\n exit 1\n fi\n ;;\n esac\n shift\ndone\n\ncd $(dirname $0)\n\nFINAL_EXITCODE=0\n\nfor pkgtest_path in package-testing/test-packages-*.sh; do\n target=\"$(basename \"${pkgtest_path##*-}\" .sh)\"\n if ./run-build-packages-one-target.sh --target \"$target\" --command \"$COMMAND\" --build-version \"$ARVADOS_BUILDING_VERSION\" $DEBUG $TEST_PACKAGES $ONLY_TEST ; then\n true\n else\n FINAL_EXITCODE=$?\n echo\n echo \"Build packages failed for $(basename $(dirname \"$dockerfile_path\"))\"\n echo\n fi\ndone\n\nif test $FINAL_EXITCODE != 0 ; then\n echo\n echo \"Build packages failed with code $FINAL_EXITCODE\" >&2\n echo\nfi\n\nexit $FINAL_EXITCODE\n" }, { "path": "build/run-build-packages-one-target.sh", "content": "#!/bin/bash\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\nread -rd \"\\000\" helpmessage < [options]\n\n--target \n Distribution to build packages for\n--command\n Build command to execute (default: use built-in Docker image command)\n--test-packages\n Run package install test script \"test-packages-[target].sh\"\n--debug\n Output debug information (default: false)\n--only-build \n Build only a specific package\n--only-test \n Test only a specific package\n--force-build\n Build even if the package exists upstream or if it has already been\n built locally\n--force-test\n Test even if there is no new untested package\n--build-version \n Version to build (default:\n \\$ARVADOS_BUILDING_VERSION-\\$ARVADOS_BUILDING_ITERATION or\n 0.1.timestamp.commithash)\n--skip-docker-build\n Don't try to build Docker images\n\nWORKSPACE=path Path to the Arvados source tree to build packages from\n\nEOF\n\nset -e\n\nif ! [[ -n \"$WORKSPACE\" ]]; then\n echo >&2 \"$helpmessage\"\n echo >&2\n echo >&2 \"Error: WORKSPACE environment variable not set\"\n echo >&2\n exit 1\nfi\n\nif ! [[ -d \"$WORKSPACE\" ]]; then\n echo >&2 \"$helpmessage\"\n echo >&2\n echo >&2 \"Error: $WORKSPACE is not a directory\"\n echo >&2\n exit 1\nfi\n\nPARSEDOPTS=$(getopt --name \"$0\" --longoptions \\\n help,debug,test-packages,target:,command:,only-test:,force-test,only-build:,force-build,arch:,build-version:,skip-docker-build \\\n -- \"\" \"$@\")\nif [ $? -ne 0 ]; then\n exit 1\nfi\n\nFORCE_BUILD=0\nCOMMAND=run-build-packages.sh\nDEBUG=\nTARGET=\n\neval set -- \"$PARSEDOPTS\"\nwhile [ $# -gt 0 ]; do\n case \"$1\" in\n --help)\n echo >&2 \"$helpmessage\"\n echo >&2\n exit 1\n ;;\n --target)\n TARGET=\"$2\"; shift\n ;;\n --only-test)\n test_packages=1\n testing_one_package=1\n packages=\"$2\"; shift\n ;;\n --force-test)\n FORCE_TEST=true\n ;;\n --force-build)\n FORCE_BUILD=1\n ;;\n --only-build)\n ONLY_BUILD=\"$2\"; shift\n ;;\n --arch)\n case \"$2\" in\n amd64) ;;\n *)\n printf \"FATAL: --arch '%s' is not supported\" \"$2\" >&2\n exit 2\n ;;\n esac\n ARCH=\"$2\"; shift\n ;;\n --debug)\n DEBUG=\" --debug\"\n ARVADOS_DEBUG=\"1\"\n ;;\n --command)\n COMMAND=\"$2\"; shift\n ;;\n --test-packages)\n test_packages=1\n ;;\n --build-version)\n if [[ -z \"$2\" ]]; then\n :\n elif ! [[ \"$2\" =~ (.*)-(.*) ]]; then\n echo >&2 \"FATAL: --build-version '$2' does not include an iteration. Try '${2}-1'?\"\n exit 1\n elif ! [[ \"$2\" =~ ^[0-9]+\\.[0-9]+\\.[0-9]+(\\.[0-9]+|)(~rc[0-9]+|~dev[0-9]+|)-[0-9]+$ ]]; then\n echo >&2 \"FATAL: --build-version '$2' is invalid, must match pattern ^[0-9]+\\.[0-9]+\\.[0-9]+(\\.[0-9]+|)(~rc[0-9]+|~dev[0-9]+|)-[0-9]+$\"\n exit 1\n else\n [[ \"$2\" =~ (.*)-(.*) ]]\n ARVADOS_BUILDING_VERSION=\"${BASH_REMATCH[1]}\"\n ARVADOS_BUILDING_ITERATION=\"${BASH_REMATCH[2]}\"\n fi\n shift\n ;;\n --skip-docker-build)\n SKIP_DOCKER_BUILD=1\n\t ;;\n --)\n if [ $# -gt 1 ]; then\n echo >&2 \"$0: unrecognized argument '$2'. Try: $0 --help\"\n exit 1\n fi\n ;;\n esac\n shift\ndone\n\nset -e\norig_umask=\"$(umask)\"\n\nif [[ -z \"$TARGET\" ]]; then\n echo \"FATAL: --target must be specified\" >&2\n exit 2\nelif [[ ! -e \"$WORKSPACE/build/package-testing/test-packages-$TARGET.sh\" ]]; then\n echo \"FATAL: unknown build target '$TARGET'\" >&2\n exit 2\nfi\n\nif [[ -n \"$ARVADOS_BUILDING_VERSION\" ]]; then\n echo \"build version='$ARVADOS_BUILDING_VERSION', package iteration='$ARVADOS_BUILDING_ITERATION'\"\nfi\n\nif [[ -n \"$test_packages\" ]]; then\n # Packages are built world-readable, so package indexes should be too,\n # especially because since 2022 apt uses an unprivileged user `_apt` to\n # retrieve everything. Ensure it has permissions to read the packages\n # when mounted as a volume inside the Docker container.\n chmod a+rx \"$WORKSPACE\" \"$WORKSPACE/packages\" \"$WORKSPACE/packages/$TARGET\"\n umask 022\n if [[ -n \"$(find $WORKSPACE/packages/$TARGET -name '*.rpm')\" ]] ; then\n CREATEREPO=\"$(command -v createrepo createrepo_c | tail -n1)\"\n if [[ -z \"$CREATEREPO\" ]]; then\n echo >&2\n echo >&2 \"Error: please install createrepo. E.g. sudo apt install createrepo-c\"\n echo >&2\n exit 1\n fi\n \"$CREATEREPO\" $WORKSPACE/packages/$TARGET\n fi\n\n if [[ -n \"$(find $WORKSPACE/packages/$TARGET -name '*.deb')\" ]] ; then\n set +e\n /usr/bin/which dpkg-scanpackages >/dev/null\n if [[ \"$?\" != \"0\" ]]; then\n echo >&2\n echo >&2 \"Error: please install dpkg-dev. E.g. sudo apt-get install dpkg-dev\"\n echo >&2\n exit 1\n fi\n /usr/bin/which apt-ftparchive >/dev/null\n if [[ \"$?\" != \"0\" ]]; then\n echo >&2\n echo >&2 \"Error: please install apt-utils. E.g. sudo apt-get install apt-utils\"\n echo >&2\n exit 1\n fi\n set -e\n (cd $WORKSPACE/packages/$TARGET\n dpkg-scanpackages --multiversion . 2> >(grep -v 'warning' 1>&2) | tee Packages | gzip -c > Packages.gz\n apt-ftparchive -o APT::FTPArchive::Release::Origin=Arvados release . > Release\n )\n fi\n\n COMMAND=\"/jenkins/package-testing/test-packages-$TARGET.sh\"\n IMAGE=\"arvados/package-test:$TARGET\"\n umask \"$orig_umask\"\nelse\n IMAGE=\"arvados/build:$TARGET\"\n COMMAND=\"bash /jenkins/$COMMAND --target $TARGET$DEBUG\"\nfi\n\nJENKINS_DIR=$(dirname \"$(readlink -e \"$0\")\")\n\nif [[ \"$SKIP_DOCKER_BUILD\" != 1 ]] ; then\n env -C \"$WORKSPACE/tools/ansible\" ansible-galaxy install -r requirements.yml\n declare -a ansible_opts=()\n if [[ -n \"$test_packages\" ]]; then\n ansible_opts+=(\n --extra-vars=arvados_build_playbook=setup-package-tests.yml\n --limit=\"arvados_pkgtest_$TARGET\"\n )\n else\n ansible_opts+=(\n --extra-vars=arvados_build_playbook=install-dev-tools.yml\n --limit=\"arvados_pkgbuild_$TARGET\"\n )\n fi\n env -C \"$WORKSPACE/tools/ansible\" ansible-playbook \\\n --inventory=files/development-docker-images.yml \\\n \"${ansible_opts[@]}\" build-docker-image.yml\n unset ansible_opts\nfi\n\nif test -z \"$packages\" ; then\n packages=\"arvados-api-server\n arvados-client\n arvados-controller\n arvados-dispatch-cloud\n arvados-dispatch-lsf\n arvados-docker-cleaner\n arvados-health\n arvados-server\n arvados-src\n arvados-sync-groups\n arvados-sync-users\n arvados-workbench2\n arvados-ws\n crunch-dispatch-local\n crunch-dispatch-slurm\n crunch-run\n keep-balance\n keep-block-check\n keep-exercise\n keep-rsync\n keep-web\n keepproxy\n keepstore\n libpam-arvados-go\n python3-arvados-cwl-runner\n python3-arvados-fuse\n python3-arvados-python-client\n python3-arvados-user-activity\n python3-arvados-cluster-activity\n python3-crunchstat-summary\"\nfi\n\nFINAL_EXITCODE=0\n\npackage_fails=\"\"\n\nmkdir -p \"$WORKSPACE/services/api/vendor/cache-$TARGET\"\n\ndocker_volume_args=(\n --mount \"type=bind,src=$JENKINS_DIR,dst=/jenkins\"\n --mount \"type=bind,src=$WORKSPACE,dst=/arvados\"\n --tmpfs /arvados/services/api/.bundle:rw,noexec,nosuid,size=1m\n --tmpfs /arvados/services/api/vendor:rw,exec,nosuid,size=1g\n --mount \"type=bind,src=$WORKSPACE/services/api/vendor/cache-$TARGET,dst=/arvados/services/api/vendor/cache\"\n)\n\nif [[ -n \"$test_packages\" ]]; then\n for p in $packages ; do\n if [[ -n \"$ONLY_BUILD\" ]] && [[ \"$p\" != \"$ONLY_BUILD\" ]]; then\n continue\n fi\n if [[ -e \"${WORKSPACE}/packages/.last_test_${TARGET}\" ]] && [[ -z \"$FORCE_TEST\" ]]; then\n MATCH=`find ${WORKSPACE}/packages/ -newer ${WORKSPACE}/packages/.last_test_${TARGET} -regex .*${TARGET}/$p.*`\n if [[ \"$MATCH\" == \"\" ]]; then\n # No new package has been built that needs testing\n echo \"Skipping $p test because no new package was built since the last test.\"\n continue\n fi\n fi\n # If we're testing all packages, we should not error out on packages that don't exist.\n # If we are testing one specific package only (i.e. --only-test was given), we should\n # error out if that package does not exist.\n if [[ -z \"$testing_one_package\" ]]; then\n MATCH=`find ${WORKSPACE}/packages/ -regextype posix-extended -regex .*${TARGET}/$p.*\\\\(deb\\\\|rpm\\\\)`\n if [[ \"$MATCH\" == \"\" ]]; then\n # No new package has been built that needs testing\n echo \"Skipping $p test because no package file is available to test.\"\n continue\n fi\n fi\n echo\n echo \"START: $p test on $IMAGE\" >&2\n if docker run \\\n --rm \\\n \"${docker_volume_args[@]}\" \\\n --env ARVADOS_DEBUG=$ARVADOS_DEBUG \\\n --env \"TARGET=$TARGET\" \\\n --env \"WORKSPACE=/arvados\" \\\n \"$IMAGE\" $COMMAND $p\n then\n echo \"OK: $p test on $IMAGE succeeded\" >&2\n else\n FINAL_EXITCODE=$?\n package_fails=\"$package_fails $p\"\n echo \"ERROR: $p test on $IMAGE failed with exit status $FINAL_EXITCODE\" >&2\n fi\n done\n\n if [[ \"$FINAL_EXITCODE\" == \"0\" ]]; then\n touch ${WORKSPACE}/packages/.last_test_${TARGET}\n fi\nelse\n echo\n echo \"START: build packages on $IMAGE\" >&2\n # Move existing packages and other files into the processed/ subdirectory\n if [[ ! -e \"${WORKSPACE}/packages/${TARGET}/processed\" ]]; then\n mkdir -p \"${WORKSPACE}/packages/${TARGET}/processed\"\n fi\n set +e\n mv -f ${WORKSPACE}/packages/${TARGET}/* ${WORKSPACE}/packages/${TARGET}/processed/ 2>/dev/null\n set -e\n # give bundle (almost) all the cores. See also the MAKE env var that is passed into the\n # docker run command below.\n # Cf. https://build.betterup.com/one-weird-trick-that-will-speed-up-your-bundle-install/\n tmpfile=$(mktemp /tmp/run-build-packages-one-target.XXXXXX)\n cores=$(let a=$(grep -c processor /proc/cpuinfo )-1; echo $a)\n printf -- \"---\\nBUNDLE_JOBS: \\\"$cores\\\"\" > $tmpfile\n # Build packages.\n if docker run \\\n --rm \\\n \"${docker_volume_args[@]}\" \\\n --mount \"type=bind,src=$tmpfile,dst=/root/.bundle/config\" \\\n --env ARVADOS_BUILDING_VERSION=\"$ARVADOS_BUILDING_VERSION\" \\\n --env ARVADOS_BUILDING_ITERATION=\"$ARVADOS_BUILDING_ITERATION\" \\\n --env ARVADOS_DEBUG=$ARVADOS_DEBUG \\\n --env \"ONLY_BUILD=$ONLY_BUILD\" \\\n --env \"FORCE_BUILD=$FORCE_BUILD\" \\\n --env \"ARCH=$ARCH\" \\\n --env \"MAKE=make --jobs $cores\" \\\n \"$IMAGE\" $COMMAND\n then\n echo\n echo \"OK: build packages on $IMAGE succeeded\" >&2\n else\n FINAL_EXITCODE=$?\n echo \"ERROR: build packages on $IMAGE failed with exit status $FINAL_EXITCODE\" >&2\n fi\n # Clean up the bundle config file\n rm -f $tmpfile\nfi\n\nif test -n \"$package_fails\" ; then\n echo \"Failed package tests:$package_fails\" >&2\nfi\n\nexit $FINAL_EXITCODE\n" }, { "path": "build/run-build-packages-python-and-ruby.sh", "content": "#!/bin/bash\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\nCOLUMNS=80\n\n. `dirname \"$(readlink -f \"$0\")\"`/run-library.sh\n\nread -rd \"\\000\" helpmessage <\n Build ruby gems (default: true)\n--python \n Build python packages (default: true)\n\nWORKSPACE=path Path to the Arvados source tree to build packages from\n\nEOF\n\nexit_cleanly() {\n trap - INT\n report_outcomes\n exit ${#failures[@]}\n}\n\ngem_wrapper() {\n local gem_name=\"$1\"; shift\n local gem_directory=\"$1\"; shift\n\n title \"Start $gem_name gem build\"\n timer_reset\n\n cd \"$gem_directory\"\n handle_ruby_gem $gem_name\n\n checkexit $? \"$gem_name gem build\"\n title \"End of $gem_name gem build (`timer`)\"\n}\n\npython_wrapper() {\n local package_name=\"$1\"; shift\n local package_directory=\"$1\"; shift\n\n title \"Start $package_name python package build\"\n timer_reset\n\n python3 -m build \"$package_directory\"\n\n checkexit $? \"$package_name python package build\"\n title \"End of $package_name python package build (`timer`)\"\n}\n\nTARGET=\nUPLOAD=0\nRUBY=1\nPYTHON=1\nDEBUG=${ARVADOS_DEBUG:-0}\n\nPARSEDOPTS=$(getopt --name \"$0\" --longoptions \\\n help,debug,ruby:,python:,upload,target: \\\n -- \"\" \"$@\")\nif [ $? -ne 0 ]; then\n exit 1\nfi\n\neval set -- \"$PARSEDOPTS\"\nwhile [ $# -gt 0 ]; do\n case \"$1\" in\n --help)\n echo >&2 \"$helpmessage\"\n echo >&2\n exit 1\n ;;\n --target)\n TARGET=\"$2\"; shift\n ;;\n --ruby)\n RUBY=\"$2\"; shift\n if [ \"$RUBY\" != \"true\" ] && [ \"$RUBY\" != \"1\" ]; then\n RUBY=0\n else\n RUBY=1\n fi\n ;;\n --python)\n PYTHON=\"$2\"; shift\n if [ \"$PYTHON\" != \"true\" ] && [ \"$PYTHON\" != \"1\" ]; then\n PYTHON=0\n else\n PYTHON=1\n fi\n ;;\n --upload)\n UPLOAD=1\n ;;\n --debug)\n DEBUG=1\n ;;\n --)\n if [ $# -gt 1 ]; then\n echo >&2 \"$0: unrecognized argument '$2'. Try: $0 --help\"\n exit 1\n fi\n ;;\n esac\n shift\ndone\n\nif ! [[ -n \"$WORKSPACE\" ]]; then\n echo >&2 \"$helpmessage\"\n echo >&2\n echo >&2 \"Error: WORKSPACE environment variable not set\"\n echo >&2\n exit 1\nfi\n\nSTDOUT_IF_DEBUG=/dev/null\nSTDERR_IF_DEBUG=/dev/null\nDASHQ_UNLESS_DEBUG=-q\nif [[ \"$DEBUG\" != 0 ]]; then\n STDOUT_IF_DEBUG=/dev/stdout\n STDERR_IF_DEBUG=/dev/stderr\n DASHQ_UNLESS_DEBUG=\nfi\n\nRUN_BUILD_PACKAGES_PATH=\"`dirname \\\"$0\\\"`\"\nRUN_BUILD_PACKAGES_PATH=\"`( cd \\\"$RUN_BUILD_PACKAGES_PATH\\\" && pwd )`\" # absolutized and normalized\nif [ -z \"$RUN_BUILD_PACKAGES_PATH\" ] ; then\n # error; for some reason, the path is not accessible\n # to the script (e.g. permissions re-evaled after suid)\n exit 1 # fail\nfi\n\ndebug_echo \"$0 is running from $RUN_BUILD_PACKAGES_PATH\"\ndebug_echo \"Workspace is $WORKSPACE\"\n\nif [ $RUBY -eq 0 ] && [ $PYTHON -eq 0 ]; then\n echo \"Nothing to do!\"\n exit 0\nfi\n\n# Make all files world-readable -- jenkins runs with umask 027, and has checked\n# out our git tree here\nchmod o+r \"$WORKSPACE\" -R\n\n# More cleanup - make sure all executables that we'll package are 755\ncd \"$WORKSPACE\"\nfind -type d -name 'bin' |xargs -I {} find {} -type f |xargs -I {} chmod 755 {}\n\n# Now fix our umask to something better suited to building and publishing\n# gems and packages\numask 0022\n\ndebug_echo \"umask is\" `umask`\n\nGEM_BUILD_FAILURES=0\nif [ $RUBY -eq 1 ]; then\n debug_echo \"Building Ruby gems\"\n gem_wrapper arvados \"$WORKSPACE/sdk/ruby\"\n gem_wrapper arvados-cli \"$WORKSPACE/sdk/cli\"\n gem_wrapper arvados-login-sync \"$WORKSPACE/services/login-sync\"\n if [ ${#failures[@]} -ne 0 ]; then\n GEM_BUILD_FAILURES=${#failures[@]}\n fi\nfi\n\nPYTHON_BUILD_FAILURES=0\nif [ $PYTHON -eq 1 ]; then\n debug_echo \"Building Python packages\"\n python_wrapper arvados-python-client \"$WORKSPACE/sdk/python\"\n python_wrapper arvados-cwl-runner \"$WORKSPACE/sdk/cwl\"\n python_wrapper arvados_fuse \"$WORKSPACE/services/fuse\"\n python_wrapper crunchstat_summary \"$WORKSPACE/tools/crunchstat-summary\"\n python_wrapper arvados-user-activity \"$WORKSPACE/tools/user-activity\"\n python_wrapper arvados-cluster-activity \"$WORKSPACE/tools/cluster-activity\"\n\n if [ $((${#failures[@]} - $GEM_BUILD_FAILURES)) -ne 0 ]; then\n PYTHON_BUILD_FAILURES=$((${#failures[@]} - $GEM_BUILD_FAILURES))\n fi\nfi\n\nif [ $UPLOAD -ne 0 ]; then\n if get_ci_scripts\n then\n checkexit $? \"get CI scripts\"\n else\n checkexit $? \"get CI scripts\"\n UPLOAD=0\n fi\nfi\n\nif [ $UPLOAD -ne 0 ]; then\n echo \"Uploading\"\n\n if [ $DEBUG > 0 ]; then\n EXTRA_UPLOAD_FLAGS=\" --verbose\"\n else\n EXTRA_UPLOAD_FLAGS=\"\"\n fi\n\n if [ ! -e \"$WORKSPACE/packages\" ]; then\n mkdir -p \"$WORKSPACE/packages\"\n fi\n\n if [ $PYTHON -eq 1 ]; then\n title \"Start upload python packages\"\n timer_reset\n\n if [ $PYTHON_BUILD_FAILURES -eq 0 ]; then\n \"$CI_DIR/run_upload_packages.py\" $EXTRA_UPLOAD_FLAGS --workspace $WORKSPACE python\n else\n echo \"Skipping python packages upload, there were errors building the packages\"\n fi\n checkexit $? \"upload python packages\"\n title \"End of upload python packages (`timer`)\"\n fi\n\n if [ $RUBY -eq 1 ]; then\n title \"Start upload ruby gems\"\n timer_reset\n\n if [ $GEM_BUILD_FAILURES -eq 0 ]; then\n \"$CI_DIR/run_upload_packages.py\" $EXTRA_UPLOAD_FLAGS --workspace $WORKSPACE gems\n else\n echo \"Skipping ruby gem upload, there were errors building the packages\"\n fi\n checkexit $? \"upload ruby gems\"\n title \"End of upload ruby gems (`timer`)\"\n fi\nfi\n\nexit_cleanly\n" }, { "path": "build/run-build-packages.sh", "content": "#!/bin/bash\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\n. \"$(dirname \"$(readlink -f \"$0\")\")\"/run-library.sh || exit 1\n\nread -rd \"\\000\" helpmessage < [options]\n\nOptions:\n\n--build-bundle-packages (default: false)\n Build api server package with vendor/bundle included\n--debug\n Output debug information (default: false)\n--target \n Distribution to build packages for\n--only-build \n Build only a specific package (or ONLY_BUILD from environment)\n--force-build\n Build even if the package exists upstream or if it has already been\n built locally\n--command\n Build command to execute (defaults to the run command defined in the\n Docker image)\n\nWORKSPACE=path Path to the Arvados source tree to build packages from\n\nEOF\n\n# Begin of user configuration\n\n# set to --no-cache-dir to disable pip caching\nCACHE_FLAG=\n\nMAINTAINER=\"Arvados Package Maintainers \"\nVENDOR=\"The Arvados Project\"\n\n# End of user configuration\n\nDEBUG=${ARVADOS_DEBUG:-0}\nFORCE_BUILD=${FORCE_BUILD:-0}\nEXITCODE=0\nCOMMAND=\nTARGET=\n\nPARSEDOPTS=$(getopt --name \"$0\" --longoptions \\\n help,build-bundle-packages,debug,target:,only-build:,arch:,force-build \\\n -- \"\" \"$@\")\nif [ $? -ne 0 ]; then\n exit 1\nfi\n\neval set -- \"$PARSEDOPTS\"\nwhile [ $# -gt 0 ]; do\n case \"$1\" in\n --help)\n echo >&2 \"$helpmessage\"\n echo >&2\n exit 1\n ;;\n --target)\n TARGET=\"$2\"; shift\n ;;\n --only-build)\n ONLY_BUILD=\"$2\"; shift\n ;;\n --force-build)\n FORCE_BUILD=1\n ;;\n --arch)\n case \"$2\" in\n amd64) ;;\n *)\n printf \"FATAL: --arch '%s' is not supported\" \"$2\" >&2\n exit 2\n ;;\n esac\n ARCH=\"$2\"; shift\n ;;\n --debug)\n DEBUG=1\n ;;\n --command)\n COMMAND=\"$2\"; shift\n ;;\n --)\n if [ $# -gt 1 ]; then\n echo >&2 \"$0: unrecognized argument '$2'. Try: $0 --help\"\n exit 1\n fi\n ;;\n esac\n shift\ndone\n\nif [[ -z \"$TARGET\" ]]; then\n echo \"FATAL: --target must be specified\" >&2\n exit 2\nelif [[ ! -e \"$WORKSPACE/build/package-testing/test-packages-$TARGET.sh\" ]]; then\n echo \"FATAL: unknown build target '$TARGET'\" >&2\n exit 2\nfi\n\nif [[ \"$COMMAND\" != \"\" ]]; then\n COMMAND=\"bash /jenkins/$COMMAND --target $TARGET\"\nfi\n\nSTDOUT_IF_DEBUG=/dev/null\nSTDERR_IF_DEBUG=/dev/null\nDASHQ_UNLESS_DEBUG=-q\nif [[ \"$DEBUG\" != 0 ]]; then\n STDOUT_IF_DEBUG=/dev/stdout\n STDERR_IF_DEBUG=/dev/stderr\n DASHQ_UNLESS_DEBUG=\nfi\n\n# The next section defines a bunch of constants used to build distro packages\n# for our Python tools. Because those packages include C extensions, they need\n# to depend on and refer to a specific minor version of Python 3. The logic\n# below should Just Work for most cases, but you can override variables for a\n# specific distro if you need to to do something weird.\n# * PYTHON3_VERSION: The major+minor version of Python we build against\n# (e.g., \"3.11\")\n# * PYTHON3_EXECUTABLE: The command to run that version of Python,\n# either a full path or something in $PATH (e.g., \"python3.11\")\n# * PYTHON3_PACKAGE: The name of the distro package that provides\n# $PYTHON3_EXECUTABLE. Our Python packages will all depend on this.\n# * PYTHON3_PKG_PREFIX: The prefix used in the names of all of our Python\n# packages. This should match distro convention.\nPYTHON3_PKG_PREFIX=python3\ncase \"$TARGET\" in\n rocky9)\n FORMAT=rpm\n PYTHON3_VERSION=3.11\n ;;\n centos*|rocky*)\n FORMAT=rpm\n ;;\n debian*|ubuntu*)\n FORMAT=deb\n ;;\n *)\n echo -e \"$0: Unknown target '$TARGET'.\\n\" >&2\n exit 1\n ;;\nesac\n: \"${PYTHON3_VERSION:=$(\"${PYTHON3_EXECUTABLE:-python3}\" -c 'import sys; print(\"{v.major}.{v.minor}\".format(v=sys.version_info))')}\"\n: \"${PYTHON3_EXECUTABLE:=python$PYTHON3_VERSION}\"\ncase \"$FORMAT\" in\n deb)\n : \"${PYTHON3_PACKAGE:=python$PYTHON3_VERSION}\"\n ;;\n rpm)\n : \"${PYTHON3_PACKAGE:=$(rpm -qf \"$(command -v \"$PYTHON3_EXECUTABLE\")\" --queryformat '%{NAME}\\n')}\"\n ;;\nesac\n\nif [[ -z \"$WORKSPACE\" ]]; then\n echo >&2 \"$helpmessage\"\n echo >&2\n echo >&2 \"Error: WORKSPACE environment variable not set\"\n echo >&2\n exit 1\nfi\n\n# Test for fpm\nfpm --version >/dev/null 2>&1\n\nif [[ $? -ne 0 ]]; then\n echo >&2 \"$helpmessage\"\n echo >&2\n echo >&2 \"Error: fpm not found\"\n echo >&2\n exit 1\nfi\n\nRUN_BUILD_PACKAGES_PATH=\"$(dirname \"$0\")\"\nRUN_BUILD_PACKAGES_PATH=\"$(cd \"$RUN_BUILD_PACKAGES_PATH\" && pwd)\" # absolutized and normalized\nif [ -z \"$RUN_BUILD_PACKAGES_PATH\" ] ; then\n # error; for some reason, the path is not accessible\n # to the script (e.g. permissions re-evaled after suid)\n exit 1 # fail\nfi\n\ndebug_echo \"$0 is running from $RUN_BUILD_PACKAGES_PATH\"\ndebug_echo \"Workspace is $WORKSPACE\"\n\n# Make all files world-readable -- jenkins runs with umask 027, and has checked\n# out our git tree here\nchmod o+r \"$WORKSPACE\" -R\n\n# More cleanup - make sure all executables that we'll package are 755\ncd \"$WORKSPACE\" || exit 1\nfind . -type d -name 'bin' -print0 |xargs -0 -I {} find {} -type f -print0 |xargs -0 -I {} chmod 755 {}\n\n# Now fix our umask to something better suited to building and publishing\n# gems and packages\numask 0022\n\ndebug_echo \"umask is\" \"$(umask)\"\n\nif [[ ! -d \"$WORKSPACE/packages/$TARGET\" ]]; then\n mkdir -p \"$WORKSPACE/packages/$TARGET\"\n chown --reference=\"$WORKSPACE\" \"$WORKSPACE/packages/$TARGET\"\nfi\n\n# Required due to CVE-2022-24765\ngit config --global --add safe.directory /arvados\n\n# Ruby gems\ndebug_echo -e \"\\nRuby gems\\n\"\n\nFPM_GEM_PREFIX=$(gem environment gemdir)\n\ncd \"$WORKSPACE/sdk/ruby\" || exit 1\nhandle_ruby_gem arvados\n\ncd \"$WORKSPACE/sdk/cli\" || exit 1\nhandle_ruby_gem arvados-cli\n\ncd \"$WORKSPACE/services/login-sync\" || exit 1\nhandle_ruby_gem arvados-login-sync\n\n# arvados-src\nhandle_arvados_src\n\n# Go packages\ndebug_echo -e \"\\nGo packages\\n\"\n\n# Go binaries\nexport GOPATH=~/go\npackage_go_binary cmd/arvados-client arvados-client \"$FORMAT\" \"$ARCH\" \\\n \"Arvados command line tool (beta)\"\npackage_go_binary cmd/arvados-server arvados-server \"$FORMAT\" \"$ARCH\" \\\n \"Arvados server daemons\"\npackage_go_binary cmd/arvados-server arvados-controller \"$FORMAT\" \"$ARCH\" \\\n \"Arvados cluster controller daemon\"\npackage_go_binary cmd/arvados-server arvados-dispatch-cloud \"$FORMAT\" \"$ARCH\" \\\n \"Arvados cluster cloud dispatch\"\npackage_go_binary cmd/arvados-server arvados-dispatch-lsf \"$FORMAT\" \"$ARCH\" \\\n \"Dispatch Arvados containers to an LSF cluster\"\npackage_go_binary services/crunch-dispatch-local crunch-dispatch-local \"$FORMAT\" \"$ARCH\" \\\n \"Dispatch Crunch containers on the local system\"\npackage_go_binary cmd/arvados-server crunch-dispatch-slurm \"$FORMAT\" \"$ARCH\" \\\n \"Dispatch Crunch containers to a SLURM cluster\"\npackage_go_binary cmd/arvados-server crunch-run \"$FORMAT\" \"$ARCH\" \\\n \"Supervise a single Crunch container\"\npackage_go_binary cmd/arvados-server arvados-health \"$FORMAT\" \"$ARCH\" \\\n \"Check health of all Arvados cluster services\"\npackage_go_binary cmd/arvados-server keep-balance \"$FORMAT\" \"$ARCH\" \\\n \"Rebalance and garbage-collect data blocks stored in Arvados Keep\"\npackage_go_binary cmd/arvados-server keepproxy \"$FORMAT\" \"$ARCH\" \\\n \"Make a Keep cluster accessible to clients that are not on the LAN\"\npackage_go_binary cmd/arvados-server keepstore \"$FORMAT\" \"$ARCH\" \\\n \"Keep storage daemon, accessible to clients on the LAN\"\npackage_go_binary cmd/arvados-server keep-web \"$FORMAT\" \"$ARCH\" \\\n \"Static web hosting service for user data stored in Arvados Keep\"\npackage_go_binary cmd/arvados-server arvados-ws \"$FORMAT\" \"$ARCH\" \\\n \"Arvados Websocket server\"\npackage_go_binary tools/sync-groups arvados-sync-groups \"$FORMAT\" \"$ARCH\" \\\n \"Synchronize remote groups into Arvados from an external source\"\npackage_go_binary tools/sync-users arvados-sync-users \"$FORMAT\" \"$ARCH\" \\\n \"Synchronize remote users into Arvados from an external source\"\npackage_go_binary tools/keep-block-check keep-block-check \"$FORMAT\" \"$ARCH\" \\\n \"Verify that all data from one set of Keep servers to another was copied\"\npackage_go_binary tools/keep-rsync keep-rsync \"$FORMAT\" \"$ARCH\" \\\n \"Copy all data from one set of Keep servers to another\"\npackage_go_binary tools/keep-exercise keep-exercise \"$FORMAT\" \"$ARCH\" \\\n \"Performance testing tool for Arvados Keep\"\npackage_go_so lib/pam pam_arvados.so libpam-arvados-go \"$FORMAT\" \"$ARCH\" \\\n \"Arvados PAM authentication module\"\n\n# Python packages\ndebug_echo -e \"\\nPython packages\\n\"\n\n# Before a Python package can be built, its dependencies must already be built.\n# This list is ordered accordingly.\nsetup_build_virtualenv\nfpm_build_virtualenv \"arvados-python-client\" \"sdk/python\" \"$FORMAT\" \"$ARCH\"\nfpm_build_virtualenv \"crunchstat-summary\" \"tools/crunchstat-summary\" \"$FORMAT\" \"$ARCH\"\nfpm_build_virtualenv \"arvados-cwl-runner\" \"sdk/cwl\" \"$FORMAT\" \"$ARCH\"\nfpm_build_virtualenv \"arvados-docker-cleaner\" \"services/dockercleaner\" \"$FORMAT\" \"$ARCH\"\nfpm_build_virtualenv \"arvados-fuse\" \"services/fuse\" \"$FORMAT\" \"$ARCH\"\nfpm_build_virtualenv \"arvados-user-activity\" \"tools/user-activity\" \"$FORMAT\" \"$ARCH\"\nfpm_build_virtualenv \"arvados-cluster-activity\" \"tools/cluster-activity\" \"$FORMAT\" \"$ARCH\"\n\n# Workbench2\npackage_workbench2\n\n# Rails packages\ndebug_echo -e \"\\nRails packages\\n\"\n\n# The rails api server package\nhandle_api_server \"$ARCH\"\n\n# clean up temporary GOPATH\nrm -rf \"$GOPATH\"\n\nexit $EXITCODE\n" }, { "path": "build/run-build-test-packages-one-target.sh", "content": "#!/bin/bash\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\nread -rd \"\\000\" helpmessage < [options]\n\n--target \n Distribution to build packages for\n--only-build \n Build only a specific package (or ONLY_BUILD from environment)\n--arch \n Build a specific architecture (or ARCH from environment, defaults to native architecture)\n--force-build\n Build even if the package exists upstream or if it has already been\n built locally\n--force-test\n Test even if there is no new untested package\n--upload\n If the build and test steps are successful, upload the packages\n to a remote apt repository (default: false)\n--debug\n Output debug information (default: false)\n--rc\n Optional Parameter to build Release Candidate\n--build-version \n Version to build (default:\n \\$ARVADOS_BUILDING_VERSION-\\$ARVADOS_BUILDING_ITERATION or\n 0.1.timestamp.commithash)\n--skip-docker-build\n Don't try to build Docker images\n\nWORKSPACE=path Path to the Arvados source tree to build packages from\n\nEOF\n\nif ! [[ -n \"$WORKSPACE\" ]]; then\n echo >&2 \"$helpmessage\"\n echo >&2\n echo >&2 \"Error: WORKSPACE environment variable not set\"\n echo >&2\n exit 1\nfi\n\nif ! [[ -d \"$WORKSPACE\" ]]; then\n echo >&2 \"$helpmessage\"\n echo >&2\n echo >&2 \"Error: $WORKSPACE is not a directory\"\n echo >&2\n exit 1\nfi\n\nPARSEDOPTS=$(getopt --name \"$0\" --longoptions \\\n help,debug,upload,rc,target:,force-test,only-build:,force-build,arch:,build-version:,skip-docker-build \\\n -- \"\" \"$@\")\nif [ $? -ne 0 ]; then\n exit 1\nfi\n\nUPLOAD=0\nUPLOAD_REPO=dev\nDEBUG=\nTARGET=\n\ndeclare -a build_args=()\n\neval set -- \"$PARSEDOPTS\"\nwhile [ $# -gt 0 ]; do\n case \"$1\" in\n --help)\n echo >&2 \"$helpmessage\"\n echo >&2\n exit 1\n ;;\n --target)\n TARGET=\"$2\"; shift\n ;;\n --force-test)\n FORCE_TEST=1\n ;;\n --force-build)\n FORCE_BUILD=1\n ;;\n --only-build)\n ONLY_BUILD=\"$2\"; shift\n ;;\n --arch)\n ARCH=\"$2\"; shift\n ;;\n --debug)\n DEBUG=\" --debug\"\n ;;\n --upload)\n UPLOAD=1\n ;;\n --rc)\n UPLOAD_REPO=testing\n ;;\n --build-version)\n build_args+=(\"$1\" \"$2\")\n shift\n ;;\n --skip-docker-build)\n SKIP_DOCKER_BUILD=1\n\t ;;\n --)\n if [ $# -gt 1 ]; then\n echo >&2 \"$0: unrecognized argument '$2'. Try: $0 --help\"\n exit 1\n fi\n ;;\n esac\n shift\ndone\n\nif [[ -z \"$TARGET\" ]]; then\n echo \"FATAL: --target must be specified\" >&2\n exit 2\nelif [[ ! -e \"$WORKSPACE/build/package-testing/test-packages-$TARGET.sh\" ]]; then\n echo \"FATAL: unknown build target '$TARGET'\" >&2\n exit 2\nfi\n\nbuild_args+=(--target \"$TARGET\")\n\nif [[ -n \"$ONLY_BUILD\" ]]; then\n build_args+=(--only-build \"$ONLY_BUILD\")\nfi\n\nif [[ -n \"$FORCE_BUILD\" ]]; then\n build_args+=(--force-build)\nfi\n\nif [[ -n \"$FORCE_TEST\" ]]; then\n build_args+=(--force-test)\nfi\n\nif [[ \"$SKIP_DOCKER_BUILD\" = 1 ]]; then\n build_args+=(--skip-docker-build)\nfi\n\nif [[ -n \"$ARCH\" ]]; then\n build_args+=(--arch \"$ARCH\")\nfi\n\nexit_cleanly() {\n trap - INT\n report_outcomes\n exit ${#failures}\n}\n\nCOLUMNS=80\n. $WORKSPACE/build/run-library.sh\n\ntitle \"Start build packages\"\ntimer_reset\n\n$WORKSPACE/build/run-build-packages-one-target.sh \"${build_args[@]}\"$DEBUG\n\ncheckexit $? \"build packages\"\ntitle \"End of build packages (`timer`)\"\n\ntitle \"Start test packages\"\ntimer_reset\n\nif [ ${#failures[@]} -eq 0 ]; then\n $WORKSPACE/build/run-build-packages-one-target.sh \"${build_args[@]}\" --test-packages$DEBUG\nelse\n echo \"Skipping package upload, there were errors building the packages\"\nfi\n\ncheckexit $? \"test packages\"\ntitle \"End of test packages (`timer`)\"\n\nif [[ \"$UPLOAD\" != 0 ]]; then\n title \"Start upload packages\"\n timer_reset\n\n get_ci_scripts\n checkexit $? \"get CI scripts\"\n\n if [ ${#failures[@]} -eq 0 ]; then\n \"$CI_DIR/run_upload_packages.py\" \\\n --repo=\"$UPLOAD_REPO\" \\\n -H jenkinsapt@apt.arvados.org \\\n --workspace=\"$WORKSPACE\" \\\n \"$TARGET\"\n checkexit $? \"upload packages\"\n else\n echo \"Skipping package upload, there were errors building and/or testing the packages\"\n fi\n title \"End of upload packages (`timer`)\"\nfi\n\nexit_cleanly\n" }, { "path": "build/run-library.sh", "content": "#!/bin/bash -xe\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\n# A library of functions shared by the various scripts in this directory.\n\n# This is the timestamp about when we merged changed to include licenses\n# with Arvados packages. We use it as a heuristic to add revisions for\n# older packages.\nLICENSE_PACKAGE_TS=20151208015500\nRAILS_PACKAGE_ITERATION=\"${ARVADOS_BUILDING_ITERATION:-1}\"\n\ndeclare -A LICENSE_FILE_NAME_MAP=(\n [agpl-3.0.txt]=\"GNU Affero General Public License version 3.0\"\n [LICENSE-2.0.txt]=\"Apache 2.0\"\n)\n\ndebug_echo () {\n echo \"$@\" >\"$STDOUT_IF_DEBUG\"\n}\n\nfind_python_program() {\n prog=\"$1\"\n shift\n for prog in \"$@\"; do\n if \"$prog\" --version >/dev/null 2>&1; then\n echo \"$prog\"\n return 0\n fi\n done\n cat >&2 <\"$STDOUT_IF_DEBUG\" 2>\"$STDERR_IF_DEBUG\"\n fi\n}\n\n# Usage: package_workbench2\npackage_workbench2() {\n local pkgname=arvados-workbench2\n local src=services/workbench2\n local dst=/var/www/arvados-workbench2/workbench2\n local description=\"Arvados Workbench 2\"\n if [[ -n \"$ONLY_BUILD\" ]] && [[ \"$pkgname\" != \"$ONLY_BUILD\" ]] ; then\n return 0\n fi\n cd \"$WORKSPACE/$src\"\n local version=\"$(version_from_git)\"\n rm -rf ./build\n NODE_ENV=production yarn install\n VERSION=\"$version\" BUILD_NUMBER=\"$(default_iteration \"$pkgname\" \"$version\" yarn)\" GIT_COMMIT=\"$(git rev-parse HEAD | head -c9)\" yarn build\n cd \"$WORKSPACE/packages/$TARGET\"\n fpm_build \"${WORKSPACE}/$src\" \"${WORKSPACE}/$src/build/=$dst\" \"$pkgname\" dir \"$version\" \\\n --license=\"GNU Affero General Public License, version 3.0\" \\\n --description=\"${description}\" \\\n --config-files=\"/etc/arvados/$pkgname/workbench2.example.json\" \\\n \"$WORKSPACE/services/workbench2/etc/arvados/workbench2/workbench2.example.json=/etc/arvados/$pkgname/workbench2.example.json\"\n}\n\ncalculate_go_package_version() {\n # $__returnvar has the nameref attribute set, which means it is a reference\n # to another variable that is passed in as the first argument to this function.\n # see https://www.gnu.org/software/bash/manual/html_node/Shell-Parameters.html\n local -n __returnvar=\"$1\"; shift\n local oldpwd=\"$PWD\"\n\n cd \"$WORKSPACE\"\n go mod download\n\n # Update the version number and build a new package if the vendor\n # bundle has changed, or the command imports anything from the\n # Arvados SDK and the SDK has changed.\n declare -a checkdirs=(go.mod go.sum)\n while [ -n \"$1\" ]; do\n checkdirs+=(\"$1\")\n shift\n done\n # Even our rails packages (version calculation happens here!) depend on a go component (arvados-server)\n # Everything depends on the build directory.\n checkdirs+=(sdk/go lib build)\n local timestamp=0\n for dir in ${checkdirs[@]}; do\n cd \"$WORKSPACE\"\n ts=\"$(timestamp_from_git \"$dir\")\"\n if [[ \"$ts\" -gt \"$timestamp\" ]]; then\n version=$(version_from_git \"$dir\")\n timestamp=\"$ts\"\n fi\n done\n cd \"$oldpwd\"\n __returnvar=\"$version\"\n}\n\n# Usage: package_go_binary services/foo arvados-foo [deb|rpm] [amd64] \"Compute foo to arbitrary precision\" [apache-2.0.txt]\npackage_go_binary() {\n local src_path=\"$1\"; shift\n local prog=\"$1\"; shift\n local package_format=\"$1\"; shift\n local target_arch=\"$1\"; shift\n local description=\"$1\"; shift\n local license_file=\"${1:-agpl-3.0.txt}\"; shift\n\n if [[ -n \"$ONLY_BUILD\" ]] && [[ \"$prog\" != \"$ONLY_BUILD\" ]]; then\n debug_echo -e \"Skipping build of $prog package.\"\n return 0\n fi\n\n native_arch=$(get_native_arch)\n\n if [[ \"$native_arch\" != \"amd64\" ]] && [[ -n \"$target_arch\" ]] && [[ \"$native_arch\" != \"$target_arch\" ]]; then\n echo \"Error: no cross compilation support for Go on $native_arch, can not build $prog for $target_arch\"\n return 1\n fi\n\n case \"$package_format-$TARGET\" in\n # Red Hat-based distributions do not support native cross compilation at\n # all (they use a qemu-based solution we haven't implemented yet).\n rpm-*)\n cross_compilation=0\n if [[ \"$native_arch\" == \"amd64\" ]] && [[ -n \"$target_arch\" ]] && [[ \"$native_arch\" != \"$target_arch\" ]]; then\n echo \"Error: no cross compilation support for Go on $native_arch for $TARGET, can not build $prog for $target_arch\"\n return 1\n fi\n ;;\n *)\n cross_compilation=1\n ;;\n esac\n\n if [[ -n \"$target_arch\" ]]; then\n archs=($target_arch)\n else\n # No target architecture specified, default to native target.\n archs=($native_arch)\n fi\n\n for ta in ${archs[@]}; do\n package_go_binary_worker \"$src_path\" \"$prog\" \"$package_format\" \"$description\" \"$native_arch\" \"$ta\" \"$license_file\"\n retval=$?\n if [[ $retval -ne 0 ]]; then\n return $retval\n fi\n done\n}\n\n# Usage: package_go_binary services/foo arvados-foo deb \"Compute foo to arbitrary precision\" [amd64] [amd64] [apache-2.0.txt]\npackage_go_binary_worker() {\n local src_path=\"$1\"; shift\n local prog=\"$1\"; shift\n local package_format=\"$1\"; shift\n local description=\"$1\"; shift\n local native_arch=\"${1:-amd64}\"; shift\n local target_arch=\"${1:-amd64}\"; shift\n local license_file=\"${1:-agpl-3.0.txt}\"; shift\n\n debug_echo \"package_go_binary $src_path as $prog (native arch: $native_arch, target arch: $target_arch)\"\n local basename=\"${src_path##*/}\"\n calculate_go_package_version go_package_version $src_path\n\n cd $WORKSPACE/packages/$TARGET\n test_package_presence \"$prog\" \"$go_package_version\" \"go\" \"\" \"$target_arch\"\n if [[ $? -ne 0 ]]; then\n return 0\n fi\n\n echo \"Building $package_format ($target_arch) package for $prog from $src_path\"\n GOARCH=${arch} go install -ldflags \"-X git.arvados.org/arvados.git/lib/cmd.version=${go_package_version} -X main.version=${go_package_version}\" \"git.arvados.org/arvados.git/$src_path\"\n\n local -a switches=()\n\n binpath=$GOPATH/bin/${basename}\n if [[ \"${target_arch}\" != \"${native_arch}\" ]]; then\n switches+=(\"-a${target_arch}\")\n binpath=\"$GOPATH/bin/linux_${target_arch}/${basename}\"\n fi\n\n case \"$package_format\" in\n # As of April 2024 we package identical Go binaries under different\n # packages and names. This upsets the build id database, so don't\n # register ourselves there.\n rpm) switches+=(--rpm-rpmbuild-define=\"_build_id_links none\") ;;\n esac\n\n systemd_unit=\"$WORKSPACE/${src_path}/${prog}.service\"\n if [[ -e \"${systemd_unit}\" ]]; then\n switches+=(\n --after-install \"${WORKSPACE}/build/go-python-package-scripts/postinst\"\n --before-remove \"${WORKSPACE}/build/go-python-package-scripts/prerm\"\n \"${systemd_unit}=/lib/systemd/system/${prog}.service\")\n fi\n switches+=(\"$WORKSPACE/${license_file}=/usr/share/doc/$prog/${license_file}\")\n\n fpm_build \"${WORKSPACE}/${src_path}\" \"$binpath=/usr/bin/${prog}\" \"${prog}\" dir \"${go_package_version}\" \"--url=https://arvados.org\" \"--license=GNU Affero General Public License, version 3.0\" \"--description=${description}\" \"${switches[@]}\"\n}\n\n# Usage: package_go_so lib/foo arvados_foo.so arvados-foo deb amd64 \"Arvados foo library\"\npackage_go_so() {\n local src_path=\"$1\"; shift\n local sofile=\"$1\"; shift\n local pkg=\"$1\"; shift\n local package_format=\"$1\"; shift\n local target_arch=\"$1\"; shift # supported: amd64\n local description=\"$1\"; shift\n\n if [[ -n \"$ONLY_BUILD\" ]] && [[ \"$pkg\" != \"$ONLY_BUILD\" ]]; then\n debug_echo -e \"Skipping build of $pkg package.\"\n return 0\n fi\n\n debug_echo \"package_go_so $src_path as $pkg\"\n\n calculate_go_package_version go_package_version $src_path\n cd $WORKSPACE/packages/$TARGET\n test_package_presence $pkg $go_package_version go || return 1\n cd $WORKSPACE/$src_path\n go build -buildmode=c-shared -o ${GOPATH}/bin/${sofile}\n cd $WORKSPACE/packages/$TARGET\n local -a fpmargs=(\n \"--url=https://arvados.org\"\n \"--license=Apache License, Version 2.0\"\n \"--description=${description}\"\n \"$WORKSPACE/apache-2.0.txt=/usr/share/doc/$pkg/apache-2.0.txt\"\n )\n if [[ -e \"$WORKSPACE/$src_path/pam-configs-arvados\" ]]; then\n fpmargs+=(\"$WORKSPACE/$src_path/pam-configs-arvados=/usr/share/doc/$pkg/pam-configs-arvados-go\")\n fi\n if [[ -e \"$WORKSPACE/$src_path/README\" ]]; then\n fpmargs+=(\"$WORKSPACE/$src_path/README=/usr/share/doc/$pkg/README\")\n fi\n fpm_build \"${WORKSPACE}/${src_path}\" \"$GOPATH/bin/${sofile}=/usr/lib/${sofile}\" \"${pkg}\" dir \"${go_package_version}\" \"${fpmargs[@]}\"\n}\n\ndefault_iteration() {\n if [[ -n \"$ARVADOS_BUILDING_VERSION\" ]]; then\n echo \"$ARVADOS_BUILDING_ITERATION\"\n return\n fi\n local package_name=\"$1\"; shift\n local package_version=\"$1\"; shift\n local package_type=\"$1\"; shift\n local iteration=1\n if [[ $package_version =~ ^0\\.1\\.([0-9]{14})(\\.|$) ]] && \\\n [[ ${BASH_REMATCH[1]} -le $LICENSE_PACKAGE_TS ]]; then\n iteration=2\n fi\n echo $iteration\n}\n\n_build_rails_package_scripts() {\n local pkgname=\"$1\"; shift\n local destdir=\"$1\"; shift\n local srcdir=\"$RUN_BUILD_PACKAGES_PATH/rails-package-scripts\"\n for scriptname in postinst prerm postrm; do\n cat \"$srcdir/$pkgname.sh\" \"$srcdir/$scriptname.sh\" \\\n >\"$destdir/$scriptname\" || return $?\n done\n}\n\nrails_package_version() {\n local pkgname=\"$1\"; shift\n local srcdir=\"$1\"; shift\n if [[ -n \"$ARVADOS_BUILDING_VERSION\" ]]; then\n echo \"$ARVADOS_BUILDING_VERSION\"\n return\n fi\n local version=\"$(version_from_git)\"\n if [ $pkgname = \"arvados-api-server\" ] ; then\n calculate_go_package_version version cmd/arvados-server \"$srcdir\"\n fi\n echo $version\n}\n\ntest_rails_package_presence() {\n local pkgname=\"$1\"; shift\n local srcdir=\"$1\"; shift\n\n if [[ -n \"$ONLY_BUILD\" ]] && [[ \"$pkgname\" != \"$ONLY_BUILD\" ]] ; then\n return 1\n fi\n\n tmppwd=`pwd`\n\n cd $srcdir\n\n local version=\"$(rails_package_version \"$pkgname\" \"$srcdir\")\"\n\n cd $tmppwd\n\n test_package_presence $pkgname $version rails \"$RAILS_PACKAGE_ITERATION\"\n}\n\nget_complete_package_name() {\n # if the errexit flag is set, unset it until this function returns\n # otherwise, the shift calls below will abort the program if optional arguments are not supplied\n if [ -o errexit ]; then\n set +e\n trap 'set -e' RETURN\n fi\n # $__returnvar has the nameref attribute set, which means it is a reference\n # to another variable that is passed in as the first argument to this function.\n # see https://www.gnu.org/software/bash/manual/html_node/Shell-Parameters.html\n local -n __returnvar=\"$1\"; shift\n local pkgname=\"$1\"; shift\n local version=\"$1\"; shift\n local pkgtype=\"$1\"; shift\n local iteration=\"$1\"; shift\n local arch=\"$1\"; shift\n if [[ \"$iteration\" == \"\" ]]; then\n iteration=\"$(default_iteration \"$pkgname\" \"$version\" \"$pkgtype\")\"\n fi\n\n if [[ \"$arch\" == \"\" ]]; then\n native_arch=$(get_native_arch)\n rpm_native_arch=\"x86_64\"\n rpm_architecture=\"$rpm_native_arch\"\n deb_architecture=\"$native_arch\"\n\n if [[ \"$pkgtype\" =~ ^(src)$ ]]; then\n rpm_architecture=\"noarch\"\n deb_architecture=\"all\"\n fi\n else\n rpm_architecture=$arch\n deb_architecture=$arch\n fi\n\n local complete_pkgname=\"${pkgname}_$version${iteration:+-$iteration}_$deb_architecture.deb\"\n if [[ \"$FORMAT\" == \"rpm\" ]]; then\n # rpm packages get iteration 1 if we don't supply one\n iteration=${iteration:-1}\n complete_pkgname=\"$pkgname-$version-${iteration}.$rpm_architecture.rpm\"\n fi\n __returnvar=${complete_pkgname}\n}\n\n# Test if the package already exists, if not return 0, if it does return 1\ntest_package_presence() {\n local pkgname=\"$1\"; shift\n local version=\"$1\"; shift\n local pkgtype=\"$1\"; shift\n local iteration=\"$1\"; shift\n local arch=\"$1\"; shift\n if [[ -n \"$ONLY_BUILD\" ]] && [[ \"$pkgname\" != \"$ONLY_BUILD\" ]] ; then\n return 1\n fi\n\n local full_pkgname\n get_complete_package_name full_pkgname \"$pkgname\" \"$version\" \"$pkgtype\" \"$iteration\" \"$arch\"\n\n # See if we can skip building the package, only if it already exists in the\n # processed/ directory. If so, move it back to the packages directory to make\n # sure it gets picked up by the test and/or upload steps.\n # Get the list of packages from the repos\n\n local pkg_url\n if [[ \"$FORCE_BUILD\" == \"1\" ]]; then\n echo \"Package $full_pkgname build forced with --force-build, building\"\n return 0\n elif [[ \"$FORMAT\" == \"deb\" ]]; then\n local codename\n case \"$TARGET\" in\n debian12) codename=bookworm ;;\n ubuntu2204) codename=jammy ;;\n ubuntu2404) codename=noble ;;\n *)\n echo \"FIXME: Don't know deb URL path for $TARGET, building\"\n return 0\n ;;\n esac\n local repo_subdir\n if [ ${pkgname:0:3} = \"lib\" ]; then\n repo_subdir=${pkgname:0:4}\n else\n repo_subdir=${pkgname:0:1}\n fi\n pkg_url=\"http://apt.arvados.org/$codename/pool/main/$repo_subdir/$pkgname/$full_pkgname\"\n else\n local rpm_root\n case \"$TARGET\" in\n rocky8 | rocky9 | rocky10 ) rpm_root=\"RHEL/${TARGET#rocky}/dev\" ;;\n *)\n echo \"FIXME: Don't know RPM URL path for $TARGET, building\"\n return 0\n ;;\n esac\n pkg_url=\"https://rpm.arvados.org/$rpm_root/$arch/$full_pkgname\"\n fi\n\n if curl -fs -o \"$WORKSPACE/packages/$TARGET/$full_pkgname\" \"$pkg_url\"; then\n echo \"Package $full_pkgname exists upstream, not rebuilding, downloading instead!\"\n return 1\n elif [[ -f \"$WORKSPACE/packages/$TARGET/processed/$full_pkgname\" ]]; then\n echo \"Package $full_pkgname exists, not rebuilding!\"\n return 1\n else\n echo \"Package $full_pkgname not found, building\"\n return 0\n fi\n}\n\nhandle_rails_package() {\n local pkgname=\"$1\"; shift\n\n if [[ -n \"$ONLY_BUILD\" ]] && [[ \"$pkgname\" != \"$ONLY_BUILD\" ]] ; then\n return 0\n fi\n local srcdir=\"$1\"; shift\n cd \"$srcdir\"\n local license_path=\"$1\"; shift\n local version=\"$(rails_package_version \"$pkgname\" \"$srcdir\")\"\n echo \"$version\" >package-build.version\n local scripts_dir=\"$(mktemp --tmpdir -d \"$pkgname-XXXXXXXX.scripts\")\" && \\\n (\n set -e\n _build_rails_package_scripts \"$pkgname\" \"$scripts_dir\"\n cd \"$srcdir\"\n mkdir -p tmp\n git rev-parse HEAD >git-commit.version\n # Prevent `bundle cache` from seeing system-wide gems and skipping\n # their download. This depends on the Bundler install set up\n # in the arvados_ruby Ansible role. See there for more background.\n export GEM_HOME=/opt/arvados-bundler\n export GEM_PATH=\"$GEM_HOME\"\n # Please make sure you read `bundle help config` carefully before you\n # modify any of these settings. Some of their names are not intuitive.\n #\n # `bundle cache` caches from Git and paths, not just rubygems.org.\n bundle config set cache_all true\n # `bundle cache` caches for all platforms listed in `Gemfile.lock`.\n bundle config set cache_all_platforms true\n # Avoid loading system-wide gems (although this seems to not work 100%).\n bundle config set disable_shared_gems true\n # `bundle cache` only downloads gems, doesn't install them.\n # Our Rails postinst script does the install step.\n bundle config set no_install true\n # Do not install gem sets unnecessary for production.\n bundle config set without development:test\n\n bundle cache\n # Configuration after this point is for the installed package but only\n # makes sense to set *after* running `bundle cache`.\n #\n # Install with deployment settings.\n bundle config set deployment true\n # Install gems to a dedicated path that is only used by RailsAPI\n # (but shared across versions for efficiency).\n bundle config set path /var/www/arvados-api/shared/vendor_bundle\n )\n if [[ 0 != \"$?\" ]] || ! cd \"$WORKSPACE/packages/$TARGET\"; then\n echo \"ERROR: $pkgname package prep failed\" >&2\n rm -rf \"$scripts_dir\"\n EXITCODE=1\n return 1\n fi\n local railsdir=\"/var/www/${pkgname%-server}/current\"\n local -a pos_args=(\"$srcdir/=$railsdir\" \"$pkgname\" dir \"$version\")\n local -a switches=(--after-install \"$scripts_dir/postinst\"\n --before-remove \"$scripts_dir/prerm\"\n --after-remove \"$scripts_dir/postrm\")\n if [[ -z \"$ARVADOS_BUILDING_VERSION\" ]]; then\n switches+=(--iteration $RAILS_PACKAGE_ITERATION)\n fi\n # For some reason fpm excludes need to not start with /.\n local exclude_root=\"${railsdir#/}\"\n for exclude in tmp log coverage Capfile\\* \\\n config/deploy\\* \\\n config/application.yml \\\n config/database.yml \\\n \\*.service; do\n switches+=(-x \"$exclude_root/$exclude\")\n done\n fpm_build \"${srcdir}\" \"${pos_args[@]}\" \"${switches[@]}\" \\\n -x \"$exclude_root/vendor/cache-*\" \\\n -x \"$exclude_root/vendor/bundle\" \"$@\" \\\n \"$license_path=$railsdir/$(basename \"$license_path\")\" \\\n \"$srcdir/arvados-railsapi.service=/lib/systemd/system/arvados-railsapi.service\"\n rm -rf \"$scripts_dir\"\n}\n\n# Usage: handle_api_server [amd64]\nhandle_api_server () {\n local target_arch=\"${1:-amd64}\"; shift\n\n if [[ -n \"$ONLY_BUILD\" ]] && [[ \"$ONLY_BUILD\" != \"arvados-api-server\" ]] ; then\n debug_echo -e \"Skipping build of arvados-api-server package.\"\n return 0\n fi\n\n native_arch=$(get_native_arch)\n if [[ \"$target_arch\" != \"$native_arch\" ]]; then\n echo \"Error: no cross compilation support for Rails yet, can not build arvados-api-server for $ARCH\"\n echo\n exit 1\n fi\n\n # Build the API server package\n test_rails_package_presence arvados-api-server \"$WORKSPACE/services/api\"\n if [[ \"$?\" == \"0\" ]]; then\n calculate_go_package_version arvados_server_version cmd/arvados-server\n arvados_server_iteration=$(default_iteration \"arvados-server\" \"$arvados_server_version\" \"go\")\n handle_rails_package arvados-api-server \"$WORKSPACE/services/api\" \\\n \"$WORKSPACE/agpl-3.0.txt\" --url=\"https://arvados.org\" \\\n --description=\"Arvados API server - Arvados is a free and open source platform for big data science.\" \\\n --license=\"GNU Affero General Public License, version 3.0\" --depends \"arvados-server = ${arvados_server_version}-${arvados_server_iteration}\"\n fi\n}\n\n# Usage: handle_arvados_src\nhandle_arvados_src () {\n if [[ -n \"$ONLY_BUILD\" ]] && [[ \"$ONLY_BUILD\" != \"arvados-src\" ]] ; then\n debug_echo -e \"Skipping build of arvados-src package.\"\n return 0\n fi\n # arvados-src\n (\n cd \"$WORKSPACE\"\n COMMIT_HASH=$(format_last_commit_here \"%H\")\n arvados_src_version=\"$(version_from_git)\"\n\n cd $WORKSPACE/packages/$TARGET\n test_package_presence arvados-src \"$arvados_src_version\" src \"\"\n\n if [[ \"$?\" == \"0\" ]]; then\n cd \"$WORKSPACE\"\n SRC_BUILD_DIR=$(mktemp -d)\n # mktemp creates the directory with 0700 permissions by default\n chmod 755 $SRC_BUILD_DIR\n git clone $DASHQ_UNLESS_DEBUG \"$WORKSPACE/.git\" \"$SRC_BUILD_DIR\"\n cd \"$SRC_BUILD_DIR\"\n\n # go into detached-head state\n git checkout $DASHQ_UNLESS_DEBUG \"$COMMIT_HASH\"\n echo \"$COMMIT_HASH\" >git-commit.version\n\n cd $WORKSPACE/packages/$TARGET\n fpm_build \"$WORKSPACE\" $SRC_BUILD_DIR/=/usr/local/arvados/src arvados-src 'dir' \"$arvados_src_version\" \"--exclude=usr/local/arvados/src/.git\" \"--url=https://arvados.org\" \"--license=GNU Affero General Public License, version 3.0\" \"--description=The Arvados source code\" \"--architecture=all\"\n\n rm -rf \"$SRC_BUILD_DIR\"\n fi\n )\n}\n\nsetup_build_virtualenv() {\n PYTHON_BUILDROOT=\"$(mktemp --directory --tmpdir pybuild.XXXXXXXX)\"\n \"$PYTHON3_EXECUTABLE\" -m venv \"$PYTHON_BUILDROOT/venv\"\n \"$PYTHON_BUILDROOT/venv/bin/pip\" install -r \"$WORKSPACE/build/requirements.build-packages.txt\"\n mkdir \"$PYTHON_BUILDROOT/wheelhouse\"\n}\n\n# Build python packages with a virtualenv built-in\n# Usage: fpm_build_virtualenv arvados-python-client sdk/python [deb|rpm] [amd64]\nfpm_build_virtualenv () {\n local pkg=$1; shift\n local pkg_dir=$1; shift\n local package_format=\"$1\"; shift\n local target_arch=\"${1:-amd64}\"; shift\n\n fpm_build_virtualenv_worker \"$pkg\" \"$pkg_dir\" \"$package_format\" amd64 amd64\n}\n\n# Build python packages with a virtualenv built-in\n# Usage: fpm_build_virtualenv_worker arvados-python-client sdk/python python3 [deb|rpm] [amd64] [amd64]\nfpm_build_virtualenv_worker () {\n PKG=$1; shift\n PKG_DIR=$1; shift\n local package_format=\"$1\"; shift\n local native_arch=\"${1:-amd64}\"; shift\n local target_arch=${1:-amd64}; shift\n\n # Set up\n STDOUT_IF_DEBUG=/dev/null\n STDERR_IF_DEBUG=/dev/null\n DASHQ_UNLESS_DEBUG=-q\n if [[ \"$DEBUG\" != \"0\" ]]; then\n STDOUT_IF_DEBUG=/dev/stdout\n STDERR_IF_DEBUG=/dev/stderr\n DASHQ_UNLESS_DEBUG=\n fi\n if [[ \"$ARVADOS_BUILDING_ITERATION\" == \"\" ]]; then\n ARVADOS_BUILDING_ITERATION=1\n fi\n\n PACKAGE=\"$PKG_DIR\"\n PACKAGE_PREFIX=$PYTHON3_PKG_PREFIX\n if [[ \"$PKG\" != \"arvados-docker-cleaner\" ]]; then\n PYTHON_PKG=$PACKAGE_PREFIX-$PKG\n else\n # Exception to our package naming convention\n PYTHON_PKG=$PKG\n fi\n\n # We must always add a wheel to our repository, even if we're not building\n # this distro package, because it might be a dependency for a later\n # package we do build.\n if [[ \"$PKG_DIR\" =~ ^.=[0-9]+\\. ]]; then\n # Not source to build, but a version to download.\n # The rest of the function expects a filesystem path, so set one afterwards.\n \"$PYTHON_BUILDROOT/venv/bin/pip\" download --dest=\"$PYTHON_BUILDROOT/wheelhouse\" \"$PKG$PKG_DIR\" \\\n && PKG_DIR=\"$PYTHON_BUILDROOT/nonexistent\"\n else\n # Make PKG_DIR absolute.\n PKG_DIR=\"$(env -C \"$WORKSPACE\" readlink -e \"$PKG_DIR\")\"\n \"$PYTHON_BUILDROOT/venv/bin/python\" -m build --outdir=\"$PYTHON_BUILDROOT/wheelhouse\" \"$PKG_DIR\"\n fi\n if [[ $? -ne 0 ]]; then\n printf \"Error, unable to download/build wheel for %s @ %s\\n\" \"$PKG\" \"$PKG_DIR\"\n exit 1\n fi\n\n if [[ -n \"$ONLY_BUILD\" ]] && [[ \"$PYTHON_PKG\" != \"$ONLY_BUILD\" ]] && [[ \"$PKG\" != \"$ONLY_BUILD\" ]]; then\n return 0\n elif ! \"$PYTHON_BUILDROOT/venv/bin/piprepo\" build \"$PYTHON_BUILDROOT/wheelhouse\"; then\n printf \"Error, unable to update local wheel repository\\n\"\n exit 1\n fi\n\n local venv_dir=\"/usr/lib/$PYTHON_PKG\"\n echo \"Creating virtualenv...\"\n if ! \"$PYTHON3_EXECUTABLE\" -m venv \"$venv_dir\"; then\n printf \"Error, unable to run\\n %s -m venv %s\\n\" \"$PYTHON3_EXECUTABLE\" \"$venv_dir\"\n exit 1\n # We must have the dependency resolver introduced in late 2020 for the rest\n # of our install process to work.\n # \n elif ! \"$venv_dir/bin/pip\" install \"pip>=20.3\"; then\n printf \"Error, unable to run\\n %s/bin/pip install 'pip>=20.3'\\n\" \"$venv_dir\"\n exit 1\n fi\n\n local pip_wheel=\"$(ls --sort=time --reverse \"$PYTHON_BUILDROOT/wheelhouse/$(echo \"$PKG\" | sed s/-/_/g)-\"*.whl | tail -n1)\"\n if [[ -z \"$pip_wheel\" ]]; then\n printf \"Error, unable to find built wheel for $PKG\\n\"\n exit 1\n elif ! \"$venv_dir/bin/pip\" install $DASHQ_UNLESS_DEBUG $CACHE_FLAG --extra-index-url=\"file://$PYTHON_BUILDROOT/wheelhouse/simple\" \"$pip_wheel\"; then\n printf \"Error, unable to run\n %s/bin/pip install $DASHQ_UNLESS_DEBUG $CACHE_FLAG --extra-index-url=file://%s %s\n\" \"$venv_dir\" \"$PYTHON_BUILDROOT/wheelhouse/simple\" \"$pip_wheel\"\n exit 1\n fi\n\n # Determine the package version from the wheel\n PYTHON_VERSION=\"$(\"$venv_dir/bin/python\" \"$WORKSPACE/build/pypkg_info.py\" metadata \"$PKG\" Version)\"\n UNFILTERED_PYTHON_VERSION=\"$(echo \"$PYTHON_VERSION\" | sed 's/\\.dev/~dev/; s/\\([0-9]\\)rc/\\1~rc/')\"\n\n # See if we actually need to build this package; does it exist already?\n # We can't do this earlier than here, because we need PYTHON_VERSION.\n if ! test_package_presence \"$PYTHON_PKG\" \"$UNFILTERED_PYTHON_VERSION\" python3 \"$ARVADOS_BUILDING_ITERATION\" \"$target_arch\"; then\n return 0\n fi\n echo \"Building $package_format ($target_arch) package for $PKG from $PKG_DIR\"\n\n local lic_key=\"$(\"$venv_dir/bin/python3\" \"$WORKSPACE/build/pypkg_info.py\" metadata \"$PKG\" License-File)\"\n local lic_desc=\"${LICENSE_FILE_NAME_MAP[$lic_key]}\"\n if [[ -z \"$lic_desc\" ]]; then\n echo \"Error, unable to determine license metadata for $PKG\" >&2\n exit 1\n fi\n # Using `env -C` sets the directory where the package is built.\n # Using `fpm --chdir` sets the root directory for source arguments.\n declare -a COMMAND_ARR=(\n env -C \"$PYTHON_BUILDROOT\" fpm\n --chdir=\"$venv_dir\"\n --name=\"$PYTHON_PKG\"\n --version=\"$UNFILTERED_PYTHON_VERSION\"\n --input-type=dir\n --output-type=\"$package_format\"\n --depends=\"$PYTHON3_PACKAGE\"\n --iteration=\"$ARVADOS_BUILDING_ITERATION\"\n --replaces=\"python-$PKG\"\n --url=\"https://arvados.org\"\n --license=\"$lic_desc\"\n )\n # Append fpm flags corresponding to Python package metadata.\n readarray -d \"\" -O \"${#COMMAND_ARR[@]}\" -t COMMAND_ARR < \\\n <(\"$venv_dir/bin/python3\" \"$WORKSPACE/build/pypkg_info.py\" \\\n --delimiter=\\\\0 --format=fpm \\\n metadata \"$PKG\" Summary)\n\n if [[ -n \"$target_arch\" ]] && [[ \"$target_arch\" != \"amd64\" ]]; then\n COMMAND_ARR+=(\"-a$target_arch\")\n fi\n\n if [[ \"$MAINTAINER\" != \"\" ]]; then\n COMMAND_ARR+=('--maintainer' \"$MAINTAINER\")\n fi\n\n if [[ \"$VENDOR\" != \"\" ]]; then\n COMMAND_ARR+=('--vendor' \"$VENDOR\")\n fi\n\n if [[ \"$DEBUG\" != \"0\" ]]; then\n COMMAND_ARR+=('--verbose' '--log' 'info')\n fi\n\n systemd_unit=\"$PKG_DIR/$PKG.service\"\n if [[ -e \"${systemd_unit}\" ]]; then\n COMMAND_ARR+=('--after-install' \"${WORKSPACE}/build/go-python-package-scripts/postinst\")\n COMMAND_ARR+=('--before-remove' \"${WORKSPACE}/build/go-python-package-scripts/prerm\")\n fi\n\n case \"$package_format\" in\n deb)\n COMMAND_ARR+=(\n # Avoid warning\n --deb-no-default-config-files\n ) ;;\n rpm)\n COMMAND_ARR+=(\n # Conflict with older packages we used to publish\n --conflicts \"rh-python36-python-$PKG\"\n # Do not generate /usr/lib/.build-id links on RH8+\n # (otherwise our packages conflict with platform-python)\n --rpm-rpmbuild-define \"_build_id_links none\"\n ) ;;\n esac\n\n # Append --depends X and other arguments specified by fpm-info.sh in\n # the package source dir. These are added last so they can override\n # the arguments added by this script.\n declare -a fpm_args=()\n declare -a fpm_depends=()\n\n fpminfo=\"$PKG_DIR/fpm-info.sh\"\n if [[ -e \"$fpminfo\" ]]; then\n echo \"Loading fpm overrides from $fpminfo\"\n if ! source \"$fpminfo\"; then\n echo \"Error, unable to source $WORKSPACE/$PKG_DIR/fpm-info.sh for $PKG\"\n exit 1\n fi\n fi\n\n for i in \"${fpm_depends[@]}\"; do\n COMMAND_ARR+=('--depends' \"$i\")\n done\n\n # make sure the systemd service file ends up in the right place\n # used by arvados-docker-cleaner\n if [[ -e \"${systemd_unit}\" ]]; then\n COMMAND_ARR+=(\"share/doc/$PKG/$PKG.service=/lib/systemd/system/$PKG.service\")\n fi\n\n COMMAND_ARR+=(\"${fpm_args[@]}\")\n\n while read -d \"\" binpath; do\n COMMAND_ARR+=(\"$binpath=/usr/$binpath\")\n done < <(\"$venv_dir/bin/python3\" \"$WORKSPACE/build/pypkg_info.py\" --delimiter=\\\\0 binfiles \"$PKG\")\n\n # the python3-arvados-cwl-runner package comes with cwltool, expose that version\n if [[ \"$PKG\" == arvados-cwl-runner ]]; then\n COMMAND_ARR+=(\"bin/cwltool=/usr/bin/cwltool\")\n fi\n\n COMMAND_ARR+=(\".=$venv_dir\")\n\n debug_echo -e \"\\n${COMMAND_ARR[@]}\\n\"\n\n FPM_RESULTS=$(\"${COMMAND_ARR[@]}\")\n FPM_EXIT_CODE=$?\n\n # if something went wrong and debug is off, print out the fpm command that errored\n if ! fpm_verify $FPM_EXIT_CODE $FPM_RESULTS && [[ \"$STDOUT_IF_DEBUG\" == \"/dev/null\" ]]; then\n echo \"fpm returned an error executing the command:\"\n echo\n echo -e \"\\n${COMMAND_ARR[@]}\\n\"\n else\n ls \"$PYTHON_BUILDROOT\"/*.\"$package_format\"\n mv \"$PYTHON_BUILDROOT\"/*.\"$package_format\" \"$WORKSPACE/packages/$TARGET/\"\n fi\n echo\n}\n\n# Build packages for everything\nfpm_build() {\n # Source dir where fpm-info.sh (if any) will be found.\n SRC_DIR=$1\n shift\n # The package source. Depending on the source type, this can be a\n # path, or the name of the package in an upstream repository (e.g.,\n # pip).\n PACKAGE=$1\n shift\n # The name of the package to build.\n PACKAGE_NAME=$1\n shift\n # The type of source package. Passed to fpm -s. Default \"dir\".\n PACKAGE_TYPE=${1:-dir}\n shift\n # Optional: the package version number. Passed to fpm -v.\n VERSION=$1\n shift\n\n if [[ -n \"$ONLY_BUILD\" ]] && [[ \"$PACKAGE_NAME\" != \"$ONLY_BUILD\" ]] && [[ \"$PACKAGE\" != \"$ONLY_BUILD\" ]] ; then\n return 0\n fi\n\n local default_iteration_value=\"$(default_iteration \"$PACKAGE\" \"$VERSION\" \"$PACKAGE_TYPE\")\"\n\n declare -a COMMAND_ARR=(\"fpm\" \"-s\" \"$PACKAGE_TYPE\" \"-t\" \"$FORMAT\")\n if [ python = \"$PACKAGE_TYPE\" ] && [ deb = \"$FORMAT\" ]; then\n # Dependencies are built from Python package metadata. Since that\n # will never refer to Debian package iterations, it doesn't make sense\n # to enforce those in the .deb dependencies.\n COMMAND_ARR+=(--deb-ignore-iteration-in-dependencies)\n fi\n\n if [[ \"$DEBUG\" != \"0\" ]]; then\n COMMAND_ARR+=('--verbose' '--log' 'info')\n fi\n\n if [[ -n \"$PACKAGE_NAME\" ]]; then\n COMMAND_ARR+=('-n' \"$PACKAGE_NAME\")\n fi\n\n if [[ \"$MAINTAINER\" != \"\" ]]; then\n COMMAND_ARR+=('--maintainer' \"$MAINTAINER\")\n fi\n\n if [[ \"$VENDOR\" != \"\" ]]; then\n COMMAND_ARR+=('--vendor' \"$VENDOR\")\n fi\n\n if [[ \"$VERSION\" != \"\" ]]; then\n COMMAND_ARR+=('-v' \"$VERSION\")\n fi\n if [[ -n \"$default_iteration_value\" ]]; then\n # We can always add an --iteration here. If another one is specified in $@,\n # that will take precedence, as desired.\n COMMAND_ARR+=(--iteration \"$default_iteration_value\")\n fi\n\n # Append --depends X and other arguments specified by fpm-info.sh in\n # the package source dir. These are added last so they can override\n # the arguments added by this script.\n declare -a fpm_args=()\n declare -a build_depends=()\n declare -a fpm_depends=()\n declare -a fpm_conflicts=()\n declare -a fpm_exclude=()\n if [[ ! -d \"$SRC_DIR\" ]]; then\n echo >&2 \"BUG: looking in wrong dir for fpm-info.sh: $pkgdir\"\n exit 1\n fi\n fpminfo=\"${SRC_DIR}/fpm-info.sh\"\n if [[ -e \"$fpminfo\" ]]; then\n debug_echo \"Loading fpm overrides from $fpminfo\"\n source \"$fpminfo\"\n fi\n for pkg in \"${build_depends[@]}\"; do\n if [[ $TARGET =~ debian|ubuntu ]]; then\n pkg_deb=$(ls \"$WORKSPACE/packages/$TARGET/$pkg_\"*.deb | sort -rg | awk 'NR==1')\n if [[ -e $pkg_deb ]]; then\n echo \"Installing build_dep $pkg from $pkg_deb\"\n dpkg -i \"$pkg_deb\"\n else\n echo \"Attemping to install build_dep $pkg using apt-get\"\n apt-get install -y \"$pkg\"\n fi\n apt-get -y -f install\n else\n pkg_rpm=$(ls \"$WORKSPACE/packages/$TARGET/$pkg\"-[0-9]*.rpm | sort -rg | awk 'NR==1')\n if [[ -e $pkg_rpm ]]; then\n echo \"Installing build_dep $pkg from $pkg_rpm\"\n rpm -i \"$pkg_rpm\"\n else\n echo \"Attemping to install build_dep $pkg\"\n rpm -i \"$pkg\"\n fi\n fi\n done\n for i in \"${fpm_depends[@]}\"; do\n COMMAND_ARR+=('--depends' \"$i\")\n done\n for i in \"${fpm_conflicts[@]}\"; do\n COMMAND_ARR+=('--conflicts' \"$i\")\n done\n for i in \"${fpm_exclude[@]}\"; do\n COMMAND_ARR+=('--exclude' \"$i\")\n done\n\n COMMAND_ARR+=(\"${fpm_args[@]}\")\n\n # Append remaining function arguments directly to fpm's command line.\n for i; do\n COMMAND_ARR+=(\"$i\")\n done\n\n COMMAND_ARR+=(\"$PACKAGE\")\n\n debug_echo -e \"\\n${COMMAND_ARR[@]}\\n\"\n\n FPM_RESULTS=$(\"${COMMAND_ARR[@]}\")\n FPM_EXIT_CODE=$?\n echo \"fpm: exit code $FPM_EXIT_CODE\" >>$STDOUT_IF_DEBUG\n echo \"$FPM_RESULTS\" >>$STDOUT_IF_DEBUG\n\n fpm_verify $FPM_EXIT_CODE $FPM_RESULTS\n\n # if something went wrong and debug is off, print out the fpm command that errored\n if [[ 0 -ne $? ]] && [[ \"$STDOUT_IF_DEBUG\" == \"/dev/null\" ]]; then\n echo -e \"\\n${COMMAND_ARR[@]}\\n\"\n fi\n}\n\n# verify build results\nfpm_verify () {\n FPM_EXIT_CODE=$1\n shift\n FPM_RESULTS=$@\n\n FPM_PACKAGE_NAME=''\n if [[ $FPM_RESULTS =~ ([A-Za-z0-9_\\.~-]*\\.)(deb|rpm) ]]; then\n FPM_PACKAGE_NAME=${BASH_REMATCH[1]}${BASH_REMATCH[2]}\n fi\n\n if [[ \"$FPM_PACKAGE_NAME\" == \"\" ]]; then\n EXITCODE=1\n echo\n echo \"Error: $PACKAGE: Unable to figure out package name from fpm results:\"\n echo\n echo $FPM_RESULTS\n echo\n return 1\n elif [[ \"$FPM_RESULTS\" =~ \"File already exists\" ]]; then\n echo \"Package $FPM_PACKAGE_NAME exists, not rebuilding\"\n return 0\n elif [[ 0 -ne \"$FPM_EXIT_CODE\" ]]; then\n EXITCODE=1\n echo \"Error building package for $1:\\n $FPM_RESULTS\"\n return 1\n fi\n}\n\ninstall_package() {\n PACKAGES=$@\n if [[ \"$FORMAT\" == \"deb\" ]]; then\n $SUDO apt-get install $PACKAGES --yes\n elif [[ \"$FORMAT\" == \"rpm\" ]]; then\n $SUDO yum -q -y install $PACKAGES\n fi\n}\n\ntitle() {\n printf '%s %s\\n' \"=======\" \"$1\"\n}\n\ncheckexit() {\n if [[ \"$1\" != \"0\" ]]; then\n title \"$2 -- FAILED\"\n failures+=(\"$2 (`timer`)\")\n else\n successes+=(\"$2 (`timer`)\")\n fi\n}\n\ntimer_reset() {\n t0=$SECONDS\n}\n\ntimer() {\n if [[ -n \"$t0\" ]]; then\n echo -n \"$(($SECONDS - $t0))s\"\n fi\n}\n\nreport_outcomes() {\n for x in \"${successes[@]}\"\n do\n echo \"Pass: $x\"\n done\n\n if [[ ${#failures[@]} == 0 ]]\n then\n if [[ ${#successes[@]} != 0 ]]; then\n echo \"All test suites passed.\"\n fi\n else\n echo \"Failures (${#failures[@]}):\"\n for x in \"${failures[@]}\"\n do\n echo \"Fail: $x\"\n done\n fi\n}\n" }, { "path": "build/run-tests.sh", "content": "#!/bin/bash\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\nCOLUMNS=80\n. `dirname \"$(readlink -f \"$0\")\"`/run-library.sh\n\nread -rd \"\\000\" helpmessage <\nexport PYTHONWARNINGS=\"ignore::FutureWarning:google.api_core._python_version_support,${PYTHONWARNINGS:-\\\ndefault::DeprecationWarning:__main__\\\n,ignore::DeprecationWarning\\\n,ignore::PendingDeprecationWarning\\\n,ignore::ImportWarning\\\n,ignore::ResourceWarning\\\n}\"\n\n# setup_ruby_environment will set this to the path of the `bundle` executable\n# it installs. This stub will cause commands to fail if they try to run before\n# that.\nBUNDLE=false\n\nshort=\nonly_install=\ntemp=\ntemp_preserve=\n\nignore_sigint=\n\nclear_temp() {\n if [[ -z \"$temp\" ]]; then\n # we did not even get as far as making a temp dir\n :\n elif [[ -z \"$temp_preserve\" ]]; then\n # Go creates readonly dirs in the module cache, which cause\n # \"rm -rf\" to fail unless we chmod first.\n chmod -R u+w \"$temp\"\n rm -rf \"$temp\"\n else\n echo \"Leaving behind temp dirs in $temp\"\n fi\n}\n\nfatal() {\n clear_temp\n echo >&2 \"Fatal: $* (encountered in ${FUNCNAME[1]} at ${BASH_SOURCE[1]} line ${BASH_LINENO[0]})\"\n exit 1\n}\n\nexit_cleanly() {\n trap - INT\n stop_services\n rotate_logfile \"$WORKSPACE/services/api/log/\" \"test.log\"\n report_outcomes\n clear_temp\n exit ${#failures}\n}\n\nsanity_checks() {\n [[ -n \"${skip[sanity]}\" ]] && return 0\n ( [[ -n \"$WORKSPACE\" ]] && [[ -d \"$WORKSPACE/services\" ]] ) \\\n || fatal \"WORKSPACE environment variable not set to a source directory (see: $0 --help)\"\n [[ -z \"$CONFIGSRC\" ]] || [[ -s \"$CONFIGSRC/config.yml\" ]] \\\n || fatal \"CONFIGSRC is $CONFIGSRC but '$CONFIGSRC/config.yml' is empty or not found (see: $0 --help)\"\n echo Checking dependencies:\n echo \"locale: ${LANG}\"\n [[ \"$(locale charmap)\" = \"UTF-8\" ]] \\\n || fatal \"Locale '${LANG}' is broken/missing. Try: echo ${LANG} | sudo tee -a /etc/locale.gen && sudo locale-gen\"\n echo -n 'ruby: '\n ruby -v \\\n || fatal \"No ruby. Install >=2.7 from package or source\"\n echo -n 'go: '\n go version \\\n || fatal \"No go binary. See http://golang.org/doc/install\"\n [[ $(go version) =~ go1.([0-9]+) ]] && [[ ${BASH_REMATCH[1]} -ge 12 ]] \\\n || fatal \"Go >= 1.12 required. See http://golang.org/doc/install\"\n echo -n 'gcc: '\n gcc --version | egrep ^gcc \\\n || fatal \"No gcc. Try: apt-get install build-essential\"\n echo -n 'fuse.h: '\n find /usr/include -path '*fuse/fuse.h' | egrep --max-count=1 . \\\n || fatal \"No fuse/fuse.h. Try: apt-get install libfuse-dev\"\n echo -n 'virtualenv: '\n python3 -m venv --help | grep -q '^usage: venv ' \\\n && echo \"venv module found\" \\\n || fatal \"No virtualenv. Try: apt-get install python3-venv\"\n which netstat \\\n || fatal \"No netstat. Try: apt-get install net-tools\"\n echo -n 'nginx: '\n PATH=\"$PATH:/sbin:/usr/sbin:/usr/local/sbin\" nginx -v \\\n || fatal \"No nginx. Try: apt-get install nginx\"\n echo -n 'npm: '\n npm --version \\\n || fatal \"No npm. Try: wget -O- https://nodejs.org/dist/v14.21.3/node-v14.21.3-linux-x64.tar.xz | sudo tar -C /usr/local -xJf - && sudo ln -s ../node-v14.21.3-linux-x64/bin/{node,npm} /usr/local/bin/\"\n echo -n 'cadaver: '\n cadaver --version | grep -w cadaver \\\n || fatal \"No cadaver. Try: apt-get install cadaver\"\n echo -n \"jq: \"\n jq --version ||\n fatal \"No jq. Try: apt-get install jq\"\n echo -n 'libcurl curl.h: '\n find /usr/include -path '*/curl/curl.h' | egrep --max-count=1 . \\\n || fatal \"No libcurl curl.h. Try: apt-get install libcurl4-gnutls-dev\"\n echo -n 'libpq libpq-fe.h: '\n find /usr/include -path '*/postgresql/libpq-fe.h' | egrep --max-count=1 . \\\n || fatal \"No libpq libpq-fe.h. Try: apt-get install libpq-dev\"\n echo -n 'libpam pam_appl.h: '\n find /usr/include -path '*/security/pam_appl.h' | egrep --max-count=1 . \\\n || fatal \"No libpam pam_appl.h. Try: apt-get install libpam0g-dev\"\n echo -n 'postgresql: '\n psql --version || fatal \"No postgresql. Try: apt-get install postgresql postgresql-client-common\"\n echo -n 'xvfb: '\n which Xvfb || fatal \"No xvfb. Try: apt-get install xvfb\"\n echo -n 'singularity: '\n singularity --version || fatal \"No singularity.\"\n echo -n 'docker client: '\n docker --version || echo \"WARNING: No docker client.\"\n echo -n 'docker server: '\n docker info --format='{{.ServerVersion}}' || echo \"WARNING: No docker server.\"\n\n if [[ \"$NEED_SDK_R\" = true ]]; then\n # R SDK stuff\n echo -n 'R: '\n which Rscript || fatal \"No Rscript. Try: apt-get install r-base\"\n echo -n 'testthat: '\n Rscript -e \"library('testthat')\" || fatal \"No testthat. Try: apt-get install r-cran-testthat\"\n # needed for roxygen2, needed for devtools, needed for R sdk\n pkg-config --exists libxml-2.0 || fatal \"No libxml2. Try: apt-get install libxml2-dev\"\n fi\n echo 'procs with /dev/fuse open:'\n find /proc/*/fd -lname /dev/fuse 2>/dev/null | cut -d/ -f3 | xargs --no-run-if-empty ps -lywww\n echo 'grep fuse /proc/self/mountinfo:'\n grep fuse /proc/self/mountinfo\n}\n\nrotate_logfile() {\n # i.e. rotate_logfile \"$WORKSPACE/services/api/log/\" \"test.log\"\n # $BUILD_NUMBER is set by Jenkins if this script is being called as part of a Jenkins run\n if [[ -f \"$1/$2\" ]]; then\n THEDATE=`date +%Y%m%d%H%M%S`\n mv \"$1/$2\" \"$1/$THEDATE-$BUILD_NUMBER-$2\"\n gzip \"$1/$THEDATE-$BUILD_NUMBER-$2\"\n fi\n}\n\ncheckpidfile() {\n svc=\"$1\"\n pid=\"$(cat \"$WORKSPACE/tmp/${svc}.pid\")\"\n if [[ -z \"$pid\" ]] || ! kill -0 \"$pid\"; then\n tail $WORKSPACE/tmp/${1}*.log\n echo \"${svc} pid ${pid} not running\"\n return 1\n fi\n echo \"${svc} pid ${pid} ok\"\n}\n\ncheckhealth() {\n svc=\"$1\"\n base=\"$(yq -r \"(.Clusters.zzzzz.Services.$svc.InternalURLs | keys)[0]\" \"$ARVADOS_CONFIG\")\"\n url=\"$base/_health/ping\"\n if ! curl -Ss -H \"Authorization: Bearer e687950a23c3a9bceec28c6223a06c79\" \"${url}\" | tee -a /dev/stderr | grep '\"OK\"'; then\n echo \"${url} failed\"\n return 1\n fi\n}\n\ncheckdiscoverydoc() {\n dd=\"https://${1}/discovery/v1/apis/arvados/v1/rest\"\n if ! (set -o pipefail; curl -fsk \"$dd\" | grep -q ^{ ); then\n echo >&2 \"ERROR: could not retrieve discovery doc from RailsAPI at $dd\"\n tail -v $WORKSPACE/tmp/railsapi.log\n return 1\n fi\n echo \"${dd} ok\"\n}\n\nstart_services() {\n if [[ -n \"$ARVADOS_TEST_API_HOST\" ]]; then\n return 0\n fi\n echo 'Starting API, controller, keepproxy, keep-web, ws, and nginx ssl proxy...'\n if [[ ! -d \"$WORKSPACE/services/api/log\" ]]; then\n mkdir -p \"$WORKSPACE/services/api/log\"\n fi\n # Remove empty api.pid file if it exists\n if [[ -f \"$WORKSPACE/tmp/api.pid\" && ! -s \"$WORKSPACE/tmp/api.pid\" ]]; then\n rm -f \"$WORKSPACE/tmp/api.pid\"\n fi\n all_services_stopped=\n fail=1\n\n cd \"$WORKSPACE\" \\\n && eval $(python3 sdk/python/tests/run_test_server.py start --auth admin) \\\n && export ARVADOS_TEST_API_HOST=\"$ARVADOS_API_HOST\" \\\n && export ARVADOS_TEST_API_INSTALLED=\"$$\" \\\n && checkpidfile api \\\n && checkdiscoverydoc $ARVADOS_API_HOST \\\n && eval $(python3 sdk/python/tests/run_test_server.py start_nginx) \\\n && checkpidfile nginx \\\n && python3 sdk/python/tests/run_test_server.py start_controller \\\n && checkpidfile controller \\\n && checkhealth Controller \\\n && checkdiscoverydoc $ARVADOS_API_HOST \\\n && python3 sdk/python/tests/run_test_server.py start_keep_proxy \\\n && checkpidfile keepproxy \\\n && python3 sdk/python/tests/run_test_server.py start_keep-web \\\n && checkpidfile keep-web \\\n && checkhealth WebDAV \\\n && python3 sdk/python/tests/run_test_server.py start_ws \\\n && checkpidfile ws \\\n && export ARVADOS_TEST_PROXY_SERVICES=1 \\\n && (env | egrep ^ARVADOS) \\\n && fail=0\n if [[ $fail != 0 ]]; then\n unset ARVADOS_TEST_API_HOST\n fi\n return $fail\n}\n\nstop_services() {\n if [[ -n \"$all_services_stopped\" ]]; then\n return\n fi\n unset ARVADOS_TEST_API_HOST ARVADOS_TEST_PROXY_SERVICES\n cd \"$WORKSPACE\" \\\n && python3 sdk/python/tests/run_test_server.py stop_nginx \\\n && python3 sdk/python/tests/run_test_server.py stop_ws \\\n && python3 sdk/python/tests/run_test_server.py stop_keep-web \\\n && python3 sdk/python/tests/run_test_server.py stop_keep_proxy \\\n && python3 sdk/python/tests/run_test_server.py stop_controller \\\n && python3 sdk/python/tests/run_test_server.py stop \\\n && all_services_stopped=1\n unset ARVADOS_CONFIG\n}\n\ninterrupt() {\n if [[ -n \"$ignore_sigint\" ]]; then\n echo >&2 \"ignored SIGINT\"\n return\n fi\n failures+=(\"($(basename $0) interrupted)\")\n exit_cleanly\n}\ntrap interrupt INT\n\nsetup_ruby_environment() {\n # When our \"bundle install\"s need to install new gems to\n # satisfy dependencies, we want them to go where \"gem install\n # --user-install\" would put them. (However, if the caller has\n # already set GEM_HOME, we assume that's where dependencies\n # should be installed, and we should leave it alone.)\n\n if [ -z \"$GEM_HOME\" ]; then\n user_gempath=\"$(gem env gempath)\"\n export GEM_HOME=\"${user_gempath%%:*}\"\n fi\n PATH=\"$(gem env gemdir)/bin:$PATH\"\n\n # When we build and install our own gems, we install them in our\n # $GEMHOME tmpdir, and we want them to be at the front of GEM_PATH and\n # PATH so integration tests prefer them over other versions that\n # happen to be installed in $user_gempath, system dirs, etc.\n\n tmpdir_gem_home=\"$(env - PATH=\"$PATH\" HOME=\"$GEMHOME\" gem env gempath | cut -f1 -d:)\"\n PATH=\"$tmpdir_gem_home/bin:$PATH\"\n export GEM_PATH=\"$tmpdir_gem_home:$(gem env gempath)\"\n\n echo \"Will install dependencies to $(gem env gemdir)\"\n echo \"Will install bundler and arvados gems to $tmpdir_gem_home\"\n echo \"Gem search path is GEM_PATH=$GEM_PATH\"\n gem install --user --no-document --conservative --version '~> 2.5.0' bundler \\\n || fatal 'install bundler'\n BUNDLE=\"$(gem contents --version '~> 2.5.0' bundler | grep -E '/(bin|exe)/bundle$' | tail -n1)\"\n if [[ ! -x \"$BUNDLE\" ]]; then\n BUNDLE=false\n fatal \"could not find 'bundle' executable after installation\"\n fi\n}\n\nwith_test_gemset() {\n GEM_HOME=\"$tmpdir_gem_home\" GEM_PATH=\"$tmpdir_gem_home\" \"$@\"\n}\n\nsetup_virtualenv() {\n if [[ -z \"${VENV3DIR:-}\" ]]; then\n fatal \"setup_virtualenv called before \\$VENV3DIR was set\"\n elif ! [[ -e \"$VENV3DIR/bin/activate\" ]]; then\n python3 -m venv \"$VENV3DIR\" || fatal \"virtualenv creation failed\"\n # Configure pip options we always want to use.\n \"$VENV3DIR/bin/pip\" config --quiet --site set global.disable-pip-version-check true\n \"$VENV3DIR/bin/pip\" config --quiet --site set global.no-input true\n \"$VENV3DIR/bin/pip\" config --quiet --site set global.no-python-version-warning true\n \"$VENV3DIR/bin/pip\" config --quiet --site set install.progress-bar off\n # If we didn't have a virtualenv before, we couldn't have started any\n # services. Set the flag used by stop_services to indicate that.\n all_services_stopped=1\n fi\n . \"$VENV3DIR/bin/activate\" || fatal \"virtualenv activation failed\"\n # We must have these in place *before* we install the PySDK below.\n pip install -r \"$WORKSPACE/build/requirements.tests.txt\" ||\n fatal \"failed to install Python requirements in virtualenv\"\n # run-tests.sh uses run_test_server.py from the Python SDK.\n do_install_once sdk/python pip || fatal \"failed to install PySDK in virtualenv\"\n}\n\ninitialize() {\n sanity_checks\n\n echo \"WORKSPACE=$WORKSPACE\"\n cd \"$WORKSPACE\"\n\n if [[ -z \"$temp\" ]]; then\n temp=\"$(mktemp -d)\"\n fi\n\n # Set up temporary install dirs (unless existing dirs were supplied)\n for tmpdir in VENV3DIR GOPATH GEMHOME R_LIBS\n do\n if [[ -z \"${!tmpdir}\" ]]; then\n eval \"$tmpdir\"=\"$temp/$tmpdir\"\n fi\n if ! [[ -d \"${!tmpdir}\" ]]; then\n mkdir \"${!tmpdir}\" || fatal \"can't create ${!tmpdir} (does $temp exist?)\"\n fi\n done\n\n rm -vf \"${WORKSPACE}/tmp/*.log\"\n\n export R_LIBS\n\n export GOPATH\n # Make sure our compiled binaries under test override anything\n # else that might be in the environment.\n export PATH=$GOPATH/bin:$PATH\n\n # Jenkins config requires that glob tmp/*.log match something. Ensure\n # that happens even if we don't end up running services that set up\n # logging.\n mkdir -p \"${WORKSPACE}/tmp/\" || fatal \"could not mkdir ${WORKSPACE}/tmp\"\n touch \"${WORKSPACE}/tmp/controller.log\" || fatal \"could not touch ${WORKSPACE}/tmp/controller.log\"\n\n unset http_proxy https_proxy no_proxy\n\n setup_ruby_environment\n setup_virtualenv\n\n echo \"PATH is $PATH\"\n}\n\ninstall_env() {\n go mod download || fatal \"Go deps failed\"\n which goimports >/dev/null || go install golang.org/x/tools/cmd/goimports@latest || fatal \"Go setup failed\"\n}\n\nretry() {\n remain=\"${repeat}\"\n while :\n do\n if ${@}; then\n if [[ \"$remain\" -gt 1 ]]; then\n remain=$((${remain}-1))\n title \"(repeating ${remain} more times)\"\n else\n break\n fi\n elif [[ \"$retry\" == 1 ]]; then\n read -p 'Try again? [Y/n] ' x\n if [[ \"$x\" != \"y\" ]] && [[ \"$x\" != \"\" ]]\n then\n break\n fi\n else\n break\n fi\n done\n}\n\ndo_test() {\n case \"${1}\" in\n services/workbench2_units | services/workbench2_integration)\n suite=services/workbench2\n ;;\n *)\n suite=\"${1}\"\n ;;\n esac\n if [[ -n \"${skip[$suite]}\" || \\\n -n \"${skip[$1]}\" || \\\n (${#only[@]} -ne 0 && ${only[$suite]} -eq 0 && ${only[$1]} -eq 0) ]]; then\n return 0\n fi\n case \"${1}\" in\n services/api)\n stop_services\n check_arvados_config \"$1\"\n ;;\n gofmt \\\n | arvados_version.py \\\n | doc \\\n | lib/boot \\\n | lib/cli \\\n | lib/cloud/azure \\\n | lib/cloud/cloudtest \\\n | lib/cloud/ec2 \\\n | lib/cmd \\\n | lib/dispatchcloud/sshexecutor \\\n | lib/dispatchcloud/worker \\\n | lib/install \\\n | services/workbench2_integration \\\n | services/workbench2_units \\\n )\n check_arvados_config \"$1\"\n # don't care whether services are running\n ;;\n *)\n check_arvados_config \"$1\"\n if ! start_services; then\n checkexit 1 \"$1 tests\"\n title \"test $1 -- failed to start services\"\n return 1\n fi\n ;;\n esac\n retry do_test_once ${@}\n}\n\ngo_ldflags() {\n version=${ARVADOS_VERSION:-$(git log -n1 --format=%H)-dev}\n echo \"-X git.arvados.org/arvados.git/lib/cmd.version=${version} -X main.version=${version} -s -w\"\n}\n\ndo_test_once() {\n unset result\n\n if [[ \"$2\" == pip && -n \"$interactive\" ]]; then\n # We test out of the virtualenv to test with full build artifacts.\n # We do this by setting --import-mode=append in pytest.ini.\n # Install the developer's latest changes to the virtualenv.\n # We need to do this before we start the test header+timer.\n do_install_once \"$1\" \"$2\" || return\n fi\n\n local -a targs=()\n case \"$1\" in\n sdk/cwl )\n # The CWL conformance/integration tests each take ~30\n # minutes. Before July 2025 they were outside the standard test\n # suite, so we deselect them by default for consistency.\n targs+=(-m \"not integration\")\n\n # The CWL conformance/integration tests expect keep\n # servers and crunch-dispatch-local.\n if ! ( env -C \"$WORKSPACE\" python3 sdk/python/tests/run_test_server.py start_keep \\\n && env -C \"$WORKSPACE\" python3 sdk/python/tests/run_test_server.py start_dispatch); then\n checkexit 1 \"$1 tests\"\n return 1\n fi\n ;;\n esac\n # Append the user's arguments to targs, respecting quoted strings.\n eval \"targs+=(${testargs[$1]})\"\n\n title \"test $1\"\n timer_reset\n\n result=\n if [[ \"$2\" == \"go\" ]]\n then\n covername=\"coverage-$(echo \"$1\" | sed -e 's/\\//_/g')\"\n coverflags=(\"-covermode=count\" \"-coverprofile=$WORKSPACE/tmp/.$covername.tmp\")\n if ! compgen -G \"$WORKSPACE/$1/*_test.go\" >/dev/null; then\n # Go 1.25, when invoked by Go 1.24 via \"toolchain go1.25\"\n # directive, fails with 'go: no such tool \"covdata\"' when\n # using $coverflags in a directory that has no tests. See\n # https://github.com/golang/go/issues/75031\n #\n # Workaround: skip coverflags when 'go test' is a no-op\n # anyway.\n coverflags=()\n fi\n testflags=()\n # We do \"go install\" here to catch compilation errors\n # before trying \"go test\". Otherwise, coverage-reporting\n # mode makes Go show the wrong line numbers when reporting\n # compilation errors.\n go install -ldflags \"$(go_ldflags)\" \"$WORKSPACE/$1\" && \\\n cd \"$WORKSPACE/$1\" && \\\n if [[ \"${#targs}\" -gt 0 ]]\n then\n # \"go test -check.vv giturl\" doesn't work, but this\n # does:\n go test ${short:+-short} ${testflags[@]} \"${targs[@]}\"\n else\n # The above form gets verbose even when testargs is\n # empty, so use this form in such cases:\n go test ${short:+-short} ${testflags[@]} ${coverflags[@]} \"git.arvados.org/arvados.git/$1\"\n fi\n result=${result:-$?}\n if [[ -f \"$WORKSPACE/tmp/.$covername.tmp\" ]]\n then\n go tool cover -html=\"$WORKSPACE/tmp/.$covername.tmp\" -o \"$WORKSPACE/tmp/$covername.html\"\n rm \"$WORKSPACE/tmp/.$covername.tmp\"\n fi\n [[ $result = 0 ]] && gofmt -e -d *.go\n elif [[ \"$2\" == \"pip\" ]]\n then\n tries=0\n while :\n do\n tries=$((${tries}+1))\n env -C \"$WORKSPACE/$1\" pytest \"${targs[@]}\"\n result=$?\n # pytest uses exit code 2 to mean \"test collection failed.\"\n # See discussion in FUSE's IntegrationTest and MountTestBase.\n if [[ ${tries} < 3 && ${result} == 2 ]]\n then\n printf '\\n*****\\n%s tests exited with code 2 -- retrying\\n*****\\n\\n' \"$1\"\n continue\n else\n break\n fi\n done\n elif [[ \"$2\" != \"\" ]]\n then\n \"test_$2\"\n else\n \"test_$1\"\n fi\n result=${result:-$?}\n checkexit $result \"$1 tests\"\n title \"test $1 -- `timer`\"\n if [[ \"$1\" == \"sdk/cwl\" ]]; then\n env -C \"$WORKSPACE\" python3 sdk/python/tests/run_test_server.py stop_keep\n env -C \"$WORKSPACE\" python3 sdk/python/tests/run_test_server.py stop_dispatch\n # Also reset test fixtures that were modified by the dispatcher\n env -C \"$WORKSPACE\" python3 sdk/python/tests/run_test_server.py database_reset\n fi\n return $result\n}\n\ncheck_arvados_config() {\n if [[ \"$1\" = \"env\" ]] ; then\n return\n fi\n if [[ -z \"$ARVADOS_CONFIG\" ]] ; then\n cd \"$WORKSPACE\"\n eval $(python3 sdk/python/tests/run_test_server.py setup_config)\n fi\n # Set all PostgreSQL connection variables, and write a .pgpass, to connect\n # to the test database, so test scripts can write `psql` commands with no\n # additional configuration.\n export PGPASSFILE=\"$WORKSPACE/tmp/.pgpass\"\n export PGDATABASE=\"$(yq -r .Clusters.zzzzz.PostgreSQL.Connection.dbname \"$ARVADOS_CONFIG\")\"\n export PGHOST=\"$(yq -r .Clusters.zzzzz.PostgreSQL.Connection.host \"$ARVADOS_CONFIG\")\"\n export PGPORT=\"$(yq -r .Clusters.zzzzz.PostgreSQL.Connection.port \"$ARVADOS_CONFIG\")\"\n export PGUSER=\"$(yq -r .Clusters.zzzzz.PostgreSQL.Connection.user \"$ARVADOS_CONFIG\")\"\n local pgpassword=\"$(yq -r .Clusters.zzzzz.PostgreSQL.Connection.password \"$ARVADOS_CONFIG\")\"\n echo \"$PGHOST:$PGPORT:$PGDATABASE:$PGUSER:$pgpassword\" >\"$PGPASSFILE\"\n chmod 0600 \"$PGPASSFILE\"\n}\n\ndo_install() {\n if [[ -n ${skip[\"install_$1\"]} || -n \"${skip[install]}\" || ( -n \"${only_install}\" && \"${only_install}\" != \"${1}\" && \"${only_install}\" != \"${2}\" ) ]]; then\n return 0\n fi\n check_arvados_config \"$1\"\n retry do_install_once ${@}\n}\n\ndo_install_once() {\n title \"install $1\"\n timer_reset\n\n result=\n if [[ \"$2\" == \"go\" ]]\n then\n go install -ldflags \"$(go_ldflags)\" \"$WORKSPACE/$1\"\n elif [[ \"$2\" == \"pip\" ]]\n then\n pip install \"$WORKSPACE/$1\"\n elif [[ \"$2\" != \"\" ]]\n then\n \"install_$2\"\n else\n \"install_$1\"\n fi\n result=${result:-$?}\n checkexit $result \"$1 install\"\n title \"install $1 -- `timer`\"\n return $result\n}\n\nbundle_install_trylocal() {\n (\n set -e\n echo \"(Running bundle install --local. 'could not find package' messages are OK.)\"\n if ! \"$BUNDLE\" install --local --no-deployment; then\n echo \"(Running bundle install again, without --local.)\"\n \"$BUNDLE\" install --no-deployment\n fi\n \"$BUNDLE\" package\n )\n}\n\ninstall_doc() {\n cd \"$WORKSPACE/doc\" \\\n && bundle_install_trylocal \\\n && rm -rf .site\n}\n\ninstall_gem() {\n gemname=$1\n srcpath=$2\n cd \"$WORKSPACE/$srcpath\" \\\n && bundle_install_trylocal \\\n && gem build \"$gemname.gemspec\" \\\n && with_test_gemset gem install --no-document $(ls -t \"$gemname\"-*.gem|head -n1)\n}\n\ninstall_sdk/ruby() {\n install_gem arvados sdk/ruby\n}\n\ninstall_sdk/ruby-google-api-client() {\n install_gem arvados-google-api-client sdk/ruby-google-api-client\n}\n\ninstall_contrib/R-sdk() {\n if [[ \"$NEED_SDK_R\" = true ]]; then\n env -C \"$WORKSPACE/contrib/R-sdk\" Rscript --vanilla install_deps.R\n fi\n}\n\ninstall_sdk/cli() {\n install_gem arvados-cli sdk/cli\n}\n\ninstall_services/login-sync() {\n install_gem arvados-login-sync services/login-sync\n}\n\ninstall_services/api() {\n stop_services\n check_arvados_config \"services/api\"\n cd \"$WORKSPACE/services/api\" \\\n && RAILS_ENV=test bundle_install_trylocal \\\n || return 1\n\n rm -f config/environments/test.rb\n cp config/environments/test.rb.example config/environments/test.rb\n\n # Clear out any lingering postgresql connections to the test\n # database, so that we can drop it. This assumes the current user\n # is a postgresql superuser.\n psql -c \"SELECT pg_terminate_backend (pg_stat_activity.pid::int) FROM pg_stat_activity WHERE pg_stat_activity.datname = '$PGDATABASE';\" 2>/dev/null\n\n mkdir -p \"$WORKSPACE/services/api/tmp/pids\"\n\n cert=\"$WORKSPACE/services/api/tmp/self-signed\"\n if [[ ! -e \"$cert.pem\" || \"$(date -r \"$cert.pem\" +%s)\" -lt 1512659226 ]]; then\n (\n dir=\"$WORKSPACE/services/api/tmp\"\n set -e\n openssl req -newkey rsa:2048 -nodes -subj '/C=US/ST=State/L=City/CN=localhost' -out \"$cert.csr\" -keyout \"$cert.key\" VERSION=n (revert and/or run a single migration; is up|down|redo)\"\n echo \"reset (...services used by integration tests)\"\n echo \"exit\"\n echo \"== Test targets:\"\n printf \"%s\\n\" \"${!testfuncargs[@]}\" | sort | column\n}\n\ndeclare -a failures\ndeclare -A skip\ndeclare -A only\ndeclare -A testargs\n\ndeclare -a pythonstuff\npythonstuff=(\n # The ordering of sdk/python, tools/crunchstat-summary, and\n # sdk/cwl here is significant. See\n # https://dev.arvados.org/issues/19744#note-26\n sdk/python\n tools/crunchstat-summary\n sdk/cwl\n services/dockercleaner\n services/fuse\n tools/cluster-activity\n)\n\ndeclare -a gostuff\nif [[ -n \"$WORKSPACE\" ]]; then\n readarray -d \"\" -t gostuff < <(\n git -C \"$WORKSPACE\" ls-files -z |\n grep -z '\\.go$' |\n xargs -0r dirname -z |\n sort -zu\n )\nfi\n\ndeclare -A testfuncargs=()\nfor testfuncname in $(declare -F | awk '\n($3 ~ /^test_/ && $3 !~ /_package_presence$/) {\n print substr($3, 6);\n}\n'); do\n testfuncargs[$testfuncname]=\"$testfuncname\"\ndone\nfor g in \"${gostuff[@]}\"; do\n testfuncargs[$g]=\"$g go\"\ndone\nfor p in \"${pythonstuff[@]}\"; do\n testfuncargs[$p]=\"$p pip\"\ndone\n\nwhile [[ -n \"$1\" ]]\ndo\n arg=\"$1\"; shift\n case \"$arg\" in\n --help)\n exec 1>&2\n echo \"$helpmessage\"\n if [[ ${#gostuff} -gt 0 ]]; then\n printf \"\\nAvailable targets:\\n\\n\"\n printf \"%s\\n\" \"${!testfuncargs[@]}\" | sort | column\n fi\n exit 1\n ;;\n --skip)\n skip[\"${1%:py3}\"]=1; shift\n ;;\n --only)\n only[\"${1%:py3}\"]=1; skip[\"${1%:py3}\"]=\"\"; shift\n ;;\n --short)\n short=1\n ;;\n --interactive)\n interactive=1\n ;;\n --skip-install)\n skip[install]=1\n ;;\n --only-install)\n only_install=\"$1\"; shift\n ;;\n --temp)\n temp=\"$1\"; shift\n temp_preserve=1\n ;;\n --leave-temp)\n temp_preserve=1\n ;;\n --repeat)\n repeat=$((${1}+0)); shift\n ;;\n --retry)\n retry=1\n ;;\n *_test=*)\n suite=\"${arg%%_test=*}\"\n args=\"${arg#*=}\"\n testargs[\"${suite%:py3}\"]=\"$args\"\n ;;\n ARVADOS_*=*)\n eval export $(echo $arg | cut -d= -f1)=\\\"$(echo $arg | cut -d= -f2-)\\\"\n ;;\n *)\n echo >&2 \"$0: Unrecognized option: '$arg'. Try: $0 --help\"\n exit 1\n ;;\n esac\ndone\n\n# R SDK installation is very slow (~360s in a clean environment) and only\n# required when testing it. Skip that step if it is not needed.\nNEED_SDK_R=true\n\nif [[ ${#only[@]} -ne 0 ]] &&\n [[ -z \"${only['contrib/R-sdk']}\" && -z \"${only['doc']}\" ]]; then\n NEED_SDK_R=false\nfi\n\nif [[ ${skip[\"contrib/R-sdk\"]} == 1 && ${skip[\"doc\"]} == 1 ]]; then\n NEED_SDK_R=false\nfi\n\nif [[ $NEED_SDK_R == false ]]; then\n echo \"R SDK not needed, it will not be installed.\"\nfi\n\ninitialize\nif [[ -z ${interactive} ]]; then\n install_all\n test_all\nelse\n skip=()\n only=()\n only_install=\"\"\n stop_services\n setnextcmd() {\n if [[ \"$TERM\" = dumb ]]; then\n # assume emacs, or something, is offering a history buffer\n # and pre-populating the command will only cause trouble\n nextcmd=\n elif [[ ! -e \"$GOPATH/bin/arvados-server\" ]]; then\n nextcmd=\"install deps\"\n else\n nextcmd=\"\"\n fi\n }\n echo\n help_interactive\n setnextcmd\n HISTFILE=\"$WORKSPACE/tmp/.history\"\n history -r\n ignore_sigint=1\n while read -p 'What next? ' -e -i \"$nextcmd\" nextcmd; do\n history -s \"$nextcmd\"\n history -w\n count=1\n if [[ \"${nextcmd}\" =~ ^[0-9] ]]; then\n read count nextcmd <<<\"${nextcmd}\"\n fi\n read verb target opts <<<\"${nextcmd}\"\n target=\"${target%/}\"\n target=\"${target/\\/:/:}\"\n # Remove old Python version suffix for backwards compatibility\n target=\"${target%:py3}\"\n case \"${verb}\" in\n \"exit\" | \"quit\")\n exit_cleanly\n ;;\n \"reset\")\n stop_services\n ;;\n \"migrate\")\n do_migrate ${target} ${opts}\n ;;\n \"bundle\")\n do_bundle ${target} ${opts}\n ;;\n \"test\" | \"install\")\n case \"$target\" in\n \"\")\n help_interactive\n ;;\n all | deps)\n ${verb}_${target}\n ;;\n *)\n testargs[\"$target\"]=\"${opts}\"\n while [ $count -gt 0 ]; do\n do_$verb ${testfuncargs[${target}]}\n let \"count=count-1\"\n done\n ;;\n esac\n ;;\n \"\" | \"help\" | *)\n help_interactive\n ;;\n esac\n if [[ ${#successes[@]} -gt 0 || ${#failures[@]} -gt 0 ]]; then\n report_outcomes\n successes=()\n failures=()\n fi\n cd \"$WORKSPACE\"\n setnextcmd\n done\n echo\nfi\nexit_cleanly\n" }, { "path": "build/version-at-commit.sh", "content": "#!/bin/bash\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\nset -e -o pipefail\ncommit=\"$1\"\ndevsuffix=\"~dev\"\n\n# automatically assign *development* version\n#\n# handles the following cases:\n#\n# * commit is on main or a development branch, the nearest tag is older\n# than commit where this branch joins main.\n# -> take greatest version tag in repo X.Y.Z and assign X.(Y+1).0\n#\n# * commit is on a release branch, the nearest tag is newer\n# than the commit where this branch joins main.\n# -> take nearest tag X.Y.Z and assign X.Y.(Z+1)\n\n# X.Y.Z releases where Z=0 are called major\n# releases and X.Y.Z releases where Z>1 are called point releases.\n#\n# The development process distinction is that X.Y.0 releases are\n# branched from main and then subsequent X.Y.Z releases cherry-pick\n# individual features from main onto the \"X.Y-staging\" branch.\n#\n# In semantic versioning terminology an \"X.Y.0\" release which only\n# increments Y is called a \"minor\" release but typically these\n# releases have significant changes that calling them \"minor\" in\n# communications with users feels misleading.\n#\n# Incrementing X is reserved for times when a release has significant\n# backwards-incompatible changes, which we don't do very often and try\n# to avoid.\n#\n# In order to assign a useful development version, we need to\n# determine if we're on the main branch (or a development branch off\n# main) or on a release branch. We do this by looking at the point\n# where the current commit history branched from main.\n#\n# If the tag for a new X+1 version appears on a release branch and not\n# directly in the history of main, the merge-base between main and the\n# release should be tagged as \"development-X.Y.Z\" so that\n# version-at-commit understands what version to assign to subsequent\n# commits on main. It is also helpful to assign development-X.Y.Z\n# tags to make git-describe provide better version strings.\n\n# 1. get the nearest tag with 'git describe'\n# 2. get the merge base between this commit and main\n# 3. if the tag is an ancestor of the merge base,\n# (tag is older than merge base) increment minor version\n# else, tag is newer than merge base, so increment point version\n\nnearest_tag=$(git describe --abbrev=0 \"$commit\")\n# We must use a remote branch here because Jenkins CI checkouts usually only\n# have the current work branch ref (and not even that if we're working by\n# commit hash). As of June 2025 everything uses origin, so,\nmerge_base=$(git merge-base origin/main \"$commit\")\n\nif git merge-base --is-ancestor \"$nearest_tag\" \"$merge_base\" ; then\n # the nearest tag appears before the merge base with main (the\n # branch point), so assume this is a tag for the previous major\n # release (or a tag with the \"development-\" prefix indicating the\n # point where a major release branched off). Subsequent\n # development versions are given the anticipated version for the\n # next major release.\n #\n # x.(y+1).0~devTIMESTAMP, where x.y.z is the newest version that does not contain $commit\n # grep reads the list of tags (-f) that contain $commit and filters them out (-v)\n # this prevents a newer tag from retroactively changing the versions of everything before it\n v=$(git tag |\n grep -vFf <(git tag --contains \"$merge_base\") |\n sed -e 's/^development-//' |\n sort --version-sort |\n awk '\nBEGIN { FS=\".\"; OFS=\".\"; }\nEND { print $1, $2+1, 0; }\n')\nelse\n # the nearest tag comes after the merge base with main (the branch\n # point). Assume this means this is a point release branch,\n # following a major release.\n #\n # x.y.(z+1)~devTIMESTAMP, where x.y.z is the latest released ancestor of $commit\n v=$(awk '\nBEGIN { FS=\".\"; OFS=\".\"; }\n{ print $1, $2, $3+1; exit; }\n' <\n\nALL=R/Arvados.R man\nSDK_VERSION!=awk '($$1 == \"Version:\"){v=$$2} END {print v}' DESCRIPTION\n\nall: $(ALL)\n\n.PHONY: api\napi: R/Arvados.R\nR/Arvados.R: arvados-v1-discovery.json generateApi.R\n\tRscript --vanilla generateApi.R\n\n# Used by arvados/doc/Rakefile.\n# Check whether we can load libraries necessary to build the package.\n.PHONY: can_run\ncan_run:\n\tRscript --vanilla -e \"library(jsonlite); library(roxygen2);\"\n\n.PHONY: clean\nclean:\n\trm -rf $(ALL) \"ArvadosR_$(SDK_VERSION).tar.gz\"\n\n.PHONY: install\ninstall:\n\tR CMD INSTALL .\n\nman: R/Arvados.R R/*.R\n\tRscript --vanilla -e \"library(roxygen2); roxygen2::roxygenize(clean=TRUE)\"\n\n.PHONY: package\npackage: \"ArvadosR_$(SDK_VERSION).tar.gz\"\n\"ArvadosR_$(SDK_VERSION).tar.gz\": $(ALL) [A-Z]* *.R tests/*.R tests/testthat/*.R tests/testthat/fakes/*.R\n\tR CMD build .\n\n.PHONY: test\ntest: $(ALL)\n\tRscript --vanilla run_test.R\n" }, { "path": "contrib/R-sdk/NAMESPACE", "content": "# Generated by roxygen2: do not edit by hand\n\nS3method(print,ArvadosFile)\nS3method(print,Collection)\nS3method(print,Subcollection)\nexport(Arvados)\nexport(ArvadosFile)\nexport(Collection)\nexport(Subcollection)\nexport(listAll)\n" }, { "path": "contrib/R-sdk/R/ArvadosFile.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\n#' R6 Class Representing a ArvadosFile\n#'\n#' @description\n#' ArvadosFile class represents a file inside Arvados collection.\n\n#' @export\nArvadosFile <- R6::R6Class(\n\n \"ArvadosFile\",\n\n public = list(\n\n #' @description\n #' Initialize new enviroment.\n #' @param name Name of the new enviroment.\n #' @return A new `ArvadosFile` object.\n #' @examples\n #' \\dontrun{\n #' myFile <- ArvadosFile$new(\"myFile\")\n #' }\n initialize = function(name)\n {\n if(name == \"\")\n stop(\"Invalid name.\")\n\n private$name <- name\n },\n\n #' @description\n #' Returns name of the file.\n #' @examples\n #' \\dontrun{\n #' arvadosFile$getName()\n #' }\n getName = function() private$name,\n\n #' @description\n #' Returns collections file content as character vector.\n #' @param fullPath Checking if TRUE.\n #' @examples\n #' \\dontrun{\n #' arvadosFile$getFileListing()\n #' }\n getFileListing = function(fullpath = TRUE)\n {\n self$getName()\n },\n\n #' @description\n #' Returns collections content size in bytes.\n #' @examples\n #' \\dontrun{\n #' arvadosFile$getSizeInBytes()\n #' }\n getSizeInBytes = function()\n {\n if(is.null(private$collection))\n return(0)\n\n REST <- private$collection$getRESTService()\n\n fileSize <- REST$getResourceSize(self$getRelativePath(),\n private$collection$uuid)\n fileSize\n },\n\n get = function(fileLikeObjectName)\n {\n return(NULL)\n },\n\n getFirst = function()\n {\n return(NULL)\n },\n\n #' @description\n #' Returns collection UUID.\n getCollection = function() private$collection,\n\n #' @description\n #' Sets new collection.\n setCollection = function(collection, setRecursively = TRUE)\n {\n private$collection <- collection\n },\n\n #' @description\n #' Returns file path relative to the root.\n getRelativePath = function()\n {\n relativePath <- c(private$name)\n parent <- private$parent\n\n while(!is.null(parent))\n {\n relativePath <- c(parent$getName(), relativePath)\n parent <- parent$getParent()\n }\n\n relativePath <- relativePath[relativePath != \"\"]\n paste0(relativePath, collapse = \"/\")\n },\n\n #' @description\n #' Returns project UUID.\n getParent = function() private$parent,\n\n #' @description\n #' Sets project collection.\n setParent = function(newParent) private$parent <- newParent,\n\n #' @description\n #' Read file content.\n #' @param contentType Type of content. Possible is \"text\", \"raw\".\n #' @param offset Describes the location of a piece of data compared to another location\n #' @param length Length of content\n #' @examples\n #' \\dontrun{\n #' collection <- Collection$new(arv, collectionUUID)\n #' arvadosFile <- collection$get(fileName)\n #' fileContent <- arvadosFile$read(\"text\")\n #' }\n read = function(contentType = \"raw\", offset = 0, length = 0)\n {\n if(is.null(private$collection))\n stop(\"ArvadosFile doesn't belong to any collection.\")\n\n if(offset < 0 || length < 0)\n stop(\"Offset and length must be positive values.\")\n\n REST <- private$collection$getRESTService()\n\n fileContent <- REST$read(self$getRelativePath(),\n private$collection$uuid,\n contentType, offset, length)\n fileContent\n },\n\n #' @description\n #' Get connection opened in \"read\" or \"write\" mode.\n #' @param rw Type of connection.\n #' @examples\n #' \\dontrun{\n #' collection <- Collection$new(arv, collectionUUID)\n #' arvadosFile <- collection$get(fileName)\n #' arvConnection <- arvadosFile$connection(\"w\")\n #' }\n connection = function(rw)\n {\n if (rw == \"r\" || rw == \"rb\")\n {\n REST <- private$collection$getRESTService()\n return(REST$getConnection(self$getRelativePath(),\n private$collection$uuid,\n rw))\n }\n else if (rw == \"w\")\n {\n private$buffer <- textConnection(NULL, \"w\")\n\n return(private$buffer)\n }\n },\n\n #' @description\n #' Write connections content to a file or override current content of the file.\n #' @examples\n #' \\dontrun{\n #' collection <- Collection$new(arv, collectionUUID)\n #' arvadosFile <- collection$get(fileName)\n #' myFile$write(\"This is new file content\")\n #' arvadosFile$flush()\n #' }\n flush = function()\n {\n v <- textConnectionValue(private$buffer)\n close(private$buffer)\n self$write(paste(v, collapse='\\n'))\n },\n\n #' @description\n #' Write to file or override current content of the file.\n #' @param content File to write.\n #' @param contentType Type of content. Possible is \"text\", \"raw\".\n #' @examples\n #' \\dontrun{\n #' collection <- Collection$new(arv, collectionUUID)\n #' arvadosFile <- collection$get(fileName)\n #' myFile$write(\"This is new file content\")\n #' }\n write = function(content, contentType = \"text/html\")\n {\n if(is.null(private$collection))\n stop(\"ArvadosFile doesn't belong to any collection.\")\n\n REST <- private$collection$getRESTService()\n\n writeResult <- REST$write(self$getRelativePath(),\n private$collection$uuid,\n content, contentType)\n writeResult\n },\n\n #' @description\n #' Moves file to a new location inside collection.\n #' @param destination Path to new folder.\n #' @examples\n #' \\dontrun{\n #' arvadosFile$move(newPath)\n #' }\n move = function(destination)\n {\n if(is.null(private$collection))\n stop(\"ArvadosFile doesn't belong to any collection.\")\n\n destination <- trimFromEnd(destination, \"/\")\n nameAndPath <- splitToPathAndName(destination)\n\n newParent <- private$collection$get(nameAndPath$path)\n\n if(is.null(newParent))\n stop(\"Unable to get destination subcollection.\")\n\n childWithSameName <- newParent$get(nameAndPath$name)\n\n if(!is.null(childWithSameName))\n stop(\"Destination already contains content with same name.\")\n\n REST <- private$collection$getRESTService()\n REST$move(self$getRelativePath(),\n paste0(newParent$getRelativePath(), \"/\", nameAndPath$name),\n private$collection$uuid)\n\n private$dettachFromCurrentParent()\n private$attachToNewParent(self, newParent)\n\n private$parent <- newParent\n private$name <- nameAndPath$name\n\n self\n },\n\n #' @description\n #' Copies file to a new location inside collection.\n #' @param destination Path to new folder.\n #' @examples\n #' \\dontrun{\n #' arvadosFile$copy(\"NewName.format\")\n #' }\n copy = function(destination)\n {\n if(is.null(private$collection))\n stop(\"ArvadosFile doesn't belong to any collection.\")\n\n destination <- trimFromEnd(destination, \"/\")\n nameAndPath <- splitToPathAndName(destination)\n\n newParent <- private$collection$get(nameAndPath$path)\n\n if(is.null(newParent))\n stop(\"Unable to get destination subcollection.\")\n\n childWithSameName <- newParent$get(nameAndPath$name)\n\n if(!is.null(childWithSameName))\n stop(\"Destination already contains content with same name.\")\n\n REST <- private$collection$getRESTService()\n REST$copy(self$getRelativePath(),\n paste0(newParent$getRelativePath(), \"/\", nameAndPath$name),\n private$collection$uuid)\n\n newFile <- self$duplicate(nameAndPath$name)\n newFile$setCollection(self$getCollection())\n private$attachToNewParent(newFile, newParent)\n newFile$setParent(newParent)\n\n newFile\n },\n\n #' @description\n #' Duplicate file and gives it a new name.\n #' @param newName New name for duplicated file.\n duplicate = function(newName = NULL)\n {\n name <- if(!is.null(newName)) newName else private$name\n newFile <- ArvadosFile$new(name)\n newFile\n }\n ),\n\n private = list(\n\n name = NULL,\n size = NULL,\n parent = NULL,\n collection = NULL,\n buffer = NULL,\n\n attachToNewParent = function(content, newParent)\n {\n # We temporary set parents collection to NULL. This will ensure that\n # add method doesn't post this file on REST.\n # We also need to set content's collection to NULL because\n # add method throws exception if we try to add content that already\n # belongs to a collection.\n\n parentsCollection <- newParent$getCollection()\n #parent$.__enclos_env__$private$children <- c(parent$.__enclos_env__$private$children, self)\n #private$parent <- parent\n content$setCollection(NULL, setRecursively = FALSE)\n newParent$setCollection(NULL, setRecursively = FALSE)\n newParent$add(content)\n content$setCollection(parentsCollection, setRecursively = FALSE)\n newParent$setCollection(parentsCollection, setRecursively = FALSE)\n },\n\n dettachFromCurrentParent = function()\n {\n # We temporary set parents collection to NULL. This will ensure that\n # remove method doesn't remove this file from REST.\n\n #private$parent$.__enclos_env__$private$removeChild(private$name)\n #private$parent <- NULL\n parent <- private$parent\n parentsCollection <- parent$getCollection()\n parent$setCollection(NULL, setRecursively = FALSE)\n parent$remove(private$name)\n parent$setCollection(parentsCollection, setRecursively = FALSE)\n }\n ),\n\n cloneable = FALSE\n)\n\n#' print.ArvadosFile\n#'\n#' Custom print function for ArvadosFile class\n#'\n#' @param x Instance of ArvadosFile class\n#' @param ... Optional arguments.\n#' @export\nprint.ArvadosFile = function(x, ...)\n{\n collection <- NULL\n relativePath <- x$getRelativePath()\n\n if(!is.null(x$getCollection()))\n {\n collection <- x$getCollection()$uuid\n relativePath <- paste0(\"/\", relativePath)\n }\n\n cat(paste0(\"Type: \", \"\\\"\", \"ArvadosFile\", \"\\\"\"), sep = \"\\n\")\n cat(paste0(\"Name: \", \"\\\"\", x$getName(), \"\\\"\"), sep = \"\\n\")\n cat(paste0(\"Relative path: \", \"\\\"\", relativePath, \"\\\"\"), sep = \"\\n\")\n cat(paste0(\"Collection: \", \"\\\"\", collection, \"\\\"\"), sep = \"\\n\")\n}\n" }, { "path": "contrib/R-sdk/R/ArvadosR.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\r\n#\r\n# SPDX-License-Identifier: Apache-2.0\r\n\r\n#' @title ArvadosR\r\n#'\r\n#' @description\r\n#'\r\n#' Arvados is an open source platform for managing, processing, and sharing genomic and other large scientific and biomedical data. With Arvados, bioinformaticians run and scale compute-intensive workflows, developers create biomedical applications, and IT administrators manage large compute and storage resources.\r\n#'\r\n#' @author \\itemize{\r\n#' \\item Lucas Di Pentima\r\n#' \\item Ward Vandewege\r\n#' \\item Fuad Muhic\r\n#' \\item Peter Amstutz\r\n#' \\item Aneta Stanczyk\r\n#' \\item Piotr Nowosielski\r\n#' \\item Brett Smith}\r\n#'\r\n#' @seealso \\itemize{\r\n#' \\item https://arvados.org\r\n#' \\item https://doc.arvados.org/sdk/R/index.html\r\n#' \\item https://github.com/arvados/arvados/tree/main/contrib/R-sdk}\r\n#'\r\n#' @name ArvadosR\r\nNULL\r\n" }, { "path": "contrib/R-sdk/R/Collection.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\n#' R6 Class Representing Arvados Collection\n#'\n#' @description\n#' Collection class provides interface for working with Arvados collections,\n#' for exaplme actions like creating, updating, moving or removing are possible.\n#'\n#' @seealso\n#' https://github.com/arvados/arvados/tree/main/contrib/R-sdk\n#'\n#' @export\n\nCollection <- R6::R6Class(\n\n \"Collection\",\n\n public = list(\n\n #' @field uuid Autentic for Collection UUID.\n uuid = NULL,\n\n #' @description\n #' Initialize new enviroment.\n #' @param api Arvados enviroment.\n #' @param uuid The UUID Autentic for Collection UUID.\n #' @return A new `Collection` object.\n #' @examples\n #' \\dontrun{\n #' collection <- Collection$new(arv, CollectionUUID)\n #' }\n initialize = function(api, uuid)\n {\n private$REST <- api$getRESTService()\n self$uuid <- uuid\n },\n\n #' @description\n #' Adds ArvadosFile or Subcollection specified by content to the collection. Used only with ArvadosFile or Subcollection.\n #' @param content Content to be added.\n #' @param relativePath Path to add content.\n add = function(content, relativePath = \"\")\n {\n if(is.null(private$tree))\n private$generateCollectionTreeStructure()\n\n if(relativePath == \"\" ||\n relativePath == \".\" ||\n relativePath == \"./\")\n {\n subcollection <- private$tree$getTree()\n }\n else\n {\n relativePath <- trimFromEnd(relativePath, \"/\")\n subcollection <- self$get(relativePath)\n }\n\n if(is.null(subcollection))\n stop(paste(\"Subcollection\", relativePath, \"doesn't exist.\"))\n\n if(\"ArvadosFile\" %in% class(content) ||\n \"Subcollection\" %in% class(content))\n {\n if(!is.null(content$getCollection()))\n stop(\"Content already belongs to a collection.\")\n\n if(content$getName() == \"\")\n stop(\"Content has invalid name.\")\n\n subcollection$add(content)\n content\n }\n else\n {\n stop(paste0(\"Expected AravodsFile or Subcollection object, got \",\n paste0(\"(\", paste0(class(content), collapse = \", \"), \")\"),\n \".\"))\n }\n },\n\n #' @description\n #' Read file content.\n #' @param file Name of the file.\n #' @param col Collection from which the file is read.\n #' @param sep Separator used in reading tsv, csv file format.\n #' @param istable Used in reading txt file to check if the file is table or not.\n #' @param fileclass Used in reading fasta file to set file class.\n #' @param Ncol Used in reading binary file to set numbers of columns in data.frame.\n #' @param Nrow Used in reading binary file to set numbers of rows in data.frame size.\n #' @examples\n #' \\dontrun{\n #' collection <- Collection$new(arv, collectionUUID)\n #' readFile <- collection$readArvFile(arvadosFile, istable = 'yes') # table\n #' readFile <- collection$readArvFile(arvadosFile, istable = 'no') # text\n #' readFile <- collection$readArvFile(arvadosFile) # xlsx, csv, tsv, rds, rdata\n #' readFile <- collection$readArvFile(arvadosFile, fileclass = 'fasta') # fasta\n #' readFile <- collection$readArvFile(arvadosFile, Ncol= 4, Nrow = 32) # binary, only numbers\n #' readFile <- collection$readArvFile(arvadosFile, Ncol = 5, Nrow = 150, istable = \"factor\") # binary with factor or text\n #' }\n readArvFile = function(file, con, sep = ',', istable = NULL, fileclass = \"SeqFastadna\", Ncol = NULL, Nrow = NULL, wantedFunction = NULL)\n {\n arvFile <- self$get(file)\n FileName <- arvFile$getName()\n FileName <- tolower(FileName)\n FileFormat <- gsub(\".*\\\\.\", \"\", FileName)\n\n # set enviroment\n ARVADOS_API_TOKEN <- Sys.getenv(\"ARVADOS_API_TOKEN\")\n ARVADOS_API_HOST <- Sys.getenv(\"ARVADOS_API_HOST\")\n my_collection <- self$uuid\n key <- gsub(\"/\", \"_\", ARVADOS_API_TOKEN)\n\n Sys.setenv(\n \"AWS_ACCESS_KEY_ID\" = key,\n \"AWS_SECRET_ACCESS_KEY\" = key,\n \"AWS_DEFAULT_REGION\" = \"collections\",\n \"AWS_S3_ENDPOINT\" = gsub(\"api[.]\", \"\", ARVADOS_API_HOST))\n\n if (FileFormat == \"txt\") {\n if (is.null(istable)){\n stop(paste('You need to paste whether it is a text or table file'))\n } else if (istable == 'no') {\n fileContent <- arvFile$read(\"text\") # used to read\n fileContent <- gsub(\"[\\r\\n]\", \" \", fileContent)\n } else if (istable == 'yes') {\n arvConnection <- arvFile$connection(\"r\") # used to make possible use different function later\n fileContent <- read.table(arvConnection)\n }\n }\n else if (FileFormat == \"xlsx\") {\n fileContent <- aws.s3::s3read_using(FUN = openxlsx::read.xlsx, object = file, bucket = my_collection)\n }\n else if (FileFormat == \"csv\" || FileFormat == \"tsv\") {\n arvConnection <- arvFile$connection(\"r\")\n if (FileFormat == \"tsv\"){\n mytable <- read.table(arvConnection, sep = '\\t')\n } else if (FileFormat == \"csv\" & sep == '\\t') {\n mytable <- read.table(arvConnection, sep = '\\t')\n } else if (FileFormat == \"csv\") {\n mytable <- read.table(arvConnection, sep = ',')\n } else {\n stop(paste('File format not supported, use arvadosFile$connection() and customise it'))\n }\n }\n else if (FileFormat == \"fasta\") {\n fastafile <- aws.s3::s3read_using(FUN = seqinr::read.fasta, as.string = TRUE, object = file, bucket = my_collection)\n }\n else if (FileFormat == \"dat\" || FileFormat == \"bin\") {\n fileContent <- gzcon(arvFile$connection(\"rb\"))\n\n # function to precess data to binary format\n read_bin.file <- function(fileContent) {\n # read binfile\n column.names <- readBin(fileContent, character(), n = Ncol)\n bindata <- readBin(fileContent, numeric(), Nrow*Ncol+Ncol)\n # check\n res <- which(bindata < 0.0000001)\n if (is.list(res)) {\n bindata <- bindata[-res]\n } else {\n bindata <- bindata\n }\n # make a dataframe\n data <- data.frame(matrix(data = NA, nrow = Nrow, ncol = Ncol))\n for (i in 1:Ncol) {\n data[,i] <- bindata[(1+Nrow*(i-1)):(Nrow*i)]\n }\n colnames(data) = column.names\n\n len <- which(is.na(data[,Ncol])) # error if sth went wrong\n if (length(len) == 0) {\n data\n } else {\n stop(paste(\"there is a factor or text in the table, customize the function by typing more arguments\"))\n }\n }\n if (is.null(Nrow) | is.null(Ncol)){\n stop(paste('You need to specify numbers of columns and rows'))\n }\n if (is.null(istable)) {\n fileContent <- read_bin.file(fileContent) # call a function\n } else if (istable == \"factor\") { # if there is a table with col name\n fileContent <- read_bin.file(fileContent)\n }\n }\n else if (FileFormat == \"rds\" || FileFormat == \"rdata\") {\n arvConnection <- arvFile$connection(\"rb\")\n mytable <- readRDS(gzcon(arvConnection))\n }\n else {\n stop(parse(('File format not supported, use arvadosFile$connection() and customise it')))\n }\n },\n\n #' @description\n #' Write file content\n #' @param name Name of the file.\n #' @param file File to be saved.\n #' @param istable Used in writing txt file to check if the file is table or not.\n #' @examples\n #' \\dontrun{\n #' collection <- Collection$new(arv, collectionUUID)\n #' writeFile <- collection$writeFile(name = \"myoutput.csv\", file = file, fileFormat = \"csv\", istable = NULL, collectionUUID = collectionUUID) # csv\n #' writeFile <- collection$writeFile(name = \"myoutput.tsv\", file = file, fileFormat = \"tsv\", istable = NULL, collectionUUID = collectionUUID) # tsv\n #' writeFile <- collection$writeFile(name = \"myoutput.fasta\", file = file, fileFormat = \"fasta\", istable = NULL, collectionUUID = collectionUUID) # fasta\n #' writeFile <- collection$writeFile(name = \"myoutputtable.txt\", file = file, fileFormat = \"txt\", istable = \"yes\", collectionUUID = collectionUUID) # txt table\n #' writeFile <- collection$writeFile(name = \"myoutputtext.txt\", file = file, fileFormat = \"txt\", istable = \"no\", collectionUUID = collectionUUID) # txt text\n #' writeFile <- collection$writeFile(name = \"myoutputbinary.dat\", file = file, fileFormat = \"dat\", collectionUUID = collectionUUID) # binary\n #' writeFile <- collection$writeFile(name = \"myoutputxlsx.xlsx\", file = file, fileFormat = \"xlsx\", collectionUUID = collectionUUID) # xlsx\n #' }\n writeFile = function(name, file, collectionUUID, fileFormat, istable = NULL, seqName = NULL)\n {\n # set enviroment\n ARVADOS_API_TOKEN <- Sys.getenv(\"ARVADOS_API_TOKEN\")\n ARVADOS_API_HOST <- Sys.getenv(\"ARVADOS_API_HOST\")\n my_collection <- self$uuid\n key <- gsub(\"/\", \"_\", ARVADOS_API_TOKEN)\n\n Sys.setenv(\n \"AWS_ACCESS_KEY_ID\" = key,\n \"AWS_SECRET_ACCESS_KEY\" = key,\n \"AWS_DEFAULT_REGION\" = \"collections\",\n \"AWS_S3_ENDPOINT\" = gsub(\"api[.]\", \"\", ARVADOS_API_HOST))\n\n # save file\n if (fileFormat == \"txt\") {\n if (istable == \"yes\") {\n aws.s3::s3write_using(file, FUN = write.table, object = name, bucket = collectionUUID)\n } else if (istable == \"no\") {\n aws.s3::s3write_using(file, FUN = writeChar, object = name, bucket = collectionUUID)\n } else {\n stop(paste(\"Specify parametr istable\"))\n }\n } else if (fileFormat == \"csv\") {\n aws.s3::s3write_using(file, FUN = write.csv, object = name, bucket = collectionUUID)\n } else if (fileFormat == \"tsv\") {\n aws.s3::s3write_using(file, FUN = write.table, row.names = FALSE, sep = \"\\t\", object = name, bucket = collectionUUID)\n } else if (fileFormat == \"fasta\") {\n aws.s3::s3write_using(file, FUN = seqinr::write.fasta, name = seqName, object = name, bucket = collectionUUID)\n } else if (fileFormat == \"xlsx\") {\n aws.s3::s3write_using(file, FUN = openxlsx::write.xlsx, object = name, bucket = collectionUUID)\n } else if (fileFormat == \"dat\" || fileFormat == \"bin\") {\n aws.s3::s3write_using(file, FUN = writeBin, object = name, bucket = collectionUUID)\n } else {\n stop(parse(('File format not supported, use arvadosFile$connection() and customise it')))\n }\n },\n\n #' @description\n #' Creates one or more ArvadosFiles and adds them to the collection at specified path.\n #' @param files Content to be created.\n #' @examples\n #' \\dontrun{\n #' collection <- arv$collections_create(name = collectionTitle, description = collectionDescription, owner_uuid = collectionOwner, properties = list(\"ROX37196928443768648\" = \"ROX37742976443830153\"))\n #' }\n create = function(files)\n {\n if(is.null(private$tree))\n private$generateCollectionTreeStructure()\n\n if(is.character(files))\n {\n sapply(files, function(file)\n {\n childWithSameName <- self$get(file)\n if(!is.null(childWithSameName))\n stop(\"Destination already contains file with same name.\")\n\n newTreeBranch <- private$tree$createBranch(file)\n private$tree$addBranch(private$tree$getTree(), newTreeBranch)\n\n private$REST$create(file, self$uuid)\n newTreeBranch$setCollection(self)\n newTreeBranch\n })\n }\n else\n {\n stop(paste0(\"Expected character vector, got \",\n paste0(\"(\", paste0(class(files), collapse = \", \"), \")\"),\n \".\"))\n }\n },\n\n #' @description\n #' Remove one or more files from the collection.\n #' @param paths Content to be removed.\n #' @examples\n #' \\dontrun{\n #' collection$remove(fileName.format)\n #' }\n remove = function(paths)\n {\n if(is.null(private$tree))\n private$generateCollectionTreeStructure()\n\n if(is.character(paths))\n {\n sapply(paths, function(filePath)\n {\n filePath <- trimFromEnd(filePath, \"/\")\n file <- self$get(filePath)\n\n if(is.null(file))\n stop(paste(\"File\", filePath, \"doesn't exist.\"))\n\n parent <- file$getParent()\n\n if(is.null(parent))\n stop(\"You can't delete root folder.\")\n\n parent$remove(file$getName())\n })\n\n \"Content removed\"\n }\n else\n {\n stop(paste0(\"Expected character vector, got \",\n paste0(\"(\", paste0(class(paths), collapse = \", \"), \")\"),\n \".\"))\n }\n },\n\n #' @description\n #' Moves ArvadosFile or Subcollection to another location in the collection.\n #' @param content Content to be moved.\n #' @param destination Path to move content.\n #' @examples\n #' \\dontrun{\n #' collection$move(\"fileName.format\", path)\n #' }\n move = function(content, destination)\n {\n if(is.null(private$tree))\n private$generateCollectionTreeStructure()\n\n content <- trimFromEnd(content, \"/\")\n\n elementToMove <- self$get(content)\n\n if(is.null(elementToMove))\n stop(\"Content you want to move doesn't exist in the collection.\")\n\n elementToMove$move(destination)\n },\n\n #' @description\n #' Copies ArvadosFile or Subcollection to another location in the collection.\n #' @param content Content to be moved.\n #' @param destination Path to move content.\n #' @examples\n #' \\dontrun{\n #' copied <- collection$copy(\"oldName.format\", \"newName.format\")\n #' }\n copy = function(content, destination)\n {\n if(is.null(private$tree))\n private$generateCollectionTreeStructure()\n\n content <- trimFromEnd(content, \"/\")\n\n elementToCopy <- self$get(content)\n\n if(is.null(elementToCopy))\n stop(\"Content you want to copy doesn't exist in the collection.\")\n\n elementToCopy$copy(destination)\n },\n\n #' @description\n #' Refreshes the environment.\n #' @examples\n #' \\dontrun{\n #' collection$refresh()\n #' }\n refresh = function()\n {\n if(!is.null(private$tree))\n {\n private$tree$getTree()$setCollection(NULL, setRecursively = TRUE)\n private$tree <- NULL\n }\n },\n\n #' @description\n #' Returns collections file content as character vector.\n #' @examples\n #' \\dontrun{\n #' list <- collection$getFileListing()\n #' }\n getFileListing = function()\n {\n if(is.null(private$tree))\n private$generateCollectionTreeStructure()\n\n content <- private$REST$getCollectionContent(self$uuid)\n content[order(tolower(content))]\n },\n\n #' @description\n #' If relativePath is valid, returns ArvadosFile or Subcollection specified by relativePath, else returns NULL.\n #' @param relativePath Path from content is taken.\n #' @examples\n #' \\dontrun{\n #' arvadosFile <- collection$get(fileName)\n #' }\n get = function(relativePath)\n {\n if(is.null(private$tree))\n private$generateCollectionTreeStructure()\n\n private$tree$getElement(relativePath)\n },\n\n getRESTService = function() private$REST,\n setRESTService = function(newRESTService) private$REST <- newRESTService\n ),\n private = list(\n\n REST = NULL,\n #' @tree beautiful tree of sth\n tree = NULL,\n fileContent = NULL,\n\n generateCollectionTreeStructure = function(relativePath = NULL)\n {\n if(is.null(self$uuid))\n stop(\"Collection uuid is not defined.\")\n\n if(is.null(private$REST))\n stop(\"REST service is not defined.\")\n\n private$fileContent <- private$REST$getCollectionContent(self$uuid, relativePath)\n private$tree <- CollectionTree$new(private$fileContent, self)\n }\n ),\n\n cloneable = FALSE\n)\n\n#' print.Collection\n#'\n#' Custom print function for Collection class\n#'\n#' @param x Instance of Collection class\n#' @param ... Optional arguments.\n#' @export\nprint.Collection = function(x, ...)\n{\n cat(paste0(\"Type: \", \"\\\"\", \"Arvados Collection\", \"\\\"\"), sep = \"\\n\")\n cat(paste0(\"uuid: \", \"\\\"\", x$uuid, \"\\\"\"), sep = \"\\n\")\n}\n" }, { "path": "contrib/R-sdk/R/CollectionTree.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\nCollectionTree <- R6::R6Class(\n \"CollectionTree\",\n public = list(\n\n pathsList = NULL,\n\n initialize = function(fileContent, collection)\n {\n self$pathsList <- fileContent\n treeBranches <- sapply(fileContent, function(filePath) self$createBranch(filePath))\n root <- Subcollection$new(\"\")\n sapply(treeBranches, function(branch) self$addBranch(root, branch))\n root$setCollection(collection)\n private$tree <- root\n },\n\n createBranch = function(filePath)\n {\n splitPath <- unlist(strsplit(filePath, \"/\", fixed = TRUE))\n branch <- NULL\n lastElementIndex <- length(splitPath)\n\n for(elementIndex in lastElementIndex:1)\n {\n if(elementIndex == lastElementIndex)\n {\n branch <- ArvadosFile$new(splitPath[[elementIndex]])\n }\n else\n {\n newFolder <- Subcollection$new(splitPath[[elementIndex]])\n newFolder$add(branch)\n branch <- newFolder\n }\n }\n\n branch\n },\n\n addBranch = function(container, node)\n {\n child <- container$get(node$getName())\n\n if(is.null(child))\n {\n # Make sure we are don't make any REST call while adding child\n collection <- container$getCollection()\n container$setCollection(NULL, setRecursively = FALSE)\n container$add(node)\n container$setCollection(collection, setRecursively = FALSE)\n }\n else\n {\n # Note: REST always returns folder name alone before other folder\n # content, so in first iteration we don't know if it's a file\n # or folder since its just a name, so we assume it's a file.\n # If we encounter that same name again we know\n # it's a folder so we need to replace ArvadosFile with Subcollection.\n if(\"ArvadosFile\" %in% class(child))\n child = private$replaceFileWithSubcollection(child)\n\n self$addBranch(child, node$getFirst())\n }\n },\n\n getElement = function(relativePath)\n {\n relativePath <- trimFromStart(relativePath, \"./\")\n relativePath <- trimFromEnd(relativePath, \"/\")\n\n if(endsWith(relativePath, \"/\"))\n relativePath <- substr(relativePath, 0, nchar(relativePath) - 1)\n\n splitPath <- unlist(strsplit(relativePath, \"/\", fixed = TRUE))\n returnElement <- private$tree\n\n for(pathFragment in splitPath)\n {\n returnElement <- returnElement$get(pathFragment)\n\n if(is.null(returnElement))\n return(NULL)\n }\n\n returnElement\n },\n\n getTree = function() private$tree\n ),\n\n private = list(\n\n tree = NULL,\n\n replaceFileWithSubcollection = function(arvadosFile)\n {\n subcollection <- Subcollection$new(arvadosFile$getName())\n fileParent <- arvadosFile$getParent()\n fileParent$remove(arvadosFile$getName())\n fileParent$add(subcollection)\n\n arvadosFile$setParent(NULL)\n\n subcollection\n }\n )\n)\n" }, { "path": "contrib/R-sdk/R/HttpParser.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\nHttpParser <- R6::R6Class(\n\n \"HttrParser\",\n\n public = list(\n\n validContentTypes = NULL,\n\n initialize = function()\n {\n self$validContentTypes <- c(\"text\", \"raw\")\n },\n\n parseJSONResponse = function(serverResponse)\n {\n parsed_response <- httr::content(serverResponse,\n as = \"parsed\",\n type = \"application/json\")\n },\n\n parseResponse = function(serverResponse, outputType)\n {\n parsed_response <- httr::content(serverResponse, as = outputType)\n },\n\n getFileNamesFromResponse = function(response, uri)\n {\n text <- rawToChar(response$content)\n doc <- XML::xmlParse(text, asText=TRUE)\n base <- paste(\"/\", strsplit(uri, \"/\")[[1]][4], \"/\", sep=\"\")\n result <- unlist(\n XML::xpathApply(doc, \"//D:response/D:href\", function(node) {\n sub(base, \"\", URLdecode(XML::xmlValue(node)), fixed=TRUE)\n })\n )\n result[result != \"\"]\n },\n\n getFileSizesFromResponse = function(response, uri)\n {\n text <- rawToChar(response$content)\n doc <- XML::xmlParse(text, asText=TRUE)\n\n base <- paste(paste(\"/\", strsplit(uri, \"/\")[[1]][-1:-3], sep=\"\", collapse=\"\"), \"/\", sep=\"\")\n result <- XML::xpathApply(doc, \"//D:response/D:propstat/D:prop/D:getcontentlength\", function(node) {\n XML::xmlValue(node)\n })\n\n unlist(result)\n }\n )\n)\n" }, { "path": "contrib/R-sdk/R/HttpRequest.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\nHttpRequest <- R6::R6Class(\n\n \"HttrRequest\",\n\n public = list(\n\n validContentTypes = NULL,\n validVerbs = NULL,\n\n initialize = function()\n {\n self$validContentTypes <- c(\"text\", \"raw\")\n self$validVerbs <- c(\"GET\", \"POST\", \"PUT\", \"DELETE\", \"PROPFIND\", \"MOVE\", \"COPY\")\n },\n\n exec = function(verb, url, headers = NULL, body = NULL, queryParams = NULL,\n retryTimes = 0)\n {\n if(!(verb %in% self$validVerbs))\n stop(\"Http verb is not valid.\")\n\n urlQuery <- self$createQuery(queryParams)\n url <- paste0(url, urlQuery)\n\n config <- httr::add_headers(unlist(headers))\n if(toString(Sys.getenv(\"ARVADOS_API_HOST_INSECURE\") == \"TRUE\"))\n config$options = list(ssl_verifypeer = 0L)\n\n response <- httr::RETRY(verb, url = url, body = body,\n config = config, times = retryTimes + 1)\n },\n\n createQuery = function(queryParams)\n {\n queryParams <- Filter(Negate(is.null), queryParams)\n\n query <- sapply(queryParams, function(param)\n {\n if(is.list(param) || length(param) > 1)\n param <- RListToPythonList(param, \",\")\n\n URLencode(as.character(param), reserved = T, repeated = T)\n\n }, USE.NAMES = TRUE)\n\n if(length(query) > 0)\n {\n query <- paste0(names(query), \"=\", query, collapse = \"&\")\n\n return(paste0(\"?\", query))\n }\n\n return(\"\")\n },\n\n getConnection = function(url, headers, openMode)\n {\n h <- curl::new_handle()\n curl::handle_setheaders(h, .list = headers)\n\n if(toString(Sys.getenv(\"ARVADOS_API_HOST_INSECURE\") == \"TRUE\"))\n curl::handle_setopt(h, ssl_verifypeer = 0L)\n\n conn <- curl::curl(url = url, open = openMode, handle = h)\n }\n ),\n\n cloneable = FALSE\n)\n" }, { "path": "contrib/R-sdk/R/RESTService.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\nRESTService <- R6::R6Class(\n\n \"RESTService\",\n\n public = list(\n\n token = NULL,\n http = NULL,\n httpParser = NULL,\n numRetries = NULL,\n\n initialize = function(token, rawHost,\n http, httpParser,\n numRetries = 0,\n webDavHostName = NULL)\n {\n self$token <- token\n self$http <- http\n self$httpParser <- httpParser\n self$numRetries <- numRetries\n\n private$rawHostName <- rawHost\n private$webDavHostName <- webDavHostName\n },\n\n setNumConnRetries = function(newNumOfRetries)\n {\n self$numRetries <- newNumOfRetries\n },\n\n getWebDavHostName = function()\n {\n if(is.null(private$webDavHostName))\n {\n publicConfigURL <- paste0(\"https://\", private$rawHostName,\n \"/arvados/v1/config\")\n\n serverResponse <- self$http$exec(\"GET\", publicConfigURL, retryTimes = self$numRetries)\n\n configDocument <- self$httpParser$parseJSONResponse(serverResponse)\n private$webDavHostName <- configDocument$Services$WebDAVDownload$ExternalURL\n\n if(is.null(private$webDavHostName))\n stop(\"Unable to find WebDAV server.\")\n }\n\n private$webDavHostName\n },\n\n create = function(files, uuid)\n {\n sapply(files, function(filePath)\n {\n private$createNewFile(filePath, uuid, \"text/html\")\n })\n },\n\n delete = function(relativePath, uuid)\n {\n fileURL <- paste0(self$getWebDavHostName(), \"c=\",\n uuid, \"/\", relativePath);\n headers <- list(Authorization = paste(\"Bearer\", self$token))\n\n serverResponse <- self$http$exec(\"DELETE\", fileURL, headers,\n retryTimes = self$numRetries)\n\n if(serverResponse$status_code < 200 || serverResponse$status_code >= 300)\n stop(paste(\"Server code:\", serverResponse$status_code))\n\n serverResponse\n },\n\n move = function(from, to, uuid)\n {\n collectionURL <- paste0(self$getWebDavHostName(), \"c=\", uuid, \"/\")\n fromURL <- paste0(collectionURL, from)\n toURL <- paste0(collectionURL, trimFromStart(to, \"/\"))\n\n headers <- list(\"Authorization\" = paste(\"Bearer\", self$token),\n \"Destination\" = toURL)\n\n serverResponse <- self$http$exec(\"MOVE\", fromURL, headers,\n retryTimes = self$numRetries)\n\n if(serverResponse$status_code < 200 || serverResponse$status_code >= 300)\n stop(paste(\"Server code:\", serverResponse$status_code))\n\n serverResponse\n },\n\n copy = function(from, to, uuid)\n {\n collectionURL <- paste0(self$getWebDavHostName(), \"c=\", uuid, \"/\")\n fromURL <- paste0(collectionURL, from)\n toURL <- paste0(collectionURL, trimFromStart(to, \"/\"))\n\n headers <- list(\"Authorization\" = paste(\"Bearer\", self$token),\n \"Destination\" = toURL)\n\n serverResponse <- self$http$exec(\"COPY\", fromURL, headers,\n retryTimes = self$numRetries)\n\n if(serverResponse$status_code < 200 || serverResponse$status_code >= 300)\n stop(paste(\"Server code:\", serverResponse$status_code))\n\n serverResponse\n },\n\n getCollectionContent = function(uuid, relativePath = NULL)\n\n {\n collectionURL <- URLencode(paste0(self$getWebDavHostName(),\n \"c=\", uuid, \"/\", relativePath))\n\n headers <- list(\"Authorization\" = paste(\"Bearer\", self$token))\n\n response <- self$http$exec(\"PROPFIND\", collectionURL, headers,\n retryTimes = self$numRetries)\n\n if(all(response == \"\"))\n stop(\"Response is empty, request may be misconfigured\")\n\n if(response$status_code < 200 || response$status_code >= 300)\n stop(paste(\"Server code:\", response$status_code))\n\n self$httpParser$getFileNamesFromResponse(response, collectionURL)\n },\n\n getResourceSize = function(relativePath, uuid)\n {\n collectionURL <- URLencode(paste0(self$getWebDavHostName(),\n \"c=\", uuid))\n\n subcollectionURL <- paste0(collectionURL, \"/\", relativePath);\n\n headers <- list(\"Authorization\" = paste(\"Bearer\", self$token))\n\n response <- self$http$exec(\"PROPFIND\", subcollectionURL, headers,\n retryTimes = self$numRetries)\n\n if(all(response == \"\"))\n stop(\"Response is empty, request may be misconfigured\")\n\n if(response$status_code < 200 || response$status_code >= 300)\n stop(paste(\"Server code:\", response$status_code))\n\n sizes <- self$httpParser$getFileSizesFromResponse(response,\n collectionURL)\n as.numeric(sizes)\n },\n\n read = function(relativePath, uuid, contentType = \"raw\", offset = 0, length = 0)\n {\n fileURL <- paste0(self$getWebDavHostName(),\n \"c=\", uuid, \"/\", relativePath);\n\n range <- paste0(\"bytes=\", offset, \"-\")\n\n if(length > 0)\n range = paste0(range, offset + length - 1)\n\n if(offset == 0 && length == 0)\n {\n headers <- list(Authorization = paste(\"Bearer\", self$token))\n }\n else\n {\n headers <- list(Authorization = paste(\"Bearer\", self$token),\n Range = range)\n }\n\n if(!(contentType %in% self$httpParser$validContentTypes))\n stop(\"Invalid contentType. Please use text or raw.\")\n\n serverResponse <- self$http$exec(\"GET\", fileURL, headers,\n retryTimes = self$numRetries)\n\n if(serverResponse$status_code < 200 || serverResponse$status_code >= 300)\n stop(paste(\"Server code:\", serverResponse$status_code))\n\n self$httpParser$parseResponse(serverResponse, contentType)\n },\n\n write = function(relativePath, uuid, content, contentType)\n {\n fileURL <- paste0(self$getWebDavHostName(),\n \"c=\", uuid, \"/\", relativePath);\n headers <- list(Authorization = paste(\"Bearer\", self$token),\n \"Content-Type\" = contentType)\n body <- content\n\n serverResponse <- self$http$exec(\"PUT\", fileURL, headers, body,\n retryTimes = self$numRetries)\n\n if(serverResponse$status_code < 200 || serverResponse$status_code >= 300)\n stop(paste(\"Server code:\", serverResponse$status_code))\n\n self$httpParser$parseResponse(serverResponse, \"text\")\n },\n\n getConnection = function(relativePath, uuid, openMode)\n {\n fileURL <- paste0(self$getWebDavHostName(),\n \"c=\", uuid, \"/\", relativePath);\n headers <- list(Authorization = paste(\"Bearer\", self$token))\n\n conn <- self$http$getConnection(fileURL, headers, openMode)\n }\n ),\n\n private = list(\n\n webDavHostName = NULL,\n rawHostName = NULL,\n\n createNewFile = function(relativePath, uuid, contentType)\n {\n fileURL <- paste0(self$getWebDavHostName(), \"c=\",\n uuid, \"/\", relativePath)\n headers <- list(Authorization = paste(\"Bearer\", self$token),\n \"Content-Type\" = contentType)\n body <- NULL\n\n serverResponse <- self$http$exec(\"PUT\", fileURL, headers, body,\n retryTimes = self$numRetries)\n\n if (serverResponse$status_code < 200){ # to wyrzuca błędy\n stop(paste(\"Server code:\", serverResponse$status_code))}\n else if (serverResponse$status_code >= 300 & serverResponse$status_code < 422) {\n stop(paste(\"Server code:\", serverResponse$status_code))}\n else if (serverResponse$status_code == 422 ) {\n stop(paste(\"Project of that name already exists. If you want to change it use project_update() instead\"))}\n\n paste(\"File created:\", relativePath)\n }\n ),\n\n cloneable = FALSE\n)\n" }, { "path": "contrib/R-sdk/R/Subcollection.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\n#' R6 Class Representing a Subcollection\n#'\n#' @description\n#' Subcollection class represents a folder inside Arvados collection.\n#' It is essentially a composite of arvadosFiles and other subcollections.\n\n#' @export\nSubcollection <- R6::R6Class(\n\n \"Subcollection\",\n\n public = list(\n\n #' @description\n #' Initialize new enviroment.\n #' @param name Name of the new enviroment.\n #' @return A new `Subcollection` object.\n initialize = function(name)\n {\n private$name <- name\n },\n\n #' @description\n #' Returns name of the file.\n getName = function() private$name,\n\n #' @description\n #' Returns Subcollection's path relative to the root.\n getRelativePath = function()\n {\n relativePath <- c(private$name)\n parent <- private$parent\n\n while(!is.null(parent))\n {\n relativePath <- c(parent$getName(), relativePath)\n parent <- parent$getParent()\n }\n\n relativePath <- relativePath[relativePath != \"\"]\n paste0(relativePath, collapse = \"/\")\n },\n\n #' @description\n #' Adds ArvadosFile or Subcollection specified by content to the Subcollection.\n #' @param content Content to be added.\n add = function(content)\n {\n if(\"ArvadosFile\" %in% class(content) ||\n \"Subcollection\" %in% class(content))\n {\n if(!is.null(content$getCollection()))\n stop(\"Content already belongs to a collection.\")\n\n if(content$getName() == \"\")\n stop(\"Content has invalid name.\")\n\n childWithSameName <- self$get(content$getName())\n\n if(!is.null(childWithSameName))\n stop(paste(\"Subcollection already contains ArvadosFile\",\n \"or Subcollection with same name.\"))\n\n if(!is.null(private$collection))\n {\n if(self$getRelativePath() != \"\")\n contentPath <- paste0(self$getRelativePath(),\n \"/\", content$getFileListing())\n else\n contentPath <- content$getFileListing()\n\n REST <- private$collection$getRESTService()\n REST$create(contentPath, private$collection$uuid)\n content$setCollection(private$collection)\n }\n\n private$children <- c(private$children, content)\n content$setParent(self)\n\n \"Content added successfully.\"\n }\n else\n {\n stop(paste0(\"Expected AravodsFile or Subcollection object, got \",\n paste0(\"(\", paste0(class(content), collapse = \", \"), \")\"),\n \".\"))\n }\n },\n\n #' @description\n #' Removes ArvadosFile or Subcollection specified by name from the Subcollection.\n #' @param name Name of the file to be removed.\n remove = function(name)\n {\n if(is.character(name))\n {\n child <- self$get(name)\n\n if(is.null(child))\n stop(paste(\"Subcollection doesn't contains ArvadosFile\",\n \"or Subcollection with specified name.\"))\n\n if(!is.null(private$collection))\n {\n REST <- private$collection$getRESTService()\n REST$delete(child$getRelativePath(), private$collection$uuid)\n\n child$setCollection(NULL)\n }\n\n private$removeChild(name)\n child$setParent(NULL)\n\n \"Content removed\"\n }\n else\n {\n stop(paste0(\"Expected character, got \",\n paste0(\"(\", paste0(class(name), collapse = \", \"), \")\"),\n \".\"))\n }\n },\n\n #' @description\n #' Returns Subcollections file content as character vector.\n #' @param fullPath Checking if the path to file exists.\n getFileListing = function(fullPath = TRUE)\n {\n content <- private$getContentAsCharVector(fullPath)\n content[order(tolower(content))]\n },\n\n #' @description\n #' Returns subcollections content size in bytes.\n getSizeInBytes = function()\n {\n if(is.null(private$collection))\n return(0)\n\n REST <- private$collection$getRESTService()\n\n fileSizes <- REST$getResourceSize(paste0(self$getRelativePath(), \"/\"),\n private$collection$uuid)\n return(sum(fileSizes))\n },\n\n #' @description\n #' Moves Subcollection to a new location inside collection.\n #' @param destination Path to move the file.\n move = function(destination)\n {\n if(is.null(private$collection))\n stop(\"Subcollection doesn't belong to any collection.\")\n\n destination <- trimFromEnd(destination, \"/\")\n nameAndPath <- splitToPathAndName(destination)\n\n newParent <- private$collection$get(nameAndPath$path)\n\n if(is.null(newParent))\n stop(\"Unable to get destination subcollection.\")\n\n childWithSameName <- newParent$get(nameAndPath$name)\n\n if(!is.null(childWithSameName))\n stop(\"Destination already contains content with same name.\")\n\n REST <- private$collection$getRESTService()\n REST$move(self$getRelativePath(),\n paste0(newParent$getRelativePath(), \"/\", nameAndPath$name),\n private$collection$uuid)\n\n private$dettachFromCurrentParent()\n private$attachToNewParent(self, newParent)\n\n private$parent <- newParent\n private$name <- nameAndPath$name\n\n self\n },\n\n #' @description\n #' Copies Subcollection to a new location inside collection.\n #' @param destination Path to copy the file.\n copy = function(destination)\n {\n if(is.null(private$collection))\n stop(\"Subcollection doesn't belong to any collection.\")\n\n destination <- trimFromEnd(destination, \"/\")\n nameAndPath <- splitToPathAndName(destination)\n\n newParent <- private$collection$get(nameAndPath$path)\n\n if(is.null(newParent) || !(\"Subcollection\" %in% class(newParent)))\n stop(\"Unable to get destination subcollection.\")\n\n childWithSameName <- newParent$get(nameAndPath$name)\n\n if(!is.null(childWithSameName))\n stop(\"Destination already contains content with same name.\")\n\n REST <- private$collection$getRESTService()\n REST$copy(self$getRelativePath(),\n paste0(newParent$getRelativePath(), \"/\", nameAndPath$name),\n private$collection$uuid)\n\n newContent <- self$duplicate(nameAndPath$name)\n newContent$setCollection(self$getCollection(), setRecursively = TRUE)\n newContent$setParent(newParent)\n private$attachToNewParent(newContent, newParent)\n\n newContent\n },\n\n #' @description\n #' Duplicate Subcollection and gives it a new name.\n #' @param newName New name for duplicated file.\n duplicate = function(newName = NULL)\n {\n name <- if(!is.null(newName)) newName else private$name\n root <- Subcollection$new(name)\n for(child in private$children)\n root$add(child$duplicate())\n\n root\n },\n\n #' @description\n #' If name is valid, returns ArvadosFile or Subcollection specified by relativePath, else returns NULL.\n #' @param name Name of the file.\n get = function(name)\n {\n for(child in private$children)\n {\n if(child$getName() == name)\n return(child)\n }\n\n return(NULL)\n },\n\n #' @description\n #' Returns files in Subcollection.\n getFirst = function()\n {\n if(length(private$children) == 0)\n return(NULL)\n\n private$children[[1]]\n },\n\n #' @description\n #' Sets Collection by its UUID.\n setCollection = function(collection, setRecursively = TRUE)\n {\n private$collection = collection\n\n if(setRecursively)\n {\n for(child in private$children)\n child$setCollection(collection)\n }\n },\n\n #' @description\n #' Returns Collection of Subcollection.\n getCollection = function() private$collection,\n\n #' @description\n #' Returns Collection UUID.\n getParent = function() private$parent,\n\n #' @description\n #' Sets new Collection.\n setParent = function(newParent) private$parent <- newParent\n ),\n\n private = list(\n\n name = NULL,\n children = NULL,\n parent = NULL,\n collection = NULL,\n\n removeChild = function(name)\n {\n numberOfChildren = length(private$children)\n if(numberOfChildren > 0)\n {\n for(childIndex in 1:numberOfChildren)\n {\n if(private$children[[childIndex]]$getName() == name)\n {\n private$children = private$children[-childIndex]\n return()\n }\n }\n }\n },\n\n attachToNewParent = function(content, newParent)\n {\n # We temporary set parents collection to NULL. This will ensure that\n # add method doesn't post this subcollection to REST.\n # We also need to set content's collection to NULL because\n # add method throws exception if we try to add content that already\n # belongs to a collection.\n parentsCollection <- newParent$getCollection()\n content$setCollection(NULL, setRecursively = FALSE)\n newParent$setCollection(NULL, setRecursively = FALSE)\n newParent$add(content)\n content$setCollection(parentsCollection, setRecursively = FALSE)\n newParent$setCollection(parentsCollection, setRecursively = FALSE)\n },\n\n dettachFromCurrentParent = function()\n {\n # We temporary set parents collection to NULL. This will ensure that\n # remove method doesn't remove this subcollection from REST.\n parent <- private$parent\n parentsCollection <- parent$getCollection()\n parent$setCollection(NULL, setRecursively = FALSE)\n parent$remove(private$name)\n parent$setCollection(parentsCollection, setRecursively = FALSE)\n },\n\n getContentAsCharVector = function(fullPath = TRUE)\n {\n content <- NULL\n\n if(fullPath)\n {\n for(child in private$children)\n content <- c(content, child$getFileListing())\n\n if(private$name != \"\")\n content <- unlist(paste0(private$name, \"/\", content))\n }\n else\n {\n for(child in private$children)\n content <- c(content, child$getName())\n }\n\n content\n }\n ),\n\n cloneable = FALSE\n)\n\n#' print.Subcollection\n#'\n#' Custom print function for Subcollection class\n#'\n#' @param x Instance of Subcollection class\n#' @param ... Optional arguments.\n#' @export\nprint.Subcollection = function(x, ...)\n{\n collection <- NULL\n relativePath <- x$getRelativePath()\n\n if(!is.null(x$getCollection()))\n {\n collection <- x$getCollection()$uuid\n\n if(!x$getName() == \"\")\n relativePath <- paste0(\"/\", relativePath)\n }\n\n cat(paste0(\"Type: \", \"\\\"\", \"Arvados Subcollection\", \"\\\"\"), sep = \"\\n\")\n cat(paste0(\"Name: \", \"\\\"\", x$getName(), \"\\\"\"), sep = \"\\n\")\n cat(paste0(\"Relative path: \", \"\\\"\", relativePath, \"\\\"\"), sep = \"\\n\")\n cat(paste0(\"Collection: \", \"\\\"\", collection, \"\\\"\"), sep = \"\\n\")\n}\n" }, { "path": "contrib/R-sdk/R/util.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\n#' listAll\n#'\n#' List all resources even if the number of items is greater than maximum API limit.\n#'\n#' @param fn Arvados method used to retrieve items from REST service.\n#' @param ... Optional arguments which will be pased to fn .\n#' @examples\n#' \\dontrun{\n#' arv <- Arvados$new(\"your Arvados token\", \"example.arvadosapi.com\")\n#' cl <- listAll(arv$collections.list, filters = list(list(\"name\", \"like\", \"test%\"))\n#' }\n#' @export \nlistAll <- function(fn, ...)\n{\n offset <- 0\n itemsAvailable <- .Machine$integer.max\n items <- c()\n\n while(length(items) < itemsAvailable)\n {\n serverResponse <- fn(offset = offset, ...)\n\n if(!is.null(serverResponse$errors))\n stop(serverResponse$errors)\n\n items <- c(items, serverResponse$items)\n offset <- length(items)\n itemsAvailable <- serverResponse$items_available\n }\n\n items\n}\n\n\n#NOTE: Package private functions\n\ntrimFromStart <- function(sample, trimCharacters)\n{\n if(startsWith(sample, trimCharacters))\n sample <- substr(sample, nchar(trimCharacters) + 1, nchar(sample))\n\n sample\n}\n\ntrimFromEnd <- function(sample, trimCharacters)\n{\n if(endsWith(sample, trimCharacters))\n sample <- substr(sample, 0, nchar(sample) - nchar(trimCharacters))\n\n sample\n}\n\nRListToPythonList <- function(RList, separator = \", \")\n{\n pythonArrayContent <- sapply(RList, function(elementInList)\n {\n if((is.vector(elementInList) || is.list(elementInList)) &&\n length(elementInList) > 1)\n {\n return(RListToPythonList(elementInList, separator))\n }\n else\n {\n return(paste0(\"\\\"\", elementInList, \"\\\"\"))\n }\n })\n\n pythonArray <- paste0(\"[\", paste0(pythonArrayContent, collapse = separator), \"]\")\n pythonArray\n}\n\nappendToStartIfNotExist <- function(sample, characters)\n{\n if(!startsWith(sample, characters))\n sample <- paste0(characters, sample)\n\n sample\n}\n\nsplitToPathAndName = function(path)\n{\n path <- appendToStartIfNotExist(path, \"/\")\n components <- unlist(stringr::str_split(path, \"/\"))\n nameAndPath <- list()\n nameAndPath$name <- components[length(components)]\n nameAndPath$path <- trimFromStart(paste0(components[-length(components)], collapse = \"/\"),\n \"/\")\n nameAndPath\n}\n" }, { "path": "contrib/R-sdk/R/zzz.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\n.onLoad <- function(libName, pkgName)\n{\n minAllowedRVersion <- \"3.3.0\"\n currentRVersion <- getRversion()\n\n if(currentRVersion < minAllowedRVersion)\n print(paste0(\"Minimum R version required to run \", pkgName, \" is \",\n minAllowedRVersion, \". Your current version is \",\n toString(currentRVersion), \". Please update R and try again.\"))\n}\n" }, { "path": "contrib/R-sdk/README.md", "content": "[comment]: # (Copyright © The Arvados Authors. All rights reserved.)\n[comment]: # ()\n[comment]: # (SPDX-License-Identifier: CC-BY-SA-3.0)\n\n# R SDK for Arvados\n\nThis SDK focuses on providing support for accessing Arvados projects, collections, and the files within collections. The API is not final and feedback is solicited from users on ways in which it could be improved.\n\n## Key Topics\n* Installation\n* Usage\n * Initializing API\n * Working with collections\n * Manipulating collection content\n * Working with Arvados projects\n * Help\n* Building the ArvadosR package\n\n## Installation\n\nMinimum R version required to run ArvadosR is 3.3.0.\n\n```r\ninstall.packages(\"ArvadosR\", repos=c(\"https://r.arvados.org\", getOption(\"repos\")[\"CRAN\"]), dependencies=TRUE)\nlibrary('ArvadosR')\n```\n\n> **Note**\n> On Linux, you may have to install supporting packages.\n>\n> On Red Hat, AlmaLinux, and Rocky Linux, this is:\n> ```\n> yum install libxml2-devel openssl-devel curl-devel\n> ```\n>\n> On Debian and Ubuntu, this is:\n> ```\n> apt-get install build-essential libxml2-dev libssl-dev libcurl4-gnutls-dev\n> ```\n\n\n## Usage\n\n### Initializing API\n\n```r\n# use environment variables ARVADOS_API_TOKEN and ARVADOS_API_HOST\narv <- Arvados$new()\n\n# provide them explicitly\narv <- Arvados$new(\"your Arvados token\", \"example.arvadosapi.com\")\n```\n\nOptionally, add `numRetries` parameter to specify number of times to retry failed service requests. Default is 0.\n\n```r\narv <- Arvados$new(\"your Arvados token\", \"example.arvadosapi.com\", numRetries = 3)\n```\n\nThis parameter can be set at any time using `setNumRetries`\n\n```r\narv$setNumRetries(5)\n```\n\n### Working with Arvados projects\n\n##### Create project:\n\n```r\nnewProject <- arv$project_create(name = \"project name\", description = \"project description\", owner_uuid = \"project UUID\", properties = NULL, ensureUniqueName = \"false\")\n```\n\n##### Update project:\n\n```r\nupdatedProject <- arv$project_update(name = \"new project name\", properties = newProperties, uuid = \"projectUUID\")\n```\n\n##### Delete a project:\n\n```r\ndeletedProject <- arv$project_delete(\"uuid\")\n```\n\n#### Find a project:\n\n##### Get a project:\n\n```r\nproject <- arv$project_get(\"uuid\")\n```\n\n##### List projects:\n\n```r\nlist subprojects of a project\nprojects <- arv$project_list(list(list(\"owner_uuid\", \"=\", \"aaaaa-j7d0g-ccccccccccccccc\")))\n\nlist projects which have names beginning with Example\nexamples <- arv$project_list(list(list(\"name\",\"like\",\"Example%\")))\n```\n\n##### List all projects even if the number of items is greater than maximum API limit:\n\n```r\nprojects <- listAll(arv$project_list, list(list(\"name\",\"like\",\"Example%\")))\n```\n\n### Working with collections\n\n#### Create a new collection:\n\n```r\nnewCollection <- arv$collections_create(name = \"collectionTitle\", description = \"collectionDescription\", ownerUUID = \"collectionOwner\", properties = Properties)\n```\n\n#### Update a collection’s metadata:\n\n```r\ncollection <- arv$collections_update(name = \"newCollectionTitle\", description = \"newCollectionDescription\", ownerUUID = \"collectionOwner\", properties = NULL, uuid = \"collectionUUID\")\n```\n\n#### Delete a collection:\n\n```r\ndeletedCollection <- arv$collections_delete(\"uuid\")\n```\n\n#### Find a collection:\n\n#### Get a collection:\n\n```r\ncollection <- arv$collections_get(\"uuid\")\n```\n\nBe aware that the result from `collections_get` is not a Collection class. The object returned from this method lets you access collection fields like “name” and “description”. The Collection class lets you access the files in the collection for reading and writing, and is described in the next section.\n\n#### List collections:\n\n```r\n# offset of 0 and default limit of 100\ncollectionList <- arv$collections_list(list(list(\"name\", \"like\", \"Test%\")))\n\ncollectionList <- arv$collections_list(list(list(\"name\", \"like\", \"Test%\")), limit = 10, offset = 2)\n\n# count of total number of items (may be more than returned due to paging)\ncollectionList$items_available\n\n# items which match the filter criteria\ncollectionList$items\n```\n\n#### List all collections even if the number of items is greater than maximum API limit:\n\n```r\ncollectionList <- listAll(arv$collections_list, list(list(\"name\", \"like\", \"Test%\")))\n```\n\n### Manipulating collection content\n\n#### Initialize a collection object:\n\n```r\ncollection <- Collection$new(arv, \"uuid\")\n```\n\n#### Get list of files:\n\n```r\nfiles <- collection$getFileListing()\n```\n\n#### Get ArvadosFile or Subcollection from internal tree-like structure:\n\n```r\narvadosFile <- collection$get(\"location/to/my/file.cpp\")\n# or\narvadosSubcollection <- collection$get(\"location/to/my/directory/\")\n```\n\n#### Read a table:\n\n```r\narvadosFile <- collection$get(\"myinput.txt\")\narvConnection <- arvadosFile$connection(\"r\")\nmytable <- read.table(arvConnection)\n```\n\n#### Write a table:\n\n```r\narvadosFile <- collection$create(\"myoutput.txt\")[[1]]\narvConnection <- arvadosFile$connection(\"w\")\nwrite.table(mytable, arvConnection)\narvadosFile$flush()\n```\n\n#### Read a table from a tab delimited file:\n\n```r\narvadosFile <- collection$get(\"myinput.txt\")\narvConnection <- arvadosFile$connection(\"r\")\nmytable <- read.delim(arvConnection)\n```\n\n#### Read a gzip compressed R object:\n\n```r\nobj <- readRDS(gzcon(coll$get(\"abc.RDS\")$connection(\"rb\")))\n```\n\n#### Write to existing file (overwrites current content of the file):\n\n```r\narvadosFile <- collection$get(\"location/to/my/file.cpp\")\narvadosFile$write(\"This is new file content\")\n```\n\n#### Read whole file or just a portion of it:\n\n```r\nfileContent <- arvadosFile$read()\nfileContent <- arvadosFile$read(\"text\")\nfileContent <- arvadosFile$read(\"raw\", offset = 1024, length = 512)\n```\n\n#### Read various file types:\n\nChooses file type based on file name extension. Recognized file extensions: 'txt', 'xlsx', 'csv', 'tsv', 'fasta', 'dat', 'bin', 'rds', 'rdata'.\n\n```r\ncollection <- Collection$new(arv, collectionUUID)\nreadFile <- collection$readArvFile(arvadosFile, istable = 'yes') # table\nreadFile <- collection$readArvFile(arvadosFile, istable = 'no') # text\nreadFile <- collection$readArvFile(arvadosFile) # xlsx, csv, tsv, rds, rdata\nreadFile <- collection$readArvFile(arvadosFile, fileclass = 'fasta') # fasta\nreadFile <- collection$readArvFile(arvadosFile, Ncol= 4, Nrow = 32) # binary data.frame, only numbers\nreadFile <- collection$readArvFile(arvadosFile, Ncol = 5, Nrow = 150, istable = \"factor\") # binary data.frame with factor or text\n```\n\n#### Get ArvadosFile or Subcollection size:\n\n```r\nsize <- arvadosFile$getSizeInBytes()\n# or\nsize <- arvadosSubcollection$getSizeInBytes()\n```\n\n#### Create new file in a collection (returns a vector of one or more ArvadosFile objects):\n\n```r\ncollection$create(files)\n```\n\n**Example**\n\n```\nmainFile <- collection$create(\"cpp/src/main.cpp\")[[1]]\nfileList <- collection$create(c(\"cpp/src/main.cpp\", \"cpp/src/util.h\"))\n```\n\n#### Delete file from a collection:\n\n```r\ncollection$remove(\"location/to/my/file.cpp\")\n```\n\nYou can remove both Subcollection and ArvadosFile. If subcollection contains more files or folders they will be removed recursively.\n\n> **Note**\n> You can also remove multiple files at once:\n> ```\n> collection$remove(c(\"path/to/my/file.cpp\", \"path/to/other/file.cpp\"))\n> ```\n\n#### Delete file or folder from a Subcollection:\n\n```r\nsubcollection <- collection$get(\"mySubcollection/\")\nsubcollection$remove(\"fileInsideSubcollection.exe\")\nsubcollection$remove(\"folderInsideSubcollection/\")\n```\n\n#### Move or rename a file or folder within a collection (moving between collections is currently not supported):\n\n##### Directly from collection\n\n```r\ncollection$move(\"folder/file.cpp\", \"file.cpp\")\n```\n\n##### Or from file\n\n```r\nfile <- collection$get(\"location/to/my/file.cpp\")\nfile$move(\"newDestination/file.cpp\")\n```\n\n##### Or from subcollection\n\n```r\nsubcollection <- collection$get(\"location/to/folder\")\nsubcollection$move(\"newDestination/folder\")\n```\n\n> **Note**\n> Make sure to include new file name in destination. In second example `file$move(“newDestination/”)` will not work.\n\n#### Copy file or folder within a collection (copying between collections is currently not supported):\n\n##### Directly from collection\n\n```r\ncollection$copy(\"folder/file.cpp\", \"file.cpp\")\n```\n\n##### Or from file\n\n```r\nfile <- collection$get(\"location/to/my/file.cpp\")\nfile$copy(\"destination/file.cpp\")\n```\n\n##### Or from subcollection\n\n```r\nsubcollection <- collection$get(\"location/to/folder\")\nsubcollection$copy(\"destination/folder\")\n```\n\n\n### Help\n\n#### View help page of Arvados classes by puting `?` before class name:\n\n```r\n?Arvados\n?Collection\n?Subcollection\n?ArvadosFile\n```\n\n#### View help page of any method defined in Arvados class by puting `?` before method name:\n\n```r\n?collections_update\n?workflows_get\n```\n\n \n\n## Building the ArvadosR package\n\n```\nmake package\n```\n\nThis will create a tarball of the ArvadosR package in the current directory.\n\n \n\n## Documentation\n\nComplete documentation, including the [User Guide](https://doc.arvados.org/user/index.html), [Installation documentation](https://doc.arvados.org/install/index.html), [Administrator documentation](https://doc.arvados.org/admin/index.html) and\n[API documentation](https://doc.arvados.org/api/index.html) is available at http://doc.arvados.org/\n\n## Community\n\nVisit [Arvados Community and Getting Help](https://doc.arvados.org/user/getting_started/community.html).\n\n## Reporting bugs\n\n[Report an issue on GitHub](https://github.com/arvados/arvados/issues/new)\n\n## Licensing\n\nArvados is Free Software. See [Arvados Free Software Licenses](https://doc.arvados.org/user/copying/copying.html) for information about the open source licenses used in Arvados.\n" }, { "path": "contrib/R-sdk/arvados-v1-discovery.json", "content": "{\n \"auth\": {\n \"oauth2\": {\n \"scopes\": {\n \"https://api.arvados.org/auth/arvados\": {\n \"description\": \"View and manage objects\"\n },\n \"https://api.arvados.org/auth/arvados.readonly\": {\n \"description\": \"View objects\"\n }\n }\n }\n },\n \"basePath\": \"/arvados/v1/\",\n \"batchPath\": \"batch\",\n \"description\": \"The API to interact with Arvados.\",\n \"discoveryVersion\": \"v1\",\n \"documentationLink\": \"http://doc.arvados.org/api/index.html\",\n \"id\": \"arvados:v1\",\n \"kind\": \"discovery#restDescription\",\n \"name\": \"arvados\",\n \"parameters\": {\n \"alt\": {\n \"type\": \"string\",\n \"description\": \"Data format for the response.\",\n \"default\": \"json\",\n \"enum\": [\n \"json\"\n ],\n \"enumDescriptions\": [\n \"Responses with Content-Type of application/json\"\n ],\n \"location\": \"query\"\n },\n \"fields\": {\n \"type\": \"string\",\n \"description\": \"Selector specifying which fields to include in a partial response.\",\n \"location\": \"query\"\n },\n \"key\": {\n \"type\": \"string\",\n \"description\": \"API key. Your API key identifies your project and provides you with API access, quota, and reports. Required unless you provide an OAuth 2.0 token.\",\n \"location\": \"query\"\n },\n \"oauth_token\": {\n \"type\": \"string\",\n \"description\": \"OAuth 2.0 token for the current user.\",\n \"location\": \"query\"\n }\n },\n \"protocol\": \"rest\",\n \"resources\": {\n \"api_client_authorizations\": {\n \"methods\": {\n \"get\": {\n \"id\": \"arvados.api_client_authorizations.get\",\n \"path\": \"api_client_authorizations/{uuid}\",\n \"httpMethod\": \"GET\",\n \"description\": \"Get a ApiClientAuthorization record by UUID.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the ApiClientAuthorization to return.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n }\n },\n \"parameterOrder\": [\n \"uuid\"\n ],\n \"response\": {\n \"$ref\": \"ApiClientAuthorization\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"list\": {\n \"id\": \"arvados.api_client_authorizations.list\",\n \"path\": \"api_client_authorizations\",\n \"httpMethod\": \"GET\",\n \"description\": \"Retrieve a ApiClientAuthorizationList.\",\n \"parameters\": {\n \"filters\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"Filters to limit which objects are returned by their attributes.\\nRefer to the [filters reference][] for more information about how to write filters.\\n\\n[filters reference]: https://doc.arvados.org/api/methods.html#filters\\n\",\n \"location\": \"query\"\n },\n \"where\": {\n \"type\": \"object\",\n \"required\": false,\n \"description\": \"An object to limit which objects are returned by their attributes.\\nThe keys of this object are attribute names.\\nEach value is either a single matching value or an array of matching values for that attribute.\\nThe `filters` parameter is more flexible and preferred.\\n\",\n \"location\": \"query\"\n },\n \"order\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"An array of strings to set the order in which matching objects are returned.\\nEach string has the format ` `.\\n`DIRECTION` can be `asc` or omitted for ascending, or `desc` for descending.\\n\",\n \"location\": \"query\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return from each matching object.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"distinct\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If this is true, and multiple objects have the same values\\nfor the attributes that you specify in the `select` parameter, then each unique\\nset of values will only be returned once in the result set.\\n\",\n \"location\": \"query\"\n },\n \"limit\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"100\",\n \"description\": \"The maximum number of objects to return in the result.\\nNote that the API may return fewer results than this if your request hits other\\nlimits set by the administrator.\\n\",\n \"location\": \"query\"\n },\n \"offset\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"0\",\n \"description\": \"Return matching objects starting from this index.\\nNote that result indexes may change if objects are modified in between a series\\nof list calls.\\n\",\n \"location\": \"query\"\n },\n \"count\": {\n \"type\": \"string\",\n \"required\": false,\n \"default\": \"exact\",\n \"description\": \"A string to determine result counting behavior. Supported values are:\\n\\n * `\\\"exact\\\"`: The response will include an `items_available` field that\\n counts the number of objects that matched this search criteria,\\n including ones not included in `items`.\\n\\n * `\\\"none\\\"`: The response will not include an `items_avaliable`\\n field. This improves performance by returning a result as soon as enough\\n `items` have been loaded for this result.\\n\\n\",\n \"location\": \"query\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster to return objects from\",\n \"location\": \"query\",\n \"required\": false\n },\n \"bypass_federation\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If true, do not return results from other clusters in the\\nfederation, only the cluster that received the request.\\nYou must be an administrator to use this flag.\\n\",\n \"location\": \"query\"\n }\n },\n \"response\": {\n \"$ref\": \"ApiClientAuthorizationList\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"create\": {\n \"id\": \"arvados.api_client_authorizations.create\",\n \"path\": \"api_client_authorizations\",\n \"httpMethod\": \"POST\",\n \"description\": \"Create a new ApiClientAuthorization.\",\n \"parameters\": {\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"ensure_unique_name\": {\n \"type\": \"boolean\",\n \"description\": \"If the given name is already used by this owner, adjust the name to ensure uniqueness instead of returning an error.\",\n \"location\": \"query\",\n \"required\": false,\n \"default\": \"false\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster where this object should be created.\",\n \"location\": \"query\",\n \"required\": false\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"api_client_authorization\": {\n \"$ref\": \"ApiClientAuthorization\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"ApiClientAuthorization\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"update\": {\n \"id\": \"arvados.api_client_authorizations.update\",\n \"path\": \"api_client_authorizations/{uuid}\",\n \"httpMethod\": \"PUT\",\n \"description\": \"Update attributes of an existing ApiClientAuthorization.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the ApiClientAuthorization to update.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"api_client_authorization\": {\n \"$ref\": \"ApiClientAuthorization\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"ApiClientAuthorization\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"delete\": {\n \"id\": \"arvados.api_client_authorizations.delete\",\n \"path\": \"api_client_authorizations/{uuid}\",\n \"httpMethod\": \"DELETE\",\n \"description\": \"Delete an existing ApiClientAuthorization.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the ApiClientAuthorization to delete.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"ApiClientAuthorization\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"create_system_auth\": {\n \"id\": \"arvados.api_client_authorizations.create_system_auth\",\n \"path\": \"api_client_authorizations/create_system_auth\",\n \"httpMethod\": \"POST\",\n \"description\": \"Create a token for the system (\\\"root\\\") user.\",\n \"parameters\": {\n \"scopes\": {\n \"type\": \"array\",\n \"required\": false,\n \"default\": \"[\\\"all\\\"]\",\n \"description\": \"An array of strings defining the scope of resources this token will be allowed to access. Refer to the [scopes reference][] for details.\\n\\n[scopes reference]: https://doc.arvados.org/api/tokens.html#scopes\\n\",\n \"location\": \"query\"\n }\n },\n \"response\": {\n \"$ref\": \"ApiClientAuthorization\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"current\": {\n \"id\": \"arvados.api_client_authorizations.current\",\n \"path\": \"api_client_authorizations/current\",\n \"httpMethod\": \"GET\",\n \"description\": \"Return all metadata for the token used to authorize this request.\",\n \"parameters\": {},\n \"response\": {\n \"$ref\": \"ApiClientAuthorization\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n }\n }\n },\n \"authorized_keys\": {\n \"methods\": {\n \"get\": {\n \"id\": \"arvados.authorized_keys.get\",\n \"path\": \"authorized_keys/{uuid}\",\n \"httpMethod\": \"GET\",\n \"description\": \"Get a AuthorizedKey record by UUID.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the AuthorizedKey to return.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n }\n },\n \"parameterOrder\": [\n \"uuid\"\n ],\n \"response\": {\n \"$ref\": \"AuthorizedKey\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"list\": {\n \"id\": \"arvados.authorized_keys.list\",\n \"path\": \"authorized_keys\",\n \"httpMethod\": \"GET\",\n \"description\": \"Retrieve a AuthorizedKeyList.\",\n \"parameters\": {\n \"filters\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"Filters to limit which objects are returned by their attributes.\\nRefer to the [filters reference][] for more information about how to write filters.\\n\\n[filters reference]: https://doc.arvados.org/api/methods.html#filters\\n\",\n \"location\": \"query\"\n },\n \"where\": {\n \"type\": \"object\",\n \"required\": false,\n \"description\": \"An object to limit which objects are returned by their attributes.\\nThe keys of this object are attribute names.\\nEach value is either a single matching value or an array of matching values for that attribute.\\nThe `filters` parameter is more flexible and preferred.\\n\",\n \"location\": \"query\"\n },\n \"order\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"An array of strings to set the order in which matching objects are returned.\\nEach string has the format ` `.\\n`DIRECTION` can be `asc` or omitted for ascending, or `desc` for descending.\\n\",\n \"location\": \"query\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return from each matching object.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"distinct\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If this is true, and multiple objects have the same values\\nfor the attributes that you specify in the `select` parameter, then each unique\\nset of values will only be returned once in the result set.\\n\",\n \"location\": \"query\"\n },\n \"limit\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"100\",\n \"description\": \"The maximum number of objects to return in the result.\\nNote that the API may return fewer results than this if your request hits other\\nlimits set by the administrator.\\n\",\n \"location\": \"query\"\n },\n \"offset\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"0\",\n \"description\": \"Return matching objects starting from this index.\\nNote that result indexes may change if objects are modified in between a series\\nof list calls.\\n\",\n \"location\": \"query\"\n },\n \"count\": {\n \"type\": \"string\",\n \"required\": false,\n \"default\": \"exact\",\n \"description\": \"A string to determine result counting behavior. Supported values are:\\n\\n * `\\\"exact\\\"`: The response will include an `items_available` field that\\n counts the number of objects that matched this search criteria,\\n including ones not included in `items`.\\n\\n * `\\\"none\\\"`: The response will not include an `items_avaliable`\\n field. This improves performance by returning a result as soon as enough\\n `items` have been loaded for this result.\\n\\n\",\n \"location\": \"query\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster to return objects from\",\n \"location\": \"query\",\n \"required\": false\n },\n \"bypass_federation\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If true, do not return results from other clusters in the\\nfederation, only the cluster that received the request.\\nYou must be an administrator to use this flag.\\n\",\n \"location\": \"query\"\n }\n },\n \"response\": {\n \"$ref\": \"AuthorizedKeyList\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"create\": {\n \"id\": \"arvados.authorized_keys.create\",\n \"path\": \"authorized_keys\",\n \"httpMethod\": \"POST\",\n \"description\": \"Create a new AuthorizedKey.\",\n \"parameters\": {\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"ensure_unique_name\": {\n \"type\": \"boolean\",\n \"description\": \"If the given name is already used by this owner, adjust the name to ensure uniqueness instead of returning an error.\",\n \"location\": \"query\",\n \"required\": false,\n \"default\": \"false\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster where this object should be created.\",\n \"location\": \"query\",\n \"required\": false\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"authorized_key\": {\n \"$ref\": \"AuthorizedKey\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"AuthorizedKey\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"update\": {\n \"id\": \"arvados.authorized_keys.update\",\n \"path\": \"authorized_keys/{uuid}\",\n \"httpMethod\": \"PUT\",\n \"description\": \"Update attributes of an existing AuthorizedKey.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the AuthorizedKey to update.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"authorized_key\": {\n \"$ref\": \"AuthorizedKey\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"AuthorizedKey\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"delete\": {\n \"id\": \"arvados.authorized_keys.delete\",\n \"path\": \"authorized_keys/{uuid}\",\n \"httpMethod\": \"DELETE\",\n \"description\": \"Delete an existing AuthorizedKey.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the AuthorizedKey to delete.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"AuthorizedKey\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n }\n }\n },\n \"collections\": {\n \"methods\": {\n \"get\": {\n \"id\": \"arvados.collections.get\",\n \"path\": \"collections/{uuid}\",\n \"httpMethod\": \"GET\",\n \"description\": \"Get a Collection record by UUID.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Collection to return.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"include_trash\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"Show collection even if its `is_trashed` attribute is true.\",\n \"location\": \"query\"\n }\n },\n \"parameterOrder\": [\n \"uuid\"\n ],\n \"response\": {\n \"$ref\": \"Collection\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"list\": {\n \"id\": \"arvados.collections.list\",\n \"path\": \"collections\",\n \"httpMethod\": \"GET\",\n \"description\": \"Retrieve a CollectionList.\",\n \"parameters\": {\n \"filters\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"Filters to limit which objects are returned by their attributes.\\nRefer to the [filters reference][] for more information about how to write filters.\\n\\n[filters reference]: https://doc.arvados.org/api/methods.html#filters\\n\",\n \"location\": \"query\"\n },\n \"where\": {\n \"type\": \"object\",\n \"required\": false,\n \"description\": \"An object to limit which objects are returned by their attributes.\\nThe keys of this object are attribute names.\\nEach value is either a single matching value or an array of matching values for that attribute.\\nThe `filters` parameter is more flexible and preferred.\\n\",\n \"location\": \"query\"\n },\n \"order\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"An array of strings to set the order in which matching objects are returned.\\nEach string has the format ` `.\\n`DIRECTION` can be `asc` or omitted for ascending, or `desc` for descending.\\n\",\n \"location\": \"query\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return from each matching object.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"distinct\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If this is true, and multiple objects have the same values\\nfor the attributes that you specify in the `select` parameter, then each unique\\nset of values will only be returned once in the result set.\\n\",\n \"location\": \"query\"\n },\n \"limit\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"100\",\n \"description\": \"The maximum number of objects to return in the result.\\nNote that the API may return fewer results than this if your request hits other\\nlimits set by the administrator.\\n\",\n \"location\": \"query\"\n },\n \"offset\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"0\",\n \"description\": \"Return matching objects starting from this index.\\nNote that result indexes may change if objects are modified in between a series\\nof list calls.\\n\",\n \"location\": \"query\"\n },\n \"count\": {\n \"type\": \"string\",\n \"required\": false,\n \"default\": \"exact\",\n \"description\": \"A string to determine result counting behavior. Supported values are:\\n\\n * `\\\"exact\\\"`: The response will include an `items_available` field that\\n counts the number of objects that matched this search criteria,\\n including ones not included in `items`.\\n\\n * `\\\"none\\\"`: The response will not include an `items_avaliable`\\n field. This improves performance by returning a result as soon as enough\\n `items` have been loaded for this result.\\n\\n\",\n \"location\": \"query\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster to return objects from\",\n \"location\": \"query\",\n \"required\": false\n },\n \"bypass_federation\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If true, do not return results from other clusters in the\\nfederation, only the cluster that received the request.\\nYou must be an administrator to use this flag.\\n\",\n \"location\": \"query\"\n },\n \"include_trash\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"Include collections whose `is_trashed` attribute is true.\",\n \"location\": \"query\"\n },\n \"include_old_versions\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"Include past collection versions.\",\n \"location\": \"query\"\n }\n },\n \"response\": {\n \"$ref\": \"CollectionList\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"create\": {\n \"id\": \"arvados.collections.create\",\n \"path\": \"collections\",\n \"httpMethod\": \"POST\",\n \"description\": \"Create a new Collection.\",\n \"parameters\": {\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"ensure_unique_name\": {\n \"type\": \"boolean\",\n \"description\": \"If the given name is already used by this owner, adjust the name to ensure uniqueness instead of returning an error.\",\n \"location\": \"query\",\n \"required\": false,\n \"default\": \"false\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster where this object should be created.\",\n \"location\": \"query\",\n \"required\": false\n },\n \"replace_files\": {\n \"type\": \"object\",\n \"description\": \"Add, delete, and replace files and directories with new content\\nand/or content from other collections. Refer to the\\n[replace_files reference][] for details.\\n\\n[replace_files reference]: https://doc.arvados.org/api/methods/collections.html#replace_files\\n\\n\",\n \"required\": false,\n \"location\": \"query\",\n \"properties\": {},\n \"additionalProperties\": {\n \"type\": \"string\"\n }\n },\n \"replace_segments\": {\n \"type\": \"object\",\n \"description\": \"Replace existing block segments in the collection with new segments.\\nRefer to the [replace_segments reference][] for details.\\n\\n[replace_segments reference]: https://doc.arvados.org/api/methods/collections.html#replace_segments\\n\\n\",\n \"required\": false,\n \"location\": \"query\",\n \"properties\": {},\n \"additionalProperties\": {\n \"type\": \"string\"\n }\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"collection\": {\n \"$ref\": \"Collection\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"Collection\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"update\": {\n \"id\": \"arvados.collections.update\",\n \"path\": \"collections/{uuid}\",\n \"httpMethod\": \"PUT\",\n \"description\": \"Update attributes of an existing Collection.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Collection to update.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"replace_files\": {\n \"type\": \"object\",\n \"description\": \"Add, delete, and replace files and directories with new content\\nand/or content from other collections. Refer to the\\n[replace_files reference][] for details.\\n\\n[replace_files reference]: https://doc.arvados.org/api/methods/collections.html#replace_files\\n\\n\",\n \"required\": false,\n \"location\": \"query\",\n \"properties\": {},\n \"additionalProperties\": {\n \"type\": \"string\"\n }\n },\n \"replace_segments\": {\n \"type\": \"object\",\n \"description\": \"Replace existing block segments in the collection with new segments.\\nRefer to the [replace_segments reference][] for details.\\n\\n[replace_segments reference]: https://doc.arvados.org/api/methods/collections.html#replace_segments\\n\\n\",\n \"required\": false,\n \"location\": \"query\",\n \"properties\": {},\n \"additionalProperties\": {\n \"type\": \"string\"\n }\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"collection\": {\n \"$ref\": \"Collection\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"Collection\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"delete\": {\n \"id\": \"arvados.collections.delete\",\n \"path\": \"collections/{uuid}\",\n \"httpMethod\": \"DELETE\",\n \"description\": \"Delete an existing Collection.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Collection to delete.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"Collection\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"provenance\": {\n \"id\": \"arvados.collections.provenance\",\n \"path\": \"collections/{uuid}/provenance\",\n \"httpMethod\": \"GET\",\n \"description\": \"Detail the provenance of a given collection.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Collection to query.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"Collection\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"used_by\": {\n \"id\": \"arvados.collections.used_by\",\n \"path\": \"collections/{uuid}/used_by\",\n \"httpMethod\": \"GET\",\n \"description\": \"Detail where a given collection has been used.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Collection to query.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"Collection\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"trash\": {\n \"id\": \"arvados.collections.trash\",\n \"path\": \"collections/{uuid}/trash\",\n \"httpMethod\": \"POST\",\n \"description\": \"Trash a collection.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Collection to update.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"Collection\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"untrash\": {\n \"id\": \"arvados.collections.untrash\",\n \"path\": \"collections/{uuid}/untrash\",\n \"httpMethod\": \"POST\",\n \"description\": \"Untrash a collection.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Collection to update.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"Collection\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n }\n }\n },\n \"computed_permissions\": {\n \"methods\": {\n \"list\": {\n \"id\": \"arvados.computed_permissions.list\",\n \"path\": \"computed_permissions\",\n \"httpMethod\": \"GET\",\n \"description\": \"Retrieve a ComputedPermissionList.\",\n \"parameters\": {\n \"filters\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"Filters to limit which objects are returned by their attributes.\\nRefer to the [filters reference][] for more information about how to write filters.\\n\\n[filters reference]: https://doc.arvados.org/api/methods.html#filters\\n\",\n \"location\": \"query\"\n },\n \"where\": {\n \"type\": \"object\",\n \"required\": false,\n \"description\": \"An object to limit which objects are returned by their attributes.\\nThe keys of this object are attribute names.\\nEach value is either a single matching value or an array of matching values for that attribute.\\nThe `filters` parameter is more flexible and preferred.\\n\",\n \"location\": \"query\"\n },\n \"order\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"An array of strings to set the order in which matching objects are returned.\\nEach string has the format ` `.\\n`DIRECTION` can be `asc` or omitted for ascending, or `desc` for descending.\\n\",\n \"location\": \"query\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return from each matching object.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"distinct\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If this is true, and multiple objects have the same values\\nfor the attributes that you specify in the `select` parameter, then each unique\\nset of values will only be returned once in the result set.\\n\",\n \"location\": \"query\"\n },\n \"limit\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"100\",\n \"description\": \"The maximum number of objects to return in the result.\\nNote that the API may return fewer results than this if your request hits other\\nlimits set by the administrator.\\n\",\n \"location\": \"query\"\n },\n \"count\": {\n \"type\": \"string\",\n \"required\": false,\n \"default\": \"exact\",\n \"description\": \"A string to determine result counting behavior. Supported values are:\\n\\n * `\\\"exact\\\"`: The response will include an `items_available` field that\\n counts the number of objects that matched this search criteria,\\n including ones not included in `items`.\\n\\n * `\\\"none\\\"`: The response will not include an `items_avaliable`\\n field. This improves performance by returning a result as soon as enough\\n `items` have been loaded for this result.\\n\\n\",\n \"location\": \"query\"\n }\n },\n \"response\": {\n \"$ref\": \"ComputedPermissionList\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n }\n }\n },\n \"containers\": {\n \"methods\": {\n \"get\": {\n \"id\": \"arvados.containers.get\",\n \"path\": \"containers/{uuid}\",\n \"httpMethod\": \"GET\",\n \"description\": \"Get a Container record by UUID.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Container to return.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n }\n },\n \"parameterOrder\": [\n \"uuid\"\n ],\n \"response\": {\n \"$ref\": \"Container\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"list\": {\n \"id\": \"arvados.containers.list\",\n \"path\": \"containers\",\n \"httpMethod\": \"GET\",\n \"description\": \"Retrieve a ContainerList.\",\n \"parameters\": {\n \"filters\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"Filters to limit which objects are returned by their attributes.\\nRefer to the [filters reference][] for more information about how to write filters.\\n\\n[filters reference]: https://doc.arvados.org/api/methods.html#filters\\n\",\n \"location\": \"query\"\n },\n \"where\": {\n \"type\": \"object\",\n \"required\": false,\n \"description\": \"An object to limit which objects are returned by their attributes.\\nThe keys of this object are attribute names.\\nEach value is either a single matching value or an array of matching values for that attribute.\\nThe `filters` parameter is more flexible and preferred.\\n\",\n \"location\": \"query\"\n },\n \"order\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"An array of strings to set the order in which matching objects are returned.\\nEach string has the format ` `.\\n`DIRECTION` can be `asc` or omitted for ascending, or `desc` for descending.\\n\",\n \"location\": \"query\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return from each matching object.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"distinct\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If this is true, and multiple objects have the same values\\nfor the attributes that you specify in the `select` parameter, then each unique\\nset of values will only be returned once in the result set.\\n\",\n \"location\": \"query\"\n },\n \"limit\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"100\",\n \"description\": \"The maximum number of objects to return in the result.\\nNote that the API may return fewer results than this if your request hits other\\nlimits set by the administrator.\\n\",\n \"location\": \"query\"\n },\n \"offset\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"0\",\n \"description\": \"Return matching objects starting from this index.\\nNote that result indexes may change if objects are modified in between a series\\nof list calls.\\n\",\n \"location\": \"query\"\n },\n \"count\": {\n \"type\": \"string\",\n \"required\": false,\n \"default\": \"exact\",\n \"description\": \"A string to determine result counting behavior. Supported values are:\\n\\n * `\\\"exact\\\"`: The response will include an `items_available` field that\\n counts the number of objects that matched this search criteria,\\n including ones not included in `items`.\\n\\n * `\\\"none\\\"`: The response will not include an `items_avaliable`\\n field. This improves performance by returning a result as soon as enough\\n `items` have been loaded for this result.\\n\\n\",\n \"location\": \"query\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster to return objects from\",\n \"location\": \"query\",\n \"required\": false\n },\n \"bypass_federation\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If true, do not return results from other clusters in the\\nfederation, only the cluster that received the request.\\nYou must be an administrator to use this flag.\\n\",\n \"location\": \"query\"\n }\n },\n \"response\": {\n \"$ref\": \"ContainerList\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"create\": {\n \"id\": \"arvados.containers.create\",\n \"path\": \"containers\",\n \"httpMethod\": \"POST\",\n \"description\": \"Create a new Container.\",\n \"parameters\": {\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"ensure_unique_name\": {\n \"type\": \"boolean\",\n \"description\": \"If the given name is already used by this owner, adjust the name to ensure uniqueness instead of returning an error.\",\n \"location\": \"query\",\n \"required\": false,\n \"default\": \"false\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster where this object should be created.\",\n \"location\": \"query\",\n \"required\": false\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"container\": {\n \"$ref\": \"Container\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"Container\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"update\": {\n \"id\": \"arvados.containers.update\",\n \"path\": \"containers/{uuid}\",\n \"httpMethod\": \"PUT\",\n \"description\": \"Update attributes of an existing Container.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Container to update.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"container\": {\n \"$ref\": \"Container\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"Container\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"delete\": {\n \"id\": \"arvados.containers.delete\",\n \"path\": \"containers/{uuid}\",\n \"httpMethod\": \"DELETE\",\n \"description\": \"Delete an existing Container.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Container to delete.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"Container\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"auth\": {\n \"id\": \"arvados.containers.auth\",\n \"path\": \"containers/{uuid}/auth\",\n \"httpMethod\": \"GET\",\n \"description\": \"Get the API client authorization token associated with this container.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Container to query.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"Container\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"lock\": {\n \"id\": \"arvados.containers.lock\",\n \"path\": \"containers/{uuid}/lock\",\n \"httpMethod\": \"POST\",\n \"description\": \"Lock a container (for a dispatcher to begin running it).\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Container to update.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"Container\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"unlock\": {\n \"id\": \"arvados.containers.unlock\",\n \"path\": \"containers/{uuid}/unlock\",\n \"httpMethod\": \"POST\",\n \"description\": \"Unlock a container (for a dispatcher to stop running it).\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Container to update.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"Container\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"update_priority\": {\n \"id\": \"arvados.containers.update_priority\",\n \"path\": \"containers/{uuid}/update_priority\",\n \"httpMethod\": \"POST\",\n \"description\": \"Recalculate and return the priority of a given container.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Container to update.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"Container\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"secret_mounts\": {\n \"id\": \"arvados.containers.secret_mounts\",\n \"path\": \"containers/{uuid}/secret_mounts\",\n \"httpMethod\": \"GET\",\n \"description\": \"Return secret mount information for the container associated with the API token authorizing this request.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Container to query.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"Container\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"current\": {\n \"id\": \"arvados.containers.current\",\n \"path\": \"containers/current\",\n \"httpMethod\": \"GET\",\n \"description\": \"Return the container record associated with the API token authorizing this request.\",\n \"parameters\": {},\n \"response\": {\n \"$ref\": \"Container\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n }\n }\n },\n \"container_requests\": {\n \"methods\": {\n \"get\": {\n \"id\": \"arvados.container_requests.get\",\n \"path\": \"container_requests/{uuid}\",\n \"httpMethod\": \"GET\",\n \"description\": \"Get a ContainerRequest record by UUID.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the ContainerRequest to return.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"include_trash\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"Show container request even if its owner project is trashed.\",\n \"location\": \"query\"\n }\n },\n \"parameterOrder\": [\n \"uuid\"\n ],\n \"response\": {\n \"$ref\": \"ContainerRequest\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"list\": {\n \"id\": \"arvados.container_requests.list\",\n \"path\": \"container_requests\",\n \"httpMethod\": \"GET\",\n \"description\": \"Retrieve a ContainerRequestList.\",\n \"parameters\": {\n \"filters\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"Filters to limit which objects are returned by their attributes.\\nRefer to the [filters reference][] for more information about how to write filters.\\n\\n[filters reference]: https://doc.arvados.org/api/methods.html#filters\\n\",\n \"location\": \"query\"\n },\n \"where\": {\n \"type\": \"object\",\n \"required\": false,\n \"description\": \"An object to limit which objects are returned by their attributes.\\nThe keys of this object are attribute names.\\nEach value is either a single matching value or an array of matching values for that attribute.\\nThe `filters` parameter is more flexible and preferred.\\n\",\n \"location\": \"query\"\n },\n \"order\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"An array of strings to set the order in which matching objects are returned.\\nEach string has the format ` `.\\n`DIRECTION` can be `asc` or omitted for ascending, or `desc` for descending.\\n\",\n \"location\": \"query\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return from each matching object.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"distinct\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If this is true, and multiple objects have the same values\\nfor the attributes that you specify in the `select` parameter, then each unique\\nset of values will only be returned once in the result set.\\n\",\n \"location\": \"query\"\n },\n \"limit\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"100\",\n \"description\": \"The maximum number of objects to return in the result.\\nNote that the API may return fewer results than this if your request hits other\\nlimits set by the administrator.\\n\",\n \"location\": \"query\"\n },\n \"offset\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"0\",\n \"description\": \"Return matching objects starting from this index.\\nNote that result indexes may change if objects are modified in between a series\\nof list calls.\\n\",\n \"location\": \"query\"\n },\n \"count\": {\n \"type\": \"string\",\n \"required\": false,\n \"default\": \"exact\",\n \"description\": \"A string to determine result counting behavior. Supported values are:\\n\\n * `\\\"exact\\\"`: The response will include an `items_available` field that\\n counts the number of objects that matched this search criteria,\\n including ones not included in `items`.\\n\\n * `\\\"none\\\"`: The response will not include an `items_avaliable`\\n field. This improves performance by returning a result as soon as enough\\n `items` have been loaded for this result.\\n\\n\",\n \"location\": \"query\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster to return objects from\",\n \"location\": \"query\",\n \"required\": false\n },\n \"bypass_federation\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If true, do not return results from other clusters in the\\nfederation, only the cluster that received the request.\\nYou must be an administrator to use this flag.\\n\",\n \"location\": \"query\"\n },\n \"include_trash\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"Include container requests whose owner project is trashed.\",\n \"location\": \"query\"\n }\n },\n \"response\": {\n \"$ref\": \"ContainerRequestList\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"create\": {\n \"id\": \"arvados.container_requests.create\",\n \"path\": \"container_requests\",\n \"httpMethod\": \"POST\",\n \"description\": \"Create a new ContainerRequest.\",\n \"parameters\": {\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"ensure_unique_name\": {\n \"type\": \"boolean\",\n \"description\": \"If the given name is already used by this owner, adjust the name to ensure uniqueness instead of returning an error.\",\n \"location\": \"query\",\n \"required\": false,\n \"default\": \"false\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster where this object should be created.\",\n \"location\": \"query\",\n \"required\": false\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"container_request\": {\n \"$ref\": \"ContainerRequest\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"ContainerRequest\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"update\": {\n \"id\": \"arvados.container_requests.update\",\n \"path\": \"container_requests/{uuid}\",\n \"httpMethod\": \"PUT\",\n \"description\": \"Update attributes of an existing ContainerRequest.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the ContainerRequest to update.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"container_request\": {\n \"$ref\": \"ContainerRequest\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"ContainerRequest\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"delete\": {\n \"id\": \"arvados.container_requests.delete\",\n \"path\": \"container_requests/{uuid}\",\n \"httpMethod\": \"DELETE\",\n \"description\": \"Delete an existing ContainerRequest.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the ContainerRequest to delete.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"ContainerRequest\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"container_status\": {\n \"id\": \"arvados.container_requests.container_status\",\n \"path\": \"container_requests/{uuid}/container_status\",\n \"httpMethod\": \"GET\",\n \"description\": \"Return scheduling details for a container request.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"required\": true,\n \"description\": \"The UUID of the container request to query.\",\n \"location\": \"query\"\n }\n },\n \"response\": {\n \"$ref\": \"ContainerRequest\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n }\n }\n },\n \"credentials\": {\n \"methods\": {\n \"get\": {\n \"id\": \"arvados.credentials.get\",\n \"path\": \"credentials/{uuid}\",\n \"httpMethod\": \"GET\",\n \"description\": \"Get a Credential record by UUID.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Credential to return.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n }\n },\n \"parameterOrder\": [\n \"uuid\"\n ],\n \"response\": {\n \"$ref\": \"Credential\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"list\": {\n \"id\": \"arvados.credentials.list\",\n \"path\": \"credentials\",\n \"httpMethod\": \"GET\",\n \"description\": \"Retrieve a CredentialList.\",\n \"parameters\": {\n \"filters\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"Filters to limit which objects are returned by their attributes.\\nRefer to the [filters reference][] for more information about how to write filters.\\n\\n[filters reference]: https://doc.arvados.org/api/methods.html#filters\\n\",\n \"location\": \"query\"\n },\n \"where\": {\n \"type\": \"object\",\n \"required\": false,\n \"description\": \"An object to limit which objects are returned by their attributes.\\nThe keys of this object are attribute names.\\nEach value is either a single matching value or an array of matching values for that attribute.\\nThe `filters` parameter is more flexible and preferred.\\n\",\n \"location\": \"query\"\n },\n \"order\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"An array of strings to set the order in which matching objects are returned.\\nEach string has the format ` `.\\n`DIRECTION` can be `asc` or omitted for ascending, or `desc` for descending.\\n\",\n \"location\": \"query\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return from each matching object.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"distinct\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If this is true, and multiple objects have the same values\\nfor the attributes that you specify in the `select` parameter, then each unique\\nset of values will only be returned once in the result set.\\n\",\n \"location\": \"query\"\n },\n \"limit\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"100\",\n \"description\": \"The maximum number of objects to return in the result.\\nNote that the API may return fewer results than this if your request hits other\\nlimits set by the administrator.\\n\",\n \"location\": \"query\"\n },\n \"offset\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"0\",\n \"description\": \"Return matching objects starting from this index.\\nNote that result indexes may change if objects are modified in between a series\\nof list calls.\\n\",\n \"location\": \"query\"\n },\n \"count\": {\n \"type\": \"string\",\n \"required\": false,\n \"default\": \"exact\",\n \"description\": \"A string to determine result counting behavior. Supported values are:\\n\\n * `\\\"exact\\\"`: The response will include an `items_available` field that\\n counts the number of objects that matched this search criteria,\\n including ones not included in `items`.\\n\\n * `\\\"none\\\"`: The response will not include an `items_avaliable`\\n field. This improves performance by returning a result as soon as enough\\n `items` have been loaded for this result.\\n\\n\",\n \"location\": \"query\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster to return objects from\",\n \"location\": \"query\",\n \"required\": false\n },\n \"bypass_federation\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If true, do not return results from other clusters in the\\nfederation, only the cluster that received the request.\\nYou must be an administrator to use this flag.\\n\",\n \"location\": \"query\"\n }\n },\n \"response\": {\n \"$ref\": \"CredentialList\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"create\": {\n \"id\": \"arvados.credentials.create\",\n \"path\": \"credentials\",\n \"httpMethod\": \"POST\",\n \"description\": \"Create a new Credential.\",\n \"parameters\": {\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"ensure_unique_name\": {\n \"type\": \"boolean\",\n \"description\": \"If the given name is already used by this owner, adjust the name to ensure uniqueness instead of returning an error.\",\n \"location\": \"query\",\n \"required\": false,\n \"default\": \"false\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster where this object should be created.\",\n \"location\": \"query\",\n \"required\": false\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"credential\": {\n \"$ref\": \"Credential\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"Credential\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"update\": {\n \"id\": \"arvados.credentials.update\",\n \"path\": \"credentials/{uuid}\",\n \"httpMethod\": \"PUT\",\n \"description\": \"Update attributes of an existing Credential.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Credential to update.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"credential\": {\n \"$ref\": \"Credential\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"Credential\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"delete\": {\n \"id\": \"arvados.credentials.delete\",\n \"path\": \"credentials/{uuid}\",\n \"httpMethod\": \"DELETE\",\n \"description\": \"Delete an existing Credential.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Credential to delete.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"Credential\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"secret\": {\n \"id\": \"arvados.credentials.secret\",\n \"path\": \"credentials/{uuid}/secret\",\n \"httpMethod\": \"GET\",\n \"description\": \"Fetch the secret part of the credential (can only be invoked by running containers).\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Credential to query.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"Credential\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n }\n }\n },\n \"groups\": {\n \"methods\": {\n \"get\": {\n \"id\": \"arvados.groups.get\",\n \"path\": \"groups/{uuid}\",\n \"httpMethod\": \"GET\",\n \"description\": \"Get a Group record by UUID.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Group to return.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"include_trash\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"Return group/project even if its `is_trashed` attribute is true.\",\n \"location\": \"query\"\n }\n },\n \"parameterOrder\": [\n \"uuid\"\n ],\n \"response\": {\n \"$ref\": \"Group\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"list\": {\n \"id\": \"arvados.groups.list\",\n \"path\": \"groups\",\n \"httpMethod\": \"GET\",\n \"description\": \"Retrieve a GroupList.\",\n \"parameters\": {\n \"filters\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"Filters to limit which objects are returned by their attributes.\\nRefer to the [filters reference][] for more information about how to write filters.\\n\\n[filters reference]: https://doc.arvados.org/api/methods.html#filters\\n\",\n \"location\": \"query\"\n },\n \"where\": {\n \"type\": \"object\",\n \"required\": false,\n \"description\": \"An object to limit which objects are returned by their attributes.\\nThe keys of this object are attribute names.\\nEach value is either a single matching value or an array of matching values for that attribute.\\nThe `filters` parameter is more flexible and preferred.\\n\",\n \"location\": \"query\"\n },\n \"order\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"An array of strings to set the order in which matching objects are returned.\\nEach string has the format ` `.\\n`DIRECTION` can be `asc` or omitted for ascending, or `desc` for descending.\\n\",\n \"location\": \"query\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return from each matching object.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"distinct\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If this is true, and multiple objects have the same values\\nfor the attributes that you specify in the `select` parameter, then each unique\\nset of values will only be returned once in the result set.\\n\",\n \"location\": \"query\"\n },\n \"limit\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"100\",\n \"description\": \"The maximum number of objects to return in the result.\\nNote that the API may return fewer results than this if your request hits other\\nlimits set by the administrator.\\n\",\n \"location\": \"query\"\n },\n \"offset\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"0\",\n \"description\": \"Return matching objects starting from this index.\\nNote that result indexes may change if objects are modified in between a series\\nof list calls.\\n\",\n \"location\": \"query\"\n },\n \"count\": {\n \"type\": \"string\",\n \"required\": false,\n \"default\": \"exact\",\n \"description\": \"A string to determine result counting behavior. Supported values are:\\n\\n * `\\\"exact\\\"`: The response will include an `items_available` field that\\n counts the number of objects that matched this search criteria,\\n including ones not included in `items`.\\n\\n * `\\\"none\\\"`: The response will not include an `items_avaliable`\\n field. This improves performance by returning a result as soon as enough\\n `items` have been loaded for this result.\\n\\n\",\n \"location\": \"query\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster to return objects from\",\n \"location\": \"query\",\n \"required\": false\n },\n \"bypass_federation\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If true, do not return results from other clusters in the\\nfederation, only the cluster that received the request.\\nYou must be an administrator to use this flag.\\n\",\n \"location\": \"query\"\n },\n \"include_trash\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"Include items whose `is_trashed` attribute is true.\",\n \"location\": \"query\"\n }\n },\n \"response\": {\n \"$ref\": \"GroupList\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"create\": {\n \"id\": \"arvados.groups.create\",\n \"path\": \"groups\",\n \"httpMethod\": \"POST\",\n \"description\": \"Create a new Group.\",\n \"parameters\": {\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"ensure_unique_name\": {\n \"type\": \"boolean\",\n \"description\": \"If the given name is already used by this owner, adjust the name to ensure uniqueness instead of returning an error.\",\n \"location\": \"query\",\n \"required\": false,\n \"default\": \"false\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster where this object should be created.\",\n \"location\": \"query\",\n \"required\": false\n },\n \"async\": {\n \"required\": false,\n \"type\": \"boolean\",\n \"location\": \"query\",\n \"default\": \"false\",\n \"description\": \"If true, cluster permission will not be updated immediately, but instead at the next configured update interval.\"\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"group\": {\n \"$ref\": \"Group\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"Group\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"update\": {\n \"id\": \"arvados.groups.update\",\n \"path\": \"groups/{uuid}\",\n \"httpMethod\": \"PUT\",\n \"description\": \"Update attributes of an existing Group.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Group to update.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"async\": {\n \"required\": false,\n \"type\": \"boolean\",\n \"location\": \"query\",\n \"default\": \"false\",\n \"description\": \"If true, cluster permission will not be updated immediately, but instead at the next configured update interval.\"\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"group\": {\n \"$ref\": \"Group\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"Group\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"delete\": {\n \"id\": \"arvados.groups.delete\",\n \"path\": \"groups/{uuid}\",\n \"httpMethod\": \"DELETE\",\n \"description\": \"Delete an existing Group.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Group to delete.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"Group\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"contents\": {\n \"id\": \"arvados.groups.contents\",\n \"path\": \"groups/contents\",\n \"httpMethod\": \"GET\",\n \"description\": \"List objects that belong to a group.\",\n \"parameters\": {\n \"filters\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"Filters to limit which objects are returned by their attributes.\\nRefer to the [filters reference][] for more information about how to write filters.\\n\\n[filters reference]: https://doc.arvados.org/api/methods.html#filters\\n\",\n \"location\": \"query\"\n },\n \"where\": {\n \"type\": \"object\",\n \"required\": false,\n \"description\": \"An object to limit which objects are returned by their attributes.\\nThe keys of this object are attribute names.\\nEach value is either a single matching value or an array of matching values for that attribute.\\nThe `filters` parameter is more flexible and preferred.\\n\",\n \"location\": \"query\"\n },\n \"order\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"An array of strings to set the order in which matching objects are returned.\\nEach string has the format ` `.\\n`DIRECTION` can be `asc` or omitted for ascending, or `desc` for descending.\\n\",\n \"location\": \"query\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return from each matching object.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"distinct\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If this is true, and multiple objects have the same values\\nfor the attributes that you specify in the `select` parameter, then each unique\\nset of values will only be returned once in the result set.\\n\",\n \"location\": \"query\"\n },\n \"limit\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"100\",\n \"description\": \"The maximum number of objects to return in the result.\\nNote that the API may return fewer results than this if your request hits other\\nlimits set by the administrator.\\n\",\n \"location\": \"query\"\n },\n \"offset\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"0\",\n \"description\": \"Return matching objects starting from this index.\\nNote that result indexes may change if objects are modified in between a series\\nof list calls.\\n\",\n \"location\": \"query\"\n },\n \"count\": {\n \"type\": \"string\",\n \"required\": false,\n \"default\": \"exact\",\n \"description\": \"A string to determine result counting behavior. Supported values are:\\n\\n * `\\\"exact\\\"`: The response will include an `items_available` field that\\n counts the number of objects that matched this search criteria,\\n including ones not included in `items`.\\n\\n * `\\\"none\\\"`: The response will not include an `items_avaliable`\\n field. This improves performance by returning a result as soon as enough\\n `items` have been loaded for this result.\\n\\n\",\n \"location\": \"query\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster to return objects from\",\n \"location\": \"query\",\n \"required\": false\n },\n \"bypass_federation\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If true, do not return results from other clusters in the\\nfederation, only the cluster that received the request.\\nYou must be an administrator to use this flag.\\n\",\n \"location\": \"query\"\n },\n \"include_trash\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"Include items whose `is_trashed` attribute is true.\",\n \"location\": \"query\"\n },\n \"uuid\": {\n \"type\": \"string\",\n \"required\": false,\n \"default\": \"\",\n \"description\": \"If given, limit the listing to objects owned by the\\nuser or group with this UUID.\",\n \"location\": \"query\"\n },\n \"recursive\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If true, include contents from child groups recursively.\",\n \"location\": \"query\"\n },\n \"include\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"An array of referenced objects to include in the `included` field of the response. Supported values in the array are:\\n\\n * `\\\"container_uuid\\\"`\\n * `\\\"owner_uuid\\\"`\\n * `\\\"collection_uuid\\\"`\\n\\n\",\n \"location\": \"query\"\n },\n \"include_old_versions\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If true, include past versions of collections in the listing.\",\n \"location\": \"query\"\n },\n \"exclude_home_project\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If true, exclude contents of the user's home project from the listing.\\nCalling this method with this flag set is how clients enumerate objects shared\\nwith the current user.\",\n \"location\": \"query\"\n }\n },\n \"response\": {\n \"$ref\": \"Group\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"shared\": {\n \"id\": \"arvados.groups.shared\",\n \"path\": \"groups/shared\",\n \"httpMethod\": \"GET\",\n \"description\": \"List groups that the current user can access via permission links.\",\n \"parameters\": {\n \"filters\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"Filters to limit which objects are returned by their attributes.\\nRefer to the [filters reference][] for more information about how to write filters.\\n\\n[filters reference]: https://doc.arvados.org/api/methods.html#filters\\n\",\n \"location\": \"query\"\n },\n \"where\": {\n \"type\": \"object\",\n \"required\": false,\n \"description\": \"An object to limit which objects are returned by their attributes.\\nThe keys of this object are attribute names.\\nEach value is either a single matching value or an array of matching values for that attribute.\\nThe `filters` parameter is more flexible and preferred.\\n\",\n \"location\": \"query\"\n },\n \"order\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"An array of strings to set the order in which matching objects are returned.\\nEach string has the format ` `.\\n`DIRECTION` can be `asc` or omitted for ascending, or `desc` for descending.\\n\",\n \"location\": \"query\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return from each matching object.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"distinct\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If this is true, and multiple objects have the same values\\nfor the attributes that you specify in the `select` parameter, then each unique\\nset of values will only be returned once in the result set.\\n\",\n \"location\": \"query\"\n },\n \"limit\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"100\",\n \"description\": \"The maximum number of objects to return in the result.\\nNote that the API may return fewer results than this if your request hits other\\nlimits set by the administrator.\\n\",\n \"location\": \"query\"\n },\n \"offset\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"0\",\n \"description\": \"Return matching objects starting from this index.\\nNote that result indexes may change if objects are modified in between a series\\nof list calls.\\n\",\n \"location\": \"query\"\n },\n \"count\": {\n \"type\": \"string\",\n \"required\": false,\n \"default\": \"exact\",\n \"description\": \"A string to determine result counting behavior. Supported values are:\\n\\n * `\\\"exact\\\"`: The response will include an `items_available` field that\\n counts the number of objects that matched this search criteria,\\n including ones not included in `items`.\\n\\n * `\\\"none\\\"`: The response will not include an `items_avaliable`\\n field. This improves performance by returning a result as soon as enough\\n `items` have been loaded for this result.\\n\\n\",\n \"location\": \"query\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster to return objects from\",\n \"location\": \"query\",\n \"required\": false\n },\n \"bypass_federation\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If true, do not return results from other clusters in the\\nfederation, only the cluster that received the request.\\nYou must be an administrator to use this flag.\\n\",\n \"location\": \"query\"\n },\n \"include_trash\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"Include items whose `is_trashed` attribute is true.\",\n \"location\": \"query\"\n },\n \"include\": {\n \"type\": \"string\",\n \"required\": false,\n \"description\": \"A string naming referenced objects to include in the `included` field of the response. Supported values are:\\n\\n * `\\\"owner_uuid\\\"`\\n\\n\",\n \"location\": \"query\"\n }\n },\n \"response\": {\n \"$ref\": \"Group\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"trash\": {\n \"id\": \"arvados.groups.trash\",\n \"path\": \"groups/{uuid}/trash\",\n \"httpMethod\": \"POST\",\n \"description\": \"Trash a group.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Group to update.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"Group\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"untrash\": {\n \"id\": \"arvados.groups.untrash\",\n \"path\": \"groups/{uuid}/untrash\",\n \"httpMethod\": \"POST\",\n \"description\": \"Untrash a group.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Group to update.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"Group\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n }\n }\n },\n \"keep_services\": {\n \"methods\": {\n \"get\": {\n \"id\": \"arvados.keep_services.get\",\n \"path\": \"keep_services/{uuid}\",\n \"httpMethod\": \"GET\",\n \"description\": \"Get a KeepService record by UUID.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the KeepService to return.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n }\n },\n \"parameterOrder\": [\n \"uuid\"\n ],\n \"response\": {\n \"$ref\": \"KeepService\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"list\": {\n \"id\": \"arvados.keep_services.list\",\n \"path\": \"keep_services\",\n \"httpMethod\": \"GET\",\n \"description\": \"Retrieve a KeepServiceList.\",\n \"parameters\": {\n \"filters\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"Filters to limit which objects are returned by their attributes.\\nRefer to the [filters reference][] for more information about how to write filters.\\n\\n[filters reference]: https://doc.arvados.org/api/methods.html#filters\\n\",\n \"location\": \"query\"\n },\n \"where\": {\n \"type\": \"object\",\n \"required\": false,\n \"description\": \"An object to limit which objects are returned by their attributes.\\nThe keys of this object are attribute names.\\nEach value is either a single matching value or an array of matching values for that attribute.\\nThe `filters` parameter is more flexible and preferred.\\n\",\n \"location\": \"query\"\n },\n \"order\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"An array of strings to set the order in which matching objects are returned.\\nEach string has the format ` `.\\n`DIRECTION` can be `asc` or omitted for ascending, or `desc` for descending.\\n\",\n \"location\": \"query\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return from each matching object.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"distinct\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If this is true, and multiple objects have the same values\\nfor the attributes that you specify in the `select` parameter, then each unique\\nset of values will only be returned once in the result set.\\n\",\n \"location\": \"query\"\n },\n \"limit\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"100\",\n \"description\": \"The maximum number of objects to return in the result.\\nNote that the API may return fewer results than this if your request hits other\\nlimits set by the administrator.\\n\",\n \"location\": \"query\"\n },\n \"offset\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"0\",\n \"description\": \"Return matching objects starting from this index.\\nNote that result indexes may change if objects are modified in between a series\\nof list calls.\\n\",\n \"location\": \"query\"\n },\n \"count\": {\n \"type\": \"string\",\n \"required\": false,\n \"default\": \"exact\",\n \"description\": \"A string to determine result counting behavior. Supported values are:\\n\\n * `\\\"exact\\\"`: The response will include an `items_available` field that\\n counts the number of objects that matched this search criteria,\\n including ones not included in `items`.\\n\\n * `\\\"none\\\"`: The response will not include an `items_avaliable`\\n field. This improves performance by returning a result as soon as enough\\n `items` have been loaded for this result.\\n\\n\",\n \"location\": \"query\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster to return objects from\",\n \"location\": \"query\",\n \"required\": false\n },\n \"bypass_federation\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If true, do not return results from other clusters in the\\nfederation, only the cluster that received the request.\\nYou must be an administrator to use this flag.\\n\",\n \"location\": \"query\"\n }\n },\n \"response\": {\n \"$ref\": \"KeepServiceList\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"create\": {\n \"id\": \"arvados.keep_services.create\",\n \"path\": \"keep_services\",\n \"httpMethod\": \"POST\",\n \"description\": \"Create a new KeepService.\",\n \"parameters\": {\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"ensure_unique_name\": {\n \"type\": \"boolean\",\n \"description\": \"If the given name is already used by this owner, adjust the name to ensure uniqueness instead of returning an error.\",\n \"location\": \"query\",\n \"required\": false,\n \"default\": \"false\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster where this object should be created.\",\n \"location\": \"query\",\n \"required\": false\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"keep_service\": {\n \"$ref\": \"KeepService\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"KeepService\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"update\": {\n \"id\": \"arvados.keep_services.update\",\n \"path\": \"keep_services/{uuid}\",\n \"httpMethod\": \"PUT\",\n \"description\": \"Update attributes of an existing KeepService.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the KeepService to update.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"keep_service\": {\n \"$ref\": \"KeepService\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"KeepService\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"delete\": {\n \"id\": \"arvados.keep_services.delete\",\n \"path\": \"keep_services/{uuid}\",\n \"httpMethod\": \"DELETE\",\n \"description\": \"Delete an existing KeepService.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the KeepService to delete.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"KeepService\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"accessible\": {\n \"id\": \"arvados.keep_services.accessible\",\n \"path\": \"keep_services/accessible\",\n \"httpMethod\": \"GET\",\n \"description\": \"List Keep services that the current client can access.\",\n \"parameters\": {},\n \"response\": {\n \"$ref\": \"KeepService\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n }\n }\n },\n \"links\": {\n \"methods\": {\n \"get\": {\n \"id\": \"arvados.links.get\",\n \"path\": \"links/{uuid}\",\n \"httpMethod\": \"GET\",\n \"description\": \"Get a Link record by UUID.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Link to return.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n }\n },\n \"parameterOrder\": [\n \"uuid\"\n ],\n \"response\": {\n \"$ref\": \"Link\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"list\": {\n \"id\": \"arvados.links.list\",\n \"path\": \"links\",\n \"httpMethod\": \"GET\",\n \"description\": \"Retrieve a LinkList.\",\n \"parameters\": {\n \"filters\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"Filters to limit which objects are returned by their attributes.\\nRefer to the [filters reference][] for more information about how to write filters.\\n\\n[filters reference]: https://doc.arvados.org/api/methods.html#filters\\n\",\n \"location\": \"query\"\n },\n \"where\": {\n \"type\": \"object\",\n \"required\": false,\n \"description\": \"An object to limit which objects are returned by their attributes.\\nThe keys of this object are attribute names.\\nEach value is either a single matching value or an array of matching values for that attribute.\\nThe `filters` parameter is more flexible and preferred.\\n\",\n \"location\": \"query\"\n },\n \"order\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"An array of strings to set the order in which matching objects are returned.\\nEach string has the format ` `.\\n`DIRECTION` can be `asc` or omitted for ascending, or `desc` for descending.\\n\",\n \"location\": \"query\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return from each matching object.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"distinct\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If this is true, and multiple objects have the same values\\nfor the attributes that you specify in the `select` parameter, then each unique\\nset of values will only be returned once in the result set.\\n\",\n \"location\": \"query\"\n },\n \"limit\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"100\",\n \"description\": \"The maximum number of objects to return in the result.\\nNote that the API may return fewer results than this if your request hits other\\nlimits set by the administrator.\\n\",\n \"location\": \"query\"\n },\n \"offset\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"0\",\n \"description\": \"Return matching objects starting from this index.\\nNote that result indexes may change if objects are modified in between a series\\nof list calls.\\n\",\n \"location\": \"query\"\n },\n \"count\": {\n \"type\": \"string\",\n \"required\": false,\n \"default\": \"exact\",\n \"description\": \"A string to determine result counting behavior. Supported values are:\\n\\n * `\\\"exact\\\"`: The response will include an `items_available` field that\\n counts the number of objects that matched this search criteria,\\n including ones not included in `items`.\\n\\n * `\\\"none\\\"`: The response will not include an `items_avaliable`\\n field. This improves performance by returning a result as soon as enough\\n `items` have been loaded for this result.\\n\\n\",\n \"location\": \"query\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster to return objects from\",\n \"location\": \"query\",\n \"required\": false\n },\n \"bypass_federation\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If true, do not return results from other clusters in the\\nfederation, only the cluster that received the request.\\nYou must be an administrator to use this flag.\\n\",\n \"location\": \"query\"\n }\n },\n \"response\": {\n \"$ref\": \"LinkList\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"create\": {\n \"id\": \"arvados.links.create\",\n \"path\": \"links\",\n \"httpMethod\": \"POST\",\n \"description\": \"Create a new Link.\",\n \"parameters\": {\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"ensure_unique_name\": {\n \"type\": \"boolean\",\n \"description\": \"If the given name is already used by this owner, adjust the name to ensure uniqueness instead of returning an error.\",\n \"location\": \"query\",\n \"required\": false,\n \"default\": \"false\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster where this object should be created.\",\n \"location\": \"query\",\n \"required\": false\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"link\": {\n \"$ref\": \"Link\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"Link\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"update\": {\n \"id\": \"arvados.links.update\",\n \"path\": \"links/{uuid}\",\n \"httpMethod\": \"PUT\",\n \"description\": \"Update attributes of an existing Link.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Link to update.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"link\": {\n \"$ref\": \"Link\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"Link\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"delete\": {\n \"id\": \"arvados.links.delete\",\n \"path\": \"links/{uuid}\",\n \"httpMethod\": \"DELETE\",\n \"description\": \"Delete an existing Link.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Link to delete.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"Link\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"get_permissions\": {\n \"id\": \"arvados.links.get_permissions\",\n \"path\": \"permissions/{uuid}\",\n \"httpMethod\": \"GET\",\n \"description\": \"List permissions granted on an Arvados object.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Link to query.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"Link\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n }\n }\n },\n \"logs\": {\n \"methods\": {\n \"get\": {\n \"id\": \"arvados.logs.get\",\n \"path\": \"logs/{uuid}\",\n \"httpMethod\": \"GET\",\n \"description\": \"Get a Log record by UUID.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Log to return.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n }\n },\n \"parameterOrder\": [\n \"uuid\"\n ],\n \"response\": {\n \"$ref\": \"Log\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"list\": {\n \"id\": \"arvados.logs.list\",\n \"path\": \"logs\",\n \"httpMethod\": \"GET\",\n \"description\": \"Retrieve a LogList.\",\n \"parameters\": {\n \"filters\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"Filters to limit which objects are returned by their attributes.\\nRefer to the [filters reference][] for more information about how to write filters.\\n\\n[filters reference]: https://doc.arvados.org/api/methods.html#filters\\n\",\n \"location\": \"query\"\n },\n \"where\": {\n \"type\": \"object\",\n \"required\": false,\n \"description\": \"An object to limit which objects are returned by their attributes.\\nThe keys of this object are attribute names.\\nEach value is either a single matching value or an array of matching values for that attribute.\\nThe `filters` parameter is more flexible and preferred.\\n\",\n \"location\": \"query\"\n },\n \"order\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"An array of strings to set the order in which matching objects are returned.\\nEach string has the format ` `.\\n`DIRECTION` can be `asc` or omitted for ascending, or `desc` for descending.\\n\",\n \"location\": \"query\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return from each matching object.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"distinct\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If this is true, and multiple objects have the same values\\nfor the attributes that you specify in the `select` parameter, then each unique\\nset of values will only be returned once in the result set.\\n\",\n \"location\": \"query\"\n },\n \"limit\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"100\",\n \"description\": \"The maximum number of objects to return in the result.\\nNote that the API may return fewer results than this if your request hits other\\nlimits set by the administrator.\\n\",\n \"location\": \"query\"\n },\n \"offset\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"0\",\n \"description\": \"Return matching objects starting from this index.\\nNote that result indexes may change if objects are modified in between a series\\nof list calls.\\n\",\n \"location\": \"query\"\n },\n \"count\": {\n \"type\": \"string\",\n \"required\": false,\n \"default\": \"exact\",\n \"description\": \"A string to determine result counting behavior. Supported values are:\\n\\n * `\\\"exact\\\"`: The response will include an `items_available` field that\\n counts the number of objects that matched this search criteria,\\n including ones not included in `items`.\\n\\n * `\\\"none\\\"`: The response will not include an `items_avaliable`\\n field. This improves performance by returning a result as soon as enough\\n `items` have been loaded for this result.\\n\\n\",\n \"location\": \"query\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster to return objects from\",\n \"location\": \"query\",\n \"required\": false\n },\n \"bypass_federation\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If true, do not return results from other clusters in the\\nfederation, only the cluster that received the request.\\nYou must be an administrator to use this flag.\\n\",\n \"location\": \"query\"\n }\n },\n \"response\": {\n \"$ref\": \"LogList\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"create\": {\n \"id\": \"arvados.logs.create\",\n \"path\": \"logs\",\n \"httpMethod\": \"POST\",\n \"description\": \"Create a new Log.\",\n \"parameters\": {\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"ensure_unique_name\": {\n \"type\": \"boolean\",\n \"description\": \"If the given name is already used by this owner, adjust the name to ensure uniqueness instead of returning an error.\",\n \"location\": \"query\",\n \"required\": false,\n \"default\": \"false\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster where this object should be created.\",\n \"location\": \"query\",\n \"required\": false\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"log\": {\n \"$ref\": \"Log\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"Log\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"update\": {\n \"id\": \"arvados.logs.update\",\n \"path\": \"logs/{uuid}\",\n \"httpMethod\": \"PUT\",\n \"description\": \"Update attributes of an existing Log.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Log to update.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"log\": {\n \"$ref\": \"Log\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"Log\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"delete\": {\n \"id\": \"arvados.logs.delete\",\n \"path\": \"logs/{uuid}\",\n \"httpMethod\": \"DELETE\",\n \"description\": \"Delete an existing Log.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Log to delete.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"Log\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n }\n }\n },\n \"users\": {\n \"methods\": {\n \"get\": {\n \"id\": \"arvados.users.get\",\n \"path\": \"users/{uuid}\",\n \"httpMethod\": \"GET\",\n \"description\": \"Get a User record by UUID.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the User to return.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n }\n },\n \"parameterOrder\": [\n \"uuid\"\n ],\n \"response\": {\n \"$ref\": \"User\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"list\": {\n \"id\": \"arvados.users.list\",\n \"path\": \"users\",\n \"httpMethod\": \"GET\",\n \"description\": \"Retrieve a UserList.\",\n \"parameters\": {\n \"filters\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"Filters to limit which objects are returned by their attributes.\\nRefer to the [filters reference][] for more information about how to write filters.\\n\\n[filters reference]: https://doc.arvados.org/api/methods.html#filters\\n\",\n \"location\": \"query\"\n },\n \"where\": {\n \"type\": \"object\",\n \"required\": false,\n \"description\": \"An object to limit which objects are returned by their attributes.\\nThe keys of this object are attribute names.\\nEach value is either a single matching value or an array of matching values for that attribute.\\nThe `filters` parameter is more flexible and preferred.\\n\",\n \"location\": \"query\"\n },\n \"order\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"An array of strings to set the order in which matching objects are returned.\\nEach string has the format ` `.\\n`DIRECTION` can be `asc` or omitted for ascending, or `desc` for descending.\\n\",\n \"location\": \"query\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return from each matching object.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"distinct\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If this is true, and multiple objects have the same values\\nfor the attributes that you specify in the `select` parameter, then each unique\\nset of values will only be returned once in the result set.\\n\",\n \"location\": \"query\"\n },\n \"limit\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"100\",\n \"description\": \"The maximum number of objects to return in the result.\\nNote that the API may return fewer results than this if your request hits other\\nlimits set by the administrator.\\n\",\n \"location\": \"query\"\n },\n \"offset\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"0\",\n \"description\": \"Return matching objects starting from this index.\\nNote that result indexes may change if objects are modified in between a series\\nof list calls.\\n\",\n \"location\": \"query\"\n },\n \"count\": {\n \"type\": \"string\",\n \"required\": false,\n \"default\": \"exact\",\n \"description\": \"A string to determine result counting behavior. Supported values are:\\n\\n * `\\\"exact\\\"`: The response will include an `items_available` field that\\n counts the number of objects that matched this search criteria,\\n including ones not included in `items`.\\n\\n * `\\\"none\\\"`: The response will not include an `items_avaliable`\\n field. This improves performance by returning a result as soon as enough\\n `items` have been loaded for this result.\\n\\n\",\n \"location\": \"query\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster to return objects from\",\n \"location\": \"query\",\n \"required\": false\n },\n \"bypass_federation\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If true, do not return results from other clusters in the\\nfederation, only the cluster that received the request.\\nYou must be an administrator to use this flag.\\n\",\n \"location\": \"query\"\n }\n },\n \"response\": {\n \"$ref\": \"UserList\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"create\": {\n \"id\": \"arvados.users.create\",\n \"path\": \"users\",\n \"httpMethod\": \"POST\",\n \"description\": \"Create a new User.\",\n \"parameters\": {\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"ensure_unique_name\": {\n \"type\": \"boolean\",\n \"description\": \"If the given name is already used by this owner, adjust the name to ensure uniqueness instead of returning an error.\",\n \"location\": \"query\",\n \"required\": false,\n \"default\": \"false\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster where this object should be created.\",\n \"location\": \"query\",\n \"required\": false\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"user\": {\n \"$ref\": \"User\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"User\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"update\": {\n \"id\": \"arvados.users.update\",\n \"path\": \"users/{uuid}\",\n \"httpMethod\": \"PUT\",\n \"description\": \"Update attributes of an existing User.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the User to update.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"bypass_federation\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If true, do not try to update the user on any other clusters in the federation,\\nonly the cluster that received the request.\\nYou must be an administrator to use this flag.\",\n \"location\": \"query\"\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"user\": {\n \"$ref\": \"User\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"User\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"delete\": {\n \"id\": \"arvados.users.delete\",\n \"path\": \"users/{uuid}\",\n \"httpMethod\": \"DELETE\",\n \"description\": \"Delete an existing User.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the User to delete.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"User\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"current\": {\n \"id\": \"arvados.users.current\",\n \"path\": \"users/current\",\n \"httpMethod\": \"GET\",\n \"description\": \"Return the user record associated with the API token authorizing this request.\",\n \"parameters\": {},\n \"response\": {\n \"$ref\": \"User\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"system\": {\n \"id\": \"arvados.users.system\",\n \"path\": \"users/system\",\n \"httpMethod\": \"GET\",\n \"description\": \"Return this cluster's system (\\\"root\\\") user record.\",\n \"parameters\": {},\n \"response\": {\n \"$ref\": \"User\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"activate\": {\n \"id\": \"arvados.users.activate\",\n \"path\": \"users/{uuid}/activate\",\n \"httpMethod\": \"POST\",\n \"description\": \"Set the `is_active` flag on a user record.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the User to update.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"User\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"setup\": {\n \"id\": \"arvados.users.setup\",\n \"path\": \"users/setup\",\n \"httpMethod\": \"POST\",\n \"description\": \"Convenience method to \\\"fully\\\" set up a user record with a virtual machine login and notification email.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"required\": false,\n \"description\": \"UUID of an existing user record to set up.\",\n \"location\": \"query\"\n },\n \"user\": {\n \"type\": \"object\",\n \"required\": false,\n \"description\": \"Attributes of a new user record to set up.\",\n \"location\": \"query\"\n },\n \"repo_name\": {\n \"type\": \"string\",\n \"required\": false,\n \"description\": \"This parameter is obsolete and ignored.\",\n \"location\": \"query\"\n },\n \"vm_uuid\": {\n \"type\": \"string\",\n \"required\": false,\n \"description\": \"If given, setup creates a login link to allow this user to access the Arvados virtual machine with this UUID.\",\n \"location\": \"query\"\n },\n \"send_notification_email\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If true, send an email to the user notifying them they can now access this Arvados cluster.\",\n \"location\": \"query\"\n }\n },\n \"response\": {\n \"$ref\": \"User\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"unsetup\": {\n \"id\": \"arvados.users.unsetup\",\n \"path\": \"users/{uuid}/unsetup\",\n \"httpMethod\": \"POST\",\n \"description\": \"Unset a user's active flag and delete associated records.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the User to update.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"User\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"merge\": {\n \"id\": \"arvados.users.merge\",\n \"path\": \"users/merge\",\n \"httpMethod\": \"POST\",\n \"description\": \"Transfer ownership of one user's data to another.\",\n \"parameters\": {\n \"new_owner_uuid\": {\n \"type\": \"string\",\n \"required\": true,\n \"description\": \"UUID of the user or group that will take ownership of data owned by the old user.\",\n \"location\": \"query\"\n },\n \"new_user_token\": {\n \"type\": \"string\",\n \"required\": false,\n \"description\": \"Valid API token for the user receiving ownership. If you use this option, it takes ownership of data owned by the user making the request.\",\n \"location\": \"query\"\n },\n \"redirect_to_new_user\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If true, authorization attempts for the old user will be redirected to the new user.\",\n \"location\": \"query\"\n },\n \"old_user_uuid\": {\n \"type\": \"string\",\n \"required\": false,\n \"description\": \"UUID of the user whose ownership is being transferred to `new_owner_uuid`. You must be an admin to use this option.\",\n \"location\": \"query\"\n },\n \"new_user_uuid\": {\n \"type\": \"string\",\n \"required\": false,\n \"description\": \"UUID of the user receiving ownership. You must be an admin to use this option.\",\n \"location\": \"query\"\n }\n },\n \"response\": {\n \"$ref\": \"User\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n }\n }\n },\n \"user_agreements\": {\n \"methods\": {\n \"get\": {\n \"id\": \"arvados.user_agreements.get\",\n \"path\": \"user_agreements/{uuid}\",\n \"httpMethod\": \"GET\",\n \"description\": \"Get a UserAgreement record by UUID.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the UserAgreement to return.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n }\n },\n \"parameterOrder\": [\n \"uuid\"\n ],\n \"response\": {\n \"$ref\": \"UserAgreement\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"list\": {\n \"id\": \"arvados.user_agreements.list\",\n \"path\": \"user_agreements\",\n \"httpMethod\": \"GET\",\n \"description\": \"Retrieve a UserAgreementList.\",\n \"parameters\": {\n \"filters\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"Filters to limit which objects are returned by their attributes.\\nRefer to the [filters reference][] for more information about how to write filters.\\n\\n[filters reference]: https://doc.arvados.org/api/methods.html#filters\\n\",\n \"location\": \"query\"\n },\n \"where\": {\n \"type\": \"object\",\n \"required\": false,\n \"description\": \"An object to limit which objects are returned by their attributes.\\nThe keys of this object are attribute names.\\nEach value is either a single matching value or an array of matching values for that attribute.\\nThe `filters` parameter is more flexible and preferred.\\n\",\n \"location\": \"query\"\n },\n \"order\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"An array of strings to set the order in which matching objects are returned.\\nEach string has the format ` `.\\n`DIRECTION` can be `asc` or omitted for ascending, or `desc` for descending.\\n\",\n \"location\": \"query\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return from each matching object.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"distinct\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If this is true, and multiple objects have the same values\\nfor the attributes that you specify in the `select` parameter, then each unique\\nset of values will only be returned once in the result set.\\n\",\n \"location\": \"query\"\n },\n \"limit\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"100\",\n \"description\": \"The maximum number of objects to return in the result.\\nNote that the API may return fewer results than this if your request hits other\\nlimits set by the administrator.\\n\",\n \"location\": \"query\"\n },\n \"offset\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"0\",\n \"description\": \"Return matching objects starting from this index.\\nNote that result indexes may change if objects are modified in between a series\\nof list calls.\\n\",\n \"location\": \"query\"\n },\n \"count\": {\n \"type\": \"string\",\n \"required\": false,\n \"default\": \"exact\",\n \"description\": \"A string to determine result counting behavior. Supported values are:\\n\\n * `\\\"exact\\\"`: The response will include an `items_available` field that\\n counts the number of objects that matched this search criteria,\\n including ones not included in `items`.\\n\\n * `\\\"none\\\"`: The response will not include an `items_avaliable`\\n field. This improves performance by returning a result as soon as enough\\n `items` have been loaded for this result.\\n\\n\",\n \"location\": \"query\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster to return objects from\",\n \"location\": \"query\",\n \"required\": false\n },\n \"bypass_federation\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If true, do not return results from other clusters in the\\nfederation, only the cluster that received the request.\\nYou must be an administrator to use this flag.\\n\",\n \"location\": \"query\"\n }\n },\n \"response\": {\n \"$ref\": \"UserAgreementList\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"create\": {\n \"id\": \"arvados.user_agreements.create\",\n \"path\": \"user_agreements\",\n \"httpMethod\": \"POST\",\n \"description\": \"Create a new UserAgreement.\",\n \"parameters\": {\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"ensure_unique_name\": {\n \"type\": \"boolean\",\n \"description\": \"If the given name is already used by this owner, adjust the name to ensure uniqueness instead of returning an error.\",\n \"location\": \"query\",\n \"required\": false,\n \"default\": \"false\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster where this object should be created.\",\n \"location\": \"query\",\n \"required\": false\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"user_agreement\": {\n \"$ref\": \"UserAgreement\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"UserAgreement\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"update\": {\n \"id\": \"arvados.user_agreements.update\",\n \"path\": \"user_agreements/{uuid}\",\n \"httpMethod\": \"PUT\",\n \"description\": \"Update attributes of an existing UserAgreement.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the UserAgreement to update.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"user_agreement\": {\n \"$ref\": \"UserAgreement\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"UserAgreement\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"delete\": {\n \"id\": \"arvados.user_agreements.delete\",\n \"path\": \"user_agreements/{uuid}\",\n \"httpMethod\": \"DELETE\",\n \"description\": \"Delete an existing UserAgreement.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the UserAgreement to delete.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"UserAgreement\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"signatures\": {\n \"id\": \"arvados.user_agreements.signatures\",\n \"path\": \"user_agreements/signatures\",\n \"httpMethod\": \"GET\",\n \"description\": \"List all user agreement signature links from a user.\",\n \"parameters\": {},\n \"response\": {\n \"$ref\": \"UserAgreement\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"sign\": {\n \"id\": \"arvados.user_agreements.sign\",\n \"path\": \"user_agreements/sign\",\n \"httpMethod\": \"POST\",\n \"description\": \"Create a signature link from the current user for a given user agreement.\",\n \"parameters\": {},\n \"response\": {\n \"$ref\": \"UserAgreement\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n }\n }\n },\n \"virtual_machines\": {\n \"methods\": {\n \"get\": {\n \"id\": \"arvados.virtual_machines.get\",\n \"path\": \"virtual_machines/{uuid}\",\n \"httpMethod\": \"GET\",\n \"description\": \"Get a VirtualMachine record by UUID.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the VirtualMachine to return.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n }\n },\n \"parameterOrder\": [\n \"uuid\"\n ],\n \"response\": {\n \"$ref\": \"VirtualMachine\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"list\": {\n \"id\": \"arvados.virtual_machines.list\",\n \"path\": \"virtual_machines\",\n \"httpMethod\": \"GET\",\n \"description\": \"Retrieve a VirtualMachineList.\",\n \"parameters\": {\n \"filters\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"Filters to limit which objects are returned by their attributes.\\nRefer to the [filters reference][] for more information about how to write filters.\\n\\n[filters reference]: https://doc.arvados.org/api/methods.html#filters\\n\",\n \"location\": \"query\"\n },\n \"where\": {\n \"type\": \"object\",\n \"required\": false,\n \"description\": \"An object to limit which objects are returned by their attributes.\\nThe keys of this object are attribute names.\\nEach value is either a single matching value or an array of matching values for that attribute.\\nThe `filters` parameter is more flexible and preferred.\\n\",\n \"location\": \"query\"\n },\n \"order\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"An array of strings to set the order in which matching objects are returned.\\nEach string has the format ` `.\\n`DIRECTION` can be `asc` or omitted for ascending, or `desc` for descending.\\n\",\n \"location\": \"query\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return from each matching object.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"distinct\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If this is true, and multiple objects have the same values\\nfor the attributes that you specify in the `select` parameter, then each unique\\nset of values will only be returned once in the result set.\\n\",\n \"location\": \"query\"\n },\n \"limit\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"100\",\n \"description\": \"The maximum number of objects to return in the result.\\nNote that the API may return fewer results than this if your request hits other\\nlimits set by the administrator.\\n\",\n \"location\": \"query\"\n },\n \"offset\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"0\",\n \"description\": \"Return matching objects starting from this index.\\nNote that result indexes may change if objects are modified in between a series\\nof list calls.\\n\",\n \"location\": \"query\"\n },\n \"count\": {\n \"type\": \"string\",\n \"required\": false,\n \"default\": \"exact\",\n \"description\": \"A string to determine result counting behavior. Supported values are:\\n\\n * `\\\"exact\\\"`: The response will include an `items_available` field that\\n counts the number of objects that matched this search criteria,\\n including ones not included in `items`.\\n\\n * `\\\"none\\\"`: The response will not include an `items_avaliable`\\n field. This improves performance by returning a result as soon as enough\\n `items` have been loaded for this result.\\n\\n\",\n \"location\": \"query\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster to return objects from\",\n \"location\": \"query\",\n \"required\": false\n },\n \"bypass_federation\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If true, do not return results from other clusters in the\\nfederation, only the cluster that received the request.\\nYou must be an administrator to use this flag.\\n\",\n \"location\": \"query\"\n }\n },\n \"response\": {\n \"$ref\": \"VirtualMachineList\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"create\": {\n \"id\": \"arvados.virtual_machines.create\",\n \"path\": \"virtual_machines\",\n \"httpMethod\": \"POST\",\n \"description\": \"Create a new VirtualMachine.\",\n \"parameters\": {\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"ensure_unique_name\": {\n \"type\": \"boolean\",\n \"description\": \"If the given name is already used by this owner, adjust the name to ensure uniqueness instead of returning an error.\",\n \"location\": \"query\",\n \"required\": false,\n \"default\": \"false\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster where this object should be created.\",\n \"location\": \"query\",\n \"required\": false\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"virtual_machine\": {\n \"$ref\": \"VirtualMachine\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"VirtualMachine\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"update\": {\n \"id\": \"arvados.virtual_machines.update\",\n \"path\": \"virtual_machines/{uuid}\",\n \"httpMethod\": \"PUT\",\n \"description\": \"Update attributes of an existing VirtualMachine.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the VirtualMachine to update.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"virtual_machine\": {\n \"$ref\": \"VirtualMachine\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"VirtualMachine\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"delete\": {\n \"id\": \"arvados.virtual_machines.delete\",\n \"path\": \"virtual_machines/{uuid}\",\n \"httpMethod\": \"DELETE\",\n \"description\": \"Delete an existing VirtualMachine.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the VirtualMachine to delete.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"VirtualMachine\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"logins\": {\n \"id\": \"arvados.virtual_machines.logins\",\n \"path\": \"virtual_machines/{uuid}/logins\",\n \"httpMethod\": \"GET\",\n \"description\": \"List login permission links for a given virtual machine.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the VirtualMachine to query.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"VirtualMachine\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"get_all_logins\": {\n \"id\": \"arvados.virtual_machines.get_all_logins\",\n \"path\": \"virtual_machines/get_all_logins\",\n \"httpMethod\": \"GET\",\n \"description\": \"List login permission links for all virtual machines.\",\n \"parameters\": {},\n \"response\": {\n \"$ref\": \"VirtualMachine\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n }\n }\n },\n \"workflows\": {\n \"methods\": {\n \"get\": {\n \"id\": \"arvados.workflows.get\",\n \"path\": \"workflows/{uuid}\",\n \"httpMethod\": \"GET\",\n \"description\": \"Get a Workflow record by UUID.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Workflow to return.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n }\n },\n \"parameterOrder\": [\n \"uuid\"\n ],\n \"response\": {\n \"$ref\": \"Workflow\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"list\": {\n \"id\": \"arvados.workflows.list\",\n \"path\": \"workflows\",\n \"httpMethod\": \"GET\",\n \"description\": \"Retrieve a WorkflowList.\",\n \"parameters\": {\n \"filters\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"Filters to limit which objects are returned by their attributes.\\nRefer to the [filters reference][] for more information about how to write filters.\\n\\n[filters reference]: https://doc.arvados.org/api/methods.html#filters\\n\",\n \"location\": \"query\"\n },\n \"where\": {\n \"type\": \"object\",\n \"required\": false,\n \"description\": \"An object to limit which objects are returned by their attributes.\\nThe keys of this object are attribute names.\\nEach value is either a single matching value or an array of matching values for that attribute.\\nThe `filters` parameter is more flexible and preferred.\\n\",\n \"location\": \"query\"\n },\n \"order\": {\n \"type\": \"array\",\n \"required\": false,\n \"description\": \"An array of strings to set the order in which matching objects are returned.\\nEach string has the format ` `.\\n`DIRECTION` can be `asc` or omitted for ascending, or `desc` for descending.\\n\",\n \"location\": \"query\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return from each matching object.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"distinct\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If this is true, and multiple objects have the same values\\nfor the attributes that you specify in the `select` parameter, then each unique\\nset of values will only be returned once in the result set.\\n\",\n \"location\": \"query\"\n },\n \"limit\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"100\",\n \"description\": \"The maximum number of objects to return in the result.\\nNote that the API may return fewer results than this if your request hits other\\nlimits set by the administrator.\\n\",\n \"location\": \"query\"\n },\n \"offset\": {\n \"type\": \"integer\",\n \"required\": false,\n \"default\": \"0\",\n \"description\": \"Return matching objects starting from this index.\\nNote that result indexes may change if objects are modified in between a series\\nof list calls.\\n\",\n \"location\": \"query\"\n },\n \"count\": {\n \"type\": \"string\",\n \"required\": false,\n \"default\": \"exact\",\n \"description\": \"A string to determine result counting behavior. Supported values are:\\n\\n * `\\\"exact\\\"`: The response will include an `items_available` field that\\n counts the number of objects that matched this search criteria,\\n including ones not included in `items`.\\n\\n * `\\\"none\\\"`: The response will not include an `items_avaliable`\\n field. This improves performance by returning a result as soon as enough\\n `items` have been loaded for this result.\\n\\n\",\n \"location\": \"query\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster to return objects from\",\n \"location\": \"query\",\n \"required\": false\n },\n \"bypass_federation\": {\n \"type\": \"boolean\",\n \"required\": false,\n \"default\": \"false\",\n \"description\": \"If true, do not return results from other clusters in the\\nfederation, only the cluster that received the request.\\nYou must be an administrator to use this flag.\\n\",\n \"location\": \"query\"\n }\n },\n \"response\": {\n \"$ref\": \"WorkflowList\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n },\n \"create\": {\n \"id\": \"arvados.workflows.create\",\n \"path\": \"workflows\",\n \"httpMethod\": \"POST\",\n \"description\": \"Create a new Workflow.\",\n \"parameters\": {\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n },\n \"ensure_unique_name\": {\n \"type\": \"boolean\",\n \"description\": \"If the given name is already used by this owner, adjust the name to ensure uniqueness instead of returning an error.\",\n \"location\": \"query\",\n \"required\": false,\n \"default\": \"false\"\n },\n \"cluster_id\": {\n \"type\": \"string\",\n \"description\": \"Cluster ID of a federated cluster where this object should be created.\",\n \"location\": \"query\",\n \"required\": false\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"workflow\": {\n \"$ref\": \"Workflow\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"Workflow\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"update\": {\n \"id\": \"arvados.workflows.update\",\n \"path\": \"workflows/{uuid}\",\n \"httpMethod\": \"PUT\",\n \"description\": \"Update attributes of an existing Workflow.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Workflow to update.\",\n \"required\": true,\n \"location\": \"path\"\n },\n \"select\": {\n \"type\": \"array\",\n \"description\": \"An array of names of attributes to return in the response.\",\n \"required\": false,\n \"location\": \"query\"\n }\n },\n \"request\": {\n \"required\": true,\n \"properties\": {\n \"workflow\": {\n \"$ref\": \"Workflow\"\n }\n }\n },\n \"response\": {\n \"$ref\": \"Workflow\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n },\n \"delete\": {\n \"id\": \"arvados.workflows.delete\",\n \"path\": \"workflows/{uuid}\",\n \"httpMethod\": \"DELETE\",\n \"description\": \"Delete an existing Workflow.\",\n \"parameters\": {\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"The UUID of the Workflow to delete.\",\n \"required\": true,\n \"location\": \"path\"\n }\n },\n \"response\": {\n \"$ref\": \"Workflow\"\n },\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\"\n ]\n }\n }\n },\n \"configs\": {\n \"methods\": {\n \"get\": {\n \"id\": \"arvados.configs.get\",\n \"path\": \"config\",\n \"httpMethod\": \"GET\",\n \"description\": \"Get this cluster's public configuration settings.\",\n \"parameters\": {},\n \"parameterOrder\": [],\n \"response\": {},\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n }\n }\n },\n \"vocabularies\": {\n \"methods\": {\n \"get\": {\n \"id\": \"arvados.vocabularies.get\",\n \"path\": \"vocabulary\",\n \"httpMethod\": \"GET\",\n \"description\": \"Get this cluster's configured vocabulary definition.\\n\\nRefer to [metadata vocabulary documentation][] for details.\\n\\n[metadata vocabulary documentation]: https://doc.arvados.org/admin/metadata-vocabulary.html\\n\\n\",\n \"parameters\": {},\n \"parameterOrder\": [],\n \"response\": {},\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n }\n }\n },\n \"sys\": {\n \"methods\": {\n \"get\": {\n \"id\": \"arvados.sys.trash_sweep\",\n \"path\": \"sys/trash_sweep\",\n \"httpMethod\": \"POST\",\n \"description\": \"Run scheduled data trash and sweep operations across this cluster's Keep services.\",\n \"parameters\": {},\n \"parameterOrder\": [],\n \"response\": {},\n \"scopes\": [\n \"https://api.arvados.org/auth/arvados\",\n \"https://api.arvados.org/auth/arvados.readonly\"\n ]\n }\n }\n }\n },\n \"revision\": \"20250402\",\n \"schemas\": {\n \"ApiClientAuthorizationList\": {\n \"id\": \"ApiClientAuthorizationList\",\n \"description\": \"A list of ApiClientAuthorization objects.\",\n \"type\": \"object\",\n \"properties\": {\n \"kind\": {\n \"type\": \"string\",\n \"description\": \"Object type. Always arvados#apiClientAuthorizationList.\",\n \"default\": \"arvados#apiClientAuthorizationList\"\n },\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"List cache version.\"\n },\n \"items\": {\n \"type\": \"array\",\n \"description\": \"An array of matching ApiClientAuthorization objects.\",\n \"items\": {\n \"$ref\": \"ApiClientAuthorization\"\n }\n }\n }\n },\n \"ApiClientAuthorization\": {\n \"id\": \"ApiClientAuthorization\",\n \"description\": \"Arvados API client authorization token\\n\\nThis resource represents an API token a user may use to authenticate an\\nArvados API request.\",\n \"type\": \"object\",\n \"uuidPrefix\": \"gj3su\",\n \"properties\": {\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"Object cache version.\"\n },\n \"api_token\": {\n \"description\": \"The secret token that can be used to authorize Arvados API requests.\",\n \"type\": \"string\"\n },\n \"created_by_ip_address\": {\n \"description\": \"The IP address of the client that created this token.\",\n \"type\": \"string\"\n },\n \"last_used_by_ip_address\": {\n \"description\": \"The IP address of the client that last used this token.\",\n \"type\": \"string\"\n },\n \"last_used_at\": {\n \"description\": \"The last time this token was used to authorize a request. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"expires_at\": {\n \"description\": \"The time after which this token is no longer valid for authorization. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"created_at\": {\n \"description\": \"The time this API client authorization was created. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"scopes\": {\n \"description\": \"An array of strings identifying HTTP methods and API paths this token is\\nauthorized to use. Refer to the [scopes reference][] for details.\\n\\n[scopes reference]: https://doc.arvados.org/api/tokens.html#scopes\\n\\n\",\n \"type\": \"Array\"\n },\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"This API client authorization's Arvados UUID, like `zzzzz-gj3su-12345abcde67890`.\"\n }\n }\n },\n \"AuthorizedKeyList\": {\n \"id\": \"AuthorizedKeyList\",\n \"description\": \"A list of AuthorizedKey objects.\",\n \"type\": \"object\",\n \"properties\": {\n \"kind\": {\n \"type\": \"string\",\n \"description\": \"Object type. Always arvados#authorizedKeyList.\",\n \"default\": \"arvados#authorizedKeyList\"\n },\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"List cache version.\"\n },\n \"items\": {\n \"type\": \"array\",\n \"description\": \"An array of matching AuthorizedKey objects.\",\n \"items\": {\n \"$ref\": \"AuthorizedKey\"\n }\n }\n }\n },\n \"AuthorizedKey\": {\n \"id\": \"AuthorizedKey\",\n \"description\": \"Arvados authorized public key\\n\\nThis resource represents a public key a user may use to authenticate themselves\\nto services on the cluster. Its primary use today is to store SSH keys for\\nvirtual machines (\\\"shell nodes\\\"). It may be extended to store other keys in\\nthe future.\",\n \"type\": \"object\",\n \"uuidPrefix\": \"fngyi\",\n \"properties\": {\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"Object cache version.\"\n },\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"This authorized key's Arvados UUID, like `zzzzz-fngyi-12345abcde67890`.\"\n },\n \"owner_uuid\": {\n \"description\": \"The UUID of the user or group that owns this authorized key.\",\n \"type\": \"string\"\n },\n \"modified_by_user_uuid\": {\n \"description\": \"The UUID of the user that last updated this authorized key.\",\n \"type\": \"string\"\n },\n \"modified_at\": {\n \"description\": \"The time this authorized key was last updated. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"name\": {\n \"description\": \"The name of this authorized key assigned by a user.\",\n \"type\": \"string\"\n },\n \"key_type\": {\n \"description\": \"A string identifying what type of service uses this key. Supported values are:\\n\\n * `\\\"SSH\\\"`\\n\\n\",\n \"type\": \"string\"\n },\n \"authorized_user_uuid\": {\n \"description\": \"The UUID of the Arvados user that is authorized by this key.\",\n \"type\": \"string\"\n },\n \"public_key\": {\n \"description\": \"The full public key, in the format referenced by `key_type`.\",\n \"type\": \"text\"\n },\n \"expires_at\": {\n \"description\": \"The time after which this key is no longer valid for authorization. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"created_at\": {\n \"description\": \"The time this authorized key was created. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n }\n }\n },\n \"CollectionList\": {\n \"id\": \"CollectionList\",\n \"description\": \"A list of Collection objects.\",\n \"type\": \"object\",\n \"properties\": {\n \"kind\": {\n \"type\": \"string\",\n \"description\": \"Object type. Always arvados#collectionList.\",\n \"default\": \"arvados#collectionList\"\n },\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"List cache version.\"\n },\n \"items\": {\n \"type\": \"array\",\n \"description\": \"An array of matching Collection objects.\",\n \"items\": {\n \"$ref\": \"Collection\"\n }\n }\n }\n },\n \"Collection\": {\n \"id\": \"Collection\",\n \"description\": \"Arvados data collection\\n\\nA collection describes how a set of files is stored in data blocks in Keep,\\nalong with associated metadata.\",\n \"type\": \"object\",\n \"uuidPrefix\": \"4zz18\",\n \"properties\": {\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"Object cache version.\"\n },\n \"owner_uuid\": {\n \"description\": \"The UUID of the user or group that owns this collection.\",\n \"type\": \"string\"\n },\n \"created_at\": {\n \"description\": \"The time this collection was created. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"modified_by_user_uuid\": {\n \"description\": \"The UUID of the user that last updated this collection.\",\n \"type\": \"string\"\n },\n \"modified_at\": {\n \"description\": \"The time this collection was last updated. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"portable_data_hash\": {\n \"description\": \"The portable data hash of this collection. This string provides a unique\\nand stable reference to these contents.\",\n \"type\": \"string\"\n },\n \"replication_desired\": {\n \"description\": \"The number of copies that should be made for data in this collection.\",\n \"type\": \"integer\"\n },\n \"replication_confirmed_at\": {\n \"description\": \"The last time the cluster confirmed that it met `replication_confirmed`\\nfor this collection. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"replication_confirmed\": {\n \"description\": \"The number of copies of data in this collection that the cluster has confirmed\\nexist in storage.\",\n \"type\": \"integer\"\n },\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"This collection's Arvados UUID, like `zzzzz-4zz18-12345abcde67890`.\"\n },\n \"manifest_text\": {\n \"description\": \"The manifest text that describes how files are constructed from data blocks\\nin this collection. Refer to the [manifest format][] reference for details.\\n\\n[manifest format]: https://doc.arvados.org/architecture/manifest-format.html\\n\\n\",\n \"type\": \"text\"\n },\n \"name\": {\n \"description\": \"The name of this collection assigned by a user.\",\n \"type\": \"string\"\n },\n \"description\": {\n \"description\": \"A longer HTML description of this collection assigned by a user.\\nAllowed HTML tags are `a`, `b`, `blockquote`, `br`, `code`,\\n`del`, `dd`, `dl`, `dt`, `em`, `h1`, `h2`, `h3`, `h4`, `h5`, `h6`, `hr`,\\n`i`, `img`, `kbd`, `li`, `ol`, `p`, `pre`,\\n`s`, `section`, `span`, `strong`, `sub`, `sup`, and `ul`.\",\n \"type\": \"string\"\n },\n \"properties\": {\n \"description\": \"A hash of arbitrary metadata for this collection.\\nSome keys may be reserved by Arvados or defined by a configured vocabulary.\\nRefer to the [metadata properties reference][] for details.\\n\\n[metadata properties reference]: https://doc.arvados.org/api/properties.html\\n\\n\",\n \"type\": \"Hash\"\n },\n \"delete_at\": {\n \"description\": \"The time this collection will be permanently deleted. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"trash_at\": {\n \"description\": \"The time this collection will be trashed. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"is_trashed\": {\n \"description\": \"A boolean flag to indicate whether or not this collection is trashed.\",\n \"type\": \"boolean\"\n },\n \"storage_classes_desired\": {\n \"description\": \"An array of strings identifying the storage class(es) that should be used\\nfor data in this collection. Storage classes are configured by the cluster administrator.\",\n \"type\": \"Array\"\n },\n \"storage_classes_confirmed\": {\n \"description\": \"An array of strings identifying the storage class(es) the cluster has\\nconfirmed have a copy of this collection's data.\",\n \"type\": \"Array\"\n },\n \"storage_classes_confirmed_at\": {\n \"description\": \"The last time the cluster confirmed that data was stored on the storage\\nclass(es) in `storage_classes_confirmed`. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"current_version_uuid\": {\n \"description\": \"The UUID of the current version of this collection.\",\n \"type\": \"string\"\n },\n \"version\": {\n \"description\": \"An integer that counts which version of a collection this record\\nrepresents. Refer to [collection versioning][] for details. This attribute is\\nread-only.\\n\\n[collection versioning]: https://doc.arvados.org/user/topics/collection-versioning.html\\n\\n\",\n \"type\": \"integer\"\n },\n \"preserve_version\": {\n \"description\": \"A boolean flag to indicate whether this specific version of this collection\\nshould be persisted in cluster storage.\",\n \"type\": \"boolean\"\n },\n \"file_count\": {\n \"description\": \"The number of files represented in this collection's `manifest_text`.\\nThis attribute is read-only.\",\n \"type\": \"integer\"\n },\n \"file_size_total\": {\n \"description\": \"The total size in bytes of files represented in this collection's `manifest_text`.\\nThis attribute is read-only.\",\n \"type\": \"integer\"\n }\n }\n },\n \"ComputedPermissionList\": {\n \"id\": \"ComputedPermissionList\",\n \"description\": \"A list of ComputedPermission objects.\",\n \"type\": \"object\",\n \"properties\": {\n \"kind\": {\n \"type\": \"string\",\n \"description\": \"Object type. Always arvados#computedPermissionList.\",\n \"default\": \"arvados#computedPermissionList\"\n },\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"List cache version.\"\n },\n \"items\": {\n \"type\": \"array\",\n \"description\": \"An array of matching ComputedPermission objects.\",\n \"items\": {\n \"$ref\": \"ComputedPermission\"\n }\n }\n }\n },\n \"ComputedPermission\": {\n \"id\": \"ComputedPermission\",\n \"description\": \"Arvados computed permission\\n\\nComputed permissions do not correspond directly to any Arvados resource, but\\nprovide a simple way to query the entire graph of permissions granted to\\nusers and groups.\",\n \"type\": \"object\",\n \"properties\": {\n \"user_uuid\": {\n \"description\": \"The UUID of the Arvados user who has this permission.\",\n \"type\": \"string\"\n },\n \"target_uuid\": {\n \"description\": \"The UUID of the Arvados object the user has access to.\",\n \"type\": \"string\"\n },\n \"perm_level\": {\n \"description\": \"A string representing the user's level of access to the target object.\\nPossible values are:\\n\\n * `\\\"can_read\\\"`\\n * `\\\"can_write\\\"`\\n * `\\\"can_manage\\\"`\\n\\n\",\n \"type\": \"string\"\n }\n }\n },\n \"ContainerList\": {\n \"id\": \"ContainerList\",\n \"description\": \"A list of Container objects.\",\n \"type\": \"object\",\n \"properties\": {\n \"kind\": {\n \"type\": \"string\",\n \"description\": \"Object type. Always arvados#containerList.\",\n \"default\": \"arvados#containerList\"\n },\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"List cache version.\"\n },\n \"items\": {\n \"type\": \"array\",\n \"description\": \"An array of matching Container objects.\",\n \"items\": {\n \"$ref\": \"Container\"\n }\n }\n }\n },\n \"Container\": {\n \"id\": \"Container\",\n \"description\": \"Arvados container record\\n\\nA container represents compute work that has been or should be dispatched,\\nalong with its results. A container can satisfy one or more container requests.\",\n \"type\": \"object\",\n \"uuidPrefix\": \"dz642\",\n \"properties\": {\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"Object cache version.\"\n },\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"This container's Arvados UUID, like `zzzzz-dz642-12345abcde67890`.\"\n },\n \"owner_uuid\": {\n \"description\": \"The UUID of the user or group that owns this container.\",\n \"type\": \"string\"\n },\n \"created_at\": {\n \"description\": \"The time this container was created. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"modified_at\": {\n \"description\": \"The time this container was last updated. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"modified_by_user_uuid\": {\n \"description\": \"The UUID of the user that last updated this container.\",\n \"type\": \"string\"\n },\n \"state\": {\n \"description\": \"A string representing the container's current execution status. Possible\\nvalues are:\\n\\n * `\\\"Queued\\\"` --- This container has not been dispatched yet.\\n * `\\\"Locked\\\"` --- A dispatcher has claimed this container in preparation to run it.\\n * `\\\"Running\\\"` --- A dispatcher is running this container.\\n * `\\\"Cancelled\\\"` --- Container execution has been cancelled by user request.\\n * `\\\"Complete\\\"` --- A dispatcher ran this container to completion and recorded the results.\\n\\n\",\n \"type\": \"string\"\n },\n \"started_at\": {\n \"description\": \" The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"finished_at\": {\n \"description\": \" The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"log\": {\n \"description\": \"The portable data hash of the Arvados collection that contains this\\ncontainer's logs.\",\n \"type\": \"string\"\n },\n \"environment\": {\n \"description\": \"A hash of string keys and values that defines the environment variables\\nfor the dispatcher to set when it executes this container.\",\n \"type\": \"Hash\"\n },\n \"cwd\": {\n \"description\": \"A string that the defines the working directory that the dispatcher should\\nuse when it executes the command inside this container.\",\n \"type\": \"string\"\n },\n \"command\": {\n \"description\": \"An array of strings that defines the command that the dispatcher should\\nexecute inside this container.\",\n \"type\": \"Array\"\n },\n \"output_path\": {\n \"description\": \"A string that defines the file or directory path where the command\\nwrites output that should be saved from this container.\",\n \"type\": \"string\"\n },\n \"mounts\": {\n \"description\": \"A hash where each key names a directory inside this container, and its\\nvalue is an object that defines the mount source for that directory. Refer\\nto the [mount types reference][] for details.\\n\\n[mount types reference]: https://doc.arvados.org/api/methods/containers.html#mount_types\\n\\n\",\n \"type\": \"Hash\"\n },\n \"runtime_constraints\": {\n \"description\": \"A hash that identifies compute resources this container requires to run\\nsuccessfully. See the [runtime constraints reference][] for details.\\n\\n[runtime constraints reference]: https://doc.arvados.org/api/methods/containers.html#runtime_constraints\\n\\n\",\n \"type\": \"Hash\"\n },\n \"output\": {\n \"description\": \"The portable data hash of the Arvados collection that contains this\\ncontainer's output file(s).\",\n \"type\": \"string\"\n },\n \"container_image\": {\n \"description\": \"The portable data hash of the Arvados collection that contains the image\\nto use for this container.\",\n \"type\": \"string\"\n },\n \"progress\": {\n \"description\": \"A float between 0.0 and 1.0 (inclusive) that represents the container's\\nexecution progress. This attribute is not implemented yet.\",\n \"type\": \"float\"\n },\n \"priority\": {\n \"description\": \"An integer between 0 and 1000 (inclusive) that represents this container's\\nscheduling priority. 0 represents a request to be cancelled. Higher\\nvalues represent higher priority. Refer to the [priority reference][] for details.\\n\\n[priority reference]: https://doc.arvados.org/api/methods/container_requests.html#priority\\n\\n\",\n \"type\": \"integer\"\n },\n \"exit_code\": {\n \"description\": \"An integer that records the Unix exit code of the `command` from a\\nfinished container.\",\n \"type\": \"integer\"\n },\n \"auth_uuid\": {\n \"description\": \"The UUID of the Arvados API client authorization token that a dispatcher\\nshould use to set up this container. This token is automatically created by\\nArvados and this attribute automatically assigned unless a container is\\ncreated with `runtime_token`.\",\n \"type\": \"string\"\n },\n \"locked_by_uuid\": {\n \"description\": \"The UUID of the Arvados API client authorization token that successfully\\nlocked this container in preparation to execute it.\",\n \"type\": \"string\"\n },\n \"scheduling_parameters\": {\n \"description\": \"A hash of scheduling parameters that should be passed to the underlying\\ndispatcher when this container is run.\\nSee the [scheduling parameters reference][] for details.\\n\\n[scheduling parameters reference]: https://doc.arvados.org/api/methods/containers.html#scheduling_parameters\\n\\n\",\n \"type\": \"Hash\"\n },\n \"runtime_status\": {\n \"description\": \"A hash with status updates from a running container.\\nRefer to the [runtime status reference][] for details.\\n\\n[runtime status reference]: https://doc.arvados.org/api/methods/containers.html#runtime_status\\n\\n\",\n \"type\": \"Hash\"\n },\n \"runtime_user_uuid\": {\n \"description\": \"The UUID of the Arvados user associated with the API client authorization\\ntoken used to run this container.\",\n \"type\": \"text\"\n },\n \"runtime_auth_scopes\": {\n \"description\": \"The `scopes` from the API client authorization token used to run this container.\",\n \"type\": \"Array\"\n },\n \"lock_count\": {\n \"description\": \"The number of times this container has been locked by a dispatcher. This\\nmay be greater than 1 if a dispatcher locks a container but then execution is\\ninterrupted for any reason.\",\n \"type\": \"integer\"\n },\n \"gateway_address\": {\n \"description\": \"A string with the address of the Arvados gateway server, in `HOST:PORT`\\nformat. This is for internal use only.\",\n \"type\": \"string\"\n },\n \"interactive_session_started\": {\n \"description\": \"This flag is set true if any user starts an interactive shell inside the\\nrunning container.\",\n \"type\": \"boolean\"\n },\n \"output_storage_classes\": {\n \"description\": \"An array of strings identifying the storage class(es) that should be set\\non the output collection of this container. Storage classes are configured by\\nthe cluster administrator.\",\n \"type\": \"Array\"\n },\n \"output_properties\": {\n \"description\": \"A hash of arbitrary metadata to set on the output collection of this container.\\nSome keys may be reserved by Arvados or defined by a configured vocabulary.\\nRefer to the [metadata properties reference][] for details.\\n\\n[metadata properties reference]: https://doc.arvados.org/api/properties.html\\n\\n\",\n \"type\": \"Hash\"\n },\n \"cost\": {\n \"description\": \"A float with the estimated cost of the cloud instance used to run this\\ncontainer. The value is `0` if cost estimation is not available on this cluster.\",\n \"type\": \"float\"\n },\n \"subrequests_cost\": {\n \"description\": \"A float with the estimated cost of all cloud instances used to run this\\ncontainer and all its subrequests. The value is `0` if cost estimation is not\\navailable on this cluster.\",\n \"type\": \"float\"\n },\n \"output_glob\": {\n \"description\": \"An array of strings of shell-style glob patterns that define which file(s)\\nand subdirectory(ies) under the `output_path` directory should be recorded in\\nthe container's final output. Refer to the [glob patterns reference][] for details.\\n\\n[glob patterns reference]: https://doc.arvados.org/api/methods/containers.html#glob_patterns\\n\\n\",\n \"type\": \"Array\"\n },\n \"service\": {\n \"description\": \"A boolean flag. If set, it informs the system that this is a long-running container\\nthat functions as a system service or web app, rather than a once-through batch operation.\",\n \"type\": \"boolean\"\n },\n \"published_ports\": {\n \"description\": \"A hash where keys are numeric TCP ports on the container which expose HTTP services. Arvados\\nwill proxy HTTP requests to these ports. Values are hashes with the following keys:\\n\\n * `\\\"access\\\"` --- One of 'private' or 'public' indicating if an Arvados API token is required to access the endpoint.\\n * `\\\"label\\\"` --- A human readable label describing the service, for display in Workbench.\\n * `\\\"initial_path\\\"` --- The relative path that should be included when constructing the URL that will be presented to the user in Workbench.\",\n \"type\": \"jsonb\"\n }\n }\n },\n \"ContainerRequestList\": {\n \"id\": \"ContainerRequestList\",\n \"description\": \"A list of ContainerRequest objects.\",\n \"type\": \"object\",\n \"properties\": {\n \"kind\": {\n \"type\": \"string\",\n \"description\": \"Object type. Always arvados#containerRequestList.\",\n \"default\": \"arvados#containerRequestList\"\n },\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"List cache version.\"\n },\n \"items\": {\n \"type\": \"array\",\n \"description\": \"An array of matching ContainerRequest objects.\",\n \"items\": {\n \"$ref\": \"ContainerRequest\"\n }\n }\n }\n },\n \"ContainerRequest\": {\n \"id\": \"ContainerRequest\",\n \"description\": \"Arvados container request\\n\\nA container request represents a user's request that Arvados do some compute\\nwork, along with full details about what work should be done. Arvados will\\nattempt to fulfill the request by mapping it to a matching container record,\\nrunning the work on demand if necessary.\",\n \"type\": \"object\",\n \"uuidPrefix\": \"xvhdp\",\n \"properties\": {\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"Object cache version.\"\n },\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"This container request's Arvados UUID, like `zzzzz-xvhdp-12345abcde67890`.\"\n },\n \"owner_uuid\": {\n \"description\": \"The UUID of the user or group that owns this container request.\",\n \"type\": \"string\"\n },\n \"created_at\": {\n \"description\": \"The time this container request was created. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"modified_at\": {\n \"description\": \"The time this container request was last updated. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"modified_by_user_uuid\": {\n \"description\": \"The UUID of the user that last updated this container request.\",\n \"type\": \"string\"\n },\n \"name\": {\n \"description\": \"The name of this container request assigned by a user.\",\n \"type\": \"string\"\n },\n \"description\": {\n \"description\": \"A longer HTML description of this container request assigned by a user.\\nAllowed HTML tags are `a`, `b`, `blockquote`, `br`, `code`,\\n`del`, `dd`, `dl`, `dt`, `em`, `h1`, `h2`, `h3`, `h4`, `h5`, `h6`, `hr`,\\n`i`, `img`, `kbd`, `li`, `ol`, `p`, `pre`,\\n`s`, `section`, `span`, `strong`, `sub`, `sup`, and `ul`.\",\n \"type\": \"text\"\n },\n \"properties\": {\n \"description\": \"A hash of arbitrary metadata for this container request.\\nSome keys may be reserved by Arvados or defined by a configured vocabulary.\\nRefer to the [metadata properties reference][] for details.\\n\\n[metadata properties reference]: https://doc.arvados.org/api/properties.html\\n\\n\",\n \"type\": \"Hash\"\n },\n \"state\": {\n \"description\": \"A string indicating where this container request is in its lifecycle.\\nPossible values are:\\n\\n * `\\\"Uncommitted\\\"` --- The container request has not been finalized and can still be edited.\\n * `\\\"Committed\\\"` --- The container request is ready to be fulfilled.\\n * `\\\"Final\\\"` --- The container request has been fulfilled or cancelled.\\n\\n\",\n \"type\": \"string\"\n },\n \"requesting_container_uuid\": {\n \"description\": \"The UUID of the container that created this container request, if any.\",\n \"type\": \"string\"\n },\n \"container_uuid\": {\n \"description\": \"The UUID of the container that fulfills this container request, if any.\",\n \"type\": \"string\"\n },\n \"container_count_max\": {\n \"description\": \"An integer that defines the maximum number of times Arvados should attempt\\nto dispatch a container to fulfill this container request.\",\n \"type\": \"integer\"\n },\n \"mounts\": {\n \"description\": \"A hash where each key names a directory inside this container, and its\\nvalue is an object that defines the mount source for that directory. Refer\\nto the [mount types reference][] for details.\\n\\n[mount types reference]: https://doc.arvados.org/api/methods/containers.html#mount_types\\n\\n\",\n \"type\": \"Hash\"\n },\n \"runtime_constraints\": {\n \"description\": \"A hash that identifies compute resources this container requires to run\\nsuccessfully. See the [runtime constraints reference][] for details.\\n\\n[runtime constraints reference]: https://doc.arvados.org/api/methods/containers.html#runtime_constraints\\n\\n\",\n \"type\": \"Hash\"\n },\n \"container_image\": {\n \"description\": \"The portable data hash of the Arvados collection that contains the image\\nto use for this container.\",\n \"type\": \"string\"\n },\n \"environment\": {\n \"description\": \"A hash of string keys and values that defines the environment variables\\nfor the dispatcher to set when it executes this container.\",\n \"type\": \"Hash\"\n },\n \"cwd\": {\n \"description\": \"A string that the defines the working directory that the dispatcher should\\nuse when it executes the command inside this container.\",\n \"type\": \"string\"\n },\n \"command\": {\n \"description\": \"An array of strings that defines the command that the dispatcher should\\nexecute inside this container.\",\n \"type\": \"Array\"\n },\n \"output_path\": {\n \"description\": \"A string that defines the file or directory path where the command\\nwrites output that should be saved from this container.\",\n \"type\": \"string\"\n },\n \"priority\": {\n \"description\": \"An integer between 0 and 1000 (inclusive) that represents this container request's\\nscheduling priority. 0 represents a request to be cancelled. Higher\\nvalues represent higher priority. Refer to the [priority reference][] for details.\\n\\n[priority reference]: https://doc.arvados.org/api/methods/container_requests.html#priority\\n\\n\",\n \"type\": \"integer\"\n },\n \"expires_at\": {\n \"description\": \"The time after which this container request will no longer be fulfilled. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"filters\": {\n \"description\": \"Filters that limit which existing containers are eligible to satisfy this\\ncontainer request. This attribute is not implemented yet and should be null.\",\n \"type\": \"text\"\n },\n \"container_count\": {\n \"description\": \"An integer that records how many times Arvados has attempted to dispatch\\na container to fulfill this container request.\",\n \"type\": \"integer\"\n },\n \"use_existing\": {\n \"description\": \"A boolean flag. If set, Arvados may choose to satisfy this container\\nrequest with an eligible container that already exists. Otherwise, Arvados will\\nsatisfy this container request with a newer container, which will usually result\\nin the container running again.\",\n \"type\": \"boolean\"\n },\n \"scheduling_parameters\": {\n \"description\": \"A hash of scheduling parameters that should be passed to the underlying\\ndispatcher when this container is run.\\nSee the [scheduling parameters reference][] for details.\\n\\n[scheduling parameters reference]: https://doc.arvados.org/api/methods/containers.html#scheduling_parameters\\n\\n\",\n \"type\": \"Hash\"\n },\n \"output_uuid\": {\n \"description\": \"The UUID of the Arvados collection that contains output for all the\\ncontainer(s) that were dispatched to fulfill this container request.\",\n \"type\": \"string\"\n },\n \"log_uuid\": {\n \"description\": \"The UUID of the Arvados collection that contains logs for all the\\ncontainer(s) that were dispatched to fulfill this container request.\",\n \"type\": \"string\"\n },\n \"output_name\": {\n \"description\": \"The name to set on the output collection of this container request.\",\n \"type\": \"string\"\n },\n \"output_ttl\": {\n \"description\": \"An integer in seconds. If greater than zero, when an output collection is\\ncreated for this container request, its `expires_at` attribute will be set this\\nfar in the future.\",\n \"type\": \"integer\"\n },\n \"output_storage_classes\": {\n \"description\": \"An array of strings identifying the storage class(es) that should be set\\non the output collection of this container request. Storage classes are configured by\\nthe cluster administrator.\",\n \"type\": \"Array\"\n },\n \"output_properties\": {\n \"description\": \"A hash of arbitrary metadata to set on the output collection of this container request.\\nSome keys may be reserved by Arvados or defined by a configured vocabulary.\\nRefer to the [metadata properties reference][] for details.\\n\\n[metadata properties reference]: https://doc.arvados.org/api/properties.html\\n\\n\",\n \"type\": \"Hash\"\n },\n \"cumulative_cost\": {\n \"description\": \"A float with the estimated cost of all cloud instances used to run\\ncontainer(s) to fulfill this container request and their subrequests.\\nThe value is `0` if cost estimation is not available on this cluster.\",\n \"type\": \"float\"\n },\n \"output_glob\": {\n \"description\": \"An array of strings of shell-style glob patterns that define which file(s)\\nand subdirectory(ies) under the `output_path` directory should be recorded in\\nthe container's final output. Refer to the [glob patterns reference][] for details.\\n\\n[glob patterns reference]: https://doc.arvados.org/api/methods/containers.html#glob_patterns\\n\\n\",\n \"type\": \"Array\"\n },\n \"service\": {\n \"description\": \"A boolean flag. If set, it informs the system that this request is for a long-running container\\nthat functions as a system service or web app, rather than a once-through batch operation.\",\n \"type\": \"boolean\"\n },\n \"published_ports\": {\n \"description\": \"A hash where keys are numeric TCP ports on the container which expose HTTP services. Arvados\\nwill proxy HTTP requests to these ports. Values are hashes with the following keys:\\n\\n * `\\\"access\\\"` --- One of 'private' or 'public' indicating if an Arvados API token is required to access the endpoint.\\n * `\\\"label\\\"` --- A human readable label describing the service, for display in Workbench.\\n * `\\\"initial_path\\\"` --- The relative path that should be included when constructing the URL that will be presented to the user in Workbench.\",\n \"type\": \"Hash\"\n }\n }\n },\n \"CredentialList\": {\n \"id\": \"CredentialList\",\n \"description\": \"A list of Credential objects.\",\n \"type\": \"object\",\n \"properties\": {\n \"kind\": {\n \"type\": \"string\",\n \"description\": \"Object type. Always arvados#credentialList.\",\n \"default\": \"arvados#credentialList\"\n },\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"List cache version.\"\n },\n \"items\": {\n \"type\": \"array\",\n \"description\": \"An array of matching Credential objects.\",\n \"items\": {\n \"$ref\": \"Credential\"\n }\n }\n }\n },\n \"Credential\": {\n \"id\": \"Credential\",\n \"description\": \"Arvados credential.\",\n \"type\": \"object\",\n \"uuidPrefix\": \"oss07\",\n \"properties\": {\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"Object cache version.\"\n },\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"This credential's Arvados UUID, like `zzzzz-oss07-12345abcde67890`.\"\n },\n \"owner_uuid\": {\n \"description\": \"The UUID of the user or group that owns this credential.\",\n \"type\": \"string\"\n },\n \"created_at\": {\n \"description\": \"The time this credential was created. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"modified_at\": {\n \"description\": \"The time this credential was last updated. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"modified_by_user_uuid\": {\n \"description\": \"The UUID of the user that last updated this credential.\",\n \"type\": \"string\"\n },\n \"name\": {\n \"description\": \"The name of this credential assigned by a user.\",\n \"type\": \"string\"\n },\n \"description\": {\n \"description\": \"A longer HTML description of this credential assigned by a user.\\nAllowed HTML tags are `a`, `b`, `blockquote`, `br`, `code`,\\n`del`, `dd`, `dl`, `dt`, `em`, `h1`, `h2`, `h3`, `h4`, `h5`, `h6`, `hr`,\\n`i`, `img`, `kbd`, `li`, `ol`, `p`, `pre`,\\n`s`, `section`, `span`, `strong`, `sub`, `sup`, and `ul`.\",\n \"type\": \"text\"\n },\n \"credential_class\": {\n \"description\": \"The type of credential being stored.\",\n \"type\": \"string\"\n },\n \"scopes\": {\n \"description\": \"The resources the credential applies to or should be used with.\",\n \"type\": \"Array\"\n },\n \"external_id\": {\n \"description\": \"The non-secret external identifier associated with a credential, e.g. a username.\",\n \"type\": \"string\"\n },\n \"expires_at\": {\n \"description\": \"Date after which the credential_secret field is no longer valid. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n }\n }\n },\n \"GroupList\": {\n \"id\": \"GroupList\",\n \"description\": \"A list of Group objects.\",\n \"type\": \"object\",\n \"properties\": {\n \"kind\": {\n \"type\": \"string\",\n \"description\": \"Object type. Always arvados#groupList.\",\n \"default\": \"arvados#groupList\"\n },\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"List cache version.\"\n },\n \"items\": {\n \"type\": \"array\",\n \"description\": \"An array of matching Group objects.\",\n \"items\": {\n \"$ref\": \"Group\"\n }\n }\n }\n },\n \"Group\": {\n \"id\": \"Group\",\n \"description\": \"Arvados group\\n\\nGroups provide a way to organize users or data together, depending on their\\n`group_class`.\",\n \"type\": \"object\",\n \"uuidPrefix\": \"j7d0g\",\n \"properties\": {\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"Object cache version.\"\n },\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"This group's Arvados UUID, like `zzzzz-j7d0g-12345abcde67890`.\"\n },\n \"owner_uuid\": {\n \"description\": \"The UUID of the user or group that owns this group.\",\n \"type\": \"string\"\n },\n \"created_at\": {\n \"description\": \"The time this group was created. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"modified_by_user_uuid\": {\n \"description\": \"The UUID of the user that last updated this group.\",\n \"type\": \"string\"\n },\n \"modified_at\": {\n \"description\": \"The time this group was last updated. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"name\": {\n \"description\": \"The name of this group assigned by a user.\",\n \"type\": \"string\"\n },\n \"description\": {\n \"description\": \"A longer HTML description of this group assigned by a user.\\nAllowed HTML tags are `a`, `b`, `blockquote`, `br`, `code`,\\n`del`, `dd`, `dl`, `dt`, `em`, `h1`, `h2`, `h3`, `h4`, `h5`, `h6`, `hr`,\\n`i`, `img`, `kbd`, `li`, `ol`, `p`, `pre`,\\n`s`, `section`, `span`, `strong`, `sub`, `sup`, and `ul`.\",\n \"type\": \"string\"\n },\n \"group_class\": {\n \"description\": \"A string representing which type of group this is. One of:\\n\\n * `\\\"filter\\\"` --- A virtual project whose contents are selected dynamically by filters.\\n * `\\\"project\\\"` --- An Arvados project that can contain collections,\\n container records, workflows, and subprojects.\\n * `\\\"role\\\"` --- A group of users that can be granted permissions in Arvados.\\n\\n\",\n \"type\": \"string\"\n },\n \"trash_at\": {\n \"description\": \"The time this group will be trashed. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"is_trashed\": {\n \"description\": \"A boolean flag to indicate whether or not this group is trashed.\",\n \"type\": \"boolean\"\n },\n \"delete_at\": {\n \"description\": \"The time this group will be permanently deleted. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"properties\": {\n \"description\": \"A hash of arbitrary metadata for this group.\\nSome keys may be reserved by Arvados or defined by a configured vocabulary.\\nRefer to the [metadata properties reference][] for details.\\n\\n[metadata properties reference]: https://doc.arvados.org/api/properties.html\\n\\n\",\n \"type\": \"Hash\"\n },\n \"frozen_by_uuid\": {\n \"description\": \"The UUID of the user that has frozen this group, if any. Frozen projects\\ncannot have their contents or metadata changed, even by admins.\",\n \"type\": \"string\"\n }\n }\n },\n \"KeepServiceList\": {\n \"id\": \"KeepServiceList\",\n \"description\": \"A list of KeepService objects.\",\n \"type\": \"object\",\n \"properties\": {\n \"kind\": {\n \"type\": \"string\",\n \"description\": \"Object type. Always arvados#keepServiceList.\",\n \"default\": \"arvados#keepServiceList\"\n },\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"List cache version.\"\n },\n \"items\": {\n \"type\": \"array\",\n \"description\": \"An array of matching KeepService objects.\",\n \"items\": {\n \"$ref\": \"KeepService\"\n }\n }\n }\n },\n \"KeepService\": {\n \"id\": \"KeepService\",\n \"description\": \"Arvados Keep service\\n\\nThis resource stores information about a single Keep service in this Arvados\\ncluster that clients can contact to retrieve and store data.\",\n \"type\": \"object\",\n \"uuidPrefix\": \"bi6l4\",\n \"properties\": {\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"Object cache version.\"\n },\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"This Keep service's Arvados UUID, like `zzzzz-bi6l4-12345abcde67890`.\"\n },\n \"owner_uuid\": {\n \"description\": \"The UUID of the user or group that owns this Keep service.\",\n \"type\": \"string\"\n },\n \"modified_by_user_uuid\": {\n \"description\": \"The UUID of the user that last updated this Keep service.\",\n \"type\": \"string\"\n },\n \"modified_at\": {\n \"description\": \"The time this Keep service was last updated. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"service_host\": {\n \"description\": \"The DNS hostname of this Keep service.\",\n \"type\": \"string\"\n },\n \"service_port\": {\n \"description\": \"The TCP port where this Keep service listens.\",\n \"type\": \"integer\"\n },\n \"service_ssl_flag\": {\n \"description\": \"A boolean flag that indicates whether or not this Keep service uses TLS/SSL.\",\n \"type\": \"boolean\"\n },\n \"service_type\": {\n \"description\": \"A string that describes which type of Keep service this is. One of:\\n\\n * `\\\"disk\\\"` --- A service that stores blocks on a local filesystem.\\n * `\\\"blob\\\"` --- A service that stores blocks in a cloud object store.\\n * `\\\"proxy\\\"` --- A keepproxy service.\\n\\n\",\n \"type\": \"string\"\n },\n \"created_at\": {\n \"description\": \"The time this Keep service was created. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"read_only\": {\n \"description\": \"A boolean flag. If set, this Keep service does not accept requests to write data\\nblocks; it only serves blocks it already has.\",\n \"type\": \"boolean\"\n }\n }\n },\n \"LinkList\": {\n \"id\": \"LinkList\",\n \"description\": \"A list of Link objects.\",\n \"type\": \"object\",\n \"properties\": {\n \"kind\": {\n \"type\": \"string\",\n \"description\": \"Object type. Always arvados#linkList.\",\n \"default\": \"arvados#linkList\"\n },\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"List cache version.\"\n },\n \"items\": {\n \"type\": \"array\",\n \"description\": \"An array of matching Link objects.\",\n \"items\": {\n \"$ref\": \"Link\"\n }\n }\n }\n },\n \"Link\": {\n \"id\": \"Link\",\n \"description\": \"Arvados object link\\n\\nA link provides a way to define relationships between Arvados objects,\\ndepending on their `link_class`.\",\n \"type\": \"object\",\n \"uuidPrefix\": \"o0j2j\",\n \"properties\": {\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"Object cache version.\"\n },\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"This link's Arvados UUID, like `zzzzz-o0j2j-12345abcde67890`.\"\n },\n \"owner_uuid\": {\n \"description\": \"The UUID of the user or group that owns this link.\",\n \"type\": \"string\"\n },\n \"created_at\": {\n \"description\": \"The time this link was created. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"modified_by_user_uuid\": {\n \"description\": \"The UUID of the user that last updated this link.\",\n \"type\": \"string\"\n },\n \"modified_at\": {\n \"description\": \"The time this link was last updated. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"tail_uuid\": {\n \"description\": \"The UUID of the Arvados object that is the target of this relationship.\",\n \"type\": \"string\"\n },\n \"link_class\": {\n \"description\": \"A string that defines which kind of link this is. One of:\\n\\n * `\\\"permission\\\"` --- This link grants a permission to the user or group\\n referenced by `head_uuid` to the object referenced by `tail_uuid`. The\\n access level is set by `name`.\\n * `\\\"star\\\"` --- This link represents a \\\"favorite.\\\" The user referenced\\n by `head_uuid` wants quick access to the object referenced by `tail_uuid`.\\n * `\\\"tag\\\"` --- This link represents an unstructured metadata tag. The object\\n referenced by `tail_uuid` has the tag defined by `name`.\\n\\n\",\n \"type\": \"string\"\n },\n \"name\": {\n \"description\": \"The primary value of this link. For `\\\"permission\\\"` links, this is one of\\n`\\\"can_read\\\"`, `\\\"can_write\\\"`, or `\\\"can_manage\\\"`.\",\n \"type\": \"string\"\n },\n \"head_uuid\": {\n \"description\": \"The UUID of the Arvados object that is the originator or actor in this\\nrelationship. May be null.\",\n \"type\": \"string\"\n },\n \"properties\": {\n \"description\": \"A hash of arbitrary metadata for this link.\\nSome keys may be reserved by Arvados or defined by a configured vocabulary.\\nRefer to the [metadata properties reference][] for details.\\n\\n[metadata properties reference]: https://doc.arvados.org/api/properties.html\\n\\n\",\n \"type\": \"Hash\"\n }\n }\n },\n \"LogList\": {\n \"id\": \"LogList\",\n \"description\": \"A list of Log objects.\",\n \"type\": \"object\",\n \"properties\": {\n \"kind\": {\n \"type\": \"string\",\n \"description\": \"Object type. Always arvados#logList.\",\n \"default\": \"arvados#logList\"\n },\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"List cache version.\"\n },\n \"items\": {\n \"type\": \"array\",\n \"description\": \"An array of matching Log objects.\",\n \"items\": {\n \"$ref\": \"Log\"\n }\n }\n }\n },\n \"Log\": {\n \"id\": \"Log\",\n \"description\": \"Arvados log record\\n\\nThis resource represents a single log record about an event in this Arvados\\ncluster. Some individual Arvados services create log records. Users can also\\ncreate custom logs.\",\n \"type\": \"object\",\n \"uuidPrefix\": \"57u5n\",\n \"properties\": {\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"Object cache version.\"\n },\n \"id\": {\n \"description\": \"The serial number of this log. You can use this in filters to query logs\\nthat were created before/after another.\",\n \"type\": \"integer\"\n },\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"This log's Arvados UUID, like `zzzzz-57u5n-12345abcde67890`.\"\n },\n \"owner_uuid\": {\n \"description\": \"The UUID of the user or group that owns this log.\",\n \"type\": \"string\"\n },\n \"modified_by_user_uuid\": {\n \"description\": \"The UUID of the user that last updated this log.\",\n \"type\": \"string\"\n },\n \"object_uuid\": {\n \"description\": \"The UUID of the Arvados object that this log pertains to, such as a user\\nor container.\",\n \"type\": \"string\"\n },\n \"event_at\": {\n \"description\": \" The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"event_type\": {\n \"description\": \"An arbitrary short string that classifies what type of log this is.\",\n \"type\": \"string\"\n },\n \"summary\": {\n \"description\": \"A text string that describes the logged event. This is the primary\\nattribute for simple logs.\",\n \"type\": \"text\"\n },\n \"properties\": {\n \"description\": \"A hash of arbitrary metadata for this log.\\nSome keys may be reserved by Arvados or defined by a configured vocabulary.\\nRefer to the [metadata properties reference][] for details.\\n\\n[metadata properties reference]: https://doc.arvados.org/api/properties.html\\n\\n\",\n \"type\": \"Hash\"\n },\n \"created_at\": {\n \"description\": \"The time this log was created. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"modified_at\": {\n \"description\": \"The time this log was last updated. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"object_owner_uuid\": {\n \"description\": \"The `owner_uuid` of the object referenced by `object_uuid` at the time\\nthis log was created.\",\n \"type\": \"string\"\n }\n }\n },\n \"UserList\": {\n \"id\": \"UserList\",\n \"description\": \"A list of User objects.\",\n \"type\": \"object\",\n \"properties\": {\n \"kind\": {\n \"type\": \"string\",\n \"description\": \"Object type. Always arvados#userList.\",\n \"default\": \"arvados#userList\"\n },\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"List cache version.\"\n },\n \"items\": {\n \"type\": \"array\",\n \"description\": \"An array of matching User objects.\",\n \"items\": {\n \"$ref\": \"User\"\n }\n }\n }\n },\n \"User\": {\n \"id\": \"User\",\n \"description\": \"Arvados user\\n\\nA user represents a single individual or role who may be authorized to access\\nthis Arvados cluster.\",\n \"type\": \"object\",\n \"uuidPrefix\": \"tpzed\",\n \"properties\": {\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"Object cache version.\"\n },\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"This user's Arvados UUID, like `zzzzz-tpzed-12345abcde67890`.\"\n },\n \"owner_uuid\": {\n \"description\": \"The UUID of the user or group that owns this user.\",\n \"type\": \"string\"\n },\n \"created_at\": {\n \"description\": \"The time this user was created. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"modified_by_user_uuid\": {\n \"description\": \"The UUID of the user that last updated this user.\",\n \"type\": \"string\"\n },\n \"modified_at\": {\n \"description\": \"The time this user was last updated. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"email\": {\n \"description\": \"This user's email address.\",\n \"type\": \"string\"\n },\n \"first_name\": {\n \"description\": \"This user's first name.\",\n \"type\": \"string\"\n },\n \"last_name\": {\n \"description\": \"This user's last name.\",\n \"type\": \"string\"\n },\n \"identity_url\": {\n \"description\": \"A URL that represents this user with the cluster's identity provider.\",\n \"type\": \"string\"\n },\n \"is_admin\": {\n \"description\": \"A boolean flag. If set, this user is an administrator of the Arvados\\ncluster, and automatically passes most permissions checks.\",\n \"type\": \"boolean\"\n },\n \"prefs\": {\n \"description\": \"A hash that stores cluster-wide user preferences.\",\n \"type\": \"Hash\"\n },\n \"is_active\": {\n \"description\": \"A boolean flag. If unset, this user is not permitted to make any Arvados\\nAPI requests.\",\n \"type\": \"boolean\"\n },\n \"username\": {\n \"description\": \"This user's Unix username on virtual machines.\",\n \"type\": \"string\"\n }\n }\n },\n \"UserAgreementList\": {\n \"id\": \"UserAgreementList\",\n \"description\": \"A list of UserAgreement objects.\",\n \"type\": \"object\",\n \"properties\": {\n \"kind\": {\n \"type\": \"string\",\n \"description\": \"Object type. Always arvados#userAgreementList.\",\n \"default\": \"arvados#userAgreementList\"\n },\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"List cache version.\"\n },\n \"items\": {\n \"type\": \"array\",\n \"description\": \"An array of matching UserAgreement objects.\",\n \"items\": {\n \"$ref\": \"UserAgreement\"\n }\n }\n }\n },\n \"UserAgreement\": {\n \"id\": \"UserAgreement\",\n \"description\": \"Arvados user agreement\\n\\nA user agreement is a collection with terms that users must agree to before\\nthey can use this Arvados cluster.\",\n \"type\": \"object\",\n \"uuidPrefix\": \"gv0sa\",\n \"properties\": {\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"Object cache version.\"\n },\n \"owner_uuid\": {\n \"description\": \"The UUID of the user or group that owns this user agreement.\",\n \"type\": \"string\"\n },\n \"created_at\": {\n \"description\": \"The time this user agreement was created. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"modified_by_user_uuid\": {\n \"description\": \"The UUID of the user that last updated this user agreement.\",\n \"type\": \"string\"\n },\n \"modified_at\": {\n \"description\": \"The time this user agreement was last updated. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"portable_data_hash\": {\n \"description\": \"The portable data hash of this user agreement. This string provides a unique\\nand stable reference to these contents.\",\n \"type\": \"string\"\n },\n \"replication_desired\": {\n \"description\": \"The number of copies that should be made for data in this user agreement.\",\n \"type\": \"integer\"\n },\n \"replication_confirmed_at\": {\n \"description\": \"The last time the cluster confirmed that it met `replication_confirmed`\\nfor this user agreement. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"replication_confirmed\": {\n \"description\": \"The number of copies of data in this user agreement that the cluster has confirmed\\nexist in storage.\",\n \"type\": \"integer\"\n },\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"This user agreement's Arvados UUID, like `zzzzz-gv0sa-12345abcde67890`.\"\n },\n \"manifest_text\": {\n \"description\": \"The manifest text that describes how files are constructed from data blocks\\nin this user agreement. Refer to the [manifest format][] reference for details.\\n\\n[manifest format]: https://doc.arvados.org/architecture/manifest-format.html\\n\\n\",\n \"type\": \"text\"\n },\n \"name\": {\n \"description\": \"The name of this user agreement assigned by a user.\",\n \"type\": \"string\"\n },\n \"description\": {\n \"description\": \"A longer HTML description of this user agreement assigned by a user.\\nAllowed HTML tags are `a`, `b`, `blockquote`, `br`, `code`,\\n`del`, `dd`, `dl`, `dt`, `em`, `h1`, `h2`, `h3`, `h4`, `h5`, `h6`, `hr`,\\n`i`, `img`, `kbd`, `li`, `ol`, `p`, `pre`,\\n`s`, `section`, `span`, `strong`, `sub`, `sup`, and `ul`.\",\n \"type\": \"string\"\n },\n \"properties\": {\n \"description\": \"A hash of arbitrary metadata for this user agreement.\\nSome keys may be reserved by Arvados or defined by a configured vocabulary.\\nRefer to the [metadata properties reference][] for details.\\n\\n[metadata properties reference]: https://doc.arvados.org/api/properties.html\\n\\n\",\n \"type\": \"Hash\"\n },\n \"delete_at\": {\n \"description\": \"The time this user agreement will be permanently deleted. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"trash_at\": {\n \"description\": \"The time this user agreement will be trashed. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"is_trashed\": {\n \"description\": \"A boolean flag to indicate whether or not this user agreement is trashed.\",\n \"type\": \"boolean\"\n },\n \"storage_classes_desired\": {\n \"description\": \"An array of strings identifying the storage class(es) that should be used\\nfor data in this user agreement. Storage classes are configured by the cluster administrator.\",\n \"type\": \"Array\"\n },\n \"storage_classes_confirmed\": {\n \"description\": \"An array of strings identifying the storage class(es) the cluster has\\nconfirmed have a copy of this user agreement's data.\",\n \"type\": \"Array\"\n },\n \"storage_classes_confirmed_at\": {\n \"description\": \"The last time the cluster confirmed that data was stored on the storage\\nclass(es) in `storage_classes_confirmed`. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"current_version_uuid\": {\n \"description\": \"The UUID of the current version of this user agreement.\",\n \"type\": \"string\"\n },\n \"version\": {\n \"description\": \"An integer that counts which version of a user agreement this record\\nrepresents. Refer to [collection versioning][] for details. This attribute is\\nread-only.\\n\\n[collection versioning]: https://doc.arvados.org/user/topics/collection-versioning.html\\n\\n\",\n \"type\": \"integer\"\n },\n \"preserve_version\": {\n \"description\": \"A boolean flag to indicate whether this specific version of this user agreement\\nshould be persisted in cluster storage.\",\n \"type\": \"boolean\"\n },\n \"file_count\": {\n \"description\": \"The number of files represented in this user agreement's `manifest_text`.\\nThis attribute is read-only.\",\n \"type\": \"integer\"\n },\n \"file_size_total\": {\n \"description\": \"The total size in bytes of files represented in this user agreement's `manifest_text`.\\nThis attribute is read-only.\",\n \"type\": \"integer\"\n }\n }\n },\n \"VirtualMachineList\": {\n \"id\": \"VirtualMachineList\",\n \"description\": \"A list of VirtualMachine objects.\",\n \"type\": \"object\",\n \"properties\": {\n \"kind\": {\n \"type\": \"string\",\n \"description\": \"Object type. Always arvados#virtualMachineList.\",\n \"default\": \"arvados#virtualMachineList\"\n },\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"List cache version.\"\n },\n \"items\": {\n \"type\": \"array\",\n \"description\": \"An array of matching VirtualMachine objects.\",\n \"items\": {\n \"$ref\": \"VirtualMachine\"\n }\n }\n }\n },\n \"VirtualMachine\": {\n \"id\": \"VirtualMachine\",\n \"description\": \"Arvados virtual machine (\\\"shell node\\\")\\n\\nThis resource stores information about a virtual machine or \\\"shell node\\\"\\nhosted on this Arvados cluster where users can log in and use preconfigured\\nArvados client tools.\",\n \"type\": \"object\",\n \"uuidPrefix\": \"2x53u\",\n \"properties\": {\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"Object cache version.\"\n },\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"This virtual machine's Arvados UUID, like `zzzzz-2x53u-12345abcde67890`.\"\n },\n \"owner_uuid\": {\n \"description\": \"The UUID of the user or group that owns this virtual machine.\",\n \"type\": \"string\"\n },\n \"modified_by_user_uuid\": {\n \"description\": \"The UUID of the user that last updated this virtual machine.\",\n \"type\": \"string\"\n },\n \"modified_at\": {\n \"description\": \"The time this virtual machine was last updated. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"hostname\": {\n \"description\": \"The DNS hostname where users should access this virtual machine.\",\n \"type\": \"string\"\n },\n \"created_at\": {\n \"description\": \"The time this virtual machine was created. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n }\n }\n },\n \"WorkflowList\": {\n \"id\": \"WorkflowList\",\n \"description\": \"A list of Workflow objects.\",\n \"type\": \"object\",\n \"properties\": {\n \"kind\": {\n \"type\": \"string\",\n \"description\": \"Object type. Always arvados#workflowList.\",\n \"default\": \"arvados#workflowList\"\n },\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"List cache version.\"\n },\n \"items\": {\n \"type\": \"array\",\n \"description\": \"An array of matching Workflow objects.\",\n \"items\": {\n \"$ref\": \"Workflow\"\n }\n }\n }\n },\n \"Workflow\": {\n \"id\": \"Workflow\",\n \"description\": \"Arvados workflow\\n\\nA workflow contains workflow definition source code that Arvados can execute\\nalong with associated metadata for users.\",\n \"type\": \"object\",\n \"uuidPrefix\": \"7fd4e\",\n \"properties\": {\n \"etag\": {\n \"type\": \"string\",\n \"description\": \"Object cache version.\"\n },\n \"uuid\": {\n \"type\": \"string\",\n \"description\": \"This workflow's Arvados UUID, like `zzzzz-7fd4e-12345abcde67890`.\"\n },\n \"owner_uuid\": {\n \"description\": \"The UUID of the user or group that owns this workflow.\",\n \"type\": \"string\"\n },\n \"created_at\": {\n \"description\": \"The time this workflow was created. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"modified_at\": {\n \"description\": \"The time this workflow was last updated. The string encodes a UTC date and time in ISO 8601 format.\",\n \"type\": \"datetime\"\n },\n \"modified_by_user_uuid\": {\n \"description\": \"The UUID of the user that last updated this workflow.\",\n \"type\": \"string\"\n },\n \"name\": {\n \"description\": \"The name of this workflow assigned by a user.\",\n \"type\": \"string\"\n },\n \"description\": {\n \"description\": \"A longer HTML description of this workflow assigned by a user.\\nAllowed HTML tags are `a`, `b`, `blockquote`, `br`, `code`,\\n`del`, `dd`, `dl`, `dt`, `em`, `h1`, `h2`, `h3`, `h4`, `h5`, `h6`, `hr`,\\n`i`, `img`, `kbd`, `li`, `ol`, `p`, `pre`,\\n`s`, `section`, `span`, `strong`, `sub`, `sup`, and `ul`.\",\n \"type\": \"text\"\n },\n \"definition\": {\n \"description\": \"A string with the CWL source of this workflow.\",\n \"type\": \"text\"\n },\n \"collection_uuid\": {\n \"description\": \"The collection this workflow is linked to, containing the definition of the workflow.\",\n \"type\": \"string\"\n }\n }\n }\n },\n \"servicePath\": \"arvados/v1/\",\n \"title\": \"Arvados API\",\n \"version\": \"v1\"\n}" }, { "path": "contrib/R-sdk/generateApi.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\nlibrary(jsonlite)\n\ngetAPIDocument <- function(loc)\n{\n if (length(grep(\"^[a-z]+://\", loc)) > 0) {\n library(httr)\n serverResponse <- httr::RETRY(\"GET\", url = loc)\n httr::content(serverResponse, as = \"parsed\", type = \"application/json\")\n } else {\n jsonlite::read_json(loc)\n }\n}\n\n#' generateAPI\n#'\n#' Autogenerate classes to interact with Arvados from the Arvados discovery document.\n#'\n#' @export\ngenerateAPI <- function(discoveryDocument)\n{\n methodResources <- discoveryDocument$resources\n resourceNames <- names(methodResources)\n\n classDoc <- genAPIClassDoc(methodResources, resourceNames)\n arvadosAPIHeader <- genAPIClassHeader()\n arvadosClassMethods <- genClassContent(methodResources, resourceNames)\n arvadosProjectMethods <- genProjectMethods(methodResources)\n arvadosAPIFooter <- genAPIClassFooter()\n\n arvadosClass <- c(classDoc,\n arvadosAPIHeader,\n arvadosClassMethods,\n arvadosProjectMethods,\n arvadosAPIFooter)\n\n fileConn <- file(\"./R/Arvados.R\", \"w\")\n writeLines(c(\n \"# Copyright (C) The Arvados Authors. All rights reserved.\",\n \"#\",\n \"# SPDX-License-Identifier: Apache-2.0\",\n \"\",\n \"#' Arvados\",\n \"#'\",\n \"#' This class implements a full REST client to the Arvados API.\",\n \"#'\"), fileConn)\n writeLines(unlist(arvadosClass), fileConn)\n close(fileConn)\n NULL\n}\n\ngenAPIClassHeader <- function()\n{\n c(\"#' @export\",\n \"Arvados <- R6::R6Class(\",\n \"\",\n \"\\t\\\"Arvados\\\",\",\n \"\",\n \"\\tpublic = list(\",\n \"\",\n \"\\t\\t#' @description Create a new Arvados API client.\",\n \"\\t\\t#' @param authToken Authentification token. If not specified ARVADOS_API_TOKEN environment variable will be used.\",\n \"\\t\\t#' @param hostName Host name. If not specified ARVADOS_API_HOST environment variable will be used.\",\n \"\\t\\t#' @param numRetries Number which specifies how many times to retry failed service requests.\",\n \"\\t\\t#' @return A new `Arvados` object.\",\n \"\\t\\tinitialize = function(authToken = NULL, hostName = NULL, numRetries = 0)\",\n \"\\t\\t{\",\n \"\\t\\t\\tif(!is.null(hostName))\",\n \"\\t\\t\\t\\tSys.setenv(ARVADOS_API_HOST = hostName)\",\n \"\",\n \"\\t\\t\\tif(!is.null(authToken))\",\n \"\\t\\t\\t\\tSys.setenv(ARVADOS_API_TOKEN = authToken)\",\n \"\",\n \"\\t\\t\\thostName <- Sys.getenv(\\\"ARVADOS_API_HOST\\\")\",\n \"\\t\\t\\ttoken <- Sys.getenv(\\\"ARVADOS_API_TOKEN\\\")\",\n \"\",\n \"\\t\\t\\tif(hostName == \\\"\\\" | token == \\\"\\\")\",\n \"\\t\\t\\t\\tstop(paste(\\\"Please provide host name and authentification token\\\",\",\n \"\\t\\t\\t\\t\\t\\t \\\"or set ARVADOS_API_HOST and ARVADOS_API_TOKEN\\\",\",\n \"\\t\\t\\t\\t\\t\\t \\\"environment variables.\\\"))\",\n \"\",\n \"\\t\\t\\tprivate$token <- token\",\n \"\\t\\t\\tprivate$host <- paste0(\\\"https://\\\", hostName, \\\"/arvados/v1/\\\")\",\n \"\\t\\t\\tprivate$numRetries <- numRetries\",\n \"\\t\\t\\tprivate$REST <- RESTService$new(token, hostName,\",\n \"\\t\\t\\t HttpRequest$new(), HttpParser$new(),\",\n \"\\t\\t\\t numRetries)\",\n \"\",\n \"\\t\\t},\\n\")\n}\n\ngenProjectMethods <- function(methodResources)\n{\n toCallArg <- function(arg) {\n callArg <- strsplit(arg, \" *=\")[[1]][1]\n paste(callArg, callArg, sep=\" = \")\n }\n toCallArgs <- function(argList) {\n paste0(Map(toCallArg, argList), collapse=\", \")\n }\n groupsMethods <- methodResources[[\"groups\"]][[\"methods\"]]\n getArgs <- getMethodArguments(groupsMethods[[\"get\"]])\n createArgs <- getMethodArguments(groupsMethods[[\"create\"]])\n updateArgs <- getMethodArguments(groupsMethods[[\"update\"]])\n listArgs <- getMethodArguments(groupsMethods[[\"list\"]])\n deleteArgs <- getMethodArguments(groupsMethods[[\"delete\"]])\n\n c(\"\\t\\t#' @description An alias for `groups_get`.\",\n getMethodParams(groupsMethods[[\"get\"]]),\n \"\\t\\t#' @return A Group object.\",\n getMethodSignature(\"project_get\", getArgs),\n \"\\t\\t{\",\n paste(\"\\t\\t\\tself$groups_get(\", toCallArgs(getArgs), \")\", sep=\"\"),\n \"\\t\\t},\",\n \"\",\n \"\\t\\t#' @description A wrapper for `groups_create` that sets `group_class=\\\"project\\\"`.\",\n getMethodParams(groupsMethods[[\"create\"]]),\n \"\\t\\t#' @return A Group object.\",\n getMethodSignature(\"project_create\", createArgs),\n \"\\t\\t{\",\n \"\\t\\t\\tgroup <- c(\\\"group_class\\\" = \\\"project\\\", group)\",\n paste(\"\\t\\t\\tself$groups_create(\", toCallArgs(createArgs), \")\", sep=\"\"),\n \"\\t\\t},\",\n \"\",\n \"\\t\\t#' @description A wrapper for `groups_update` that sets `group_class=\\\"project\\\"`.\",\n getMethodParams(groupsMethods[[\"update\"]]),\n \"\\t\\t#' @return A Group object.\",\n getMethodSignature(\"project_update\", updateArgs),\n \"\\t\\t{\",\n \"\\t\\t\\tgroup <- c(\\\"group_class\\\" = \\\"project\\\", group)\",\n paste(\"\\t\\t\\tself$groups_update(\", toCallArgs(updateArgs), \")\", sep=\"\"),\n \"\\t\\t},\",\n \"\",\n \"\\t\\t#' @description A wrapper for `groups_list` that adds a filter for `group_class=\\\"project\\\"`.\",\n getMethodParams(groupsMethods[[\"list\"]]),\n \"\\t\\t#' @return A GroupList object.\",\n getMethodSignature(\"project_list\", listArgs),\n \"\\t\\t{\",\n \"\\t\\t\\tfilters[[length(filters) + 1]] <- list(\\\"group_class\\\", \\\"=\\\", \\\"project\\\")\",\n paste(\"\\t\\t\\tself$groups_list(\", toCallArgs(listArgs), \")\", sep=\"\"),\n \"\\t\\t},\",\n \"\",\n \"\\t\\t#' @description An alias for `groups_delete`.\",\n getMethodParams(groupsMethods[[\"delete\"]]),\n \"\\t\\t#' @return A Group object.\",\n getMethodSignature(\"project_delete\", deleteArgs),\n \"\\t\\t{\",\n paste(\"\\t\\t\\tself$groups_delete(\", toCallArgs(deleteArgs), \")\", sep=\"\"),\n \"\\t\\t},\",\n \"\",\n \"\\t\\t#' @description Test whether or not a project exists.\",\n getMethodParams(groupsMethods[[\"get\"]]),\n getMethodSignature(\"project_exist\", getArgs),\n \"\\t\\t{\",\n paste(\"\\t\\t\\tresult <- try(self$groups_get(\", toCallArgs(getArgs), \"))\", sep=\"\"),\n \"\\t\\t\\tif(inherits(result, \\\"try-error\\\"))\",\n \"\\t\\t\\t\\texists <- FALSE\",\n \"\\t\\t\\telse\",\n \"\\t\\t\\t\\texists <- result['group_class'] == \\\"project\\\"\",\n \"\\t\\t\\tcat(format(exists))\",\n \"\\t\\t},\",\n \"\",\n \"\\t\\t#' @description A convenience wrapper for `project_update` to set project metadata properties.\",\n \"\\t\\t#' @param listProperties List of new properties.\",\n \"\\t\\t#' @param uuid UUID of the project to update.\",\n \"\\t\\t#' @return A Group object.\",\n \"\\t\\tproject_properties_set = function(listProperties, uuid)\",\n \"\\t\\t{\",\n \"\\t\\t\\tself$project_update(list(\\\"properties\\\" = listProperties), uuid)\",\n \"\\t\\t},\",\n \"\",\n \"\\t\\t#' @description Get a project and update it with additional properties.\",\n \"\\t\\t#' @param properties List of new properties.\",\n \"\\t\\t#' @param uuid UUID of the project to update.\",\n \"\\t\\t#' @return A Group object.\",\n \"\\t\\tproject_properties_append = function(properties, uuid)\",\n \"\\t\\t{\",\n \"\\t\\t\\tproj <- private$get_project_by_list(uuid, list('uuid', 'properties'))\",\n \"\\t\\t\\tnewListOfProperties <- c(proj$properties, properties)\",\n \"\\t\\t\\tuniqueProperties <- unique(unlist(newListOfProperties))\",\n \"\\t\\t\\tnewProperties <- suppressWarnings(newListOfProperties[which(newListOfProperties == uniqueProperties)])\",\n \"\\t\\t\\tself$project_properties_set(newProperties, proj$uuid)\",\n \"\\t\\t},\",\n \"\",\n \"\\t\\t#' @description Get properties of a project.\",\n \"\\t\\t#' @param uuid The UUID of the project to query.\",\n \"\\t\\tproject_properties_get = function(uuid)\",\n \"\\t\\t{\",\n \"\\t\\t\\tprivate$get_project_by_list(uuid, list('uuid', 'properties'))$properties\",\n \"\\t\\t},\",\n \"\",\n \"\\t\\t#' @description Delete one property from a project by name.\",\n \"\\t\\t#' @param oneProp Name of the property to delete.\",\n \"\\t\\t#' @param uuid The UUID of the project to update.\",\n \"\\t\\t#' @return A Group object.\",\n \"\\t\\tproject_properties_delete = function(oneProp, uuid)\",\n \"\\t\\t{\",\n \"\\t\\t\\tprojProp <- self$project_properties_get(uuid)\",\n \"\\t\\t\\tprojProp[[oneProp]] <- NULL\",\n \"\\t\\t\\tself$project_properties_set(projProp, uuid)\",\n \"\\t\\t},\",\n \"\",\n \"\\t\\t#' @description Convenience wrapper of `links_list` to create a permission link.\",\n \"\\t\\t#' @param type The type of permission: one of `'can_read'`, `'can_write'`, or `'can_manage'`.\",\n \"\\t\\t#' @param uuid The UUID of the object to grant permission to.\",\n \"\\t\\t#' @param user The UUID of the user or group who receives this permission.\",\n \"\\t\\t#' @return A Link object if one was updated, else NULL.\",\n \"\\t\\tproject_permission_give = function(type, uuid, user)\",\n \"\\t\\t{\",\n \"\\t\\t\\tlink <- list(\",\n \"\\t\\t\\t\\t'link_class' = 'permission',\",\n \"\\t\\t\\t\\t'name' = type,\",\n \"\\t\\t\\t\\t'head_uuid' = uuid,\",\n \"\\t\\t\\t\\t'tail_uuid' = user)\",\n \"\\t\\t\\tself$links_create(link)\",\n \"\\t\\t},\",\n \"\",\n \"\\t\\t#' @description Find an existing permission link and update its level.\",\n \"\\t\\t#' @param typeOld The type of permission to find: one of `'can_read'`, `'can_write'`, or `'can_manage'`.\",\n \"\\t\\t#' @param typeNew The type of permission to set: one of `'can_read'`, `'can_write'`, or `'can_manage'`.\",\n \"\\t\\t#' @param uuid The UUID of the object to grant permission to.\",\n \"\\t\\t#' @param user The UUID of the user or group who receives this permission.\",\n \"\\t\\t#' @return A Link object if one was updated, else NULL.\",\n \"\\t\\tproject_permission_update = function(typeOld, typeNew, uuid, user)\",\n \"\\t\\t{\",\n \"\\t\\t\\tlinks <- self$links_list(filters = list(\",\n \"\\t\\t\\t\\t\\tlist('link_class', '=', 'permission'),\",\n \"\\t\\t\\t\\t\\tlist('name', '=', typeOld),\",\n \"\\t\\t\\t\\t\\tlist('head_uuid', '=', uuid),\",\n \"\\t\\t\\t\\t\\tlist('tail_uuid', '=', user)\",\n \"\\t\\t\\t\\t), select=list('uuid'), count = 'none')$items\",\n \"\\t\\t\\tif (length(links) == 0) {\",\n \"\\t\\t\\t\\tcat(format('No permission granted'))\",\n \"\\t\\t\\t} else {\",\n \"\\t\\t\\t\\tself$links_update(list('name' = typeNew), links[[1]]$uuid)\",\n \"\\t\\t\\t}\",\n \"\\t\\t},\",\n \"\",\n \"\\t\\t#' @description Delete an existing permission link.\",\n \"\\t\\t#' @param type The type of permission to delete: one of `'can_read'`, `'can_write'`, or `'can_manage'`.\",\n \"\\t\\t#' @param uuid The UUID of the object to grant permission to.\",\n \"\\t\\t#' @param user The UUID of the user or group who receives this permission.\",\n \"\\t\\t#' @return A Link object if one was deleted, else NULL.\",\n \"\\t\\tproject_permission_delete = function(type, uuid, user)\",\n \"\\t\\t{\",\n \"\\t\\t\\tlinks <- self$links_list(filters = list(\",\n \"\\t\\t\\t\\t\\tlist('link_class', '=', 'permission'),\",\n \"\\t\\t\\t\\t\\tlist('name', '=', type),\",\n \"\\t\\t\\t\\t\\tlist('head_uuid', '=', uuid),\",\n \"\\t\\t\\t\\t\\tlist('tail_uuid', '=', user)\",\n \"\\t\\t\\t\\t), select=list('uuid'), count = 'none')$items\",\n \"\\t\\t\\tif (length(links) == 0) {\",\n \"\\t\\t\\t\\tcat(format('No permission granted'))\",\n \"\\t\\t\\t} else {\",\n \"\\t\\t\\t\\tself$links_delete(links[[1]]$uuid)\",\n \"\\t\\t\\t}\",\n \"\\t\\t},\",\n \"\",\n \"\\t\\t#' @description Check for an existing permission link.\",\n \"\\t\\t#' @param type The type of permission to check: one of `'can_read'`, `'can_write'`, `'can_manage'`, or `NULL` (the default).\",\n \"\\t\\t#' @param uuid The UUID of the object to check permission on.\",\n \"\\t\\t#' @param user The UUID of the user or group to check permission for.\",\n \"\\t\\t#' @return If `type` is `NULL`, the list of matching permission links.\",\n \"\\t\\t#' Otherwise, prints and invisibly returns the level of the found permission link.\",\n \"\\t\\tproject_permission_check = function(uuid, user, type = NULL)\",\n \"\\t\\t{\",\n \"\\t\\t\\tfilters <- list(\",\n \"\\t\\t\\t\\tlist('link_class', '=', 'permission'),\",\n \"\\t\\t\\t\\tlist('head_uuid', '=', uuid),\",\n \"\\t\\t\\t\\tlist('tail_uuid', '=', user))\",\n \"\\t\\t\\tif (!is.null(type)) {\",\n \"\\t\\t\\t\\tfilters <- c(filters, list(list('name', '=', type)))\",\n \"\\t\\t\\t}\",\n \"\\t\\t\\tlinks <- self$links_list(filters = filters, count='none')$items\",\n \"\\t\\t\\tif (is.null(type)) {\",\n \"\\t\\t\\t\\tlinks\",\n \"\\t\\t\\t} else {\",\n \"\\t\\t\\t\\tprint(links[[1]]$name)\",\n \"\\t\\t\\t}\",\n \"\\t\\t},\",\n \"\")\n}\n\ngenClassContent <- function(methodResources, resourceNames)\n{\n arvadosMethods <- Map(function(resource, resourceName)\n {\n methodNames <- names(resource$methods)\n\n functions <- Map(function(methodMetaData, methodName)\n {\n #NOTE: Index, show and destroy are aliases for the preferred names\n # \"list\", \"get\" and \"delete\". Until they are removed from discovery\n # document we will filter them here.\n if(methodName %in% c(\"index\", \"show\", \"destroy\"))\n return(NULL)\n\n methodName <- paste0(resourceName, \"_\", methodName)\n unlist(c(\n getMethodDoc(methodName, methodMetaData),\n createMethod(methodName, methodMetaData)\n ))\n\n }, resource$methods, methodNames)\n\n unlist(unname(functions))\n\n }, methodResources, resourceNames)\n\n arvadosMethods\n}\n\ngenAPIClassFooter <- function()\n{\n c(\"\\t\\t#' @description Return the host name of this client's Arvados API server.\",\n \"\\t\\t#' @return Hostname string.\",\n \"\\t\\tgetHostName = function() private$host,\",\n \"\",\n \"\\t\\t#' @description Return the Arvados API token used by this client.\",\n \"\\t\\t#' @return API token string.\",\n \"\\t\\tgetToken = function() private$token,\",\n \"\",\n \"\\t\\t#' @description Set the RESTService object used by this client.\",\n \"\\t\\tsetRESTService = function(newREST) private$REST <- newREST,\",\n \"\",\n \"\\t\\t#' @description Return the RESTService object used by this client.\",\n \"\\t\\t#' @return RESTService object.\",\n \"\\t\\tgetRESTService = function() private$REST\",\n \"\\t),\",\n \"\",\n \"\\tprivate = list(\",\n \"\\t\\ttoken = NULL,\",\n \"\\t\\thost = NULL,\",\n \"\\t\\tREST = NULL,\",\n \"\\t\\tnumRetries = NULL,\",\n \"\\t\\tget_project_by_list = function(uuid, select = NULL)\",\n \"\\t\\t{\",\n \"\\t\\t\\tself$groups_list(\",\n \"\\t\\t\\t\\tfilters = list(list('uuid', '=', uuid), list('group_class', '=', 'project')),\",\n \"\\t\\t\\t\\tselect = select,\",\n \"\\t\\t\\t\\tcount = 'none'\",\n \"\\t\\t\\t)$items[[1]]\",\n \"\\t\\t}\",\n \"\\t),\",\n \"\",\n \"\\tcloneable = FALSE\",\n \")\")\n}\n\ncreateMethod <- function(name, methodMetaData)\n{\n args <- getMethodArguments(methodMetaData)\n signature <- getMethodSignature(name, args)\n body <- getMethodBody(methodMetaData)\n\n c(signature,\n \"\\t\\t{\",\n body,\n \"\\t\\t},\\n\")\n}\n\nnormalizeParamName <- function(name)\n{\n # Downcase the first letter\n name <- sub(\"^(\\\\w)\", \"\\\\L\\\\1\", name, perl=TRUE)\n # Convert snake_case to camelCase\n gsub(\"_(uuid\\\\b|id\\\\b|\\\\w)\", \"\\\\U\\\\1\", name, perl=TRUE)\n}\n\ngetMethodArguments <- function(methodMetaData)\n{\n request <- methodMetaData$request\n requestArgs <- NULL\n\n if(!is.null(request))\n {\n resourceName <- normalizeParamName(request$properties[[1]][[1]])\n\n if(request$required)\n requestArgs <- resourceName\n else\n requestArgs <- paste(resourceName, \"=\", \"NULL\")\n }\n\n argNames <- names(methodMetaData$parameters)\n\n args <- sapply(argNames, function(argName)\n {\n arg <- methodMetaData$parameters[[argName]]\n argName <- normalizeParamName(argName)\n\n if(!arg$required)\n {\n return(paste(argName, \"=\", \"NULL\"))\n }\n\n argName\n })\n\n c(requestArgs, args)\n}\n\ngetMethodSignature <- function(methodName, args)\n{\n collapsedArgs <- paste0(args, collapse = \", \")\n lineLengthLimit <- 40\n\n if(nchar(collapsedArgs) > lineLengthLimit)\n {\n return(paste0(\"\\t\\t\",\n formatArgs(paste(methodName, \"= function(\"),\n \"\\t\", args, \")\", lineLengthLimit)))\n }\n else\n {\n return(paste0(\"\\t\\t\", methodName, \" = function(\", collapsedArgs, \")\"))\n }\n}\n\ngetMethodBody <- function(methodMetaData)\n{\n url <- getRequestURL(methodMetaData)\n headers <- getRequestHeaders()\n requestQueryList <- getRequestQueryList(methodMetaData)\n requestBody <- getRequestBody(methodMetaData)\n request <- getRequest(methodMetaData)\n response <- getResponse(methodMetaData)\n errorCheck <- getErrorCheckingCode(methodMetaData)\n returnStatement <- getReturnObject()\n\n body <- c(url,\n headers,\n requestQueryList, \"\",\n requestBody, \"\",\n request, response, \"\",\n errorCheck, \"\",\n returnStatement)\n\n paste0(\"\\t\\t\\t\", body)\n}\n\ngetRequestURL <- function(methodMetaData)\n{\n endPoint <- methodMetaData$path\n endPoint <- stringr::str_replace_all(endPoint, \"\\\\{\", \"${\")\n url <- c(paste0(\"endPoint <- stringr::str_interp(\\\"\", endPoint, \"\\\")\"),\n paste0(\"url <- paste0(private$host, endPoint)\"))\n url\n}\n\ngetRequestHeaders <- function()\n{\n c(\"headers <- list(Authorization = paste(\\\"Bearer\\\", private$token), \",\n \" \\\"Content-Type\\\" = \\\"application/json\\\")\")\n}\n\ngetRequestQueryList <- function(methodMetaData)\n{\n queryArgs <- names(Filter(function(arg) arg$location == \"query\",\n methodMetaData$parameters))\n\n if(length(queryArgs) == 0)\n return(\"queryArgs <- NULL\")\n\n queryArgs <- sapply(queryArgs, function(arg) {\n arg <- normalizeParamName(arg)\n paste(arg, \"=\", arg)\n })\n collapsedArgs <- paste0(queryArgs, collapse = \", \")\n\n lineLengthLimit <- 40\n\n if(nchar(collapsedArgs) > lineLengthLimit)\n return(formatArgs(\"queryArgs <- list(\", \"\\t\\t\\t\\t \", queryArgs, \")\",\n lineLengthLimit))\n else\n return(paste0(\"queryArgs <- list(\", collapsedArgs, \")\"))\n}\n\ngetRequestBody <- function(methodMetaData)\n{\n request <- methodMetaData$request\n\n if(is.null(request) || !request$required)\n return(\"body <- NULL\")\n\n resourceName <- normalizeParamName(request$properties[[1]][[1]])\n\n requestParameterName <- names(request$properties)[1]\n\n c(paste0(\"if(length(\", resourceName, \") > 0)\"),\n paste0(\"\\tbody <- jsonlite::toJSON(list(\", resourceName, \" = \", resourceName, \"), \"),\n \"\\t auto_unbox = TRUE)\",\n \"else\",\n \"\\tbody <- NULL\")\n}\n\ngetRequest <- function(methodMetaData)\n{\n method <- methodMetaData$httpMethod\n c(paste0(\"response <- private$REST$http$exec(\\\"\", method, \"\\\", url, headers, body,\"),\n \" queryArgs, private$numRetries)\")\n}\n\ngetResponse <- function(methodMetaData)\n{\n \"resource <- private$REST$httpParser$parseJSONResponse(response)\"\n}\n\ngetErrorCheckingCode <- function(methodMetaData)\n{\n if (\"ensure_unique_name\" %in% names(methodMetaData$parameters)) {\n body <- c(\"\\tif (identical(sub('Entity:.*', '', resource$errors), '//railsapi.internal/arvados/v1/collections: 422 Unprocessable ')) {\",\n \"\\t\\tresource <- cat(format('An object with the given name already exists with this owner. If you want to update it use the update method instead'))\",\n \"\\t} else {\",\n \"\\t\\tstop(resource$errors)\",\n \"\\t}\")\n } else {\n body <- \"\\tstop(resource$errors)\"\n }\n c(\"if(!is.null(resource$errors)) {\", body, \"}\")\n}\n\ngetReturnObject <- function()\n{\n \"resource\"\n}\n\ngenAPIClassDoc <- function(methodResources, resourceNames)\n{\n c(\"#' @examples\",\n \"#' \\\\dontrun{\",\n \"#' arv <- Arvados$new(\\\"your Arvados token\\\", \\\"example.arvadosapi.com\\\")\",\n \"#'\",\n \"#' collection <- arv$collections.get(\\\"uuid\\\")\",\n \"#'\",\n \"#' collectionList <- arv$collections.list(list(list(\\\"name\\\", \\\"like\\\", \\\"Test%\\\")))\",\n \"#' collectionList <- listAll(arv$collections.list, list(list(\\\"name\\\", \\\"like\\\", \\\"Test%\\\")))\",\n \"#'\",\n \"#' deletedCollection <- arv$collections.delete(\\\"uuid\\\")\",\n \"#'\",\n \"#' updatedCollection <- arv$collections.update(list(name = \\\"New name\\\", description = \\\"New description\\\"),\",\n \"#' \\\"uuid\\\")\",\n \"#'\",\n \"#' createdCollection <- arv$collections.create(list(name = \\\"Example\\\",\",\n \"#' description = \\\"This is a test collection\\\"))\",\n \"#' }\",\n \"\")\n}\n\ngetAPIClassMethodList <- function(methodResources, resourceNames)\n{\n methodList <- unlist(unname(Map(function(resource, resourceName)\n {\n methodNames <- names(resource$methods)\n paste0(resourceName,\n \".\",\n methodNames[!(methodNames %in% c(\"index\", \"show\", \"destroy\"))])\n\n }, methodResources, resourceNames)))\n\n hardcodedMethods <- c(\"projects.create\", \"projects.get\",\n \"projects.list\", \"projects.update\", \"projects.delete\")\n paste0(\"#' \\t\\\\item{}{\\\\code{\\\\link{\", sort(c(methodList, hardcodedMethods)), \"}}}\")\n}\n\ngetMethodDoc <- function(methodName, methodMetaData)\n{\n description <- paste(\"\\t\\t#' @description\", gsub(\"\\n\", \"\\n\\t\\t#' \", methodMetaData$description))\n params <- getMethodParams(methodMetaData)\n returnValue <- paste(\"\\t\\t#' @return\", methodMetaData$response[[\"$ref\"]], \"object.\")\n\n c(description, params, returnValue)\n}\n\ngetMethodParams <- function(methodMetaData)\n{\n request <- methodMetaData$request\n requestDoc <- NULL\n\n if(!is.null(request))\n {\n requestDoc <- unname(unlist(sapply(request$properties, function(prop)\n {\n className <- sapply(prop, function(ref) ref)\n objectName <- normalizeParamName(className)\n paste(\"\\t\\t#' @param\", objectName, className, \"object.\")\n })))\n }\n\n argNames <- names(methodMetaData$parameters)\n\n argsDoc <- unname(unlist(sapply(argNames, function(argName)\n {\n arg <- methodMetaData$parameters[[argName]]\n paste(\"\\t\\t#' @param\",\n normalizeParamName(argName),\n gsub(\"\\n\", \"\\n\\t\\t#' \", arg$description)\n )\n })))\n\n c(requestDoc, argsDoc)\n}\n\n#NOTE: Utility functions:\n\n# This function is used to split very long lines of code into smaller chunks.\n# This is usually the case when we pass a lot of named argumets to a function.\nformatArgs <- function(prependAtStart, prependToEachSplit,\n args, appendAtEnd, lineLength)\n{\n if(length(args) > 1)\n {\n args[1:(length(args) - 1)] <- paste0(args[1:(length(args) - 1)], \",\")\n }\n\n args[1] <- paste0(prependAtStart, args[1])\n args[length(args)] <- paste0(args[length(args)], appendAtEnd)\n\n argsLength <- length(args)\n argLines <- list()\n index <- 1\n\n while(index <= argsLength)\n {\n line <- args[index]\n index <- index + 1\n\n while(nchar(line) < lineLength && index <= argsLength)\n {\n line <- paste(line, args[index])\n index <- index + 1\n }\n\n argLines <- c(argLines, line)\n }\n\n argLines <- unlist(argLines)\n argLinesLen <- length(argLines)\n\n if(argLinesLen > 1)\n argLines[2:argLinesLen] <- paste0(prependToEachSplit, argLines[2:argLinesLen])\n\n argLines\n}\n\nargs <- commandArgs(TRUE)\nif (length(args) == 0) {\n loc <- \"arvados-v1-discovery.json\"\n} else {\n loc <- args[[1]]\n}\ndiscoveryDocument <- getAPIDocument(loc)\ngenerateAPI(discoveryDocument)\n" }, { "path": "contrib/R-sdk/install_deps.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\noptions(repos=structure(c(CRAN=\"https://cloud.r-project.org/\")))\nif (!requireNamespace(\"devtools\")) {\n install.packages(\"devtools\")\n}\nif (!requireNamespace(\"roxygen2\")) {\n install.packages(\"roxygen2\")\n}\nif (!requireNamespace(\"knitr\")) {\n install.packages(\"knitr\")\n}\nif (!requireNamespace(\"markdown\")) {\n install.packages(\"markdown\")\n}\nif (!requireNamespace(\"XML\")) {\n install.packages(\"XML\")\n}\n\ndevtools::install_dev_deps()\n" }, { "path": "contrib/R-sdk/run_test.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\ndevtools::check()\n\nresults <- devtools::test()\nany_error <- any(as.data.frame(results)$error)\nif (any_error) {\n q(\"no\", 1)\n} else {\n q(\"no\", 0)\n}\n" }, { "path": "contrib/R-sdk/tests/testthat/fakes/FakeArvados.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\nFakeArvados <- R6::R6Class(\n\n \"FakeArvados\",\n\n public = list(\n\n token = NULL,\n host = NULL,\n webdavHost = NULL,\n http = NULL,\n httpParser = NULL,\n REST = NULL,\n\n initialize = function(token = NULL,\n host = NULL,\n webdavHost = NULL,\n http = NULL,\n httpParser = NULL)\n {\n self$token <- token\n self$host <- host\n self$webdavHost <- webdavHost\n self$http <- http\n self$httpParser <- httpParser\n },\n\n getToken = function() self$token,\n getHostName = function() self$host,\n getHttpClient = function() self$http,\n getHttpParser = function() self$httpParser,\n getWebDavHostName = function() self$webdavHost\n ),\n\n cloneable = FALSE\n)\n" }, { "path": "contrib/R-sdk/tests/testthat/fakes/FakeHttpParser.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\nFakeHttpParser <- R6::R6Class(\n\n \"FakeHttrParser\",\n\n public = list(\n\n validContentTypes = NULL,\n parserCallCount = NULL,\n\n initialize = function()\n {\n self$parserCallCount <- 0\n self$validContentTypes <- c(\"text\", \"raw\")\n },\n\n parseJSONResponse = function(serverResponse)\n {\n self$parserCallCount <- self$parserCallCount + 1\n\n if(!is.null(serverResponse$content))\n return(serverResponse$content)\n\n serverResponse\n },\n\n parseResponse = function(serverResponse, outputType)\n {\n self$parserCallCount <- self$parserCallCount + 1\n\n if(!is.null(serverResponse$content))\n return(serverResponse$content)\n\n serverResponse\n },\n\n getFileNamesFromResponse = function(serverResponse, uri)\n {\n self$parserCallCount <- self$parserCallCount + 1\n\n if(!is.null(serverResponse$content))\n return(serverResponse$content)\n\n serverResponse\n },\n\n getFileSizesFromResponse = function(serverResponse, uri)\n {\n self$parserCallCount <- self$parserCallCount + 1\n\n if(!is.null(serverResponse$content))\n return(serverResponse$content)\n\n serverResponse\n }\n )\n)\n" }, { "path": "contrib/R-sdk/tests/testthat/fakes/FakeHttpRequest.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\nFakeHttpRequest <- R6::R6Class(\n\n \"FakeHttpRequest\",\n\n public = list(\n\n serverMaxElementsPerRequest = NULL,\n\n content = NULL,\n expectedURL = NULL,\n URLIsProperlyConfigured = NULL,\n expectedQueryFilters = NULL,\n queryFiltersAreCorrect = NULL,\n requestHeaderContainsAuthorizationField = NULL,\n requestHeaderContainsDestinationField = NULL,\n requestHeaderContainsRangeField = NULL,\n requestHeaderContainsContentTypeField = NULL,\n JSONEncodedBodyIsProvided = NULL,\n requestBodyIsProvided = NULL,\n\n numberOfGETRequests = NULL,\n numberOfDELETERequests = NULL,\n numberOfPUTRequests = NULL,\n numberOfPOSTRequests = NULL,\n numberOfMOVERequests = NULL,\n numberOfCOPYRequests = NULL,\n numberOfgetConnectionCalls = NULL,\n\n initialize = function(expectedURL = NULL,\n serverResponse = NULL,\n expectedFilters = NULL)\n {\n if(is.null(serverResponse))\n {\n self$content <- list()\n self$content$status_code <- 200\n }\n else\n self$content <- serverResponse\n\n self$expectedURL <- expectedURL\n self$URLIsProperlyConfigured <- FALSE\n self$expectedQueryFilters <- expectedFilters\n self$queryFiltersAreCorrect <- FALSE\n self$requestHeaderContainsAuthorizationField <- FALSE\n self$requestHeaderContainsDestinationField <- FALSE\n self$requestHeaderContainsRangeField <- FALSE\n self$requestHeaderContainsContentTypeField <- FALSE\n self$JSONEncodedBodyIsProvided <- FALSE\n self$requestBodyIsProvided <- FALSE\n\n self$numberOfGETRequests <- 0\n self$numberOfDELETERequests <- 0\n self$numberOfPUTRequests <- 0\n self$numberOfPOSTRequests <- 0\n self$numberOfMOVERequests <- 0\n self$numberOfCOPYRequests <- 0\n\n self$numberOfgetConnectionCalls <- 0\n\n self$serverMaxElementsPerRequest <- 5\n },\n\n exec = function(verb, url, headers = NULL, body = NULL, query = NULL,\n limit = NULL, offset = NULL, retryTimes = 0)\n {\n private$validateURL(url)\n private$validateHeaders(headers)\n private$validateFilters(queryFilters)\n private$validateBody(body)\n\n if(verb == \"GET\")\n self$numberOfGETRequests <- self$numberOfGETRequests + 1\n else if(verb == \"POST\")\n self$numberOfPOSTRequests <- self$numberOfPOSTRequests + 1\n else if(verb == \"PUT\")\n self$numberOfPUTRequests <- self$numberOfPUTRequests + 1\n else if(verb == \"DELETE\")\n self$numberOfDELETERequests <- self$numberOfDELETERequests + 1\n else if(verb == \"MOVE\")\n self$numberOfMOVERequests <- self$numberOfMOVERequests + 1\n else if(verb == \"COPY\")\n self$numberOfCOPYRequests <- self$numberOfCOPYRequests + 1\n else if(verb == \"PROPFIND\")\n {\n return(self$content)\n }\n\n if(!is.null(self$content$items_available))\n return(private$getElements(offset, limit))\n else\n return(self$content)\n },\n\n getConnection = function(url, headers, openMode)\n {\n self$numberOfgetConnectionCalls <- self$numberOfgetConnectionCalls + 1\n c(url, headers, openMode)\n }\n ),\n\n private = list(\n\n validateURL = function(url)\n {\n if(!is.null(self$expectedURL) && url == self$expectedURL)\n self$URLIsProperlyConfigured <- TRUE\n },\n\n validateHeaders = function(headers)\n {\n if(!is.null(headers$Authorization))\n self$requestHeaderContainsAuthorizationField <- TRUE\n\n if(!is.null(headers$Destination))\n self$requestHeaderContainsDestinationField <- TRUE\n\n if(!is.null(headers$Range))\n self$requestHeaderContainsRangeField <- TRUE\n\n if(!is.null(headers[[\"Content-Type\"]]))\n self$requestHeaderContainsContentTypeField <- TRUE\n },\n\n validateBody = function(body)\n {\n if(!is.null(body))\n {\n self$requestBodyIsProvided <- TRUE\n\n if(class(body) == \"json\")\n self$JSONEncodedBodyIsProvided <- TRUE\n }\n },\n\n validateFilters = function(filters)\n {\n if(!is.null(self$expectedQueryFilters) &&\n !is.null(filters) &&\n all.equal(unname(filters), self$expectedQueryFilters))\n {\n self$queryFiltersAreCorrect <- TRUE\n }\n },\n\n getElements = function(offset, limit)\n {\n start <- 1\n elementCount <- self$serverMaxElementsPerRequest\n\n if(!is.null(offset))\n {\n if(offset > self$content$items_available)\n stop(\"Invalid offset\")\n\n start <- offset + 1\n }\n\n if(!is.null(limit))\n if(limit < self$serverMaxElementsPerRequest)\n elementCount <- limit - 1\n\n\n serverResponse <- list()\n serverResponse$items_available <- self$content$items_available\n serverResponse$items <- self$content$items[start:(start + elementCount - 1)]\n\n if(start + elementCount > self$content$items_available)\n {\n elementCount = self$content$items_available - start\n serverResponse$items <- self$content$items[start:(start + elementCount)]\n }\n\n serverResponse\n }\n ),\n\n cloneable = FALSE\n)\n" }, { "path": "contrib/R-sdk/tests/testthat/fakes/FakeRESTService.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\nFakeRESTService <- R6::R6Class(\n\n \"FakeRESTService\",\n\n public = list(\n\n getResourceCallCount = NULL,\n createResourceCallCount = NULL,\n listResourcesCallCount = NULL,\n deleteResourceCallCount = NULL,\n updateResourceCallCount = NULL,\n fetchAllItemsCallCount = NULL,\n\n createCallCount = NULL,\n deleteCallCount = NULL,\n moveCallCount = NULL,\n copyCallCount = NULL,\n getCollectionContentCallCount = NULL,\n getResourceSizeCallCount = NULL,\n readCallCount = NULL,\n writeCallCount = NULL,\n getConnectionCallCount = NULL,\n writeBuffer = NULL,\n filtersAreConfiguredCorrectly = NULL,\n bodyIsConfiguredCorrectly = NULL,\n expectedFilterContent = NULL,\n\n collectionContent = NULL,\n returnContent = NULL,\n\n initialize = function(collectionContent = NULL, returnContent = NULL,\n expectedFilterContent = NULL)\n {\n self$getResourceCallCount <- 0\n self$createResourceCallCount <- 0\n self$listResourcesCallCount <- 0\n self$deleteResourceCallCount <- 0\n self$updateResourceCallCount <- 0\n self$fetchAllItemsCallCount <- 0\n\n self$createCallCount <- 0\n self$deleteCallCount <- 0\n self$moveCallCount <- 0\n self$copyCallCount <- 0\n self$getCollectionContentCallCount <- 0\n self$getResourceSizeCallCount <- 0\n self$readCallCount <- 0\n self$writeCallCount <- 0\n self$getConnectionCallCount <- 0\n self$filtersAreConfiguredCorrectly <- FALSE\n self$bodyIsConfiguredCorrectly <- FALSE\n\n self$collectionContent <- collectionContent\n self$returnContent <- returnContent\n self$expectedFilterContent <- expectedFilterContent\n },\n\n getWebDavHostName = function()\n {\n },\n\n getResource = function(resource, uuid)\n {\n self$getResourceCallCount <- self$getResourceCallCount + 1\n self$returnContent\n },\n\n listResources = function(resource, filters = NULL, limit = 100, offset = 0)\n {\n self$listResourcesCallCount <- self$listResourcesCallCount + 1\n\n if(!is.null(self$expectedFilterContent) && !is.null(filters))\n if(all.equal(filters, self$expectedFilterContent))\n self$filtersAreConfiguredCorrectly <- TRUE\n\n self$returnContent\n },\n\n fetchAllItems = function(resourceURL, filters)\n {\n self$fetchAllItemsCallCount <- self$fetchAllItemsCallCount + 1\n\n if(!is.null(self$expectedFilterContent) && !is.null(filters))\n if(all.equal(filters, self$expectedFilterContent))\n self$filtersAreConfiguredCorrectly <- TRUE\n\n self$returnContent\n },\n\n deleteResource = function(resource, uuid)\n {\n self$deleteResourceCallCount <- self$deleteResourceCallCount + 1\n self$returnContent\n },\n\n updateResource = function(resource, uuid, newContent)\n {\n self$updateResourceCallCount <- self$updateResourceCallCount + 1\n\n if(!is.null(self$returnContent) && !is.null(newContent))\n if(all.equal(newContent, self$returnContent))\n self$bodyIsConfiguredCorrectly <- TRUE\n\n self$returnContent\n },\n\n createResource = function(resource, content)\n {\n self$createResourceCallCount <- self$createResourceCallCount + 1\n\n if(!is.null(self$returnContent) && !is.null(content))\n if(all.equal(content, self$returnContent))\n self$bodyIsConfiguredCorrectly <- TRUE\n\n self$returnContent\n },\n\n create = function(files, uuid)\n {\n self$createCallCount <- self$createCallCount + 1\n self$returnContent\n },\n\n delete = function(relativePath, uuid)\n {\n self$deleteCallCount <- self$deleteCallCount + 1\n self$returnContent\n },\n\n move = function(from, to, uuid)\n {\n self$moveCallCount <- self$moveCallCount + 1\n self$returnContent\n },\n\n copy = function(from, to, uuid)\n {\n self$copyCallCount <- self$copyCallCount + 1\n self$returnContent\n },\n\n getCollectionContent = function(uuid, relativePath = NULL)\n {\n self$getCollectionContentCallCount <- self$getCollectionContentCallCount + 1\n if (!is.null(relativePath)) {\n self$collectionContent[startsWith(self$collectionContent, relativePath)]\n } else {\n self$collectionContent\n }\n },\n\n getResourceSize = function(uuid, relativePathToResource)\n {\n self$getResourceSizeCallCount <- self$getResourceSizeCallCount + 1\n self$returnContent\n },\n\n read = function(relativePath, uuid, contentType = \"text\", offset = 0, length = 0)\n {\n self$readCallCount <- self$readCallCount + 1\n self$returnContent\n },\n\n write = function(relativePath, uuid, content, contentType)\n {\n self$writeBuffer <- content\n self$writeCallCount <- self$writeCallCount + 1\n self$returnContent\n },\n\n getConnection = function(uuid, relativePath, openMode)\n {\n self$getConnectionCallCount <- self$getConnectionCallCount + 1\n self$returnContent\n }\n ),\n\n cloneable = FALSE\n)\n" }, { "path": "contrib/R-sdk/tests/testthat/test-ArvadosFile.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\nsource(\"fakes/FakeRESTService.R\")\n\ncontext(\"ArvadosFile\")\n\ntest_that(\"constructor raises error if file name is empty string\", {\n\n expect_that(ArvadosFile$new(\"\"), throws_error(\"Invalid name.\"))\n})\n\ntest_that(\"getFileListing always returns file name\", {\n\n dog <- ArvadosFile$new(\"dog\")\n\n expect_that(dog$getFileListing(), equals(\"dog\"))\n})\n\ntest_that(\"get always returns NULL\", {\n\n dog <- ArvadosFile$new(\"dog\")\n\n responseIsNull <- is.null(dog$get(\"something\"))\n expect_true(responseIsNull)\n})\n\ntest_that(\"getFirst always returns NULL\", {\n\n dog <- ArvadosFile$new(\"dog\")\n\n responseIsNull <- is.null(dog$getFirst())\n expect_true(responseIsNull)\n})\n\ntest_that(paste(\"getSizeInBytes returns zero if arvadosFile\",\n \"is not part of a collection\"), {\n\n dog <- ArvadosFile$new(\"dog\")\n\n expect_that(dog$getSizeInBytes(), equals(0))\n})\n\ntest_that(paste(\"getSizeInBytes delegates size calculation\",\n \"to REST service class\"), {\n\n collectionContent <- c(\"animal\", \"animal/fish\")\n returnSize <- 100\n fakeREST <- FakeRESTService$new(collectionContent, returnSize)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n fish <- collection$get(\"animal/fish\")\n\n resourceSize <- fish$getSizeInBytes()\n\n expect_that(resourceSize, equals(100))\n})\n\ntest_that(\"getRelativePath returns path relative to the tree root\", {\n\n animal <- Subcollection$new(\"animal\")\n fish <- Subcollection$new(\"fish\")\n shark <- ArvadosFile$new(\"shark\")\n\n animal$add(fish)\n fish$add(shark)\n\n expect_that(shark$getRelativePath(), equals(\"animal/fish/shark\"))\n})\n\ntest_that(\"read raises exception if file doesn't belong to a collection\", {\n\n dog <- ArvadosFile$new(\"dog\")\n\n expect_that(dog$read(),\n throws_error(\"ArvadosFile doesn't belong to any collection.\"))\n})\n\ntest_that(\"read raises exception offset or length is negative number\", {\n\n collectionContent <- c(\"animal\", \"animal/fish\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n fish <- collection$get(\"animal/fish\")\n\n expect_that(fish$read(contentType = \"text\", offset = -1),\n throws_error(\"Offset and length must be positive values.\"))\n expect_that(fish$read(contentType = \"text\", length = -1),\n throws_error(\"Offset and length must be positive values.\"))\n expect_that(fish$read(contentType = \"text\", offset = -1, length = -1),\n throws_error(\"Offset and length must be positive values.\"))\n})\n\ntest_that(\"read delegates reading operation to REST service class\", {\n\n collectionContent <- c(\"animal\", \"animal/fish\")\n readContent <- \"my file\"\n fakeREST <- FakeRESTService$new(collectionContent, readContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n fish <- collection$get(\"animal/fish\")\n\n fileContent <- fish$read(\"text\")\n\n expect_that(fileContent, equals(\"my file\"))\n expect_that(fakeREST$readCallCount, equals(1))\n})\n\ntest_that(paste(\"connection delegates connection creation ro RESTService class\",\n \"which returns curl connection opened in read mode when\",\n \"'r' of 'rb' is passed as argument\"), {\n\n collectionContent <- c(\"animal\", \"animal/fish\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n fish <- collection$get(\"animal/fish\")\n\n connection <- fish$connection(\"r\")\n\n expect_that(fakeREST$getConnectionCallCount, equals(1))\n})\n\ntest_that(paste(\"connection returns textConnection opened\",\n \"in write mode when 'w' is passed as argument\"), {\n\n collectionContent <- c(\"animal\", \"animal/fish\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n fish <- collection$get(\"animal/fish\")\n\n connection <- fish$connection(\"w\")\n\n writeLines(\"file\", connection)\n writeLines(\"content\", connection)\n\n writeResult <- textConnectionValue(connection)\n\n expect_that(writeResult[1], equals(\"file\"))\n expect_that(writeResult[2], equals(\"content\"))\n})\n\ntest_that(\"flush sends data stored in a connection to a REST server\", {\n\n collectionContent <- c(\"animal\", \"animal/fish\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n fish <- collection$get(\"animal/fish\")\n\n connection <- fish$connection(\"w\")\n\n writeLines(\"file content\", connection)\n\n fish$flush()\n\n expect_that(fakeREST$writeBuffer, equals(\"file content\"))\n})\n\ntest_that(\"write raises exception if file doesn't belong to a collection\", {\n\n dog <- ArvadosFile$new(\"dog\")\n\n expect_that(dog$write(),\n throws_error(\"ArvadosFile doesn't belong to any collection.\"))\n})\n\ntest_that(\"write delegates writing operation to REST service class\", {\n\n\n collectionContent <- c(\"animal\", \"animal/fish\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n fish <- collection$get(\"animal/fish\")\n\n fileContent <- fish$write(\"new file content\")\n\n expect_that(fakeREST$writeBuffer, equals(\"new file content\"))\n})\n\ntest_that(paste(\"move raises exception if arvados file\",\n \"doesn't belong to any collection\"), {\n\n animal <- ArvadosFile$new(\"animal\")\n\n expect_that(animal$move(\"new/location\"),\n throws_error(\"ArvadosFile doesn't belong to any collection.\"))\n})\n\ntest_that(paste(\"move raises exception if newLocationInCollection\",\n \"parameter is invalid\"), {\n\n collectionContent <- c(\"animal\",\n \"animal/fish\",\n \"animal/dog\",\n \"animal/fish/shark\",\n \"ball\")\n\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n\n collection <- Collection$new(api, \"myUUID\")\n dog <- collection$get(\"animal/dog\")\n\n expect_that(dog$move(\"objects/dog\"),\n throws_error(\"Unable to get destination subcollection.\"))\n})\n\ntest_that(\"move raises exception if new location contains content with the same name\", {\n\n\n collectionContent <- c(\"animal\",\n \"animal/fish\",\n \"animal/dog\",\n \"animal/fish/shark\",\n \"dog\")\n\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n dog <- collection$get(\"animal/dog\")\n\n expect_that(dog$move(\"dog\"),\n throws_error(\"Destination already contains content with same name.\"))\n\n})\n\ntest_that(\"move moves arvados file inside collection tree\", {\n\n collectionContent <- c(\"animal\",\n \"animal/fish\",\n \"animal/dog\",\n \"animal/fish/shark\",\n \"ball\")\n\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n dog <- collection$get(\"animal/dog\")\n\n dog$move(\"dog\")\n dogIsNullOnOldLocation <- is.null(collection$get(\"animal/dog\"))\n dogExistsOnNewLocation <- !is.null(collection$get(\"dog\"))\n\n expect_true(dogIsNullOnOldLocation)\n expect_true(dogExistsOnNewLocation)\n})\n\ntest_that(paste(\"copy raises exception if arvados file\",\n \"doesn't belong to any collection\"), {\n\n animal <- ArvadosFile$new(\"animal\")\n\n expect_that(animal$copy(\"new/location\"),\n throws_error(\"ArvadosFile doesn't belong to any collection.\"))\n})\n\ntest_that(paste(\"copy raises exception if location parameter is invalid\"), {\n\n collectionContent <- c(\"animal\",\n \"animal/fish\",\n \"animal/dog\",\n \"animal/fish/shark\",\n \"ball\")\n\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n\n collection <- Collection$new(api, \"myUUID\")\n dog <- collection$get(\"animal/dog\")\n\n expect_that(dog$copy(\"objects/dog\"),\n throws_error(\"Unable to get destination subcollection.\"))\n})\n\ntest_that(\"copy raises exception if new location contains content with the same name\", {\n\n\n collectionContent <- c(\"animal\",\n \"animal/fish\",\n \"animal/dog\",\n \"animal/fish/shark\",\n \"dog\")\n\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n dog <- collection$get(\"animal/dog\")\n\n expect_that(dog$copy(\"dog\"),\n throws_error(\"Destination already contains content with same name.\"))\n\n})\n\ntest_that(\"copy copies arvados file inside collection tree\", {\n\n collectionContent <- c(\"animal\",\n \"animal/fish\",\n \"animal/dog\",\n \"animal/fish/shark\",\n \"ball\")\n\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n dog <- collection$get(\"animal/dog\")\n\n dog$copy(\"dog\")\n dogExistsOnOldLocation <- !is.null(collection$get(\"animal/dog\"))\n dogExistsOnNewLocation <- !is.null(collection$get(\"dog\"))\n\n expect_true(dogExistsOnOldLocation)\n expect_true(dogExistsOnNewLocation)\n})\n\ntest_that(\"duplicate performs deep cloning of Arvados file\", {\n arvFile <- ArvadosFile$new(\"foo\")\n newFile1 <- arvFile$duplicate()\n newFile2 <- arvFile$duplicate(\"bar\")\n\n expect_that(newFile1$getFileListing(), equals(arvFile$getFileListing()))\n expect_that(newFile2$getFileListing(), equals(c(\"bar\")))\n})\n" }, { "path": "contrib/R-sdk/tests/testthat/test-Collection.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\nsource(\"fakes/FakeRESTService.R\")\n\ncontext(\"Collection\")\n\ntest_that(paste(\"constructor creates file tree from text content\",\n \"retreived form REST service\"), {\n\n collectionContent <- c(\"animal\", \"animal/fish\", \"ball\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n\n root <- collection$get(\"\")\n\n expect_that(fakeREST$getCollectionContentCallCount, equals(1))\n expect_that(root$getName(), equals(\"\"))\n})\n\ntest_that(paste(\"add raises exception if passed argumet is not\",\n \"ArvadosFile or Subcollection\"), {\n\n collectionContent <- c(\"animal\", \"animal/fish\", \"ball\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n\n newNumber <- 10\n\n expect_that(collection$add(newNumber),\n throws_error(paste(\"Expected AravodsFile or Subcollection\",\n \"object, got (numeric).\"), fixed = TRUE))\n})\n\ntest_that(\"add raises exception if relative path is not valid\", {\n\n collectionContent <- c(\"animal\", \"animal/fish\", \"ball\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n\n newPen <- ArvadosFile$new(\"pen\")\n\n expect_that(collection$add(newPen, \"objects\"),\n throws_error(\"Subcollection objects doesn't exist.\",\n fixed = TRUE))\n})\n\ntest_that(\"add raises exception if content name is empty string\", {\n\n collectionContent <- c(\"animal\", \"animal/fish\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n\n rootFolder <- Subcollection$new(\"\")\n\n expect_that(collection$add(rootFolder),\n throws_error(\"Content has invalid name.\", fixed = TRUE))\n})\n\ntest_that(paste(\"add adds ArvadosFile or Subcollection\",\n \"to local tree structure and remote REST service\"), {\n\n collectionContent <- c(\"animal\", \"animal/fish\", \"ball\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n\n newDog <- ArvadosFile$new(\"dog\")\n collection$add(newDog, \"animal\")\n\n dog <- collection$get(\"animal/dog\")\n dogExistsInCollection <- !is.null(dog) && dog$getName() == \"dog\"\n\n expect_true(dogExistsInCollection)\n expect_that(fakeREST$createCallCount, equals(1))\n})\n\ntest_that(\"create raises exception if passed argumet is not character vector\", {\n\n collectionContent <- c(\"animal\", \"animal/fish\", \"ball\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n\n expect_that(collection$create(10),\n throws_error(\"Expected character vector, got (numeric).\",\n fixed = TRUE))\n})\n\ntest_that(paste(\"create adds files specified by fileNames\",\n \"to local tree structure and remote REST service\"), {\n\n fakeREST <- FakeRESTService$new()\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n\n collection$create(c(\"animal/dog\", \"animal/cat\"))\n\n dog <- collection$get(\"animal/dog\")\n cat <- collection$get(\"animal/cat\")\n dogExistsInCollection <- !is.null(dog) && dog$getName() == \"dog\"\n catExistsInCollection <- !is.null(cat) && cat$getName() == \"cat\"\n\n expect_true(dogExistsInCollection)\n expect_true(catExistsInCollection)\n expect_that(fakeREST$createCallCount, equals(2))\n})\n\ntest_that(\"remove raises exception if passed argumet is not character vector\", {\n\n collectionContent <- c(\"animal\", \"animal/fish\", \"ball\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n\n expect_that(collection$remove(10),\n throws_error(\"Expected character vector, got (numeric).\",\n fixed = TRUE))\n})\n\ntest_that(\"remove raises exception if user tries to remove root folder\", {\n\n collectionContent <- c(\"animal\", \"animal/fish\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n\n expect_that(collection$remove(\"\"),\n throws_error(\"You can't delete root folder.\", fixed = TRUE))\n})\n\ntest_that(paste(\"remove removes files specified by paths\",\n \"from local tree structure and from remote REST service\"), {\n\n collectionContent <- c(\"animal\", \"animal/fish\", \"animal/dog\", \"animal/cat\", \"ball\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n\n collection$remove(c(\"animal/dog\", \"animal/cat\"))\n\n dog <- collection$get(\"animal/dog\")\n cat <- collection$get(\"animal/dog\")\n dogExistsInCollection <- !is.null(dog) && dog$getName() == \"dog\"\n catExistsInCollection <- !is.null(cat) && cat$getName() == \"cat\"\n\n expect_false(dogExistsInCollection)\n expect_false(catExistsInCollection)\n expect_that(fakeREST$deleteCallCount, equals(2))\n})\n\ntest_that(paste(\"move moves content to a new location inside file tree\",\n \"and on REST service\"), {\n\n collectionContent <- c(\"animal\", \"animal/dog\", \"ball\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n\n collection$move(\"animal/dog\", \"dog\")\n\n dogIsNullOnOldLocation <- is.null(collection$get(\"animal/dog\"))\n dogExistsOnNewLocation <- !is.null(collection$get(\"dog\"))\n\n expect_true(dogIsNullOnOldLocation)\n expect_true(dogExistsOnNewLocation)\n expect_that(fakeREST$moveCallCount, equals(1))\n})\n\ntest_that(\"move raises exception if new location is not valid\", {\n\n collectionContent <- c(\"animal\", \"animal/fish\", \"ball\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n\n expect_that(collection$move(\"fish\", \"object\"),\n throws_error(\"Content you want to move doesn't exist in the collection.\",\n fixed = TRUE))\n})\n\ntest_that(\"getFileListing returns sorted collection content received from REST service\", {\n\n collectionContent <- c(\"animal\", \"animal/fish\", \"ball\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n\n contentMatchExpected <- all(collection$getFileListing() ==\n c(\"animal\", \"animal/fish\", \"ball\"))\n\n expect_true(contentMatchExpected)\n #2 calls because Collection$new calls getFileListing once\n expect_that(fakeREST$getCollectionContentCallCount, equals(2))\n\n})\n\ntest_that(\"get returns arvados file or subcollection from internal tree structure\", {\n\n collectionContent <- c(\"animal\", \"animal/fish\", \"ball\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n\n fish <- collection$get(\"animal/fish\")\n fishIsNotNull <- !is.null(fish)\n\n expect_true(fishIsNotNull)\n expect_that(fish$getName(), equals(\"fish\"))\n\n ball <- collection$get(\"ball\")\n ballIsNotNull <- !is.null(ball)\n\n expect_true(ballIsNotNull)\n expect_that(ball$getName(), equals(\"ball\"))\n})\n\ntest_that(paste(\"copy copies content to a new location inside file tree\",\n \"and on REST service\"), {\n\n collectionContent <- c(\"animal\", \"animal/dog\", \"ball\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n\n collection$copy(\"animal/dog\", \"dog\")\n\n dogExistsOnOldLocation <- !is.null(collection$get(\"animal/dog\"))\n dogExistsOnNewLocation <- !is.null(collection$get(\"dog\"))\n\n expect_true(dogExistsOnOldLocation)\n expect_true(dogExistsOnNewLocation)\n expect_that(fakeREST$copyCallCount, equals(1))\n})\n\ntest_that(\"copy raises exception if new location is not valid\", {\n\n collectionContent <- c(\"animal\", \"animal/fish\", \"ball\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n\n expect_that(collection$copy(\"fish\", \"object\"),\n throws_error(\"Content you want to copy doesn't exist in the collection.\",\n fixed = TRUE))\n})\n\ntest_that(\"refresh invalidates current tree structure\", {\n\n collectionContent <- c(\"animal\", \"animal/fish\", \"ball\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"aaaaa-j7d0g-ccccccccccccccc\")\n\n # Before refresh\n fish <- collection$get(\"animal/fish\")\n expect_that(fish$getName(), equals(\"fish\"))\n expect_that(fish$getCollection()$uuid, equals(\"aaaaa-j7d0g-ccccccccccccccc\"))\n\n collection$refresh()\n\n # After refresh\n expect_that(fish$getName(), equals(\"fish\"))\n expect_true(is.null(fish$getCollection()))\n})\n" }, { "path": "contrib/R-sdk/tests/testthat/test-CollectionTree.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\ncontext(\"CollectionTree\")\n\ntest_that(\"constructor creates file tree from character array properly\", {\n\n collection <- \"myCollection\"\n characterArray <- c(\"animal\",\n \"animal/dog\",\n \"boat\")\n\n collectionTree <- CollectionTree$new(characterArray, collection)\n\n root <- collectionTree$getTree()\n animal <- collectionTree$getElement(\"animal\")\n dog <- collectionTree$getElement(\"animal/dog\")\n boat <- collectionTree$getElement(\"boat\")\n\n rootHasNoParent <- is.null(root$getParent())\n rootIsOfTypeSubcollection <- \"Subcollection\" %in% class(root)\n animalIsOfTypeSubcollection <- \"Subcollection\" %in% class(animal)\n dogIsOfTypeArvadosFile <- \"ArvadosFile\" %in% class(dog)\n boatIsOfTypeArvadosFile <- \"ArvadosFile\" %in% class(boat)\n animalsParentIsRoot <- animal$getParent()$getName() == root$getName()\n animalContainsDog <- animal$getFirst()$getName() == dog$getName()\n dogsParentIsAnimal <- dog$getParent()$getName() == animal$getName()\n boatsParentIsRoot <- boat$getParent()$getName() == root$getName()\n\n allElementsBelongToSameCollection <- root$getCollection() == \"myCollection\" &&\n animal$getCollection() == \"myCollection\" &&\n dog$getCollection() == \"myCollection\" &&\n boat$getCollection() == \"myCollection\"\n\n expect_that(root$getName(), equals(\"\"))\n expect_true(rootIsOfTypeSubcollection)\n expect_true(rootHasNoParent)\n expect_true(animalIsOfTypeSubcollection)\n expect_true(animalsParentIsRoot)\n expect_true(animalContainsDog)\n expect_true(dogIsOfTypeArvadosFile)\n expect_true(dogsParentIsAnimal)\n expect_true(boatIsOfTypeArvadosFile)\n expect_true(boatsParentIsRoot)\n expect_true(allElementsBelongToSameCollection)\n})\n\ntest_that(\"getElement returns element from tree if element exists on specified path\", {\n\n collection <- \"myCollection\"\n characterArray <- c(\"animal\",\n \"animal/dog\",\n \"boat\")\n\n collectionTree <- CollectionTree$new(characterArray, collection)\n\n dog <- collectionTree$getElement(\"animal/dog\")\n\n expect_that(dog$getName(), equals(\"dog\"))\n})\n\ntest_that(\"getElement returns NULL from tree if element doesn't exists on specified path\", {\n\n collection <- \"myCollection\"\n characterArray <- c(\"animal\",\n \"animal/dog\",\n \"boat\")\n\n collectionTree <- CollectionTree$new(characterArray, collection)\n\n fish <- collectionTree$getElement(\"animal/fish\")\n fishIsNULL <- is.null(fish)\n\n expect_true(fishIsNULL)\n})\n\ntest_that(\"getElement trims ./ from start of relativePath\", {\n\n collection <- \"myCollection\"\n characterArray <- c(\"animal\",\n \"animal/dog\",\n \"boat\")\n\n collectionTree <- CollectionTree$new(characterArray, collection)\n\n dog <- collectionTree$getElement(\"animal/dog\")\n dogWithDotSlash <- collectionTree$getElement(\"./animal/dog\")\n\n expect_that(dogWithDotSlash$getName(), equals(dog$getName()))\n})\n\ntest_that(\"getElement trims / from end of relativePath\", {\n\n collection <- \"myCollection\"\n characterArray <- c(\"animal\",\n \"animal/dog\",\n \"boat\")\n\n collectionTree <- CollectionTree$new(characterArray, collection)\n\n animal <- collectionTree$getElement(\"animal\")\n animalWithSlash <- collectionTree$getElement(\"animal/\")\n\n expect_that(animalWithSlash$getName(), equals(animal$getName()))\n})\n" }, { "path": "contrib/R-sdk/tests/testthat/test-HttpParser.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\ncontext(\"Http Parser\")\n\n\ntest_that(\"parseJSONResponse generates and returns JSON object from server response\", {\n\n JSONContent <- \"{\\\"bar\\\":{\\\"foo\\\":[10]}}\"\n serverResponse <- list()\n serverResponse$content <- charToRaw(JSONContent)\n serverResponse$headers[[\"Content-Type\"]] <- \"application/json; charset=utf-8\"\n class(serverResponse) <- c(\"response\")\n\n parser <- HttpParser$new()\n\n result <- parser$parseJSONResponse(serverResponse)\n barExists <- !is.null(result$bar)\n\n expect_true(barExists)\n expect_that(unlist(result$bar$foo), equals(10))\n})\n\ntest_that(paste(\"parseResponse generates and returns character vector\",\n \"from server response if outputType is text\"), {\n\n content <- \"random text\"\n serverResponse <- list()\n serverResponse$content <- charToRaw(content)\n serverResponse$headers[[\"Content-Type\"]] <- \"text/plain; charset=utf-8\"\n class(serverResponse) <- c(\"response\")\n\n parser <- HttpParser$new()\n parsedResponse <- parser$parseResponse(serverResponse, \"text\")\n\n expect_that(parsedResponse, equals(\"random text\"))\n})\n\n\nwebDAVResponseSample =\n paste0(\"/c=aaaaa-bbbbb-ccccccccccccccc/Fri, 11 Jan 2018 1\",\n \"1:11:11 GMTHTTP/1.1 200 OK/c=aaaaa-bbb\",\n \"bb-ccccccccccccccc/myFile.exeFri, 12 Jan 2018\",\n \" 22:22:22 GMTtext/x-c++src\",\n \"; charset=utf-8myFile.exe25\\\"123b12dd1234567890\\\"\",\n \"HTTP/1.1 200 OK\")\n\n\n\ntest_that(paste(\"getFileNamesFromResponse returns file names belonging to specific\",\n \"collection parsed from webDAV server response\"), {\n\n serverResponse <- list()\n serverResponse$content <- charToRaw(webDAVResponseSample)\n serverResponse$headers[[\"Content-Type\"]] <- \"text/xml; charset=utf-8\"\n class(serverResponse) <- c(\"response\")\n url <- URLencode(\"https://webdav/c=aaaaa-bbbbb-ccccccccccccccc\")\n\n parser <- HttpParser$new()\n result <- parser$getFileNamesFromResponse(serverResponse, url)\n expectedResult <- \"myFile.exe\"\n resultMatchExpected <- all.equal(result, expectedResult)\n\n expect_true(resultMatchExpected)\n})\n\ntest_that(paste(\"getFileSizesFromResponse returns file sizes\",\n \"parsed from webDAV server response\"), {\n\n serverResponse <- list()\n serverResponse$content <- charToRaw(webDAVResponseSample)\n serverResponse$headers[[\"Content-Type\"]] <- \"text/xml; charset=utf-8\"\n class(serverResponse) <- c(\"response\")\n url <- URLencode(\"https://webdav/c=aaaaa-bbbbb-ccccccccccccccc\")\n\n parser <- HttpParser$new()\n expectedResult <- \"25\"\n result <- parser$getFileSizesFromResponse(serverResponse, url)\n resultMatchExpected <- result == expectedResult\n\n expect_true(resultMatchExpected)\n})\n" }, { "path": "contrib/R-sdk/tests/testthat/test-HttpRequest.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\ncontext(\"Http Request\")\n\n\ntest_that(\"execute raises exception if http verb is not valid\", {\n\n http <- HttpRequest$new()\n expect_that(http$exec(\"FAKE VERB\", \"url\"),\n throws_error(\"Http verb is not valid.\"))\n})\n\ntest_that(\"createQuery generates and encodes query portion of http\", {\n\n http <- HttpRequest$new()\n queryParams <- list()\n queryParams$filters <- list(list(\"color\", \"=\", \"red\"))\n queryParams$limit <- 20\n queryParams$offset <- 50\n expect_that(http$createQuery(queryParams),\n equals(paste0(\"?filters=%5B%5B%22color%22%2C%22%3D%22%2C%22red\",\n \"%22%5D%5D&limit=20&offset=50\")))\n})\n\ntest_that(\"createQuery generates and empty string when queryParams is an empty list\", {\n\n http <- HttpRequest$new()\n expect_that(http$createQuery(list()), equals(\"\"))\n})\n\ntest_that(\"exec calls httr functions correctly\", {\n httrNamespace <- getNamespace(\"httr\")\n\n # Monkeypatch httr functions and assert that they are called later\n add_headersCalled <- FALSE\n unlockBinding(\"add_headers\", httrNamespace)\n newAddHeaders <- function(h)\n {\n add_headersCalled <<- TRUE\n list()\n }\n httrNamespace$add_headers <- newAddHeaders\n lockBinding(\"add_headers\", httrNamespace)\n\n expectedConfig <- list()\n retryCalled <- FALSE\n unlockBinding(\"RETRY\", httrNamespace)\n newRETRY <- function(verb, url, body, config, times)\n {\n retryCalled <<- TRUE\n expectedConfig <<- config\n }\n httrNamespace$RETRY <- newRETRY\n lockBinding(\"RETRY\", httrNamespace)\n\n Sys.setenv(\"ARVADOS_API_HOST_INSECURE\" = TRUE)\n http <- HttpRequest$new()\n http$exec(\"GET\", \"url\")\n\n expect_true(add_headersCalled)\n expect_true(retryCalled)\n expect_that(expectedConfig$options, equals(list(ssl_verifypeer = 0L)))\n})\n\ntest_that(\"getConnection calls curl functions correctly\", {\n curlNamespace <- getNamespace(\"curl\")\n\n # Monkeypatch curl functions and assert that they are called later\n curlCalled <- FALSE\n unlockBinding(\"curl\", curlNamespace)\n newCurl <- function(url, open, handle) curlCalled <<- TRUE\n curlNamespace$curl <- newCurl\n lockBinding(\"curl\", curlNamespace)\n\n new_handleCalled <- FALSE\n unlockBinding(\"new_handle\", curlNamespace)\n newHandleFun <- function()\n {\n new_handleCalled <<- TRUE\n list()\n }\n curlNamespace$new_handle <- newHandleFun\n lockBinding(\"new_handle\", curlNamespace)\n\n handle_setheadersCalled <- FALSE\n unlockBinding(\"handle_setheaders\", curlNamespace)\n newHandleSetHeaders <- function(h, .list) handle_setheadersCalled <<- TRUE\n curlNamespace$handle_setheaders <- newHandleSetHeaders\n lockBinding(\"handle_setheaders\", curlNamespace)\n\n handle_setoptCalled <- FALSE\n unlockBinding(\"handle_setopt\", curlNamespace)\n newHandleSetOpt <- function(h, ssl_verifypeer) handle_setoptCalled <<- TRUE\n curlNamespace$handle_setopt <- newHandleSetOpt\n lockBinding(\"handle_setopt\", curlNamespace)\n\n\n Sys.setenv(\"ARVADOS_API_HOST_INSECURE\" = TRUE)\n http <- HttpRequest$new()\n http$getConnection(\"location\", list(), \"r\")\n\n expect_true(new_handleCalled)\n expect_true(handle_setheadersCalled)\n expect_true(handle_setoptCalled)\n expect_true(curlCalled)\n})\n" }, { "path": "contrib/R-sdk/tests/testthat/test-RESTService.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\nsource(\"fakes/FakeArvados.R\")\nsource(\"fakes/FakeHttpRequest.R\")\nsource(\"fakes/FakeHttpParser.R\")\n\ncontext(\"REST service\")\n\ntest_that(\"getWebDavHostName calls REST service properly\", {\n\n expectedURL <- \"https://host/arvados/v1/config\"\n serverResponse <- list(Services = list(WebDAVDownload = list(ExternalURL = \"https://myWebDavServer.com\")))\n httpRequest <- FakeHttpRequest$new(expectedURL, serverResponse)\n\n REST <- RESTService$new(\"token\", \"host\",\n httpRequest, FakeHttpParser$new())\n\n REST$getWebDavHostName()\n\n expect_true(httpRequest$URLIsProperlyConfigured)\n expect_false(httpRequest$requestHeaderContainsAuthorizationField)\n expect_that(httpRequest$numberOfGETRequests, equals(1))\n})\n\ntest_that(\"getWebDavHostName returns webDAV host name properly\", {\n\n serverResponse <- list(Services = list(WebDAVDownload = list(ExternalURL = \"https://myWebDavServer.com\")))\n httpRequest <- FakeHttpRequest$new(expectedURL = NULL, serverResponse)\n\n REST <- RESTService$new(\"token\", \"host\",\n httpRequest, FakeHttpParser$new())\n\n expect_that(\"https://myWebDavServer.com\", equals(REST$getWebDavHostName()))\n})\n\ntest_that(\"create calls REST service properly\", {\n\n uuid <- \"aaaaa-j7d0g-ccccccccccccccc\"\n expectedURL <- \"https://webDavHost/c=aaaaa-j7d0g-ccccccccccccccc/file\"\n fakeHttp <- FakeHttpRequest$new(expectedURL)\n fakeHttpParser <- FakeHttpParser$new()\n\n REST <- RESTService$new(\"token\", \"https://host/\",\n fakeHttp, fakeHttpParser,\n 0, \"https://webDavHost/\")\n\n REST$create(\"file\", uuid)\n\n expect_true(fakeHttp$URLIsProperlyConfigured)\n expect_true(fakeHttp$requestHeaderContainsAuthorizationField)\n expect_that(fakeHttp$numberOfPUTRequests, equals(1))\n})\n\ntest_that(\"create raises exception if server response code is not between 200 and 300\", {\n\n uuid <- \"aaaaa-j7d0g-ccccccccccccccc\"\n response <- list()\n response$status_code <- 404\n fakeHttp <- FakeHttpRequest$new(serverResponse = response)\n\n REST <- RESTService$new(\"token\", \"https://host/\",\n fakeHttp, HttpParser$new(),\n 0, \"https://webDavHost/\")\n\n expect_that(REST$create(\"file\", uuid),\n throws_error(\"Server code: 404\"))\n})\n\ntest_that(\"delete calls REST service properly\", {\n\n uuid <- \"aaaaa-j7d0g-ccccccccccccccc\"\n expectedURL <- \"https://webDavHost/c=aaaaa-j7d0g-ccccccccccccccc/file\"\n fakeHttp <- FakeHttpRequest$new(expectedURL)\n fakeHttpParser <- FakeHttpParser$new()\n\n REST <- RESTService$new(\"token\", \"https://host/\",\n fakeHttp, fakeHttpParser,\n 0, \"https://webDavHost/\")\n\n REST$delete(\"file\", uuid)\n\n expect_true(fakeHttp$URLIsProperlyConfigured)\n expect_true(fakeHttp$requestHeaderContainsAuthorizationField)\n expect_that(fakeHttp$numberOfDELETERequests, equals(1))\n})\n\ntest_that(\"delete raises exception if server response code is not between 200 and 300\", {\n\n uuid <- \"aaaaa-j7d0g-ccccccccccccccc\"\n response <- list()\n response$status_code <- 404\n fakeHttp <- FakeHttpRequest$new(serverResponse = response)\n\n REST <- RESTService$new(\"token\", \"https://host/\",\n fakeHttp, HttpParser$new(),\n 0, \"https://webDavHost/\")\n\n expect_that(REST$delete(\"file\", uuid),\n throws_error(\"Server code: 404\"))\n})\n\ntest_that(\"move calls REST service properly\", {\n\n uuid <- \"aaaaa-j7d0g-ccccccccccccccc\"\n expectedURL <- \"https://webDavHost/c=aaaaa-j7d0g-ccccccccccccccc/file\"\n fakeHttp <- FakeHttpRequest$new(expectedURL)\n fakeHttpParser <- FakeHttpParser$new()\n\n REST <- RESTService$new(\"token\", \"https://host/\",\n fakeHttp, fakeHttpParser,\n 0, \"https://webDavHost/\")\n\n REST$move(\"file\", \"newDestination/file\", uuid)\n\n expect_true(fakeHttp$URLIsProperlyConfigured)\n expect_true(fakeHttp$requestHeaderContainsAuthorizationField)\n expect_true(fakeHttp$requestHeaderContainsDestinationField)\n expect_that(fakeHttp$numberOfMOVERequests, equals(1))\n})\n\ntest_that(\"move raises exception if server response code is not between 200 and 300\", {\n\n uuid <- \"aaaaa-j7d0g-ccccccccccccccc\"\n response <- list()\n response$status_code <- 404\n fakeHttp <- FakeHttpRequest$new(serverResponse = response)\n\n REST <- RESTService$new(\"token\", \"https://host/\",\n fakeHttp, HttpParser$new(),\n 0, \"https://webDavHost/\")\n\n expect_that(REST$move(\"file\", \"newDestination/file\", uuid),\n throws_error(\"Server code: 404\"))\n})\n\ntest_that(\"copy calls REST service properly\", {\n\n uuid <- \"aaaaa-j7d0g-ccccccccccccccc\"\n expectedURL <- \"https://webDavHost/c=aaaaa-j7d0g-ccccccccccccccc/file\"\n fakeHttp <- FakeHttpRequest$new(expectedURL)\n fakeHttpParser <- FakeHttpParser$new()\n\n REST <- RESTService$new(\"token\", \"https://host/\",\n fakeHttp, fakeHttpParser,\n 0, \"https://webDavHost/\")\n\n REST$copy(\"file\", \"newDestination/file\", uuid)\n\n expect_true(fakeHttp$URLIsProperlyConfigured)\n expect_true(fakeHttp$requestHeaderContainsAuthorizationField)\n expect_true(fakeHttp$requestHeaderContainsDestinationField)\n expect_that(fakeHttp$numberOfCOPYRequests, equals(1))\n})\n\ntest_that(\"copy raises exception if server response code is not between 200 and 300\", {\n\n uuid <- \"aaaaa-j7d0g-ccccccccccccccc\"\n response <- list()\n response$status_code <- 404\n fakeHttp <- FakeHttpRequest$new(serverResponse = response)\n\n REST <- RESTService$new(\"token\", \"https://host/\",\n fakeHttp, HttpParser$new(),\n 0, \"https://webDavHost/\")\n\n expect_that(REST$copy(\"file\", \"newDestination/file\", uuid),\n throws_error(\"Server code: 404\"))\n})\n\ntest_that(\"getCollectionContent retreives correct content from WebDAV server\", {\n\n uuid <- \"aaaaa-j7d0g-ccccccccccccccc\"\n expectedURL <- \"https://webDavHost/c=aaaaa-j7d0g-ccccccccccccccc\"\n returnContent <- list()\n returnContent$status_code <- 200\n returnContent$content <- c(\"animal\", \"animal/dog\", \"ball\")\n\n fakeHttp <- FakeHttpRequest$new(expectedURL, returnContent)\n\n REST <- RESTService$new(\"token\", \"https://host/\",\n fakeHttp, FakeHttpParser$new(),\n 0, \"https://webDavHost/\")\n\n returnResult <- REST$getCollectionContent(uuid)\n returnedContentMatchExpected <- all.equal(returnResult,\n c(\"animal\", \"animal/dog\", \"ball\"))\n\n expect_true(returnedContentMatchExpected)\n expect_true(fakeHttp$requestHeaderContainsAuthorizationField)\n})\n\ntest_that(\"getCollectionContent raises exception if server returns empty response\", {\n\n uuid <- \"aaaaa-j7d0g-ccccccccccccccc\"\n response <- \"\"\n fakeHttp <- FakeHttpRequest$new(serverResponse = response)\n\n REST <- RESTService$new(\"token\", \"https://host/\",\n fakeHttp, FakeHttpParser$new(),\n 0, \"https://webDavHost/\")\n\n expect_that(REST$getCollectionContent(uuid),\n throws_error(\"Response is empty, request may be misconfigured\"))\n})\n\ntest_that(\"getCollectionContent parses server response\", {\n\n uuid <- \"aaaaa-j7d0g-ccccccccccccccc\"\n fakeHttpParser <- FakeHttpParser$new()\n REST <- RESTService$new(\"token\", \"https://host/\",\n FakeHttpRequest$new(), fakeHttpParser,\n 0, \"https://webDavHost/\")\n\n REST$getCollectionContent(uuid)\n\n expect_that(fakeHttpParser$parserCallCount, equals(1))\n})\n\ntest_that(\"getCollectionContent raises exception if server returns empty response\", {\n\n uuid <- \"aaaaa-j7d0g-ccccccccccccccc\"\n response <- \"\"\n fakeHttp <- FakeHttpRequest$new(serverResponse = response)\n\n REST <- RESTService$new(\"token\", \"https://host/\",\n fakeHttp, FakeHttpParser$new(),\n 0, \"https://webDavHost/\")\n\n expect_that(REST$getCollectionContent(uuid),\n throws_error(\"Response is empty, request may be misconfigured\"))\n})\n\ntest_that(paste(\"getCollectionContent raises exception if server\",\n \"response code is not between 200 and 300\"), {\n\n uuid <- \"aaaaa-j7d0g-ccccccccccccccc\"\n response <- list()\n response$status_code <- 404\n fakeHttp <- FakeHttpRequest$new(serverResponse = response)\n\n REST <- RESTService$new(\"token\", \"https://host/\",\n fakeHttp, HttpParser$new(),\n 0, \"https://webDavHost/\")\n\n expect_that(REST$getCollectionContent(uuid),\n throws_error(\"Server code: 404\"))\n})\n\n\ntest_that(\"getResourceSize calls REST service properly\", {\n\n uuid <- \"aaaaa-j7d0g-ccccccccccccccc\"\n expectedURL <- \"https://webDavHost/c=aaaaa-j7d0g-ccccccccccccccc/file\"\n response <- list()\n response$status_code <- 200\n response$content <- c(6, 2, 931, 12003)\n fakeHttp <- FakeHttpRequest$new(expectedURL, response)\n\n REST <- RESTService$new(\"token\", \"https://host/\",\n fakeHttp, FakeHttpParser$new(),\n 0, \"https://webDavHost/\")\n\n returnResult <- REST$getResourceSize(\"file\", uuid)\n returnedContentMatchExpected <- all.equal(returnResult,\n c(6, 2, 931, 12003))\n\n expect_true(fakeHttp$URLIsProperlyConfigured)\n expect_true(fakeHttp$requestHeaderContainsAuthorizationField)\n expect_true(returnedContentMatchExpected)\n})\n\ntest_that(\"getResourceSize raises exception if server returns empty response\", {\n\n uuid <- \"aaaaa-j7d0g-ccccccccccccccc\"\n response <- \"\"\n fakeHttp <- FakeHttpRequest$new(serverResponse = response)\n\n REST <- RESTService$new(\"token\", \"https://host/\",\n fakeHttp, FakeHttpParser$new(),\n 0, \"https://webDavHost/\")\n\n expect_that(REST$getResourceSize(\"file\", uuid),\n throws_error(\"Response is empty, request may be misconfigured\"))\n})\n\ntest_that(paste(\"getResourceSize raises exception if server\",\n \"response code is not between 200 and 300\"), {\n\n uuid <- \"aaaaa-j7d0g-ccccccccccccccc\"\n response <- list()\n response$status_code <- 404\n fakeHttp <- FakeHttpRequest$new(serverResponse = response)\n\n REST <- RESTService$new(\"token\", \"https://host/\",\n fakeHttp, HttpParser$new(),\n 0, \"https://webDavHost/\")\n\n expect_that(REST$getResourceSize(\"file\", uuid),\n throws_error(\"Server code: 404\"))\n})\n\ntest_that(\"getResourceSize parses server response\", {\n\n uuid <- \"aaaaa-j7d0g-ccccccccccccccc\"\n fakeHttpParser <- FakeHttpParser$new()\n REST <- RESTService$new(\"token\", \"https://host/\",\n FakeHttpRequest$new(), fakeHttpParser,\n 0, \"https://webDavHost/\")\n\n REST$getResourceSize(\"file\", uuid)\n\n expect_that(fakeHttpParser$parserCallCount, equals(1))\n})\n\ntest_that(\"read calls REST service properly\", {\n\n uuid <- \"aaaaa-j7d0g-ccccccccccccccc\"\n expectedURL <- \"https://webDavHost/c=aaaaa-j7d0g-ccccccccccccccc/file\"\n serverResponse <- list()\n serverResponse$status_code <- 200\n serverResponse$content <- \"file content\"\n\n fakeHttp <- FakeHttpRequest$new(expectedURL, serverResponse)\n\n REST <- RESTService$new(\"token\", \"https://host/\",\n fakeHttp, FakeHttpParser$new(),\n 0, \"https://webDavHost/\")\n\n returnResult <- REST$read(\"file\", uuid, \"text\", 1024, 512)\n\n expect_true(fakeHttp$URLIsProperlyConfigured)\n expect_true(fakeHttp$requestHeaderContainsAuthorizationField)\n expect_true(fakeHttp$requestHeaderContainsRangeField)\n expect_that(returnResult, equals(\"file content\"))\n})\n\ntest_that(\"read raises exception if server response code is not between 200 and 300\", {\n\n uuid <- \"aaaaa-j7d0g-ccccccccccccccc\"\n response <- list()\n response$status_code <- 404\n fakeHttp <- FakeHttpRequest$new(serverResponse = response)\n\n REST <- RESTService$new(\"token\", \"https://host/\",\n fakeHttp, HttpParser$new(),\n 0, \"https://webDavHost/\")\n\n expect_that(REST$read(\"file\", uuid),\n throws_error(\"Server code: 404\"))\n})\n\ntest_that(\"read raises exception if contentType is not valid\", {\n\n uuid <- \"aaaaa-j7d0g-ccccccccccccccc\"\n fakeHttp <- FakeHttpRequest$new()\n\n REST <- RESTService$new(\"token\", \"https://host/\",\n fakeHttp, HttpParser$new(),\n 0, \"https://webDavHost/\")\n\n expect_that(REST$read(\"file\", uuid, \"some invalid content type\"),\n throws_error(\"Invalid contentType. Please use text or raw.\"))\n})\n\ntest_that(\"read parses server response\", {\n\n uuid <- \"aaaaa-j7d0g-ccccccccccccccc\"\n fakeHttpParser <- FakeHttpParser$new()\n REST <- RESTService$new(\"token\", \"https://host/\",\n FakeHttpRequest$new(), fakeHttpParser,\n 0, \"https://webDavHost/\")\n\n REST$read(\"file\", uuid, \"text\", 1024, 512)\n\n expect_that(fakeHttpParser$parserCallCount, equals(1))\n})\n\ntest_that(\"write calls REST service properly\", {\n\n fileContent <- \"new file content\"\n uuid <- \"aaaaa-j7d0g-ccccccccccccccc\"\n expectedURL <- \"https://webDavHost/c=aaaaa-j7d0g-ccccccccccccccc/file\"\n fakeHttp <- FakeHttpRequest$new(expectedURL)\n\n REST <- RESTService$new(\"token\", \"https://host/\",\n fakeHttp, FakeHttpParser$new(),\n 0, \"https://webDavHost/\")\n\n REST$write(\"file\", uuid, fileContent, \"text/html\")\n\n expect_true(fakeHttp$URLIsProperlyConfigured)\n expect_true(fakeHttp$requestBodyIsProvided)\n expect_true(fakeHttp$requestHeaderContainsAuthorizationField)\n expect_true(fakeHttp$requestHeaderContainsContentTypeField)\n})\n\ntest_that(\"write raises exception if server response code is not between 200 and 300\", {\n\n uuid <- \"aaaaa-j7d0g-ccccccccccccccc\"\n fileContent <- \"new file content\"\n response <- list()\n response$status_code <- 404\n fakeHttp <- FakeHttpRequest$new(serverResponse = response)\n\n REST <- RESTService$new(\"token\", \"https://host/\",\n fakeHttp, HttpParser$new(),\n 0, \"https://webDavHost/\")\n\n expect_that(REST$write(\"file\", uuid, fileContent, \"text/html\"),\n throws_error(\"Server code: 404\"))\n})\n\ntest_that(\"getConnection calls REST service properly\", {\n uuid <- \"aaaaa-j7d0g-ccccccccccccccc\"\n fakeHttp <- FakeHttpRequest$new()\n\n REST <- RESTService$new(\"token\", \"https://host/\",\n fakeHttp, FakeHttpParser$new(),\n 0, \"https://webDavHost/\")\n\n REST$getConnection(\"file\", uuid, \"r\")\n\n expect_that(fakeHttp$numberOfgetConnectionCalls, equals(1))\n})\n" }, { "path": "contrib/R-sdk/tests/testthat/test-Subcollection.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\nsource(\"fakes/FakeRESTService.R\")\n\ncontext(\"Subcollection\")\n\ntest_that(\"getRelativePath returns path relative to the tree root\", {\n\n animal <- Subcollection$new(\"animal\")\n\n fish <- Subcollection$new(\"fish\")\n animal$add(fish)\n\n expect_that(animal$getRelativePath(), equals(\"animal\"))\n expect_that(fish$getRelativePath(), equals(\"animal/fish\"))\n})\n\ntest_that(paste(\"getFileListing by default returns sorted path of all files\",\n \"relative to the current subcollection\"), {\n\n animal <- Subcollection$new(\"animal\")\n fish <- Subcollection$new(\"fish\")\n shark <- ArvadosFile$new(\"shark\")\n blueFish <- ArvadosFile$new(\"blueFish\")\n\n animal$add(fish)\n fish$add(shark)\n fish$add(blueFish)\n\n result <- animal$getFileListing()\n\n #expect sorted array\n expectedResult <- c(\"animal/fish/blueFish\", \"animal/fish/shark\")\n\n resultsMatch <- length(expectedResult) == length(result) &&\n all(expectedResult == result)\n\n expect_true(resultsMatch)\n})\n\ntest_that(paste(\"getFileListing returns sorted names of all direct children\",\n \"if fullPath is set to FALSE\"), {\n\n animal <- Subcollection$new(\"animal\")\n fish <- Subcollection$new(\"fish\")\n shark <- ArvadosFile$new(\"shark\")\n dog <- ArvadosFile$new(\"dog\")\n\n animal$add(fish)\n animal$add(dog)\n fish$add(shark)\n\n result <- animal$getFileListing(fullPath = FALSE)\n expectedResult <- c(\"dog\", \"fish\")\n\n resultsMatch <- length(expectedResult) == length(result) &&\n all(expectedResult == result)\n\n expect_true(resultsMatch)\n})\n\ntest_that(\"add adds content to inside collection tree\", {\n\n animal <- Subcollection$new(\"animal\")\n fish <- Subcollection$new(\"fish\")\n dog <- ArvadosFile$new(\"dog\")\n\n animal$add(fish)\n animal$add(dog)\n\n animalContainsFish <- animal$get(\"fish\")$getName() == fish$getName()\n animalContainsDog <- animal$get(\"dog\")$getName() == dog$getName()\n\n expect_true(animalContainsFish)\n expect_true(animalContainsDog)\n})\n\ntest_that(\"add raises exception if content name is empty string\", {\n\n animal <- Subcollection$new(\"animal\")\n rootFolder <- Subcollection$new(\"\")\n\n expect_that(animal$add(rootFolder),\n throws_error(\"Content has invalid name.\", fixed = TRUE))\n})\n\ntest_that(paste(\"add raises exception if ArvadosFile/Subcollection\",\n \"with same name already exists in the subcollection\"), {\n\n animal <- Subcollection$new(\"animal\")\n fish <- Subcollection$new(\"fish\")\n secondFish <- Subcollection$new(\"fish\")\n thirdFish <- ArvadosFile$new(\"fish\")\n\n animal$add(fish)\n\n expect_that(animal$add(secondFish),\n throws_error(paste(\"Subcollection already contains ArvadosFile or\",\n \"Subcollection with same name.\"), fixed = TRUE))\n expect_that(animal$add(thirdFish),\n throws_error(paste(\"Subcollection already contains ArvadosFile or\",\n \"Subcollection with same name.\"), fixed = TRUE))\n})\n\ntest_that(paste(\"add raises exception if passed argument is\",\n \"not ArvadosFile or Subcollection\"), {\n\n animal <- Subcollection$new(\"animal\")\n number <- 10\n\n expect_that(animal$add(number),\n throws_error(paste(\"Expected AravodsFile or Subcollection object,\",\n \"got (numeric).\"), fixed = TRUE))\n})\n\ntest_that(paste(\"add post content to a REST service\",\n \"if subcollection belongs to a collection\"), {\n\n collectionContent <- c(\"animal\", \"animal/fish\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n\n collection <- Collection$new(api, \"myUUID\")\n animal <- collection$get(\"animal\")\n dog <- ArvadosFile$new(\"dog\")\n\n animal$add(dog)\n\n expect_that(fakeREST$createCallCount, equals(1))\n})\n\ntest_that(\"remove removes content from subcollection\", {\n\n animal <- Subcollection$new(\"animal\")\n fish <- Subcollection$new(\"fish\")\n\n animal$add(fish)\n animal$remove(\"fish\")\n\n returnValueAfterRemovalIsNull <- is.null(animal$get(\"fish\"))\n\n expect_true(returnValueAfterRemovalIsNull)\n})\n\ntest_that(paste(\"remove raises exception\",\n \"if content to remove doesn't exist in the subcollection\"), {\n\n animal <- Subcollection$new(\"animal\")\n\n expect_that(animal$remove(\"fish\"),\n throws_error(paste(\"Subcollection doesn't contains ArvadosFile\",\n \"or Subcollection with specified name.\")))\n})\n\ntest_that(\"remove raises exception if passed argument is not character vector\", {\n\n animal <- Subcollection$new(\"animal\")\n number <- 10\n\n expect_that(animal$remove(number),\n throws_error(paste(\"Expected character,\",\n \"got (numeric).\"), fixed = TRUE))\n})\n\ntest_that(paste(\"remove removes content from REST service\",\n \"if subcollection belongs to a collection\"), {\n\n collectionContent <- c(\"animal\", \"animal/fish\", \"animal/dog\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n animal <- collection$get(\"animal\")\n\n animal$remove(\"fish\")\n\n expect_that(fakeREST$deleteCallCount, equals(1))\n})\n\ntest_that(paste(\"get returns ArvadosFile or Subcollection\",\n \"if file or folder with given name exists\"), {\n\n animal <- Subcollection$new(\"animal\")\n fish <- Subcollection$new(\"fish\")\n dog <- ArvadosFile$new(\"dog\")\n\n animal$add(fish)\n animal$add(dog)\n\n returnedFish <- animal$get(\"fish\")\n returnedDog <- animal$get(\"dog\")\n\n returnedFishIsSubcollection <- \"Subcollection\" %in% class(returnedFish)\n returnedDogIsArvadosFile <- \"ArvadosFile\" %in% class(returnedDog)\n\n expect_true(returnedFishIsSubcollection)\n expect_that(returnedFish$getName(), equals(\"fish\"))\n\n expect_true(returnedDogIsArvadosFile)\n expect_that(returnedDog$getName(), equals(\"dog\"))\n})\n\ntest_that(paste(\"get returns NULL if file or folder\",\n \"with given name doesn't exists\"), {\n\n animal <- Subcollection$new(\"animal\")\n fish <- Subcollection$new(\"fish\")\n\n animal$add(fish)\n\n returnedDogIsNull <- is.null(animal$get(\"dog\"))\n\n expect_true(returnedDogIsNull)\n})\n\ntest_that(\"getFirst returns first child in the subcollection\", {\n\n animal <- Subcollection$new(\"animal\")\n fish <- Subcollection$new(\"fish\")\n\n animal$add(fish)\n\n expect_that(animal$getFirst()$getName(), equals(\"fish\"))\n})\n\ntest_that(\"getFirst returns NULL if subcollection contains no children\", {\n\n animal <- Subcollection$new(\"animal\")\n\n returnedElementIsNull <- is.null(animal$getFirst())\n\n expect_true(returnedElementIsNull)\n})\n\ntest_that(paste(\"setCollection by default sets collection\",\n \"filed of subcollection and all its children\"), {\n\n animal <- Subcollection$new(\"animal\")\n fish <- Subcollection$new(\"fish\")\n animal$add(fish)\n\n animal$setCollection(\"myCollection\")\n\n expect_that(animal$getCollection(), equals(\"myCollection\"))\n expect_that(fish$getCollection(), equals(\"myCollection\"))\n})\n\ntest_that(paste(\"setCollection sets collection filed of subcollection only\",\n \"if parameter setRecursively is set to FALSE\"), {\n\n animal <- Subcollection$new(\"animal\")\n fish <- Subcollection$new(\"fish\")\n animal$add(fish)\n\n animal$setCollection(\"myCollection\", setRecursively = FALSE)\n fishCollectionIsNull <- is.null(fish$getCollection())\n\n expect_that(animal$getCollection(), equals(\"myCollection\"))\n expect_true(fishCollectionIsNull)\n})\n\ntest_that(paste(\"move raises exception if subcollection\",\n \"doesn't belong to any collection\"), {\n\n animal <- Subcollection$new(\"animal\")\n\n expect_that(animal$move(\"new/location\"),\n throws_error(\"Subcollection doesn't belong to any collection\"))\n})\n\ntest_that(\"move raises exception if new location contains content with the same name\", {\n\n collectionContent <- c(\"animal\",\n \"animal/fish\",\n \"animal/dog\",\n \"animal/fish/shark\",\n \"fish\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n fish <- collection$get(\"animal/fish\")\n\n expect_that(fish$move(\"fish\"),\n throws_error(\"Destination already contains content with same name.\"))\n\n})\n\ntest_that(paste(\"move raises exception if newLocationInCollection\",\n \"parameter is invalid\"), {\n\n collectionContent <- c(\"animal\",\n \"animal/fish\",\n \"animal/dog\",\n \"animal/fish/shark\",\n \"ball\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n\n collection <- Collection$new(api, \"myUUID\")\n fish <- collection$get(\"animal/fish\")\n\n expect_that(fish$move(\"objects/dog\"),\n throws_error(\"Unable to get destination subcollection.\"))\n})\n\ntest_that(\"move moves subcollection inside collection tree\", {\n\n collectionContent <- c(\"animal\",\n \"animal/fish\",\n \"animal/dog\",\n \"animal/fish/shark\",\n \"ball\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n fish <- collection$get(\"animal/fish\")\n\n fish$move(\"fish\")\n fishIsNullOnOldLocation <- is.null(collection$get(\"animal/fish\"))\n fishExistsOnNewLocation <- !is.null(collection$get(\"fish\"))\n\n expect_true(fishIsNullOnOldLocation)\n expect_true(fishExistsOnNewLocation)\n})\n\ntest_that(paste(\"getSizeInBytes returns zero if subcollection\",\n \"is not part of a collection\"), {\n\n animal <- Subcollection$new(\"animal\")\n\n expect_that(animal$getSizeInBytes(), equals(0))\n})\n\ntest_that(paste(\"getSizeInBytes delegates size calculation\",\n \"to REST service class\"), {\n\n collectionContent <- c(\"animal\", \"animal/fish\")\n returnSize <- 100\n fakeREST <- FakeRESTService$new(collectionContent, returnSize)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n animal <- collection$get(\"animal\")\n\n resourceSize <- animal$getSizeInBytes()\n\n expect_that(resourceSize, equals(100))\n})\n\n#########################\ntest_that(paste(\"copy raises exception if subcollection\",\n \"doesn't belong to any collection\"), {\n\n animal <- Subcollection$new(\"animal\")\n\n expect_that(animal$copy(\"new/location\"),\n throws_error(\"Subcollection doesn't belong to any collection.\"))\n})\n\ntest_that(\"copy raises exception if new location contains content with the same name\", {\n\n collectionContent <- c(\"animal\",\n \"animal/fish\",\n \"animal/dog\",\n \"animal/fish/shark\",\n \"fish\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n fish <- collection$get(\"animal/fish\")\n\n expect_that(fish$copy(\"fish\"),\n throws_error(\"Destination already contains content with same name.\"))\n\n})\n\ntest_that(paste(\"copy raises exception if location parameter is invalid\"), {\n\n collectionContent <- c(\"animal\",\n \"animal/fish\",\n \"animal/dog\",\n \"animal/fish/shark\",\n \"ball\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n\n collection <- Collection$new(api, \"myUUID\")\n fish <- collection$get(\"animal/fish\")\n\n expect_that(fish$copy(\"objects/dog\"),\n throws_error(\"Unable to get destination subcollection.\"))\n})\n\ntest_that(\"copy copies subcollection inside collection tree\", {\n\n collectionContent <- c(\"animal\",\n \"animal/fish\",\n \"animal/dog\",\n \"animal/fish/shark\",\n \"ball\")\n fakeREST <- FakeRESTService$new(collectionContent)\n\n api <- Arvados$new(\"myToken\", \"myHostName\")\n api$setRESTService(fakeREST)\n collection <- Collection$new(api, \"myUUID\")\n fish <- collection$get(\"animal/fish\")\n\n fish$copy(\"fish\")\n fishExistsOnOldLocation <- !is.null(collection$get(\"animal/fish\"))\n fishExistsOnNewLocation <- !is.null(collection$get(\"fish\"))\n\n expect_true(fishExistsOnOldLocation)\n expect_true(fishExistsOnNewLocation)\n})\n\ntest_that(\"duplicate performs deep cloning of Subcollection\", {\n foo <- ArvadosFile$new(\"foo\")\n bar <- ArvadosFile$new(\"bar\")\n sub <- Subcollection$new(\"qux\")\n sub$add(foo)\n sub$add(bar)\n\n newSub1 <- sub$duplicate()\n newSub2 <- sub$duplicate(\"quux\")\n\n expect_that(newSub1$getFileListing(), equals(sub$getFileListing()))\n expect_that(sort(newSub2$getFileListing()), equals(c(\"quux/bar\", \"quux/foo\")))\n})\n" }, { "path": "contrib/R-sdk/tests/testthat/test-util.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\ncontext(\"Utility function\")\n\ntest_that(\"listAll always returns all resource items from server\", {\n\n serverResponseLimit <- 3\n itemsAvailable <- 8\n items <- list(\"collection1\", \"collection2\", \"collection3\", \"collection4\",\n \"collection5\", \"collection6\", \"collection7\", \"collection8\")\n\n testFunction <- function(offset, ...)\n {\n response <- list()\n response$items_available <- itemsAvailable\n\n maxIndex <- offset + serverResponseLimit\n lastElementIndex <- if(maxIndex < itemsAvailable) maxIndex else itemsAvailable\n\n response$items <- items[(offset + 1):lastElementIndex]\n response\n }\n\n result <- listAll(testFunction)\n\n expect_that(length(result), equals(8))\n})\n\ntest_that(\"trimFromStart trims string correctly if string starts with trimCharacters\", {\n\n sample <- \"./something/random\"\n trimCharacters <- \"./something/\"\n\n result <- trimFromStart(sample, trimCharacters)\n\n expect_that(result, equals(\"random\"))\n})\n\ntest_that(\"trimFromStart returns original string if string doesn't starts with trimCharacters\", {\n\n sample <- \"./something/random\"\n trimCharacters <- \"./nothing/\"\n\n result <- trimFromStart(sample, trimCharacters)\n\n expect_that(result, equals(\"./something/random\"))\n})\n\ntest_that(\"trimFromEnd trims string correctly if string ends with trimCharacters\", {\n\n sample <- \"./something/random\"\n trimCharacters <- \"/random\"\n\n result <- trimFromEnd(sample, trimCharacters)\n\n expect_that(result, equals(\"./something\"))\n})\n\ntest_that(\"trimFromEnd returns original string if string doesn't end with trimCharacters\", {\n\n sample <- \"./something/random\"\n trimCharacters <- \"specific\"\n\n result <- trimFromStart(sample, trimCharacters)\n\n expect_that(result, equals(\"./something/random\"))\n})\n\ntest_that(\"RListToPythonList converts nested R list to char representation of Python list\", {\n\n sample <- list(\"insert\", list(\"random\", list(\"text\")), list(\"here\"))\n\n result <- RListToPythonList(sample)\n resultWithSeparator <- RListToPythonList(sample, separator = \",+\")\n\n expect_that(result, equals(\"[\\\"insert\\\", [\\\"random\\\", \\\"text\\\"], \\\"here\\\"]\"))\n expect_that(resultWithSeparator,\n equals(\"[\\\"insert\\\",+[\\\"random\\\",+\\\"text\\\"],+\\\"here\\\"]\"))\n})\n\ntest_that(\"appendToStartIfNotExist appends characters to beginning of a string\", {\n\n sample <- \"New Year\"\n charactersToAppend <- \"Happy \"\n\n result <- appendToStartIfNotExist(sample, charactersToAppend)\n\n expect_that(result, equals(\"Happy New Year\"))\n})\n\ntest_that(paste(\"appendToStartIfNotExist returns original string if string\",\n \"doesn't start with specified characters\"), {\n\n sample <- \"Happy New Year\"\n charactersToAppend <- \"Happy\"\n\n result <- appendToStartIfNotExist(sample, charactersToAppend)\n\n expect_that(result, equals(\"Happy New Year\"))\n})\n\ntest_that(paste(\"splitToPathAndName splits relative path to file/folder\",\n \"name and rest of the path\"), {\n\n relativePath <- \"path/to/my/file.exe\"\n\n result <- splitToPathAndName( relativePath)\n\n expect_that(result$name, equals(\"file.exe\"))\n expect_that(result$path, equals(\"path/to/my\"))\n})\n" }, { "path": "contrib/R-sdk/tests/testthat.R", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\nlibrary(testthat)\nlibrary(ArvadosR)\n\ntest_check(\"ArvadosR\")\n" }, { "path": "contrib/README.md", "content": "## Arvados Client Contributions\n\n\n\nThis directory contains client libraries and tools that can be used with an Arvados cluster. Core components of Arvados are regularly tested together to ensure they work in concert. These tools receive less frequent testing before release. The Arvados team is happy to receive bug reports and contributions to help improve them. However, bugs in these components will never be considered release-critical.\n\n* `arvados-bootstrap`: Scripts to initialize an Arvados cluster with data\n\n* `arvbash`: Arvados utility functions for the bash shell\n\n* `java-sdk-v2`: Java client SDK for Arvados\n\n* `R-sdk`: R client SDK for Arvados\n" }, { "path": "contrib/arvados-bootstrap/LICENSE-2.0.txt", "content": "\n Apache License\n Version 2.0, January 2004\n http://www.apache.org/licenses/\n\n TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n 1. Definitions.\n\n \"License\" shall mean the terms and conditions for use, reproduction,\n and distribution as defined by Sections 1 through 9 of this document.\n\n \"Licensor\" shall mean the copyright owner or entity authorized by\n the copyright owner that is granting the License.\n\n \"Legal Entity\" shall mean the union of the acting entity and all\n other entities that control, are controlled by, or are under common\n control with that entity. For the purposes of this definition,\n \"control\" means (i) the power, direct or indirect, to cause the\n direction or management of such entity, whether by contract or\n otherwise, or (ii) ownership of fifty percent (50%) or more of the\n outstanding shares, or (iii) beneficial ownership of such entity.\n\n \"You\" (or \"Your\") shall mean an individual or Legal Entity\n exercising permissions granted by this License.\n\n \"Source\" form shall mean the preferred form for making modifications,\n including but not limited to software source code, documentation\n source, and configuration files.\n\n \"Object\" form shall mean any form resulting from mechanical\n transformation or translation of a Source form, including but\n not limited to compiled object code, generated documentation,\n and conversions to other media types.\n\n \"Work\" shall mean the work of authorship, whether in Source or\n Object form, made available under the License, as indicated by a\n copyright notice that is included in or attached to the work\n (an example is provided in the Appendix below).\n\n \"Derivative Works\" shall mean any work, whether in Source or Object\n form, that is based on (or derived from) the Work and for which the\n editorial revisions, annotations, elaborations, or other modifications\n represent, as a whole, an original work of authorship. For the purposes\n of this License, Derivative Works shall not include works that remain\n separable from, or merely link (or bind by name) to the interfaces of,\n the Work and Derivative Works thereof.\n\n \"Contribution\" shall mean any work of authorship, including\n the original version of the Work and any modifications or additions\n to that Work or Derivative Works thereof, that is intentionally\n submitted to Licensor for inclusion in the Work by the copyright owner\n or by an individual or Legal Entity authorized to submit on behalf of\n the copyright owner. For the purposes of this definition, \"submitted\"\n means any form of electronic, verbal, or written communication sent\n to the Licensor or its representatives, including but not limited to\n communication on electronic mailing lists, source code control systems,\n and issue tracking systems that are managed by, or on behalf of, the\n Licensor for the purpose of discussing and improving the Work, but\n excluding communication that is conspicuously marked or otherwise\n designated in writing by the copyright owner as \"Not a Contribution.\"\n\n \"Contributor\" shall mean Licensor and any individual or Legal Entity\n on behalf of whom a Contribution has been received by Licensor and\n subsequently incorporated within the Work.\n\n 2. Grant of Copyright License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n copyright license to reproduce, prepare Derivative Works of,\n publicly display, publicly perform, sublicense, and distribute the\n Work and such Derivative Works in Source or Object form.\n\n 3. Grant of Patent License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n (except as stated in this section) patent license to make, have made,\n use, offer to sell, sell, import, and otherwise transfer the Work,\n where such license applies only to those patent claims licensable\n by such Contributor that are necessarily infringed by their\n Contribution(s) alone or by combination of their Contribution(s)\n with the Work to which such Contribution(s) was submitted. If You\n institute patent litigation against any entity (including a\n cross-claim or counterclaim in a lawsuit) alleging that the Work\n or a Contribution incorporated within the Work constitutes direct\n or contributory patent infringement, then any patent licenses\n granted to You under this License for that Work shall terminate\n as of the date such litigation is filed.\n\n 4. Redistribution. You may reproduce and distribute copies of the\n Work or Derivative Works thereof in any medium, with or without\n modifications, and in Source or Object form, provided that You\n meet the following conditions:\n\n (a) You must give any other recipients of the Work or\n Derivative Works a copy of this License; and\n\n (b) You must cause any modified files to carry prominent notices\n stating that You changed the files; and\n\n (c) You must retain, in the Source form of any Derivative Works\n that You distribute, all copyright, patent, trademark, and\n attribution notices from the Source form of the Work,\n excluding those notices that do not pertain to any part of\n the Derivative Works; and\n\n (d) If the Work includes a \"NOTICE\" text file as part of its\n distribution, then any Derivative Works that You distribute must\n include a readable copy of the attribution notices contained\n within such NOTICE file, excluding those notices that do not\n pertain to any part of the Derivative Works, in at least one\n of the following places: within a NOTICE text file distributed\n as part of the Derivative Works; within the Source form or\n documentation, if provided along with the Derivative Works; or,\n within a display generated by the Derivative Works, if and\n wherever such third-party notices normally appear. The contents\n of the NOTICE file are for informational purposes only and\n do not modify the License. You may add Your own attribution\n notices within Derivative Works that You distribute, alongside\n or as an addendum to the NOTICE text from the Work, provided\n that such additional attribution notices cannot be construed\n as modifying the License.\n\n You may add Your own copyright statement to Your modifications and\n may provide additional or different license terms and conditions\n for use, reproduction, or distribution of Your modifications, or\n for any such Derivative Works as a whole, provided Your use,\n reproduction, and distribution of the Work otherwise complies with\n the conditions stated in this License.\n\n 5. Submission of Contributions. Unless You explicitly state otherwise,\n any Contribution intentionally submitted for inclusion in the Work\n by You to the Licensor shall be under the terms and conditions of\n this License, without any additional terms or conditions.\n Notwithstanding the above, nothing herein shall supersede or modify\n the terms of any separate license agreement you may have executed\n with Licensor regarding such Contributions.\n\n 6. Trademarks. This License does not grant permission to use the trade\n names, trademarks, service marks, or product names of the Licensor,\n except as required for reasonable and customary use in describing the\n origin of the Work and reproducing the content of the NOTICE file.\n\n 7. Disclaimer of Warranty. Unless required by applicable law or\n agreed to in writing, Licensor provides the Work (and each\n Contributor provides its Contributions) on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n implied, including, without limitation, any warranties or conditions\n of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n PARTICULAR PURPOSE. You are solely responsible for determining the\n appropriateness of using or redistributing the Work and assume any\n risks associated with Your exercise of permissions under this License.\n\n 8. Limitation of Liability. In no event and under no legal theory,\n whether in tort (including negligence), contract, or otherwise,\n unless required by applicable law (such as deliberate and grossly\n negligent acts) or agreed to in writing, shall any Contributor be\n liable to You for damages, including any direct, indirect, special,\n incidental, or consequential damages of any character arising as a\n result of this License or out of the use or inability to use the\n Work (including but not limited to damages for loss of goodwill,\n work stoppage, computer failure or malfunction, or any and all\n other commercial damages or losses), even if such Contributor\n has been advised of the possibility of such damages.\n\n 9. Accepting Warranty or Additional Liability. While redistributing\n the Work or Derivative Works thereof, You may choose to offer,\n and charge a fee for, acceptance of support, warranty, indemnity,\n or other liability obligations and/or rights consistent with this\n License. However, in accepting such obligations, You may act only\n on Your own behalf and on Your sole responsibility, not on behalf\n of any other Contributor, and only if You agree to indemnify,\n defend, and hold each Contributor harmless for any liability\n incurred by, or claims asserted against, such Contributor by reason\n of your accepting any such warranty or additional liability.\n\n END OF TERMS AND CONDITIONS\n\n APPENDIX: How to apply the Apache License to your work.\n\n To apply the Apache License to your work, attach the following\n boilerplate notice, with the fields enclosed by brackets \"[]\"\n replaced with your own identifying information. (Don't include\n the brackets!) The text should be enclosed in the appropriate\n comment syntax for the file format. We also recommend that a\n file or class name and description of purpose be included on the\n same \"printed page\" as the copyright notice for easier\n identification within third-party archives.\n\n Copyright [yyyy] [name of copyright owner]\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n" }, { "path": "contrib/arvados-bootstrap/README.md", "content": "# Arvados Bootstrap Tools\n\n\n\n## Introduction\n\nThis package provides scripts to initialize an Arvados cluster with data, built on top of the Python SDK. From inside this directory, you can install it by running:\n\n pipx install .\n\nor if you're managing your own virtualenvs and have one activated:\n\n pip install .\n\n## arv-export\n\n`arv-export` saves records from a running Arvados cluster to the directory where you run it. It finds Arvados credentials the same way arv-copy does, by reading `~/.config/arvados/ZZZZZ.conf`, where `ZZZZZ` is a five-alphanumeric cluster ID.\n\n`cd` to a directory where you want to save data and run:\n\n arv-export [other options] OBJECT_UUID\n\nThis will subdirectories inside the current directory with data from Arvados. You can load this data with `arv-import` as described in the next section.\n\n## arv-import\n\n`arv-import` creates records on an Arvados cluster from records previously saved by `arv-export`. It finds Arvados credentials the same way arv-copy does, by reading `~/.config/arvados/ZZZZZ.conf`, where `ZZZZZ` is a five-alphanumeric cluster ID.\n\n`cd` to a directory where you previously saved data with `arv-export` and run:\n\n arv-import [--project-uuid=UUID] [--no-block-copy] [other options] OBJECT_UUID\n\n`OBJECT_UUID` should match a UUID you exported with `arv-export`.\n\n### Using --no-block-copy\n\nIf you have administrator access to the destination cluster, then you have the option to write Keep blocks directly to the underlying storage and skip the normal upload using the `--no-block-copy` option. This is normally faster than uploading the blocks via HTTP, but you are entirely responsible for the separate data transfer. For example, if you use a standard filesystem-backed Keep volume, you might run:\n\n rsync -r arv-export-data/keep/ root@keep.xurid.example:/var/lib/arvados/keep-data/\n\nThe exact process will vary by Keep volume and system configuration. Documenting all the possibilities is outside the scope of this document.\n\n## arv-seed\n\n### Synopsis\n\narv-seed is a script to bulk create Arvados objects from JSON files.\n\n arv-seed [options] DIRECTORY [directory ...]\n\n### Configuration\n\nBy default, when running as root, this tool will read the cluster\nconfiguration file `$ARVADOS_CONFIG` (default `/etc/arvados/config.yml`),\nsearch for exactly one cluster configuration with a `Controller` endpoint and\n`SystemRootToken` configured, and use that.\n\nWhen running as a non-root user, this tool will search for user credentials\nthe same way as other Arvados command-line tools.\n\nYou can control how to load credentials using the `--client-from` option.\n\n### Input\n\nEach directory will be scanned for files named `NAME.TYPE.json`. `NAME` is any\nname you like. `TYPE` is the name of an Arvados API resource type, like `group`,\n`collection`, or `container_request`. `TYPE` can be spelled with any\npunctuation, use CamelCase or not, and be singular or plural.\n\nInput can be further controlled with \"base\" JSON that sets attributes for all\nobjects as well as additional parameters for the Arvados create method. Refer\nto the `--help` output for `--base-object` and `--parameters` for details.\n\n### Output\n\nWhen finished, the tool writes JSON output like this to stdout:\n\n {\n \"created\": {\"/path1\": {… Arvados object…}, …},\n \"failed\": {\"/path2\": \"error message\", …}\n }\n\nFor both `created` and `failed`, each key is the absolute path of a JSON file\nthat the tool read. For `created`, each value is the object that Arvados\nreturned after creation. For `failed`, each value is an error message that\ndescribes why no object could be created.\n\n### Logging\n\nThe tool always logs to syslog. It also logs to stderr if `$TERM` is set.\nControl what gets logged with the `--loglevel` option.\n\n### Exit codes\n\narv-seed uses the following exit codes:\n\n* 0: Created all objects successfully (at least one)\n* 1: Early internal error\n* 2: Incorrect command line arguments\n* 11: Created no objects successfully (at least one)\n* 12: Mixed results: some objects were created, others failed\n* 66: Did not find any JSON input files (`EX_NOINPUT`)\n* 70: Internal error (`EX_SOFTWARE`)\n* 78: Could not initialize from configuration (`EX_CONFIG`)\n\n### Example\n\nRead JSON from `~/arv-seed` and create them all in the given directory:\n\n arv-seed --base='{\"owner_uuid\":\"zzzzz-j7d0g-12345abcde67890\"}' ~/arv-seed\n\n### systemd service example\n\n [Unit]\n After=arvados-railsapi.service arvados-controller.service network-online.target\n\n [Service]\n Type=oneshot\n StandardOutput=file:%t/%N.json\n ExecStart=/opt/arvados-bootstrap/bin/arv-seed /usr/local/share/arv-seed\n\n## arv-federation-migrate\n\n### Introduction\n\nWhen using multiple Arvados clusters before a federation, a user would have to create a separate account on each cluster. Unfortunately, because each account represents a separate \"identity\", in this system permissions granted to a user on one cluster do not transfer to another cluster, even if the accounts are associated with the same user.\n\nTo address this, Arvados supports \"federated user accounts\". A federated user account is associated with a specific \"home\" cluster, and can be used to access other clusters in the federation that trust the home cluster. When a user arrives at another cluster's Workbench, they select and log in to their home cluster, and then are returned to the starting cluster logged in with the federated user account.\n\nWhen setting up federation capabilities on existing clusters, some users might already have accounts on multiple clusters. In order to have a single federated identity, users should be assigned a \"home\" cluster, and accounts associated with that user on the other (non-home) clusters should be migrated to the new federated user account. The @arv-federation-migrate@ tool assists with this.\n\nThis tool is designed to help an administrator who has access to all clusters in a federation to migrate users who have multiple accounts to a single federated account.\n\nAs part of migrating a user, any data or permissions associated with old user accounts will be reassigned to the federated account.\n\n### Step 1: Get a user report\n\n#### With a LoginCluster\n\nWhen using centralized user database as specified by `LoginCluster` in the config file.\n\nSet the `ARVADOS_API_HOST` and `ARVADOS_API_TOKEN` environment variables to be an admin user on cluster in `LoginCluster` . It will automatically determine the other clusters that are listed in the federation.\n\nNext, run `arv-federation-migrate` with the `--report` flag:\n\n $ arv-federation-migrate --report users.csv\n Getting user list from x6b1s\n Getting user list from x3982\n Wrote users.csv\n\n#### Without a LoginCluster\n\nThe first step is to create `tokens.csv` and list each cluster and API token to access the cluster. API tokens must be trusted tokens with administrator access. This is a simple comma separated value file and can be created in a text editor. Example:\n\n x3982.arvadosapi.com,v2/x3982-gj3su-sb6meh2jf145s7x/98d40d70d8862e33d7398213435d1a71a96cf870\n x6b1s.arvadosapi.com,v2/x6b1s-gj3su-dxc87btfv5kg91z/5575d980d3ff6231bb0c692281c42a7541c59417\n\nNext, run `arv-federation-migrate` with the `--tokens` and `--report` flags:\n\n $ arv-federation-migrate --tokens tokens.csv --report users.csv\n Reading tokens.csv\n Getting user list from x6b1s\n Getting user list from x3982\n Wrote users.csv\n\n### Step 2: Update the user report\n\nThis will produce a report of users across all clusters listed in `tokens.csv`, sorted by email address. This file can be loaded into a text editor or spreadsheet program for ease of viewing and editing.\n\n email,username,user uuid,primary cluster/user\n person_a@example.com,person_a,x6b1s-tpzed-hb5n7doogwhk6cf,x6b1s\n person_b@example.com,person_b,x3982-tpzed-1vl3k7knf7qihbe,\n person_b@example.com,person_b,x6b1s-tpzed-w4nhkx2rmrhlr54,\n\nThe fourth column describes that user's home cluster. If a user only has one account (identified by email address), the column will be filled in and there is nothing to do. If the column is blank, that means there is more than one Arvados account associated with the user. Edit the file and provide the desired home cluster for each user as necessary (note: if there is a LoginCluster, all users will be migrated to the LoginCluster). It is also possible to change the desired username for a user. In this example, `person_b@example.com` is assigned the home cluster `x3982`.\n\n email,username,user uuid,primary cluster/user\n person_a@example.com,person_a,x6b1s-tpzed-hb5n7doogwhk6cf,x6b1s\n person_b@example.com,person_b,x3982-tpzed-1vl3k7knf7qihbe,x3982\n person_b@example.com,person_b,x6b1s-tpzed-w4nhkx2rmrhlr54,x3982\n\n### Step 3: Migrate users\n\nTo avoid disruption, advise users to log out and avoid running workflows while performing the migration.\n\nAfter updating `users.csv`, you can preview the migration using the `--dry-run` option (add `--tokens tokens.csv` if not using LoginCluster). This will print out what actions the migration will take (as if it were happening) and report possible problems, but not make any actual changes on any cluster:\n\n $ arv-federation-migrate --dry-run users.csv\n (person_b@example.com) Migrating x6b1s-tpzed-w4nhkx2rmrhlr54 to x3982-tpzed-1vl3k7knf7qihbe\n\nExecute the migration using the `--migrate` option (add `--tokens tokens.csv` if not using LoginCluster):\n\n $ arv-federation-migrate --migrate users.csv\n (person_b@example.com) Migrating x6b1s-tpzed-w4nhkx2rmrhlr54 to x3982-tpzed-1vl3k7knf7qihbe\n\nAfter migration, users should select their home cluster when logging into Arvados Workbench. If a user attempts to log into a migrated user account, they will be redirected to log in with their home cluster.\n" }, { "path": "contrib/arvados-bootstrap/pyproject.toml", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\n[build-system]\nrequires = [\"setuptools ~= 80.9\"]\nbuild-backend = \"setuptools.build_meta\"\n\n[project]\nname = \"arvados-bootstrap\"\nversion = \"3.2.1\"\ndependencies = [\n \"arvados-python-client == 3.2.1\",\n]\ndescription = \"Tools to bootstrap an Arvados cluster\"\nauthors = [\n {name = \"Arvados\", email = \"info@arvados.org\"},\n]\nclassifiers = [\n \"Development Status :: 4 - Beta\",\n \"Environment :: Console\",\n \"Intended Audience :: Science/Research\",\n \"Operating System :: POSIX\",\n \"Programming Language :: Python :: 3\",\n \"Programming Language :: Python :: 3.10\",\n \"Programming Language :: Python :: 3.11\",\n \"Programming Language :: Python :: 3.12\",\n \"Programming Language :: Python :: 3.13\",\n]\nlicense = \"Apache-2.0\"\nlicense-files = [\n \"LICENSE-2.0.txt\",\n]\nreadme = \"README.md\"\nrequires-python = \"~= 3.10\"\n\n[project.scripts]\narv-export = \"arv_bootstrap.export_import:export_main\"\narv-federation-migrate = \"arv_bootstrap.federation_migrate:main\"\narv-import = \"arv_bootstrap.export_import:import_main\"\narv-seed = \"arv_bootstrap.seed:main\"\n\n[project.urls]\nHomepage = \"https://arvados.org\"\nDocumentation = \"https://doc.arvados.org\"\nRepository = \"https://github.com/arvados/arvados\"\nIssues = \"https://github.com/arvados/arvados/issues\"\nChangelog = \"https://arvados.org/releases/\"\n\n[tool.setuptools.packages.find]\nwhere = [\"src\"]\n" }, { "path": "contrib/arvados-bootstrap/src/arv_bootstrap/__init__.py", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n" }, { "path": "contrib/arvados-bootstrap/src/arv_bootstrap/export_import.py", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\nimport argparse\nimport importlib.metadata\nimport logging\nimport os\nimport re\nimport sys\n\nimport arvados.commands.arv_copy as arv_copy\nimport arvados.commands._util as arv_cmd\nimport arvados.util as arv_util\n\nfrom . import stubapi\n\nlogger = logging.getLogger('arvados.arv-export-import')\n\nclass ArgumentParser(argparse.ArgumentParser):\n @classmethod\n def _base_options(cls, cmdname=sys.argv[0]):\n opts = argparse.ArgumentParser(add_help=False)\n opts.add_argument(\n '--version',\n action='version',\n version=f'{cmdname} {importlib.metadata.version(\"arvados-bootstrap\")}',\n help='Print version and exit.',\n )\n opts.add_argument(\n '--verbose', '-v',\n dest='verbose',\n action='store_true',\n help='Verbose output.',\n )\n return opts\n\n @classmethod\n def _common_options(cls, verb):\n opts = cls._base_options(f'arv-{verb}')\n opts.add_argument(\n '--force', '-f',\n action='store_true',\n help=f\"\"\"{verb.capitalize()} even if the object has already been {verb}ed.\n\"\"\")\n opts.add_argument(\n '--recursive',\n action='store_true',\n help=f\"\"\"Recursively {verb} any dependencies for this object\nand subprojects. (default)\n\"\"\")\n opts.add_argument(\n '--no-recursive',\n dest='recursive',\n action='store_false',\n help=f\"\"\"Do not {verb} any dependencies or subprojects.\n\"\"\")\n\n opts.add_argument(\n '--block-copy',\n dest='keep_block_copy',\n action='store_true',\n help=f\"\"\"Copy Keep blocks when {verb}ing collections. (default)\n\"\"\")\n opts.add_argument(\n '--no-block-copy',\n dest='keep_block_copy',\n action='store_false',\n help=f\"\"\"Do not copy Keep blocks when {verb}ing collections.\nMust have administrator privileges to import collections.\n\"\"\")\n opts.add_argument(\n 'object_uuid',\n help=f\"\"\"The UUID of the collection or project to {verb}.\n\"\"\")\n return opts\n\n @classmethod\n def _import_options(cls):\n opts = cls._common_options('import')\n opts.add_argument(\n '--project-uuid',\n help=\"\"\"The UUID of the project at the destination to which the\ncollection or project should be imported.\n\"\"\")\n opts.add_argument(\n '--storage-classes',\n type=arv_cmd.UniqueSplit(),\n help=\"\"\"Comma-separated list of storage classes to be used when\nsaving data to the destinaton Arvados instance.\n\"\"\")\n opts.add_argument(\n '--replication',\n type=arv_cmd.RangedValue(int, range(1, sys.maxsize)),\n metavar='N',\n help=\"\"\"\nNumber of replicas per storage class for the copied collections at the destination.\nIf not provided (or if provided with invalid value),\nuse the destination's default replication-level setting (if found),\nor the fallback value 2.\n\"\"\")\n return opts\n\n def _set_common_defaults(self):\n self.set_defaults(\n # Common defaults should use the \"safer\" value.\n export_all_fields=False,\n force=False,\n keep_block_copy=True,\n prefer_cached_downloads=False,\n project_uuid=None,\n progress=None,\n recursive=True,\n varying_url_params=\"\",\n )\n\n @classmethod\n def export_parser(cls):\n parser = cls(\n description=f\"Export Arvados objects to a local filesystem\",\n parents=[cls._common_options('export'), arv_cmd.retry_opt],\n )\n parser._set_common_defaults()\n parser.set_defaults(\n export_all_fields=True,\n progress=True,\n replication=1,\n storage_classes=[],\n )\n return parser\n\n @classmethod\n def import_parser(cls):\n parser = cls(\n description=f\"Import Arvados objects from a local filesystem\",\n parents=[cls._import_options(), arv_cmd.retry_opt],\n )\n parser._set_common_defaults()\n return parser\n\n\ndef setup_logging(name, args):\n global logger\n arvlogger = logging.getLogger('arvados')\n logger = arvlogger.getChild(name)\n if args.verbose:\n arvlogger.setLevel(logging.DEBUG)\n else:\n arvlogger.setLevel(logging.INFO)\n arvlogger.getChild('keep').setLevel(logging.WARNING)\n\n\ndef transfer(src_arv, dst_arv, args, verb):\n if re.match(arv_util.collection_uuid_pattern, args.object_uuid):\n result = arv_copy.copy_collection(args.object_uuid, src_arv, dst_arv, args)\n elif re.match(arv_util.group_uuid_pattern, args.object_uuid):\n result = arv_copy.copy_project(args.object_uuid, src_arv, dst_arv, args.project_uuid, args)\n else:\n logger.error(\"Unsupported object type for %s: %s\", verb, args.object_uuid)\n return os.EX_DATAERR\n if error := result.get('partial_error'):\n logger.error(\n \"Error copying %s: %s\",\n args.object_uuid,\n result if logger.isEnabledFor(logging.DEBUG) else error,\n )\n return os.EX_IOERR\n return os.EX_OK\n\n\ndef export_main(arglist=None):\n args = ArgumentParser.export_parser().parse_args(arglist)\n setup_logging('arv-export', args)\n src_arv = arv_copy.api_for_instance(args.object_uuid[:5], args.retries)\n dst_arv = stubapi.StubArvadosAPI.for_cwd()\n return transfer(src_arv, dst_arv, args, 'export')\n\n\ndef import_main(arglist=None):\n args = ArgumentParser.import_parser().parse_args(arglist)\n setup_logging('arv-import', args)\n src_arv = stubapi.StubArvadosAPI.for_cwd()\n try:\n dst_id = args.project_uuid[:5]\n except TypeError:\n dst_id = ''\n dst_arv = arv_copy.api_for_instance(dst_id, args.retries)\n if args.project_uuid is None:\n args.project_uuid = dst_arv.users().current().execute()['uuid']\n if args.replication is None:\n try:\n args.replication = int(dst_arv.config()[\"Collections\"][\"DefaultReplication\"])\n except (KeyError, TypeError, ValueError):\n args.replication = 2\n return transfer(src_arv, dst_arv, args, 'import')\n" }, { "path": "contrib/arvados-bootstrap/src/arv_bootstrap/federation_migrate.py", "content": "#!/usr/bin/env python3\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\n# Migration tool for merging user accounts belonging to the same user\n# but on separate clusters to use a single user account managed by a\n# specific cluster.\n\nimport argparse\nimport csv\nimport hashlib\nimport hmac\nimport importlib.metadata\nimport os\nimport re\nimport sys\nimport urllib.parse\n\nimport arvados\nimport arvados.commands._util as arv_cmd\nimport arvados.util\nimport arvados.errors\n\nEMAIL=0\nUSERNAME=1\nUUID=2\nHOMECLUSTER=3\n\ndef connect_clusters(args):\n clusters = {}\n errors = []\n loginCluster = None\n if args.tokens:\n print(\"Reading %s\" % args.tokens)\n with open(args.tokens, \"rt\") as f:\n for r in csv.reader(f):\n if len(r) != 2:\n continue\n host = r[0]\n token = r[1]\n print(\"Contacting %s\" % (host))\n arv = arvados.api(host=host, token=token, cache=False, num_retries=args.retries)\n clusters[arv._rootDesc[\"uuidPrefix\"]] = arv\n else:\n arv = arvados.api(cache=False, num_retries=args.retries)\n rh = arv._rootDesc[\"remoteHosts\"]\n tok = arv.api_client_authorizations().current().execute()\n token = \"v2/%s/%s\" % (tok[\"uuid\"], tok[\"api_token\"])\n\n for k,v in rh.items():\n arv = arvados.api(host=v, token=token, cache=False, insecure=os.environ.get(\"ARVADOS_API_HOST_INSECURE\"))\n clusters[k] = arv\n\n for _, arv in clusters.items():\n config = arv.configs().get().execute()\n if config[\"Login\"][\"LoginCluster\"] != \"\" and loginCluster is None:\n loginCluster = config[\"Login\"][\"LoginCluster\"]\n\n print(\"Checking that the federation is well connected\")\n for arv in clusters.values():\n config = arv.configs().get().execute()\n if loginCluster and config[\"Login\"][\"LoginCluster\"] != loginCluster and config[\"ClusterID\"] != loginCluster:\n errors.append(\"Inconsistent login cluster configuration, expected '%s' on %s but was '%s'\" % (loginCluster, config[\"ClusterID\"], config[\"Login\"][\"LoginCluster\"]))\n continue\n\n if arv._rootDesc[\"revision\"] < \"20200331\":\n errors.append(\"Arvados API server revision on cluster '%s' is too old, must be updated to at least Arvados 2.0.2 before running migration.\" % config[\"ClusterID\"])\n continue\n\n try:\n cur = arv.users().current().execute()\n except arvados.errors.ApiError as e:\n errors.append(\"checking token for %s %s\" % (arv._rootDesc[\"rootUrl\"], e))\n continue\n\n if not cur[\"is_admin\"]:\n errors.append(\"User %s is not admin on %s\" % (cur[\"uuid\"], arv._rootDesc[\"uuidPrefix\"]))\n continue\n\n for r in clusters:\n if r != arv._rootDesc[\"uuidPrefix\"] and r not in arv._rootDesc[\"remoteHosts\"]:\n errors.append(\"%s is missing from remoteHosts of %s\" % (r, arv._rootDesc[\"uuidPrefix\"]))\n for r in arv._rootDesc[\"remoteHosts\"]:\n if r != \"*\" and r not in clusters:\n print(\"WARNING: %s is federated with %s but %s is missing from the tokens file or the token is invalid\" % (arv._rootDesc[\"uuidPrefix\"], r, r))\n\n return clusters, errors, loginCluster\n\n\ndef fetch_users(clusters, loginCluster):\n rows = []\n by_email = {}\n by_username = {}\n\n users = [\n user\n for prefix, arv in clusters.items()\n for user in arvados.util.keyset_list_all(arv.users().list, bypass_federation=True)\n if user['uuid'].startswith(prefix)\n ]\n\n # Users list is sorted by email\n # Go through users and collect users with same email\n # when we see a different email (or get to the end)\n # call add_accum_rows() to generate the report rows with\n # the \"home cluster\" set, and also fill in the by_email table.\n\n users.sort(key=lambda u: (u[\"email\"], u[\"username\"] or \"\", u[\"uuid\"]))\n\n accum = []\n lastemail = None\n\n def add_accum_rows():\n homeuuid = None\n for a in accum:\n uuids = set(a[\"uuid\"] for a in accum)\n homeuuid = ((len(uuids) == 1) and uuids.pop()) or \"\"\n for a in accum:\n r = (a[\"email\"], a[\"username\"], a[\"uuid\"], loginCluster or homeuuid[0:5])\n by_email.setdefault(a[\"email\"], {})\n by_email[a[\"email\"]][a[\"uuid\"]] = r\n homeuuid_and_username = \"%s::%s\" % (r[HOMECLUSTER], a[\"username\"])\n if homeuuid_and_username not in by_username:\n by_username[homeuuid_and_username] = a[\"email\"]\n elif by_username[homeuuid_and_username] != a[\"email\"]:\n print(\"ERROR: the username '%s' is listed for both '%s' and '%s' on cluster '%s'\" % (r[USERNAME], r[EMAIL], by_username[homeuuid_and_username], r[HOMECLUSTER]))\n exit(1)\n rows.append(r)\n\n for u in users:\n if u[\"uuid\"].endswith(\"-anonymouspublic\") or u[\"uuid\"].endswith(\"-000000000000000\"):\n continue\n if lastemail == None:\n lastemail = u[\"email\"]\n if u[\"email\"] == lastemail:\n accum.append(u)\n else:\n add_accum_rows()\n lastemail = u[\"email\"]\n accum = [u]\n\n add_accum_rows()\n\n return rows, by_email, by_username\n\n\ndef read_migrations(args, by_email, by_username):\n rows = []\n with open(args.migrate or args.dry_run, \"rt\") as f:\n for r in csv.reader(f):\n if r[EMAIL] == \"email\":\n continue\n by_email.setdefault(r[EMAIL], {})\n by_email[r[EMAIL]][r[UUID]] = r\n\n homeuuid_and_username = \"%s::%s\" % (r[HOMECLUSTER], r[USERNAME])\n if homeuuid_and_username not in by_username:\n by_username[homeuuid_and_username] = r[EMAIL]\n elif by_username[homeuuid_and_username] != r[EMAIL]:\n print(\"ERROR: the username '%s' is listed for both '%s' and '%s' on cluster '%s'\" % (r[USERNAME], r[EMAIL], by_username[homeuuid_and_username], r[HOMECLUSTER]))\n exit(1)\n\n rows.append(r)\n return rows\n\ndef update_username(args, email, user_uuid, username, migratecluster, migratearv):\n print(\"(%s) Updating username of %s to '%s' on %s\" % (email, user_uuid, username, migratecluster))\n if args.dry_run:\n return\n try:\n conflicts = migratearv.users().list(filters=[[\"username\", \"=\", username]], bypass_federation=True).execute()\n if conflicts[\"items\"]:\n # There's already a user with the username, move the old user out of the way\n migratearv.users().update(uuid=conflicts[\"items\"][0][\"uuid\"],\n bypass_federation=True,\n body={\"user\": {\"username\": username+\"migrate\"}}).execute()\n migratearv.users().update(uuid=user_uuid,\n bypass_federation=True,\n body={\"user\": {\"username\": username}}).execute()\n except arvados.errors.ApiError as e:\n print(\"(%s) Error updating username of %s to '%s' on %s: %s\" % (email, user_uuid, username, migratecluster, e))\n\n\ndef choose_new_user(args, by_email, email, userhome, username, old_user_uuid, clusters):\n candidates = []\n conflict = False\n for b in by_email[email].values():\n if b[2].startswith(userhome):\n candidates.append(b)\n if b[1] != username and b[3] == userhome:\n print(\"(%s) Cannot migrate %s, conflicting usernames %s and %s\" % (email, old_user_uuid, b[1], username))\n conflict = True\n break\n if conflict:\n return None\n if len(candidates) == 0:\n if len(userhome) == 5 and userhome not in clusters:\n print(\"(%s) Cannot migrate %s, unknown home cluster %s (typo?)\" % (email, old_user_uuid, userhome))\n return None\n print(\"(%s) No user listed with same email to migrate %s to %s, will create new user with username '%s'\" % (email, old_user_uuid, userhome, username))\n if not args.dry_run:\n oldhomecluster = old_user_uuid[0:5]\n oldhomearv = clusters[oldhomecluster]\n newhomecluster = userhome[0:5]\n homearv = clusters[userhome]\n user = None\n try:\n olduser = oldhomearv.users().get(uuid=old_user_uuid).execute()\n conflicts = homearv.users().list(filters=[[\"username\", \"=\", username]],\n bypass_federation=True).execute()\n if conflicts[\"items\"]:\n homearv.users().update(\n uuid=conflicts[\"items\"][0][\"uuid\"],\n bypass_federation=True,\n body={\"user\": {\"username\": username+\"migrate\"}}).execute()\n user = homearv.users().create(\n body={\"user\": {\n \"email\": email,\n \"first_name\": olduser[\"first_name\"],\n \"last_name\": olduser[\"last_name\"],\n \"username\": username,\n \"is_active\": olduser[\"is_active\"]}}).execute()\n except arvados.errors.ApiError as e:\n print(\"(%s) Could not create user: %s\" % (email, str(e)))\n return None\n\n tup = (email, username, user[\"uuid\"], userhome)\n else:\n # dry run\n tup = (email, username, \"%s-tpzed-xfakexfakexfake\" % (userhome[0:5]), userhome)\n by_email[email][tup[2]] = tup\n candidates.append(tup)\n if len(candidates) > 1:\n print(\"(%s) Multiple users listed to migrate %s to %s, use full uuid\" % (email, old_user_uuid, userhome))\n return None\n return candidates[0][2]\n\n\ndef activate_remote_user(args, email, homearv, migratearv, old_user_uuid, new_user_uuid):\n # create a token for the new user and salt it for the\n # migration cluster, then use it to access the migration\n # cluster as the new user once before merging to ensure\n # the new user is known on that cluster.\n migratecluster = migratearv._rootDesc[\"uuidPrefix\"]\n try:\n if not args.dry_run:\n newtok = homearv.api_client_authorizations().create(body={\n \"api_client_authorization\": {'owner_uuid': new_user_uuid}}).execute()\n else:\n newtok = {\"uuid\": \"dry-run\", \"api_token\": \"12345\"}\n except arvados.errors.ApiError as e:\n print(\"(%s) Could not create API token for %s: %s\" % (email, new_user_uuid, e))\n return None\n\n try:\n findolduser = migratearv.users().list(filters=[[\"uuid\", \"=\", old_user_uuid]], bypass_federation=True).execute()\n if len(findolduser[\"items\"]) == 0:\n return False\n if len(findolduser[\"items\"]) == 1:\n olduser = findolduser[\"items\"][0]\n else:\n print(\"(%s) Unexpected result\" % (email))\n return None\n except arvados.errors.ApiError as e:\n print(\"(%s) Could not retrieve user %s from %s, user may have already been migrated: %s\" % (email, old_user_uuid, migratecluster, e))\n return None\n\n salted = 'v2/' + newtok[\"uuid\"] + '/' + hmac.new(newtok[\"api_token\"].encode(),\n msg=migratecluster.encode(),\n digestmod=hashlib.sha1).hexdigest()\n try:\n ru = urllib.parse.urlparse(migratearv._rootDesc[\"rootUrl\"])\n if not args.dry_run:\n newuser = arvados.api(host=ru.netloc, token=salted,\n insecure=os.environ.get(\"ARVADOS_API_HOST_INSECURE\")).users().current().execute()\n else:\n newuser = {\"is_active\": True, \"username\": email.split('@')[0], \"is_admin\": False}\n except arvados.errors.ApiError as e:\n print(\"(%s) Error getting user info for %s from %s: %s\" % (email, new_user_uuid, migratecluster, e))\n return None\n\n if not newuser[\"is_active\"] and olduser[\"is_active\"]:\n print(\"(%s) Activating user %s on %s\" % (email, new_user_uuid, migratecluster))\n try:\n if not args.dry_run:\n migratearv.users().update(uuid=new_user_uuid, bypass_federation=True,\n body={\"is_active\": True}).execute()\n except arvados.errors.ApiError as e:\n print(\"(%s) Could not activate user %s on %s: %s\" % (email, new_user_uuid, migratecluster, e))\n return None\n\n if olduser[\"is_admin\"] and not newuser[\"is_admin\"]:\n print(\"(%s) Not migrating %s because user is admin but target user %s is not admin on %s. Please ensure the user admin status is the same on both clusters. Note that a federated admin account has admin privileges on the entire federation.\" % (email, old_user_uuid, new_user_uuid, migratecluster))\n return None\n\n return newuser\n\ndef migrate_user(args, migratearv, email, new_user_uuid, old_user_uuid):\n if args.dry_run:\n return\n try:\n new_owner_uuid = new_user_uuid\n if args.data_into_subproject:\n grp = migratearv.groups().create(body={\n \"owner_uuid\": new_user_uuid,\n \"name\": \"Migrated from %s (%s)\" % (email, old_user_uuid),\n \"group_class\": \"project\"\n }, ensure_unique_name=True).execute()\n new_owner_uuid = grp[\"uuid\"]\n migratearv.users().merge(old_user_uuid=old_user_uuid,\n new_user_uuid=new_user_uuid,\n new_owner_uuid=new_owner_uuid,\n redirect_to_new_user=True).execute()\n except arvados.errors.ApiError as e:\n name_collision = re.search(r'Key \\(owner_uuid, name\\)=\\((.*?), (.*?)\\) already exists\\.\\n.*UPDATE \"(.*?)\"', e._get_reason())\n if name_collision:\n target_owner, rsc_name, rsc_type = name_collision.groups()\n print(\"(%s) Cannot migrate to %s because both origin and target users have a %s named '%s'. Please rename the conflicting items or use --data-into-subproject to migrate all users' data into a special subproject.\" % (email, target_owner, rsc_type[:-1], rsc_name))\n else:\n print(\"(%s) Skipping user migration because of error: %s\" % (email, e))\n\n\ndef main():\n parser = argparse.ArgumentParser(\n description='Migrate users to federated identity, see https://doc.arvados.org/admin/merge-remote-account.html',\n parents=[arv_cmd.retry_opt],\n )\n parser.add_argument(\n '--version',\n action='version',\n version=f\"{sys.argv[0]} {importlib.metadata.version('arvados-bootstrap')}\",\n help='Print version and exit.')\n parser.add_argument('--tokens', type=str, metavar='FILE', required=False, help=\"Read tokens from FILE. Not needed when using LoginCluster.\")\n parser.add_argument('--data-into-subproject', action=\"store_true\", help=\"Migrate user's data into a separate subproject. This can be used to avoid name collisions from within an account.\")\n group = parser.add_mutually_exclusive_group(required=True)\n group.add_argument('--report', type=str, metavar='FILE', help=\"Generate report .csv file listing users by email address and their associated Arvados accounts.\")\n group.add_argument('--migrate', type=str, metavar='FILE', help=\"Consume report .csv and migrate users to designated Arvados accounts.\")\n group.add_argument('--dry-run', type=str, metavar='FILE', help=\"Consume report .csv and report how user would be migrated to designated Arvados accounts.\")\n group.add_argument('--check', action=\"store_true\", help=\"Check that tokens are usable and the federation is well connected.\")\n args = parser.parse_args()\n\n clusters, errors, loginCluster = connect_clusters(args)\n\n if errors:\n for e in errors:\n print(\"ERROR: \"+str(e))\n exit(1)\n\n if args.check:\n print(\"Tokens file passed checks\")\n exit(0)\n\n rows, by_email, by_username = fetch_users(clusters, loginCluster)\n\n if args.report:\n out = csv.writer(open(args.report, \"wt\"))\n out.writerow((\"email\", \"username\", \"user uuid\", \"home cluster\"))\n for r in rows:\n out.writerow(r)\n print(\"Wrote %s\" % args.report)\n return\n\n if args.migrate or args.dry_run:\n if args.dry_run:\n print(\"Performing dry run\")\n\n rows = read_migrations(args, by_email, by_username)\n\n for r in rows:\n email = r[EMAIL]\n username = r[USERNAME]\n old_user_uuid = r[UUID]\n userhome = r[HOMECLUSTER]\n\n if userhome == \"\":\n print(\"(%s) Skipping %s, no home cluster specified\" % (email, old_user_uuid))\n if old_user_uuid.startswith(userhome):\n migratecluster = old_user_uuid[0:5]\n migratearv = clusters[migratecluster]\n if migratearv.users().get(uuid=old_user_uuid).execute()[\"username\"] != username:\n update_username(args, email, old_user_uuid, username, migratecluster, migratearv)\n continue\n\n new_user_uuid = choose_new_user(args, by_email, email, userhome, username, old_user_uuid, clusters)\n if new_user_uuid is None:\n continue\n\n remote_users = {}\n got_error = False\n for migratecluster in clusters:\n # cluster where the migration is happening\n migratearv = clusters[migratecluster]\n\n # the user's new home cluster\n newhomecluster = userhome[0:5]\n homearv = clusters[newhomecluster]\n\n newuser = activate_remote_user(args, email, homearv, migratearv, old_user_uuid, new_user_uuid)\n if newuser is None:\n got_error = True\n remote_users[migratecluster] = newuser\n\n if not got_error:\n for migratecluster in clusters:\n migratearv = clusters[migratecluster]\n newuser = remote_users[migratecluster]\n if newuser is False:\n continue\n\n print(\"(%s) Migrating %s to %s on %s\" % (email, old_user_uuid, new_user_uuid, migratecluster))\n\n migrate_user(args, migratearv, email, new_user_uuid, old_user_uuid)\n\n if newuser['username'] != username:\n update_username(args, email, new_user_uuid, username, migratecluster, migratearv)\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "contrib/arvados-bootstrap/src/arv_bootstrap/seed.py", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\nimport argparse\nimport contextlib\nimport dataclasses\nimport functools\nimport json\nimport logging\nimport logging.handlers\nimport pathlib\nimport os\nimport re\nimport sys\nimport traceback\nimport urllib.parse\n\nfrom collections import abc\n\nimport arvados\nimport arvados.commands._util as cmd_util\n\nlogger = logging.getLogger('arvados.commands.seed')\n_root_logger = logging.getLogger()\n\ndef is_mapping(arg):\n return isinstance(arg, abc.Mapping)\n\n\n@dataclasses.dataclass\nclass ExceptHook:\n logger: logging.Logger\n exit_code: int = os.EX_SOFTWARE\n\n def __call__(self, exc_type, exc_value, exc_tb):\n self.logger.critical(\n \"internal %s: %s\", exc_type.__name__, exc_value,\n exc_info=self.logger.isEnabledFor(logging.DEBUG),\n )\n raise SystemExit(self.exit_code)\n\n @contextlib.contextmanager\n def using_exit_code(self, exit_code):\n orig_code = self.exit_code\n self.exit_code = exit_code\n # We intentionally *don't* want to use `finally` here, because we don't\n # want to restore the original code for an unhandled exception.\n yield self\n self.exit_code = orig_code\n\n\nclass Path(pathlib.Path):\n _flavour = pathlib._posix_flavour\n\n def __format__(self, format_spec=''):\n if format_spec.endswith('`'):\n return f\"`{super().__format__(format_spec[:-1])}`\"\n else:\n return super().__format__(format_spec)\n\n\n@dataclasses.dataclass\nclass ArvadosResources:\n _resources: abc.Mapping\n\n @classmethod\n def from_client(cls, arv_client):\n return cls(arv_client._resourceDesc['resources'])\n\n @staticmethod\n def _sep_caps(match):\n return '_'.join(match.group(0).lower())\n\n def singular_name(self, name):\n if name == 'sys':\n return name\n elif name.endswith('ies'):\n return f'{name[:-3]}y'\n else:\n return name.removesuffix('s')\n\n def canonical_name(self, name):\n s = re.sub(r'\\W', '_', name)\n s = re.sub(r'[a-z][A-Z]', self._sep_caps, s)\n s = s.lower()\n if s in self._resources:\n return s\n elif s.endswith('y'):\n s = f'{s[:1]}ies'\n else:\n s += 's'\n if s in self._resources:\n return s\n raise ValueError(f\"no resource found for {name!r}\")\n\n def parameters(self, resource_name, method_name):\n return self._resources[resource_name]['methods'][method_name]['parameters']\n\n\n@dataclasses.dataclass\nclass DirectoryLoader:\n arv_client: arvados.api.ThreadSafeAPIClient\n base: abc.Mapping | None\n params: abc.Mapping | None\n resources: ArvadosResources\n\n OBJECT_BASE_PATH = Path('arvados_seed_object.json')\n PARAMETERS_PATH = Path('arvados_seed_parameters.json')\n\n @classmethod\n def from_args(cls, args):\n arv_client = arvados.api.api(**args.api_kwargs)\n return cls(\n arv_client,\n args.object_base,\n args.parameters,\n ArvadosResources.from_client(arv_client),\n )\n\n def _load_defaults(self, instance_defaults, path):\n if instance_defaults is not None:\n return instance_defaults\n try:\n with path.open('rb') as json_file:\n defaults = json.load(json_file)\n except FileNotFoundError:\n defaults = {}\n if not is_mapping(defaults):\n raise ValueError(f\"{path:`} does not contain a JSON object\")\n return defaults\n\n def _create_from(self, json_path, base, base_params):\n prefix, _, rname = json_path.stem.rpartition('.')\n if not prefix:\n raise ValueError(f\"{path:`} does have an object type in the name\")\n rname = self.resources.canonical_name(rname)\n\n kwargs = dict(base_params)\n if 'ensure_unique_name' in self.resources.parameters(rname, 'create'):\n kwargs.setdefault('ensure_unique_name', True)\n\n with json_path.open('rb') as json_file:\n json_body = json.load(json_file)\n kwargs['body'] = {self.resources.singular_name(rname): base | json_body}\n\n resource = getattr(self.arv_client, rname)\n return resource().create(**kwargs).execute()\n\n def build_from(self, dir_path):\n base = self._load_defaults(self.base, dir_path / self.OBJECT_BASE_PATH)\n base_params = self._load_defaults(self.params, dir_path / self.PARAMETERS_PATH)\n created = {}\n failed = {}\n for path in sorted(dir_path.glob('*.json')):\n path_key = str(path.absolute())\n try:\n result = self._create_from(path, base, base_params)\n except Exception as err:\n logger.warning(\n \"failed to load %s: %s\", path, err,\n exc_info=logger.isEnabledFor(logging.DEBUG),\n )\n failed[path_key] = str(err)\n else:\n created[path_key] = result\n return (created, failed)\n\n\nclass ConfigLoader:\n DEFAULT_CONFIG_PATH = Path('/etc/arvados/config.yml')\n DISCOVERY_SERVICE_PATH = 'discovery/v1/apis/{api}/{apiVersion}/rest'\n\n @classmethod\n def _load_yaml(cls, path):\n try:\n with open(path, 'rb') as yaml_file:\n result = yaml.safe_load(yaml_file)\n except OSError as err:\n raise ValueError(f\"error reading {path:`}: {err}\") from None\n if not is_mapping(result):\n raise ValueError(f\"{path:`} is not a YAML object\")\n return result\n\n @classmethod\n def _cluster_config_path(cls):\n return Path(os.environ.get('ARVADOS_CONFIG', cls.DEFAULT_CONFIG_PATH))\n\n @classmethod\n def _from_one_cluster(cls, config):\n try:\n controller_url = config['Services']['Controller']['ExternalURL']\n token = config['SystemRootToken']\n except (KeyError, TypeError) as err:\n raise ValueError(f\"error loading cluster configuration: {err}\") from None\n try:\n insecure = config['TLS']['Insecure']\n except (KeyError, TypeError):\n insecure = False\n return {\n 'version': 'v1',\n 'discoveryServiceUrl': urllib.parse.urljoin(controller_url, cls.DISCOVERY_SERVICE_PATH),\n 'token': token,\n 'insecure': insecure,\n }\n\n @classmethod\n def from_cluster(cls, arg):\n path = cls._cluster_config_path()\n whole_config = cls._load_yaml(path)\n try:\n configs = whole_config['Clusters'].items()\n except (AttributeError, KeyError, TypeError) as err:\n raise ValueError(f\"error loading clusters configuration: {err}\") from None\n kwargs = None\n kwargs_id = None\n for cluster_id, config in configs:\n try:\n new_kwargs = cls._from_one_cluster(config)\n except ValueError:\n continue\n if kwargs is None:\n kwargs = new_kwargs\n kwargs_id = cluster_id\n else:\n raise ValueError(\n f\"{path:`} has configuration for both {kwargs_id} and {cluster_id} - \"\n \"specify a cluster ID\",\n )\n if kwargs is None:\n raise ValueError(f\"no usable cluster configuration found in {path:`}\")\n else:\n return kwargs\n\n @classmethod\n def from_cluster_id(cls, arg):\n path = cls._cluster_config_path()\n whole_config = cls._load_yaml(path)\n try:\n config = whole_config['Clusters'][arg]\n except (AttributeError, KeyError, TypeError) as err:\n raise ValueError(f\"error loading {arg} configuration from {path:`}: {err}\") from None\n return self._from_one_cluster(config)\n\n @classmethod\n def from_env(cls, arg):\n arvados.config.initialize('')\n return cls.from_user()\n\n @classmethod\n def from_user(cls, arg):\n return arvados.api.api_kwargs_from_config('v1')\n\n @classmethod\n def parse_arg(cls, arg):\n try:\n constructor = getattr(cls, f'from_{arg}')\n except AttributeError:\n if re.fullmatch(r'^[a-z0-9]$', arg):\n constructor = cls.from_cluster_id\n else:\n raise ValueError(f\"invalid configuration source {arg!r}\") from None\n return constructor(arg)\n\n @classmethod\n def default_config(cls):\n if os.geteuid() == 0:\n return cls.from_cluster(None)\n else:\n return cls.from_user(None)\n\n\ndef parse_loglevel(arg):\n try:\n return logging.getLevelNamesMapping()[arg.upper()]\n except KeyError:\n raise ValueError(f\"invalid log level {arg!r}\") from None\n\n\ndef validate_mapping(arg):\n if is_mapping(arg):\n return arg\n else:\n raise ValueError(\"value is not a JSON object\")\n\n\ndef parse_arguments(arglist=None):\n parser = argparse.ArgumentParser(\n prog=\"arv-seed\",\n description=\"Create multiple Arvados objects from a directory of JSON files\",\n )\n parser.add_argument(\n '--client-from',\n metavar='SOURCE',\n type=ConfigLoader.parse_arg,\n dest='api_kwargs',\n help=\"\"\"\nWhere to find the Arvados API server and token. Specify one of a cluster ID,\n`cluster`, `env`, or `user`. The first two options load the cluster configuration\nfile from `$ARVADOS_CONFIG` or `/etc/arvados/config.yml`.\n\"\"\")\n parser.add_argument(\n '--loglevel',\n metavar='LEVEL',\n type=parse_loglevel,\n default=logging.INFO,\n help=\"\"\"\nThe name of a log level like `debug`, `info`, `warning`, or `error`\n\"\"\")\n parser.add_argument(\n '--object-base', '--base',\n metavar='BASE_JSON',\n type=cmd_util.JSONArgument(validate_mapping, \"JSON object\"),\n help=\"\"\"\nJSON object or path to set common attributes for all created objects.\nIf not set, will try to read `arvados_seed_object.json` in each directory.\n\"\"\")\n parser.add_argument(\n '--parameters', '--params',\n metavar='PARAMS_JSON',\n type=cmd_util.JSONArgument(validate_mapping, \"JSON object\"),\n help=\"\"\"\nJSON object or path to set parameters when creating objects.\nIf not set, will try to read `arvados_seed_parameters.json` in each directory.\n\"\"\")\n parser.add_argument(\n 'dir_paths',\n metavar='DIRECTORY',\n type=Path,\n nargs=argparse.ONE_OR_MORE,\n help=\"\"\"\nDirectory to read object JSON files from. Object files must be named\n`..json`, where `type` is an Arvados API resource type.\n\"\"\")\n args = parser.parse_args(arglist)\n if args.api_kwargs is None:\n args.api_kwargs = ConfigLoader.default_config()\n return args\n\n\ndef add_log_handlers(logger, stderr=sys.stderr):\n syslog = logging.handlers.SysLogHandler('/dev/log')\n syslog.setFormatter(logging.Formatter('[%(name)s] %(message)s'))\n logger.addHandler(syslog)\n if os.environ.get('TERM'):\n stream = logging.StreamHandler(stderr)\n stream.setFormatter(logging.Formatter(\n '[%(asctime)s] arv-seed: %(levelname)s: %(message)s',\n '%Y-%m-%d %H:%M:%S',\n ))\n logger.addHandler(stream)\n\n\ndef main(\n arglist=None,\n *,\n stdout=sys.stdout,\n stderr=sys.stderr,\n is_main=Path(sys.argv[0]).stem == 'arv-seed',\n):\n if is_main:\n add_log_handlers(_root_logger)\n sys.excepthook = ExceptHook(logger)\n arvados.logger.removeHandler(arvados.logging.log_handler)\n args = parse_arguments(arglist)\n if is_main:\n _root_logger.setLevel(args.loglevel)\n setup_ctx = sys.excepthook.using_exit_code(os.EX_CONFIG)\n else:\n logger.setLevel(args.loglevel)\n setup_ctx = contextlib.nullcontext()\n with setup_ctx:\n loader = DirectoryLoader.from_args(args)\n\n created = {}\n failed = {}\n for dir_path in args.dir_paths:\n try:\n dir_created, dir_failed = loader.build_from(dir_path)\n except Exception as err:\n logger.warning(\n \"failed to load directory %s: %s\", dir_path, err,\n exc_info=logger.isEnabledFor(logging.DEBUG),\n )\n failed[dir_path] = err\n else:\n created.update(dir_created)\n failed.update(dir_failed)\n json.dump({'created': created, 'failed': failed}, stdout)\n print(file=stdout)\n\n if created and failed:\n return 12\n elif failed:\n return 11\n elif created:\n return os.EX_OK\n else:\n return os.EX_NOINPUT\n" }, { "path": "contrib/arvados-bootstrap/src/arv_bootstrap/stubapi.py", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\nimport functools\nimport hashlib\nimport json\nimport logging\nimport operator\nimport os\n\nimport arvados.util\n\n_FILTER_OPS = {\n '=': operator.eq,\n '>=': operator.ge,\n '>': operator.gt,\n '<=': operator.le,\n '<': operator.lt,\n '!=': operator.ne,\n '<>': operator.ne,\n}\n\nclass DeferExecution:\n def __init__(self, fn):\n self._fn = fn\n\n def execute(self, *, num_retries=None):\n return self._fn()\n\n\ndef defer_execution(f):\n @functools.wraps(f)\n def wrapper(*args, **kwds):\n return DeferExecution(functools.partial(f, *args, **kwds))\n return wrapper\n\n\nclass StubKeepClient:\n def __init__(self, basedir):\n self._basedir = basedir\n\n def get(self, locator):\n blockdir = os.path.join(self._basedir, locator[0:3])\n filepath = os.path.join(blockdir, arvados.KeepLocator(locator).md5sum)\n with open(filepath, \"rb\") as fr:\n return fr.read()\n\n def put(self, data, copies=2, num_retries=None, request_id=None, classes=None):\n md5 = hashlib.md5(data).hexdigest()\n locator = '%s+%d' % (md5, len(data))\n\n blockdir = os.path.join(self._basedir, locator[0:3])\n os.makedirs(blockdir, exist_ok=True)\n filepath = os.path.join(blockdir, md5)\n\n with open(os.path.join(filepath + '.tmp'), 'wb') as f:\n f.write(data)\n os.rename(os.path.join(filepath + '.tmp'),\n os.path.join(filepath))\n return locator\n\n\ndef match_filter(fl, obj):\n key, op_key, val = fl\n try:\n op_func = _FILTER_OPS[op_key]\n except KeyError:\n raise NotImplementedError(f\"unsupported filter operator {op_key}\") from None\n else:\n return op_func(obj[key], val)\n\n\nclass StubArvadosResources:\n def __init__(self, basedir, resource_type):\n self._basedir = basedir\n self._resource_type = resource_type\n self._logger = logging.getLogger(f'arvados.stubapi.{resource_type}')\n\n @defer_execution\n def get(self, *, uuid=\"\"):\n with open(os.path.join(self._basedir, uuid), \"rt\") as fr:\n return json.load(fr)\n\n @defer_execution\n def create(self, *, body=None, ensure_unique_name=None):\n if self._resource_type in body:\n body = body[self._resource_type]\n with open(os.path.join(self._basedir, body[\"uuid\"]), \"wt\") as fw:\n json.dump(body, fw, indent=2)\n\n return body\n\n @defer_execution\n def update(self, *, uuid=\"\", body=None):\n if self._resource_type in body:\n body = body[self._resource_type]\n with open(os.path.join(self._basedir, uuid), \"rt\") as fr:\n obj = json.load(fr)\n\n for k,v in body.items():\n obj[k] = v\n\n with open(os.path.join(self._basedir, uuid), \"wt\") as fw:\n json.dump(obj, fw, indent=2)\n\n return obj\n\n @defer_execution\n def list(self, *, filters=None, limit=None, count=None, order=None):\n items = []\n for dirent in os.scandir(self._basedir):\n if not arvados.util.uuid_pattern.match(dirent.name) or not dirent.is_file():\n continue\n\n with open(os.path.join(self._basedir, dirent.name), \"rt\") as fr:\n obj = json.load(fr)\n\n if all(match_filter(f, obj) for f in filters):\n items.append(obj)\n\n if order:\n if len(order) == 1:\n k1, r1 = order[0].split(' ')\n keycomp = lambda x: x[k1]\n elif len(order) == 2:\n k1, r1 = order[0].split(' ')\n k2, r2 = order[1].split(' ')\n if r1 != r2:\n raise NotImplementedError(\"Can't have secondary sort column in opposite direction\")\n keycomp = lambda x: (x[k1], x[k2])\n\n items.sort(key=keycomp, reverse=(r1=='desc'))\n\n if limit is not None:\n items = items[0:limit]\n\n return {\n \"items\": items,\n \"items_available\": len(items)\n }\n\n\nclass StubArvadosAPI:\n def __init__(self, basedir):\n self._basedir = basedir\n\n os.makedirs(os.path.join(self._basedir, \"keep\"), exist_ok=True)\n os.makedirs(os.path.join(self._basedir, \"arvados/v1/collections\"), exist_ok=True)\n os.makedirs(os.path.join(self._basedir, \"arvados/v1/links\"), exist_ok=True)\n os.makedirs(os.path.join(self._basedir, \"arvados/v1/groups\"), exist_ok=True)\n os.makedirs(os.path.join(self._basedir, \"arvados/v1/workflows\"), exist_ok=True)\n\n self.keep = StubKeepClient(os.path.join(self._basedir, \"keep\"))\n\n @classmethod\n def for_cwd(cls):\n return cls(os.getcwd())\n\n def collections(self):\n return StubArvadosResources(os.path.join(self._basedir, \"arvados/v1/collections\"), \"collection\")\n\n def links(self):\n return StubArvadosResources(os.path.join(self._basedir, \"arvados/v1/links\"), \"link\")\n\n def groups(self):\n return StubArvadosResources(os.path.join(self._basedir, \"arvados/v1/groups\"), \"group\")\n\n def workflows(self):\n return StubArvadosResources(os.path.join(self._basedir, \"arvados/v1/workflows\"), \"workflow\")\n" }, { "path": "contrib/arvbash/arvbash.sh", "content": "#!/bin/bash\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\n# bash functions for managing Arvados tokens and other conveniences.\n\nread -rd \"\\000\" helpmessage <\n Set ARVADOS_API_HOST and ARVADOS_API_TOKEN in the current environment based on\n $HOME/.config/arvados/.conf\n With no arguments, print current API host and available Arvados configurations.\n\narvsave \n Save current values of ARVADOS_API_HOST and ARVADOS_API_TOKEN in the current environment to\n $HOME/.config/arvados/.conf\n\narvrm \n Delete $HOME/.config/arvados/.conf\n\narvopen \n Open an Arvados uuid in web browser (http://arvadosapi.com)\n\narvissue \n Open an Arvados ticket in web browser (http://dev.arvados.org)\n\nEOF\n\nif [[ \"$1\" = \"--install\" ]] ; then\n this=$(readlink -f $0)\n if ! grep \". $this\" ~/.bashrc >/dev/null ; then\n echo \". $this\" >> ~/.bashrc\n echo \"Installed into ~/.bashrc\"\n else\n echo \"Already installed in ~/.bashrc\"\n fi\nelif ! [[ $0 =~ bash$ ]] ; then\n echo \"$helpmessage\"\nfi\n\nHISTIGNORE=$HISTIGNORE:'export ARVADOS_API_TOKEN=*'\n\narvswitch() {\n if [[ -n \"$1\" ]] ; then\n if [[ -f $HOME/.config/arvados/$1.conf ]] ; then\n unset ARVADOS_API_HOST_INSECURE\n for a in $(cat $HOME/.config/arvados/$1.conf) ; do export $a ; done\n echo \"Switched to $1\"\n else\n echo \"$1 unknown\"\n fi\n else\n echo \"Switch Arvados environment conf\"\n\techo \"Current host: ${ARVADOS_API_HOST}\"\n echo \"Usage: arvswitch \"\n echo \"Available confs:\" $((cd $HOME/.config/arvados && ls --indicator-style=none *.conf) | rev | cut -c6- | rev)\n fi\n}\n\narvsave() {\n if [[ -n \"$1\" ]] ; then\n\ttouch $HOME/.config/arvados/$1.conf\n\tchmod 0600 $HOME/.config/arvados/$1.conf\n env | grep ARVADOS_ > $HOME/.config/arvados/$1.conf\n else\n echo \"Save current Arvados environment variables to conf file\"\n echo \"Usage: arvsave \"\n fi\n}\n\narvrm() {\n if [[ -n \"$1\" ]] ; then\n if [[ -f $HOME/.config/arvados/$1.conf ]] ; then\n rm $HOME/.config/arvados/$1.conf\n else\n echo \"$1 unknown\"\n fi\n else\n echo \"Delete Arvados environment conf\"\n echo \"Usage: arvrm \"\n fi\n}\n\narvopen() {\n if [[ -n \"$1\" ]] ; then\n xdg-open https://arvadosapi.com/$1\n else\n echo \"Open Arvados uuid in browser\"\n echo \"Usage: arvopen \"\n fi\n}\n\narvissue() {\n if [[ -n \"$1\" ]] ; then\n xdg-open https://dev.arvados.org/issues/$1\n else\n echo \"Open Arvados issue in browser\"\n echo \"Usage: arvissue \"\n fi\n}\n" }, { "path": "contrib/java-sdk-v2/.gitignore", "content": "/.gradle/\n/bin/\n/build/\n.project\n.classpath\n/.settings/\n.DS_Store\n/.idea/\n/out/\n" }, { "path": "contrib/java-sdk-v2/.licenseignore", "content": ".licenseignore\nagpl-3.0.txt\napache-2.0.txt\nCOPYING" }, { "path": "contrib/java-sdk-v2/COPYING", "content": "Unless indicated otherwise in the header of the file, the files in this\nrepository are dual-licensed AGPL-3.0 and Apache-2.0\n\nIndividual files contain an SPDX tag that indicates the license for the file.\ndual-licensed files use the following tag:\n\n SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n\nThis enables machine processing of license information based on the SPDX\nLicense Identifiers that are available here: http://spdx.org/licenses/\n\nThe full license text for each license is available in this directory:\n\n AGPL-3.0: agpl-3.0.txt\n Apache-2.0: apache-2.0.txt\n" }, { "path": "contrib/java-sdk-v2/README.md", "content": "```\nCopyright (C) The Arvados Authors. All rights reserved.\n \nSPDX-License-Identifier: CC-BY-SA-3.0\n```\n\n# Arvados Java SDK\n\n##### About\nArvados Java Client allows to access Arvados servers and uses two APIs:\n* lower level [Keep Server API](https://doc.arvados.org/api/index.html)\n* higher level [Keep-Web API](https://godoc.org/github.com/arvados/arvados/services/keep-web) (when needed)\n\n##### Required Java version\nThis SDK requires Java 8+\n\n##### Logging\n\nSLF4J is used for logging. Concrete logging framework and configuration must be provided by a client.\n\n##### Configuration\n\n[TypeSafe Configuration](https://github.com/lightbend/config) is used for configuring this library.\n\nPlease, have a look at java/resources/reference.conf for default values provided with this library.\n\n* **keepweb-host** - change to host of your Keep-Web installation\n* **keepweb-port** - change to port of your Keep-Web installation\n* **host** - change to host of your Arvados installation\n* **port** - change to port of your Arvados installation\n* **token** - authenticates registered user, one must provide\n [token obtained from Arvados Workbench](https://doc.arvados.org/user/reference/api-tokens.html)\n* **protocol** - don't change to unless really needed\n* **host-insecure** - insecure communication with Arvados (ignores SSL certificate verification), \n don't change to *true* unless really needed\n* **split-size** - size of chunk files in megabytes\n* **temp-dir** - temporary chunk files storage\n* **copies** - amount of chunk files duplicates per Keep server\n* **retries** - in case of chunk files send failure this should allow to repeat send \n (*NOTE*: this parameter is not used at the moment but was left for future improvements)\n\nIn order to override default settings one can create application.conf file in an application.\nExample: src/test/resources/application.conf.\n\nAlternatively ExternalConfigProvider class can be used to pass configuration via code. \nExternalConfigProvider comes with a builder and all of the above values must be provided in order for it to work properly.\n\nArvadosFacade has two constructors, one without arguments that uses values from reference.conf and second one \ntaking ExternalConfigProvider as an argument.\n\n##### API clients\n\nAll API clients inherit from BaseStandardApiClient. This class contains implementation of all \ncommon methods as described in http://doc.arvados.org/api/methods.html.\n\nParameters provided to common or specific methods are String UUID or fields wrapped in Java objects. For example:\n\n```java\nString uuid = \"ardev-4zz18-rxcql7qwyakg1r1\";\n\nCollection actual = client.get(uuid);\n```\n\n```java\nListArgument listArgument = ListArgument.builder()\n .filters(Arrays.asList(\n Filter.of(\"owner_uuid\", Operator.LIKE, \"ardev%\"),\n Filter.of(\"name\", Operator.LIKE, \"Super%\"),\n Filter.of(\"portable_data_hash\", Operator.IN, Lists.newArrayList(\"54f6d9f59065d3c009d4306660989379+65\")\n )))\n .build();\n\nCollectionList actual = client.list(listArgument);\n```\n\nNon-standard API clients must inherit from BaseApiClient. \nFor example: KeepServerApiClient communicates directly with Keep servers using exclusively non-common methods.\n\n##### Business logic\n\nMore advanced API data handling could be implemented as *Facade* classes. \nIn current version functionalities provided by SDK are handled by *ArvadosFacade*.\nThey include:\n* **downloading single file from collection** - using Keep-Web\n* **downloading whole collection** - using Keep-Web or Keep Server API\n* **listing file info from certain collection** - information is returned as list of *FileTokens* providing file details\n* **uploading single file** - to either new or existing collection\n* **uploading list of files** - to either new or existing collection\n* **creating an empty collection**\n* **getting current user info**\n* **listing current user's collections**\n* **creating new project**\n* **deleting certain collection**\n\n##### Note regarding Keep-Web\n\nCurrent version requires both Keep Web and standard Keep Server API configured in order to use Keep-Web functionalities.\n\n##### Integration tests\n\nIn order to run integration tests all fields within following configuration file must be provided: \n```java\nsrc/test/resources/integration-test-appliation.conf \n```\nParameter **integration-tests.project-uuid** should contain UUID of one project available to user,\nwhose token was provided within configuration file. \n\nIntegration tests require connection to real Arvados server.\n\n##### Note regarding file naming\n\nWhile uploading via this SDK all uploaded files within single collection must have different names.\nThis applies also to uploading files to already existing collection. \nRenaming files with duplicate names is not implemented in current version.\n\n##### Building with Gradle\n\nThe Arvados Java SDK is built with `gradle`. Common development build tasks are:\n\n* `clean`\n* `test`\n* `jar` (build the jar files, including documentation)\n* `install`\n" }, { "path": "contrib/java-sdk-v2/agpl-3.0.txt", "content": " GNU AFFERO GENERAL PUBLIC LICENSE\n Version 3, 19 November 2007\n\n Copyright (C) 2007 Free Software Foundation, Inc. \n Everyone is permitted to copy and distribute verbatim copies\n of this license document, but changing it is not allowed.\n\n Preamble\n\n The GNU Affero General Public License is a free, copyleft license for\nsoftware and other kinds of works, specifically designed to ensure\ncooperation with the community in the case of network server software.\n\n The licenses for most software and other practical works are designed\nto take away your freedom to share and change the works. By contrast,\nour General Public Licenses are intended to guarantee your freedom to\nshare and change all versions of a program--to make sure it remains free\nsoftware for all its users.\n\n When we speak of free software, we are referring to freedom, not\nprice. Our General Public Licenses are designed to make sure that you\nhave the freedom to distribute copies of free software (and charge for\nthem if you wish), that you receive source code or can get it if you\nwant it, that you can change the software or use pieces of it in new\nfree programs, and that you know you can do these things.\n\n Developers that use our General Public Licenses protect your rights\nwith two steps: (1) assert copyright on the software, and (2) offer\nyou this License which gives you legal permission to copy, distribute\nand/or modify the software.\n\n A secondary benefit of defending all users' freedom is that\nimprovements made in alternate versions of the program, if they\nreceive widespread use, become available for other developers to\nincorporate. Many developers of free software are heartened and\nencouraged by the resulting cooperation. However, in the case of\nsoftware used on network servers, this result may fail to come about.\nThe GNU General Public License permits making a modified version and\nletting the public access it on a server without ever releasing its\nsource code to the public.\n\n The GNU Affero General Public License is designed specifically to\nensure that, in such cases, the modified source code becomes available\nto the community. It requires the operator of a network server to\nprovide the source code of the modified version running there to the\nusers of that server. Therefore, public use of a modified version, on\na publicly accessible server, gives the public access to the source\ncode of the modified version.\n\n An older license, called the Affero General Public License and\npublished by Affero, was designed to accomplish similar goals. This is\na different license, not a version of the Affero GPL, but Affero has\nreleased a new version of the Affero GPL which permits relicensing under\nthis license.\n\n The precise terms and conditions for copying, distribution and\nmodification follow.\n\n TERMS AND CONDITIONS\n\n 0. Definitions.\n\n \"This License\" refers to version 3 of the GNU Affero General Public License.\n\n \"Copyright\" also means copyright-like laws that apply to other kinds of\nworks, such as semiconductor masks.\n\n \"The Program\" refers to any copyrightable work licensed under this\nLicense. Each licensee is addressed as \"you\". \"Licensees\" and\n\"recipients\" may be individuals or organizations.\n\n To \"modify\" a work means to copy from or adapt all or part of the work\nin a fashion requiring copyright permission, other than the making of an\nexact copy. The resulting work is called a \"modified version\" of the\nearlier work or a work \"based on\" the earlier work.\n\n A \"covered work\" means either the unmodified Program or a work based\non the Program.\n\n To \"propagate\" a work means to do anything with it that, without\npermission, would make you directly or secondarily liable for\ninfringement under applicable copyright law, except executing it on a\ncomputer or modifying a private copy. Propagation includes copying,\ndistribution (with or without modification), making available to the\npublic, and in some countries other activities as well.\n\n To \"convey\" a work means any kind of propagation that enables other\nparties to make or receive copies. Mere interaction with a user through\na computer network, with no transfer of a copy, is not conveying.\n\n An interactive user interface displays \"Appropriate Legal Notices\"\nto the extent that it includes a convenient and prominently visible\nfeature that (1) displays an appropriate copyright notice, and (2)\ntells the user that there is no warranty for the work (except to the\nextent that warranties are provided), that licensees may convey the\nwork under this License, and how to view a copy of this License. If\nthe interface presents a list of user commands or options, such as a\nmenu, a prominent item in the list meets this criterion.\n\n 1. Source Code.\n\n The \"source code\" for a work means the preferred form of the work\nfor making modifications to it. \"Object code\" means any non-source\nform of a work.\n\n A \"Standard Interface\" means an interface that either is an official\nstandard defined by a recognized standards body, or, in the case of\ninterfaces specified for a particular programming language, one that\nis widely used among developers working in that language.\n\n The \"System Libraries\" of an executable work include anything, other\nthan the work as a whole, that (a) is included in the normal form of\npackaging a Major Component, but which is not part of that Major\nComponent, and (b) serves only to enable use of the work with that\nMajor Component, or to implement a Standard Interface for which an\nimplementation is available to the public in source code form. A\n\"Major Component\", in this context, means a major essential component\n(kernel, window system, and so on) of the specific operating system\n(if any) on which the executable work runs, or a compiler used to\nproduce the work, or an object code interpreter used to run it.\n\n The \"Corresponding Source\" for a work in object code form means all\nthe source code needed to generate, install, and (for an executable\nwork) run the object code and to modify the work, including scripts to\ncontrol those activities. However, it does not include the work's\nSystem Libraries, or general-purpose tools or generally available free\nprograms which are used unmodified in performing those activities but\nwhich are not part of the work. For example, Corresponding Source\nincludes interface definition files associated with source files for\nthe work, and the source code for shared libraries and dynamically\nlinked subprograms that the work is specifically designed to require,\nsuch as by intimate data communication or control flow between those\nsubprograms and other parts of the work.\n\n The Corresponding Source need not include anything that users\ncan regenerate automatically from other parts of the Corresponding\nSource.\n\n The Corresponding Source for a work in source code form is that\nsame work.\n\n 2. Basic Permissions.\n\n All rights granted under this License are granted for the term of\ncopyright on the Program, and are irrevocable provided the stated\nconditions are met. This License explicitly affirms your unlimited\npermission to run the unmodified Program. The output from running a\ncovered work is covered by this License only if the output, given its\ncontent, constitutes a covered work. This License acknowledges your\nrights of fair use or other equivalent, as provided by copyright law.\n\n You may make, run and propagate covered works that you do not\nconvey, without conditions so long as your license otherwise remains\nin force. You may convey covered works to others for the sole purpose\nof having them make modifications exclusively for you, or provide you\nwith facilities for running those works, provided that you comply with\nthe terms of this License in conveying all material for which you do\nnot control copyright. Those thus making or running the covered works\nfor you must do so exclusively on your behalf, under your direction\nand control, on terms that prohibit them from making any copies of\nyour copyrighted material outside their relationship with you.\n\n Conveying under any other circumstances is permitted solely under\nthe conditions stated below. Sublicensing is not allowed; section 10\nmakes it unnecessary.\n\n 3. Protecting Users' Legal Rights From Anti-Circumvention Law.\n\n No covered work shall be deemed part of an effective technological\nmeasure under any applicable law fulfilling obligations under article\n11 of the WIPO copyright treaty adopted on 20 December 1996, or\nsimilar laws prohibiting or restricting circumvention of such\nmeasures.\n\n When you convey a covered work, you waive any legal power to forbid\ncircumvention of technological measures to the extent such circumvention\nis effected by exercising rights under this License with respect to\nthe covered work, and you disclaim any intention to limit operation or\nmodification of the work as a means of enforcing, against the work's\nusers, your or third parties' legal rights to forbid circumvention of\ntechnological measures.\n\n 4. Conveying Verbatim Copies.\n\n You may convey verbatim copies of the Program's source code as you\nreceive it, in any medium, provided that you conspicuously and\nappropriately publish on each copy an appropriate copyright notice;\nkeep intact all notices stating that this License and any\nnon-permissive terms added in accord with section 7 apply to the code;\nkeep intact all notices of the absence of any warranty; and give all\nrecipients a copy of this License along with the Program.\n\n You may charge any price or no price for each copy that you convey,\nand you may offer support or warranty protection for a fee.\n\n 5. Conveying Modified Source Versions.\n\n You may convey a work based on the Program, or the modifications to\nproduce it from the Program, in the form of source code under the\nterms of section 4, provided that you also meet all of these conditions:\n\n a) The work must carry prominent notices stating that you modified\n it, and giving a relevant date.\n\n b) The work must carry prominent notices stating that it is\n released under this License and any conditions added under section\n 7. This requirement modifies the requirement in section 4 to\n \"keep intact all notices\".\n\n c) You must license the entire work, as a whole, under this\n License to anyone who comes into possession of a copy. This\n License will therefore apply, along with any applicable section 7\n additional terms, to the whole of the work, and all its parts,\n regardless of how they are packaged. This License gives no\n permission to license the work in any other way, but it does not\n invalidate such permission if you have separately received it.\n\n d) If the work has interactive user interfaces, each must display\n Appropriate Legal Notices; however, if the Program has interactive\n interfaces that do not display Appropriate Legal Notices, your\n work need not make them do so.\n\n A compilation of a covered work with other separate and independent\nworks, which are not by their nature extensions of the covered work,\nand which are not combined with it such as to form a larger program,\nin or on a volume of a storage or distribution medium, is called an\n\"aggregate\" if the compilation and its resulting copyright are not\nused to limit the access or legal rights of the compilation's users\nbeyond what the individual works permit. Inclusion of a covered work\nin an aggregate does not cause this License to apply to the other\nparts of the aggregate.\n\n 6. Conveying Non-Source Forms.\n\n You may convey a covered work in object code form under the terms\nof sections 4 and 5, provided that you also convey the\nmachine-readable Corresponding Source under the terms of this License,\nin one of these ways:\n\n a) Convey the object code in, or embodied in, a physical product\n (including a physical distribution medium), accompanied by the\n Corresponding Source fixed on a durable physical medium\n customarily used for software interchange.\n\n b) Convey the object code in, or embodied in, a physical product\n (including a physical distribution medium), accompanied by a\n written offer, valid for at least three years and valid for as\n long as you offer spare parts or customer support for that product\n model, to give anyone who possesses the object code either (1) a\n copy of the Corresponding Source for all the software in the\n product that is covered by this License, on a durable physical\n medium customarily used for software interchange, for a price no\n more than your reasonable cost of physically performing this\n conveying of source, or (2) access to copy the\n Corresponding Source from a network server at no charge.\n\n c) Convey individual copies of the object code with a copy of the\n written offer to provide the Corresponding Source. This\n alternative is allowed only occasionally and noncommercially, and\n only if you received the object code with such an offer, in accord\n with subsection 6b.\n\n d) Convey the object code by offering access from a designated\n place (gratis or for a charge), and offer equivalent access to the\n Corresponding Source in the same way through the same place at no\n further charge. You need not require recipients to copy the\n Corresponding Source along with the object code. If the place to\n copy the object code is a network server, the Corresponding Source\n may be on a different server (operated by you or a third party)\n that supports equivalent copying facilities, provided you maintain\n clear directions next to the object code saying where to find the\n Corresponding Source. Regardless of what server hosts the\n Corresponding Source, you remain obligated to ensure that it is\n available for as long as needed to satisfy these requirements.\n\n e) Convey the object code using peer-to-peer transmission, provided\n you inform other peers where the object code and Corresponding\n Source of the work are being offered to the general public at no\n charge under subsection 6d.\n\n A separable portion of the object code, whose source code is excluded\nfrom the Corresponding Source as a System Library, need not be\nincluded in conveying the object code work.\n\n A \"User Product\" is either (1) a \"consumer product\", which means any\ntangible personal property which is normally used for personal, family,\nor household purposes, or (2) anything designed or sold for incorporation\ninto a dwelling. In determining whether a product is a consumer product,\ndoubtful cases shall be resolved in favor of coverage. For a particular\nproduct received by a particular user, \"normally used\" refers to a\ntypical or common use of that class of product, regardless of the status\nof the particular user or of the way in which the particular user\nactually uses, or expects or is expected to use, the product. A product\nis a consumer product regardless of whether the product has substantial\ncommercial, industrial or non-consumer uses, unless such uses represent\nthe only significant mode of use of the product.\n\n \"Installation Information\" for a User Product means any methods,\nprocedures, authorization keys, or other information required to install\nand execute modified versions of a covered work in that User Product from\na modified version of its Corresponding Source. The information must\nsuffice to ensure that the continued functioning of the modified object\ncode is in no case prevented or interfered with solely because\nmodification has been made.\n\n If you convey an object code work under this section in, or with, or\nspecifically for use in, a User Product, and the conveying occurs as\npart of a transaction in which the right of possession and use of the\nUser Product is transferred to the recipient in perpetuity or for a\nfixed term (regardless of how the transaction is characterized), the\nCorresponding Source conveyed under this section must be accompanied\nby the Installation Information. But this requirement does not apply\nif neither you nor any third party retains the ability to install\nmodified object code on the User Product (for example, the work has\nbeen installed in ROM).\n\n The requirement to provide Installation Information does not include a\nrequirement to continue to provide support service, warranty, or updates\nfor a work that has been modified or installed by the recipient, or for\nthe User Product in which it has been modified or installed. Access to a\nnetwork may be denied when the modification itself materially and\nadversely affects the operation of the network or violates the rules and\nprotocols for communication across the network.\n\n Corresponding Source conveyed, and Installation Information provided,\nin accord with this section must be in a format that is publicly\ndocumented (and with an implementation available to the public in\nsource code form), and must require no special password or key for\nunpacking, reading or copying.\n\n 7. Additional Terms.\n\n \"Additional permissions\" are terms that supplement the terms of this\nLicense by making exceptions from one or more of its conditions.\nAdditional permissions that are applicable to the entire Program shall\nbe treated as though they were included in this License, to the extent\nthat they are valid under applicable law. If additional permissions\napply only to part of the Program, that part may be used separately\nunder those permissions, but the entire Program remains governed by\nthis License without regard to the additional permissions.\n\n When you convey a copy of a covered work, you may at your option\nremove any additional permissions from that copy, or from any part of\nit. (Additional permissions may be written to require their own\nremoval in certain cases when you modify the work.) You may place\nadditional permissions on material, added by you to a covered work,\nfor which you have or can give appropriate copyright permission.\n\n Notwithstanding any other provision of this License, for material you\nadd to a covered work, you may (if authorized by the copyright holders of\nthat material) supplement the terms of this License with terms:\n\n a) Disclaiming warranty or limiting liability differently from the\n terms of sections 15 and 16 of this License; or\n\n b) Requiring preservation of specified reasonable legal notices or\n author attributions in that material or in the Appropriate Legal\n Notices displayed by works containing it; or\n\n c) Prohibiting misrepresentation of the origin of that material, or\n requiring that modified versions of such material be marked in\n reasonable ways as different from the original version; or\n\n d) Limiting the use for publicity purposes of names of licensors or\n authors of the material; or\n\n e) Declining to grant rights under trademark law for use of some\n trade names, trademarks, or service marks; or\n\n f) Requiring indemnification of licensors and authors of that\n material by anyone who conveys the material (or modified versions of\n it) with contractual assumptions of liability to the recipient, for\n any liability that these contractual assumptions directly impose on\n those licensors and authors.\n\n All other non-permissive additional terms are considered \"further\nrestrictions\" within the meaning of section 10. If the Program as you\nreceived it, or any part of it, contains a notice stating that it is\ngoverned by this License along with a term that is a further\nrestriction, you may remove that term. If a license document contains\na further restriction but permits relicensing or conveying under this\nLicense, you may add to a covered work material governed by the terms\nof that license document, provided that the further restriction does\nnot survive such relicensing or conveying.\n\n If you add terms to a covered work in accord with this section, you\nmust place, in the relevant source files, a statement of the\nadditional terms that apply to those files, or a notice indicating\nwhere to find the applicable terms.\n\n Additional terms, permissive or non-permissive, may be stated in the\nform of a separately written license, or stated as exceptions;\nthe above requirements apply either way.\n\n 8. Termination.\n\n You may not propagate or modify a covered work except as expressly\nprovided under this License. Any attempt otherwise to propagate or\nmodify it is void, and will automatically terminate your rights under\nthis License (including any patent licenses granted under the third\nparagraph of section 11).\n\n However, if you cease all violation of this License, then your\nlicense from a particular copyright holder is reinstated (a)\nprovisionally, unless and until the copyright holder explicitly and\nfinally terminates your license, and (b) permanently, if the copyright\nholder fails to notify you of the violation by some reasonable means\nprior to 60 days after the cessation.\n\n Moreover, your license from a particular copyright holder is\nreinstated permanently if the copyright holder notifies you of the\nviolation by some reasonable means, this is the first time you have\nreceived notice of violation of this License (for any work) from that\ncopyright holder, and you cure the violation prior to 30 days after\nyour receipt of the notice.\n\n Termination of your rights under this section does not terminate the\nlicenses of parties who have received copies or rights from you under\nthis License. If your rights have been terminated and not permanently\nreinstated, you do not qualify to receive new licenses for the same\nmaterial under section 10.\n\n 9. Acceptance Not Required for Having Copies.\n\n You are not required to accept this License in order to receive or\nrun a copy of the Program. Ancillary propagation of a covered work\noccurring solely as a consequence of using peer-to-peer transmission\nto receive a copy likewise does not require acceptance. However,\nnothing other than this License grants you permission to propagate or\nmodify any covered work. These actions infringe copyright if you do\nnot accept this License. Therefore, by modifying or propagating a\ncovered work, you indicate your acceptance of this License to do so.\n\n 10. Automatic Licensing of Downstream Recipients.\n\n Each time you convey a covered work, the recipient automatically\nreceives a license from the original licensors, to run, modify and\npropagate that work, subject to this License. You are not responsible\nfor enforcing compliance by third parties with this License.\n\n An \"entity transaction\" is a transaction transferring control of an\norganization, or substantially all assets of one, or subdividing an\norganization, or merging organizations. If propagation of a covered\nwork results from an entity transaction, each party to that\ntransaction who receives a copy of the work also receives whatever\nlicenses to the work the party's predecessor in interest had or could\ngive under the previous paragraph, plus a right to possession of the\nCorresponding Source of the work from the predecessor in interest, if\nthe predecessor has it or can get it with reasonable efforts.\n\n You may not impose any further restrictions on the exercise of the\nrights granted or affirmed under this License. For example, you may\nnot impose a license fee, royalty, or other charge for exercise of\nrights granted under this License, and you may not initiate litigation\n(including a cross-claim or counterclaim in a lawsuit) alleging that\nany patent claim is infringed by making, using, selling, offering for\nsale, or importing the Program or any portion of it.\n\n 11. Patents.\n\n A \"contributor\" is a copyright holder who authorizes use under this\nLicense of the Program or a work on which the Program is based. The\nwork thus licensed is called the contributor's \"contributor version\".\n\n A contributor's \"essential patent claims\" are all patent claims\nowned or controlled by the contributor, whether already acquired or\nhereafter acquired, that would be infringed by some manner, permitted\nby this License, of making, using, or selling its contributor version,\nbut do not include claims that would be infringed only as a\nconsequence of further modification of the contributor version. For\npurposes of this definition, \"control\" includes the right to grant\npatent sublicenses in a manner consistent with the requirements of\nthis License.\n\n Each contributor grants you a non-exclusive, worldwide, royalty-free\npatent license under the contributor's essential patent claims, to\nmake, use, sell, offer for sale, import and otherwise run, modify and\npropagate the contents of its contributor version.\n\n In the following three paragraphs, a \"patent license\" is any express\nagreement or commitment, however denominated, not to enforce a patent\n(such as an express permission to practice a patent or covenant not to\nsue for patent infringement). To \"grant\" such a patent license to a\nparty means to make such an agreement or commitment not to enforce a\npatent against the party.\n\n If you convey a covered work, knowingly relying on a patent license,\nand the Corresponding Source of the work is not available for anyone\nto copy, free of charge and under the terms of this License, through a\npublicly available network server or other readily accessible means,\nthen you must either (1) cause the Corresponding Source to be so\navailable, or (2) arrange to deprive yourself of the benefit of the\npatent license for this particular work, or (3) arrange, in a manner\nconsistent with the requirements of this License, to extend the patent\nlicense to downstream recipients. \"Knowingly relying\" means you have\nactual knowledge that, but for the patent license, your conveying the\ncovered work in a country, or your recipient's use of the covered work\nin a country, would infringe one or more identifiable patents in that\ncountry that you have reason to believe are valid.\n\n If, pursuant to or in connection with a single transaction or\narrangement, you convey, or propagate by procuring conveyance of, a\ncovered work, and grant a patent license to some of the parties\nreceiving the covered work authorizing them to use, propagate, modify\nor convey a specific copy of the covered work, then the patent license\nyou grant is automatically extended to all recipients of the covered\nwork and works based on it.\n\n A patent license is \"discriminatory\" if it does not include within\nthe scope of its coverage, prohibits the exercise of, or is\nconditioned on the non-exercise of one or more of the rights that are\nspecifically granted under this License. You may not convey a covered\nwork if you are a party to an arrangement with a third party that is\nin the business of distributing software, under which you make payment\nto the third party based on the extent of your activity of conveying\nthe work, and under which the third party grants, to any of the\nparties who would receive the covered work from you, a discriminatory\npatent license (a) in connection with copies of the covered work\nconveyed by you (or copies made from those copies), or (b) primarily\nfor and in connection with specific products or compilations that\ncontain the covered work, unless you entered into that arrangement,\nor that patent license was granted, prior to 28 March 2007.\n\n Nothing in this License shall be construed as excluding or limiting\nany implied license or other defenses to infringement that may\notherwise be available to you under applicable patent law.\n\n 12. No Surrender of Others' Freedom.\n\n If conditions are imposed on you (whether by court order, agreement or\notherwise) that contradict the conditions of this License, they do not\nexcuse you from the conditions of this License. If you cannot convey a\ncovered work so as to satisfy simultaneously your obligations under this\nLicense and any other pertinent obligations, then as a consequence you may\nnot convey it at all. For example, if you agree to terms that obligate you\nto collect a royalty for further conveying from those to whom you convey\nthe Program, the only way you could satisfy both those terms and this\nLicense would be to refrain entirely from conveying the Program.\n\n 13. Remote Network Interaction; Use with the GNU General Public License.\n\n Notwithstanding any other provision of this License, if you modify the\nProgram, your modified version must prominently offer all users\ninteracting with it remotely through a computer network (if your version\nsupports such interaction) an opportunity to receive the Corresponding\nSource of your version by providing access to the Corresponding Source\nfrom a network server at no charge, through some standard or customary\nmeans of facilitating copying of software. This Corresponding Source\nshall include the Corresponding Source for any work covered by version 3\nof the GNU General Public License that is incorporated pursuant to the\nfollowing paragraph.\n\n Notwithstanding any other provision of this License, you have\npermission to link or combine any covered work with a work licensed\nunder version 3 of the GNU General Public License into a single\ncombined work, and to convey the resulting work. The terms of this\nLicense will continue to apply to the part which is the covered work,\nbut the work with which it is combined will remain governed by version\n3 of the GNU General Public License.\n\n 14. Revised Versions of this License.\n\n The Free Software Foundation may publish revised and/or new versions of\nthe GNU Affero General Public License from time to time. Such new versions\nwill be similar in spirit to the present version, but may differ in detail to\naddress new problems or concerns.\n\n Each version is given a distinguishing version number. If the\nProgram specifies that a certain numbered version of the GNU Affero General\nPublic License \"or any later version\" applies to it, you have the\noption of following the terms and conditions either of that numbered\nversion or of any later version published by the Free Software\nFoundation. If the Program does not specify a version number of the\nGNU Affero General Public License, you may choose any version ever published\nby the Free Software Foundation.\n\n If the Program specifies that a proxy can decide which future\nversions of the GNU Affero General Public License can be used, that proxy's\npublic statement of acceptance of a version permanently authorizes you\nto choose that version for the Program.\n\n Later license versions may give you additional or different\npermissions. However, no additional obligations are imposed on any\nauthor or copyright holder as a result of your choosing to follow a\nlater version.\n\n 15. Disclaimer of Warranty.\n\n THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY\nAPPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT\nHOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM \"AS IS\" WITHOUT WARRANTY\nOF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,\nTHE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR\nPURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM\nIS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF\nALL NECESSARY SERVICING, REPAIR OR CORRECTION.\n\n 16. Limitation of Liability.\n\n IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING\nWILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS\nTHE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY\nGENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE\nUSE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF\nDATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD\nPARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),\nEVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF\nSUCH DAMAGES.\n\n 17. Interpretation of Sections 15 and 16.\n\n If the disclaimer of warranty and limitation of liability provided\nabove cannot be given local legal effect according to their terms,\nreviewing courts shall apply local law that most closely approximates\nan absolute waiver of all civil liability in connection with the\nProgram, unless a warranty or assumption of liability accompanies a\ncopy of the Program in return for a fee.\n\n END OF TERMS AND CONDITIONS\n\n How to Apply These Terms to Your New Programs\n\n If you develop a new program, and you want it to be of the greatest\npossible use to the public, the best way to achieve this is to make it\nfree software which everyone can redistribute and change under these terms.\n\n To do so, attach the following notices to the program. It is safest\nto attach them to the start of each source file to most effectively\nstate the exclusion of warranty; and each file should have at least\nthe \"copyright\" line and a pointer to where the full notice is found.\n\n \n Copyright (C) \n\n This program is free software: you can redistribute it and/or modify\n it under the terms of the GNU Affero General Public License as published by\n the Free Software Foundation, either version 3 of the License, or\n (at your option) any later version.\n\n This program is distributed in the hope that it will be useful,\n but WITHOUT ANY WARRANTY; without even the implied warranty of\n MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n GNU Affero General Public License for more details.\n\n You should have received a copy of the GNU Affero General Public License\n along with this program. If not, see .\n\nAlso add information on how to contact you by electronic and paper mail.\n\n If your software can interact with users remotely through a computer\nnetwork, you should also make sure that it provides a way for users to\nget its source. For example, if your program is a web application, its\ninterface could display a \"Source\" link that leads users to an archive\nof the code. There are many ways you could offer source, and different\nsolutions will be better for different programs; see section 13 for the\nspecific requirements.\n\n You should also get your employer (if you work as a programmer) or school,\nif any, to sign a \"copyright disclaimer\" for the program, if necessary.\nFor more information on this, and how to apply and follow the GNU AGPL, see\n.\n" }, { "path": "contrib/java-sdk-v2/apache-2.0.txt", "content": "\n Apache License\n Version 2.0, January 2004\n http://www.apache.org/licenses/\n\n TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n 1. Definitions.\n\n \"License\" shall mean the terms and conditions for use, reproduction,\n and distribution as defined by Sections 1 through 9 of this document.\n\n \"Licensor\" shall mean the copyright owner or entity authorized by\n the copyright owner that is granting the License.\n\n \"Legal Entity\" shall mean the union of the acting entity and all\n other entities that control, are controlled by, or are under common\n control with that entity. For the purposes of this definition,\n \"control\" means (i) the power, direct or indirect, to cause the\n direction or management of such entity, whether by contract or\n otherwise, or (ii) ownership of fifty percent (50%) or more of the\n outstanding shares, or (iii) beneficial ownership of such entity.\n\n \"You\" (or \"Your\") shall mean an individual or Legal Entity\n exercising permissions granted by this License.\n\n \"Source\" form shall mean the preferred form for making modifications,\n including but not limited to software source code, documentation\n source, and configuration files.\n\n \"Object\" form shall mean any form resulting from mechanical\n transformation or translation of a Source form, including but\n not limited to compiled object code, generated documentation,\n and conversions to other media types.\n\n \"Work\" shall mean the work of authorship, whether in Source or\n Object form, made available under the License, as indicated by a\n copyright notice that is included in or attached to the work\n (an example is provided in the Appendix below).\n\n \"Derivative Works\" shall mean any work, whether in Source or Object\n form, that is based on (or derived from) the Work and for which the\n editorial revisions, annotations, elaborations, or other modifications\n represent, as a whole, an original work of authorship. For the purposes\n of this License, Derivative Works shall not include works that remain\n separable from, or merely link (or bind by name) to the interfaces of,\n the Work and Derivative Works thereof.\n\n \"Contribution\" shall mean any work of authorship, including\n the original version of the Work and any modifications or additions\n to that Work or Derivative Works thereof, that is intentionally\n submitted to Licensor for inclusion in the Work by the copyright owner\n or by an individual or Legal Entity authorized to submit on behalf of\n the copyright owner. For the purposes of this definition, \"submitted\"\n means any form of electronic, verbal, or written communication sent\n to the Licensor or its representatives, including but not limited to\n communication on electronic mailing lists, source code control systems,\n and issue tracking systems that are managed by, or on behalf of, the\n Licensor for the purpose of discussing and improving the Work, but\n excluding communication that is conspicuously marked or otherwise\n designated in writing by the copyright owner as \"Not a Contribution.\"\n\n \"Contributor\" shall mean Licensor and any individual or Legal Entity\n on behalf of whom a Contribution has been received by Licensor and\n subsequently incorporated within the Work.\n\n 2. Grant of Copyright License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n copyright license to reproduce, prepare Derivative Works of,\n publicly display, publicly perform, sublicense, and distribute the\n Work and such Derivative Works in Source or Object form.\n\n 3. Grant of Patent License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n (except as stated in this section) patent license to make, have made,\n use, offer to sell, sell, import, and otherwise transfer the Work,\n where such license applies only to those patent claims licensable\n by such Contributor that are necessarily infringed by their\n Contribution(s) alone or by combination of their Contribution(s)\n with the Work to which such Contribution(s) was submitted. If You\n institute patent litigation against any entity (including a\n cross-claim or counterclaim in a lawsuit) alleging that the Work\n or a Contribution incorporated within the Work constitutes direct\n or contributory patent infringement, then any patent licenses\n granted to You under this License for that Work shall terminate\n as of the date such litigation is filed.\n\n 4. Redistribution. You may reproduce and distribute copies of the\n Work or Derivative Works thereof in any medium, with or without\n modifications, and in Source or Object form, provided that You\n meet the following conditions:\n\n (a) You must give any other recipients of the Work or\n Derivative Works a copy of this License; and\n\n (b) You must cause any modified files to carry prominent notices\n stating that You changed the files; and\n\n (c) You must retain, in the Source form of any Derivative Works\n that You distribute, all copyright, patent, trademark, and\n attribution notices from the Source form of the Work,\n excluding those notices that do not pertain to any part of\n the Derivative Works; and\n\n (d) If the Work includes a \"NOTICE\" text file as part of its\n distribution, then any Derivative Works that You distribute must\n include a readable copy of the attribution notices contained\n within such NOTICE file, excluding those notices that do not\n pertain to any part of the Derivative Works, in at least one\n of the following places: within a NOTICE text file distributed\n as part of the Derivative Works; within the Source form or\n documentation, if provided along with the Derivative Works; or,\n within a display generated by the Derivative Works, if and\n wherever such third-party notices normally appear. The contents\n of the NOTICE file are for informational purposes only and\n do not modify the License. You may add Your own attribution\n notices within Derivative Works that You distribute, alongside\n or as an addendum to the NOTICE text from the Work, provided\n that such additional attribution notices cannot be construed\n as modifying the License.\n\n You may add Your own copyright statement to Your modifications and\n may provide additional or different license terms and conditions\n for use, reproduction, or distribution of Your modifications, or\n for any such Derivative Works as a whole, provided Your use,\n reproduction, and distribution of the Work otherwise complies with\n the conditions stated in this License.\n\n 5. Submission of Contributions. Unless You explicitly state otherwise,\n any Contribution intentionally submitted for inclusion in the Work\n by You to the Licensor shall be under the terms and conditions of\n this License, without any additional terms or conditions.\n Notwithstanding the above, nothing herein shall supersede or modify\n the terms of any separate license agreement you may have executed\n with Licensor regarding such Contributions.\n\n 6. Trademarks. This License does not grant permission to use the trade\n names, trademarks, service marks, or product names of the Licensor,\n except as required for reasonable and customary use in describing the\n origin of the Work and reproducing the content of the NOTICE file.\n\n 7. Disclaimer of Warranty. Unless required by applicable law or\n agreed to in writing, Licensor provides the Work (and each\n Contributor provides its Contributions) on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n implied, including, without limitation, any warranties or conditions\n of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n PARTICULAR PURPOSE. You are solely responsible for determining the\n appropriateness of using or redistributing the Work and assume any\n risks associated with Your exercise of permissions under this License.\n\n 8. Limitation of Liability. In no event and under no legal theory,\n whether in tort (including negligence), contract, or otherwise,\n unless required by applicable law (such as deliberate and grossly\n negligent acts) or agreed to in writing, shall any Contributor be\n liable to You for damages, including any direct, indirect, special,\n incidental, or consequential damages of any character arising as a\n result of this License or out of the use or inability to use the\n Work (including but not limited to damages for loss of goodwill,\n work stoppage, computer failure or malfunction, or any and all\n other commercial damages or losses), even if such Contributor\n has been advised of the possibility of such damages.\n\n 9. Accepting Warranty or Additional Liability. While redistributing\n the Work or Derivative Works thereof, You may choose to offer,\n and charge a fee for, acceptance of support, warranty, indemnity,\n or other liability obligations and/or rights consistent with this\n License. However, in accepting such obligations, You may act only\n on Your own behalf and on Your sole responsibility, not on behalf\n of any other Contributor, and only if You agree to indemnify,\n defend, and hold each Contributor harmless for any liability\n incurred by, or claims asserted against, such Contributor by reason\n of your accepting any such warranty or additional liability.\n\n END OF TERMS AND CONDITIONS\n\n APPENDIX: How to apply the Apache License to your work.\n\n To apply the Apache License to your work, attach the following\n boilerplate notice, with the fields enclosed by brackets \"[]\"\n replaced with your own identifying information. (Don't include\n the brackets!) The text should be enclosed in the appropriate\n comment syntax for the file format. We also recommend that a\n file or class name and description of purpose be included on the\n same \"printed page\" as the copyright notice for easier\n identification within third-party archives.\n\n Copyright [yyyy] [name of copyright owner]\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n" }, { "path": "contrib/java-sdk-v2/build.gradle", "content": "apply plugin: 'java-library'\napply plugin: 'eclipse'\napply plugin: 'idea'\napply plugin: 'maven'\napply plugin: 'signing'\n\n\nrepositories {\n mavenCentral()\n}\n\ndependencies {\n api 'com.squareup.okhttp3:okhttp:3.9.1'\n api 'com.fasterxml.jackson.core:jackson-databind:2.9.2'\n api 'com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.9.2'\n api 'com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.9.2'\n api 'commons-codec:commons-codec:1.11'\n api 'commons-io:commons-io:2.6'\n api 'com.google.guava:guava:23.4-jre'\n api 'org.slf4j:slf4j-api:1.7.25'\n api 'com.typesafe:config:1.3.2'\n \n testImplementation 'junit:junit:4.12'\n testImplementation 'org.mockito:mockito-core:5.17.0'\n testImplementation 'org.assertj:assertj-core:3.8.0'\n testImplementation 'com.squareup.okhttp3:mockwebserver:3.9.1'\n}\n\ntest {\n useJUnit {\n excludeCategories 'org.arvados.client.junit.categories.IntegrationTests'\n }\n\n\ttestLogging {\n\t events \"passed\", \"skipped\", \"failed\"\n\t afterSuite { desc, result ->\n\t if (!desc.parent) { // will match the outermost suite\n\t println \"\\n---- Test results ----\"\n\t println \"${result.resultType} (${result.testCount} tests, ${result.successfulTestCount} successes, ${result.failedTestCount} failures, ${result.skippedTestCount} skipped)\"\n\t println \"\"\n\t }\n\t }\n\t}\n}\n\ntask integrationTest(type: Test) {\n useJUnit {\n includeCategories 'org.arvados.client.junit.categories.IntegrationTests'\n }\n}\n\ntask javadocJar(type: Jar) {\n classifier = 'javadoc'\n from javadoc\n}\n\ntask sourcesJar(type: Jar) {\n classifier = 'sources'\n from sourceSets.main.allSource\n}\n\nartifacts {\n archives javadocJar, sourcesJar\n}\n\nsigning {\n sign configurations.archives\n}\n\nuploadArchives {\n repositories {\n mavenDeployer {\n beforeDeployment { MavenDeployment deployment -> signing.signPom(deployment) }\n\n repository(url: \"https://ossrh-staging-api.central.sonatype.com/service/local/staging/deploy/maven2\") {\n authentication(userName: ossrhUsername, password: ossrhPassword)\n }\n\n snapshotRepository(url: \"https://ossrh-staging-api.central.sonatype.com/content/repositories/snapshots\") {\n authentication(userName: ossrhUsername, password: ossrhPassword)\n }\n\n pom.project {\n name 'Arvados Java SDK'\n packaging 'jar'\n groupId 'org.arvados'\n description 'Arvados Java SDK'\n url 'https://github.com/arvados/arvados'\n \n scm {\n url 'scm:git@https://github.com/arvados/arvados.git'\n connection 'scm:git@https://github.com/arvados/arvados.git'\n developerConnection 'scm:git@https://github.com/arvados/arvados.git'\n }\n\n licenses {\n license {\n name 'The Apache License, Version 2.0'\n url 'http://www.apache.org/licenses/LICENSE-2.0.txt'\n }\n }\n\n developers {\n developer {\n id 'veritasgenetics'\n name 'Veritas Genetics'\n email 'ops@veritasgenetics.com'\n }\n }\n }\n }\n }\n}\n" }, { "path": "contrib/java-sdk-v2/gradle.properties", "content": "/*\n Copyright (C) The Arvados Authors. All rights reserved.\n\n SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n*/\n\nossrhUsername = ''\nossrhPassword = ''\n" }, { "path": "contrib/java-sdk-v2/settings.gradle", "content": "rootProject.name = 'arvados-java-sdk'\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/client/BaseApiClient.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.client;\n\nimport com.fasterxml.jackson.databind.ObjectMapper;\nimport org.arvados.client.exception.ArvadosApiException;\nimport org.arvados.client.api.client.factory.OkHttpClientFactory;\nimport org.arvados.client.api.model.ApiError;\nimport org.arvados.client.config.ConfigProvider;\nimport okhttp3.OkHttpClient;\nimport okhttp3.Request;\nimport okhttp3.Response;\nimport okhttp3.ResponseBody;\nimport org.slf4j.Logger;\n\nimport java.io.IOException;\nimport java.io.UnsupportedEncodingException;\nimport java.net.URLDecoder;\nimport java.nio.charset.StandardCharsets;\nimport java.util.Objects;\nimport java.util.concurrent.TimeUnit;\n\nabstract class BaseApiClient {\n\n static final ObjectMapper MAPPER = new ObjectMapper().findAndRegisterModules();\n\n final OkHttpClient client;\n final ConfigProvider config;\n private final Logger log = org.slf4j.LoggerFactory.getLogger(BaseApiClient.class);\n\n BaseApiClient(ConfigProvider config) {\n this.config = config;\n this.client = OkHttpClientFactory.INSTANCE.create(config.isApiHostInsecure())\n\t .newBuilder()\n\t .connectTimeout(config.getConnectTimeout(), TimeUnit.MILLISECONDS)\n\t .readTimeout(config.getReadTimeout(), TimeUnit.MILLISECONDS)\n\t .writeTimeout(config.getWriteTimeout(), TimeUnit.MILLISECONDS)\n\t .build();\n }\n\n Request.Builder getRequestBuilder() {\n return new Request.Builder()\n .addHeader(\"authorization\", String.format(\"Bearer %s\", config.getApiToken()))\n .addHeader(\"cache-control\", \"no-cache\");\n }\n\n String newCall(Request request) {\n return (String) getResponseBody(request, body -> body.string().trim());\n }\n\n byte[] newFileCall(Request request) {\n return (byte[]) getResponseBody(request, ResponseBody::bytes);\n }\n\n private Object getResponseBody(Request request, Command command) {\n try {\n log.debug(URLDecoder.decode(request.toString(), StandardCharsets.UTF_8.name()));\n } catch (UnsupportedEncodingException e) {\n throw new ArvadosApiException(e);\n }\n\n try (Response response = client.newCall(request).execute()) {\n ResponseBody responseBody = response.body();\n\n if (!response.isSuccessful()) {\n String errorBody = Objects.requireNonNull(responseBody).string();\n if (errorBody == null || errorBody.length() == 0) {\n throw new ArvadosApiException(String.format(\"Error code %s with message: %s\", response.code(), response.message()));\n }\n ApiError apiError = MAPPER.readValue(errorBody, ApiError.class);\n throw new ArvadosApiException(String.format(\"Error code %s with messages: %s\", response.code(), apiError.getErrors()));\n }\n return command.readResponseBody(responseBody);\n } catch (IOException e) {\n throw new ArvadosApiException(e);\n }\n }\n\n private interface Command {\n Object readResponseBody(ResponseBody body) throws IOException;\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/client/BaseStandardApiClient.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.client;\n\nimport com.fasterxml.jackson.core.JsonProcessingException;\nimport com.fasterxml.jackson.core.type.TypeReference;\nimport com.fasterxml.jackson.databind.ObjectWriter;\nimport okhttp3.MediaType;\nimport okhttp3.HttpUrl;\nimport okhttp3.HttpUrl.Builder;\nimport okhttp3.Request;\nimport okhttp3.RequestBody;\nimport org.arvados.client.exception.ArvadosApiException;\nimport org.arvados.client.api.model.Item;\nimport org.arvados.client.api.model.ItemList;\nimport org.arvados.client.api.model.argument.ListArgument;\nimport org.arvados.client.config.ConfigProvider;\nimport org.slf4j.Logger;\n\nimport java.io.IOException;\nimport java.util.Map;\n\npublic abstract class BaseStandardApiClient extends BaseApiClient {\n\n protected static final MediaType JSON = MediaType.parse(com.google.common.net.MediaType.JSON_UTF_8.toString());\n private final Logger log = org.slf4j.LoggerFactory.getLogger(BaseStandardApiClient.class);\n\n BaseStandardApiClient(ConfigProvider config) {\n super(config);\n }\n\n public L list(ListArgument listArguments) {\n log.debug(\"Get list of {}\", getType().getSimpleName());\n Builder urlBuilder = getUrlBuilder();\n addQueryParameters(urlBuilder, listArguments);\n HttpUrl url = urlBuilder.build();\n Request request = getRequestBuilder().url(url).build();\n return callForList(request);\n }\n \n public L list() {\n return list(ListArgument.builder().build());\n }\n\n public T get(String uuid) {\n log.debug(\"Get {} by UUID {}\", getType().getSimpleName(), uuid);\n HttpUrl url = getUrlBuilder().addPathSegment(uuid).build();\n Request request = getRequestBuilder().get().url(url).build();\n return callForType(request);\n }\n\n public T create(T type) {\n log.debug(\"Create {}\", getType().getSimpleName());\n String json = mapToJson(type);\n RequestBody body = RequestBody.create(JSON, json);\n Request request = getRequestBuilder().post(body).build();\n return callForType(request);\n }\n\n public T delete(String uuid) {\n log.debug(\"Delete {} by UUID {}\", getType().getSimpleName(), uuid);\n HttpUrl url = getUrlBuilder().addPathSegment(uuid).build();\n Request request = getRequestBuilder().delete().url(url).build();\n return callForType(request);\n }\n\n public T update(T type) {\n String uuid = type.getUuid();\n log.debug(\"Update {} by UUID {}\", getType().getSimpleName(), uuid);\n String json = mapToJson(type);\n RequestBody body = RequestBody.create(JSON, json);\n HttpUrl url = getUrlBuilder().addPathSegment(uuid).build();\n Request request = getRequestBuilder().put(body).url(url).build();\n return callForType(request);\n }\n\n @Override\n Request.Builder getRequestBuilder() {\n return super.getRequestBuilder().url(getUrlBuilder().build());\n }\n\n HttpUrl.Builder getUrlBuilder() {\n return new HttpUrl.Builder()\n .scheme(config.getApiProtocol())\n .host(config.getApiHost())\n .port(config.getApiPort())\n .addPathSegment(\"arvados\")\n .addPathSegment(\"v1\")\n .addPathSegment(getResource());\n }\n\n TL call(Request request, Class cls) {\n String bodyAsString = newCall(request);\n try {\n return mapToObject(bodyAsString, cls);\n } catch (IOException e) {\n throw new ArvadosApiException(\"A problem occurred while parsing JSON data\", e);\n }\n }\n\n private TL mapToObject(String content, Class cls) throws IOException {\n return MAPPER.readValue(content, cls);\n }\n\n protected String mapToJson(TL type) {\n ObjectWriter writer = MAPPER.writer().withDefaultPrettyPrinter();\n try {\n return writer.writeValueAsString(type);\n } catch (JsonProcessingException e) {\n log.error(e.getMessage());\n return null;\n }\n }\n\n T callForType(Request request) {\n return call(request, getType());\n }\n\n L callForList(Request request) {\n return call(request, getListType());\n }\n\n abstract String getResource();\n\n abstract Class getType();\n\n abstract Class getListType();\n \n Request getNoArgumentMethodRequest(String method) {\n HttpUrl url = getUrlBuilder().addPathSegment(method).build();\n return getRequestBuilder().get().url(url).build();\n }\n \n RequestBody getJsonRequestBody(Object object) {\n return RequestBody.create(JSON, mapToJson(object));\n }\n \n void addQueryParameters(Builder urlBuilder, Object object) {\n Map queryMap = MAPPER.convertValue(object, new TypeReference>() {});\n queryMap.keySet().forEach(key -> {\n Object type = queryMap.get(key);\n if (!(type instanceof String)) {\n type = mapToJson(type);\n }\n urlBuilder.addQueryParameter(key, (String) type);\n });\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/client/CollectionsApiClient.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.client;\n\nimport org.arvados.client.api.model.Collection;\nimport org.arvados.client.api.model.CollectionList;\nimport org.arvados.client.api.model.CollectionReplaceFiles;\nimport org.arvados.client.config.ConfigProvider;\nimport org.slf4j.Logger;\n\nimport okhttp3.HttpUrl;\nimport okhttp3.Request;\nimport okhttp3.RequestBody;\n\npublic class CollectionsApiClient extends BaseStandardApiClient {\n\n private static final String RESOURCE = \"collections\";\n\n private final Logger log = org.slf4j.LoggerFactory.getLogger(CollectionsApiClient.class);\n\n public CollectionsApiClient(ConfigProvider config) {\n super(config);\n }\n \n @Override\n public Collection create(Collection type) {\n Collection newCollection = super.create(type);\n log.debug(String.format(\"New collection '%s' with UUID %s has been created\", newCollection.getName(), newCollection.getUuid()));\n return newCollection;\n }\n\n public Collection update(String collectionUUID, CollectionReplaceFiles replaceFilesRequest) {\n String json = mapToJson(replaceFilesRequest);\n RequestBody body = RequestBody.create(JSON, json);\n HttpUrl url = getUrlBuilder().addPathSegment(collectionUUID).build();\n Request request = getRequestBuilder().put(body).url(url).build();\n return callForType(request);\n }\n\n @Override\n String getResource() {\n return RESOURCE;\n }\n\n @Override\n Class getType() {\n return Collection.class;\n }\n\n @Override\n Class getListType() {\n return CollectionList.class;\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/client/ConfigApiClient.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.client;\n\nimport com.fasterxml.jackson.databind.ObjectMapper;\nimport okhttp3.OkHttpClient;\nimport okhttp3.Request;\nimport okhttp3.Response;\nimport org.arvados.client.api.client.factory.OkHttpClientFactory;\nimport org.arvados.client.api.model.ArvadosConfig;\nimport org.arvados.client.exception.ArvadosApiException;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport java.io.IOException;\nimport java.util.concurrent.TimeUnit;\n\npublic class ConfigApiClient {\n\n private static final Logger log = LoggerFactory.getLogger(ConfigApiClient.class);\n private static final ObjectMapper MAPPER = new ObjectMapper().findAndRegisterModules();\n private static final String CONFIG_ENDPOINT = \"/arvados/v1/config\";\n\n private final OkHttpClient client;\n private final String baseUrl;\n\n public ConfigApiClient(String protocol, String host, int port, boolean insecure) {\n this.baseUrl = String.format(\"%s://%s:%d\", protocol, host, port);\n this.client = OkHttpClientFactory.INSTANCE.create(insecure)\n .newBuilder()\n .connectTimeout(10, TimeUnit.SECONDS)\n .readTimeout(10, TimeUnit.SECONDS)\n .build();\n }\n\n public ArvadosConfig fetchConfig() throws ArvadosApiException {\n String url = baseUrl + CONFIG_ENDPOINT;\n Request request = new Request.Builder()\n .url(url)\n .get()\n .build();\n\n try (Response response = client.newCall(request).execute()) {\n if (!response.isSuccessful()) {\n String errorMessage = String.format(\"Failed to fetch config from %s. Status: %d\",\n url, response.code());\n log.error(errorMessage);\n throw new ArvadosApiException(errorMessage);\n }\n\n String responseBody = response.body() != null ? response.body().string() : \"\";\n return MAPPER.readValue(responseBody, ArvadosConfig.class);\n\n } catch (IOException e) {\n String errorMessage = String.format(\"Error fetching config from %s: %s\",\n url, e.getMessage());\n log.error(errorMessage, e);\n throw new ArvadosApiException(errorMessage, e);\n }\n }\n}" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/client/CountingFileRequestBody.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.client;\n\nimport okio.BufferedSink;\nimport okio.Okio;\nimport okio.Source;\n\nimport java.io.File;\n\n/**\n * Based on:\n * {@link} https://gist.github.com/eduardb/dd2dc530afd37108e1ac\n */\npublic class CountingFileRequestBody extends CountingRequestBody {\n\n CountingFileRequestBody(final File file, final ProgressListener listener) {\n super(file, listener);\n }\n\n @Override\n public long contentLength() {\n return requestBodyData.length();\n }\n\n @Override\n public void writeTo(BufferedSink sink) {\n try (Source source = Okio.source(requestBodyData)) {\n long total = 0;\n long read;\n\n while ((read = source.read(sink.buffer(), SEGMENT_SIZE)) != -1) {\n total += read;\n sink.flush();\n listener.updateProgress(total);\n\n }\n } catch (RuntimeException rethrown) {\n throw rethrown;\n } catch (Exception ignored) {\n //ignore\n }\n }\n}" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/client/CountingRequestBody.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.client;\n\nimport okhttp3.MediaType;\nimport okhttp3.RequestBody;\nimport org.slf4j.Logger;\n\nabstract class CountingRequestBody extends RequestBody {\n\n protected static final int SEGMENT_SIZE = 2048; // okio.Segment.SIZE\n protected static final MediaType CONTENT_BINARY = MediaType.parse(com.google.common.net.MediaType.OCTET_STREAM.toString());\n\n protected final ProgressListener listener;\n\n protected final T requestBodyData;\n\n CountingRequestBody(T file, final ProgressListener listener) {\n this.requestBodyData = file;\n this.listener = listener;\n }\n\n @Override\n public MediaType contentType() {\n return CONTENT_BINARY;\n }\n\n static class TransferData {\n\n private final Logger log = org.slf4j.LoggerFactory.getLogger(TransferData.class);\n private int progressValue;\n private long totalSize;\n\n TransferData(long totalSize) {\n this.progressValue = 0;\n this.totalSize = totalSize;\n }\n\n void updateTransferProgress(long transferred) {\n float progress = (transferred / (float) totalSize) * 100;\n if (progressValue != (int) progress) {\n progressValue = (int) progress;\n log.debug(\"{} / {} / {}%\", transferred, totalSize, progressValue);\n }\n }\n }\n}" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/client/CountingStreamRequestBody.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.client;\n\nimport okio.BufferedSink;\nimport okio.Okio;\nimport okio.Source;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.io.InputStream;\n\npublic class CountingStreamRequestBody extends CountingRequestBody {\n\n CountingStreamRequestBody(final InputStream inputStream, final ProgressListener listener) {\n super(inputStream, listener);\n }\n\n @Override\n public long contentLength() throws IOException {\n return requestBodyData.available();\n }\n\n @Override\n public void writeTo(BufferedSink sink) {\n try (Source source = Okio.source(requestBodyData)) {\n long total = 0;\n long read;\n\n while ((read = source.read(sink.buffer(), SEGMENT_SIZE)) != -1) {\n total += read;\n sink.flush();\n listener.updateProgress(total);\n\n }\n } catch (RuntimeException rethrown) {\n throw rethrown;\n } catch (Exception ignored) {\n //ignore\n }\n }\n}" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/client/GroupsApiClient.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.client;\n\nimport okhttp3.HttpUrl;\nimport okhttp3.HttpUrl.Builder;\nimport okhttp3.Request;\nimport okhttp3.RequestBody;\nimport org.arvados.client.api.model.Group;\nimport org.arvados.client.api.model.GroupList;\nimport org.arvados.client.api.model.argument.ContentsGroup;\nimport org.arvados.client.api.model.argument.ListArgument;\nimport org.arvados.client.api.model.argument.UntrashGroup;\nimport org.arvados.client.config.ConfigProvider;\nimport org.slf4j.Logger;\n\npublic class GroupsApiClient extends BaseStandardApiClient {\n\n private static final String RESOURCE = \"groups\";\n private final Logger log = org.slf4j.LoggerFactory.getLogger(GroupsApiClient.class);\n\n public GroupsApiClient(ConfigProvider config) {\n super(config);\n }\n\n public GroupList contents(ContentsGroup contentsGroup) {\n log.debug(\"Get {} contents\", getType().getSimpleName());\n Builder urlBuilder = getUrlBuilder().addPathSegment(\"contents\");\n addQueryParameters(urlBuilder, contentsGroup);\n HttpUrl url = urlBuilder.build();\n Request request = getRequestBuilder().url(url).build();\n return callForList(request);\n }\n\n public GroupList contents(ListArgument listArguments) {\n this.log.debug(\"Get {} contents\", this.getType().getSimpleName());\n HttpUrl.Builder urlBuilder = this.getUrlBuilder().addPathSegment(\"contents\");\n this.addQueryParameters(urlBuilder, listArguments);\n HttpUrl url = urlBuilder.build();\n Request request = this.getRequestBuilder().url(url).build();\n return callForList(request);\n }\n\n public Group untrash(UntrashGroup untrashGroup) {\n log.debug(\"Untrash {} by UUID {}\", getType().getSimpleName(), untrashGroup.getUuid());\n HttpUrl url = getUrlBuilder().addPathSegment(untrashGroup.getUuid()).addPathSegment(\"untrash\").build();\n RequestBody requestBody = getJsonRequestBody(untrashGroup);\n Request request = getRequestBuilder().post(requestBody).url(url).build();\n return callForType(request);\n }\n\n @Override\n String getResource() {\n return RESOURCE;\n }\n\n @Override\n Class getType() {\n return Group.class;\n }\n\n @Override\n Class getListType() {\n return GroupList.class;\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/client/KeepWebApiClient.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.client;\n\nimport okhttp3.HttpUrl;\nimport okhttp3.Request;\nimport okhttp3.RequestBody;\nimport okhttp3.Response;\nimport okhttp3.ResponseBody;\n\nimport org.arvados.client.config.ConfigProvider;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.io.InputStream;\n\npublic class KeepWebApiClient extends BaseApiClient {\n\n public KeepWebApiClient(ConfigProvider config) {\n super(config);\n }\n\n public byte[] download(String collectionUuid, String filePathName) {\n Request request = getRequestBuilder()\n .url(getUrlBuilder(collectionUuid,filePathName).build())\n .get()\n .build();\n\n return newFileCall(request);\n }\n\n public InputStream get(String collectionUuid, String filePathName, long start, Long end) throws IOException {\n Request.Builder builder = this.getRequestBuilder();\n String rangeValue = \"bytes=\" + start + \"-\";\n if (end != null) {\n rangeValue += end;\n }\n builder.addHeader(\"Range\", rangeValue);\n Request request = builder.url(this.getUrlBuilder(collectionUuid, filePathName).build()).get().build();\n Response response = client.newCall(request).execute();\n if (!response.isSuccessful()) {\n response.close();\n throw new IOException(\"Failed to download file: \" + response);\n }\n ResponseBody body = response.body();\n if (body == null) {\n response.close();\n throw new IOException(\"Response body is null for request: \" + request);\n }\n return body.byteStream();\n }\n\n public String delete(String collectionUuid, String filePathName) {\n Request request = getRequestBuilder()\n .url(getUrlBuilder(collectionUuid, filePathName).build())\n .delete()\n .build();\n\n return newCall(request);\n }\n\n public String upload(String collectionUuid, File file, ProgressListener progressListener) {\n RequestBody requestBody = new CountingFileRequestBody(file, progressListener);\n\n Request request = getRequestBuilder()\n .url(getUrlBuilder(collectionUuid, file.getName()).build())\n .put(requestBody)\n .build();\n return newCall(request);\n }\n\n public String upload(String collectionUuid, InputStream inputStream, String fileName, ProgressListener progressListener) {\n RequestBody requestBody = new CountingStreamRequestBody(inputStream, progressListener);\n\n Request request = getRequestBuilder()\n .url(getUrlBuilder(collectionUuid, fileName).build())\n .put(requestBody)\n .build();\n return newCall(request);\n }\n\n private HttpUrl.Builder getUrlBuilder(String collectionUuid, String filePathName) {\n return new HttpUrl.Builder()\n .scheme(config.getApiProtocol())\n .host(config.getKeepWebHost())\n .port(config.getKeepWebPort())\n .addPathSegment(\"c=\" + collectionUuid)\n .addPathSegment(filePathName);\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/client/LinksApiClient.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.client;\n\nimport org.arvados.client.api.model.Link;\nimport org.arvados.client.api.model.LinkList;\nimport org.arvados.client.config.ConfigProvider;\n\npublic class LinksApiClient extends BaseStandardApiClient {\n\n private static final String RESOURCE = \"links\";\n\n public LinksApiClient(ConfigProvider config) {\n super(config);\n }\n\n @Override\n String getResource() {\n return RESOURCE;\n }\n\n @Override\n Class getType() {\n return Link.class;\n }\n\n @Override\n Class getListType() {\n return LinkList.class;\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/client/ProgressListener.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.client;\n\n@FunctionalInterface\npublic interface ProgressListener {\n\n void updateProgress(long uploadedBytes);\n}" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/client/UsersApiClient.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.client;\n\nimport okhttp3.Request;\nimport org.arvados.client.api.model.User;\nimport org.arvados.client.api.model.UserList;\nimport org.arvados.client.config.ConfigProvider;\nimport org.slf4j.Logger;\n\npublic class UsersApiClient extends BaseStandardApiClient {\n\n private static final String RESOURCE = \"users\";\n private final Logger log = org.slf4j.LoggerFactory.getLogger(UsersApiClient.class);\n\n public UsersApiClient(ConfigProvider config) {\n super(config);\n }\n\n public User current() {\n log.debug(\"Get current {}\", getType().getSimpleName());\n Request request = getNoArgumentMethodRequest(\"current\");\n return callForType(request);\n }\n\n public User system() {\n log.debug(\"Get system {}\", getType().getSimpleName());\n Request request = getNoArgumentMethodRequest(\"system\");\n return callForType(request);\n }\n\n @Override\n String getResource() {\n return RESOURCE;\n }\n\n @Override\n Class getType() {\n return User.class;\n }\n\n @Override\n Class getListType() {\n return UserList.class;\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/client/factory/OkHttpClientFactory.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.client.factory;\n\nimport com.google.common.base.Suppliers;\nimport okhttp3.OkHttpClient;\nimport org.arvados.client.exception.ArvadosClientException;\nimport org.slf4j.Logger;\n\nimport javax.net.ssl.SSLContext;\nimport javax.net.ssl.SSLSocketFactory;\nimport javax.net.ssl.TrustManager;\nimport javax.net.ssl.X509TrustManager;\nimport java.security.KeyManagementException;\nimport java.security.NoSuchAlgorithmException;\nimport java.security.SecureRandom;\nimport java.security.cert.X509Certificate;\nimport java.util.function.Supplier;\n\n/**\n * {@link OkHttpClient} instance factory that builds and configures client instances sharing\n * the common resource pool: this is the recommended approach to optimize resource usage.\n */\npublic final class OkHttpClientFactory {\n public static final OkHttpClientFactory INSTANCE = new OkHttpClientFactory();\n private final Logger log = org.slf4j.LoggerFactory.getLogger(OkHttpClientFactory.class);\n private final OkHttpClient clientSecure = new OkHttpClient();\n private final Supplier clientUnsecure =\n Suppliers.memoize(this::getDefaultClientAcceptingAllCertificates);\n\n private OkHttpClientFactory() { /* singleton */}\n\n public OkHttpClient create(boolean apiHostInsecure) {\n return apiHostInsecure ? getDefaultUnsecureClient() : getDefaultClient();\n }\n\n /**\n * @return default secure {@link OkHttpClient} with shared resource pool.\n */\n public OkHttpClient getDefaultClient() {\n return clientSecure;\n }\n\n /**\n * @return default {@link OkHttpClient} with shared resource pool\n * that will accept all SSL certificates by default.\n */\n public OkHttpClient getDefaultUnsecureClient() {\n return clientUnsecure.get();\n }\n\n /**\n * @return default {@link OkHttpClient.Builder} with shared resource pool.\n */\n public OkHttpClient.Builder getDefaultClientBuilder() {\n return clientSecure.newBuilder();\n }\n\n /**\n * @return default {@link OkHttpClient.Builder} with shared resource pool\n * that is preconfigured to accept all SSL certificates.\n */\n public OkHttpClient.Builder getDefaultUnsecureClientBuilder() {\n return clientUnsecure.get().newBuilder();\n }\n\n private OkHttpClient getDefaultClientAcceptingAllCertificates() {\n log.warn(\"Creating unsafe OkHttpClient. All SSL certificates will be accepted.\");\n try {\n // Create a trust manager that does not validate certificate chains\n final TrustManager[] trustAllCerts = {createX509TrustManager()};\n\n // Install the all-trusting trust manager\n SSLContext sslContext = SSLContext.getInstance(\"SSL\");\n sslContext.init(null, trustAllCerts, new SecureRandom());\n // Create an ssl socket factory with our all-trusting manager\n final SSLSocketFactory sslSocketFactory = sslContext.getSocketFactory();\n\n // Create the OkHttpClient.Builder with shared resource pool\n final OkHttpClient.Builder builder = clientSecure.newBuilder();\n builder.sslSocketFactory(sslSocketFactory, (X509TrustManager) trustAllCerts[0]);\n builder.hostnameVerifier((hostname, session) -> true);\n return builder.build();\n } catch (NoSuchAlgorithmException | KeyManagementException e) {\n throw new ArvadosClientException(\"Error establishing SSL context\", e);\n }\n }\n\n private static X509TrustManager createX509TrustManager() {\n return new X509TrustManager() {\n\n @Override\n public void checkClientTrusted(X509Certificate[] chain, String authType) {\n }\n\n @Override\n public void checkServerTrusted(X509Certificate[] chain, String authType) {\n }\n\n @Override\n public X509Certificate[] getAcceptedIssuers() {\n return new X509Certificate[]{};\n }\n };\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/model/ApiError.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.model;\n\nimport com.fasterxml.jackson.annotation.JsonIgnoreProperties;\nimport com.fasterxml.jackson.annotation.JsonInclude;\nimport com.fasterxml.jackson.annotation.JsonProperty;\nimport com.fasterxml.jackson.annotation.JsonPropertyOrder;\n\nimport java.util.List;\n\n@JsonInclude(JsonInclude.Include.NON_NULL)\n@JsonIgnoreProperties(ignoreUnknown = true)\n@JsonPropertyOrder({ \"errors\", \"error_token\" })\npublic class ApiError {\n\n @JsonProperty(\"errors\")\n private List errors;\n @JsonProperty(\"error_token\")\n private String errorToken;\n\n public List getErrors() {\n return this.errors;\n }\n\n public String getErrorToken() {\n return this.errorToken;\n }\n\n public void setErrors(List errors) {\n this.errors = errors;\n }\n\n public void setErrorToken(String errorToken) {\n this.errorToken = errorToken;\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/model/ArvadosConfig.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.model;\n\nimport com.fasterxml.jackson.annotation.JsonIgnoreProperties;\nimport com.fasterxml.jackson.annotation.JsonProperty;\n\n@JsonIgnoreProperties(ignoreUnknown = true)\npublic class ArvadosConfig {\n\n @JsonProperty(\"Services\")\n private Services services;\n\n public Services getServices() {\n return services;\n }\n\n public void setServices(Services services) {\n this.services = services;\n }\n\n @JsonIgnoreProperties(ignoreUnknown = true)\n public static class Services {\n\n @JsonProperty(\"WebDAVDownload\")\n private WebDAVDownload webDAVDownload;\n\n public WebDAVDownload getWebDAVDownload() {\n return webDAVDownload;\n }\n\n public void setWebDAVDownload(WebDAVDownload webDAVDownload) {\n this.webDAVDownload = webDAVDownload;\n }\n }\n\n @JsonIgnoreProperties(ignoreUnknown = true)\n public static class WebDAVDownload {\n\n @JsonProperty(\"ExternalURL\")\n private String externalURL;\n\n public String getExternalURL() {\n return externalURL;\n }\n\n public void setExternalURL(String externalURL) {\n this.externalURL = externalURL;\n }\n }\n}" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/model/Collection.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.model;\n\nimport com.fasterxml.jackson.annotation.JsonIgnoreProperties;\nimport com.fasterxml.jackson.annotation.JsonInclude;\nimport com.fasterxml.jackson.annotation.JsonProperty;\nimport com.fasterxml.jackson.annotation.JsonPropertyOrder;\n\nimport java.time.LocalDateTime;\n\n@JsonInclude(JsonInclude.Include.NON_NULL)\n@JsonIgnoreProperties(ignoreUnknown = true)\n@JsonPropertyOrder({ \"portable_data_hash\", \"replication_desired\", \"replication_confirmed_at\", \"replication_confirmed\", \"manifest_text\", \n \"name\", \"description\", \"properties\", \"delete_at\", \"trash_at\", \"is_trashed\" })\npublic class Collection extends Item {\n\n @JsonProperty(\"portable_data_hash\")\n private String portableDataHash;\n @JsonProperty(\"replication_desired\")\n private Integer replicationDesired;\n @JsonProperty(\"replication_confirmed_at\")\n private LocalDateTime replicationConfirmedAt;\n @JsonProperty(\"replication_confirmed\")\n private Integer replicationConfirmed;\n @JsonProperty(\"manifest_text\")\n private String manifestText;\n @JsonProperty(\"name\")\n private String name;\n @JsonProperty(\"description\")\n private String description;\n @JsonProperty(\"properties\")\n private Object properties;\n @JsonProperty(\"delete_at\")\n private LocalDateTime deleteAt;\n @JsonProperty(\"trash_at\")\n private LocalDateTime trashAt;\n @JsonProperty(\"is_trashed\")\n private Boolean trashed;\n\n public String getPortableDataHash() {\n return this.portableDataHash;\n }\n\n public Integer getReplicationDesired() {\n return this.replicationDesired;\n }\n\n public LocalDateTime getReplicationConfirmedAt() {\n return this.replicationConfirmedAt;\n }\n\n public Integer getReplicationConfirmed() {\n return this.replicationConfirmed;\n }\n\n public String getManifestText() {\n return this.manifestText;\n }\n\n public String getName() {\n return this.name;\n }\n\n public String getDescription() {\n return this.description;\n }\n\n public Object getProperties() {\n return this.properties;\n }\n\n public LocalDateTime getDeleteAt() {\n return this.deleteAt;\n }\n\n public LocalDateTime getTrashAt() {\n return this.trashAt;\n }\n\n public Boolean getTrashed() {\n return this.trashed;\n }\n\n public void setPortableDataHash(String portableDataHash) {\n this.portableDataHash = portableDataHash;\n }\n\n public void setReplicationDesired(Integer replicationDesired) {\n this.replicationDesired = replicationDesired;\n }\n\n public void setReplicationConfirmedAt(LocalDateTime replicationConfirmedAt) {\n this.replicationConfirmedAt = replicationConfirmedAt;\n }\n\n public void setReplicationConfirmed(Integer replicationConfirmed) {\n this.replicationConfirmed = replicationConfirmed;\n }\n\n public void setManifestText(String manifestText) {\n this.manifestText = manifestText;\n }\n\n public void setName(String name) {\n this.name = name;\n }\n\n public void setDescription(String description) {\n this.description = description;\n }\n\n public void setProperties(Object properties) {\n this.properties = properties;\n }\n\n public void setDeleteAt(LocalDateTime deleteAt) {\n this.deleteAt = deleteAt;\n }\n\n public void setTrashAt(LocalDateTime trashAt) {\n this.trashAt = trashAt;\n }\n\n public void setTrashed(Boolean trashed) {\n this.trashed = trashed;\n }\n\n public String toString() {\n return \"Collection(portableDataHash=\" + this.getPortableDataHash() + \", replicationDesired=\" + this.getReplicationDesired() + \", replicationConfirmedAt=\" + this.getReplicationConfirmedAt() + \", replicationConfirmed=\" + this.getReplicationConfirmed() + \", manifestText=\" + this.getManifestText() + \", name=\" + this.getName() + \", description=\" + this.getDescription() + \", properties=\" + this.getProperties() + \", deleteAt=\" + this.getDeleteAt() + \", trashAt=\" + this.getTrashAt() + \", trashed=\" + this.getTrashed() + \")\";\n }\n}" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/model/CollectionList.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.model;\n\nimport com.fasterxml.jackson.annotation.JsonIgnoreProperties;\nimport com.fasterxml.jackson.annotation.JsonInclude;\nimport com.fasterxml.jackson.annotation.JsonProperty;\nimport com.fasterxml.jackson.annotation.JsonPropertyOrder;\n\nimport java.util.List;\n\n@JsonInclude(JsonInclude.Include.NON_NULL)\n@JsonIgnoreProperties(ignoreUnknown = true)\n@JsonPropertyOrder({ \"items\" })\npublic class CollectionList extends ItemList {\n\n @JsonProperty(\"items\")\n private List items;\n\n public List getItems() {\n return this.items;\n }\n\n public void setItems(List items) {\n this.items = items;\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/model/CollectionReplaceFiles.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.model;\n\nimport com.fasterxml.jackson.annotation.JsonInclude;\nimport com.fasterxml.jackson.annotation.JsonIgnoreProperties;\nimport com.fasterxml.jackson.annotation.JsonProperty;\n\nimport java.util.HashMap;\nimport java.util.Map;\n\n@JsonInclude(JsonInclude.Include.NON_NULL)\n@JsonIgnoreProperties(ignoreUnknown = true)\npublic class CollectionReplaceFiles {\n\n @JsonProperty(\"collection\")\n private CollectionOptions collectionOptions;\n\n @JsonProperty(\"replace_files\")\n private Map replaceFiles;\n\n public CollectionReplaceFiles() {\n this.collectionOptions = new CollectionOptions();\n this.replaceFiles = new HashMap<>();\n }\n\n public void addFileReplacement(String targetPath, String sourcePath) {\n this.replaceFiles.put(targetPath, sourcePath);\n }\n\n @JsonInclude(JsonInclude.Include.NON_NULL)\n @JsonIgnoreProperties(ignoreUnknown = true)\n public static class CollectionOptions {\n @JsonProperty(\"preserve_version\")\n private boolean preserveVersion;\n\n public CollectionOptions() {\n this.preserveVersion = true;\n }\n\n public boolean isPreserveVersion() {\n return preserveVersion;\n }\n\n public void setPreserveVersion(boolean preserveVersion) {\n this.preserveVersion = preserveVersion;\n }\n }\n\n public CollectionOptions getCollectionOptions() {\n return collectionOptions;\n }\n\n public void setCollectionOptions(CollectionOptions collectionOptions) {\n this.collectionOptions = collectionOptions;\n }\n\n public Map getReplaceFiles() {\n return replaceFiles;\n }\n\n public void setReplaceFiles(Map replaceFiles) {\n this.replaceFiles = replaceFiles;\n }\n}" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/model/Group.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.model;\n\nimport com.fasterxml.jackson.annotation.JsonIgnoreProperties;\nimport com.fasterxml.jackson.annotation.JsonInclude;\nimport com.fasterxml.jackson.annotation.JsonProperty;\nimport com.fasterxml.jackson.annotation.JsonPropertyOrder;\n\nimport java.time.LocalDateTime;\nimport java.util.List;\n\n@JsonInclude(JsonInclude.Include.NON_NULL)\n@JsonIgnoreProperties(ignoreUnknown = true)\n@JsonPropertyOrder({ \"command\", \"container_count\", \"container_count_max\", \"container_image\", \"container_uuid\", \"cwd\", \"environment\", \"expires_at\", \n \"filters\", \"log_uuid\", \"mounts\", \"output_name\", \"output_path\", \"output_uuid\", \"output_ttl\", \"priority\", \"properties\", \"requesting_container_uuid\", \n \"runtime_constraints\", \"scheduling_parameters\", \"state\", \"use_existing\" })\npublic class Group extends Item {\n\n @JsonProperty(\"name\")\n private String name;\n @JsonProperty(\"group_class\")\n private String groupClass;\n @JsonProperty(\"description\")\n private String description;\n @JsonProperty(value = \"writable_by\", access = JsonProperty.Access.WRITE_ONLY)\n private List writableBy;\n @JsonProperty(\"delete_at\")\n private LocalDateTime deleteAt;\n @JsonProperty(\"trash_at\")\n private LocalDateTime trashAt;\n @JsonProperty(\"is_trashed\")\n private Boolean isTrashed;\n @JsonProperty(\"command\")\n private List command;\n @JsonProperty(\"container_count\")\n private Integer containerCount;\n @JsonProperty(\"container_count_max\")\n private Integer containerCountMax;\n @JsonProperty(\"container_image\")\n private String containerImage;\n @JsonProperty(\"container_uuid\")\n private String containerUuid;\n @JsonProperty(\"cwd\")\n private String cwd;\n @JsonProperty(\"environment\")\n private Object environment;\n @JsonProperty(\"expires_at\")\n private LocalDateTime expiresAt;\n @JsonProperty(\"filters\")\n private List filters;\n @JsonProperty(\"log_uuid\")\n private String logUuid;\n @JsonProperty(\"mounts\")\n private Object mounts;\n @JsonProperty(\"output_name\")\n private String outputName;\n @JsonProperty(\"output_path\")\n private String outputPath;\n @JsonProperty(\"output_uuid\")\n private String outputUuid;\n @JsonProperty(\"output_ttl\")\n private Integer outputTtl;\n @JsonProperty(\"priority\")\n private Integer priority;\n @JsonProperty(\"properties\")\n private Object properties;\n @JsonProperty(\"requesting_container_uuid\")\n private String requestingContainerUuid;\n @JsonProperty(\"runtime_constraints\")\n private RuntimeConstraints runtimeConstraints;\n @JsonProperty(\"scheduling_parameters\")\n private Object schedulingParameters;\n @JsonProperty(\"state\")\n private String state;\n @JsonProperty(\"use_existing\")\n private Boolean useExisting;\n\n public String getName() {\n return this.name;\n }\n\n public String getGroupClass() {\n return this.groupClass;\n }\n\n public String getDescription() {\n return this.description;\n }\n\n public List getWritableBy() {\n return this.writableBy;\n }\n\n public LocalDateTime getDeleteAt() {\n return this.deleteAt;\n }\n\n public LocalDateTime getTrashAt() {\n return this.trashAt;\n }\n\n public Boolean getIsTrashed() {\n return this.isTrashed;\n }\n\n public List getCommand() {\n return this.command;\n }\n\n public Integer getContainerCount() {\n return this.containerCount;\n }\n\n public Integer getContainerCountMax() {\n return this.containerCountMax;\n }\n\n public String getContainerImage() {\n return this.containerImage;\n }\n\n public String getContainerUuid() {\n return this.containerUuid;\n }\n\n public String getCwd() {\n return this.cwd;\n }\n\n public Object getEnvironment() {\n return this.environment;\n }\n\n public LocalDateTime getExpiresAt() {\n return this.expiresAt;\n }\n\n public List getFilters() {\n return this.filters;\n }\n\n public String getLogUuid() {\n return this.logUuid;\n }\n\n public Object getMounts() {\n return this.mounts;\n }\n\n public String getOutputName() {\n return this.outputName;\n }\n\n public String getOutputPath() {\n return this.outputPath;\n }\n\n public String getOutputUuid() {\n return this.outputUuid;\n }\n\n public Integer getOutputTtl() {\n return this.outputTtl;\n }\n\n public Integer getPriority() {\n return this.priority;\n }\n\n public Object getProperties() {\n return this.properties;\n }\n\n public String getRequestingContainerUuid() {\n return this.requestingContainerUuid;\n }\n\n public RuntimeConstraints getRuntimeConstraints() {\n return this.runtimeConstraints;\n }\n\n public Object getSchedulingParameters() {\n return this.schedulingParameters;\n }\n\n public String getState() {\n return this.state;\n }\n\n public Boolean getUseExisting() {\n return this.useExisting;\n }\n\n public void setName(String name) {\n this.name = name;\n }\n\n public void setGroupClass(String groupClass) {\n this.groupClass = groupClass;\n }\n\n public void setDescription(String description) {\n this.description = description;\n }\n\n public void setWritableBy(List writableBy) {\n this.writableBy = writableBy;\n }\n\n public void setDeleteAt(LocalDateTime deleteAt) {\n this.deleteAt = deleteAt;\n }\n\n public void setTrashAt(LocalDateTime trashAt) {\n this.trashAt = trashAt;\n }\n\n public void setIsTrashed(Boolean isTrashed) {\n this.isTrashed = isTrashed;\n }\n\n public void setCommand(List command) {\n this.command = command;\n }\n\n public void setContainerCount(Integer containerCount) {\n this.containerCount = containerCount;\n }\n\n public void setContainerCountMax(Integer containerCountMax) {\n this.containerCountMax = containerCountMax;\n }\n\n public void setContainerImage(String containerImage) {\n this.containerImage = containerImage;\n }\n\n public void setContainerUuid(String containerUuid) {\n this.containerUuid = containerUuid;\n }\n\n public void setCwd(String cwd) {\n this.cwd = cwd;\n }\n\n public void setEnvironment(Object environment) {\n this.environment = environment;\n }\n\n public void setExpiresAt(LocalDateTime expiresAt) {\n this.expiresAt = expiresAt;\n }\n\n public void setFilters(List filters) {\n this.filters = filters;\n }\n\n public void setLogUuid(String logUuid) {\n this.logUuid = logUuid;\n }\n\n public void setMounts(Object mounts) {\n this.mounts = mounts;\n }\n\n public void setOutputName(String outputName) {\n this.outputName = outputName;\n }\n\n public void setOutputPath(String outputPath) {\n this.outputPath = outputPath;\n }\n\n public void setOutputUuid(String outputUuid) {\n this.outputUuid = outputUuid;\n }\n\n public void setOutputTtl(Integer outputTtl) {\n this.outputTtl = outputTtl;\n }\n\n public void setPriority(Integer priority) {\n this.priority = priority;\n }\n\n public void setProperties(Object properties) {\n this.properties = properties;\n }\n\n public void setRequestingContainerUuid(String requestingContainerUuid) {\n this.requestingContainerUuid = requestingContainerUuid;\n }\n\n public void setRuntimeConstraints(RuntimeConstraints runtimeConstraints) {\n this.runtimeConstraints = runtimeConstraints;\n }\n\n public void setSchedulingParameters(Object schedulingParameters) {\n this.schedulingParameters = schedulingParameters;\n }\n\n public void setState(String state) {\n this.state = state;\n }\n\n public void setUseExisting(Boolean useExisting) {\n this.useExisting = useExisting;\n }\n\n public String toString() {\n return \"Group(name=\" + this.getName() + \", groupClass=\" + this.getGroupClass() + \", description=\" + this.getDescription() + \", writableBy=\" + this.getWritableBy() + \", deleteAt=\" + this.getDeleteAt() + \", trashAt=\" + this.getTrashAt() + \", isTrashed=\" + this.getIsTrashed() + \", command=\" + this.getCommand() + \", containerCount=\" + this.getContainerCount() + \", containerCountMax=\" + this.getContainerCountMax() + \", containerImage=\" + this.getContainerImage() + \", containerUuid=\" + this.getContainerUuid() + \", cwd=\" + this.getCwd() + \", environment=\" + this.getEnvironment() + \", expiresAt=\" + this.getExpiresAt() + \", filters=\" + this.getFilters() + \", logUuid=\" + this.getLogUuid() + \", mounts=\" + this.getMounts() + \", outputName=\" + this.getOutputName() + \", outputPath=\" + this.getOutputPath() + \", outputUuid=\" + this.getOutputUuid() + \", outputTtl=\" + this.getOutputTtl() + \", priority=\" + this.getPriority() + \", properties=\" + this.getProperties() + \", requestingContainerUuid=\" + this.getRequestingContainerUuid() + \", runtimeConstraints=\" + this.getRuntimeConstraints() + \", schedulingParameters=\" + this.getSchedulingParameters() + \", state=\" + this.getState() + \", useExisting=\" + this.getUseExisting() + \")\";\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/model/GroupList.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.model;\n\nimport com.fasterxml.jackson.annotation.JsonIgnoreProperties;\nimport com.fasterxml.jackson.annotation.JsonInclude;\nimport com.fasterxml.jackson.annotation.JsonProperty;\nimport com.fasterxml.jackson.annotation.JsonPropertyOrder;\n\nimport java.util.List;\n\n@JsonInclude(JsonInclude.Include.NON_NULL)\n@JsonIgnoreProperties(ignoreUnknown = true)\n@JsonPropertyOrder({ \"items\" })\npublic class GroupList extends ItemList {\n\n @JsonProperty(\"items\")\n private List items;\n\n public List getItems() {\n return this.items;\n }\n\n public void setItems(List items) {\n this.items = items;\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/model/Item.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.model;\n\nimport com.fasterxml.jackson.annotation.JsonIgnoreProperties;\nimport com.fasterxml.jackson.annotation.JsonInclude;\nimport com.fasterxml.jackson.annotation.JsonProperty;\nimport com.fasterxml.jackson.annotation.JsonPropertyOrder;\n\nimport java.time.LocalDateTime;\n\n@JsonInclude(JsonInclude.Include.NON_NULL)\n@JsonIgnoreProperties(ignoreUnknown = true)\n@JsonPropertyOrder({ \"kind\", \"etag\", \"uuid\", \"owner_uuid\", \"created_at\", \"modified_by_client_uuid\",\n \"modified_by_user_uuid\", \"modified_at\", \"updated_at\" })\npublic abstract class Item {\n\n @JsonProperty(\"kind\")\n private String kind;\n @JsonProperty(\"etag\")\n private String etag;\n @JsonProperty(\"uuid\")\n private String uuid;\n @JsonProperty(\"owner_uuid\")\n private String ownerUuid;\n @JsonProperty(\"created_at\")\n private LocalDateTime createdAt;\n @JsonProperty(\"modified_by_client_uuid\")\n private String modifiedByClientUuid;\n @JsonProperty(\"modified_by_user_uuid\")\n private String modifiedByUserUuid;\n @JsonProperty(\"modified_at\")\n private LocalDateTime modifiedAt;\n @JsonProperty(\"updated_at\")\n private LocalDateTime updatedAt;\n\n public String getKind() {\n return this.kind;\n }\n\n public String getEtag() {\n return this.etag;\n }\n\n public String getUuid() {\n return this.uuid;\n }\n\n public String getOwnerUuid() {\n return this.ownerUuid;\n }\n\n public LocalDateTime getCreatedAt() {\n return this.createdAt;\n }\n\n public String getModifiedByClientUuid() {\n return this.modifiedByClientUuid;\n }\n\n public String getModifiedByUserUuid() {\n return this.modifiedByUserUuid;\n }\n\n public LocalDateTime getModifiedAt() {\n return this.modifiedAt;\n }\n\n public LocalDateTime getUpdatedAt() {\n return this.updatedAt;\n }\n\n public void setKind(String kind) {\n this.kind = kind;\n }\n\n public void setEtag(String etag) {\n this.etag = etag;\n }\n\n public void setUuid(String uuid) {\n this.uuid = uuid;\n }\n\n public void setOwnerUuid(String ownerUuid) {\n this.ownerUuid = ownerUuid;\n }\n\n public void setCreatedAt(LocalDateTime createdAt) {\n this.createdAt = createdAt;\n }\n\n public void setModifiedByClientUuid(String modifiedByClientUuid) {\n this.modifiedByClientUuid = modifiedByClientUuid;\n }\n\n public void setModifiedByUserUuid(String modifiedByUserUuid) {\n this.modifiedByUserUuid = modifiedByUserUuid;\n }\n\n public void setModifiedAt(LocalDateTime modifiedAt) {\n this.modifiedAt = modifiedAt;\n }\n\n public void setUpdatedAt(LocalDateTime updatedAt) {\n this.updatedAt = updatedAt;\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/model/ItemList.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.model;\n\nimport com.fasterxml.jackson.annotation.JsonIgnoreProperties;\nimport com.fasterxml.jackson.annotation.JsonInclude;\nimport com.fasterxml.jackson.annotation.JsonProperty;\nimport com.fasterxml.jackson.annotation.JsonPropertyOrder;\n\n@JsonInclude(JsonInclude.Include.NON_NULL)\n@JsonIgnoreProperties(ignoreUnknown = true)\n@JsonPropertyOrder({ \"kind\", \"etag\", \"offset\", \"limit\", \"items_available\" })\npublic class ItemList {\n\n @JsonProperty(\"kind\")\n private String kind;\n @JsonProperty(\"etag\")\n private String etag;\n @JsonProperty(\"offset\")\n private Object offset;\n @JsonProperty(\"limit\")\n private Object limit;\n @JsonProperty(\"items_available\")\n private Integer itemsAvailable;\n\n public String getKind() {\n return this.kind;\n }\n\n public String getEtag() {\n return this.etag;\n }\n\n public Object getOffset() {\n return this.offset;\n }\n\n public Object getLimit() {\n return this.limit;\n }\n\n public Integer getItemsAvailable() {\n return this.itemsAvailable;\n }\n\n public void setKind(String kind) {\n this.kind = kind;\n }\n\n public void setEtag(String etag) {\n this.etag = etag;\n }\n\n public void setOffset(Object offset) {\n this.offset = offset;\n }\n\n public void setLimit(Object limit) {\n this.limit = limit;\n }\n\n public void setItemsAvailable(Integer itemsAvailable) {\n this.itemsAvailable = itemsAvailable;\n }\n}" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/model/KeepService.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.model;\n\nimport com.fasterxml.jackson.annotation.*;\n\n@JsonInclude(JsonInclude.Include.NON_NULL)\n@JsonIgnoreProperties(ignoreUnknown = true)\n@JsonPropertyOrder({ \"service_host\", \"service_port\", \"service_ssl_flag\", \"service_type\", \"read_only\" })\npublic class KeepService extends Item {\n\n @JsonProperty(\"service_host\")\n private String serviceHost;\n @JsonProperty(\"service_port\")\n private Integer servicePort;\n @JsonProperty(\"service_ssl_flag\")\n private Boolean serviceSslFlag;\n @JsonProperty(\"service_type\")\n private String serviceType;\n @JsonProperty(\"read_only\")\n private Boolean readOnly;\n @JsonIgnore\n private String serviceRoot;\n\n public String getServiceHost() {\n return this.serviceHost;\n }\n\n public Integer getServicePort() {\n return this.servicePort;\n }\n\n public Boolean getServiceSslFlag() {\n return this.serviceSslFlag;\n }\n\n public String getServiceType() {\n return this.serviceType;\n }\n\n public Boolean getReadOnly() {\n return this.readOnly;\n }\n\n public String getServiceRoot() {\n return this.serviceRoot;\n }\n\n public void setServiceHost(String serviceHost) {\n this.serviceHost = serviceHost;\n }\n\n public void setServicePort(Integer servicePort) {\n this.servicePort = servicePort;\n }\n\n public void setServiceSslFlag(Boolean serviceSslFlag) {\n this.serviceSslFlag = serviceSslFlag;\n }\n\n public void setServiceType(String serviceType) {\n this.serviceType = serviceType;\n }\n\n public void setReadOnly(Boolean readOnly) {\n this.readOnly = readOnly;\n }\n\n public void setServiceRoot(String serviceRoot) {\n this.serviceRoot = serviceRoot;\n }\n}" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/model/KeepServiceList.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.model;\n\nimport com.fasterxml.jackson.annotation.JsonIgnoreProperties;\nimport com.fasterxml.jackson.annotation.JsonInclude;\nimport com.fasterxml.jackson.annotation.JsonProperty;\nimport com.fasterxml.jackson.annotation.JsonPropertyOrder;\n\nimport java.util.List;\n\n@JsonInclude(JsonInclude.Include.NON_NULL)\n@JsonIgnoreProperties(ignoreUnknown = true)\n@JsonPropertyOrder({ \"items\" })\npublic class KeepServiceList extends ItemList {\n\n @JsonProperty(\"items\")\n private List items;\n\n public List getItems() {\n return this.items;\n }\n\n public void setItems(List items) {\n this.items = items;\n }\n}" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/model/Link.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.model;\n\nimport com.fasterxml.jackson.annotation.JsonIgnoreProperties;\nimport com.fasterxml.jackson.annotation.JsonInclude;\nimport com.fasterxml.jackson.annotation.JsonProperty;\nimport com.fasterxml.jackson.annotation.JsonPropertyOrder;\n\n\n@JsonInclude(JsonInclude.Include.NON_NULL)\n@JsonIgnoreProperties(ignoreUnknown = true)\n@JsonPropertyOrder({\"name\", \"head_kind\", \"head_uuid\", \"link_class\"})\npublic class Link extends Item {\n\n @JsonProperty(\"name\")\n private String name;\n @JsonProperty(value = \"head_kind\", access = JsonProperty.Access.WRITE_ONLY)\n private String headKind;\n @JsonProperty(\"head_uuid\")\n private String headUuid;\n @JsonProperty(\"tail_uuid\")\n private String tailUuid;\n @JsonProperty(value = \"tail_kind\", access = JsonProperty.Access.WRITE_ONLY)\n private String tailKind;\n @JsonProperty(\"link_class\")\n private String linkClass;\n\n public String getName() {\n return name;\n }\n\n public String getHeadKind() {\n return headKind;\n }\n\n public String getHeadUuid() {\n return headUuid;\n }\n\n public String getTailUuid() {\n return tailUuid;\n }\n\n public String getTailKind() {\n return tailKind;\n }\n\n public String getLinkClass() {\n return linkClass;\n }\n\n public void setName(String name) {\n this.name = name;\n }\n\n public void setHeadKind(String headKind) {\n this.headKind = headKind;\n }\n\n public void setHeadUuid(String headUuid) {\n this.headUuid = headUuid;\n }\n\n public void setTailUuid(String tailUuid) {\n this.tailUuid = tailUuid;\n }\n\n public void setTailKind(String tailKind) {\n this.tailKind = tailKind;\n }\n\n public void setLinkClass(String linkClass) {\n this.linkClass = linkClass;\n }\n\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/model/LinkList.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.model;\n\nimport com.fasterxml.jackson.annotation.JsonIgnoreProperties;\nimport com.fasterxml.jackson.annotation.JsonInclude;\nimport com.fasterxml.jackson.annotation.JsonProperty;\nimport com.fasterxml.jackson.annotation.JsonPropertyOrder;\n\nimport java.util.List;\n\n@JsonInclude(JsonInclude.Include.NON_NULL)\n@JsonIgnoreProperties(ignoreUnknown = true)\n@JsonPropertyOrder({ \"items\" })\npublic class LinkList extends ItemList {\n\n @JsonProperty(\"items\")\n private List items;\n\n public List getItems() {\n return this.items;\n }\n\n public void setItems(List items) {\n this.items = items;\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/model/RuntimeConstraints.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.model;\n\nimport com.fasterxml.jackson.annotation.JsonIgnoreProperties;\nimport com.fasterxml.jackson.annotation.JsonInclude;\nimport com.fasterxml.jackson.annotation.JsonProperty;\nimport com.fasterxml.jackson.annotation.JsonPropertyOrder;\n\n@JsonInclude(JsonInclude.Include.NON_NULL)\n@JsonIgnoreProperties(ignoreUnknown = true)\n@JsonPropertyOrder({ \"API\", \"vcpus\", \"ram\", \"keep_cache_ram\" })\npublic class RuntimeConstraints {\n\n @JsonProperty(\"API\")\n private Boolean api;\n @JsonProperty(\"vcpus\")\n private Integer vcpus;\n @JsonProperty(\"ram\")\n private Long ram;\n @JsonProperty(\"keep_cache_ram\")\n private Long keepCacheRam;\n\n public Boolean getApi() {\n return this.api;\n }\n\n public Integer getVcpus() {\n return this.vcpus;\n }\n\n public Long getRam() {\n return this.ram;\n }\n\n public Long getKeepCacheRam() {\n return this.keepCacheRam;\n }\n\n public void setApi(Boolean api) {\n this.api = api;\n }\n\n public void setVcpus(Integer vcpus) {\n this.vcpus = vcpus;\n }\n\n public void setRam(Long ram) {\n this.ram = ram;\n }\n\n public void setKeepCacheRam(Long keepCacheRam) {\n this.keepCacheRam = keepCacheRam;\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/model/User.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.model;\n\nimport com.fasterxml.jackson.annotation.JsonIgnoreProperties;\nimport com.fasterxml.jackson.annotation.JsonInclude;\nimport com.fasterxml.jackson.annotation.JsonProperty;\nimport com.fasterxml.jackson.annotation.JsonPropertyOrder;\n\nimport java.util.List;\n\n@JsonInclude(JsonInclude.Include.NON_NULL)\n@JsonIgnoreProperties(ignoreUnknown = true)\n@JsonPropertyOrder({ \"email\", \"username\", \"full_name\", \"first_name\", \"last_name\", \"identity_url\", \"is_active\", \"is_admin\", \"is_invited\", \n \"prefs\", \"writable_by\" })\npublic class User extends Item {\n\n @JsonProperty(\"email\")\n private String email;\n @JsonProperty(\"username\")\n private String username;\n @JsonProperty(\"full_name\")\n private String fullName;\n @JsonProperty(\"first_name\")\n private String firstName;\n @JsonProperty(\"last_name\")\n private String lastName;\n @JsonProperty(\"identity_url\")\n private String identityUrl;\n @JsonProperty(\"is_active\")\n private Boolean isActive;\n @JsonProperty(\"is_admin\")\n private Boolean isAdmin;\n @JsonProperty(\"is_invited\")\n private Boolean isInvited;\n @JsonProperty(\"prefs\")\n private Object prefs;\n @JsonProperty(\"writable_by\")\n private List writableBy;\n\n public String getEmail() {\n return this.email;\n }\n\n public String getUsername() {\n return this.username;\n }\n\n public String getFullName() {\n return this.fullName;\n }\n\n public String getFirstName() {\n return this.firstName;\n }\n\n public String getLastName() {\n return this.lastName;\n }\n\n public String getIdentityUrl() {\n return this.identityUrl;\n }\n\n public Boolean getIsActive() {\n return this.isActive;\n }\n\n public Boolean getIsAdmin() {\n return this.isAdmin;\n }\n\n public Boolean getIsInvited() {\n return this.isInvited;\n }\n\n public Object getPrefs() {\n return this.prefs;\n }\n\n public List getWritableBy() {\n return this.writableBy;\n }\n\n public void setEmail(String email) {\n this.email = email;\n }\n\n public void setUsername(String username) {\n this.username = username;\n }\n\n public void setFullName(String fullName) {\n this.fullName = fullName;\n }\n\n public void setFirstName(String firstName) {\n this.firstName = firstName;\n }\n\n public void setLastName(String lastName) {\n this.lastName = lastName;\n }\n\n public void setIdentityUrl(String identityUrl) {\n this.identityUrl = identityUrl;\n }\n\n public void setIsActive(Boolean isActive) {\n this.isActive = isActive;\n }\n\n public void setIsAdmin(Boolean isAdmin) {\n this.isAdmin = isAdmin;\n }\n\n public void setIsInvited(Boolean isInvited) {\n this.isInvited = isInvited;\n }\n\n public void setPrefs(Object prefs) {\n this.prefs = prefs;\n }\n\n public void setWritableBy(List writableBy) {\n this.writableBy = writableBy;\n }\n\n public String toString() {\n return \"User(email=\" + this.getEmail() + \", username=\" + this.getUsername() + \", fullName=\" + this.getFullName() + \", firstName=\" + this.getFirstName() + \", lastName=\" + this.getLastName() + \", identityUrl=\" + this.getIdentityUrl() + \", isActive=\" + this.getIsActive() + \", isAdmin=\" + this.getIsAdmin() + \", isInvited=\" + this.getIsInvited() + \", prefs=\" + this.getPrefs() + \", writableBy=\" + this.getWritableBy() + \")\";\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/model/UserList.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.model;\n\nimport com.fasterxml.jackson.annotation.JsonIgnoreProperties;\nimport com.fasterxml.jackson.annotation.JsonInclude;\nimport com.fasterxml.jackson.annotation.JsonProperty;\nimport com.fasterxml.jackson.annotation.JsonPropertyOrder;\n\nimport java.util.List;\n\n@JsonInclude(JsonInclude.Include.NON_NULL)\n@JsonIgnoreProperties(ignoreUnknown = true)\n@JsonPropertyOrder({ \"items\" })\npublic class UserList extends ItemList {\n\n @JsonProperty(\"items\")\n private List items;\n\n public List getItems() {\n return this.items;\n }\n\n public void setItems(List items) {\n this.items = items;\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/model/argument/Argument.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.model.argument;\n\nimport com.fasterxml.jackson.annotation.JsonIgnore;\n\npublic abstract class Argument {\n\n @JsonIgnore\n private String uuid;\n\n public String getUuid() {\n return this.uuid;\n }\n\n public void setUuid(String uuid) {\n this.uuid = uuid;\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/model/argument/ContentsGroup.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.model.argument;\n\nimport com.fasterxml.jackson.annotation.JsonInclude;\nimport com.fasterxml.jackson.annotation.JsonProperty;\nimport com.fasterxml.jackson.annotation.JsonPropertyOrder;\n\nimport java.util.List;\n\n@JsonInclude(JsonInclude.Include.NON_NULL)\n@JsonPropertyOrder({ \"limit\", \"order\", \"filters\", \"recursive\" })\npublic class ContentsGroup extends Argument {\n\n @JsonProperty(\"limit\")\n private Integer limit;\n\n @JsonProperty(\"order\")\n private String order;\n\n @JsonProperty(\"filters\")\n private List filters;\n\n @JsonProperty(\"recursive\")\n private Boolean recursive;\n\n public Integer getLimit() {\n return this.limit;\n }\n\n public String getOrder() {\n return this.order;\n }\n\n public List getFilters() {\n return this.filters;\n }\n\n public Boolean getRecursive() {\n return this.recursive;\n }\n\n public void setLimit(Integer limit) {\n this.limit = limit;\n }\n\n public void setOrder(String order) {\n this.order = order;\n }\n\n public void setFilters(List filters) {\n this.filters = filters;\n }\n\n public void setRecursive(Boolean recursive) {\n this.recursive = recursive;\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/model/argument/Filter.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.model.argument;\n\nimport com.fasterxml.jackson.annotation.JsonFormat;\nimport com.fasterxml.jackson.annotation.JsonInclude;\nimport com.fasterxml.jackson.annotation.JsonProperty;\nimport com.fasterxml.jackson.annotation.JsonPropertyOrder;\n\n@JsonFormat(shape = JsonFormat.Shape.ARRAY)\n@JsonInclude(JsonInclude.Include.NON_NULL)\n@JsonPropertyOrder({ \"attribute\", \"operator\", \"operand\" })\npublic class Filter {\n\n @JsonProperty(\"attribute\")\n private String attribute;\n\n @JsonProperty(\"operator\")\n private Operator operator;\n\n @JsonProperty(\"operand\")\n private Object operand;\n\n private Filter(String attribute, Operator operator, Object operand) {\n this.attribute = attribute;\n this.operator = operator;\n this.operand = operand;\n }\n\n public static Filter of(String attribute, Operator operator, Object operand) {\n return new Filter(attribute, operator, operand);\n }\n\n public String getAttribute() {\n return this.attribute;\n }\n\n public Operator getOperator() {\n return this.operator;\n }\n\n public Object getOperand() {\n return this.operand;\n }\n\n public boolean equals(Object o) {\n if (o == this) return true;\n if (!(o instanceof Filter)) return false;\n final Filter other = (Filter) o;\n final Object this$attribute = this.getAttribute();\n final Object other$attribute = other.getAttribute();\n if (this$attribute == null ? other$attribute != null : !this$attribute.equals(other$attribute)) return false;\n final Object this$operator = this.getOperator();\n final Object other$operator = other.getOperator();\n if (this$operator == null ? other$operator != null : !this$operator.equals(other$operator)) return false;\n final Object this$operand = this.getOperand();\n final Object other$operand = other.getOperand();\n if (this$operand == null ? other$operand != null : !this$operand.equals(other$operand)) return false;\n return true;\n }\n\n public int hashCode() {\n final int PRIME = 59;\n int result = 1;\n final Object $attribute = this.getAttribute();\n result = result * PRIME + ($attribute == null ? 43 : $attribute.hashCode());\n final Object $operator = this.getOperator();\n result = result * PRIME + ($operator == null ? 43 : $operator.hashCode());\n final Object $operand = this.getOperand();\n result = result * PRIME + ($operand == null ? 43 : $operand.hashCode());\n return result;\n }\n\n public String toString() {\n return \"Filter(attribute=\" + this.getAttribute() + \", operator=\" + this.getOperator() + \", operand=\" + this.getOperand() + \")\";\n }\n\n public enum Operator {\n\n @JsonProperty(\"<\")\n LESS,\n\n @JsonProperty(\"<=\")\n LESS_EQUALS,\n\n @JsonProperty(\">=\")\n MORE_EQUALS,\n\n @JsonProperty(\">\")\n MORE,\n\n @JsonProperty(\"like\")\n LIKE,\n\n @JsonProperty(\"ilike\")\n ILIKE,\n\n @JsonProperty(\"=\")\n EQUALS,\n\n @JsonProperty(\"!=\")\n NOT_EQUALS,\n\n @JsonProperty(\"in\")\n IN,\n\n @JsonProperty(\"not in\")\n NOT_IN,\n\n @JsonProperty(\"is_a\")\n IS_A,\n\n @JsonProperty(\"exists\")\n EXISTS\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/model/argument/ListArgument.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.model.argument;\n\nimport com.fasterxml.jackson.annotation.JsonInclude;\nimport com.fasterxml.jackson.annotation.JsonProperty;\nimport com.fasterxml.jackson.annotation.JsonPropertyOrder;\n\nimport java.util.List;\n\n@JsonInclude(JsonInclude.Include.NON_NULL)\n@JsonPropertyOrder({ \"limit\", \"offset\", \"filters\", \"order\", \"select\", \"distinct\", \"count\", \"exclude_home_project\", \"include_old_versions\", \"include_trash\" })\npublic class ListArgument extends Argument {\n\n @JsonProperty(\"limit\")\n private Integer limit;\n\n @JsonProperty(\"offset\")\n private Integer offset;\n \n @JsonProperty(\"filters\")\n private List filters;\n\n @JsonProperty(\"order\")\n private List order;\n\n @JsonProperty(\"select\")\n private List select;\n\n @JsonProperty(\"distinct\")\n private Boolean distinct;\n\n @JsonProperty(\"count\")\n private Count count;\n\n @JsonProperty(\"exclude_home_project\")\n private Boolean excludeHomeProject;\n\n @JsonProperty(\"include_old_versions\")\n private Boolean includeOldVersions;\n\n @JsonProperty(\"include_trash\")\n private Boolean includeTrash;\n\n ListArgument(\n Integer limit, Integer offset, List filters, List order, List select,\n Boolean distinct, Count count, Boolean excludeHomeProject, Boolean includeOldVersions,\n Boolean includeTrash\n ) {\n this.limit = limit;\n this.offset = offset;\n this.filters = filters;\n this.order = order;\n this.select = select;\n this.distinct = distinct;\n this.count = count;\n this.excludeHomeProject = excludeHomeProject;\n this.includeOldVersions = includeOldVersions;\n this.includeTrash = includeTrash;\n }\n\n public static ListArgumentBuilder builder() {\n return new ListArgumentBuilder();\n }\n\n public enum Count {\n \n @JsonProperty(\"exact\")\n EXACT,\n \n @JsonProperty(\"none\")\n NONE\n }\n\n public static class ListArgumentBuilder {\n private Integer limit;\n private Integer offset;\n private List filters;\n private List order;\n private List select;\n private Boolean distinct;\n private Count count;\n private Boolean excludeHomeProject;\n private Boolean includeOldVersions;\n private Boolean includeTrash;\n\n ListArgumentBuilder() {\n }\n\n public ListArgumentBuilder limit(Integer limit) {\n this.limit = limit;\n return this;\n }\n\n public ListArgumentBuilder offset(Integer offset) {\n this.offset = offset;\n return this;\n }\n\n public ListArgumentBuilder filters(List filters) {\n this.filters = filters;\n return this;\n }\n\n public ListArgumentBuilder order(List order) {\n this.order = order;\n return this;\n }\n\n public ListArgumentBuilder select(List select) {\n this.select = select;\n return this;\n }\n\n public ListArgumentBuilder distinct(Boolean distinct) {\n this.distinct = distinct;\n return this;\n }\n\n public ListArgumentBuilder count(Count count) {\n this.count = count;\n return this;\n }\n\n public ListArgument.ListArgumentBuilder excludeHomeProject(Boolean excludeHomeProject) {\n this.excludeHomeProject = excludeHomeProject;\n return this;\n }\n\n public ListArgument.ListArgumentBuilder includeOldVersions(Boolean includeOldVersions) {\n this.includeOldVersions = includeOldVersions;\n return this;\n }\n\n public ListArgument.ListArgumentBuilder includeTrash(Boolean includeTrash) {\n this.includeTrash = includeTrash;\n return this;\n }\n\n public ListArgument build() {\n return new ListArgument(limit, offset, filters, order, select, distinct, count, excludeHomeProject, includeOldVersions, includeTrash);\n }\n\n public String toString() {\n return \"ListArgument.ListArgumentBuilder(limit=\" + this.limit +\n \", offset=\" + this.offset + \", filters=\" + this.filters +\n \", order=\" + this.order + \", select=\" + this.select +\n \", distinct=\" + this.distinct + \", count=\" + this.count +\n \", excludeHomeProject=\" + this.excludeHomeProject +\n \", includeOldVersions=\" + this.includeOldVersions +\n \", includeTrash=\" + this.includeTrash +\n \")\";\n }\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/api/model/argument/UntrashGroup.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.model.argument;\n\nimport com.fasterxml.jackson.annotation.JsonInclude;\nimport com.fasterxml.jackson.annotation.JsonProperty;\nimport com.fasterxml.jackson.annotation.JsonPropertyOrder;\n\n@JsonInclude(JsonInclude.Include.NON_NULL)\n@JsonPropertyOrder({ \"ensure_unique_name\" })\npublic class UntrashGroup extends Argument {\n\n @JsonProperty(\"ensure_unique_name\")\n private Boolean ensureUniqueName;\n\n public Boolean getEnsureUniqueName() {\n return this.ensureUniqueName;\n }\n\n public void setEnsureUniqueName(Boolean ensureUniqueName) {\n this.ensureUniqueName = ensureUniqueName;\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/common/Characters.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.common;\n\npublic final class Characters {\n\n private Characters() {}\n\n public static final String SPACE = \"\\\\040\";\n public static final String NEW_LINE = \"\\n\";\n public static final String SLASH = \"/\";\n public static final String DOT = \".\";\n public static final String COLON = \":\";\n public static final String PERCENT = \"%\";\n public static final String QUOTE = \"\\\"\";\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/common/Headers.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.common;\n\npublic final class Headers {\n\n private Headers() {}\n \n public static final String X_KEEP_DESIRED_REPLICAS = \"X-Keep-Desired-Replicas\";\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/common/Patterns.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.common;\n\npublic final class Patterns {\n\n public static final String HINT_PATTERN = \"^[A-Z][A-Za-z0-9@_-]+$\";\n public static final String FILE_TOKEN_PATTERN = \"(\\\\d+:\\\\d+:\\\\S+)\";\n public static final String LOCATOR_PATTERN = \"([0-9a-f]{32})\\\\+([0-9]+)(\\\\+[A-Z][-A-Za-z0-9@_]*)*\";\n public static final String GROUP_UUID_PATTERN = \"[a-z0-9]{5}-j7d0g-[a-z0-9]{15}\";\n public static final String USER_UUID_PATTERN = \"[a-z0-9]{5}-tpzed-[a-z0-9]{15}\";\n\n private Patterns() {}\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/config/ConfigProvider.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.config;\n\nimport java.io.File;\n\npublic interface ConfigProvider {\n\n //API\n boolean isApiHostInsecure();\n\n String getKeepWebHost();\n\n int getKeepWebPort();\n\n String getApiHost();\n\n int getApiPort();\n\n String getApiToken();\n\n String getApiProtocol();\n\n int getConnectTimeout();\n\n int getReadTimeout();\n\n int getWriteTimeout();\n\n //FILE UPLOAD\n int getFileSplitSize();\n\n File getFileSplitDirectory();\n\n int getNumberOfCopies();\n\n int getNumberOfRetries();\n\n\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/config/ExternalConfigProvider.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.config;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport java.io.File;\n\npublic class ExternalConfigProvider implements ConfigProvider {\n\n private static final Logger log = LoggerFactory.getLogger(ExternalConfigProvider.class);\n private static final int DEFAULT_CONNECTION_TIMEOUT = 60000;\n private static final int DEFAULT_READ_TIMEOUT = 60000;\n private static final int DEFAULT_WRITE_TIMEOUT = 60000;\n\n private final boolean apiHostInsecure;\n private final String keepWebHost;\n private final int keepWebPort;\n private final String apiHost;\n private final int apiPort;\n private final String apiToken;\n private final String apiProtocol;\n private final int fileSplitSize;\n private final File fileSplitDirectory;\n private final int numberOfCopies;\n private final int numberOfRetries;\n private final int connectTimeout;\n private final int readTimeout;\n private final int writeTimeout;\n\n ExternalConfigProvider(boolean apiHostInsecure, String keepWebHost, int keepWebPort, String apiHost, int apiPort,\n String apiToken, String apiProtocol, int fileSplitSize, File fileSplitDirectory,\n int numberOfCopies, int numberOfRetries) {\n this.apiHostInsecure = apiHostInsecure;\n this.keepWebHost = keepWebHost;\n this.keepWebPort = keepWebPort;\n this.apiHost = apiHost;\n this.apiPort = apiPort;\n this.apiToken = apiToken;\n this.apiProtocol = apiProtocol;\n this.fileSplitSize = fileSplitSize;\n this.fileSplitDirectory = fileSplitDirectory;\n this.numberOfCopies = numberOfCopies;\n this.numberOfRetries = numberOfRetries;\n this.connectTimeout = DEFAULT_CONNECTION_TIMEOUT;\n this.readTimeout = DEFAULT_READ_TIMEOUT;\n this.writeTimeout = DEFAULT_WRITE_TIMEOUT;\n }\n\n ExternalConfigProvider(boolean apiHostInsecure, String keepWebHost, int keepWebPort, String apiHost, int apiPort,\n String apiToken, String apiProtocol, int fileSplitSize, File fileSplitDirectory,\n int numberOfCopies, int numberOfRetries,\n int connectTimeout, int readTimeout, int writeTimeout) {\n this.apiHostInsecure = apiHostInsecure;\n this.keepWebHost = keepWebHost;\n this.keepWebPort = keepWebPort;\n this.apiHost = apiHost;\n this.apiPort = apiPort;\n this.apiToken = apiToken;\n this.apiProtocol = apiProtocol;\n this.fileSplitSize = fileSplitSize;\n this.fileSplitDirectory = fileSplitDirectory;\n this.numberOfCopies = numberOfCopies;\n this.numberOfRetries = numberOfRetries;\n this.connectTimeout = connectTimeout;\n this.readTimeout = readTimeout;\n this.writeTimeout = writeTimeout;\n }\n\n public static ExternalConfigProviderBuilder builder() {\n return new ExternalConfigProviderBuilder();\n }\n\n @Override\n public String toString() {\n return \"ExternalConfigProvider{\" +\n \"apiHostInsecure=\" + apiHostInsecure +\n \", keepWebHost='\" + keepWebHost + '\\'' +\n \", keepWebPort=\" + keepWebPort +\n \", apiHost='\" + apiHost + '\\'' +\n \", apiPort=\" + apiPort +\n \", apiToken='\" + apiToken + '\\'' +\n \", apiProtocol='\" + apiProtocol + '\\'' +\n \", fileSplitSize=\" + fileSplitSize +\n \", fileSplitDirectory=\" + fileSplitDirectory +\n \", numberOfCopies=\" + numberOfCopies +\n \", numberOfRetries=\" + numberOfRetries +\n '}';\n }\n\n public boolean isApiHostInsecure() {\n return this.apiHostInsecure;\n }\n\n public String getKeepWebHost() {\n return this.keepWebHost;\n }\n\n public int getKeepWebPort() {\n return this.keepWebPort;\n }\n\n public String getApiHost() {\n return this.apiHost;\n }\n\n public int getApiPort() {\n return this.apiPort;\n }\n\n public String getApiToken() {\n return this.apiToken;\n }\n\n public String getApiProtocol() {\n return this.apiProtocol;\n }\n\n public int getFileSplitSize() {\n return this.fileSplitSize;\n }\n\n public File getFileSplitDirectory() {\n return this.fileSplitDirectory;\n }\n\n public int getNumberOfCopies() {\n return this.numberOfCopies;\n }\n\n public int getNumberOfRetries() {\n return this.numberOfRetries;\n }\n\n public int getConnectTimeout() {\n return this.connectTimeout;\n }\n\n public int getReadTimeout() {\n return this.readTimeout;\n }\n\n public int getWriteTimeout() {\n return this.writeTimeout;\n }\n\n public static class ExternalConfigProviderBuilder {\n private boolean apiHostInsecure;\n private String keepWebHost;\n private int keepWebPort;\n private String apiHost;\n private int apiPort;\n private String apiToken;\n private String apiProtocol;\n private int fileSplitSize;\n private File fileSplitDirectory;\n private int numberOfCopies;\n private int numberOfRetries;\n private int connectTimeout = DEFAULT_CONNECTION_TIMEOUT;\n private int readTimeout = DEFAULT_READ_TIMEOUT;\n private int writeTimeout = DEFAULT_WRITE_TIMEOUT;\n private boolean autoFetchWebDAV = true;\n\n ExternalConfigProviderBuilder() {\n }\n\n public ExternalConfigProvider.ExternalConfigProviderBuilder apiHostInsecure(boolean apiHostInsecure) {\n this.apiHostInsecure = apiHostInsecure;\n return this;\n }\n\n public ExternalConfigProvider.ExternalConfigProviderBuilder keepWebHost(String keepWebHost) {\n this.keepWebHost = keepWebHost;\n return this;\n }\n\n public ExternalConfigProvider.ExternalConfigProviderBuilder keepWebPort(int keepWebPort) {\n this.keepWebPort = keepWebPort;\n return this;\n }\n\n public ExternalConfigProvider.ExternalConfigProviderBuilder apiHost(String apiHost) {\n this.apiHost = apiHost;\n return this;\n }\n\n public ExternalConfigProvider.ExternalConfigProviderBuilder apiPort(int apiPort) {\n this.apiPort = apiPort;\n return this;\n }\n\n public ExternalConfigProvider.ExternalConfigProviderBuilder apiToken(String apiToken) {\n this.apiToken = apiToken;\n return this;\n }\n\n public ExternalConfigProvider.ExternalConfigProviderBuilder apiProtocol(String apiProtocol) {\n this.apiProtocol = apiProtocol;\n return this;\n }\n\n public ExternalConfigProvider.ExternalConfigProviderBuilder fileSplitSize(int fileSplitSize) {\n this.fileSplitSize = fileSplitSize;\n return this;\n }\n\n public ExternalConfigProvider.ExternalConfigProviderBuilder fileSplitDirectory(File fileSplitDirectory) {\n this.fileSplitDirectory = fileSplitDirectory;\n return this;\n }\n\n public ExternalConfigProvider.ExternalConfigProviderBuilder numberOfCopies(int numberOfCopies) {\n this.numberOfCopies = numberOfCopies;\n return this;\n }\n\n public ExternalConfigProvider.ExternalConfigProviderBuilder numberOfRetries(int numberOfRetries) {\n this.numberOfRetries = numberOfRetries;\n return this;\n }\n\n public ExternalConfigProvider.ExternalConfigProviderBuilder connectTimeout(int connectTimeout) {\n this.connectTimeout = connectTimeout;\n return this;\n }\n\n public ExternalConfigProvider.ExternalConfigProviderBuilder readTimeout(int readTimeout) {\n this.readTimeout = readTimeout;\n return this;\n }\n\n public ExternalConfigProvider.ExternalConfigProviderBuilder writeTimeout(int writeTimeout) {\n this.writeTimeout = writeTimeout;\n return this;\n }\n\n public ExternalConfigProvider.ExternalConfigProviderBuilder autoFetchWebDAV(boolean autoFetchWebDAV) {\n this.autoFetchWebDAV = autoFetchWebDAV;\n return this;\n }\n\n public ExternalConfigProvider build() {\n if (shouldAutoFetchWebDAV()) {\n autoFetchWebDAVConfiguration();\n }\n\n validateWebDAVConfiguration();\n\n return new ExternalConfigProvider(\n apiHostInsecure, keepWebHost, keepWebPort, apiHost,\n apiPort, apiToken, apiProtocol, fileSplitSize, fileSplitDirectory,\n numberOfCopies, numberOfRetries, connectTimeout, readTimeout, writeTimeout\n );\n }\n\n private boolean shouldAutoFetchWebDAV() {\n return autoFetchWebDAV &&\n (keepWebHost == null || keepWebHost.isEmpty());\n }\n\n private void autoFetchWebDAVConfiguration() {\n WebDAVConfigFetcher fetcher = new WebDAVConfigFetcher(\n apiProtocol, apiHost, apiPort, apiHostInsecure\n );\n\n WebDAVConfigFetcher.WebDAVConfig config = fetcher.fetch();\n\n if (config != null) {\n keepWebHost = config.getHost();\n keepWebPort = config.getPort();\n }\n }\n\n private void validateWebDAVConfiguration() {\n if (keepWebHost == null || keepWebHost.isEmpty()) {\n log.warn(\"WebDAV host is not configured. File operations may not work properly. Consider providing keepWebHost/keepWebPort or ensuring the Arvados API config endpoint is accessible.\");\n }\n }\n\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/config/FileConfigProvider.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.config;\n\nimport com.typesafe.config.Config;\nimport com.typesafe.config.ConfigFactory;\n\nimport java.io.File;\n\npublic class FileConfigProvider implements ConfigProvider {\n\n private static final String DEFAULT_PATH = \"arvados\";\n private final Config config;\n\n public FileConfigProvider() {\n config = ConfigFactory.load().getConfig(DEFAULT_PATH);\n }\n\n public FileConfigProvider(final String configFile) {\n config = (configFile != null) ?\n ConfigFactory.load(configFile).getConfig(DEFAULT_PATH) : ConfigFactory.load().getConfig(DEFAULT_PATH);\n }\n\n public Config getConfig() {\n return config;\n }\n\n private File getFile(String path) {\n return new File(config.getString(path));\n }\n\n private int getInt(String path) {\n return config.getInt(path);\n }\n\n private boolean getBoolean(String path) {\n return config.getBoolean(path);\n }\n\n private String getString(String path) {\n return config.getString(path);\n }\n\n @Override\n public boolean isApiHostInsecure() {\n return this.getBoolean(\"api.host-insecure\");\n }\n\n @Override\n public String getKeepWebHost() {\n return this.getString(\"api.keepweb-host\");\n }\n\n @Override\n public int getKeepWebPort() {\n return this.getInt(\"api.keepweb-port\");\n }\n\n @Override\n public String getApiHost() {\n return this.getString(\"api.host\");\n }\n\n @Override\n public int getApiPort() {\n return this.getInt(\"api.port\");\n }\n\n @Override\n public String getApiToken() {\n return this.getString(\"api.token\");\n }\n\n @Override\n public String getApiProtocol() {\n return this.getString(\"api.protocol\");\n }\n\n @Override\n public int getFileSplitSize() {\n return this.getInt(\"split-size\");\n }\n\n @Override\n public File getFileSplitDirectory() {\n return this.getFile(\"temp-dir\");\n }\n\n @Override\n public int getNumberOfCopies() {\n return this.getInt(\"copies\");\n }\n\n @Override\n public int getNumberOfRetries() {\n return this.getInt(\"retries\");\n }\n\n public String getIntegrationTestProjectUuid() {\n return this.getString(\"integration-tests.project-uuid\");\n }\n\n @Override\n public int getConnectTimeout() {\n return this.getInt(\"connectTimeout\");\n }\n\n @Override\n public int getReadTimeout() {\n return this.getInt(\"readTimeout\");\n }\n\n @Override\n public int getWriteTimeout() {\n return this.getInt(\"writeTimeout\");\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/config/WebDAVConfigFetcher.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.config;\n\nimport org.arvados.client.api.client.ConfigApiClient;\nimport org.arvados.client.api.model.ArvadosConfig;\nimport org.arvados.client.exception.ArvadosApiException;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport java.net.MalformedURLException;\nimport java.net.URL;\n\npublic class WebDAVConfigFetcher {\n \n private static final Logger log = LoggerFactory.getLogger(WebDAVConfigFetcher.class);\n private static final int DEFAULT_HTTPS_PORT = 443;\n private static final int DEFAULT_HTTP_PORT = 80;\n \n private final String apiProtocol;\n private final String apiHost;\n private final int apiPort;\n private final boolean apiHostInsecure;\n \n public WebDAVConfigFetcher(String apiProtocol, String apiHost, int apiPort, boolean apiHostInsecure) {\n this.apiProtocol = apiProtocol != null ? apiProtocol : \"https\";\n this.apiHost = apiHost;\n this.apiPort = apiPort > 0 ? apiPort : (this.apiProtocol.equals(\"https\") ? DEFAULT_HTTPS_PORT : DEFAULT_HTTP_PORT);\n this.apiHostInsecure = apiHostInsecure;\n }\n \n public WebDAVConfig fetch() {\n if (!isConfigured()) {\n log.debug(\"API host not configured, skipping WebDAV auto-fetch\");\n return null;\n }\n \n try {\n log.info(\"Attempting to auto-fetch WebDAV configuration from Arvados API\");\n \n ArvadosConfig config = fetchArvadosConfig();\n String webDavUrl = extractWebDAVUrl(config);\n \n if (webDavUrl == null) {\n log.debug(\"No WebDAV URL found in Arvados config\");\n return null;\n }\n \n return parseWebDAVUrl(webDavUrl);\n \n } catch (ArvadosApiException e) {\n log.warn(\"Failed to auto-fetch WebDAV configuration: {}. \" +\n \"You may need to configure keepWebHost and keepWebPort manually.\", \n e.getMessage());\n } catch (Exception e) {\n log.warn(\"Unexpected error while auto-fetching WebDAV configuration: {}. \" +\n \"You may need to configure keepWebHost and keepWebPort manually.\", \n e.getMessage());\n }\n \n return null;\n }\n \n private boolean isConfigured() {\n return apiHost != null && !apiHost.isEmpty();\n }\n \n private ArvadosConfig fetchArvadosConfig() throws ArvadosApiException {\n ConfigApiClient configClient = new ConfigApiClient(\n apiProtocol, apiHost, apiPort, apiHostInsecure\n );\n return configClient.fetchConfig();\n }\n \n private String extractWebDAVUrl(ArvadosConfig config) {\n if (config == null || config.getServices() == null) {\n return null;\n }\n \n ArvadosConfig.WebDAVDownload webDav = config.getServices().getWebDAVDownload();\n if (webDav == null) {\n return null;\n }\n \n return webDav.getExternalURL();\n }\n \n private WebDAVConfig parseWebDAVUrl(String webDavUrl) {\n if (webDavUrl == null || webDavUrl.isEmpty()) {\n return null;\n }\n \n try {\n URL url = new URL(webDavUrl);\n String host = url.getHost();\n int port = url.getPort();\n \n // Use default port based on protocol if not specified\n if (port == -1) {\n port = \"https\".equals(url.getProtocol()) ? DEFAULT_HTTPS_PORT : DEFAULT_HTTP_PORT;\n }\n \n log.info(\"Successfully auto-configured WebDAV: host={}, port={}\", host, port);\n return new WebDAVConfig(host, port);\n \n } catch (MalformedURLException e) {\n log.warn(\"Failed to parse WebDAV URL '{}': {}\", webDavUrl, e.getMessage());\n return null;\n }\n }\n \n public static class WebDAVConfig {\n private final String host;\n private final int port;\n \n public WebDAVConfig(String host, int port) {\n this.host = host;\n this.port = port;\n }\n \n public String getHost() {\n return host;\n }\n \n public int getPort() {\n return port;\n }\n }\n}" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/exception/ArvadosApiException.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.exception;\n\npublic class ArvadosApiException extends ArvadosClientException {\n\n private static final long serialVersionUID = 1L;\n\n public ArvadosApiException(String message) {\n super(message);\n }\n \n public ArvadosApiException(String message, Throwable cause) {\n super(message, cause);\n }\n \n public ArvadosApiException(Throwable cause) {\n super(cause);\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/exception/ArvadosClientException.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.exception;\n\n/**\n * Parent exception for all exceptions in library.\n * More specific exceptions like ArvadosApiException extend this class.\n */\npublic class ArvadosClientException extends RuntimeException {\n\n public ArvadosClientException(String message) {\n super(message);\n }\n\n public ArvadosClientException(String message, Throwable cause) {\n super(message, cause);\n }\n\n public ArvadosClientException(Throwable cause) {\n super(cause);\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/facade/ArvadosFacade.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.facade;\n\nimport com.google.common.collect.Lists;\nimport org.arvados.client.api.client.CollectionsApiClient;\nimport org.arvados.client.api.client.GroupsApiClient;\nimport org.arvados.client.api.client.KeepWebApiClient;\nimport org.arvados.client.api.client.UsersApiClient;\nimport org.arvados.client.api.model.*;\nimport org.arvados.client.api.model.argument.Filter;\nimport org.arvados.client.api.model.argument.ListArgument;\nimport org.arvados.client.config.FileConfigProvider;\nimport org.arvados.client.config.ConfigProvider;\nimport org.arvados.client.logic.collection.FileToken;\nimport org.arvados.client.logic.collection.ManifestDecoder;\nimport org.arvados.client.logic.keep.FileDownloader;\nimport org.arvados.client.logic.keep.FileUploader;\nimport org.slf4j.Logger;\n\nimport java.io.File;\nimport java.util.Arrays;\nimport java.util.Collections;\nimport java.util.List;\nimport java.util.Map;\n\npublic class ArvadosFacade {\n\n private final ConfigProvider config;\n private final Logger log = org.slf4j.LoggerFactory.getLogger(ArvadosFacade.class);\n private CollectionsApiClient collectionsApiClient;\n private GroupsApiClient groupsApiClient;\n private UsersApiClient usersApiClient;\n private FileDownloader fileDownloader;\n private FileUploader fileUploader;\n private static final String PROJECT = \"project\";\n private static final String SUBPROJECT = \"sub-project\";\n\n public ArvadosFacade(ConfigProvider config) {\n this.config = config;\n setFacadeFields();\n }\n\n public ArvadosFacade() {\n this.config = new FileConfigProvider();\n setFacadeFields();\n }\n\n private void setFacadeFields() {\n collectionsApiClient = new CollectionsApiClient(config);\n groupsApiClient = new GroupsApiClient(config);\n usersApiClient = new UsersApiClient(config);\n ManifestDecoder manifestDecoder = new ManifestDecoder();\n KeepWebApiClient keepWebApiClient = new KeepWebApiClient(config);\n fileDownloader = new FileDownloader(manifestDecoder, collectionsApiClient, keepWebApiClient);\n fileUploader = new FileUploader(keepWebApiClient, collectionsApiClient, config);\n }\n\n /**\n * This method downloads single file from collection using Arvados Keep-Web.\n * File is saved on a drive in specified location and returned.\n *\n * @param filePathName path to the file in collection. If requested file is stored\n * directly in collection (not within its subdirectory) this\n * would be just the name of file (ex. 'file.txt').\n * Otherwise full file path must be passed (ex. 'folder/file.txt')\n * @param collectionUuid uuid of collection containing requested file\n * @param pathToDownloadFolder path to location in which file should be saved.\n * Passed location must be a directory in which file of\n * that name does not already exist.\n * @return downloaded file\n */\n public File downloadFile(String filePathName, String collectionUuid, String pathToDownloadFolder) {\n return fileDownloader.downloadSingleFileUsingKeepWeb(filePathName, collectionUuid, pathToDownloadFolder);\n }\n\n /**\n * This method downloads all files from collection.\n * Directory named by collection uuid is created in specified location,\n * files are saved on a drive in this directory and list with downloaded\n * files is returned.\n *\n * @param collectionUuid uuid of collection from which files are downloaded\n * @param pathToDownloadFolder path to location in which files should be saved.\n * New folder named by collection uuid, containing\n * downloaded files, is created in this location.\n * Passed location must be a directory in which folder\n * of that name does not already exist.\n * @param usingKeepWeb if set to true files will be downloaded using Keep Web.\n * If set to false files will be downloaded using Keep Server API.\n * @return list containing downloaded files\n */\n public List downloadCollectionFiles(String collectionUuid, String pathToDownloadFolder, boolean usingKeepWeb) {\n if (usingKeepWeb)\n return fileDownloader.downloadFilesFromCollectionUsingKeepWeb(collectionUuid, pathToDownloadFolder);\n return fileDownloader.downloadFilesFromCollection(collectionUuid, pathToDownloadFolder);\n }\n\n /**\n * Lists all FileTokens (objects containing information about files) for\n * specified collection.\n * Information in each FileToken includes file path, name, size and position\n * in data stream\n *\n * @param collectionUuid uuid of collection for which FileTokens are listed\n * @return list containing FileTokens for each file in specified collection\n */\n public List listFileInfoFromCollection(String collectionUuid) {\n return fileDownloader.listFileInfoFromCollection(collectionUuid);\n }\n\n /**\n * Creates and uploads new collection containing passed files.\n * Created collection has a default name and is uploaded to user's 'Home' project.\n *\n * @see ArvadosFacade#upload(List, String, String)\n * @param files list of files to be uploaded within new collection\n * @return collection object mapped from JSON that is returned from server after successful upload\n */\n public Collection upload(List files) {\n return upload(files, null, null);\n }\n\n /**\n * Creates and uploads new collection containing a single file.\n * Created collection has a default name and is uploaded to user's 'Home' project.\n *\n * @see ArvadosFacade#upload(List, String, String)\n * @param file file to be uploaded\n * @return collection object mapped from JSON that is returned from server after successful upload\n */\n public Collection upload(File file) {\n return upload(Collections.singletonList(file), null, null);\n }\n\n /**\n * Uploads new collection with specified name and containing selected files\n * to an existing project.\n *\n * @param sourceFiles list of files to be uploaded within new collection\n * @param collectionName name for the newly created collection.\n * Collection with that name cannot be already created\n * in specified project. If null is passed\n * then collection name is set to default, containing\n * phrase 'New Collection' and a timestamp.\n * @param projectUuid uuid of the project in which created collection is to be included.\n * If null is passed then collection is uploaded to user's 'Home' project.\n * @return collection object mapped from JSON that is returned from server after successful upload\n */\n public Collection upload(List sourceFiles, String collectionName, String projectUuid) {\n return fileUploader.upload(sourceFiles, collectionName, projectUuid);\n }\n\n /**\n * Uploads a file to a specified collection.\n *\n * @see ArvadosFacade#uploadToExistingCollection(List, String)\n * @param file file to be uploaded to existing collection. Filenames must be unique\n * in comparison with files already existing within collection.\n * @param collectionUUID UUID of collection to which files should be uploaded\n * @return collection object mapped from JSON that is returned from server after successful upload\n */\n public Collection uploadToExistingCollection(File file, String collectionUUID) {\n return fileUploader.uploadToExistingCollection(Collections.singletonList(file), collectionUUID);\n }\n\n /**\n * Uploads multiple files to an existing collection.\n *\n * @param files list of files to be uploaded to existing collection.\n * File names must be unique - both within passed list and\n * in comparison with files already existing within collection.\n * @param collectionUUID UUID of collection to which files should be uploaded\n * @return collection object mapped from JSON that is returned from server after successful upload\n */\n public Collection uploadToExistingCollection(List files, String collectionUUID) {\n return fileUploader.uploadToExistingCollection(files, collectionUUID);\n }\n\n /**\n * Creates and uploads new empty collection to specified project.\n *\n * @param collectionName name for the newly created collection.\n * Collection with that name cannot be already created\n * in specified project.\n * @param projectUuid uuid of project that will contain uploaded empty collection.\n * To select home project pass current user's uuid from getCurrentUser()\n * @return collection object mapped from JSON that is returned from server after successful upload\n * @see ArvadosFacade#getCurrentUser()\n */\n public Collection createEmptyCollection(String collectionName, String projectUuid) {\n Collection collection = new Collection();\n collection.setOwnerUuid(projectUuid);\n collection.setName(collectionName);\n return collectionsApiClient.create(collection);\n }\n\n /**\n * Uploads multiple files to an existing collection.\n *\n * @param collectionUUID UUID of collection to which the files are to be copied\n * @param files map of files to be copied to existing collection.\n * The map consists of a pair in the form of a filename and a filename\n * along with the Portable data hash\n * @return collection object mapped from JSON that is returned from server after successful copied\n */\n public Collection updateWithReplaceFiles(String collectionUUID, Map files) {\n CollectionReplaceFiles replaceFilesRequest = new CollectionReplaceFiles();\n replaceFilesRequest.getReplaceFiles().putAll(files);\n return collectionsApiClient.update(collectionUUID, replaceFilesRequest);\n }\n\n /**\n * Returns current user information based on Api Token provided via configuration\n *\n * @return user object mapped from JSON that is returned from server based on provided Api Token.\n * It contains information about user who has this token assigned.\n */\n public User getCurrentUser() {\n return usersApiClient.current();\n }\n\n /**\n * Gets uuid of current user based on api Token provided in configuration and uses it to list all\n * projects that this user owns in Arvados.\n *\n * @return GroupList containing all groups that current user is owner of.\n * @see ArvadosFacade#getCurrentUser()\n */\n public GroupList showGroupsOwnedByCurrentUser() {\n ListArgument listArgument = ListArgument.builder()\n .filters(Arrays.asList(\n Filter.of(\"owner_uuid\", Filter.Operator.LIKE, getCurrentUser().getUuid()),\n Filter.of(\"group_class\", Filter.Operator.IN, Lists.newArrayList(PROJECT, SUBPROJECT)\n )))\n .build();\n GroupList groupList = groupsApiClient.list(listArgument);\n log.debug(\"Groups owned by user:\");\n groupList.getItems().forEach(m -> log.debug(m.getUuid() + \" -- \" + m.getName()));\n\n return groupList;\n }\n\n /**\n * Gets uuid of current user based on api Token provided in configuration and uses it to list all\n * projects that this user has read access to in Arvados.\n *\n * @return GroupList containing all groups that current user has read access to.\n */\n public GroupList showGroupsAccessibleByCurrentUser() {\n ListArgument listArgument = ListArgument.builder()\n .filters(Collections.singletonList(\n Filter.of(\"group_class\", Filter.Operator.IN, Lists.newArrayList(PROJECT, SUBPROJECT)\n )))\n .build();\n GroupList groupList = groupsApiClient.list(listArgument);\n log.debug(\"Groups accessible by user:\");\n groupList.getItems().forEach(m -> log.debug(m.getUuid() + \" -- \" + m.getName()));\n\n return groupList;\n }\n\n /**\n * Filters all collections from selected project and returns list of those that contain passed String in their name.\n * Operator \"LIKE\" is used so in order to obtain certain collection it is sufficient to pass just part of its name.\n * Returned collections in collectionList are ordered by date of creation (starting from oldest one).\n *\n * @param collectionName collections containing this param in their name will be returned.\n * Passing a wildcard is possible - for example passing \"a%\" searches for\n * all collections starting with \"a\".\n * @param projectUuid uuid of project in which will be searched for collections with given name. To search home\n * project provide user uuid (from getCurrentUser())\n * @return object CollectionList containing all collections matching specified name criteria\n * @see ArvadosFacade#getCurrentUser()\n */\n public CollectionList getCollectionsFromProjectByName(String collectionName, String projectUuid) {\n ListArgument listArgument = ListArgument.builder()\n .filters(Arrays.asList(\n Filter.of(\"owner_uuid\", Filter.Operator.LIKE, projectUuid),\n Filter.of(\"name\", Filter.Operator.LIKE, collectionName)\n ))\n .order(Collections.singletonList(\"created_at\"))\n .build();\n\n return collectionsApiClient.list(listArgument);\n }\n\n /**\n * Gets project details by uuid.\n *\n * @param projectUuid uuid of project\n * @return Group object containing information about project\n */\n public Group getProjectByUuid(String projectUuid) {\n Group project = groupsApiClient.get(projectUuid);\n log.debug(\"Retrieved \" + project.getName() + \" with UUID: \" + project.getUuid());\n return project;\n }\n\n /**\n * Creates new project that will be a subproject of \"home\" for current user.\n *\n * @param projectName name for the newly created project\n * @return Group object containing information about created project\n * (mapped from JSON returned from server after creating the project)\n */\n public Group createNewProject(String projectName) {\n Group project = new Group();\n project.setName(projectName);\n project.setGroupClass(PROJECT);\n Group createdProject = groupsApiClient.create(project);\n log.debug(\"Project \" + createdProject.getName() + \" created with UUID: \" + createdProject.getUuid());\n return createdProject;\n }\n\n\n /**\n * Creates new project that will be a subproject of \"home\" for the specified owner.\n *\n * @param ownerUuid uuid of owner for subproject\n * @param projectName name for the newly created subproject\n * @return Group object containing information about created project\n * (mapped from JSON returned from server after creating the project)\n */\n public Group createNewSubProject(String ownerUuid, String projectName) {\n Group project = new Group();\n project.setName(projectName);\n project.setGroupClass(PROJECT);\n project.setOwnerUuid(ownerUuid);\n Group createdProject = groupsApiClient.create(project);\n this.log.debug(\"Project \" + createdProject.getName() + \" created with UUID: \" + createdProject.getUuid());\n return createdProject;\n }\n\n /**\n * Deletes collection with specified uuid.\n *\n * @param collectionUuid uuid of collection to be deleted. User whose token is provided in configuration\n * must be authorized to delete such collection.\n * @return collection object with deleted collection (mapped from JSON returned from server after deleting the collection)\n */\n public Collection deleteCollection(String collectionUuid) {\n Collection deletedCollection = collectionsApiClient.delete(collectionUuid);\n log.debug(\"Collection: \" + collectionUuid + \" deleted.\");\n return deletedCollection;\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/logic/collection/CollectionFactory.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.logic.collection;\n\nimport org.arvados.client.api.client.GroupsApiClient;\nimport org.arvados.client.api.client.UsersApiClient;\nimport org.arvados.client.exception.ArvadosApiException;\nimport org.arvados.client.api.model.Collection;\nimport org.arvados.client.common.Patterns;\nimport org.arvados.client.config.FileConfigProvider;\nimport org.arvados.client.config.ConfigProvider;\nimport org.arvados.client.exception.ArvadosClientException;\n\nimport java.time.LocalDateTime;\nimport java.time.format.DateTimeFormatter;\nimport java.util.Optional;\n\npublic class CollectionFactory {\n\n private ConfigProvider config;\n private UsersApiClient usersApiClient;\n private GroupsApiClient groupsApiClient;\n\n private final String name;\n private final String projectUuid;\n\n private CollectionFactory(ConfigProvider config, String name, String projectUuid) {\n this.name = name;\n this.projectUuid = projectUuid;\n this.config = config;\n setApiClients();\n }\n\n public static CollectionFactoryBuilder builder() {\n return new CollectionFactoryBuilder();\n }\n\n private void setApiClients() {\n if(this.config == null) this.config = new FileConfigProvider();\n\n this.usersApiClient = new UsersApiClient(config);\n this.groupsApiClient = new GroupsApiClient(config);\n }\n\n public Collection create() {\n Collection newCollection = new Collection();\n newCollection.setName(getNameOrDefault(name));\n newCollection.setOwnerUuid(getDesiredProjectUuid(projectUuid));\n\n return newCollection;\n }\n\n private String getNameOrDefault(String name) {\n return Optional.ofNullable(name).orElseGet(() -> {\n LocalDateTime dateTime = LocalDateTime.now();\n DateTimeFormatter formatter = DateTimeFormatter.ofPattern(\"Y-MM-dd HH:mm:ss.SSS\");\n return String.format(\"New Collection (%s)\", dateTime.format(formatter));\n });\n }\n\n public String getDesiredProjectUuid(String projectUuid) {\n try {\n if (projectUuid == null || projectUuid.length() == 0){\n return usersApiClient.current().getUuid();\n } else if (projectUuid.matches(Patterns.USER_UUID_PATTERN)) {\n return usersApiClient.get(projectUuid).getUuid();\n } else if (projectUuid.matches(Patterns.GROUP_UUID_PATTERN)) {\n return groupsApiClient.get(projectUuid).getUuid();\n }\n } catch (ArvadosApiException e) {\n throw new ArvadosClientException(String.format(\"An error occurred while getting project by UUID %s\", projectUuid));\n }\n throw new ArvadosClientException(String.format(\"No project with %s UUID found\", projectUuid));\n }\n\n public static class CollectionFactoryBuilder {\n private ConfigProvider config;\n private String name;\n private String projectUuid;\n\n CollectionFactoryBuilder() {\n }\n\n public CollectionFactoryBuilder config(ConfigProvider config) {\n this.config = config;\n return this;\n }\n\n public CollectionFactoryBuilder name(String name) {\n this.name = name;\n return this;\n }\n\n public CollectionFactoryBuilder projectUuid(String projectUuid) {\n this.projectUuid = projectUuid;\n return this;\n }\n\n public CollectionFactory build() {\n return new CollectionFactory(config, name, projectUuid);\n }\n\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/logic/collection/FileToken.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.logic.collection;\n\nimport com.google.common.base.Strings;\nimport org.arvados.client.common.Characters;\n\npublic class FileToken {\n\n private long filePosition;\n private long fileSize;\n private String fileName;\n private String path;\n\n public FileToken(String fileTokenInfo) {\n splitFileTokenInfo(fileTokenInfo);\n }\n\n public FileToken(String fileTokenInfo, String path) {\n splitFileTokenInfo(fileTokenInfo);\n this.path = path;\n }\n\n private void splitFileTokenInfo(String fileTokenInfo) {\n String[] tokenPieces = fileTokenInfo.split(\":\");\n this.filePosition = Long.parseLong(tokenPieces[0]);\n this.fileSize = Long.parseLong(tokenPieces[1]);\n this.fileName = tokenPieces[2].replace(Characters.SPACE, \" \");\n }\n\n @Override\n public String toString() {\n return filePosition + \":\" + fileSize + \":\" + fileName;\n }\n\n public String getFullPath() {\n return Strings.isNullOrEmpty(path) ? fileName : path + fileName;\n }\n\n public long getFilePosition() {\n return this.filePosition;\n }\n\n public long getFileSize() {\n return this.fileSize;\n }\n\n public String getFileName() {\n return this.fileName;\n }\n\n public String getPath() {\n return this.path;\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/logic/collection/ManifestDecoder.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.logic.collection;\n\nimport org.arvados.client.common.Characters;\nimport org.arvados.client.exception.ArvadosClientException;\nimport org.arvados.client.logic.keep.KeepLocator;\n\nimport java.util.ArrayList;\nimport java.util.Arrays;\nimport java.util.LinkedList;\nimport java.util.List;\nimport java.util.Objects;\n\nimport static java.util.stream.Collectors.toList;\nimport static org.arvados.client.common.Patterns.FILE_TOKEN_PATTERN;\nimport static org.arvados.client.common.Patterns.LOCATOR_PATTERN;\n\npublic class ManifestDecoder {\n\n public List decode(String manifestText) {\n\n if (manifestText == null || manifestText.isEmpty()) {\n throw new ArvadosClientException(\"Manifest text cannot be empty.\");\n }\n\n List manifestStreams = new ArrayList<>(Arrays.asList(manifestText.split(\"\\\\n\")));\n if (!manifestStreams.get(0).startsWith(\". \")) {\n throw new ArvadosClientException(\"Invalid first path component (expecting \\\".\\\")\");\n }\n\n return manifestStreams.stream()\n .map(this::decodeSingleManifestStream)\n .collect(toList());\n }\n\n private ManifestStream decodeSingleManifestStream(String manifestStream) {\n Objects.requireNonNull(manifestStream, \"Manifest stream cannot be empty.\");\n\n LinkedList manifestPieces = new LinkedList<>(Arrays.asList(manifestStream.split(\"\\\\s+\")));\n String streamName = manifestPieces.poll();\n String path = \".\".equals(streamName) ? \"\" : streamName.substring(2).concat(Characters.SLASH);\n\n List keepLocators = manifestPieces\n .stream()\n .filter(p -> p.matches(LOCATOR_PATTERN))\n .map(this::getKeepLocator)\n .collect(toList());\n\n\n List fileTokens = manifestPieces.stream()\n .skip(keepLocators.size())\n .filter(p -> p.matches(FILE_TOKEN_PATTERN))\n .map(p -> new FileToken(p, path))\n .collect(toList());\n\n return new ManifestStream(streamName, keepLocators, fileTokens);\n\n }\n\n private KeepLocator getKeepLocator(String locatorString ) {\n try {\n return new KeepLocator(locatorString);\n } catch (Exception e) {\n throw new RuntimeException(e);\n }\n }\n\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/logic/collection/ManifestFactory.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.logic.collection;\n\nimport com.google.common.collect.ImmutableList;\nimport org.arvados.client.common.Characters;\n\nimport java.io.File;\nimport java.util.Collection;\nimport java.util.List;\nimport java.util.stream.Collectors;\n\npublic class ManifestFactory {\n\n private Collection files;\n private List locators;\n\n ManifestFactory(Collection files, List locators) {\n this.files = files;\n this.locators = locators;\n }\n\n public static ManifestFactoryBuilder builder() {\n return new ManifestFactoryBuilder();\n }\n\n public String create() {\n ImmutableList.Builder builder = new ImmutableList.Builder()\n .add(Characters.DOT)\n .addAll(locators);\n long filePosition = 0;\n for (File file : files) {\n builder.add(String.format(\"%d:%d:%s\", filePosition, file.length(), file.getName().replace(\" \", Characters.SPACE)));\n filePosition += file.length();\n }\n String manifest = builder.build().stream().collect(Collectors.joining(\" \")).concat(Characters.NEW_LINE);\n return manifest;\n }\n\n public static class ManifestFactoryBuilder {\n private Collection files;\n private List locators;\n\n ManifestFactoryBuilder() {\n }\n\n public ManifestFactory.ManifestFactoryBuilder files(Collection files) {\n this.files = files;\n return this;\n }\n\n public ManifestFactory.ManifestFactoryBuilder locators(List locators) {\n this.locators = locators;\n return this;\n }\n\n public ManifestFactory build() {\n return new ManifestFactory(files, locators);\n }\n\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/logic/collection/ManifestStream.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.logic.collection;\n\nimport org.arvados.client.logic.keep.KeepLocator;\n\nimport java.util.List;\nimport java.util.stream.Collectors;\nimport java.util.stream.Stream;\n\npublic class ManifestStream {\n\n private String streamName;\n private List keepLocators;\n private List fileTokens;\n\n public ManifestStream(String streamName, List keepLocators, List fileTokens) {\n this.streamName = streamName;\n this.keepLocators = keepLocators;\n this.fileTokens = fileTokens;\n }\n\n @Override\n public String toString() {\n return streamName + \" \" + Stream.concat(keepLocators.stream().map(KeepLocator::toString), fileTokens.stream().map(FileToken::toString))\n .collect(Collectors.joining(\" \"));\n }\n\n public String getStreamName() {\n return this.streamName;\n }\n\n public List getKeepLocators() {\n return this.keepLocators;\n }\n\n public List getFileTokens() {\n return this.fileTokens;\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/logic/keep/FileDownloader.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.logic.keep;\n\nimport com.google.common.collect.Lists;\nimport org.arvados.client.api.client.CollectionsApiClient;\nimport org.arvados.client.api.client.KeepWebApiClient;\nimport org.arvados.client.api.model.Collection;\nimport org.arvados.client.common.Characters;\nimport org.arvados.client.exception.ArvadosClientException;\nimport org.arvados.client.logic.collection.FileToken;\nimport org.arvados.client.logic.collection.ManifestDecoder;\nimport org.arvados.client.logic.collection.ManifestStream;\nimport org.arvados.client.logic.keep.exception.DownloadFolderAlreadyExistsException;\nimport org.arvados.client.logic.keep.exception.FileAlreadyExistsException;\nimport org.slf4j.Logger;\n\nimport java.io.File;\nimport java.io.FileOutputStream;\nimport java.io.IOException;\nimport java.io.InputStream;\nimport java.io.RandomAccessFile;\nimport java.nio.file.Files;\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.concurrent.CompletableFuture;\nimport java.util.stream.Collectors;\nimport java.util.stream.Stream;\n\npublic class FileDownloader {\n\n private final ManifestDecoder manifestDecoder;\n private final CollectionsApiClient collectionsApiClient;\n private final KeepWebApiClient keepWebApiClient;\n private final Logger log = org.slf4j.LoggerFactory.getLogger(FileDownloader.class);\n\n public FileDownloader(ManifestDecoder manifestDecoder, CollectionsApiClient collectionsApiClient, KeepWebApiClient keepWebApiClient) {\n this.manifestDecoder = manifestDecoder;\n this.collectionsApiClient = collectionsApiClient;\n this.keepWebApiClient = keepWebApiClient;\n }\n\n public List listFileInfoFromCollection(String collectionUuid) {\n Collection requestedCollection = collectionsApiClient.get(collectionUuid);\n String manifestText = requestedCollection.getManifestText();\n\n // decode manifest text and get list of all FileTokens for this collection\n return manifestDecoder.decode(manifestText)\n .stream()\n .flatMap(p -> p.getFileTokens().stream())\n .collect(Collectors.toList());\n }\n\n public File downloadSingleFileUsingKeepWeb(String filePathName, String collectionUuid, String pathToDownloadFolder) {\n FileToken fileToken = getFileTokenFromCollection(filePathName, collectionUuid);\n if (fileToken == null) {\n throw new ArvadosClientException(String.format(\"%s not found in Collection with UUID %s\", filePathName, collectionUuid));\n }\n\n File downloadedFile = checkIfFileExistsInTargetLocation(fileToken, pathToDownloadFolder);\n try (FileOutputStream fos = new FileOutputStream(downloadedFile)) {\n fos.write(keepWebApiClient.download(collectionUuid, filePathName));\n } catch (IOException e) {\n throw new ArvadosClientException(String.format(\"Unable to write down file %s\", fileToken.getFileName()), e);\n }\n return downloadedFile;\n }\n\n public File downloadFileWithResume(String collectionUuid, String fileName, String pathToDownloadFolder, long start, Long end) throws IOException {\n if (end != null && end < start) {\n throw new IllegalArgumentException(\"End index must be greater than or equal to the start index\");\n }\n\n File destinationFile = new File(pathToDownloadFolder, fileName);\n\n if (!destinationFile.exists()) {\n boolean isCreated = destinationFile.createNewFile();\n if (!isCreated) {\n throw new IOException(\"Failed to create new file: \" + destinationFile.getAbsolutePath());\n }\n }\n\n try (RandomAccessFile outputFile = new RandomAccessFile(destinationFile, \"rw\");\n InputStream inputStream = keepWebApiClient.get(collectionUuid, fileName, start, end)) {\n outputFile.seek(start);\n\n long remaining = (end == null) ? Long.MAX_VALUE : end - start + 1;\n byte[] buffer = new byte[4096];\n int bytesRead;\n while ((bytesRead = inputStream.read(buffer)) != -1 && remaining > 0) {\n int bytesToWrite = (int) Math.min(bytesRead, remaining);\n outputFile.write(buffer, 0, bytesToWrite);\n remaining -= bytesToWrite;\n }\n }\n\n return destinationFile;\n }\n\n public List downloadFilesFromCollectionUsingKeepWeb(String collectionUuid, String pathToDownloadFolder) {\n String collectionTargetDir = setTargetDirectory(collectionUuid, pathToDownloadFolder).getAbsolutePath();\n List fileTokens = listFileInfoFromCollection(collectionUuid);\n\n List> futures = Lists.newArrayList();\n for (FileToken fileToken : fileTokens) {\n futures.add(CompletableFuture.supplyAsync(() -> this.downloadOneFileFromCollectionUsingKeepWeb(fileToken, collectionUuid, collectionTargetDir)));\n }\n\n @SuppressWarnings(\"unchecked\")\n CompletableFuture[] array = futures.toArray(new CompletableFuture[0]);\n return Stream.of(array)\n .map(CompletableFuture::join).collect(Collectors.toList());\n }\n\n private FileToken getFileTokenFromCollection(String filePathName, String collectionUuid) {\n return listFileInfoFromCollection(collectionUuid)\n .stream()\n .filter(p -> (p.getFullPath()).equals(filePathName))\n .findFirst()\n .orElse(null);\n }\n\n private File checkIfFileExistsInTargetLocation(FileToken fileToken, String pathToDownloadFolder) {\n String fileName = fileToken.getFileName();\n\n File downloadFile = new File(pathToDownloadFolder + Characters.SLASH + fileName);\n if (downloadFile.exists()) {\n throw new FileAlreadyExistsException(String.format(\"File %s exists in location %s\", fileName, pathToDownloadFolder));\n } else {\n return downloadFile;\n }\n }\n\n private File downloadOneFileFromCollectionUsingKeepWeb(FileToken fileToken, String collectionUuid, String pathToDownloadFolder) {\n String filePathName = fileToken.getPath() + fileToken.getFileName();\n File downloadedFile = new File(pathToDownloadFolder + Characters.SLASH + filePathName);\n downloadedFile.getParentFile().mkdirs();\n\n try (FileOutputStream fos = new FileOutputStream(downloadedFile)) {\n fos.write(keepWebApiClient.download(collectionUuid, filePathName));\n } catch (IOException e) {\n throw new RuntimeException(e);\n }\n return downloadedFile;\n }\n\n public List downloadFilesFromCollection(String collectionUuid, String pathToDownloadFolder) {\n\n // download requested collection and extract manifest text\n Collection requestedCollection = collectionsApiClient.get(collectionUuid);\n String manifestText = requestedCollection.getManifestText();\n\n // if directory with this collectionUUID does not exist - create one\n // if exists - abort (throw exception)\n File collectionTargetDir = setTargetDirectory(collectionUuid, pathToDownloadFolder);\n\n // decode manifest text and create list of ManifestStream objects containing KeepLocators and FileTokens\n List manifestStreams = manifestDecoder.decode(manifestText);\n\n //list of all downloaded files that will be returned by this method\n List downloadedFilesFromCollection = new ArrayList<>();\n\n // download files for each manifest stream\n for (ManifestStream manifestStream : manifestStreams)\n downloadedFilesFromCollection.addAll(downloadFilesFromSingleManifestStream(collectionUuid, manifestStream, collectionTargetDir));\n\n log.debug(String.format(\"Total of: %d files downloaded\", downloadedFilesFromCollection.size()));\n return downloadedFilesFromCollection;\n }\n\n private File setTargetDirectory(String collectionUUID, String pathToDownloadFolder) {\n //local directory to save downloaded files\n File collectionTargetDir = new File(pathToDownloadFolder + Characters.SLASH + collectionUUID);\n if (collectionTargetDir.exists()) {\n throw new DownloadFolderAlreadyExistsException(String.format(\"Directory for collection UUID %s already exists\", collectionUUID));\n } else {\n collectionTargetDir.mkdirs();\n }\n return collectionTargetDir;\n }\n\n private List downloadFilesFromSingleManifestStream(String collectionUuid, ManifestStream manifestStream, File collectionTargetDir){\n List downloadedFiles = new ArrayList<>();\n\n for (FileToken fileToken : manifestStream.getFileTokens()) {\n File downloadedFile = new File(collectionTargetDir.getAbsolutePath() + Characters.SLASH + fileToken.getFullPath()); //create file\n downloadedFile.getParentFile().mkdirs();\n\n try {\n byte[] download = keepWebApiClient.download(collectionUuid, fileToken.getFileName());\n Files.write(downloadedFile.toPath(), download);\n } catch (IOException | ArvadosClientException e) {\n throw new ArvadosClientException(String.format(\"Unable to write down file %s\", fileToken.getFileName()), e);\n }\n\n downloadedFiles.add(downloadedFile);\n log.debug(String.format(\"File %d / %d downloaded from manifest stream\",\n manifestStream.getFileTokens().indexOf(fileToken) + 1,\n manifestStream.getFileTokens().size()));\n }\n return downloadedFiles;\n }\n\n}" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/logic/keep/FileUploader.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.logic.keep;\n\nimport org.arvados.client.api.client.CollectionsApiClient;\nimport org.arvados.client.api.client.KeepWebApiClient;\nimport org.arvados.client.api.model.Collection;\nimport org.arvados.client.config.ConfigProvider;\nimport org.arvados.client.logic.collection.CollectionFactory;\nimport org.slf4j.Logger;\n\nimport java.io.File;\nimport java.util.List;\n\npublic class FileUploader {\n\n private final KeepWebApiClient keepWebApiClient;\n private final CollectionsApiClient collectionsApiClient;\n private final ConfigProvider config;\n private final Logger log = org.slf4j.LoggerFactory.getLogger(FileUploader.class);\n\n public FileUploader(KeepWebApiClient keepWebApiClient, CollectionsApiClient collectionsApiClient, ConfigProvider config) {\n this.keepWebApiClient = keepWebApiClient;\n this.collectionsApiClient = collectionsApiClient;\n this.config = config;\n }\n\n public Collection upload(List sourceFiles, String collectionName, String projectUuid) {\n Collection newCollection = CollectionFactory.builder()\n .config(config)\n .name(collectionName)\n .projectUuid(projectUuid)\n .build()\n .create();\n\n newCollection = collectionsApiClient.create(newCollection);\n String newCollectionId = newCollection.getUuid();\n\n sourceFiles.forEach(file -> uploadFile(newCollectionId, file));\n\n return collectionsApiClient.get(newCollection.getUuid());\n }\n\n private void uploadFile(String collectionUuid, File file) {\n keepWebApiClient.upload(collectionUuid, file, (progress) -> log.info(\"Uploaded {} bytes for file: {}\", progress, file.getName()));\n }\n\n public Collection uploadToExistingCollection(List files, String collectionUuid) {\n files.forEach(file -> uploadFile(collectionUuid, file));\n\n return collectionsApiClient.get(collectionUuid);\n }\n\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/logic/keep/KeepLocator.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.logic.keep;\n\nimport org.arvados.client.exception.ArvadosClientException;\n\nimport java.time.Instant;\nimport java.time.LocalDateTime;\nimport java.time.ZoneOffset;\nimport java.util.ArrayList;\nimport java.util.Arrays;\nimport java.util.LinkedList;\nimport java.util.List;\nimport java.util.Objects;\nimport java.util.stream.Collectors;\nimport java.util.stream.Stream;\n\nimport static org.arvados.client.common.Patterns.HINT_PATTERN;\n\npublic class KeepLocator {\n\n private final List hints = new ArrayList<>();\n private String permSig;\n private LocalDateTime permExpiry;\n private final String md5sum;\n private final Integer size;\n\n public KeepLocator(String locatorString) {\n LinkedList pieces = new LinkedList<>(Arrays.asList(locatorString.split(\"\\\\+\")));\n\n md5sum = pieces.poll();\n size = Integer.valueOf(Objects.requireNonNull(pieces.poll()));\n\n for (String hint : pieces) {\n if (!hint.matches(HINT_PATTERN)) {\n throw new ArvadosClientException(String.format(\"invalid hint format: %s\", hint));\n } else if (hint.startsWith(\"A\")) {\n parsePermissionHint(hint);\n } else {\n hints.add(hint);\n }\n }\n }\n\n public List getHints() {\n return hints;\n }\n\n public String getMd5sum() {\n return md5sum;\n }\n\n @Override\n public String toString() {\n return Stream.concat(Stream.of(md5sum, size.toString(), permissionHint()), hints.stream())\n .filter(Objects::nonNull)\n .collect(Collectors.joining(\"+\"));\n }\n\n public String stripped() {\n return size != null ? String.format(\"%s+%d\", md5sum, size) : md5sum;\n }\n\n public String permissionHint() {\n if (permSig == null || permExpiry == null) {\n return null;\n }\n\n long timestamp = permExpiry.toEpochSecond(ZoneOffset.UTC);\n String signTimestamp = Long.toHexString(timestamp);\n return String.format(\"A%s@%s\", permSig, signTimestamp);\n }\n\n private void parsePermissionHint(String hint) {\n String[] hintSplit = hint.substring(1).split(\"@\", 2);\n permSig = hintSplit[0];\n\n int permExpiryDecimal = Integer.parseInt(hintSplit[1], 16);\n permExpiry = LocalDateTime.ofInstant(Instant.ofEpochSecond(permExpiryDecimal), ZoneOffset.UTC);\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/logic/keep/exception/DownloadFolderAlreadyExistsException.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.logic.keep.exception;\n\nimport org.arvados.client.exception.ArvadosClientException;\n\n/**\n * Exception indicating that directory with given name was already created in specified location.\n *\n *

This exception will be thrown during an attempt to download all files from certain\n * collection to a location that already contains folder named by this collection's UUID.

\n */\npublic class DownloadFolderAlreadyExistsException extends ArvadosClientException {\n\n public DownloadFolderAlreadyExistsException(String message) {\n super(message);\n }\n\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/logic/keep/exception/FileAlreadyExistsException.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.logic.keep.exception;\n\nimport org.arvados.client.exception.ArvadosClientException;\n\n/**\n * Signals that an attempt to download a file with given name has failed for a specified\n * download location.\n *\n *

This exception will be thrown during an attempt to download single file to a location\n * that already contains file with given name

\n */\npublic class FileAlreadyExistsException extends ArvadosClientException {\n\n public FileAlreadyExistsException(String message) { super(message); }\n\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/utils/FileMerge.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.utils;\n\nimport java.io.BufferedOutputStream;\nimport java.io.File;\nimport java.io.FileOutputStream;\nimport java.io.IOException;\nimport java.nio.file.Files;\nimport java.util.Collection;\n\npublic class FileMerge {\n\n public static void merge(Collection files, File targetFile) throws IOException {\n try (FileOutputStream fos = new FileOutputStream(targetFile); BufferedOutputStream mergingStream = new BufferedOutputStream(fos)) {\n for (File file : files) {\n Files.copy(file.toPath(), mergingStream);\n }\n }\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/main/java/org/arvados/client/utils/FileSplit.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.utils;\n\nimport org.apache.commons.io.FileUtils;\n\nimport java.io.*;\nimport java.util.ArrayList;\nimport java.util.List;\n\n/**\n * Based on:\n * {@link} https://stackoverflow.com/questions/10864317/how-to-break-a-file-into-pieces-using-java\n */\npublic class FileSplit {\n\n public static List split(File f, File dir, int splitSize) throws IOException {\n int partCounter = 1;\n\n long sizeOfFiles = splitSize * FileUtils.ONE_MB;\n byte[] buffer = new byte[(int) sizeOfFiles];\n\n List files = new ArrayList<>();\n String fileName = f.getName();\n\n try (FileInputStream fis = new FileInputStream(f); BufferedInputStream bis = new BufferedInputStream(fis)) {\n int bytesAmount = 0;\n while ((bytesAmount = bis.read(buffer)) > 0) {\n String filePartName = String.format(\"%s.%03d\", fileName, partCounter++);\n File newFile = new File(dir, filePartName);\n try (FileOutputStream out = new FileOutputStream(newFile)) {\n out.write(buffer, 0, bytesAmount);\n }\n files.add(newFile);\n }\n }\n return files;\n }\n}" }, { "path": "contrib/java-sdk-v2/src/main/resources/reference.conf", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n#\n# Arvados client default configuration\n#\n# Remarks:\n# * While providing data remove apostrophes (\"\") from each line\n# * See Arvados documentation for information how to obtain a token:\n# https://doc.arvados.org/user/reference/api-tokens.html\n#\n\narvados {\n api {\n \tkeepweb-host = localhost\n \tkeepweb-port = 8000\n \thost = localhost\n \tport = 8000\n \ttoken = \"\"\n \tprotocol = https\n \thost-insecure = false\n }\n split-size = 64\n temp-dir = /tmp/file-split\n copies = 2\n retries = 0\n connectTimeout = 60000\n readTimeout = 60000\n writeTimeout = 60000\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/api/client/BaseStandardApiClientTest.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.client;\n\nimport okhttp3.HttpUrl;\nimport org.arvados.client.api.model.Item;\nimport org.arvados.client.api.model.ItemList;\nimport org.arvados.client.test.utils.ArvadosClientUnitTest;\nimport org.junit.Test;\nimport org.junit.runner.RunWith;\nimport org.mockito.Spy;\nimport org.mockito.junit.MockitoJUnitRunner;\n\nimport static org.assertj.core.api.Assertions.assertThat;\n\n@RunWith(MockitoJUnitRunner.class)\npublic class BaseStandardApiClientTest extends ArvadosClientUnitTest {\n\n @Spy\n private BaseStandardApiClient client = new BaseStandardApiClient(CONFIG) {\n @Override\n String getResource() {\n return \"resource\";\n }\n\n @Override\n Class getType() {\n return null;\n }\n\n @Override\n Class getListType() {\n return null;\n }\n };\n\n @Test\n public void urlBuilderBuildsExpectedUrlFormat() {\n // when\n HttpUrl.Builder actual = client.getUrlBuilder();\n\n // then\n assertThat(actual.build().toString()).isEqualTo(\"http://localhost:9000/arvados/v1/resource\");\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/api/client/CollectionsApiClientTest.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.client;\n\nimport java.io.IOException;\nimport java.util.HashMap;\nimport java.util.Map;\n\nimport com.fasterxml.jackson.databind.ObjectMapper;\nimport com.fasterxml.jackson.databind.SerializationFeature;\nimport okhttp3.mockwebserver.RecordedRequest;\nimport org.arvados.client.api.model.Collection;\nimport org.arvados.client.api.model.CollectionList;\nimport org.arvados.client.api.model.CollectionReplaceFiles;\nimport org.arvados.client.test.utils.RequestMethod;\nimport org.arvados.client.test.utils.ArvadosClientMockedWebServerTest;\nimport org.junit.Before;\nimport org.junit.Test;\n\nimport static org.arvados.client.test.utils.ApiClientTestUtils.*;\nimport static org.assertj.core.api.Assertions.assertThat;\nimport static org.junit.Assert.assertEquals;\n\npublic class CollectionsApiClientTest extends ArvadosClientMockedWebServerTest {\n\n private static final String RESOURCE = \"collections\";\n private static final String TEST_COLLECTION_NAME = \"Super Collection\";\n private static final String TEST_COLLECTION_UUID = \"test-collection-uuid\";\n private ObjectMapper objectMapper;\n private CollectionsApiClient client;\n\n @Before\n public void setUp() {\n objectMapper = new ObjectMapper();\n objectMapper.configure(SerializationFeature.ORDER_MAP_ENTRIES_BY_KEYS, true);\n client = new CollectionsApiClient(CONFIG);\n }\n\n @Test\n public void listCollections() throws Exception {\n\n // given\n server.enqueue(getResponse(\"collections-list\"));\n\n // when\n CollectionList actual = client.list();\n\n // then\n RecordedRequest request = server.takeRequest();\n assertAuthorizationHeader(request);\n assertRequestPath(request, RESOURCE);\n assertRequestMethod(request, RequestMethod.GET);\n assertThat(actual.getItemsAvailable()).isEqualTo(41);\n }\n\n @Test\n public void getCollection() throws Exception {\n\n // given\n server.enqueue(getResponse(\"collections-get\"));\n\n String uuid = \"112ci-4zz18-p51w7z3fpopo6sm\";\n\n // when\n Collection actual = client.get(uuid);\n\n // then\n RecordedRequest request = server.takeRequest();\n assertAuthorizationHeader(request);\n assertRequestPath(request, RESOURCE + \"/\" + uuid);\n assertRequestMethod(request, RequestMethod.GET);\n assertThat(actual.getUuid()).isEqualTo(uuid);\n assertThat(actual.getPortableDataHash()).isEqualTo(\"6c4106229b08fe25f48b3a7a8289dd46+143\");\n }\n\n @Test\n public void createCollection() throws Exception {\n\n // given\n server.enqueue(getResponse(\"collections-create-simple\"));\n\n String name = TEST_COLLECTION_NAME;\n \n Collection collection = new Collection();\n collection.setName(name);\n\n // when\n Collection actual = client.create(collection);\n\n // then\n RecordedRequest request = server.takeRequest();\n assertAuthorizationHeader(request);\n assertRequestPath(request, RESOURCE);\n assertRequestMethod(request, RequestMethod.POST);\n assertThat(actual.getName()).isEqualTo(name);\n assertThat(actual.getPortableDataHash()).isEqualTo(\"d41d8cd98f00b204e9800998ecf8427e+0\");\n assertThat(actual.getManifestText()).isEmpty();\n }\n\n @Test\n public void createCollectionWithManifest() throws Exception {\n\n // given\n server.enqueue(getResponse(\"collections-create-manifest\"));\n\n String name = TEST_COLLECTION_NAME;\n String manifestText = \". 7df44272090cee6c0732382bba415ee9+70+Aa5ece4560e3329315165b36c239b8ab79c888f8a@5a1d5708 0:70:README.md\\n\";\n \n Collection collection = new Collection();\n collection.setName(name);\n collection.setManifestText(manifestText);\n\n // when\n Collection actual = client.create(collection);\n\n // then\n RecordedRequest request = server.takeRequest();\n assertAuthorizationHeader(request);\n assertRequestPath(request, RESOURCE);\n assertRequestMethod(request, RequestMethod.POST);\n assertThat(actual.getName()).isEqualTo(name);\n assertThat(actual.getPortableDataHash()).isEqualTo(\"d41d8cd98f00b204e9800998ecf8427e+0\");\n assertThat(actual.getManifestText()).isEqualTo(manifestText);\n }\n\n @Test\n public void testUpdateWithReplaceFiles() throws IOException, InterruptedException {\n // given\n server.enqueue(getResponse(\"collections-create-manifest\"));\n\n Map files = new HashMap<>();\n files.put(\"targetPath1\", \"sourcePath1\");\n files.put(\"targetPath2\", \"sourcePath2\");\n\n CollectionReplaceFiles replaceFilesRequest = new CollectionReplaceFiles();\n replaceFilesRequest.setReplaceFiles(files);\n\n // when\n Collection actual = client.update(TEST_COLLECTION_UUID, replaceFilesRequest);\n\n // then\n RecordedRequest request = server.takeRequest();\n assertAuthorizationHeader(request);\n assertRequestPath(request, \"collections/test-collection-uuid\");\n assertRequestMethod(request, RequestMethod.PUT);\n assertThat(actual.getPortableDataHash()).isEqualTo(\"d41d8cd98f00b204e9800998ecf8427e+0\");\n\n String actualRequestBody = request.getBody().readUtf8();\n Map actualRequestMap = objectMapper.readValue(actualRequestBody, Map.class);\n\n Map expectedRequestMap = new HashMap<>();\n Map collectionOptionsMap = new HashMap<>();\n collectionOptionsMap.put(\"preserve_version\", true);\n\n Map replaceFilesMap = new HashMap<>();\n replaceFilesMap.put(\"targetPath1\", \"sourcePath1\");\n replaceFilesMap.put(\"targetPath2\", \"sourcePath2\");\n\n expectedRequestMap.put(\"collection\", collectionOptionsMap);\n expectedRequestMap.put(\"replace_files\", replaceFilesMap);\n\n String expectedJson = objectMapper.writeValueAsString(expectedRequestMap);\n String actualJson = objectMapper.writeValueAsString(actualRequestMap);\n assertEquals(expectedJson, actualJson);\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/api/client/GroupsApiClientTest.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.client;\n\nimport com.google.common.collect.Lists;\nimport okhttp3.mockwebserver.RecordedRequest;\nimport org.arvados.client.api.model.Group;\nimport org.arvados.client.api.model.GroupList;\nimport org.arvados.client.api.model.argument.Filter;\nimport org.arvados.client.api.model.argument.ListArgument;\nimport org.arvados.client.test.utils.RequestMethod;\nimport org.arvados.client.test.utils.ArvadosClientMockedWebServerTest;\nimport org.junit.Test;\n\nimport java.util.Arrays;\n\nimport static org.arvados.client.test.utils.ApiClientTestUtils.*;\nimport static org.junit.Assert.assertEquals;\n\npublic class GroupsApiClientTest extends ArvadosClientMockedWebServerTest {\n private static final String RESOURCE = \"groups\";\n private GroupsApiClient client = new GroupsApiClient(CONFIG);\n\n @Test\n public void listGroups() throws Exception {\n\n // given\n server.enqueue(getResponse(\"groups-list\"));\n\n // when\n GroupList actual = client.list();\n\n // then\n RecordedRequest request = server.takeRequest();\n assertAuthorizationHeader(request);\n assertRequestPath(request, RESOURCE);\n assertRequestMethod(request, RequestMethod.GET);\n assertEquals(20, actual.getItems().size());\n }\n\n @Test\n public void listProjectsByOwner() throws Exception {\n\n // given\n server.enqueue(getResponse(\"groups-list\"));\n String ownerUuid = \"ardev-tpzed-n3kzq4fvoks3uw4\";\n String filterSubPath = \"?filters=[%20[%20%22owner_uuid%22,%20%22like%22,%20%22ardev-tpzed-n3kzq4fvoks3uw4%22%20],%20\" +\n \"[%20%22group_class%22,%20%22in%22,%20[%20%22project%22,%20%22sub-project%22%20]%20]%20]\";\n\n // when\n ListArgument listArgument = ListArgument.builder()\n .filters(Arrays.asList(\n Filter.of(\"owner_uuid\", Filter.Operator.LIKE, ownerUuid),\n Filter.of(\"group_class\", Filter.Operator.IN, Lists.newArrayList(\"project\", \"sub-project\")\n )))\n .build();\n GroupList actual = client.list(listArgument);\n\n // then\n RecordedRequest request = server.takeRequest();\n assertAuthorizationHeader(request);\n assertRequestPath(request, RESOURCE + filterSubPath);\n assertRequestMethod(request, RequestMethod.GET);\n assertEquals(20, actual.getItems().size());\n }\n\n @Test\n public void getGroup() throws Exception {\n\n // given\n server.enqueue(getResponse(\"groups-get\"));\n\n String uuid = \"ardev-j7d0g-bmg3pfqtx3ivczp\";\n\n // when\n Group actual = client.get(uuid);\n\n // then\n RecordedRequest request = server.takeRequest();\n assertAuthorizationHeader(request);\n assertRequestPath(request, RESOURCE + \"/\" + uuid);\n assertRequestMethod(request, RequestMethod.GET);\n assertEquals(uuid, actual.getUuid());\n assertEquals(\"3hw0vk4mbl0ofvia5k6x4dwrx\", actual.getEtag());\n assertEquals(\"ardev-tpzed-n3kzq4fvoks3uw4\", actual.getOwnerUuid());\n assertEquals(\"TestGroup1\", actual.getName());\n assertEquals(\"project\", actual.getGroupClass());\n\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/api/client/KeepWebApiClientTest.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.client;\n\nimport org.arvados.client.test.utils.ArvadosClientMockedWebServerTest;\nimport org.junit.Test;\n\nimport java.io.ByteArrayOutputStream;\nimport java.io.File;\nimport java.io.IOException;\nimport java.io.InputStream;\nimport java.nio.file.Files;\n\nimport okhttp3.mockwebserver.MockResponse;\nimport okio.Buffer;\n\nimport static org.arvados.client.test.utils.ApiClientTestUtils.getResponse;\nimport static org.assertj.core.api.Assertions.assertThat;\nimport static org.junit.Assert.assertArrayEquals;\nimport static org.junit.Assert.assertNotNull;\n\npublic class KeepWebApiClientTest extends ArvadosClientMockedWebServerTest {\n\n private final KeepWebApiClient client = new KeepWebApiClient(CONFIG);\n\n @Test\n public void uploadFile() throws Exception {\n // given\n String collectionUuid = \"112ci-4zz18-p51w7z3fpopo6sm\";\n File file = Files.createTempFile(\"keep-upload-test\", \"txt\").toFile();\n Files.write(file.toPath(), \"test data\".getBytes());\n\n server.enqueue(getResponse(\"keep-client-upload-response\"));\n\n // when\n String uploadResponse = client.upload(collectionUuid, file, uploadedBytes -> System.out.printf(\"Uploaded bytes: %s/%s%n\", uploadedBytes, file.length()));\n\n // then\n assertThat(uploadResponse).isEqualTo(\"Created\");\n }\n\n @Test\n public void downloadPartialIsPerformedSuccessfully() throws Exception {\n // given\n String collectionUuid = \"some-collection-uuid\";\n String filePathName = \"sample-file-path\";\n long start = 1024;\n Long end = null;\n\n byte[] expectedData = \"test data\".getBytes();\n\n try (Buffer buffer = new Buffer().write(expectedData)) {\n server.enqueue(new MockResponse().setBody(buffer));\n\n // when\n InputStream inputStream = client.get(collectionUuid, filePathName, start, end);\n byte[] actualData = inputStreamToByteArray(inputStream);\n\n // then\n assertNotNull(actualData);\n assertArrayEquals(expectedData, actualData);\n }\n }\n\n private byte[] inputStreamToByteArray(InputStream inputStream) throws IOException {\n ByteArrayOutputStream buffer = new ByteArrayOutputStream();\n int nRead;\n byte[] data = new byte[1024];\n while ((nRead = inputStream.read(data, 0, data.length)) != -1) {\n buffer.write(data, 0, nRead);\n }\n buffer.flush();\n return buffer.toByteArray();\n }\n\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/api/client/LinkApiClientTest.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.client;\n\nimport okhttp3.mockwebserver.RecordedRequest;\nimport org.arvados.client.api.model.Link;\nimport org.arvados.client.api.model.LinkList;\nimport org.arvados.client.test.utils.ArvadosClientMockedWebServerTest;\nimport org.arvados.client.test.utils.RequestMethod;\nimport org.junit.Test;\n\nimport static org.arvados.client.test.utils.ApiClientTestUtils.assertAuthorizationHeader;\nimport static org.arvados.client.test.utils.ApiClientTestUtils.assertRequestMethod;\nimport static org.arvados.client.test.utils.ApiClientTestUtils.assertRequestPath;\nimport static org.arvados.client.test.utils.ApiClientTestUtils.getResponse;\nimport static org.assertj.core.api.Assertions.assertThat;\nimport static org.junit.Assert.assertEquals;\n\npublic class LinkApiClientTest extends ArvadosClientMockedWebServerTest {\n\n private static final String RESOURCE = \"links\";\n\n private final LinksApiClient client = new LinksApiClient(CONFIG);\n\n @Test\n public void listLinks() throws Exception {\n // given\n server.enqueue(getResponse(\"links-list\"));\n\n // when\n LinkList actual = client.list();\n\n // then\n RecordedRequest request = server.takeRequest();\n assertAuthorizationHeader(request);\n assertRequestPath(request, RESOURCE);\n assertRequestMethod(request, RequestMethod.GET);\n assertThat(actual.getItemsAvailable()).isEqualTo(2);\n }\n\n @Test\n public void getLink() throws Exception {\n // given\n server.enqueue(getResponse(\"links-get\"));\n\n String uuid = \"arkau-o0j2j-huxuaxbi46s1yml\";\n\n // when\n Link actual = client.get(uuid);\n\n // then\n RecordedRequest request = server.takeRequest();\n assertAuthorizationHeader(request);\n assertRequestPath(request, RESOURCE + \"/\" + uuid);\n assertRequestMethod(request, RequestMethod.GET);\n assertEquals(actual.getUuid(), uuid);\n assertEquals(actual.getName(), \"can_read\");\n assertEquals(actual.getHeadKind(), \"arvados#group\");\n assertEquals(actual.getHeadUuid(), \"arkau-j7d0g-fcedae2076pw56h\");\n assertEquals(actual.getTailUuid(), \"ardev-tpzed-n3kzq4fvoks3uw4\");\n assertEquals(actual.getTailKind(), \"arvados#user\");\n assertEquals(actual.getLinkClass(), \"permission\");\n }\n\n @Test\n public void createLink() throws Exception {\n // given\n server.enqueue(getResponse(\"links-create\"));\n\n String name = \"Star Link\";\n\n Link collection = new Link();\n collection.setName(name);\n\n // when\n Link actual = client.create(collection);\n\n // then\n RecordedRequest request = server.takeRequest();\n assertAuthorizationHeader(request);\n assertRequestPath(request, RESOURCE);\n assertRequestMethod(request, RequestMethod.POST);\n assertThat(actual.getName()).isEqualTo(name);\n assertEquals(actual.getName(), name);\n assertEquals(actual.getUuid(), \"arkau-o0j2j-huxuaxbi46s1yml\");\n assertEquals(actual.getHeadKind(), \"arvados#group\");\n assertEquals(actual.getHeadUuid(), \"arkau-j7d0g-fcedae2076pw56h\");\n assertEquals(actual.getTailUuid(), \"ardev-tpzed-n3kzq4fvoks3uw4\");\n assertEquals(actual.getTailKind(), \"arvados#user\");\n assertEquals(actual.getLinkClass(), \"star\");\n }\n}" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/api/client/UsersApiClientTest.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.client;\n\nimport okhttp3.mockwebserver.RecordedRequest;\nimport org.arvados.client.api.model.User;\nimport org.arvados.client.api.model.UserList;\nimport org.arvados.client.test.utils.RequestMethod;\nimport org.arvados.client.test.utils.ArvadosClientMockedWebServerTest;\nimport org.junit.Test;\n\nimport static org.arvados.client.common.Characters.SLASH;\nimport static org.arvados.client.test.utils.ApiClientTestUtils.*;\nimport static org.assertj.core.api.Assertions.assertThat;\n\npublic class UsersApiClientTest extends ArvadosClientMockedWebServerTest {\n\n private static final String RESOURCE = \"users\";\n private static final String USER_UUID = \"ardev-tpzed-q6dvn7sby55up1b\";\n\n private UsersApiClient client = new UsersApiClient(CONFIG);\n\n @Test\n public void listUsers() throws Exception {\n\n // given\n server.enqueue(getResponse(\"users-list\"));\n\n // when\n UserList actual = client.list();\n\n // then\n RecordedRequest request = server.takeRequest();\n assertAuthorizationHeader(request);\n assertRequestPath(request, RESOURCE);\n assertRequestMethod(request, RequestMethod.GET);\n assertThat(actual.getItemsAvailable()).isEqualTo(13);\n }\n\n @Test\n public void getUser() throws Exception {\n\n // given\n server.enqueue(getResponse(\"users-get\"));\n\n // when\n User actual = client.get(USER_UUID);\n\n // then\n RecordedRequest request = server.takeRequest();\n assertAuthorizationHeader(request);\n assertRequestPath(request, RESOURCE + SLASH + USER_UUID);\n assertRequestMethod(request, RequestMethod.GET);\n assertThat(actual.getUuid()).isEqualTo(USER_UUID);\n }\n\n @Test\n public void getCurrentUser() throws Exception {\n\n // given\n server.enqueue(getResponse(\"users-get\"));\n\n // when\n User actual = client.current();\n\n // then\n RecordedRequest request = server.takeRequest();\n assertAuthorizationHeader(request);\n assertRequestPath(request, RESOURCE + SLASH + \"current\");\n assertRequestMethod(request, RequestMethod.GET);\n assertThat(actual.getUuid()).isEqualTo(USER_UUID);\n }\n\n @Test\n public void getSystemUser() throws Exception {\n\n // given\n server.enqueue(getResponse(\"users-system\"));\n\n // when\n User actual = client.system();\n\n // then\n RecordedRequest request = server.takeRequest();\n assertAuthorizationHeader(request);\n assertRequestPath(request, RESOURCE + SLASH + \"system\");\n assertRequestMethod(request, RequestMethod.GET);\n assertThat(actual.getUuid()).isEqualTo(\"ardev-tpzed-000000000000000\");\n }\n\n @Test\n public void createUser() throws Exception {\n\n // given\n server.enqueue(getResponse(\"users-create\"));\n\n String firstName = \"John\";\n String lastName = \"Wayne\";\n String fullName = String.format(\"%s %s\", firstName, lastName);\n String username = String.format(\"%s%s\", firstName, lastName).toLowerCase();\n\n User user = new User();\n user.setFirstName(firstName);\n user.setLastName(lastName);\n user.setFullName(fullName);\n user.setUsername(username);\n\n // when\n User actual = client.create(user);\n\n // then\n RecordedRequest request = server.takeRequest();\n assertAuthorizationHeader(request);\n assertRequestPath(request, RESOURCE);\n assertRequestMethod(request, RequestMethod.POST);\n assertThat(actual.getFirstName()).isEqualTo(firstName);\n assertThat(actual.getLastName()).isEqualTo(lastName);\n assertThat(actual.getFullName()).isEqualTo(fullName);\n assertThat(actual.getUsername()).isEqualTo(username);\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/api/client/factory/OkHttpClientFactoryTest.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.api.client.factory;\n\nimport okhttp3.OkHttpClient;\nimport okhttp3.Request;\nimport okhttp3.Response;\nimport okhttp3.mockwebserver.MockResponse;\nimport org.arvados.client.test.utils.ArvadosClientMockedWebServerTest;\nimport org.junit.Assert;\nimport org.junit.Test;\nimport org.junit.runner.RunWith;\nimport org.mockito.junit.MockitoJUnitRunner;\n\nimport javax.net.ssl.KeyManagerFactory;\nimport javax.net.ssl.SSLContext;\nimport javax.net.ssl.SSLSocketFactory;\nimport javax.net.ssl.TrustManagerFactory;\nimport java.io.FileInputStream;\nimport java.security.KeyStore;\n\n\n@RunWith(MockitoJUnitRunner.class)\npublic class OkHttpClientFactoryTest extends ArvadosClientMockedWebServerTest {\n\n @Test(expected = javax.net.ssl.SSLHandshakeException.class)\n public void secureOkHttpClientIsCreated() throws Exception {\n\n // given\n OkHttpClientFactory factory = OkHttpClientFactory.INSTANCE;\n // * configure HTTPS server\n SSLSocketFactory sf = getSSLSocketFactoryWithSelfSignedCertificate();\n server.useHttps(sf, false);\n server.enqueue(new MockResponse().setBody(\"OK\"));\n // * prepare client HTTP request\n Request request = new Request.Builder()\n .url(\"https://localhost:9000/\")\n .build();\n\n // when - then (SSL certificate is verified)\n OkHttpClient actual = factory.create(false);\n Response response = actual.newCall(request).execute();\n }\n\n @Test\n public void insecureOkHttpClientIsCreated() throws Exception {\n // given\n OkHttpClientFactory factory = OkHttpClientFactory.INSTANCE;\n // * configure HTTPS server\n SSLSocketFactory sf = getSSLSocketFactoryWithSelfSignedCertificate();\n server.useHttps(sf, false);\n server.enqueue(new MockResponse().setBody(\"OK\"));\n // * prepare client HTTP request\n Request request = new Request.Builder()\n .url(\"https://localhost:9000/\")\n .build();\n\n // when (SSL certificate is not verified)\n OkHttpClient actual = factory.create(true);\n Response response = actual.newCall(request).execute();\n\n // then\n Assert.assertEquals(response.body().string(),\"OK\");\n }\n\n\n /*\n This ugly boilerplate is needed to enable self signed certificate.\n\n It requires selfsigned.keystore.jks file. It was generated with:\n keytool -genkey -v -keystore mystore.keystore.jks -alias alias_name -keyalg RSA -keysize 2048 -validity 10000\n */\n public SSLSocketFactory getSSLSocketFactoryWithSelfSignedCertificate() throws Exception {\n\n FileInputStream stream = new FileInputStream(\"src/test/resources/selfsigned.keystore.jks\");\n char[] serverKeyStorePassword = \"123456\".toCharArray();\n KeyStore serverKeyStore = KeyStore.getInstance(KeyStore.getDefaultType());\n serverKeyStore.load(stream, serverKeyStorePassword);\n\n String kmfAlgorithm = KeyManagerFactory.getDefaultAlgorithm();\n KeyManagerFactory kmf = KeyManagerFactory.getInstance(kmfAlgorithm);\n kmf.init(serverKeyStore, serverKeyStorePassword);\n\n TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(kmfAlgorithm);\n trustManagerFactory.init(serverKeyStore);\n\n SSLContext sslContext = SSLContext.getInstance(\"SSL\");\n sslContext.init(kmf.getKeyManagers(), trustManagerFactory.getTrustManagers(), null);\n return sslContext.getSocketFactory();\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/config/ExternalConfigProviderTest.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.config;\n\nimport okhttp3.mockwebserver.MockResponse;\nimport okhttp3.mockwebserver.MockWebServer;\nimport org.junit.After;\nimport org.junit.Before;\nimport org.junit.Test;\n\nimport java.io.IOException;\n\nimport static org.assertj.core.api.Assertions.assertThat;\n\npublic class ExternalConfigProviderTest {\n\n private MockWebServer mockServer;\n\n @Before\n public void setUp() throws IOException {\n mockServer = new MockWebServer();\n mockServer.start();\n }\n\n @After\n public void tearDown() throws IOException {\n mockServer.shutdown();\n }\n\n @Test\n public void testAutoFetchWebDAVConfiguration() {\n // Given\n String configResponse = \"{\\n\" +\n \" \\\"Services\\\": {\\n\" +\n \" \\\"WebDAVDownload\\\": {\\n\" +\n \" \\\"ExternalURL\\\": \\\"https://download.example.com:9000/\\\"\\n\" +\n \" }\\n\" +\n \" }\\n\" +\n \"}\";\n\n mockServer.enqueue(new MockResponse()\n .setResponseCode(200)\n .setBody(configResponse)\n .addHeader(\"Content-Type\", \"application/json\"));\n\n // When\n ExternalConfigProvider provider = ExternalConfigProvider.builder()\n .apiHost(mockServer.getHostName())\n .apiPort(mockServer.getPort())\n .apiProtocol(\"http\")\n .apiToken(\"test-token\")\n .build();\n\n // Then\n assertThat(provider.getKeepWebHost()).isEqualTo(\"download.example.com\");\n assertThat(provider.getKeepWebPort()).isEqualTo(9000);\n }\n\n @Test\n public void testAutoFetchWebDAVConfigurationWithDefaultHttpsPort() {\n // Given\n String configResponse = \"{\\n\" +\n \" \\\"Services\\\": {\\n\" +\n \" \\\"WebDAVDownload\\\": {\\n\" +\n \" \\\"ExternalURL\\\": \\\"https://download.example.com/\\\"\\n\" +\n \" }\\n\" +\n \" }\\n\" +\n \"}\";\n\n mockServer.enqueue(new MockResponse()\n .setResponseCode(200)\n .setBody(configResponse)\n .addHeader(\"Content-Type\", \"application/json\"));\n\n // When\n ExternalConfigProvider provider = ExternalConfigProvider.builder()\n .apiHost(mockServer.getHostName())\n .apiPort(mockServer.getPort())\n .apiProtocol(\"http\")\n .apiToken(\"test-token\")\n .build();\n\n // Then\n assertThat(provider.getKeepWebHost()).isEqualTo(\"download.example.com\");\n assertThat(provider.getKeepWebPort()).isEqualTo(443);\n }\n\n @Test\n public void testAutoFetchWebDAVConfigurationWithDefaultHttpPort() {\n // Given\n String configResponse = \"{\\n\" +\n \" \\\"Services\\\": {\\n\" +\n \" \\\"WebDAVDownload\\\": {\\n\" +\n \" \\\"ExternalURL\\\": \\\"http://download.example.com/\\\"\\n\" +\n \" }\\n\" +\n \" }\\n\" +\n \"}\";\n\n mockServer.enqueue(new MockResponse()\n .setResponseCode(200)\n .setBody(configResponse)\n .addHeader(\"Content-Type\", \"application/json\"));\n\n // When\n ExternalConfigProvider provider = ExternalConfigProvider.builder()\n .apiHost(mockServer.getHostName())\n .apiPort(mockServer.getPort())\n .apiProtocol(\"http\")\n .apiToken(\"test-token\")\n .build();\n\n // Then\n assertThat(provider.getKeepWebHost()).isEqualTo(\"download.example.com\");\n assertThat(provider.getKeepWebPort()).isEqualTo(80);\n }\n\n @Test\n public void testManualConfigurationTakesPrecedence() {\n // Given - server returns config but we provide manual values\n String configResponse = \"{\\n\" +\n \" \\\"Services\\\": {\\n\" +\n \" \\\"WebDAVDownload\\\": {\\n\" +\n \" \\\"ExternalURL\\\": \\\"https://auto.example.com/\\\"\\n\" +\n \" }\\n\" +\n \" }\\n\" +\n \"}\";\n\n mockServer.enqueue(new MockResponse()\n .setResponseCode(200)\n .setBody(configResponse)\n .addHeader(\"Content-Type\", \"application/json\"));\n\n // When - manual configuration is provided\n ExternalConfigProvider provider = ExternalConfigProvider.builder()\n .apiHost(mockServer.getHostName())\n .apiPort(mockServer.getPort())\n .apiProtocol(\"http\")\n .apiToken(\"test-token\")\n .keepWebHost(\"manual.example.com\")\n .keepWebPort(8080)\n .build();\n\n // Then - manual values should be used\n assertThat(provider.getKeepWebHost()).isEqualTo(\"manual.example.com\");\n assertThat(provider.getKeepWebPort()).isEqualTo(8080);\n }\n\n @Test\n public void testAutoFetchDisabled() {\n // When - auto-fetch is explicitly disabled\n ExternalConfigProvider provider = ExternalConfigProvider.builder()\n .apiHost(\"api.example.com\")\n .apiPort(443)\n .apiProtocol(\"https\")\n .apiToken(\"test-token\")\n .autoFetchWebDAV(false)\n .build();\n\n // Then - keepWeb values should be null/0\n assertThat(provider.getKeepWebHost()).isNull();\n assertThat(provider.getKeepWebPort()).isEqualTo(0);\n }\n\n @Test\n public void testHandlesApiError() {\n // Given - server returns error\n mockServer.enqueue(new MockResponse()\n .setResponseCode(500)\n .setBody(\"Internal Server Error\"));\n\n // When\n ExternalConfigProvider provider = ExternalConfigProvider.builder()\n .apiHost(mockServer.getHostName())\n .apiPort(mockServer.getPort())\n .apiProtocol(\"http\")\n .apiToken(\"test-token\")\n .build();\n\n // Then - should handle gracefully, keepWeb values should be null/0\n assertThat(provider.getKeepWebHost()).isNull();\n assertThat(provider.getKeepWebPort()).isEqualTo(0);\n }\n\n @Test\n public void testHandlesMalformedResponse() {\n // Given - server returns malformed JSON\n mockServer.enqueue(new MockResponse()\n .setResponseCode(200)\n .setBody(\"{ invalid json ]\")\n .addHeader(\"Content-Type\", \"application/json\"));\n\n // When\n ExternalConfigProvider provider = ExternalConfigProvider.builder()\n .apiHost(mockServer.getHostName())\n .apiPort(mockServer.getPort())\n .apiProtocol(\"http\")\n .apiToken(\"test-token\")\n .build();\n\n // Then - should handle gracefully\n assertThat(provider.getKeepWebHost()).isNull();\n assertThat(provider.getKeepWebPort()).isEqualTo(0);\n }\n\n @Test\n public void testHandlesMissingWebDAVInResponse() {\n // Given - server returns config without WebDAV section\n String configResponse = \"{\\n\" +\n \" \\\"Services\\\": {\\n\" +\n \" }\\n\" +\n \"}\";\n\n mockServer.enqueue(new MockResponse()\n .setResponseCode(200)\n .setBody(configResponse)\n .addHeader(\"Content-Type\", \"application/json\"));\n\n // When\n ExternalConfigProvider provider = ExternalConfigProvider.builder()\n .apiHost(mockServer.getHostName())\n .apiPort(mockServer.getPort())\n .apiProtocol(\"http\")\n .apiToken(\"test-token\")\n .build();\n\n // Then - should handle gracefully\n assertThat(provider.getKeepWebHost()).isNull();\n assertThat(provider.getKeepWebPort()).isEqualTo(0);\n }\n}" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/config/WebDAVConfigFetcherTest.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.config;\n\nimport okhttp3.mockwebserver.MockResponse;\nimport okhttp3.mockwebserver.MockWebServer;\nimport org.junit.After;\nimport org.junit.Before;\nimport org.junit.Test;\n\nimport java.io.IOException;\n\nimport static org.assertj.core.api.Assertions.assertThat;\n\npublic class WebDAVConfigFetcherTest {\n\n private MockWebServer mockServer;\n\n @Before\n public void setUp() throws IOException {\n mockServer = new MockWebServer();\n mockServer.start();\n }\n\n @After\n public void tearDown() throws IOException {\n mockServer.shutdown();\n }\n\n @Test\n public void testFetchWithValidConfig() {\n // Given\n String configResponse = \"{\\n\" +\n \" \\\"Services\\\": {\\n\" +\n \" \\\"WebDAVDownload\\\": {\\n\" +\n \" \\\"ExternalURL\\\": \\\"https://download.example.com:9000/\\\"\\n\" +\n \" }\\n\" +\n \" }\\n\" +\n \"}\";\n\n mockServer.enqueue(new MockResponse()\n .setResponseCode(200)\n .setBody(configResponse)\n .addHeader(\"Content-Type\", \"application/json\"));\n\n // When\n WebDAVConfigFetcher fetcher = new WebDAVConfigFetcher(\n \"http\", mockServer.getHostName(), mockServer.getPort(), false\n );\n WebDAVConfigFetcher.WebDAVConfig config = fetcher.fetch();\n\n // Then\n assertThat(config).isNotNull();\n assertThat(config.getHost()).isEqualTo(\"download.example.com\");\n assertThat(config.getPort()).isEqualTo(9000);\n }\n\n @Test\n public void testFetchWithDefaultHttpsPort() {\n // Given\n String configResponse = \"{\\n\" +\n \" \\\"Services\\\": {\\n\" +\n \" \\\"WebDAVDownload\\\": {\\n\" +\n \" \\\"ExternalURL\\\": \\\"https://download.example.com/\\\"\\n\" +\n \" }\\n\" +\n \" }\\n\" +\n \"}\";\n\n mockServer.enqueue(new MockResponse()\n .setResponseCode(200)\n .setBody(configResponse)\n .addHeader(\"Content-Type\", \"application/json\"));\n\n // When\n WebDAVConfigFetcher fetcher = new WebDAVConfigFetcher(\n \"http\", mockServer.getHostName(), mockServer.getPort(), false\n );\n WebDAVConfigFetcher.WebDAVConfig config = fetcher.fetch();\n\n // Then\n assertThat(config).isNotNull();\n assertThat(config.getHost()).isEqualTo(\"download.example.com\");\n assertThat(config.getPort()).isEqualTo(443);\n }\n\n @Test\n public void testFetchWithDefaultHttpPort() {\n // Given\n String configResponse = \"{\\n\" +\n \" \\\"Services\\\": {\\n\" +\n \" \\\"WebDAVDownload\\\": {\\n\" +\n \" \\\"ExternalURL\\\": \\\"http://download.example.com/\\\"\\n\" +\n \" }\\n\" +\n \" }\\n\" +\n \"}\";\n\n mockServer.enqueue(new MockResponse()\n .setResponseCode(200)\n .setBody(configResponse)\n .addHeader(\"Content-Type\", \"application/json\"));\n\n // When\n WebDAVConfigFetcher fetcher = new WebDAVConfigFetcher(\n \"http\", mockServer.getHostName(), mockServer.getPort(), false\n );\n WebDAVConfigFetcher.WebDAVConfig config = fetcher.fetch();\n\n // Then\n assertThat(config).isNotNull();\n assertThat(config.getHost()).isEqualTo(\"download.example.com\");\n assertThat(config.getPort()).isEqualTo(80);\n }\n\n @Test\n public void testFetchWithApiError() {\n // Given\n mockServer.enqueue(new MockResponse()\n .setResponseCode(500)\n .setBody(\"Internal Server Error\"));\n\n // When\n WebDAVConfigFetcher fetcher = new WebDAVConfigFetcher(\n \"http\", mockServer.getHostName(), mockServer.getPort(), false\n );\n WebDAVConfigFetcher.WebDAVConfig config = fetcher.fetch();\n\n // Then\n assertThat(config).isNull();\n }\n\n @Test\n public void testFetchWithMalformedJson() {\n // Given\n mockServer.enqueue(new MockResponse()\n .setResponseCode(200)\n .setBody(\"{ invalid json ]\")\n .addHeader(\"Content-Type\", \"application/json\"));\n\n // When\n WebDAVConfigFetcher fetcher = new WebDAVConfigFetcher(\n \"http\", mockServer.getHostName(), mockServer.getPort(), false\n );\n WebDAVConfigFetcher.WebDAVConfig config = fetcher.fetch();\n\n // Then\n assertThat(config).isNull();\n }\n\n @Test\n public void testFetchWithMissingWebDAVSection() {\n // Given\n String configResponse = \"{\\n\" +\n \" \\\"Services\\\": {\\n\" +\n \" }\\n\" +\n \"}\";\n\n mockServer.enqueue(new MockResponse()\n .setResponseCode(200)\n .setBody(configResponse)\n .addHeader(\"Content-Type\", \"application/json\"));\n\n // When\n WebDAVConfigFetcher fetcher = new WebDAVConfigFetcher(\n \"http\", mockServer.getHostName(), mockServer.getPort(), false\n );\n WebDAVConfigFetcher.WebDAVConfig config = fetcher.fetch();\n\n // Then\n assertThat(config).isNull();\n }\n\n @Test\n public void testFetchWithNullApiHost() {\n // When\n WebDAVConfigFetcher fetcher = new WebDAVConfigFetcher(\n \"https\", null, 443, false\n );\n WebDAVConfigFetcher.WebDAVConfig config = fetcher.fetch();\n\n // Then\n assertThat(config).isNull();\n }\n\n @Test\n public void testFetchWithEmptyApiHost() {\n // When\n WebDAVConfigFetcher fetcher = new WebDAVConfigFetcher(\n \"https\", \"\", 443, false\n );\n WebDAVConfigFetcher.WebDAVConfig config = fetcher.fetch();\n\n // Then\n assertThat(config).isNull();\n }\n\n @Test\n public void testFetchWithInvalidWebDAVUrl() {\n // Given\n String configResponse = \"{\\n\" +\n \" \\\"Services\\\": {\\n\" +\n \" \\\"WebDAVDownload\\\": {\\n\" +\n \" \\\"ExternalURL\\\": \\\"not-a-valid-url\\\"\\n\" +\n \" }\\n\" +\n \" }\\n\" +\n \"}\";\n\n mockServer.enqueue(new MockResponse()\n .setResponseCode(200)\n .setBody(configResponse)\n .addHeader(\"Content-Type\", \"application/json\"));\n\n // When\n WebDAVConfigFetcher fetcher = new WebDAVConfigFetcher(\n \"http\", mockServer.getHostName(), mockServer.getPort(), false\n );\n WebDAVConfigFetcher.WebDAVConfig config = fetcher.fetch();\n\n // Then\n assertThat(config).isNull();\n }\n}" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/facade/ArvadosFacadeIntegrationTest.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.facade;\n\nimport org.apache.commons.io.FileUtils;\nimport org.arvados.client.api.model.Collection;\nimport org.arvados.client.common.Characters;\nimport org.arvados.client.config.ExternalConfigProvider;\nimport org.arvados.client.junit.categories.IntegrationTests;\nimport org.arvados.client.logic.collection.FileToken;\nimport org.arvados.client.test.utils.ArvadosClientIntegrationTest;\nimport org.arvados.client.test.utils.FileTestUtils;\nimport org.junit.After;\nimport org.junit.Assert;\nimport org.junit.Before;\nimport org.junit.Test;\nimport org.junit.experimental.categories.Category;\n\nimport java.io.File;\nimport java.util.Collections;\nimport java.util.List;\nimport java.util.UUID;\n\nimport static org.arvados.client.test.utils.FileTestUtils.FILE_DOWNLOAD_TEST_DIR;\nimport static org.arvados.client.test.utils.FileTestUtils.FILE_SPLIT_TEST_DIR;\nimport static org.arvados.client.test.utils.FileTestUtils.TEST_FILE;\nimport static org.assertj.core.api.Assertions.assertThat;\nimport static org.junit.Assert.assertEquals;\nimport static org.junit.Assert.assertTrue;\n\n@Category(IntegrationTests.class)\npublic class ArvadosFacadeIntegrationTest extends ArvadosClientIntegrationTest {\n\n\n private static final String COLLECTION_NAME = \"Test collection \" + UUID.randomUUID().toString();\n private String collectionUuid;\n\n @Before\n public void setUp() throws Exception {\n FileTestUtils.createDirectory(FILE_SPLIT_TEST_DIR);\n FileTestUtils.createDirectory(FILE_DOWNLOAD_TEST_DIR);\n }\n\n @Test\n public void uploadOfFileIsPerformedSuccessfully() throws Exception {\n // given\n File file = FileTestUtils.generateFile(TEST_FILE, FileTestUtils.ONE_FOURTH_GB / 200);\n\n // when\n Collection actual = FACADE.upload(Collections.singletonList(file), COLLECTION_NAME, PROJECT_UUID);\n collectionUuid = actual.getUuid();\n\n // then\n assertThat(actual.getName()).contains(\"Test collection\");\n assertThat(actual.getManifestText()).contains(file.length() + Characters.COLON + file.getName());\n }\n\n @Test\n public void uploadOfFilesIsPerformedSuccessfully() throws Exception {\n // given\n List files = FileTestUtils.generatePredefinedFiles();\n files.addAll(FileTestUtils.generatePredefinedFiles());\n\n // when\n Collection actual = FACADE.upload(files, COLLECTION_NAME, PROJECT_UUID);\n collectionUuid = actual.getUuid();\n\n // then\n assertThat(actual.getName()).contains(\"Test collection\");\n files.forEach(f -> assertThat(actual.getManifestText()).contains(f.length() + Characters.COLON + f.getName().replace(\" \", Characters.SPACE)));\n }\n\n @Test\n public void uploadToExistingCollectionIsPerformedSuccessfully() throws Exception {\n // given\n File file = FileTestUtils.generateFile(TEST_FILE, FileTestUtils.ONE_EIGTH_GB / 500);\n Collection existing = createTestCollection();\n\n // when\n Collection actual = FACADE.uploadToExistingCollection(Collections.singletonList(file), collectionUuid);\n\n // then\n assertEquals(collectionUuid, actual.getUuid());\n assertThat(actual.getManifestText()).contains(file.length() + Characters.COLON + file.getName());\n }\n\n @Test\n public void uploadWithExternalConfigProviderWorksProperly() throws Exception {\n //given\n ArvadosFacade facade = new ArvadosFacade(buildExternalConfig());\n File file = FileTestUtils.generateFile(TEST_FILE, FileTestUtils.ONE_FOURTH_GB / 200);\n\n //when\n Collection actual = facade.upload(Collections.singletonList(file), COLLECTION_NAME, PROJECT_UUID);\n collectionUuid = actual.getUuid();\n\n //then\n assertThat(actual.getName()).contains(\"Test collection\");\n assertThat(actual.getManifestText()).contains(file.length() + Characters.COLON + file.getName());\n }\n\n @Test\n public void creationOfEmptyCollectionPerformedSuccesfully() {\n // given\n String collectionName = \"Empty collection \" + UUID.randomUUID().toString();\n\n // when\n Collection actual = FACADE.createEmptyCollection(collectionName, PROJECT_UUID);\n collectionUuid = actual.getUuid();\n\n // then\n assertEquals(collectionName, actual.getName());\n assertEquals(PROJECT_UUID, actual.getOwnerUuid());\n }\n\n @Test\n public void fileTokensAreListedFromCollection() throws Exception {\n //given\n List files = uploadTestFiles();\n\n //when\n List actual = FACADE.listFileInfoFromCollection(collectionUuid);\n\n //then\n assertEquals(files.size(), actual.size());\n for (int i = 0; i < files.size(); i++) {\n assertEquals(files.get(i).length(), actual.get(i).getFileSize());\n }\n }\n\n @Test\n public void downloadOfFilesPerformedSuccessfully() throws Exception {\n //given\n List files = uploadTestFiles();\n File destination = new File(FILE_DOWNLOAD_TEST_DIR + Characters.SLASH + collectionUuid);\n\n //when\n List actual = FACADE.downloadCollectionFiles(collectionUuid, FILE_DOWNLOAD_TEST_DIR, false);\n\n //then\n assertEquals(files.size(), actual.size());\n assertTrue(destination.exists());\n assertThat(actual).allMatch(File::exists);\n for (int i = 0; i < files.size(); i++) {\n assertEquals(files.get(i).length(), actual.get(i).length());\n }\n }\n\n @Test\n public void downloadOfFilesPerformedSuccessfullyUsingKeepWeb() throws Exception {\n //given\n List files = uploadTestFiles();\n File destination = new File(FILE_DOWNLOAD_TEST_DIR + Characters.SLASH + collectionUuid);\n\n //when\n List actual = FACADE.downloadCollectionFiles(collectionUuid, FILE_DOWNLOAD_TEST_DIR, true);\n\n //then\n assertEquals(files.size(), actual.size());\n assertTrue(destination.exists());\n assertThat(actual).allMatch(File::exists);\n for (int i = 0; i < files.size(); i++) {\n assertEquals(files.get(i).length(), actual.get(i).length());\n }\n }\n\n @Test\n public void singleFileIsDownloadedSuccessfullyUsingKeepWeb() throws Exception {\n //given\n File file = uploadSingleTestFile(false);\n\n //when\n File actual = FACADE.downloadFile(file.getName(), collectionUuid, FILE_DOWNLOAD_TEST_DIR);\n\n //then\n assertThat(actual).exists();\n assertThat(actual.length()).isEqualTo(file.length());\n }\n\n @Test\n public void downloadOfOneFileSplittedToMultipleLocatorsPerformedSuccesfully() throws Exception {\n //given\n File file = uploadSingleTestFile(true);\n\n List actual = FACADE.downloadCollectionFiles(collectionUuid, FILE_DOWNLOAD_TEST_DIR, false);\n\n Assert.assertEquals(1, actual.size());\n assertThat(actual.get(0).length()).isEqualTo(file.length());\n }\n\n @Test\n public void downloadWithExternalConfigProviderWorksProperly() throws Exception {\n //given\n ArvadosFacade facade = new ArvadosFacade(buildExternalConfig());\n List files = uploadTestFiles();\n //when\n List actual = facade.downloadCollectionFiles(collectionUuid, FILE_DOWNLOAD_TEST_DIR, false);\n\n //then\n assertEquals(files.size(), actual.size());\n assertThat(actual).allMatch(File::exists);\n for (int i = 0; i < files.size(); i++) {\n assertEquals(files.get(i).length(), actual.get(i).length());\n }\n }\n\n private ExternalConfigProvider buildExternalConfig() {\n return ExternalConfigProvider\n .builder()\n .apiHostInsecure(CONFIG.isApiHostInsecure())\n .keepWebHost(CONFIG.getKeepWebHost())\n .keepWebPort(CONFIG.getKeepWebPort())\n .apiHost(CONFIG.getApiHost())\n .apiPort(CONFIG.getApiPort())\n .apiToken(CONFIG.getApiToken())\n .apiProtocol(CONFIG.getApiProtocol())\n .fileSplitSize(CONFIG.getFileSplitSize())\n .fileSplitDirectory(CONFIG.getFileSplitDirectory())\n .numberOfCopies(CONFIG.getNumberOfCopies())\n .numberOfRetries(CONFIG.getNumberOfRetries())\n .connectTimeout(CONFIG.getConnectTimeout())\n .readTimeout(CONFIG.getReadTimeout())\n .writeTimeout(CONFIG.getWriteTimeout())\n .build();\n }\n\n private Collection createTestCollection() {\n Collection collection = FACADE.createEmptyCollection(COLLECTION_NAME, PROJECT_UUID);\n collectionUuid = collection.getUuid();\n return collection;\n }\n\n private List uploadTestFiles() throws Exception{\n createTestCollection();\n List files = FileTestUtils.generatePredefinedFiles();\n FACADE.uploadToExistingCollection(files, collectionUuid);\n return files;\n }\n\n private File uploadSingleTestFile(boolean bigFile) throws Exception{\n createTestCollection();\n Long fileSize = bigFile ? FileUtils.ONE_MB * 70 : FileTestUtils.ONE_EIGTH_GB / 100;\n File file = FileTestUtils.generateFile(TEST_FILE, fileSize);\n FACADE.uploadToExistingCollection(Collections.singletonList(file), collectionUuid);\n return file;\n }\n\n @After\n public void tearDown() throws Exception {\n FileTestUtils.cleanDirectory(FILE_SPLIT_TEST_DIR);\n FileTestUtils.cleanDirectory(FILE_DOWNLOAD_TEST_DIR);\n\n if(collectionUuid != null)\n FACADE.deleteCollection(collectionUuid);\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/facade/ArvadosFacadeTest.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.facade;\n\nimport com.fasterxml.jackson.databind.ObjectMapper;\nimport com.fasterxml.jackson.databind.ObjectWriter;\nimport okhttp3.mockwebserver.MockResponse;\nimport okio.Buffer;\nimport org.apache.commons.io.FileUtils;\nimport org.arvados.client.api.model.Collection;\nimport org.arvados.client.api.model.KeepService;\nimport org.arvados.client.api.model.KeepServiceList;\nimport org.arvados.client.common.Characters;\nimport org.arvados.client.test.utils.ArvadosClientMockedWebServerTest;\nimport org.arvados.client.test.utils.FileTestUtils;\nimport org.junit.After;\nimport org.junit.Before;\nimport org.junit.Test;\nimport org.junit.Ignore;\n\nimport java.io.File;\nimport java.nio.charset.Charset;\nimport java.nio.file.Files;\nimport java.util.ArrayList;\nimport java.util.Arrays;\nimport java.util.List;\nimport java.util.stream.Collectors;\n\nimport static org.arvados.client.test.utils.ApiClientTestUtils.getResponse;\nimport static org.arvados.client.test.utils.FileTestUtils.*;\nimport static org.assertj.core.api.Assertions.assertThat;\nimport static org.junit.Assert.assertEquals;\nimport static org.junit.Assert.assertTrue;\n\npublic class ArvadosFacadeTest extends ArvadosClientMockedWebServerTest {\n\n ArvadosFacade facade = new ArvadosFacade(CONFIG);\n\n @Before\n public void setUp() throws Exception {\n FileTestUtils.createDirectory(FILE_SPLIT_TEST_DIR);\n FileTestUtils.createDirectory(FILE_DOWNLOAD_TEST_DIR);\n }\n\n @Test\n @Ignore(\"Failing test #15041\")\n public void uploadIsPerformedSuccessfullyUsingDiskOnlyKeepServices() throws Exception {\n\n // given\n String keepServicesAccessible = setMockedServerPortToKeepServices(\"keep-services-accessible-disk-only\");\n server.enqueue(new MockResponse().setBody(keepServicesAccessible));\n\n String blockLocator = \"7df44272090cee6c0732382bba415ee9\";\n String signedBlockLocator = blockLocator + \"+70+A189a93acda6e1fba18a9dffd42b6591cbd36d55d@5a1c17b6\";\n for (int i = 0; i < 8; i++) {\n server.enqueue(new MockResponse().setBody(signedBlockLocator));\n }\n server.enqueue(getResponse(\"users-get\"));\n server.enqueue(getResponse(\"collections-create-manifest\"));\n\n FileTestUtils.generateFile(TEST_FILE, FileTestUtils.ONE_FOURTH_GB);\n\n // when\n Collection actual = facade.upload(Arrays.asList(new File(TEST_FILE)), \"Super Collection\", null);\n\n // then\n assertThat(actual.getName()).contains(\"Super Collection\");\n }\n\n @Test\n public void uploadIsPerformedSuccessfully() throws Exception {\n\n // given\n // First response: get current user (called by CollectionFactory when projectUuid is null)\n server.enqueue(getResponse(\"users-get\"));\n\n // Second response: create collection\n server.enqueue(getResponse(\"collections-create-manifest\"));\n\n // Third response: upload file to KeepWeb (it returns empty response)\n server.enqueue(new MockResponse().setBody(\"\"));\n\n // Fourth response: get the updated collection\n server.enqueue(getResponse(\"collections-create-manifest\"));\n\n FileTestUtils.generateFile(TEST_FILE, FileTestUtils.ONE_FOURTH_GB);\n\n // when\n Collection actual = facade.upload(Arrays.asList(new File(TEST_FILE)), \"Super Collection\", null);\n\n // then\n assertThat(actual.getName()).contains(\"Super Collection\");\n }\n\n @Test\n public void downloadOfWholeCollectionIsPerformedSuccessfully() throws Exception {\n\n //given\n String collectionUuid = \"ardev-4zz18-jk5vo4uo9u5vj52\";\n server.enqueue(getResponse(\"collections-download-file\"));\n\n // Mock KeepWeb API responses for each file\n List files = generatePredefinedFiles();\n for (File f : files) {\n server.enqueue(new MockResponse().setBody(new Buffer().write(Files.readAllBytes(f.toPath()))));\n }\n\n //when\n List downloadedFiles = facade.downloadCollectionFiles(collectionUuid, FILE_DOWNLOAD_TEST_DIR, false);\n\n //then\n File collectionDestination = new File(FILE_DOWNLOAD_TEST_DIR + Characters.SLASH + collectionUuid);\n assertEquals(3, downloadedFiles.size());\n assertTrue(collectionDestination.exists());\n assertThat(downloadedFiles).allMatch(File::exists);\n assertEquals(files.stream().map(File::getName).collect(Collectors.toList()), downloadedFiles.stream().map(File::getName).collect(Collectors.toList()));\n assertEquals(files.stream().map(File::length).collect(Collectors.toList()), downloadedFiles.stream().map(File::length).collect(Collectors.toList()));\n }\n\n @Test\n public void downloadOfWholeCollectionUsingKeepWebPerformedSuccessfully() throws Exception {\n\n //given\n String collectionUuid = \"ardev-4zz18-jk5vo4uo9u5vj52\";\n server.enqueue(getResponse(\"collections-download-file\"));\n\n List files = generatePredefinedFiles();\n for (File f : files) {\n server.enqueue(new MockResponse().setBody(new Buffer().write(FileUtils.readFileToByteArray(f))));\n }\n\n //when\n List downloadedFiles = facade.downloadCollectionFiles(collectionUuid, FILE_DOWNLOAD_TEST_DIR, true);\n\n //then\n assertEquals(3, downloadedFiles.size());\n assertThat(downloadedFiles).allMatch(File::exists);\n assertEquals(files.stream().map(File::getName).collect(Collectors.toList()), downloadedFiles.stream().map(File::getName).collect(Collectors.toList()));\n assertTrue(downloadedFiles.stream().map(File::length).collect(Collectors.toList()).containsAll(files.stream().map(File::length).collect(Collectors.toList())));\n }\n\n @Test\n public void downloadOfSingleFilePerformedSuccessfully() throws Exception {\n\n //given\n String collectionUuid = \"ardev-4zz18-jk5vo4uo9u5vj52\";\n server.enqueue(getResponse(\"collections-download-file\"));\n\n File file = generatePredefinedFiles().get(0);\n byte[] fileData = FileUtils.readFileToByteArray(file);\n server.enqueue(new MockResponse().setBody(new Buffer().write(fileData)));\n\n //when\n File downloadedFile = facade.downloadFile(file.getName(), collectionUuid, FILE_DOWNLOAD_TEST_DIR);\n\n //then\n assertTrue(downloadedFile.exists());\n assertEquals(file.getName(), downloadedFile.getName());\n assertEquals(file.length(), downloadedFile.length());\n }\n\n private String setMockedServerPortToKeepServices(String jsonPath) throws Exception {\n\n ObjectMapper mapper = new ObjectMapper().findAndRegisterModules();\n String filePath = String.format(\"src/test/resources/org/arvados/client/api/client/%s.json\", jsonPath);\n File jsonFile = new File(filePath);\n String json = FileUtils.readFileToString(jsonFile, Charset.defaultCharset());\n KeepServiceList keepServiceList = mapper.readValue(json, KeepServiceList.class);\n List items = keepServiceList.getItems();\n for (KeepService keepService : items) {\n keepService.setServicePort(server.getPort());\n }\n ObjectWriter writer = mapper.writer().withDefaultPrettyPrinter();\n return writer.writeValueAsString(keepServiceList);\n }\n\n //Method to copy multiple byte[] arrays into one byte[] array\n private byte[] addAll(byte[] array1, byte[] array2) {\n byte[] joinedArray = new byte[array1.length + array2.length];\n System.arraycopy(array1, 0, joinedArray, 0, array1.length);\n System.arraycopy(array2, 0, joinedArray, array1.length, array2.length);\n return joinedArray;\n }\n\n @After\n public void tearDown() throws Exception {\n FileTestUtils.cleanDirectory(FILE_SPLIT_TEST_DIR);\n FileTestUtils.cleanDirectory(FILE_DOWNLOAD_TEST_DIR);\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/junit/categories/IntegrationTests.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.junit.categories;\n\npublic interface IntegrationTests {}\n" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/logic/collection/FileTokenTest.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.logic.collection;\n\nimport org.arvados.client.common.Characters;\nimport org.junit.Assert;\nimport org.junit.Test;\n\npublic class FileTokenTest {\n\n public static final String FILE_TOKEN_INFO = \"0:1024:test-file1\";\n public static final int FILE_POSITION = 0;\n public static final long FILE_LENGTH = 1024L;\n public static final String FILE_NAME = \"test-file1\";\n public static final String FILE_PATH = \"c\" + Characters.SLASH;\n\n private static FileToken fileToken = new FileToken(FILE_TOKEN_INFO);\n private static FileToken fileTokenWithPath = new FileToken(FILE_TOKEN_INFO, FILE_PATH);\n\n @Test\n public void tokenInfoIsDividedCorrectly(){\n Assert.assertEquals(FILE_NAME, fileToken.getFileName());\n Assert.assertEquals(FILE_POSITION, fileToken.getFilePosition());\n Assert.assertEquals(FILE_LENGTH, fileToken.getFileSize());\n }\n\n @Test\n public void toStringReturnsOriginalFileTokenInfo(){\n Assert.assertEquals(FILE_TOKEN_INFO, fileToken.toString());\n }\n\n @Test\n public void fullPathIsReturnedProperly(){\n Assert.assertEquals(FILE_NAME, fileToken.getFullPath());\n Assert.assertEquals(FILE_PATH + FILE_NAME, fileTokenWithPath.getFullPath());\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/logic/collection/ManifestDecoderTest.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.logic.collection;\n\nimport org.arvados.client.exception.ArvadosClientException;\nimport org.junit.Assert;\nimport org.junit.Test;\n\nimport java.util.List;\n\nimport static junit.framework.TestCase.fail;\n\npublic class ManifestDecoderTest {\n\n private ManifestDecoder manifestDecoder = new ManifestDecoder();\n\n private static final String ONE_LINE_MANIFEST_TEXT = \". \" +\n \"eff999f3b5158331eb44a9a93e3b36e1+67108864+Aad3839bea88bce22cbfe71cf4943de7dab3ea52a@5826180f \" +\n \"db141bfd11f7da60dce9e5ee85a988b8+34038725+Ae8f48913fed782cbe463e0499ab37697ee06a2f8@5826180f \" +\n \"0:101147589:rna.SRR948778.bam\" +\n \"\\\\n\";\n\n private static final String MULTIPLE_LINES_MANIFEST_TEXT = \". \" +\n \"930625b054ce894ac40596c3f5a0d947+33 \" +\n \"0:0:a 0:0:b 0:33:output.txt\\n\" +\n \"./c d41d8cd98f00b204e9800998ecf8427e+0 0:0:d\";\n\n private static final String MANIFEST_TEXT_WITH_INVALID_FIRST_PATH_COMPONENT = \"a\" + ONE_LINE_MANIFEST_TEXT;\n\n\n @Test\n public void allLocatorsAndFileTokensAreExtractedFromSimpleManifest() {\n\n List actual = manifestDecoder.decode(ONE_LINE_MANIFEST_TEXT);\n\n // one manifest stream\n Assert.assertEquals(1, actual.size());\n\n ManifestStream manifest = actual.get(0);\n // two locators\n Assert.assertEquals(2, manifest.getKeepLocators().size());\n // one file token\n Assert.assertEquals(1, manifest.getFileTokens().size());\n\n }\n\n @Test\n public void allLocatorsAndFileTokensAreExtractedFromComplexManifest() {\n\n List actual = manifestDecoder.decode(MULTIPLE_LINES_MANIFEST_TEXT);\n\n // two manifest streams\n Assert.assertEquals(2, actual.size());\n\n // first stream - 1 locator and 3 file tokens\n ManifestStream firstManifestStream = actual.get(0);\n Assert.assertEquals(1, firstManifestStream.getKeepLocators().size());\n Assert.assertEquals(3, firstManifestStream.getFileTokens().size());\n\n // second stream - 1 locator and 1 file token\n ManifestStream secondManifestStream = actual.get(1);\n Assert.assertEquals(1, secondManifestStream.getKeepLocators().size());\n Assert.assertEquals(1, secondManifestStream.getFileTokens().size());\n }\n\n @Test\n public void manifestTextWithInvalidStreamNameThrowsException() {\n\n try {\n List actual = manifestDecoder.decode(MANIFEST_TEXT_WITH_INVALID_FIRST_PATH_COMPONENT);\n fail();\n } catch (ArvadosClientException e) {\n Assert.assertEquals(\"Invalid first path component (expecting \\\".\\\")\", e.getMessage());\n }\n\n }\n\n @Test\n public void emptyManifestTextThrowsException() {\n String emptyManifestText = null;\n\n try {\n List actual = manifestDecoder.decode(emptyManifestText);\n fail();\n } catch (ArvadosClientException e) {\n Assert.assertEquals(\"Manifest text cannot be empty.\", e.getMessage());\n }\n\n emptyManifestText = \"\";\n try {\n List actual = manifestDecoder.decode(emptyManifestText);\n fail();\n } catch (ArvadosClientException e) {\n Assert.assertEquals(\"Manifest text cannot be empty.\", e.getMessage());\n }\n\n }\n\n\n\n\n\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/logic/collection/ManifestFactoryTest.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.logic.collection;\n\nimport org.arvados.client.test.utils.FileTestUtils;\nimport org.assertj.core.util.Lists;\nimport org.junit.Test;\nimport org.junit.Ignore;\n\nimport java.io.File;\nimport java.util.List;\n\nimport static org.assertj.core.api.Assertions.assertThat;\n\npublic class ManifestFactoryTest {\n\n @Test\n @Ignore(\"Failing test #15041\")\n public void manifestIsCreatedAsExpected() throws Exception {\n\n // given\n List files = FileTestUtils.generatePredefinedFiles();\n List locators = Lists.newArrayList(\"a\", \"b\", \"c\");\n ManifestFactory factory = ManifestFactory.builder()\n .files(files)\n .locators(locators)\n .build();\n\n // when\n String actual = factory.create();\n\n // then\n assertThat(actual).isEqualTo(\". a b c 0:1024:test-file1 1024:20480:test-file2 21504:1048576:test-file\\\\0403\\n\");\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/logic/collection/ManifestStreamTest.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.logic.collection;\n\n\nimport org.junit.Assert;\nimport org.junit.Test;\n\nimport java.util.List;\n\npublic class ManifestStreamTest {\n\n private ManifestDecoder manifestDecoder = new ManifestDecoder();\n\n @Test\n public void toStringReturnsProperlyConnectedManifestStream() throws Exception{\n String encodedManifest = \". eff999f3b5158331eb44a9a93e3b36e1+67108864 db141bfd11f7da60dce9e5ee85a988b8+34038725 0:101147589:rna.SRR948778.bam\\\\n\\\"\";\n List manifestStreams = manifestDecoder.decode(encodedManifest);\n Assert.assertEquals(encodedManifest, manifestStreams.get(0).toString());\n\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/logic/keep/FileDownloaderTest.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.logic.keep;\n\nimport com.fasterxml.jackson.databind.ObjectMapper;\nimport org.arvados.client.api.client.CollectionsApiClient;\nimport org.arvados.client.api.client.KeepWebApiClient;\nimport org.arvados.client.api.model.Collection;\nimport org.arvados.client.common.Characters;\nimport org.arvados.client.logic.collection.FileToken;\nimport org.arvados.client.logic.collection.ManifestDecoder;\nimport org.arvados.client.logic.collection.ManifestStream;\nimport org.arvados.client.test.utils.FileTestUtils;\nimport org.apache.commons.io.FileUtils;\nimport org.junit.After;\nimport org.junit.Before;\nimport org.junit.Test;\nimport org.junit.runner.RunWith;\nimport org.mockito.InjectMocks;\nimport org.mockito.Mock;\nimport org.mockito.junit.MockitoJUnitRunner;\n\nimport java.io.ByteArrayInputStream;\nimport java.io.File;\nimport java.io.IOException;\nimport java.io.InputStream;\nimport java.nio.charset.StandardCharsets;\nimport java.nio.file.Files;\nimport java.util.ArrayList;\nimport java.util.Arrays;\nimport java.util.List;\n\nimport static org.arvados.client.test.utils.FileTestUtils.*;\nimport static org.assertj.core.api.Assertions.assertThat;\nimport static org.junit.Assert.assertArrayEquals;\nimport static org.junit.Assert.assertEquals;\nimport static org.junit.Assert.assertNotNull;\nimport static org.junit.Assert.assertTrue;\nimport static org.mockito.Mockito.when;\n\n@RunWith(MockitoJUnitRunner.class)\npublic class FileDownloaderTest {\n\n static final ObjectMapper MAPPER = new ObjectMapper().findAndRegisterModules();\n private Collection collectionToDownload;\n private ManifestStream manifestStream;\n\n @Mock\n private CollectionsApiClient collectionsApiClient;\n @Mock\n private KeepWebApiClient keepWebApiClient;\n @Mock\n private ManifestDecoder manifestDecoder;\n @InjectMocks\n private FileDownloader fileDownloader;\n\n @Before\n public void setUp() throws Exception {\n FileTestUtils.createDirectory(FILE_SPLIT_TEST_DIR);\n FileTestUtils.createDirectory(FILE_DOWNLOAD_TEST_DIR);\n\n collectionToDownload = prepareCollection();\n manifestStream = prepareManifestStream();\n }\n\n @Test\n public void downloadingAllFilesFromCollectionWorksProperly() throws Exception {\n // given\n List files = generatePredefinedFiles();\n\n //having\n when(collectionsApiClient.get(collectionToDownload.getUuid())).thenReturn(collectionToDownload);\n when(manifestDecoder.decode(collectionToDownload.getManifestText())).thenReturn(Arrays.asList(manifestStream));\n\n // Mock download responses for all three files based on the file tokens\n when(keepWebApiClient.download(collectionToDownload.getUuid(), \"test-file1\")).thenReturn(FileUtils.readFileToByteArray(files.get(0)));\n when(keepWebApiClient.download(collectionToDownload.getUuid(), \"test-file2\")).thenReturn(FileUtils.readFileToByteArray(files.get(1)));\n when(keepWebApiClient.download(collectionToDownload.getUuid(), \"test-file 3\")).thenReturn(FileUtils.readFileToByteArray(files.get(2)));\n\n //when\n List downloadedFiles = fileDownloader.downloadFilesFromCollection(collectionToDownload.getUuid(), FILE_DOWNLOAD_TEST_DIR);\n\n //then\n assertEquals(3, downloadedFiles.size()); // 3 files downloaded\n\n File collectionDir = new File(FILE_DOWNLOAD_TEST_DIR + Characters.SLASH + collectionToDownload.getUuid());\n assertTrue(collectionDir.exists()); // collection directory created\n\n // 3 files correctly saved\n assertThat(downloadedFiles).allMatch(File::exists);\n\n // Verify file contents match\n File downloaded1 = new File(collectionDir + Characters.SLASH + \"test-file1\");\n File downloaded2 = new File(collectionDir + Characters.SLASH + \"test-file2\");\n File downloaded3 = new File(collectionDir + Characters.SLASH + \"test-file 3\");\n\n assertArrayEquals(FileUtils.readFileToByteArray(downloaded1), FileUtils.readFileToByteArray(files.get(0)));\n assertArrayEquals(FileUtils.readFileToByteArray(downloaded2), FileUtils.readFileToByteArray(files.get(1)));\n assertArrayEquals(FileUtils.readFileToByteArray(downloaded3), FileUtils.readFileToByteArray(files.get(2)));\n }\n\n @Test\n public void downloadingSingleFileFromKeepWebWorksCorrectly() throws Exception{\n //given\n File file = generatePredefinedFiles().get(0);\n\n //having\n when(collectionsApiClient.get(collectionToDownload.getUuid())).thenReturn(collectionToDownload);\n when(manifestDecoder.decode(collectionToDownload.getManifestText())).thenReturn(Arrays.asList(manifestStream));\n when(keepWebApiClient.download(collectionToDownload.getUuid(), file.getName())).thenReturn(FileUtils.readFileToByteArray(file));\n\n //when\n File downloadedFile = fileDownloader.downloadSingleFileUsingKeepWeb(file.getName(), collectionToDownload.getUuid(), FILE_DOWNLOAD_TEST_DIR);\n\n //then\n assertTrue(downloadedFile.exists());\n assertEquals(file.getName(), downloadedFile.getName());\n assertArrayEquals(FileUtils.readFileToByteArray(downloadedFile), FileUtils.readFileToByteArray(file));\n }\n\n @Test\n public void testDownloadFileWithResume() throws Exception {\n //given\n String collectionUuid = \"some-collection-uuid\";\n String expectedDataString = \"testData\";\n String fileName = \"sample-file-name\";\n long start = 0;\n Long end = null;\n\n InputStream inputStream = new ByteArrayInputStream(expectedDataString.getBytes());\n\n when(keepWebApiClient.get(collectionUuid, fileName, start, end)).thenReturn(inputStream);\n\n //when\n File downloadedFile = fileDownloader.downloadFileWithResume(collectionUuid, fileName, FILE_DOWNLOAD_TEST_DIR, start, end);\n\n //then\n assertNotNull(downloadedFile);\n assertTrue(downloadedFile.exists());\n String actualDataString = Files.readString(downloadedFile.toPath());\n assertEquals(\"The content of the file does not match the expected data.\", expectedDataString, actualDataString);\n }\n\n @After\n public void tearDown() throws Exception {\n FileTestUtils.cleanDirectory(FILE_SPLIT_TEST_DIR);\n FileTestUtils.cleanDirectory(FILE_DOWNLOAD_TEST_DIR);\n }\n\n private Collection prepareCollection() throws IOException {\n // collection that will be returned by mocked collectionsApiClient\n String filePath = \"src/test/resources/org/arvados/client/api/client/collections-download-file.json\";\n File jsonFile = new File(filePath);\n return MAPPER.readValue(jsonFile, Collection.class);\n }\n\n private ManifestStream prepareManifestStream() throws Exception {\n // manifestStream that will be returned by mocked manifestDecoder\n List fileTokens = new ArrayList<>();\n fileTokens.add(new FileToken(\"0:1024:test-file1\"));\n fileTokens.add(new FileToken(\"1024:20480:test-file2\"));\n fileTokens.add(new FileToken(\"21504:1048576:test-file\\\\0403\"));\n\n KeepLocator keepLocator = new KeepLocator(\"163679d58edaadc28db769011728a72c+1070080+A3acf8c1fe582c265d2077702e4a7d74fcc03aba8@5aa4fdeb\");\n return new ManifestStream(\".\", Arrays.asList(keepLocator), fileTokens);\n }\n\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/logic/keep/KeepLocatorTest.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.logic.keep;\n\nimport org.junit.Test;\n\nimport static org.assertj.core.api.Assertions.assertThat;\n\npublic class KeepLocatorTest {\n\n private KeepLocator locator;\n\n @Test\n public void md5sumIsExtracted() throws Exception {\n\n // given\n locator = new KeepLocator(\"7df44272090cee6c0732382bba415ee9+70\");\n\n // when\n String actual = locator.getMd5sum();\n\n // then\n assertThat(actual).isEqualTo(\"7df44272090cee6c0732382bba415ee9\");\n }\n\n @Test\n public void locatorIsStrippedWithMd5sumAndSize() throws Exception {\n\n // given\n locator = new KeepLocator(\"7df44272090cee6c0732382bba415ee9+70\");\n\n // when\n String actual = locator.stripped();\n\n // then\n assertThat(actual).isEqualTo(\"7df44272090cee6c0732382bba415ee9+70\");\n }\n\n\n @Test\n public void locatorToStringProperlyShowing() throws Exception {\n\n // given\n locator = new KeepLocator(\"7df44272090cee6c0732382bba415ee9+70+Ae8f48913fed782cbe463e0499ab37697ee06a2f8@5826180f\");\n\n // when\n String actual = locator.toString();\n\n // then\n assertThat(actual).isEqualTo(\"7df44272090cee6c0732382bba415ee9+70+Ae8f48913fed782cbe463e0499ab37697ee06a2f8@5826180f\");\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/test/utils/ApiClientTestUtils.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.test.utils;\n\nimport org.arvados.client.config.FileConfigProvider;\nimport okhttp3.mockwebserver.MockResponse;\nimport okhttp3.mockwebserver.RecordedRequest;\nimport org.apache.commons.io.FileUtils;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.nio.charset.Charset;\n\nimport static org.assertj.core.api.Assertions.assertThat;\n\npublic final class ApiClientTestUtils {\n\n static final String BASE_URL = \"/arvados/v1/\";\n\n private ApiClientTestUtils() {}\n\n public static MockResponse getResponse(String filename) throws IOException {\n String filePath = String.format(\"src/test/resources/org/arvados/client/api/client/%s.json\", filename);\n File jsonFile = new File(filePath);\n String json = FileUtils.readFileToString(jsonFile, Charset.defaultCharset());\n return new MockResponse().setBody(json);\n }\n\n public static void assertAuthorizationHeader(RecordedRequest request) {\n assertThat(request.getHeader(\"authorization\")).isEqualTo(\"Bearer \" + new FileConfigProvider().getApiToken());\n }\n\n public static void assertRequestPath(RecordedRequest request, String subPath) {\n assertThat(request.getPath()).isEqualTo(BASE_URL + subPath);\n }\n\n public static void assertRequestMethod(RecordedRequest request, RequestMethod requestMethod) {\n assertThat(request.getMethod()).isEqualTo(requestMethod.name());\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/test/utils/ArvadosClientIntegrationTest.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.test.utils;\n\nimport org.arvados.client.config.FileConfigProvider;\nimport org.arvados.client.facade.ArvadosFacade;\nimport org.junit.BeforeClass;\n\nimport static org.junit.Assert.assertTrue;\n\npublic class ArvadosClientIntegrationTest {\n\n protected static final FileConfigProvider CONFIG = new FileConfigProvider(\"integration-tests-application.conf\");\n protected static final ArvadosFacade FACADE = new ArvadosFacade(CONFIG);\n protected static final String PROJECT_UUID = CONFIG.getIntegrationTestProjectUuid();\n\n @BeforeClass\n public static void validateConfiguration(){\n String msg = \" info must be provided in configuration\";\n CONFIG.getConfig().entrySet()\n .forEach(e -> assertTrue(\"Parameter \" + e.getKey() + msg, !e.getValue().render().equals(\"\\\"\\\"\")));\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/test/utils/ArvadosClientMockedWebServerTest.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.test.utils;\n\nimport okhttp3.mockwebserver.MockWebServer;\nimport org.junit.After;\nimport org.junit.Before;\n\npublic class ArvadosClientMockedWebServerTest extends ArvadosClientUnitTest {\n private static final int PORT = CONFIG.getApiPort();\n protected MockWebServer server = new MockWebServer();\n\n @Before\n public void setUpServer() throws Exception {\n server.start(PORT);\n }\n \n @After\n public void tearDownServer() throws Exception {\n server.shutdown();\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/test/utils/ArvadosClientUnitTest.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.test.utils;\n\nimport org.arvados.client.config.FileConfigProvider;\nimport org.junit.BeforeClass;\n\nimport static org.junit.Assert.assertTrue;\n\npublic class ArvadosClientUnitTest {\n\n protected static final FileConfigProvider CONFIG = new FileConfigProvider(\"application.conf\");\n\n @BeforeClass\n public static void validateConfiguration(){\n String msg = \" info must be provided in configuration\";\n CONFIG.getConfig().entrySet().forEach(e -> assertTrue(\"Parameter \" + e.getKey() + msg, !e.getValue().render().equals(\"\\\"\\\"\")));\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/test/utils/FileTestUtils.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.test.utils;\n\nimport org.apache.commons.io.FileUtils;\nimport org.assertj.core.util.Lists;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.io.RandomAccessFile;\nimport java.util.List;\n\npublic class FileTestUtils {\n\n public static final String FILE_SPLIT_TEST_DIR = \"/tmp/file-split\";\n public static final String FILE_DOWNLOAD_TEST_DIR = \"/tmp/arvados-downloaded\";\n public static final String TEST_FILE = FILE_SPLIT_TEST_DIR + \"/test-file\";\n public static long ONE_FOURTH_GB = FileUtils.ONE_GB / 4;\n public static long ONE_EIGTH_GB = FileUtils.ONE_GB / 8;\n public static long HALF_GB = FileUtils.ONE_GB / 2;\n public static int FILE_SPLIT_SIZE = 64;\n\n public static void createDirectory(String path) throws Exception {\n new File(path).mkdirs();\n }\n\n public static void cleanDirectory(String directory) throws Exception {\n FileUtils.cleanDirectory(new File(directory));\n }\n \n public static File generateFile(String path, long length) throws IOException {\n RandomAccessFile testFile = new RandomAccessFile(path, \"rwd\");\n testFile.setLength(length);\n testFile.close();\n return new File(path);\n }\n \n public static List generatePredefinedFiles() throws IOException {\n return Lists.newArrayList(\n generateFile(TEST_FILE + 1, FileUtils.ONE_KB),\n generateFile(TEST_FILE + 2, FileUtils.ONE_KB * 20),\n generateFile(TEST_FILE + \" \" + 3, FileUtils.ONE_MB)\n );\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/test/utils/RequestMethod.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.test.utils;\n\npublic enum RequestMethod {\n \n GET, POST, PUT, DELETE\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/utils/FileMergeTest.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.utils;\n\nimport org.arvados.client.test.utils.FileTestUtils;\nimport org.junit.After;\nimport org.junit.Before;\nimport org.junit.Test;\n\nimport java.io.File;\nimport java.util.List;\n\nimport static org.arvados.client.test.utils.FileTestUtils.*;\nimport static org.assertj.core.api.Assertions.assertThat;\n\npublic class FileMergeTest {\n\n @Before\n public void setUp() throws Exception {\n FileTestUtils.createDirectory(FILE_SPLIT_TEST_DIR);\n }\n\n @Test\n public void fileChunksAreMergedIntoOneFile() throws Exception {\n\n // given\n FileTestUtils.generateFile(TEST_FILE, FileTestUtils.ONE_EIGTH_GB);\n\n List files = FileSplit.split(new File(TEST_FILE), new File(FILE_SPLIT_TEST_DIR), FILE_SPLIT_SIZE);\n File targetFile = new File(TEST_FILE);\n\n // when\n FileMerge.merge(files, targetFile);\n\n // then\n assertThat(targetFile.length()).isEqualTo(FileTestUtils.ONE_EIGTH_GB);\n }\n\n @After\n public void tearDown() throws Exception {\n FileTestUtils.cleanDirectory(FILE_SPLIT_TEST_DIR);\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/java/org/arvados/client/utils/FileSplitTest.java", "content": "/*\n * Copyright (C) The Arvados Authors. All rights reserved.\n *\n * SPDX-License-Identifier: AGPL-3.0 OR Apache-2.0\n *\n */\n\npackage org.arvados.client.utils;\n\nimport org.arvados.client.test.utils.FileTestUtils;\nimport org.junit.After;\nimport org.junit.Before;\nimport org.junit.Test;\n\nimport java.io.File;\nimport java.util.List;\n\nimport static org.arvados.client.test.utils.FileTestUtils.*;\nimport static org.assertj.core.api.Assertions.assertThat;\n\npublic class FileSplitTest {\n\n @Before\n public void setUp() throws Exception {\n FileTestUtils.createDirectory(FILE_SPLIT_TEST_DIR);\n }\n\n @Test\n public void fileIsDividedIntoSmallerChunks() throws Exception {\n\n // given\n int expectedSize = 2;\n int expectedFileSizeInBytes = 67108864;\n FileTestUtils.generateFile(TEST_FILE, FileTestUtils.ONE_EIGTH_GB);\n\n // when\n List actual = FileSplit.split(new File(TEST_FILE), new File(FILE_SPLIT_TEST_DIR), FILE_SPLIT_SIZE);\n\n // then\n assertThat(actual).hasSize(expectedSize);\n assertThat(actual).allMatch(a -> a.length() == expectedFileSizeInBytes);\n }\n\n @After\n public void tearDown() throws Exception {\n FileTestUtils.cleanDirectory(FILE_SPLIT_TEST_DIR);\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/resources/application.conf", "content": "# configuration for unit tests\n\narvados {\n api {\n port = 9000\n keepweb-port = 9000\n token = 1m69yw9m2wanubzyfkb1e9icplqhtr2r969bu9rnzqbqhb7cnb\n protocol = \"http\"\n }\n}" }, { "path": "contrib/java-sdk-v2/src/test/resources/integration-tests-application.conf", "content": "# Configuration for integration tests\n#\n# Remarks:\n# * For example see integration-tests-application.conf.example\n# * While providing data remove apostrophes (\"\") from each line\n# * See Arvados documentation for information how to obtain a token:\n# https://doc.arvados.org/user/reference/api-tokens.html\n#\n\narvados {\n api {\n keepweb-host = \"\"\n keepweb-port = 443\n host = \"\"\n port = 443\n token = \"\"\n protocol = https\n host-insecure = false\n }\n integration-tests {\n project-uuid = \"\"\n }\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/resources/integration-tests-application.conf.example", "content": "# example configuration for integration tests\n\narvados {\n api {\n keepweb-host = collections.ardev.mycompany.com\n keepweb-port = 443\n host = api.ardev.mycompany.com\n port = 443\n token = mytoken\n protocol = https\n host-insecure = false\n }\n integration-tests {\n project-uuid = ardev-j7d0g-aa123f81q6y7skk\n }\n}" }, { "path": "contrib/java-sdk-v2/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker", "content": "mock-maker-inline" }, { "path": "contrib/java-sdk-v2/src/test/resources/org/arvados/client/api/client/collections-create-manifest.json", "content": "{\n \"kind\": \"arvados#collection\",\n \"etag\": \"bqoujj7oybdx0jybwvtsebj7y\",\n \"uuid\": \"112ci-4zz18-12tncxzptzbec1p\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-21T13:38:56.521853000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-21T13:38:56.521853000Z\",\n \"name\": \"Super Collection\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"d41d8cd98f00b204e9800998ecf8427e+0\",\n \"manifest_text\": \". 7df44272090cee6c0732382bba415ee9+70+Aa5ece4560e3329315165b36c239b8ab79c888f8a@5a1d5708 0:70:README.md\\n\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n}" }, { "path": "contrib/java-sdk-v2/src/test/resources/org/arvados/client/api/client/collections-create-simple.json", "content": "{\n \"kind\": \"arvados#collection\",\n \"etag\": \"bqoujj7oybdx0jybwvtsebj7y\",\n \"uuid\": \"112ci-4zz18-12tncxzptzbec1p\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-21T13:38:56.521853000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-21T13:38:56.521853000Z\",\n \"name\": \"Super Collection\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"d41d8cd98f00b204e9800998ecf8427e+0\",\n \"manifest_text\": \"\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n}\n" }, { "path": "contrib/java-sdk-v2/src/test/resources/org/arvados/client/api/client/collections-download-file.json", "content": "{\n \"kind\": \"arvados#collection\",\n \"etag\": \"2vm76dxmzr23u9774iguuxsrg\",\n \"uuid\": \"ardev-4zz18-jk5vo4uo9u5vj52\",\n \"owner_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"created_at\": \"2018-02-19T11:00:00.852389000Z\",\n \"modified_by_client_uuid\": \"ardev-ozdt8-97tzh5x96spqkay\",\n \"modified_by_user_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"modified_at\": \"2018-02-19T11:00:00.852389000Z\",\n \"name\": \"New Collection (2018-02-19 12:00:00.273)\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"49581091dfad651945c12b08d4735d88+112\",\n \"manifest_text\": \". 163679d58edaadc28db769011728a72c+1070080+A3acf8c1fe582c265d2077702e4a7d74fcc03aba8@5aa4fdeb 0:1024:test-file1 1024:20480:test-file2 21504:1048576:test-file\\\\0403\\n\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n}" }, { "path": "contrib/java-sdk-v2/src/test/resources/org/arvados/client/api/client/collections-get.json", "content": "{\n \"kind\": \"arvados#collection\",\n \"etag\": \"52tk5yg024cwhkkcidu3zcmj2\",\n \"uuid\": \"112ci-4zz18-p51w7z3fpopo6sm\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-15T10:36:03.554356000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-15T10:36:03.554356000Z\",\n \"name\": \"Collection With Manifest #2\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"6c4106229b08fe25f48b3a7a8289dd46+143\",\n \"manifest_text\": \". 66c9daa69630e092e9ce554b7aae8a20+524288+A4a15ffea58f259e09f68d3f7eea29942750a79d0@5a269ff6 435f38dd384b06c248feabee0cabca52+524288+A8a99e8148bd368c49901526098901bb7d7890c3b@5a269ff6 dc5b6c104aab35fff6d70a4dadc28d37+391727+Ab0662d549c422c983fccaad02b4ade7b48a8255b@5a269ff6 0:1440303:lombok.jar\\n\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n}" }, { "path": "contrib/java-sdk-v2/src/test/resources/org/arvados/client/api/client/collections-list.json", "content": "{\n \"kind\": \"arvados#collectionList\",\n \"etag\": \"\",\n \"offset\": 0,\n \"limit\": 100,\n \"items\": [\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"8xyiwnih5b5vzmj5sa33348a7\",\n \"uuid\": \"112ci-4zz18-x6xfmvz0chnkzgv\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-15T13:06:36.934337000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-15T13:06:36.934337000Z\",\n \"name\": \"Collection With Manifest #3\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"6c4106229b08fe25f48b3a7a8289dd46+143\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"8cmhep8aixe4p42pxjoct5502\",\n \"uuid\": \"112ci-4zz18-p51w7z3fpopo6sm\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-15T10:36:03.554356000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-15T10:36:03.554356000Z\",\n \"name\": \"Collection With Manifest #2\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"6c4106229b08fe25f48b3a7a8289dd46+143\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"de2ol2dyvsba3mn46al760cyg\",\n \"uuid\": \"112ci-4zz18-xb6gf2yraln7cwa\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-15T09:32:44.146172000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-15T09:32:44.146172000Z\",\n \"name\": \"New collection\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"d41d8cd98f00b204e9800998ecf8427e+0\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"dby68gd0vatvi090cu0axvtq3\",\n \"uuid\": \"112ci-4zz18-r5jfktpn3a9o0ap\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-14T13:00:35.431046000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-14T13:00:35.431046000Z\",\n \"name\": \"Collection With Manifest #1\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"3c59518bf8e1100d420488d822682b4a+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"2b34uzau862w862a2rv36agv6\",\n \"uuid\": \"112ci-4zz18-nqxk8xjn6mtskzt\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-14T12:59:34.767068000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-14T12:59:34.767068000Z\",\n \"name\": \"Empty Collection #2\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"d41d8cd98f00b204e9800998ecf8427e+0\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"60aywazztwfspnasltufcjxpa\",\n \"uuid\": \"112ci-4zz18-rs9bcf5qnyfjrkm\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-14T12:52:33.124452000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-14T12:52:33.124452000Z\",\n \"name\": \"Empty Collection #1\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"d41d8cd98f00b204e9800998ecf8427e+0\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"1jward6snif3tsjzftxh8hvwh\",\n \"uuid\": \"112ci-4zz18-af656lee4kv7q2m\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-14T12:09:05.319319000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-14T12:09:05.319319000Z\",\n \"name\": \"create example\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"d41d8cd98f00b204e9800998ecf8427e+0\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"zs2n4zliu6nb5yk3rw6h5ugw\",\n \"uuid\": \"112ci-4zz18-y2zqix7k9an7nro\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-13T16:59:02.299257000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-13T16:59:02.299257000Z\",\n \"name\": \"Saved at 2017-11-13 16:59:01 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"5ba3fc508718fabfa20d24390fe31856+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"eijhemzgy44ofmu0dtrowl604\",\n \"uuid\": \"112ci-4zz18-wq77jfi62u5i4rv\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-13T16:58:10.637548000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-13T16:58:10.637548000Z\",\n \"name\": \"Saved at 2017-11-13 16:58:07 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"5ba3fc508718fabfa20d24390fe31856+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"1oq7ye0gfbf3ih6y864w3n683\",\n \"uuid\": \"112ci-4zz18-unaeckkjgeg7ui0\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-10T09:43:07.583862000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-10T09:43:07.583862000Z\",\n \"name\": \"Saved at 2017-11-10 09:43:03 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"5ba3fc508718fabfa20d24390fe31856+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"4qmqlro878yx8q7ikhilo8qwn\",\n \"uuid\": \"112ci-4zz18-5y6atonkxq55lms\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T12:46:15.245770000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T12:46:15.245770000Z\",\n \"name\": \"Saved at 2017-11-09 12:46:13 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"5ba3fc508718fabfa20d24390fe31856+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"91v698hngoz241c38bbmh0ogc\",\n \"uuid\": \"112ci-4zz18-b3fjqd01pxjvseo\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T11:54:07.259998000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T11:54:07.259998000Z\",\n \"name\": \"Saved at 2017-11-09 11:54:04 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"5ba3fc508718fabfa20d24390fe31856+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"215t842ckrrgjpxrxr4j0gsui\",\n \"uuid\": \"112ci-4zz18-cwfxl8h41q18n65\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T11:49:38.276888000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T11:49:38.276888000Z\",\n \"name\": \"Saved at 2017-11-09 11:49:35 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"5ba3fc508718fabfa20d24390fe31856+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"90z6i3oqv197osng3wvjjir3t\",\n \"uuid\": \"112ci-4zz18-uv4xu08739tn1vy\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T11:43:05.917513000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T11:43:05.917513000Z\",\n \"name\": \"Saved at 2017-11-09 11:43:05 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"5ba3fc508718fabfa20d24390fe31856+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"5lcf6wvc3wypwobswdz22wen\",\n \"uuid\": \"112ci-4zz18-pzisn8c5mefzczv\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T11:40:38.804718000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T11:40:38.804718000Z\",\n \"name\": \"Saved at 2017-11-09 11:40:36 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"5ba3fc508718fabfa20d24390fe31856+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"98s08xew49avui1gy3mzit8je\",\n \"uuid\": \"112ci-4zz18-mj24uwtnqqrno27\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T11:40:25.189869000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T11:40:25.189869000Z\",\n \"name\": \"Saved at 2017-11-09 11:40:24 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"5ba3fc508718fabfa20d24390fe31856+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"a09wnvl4i51xqx7u9yf4qbi94\",\n \"uuid\": \"112ci-4zz18-oco162516upgqng\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T11:39:04.148785000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T11:39:04.148785000Z\",\n \"name\": \"Saved at 2017-11-09 11:39:03 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"5ba3fc508718fabfa20d24390fe31856+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"4ee2xudbc5rkr597drgu9tg10\",\n \"uuid\": \"112ci-4zz18-tlze7dgczsdwkep\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T11:37:59.478975000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T11:37:59.478975000Z\",\n \"name\": \"Saved at 2017-11-09 11:37:58 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"5ba3fc508718fabfa20d24390fe31856+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"5aa3evnbceo3brnps2e1sq8ts\",\n \"uuid\": \"112ci-4zz18-nq0kxi9d7w64la1\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T11:32:23.329259000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T11:32:23.329259000Z\",\n \"name\": \"Saved at 2017-11-09 11:32:22 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"5ba3fc508718fabfa20d24390fe31856+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"97vicgogv8bovmk4s2jymsdq\",\n \"uuid\": \"112ci-4zz18-fks9mewtw155pvx\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T11:30:17.589462000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T11:30:17.589462000Z\",\n \"name\": \"Saved at 2017-11-09 11:30:17 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"5ba3fc508718fabfa20d24390fe31856+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"btktwjclv063s1rd6duvk51v3\",\n \"uuid\": \"112ci-4zz18-kp356e0q2wdl2df\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T11:29:26.820481000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T11:29:26.820481000Z\",\n \"name\": \"Saved at 2017-11-09 11:29:25 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"5ba3fc508718fabfa20d24390fe31856+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"bob83na42pufqli1a5buxryvm\",\n \"uuid\": \"112ci-4zz18-0ey8ob38xf7surq\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T11:08:53.781498000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T11:08:53.781498000Z\",\n \"name\": \"Saved at 2017-11-09 11:08:52 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"5ba3fc508718fabfa20d24390fe31856+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"7pl1x327eeutqtsjppdj284g8\",\n \"uuid\": \"112ci-4zz18-wu2n0fv3cewna1n\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T11:08:33.423284000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T11:08:33.423284000Z\",\n \"name\": \"Saved at 2017-11-09 11:08:33 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"5ba3fc508718fabfa20d24390fe31856+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"2wg1wn2o18ubrgbhbqwwsslhf\",\n \"uuid\": \"112ci-4zz18-hyybo6yuvkx4hrm\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T10:44:53.096798000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T10:44:53.096798000Z\",\n \"name\": \"Saved at 2017-11-09 10:44:51 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"5ba3fc508718fabfa20d24390fe31856+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"8jk0at4e69cwjyjamvm4wz2oj\",\n \"uuid\": \"112ci-4zz18-h3gjq7gzd4syanw\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T10:41:31.278281000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T10:41:31.278281000Z\",\n \"name\": \"Saved at 2017-11-09 10:41:30 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"5ba3fc508718fabfa20d24390fe31856+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"be57zhzufz2hp1tbdwidoro5j\",\n \"uuid\": \"112ci-4zz18-jinwyyaeigjs1yg\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T10:41:07.083017000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T10:41:07.083017000Z\",\n \"name\": \"Saved at 2017-11-09 10:41:06 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"5ba3fc508718fabfa20d24390fe31856+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"29lj2roie4cygo5ffgrduflly\",\n \"uuid\": \"112ci-4zz18-etf8aghyxlfxvo1\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T10:40:31.710865000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T10:40:31.710865000Z\",\n \"name\": \"Saved at 2017-11-09 10:40:31 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"5ba3fc508718fabfa20d24390fe31856+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"6div78e1nhusii4x1xkp3rg2v\",\n \"uuid\": \"112ci-4zz18-jtbn4edpkkhbm9b\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T10:39:36.999602000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T10:39:36.999602000Z\",\n \"name\": \"Saved at 2017-11-09 10:39:36 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"5ba3fc508718fabfa20d24390fe31856+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"12wlbsxlmy3sze4v2m0ua7ake\",\n \"uuid\": \"112ci-4zz18-whdleimp34hiqp6\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T10:19:52.879907000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T10:19:52.879907000Z\",\n \"name\": \"Saved at 2017-11-09 10:19:52 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"5ba3fc508718fabfa20d24390fe31856+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"9bv1bw9afb3w84gu55uzcgd6h\",\n \"uuid\": \"112ci-4zz18-kj8dz72zpo5kbtm\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T10:16:31.558621000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T10:16:31.558621000Z\",\n \"name\": \"Saved at 2017-11-09 10:16:30 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"5ba3fc508718fabfa20d24390fe31856+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"683d77tvlhe97etk9bk2bx8ds\",\n \"uuid\": \"112ci-4zz18-tr306nau9hrr437\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T09:59:44.978811000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T09:59:44.978811000Z\",\n \"name\": \"Saved at 2017-11-09 09:59:44 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"67cbebb9f739b6b06ca056d21115cf43+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"1m34v9jbna2v7gv7auio54i8w\",\n \"uuid\": \"112ci-4zz18-oxuk69569mxztp0\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T09:59:30.774888000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T09:59:30.774888000Z\",\n \"name\": \"Saved at 2017-11-09 09:59:30 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"67cbebb9f739b6b06ca056d21115cf43+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"7l2a9fhqmxg7ghn7osx0s19v4\",\n \"uuid\": \"112ci-4zz18-wf8sl6xbyfwjyer\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T09:58:21.496088000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T09:58:21.496088000Z\",\n \"name\": \"Saved at 2017-11-09 09:58:20 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"67cbebb9f739b6b06ca056d21115cf43+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"33dw426fhs2vlb50b6301ukn0\",\n \"uuid\": \"112ci-4zz18-drpia2es1hp9ydi\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T09:56:08.506505000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T09:56:08.506505000Z\",\n \"name\": \"Saved at 2017-11-09 09:56:08 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"67cbebb9f739b6b06ca056d21115cf43+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"2437tnhn2gmti52lpm8nfq9ct\",\n \"uuid\": \"112ci-4zz18-5b4px2i2dwyidfi\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T09:54:06.651026000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T09:54:06.651026000Z\",\n \"name\": \"Saved at 2017-11-09 09:54:06 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"67cbebb9f739b6b06ca056d21115cf43+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"7e0k48zu93o57zudxjp1yrgjq\",\n \"uuid\": \"112ci-4zz18-94oslnwnxe1f9wp\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T09:40:04.240297000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T09:40:04.240297000Z\",\n \"name\": \"Saved at 2017-11-09 09:39:58 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"67cbebb9f739b6b06ca056d21115cf43+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"cuirr803f54e89reakuq50oaq\",\n \"uuid\": \"112ci-4zz18-2fk0d5d4jjc1fmq\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T09:36:14.952671000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T09:36:14.952671000Z\",\n \"name\": \"Saved at 2017-11-09 09:36:08 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"67cbebb9f739b6b06ca056d21115cf43+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"3bi5xd8ezxrazk5266cwzn4s4\",\n \"uuid\": \"112ci-4zz18-xp9pu81xyc5h422\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T09:35:29.552746000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T09:35:29.552746000Z\",\n \"name\": \"Saved at 2017-11-09 09:35:29 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"67cbebb9f739b6b06ca056d21115cf43+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"59uaoxy6uh82i6lrvr3ht8gz1\",\n \"uuid\": \"112ci-4zz18-znb4lo0if2as58c\",\n \"owner_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"created_at\": \"2017-11-09T09:31:08.109971000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-09T09:31:08.109971000Z\",\n \"name\": \"Saved at 2017-11-09 09:31:06 UTC by VirtualBox\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"67cbebb9f739b6b06ca056d21115cf43+53\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"dksrh8jznxoaidl29i1vv5904\",\n \"uuid\": \"112ci-4zz18-6pvl5ea5u932qzi\",\n \"owner_uuid\": \"112ci-j7d0g-tw71k7mxii6fqgx\",\n \"created_at\": \"2017-11-08T12:48:32.238698000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-f4633qdjs6w8zcy\",\n \"modified_by_user_uuid\": \"112ci-tpzed-nd84czdo4iea1mz\",\n \"modified_at\": \"2017-11-08T12:50:23.946608000Z\",\n \"name\": \"New collection\",\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"18c037c51c3f74be53ea2b115afd0c5f+69\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#collection\",\n \"etag\": \"1w1rhhd6oql4ceb7h9t16sf0q\",\n \"uuid\": \"112ci-4zz18-wq5pyrxfv1t9isu\",\n \"owner_uuid\": \"112ci-j7d0g-anonymouspublic\",\n \"created_at\": \"2017-11-03T10:03:20.364737000Z\",\n \"modified_by_client_uuid\": null,\n \"modified_by_user_uuid\": \"112ci-tpzed-000000000000000\",\n \"modified_at\": \"2017-11-03T10:03:20.364737000Z\",\n \"name\": null,\n \"description\": null,\n \"properties\": {},\n \"portable_data_hash\": \"d41d8cd98f00b204e9800998ecf8427e+0\",\n \"replication_desired\": null,\n \"replication_confirmed\": null,\n \"replication_confirmed_at\": null,\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n }\n ],\n \"items_available\": 41\n}" }, { "path": "contrib/java-sdk-v2/src/test/resources/org/arvados/client/api/client/groups-get.json", "content": "{\n \"kind\": \"arvados#group\",\n \"etag\": \"3hw0vk4mbl0ofvia5k6x4dwrx\",\n \"uuid\": \"ardev-j7d0g-bmg3pfqtx3ivczp\",\n \"owner_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"created_at\": \"2018-03-29T11:09:05.984597000Z\",\n \"modified_by_client_uuid\": \"ardev-ozdt8-97tzh5x96spqkay\",\n \"modified_by_user_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"modified_at\": \"2018-03-29T11:09:05.984597000Z\",\n \"name\": \"TestGroup1\",\n \"group_class\": \"project\",\n \"description\": null,\n \"writable_by\": [\n \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"ardev-tpzed-n3kzq4fvoks3uw4\"\n ],\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n}" }, { "path": "contrib/java-sdk-v2/src/test/resources/org/arvados/client/api/client/groups-list.json", "content": "{\n \"kind\": \"arvados#groupList\",\n \"etag\": \"\",\n \"offset\": 0,\n \"limit\": 100,\n \"items\": [\n {\n \"kind\": \"arvados#group\",\n \"etag\": \"68vubv3iw7663763bozxebmyf\",\n \"uuid\": \"ardev-j7d0g-ylx7wnu1moge2di\",\n \"owner_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"created_at\": \"2018-04-18T09:09:21.126649000Z\",\n \"modified_by_client_uuid\": \"ardev-ozdt8-97tzh5x96spqkay\",\n \"modified_by_user_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"modified_at\": \"2018-04-18T09:09:21.126649000Z\",\n \"name\": \"TestProject1\",\n \"group_class\": \"project\",\n \"description\": null,\n \"writable_by\": [\n \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"ardev-tpzed-n3kzq4fvoks3uw4\"\n ],\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#group\",\n \"etag\": \"68q7r8r37u9hckr2zsynvton3\",\n \"uuid\": \"ardev-j7d0g-mnzhga726itrbrq\",\n \"owner_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"created_at\": \"2018-04-17T12:11:24.389594000Z\",\n \"modified_by_client_uuid\": \"ardev-ozdt8-97tzh5x96spqkay\",\n \"modified_by_user_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"modified_at\": \"2018-04-17T12:11:24.389594000Z\",\n \"name\": \"TestProject2\",\n \"group_class\": \"project\",\n \"description\": null,\n \"writable_by\": [\n \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"ardev-tpzed-n3kzq4fvoks3uw4\"\n ],\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#group\",\n \"etag\": \"ef4vzx5gyudkrg9zml0zdv6qu\",\n \"uuid\": \"ardev-j7d0g-0w9m1sz46ljtdnm\",\n \"owner_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"created_at\": \"2018-04-17T12:08:39.066802000Z\",\n \"modified_by_client_uuid\": \"ardev-ozdt8-97tzh5x96spqkay\",\n \"modified_by_user_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"modified_at\": \"2018-04-17T12:08:39.066802000Z\",\n \"name\": \"TestProject3\",\n \"group_class\": \"project\",\n \"description\": null,\n \"writable_by\": [\n \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"ardev-tpzed-n3kzq4fvoks3uw4\"\n ],\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#group\",\n \"etag\": \"6h6h4ta6yyf9058delxk8fnqs\",\n \"uuid\": \"ardev-j7d0g-r20iem5ou6h5wao\",\n \"owner_uuid\": \"ardev-j7d0g-j7drd8yikkp6evd\",\n \"created_at\": \"2018-04-17T12:03:39.647244000Z\",\n \"modified_by_client_uuid\": \"ardev-ozdt8-97tzh5x96spqkay\",\n \"modified_by_user_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"modified_at\": \"2018-04-17T12:03:39.647244000Z\",\n \"name\": \"TestProject4\",\n \"group_class\": \"project\",\n \"description\": null,\n \"writable_by\": [\n \"ardev-j7d0g-j7drd8yikkp6evd\",\n \"ardev-tpzed-n3kzq4fvoks3uw4\"\n ],\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#group\",\n \"etag\": \"6se2y8f9o7uu06pbopgq56xds\",\n \"uuid\": \"ardev-j7d0g-j7drd8yikkp6evd\",\n \"owner_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"created_at\": \"2018-04-17T11:58:31.339515000Z\",\n \"modified_by_client_uuid\": \"ardev-ozdt8-97tzh5x96spqkay\",\n \"modified_by_user_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"modified_at\": \"2018-04-17T11:58:31.339515000Z\",\n \"name\": \"TestProject5\",\n \"group_class\": \"project\",\n \"description\": null,\n \"writable_by\": [\n \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"ardev-tpzed-n3kzq4fvoks3uw4\"\n ],\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#group\",\n \"etag\": \"2si26vaig3vig9266pqkqh2gy\",\n \"uuid\": \"ardev-j7d0g-kh1g7i5va870xt0\",\n \"owner_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"created_at\": \"2018-04-17T10:56:54.391676000Z\",\n \"modified_by_client_uuid\": \"ardev-ozdt8-97tzh5x96spqkay\",\n \"modified_by_user_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"modified_at\": \"2018-04-17T10:56:54.391676000Z\",\n \"name\": \"TestProject6\",\n \"group_class\": \"project\",\n \"description\": null,\n \"writable_by\": [\n \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"ardev-tpzed-n3kzq4fvoks3uw4\"\n ],\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#group\",\n \"etag\": \"edgnz6q0vt2u3o13ujtfohb75\",\n \"uuid\": \"ardev-j7d0g-sclkdyuwm4h2m78\",\n \"owner_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"created_at\": \"2018-04-17T10:27:15.914517000Z\",\n \"modified_by_client_uuid\": \"ardev-ozdt8-97tzh5x96spqkay\",\n \"modified_by_user_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"modified_at\": \"2018-04-17T10:27:15.914517000Z\",\n \"name\": \"TestProject7\",\n \"group_class\": \"project\",\n \"description\": null,\n \"writable_by\": [\n \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"ardev-tpzed-n3kzq4fvoks3uw4\"\n ],\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#group\",\n \"etag\": \"39ig9ttgec6lbe096uetn2cb9\",\n \"uuid\": \"ardev-j7d0g-593khc577zuyyhe\",\n \"owner_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"created_at\": \"2018-04-17T10:27:03.858203000Z\",\n \"modified_by_client_uuid\": \"ardev-ozdt8-97tzh5x96spqkay\",\n \"modified_by_user_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"modified_at\": \"2018-04-17T10:27:03.858203000Z\",\n \"name\": \"TestProject8\",\n \"group_class\": \"project\",\n \"description\": null,\n \"writable_by\": [\n \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"ardev-tpzed-n3kzq4fvoks3uw4\"\n ],\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#group\",\n \"etag\": \"1dpr8v6tx6pta0fozq93eyeou\",\n \"uuid\": \"ardev-j7d0g-iotds0tm559dbz7\",\n \"owner_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"created_at\": \"2018-04-17T10:26:25.180623000Z\",\n \"modified_by_client_uuid\": \"ardev-ozdt8-97tzh5x96spqkay\",\n \"modified_by_user_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"modified_at\": \"2018-04-17T10:26:25.180623000Z\",\n \"name\": \"TestProject9\",\n \"group_class\": \"project\",\n \"description\": null,\n \"writable_by\": [\n \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"ardev-tpzed-n3kzq4fvoks3uw4\"\n ],\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#group\",\n \"etag\": \"dizbavs2opfe1wpx6thocfki0\",\n \"uuid\": \"ardev-j7d0g-gbqay74778tonb8\",\n \"owner_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"created_at\": \"2018-04-17T10:26:06.435961000Z\",\n \"modified_by_client_uuid\": \"ardev-ozdt8-97tzh5x96spqkay\",\n \"modified_by_user_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"modified_at\": \"2018-04-17T10:26:06.435961000Z\",\n \"name\": \"TestProject10\",\n \"group_class\": \"project\",\n \"description\": null,\n \"writable_by\": [\n \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"ardev-tpzed-n3kzq4fvoks3uw4\"\n ],\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#group\",\n \"etag\": \"6xue8m3lx9qpptfvdf13val5t\",\n \"uuid\": \"ardev-j7d0g-fmq1t0jlznehbdm\",\n \"owner_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"created_at\": \"2018-04-17T10:25:55.546399000Z\",\n \"modified_by_client_uuid\": \"ardev-ozdt8-97tzh5x96spqkay\",\n \"modified_by_user_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"modified_at\": \"2018-04-17T10:25:55.546399000Z\",\n \"name\": \"TestProject11\",\n \"group_class\": \"project\",\n \"description\": null,\n \"writable_by\": [\n \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"ardev-tpzed-n3kzq4fvoks3uw4\"\n ],\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#group\",\n \"etag\": \"2gqix9e4m023usi9exhrsjx6z\",\n \"uuid\": \"ardev-j7d0g-vxju56ch64u51gq\",\n \"owner_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"created_at\": \"2018-04-16T14:09:49.700566000Z\",\n \"modified_by_client_uuid\": \"ardev-ozdt8-97tzh5x96spqkay\",\n \"modified_by_user_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"modified_at\": \"2018-04-16T14:09:49.700566000Z\",\n \"name\": \"TestProject12\",\n \"group_class\": \"project\",\n \"description\": null,\n \"writable_by\": [\n \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"ardev-tpzed-n3kzq4fvoks3uw4\"\n ],\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#group\",\n \"etag\": \"73n8x82814o6ihld0kltf468d\",\n \"uuid\": \"ardev-j7d0g-g8m4w0d22gv6fbj\",\n \"owner_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"created_at\": \"2018-04-11T15:02:35.016850000Z\",\n \"modified_by_client_uuid\": \"ardev-ozdt8-97tzh5x96spqkay\",\n \"modified_by_user_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"modified_at\": \"2018-04-11T15:02:35.016850000Z\",\n \"name\": \"TestProject13\",\n \"group_class\": \"project\",\n \"description\": null,\n \"writable_by\": [\n \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"ardev-tpzed-n3kzq4fvoks3uw4\"\n ],\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#group\",\n \"etag\": \"91f7uwq7pj3d3ez1u4smjg3ch\",\n \"uuid\": \"ardev-j7d0g-lstqed4y78khaqm\",\n \"owner_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"created_at\": \"2018-04-06T15:29:27.754408000Z\",\n \"modified_by_client_uuid\": \"ardev-ozdt8-97tzh5x96spqkay\",\n \"modified_by_user_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"modified_at\": \"2018-04-06T15:29:27.754408000Z\",\n \"name\": \"TestProject14\",\n \"group_class\": \"project\",\n \"description\": null,\n \"writable_by\": [\n \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"ardev-tpzed-n3kzq4fvoks3uw4\"\n ],\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#group\",\n \"etag\": \"7dbxhvbcfaogwnvo8k4mtqthk\",\n \"uuid\": \"ardev-j7d0g-0jbezvnq8i07l7p\",\n \"owner_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"created_at\": \"2018-04-05T09:32:46.946417000Z\",\n \"modified_by_client_uuid\": \"ardev-ozdt8-97tzh5x96spqkay\",\n \"modified_by_user_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"modified_at\": \"2018-04-05T09:32:46.946417000Z\",\n \"name\": \"TestProject15\",\n \"group_class\": \"project\",\n \"description\": null,\n \"writable_by\": [\n \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"ardev-tpzed-n3kzq4fvoks3uw4\"\n ],\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#group\",\n \"etag\": \"dhfu203rckzdzvx832wm7jv59\",\n \"uuid\": \"ardev-j7d0g-72dxer22g6iltqz\",\n \"owner_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"created_at\": \"2018-03-29T11:27:02.482218000Z\",\n \"modified_by_client_uuid\": \"ardev-ozdt8-97tzh5x96spqkay\",\n \"modified_by_user_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"modified_at\": \"2018-03-29T13:17:00.045606000Z\",\n \"name\": \"TestProject16\",\n \"group_class\": \"project\",\n \"description\": null,\n \"writable_by\": [\n \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"ardev-tpzed-n3kzq4fvoks3uw4\"\n ],\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#group\",\n \"etag\": \"7l9oxbdf4e1m9ddnujokf7czz\",\n \"uuid\": \"ardev-j7d0g-nebzwquxtq1v3o5\",\n \"owner_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"created_at\": \"2018-03-29T11:11:26.235411000Z\",\n \"modified_by_client_uuid\": \"ardev-ozdt8-97tzh5x96spqkay\",\n \"modified_by_user_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"modified_at\": \"2018-03-29T11:11:26.235411000Z\",\n \"name\": \"TestProject17\",\n \"group_class\": \"project\",\n \"description\": null,\n \"writable_by\": [\n \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"ardev-tpzed-n3kzq4fvoks3uw4\"\n ],\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#group\",\n \"etag\": \"83862x2o4453mja2rvypjl5gv\",\n \"uuid\": \"ardev-j7d0g-5589c8dmxevecqh\",\n \"owner_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"created_at\": \"2018-03-29T11:10:58.496482000Z\",\n \"modified_by_client_uuid\": \"ardev-ozdt8-97tzh5x96spqkay\",\n \"modified_by_user_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"modified_at\": \"2018-03-29T11:10:58.496482000Z\",\n \"name\": \"TestProject18\",\n \"group_class\": \"project\",\n \"description\": null,\n \"writable_by\": [\n \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"ardev-tpzed-n3kzq4fvoks3uw4\"\n ],\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#group\",\n \"etag\": \"3hw0vk4mbl0ofvia5k6x4dwrx\",\n \"uuid\": \"ardev-j7d0g-bmg3pfqtx3ivczp\",\n \"owner_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"created_at\": \"2018-03-29T11:09:05.984597000Z\",\n \"modified_by_client_uuid\": \"ardev-ozdt8-97tzh5x96spqkay\",\n \"modified_by_user_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"modified_at\": \"2018-03-29T11:09:05.984597000Z\",\n \"name\": \"TestProject19\",\n \"group_class\": \"project\",\n \"description\": null,\n \"writable_by\": [\n \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"ardev-tpzed-n3kzq4fvoks3uw4\"\n ],\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n },\n {\n \"kind\": \"arvados#group\",\n \"etag\": \"6p9xbxpttj782mpqs537gfvc6\",\n \"uuid\": \"ardev-j7d0g-mfitz2oa4rpycou\",\n \"owner_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"created_at\": \"2018-03-29T11:00:19.809612000Z\",\n \"modified_by_client_uuid\": \"ardev-ozdt8-97tzh5x96spqkay\",\n \"modified_by_user_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"modified_at\": \"2018-03-29T11:00:19.809612000Z\",\n \"name\": \"TestProject20\",\n \"group_class\": \"project\",\n \"description\": null,\n \"writable_by\": [\n \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"ardev-tpzed-n3kzq4fvoks3uw4\"\n ],\n \"delete_at\": null,\n \"trash_at\": null,\n \"is_trashed\": false\n }\n ],\n \"items_available\": 20\n}" }, { "path": "contrib/java-sdk-v2/src/test/resources/org/arvados/client/api/client/keep-client-test-file.txt", "content": "Sample text file to test keep client." }, { "path": "contrib/java-sdk-v2/src/test/resources/org/arvados/client/api/client/keep-client-upload-response.json", "content": "Created" }, { "path": "contrib/java-sdk-v2/src/test/resources/org/arvados/client/api/client/keep-services-accessible-disk-only.json", "content": "{\n \"kind\": \"arvados#keepServiceList\",\n \"etag\": \"\",\n \"offset\": null,\n \"limit\": null,\n \"items\": [\n {\n \"kind\": \"arvados#keepService\",\n \"etag\": \"bjzh7og2d9z949lbd38vnnslt\",\n \"uuid\": \"112ci-bi6l4-hv02fg8sbti8ykk\",\n \"owner_uuid\": \"112ci-tpzed-000000000000000\",\n \"created_at\": \"2017-11-03T10:04:48.314229000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-xxy0ipzwti8gnmt\",\n \"modified_by_user_uuid\": \"112ci-tpzed-000000000000000\",\n \"modified_at\": \"2017-11-03T10:04:48.314229000Z\",\n \"service_host\": \"localhost\",\n \"service_port\": 9000,\n \"service_ssl_flag\": false,\n \"service_type\": \"disk\",\n \"read_only\": false\n },\n {\n \"kind\": \"arvados#keepService\",\n \"etag\": \"7m64l69kko4bytpsykf8cay7t\",\n \"uuid\": \"112ci-bi6l4-f0r03wrqymotwql\",\n \"owner_uuid\": \"112ci-tpzed-000000000000000\",\n \"created_at\": \"2017-11-03T10:04:48.351577000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-xxy0ipzwti8gnmt\",\n \"modified_by_user_uuid\": \"112ci-tpzed-000000000000000\",\n \"modified_at\": \"2017-11-03T10:04:48.351577000Z\",\n \"service_host\": \"localhost\",\n \"service_port\": 9001,\n \"service_ssl_flag\": false,\n \"service_type\": \"disk\",\n \"read_only\": false\n }\n ],\n \"items_available\": 2\n}" }, { "path": "contrib/java-sdk-v2/src/test/resources/org/arvados/client/api/client/keep-services-accessible.json", "content": "{\n \"kind\": \"arvados#keepServiceList\",\n \"etag\": \"\",\n \"offset\": null,\n \"limit\": null,\n \"items\": [\n {\n \"kind\": \"arvados#keepService\",\n \"etag\": \"bjzh7og2d9z949lbd38vnnslt\",\n \"uuid\": \"112ci-bi6l4-hv02fg8sbti8ykk\",\n \"owner_uuid\": \"112ci-tpzed-000000000000000\",\n \"created_at\": \"2017-11-03T10:04:48.314229000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-xxy0ipzwti8gnmt\",\n \"modified_by_user_uuid\": \"112ci-tpzed-000000000000000\",\n \"modified_at\": \"2017-11-03T10:04:48.314229000Z\",\n \"service_host\": \"localhost\",\n \"service_port\": 9000,\n \"service_ssl_flag\": false,\n \"service_type\": \"disk\",\n \"read_only\": false\n },\n {\n \"kind\": \"arvados#keepService\",\n \"etag\": \"7m64l69kko4bytpsykf8cay7t\",\n \"uuid\": \"112ci-bi6l4-f0r03wrqymotwql\",\n \"owner_uuid\": \"112ci-tpzed-000000000000000\",\n \"created_at\": \"2017-11-03T10:04:48.351577000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-xxy0ipzwti8gnmt\",\n \"modified_by_user_uuid\": \"112ci-tpzed-000000000000000\",\n \"modified_at\": \"2017-11-03T10:04:48.351577000Z\",\n \"service_host\": \"localhost\",\n \"service_port\": 9000,\n \"service_ssl_flag\": false,\n \"service_type\": \"gpfs\",\n \"read_only\": false\n }\n ],\n \"items_available\": 2\n}" }, { "path": "contrib/java-sdk-v2/src/test/resources/org/arvados/client/api/client/keep-services-get.json", "content": "{\n \"kind\": \"arvados#keepService\",\n \"etag\": \"bjzh7og2d9z949lbd38vnnslt\",\n \"uuid\": \"112ci-bi6l4-hv02fg8sbti8ykk\",\n \"owner_uuid\": \"112ci-tpzed-000000000000000\",\n \"created_at\": \"2017-11-03T10:04:48.314229000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-xxy0ipzwti8gnmt\",\n \"modified_by_user_uuid\": \"112ci-tpzed-000000000000000\",\n \"modified_at\": \"2017-11-03T10:04:48.314229000Z\",\n \"service_host\": \"10.0.2.15\",\n \"service_port\": 9000,\n \"service_ssl_flag\": false,\n \"service_type\": \"disk\",\n \"read_only\": false\n}" }, { "path": "contrib/java-sdk-v2/src/test/resources/org/arvados/client/api/client/keep-services-list.json", "content": "{\n \"kind\": \"arvados#keepServiceList\",\n \"etag\": \"\",\n \"offset\": 0,\n \"limit\": 100,\n \"items\": [\n {\n \"kind\": \"arvados#keepService\",\n \"etag\": \"7m64l69kko4bytpsykf8cay7t\",\n \"uuid\": \"112ci-bi6l4-f0r03wrqymotwql\",\n \"owner_uuid\": \"112ci-tpzed-000000000000000\",\n \"created_at\": \"2017-11-03T10:04:48.351577000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-xxy0ipzwti8gnmt\",\n \"modified_by_user_uuid\": \"112ci-tpzed-000000000000000\",\n \"modified_at\": \"2017-11-03T10:04:48.351577000Z\",\n \"service_host\": \"10.0.2.15\",\n \"service_port\": 9000,\n \"service_ssl_flag\": false,\n \"service_type\": \"disk\",\n \"read_only\": false\n },\n {\n \"kind\": \"arvados#keepService\",\n \"etag\": \"bjzh7og2d9z949lbd38vnnslt\",\n \"uuid\": \"112ci-bi6l4-hv02fg8sbti8ykk\",\n \"owner_uuid\": \"112ci-tpzed-000000000000000\",\n \"created_at\": \"2017-11-03T10:04:48.314229000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-xxy0ipzwti8gnmt\",\n \"modified_by_user_uuid\": \"112ci-tpzed-000000000000000\",\n \"modified_at\": \"2017-11-03T10:04:48.314229000Z\",\n \"service_host\": \"10.0.2.15\",\n \"service_port\": 9001,\n \"service_ssl_flag\": false,\n \"service_type\": \"disk\",\n \"read_only\": false\n },\n {\n \"kind\": \"arvados#keepService\",\n \"etag\": \"4be61qkpt6nzdfff4vj9nkpmj\",\n \"uuid\": \"112ci-bi6l4-ko27cfbsf2ssx2m\",\n \"owner_uuid\": \"112ci-tpzed-000000000000000\",\n \"created_at\": \"2017-11-03T10:04:36.355045000Z\",\n \"modified_by_client_uuid\": \"112ci-ozdt8-xxy0ipzwti8gnmt\",\n \"modified_by_user_uuid\": \"112ci-tpzed-000000000000000\",\n \"modified_at\": \"2017-11-03T10:04:36.355045000Z\",\n \"service_host\": \"10.0.2.15\",\n \"service_port\": 9002,\n \"service_ssl_flag\": false,\n \"service_type\": \"proxy\",\n \"read_only\": false\n }\n ],\n \"items_available\": 3\n}" }, { "path": "contrib/java-sdk-v2/src/test/resources/org/arvados/client/api/client/keep-services-not-accessible.json", "content": "{\n \"kind\": \"arvados#keepServiceList\",\n \"etag\": \"\",\n \"offset\": null,\n \"limit\": null,\n \"items\": [],\n \"items_available\": 0\n}" }, { "path": "contrib/java-sdk-v2/src/test/resources/org/arvados/client/api/client/links-create.json", "content": "{\n \"kind\": \"arvados#link\",\n \"etag\": \"zw1rlnbig0kpm9btw8us3pn9\",\n \"uuid\": \"arkau-o0j2j-huxuaxbi46s1yml\",\n \"owner_uuid\": \"arkau-tpzed-000000000000000\",\n \"created_at\": \"2021-11-30T08:45:04.373354745Z\",\n \"modified_by_client_uuid\": null,\n \"modified_by_user_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"modified_at\": \"2021-11-30T08:45:04.374489000Z\",\n \"tail_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"link_class\": \"star\",\n \"name\": \"Star Link\",\n \"head_uuid\": \"arkau-j7d0g-fcedae2076pw56h\",\n \"head_kind\": \"arvados#group\",\n \"tail_kind\": \"arvados#user\",\n \"properties\": {}\n}" }, { "path": "contrib/java-sdk-v2/src/test/resources/org/arvados/client/api/client/links-get.json", "content": "{\n \"kind\": \"arvados#link\",\n \"etag\": \"zw1rlnbig0kpm9btw8us3pn9\",\n \"uuid\": \"arkau-o0j2j-huxuaxbi46s1yml\",\n \"owner_uuid\": \"arkau-tpzed-000000000000000\",\n \"created_at\": \"2021-11-30T08:45:04.373354745Z\",\n \"modified_by_client_uuid\": null,\n \"modified_by_user_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"modified_at\": \"2021-11-30T08:45:04.374489000Z\",\n \"tail_uuid\": \"ardev-tpzed-n3kzq4fvoks3uw4\",\n \"link_class\": \"permission\",\n \"name\": \"can_read\",\n \"head_uuid\": \"arkau-j7d0g-fcedae2076pw56h\",\n \"head_kind\": \"arvados#group\",\n \"tail_kind\": \"arvados#user\",\n \"properties\": {}\n}" }, { "path": "contrib/java-sdk-v2/src/test/resources/org/arvados/client/api/client/links-list.json", "content": "{\n \"kind\": \"arvados#linkList\",\n \"etag\": \"\",\n \"offset\": 0,\n \"limit\": 100,\n \"items\": [\n {\n \"kind\": \"arvados#link\",\n \"etag\": \"dkhtr9tvp9zfy0d90xjn7w1t7\",\n \"uuid\": \"arkau-o0j2j-x2b4rdadxs2fizn\",\n \"owner_uuid\": \"arkau-j7d0g-publicfavorites\",\n \"created_at\": \"2021-10-27T12:00:06.607794000Z\",\n \"modified_by_client_uuid\": null,\n \"modified_by_user_uuid\": \"arlog-tpzed-fyiau9qwo7ytntu\",\n \"modified_at\": \"2021-10-27T12:00:06.609840000Z\",\n \"tail_uuid\": \"arkau-j7d0g-publicfavorites\",\n \"link_class\": \"star\",\n \"name\": \"pRED Data Commons Service - Open access\",\n \"head_uuid\": \"arkau-j7d0g-sfhw8b1uson0hwh\",\n \"head_kind\": \"arvados#group\",\n \"tail_kind\": \"arvados#group\",\n \"properties\": {}\n },\n {\n \"kind\": \"arvados#link\",\n \"etag\": \"9nt0c2xn5oz1jzjzawlycmehz\",\n \"uuid\": \"arkau-o0j2j-r5am4lz9gnu488k\",\n \"owner_uuid\": \"arkau-j7d0g-publicfavorites\",\n \"created_at\": \"2021-06-23T14:58:06.189520000Z\",\n \"modified_by_client_uuid\": null,\n \"modified_by_user_uuid\": \"arlog-tpzed-xzjyeljl6co7vlz\",\n \"modified_at\": \"2021-06-23T14:58:06.196208000Z\",\n \"tail_uuid\": \"arkau-j7d0g-publicfavorites\",\n \"link_class\": \"star\",\n \"name\": \"Open Targets Genetics\",\n \"head_uuid\": \"arkau-j7d0g-pj5wysmpy5wn8yo\",\n \"head_kind\": \"arvados#group\",\n \"tail_kind\": \"arvados#group\",\n \"properties\": {}\n }\n ],\n \"items_available\": 2\n}" }, { "path": "contrib/java-sdk-v2/src/test/resources/org/arvados/client/api/client/users-create.json", "content": "{\n \"kind\": \"arvados#user\",\n \"etag\": \"b21emst9eu9u1wdpqcz6la583\",\n \"uuid\": \"ardev-tpzed-q6dvn7sby55up1b\",\n \"owner_uuid\": \"ardev-tpzed-000000000000000\",\n \"created_at\": \"2017-10-30T19:42:43.324740000Z\",\n \"modified_by_client_uuid\": \"ardev-ozdt8-97tzh5x96spqkay\",\n \"modified_by_user_uuid\": \"ardev-tpzed-o3km4ug9jhs189j\",\n \"modified_at\": \"2017-10-31T09:01:03.985749000Z\",\n \"email\": \"example@email.com\",\n \"username\": \"johnwayne\",\n \"full_name\": \"John Wayne\",\n \"first_name\": \"John\",\n \"last_name\": \"Wayne\",\n \"identity_url\": \"ardev-tpzed-r09t5ztf5qd3rlj\",\n \"is_active\": true,\n \"is_admin\": null,\n \"is_invited\": true,\n \"prefs\": {},\n \"writable_by\": [\n \"ardev-tpzed-000000000000000\",\n \"ardev-tpzed-q6dvn7sby55up1b\",\n \"ardev-j7d0g-000000000000000\"\n ]\n}" }, { "path": "contrib/java-sdk-v2/src/test/resources/org/arvados/client/api/client/users-get.json", "content": "{\n \"kind\": \"arvados#user\",\n \"etag\": \"b21emst9eu9u1wdpqcz6la583\",\n \"uuid\": \"ardev-tpzed-q6dvn7sby55up1b\",\n \"owner_uuid\": \"ardev-tpzed-000000000000000\",\n \"created_at\": \"2017-10-30T19:42:43.324740000Z\",\n \"modified_by_client_uuid\": \"ardev-ozdt8-97tzh5x96spqkay\",\n \"modified_by_user_uuid\": \"ardev-tpzed-o3km4ug9jhs189j\",\n \"modified_at\": \"2017-10-31T09:01:03.985749000Z\",\n \"email\": \"example@email.com\",\n \"username\": \"johnwayne\",\n \"full_name\": \"John Wayne\",\n \"first_name\": \"John\",\n \"last_name\": \"Wayne\",\n \"identity_url\": \"ardev-tpzed-r09t5ztf5qd3rlj\",\n \"is_active\": true,\n \"is_admin\": null,\n \"is_invited\": true,\n \"prefs\": {},\n \"writable_by\": [\n \"ardev-tpzed-000000000000000\",\n \"ardev-tpzed-q6dvn7sby55up1b\",\n \"ardev-j7d0g-000000000000000\"\n ]\n}" }, { "path": "contrib/java-sdk-v2/src/test/resources/org/arvados/client/api/client/users-list.json", "content": "{\n \"kind\": \"arvados#userList\",\n \"etag\": \"\",\n \"offset\": 0,\n \"limit\": 100,\n \"items\": [\n {\n \"kind\": \"arvados#user\",\n \"uuid\": \"ardev-tpzed-12389ux30402est\",\n \"email\": \"test.user@email.com\",\n \"first_name\": \"Test\",\n \"last_name\": \"User\",\n \"is_active\": true\n },\n {\n \"kind\": \"arvados#user\",\n \"uuid\": \"ardev-tpzed-123vn7sby55up1b\",\n \"email\": \"test.user1@email.com\",\n \"first_name\": \"Test1\",\n \"last_name\": \"User1\",\n \"is_active\": true\n },\n {\n \"kind\": \"arvados#user\",\n \"uuid\": \"ardev-tpzed-123g70lq1m3c6fz\",\n \"email\": \"test.user2@email.com\",\n \"first_name\": \"Test2\",\n \"last_name\": \"User2\",\n \"is_active\": true\n },\n {\n \"kind\": \"arvados#user\",\n \"uuid\": \"ardev-tpzed-1233zsoudkgq92e\",\n \"email\": \"test.user3@email.com\",\n \"first_name\": \"Test3\",\n \"last_name\": \"User3\",\n \"is_active\": true\n },\n {\n \"kind\": \"arvados#user\",\n \"uuid\": \"ardev-tpzed-1234xjvs0clppd3\",\n \"email\": \"test.user4@email.com\",\n \"first_name\": \"Test4\",\n \"last_name\": \"User4\",\n \"is_active\": true\n },\n {\n \"kind\": \"arvados#user\",\n \"uuid\": \"ardev-tpzed-123bpggscmn6z8m\",\n \"email\": \"test.user5@email.com\",\n \"first_name\": \"Test5\",\n \"last_name\": \"User5\",\n \"is_active\": true\n },\n {\n \"kind\": \"arvados#user\",\n \"uuid\": \"ardev-tpzed-1231uysivaz6ipi\",\n \"email\": \"test.user6@email.com\",\n \"first_name\": \"Test6\",\n \"last_name\": \"User6\",\n \"is_active\": true\n },\n {\n \"kind\": \"arvados#user\",\n \"uuid\": \"ardev-tpzed-123b0a1wu0q6cm4\",\n \"email\": \"test.user7@email.com\",\n \"first_name\": \"Test7\",\n \"last_name\": \"User7\",\n \"is_active\": true\n },\n {\n \"kind\": \"arvados#user\",\n \"uuid\": \"ardev-tpzed-123bz6n6si24t6v\",\n \"email\": \"test.user8@email.com\",\n \"first_name\": \"Test8\",\n \"last_name\": \"User8\",\n \"is_active\": true\n },\n {\n \"kind\": \"arvados#user\",\n \"uuid\": \"ardev-tpzed-123lxhzifligheu\",\n \"email\": \"test.user9@email.com\",\n \"first_name\": \"Test9\",\n \"last_name\": \"User9\",\n \"is_active\": true\n },\n {\n \"kind\": \"arvados#user\",\n \"uuid\": \"ardev-tpzed-123gaz31qbopewh\",\n \"email\": \"test.user10@email.com\",\n \"first_name\": \"Test10\",\n \"last_name\": \"User10\",\n \"is_active\": true\n },\n {\n \"kind\": \"arvados#user\",\n \"uuid\": \"ardev-tpzed-123dmcf65z973uo\",\n \"email\": \"test.user11@email.com\",\n \"first_name\": \"Test11\",\n \"last_name\": \"User11\",\n \"is_active\": true\n },\n {\n \"kind\": \"arvados#user\",\n \"uuid\": \"ardev-tpzed-1239y3lj7ybpyg8\",\n \"email\": \"test.user12@email.com\",\n \"first_name\": \"Test12\",\n \"last_name\": \"User12\",\n \"is_active\": true\n }\n\n ],\n \"items_available\": 13\n}" }, { "path": "contrib/java-sdk-v2/src/test/resources/org/arvados/client/api/client/users-system.json", "content": "{\n \"kind\": \"arvados#user\",\n \"etag\": \"2ehmra38iwfuexvz1cjno5xua\",\n \"uuid\": \"ardev-tpzed-000000000000000\",\n \"owner_uuid\": \"ardev-tpzed-000000000000000\",\n \"created_at\": \"2016-10-19T07:48:04.838534000Z\",\n \"modified_by_client_uuid\": null,\n \"modified_by_user_uuid\": \"ardev-tpzed-000000000000000\",\n \"modified_at\": \"2016-10-19T07:48:04.833164000Z\",\n \"email\": \"root\",\n \"username\": null,\n \"full_name\": \"root\",\n \"first_name\": \"root\",\n \"last_name\": \"\",\n \"identity_url\": null,\n \"is_active\": true,\n \"is_admin\": true,\n \"is_invited\": true,\n \"prefs\": {},\n \"writable_by\": [\n \"ardev-tpzed-000000000000000\"\n ]\n}" }, { "path": "doc/Gemfile", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: CC-BY-SA-3.0\n\nsource 'https://rubygems.org'\n\ngem 'zenweb'\ngem 'liquid', '~>4.0.0'\ngem 'RedCloth'\ngem 'colorize'\n" }, { "path": "doc/README.textile", "content": "###. Copyright (C) The Arvados Authors. All rights reserved.\n....\n.... SPDX-License-Identifier: CC-BY-SA-3.0\n\nh1. Arvados documentation\n\nThis is the source code for \"doc.arvados.org\":http://doc.arvados.org.\n\nHere's how to build the HTML pages locally so you can preview your updates before you commit and push.\n\nAdditional information is available on the \"'Documentation' page on the Arvados wiki\":https://dev.arvados.org/projects/arvados/wiki/Documentation.\n\nh2. Install dependencies\n\nTo build the core Arvados documentation:\n\n
\narvados/doc$ sudo apt-get install build-essential libcurl4-openssl-dev libgnutls28-dev libssl-dev\narvados/doc$ bundle install\n
\n\nSDK reference documentation has additional, optional build requirements.\n\nh3. Java SDK documentation\n\n
\n$ sudo apt install gradle\n
\n\nh3. Python SDK documentation\n\n
\narvados/doc$ sudo apt install python3-venv\narvados/doc$ python3 -m venv .venv\narvados/doc$ .venv/bin/pip install pdoc setuptools\n
\n\nThen you must activate the virtualenv (e.g., run @. .venv/bin/activate@) before you run the @bundle exec rake@ commands below.\n\nh3. R SDK documentation\n\n
\n$ sudo apt install r-cran-devtools r-cran-roxygen2 r-cran-knitr r-cran-markdown r-cran-xml\n
\n\nh2. Generate HTML pages\n\n
\narvados/doc$ bundle exec rake\n
\n\nAlternately, to make the documentation browsable on the local filesystem:\n\n
\narvados/doc$ bundle exec rake generate baseurl=$PWD/.site\n
\n\nh3. Selecting SDK documentation to build\n\nBy default, the build process will try to detect what SDK documentation it can build, build all that, and skip the rest. You can specify exactly what you want to build using the @sdks@ environment variable. This is a list of comma- or space-separated SDKs you wanted to build documentation for. Valid values are @java@, @python@, @r@, @all@, or @none@. @all@ is a shortcut for listing all the valid SDKs. @none@ means do not build documentation for any SDK. For example, to build documentation for the Java and Python SDKs, but skip R:\n\n
\narvados/doc$ bundle exec rake generate baseurl=$PWD/.site sdks=java,python\n
\n\nSpecifying @sdks@ skips the build detection logic. If the Rakefile cannot build the requested SDK documentation, the build will fail.\n\nFor backwards compatibility, if you do not specify @sdks@, but the @NO_SDK@ environment variable is set, or the @no-sdk@ file exists, the build will run as if you set @sdks=none@.\n\nh2. Run linkchecker\n\nIf you have \"Linkchecker\":http://wummel.github.io/linkchecker/ installed on\nyour system, you can run it against the documentation:\n\n
\narvados/doc$ bundle exec rake linkchecker baseurl=file://$PWD/.site\n
\n\nPlease note that this will regenerate your $PWD/.site directory.\n\nh2. Preview HTML pages\n\n
\narvados/doc$ bundle exec rake run\n[2014-03-10 09:03:41] INFO  WEBrick 1.3.1\n[2014-03-10 09:03:41] INFO  ruby 2.1.1 (2014-02-24) [x86_64-linux]\n[2014-03-10 09:03:41] INFO  WEBrick::HTTPServer#start: pid=8926 port=8000\n
\n\nPreview the rendered pages at \"http://localhost:8000\":http://localhost:8000.\n\nh2. Publish HTML pages inside Workbench\n\n(or some other web site)\n\nYou can set @baseurl@ (the URL prefix for all internal links), @arvados_cluster_uuid@, @arvados_api_host@ and @arvados_workbench_host@ without changing @_config.yml@:\n\n
\narvados/doc$ bundle exec rake generate baseurl=/doc arvados_api_host=xyzzy.arvadosapi.com\n
\n\nh2. Delete generated files\n\n
\narvados/doc$ bundle exec rake realclean\n
\n" }, { "path": "doc/Rakefile", "content": "#!/usr/bin/env rake\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: CC-BY-SA-3.0\n\n# As a convenience to the documentation writer, you can touch a file\n# called 'no-sdk' in the 'doc' directory and it will suppress\n# generating the documentation for the SDKs, which (the R docs\n# especially) take a fair bit of time and slow down the edit-preview\n# cycle.\n#\n# To generate and view the documentation locally, run this command\n#\n# rake && sensible-browser .site/index.html\n#\n# Or alternatively:\n#\n# baseurl=http://localhost:8000 rake && rake run\n#\n# and then visit http://localhost:8000 in a browser.\n\nrequire \"uri\"\n\nrequire \"rubygems\"\nrequire \"colorize\"\n\ndef can_run?(*command, **options)\n options = {\n :in => :close,\n :out => [File::NULL, \"w\"],\n }.merge(options)\n system(*command, **options)\nend\n\nclass JavaSDK\n def self.build_path\n \"contrib/java-sdk-v2\"\n end\n\n def self.can_build?\n can_run?(\"gradle\", \"--version\")\n end\n\n def self.doc_path\n \"sdk/java-v2\"\n end\nend\n\nclass PythonSDK\n def self.build_path\n \"sdk/python/arvados\"\n end\n\n def self.can_build?\n can_run?(\"./pysdk_pdoc.py\", \"--version\")\n end\n\n def self.doc_path\n \"sdk/python/arvados\"\n end\nend\n\nclass RSDK\n def self.build_path\n \"contrib/R-sdk\"\n end\n\n def self.can_build?\n can_run?(\"make\", \"can_run\", chdir: File.join(\"..\", self.build_path))\n end\n\n def self.doc_path\n \"sdk/R\"\n end\nend\n\n$build_sdks = begin\n no_sdk_env = ENV.fetch(\"NO_SDK\", \"\")\n sdks_env = ENV.fetch(\"sdks\", \"\")\n all_sdks = Hash[[JavaSDK, PythonSDK, RSDK].map { |c| [c.name, c] }]\n\n if no_sdk_env != \"\" and sdks_env != \"\"\n fail \"both NO_SDK and sdks defined in environment\"\n elsif sdks_env != \"\"\n # Nothing to do\n elsif no_sdk_env != \"\" or File.exist?(\"no-sdk\")\n sdks_env = \"none\"\n end\n\n if sdks_env == \"\"\n all_sdks.each_pair.filter_map do |name, sdk|\n if sdk.can_build?\n sdk\n else\n puts \"Warning: cannot build #{name.gsub(/SDK$/, ' SDK')} documentation, skipping\".colorize(:light_red)\n end\n end\n else\n wanted_sdks = []\n sdks_env.split(/\\s*[,\\s]\\s*/).each do |key|\n key = \"#{key.capitalize}SDK\"\n if key == \"AllSDK\"\n wanted_sdks = all_sdks.values\n elsif key == \"NoneSDK\"\n wanted_sdks.clear\n elsif sdk = all_sdks[key]\n wanted_sdks << sdk\n else\n fail \"cannot build documentation for unknown #{key}\"\n end\n end\n wanted_sdks\n end\nend\n\nmodule Zenweb\n class Site\n @binary_files = %w[png jpg gif eot svg ttf woff2? ico pdf m4a t?gz xlsx]\n end\nend\n\ntask :generate => [ :realclean, 'sdk/python/arvados.html', 'sdk/R/arvados/index.html', 'sdk/java-v2/javadoc/index.html' ] do\n vars = ['baseurl', 'arvados_cluster_uuid', 'arvados_api_host', 'arvados_workbench_host']\n if ! ENV.key?('baseurl') || ENV['baseurl'] == \"\"\n if !ENV.key?('WORKSPACE') || ENV['WORKSPACE'] == \"\"\n puts \"The `baseurl` variable was not specified and the `WORKSPACE` environment variable is not set. Defaulting `baseurl` to file://#{pwd}/.site\"\n ENV['baseurl'] = \"file://#{pwd}/.site/\"\n else\n puts \"The `baseurl` variable was not specified, defaulting to a value derived from the `WORKSPACE` environment variable\"\n ENV['baseurl'] = \"file://#{ENV['WORKSPACE']}/doc/.site/\"\n end\n end\n vars.each do |v|\n if ENV[v]\n website.config.h[v] = ENV[v]\n end\n end\nend\n\nfile [\"install/new_cluster_checklist_Azure.xlsx\", \"install/new_cluster_checklist_AWS.xlsx\"] do |t|\n cp(t, t)\nend\n\nfile \"sdk/python/arvados.html\" do |t|\n next unless $build_sdks.include?(PythonSDK)\n raise unless system(\"pip\", \"install\", \"../sdk/python\",\n out: :err)\n raise unless system(\"python3\", \"pysdk_pdoc.py\",\n out: :err)\nend\n\nfile \"sdk/R/arvados/index.html\" do |t|\n next unless $build_sdks.include?(RSDK)\n Dir.mkdir(\"sdk/R\")\n Dir.mkdir(\"sdk/R/arvados\")\n cp('css/R.css', 'sdk/R/arvados')\n raise unless system(\"make\", \"man\", chdir: \"../contrib/R-sdk\", out: :err)\n docnames = Dir.glob(\"../contrib/R-sdk/man/*.Rd\").map { |rd| File.basename(rd, \".Rd\") }.sort\n docnames.each do |basename|\n raise unless system(\n \"R\", \"CMD\", \"Rdconv\", \"--type=html\", \"man/#{basename}.Rd\",\n chdir: \"../contrib/R-sdk\",\n out: [\"sdk/R/arvados/#{basename}.html\", \"w\"],\n )\n end\n\n File.open(\"sdk/R/index.html.md\", \"w\") do |fn|\n fn.write(<<-EOF\n---\nlayout: default\nnavsection: sdk\nnavmenu: R\ntitle: \"R SDK Overview\"\n...\n\nEOF\n )\n File.open(\"../contrib/R-sdk/README.md\", \"r\") do |rd|\n fn.write(rd.read)\n end\n end\n\n File.open(\"sdk/R/arvados/index.html.textile.liquid\", \"w\") do |fn|\n fn.write(<<-EOF\n---\nlayout: default\nnavsection: sdk\nnavmenu: R\ntitle: \"R Reference\"\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nEOF\n )\n docnames.each do |basename|\n fn.printf(\"* \\\"%s\\\":%s.html\\n\", basename, basename)\n end\n end\nend\n\nfile \"sdk/java-v2/javadoc/index.html\" do |t|\n next unless $build_sdks.include?(JavaSDK)\n tgt = Dir.pwd\n docfiles = []\n Dir.chdir(\"../contrib/java-sdk-v2\") do\n STDERR.puts `gradle javadoc 2>&1`\n raise if $? != 0\n puts `sed -i \"s/@import.*dejavu.css.*//g\" build/docs/javadoc/stylesheet.css`\n raise if $? != 0\n end\n cp_r(\"../contrib/java-sdk-v2/build/docs/javadoc\", \"sdk/java-v2\")\n raise if $? != 0\nend\n\ntask :linkchecker => [ :generate ] do\n # we need --check-extern to check relative links, weird but true\n opts = [\n \"--check-extern\",\n \"--ignore-url=!^file://\",\n ]\n ([JavaSDK, PythonSDK, RSDK] - $build_sdks).map(&:doc_path).each do |sdk_path|\n sdk_url = URI.join(ENV[\"baseurl\"], sdk_path)\n url_re = Regexp.escape(sdk_url.to_s)\n opts << \"--ignore-url=^#{url_re}[./]\"\n end\n result = system(\n \"linkchecker\", *opts, \"index.html\",\n chdir: \".site\",\n )\n if result.nil?\n fail \"could not run linkchecker command (is it installed?)\"\n elsif !result\n fail \"linkchecker exited #{$?.exitstatus}\"\n end\nend\n\ntask :import_vscode_training do\n Dir.chdir(\"user\") do\n rm_rf \"arvados-vscode-cwl-training\"\n `git clone https://github.com/arvados/arvados-vscode-cwl-training`\n githash = `git --git-dir arvados-vscode-cwl-training/.git log -n1 --format=%H HEAD`\n File.open(\"cwl/arvados-vscode-training.html.md.liquid\", \"w\") do |fn|\n File.open(\"arvados-vscode-cwl-training/README.md\", \"r\") do |rd|\n fn.write(<<-EOF\n---\nlayout: default\nnavsection: userguide\ntitle: \"Developing CWL Workflows with VSCode\"\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n\nImported from https://github.com/arvados/arvados-vscode-cwl-training\ngit hash: #{githash}\n{% endcomment %}\n\nEOF\n )\n fn.write(rd.read())\n end\n end\n rm_rf \"arvados-vscode-cwl-training\"\n end\nend\n\ntask :clean do\n rm_rf \"sdk/python/arvados\"\n rm_f \"sdk/python/arvados.html\"\n rm_f \"sdk/python/index.html\"\n rm_rf \"sdk/R\"\n rm_rf \"sdk/java-v2/javadoc\"\nend\n\nrequire \"zenweb/tasks\"\nload \"zenweb-textile.rb\"\nload \"zenweb-liquid.rb\"\nload \"zenweb-fix-body.rb\"\n\ntask :extra_wirings do\n $website.pages[\"sdk/python/python.html.textile.liquid\"].depends_on(\"sdk/python/arvados.html\")\nend\n" }, { "path": "doc/_config.yml", "content": "# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: CC-BY-SA-3.0\n\n# baseurl is the location of the generated site from the browser's\n# perspective (e.g., http://doc.arvados.org or\n# file:///tmp/arvados/doc/.site). You can also set these on the\n# command line: $ rake generate baseurl=/example\n# arvados_api_host=example.com\n\nbaseurl:\ncurrent_version:\nall_versions:\nlatest_version:\narvados_api_host: localhost\narvados_cluster_uuid: local\narvados_workbench_host: http://localhost\ngoogle_analytics: \"G-EFLSBXJ5SQ\"\nmatomo_analytics_url: \"https://piwik.arvados.org\"\nmatomo_analytics_siteid: \"3\"\n\nexclude: [\"Rakefile\", \"tmp\", \"vendor\"]\n\nnavbar:\n userguide:\n - Welcome:\n - user/index.html.textile.liquid\n - user/getting_started/community.html.textile.liquid\n - Walkthough:\n - user/getting_started/workbench.html.textile.liquid\n - user/tutorials/wgs-tutorial.html.textile.liquid\n - Working at the Command Line:\n - user/getting_started/setup-cli.html.textile.liquid\n - user/reference/api-tokens.html.textile.liquid\n - user/getting_started/check-environment.html.textile.liquid\n - Working with data sets:\n - user/tutorials/tutorial-projects.html.textile.liquid\n - user/tutorials/tutorial-keep.html.textile.liquid\n - user/tutorials/tutorial-keep-get.html.textile.liquid\n - user/tutorials/tutorial-keep-collection-lifecycle.html.textile.liquid\n - user/topics/arv-copy.html.textile.liquid\n - user/tutorials/tutorial-keep-mount-gnu-linux.html.textile.liquid\n - user/tutorials/tutorial-keep-mount-os-x.html.textile.liquid\n - user/tutorials/tutorial-keep-mount-windows.html.textile.liquid\n - user/topics/collection-versioning.html.textile.liquid\n - user/topics/storage-classes.html.textile.liquid\n - Data Analysis with Workflows:\n - user/tutorials/tutorial-workflow-workbench.html.textile.liquid\n - user/cwl/cwl-runner.html.textile.liquid\n - user/cwl/crunchstat-summary.html.textile.liquid\n - user/debugging/container-shell-access.html.textile.liquid\n - user/topics/external-inputs.html.textile.liquid\n - user/topics/service-containers.html.textile.liquid\n - user/cwl/costanalyzer.html.textile.liquid\n - user/cwl/federated-workflows.html.textile.liquid\n - user/cwl/cwl-run-options.html.textile.liquid\n - Common Workflow Language:\n - user/cwl/rnaseq-cwl-training.html.textile.liquid\n - user/cwl/arvados-vscode-training.html.md.liquid\n - user/topics/arv-docker.html.textile.liquid\n - user/cwl/cwl-style.html.textile.liquid\n - user/tutorials/writing-cwl-workflow.html.textile.liquid\n - user/cwl/cwl-extensions.html.textile.liquid\n - user/cwl/cwl-versions.html.textile.liquid\n - Access an Arvados virtual machine:\n - user/getting_started/vm-login-with-webshell.html.textile.liquid\n - user/getting_started/ssh-access-unix.html.textile.liquid\n - user/getting_started/ssh-access-windows.html.textile.liquid\n - Reference:\n - user/topics/workbench-migration.html.textile.liquid\n - user/topics/link-accounts.html.textile.liquid\n - user/reference/cookbook.html.textile.liquid\n - Arvados License:\n - user/copying/copying.html.textile.liquid\n - user/copying/agpl-3.0.html\n - user/copying/LICENSE-2.0.html\n - user/copying/by-sa-3.0.html\n sdk:\n - Overview:\n - sdk/index.html.textile.liquid\n - Python:\n - sdk/python/sdk-python.html.textile.liquid\n - sdk/python/api-client.html.textile.liquid\n - sdk/python/cookbook.html.textile.liquid\n - sdk/python/python.html.textile.liquid\n - sdk/python/arvados-cwl-runner.html.textile.liquid\n - sdk/python/events.html.textile.liquid\n - Command line tools (CLI SDK):\n - sdk/cli/install.html.textile.liquid\n - sdk/cli/index.html.textile.liquid\n - sdk/cli/reference.html.textile.liquid\n - sdk/cli/subcommands.html.textile.liquid\n - FUSE Driver:\n - sdk/fuse/install.html.textile.liquid\n - sdk/fuse/options.html.textile.liquid\n - Go:\n - sdk/go/index.html.textile.liquid\n - sdk/go/example.html.textile.liquid\n - Java:\n - sdk/java-v2/index.html.textile.liquid\n - sdk/java-v2/example.html.textile.liquid\n - sdk/java-v2/javadoc.html.textile.liquid\n - R:\n - sdk/R/index.html.md\n - sdk/R/arvados/index.html.textile.liquid\n - Ruby:\n - sdk/ruby/index.html.textile.liquid\n - sdk/ruby/example.html.textile.liquid\n api:\n - Concepts:\n - api/index.html.textile.liquid\n - api/tokens.html.textile.liquid\n - api/requests.html.textile.liquid\n - api/methods.html.textile.liquid\n - api/resources.html.textile.liquid\n - Permission and authentication:\n - api/methods/users.html.textile.liquid\n - api/methods/groups.html.textile.liquid\n - api/methods/api_client_authorizations.html.textile.liquid\n - api/methods/links.html.textile.liquid\n - api/methods/computed_permissions.html.textile.liquid\n - api/methods/authorized_keys.html.textile.liquid\n - api/methods/credentials.html.textile.liquid\n - api/methods/user_agreements.html.textile.liquid\n - api/methods/virtual_machines.html.textile.liquid\n - Data management:\n - api/keep-webdav.html.textile.liquid\n - api/keep-s3.html.textile.liquid\n - api/keep-web-urls.html.textile.liquid\n - api/projects.html.textile.liquid\n - api/properties.html.textile.liquid\n - api/methods/collections.html.textile.liquid\n - api/methods/logs.html.textile.liquid\n - api/methods/keep_services.html.textile.liquid\n - Container engine:\n - api/methods/container_requests.html.textile.liquid\n - api/methods/containers.html.textile.liquid\n - api/methods/workflows.html.textile.liquid\n - api/dispatch.html.textile.liquid\n architecture:\n - Topics:\n - architecture/index.html.textile.liquid\n - Storage in Keep:\n - architecture/storage.html.textile.liquid\n - architecture/keep-components-overview.html.textile.liquid\n - architecture/keep-clients.html.textile.liquid\n - architecture/keep-data-lifecycle.html.textile.liquid\n - architecture/manifest-format.html.textile.liquid\n - Computation with Crunch:\n - api/execution.html.textile.liquid\n - architecture/dispatchcloud.html.textile.liquid\n - architecture/hpc.html.textile.liquid\n - architecture/singularity.html.textile.liquid\n - Other:\n - api/permission-model.html.textile.liquid\n - architecture/federation.html.textile.liquid\n admin:\n - Topics:\n - admin/index.html.textile.liquid\n - Users and Groups:\n - admin/user-management.html.textile.liquid\n - admin/user-management-cli.html.textile.liquid\n - admin/group-management.html.textile.liquid\n - admin/reassign-ownership.html.textile.liquid\n - admin/link-accounts.html.textile.liquid\n - admin/federation.html.textile.liquid\n - admin/migrating-providers.html.textile.liquid\n - user/topics/arvados-sync-external-sources.html.textile.liquid\n - admin/scoped-tokens.html.textile.liquid\n - admin/token-expiration-policy.html.textile.liquid\n - Monitoring:\n - admin/logging.html.textile.liquid\n - admin/metrics.html.textile.liquid\n - admin/health-checks.html.textile.liquid\n - admin/inspect.html.textile.liquid\n - admin/diagnostics.html.textile.liquid\n - admin/management-token.html.textile.liquid\n - admin/user-activity.html.textile.liquid\n - admin/memory-cpu-profiling.html.textile.liquid\n - Data Management:\n - admin/collection-versioning.html.textile.liquid\n - admin/collection-managed-properties.html.textile.liquid\n - admin/restricting-upload-download.html.textile.liquid\n - admin/keep-balance.html.textile.liquid\n - admin/controlling-container-reuse.html.textile.liquid\n - admin/logs-table-management.html.textile.liquid\n - admin/metadata-vocabulary.html.textile.liquid\n - admin/storage-classes.html.textile.liquid\n - admin/keep-recovering-data.html.textile.liquid\n - admin/keep-measuring-deduplication.html.textile.liquid\n - admin/keep-faster-gc-s3.html.textile.liquid\n - Cloud:\n - admin/spot-instances.html.textile.liquid\n - admin/cloudtest.html.textile.liquid\n - admin/dispatch.html.textile.liquid\n installguide:\n - Overview:\n - install/index.html.textile.liquid\n - Arvados Installer:\n - install/install-single-host.html.textile.liquid\n - install/install-multi-host.html.textile.liquid\n - Manual installation:\n - install/install-manual-prerequisites.html.textile.liquid\n - install/packages.html.textile.liquid\n - Configuration:\n - install/config.html.textile.liquid\n - admin/config-urls.html.textile.liquid\n - admin/config.html.textile.liquid\n - Maintenance and upgrading:\n - admin/upgrading.html.textile.liquid\n - admin/maintenance-and-upgrading.html.textile.liquid\n - Core:\n - install/install-api-server.html.textile.liquid\n - install/diagnostics.html.textile.liquid\n - Keep:\n - install/install-keepstore.html.textile.liquid\n - install/configure-fs-storage.html.textile.liquid\n - install/configure-s3-object-storage.html.textile.liquid\n - install/configure-azure-blob-storage.html.textile.liquid\n - install/install-keepproxy.html.textile.liquid\n - install/install-keep-web.html.textile.liquid\n - install/install-keep-balance.html.textile.liquid\n - User interface:\n - install/setup-login.html.textile.liquid\n - install/install-ws.html.textile.liquid\n - install/install-workbench2-app.html.textile.liquid\n - install/workbench.html.textile.liquid\n - Additional services:\n - install/install-shell-server.html.textile.liquid\n - install/install-webshell.html.textile.liquid\n - Containers API (cloud):\n - install/crunch2-cloud/install-compute-node.html.textile.liquid\n - install/crunch2-cloud/install-dispatch-cloud.html.textile.liquid\n - Compute nodes (Slurm or LSF):\n - install/crunch2/install-compute-node-docker.html.textile.liquid\n - install/crunch2/install-compute-node-singularity.html.textile.liquid\n - Containers API (Slurm):\n - install/crunch2-slurm/install-dispatch.html.textile.liquid\n - install/crunch2-slurm/configure-slurm.html.textile.liquid\n - install/crunch2-slurm/install-test.html.textile.liquid\n - Containers API (LSF):\n - install/crunch2-lsf/install-dispatch.html.textile.liquid\n - Additional configuration:\n - install/container-shell-access.html.textile.liquid\n - External dependencies:\n - install/install-postgresql.html.textile.liquid\n - install/ruby.html.textile.liquid\n - install/nginx.html.textile.liquid\n - install/install-docker.html.textile.liquid\n" }, { "path": "doc/_includes/_admin_list_collections_without_property_py.liquid", "content": "#!/usr/bin/env python3\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\nimport arvados\nimport arvados.util as util\n\nfilters = [['properties.responsible_person_uuid', 'exists', False]]\ncols = util.keyset_list_all(arvados.api().collections().list, filters=filters, select=['uuid', 'name'], order='uuid')\n\nprint('Found {} collections:'.format(len(cols)))\nfor c in cols:\n print('{}, \"{}\"'.format(c['uuid'], c['name']))\n" }, { "path": "doc/_includes/_admin_set_property_to_collections_under_project_py.liquid", "content": "#!/usr/bin/env python3\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\nimport arvados\nimport arvados.util as util\n\ndef get_subproject_uuids(api, root_uuid):\n uuids = []\n groups = util.keyset_list_all(api.groups().list, filters=[['owner_uuid', '=', '{}'.format(root_uuid)]], select=['uuid'], order='uuid')\n for g in groups:\n uuids += ([g['uuid']] + get_subproject_uuids(api, g['uuid']))\n return uuids\n\ndef get_cols(api, filters):\n cols = util.keyset_list_all(api.collections().list, filters=filters, select=['uuid', 'properties'], order='uuid')\n return cols\n\n# Search for collections on project hierarchy rooted at root_uuid\nroot_uuid = 'zzzzz-j7d0g-ppppppppppppppp'\n# Set the property to the UUID below\nresponsible_uuid = 'zzzzz-tpzed-xxxxxxxxxxxxxxx'\n\napi = arvados.api()\nfor p_uuid in [root_uuid] + get_subproject_uuids(api, root_uuid):\n f = [['properties.responsible_person_uuid', 'exists', False],\n ['owner_uuid', '=', p_uuid]]\n cols = get_cols(api, f)\n print('Found {} collections owned by {}'.format(len(cols), p_uuid))\n for c in cols:\n print(' - Updating collection {}'.format(c['uuid']))\n props = c['properties']\n props['responsible_person_uuid'] = responsible_uuid\n api.collections().update(uuid=c['uuid'], body={'properties': props}).execute()\n" }, { "path": "doc/_includes/_admin_update_collection_property_py.liquid", "content": "#!/usr/bin/env python3\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\nimport arvados\nimport arvados.util as util\n\nold_uuid = 'zzzzz-tpzed-xxxxxxxxxxxxxxx'\nnew_uuid = 'zzzzz-tpzed-yyyyyyyyyyyyyyy'\n\napi = arvados.api()\nfilters = [['properties.responsible_person_uuid', '=', '{}'.format(old_uuid)]]\ncols = util.keyset_list_all(api.collections().list, filters=filters, select=['uuid', 'properties'], order='uuid')\n\nprint('Found {} collections'.format(len(cols)))\nfor c in cols:\n print('Updating collection {}'.format(c['uuid']))\n props = c['properties']\n props['responsible_person_uuid'] = new_uuid\n api.collections().update(uuid=c['uuid'], body={'properties': props}).execute()\n" }, { "path": "doc/_includes/_assign_volume_uuid.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nNote that each volume has a UUID, like @zzzzz-nyw5e-0123456789abcde@. You assign these manually: replace @zzzzz@ with your Cluster ID, and replace @0123456789abcde@ with an arbitrary unique string of 15 alphanumerics. Once assigned, UUIDs should not be changed.\n\nEssential configuration values are highlighted in red. Remaining parameters are provided for documentation, with their default values." }, { "path": "doc/_includes/_branchname.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n{% if site.current_version and site.current_version != 'main' %}\n{% assign branchname = site.current_version | slice: 1, 3 | append: '-release' %}\n{% else %}\n{% assign branchname = 'main' %}\n{% endif %}\n" }, { "path": "doc/_includes/_container_glob_patterns.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nh2. Glob patterns\n\nEach pattern in the @output_glob@ array can include the following special terms:\n\ntable(table table-bordered table-condensed).\n|@*@|matches any sequence of non-@/@ characters|\n|@?@|matches any single non-@/@ character|\n|@[abcde]@ or @[a-e]@|matches any non-@/@ character in @abcde@|\n|@[^abcde]@ or @[^a-e]@ or\n@[!abcde]@ or @[!a-e]@|matches any non-@/@ character other than @abcde@|\n|@/**/@|matches zero or more levels of subdirectories|\n|@**/@|at the beginning of a pattern, matches zero or more directories|\n|@/**@|at the end of a pattern, matches any file in any subdirectory|\n\nExample patterns:\n\ntable(table table-bordered table-condensed).\n|@*.txt@|matches files with extension @.txt@ at the top level|\n|@foo/**@|matches the entire tree rooted at @foo@ in the top level|\n|@**/fo[og]@|matches all files named @foo@ or @fog@ anywhere in the tree|\n|@foo/**/*.txt@|matches all files with extension @.txt@ anywhere in the tree rooted at @foo@ in the top level|\n" }, { "path": "doc/_includes/_container_published_ports.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nh2. Published ports\n\nContainers can expose web services. These can be interactive web applications such as Jupyter notebooks or AI chats, or HTTP-based APIs. Arvados acts as a reverse proxy, forwarding HTTP requests to the container and relaying responses back to the client. The external URL will be one of the following, depending on how Arvados is configured (see ContainerWebServices in the \"default config file\":{{site.baseurl}}/admin/config.html):\n\n* @https://-.containers.zzzzz.example.com/@ where @@ is the container UUID and @@ is the port where the container process is listening\n* @https://containers.zzzzz.example.com:/@ where @@ is a dynamically assigned port\n\nTo accept requests, the container should listen on 0.0.0.0 (listening on localhost or 127.0.0.1 will _not_ work) and handle plain text HTTP/1.1 requests.\n\nThe @published_ports@ attribute of the container request record advertises which ports on the container should be available to external clients.\n\nThe value @published_ports@ is a hash. Each key in the hash is a port number that the container is listening on. Each entry in the hash has three keys described here:\n\ntable(table table-bordered table-condensed).\n|_. Key|_. Type|_. Description|\n|access|string|One of \"public\" or \"private\". If \"private\", the client connecting to the container must provide an Arvados API for the user who submitted the container request(s) corresponding to the container. The token is provided as a query parameter @?arvados_api_token=...@. Arvados will consume the query parameter and respond with a redirect and a cookie used to authenticate subsequent requests. If \"public\", no authentication is required.|\n|label|string|A string that will be displayed to the user on Workbench describing the service. Cannot be empty.|\n|initial_path|string|The relative path that should be included when constructing the URL that will be presented in Workbench and in the @initial_url@ field described below. May include any or none of path, fragment and query parameter parts of the URL, or be blank. Leading slash is optional.|\n\nThe @published_ports@ attribute of the container record is a copy of the corresponding container request attribute, with the following entries added for each exposed port when the container enters @Running@ state:\n\ntable(table table-bordered table-condensed).\n|_. Key|_. Type|_. Description|_. Examples|\n|base_url|string|The external URL where the service is reachable.|@https://zzzzz-dz642-abcdefghijklmno-80.containers.zzzzz.example.com/@\n@https://containers.zzzzz.example.com:2000/@|\n|initial_url|string|The external URL with @initial_path@ applied.|@https://zzzzz-dz642-abcdefghijklmno-80.containers.zzzzz.example.com/index.html?start=true@\n@https://containers.zzzzz.example.com:2000/index.html?start=true@|\n|external_port|integer|The dynamically assigned external port if applicable, otherwise null.|@2000@\n@null@|\n\nExample @published_ports@ attribute for a container request:\n\n
\n{\n  \"published_ports\": {\n    \"80\": {\n      \"access\": \"private\",\n      \"label\": \"Jupyter notebook instance\",\n      \"initial_path\": \"?path=example.ipynb\"\n    }\n  }\n}\n
\n\nExample @published_ports@ attribute for a running container:\n\n
\n{\n  \"published_ports\": {\n    \"80\": {\n      \"access\": \"private\",\n      \"label\": \"Jupyter notebook instance\",\n      \"initial_path\": \"?path=example.ipynb\",\n      \"external_port\": 2025,\n      \"base_url\": \"https://containers.zzzzz.example.com:2025/\",\n      \"initial_url\": \"https://containers.zzzzz.example.com:2025/?path=example.ipynb\"\n    }\n  }\n}\n
\n\nh3. Accessing unpublished ports\n\nIf the @Services.ContainerWebServices.ExternalURL@ config entry is a wildcard, it is possible to connect to _any_ port in a running container, whether or not it is listed in @published_ports@, by providing the container request or container UUID and the listening port number as @-@ in place of the @*@ wildcard in the URL, _i.e._, @https://-.containers.zzzzz.example.com/@.\n\nUnpublished ports are not displayed in Workbench and have a default acccess level of \"private\".\n" }, { "path": "doc/_includes/_container_runtime_constraints.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nh2. Runtime constraints\n\nRuntime constraints restrict the container's access to compute resources and the outside world (in addition to its explicitly stated inputs and output).\n\ntable(table table-bordered table-condensed).\n|_. Key|_. Type|_. Description|_. Notes|\n|ram|integer|Number of ram bytes to be used to run this process.|Optional. However, a ContainerRequest that is in \"Committed\" state must provide this.|\n|vcpus|integer|Number of cores to be used to run this process.|Optional. However, a ContainerRequest that is in \"Committed\" state must provide this.|\n|keep_cache_disk|integer|When the container process accesses data from Keep via the filesystem, that data will be cached on disk, up to this amount in bytes.|Optional. If your cluster is configured to use a disk cache by default, the default size will match your @ram@ constraint, bounded between 2GiB and 32GiB.|\n|keep_cache_ram|integer|When the container process accesses data from Keep via the filesystem, that data will be cached in memory, up to this amount in bytes.|Optional. If your cluster is configured to use a RAM cache by default, the administrator sets a default cache size.|\n|API|boolean|When set, ARVADOS_API_HOST and ARVADOS_API_TOKEN will be set, and container will have networking enabled to access the Arvados API server.|Optional.|\n|gpu|object|Request GPU support, see below|Optional.|\n|cuda|object|Old way to request CUDA GPU support, included for backwards compatability only. Use the 'gpu' field instead.|Deprecated.|\n\nh3. GPU support\n\ntable(table table-bordered table-condensed).\n|_. Key|_. Type|_. Description|_. Notes|\n|stack|string|One of 'cuda' or 'rocm' to request Nvidia or AMD GPU support.||\n|device_count|int|Number of GPUs to request.|Count greater than 0 enables GPU support.|\n|driver_version|string|Minimum driver version, in \"X.Y\" format.|Required when device_count > 0|\n|hardware_target|array of strings|For CUDA: a single item with minimum CUDA hardware capability, in \"X.Y\" format, or multiple items listing CUDA specific hardware capability versions, one of which must be an exact match on the compute node the container is scheduled on.\nFor ROCm: A list of one or more hardware targets (e.g. gfx1100) corresponding to the GPU architectures supported by the container. To be scheduled, at least one item in this list must match the @HardwareTarget@ of one of the cluster's @InstanceTypes@.|Required when device_count > 0|\n|vram|int|Amount of VRAM to request, in bytes.||\n\nh3. CUDA support (deprecated)\n\nNote. This API is deprecated. Use the 'gpu' API instead.\n\ntable(table table-bordered table-condensed).\n|_. Key|_. Type|_. Description|_. Notes|\n|device_count|int|Number of GPUs to request.|Count greater than 0 enables CUDA GPU support.|\n|driver_version|string|Minimum CUDA driver version, in \"X.Y\" format.|Required when device_count > 0|\n|hardware_capability|string|Minimum CUDA hardware capability, in \"X.Y\" format.|Required when device_count > 0|\n" }, { "path": "doc/_includes/_container_scheduling_parameters.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nh2. Scheduling parameters\n\nParameters to be passed to the container scheduler (e.g., Slurm) when running a container.\n\ntable(table table-bordered table-condensed).\n|_. Key|_. Type|_. Description|_. Notes|\n|partitions|array of strings|The names of one or more compute partitions that may run this container. If not provided, the system will choose where to run the container.|Optional.|\n|preemptible|boolean|If true, the dispatcher should use a preemptible cloud node instance (eg: AWS Spot Instance) to run this container. Whether a preemptible instance is actually used \"depends on cluster configuration.\":{{site.baseurl}}/admin/spot-instances.html|Optional. Default is false.|\n|max_run_time|integer|Maximum running time (in seconds) that this container will be allowed to run before being cancelled.|Optional. Default is 0 (no limit).|\n" }, { "path": "doc/_includes/_contrib_component.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n{% include 'notebox_begin' %}\n{{ component_name|default: 'This component' }} is an Arvados client contribution. It is supported by the Arvados development team and we are happy to receive contributions for it, but it receives less testing than core components and bug reports may get lower priority.\n{% include 'notebox_end' %}\n" }, { "path": "doc/_includes/_download_installer.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n{% include 'branchname' %}\n\nThis is a package-based installation method, however the installation script is currently distributed in source form via @git@. We recommend checking out the git tree on your local workstation, not directly on the target(s) where you want to install and run Arvados.\n\n\n
git clone https://github.com/arvados/arvados.git\ncd arvados\ngit checkout {{ branchname }}\ncd tools/salt-install\n
\n\n\nThe @install.sh@ and @provision.sh@ scripts will help you deploy Arvados by preparing your environment to be able to run the installer, then running it. The actual installer is located in the \"arvados-formula git repository\":https://github.com/arvados/arvados-formula/tree/refs/heads/{{ branchname }} and will be cloned during the running of the @provision.sh@ script. The installer is built using \"Saltstack\":https://saltproject.io/ and @provision.sh@ performs the install using masterless mode.\n\nh2(#copy_config). Initialize the installer\n\nReplace \"xarv1\" with the cluster id you selected earlier.\n\nThis creates a git repository in @~/setup-arvados-xarv1@. The @installer.sh@ will record all the configuration changes you make, as well as using @git push@ to synchronize configuration edits if you have multiple nodes.\n\nImportant! Once you have initialized the installer directory, all further commands must be run with @~/setup-arvados-${CLUSTER}@ as the current working directory.\n\nh3. Using Terraform (AWS specific)\n\nIf you are going to use Terraform to set up the infrastructure on AWS, you first need to install the \"Terraform CLI\":https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli and the \"AWS CLI\":https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html tool. Then you can initialize the installer.\n\n\n
CLUSTER=xarv1\n./installer.sh initialize ~/setup-arvados-${CLUSTER} {{local_params_src}} {{config_examples_src}} {{terraform_src}}\ncd ~/setup-arvados-${CLUSTER}\n
\n\n\nh3. Without Terraform\n\n\n
CLUSTER=xarv1\n./installer.sh initialize ~/setup-arvados-${CLUSTER} {{local_params_src}} {{config_examples_src}}\ncd ~/setup-arvados-${CLUSTER}\n
\n\n" }, { "path": "doc/_includes/_example_sdk_go.liquid", "content": "// Copyright (C) The Arvados Authors. All rights reserved.\n// \n// SPDX-License-Identifier: CC-BY-SA-3.0\n\npackage main\n\n\n// *******************\n// Import the modules.\n//\n// Our examples don't use keepclient, but they do use fmt and log to\n// display output.\n\nimport (\n\t\"fmt\"\n\t\"git.arvados.org/arvados.git/sdk/go/arvadosclient\"\n\t\"log\"\n)\n\nfunc main() {\n\n\n\t// ********************************\n\t// Set up an API client user agent.\n\t//\n\n\tarv, err := arvadosclient.MakeArvadosClient()\n\tif err != nil {\n\t\tlog.Fatalf(\"Error setting up arvados client %s\", err.Error())\n\t}\n\n\n\t// *****************************************\n\t// Print the full name of the current user.\n\t//\n\n\ttype user struct {\n\t\t// Remember to start each field name with a capital letter,\n\t\t// otherwise it won't get populated by the arvados client because\n\t\t// the field will be invisible to it.\n\t\tUuid string `json:\"uuid\"`\n\t\tFullName string `json:\"full_name\"`\n\t}\n\n\tvar u user\n\terr = arv.Call(\"GET\", \"users\", \"\", \"current\", nil, &u)\n\n\tif err != nil {\n\t\tlog.Fatalf(\"error querying current user\", err.Error())\n\t}\n\n\tlog.Printf(\"Logged in as %s (uuid %s)\", u.FullName, u.Uuid)\n\n\n\t// ********************************************************\n\t// Print all fields from the first five collections returned.\n\t//\n\t// Note that some fields, are not returned by default and have to be\n\t// requested. See below for an example.\n\n\tvar results map[string]interface{}\n\n\tparams := arvadosclient.Dict{\"limit\": 5}\n\n\terr = arv.List(\"collections\", params, &results)\n\tif err != nil {\n\t\tlog.Fatalf(\"error querying collections\", err.Error())\n\t}\n\n\tprintArvadosResults(results)\n\n\n\t// *********************************************************\n\t// Print some fields from the first two collections returned.\n\t//\n\t// We also print manifest_test, which has to be explicitly requested.\n\t//\n\n\tcollection_fields_wanted := []string{\"manifest_text\", \"owner_uuid\", \"uuid\"}\n\tparams = arvadosclient.Dict{\"limit\": 2, \"select\": collection_fields_wanted}\n\n\terr = arv.List(\"collections\", params, &results)\n\tif err != nil {\n\t\tlog.Fatalf(\"error querying collections\", err.Error())\n\t}\n\n\tprintArvadosResults(results)\n}\n\n\n// A helper method which will print out a result map returned by\n// arvadosclient.\nfunc printArvadosResults(results map[string]interface{}) {\n\tfor key, value := range results {\n\t\t// \"items\", if it exists, holds a map.\n\t\t// So we print it prettily below.\n\t\tif key != \"items\" {\n\t\t\tfmt.Println(key, \":\", value)\n\t\t}\n\t}\n\n\tif value, ok := results[\"items\"]; ok {\n\t\titems := value.([]interface{})\n\t\tfor index, item := range items {\n\t\t\tfmt.Println(\"=========== \", index, \" ===========\")\n\t\t\titem_map := item.(map[string]interface{})\n\t\t\tif len(item_map) == 0 {\n\t\t\t\tfmt.Println(\"item\", index, \": empty map\")\n\t\t\t} else {\n\t\t\t\tfor k, v := range item_map {\n\t\t\t\t\tfmt.Println(index, k, \":\", v)\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n}\n" }, { "path": "doc/_includes/_google_analytics.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n\n" }, { "path": "doc/_includes/_hpc_max_gateway_tunnels.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nh3(#MaxGatewayTunnels). API.MaxGatewayTunnels\n\nEach Arvados container that runs on your HPC cluster will bring up a long-lived connection to the Arvados controller and keep it open for the entire duration of the container. This connection is used to access real-time container logs from Workbench, and to enable the \"container shell\":{{site.baseurl}}/install/container-shell-access.html feature.\n\nSet the @MaxGatewayTunnels@ config entry high enough to accommodate the maximum number of containers you expect to run concurrently on your HPC cluster, plus incoming container shell sessions.\n\n\n
    API:\n      MaxGatewayTunnels: 2000
\n\n\nAlso, configure Nginx (and any other HTTP proxies or load balancers running between the HPC and Arvados controller) to allow the expected number of connections, i.e., @MaxConcurrentRequests + MaxQueuedRequests + MaxGatewayTunnels@.\n" }, { "path": "doc/_includes/_html_tags.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThe following HTML tags are permitted: *a*, *b*, *blockquote*, *br*, *code*, *del*, *dd*, *dl*, *dt*, *em*, *h1*, *h2*, *h3*, *h4*, *h5*, *h6*, *hr*, *i*, *img*, *kbd*, *li*, *ol*, *p*, *pre*, *s*, *del*, *section*, *span*, *strong*, *sub*, *sup*, and *ul*.\n\nThe following HTML attributes are permitted: *src*, *width*, *height*, *href*, *alt*, *title*, and *style*.\n\nAll styling must be made in-line with the style attribute. Disallowed tags and attributes will not render." }, { "path": "doc/_includes/_install_ansible.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n{{ header_level|default: 'h3' }}(#install-ansible-pipx). Option 1. Install Ansible with pipx\n\nThe pipx tool is packaged in many of our supported distributions. You can install it on Debian/Ubuntu by running:\n\n\n
# apt install pipx\n
\n\n\nOr install it on Red Hat/AlmaLinux/Rocky Linux by running:\n\n\n
# dnf install pipx
\n\n\n\n{% include 'notebox_begin' %}\nIf the pipx package is not found, it is not available for your distribution. Instead \"install Ansible with virtualenv and pip\":#install-ansible-venv.\n{% include 'notebox_end' %}\n\nAfter pipx is installed, install Ansible by running:\n\n\n
$ arvados/tools/ansible/install-ansible.sh\n  installed package ansible-core 2.15.13, installed using Python 3.11.2\n  These apps are now globally available\n    - ansible\n    - ansible-config\n    - ansible-connection\n    - ansible-console\n    - ansible-doc\n    - ansible-galaxy\n    - ansible-inventory\n    - ansible-playbook\n    - ansible-pull\n    - ansible-test\n    - ansible-vault\ndone! ✨ 🌟 ✨\n\n[…]\n\nAnsible successfully installed!\n
\n\n\nIf this script reports the final success message, skip the next section.\n\n{{ header_level|default: 'h3' }}(#install-ansible-venv). Option 2. Install Ansible in a virtualenv\n\nThis method works on all of our supported distributions, but requires you to configure a lot of paths manually. Install Python and virtualenv on Debian/Ubuntu by running:\n\n\n
# apt install python3-venv\n
\n\n\nOr install it on Red Hat/AlmaLinux/Rocky Linux by running:\n\n\n
# dnf install python3\n
\n\n\nNext, set up a virtualenv. If you want to install this somewhere other than @~/arvados-ansible@, you may change that path each time it appears.\n\n\n
$ arvados/tools/ansible/install-ansible.sh ~/arvados-ansible\nCollecting ansible-core~=2.15.13\n[…]\n\nAnsible successfully installed!\n
\n\n\nFinally, add all the Ansible tools to your executable path. If you keep personal executables somewhere other than @~/.local/bin@, you may change that path.\n\n\n
$ ln -st ~/.local/bin ~/arvados-ansible/bin/ansible*\n
\n\n\nAlternatively, you may reconfigure your shell to add $HOME/arvados-ansible/bin to the end of your @$PATH@ variable.\n" }, { "path": "doc/_includes/_install_ca_cert.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nh3. Web Browser\n\nInstalling the root certificate into your web browser will prevent security errors when accessing Arvados services with your web browser.\n\nh4. Chrome\n\n# Go to \"Settings → Privacy and Security → Security → Manage Certificates\" or enter @chrome://settings/certificates@ in the URL bar.\n# *Click on the \"Authorities\" tab* (it is not selected by default)\n# Click on the \"Import\" button\n# Choose @{{ca_cert_name}}@\n# Tick the checkbox next to \"Trust this certificate for identifying websites\"\n# Hit OK\n# The certificate should appear in the list of Authorities under \"Arvados\"\n\nh4. Firefox\n\n# Go to \"Preferences → Privacy & Security\" or enter @about:preferences#privacy@ in the URL bar\n# Scroll down to the *Certificates* section\n# Click on the button \"View Certificates...\".\n# Make sure the \"Authorities\" tab is selected\n# Press the \"Import...\" button.\n# Choose @{{ca_cert_name}}@\n# Tick the checkbox next to \"Trust this CA to identify websites\"\n# Hit OK\n# The certificate should appear in the list of Authorities under \"Arvados\"\n\nh4. Other browsers (Safari, etc)\n\nThe process will be similar to that of Chrome and Firefox, but the exact user interface will be different. If you can't figure it out, try searching for \"how do I install a custom certificate authority in (my browser)\".\n\nh3. Installation on Linux OS certificate storage\n\nTo access your Arvados instance using command line clients (such as @arv-get@ and @arv-put@) without security errors, install the certificate into the OS certificate storage.\n\nh4. Debian/Ubuntu\n\n*Important* the certificate file added to @ca-certificates@ must have the extension @.crt@ or it won't be recognized.\n\n\n
cp {{ca_cert_name}} /usr/local/share/ca-certificates/arvados-snakeoil-ca.crt\n/usr/sbin/update-ca-certificates\n
\n\n\nh4. Red Hat, AlmaLinux, and Rocky Linux\n\n\n
cp {{ca_cert_name}} /etc/pki/ca-trust/source/anchors/\n/usr/bin/update-ca-trust\n
\n\n" }, { "path": "doc/_includes/_install_compute_docker.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nh2(#cgroups). Configure Linux cgroups accounting\n\nLinux can report what compute resources are used by processes in a specific cgroup or Docker container. Crunch can use these reports to share that information with users running compute work. This can help workflow authors debug and optimize their workflows.\n\nTo enable cgroups accounting, you must boot Linux with the command line parameters @cgroup_enable=memory swapaccount=1@.\n\nCurrently Arvados is not compatible with the new cgroups accounting, also known as cgroups v2. Currently, all supported GNU/Linux distributions don't use cgroups v2 as default\nIf you are using a distribution in the compute nodes that ships with cgroups v2 enabled, make sure to disable it by booting Linux with the command line parameters @systemd.unified_cgroup_hierarchy=0@.\n\nAfter making changes, reboot the system to make these changes effective.\n\nh3. Red Hat, AlmaLinux, and Rocky Linux\n\n\n
~$ sudo grubby --update-kernel=ALL --args='cgroup_enable=memory swapaccount=1 systemd.unified_cgroup_hierarchy=0'\n
\n\n\nh3. Debian and Ubuntu\n\nOpen the file @/etc/default/grub@ in an editor. Find where the string @GRUB_CMDLINE_LINUX@ is set. Add @cgroup_enable=memory swapaccount=1 systemd.unified_cgroup_hierarchy=0@ to that string. Save the file and exit the editor. Then run:\n\n\n
~$ sudo update-grub\n
\n\n\nh2(#install_docker). Install Docker\n\nCompute nodes must have Docker installed to run containers. This requires a relatively recent version of Linux (at least upstream version 3.10, or a distribution version with the appropriate patches backported). Follow the \"Docker Engine installation documentation\":https://docs.docker.com/install/ for your distribution.\n\nMake sure Docker is enabled to start on boot:\n\n\n
# systemctl enable --now docker\n
\n\n\nh2(#configure_docker_daemon). Configure the Docker daemon\n\nDepending on your anticipated workload or cluster configuration, you may need to tweak Docker options.\n\nFor information about how to set configuration options for the Docker daemon, see https://docs.docker.com/config/daemon/systemd/\n\nh3. Changing ulimits\n\nDocker containers inherit ulimits from the Docker daemon. However, the ulimits for a single Unix daemon may not accommodate a long-running Crunch job. You may want to increase default limits for compute containers by passing @--default-ulimit@ options to the Docker daemon. For example, to allow containers to open 10,000 files, set @--default-ulimit nofile=10000:10000@.\n\nh2. Troubleshooting\n\nh3. Workflows fail with @ValidationException: Not found: '/var/lib/cwl/workflow.json#main'@\n\nA possible configuration error is having Docker installed as a @snap@ package rather than a @deb@ package. This is a problem because @snap@ packages are partially containerized and may have a different view of the filesystem than @crunch-run@. This will produce confusing problems, for example, directory bind mounts sent to Docker that are empty (instead of containing the intended files) and resulting in unexpected \"file not found\" errors.\n\nTo check for this situation, run @snap list@ and look for @docker@. If found, run @snap remove docker@ and follow the instructions to above to \"install Docker Engine\":#install_docker .\n" }, { "path": "doc/_includes/_install_compute_fuse.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nh2(#fuse). Update fuse.conf\n\nFUSE must be configured with the @user_allow_other@ option enabled for Crunch to set up Keep mounts that are readable by containers. Install this file as @/etc/fuse.conf@:\n\n\n
\n# Allow non-root users to specify the 'allow_other' or 'allow_root'\n# mount options.\nuser_allow_other\n
\n\n" }, { "path": "doc/_includes/_install_cuda.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nh2(#cuda). Install NVIDA CUDA Toolkit (optional)\n\nIf you want to use NVIDIA GPUs, \"install the CUDA toolkit\":https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html and the \"NVIDIA Container Toolkit\":https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html.\n" }, { "path": "doc/_includes/_install_debian_key.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n\n
# install -d /etc/apt/keyrings\n# curl -fsSL -o /etc/apt/keyrings/arvados.asc https://apt.arvados.org/pubkey.gpg\n
\n\n" }, { "path": "doc/_includes/_install_docker_cleaner.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nh2(#docker-cleaner). Update docker-cleaner.json\n\nThe @arvados-docker-cleaner@ program removes least recently used Docker images as needed to keep disk usage below a configured limit.\n\nCreate a file @/etc/arvados/docker-cleaner/docker-cleaner.json@ in an editor, with the following contents.\n\n\n
{\n    \"Quota\": \"10G\",\n    \"RemoveStoppedContainers\": \"always\"\n}\n
\n\n\n*Choosing a quota:* Most deployments will want a quota that's at least 10G. From there, a larger quota can help reduce compute overhead by preventing reloading the same Docker image repeatedly, but will leave less space for other files on the same storage (usually Docker volumes). Make sure the quota is less than the total space available for Docker images.\n\n{% include 'notebox_begin' %}\nThis also removes all containers as soon as they exit, as if they were run with @docker run --rm@. If you need to debug or inspect containers after they stop, temporarily stop arvados-docker-cleaner or configure it with @\"RemoveStoppedContainers\":\"never\"@.\n{% include 'notebox_end' %}\n" }, { "path": "doc/_includes/_install_packages.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n\npackages_to_install should be a list\nfallback on arvados_component if not defined\n{% endcomment %}\n\n{% if package_to_install == nil %}\n {% assign packages_to_install = arvados_component | split: \" \" %}\n{% endif %}\n\nh2(#install-packages). Install {{packages_to_install | join: \" and \" }}\n\nh3. Red Hat, AlmaLinux, and Rocky Linux\n\n\n
# dnf install {{packages_to_install | join: \" \"}}\n
\n\n\nh3. Debian and Ubuntu\n\n\n
# apt install {{packages_to_install  | join \" \"}}\n
\n\n" }, { "path": "doc/_includes/_install_postgres_database.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n
    \n
  1. Start a shell for the postgres user:\n
    # su postgres
    \n
    2. \n
  2. Generate a new database password:\n
    postgres$ tr -dc 0-9a-zA-Z </dev/urandom | head -c25; echo\nyourgeneratedpassword\n
     Record this. You'll need it when you set up the Rails server later.\n
    3. \n
  3. Create a database user with the password you generated:\n 
    postgres$ createuser --encrypted --no-createrole --no-superuser --pwprompt {{service_role}}\n  Enter password for new role: yourgeneratedpassword\n  Enter it again: yourgeneratedpassword
    \n
    4. \n
  4. Create a database owned by the new user:\n 
    postgres$ createdb {{service_database}} -T template0 -E UTF8 -O {{service_role}}
    \n
    5. \n{% if use_contrib %}\n
  5. Enable the pg_trgm extension\n 
    postgres$ psql {{service_database}} -c \"CREATE EXTENSION IF NOT EXISTS pg_trgm\"
    \n
    6. \n{% endif %}\n
  6. Exit the postgres user shell:\n 
    postgres$ exit
    \n
    7. \n
\n" }, { "path": "doc/_includes/_install_rails_command.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n{% comment %}\nThis template recognizes four variables:\n* railshost: The hostname included in the prompt, to let the user know where to run the command. If this is the empty string, no hostname will be displayed. Default \"apiserver\".\n* railsdir: The directory included in the prompt, to let the user know where to run the command. Default \"/var/www/arvados-api/current\".\n* railscmd: The full command to run. Default \"bin/rails console\".\n* railsout: The expected output of the command, if any.\n{% endcomment %} Change *@webserver-user@* to the user that runs your web server process. If you install Phusion Passenger as we recommend, this is *@www-data@* on Debian-based systems, and *@nginx@* on Red Hat-based systems.\n\n{% unless railshost %}\n {% assign railshost = \"apiserver\" %}\n{% endunless %}\n\n{% unless (railshost == \"\") or (railshost contains \":\") %}\n {% capture railshost %}{{railshost}}:{% endcapture %}\n{% endunless %}\n\n{% unless railsdir %}\n {% assign railsdir = \"/var/www/arvados-api/current\" %}\n{% endunless %}\n\n{% unless railscmd %}\n {% assign railscmd = \"bin/rails console\" %}\n{% endunless %}\n\n\n
{{railshost}}~$ cd {{railsdir}}\n{{railshost}}{{railsdir}}$ sudo -u webserver-user RAILS_ENV=production {{railscmd}}\n{% if railsout %}{{railsout}}\n{% endif %}
\n\n" }, { "path": "doc/_includes/_install_ruby_and_bundler.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nRuby 2.7 or newer is required.\n\nh2. Red Hat, AlmaLinux, and Rocky Linux\n\nVersion 8 of these distributions provides Ruby 2.7. You can install it by running:\n\n\n
# dnf module enable ruby:2.7\n# dnf install --enablerepo=devel ruby ruby-devel
\n\n\nh2. Debian and Ubuntu\n\nAll supported versions of Debian and Ubuntu include a version of Ruby you can use with Arvados.\n\n\n
# apt --no-install-recommends install ruby ruby-dev
\n\n" }, { "path": "doc/_includes/_matomo_analytics.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n\n\n\n\n" }, { "path": "doc/_includes/_metadata_vocabulary_example.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}{\n \"strict_tags\": false,\n \"tags\": {\n \"IDTAGANIMALS\": {\n \"strict\": false,\n \"labels\": [{\"label\": \"Animal\" }, {\"label\": \"Creature\"}, {\"label\": \"Species\"}],\n \"values\": {\n \"IDVALANIMALS1\": { \"labels\": [{\"label\": \"Human\"}, {\"label\": \"Homo sapiens\"}] },\n \"IDVALANIMALS2\": { \"labels\": [{\"label\": \"Dog\"}, {\"label\": \"Canis lupus familiaris\"}] },\n \"IDVALANIMALS3\": { \"labels\": [{\"label\": \"Elephant\"}, {\"label\": \"Loxodonta\"}] },\n \"IDVALANIMALS4\": { \"labels\": [{\"label\": \"Eagle\"}, {\"label\": \"Haliaeetus leucocephalus\"}] }\n }\n },\n \"IDTAGCOMMENT\": {\n \"labels\": [{\"label\": \"Comment\"}, {\"label\": \"Suggestion\"}]\n },\n \"IDTAGIMPORTANCES\": {\n \"strict\": true,\n \"labels\": [{\"label\": \"Importance\"}, {\"label\": \"Priority\"}],\n \"values\": {\n \"IDVALIMPORTANCES1\": { \"labels\": [{\"label\": \"Critical\"}, {\"label\": \"Urgent\"}, {\"label\": \"High\"}] },\n \"IDVALIMPORTANCES2\": { \"labels\": [{\"label\": \"Normal\"}, {\"label\": \"Moderate\"}] },\n \"IDVALIMPORTANCES3\": { \"labels\": [{\"label\": \"Low\"}] }\n }\n }\n }\n}" }, { "path": "doc/_includes/_mount_types.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nh2. Mount types\n\nThe \"mounts\" hash is the primary mechanism for adding data to the container at runtime (beyond what is already in the container image).\n\nEach value of the \"mounts\" hash is itself a hash, whose \"kind\" key determines the handler used to attach data to the container.\n\ntable(table table-bordered table-condensed).\n|_. Mount type|_. Kind|_. Description|_. Examples|\n|Arvados data collection|@collection@|@\"portable_data_hash\"@ _or_ @\"uuid\"@ _may_ be provided. If not provided, a new collection will be created. This is useful when @\"writable\":true@ and the container's @output_path@ is (or is a subdirectory of) this mount target.\n@\"writable\"@ may be provided with a @true@ or @false@ to indicate the path must (or must not) be writable. If not specified, the system can choose.\n@\"path\"@ may be provided, and defaults to @\"/\"@.\nAt container startup, the target path will have the same directory structure as the given path within the collection. Even if the files/directories are writable in the container, modifications will _not_ be saved back to the original collections when the container ends.|
{\n \"kind\":\"collection\",\n \"uuid\":\"...\",\n \"path\":\"/foo.txt\"\n}\n{\n \"kind\":\"collection\",\n \"uuid\":\"...\"\n}
|\n|Temporary directory|@tmp@|@\"capacity\"@: capacity (in bytes) of the storage device.\n@\"device_type\"@ (optional, default \"network\"): one of @{\"ram\", \"ssd\", \"disk\", \"network\"}@ indicating the acceptable level of performance. (*note: not yet implemented as of v1.5*)\nAt container startup, the target path will be empty. When the container finishes, the content will be discarded. This will be backed by a storage mechanism no slower than the specified type.|
{\n \"kind\":\"tmp\",\n \"capacity\":100000000000\n}\n{\n \"kind\":\"tmp\",\n \"capacity\":1000000000,\n \"device_type\":\"ram\"\n}
|\n|Keep|@keep@|Expose all readable collections via arv-mount.\nRequires suitable runtime constraints.|
{\n \"kind\":\"keep\"\n}
|\n|Mounted file or directory|@file@|@\"path\"@: absolute path (inside the container) of a file or directory that is (or is inside) another mount target.\nCan be used for \"stdin\" and \"stdout\" targets.|
{\n \"kind\":\"file\",\n \"path\":\"/mounted_tmp/a.out\"\n}
|\n|JSON document|@json@|A JSON-encoded string, array, or object.|
{\n \"kind\":\"json\",\n \"content\":{\"foo\":\"bar\"}\n}
|\n|Plain text|@text@|A plain text string.|
{\n \"kind\":\"text\",\n \"content\":\"foo bar\\n\"\n}
|\n\nh2(#pre-populate-output). Pre-populate output using Mount points\n\nWhen a container's output_path is a tmp mount backed by local disk, this output directory can be pre-populated with content from existing collections. This content can be specified by mounting collections at mount points that are subdirectories of output_path. Certain restrictions apply:\n\n1. Only mount points of kind @collection@ are supported.\n\n2. Mount points underneath output_path which have @\"writable\":true@ are copied into output_path during container initialization and may be updated, renamed, or deleted by the running container. The original collection is not modified. On container completion, files remaining in the output are saved to the output collection. The mount at output_path must be big enough to accommodate copies of the inner writable mounts.\n\n3. If any such mount points are configured as @\"exclude_from_output\":true@, they will be excluded from the output.\n\nIf any process in the container tries to modify, remove, or rename these mount points or anything underneath them, the operation will fail and the container output and the underlying collections used to pre-populate are unaffected.\n\nh3. Example mount point configurations\n\nAll the below examples are based on this collection:\n
\nportable_data_hash cdfbe2e823222d26483d52e5089d553c+175\n\nmanifest_text: ./alice 03032680d3fa0561ef4f85071140861e+13+A04e9d06459cda00aa997565bd78001061cf5bffb@58ab593d 0:13:hello.txt\\n./bob d820b9df970e1b498e7723c50b107e1b+11+A42d162a60210479d1cfaf9fbb98d494ac6322ae6@58ab593d 0:11:hello.txt\\n./carol cf72b172ff969250ae14a893a6745440+13+A476a2fd39e14e9c03af3076bd17e3612c075ff66@58ab593d 0:13:hello.txt\\n\n
\n\ntable(table table-bordered table-condensed).\n|{width:40%}. *Mount point*|{width:30%}. *Description*|{width:30%}. *Resulting collection manifest text*|\n|
\"mounts\": {\n  \"/tmp/foo\": {\n    \"kind\": \"collection\",\n    \"portable_data_hash\": \"cdfbe2...+175\"\n  },\n},\n\"output_path\": \"/tmp\"\n
|No path specified and hence the entire collection will be mounted.|./*foo/*alice 030326... 0:13:hello.txt\\n\n./*foo/*bob d820b9... 0:11:hello.txt\\n\n./*foo/*carol cf72b1... 0:13:hello.txt\\n\n*Note:* Here the \".\" in streams is replaced with *foo*.|\n|
\"mounts\": {\n  \"/tmp/foo/bar\": {\n    \"kind\": \"collection\",\n    \"portable_data_hash\": \"cdfbe2...+175\"\n    \"path\": \"alice\"\n  },\n},\n\"output_path\": \"/tmp\"\n
|Specified path refers to the subdirectory *alice* in the collection.|./*foo/bar* 030326... 0:13:hello.txt\\n\n*Note:* only the manifest text segment for the subdirectory *alice* is included after replacing the subdirectory *alice* with *foo/bar*.|\n|
\"mounts\": {\n  \"/tmp/foo/bar\": {\n    \"kind\": \"collection\",\n    \"portable_data_hash\": \"cdfbe2...+175\"\n    \"path\": \"alice/hello.txt\"\n  },\n},\n\"output_path\": \"/tmp\"\n
|Specified path refers to the file *hello.txt* in the *alice* subdirectory|./*foo* 030326... 0:13:*bar*\\n\n*Note:* Here the subdirectory *alice* is replaced with *foo* and the filename *hello.txt* from this subdirectory is replaced with *bar*.|\n\nh2(#symlinks-in-output). Symlinks in output\n\nWhen a container's output_path is a tmp mount backed by local disk, this output directory can contain symlinks to other files in the output directory, or to collection mount points. If the symlink leads to a collection mount, efficiently copy the collection into the output collection. Symlinks leading to files or directories are expanded and created as regular files in the output collection. Further, whether symlinks are relative or absolute, every symlink target (even targets that are symlinks themselves) must point to a path in either the output directory or a collection mount.\n" }, { "path": "doc/_includes/_multi_host_install_custom_certificates.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nYou will need certificates for each DNS name and DNS wildcard previously listed in the \"DNS hostnames for each service\":#DNS .\n\nTo simplify certificate management, we recommend creating a single certificate for all of the hostnames, or creating a wildcard certificate that covers all possible hostnames (with the following patterns in subjectAltName):\n\n
\nxarv1.example.com\n*.xarv1.example.com\n*.collections.xarv1.example.com\n*.containers.xarv1.example.com\n
\n\n(Replacing @xarv1.example.com@ with your own @${DOMAIN}@)\n\nCopy your certificates to the directory specified with the variable @CUSTOM_CERTS_DIR@ in the remote directory where you copied the @provision.sh@ script. The provision script will find the certificates there.\n\nThe script expects cert/key files with these basenames (matching the role except for keepweb, which is split in both download / collections):\n\n# @balancer@ -- Optional on multi-node installations\n# @collections@ -- Part of keepweb, must be a wildcard for @*.collections.${DOMAIN}@\n# @controller@ -- Must be valid for @${DOMAIN}@ and @*.containers.${DOMAIN}@\n# @download@ -- Part of keepweb\n# @grafana@ -- Service available by default on multi-node installations\n# @keepproxy@ -- Corresponds to default domain @keep.${DOMAIN}@\n# @prometheus@ -- Service available by default on multi-node installations\n# @webshell@\n# @websocket@ -- Corresponds to default domain @ws.${DOMAIN}@\n# @workbench@\n# @workbench2@\n\nFor example, for the @keepproxy@ service the script will expect to find this certificate:\n\n\n
${CUSTOM_CERTS_DIR}/keepproxy.crt\n${CUSTOM_CERTS_DIR}/keepproxy.key\n
\n\n\nMake sure that all the FQDNs that you will use for the public-facing applications (API/controller, Workbench, Keepproxy/Keepweb) are reachable.\n\nNote: because the installer currently looks for a different certificate file for each service, if you use a single certificate, we recommend creating a symlink for each certificate and key file to the primary certificate and key, e.g.\n\n\n
ln -s xarv1.crt ${CUSTOM_CERTS_DIR}/controller.crt\nln -s xarv1.key ${CUSTOM_CERTS_DIR}/controller.key\nln -s xarv1.crt ${CUSTOM_CERTS_DIR}/keepproxy.crt\nln -s xarv1.key ${CUSTOM_CERTS_DIR}/keepproxy.key\n...\n
\n\n" }, { "path": "doc/_includes/_navbar_left.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n{% assign highlighturl = \"\" %}\n{% for section in site.navbar[page.navsection] %}\n {% for entry in section %}\n {% comment %}\n Want to highlight the current page on the left nav.\n But some pages have been renamed with a symlink from the old page to the new one.\n Then the URL won't match.\n So if the URL doesn't match, as a fallback look for a page with a matching title.\n {% endcomment %}\n\n {% for item in entry[1] %}\n {% if site.pages[item].url == page.url %}\n {% assign highlighturl = site.pages[item].url %}\n {% endif %}\n {% endfor %}\n\n {% if highlighturl == \"\" %}\n {% for item in entry[1] %}\n {% if site.pages[item].title == page.title %}\n {% assign highlighturl = site.pages[item].url %}\n {% endif %}\n {% endfor %}\n {% endif %}\n {% endfor %}\n{% endfor %}\n\n
\n
\n
    \n {% for section in site.navbar[page.navsection] %}\n {% for entry in section %}\n
  1. {{ entry[0] }}\n\t
      \n {% for item in entry[1] %}\n {% assign p = site.pages[item] %}\n
    1. \n {{ p.title }}
      2. \n {% endfor %}\n
    \n {% endfor %}\n {% endfor %}\n
\n
\n
\n" }, { "path": "doc/_includes/_navbar_top.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n
\n
\n
\n \n Arvados™ Docs\n
\n
\n \n\n
\n
\n
\n \n
\n \n
\n \n
\n
\n
\n
\n\n{% comment %}\nThe div with id \"old-version-warning\" is automatically enabled by the Apache\nconfig under certain conditions, as described at\nhttps://dev.arvados.org/projects/ops/wiki/Docarvadosorg. Changing the class\nshould be safe, but please double check the matching and substitution that is\ndone in the Apache config to be sure before you modify this div.\n{% endcomment %}\n
\n WARNING - you are viewing the documentation for an old version of Arvados. For the latest version, click here.\n
\n
\n
Hey! You can use the annotation sidebar from hypothes.is to make public comments and private notes\n →
\n \n
\n\n \n\n
\n
\n" }, { "path": "doc/_includes/_notebox_begin.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n
\n \n

Note:

\n" }, { "path": "doc/_includes/_notebox_begin_warning.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n
\n

Note:

\n" }, { "path": "doc/_includes/_notebox_end.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n
\n" }, { "path": "doc/_includes/_restart_api.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nh2(#restart-api). Restart the API server and controller\n\n*Make sure the cluster config file is up to date on the API server host* then restart the API server and controller processes to ensure the configuration changes are visible to the whole cluster.\n\n\n
# systemctl restart nginx arvados-controller\n# arvados-server check\n
\n\n" }, { "path": "doc/_includes/_setup_debian_repo.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n\npackages_to_install may be a space-separated string\n{% endcomment %}\n\nSet up the Arvados package repository\n{%- if packages_to_install == nil %}\n{%- elsif packages_to_install contains \" \" %} and install the packages\n{%- else %} and install @{{ packages_to_install }}@\n{%- endif %} by running these commands:\n\n\n
# install -d /etc/apt/keyrings\n# curl -fsSL -o /etc/apt/keyrings/arvados.asc https://apt.arvados.org/pubkey.gpg\n# declare $(grep \"^VERSION_CODENAME=\" /etc/os-release || echo VERSION_CODENAME=MISSING)\n# tee /etc/apt/sources.list.d/arvados.sources >/dev/null <<EOF\nTypes: deb\nURIs: https://apt.arvados.org/$VERSION_CODENAME\nSuites: $VERSION_CODENAME\nComponents: main\nSigned-by: /etc/apt/keyrings/arvados.asc\nEOF\n# apt update\n{%- if packages_to_install != nil %}\n# apt install {{ packages_to_install }}\n{% endif -%}\n
\n\n" }, { "path": "doc/_includes/_setup_redhat_repo.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n\nmodules_to_enable and packages_to_install may be space-separated strings\n{% endcomment %}\n\n{%- if modules_to_enable != nil %}\n{% include 'notebox_begin_warning' %}\n\nArvados tools require newer language runtimes than the default versions included with these distributions. These instructions will **upgrade language runtimes for the entire system**. Check that won't interfere with any existing software before you proceed.\n\n{% include 'notebox_end' %}\n{% endif -%}\n\nSet up the Arvados package repository\n{%- if packages_to_install == nil %}\n{%- elsif packages_to_install contains \" \" %} and install the packages\n{%- else %} and install @{{ packages_to_install }}@\n{%- endif %} by running these commands:\n\n\n
# tee /etc/yum.repos.d/arvados.repo >/dev/null <<'EOF'\n[arvados]\nname=Arvados\nbaseurl=https://rpm.arvados.org/RHEL/$releasever/os/$basearch/\ngpgcheck=1\ngpgkey=https://rpm.arvados.org/RHEL/$releasever/RPM-GPG-KEY-arvados\nEOF\n{%- if modules_to_enable != nil %}\n# dnf module enable {{ modules_to_enable }}\n{% endif -%}\n{%- if packages_to_install != nil -%}\n# dnf install {{ packages_to_install }}\n{% endif -%}\n
\n\n" }, { "path": "doc/_includes/_singularity_mksquashfs_configuration.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n{{ mksquashfs_header|default: \"h2\" }}(#singularity_mksquashfs_configuration). Singularity mksquashfs configuration\n\n{% if show_docker_warning != nil %}\n{% include 'notebox_begin_warning' %}\nThis section is only relevant when using Singularity. Skip this section when using Docker.\n{% include 'notebox_end' %}\n{% endif %}\n\nDocker images are converted on the fly by @mksquashfs@, which can consume a considerable amount of RAM. The RAM usage of mksquashfs can be restricted in @/etc/singularity/singularity.conf@ with a line like @mksquashfs mem = 256M@. The amount of memory made available for mksquashfs should be configured lower than the smallest amount of memory requested by a container on the cluster to avoid the conversion being killed for using too much memory. The default memory allocation in CWL is 256M, so that is also a good choice for the @mksquashfs mem@ setting.\n" }, { "path": "doc/_includes/_ssh_addkey.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n\nYou may now proceed to \"adding your key to the Arvados Workbench.\":#workbench\n\nh1(#workbench). Adding your key to Arvados Workbench\n\nIn the Workbench top navigation menu, click on the dropdown menu icon to access the Account Management menu. Then, click on the menu item *Ssh keys* to go to the *SSH keys* page. Click on the + ADD NEW SSH KEY button in the upper-right on that page. You will see a popup as shown in this screenshot:\n\n!{width: 100%;}{{ site.baseurl }}/images/ssh-adding-public-key.png!\n\nPaste your _public_ key into the text area labeled *Public Key*, and click on the ADD NEW SSH KEY button in lower-right. You are now ready to \"log into an Arvados VM\":#login.\n\nh1(#login). Using SSH to log into an Arvados VM\n\nTo see a list of virtual machines that you have access to, click on the dropdown menu icon in the upper right corner of the top navigation menu to access the Account Management menu. Then, click on the menu item *Virtual Machines*.\n\nYou will then see a page that lists the virtual machines you can access. The *Host name* column lists the name of each available VM. The *Login name* column lists your login name on that VM. The *Command line* column provides a sample @ssh@ command.\n\n!{width: 100%;}{{ site.baseurl }}/images/vm-access-with-webshell.png!\n\nAt the bottom of the page there may be additional instructions for connecting your specific Arvados instance. If so, follow your site-specific instructions. If there are no site-specific instructions, you can probably connect directly with @ssh@.\n\nThe following are generic instructions. In these examples, the login name will be *_you_* and the host domain will be *_ClusterID.example.com_*. Replace these with your login name and hostname as appropriate.\n" }, { "path": "doc/_includes/_ssh_intro.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n\nArvados requires a public SSH key in order to securely log in to an Arvados VM instance. The three sections below help you get started:\n\n# \"Getting your SSH key\":#gettingkey\n# \"Adding your key to Arvados Workbench\":#workbench\n# \"Using SSH to log into an Arvados VM instance\":#login\n\n" }, { "path": "doc/_includes/_ssl_config_multi.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nh2(#certificates). Choose the SSL/TLS configuration (SSL_MODE)\n\nArvados requires a valid TLS certificate to work correctly. This installer supports these options:\n\n# @lets-encrypt@: \"automatically obtain and install an SSL certificates for your hostnames\":#lets-encrypt\n# @bring-your-own@: \"supply your own certificates in the @certs@ directory\":#bring-your-own\n\nh3(#lets-encrypt). Using a Let's Encrypt certificate\n\nIn the default configuration, this installer gets a valid certificate via Let's Encrypt. If you have the CLUSTER.DOMAIN domain in a route53 zone, you can set USE_LETSENCRYPT_ROUTE53 to YES and supply appropriate credentials so that Let's Encrypt can use dns-01 validation to get the appropriate certificates.\n\n\n
SSL_MODE=\"lets-encrypt\"\nUSE_LETSENCRYPT_ROUTE53=\"yes\"\nLE_AWS_REGION=\"us-east-1\"\nLE_AWS_ACCESS_KEY_ID=\"AKIABCDEFGHIJKLMNOPQ\"\nLE_AWS_SECRET_ACCESS_KEY=\"thisistherandomstringthatisyoursecretkey\"\n
\n\n\nPlease note that when using AWS, EC2 instances can have a default hostname that ends with amazonaws.com. Let's Encrypt has a blacklist of domain names for which it will not issue certificates, and that blacklist includes the amazonaws.com domain, which means the default hostname can not be used to get a certificate from Let's Encrypt.\n\nh3(#bring-your-own). Bring your own certificates\n\nTo supply your own certificates, change the configuration like this:\n\n\n
SSL_MODE=\"bring-your-own\"\n
\n\n\n{% include 'multi_host_install_custom_certificates' %}\n\nAll certificate files will be used by nginx. You may need to include intermediate certificates in your certificate files. See \"the nginx documentation\":http://nginx.org/en/docs/http/configuring_https_servers.html#chains for more details.\n\nh4(#secure-tls-keys). Securing your TLS certificate keys (AWS specific) (optional)\n\nWhen using @SSL_MODE=bring-your-own@, you can keep your TLS certificate keys encrypted on the server nodes. This reduces the risk of certificate leaks from node disk volumes snapshots or backups.\n\nThis feature is currently implemented in AWS by providing the certificate keys’ password via Amazon’s \"Secrets Manager\":https://aws.amazon.com/es/secrets-manager/ service, and installing appropriate services on the nodes that provide this password to nginx via a file that only lives in system RAM.\n\nIf you use the installer's Terraform code, the secret and related permission cloud resources are created automatically, and you can customize the secret's name by editing @terraform/services/terraform.tfvars@ and setting its suffix in @ssl_password_secret_name_suffix@.\n\nIn @local.params@ you need to set @SSL_KEY_ENCRYPTED@ to @yes@ and change the default values for @SSL_KEY_AWS_SECRET_NAME@ and @SSL_KEY_AWS_REGION@ if necessary.\n\nThen, if your certificate key file is not yet encrypted, you can generated an encrypted version of it by running the @openssl@ command as follows:\n\n\n
openssl rsa -aes256 -in your.key -out your.encrypted.key\n
\n\n(this will ask you to type the encryption password)\n\nThis encrypted key file will be the one needed to be copied to the @${CUSTOM_CERTS_DIR}@ directory, instead of the plain key file.\n\nIn order to allow the appropriate nodes decrypt the key file, you should set the password on Amazon Secrets Manager. There're a couple way this can be done:\n\n# Through AWS web interface may be the easiest, just make sure to set it as \"plain text\" instead of JSON.\n# By using the AWS CLI tools, for example:\n\n
aws secretsmanager put-secret-value --secret-id pkey-pwd --secret-string \"p455w0rd\" --region us-east-1\n
\nWhere @pkey-pwd@ should match with what's set in @SSL_KEY_AWS_SECRET_NAME@ and @us-east-1@ with what's set in @SSL_KEY_AWS_REGION@.\n\nTake into account that the AWS secret should be set before running @installer.sh deploy@ to avoid any failures when trying to start the @nginx@ servers.\n\nIf you ever need to change the encryption password on a running cluster, you should first change the secret's value on AWS, and only then copy the newly encrypted key file to @${CUSTOM_CERTS_DIR}@ and re-run the deploy command." }, { "path": "doc/_includes/_ssl_config_single.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nh2(#certificates). Choose the SSL configuration (SSL_MODE)\n\nArvados requires an SSL certificate to work correctly. This installer supports these options:\n\n* @self-signed@: let the installer create a self-signed certificate\n* @lets-encrypt@: automatically obtain and install an SSL certificate for your hostname\n* @bring-your-own@: supply your own certificate in the `certs` directory\n\nh3(#self-signed). Using a self-signed certificate\n\nIn the default configuration, this installer uses self-signed certificate(s):\n\n\n
SSL_MODE=\"self-signed\"\n
\n\n\nThis works everywhere and does not require that you have a domain name. However, after installation, users will need to \"install the self-signed root certificate in the browser.\":#ca_root_certificate\"\n\nh3(#lets-encrypt). Using a Let's Encrypt certificate\n\nTo automatically get a valid certificate via Let's Encrypt, change the configuration like this:\n\n\n
SSL_MODE=\"lets-encrypt\"\n
\n\n\nThis requires that you have a \"real\" hostname that you control. The hostname for your Arvados cluster must be defined in @HOSTNAME_EXT@ and resolve to the public IP address of your Arvados instance, so that Let's Encrypt can validate the domainname ownership and issue the certificate.\n\nWhen using AWS, EC2 instances can have a default hostname that ends with amazonaws.com. Let's Encrypt has a blacklist of domain names for which it will not issue certificates, and that blacklist includes the amazonaws.com domain, which means the default hostname can not be used to get a certificate from Let's Encrypt.\n\nh3(#bring-your-own). Bring your own certificate\n\nTo supply your own certificate, change the configuration like this:\n\n\n
SSL_MODE=\"bring-your-own\"\n
\n\n\nCopy your certificate files to the directory specified with the variable @CUSTOM_CERTS_DIR@. The provision script will find it there. The certificate and its key need to be copied to a file named after @HOSTNAME_EXT@. For example, if @HOSTNAME_EXT@ is defined as @my-arvados.example.net@, the script will look for\n\n\n
${CUSTOM_CERTS_DIR}/my-arvados.example.net.crt\n${CUSTOM_CERTS_DIR}/my-arvados.example.net.key\n
\n\n\nAll certificate files will be used by nginx. You may need to include intermediate certificates in your certificate file. See \"the nginx documentation\":http://nginx.org/en/docs/http/configuring_https_servers.html#chains for more details.\n" }, { "path": "doc/_includes/_start_service.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nh2(#start-service). Start the service\n\n\n
# systemctl enable --now {{arvados_component}}\n# systemctl status {{arvados_component}}\n[...]\n
\n\n\nIf @systemctl status@ indicates it is not running, use @journalctl@ to check logs for errors:\n\n\n
# journalctl --since -5min -u {{ arvados_component | split: ' ' | join: ' -u ' }}\n
\n\n" }, { "path": "doc/_includes/_supportedlinux.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\ntable(table table-bordered table-condensed).\n|_. *Supported Linux Distributions*|\n|AlmaLinux 10 (since 10.0)|\n|AlmaLinux 9 (since 9.2)|\n|AlmaLinux 8 (since 8.8)|\n|Debian 12 (\"bookworm\")|\n|Red Hat Enterprise Linux 10 (since 10.0)|\n|Red Hat Enterprise Linux 9 (since 9.2)|\n|Red Hat Enterprise Linux 8 (since 8.8)|\n|Rocky Linux 10 (since 10.0)|\n|Rocky Linux 9 (since 9.2)|\n|Rocky Linux 8 (since 8.8)|\n|Ubuntu 24.04 (\"noble\")|\n|Ubuntu 22.04 (\"jammy\")|\n" }, { "path": "doc/_includes/_tutorial_expectations.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n{% include 'notebox_begin' %}\nThis tutorial assumes that you have access to \"Arvados command line tools\":{{ site.baseurl }}/user/getting_started/setup-cli.html, configured your \"API token\":{{site.baseurl}}/user/reference/api-tokens.html, and confirmed a \"working environment\":{{site.baseurl}}/user/getting_started/check-environment.html.\n{% include 'notebox_end' %}\n" }, { "path": "doc/_includes/_tutorial_expectations_workstation.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n{% include 'notebox_begin' %}\nThis tutorial assumes that you have installed the Arvados \"Command line SDK\":{{site.baseurl}}/sdk/cli/install.html and \"Python SDK\":{{site.baseurl}}/sdk/python/sdk-python.html on your workstation and have a \"working environment.\":{{site.baseurl}}/user/getting_started/check-environment.html\n{% include 'notebox_end' %}\n" }, { "path": "doc/_includes/_tutorial_hello_cwl.liquid", "content": "#!/usr/bin/env cwl-runner\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: CC-BY-SA-3.0\n\ncwlVersion: v1.0\nclass: CommandLineTool\ninputs: []\noutputs: []\narguments: [\"echo\", \"hello world!\"]\n" }, { "path": "doc/_includes/_webring.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n{% assign n = 0 %}\n{% assign prev = \"\" %}\n{% assign nx = 0 %}\n{% for section in site.navbar[page.navsection] %}\n {% for entry in section %}\n {% for item in entry[1] %} \n {% assign p = site.pages[item] %}\n {% if nx == 1 %}\n
\n {% if prev != \"\" %}\n Previous: {{ prev.title }}\n {% endif %}\n Next: {{ p.title }}\n {% assign nx = 0 %}\n {% assign n = 1 %}\n {% endif %}\n {% if p.url == page.url %}\n {% assign nx = 1 %}\n {% else %}\n {% assign prev = p %}\n {% endif %}\n {% endfor %}\n {% endfor %}\n{% endfor %}\n{% if n == 0 && prev != \"\" %}\n
\n Previous: {{ prev.title }}\n {% assign n = 1 %}\n{% endif %}" }, { "path": "doc/_includes/_what_is_cwl.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThe \"Common Workflow Language (CWL)\":http://commonwl.org is a multi-vendor open standard for describing analysis tools and workflows that are portable across a variety of platforms. CWL is the primary way to develop and run workflows for Arvados. Arvados supports versions \"v1.0\":http://commonwl.org/v1.0, \"v1.1\":http://commonwl.org/v1.1, and \"v1.2\":http://commonwl.org/v1.2 of the CWL standard.\n" }, { "path": "doc/_layouts/default.html.liquid", "content": "{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n\n\n \n \n {% unless page.title == \"Arvados | Documentation\" %} Arvados {% if page.navmenu %}| {{ page.navmenu }} {% endif %} | {% endunless %}{{ page.title }}\n \n \n \n {% if site.current_version != site.latest_version %}\n \n {% endif %}\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n {% include 'matomo_analytics' %}\n {% include 'google_analytics' %}\n\n \n \n {% include 'navbar_top' %}\n\n {% if page.navsection == 'top' or page.no_nav_left %}\n {{ content }}\n {% else %}\n\n
\n\n
\n {% include 'navbar_left' %}\n
\n

{{ page.title }}

\n {{ content }}\n {% include 'webring' %}\n
\n
\n\n
\n\n
\n {% endif %}\n\n{% if page.no_nav_left %}\n{% else %}\n

\nThe content of this documentation is licensed under the\nCreative\n Commons Attribution-Share Alike 3.0 United States licence.
\nCode samples in this documentation are licensed under the\nApache License, Version 2.0.\n

\n{% endif %}\n\n \n\n" }, { "path": "doc/admin/cloudtest.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: Testing cloud configuration\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThe @arvados-server@ package includes a @cloudtest@ tool that checks compatibility between your Arvados configuration, your cloud driver, your cloud provider's API, your cloud provider's VM instances, and the worker image you use with the \"cloud dispatcher\":../install/crunch2-cloud/install-dispatch-cloud.html.\n\n@arvados-server cloudtest@ performs the following steps:\n# Create a new instance\n# Wait for it to finish booting\n# Run a shell command on the new instance (optional)\n# Pause while you log in to the new instance and do other tests yourself (optional)\n# Shut down the instance\n\nThis is an easy way to expose problems like these:\n* Configured cloud credentials don't work\n* Configured image types don't work\n* Configured driver is not compatible with your cloud API/region\n* Newly created instances are not usable due to a network problem or misconfiguration\n* Newly created instances do not accept the configured SSH private key\n* Selected machine image does not boot properly\n* Selected machine image is incompatible with some instance types\n* Driver has bugs\n\nh2. Typical uses\n\nBefore bringing up the @arvados-dispatch-cloud@ service for the first time, we recommend running @cloudtest@ to check your configuration:\n\n
\n$ arvados-server cloudtest -command \"crunch-run --list\"\n
\n\nBefore updating your configuration to use a new VM image, we recommend running @cloudtest@ with the new image:\n\n
\n$ arvados-server cloudtest -image-id new_image_id -command \"crunch-run --list\"\n
\n\nAfter adding an instance type to your configuration, we recommend running @cloudtest@ with the new instance type:\n\n
\n$ arvados-server cloudtest -instance-type new_instance_type_name\n
\n\nFor a full list of options, use the @-help@ flag:\n\n
\n$ arvados-server cloudtest -help\nUsage:\n  -command string\n        Run an interactive shell command on the test instance when it boots\n  -config file\n        Site configuration file (default \"/etc/arvados/config.yml\")\n  -destroy-existing\n        Destroy any existing instances tagged with our InstanceSetID, instead of erroring out\n  -image-id string\n        Image ID to use when creating the test instance (if empty, use cluster config)\n  -instance-set-id value\n        InstanceSetID tag value to use on the test instance (default \"cloudtest-user@hostname.example\")\n  -instance-type string\n        Instance type to create (if empty, use cheapest type in config)\n  -pause-before-destroy\n        Prompt and wait before destroying the test instance\n
\n" }, { "path": "doc/admin/collection-managed-properties.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: Configuring collection's managed properties\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nCollection's managed properties allow a cluster administrator to enable some special behaviors regarding properties at creation & update times.\nThis page describes how to enable and configure these behaviors on the API server.\n\nh3. API Server configuration\n\nThe @Collections.ManagedProperties@ setting from the @config.yml@ file is used for enabling any of the following behaviors:\n\nh4. Pre-assigned property key & value\n\nFor every newly created collection, assign a predefined key/value pair if it isn't already passed at creation time:\n\n
\nCollections:\n  ManagedProperties:\n    foo: {Value: bar}\n
\n\nh4. Original owner UUID\n\nThis behavior will assign to a property key the UUID of the user who owns the collection's contaning project.\n\n
\nCollections:\n  ManagedProperties:\n    responsible_person_uuid: {Function: original_owner}\n
\n\nh4. Protected properties\n\nIf there's a need to prevent a non-admin user from modifying a specific property, even by its owner, the @Protected@ attribute can be set to @true@, like so:\n\n
\nCollections:\n  ManagedProperties:\n    sample_id: {Protected: true}\n
\n\nThis configuration won't assign a @sample_id@ property on collection creation, but if the user adds it to any collection, its value is protected from that point on.\n\nAnother use case would be to protect properties that were automatically assigned by the system:\n\n
\nCollections:\n  ManagedProperties:\n    responsible_person_uuid: {Function: original_owner, Protected: true}\n
\n\nIf missing, the @Protected@ attribute it’s assumed as being @false@ by default.\n\nh3. Supporting example scripts\n\nWhen enabling this feature, there may be pre-existing collections that won't have the managed properties just configured. The following script examples may be helpful to sync these older collections.\n\nFor the following examples we assume that the @responsible_person_uuid@ property is set as @{Function: original_owner, Protected: true}@.\n\nh4. List uuid/names of collections without @responsible_person_uuid@ property\n\nThe collection's managed properties feature assigns the configured properties to newly created collections. This means that previously existing collections won't get the default properties and if needed, they should be assigned manually.\n\nThe following example script outputs a listing of collection UUIDs and names of those collections that don't include the @responsible_person_uuid@ property.\n\n{% codeblock as python %}\n{% include 'admin_list_collections_without_property_py' %}\n{% endcodeblock %}\n\nh4. Update the @responsible_person_uuid@ property from nil to X in the project hierarchy rooted at P\n\nWhen enabling @responsible_person_uuid@, new collections will get this property's value set to the user who owns the root project where the collection is placed, but older collections won't have the property set. The following example script allows an administrator to set the @responsible_person_uuid@ property to collections below a certaing project hierarchy.\n\n{% codeblock as python %}\n{% include 'admin_set_property_to_collections_under_project_py' %}\n{% endcodeblock %}\n\nh4. Update the @responsible_person_uuid@ property from X to Y on all collections\n\nThis example can be useful to change responsibility from one user to another.\n\nPlease note that the following code should run with admin privileges, assuming that the managed property is @Protected@.\n\n{% codeblock as python %}\n{% include 'admin_update_collection_property_py' %}\n{% endcodeblock %}\n\n" }, { "path": "doc/admin/collection-versioning.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: Configuring collection versioning\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThis page describes how to enable and configure the collection versioning feature on the API server.\n\nh3. Configuration\n\nThere are 2 configuration settings in the @Collections@ section of @config.yml@ that control this feature.\n\n
\n    Collections:\n      # If true, enable collection versioning.\n      # When a collection's preserve_version field is true or the current version\n      # is older than the amount of seconds defined on PreserveVersionIfIdle,\n      # a snapshot of the collection's previous state is created and linked to\n      # the current collection.\n      CollectionVersioning: true\n\n      # This setting control the auto-save aspect of collection versioning, and can be set to:\n      #   0s = auto-create a new version on every update.\n      #  -1s = never auto-create new versions.\n      # > 0s = auto-create a new version when older than the specified number of seconds.\n      PreserveVersionIfIdle: 10s\n
\n\nNote that if you set @CollectionVersioning@ to @false@ after being enabled, old versions will still be accessible, but further changes will not be versioned.\n\nh3. Using collection versioning\n\n\"Discussed in the user guide\":{{site.baseurl}}/user/topics/collection-versioning.html\n" }, { "path": "doc/admin/config-urls.html.textile.liquid", "content": "---\nlayout: default\nnavsection: installguide\ntitle: InternalURLs and ExternalURL\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThe Arvados configuration is stored at @/etc/arvados/config.yml@. See the \"Configuration reference\":config.html for more detail.\n\nThe @Services@ section lists a number of Arvados services, each with an @InternalURLs@ and/or @ExternalURL@ configuration key. This document explains the precise meaning of these configuration keys, and how they are used by the Arvados services.\n\nThe @ExternalURL@ is the address where the service should be reachable by clients, both from inside and from outside the Arvados cluster. Some services do not expose an Arvados API, only Prometheus metrics. In that case, @ExternalURL@ is not used.\n\nThe keys under @InternalURLs@ are the URLs through which Arvados system components can connect to one another, including the reverse proxy (e.g. Nginx) that fronts Arvados services. The exception is the @Keepstore@ service, where clients on the local network connect directly to @Keepstore.InternalURLs@ (while clients from outside networks connect to @Keepproxy.ExternalURL@). If a service is not fronted by a reverse proxy, e.g. when its endpoint only exposes Prometheus metrics, the intention is that metrics are collected directly from the endpoints defined in @InternalURLs@.\n\nEach entry in the @InternalURLs@ section may also indicate a @ListenURL@ to determine the protocol, address/interface, and port where the service process will listen, in case the desired listening address differs from the @InternalURLs@ key itself -- for example, when passing internal traffic through a reverse proxy.\n\nIf the Arvados service lives behind a reverse proxy (e.g. Nginx), configuring the reverse proxy and the @InternalURLs@ and @ExternalURL@ values must be done in concert.\n\nh2. Overview\n\n
\ntable(table table-bordered table-condensed).\n|_.Service |_.ExternalURL required? |_.InternalURLs required?|_.InternalURLs must be reachable from other cluster nodes?|_.Note|\n|railsapi |no |yes|no ^1^|InternalURLs only used by Controller|\n|controller |yes |yes|yes ^2,4^|InternalURLs used by reverse proxy and container shell connections|\n|arvados-dispatch-cloud|no |yes|no ^3^|InternalURLs only used to expose Prometheus metrics|\n|arvados-dispatch-lsf|no |yes|no ^3^|InternalURLs only used to expose Prometheus metrics|\n|container web services|yes |no |no |controller's InternalURLs are used by reverse proxy (e.g. Nginx)|\n|git-ssh |yes |no |no ||\n|keepproxy |yes |yes|no ^2^|InternalURLs only used by reverse proxy (e.g. Nginx)|\n|keepstore |no |yes|yes |All clients connect to InternalURLs|\n|keep-balance |no |yes|no ^3^|InternalURLs only used to expose Prometheus metrics|\n|keep-web |yes |yes|yes ^5^|InternalURLs used by reverse proxy and container log API|\n|websocket |yes |yes|no ^2^|InternalURLs only used by reverse proxy (e.g. Nginx)|\n|workbench2 |yes |no|no ||\n
\n\n^1^ If @Controller@ runs on a different host than @RailsAPI@, the @InternalURLs@ will need to be reachable from the host that runs @Controller@.\n^2^ If the reverse proxy (e.g. Nginx) does not run on the same host as the Arvados service it fronts, the @InternalURLs@ will need to be reachable from the host that runs the reverse proxy.\n^3^ If the Prometheus metrics are not collected from the same machine that runs the service, the @InternalURLs@ will need to be reachable from the host that collects the metrics.\n^4^ If dispatching containers to HPC (Slurm/LSF) and there are multiple @Controller@ services, they must be able to connect to one another using their InternalURLs, otherwise the \"tunnel connections\":{{site.baseurl}}/architecture/hpc.html enabling \"container shell access\":{{site.baseurl}}/install/container-shell-access.html will not work.\n^5^ All URLs in @Services.WebDAV.InternalURLs@ must be reachable by all Controller services. Alternatively, each entry in @Services.Controller.InternalURLs@ must have a corresponding entry in @Services.WebDAV.InternalURLs@ with the same hostname.\n\nWhen @InternalURLs@ do not need to be reachable from other nodes, it is most secure to use loopback addresses as @InternalURLs@, e.g. @http://127.0.0.1:9005@.\n\nIt is recommended to use a split-horizon DNS setup where the hostnames specified in @ExternalURL@ resolve to an internal IP address from inside the Arvados cluster, and a publicly routed external IP address when resolved from outside the cluster. This simplifies firewalling and provides optimally efficient traffic routing. In a cloud environment where traffic that flows via public IP addresses is charged, using split horizon DNS can also avoid unnecessary expense.\n\nh2. Examples\n\nThe remainder of this document walks through a number of examples to provide more detail.\n\nh3. Keep-balance\n\nConsider this section for the @Keep-balance@ service:\n\n{% codeblock as yaml %}\n Keepbalance:\n InternalURLs:\n \"http://ip-10-0-1-233.internal:9005/\": {}\n{% endcodeblock %}\n\n@Keep-balance@ has an API endpoint, but it is only used to expose \"Prometheus\":https://prometheus.io metrics.\n\nThere is no @ExternalURL@ key because @Keep-balance@ does not have an Arvados API, no Arvados services need to connect to @Keep-balance@.\n\nThe value for @InternalURLs@ tells the @Keep-balance@ service to start up and listen on port 9005, if it is started on a host where @ip-10-0-1-233.internal@ resolves to a local IP address. If @Keep-balance@ is started on a machine where the @ip-10-0-1-233.internal@ hostname does not resolve to a local IP address, it would refuse to start up, because it would not be able to find a local IP address to listen on.\n\nIt is also possible to use IP addresses in @InternalURLs@, for example:\n\n{% codeblock as yaml %}\n Keepbalance:\n InternalURLs:\n \"http://127.0.0.1:9005/\": {}\n{% endcodeblock %}\n\nIn this example, @Keep-balance@ would start up and listen on port 9005 at the @127.0.0.1@ IP address. Prometheus would only be able to access the @Keep-balance@ metrics if it could reach that IP and port, e.g. if it runs on the same machine.\n\nFinally, it is also possible to listen on all interfaces, for example:\n\n{% codeblock as yaml %}\n Keepbalance:\n InternalURLs:\n \"http://0.0.0.0:9005/\": {}\n{% endcodeblock %}\n\nIn this case, @Keep-balance@ will listen on port 9005 on all IP addresses local to the machine.\n\nh3. Keepstore\n\nConsider this section for the @Keepstore@ service:\n\n{% codeblock as yaml %}\n Keepstore:\n InternalURLs:\n \"http://keep0.ClusterID.example.com:25107\": {}\n \"http://keep1.ClusterID.example.com:25107\": {}\n{% endcodeblock %}\n\nThere is no @ExternalURL@ key because @Keepstore@ is only accessed from inside the Arvados cluster. For access from outside, all traffic goes via @Keepproxy@.\n\nWhen @Keepstore@ is installed on the host where @keep0.ClusterID.example.com@ resolves to a local IP address, it will listen on port 25107 on that IP address. Likewise on the @keep1.ClusterID.example.com@ host. On all other systems, @Keepstore@ will refuse to start.\n\nh3. Keepproxy\n\nConsider this section for the @Keepproxy@ service:\n\n{% codeblock as yaml %}\n Keepproxy:\n ExternalURL: https://keep.ClusterID.example.com\n InternalURLs:\n \"http://localhost:25107\": {}\n{% endcodeblock %}\n\nThe @ExternalURL@ advertised is @https://keep.ClusterID.example.com@. The @Keepproxy@ service will start up on @localhost@ port 25107, however. This is possible because we also configure Nginx to terminate SSL and sit in front of the @Keepproxy@ service:\n\n
upstream keepproxy {\n  server                127.0.0.1:25107;\n}\n\nserver {\n  listen                  443 ssl;\n  server_name             keep.ClusterID.example.com;\n\n  proxy_connect_timeout   90s;\n  proxy_read_timeout      300s;\n  proxy_set_header        X-Real-IP $remote_addr;\n  proxy_http_version      1.1;\n  proxy_request_buffering off;\n  proxy_max_temp_file_size 0;\n\n  ssl_certificate     /YOUR/PATH/TO/cert.pem;\n  ssl_certificate_key /YOUR/PATH/TO/cert.key;\n\n  # Clients need to be able to upload blocks of data up to 64MiB in size.\n  client_max_body_size    64m;\n\n  location / {\n    proxy_pass            http://keepproxy;\n  }\n}\n
\n\nIf a client connects to the @Keepproxy@ service, it will talk to Nginx which will reverse proxy the traffic to the @Keepproxy@ service.\n\nh3. API server\n\nConsider this section for the @RailsAPI@ service:\n\n{% codeblock as yaml %}\n RailsAPI:\n InternalURLs:\n \"http://localhost:8004\": {}\n{% endcodeblock %}\n\nThere is no @ExternalURL@ defined because the @RailsAPI@ is not directly accessible and does not need to advertise a URL: all traffic to it flows via @Controller@, which is the only client that talks to it.\n\nThe @RailsAPI@ service is also a Rails application, and its listening host and port are set in the @arvados-railsapi.service@ unit definition:\n\n\n
# systemctl cat arvados-railsapi.service\n[...]\n[Service]\nEnvironment=PASSENGER_ADDRESS=localhost\nEnvironment=PASSENGER_PORT=8004\n[...]\n
\n\nSo then, why is there a need to specify @InternalURLs@ for the @RailsAPI@ service? It is there because this is how the @Controller@ service locates the @RailsAPI@ service it should talk to. Since this connection is internal to the Arvados cluster, @Controller@ uses @InternalURLs@ to find the @RailsAPI@ endpoint.\n\nh3. Controller\n\nConsider this section for the @Controller@ service:\n\n{% codeblock as yaml %}\n Controller:\n InternalURLs:\n \"https://ctrl-0.internal\":\n ListenURL: \"http://localhost:8003\"\n ExternalURL: \"https://ClusterID.example.com\"\n{% endcodeblock %}\n\nThe @ExternalURL@ advertised to clients is @https://ClusterID.example.com@. The @arvados-controller@ process will listen on @localhost@ port 8003. Other Arvados service processes in the cluster can connect to this specific controller instance, using the URL @https://ctrl-0.internal@. Container web service traffic at @https://*.containers.ClusterID.example.com@ is also handled by the same @arvados-controller@ process. Nginx is configured to sit in front of the @Controller@ service and terminate TLS:\n\n
\n# This is the port where nginx expects to contact arvados-controller.\nupstream controller {\n  server     localhost:8003  fail_timeout=10s;\n}\n\nserver {\n  # This configures the public https port that clients will actually connect to,\n  # the request is reverse proxied to the upstream 'controller'\n\n  listen       443 ssl;\n  server_name  ClusterID.example.com\n               ctrl-0.internal\n               *.containers.ClusterID.example.com;\n\n  ssl_certificate     /YOUR/PATH/TO/cert.pem;\n  ssl_certificate_key /YOUR/PATH/TO/cert.key;\n\n  # Refer to the comment about this setting in the passenger (arvados\n  # api server) section of your Nginx configuration.\n  client_max_body_size 128m;\n\n  location / {\n    proxy_pass               http://controller;\n    proxy_redirect           off;\n    proxy_connect_timeout    90s;\n    proxy_read_timeout       300s;\n    proxy_max_temp_file_size 0;\n    proxy_request_buffering  off;\n    proxy_buffering          off;\n    proxy_http_version       1.1;\n\n    proxy_set_header      Host              $http_host;\n    proxy_set_header      Upgrade           $http_upgrade;\n    proxy_set_header      Connection        \"upgrade\";\n    proxy_set_header      X-External-Client $external_client;\n    proxy_set_header      X-Forwarded-For   $proxy_add_x_forwarded_for;\n    proxy_set_header      X-Forwarded-Proto https;\n    proxy_set_header      X-Real-IP         $remote_addr;\n  }\n}\n
\n\nIf the host part of @ListenURL@ is ambiguous, in the sense that more than one system host is able to listen on that address (e.g., @localhost@), configure each host's startup scripts to set the environment variable @ARVADOS_SERVICE_INTERNAL_URL@ to the @InternalURLs@ key that will reach that host. In the example above, this would be @ARVADOS_SERVICE_INTERNAL_URL=https://ctrl-0.internal@.\n\nIf the cluster has just a single node running all of the Arvados server processes, configuration can be simplified:\n\n{% codeblock as yaml %}\n Controller:\n InternalURLs:\n \"http://localhost:8003\": {}\n ExternalURL: \"https://ClusterID.example.com\"\n{% endcodeblock %}\n" }, { "path": "doc/admin/config.html.textile.liquid", "content": "---\nlayout: default\nnavsection: installguide\ntitle: Configuration reference\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThe Arvados configuration is stored at @/etc/arvados/config.yml@\n\n{% codeblock as yaml %}\n{% include 'config_default_yml' %}\n{% endcodeblock %}\n" }, { "path": "doc/admin/controlling-container-reuse.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: Preventing container reuse\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nSometimes a container exited successfully but produced bad output, and re-running the workflow will cause it to re-use the bad container instead of running a new container. One way to deal with this is to re-run the entire workflow with reuse disabled. Another way is for the workflow author to tweak the input data or workflow so that on re-run it produces a distinct container request. However, for large or complex workflows both these options may be impractical.\n\nTo prevent an individual container from being reused in later workflows, an admin can manually change the state of the bad container record from @Complete@ to @Cancelled@. The following @arv@ command demonstrates how change a container state to @Cancelled@, where @xxxxx-xxxxx-xxxxxxxxxxxxxxx@ is the @UUID@ of the container:\n\n
arv container update -u xxxxx-xxxxx-xxxxxxxxxxxxxxx -c '{\"state\":\"Cancelled\"}'
\n" }, { "path": "doc/admin/diagnostics.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: Diagnostics\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThe @arvados-client diagnostics@ command exercises basic cluster functionality, and identifies some common installation and configuration problems. Especially after upgrading or reconfiguring Arvados or server/network infrastructure, it can be the quickest way to identify problems.\n\nh2. Using system privileges\n\nOn a server node, it is easiest to run the diagnostics command with system privileges. The word @sudo@ here instructs the @arvados-client@ command to load @Controller.ExternalURL@ and @SystemRootToken@ from @/etc/arvados/config.yml@ and use those credentials to run tests with system privileges.\n\nWhen run this way, diagnostics will also include \"health checks\":health-checks.html.\n\n
\n# arvados-client sudo diagnostics\n
\n\nh2. Using regular user privileges\n\nOn any node (server node, shell node, or a workstation outside the system network), you can also run diagnostics by setting the usual @ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ environment variables. Typically this is done with a regular user account.\n\n
\n$ export ARVADOS_API_HOST=zzzzz.arvadosapi.com\n$ export ARVADOS_API_TOKEN=xxxxxxxxxx\n$ arvados-client diagnostics\n
\n\nh2. Internal/external client detection\n\nThe diagnostics output indicates whether its client connection is categorized by the server as internal or external. If you run diagnostics automatically with cron or a monitoring tool, you can use the @-internal-client@ or @-external-client@ flag to specify how you _expect_ the client to be categorized, and the test will fail otherwise. Example:\n\n
\n# arvados-client sudo diagnostics -internal-client\n[...]\n\n--- cut here --- error summary ---\n\nERROR     60: checking internal/external client detection (11 ms): expecting internal=true external=false, but found internal=false external=true\n
\n\nh2(#container-options). Container-running options\n\nBy default, the @diagnostics@ command builds a custom Docker image containing a copy of its own binary, and uses that image to run diagnostic checks from inside an Arvados container. This can help detect problems like lack of network connectivity between containers and Arvados cluster services.\n\nThe default approach works well if the client host (i.e., the host where you invoke @arvados-client diagnostics@) meets certain conditions:\n* Docker is installed and working (so the diagnostics command can run @docker build@ and @docker save@).\n* Its hardware and kernel are similar to the cluster's compute instances (so the @arvados-client@ binary and the custom-built Docker image are compatible with the compute instances).\n* Network bandwidth supports uploading the Docker image (about 100 megabytes) in less than a minute.\n\nThe following options provide flexibility in case the default approach is not suitable.\n* @-priority=0@ skips the container-running part of the diagnostics suite.\n* @-docker-image=\"hello-world\"@ uses a tiny \"hello world\" image that is already embedded in the @arvados-client@ binary. This works even if the client host does not have any docker tools installed, and it minimizes the data transferred during the diagnostics suite. It provides less test coverage than the default option, but it will at least check that it is possible to run a container on the cluster.\n* @-docker-image=X@ (where @X@ is a Docker image name or a portable data hash) uses a Docker image that has already been uploaded to your Arvados cluster using @arv keep docker@. In this case the diagnostics tool will run a container with the command @echo {timestamp}@.\n* @-docker-image-from=NAME@ builds a custom Docker image on the fly as described above, but using the specified image as a base instead of the default @debian:slim-stable@ image. Note that the build recipe runs commands like @apt-get install [...] libfuse2 ca-certificates@ so only Debian-based base images are supported. For more flexibility, use one of the above @-docker-image=...@ options.\n* @-timeout=2m@ extends the time limit for each HTTP request made by the diagnostics suite, including the process of uploading a custom-built Docker image, to 2 minutes (the default HTTP request timeout is 10 seconds, and the default upload time limit is either the HTTP timeout or 1 minute, whichever is longer).\n\nh2. Example output\n\n
\n# arvados-client sudo diagnostics\nINFO       5: running health check (same as `arvados-server check`)\nINFO      10: getting discovery document from https://zzzzz.arvadosapi.com/discovery/v1/apis/arvados/v1/rest\nINFO      20: getting exported config from https://zzzzz.arvadosapi.com/arvados/v1/config\nINFO      30: getting current user record\nINFO      40: connecting to service endpoint https://keep.zzzzz.arvadosapi.com/\nINFO      41: connecting to service endpoint https://*.collections.zzzzz.arvadosapi.com/\nINFO      42: connecting to service endpoint https://download.zzzzz.arvadosapi.com/\nINFO      43: connecting to service endpoint wss://ws.zzzzz.arvadosapi.com/websocket\nINFO      44: connecting to service endpoint https://workbench.zzzzz.arvadosapi.com/\nINFO      45: connecting to service endpoint https://workbench2.zzzzz.arvadosapi.com/\nINFO      50: checking CORS headers at https://zzzzz.arvadosapi.com/\nINFO      51: checking CORS headers at https://keep.zzzzz.arvadosapi.com/d41d8cd98f00b204e9800998ecf8427e+0\nINFO      52: checking CORS headers at https://download.zzzzz.arvadosapi.com/\nINFO      60: checking internal/external client detection\nINFO      61: reading+writing via keep service at https://keep.zzzzz.arvadosapi.com:443/\nINFO      80: finding/creating \"scratch area for diagnostics\" project\nINFO      90: creating temporary collection\nINFO     100: uploading file via webdav\nINFO     110: checking WebDAV ExternalURL wildcard (https://*.collections.zzzzz.arvadosapi.com/)\nINFO     120: downloading from webdav (https://d41d8cd98f00b204e9800998ecf8427e-0.collections.zzzzz.arvadosapi.com/foo)\nINFO     121: downloading from webdav (https://d41d8cd98f00b204e9800998ecf8427e-0.collections.zzzzz.arvadosapi.com/sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412.tar)\nINFO     122: downloading from webdav (https://download.zzzzz.arvadosapi.com/c=d41d8cd98f00b204e9800998ecf8427e+0/_/foo)\nINFO     123: downloading from webdav (https://download.zzzzz.arvadosapi.com/c=d41d8cd98f00b204e9800998ecf8427e+0/_/sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412.tar)\nINFO     124: downloading from webdav (https://a15a27cbc1c7d2d4a0d9e02529aaec7e-128.collections.zzzzz.arvadosapi.com/sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412.tar)\nINFO     125: downloading from webdav (https://download.zzzzz.arvadosapi.com/c=zzzzz-4zz18-twitqma8mbvwydy/_/sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412.tar)\nINFO     130: getting list of virtual machines\nINFO     150: connecting to webshell service\nINFO     160: running a container\nINFO      ... container request submitted, waiting up to 10m for container to run\nINFO    9990: deleting temporary collection\n
\n" }, { "path": "doc/admin/dispatch.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: Cloud dispatcher\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThe @arvados-server@ program provides subcommands for accessing the \"cloud dispatcher's management API\":{{site.baseurl}}/api/dispatch.html interactively.\n\nh2. List instances\n\n@arvados-server instance list@\n\nDisplay a list of instances managed by the dispatcher.\n\nA placeholder @-@ in the @instance@ column indicates that an instance has been requested but the cloud provider has not yet returned a response with an instance ID.\n\nA placeholder @-@ in the @address@ column indicates that an instance has been requested but has not yet been assigned an IP address by the cloud provider.\n\nA placeholder @-@ in the @running-containers@ column indicates that the instance is not running any containers.\n\nUse the @-header@ flag to display column names.\n\n
# arvados-server instance list -header\ninstance\taddress\tstate\tidle-behavior\tconfig-type\tprovider-type\tprice\trunning-containers\ni-03d59cfcfacf307ff\t10.253.254.184\trunning\trun\tc5large\tc5.large\t0.085000\ttordo-dz642-r6fz90awybvywr6\ni-0df614e93e4170ae7\t10.253.254.157\tbooting\trun\tt3small\tt3.small\t0.020800\t-\n
\n\nh2. Drain instances\n\n
# arvados-server instance drain  [instance-id ...]
\n\nSet the indicated instances' idle behavior to @drain@. Containers currently running will be allowed to continue, but when each instance becomes idle, it will be shut down.\n\nh2. Hold instances\n\n
# arvados-server instance hold  [instance-id ...]
\n\nSet the indicated instances' idle behavior to @hold@. The instances will not be shut down automatically. Containers currently running will be allowed to continue, but no new containers will be scheduled.\n\nh2. Run instances\n\n
# arvados-server instance run  [instance-id ...]
\n\nSet the indicated instances' idle behavior to @run@ (the normal behavior). When the instances become idle, they will be eligible to run new containers. They will be shut down automatically when the configured idle threshold is reached.\n\nh2. Kill instances\n\n
# arvados-server instance kill [-reason \"...\"]  [instance-id ...]
\n\nShut down the indicated instances immediately, abandoning/failing any containers they are currently running.\n\nThe provided reason string will appear in the dispatcher’s log.\n" }, { "path": "doc/admin/federation.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: Configuring federation\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThis page describes how to enable and configure federation capabilities between clusters.\n\nAn overview on how this feature works is discussed in the \"architecture section\":{{site.baseurl}}/architecture/federation.html\n\nh2. Configuration\n\nTo enable a cluster to communicate with other clusters, some settings need to be added to the @config.yml@ file. Federated clusters are identified by listing the cluster-to-hostname mapping in the @RemoteClusters@ section.\n\nHere is an example of the settings that should be added to the @/etc/arvados/config.yml@ file:\n\n
\nClusters:\n  clsr1:\n    RemoteClusters:\n      clsr2:\n        Host: api.cluster2.example\n        Proxy: true\n\tActivateUsers: true\n      clsr3:\n        Host: api.cluster3.example\n        Proxy: true\n\tActivateUsers: false\n
\n\nSimilar settings should be added to @clsr2@ & @clsr3@ hosts, so that all clusters in the federation can talk to each other.\n\nThe @ActivateUsers@ setting indicates whether users from a given cluster are automatically activated or they require manual activation. User activation is covered in more detail in the \"user activation section\":{{site.baseurl}}/admin/user-management.html. In the current example, users from @clsr2@ would be automatically activated but users from @clsr3@ would require an admin to activate the account.\n\nNote: The @Proxy:@ variable is intended for future use, and should always be set to @true@.\n\nh2(#LoginCluster). User management\n\nA federation of clusters can be configured to use a separate user database per cluster, or delegate a central cluster to manage the database.\n\nh3. Peer federation\n\nIf clusters belong to separate organizations, each cluster will have its own user database for the members of that organization. Through federation, a user from one organization can be granted access to the cluster of another organization. The admin of the second cluster can control access on a individual basis by choosing to activate or deactivate accounts from other organizations.\n\nh3. Centralized (LoginCluster) federation\n\nIf all clusters belong to the same organization, and users in that organization should have access to all the clusters, user management can be simplified by setting the @LoginCluster@ which manages the user database used by all other clusters in the federation. To do this, choose one cluster in the federation which will be the 'login cluster'. Set the @Login.LoginCluster@ configuration value on all clusters in the federation to the cluster id of the login cluster. After setting @LoginCluster@, restart arvados-api-server and arvados-controller.\n\n
\nClusters:\n  clsr2:\n    Login:\n      LoginCluster: clsr1\n
\n\nThe @LoginCluster@ configuration redirects all user logins to the LoginCluster, and the LoginCluster will issue API tokens which will be accepted by the federation. Users are activated or deactivated across the entire federation based on their status on the login cluster.\n\nNote: tokens issued by the login cluster need to be periodically re-validated when used on other clusters in the federation. The period between revalidation attempts is configured with @Login.RemoteTokenRefresh@. The default is 5 minutes. A longer period reduces overhead from validating tokens, but means it may take longer for other clusters to notice when a token has been revoked or a user has changed status (being activated/deactivated, admin flag changed).\n\nTo migrate users of existing clusters with separate user databases to use a single LoginCluster, a script @arv-federation-migrate@ is available in @contrib/arvados-bootstrap@.\n\nh2. Groups\n\nIn order for a user to see (and be able to share with) other users, the admin needs to create a \"can_read\" permission link from the user to either the \"All users\" group, or another group that grants visibility to a subset of users.\n\nIn a peer federation, this means that for a user that has joined a second cluster, that user needs to be added to the \"All users\" group on the second cluster as well, to be able to share with other users.\n\nIn a LoginCluster federation, all visibility of users to share with other users is set by the LoginCluster. It is not necessary to add users to \"All users\" on the other clusters.\n\nh3. Trusted clients\n\nWhen a cluster is configured to use a LoginCluster, the login flow goes to the LoginCluster to log in and issue a token, then returns the user to the starting workbench. In this case, you want to configure the LoginCluster to \"trust\" the workbench instances associated with the other clusters.\n\n
\nClusters:\n  clsr1:\n    Login:\n      TrustedClients:\n        \"https://workbench.cluster2.example\": {}\n        \"https://workbench2.cluster2.example\": {}\n        \"https://workbench.cluster3.example\": {}\n        \"https://workbench2.cluster3.example\": {}\n
\n\nh2. Testing\n\nFollowing the above example, let's suppose @clsr1@ is our \"home cluster\", that is to say, we use our @clsr1@ user account as our federated identity and both @clsr2@ and @clsr3@ remote clusters are set up to allow users from @clsr1@ and to auto-activate them. The first thing to do would be to log into a remote workbench using the local user token. This can be done following these steps:\n\n1. Log into the local workbench and get the user token\n2. Visit the remote workbench specifying the local user token by URL: @https://workbench.cluster2.example?api_token=token_from_clsr1@\n3. You should now be logged into @clsr2@ with your account from @clsr1@\n\nTo further test the federation setup, you can create a collection on @clsr2@, uploading some files and copying its UUID. Next, logged into a shell node on your home cluster you should be able to get that collection by running:\n\n
\nuser@clsr1:~$ arv collection get --uuid clsr2-xvhdp-xxxxxxxxxxxxxxx\n
\n\nThe returned collection metadata should show the local user's uuid on the @owner_uuid@ field. This tests that the @arvados-controller@ service is proxying requests correctly.\n\nOne last test may be performed, to confirm that the @keepstore@ services also recognize remote cluster prefixes and proxy the requests. You can ask for the previously created collection using any of the usual tools, for example:\n\n
\nuser@clsr1:~$ arv-get clsr2-xvhdp-xxxxxxxxxxxxxxx/uploaded_file .\n
\n" }, { "path": "doc/admin/group-management.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: Role group management at the CLI\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThis page describes how to manage groups at the command line. You should be familiar with the \"permission system\":{{site.baseurl}}/api/permission-model.html .\n\nh2. Create a role group\n\nUser groups are entries in the \"groups\" table with @\"group_class\": \"role\"@.\n\n
\narv group create --group '{\"name\": \"My new group\", \"group_class\": \"role\"}'\n
\n\nh2(#add). Add a user to a role group\n\nThere are two separate permissions associated with group membership. The first link grants the user @can_manage@ permission to manage things that the group can manage. The second link grants permission for other users of the group to see that this user is part of the group.\n\n
\narv link create --link '{\n  \"link_class\": \"permission\",\n  \"name\": \"can_manage\",\n  \"tail_uuid\": \"the_user_uuid\",\n  \"head_uuid\": \"the_group_uuid\"}'\n\narv link create --link '{\n  \"link_class\": \"permission\",\n  \"name\": \"can_read\",\n  \"tail_uuid\": \"the_group_uuid\",\n  \"head_uuid\": \"the_user_uuid\"}'\n
\n\nA user can also be given read-only access to a group. In that case, the first link should be created with @can_read@ instead of @can_manage@.\n\nh2. List role groups\n\n
\narv group list --filters '[[\"group_class\", \"=\", \"role\"]]'\n
\n\nh2. List members of a role group\n\nUse the command \"jq\":https://stedolan.github.io/jq/ to extract the tail_uuid of each permission link which has the user uuid.\n\n
\narv link list --filters '[[\"link_class\", \"=\", \"permission\"],\n  [\"head_uuid\", \"=\", \"the_group_uuid\"]]' | jq .items[].tail_uuid\n
\n\nh2(#share-project). Share a project with a role group\n\nMembers of the role group will have access to the project based on their level of access to the role group.\n\n
\narv link create --link '{\n  \"link_class\": \"permission\",\n  \"name\": \"can_manage\",\n  \"tail_uuid\": \"the_group_uuid\",\n  \"head_uuid\": \"the_project_uuid\"}'\n
\n\nA project can also be shared read-only. In that case, the link @name@ should be @can_read@ instead of @can_manage@.\n\nh2. List things shared with the group\n\nUse the command \"jq\":https://stedolan.github.io/jq/ to extract the head_uuid of each permission link which has the object uuid.\n\n
\narv link list --filters '[[\"link_class\", \"=\", \"permission\"],\n  [\"tail_uuid\", \"=\", \"the_group_uuid\"]]' | jq .items[].head_uuid\n
\n\nh2(#stop-sharing-project). Stop sharing a project with a group\n\nThis will remove access for members of the group.\n\nThe first step is to find the permission link objects. The second step is to delete them.\n\n
\narv --format=uuid link list --filters '[[\"link_class\", \"=\", \"permission\"],\n  [\"tail_uuid\", \"=\", \"the_group_uuid\"], [\"head_uuid\", \"=\", \"the_project_uuid\"]]'\n\narv link delete --uuid each_link_uuid\n
\n\nh2. Remove user from a role group\n\nThe first step is to find the permission link objects. The second step is to delete them.\n\n
\narv --format=uuid link list --filters '[[\"link_class\", \"=\", \"permission\"],\n  [\"tail_uuid\", \"in\", [\"the_user_uuid\", \"the_group_uuid\"]],\n  [\"head_uuid\", \"in\", [\"the_user_uuid\", \"the_group_uuid\"]]'\n\narv link delete --uuid each_link_uuid\n
\n" }, { "path": "doc/admin/health-checks.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: Health checks\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nHealth check endpoints are found at @/_health/ping@ on many Arvados services. The purpose of the health check is to offer a simple method of determining if a service can be reached and allow the service to self-report any problems, suitable for integrating into operational alert systems.\n\nTo access health check endpoints, services must be configured with a \"management token\":management-token.html .\n\nHealth check endpoints return a JSON object with the field @health@. This has a value of either @OK@ or @ERROR@. On error, it may also include a field @error@ with additional information. Examples:\n\n
\n{\n  \"health\": \"OK\"\n}\n
\n\n
\n{\n  \"health\": \"ERROR\"\n  \"error\": \"Inverted polarity in the warp core\"\n}\n
\n\nh2. Health check aggregator\n\nThe service @arvados-health@ performs health checks on all configured services and returns a single value of @OK@ or @ERROR@ for the entire cluster. It exposes the endpoint @/_health/all@ .\n\nThe healthcheck aggregator uses the @Services@ section of the cluster-wide @config.yml@ configuration file.\n\nh2. Health check command\n\nThe @arvados-server check@ command is another way to perform the same health checks as the health check aggregator service. It does not depend on the aggregator service.\n\nIf all checks pass, it writes @health check OK@ to stderr (unless the @-quiet@ flag is used) and exits 0. Otherwise, it writes error messages to stderr and exits with error status.\n\n@arvados-server check -yaml@ outputs a YAML document on stdout with additional details about each service endpoint that was checked.\n\n{% codeblock as yaml %}\nChecks:\n \"arvados-api-server+http://localhost:8004/_health/ping\":\n ClockTime: \"2024-12-13T14:38:25Z\"\n ConfigSourceSHA256: 5a2b21ce0aeeeebcaf623329871b4628772446d4684ab0f89da4a2cbc7b3f17c\n ConfigSourceTimestamp: \"2024-12-12T11:14:06.487848-05:00\"\n HTTPStatusCode: 200\n Health: OK\n Response:\n health: OK\n ResponseTime: 0.051136\n Server: nginx/1.26.1 + Phusion Passenger(R) 6.0.23\n Version: 3.0.0\n \"arvados-controller+http://localhost:8003/_health/ping\":\n ClockTime: \"2024-12-13T14:38:25Z\"\n ConfigSourceSHA256: 5a2b21ce0aeeeebcaf623329871b4628772446d4684ab0f89da4a2cbc7b3f17c\n ConfigSourceTimestamp: \"2024-12-12T11:14:06.487848-05:00\"\n HTTPStatusCode: 200\n Health: OK\n Response:\n health: OK\n ResponseTime: 0.014869\n Server: \"\"\n Version: 3.0.0 (go1.21.10)\n# ...\n{% endcodeblock %}\n" }, { "path": "doc/admin/index.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: \"Arvados admin overview\"\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThis section describes how to administer an Arvados cluster. Cluster admins should already be familiar with the \"Arvados architecture.\":{{site.baseurl}}/architecture/index.html For instructions on installing and configuring an Arvados cluster, see the \"install guide.\":{{site.baseurl}}/install/index.html\n" }, { "path": "doc/admin/inspect.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: Inspecting active requests\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nMost Arvados services publish a snapshot of HTTP requests currently being serviced at @/_inspect/requests@. This can be useful for troubleshooting slow requests and understanding high server load conditions.\n\nTo access snapshots, services must be configured with a \"management token\":management-token.html. When accessing this endpoint, prefix the management token with @\"Bearer \"@ and supply it in the @Authorization@ request header.\n\nIn an interactive setting, use the @jq@ tool to format the JSON response.\n\n
curl -sfH \"Authorization: Bearer your_management_token_goes_here\" \"https://0.0.0.0:25107/_inspect/requests\" | jq .\n
\n\ntable(table table-bordered table-condensed table-hover){width:40em}.\n|_. Component|_. Provides @/_inspect/requests@ endpoint|\n|arvados-api-server||\n|arvados-controller|✓|\n|arvados-dispatch-cloud|✓|\n|arvados-dispatch-lsf|✓|\n|arvados-ws|✓|\n|composer||\n|keepproxy|✓|\n|keepstore|✓|\n|keep-balance|✓|\n|keep-web|✓|\n|workbench2||\n\nh2. Report fields\n\nMost fields are self explanatory.\n\nThe @Host@ field reports the virtual host specified in the incoming HTTP request.\n\nThe @RemoteAddr@ field reports the source of the incoming TCP connection, which is typically a local address associated with the Nginx proxy service.\n\nThe @XForwardedFor@ field reports the value of the \"@X-Forwarded-For@ header\":https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/X-Forwarded-For in the request.\n\nThe @Elapsed@ field reports the number of seconds since the incoming HTTP request headers were received.\n\nh2. Example response\n\n
\n[\n  {\n    \"RequestID\": \"req-1vzzj6nwrki0rd2hj08a\",\n    \"Method\": \"GET\",\n    \"Host\": \"tordo.arvadosapi.com\",\n    \"URL\": \"/arvados/v1/groups?order=name+asc&filters=[[%22owner_uuid%22,%22%3D%22,%22zzzzz-tpzed-aaaaaaaaaaaaaaa%22],[%22group_class%22,%22in%22,[%22project%22,%22filter%22]]]\",\n    \"RemoteAddr\": \"127.0.0.1:55822\",\n    \"XForwardedFor\": \"192.168.0.111, 10.0.0.123\",\n    \"Elapsed\": 0.006363228\n  },\n  {\n    \"RequestID\": \"req-1wrof2b2wlj5s1rao4u3\",\n    \"Method\": \"GET\",\n    \"Host\": \"tordo.arvadosapi.com\",\n    \"URL\": \"/arvados/v1/users/current\",\n    \"RemoteAddr\": \"127.0.0.1:55814\",\n    \"XForwardedFor\": \"192.168.0.222, 10.0.0.123\",\n    \"Elapsed\": 0.04796585\n  }\n]\n
\n" }, { "path": "doc/admin/keep-balance.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: Balancing Keep servers\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThis page describes how to balance keepstore servers using keep-balance. Keep-balance creates new copies of under-replicated blocks, deletes excess copies of over-replicated and unreferenced blocks, and moves blocks to better positions (e.g. after adding new keepstore servers) so clients find them faster.\n\nSee \"the Keep-balance install docs\":{{site.baseurl}}/install/install-keep-balance.html for installation instructions.\n\nh3. Data deletion\n\nThe keep-balance service determines which blocks are candidates for deletion and instructs the keepstore to move those blocks to the trash. When a block is newly written, it is protected from deletion for the duration in @BlobSigningTTL@. During this time, it cannot be trashed or deleted.\n\nIf keep-balance instructs keepstore to trash a block which is older than @BlobSigningTTL@, and @BlobTrashLifetime@ is non-zero, the block will be moved to \"trash\". A block which is in the trash is no longer accessible by read requests, but has not yet been permanently deleted. Blocks which are in the trash may be recovered using the \"untrash\" API endpoint. Blocks are permanently deleted after they have been in the trash for the duration in @BlobTrashLifetime@.\n\nKeep-balance is also responsible for balancing the distribution of blocks across keepstore servers by asking servers to pull blocks from other servers (as determined by their \"storage class\":{{site.baseurl}}/admin/storage-classes.html and \"rendezvous hashing order\":{{site.baseurl}}/architecture/keep-clients.html#rendezvous). Pulling a block makes a copy. If a block is overreplicated (i.e. there are excess copies) after pulling, it will be subsequently trashed and deleted on the original server, subject to @BlobTrash@ and @BlobTrashLifetime@ settings.\n\nh3. Scanning\n\nBy default, keep-balance operates periodically, i.e. do a scan/balance operation, sleep, repeat.\n\nThe @Collections.BalancePeriod@ value in @/etc/arvados/config.yml@ determines the interval between start times of successive scan/balance operations. If an operation takes longer than the @Collections.BalancePeriod@, the next operation will follow it immediately. If SIGUSR1 is received during an idle period between operations, the next operation will start immediately.\n\nKeep-balance can also be run with the @-once@ flag to do a single scan/balance operation and then exit. The exit code will be zero if the operation was successful.\n\nh3. Additional configuration\n\nFor configuring resource usage tuning and lost block reporting, please see the @Collections.BlobMissingReport@, @Collections.BalanceCollectionBatch@, @Collections.BalanceCollectionBuffers@ option in the \"default config.yml file\":{{site.baseurl}}/admin/config.html.\n\nThe @Collections.BalancePullLimit@ and @Collections.BalanceTrashLimit@ configuration entries determine the maximum number of pull and trash operations keep-balance will attempt to apply on each keepstore server. If both values are zero, keep-balance will operate in \"dry run\" mode, where all changes are computed but none are committed.\n\nh3. Limitations\n\nKeep-balance does not attempt to discover whether committed pull and trash requests ever get carried out -- only that they are accepted by the Keep services. If some services are full, new copies of under-replicated blocks might never get made, only repeatedly requested.\n" }, { "path": "doc/admin/keep-faster-gc-s3.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: \"Faster garbage collection in S3\"\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nWhen there is a large number of unneeded blocks stored in an S3 bucket, particularly when using @PrefixLength: 0@, the speed of garbage collection can be severely limited by AWS API rate limits and Arvados's multi-step trash/delete process.\n\nThe multi-step trash/delete process can be short-circuited by setting @BlobTrashLifetime@ to zero and enabling @UnsafeDelete@ on S3-backed volumes. However, on an actively used cluster such a configuration *can result in data loss* in the rare case where a given block is trashed and then rewritten soon afterward, and S3 processes the write and delete requests in the opposite order.\n\nThe following steps can be used to temporarily disable writes on an S3 bucket to enable faster garbage collection without data loss or service interruption. Note that garbage collection on other S3 volumes will be temporarily disabled during this procedure.\n# Create a new S3 bucket and configure it as an additional volume (this step may be skipped if the configuration already has enough writable volumes that clients will still be able to write blocks while the target volume is read-only). We recommend using @PrefixLength: 3@ for the new volume because this results in a much higher rate limit for I/O and garbage collection operations compared to the default @PrefixLength: 0@. If the target volume configuration specifies @StorageClasses@, use the same values for the new volume.\n# Shut down the @keep-balance@ service.\n# Update your configuration as follows: 
\n  Collections:\n    BlobTrashLifetime: 0\n    BalancePullLimit: 0\n  [...]\n  Volumes:\n    target-volume-uuid:\n      ReadOnly: true\n      AllowTrashWhenReadOnly: true\n      DriverParameters:\n        UnsafeDelete: true\n
 Note that @BlobTrashLifetime: 0@ instructs keepstore to delete unneeded blocks outright (bypassing the recoverable trash phase); however, in this mode it will normally not trash any blocks at all on an S3 volume due to the safety issue mentioned above, unless the volume is configured with @UnsafeDelete: true@.\n# Restart all @keepstore@ services with the updated configuration.\n# Start the @keep-balance@ service.\n# Objects will be deleted immediately instead of being first copied to trash on the S3 volume, which should significantly speed up cleanup of trashed objects. Monitor progress by watching @keep-balance@ logs and metrics. When garbage collection is complete, keep-balance logs will show an empty changeset: 
zzzzz-bi6l4-0123456789abcdef (keep0.zzzzz.arvadosapi.com:25107, disk): ChangeSet{Pulls:0, Trashes:0}
\n# Remove the @UnsafeDelete@ configuration entry on the target volume.\n# Remove the @BlobTrashLifetime@ configuration entry (or restore it to its previous value).\n# If the target volume has @PrefixLength: 0@ and the new volume has @PrefixLength: 3@, skip the next two steps: new data will be stored on the new volume, some existing data will be moved automatically to other volumes, and some will be left on the target volume as long as it's needed.\n# If you want to resume writing new data to the target volume, revert to @ReadOnly: false@ and @AllowTrashWhenReadOnly: false@ on the target volume.\n# If you want to stop writing new data to the newly created volume, set @ReadOnly: true@ and @AllowTrashWhenReadOnly: true@ on the new volume.\n# Remove the @BalancePullLimit@ configuration entry (or restore its previous value), and restart @keep-balance@.\n# Restart all @keepstore@ services with the updated configuration.\n" }, { "path": "doc/admin/keep-measuring-deduplication.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: \"Measuring deduplication\"\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThe @arvados-client@ tool can be used to generate a deduplication report across an arbitrary number of collections. It can be installed from packages (@apt install arvados-client@ or @dnf install arvados-client@).\n\nh2(#syntax). Syntax\n\n\n
~$ arvados-client deduplication-report -h\nUsage:\n  arvados-client deduplication-report [options ...]   ...\n\n  arvados-client deduplication-report [options ...] , \\\n     , ...\n\n  This program analyzes the overlap in blocks used by 2 or more collections. It\n  prints a deduplication report that shows the nominal space used by the\n  collections, as well as the actual size and the amount of space that is saved\n  by Keep's deduplication.\n\n  The list of collections may be provided in two ways. A list of collection\n  uuids is sufficient. Alternatively, the PDH for each collection may also be\n  provided. This is will greatly speed up operation when the list contains\n  multiple collections with the same PDH.\n\n  Exit status will be zero if there were no errors generating the report.\n\nExample:\n\n  Use the 'arv' and 'jq' commands to get the list of the 100\n  largest collections and generate the deduplication report:\n\n  arv collection list --order 'file_size_total desc' --limit 100 | \\\n    jq -r '.items[] | [.portable_data_hash,.uuid] |@csv' | \\\n    sed -e 's/\"//g'|tr '\\n' ' ' | \\\n    xargs arvados-client deduplication-report\n\nOptions:\n  -log-level string\n      logging level (debug, info, ...) (default \"info\")\n\n
\n\n\nThe usual environment variables (@ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@) need to be set for the deduplication report to be be generated. To get cluster-wide results, an admin token will need to be supplied. Users can also run this report, but only collections their token is able to read will be included.\n\nExample output (with uuids and portable data hashes obscured) from a small Arvados cluster:\n\n\n
~$ arv collection list --order 'file_size_total desc' --limit 10 | jq -r '.items[] | [.portable_data_hash,.uuid] |@csv' |sed -e 's/\"//g'|tr '\\n' ' ' |xargs arvados-client deduplication-report\nCollection _____-_____-_______________: pdh ________________________________+5003343; nominal size 7382073267640 (6.7 TiB); file count 2796\nCollection _____-_____-_______________: pdh ________________________________+4961919; nominal size 6989909625775 (6.4 TiB); file count 5592\nCollection _____-_____-_______________: pdh ________________________________+1903643; nominal size 2677933564052 (2.4 TiB); file count 2796\nCollection _____-_____-_______________: pdh ________________________________+1903643; nominal size 2677933564052 (2.4 TiB); file count 2796\nCollection _____-_____-_______________: pdh ________________________________+137710; nominal size 191858151583 (179 GiB); file count 201\nCollection _____-_____-_______________: pdh ________________________________+137636; nominal size 191858101962 (179 GiB); file count 200\nCollection _____-_____-_______________: pdh ________________________________+135350; nominal size 191715427388 (178 GiB); file count 201\nCollection _____-_____-_______________: pdh ________________________________+135276; nominal size 191715384167 (178 GiB); file count 200\nCollection _____-_____-_______________: pdh ________________________________+135350; nominal size 191707276684 (178 GiB); file count 201\nCollection _____-_____-_______________: pdh ________________________________+135276; nominal size 191707233463 (178 GiB); file count 200\n\nCollections:                              10\nNominal size of stored data:  20878411596766 bytes (19 TiB)\nActual size of stored data:   17053104444050 bytes (16 TiB)\nSaved by Keep deduplication:   3825307152716 bytes (3.5 TiB)\n\n\n
\n\n" }, { "path": "doc/admin/keep-recovering-data.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: \"Recovering data\"\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nArvados has several features to prevent accidental loss or deletion of data, but accidents can happen. This page lays out the options to recover deleted or overwritten collections.\n\nFor more detail on the data lifecycle in Arvados, see the \"Data lifecycle\":{{ site.baseurl }}/architecture/keep-data-lifecycle.html page.\n\nh2(#check_the_trash). Check the trash\n\nWhen a collection is deleted, it is moved to the trash. It will remain there for the duration of @Collections.DefaultTrashLifetime@, and it can be untrashed via workbench or with the cli tools, as described in \"Recovering trashed collections\":{{ site.baseurl }}/user/tutorials/tutorial-keep-collection-lifecycle.html#trash-recovery.\n\nh2(#check_other_collections). Check for other collections with the same PDH\n\nMultiple collections may share a _portable data hash_, i.e. have the same contents. If another collection exists with the same portable data hash, recovering data is not necessary, everything is still stored in Keep. A new copy of the collection can be made to make the data available in the correct project and with the correct permissions.\n\nh2(#check_collection_versioning). Consider collection versioning\n\nArvados supports collection versioning. If it has not been \"disabled\":{{ site.baseurl }}/admin/collection-versioning.html on your cluster, the deleted collection may be recoverable from an older version. See \"Using collection versioning\":{{ site.baseurl }}/user/topics/collection-versioning.html for details.\n\nh2(#recover_collection). Recovering collections\n\nWhen all the above options fail, it may still be possible to recover a collection that has been deleted.\n\nTo recover a collection the manifest is required. Arvados has a built-in audit log, which consists of a row added to the \"logs\" table in the PostgreSQL database each time an Arvados object is created, modified, or deleted. Collection manifests are included, unless they are listed in the @AuditLogs.UnloggedAttributes@ configuration parameter. The audit log is retained for up to @AuditLogs.MaxAge@.\n\nIn some cases, it is possible to recover files that have been lost by modifying or deleting a collection.\n\nPossibility of recovery depends on many factors, including:\n* Whether the collection manifest is still available, e.g., in an audit log entry\n* Whether the data blocks are also referenced by other collections\n* Whether the data blocks have been unreferenced long enough to be marked for deletion/trash by keep-balance\n* Blob signature TTL, trash lifetime, trash check interval, and other config settings\n\nTo attempt recovery of a previous version of a deleted/modified collection, use the @arvados-server recover-collection@ command. It should be run on one of your server nodes where the @arvados-server@ package is installed and the @/etc/arvados/config.yml@ file is up to date.\n\nSpecify the collection you want to recover by passing either the UUID of an audit log entry, or a file containing the manifest.\n\nIf recovery is successful, the @recover-collection@ program saves the recovered data a new collection belonging to the system user, and prints the new collection's UUID on stdout.\n\n
\n# arvados-server recover-collection 9tee4-57u5n-nb5awmk1pahac2t\nINFO[2020-06-05T19:52:29.557761245Z] loaded log entry                              logged_event_time=\"2020-06-05 16:48:01.438791 +0000 UTC\" logged_event_type=update old_collection_uuid=9tee4-4zz18-1ex26g95epmgw5w src=9tee4-57u5n-nb5awmk1pahac2t\nINFO[2020-06-05T19:52:29.642145127Z] recovery succeeded                            UUID=9tee4-4zz18-5trfp4k4xxg97f1 src=9tee4-57u5n-nb5awmk1pahac2t\n9tee4-4zz18-5trfp4k4xxg97f1\nINFO[2020-06-05T19:52:29.644699436Z] exiting\n
\n\nIn this example, the original data has been restored and saved in a new collection with UUID @9tee4-4zz18-5trfp4k4xxg97f1@.\n\nFor more options, run @arvados-server recover-collection -help@.\n\nh2(#untrashing_lost_blocks). Untrashing lost blocks\n\nIn some cases it is possible to recover data blocks that were trashed erroneously by @keep-balance@ (e.g. due to an install/config error).\n\nIf you suspect blocks have been trashed erroneously, you should immediately:\n\n* On all keepstore servers: set @BlobTrashCheckInterval@ to a long time like 2400h\n* On all keepstore servers: restart keepstore\n* Stop the keep-balance service\n\nWhen you think you have corrected the underlying problem, you should:\n\n* Set @Collections.BlobMissingReport@ to a suitable value (perhaps \"/tmp/keep-balance-lost-blocks.txt\").\n* Start @keep-balance@\n* After @keep-balance@ completes its first sweep, inspect /tmp/keep-balance-lost-blocks.txt. If it's not empty, you can request all keepstores to untrash any blocks that are still recoverable with a script like this:\n\n\n
\n#!/bin/bash\nset -e\n\n# see Client.AuthToken in /etc/arvados/keep-balance/keep-balance.yml\ntoken=xxxxxxx-your-system-auth-token-xxxxxxx\n\n# all keep server hostnames\nhosts=(keep0 keep1 keep2 keep3 keep4 keep5)\n\nwhile read hash pdhs; do\n  echo \"${hash}\"\n  for h in ${hosts[@]}; do\n    if curl -fgs -H \"Authorization: Bearer $token\" -X PUT \"http://${h}:25107/untrash/$hash\"; then\n      echo \"${hash} ok ${host}\"\n    fi\n  done\ndone < /tmp/keep-balance-lost-blocks.txt\n\n
\n\n\nAny blocks which were successfully untrashed can be removed from the list of blocks and collections which need to be recovered.\n\nh2(#regenerating_lost_blocks). Regenerating lost blocks\n\nFor blocks which were trashed long enough ago that they've been deleted, it may be possible to regenerate them by rerunning the workflows which generated them. To do this, the process is:\n\n* Delete the affected collections so that job reuse doesn't attempt to reuse them (it's likely that if one block is missing, they all are, so they're unlikely to contain any useful data)\n* Resubmit any container requests for which you want the output collections regenerated\n\nThe Arvados repository contains a tool that can be used to generate a report to help with this task at \"arvados/tools/keep-xref/keep-xref.py\":https://github.com/arvados/arvados/blob/main/tools/keep-xref/keep-xref.py\n" }, { "path": "doc/admin/link-accounts.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: \"Link user accounts\"\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nIf a user needs to log in to Arvados with a upstream account or provider, they may end up with two Arvados user accounts. If the user still has the ability to log in with the old account, they can use the \"self-serve account linking\":{{site.baseurl}}/user/topics/link-accounts.html feature of workbench. However, if the user does not have the ability to log in with both upstream accounts, the admin can also link the accounts using the command line.\n\nbq. NOTE: self-serve account linking is currently not supported on LoginCluster federations and needs to be performed manually by the site admin.\n\nh3. Step 1: Determine user uuids\n\nUser uuids can be determined by browsing workbench or using @arv user list@ at the command line.\n\nAccount linking works by recording in the database that a log in to the \"old\" account should redirected and treated as a login to the \"new\" account.\n\nThe \"old\" account is the Arvados account that will be redirected.\n\nThe \"new\" account is the user that the \"old\" account is redirected to. As part of account linking any Arvados records owned by the \"old\" account is also transferred to the \"new\" account.\n\nCounter-intuitively, if you do not want the account uuid of the user to change, the \"new\" account should be the pre-existing account, and the \"old\" account should be the redundant second account that was more recently created. This means \"old\" and \"new\" are opposite from their expected chronological meaning. In this case, the use of \"old\" and \"new\" reflect the direction of transfer of ownership -- the login was associated with the \"old\" user account, but will be associated with the \"new\" user account.\n\nIn the example below, @zzzzz-tpzed-3kz0nwtjehhl0u4@ is the \"old\" account (the pre-existing account we want to keep) and @zzzzz-tpzed-fr97h9t4m5jffxs@ is the \"new\" account (the redundant account we want to merge into the existing account).\n\nh3. Step 2: Create a project\n\nCreate a project owned by the \"new\" account that will hold any data owned by the \"old\" account.\n\n
\n$ arv --format=uuid group create --group '{\"group_class\": \"project\", \"name\": \"Data from old user\", \"owner_uuid\": \"zzzzz-tpzed-fr97h9t4m5jffxs\"}'\nzzzzz-j7d0g-mczqiguhil13083\n
\n\nh3. Step 3: Merge \"old\" user to \"new\" user\n\nThe @user merge@ method redirects login and reassigns data from the \"old\" account to the \"new\" account.\n\n
\n$ arv user merge  --redirect-to-new-user \\\n  --old-user-uuid=zzzzz-tpzed-3kz0nwtjehhl0u4 \\\n  --new-user-uuid=zzzzz-tpzed-fr97h9t4m5jffxs \\\n  --new-owner-uuid=zzzzz-j7d0g-mczqiguhil13083 \\\n
\n\nNote that authorization credentials (API tokens, ssh keys) are also transferred to the \"new\" account, so credentials used to access the \"old\" account work with the \"new\" account.\n" }, { "path": "doc/admin/logging.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: Logging\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nMost Arvados services write JSON-format structured logs to stderr, which can be parsed by any operational tools that support JSON.\n\nh2. Request ids\n\nUsing a distributed system with several services working together sometimes makes it difficult to find the root cause of errors, as one single client request usually means several different requests to more than one service.\n\nTo deal with this difficulty, Arvados creates a request ID that gets carried over different services as the requests take place. This ID has a specific format and it's comprised of the prefix \"@req-@\" followed by 20 random alphanumeric characters:\n\n
req-frdyrcgdh4rau1ajiq5q
\n\nThis ID gets propagated via an HTTP @X-Request-Id@ header, and gets logged on every service.\n\nh3. API Server error reporting and logging\n\nIn addition to providing the request ID on every HTTP response, the API Server adds it to every error message so that all clients show enough information to the user to be able to track a particular issue. As an example, let's suppose that we get the following error when trying to create a collection using the CLI tools:\n\n
\n$ arv collection create --collection '{}'\nError: # (req-ku5ct9ehw0y71f1c5p79)\n
\n\nThe API Server logs every request in JSON format on the @production.log@ (usually under @/var/www/arvados-api/current/log/@ when installing from packages) file, so we can retrieve more information about this by using @grep@ and @jq@ tools:\n\n
\n# grep req-ku5ct9ehw0y71f1c5p79 /var/www/arvados-api/current/log/production.log | jq .\n{\n  \"method\": \"POST\",\n  \"path\": \"/arvados/v1/collections\",\n  \"format\": \"json\",\n  \"controller\": \"Arvados::V1::CollectionsController\",\n  \"action\": \"create\",\n  \"status\": 422,\n  \"duration\": 1.52,\n  \"view\": 0.25,\n  \"db\": 0,\n  \"request_id\": \"req-ku5ct9ehw0y71f1c5p79\",\n  \"client_ipaddr\": \"127.0.0.1\",\n  \"client_auth\": \"zzzzz-gj3su-jllemyj9v3s5emu\",\n  \"exception\": \"#\",\n  \"exception_backtrace\": \"/var/www/arvados-api/current/app/controllers/arvados/v1/collections_controller.rb:43:in `create'\\n/var/lib/gems/ruby/2.3.0/gems/actionpack-5.0.7.2/lib/action_controller/metal/basic_implicit_render.rb:4:in `send_action'\\n ...[snipped]\",\n  \"params\": {\n    \"collection\": \"{}\",\n    \"_profile\": \"true\",\n    \"cluster_id\": \"\",\n    \"collection_given\": \"true\",\n    \"ensure_unique_name\": \"false\",\n    \"help\": \"false\"\n  },\n  \"@timestamp\": \"2019-07-15T16:40:41.726634182Z\",\n  \"@version\": \"1\",\n  \"message\": \"[422] POST /arvados/v1/collections (Arvados::V1::CollectionsController#create)\"\n}\n
\n\nWhen logging a request that produced an error, the API Server adds @exception@ and @exception_backtrace@ keys to the JSON log. The latter includes the complete error stack trace as a string, and can be displayed in a more readable form like so:\n\n
\n# grep req-ku5ct9ehw0y71f1c5p79 /var/www/arvados-api/current/log/production.log | jq -r .exception_backtrace\n/var/www/arvados-api/current/app/controllers/arvados/v1/collections_controller.rb:43:in `create'\n/var/lib/gems/ruby/2.3.0/gems/actionpack-5.0.7.2/lib/action_controller/metal/basic_implicit_render.rb:4:in `send_action'\n/var/lib/gems/ruby/2.3.0/gems/actionpack-5.0.7.2/lib/abstract_controller/base.rb:188:in `process_action'\n/var/lib/gems/ruby/2.3.0/gems/actionpack-5.0.7.2/lib/action_controller/metal/rendering.rb:30:in `process_action'\n/var/lib/gems/ruby/2.3.0/gems/actionpack-5.0.7.2/lib/abstract_controller/callbacks.rb:20:in `block in process_action'\n/var/lib/gems/ruby/2.3.0/gems/activesupport-5.0.7.2/lib/active_support/callbacks.rb:126:in `call'\n...\n
\n" }, { "path": "doc/admin/logs-table-management.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: \"Logs table management\"\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThis page aims to provide insight about managing the ever growing API Server's logs table.\n\nh3. Logs table purpose & behavior\n\nThis database table is accessed via \"the @logs@ endpoint.\":../api/methods/logs.html\n\nThis table currently serves several purposes:\n\n* Audit logging, permitting admins and users to look up the time and details of past changes to Arvados objects.\n* Logging other system events, specifically \"file uploads and downloads from keep-web.\":restricting-upload-download.html#audit_logs\n* The source for cache-invalidation events, published through websockets to Workbench to refresh the view. It can also be monitored by the Python SDK \"events module.\":../sdk/python/events.html\n* Prior to Arvados 2.7, it was used a staging area for stdout/stderr text coming from users' containers, permitting users to see what their containers are doing while they are still running (i.e., before those text files are written to Keep). Starting with Arvados 2.7, this is superseded by a more efficient mechanism, so these logs are disabled by default. See \"2.7.0 upgrade notes\":upgrading.html#v2_7_0 for details.\n\nAs a result, this table grows indefinitely, even on sites where policy does not require an audit log; making backups, migrations, and upgrades unnecessarily slow and painful.\n\nh3. Configuration\n\nTo solve the problem mentioned above, the @AuditLogs@ section of @config.yml@ offers several options to limit the amount of log information stored on the table:\n\n
\n    AuditLogs:\n      # Time to keep audit logs. (An audit log is a row added\n      # to the \"logs\" table in the PostgreSQL database each time an\n      # Arvados object is created, modified, or deleted.)\n      #\n      # Currently, websocket event notifications rely on audit logs, so\n      # this should not be set lower than 5 minutes.\n      MaxAge: 336h\n\n      # Maximum number of log rows to delete in a single SQL transaction,\n      # to prevent surprises and avoid bad database behavior\n      # (especially the first time the cleanup job runs on an existing\n      # cluster with a huge backlog) a maximum number of rows to\n      # delete in a single transaction.\n      #\n      # If MaxDeleteBatch is 0, log entries will never be\n      # deleted by Arvados. Cleanup can be done by an external process\n      # without affecting any Arvados system processes, as long as very\n      # recent (<5 minutes old) logs are not deleted.\n      #\n      # 100000 is a reasonable batch size for most sites.\n      MaxDeleteBatch: 0\n\n      # Attributes to suppress in events and audit logs.  Notably,\n      # specifying {\"manifest_text\": {}} here typically makes the database\n      # smaller and faster.\n      #\n      # Warning: Using any non-empty value here can have undesirable side\n      # effects for any client or component that relies on event logs.\n      # Use at your own risk.\n      UnloggedAttributes: {}\n
\n\n\nh3. Additional consideration\n\nDepending on the local installation's audit requirements, the cluster admins should plan for an external backup procedure before enabling this feature, as this information is not replicated anywhere else.\n" }, { "path": "doc/admin/maintenance-and-upgrading.html.textile.liquid", "content": "---\nlayout: default\nnavsection: installguide\ntitle: Maintenance and upgrading\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n# \"Commercial support\":#commercial_support\n# \"Maintaining Arvados\":#maintaining\n## \"Modification of the config.yml file\":#configuration\n## \"Distributing the configuration file\":#distribution\n## \"Restart the services affected by the change\":#restart\n# \"Upgrading Arvados\":#upgrading\n\nh2(#commercial_support). Commercial support\n\nArvados is \"100% open source software\":{{site.baseurl}}/user/copying/copying.html. Anyone can download, install, maintain and upgrade it. However, if this is not something you want to spend your time and energy doing, \"Curii Corporation\":https://curii.com provides managed Arvados installations as well as commercial support for Arvados. Please contact \"info@curii.com\":mailto:info@curii.com for more information.\n\nIf you'd prefer to do things yourself, a few starting points for maintaining and upgrading Arvados can be found below.\n\nh2(#maintaining). Maintaining Arvados\n\nAfter Arvados is installed, periodic configuration changes may be required to adapt the software to your needs. Arvados uses a unified configuration file, which is normally found at @/etc/arvados/config.yml@.\n\nMaking a configuration change to Arvados typically involves three steps:\n\n* modification of the @config.yml@ file\n* distribution of the modified file to the machines in the cluster\n* restarting of the services affected by the change\n\nh3(#configchange). Modification of the @config.yml@ file\n\nConsult the \"configuration reference\":{{site.baseurl}}/admin/config.html or another part of the documentation to identify the change to be made.\n\nPreserve a copy of your existing configuration file as a backup, and make the desired modification.\n\nRun @arvados-server config-check@ to make sure the configuration file has no errors and no warnings.\n\nh3(#distribution). Distribute the configuration file\n\nIt is very important to keep the @config.yml@ file in sync between all the Arvados system nodes, to avoid issues with services running on different versions of the configuration.\n\nWe provide \"installer.sh\":../install/install-multi-host.html#installation to distribute config changes. You may also do your own orchestration e.g. @scp@, configuration management software, etc.\n\nh3(#restart). Restart the services affected by the change\n\nIf you know which Arvados service uses the specific configuration that was modified, restart those services. When in doubt, restart all Arvados system services.\n\nTo check for services that have not restarted since the configuration file was updated, run the @arvados-server check@ command on each system node.\n\nTo test functionality and check for common problems, run the @arvados-client sudo diagnostics@ command on a system node.\n\nh2(#upgrading). Upgrading Arvados\n\nUpgrading Arvados typically involves the following steps:\n\n# consult the \"upgrade notes\":{{site.baseurl}}/admin/upgrading.html and the \"release notes\":https://arvados.org/releases/ for the release you want to upgrade to\n# Wait for the cluster to be idle and stop Arvados services.\n# Make a backup of your database, as a precaution.\n# update the configuration file for the new release, if necessary (see \"Maintaining Arvados\":#maintaining above)\n# Update compute nodes\n## (cloud) Rebuild and deploy the \"compute node image\":{{site.baseurl}}/install/crunch2-cloud/install-compute-node.html\n## (slurm/LSF) Upgrade the @python3-arvados-fuse@ package used on your compute nodes\n# Install new packages using @apt upgrade@ or @dnf upgrade@.\n# Wait for package installation scripts as they perform any necessary data migrations.\n# Run @arvados-server config-check@ to detect configuration errors or deprecated entries.\n# Verify that the Arvados services were restarted as part of the package upgrades.\n# Run @arvados-server check@ to detect services that did not restart properly.\n# Run @arvados-client sudo diagnostics@ to test functionality.\n" }, { "path": "doc/admin/management-token.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: Management token\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nTo enable and collect health checks and metrics, services must be configured with a \"management token\".\n\nServices must have ManagementToken configured. This is used to authorize access monitoring endpoints. If ManagementToken is not configured, monitoring endpoints will return the error @404 disabled@.\n\nTo access a monitoring endpoint, the requester must provide the HTTP header @Authorization: Bearer (ManagementToken)@.\n\nh2. API server and other services\n\nThe following services also support monitoring.\n\n* API server\n* controller\n* keep-balance\n* keepproxy\n* keepstore\n* keep-web\n* arvados-ws \n\nSet @ManagementToken@ in the appropriate section of @/etc/arvados/config.yml@.\n\n\n
Clusters:\n  ClusterID:\n    # Token to be included in all healthcheck requests. Disabled by default.\n    # Server expects request header of the format \"Authorization: Bearer xxx\"\n    ManagementToken: xxx\n
\n\n" }, { "path": "doc/admin/memory-cpu-profiling.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: Memory and CPU profiling\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nArvados system services (other than the Rails API server) have an option to provide live profiling data on an HTTP endpoint. This can be analyzed with the @go tool pprof@ program from the Go runtime to help identify memory and CPU usage issues. The @go tool pprof@ program can either connect directly to the profiling endpoint, or read a snapshot from disk.\n\nEnable profiling by choosing a listening address and adding @-pprof
@ to the @EXTRA_OPTS@ environment variable in the systemd service.\n\nExample:\n\n
$ sudo systemctl edit keep-balance\n### Editing /etc/systemd/system/keep-balance.service.d/override.conf\n### Anything between here and the comment below will become the new contents of the file\n[Service]\nEnvironment=\"EXTRA_OPTS=-pprof 127.0.0.1:3333\"\n### Lines below this comment will be discarded\n[...]\n
\n\nRestart the service.\n\n
$ sudo systemctl restart keep-balance\n
\n\nSave a snapshot of the program's active memory usage after garbage collection.\n\n
$ curl 'http://localhost:3333/debug/pprof/heap?gc=1' > /tmp/pprof.gz\n
\n\nThe following analysis steps can be done on the server itself or on a different machine.\n\nTo get the @go tool pprof@ command, install the Go runtime from OS packages or from the \"Go download page\":https://go.dev/doc/install.\n\n
$ sudo apt install golang\n
\n\nRun the @go tool pprof@ command to summarize the snapshot.\n\n
$ go tool pprof -top /tmp/pprof.gz\nFile: keep-balance\nBuild ID: edd0405c97f4235473dba21b7c7fd52c8f755cde\nType: inuse_space\nTime: Nov 3, 2025 at 11:12am (EST)\nShowing nodes accounting for 443.71MB, 98.02% of 452.67MB total\nDropped 35 nodes (cum <= 2.26MB)\n      flat  flat%   sum%        cum   cum%\n  217.54MB 48.06% 48.06%   217.54MB 48.06%  git.arvados.org/arvados.git/services/keep-balance.(*BlockState).increaseDesired (inline)\n   85.46MB 18.88% 66.94%    85.46MB 18.88%  git.arvados.org/arvados.git/services/keep-balance.(*BlockStateMap).get (inline)\n   66.07MB 14.60% 81.53%    76.58MB 16.92%  git.arvados.org/arvados.git/sdk/go/arvados.(*Collection).SizedDigests\n   19.87MB  4.39% 85.92%    20.87MB  4.61%  github.com/lib/pq.textDecode\n      13MB  2.87% 88.79%       13MB  2.87%  git.arvados.org/arvados.git/services/keep-balance.(*BlockState).addReplica (inline)\n   10.51MB  2.32% 91.12%    10.51MB  2.32%  bytes.genSplit\n   10.25MB  2.26% 93.38%    10.25MB  2.26%  github.com/lib/pq.(*conn).recvMessage\n    7.50MB  1.66% 95.04%    53.63MB 11.85%  git.arvados.org/arvados.git/services/keep-balance.EachCollection\n       5MB  1.10% 96.14%    13.50MB  2.98%  encoding/json.Unmarshal\n    4.50MB  0.99% 97.14%     4.50MB  0.99%  encoding/json.(*scanner).pushParseState\n       4MB  0.88% 98.02%        4MB  0.88%  encoding/json.(*decodeState).literalStore\n         0     0% 98.02%    10.51MB  2.32%  bytes.Split (inline)\n         0     0% 98.02%    31.13MB  6.88%  database/sql.(*Rows).Next\n         0     0% 98.02%    31.13MB  6.88%  database/sql.(*Rows).Next.func1\n         0     0% 98.02%    31.13MB  6.88%  database/sql.(*Rows).nextLocked\n         0     0% 98.02%    31.13MB  6.88%  database/sql.withLock\n         0     0% 98.02%        4MB  0.88%  encoding/json.(*decodeState).array\n         0     0% 98.02%        4MB  0.88%  encoding/json.(*decodeState).unmarshal\n         0     0% 98.02%        4MB  0.88%  encoding/json.(*decodeState).value\n         0     0% 98.02%     4.50MB  0.99%  encoding/json.checkValid\n         0     0% 98.02%     4.50MB  0.99%  encoding/json.stateBeginValue\n         0     0% 98.02%     2.46MB  0.54%  git.arvados.org/arvados.git/lib/service.(*command).RunCommand.ifCollectionInHost.func9\n         0     0% 98.02%     2.46MB  0.54%  git.arvados.org/arvados.git/sdk/go/httpserver.(*metrics).ServeAPI.RequireLiteralToken.func3\n...\n
\n\nThe @go tool pprof@ command can also connect directly to the profiling endpoint. In this mode, by default it will also save a snapshot in @$HOME/pprof/@.\n\nTo connect directly to the profiling endpoint and display a sampling of CPU usage over a 2-second interval:\n\n
$ go tool pprof -top 'http://localhost:3333/debug/pprof/profile?seconds=2'\nFetching profile over HTTP from http://localhost:3333/debug/pprof/profile?seconds=2\nSaved profile in /home/username/pprof/pprof.keep-balance.samples.cpu.001.pb.gz\nFile: keep-balance\nBuild ID: edd0405c97f4235473dba21b7c7fd52c8f755cde\nType: cpu\nTime: Nov 3, 2025 at 11:12am (EST)\nDuration: 2.19s, Total samples = 2.98s (136.17%)\nShowing nodes accounting for 2.57s, 86.24% of 2.98s total\nDropped 73 nodes (cum <= 0.01s)\n      flat  flat%   sum%        cum   cum%\n     0.34s 11.41% 11.41%      0.36s 12.08%  runtime.findObject\n     0.15s  5.03% 16.44%      0.32s 10.74%  regexp.(*Regexp).doOnePass\n     0.13s  4.36% 20.81%      0.13s  4.36%  runtime.(*mspan).heapBitsSmallForAddr\n     0.12s  4.03% 24.83%      0.12s  4.03%  runtime.(*gcBits).bitp (inline)\n     0.11s  3.69% 28.52%      0.11s  3.69%  regexp/syntax.(*Inst).MatchRunePos\n     0.08s  2.68% 31.21%      0.09s  3.02%  runtime.(*mspan).writeHeapBitsSmall\n     0.07s  2.35% 33.56%      0.07s  2.35%  runtime.nextFreeFast (inline)\n     0.06s  2.01% 35.57%      0.06s  2.01%  runtime.futex\n     0.06s  2.01% 37.58%      0.06s  2.01%  runtime.memclrNoHeapPointers\n     0.05s  1.68% 39.26%      0.05s  1.68%  indexbytebody\n     0.05s  1.68% 40.94%      0.05s  1.68%  internal/runtime/syscall.Syscall6\n     0.05s  1.68% 42.62%      0.43s 14.43%  runtime.mallocgc\n     0.05s  1.68% 44.30%      0.05s  1.68%  runtime.memmove\n     0.05s  1.68% 45.97%      0.71s 23.83%  runtime.scanobject\n     0.05s  1.68% 47.65%      0.05s  1.68%  runtime.usleep\n     0.04s  1.34% 48.99%      0.26s  8.72%  runtime.mallocgcSmallScanNoHeader\n     0.04s  1.34% 50.34%      0.04s  1.34%  runtime.rand\n     0.03s  1.01% 51.34%      0.03s  1.01%  crypto/internal/fips140/aes/gcm.gcmAesDec\n     0.03s  1.01% 52.35%      0.57s 19.13%  database/sql.(*Rows).nextLocked\n     0.03s  1.01% 53.36%      0.13s  4.36%  database/sql.convertAssignRows\n     0.03s  1.01% 54.36%      0.56s 18.79%  git.arvados.org/arvados.git/sdk/go/arvados.(*Collection).SizedDigests\n     0.03s  1.01% 55.37%      0.18s  6.04%  git.arvados.org/arvados.git/services/keep-balance.(*BlockStateMap).GetConfirmedReplication\n...\n
\n\n@http://localhost:3333/debug/pprof/@ serves an HTML page with a list of available profiles:\n* @allocs@ -- A sampling of all past memory allocations\n* @block@ -- Stack traces that led to blocking on synchronization primitives\n* @cmdline@ -- The command line invocation of the current program\n* @goroutine@ -- Stack traces of all current goroutines. Use debug=2 as a query parameter to export in the same format as an unrecovered panic.\n* @heap@ -- A sampling of memory allocations of live objects. You can specify the gc GET parameter to run GC before taking the heap sample.\n* @mutex@ -- Stack traces of holders of contended mutexes\n* @profile@ -- CPU profile. You can specify the duration in the seconds GET parameter. After you get the profile file, use the go tool pprof command to investigate the profile.\n* @symbol@ -- Maps given program counters to function names. Counters can be specified in a GET raw query or POST body, multiple counters are separated by '+'.\n* @threadcreate@ -- Stack traces that led to the creation of new OS threads\n* @trace@ -- A trace of execution of the current program. You can specify the duration in the seconds GET parameter. After you get the trace file, use the go tool trace command to investigate the trace.\n\nAdditional resources:\n* \"pprof tool documentation\":https://github.com/google/pprof/blob/main/doc/README.md\n* \"Go profiling data endpoint documentation\":https://pkg.go.dev/net/http/pprof\n" }, { "path": "doc/admin/metadata-vocabulary.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: Metadata vocabulary\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nMany Arvados objects (like collections and projects) can store metadata as properties that in turn can be used in searches allowing a flexible way of organizing data inside the system.\n\nArvados enables the site administrator to set up a formal metadata vocabulary definition so that users can select from predefined key/value pairs of properties, offering the possibility to add different terms for the same concept on clients' UI such as workbench2.\n\nThe Controller service loads and caches the configured vocabulary file in memory at startup time, exporting it on a particular endpoint. From time to time, it'll check for updates in the local copy and refresh its cache if validation passes.\n\nh2. Configuration\n\nThe site administrator should place the JSON vocabulary file on the same host as the controller service and set up the config file as follows:\n\n\n
Cluster:\n  zzzzz:\n    API:\n      VocabularyPath: /etc/arvados/vocabulary.json\n
\n\n\nh2. Definition format\n\nThe JSON file describes the available keys and values and if the user is allowed to enter free text not defined by the vocabulary.\n\nKeys and values are indexed by identifiers so that the concept of a term is preserved even if vocabulary labels are changed.\n\nThe following is an example of a vocabulary definition:\n\n{% codeblock as json %}\n{% include 'metadata_vocabulary_example' %}\n{% endcodeblock %}\n\nFor clients to be able to query the vocabulary definition, a special endpoint is exposed on the @controller@ service: @/arvados/v1/vocabulary@. This endpoint doesn't require authentication and returns the vocabulary definition in JSON format.\n\nIf the @strict_tags@ flag at the root level is @true@, it will restrict the users from saving property keys other than the ones defined in the vocabulary. This restriction is enforced at the backend level to ensure consistency across different clients.\n\nInside the @tags@ member, IDs are defined (@IDTAGANIMALS@, @IDTAGCOMMENT@, @IDTAGIMPORTANCES@) and can have any format that the current application requires. Every key will declare at least a @labels@ list with zero or more label objects.\n\nThe @strict@ flag inside a tag definition operates the same as the @strict_tags@ root member, but at the individual tag level. When @strict@ is @true@, a tag’s value options are limited to those defined by the vocabulary.\n\nThe @values@ member is optional and is used to define valid key/label pairs when applicable. In the example above, @IDTAGCOMMENT@ allows open-ended text by only defining the tag's ID and labels and leaving out @values@.\n\nWhen any key or value has more than one label option, Workbench2's user interface will allow the user to select any of the options. But because only the IDs are saved in the system, when the property is displayed in the user interface, the label shown will be the first of each group defined in the vocabulary file. For example, the user could select the property key @Species@ and @Homo sapiens@ as its value, but the user interface will display it as @Animal: Human@ because those labels are the first in the vocabulary definition.\n\nInternally, Workbench2 uses the IDs to do property based searches, so if the user searches by @Animal: Human@ or @Species: Homo sapiens@, both will return the same results.\n\nh2. Definition validation\n\nBecause the vocabulary definition is prone to syntax or logical errors, the @controller@ service needs to do some validation before answering requests. If the vocabulary validation fails, the service won't start.\nThe site administrator can make sure the vocabulary file is correct before even trying to start the @controller@ service by running @arvados-server config-check@. When the vocabulary definition isn't correct, the administrator will get a list of issues like the one below:\n\n\n
# arvados-server config-check -config /etc/arvados/config.yml\nError loading vocabulary file \"/etc/arvados/vocabulary.json\" for cluster zzzzz:\nduplicate JSON key \"tags.IDTAGFRUITS.values.IDVALFRUITS1\"\ntag key \"IDTAGCOMMENT\" is configured as strict but doesn't provide values\ntag value label \"Banana\" for pair (\"IDTAGFRUITS\":\"IDVALFRUITS8\") already seen on value \"IDVALFRUITS4\"\nexit status 1\n
\n\n\nbq. NOTE: These validation checks are performed only on the node that hosts the vocabulary file defined on the configuration. As the same configuration file is shared between different nodes, those who don't host the file won't produce spurious errors when running @arvados-server config-check@.\n\nh2. Live updates\n\nSometimes it may be necessary to modify the vocabulary definition in a running production environment.\nWhen a change is detected, the @controller@ service will automatically attempt to load the new vocabulary and check its validity before making it active.\nIf the new vocabulary has some issue, the last valid one will keep being active. The service will export any errors on its health endpoint so that a monitoring solution can send an alert appropriately.\nWith the above mechanisms in place, no outages should occur from making typos or other errors when updating the vocabulary file.\n\nh2. Health status\n\nTo be able for the administrator to guarantee the system's metadata integrity, the @controller@ service exports a specific health endpoint for the vocabulary at @/_health/vocabulary@.\nAs a first measure, the service won't start if the vocabulary file is incorrect. Once running, if there are updates (that may even be periodical), the service needs to keep running while notifying the operator that some fixing is in order.\nAn example of a vocabulary health error is included below:\n\n\n
$ curl --silent -H \"Authorization: Bearer xxxtokenxxx\" https://controller/_health/vocabulary | jq .\n{\n  \"error\": \"while loading vocabulary file \\\"/etc/arvados/vocabulary.json\\\": duplicate JSON key \\\"tags.IDTAGSIZES.values.IDVALSIZES3\\\"\",\n  \"health\": \"ERROR\"\n}\n
\n\n\nh2. Client support\n\nWorkbench2 currently takes advantage of this vocabulary definition by providing an easy-to-use interface for searching and applying metadata to different objects in the system. Because the definition file only resides on the @controller@ node, and Workbench2 is just a static web application run by every users' web browser, there's a mechanism in place that allows Workbench2 and any other client to request the active vocabulary.\n\nThe @controller@ service provides an unauthenticated endpoint at @/arvados/v1/vocabulary@ where it exports the contents of the vocabulary JSON file:\n\n\n
$ curl --silent https://controller/arvados/v1/vocabulary | jq .\n{\n  \"kind\": \"arvados#vocabulary\",\n  \"strict_tags\": false,\n  \"tags\": {\n    \"IDTAGANIMALS\": {\n      \"labels\": [\n        {\n          \"label\": \"Animal\"\n        },\n        {\n          \"label\": \"Creature\"\n        }\n      ],\n      \"strict\": false,\n...\n}\n
\n\n\nAlthough the vocabulary enforcement is done on the backend side, clients can use this information to provide helping features to users, like doing ID-to-label translations, preemptive error checking, etc.\n\nh2. Properties migration\n\nAfter installing the new vocabulary definition, it may be necessary to migrate preexisting properties that were set up using literal strings. This can be a big task depending on the number of properties on the vocabulary and the amount of collections and projects on the cluster.\n\nTo help with this task we provide below a migration example script that accepts the new vocabulary definition file as an input, and uses the @ARVADOS_API_TOKEN@ and @ARVADOS_API_HOST@ environment variables to connect to the cluster, search for every collection and group that has properties with labels defined on the vocabulary file, and migrates them to the corresponding identifiers.\n\nThis script will not run if the vocabulary file has duplicated labels for different keys or for different values inside a key, this is a failsafe mechanism to avoid migration errors.\n\nPlease take into account that this script requires admin credentials. It also offers a @--dry-run@ flag that will report what changes are required without applying them, so it can be reviewed by an administrator.\n\nAlso, take into consideration that this example script does case-sensitive matching on labels.\n\n{% codeblock as python %}\n{% include 'vocabulary_migrate_py' %}\n{% endcodeblock %}\n" }, { "path": "doc/admin/metrics.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: Metrics\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nSome Arvados services publish Prometheus/OpenMetrics-compatible metrics at @/metrics@. Metrics can help you understand how components perform under load, find performance bottlenecks, and detect and diagnose problems.\n\nTo access metrics endpoints, services must be configured with a \"management token\":management-token.html. When accessing a metrics endpoint, prefix the management token with @\"Bearer \"@ and supply it in the @Authorization@ request header.\n\n
curl -sfH \"Authorization: Bearer your_management_token_goes_here\" \"https://0.0.0.0:25107/metrics\"\n
\n\nThe plain text export format includes \"help\" messages with a description of each reported metric.\n\nWhen configuring Prometheus, use a @bearer_token@ or @bearer_token_file@ option to authenticate requests.\n\n
scrape_configs:\n  - job_name: keepstore\n    bearer_token: your_management_token_goes_here\n    static_configs:\n    - targets:\n      - \"keep0.ClusterID.example.com:25107\"\n
\n\ntable(table table-bordered table-condensed table-hover).\n|_. Component|_. Metrics endpoint|\n|arvados-api-server|✓|\n|arvados-controller|✓|\n|arvados-dispatch-cloud|✓|\n|arvados-dispatch-lsf|✓|\n|arvados-ws|✓|\n|keepproxy|✓|\n|keepstore|✓|\n|keep-balance|✓|\n|keep-web|✓|\n|workbench2||\n" }, { "path": "doc/admin/migrating-providers.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: Changing upstream login providers\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nWhen a user logs in to Arvados, their email address (as returned by the authentication provider) is used as the primary key for their Arvados account.\n\nIf you reconfigure Arvados to use a different authentication provider after some users have created accounts, you should either ensure the new provider returns the same email addresses as the old one, or update your Arvados users' @email@ attributes to match the email addresses returned by the new provider.\n\nOtherwise, next time users log in, they will be given new accounts instead of logging in to their existing accounts.\n" }, { "path": "doc/admin/reassign-ownership.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: \"Reassign user data ownership\"\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nIf a user leaves an organization and stops using their Arvados account, it may be desirable to reassign the data owned by that user to another user to maintain easy access.\n\nThis is currently a command line based, admin-only feature.\n\nh3. Step 1: Determine user uuids\n\nUser uuids can be determined by browsing workbench or using @arv user list@ at the command line.\n\nThe \"old user\" is the user that is leaving the organization.\n\nThe \"new user\" is the user that will gain ownership of the old user's data. This includes collections, projects, container requests, workflows, and git repositories owned by the old user. It also transfers any permissions granted to the old user, to the new user.\n\nIn the example below, @x1u39-tpzed-3kz0nwtjehhl0u4@ is the old user and @x1u39-tpzed-fr97h9t4m5jffxs@ is the new user.\n\nh3. Step 2: Create a project\n\nCreate a project owned by the new user that will hold the data from the old user.\n\n
\n$ arv --format=uuid group create --group '{\"group_class\": \"project\", \"name\": \"Data from old user\", \"owner_uuid\": \"x1u39-tpzed-fr97h9t4m5jffxs\"}'\nx1u39-j7d0g-mczqiguhil13083\n
\n\nh3. Step 3: Reassign data from the old user to the new user and project\n\nThe @user merge@ method reassigns data from the old user to the new user.\n\n
\n$ arv user merge --old-user-uuid=x1u39-tpzed-3kz0nwtjehhl0u4 \\\n  --new-user-uuid=x1u39-tpzed-fr97h9t4m5jffxs \\\n  --new-owner-uuid=x1u39-j7d0g-mczqiguhil13083\n
\n\nAfter reassigning data, use @unsetup@ to deactivate the old user's account.\n\n
\n$ arv user unsetup --uuid=x1u39-tpzed-3kz0nwtjehhl0u4\n
\n\nNote that authorization credentials (API tokens, ssh keys) are *not* transferred to the new user, as this would potentially give the old user access to the new user's account.\n" }, { "path": "doc/admin/restricting-upload-download.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: Restricting upload or download\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nFor some use cases, you may want to limit the ability of users to upload or download data from outside the cluster. (By \"outside\" we mean from networks other than the cluster's own private network). For example, this makes it possible to share restricted data sets with users so that they may run their own data analysis on the cluster, while preventing them from easily downloading the data set to their local workstation.\n\nThis feature exists in addition to the existing Arvados permission system. Users can only download from collections they have @read@ access to, and can only upload to projects and collections they have @write@ access to.\n\nThere are two services involved in accessing data from outside the cluster.\n\nh2. Keepproxy Permissions\n\nPermitting @keepproxy@ makes it possible to use @arv-put@ and @arv-get@. It works in terms of individual 64 MiB keep blocks. It prints a log line each time a user uploads or downloads an individual block. Those logs are usually stored by @journald@ or @syslog@.\n\nThe default policy allows anyone to upload or download.\n\n
\n    Collections:\n      KeepproxyPermission:\n        User:\n          Download: true\n          Upload: true\n        Admin:\n          Download: true\n          Upload: true\n
\n\nh2. WebDAV and S3 API Permissions\n\nPermitting @WebDAV@ makes it possible to use WebDAV, S3 API, and upload/download with Workbench 2. It works in terms of individual files. It prints a log each time a user uploads or downloads a file (\"subject to throttling discussed below\":#throttling). When @WebDAVLogEvents@ (default true) is enabled, it also adds an entry into the API server @logs@ table.\n\nWhen a user attempts to upload or download from a service without permission, they will receive a @403 Forbidden@ response. This only applies to file content.\n\nDenying download permission does not deny access to access to XML file listings with PROPFIND, or auto-generated HTML documents containing file listings.\n\nDenying upload permission does not deny other operations that modify collections without directly accessing file content, such as MOVE and COPY.\n\nThe default policy allows anyone to upload or download.\n\n
\n    Collections:\n      WebDAVPermission:\n        User:\n          Download: true\n          Upload: true\n        Admin:\n          Download: true\n          Upload: true\n      WebDAVLogEvents: true\n      WebDAVLogDownloadInterval: 30s\n
\n\nWhen a user or admin creates a sharing link, a custom scoped token is embedded in that link. This effectively allows anonymous user access to the associated data via that link. These custom scoped tokens are always treated as user tokens for the purposes of restricting download access, even when created by an admin user. In other words, these custom scoped tokens, when used in a sharing link, are always subject to the value of the @WebDAVPermission/User/Download@ configuration setting.\n\nIf that custom scoped token is used with @arv-get@, its use will be subject to the value of the @KeepproxyPermission/User/Download@ configuration setting.\n\nh2. Shell node and container permissions\n\nBe aware that even when upload and download from outside the network is not allowed, a user who has access to a shell node or runs a container still has internal access to Keep. (This is necessary to be able to run workflows). From the shell node or container, a user could send data outside the network by some other method, although this requires more intent than accidentally clicking on a link and downloading a file. It is possible to set up a firewall to prevent shell and compute nodes from making connections to hosts outside the private network. Exactly how to configure firewalls is out of scope for this page, as it depends on the specific network infrastructure of your cluster.\n\nh2. Choosing a policy\n\nThis distinction between WebDAV and Keepproxy is important for auditing. WebDAV records 'upload' and 'download' events on the API server that are included in the \"User Activity Report\":user-activity.html, whereas @keepproxy@ only logs upload and download of individual blocks, which require a reverse lookup to determine the collection(s) and file(s) a block is associated with.\n\nYou set separate permissions for @WebDAV@ and @Keepproxy@, with separate policies for regular users and admin users.\n\nThese policies apply to only access from outside the cluster, using Workbench or Arvados CLI tools.\n\nThe @WebDAVLogEvents@ option should be enabled if you intend to the run the \"User Activity Report\":user-activity.html. If you don't need audits, or you are running a site that is mostly serving public data to anonymous downloaders, you can disable it to avoid the extra API server request.\n\nh3. Audited downloads\n\nFor ease of access auditing, this policy prevents downloads using @arv-get@. Downloads through WebDAV and S3 API are permitted, but logged. Uploads are allowed.\n\n
\n    Collections:\n      WebDAVPermission:\n        User:\n          Download: true\n          Upload: true\n        Admin:\n          Download: true\n          Upload: true\n\n      KeepproxyPermission:\n        User:\n          Download: false\n          Upload: true\n        Admin:\n          Download: false\n          Upload: true\n      WebDAVLogEvents: true\n
\n\nh3. Disallow downloads by regular users\n\nThis policy prevents regular users (non-admin) from downloading data. Uploading is allowed. This supports the case where restricted data sets are shared with users so that they may run their own data analysis on the cluster, while preventing them from downloading the data set to their local workstation. Be aware that users won't be able to download the results of their analysis, either, requiring an admin in the loop or some other process to release results.\n\n
\n    Collections:\n      WebDAVPermission:\n        User:\n          Download: false\n          Upload: true\n        Admin:\n          Download: true\n          Upload: true\n\n      KeepproxyPermission:\n        User:\n          Download: false\n          Upload: true\n        Admin:\n          Download: true\n          Upload: true\n      WebDAVLogEvents: true\n
\n\nh3. Disallow uploads by regular users\n\nThis policy is suitable for an installation where data is being shared with a group of users who are allowed to download the data, but not permitted to store their own data on the cluster.\n\n
\n    Collections:\n      WebDAVPermission:\n        User:\n          Download: true\n          Upload: false\n        Admin:\n          Download: true\n          Upload: true\n\n      KeepproxyPermission:\n        User:\n          Download: true\n          Upload: false\n        Admin:\n          Download: true\n          Upload: true\n      WebDAVLogEvents: true\n
\n\n\nh2(#audit_log). Accessing the audit log\n\nWhen @WebDAVLogEvents@ is enabled, uploads and downloads of files are logged in the Arvados audit log. These events are included in the \"User Activity Report\":user-activity.html. The audit log can also be accessed via the API, SDKs or command line. For example, to show the 100 most recent file downloads:\n\n
\narv log list --filters '[[\"event_type\",\"=\",\"file_download\"]]' -o 'created_at desc' -l 100\n
\n\nFor uploads, use the @file_upload@ event type.\n\nNote that this only covers upload and download activity via WebDAV, S3, and Workbench 2.\n\nThe @arv-get@ and @arv-put@ tools upload via @Keepproxy@, which does not log activity to the audit log because it operates at the block level, not the file level. @Keepproxy@ records the uuid of the user that owns the token used in the request in its system logs. Those logs are usually stored by @journald@ or @syslog@. A typical log line for such a block download looks like this:\n\n
\nJul 20 15:03:38 keep.xxxx1.arvadosapi.com keepproxy[63828]: {\"level\":\"info\",\"locator\":\"abcdefghijklmnopqrstuvwxyz012345+53251584\",\"msg\":\"Block download\",\"time\":\"2021-07-20T15:03:38.458792300Z\",\"user_full_name\":\"Albert User\",\"user_uuid\":\"ce8i5-tpzed-abcdefghijklmno\"}\n
\n\nIt is possible to do a reverse lookup from the locator to find all matching collections: the @manifest_text@ field of a collection lists all the block locators that are part of the collection. The @manifest_text@ field also provides the relevant filename in the collection. Because this lookup is rather involved and there is no automated tool to do it, we recommend disabling @KeepproxyPermission.User.Download@ and @KeepproxyPermission.User.Upload@ for sites where the audit log is important and @arv-get@ and @arv-put@ are not essential.\n\nh3(#throttling). WebDAV download log throttling\n\nIf a client requests partial content past the start of a file, and a request from the same client for the same file was logged within the last time interval configured by @WebDAVLogDownloadInterval@, @keep-web@ will not write a new log. This throttling applies to both printed and API server logs. The default value of 30 seconds reduces log output when clients like @aws s3 cp@ download one file in small chunks in parallel. Administrators can set this setting to @0@ to disable log throttling. This setting lets administrators choose how they want to balance full auditability against logging overhead: a shorter interval means more download requests are logged, with all the overhead that entails.\n" }, { "path": "doc/admin/scoped-tokens.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: Securing API access with scoped tokens\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nBy default, Arvados API tokens grant unlimited access to a user account, and admin account tokens have unlimited access to the whole system. If you want to grant restricted access to a user account, you can create a \"scoped token\" which is an Arvados API token which is limited to accessing specific APIs.\n\nOne use of token scopes is to grant access to data, such as a collection, to users who do not have an Arvados accounts on your cluster. This is done by creating scoped token that only allows getting a specific record. An example of this is \"creating a collection sharing link.\":{{site.baseurl}}/sdk/python/cookbook.html#sharing_link\n\nAnother example is situations where admin access is required but there is risk of the token being compromised. Setting a scope prevents the token from being used for any action other than the specific action the token is intended for. For example, \"synchronizing user accounts on a shell node.\":{{site.baseurl}}/install/install-shell-server.html#scoped-token\n\nh2. Defining scopes\n\nA \"scope\" consists of a HTTP method and API path. A token can have multiple scopes. Token scopes act as a whitelist, and the API server checks the HTTP method and the API path of every request against the scopes of the request token. Scopes are also described on the \"API Authorization\":{{site.baseurl}}/api/tokens.html#scopes page of the \"API documentation\":{{site.baseurl}}/api/index.html.\n\nThese examples use @/arvados/v1/collections@, but can be applied to any endpoint. Consult the \"API documentation\":{{site.baseurl}}/api/index.html to determine the endpoints for specific methods.\n\nThe scope @[\"GET\", \"/arvados/v1/collections\"]@ will allow only GET or HEAD requests for the list of collections. Any other HTTP method or path (including requests for a specific collection record, eg a request with path @/arvados/v1/collections/zzzzz-4zz18-0123456789abcde@) will return a permission error.\n\nA trailing slash in a scope is signficant. The scope @[\"GET\", \"/arvados/v1/collections/\"]@ will allow only GET or HEAD requests *starting with* @/arvados/v1/collections/@. A request for an individual record path @/arvados/v1/collections/zzzzz-4zz18-0123456789abcde@) is allowed but a request to list collections (@/arvados/v1/collections@) will be denied because it does not end with @/@ (API requests with a trailing @/@ will have the slash stripped before the scope is checked.)\n\nThe scope can include an object uuid. The scope @[\"GET\", \"/arvados/v1/collections/zzzzz-4zz18-0123456789abcde\"]@ only permits requests to read the record @zzzzz-4zz18-0123456789abcde@.\n\nSince a token can have multiple scopes, use @[[\"GET\", \"/arvados/v1/collections\"], [\"GET\", \"/arvados/v1/collections/\"]]@ to allow both listing collections and fetching individual collection records. This will reject requests to create or change collections, or access any other API method.\n\nObject create calls use the @POST@ method. A scope of @[\"POST\", \"/arvados/v1/collections\"]@ will allow creating collections, but not reading, listing or updating them (or accessing anything else).\n\nObject update calls use the @PATCH@ method. A scope of @[\"PATCH\", \"/arvados/v1/collections/\"]@ will allow updating collections, but not listing or creating them. (Note: while GET requests are denied an object can be read indirectly by using an empty PATCH which will return the unmodified object as the result).\n\nSimilarly, you can use a scope of @[\"PATCH\", \"/arvados/v1/collections/zzzzz-4zz18-0123456789abcde\"]@ to restrict updates to a single collection.\n\nThere is one special exception to the scope rules: a valid token is always allowed to issue a request to \"@GET /arvados/v1/api_client_authorizations/current@\":{{ site.baseurl }}/api/methods/api_client_authorizations.html#current regardless of its scopes. This allows clients to reliably determine whether a request failed because a token is invalid, or because the token is not permitted to perform a particular request. The API server itself needs to be able to do this to validate tokens issued by other clusters in a federation.\n\nh2. Creating a scoped token\n\nA scoped token can be created at the command line:\n\n\n
$ arv api_client_authorization create --api-client-authorization '{\"scopes\": [[\"GET\", \"/arvados/v1/collections\"], [\"GET\", \"/arvados/v1/collections/\"]]}'\n{\n \"kind\":\"arvados#apiClientAuthorization\",\n \"etag\":\"9yk144t0v6cvyp0342exoh2vq\",\n \"uuid\":\"zzzzz-gj3su-bizbsw0mx5pju3w\",\n \"owner_uuid\":\"zzzzz-tpzed-fr97h9t4m5jffxs\",\n \"created_at\":\"2020-03-12T20:36:12.517375422Z\",\n \"modified_by_user_uuid\":null,\n \"modified_at\":null,\n \"api_token\":\"5a74htnoqwkhtfo2upekpfbsg04hv7cy5v4nowf7dtpxer086m\",\n \"created_by_ip_address\":null,\n \"expires_at\":null,\n \"last_used_at\":null,\n \"last_used_by_ip_address\":null,\n \"scopes\":[\n  [\n   \"GET\",\n   \"/arvados/v1/collections\"\n  ],\n  [\n   \"GET\",\n   \"/arvados/v1/collections/\"\n  ]\n ]\n}\n
\n\n\nThe response will include @api_token@ field which is the newly issued secret token. It can be passed directly to the API server that issued it, or can be used to construct a @v2@ token. A @v2@ format token is required if the token will be used to access other clusters in an Arvados federation. An Arvados @v2@ format token consists of three fields separate by slashes: the prefix @v2@, followed by the token uuid, followed by the token secret. For example: @v2/x1u39-gj3su-bizbsw0mx5pju3w/5a74htnoqwkhtfo2upekpfbsg04hv7cy5v4nowf7dtpxer086m@.\n" }, { "path": "doc/admin/spot-instances.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: Using Preemptible instances\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThis page describes how to enable preemptible instances. Preemptible instances typically offer lower cost computation with a tradeoff of lower service guarantees. If a compute node is preempted, Arvados will restart the computation on a new instance.\n\nCurrently Arvados supports preemptible instances using AWS and Azure spot instances.\n\nh2. Configuration\n\nFirst, configure some @InstanceTypes@ that have @Preemptible: true@. For a preemptible instance, @Price@ determines the maximum bid price; the actual price paid is dynamic and will likely be lower.\n\nTypically you want to add both preemptible and non-preemptible entries for each cloud provider VM type. To do this automatically, use @PreemptiblePriceFactor@ to enable a preemptible version of each listed type, using the given factor to set the maximum bid price relative to the non-preemptible price. Alternatively, you can configure preemptible instance types explicitly. For example, the following two configurations are equivalent:\n\n
\nClusters:\n  ClusterID:\n    Containers:\n      PreemptiblePriceFactor: 0.8\n    InstanceTypes:\n      m4.large:\n        ProviderType: m4.large\n        VCPUs: 2\n        RAM: 8GiB\n        AddedScratch: 32GB\n        Price: 0.1\n
\n\n
\nClusters:\n  ClusterID:\n    InstanceTypes:\n      m4.large:\n        ProviderType: m4.large\n        VCPUs: 2\n        RAM: 8GiB\n        AddedScratch: 32GB\n        Price: 0.1\n      m4.large.preemptible:\n        Preemptible: true\n        ProviderType: m4.large\n        VCPUs: 2\n        RAM: 8GiB\n        AddedScratch: 32GB\n        Price: 0.08\n
\n\nNext, you can choose to enable automatic use of preemptible instances:\n\n
\nClusters:\n  ClusterID:\n    Containers:\n      AlwaysUsePreemptibleInstances: true\n
\n\nIf @AlwaysUsePreemptibleInstances@ is \"true\", child containers (workflow steps) will always select preemptible instances, regardless of user option.\n\nIf @AlwaysUsePreemptibleInstances@ is \"false\" (the default) or unspecified, preemptible instance are \"used when requested by the user.\":{{site.baseurl}}/user/cwl/cwl-run-options.html#preemptible\n\nNote that regardless of the value of @AlwaysUsePreemptibleInstances@, the top level workflow runner container always runs in a reserved (non-preemptible) instance, to avoid situations where the workflow runner is killed requiring the entire to be restarted.\n\nNo additional configuration is required, \"arvados-dispatch-cloud\":{{site.baseurl}}/install/crunch2-cloud/install-dispatch-cloud.html will now start preemptible instances where appropriate.\n\nh3. Cost Tracking\n\nPreemptible instances prices are declared at instance request time and defined by the maximum price that the user is willing to pay per hour. By default, this price is the same amount as the on-demand version of each instance type, and this setting is the one that @arvados-dispatch-cloud@ uses for now, as it doesn't include any pricing data to the spot instance request.\n\nFor AWS, the real price that a spot instance has at any point in time is discovered at the end of each usage hour, depending on instance demand. For this reason, AWS provides a data feed subscription to get hourly logs, as described on \"Amazon's User Guide\":https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-data-feeds.html.\n\nh2. Preemptible instances on AWS\n\nFor general information, see \"using Amazon EC2 spot instances\":https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html.\n\nh3. Permissions\n\nWhen requesting spot instances, Amazon's API may return an authorization error depending on how users and permissions are set on the account. If this is the case check logs for this error:\n\n
\nBaseHTTPError: AuthFailure.ServiceLinkedRoleCreationNotPermitted: The provided credentials do not have permission to create the service-linked role for EC2 Spot Instances.\n
\n\nThe account needs to have a service linked role created. This can be done by logging into the AWS account, go to _IAM Management_ → _Roles_ and create the @AWSServiceRoleForEC2Spot@ role by clicking on the @Create@ button, selecting @EC2@ service and @EC2 - Spot Instances@ use case.\n\nh3. Interruption notices\n\nWhen running a container on a spot instance, Arvados monitors the EC2 metadata endpoint for interruption notices. When an interruption notice is received, it is reported in a log entry in the @crunch-run.txt@ file as well as @warning@ and @preemptionNotice@ keys in the @runtime_status@ field of the affected container.\n\nExample excerpt from @crunch-run.txt@:\n\n
\n2023-02-21T21:12:42.350719824Z Cloud provider scheduled instance stop at 2023-02-21T21:14:42Z\n
\n\nExample @runtime_status@:\n\n
\n{\n  \"warning\": \"preemption notice\",\n  \"warningDetail\": \"Cloud provider scheduled instance stop at 2023-02-21T21:14:42Z\",\n  \"preemptionNotice\": \"Cloud provider scheduled instance stop at 2023-02-21T21:14:42Z\"\n}\n
\n\nh2. Preemptible instances on Azure\n\nFor general information, see \"Use Spot VMs in Azure\":https://docs.microsoft.com/en-us/azure/virtual-machines/spot-vms.\n\nWhen starting preemptible instances on Azure, Arvados configures the eviction policy to 'delete', with max price set to '-1'. This has the effect that preemptible VMs will not be evicted for pricing reasons. The price paid for the instance will be the current spot price for the VM type, up to a maximum of the price for a standard, non-spot VM of that type.\n\nPlease note that Azure provides no SLA for preemptible instances. Even in this configuration, preemptible instances can still be evicted for capacity reasons. If that happens and a container is aborted, Arvados will try to restart it, subject to the usual retry rules.\n\nSpot pricing is not available on 'B-series' VMs, those should not be defined in the configuration file with the _Preemptible_ flag set to true. Spot instances have a separate quota pool, make sure you have sufficient quota available.\n" }, { "path": "doc/admin/storage-classes.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: Configuring storage classes\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nStorage classes (alternately known as \"storage tiers\") allow you to control which volumes should be used to store particular collection data blocks. This can be used to implement data storage policies such as moving data to archival storage.\n\nIn the default Arvados configuration, with no storage classes specified in the configuration file, all volumes belong to a single implicit storage class called \"default\". Apart from that, names of storage classes are internal to the cluster and decided by the administrator. Other than the implicit \"default\" class, Arvados currently does not define any standard storage class names.\n\nTo use multiple storage classes, update the @StorageClasses@ and @Volumes@ sections of your configuration file.\n* Every storage class you use (including \"default\") must be defined in the @StorageClasses@ section.\n* The @StorageClasses@ section must use @Default: true@ to indicate at least one default storage class. When a client/user does not specify storage classes when creating a new collection, the default storage classes are used implicitly.\n* If some storage classes are faster or cheaper to access than others, assign a higher @Priority@ to the faster ones. When reading data, volumes with high priority storage classes are searched first.\n\nExample:\n\n
\n    StorageClasses:\n\n      default:\n        # When reading a block that is stored on multiple volumes,\n        # prefer a volume with this class.\n        Priority: 20\n\n        # When a client does not specify a storage class when saving a\n        # new collection, use this one.\n        Default: true\n\n      archival:\n        Priority: 10\n\n    Volumes:\n\n      ClusterID-nyw5e-000000000000000:\n        # This volume is in the \"default\" storage class.\n        StorageClasses:\n          default: true\n\n      ClusterID-nyw5e-000000000000001:\n        # This volume is in the \"archival\" storage class.\n        StorageClasses:\n          archival: true\n
\n\nRefer to the \"configuration reference\":{{site.baseurl}}/admin/config.html for more details.\n\nh3. Using storage classes\n\n\"Discussed in the user guide\":{{site.baseurl}}/user/topics/storage-classes.html\n\nh3. Storage management notes\n\nWhen uploading data, if a data block cannot be uploaded to all desired storage classes, it will result in a fatal error. Data blocks will not be uploaded to volumes that do not have the desired storage class.\n\nIf you change the storage classes for a collection, the data is not moved immediately. The \"keep-balance\":{{site.baseurl}}/install/install-keep-balance.html service is responsible for deciding which blocks should be placed on which keepstore volumes. As part of the rebalancing behavior, it will determine where a block should go in order to satisfy the desired storage classes, and issue pull requests to copy the block from its original volume to the desired volume. The block will subsequently be moved to trash on the original volume.\n\nIf a block is assigned to multiple storage classes, the block will be stored on @desired_replication@ number of volumes for storage class, even if that results in overreplication.\n\nIf a collection has a desired storage class which is not available in any keepstore volume, the collection's blocks will remain in place, and an error will appear in the @keep-balance@ logs.\n" }, { "path": "doc/admin/token-expiration-policy.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: Automatic logout and token expiration\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nWhen a user logs in to Workbench, they receive a newly created token (a long string of random characters) which grants access to the Arvados API on behalf of that user. In the default configuration, this token does not expire until the user explicitly logs out.\n\nSecurity policies, such as those required to comply with regulations such as HIPAA and GxP, may include policies for \"automatic logoff\". In order to limit the window of risk associated with unauthorized access of the desktop of an Arvados user, or a token being leaked, Arvados offers options for automatic logout from the web app, and to configure access tokens to expire by default.\n\nThe @Workbench.IdleTimeout@, @Login.TokenLifetime@, and @API.MaxTokenLifetime@ options give the administrator ways to control automatic expiration of tokens granted through the login flow.\n\nIf you are looking for information on how to expire a token manually, see how to \"delete a single token\":user-management-cli.html#delete-token and \"delete all tokens belonging to a user\":user-management-cli.html#delete-all-tokens .\n\nh2. Automatic logout\n\nUse @Workbench.IdleTimeout@ to configure Workbench 2 for automatic logout after a period of idle time. For example, this configuration would log the user out after five minutes of no keyboard or pointer activity:\n\n
\nClusters:\n  zzzzz:\n    ...\n    Workbench:\n      IdleTimeout: 5m\n    ...\n
\n\nWhen idle timeout is set, several behaviors and considerations apply:\n\n* The user will be automatically logged out after a period of inactivity. When the automatic logout happens, the token associated with that session will be revoked.\n* Users should use the \"open in new tab\" functionality of Workbench 2. This will share the same token between tabs without requiring the user to log in again. Logging out will apply to all browser tabs that use the same token.\n* If the user closes a Workbench tab without first logging out, the browser will forget the token, but not expire the token (this is desirable if the user has several tabs open).\n* If the user closes all Workbench tabs, they will be required to log in again.\n* This only affects browser behavior. Automatic logout should be used together automatic token expiration described below.\n\nThe default value for @Workbench.IdleTimeout@ is zero, which disables auto-logout.\n\nh2. Automatic expiration of login tokens\n\nUse @Login.TokenLifetime@ to set the lifetime for tokens issued through the login process. This is the maximum amount of time a user can maintain a session before having to log in again. This setting applies to both regular and admin user logins. Here is an example configuration that would require the user to log in again after 12 hours:\n\n
\nClusters:\n  zzzzz:\n    ...\n    Login:\n      TokenLifetime: 12h\n    ...\n
\n\nThis is independent of @Workbench.IdleTimeout@. Even if Workbench auto-logout is disabled, this option will ensure that the user is always required to log in again after the configured amount of time.\n\nThe default value of @Login.TokenLifetime@ is zero, meaning login tokens do not expire (unless @API.MaxTokenLifetime@ is set).\n\nh2. Untrusted login tokens\n\n
\nClusters:\n  zzzzz:\n    ...\n    Login:\n      IssueTrustedTokens: false\n    ...\n
\n\nWhen @IssueTrustedTokens@ is @false@, tokens are \"untrusted\" and cannot be used to list other tokens issued to the same user, nor to grant new tokens. This prevents an attacker from leveraging a leaked token to aquire other tokens, but also interferes with some Workbench features that create new tokens on behalf of the user.\n\nh2. Automatic expiration of all tokens\n\nUse @API.MaxTokenLifetime@ to set the maximum lifetime for any access token created by regular (non-admin) users. For example, this configuration would require that all tokens expire after 24 hours:\n\n
\nClusters:\n  zzzzz:\n    ...\n    API:\n      MaxTokenLifetime: 24h\n    ...\n
\n\nTokens created without an explicit expiration time, or that exceed maximum lifetime, will be set to @API.MaxTokenLifetime@.\n\nSimilar to @Login.TokenLifetime@, this option ensures that the user is always required to log in again after the configured amount of time.\n\nUnlike @Login.TokenLifetime@, this applies to all API operations that manipulate tokens, regardless of whether the token was created by logging in, or by using the API. If @Login.TokenLifetime@ is greater than @API.MaxTokenLifetime@, MaxTokenLifetime takes precedence.\n\nAdmin users are permitted to create tokens with expiration times further in the future than @MaxTokenLifetime@.\n\nThe default value @MaxTokenLifetime@ is zero, which means there is no maximum token lifetime.\n\nh2. Choosing a policy\n\n@Workbench.IdleTimeout@ only affects browser behavior. It is strongly recommended that automatic browser logout be used together with @Login.TokenLifetime@, which is enforced on API side.\n\n@IssueTrustedTokens: true@ (default value) is less restrictive. Be aware that an unrestricted token can be \"refreshed\" to gain access for an indefinite period. This means, during the window that the token is valid, the user is permitted to create a new token, which will have a new expiration further in the future (of course, once the token has expired, this is no longer possible). Unrestricted tokens are required for some Workbench features, as well as ease of use in other contexts, such as the Arvados command line. This option is recommended if many users will interact with the system through the command line.\n\n@IssueTrustedTokens: false@ is more restrictive. A token obtained by logging into Workbench cannot be \"refreshed\" to gain access for an indefinite period. However, it interferes with some Workbench features, as well as ease of use in other contexts, such as the Arvados command line. This option is recommended only if most users will only ever interact with the system through Workbench or WebShell. With this configuration, it is still possible to \"create a token at the command line\":user-management-cli.html#create-token using the @SystemRootToken@.\n\nIn every case, admin users may always create tokens with expiration dates far in the future.\n\nThese policies do not apply to tokens created by the API server for the purposes of authorizing a container to run, as those tokens are automatically expired when the container is finished.\n\nh2. Applying policy to existing tokens\n\nIf you have an existing Arvados installation and want to set a token lifetime policy, there may be long-lived user tokens already granted. The administrator can use the following @rake@ tasks to enforce the new policy.\n\nThe @db:check_long_lived_tokens@ task will list which users have tokens with no expiration date.\n\n\n
# bin/rake db:check_long_lived_tokens\nFound 6 long-lived tokens from users:\nuser2,user2@example.com,zzzzz-tpzed-5vzt5wc62k46p6r\nadmin,admin@example.com,zzzzz-tpzed-6drplgwq9nm5cox\nuser1,user1@example.com,zzzzz-tpzed-ftz2tfurbpf7xox\n
\n\n\nTo apply the new policy to existing tokens, use the @db:fix_long_lived_tokens@ task.\n\n\n
# bin/rake db:fix_long_lived_tokens\nSetting token expiration to: 2020-08-25 03:30:50 +0000\n6 tokens updated.\n
\n\n\nNOTE: These rake tasks adjust the expiration of all tokens except those belonging to the system root user (@zzzzz-tpzed-000000000000000@). If you have tokens used by automated service accounts that need to be long-lived, you can \"create tokens that don't expire using the command line\":user-management-cli.html#create-token .\n" }, { "path": "doc/admin/upgrading.html.textile.liquid", "content": "---\nlayout: default\nnavsection: installguide\ntitle: \"Arvados upgrade notes\"\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nFor Arvados administrators, this page will cover what you need to know and do in order to ensure a smooth upgrade of your Arvados installation. For general release notes covering features added and bugs fixed, see \"Arvados releases\":https://arvados.org/releases.\n\nUpgrade instructions can be found at \"Maintenance and upgrading\":{{site.baseurl}}/admin/maintenance-and-upgrading.html#upgrading.\n\nh2. Upgrade notes\n\nSome versions introduce changes that require special attention when upgrading: e.g., there is a new service to install, or there is a change to the default configuration that you might need to override in order to preserve the old behavior. These notes are listed below, organized by release version. Scroll down to the version number you are upgrading to.\n\n{% comment %}\nNote to developers: Add new items at the top. Include the date, issue number, commit, and considerations/instructions for those about to upgrade.\n\nTODO: extract this information based on git commit messages and generate changelogs / release notes automatically.\n{% endcomment %}\n\n\n
\n\n\nh2(#main). development main\n\n\"previous: Upgrading to 3.2.1\":#v3_2_1\n\nh3. @CUDA@ configuration in @InstanceTypes@ is no longer supported\n\nArvados 3.1 introduced general \"GPU\" instance type settings and runtime constraints to replace vendor-specific \"CUDA\" settings. To support new cloud dispatch features, Arvados 3.3 removes support for the old @CUDA@ configuration sections of the cluster's configured @InstanceTypes@. Before you upgrade to Arvados 3.3, \"update your cluster's configured @InstanceTypes@ with @GPU@ sections\":#v3_1_0_instance_type_cuda. Make sure that each @GPU@ section has an accurate @VRAM@ setting. Without this, the cloud dispatcher may not consider your instance types suitable for GPU-accelerated workflows. You can update and test your current configuration before you start with the rest of your upgrade planning.\n\n@cuda@ runtime constraints in container requests are still deprecated but supported in this release.\n\nh3. Cloud dispatcher runs multiple containers per instance\n\nThe cloud dispatcher will now run multiple containers at once on an instance if it has enough RAM and VCPUs to do so _and_ the instance type is suitable for each container individually.\n\nThese conditions can arise in the following cases:\n* the configured @MaximumPriceFactor@ is large enough that an instance type with more than the exact number of VCPUs specified by a container is considered suitable; or\n* no configured instance type has exactly the number of VCPUs specified by a container, _i.e.,_ every suitable instance type has one or more spare VCPUs available to run another container alongside it.\n\nTo disable the new behavior and ensure that each container runs on a dedicated instance even in the above cases, set the new configuration entry @Containers.MaxRunningContainersPerInstance@ to @1@.\n\nh2(#v3_2_1). v3.2.1 (2026-03-02)\n\n\"previous: Upgrading to 3.2.0\":#v3_2_0\n\nh3. Configuration URLs are stricter about bracketed addresses\n\nThe new version of Go used to build this release has stricter URL parsing: _only_ an IPv6 address can appear inside brackets around the network location of the URL. If your cluster configuration includes URLs with brackets around DNS hostnames or IPv4 addresses, remove those brackets before upgrading.\n\nh2(#v3_2_0). v3.2.0 (2025-11-03)\n\n\"previous: Upgrading to 3.1.2\":#v3_1_2\n\nh3. Debian 11 and Ubuntu 20.04 are no longer supported\n\nArvados 3.2 no longer supports some of the older distributions supported by Arvados 3.1: Debian 11 \"bullseye\" and Ubuntu 20.04 \"focal.\" If you are running Arvados on any of these distributions, you must first upgrade to a supported distribution before you upgrade to Arvados 3.2.\n\nArvados 3.1 supports Debian 12 \"bookworm\" and Ubuntu 22.04 \"jammy.\" You can upgrade your Arvados cluster to one of those releases, then proceed to upgrade Arvados to 3.2.\n\nThe list of distributions supported by Arvados 3.2 can be found on the \"planning and prerequisites page\":{{site.baseurl}}/install/install-manual-prerequisites.html#supportedlinux of the install guide.\n\nh3. RPMs now require Red Hat/AlmaLinux/Rocky Linux 8.8 or later\n\nOur packages for Red Hat/AlmaLinux/Rocky Linux 8 now depend on appstreams in version 8.8, particularly Python 3.11. Please make sure you are running at least version 8.8 of your distribution before you upgrade to Arvados 3.2.0. If not, you should follow your distributor's instructions to upgrade from your current 8.x release to 8.8 or later before you upgrade Arvados.\n\nh3. New GPG key URL for Red Hat, AlmaLinux, and Rocky Linux\n\nAs part of adding support for the RHEL 9 family of distributions, we have started using a new signing key for packages. For these distributions, the key corresponding to your distribution is now available at a URL that includes the release version. Before you upgrade, on each system where you have the Arvados package repository installed, edit the file with that repository configuration, usually @/etc/yum.repos.d/arvados.repo@. Find the line that defines @gpgkey@:\n\n\n
[arvados]\n…\ngpgkey=https://rpm.arvados.org/RHEL/RPM-GPG-KEY-arvados\n
\n\n\nEdit this line to add @$releasever/@ after @RHEL/@, so it looks like this:\n\n\n
gpgkey=https://rpm.arvados.org/RHEL/$releasever/RPM-GPG-KEY-arvados\n
\n\n\nThen save and close the file. The old key URL still works, so this step is not required to upgrade Arvados itself. However, doing it now will help ensure you retain access to the Arvados repositories next time you upgrade your distribution.\n\nh3. SbatchArgumentsList (SLURM) configuration semantics have changed\n\n@Containers.SLURM.SbatchArgumentsList@ must now specify arguments that were previously added implicitly. Also, the @%@ character invokes template behavior. If your current configuration looks like this:\n\n\n
SbatchArgumentsList: [\"--clusters=all\"]\n
\n\n\nYou must update it to add the arguments that were previously added implicitly:\n\n\n
SbatchArgumentsList: [\"--job-name=%U\", \"--mem=%M\", \"--cpus-per-task=%C\", \"--tmp=%T\", \"--constraint=instancetype=%I\", \"--partition=%P\", \"--clusters=all\"]\n
\n\n\nIf your configuration file does not have an @SbatchArgumentsList@ entry, you do not need to add one.\n\nh3. @Users.SendUserSetupNotificationEmail@ is disabled by default\n\nIf you want to preserve the old default behavior of sending an email to each user when their account has been set up, update your configuration file accordingly.\n\n
\nUsers:\n  SendUserSetupNotificationEmail: true\n
\n\nh3. Admin container shell access is enabled by default\n\n\"Container shell access\":{{ site.baseurl }}/user/debugging/container-shell-access.html by admin users is now enabled by default to make it easier to diagnose workflow issues on new deployments. If you prefer to leave it disabled, update your configuration file accordingly.\n\n
\nContainers:\n  ShellAccess:\n    Admin: false\n
\n\nContainer shell access for non-admin users is still disabled by default.\n\nh3. Configure ExternalURL, DNS, and TLS for container web services\n\nArvados now allows external clients to connect to HTTP services running in containers. To enable this feature:\n* Add a @Services.ContainerWebServices.ExternalURL@ entry to @/etc/arvados/config.yml@ with a wildcard URL, e.g., @https://*.containers.ClusterID.example.com/@\n* Add the wildcard name to the @server_name@ directive in the controller section of your Nginx configuration, e.g., @server_name ClusterID.example.com *.containers.ClusterID.example.com;@\n* Add wildcard DNS records so @*.containers.ClusterID.example.com@ names resolve to the same address(es) as your controller's external URL\n* Update the TLS certificate used by Nginx for @ClusterID.example.com@ so it also validates for @*.containers.ClusterID.example.com@\n\nh3. Loki credentials in @local.params.secrets@ are no longer needed\n\nSalt installer's Terraform code replaces the use of AWS access key and secret for Loki's S3 bucket with equivalent permissions through an instance profile. Once applied, the credentials in the @local.params.secrets@ file will be invalid and can be safely removed.\n\nh3. arvbox and @arvados-server install@ are no longer supported\n\nArvados 3.2 no longer includes the arvbox Docker image and associated tooling. The @arvados-server install@ subcommand has also been removed from this release.\n\nIf you were using arvbox in demo mode, consider installing on a Debian-based virtual machine with our \"single-node Ansible installer\":{{ site.baseurl }}/install/install-single-host.html.\n\nIf you were using arvbox or @arvados-server install@ for development, we now provide an Ansible playbook to install development dependencies on a Debian-based system. Our \"Hacking Prerequisites wiki\":https://dev.arvados.org/projects/arvados/wiki/Hacking_prerequisites has instructions for how to use it.\n\nh2(#v3_1_2). v3.1.2 (2025-05-27)\n\n\"previous: Upgrading to 3.1.1\":#v3_1_1\n\nThere are no changes that require administrator attention in this release.\n\nh2(#v3_1_1). v3.1.1 (2025-04-14)\n\n\"previous: Upgrading to 3.1.0\":#v3_1_0\n\nh3. Clusters using cloud dispatch should rebuild a compute node image\n\nArvados 3.1.1 fixes a handful of bugs in installation tools, particularly for deployments on Ubuntu. If you have already successfully upgraded to 3.1.0, the only thing in this release that affects you is a bug fix in the compute node image builder for cloud deployments. If your cluster uses @arvados-dispatch-cloud@, you should \"build a new compute node following our install guide\":{{ site.baseurl }}/install/crunch2-cloud/install-compute-node.html and configure your cluster to use it. You do not need to upgrade any cluster services; there are no changes to them since 3.1.0.\n\nh2(#v3_1_0). v3.1.0 (2025-03-20)\n\n\"previous: Upgrading to 3.0.0\":#v3_0_0\n\nh3. Rails API server now runs standalone\n\nThe Arvados Rails API server now runs from a standalone Passenger server to simplify deployment. Before upgrading, existing deployments should remove the Rails API server from their nginx configuration. e.g., remove the entire @server@ block with @root /var/www/arvados-api/current/public@ from @/etc/nginx/conf.d/arvados-api-and-controller.conf@. If you customized this deployment at all, the \"updated install instructions\":{{ site.baseurl }}/install/install-api-server.html#railsapi-config explain how to customize the standalone Passenger server. Finally, you'll need to enable the new service by running:\n\n\n
# systemctl enable --now arvados-railsapi.service\n
\n\nh3. Rails API server needs PowerTools on Red Hat, AlmaLinux, and Rocky Linux\n\nThe Arvados Rails API server now needs to be able to link against @libyaml@ development headers. On Red Hat, AlmaLinux, and Rocky Linux, these are provided by the @libyaml-devel@ package in the PowerTools repository. Before you upgrade, make sure you have this repository enabled on the host where you run the Rails API server by running:\n\n\n
# dnf config-manager --set-enabled powertools\n
\n\nh3. \"cuda\" runtime constraint is deprecated in favor of \"gpu\"\n\nArvados 3.1.0 adds support for containers that use AMD ROCm alongside our existing support for NVIDIA CUDA. As part of this, the @cuda@ runtime constraint has been deprecated and replaced with a more general @gpu@ constraint. The requested type of GPU is named in the @stack@ field of this object. Other fields have been carried over from @cuda@ and work the same way. Refer to the \"runtime constraints reference\":{{ site.baseurl }}/api/methods/container_requests.html#runtime_constraints for details.\n\nIf client software creates or updates a container request with a @cuda@ runtime constraint, the Arvados API server will automatically translate that to a @gpu@ constraint. This client software should still be updated to specify a @gpu@ runtime constraint, but you can safely upgrade to Arvados 3.1.0 and do these updates opportunistically.\n\nClient software that reads and reports runtime constraints (like Workbench does) must be updated to read the new @gpu@ constraint. The @cuda@ constraint will no longer appear in API responses.\n\nh3(#v3_1_0_instance_type_cuda). Generalized configuration for GPU compute nodes\n\nAs part of adding support for AMD GPUs in Arvados 3.1, the @CUDA@ section of @InstanceType@ definitions is now deprecated in favor of a new @GPU@ section that is generalized for both CUDA and ROCm.\n\nWhere previously there would be a @CUDA@ section:\n\n
\n  InstanceTypes:\n     gpuInstance:\n       CUDA:\n         DriverVersion: \"11.0\"\n         HardwareCapability: \"9.0\"\n         DeviceCount: 1\n
\n\nThe configuration file should now be updated to use a @GPU@ section:\n\n* Rename the section from @CUDA@ to @GPU@.\n* Rename the setting @HardwareCapability@ to @HardwareTarget@. The value can remain unchanged.\n* Add the setting @Stack: \"cuda\"@.\n* Add a @VRAM@ setting that defines the amount of RAM available on the GPU.\n\n\n
  InstanceTypes:\n     gpuInstance:\n       GPU:\n         Stack: \"cuda\"\n         DriverVersion: \"11.0\"\n         HardwareTarget: \"9.0\"\n         DeviceCount: 1\n         VRAM: 8GiB\n
\n\n\nTo minimize disruption, the config loader will continue to accept the deprecated @CUDA@ field and a emit warning. Admins are advised to update the configuration file as the legacy field will be removed in a future version.\n\nh3. BsubCUDAArguments renamed to BsubGPUArguments\n\nThe configuration item @Containers.LSF.BsubCUDAArguments@ has been renamed to @Containers.LSF.BsubGPUArguments@. There is no change in content. To minimize disruption, the config loader will continue to accept the deprecated @BsubCUDAArguments@ field and a emit warning. Admins are advised to update the configuration file as the legacy field will be removed in a future version.\n\nh2(#v3_0_0). v3.0.0 (2024-11-12)\n\n\"previous: Upgrading to 2.7.4\":#v2_7_4\n\nh3. Debian 10 and Ubuntu 18.04 are no longer supported\n\nArvados 3.0 no longer supports some of the older distributions supported by Arvados 2.7: Debian 10 \"buster\" and Ubuntu 18.04 \"bionic.\" If you are running Arvados on any of these distributions, you must first upgrade to a supported distribution before you upgrade to Arvados 3.0.\n\nArvados 2.7 supports Debian 11 \"bullseye\" and Ubuntu 20.04 \"focal.\" You can upgrade your Arvados cluster to one of those releases, then proceed to upgrade Arvados to 3.0.\n\nThe list of distributions supported by Arvados 3.0 can be found on \"Planning and prerequisites.\":{{site.baseurl}}/install/install-manual-prerequisites.html#supportedlinux\n\nh3. Red Hat 8 package dependency on package streams\n\nThe Red Hat 8 package of the Rails API server now depends on the Ruby 3.1 stream, and the various Python packages now depend on the Python 3.9 stream. Plan for these streams to be activated and installed automatically during your upgrade.\n\nh3. RVM is no longer supported\n\nSome Arvados packages, most notably the Rails API server package @arvados-api-server@, would check whether RVM is installed on the system, and invoke Ruby commands through it if so. Arvados 3.0 no longer specially supports RVM. Instead, Arvados 3.0 supports all the different versions of Ruby that are packaged in our supported distributions, mitigating the need to support separate Ruby installations. Package scripts run plain @ruby@ and @gem@ commands and expect they come from a supported version.\n\nIf you have a custom install that requires a different version of Ruby than the one included with your distribution, you must configure your system to ensure package scripts find that version of @ruby@ before any others. For example, you might do this on Debian-based distributions by customizing apt's @DPkg::Path@ setting.\n\nh3. Keep-web requires PostgreSQL database access\n\nThe keep-web service now connects directly to the PostgreSQL database. Make sure these connections are supported by your network firewall rules, PostgreSQL connection settings, and PostgreSQL server configuration (in @pg_hba.conf@) as shown in the \"PostgreSQL install instructions\":{{site.baseurl}}/install/install-postgresql.html.\n\nh3. Slow migration on upgrade\n\nThis upgrade includes a database schema update to rebuild full text search indexes to remove UUID and portable data hash column data. This will provide better search results to users and take less space on the database, but plan for the @arvados-api-server@ package upgrade to take longer than usual.\n\nh3. WebDAV service uses @/var/cache@ for file content\n\nWhen running as root, @keep-web@ now stores copies of recently accessed data blocks in @/var/cache/arvados/keep@ instead of in memory. This directory is created automatically. The default cache size is 10% of the filesystem size. Use the new @Collections.WebDAVCache.DiskCacheSize@ config to specify a different percentage or an absolute size. If @keep-web@ is not running as root, it will store the cache in @$HOME/.cache/arvados/keep@.\n\nIf the previously supported @MaxBlockEntries@ config is present, remove it to avoid warning messages at startup.\n\nh3. Python SDK reorganization of internal classes and modules\n\nWe have reorganized the Python SDK to make it clearer which APIs are intended to be public, and make it easier to find documentation for them. As part of this work, some modules that only included internal support code have been moved, most notably @arvados.diskcache@, @arvados.http_to_keep@, and @arvados.timer@.\n\nIf you need immediate access to these modules, you can find them under @arvados._internal@, but we do not intend to support them as part of our public SDK API, so they may change or be removed entirely in future versions. If you've written client software that relies on these modules, please \"file an issue\":https://github.com/arvados/arvados/issues/new to let us know so we can figure out how best to support you.\n\nh3. Virtual environments inside distribution Python packages have moved\n\nThe distribution packages that we publish for Python packages include an entire virtualenv with all required libraries. In Arvados 3.0 these virtualenvs have moved from @/usr/share/python3/dist/PACKAGE_NAME@ to @/usr/lib/PACKAGE_NAME@ to prevent conflicts with distribution packages and better conform to filesystem standards.\n\nIf you only run the executables installed by these packages, you don't need to change anything. Those are still installed under @/usr/bin@ and will use the new location when you upgrade. If you have written your own scripts or tools that rely on these virtualenvs, you may need to update those with the new location. For example, if you have a shell script that activates the virtualenv by running:\n\n
source /usr/share/python3/dist/python3-arvados-python-client/bin/activate
\n\nYou must update it to:\n\n\n
source /usr/lib/python3-arvados-python-client/bin/activate
\n\n\nIf you have a Python script with this shebang line:\n\n
#!/usr/share/python3/dist/python3-arvados-python-client/bin/python
\n\nYou must update it to:\n\n\n
#!/usr/lib/python3-arvados-python-client/bin/python
\n\n\nh3. costanalyzer subcommand replaced by Arvados cluster activity tool\n\nThe functionality of @arvados-client costanalyzer@ has been replaced by a new @arvados-cluster-activity@ tool. More information can be found at \"Analyzing workflow cost\":{{site.baseurl}}/user/cwl/costanalyzer.html .\n\nh3. @arv-migrate-docker19@ tool removed\n\nThe @arv-migrate-docker19@ tool that updates images from Docker 1.9 to be used with Docker 1.10+ (released February 2016) has been removed. In the unlikely event you still need to \"run this migration\":https://doc.arvados.org/v2.7/install/migrate-docker19.html, please do so before you upgrade to Arvados 3.0.\n\nh3. Legacy APIs and response fields have been removed\n\nThe following APIs have been removed:\n* \"api_clients\":https://doc.arvados.org/v2.7/api/methods/api_clients.html\n* \"humans\":https://doc.arvados.org/v2.7/api/methods/humans.html\n* \"jobs\":https://doc.arvados.org/v2.7/api/methods/jobs.html\n* \"job_tasks\":https://doc.arvados.org/v2.7/api/methods/job_tasks.html\n* \"nodes\":https://doc.arvados.org/v2.7/api/methods/nodes.html\n* \"pipeline_instances\":https://doc.arvados.org/v2.7/api/methods/pipeline_instances.html\n* \"pipeline_templates\":https://doc.arvados.org/v2.7/api/methods/pipeline_templates.html\n* \"repositories\":https://doc.arvados.org/v2.7/api/methods/repositories.html, and \"keep_disks\":https://doc.arvados.org/v2.7/api/methods/keep_disks.html\n* \"specimens\":https://doc.arvados.org/v2.7/api/methods/specimens.html\n* \"traits\":https://doc.arvados.org/v2.7/api/methods/traits.html\n\nThe following fields are no longer returned in API responses.\n* @api_client_id@, @user_id@, @default_owner_uuid@ (\"api_client_authorizations\":{{site.baseurl}}/api/methods/api_client_authorizations.html API)\n* @modified_by_client_uuid@ (all APIs)\n\nh3. Configuration entries have been removed or renamed\n\nThe following configuration keys have been renamed or removed. Renamed keys will still be loaded if they appear with their old names, but you should update your @/etc/arvados/config.yml@ file to avoid warnings when services start up.\n* @API.LogCreateRequestFraction@ has been removed\n* @Containers.JobsAPI.Enable@ has been removed\n* @Mail.EmailFrom@ has been removed\n* @Mail.IssueReporterEmailFrom@ has been removed\n* @Mail.IssueReporterEmailTo@ has been removed\n* @Mail.MailchimpAPIKey@ has been removed\n* @Mail.MailchimpListID@ has been removed\n* @Mail.SendUserSetupNotificationEmail@ has moved to @Users.SendUserSetupNotificationEmail@\n* @Mail.SupportEmailAddress@ has moved to @Users.SupportEmailAddress@\n\nh3. S3 volume IAMRole configuration entry has been removed\n\nThe @Volumes.*.DriverParameters.IAMRole@ configuration entry for S3 volumes has been removed. You should remove it from your @/etc/arvados/config.yml@ file to avoid warnings when services start up. As before, if @AccessKeyID@ and @SecretAccessKey@ are blank, keepstore will retrieve IAM role credentials from instance metadata. Previously, documentation indicated that keepstore would refuse to use the IAM credentials if @IAMRole@ was specified and did not match the instance metadata, but that check has not been working for some time.\n\nh3. Legacy container logging system has been removed\n\nThe following configuration keys are no longer supported. Remove them from your @/etc/arvados/config.yml@ file to avoid warnings when services start up.\n* @Containers.Logging.LimitLogBytesPerJob@\n* @Containers.Logging.LogBytesPerEvent@\n* @Containers.Logging.LogPartialLineThrottlePeriod@\n* @Containers.Logging.LogSecondsBetweenEvents@\n* @Containers.Logging.LogThrottleBytes@\n* @Containers.Logging.LogThrottleLines@\n* @Containers.Logging.LogThrottlePeriod@\n* @Containers.Logging.MaxAge@\n* @Containers.Logging.SweepInterval@\n\nAny container logging content remaining in the database from the legacy system will be deleted.\n\nh2(#v2_7_4). v2.7.4 (2024-07-08)\n\n\"previous: Upgrading to 2.7.3\":#v2_7_3\n\nStarting from 2.7.4, Arvados no longer supports CentOS. CentOS users should migrate to an Arvados-supported version of Red Hat Enterprise Linux (RHEL), Rocky Linux or AlmaLinux.\n\nThere are no other configuration changes requiring administrator attention in this release.\n\nh2(#v2_7_3). v2.7.3 (2024-05-24)\n\n\"previous: Upgrading to 2.7.2\":#v2_7_2\n\nThere are no configuration changes requiring administrator attention in this release.\n\nh2(#v2_7_2). v2.7.2 (2024-04-09)\n\n\"previous: Upgrading to 2.7.1\":#v2_7_1\n\nh3. Check MaxGatewayTunnels config\n\nIf you use the LSF or Slurm dispatcher, ensure the new @API.MaxGatewayTunnels@ config entry is high enough to support the size of your cluster. See \"LSF docs\":{{site.baseurl}}/install/crunch2-lsf/install-dispatch.html#MaxGatewayTunnels or \"Slurm docs\":{{site.baseurl}}/install/crunch2-slurm/install-dispatch.html#MaxGatewayTunnels for details.\n\nh3. New LSF dispatcher config items MaxRunTimeOverhead and MaxRunTimeDefault\n\nNow supports configuration parameter @Containers.LSF.MaxRunTimeDefault@ as the default value for @max_run_time@ for containers that do not specify a time limit (using CWL @ToolTimeLimit@).\n\nNow supports configuration parameter @Containers.LSF.MaxRunTimeOverhead@ so that when @scheduling_constraints.max_run_time@ or @MaxRunTimeDefault@ are non-zero, this adds time to account for crunch-run startup/shutdown overhead.\n\nh2(#v2_7_1). v2.7.1 (2023-12-12)\n\n\"previous: Upgrading to 2.7.0\":#v2_7_0\n\nh3. Separate configs for MaxConcurrentRequests and MaxConcurrentRailsRequests\n\nThe default configuration value @API.MaxConcurrentRequests@ (the number of concurrent requests that will be processed by a single instance of an arvados service process) is raised from 8 to 64.\n\nA new configuration key @API.MaxConcurrentRailsRequests@ (default 8) limits the number of concurrent requests processed by a RailsAPI service process.\n\nh3. Remove Workbench1 packages after upgrading the salt installer\n\nIf you installed a previous version of Arvados with the Salt installer, and you upgrade your installer to upgrade the cluster, you should uninstall the @arvados-workbench@ package from the workbench instance afterwards.\n\nh3. Remove Workbench1 packages and configuration\n\nThe Workbench1 application has been removed from the Arvados distribution. We recommend the following follow-up steps.\n* Remove the Workbench1 package from any service node where it is installed (e.g., @apt remove arvados-workbench@).\n* In your Nginx configuration, add your Workbench1 URL host (from @Services.Workbench1.ExternalURL@) to the @server_name@ directive in the Workbench2 section. For example: 
server {\n  listen 443 ssl;\n  server_name workbench.ClusterID.example.com workbench2.ClusterID.example.com;\n  ...\n}
\n* In your Nginx configuration, remove the @upstream@ and @server@ sections for Workbench1.\n* Remove the @Services.Workbench1.InternalURLs@ section of your configuration file. (Do not remove @ExternalURL@.)\n* Run @arvados-server config-check@ to identify any Workbench1-specific entries in your configuration file, and remove them.\n\nh3. Check implications of Containers.MaximumPriceFactor 1.5\n\nWhen scheduling a container, Arvados now considers using instance types other than the lowest-cost type consistent with the container's resource constraints. If a larger instance is already running and idle, or the cloud provider reports that the optimal instance type is not currently available, Arvados will select a larger instance type, provided the cost does not exceed 1.5x the optimal instance type cost.\n\nThis will typically reduce overall latency for containers and reduce instance booting/shutdown overhead, but may increase costs depending on workload and instance availability. To avoid this behavior, configure @Containers.MaximumPriceFactor: 1.0@.\n\nh3. Synchronize keepstore and keep-balance upgrades\n\nThe internal communication between keepstore and keep-balance about read-only volumes has changed. After keep-balance is upgraded, old versions of keepstore will be treated as read-only. We recommend upgrading and restarting all keepstore services first, then upgrading and restarting keep-balance.\n\nh3. Separate configs for MaxConcurrentRequests and MaxConcurrentRailsRequests\n\nThe default configuration value @API.MaxConcurrentRequests@ (the number of concurrent requests that will be processed by a single instance of an arvados service process) is raised from 8 to 64.\n\nA new configuration key @API.MaxConcurrentRailsRequests@ (default 8) limits the number of concurrent requests processed by a RailsAPI service process.\n\nh2(#v2_7_0). v2.7.0 (2023-09-21)\n\n\"previous: Upgrading to 2.6.3\":#v2_6_3\n\nh3. New system for live container logs\n\nStarting with Arvados 2.7, a new system for fetching live container logs is in place. This system features significantly reduced database load compared to previous releases. When Workbench or another application needs to access the logs of a process (running or completed), they should use the \"log endpoint of container_requests\":{{ site.baseurl }}/api/methods/container_requests.html which forwards requests to the running container. This supersedes the previous system where compute processes would send all of their logs to the database, which produced significant load.\n\nThe legacy logging system is now disabled by default for all installations with the setting @Containers.Logging.LimitLogBytesForJob: 0@. If you have an existing Arvados installation where you have customized this value and do not need the legacy container logging system, we recommend removing @LimitLogBytesForJob@ from your configuration.\n\nIf you need to re-enable the legacy logging system, set @Containers.Logging.LimitLogBytesForJob@ to a positive value (the previous default was @Containers.Logging.LimitLogBytesForJob: 67108864@).\n\nh3. Workbench 1 deprecated\n\nThe original Arvados Workbench application (referred to as \"Workbench 1\") is deprecated and will be removed in a future major version of Arvados. Users are advised to migrate to \"Workbench 2\". Starting with this release, new installations of Arvados will only set up Workbench 2 and no longer include Workbench 1 by default.\n\nIt is also important to note that Workbench 1 only supports the legacy logging system, which is now disabled by default. If you need to re-enable the legacy logging system, see above.\n\nh3. Multi-node installer's domain name configuration changes\n\nThe @domain_name@ variable at @terraform/vpc/terraform.tfvars@ and @DOMAIN@ variable at @local.params@ changed their meaning. In previous versions they were used in combination with @cluster_name@ and @CLUSTER@ to build the cluster's domain name (e.g.: @cluster_name@.@domain_name@). To allow the use of any arbitrary cluster domain, now we don't enforce using the cluster prefix as part of the domain, so @domain_name@ and @DOMAIN@ need to hold the entire domain for the given cluster.\nFor example, if @cluster_name@ is set to @\"xarv1\"@ and @domain_name@ was previously set to @\"example.com\"@, it should now be set to @\"xarv1.example.com\"@ to keep using the same cluster domain.\n\nh3. Crunchstat log format change\n\nThe reported number of CPUs available in a container is now formatted in @crunchstat.txt@ log files and @crunchstat-summary@ text reports as a floating-point number rather than an integer (@2.00 cpus@ rather than @2 cpus@). Programs that parse these files may need to be updated accordingly.\n\nh3. arvados-login-sync configuration changes, including ignored groups\n\nIn the @Users@ section of your cluster configuration, there are now several options to control what system resources are or are not managed by @arvados-login-sync@. These options all have names that begin with @Sync@.\n\nThe defaults for all of these options match the previous behavior of @arvados-login-sync@ _except_ for @SyncIgnoredGroups@. This list names groups that @arvados-login-sync@ will never modify by adding or removing members. As a security precaution, the default list names security-sensitive system groups on Debian- and Red Hat-based distributions. If you are using Arvados to manage system group membership on shell nodes, especially @sudo@ or @wheel@, you may want to provide your own list. Set @SyncIgnoredGroups: []@ to restore the original behavior of ignoring no groups.\n\nh3. API clients can always retrieve their current token, regardless of scopes\n\nWe have introduced a small exception to the previous behavior of \"Arvados API token scopes\":{{ site.baseurl }}/admin/scoped-tokens.html in this release. A valid token is now always allowed to issue a request to \"@GET /arvados/v1/api_client_authorizations/current@\":{{ site.baseurl }}/api/methods/api_client_authorizations.html#current regardless of its scopes. This allows clients to reliably determine whether a request failed because a token is invalid, or because the token is not permitted to perform a particular request. The API server itself needs to be able to do this to validate tokens issued by other clusters in a federation.\n\nh3. Deprecated/legacy APIs slated for removal\n\nThe legacy APIs \"humans\":https://doc.arvados.org/v2.7/api/methods/humans.html, \"specimens\":https://doc.arvados.org/v2.7/api/methods/specimens.html, \"traits\":https://doc.arvados.org/v2.7/api/methods/traits.html, \"jobs\":https://doc.arvados.org/v2.7/api/methods/jobs.html, \"job_tasks\":https://doc.arvados.org/v2.7/api/methods/job_tasks.html, \"pipeline_instances\":https://doc.arvados.org/v2.7/api/methods/pipeline_instances.html, \"pipeline_templates\":https://doc.arvados.org/v2.7/api/methods/pipeline_templates.html, \"nodes\":https://doc.arvados.org/v2.7/api/methods/nodes.html, \"repositories\":https://doc.arvados.org/v2.7/api/methods/repositories.html, and \"keep_disks\":https://doc.arvados.org/v2.7/api/methods/keep_disks.html are deprecated and will be removed in a future major version of Arvados.\n\nIn addition, the @default_owner_uuid@, @api_client_id@, and @user_id@ fields of \"api_client_authorizations\":../api/methods/api_client_authorizations.html are deprecated and will be removed from @api_client_authorization@ responses in a future major version of Arvados. This should not affect clients as @default_owner_uuid@ was never implemented, and @api_client_id@ and @user_id@ returned internal ids that were not meaningful or usable with any other API call.\n\nh3. UseAWSS3v2Driver option removed\n\nThe old \"v1\" S3 driver for keepstore has been removed. The new \"v2\" implementation, which has been the default since Arvados 2.5.0, is always used. The @Volumes.*.DriverParameters.UseAWSS3v2Driver@ configuration key is no longer recognized. If your config file uses it, remove it to avoid warning messages at startup.\n\nh2(#v2_6_3). v2.6.3 (2023-06-06)\n\nh3. Python SDK automatically retries failed requests much more\n\nThe Python SDK has always provided functionality to retry API requests that fail due to temporary problems like network failures, by passing @num_retries=N@ to a request's @execute()@ method. In this release, API client constructor functions like @arvados.api@ also accept a @num_retries@ argument. This value is stored on the client object and used as a floor for all API requests made with this client. This allows developers to set their preferred retry strategy once, without having to pass it to each @execute()@ call.\n\nThe default value for @num_retries@ in API constructor functions is 10. This means that an API request that repeatedly encounters temporary problems may spend up to about 35 minutes retrying in the worst case. We believe this is an appropriate default for most users, where eventual success is a much greater concern than responsiveness. If you have client applications where this is undesirable, update them to pass a lower @num_retries@ value to the constructor function. You can even pass @num_retries=0@ to have the API client act as it did before, like this:\n\n{% codeblock as python %}\nimport arvados\narv_client = arvados.api('v1', num_retries=0, ...)\n{% endcodeblock %}\n\nThe first time the Python SDK fetches an Arvados API discovery document, it will ensure that @googleapiclient.http@ logs are handled so you have a way to know about early problems that are being retried. If you prefer to handle these logs your own way, just ensure that the @googleapiclient.http@ logger (or a parent logger) has a handler installed before you call any Arvados API client constructor.\n\nh2(#v2_6_2). v2.6.2 (2023-05-22)\n\n\"previous: Upgrading to 2.6.1\":#v2_6_1\n\nThis version introduces a new API feature which is used by Workbench 2 to improve page loading performance. To avoid any errors using the new Workbench with an old API server, be sure to upgrade the API server before upgrading Workbench 2.\n\nh2(#v2_6_1). v2.6.1 (2023-04-17)\n\n\"previous: Upgrading to 2.6.0\":#v2_6_0\n\nh3. Performance improvement for permission row de-duplication migration\n\nThe migration which de-duplicates permission links has been optimized. We recommend upgrading from 2.5.0 directly to 2.6.1 in order to avoid the slow permission de-deplication migration in 2.6.0.\n\nYou should still plan for the arvados-api-server package upgrade to take longer than usual due to the database schema update changing the integer id column in each table from 32-bit to 64-bit.\n\nh2(#v2_6_0). v2.6.0 (2023-04-06)\n\n\"previous: Upgrading to 2.5.0\":#v2_5_0\n\nh3. WebDAV InternalURLs must be reachable from controller nodes\n\nEnsure your internal keep-web service addresses are listed in the @Services.WebDAV.InternalURLs@ section of your configuration file, and reachable from controller processes, as noted on the \"updated install page\":{{site.baseurl}}/admin/config-urls.html.\n\nh3. Slow migration on upgrade\n\nImportant! This upgrade includes a database schema update changing the integer id column in each table from 32-bit to 64-bit. Because it touches every row in the table, on moderate to large sized installations *this may be very slow* (on the order of hours). Plan for the arvados-api-server package upgrade to take longer than usual.\n\nh3. Default request concurrency, new limit on log requests\n\nThe configuration value @API.MaxConcurrentRequests@ (the number of concurrent requests that will be accepted by a single instance of arvados-controller) now has a default value of 64, instead of being unlimited.\n\nNew configuration value @API.LogCreateRequestFraction@ (default 0.50) limits requests that post live container logs to the API server, to avoid situations where log messages crowd out other more important requests.\n\nh3. New limit on concurrent workflows\n\nNew configuration options @CloudVMs.SupervisorFraction@ (default 0.30) limits the number of concurrent workflow supervisors, to avoid situations where too many workflow runners crowds out actual workers.\n\nh3. Default limit for cloud VM instances\n\nThere is a new configuration entry @CloudVMs.MaxInstances@ (default 64) that limits the number of VMs the cloud dispatcher will run at a time. This may need to be adjusted to suit your anticipated workload.\n\nUsing the obsolete configuration entry @MaxCloudVMs@, which was previously accepted in config files but not obeyed, will now result in a deprecation warning.\n\nh3. Default frequency for running keep-balance has changed\n\nThe frequency that @keep-balance@ will run (@Collections.BalancePeriod@) has been changed from every 10 minutes to every 6 hours.\n\nh2(#v2_5_0). v2.5.0 (2022-12-22)\n\n\"previous: Upgrading to 2.4.4\":#v2_4_4\n\nh3. Dispatchers require PostgreSQL database access\n\nAll dispatchers (cloud, LSF, and Slurm) now connect directly to the PostgreSQL database. Make sure these connections are supported by your network firewall rules, PostgreSQL connection settings, and PostgreSQL server configuration (in @pg_hba.conf@) as shown in the \"PostgreSQL install instructions\":{{site.baseurl}}/install/install-postgresql.html.\n\nh3. Google or OpenID Connect login restricted to trusted clients\n\nIf you use OpenID Connect or Google login, and your cluster serves as the @LoginCluster@ in a federation _or_ your users log in from a web application other than the Workbench1 and Workbench2 @ExternalURL@ addresses in your configuration file, the additional web application URLs (e.g., the other clusters' Workbench addresses) must be listed explicitly in @Login.TrustedClients@, otherwise login will fail. Previously, login would succeed with a less-privileged token.\n\nh3. New keepstore S3 driver enabled by default\n\nA more actively maintained S3 client library is now enabled by default for keeepstore services. The previous driver is still available for use in case of unknown issues. To use the old driver, set @DriverParameters.UseAWSS3v2Driver@ to @false@ on the appropriate @Volumes@ config entries.\n\nh3. Old container logs are automatically deleted from PostgreSQL\n\nCached copies of log entries from containers that finished more than 1 month ago are now deleted automatically (this only affects the \"live\" logs saved in the PostgreSQL database, not log collections saved in Keep). If you have an existing cron job that runs @rake db:delete_old_container_logs@, you can remove it. See configuration options @Containers.Logging.MaxAge@ and @Containers.Logging.SweepInterval@.\n\nh3. Fixed salt installer template file to support container shell access\n\nIf you manage your cluster using the salt installer, you may want to update it to the latest version, use the appropriate @config_examples@ subdirectory and re-reploy with your custom @local.params@ file so that the @arvados-controller@'s @nginx@ configuration file gets fixed.\n\nh3. Login-sync script requires configuration update on LoginCluster federations\n\nIf you have @arvados-login-sync@ running on a satellite cluster, please update the environment variable settings by removing the @LOGINCLUSTER_ARVADOS_API_*@ variables and setting @ARVADOS_API_TOKEN@ to a LoginCluster's admin token, as described on the \"updated install page\":{{site.baseurl}}/install/install-shell-server.html#arvados-login-sync.\n\nh3. Renamed keep-web metrics and WebDAV configs\n\nMetrics previously reported by keep-web (@arvados_keepweb_collectioncache_requests@, @..._hits@, @..._pdh_hits@, @..._api_calls@, @..._cached_manifests@, and @arvados_keepweb_sessions_cached_collection_bytes@) have been replaced with @arvados_keepweb_cached_session_bytes@.\n\nThe config entries @Collections.WebDAVCache.UUIDTTL@, @...MaxCollectionEntries@, and @...MaxUUIDEntries@ are no longer used, and should be removed from your config file.\n\nh2(#v2_4_4). v2.4.4 (2022-11-18)\n\n\"previous: Upgrading to 2.4.3\":#v2_4_3\n\nThis update only consists of improvements to @arvados-cwl-runner@. There are no changes to backend services.\n\nh2(#v2_4_3). v2.4.3 (2022-09-21)\n\n\"previous: Upgrading to 2.4.2\":#v2_4_2\n\nh3. Fixed PAM authentication security vulnerability\n\nIn Arvados 2.4.2 and earlier, when using PAM authentication, if a user presented valid credentials but the account is disabled or otherwise not allowed to access the host, it would still be accepted for access to Arvados. From 2.4.3 onwards, Arvados now also checks that the account is permitted to access the host before completing the PAM login process.\n\nOther authentication methods (LDAP, OpenID Connect) are not affected by this flaw.\n\nh2(#v2_4_2). v2.4.2 (2022-08-09)\n\n\"previous: Upgrading to 2.4.1\":#v2_4_1\n\nh3. GHSL-2022-063\n\nGitHub Security Lab (GHSL) reported a remote code execution (RCE) vulnerability in the Arvados Workbench that allows authenticated attackers to execute arbitrary code via specially crafted JSON payloads.\n\nThis vulnerability is fixed in 2.4.2 (\"#19316\":https://dev.arvados.org/issues/19316).\n\nIt is likely that this vulnerability exists in all versions of Arvados up to 2.4.1.\n\nThis vulnerability is specific to the Ruby on Rails Workbench application (\"Workbench 1\"). We do not believe any other Arvados components, including the TypesScript browser-based Workbench application (\"Workbench 2\") or API Server, are vulnerable to this attack.\n\nh3. CVE-2022-31163 and CVE-2022-32224\n\nAs a precaution, Arvados 2.4.2 has includes security updates for Ruby on Rails and the TZInfo Ruby gem. However, there are no known exploits in Arvados based on these CVEs.\n\nh3. Disable Sharing URLs UI\n\nThere is now a configuration option @Workbench.DisableSharingURLsUI@ for admins to disable the user interface for \"sharing link\" feature (URLs which can be sent to users to access the data in a specific collection in Arvados without an Arvados account), for organizations where sharing links violate their data sharing policy.\n\nh2(#v2_4_1). v2.4.1 (2022-06-02)\n\n\"previous: Upgrading to 2.4.0\":#v2_4_0\n\nh3. Slurm dispatcher requires configuration update\n\nIf you use the Slurm dispatcher (@crunch-dispatch-slurm@) you must add a @Services.DispatchSLURM.InternalURLs@ section to your configuration file, as shown on the \"updated install page\":{{site.baseurl}}/install/crunch2-slurm/install-dispatch.html.\n\nh3. New proxy parameters for arvados-controller\n\nWe now recommend disabling nginx proxy caching for arvados-controller, to avoid truncation of large responses.\n\nIn your Nginx configuration file (@/etc/nginx/conf.d/arvados-api-and-controller.conf@), add the following lines to the @location /@ block with @http://controller@ (see \"Update nginx configuration\":{{site.baseurl}}/install/install-api-server.html#update-nginx for an example) and reload/restart Nginx (@sudo nginx -s reload@).\n\n
\n    proxy_max_temp_file_size 0;\n    proxy_request_buffering  off;\n    proxy_buffering          off;\n    proxy_http_version       1.1;\n
\n\nh3. Now recommending Singularity 3.9.9\n\nThe compute image \"build script\":{{site.baseurl}}/install/crunch2-cloud/install-compute-node.html now installs Singularity 3.9.9 instead of 3.7.4. The newer version includes a bugfix that should resolve \"intermittent loopback device errors\":https://dev.arvados.org/issues/18489 when running containers.\n\nh3. Changes to @arvados-cwl-runner --create-workflow@ and @--update-workflow@\n\nWhen using @arvados-cwl-runner --create-workflow@ or @--update-workflow@, by default it will now make a copy of all collection and Docker image dependencies in the target project. Running workflows retains the old behavior (use the dependencies wherever they are found). The can be controlled explicit with @--copy-deps@ and @--no-copy-deps@.\n\nh2(#v2_4_0). v2.4.0 (2022-04-08)\n\n\"previous: Upgrading to 2.3.1\":#v2_3_1\n\nh3. Default result order changed\n\nWhen requesting a list of objects without an explicit @order@ parameter, the default order has changed from @modified_at desc, uuid asc@ to @modified_at desc, uuid desc@. This means that if two objects have identical @modified_at@ timestamps, the tiebreaker will now be based on @uuid@ in decending order where previously it would be ascending order. The practical effect of this should be minor; with microsecond precision it is unusual to have two records with exactly the same timestamp, and order-sensitive queries should already provide an explicit @order@ parameter.\n\nh3. Ubuntu 18.04 Arvados Python packages now depend on python-3.8\n\nUbuntu 18.04 ships with Python 3.6 as the default version of Python 3. Ubuntu also ships a version of Python 3.8, and the Arvados Python packages (@python3-arvados-cwl-runner@, @python3-arvados-fuse@, @python3-arvados-python-client@, @python3-arvados-user-activity@ and @python3-crunchstat-summary@) now depend on the @python-3.8@ system package.\n\nThis means that they are now installed under @/usr/share/python3.8@ (before, the path was @/usr/share/python3@). If you rely on the @python3@ executable from the packages (e.g. to load a virtualenv), you may need to update the path to that executable.\n\nh3. Minimum supported Ruby version is now 2.6\n\nThe minimum supported Ruby version is now 2.6. If you are running Arvados on Debian 10 or Ubuntu 18.04, you may need to switch to using RVM or upgrade your OS. See \"Install Ruby and Bundler\":../install/ruby.html for more information.\n\nh3. Anonymous token changes\n\nThe anonymous token configured in @Users.AnonymousUserToken@ must now be 32 characters or longer. This was already the suggestion in the documentation, now it is enforced. The @script/get_anonymous_user_token.rb@ script that was needed to register the anonymous user token in the database has been removed. Registration of the anonymous token is no longer necessary.\n\nh3. Preemptible instance support changes\n\nThe @Containers.UsePreemptibleInstances@ option has been renamed to @Containers.AlwaysUsePreemptibleInstances@ and has the same behavior when @true@ and one or more preemptible instances are configured. However, a value of @false@ no longer disables support for preemptible instances, instead users can now enable use of preemptible instances at the level of an individual workflow or workflow step.\n\nIn addition, there is a new configuration option @Containers.PreemptiblePriceFactor@ will automatically add a preemptible instance type corresponding to each regular instance type. See \"Using Preemptible instances\":spot-instances.html for details.\n\nh3. Default LSF arguments have changed\n\nIf you use LSF and your configuration specifies @Containers.LSF.BsubArgumentsList@, you should update it to include the new arguments (@\"-R\", \"select[mem>=%MMB]\", ...@, see \"configuration reference\":{{site.baseurl}}/admin/config.html). Otherwise, containers that are too big to run on any LSF host will remain in the LSF queue instead of being cancelled.\n\nh3. Support for NVIDIA CUDA GPUs\n\nArvados now supports requesting NVIDIA CUDA GPUs for cloud and LSF (Slurm is currently not supported). To be able to request GPU nodes, some additional configuration is needed:\n\n\"Including GPU support in cloud compute node image\":{{site.baseurl}}/install/crunch2-cloud/install-compute-node.html#nvidia\n\n\"Configure cloud dispatcher for GPU support\":{{site.baseurl}}/install/crunch2-cloud/install-dispatch-cloud.html#GPUsupport\n\n\"LSF GPU configuration\":{{site.baseurl}}/install/crunch2-lsf/install-dispatch.html\n\nh3. Role groups are visible to all users by default\n\nThe permission model has changed such that all role groups are visible to all active users. This enables users to share objects with groups they don't belong to. To preserve the previous behavior, where role groups are only visible to members and admins, add @RoleGroupsVisibleToAll: false@ to the @Users@ section of your configuration file.\n\nh3. Previously trashed role groups will be deleted\n\nDue to a bug in previous versions, the @DELETE@ operation on a role group caused the group to be flagged as trash in the database, but continue to grant permissions regardless. After upgrading, any role groups that had been trashed this way will be deleted. This might surprise some users if they were relying on permissions that were still in effect due to this bug. Future @DELETE@ operations on a role group will immediately delete the group and revoke the associated permissions.\n\nh3. Dedicated keepstore process for each container\n\nWhen Arvados runs a container via @arvados-dispatch-cloud@, the @crunch-run@ supervisor process now brings up its own keepstore server to handle I/O for mounted collections, outputs, and logs. With the default configuration, the keepstore process allocates one 64 MiB block buffer per VCPU requested by the container. For most workloads this will increase throughput, reduce total network traffic, and make it possible to run more containers at once without provisioning additional keepstore nodes to handle the I/O load.\n* If you have containers that can effectively handle multiple I/O threads per VCPU, consider increasing the @Containers.LocalKeepBlobBuffersPerVCPU@ value.\n* If you already have a robust permanent keepstore infrastructure, you can set @Containers.LocalKeepBlobBuffersPerVCPU@ to 0 to disable this feature and preserve the previous behavior of sending container I/O traffic to your separately provisioned keepstore servers.\n* This feature is enabled only if no volumes use @AccessViaHosts@, and no volumes have underlying @Replication@ less than @Collections.DefaultReplication@. If the feature is configured but cannot be enabled due to an incompatible volume configuration, this will be noted in the @crunch-run.txt@ file in the container log.\n\nh2(#v2_3_1). v2.3.1 (2021-11-24)\n\n\"previous: Upgrading to 2.3.0\":#v2_3_0\n\nh3. Users are visible to other users by default\n\nWhen a new user is set up (either via @AutoSetupNewUsers@ config or via Workbench admin interface) the user immediately becomes visible to other users. To revert to the previous behavior, where the administrator must add two users to the same group using the Workbench admin interface in order for the users to see each other, change the new @Users.ActivatedUsersAreVisibleToOthers@ config to @false@.\n\nh3. Backend support for vocabulary checking\n\nIf your installation uses the vocabulary feature on Workbench2, you will need to update the cluster configuration by moving the vocabulary definition file to the node where @controller@ runs, and set the @API.VocabularyPath@ configuration parameter to the local path where the file was placed.\nThis will enable the vocabulary checking cluster-wide, including Workbench2. The @Workbench.VocabularyURL@ configuration parameter is deprecated and will be removed in a future release.\nYou can read more about how this feature works on the \"admin page\":{{site.baseurl}}/admin/metadata-vocabulary.html.\n\nh2(#v2_3_0). v2.3.0 (2021-10-27)\n\n\"previous: Upgrading to 2.2.0\":#v2_2_0\n\nh3. Ubuntu 18.04 packages for arvados-api-server and arvados-workbench now conflict with ruby-bundler\n\nUbuntu 18.04 ships with Bundler version 1.16.1, which is no longer compatible with the Gemfiles in the Arvados packages (made with Bundler 2.2.19). The Ubuntu 18.04 packages for arvados-api-server and arvados-workbench now conflict with the ruby-bundler package to work around this issue. The post-install scripts for arvados-api-server and arvados-workbench install the proper version of Bundler as a gem.\n\nh3. Removed unused @update_uuid@ endpoint for users.\n\nThe @update_uuid@ endpoint was superseded by the \"link accounts feature\":{{site.baseurl}}/admin/link-accounts.html, so it's no longer available.\n\nh3. Removed deprecated '@@' search operator\n\nThe '@@' full text search operator, previously deprecated, has been removed. To perform a string search across multiple columns, use the 'ilike' operator on 'any' column as described in the \"available list method filter section\":{{site.baseurl}}/api/methods.html#substringsearchfilter of the API documentation.\n\nh3. Storage classes must be defined explicitly\n\nIf your configuration uses the StorageClasses attribute on any Keep volumes, you must add a new @StorageClasses@ section that lists all of your storage classes. Refer to the updated documentation about \"configuring storage classes\":{{site.baseurl}}/admin/storage-classes.html for details.\n\nh3. keep-balance requires access to PostgreSQL\n\nMake sure the keep-balance process can connect to your PostgreSQL server using the settings in your config file. (In previous versions, keep-balance accessed the database through controller instead of connecting to the database server directly.)\n\nh3. crunch-dispatch-local now requires config.yml\n\nThe @crunch-dispatch-local@ dispatcher now reads the API host and token from the system wide @/etc/arvados/config.yml@ . It will fail to start that file is not found or not readable.\n\nh3. Multi-file docker image collections\n\nTypically a docker image collection contains a single @.tar@ file at the top level. Handling of atypical cases has changed. If a docker image collection contains files with extensions other than @.tar@, they will be ignored (previously they could cause errors). If a docker image collection contains multiple @.tar@ files, it will cause an error at runtime, \"cannot choose from multiple tar files in image collection\" (previously one of the @.tar@ files was selected). Subdirectories are ignored. The @arv keep docker@ command always creates a collection with a single @.tar@ file, and never uses subdirectories, so this change will not affect most users.\n\nh2(#v2_2_0). v2.2.0 (2021-06-03)\n\n\"previous: Upgrading to 2.1.0\":#v2_1_0\n\nh3. New spelling of S3 credential configs\n\nIf you use the S3 driver for Keep volumes and specify credentials in your configuration file (as opposed to using an IAM role), you should change the spelling of the @AccessKey@ and @SecretKey@ config keys to @AccessKeyID@ and @SecretAccessKey@. If you don't update them, the previous spellings will still be accepted, but warnings will be logged at server startup.\n\nh3. New proxy parameters for arvados-controller\n\nIn your Nginx configuration file (@/etc/nginx/conf.d/arvados-api-and-controller.conf@), add the following lines to the @location /@ block with @http://controller@ (see \"Update nginx configuration\":{{site.baseurl}}/install/install-api-server.html#update-nginx for an example) and reload/restart Nginx (@sudo nginx -s reload@).\n\n
\n    proxy_set_header      Upgrade           $http_upgrade;\n    proxy_set_header      Connection        \"upgrade\";\n
\n\nh3. Changes on the collection's @preserve_version@ attribute semantics\n\nThe @preserve_version@ attribute on collections was originally designed to allow clients to persist a preexisting collection version. This forced clients to make 2 requests if the intention is to \"make this set of changes in a new version that will be kept\", so we have changed the semantics to do just that: When passing @preserve_version=true@ along with other collection updates, the current version is persisted and also the newly created one will be persisted on the next update.\n\nh3. System token requirements\n\nSystem services now log a warning at startup if any of the system tokens (@ManagementToken@, @SystemRootToken@, and @Collections.BlobSigningKey@) are less than 32 characters, or contain characters other than a-z, A-Z, and 0-9. After upgrading, run @arvados-server config-check@ and update your configuration file if needed to resolve any warnings.\n\nThe @API.RailsSessionSecretToken@ configuration key has been removed. Delete this entry from your configuration file after upgrading.\n\nh3. Centos7 Python 3 dependency upgraded to python3\n\nNow that Python 3 is part of the base repository in CentOS 7, the Python 3 dependency for Centos7 Arvados packages was changed from SCL rh-python36 to python3.\n\nh3. ForceLegacyAPI14 option removed\n\nThe ForceLegacyAPI14 configuration option has been removed. In the unlikely event it is mentioned in your config file, remove it to avoid \"deprecated/unknown config\" warning logs.\n\nh2(#v2_1_0). v2.1.0 (2020-10-13)\n\n\"previous: Upgrading to 2.0.0\":#v2_0_0\n\nh3. LoginCluster conflicts with other Login providers\n\nA satellite cluster that delegates its user login to a central user database must only have `Login.LoginCluster` set, or it will return an error. This is a change in behavior, previously it would return an error if another login provider was _not_ configured, even though the provider would never be used.\n\nh3. Minimum supported Python version is now 3.5\n\nWe no longer publish Python 2 based distribution packages for our Python components. There are equivalent packages based on Python 3, but their names are slightly different. If you were using the Python 2 based packages, you can install the Python 3 based package for a drop in replacement. On Debian and Ubuntu:\n\n
\n    apt remove python-arvados-fuse && apt install python3-arvados-fuse\n    apt remove python-arvados-python-client && apt install python3-arvados-python-client\n    apt remove python-arvados-cwl-runner && apt install python3-arvados-cwl-runner\n    apt remove python-crunchstat-summary && apt install python3-crunchstat-summary\n    apt remove python-cwltest && apt install python3-cwltest\n
\n\nOn CentOS:\n\n
\n    yum remove python-arvados-fuse && yum install python3-arvados-fuse\n    yum remove python-arvados-python-client && yum install python3-arvados-python-client\n    yum remove python-arvados-cwl-runner && yum install python3-arvados-cwl-runner\n    yum remove python-crunchstat-summary && yum install python3-crunchstat-summary\n    yum remove python-cwltest && yum install python3-cwltest\n
\n\nh3. Minimum supported Ruby version is now 2.5\n\nThe minimum supported Ruby version is now 2.5. If you are running Arvados on Debian 9 or Ubuntu 16.04, you may need to switch to using RVM or upgrade your OS. See \"Install Ruby and Bundler\":../install/ruby.html for more information.\n\nh3. Removing libpam-arvados, replaced with libpam-arvados-go\n\nThe Python-based PAM package has been replaced with a version written in Go. See \"using PAM for authentication\":{{site.baseurl}}/install/setup-login.html#pam for details.\n\nh3. Removing sso-provider\n\nThe SSO (single sign-on) component is deprecated and will not be supported in future releases. Existing configurations will continue to work in this release, but you should switch to one of the built-in authentication mechanisms as soon as possible. See \"setting up web based login\":{{site.baseurl}}/install/setup-login.html for details.\n\nAfter migrating your configuration, uninstall the @arvados-sso-provider@ package.\n\nh3. S3 signatures\n\nKeepstore now uses \"V4 signatures\":https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html by default for S3 requests. If you are using Amazon S3, no action is needed; all regions support V4 signatures. If you are using a different S3-compatible service that does not support V4 signatures, add @V2Signature: true@ to your volume driver parameters to preserve the old behavior. See \"configuring S3 object storage\":{{site.baseurl}}/install/configure-s3-object-storage.html.\n\nh3. New permission system constraints\n\nSome constraints on the permission system have been added, in particular @role@ and @project@ group types now have distinct behavior. These constraints were already de-facto imposed by the Workbench UI, so on most installations the only effect of this migration will be to reassign @role@ groups to the system user and create a @can_manage@ permission link for the previous owner.\n\n# The @group_class@ field must be either @role@ or @project@. Invalid group_class are migrated to @role@.\n# A @role@ cannot own things. Anything owned by a role is migrated to a @can_manage@ link and reassigned to the system user.\n# Only @role@ and @user@ can have outgoing permission links. Permission links originating from projects are deleted by the migration.\n# A @role@ is always owned by the system_user. When a group is created, it creates a @can_manage@ link for the object that would have been assigned to @owner_uuid@. Migration adds @can_manage@ links and reassigns roles to the system user. This also has the effect of requiring that all @role@ groups have unique names on the system. If there is a name collision during migration, roles will be renamed to ensure they are unique.\n# A permission link can have the permission level (@name@) updated but not @head_uuid@, @tail_uuid@ or @link_class@.\n\nThe @arvados-sync-groups@ tool has been updated to reflect these constraints, so it is important to use the version of @arvados-sync-groups@ that matches the API server version.\n\nBefore upgrading, use the following commands to find out which groups and permissions in your database will be automatically modified or deleted during the upgrade.\n\nTo determine which groups have invalid @group_class@ (these will be migrated to @role@ groups):\n\n
\narv group list --filters '[[\"group_class\", \"not in\", [\"project\", \"role\"]]]'\n
\n\nTo list all @role@ groups, which will be reassigned to the system user (unless @owner_uuid@ is already the system user):\n\n
\narv group list --filters '[[\"group_class\", \"=\", \"role\"]]'\n
\n\nTo list which @project@ groups have outgoing permission links (such links are now invalid and will be deleted by the migration):\n\n
\nfor uuid in $(arv link list --filters '[[\"link_class\", \"=\", \"permission\"], [\"tail_uuid\", \"like\", \"%-j7d0g-%\"]]' |\n              jq -r .items[].tail_uuid | sort | uniq) ; do\n   arv group list --filters '[[\"group_class\", \"=\", \"project\"], [\"uuid\", \"=\", \"'$uuid'\"]]' | jq .items\ndone\n
\n\nh4. \"Public favorites\" moved to their own project\n\nAs a side effect of new permission system constraints, \"star\" links (indicating shortcuts in Workbench) that were previously owned by \"All users\" (which is now a \"role\" and cannot own things) will be migrated to a new system project called \"Public favorites\" which is readable by the \"Anonymous users\" role.\n\nh2(#v2_0_0). v2.0.0 (2020-02-07)\n\n\"previous: Upgrading to 1.4.1\":#v1_4_1\n\nArvados 2.0 is a major upgrade, with many changes. Please read these upgrade notes carefully before you begin.\n\nh3. Migrating to centralized config.yml\n\nSee \"Migrating Configuration\":https://doc.arvados.org/v2.1/admin/config-migration.html for notes on migrating legacy per-component configuration files to the new centralized @/etc/arvados/config.yml@.\n\nTo ensure a smooth transition, the per-component config files continue to be read, and take precedence over the centralized configuration. Your cluster should continue to function after upgrade but before doing the full configuration migration. However, several services (keepstore, keep-web, keepproxy) require a minimal `/etc/arvados/config.yml` to start:\n\n
\nClusters:\n  zzzzz:\n    Services:\n      Controller:\n        ExternalURL: \"https://zzzzz.example.com\"\n
\n\nh3. Keep-balance configuration migration\n\n(feature \"#14714\":https://dev.arvados.org/issues/14714 ) The keep-balance service can now be configured using the centralized configuration file at @/etc/arvados/config.yml@. The following command line and configuration options have changed.\n\nYou can no longer specify types of keep services to balance via the @KeepServiceTypes@ config option in the legacy config at @/etc/arvados/keep-balance/keep-balance.yml@. If you are still using the legacy config and @KeepServiceTypes@ has a value other than \"disk\", keep-balance will produce an error.\n\nYou can no longer specify individual keep services to balance via the @config.KeepServiceList@ command line option or @KeepServiceList@ legacy config option. Instead, keep-balance will operate on all keepstore servers with @service_type:disk@ as reported by the @arv keep_service list@ command. If you are still using the legacy config, @KeepServiceList@ should be removed or keep-balance will produce an error.\n\nPlease see the \"config migration guide\":https://doc.arvados.org/v2.1/admin/config-migration.html and \"keep-balance install guide\":{{site.baseurl}}/install/install-keep-balance.html for more details.\n\nh3. Arv-git-httpd configuration migration\n\n(feature \"#14712\":https://dev.arvados.org/issues/14712 ) The arv-git-httpd package can now be configured using the centralized configuration file at @/etc/arvados/config.yml@. Configuration via individual command line arguments is no longer available. Please see \"arv-git-httpd's config migration guide\":https://doc.arvados.org/v2.1/admin/config-migration.html#arv-git-httpd for more details.\n\nh3. Keepstore and keep-web configuration migration\n\nkeepstore and keep-web no longer support configuration via (previously deprecated) command line configuration flags and environment variables.\n\nkeep-web now supports the legacy @keep-web.yml@ config format (used by Arvados 1.4) and the new cluster config file format. Please check \"keep-web's install guide\":{{site.baseurl}}/install/install-keep-web.html for more details.\n\nkeepstore now supports the legacy @keepstore.yml@ config format (used by Arvados 1.4) and the new cluster config file format. Please check the \"keepstore config migration notes\":https://doc.arvados.org/v2.1/admin/config-migration.html#keepstore and \"keepstore install guide\":{{site.baseurl}}/install/install-keepstore.html for more details.\n\nh3. Keepproxy configuration migration\n\n(feature \"#14715\":https://dev.arvados.org/issues/14715 ) Keepproxy can now be configured using the centralized config at @/etc/arvados/config.yml@. Configuration via individual command line arguments is no longer available and the @DisableGet@, @DisablePut@, and @PIDFile@ configuration options are no longer supported. If you are still using the legacy config and @DisableGet@ or @DisablePut@ are set to true or @PIDFile@ has a value, keepproxy will produce an error and fail to start. Please see \"keepproxy's config migration guide\":https://doc.arvados.org/v2.1/admin/config-migration.html#keepproxy for more details.\n\nh3. Delete \"keep_services\" records\n\nAfter all keepproxy and keepstore configurations have been migrated to the centralized configuration file, all keep_services records you added manually during installation should be removed. System logs from keepstore and keepproxy at startup, as well as the output of @arvados-server config-check@, will remind you to do this.\n\n
$ export ARVADOS_API_HOST=...\n$ export ARVADOS_API_TOKEN=...\n$ arv --format=uuid keep_service list | xargs -n1 arv keep_service delete --uuid\n
\n\nOnce these old records are removed, @arv keep_service list@ will instead return the services listed under Services/Keepstore/InternalURLs and Services/Keepproxy/ExternalURL in your centralized configuration file.\n\nh3. Enabling Postgres trigram indexes\n\nFeature \"#15106\":https://dev.arvados.org/issues/15106 improves the speed and functionality of full text search by introducing trigram indexes on text searchable database columns via a migration. Prior to updating, you must first install the postgresql-contrib package on your system and subsequently run the CREATE EXTENSION pg_trgm SQL command on the arvados_production database as a postgres superuser.\n\nThe \"postgres-contrib package\":https://www.postgresql.org/docs/10/contrib.html has been supported since PostgreSQL version 9.4. The version of the contrib package should match the version of your PostgreSQL installation. Using 9.5 as an example, the package can be installed and the extension enabled using the following:\n\nCentos 7\n\n
~$ sudo yum install -y postgresql95-contrib\n~$ su - postgres -c \"psql -d 'arvados_production' -c 'CREATE EXTENSION IF NOT EXISTS pg_trgm'\"\n
\n\n\nRHEL 7\n\n
~$ sudo yum install -y rh-postgresql95-postgresql-contrib\n~$ su - postgres -c \"psql -d 'arvados_production' -c 'CREATE EXTENSION IF NOT EXISTS pg_trgm'\"\n
\n\n\nDebian or Ubuntu\n\n
~$ sudo apt-get install -y postgresql-contrib-9.5\n~$ sudo -u postgres psql -d 'arvados_production' -c 'CREATE EXTENSION IF NOT EXISTS pg_trgm'\n
\n\n\nSubsequently, the psql -d 'arvados_production' -c '\\dx' command will display the installed extensions for the arvados_production database. This list should now contain @pg_trgm@.\n\nh3. New Workbench 2\n\nWorkbench 2 is now ready for regular use. Follow the instructions to \"install workbench 2\":../install/install-workbench2-app.html\n\nh3. New property vocabulary format for Workbench2\n\n(feature \"#14151\":https://dev.arvados.org/issues/14151) Workbench2 supports a new vocabulary format and it isn't compatible with the previous one, please read the \"metadata vocabulary format admin page\":{{site.baseurl}}/admin/metadata-vocabulary.html for more information.\n\nh3. Cloud installations only: node manager replaced by arvados-dispatch-cloud\n\nNode manager is deprecated and replaced by @arvados-dispatch-cloud@. No automated config migration is available. Follow the instructions to \"install the cloud dispatcher\":../install/crunch2-cloud/install-dispatch-cloud.html\n\n*Only one dispatch process should be running at a time.* If you are migrating a system that currently runs Node manager and @crunch-dispatch-slurm@, it is safest to remove the @crunch-dispatch-slurm@ service entirely before installing @arvados-dispatch-cloud@.\n\n\n
~$ sudo systemctl --now disable crunch-dispatch-slurm\n~$ sudo apt-get remove crunch-dispatch-slurm\n
\n\n\nh3. Jobs API is read-only\n\n(task \"#15133\":https://dev.arvados.org/issues/15133 ) The legacy 'jobs' API is now read-only. It has been superceded since Arvados 1.1 by containers / container_requests (aka crunch v2). Arvados installations since the end of 2017 (v1.1.0) have probably only used containers, and are unaffected by this change.\n\nSo that older Arvados sites don't lose access to legacy records, the API has been converted to read-only. Creating and updating jobs (and related types job_task, pipeline_template and pipeline_instance) is disabled and much of the business logic related has been removed, along with various other code specific to the jobs API. Specifically, the following programs associated with the jobs API have been removed: @crunch-dispatch.rb@, @crunch-job@, @crunchrunner@, @arv-run-pipeline-instance@, @arv-run@.\n\nh3. \"/\" prohibited in collection and project names\n\n(issue \"#15836\":https://dev.arvados.org/issues/15836) By default, Arvados now rejects new names containing the @/@ character when creating or renaming collections and projects. Previously, these names were permitted, but the resulting objects were invisible in the WebDAV \"home\" tree. If you prefer, you can restore the previous behavior, and optionally configure a substitution string to make the affected objects accessible via WebDAV. See @ForwardSlashNameSubstitution@ in the \"configuration reference\":config.html.\n\nh3. No longer stripping ':' from strings in serialized database columns\n\n(bug \"#15311\":https://dev.arvados.org/issues/15311 ) Strings read from serialized columns in the database with a leading ':' would have the ':' stripped after loading the record. This behavior existed due to legacy serialization behavior which stored Ruby symbols with a leading ':'. Unfortunately this corrupted fields where the leading \":\" was intentional. This behavior has been removed.\n\nYou can test if any records in your database are affected by going to the API server directory and running @bundle exec rake symbols:check@. This will report which records contain fields with a leading ':' that would previously have been stripped. If there are records to be updated, you can update the database using @bundle exec rake symbols:stringify@.\n\nh3. Scoped tokens should use PATCH for updates\n\nThe API server accepts both PUT and PATCH for updates, but they will be normalized to PATCH by arvados-controller. Scoped tokens should be updated accordingly.\n\n\n\nh2(#v1_4_1). v1.4.1 (2019-09-20)\n\n\"previous: Upgrading to 1.4.0\":#v1_4_0\n\nh3. Centos7 Python 3 dependency upgraded to rh-python36\n\nThe Python 3 dependency for Centos7 Arvados packages was upgraded from rh-python35 to rh-python36.\n\nh2(#v1_4_0). v1.4.0 (2019-06-05)\n\n\"previous: Upgrading to 1.3.3\":#v1_3_3\n\nh3. Populating the new file_count and file_size_total columns on the collections table\n\nAs part of story \"#14484\":https://dev.arvados.org/issues/14484, two new columns were added to the collections table in a database migration. If your installation has a large collections table, this migration may take some time. We've seen it take ~5 minutes on an installation with 250k collections, but your mileage may vary.\n\nThe new columns are initialized with a zero value. In order to populate them, it is necessary to run a script called populate-file-info-columns-in-collections.rb from the scripts directory of the API server. This can be done out of band, ideally directly after the API server has been upgraded to v1.4.0.\n\nh3. Stricter collection manifest validation on the API server\n\nAs a consequence of \"#14482\":https://dev.arvados.org/issues/14482, the Ruby SDK does a more rigorous collection manifest validation. Collections created after 2015-05 are unlikely to be invalid, however you may check for invalid manifests using the script below.\n\nYou could set up a new rvm gemset and install the specific arvados gem for testing, like so:\n\n\n
~$ rvm gemset create rubysdk-test\n~$ rvm gemset use rubysdk-test\n~$ gem install arvados -v 1.3.1.20190301212059\n
\n\n\nNext, you can run the following script using admin credentials, it will scan the whole collection database and report any collection that didn't pass the check:\n\n{% codeblock as ruby %}\nrequire 'arvados'\nrequire 'arvados/keep'\n\napi = Arvados.new\noffset = 0\nbatch_size = 100\ninvalid = []\n\nwhile true\n begin\n req = api.collection.index(\n :select => [:uuid, :created_at, :manifest_text],\n :include_trash => true, :include_old_versions => true,\n :limit => batch_size, :offset => offset)\n rescue\n invalid.each {|c| puts \"#{c[:uuid]} (Created at #{c[:created_at]}): #{c[:error]}\" }\n raise\n end\n\n req[:items].each do |col|\n begin\n Keep::Manifest.validate! col[:manifest_text]\n rescue Exception => e\n puts \"Collection #{col[:uuid]} manifest not valid\"\n invalid << {uuid: col[:uuid], error: e, created_at: col[:created_at]}\n end\n end\n puts \"Checked #{offset} / #{req[:items_available]} - Invalid: #{invalid.size}\"\n offset += req[:limit]\n break if offset > req[:items_available]\nend\n\nif invalid.empty?\n puts \"No invalid collection manifests found\"\nelse\n invalid.each {|c| puts \"#{c[:uuid]} (Created at #{c[:created_at]}): #{c[:error]}\" }\nend\n{% endcodeblock %}\n\nThe script will return a final report enumerating any invalid collection by UUID, with its creation date and error message so you can take the proper correction measures, if needed.\n\nh3. Python packaging change\n\nAs part of story \"#9945\":https://dev.arvados.org/issues/9945, the distribution packaging (deb/rpm) of our Python packages has changed. These packages now include a built-in virtualenv to reduce dependencies on system packages. We have also stopped packaging and publishing backports for all the Python dependencies of our packages, as they are no longer needed.\n\nOne practical consequence of this change is that the use of the Arvados Python SDK (aka \"import arvados\") will require a tweak if the SDK was installed from a distribution package. It now requires the loading of the virtualenv environment from our packages. The \"Install documentation for the Arvados Python SDK\":{{ site.baseurl }}/sdk/python/sdk-python.html reflects this change. This does not affect the use of the command line tools (e.g. arv-get, etc.).\n\nPython scripts that rely on the distribution Arvados Python SDK packages to import the Arvados SDK will need to be tweaked to load the correct Python environment.\n\nThis can be done by activating the virtualenv outside of the script:\n\n\n
~$ source /usr/share/python2.7/dist/python-arvados-python-client/bin/activate\n(python-arvados-python-client) ~$ path-to-the-python-script\n
\n\n\nOr alternatively, by updating the shebang line at the start of the script to:\n\n\n
\n#!/usr/share/python2.7/dist/python-arvados-python-client/bin/python\n
\n\n\nh3. python-arvados-cwl-runner deb/rpm package now conflicts with python-cwltool deb/rpm package\n\nAs part of story \"#9945\":https://dev.arvados.org/issues/9945, the distribution packaging (deb/rpm) of our Python packages has changed. The python-arvados-cwl-runner package now includes a version of cwltool. If present, the python-cwltool and cwltool distribution packages will need to be uninstalled before the python-arvados-cwl-runner deb or rpm package can be installed.\n\nh3. Centos7 Python 3 dependency upgraded to rh-python35\n\nAs part of story \"#9945\":https://dev.arvados.org/issues/9945, the Python 3 dependency for Centos7 Arvados packages was upgraded from SCL python33 to rh-python35.\n\nh3. Centos7 package for libpam-arvados depends on the python-pam package, which is available from EPEL\n\nAs part of story \"#9945\":https://dev.arvados.org/issues/9945, it was discovered that the Centos7 package for libpam-arvados was missing a dependency on the python-pam package, which is available from the EPEL repository. The dependency has been added to the libpam-arvados package. This means that going forward, the EPEL repository will need to be enabled to install libpam-arvados on Centos7.\n\nh3. New configuration\n\nArvados is migrating to a centralized configuration file for all components. During the migration, legacy configuration files will continue to be loaded. See \"Migrating Configuration\":https://doc.arvados.org/v2.1/admin/config-migration.html for details.\n\nh2(#v1_3_3). v1.3.3 (2019-05-14)\n\n\"previous: Upgrading to 1.3.0\":#v1_3_0\n\nThis release corrects a potential data loss issue, if you are running Arvados 1.3.0 or 1.3.1 we strongly recommended disabling @keep-balance@ until you can upgrade to 1.3.3 or 1.4.0. With keep-balance disabled, there is no chance of data loss.\n\nWe've put together a \"wiki page\":https://dev.arvados.org/projects/arvados/wiki/Recovering_lost_data which outlines how to recover blocks which have been put in the trash, but not yet deleted, as well as how to identify any collections which have missing blocks so that they can be regenerated. The keep-balance component has been enhanced to provide a list of missing blocks and affected collections and we've provided a \"utility script\":https://github.com/arvados/arvados/blob/main/tools/keep-xref/keep-xref.py which can be used to identify the workflows that generated those collections and who ran those workflows, so that they can be rerun.\n\nh2(#v1_3_0). v1.3.0 (2018-12-05)\n\n\"previous: Upgrading to 1.2\":#v1_2_0\n\nThis release includes several database migrations, which will be executed automatically as part of the API server upgrade. On large Arvados installations, these migrations will take a while. We've seen the upgrade take 30 minutes or more on installations with a lot of collections.\n\nThe @arvados-controller@ component now requires the /etc/arvados/config.yml file to be present.\n\nSupport for the deprecated \"jobs\" API is broken in this release. Users who rely on it should not upgrade. This will be fixed in an upcoming 1.3.1 patch release, however users are encouraged to migrate as support for the \"jobs\" API will be dropped in an upcoming release. Users who are already using the \"containers\" API are not affected.\n\nh2(#v1_2_1). v1.2.1 (2018-11-26)\n\nThere are no special upgrade notes for this release.\n\nh2(#v1_2_0). v1.2.0 (2018-09-05)\n\n\"previous: Upgrading to 1.1.2 or 1.1.3\":#v1_1_2\n\nh3. Regenerate Postgres table statistics\n\nIt is recommended to regenerate the table statistics for Postgres after upgrading to v1.2.0. If autovacuum is enabled on your installation, this script would do the trick:\n\n
\n#!/bin/bash\n\nset -e\nset -u\n\ntables=`echo \"\\dt\" | psql arvados_production | grep public|awk -e '{print $3}'`\n\nfor t in $tables; do\n    echo \"echo 'analyze $t' | psql arvados_production\"\n    time echo \"analyze $t\" | psql arvados_production\ndone\n
\n\nIf you also need to do the vacuum, you could adapt the script to run 'vacuum analyze' instead of 'analyze'.\n\nh3. New component: arvados-controller\n\nCommit \"db5107dca\":https://dev.arvados.org/projects/arvados/repository/revisions/db5107dca adds a new system service, arvados-controller. More detail is available in story \"#13496\":https://dev.arvados.org/issues/13497.\n\nTo add the Arvados Controller to your system please refer to the \"installation instructions\":../install/install-api-server.html after upgrading your system to 1.2.0.\n\nVerify your setup by confirming that API calls appear in the controller's logs (_e.g._, @journalctl -fu arvados-controller@) while loading a workbench page.\n\nh2(#v1_1_4). v1.1.4 (2018-04-10)\n\n\"previous: Upgrading to 1.1.3\":#v1_1_3\n\nh3. arvados-cwl-runner regressions (2018-04-05)\n\nSecondary files missing from toplevel workflow inputs\n\nThis only affects workflows that rely on implicit discovery of secondaryFiles.\n\nIf a workflow input does not declare @secondaryFiles@ corresponding to the @secondaryFiles@ of workflow steps which use the input, the workflow would inconsistently succeed or fail depending on whether the input values were specified as local files or referenced an existing collection (and whether the existing collection contained the secondary files or not). To ensure consistent behavior, the workflow is now required to declare in the top level workflow inputs any secondaryFiles that are expected by workflow steps.\n\nAs an example, the following workflow will fail because the @toplevel_input@ does not declare the @secondaryFiles@ that are expected by @step_input@:\n\n
\nclass: Workflow\ncwlVersion: v1.0\ninputs:\n  toplevel_input: File\noutputs: []\nsteps:\n  step1:\n    in:\n      step_input: toplevel_input\n    out: []\n    run:\n      id: sub\n      class: CommandLineTool\n      inputs:\n        step_input:\n          type: File\n          secondaryFiles:\n            - .idx\n      outputs: []\n      baseCommand: echo\n
\n\nWhen run, this produces an error like this:\n\n
\ncwltool ERROR: [step step1] Cannot make job: Missing required secondary file 'hello.txt.idx' from file object: {\n    \"basename\": \"hello.txt\",\n    \"class\": \"File\",\n    \"location\": \"keep:ade9d0e032044bd7f58daaecc0d06bc6+51/hello.txt\",\n    \"size\": 0,\n    \"nameroot\": \"hello\",\n    \"nameext\": \".txt\",\n    \"secondaryFiles\": []\n}\n
\n\nTo fix this error, add the appropriate @secondaryFiles@ section to @toplevel_input@\n\n\n
class: Workflow\ncwlVersion: v1.0\ninputs:\n  toplevel_input:\n    type: File\n    secondaryFiles:\n      - .idx\noutputs: []\nsteps:\n  step1:\n    in:\n      step_input: toplevel_input\n    out: []\n    run:\n      id: sub\n      class: CommandLineTool\n      inputs:\n        step_input:\n          type: File\n          secondaryFiles:\n            - .idx\n      outputs: []\n      baseCommand: echo\n
\n\n\nThis bug has been fixed in Arvados release v1.2.0.\n\nSecondary files on default file inputs\n\n@File@ inputs that have default values and also expect @secondaryFiles@ and will fail to upload default @secondaryFiles@. As an example, the following case will fail:\n\n
\nclass: CommandLineTool\ninputs:\n  step_input:\n    type: File\n    secondaryFiles:\n      - .idx\n    default:\n      class: File\n      location: hello.txt\noutputs: []\nbaseCommand: echo\n
\n\nWhen run, this produces an error like this:\n\n
\n2018-05-03 10:58:47 cwltool ERROR: Unhandled error, try again with --debug for more information:\n  [Errno 2] File not found: u'hello.txt.idx'\n
\n\nTo fix this, manually upload the primary and secondary files to keep and explicitly declare @secondaryFiles@ on the default primary file:\n\n\n
class: CommandLineTool\ninputs:\n  step_input:\n    type: File\n    secondaryFiles:\n      - .idx\n    default:\n      class: File\n      location: keep:4d8a70b1e63b2aad6984e40e338e2373+69/hello.txt\n      secondaryFiles:\n       - class: File\n         location: keep:4d8a70b1e63b2aad6984e40e338e2373+69/hello.txt.idx\noutputs: []\nbaseCommand: echo\n
\n\n\nThis bug has been fixed in Arvados release v1.2.0.\n\nh2(#v1_1_3). v1.1.3 (2018-02-08)\n\nThere are no special upgrade notes for this release.\n\nh2(#v1_1_2). v1.1.2 (2017-12-22)\n\n\"previous: Upgrading to 1.1.0 or 1.1.1\":#v1_1_0\n\nh3. The minimum version for Postgres is now 9.4 (2017-12-08)\n\nAs part of story \"#11908\":https://dev.arvados.org/issues/11908, commit \"8f987a9271\":https://dev.arvados.org/projects/arvados/repository/revisions/8f987a9271 introduces a dependency on Postgres 9.4. Previously, Arvados required Postgres 9.3.\n\n* Debian 8 (pg 9.4) and Debian 9 (pg 9.6) do not require an upgrade\n* Ubuntu 16.04 (pg 9.5) does not require an upgrade\n* Ubuntu 14.04 (pg 9.3) requires upgrade to Postgres 9.4: https://www.postgresql.org/download/linux/ubuntu/\n* CentOS 7 and RHEL7 (pg 9.2) require upgrade to Postgres 9.4. It is necessary to migrate of the contents of your database: https://www.postgresql.org/docs/9.0/static/migration.html\n*# Create a database backup using @pg_dump@\n*# Install the @rh-postgresql94@ backport package from either Software Collections: http://doc.arvados.org/install/install-postgresql.html or the Postgres developers: https://www.postgresql.org/download/linux/redhat/\n*# Restore from the backup using @psql@\n\nh2(#v1_1_1). v1.1.1 (2017-11-30)\n\nThere are no special upgrade notes for this release.\n\nh2(#v1_1_0). v1.1.0 (2017-10-24)\n\nh3. The minimum version for Postgres is now 9.3 (2017-09-25)\n\nAs part of story \"#12032\":https://dev.arvados.org/issues/12032, commit \"68bdf4cbb1\":https://dev.arvados.org/projects/arvados/repository/revisions/68bdf4cbb1 introduces a dependency on Postgres 9.3. Previously, Arvados required Postgres 9.1.\n\n* Debian 8 (pg 9.4) and Debian 9 (pg 9.6) do not require an upgrade\n* Ubuntu 16.04 (pg 9.5) does not require an upgrade\n* Ubuntu 14.04 (pg 9.3) is compatible, however upgrading to Postgres 9.4 is recommended: https://www.postgresql.org/download/linux/ubuntu/\n* CentOS 7 and RHEL7 (pg 9.2) should upgrade to Postgres 9.4. It is necessary to migrate of the contents of your database: https://www.postgresql.org/docs/9.0/static/migration.html\n*# Create a database backup using @pg_dump@\n*# Install the @rh-postgresql94@ backport package from either Software Collections: http://doc.arvados.org/install/install-postgresql.html or the Postgres developers: https://www.postgresql.org/download/linux/redhat/\n*# Restore from the backup using @psql@\n\nh2(#older). Older versions\n\nh3. Upgrade slower than usual (2017-06-30)\n\nAs part of story \"#11807\":https://dev.arvados.org/issues/11807, commit \"55aafbb\":https://dev.arvados.org/projects/arvados/repository/revisions/55aafbb converts old \"jobs\" database records from YAML to JSON, making the upgrade process slower than usual.\n\n* The migration can take some time if your database contains a substantial number of YAML-serialized rows (i.e., you installed Arvados before March 3, 2017 \"660a614\":https://dev.arvados.org/projects/arvados/repository/revisions/660a614 and used the jobs/pipelines APIs). Otherwise, the upgrade will be no slower than usual.\n* The conversion runs as a database migration, i.e., during the deb/rpm package upgrade process, while your API server is unavailable.\n* Expect it to take about 1 minute per 20K jobs that have ever been created/run.\n\nh3. Service discovery overhead change in keep-web (2017-06-05)\n\nAs part of story \"#9005\":https://dev.arvados.org/issues/9005, commit \"cb230b0\":https://dev.arvados.org/projects/arvados/repository/revisions/cb230b0 reduces service discovery overhead in keep-web requests.\n\n* When upgrading keep-web _or keepproxy_ to/past this version, make sure to update API server as well. Otherwise, a bad token in a request can cause keep-web to fail future requests until either keep-web restarts or API server gets upgraded.\n\nh3. Node manager now has an http endpoint for management (2017-04-12)\n\nAs part of story \"#11349\":https://dev.arvados.org/issues/11349, commit \"2c094e2\":https://dev.arvados.org/projects/arvados/repository/revisions/2c094e2 adds a \"management\" http server to nodemanager.\n\n* To enable it, add to your configuration file: 
[Manage]\n  address = 127.0.0.1\n  port = 8989
\n* The server responds to @http://{address}:{port}/status.json@ with a summary of how many nodes are in each state (booting, busy, shutdown, etc.)\n\nh3. New websockets component (2017-03-23)\n\nAs part of story \"#10766\":https://dev.arvados.org/issues/10766, commit \"e8cc0d7\":https://dev.arvados.org/projects/arvados/repository/revisions/e8cc0d7 replaces puma with arvados-ws as the recommended websocket server.\n* See http://doc.arvados.org/install/install-ws.html for install/upgrade instructions.\n* Remove the old puma server after the upgrade is complete. Example, with runit: 
\n$ sudo sv down /etc/sv/puma\n$ sudo rm -r /etc/sv/puma\n
Example, with systemd: 
\n$ systemctl disable puma\n$ systemctl stop puma\n
\n\nh3. Change of database encoding for hashes and arrays (2017-03-06)\n\nAs part of story \"#11168\":https://dev.arvados.org/issues/11168, commit \"660a614\":https://dev.arvados.org/projects/arvados/repository/revisions/660a614 uses JSON instead of YAML to encode hashes and arrays in the database.\n\n* Aside from a slight performance improvement, this should have no externally visible effect.\n* Downgrading past this version is not supported, and is likely to cause errors. If this happens, the solution is to upgrade past this version.\n* After upgrading, make sure to restart puma and crunch-dispatch-* processes.\n\nh3. Docker image format compatibility check (2017-02-03)\n\nAs part of story \"#10969\":https://dev.arvados.org/issues/10969, commit \"74a9dec\":https://dev.arvados.org/projects/arvados/repository/revisions/74a9dec introduces a Docker image format compatibility check: the @arv keep docker@ command prevents users from inadvertently saving docker images that compute nodes won't be able to run.\n* If your compute nodes run a version of *docker older than 1.10* you must override the default by adding to your API server configuration (@/etc/arvados/api/application.yml@): 
docker_image_formats: [\"v1\"]
\n* Refer to the comments above @docker_image_formats@ in @/var/www/arvados-api/current/config/application.default.yml@ or source:services/api/config/application.default.yml or issue \"#10969\":https://dev.arvados.org/issues/10969 for more detail.\n* *NOTE:* This does *not* include any support for migrating existing Docker images from v1 to v2 format. This will come later: for now, sites running Docker 1.9 or earlier should still *avoid upgrading Docker further than 1.9.*\n\nh3. Debian and RPM packages now have systemd unit files (2016-09-27)\n\nSeveral Debian and RPM packages -- keep-balance (\"d9eec0b\":https://dev.arvados.org/projects/arvados/repository/revisions/d9eec0b), keep-web (\"3399e63\":https://dev.arvados.org/projects/arvados/repository/revisions/3399e63), keepproxy (\"6de67b6\":https://dev.arvados.org/projects/arvados/repository/revisions/6de67b6), and arvados-git-httpd (\"9e27ddf\":https://dev.arvados.org/projects/arvados/repository/revisions/9e27ddf) -- now enable their respective components using systemd. These components prefer YAML configuration files over command line flags (\"3bbe1cd\":https://dev.arvados.org/projects/arvados/repository/revisions/3bbe1cd).\n\n* On Debian-based systems using systemd, services are enabled automatically when packages are installed.\n* On RedHat-based systems using systemd, unit files are installed but services must be enabled explicitly: e.g., \"sudo systemctl enable keep-web; sudo systemctl start keep-web\".\n* The new systemd-supervised services will not start up successfully until configuration files are installed in /etc/arvados/: e.g., \"Sep 26 18:23:55 62751f5bb946 keep-web[74]: 2016/09/26 18:23:55 open /etc/arvados/keep-web/keep-web.yml: no such file or directory\"\n* To migrate from runit to systemd after installing the new packages, we recommend the following procedure:\n*# Bring down the runit service: \"sv down /etc/sv/keep-web\"\n*# Create a JSON configuration file (e.g., /etc/arvados/keep-web/keep-web.yml -- see \"keep-web -help\")\n*# Ensure the service is running correctly under systemd: \"systemctl status keep-web\" / \"journalctl -u keep-web\"\n*# Remove the runit service so it doesn't start at next boot\n* Affected services:\n** keep-balance - /etc/arvados/keep-balance/keep-balance.yml\n** keep-web - /etc/arvados/keep-web/keep-web.yml\n** keepproxy - /etc/arvados/keepproxy/keepproxy.yml\n** arvados-git-httpd - /etc/arvados/arv-git-httpd/arv-git-httpd.yml\n\nh3. Installation paths for Python modules and script changed (2016-05-31)\n\nCommits \"ae72b172c8\":https://dev.arvados.org/projects/arvados/repository/revisions/ae72b172c8 and \"3aae316c25\":https://dev.arvados.org/projects/arvados/repository/revisions/3aae316c25 change the filesystem location where Python modules and scripts are installed.\n\n* Previous packages installed these files to the distribution's preferred path under @/usr/local@ (or the equivalent location in a Software Collection). Now they get installed to a path under @/usr@. This improves compatibility with other Python packages provided by the distribution. See \"#9242\":https://dev.arvados.org/issues/9242 for more background.\n* If you simply import Python modules from scripts, or call Python tools relying on $PATH, you don't need to make any changes. If you have hardcoded full paths to some of these files (e.g., in symbolic links or configuration files), you will need to update those paths after this upgrade.\n\nh3. Crunchrunner package is required on compute and shell nodes (2016-04-25)\n\nCommit \"eebcb5e\":https://dev.arvados.org/projects/arvados/repository/revisions/eebcb5e requires the crunchrunner package to be installed on compute nodes and shell nodes in order to run CWL workflows.\n\n* On each Debian-based compute node and shell node, run: @sudo apt-get install crunchrunner@\n* On each Red Hat-based compute node and shell node, run: @sudo yum install crunchrunner@\n\nh3. Keep permission signature algorithm change (2016-04-21)\n\nCommit \"3c88abd\":https://dev.arvados.org/projects/arvados/repository/revisions/3c88abd changes the Keep permission signature algorithm.\n\n* All software components that generate signatures must be upgraded together. These are: keepstore, API server, keep-block-check, and keep-rsync. For example, if keepstore < 0.1.20160421183420 but API server >= 0.1.20160421183420, clients will not be able to read or write data in Keep.\n* Jobs and client operations that are in progress during the upgrade (including arv-put's \"resume cache\") will fail.\n\nh3. Workbench's \"Getting Started\" popup disabled by default (2015-01-05)\n\nCommit \"e1276d6e\":https://dev.arvados.org/projects/arvados/repository/revisions/e1276d6e disables Workbench's \"Getting Started\" popup by default.\n\n* If you want new users to continue seeing this popup, set @enable_getting_started_popup: true@ in Workbench's @application.yml@ configuration.\n\nh3. Crunch jobs now have access to Keep-backed writable scratch storage (2015-12-03)\n\nCommit \"5590c9ac\":https://dev.arvados.org/projects/arvados/repository/revisions/5590c9ac makes a Keep-backed writable scratch directory available in crunch jobs (see \"#7751\":https://dev.arvados.org/issues/7751)\n\n* All compute nodes must be upgraded to arvados-fuse >= 0.1.2015112518060 because crunch-job uses some new arv-mount flags (--mount-tmp, --mount-by-pdh) introduced in merge \"346a558\":https://dev.arvados.org/projects/arvados/repository/revisions/346a558\n* Jobs will fail if the API server (in particular crunch-job from the arvados-cli gem) is upgraded without upgrading arvados-fuse on compute nodes.\n\nh3. Recommended configuration change for keep-web (2015-11-11)\n\nCommit \"1e2ace5\":https://dev.arvados.org/projects/arvados/repository/revisions/1e2ace5 changes recommended config for keep-web (see \"#5824\":https://dev.arvados.org/issues/5824)\n\n* proxy/dns/ssl config should be updated to route \"https://download.ClusterID.example.com/\" requests to keep-web (alongside the existing \"collections\" routing)\n* keep-web command line adds @-attachment-only-host download.ClusterID.example.com@\n* Workbench config adds @keep_web_download_url@\n* More info on the (still beta/non-TOC-linked) \"keep-web doc page\":http://doc.arvados.org/install/install-keep-web.html\n\nh3. Stopped containers are now automatically removed on compute nodes (2015-11-04)\n\nCommit \"1d1c6de\":https://dev.arvados.org/projects/arvados/repository/revisions/1d1c6de removes stopped containers (see \"#7444\":https://dev.arvados.org/issues/7444)\n\n* arvados-docker-cleaner removes _all_ docker containers as soon as they exit, effectively making @docker run@ default to @--rm@. If you run arvados-docker-cleaner on a host that does anything other than run crunch-jobs, and you still want to be able to use @docker start@, read the \"new doc page\":http://doc.arvados.org/install/install-compute-node.html to learn how to turn this off before upgrading.\n\nh3. New keep-web service (2015-11-04)\n\nCommit \"21006cf\":https://dev.arvados.org/projects/arvados/repository/revisions/21006cf adds a new keep-web service (see \"#5824\":https://dev.arvados.org/issues/5824).\n\n* Nothing relies on keep-web yet, but early adopters can install it now by following http://doc.arvados.org/install/install-keep-web.html (it is not yet linked in the TOC).\n\n\n
\n\n" }, { "path": "doc/admin/user-activity.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: \"User activity report\"\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThe @arv-user-activity@ tool generates a summary report of user activity on an Arvados instance based on the audit logs (the @logs@ table).\n\nh2. Installation\n\nh2. Option 1: Install from a distribution package\n\nThis installation method is recommended to make the CLI tools available system-wide. It can coexist with the installation method described in option 2, below.\n\nFirst, configure the \"Arvados package repositories\":{{ site.baseurl }}/install/packages.html\n\n{% assign arvados_component = 'python3-arvados-user-activity' %}\n\n{% include 'install_packages' %}\n\nh2. Option 2: Install from source\n\nStep 1: Check out the arvados source code\n\nStep 2: Change directory to @arvados/tools/user-activity@\n\nStep 3: Run @pip install .@ in an appropriate installation environment, such as a @virtualenv@.\n\nNote: depends on the \"Arvados Python SDK\":{{ site.baseurl }}/sdk/python/sdk-python.html and its associated build prerequisites (e.g. @pycurl@).\n\nh2. Usage\n\nSet ARVADOS_API_HOST to the api server of the cluster for which the report should be generated. ARVADOS_API_TOKEN needs to be a \"v2 token\":../admin/scoped-tokens.html for an admin user, or the system root token. Please note that in a login cluster federation, the token needs to be issued by the login cluster, but the report should be generated against the API server of the cluster for which it is desired. In other words, ARVADOS_API_HOST would point at the satellite cluster for which the report is desired, but ARVADOS_API_TOKEN would be a token that belongs to a login cluster user, or the login cluster's system root token.\n\nRun the tool with the option @--days@ giving the number of days to report on. It will request activity logs from the API and generate a summary report on standard output.\n\nExample run:\n\n
\n$ bin/arv-user-activity --days 14\nUser activity on pirca between 2020-11-10 16:42 and 2020-11-24 16:42\n\nPeter Amstutz  (https://workbench.pirca.arvadosapi.com/users/jutro-tpzed-a4qnxq3pcfcgtkz)\n  organization: \"Curii\"\n  role: \"Software Developer\"\n\n  2020-11-10 16:51-05:00 to 2020-11-11 13:51-05:00 (21:00) Account activity\n  2020-11-13 13:47-05:00 to 2020-11-14 03:32-05:00 (13:45) Account activity\n  2020-11-14 04:33-05:00 to 2020-11-15 20:33-05:00 (40:00) Account activity\n  2020-11-15 21:34-05:00 to 2020-11-16 13:34-05:00 (16:00) Account activity\n  2020-11-16 16:21-05:00 to 2020-11-16 16:28-05:00 (00:07) Account activity\n  2020-11-17 15:49-05:00 to 2020-11-17 15:49-05:00 (00:00) Account activity\n  2020-11-17 15:51-05:00 Created project \"New project\" (pirca-j7d0g-7bxvkyr4khfa1a4)\n  2020-11-17 15:51-05:00 Updated project \"Test run\" (pirca-j7d0g-7bxvkyr4khfa1a4)\n  2020-11-17 15:51-05:00 Ran container \"bwa-mem.cwl container\" (pirca-xvhdp-xf2w8dkk17jkk5r)\n  2020-11-17 15:51-05:00 to 2020-11-17 15:51-05:00 (0:00) Account activity\n  2020-11-17 15:53-05:00 Ran container \"WGS processing workflow scattered over samples container\" (pirca-xvhdp-u7bm0wdy6lq4r8k)\n  2020-11-17 15:53-05:00 to 2020-11-17 15:54-05:00 (00:01) Account activity\n  2020-11-17 15:55-05:00 Created collection \"output for pirca-dz642-36ffk81c8zzopxz\" (pirca-4zz18-np35gw690ndzzk7)\n  2020-11-17 15:55-05:00 to 2020-11-17 15:55-05:00 (0:00) Account activity\n  2020-11-17 15:55-05:00 Created collection \"Output of main\" (pirca-4zz18-oiiymetwhnnhhwc)\n  2020-11-17 15:55-05:00 Tagged pirca-4zz18-oiiymetwhnnhhwc\n  2020-11-17 15:55-05:00 Updated collection \"Output of main\" (pirca-4zz18-oiiymetwhnnhhwc)\n  2020-11-17 15:55-05:00 to 2020-11-17 16:04-05:00 (00:09) Account activity\n  2020-11-17 16:04-05:00 Created collection \"Output of main\" (pirca-4zz18-f6n9n89e3dhtwvl)\n  2020-11-17 16:04-05:00 Tagged pirca-4zz18-f6n9n89e3dhtwvl\n  2020-11-17 16:04-05:00 Updated collection \"Output of main\" (pirca-4zz18-f6n9n89e3dhtwvl)\n  2020-11-17 16:04-05:00 to 2020-11-17 17:55-05:00 (01:51) Account activity\n  2020-11-17 20:09-05:00 to 2020-11-17 20:09-05:00 (00:00) Account activity\n  2020-11-17 21:35-05:00 to 2020-11-17 21:35-05:00 (00:00) Account activity\n  2020-11-18 10:09-05:00 to 2020-11-18 11:00-05:00 (00:51) Account activity\n  2020-11-18 14:37-05:00 Untagged pirca-4zz18-st8yzjan1nhxo1a\n  2020-11-18 14:37-05:00 Deleted collection \"Output of main\" (pirca-4zz18-st8yzjan1nhxo1a)\n  2020-11-18 17:44-05:00 to 2020-11-18 17:44-05:00 (00:00) Account activity\n  2020-11-19 12:18-05:00 to 2020-11-19 12:19-05:00 (00:01) Account activity\n  2020-11-19 13:57-05:00 to 2020-11-19 14:21-05:00 (00:24) Account activity\n  2020-11-20 09:48-05:00 to 2020-11-20 22:51-05:00 (13:03) Account activity\n  2020-11-20 23:52-05:00 to 2020-11-22 22:32-05:00 (46:40) Account activity\n  2020-11-22 23:37-05:00 to 2020-11-23 13:52-05:00 (14:15) Account activity\n  2020-11-23 14:53-05:00 to 2020-11-24 11:58-05:00 (21:05) Account activity\n  2020-11-24 15:06-05:00 to 2020-11-24 16:38-05:00 (01:32) Account activity\n\nMarc Rubenfield  (https://workbench.pirca.arvadosapi.com/users/jutro-tpzed-v9s9q97pgydh1yf)\n  2020-11-11 12:27-05:00 Untagged pirca-4zz18-xmq257bsla4kdco\n  2020-11-11 12:27-05:00 Deleted collection \"Output of main\" (pirca-4zz18-xmq257bsla4kdco)\n\nWard Vandewege  (https://workbench.pirca.arvadosapi.com/users/jutro-tpzed-9z6foyez9ydn2hl)\n  organization: \"Curii Corporation, Inc.\"\n  organization_email: \"ward@curii.com\"\n  role: \"System Administrator\"\n  website_url: \"https://curii.com\"\n\n  2020-11-19 19:30-05:00 to 2020-11-19 19:46-05:00 (00:16) Account activity\n  2020-11-20 10:51-05:00 to 2020-11-20 11:26-05:00 (00:35) Account activity\n  2020-11-24 12:01-05:00 to 2020-11-24 13:01-05:00 (01:00) Account activity\n
\n" }, { "path": "doc/admin/user-management-cli.html.textile.liquid", "content": "---\nlayout: default\nnavsection: admin\ntitle: User management at the CLI\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nInitial setup\n\n
\nARVADOS_API_HOST={{ site.arvados_api_host }}\nARVADOS_API_TOKEN=1234567890qwertyuiopasdfghjklzxcvbnm1234567890zzzz\n
\n\nIn these examples, @zzzzz-tpzed-3kz0nwtjehhl0u4@ is the sample user account. Replace with the uuid of the user you wish to manipulate.\n\nSee \"user management\":{{site.baseurl}}/admin/user-management.html for an overview of how to use these commands.\n\nh3. Setup a user\n\nThis creates a default git repository and VM login. Enables user to self-activate using Workbench.\n\n\n
$ arv user setup --uuid zzzzz-tpzed-3kz0nwtjehhl0u4\n
\n\n\n\nh3. Deactivate user\n\n\n
$ arv user unsetup --uuid zzzzz-tpzed-3kz0nwtjehhl0u4\n
\n\n\n\nWhen deactivating a user, you may also want to \"reassign ownership of their data\":{{site.baseurl}}/admin/reassign-ownership.html .\n\nh3(#activate-user). Directly activate user\n\n\n
$ arv user update --uuid \"zzzzz-tpzed-3kz0nwtjehhl0u4\" --user '{\"is_active\":true}'\n
\n\n\nNote: this bypasses user agreements checks, and does not set up the user with a default git repository or VM login.\n\nh3(#create-token). Create a token for a user\n\nAs an admin, you can create tokens for other users.\n\n\n
$ arv api_client_authorization create --api-client-authorization '{\"owner_uuid\": \"zzzzz-tpzed-fr97h9t4m5jffxs\"}'\n{\n \"kind\":\"arvados#apiClientAuthorization\",\n \"etag\":\"9yk144t0v6cvyp0342exoh2vq\",\n \"uuid\":\"zzzzz-gj3su-yyyyyyyyyyyyyyy\",\n \"owner_uuid\":\"zzzzz-tpzed-fr97h9t4m5jffxs\",\n \"created_at\":\"2020-03-12T20:36:12.517375422Z\",\n \"modified_by_user_uuid\":null,\n \"modified_at\":null,\n \"api_token\":\"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\",\n \"created_by_ip_address\":null,\n \"expires_at\":null,\n \"last_used_at\":null,\n \"last_used_by_ip_address\":null,\n \"scopes\":[\"all\"]\n}\n
\n\n\n\nTo get the token string, combine the values of @uuid@ and @api_token@ in the form \"v2/$uuid/$api_token\". In this example the string that goes in @ARVADOS_API_TOKEN@ would be:\n\n
\nARVADOS_API_TOKEN=v2/zzzzz-gj3su-yyyyyyyyyyyyyyy/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n
\n\nh3(#delete-token). Delete a single token\n\nAs a user or admin, if you need to revoke a specific, known token, for example a token that may have been leaked to an unauthorized party, you can delete it at the command line.\n\nFirst, determine the token UUID. If it is a \"v2\" format token (starts with \"v2/\") then the token UUID is middle section between the two slashes. For example:\n\n
\nv2/zzzzz-gj3su-yyyyyyyyyyyyyyy/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n
\n\nthe UUID is \"zzzzz-gj3su-yyyyyyyyyyyyyyy\" and you can skip to the next step.\n\nIf you have a \"bare\" token (only the secret part) then, as an admin, you need to query the token to get the uuid:\n\n
\n$ ARVADOS_API_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx arv --format=uuid api_client_authorization current\nzzzzz-gj3su-yyyyyyyyyyyyyyy\n
\n\nNow you can delete the token:\n\n
\n$ ARVADOS_API_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx arv api_client_authorization delete --uuid zzzzz-gj3su-yyyyyyyyyyyyyyy\n
\n\nh3(#delete-all-tokens). Delete all tokens belonging to a user\n\nFirst, \"obtain a valid token for the user.\":#create-token\n\nThen, use that token to get all the user's tokens, and delete each one:\n\n
\n$ ARVADOS_API_TOKEN=xxxxtoken-belonging-to-user-whose-tokens-will-be-deletedxxxxxxxx ; \\\nfor uuid in $(arv --format=uuid api_client_authorization list) ; do \\\narv api_client_authorization delete --uuid $uuid ; \\\ndone\n
\n\nh2. Adding Permissions\n\nh3(#vm-login). VM login\n\nGive @$user_uuid@ permission to log in to @$vm_uuid@ as @$target_username@ and make sure that @$target_username@ is a member of the @docker@ group\n\n
\nuser_uuid=xxxxxxxchangeme\nvm_uuid=xxxxxxxchangeme\ntarget_username=xxxxxxxchangeme\n\nread -rd $'\\000' newlink <\n"
  },
  {
    "path": "doc/admin/user-management.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: admin\ntitle: User management\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n# \"Authentication\":#authentication\n## \"Federated Authentication\":#federated_auth\n# \"User activation\":#user_activation\n# \"User agreements and self-activation\":#user_agreements\n# \"User profile\":#user_profile\n# \"User visibility\":#user_visibility\n# \"Pre-setup user by email address\":#pre-activated\n# \"Pre-activate federated user\":#pre-activated-fed\n# \"Auto-setup federated users from trusted clusters\":#auto_setup_federated\n# \"Activation flows\":#activation_flows\n## \"Private instance\":#activation_flow_private\n## \"Federated instance\":#federated\n## \"Open instance\":#activation_flow_open\n# \"Service Accounts\":#service_accounts\n\n{% comment %}\nTODO: Link to relevant workbench documentation when it gets written\n{% endcomment %}\n\nThis page describes how user accounts are created, set up and activated.\n\nh2(#authentication). Authentication\n\n\"Browser login and management of API tokens is described here.\":{{site.baseurl}}/api/tokens.html\n\nAfter completing the log in and authentication process, the API server receives a user record from the upstream identity provider (Google, LDAP, etc) consisting of the user's name, primary email address, alternate email addresses, and optional unique provider identifier (@identity_url@).\n\nIf a provider identifier is given, the API server searches for a matching user record.\n\nIf a provider identifier is not given, no match is found, it next searches by primary email and then alternate email address.  This enables \"provider migration\":migrating-providers.html and \"pre-activated accounts.\":#pre-activated\n\nIf no user account is found, a new user account is created with the information from the identity provider.\n\nIf a user account has been \"linked\":{{site.baseurl}}/user/topics/link-accounts.html or migrated the API server may follow internal redirects (@redirect_to_user_uuid@) to select the linked or migrated user account.\n\nh3(#federated_auth). Federated Authentication\n\nA federated user follows a slightly different flow.  The client presents a token issued by the remote cluster.  The local API server contacts the remote cluster to verify the user's identity.  This results in a user object (representing the remote user) being created on the local cluster.  If the user cannot be verified, the token will be rejected.  If the user is inactive on the remote cluster, a user record will be created, but it will also be inactive.\n\nh2(#user_activation). User activation\n\nThis section describes the different user account states.\n\n!(side){{site.baseurl}}/images/user-account-states.svg!\n\nnotextile. 
\n\n# A new user record is not set up, and not active.  An inactive user cannot create or update any object, but can read Arvados objects that the user account has permission to read (such as publicly available items readable by the \"anonymous\" user).\n# Using Workbench or the \"command line\":{{site.baseurl}}/admin/user-management-cli.html, the admin invokes @setup@ on the user.  The setup method adds the user to the \"All users\" group.\n- If \"Users.AutoSetupNewUsers\":config.html is true, this happens automatically during user creation, so in that case new users start at step (3).\n- If \"Users.AutoSetupNewUsersWithVmUUID\":config.html is set, the user is given login permission to the specified shell node\n# User is set up, but still not yet active.  The browser presents \"user agreements\":#user_agreements (if any) and then invokes the user @activate@ method on the user's behalf.\n# The user @activate@ method checks that all \"user agreements\":#user_agreements are signed.  If so, or there are no user agreements, the user is activated.\n# The user is active.  User has normal access to the system.\n# From steps (1) and (3), an admin user can directly update the @is_active@ flag.  This bypasses enforcement that user agreements are signed.\nIf the user was not yet set up (still in step (1)), it adds the user to the \"All users\", but bypasses creating default git repository and assigning default VM access.\n# An existing user can have their access revoked using @unsetup@ and \"ownership reassigned\":reassign-ownership.html .\nUnsetup removes the user from the \"All users\" group and makes them inactive, preventing them from re-activating themselves.\n\"Ownership reassignment\":reassign-ownership.html moves any objects or permission from the old user to a new user and deletes any credentials for the old user.\n\nnotextile. 
\n\nUser management can be performed through the web using Workbench or the command line.  See \"user management at the CLI\":{{site.baseurl}}/admin/user-management-cli.html for specific examples.\n\nh2(#user_agreements). User agreements and self-activation\n\nThe @activate@ method of the users controller checks if the user account is part of the \"All Users\" group and whether the user has \"signed\" all the user agreements.\n\nUser agreements are accessed through the \"user_agreements API\":{{site.baseurl}}/api/methods/user_agreements.html .  This returns a list of collection records.\n\nThe user agreements that users are required to sign should be added to the @links@ table this way:\n\n
\n$ arv link create --link '{\n  \"link_class\": \"signature\",\n  \"name\": \"require\",\n  \"tail_uuid\": \"*system user uuid*\",\n  \"head_uuid: \"*collection uuid*\"\n}'\n
\n\nThe collection should contain a single HTML file with the user agreement text.\n\nWorkbench displays the clickthrough agreements which the user can \"sign\".\n\nThe @user_agreements/sign@ endpoint creates a Link object:\n\n
\n{\n  \"link_class\": \"signature\"\n  \"name\": \"click\",\n  \"tail_uuid\": \"*user uuid*\",\n  \"head_uuid: \"*collection uuid*\"\n}\n
\n\nThe @user_agreements/signatures@ endpoint returns the list of Link objects that represent signatures by the current user (created by @sign@).\n\nh2(#user_profile). User profile\n\nThe fields making up the user profile are described in @Workbench.UserProfileFormFields@ .  See \"Configuration reference\":config.html .\n\nThe user profile is checked by workbench after checking if user agreements need to be signed.  The values entered are stored in the @properties@ field on the user object.  Unlike user agreements, the requirement to fill out the user profile is not enforced by the API server.\n\nh2(#user_visibility). User visibility\n\nInitially, a user is not part of any groups and will not be able to interact with other users on the system.  The admin should determine who the user is permited to interact with and use Workbench or the \"command line\":group-management.html#add to create and add the user to the appropriate group(s).\n\nh2(#pre-activated). Pre-setup user by email address\n\nYou may create a user account for a user that has not yet logged in, and identify the user by email address.\n\n1. As an admin, create a user object:\n\n
\n$ arv --format=uuid user create --user '{\"email\": \"foo@example.com\", \"username\": \"foo\"}'\nclsr1-tpzed-1234567890abcdf\n$ arv user setup --uuid clsr1-tpzed-1234567890abcdf\n
\n\n2. When the user logs in the first time, the email address will be recognized and the user will be associated with the existing user object.\n\nh2(#pre-activated-fed). Pre-activate federated user\n\n1. As admin, create a user object with the @uuid@ of the federated user (this is the user's uuid on their home cluster, called @clsr2@ in this example):\n\n
\n$ arv user create --user '{\"uuid\": \"clsr2-tpzed-1234567890abcdf\", \"email\": \"foo@example.com\", \"username\": \"foo\", \"is_active\": true}'\n
\n\n2. When the user logs in, they will be associated with the existing user object.\n\nh2(#auto_setup_federated). Auto-setup federated users from trusted clusters\n\nBy setting @ActivateUsers: true@ for each federated cluster in @RemoteClusters@, a federated user from one of the listed clusters will be automatically set up and activated on this cluster.  See configuration example in \"Federated instance\":#federated .\n\nh2(#activation_flows). Activation flows\n\nh3(#activation_flow_private). Private instance\n\nPolicy: users must be manually set up by the admin.\n\nHere is the configuration for this policy.  This is also the default if not provided.\n\n
\nUsers:\n  AutoSetupNewUsers: false\n
\n\n# User is created.  Not set up.  @is_active@ is false.\n# Workbench checks @is_invited@ and finds it is false.  User gets \"inactive user\" page.\n# Admin goes to user page and clicks \"setup user\" or sets @is_active@ to true.\n# On refreshing workbench, the user is able to self-activate after signing clickthrough agreements (if any).\n# Alternately, directly setting @is_active@ to true also sets up the user, but skips clickthrough agreements (because the user is already active).\n\nh3(#federated). Federated instance\n\nPolicy: users from other clusters in the federation are activated, users from outside the federation must be manually approved.\n\nHere is the configuration for this policy and an example remote cluster @clsr2@.\n\n
\nUsers:\n  AutoSetupNewUsers: false\nRemoteClusters:\n  clsr2:\n    ActivateUsers: true\n
\n\n# Federated user arrives claiming to be from cluster 'clsr2'\n# API server authenticates user as being from cluster 'clsr2'\n# Because 'clsr2' has @ActivateUsers@ the user is set up and activated.\n# User can immediately start using Workbench.\n\nh3(#activation_flow_open). Open instance\n\nPolicy: anybody who shows up and signs the agreements is activated.\n\n
\nUsers:\n  AutoSetupNewUsers: true\n
\n\n\"Set up user agreements\":#user_agreements by creating \"signature\" \"require\" links as described earlier.\n\n# User is created and auto-setup.  At this point, @is_active@ is false, but user has been added to \"All users\" group.\n# Workbench checks @is_invited@ and finds it is true, because the user is a member of \"All users\" group.\n# Workbench presents user with list of user agreements, user reads and clicks \"sign\" for each one.\n# Workbench tries to activate user.\n# User is activated.\n\nh2(#service_accounts). Service Accounts\n\nFor automation purposes, you can create service accounts that aren't tied to an external authorization system. These kind of accounts don't really differ much from standard user accounts, they just cannot be accessed through a normal login mechanism.\n\nAs an admin, you can create accounts like described in the \"user pre-setup section above\":#pre-activated and then \"activate them by updating its @is_active@ field\":{{site.baseurl}}/admin/user-management-cli.html#activate-user.\n\nOnce a service account is created you can \"use an admin account to set up a token\":{{site.baseurl}}/admin/user-management-cli.html#create-token for it, so that the required automations can authenticate. Note that these tokens support having a limited lifetime by using the @expires_at@ field and also \"limited scope\":{{site.baseurl}}/admin/scoped-tokens.html, if required by your security policies. You can read more about them at \"the API reference page\":{{site.baseurl}}/api/methods/api_client_authorizations.html."
  },
  {
    "path": "doc/api/dispatch.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\nnavmenu: API Methods\ntitle: \"cloud dispatcher\"\n\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThe cloud dispatcher provides several management/diagnostic APIs, intended to be used by a system administrator.\n\n{% include 'notebox_begin' %}\nThe \"@arvados-server instance@ subcommand\":{{site.baseurl}}/admin/dispatch.html provides a command-line interface for the most commonly used parts of this API.\n{% include 'notebox_end' %}\n\nThese APIs are not normally exposed to external clients. To use them, connect directly to the dispatcher's internal URL (see Services.DispatchCloud.InternalURLs in the cluster config file). All requests must include the cluster's management token (@ManagementToken@ in the cluster config file).\n\nExample:\n\n
curl -H \"Authorization: Bearer $management_token\" http://localhost:9006/arvados/v1/dispatch/containers
\n\nThese APIs are not available via @arv@ CLI tool.\n\nNote: the term \"instance\" here refers to a virtual machine provided by a cloud computing service. The alternate terms \"cloud VM\", \"compute node\", and \"worker node\" are sometimes used as well in config files, documentation, and log messages.\n\nh3. List containers\n\n@GET /arvados/v1/dispatch/containers@\n\nReturn a list of containers that are either ready to dispatch, or being started/monitored by the dispatcher.\n\nEach entry in the returned list of @items@ includes:\n* an @instance_type@ entry with the name and attributes of the instance type that will be used to schedule the container (chosen from the @InstanceTypes@ section of your cluster config file); and\n* a @container@ entry with selected attributes of the container itself, including @uuid@, @priority@, @runtime_constraints@, and @state@. Other fields of the container records are not loaded by the dispatcher, and will have empty/zero values here (e.g., @{...,\"created_at\":\"0001-01-01T00:00:00Z\",\"command\":[],...}@).\n* a @scheduling_status@ field with a brief explanation of the container's status in the dispatch queue, or an empty string if scheduling is not applicable, e.g., the container has already started running.\n\nExample response:\n\n
{\n  \"items\": [\n    {\n      \"container\": {\n        \"uuid\": \"zzzzz-dz642-xz68ptr62m49au7\",\n        ...\n        \"priority\": 562948375092493200,\n        ...\n        \"state\": \"Locked\",\n        ...\n      },\n      \"instance_type\": {\n        \"Name\": \"Standard_E2s_v3\",\n        \"ProviderType\": \"Standard_E2s_v3\",\n        \"VCPUs\": 2,\n        \"RAM\": 17179869184,\n        \"Scratch\": 32000000000,\n        \"IncludedScratch\": 32000000000,\n        \"AddedScratch\": 0,\n        \"Price\": 0.146,\n        \"Preemptible\": false\n      },\n      \"scheduling_status\": \"Waiting for a Standard_E2s_v3 instance to boot and be ready to accept work.\"\n    },\n    ...\n  ]\n}
\n\nh3. Get specified container\n\n@GET /arvados/v1/dispatch/container?container_uuid={uuid}@\n\nReturn the same information as \"list containers\" above, but for a single specified container.\n\nExample response:\n\n
{\n  \"container\": {\n    ...\n  },\n  \"instance_type\": {\n    ...\n  },\n  \"scheduling_status\": \"Waiting for a Standard_E2s_v3 instance to boot and be ready to accept work.\"\n}
\n\nh3. Terminate a container\n\n@POST /arvados/v1/dispatch/containers/kill?container_uuid={uuid}&reason={string}@\n\nMake a single attempt to terminate the indicated container on the relevant instance. (The caller can implement a delay-and-retry loop if needed.)\n\nA container terminated this way will end with state @Cancelled@ if its docker container had already started, or @Queued@ if it was terminated while setting up the runtime environment.\n\nThe provided @reason@ string will appear in the dispatcher's log, but not in the user-visible container log.\n\nIf the provided @container_uuid@ is not scheduled/running on an instance, the response status will be 404.\n\nh3. List instances\n\n@GET /arvados/v1/dispatch/instances@\n\nReturn a list of cloud instances.\n\nExample response:\n\n
{\n  \"items\": [\n    {\n      \"instance\": \"/subscriptions/abcdefab-abcd-abcd-abcd-abcdefabcdef/resourceGroups/zzzzz/providers/Microsoft.Compute/virtualMachines/compute-abcdef0123456789abcdef0123456789-abcdefghijklmno\",\n      \"address\": \"10.23.45.67\",\n      \"price\": 0.073,\n      \"arvados_instance_type\": \"Standard_DS1_v2\",\n      \"provider_instance_type\": \"Standard_DS1_v2\",\n      \"last_container_uuid\": \"zzzzz-dz642-vp7scm21telkadq\",\n      \"last_busy\": \"2020-01-13T15:20:21.775019617Z\",\n      \"running_container_uuids\": [\"zzzzz-dz642-vp7scm21telkadq\"],\n      \"worker_state\": \"running\",\n      \"idle_behavior\": \"run\"\n    },\n    ...\n}
\n\nThe @instance@ value is the instance's identifier, assigned by the cloud provider. It can be used with the instance APIs below.\n\nThe @last_container_uuid@ value indicates the most recently started container, if any. (It does not necessarily indicate that the container is still running.)\n\nThe @worker_state@ value indicates the instance's capability to run containers.\n* @unknown@: instance was not created by this dispatcher, and a boot probe has not yet succeeded (this state typically appears briefly after the dispatcher restarts).\n* @booting@: cloud provider says the instance exists, but a boot probe has not yet succeeded.\n* @idle@: instance is idle and ready to run a container.\n* @running@: instance is running one or more containers.\n* @shutdown@: cloud provider has been instructed to terminate the instance.\n\nThe @idle_behavior@ value determines what the dispatcher will do with the instance when it is idle; see hold/drain/run APIs below.\n\nh3. Hold an instance\n\n@POST /arvados/v1/dispatch/instances/hold?instance_id={instance}@\n\nSet the indicated instance's idle behavior to @hold@. The instance will not be shut down automatically. If any containers are currently running, they will be allowed to continue, but no new containers will be scheduled.\n\nh3. Drain an instance\n\n@POST /arvados/v1/dispatch/instances/drain?instance_id={instance}@\n\nSet the indicated instance's idle behavior to @drain@. If any containers are currently running, they will be allowed to continue, but when the instance becomes idle, it will be shut down.\n\nh3. Resume an instance\n\n@POST /arvados/v1/dispatch/instances/run?instance_id={instance}@\n\nSet the indicated instance's idle behavior to @run@ (the normal behavior). When it becomes idle, it will be eligible to run new containers. It will be shut down automatically when the configured idle threshold is reached.\n\nh3. Shut down an instance\n\n@POST /arvados/v1/dispatch/instances/kill?instance_id={instance}&reason={string}@\n\nTerminate the indicated instance.\n\nIf any containers are running on the instance, they will be killed too; no effort is made to wait for them to end gracefully.\n\nThe provided @reason@ string will appear in the dispatcher's log.\n"
  },
  {
    "path": "doc/api/execution.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: architecture\ntitle: Computing with Crunch\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nCrunch is the name for the Arvados system for managing computation.  It provides an abstract API to various clouds and HPC resource allocation and scheduling systems, and integrates closely with Keep storage and the Arvados permission system.\n\nh2. Container API\n\n# To submit work, create a \"container request\":{{site.baseurl}}/api/methods/container_requests.html in the @Committed@ state.\n# The system will fufill the container request by creating or reusing a \"Container object\":{{site.baseurl}}/api/methods/containers.html and assigning it to the @container_uuid@ field.  If the same request has been submitted in the past, it may reuse an existing container.  The reuse behavior can be suppressed with @use_existing: false@ in the container request.\n# The dispatcher process will notice a new container in @Queued@ state and submit a container executor to the underlying work queuing system (such as Slurm).\n# The container executes.  Upon termination the container goes into the  @Complete@ state.  If the container execution was interrupted or lost due to system failure, it will go into the @Cancelled@ state.\n# When the container associated with the container request is completed, the container request will go into the @Final@ state.\n# The @output_uuid@ field of the container request contains the uuid of output collection produced by container request.\n\n!(full-width){{site.baseurl}}/images/Crunch_dispatch.svg!\n\nh2(#RAM). Understanding RAM requests for containers\n\nThe @runtime_constraints@ section of a container specifies working RAM (@ram@) and Keep cache (@keep_cache_ram@).  If not specified, containers get a default Keep cache (@container_default_keep_cache_ram@, default 256 MiB).  The total RAM requested for a container is the sum of working RAM, Keep cache, and an additional RAM reservation configured by the admin (@ReserveExtraRAM@ in the dispatcher configuration, default zero).\n\nThe total RAM request is used to schedule containers onto compute nodes.  RAM allocation limits are enforced using kernel controls such as cgroups.  A container which requests 1 GiB RAM will only be permitted to allocate up to 1 GiB of RAM, even if scheduled on a 4 GiB node.  On HPC systems, a multi-core node may run multiple containers at a time.\n\nWhen running on the cloud, the memory request (along with CPU and disk) is used to select (and possibly boot) an instance type with adequate resources to run the container.  Instance type RAM is derated 5% from the published specification to accomodate virtual machine, kernel and system services overhead.\n\nh3. Calculate minimum instance type RAM for a container\n\n    (RAM request + Keep cache + ReserveExtraRAM) * (100/95)\n\nFor example, for a 3 GiB request, default Keep cache, and no extra RAM reserved:\n\n    (3072 + 256) * 1.0526 = 3494 MiB\n\nTo run this container, the instance type must have a published RAM size of at least 3494 MiB.\n\nh3. Calculate the maximum requestable RAM for an instance type\n\n    (Instance type RAM * (95/100)) - Keep cache - ReserveExtraRAM\n\nFor example, for a 3.75 GiB node, default Keep cache, and no extra RAM reserved:\n\n    (3840 * 0.95) - 256 = 3392 MiB\n\nTo run on this instance type, the container can request at most 3392 MiB of working RAM.\n"
  },
  {
    "path": "doc/api/index.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\ntitle: API Reference\n\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThis reference describes the semantics of Arvados resources and how to programatically access Arvados via its REST API.  Each resource listed in this section is exposed on the Arvados API server under the @/arvados/v1/@ path prefix, for example, @https://{{ site.arvados_api_host }}/arvados/v1/collections@.\n\nh2. Discovery document\n\nThe API server publishes a machine-readable description of its endpoints and some additional site configuration values via a JSON-formatted discovery document.  This is available at @/discovery/v1/apis/arvados/v1/rest@, for example @https://{{ site.arvados_api_host }}/discovery/v1/apis/arvados/v1/rest@.  Some Arvados SDKs use the discovery document to generate language bindings.\n\nh2. Exported configuration\n\nThe Controller exposes a subset of the cluster's configuration and makes it available to clients in JSON format. This public config includes valuable information like several service's URLs, timeout settings, etc. and it is available at @/arvados/v1/config@, for example @https://{{ site.arvados_api_host }}/arvados/v1/config@. Workbench is one example of a client using this information, as it's a client-side application and doesn't have access to the cluster's config file.\n\nh2. Exported vocabulary definition\n\nWhen configured, the Controller also exports the \"metadata vocabulary definition\":{{site.baseurl}}/admin/metadata-vocabulary.html in JSON format. This functionality is useful for clients like Workbench and the Python SDK to translate between identifiers and human-readable labels when reading and writing objects on the system. This is available at @/arvados/v1/vocabulary@, for example @https://{{ site.arvados_api_host }}/arvados/v1/vocabulary@.\n"
  },
  {
    "path": "doc/api/keep-s3.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\nnavmenu: API Methods\ntitle: \"S3 API\"\n\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThe Simple Storage Service (S3) API is a de-facto standard for object storage originally developed by Amazon Web Services.  Arvados supports accessing files in Keep using the S3 API.\n\nS3 is supported by many \"cloud native\" applications, and client libraries exist in many languages for programmatic access.\n\nh3. Endpoints and Buckets\n\nTo access Arvados S3 using an S3 client library, you must tell it to use the URL of the keep-web server (this is @Services.WebDAVDownload.ExternalURL@ in the public configuration) as the custom endpoint.  The keep-web server will decide to treat it as an S3 API request based on the presence of an AWS-format Authorization header.  Requests without an Authorization header, or differently formatted Authorization, will be treated as \"WebDAV\":keep-webdav.html .\n\nThe \"bucket name\" is an Arvados collection uuid, portable data hash, or project uuid.\n\nPath-style and virtual host-style requests are supported.\n* A path-style request uses the hostname indicated by @Services.WebDAVDownload.ExternalURL@, with the bucket name in the first path segment: @https://download.example.com/zzzzz-4zz18-asdfgasdfgasdfg/@.\n* A virtual host-style request uses the hostname pattern indicated by @Services.WebDAV.ExternalURL@, with a bucket name in place of the leading @*@: @https://zzzzz-4zz18-asdfgasdfgasdfg.collections.example.com/@.\n\nIf you have wildcard DNS, TLS, and routing set up, an S3 client configured with endpoint @collections.example.com@ should work regardless of which request style it uses.\n\nh3. Supported Operations\n\nh4. ListObjects\n\nSupports the following request query parameters:\n\n* delimiter\n* marker\n* max-keys\n* prefix\n\nh4. GetObject\n\nSupports the @Range@ header.\n\nh4. PutObject\n\nCan be used to create or replace a file in a collection.\n\nAn empty PUT with a trailing slash and @Content-Type: application/x-directory@ will create a directory within a collection if Arvados configuration option @Collections.S3FolderObjects@ is true.\n\nMissing parent/intermediate directories within a collection are created automatically.\n\nCannot be used to create a collection or project.\n\nh4. DeleteObject\n\nCan be used to remove files from a collection.\n\nIf used on a directory marker, it will delete the directory only if the directory is empty.\n\nh4. HeadBucket\n\nCan be used to determine if a bucket exists and if client has read access to it.\n\nh4. HeadObject\n\nCan be used to determine if an object exists and if client has read access to it.\n\nh4. GetBucketVersioning\n\nBucket versioning is presently not supported, so this will always respond that bucket versioning is not enabled.\n\nh3. Accessing collection/project properties as metadata\n\nGetObject, HeadObject, and HeadBucket return Arvados object properties as S3 metadata headers, e.g., @X-Amz-Meta-Foo: bar@.\n\nIf the requested path indicates a file or directory placeholder inside a collection, or the top level of a collection, GetObject and HeadObject return the collection properties.\n\nIf the requested path indicates a directory placeholder corresponding to a project, GetObject and HeadObject return the properties of the project.\n\nHeadBucket returns the properties of the collection or project corresponding to the bucket name.\n\nNon-string property values are returned in a JSON representation, e.g., @[\"foo\",\"bar\"]@.\n\nAs in Amazon S3, property values containing non-ASCII characters are returned in BASE64-encoded form as described in RFC 2047, e.g., @=?UTF-8?b?4pu1?=@.\n\nGetBucketTagging and GetObjectTagging APIs are _not_ supported.\n\nIt is not possible to modify collection or project properties using the S3 API.\n\nh3. Authorization mechanisms\n\nKeep-web accepts AWS Signature Version 4 (AWS4-HMAC-SHA256) as well as the older V2 AWS signature.\n\nIf your client uses V4 signatures exclusively _and_ your Arvados token was issued by the same cluster you are connecting to, you can use the Arvados token's UUID part as your S3 Access Key, and its secret part as your S3 Secret Key. This is preferred, where applicable.\n\nExample using cluster @zzzzz@:\n* Arvados token: @v2/zzzzz-gj3su-yyyyyyyyyyyyyyy/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@\n* Access Key: @zzzzz-gj3su-yyyyyyyyyyyyyyy@\n* Secret Key: @xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@\n\nIn all other cases, replace every @/@ character in your Arvados token with @_@, and use the resulting string as both Access Key and Secret Key.\n\nExample using a cluster other than @zzzzz@ _or_ an S3 client that uses V2 signatures:\n* Arvados token: @v2/zzzzz-gj3su-yyyyyyyyyyyyyyy/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@\n* Access Key: @v2_zzzzz-gj3su-yyyyyyyyyyyyyyy_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@\n* Secret Key: @v2_zzzzz-gj3su-yyyyyyyyyyyyyyy_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@\n"
  },
  {
    "path": "doc/api/keep-web-urls.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\nnavmenu: API Methods\ntitle: \"Keep-web URL patterns\"\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nFiles served by @keep-web@ can be rendered directly in the browser, or @keep-web@ can instruct the browser to only download the file.\n\nWhen serving files that will render directly in the browser, it is important to properly configure the keep-web service to migitate cross-site-scripting (XSS) attacks.  A HTML page can be stored in a collection.  If an attacker causes a victim to visit that page through Workbench, the HTML will be rendered by the browser.  If all collections are served at the same domain, the browser will consider collections as coming from the same origin, which will grant access to the same browsing data (cookies and local storage).  This would enable malicious Javascript on that page to access Arvados on behalf of the victim.\n\nThis can be mitigated by having separate domains for each collection, or limiting preview to circumstances where the collection is not accessed with the user's regular full-access token.  For clusters where this risk is acceptable, this protection can also be turned off by setting the @Collections/TrustAllContent@ configuration flag to true, see the \"configuration reference\":../admin/config.html for more detail.\n\nThe following \"same origin\" URL patterns are supported for public collections and collections shared anonymously via secret links (i.e., collections which can be served by keep-web without making use of any implicit credentials like cookies). See \"Same-origin URLs\" below.\n\n
\nhttp://collections.example.com/c=uuid_or_pdh/path/file.txt\nhttp://collections.example.com/c=uuid_or_pdh/t=TOKEN/path/file.txt\n
\n\nThe following \"multiple origin\" URL patterns are supported for all collections:\n\n
\nhttp://uuid_or_pdh--collections.example.com/path/file.txt\nhttp://uuid_or_pdh--collections.example.com/t=TOKEN/path/file.txt\n
\n\nIn the \"multiple origin\" form, the string @--@ can be replaced with @.@ with identical results (assuming the downstream proxy is configured accordingly). These two are equivalent:\n\n
\nhttp://uuid_or_pdh--collections.example.com/path/file.txt\nhttp://uuid_or_pdh.collections.example.com/path/file.txt\n
\n\nThe first form (with @--@ instead of @.@) avoids the cost and effort of deploying a wildcard TLS certificate for @*.collections.example.com@ at sites that already have a wildcard certificate for @*.example.com@ . The second form is likely to be easier to configure, and more efficient to run, on a downstream proxy.\n\nIn all of the above forms, the @collections.example.com@ part can be anything at all: keep-web itself ignores everything after the first @.@ or @--@. (Of course, in order for clients to connect at all, DNS and any relevant proxies must be configured accordingly.)\n\nIn all of the above forms, the @uuid_or_pdh@ part can be either a collection UUID or a portable data hash with the @+@ character optionally replaced by @-@ . (When @uuid_or_pdh@ appears in the domain name, replacing @+@ with @-@ is mandatory, because @+@ is not a valid character in a domain name.)\n\nIn all of the above forms, a top level directory called @_@ is skipped. In cases where the @path/file.txt@ part might start with @t=@ or @c=@ or @_/@, links should be constructed with a leading @_/@ to ensure the top level directory is not interpreted as a token or collection ID.\n\nAssuming there is a collection with UUID @zzzzz-4zz18-znfnqtbbv4spc3w@ and portable data hash @1f4b0bc7583c2a7f9102c395f4ffc5e3+45@, the following URLs are interchangeable:\n\n
\nhttp://zzzzz-4zz18-znfnqtbbv4spc3w.collections.example.com/foo/bar.txt\nhttp://zzzzz-4zz18-znfnqtbbv4spc3w.collections.example.com/_/foo/bar.txt\nhttp://zzzzz-4zz18-znfnqtbbv4spc3w--collections.example.com/_/foo/bar.txt\n
\n\nThe following URLs are read-only, but will return the same content as above:\n\n
\nhttp://1f4b0bc7583c2a7f9102c395f4ffc5e3-45--foo.example.com/foo/bar.txt\nhttp://1f4b0bc7583c2a7f9102c395f4ffc5e3-45--.invalid/foo/bar.txt\nhttp://collections.example.com/by_id/1f4b0bc7583c2a7f9102c395f4ffc5e3%2B45/foo/bar.txt\nhttp://collections.example.com/by_id/zzzzz-4zz18-znfnqtbbv4spc3w/foo/bar.txt\n
\n\nIf the collection is named \"MyCollection\" and located in a project called \"MyProject\" which is in the home project of a user with username is \"bob\", the following read-only URL is also available when authenticating as bob:\n\npre. http://collections.example.com/users/bob/MyProject/MyCollection/foo/bar.txt\n\nAn additional form is supported specifically to make it more convenient to maintain support for existing Workbench download links:\n\npre. http://collections.example.com/collections/download/uuid_or_pdh/TOKEN/foo/bar.txt\n\nA regular Workbench \"download\" link is also accepted, but credentials passed via cookie, header, etc. are ignored. Only public data can be served this way:\n\npre. http://collections.example.com/collections/uuid_or_pdh/foo/bar.txt\n\nh2(#same-site). Same-site requirements for requests with tokens\n\nAlthough keep-web doesn't care about the domain part of the URL, the clients do: especially when rendering inline content.\n\nWhen a client passes a token in the URL, keep-web sends a redirect response placing the token in a @Set-Cookie@ header with the @SameSite=Lax@ attribute. The browser will ignore the cookie if it's not coming from a _same-site_ request, and thus its subsequent request will fail with a @401 Unauthorized@ error.\n\nThis mainly affects Workbench's ability to show inline content, so it should be taken into account when configuring both services' URL schemes.\n\nYou can read more about the definition of a _same-site_ request at the \"RFC 6265bis-03 page\":https://tools.ietf.org/html/draft-ietf-httpbis-rfc6265bis-03#section-5.2\n"
  },
  {
    "path": "doc/api/keep-webdav.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\nnavmenu: API Methods\ntitle: \"WebDAV\"\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n\"Web Distributed Authoring and Versioning (WebDAV)\":https://tools.ietf.org/html/rfc4918 is an IETF standard set of extensions to HTTP to manipulate and retrieve hierarchical web resources, similar to directories in a file system.  Arvados supports accessing files in Keep using WebDAV.\n\nMost major operating systems include built-in support for mounting WebDAV resources as network file systems, see user guide sections for \"Windows\":{{site.baseurl}}/user/tutorials/tutorial-keep-mount-windows.html, \"macOS\":{{site.baseurl}}/user/tutorials/tutorial-keep-mount-os-x.html, or \"Linux (GNOME)\":{{site.baseurl}}/user/tutorials/tutorial-keep-mount-gnu-linux.html#gnome.  WebDAV is also supported by various standalone storage browser applications such as \"Cyberduck\":https://cyberduck.io/ and client libraries exist in many languages for programmatic access.\n\nKeep-web provides read/write HTTP (WebDAV) access to files stored in Keep. It serves public data to anonymous and unauthenticated clients, and serves private data to clients that supply Arvados API tokens.\n\nh3. Supported Operations\n\nSupports WebDAV HTTP methods @GET@, @PUT@, @DELETE@, @PROPFIND@, @COPY@, and @MOVE@.\n\nDoes not support @LOCK@ or @UNLOCK@.  These methods will be accepted, but are no-ops.\n\nh3. Browsing\n\nRequests can be authenticated a variety of ways as described below in \"Authentication mechanisms\":#auth .  An unauthenticated request will return a 401 Unauthorized response with a @WWW-Authenticate@ header indicating \"support for RFC 7617 Basic Authentication\":https://tools.ietf.org/html/rfc7617 .\n\nGetting a listing from keep-web starting at the root path @/@ will return two folders, @by_id@ and @users@.\n\nThe @by_id@ folder will return an empty listing.  However, a path which starts with /by_id/ followed by a collection uuid, portable data hash, or project uuid will return the listing of that object.\n\nThe @users@ folder will return a listing of the users for whom the client has permission to read the \"home\" project of that user.  Browsing an individual user will return the collections and projects directly owned by that user.  Browsing those collections and projects return listings of the files, directories, collections, and subprojects they contain, and so forth.\n\nIn addition to the @/by_id/@ path prefix, the collection or project can be specified using a path prefix of @/c=/@ or (if the cluster is properly configured) as a virtual host.  This is described on \"Keep-web URLs\":keep-web-urls.html\n\nIt is possible for a project or a \"filter group\":methods/groups.html#filter to appear as its own descendant in the @by_id@ and @users@ tree (a filter group may match itself, its own ancestor, another filter group that matches its ancestor, etc). When this happens, the descendant appears as an empty read-only directory. For example, if filter group @f@ matches its own parent @p@:\n* @/users/example/p/f@ will show the filter group's contents (matched projects and collections).\n* @/users/example/p/f/p@ will appear as an empty directory.\n* @/by_id/uuid_of_f/p@ will show the parent project's contents, including @f@.\n* @/by_id/uuid_of_f/p/f@ will appear as an empty directory.\n\nh3(#zip). Downloading ZIP archives\n\nKeep-web can produce an uncompressed ZIP archive of a collection, or a subset of a collection.\n\nTo request a ZIP archive:\n* The request must include an @Accept: application/zip@ header _or_ @?accept=application/zip&disposition=attachment@ in the query.\n* The request URI must specify the root directory of a collection, e.g., @/by_id//@.  See \"Keep-web URLs\":keep-web-urls.html for more examples.\n\nTo download a subset of a collection, the request can specify one or more pathnames relative to the collection directory:\n* A @files@ parameter in the query of a @GET@ request, e.g., @https://.collections.example.com/?files=file1&files=file2@,\n* A @files@ parameter in the body of a @POST@ request with a @Content-Type: application/x-www-form-urlencoded@ header, or\n* The value of a @files@ key in a JSON object in the body of a @POST@ request with a @Content-Type: application/json@ header, e.g., @{\"files\":[\"file1\",\"file2\"]}@.\n\nKeep-web returns an error if one of the specified paths does not exist in the requested collection.\n\nThe ZIP archive comment will include a download URL with the collection UUID or portable data hash, e.g., \"Downloaded from https://collections.example.com/by_id/zzzzz-4zz18-0pg114rezrbz46u/\".\n\nThe ZIP archive will also include collection metadata if the request sets an @include_collection_metadata@ parameter, e.g., @https://.collections.example.com/?include_collection_metadata=true@. The resulting ZIP archive will also include a file named @collection.json@ containing the collection's metadata (UUID, name, description, portable data hash, properties, creation time, modification time) and information about the user who last modified it (UUID, full name, username, and email). If the collection is specified by portable data hash rather than name or UUID, @collection.json@ will contain only the portable data hash.\n\nExample @collection.json@ content:\n\n
\n{\n  \"created_at\":\"2025-04-28T19:50:49.046969000Z\",\n  \"description\":\"Description of test collection\\n\",\n  \"modified_at\":\"2025-04-28T19:50:49.093166000Z\",\n  \"modified_by_user\":{\n    \"email\":\"example@example.com\",\n    \"full_name\":\"Example Name\",\n    \"username\":\"example\",\n    \"uuid\":\"zzzzz-tpzed-xurymjxw79nv3jz\"\n  },\n  \"name\":\"collection name\",\n  \"portable_data_hash\":\"6acf043b102afcf04e3be2443e7ea2ba+223\",\n  \"properties\":{\n    \"key\":\"value\"\n  },\n  \"uuid\":\"zzzzz-4zz18-0pg114rezrbz46u\"\n}\n
\n\nThe request can also include a @download_filename@ parameter with a desired name for the downloaded zip file. This filename will be included in the @Content-Disposition@ response header. If this parameter is not provided, the filename suggested in the response header will be based on the collection name or portable data hash:\n* @{collection name}.zip@ if downloading an entire collection\n* @{collection name} - {file name}.zip@ if a single file was specified in the request\n* @{collection name} - 3 files.zip@ if a directory or multiple files were specified in the request\n* @{portable data hash}.zip@, @{portable data hash} - {file name}.zip@, etc., if the source collection was specified by portable data hash rather than name or UUID\n\nExample request:\n\n
\nGET /by_id/zzzzz-4zz18-0pg114rezrbz46u\nAccept: application/zip\nContent-Type: application/json\n\n{\n  \"download_filename\": \"odd-numbered files and directories.zip\",\n  \"files\": [\n    \"file1.txt\",\n    \"file3.bin\",\n    \"dir5\"\n  ],\n  \"include_collection_metadata\": true\n}\n
\n\nh3(#auth). Authentication mechanisms\n\nA token can be provided in an Authorization header as a @Bearer@ token:\n\n
\nAuthorization: Bearer o07j4px7RlJK4CuMYp7C0LDT4CzR1J1qBE5Avo7eCcUjOTikxK\n
\n\nA token can also be provided with \"RFC 7617 Basic Authentication\":https://tools.ietf.org/html/rfc7617 in this case, the payload is formatted as @username:token@ and encoded with base64.  The username must be non-empty, but is ignored.  In this example, the username is \"user\":\n\n
\nAuthorization: Basic dXNlcjpvMDdqNHB4N1JsSks0Q3VNWXA3QzBMRFQ0Q3pSMUoxcUJFNUF2bzdlQ2NVak9UaWt4Swo=\n
\n\nA base64-encoded token can be provided in a cookie named \"api_token\":\n\n
\nCookie: api_token=bzA3ajRweDdSbEpLNEN1TVlwN0MwTERUNEN6UjFKMXFCRTVBdm83ZUNjVWpPVGlreEs=\n
\n\nA token can be provided in an URL-encoded query string:\n\n
\nGET /foo/bar.txt?api_token=o07j4px7RlJK4CuMYp7C0LDT4CzR1J1qBE5Avo7eCcUjOTikxK\n
\n\nA token can be provided in a URL-encoded path (as described in the previous section):\n\n
\nGET /t=o07j4px7RlJK4CuMYp7C0LDT4CzR1J1qBE5Avo7eCcUjOTikxK/_/foo/bar.txt\n
\n\nA suitably encoded token can be provided in a POST body if the request has a content type of application/x-www-form-urlencoded or multipart/form-data:\n\n
\nPOST /foo/bar.txt\nContent-Type: application/x-www-form-urlencoded\n[...]\napi_token=o07j4px7RlJK4CuMYp7C0LDT4CzR1J1qBE5Avo7eCcUjOTikxK\n
\n\nIf a token is provided in a query string or in a POST request, the response is an HTTP 303 redirect to an equivalent GET request, with the token stripped from the query string and added to a cookie instead.\n\nh3. Indexes\n\nKeep-web returns a generic HTML index listing when a directory is requested with the GET method. It does not serve a default file like \"index.html\". Directory listings are also returned for WebDAV PROPFIND requests.\n\nh3. Range requests\n\nKeep-web supports partial resource reads using the HTTP @Range@ header as specified in \"RFC 7233\":https://tools.ietf.org/html/rfc7233 .\n\nh3. Compatibility\n\nClient-provided authorization tokens are ignored if the client does not provide a @Host@ header.\n\nIn order to use the query string or a POST form authorization mechanisms, the client must follow 303 redirects; the client must accept cookies with a 303 response and send those cookies when performing the redirect; and either the client or an intervening proxy must resolve a relative URL (\"//host/path\") if given in a response Location header.\n\nh3. Intranet mode\n\nNormally, Keep-web accepts requests for multiple collections using the same host name, provided the client's credentials are not being used. This provides insufficient XSS protection in an installation where the \"anonymously accessible\" data is not truly public, but merely protected by network topology.\n\nIn such cases -- for example, a site which is not reachable from the internet, where some data is world-readable from Arvados's perspective but is intended to be available only to users within the local network -- the downstream proxy should configured to return 401 for all paths beginning with \"/c=\".\n\nh3. Same-origin URLs\n\nWithout the same-origin protection outlined above, a web page stored in collection X could execute JavaScript code that uses the current viewer's credentials to download additional data from collection Y -- data which is accessible to the current viewer, but not to the author of collection X -- from the same origin (``https://collections.example.com/'') and upload it to some other site chosen by the author of collection X.\n"
  },
  {
    "path": "doc/api/methods/api_client_authorizations.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\nnavmenu: API Methods\ntitle: \"api_client_authorizations\"\n\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nAPI endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/api_client_authorizations@\n\nObject type: @gj3su@\n\nExample UUID: @zzzzz-gj3su-0123456789abcde@\n\nh2. Resource\n\nThe @api_client_authorizations@ resource stores the API tokens that have been issued to permit access the API server.\n\nAn ApiClientAuthorization is *not* a generic Arvados resource.  The full list of properties that belong to an ApiClientAuthorization is:\n\ntable(table table-bordered table-condensed).\n|_. Attribute|_. Type|_. Description|_. Example|\n|uuid|string|An identifier used to refer to the token without exposing the actual token.||\n|api_token|string|The actual token string that is expected in the Authorization header.||\n|created_by_ip_address|string|-||\n|last_used_by_ip_address|string|The network address of the most recent client using this token.||\n|last_used_at|datetime|Timestamp of the most recent request using this token.||\n|expires_at|datetime|Time at which the token is no longer valid.  May be set to a time in the past in order to immediately expire a token.||\n|owner_uuid|string|The user associated with the token.  All operations using this token are checked against the permissions of this user.||\n|scopes|array|A list of resources this token is allowed to access.  A scope of [\"all\"] allows all resources.  See \"API Authorization\":{{site.baseurl}}/api/tokens.html#scopes for details.||\n\nh2. Methods\n\nSee \"Common resource methods\":{{site.baseurl}}/api/methods.html for more information about @create@, @delete@, @get@, @list@, and @update@.\n\nRequired arguments are displayed in %{background:#ccffcc}green%.\n\nh3(#create). create\n\nCreate a new ApiClientAuthorization.\n\nRegular users may only create self-owned API tokens, but may provide a restricted \"scope\":{{site.baseurl}}/api/tokens.html#scopes .  Administrators may create API tokens corresponding to any user.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n|api_client_authorization|object||query||\n\nh3. create_system_auth\n\ncreate_system_auth api_client_authorizations\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n|scopes|array||query||\n\nh3(#current). current\n\nReturn the full record associated with the provided API token. This endpoint is often used to check the validity of a given token.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n\nh3. delete\n\nDelete an existing ApiClientAuthorization.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the ApiClientAuthorization in question.|path||\n\nh3. get\n\nGets an ApiClientAuthorization's metadata by UUID.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the ApiClientAuthorization in question.|path||\n\nh3. list\n\nList api_client_authorizations.\n\nSee \"common resource list method.\":{{site.baseurl}}/api/methods.html#index\n\nh3. update\n\nUpdate attributes of an existing ApiClientAuthorization.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the ApiClientAuthorization in question.|path||\n|api_client_authorization|object||query||\n"
  },
  {
    "path": "doc/api/methods/authorized_keys.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\nnavmenu: API Methods\ntitle: \"authorized_keys\"\n\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nAPI endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/authorized_keys@\n\nObject type: @fngyi@\n\nExample UUID: @zzzzz-fngyi-0123456789abcde@\n\nh2. Resource\n\nThe authorized_keys resource stores SSH public keys which grant access to virtual machines or git repositories on the Arvados cluster as the user in @authorized_user_uuid@.\n\nEach AuthorizedKey has, in addition to the \"Common resource fields\":{{site.baseurl}}/api/resources.html:\n\ntable(table table-bordered table-condensed).\n|_. Attribute|_. Type|_. Description|_. Example|\n|name|string|A name to help the user manage their keys.||\n|key_type|string|Public key type, currently only supports \"SSH\"||\n|authorized_user_uuid|string|The user to which this key belongs.  Authentication using this key authenticates as this user.||\n|public_key|text|The actual public key material, e.g., from @~/.ssh/id_rsa.pub@||\n|expires_at|datetime|Expiration date after which the key is no longer valid.||\n\nh2. Methods\n\nSee \"Common resource methods\":{{site.baseurl}}/api/methods.html for more information about @create@, @delete@, @get@, @list@, and @update@.\n\nRequired arguments are displayed in %{background:#ccffcc}green%.\n\nh3. create\n\nCreate a new AuthorizedKey.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n|authorized_key|object||query||\n\nh3. delete\n\nDelete an existing AuthorizedKey.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the AuthorizedKey in question.|path||\n\nh3. get\n\nGets a AuthorizedKey's metadata by UUID.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the AuthorizedKey in question.|path||\n\nh3. list\n\nList authorized_keys.\n\nSee \"common resource list method.\":{{site.baseurl}}/api/methods.html#index\n\nh3. update\n\nUpdate attributes of an existing AuthorizedKey.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the AuthorizedKey in question.|path||\n|authorized_key|object||query||\n"
  },
  {
    "path": "doc/api/methods/collections.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\nnavmenu: API Methods\ntitle: \"collections\"\n\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nAPI endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/collections@\n\nObject type: @4zz18@\n\nExample UUID: @zzzzz-4zz18-0123456789abcde@\n\nh2. Resource\n\nCollections describe sets of files in terms of data blocks stored in Keep.  See \"Keep - Content-Addressable Storage\":{{site.baseurl}}/architecture/storage.html and \"using collection versioning\":../../user/topics/collection-versioning.html for details.\n\nEach collection has, in addition to the \"Common resource fields\":{{site.baseurl}}/api/resources.html:\n\ntable(table table-bordered table-condensed).\n|_. Attribute|_. Type|_. Description|_. Example|\n|name|string|||\n|description|text|Free text description of the group.  Allows \"HTML formatting.\":{{site.baseurl}}/api/resources.html#descriptions||\n|properties|hash|User-defined metadata, may be used in queries using \"subproperty filters\":{{site.baseurl}}/api/methods.html#subpropertyfilters ||\n|portable_data_hash|string|The MD5 sum of the manifest text stripped of block hints other than the size hint.||\n|manifest_text|text|The manifest describing how to assemble blocks into files, in the \"Arvados manifest format\":{{site.baseurl}}/architecture/manifest-format.html||\n|replication_desired|number|Minimum storage replication level desired for each data block referenced by this collection. A value of @null@ signifies that the site default replication level (typically 2) is desired.|@2@|\n|replication_confirmed|number|Replication level most recently confirmed by the storage system. This field is null when a collection is first created, and is reset to null when the manifest_text changes in a way that introduces a new data block. An integer value indicates the replication level of the _least replicated_ data block in the collection.|@2@, null|\n|replication_confirmed_at|datetime|When @replication_confirmed@ was confirmed. If @replication_confirmed@ is null, this field is also null.||\n|storage_classes_desired|list|An optional list of storage class names where the blocks should be saved. If not provided, the cluster's default storage class(es) will be set.|@['archival']@|\n|storage_classes_confirmed|list|Storage classes most recently confirmed by the storage system. This field is an empty list when a collection is first created.|@'archival']@, @[]@|\n|storage_classes_confirmed_at|datetime|When @storage_classes_confirmed@ was confirmed. If @storage_classes_confirmed@ is @[]@, this field is null.||\n|trash_at|datetime|If @trash_at@ is non-null and in the past, this collection will be hidden from API calls.  May be untrashed.||\n|delete_at|datetime|If @delete_at@ is non-null and in the past, the collection may be permanently deleted.||\n|is_trashed|boolean|True if @trash_at@ is in the past, false if not.||\n|current_version_uuid|string|UUID of the collection's current version. On new collections, it'll be equal to the @uuid@ attribute.||\n|version|number|Version number, starting at 1 on new collections. This attribute is read-only.||\n|preserve_version|boolean|When set to true on a current version, it will be persisted. When passing @true@ as part of a bigger update call, both current and newly created versions are persisted.||\n|file_count|number|The total number of files in the collection. This attribute is read-only.||\n|file_size_total|number|The sum of the file sizes in the collection. This attribute is read-only.||\n\nh3. Conditions of creating a Collection\n\nIf a new @portable_data_hash@ is specified when creating or updating a Collection, it must match the cryptographic digest of the supplied @manifest_text@.\n\nh3. Side effects of creating a Collection\n\nReferenced blocks are protected from garbage collection in Keep.\n\nData can be shared with other users via the Arvados permission model.\n\nh3(#trashing). Trashing collections\n\nCollections can be trashed by updating the record and setting the @trash_at@ field, or with the \"delete\":#delete method.  The delete method sets @trash_at@ to \"now\".\n\nThe value of @trash_at@ can be set to a time in the future as a feature to automatically expire collections.\n\nWhen @trash_at@ is set, @delete_at@ will also be set.  Normally @delete_at = trash_at + Collections.DefaultTrashLifetime@.  When the @trash_at@ time is past but @delete_at@ is in the future, the trashed collection is invisible to most API calls unless the @include_trash@ parameter is true.  Collections in the trashed state can be \"untrashed\":#untrash so long as @delete_at@ has not past.  Collections are also trashed if they are contained in a \"trashed group\":groups.html#trashing\n\nOnce @delete_at@ is past, the collection and all of its previous versions will be deleted permanently and can no longer be untrashed.\n\nh3(#replace_files). Using \"replace_files\" to create or update a collection\n\nThe @replace_files@ option can be used with the \"create\":#create and \"update\":#update APIs to efficiently and atomically copy individual files and directory trees from other collections, copy/rename/delete items within an existing collection, and add new items to a collection.\n\n@replace_files@ keys indicate target paths in the new collection, and values specify sources that should be copied to the target paths.\n* Each target path must be an absolute canonical path beginning with @/@. It must not contain @.@ or @..@ components, consecutive @/@ characters, or a trailing @/@ after the final component.\n* Each source must be one of the following:\n** an empty string (signifying that the target path is to be deleted),\n** @/@ where @@ is the portable data hash of a collection on the cluster and @@ is a file or directory in that collection,\n** @manifest_text/@ where @@ is an existing file or directory in a collection supplied in the @manifest_text@ attribute in the request, or\n** @current/@ where @@ is an existing file or directory in the collection being updated.\n\nIn an @update@ request, sources may reference the current portable data hash of the collection being updated. However, in many cases it is more appropriate to use a @current/@ source instead, to ensure the latest content is used even if the collection has been updated since the PDH was last retrieved.\n\nh4(#replace_files-delete). Delete a file\n\nDelete @foo.txt@.\n\n
\n\"replace_files\": {\n  \"/foo.txt\": \"\"\n}\n
\n\nh4(#replace_files-rename). Rename a file\n\nRename @foo.txt@ to @bar.txt@.\n\n
\n\"replace_files\": {\n  \"/foo.txt\": \"\",\n  \"/bar.txt\": \"current/foo.txt\"\n}\n
\n\nh4(#replace_files-swap). Swap files\n\nSwap contents of files @foo@ and @bar@.\n\n
\n\"replace_files\": {\n  \"/foo\": \"current/bar\",\n  \"/bar\": \"current/foo\"\n}\n
\n\nh4(#replace_files-add). Add a file\n\n
\n\"replace_files\": {\n  \"/new_directory/new_file.txt\": \"manifest_text/new_file.txt\"\n},\n\"collection\": {\n  \"manifest_text\": \". acbd18db4cc2f85cedef654fccc4a4d8+3+A82740cd577ff5745925af5780de5992cbb25d937@668efec4 0:3:new_file.txt\\n\"\n}\n
\n\nh4(#replace_files-replace). Replace all content with new content\n\nNote this is equivalent to omitting the @replace_files@ argument.\n\n
\n\"replace_files\": {\n  \"/\": \"manifest_text/\"\n},\n\"collection\": {\n  \"manifest_text\": \"./new_directory acbd18db4cc2f85cedef654fccc4a4d8+3+A82740cd577ff5745925af5780de5992cbb25d937@668efec4 0:3:new_file.txt\\n\"\n}\n
\n\nh4(#replace_files-rename-and-replace). Atomic rename and replace\n\nRename @current_file.txt@ to @old_file.txt@ and replace @current_file.txt@ with new content, all in a single atomic operation.\n\n
\n\"replace_files\": {\n  \"/current_file.txt\": \"manifest_text/new_file.txt\",\n  \"/old_file.txt\": \"current/current_file.txt\"\n},\n\"collection\": {\n  \"manifest_text\": \". acbd18db4cc2f85cedef654fccc4a4d8+3+A82740cd577ff5745925af5780de5992cbb25d937@668efec4 0:3:new_file.txt\\n\"\n}\n
\n\nh4(#replace_files-combine). Combine collections\n\nDelete all current content, then copy content from other collections into new subdirectories.\n\n
\n\"replace_files\": {\n  \"/\": \"\",\n  \"/copy of collection 1\": \"1f4b0bc7583c2a7f9102c395f4ffc5e3+45/\",\n  \"/copy of collection 2\": \"ea10d51bcf88862dbcc36eb292017dfd+45/\"\n}\n
\n\nh4(#replace_files-extract-subdirectory). Extract a subdirectory\n\nReplace all current content with a copy of a subdirectory from another collection.\n\n
\n\"replace_files\": {\n  \"/\": \"1f4b0bc7583c2a7f9102c395f4ffc5e3+45/subdir\"\n}\n
\n\nh4(#replace_files-usage-restrictions). Usage restrictions\n\nA target path with a non-empty source cannot be the ancestor of another target path in the same request. For example, the following request is invalid:\n\n
\n\"replace_files\": {\n  \"/foo\": \"fa7aeb5140e2848d39b416daeef4ffc5+45/\",\n  \"/foo/this_will_return_an_error\": \"\"\n}\n
\n\nIt is an error to supply a non-empty @manifest_text@ that is unused, i.e., the @replace_files@ argument does not contain any values beginning with @\"manifest_text/\"@. For example, the following request is invalid:\n\n
\n\"replace_files\": {\n  \"/foo\": \"current/bar\"\n},\n\"collection\": {\n  \"manifest_text\": \". acbd18db4cc2f85cedef654fccc4a4d8+3+A82740cd577ff5745925af5780de5992cbb25d937@668efec4 0:3:new_file.txt\\n\"\n}\n
\n\nCollections on other clusters in a federation cannot be used as sources. Each source must exist on the current cluster and be readable by the current user.\n\nSimilarly, if @manifest_text@ is provided, it must only reference data blocks that are stored on the current cluster. This API does not copy data from other clusters in a federation.\n\nh3(#replace_segments). Using \"replace_segments\" to repack file data\n\nThe @replace_segments@ option can be used with the \"create\":#create or \"update\":#update API to atomically apply a new file packing, typically with the goal of replacing a number of small blocks with one larger block. The repacking is specified in terms of _block segments_: a block segment is a portion of a stored block that is referenced by a file in a manifest.\n\n@replace_segments@ keys indicate existing block segments in the collection, and values specify replacement segments.\n* Each segment is specified as space-separated tokens: @\"locator offset length\"@ where @locator@ is a signed block locator and @offset@ and @length@ are decimal-encoded integers specifying a portion of the block that is referenced in the collection.\n* Each replacement block locator must be properly signed (just as if it appeared in a @manifest_text@).\n* Each existing block segment must correspond to an entire contiguous portion of a block referenced by a single file (splitting existing segments is not supported).\n* If a segment to be replaced does not match any existing block segment in the manifest, that segment _and all other @replace_segments@ entries referencing the same replacement block_ will be skipped. Other replacements will still be applied. Replacements that are skipped for this reason do not cause the request to fail. This rule ensures that when concurrent clients compute different repackings and request similar replacements such as @a,b,c,d,e → X@ and @a,b,c,d,e,f → Y@, the resulting manifest references @X@ or @Y@ but not both. Otherwise, the effect could be @a,b,c,d,e → X, f → Y@ where @Y@ is just an inefficient way to reference the same data as @f@.\n\nThe @replace_files@ and @manifest_text@ options, if present, are applied before @replace_segments@. This means @replace_segments@ can apply to blocks from @manifest_text@ and/or other collections referenced by @replace_files@.\n\nIn the following example, two files were originally saved by writing two small blocks (@c410@ and @c93e@). After concatenating the two small blocks and writing a single larger block @ca9c@, the manifest is being updated to reference the larger block.\n\n
\n\"collection\": {\n  \"manifest_text\": \". c4103f122d27677c9db144cae1394a66+2+A3d02f1f3d8a622b2061ad5afe4853dbea42039e2@674dd351 693e9af84d3dfcc71e640e005bdc5e2e+3+A6528480b63d90a24b60b2ee2409040f050cc5d0c@674dd351 0:2:file1.txt 2:3:file2.txt\\n\"\n},\n\"replace_segments\": {\n  \"c4103f122d27677c9db144cae1394a66+2+A3d02f1f3d8a622b2061ad5afe4853dbea42039e2@674dd351 0 2\": \"ca9c491ac66b2c62500882e93f3719a8+5+A312fea6de5807e9e77d844450d36533a599c40f1@674dd351 0 2\",\n  \"693e9af84d3dfcc71e640e005bdc5e2e+3+A6528480b63d90a24b60b2ee2409040f050cc5d0c@674dd351 0 3\": \"ca9c491ac66b2c62500882e93f3719a8+5+A312fea6de5807e9e77d844450d36533a599c40f1@674dd351 2 3\"\n}\n
\n\nResulting manifest:\n\n
\n. ca9c491ac66b2c62500882e93f3719a8+5+A312fea6de5807e9e77d844450d36533a599c40f1@674dd351 0:2:file1.txt 2:3:file2.txt\n
\n\nh2. Methods\n\nSee \"Common resource methods\":{{site.baseurl}}/api/methods.html for more information about @create@, @delete@, @get@, @list@, and @update@.\n\nRequired arguments are displayed in %{background:#ccffcc}green%.\n\nSupports federated @get@ only, which may be called with either a uuid or a portable data hash.  When requesting a portable data hash which is not available on the home cluster, the query is forwarded to all the clusters listed in @RemoteClusters@ and returns the first successful result.\n\nh3(#create). create\n\nCreate a new Collection.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n|collection|object||query||\n|replace_files|object|Initialize files and directories with new content and/or content from other collections|query||\n|replace_segments|object|Repack the collection by substituting data blocks|query||\n\nThe new collection's content can be initialized by providing a @manifest_text@ key in the provided @collection@ object, or by \"using the @replace_files@ option\":#replace_files.\n\nAn alternative file packing can be applied atomically \"using the @replace_segments@ option\":#replace_segments.\n\nh3(#delete). delete\n\nPut a Collection in the trash.  This sets the @trash_at@ field to @now@ and @delete_at@ field to @now@ + token TTL.  A trashed collection is invisible to most API calls unless the @include_trash@ parameter is true.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the Collection in question.|path||\n\nh3. get\n\nGets a Collection's metadata by UUID or portable data hash.  When making a request by portable data hash, attributes other than @portable_data_hash@, @manifest_text@, and @trash_at@ are not returned, even when requested explicitly using the @select@ parameter.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID or portable data hash of the Collection in question.|path||\n\nh3. list\n\nList collections.\n\nSee \"common resource list method.\":{{site.baseurl}}/api/methods.html#index\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n|include_trash|boolean (default false)|Include trashed collections.|query||\n|include_old_versions|boolean (default false)|Include past versions of the collection(s) being listed, if any.|query||\n\nNote: Because adding access tokens to manifests can be computationally expensive, the @manifest_text@ field is not included in results by default.  If you need it, pass a @select@ parameter that includes @manifest_text@.\n\nh4. Searching Collections for names of file or directories\n\nYou can search collections for specific file or directory names (whole or part) using the following filter in a @list@ query.\n\n
\nfilters: [[\"file_names\", \"ilike\", \"%sample1234.fastq%\"]]\n
\n\nNote: @file_names@ is a hidden field used for indexing.  It is not returned by any API call.  On the client, you can programmatically enumerate all the files in a collection using @arv-ls@, the Python SDK @Collection@ class, Go SDK @FileSystem@ struct, the WebDAV API, or the S3-compatible API.\n\nAs of this writing (Arvados 2.4), you can also search for directory paths, but _not_ complete file paths.\n\nIn other words, this will work (when @dir3@ is a directory):\n\n
\nfilters: [[\"file_names\", \"ilike\", \"%dir1/dir2/dir3%\"]]\n
\n\nHowever, this will _not_ return the desired results (where @sample1234.fastq@ is a file):\n\n
\nfilters: [[\"file_names\", \"ilike\", \"%dir1/dir2/dir3/sample1234.fastq%\"]]\n
\n\nAs a workaround, you can search for both the directory path and file name separately, and then filter on the client side.\n\n
\nfilters: [[\"file_names\", \"ilike\", \"%dir1/dir2/dir3%\"], [\"file_names\", \"ilike\", \"%sample1234.fastq%\"]]\n
\n\nh3(#update). update\n\nUpdate attributes of an existing Collection.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the Collection in question.|path||\n|collection|object||query||\n|replace_files|object|Add, delete, and replace files and directories with new content and/or content from other collections|query||\n|replace_segments|object|Repack the collection by substituting data blocks|query||\n\nThe collection's existing content can be replaced entirely by providing a @manifest_text@ key in the provided @collection@ object, or updated in place by \"using the @replace_files@ option\":#replace_files.\n\nAn alternative file packing can be applied atomically \"using the @replace_segments@ option\":#replace_segments.\n\nh3(#untrash). untrash\n\nRemove a Collection from the trash.  This sets the @trash_at@ and @delete_at@ fields to @null@.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the Collection to untrash.|path||\n|ensure_unique_name|boolean (default false)|Rename collection uniquely if untrashing it would fail with a unique name conflict.|query||\n\n\nh3. provenance\n\nReturns a list of objects in the database that directly or indirectly contributed to producing this collection, such as the container request that produced this collection as output.\n\nThe general algorithm is:\n\n# Visit the container request that produced this collection (via @output_uuid@ or @log_uuid@ attributes of the container request)\n# Visit the input collections to that container request (via @mounts@ and @container_image@ of the container request)\n# Iterate until there are no more objects to visit\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the Collection to get provenance.|path||\n\nh3. used_by\n\nReturns a list of objects in the database this collection directly or indirectly contributed to, such as containers that takes this collection as input.\n\nThe general algorithm is:\n\n# Visit containers that take this collection as input (via @mounts@ or @container_image@ of the container)\n# Visit collections produced by those containers (via @output@ or @log@ of the container)\n# Iterate until there are no more objects to visit\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the Collection to get usage.|path||\n"
  },
  {
    "path": "doc/api/methods/computed_permissions.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\nnavmenu: API Methods\ntitle: \"computed_permissions\"\n\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nAPI endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/computed_permissions@\n\nh2. Resource\n\nComputed permissions are entries from the internal cache of the highest permission level each user has on each permission target.\n\nEach entry has the following attributes:\n\ntable(table table-bordered table-condensed).\n|_. Attribute|_. Type|_. Description|\n|user_uuid|string|An individual user.|\n|target_uuid|string|An object (role group, project group, collection, etc.) on which the user has implicit or explicit permission.|\n|perm_level|string|@can_read@, @can_write@, or @can_manage@|\n\nThere is only one row for a given (@user_uuid@, @target_uuid@) pair.\n\nComputed permissions cannot be created or updated directly. To change permissions, use \"groups\":groups.html and \"links\":links.html APIs as described in the \"permission model\":../permission-model.html.\n\nh2. Method\n\nh3. list\n\n@GET /arvados/v1/computed_permissions@\n\nList computed permissions.\n\nThe computed permissions API accepts the arguments described in the \"common resource list method\":{{site.baseurl}}/api/methods.html#index with the following exceptions:\n* It is an error to supply a non-zero @offset@ argument.\n* The default value for @order@ is @[\"user_uuid\", \"target_uuid\"]@.\n* The default value for @count@ is @\"none\"@ and no other values are accepted.\n"
  },
  {
    "path": "doc/api/methods/container_requests.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\nnavmenu: API Methods\ntitle: \"container_requests\"\n\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nAPI endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/container_requests@\n\nObject type: @xvhdp@\n\nExample UUID: @zzzzz-xvhdp-0123456789abcde@\n\nh2. Resource\n\nA container request is a request for the Arvados cluster to perform some computational work.  See \"computing with Crunch\":{{site.baseurl}}/api/execution.html for details.\n\nEach ContainerRequest offers the following attributes, in addition to the \"Common resource fields\":{{site.baseurl}}/api/resources.html:\n\nAll attributes are optional, unless otherwise marked as required.\n\ntable(table table-bordered table-condensed).\n|_. Attribute|_. Type|_. Description|_. Notes|\n|name|string|The name of the container_request.||\n|description|string|The description of the container_request.  Allows \"HTML formatting.\":{{site.baseurl}}/api/resources.html#descriptions ||\n|properties|hash|User-defined metadata that does not affect how the container is run.  May be used in queries using \"subproperty filters\":{{site.baseurl}}/api/methods.html#subpropertyfilters||\n|state|string|The allowed states are \"Uncommitted\", \"Committed\", and \"Final\".|Once a request is Committed, the only attributes that can be modified are priority, container_uuid, and container_count_max. A request in the \"Final\" state cannot have any of its functional parts modified (i.e., only name, description, and properties fields can be modified).|\n|requesting_container_uuid|string|The uuid of the parent container that created this container_request, if any. Represents a process tree.|The priority of this container_request is inherited from the parent container, if the parent container is cancelled, this container_request will be cancelled as well.|\n|container_uuid|string|The uuid of the container that satisfies this container_request. The system may return a preexisting Container that matches the container request criteria. See \"Container reuse\":#container_reuse for more details.|Container reuse is the default behavior, but may be disabled with @use_existing: false@ to always create a new container.|\n|container_count_max|integer|Maximum number of containers to start, i.e., the maximum number of \"attempts\" to be made.||\n|mounts|hash|Objects to attach to the container's filesystem and stdin/stdout.|See \"Mount types\":#mount_types for more details.|\n|secret_mounts|hash|Objects to attach to the container's filesystem.  Only \"json\" or \"text\" mount types allowed.|Not returned in API responses. Reset to empty when state is \"Complete\" or \"Cancelled\".|\n|runtime_constraints|hash|Restrict the container's access to compute resources and the outside world.|Required when in \"Committed\" state. e.g.,
{\n  \"ram\":12000000000,\n  \"vcpus\":2,\n  \"API\":true\n}
See \"Runtime constraints\":#runtime_constraints for more details.|\n|scheduling_parameters|hash|Parameters to be passed to the container scheduler when running this container.|e.g.,
{\n\"partitions\":[\"fastcpu\",\"vfastcpu\"]\n}
See \"Scheduling parameters\":#scheduling_parameters for more details.|\n|container_image|string|Portable data hash of a collection containing the docker image to run the container.|Required.|\n|environment|hash|Environment variables and values that should be set in the container environment (@docker run --env@). This augments and (when conflicts exist) overrides environment variables given in the image's Dockerfile.||\n|cwd|string|Initial working directory, given as an absolute path (in the container) or a path relative to the WORKDIR given in the image's Dockerfile.|Optional. If omitted or blank, @\".\"@ is implied.|\n|command|array of strings|Command to execute in the container.|Required. e.g., @[\"echo\",\"hello\"]@|\n|output_path|string|Path to a directory or file inside the container that should be preserved as container's output when it finishes. This path must be one of the mount targets. For best performance, point output_path to a writable collection mount.  See \"Pre-populate output using Mount points\":#pre-populate-output for details regarding optional output pre-population using mount points and \"Symlinks in output\":#symlinks-in-output for additional details.|Required.|\n|output_glob|array of strings|Glob patterns determining which files (of those present in the output directory when the container finishes) will be included in the output collection. If multiple patterns are given, files that match any pattern are included. If null or empty, all files will be included.|e.g., @[\"**/*.vcf\", \"**/*.vcf.gz\"]@\nSee \"Glob patterns\":#glob_patterns for more details.|\n|output_name|string|Desired name for the output collection. If null or empty, a name will be assigned automatically.||\n|output_ttl|integer|Desired lifetime for the output collection, in seconds. If zero, the output collection will not be deleted automatically.||\n|priority|integer|Range 0-1000.  Indicate scheduling order preference.|Clients are expected to submit container requests with zero priority in order to preview the container that will be used to satisfy it. Priority can be null if and only if state!=\"Committed\".  See \"priority below for more details.\":#priority |\n|expires_at|datetime|After this time, priority is considered to be zero.|Not yet implemented.|\n|use_existing|boolean|If possible, use an existing (non-failed) container to satisfy the request instead of creating a new one.|Default is true.|\n|log_uuid|string|Log collection containing log messages provided by the scheduler and crunch processes.|Null if the container has not yet started running.\nTo retrieve logs in real time while the container is running, use the log API (see below).|\n|output_uuid|string|Output collection created when the container finished successfully.|Null if the container has failed or not yet completed.|\n|filters|string|Additional constraints for satisfying the container_request, given in the same form as the filters parameter accepted by the container_requests.list API.|This attribute is not implemented yet. The value should always be null.|\n|runtime_token|string|A v2 token to be passed into the container itself, used to access Keep-backed mounts, etc.  |Not returned in API responses.  Reset to null when state is \"Complete\" or \"Cancelled\".|\n|runtime_user_uuid|string|The user permission that will be granted to this container.||\n|runtime_auth_scopes|array of string|The scopes associated with the auth token used to run this container.||\n|output_storage_classes|array of strings|The storage classes that will be used for the log and output collections of this container request|If omitted, the cluster's configured default storage classes are used.|\n|output_properties|hash|User metadata properties to set on the output collection.  The output collection will also have default properties \"type\" (\"intermediate\" or \"output\") and \"container_request\" (the uuid of container request that produced the collection).|\n|cumulative_cost|number|Estimated cost of the cloud VMs used to satisfy the request, including retried attempts and completed subrequests, but not including reused containers.|0 if container was reused or VM price information was not available.|\n|service|boolean|Indicates that this container is a long-lived service rather than a once-through batch job.  Incompatible with @use_existing@||\n|published_ports|hash|Web service ports that are published by this container.  See \"published ports\":#published_ports below.||\n\nh2(#lifecycle). Container request lifecycle\n\nA container request may be created in the Committed state, or created in the Uncommitted state and then moved into the Committed state.\n\nOnce a request is in the Committed state, Arvados locates a suitable existing container or schedules a new one. When the assigned container finishes, the request state changes to Final.\n\nA client may cancel a committed request early (before the assigned container finishes) by setting the request priority to zero.\n\n!{max-width:60em;}{{site.baseurl}}/api/methods/container_request_lifecycle.svg!\n{% comment %}\n# svg generated using `graphviz -Tsvg -O`\ndigraph {\n    graph [nojustify=true] [labeljust=l]\n\n    invisiblestart [label = \"\"] [color=white] [group=lifecycle];\n    node [color=black] [fillcolor=white] [style=filled] [shape=box] [nojustify=true];\n    uncommitted [label = \"container request:\\l   state=Uncommitted\\l\"] [fillcolor=lightgrey] [group=lifecycle];\n    {\n        rank=same;\n        committed [label = \"container request:\\l   state=Committed\\l   priority>0\\l\"] [group=lifecycle];\n        reused [label = \"container request:\\l   state=Final\\lcontainer:\\l   state=Complete\\l(reused existing container)\\l\"] [fillcolor=lightblue] [group=endstate];\n    }\n    invisiblestart -> uncommitted [label = \"   user creates container request\\l\"] [color=navy] [fontcolor=navy];\n    uncommitted -> committed [label = \"   user updates to\\l      state=Committed, priority>0\\l\"] [color=navy] [fontcolor=navy];\n    queued [label = \"container request:\\l   state=Committed\\l   priority>0\\lcontainer:\\l   state=Queued\\l\"] [group=lifecycle];\n    committed -> queued [label = \"   Arvados creates a new container\\l\"];\n    {\n        rank=same;\n        locked [label = \"container request:\\l   state=Committed\\l   priority>0\\lcontainer:\\l   state=Locked\\l\"] [group=lifecycle];\n        latecancelled [label = \"container request:\\l   state=Final\\lcontainer:\\l   state=Cancelled\\l\"] [fillcolor=lightblue] [group=endstate];\n    }\n    queued -> locked [label = \"   Arvados is ready to dispatch the container\\l\"];\n    {\n        rank=same;\n        running [label = \"container request:\\l   state=Committed\\l   priority>0\\lcontainer:\\l   state=Running\\l\"] [group=lifecycle];\n        containerfailed [label = \"container request:\\l   state=Final\\lcontainer:\\l   state=Complete\\l   exit_code≠0\\l\"] [fillcolor=lightblue] [group=endstate];\n    }\n    locked -> running [label = \"   Arvados starts the container process\\l\"];\n    containerfinished [label = \"container request:\\l   state=Final\\lcontainer:\\l   state=Complete\\l   exit_code=0\\l\"] [fillcolor=lightblue] [group=lifecycle];\n\n    committed -> reused [label = \"Arvados selects an existing container\"] [constraint=false] [labeldistance=0.5];\n    queued -> latecancelled [label = \"user updates to priority=0\"] [color=navy] [fontcolor=navy];\n    locked -> latecancelled [label = \"user updates to priority=0\"] [color=navy] [fontcolor=navy] [constraint=false];\n    running -> latecancelled [label = \"user updates to priority=0\"] [color=navy] [fontcolor=navy] [constraint=false];\n    running -> containerfailed [label = \"container process fails\"];\n    running -> containerfinished [label = \"   container process succeeds\\l\"];\n\n    # layout hacks\n    reused -> latecancelled [style=invis];\n    latecancelled -> containerfailed [style=invis];\n}\n{% endcomment %}\n\nh2(#priority). Priority\n\nThe @priority@ field has a range of 0-1000.\n\nPriority 0 means no container should run on behalf of this request, and containers already running will be terminated (setting container priority to 0 is the cancel operation.)\n\nPriority 1 is the lowest priority.\n\nPriority 1000 is the highest priority.\n\nThe actual order that containers execute is determined by the underlying scheduling software (e.g. Slurm) and may be based on a combination of container priority, submission time, available resources, and other factors.\n\nIn the current implementation, the magnitude of difference in priority between two containers affects the weight of priority vs age in determining scheduling order.  If two containers have only a small difference in priority (for example, 500 and 501) and the lower priority container has a longer queue time, the lower priority container may be scheduled before the higher priority container.  Use a greater magnitude difference (for example, 500 and 600) to give higher weight to priority over queue time.\n\nh2(#mount_types). {% include 'mount_types' %}\n\nh2(#runtime_constraints). {% include 'container_runtime_constraints' %}\n\nh2(#scheduling_parameters). {% include 'container_scheduling_parameters' %}\n\nh2(#glob_patterns). {% include 'container_glob_patterns' %}\n\nh2(#published_ports). {% include 'container_published_ports' %}\n\nh2(#container_reuse). Container reuse\n\nWhen a container request is \"Committed\", the system will try to find and reuse an existing Container with the same command, cwd, environment, output_path, container_image, mounts, secret_mounts, runtime_constraints, runtime_user_uuid, and runtime_auth_scopes being requested.\n\n* The serialized fields environment, mounts, and runtime_constraints are normalized when searching.\n* The system will also search for containers with minor variations in the keep_cache_disk and keep_cache_ram runtime_constraints that should not affect the result. This searches for other common values for those constraints, so a container that used a non-default value for these constraints may not be reused by later container requests that use a different value.\n\nIn order of preference, the system will use:\n\n* The first matching container to have finished successfully (i.e., reached state \"Complete\" with an exit_code of 0) whose log and output collections are still available.\n* The oldest matching \"Running\" container with the highest progress, i.e., the container that is most likely to finish first.\n* The oldest matching \"Locked\" container with the highest priority, i.e., the container that is most likely to start first.\n* The oldest matching \"Queued\" container with the highest priority, i.e,, the container that is most likely to start first.\n* A new container.\n\nh2(#cancel_container). Canceling a container request\n\nA container request may be canceled by setting its priority to 0, using an update call.\n\nWhen a container request is canceled, it will still reflect the state of the Container it is associated with via the container_uuid attribute. If that Container is being reused by any other container_requests that are still active, i.e., not yet canceled, that Container may continue to run or be scheduled to run by the system in future. However, if no other container_requests are using that Container, then the Container will get canceled as well.\n\nh2. Methods\n\nSee \"Common resource methods\":{{site.baseurl}}/api/methods.html for more information about @create@, @delete@, @get@, @list@, and @update@.\n\nRequired arguments are displayed in %{background:#ccffcc}green%.\n\nSupports federated @create@, @delete@, @get@, @list@, and @update@.\n\nh2(#create). create\n\nCreate a new container request.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|container_request|object|Container request resource.|request body||\n|cluster_id|string|The federated cluster to submit the container request.|query||\n\nThe request body must include the required attributes @command@, @container_image@, @mounts@, and @output_path@. It can also include other attributes such as @environment@, @published_ports@, @runtime_constraints@, and @scheduling_parameters@.\n\nh3. delete\n\nDelete an existing container request.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the container request in question.|path||\n\nh3. get\n\nGet a container request's metadata by UUID.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the container request in question.|path||\n\nh3. list\n\nList container requests.\n\nSee \"common resource list method.\":{{site.baseurl}}/api/methods.html#index\n\nThe @filters@ argument can also filter on attributes of the container referenced by @container_uuid@. For example, @[[\"container.state\", \"=\", \"Running\"]]@ will match any container request whose container is running now.\n\nh3. update\n\nUpdate attributes of an existing container request.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the container request in question.|path||\n|container_request|object||query||\n\n{% include 'notebox_begin' %}\nSetting the priority of a committed container_request to 0 may cancel a running container assigned for it.\nSee \"Canceling a container request\":{{site.baseurl}}/api/methods/container_requests.html#cancel_container for further details.\n{% include 'notebox_end' %}\n\nh3(#container_status). container_status\n\nGet container status.\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |\n{background:#ccffcc}.|uuid|string|The UUID of the container request in question.|path|\n\nExample request: @GET /arvados/v1/container_requests/zzzzz-xvdhp-0123456789abcde/container_status@\n\nResponse attributes:\n\ntable(table table-bordered table-condensed).\n|_. Attribute|_. Type|_. Description|_. Examples|\n|uuid|string|The UUID of the container assigned to this request.||\n|state|string|The state of the container assigned to this request (see \"container resource attributes\":containers.html).||\n|scheduling_status|string|A brief explanation of the container's status in the dispatch queue, or an empty string if scheduling is not applicable, e.g., the container is running or finished.|@waiting for cloud resources: queue position 3@\n@creating new instance@\n@preparing runtime environment@|\n\nh3(#log). log\n\nGet container log data using WebDAV methods.\n\nThis API retrieves data from the container request's log collection. It can be used at any time in the container request lifecycle.\n* Before a container has been assigned (the request is @Uncommitted@) it returns an empty directory.\n* While the container is @Queued@ or @Locked@, it returns an empty directory.\n* While the container is @Running@, @.../log/{container_uuid}/@ returns real-time logging data.\n* While the container is @Complete@ or @Cancelled@, @.../log/{container_uuid}/@ returns the final log collection.\n\nIf a request results in multiple containers being run (see @container_count_max@ above), the logs from prior attempts remain available at @.../log/{old_container_uuid}/@.\n\nCurrently, this API has a limitation that a directory listing at the top level @/arvados/v1/container_requests/{uuid}/log/@ does not reveal the per-container subdirectories. Instead, clients should look up the container request record and use the @container_uuid@ attribute to request files and directory listings under the per-container directory, as in the examples below.\n\nThis API supports the @Range@ request header, so it can be used to poll for and retrieve logs incrementally while the container is running.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|method|string|Read-only WebDAV method|HTTP method|@GET@, @OPTIONS@, @PROPFIND@|\n{background:#ccffcc}.|uuid|string|The UUID of the container request.|path|zzzzz-xvdhp-0123456789abcde|\n{background:#ccffcc}.|path|string|Path to a file in the log collection.|path|@/zzzzz-dz642-0123456789abcde/stderr.txt@|\n\nExamples:\n* @GET /arvados/v1/container_requests/zzzzz-xvdhp-0123456789abcde/log/zzzzz-dz642-0123456789abcde/stderr.txt@\n* @PROPFIND /arvados/v1/container_requests/zzzzz-xvdhp-0123456789abcde/log/zzzzz-dz642-0123456789abcde/@\n"
  },
  {
    "path": "doc/api/methods/containers.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\nnavmenu: API Methods\ntitle: \"containers\"\n\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nAPI endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/containers@\n\nObject type: @dz642@\n\nExample UUID: @zzzzz-dz642-0123456789abcde@\n\nh2. Resource\n\nA container is work order to be dispatched to an Arvados cluster to perform some computational work.  A container is created in response to a container request.  See \"computing with Crunch\":{{site.baseurl}}/api/execution.html for details.\n\nEach Container offers the following attributes, in addition to the \"Common resource fields\":{{site.baseurl}}/api/resources.html:\n\ntable(table table-bordered table-condensed).\n|_. Attribute|_. Type|_. Description|_. Notes|\n|state|string|The allowed states are \"Queued\", \"Locked\", \"Running\", \"Cancelled\" and \"Complete\".|See \"Container states\":#container_states for more details.|\n|started_at|datetime|When this container started running.|Null if container has not yet started.|\n|finished_at|datetime|When this container finished.|Null if container has not yet finished.|\n|log|string|Portable data hash of a collection containing the log messages produced when executing the container.|Null if container has not yet started. The Crunch system will periodically update this field for a running container.|\n|environment|hash|Environment variables and values that should be set in the container environment (@docker run --env@). This augments and (when conflicts exist) overrides environment variables given in the image's Dockerfile.|Must be equal to a ContainerRequest's environment in order to satisfy the ContainerRequest.|\n|cwd|string|Initial working directory.|Must be equal to a ContainerRequest's cwd in order to satisfy the ContainerRequest.|\n|command|array of strings|Command to execute.| Must be equal to a ContainerRequest's command in order to satisfy the ContainerRequest.|\n|output_path|string|Path to a directory or file inside the container that should be preserved as this container's output when it finishes.|Must be equal to a ContainerRequest's output_path in order to satisfy the ContainerRequest.|\n|output_glob|array of strings|Glob patterns determining which files will be included in the output collection. See corresponding attribute in the \"container_requests resource\":container_requests.html.|Must be equal to a ContainerRequest's output_glob in order to satisfy the ContainerRequest. See \"Glob patterns\":#glob_patterns for more details.|\n|mounts|hash|Must contain the same keys as the ContainerRequest being satisfied. Each value must be within the range of values described in the ContainerRequest at the time the Container is assigned to the ContainerRequest.|See \"Mount types\":#mount_types for more details.|\n|secret_mounts|hash|Must contain the same keys as the ContainerRequest being satisfied. Each value must be within the range of values described in the ContainerRequest at the time the Container is assigned to the ContainerRequest.|Not returned in API responses. Reset to empty when state is \"Complete\" or \"Cancelled\".|\n|runtime_constraints|hash|Compute resources, and access to the outside world, that are / were available to the container.\nGenerally this will contain additional keys that are not present in any corresponding ContainerRequests: for example, even if no ContainerRequests specified constraints on the number of CPU cores, the number of cores actually used will be recorded here.|e.g.,\n
{\n  \"ram\":12000000000,\n  \"vcpus\":2,\n  \"API\":true\n}
See \"Runtime constraints\":#runtime_constraints for more details.|\n|runtime_status|hash|Information related to the container's run, including its steps. Some keys have specific meaning and are described later in this page.|e.g.,\n
{\n  \"error\": \"This container won't be successful because at least one step has already failed.\"\n}
See \"Runtime status\":#runtime_status for more details.|\n|scheduling_parameters|hash|Parameters to be passed to the container scheduler when running this container.|e.g.,
{\n\"partitions\":[\"fastcpu\",\"vfastcpu\"]\n}
See \"Scheduling parameters\":#scheduling_parameters for more details.|\n|output|string|Portable data hash of the output collection.|Null if the container is not yet finished.|\n|container_image|string|Portable data hash of a collection containing the docker image used to run the container.||\n|progress|number|A number between 0.0 and 1.0 describing the fraction of work done.||\n|priority|integer|Range 0-1000.  Indicate scheduling order preference.|Currently assigned by the system as the max() of the priorities of all associated ContainerRequests.  See \"container request priority\":container_requests.html#priority.|\n|exit_code|integer|Process exit code.|Null if container process has not exited yet.|\n|auth_uuid|string|UUID of a token to be passed into the container itself, used to access Keep-backed mounts, etc.  Automatically assigned.|Null if state∉{\"Locked\",\"Running\"} or if @runtime_token@ was provided.|\n|locked_by_uuid|string|UUID of a token, indicating which dispatch process changed state to Locked. If null, any token can be used to lock. If not null, only the indicated token can modify this container.|Null if state∉{\"Locked\",\"Running\"}|\n|runtime_token|string|A v2 token to be passed into the container itself, used to access Keep-backed mounts, etc.|Not returned in API responses.  Reset to null when state is \"Complete\" or \"Cancelled\".|\n|gateway_address|string|Address (host:port) of gateway server.|Internal use only.|\n|interactive_session_started|boolean|Indicates whether @arvados-client shell@ has been used to run commands in the container, which may have altered the container's behavior and output.||\n|output_storage_classes|array of strings|The storage classes that will be used for the log and output collections of this container||\n|output_properties|hash|User metadata properties to set on the output collection.|\n|cost|number|Estimated cost of the cloud VM used to run the container.|0 if not available.|\n|subrequests_cost|number|Total estimated cumulative cost of container requests submitted by this container.|0 if not available.|\n|service|boolean|Indicates that this container is a long-lived service rather than a once-through batch job.  Incompatible with @use_existing@||\n|published_ports|hash|Web service ports that are published by this container.  See \"published ports\":#published_ports below.||\n\nh2(#container_states). Container states\n\ntable(table table-bordered table-condensed).\n|_. State value|_. Description|_. Allowed next|\n|Queued|Waiting for a dispatcher to lock it and try to run the container.|Locked, Cancelled|\n|Locked|A dispatcher has \"taken\" the container and is allocating resources for it. The container has not started yet.|Queued, Running, Cancelled|\n|Running|Resources have been allocated and the contained process has been started (or is about to start). Crunch-run _must_ set state to Running _before_ there is any possibility that user code will run in the container.|Complete, Cancelled|\n|Complete|Container was running, and the contained process/command has exited.|Cancelled|\n|Cancelled|The container did not run long enough to produce an exit code. This includes cases where the container didn't even start, cases where the container was interrupted/killed before it exited by itself (e.g., priority changed to 0), and cases where some problem prevented the system from capturing the contained process's exit status (exit code and output).|-|\n\nSee \"Controlling container reuse\":{{site.baseurl}}/admin/controlling-container-reuse.html for details about changing state from @Complete@ to @Cancelled@\n\nh2(#mount_types). {% include 'mount_types' %}\n\nh2(#runtime_constraints). {% include 'container_runtime_constraints' %}\n\nh2(#runtime_status). Runtime status\n\nRuntime status provides container's relevant information about its progress even while it's still in Running state. This is used to avoid reusing containers that have not yet failed but will definitely do, and also for easier workflow debugging.\n\nThe following keys have well known meanings:\n\ntable(table table-bordered table-condensed).\n|_. Key|_. Type|_. Description|_. Notes|\n|error|string|The existance of this key indicates the container will definitely fail, or has already failed.|Optional.|\n|warning|string|Indicates something unusual happened or is currently happening, but isn't considered fatal.|Optional.|\n|activity|string|A message for the end user about what state the container is currently in.|Optional.|\n|errorDetail|string|Additional structured error details.|Optional.|\n|warningDetail|string|Additional structured warning details.|Optional.|\n|preemptionNotice|string|Details about any cloud provider scheduled interruption to the instance running this container.|Existence of this key indicates the container likely was (or will soon be) @Cancelled@ due to an instance interruption.|\n\nh2(#scheduling_parameters). {% include 'container_scheduling_parameters' %}\n\nh2(#glob_patterns). {% include 'container_glob_patterns' %}\n\nh2(#published_ports). {% include 'container_published_ports' %}\n\nh2. Methods\n\nSee \"Common resource methods\":{{site.baseurl}}/api/methods.html for more information about @create@, @delete@, @get@, @list@, and @update@.\n\nRequired arguments are displayed in %{background:#ccffcc}green%.\n\nSupports federated @get@ and @list@.\n\nh3(#create). create\n\nCreate a new Container.\n\nThis API requires admin privileges. In normal operation, it should not be used at all.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|container|object|Container resource|request body||\n\nh3. delete\n\nDelete a Container.\n\nThis API requires admin privileges. In normal operation, it should not be used at all. API clients like Workbench might not work correctly when a container request references a container that has been deleted.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the Container in question.|path||\n\nh3. get\n\nGet a Container's metadata by UUID.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the Container in question.|path||\n\nh3. list\n\nList containers.\n\nSee \"common resource list method.\":{{site.baseurl}}/api/methods.html#index\n\nh3. update\n\nUpdate attributes of an existing Container.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the Container in question.|path||\n|container|object||query||\n\nh3. auth\n\nGet the api_client_authorization record indicated by this container's auth_uuid, which belongs to the container's locked_by_uuid.\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string||path||\n"
  },
  {
    "path": "doc/api/methods/credentials.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\nnavmenu: API Methods\ntitle: \"credentials\"\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nAPI endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/credentials@\n\nObject type: @oss07@\n\nExample UUID: @zzzzz-oss07-0123456789abcde@\n\nh2. Resource\n\nStores a credential, such as a username/password or API token, for use by running containers to access an external resource on the user's behalf.\n\nEach Credential offers the following attributes, in addition to the \"Common resource fields\":{{site.baseurl}}/api/resources.html:\n\ntable(table table-bordered table-condensed).\n|_. Attribute|_. Type|_. Description|\n|name|string|Name for the credential, unique by owner.|\n|description|string|(optional) Free text description of this credential.|\n|credential_class|string|The type of credential stored in this record. See below for more information.|\n|scopes|array of string|(optional) One or more specific resources this credential applies to.|\n|external_id|string|The non-secret part of the credential.|\n|secret|string|The secret part of the credential that should kept hidden where possible.|\n|expires_at|timestamp|Date at which the @secret@ field is not longer valid and can no longer be accessed (and may be scrubbed from the database).  If @expires_at@ has past, any attempts to access the @secret@ endpoint (see below) also return an error.|\n\nThe @secret@ field can be set when the record is created or updated by users with at @can_write@ permission, however the value of @secret@ is not returned in the regular @get@ or @list@ API calls, and cannot be used in queries.\n\nCredentials can be read using an Arvados token issued to a container running on behalf of a user who has @can_read@ permission to the credential, using the @secret@ API call (see below).  Calling the @secret@ API with a regular Arvados token (i.e. not associated with a running container) will return a permission denied error.\n\nThis design is intended to minimize accidental exposure of the secret material, but does not inherently protect it from users who have been given @can_read@ access, since it is necessary for code running on those user's behalf to access the secret in order to make use of it.\n\nAs of Arvados 3.2, all credentials are owned by the system user and the @name@ field must be unique on a given Arvados instance.  Credentials are shared using normal permission links.\n\nh2. Credential classes\n\nThe @credential_class@ field is used to identify what kind of credential is stored and how to interpret the other fields of the record. Some credential classes, like @aws_access_key@, are reserved and must be prefixed with @arv:@. Being reserved means that each scope in the associated @scopes@ field is checked to ensure that it is valid for that credential class.\n\nh3. aws_access_key\n\ntable(table table-bordered table-condensed).\n|_. Attribute|_. Description|\n|credential_class|String \"arv:aws_access_key\"|\n|scopes|A list of S3 buckets (in the form \"s3://bucketname\") to which these credentials grant access. The special value \"s3://*\" means this credential can be used for any bucket.|\n|external_id|The value of \"aws_access_key_id\" from @~/.aws/credentials@|\n|secret|The value of \"aws_secret_access_key\" from @~/.aws/credentials@|\n\nh2. Methods\n\nSee \"Common resource methods\":{{site.baseurl}}/api/methods.html for more information about @create@, @delete@, @get@, @list@, and @update@.\n\nRequired arguments are displayed in %{background:#ccffcc}green%.\n\nh3. create\n\nCreate a new Credential.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|credential|object|Credential resource|request body||\n\nh3. delete\n\nDelete an existing Credential.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the Credential in question.|path||\n\nh3. get\n\nGet a credential by UUID.  The @secret@ field is not returned in @get@ API calls.  To get the value of @secret@, use the @secret@ API call.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the Credential in question.|path||\n\nh3. list\n\nList credentials.  The @secret@ field is not returned in @list@ API calls, and cannot be used in queries.  To get the value of @secret@, use the @secret@ API call.\n\nSee \"common resource list method.\":{{site.baseurl}}/api/methods.html#index\n\nh3. update\n\nUpdate attributes of an existing credential.  May be used to update the value of @secret@.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the Credential in question.|path||\n|credential|object||query||\n\nh3. secret\n\nGet the value of @secret@.  Returns a JSON object in the form @{\"external_id\": \"...\", \"secret\": \"...\"}@.\n\nOnly permitted when called with a Arvados token issued to a container running on behalf of a user who has @can_read@ permission to the credential.  Calling this API with a regular Arvados token (i.e. not associated with a running container) will return a permission denied error.\n\nIf @expires_at@ has passed, this endpoint will return an error.\n\nCalls to the @secret@ API endpoint are logged as @event_type: secret_access@ in the audit log table.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the Credential in question.|path||\n"
  },
  {
    "path": "doc/api/methods/groups.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\nnavmenu: API Methods\ntitle: \"groups\"\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nAPI endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/groups@\n\nObject type: @j7d0g@\n\nExample UUID: @zzzzz-j7d0g-0123456789abcde@\n\nh2. Resource\n\nGroups provides a way to apply the same permissions to a set of Arvados objects.  See \"permission model\":{{site.baseurl}}/api/permission-model.html for details.\n\nEach Group has, in addition to the \"Common resource fields\":{{site.baseurl}}/api/resources.html:\n\ntable(table table-bordered table-condensed).\n|_. Attribute|_. Type|_. Description|_. Example|\n|name|string|||\n|group_class|string|Type of group. @project@ and @filter@ indicate that the group should be displayed by Workbench and arv-mount as a project for organizing and naming objects. @role@ is used as part of the \"permission system\":{{site.baseurl}}/api/permission-model.html. |@\"filter\"@\n@\"project\"@\n@\"role\"@|\n|description|text|Free text description of the group.  Allows \"HTML formatting.\":{{site.baseurl}}/api/resources.html#descriptions ||\n|properties|hash|User-defined metadata, may be used in queries using \"subproperty filters\":{{site.baseurl}}/api/methods.html#subpropertyfilters ||\n|can_write|boolean|True if the current user has write permission on this group.||\n|can_manage|boolean|True if the current user has manage permission on this group.||\n|trash_at|datetime|If @trash_at@ is non-null and in the past, this group and all objects directly or indirectly owned by the group will be hidden from API calls.  May be untrashed as long as @delete_at@ is in the future.||\n|delete_at|datetime|If @delete_at@ is non-null and in the past, the group and all objects directly or indirectly owned by the group may be permanently deleted.||\n|is_trashed|datetime|True if @trash_at@ is in the past, false if not.||\n|frozen_by_uuid|string|For a frozen project, indicates the user who froze the project; null in all other cases. When a project is frozen, no further changes can be made to the project or its contents, even by admins. Attempting to add new items or modify, rename, move, trash, or delete the project or its contents, including any subprojects, will return an error.||\n\nh2. Group types and states\n\nh3(#project). Project groups\n\nGroups with @group_class: project@ are used to organize objects and subprojects through ownership.  When \"trashed or deleted\":#trashing, all items owned by the project (including subprojects, collections, or container requests) as well as permissions (permission links) granted to the project are also trashed or deleted.\n\nh3(#role). Role groups\n\nGroups with @group_class: role@ are used to grant permissions to users (or other groups) through permission links.  Role groups can confer \"can_manage\" permission but cannot directly own objects.  When \"trashed and deleted\":#trashing group membership and permission grants (expressed as permission links) are deleted as well.\n\nh3(#filter). Filter groups\n\nGroups with @group_class: filter@ groups are virtual groups; they can not own other objects, but instead their contents (as returned by the \"contents\":#contents API method) are defined by a query. Filter groups have a special @properties@ field named @filters@, which must be an array of filter conditions. See \"list method filters\":{{site.baseurl}}/api/methods.html#filters for details on the syntax of valid filters, but keep in mind that the attributes must include the object type (@collections@, @container_requests@, @groups@, @workflows@), separated with a dot from the field to be filtered on.\n\nFilters are applied with an implied *and* between them, but each filter only applies to the object type specified. The results are subject to the usual access controls - they are a subset of all objects the user can see. Here is an example:\n\n
\n \"properties\":{\n  \"filters\":[\n   [\n    \"groups.name\",\n    \"like\",\n    \"Public%\"\n   ]\n  ]\n },\n
\n\nThis @filter@ group will return all groups (projects) that have a name starting with the word @Public@ and are visible to the user issuing the query. Because groups can contain many types of object, it will also return all objects of other types that the user can see.\n\nThe 'is_a' filter operator is of particular interest to limit the @filter@ group 'content' to the desired object(s). When the 'is_a' operator is used, the attribute must be 'uuid'. The operand may be a string or an array which means objects of either type will match the filter. This example will return all groups (projects) that have a name starting with the word @Public@, as well as all collections that are in the project with uuid @zzzzz-j7d0g-0123456789abcde@.\n\n
\n \"properties\":{\n  \"filters\":[\n   [\n    \"groups.name\",\n    \"like\",\n    \"Public%\"\n   ],\n   [\n    \"collections.owner_uuid\",\n    \"=\",\n    \"zzzzz-j7d0g-0123456789abcde\"\n   ],\n   [\n    \"uuid\",\n    \"is_a\",\n    [\n     \"arvados#group\",\n     \"arvados#collection\"\n    ]\n   ]\n  ]\n },\n 
\n\n\"Trashed or deleting\":#trashing a filter group causes the group itself to be hidden or deleted, but has no effect on the items returned in \"contents\", i.e. the database objects in \"contents\" are not hidden or deleted and may be accessed by other means.\n\nh3(#trashing). Trashing groups\n\nGroups can be trashed by updating the record and setting the @trash_at@ field, or with the \"delete\":#delete method.  The delete method sets @trash_at@ to \"now\".\n\nThe value of @trash_at@ can be set to a time in the future as a feature to automatically expire groups.\n\nWhen @trash_at@ is set, @delete_at@ will also be set.  Normally @delete_at = trash_at + Collections.DefaultTrashLifetime@ for projects and filter groups, and @delete_at = trash_at@ for role groups.  When the @trash_at@ time is past but @delete_at@ is in the future, the trashed group is invisible to most API calls unless the @include_trash@ parameter is true.  All objects directly or indirectly owned by the group (including subprojects, collections, or container requests) are considered trashed as well.  Groups in the trashed state can be \"untrashed\":#untrash so long as @delete_at@ has not past.\n\nOnce @delete_at@ is past, the group will be deleted permanently and can no longer be untrashed.  Different group types have different behavior when deleted, described above.\n\nNote: like other groups, \"role\" groups may have @trash_at@ set to date in the future, however roles groups are required to have @delete_at = trash_at@, so the trash time and delete time expire at the same time.  This means once @trash_at@ expires the role group is deleted immediately.  Role groups with @trash_at@ set can only be \"untrashed\":#untrash before they expire.\n\nh3(#frozen). Frozen projects\n\nA user with @manage@ permission can set the @frozen_by_uuid@ attribute of a @project@ group to their own user UUID. Once this is done, no further changes can be made to the project or its contents, including subprojects.\n\nThe @frozen_by_uuid@ attribute can be cleared by an admin user. It can also be cleared by a user with @manage@ permission, unless the @API.UnfreezeProjectRequiresAdmin@ configuration setting is active.\n\nThe optional @API.FreezeProjectRequiresDescription@ and @API.FreezeProjectRequiresProperties@ configuration settings can be used to prevent users from freezing projects that have empty @description@ and/or empty @properties@ entries.\n\nh2. Methods\n\nSee \"Common resource methods\":{{site.baseurl}}/api/methods.html for more information about @create@, @delete@, @get@, @list@, and @update@.\n\nRequired arguments are displayed in %{background:#ccffcc}green%.\n\nh3(#contents). contents\n\nRetrieve a list of items owned by the group or user.  Use \"recursive\" to list objects within subprojects as well.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the group or user to enumerate. If this is a user UUID, this method returns the contents of that user's home project.|path||\n|limit|integer (default 100)|Maximum number of items to return.|query||\n|order|array|Attributes to use as sort keys to determine the order resources are returned, each optionally followed by @asc@ or @desc@ to indicate ascending or descending order. Sort within a resource type by prefixing the attribute with the resource name and a period.|query|@[\"collections.modified_at desc\"]@|\n|filters|array|Conditions for filtering items.|query|@[[\"uuid\", \"is_a\", \"arvados#job\"]]@|\n|recursive|boolean (default false)|Include items owned by subprojects.|query|@true@|\n|exclude_home_project|boolean (default false)|Only return items which are visible to the user but not accessible within the user's home project.  Use this to get a list of items that are shared with the user.  Uses the logic described under the \"shared\" endpoint.|query|@true@|\n|include|array|Look up objects referenced by the indicated fields and include them in the response. Only \"owner_uuid\", \"container_uuid\" and \"collection_uuid\" are supported. If \"owner_uuid\" is given, the parent project or user will be returned. If \"container_uuid\" is given and container requests are returned in the response, the corresponding container records will also be returned.  If \"collection_uuid\" is given and workflows are returned in the response, the collection records will also be returned. These referenced objects will be returned in the \"included\" field of the response. For compatibility, a string @\"owner_uuid\"@ is accepted as equivalent to @[\"owner_uuid\"]@.|query|@\"owner_uuid\"@\n@[\"owner_uuid\",\"container_uuid\"]@|\n|include_trash|boolean (default false)|Include trashed objects.|query|@true@|\n|include_old_versions|boolean (default false)|Include past versions of the collections being listed.|query|@true@|\n|select|array|Attributes of each object to return in the response. Specify an unqualified name like @uuid@ to select that attribute on all object types, or a qualified name like @collections.name@ to select that attribute on objects of the specified type. By default, all available attributes are returned, except on collections, where @manifest_text@ is not returned and cannot be selected due to an implementation limitation. This limitation may be removed in the future.|query|@[\"uuid\", \"collections.name\"]@|\n\nNotes:\n\nBecause adding access tokens to manifests can be computationally expensive, the @manifest_text@ field is not included in listed collections.  If you need it, request a \"list of collections\":{{site.baseurl}}/api/methods/collections.html with the filter @[\"owner_uuid\", \"=\", GROUP_UUID]@, and @\"manifest_text\"@ listed in the select parameter.\n\nUse filters with the attribute format @.@ to filter items of a specific type. For example: @[\"container_requests.state\", \"=\", \"Final\"]@ to filter @container_requests@ where @state@ is @Final@. All other types of items owned by this group will be unimpacted by this filter and will still be included.\n\nWhen called with “include=owner_uuid”, the @included@ field of the response is populated with users, projects, or other groups that own the objects returned in @items@.  This can be used to fetch an object and its parent with a single API call.\n\nWhen called with “include=container_uuid”, the @included@ field of the response is populated with the container associated with each container request in the response.\n\n\nh3. create\n\nCreate a new Group.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n|group|object||query||\n|async|boolean (default false)|Defer the permissions graph update by a configured number of seconds. (By default, @async_permissions_update_interval@ is 20 seconds). On success, the response is 202 (Accepted).|query|@true@|\n\nh3(#delete). delete\n\nPut a Group in the trash.  See \"Trashing groups\":#trashing for details.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the Group in question.|path||\n\nh3. get\n\nGets a Group's metadata by UUID.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the Group in question.|path||\n\nh3. list\n\nList groups.\n\nSee \"common resource list method.\":{{site.baseurl}}/api/methods.html#index\n\nh3. show\n\nshow groups\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string||path||\n\nh3. update\n\nUpdate attributes of an existing Group.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the Group in question.|path||\n|group|object||query||\n|async|boolean (default false)|Defer the permissions graph update by a configured number of seconds. (By default, @async_permissions_update_interval@ is 20 seconds). On success, the response is 202 (Accepted).|query|@true@|\n\nh3(#untrash). untrash\n\nRemove a Group from the trash.  Only valid when @delete_at@ is in the future.  This sets the @trash_at@ and @delete_at@ fields to @null@.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the Group to untrash.|path||\n|ensure_unique_name|boolean (default false)|Rename project uniquely if untrashing it would fail with a unique name conflict.|query||\n\nh3(#shared). shared\n\nThis endpoint returns the toplevel set of groups to which access is granted through a chain of one or more permission links rather than through direct ownership by the current user account.  This is useful for clients which wish to browse the list of projects the user has permission to read which are not part of the \"home\" project tree.  Similar behavior is also available with the @exclude_home_project@ option of the \"contents\" endpoint.\n\nSpecifically, the logic is:\n\n
\nselect groups that are readable by current user AND\n    (the owner_uuid is a user (but not the current user) OR\n     the owner_uuid is not readable by the current user OR\n     the owner_uuid is a group but group_class is not a project)\n
\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n|include|string|If provided with the value \"owner_uuid\", this will return owner objects in the @included@ field of the response.|query||\n\nNotes:\n\nWhen called with “include=owner_uuid”, the @included@ field of the response is populated with users and non-project groups that own the objects returned in @items@.\n\nIn addition to the \"include\" parameter this endpoint also supports the same parameters as the \"list method.\":{{site.baseurl}}/api/methods.html#index\n"
  },
  {
    "path": "doc/api/methods/keep_services.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\nnavmenu: API Methods\ntitle: \"keep_services\"\n\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nAPI endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/keep_services@\n\nObject type: @bi6l4@\n\nExample UUID: @zzzzz-bi6l4-0123456789abcde@\n\nh2. Resource\n\nThe keep_services resource keep clients to discover storage servers and proxies available on the cluster for persistent storage and retrieval of keep blocks.\n\nEach KeepService has, in addition to the \"Common resource fields\":{{site.baseurl}}/api/resources.html:\n\ntable(table table-bordered table-condensed).\n|_. Attribute|_. Type|_. Description|_. Example|\n|service_host|string|hostname of the server||\n|service_port|integer|TCP port of the service||\n|service_ssl_flag|boolean|if the server uses SSL||\n|service_type|string|The service type, one of \"disk\", \"blob\" (cloud object store) or \"proxy\" (keepproxy)||\n\nh2. Methods\n\nSee \"Common resource methods\":{{site.baseurl}}/api/methods.html for more information about @create@, @delete@, @get@, @list@, and @update@.\n\nRequired arguments are displayed in %{background:#ccffcc}green%.\n\nh3. accessible\n\nGet a list of keep services that are accessible to the requesting client.  Unlike @list@, this is context-sensitive based on the requester, for example providing the list of actual Keep servers when inside the cluster, but providing a proxy service if client contacts Arvados from outside the cluster.\n\nh3. create\n\nCreate a new KeepService.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n|keep_service|object||query||\n\nh3. delete\n\nDelete an existing KeepService.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the KeepService in question.|path||\n\nh3. get\n\nGets a KeepService's metadata by UUID.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the KeepService in question.|path||\n\nh3. list\n\nList keep_services.\n\nSee \"common resource list method.\":{{site.baseurl}}/api/methods.html#index\n\nh3. update\n\nUpdate attributes of an existing KeepService.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the KeepService in question.|path||\n|keep_service|object||query||\n"
  },
  {
    "path": "doc/api/methods/links.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\nnavmenu: API Methods\ntitle: \"links\"\n\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nAPI endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/links@\n\nObject type: @o0j2j@\n\nExample UUID: @zzzzz-o0j2j-0123456789abcde@\n\nh2. Resource\n\nLinks are an extensible way to describe relationships between Arvados objects and metadata about individual objects.\n\nEach link has, in addition to the \"Common resource fields\":{{site.baseurl}}/api/resources.html:\n\ntable(table table-bordered table-condensed).\n|_. Attribute|_. Type|_. Description|\n|head_uuid|string|The object being described or acted on.|\n|tail_uuid|string|The origin or actor in the description or action (may be null).|\n|link_class|string|Type of link|\n|name|string|Primary value of the link.|\n|properties|hash|Additional information, expressed as a key→value hash. Key: string. Value: string, number, array, or hash.  May be used in queries using \"subproperty filters\":{{site.baseurl}}/api/methods.html#subpropertyfilters|\n\nh2. Link classes\n\nSome classes are pre-defined by convention and have standard meanings attached to names.\n\nh3. permission\n\nThe significance of permission links is discussed in the \"permission links\":{{site.baseurl}}/api/permission-model.html#links section of the permission model documentation.\n\nh3. star\n\nA **star** link is a shortcut to a project that is displayed in the user interface (Workbench) as \"favorites\".  Users can mark their own favorites (implemented by creating or deleting **star** links).\n\nAn admin can also create **star** links owned by the \"Public favorites\" project.  These are favorites will be displayed to all users that have permission to read the project that has been favorited.\n\nThe schema for a star link is:\n\ntable(table table-bordered table-condensed).\n|_. Field|_. Value|_. Description|\n|owner_uuid|user or group uuid|Either the user that owns the favorite, or the \"Public favorites\" group.|\n|tail_uuid|user or group uuid|Should be the same as owner_uuid|\n|head_uuid|project uuid|The project being favorited|\n|link_class|string of value \"star\"|Indicates this represents a link to a user favorite|\n\nh4. Creating a public favorite\n\n@owner_uuid@ is either an individual user, or the \"Public favorites\" group.  The @head_uuid@ is the project being favorited.\n\n
\n$ linkuuid=$(arv --format=uuid link create --link '{\n    \"link_class\": \"star\",\n    \"owner_uuid\": \"zzzzz-j7d0g-publicfavorites\",\n    \"tail_uuid\": \"zzzzz-j7d0g-publicfavorites\",\n    \"head_uuid\":  \"zzzzz-j7d0g-theprojectuuid\"}')\n
\n\nh4. Removing a favorite\n\n
\n$ arv link delete --uuid zzzzz-o0j2j-thestarlinkuuid\n
\n\nh4. Listing favorites\n\nTo list all 'star' links that will be displayed for a user:\n\n
\n$ arv link list --filters '[\n  [\"link_class\", \"=\", \"star\"],\n  [\"tail_uuid\", \"in\", [\"zzzzz-j7d0g-publicfavorites\", \"zzzzz-tpzed-currentuseruuid\"]]]'\n
\n\nh3. tag\n\nA **tag** link describes an object using an unparsed plain text string.  Tags can be used to annotate objects that are not directly editable by the user, like collections and objects shared as read-only.\n\ntable(table table-bordered table-condensed).\n|_. tail_type→head_type|_. name→head_uuid {properties}|\n|→Collection           | _tag name_ → _collection uuid_|\n|→Job                  | _tag name_ → _job uuid_|\n\nh3. published_port\n\nA **published_port** link enables external access to container ports via user-defined domain names.\n\nIf the cluster is configured as follows to forward HTTP requests from external clients to container ports:\n\n
\nServices:\n  ContainerWebServices:\n    ExternalURL: https://*.containers.zzzzz.example.com/\n
\n\nA user can create the following link to route HTTP requests like @https://servicename.containers.zzzzz.example.com/@ to port 12345 in the container running for container request @zzzzz-xvhdp-012340123401234@:\n\n
\n{\n  \"link_class\" \"published_port\",\n  \"head_uuid\": \"zzzzz-xvhdp-012340123401234\",\n  \"name\": \"servicename\",\n  \"properties\": {\n    \"port\": 12345\n  }\n}\n
\n\nRefer to the \"documentation about published ports\":container_requests.html#published_ports for additional information.\n\nh2. Methods\n\nSee \"Common resource methods\":{{site.baseurl}}/api/methods.html for more information about @create@, @delete@, @get@, @list@, and @update@.\n\nRequired arguments are displayed in %{background:#ccffcc}green%.\n\nh3. create\n\nCreate a new Link.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n|link|object||query||\n\nWhen you create a new permission link with the same @head_uuid@ and @tail_uuid@ as an existing permission link, the API returns the existing link instead of creating a new one. If the requested permission level is higher than the existing link, the existing link is updated accordingly. Otherwise the existing link is returned unchanged.\n\nh3. delete\n\nDelete an existing Link.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the Link in question.|path||\n\nWhen you delete a permission link, any other existing permission links that have the same @head_uuid@ and @tail_uuid@ are also deleted.\n\nh3. get\n\nGets a Link's metadata by UUID.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the Link in question.|path||\n\nh3. list\n\nList links.\n\nSee \"common resource list method.\":{{site.baseurl}}/api/methods.html#index\n\nh3. update\n\nUpdate attributes of an existing Link.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the Link in question.|path||\n|link|object||query||\n\nWhen you update a permission link such that it has the same @head_uuid@ and @tail_uuid@ as one or more existing permission links, the API deletes the other links. If the highest permission level among the deleted links was higher than the newly updated link, the updated link's permission level is increased accordingly.\n\nh3. get_permissions\n\nGet all permission links that point directly to given UUID (in the head_uuid field).  The requesting user must have @can_manage@ permission or be an admin.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the object.|path||\n"
  },
  {
    "path": "doc/api/methods/logs.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\nnavmenu: API Methods\ntitle: \"logs\"\n\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nAPI endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/logs@\n\nObject type: @57u5n@\n\nExample UUID: @zzzzz-57u5n-0123456789abcde@\n\nh2. Resource\n\nEach Log has, in addition to the \"Common resource fields\":{{site.baseurl}}/api/resources.html:\n\ntable(table table-bordered table-condensed).\n|_. Attribute|_. Type|_. Description|_. Example|\n|object_uuid|string|The arvados object that is the subject of the log.||\n|event_at|datetime|||\n|event_type|string|A user-defined category or type for this event.|@LOGIN@|\n|summary|text|||\n|properties|hash|||\n\nh3. Creation\n\nAny user may create Log entries for any event they find useful. User-generated Logs have no intrinsic meaning to other users or to the Arvados system itself; it is up to each user to choose appropriate log event types and summaries for their project.\n\nh3. System Logs\n\nArvados uses Logs to record creation, deletion, and updates of other Arvados resources.\n\nh2. Methods\n\nSee \"Common resource methods\":{{site.baseurl}}/api/methods.html for more information about @create@, @delete@, @get@, @list@, and @update@.\n\nRequired arguments are displayed in %{background:#ccffcc}green%.\n\nh3. create\n\nCreate a new log entry.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n|log|object||query||\n\nh3. delete\n\nDelete an existing log entry. This method can only be used by privileged (system administrator) users.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the log entry in question.|path||\n\nh3. get\n\nRetrieve a log entry.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the log entry in question.|path||\n\nh3. list\n\nList log entries.\n\nSee \"common resource list method.\":{{site.baseurl}}/api/methods.html#index\n\nh3. update\n\nUpdate attributes of an existing log entry. This method can only be used by privileged (system administrator) users.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the log entry in question.|path||\n|log|object||query||\n"
  },
  {
    "path": "doc/api/methods/user_agreements.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\nnavmenu: API Methods\ntitle: \"user_agreements\"\n\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nAPI endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/user_agreements@\n\nh2. Resource\n\nThis provides an API for inactive users to sign clickthrough agreements prior to being activated.\n\nh2. Methods\n\nRequired arguments are displayed in %{background:#ccffcc}green%.\n\nh3. list\n\nList user agreements.  This is a list of collections which contain HTML files with the text of the clickthrough agreement(s) which can be rendered by Workbench.\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n\nh3. signatures\n\nList user agreements that have already been signed.  These are recorded as link objects of @{\"link_class\": \"signature\", \"name\": \"click\"}@.\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n\nh3. sign\n\nSign a user agreement.\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the user agreement collection.|path||\n"
  },
  {
    "path": "doc/api/methods/users.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\nnavmenu: API Methods\ntitle: \"users\"\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nAPI endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/users@\n\nObject type: @tpzed@\n\nExample UUID: @zzzzz-tpzed-0123456789abcde@\n\nh2. Resource\n\nUsers represent individuals with access to the Arvados cluster.\n\nEach User has, in addition to the \"Common resource fields\":{{site.baseurl}}/api/resources.html:\n\ntable(table table-bordered table-condensed).\n|_. Attribute|_. Type|_. Description|_. Example|\n|email|string|||\n|username|string|The username used for the user's git repositories and virtual machine logins.  Usernames must start with a letter, and contain only alphanumerics.  When a new user is created, a default username is set from their e-mail address.  Only administrators may change the username.||\n|first_name|string|||\n|last_name|string|||\n|identity_url|string|||\n|is_admin|boolean|||\n|prefs|hash|||\n|is_active|boolean|||\n\nh2. Methods\n\nSee \"Common resource methods\":{{site.baseurl}}/api/methods.html for more information about @create@, @delete@, @get@, @list@, and @update@.\n\nRequired arguments are displayed in %{background:#ccffcc}green%.\n\nh3. create\n\nCreate a new User.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n|user|object||query||\n\nh3(#current). current\n\nGet the user associated with the provided API token.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n\nh3. delete\n\nDelete an existing User.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the User in question.|path||\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string||path||\n\nh3. get\n\nGets a User's metadata by UUID.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the User in question.|path||\n\nh3. list\n\nList users.\n\nSee \"common resource list method.\":{{site.baseurl}}/api/methods.html#index\n\nh3. system\n\nGet the user record for the \"system user.\":{{site.baseurl}}/api/permission-model.html#system\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n\nh3. update\n\nUpdate attributes of an existing User.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the User in question.|path||\n|user|object|The new attributes.|query||\n\nh3. setup\n\nSet up a user.  Adds the user to the \"All users\" group.  Enables the user to invoke @activate@.  See \"user management\":{{site.baseurl}}/admin/user-management.html for details.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the User in question.|query||\n\nh3. activate\n\nCheck that a user has is set up and has signed all the user agreements.  If so, activate the user.  Users can invoke this for themselves.  See \"user agreements\":{{site.baseurl}}/admin/user-management.html#user_agreements for details.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the User in question.|query||\n\nh3. unsetup\n\nRemove the user from the \"All users\" group and deactivate the user.  See \"user management\":{{site.baseurl}}/admin/user-management.html for details.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the User in question.|path||\n\nh3. merge\n\nTransfer ownership of data from the \"old\" user account to the \"new\" user account.  When @redirect_to_new_user@ is @true@ this also causes logins to the \"old\" account to be redirected to the \"new\" account.  The \"old\" user account that was redirected becomes invisible in user listings.\n\nSee \"Merge user accounts\":{{site.baseurl}}/admin/link-accounts.html, \"Reassign user data ownership\":{{site.baseurl}}/admin/reassign-ownership.html, and \"Linking alternate login accounts\":{{site.baseurl}}/user/topics/link-accounts.html for examples of how this method is used.\n\nMust supply either @new_user_token@ (the currently authorized user will be the \"old\" user), or both @new_user_uuid@ and @old_user_uuid@ (the currently authorized user must be an admin).\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n|new_user_token|string|A valid token for the \"new\" user|query||\n|new_user_uuid|uuid|The uuid of the \"new\" account|query||\n|old_user_uuid|uuid|The uuid of the \"old\" account|query||\n|new_owner_uuid|uuid|The uuid of a project to which objects owned by the \"old\" user will be reassigned.|query||\n|redirect_to_new_user|boolean|If true, also redirect login and reassign authorization credentials from \"old\" user to the \"new\" user|query||\n\nh3. authenticate\n\nCreate a new API token based on username/password credentials.  Returns an \"API client authorization\":api_client_authorizations.html object containing the API token, or an \"error object.\":../requests.html#errors\n\nValid credentials are determined by the choice of \"configured login backend.\":{{site.baseurl}}/install/setup-login.html\n\nNote: this endpoint cannot be used with login backends that use web-based third party authentication, such as Google or OpenID Connect.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|username|string|The username.|body||\n{background:#ccffcc}.|password|string|The password.|body||\n"
  },
  {
    "path": "doc/api/methods/virtual_machines.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\nnavmenu: API Methods\ntitle: \"virtual_machines\"\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nAPI endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/virtual_machines@\n\nObject type: @2x53u@\n\nExample UUID: @zzzzz-2x53u-0123456789abcde@\n\nh2. Resource\n\nThe virtual_machines resource lists compute resources in the Arvados cluster to which a user may log in to get an interactive shell (via ssh or webshell).\n\nEach VirtualMachine has, in addition to the \"Common resource fields\":{{site.baseurl}}/api/resources.html:\n\ntable(table table-bordered table-condensed).\n|_. Attribute|_. Type|_. Description|_. Example|\n|hostname|string|||\n\nh2. Methods\n\nSee \"Common resource methods\":{{site.baseurl}}/api/methods.html for more information about @create@, @delete@, @get@, @list@, and @update@.\n\nRequired arguments are displayed in %{background:#ccffcc}green%.\n\nh3. create\n\nCreate a new VirtualMachine.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n|virtual_machine|object||query||\n\nh3. delete\n\nDelete an existing VirtualMachine.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the VirtualMachine in question.|path||\n\nh3. get\n\nGets a VirtualMachine's metadata by UUID.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the VirtualMachine in question.|path||\n\nh3(#logins). logins\n\nGet a list of SSH keys and account names that should be able to log in to a given virtual machine.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string||path||\n\nThe response is an object with the field @items@ containing an array of objects in the following format:\n\ntable(table table-bordered table-condensed).\n|_. Key|_. Value type|_. Description|_. Example|\n|username|string|Name of the Unix login account to which the user should be able to log in|@\"jsmith\"@|\n|hostname|string|Hostname of the virtual machine|@\"shell.xyzzy.arvadosapi.com\"@|\n|public_key|string|SSH public key|@\"ssh-rsa AAAAB3NzaC1yc2E...\"@|\n|user_uuid|string|UUID of the user who should be able to log in|@\"xyzzy-tpzed-mv4d7dy7n91te11\"@|\n|virtual_machine_uuid|string|UUID of the \"virtual machine resource\":{{site.baseurl}}/api/methods/virtual_machines.html|@\"zzzzz-2x53u-kvszmclnbjuv8xc\"@|\n|authorized_key_uuid|string|UUID of the \"authorized key resource\":{{site.baseurl}}/api/methods/authorized_keys.html|@\"zzzzz-fngyi-v9p0cyfmjxbio64\"@|\n\nh3. get_all_logins\n\nGet a list of SSH keys and account names that should be able to log in for every virtual machine in the system.\n\nArguments: none.\n\nThe response has the same format as the response to the \"logins method\":#logins above.\n\nh3. list\n\nList virtual_machines.\n\nSee \"common resource list method.\":{{site.baseurl}}/api/methods.html#index\n\nh3. update\n\nUpdate attributes of an existing VirtualMachine.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the VirtualMachine in question.|path||\n|virtual_machine|object||query||\n"
  },
  {
    "path": "doc/api/methods/workflows.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\nnavmenu: API Methods\ntitle: \"workflows\"\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nAPI endpoint base: @https://{{ site.arvados_api_host }}/arvados/v1/workflows@\n\nObject type: @7fd4e@\n\nExample UUID: @zzzzz-7fd4e-0123456789abcde@\n\nh2. Resource\n\nStores a \"Common Workflow Language\":http://commonwl.org (CWL) computational workflow that can be searched for, browsed and executed (submitted to Crunch) from the workbench.\n\nEach Workflow offers the following optional attributes, in addition to the \"Common resource fields\":{{site.baseurl}}/api/resources.html:\n\ntable(table table-bordered table-condensed).\n|_. Attribute|_. Type|_. Description|_. Example|\n|name|string|If not specified, will be set to any \"name\" from the \"definition\" attribute.||\n|description|string|If not specified, will be set to any \"description\" from the \"definition\" attribute.||\n|definition|string|A \"Common Workflow Language\" document.|Visit \"Common Workflow Language\":http://www.commonwl.org/ for details.|\n|collection_uuid|string|This attribute is always null. It is reserved for future development. {% comment until 23057 %} If non-null, a linked workflow definition stored in a Collection.  See below. {% endcomment %}||\n\n{% comment until 23057 %}\nh2. Workflows linked to Collections\n\nIf @collection_uuid@ is set, this significantly changes the behavior of the workflow record.\n\nThe linked Collection must have the following properties.  These are extracted from and must be synchronized with the workflow in @arv:workflowMain@. They are copied into the workflow collection's @properties@ for ease of processing by client tools such as Workbench.\n\ntable(table table-bordered table-condensed).\n|_. Attribute|_. Type|_. Description|\n|type|string|Value must be 'workflow'|\n|arv:workflowMain|string|The file path within the collection that is the top-level workflow that will be launched.|\n|arv:cwl_inputs|array of object|Array of \"workflow input parameters\":https://www.commonwl.org/v1.2/Workflow.html#WorkflowInputParameter in \"fully expanded form\":https://www.commonwl.org/v1.2/SchemaSalad.html#Document_preprocessing |\n|arv:cwl_outputs|array of object|Array of \"workflow output parameters\":https://www.commonwl.org/v1.2/Workflow.html#WorkflowOutputParameter in \"fully expanded form\":https://www.commonwl.org/v1.2/SchemaSalad.html#Document_preprocessing |\n|arv:cwl_requirements|array of object|Array of \"workflow process requirements\":https://www.commonwl.org/v1.2/Workflow.html#Workflow in \"fully expanded form\":https://www.commonwl.org/v1.2/SchemaSalad.html#Document_preprocessing (in particular, this must list requirements that affect initial launching of the workflow such as \"WorkflowRunnerResources\":{{site.baseurl}}/user/cwl/cwl-extensions.html ).|\n|arv:cwl_hints|array of object|Array of \"workflow process hints\":https://www.commonwl.org/v1.2/Workflow.html#Workflow in \"fully expanded form\":https://www.commonwl.org/v1.2/SchemaSalad.html#Document_preprocessing (in particular, this must list hints that affect initial launching of the workflow such as \"WorkflowRunnerResources\":{{site.baseurl}}/user/cwl/cwl-extensions.html ).|\n\nWhen @collection_uuid@ is set, the workflow record @name@, @description@, @definition@ and @owner_uuid@ are all set from the linked collection.  The workflow record can no longer be updated directly, but changes to the linked collection will be reflected in the workflow record.  Trashing the linked collection will cause the workflow record to become trashed and eventually deleted as well.  The workflow record cannot be un-linked from a collection, only deleted and re-created.\n\nWhen a workflow is linked to a collection, the collection can be queried and fetched together with the workflow.  The @filters@ argument can filter on attributes of the collection referenced by @collection_uuid@. For example, @[[\"collection.properties.category\", \"=\", \"WGS\"]]@ will match workflow definitions linked to collections that have a \"category\" property with the value \"WGS\".  When using the \"group contents\":groups.html#contents API to fetch workflow records, in addition the previously-described filters, you can use @include=[\"collection_uuid\"]@ to include the collection records corresponding to the @collection_uuid@ of the workflow records in the response.\n{% endcomment %}\n\nh2. Methods\n\nSee \"Common resource methods\":{{site.baseurl}}/api/methods.html for more information about @create@, @delete@, @get@, @list@, and @update@.\n\nRequired arguments are displayed in %{background:#ccffcc}green%.\n\nSupports federated @create@, @delete@, @get@, @list@, and @update@.\n\nh3. create\n\nCreate a new Workflow.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|workflow|object|Workflow resource|request body||\n\nh3. delete\n\nDelete an existing Workflow.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the Workflow in question.|path||\n\nh3. get\n\nGet a Workflow's metadata by UUID.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the Workflow in question.|path||\n\nh3. list\n\nList workflows.\n\nSee \"common resource list method.\":{{site.baseurl}}/api/methods.html#index\n\n{% comment until 23057 %}\nThe @filters@ argument can filter on attributes of the collection referenced by @collection_uuid@. For example, @[[\"collection.properties.category\", \"=\", \"WGS\"]]@ will match workflow definitions linked to collections that have a \"category\" property with the value \"WGS\".\n{% endcomment %}\n\nh3. update\n\nUpdate attributes of an existing Workflow.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |_. Example |\n{background:#ccffcc}.|uuid|string|The UUID of the Workflow in question.|path||\n|workflow|object||query||\n"
  },
  {
    "path": "doc/api/methods.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\nnavmenu: Concepts\ntitle: Common resource methods\n\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThe following methods are available for most resources.  Some resources may limit who can perform certain operations.  Consult documentation for individual resource types for details.\n\nThe methods are relative to the base URI, e.g., @/arvados/v1/resource_type@.  For arguments specifying a *Location* of @path@, the value of the argument is incorporated into the path portion of the URI.  For example, a @uuid@ of @aaaaa-bbbbb-ccccccccccccccc@ in a path position yields a URI of @/arvados/v1/resource_type/aaaaa-bbbbb-ccccccccccccccc@.\n\nArguments specifying a *Location* of \"query\" are incorporated into the query portion of the URI or request body.  For example, @/arvados/v1/resource_type?count=none@.\n\nCertain method calls on certain object types support \"federation\":{{site.baseurl}}/architecture/federation.html: the ability to operate on objects owned by different clusters.   API pages for specific object types list which federated operations are supported for that type (if any) in the \"Methods\" section.  Methods which implicitly include a cluster ID (such as @GET@ on a specific UUID, using the UUID prefix) will be directed to the appropriate cluster.  Methods that don't implicitly include the cluster ID (such as @create@) use the @cluster_id@ query parameter to specify which cluster to direct the request.\n\n* \"create\":#create\n* \"delete\":#delete\n* \"get\":#get\n* \"list\":#index\n** \"Available list method filters\":#filters\n*** \"Filtering using substring search\":#substringsearchfilter\n*** \"Filtering on subproperties\":#subpropertyfilters\n*** \"Filtering using boolean expressions\":#filterexpression\n** \"Federated listing\":#federated-list\n** \"Results of list method\":#list-results\n* \"update\":#update\n\nh2(#create). create\n\nThe @create@ method creates a new object of the specified type.  Note that:\n\n* Only the listed attributes (and \"standard metadata\":resources.html) are set\n* Unset attributes will get default values\n* The attributes of a given resource type are fixed (you cannot introduce new toplevel attributes)\n\nThis method corresponds to the HTTP request @POST /arvados/v1/resource_type@.  A successful create call returns a copy of the new object.\n\nTo create an object on a remote cluster (federated create), provide the @cluster_id@ of the target cluster.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |\n|{resource_type}|object|Name is the singular form of the resource type, e.g., for the \"collections\" resource, this argument is \"collection\"|body|\n|{cluster_id}|string|Optional, the cluster on which to create the object if not the current cluster.|query|\n|select  |array  |Attributes of the new object to return in the response (by default, all available attributes are returned).\nExample: @[\"uuid\",\"name\",\"modified_at\"]@|query|\n\nh2(#delete). delete\n\nThe @delete@ method deletes an object of the specified type.  It corresponds to the HTTP request @DELETE /arvados/v1/resource_type/uuid@.  A successful delete call returns a copy of the deleted object.\n\nThe cluster ID portion of the @uuid@ is used to determine which cluster owns the object, a federated delete request will be routed to that cluster.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |\n{background:#ccffcc}.|uuid|string|The UUID of the object in question.|path|\n|select  |array  |Attributes of the deleted object to return in the response (by default, all available attributes are returned).\nExample: @[\"uuid\",\"name\",\"modified_at\"]@|query|\n\nh2(#get). get\n\nThe @get@ method gets a single object with the specified @uuid@.  It corresponds to the HTTP request @GET /arvados/v1/resource_type/uuid@.\n\nThe cluster ID portion of the @uuid@ is used to determine which cluster owns the object, a federated get request will be routed to that cluster.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |\n{background:#ccffcc}.|uuid|string|The UUID of the object in question.|path|\n|select  |array  |Attributes of the object to return in the response (by default, all available attributes are returned).\nExample: @[\"uuid\",\"name\",\"modified_at\"]@|query|\n\nh2(#index). list\n\nThe @list@ method requests an list of resources of that type.  It corresponds to the HTTP request @GET /arvados/v1/resource_type@.  All resources support the @list@ method unless otherwise noted.\n\nArguments:\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |\n|limit   |integer|Maximum number of resources to return.  If not provided, server will provide a default limit.  Server may also impose a maximum number of records that can be returned in a single request.|query|\n|offset  |integer|Skip the first 'offset' number of resources that would be returned under the given filter conditions.|query|\n|filters |array  |\"Conditions for selecting resources to return.\":#filters|query|\n|order   |array  |Attributes to use as sort keys to determine the order resources are returned, each optionally followed by @asc@ or @desc@ to indicate ascending or descending order.  (If not specified, it will be ascending).\nExample: @[\"head_uuid asc\",\"modified_at desc\"]@\nDefault: @[\"modified_at desc\", \"uuid asc\"]@|query|\n|select  |array  |Attributes of each object to return in the response (by default, all available attributes are returned, except collections, which do not return @manifest_text@ unless explicitly selected).\nExample: @[\"uuid\",\"name\",\"modified_at\"]@|query|\n|distinct|boolean|When returning multiple records whose selected attributes (see @select@) are equal, return them as a single response entry.\nDefault is @false@.|query|\n|count|string|@\"exact\"@ (default): Include an @items_available@ response field giving the number of distinct matching items that can be retrieved (irrespective of @limit@ and @offset@ arguments).\n@\"none\"@: Omit the @items_available@ response field. This option will produce a faster response.|query|\n\nh3(#filters). Available list method filters\n\nThe value of the @filters@ parameter is an array of conditions. The @list@ method returns only the resources that satisfy all of the given conditions. In other words, the conjunction @AND@ is implicit.\n\nEach condition is expressed as an array with three elements: @[attribute, operator, operand]@.\n\ntable(table table-bordered table-condensed).\n|_. Index|_. Element|_. Type|_. Description|_. Examples|\n|0|attribute|string|Name of the attribute to compare (or \"any\" to return resources with any matching attribute)|@script_version@, @head_uuid@, @any@|\n|1|operator|string|Comparison operator|@>@, @>=@, @like@, @not in@|\n|2|operand|string, array, or null|Value to compare with the resource attribute|@\"d00220fb%\"@, @\"1234\"@, @[\"foo\",\"bar\"]@, @nil@|\n\nThe following operators are available.\n\ntable(table table-bordered table-condensed).\n|_. Operator|_. Operand type|_. Description|_. Example|\n|@=@, @!=@, @<>@|string, number, timestamp, JSON-encoded array, JSON-encoded object, or null|Equality comparison|@[\"tail_uuid\",\"=\",\"xyzzy-j7d0g-fffffffffffffff\"]@\n@[\"tail_uuid\",\"!=\",null]@\n@[\"storage_classes_desired\",\"=\",\"[\\\"default\\\"]\"]@|\n|@<@, @<=@, @>=@, @>@|string, number, or timestamp|Ordering comparison|@[\"script_version\",\">\",\"123\"]@|\n|@like@, @ilike@|string|SQL pattern match.  Single character match is @_@ and wildcard is @%@. The @ilike@ operator is case-insensitive|@[\"script_version\",\"like\",\"d00220fb%\"]@|\n|@in@, @not in@|array of strings or integers|Set membership|@[\"script_version\",\"in\",[\"main\",\"d00220fb38d4b85ca8fc28a8151702a2b9d1dec5\"]]@|\n|@is_a@|string|Arvados object type|@[\"head_uuid\",\"is_a\",\"arvados#collection\"]@|\n|@exists@|string|Presence of subproperty|@[\"properties\",\"exists\",\"my_subproperty\"]@|\n|@contains@|string, array of strings|Presence of one or more keys or array elements|@[\"storage_classes_desired\", \"contains\", [\"foo\", \"bar\"]]@ (matches both @[\"foo\", \"bar\"]@ and @[\"foo\", \"bar\", \"baz\"]@)\n(note @[..., \"contains\", \"foo\"]@ is also accepted, and is equivalent to @[..., \"contains\", [\"foo\"]]@)|\n\nh4(#substringsearchfilter). Filtering using substring search\n\nResources can also be filtered by searching for a substring in attributes of type @string@, @array of strings@, @text@, and @hash@, which are indexed in the database specifically for search. To use substring search, the filter must:\n\n* Specify @any@ as the attribute\n* Use either the @like@ or @ilike@ operator\n* Have an operand of type @string@ that is wrapped in the SQL pattern match wildcard character @%@\n\nFor example, the @[\"any\", \"like\", \"%foo%\"]@ filter will return all resources that contain @foo@ in the content of at least one attribute of the previously defined types. This is the recommended way to do keyword and file name search across the entire database. Note that only exact substring matches are returned and results are unranked and returned in the order specified by the @list@ @order@ argument.\n\nh4(#subpropertyfilters). Filtering on subproperties\n\nSome record types have an additional @properties@ attribute that allows recording and filtering on additional key-value pairs.  To filter on a subproperty, the value in the @attribute@ position has the form @properties.user_property@.  You may also use JSON-LD / RDF style URIs for property keys by enclosing them in @<...>@ for example @properties.@.  Alternately you may also provide a JSON-LD \"@context\" field, however at this time JSON-LD contexts are not interpreted by Arvados.\n\ntable(table table-bordered table-condensed).\n|_. Operator|_. Operand type|_. Description|_. Example|\n|@=@, @!=@|string, number or boolean|Equality comparison|@[\"properties.my_subproperty\", \"=\", \"fizzy whizy sparkle pop\"]@|\n|@<@, @<=@, @>=@, @>@|string or number|Ordering comparison|@[\"properties.my_subproperty\", \"<\", 3]@|\n|@like@, @ilike@|string|SQL pattern match, single character match is @_@ and wildcard is @%@, ilike is case-insensitive|@[\"properties.my_subproperty\", \"like\", \"d00220fb%\"]@|\n|@in@, @not in@|array of strings|Set membership|@[\"properties.my_subproperty\", \"in\", [\"fizz\", \"buzz\"]]@|\n|@exists@|boolean|Test if a subproperty is present or not (determined by operand).|@[\"properties.my_subproperty\", \"exists\", true]@|\n|@contains@|string, number|Filter where subproperty has a value either by exact match or value is element of subproperty list.|@[\"properties.foo\", \"contains\", \"bar\"]@ will find both @{\"foo\": \"bar\"}@ and @{\"foo\": [\"bar\", \"baz\"]}@.|\n\nNote that exclusion filters @!=@ and @not in@ will return records for which the property is not defined at all.  To restrict filtering to records on which the subproperty is defined, combine with an @exists@ filter.\n\nh4(#filterexpression). Filtering using boolean expressions\n\nIn addition to the three-element array form described above, a string containing a boolean expression is also accepted. The following restrictions apply:\n* The expression must contain exactly one operator.\n* The operator must be @=@, @<@, @<=@, @>@, or @>=@.\n* There must be exactly one pair of parentheses, surrounding the entire expression.\n* Each operand must be the name of a numeric attribute like @replication_desired@ (literal values like @3@ and non-numeric attributes like @uuid@ are not accepted).\n* The expression must not contain whitespace other than an ASCII space (newline and tab characters are not accepted).\n\nExamples:\n* @(replication_desired > replication_confirmed)@\n* @(replication_desired = replication_confirmed)@\n\nBoth types of filter (boolean expressions and @[attribute, operator, operand]@ filters) can be combined in the same API call. Example:\n* @{\"filters\": [\"(replication_desired > replication_confirmed)\", [\"replication_desired\", \"<\", 2]]}@\n\nh3(#federated-list). Federated listing\n\nFederated listing forwards a request to multiple clusters and combines the results.  Currently only a very restricted form of the \"list\" method is supported.\n\nTo query multiple clusters, the list request must:\n\n* Have filters only matching @[[\"uuid\", \"in\", [...]]@ or @[\"uuid\", \"=\", \"...\"]@\n* Specify @count=none@\n* Not specify @limit@, @offset@ or @order@\n* Not request more items than the maximum response size\n\nThis form may be used to request a specific list of objects by UUID which are owned by multiple clusters.\n\nh3(#list-results). Results of list method\n\nA successful call to list will return the following object.\n\ntable(table table-bordered table-condensed).\n|_. Attribute |_. Type |_. Description |\n|kind|string|type of objects returned|\n|offset|integer|query offset in effect|\n|limit|integer|query limit in effect|\n|items|array|actual query payload, an array of resource objects|\n|items_available|integer|total items available matching query|\n\nh2(#update). update\n\nThe @update@ method updates fields on the object with the specified @uuid@.  It corresponds to the HTTP request @PUT /arvados/v1/resource_type/uuid@.  Note that only the listed attributes (and \"standard metadata\":resources.html) are updated, unset attributes will retain their previous values, and the attributes of a given resource type are fixed (you cannot introduce new toplevel attributes).  Also note that updates replace the value of the attribute, so if an attribute has an object value, the entire object is replaced.  A successful update call returns the updated copy of the object.\n\nThe cluster ID portion of the @uuid@ is used to determine which cluster owns the object, a federated update request will be routed to that cluster.\n\ntable(table table-bordered table-condensed).\n|_. Argument |_. Type |_. Description |_. Location |\n{background:#ccffcc}.|uuid|string|The UUID of the resource in question.|path||\n|{resource_type}|object||query||\n|select  |array  |Attributes of the updated object to return in the response (by default, all available attributes are returned).\nExample: @[\"uuid\",\"name\",\"modified_at\"]@|query|\n"
  },
  {
    "path": "doc/api/permission-model.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: architecture\nnavmenu: Concepts\ntitle: \"Permission model\"\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThere are four levels of permission: *none*, *can_read*, *can_write*, and *can_manage*.\n\n* *none* is the default state when there are no other permission grants.\n** the object is not included in any list query response.\n** direct queries of the object by uuid return 404 Not Found.\n** Link objects require valid identifiers in @head_uuid@ and @tail_uuid@, so an attempt to create a Link that references an unreadable object will return an error indicating the object is not found.\n* *can_read* grants read-only access to the record.  Attempting to update or delete the record returns an error.\n** *can_read* does not allow a reader to see any permission grants on the object except the object's owner_uuid and the reader's own permissions.\n* *can_write* permits changes to the record, including changing ownership and deleting the object.\n** *can_write* cannot read, create, update or delete permission links associated with the object.\n** *can_write* also implies *can_read*.\n* *can_manage* permits the user to read, create, update and delete permission links whose @head_uuid@ is this object's @uuid@.\n** *can_manage* also implies *can_write* and *can_read*.\n\nh2. Ownership\n\nAll Arvados objects have an @owner_uuid@ field. Valid uuid types for @owner_uuid@ are \"User\" and \"Group\".  In the case of a Group, the @group_class@ must be \"project\".\n\nThe User or Group specified by @owner_uuid@ has *can_manage* permission on the object.  This permission is one way: an object that is owned does not get any special permissions on the User or Group that owns it.\n\nTo change the @owner_uuid@ field, it is necessary to have @can_write@ permission on both the current owner and the new owner.\n\nh2(#links). Permission links\n\nA permission link is a link object with:\n\n* @owner_uuid@ of the system user.\n* @link_class@ \"permission\"\n* @name@ one of *can_read*, *can_write*, *can_manage* or *can_login*\n* @head_uuid@ of some Arvados object\n* @tail_uuid@ of a User or Group.  For Group, the @group_class@ must be a \"role\".\n\nThis grants the permission in @name@ for @tail_uuid@ accessing @head_uuid@.\n\nIf a User has *can_manage* permission on some object, the user has the ability to read, create, update and delete permission links with @head_uuid@ of the managed object.  In other words, the user has the ability to modify the permission grants on the object.\n\nThe *can_login* @name@ is only meaningful on a permission link with with @tail_uuid@ a user UUID and @head_uuid@ a Virtual Machine UUID. A permission link of this type gives the user UUID permission to log into the Virtual Machine UUID. The username for the VM is specified in the @properties@ field. Group membership can be specified that way as well, optionally. See the \"VM login section on the 'User management at the CLI' page\":{{ site.baseurl }}/admin/user-management-cli.html#vm-login for an example.\n\nh3. Transitive permissions\n\nPermissions can be obtained indirectly through nested ownership (*can_manage*) or by following multiple permission links.\n\n* If a User X owns project A, and project A owns project B, then User X *can_manage* project B.\n* If a User X *can_read* role A, and role A *can_read* Object B, then User X *can_read* Object B.\n* Permissions are narrowed to the least powerful permission on the path.\n** If User X *can_write* role A, and role A *can_read* Object B, then User X *can_read* Object B.\n** If User X *can_read* role A, and role A *can_write* Object B, then User X *can_read* Object B.\n\nh2. Projects and Roles\n\nA \"project\" is a subtype of Group that is displayed as a \"Project\" in Workbench, and as a directory by @arv-mount@.\n* A project can own things (appear in @owner_uuid@)\n* A project can be owned by a user or another project.\n* The name of a project is unique only among projects and filters with the same owner_uuid.\n* Projects can be targets (@head_uuid@) of permission links, but not origins (@tail_uuid@).  Putting a project in a @tail_uuid@ field is an error.\n\nA \"filter\" is a subtype of Group that is displayed as a \"Project\" in Workbench, and as a directory by @arv-mount@. See \"the groups API documentation\":{{ site.baseurl }}/api/methods/groups.html for more information.\n* A filter group cannot own things (cannot appear in @owner_uuid@).  Putting a filter group in an @owner_uuid@ field is an error.\n* A filter group can be owned by a user or a project.\n* The name of a filter is unique only among projects and filters with the same owner_uuid.\n* Filters can be targets (@head_uuid@) of permission links, but not origins (@tail_uuid@).  Putting a filter in a @tail_uuid@ field is an error.\n\nA \"role\" is a subtype of Group that is treated in Workbench as a group of users who have permissions in common (typically an organizational group).\n* A role cannot own things (cannot appear in @owner_uuid@).  Putting a role in an @owner_uuid@ field is an error.\n* All roles are owned by the system user.\n* The name of a role is unique across a single Arvados cluster.\n* Roles can be both targets (@head_uuid@) and origins (@tail_uuid@) of permission links.\n* By default, all roles are visible to all active users. However, if the configuration entry @Users.RoleGroupsVisibleToAll@ is @false@, visibility is determined by normal permission rules, _i.e._, a role is only visible to users who have that role, and to admins.\n* By default, any user can create a new role. However, if the configuration entry @Users.CanCreateRoleGroups@ is @false@, only admins can create roles.\n\nh3. Access through Roles\n\nA \"role\" consists of a set of users or other roles that have that role, and a set of permissions (primarily read/write/manage access to projects) the role grants.\n\nIf there is a permission link stating that user A *can_write* role R, then we say A has role R.  This means user A has up to *can_write* access to everything the role has access to.\n\nBecause permissions are one-way, the links A *can_write* R and B *can_write* R does not imply that user A and B will be able to see each other.  For users in a role to see each other, read permission should be added going in the opposite direction: R *can_read* A and R *can_read* B.\n\nIf a user needs to be able to manipulate permissions of objects that are accessed through the role (for example, to share project P with a user outside the role), then role R must have *can_manage* permission on project P (R *can_manage* P) and the user must be granted *can_manage* permission on R (A *can_manage* R).\n\nh2. Special cases\n\nLog table objects are additionally readable based on whether the User has *can_read* permission on @object_uuid@ (User can access log history about objects it can read).  To retain the integrity of the log, the log table denies all update or delete operations.\n\nPermission links where @tail_uuid@ is a User allow *can_read* on the link record by that user (User can discover her own permission grants.)\n\nAt least *can_read* on a Collection grants permission to read the blocks that make up the collection (API server returns signed blocks).\n\nA user can only read a container record if the user has read permission to a container_request with that container_uuid.\n\n*can_read* and *can_write* access on a user grants access to the user record, but not anything owned by the user.\n*can_manage* access to a user grants can_manage access to the user, _and everything owned by that user_ .\nIf a user A *can_read* role R, and role R *can_manage* user B, then user A *can_read* user B _and everything owned by that user_ .\n\nModifying a role group requires *can_manage* permission (by contrast, *can_write* is sufficient to modify project groups and other object types).\n\nh2(#system). System user and group\n\nA privileged user account exists for the use by internal Arvados components.  This user manages system objects which should not be \"owned\" by any particular user.  The system user uuid is @{siteprefix}-tpzed-000000000000000@.\n\nh2. Anoymous user and group\n\nAn Arvados site may be configured to allow users to browse resources without requiring a login.  In this case, permissions for non-logged-in users are associated with the \"anonymous\" user.  To make objects visible to anyone (both logged-in and non-logged-in users), they can be shared with the \"anonymous\" role.  Note that objects shared with the \"anonymous\" user will only be visible to non-logged-in users!\n\nThe anonymous user uuid is @{siteprefix}-tpzed-anonymouspublic@.  The anonymous group uuid is @{siteprefix}-j7d0g-anonymouspublic@.\n\nh2. Example\n\n!(full-width){{site.baseurl}}/images/Arvados_Permissions.svg!\n"
  },
  {
    "path": "doc/api/projects.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\ntitle: \"Projects and filter groups\"\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nArvados @projects@ are used to organize objects. Projects can contain @collections@, @container requests@, @workflows@, etc. Projects can also contain other projects. An object is part of a project if the @owner_uuid@ of the object is set to the uuid of the project.\n\nProjects are implemented as a subtype of the Arvados @group@ object type, with @group_class@ set to the value \"project\". More information is available in the \"groups API reference\":{{ site.baseurl }}/api/methods/groups.html.\n\nProjects can be manipulated via Workbench, the cli tools, the SDKs, and the Arvados APIs.\n\nh2. The home project\n\nEach user has a @home project@, which is implemented differently. This is a virtual project that is comprised of all objects owned by the user, in other words, all objects with the @owner_uuid@ set to the @uuid@ of the user. The home project is accessible via Workbench, which makes it easy view its contents and to move objects from and to the home project. The home project is also accessible via FUSE, WebDAV and the S3 interface.\n\nThe same thing can be done via the APIs. To put something in a user's home project via the cli or SDKs, one would set the @owner_uuid@ of the object to the user's @uuid@. This also implies that this user now has full ownership and control over that object.\n\nThe contents of the home project can be accessed with the @group contents@ API, e.g. via the cli with this command:\n
arv group contents --uuid zzzzz-tpzed-123456789012345
\nIn this command, `zzzzz-tpzed-123456789012345` is a @user@ uuid, which is unusual because we are using it as the argument to a @groups@ API. The @group contents@ API is normally used with a @group@ uuid.\n\nBecause the home project is a virtual project, other operations via the @groups@ API are not supported.\n\nh2(#filtergroups). Filter groups\n\nFilter groups are another type of virtual project. They are implemented as an Arvados @group@ object with @group_class@ set to the value \"filter\".\n\nFilter groups define one or more filters which are applied to all objects that the current user can see, and returned as the contents of the @group@. Filter groups are described in more detail in the \"groups API reference\":{{site.baseurl}}/api/methods/groups.html, and the rules for creating valid filters are the same as for \"list method filters\":{{site.baseurl}}/api/methods.html#filters.\n\nFilter groups are accessible (read-only) via Workbench and the Arvados FUSE mount, WebDAV and S3 interface. Filter groups must currently be defined via the API, SDK or cli, there is no Workbench support yet.\n\nAs an example, create a filter group with the @arv@ cli:\n\n\n
~$  FILTER_GROUP_UUID=`arv -s group create --group '{\n    \"group_class\":\"filter\",\n    \"name\":\"my filter group\",\n    \"properties\":{\n      \"filters\":\n        [\n          [\"collections.name\",\"ilike\",\"%test%\"],\n          [\"uuid\",\"is_a\",\"arvados#collection\"]\n        ]\n      }\n    }'`\n\n
\n\nThis filter group will contain all collections visible to the current user whose name matches the word @test@ (case insensitive).\n\nTo see how this works via the keep FUSE mount, create a few matching (and non-matching) collections:\n\n\n
~$ arv collection create --collection '{\"name\":\"empty test collection 1\"}'\n~$ arv collection create --collection '{\"name\":\"another empty collection\"}'\n~$ arv collection create --collection '{\"name\":\"empty Test collection 2\"}'\n~$ mkdir -p keep\n~$ arv-mount keep\n~$ ls keep/by_id/$FILTER_GROUP_UUID/ -C1\n'empty test collection 1'\n'empty Test collection 2'\n
\n\n"
  },
  {
    "path": "doc/api/properties.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\ntitle: \"Metadata properties\"\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nArvados allows you to attach arbitrary properties to \"collection\":methods/collections.html, \"container_request\":methods/container_requests.html, \"link\":methods/links.html and \"group\":methods/groups.html records that have a @properties@ field.  These are key-value pairs, where the value is a valid JSON type (string, number, null, boolean, array, object).\n\nSearching for records using properties is described in \"Filtering on subproperties\":methods.html#subpropertyfilters .\n\nh2. Controlling user-supplied properties\n\nArvados can be configured with a vocabulary file that lists valid properties and the range of valid values for those properties.  This is described in \"Metadata vocabulary\":{{site.baseurl}}/admin/metadata-vocabulary.html .\n\nArvados offers options to set properties automatically and/or prevent certain properties, once set, from being changed by non-admin users.  This is described in \"Configuring collection's managed properties\":{{site.baseurl}}/admin/collection-managed-properties.html .\n\nThe admin can require that certain properties must be non-empty before \"freezing a project\":methods/groups.html#frozen .\n\nh2. Reserved properties\n\nComponents that ship with Arvados may automatically set properties on objects. These usually help track provenance or provide additional link metadata. These properties usually have a key that starts with @arv:@, and can always be set even when the system is configured with a strict vocabulary.\n\ntable(table table-bordered table-condensed).\n|_. Property name|_. Appears on|_. Value type|_.Description|\n{% comment %}\nThe arv:git* container properties, and the associated Git commands, primarily come from arvados_cwl.executor.ArvCwlExecutor.get_git_info.\n{% endcomment -%}\n|arv:gitBranch|container request, collection of type=workflow|string|When @arvados-cwl-runner@ is run from a Git checkout, this property is set with the name of the branch checked out (the output of @git rev-parse --abbrev-ref HEAD@)|\n|arv:gitCommitter|container request, collection of type=workflow|string|When @arvados-cwl-runner@ is run from a Git checkout, this property is set with the name and email address of the committer of the most recent commit (the output of @git log --format='%cn <%ce>' -n1 HEAD@)|\n|arv:gitCommit|container request, collection of type=workflow|string|When @arvados-cwl-runner@ is run from a Git checkout, this property is set with the full checksum of the most recent commit (the output of @git log --format='%H' -n1 HEAD@)|\n|arv:gitDate|container request, collection of type=workflow|string|When @arvados-cwl-runner@ is run from a Git checkout, this property is set with the commit date of the most recent commit in RFC 2822 format (the output of @git log --format='%cD' -n1 HEAD@)|\n|arv:gitDescribe|container request, collection of type=workflow|string|When @arvados-cwl-runner@ is run from a Git checkout, this property is set with the name of the most recent tag that is reachable from the most recent commit (the output of @git describe --always --tags@)|\n|arv:gitOrigin|container request, collection of type=workflow|string|When @arvados-cwl-runner@ is run from a Git checkout, this property is set with the URL of the remote named @origin@, if set (the output of @git remote get-url origin@)|\n|arv:gitPath|container request, collection of type=workflow|string|When @arvados-cwl-runner@ is run from a Git checkout, this property is set with the absolute path of the checkout on the filesystem|\n|arv:gitStatus|container request, collection of type=workflow|string|When @arvados-cwl-runner@ is run from a Git checkout, this property is set with a machine-readable summary of files modified in the checkout since the most recent commit (the output of @git status --untracked-files=no --porcelain@)|\n|arv:workflowMain|collection of type=workflow|string|Set on a collection containing a workflow created by @arvados-cwl-runner --create-workflow@, this is a relative reference inside the collection to the entry point of the workflow.|\n|arv:failed_container_resubmitted|container request|uuid|Set on container requests that were automatically resubmitted by the workflow runner with modified run options, such as when using the @PreemptionBehavior@ or @OutOfMemoryRetry@ CWL extensions.  Set to the uuid of the new, resubmitted container request.|\n\nThe following system properties predate the @arv:@ key prefix, but are still reserved and can always be set.\n\ntable(table table-bordered table-condensed).\n|_. Property name|_. Appears on|_. Value type|_.Description|\n|type|collection|string|Appears on collections to indicates the contents or usage. See \"Collection type values\":#collectiontype below for details.|\n|container_request|collection|string|The UUID of the container request that produced an output or log collection.|\n|docker-image-repo-tag|collection|string|For collections containing a Docker image, the repo/name:tag identifier|\n|container_uuid|collection|string|The UUID of the container that produced a collection (set on collections with type=log)|\n|container|collection|string|(legacy) The UUID of the container that produced a collection.  Set on intermediate collections created by arvados-cwl-runner.  Starting with Arvados 2.6.0 arvados-cwl-runner uses @container_uuid@ instead, but older versions may still set the @container@ property.|\n|cwl_input|container_request|object|On an intermediate container request, the CWL workflow-level input parameters used to generate the container request|\n|cwl_output|container_request|object|On an intermediate container request, the CWL workflow-level output parameters collected from the container request|\n|template_uuid|container_request|string|For a workflow runner container request, the workflow record that was used to launch it.|\n|workflowName|container_request|string|For a workflow runner container request, the \"name\" of the workflow record in @template_uuid@ at the time of launch (used for display only).|\n|username|link|string|For a \"can_login\":permission-model.html#links permission link, the unix username on the VM that the user will have.|\n|groups|link|array of string|For a \"can_login\":permission-model.html#links permission link, the unix groups on the VM that the user will be added to.|\n|image_timestamp|link|string|When resolving a Docker image name and multiple links are found with @link_class=docker_image_repo+tag@ and same @link_name@, the @image_timestamp@ is used to determine precedence (most recent wins).|\n|filters|group|array of array of string|Used to define \"filter groups\":projects.html#filtergroup|\n\nh3(#collectiontype). Collection \"type\" values\n\nMeaningful values of the @type@ property.  These are recognized by Workbench when filtering on types of collections from the project content listing.\n\ntable(table table-bordered table-condensed).\n|_. Type|_.Description|\n|log|The collection contains log files from a container run.|\n|output|The collection contains the output of a top-level container run (this is a container request where @requesting_container_uuid@  is null).|\n|intermediate|The collection contains the output of a child container run (this is a container request where @requesting_container_uuid@ is non-empty).|\n|workflow|A collection created by @arvados-cwl-runner --create-workflow@ containing a workflow definition.|\n"
  },
  {
    "path": "doc/api/requests.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\nnavmenu: Concepts\ntitle: REST API syntax\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nArvados exposes a REST API using standard HTTP requests.\n\nh3. HTTP Method\n\nUse @GET@ to request individual resources or lists of resources.\n\nUse @POST@ to create new resources.\n\nUse @PUT@ to update an existing resource.\n\nUse @DELETE@ to remove an existing resource.\n\nAs a special case, a @POST@ with the query parameter @_method=GET@ will be treated as a GET request.  This makes it possible to issue @GET@ requests where the query string exceeds the maximum request URI length, by putting the query string in the body of the request.\n\nh3. Request URI\n\nThe URI portion of the request identifies the specific resource to operate on.  For example, operations on \"collections\":{{site.baseurl}}/api/methods/collections.html use the @https://{{ site.arvados_api_host }}/arvados/v1/collections@ request URI prefix.\n\nh3. Authorization header\n\nEvery request must include an API token.  This identifies the user making the request for the purposes of access control.  In addition, tokens may be further \"restricted in scope\":{{site.baseurl}}/api/methods/api_client_authorizations.html#scope to only access certain API endpoints.\n\nAPI requests must provide the API token using the @Authorization@ header in the following format:\n\n
\n$ curl -v -H \"Authorization: Bearer xxxxapitokenxxxx\" https://192.168.5.2:8000/arvados/v1/collections\n> GET /arvados/v1/collections HTTP/1.1\n> ...\n> Authorization: Bearer xxxxapitokenxxxx\n> ...\n
\n\nOn a cluster configured to use an OpenID Connect provider (other than Google) as a login backend, Arvados can be configured to accept an OpenID Connect access token in place of an Arvados API token. OIDC access tokens are also accepted by a cluster that delegates login to another cluster (LoginCluster) which in turn has this feature configured. See @Login.OpenIDConnect.AcceptAccessTokenScope@ in the \"default config.yml file\":{{site.baseurl}}/admin/config.html for details.\n\n
\n$ curl -v -H \"Authorization: Bearer xxxx-openid-connect-access-token-xxxx\" https://192.168.5.2:8000/arvados/v1/collections\n
\n\nh3. Parameters\n\nRequest parameters may be provided in one of two ways.  They may be provided in the \"query\" section of request URI, or they may be provided in the body of the request with application/x-www-form-urlencoded encoding.  If parameters are provided in both places, their values will be merged.  Parameter names must be unique.  If a parameter appears multiple times, the behavior is undefined.\n\nStructured and nested parameter values must be provided as urlencoded JSON.\n\nh3. Result\n\nResults are returned JSON-encoded in the response body.\n\nh3(#errors). Errors\n\nIf a request cannot be fulfilled, the API will return 4xx or 5xx HTTP status code.  Be aware that the API server may return a 404 (Not Found) status for resources that exist but for which the client does not have read access.  The API will also return an error record:\n\ntable(table table-bordered table-condensed).\n|*Parameter name*|*Value*|*Description*|\n|errors|array|An array of one or more error messages|\n|error_token|string|a unique identifier used to correlate the error in the API server logs|\n\nh2. Examples\n\nh3. Create a new record\n\n
\n$ curl -v -X POST --data-urlencode 'collection={\"name\":\"empty collection\"}' -H \"Authorization: Bearer oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr\" https://192.168.5.2:8000/arvados/v1/collections | jq .\n> POST /arvados/v1/collections HTTP/1.1\n> User-Agent: curl/7.38.0\n> Host: 192.168.5.2:8000\n> Accept: */*\n> Authorization: Bearer oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr\n> Content-Length: 54\n> Content-Type: application/x-www-form-urlencoded\n>\n} [data not shown]\n< HTTP/1.1 200 OK\n< Content-Type: application/json; charset=utf-8\n< Transfer-Encoding: chunked\n< Connection: keep-alive\n< Status: 200 OK\n< Access-Control-Allow-Origin: *\n< Access-Control-Allow-Methods: GET, HEAD, PUT, POST, DELETE\n< Access-Control-Allow-Headers: Authorization\n< Access-Control-Max-Age: 86486400\n< X-UA-Compatible: IE=Edge,chrome=1\n< ETag: \"2ec9ef5151c1f7a1486ad169c33ae462\"\n< Cache-Control: max-age=0, private, must-revalidate\n< Set-Cookie: _server_session=BAh7BkkiD3Nlc3Npb25faWQGOgZFVEkiJTIwMjQ1NTE5YmEwMzU1MGZkMTBmYmY1YzllY2ZiMjFlBjsAVA%3D%3D--653bc9c20899d48ee8523e18d9a4c1cde0702577; path=/; HttpOnly\n< X-Request-Id: 56aa10bc49097f3b44d3ed946bf0e61e\n< X-Runtime: 0.049951\n< X-Powered-By: Phusion Passenger 4.0.41\n< Date: Fri, 28 Oct 2016 19:20:09 GMT\n< Server: nginx/1.4.7 + Phusion Passenger 4.0.41\n<\n{\n  \"kind\": \"arvados#collection\",\n  \"etag\": \"c5ifrv1ox2tu6alb559ymtkb7\",\n  \"uuid\": \"962eh-4zz18-m1ma0mxxfg3mbcc\",\n  \"owner_uuid\": \"962eh-tpzed-000000000000000\",\n  \"created_at\": \"2016-10-28T19:20:09.320771531Z\",\n  \"modified_by_user_uuid\": \"962eh-tpzed-000000000000000\",\n  \"modified_at\": \"2016-10-28T19:20:09.319661000Z\",\n  \"name\": \"empty collection\",\n  \"description\": null,\n  \"properties\": {},\n  \"portable_data_hash\": \"d41d8cd98f00b204e9800998ecf8427e+0\",\n  \"manifest_text\": \"\",\n  \"replication_desired\": null,\n  \"replication_confirmed\": null,\n  \"replication_confirmed_at\": null,\n  \"expires_at\": null\n}\n
\n\nh3. Delete a record\n\n
\n$ curl -X DELETE -v -H \"Authorization: Bearer oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr\" https://192.168.5.2:8000/arvados/v1/collections/962eh-4zz18-m1ma0mxxfg3mbcc | jq .\n> DELETE /arvados/v1/collections/962eh-4zz18-m1ma0mxxfg3mbcc HTTP/1.1\n> User-Agent: curl/7.38.0\n> Host: 192.168.5.2:8000\n> Accept: */*\n> Authorization: Bearer oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr\n>\n< HTTP/1.1 200 OK\n< Content-Type: application/json; charset=utf-8\n< Transfer-Encoding: chunked\n< Connection: keep-alive\n< Status: 200 OK\n< Access-Control-Allow-Origin: *\n< Access-Control-Allow-Methods: GET, HEAD, PUT, POST, DELETE\n< Access-Control-Allow-Headers: Authorization\n< Access-Control-Max-Age: 86486400\n< X-UA-Compatible: IE=Edge,chrome=1\n< ETag: \"1e8f72802cf1a6d0a5c4a1ebbfcc46a9\"\n< Cache-Control: max-age=0, private, must-revalidate\n< Set-Cookie: _server_session=BAh7BkkiD3Nlc3Npb25faWQGOgZFVEkiJTc2NDYyY2M0NTNlNmU3M2Y2M2E3YmFiMWQ1MTEyZGZkBjsAVA%3D%3D--d28c7dd640bd24e2b12f01e77088072138dcf145; path=/; HttpOnly\n< X-Request-Id: e66fd3ab825bdb87301f5456161fb641\n< X-Runtime: 0.028788\n< X-Powered-By: Phusion Passenger 4.0.41\n< Date: Fri, 28 Oct 2016 19:33:31 GMT\n< Server: nginx/1.4.7 + Phusion Passenger 4.0.41\n<\n{\n  \"kind\": \"arvados#collection\",\n  \"etag\": \"c5ifrv1ox2tu6alb559ymtkb7\",\n  \"uuid\": \"962eh-4zz18-m1ma0mxxfg3mbcc\",\n  \"owner_uuid\": \"962eh-tpzed-000000000000000\",\n  \"created_at\": \"2016-10-28T19:20:09.320771000Z\",\n  \"modified_by_user_uuid\": \"962eh-tpzed-000000000000000\",\n  \"modified_at\": \"2016-10-28T19:20:09.319661000Z\",\n  \"name\": \"empty collection\",\n  \"description\": null,\n  \"properties\": {},\n  \"portable_data_hash\": \"d41d8cd98f00b204e9800998ecf8427e+0\",\n  \"manifest_text\": \"\",\n  \"replication_desired\": null,\n  \"replication_confirmed\": null,\n  \"replication_confirmed_at\": null,\n  \"expires_at\": null\n}\n
\n\nh3. Get a specific record\n\n
\n$ curl -v -H \"Authorization: Bearer oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr\" https://192.168.5.2:8000/arvados/v1/collections/962eh-4zz18-xi32mpz2621o8km | jq .\n> GET /arvados/v1/collections/962eh-4zz18-xi32mpz2621o8km HTTP/1.1\n> User-Agent: curl/7.38.0\n> Host: 192.168.5.2:8000\n> Accept: */*\n> Authorization: Bearer oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr\n>\n< HTTP/1.1 200 OK\n< Content-Type: application/json; charset=utf-8\n< Transfer-Encoding: chunked\n< Connection: keep-alive\n< Status: 200 OK\n< Access-Control-Allow-Origin: *\n< Access-Control-Allow-Methods: GET, HEAD, PUT, POST, DELETE\n< Access-Control-Allow-Headers: Authorization\n< Access-Control-Max-Age: 86486400\n< X-UA-Compatible: IE=Edge,chrome=1\n< ETag: \"fec2ddf433a352e5a2b5d356abd6d3d4\"\n< Cache-Control: max-age=0, private, must-revalidate\n< X-Request-Id: 40b447507ff202ae9a0b0b3e0ebe98da\n< X-Runtime: 0.011404\n< X-Powered-By: Phusion Passenger 4.0.41\n< Date: Fri, 28 Oct 2016 18:59:09 GMT\n< Server: nginx/1.4.7 + Phusion Passenger 4.0.41\n<\n{\n  \"kind\": \"arvados#collection\",\n  \"etag\": \"3mmn0s9e1z5s5opfofmtb9k8p\",\n  \"uuid\": \"962eh-4zz18-xi32mpz2621o8km\",\n  \"owner_uuid\": \"962eh-tpzed-000000000000000\",\n  \"created_at\": \"2016-10-27T14:47:43.792587000Z\",\n  \"modified_by_user_uuid\": \"962eh-tpzed-000000000000000\",\n  \"modified_at\": \"2016-10-27T14:47:43.792166000Z\",\n  \"name\": \"Saved at 2016-10-27 14:47:43 UTC by peter@debian\",\n  \"description\": null,\n  \"properties\": {},\n  \"portable_data_hash\": \"93a45073511646a5c3e2f4953fcf6f61+116\",\n  \"manifest_text\": \". eff999f3b5158331eb44a9a93e3b36e1+67108864+Aad3839bea88bce22cbfe71cf4943de7dab3ea52a@5826180f db141bfd11f7da60dce9e5ee85a988b8+34038725+Ae8f48913fed782cbe463e0499ab37697ee06a2f8@5826180f 0:101147589:rna.SRR948778.bam\\n\",\n  \"replication_desired\": null,\n  \"replication_confirmed\": null,\n  \"replication_confirmed_at\": null,\n  \"expires_at\": null\n}\n
\n\nh3. List records and filter by date\n\n(Note, return result is truncated).\n\n
\n$ curl -v -G --data-urlencode 'filters=[[\"created_at\",\">\",\"2016-11-08T21:38:24.124834000Z\"]]' -H \"Authorization: Bearer oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr\" https://192.168.5.2:8000/arvados/v1/collections | jq .\n> GET /arvados/v1/collections?filters=%5B%5B%22uuid%22%2C%20%22%3D%22%2C%20%22962eh-4zz18-xi32mpz2621o8km%22%5D%5D HTTP/1.1\n> User-Agent: curl/7.38.0\n> Host: 192.168.5.2:8000\n> Accept: */*\n> Authorization: Bearer oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr\n>\n< HTTP/1.1 200 OK\n< Content-Type: application/json; charset=utf-8\n< Transfer-Encoding: chunked\n< Connection: keep-alive\n< Status: 200 OK\n< Access-Control-Allow-Origin: *\n< Access-Control-Allow-Methods: GET, HEAD, PUT, POST, DELETE\n< Access-Control-Allow-Headers: Authorization\n< Access-Control-Max-Age: 86486400\n< X-UA-Compatible: IE=Edge,chrome=1\n< ETag: \"76345ef24952f073acc3a0c550241d4e\"\n< Cache-Control: max-age=0, private, must-revalidate\n< X-Request-Id: d34b8ede4ffc707d8ed172dc2f47ff5e\n< X-Runtime: 0.012727\n< X-Powered-By: Phusion Passenger 4.0.41\n< Date: Fri, 28 Oct 2016 19:08:52 GMT\n< Server: nginx/1.4.7 + Phusion Passenger 4.0.41\n<\n{\n  \"kind\": \"arvados#collectionList\",\n  \"etag\": \"\",\n  \"self_link\": \"\",\n  \"offset\": 0,\n  \"limit\": 100,\n  \"items\": [\n    {\n      \"kind\": \"arvados#collection\",\n      \"etag\": \"bvgrrsg63zsenb9wnpnp0nsgl\",\n      \"uuid\": \"962eh-4zz18-ybggo9im899vv60\",\n      \"owner_uuid\": \"962eh-tpzed-000000000000000\",\n      \"created_at\": \"2016-11-08T21:47:36.937106000Z\",\n      \"modified_by_user_uuid\": \"962eh-tpzed-000000000000000\",\n      \"modified_at\": \"2016-11-08T21:47:36.936625000Z\",\n      \"name\": \"Log from cwl-runner job 962eh-8i9sb-45jww0k15fi5ldd\",\n      \"description\": null,\n      \"properties\": {},\n      \"portable_data_hash\": \"a7820b94717eff86229927565fedbd72+85\",\n      \"replication_desired\": null,\n      \"replication_confirmed\": null,\n      \"replication_confirmed_at\": null,\n      \"expires_at\": null\n    },\n   ...\n    {\n      \"kind\": \"arvados#collection\",\n      \"etag\": \"2fa07dx52lux8wa1loehwyrc5\",\n      \"uuid\": \"962eh-4zz18-37i1tfl5de5ild9\",\n      \"owner_uuid\": \"962eh-tpzed-000000000000000\",\n      \"created_at\": \"2016-11-08T21:38:46.717798000Z\",\n      \"modified_by_user_uuid\": \"962eh-tpzed-000000000000000\",\n      \"modified_at\": \"2016-11-08T21:38:46.717409000Z\",\n      \"name\": null,\n      \"description\": null,\n      \"properties\": {},\n      \"portable_data_hash\": \"9d43d4c8328640446f6e252cda584e7e+54\",\n      \"replication_desired\": null,\n      \"replication_confirmed\": null,\n      \"replication_confirmed_at\": null,\n      \"expires_at\": null\n    }\n  ],\n  \"items_available\": 99\n}\n
\n\nh3. Update a field\n\n
\n$ curl -v -X PUT --data-urlencode 'collection={\"name\":\"rna.SRR948778.bam\"}' -H \"Authorization: Bearer oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr\" https://192.168.5.2:8000/arvados/v1/collections/962eh-4zz18-xi32mpz2621o8km | jq .\n> PUT /arvados/v1/collections/962eh-4zz18-xi32mpz2621o8km HTTP/1.1\n> User-Agent: curl/7.38.0\n> Host: 192.168.5.2:8000\n> Accept: */*\n> Authorization: Bearer oz0os4nyudswvglxhdlnrgnuelxptmj7qu7dpwvyz3g9ocqtr\n> Content-Length: 53\n> Content-Type: application/x-www-form-urlencoded\n>\n} [data not shown]\n< HTTP/1.1 200 OK\n< Content-Type: application/json; charset=utf-8\n< Transfer-Encoding: chunked\n< Connection: keep-alive\n< Status: 200 OK\n< Access-Control-Allow-Origin: *\n< Access-Control-Allow-Methods: GET, HEAD, PUT, POST, DELETE\n< Access-Control-Allow-Headers: Authorization\n< Access-Control-Max-Age: 86486400\n< X-UA-Compatible: IE=Edge,chrome=1\n< ETag: \"fbb50d2847426eab793e3fcf346ca9eb\"\n< Cache-Control: max-age=0, private, must-revalidate\n< Set-Cookie: _server_session=BAh7BkkiD3Nlc3Npb25faWQGOgZFVEkiJWI3NjFjMzVjMGI5OGExYmNjZDg0ZTg5MjZhMzcwMDE1BjsAVA%3D%3D--0e005d71fad15cb366e47361c38474b7447ba155; path=/; HttpOnly\n< X-Request-Id: 76d3cb3c0995af6133b0a73a64f57354\n< X-Runtime: 0.030756\n< X-Powered-By: Phusion Passenger 4.0.41\n< Date: Fri, 28 Oct 2016 19:15:16 GMT\n< Server: nginx/1.4.7 + Phusion Passenger 4.0.41\n<\n{\n  \"kind\": \"arvados#collection\",\n  \"etag\": \"51509hhxo9qqjxqewnoz1b7og\",\n  \"uuid\": \"962eh-4zz18-xi32mpz2621o8km\",\n  \"owner_uuid\": \"962eh-tpzed-000000000000000\",\n  \"created_at\": \"2016-10-27T14:47:43.792587000Z\",\n  \"modified_by_user_uuid\": \"962eh-tpzed-000000000000000\",\n  \"modified_at\": \"2016-10-28T19:15:16.137814000Z\",\n  \"name\": \"rna.SRR948778.bam\",\n  \"description\": null,\n  \"properties\": {},\n  \"portable_data_hash\": \"93a45073511646a5c3e2f4953fcf6f61+116\",\n  \"manifest_text\": \". eff999f3b5158331eb44a9a93e3b36e1+67108864+Acca57af82cc18c5dfa47bdfd16e335fccd09dfa5@582618c4 db141bfd11f7da60dce9e5ee85a988b8+34038725+A7764f122f41f92c2d5bde1852fcdd1bea5f8bd78@582618c4 0:101147589:rna.SRR948778.bam\\n\",\n  \"replication_desired\": null,\n  \"replication_confirmed\": null,\n  \"replication_confirmed_at\": null,\n  \"expires_at\": null\n}\n
\n"
  },
  {
    "path": "doc/api/resources.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\nnavmenu: Concepts\ntitle: Common resource fields\n\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThis page describes the common attributes shared by most or all Arvados resources.\n\nh2(#resource). Resource\n\ntable(table table-bordered table-condensed).\n|_. Attribute |_. Type |_. Description |_. Example|\n|uuid|string|universally unique object identifier.  Set on @create@.|@mk2qn-4zz18-w3anr2hk2wgfpuo@|\n|owner_uuid|string|UUID of owner (must be a User or Group), set on @create@.  Controls who may access the resource. Ownership may be changed explicitly with @update@, see \"permission model\":{{site.baseurl}}/api/permission-model.html for details.|@mk2qn-tpzed-a4lcehql0dv2u25@|\n|name|string|Human-assigned name.  Not present on all object types, check individual API page.  Uniqueness constraint varys by object type.||\n|description|string|Free text description of the object.  Not present on all object types, check individual API page.  May be HTML formatted, \"see below for valid HTML tags and attributes\":#descriptions .||\n|created_at|datetime|When resource was created.  Set on @create@.|@2013-01-21T22:17:39Z@|\n|modified_at|datetime|When resource was last modified.  Set on @create@ and @update@.|@2013-01-25T22:29:32Z@|\n|modified_by_user_uuid|string|The owner of the API token used to authenticate the @create@ or @update@ request.|@mk2qn-tpzed-a4lcehql0dv2u25@|\n|kind|string|@arvados#{resource_type}@|@arvados#collection@|\n|etag|string|The ETag[1] of the resource|@1xlmizzjq7wro3dlb2dirf505@|\n\nh2. Object UUID\n\nEach object is assigned a UUID.  This has the format @aaaaa-bbbbb-ccccccccccccccc@.\n\n# The first field (@aaaaa@ in the example) is the site prefix.  This is unique to a specific Arvados installation.\n# The second field (@bbbbb@ in the example) is the object type.\n# The third field (@ccccccccccccccc@ in the example) uniquely identifies the object.\n\nh2(#descriptions). Descriptions\n\n{% include 'html_tags' %}\n\nh2. Timestamps\n\nAll Arvados timestamps follow ISO 8601 datetime format with fractional seconds (microsecond precision).  All timestamps are UTC.  Date format: @YYYY-mm-ddTHH:MM:SS.SSSSZ@ example date: @2016-11-08T21:38:24.124834000Z@.\n\nh2. ETags\n\nfn1. Each response includes an ETag, a string which changes when the resource changes.  Clients can use this to check whether a resource has changed since they last retrieved it.  If a previous ETag is provided along with a request, and the resource has not changed since, the server may return a \"not modified\" response.\n"
  },
  {
    "path": "doc/api/tokens.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: api\ntitle: API Authorization\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nAll requests to the API server must have an API token.  API tokens can be issued by going though the login flow, or created via the API.  At this time, only browser based applications can perform login from email/password.  Command line applications and services must use an API token provided via the @ARVADOS_API_TOKEN@ environment variable or configuration file.\n\nh2. Login\n\nBrowser based applications can log in using one of the two possible flows:\n\nh3. Authenticate via a third party\n\n# The web application instructs the user to click on a link to the @/login@ endpoint on the API server.  This link should include the @return_to@ parameter in the query portion of the URL.  For example @https://{{ site.arvados_api_host }}/login?return_to=XXX@ where @return_to=XXX@ is a page in the web application.\n# The @/login@ endpoint redirects the user to the configured third party authentication provider (e.g. Google or other OpenID Connect provider).\n# The user logs in to the third party provider, then they are redirected back to the API server.\n# The API server authenticates the user, issues a new API token, and redirects the browser to the URL provided in @return_to=XXX@ with the addition of @?api_token=xxxxapitokenxxxx@.\n# The web application gets the authorization token from the query and uses it to access the API server on the user's behalf.\n\nh3. Direct username/password authentication\n\n# The web application presents username and password fields.\n# When the submit button is pressed, using Javascript, the browser sends a POST request to @/arvados/v1/users/authenticate@\n** The request payload type is @application/javascript@\n** The request body is a JSON object with @username@ and @password@ fields.\n# The API server receives the username and password, authenticates them with the upstream provider (such as LDAP or PAM), and responds with the @api_client_authorization@ object for the new API token.\n# The web application receives the authorization token in the response and uses it to access the API server on the user's behalf.\n\nh3. Using an OpenID Connect access token\n\nA cluster that uses OpenID Connect as a login provider can be configured to accept OIDC access tokens as well as Arvados API tokens (this is disabled by default; see @Login.OpenIDConnect.AcceptAccessToken@ in the \"default config.yml file\":{{site.baseurl}}/admin/config.html).\n# The client obtains an access token from the OpenID Connect provider via some method outside of Arvados.\n# The client presents the access token with an Arvados API request (e.g., request header @Authorization: Bearer xxxxaccesstokenxxxx@).\n# Depending on configuration, the API server decodes the access token (which must be a signed JWT) and confirms that it includes the required scope (see @Login.OpenIDConnect.AcceptAccessTokenScope@ in the \"default config.yml file\":{{site.baseurl}}/admin/config.html).\n# The API server uses the provider's UserInfo endpoint to validate the presented token.\n# If the token is valid, it is cached in the Arvados database and accepted in subsequent API calls for the next 10 minutes.\n\nh3. Diagram\n\n!{{site.baseurl}}/images/Session_Establishment.svg!\n\nh2. User activation\n\n\"Creation and activation of new users is described here.\":{{site.baseurl}}/admin/user-management.html\n\nh2. Creating tokens via the API\n\nThe browser login method above issues a new token.  Using that token, it is possible to make API calls to create additional tokens.  To do so, use the @create@ method of the \"API client authorizations\":{{site.baseurl}}/api/methods/api_client_authorizations.html resource.\n\nh2(#scopes). Scopes\n\nScopes can restrict a token so it may only access certain resources.  This is in addition to normal permission checks for the user associated with the token.\n\nEach entry in scopes consists of a @request_method@ and @request_path@.  The @request_method@ is a HTTP method (one of @GET@, @POST@, @PATCH@ or @DELETE@) and @request_path@ is the request URI.  A given request is permitted if it matches a scopes exactly, or the scope ends with @/@ and the request string is a prefix of the scope.\n\nAs a special case, a scope of @[\"all\"]@ allows all resources.  This is the default if no scope is given.\n\nA valid token is always allowed to issue a request to \"@GET /arvados/v1/api_client_authorizations/current@\":{{ site.baseurl }}/api/methods/api_client_authorizations.html#current regardless of its scopes.\n\nUsing scopes is also described on the \"Securing API access with scoped tokens\":{{site.baseurl}}/admin/scoped-tokens.html page of the admin documentation.\n\nh3. Scope examples\n\nA scope of @GET /arvados/v1/collections@ permits listing collections.\n\n* Requests with different methods, such as creating a new collection using @POST /arvados/v1/collections@, will be rejected.\n* Requests to access other resources, such as @GET /arvados/v1/groups@, will be rejected (except \"@GET /arvados/v1/api_client_authorizations/current@\":{{ site.baseurl }}/api/methods/api_client_authorizations.html#current, which is always allowed).\n* Be aware that requests for specific records, such as @GET /arvados/v1/collections/962eh-4zz18-xi32mpz2621o8km@ will also be rejected.  This is because the scope @GET /arvados/v1/collections@ does not end in @/@\n\nA scope of @GET /arvados/v1/collections/@ (with @/@ suffix) will permit access to individual collections.\n\n* The request @GET /arvados/v1/collections/962eh-4zz18-xi32mpz2621o8km@ will succeed\n* Be aware that requests for listing @GET /arvados/v1/collections@ (no @/@ suffix) will be rejected, because it is not a match with the rule @GET /arvados/v1/collections/@\n* A listing request @GET /arvados/v1/collections/@ will have the trailing @/@ suffix trimmed before the scope check, as a result it will not match the rule @GET /arvados/v1/collections/@.\n\nTo allow both listing objects and requesting individual objects, include both in the scope: @[\"GET /arvados/v1/collections\", \"GET /arvados/v1/collections/\"]@\n\nA narrow scope such as @GET /arvados/v1/collections/962eh-4zz18-xi32mpz2621o8km@ will disallow listing objects as well as disallow requesting any object other than those listed in the scope.\n"
  },
  {
    "path": "doc/architecture/dispatchcloud.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: architecture\ntitle: Dispatching containers to cloud VMs\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThe arvados-dispatch-cloud component runs Arvados user containers on generic public cloud infrastructure by automatically creating and destroying VMs (“instances”) of various sizes according to demand, preparing the instances’ runtime environments, and running containers on them.\n\nThis does not use a cloud provider’s container-execution service.\n\nh2. Overview\n\nIn this diagram, the black edges show interactions involved in starting a VM instance and running a container. The blue edges show the “container shell” communication channel.\n\n!{max-width:40em}{{site.baseurl}}/architecture/dispatchcloud.svg!\n\n{% comment %}\n# svg generated using https://dreampuf.github.io/\ndigraph {\n    subgraph cluster_cloudvm {\n        node [color=black] [fillcolor=white] [style=filled];\n        style = filled;\n        color = lightgrey;\n        label = \"cloud instance (VM)\";\n        \"SSH server\" -> \"crunch-run\" [label = \"start crunch-run\"];\n        \"crunch-run\" -> docker [label = \"create container\"];\n        \"crunch-run\" -> docker [label = \"shell\"] [color = blue] [fontcolor = blue];\n        \"crunch-run\" -> container [label = \"tcp/http\"] [color = blue] [fontcolor = blue];\n        docker -> container;\n    }\n    \"cloud provider\" [shape=box] [style=dashed];\n    dispatcher -> controller [label = \"get container queue\"];\n    dispatcher -> \"cloud provider\" [label = \"create/destroy/list VMs\"];\n    \"cloud provider\" -> \"SSH server\" [label = \"add authorized_keys\"];\n    \"crunch-run\" -> controller [label = \"update\\ngateway ip:port,\\ncontainer state,\\noutput, ...\"];\n    client -> controller [label = \"shell/tcp/http (https tunnel)\"] [color = blue] [fontcolor = blue];\n    controller -> \"crunch-run\" [label = \"shell/tcp/http (https tunnel)\"] [color = blue] [fontcolor = blue];\n    dispatcher -> \"SSH server\" [label = \"start crunch-run\"];\n}\n{% endcomment %}\n\nh2. Scheduling\n\nThe dispatcher periodically polls the \"containers API\":{{site.baseurl}}/api/methods/containers.html to get a list of containers that are ready to run. Whenever this list changes, the dispatcher runs a scheduling loop that determines the set of suitable instance types for each container, allocates the highest priority containers to instances with sufficient unused resources, requests new instances if needed, and shuts down instances that have been idle for longer than the configured idle timeout. It will run multiple containers on an instance if it has enough RAM and CPUs to do so _and_ the instance type is suitable for each container individually.\n\nThe lowest-priced instance type with enough resources to run a given container is always suitable. Other instance types that have enough resources to run the container, and whose prices are within @MaximumPriceFactor@ of that lowest-priced type, are also suitable. The dispatcher will select a suitable instance type other than the lowest-priced one when:\n* the lowest-priced instance that is _already running or requested,_ and has sufficient resources, is one of the suitable types (_e.g.,_ it just finished running a container that needed a higher-priced type), whereas in order to use the lowest-priced type the dispatcher would need to request a new instance, or\n* the cloud provider indicates that the lowest-priced suitable type is not available (_e.g.,_ due to a per-instance-type quota restriction).\n\nh2. Creating instances\n\nWhen creating a new instance, the dispatcher uses the cloud provider’s metadata feature to add a tag with key “InstanceSetID” and a value derived from its Arvados authentication token. This enables the dispatcher to recognize and reconnect to existing instances that belong to it, and continue monitoring existing containers, after a restart or upgrade.\n\nWhen using the Azure cloud service, the dispatcher needs to first create a new network interface, then attach it to a new instance. The network interface is also tagged with “InstanceSetID”.\n\nIf the cloud provider returns a rate-limiting error when creating a new instance, the dispatcher avoids requesting new instances for a short period, and shuts down idle nodes more aggressively (i.e., without waiting for the usual idle timeout to elapse) until a new instance is successfully created.\n\nh2. Recovering state after a restart\n\nRestarting the dispatcher does not interrupt containers that are already running. When the dispatcher starts up, it gets the cloud provider’s current list of instances that have the expected InstanceSetID tag value. It ignores instances without that tag, so it won’t interfere with other VM instances in the same cloud account. It runs the boot probe command on each instance, checks for containers that were started by a previous invocation and are still running, and resumes monitoring. Before dispatching any new containers to a previously existing instance, it ensures the crunch-run program is updated if needed.\n\nh2. Instance boot process\n\nWhen the cloud provider indicates that a new instance has been created, the dispatcher connects to the instance’s SSH service (see “instance control channel” below) and executes the configured boot probe command. If this fails, the dispatcher retries until the configured boot timeout is reached, then shuts down the instance. When the boot probe succeeds, the dispatcher copies the crunch-run program to the instance, and runs it to check for running containers before reporting the instance’s state as “idle” or “busy”. (Normally of course a freshly booted instance has no containers running, but this covers the case where the dispatcher itself has restarted and containers submitted by the previous dispatcher process are still running.)\n\nThe dispatcher and crunch-run programs are both packaged in a single executable file: when dispatcher copies crunch-run to an instance, it is really copying itself. This ensures the dispatcher is always using the version of crunch-run that it expects.\n\nh2. Boot probe command\n\nThe purpose of the boot probe command is to ensure the dispatcher does not try to schedule containers on an instance before the instance is ready, even if its SSH daemon comes up early in the boot process. The default boot probe command, @systemctl is-system-running@, is appropriate for images that use @systemd@ to manage the boot process. Another approach is to use a custom startup script in the VM image that writes a file when it finishes, and a boot probe command that checks for that file, such as @cat /var/run/boot.complete@.\n\nh2. Automatic instance shutdown\n\nNormally, the dispatcher shuts down any instance that has remained idle for 1 minute (see TimeoutIdle configuration) but there are some exceptions to this rule. If the cloud provider returns a quota error when trying to create a new instance, the dispatcher shuts down idle nodes right away, in case the idle nodes are contributing to the quota. Also, the operator can use the management API to set an instance’s idle behavior to “drain” or “hold”. “Drain” shuts down the instance as soon as it becomes idle, which can be used to recycle a suspect node without interrupting a running container. “Hold” keeps the instance alive indefinitely without scheduling additional containers on it, which can be used to investigate problems like a failed startup script.\n\nEach instance is tagged with its current idle behavior (using the tag name “IdleBehavior”), which makes it visible in the cloud provider’s console and ensures the behavior is retained if dispatcher restarts.\n\nh2. Management API\n\nThe dispatcher provides an HTTP management interface, which provides the operator with more visibility and control for purposes of troubleshooting and monitoring. APIs are provided to return details of current VM instances and running/scheduled containers as seen by the dispatcher, immediately terminate containers and instances, and control the on-idle behavior of instances. This interface also provides Prometheus metrics. See the \"cloud dispatcher management API\":{{site.baseurl}}/api/dispatch.html documentation for details.\n\nh2. Instance control channel (SSH)\n\nThe dispatcher uses a multiplexed SSH connection to monitor instance boot progress, install the crunch-run supervisor program, start and stop containers, and detect crashed containers and failing instances. It establishes a persistent SSH connection to each cloud instance when the instance first appears, retrying/reconnecting as needed.\n\nCloud VMs typically generate a random SSH host key at boot time, making host key verification impossible. To provide some assurance the dispatcher is connecting to the intended instance, when it creates a new instance the dispatcher generates a random “instance secret”, uses the cloud provider’s bootstrap command feature to save it in @/var/run/arvados-instance-secret@ on the new instance, and executes @cat /var/run/arvados-instance-secret@ to verify the instance’ identity when first connecting to its SSH server. Each instance is also tagged with its instance secret, so it can still be verified after a dispatcher restart.\n\nh2. Container communication channel (https tunnel)\n\nThe crunch-run program runs a gateway server which facilitates the “container shell” feature without sending traffic through the dispatcher process. The gateway server accepts TLS connections from arvados-controller on a dynamic TCP port (typically in the range 32768-60999, see @sysctl net.ipv4.ip_local_port_range@). Crunch-run saves the selected port, along with the external IP address of the VM instance as seen by the dispatcher, in the @gateway_address@ field in the container record so arvados-controller can connect to it.\n\nOn the client host (typically a shell node or a user’s workstation) the @arvados-client shell@ command sends an https “connect” request to arvados-controller, which sends an https “connect” request to the gateway server. These tunnels convey SSH protocol traffic between the user’s SSH client and crunch-run’s built-in SSH server, which uses @docker exec@ to run commands inside the container.\n\nArvados-controller and crunch-run gateway server authenticate each other using a self-signed certificate and a shared secret based on the cluster-wide @SystemRootToken@. If that token changes (and the dispatcher restarts to load the new token) while a container is running, the container will stop accepting container shell traffic.\n\nh2. Scaling\n\nArchitecturally, the dispatcher is _designed_ to accommodate multiple concurrent dispatcher processes on multiple hosts, each using a different authorization token, but such a configuration is not yet supported. Currently, each cluster should run a single dispatcher process. A single process can support thousands of concurrent VM instances.\n"
  },
  {
    "path": "doc/architecture/federation.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: architecture\ntitle: \"Federation\"\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nArvados federation enables clients to transparently read, create and manipulate objects and collections across clusters in different regions or organizations.  Federation supports workfows that integrate and analyze data across multiple clusters by sending computation to where the data is, reducing the technical and legal barriers to analyzing large, sensitive data sets.\n\n_This feature is under development.  Support for federation is limited to certain types of requests.  The behaviors described here should not be interpreted as a stable API._\n\nDetailed configuration information is available on the \"federation admin section\":{{site.baseurl}}/admin/federation.html.\n\nh2(#cluster_id). Cluster identifiers\n\nClusters are identified by a five-digit alphanumeric id (numbers and lowercase letters).  There are 36 ^5^ = 60466176 possible cluster identifiers.\n\n* For automated test purposes, use \"z****\"\n* For experimental/local-only/private clusters that won't ever be visible on the public Internet, use \"x****\"\n* For long-lived clusters, we recommend reserving a cluster id.  Contact \"info@curii.com\":mailto:info@curii.com for more information.\n\nCluster identifiers are mapped API server hosts one of two ways:\n\n* Through DNS resolution, under the @arvadosapi.com@ domain.  For example, the API server for the cluster @pirca@ can be found at @pirca.arvadosapi.com@.  To register a cluster id for free under @arvadosapi.com@, contact \"info@curii.com\":mailto:info@curii.com\n* Through explicit configuration:\n\nThe @RemoteClusters@ section of @/etc/arvados/config.yml@ (for arvados-controller)\n\n
\nClusters:\n  clsr1:\n    RemoteClusters:\n      clsr2:\n        Host: api.cluster2.example\n        Proxy: true\n      clsr3:\n        Host: api.cluster3.example\n        Proxy: true\n
\n\nIn this example, the cluster @clsr1@ is configured to contact @api.cluster2.example@ for requests involving @clsr2@ and @api.cluster3.example@ for requests involving @clsr3@.\n\nh2(#identity). Identity\n\nThe goal is for a federated user to have a single identity across the cluster federation.  This identity is a user account on a specific \"home cluster\".  When arvados-controller contacts a remote cluster, the remote cluster verifies the user's identity (see below) and then creates a mirror of the user account with the same uuid of the user's home cluster.  On the remote cluster, permissions can then be granted to the federated user, and the federated user can create and own objects.\n\nh3. Peer federation: Authenticating remote users with salted tokens\n\nWhen making a request to the home cluster, authorization is established by looking up the API token in the @api_client_authorizations@ table to determine the user identity.  When making a request to a remote cluster, we need to provide an API token which can be used to establish the user's identity.  The remote cluster will connect back to the home cluster to determine if the token valid and the user it corresponds to.  However, we do not want to send along the same API token used for the original request.  If the remote cluster is malicious or compromised, sending along user's regular token would compromise the user account on the home cluster.  Instead, the controller sends a \"salted token\".  The salted token is restricted to only to fetching the user account and group membership.  The salted token consists of the uuid of the token in @api_client_authorizations@ and the SHA1 HMAC of the original token and the cluster id of remote cluster.  To verify the token, the remote cluster contacts the home cluster and provides the token uuid, the hash, and its cluster id.  The home cluster uses the uuid to look up the token re-computes the SHA1 HMAC of the original token and cluster id.  If that hash matches, then the token is valid.  To avoid having to re-validate the token on every request, it is cached for a short period.\n\nThe security properties of this scheme are:\n\n* The salted token does not grant access on the home cluster beyond what is needed to verify user identity\n* Revoking a token on the home cluster also revokes it for remote clusters (after the cache period)\n* A salted token given to a malicious/compromised cluster cannot be used to gain access to the user account on another remote cluster\n\nh3. LoginCluster federation: Centralized user database\n\nIn a LoginCluster federation, there is a central \"home\" called the LoginCluster, and one or more \"satellite\" clusters.  The satellite clusters delegate their user management to the LoginCluster.  Unlike the peer federation, satellite clusters implicitly trust the home cluster, so the \"salted token\" scheme is not used.  Users arriving at a satellite cluster are redirected to the home cluster for login, the user token is issued by the LoginCluster, and then the user is sent back to the satellite cluster.   Tokens issued by the LoginCluster are accepted by all clusters in the federation.  All requests for user records on a satellite cluster is forwarded to the LoginCluster.\n\nh2(#retrieval). Federated records\n\n!(full-width){{site.baseurl}}/images/arvados_federation.svg!\n\nh3. Retrieving and updating records\n\nIn the REST API, GET and PUT/PATCH requests are used to fetch and update records.\n\n# the client begins by making a request to the home arvados-controller to retrieve or update a specific record owned by a remote cluster\n# arvados-controller determines the 5-digit cluster id from the first part of the uuid string\n# arvados-controller determines the API server host corresponding to the cluster id\n# arvados-controller creates a \"salted\" token by combining the API token used for the request and the target cluster id\n# arvados-controller contacts the remote cluster to request the desired record, providing the salted token\n# the remote cluster verifies the salted token\n# the remote cluster processes the request and returns a response\n# arvados-controller forwards the response to the client\n\nh3. Creating records\n\nIn the REST API, POST requests create new records, so there is no uuid to use for the cluster id.  In this case, to create an object on a remote cluster, the request includes the @cluster_id@ parameter.  The flow is otherwise the same as described above.\n\nh3. Collections and Keep block retrieval\n\nEach collection record has @manifest_text@, which describes how to reassemble keep blocks into files as described in the \"Manifest format\":{{site.baseurl}}/architecture/manifest-format.html.  Each block identifier in the manifest has an added signature which is used to confirm permission to read the block.  To read a block from a keepstore server, the client must provide the block identifier, the signature, and the same API token used to retrieve the collection record.\n\nSee \"Federation signatures\":{{site.baseurl}}/architecture/manifest-format.html#federationsignatures for details on how federation affects block signatures.\n"
  },
  {
    "path": "doc/architecture/hpc.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: architecture\ntitle: Dispatching containers to HPC\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nArvados can be configured to run containers on an HPC cluster using Slurm or LSF, as an alternative to \"dispatching to cloud VMs\":dispatchcloud.html.\n\nIn this configuration, the appropriate Arvados dispatcher service -- @crunch-dispatch-slurm@ or @arvados-dispatch-lsf@ -- picks up each container as it appears in the Arvados queue and submits a short shell script as a batch job to the HPC job queue. The shell script executes the @crunch-run@ container supervisor which retrieves the container specification from the Arvados controller, starts an arv-mount process, runs the container using @docker exec@ or @singularity exec@, and sends updates (logs, outputs, exit code, etc.) back to the Arvados controller.\n\nh2. Container communication channel (reverse https tunnel)\n\nThe crunch-run program runs a gateway server to facilitate the “container shell” feature. However, depending on the site's network topology, the Arvados controller may not be able to connect directly to the compute node where a given crunch-run process is running.\n\nInstead, in the HPC configuration, crunch-run connects to the Arvados controller at startup and sets up a multiplexed tunnel, allowing the controller process to connect to crunch-run's gateway server without initiating a connection to the compute node, or even knowing the compute node's IP address.\n\nThis means that when a client requests a container shell connection, the traffic goes through two or three servers:\n# The client connects to a controller host C1.\n# If the multiplexed tunnel is connected to a different controller host C2, then C1 proxies the incoming request to C2, using C2's InternalURL.\n# The controller host (C1 or C2) uses the multiplexed tunnel to connect to crunch-run's container gateway.\n\nh2. Scaling\n\nThe @API.MaxConcurrentRequests@ configuration should not be set too low, or the long-lived tunnel connections can starve other clients.\n"
  },
  {
    "path": "doc/architecture/index.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: architecture\ntitle: \"Arvados components\"\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n# \"Services\":#Services\n# \"Arvados-server\":#Arvados-server\n# \"SDK\":#SDK\n# \"Tools\":#Tools\n# \"Arvados-client\":#Arvados-client\n\n!(full-width){{site.baseurl}}/images/Arvados_arch.svg!\n\nh3(#Services). Services\n\nLocated in @arvados/services@.  Many services have been incorporated into @arvados-server@, see below.\n\ntable(table table-bordered table-condensed).\n|_. Component|_. Description|\n|api|Along with Controller, the API server is the core of Arvados.  It is backed by a Postgres database and manages information such as metadata for storage, a record of submitted compute jobs, users, groups, and associated permissions.|\n|crunch-dispatch-local|Get compute requests submitted to the API server and execute them locally.|\n|dockercleaner|Daemon for cleaning up Docker containers and images.|\n|fuse|Filesystem in Userspace (FUSE) enabling users to mount Keep collections as a filesystem.|\n|login-sync|Synchronize virtual machine users with Arvados users and permissions.|\n|workbench2|Web application providing user interface to Arvados services.|\n\nh3(#Arvados-server). Arvados-server\n\nLocated in @cmd/arvados-server@.  It consists of a single @arvados-server@ binary with a number of different subcommands.  Although the binary itself is monolithic, subcommands are each a standalone service and only handle requests for that specific service, i.e. a @arvados-server controller@ process will not respond to requests intended for a @arvados-server keep-web@.\n\ntable(table table-bordered table-condensed).\n|_. Subcommand|_. Description |\n|boot|Boot an Arvados cluster from source, used by automated testing.|\n|check|Contact the a health check endpoint on services and print a report.|\n|cloudtest|Diagnostic tool which attempts to start a cloud instance using the current settings in the config file.|\n|config-check|Check that the config file is valid.|\n|config-defaults|Dump the default config options.|\n|config-dump|Dump the active config options that would be used by the other @arvados-server@ commands.|\n|controller|Controller works with the API server to make up the core of Arvados.  It intercepts requests and implements additional features such as federation.|\n|crunch-run|Dispatched by crunch-dispatch, executes a single compute run: setting up a Docker container, running it, and collecting the output.|\n|crunchstat|Run a program and collect resource usage stats using cgroups.|\n|dispatch-cloud|Get compute requests submitted to the API server and schedule them on elastic cloud compute, creating and destroying cloud based virtual machines on demand.|\n|dispatch-lsf|Get compute requests submitted to the API server and submit them to LSF HPC scheduler.|\n|dispatch-slurm|Get compute requests submitted to the API server and submit them to SLURM HPC scheduler.|\n|health|Service that aggregates the other health check results to provide a single cluster-wide health status.|\n|install|Install development dependencies to be able to build and run Arvados from source.|\n|init|Create an initial configuration file for a new cluster and perform database setup.|\n|keep-balance|Perform storage utilization reporting, optimization and garbage collection.  Moves data blocks to their optimum location, ensures correct replication and storage class, and trashes unreferenced blocks.|\n|keep-web|Provides high-level to files in collections as either a WebDAV or S3-compatible API endpoint.|\n|keepproxy|Provides low-level access to keepstore services (block-level data access) for clients outside the internal (private) network.|\n|keepstore|Provides access to underlying storage (filesystem or object storage such as Amazon S3 or Azure Blob) with Arvados permissions.|\n|recover-collection|Recovers deleted collections. Recovery is possible when the collection's manifest is still available and all of its data blocks are still available or recoverable.|\n|workbench2|Serve the HTML/Javascript for the single-page Workbench application.|\n|ws|Publishes API server change events over websockets.|\n\nh3(#SDK). SDK\n\nThe @arv@ command is located in @arvados/sdk/ruby@, the @arv-*@ tools are located in @arvados/sdk/python@.\n\ntable(table table-bordered table-condensed).\n|_. Component|_. Description |\n|arv|Provides command line access to API, also provides some purpose utilities.|\n|arv-copy|Copy a collection from one cluster to another|\n|arv-get|Get files from a collection.|\n|arv-keepdocker|Upload Docker images from local Docker daemon to Keep.|\n|arv-ls|List files in a collection|\n|arv-put|Upload files to a collection.|\n|arv-ws|Print events from Arvados websocket event source.|\n\nh3(#Tools). Tools\n\nLocated in @arvados/tools@.\n\ntable(table table-bordered table-condensed).\n|_. Component|_. Description |\n|cluster-activity|Generate a HTML and/or CSV report of cluster activity over a time period.|\n|crunchstat-summary|Read execution metrics (cpu %, ram, network, etc) collected from a compute container and produce a report.|\n|keep-block-check|Given a list of keep block locators, check that each block exists on one of the configured keepstore servers and verify the block hash.|\n|keep-exercise|Benchmarking tool to test throughput and reliability of keepstores under various usage patterns.|\n|keep-rsync|Get lists of blocks from two clusters, copy blocks which exist on source cluster but are missing from destination cluster.|\n|sync-groups|Takes a CSV file listing with rows in the form (group, user, permission) records and synchronize membership in Arvados groups.|\n|sync-users|Takes a CSV file listing with rows in the form (email, first name, last name, active, admin) and synchronize Arvados users.|\n|user-activity|Generate a text report of user activity over a time period.|\n\nh3(#Arvados-client). Arvados-client\n\nLocated in @cmd/arvados-client@.  It consists of a single @arvados-client@ binary with a number of different subcommands.\n\ntable(table table-bordered table-condensed).\n|_. Subcommand|_. Description |\n|connect-ssh|Connects stdin/stdout to a container's gateway server. It is intended to be invoked with OpenSSH client's ProxyCommand config.|\n|deduplication-report|Analyzes the overlap in blocks used by 2 or more collections. It prints a deduplication report that shows the nominal space used by the collections, as well as the actual size and the amount of space that is saved by Keep's deduplication.|\n|diagnostics|Perform cluster diagnostics to check that all the services are available and responding normally to requests.|\n|logs|Prints live streaming logs for a container.|\n|mount|Alternate Keep FUSE mount written in Go.|\n|shell|Connects the terminal to an interactive shell on a running container.|\n|sudo|Runs another command using API connection info and SystemRootToken from the system config file instead of the caller's environment vars.|\n"
  },
  {
    "path": "doc/architecture/keep-clients.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: architecture\ntitle: Keep clients\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nKeep clients are applications such as @arv-get@, @arv-put@ and @arv-mount@ which store and retrieve data from Keep.  In doing so, these programs interact with both the API server (which stores file metadata in the form of @collection@ objects) and individual @keepstore@ servers (which store the actual data blocks).\n\n!(full-width){{site.baseurl}}/images/Keep_reading_writing_block.svg!\n\nh2. Storing a file\n\n# The client discovers keep servers (or proxies) using the @accessible@ method on \"keep_services\":{{site.baseurl}}/api/methods/keep_services.html\n# Data is split into 64 MiB blocks and the MD5 hash is computed for each block.\n# The client uploads each block to one or more Keep servers, based on the number of desired replicas.  The priority order is determined using rendezvous hashing, described below.\n# The Keep server returns a block locator (the MD5 sum of the block) and a \"signed token\" which the client can use as proof of knowledge for the block.\n# The client constructs a @manifest@ which lists the blocks by MD5 hash and how to reassemble them into the original files.\n# The client creates a \"collection\":{{site.baseurl}}/api/methods/collections.html and provides the @manifest_text@\n# The API server accepts the collection after validating the signed tokens (proof of knowledge) for each block.\n\nh2. Fetching a file\n\n# The client requests a @collection@ object including @manifest_text@ from the APIs server\n# The server adds \"token signatures\" to the @manifest_text@ and returns it to the client.\n# The client discovers keep servers (or proxies) using the @accessible@ method on \"keep_services\":{{site.baseurl}}/api/methods/keep_services.html\n# For each data block, the client chooses the highest priority server using rendezvous hashing, described below.\n# The client sends the data block request to the keep server, along with the token signature from the API which proves to Keep servers that the client is permitted to read a given block.\n# The server provides the block data after validating the token signature for the block (if the server does not have the block, it returns a 404 and the client tries the next highest priority server)\n\nh2(#rendezvous). Rendezvous hashing\n!(full-width){{site.baseurl}}/images/Keep_rendezvous_hashing.svg!\n\nEach @keep_service@ resource has an assigned uuid.  To determine priority assignments of blocks to servers, for each keep service compute the MD5 sum of the string concatenation of the block locator (hex-coded hash part only) and service uuid, then sort this list in descending order.  Blocks are preferentially placed on servers with the highest weight.\n\n"
  },
  {
    "path": "doc/architecture/keep-components-overview.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: architecture\ntitle: Keep components overview\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nKeep has a number of components. This page describes each component and the role it plays.\n\nh3. Keep clients for data access\n\nIn order to access data in Keep, a client is needed to store data in and retrieve data from Keep. Different types of Keep clients exist:\n* a command line client like \"@arv-get@\":{{ site.baseurl }}/user/tutorials/tutorial-keep-get.html#download-using-arv or \"@arv-put@\":{{ site.baseurl }}/user/tutorials/tutorial-keep.html#upload-using-command\n* a FUSE mount provided by \"@arv-mount@\":{{ site.baseurl }}/user/tutorials/tutorial-keep-mount-gnu-linux.html\n* a WebDAV mount provided by @keep-web@\n* an S3-compatible endpoint provided by @keep-web@\n* programmatic access via the \"Arvados SDKs\":{{ site.baseurl }}/sdk/index.html\n\nIn essense, these clients all do the same thing: they translate file and directory references into requests for Keep blocks and collection manifests. How Keep clients work, and how they use rendezvous hashing, is described in greater detail in \"the next section\":{{ site.baseurl }}/architecture/keep-clients.html.\n\nFor example, when a request comes in to read a file from Keep, the client will\n* request the collection object (including its manifest) from the API server\n* look up the file in the collection manifest, and retrieve the hashes of the block(s) that contain its content\n* ask the keepstore(s) for the block hashes\n* return the contents of the file to the requestor\n\nAll of those steps are subject to access control, which applies at the level of the collection: in the example above, the API server and the keepstore daemons verify that the client has permission to read the collection, and will reject the request if it does not.\n\nh3. API server\n\nThe API server stores collection objects and all associated metadata. That includes data about where the blocks for a collection are to be stored, e.g. when \"storage classes\":{{ site.baseurl }}/admin/storage-classes.html are configured, as well as the desired and confirmed replication count for each block. It also stores the ACLs that control access to the collections. Finally, the API server provides Keep clients with time-based block signatures for access.\n\nh3. Keepstore\n\nThe @keepstore@ daemon is Keep's workhorse, the storage server that stores and retrieves data from an underlying storage system. Keepstore exposes an HTTP REST API. Keepstore only handles requests for blocks. Because blocks are content-addressed, they can be written and deleted, but there is no _update_ operation: blocks are immutable.\n\nSo what happens if the content of a file changes? When a client changes a file, it first writes any new blocks to the keepstore(s). Then, it updates the manifest for the collection the file belongs to with the references to the new blocks.\n\nA keepstore can store its blocks in object storage (S3 or an S3-compatible system, or Azure Blob Storage). It can also store blocks on a POSIX file system. A keepstore can be configured with multiple storage volumes. Each keepstore volume is configured with a replication number; e.g. a POSIX file system backed by a single disk would have a replication factor of 1, while an Azure 'LRS'  storage volume could be configured with a replication factor of 3 (that is how many copies LRS stores under the hood, according to the Azure documentation).\n\nBy default, Arvados uses a replication factor of 2. See the @DefaultReplication@ configuration parameter in \"the configuration reference\":https://doc.arvados.org/admin/config.html. Additionally, each collection can be configured with its own replication factor. It's worth noting that it is the responsibility of the Keep clients to make sure that all blocks are stored subject to their desired replica count, which is derived from the collections the blocks belong to. @keepstore@ itself does not provide replication; all it does is store blocks on the volumes it knows about. The @keepproxy@ and @keep-balance@ processes (see below) make sure that blocks are replicated properly.\n\nThe maximum block size for @keepstore@ is 64 MiB, and keep clients typically combine small files into larger blocks. In a typical Arvados installation, the majority of blocks stored in Keep will be 64 MiB, though some fraction will be smaller.\n\nh3. Keepproxy\n\nThe @keepproxy@ server is a gateway into your Keep storage. Unlike the Keepstore servers, which are only accessible on the local LAN, Keepproxy is suitable for clients located elsewhere on the internet. A client writing through Keepproxy only writes one copy of each block; the Keepproxy server will write additional copies of the data to the Keepstore servers, to fulfill the requested replication factor. Keepproxy also checks API token validity before processing requests.\n\nh3. Keep-web\n\nThe @keep-web@ server provides read/write access to files stored in Keep using the HTTP, WebDAV and S3 protocols. This makes it easy to access files in Keep from a browser, or mount Keep as a network folder using WebDAV support in various operating systems. It serves public data to unauthenticated clients, and serves private data to clients that supply Arvados API tokens.\n\nh3. Keep-balance\n\nKeep is a garbage-collected system. When a block is no longer referenced in any collection manifest in the system, it becomes eligible for garbage collection. When the desired replication factor for a block (derived from the default replication factor, in addition to the replication factor of any collection(s) the block belongs to) does not match reality, the number of copies stored in the available Keepstore servers needs to be adjusted.\n\nThe @keep-balance@ program takes care of these things. It runs as a service, and wakes up periodically to do a scan of the system and send instructions to the Keepstore servers. That process is described in more detail at \"Balancing Keep servers\":https://doc.arvados.org/admin/keep-balance.html.\n"
  },
  {
    "path": "doc/architecture/keep-data-lifecycle.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: architecture\ntitle: \"Data lifecycle\"\n...\n\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nh2(#overview). Overview\n\nArvados collections consist of a \"manifest\":{{site.baseurl}}/architecture/manifest-format.html and the data blocks referenced in that manifest. Manifests are stored in the PosgreSQL database, @data blocks@ are stored by a @keepstore@.\n\nData blocks are frequently shared between collections. Each collection has its own @manifest@. Collection manifests and data blocks have a separate lifecycle, which is described in detail below.\n\nh2(#collection_lifecycle). Collection lifecycle\n\nDuring its lifetime, a collection can be in various states. These states are *persisted*, *expiring*, *trashed*  and *permanently deleted*.\n\nThe nominal state is *persisted* which means the data can be can be accessed normally and will be retained indefinitely.\n\nA collection is *expiring* when it has a *trash_at* time in the future. An expiring collection can be accessed as normal, but is scheduled to be trashed automatically at the *trash_at* time.\n\nA collection is *trashed* when it has a *trash_at* time in the past. The *is_trashed* attribute will also be \"true\". The delete operation immediately puts the collection in the trash by setting the *trash_at* time to \"now\", and *delete_at* defaults to \"now\" + @Collections.DefaultTrashLifetime@. Once trashed, the collection is no longer readable through normal data access APIs. The collection will have *delete_at* set to some time in the future. The trashed collection is recoverable until the *delete_at* time passes, at which point the collection is permanently deleted.\n\nSee \"Recovering trashed collections\":{{ site.baseurl }}/user/tutorials/tutorial-keep-collection-lifecycle.html#trash-recovery for instructions to recover trashed collections.\n\nh3(#collection_attributes). Collection lifecycle attributes\n\nAs listed above the attributes that are used to manage a collection lifecycle are *is_trashed*, *trash_at*, and *delete_at*. The table below lists the values of these attributes and how they influence the state of a collection and its accessibility.\n\ntable(table table-bordered table-condensed).\n|_. collection state|_. is_trashed|_. trash_at|_. delete_at|_. get|_. list|_. list?include_trash=true|_. can be modified|\n|persisted collection|false |null |null |yes |yes |yes |yes |\n|expiring collection|false |future |future |yes  |yes |yes |yes |\n|trashed collection|true |past |future |no |no |yes |only is_trashed, trash_at and delete_at attributes|\n|deleted collection|true|past |past |no |no |no |no |\n\nh2(#block_lifecycle). Block lifecycle\n\nDuring its lifetime, a data block can be in various states. These states are *persisted*, *unreferenced*, *trashed* and *permanently deleted*.\n\nThe nominal state is *persisted* which means the block can be can be retrieved normally from a @keepstore@ process.\n\nA block is *unreferenced* when there are no collection manifests in the PostgreSQL collections table that reference it. The block can still be retrieved normally from a @keepstore@ process, e.g. by creating a new collection with a manifest that references the hash of the block. Unreferenced blocks will be moved to the *trashed* state by @keep-balance@ after @BlobSigningTTL@, if @BlobTrash@ is enabled and @keep-balance@ is running and configured to send trash lists to the keepstores.\n\nA block is *trashed* when @keep-balance@ has asked a @keepstore@ to move it to its trash and @BlobTrash@ is enabled. It will stay there for a period of time, subject to the @BlobTrashLifetime@ settings.\n\nA block is *permanently deleted* on the first wakeup of its @keepstore@ trash process after the block has spent @BlobTrashLifetime@ in that keepstore's trash. The trash process wakes up with a frequency defined by the @BlobTrashCheckInterval@.\n\ntable(table table-bordered table-condensed).\n|_. block state|_. duration|_. retrievable via Keep|_. can be recovered|\n|persisted block|indefinitely|yes |n/a |\n|unreferenced block|@BlobSigningTTL@ + up to @BalancePeriod@ + duration of keep-balance run|yes |n/a |\n|trashed block|@BlobTrashLifetime@ + up to @BlobTrashCheckInterval@|no |yes |\n|deleted block||no |no |\n"
  },
  {
    "path": "doc/architecture/manifest-format.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: architecture\ntitle: Manifest format\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nEach collection record has a @manifest_text@ field, which describes how to reassemble keep blocks into files. Each block identifier in the manifest has an added signature which is used to confirm permission to read the block.  To read a block from a keepstore server, the client must provide the block identifier, the signature, and the same API token used to retrieve the collection record.\n\n!(full-width){{site.baseurl}}/images/Keep_manifests.svg!\n\nh2. Manifest v1\n\nA manifest is utf-8 encoded text, consisting of zero or more newline-terminated streams.\n\n
\nmanifest       ::= stream*\nstream         ::= stream-name (\" \" locator)+ (\" \" file-segment)+ \"\\n\"\nstream-name    ::= \".\" (\"/\" path-component)*\npath-component ::= +\nfile-segment   ::= position \":\" size \":\" filename\nposition       ::= [0-9]+\nsize           ::= [0-9]+\nfilename       ::= path-component (\"/\" path-component)*\n
\n\nNotes:\n\n* The first token is the stream name, consisting of one or more path components, delimited by @\"/\"@.\n** The first path component is always @\".\"@.\n** No path component is empty.\n** No path component following the first one can be \".\" or \"..\".\n** The stream name never begins or ends with @\"/\"@.\n* The next N tokens are \"keep locators\":#locator\n** These describe the \"data stream\".  By logically concatenating the blocks in the order that they appear, we can refer to \"positions\" in the data stream.\n* File tokens come after the sequence of keep locators.\n** A file token has three parts, delimited by @\":\"@: position, size, filename.\n** Position and size are given in decimal\n** The position is the position in the data stream\n** The size is the count of bytes following the position in the data stream.  A file size may cross multiple blocks in the data stream.\n** Filename may contain @\"/\"@ characters, but must not start or end with @\"/\"@, and must not contain @\"//\"@.\n** Filename components (delimited by @\"/\"@) must not be @\".\"@ or @\"..\"@.\n** There may be multiple file tokens.\n\nIt is legal to have multiple file tokens in the manifest (possible across different streams) with the same combined path name @stream name + \"/\" + filename@.  This must be interpreted as a concatenation of file content, in the order that the file tokens appear in the manifest.\n\nSpaces are represented by the escape sequence @\\040@.  Spaces in stream names and filenames must be translated when reading and writing manifests.  A manifest may not contain TAB characters, nor other ASCII whitespace characters or control codes other than the spaces or newlines used as delimiters specified above.  A manifest always ends with a newline -- except the empty (zero-length) string, which is a valid manifest.\n\nh3. Normalized manifest v1\n\nA normalized manifest is a manifest that meets the following additional restrictions:\n\n* Streams are in alphanumeric order.\n* Each stream name is unique within the manifest.\n* Files within a stream are listed in alphanumeric order.\n* Blocks within a stream are ordered based on order of file tokens of the stream.  A given block is listed at most once in a stream.\n* Filename must not contain @\"/\"@ (the stream name represents the path prefix)\n\nh3. Estimating manifest size\n\nHere's a formula for estimating manifest size as stored in the database, assuming efficiently packed blocks.\n\n
\nmanifest_size =\n   + (total data size / 64 MB) * 40\n   + sum(number of files * 20)\n   + sum(size of all directory paths)\n   + sum(size of all file names)\n
\n\nHere is the size when including block signatures.  The block signatures authorize access to fetch each block from a Keep server, as described below.  The signed manifest text is what is actually transferred to/from the API server and stored in RAM by @arv-mount@.  The effective upper limit on how large a collection manifest can be is determined by @API.MaxRequestSize@ in @config.yml@ as well as the maximum request size configuration in your reverse proxy or load balancer (e.g. @client_max_body_size@ in Nginx).\n\n
\nmanifest_size =\n   + (total data size / 64 MB) * 94\n   + sum(number of files * 20)\n   + sum(size of all directory paths)\n   + sum(size of all file names)\n
\n\nh3. Example manifests\n\nA manifest with four files in two directories:\n\n
\n. 930625b054ce894ac40596c3f5a0d947+33 0:0:a 0:0:b 0:33:output.txt\n./c d41d8cd98f00b204e9800998ecf8427e+0 0:0:d\n
\n\nThe same manifest with permission signatures on each block:\n\n
\n. 930625b054ce894ac40596c3f5a0d947+33+A1f27a35dd9af37191d63ad8eb8985624451e7b79@5835c8bc 0:0:a 0:0:b 0:33:output.txt\n./c d41d8cd98f00b204e9800998ecf8427e+0+A27117dcd30c013a6e85d6d74c9a50179a1446efa@5835c8bc 0:0:d\n
\n\nA manifest containing a file consisting of multiple blocks and a space in the file name:\n\n
\n. c449ed86671e4a34a8b8b9430850beba+67108864 09fcfea01c3a141b89dd0dcfa1b7768e+22534144 0:89643008:Docker\\040image.tar\n
\nh2(#locator). Keep locator format\n\nBNF notation for a valid Keep locator string (with hints).  For example: *d41d8cd98f00b204e9800998ecf8427e+0+Z+Ada39a3ee5e6b4b0d3255bfef95601890afd80709@53bed294*\n\n
\nlocator          ::= sized-digest hint*\nsized-digest     ::= digest size-hint\ndigest           ::= <32 lowercase hexadecimal digits>\nsize-hint        ::= \"+\" [0-9]+\nhint             ::= \"+\" hint-type hint-content\nhint-type        ::= [A-Z]+\nhint-content     ::= [A-Za-z0-9@_-]*\nsign-hint        ::= \"+A\" <40 lowercase hexadecimal digits> \"@\" sign-timestamp\nremote-sign-hint ::= \"+R\" [A-Za-z0-9]{5} \"-\" <40 lowercase hexadecimal digits> \"@\" sign-timestamp\nsign-timestamp   ::= <8 lowercase hexadecimal digits>\n
\n\nh3. Regular expression to validate locator\n\n
\n/^([0-9a-f]{32})\\+([0-9]+)(\\+[A-Z][-A-Za-z0-9@_]*)*$/\n
\n\nh3. Valid locators\n\ntable(table table-bordered table-condensed).\n|@d41d8cd98f00b204e9800998ecf8427e+0@|\n|@d41d8cd98f00b204e9800998ecf8427e+0+Z@|\n|d41d8cd98f00b204e9800998ecf8427e+0+Z+Ada39a3ee5e6b4b0d3255bfef95601890afd80709@53bed294|\n|930625b054ce894ac40596c3f5a0d947+33+Rzzzzz-1f27a35dd9af37191d63ad8eb8985624451e7b79@5835c8bc|\n\nh3. Invalid locators\n\ntable(table table-bordered table-condensed).\n||Why|\n|@d41d8cd98f00b204e9800998ecf8427e@|No size hint|\n|@d41d8cd98f00b204e9800998ecf8427e+Z+0@|Other hint before size hint|\n|@d41d8cd98f00b204e9800998ecf8427e+0+0@|Multiple size hints|\n|@d41d8cd98f00b204e9800998ecf8427e+0+z@|Hint does not start with uppercase letter|\n|@d41d8cd98f00b204e9800998ecf8427e+0+Zfoo*bar@|Hint contains invalid character @*@|\n\nh3(#token_signatures). Token signatures\n\nA token signature (sign-hint) provides proof-of-access for a data block.  It is computed by taking a SHA1 HMAC of the blob signing token (a shared secret between the API server and keep servers), block digest, current API token, expiration timestamp, and blob signature TTL.\n\nWhen communicating with the @keepstore@ to fetch a block, or the API server to create or update a collection, the service computes the expected token signature for each block and compares it to the token signature that was presented by the client.  Keep clients receive valid block signatures when uploading a block to a keep store (getting back a signed token as proof of knowledge) or, from the API server, getting the manifest text of a collection on which the user has read permission.\n\nSecurity of a token signature is derived from the following characteristics:\n\n# Valid signatures can only be generated by entities that know the shared secret (the \"blob signing token\")\n# A signature can only be used by an entity that also know the API token that was used to generate it.\n# It expires after a set date (the expiration time, based on the \"blob signature time-to-live (TTL)\")\n\nh3(#federationsignatures). Federation and signatures\n\nWhen a collection record is returned through a federation request, the keep blocks listed in the manifest may not be available on the local cluster, and the keep block signatures returned by the remote cluster are not valid for the local cluster.  To solve this, @arvados-controller@ rewrites the signatures in the manifest to \"remote cluster\" signatures.\n\nA local signature comes after the block identifier and block size, and starts with @+A@:\n\n930625b054ce894ac40596c3f5a0d947+33+A1f27a35dd9af37191d63ad8eb8985624451e7b79@5835c8bc\n\nA remote cluster signature starts with @+R@, then the cluster id of the cluster it originated from (@zzzzz@ in this example), a dash, and then the original signature:\n\n930625b054ce894ac40596c3f5a0d947+33+Rzzzzz-1f27a35dd9af37191d63ad8eb8985624451e7b79@5835c8bc\n\nWhen the client provides a remote-signed block locator to keepstore, the keepstore proxies the request to the remote cluster.\n\n# keepstore determines the cluster id to contact from the first part of the @+R@ signature\n# creates a salted token using the API token and cluster id\n# contacts the \"accessible\" endpoint on the remote cluster to determine the remote cluster's keepstore or keepproxy hosts\n# converts the remote signature @+R@ back to a local signature @+A@\n# contacts the remote keepstore or keepproxy host and requests the block using the local signature\n# returns the block contents back to the client\n\nh3(#example). Example\n\nThis example uses @c1bad4b39ca5a924e481008009d94e32+210@, which is the content hash of a @collection@ that was added to Keep in \"how to upload data\":{{ site.baseurl }}/user/tutorials/tutorial-keep.html.  Get the collection manifest using @arv-get@:\n\n\n
~$ arv-get c1bad4b39ca5a924e481008009d94e32+210\n. 204e43b8a1185621ca55a94839582e6f+67108864+Aasignatureforthisblockaaaaaaaaaaaaaaaaaa@5f612ee6 b9677abbac956bd3e86b1deb28dfac03+67108864+Aasignatureforthisblockbbbbbbbbbbbbbbbbbb@5f612ee6 fc15aff2a762b13f521baf042140acec+67108864+Aasignatureforthisblockcccccccccccccccccc@5f612ee6 323d2a3ce20370c4ca1d3462a344f8fd+25885655+Aasignatureforthisblockdddddddddddddddddd@5f612ee6 0:227212247:var-GS000016015-ASM.tsv.bz2\n
\n\n\nThis collection includes a single file @var-GS000016015-ASM.tsv.bz2@ which is 227212247 bytes long. It is stored using four sequential data blocks with hashes @204e43b8a1185621ca55a94839582e6f+67108864@, @b9677abbac956bd3e86b1deb28dfac03+67108864@, @fc15aff2a762b13f521baf042140acec+67108864@, and @323d2a3ce20370c4ca1d3462a344f8fd+25885655@. Each of the block hashes is followed by the rest of their \"locator\":#locator.\n\nUse @arv-get@ to download the first data block:\n\nnotextile. 
~$ arv-get 204e43b8a1185621ca55a94839582e6f+67108864+Aasignatureforthisblockaaaaaaaaaaaaaaaaaa@5f612ee6 > block1
\n\nInspect the size and compute the MD5 hash of @block1@:\n\n\n
~$ ls -l block1\n-rw-r--r-- 1 you group 67108864 Dec  9 20:14 block1\n~$ md5sum block1\n204e43b8a1185621ca55a94839582e6f  block1\n
\n\n\nAs expected, the md5sum of the contents of the block matches the @digest@ part of the \"locator\":#locator, and the size of the contents matches the @size-hint@.\n"
  },
  {
    "path": "doc/architecture/singularity.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: architecture\ntitle: Singularity\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nArvados can be configured to use \"Singularity\":https://sylabs.io/singularity/ instead of Docker to execute containers on cloud nodes or a Slurm/LSF cluster. Singularity may be preferable due to its simpler installation and lack of long-running daemon process and special system users/groups. For on premises Slurm/LSF clusters, see the \"Set up a compute node with Singularity\":{{ site.baseurl }}/install/crunch2/install-compute-node-singularity.html page. For cloud compute clusters, see the \"Build a cloud compute node image\":{{ site.baseurl }}/install/crunch2-cloud/install-compute-node.html page.\n\nh2. Design overview\n\nWhen Arvados is configured to use Singularity as the runtime engine for Crunch, containers are executed by Singularity. The images specified in workflows and tool definitions must be Docker images uploaded via @arv-keepdocker@ or @arvados-cwl-runner@. When Singularity is the runtime engine, these images are converted to Singularity format (@.sif@) at runtime, as needed.\n\nTo avoid repeating this conversion work unnecessarily, the @.sif@ files are cached in @Keep@. This is done on a per-user basis. If it does not exist yet, a new Arvados project named @.cache@ is automatically created in the user's home project. Similarly, a subproject named @auto-generated singularity images@ will be created in the @.cache@ project. The automatically generated @.sif@ files are stored in collections in that project, with an expiration date two weeks in the future. If the cached image exists when Crunch runs a new container, the expiration date will be pushed out, so that it is always 2 weeks in the future from the most recent start of a container using the image.\n\nIt is safe to empty out or even remove the .cache project or any of its contents; if necessary the cache projects and the @.sif@ files will automatically be regenerated.\n\nh2. Notes\n\n* Programs running in Singularity containers may behave differently than when run in Docker, due to differences between Singularity and Docker. For example, the root (image) filesystem is read-only in a Singularity container. Programs that attempt to write outside a designated output or temporary directory are likely to fail.\n\n* When using Singularity as the runtime engine, the compute node needs to have a compatible Singularity executable installed, as well as the @mksquashfs@ program used to convert Docker images to Singularity's @.sif@ format. The Arvados \"compute node image build script\":{{ site.baseurl }}/install/crunch2-cloud/install-compute-node.html includes these executables since Arvados 2.3.0.\n\nh2. Limitations\n\nArvados @Singularity@ support is a work in progress. These are the current limitations of the implementation:\n\n* Even when using the Singularity runtime, users' container images are expected to be saved in Docker format. Specifying a @.sif@ file as an image when submitting a container request is not yet supported.\n* Arvados' Singularity implementation does not yet limit the amount of memory available in a container. Each container will have access to all memory on the host where it runs, unless memory use is restricted by Slurm/LSF.\n* The Docker ENTRYPOINT instruction is ignored.\n* Arvados is tested with Singularity version 3.10.4. Other versions may not work.\n"
  },
  {
    "path": "doc/architecture/storage.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: architecture\ntitle: Introduction to Keep\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nKeep is a content-addressable storage system that yields high performance for I/O-bound workloads. Keep is designed to run on low-cost commodity hardware or cloud services and is tightly integrated with the rest of the Arvados system. It provides high fault tolerance and high aggregate performance to a large number of clients.\n\nh2. Design goals and core features\n\n* *Scale* - Keep installations are managing petabytes of data today. Keep scales horizontally.\n\n* *Data deduplication* - Keep automatically deduplicates data through its use of content addressing.\n\n* *Flexibility* - Keep can store data in S3, S3-compatible storage systems (e.g. Ceph) and Azure blob storage. Keep can also store data on POSIX file systems.\n\n* *Fault-Tolerance* - Errors and failure are expected. Keep has redundancy and recovery capabilities at its core.\n\n* *Optimized for Aggregate Throughput* - Like S3 and Azure blob storage, Keep is optimized for aggregate throughput. This is optimal in a scenario with many reader/writer processes.\n\n* *Complex Data Management* - Keep operates well in environments where there are many independent users accessing the same data or users who want to organize data in many different ways. Keep facilitates data sharing without expecting users either to agree with one another about directory structures or to create redundant copies of the data.\n\n* *Security* - Keep works well combined with encryption at rest and transport encryption. All data is managed through @collection@ objects, which implement a rich \"permission model\":{{site.baseurl}}/api/permission-model.html.\n\nh2. How Keep works\n\nKeep is a content-addressable file system.  This means that files are managed using special unique identifiers derived from the _contents_ of the file (specifically, the MD5 hash), rather than human-assigned file names.  This has a number of advantages:\n* Files can be stored and replicated across a cluster of servers without requiring a central name server.\n* Both the server and client systematically validate data integrity because the checksum is built into the identifier.\n* Data duplication is minimized—two files with the same contents will have in the same identifier, and will not be stored twice.\n* It avoids data race conditions, since an identifier always points to the same data.\n\nIn Keep, information is stored in @data blocks@.  Data blocks are normally between 1 byte and 64 megabytes in size.  If a file exceeds the maximum size of a single data block, the file will be split across multiple data blocks until the entire file can be stored.  These data blocks may be stored and replicated across multiple disks, servers, or clusters.  Each data block has its own identifier for the contents of that specific data block.\n\nIn order to reassemble the file, Keep stores a @collection@ manifest which lists in sequence the data blocks that make up the original file.  A @manifest@ may store the information for multiple files, including a directory structure. See \"manifest format\":{{site.baseurl}}/architecture/manifest-format.html for more information on how manifests are structured.\n"
  },
  {
    "path": "doc/css/R.css",
    "content": "body {\n    background: white;\n    color: black;\n}\n\na:link {\n    background: white;\n    color: blue;\n}\n\na:visited {\n    background: white;\n    color: rgb(50%, 0%, 50%);\n}\n\nh1 {\n    background: white;\n    color: rgb(55%, 55%, 55%);\n    font-family: monospace;\n    font-size: x-large;\n    text-align: center;\n}\n\nh2 {\n    background: white;\n    color: rgb(40%, 40%, 40%);\n    font-family: monospace;\n    font-size: large;\n    text-align: center;\n}\n\nh3 {\n    background: white;\n    color: rgb(40%, 40%, 40%);\n    font-family: monospace;\n    font-size: large;\n}\n\nh4 {\n    background: white;\n    color: rgb(40%, 40%, 40%);\n    font-family: monospace;\n    font-style: italic;\n    font-size: large;\n}\n\nh5 {\n    background: white;\n    color: rgb(40%, 40%, 40%);\n    font-family: monospace;\n}\n\nh6 {\n    background: white;\n    color: rgb(40%, 40%, 40%);\n    font-family: monospace;\n    font-style: italic;\n}\n\t\t\nimg.toplogo {\n    width: 4em;\n    vertical-align: middle;\n}\n\nimg.arrow {\n    width: 30px;\n    height: 30px;\n    border: 0;\n}\n\nspan.acronym {\n    font-size: small;\n}\n\nspan.env {\n    font-family: monospace;\n}\n\nspan.file {\n    font-family: monospace;\n}\n\nspan.option{\n    font-family: monospace;\n}\n\nspan.pkg {\n    font-weight: bold;\n}\n\nspan.samp{\n    font-family: monospace;\n}\n\ndiv.vignettes a:hover {\n    background: rgb(85%, 85%, 85%);\n}\n"
  },
  {
    "path": "doc/css/badges.css",
    "content": "/* Copyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0 */\n\n/* Colors\n * Contextual variations of badges\n * Bootstrap 3.0 removed contexts for badges, we re-introduce them, based on what is done for labels\n */\n\n.badge.badge-error {\n  background-color: #b94a48;\n}\n\n.badge.badge-warning {\n  background-color: #f89406;\n}\n\n.badge.badge-success {\n  background-color: #468847;\n}\n\n.badge.badge-info {\n  background-color: #3a87ad;\n}\n\n.badge.badge-inverse {\n  background-color: #333333;\n}\n\n.badge.badge-alert {\n    background: red;\n}\n"
  },
  {
    "path": "doc/css/bootstrap-theme.css",
    "content": "/*!\n * Bootstrap v3.1.0 (http://getbootstrap.com)\n * Copyright 2011-2014 Twitter, Inc.\n * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)\n */\n\n.btn-default,\n.btn-primary,\n.btn-success,\n.btn-info,\n.btn-warning,\n.btn-danger {\n  text-shadow: 0 -1px 0 rgba(0, 0, 0, .2);\n  -webkit-box-shadow: inset 0 1px 0 rgba(255, 255, 255, .15), 0 1px 1px rgba(0, 0, 0, .075);\n          box-shadow: inset 0 1px 0 rgba(255, 255, 255, .15), 0 1px 1px rgba(0, 0, 0, .075);\n}\n.btn-default:active,\n.btn-primary:active,\n.btn-success:active,\n.btn-info:active,\n.btn-warning:active,\n.btn-danger:active,\n.btn-default.active,\n.btn-primary.active,\n.btn-success.active,\n.btn-info.active,\n.btn-warning.active,\n.btn-danger.active {\n  -webkit-box-shadow: inset 0 3px 5px rgba(0, 0, 0, .125);\n          box-shadow: inset 0 3px 5px rgba(0, 0, 0, .125);\n}\n.btn:active,\n.btn.active {\n  background-image: none;\n}\n.btn-default {\n  text-shadow: 0 1px 0 #fff;\n  background-image: -webkit-linear-gradient(top, #fff 0%, #e0e0e0 100%);\n  background-image:         linear-gradient(to bottom, #fff 0%, #e0e0e0 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#ffffffff', endColorstr='#ffe0e0e0', GradientType=0);\n  filter: progid:DXImageTransform.Microsoft.gradient(enabled = false);\n  background-repeat: repeat-x;\n  border-color: #dbdbdb;\n  border-color: #ccc;\n}\n.btn-default:hover,\n.btn-default:focus {\n  background-color: #e0e0e0;\n  background-position: 0 -15px;\n}\n.btn-default:active,\n.btn-default.active {\n  background-color: #e0e0e0;\n  border-color: #dbdbdb;\n}\n.btn-primary {\n  background-image: -webkit-linear-gradient(top, #428bca 0%, #2d6ca2 100%);\n  background-image:         linear-gradient(to bottom, #428bca 0%, #2d6ca2 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#ff428bca', endColorstr='#ff2d6ca2', GradientType=0);\n  filter: progid:DXImageTransform.Microsoft.gradient(enabled = false);\n  background-repeat: repeat-x;\n  border-color: #2b669a;\n}\n.btn-primary:hover,\n.btn-primary:focus {\n  background-color: #2d6ca2;\n  background-position: 0 -15px;\n}\n.btn-primary:active,\n.btn-primary.active {\n  background-color: #2d6ca2;\n  border-color: #2b669a;\n}\n.btn-success {\n  background-image: -webkit-linear-gradient(top, #5cb85c 0%, #419641 100%);\n  background-image:         linear-gradient(to bottom, #5cb85c 0%, #419641 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#ff5cb85c', endColorstr='#ff419641', GradientType=0);\n  filter: progid:DXImageTransform.Microsoft.gradient(enabled = false);\n  background-repeat: repeat-x;\n  border-color: #3e8f3e;\n}\n.btn-success:hover,\n.btn-success:focus {\n  background-color: #419641;\n  background-position: 0 -15px;\n}\n.btn-success:active,\n.btn-success.active {\n  background-color: #419641;\n  border-color: #3e8f3e;\n}\n.btn-info {\n  background-image: -webkit-linear-gradient(top, #5bc0de 0%, #2aabd2 100%);\n  background-image:         linear-gradient(to bottom, #5bc0de 0%, #2aabd2 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#ff5bc0de', endColorstr='#ff2aabd2', GradientType=0);\n  filter: progid:DXImageTransform.Microsoft.gradient(enabled = false);\n  background-repeat: repeat-x;\n  border-color: #28a4c9;\n}\n.btn-info:hover,\n.btn-info:focus {\n  background-color: #2aabd2;\n  background-position: 0 -15px;\n}\n.btn-info:active,\n.btn-info.active {\n  background-color: #2aabd2;\n  border-color: #28a4c9;\n}\n.btn-warning {\n  background-image: -webkit-linear-gradient(top, #f0ad4e 0%, #eb9316 100%);\n  background-image:         linear-gradient(to bottom, #f0ad4e 0%, #eb9316 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#fff0ad4e', endColorstr='#ffeb9316', GradientType=0);\n  filter: progid:DXImageTransform.Microsoft.gradient(enabled = false);\n  background-repeat: repeat-x;\n  border-color: #e38d13;\n}\n.btn-warning:hover,\n.btn-warning:focus {\n  background-color: #eb9316;\n  background-position: 0 -15px;\n}\n.btn-warning:active,\n.btn-warning.active {\n  background-color: #eb9316;\n  border-color: #e38d13;\n}\n.btn-danger {\n  background-image: -webkit-linear-gradient(top, #d9534f 0%, #c12e2a 100%);\n  background-image:         linear-gradient(to bottom, #d9534f 0%, #c12e2a 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#ffd9534f', endColorstr='#ffc12e2a', GradientType=0);\n  filter: progid:DXImageTransform.Microsoft.gradient(enabled = false);\n  background-repeat: repeat-x;\n  border-color: #b92c28;\n}\n.btn-danger:hover,\n.btn-danger:focus {\n  background-color: #c12e2a;\n  background-position: 0 -15px;\n}\n.btn-danger:active,\n.btn-danger.active {\n  background-color: #c12e2a;\n  border-color: #b92c28;\n}\n.thumbnail,\n.img-thumbnail {\n  -webkit-box-shadow: 0 1px 2px rgba(0, 0, 0, .075);\n          box-shadow: 0 1px 2px rgba(0, 0, 0, .075);\n}\n.dropdown-menu > li > a:hover,\n.dropdown-menu > li > a:focus {\n  background-color: #e8e8e8;\n  background-image: -webkit-linear-gradient(top, #f5f5f5 0%, #e8e8e8 100%);\n  background-image:         linear-gradient(to bottom, #f5f5f5 0%, #e8e8e8 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#fff5f5f5', endColorstr='#ffe8e8e8', GradientType=0);\n  background-repeat: repeat-x;\n}\n.dropdown-menu > .active > a,\n.dropdown-menu > .active > a:hover,\n.dropdown-menu > .active > a:focus {\n  background-color: #357ebd;\n  background-image: -webkit-linear-gradient(top, #428bca 0%, #357ebd 100%);\n  background-image:         linear-gradient(to bottom, #428bca 0%, #357ebd 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#ff428bca', endColorstr='#ff357ebd', GradientType=0);\n  background-repeat: repeat-x;\n}\n.navbar-default {\n  background-image: -webkit-linear-gradient(top, #fff 0%, #f8f8f8 100%);\n  background-image:         linear-gradient(to bottom, #fff 0%, #f8f8f8 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#ffffffff', endColorstr='#fff8f8f8', GradientType=0);\n  filter: progid:DXImageTransform.Microsoft.gradient(enabled = false);\n  background-repeat: repeat-x;\n  border-radius: 4px;\n  -webkit-box-shadow: inset 0 1px 0 rgba(255, 255, 255, .15), 0 1px 5px rgba(0, 0, 0, .075);\n          box-shadow: inset 0 1px 0 rgba(255, 255, 255, .15), 0 1px 5px rgba(0, 0, 0, .075);\n}\n.navbar-default .navbar-nav > .active > a {\n  background-image: -webkit-linear-gradient(top, #ebebeb 0%, #f3f3f3 100%);\n  background-image:         linear-gradient(to bottom, #ebebeb 0%, #f3f3f3 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#ffebebeb', endColorstr='#fff3f3f3', GradientType=0);\n  background-repeat: repeat-x;\n  -webkit-box-shadow: inset 0 3px 9px rgba(0, 0, 0, .075);\n          box-shadow: inset 0 3px 9px rgba(0, 0, 0, .075);\n}\n.navbar-brand,\n.navbar-nav > li > a {\n  text-shadow: 0 1px 0 rgba(255, 255, 255, .25);\n}\n.navbar-inverse {\n  background-image: -webkit-linear-gradient(top, #3c3c3c 0%, #222 100%);\n  background-image:         linear-gradient(to bottom, #3c3c3c 0%, #222 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#ff3c3c3c', endColorstr='#ff222222', GradientType=0);\n  filter: progid:DXImageTransform.Microsoft.gradient(enabled = false);\n  background-repeat: repeat-x;\n}\n.navbar-inverse .navbar-nav > .active > a {\n  background-image: -webkit-linear-gradient(top, #222 0%, #282828 100%);\n  background-image:         linear-gradient(to bottom, #222 0%, #282828 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#ff222222', endColorstr='#ff282828', GradientType=0);\n  background-repeat: repeat-x;\n  -webkit-box-shadow: inset 0 3px 9px rgba(0, 0, 0, .25);\n          box-shadow: inset 0 3px 9px rgba(0, 0, 0, .25);\n}\n.navbar-inverse .navbar-brand,\n.navbar-inverse .navbar-nav > li > a {\n  text-shadow: 0 -1px 0 rgba(0, 0, 0, .25);\n}\n.navbar-static-top,\n.navbar-fixed-top,\n.navbar-fixed-bottom {\n  border-radius: 0;\n}\n.alert {\n  text-shadow: 0 1px 0 rgba(255, 255, 255, .2);\n  -webkit-box-shadow: inset 0 1px 0 rgba(255, 255, 255, .25), 0 1px 2px rgba(0, 0, 0, .05);\n          box-shadow: inset 0 1px 0 rgba(255, 255, 255, .25), 0 1px 2px rgba(0, 0, 0, .05);\n}\n.alert-success {\n  background-image: -webkit-linear-gradient(top, #dff0d8 0%, #c8e5bc 100%);\n  background-image:         linear-gradient(to bottom, #dff0d8 0%, #c8e5bc 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#ffdff0d8', endColorstr='#ffc8e5bc', GradientType=0);\n  background-repeat: repeat-x;\n  border-color: #b2dba1;\n}\n.alert-info {\n  background-image: -webkit-linear-gradient(top, #d9edf7 0%, #b9def0 100%);\n  background-image:         linear-gradient(to bottom, #d9edf7 0%, #b9def0 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#ffd9edf7', endColorstr='#ffb9def0', GradientType=0);\n  background-repeat: repeat-x;\n  border-color: #9acfea;\n}\n.alert-warning {\n  background-image: -webkit-linear-gradient(top, #fcf8e3 0%, #f8efc0 100%);\n  background-image:         linear-gradient(to bottom, #fcf8e3 0%, #f8efc0 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#fffcf8e3', endColorstr='#fff8efc0', GradientType=0);\n  background-repeat: repeat-x;\n  border-color: #f5e79e;\n}\n.alert-danger {\n  background-image: -webkit-linear-gradient(top, #f2dede 0%, #e7c3c3 100%);\n  background-image:         linear-gradient(to bottom, #f2dede 0%, #e7c3c3 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#fff2dede', endColorstr='#ffe7c3c3', GradientType=0);\n  background-repeat: repeat-x;\n  border-color: #dca7a7;\n}\n.progress {\n  background-image: -webkit-linear-gradient(top, #ebebeb 0%, #f5f5f5 100%);\n  background-image:         linear-gradient(to bottom, #ebebeb 0%, #f5f5f5 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#ffebebeb', endColorstr='#fff5f5f5', GradientType=0);\n  background-repeat: repeat-x;\n}\n.progress-bar {\n  background-image: -webkit-linear-gradient(top, #428bca 0%, #3071a9 100%);\n  background-image:         linear-gradient(to bottom, #428bca 0%, #3071a9 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#ff428bca', endColorstr='#ff3071a9', GradientType=0);\n  background-repeat: repeat-x;\n}\n.progress-bar-success {\n  background-image: -webkit-linear-gradient(top, #5cb85c 0%, #449d44 100%);\n  background-image:         linear-gradient(to bottom, #5cb85c 0%, #449d44 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#ff5cb85c', endColorstr='#ff449d44', GradientType=0);\n  background-repeat: repeat-x;\n}\n.progress-bar-info {\n  background-image: -webkit-linear-gradient(top, #5bc0de 0%, #31b0d5 100%);\n  background-image:         linear-gradient(to bottom, #5bc0de 0%, #31b0d5 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#ff5bc0de', endColorstr='#ff31b0d5', GradientType=0);\n  background-repeat: repeat-x;\n}\n.progress-bar-warning {\n  background-image: -webkit-linear-gradient(top, #f0ad4e 0%, #ec971f 100%);\n  background-image:         linear-gradient(to bottom, #f0ad4e 0%, #ec971f 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#fff0ad4e', endColorstr='#ffec971f', GradientType=0);\n  background-repeat: repeat-x;\n}\n.progress-bar-danger {\n  background-image: -webkit-linear-gradient(top, #d9534f 0%, #c9302c 100%);\n  background-image:         linear-gradient(to bottom, #d9534f 0%, #c9302c 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#ffd9534f', endColorstr='#ffc9302c', GradientType=0);\n  background-repeat: repeat-x;\n}\n.list-group {\n  border-radius: 4px;\n  -webkit-box-shadow: 0 1px 2px rgba(0, 0, 0, .075);\n          box-shadow: 0 1px 2px rgba(0, 0, 0, .075);\n}\n.list-group-item.active,\n.list-group-item.active:hover,\n.list-group-item.active:focus {\n  text-shadow: 0 -1px 0 #3071a9;\n  background-image: -webkit-linear-gradient(top, #428bca 0%, #3278b3 100%);\n  background-image:         linear-gradient(to bottom, #428bca 0%, #3278b3 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#ff428bca', endColorstr='#ff3278b3', GradientType=0);\n  background-repeat: repeat-x;\n  border-color: #3278b3;\n}\n.panel {\n  -webkit-box-shadow: 0 1px 2px rgba(0, 0, 0, .05);\n          box-shadow: 0 1px 2px rgba(0, 0, 0, .05);\n}\n.panel-default > .panel-heading {\n  background-image: -webkit-linear-gradient(top, #f5f5f5 0%, #e8e8e8 100%);\n  background-image:         linear-gradient(to bottom, #f5f5f5 0%, #e8e8e8 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#fff5f5f5', endColorstr='#ffe8e8e8', GradientType=0);\n  background-repeat: repeat-x;\n}\n.panel-primary > .panel-heading {\n  background-image: -webkit-linear-gradient(top, #428bca 0%, #357ebd 100%);\n  background-image:         linear-gradient(to bottom, #428bca 0%, #357ebd 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#ff428bca', endColorstr='#ff357ebd', GradientType=0);\n  background-repeat: repeat-x;\n}\n.panel-success > .panel-heading {\n  background-image: -webkit-linear-gradient(top, #dff0d8 0%, #d0e9c6 100%);\n  background-image:         linear-gradient(to bottom, #dff0d8 0%, #d0e9c6 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#ffdff0d8', endColorstr='#ffd0e9c6', GradientType=0);\n  background-repeat: repeat-x;\n}\n.panel-info > .panel-heading {\n  background-image: -webkit-linear-gradient(top, #d9edf7 0%, #c4e3f3 100%);\n  background-image:         linear-gradient(to bottom, #d9edf7 0%, #c4e3f3 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#ffd9edf7', endColorstr='#ffc4e3f3', GradientType=0);\n  background-repeat: repeat-x;\n}\n.panel-warning > .panel-heading {\n  background-image: -webkit-linear-gradient(top, #fcf8e3 0%, #faf2cc 100%);\n  background-image:         linear-gradient(to bottom, #fcf8e3 0%, #faf2cc 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#fffcf8e3', endColorstr='#fffaf2cc', GradientType=0);\n  background-repeat: repeat-x;\n}\n.panel-danger > .panel-heading {\n  background-image: -webkit-linear-gradient(top, #f2dede 0%, #ebcccc 100%);\n  background-image:         linear-gradient(to bottom, #f2dede 0%, #ebcccc 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#fff2dede', endColorstr='#ffebcccc', GradientType=0);\n  background-repeat: repeat-x;\n}\n.well {\n  background-image: -webkit-linear-gradient(top, #e8e8e8 0%, #f5f5f5 100%);\n  background-image:         linear-gradient(to bottom, #e8e8e8 0%, #f5f5f5 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#ffe8e8e8', endColorstr='#fff5f5f5', GradientType=0);\n  background-repeat: repeat-x;\n  border-color: #dcdcdc;\n  -webkit-box-shadow: inset 0 1px 3px rgba(0, 0, 0, .05), 0 1px 0 rgba(255, 255, 255, .1);\n          box-shadow: inset 0 1px 3px rgba(0, 0, 0, .05), 0 1px 0 rgba(255, 255, 255, .1);\n}\n/*# sourceMappingURL=bootstrap-theme.css.map */\n"
  },
  {
    "path": "doc/css/bootstrap.css",
    "content": "/*!\n * Bootstrap v3.1.0 (http://getbootstrap.com)\n * Copyright 2011-2014 Twitter, Inc.\n * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)\n */\n\n/*! normalize.css v3.0.0 | MIT License | git.io/normalize */\nhtml {\n  font-family: sans-serif;\n  -webkit-text-size-adjust: 100%;\n      -ms-text-size-adjust: 100%;\n}\nbody {\n  margin: 0;\n}\narticle,\naside,\ndetails,\nfigcaption,\nfigure,\nfooter,\nheader,\nhgroup,\nmain,\nnav,\nsection,\nsummary {\n  display: block;\n}\naudio,\ncanvas,\nprogress,\nvideo {\n  display: inline-block;\n  vertical-align: baseline;\n}\naudio:not([controls]) {\n  display: none;\n  height: 0;\n}\n[hidden],\ntemplate {\n  display: none;\n}\na {\n  background: transparent;\n}\na:active,\na:hover {\n  outline: 0;\n}\nabbr[title] {\n  border-bottom: 1px dotted;\n}\nb,\nstrong {\n  font-weight: bold;\n}\ndfn {\n  font-style: italic;\n}\nh1 {\n  margin: .67em 0;\n  font-size: 2em;\n}\nmark {\n  color: #000;\n  background: #ff0;\n}\nsmall {\n  font-size: 80%;\n}\nsub,\nsup {\n  position: relative;\n  font-size: 75%;\n  line-height: 0;\n  vertical-align: baseline;\n}\nsup {\n  top: -.5em;\n}\nsub {\n  bottom: -.25em;\n}\nimg {\n  border: 0;\n}\nsvg:not(:root) {\n  overflow: hidden;\n}\nfigure {\n  margin: 1em 40px;\n}\nhr {\n  height: 0;\n  -moz-box-sizing: content-box;\n       box-sizing: content-box;\n}\npre {\n  overflow: auto;\n}\ncode,\nkbd,\npre,\nsamp {\n  font-family: monospace, monospace;\n  font-size: 1em;\n}\nbutton,\ninput,\noptgroup,\nselect,\ntextarea {\n  margin: 0;\n  font: inherit;\n  color: inherit;\n}\nbutton {\n  overflow: visible;\n}\nbutton,\nselect {\n  text-transform: none;\n}\nbutton,\nhtml input[type=\"button\"],\ninput[type=\"reset\"],\ninput[type=\"submit\"] {\n  -webkit-appearance: button;\n  cursor: pointer;\n}\nbutton[disabled],\nhtml input[disabled] {\n  cursor: default;\n}\nbutton::-moz-focus-inner,\ninput::-moz-focus-inner {\n  padding: 0;\n  border: 0;\n}\ninput {\n  line-height: normal;\n}\ninput[type=\"checkbox\"],\ninput[type=\"radio\"] {\n  box-sizing: border-box;\n  padding: 0;\n}\ninput[type=\"number\"]::-webkit-inner-spin-button,\ninput[type=\"number\"]::-webkit-outer-spin-button {\n  height: auto;\n}\ninput[type=\"search\"] {\n  -webkit-box-sizing: content-box;\n     -moz-box-sizing: content-box;\n          box-sizing: content-box;\n  -webkit-appearance: textfield;\n}\ninput[type=\"search\"]::-webkit-search-cancel-button,\ninput[type=\"search\"]::-webkit-search-decoration {\n  -webkit-appearance: none;\n}\nfieldset {\n  padding: .35em .625em .75em;\n  margin: 0 2px;\n  border: 1px solid #c0c0c0;\n}\nlegend {\n  padding: 0;\n  border: 0;\n}\ntextarea {\n  overflow: auto;\n}\noptgroup {\n  font-weight: bold;\n}\ntable {\n  border-spacing: 0;\n  border-collapse: collapse;\n}\ntd,\nth {\n  padding: 0;\n}\n@media print {\n  * {\n    color: #000 !important;\n    text-shadow: none !important;\n    background: transparent !important;\n    box-shadow: none !important;\n  }\n  a,\n  a:visited {\n    text-decoration: underline;\n  }\n  a[href]:after {\n    content: \" (\" attr(href) \")\";\n  }\n  abbr[title]:after {\n    content: \" (\" attr(title) \")\";\n  }\n  a[href^=\"javascript:\"]:after,\n  a[href^=\"#\"]:after {\n    content: \"\";\n  }\n  pre,\n  blockquote {\n    border: 1px solid #999;\n\n    page-break-inside: avoid;\n  }\n  thead {\n    display: table-header-group;\n  }\n  tr,\n  img {\n    page-break-inside: avoid;\n  }\n  img {\n    max-width: 100% !important;\n  }\n  p,\n  h2,\n  h3 {\n    orphans: 3;\n    widows: 3;\n  }\n  h2,\n  h3 {\n    page-break-after: avoid;\n  }\n  select {\n    background: #fff !important;\n  }\n  .navbar {\n    display: none;\n  }\n  .table td,\n  .table th {\n    background-color: #fff !important;\n  }\n  .btn > .caret,\n  .dropup > .btn > .caret {\n    border-top-color: #000 !important;\n  }\n  .label {\n    border: 1px solid #000;\n  }\n  .table {\n    border-collapse: collapse !important;\n  }\n  .table-bordered th,\n  .table-bordered td {\n    border: 1px solid #ddd !important;\n  }\n}\n* {\n  -webkit-box-sizing: border-box;\n     -moz-box-sizing: border-box;\n          box-sizing: border-box;\n}\n*:before,\n*:after {\n  -webkit-box-sizing: border-box;\n     -moz-box-sizing: border-box;\n          box-sizing: border-box;\n}\nhtml {\n  font-size: 62.5%;\n\n  -webkit-tap-highlight-color: rgba(0, 0, 0, 0);\n}\nbody {\n  font-family: \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n  font-size: 14px;\n  line-height: 1.428571429;\n  color: #333;\n  background-color: #fff;\n}\ninput,\nbutton,\nselect,\ntextarea {\n  font-family: inherit;\n  font-size: inherit;\n  line-height: inherit;\n}\na {\n  color: #428bca;\n  text-decoration: none;\n}\na:hover,\na:focus {\n  color: #2a6496;\n  text-decoration: underline;\n}\na:focus {\n  outline: thin dotted;\n  outline: 5px auto -webkit-focus-ring-color;\n  outline-offset: -2px;\n}\nfigure {\n  margin: 0;\n}\nimg {\n  vertical-align: middle;\n}\n.img-responsive {\n  display: block;\n  max-width: 100%;\n  height: auto;\n}\n.img-rounded {\n  border-radius: 6px;\n}\n.img-thumbnail {\n  display: inline-block;\n  max-width: 100%;\n  height: auto;\n  padding: 4px;\n  line-height: 1.428571429;\n  background-color: #fff;\n  border: 1px solid #ddd;\n  border-radius: 4px;\n  -webkit-transition: all .2s ease-in-out;\n          transition: all .2s ease-in-out;\n}\n.img-circle {\n  border-radius: 50%;\n}\nhr {\n  margin-top: 20px;\n  margin-bottom: 20px;\n  border: 0;\n  border-top: 1px solid #eee;\n}\n.sr-only {\n  position: absolute;\n  width: 1px;\n  height: 1px;\n  padding: 0;\n  margin: -1px;\n  overflow: hidden;\n  clip: rect(0, 0, 0, 0);\n  border: 0;\n}\nh1,\nh2,\nh3,\nh4,\nh5,\nh6,\n.h1,\n.h2,\n.h3,\n.h4,\n.h5,\n.h6 {\n  font-family: inherit;\n  font-weight: 500;\n  line-height: 1.1;\n  color: inherit;\n}\nh1 small,\nh2 small,\nh3 small,\nh4 small,\nh5 small,\nh6 small,\n.h1 small,\n.h2 small,\n.h3 small,\n.h4 small,\n.h5 small,\n.h6 small,\nh1 .small,\nh2 .small,\nh3 .small,\nh4 .small,\nh5 .small,\nh6 .small,\n.h1 .small,\n.h2 .small,\n.h3 .small,\n.h4 .small,\n.h5 .small,\n.h6 .small {\n  font-weight: normal;\n  line-height: 1;\n  color: #999;\n}\nh1,\n.h1,\nh2,\n.h2,\nh3,\n.h3 {\n  margin-top: 20px;\n  margin-bottom: 10px;\n}\nh1 small,\n.h1 small,\nh2 small,\n.h2 small,\nh3 small,\n.h3 small,\nh1 .small,\n.h1 .small,\nh2 .small,\n.h2 .small,\nh3 .small,\n.h3 .small {\n  font-size: 65%;\n}\nh4,\n.h4,\nh5,\n.h5,\nh6,\n.h6 {\n  margin-top: 10px;\n  margin-bottom: 10px;\n}\nh4 small,\n.h4 small,\nh5 small,\n.h5 small,\nh6 small,\n.h6 small,\nh4 .small,\n.h4 .small,\nh5 .small,\n.h5 .small,\nh6 .small,\n.h6 .small {\n  font-size: 75%;\n}\nh1,\n.h1 {\n  font-size: 36px;\n}\nh2,\n.h2 {\n  font-size: 30px;\n}\nh3,\n.h3 {\n  font-size: 24px;\n}\nh4,\n.h4 {\n  font-size: 18px;\n}\nh5,\n.h5 {\n  font-size: 14px;\n}\nh6,\n.h6 {\n  font-size: 12px;\n}\np {\n  margin: 0 0 10px;\n}\n.lead {\n  margin-bottom: 20px;\n  font-size: 16px;\n  font-weight: 200;\n  line-height: 1.4;\n}\n@media (min-width: 768px) {\n  .lead {\n    font-size: 21px;\n  }\n}\nsmall,\n.small {\n  font-size: 85%;\n}\ncite {\n  font-style: normal;\n}\n.text-left {\n  text-align: left;\n}\n.text-right {\n  text-align: right;\n}\n.text-center {\n  text-align: center;\n}\n.text-justify {\n  text-align: justify;\n}\n.text-muted {\n  color: #999;\n}\n.text-primary {\n  color: #428bca;\n}\na.text-primary:hover {\n  color: #3071a9;\n}\n.text-success {\n  color: #3c763d;\n}\na.text-success:hover {\n  color: #2b542c;\n}\n.text-info {\n  color: #31708f;\n}\na.text-info:hover {\n  color: #245269;\n}\n.text-warning {\n  color: #8a6d3b;\n}\na.text-warning:hover {\n  color: #66512c;\n}\n.text-danger {\n  color: #a94442;\n}\na.text-danger:hover {\n  color: #843534;\n}\n.bg-primary {\n  color: #fff;\n  background-color: #428bca;\n}\na.bg-primary:hover {\n  background-color: #3071a9;\n}\n.bg-success {\n  background-color: #dff0d8;\n}\na.bg-success:hover {\n  background-color: #c1e2b3;\n}\n.bg-info {\n  background-color: #d9edf7;\n}\na.bg-info:hover {\n  background-color: #afd9ee;\n}\n.bg-warning {\n  background-color: #fcf8e3;\n}\na.bg-warning:hover {\n  background-color: #f7ecb5;\n}\n.bg-danger {\n  background-color: #f2dede;\n}\na.bg-danger:hover {\n  background-color: #e4b9b9;\n}\n.page-header {\n  padding-bottom: 9px;\n  margin: 40px 0 20px;\n  border-bottom: 1px solid #eee;\n}\nul,\nol {\n  margin-top: 0;\n  margin-bottom: 10px;\n}\nul ul,\nol ul,\nul ol,\nol ol {\n  margin-bottom: 0;\n}\n.list-unstyled {\n  padding-left: 0;\n  list-style: none;\n}\n.list-inline {\n  padding-left: 0;\n  list-style: none;\n}\n.list-inline > li {\n  display: inline-block;\n  padding-right: 5px;\n  padding-left: 5px;\n}\n.list-inline > li:first-child {\n  padding-left: 0;\n}\ndl {\n  margin-top: 0;\n  margin-bottom: 20px;\n}\ndt,\ndd {\n  line-height: 1.428571429;\n}\ndt {\n  font-weight: bold;\n}\ndd {\n  margin-left: 0;\n}\n@media (min-width: 768px) {\n  .dl-horizontal dt {\n    float: left;\n    width: 160px;\n    overflow: hidden;\n    clear: left;\n    text-align: right;\n    text-overflow: ellipsis;\n    white-space: nowrap;\n  }\n  .dl-horizontal dd {\n    margin-left: 180px;\n  }\n}\nabbr[title],\nabbr[data-original-title] {\n  cursor: help;\n  border-bottom: 1px dotted #999;\n}\n.initialism {\n  font-size: 90%;\n  text-transform: uppercase;\n}\nblockquote {\n  padding: 10px 20px;\n  margin: 0 0 20px;\n  font-size: 17.5px;\n  border-left: 5px solid #eee;\n}\nblockquote p:last-child,\nblockquote ul:last-child,\nblockquote ol:last-child {\n  margin-bottom: 0;\n}\nblockquote footer,\nblockquote small,\nblockquote .small {\n  display: block;\n  font-size: 80%;\n  line-height: 1.428571429;\n  color: #999;\n}\nblockquote footer:before,\nblockquote small:before,\nblockquote .small:before {\n  content: '\\2014 \\00A0';\n}\n.blockquote-reverse,\nblockquote.pull-right {\n  padding-right: 15px;\n  padding-left: 0;\n  text-align: right;\n  border-right: 5px solid #eee;\n  border-left: 0;\n}\n.blockquote-reverse footer:before,\nblockquote.pull-right footer:before,\n.blockquote-reverse small:before,\nblockquote.pull-right small:before,\n.blockquote-reverse .small:before,\nblockquote.pull-right .small:before {\n  content: '';\n}\n.blockquote-reverse footer:after,\nblockquote.pull-right footer:after,\n.blockquote-reverse small:after,\nblockquote.pull-right small:after,\n.blockquote-reverse .small:after,\nblockquote.pull-right .small:after {\n  content: '\\00A0 \\2014';\n}\nblockquote:before,\nblockquote:after {\n  content: \"\";\n}\naddress {\n  margin-bottom: 20px;\n  font-style: normal;\n  line-height: 1.428571429;\n}\ncode,\nkbd,\npre,\nsamp {\n  font-family: Menlo, Monaco, Consolas, \"Courier New\", monospace;\n}\ncode {\n  padding: 2px 4px;\n  font-size: 90%;\n  color: #c7254e;\n  white-space: nowrap;\n  background-color: #f9f2f4;\n  border-radius: 4px;\n}\nkbd {\n  padding: 2px 4px;\n  font-size: 90%;\n  color: #fff;\n  background-color: #333;\n  border-radius: 3px;\n  box-shadow: inset 0 -1px 0 rgba(0, 0, 0, .25);\n}\npre {\n  display: block;\n  padding: 9.5px;\n  margin: 0 0 10px;\n  font-size: 13px;\n  line-height: 1.428571429;\n  color: #333;\n  word-break: break-all;\n  word-wrap: break-word;\n  background-color: #f5f5f5;\n  border: 1px solid #ccc;\n  border-radius: 4px;\n}\npre code {\n  padding: 0;\n  font-size: inherit;\n  color: inherit;\n  white-space: pre-wrap;\n  background-color: transparent;\n  border-radius: 0;\n}\n.pre-scrollable {\n  max-height: 340px;\n  overflow-y: scroll;\n}\n.container {\n  padding-right: 15px;\n  padding-left: 15px;\n  margin-right: auto;\n  margin-left: auto;\n}\n@media (min-width: 768px) {\n  .container {\n    width: 750px;\n  }\n}\n@media (min-width: 992px) {\n  .container {\n    width: 970px;\n  }\n}\n@media (min-width: 1200px) {\n  .container {\n    width: 1170px;\n  }\n}\n.container-fluid {\n  padding-right: 15px;\n  padding-left: 15px;\n  margin-right: auto;\n  margin-left: auto;\n}\n.row {\n  margin-right: -15px;\n  margin-left: -15px;\n}\n.col-xs-1, .col-sm-1, .col-md-1, .col-lg-1, .col-xs-2, .col-sm-2, .col-md-2, .col-lg-2, .col-xs-3, .col-sm-3, .col-md-3, .col-lg-3, .col-xs-4, .col-sm-4, .col-md-4, .col-lg-4, .col-xs-5, .col-sm-5, .col-md-5, .col-lg-5, .col-xs-6, .col-sm-6, .col-md-6, .col-lg-6, .col-xs-7, .col-sm-7, .col-md-7, .col-lg-7, .col-xs-8, .col-sm-8, .col-md-8, .col-lg-8, .col-xs-9, .col-sm-9, .col-md-9, .col-lg-9, .col-xs-10, .col-sm-10, .col-md-10, .col-lg-10, .col-xs-11, .col-sm-11, .col-md-11, .col-lg-11, .col-xs-12, .col-sm-12, .col-md-12, .col-lg-12 {\n  position: relative;\n  min-height: 1px;\n  padding-right: 15px;\n  padding-left: 15px;\n}\n.col-xs-1, .col-xs-2, .col-xs-3, .col-xs-4, .col-xs-5, .col-xs-6, .col-xs-7, .col-xs-8, .col-xs-9, .col-xs-10, .col-xs-11, .col-xs-12 {\n  float: left;\n}\n.col-xs-12 {\n  width: 100%;\n}\n.col-xs-11 {\n  width: 91.66666666666666%;\n}\n.col-xs-10 {\n  width: 83.33333333333334%;\n}\n.col-xs-9 {\n  width: 75%;\n}\n.col-xs-8 {\n  width: 66.66666666666666%;\n}\n.col-xs-7 {\n  width: 58.333333333333336%;\n}\n.col-xs-6 {\n  width: 50%;\n}\n.col-xs-5 {\n  width: 41.66666666666667%;\n}\n.col-xs-4 {\n  width: 33.33333333333333%;\n}\n.col-xs-3 {\n  width: 25%;\n}\n.col-xs-2 {\n  width: 16.666666666666664%;\n}\n.col-xs-1 {\n  width: 8.333333333333332%;\n}\n.col-xs-pull-12 {\n  right: 100%;\n}\n.col-xs-pull-11 {\n  right: 91.66666666666666%;\n}\n.col-xs-pull-10 {\n  right: 83.33333333333334%;\n}\n.col-xs-pull-9 {\n  right: 75%;\n}\n.col-xs-pull-8 {\n  right: 66.66666666666666%;\n}\n.col-xs-pull-7 {\n  right: 58.333333333333336%;\n}\n.col-xs-pull-6 {\n  right: 50%;\n}\n.col-xs-pull-5 {\n  right: 41.66666666666667%;\n}\n.col-xs-pull-4 {\n  right: 33.33333333333333%;\n}\n.col-xs-pull-3 {\n  right: 25%;\n}\n.col-xs-pull-2 {\n  right: 16.666666666666664%;\n}\n.col-xs-pull-1 {\n  right: 8.333333333333332%;\n}\n.col-xs-pull-0 {\n  right: 0;\n}\n.col-xs-push-12 {\n  left: 100%;\n}\n.col-xs-push-11 {\n  left: 91.66666666666666%;\n}\n.col-xs-push-10 {\n  left: 83.33333333333334%;\n}\n.col-xs-push-9 {\n  left: 75%;\n}\n.col-xs-push-8 {\n  left: 66.66666666666666%;\n}\n.col-xs-push-7 {\n  left: 58.333333333333336%;\n}\n.col-xs-push-6 {\n  left: 50%;\n}\n.col-xs-push-5 {\n  left: 41.66666666666667%;\n}\n.col-xs-push-4 {\n  left: 33.33333333333333%;\n}\n.col-xs-push-3 {\n  left: 25%;\n}\n.col-xs-push-2 {\n  left: 16.666666666666664%;\n}\n.col-xs-push-1 {\n  left: 8.333333333333332%;\n}\n.col-xs-push-0 {\n  left: 0;\n}\n.col-xs-offset-12 {\n  margin-left: 100%;\n}\n.col-xs-offset-11 {\n  margin-left: 91.66666666666666%;\n}\n.col-xs-offset-10 {\n  margin-left: 83.33333333333334%;\n}\n.col-xs-offset-9 {\n  margin-left: 75%;\n}\n.col-xs-offset-8 {\n  margin-left: 66.66666666666666%;\n}\n.col-xs-offset-7 {\n  margin-left: 58.333333333333336%;\n}\n.col-xs-offset-6 {\n  margin-left: 50%;\n}\n.col-xs-offset-5 {\n  margin-left: 41.66666666666667%;\n}\n.col-xs-offset-4 {\n  margin-left: 33.33333333333333%;\n}\n.col-xs-offset-3 {\n  margin-left: 25%;\n}\n.col-xs-offset-2 {\n  margin-left: 16.666666666666664%;\n}\n.col-xs-offset-1 {\n  margin-left: 8.333333333333332%;\n}\n.col-xs-offset-0 {\n  margin-left: 0;\n}\n@media (min-width: 768px) {\n  .col-sm-1, .col-sm-2, .col-sm-3, .col-sm-4, .col-sm-5, .col-sm-6, .col-sm-7, .col-sm-8, .col-sm-9, .col-sm-10, .col-sm-11, .col-sm-12 {\n    float: left;\n  }\n  .col-sm-12 {\n    width: 100%;\n  }\n  .col-sm-11 {\n    width: 91.66666666666666%;\n  }\n  .col-sm-10 {\n    width: 83.33333333333334%;\n  }\n  .col-sm-9 {\n    width: 75%;\n  }\n  .col-sm-8 {\n    width: 66.66666666666666%;\n  }\n  .col-sm-7 {\n    width: 58.333333333333336%;\n  }\n  .col-sm-6 {\n    width: 50%;\n  }\n  .col-sm-5 {\n    width: 41.66666666666667%;\n  }\n  .col-sm-4 {\n    width: 33.33333333333333%;\n  }\n  .col-sm-3 {\n    width: 25%;\n  }\n  .col-sm-2 {\n    width: 16.666666666666664%;\n  }\n  .col-sm-1 {\n    width: 8.333333333333332%;\n  }\n  .col-sm-pull-12 {\n    right: 100%;\n  }\n  .col-sm-pull-11 {\n    right: 91.66666666666666%;\n  }\n  .col-sm-pull-10 {\n    right: 83.33333333333334%;\n  }\n  .col-sm-pull-9 {\n    right: 75%;\n  }\n  .col-sm-pull-8 {\n    right: 66.66666666666666%;\n  }\n  .col-sm-pull-7 {\n    right: 58.333333333333336%;\n  }\n  .col-sm-pull-6 {\n    right: 50%;\n  }\n  .col-sm-pull-5 {\n    right: 41.66666666666667%;\n  }\n  .col-sm-pull-4 {\n    right: 33.33333333333333%;\n  }\n  .col-sm-pull-3 {\n    right: 25%;\n  }\n  .col-sm-pull-2 {\n    right: 16.666666666666664%;\n  }\n  .col-sm-pull-1 {\n    right: 8.333333333333332%;\n  }\n  .col-sm-pull-0 {\n    right: 0;\n  }\n  .col-sm-push-12 {\n    left: 100%;\n  }\n  .col-sm-push-11 {\n    left: 91.66666666666666%;\n  }\n  .col-sm-push-10 {\n    left: 83.33333333333334%;\n  }\n  .col-sm-push-9 {\n    left: 75%;\n  }\n  .col-sm-push-8 {\n    left: 66.66666666666666%;\n  }\n  .col-sm-push-7 {\n    left: 58.333333333333336%;\n  }\n  .col-sm-push-6 {\n    left: 50%;\n  }\n  .col-sm-push-5 {\n    left: 41.66666666666667%;\n  }\n  .col-sm-push-4 {\n    left: 33.33333333333333%;\n  }\n  .col-sm-push-3 {\n    left: 25%;\n  }\n  .col-sm-push-2 {\n    left: 16.666666666666664%;\n  }\n  .col-sm-push-1 {\n    left: 8.333333333333332%;\n  }\n  .col-sm-push-0 {\n    left: 0;\n  }\n  .col-sm-offset-12 {\n    margin-left: 100%;\n  }\n  .col-sm-offset-11 {\n    margin-left: 91.66666666666666%;\n  }\n  .col-sm-offset-10 {\n    margin-left: 83.33333333333334%;\n  }\n  .col-sm-offset-9 {\n    margin-left: 75%;\n  }\n  .col-sm-offset-8 {\n    margin-left: 66.66666666666666%;\n  }\n  .col-sm-offset-7 {\n    margin-left: 58.333333333333336%;\n  }\n  .col-sm-offset-6 {\n    margin-left: 50%;\n  }\n  .col-sm-offset-5 {\n    margin-left: 41.66666666666667%;\n  }\n  .col-sm-offset-4 {\n    margin-left: 33.33333333333333%;\n  }\n  .col-sm-offset-3 {\n    margin-left: 25%;\n  }\n  .col-sm-offset-2 {\n    margin-left: 16.666666666666664%;\n  }\n  .col-sm-offset-1 {\n    margin-left: 8.333333333333332%;\n  }\n  .col-sm-offset-0 {\n    margin-left: 0;\n  }\n}\n@media (min-width: 992px) {\n  .col-md-1, .col-md-2, .col-md-3, .col-md-4, .col-md-5, .col-md-6, .col-md-7, .col-md-8, .col-md-9, .col-md-10, .col-md-11, .col-md-12 {\n    float: left;\n  }\n  .col-md-12 {\n    width: 100%;\n  }\n  .col-md-11 {\n    width: 91.66666666666666%;\n  }\n  .col-md-10 {\n    width: 83.33333333333334%;\n  }\n  .col-md-9 {\n    width: 75%;\n  }\n  .col-md-8 {\n    width: 66.66666666666666%;\n  }\n  .col-md-7 {\n    width: 58.333333333333336%;\n  }\n  .col-md-6 {\n    width: 50%;\n  }\n  .col-md-5 {\n    width: 41.66666666666667%;\n  }\n  .col-md-4 {\n    width: 33.33333333333333%;\n  }\n  .col-md-3 {\n    width: 25%;\n  }\n  .col-md-2 {\n    width: 16.666666666666664%;\n  }\n  .col-md-1 {\n    width: 8.333333333333332%;\n  }\n  .col-md-pull-12 {\n    right: 100%;\n  }\n  .col-md-pull-11 {\n    right: 91.66666666666666%;\n  }\n  .col-md-pull-10 {\n    right: 83.33333333333334%;\n  }\n  .col-md-pull-9 {\n    right: 75%;\n  }\n  .col-md-pull-8 {\n    right: 66.66666666666666%;\n  }\n  .col-md-pull-7 {\n    right: 58.333333333333336%;\n  }\n  .col-md-pull-6 {\n    right: 50%;\n  }\n  .col-md-pull-5 {\n    right: 41.66666666666667%;\n  }\n  .col-md-pull-4 {\n    right: 33.33333333333333%;\n  }\n  .col-md-pull-3 {\n    right: 25%;\n  }\n  .col-md-pull-2 {\n    right: 16.666666666666664%;\n  }\n  .col-md-pull-1 {\n    right: 8.333333333333332%;\n  }\n  .col-md-pull-0 {\n    right: 0;\n  }\n  .col-md-push-12 {\n    left: 100%;\n  }\n  .col-md-push-11 {\n    left: 91.66666666666666%;\n  }\n  .col-md-push-10 {\n    left: 83.33333333333334%;\n  }\n  .col-md-push-9 {\n    left: 75%;\n  }\n  .col-md-push-8 {\n    left: 66.66666666666666%;\n  }\n  .col-md-push-7 {\n    left: 58.333333333333336%;\n  }\n  .col-md-push-6 {\n    left: 50%;\n  }\n  .col-md-push-5 {\n    left: 41.66666666666667%;\n  }\n  .col-md-push-4 {\n    left: 33.33333333333333%;\n  }\n  .col-md-push-3 {\n    left: 25%;\n  }\n  .col-md-push-2 {\n    left: 16.666666666666664%;\n  }\n  .col-md-push-1 {\n    left: 8.333333333333332%;\n  }\n  .col-md-push-0 {\n    left: 0;\n  }\n  .col-md-offset-12 {\n    margin-left: 100%;\n  }\n  .col-md-offset-11 {\n    margin-left: 91.66666666666666%;\n  }\n  .col-md-offset-10 {\n    margin-left: 83.33333333333334%;\n  }\n  .col-md-offset-9 {\n    margin-left: 75%;\n  }\n  .col-md-offset-8 {\n    margin-left: 66.66666666666666%;\n  }\n  .col-md-offset-7 {\n    margin-left: 58.333333333333336%;\n  }\n  .col-md-offset-6 {\n    margin-left: 50%;\n  }\n  .col-md-offset-5 {\n    margin-left: 41.66666666666667%;\n  }\n  .col-md-offset-4 {\n    margin-left: 33.33333333333333%;\n  }\n  .col-md-offset-3 {\n    margin-left: 25%;\n  }\n  .col-md-offset-2 {\n    margin-left: 16.666666666666664%;\n  }\n  .col-md-offset-1 {\n    margin-left: 8.333333333333332%;\n  }\n  .col-md-offset-0 {\n    margin-left: 0;\n  }\n}\n@media (min-width: 1200px) {\n  .col-lg-1, .col-lg-2, .col-lg-3, .col-lg-4, .col-lg-5, .col-lg-6, .col-lg-7, .col-lg-8, .col-lg-9, .col-lg-10, .col-lg-11, .col-lg-12 {\n    float: left;\n  }\n  .col-lg-12 {\n    width: 100%;\n  }\n  .col-lg-11 {\n    width: 91.66666666666666%;\n  }\n  .col-lg-10 {\n    width: 83.33333333333334%;\n  }\n  .col-lg-9 {\n    width: 75%;\n  }\n  .col-lg-8 {\n    width: 66.66666666666666%;\n  }\n  .col-lg-7 {\n    width: 58.333333333333336%;\n  }\n  .col-lg-6 {\n    width: 50%;\n  }\n  .col-lg-5 {\n    width: 41.66666666666667%;\n  }\n  .col-lg-4 {\n    width: 33.33333333333333%;\n  }\n  .col-lg-3 {\n    width: 25%;\n  }\n  .col-lg-2 {\n    width: 16.666666666666664%;\n  }\n  .col-lg-1 {\n    width: 8.333333333333332%;\n  }\n  .col-lg-pull-12 {\n    right: 100%;\n  }\n  .col-lg-pull-11 {\n    right: 91.66666666666666%;\n  }\n  .col-lg-pull-10 {\n    right: 83.33333333333334%;\n  }\n  .col-lg-pull-9 {\n    right: 75%;\n  }\n  .col-lg-pull-8 {\n    right: 66.66666666666666%;\n  }\n  .col-lg-pull-7 {\n    right: 58.333333333333336%;\n  }\n  .col-lg-pull-6 {\n    right: 50%;\n  }\n  .col-lg-pull-5 {\n    right: 41.66666666666667%;\n  }\n  .col-lg-pull-4 {\n    right: 33.33333333333333%;\n  }\n  .col-lg-pull-3 {\n    right: 25%;\n  }\n  .col-lg-pull-2 {\n    right: 16.666666666666664%;\n  }\n  .col-lg-pull-1 {\n    right: 8.333333333333332%;\n  }\n  .col-lg-pull-0 {\n    right: 0;\n  }\n  .col-lg-push-12 {\n    left: 100%;\n  }\n  .col-lg-push-11 {\n    left: 91.66666666666666%;\n  }\n  .col-lg-push-10 {\n    left: 83.33333333333334%;\n  }\n  .col-lg-push-9 {\n    left: 75%;\n  }\n  .col-lg-push-8 {\n    left: 66.66666666666666%;\n  }\n  .col-lg-push-7 {\n    left: 58.333333333333336%;\n  }\n  .col-lg-push-6 {\n    left: 50%;\n  }\n  .col-lg-push-5 {\n    left: 41.66666666666667%;\n  }\n  .col-lg-push-4 {\n    left: 33.33333333333333%;\n  }\n  .col-lg-push-3 {\n    left: 25%;\n  }\n  .col-lg-push-2 {\n    left: 16.666666666666664%;\n  }\n  .col-lg-push-1 {\n    left: 8.333333333333332%;\n  }\n  .col-lg-push-0 {\n    left: 0;\n  }\n  .col-lg-offset-12 {\n    margin-left: 100%;\n  }\n  .col-lg-offset-11 {\n    margin-left: 91.66666666666666%;\n  }\n  .col-lg-offset-10 {\n    margin-left: 83.33333333333334%;\n  }\n  .col-lg-offset-9 {\n    margin-left: 75%;\n  }\n  .col-lg-offset-8 {\n    margin-left: 66.66666666666666%;\n  }\n  .col-lg-offset-7 {\n    margin-left: 58.333333333333336%;\n  }\n  .col-lg-offset-6 {\n    margin-left: 50%;\n  }\n  .col-lg-offset-5 {\n    margin-left: 41.66666666666667%;\n  }\n  .col-lg-offset-4 {\n    margin-left: 33.33333333333333%;\n  }\n  .col-lg-offset-3 {\n    margin-left: 25%;\n  }\n  .col-lg-offset-2 {\n    margin-left: 16.666666666666664%;\n  }\n  .col-lg-offset-1 {\n    margin-left: 8.333333333333332%;\n  }\n  .col-lg-offset-0 {\n    margin-left: 0;\n  }\n}\ntable {\n  max-width: 100%;\n  background-color: transparent;\n}\nth {\n  text-align: left;\n}\n.table {\n  width: 100%;\n  margin-bottom: 20px;\n}\n.table > thead > tr > th,\n.table > tbody > tr > th,\n.table > tfoot > tr > th,\n.table > thead > tr > td,\n.table > tbody > tr > td,\n.table > tfoot > tr > td {\n  padding: 8px;\n  line-height: 1.428571429;\n  vertical-align: top;\n  border-top: 1px solid #ddd;\n}\n.table > thead > tr > th {\n  vertical-align: bottom;\n  border-bottom: 2px solid #ddd;\n}\n.table > caption + thead > tr:first-child > th,\n.table > colgroup + thead > tr:first-child > th,\n.table > thead:first-child > tr:first-child > th,\n.table > caption + thead > tr:first-child > td,\n.table > colgroup + thead > tr:first-child > td,\n.table > thead:first-child > tr:first-child > td {\n  border-top: 0;\n}\n.table > tbody + tbody {\n  border-top: 2px solid #ddd;\n}\n.table .table {\n  background-color: #fff;\n}\n.table-condensed > thead > tr > th,\n.table-condensed > tbody > tr > th,\n.table-condensed > tfoot > tr > th,\n.table-condensed > thead > tr > td,\n.table-condensed > tbody > tr > td,\n.table-condensed > tfoot > tr > td {\n  padding: 5px;\n}\n.table-bordered {\n  border: 1px solid #ddd;\n}\n.table-bordered > thead > tr > th,\n.table-bordered > tbody > tr > th,\n.table-bordered > tfoot > tr > th,\n.table-bordered > thead > tr > td,\n.table-bordered > tbody > tr > td,\n.table-bordered > tfoot > tr > td {\n  border: 1px solid #ddd;\n}\n.table-bordered > thead > tr > th,\n.table-bordered > thead > tr > td {\n  border-bottom-width: 2px;\n}\n.table-striped > tbody > tr:nth-child(odd) > td,\n.table-striped > tbody > tr:nth-child(odd) > th {\n  background-color: #f9f9f9;\n}\n.table-hover > tbody > tr:hover > td,\n.table-hover > tbody > tr:hover > th {\n  background-color: #f5f5f5;\n}\ntable col[class*=\"col-\"] {\n  position: static;\n  display: table-column;\n  float: none;\n}\ntable td[class*=\"col-\"],\ntable th[class*=\"col-\"] {\n  position: static;\n  display: table-cell;\n  float: none;\n}\n.table > thead > tr > td.active,\n.table > tbody > tr > td.active,\n.table > tfoot > tr > td.active,\n.table > thead > tr > th.active,\n.table > tbody > tr > th.active,\n.table > tfoot > tr > th.active,\n.table > thead > tr.active > td,\n.table > tbody > tr.active > td,\n.table > tfoot > tr.active > td,\n.table > thead > tr.active > th,\n.table > tbody > tr.active > th,\n.table > tfoot > tr.active > th {\n  background-color: #f5f5f5;\n}\n.table-hover > tbody > tr > td.active:hover,\n.table-hover > tbody > tr > th.active:hover,\n.table-hover > tbody > tr.active:hover > td,\n.table-hover > tbody > tr.active:hover > th {\n  background-color: #e8e8e8;\n}\n.table > thead > tr > td.success,\n.table > tbody > tr > td.success,\n.table > tfoot > tr > td.success,\n.table > thead > tr > th.success,\n.table > tbody > tr > th.success,\n.table > tfoot > tr > th.success,\n.table > thead > tr.success > td,\n.table > tbody > tr.success > td,\n.table > tfoot > tr.success > td,\n.table > thead > tr.success > th,\n.table > tbody > tr.success > th,\n.table > tfoot > tr.success > th {\n  background-color: #dff0d8;\n}\n.table-hover > tbody > tr > td.success:hover,\n.table-hover > tbody > tr > th.success:hover,\n.table-hover > tbody > tr.success:hover > td,\n.table-hover > tbody > tr.success:hover > th {\n  background-color: #d0e9c6;\n}\n.table > thead > tr > td.info,\n.table > tbody > tr > td.info,\n.table > tfoot > tr > td.info,\n.table > thead > tr > th.info,\n.table > tbody > tr > th.info,\n.table > tfoot > tr > th.info,\n.table > thead > tr.info > td,\n.table > tbody > tr.info > td,\n.table > tfoot > tr.info > td,\n.table > thead > tr.info > th,\n.table > tbody > tr.info > th,\n.table > tfoot > tr.info > th {\n  background-color: #d9edf7;\n}\n.table-hover > tbody > tr > td.info:hover,\n.table-hover > tbody > tr > th.info:hover,\n.table-hover > tbody > tr.info:hover > td,\n.table-hover > tbody > tr.info:hover > th {\n  background-color: #c4e3f3;\n}\n.table > thead > tr > td.warning,\n.table > tbody > tr > td.warning,\n.table > tfoot > tr > td.warning,\n.table > thead > tr > th.warning,\n.table > tbody > tr > th.warning,\n.table > tfoot > tr > th.warning,\n.table > thead > tr.warning > td,\n.table > tbody > tr.warning > td,\n.table > tfoot > tr.warning > td,\n.table > thead > tr.warning > th,\n.table > tbody > tr.warning > th,\n.table > tfoot > tr.warning > th {\n  background-color: #fcf8e3;\n}\n.table-hover > tbody > tr > td.warning:hover,\n.table-hover > tbody > tr > th.warning:hover,\n.table-hover > tbody > tr.warning:hover > td,\n.table-hover > tbody > tr.warning:hover > th {\n  background-color: #faf2cc;\n}\n.table > thead > tr > td.danger,\n.table > tbody > tr > td.danger,\n.table > tfoot > tr > td.danger,\n.table > thead > tr > th.danger,\n.table > tbody > tr > th.danger,\n.table > tfoot > tr > th.danger,\n.table > thead > tr.danger > td,\n.table > tbody > tr.danger > td,\n.table > tfoot > tr.danger > td,\n.table > thead > tr.danger > th,\n.table > tbody > tr.danger > th,\n.table > tfoot > tr.danger > th {\n  background-color: #f2dede;\n}\n.table-hover > tbody > tr > td.danger:hover,\n.table-hover > tbody > tr > th.danger:hover,\n.table-hover > tbody > tr.danger:hover > td,\n.table-hover > tbody > tr.danger:hover > th {\n  background-color: #ebcccc;\n}\n@media (max-width: 767px) {\n  .table-responsive {\n    width: 100%;\n    margin-bottom: 15px;\n    overflow-x: scroll;\n    overflow-y: hidden;\n    -webkit-overflow-scrolling: touch;\n    -ms-overflow-style: -ms-autohiding-scrollbar;\n    border: 1px solid #ddd;\n  }\n  .table-responsive > .table {\n    margin-bottom: 0;\n  }\n  .table-responsive > .table > thead > tr > th,\n  .table-responsive > .table > tbody > tr > th,\n  .table-responsive > .table > tfoot > tr > th,\n  .table-responsive > .table > thead > tr > td,\n  .table-responsive > .table > tbody > tr > td,\n  .table-responsive > .table > tfoot > tr > td {\n    white-space: nowrap;\n  }\n  .table-responsive > .table-bordered {\n    border: 0;\n  }\n  .table-responsive > .table-bordered > thead > tr > th:first-child,\n  .table-responsive > .table-bordered > tbody > tr > th:first-child,\n  .table-responsive > .table-bordered > tfoot > tr > th:first-child,\n  .table-responsive > .table-bordered > thead > tr > td:first-child,\n  .table-responsive > .table-bordered > tbody > tr > td:first-child,\n  .table-responsive > .table-bordered > tfoot > tr > td:first-child {\n    border-left: 0;\n  }\n  .table-responsive > .table-bordered > thead > tr > th:last-child,\n  .table-responsive > .table-bordered > tbody > tr > th:last-child,\n  .table-responsive > .table-bordered > tfoot > tr > th:last-child,\n  .table-responsive > .table-bordered > thead > tr > td:last-child,\n  .table-responsive > .table-bordered > tbody > tr > td:last-child,\n  .table-responsive > .table-bordered > tfoot > tr > td:last-child {\n    border-right: 0;\n  }\n  .table-responsive > .table-bordered > tbody > tr:last-child > th,\n  .table-responsive > .table-bordered > tfoot > tr:last-child > th,\n  .table-responsive > .table-bordered > tbody > tr:last-child > td,\n  .table-responsive > .table-bordered > tfoot > tr:last-child > td {\n    border-bottom: 0;\n  }\n}\nfieldset {\n  min-width: 0;\n  padding: 0;\n  margin: 0;\n  border: 0;\n}\nlegend {\n  display: block;\n  width: 100%;\n  padding: 0;\n  margin-bottom: 20px;\n  font-size: 21px;\n  line-height: inherit;\n  color: #333;\n  border: 0;\n  border-bottom: 1px solid #e5e5e5;\n}\nlabel {\n  display: inline-block;\n  margin-bottom: 5px;\n  font-weight: bold;\n}\ninput[type=\"search\"] {\n  -webkit-box-sizing: border-box;\n     -moz-box-sizing: border-box;\n          box-sizing: border-box;\n}\ninput[type=\"radio\"],\ninput[type=\"checkbox\"] {\n  margin: 4px 0 0;\n  margin-top: 1px \\9;\n  /* IE8-9 */\n  line-height: normal;\n}\ninput[type=\"file\"] {\n  display: block;\n}\ninput[type=\"range\"] {\n  display: block;\n  width: 100%;\n}\nselect[multiple],\nselect[size] {\n  height: auto;\n}\ninput[type=\"file\"]:focus,\ninput[type=\"radio\"]:focus,\ninput[type=\"checkbox\"]:focus {\n  outline: thin dotted;\n  outline: 5px auto -webkit-focus-ring-color;\n  outline-offset: -2px;\n}\noutput {\n  display: block;\n  padding-top: 7px;\n  font-size: 14px;\n  line-height: 1.428571429;\n  color: #555;\n}\n.form-control {\n  display: block;\n  width: 100%;\n  height: 34px;\n  padding: 6px 12px;\n  font-size: 14px;\n  line-height: 1.428571429;\n  color: #555;\n  background-color: #fff;\n  background-image: none;\n  border: 1px solid #ccc;\n  border-radius: 4px;\n  -webkit-box-shadow: inset 0 1px 1px rgba(0, 0, 0, .075);\n          box-shadow: inset 0 1px 1px rgba(0, 0, 0, .075);\n  -webkit-transition: border-color ease-in-out .15s, box-shadow ease-in-out .15s;\n          transition: border-color ease-in-out .15s, box-shadow ease-in-out .15s;\n}\n.form-control:focus {\n  border-color: #66afe9;\n  outline: 0;\n  -webkit-box-shadow: inset 0 1px 1px rgba(0,0,0,.075), 0 0 8px rgba(102, 175, 233, .6);\n          box-shadow: inset 0 1px 1px rgba(0,0,0,.075), 0 0 8px rgba(102, 175, 233, .6);\n}\n.form-control:-moz-placeholder {\n  color: #999;\n}\n.form-control::-moz-placeholder {\n  color: #999;\n  opacity: 1;\n}\n.form-control:-ms-input-placeholder {\n  color: #999;\n}\n.form-control::-webkit-input-placeholder {\n  color: #999;\n}\n.form-control[disabled],\n.form-control[readonly],\nfieldset[disabled] .form-control {\n  cursor: not-allowed;\n  background-color: #eee;\n  opacity: 1;\n}\ntextarea.form-control {\n  height: auto;\n}\ninput[type=\"date\"] {\n  line-height: 34px;\n}\n.form-group {\n  margin-bottom: 15px;\n}\n.radio,\n.checkbox {\n  display: block;\n  min-height: 20px;\n  padding-left: 20px;\n  margin-top: 10px;\n  margin-bottom: 10px;\n}\n.radio label,\n.checkbox label {\n  display: inline;\n  font-weight: normal;\n  cursor: pointer;\n}\n.radio input[type=\"radio\"],\n.radio-inline input[type=\"radio\"],\n.checkbox input[type=\"checkbox\"],\n.checkbox-inline input[type=\"checkbox\"] {\n  float: left;\n  margin-left: -20px;\n}\n.radio + .radio,\n.checkbox + .checkbox {\n  margin-top: -5px;\n}\n.radio-inline,\n.checkbox-inline {\n  display: inline-block;\n  padding-left: 20px;\n  margin-bottom: 0;\n  font-weight: normal;\n  vertical-align: middle;\n  cursor: pointer;\n}\n.radio-inline + .radio-inline,\n.checkbox-inline + .checkbox-inline {\n  margin-top: 0;\n  margin-left: 10px;\n}\ninput[type=\"radio\"][disabled],\ninput[type=\"checkbox\"][disabled],\n.radio[disabled],\n.radio-inline[disabled],\n.checkbox[disabled],\n.checkbox-inline[disabled],\nfieldset[disabled] input[type=\"radio\"],\nfieldset[disabled] input[type=\"checkbox\"],\nfieldset[disabled] .radio,\nfieldset[disabled] .radio-inline,\nfieldset[disabled] .checkbox,\nfieldset[disabled] .checkbox-inline {\n  cursor: not-allowed;\n}\n.input-sm {\n  height: 30px;\n  padding: 5px 10px;\n  font-size: 12px;\n  line-height: 1.5;\n  border-radius: 3px;\n}\nselect.input-sm {\n  height: 30px;\n  line-height: 30px;\n}\ntextarea.input-sm,\nselect[multiple].input-sm {\n  height: auto;\n}\n.input-lg {\n  height: 46px;\n  padding: 10px 16px;\n  font-size: 18px;\n  line-height: 1.33;\n  border-radius: 6px;\n}\nselect.input-lg {\n  height: 46px;\n  line-height: 46px;\n}\ntextarea.input-lg,\nselect[multiple].input-lg {\n  height: auto;\n}\n.has-feedback {\n  position: relative;\n}\n.has-feedback .form-control {\n  padding-right: 42.5px;\n}\n.has-feedback .form-control-feedback {\n  position: absolute;\n  top: 25px;\n  right: 0;\n  display: block;\n  width: 34px;\n  height: 34px;\n  line-height: 34px;\n  text-align: center;\n}\n.has-success .help-block,\n.has-success .control-label,\n.has-success .radio,\n.has-success .checkbox,\n.has-success .radio-inline,\n.has-success .checkbox-inline {\n  color: #3c763d;\n}\n.has-success .form-control {\n  border-color: #3c763d;\n  -webkit-box-shadow: inset 0 1px 1px rgba(0, 0, 0, .075);\n          box-shadow: inset 0 1px 1px rgba(0, 0, 0, .075);\n}\n.has-success .form-control:focus {\n  border-color: #2b542c;\n  -webkit-box-shadow: inset 0 1px 1px rgba(0, 0, 0, .075), 0 0 6px #67b168;\n          box-shadow: inset 0 1px 1px rgba(0, 0, 0, .075), 0 0 6px #67b168;\n}\n.has-success .input-group-addon {\n  color: #3c763d;\n  background-color: #dff0d8;\n  border-color: #3c763d;\n}\n.has-success .form-control-feedback {\n  color: #3c763d;\n}\n.has-warning .help-block,\n.has-warning .control-label,\n.has-warning .radio,\n.has-warning .checkbox,\n.has-warning .radio-inline,\n.has-warning .checkbox-inline {\n  color: #8a6d3b;\n}\n.has-warning .form-control {\n  border-color: #8a6d3b;\n  -webkit-box-shadow: inset 0 1px 1px rgba(0, 0, 0, .075);\n          box-shadow: inset 0 1px 1px rgba(0, 0, 0, .075);\n}\n.has-warning .form-control:focus {\n  border-color: #66512c;\n  -webkit-box-shadow: inset 0 1px 1px rgba(0, 0, 0, .075), 0 0 6px #c0a16b;\n          box-shadow: inset 0 1px 1px rgba(0, 0, 0, .075), 0 0 6px #c0a16b;\n}\n.has-warning .input-group-addon {\n  color: #8a6d3b;\n  background-color: #fcf8e3;\n  border-color: #8a6d3b;\n}\n.has-warning .form-control-feedback {\n  color: #8a6d3b;\n}\n.has-error .help-block,\n.has-error .control-label,\n.has-error .radio,\n.has-error .checkbox,\n.has-error .radio-inline,\n.has-error .checkbox-inline {\n  color: #a94442;\n}\n.has-error .form-control {\n  border-color: #a94442;\n  -webkit-box-shadow: inset 0 1px 1px rgba(0, 0, 0, .075);\n          box-shadow: inset 0 1px 1px rgba(0, 0, 0, .075);\n}\n.has-error .form-control:focus {\n  border-color: #843534;\n  -webkit-box-shadow: inset 0 1px 1px rgba(0, 0, 0, .075), 0 0 6px #ce8483;\n          box-shadow: inset 0 1px 1px rgba(0, 0, 0, .075), 0 0 6px #ce8483;\n}\n.has-error .input-group-addon {\n  color: #a94442;\n  background-color: #f2dede;\n  border-color: #a94442;\n}\n.has-error .form-control-feedback {\n  color: #a94442;\n}\n.form-control-static {\n  margin-bottom: 0;\n}\n.help-block {\n  display: block;\n  margin-top: 5px;\n  margin-bottom: 10px;\n  color: #737373;\n}\n@media (min-width: 768px) {\n  .form-inline .form-group {\n    display: inline-block;\n    margin-bottom: 0;\n    vertical-align: middle;\n  }\n  .form-inline .form-control {\n    display: inline-block;\n    width: auto;\n    vertical-align: middle;\n  }\n  .form-inline .control-label {\n    margin-bottom: 0;\n    vertical-align: middle;\n  }\n  .form-inline .radio,\n  .form-inline .checkbox {\n    display: inline-block;\n    padding-left: 0;\n    margin-top: 0;\n    margin-bottom: 0;\n    vertical-align: middle;\n  }\n  .form-inline .radio input[type=\"radio\"],\n  .form-inline .checkbox input[type=\"checkbox\"] {\n    float: none;\n    margin-left: 0;\n  }\n  .form-inline .has-feedback .form-control-feedback {\n    top: 0;\n  }\n}\n.form-horizontal .control-label,\n.form-horizontal .radio,\n.form-horizontal .checkbox,\n.form-horizontal .radio-inline,\n.form-horizontal .checkbox-inline {\n  padding-top: 7px;\n  margin-top: 0;\n  margin-bottom: 0;\n}\n.form-horizontal .radio,\n.form-horizontal .checkbox {\n  min-height: 27px;\n}\n.form-horizontal .form-group {\n  margin-right: -15px;\n  margin-left: -15px;\n}\n.form-horizontal .form-control-static {\n  padding-top: 7px;\n}\n@media (min-width: 768px) {\n  .form-horizontal .control-label {\n    text-align: right;\n  }\n}\n.form-horizontal .has-feedback .form-control-feedback {\n  top: 0;\n  right: 15px;\n}\n.btn {\n  display: inline-block;\n  padding: 6px 12px;\n  margin-bottom: 0;\n  font-size: 14px;\n  font-weight: normal;\n  line-height: 1.428571429;\n  text-align: center;\n  white-space: nowrap;\n  vertical-align: middle;\n  cursor: pointer;\n  -webkit-user-select: none;\n     -moz-user-select: none;\n      -ms-user-select: none;\n       -o-user-select: none;\n          user-select: none;\n  background-image: none;\n  border: 1px solid transparent;\n  border-radius: 4px;\n}\n.btn:focus {\n  outline: thin dotted;\n  outline: 5px auto -webkit-focus-ring-color;\n  outline-offset: -2px;\n}\n.btn:hover,\n.btn:focus {\n  color: #333;\n  text-decoration: none;\n}\n.btn:active,\n.btn.active {\n  background-image: none;\n  outline: 0;\n  -webkit-box-shadow: inset 0 3px 5px rgba(0, 0, 0, .125);\n          box-shadow: inset 0 3px 5px rgba(0, 0, 0, .125);\n}\n.btn.disabled,\n.btn[disabled],\nfieldset[disabled] .btn {\n  pointer-events: none;\n  cursor: not-allowed;\n  filter: alpha(opacity=65);\n  -webkit-box-shadow: none;\n          box-shadow: none;\n  opacity: .65;\n}\n.btn-default {\n  color: #333;\n  background-color: #fff;\n  border-color: #ccc;\n}\n.btn-default:hover,\n.btn-default:focus,\n.btn-default:active,\n.btn-default.active,\n.open .dropdown-toggle.btn-default {\n  color: #333;\n  background-color: #ebebeb;\n  border-color: #adadad;\n}\n.btn-default:active,\n.btn-default.active,\n.open .dropdown-toggle.btn-default {\n  background-image: none;\n}\n.btn-default.disabled,\n.btn-default[disabled],\nfieldset[disabled] .btn-default,\n.btn-default.disabled:hover,\n.btn-default[disabled]:hover,\nfieldset[disabled] .btn-default:hover,\n.btn-default.disabled:focus,\n.btn-default[disabled]:focus,\nfieldset[disabled] .btn-default:focus,\n.btn-default.disabled:active,\n.btn-default[disabled]:active,\nfieldset[disabled] .btn-default:active,\n.btn-default.disabled.active,\n.btn-default[disabled].active,\nfieldset[disabled] .btn-default.active {\n  background-color: #fff;\n  border-color: #ccc;\n}\n.btn-default .badge {\n  color: #fff;\n  background-color: #333;\n}\n.btn-primary {\n  color: #fff;\n  background-color: #428bca;\n  border-color: #357ebd;\n}\n.btn-primary:hover,\n.btn-primary:focus,\n.btn-primary:active,\n.btn-primary.active,\n.open .dropdown-toggle.btn-primary {\n  color: #fff;\n  background-color: #3276b1;\n  border-color: #285e8e;\n}\n.btn-primary:active,\n.btn-primary.active,\n.open .dropdown-toggle.btn-primary {\n  background-image: none;\n}\n.btn-primary.disabled,\n.btn-primary[disabled],\nfieldset[disabled] .btn-primary,\n.btn-primary.disabled:hover,\n.btn-primary[disabled]:hover,\nfieldset[disabled] .btn-primary:hover,\n.btn-primary.disabled:focus,\n.btn-primary[disabled]:focus,\nfieldset[disabled] .btn-primary:focus,\n.btn-primary.disabled:active,\n.btn-primary[disabled]:active,\nfieldset[disabled] .btn-primary:active,\n.btn-primary.disabled.active,\n.btn-primary[disabled].active,\nfieldset[disabled] .btn-primary.active {\n  background-color: #428bca;\n  border-color: #357ebd;\n}\n.btn-primary .badge {\n  color: #428bca;\n  background-color: #fff;\n}\n.btn-success {\n  color: #fff;\n  background-color: #5cb85c;\n  border-color: #4cae4c;\n}\n.btn-success:hover,\n.btn-success:focus,\n.btn-success:active,\n.btn-success.active,\n.open .dropdown-toggle.btn-success {\n  color: #fff;\n  background-color: #47a447;\n  border-color: #398439;\n}\n.btn-success:active,\n.btn-success.active,\n.open .dropdown-toggle.btn-success {\n  background-image: none;\n}\n.btn-success.disabled,\n.btn-success[disabled],\nfieldset[disabled] .btn-success,\n.btn-success.disabled:hover,\n.btn-success[disabled]:hover,\nfieldset[disabled] .btn-success:hover,\n.btn-success.disabled:focus,\n.btn-success[disabled]:focus,\nfieldset[disabled] .btn-success:focus,\n.btn-success.disabled:active,\n.btn-success[disabled]:active,\nfieldset[disabled] .btn-success:active,\n.btn-success.disabled.active,\n.btn-success[disabled].active,\nfieldset[disabled] .btn-success.active {\n  background-color: #5cb85c;\n  border-color: #4cae4c;\n}\n.btn-success .badge {\n  color: #5cb85c;\n  background-color: #fff;\n}\n.btn-info {\n  color: #fff;\n  background-color: #5bc0de;\n  border-color: #46b8da;\n}\n.btn-info:hover,\n.btn-info:focus,\n.btn-info:active,\n.btn-info.active,\n.open .dropdown-toggle.btn-info {\n  color: #fff;\n  background-color: #39b3d7;\n  border-color: #269abc;\n}\n.btn-info:active,\n.btn-info.active,\n.open .dropdown-toggle.btn-info {\n  background-image: none;\n}\n.btn-info.disabled,\n.btn-info[disabled],\nfieldset[disabled] .btn-info,\n.btn-info.disabled:hover,\n.btn-info[disabled]:hover,\nfieldset[disabled] .btn-info:hover,\n.btn-info.disabled:focus,\n.btn-info[disabled]:focus,\nfieldset[disabled] .btn-info:focus,\n.btn-info.disabled:active,\n.btn-info[disabled]:active,\nfieldset[disabled] .btn-info:active,\n.btn-info.disabled.active,\n.btn-info[disabled].active,\nfieldset[disabled] .btn-info.active {\n  background-color: #5bc0de;\n  border-color: #46b8da;\n}\n.btn-info .badge {\n  color: #5bc0de;\n  background-color: #fff;\n}\n.btn-warning {\n  color: #fff;\n  background-color: #f0ad4e;\n  border-color: #eea236;\n}\n.btn-warning:hover,\n.btn-warning:focus,\n.btn-warning:active,\n.btn-warning.active,\n.open .dropdown-toggle.btn-warning {\n  color: #fff;\n  background-color: #ed9c28;\n  border-color: #d58512;\n}\n.btn-warning:active,\n.btn-warning.active,\n.open .dropdown-toggle.btn-warning {\n  background-image: none;\n}\n.btn-warning.disabled,\n.btn-warning[disabled],\nfieldset[disabled] .btn-warning,\n.btn-warning.disabled:hover,\n.btn-warning[disabled]:hover,\nfieldset[disabled] .btn-warning:hover,\n.btn-warning.disabled:focus,\n.btn-warning[disabled]:focus,\nfieldset[disabled] .btn-warning:focus,\n.btn-warning.disabled:active,\n.btn-warning[disabled]:active,\nfieldset[disabled] .btn-warning:active,\n.btn-warning.disabled.active,\n.btn-warning[disabled].active,\nfieldset[disabled] .btn-warning.active {\n  background-color: #f0ad4e;\n  border-color: #eea236;\n}\n.btn-warning .badge {\n  color: #f0ad4e;\n  background-color: #fff;\n}\n.btn-danger {\n  color: #fff;\n  background-color: #d9534f;\n  border-color: #d43f3a;\n}\n.btn-danger:hover,\n.btn-danger:focus,\n.btn-danger:active,\n.btn-danger.active,\n.open .dropdown-toggle.btn-danger {\n  color: #fff;\n  background-color: #d2322d;\n  border-color: #ac2925;\n}\n.btn-danger:active,\n.btn-danger.active,\n.open .dropdown-toggle.btn-danger {\n  background-image: none;\n}\n.btn-danger.disabled,\n.btn-danger[disabled],\nfieldset[disabled] .btn-danger,\n.btn-danger.disabled:hover,\n.btn-danger[disabled]:hover,\nfieldset[disabled] .btn-danger:hover,\n.btn-danger.disabled:focus,\n.btn-danger[disabled]:focus,\nfieldset[disabled] .btn-danger:focus,\n.btn-danger.disabled:active,\n.btn-danger[disabled]:active,\nfieldset[disabled] .btn-danger:active,\n.btn-danger.disabled.active,\n.btn-danger[disabled].active,\nfieldset[disabled] .btn-danger.active {\n  background-color: #d9534f;\n  border-color: #d43f3a;\n}\n.btn-danger .badge {\n  color: #d9534f;\n  background-color: #fff;\n}\n.btn-link {\n  font-weight: normal;\n  color: #428bca;\n  cursor: pointer;\n  border-radius: 0;\n}\n.btn-link,\n.btn-link:active,\n.btn-link[disabled],\nfieldset[disabled] .btn-link {\n  background-color: transparent;\n  -webkit-box-shadow: none;\n          box-shadow: none;\n}\n.btn-link,\n.btn-link:hover,\n.btn-link:focus,\n.btn-link:active {\n  border-color: transparent;\n}\n.btn-link:hover,\n.btn-link:focus {\n  color: #2a6496;\n  text-decoration: underline;\n  background-color: transparent;\n}\n.btn-link[disabled]:hover,\nfieldset[disabled] .btn-link:hover,\n.btn-link[disabled]:focus,\nfieldset[disabled] .btn-link:focus {\n  color: #999;\n  text-decoration: none;\n}\n.btn-lg {\n  padding: 10px 16px;\n  font-size: 18px;\n  line-height: 1.33;\n  border-radius: 6px;\n}\n.btn-sm {\n  padding: 5px 10px;\n  font-size: 12px;\n  line-height: 1.5;\n  border-radius: 3px;\n}\n.btn-xs {\n  padding: 1px 5px;\n  font-size: 12px;\n  line-height: 1.5;\n  border-radius: 3px;\n}\n.btn-block {\n  display: block;\n  width: 100%;\n  padding-right: 0;\n  padding-left: 0;\n}\n.btn-block + .btn-block {\n  margin-top: 5px;\n}\ninput[type=\"submit\"].btn-block,\ninput[type=\"reset\"].btn-block,\ninput[type=\"button\"].btn-block {\n  width: 100%;\n}\n.fade {\n  opacity: 0;\n  -webkit-transition: opacity .15s linear;\n          transition: opacity .15s linear;\n}\n.fade.in {\n  opacity: 1;\n}\n.collapse {\n  display: none;\n}\n.collapse.in {\n  display: block;\n}\n.collapsing {\n  position: relative;\n  height: 0;\n  overflow: hidden;\n  -webkit-transition: height .35s ease;\n          transition: height .35s ease;\n}\n@font-face {\n  font-family: 'Glyphicons Halflings';\n\n  src: url('../fonts/glyphicons-halflings-regular.eot');\n  src: url('../fonts/glyphicons-halflings-regular.eot?#iefix') format('embedded-opentype'), url('../fonts/glyphicons-halflings-regular.woff') format('woff'), url('../fonts/glyphicons-halflings-regular.ttf') format('truetype'), url('../fonts/glyphicons-halflings-regular.svg#glyphicons_halflingsregular') format('svg');\n}\n.glyphicon {\n  position: relative;\n  top: 1px;\n  display: inline-block;\n  font-family: 'Glyphicons Halflings';\n  font-style: normal;\n  font-weight: normal;\n  line-height: 1;\n\n  -webkit-font-smoothing: antialiased;\n  -moz-osx-font-smoothing: grayscale;\n}\n.glyphicon-asterisk:before {\n  content: \"\\2a\";\n}\n.glyphicon-plus:before {\n  content: \"\\2b\";\n}\n.glyphicon-euro:before {\n  content: \"\\20ac\";\n}\n.glyphicon-minus:before {\n  content: \"\\2212\";\n}\n.glyphicon-cloud:before {\n  content: \"\\2601\";\n}\n.glyphicon-envelope:before {\n  content: \"\\2709\";\n}\n.glyphicon-pencil:before {\n  content: \"\\270f\";\n}\n.glyphicon-glass:before {\n  content: \"\\e001\";\n}\n.glyphicon-music:before {\n  content: \"\\e002\";\n}\n.glyphicon-search:before {\n  content: \"\\e003\";\n}\n.glyphicon-heart:before {\n  content: \"\\e005\";\n}\n.glyphicon-star:before {\n  content: \"\\e006\";\n}\n.glyphicon-star-empty:before {\n  content: \"\\e007\";\n}\n.glyphicon-user:before {\n  content: \"\\e008\";\n}\n.glyphicon-film:before {\n  content: \"\\e009\";\n}\n.glyphicon-th-large:before {\n  content: \"\\e010\";\n}\n.glyphicon-th:before {\n  content: \"\\e011\";\n}\n.glyphicon-th-list:before {\n  content: \"\\e012\";\n}\n.glyphicon-ok:before {\n  content: \"\\e013\";\n}\n.glyphicon-remove:before {\n  content: \"\\e014\";\n}\n.glyphicon-zoom-in:before {\n  content: \"\\e015\";\n}\n.glyphicon-zoom-out:before {\n  content: \"\\e016\";\n}\n.glyphicon-off:before {\n  content: \"\\e017\";\n}\n.glyphicon-signal:before {\n  content: \"\\e018\";\n}\n.glyphicon-cog:before {\n  content: \"\\e019\";\n}\n.glyphicon-trash:before {\n  content: \"\\e020\";\n}\n.glyphicon-home:before {\n  content: \"\\e021\";\n}\n.glyphicon-file:before {\n  content: \"\\e022\";\n}\n.glyphicon-time:before {\n  content: \"\\e023\";\n}\n.glyphicon-road:before {\n  content: \"\\e024\";\n}\n.glyphicon-download-alt:before {\n  content: \"\\e025\";\n}\n.glyphicon-download:before {\n  content: \"\\e026\";\n}\n.glyphicon-upload:before {\n  content: \"\\e027\";\n}\n.glyphicon-inbox:before {\n  content: \"\\e028\";\n}\n.glyphicon-play-circle:before {\n  content: \"\\e029\";\n}\n.glyphicon-repeat:before {\n  content: \"\\e030\";\n}\n.glyphicon-refresh:before {\n  content: \"\\e031\";\n}\n.glyphicon-list-alt:before {\n  content: \"\\e032\";\n}\n.glyphicon-lock:before {\n  content: \"\\e033\";\n}\n.glyphicon-flag:before {\n  content: \"\\e034\";\n}\n.glyphicon-headphones:before {\n  content: \"\\e035\";\n}\n.glyphicon-volume-off:before {\n  content: \"\\e036\";\n}\n.glyphicon-volume-down:before {\n  content: \"\\e037\";\n}\n.glyphicon-volume-up:before {\n  content: \"\\e038\";\n}\n.glyphicon-qrcode:before {\n  content: \"\\e039\";\n}\n.glyphicon-barcode:before {\n  content: \"\\e040\";\n}\n.glyphicon-tag:before {\n  content: \"\\e041\";\n}\n.glyphicon-tags:before {\n  content: \"\\e042\";\n}\n.glyphicon-book:before {\n  content: \"\\e043\";\n}\n.glyphicon-bookmark:before {\n  content: \"\\e044\";\n}\n.glyphicon-print:before {\n  content: \"\\e045\";\n}\n.glyphicon-camera:before {\n  content: \"\\e046\";\n}\n.glyphicon-font:before {\n  content: \"\\e047\";\n}\n.glyphicon-bold:before {\n  content: \"\\e048\";\n}\n.glyphicon-italic:before {\n  content: \"\\e049\";\n}\n.glyphicon-text-height:before {\n  content: \"\\e050\";\n}\n.glyphicon-text-width:before {\n  content: \"\\e051\";\n}\n.glyphicon-align-left:before {\n  content: \"\\e052\";\n}\n.glyphicon-align-center:before {\n  content: \"\\e053\";\n}\n.glyphicon-align-right:before {\n  content: \"\\e054\";\n}\n.glyphicon-align-justify:before {\n  content: \"\\e055\";\n}\n.glyphicon-list:before {\n  content: \"\\e056\";\n}\n.glyphicon-indent-left:before {\n  content: \"\\e057\";\n}\n.glyphicon-indent-right:before {\n  content: \"\\e058\";\n}\n.glyphicon-facetime-video:before {\n  content: \"\\e059\";\n}\n.glyphicon-picture:before {\n  content: \"\\e060\";\n}\n.glyphicon-map-marker:before {\n  content: \"\\e062\";\n}\n.glyphicon-adjust:before {\n  content: \"\\e063\";\n}\n.glyphicon-tint:before {\n  content: \"\\e064\";\n}\n.glyphicon-edit:before {\n  content: \"\\e065\";\n}\n.glyphicon-share:before {\n  content: \"\\e066\";\n}\n.glyphicon-check:before {\n  content: \"\\e067\";\n}\n.glyphicon-move:before {\n  content: \"\\e068\";\n}\n.glyphicon-step-backward:before {\n  content: \"\\e069\";\n}\n.glyphicon-fast-backward:before {\n  content: \"\\e070\";\n}\n.glyphicon-backward:before {\n  content: \"\\e071\";\n}\n.glyphicon-play:before {\n  content: \"\\e072\";\n}\n.glyphicon-pause:before {\n  content: \"\\e073\";\n}\n.glyphicon-stop:before {\n  content: \"\\e074\";\n}\n.glyphicon-forward:before {\n  content: \"\\e075\";\n}\n.glyphicon-fast-forward:before {\n  content: \"\\e076\";\n}\n.glyphicon-step-forward:before {\n  content: \"\\e077\";\n}\n.glyphicon-eject:before {\n  content: \"\\e078\";\n}\n.glyphicon-chevron-left:before {\n  content: \"\\e079\";\n}\n.glyphicon-chevron-right:before {\n  content: \"\\e080\";\n}\n.glyphicon-plus-sign:before {\n  content: \"\\e081\";\n}\n.glyphicon-minus-sign:before {\n  content: \"\\e082\";\n}\n.glyphicon-remove-sign:before {\n  content: \"\\e083\";\n}\n.glyphicon-ok-sign:before {\n  content: \"\\e084\";\n}\n.glyphicon-question-sign:before {\n  content: \"\\e085\";\n}\n.glyphicon-info-sign:before {\n  content: \"\\e086\";\n}\n.glyphicon-screenshot:before {\n  content: \"\\e087\";\n}\n.glyphicon-remove-circle:before {\n  content: \"\\e088\";\n}\n.glyphicon-ok-circle:before {\n  content: \"\\e089\";\n}\n.glyphicon-ban-circle:before {\n  content: \"\\e090\";\n}\n.glyphicon-arrow-left:before {\n  content: \"\\e091\";\n}\n.glyphicon-arrow-right:before {\n  content: \"\\e092\";\n}\n.glyphicon-arrow-up:before {\n  content: \"\\e093\";\n}\n.glyphicon-arrow-down:before {\n  content: \"\\e094\";\n}\n.glyphicon-share-alt:before {\n  content: \"\\e095\";\n}\n.glyphicon-resize-full:before {\n  content: \"\\e096\";\n}\n.glyphicon-resize-small:before {\n  content: \"\\e097\";\n}\n.glyphicon-exclamation-sign:before {\n  content: \"\\e101\";\n}\n.glyphicon-gift:before {\n  content: \"\\e102\";\n}\n.glyphicon-leaf:before {\n  content: \"\\e103\";\n}\n.glyphicon-fire:before {\n  content: \"\\e104\";\n}\n.glyphicon-eye-open:before {\n  content: \"\\e105\";\n}\n.glyphicon-eye-close:before {\n  content: \"\\e106\";\n}\n.glyphicon-warning-sign:before {\n  content: \"\\e107\";\n}\n.glyphicon-plane:before {\n  content: \"\\e108\";\n}\n.glyphicon-calendar:before {\n  content: \"\\e109\";\n}\n.glyphicon-random:before {\n  content: \"\\e110\";\n}\n.glyphicon-comment:before {\n  content: \"\\e111\";\n}\n.glyphicon-magnet:before {\n  content: \"\\e112\";\n}\n.glyphicon-chevron-up:before {\n  content: \"\\e113\";\n}\n.glyphicon-chevron-down:before {\n  content: \"\\e114\";\n}\n.glyphicon-retweet:before {\n  content: \"\\e115\";\n}\n.glyphicon-shopping-cart:before {\n  content: \"\\e116\";\n}\n.glyphicon-folder-close:before {\n  content: \"\\e117\";\n}\n.glyphicon-folder-open:before {\n  content: \"\\e118\";\n}\n.glyphicon-resize-vertical:before {\n  content: \"\\e119\";\n}\n.glyphicon-resize-horizontal:before {\n  content: \"\\e120\";\n}\n.glyphicon-hdd:before {\n  content: \"\\e121\";\n}\n.glyphicon-bullhorn:before {\n  content: \"\\e122\";\n}\n.glyphicon-bell:before {\n  content: \"\\e123\";\n}\n.glyphicon-certificate:before {\n  content: \"\\e124\";\n}\n.glyphicon-thumbs-up:before {\n  content: \"\\e125\";\n}\n.glyphicon-thumbs-down:before {\n  content: \"\\e126\";\n}\n.glyphicon-hand-right:before {\n  content: \"\\e127\";\n}\n.glyphicon-hand-left:before {\n  content: \"\\e128\";\n}\n.glyphicon-hand-up:before {\n  content: \"\\e129\";\n}\n.glyphicon-hand-down:before {\n  content: \"\\e130\";\n}\n.glyphicon-circle-arrow-right:before {\n  content: \"\\e131\";\n}\n.glyphicon-circle-arrow-left:before {\n  content: \"\\e132\";\n}\n.glyphicon-circle-arrow-up:before {\n  content: \"\\e133\";\n}\n.glyphicon-circle-arrow-down:before {\n  content: \"\\e134\";\n}\n.glyphicon-globe:before {\n  content: \"\\e135\";\n}\n.glyphicon-wrench:before {\n  content: \"\\e136\";\n}\n.glyphicon-tasks:before {\n  content: \"\\e137\";\n}\n.glyphicon-filter:before {\n  content: \"\\e138\";\n}\n.glyphicon-briefcase:before {\n  content: \"\\e139\";\n}\n.glyphicon-fullscreen:before {\n  content: \"\\e140\";\n}\n.glyphicon-dashboard:before {\n  content: \"\\e141\";\n}\n.glyphicon-paperclip:before {\n  content: \"\\e142\";\n}\n.glyphicon-heart-empty:before {\n  content: \"\\e143\";\n}\n.glyphicon-link:before {\n  content: \"\\e144\";\n}\n.glyphicon-phone:before {\n  content: \"\\e145\";\n}\n.glyphicon-pushpin:before {\n  content: \"\\e146\";\n}\n.glyphicon-usd:before {\n  content: \"\\e148\";\n}\n.glyphicon-gbp:before {\n  content: \"\\e149\";\n}\n.glyphicon-sort:before {\n  content: \"\\e150\";\n}\n.glyphicon-sort-by-alphabet:before {\n  content: \"\\e151\";\n}\n.glyphicon-sort-by-alphabet-alt:before {\n  content: \"\\e152\";\n}\n.glyphicon-sort-by-order:before {\n  content: \"\\e153\";\n}\n.glyphicon-sort-by-order-alt:before {\n  content: \"\\e154\";\n}\n.glyphicon-sort-by-attributes:before {\n  content: \"\\e155\";\n}\n.glyphicon-sort-by-attributes-alt:before {\n  content: \"\\e156\";\n}\n.glyphicon-unchecked:before {\n  content: \"\\e157\";\n}\n.glyphicon-expand:before {\n  content: \"\\e158\";\n}\n.glyphicon-collapse-down:before {\n  content: \"\\e159\";\n}\n.glyphicon-collapse-up:before {\n  content: \"\\e160\";\n}\n.glyphicon-log-in:before {\n  content: \"\\e161\";\n}\n.glyphicon-flash:before {\n  content: \"\\e162\";\n}\n.glyphicon-log-out:before {\n  content: \"\\e163\";\n}\n.glyphicon-new-window:before {\n  content: \"\\e164\";\n}\n.glyphicon-record:before {\n  content: \"\\e165\";\n}\n.glyphicon-save:before {\n  content: \"\\e166\";\n}\n.glyphicon-open:before {\n  content: \"\\e167\";\n}\n.glyphicon-saved:before {\n  content: \"\\e168\";\n}\n.glyphicon-import:before {\n  content: \"\\e169\";\n}\n.glyphicon-export:before {\n  content: \"\\e170\";\n}\n.glyphicon-send:before {\n  content: \"\\e171\";\n}\n.glyphicon-floppy-disk:before {\n  content: \"\\e172\";\n}\n.glyphicon-floppy-saved:before {\n  content: \"\\e173\";\n}\n.glyphicon-floppy-remove:before {\n  content: \"\\e174\";\n}\n.glyphicon-floppy-save:before {\n  content: \"\\e175\";\n}\n.glyphicon-floppy-open:before {\n  content: \"\\e176\";\n}\n.glyphicon-credit-card:before {\n  content: \"\\e177\";\n}\n.glyphicon-transfer:before {\n  content: \"\\e178\";\n}\n.glyphicon-cutlery:before {\n  content: \"\\e179\";\n}\n.glyphicon-header:before {\n  content: \"\\e180\";\n}\n.glyphicon-compressed:before {\n  content: \"\\e181\";\n}\n.glyphicon-earphone:before {\n  content: \"\\e182\";\n}\n.glyphicon-phone-alt:before {\n  content: \"\\e183\";\n}\n.glyphicon-tower:before {\n  content: \"\\e184\";\n}\n.glyphicon-stats:before {\n  content: \"\\e185\";\n}\n.glyphicon-sd-video:before {\n  content: \"\\e186\";\n}\n.glyphicon-hd-video:before {\n  content: \"\\e187\";\n}\n.glyphicon-subtitles:before {\n  content: \"\\e188\";\n}\n.glyphicon-sound-stereo:before {\n  content: \"\\e189\";\n}\n.glyphicon-sound-dolby:before {\n  content: \"\\e190\";\n}\n.glyphicon-sound-5-1:before {\n  content: \"\\e191\";\n}\n.glyphicon-sound-6-1:before {\n  content: \"\\e192\";\n}\n.glyphicon-sound-7-1:before {\n  content: \"\\e193\";\n}\n.glyphicon-copyright-mark:before {\n  content: \"\\e194\";\n}\n.glyphicon-registration-mark:before {\n  content: \"\\e195\";\n}\n.glyphicon-cloud-download:before {\n  content: \"\\e197\";\n}\n.glyphicon-cloud-upload:before {\n  content: \"\\e198\";\n}\n.glyphicon-tree-conifer:before {\n  content: \"\\e199\";\n}\n.glyphicon-tree-deciduous:before {\n  content: \"\\e200\";\n}\n.caret {\n  display: inline-block;\n  width: 0;\n  height: 0;\n  margin-left: 2px;\n  vertical-align: middle;\n  border-top: 4px solid;\n  border-right: 4px solid transparent;\n  border-left: 4px solid transparent;\n}\n.dropdown {\n  position: relative;\n}\n.dropdown-toggle:focus {\n  outline: 0;\n}\n.dropdown-menu {\n  position: absolute;\n  top: 100%;\n  left: 0;\n  z-index: 1000;\n  display: none;\n  float: left;\n  min-width: 160px;\n  padding: 5px 0;\n  margin: 2px 0 0;\n  font-size: 14px;\n  list-style: none;\n  background-color: #fff;\n  background-clip: padding-box;\n  border: 1px solid #ccc;\n  border: 1px solid rgba(0, 0, 0, .15);\n  border-radius: 4px;\n  -webkit-box-shadow: 0 6px 12px rgba(0, 0, 0, .175);\n          box-shadow: 0 6px 12px rgba(0, 0, 0, .175);\n}\n.dropdown-menu.pull-right {\n  right: 0;\n  left: auto;\n}\n.dropdown-menu .divider {\n  height: 1px;\n  margin: 9px 0;\n  overflow: hidden;\n  background-color: #e5e5e5;\n}\n.dropdown-menu > li > a {\n  display: block;\n  padding: 3px 20px;\n  clear: both;\n  font-weight: normal;\n  line-height: 1.428571429;\n  color: #333;\n  white-space: nowrap;\n}\n.dropdown-menu > li > a:hover,\n.dropdown-menu > li > a:focus {\n  color: #262626;\n  text-decoration: none;\n  background-color: #f5f5f5;\n}\n.dropdown-menu > .active > a,\n.dropdown-menu > .active > a:hover,\n.dropdown-menu > .active > a:focus {\n  color: #fff;\n  text-decoration: none;\n  background-color: #428bca;\n  outline: 0;\n}\n.dropdown-menu > .disabled > a,\n.dropdown-menu > .disabled > a:hover,\n.dropdown-menu > .disabled > a:focus {\n  color: #999;\n}\n.dropdown-menu > .disabled > a:hover,\n.dropdown-menu > .disabled > a:focus {\n  text-decoration: none;\n  cursor: not-allowed;\n  background-color: transparent;\n  background-image: none;\n  filter: progid:DXImageTransform.Microsoft.gradient(enabled = false);\n}\n.open > .dropdown-menu {\n  display: block;\n}\n.open > a {\n  outline: 0;\n}\n.dropdown-menu-right {\n  right: 0;\n  left: auto;\n}\n.dropdown-menu-left {\n  right: auto;\n  left: 0;\n}\n.dropdown-header {\n  display: block;\n  padding: 3px 20px;\n  font-size: 12px;\n  line-height: 1.428571429;\n  color: #999;\n}\n.dropdown-backdrop {\n  position: fixed;\n  top: 0;\n  right: 0;\n  bottom: 0;\n  left: 0;\n  z-index: 990;\n}\n.pull-right > .dropdown-menu {\n  right: 0;\n  left: auto;\n}\n.dropup .caret,\n.navbar-fixed-bottom .dropdown .caret {\n  content: \"\";\n  border-top: 0;\n  border-bottom: 4px solid;\n}\n.dropup .dropdown-menu,\n.navbar-fixed-bottom .dropdown .dropdown-menu {\n  top: auto;\n  bottom: 100%;\n  margin-bottom: 1px;\n}\n@media (min-width: 768px) {\n  .navbar-right .dropdown-menu {\n    right: 0;\n    left: auto;\n  }\n  .navbar-right .dropdown-menu-left {\n    right: auto;\n    left: 0;\n  }\n}\n.btn-group,\n.btn-group-vertical {\n  position: relative;\n  display: inline-block;\n  vertical-align: middle;\n}\n.btn-group > .btn,\n.btn-group-vertical > .btn {\n  position: relative;\n  float: left;\n}\n.btn-group > .btn:hover,\n.btn-group-vertical > .btn:hover,\n.btn-group > .btn:focus,\n.btn-group-vertical > .btn:focus,\n.btn-group > .btn:active,\n.btn-group-vertical > .btn:active,\n.btn-group > .btn.active,\n.btn-group-vertical > .btn.active {\n  z-index: 2;\n}\n.btn-group > .btn:focus,\n.btn-group-vertical > .btn:focus {\n  outline: none;\n}\n.btn-group .btn + .btn,\n.btn-group .btn + .btn-group,\n.btn-group .btn-group + .btn,\n.btn-group .btn-group + .btn-group {\n  margin-left: -1px;\n}\n.btn-toolbar {\n  margin-left: -5px;\n}\n.btn-toolbar .btn-group,\n.btn-toolbar .input-group {\n  float: left;\n}\n.btn-toolbar > .btn,\n.btn-toolbar > .btn-group,\n.btn-toolbar > .input-group {\n  margin-left: 5px;\n}\n.btn-group > .btn:not(:first-child):not(:last-child):not(.dropdown-toggle) {\n  border-radius: 0;\n}\n.btn-group > .btn:first-child {\n  margin-left: 0;\n}\n.btn-group > .btn:first-child:not(:last-child):not(.dropdown-toggle) {\n  border-top-right-radius: 0;\n  border-bottom-right-radius: 0;\n}\n.btn-group > .btn:last-child:not(:first-child),\n.btn-group > .dropdown-toggle:not(:first-child) {\n  border-top-left-radius: 0;\n  border-bottom-left-radius: 0;\n}\n.btn-group > .btn-group {\n  float: left;\n}\n.btn-group > .btn-group:not(:first-child):not(:last-child) > .btn {\n  border-radius: 0;\n}\n.btn-group > .btn-group:first-child > .btn:last-child,\n.btn-group > .btn-group:first-child > .dropdown-toggle {\n  border-top-right-radius: 0;\n  border-bottom-right-radius: 0;\n}\n.btn-group > .btn-group:last-child > .btn:first-child {\n  border-top-left-radius: 0;\n  border-bottom-left-radius: 0;\n}\n.btn-group .dropdown-toggle:active,\n.btn-group.open .dropdown-toggle {\n  outline: 0;\n}\n.btn-group-xs > .btn {\n  padding: 1px 5px;\n  font-size: 12px;\n  line-height: 1.5;\n  border-radius: 3px;\n}\n.btn-group-sm > .btn {\n  padding: 5px 10px;\n  font-size: 12px;\n  line-height: 1.5;\n  border-radius: 3px;\n}\n.btn-group-lg > .btn {\n  padding: 10px 16px;\n  font-size: 18px;\n  line-height: 1.33;\n  border-radius: 6px;\n}\n.btn-group > .btn + .dropdown-toggle {\n  padding-right: 8px;\n  padding-left: 8px;\n}\n.btn-group > .btn-lg + .dropdown-toggle {\n  padding-right: 12px;\n  padding-left: 12px;\n}\n.btn-group.open .dropdown-toggle {\n  -webkit-box-shadow: inset 0 3px 5px rgba(0, 0, 0, .125);\n          box-shadow: inset 0 3px 5px rgba(0, 0, 0, .125);\n}\n.btn-group.open .dropdown-toggle.btn-link {\n  -webkit-box-shadow: none;\n          box-shadow: none;\n}\n.btn .caret {\n  margin-left: 0;\n}\n.btn-lg .caret {\n  border-width: 5px 5px 0;\n  border-bottom-width: 0;\n}\n.dropup .btn-lg .caret {\n  border-width: 0 5px 5px;\n}\n.btn-group-vertical > .btn,\n.btn-group-vertical > .btn-group,\n.btn-group-vertical > .btn-group > .btn {\n  display: block;\n  float: none;\n  width: 100%;\n  max-width: 100%;\n}\n.btn-group-vertical > .btn-group > .btn {\n  float: none;\n}\n.btn-group-vertical > .btn + .btn,\n.btn-group-vertical > .btn + .btn-group,\n.btn-group-vertical > .btn-group + .btn,\n.btn-group-vertical > .btn-group + .btn-group {\n  margin-top: -1px;\n  margin-left: 0;\n}\n.btn-group-vertical > .btn:not(:first-child):not(:last-child) {\n  border-radius: 0;\n}\n.btn-group-vertical > .btn:first-child:not(:last-child) {\n  border-top-right-radius: 4px;\n  border-bottom-right-radius: 0;\n  border-bottom-left-radius: 0;\n}\n.btn-group-vertical > .btn:last-child:not(:first-child) {\n  border-top-left-radius: 0;\n  border-top-right-radius: 0;\n  border-bottom-left-radius: 4px;\n}\n.btn-group-vertical > .btn-group:not(:first-child):not(:last-child) > .btn {\n  border-radius: 0;\n}\n.btn-group-vertical > .btn-group:first-child:not(:last-child) > .btn:last-child,\n.btn-group-vertical > .btn-group:first-child:not(:last-child) > .dropdown-toggle {\n  border-bottom-right-radius: 0;\n  border-bottom-left-radius: 0;\n}\n.btn-group-vertical > .btn-group:last-child:not(:first-child) > .btn:first-child {\n  border-top-left-radius: 0;\n  border-top-right-radius: 0;\n}\n.btn-group-justified {\n  display: table;\n  width: 100%;\n  table-layout: fixed;\n  border-collapse: separate;\n}\n.btn-group-justified > .btn,\n.btn-group-justified > .btn-group {\n  display: table-cell;\n  float: none;\n  width: 1%;\n}\n.btn-group-justified > .btn-group .btn {\n  width: 100%;\n}\n[data-toggle=\"buttons\"] > .btn > input[type=\"radio\"],\n[data-toggle=\"buttons\"] > .btn > input[type=\"checkbox\"] {\n  display: none;\n}\n.input-group {\n  position: relative;\n  display: table;\n  border-collapse: separate;\n}\n.input-group[class*=\"col-\"] {\n  float: none;\n  padding-right: 0;\n  padding-left: 0;\n}\n.input-group .form-control {\n  float: left;\n  width: 100%;\n  margin-bottom: 0;\n}\n.input-group-lg > .form-control,\n.input-group-lg > .input-group-addon,\n.input-group-lg > .input-group-btn > .btn {\n  height: 46px;\n  padding: 10px 16px;\n  font-size: 18px;\n  line-height: 1.33;\n  border-radius: 6px;\n}\nselect.input-group-lg > .form-control,\nselect.input-group-lg > .input-group-addon,\nselect.input-group-lg > .input-group-btn > .btn {\n  height: 46px;\n  line-height: 46px;\n}\ntextarea.input-group-lg > .form-control,\ntextarea.input-group-lg > .input-group-addon,\ntextarea.input-group-lg > .input-group-btn > .btn,\nselect[multiple].input-group-lg > .form-control,\nselect[multiple].input-group-lg > .input-group-addon,\nselect[multiple].input-group-lg > .input-group-btn > .btn {\n  height: auto;\n}\n.input-group-sm > .form-control,\n.input-group-sm > .input-group-addon,\n.input-group-sm > .input-group-btn > .btn {\n  height: 30px;\n  padding: 5px 10px;\n  font-size: 12px;\n  line-height: 1.5;\n  border-radius: 3px;\n}\nselect.input-group-sm > .form-control,\nselect.input-group-sm > .input-group-addon,\nselect.input-group-sm > .input-group-btn > .btn {\n  height: 30px;\n  line-height: 30px;\n}\ntextarea.input-group-sm > .form-control,\ntextarea.input-group-sm > .input-group-addon,\ntextarea.input-group-sm > .input-group-btn > .btn,\nselect[multiple].input-group-sm > .form-control,\nselect[multiple].input-group-sm > .input-group-addon,\nselect[multiple].input-group-sm > .input-group-btn > .btn {\n  height: auto;\n}\n.input-group-addon,\n.input-group-btn,\n.input-group .form-control {\n  display: table-cell;\n}\n.input-group-addon:not(:first-child):not(:last-child),\n.input-group-btn:not(:first-child):not(:last-child),\n.input-group .form-control:not(:first-child):not(:last-child) {\n  border-radius: 0;\n}\n.input-group-addon,\n.input-group-btn {\n  width: 1%;\n  white-space: nowrap;\n  vertical-align: middle;\n}\n.input-group-addon {\n  padding: 6px 12px;\n  font-size: 14px;\n  font-weight: normal;\n  line-height: 1;\n  color: #555;\n  text-align: center;\n  background-color: #eee;\n  border: 1px solid #ccc;\n  border-radius: 4px;\n}\n.input-group-addon.input-sm {\n  padding: 5px 10px;\n  font-size: 12px;\n  border-radius: 3px;\n}\n.input-group-addon.input-lg {\n  padding: 10px 16px;\n  font-size: 18px;\n  border-radius: 6px;\n}\n.input-group-addon input[type=\"radio\"],\n.input-group-addon input[type=\"checkbox\"] {\n  margin-top: 0;\n}\n.input-group .form-control:first-child,\n.input-group-addon:first-child,\n.input-group-btn:first-child > .btn,\n.input-group-btn:first-child > .btn-group > .btn,\n.input-group-btn:first-child > .dropdown-toggle,\n.input-group-btn:last-child > .btn:not(:last-child):not(.dropdown-toggle),\n.input-group-btn:last-child > .btn-group:not(:last-child) > .btn {\n  border-top-right-radius: 0;\n  border-bottom-right-radius: 0;\n}\n.input-group-addon:first-child {\n  border-right: 0;\n}\n.input-group .form-control:last-child,\n.input-group-addon:last-child,\n.input-group-btn:last-child > .btn,\n.input-group-btn:last-child > .btn-group > .btn,\n.input-group-btn:last-child > .dropdown-toggle,\n.input-group-btn:first-child > .btn:not(:first-child),\n.input-group-btn:first-child > .btn-group:not(:first-child) > .btn {\n  border-top-left-radius: 0;\n  border-bottom-left-radius: 0;\n}\n.input-group-addon:last-child {\n  border-left: 0;\n}\n.input-group-btn {\n  position: relative;\n  font-size: 0;\n  white-space: nowrap;\n}\n.input-group-btn > .btn {\n  position: relative;\n}\n.input-group-btn > .btn + .btn {\n  margin-left: -1px;\n}\n.input-group-btn > .btn:hover,\n.input-group-btn > .btn:focus,\n.input-group-btn > .btn:active {\n  z-index: 2;\n}\n.input-group-btn:first-child > .btn,\n.input-group-btn:first-child > .btn-group {\n  margin-right: -1px;\n}\n.input-group-btn:last-child > .btn,\n.input-group-btn:last-child > .btn-group {\n  margin-left: -1px;\n}\n.nav {\n  padding-left: 0;\n  margin-bottom: 0;\n  list-style: none;\n}\n.nav > li {\n  position: relative;\n  display: block;\n}\n.nav > li > a {\n  position: relative;\n  display: block;\n  padding: 10px 15px;\n}\n.nav > li > a:hover,\n.nav > li > a:focus {\n  text-decoration: none;\n  background-color: #eee;\n}\n.nav > li.disabled > a {\n  color: #999;\n}\n.nav > li.disabled > a:hover,\n.nav > li.disabled > a:focus {\n  color: #999;\n  text-decoration: none;\n  cursor: not-allowed;\n  background-color: transparent;\n}\n.nav .open > a,\n.nav .open > a:hover,\n.nav .open > a:focus {\n  background-color: #eee;\n  border-color: #428bca;\n}\n.nav .nav-divider {\n  height: 1px;\n  margin: 9px 0;\n  overflow: hidden;\n  background-color: #e5e5e5;\n}\n.nav > li > a > img {\n  max-width: none;\n}\n.nav-tabs {\n  border-bottom: 1px solid #ddd;\n}\n.nav-tabs > li {\n  float: left;\n  margin-bottom: -1px;\n}\n.nav-tabs > li > a {\n  margin-right: 2px;\n  line-height: 1.428571429;\n  border: 1px solid transparent;\n  border-radius: 4px 4px 0 0;\n}\n.nav-tabs > li > a:hover {\n  border-color: #eee #eee #ddd;\n}\n.nav-tabs > li.active > a,\n.nav-tabs > li.active > a:hover,\n.nav-tabs > li.active > a:focus {\n  color: #555;\n  cursor: default;\n  background-color: #fff;\n  border: 1px solid #ddd;\n  border-bottom-color: transparent;\n}\n.nav-tabs.nav-justified {\n  width: 100%;\n  border-bottom: 0;\n}\n.nav-tabs.nav-justified > li {\n  float: none;\n}\n.nav-tabs.nav-justified > li > a {\n  margin-bottom: 5px;\n  text-align: center;\n}\n.nav-tabs.nav-justified > .dropdown .dropdown-menu {\n  top: auto;\n  left: auto;\n}\n@media (min-width: 768px) {\n  .nav-tabs.nav-justified > li {\n    display: table-cell;\n    width: 1%;\n  }\n  .nav-tabs.nav-justified > li > a {\n    margin-bottom: 0;\n  }\n}\n.nav-tabs.nav-justified > li > a {\n  margin-right: 0;\n  border-radius: 4px;\n}\n.nav-tabs.nav-justified > .active > a,\n.nav-tabs.nav-justified > .active > a:hover,\n.nav-tabs.nav-justified > .active > a:focus {\n  border: 1px solid #ddd;\n}\n@media (min-width: 768px) {\n  .nav-tabs.nav-justified > li > a {\n    border-bottom: 1px solid #ddd;\n    border-radius: 4px 4px 0 0;\n  }\n  .nav-tabs.nav-justified > .active > a,\n  .nav-tabs.nav-justified > .active > a:hover,\n  .nav-tabs.nav-justified > .active > a:focus {\n    border-bottom-color: #fff;\n  }\n}\n.nav-pills > li {\n  float: left;\n}\n.nav-pills > li > a {\n  border-radius: 4px;\n}\n.nav-pills > li + li {\n  margin-left: 2px;\n}\n.nav-pills > li.active > a,\n.nav-pills > li.active > a:hover,\n.nav-pills > li.active > a:focus {\n  color: #fff;\n  background-color: #428bca;\n}\n.nav-stacked > li {\n  float: none;\n}\n.nav-stacked > li + li {\n  margin-top: 2px;\n  margin-left: 0;\n}\n.nav-justified {\n  width: 100%;\n}\n.nav-justified > li {\n  float: none;\n}\n.nav-justified > li > a {\n  margin-bottom: 5px;\n  text-align: center;\n}\n.nav-justified > .dropdown .dropdown-menu {\n  top: auto;\n  left: auto;\n}\n@media (min-width: 768px) {\n  .nav-justified > li {\n    display: table-cell;\n    width: 1%;\n  }\n  .nav-justified > li > a {\n    margin-bottom: 0;\n  }\n}\n.nav-tabs-justified {\n  border-bottom: 0;\n}\n.nav-tabs-justified > li > a {\n  margin-right: 0;\n  border-radius: 4px;\n}\n.nav-tabs-justified > .active > a,\n.nav-tabs-justified > .active > a:hover,\n.nav-tabs-justified > .active > a:focus {\n  border: 1px solid #ddd;\n}\n@media (min-width: 768px) {\n  .nav-tabs-justified > li > a {\n    border-bottom: 1px solid #ddd;\n    border-radius: 4px 4px 0 0;\n  }\n  .nav-tabs-justified > .active > a,\n  .nav-tabs-justified > .active > a:hover,\n  .nav-tabs-justified > .active > a:focus {\n    border-bottom-color: #fff;\n  }\n}\n.tab-content > .tab-pane {\n  display: none;\n}\n.tab-content > .active {\n  display: block;\n}\n.nav-tabs .dropdown-menu {\n  margin-top: -1px;\n  border-top-left-radius: 0;\n  border-top-right-radius: 0;\n}\n.navbar {\n  position: relative;\n  min-height: 50px;\n  margin-bottom: 20px;\n  border: 1px solid transparent;\n}\n@media (min-width: 768px) {\n  .navbar {\n    border-radius: 4px;\n  }\n}\n@media (min-width: 768px) {\n  .navbar-header {\n    float: left;\n  }\n}\n.navbar-collapse {\n  max-height: 340px;\n  padding-right: 15px;\n  padding-left: 15px;\n  overflow-x: visible;\n  -webkit-overflow-scrolling: touch;\n  border-top: 1px solid transparent;\n  box-shadow: inset 0 1px 0 rgba(255, 255, 255, .1);\n}\n.navbar-collapse.in {\n  overflow-y: auto;\n}\n@media (min-width: 768px) {\n  .navbar-collapse {\n    width: auto;\n    border-top: 0;\n    box-shadow: none;\n  }\n  .navbar-collapse.collapse {\n    display: block !important;\n    height: auto !important;\n    padding-bottom: 0;\n    overflow: visible !important;\n  }\n  .navbar-collapse.in {\n    overflow-y: visible;\n  }\n  .navbar-fixed-top .navbar-collapse,\n  .navbar-static-top .navbar-collapse,\n  .navbar-fixed-bottom .navbar-collapse {\n    padding-right: 0;\n    padding-left: 0;\n  }\n}\n.container > .navbar-header,\n.container-fluid > .navbar-header,\n.container > .navbar-collapse,\n.container-fluid > .navbar-collapse {\n  margin-right: -15px;\n  margin-left: -15px;\n}\n@media (min-width: 768px) {\n  .container > .navbar-header,\n  .container-fluid > .navbar-header,\n  .container > .navbar-collapse,\n  .container-fluid > .navbar-collapse {\n    margin-right: 0;\n    margin-left: 0;\n  }\n}\n.navbar-static-top {\n  z-index: 1000;\n  border-width: 0 0 1px;\n}\n@media (min-width: 768px) {\n  .navbar-static-top {\n    border-radius: 0;\n  }\n}\n.navbar-fixed-top,\n.navbar-fixed-bottom {\n  position: fixed;\n  right: 0;\n  left: 0;\n  z-index: 1030;\n}\n@media (min-width: 768px) {\n  .navbar-fixed-top,\n  .navbar-fixed-bottom {\n    border-radius: 0;\n  }\n}\n.navbar-fixed-top {\n  top: 0;\n  border-width: 0 0 1px;\n}\n.navbar-fixed-bottom {\n  bottom: 0;\n  margin-bottom: 0;\n  border-width: 1px 0 0;\n}\n.navbar-brand {\n  float: left;\n  height: 20px;\n  padding: 15px 15px;\n  font-size: 18px;\n  line-height: 20px;\n}\n.navbar-brand:hover,\n.navbar-brand:focus {\n  text-decoration: none;\n}\n@media (min-width: 768px) {\n  .navbar > .container .navbar-brand,\n  .navbar > .container-fluid .navbar-brand {\n    margin-left: -15px;\n  }\n}\n.navbar-toggle {\n  position: relative;\n  float: right;\n  padding: 9px 10px;\n  margin-top: 8px;\n  margin-right: 15px;\n  margin-bottom: 8px;\n  background-color: transparent;\n  background-image: none;\n  border: 1px solid transparent;\n  border-radius: 4px;\n}\n.navbar-toggle:focus {\n  outline: none;\n}\n.navbar-toggle .icon-bar {\n  display: block;\n  width: 22px;\n  height: 2px;\n  border-radius: 1px;\n}\n.navbar-toggle .icon-bar + .icon-bar {\n  margin-top: 4px;\n}\n@media (min-width: 768px) {\n  .navbar-toggle {\n    display: none;\n  }\n}\n.navbar-nav {\n  margin: 7.5px -15px;\n}\n.navbar-nav > li > a {\n  padding-top: 10px;\n  padding-bottom: 10px;\n  line-height: 20px;\n}\n@media (max-width: 767px) {\n  .navbar-nav .open .dropdown-menu {\n    position: static;\n    float: none;\n    width: auto;\n    margin-top: 0;\n    background-color: transparent;\n    border: 0;\n    box-shadow: none;\n  }\n  .navbar-nav .open .dropdown-menu > li > a,\n  .navbar-nav .open .dropdown-menu .dropdown-header {\n    padding: 5px 15px 5px 25px;\n  }\n  .navbar-nav .open .dropdown-menu > li > a {\n    line-height: 20px;\n  }\n  .navbar-nav .open .dropdown-menu > li > a:hover,\n  .navbar-nav .open .dropdown-menu > li > a:focus {\n    background-image: none;\n  }\n}\n@media (min-width: 768px) {\n  .navbar-nav {\n    float: left;\n    margin: 0;\n  }\n  .navbar-nav > li {\n    float: left;\n  }\n  .navbar-nav > li > a {\n    padding-top: 15px;\n    padding-bottom: 15px;\n  }\n  .navbar-nav.navbar-right:last-child {\n    margin-right: -15px;\n  }\n}\n@media (min-width: 768px) {\n  .navbar-left {\n    float: left !important;\n  }\n  .navbar-right {\n    float: right !important;\n  }\n}\n.navbar-form {\n  padding: 10px 15px;\n  margin-top: 8px;\n  margin-right: -15px;\n  margin-bottom: 8px;\n  margin-left: -15px;\n  border-top: 1px solid transparent;\n  border-bottom: 1px solid transparent;\n  -webkit-box-shadow: inset 0 1px 0 rgba(255, 255, 255, .1), 0 1px 0 rgba(255, 255, 255, .1);\n          box-shadow: inset 0 1px 0 rgba(255, 255, 255, .1), 0 1px 0 rgba(255, 255, 255, .1);\n}\n@media (min-width: 768px) {\n  .navbar-form .form-group {\n    display: inline-block;\n    margin-bottom: 0;\n    vertical-align: middle;\n  }\n  .navbar-form .form-control {\n    display: inline-block;\n    width: auto;\n    vertical-align: middle;\n  }\n  .navbar-form .control-label {\n    margin-bottom: 0;\n    vertical-align: middle;\n  }\n  .navbar-form .radio,\n  .navbar-form .checkbox {\n    display: inline-block;\n    padding-left: 0;\n    margin-top: 0;\n    margin-bottom: 0;\n    vertical-align: middle;\n  }\n  .navbar-form .radio input[type=\"radio\"],\n  .navbar-form .checkbox input[type=\"checkbox\"] {\n    float: none;\n    margin-left: 0;\n  }\n  .navbar-form .has-feedback .form-control-feedback {\n    top: 0;\n  }\n}\n@media (max-width: 767px) {\n  .navbar-form .form-group {\n    margin-bottom: 5px;\n  }\n}\n@media (min-width: 768px) {\n  .navbar-form {\n    width: auto;\n    padding-top: 0;\n    padding-bottom: 0;\n    margin-right: 0;\n    margin-left: 0;\n    border: 0;\n    -webkit-box-shadow: none;\n            box-shadow: none;\n  }\n  .navbar-form.navbar-right:last-child {\n    margin-right: -15px;\n  }\n}\n.navbar-nav > li > .dropdown-menu {\n  margin-top: 0;\n  border-top-left-radius: 0;\n  border-top-right-radius: 0;\n}\n.navbar-fixed-bottom .navbar-nav > li > .dropdown-menu {\n  border-bottom-right-radius: 0;\n  border-bottom-left-radius: 0;\n}\n.navbar-btn {\n  margin-top: 8px;\n  margin-bottom: 8px;\n}\n.navbar-btn.btn-sm {\n  margin-top: 10px;\n  margin-bottom: 10px;\n}\n.navbar-btn.btn-xs {\n  margin-top: 14px;\n  margin-bottom: 14px;\n}\n.navbar-text {\n  margin-top: 15px;\n  margin-bottom: 15px;\n}\n@media (min-width: 768px) {\n  .navbar-text {\n    float: left;\n    margin-right: 15px;\n    margin-left: 15px;\n  }\n  .navbar-text.navbar-right:last-child {\n    margin-right: 0;\n  }\n}\n.navbar-default {\n  background-color: #f8f8f8;\n  border-color: #e7e7e7;\n}\n.navbar-default .navbar-brand {\n  color: #777;\n}\n.navbar-default .navbar-brand:hover,\n.navbar-default .navbar-brand:focus {\n  color: #5e5e5e;\n  background-color: transparent;\n}\n.navbar-default .navbar-text {\n  color: #777;\n}\n.navbar-default .navbar-nav > li > a {\n  color: #777;\n}\n.navbar-default .navbar-nav > li > a:hover,\n.navbar-default .navbar-nav > li > a:focus {\n  color: #333;\n  background-color: transparent;\n}\n.navbar-default .navbar-nav > .active > a,\n.navbar-default .navbar-nav > .active > a:hover,\n.navbar-default .navbar-nav > .active > a:focus {\n  color: #555;\n  background-color: #e7e7e7;\n}\n.navbar-default .navbar-nav > .disabled > a,\n.navbar-default .navbar-nav > .disabled > a:hover,\n.navbar-default .navbar-nav > .disabled > a:focus {\n  color: #ccc;\n  background-color: transparent;\n}\n.navbar-default .navbar-toggle {\n  border-color: #ddd;\n}\n.navbar-default .navbar-toggle:hover,\n.navbar-default .navbar-toggle:focus {\n  background-color: #ddd;\n}\n.navbar-default .navbar-toggle .icon-bar {\n  background-color: #888;\n}\n.navbar-default .navbar-collapse,\n.navbar-default .navbar-form {\n  border-color: #e7e7e7;\n}\n.navbar-default .navbar-nav > .open > a,\n.navbar-default .navbar-nav > .open > a:hover,\n.navbar-default .navbar-nav > .open > a:focus {\n  color: #555;\n  background-color: #e7e7e7;\n}\n@media (max-width: 767px) {\n  .navbar-default .navbar-nav .open .dropdown-menu > li > a {\n    color: #777;\n  }\n  .navbar-default .navbar-nav .open .dropdown-menu > li > a:hover,\n  .navbar-default .navbar-nav .open .dropdown-menu > li > a:focus {\n    color: #333;\n    background-color: transparent;\n  }\n  .navbar-default .navbar-nav .open .dropdown-menu > .active > a,\n  .navbar-default .navbar-nav .open .dropdown-menu > .active > a:hover,\n  .navbar-default .navbar-nav .open .dropdown-menu > .active > a:focus {\n    color: #555;\n    background-color: #e7e7e7;\n  }\n  .navbar-default .navbar-nav .open .dropdown-menu > .disabled > a,\n  .navbar-default .navbar-nav .open .dropdown-menu > .disabled > a:hover,\n  .navbar-default .navbar-nav .open .dropdown-menu > .disabled > a:focus {\n    color: #ccc;\n    background-color: transparent;\n  }\n}\n.navbar-default .navbar-link {\n  color: #777;\n}\n.navbar-default .navbar-link:hover {\n  color: #333;\n}\n.navbar-inverse {\n  background-color: #222;\n  border-color: #080808;\n}\n.navbar-inverse .navbar-brand {\n  color: #999;\n}\n.navbar-inverse .navbar-brand:hover,\n.navbar-inverse .navbar-brand:focus {\n  color: #fff;\n  background-color: transparent;\n}\n.navbar-inverse .navbar-text {\n  color: #999;\n}\n.navbar-inverse .navbar-nav > li > a {\n  color: #999;\n}\n.navbar-inverse .navbar-nav > li > a:hover,\n.navbar-inverse .navbar-nav > li > a:focus {\n  color: #fff;\n  background-color: transparent;\n}\n.navbar-inverse .navbar-nav > .active > a,\n.navbar-inverse .navbar-nav > .active > a:hover,\n.navbar-inverse .navbar-nav > .active > a:focus {\n  color: #fff;\n  background-color: #080808;\n}\n.navbar-inverse .navbar-nav > .disabled > a,\n.navbar-inverse .navbar-nav > .disabled > a:hover,\n.navbar-inverse .navbar-nav > .disabled > a:focus {\n  color: #444;\n  background-color: transparent;\n}\n.navbar-inverse .navbar-toggle {\n  border-color: #333;\n}\n.navbar-inverse .navbar-toggle:hover,\n.navbar-inverse .navbar-toggle:focus {\n  background-color: #333;\n}\n.navbar-inverse .navbar-toggle .icon-bar {\n  background-color: #fff;\n}\n.navbar-inverse .navbar-collapse,\n.navbar-inverse .navbar-form {\n  border-color: #101010;\n}\n.navbar-inverse .navbar-nav > .open > a,\n.navbar-inverse .navbar-nav > .open > a:hover,\n.navbar-inverse .navbar-nav > .open > a:focus {\n  color: #fff;\n  background-color: #080808;\n}\n@media (max-width: 767px) {\n  .navbar-inverse .navbar-nav .open .dropdown-menu > .dropdown-header {\n    border-color: #080808;\n  }\n  .navbar-inverse .navbar-nav .open .dropdown-menu .divider {\n    background-color: #080808;\n  }\n  .navbar-inverse .navbar-nav .open .dropdown-menu > li > a {\n    color: #999;\n  }\n  .navbar-inverse .navbar-nav .open .dropdown-menu > li > a:hover,\n  .navbar-inverse .navbar-nav .open .dropdown-menu > li > a:focus {\n    color: #fff;\n    background-color: transparent;\n  }\n  .navbar-inverse .navbar-nav .open .dropdown-menu > .active > a,\n  .navbar-inverse .navbar-nav .open .dropdown-menu > .active > a:hover,\n  .navbar-inverse .navbar-nav .open .dropdown-menu > .active > a:focus {\n    color: #fff;\n    background-color: #080808;\n  }\n  .navbar-inverse .navbar-nav .open .dropdown-menu > .disabled > a,\n  .navbar-inverse .navbar-nav .open .dropdown-menu > .disabled > a:hover,\n  .navbar-inverse .navbar-nav .open .dropdown-menu > .disabled > a:focus {\n    color: #444;\n    background-color: transparent;\n  }\n}\n.navbar-inverse .navbar-link {\n  color: #999;\n}\n.navbar-inverse .navbar-link:hover {\n  color: #fff;\n}\n.breadcrumb {\n  padding: 8px 15px;\n  margin-bottom: 20px;\n  list-style: none;\n  background-color: #f5f5f5;\n  border-radius: 4px;\n}\n.breadcrumb > li {\n  display: inline-block;\n}\n.breadcrumb > li + li:before {\n  padding: 0 5px;\n  color: #ccc;\n  content: \"/\\00a0\";\n}\n.breadcrumb > .active {\n  color: #999;\n}\n.pagination {\n  display: inline-block;\n  padding-left: 0;\n  margin: 20px 0;\n  border-radius: 4px;\n}\n.pagination > li {\n  display: inline;\n}\n.pagination > li > a,\n.pagination > li > span {\n  position: relative;\n  float: left;\n  padding: 6px 12px;\n  margin-left: -1px;\n  line-height: 1.428571429;\n  color: #428bca;\n  text-decoration: none;\n  background-color: #fff;\n  border: 1px solid #ddd;\n}\n.pagination > li:first-child > a,\n.pagination > li:first-child > span {\n  margin-left: 0;\n  border-top-left-radius: 4px;\n  border-bottom-left-radius: 4px;\n}\n.pagination > li:last-child > a,\n.pagination > li:last-child > span {\n  border-top-right-radius: 4px;\n  border-bottom-right-radius: 4px;\n}\n.pagination > li > a:hover,\n.pagination > li > span:hover,\n.pagination > li > a:focus,\n.pagination > li > span:focus {\n  color: #2a6496;\n  background-color: #eee;\n  border-color: #ddd;\n}\n.pagination > .active > a,\n.pagination > .active > span,\n.pagination > .active > a:hover,\n.pagination > .active > span:hover,\n.pagination > .active > a:focus,\n.pagination > .active > span:focus {\n  z-index: 2;\n  color: #fff;\n  cursor: default;\n  background-color: #428bca;\n  border-color: #428bca;\n}\n.pagination > .disabled > span,\n.pagination > .disabled > span:hover,\n.pagination > .disabled > span:focus,\n.pagination > .disabled > a,\n.pagination > .disabled > a:hover,\n.pagination > .disabled > a:focus {\n  color: #999;\n  cursor: not-allowed;\n  background-color: #fff;\n  border-color: #ddd;\n}\n.pagination-lg > li > a,\n.pagination-lg > li > span {\n  padding: 10px 16px;\n  font-size: 18px;\n}\n.pagination-lg > li:first-child > a,\n.pagination-lg > li:first-child > span {\n  border-top-left-radius: 6px;\n  border-bottom-left-radius: 6px;\n}\n.pagination-lg > li:last-child > a,\n.pagination-lg > li:last-child > span {\n  border-top-right-radius: 6px;\n  border-bottom-right-radius: 6px;\n}\n.pagination-sm > li > a,\n.pagination-sm > li > span {\n  padding: 5px 10px;\n  font-size: 12px;\n}\n.pagination-sm > li:first-child > a,\n.pagination-sm > li:first-child > span {\n  border-top-left-radius: 3px;\n  border-bottom-left-radius: 3px;\n}\n.pagination-sm > li:last-child > a,\n.pagination-sm > li:last-child > span {\n  border-top-right-radius: 3px;\n  border-bottom-right-radius: 3px;\n}\n.pager {\n  padding-left: 0;\n  margin: 20px 0;\n  text-align: center;\n  list-style: none;\n}\n.pager li {\n  display: inline;\n}\n.pager li > a,\n.pager li > span {\n  display: inline-block;\n  padding: 5px 14px;\n  background-color: #fff;\n  border: 1px solid #ddd;\n  border-radius: 15px;\n}\n.pager li > a:hover,\n.pager li > a:focus {\n  text-decoration: none;\n  background-color: #eee;\n}\n.pager .next > a,\n.pager .next > span {\n  float: right;\n}\n.pager .previous > a,\n.pager .previous > span {\n  float: left;\n}\n.pager .disabled > a,\n.pager .disabled > a:hover,\n.pager .disabled > a:focus,\n.pager .disabled > span {\n  color: #999;\n  cursor: not-allowed;\n  background-color: #fff;\n}\n.label {\n  display: inline;\n  padding: .2em .6em .3em;\n  font-size: 75%;\n  font-weight: bold;\n  line-height: 1;\n  color: #fff;\n  text-align: center;\n  white-space: nowrap;\n  vertical-align: baseline;\n  border-radius: .25em;\n}\n.label[href]:hover,\n.label[href]:focus {\n  color: #fff;\n  text-decoration: none;\n  cursor: pointer;\n}\n.label:empty {\n  display: none;\n}\n.btn .label {\n  position: relative;\n  top: -1px;\n}\n.label-default {\n  background-color: #999;\n}\n.label-default[href]:hover,\n.label-default[href]:focus {\n  background-color: #808080;\n}\n.label-primary {\n  background-color: #428bca;\n}\n.label-primary[href]:hover,\n.label-primary[href]:focus {\n  background-color: #3071a9;\n}\n.label-success {\n  background-color: #5cb85c;\n}\n.label-success[href]:hover,\n.label-success[href]:focus {\n  background-color: #449d44;\n}\n.label-info {\n  background-color: #5bc0de;\n}\n.label-info[href]:hover,\n.label-info[href]:focus {\n  background-color: #31b0d5;\n}\n.label-warning {\n  background-color: #f0ad4e;\n}\n.label-warning[href]:hover,\n.label-warning[href]:focus {\n  background-color: #ec971f;\n}\n.label-danger {\n  background-color: #d9534f;\n}\n.label-danger[href]:hover,\n.label-danger[href]:focus {\n  background-color: #c9302c;\n}\n.badge {\n  display: inline-block;\n  min-width: 10px;\n  padding: 3px 7px;\n  font-size: 12px;\n  font-weight: bold;\n  line-height: 1;\n  color: #fff;\n  text-align: center;\n  white-space: nowrap;\n  vertical-align: baseline;\n  background-color: #999;\n  border-radius: 10px;\n}\n.badge:empty {\n  display: none;\n}\n.btn .badge {\n  position: relative;\n  top: -1px;\n}\n.btn-xs .badge {\n  top: 0;\n  padding: 1px 5px;\n}\na.badge:hover,\na.badge:focus {\n  color: #fff;\n  text-decoration: none;\n  cursor: pointer;\n}\na.list-group-item.active > .badge,\n.nav-pills > .active > a > .badge {\n  color: #428bca;\n  background-color: #fff;\n}\n.nav-pills > li > a > .badge {\n  margin-left: 3px;\n}\n.jumbotron {\n  padding: 30px;\n  margin-bottom: 30px;\n  color: inherit;\n  background-color: #eee;\n}\n.jumbotron h1,\n.jumbotron .h1 {\n  color: inherit;\n}\n.jumbotron p {\n  margin-bottom: 15px;\n  font-size: 21px;\n  font-weight: 200;\n}\n.container .jumbotron {\n  border-radius: 6px;\n}\n.jumbotron .container {\n  max-width: 100%;\n}\n@media screen and (min-width: 768px) {\n  .jumbotron {\n    padding-top: 48px;\n    padding-bottom: 48px;\n  }\n  .container .jumbotron {\n    padding-right: 60px;\n    padding-left: 60px;\n  }\n  .jumbotron h1,\n  .jumbotron .h1 {\n    font-size: 63px;\n  }\n}\n.thumbnail {\n  display: block;\n  padding: 4px;\n  margin-bottom: 20px;\n  line-height: 1.428571429;\n  background-color: #fff;\n  border: 1px solid #ddd;\n  border-radius: 4px;\n  -webkit-transition: all .2s ease-in-out;\n          transition: all .2s ease-in-out;\n}\n.thumbnail > img,\n.thumbnail a > img {\n  display: block;\n  max-width: 100%;\n  height: auto;\n  margin-right: auto;\n  margin-left: auto;\n}\na.thumbnail:hover,\na.thumbnail:focus,\na.thumbnail.active {\n  border-color: #428bca;\n}\n.thumbnail .caption {\n  padding: 9px;\n  color: #333;\n}\n.alert {\n  padding: 15px;\n  margin-bottom: 20px;\n  border: 1px solid transparent;\n  border-radius: 4px;\n}\n.alert h4 {\n  margin-top: 0;\n  color: inherit;\n}\n.alert .alert-link {\n  font-weight: bold;\n}\n.alert > p,\n.alert > ul {\n  margin-bottom: 0;\n}\n.alert > p + p {\n  margin-top: 5px;\n}\n.alert-dismissable {\n  padding-right: 35px;\n}\n.alert-dismissable .close {\n  position: relative;\n  top: -2px;\n  right: -21px;\n  color: inherit;\n}\n.alert-success {\n  color: #3c763d;\n  background-color: #dff0d8;\n  border-color: #d6e9c6;\n}\n.alert-success hr {\n  border-top-color: #c9e2b3;\n}\n.alert-success .alert-link {\n  color: #2b542c;\n}\n.alert-info {\n  color: #31708f;\n  background-color: #edf6fa;\n  border-color: #bce8f1;\n}\n.alert-info hr {\n  border-top-color: #a6e1ec;\n}\n.alert-info .alert-link {\n  color: #245269;\n  font-weight: bold;\n}\n.alert-info a {\n  font-weight: bold;\n}\n.alert-warning {\n  color: #8a6d3b;\n  background-color: #fcf8e3;\n  border-color: #faebcc;\n}\n.alert-warning hr {\n  border-top-color: #f7e1b5;\n}\n.alert-warning .alert-link {\n  color: #66512c;\n}\n.alert-danger {\n  color: #a94442;\n  background-color: #f2dede;\n  border-color: #ebccd1;\n}\n.alert-danger hr {\n  border-top-color: #e4b9c0;\n}\n.alert-danger .alert-link {\n  color: #843534;\n}\n@-webkit-keyframes progress-bar-stripes {\n  from {\n    background-position: 40px 0;\n  }\n  to {\n    background-position: 0 0;\n  }\n}\n@keyframes progress-bar-stripes {\n  from {\n    background-position: 40px 0;\n  }\n  to {\n    background-position: 0 0;\n  }\n}\n.progress {\n  height: 20px;\n  margin-bottom: 20px;\n  overflow: hidden;\n  background-color: #f5f5f5;\n  border-radius: 4px;\n  -webkit-box-shadow: inset 0 1px 2px rgba(0, 0, 0, .1);\n          box-shadow: inset 0 1px 2px rgba(0, 0, 0, .1);\n}\n.progress-bar {\n  float: left;\n  width: 0;\n  height: 100%;\n  font-size: 12px;\n  line-height: 20px;\n  color: #fff;\n  text-align: center;\n  background-color: #428bca;\n  -webkit-box-shadow: inset 0 -1px 0 rgba(0, 0, 0, .15);\n          box-shadow: inset 0 -1px 0 rgba(0, 0, 0, .15);\n  -webkit-transition: width .6s ease;\n          transition: width .6s ease;\n}\n.progress-striped .progress-bar {\n  background-image: -webkit-linear-gradient(45deg, rgba(255, 255, 255, .15) 25%, transparent 25%, transparent 50%, rgba(255, 255, 255, .15) 50%, rgba(255, 255, 255, .15) 75%, transparent 75%, transparent);\n  background-image:         linear-gradient(45deg, rgba(255, 255, 255, .15) 25%, transparent 25%, transparent 50%, rgba(255, 255, 255, .15) 50%, rgba(255, 255, 255, .15) 75%, transparent 75%, transparent);\n  background-size: 40px 40px;\n}\n.progress.active .progress-bar {\n  -webkit-animation: progress-bar-stripes 2s linear infinite;\n          animation: progress-bar-stripes 2s linear infinite;\n}\n.progress-bar-success {\n  background-color: #5cb85c;\n}\n.progress-striped .progress-bar-success {\n  background-image: -webkit-linear-gradient(45deg, rgba(255, 255, 255, .15) 25%, transparent 25%, transparent 50%, rgba(255, 255, 255, .15) 50%, rgba(255, 255, 255, .15) 75%, transparent 75%, transparent);\n  background-image:         linear-gradient(45deg, rgba(255, 255, 255, .15) 25%, transparent 25%, transparent 50%, rgba(255, 255, 255, .15) 50%, rgba(255, 255, 255, .15) 75%, transparent 75%, transparent);\n}\n.progress-bar-info {\n  background-color: #5bc0de;\n}\n.progress-striped .progress-bar-info {\n  background-image: -webkit-linear-gradient(45deg, rgba(255, 255, 255, .15) 25%, transparent 25%, transparent 50%, rgba(255, 255, 255, .15) 50%, rgba(255, 255, 255, .15) 75%, transparent 75%, transparent);\n  background-image:         linear-gradient(45deg, rgba(255, 255, 255, .15) 25%, transparent 25%, transparent 50%, rgba(255, 255, 255, .15) 50%, rgba(255, 255, 255, .15) 75%, transparent 75%, transparent);\n}\n.progress-bar-warning {\n  background-color: #f0ad4e;\n}\n.progress-striped .progress-bar-warning {\n  background-image: -webkit-linear-gradient(45deg, rgba(255, 255, 255, .15) 25%, transparent 25%, transparent 50%, rgba(255, 255, 255, .15) 50%, rgba(255, 255, 255, .15) 75%, transparent 75%, transparent);\n  background-image:         linear-gradient(45deg, rgba(255, 255, 255, .15) 25%, transparent 25%, transparent 50%, rgba(255, 255, 255, .15) 50%, rgba(255, 255, 255, .15) 75%, transparent 75%, transparent);\n}\n.progress-bar-danger {\n  background-color: #d9534f;\n}\n.progress-striped .progress-bar-danger {\n  background-image: -webkit-linear-gradient(45deg, rgba(255, 255, 255, .15) 25%, transparent 25%, transparent 50%, rgba(255, 255, 255, .15) 50%, rgba(255, 255, 255, .15) 75%, transparent 75%, transparent);\n  background-image:         linear-gradient(45deg, rgba(255, 255, 255, .15) 25%, transparent 25%, transparent 50%, rgba(255, 255, 255, .15) 50%, rgba(255, 255, 255, .15) 75%, transparent 75%, transparent);\n}\n.media,\n.media-body {\n  overflow: hidden;\n  zoom: 1;\n}\n.media,\n.media .media {\n  margin-top: 15px;\n}\n.media:first-child {\n  margin-top: 0;\n}\n.media-object {\n  display: block;\n}\n.media-heading {\n  margin: 0 0 5px;\n}\n.media > .pull-left {\n  margin-right: 10px;\n}\n.media > .pull-right {\n  margin-left: 10px;\n}\n.media-list {\n  padding-left: 0;\n  list-style: none;\n}\n.list-group {\n  padding-left: 0;\n  margin-bottom: 20px;\n}\n.list-group-item {\n  position: relative;\n  display: block;\n  padding: 10px 15px;\n  margin-bottom: -1px;\n  background-color: #fff;\n  border: 1px solid #ddd;\n}\n.list-group-item:first-child {\n  border-top-left-radius: 4px;\n  border-top-right-radius: 4px;\n}\n.list-group-item:last-child {\n  margin-bottom: 0;\n  border-bottom-right-radius: 4px;\n  border-bottom-left-radius: 4px;\n}\n.list-group-item > .badge {\n  float: right;\n}\n.list-group-item > .badge + .badge {\n  margin-right: 5px;\n}\na.list-group-item {\n  color: #555;\n}\na.list-group-item .list-group-item-heading {\n  color: #333;\n}\na.list-group-item:hover,\na.list-group-item:focus {\n  text-decoration: none;\n  background-color: #f5f5f5;\n}\na.list-group-item.active,\na.list-group-item.active:hover,\na.list-group-item.active:focus {\n  z-index: 2;\n  color: #fff;\n  background-color: #428bca;\n  border-color: #428bca;\n}\na.list-group-item.active .list-group-item-heading,\na.list-group-item.active:hover .list-group-item-heading,\na.list-group-item.active:focus .list-group-item-heading {\n  color: inherit;\n}\na.list-group-item.active .list-group-item-text,\na.list-group-item.active:hover .list-group-item-text,\na.list-group-item.active:focus .list-group-item-text {\n  color: #e1edf7;\n}\n.list-group-item-success {\n  color: #3c763d;\n  background-color: #dff0d8;\n}\na.list-group-item-success {\n  color: #3c763d;\n}\na.list-group-item-success .list-group-item-heading {\n  color: inherit;\n}\na.list-group-item-success:hover,\na.list-group-item-success:focus {\n  color: #3c763d;\n  background-color: #d0e9c6;\n}\na.list-group-item-success.active,\na.list-group-item-success.active:hover,\na.list-group-item-success.active:focus {\n  color: #fff;\n  background-color: #3c763d;\n  border-color: #3c763d;\n}\n.list-group-item-info {\n  color: #31708f;\n  background-color: #d9edf7;\n}\na.list-group-item-info {\n  color: #31708f;\n}\na.list-group-item-info .list-group-item-heading {\n  color: inherit;\n}\na.list-group-item-info:hover,\na.list-group-item-info:focus {\n  color: #31708f;\n  background-color: #c4e3f3;\n}\na.list-group-item-info.active,\na.list-group-item-info.active:hover,\na.list-group-item-info.active:focus {\n  color: #fff;\n  background-color: #31708f;\n  border-color: #31708f;\n}\n.list-group-item-warning {\n  color: #8a6d3b;\n  background-color: #fcf8e3;\n}\na.list-group-item-warning {\n  color: #8a6d3b;\n}\na.list-group-item-warning .list-group-item-heading {\n  color: inherit;\n}\na.list-group-item-warning:hover,\na.list-group-item-warning:focus {\n  color: #8a6d3b;\n  background-color: #faf2cc;\n}\na.list-group-item-warning.active,\na.list-group-item-warning.active:hover,\na.list-group-item-warning.active:focus {\n  color: #fff;\n  background-color: #8a6d3b;\n  border-color: #8a6d3b;\n}\n.list-group-item-danger {\n  color: #a94442;\n  background-color: #f2dede;\n}\na.list-group-item-danger {\n  color: #a94442;\n}\na.list-group-item-danger .list-group-item-heading {\n  color: inherit;\n}\na.list-group-item-danger:hover,\na.list-group-item-danger:focus {\n  color: #a94442;\n  background-color: #ebcccc;\n}\na.list-group-item-danger.active,\na.list-group-item-danger.active:hover,\na.list-group-item-danger.active:focus {\n  color: #fff;\n  background-color: #a94442;\n  border-color: #a94442;\n}\n.list-group-item-heading {\n  margin-top: 0;\n  margin-bottom: 5px;\n}\n.list-group-item-text {\n  margin-bottom: 0;\n  line-height: 1.3;\n}\n.panel {\n  margin-bottom: 20px;\n  background-color: #fff;\n  border: 1px solid transparent;\n  border-radius: 4px;\n  -webkit-box-shadow: 0 1px 1px rgba(0, 0, 0, .05);\n          box-shadow: 0 1px 1px rgba(0, 0, 0, .05);\n}\n.panel-body {\n  padding: 15px;\n}\n.panel > .list-group {\n  margin-bottom: 0;\n}\n.panel > .list-group .list-group-item {\n  border-width: 1px 0;\n  border-radius: 0;\n}\n.panel > .list-group .list-group-item:first-child {\n  border-top: 0;\n}\n.panel > .list-group .list-group-item:last-child {\n  border-bottom: 0;\n}\n.panel > .list-group:first-child .list-group-item:first-child {\n  border-top-left-radius: 3px;\n  border-top-right-radius: 3px;\n}\n.panel > .list-group:last-child .list-group-item:last-child {\n  border-bottom-right-radius: 3px;\n  border-bottom-left-radius: 3px;\n}\n.panel-heading + .list-group .list-group-item:first-child {\n  border-top-width: 0;\n}\n.panel > .table,\n.panel > .table-responsive > .table {\n  margin-bottom: 0;\n}\n.panel > .table:first-child > thead:first-child > tr:first-child td:first-child,\n.panel > .table-responsive:first-child > .table:first-child > thead:first-child > tr:first-child td:first-child,\n.panel > .table:first-child > tbody:first-child > tr:first-child td:first-child,\n.panel > .table-responsive:first-child > .table:first-child > tbody:first-child > tr:first-child td:first-child,\n.panel > .table:first-child > thead:first-child > tr:first-child th:first-child,\n.panel > .table-responsive:first-child > .table:first-child > thead:first-child > tr:first-child th:first-child,\n.panel > .table:first-child > tbody:first-child > tr:first-child th:first-child,\n.panel > .table-responsive:first-child > .table:first-child > tbody:first-child > tr:first-child th:first-child {\n  border-top-left-radius: 3px;\n}\n.panel > .table:first-child > thead:first-child > tr:first-child td:last-child,\n.panel > .table-responsive:first-child > .table:first-child > thead:first-child > tr:first-child td:last-child,\n.panel > .table:first-child > tbody:first-child > tr:first-child td:last-child,\n.panel > .table-responsive:first-child > .table:first-child > tbody:first-child > tr:first-child td:last-child,\n.panel > .table:first-child > thead:first-child > tr:first-child th:last-child,\n.panel > .table-responsive:first-child > .table:first-child > thead:first-child > tr:first-child th:last-child,\n.panel > .table:first-child > tbody:first-child > tr:first-child th:last-child,\n.panel > .table-responsive:first-child > .table:first-child > tbody:first-child > tr:first-child th:last-child {\n  border-top-right-radius: 3px;\n}\n.panel > .table:last-child > tbody:last-child > tr:last-child td:first-child,\n.panel > .table-responsive:last-child > .table:last-child > tbody:last-child > tr:last-child td:first-child,\n.panel > .table:last-child > tfoot:last-child > tr:last-child td:first-child,\n.panel > .table-responsive:last-child > .table:last-child > tfoot:last-child > tr:last-child td:first-child,\n.panel > .table:last-child > tbody:last-child > tr:last-child th:first-child,\n.panel > .table-responsive:last-child > .table:last-child > tbody:last-child > tr:last-child th:first-child,\n.panel > .table:last-child > tfoot:last-child > tr:last-child th:first-child,\n.panel > .table-responsive:last-child > .table:last-child > tfoot:last-child > tr:last-child th:first-child {\n  border-bottom-left-radius: 3px;\n}\n.panel > .table:last-child > tbody:last-child > tr:last-child td:last-child,\n.panel > .table-responsive:last-child > .table:last-child > tbody:last-child > tr:last-child td:last-child,\n.panel > .table:last-child > tfoot:last-child > tr:last-child td:last-child,\n.panel > .table-responsive:last-child > .table:last-child > tfoot:last-child > tr:last-child td:last-child,\n.panel > .table:last-child > tbody:last-child > tr:last-child th:last-child,\n.panel > .table-responsive:last-child > .table:last-child > tbody:last-child > tr:last-child th:last-child,\n.panel > .table:last-child > tfoot:last-child > tr:last-child th:last-child,\n.panel > .table-responsive:last-child > .table:last-child > tfoot:last-child > tr:last-child th:last-child {\n  border-bottom-right-radius: 3px;\n}\n.panel > .panel-body + .table,\n.panel > .panel-body + .table-responsive {\n  border-top: 1px solid #ddd;\n}\n.panel > .table > tbody:first-child > tr:first-child th,\n.panel > .table > tbody:first-child > tr:first-child td {\n  border-top: 0;\n}\n.panel > .table-bordered,\n.panel > .table-responsive > .table-bordered {\n  border: 0;\n}\n.panel > .table-bordered > thead > tr > th:first-child,\n.panel > .table-responsive > .table-bordered > thead > tr > th:first-child,\n.panel > .table-bordered > tbody > tr > th:first-child,\n.panel > .table-responsive > .table-bordered > tbody > tr > th:first-child,\n.panel > .table-bordered > tfoot > tr > th:first-child,\n.panel > .table-responsive > .table-bordered > tfoot > tr > th:first-child,\n.panel > .table-bordered > thead > tr > td:first-child,\n.panel > .table-responsive > .table-bordered > thead > tr > td:first-child,\n.panel > .table-bordered > tbody > tr > td:first-child,\n.panel > .table-responsive > .table-bordered > tbody > tr > td:first-child,\n.panel > .table-bordered > tfoot > tr > td:first-child,\n.panel > .table-responsive > .table-bordered > tfoot > tr > td:first-child {\n  border-left: 0;\n}\n.panel > .table-bordered > thead > tr > th:last-child,\n.panel > .table-responsive > .table-bordered > thead > tr > th:last-child,\n.panel > .table-bordered > tbody > tr > th:last-child,\n.panel > .table-responsive > .table-bordered > tbody > tr > th:last-child,\n.panel > .table-bordered > tfoot > tr > th:last-child,\n.panel > .table-responsive > .table-bordered > tfoot > tr > th:last-child,\n.panel > .table-bordered > thead > tr > td:last-child,\n.panel > .table-responsive > .table-bordered > thead > tr > td:last-child,\n.panel > .table-bordered > tbody > tr > td:last-child,\n.panel > .table-responsive > .table-bordered > tbody > tr > td:last-child,\n.panel > .table-bordered > tfoot > tr > td:last-child,\n.panel > .table-responsive > .table-bordered > tfoot > tr > td:last-child {\n  border-right: 0;\n}\n.panel > .table-bordered > thead > tr:first-child > th,\n.panel > .table-responsive > .table-bordered > thead > tr:first-child > th,\n.panel > .table-bordered > tbody > tr:first-child > th,\n.panel > .table-responsive > .table-bordered > tbody > tr:first-child > th,\n.panel > .table-bordered > tfoot > tr:first-child > th,\n.panel > .table-responsive > .table-bordered > tfoot > tr:first-child > th,\n.panel > .table-bordered > thead > tr:first-child > td,\n.panel > .table-responsive > .table-bordered > thead > tr:first-child > td,\n.panel > .table-bordered > tbody > tr:first-child > td,\n.panel > .table-responsive > .table-bordered > tbody > tr:first-child > td,\n.panel > .table-bordered > tfoot > tr:first-child > td,\n.panel > .table-responsive > .table-bordered > tfoot > tr:first-child > td {\n  border-top: 0;\n}\n.panel > .table-bordered > thead > tr:last-child > th,\n.panel > .table-responsive > .table-bordered > thead > tr:last-child > th,\n.panel > .table-bordered > tbody > tr:last-child > th,\n.panel > .table-responsive > .table-bordered > tbody > tr:last-child > th,\n.panel > .table-bordered > tfoot > tr:last-child > th,\n.panel > .table-responsive > .table-bordered > tfoot > tr:last-child > th,\n.panel > .table-bordered > thead > tr:last-child > td,\n.panel > .table-responsive > .table-bordered > thead > tr:last-child > td,\n.panel > .table-bordered > tbody > tr:last-child > td,\n.panel > .table-responsive > .table-bordered > tbody > tr:last-child > td,\n.panel > .table-bordered > tfoot > tr:last-child > td,\n.panel > .table-responsive > .table-bordered > tfoot > tr:last-child > td {\n  border-bottom: 0;\n}\n.panel > .table-responsive {\n  margin-bottom: 0;\n  border: 0;\n}\n.panel-heading {\n  padding: 10px 15px;\n  border-bottom: 1px solid transparent;\n  border-top-left-radius: 3px;\n  border-top-right-radius: 3px;\n}\n.panel-heading > .dropdown .dropdown-toggle {\n  color: inherit;\n}\n.panel-title {\n  margin-top: 0;\n  margin-bottom: 0;\n  font-size: 16px;\n  color: inherit;\n}\n.panel-title > a {\n  color: inherit;\n}\n.panel-footer {\n  padding: 10px 15px;\n  background-color: #f5f5f5;\n  border-top: 1px solid #ddd;\n  border-bottom-right-radius: 3px;\n  border-bottom-left-radius: 3px;\n}\n.panel-group {\n  margin-bottom: 20px;\n}\n.panel-group .panel {\n  margin-bottom: 0;\n  overflow: hidden;\n  border-radius: 4px;\n}\n.panel-group .panel + .panel {\n  margin-top: 5px;\n}\n.panel-group .panel-heading {\n  border-bottom: 0;\n}\n.panel-group .panel-heading + .panel-collapse .panel-body {\n  border-top: 1px solid #ddd;\n}\n.panel-group .panel-footer {\n  border-top: 0;\n}\n.panel-group .panel-footer + .panel-collapse .panel-body {\n  border-bottom: 1px solid #ddd;\n}\n.panel-default {\n  border-color: #ddd;\n}\n.panel-default > .panel-heading {\n  color: #333;\n  background-color: #f5f5f5;\n  border-color: #ddd;\n}\n.panel-default > .panel-heading + .panel-collapse .panel-body {\n  border-top-color: #ddd;\n}\n.panel-default > .panel-footer + .panel-collapse .panel-body {\n  border-bottom-color: #ddd;\n}\n.panel-primary {\n  border-color: #428bca;\n}\n.panel-primary > .panel-heading {\n  color: #fff;\n  background-color: #428bca;\n  border-color: #428bca;\n}\n.panel-primary > .panel-heading + .panel-collapse .panel-body {\n  border-top-color: #428bca;\n}\n.panel-primary > .panel-footer + .panel-collapse .panel-body {\n  border-bottom-color: #428bca;\n}\n.panel-success {\n  border-color: #d6e9c6;\n}\n.panel-success > .panel-heading {\n  color: #3c763d;\n  background-color: #dff0d8;\n  border-color: #d6e9c6;\n}\n.panel-success > .panel-heading + .panel-collapse .panel-body {\n  border-top-color: #d6e9c6;\n}\n.panel-success > .panel-footer + .panel-collapse .panel-body {\n  border-bottom-color: #d6e9c6;\n}\n.panel-info {\n  border-color: #bce8f1;\n}\n.panel-info > .panel-heading {\n  color: #31708f;\n  background-color: #d9edf7;\n  border-color: #bce8f1;\n}\n.panel-info > .panel-heading + .panel-collapse .panel-body {\n  border-top-color: #bce8f1;\n}\n.panel-info > .panel-footer + .panel-collapse .panel-body {\n  border-bottom-color: #bce8f1;\n}\n.panel-warning {\n  border-color: #faebcc;\n}\n.panel-warning > .panel-heading {\n  color: #8a6d3b;\n  background-color: #fcf8e3;\n  border-color: #faebcc;\n}\n.panel-warning > .panel-heading + .panel-collapse .panel-body {\n  border-top-color: #faebcc;\n}\n.panel-warning > .panel-footer + .panel-collapse .panel-body {\n  border-bottom-color: #faebcc;\n}\n.panel-danger {\n  border-color: #ebccd1;\n}\n.panel-danger > .panel-heading {\n  color: #a94442;\n  background-color: #f2dede;\n  border-color: #ebccd1;\n}\n.panel-danger > .panel-heading + .panel-collapse .panel-body {\n  border-top-color: #ebccd1;\n}\n.panel-danger > .panel-footer + .panel-collapse .panel-body {\n  border-bottom-color: #ebccd1;\n}\n.well {\n  min-height: 20px;\n  padding: 19px;\n  margin-bottom: 20px;\n  background-color: #f5f5f5;\n  border: 1px solid #e3e3e3;\n  border-radius: 4px;\n  -webkit-box-shadow: inset 0 1px 1px rgba(0, 0, 0, .05);\n          box-shadow: inset 0 1px 1px rgba(0, 0, 0, .05);\n}\n.well blockquote {\n  border-color: #ddd;\n  border-color: rgba(0, 0, 0, .15);\n}\n.well-lg {\n  padding: 24px;\n  border-radius: 6px;\n}\n.well-sm {\n  padding: 9px;\n  border-radius: 3px;\n}\n.close {\n  float: right;\n  font-size: 21px;\n  font-weight: bold;\n  line-height: 1;\n  color: #000;\n  text-shadow: 0 1px 0 #fff;\n  filter: alpha(opacity=20);\n  opacity: .2;\n}\n.close:hover,\n.close:focus {\n  color: #000;\n  text-decoration: none;\n  cursor: pointer;\n  filter: alpha(opacity=50);\n  opacity: .5;\n}\nbutton.close {\n  -webkit-appearance: none;\n  padding: 0;\n  cursor: pointer;\n  background: transparent;\n  border: 0;\n}\n.modal-open {\n  overflow: hidden;\n}\n.modal {\n  position: fixed;\n  top: 0;\n  right: 0;\n  bottom: 0;\n  left: 0;\n  z-index: 1050;\n  display: none;\n  overflow: auto;\n  overflow-y: scroll;\n  -webkit-overflow-scrolling: touch;\n  outline: 0;\n}\n.modal.fade .modal-dialog {\n  -webkit-transition: -webkit-transform .3s ease-out;\n     -moz-transition:    -moz-transform .3s ease-out;\n       -o-transition:      -o-transform .3s ease-out;\n          transition:         transform .3s ease-out;\n  -webkit-transform: translate(0, -25%);\n      -ms-transform: translate(0, -25%);\n          transform: translate(0, -25%);\n}\n.modal.in .modal-dialog {\n  -webkit-transform: translate(0, 0);\n      -ms-transform: translate(0, 0);\n          transform: translate(0, 0);\n}\n.modal-dialog {\n  position: relative;\n  width: auto;\n  margin: 10px;\n}\n.modal-content {\n  position: relative;\n  background-color: #fff;\n  background-clip: padding-box;\n  border: 1px solid #999;\n  border: 1px solid rgba(0, 0, 0, .2);\n  border-radius: 6px;\n  outline: none;\n  -webkit-box-shadow: 0 3px 9px rgba(0, 0, 0, .5);\n          box-shadow: 0 3px 9px rgba(0, 0, 0, .5);\n}\n.modal-backdrop {\n  position: fixed;\n  top: 0;\n  right: 0;\n  bottom: 0;\n  left: 0;\n  z-index: 1040;\n  background-color: #000;\n}\n.modal-backdrop.fade {\n  filter: alpha(opacity=0);\n  opacity: 0;\n}\n.modal-backdrop.in {\n  filter: alpha(opacity=50);\n  opacity: .5;\n}\n.modal-header {\n  min-height: 16.428571429px;\n  padding: 15px;\n  border-bottom: 1px solid #e5e5e5;\n}\n.modal-header .close {\n  margin-top: -2px;\n}\n.modal-title {\n  margin: 0;\n  line-height: 1.428571429;\n}\n.modal-body {\n  position: relative;\n  padding: 20px;\n}\n.modal-footer {\n  padding: 19px 20px 20px;\n  margin-top: 15px;\n  text-align: right;\n  border-top: 1px solid #e5e5e5;\n}\n.modal-footer .btn + .btn {\n  margin-bottom: 0;\n  margin-left: 5px;\n}\n.modal-footer .btn-group .btn + .btn {\n  margin-left: -1px;\n}\n.modal-footer .btn-block + .btn-block {\n  margin-left: 0;\n}\n@media (min-width: 768px) {\n  .modal-dialog {\n    width: 600px;\n    margin: 30px auto;\n  }\n  .modal-content {\n    -webkit-box-shadow: 0 5px 15px rgba(0, 0, 0, .5);\n            box-shadow: 0 5px 15px rgba(0, 0, 0, .5);\n  }\n  .modal-sm {\n    width: 300px;\n  }\n  .modal-lg {\n    width: 900px;\n  }\n}\n.tooltip {\n  position: absolute;\n  z-index: 1030;\n  display: block;\n  font-size: 12px;\n  line-height: 1.4;\n  visibility: visible;\n  filter: alpha(opacity=0);\n  opacity: 0;\n}\n.tooltip.in {\n  filter: alpha(opacity=90);\n  opacity: .9;\n}\n.tooltip.top {\n  padding: 5px 0;\n  margin-top: -3px;\n}\n.tooltip.right {\n  padding: 0 5px;\n  margin-left: 3px;\n}\n.tooltip.bottom {\n  padding: 5px 0;\n  margin-top: 3px;\n}\n.tooltip.left {\n  padding: 0 5px;\n  margin-left: -3px;\n}\n.tooltip-inner {\n  max-width: 200px;\n  padding: 3px 8px;\n  color: #fff;\n  text-align: center;\n  text-decoration: none;\n  background-color: #000;\n  border-radius: 4px;\n}\n.tooltip-arrow {\n  position: absolute;\n  width: 0;\n  height: 0;\n  border-color: transparent;\n  border-style: solid;\n}\n.tooltip.top .tooltip-arrow {\n  bottom: 0;\n  left: 50%;\n  margin-left: -5px;\n  border-width: 5px 5px 0;\n  border-top-color: #000;\n}\n.tooltip.top-left .tooltip-arrow {\n  bottom: 0;\n  left: 5px;\n  border-width: 5px 5px 0;\n  border-top-color: #000;\n}\n.tooltip.top-right .tooltip-arrow {\n  right: 5px;\n  bottom: 0;\n  border-width: 5px 5px 0;\n  border-top-color: #000;\n}\n.tooltip.right .tooltip-arrow {\n  top: 50%;\n  left: 0;\n  margin-top: -5px;\n  border-width: 5px 5px 5px 0;\n  border-right-color: #000;\n}\n.tooltip.left .tooltip-arrow {\n  top: 50%;\n  right: 0;\n  margin-top: -5px;\n  border-width: 5px 0 5px 5px;\n  border-left-color: #000;\n}\n.tooltip.bottom .tooltip-arrow {\n  top: 0;\n  left: 50%;\n  margin-left: -5px;\n  border-width: 0 5px 5px;\n  border-bottom-color: #000;\n}\n.tooltip.bottom-left .tooltip-arrow {\n  top: 0;\n  left: 5px;\n  border-width: 0 5px 5px;\n  border-bottom-color: #000;\n}\n.tooltip.bottom-right .tooltip-arrow {\n  top: 0;\n  right: 5px;\n  border-width: 0 5px 5px;\n  border-bottom-color: #000;\n}\n.popover {\n  position: absolute;\n  top: 0;\n  left: 0;\n  z-index: 1010;\n  display: none;\n  max-width: 276px;\n  padding: 1px;\n  text-align: left;\n  white-space: normal;\n  background-color: #fff;\n  background-clip: padding-box;\n  border: 1px solid #ccc;\n  border: 1px solid rgba(0, 0, 0, .2);\n  border-radius: 6px;\n  -webkit-box-shadow: 0 5px 10px rgba(0, 0, 0, .2);\n          box-shadow: 0 5px 10px rgba(0, 0, 0, .2);\n}\n.popover.top {\n  margin-top: -10px;\n}\n.popover.right {\n  margin-left: 10px;\n}\n.popover.bottom {\n  margin-top: 10px;\n}\n.popover.left {\n  margin-left: -10px;\n}\n.popover-title {\n  padding: 8px 14px;\n  margin: 0;\n  font-size: 14px;\n  font-weight: normal;\n  line-height: 18px;\n  background-color: #f7f7f7;\n  border-bottom: 1px solid #ebebeb;\n  border-radius: 5px 5px 0 0;\n}\n.popover-content {\n  padding: 9px 14px;\n}\n.popover .arrow,\n.popover .arrow:after {\n  position: absolute;\n  display: block;\n  width: 0;\n  height: 0;\n  border-color: transparent;\n  border-style: solid;\n}\n.popover .arrow {\n  border-width: 11px;\n}\n.popover .arrow:after {\n  content: \"\";\n  border-width: 10px;\n}\n.popover.top .arrow {\n  bottom: -11px;\n  left: 50%;\n  margin-left: -11px;\n  border-top-color: #999;\n  border-top-color: rgba(0, 0, 0, .25);\n  border-bottom-width: 0;\n}\n.popover.top .arrow:after {\n  bottom: 1px;\n  margin-left: -10px;\n  content: \" \";\n  border-top-color: #fff;\n  border-bottom-width: 0;\n}\n.popover.right .arrow {\n  top: 50%;\n  left: -11px;\n  margin-top: -11px;\n  border-right-color: #999;\n  border-right-color: rgba(0, 0, 0, .25);\n  border-left-width: 0;\n}\n.popover.right .arrow:after {\n  bottom: -10px;\n  left: 1px;\n  content: \" \";\n  border-right-color: #fff;\n  border-left-width: 0;\n}\n.popover.bottom .arrow {\n  top: -11px;\n  left: 50%;\n  margin-left: -11px;\n  border-top-width: 0;\n  border-bottom-color: #999;\n  border-bottom-color: rgba(0, 0, 0, .25);\n}\n.popover.bottom .arrow:after {\n  top: 1px;\n  margin-left: -10px;\n  content: \" \";\n  border-top-width: 0;\n  border-bottom-color: #fff;\n}\n.popover.left .arrow {\n  top: 50%;\n  right: -11px;\n  margin-top: -11px;\n  border-right-width: 0;\n  border-left-color: #999;\n  border-left-color: rgba(0, 0, 0, .25);\n}\n.popover.left .arrow:after {\n  right: 1px;\n  bottom: -10px;\n  content: \" \";\n  border-right-width: 0;\n  border-left-color: #fff;\n}\n.carousel {\n  position: relative;\n}\n.carousel-inner {\n  position: relative;\n  width: 100%;\n  overflow: hidden;\n}\n.carousel-inner > .item {\n  position: relative;\n  display: none;\n  -webkit-transition: .6s ease-in-out left;\n          transition: .6s ease-in-out left;\n}\n.carousel-inner > .item > img,\n.carousel-inner > .item > a > img {\n  display: block;\n  max-width: 100%;\n  height: auto;\n  line-height: 1;\n}\n.carousel-inner > .active,\n.carousel-inner > .next,\n.carousel-inner > .prev {\n  display: block;\n}\n.carousel-inner > .active {\n  left: 0;\n}\n.carousel-inner > .next,\n.carousel-inner > .prev {\n  position: absolute;\n  top: 0;\n  width: 100%;\n}\n.carousel-inner > .next {\n  left: 100%;\n}\n.carousel-inner > .prev {\n  left: -100%;\n}\n.carousel-inner > .next.left,\n.carousel-inner > .prev.right {\n  left: 0;\n}\n.carousel-inner > .active.left {\n  left: -100%;\n}\n.carousel-inner > .active.right {\n  left: 100%;\n}\n.carousel-control {\n  position: absolute;\n  top: 0;\n  bottom: 0;\n  left: 0;\n  width: 15%;\n  font-size: 20px;\n  color: #fff;\n  text-align: center;\n  text-shadow: 0 1px 2px rgba(0, 0, 0, .6);\n  filter: alpha(opacity=50);\n  opacity: .5;\n}\n.carousel-control.left {\n  background-image: -webkit-linear-gradient(left, color-stop(rgba(0, 0, 0, .5) 0%), color-stop(rgba(0, 0, 0, .0001) 100%));\n  background-image:         linear-gradient(to right, rgba(0, 0, 0, .5) 0%, rgba(0, 0, 0, .0001) 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#80000000', endColorstr='#00000000', GradientType=1);\n  background-repeat: repeat-x;\n}\n.carousel-control.right {\n  right: 0;\n  left: auto;\n  background-image: -webkit-linear-gradient(left, color-stop(rgba(0, 0, 0, .0001) 0%), color-stop(rgba(0, 0, 0, .5) 100%));\n  background-image:         linear-gradient(to right, rgba(0, 0, 0, .0001) 0%, rgba(0, 0, 0, .5) 100%);\n  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#00000000', endColorstr='#80000000', GradientType=1);\n  background-repeat: repeat-x;\n}\n.carousel-control:hover,\n.carousel-control:focus {\n  color: #fff;\n  text-decoration: none;\n  filter: alpha(opacity=90);\n  outline: none;\n  opacity: .9;\n}\n.carousel-control .icon-prev,\n.carousel-control .icon-next,\n.carousel-control .glyphicon-chevron-left,\n.carousel-control .glyphicon-chevron-right {\n  position: absolute;\n  top: 50%;\n  z-index: 5;\n  display: inline-block;\n}\n.carousel-control .icon-prev,\n.carousel-control .glyphicon-chevron-left {\n  left: 50%;\n}\n.carousel-control .icon-next,\n.carousel-control .glyphicon-chevron-right {\n  right: 50%;\n}\n.carousel-control .icon-prev,\n.carousel-control .icon-next {\n  width: 20px;\n  height: 20px;\n  margin-top: -10px;\n  margin-left: -10px;\n  font-family: serif;\n}\n.carousel-control .icon-prev:before {\n  content: '\\2039';\n}\n.carousel-control .icon-next:before {\n  content: '\\203a';\n}\n.carousel-indicators {\n  position: absolute;\n  bottom: 10px;\n  left: 50%;\n  z-index: 15;\n  width: 60%;\n  padding-left: 0;\n  margin-left: -30%;\n  text-align: center;\n  list-style: none;\n}\n.carousel-indicators li {\n  display: inline-block;\n  width: 10px;\n  height: 10px;\n  margin: 1px;\n  text-indent: -999px;\n  cursor: pointer;\n  background-color: #000 \\9;\n  background-color: rgba(0, 0, 0, 0);\n  border: 1px solid #fff;\n  border-radius: 10px;\n}\n.carousel-indicators .active {\n  width: 12px;\n  height: 12px;\n  margin: 0;\n  background-color: #fff;\n}\n.carousel-caption {\n  position: absolute;\n  right: 15%;\n  bottom: 20px;\n  left: 15%;\n  z-index: 10;\n  padding-top: 20px;\n  padding-bottom: 20px;\n  color: #fff;\n  text-align: center;\n  text-shadow: 0 1px 2px rgba(0, 0, 0, .6);\n}\n.carousel-caption .btn {\n  text-shadow: none;\n}\n@media screen and (min-width: 768px) {\n  .carousel-control .glyphicons-chevron-left,\n  .carousel-control .glyphicons-chevron-right,\n  .carousel-control .icon-prev,\n  .carousel-control .icon-next {\n    width: 30px;\n    height: 30px;\n    margin-top: -15px;\n    margin-left: -15px;\n    font-size: 30px;\n  }\n  .carousel-caption {\n    right: 20%;\n    left: 20%;\n    padding-bottom: 30px;\n  }\n  .carousel-indicators {\n    bottom: 20px;\n  }\n}\n.clearfix:before,\n.clearfix:after,\n.container:before,\n.container:after,\n.container-fluid:before,\n.container-fluid:after,\n.row:before,\n.row:after,\n.form-horizontal .form-group:before,\n.form-horizontal .form-group:after,\n.btn-toolbar:before,\n.btn-toolbar:after,\n.btn-group-vertical > .btn-group:before,\n.btn-group-vertical > .btn-group:after,\n.nav:before,\n.nav:after,\n.navbar:before,\n.navbar:after,\n.navbar-header:before,\n.navbar-header:after,\n.navbar-collapse:before,\n.navbar-collapse:after,\n.pager:before,\n.pager:after,\n.panel-body:before,\n.panel-body:after,\n.modal-footer:before,\n.modal-footer:after {\n  display: table;\n  content: \" \";\n}\n.clearfix:after,\n.container:after,\n.container-fluid:after,\n.row:after,\n.form-horizontal .form-group:after,\n.btn-toolbar:after,\n.btn-group-vertical > .btn-group:after,\n.nav:after,\n.navbar:after,\n.navbar-header:after,\n.navbar-collapse:after,\n.pager:after,\n.panel-body:after,\n.modal-footer:after {\n  clear: both;\n}\n.center-block {\n  display: block;\n  margin-right: auto;\n  margin-left: auto;\n}\n.pull-right {\n  float: right !important;\n}\n.pull-left {\n  float: left !important;\n}\n.hide {\n  display: none !important;\n}\n.show {\n  display: block !important;\n}\n.invisible {\n  visibility: hidden;\n}\n.text-hide {\n  font: 0/0 a;\n  color: transparent;\n  text-shadow: none;\n  background-color: transparent;\n  border: 0;\n}\n.hidden {\n  display: none !important;\n  visibility: hidden !important;\n}\n.affix {\n  position: fixed;\n}\n@-ms-viewport {\n  width: device-width;\n}\n.visible-xs,\ntr.visible-xs,\nth.visible-xs,\ntd.visible-xs {\n  display: none !important;\n}\n@media (max-width: 767px) {\n  .visible-xs {\n    display: block !important;\n  }\n  table.visible-xs {\n    display: table;\n  }\n  tr.visible-xs {\n    display: table-row !important;\n  }\n  th.visible-xs,\n  td.visible-xs {\n    display: table-cell !important;\n  }\n}\n.visible-sm,\ntr.visible-sm,\nth.visible-sm,\ntd.visible-sm {\n  display: none !important;\n}\n@media (min-width: 768px) and (max-width: 991px) {\n  .visible-sm {\n    display: block !important;\n  }\n  table.visible-sm {\n    display: table;\n  }\n  tr.visible-sm {\n    display: table-row !important;\n  }\n  th.visible-sm,\n  td.visible-sm {\n    display: table-cell !important;\n  }\n}\n.visible-md,\ntr.visible-md,\nth.visible-md,\ntd.visible-md {\n  display: none !important;\n}\n@media (min-width: 992px) and (max-width: 1199px) {\n  .visible-md {\n    display: block !important;\n  }\n  table.visible-md {\n    display: table;\n  }\n  tr.visible-md {\n    display: table-row !important;\n  }\n  th.visible-md,\n  td.visible-md {\n    display: table-cell !important;\n  }\n}\n.visible-lg,\ntr.visible-lg,\nth.visible-lg,\ntd.visible-lg {\n  display: none !important;\n}\n@media (min-width: 1200px) {\n  .visible-lg {\n    display: block !important;\n  }\n  table.visible-lg {\n    display: table;\n  }\n  tr.visible-lg {\n    display: table-row !important;\n  }\n  th.visible-lg,\n  td.visible-lg {\n    display: table-cell !important;\n  }\n}\n@media (max-width: 767px) {\n  .hidden-xs,\n  tr.hidden-xs,\n  th.hidden-xs,\n  td.hidden-xs {\n    display: none !important;\n  }\n}\n@media (min-width: 768px) and (max-width: 991px) {\n  .hidden-sm,\n  tr.hidden-sm,\n  th.hidden-sm,\n  td.hidden-sm {\n    display: none !important;\n  }\n}\n@media (min-width: 992px) and (max-width: 1199px) {\n  .hidden-md,\n  tr.hidden-md,\n  th.hidden-md,\n  td.hidden-md {\n    display: none !important;\n  }\n}\n@media (min-width: 1200px) {\n  .hidden-lg,\n  tr.hidden-lg,\n  th.hidden-lg,\n  td.hidden-lg {\n    display: none !important;\n  }\n}\n.visible-print,\ntr.visible-print,\nth.visible-print,\ntd.visible-print {\n  display: none !important;\n}\n@media print {\n  .visible-print {\n    display: block !important;\n  }\n  table.visible-print {\n    display: table;\n  }\n  tr.visible-print {\n    display: table-row !important;\n  }\n  th.visible-print,\n  td.visible-print {\n    display: table-cell !important;\n  }\n}\n@media print {\n  .hidden-print,\n  tr.hidden-print,\n  th.hidden-print,\n  td.hidden-print {\n    display: none !important;\n  }\n}\n/*# sourceMappingURL=bootstrap.css.map */\n"
  },
  {
    "path": "doc/css/button-override.css",
    "content": "/* Copyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0 */\n\n.btn:hover,\n.btn:focus,\n.btn:active,\n.btn.active,\n.open .dropdown-toggle.btn {\n  opacity: 0.4;\n}\n"
  },
  {
    "path": "doc/css/carousel-override.css",
    "content": "/* Copyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0 */\n\n.carousel-control {\n  width: 5%;\n}\n\n.carousel-caption {\n  position: static;\n  background: rgba(0,0,0,0.6);\n  color: white;\n  padding-bottom: 35px;\n  padding-left: 1em;\n  padding-right: 1em;\n  padding-top: 15px;\n}\n\n.carousel {\n  overflow: hidden;\n  border-radius: 5px;\n  max-width: 900px;\n  margin: 1em;\n}\n\n.carousel-indicators {\n  bottom: 0px;\n}\n\n"
  },
  {
    "path": "doc/css/code.css",
    "content": "/* Copyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0 */\n\ntable.code {\n    font-family: Menlo,Monaco,Consolas,\"Courier New\",monospace;\n    display: block;\n    padding: 9.5px;\n    margin: 0px 0px 10px;\n    font-size: 13px;\n    line-height: 1.42857;\n    color: rgb(51, 51, 51);\n    word-break: break-all;\n    word-wrap: break-word;\n    background-color: rgb(245, 245, 245);\n    border: 1px solid rgb(204, 204, 204);\n    border-radius: 4px 4px 4px 4px;\n}\n\ntable.code tr td {\n    white-space: pre;\n}\n\ntable.code tr td:nth-child(2) {\n    color: #d14;\n    padding-left: .5em;\n}\n\n.userinput {\n    color: #d14;\n}\n\ntable.CodeRay {\n    margin-left: 3em;\n    width: calc(100% - 6em);\n}\n\ntd.line-numbers {\n    width: 2em;\n}\n\n.releasenotes h2 { margin-top: 1.5em; text-decoration: underline; }\n"
  },
  {
    "path": "doc/css/font-awesome.css",
    "content": "/*!\n *  Font Awesome 4.1.0 by @davegandy - http://fontawesome.io - @fontawesome\n *  License - http://fontawesome.io/license (Font: SIL OFL 1.1, CSS: MIT License)\n */\n/* FONT PATH\n * -------------------------- */\n@font-face {\n  font-family: 'FontAwesome';\n  src: url('../fonts/fontawesome-webfont.eot?v=4.1.0');\n  src: url('../fonts/fontawesome-webfont.eot?#iefix&v=4.1.0') format('embedded-opentype'), url('../fonts/fontawesome-webfont.woff?v=4.1.0') format('woff'), url('../fonts/fontawesome-webfont.ttf?v=4.1.0') format('truetype'), url('../fonts/fontawesome-webfont.svg?v=4.1.0#fontawesomeregular') format('svg');\n  font-weight: normal;\n  font-style: normal;\n}\n.fa {\n  display: inline-block;\n  font-family: 'FontAwesome', sans-serif;\n  font-style: normal;\n  font-weight: normal;\n  line-height: 1;\n  -webkit-font-smoothing: antialiased;\n  -moz-osx-font-smoothing: grayscale;\n}\n/* makes the font 33% larger relative to the icon container */\n.fa-lg {\n  font-size: 1.33333333em;\n  line-height: 0.75em;\n  vertical-align: -15%;\n}\n.fa-2x {\n  font-size: 2em;\n}\n.fa-3x {\n  font-size: 3em;\n}\n.fa-4x {\n  font-size: 4em;\n}\n.fa-5x {\n  font-size: 5em;\n}\n.fa-fw {\n  width: 1.28571429em;\n  text-align: center;\n}\n.fa-ul {\n  padding-left: 0;\n  margin-left: 2.14285714em;\n  list-style-type: none;\n}\n.fa-ul > li {\n  position: relative;\n}\n.fa-li {\n  position: absolute;\n  left: -2.14285714em;\n  width: 2.14285714em;\n  top: 0.14285714em;\n  text-align: center;\n}\n.fa-li.fa-lg {\n  left: -1.85714286em;\n}\n.fa-border {\n  padding: .2em .25em .15em;\n  border: solid 0.08em #eeeeee;\n  border-radius: .1em;\n}\n.pull-right {\n  float: right;\n}\n.pull-left {\n  float: left;\n}\n.fa.pull-left {\n  margin-right: .3em;\n}\n.fa.pull-right {\n  margin-left: .3em;\n}\n.fa-spin {\n  -webkit-animation: spin 2s infinite linear;\n  -moz-animation: spin 2s infinite linear;\n  -o-animation: spin 2s infinite linear;\n  animation: spin 2s infinite linear;\n}\n@-moz-keyframes spin {\n  0% {\n    -moz-transform: rotate(0deg);\n  }\n  100% {\n    -moz-transform: rotate(359deg);\n  }\n}\n@-webkit-keyframes spin {\n  0% {\n    -webkit-transform: rotate(0deg);\n  }\n  100% {\n    -webkit-transform: rotate(359deg);\n  }\n}\n@-o-keyframes spin {\n  0% {\n    -o-transform: rotate(0deg);\n  }\n  100% {\n    -o-transform: rotate(359deg);\n  }\n}\n@keyframes spin {\n  0% {\n    -webkit-transform: rotate(0deg);\n    transform: rotate(0deg);\n  }\n  100% {\n    -webkit-transform: rotate(359deg);\n    transform: rotate(359deg);\n  }\n}\n.fa-rotate-90 {\n  filter: progid:DXImageTransform.Microsoft.BasicImage(rotation=1);\n  -webkit-transform: rotate(90deg);\n  -moz-transform: rotate(90deg);\n  -ms-transform: rotate(90deg);\n  -o-transform: rotate(90deg);\n  transform: rotate(90deg);\n}\n.fa-rotate-180 {\n  filter: progid:DXImageTransform.Microsoft.BasicImage(rotation=2);\n  -webkit-transform: rotate(180deg);\n  -moz-transform: rotate(180deg);\n  -ms-transform: rotate(180deg);\n  -o-transform: rotate(180deg);\n  transform: rotate(180deg);\n}\n.fa-rotate-270 {\n  filter: progid:DXImageTransform.Microsoft.BasicImage(rotation=3);\n  -webkit-transform: rotate(270deg);\n  -moz-transform: rotate(270deg);\n  -ms-transform: rotate(270deg);\n  -o-transform: rotate(270deg);\n  transform: rotate(270deg);\n}\n.fa-flip-horizontal {\n  filter: progid:DXImageTransform.Microsoft.BasicImage(rotation=0, mirror=1);\n  -webkit-transform: scale(-1, 1);\n  -moz-transform: scale(-1, 1);\n  -ms-transform: scale(-1, 1);\n  -o-transform: scale(-1, 1);\n  transform: scale(-1, 1);\n}\n.fa-flip-vertical {\n  filter: progid:DXImageTransform.Microsoft.BasicImage(rotation=2, mirror=1);\n  -webkit-transform: scale(1, -1);\n  -moz-transform: scale(1, -1);\n  -ms-transform: scale(1, -1);\n  -o-transform: scale(1, -1);\n  transform: scale(1, -1);\n}\n.fa-stack {\n  position: relative;\n  display: inline-block;\n  width: 2em;\n  height: 2em;\n  line-height: 2em;\n  vertical-align: middle;\n}\n.fa-stack-1x,\n.fa-stack-2x {\n  position: absolute;\n  left: 0;\n  width: 100%;\n  text-align: center;\n}\n.fa-stack-1x {\n  line-height: inherit;\n}\n.fa-stack-2x {\n  font-size: 2em;\n}\n.fa-inverse {\n  color: #ffffff;\n}\n/* Font Awesome uses the Unicode Private Use Area (PUA) to ensure screen\n   readers do not read off random characters that represent icons */\n.fa-glass:before {\n  content: \"\\f000\";\n}\n.fa-music:before {\n  content: \"\\f001\";\n}\n.fa-search:before {\n  content: \"\\f002\";\n}\n.fa-envelope-o:before {\n  content: \"\\f003\";\n}\n.fa-heart:before {\n  content: \"\\f004\";\n}\n.fa-star:before {\n  content: \"\\f005\";\n}\n.fa-star-o:before {\n  content: \"\\f006\";\n}\n.fa-user:before {\n  content: \"\\f007\";\n}\n.fa-film:before {\n  content: \"\\f008\";\n}\n.fa-th-large:before {\n  content: \"\\f009\";\n}\n.fa-th:before {\n  content: \"\\f00a\";\n}\n.fa-th-list:before {\n  content: \"\\f00b\";\n}\n.fa-check:before {\n  content: \"\\f00c\";\n}\n.fa-times:before {\n  content: \"\\f00d\";\n}\n.fa-search-plus:before {\n  content: \"\\f00e\";\n}\n.fa-search-minus:before {\n  content: \"\\f010\";\n}\n.fa-power-off:before {\n  content: \"\\f011\";\n}\n.fa-signal:before {\n  content: \"\\f012\";\n}\n.fa-gear:before,\n.fa-cog:before {\n  content: \"\\f013\";\n}\n.fa-trash-o:before {\n  content: \"\\f014\";\n}\n.fa-home:before {\n  content: \"\\f015\";\n}\n.fa-file-o:before {\n  content: \"\\f016\";\n}\n.fa-clock-o:before {\n  content: \"\\f017\";\n}\n.fa-road:before {\n  content: \"\\f018\";\n}\n.fa-download:before {\n  content: \"\\f019\";\n}\n.fa-arrow-circle-o-down:before {\n  content: \"\\f01a\";\n}\n.fa-arrow-circle-o-up:before {\n  content: \"\\f01b\";\n}\n.fa-inbox:before {\n  content: \"\\f01c\";\n}\n.fa-play-circle-o:before {\n  content: \"\\f01d\";\n}\n.fa-rotate-right:before,\n.fa-repeat:before {\n  content: \"\\f01e\";\n}\n.fa-refresh:before {\n  content: \"\\f021\";\n}\n.fa-list-alt:before {\n  content: \"\\f022\";\n}\n.fa-lock:before {\n  content: \"\\f023\";\n}\n.fa-flag:before {\n  content: \"\\f024\";\n}\n.fa-headphones:before {\n  content: \"\\f025\";\n}\n.fa-volume-off:before {\n  content: \"\\f026\";\n}\n.fa-volume-down:before {\n  content: \"\\f027\";\n}\n.fa-volume-up:before {\n  content: \"\\f028\";\n}\n.fa-qrcode:before {\n  content: \"\\f029\";\n}\n.fa-barcode:before {\n  content: \"\\f02a\";\n}\n.fa-tag:before {\n  content: \"\\f02b\";\n}\n.fa-tags:before {\n  content: \"\\f02c\";\n}\n.fa-book:before {\n  content: \"\\f02d\";\n}\n.fa-bookmark:before {\n  content: \"\\f02e\";\n}\n.fa-print:before {\n  content: \"\\f02f\";\n}\n.fa-camera:before {\n  content: \"\\f030\";\n}\n.fa-font:before {\n  content: \"\\f031\";\n}\n.fa-bold:before {\n  content: \"\\f032\";\n}\n.fa-italic:before {\n  content: \"\\f033\";\n}\n.fa-text-height:before {\n  content: \"\\f034\";\n}\n.fa-text-width:before {\n  content: \"\\f035\";\n}\n.fa-align-left:before {\n  content: \"\\f036\";\n}\n.fa-align-center:before {\n  content: \"\\f037\";\n}\n.fa-align-right:before {\n  content: \"\\f038\";\n}\n.fa-align-justify:before {\n  content: \"\\f039\";\n}\n.fa-list:before {\n  content: \"\\f03a\";\n}\n.fa-dedent:before,\n.fa-outdent:before {\n  content: \"\\f03b\";\n}\n.fa-indent:before {\n  content: \"\\f03c\";\n}\n.fa-video-camera:before {\n  content: \"\\f03d\";\n}\n.fa-photo:before,\n.fa-image:before,\n.fa-picture-o:before {\n  content: \"\\f03e\";\n}\n.fa-pencil:before {\n  content: \"\\f040\";\n}\n.fa-map-marker:before {\n  content: \"\\f041\";\n}\n.fa-adjust:before {\n  content: \"\\f042\";\n}\n.fa-tint:before {\n  content: \"\\f043\";\n}\n.fa-edit:before,\n.fa-pencil-square-o:before {\n  content: \"\\f044\";\n}\n.fa-share-square-o:before {\n  content: \"\\f045\";\n}\n.fa-check-square-o:before {\n  content: \"\\f046\";\n}\n.fa-arrows:before {\n  content: \"\\f047\";\n}\n.fa-step-backward:before {\n  content: \"\\f048\";\n}\n.fa-fast-backward:before {\n  content: \"\\f049\";\n}\n.fa-backward:before {\n  content: \"\\f04a\";\n}\n.fa-play:before {\n  content: \"\\f04b\";\n}\n.fa-pause:before {\n  content: \"\\f04c\";\n}\n.fa-stop:before {\n  content: \"\\f04d\";\n}\n.fa-forward:before {\n  content: \"\\f04e\";\n}\n.fa-fast-forward:before {\n  content: \"\\f050\";\n}\n.fa-step-forward:before {\n  content: \"\\f051\";\n}\n.fa-eject:before {\n  content: \"\\f052\";\n}\n.fa-chevron-left:before {\n  content: \"\\f053\";\n}\n.fa-chevron-right:before {\n  content: \"\\f054\";\n}\n.fa-plus-circle:before {\n  content: \"\\f055\";\n}\n.fa-minus-circle:before {\n  content: \"\\f056\";\n}\n.fa-times-circle:before {\n  content: \"\\f057\";\n}\n.fa-check-circle:before {\n  content: \"\\f058\";\n}\n.fa-question-circle:before {\n  content: \"\\f059\";\n}\n.fa-info-circle:before {\n  content: \"\\f05a\";\n}\n.fa-crosshairs:before {\n  content: \"\\f05b\";\n}\n.fa-times-circle-o:before {\n  content: \"\\f05c\";\n}\n.fa-check-circle-o:before {\n  content: \"\\f05d\";\n}\n.fa-ban:before {\n  content: \"\\f05e\";\n}\n.fa-arrow-left:before {\n  content: \"\\f060\";\n}\n.fa-arrow-right:before {\n  content: \"\\f061\";\n}\n.fa-arrow-up:before {\n  content: \"\\f062\";\n}\n.fa-arrow-down:before {\n  content: \"\\f063\";\n}\n.fa-mail-forward:before,\n.fa-share:before {\n  content: \"\\f064\";\n}\n.fa-expand:before {\n  content: \"\\f065\";\n}\n.fa-compress:before {\n  content: \"\\f066\";\n}\n.fa-plus:before {\n  content: \"\\f067\";\n}\n.fa-minus:before {\n  content: \"\\f068\";\n}\n.fa-asterisk:before {\n  content: \"\\f069\";\n}\n.fa-exclamation-circle:before {\n  content: \"\\f06a\";\n}\n.fa-gift:before {\n  content: \"\\f06b\";\n}\n.fa-leaf:before {\n  content: \"\\f06c\";\n}\n.fa-fire:before {\n  content: \"\\f06d\";\n}\n.fa-eye:before {\n  content: \"\\f06e\";\n}\n.fa-eye-slash:before {\n  content: \"\\f070\";\n}\n.fa-warning:before,\n.fa-exclamation-triangle:before {\n  content: \"\\f071\";\n}\n.fa-plane:before {\n  content: \"\\f072\";\n}\n.fa-calendar:before {\n  content: \"\\f073\";\n}\n.fa-random:before {\n  content: \"\\f074\";\n}\n.fa-comment:before {\n  content: \"\\f075\";\n}\n.fa-magnet:before {\n  content: \"\\f076\";\n}\n.fa-chevron-up:before {\n  content: \"\\f077\";\n}\n.fa-chevron-down:before {\n  content: \"\\f078\";\n}\n.fa-retweet:before {\n  content: \"\\f079\";\n}\n.fa-shopping-cart:before {\n  content: \"\\f07a\";\n}\n.fa-folder:before {\n  content: \"\\f07b\";\n}\n.fa-folder-open:before {\n  content: \"\\f07c\";\n}\n.fa-arrows-v:before {\n  content: \"\\f07d\";\n}\n.fa-arrows-h:before {\n  content: \"\\f07e\";\n}\n.fa-bar-chart-o:before {\n  content: \"\\f080\";\n}\n.fa-twitter-square:before {\n  content: \"\\f081\";\n}\n.fa-facebook-square:before {\n  content: \"\\f082\";\n}\n.fa-camera-retro:before {\n  content: \"\\f083\";\n}\n.fa-key:before {\n  content: \"\\f084\";\n}\n.fa-gears:before,\n.fa-cogs:before {\n  content: \"\\f085\";\n}\n.fa-comments:before {\n  content: \"\\f086\";\n}\n.fa-thumbs-o-up:before {\n  content: \"\\f087\";\n}\n.fa-thumbs-o-down:before {\n  content: \"\\f088\";\n}\n.fa-star-half:before {\n  content: \"\\f089\";\n}\n.fa-heart-o:before {\n  content: \"\\f08a\";\n}\n.fa-sign-out:before {\n  content: \"\\f08b\";\n}\n.fa-linkedin-square:before {\n  content: \"\\f08c\";\n}\n.fa-thumb-tack:before {\n  content: \"\\f08d\";\n}\n.fa-external-link:before {\n  content: \"\\f08e\";\n}\n.fa-sign-in:before {\n  content: \"\\f090\";\n}\n.fa-trophy:before {\n  content: \"\\f091\";\n}\n.fa-github-square:before {\n  content: \"\\f092\";\n}\n.fa-upload:before {\n  content: \"\\f093\";\n}\n.fa-lemon-o:before {\n  content: \"\\f094\";\n}\n.fa-phone:before {\n  content: \"\\f095\";\n}\n.fa-square-o:before {\n  content: \"\\f096\";\n}\n.fa-bookmark-o:before {\n  content: \"\\f097\";\n}\n.fa-phone-square:before {\n  content: \"\\f098\";\n}\n.fa-twitter:before {\n  content: \"\\f099\";\n}\n.fa-facebook:before {\n  content: \"\\f09a\";\n}\n.fa-github:before {\n  content: \"\\f09b\";\n}\n.fa-unlock:before {\n  content: \"\\f09c\";\n}\n.fa-credit-card:before {\n  content: \"\\f09d\";\n}\n.fa-rss:before {\n  content: \"\\f09e\";\n}\n.fa-hdd-o:before {\n  content: \"\\f0a0\";\n}\n.fa-bullhorn:before {\n  content: \"\\f0a1\";\n}\n.fa-bell:before {\n  content: \"\\f0f3\";\n}\n.fa-certificate:before {\n  content: \"\\f0a3\";\n}\n.fa-hand-o-right:before {\n  content: \"\\f0a4\";\n}\n.fa-hand-o-left:before {\n  content: \"\\f0a5\";\n}\n.fa-hand-o-up:before {\n  content: \"\\f0a6\";\n}\n.fa-hand-o-down:before {\n  content: \"\\f0a7\";\n}\n.fa-arrow-circle-left:before {\n  content: \"\\f0a8\";\n}\n.fa-arrow-circle-right:before {\n  content: \"\\f0a9\";\n}\n.fa-arrow-circle-up:before {\n  content: \"\\f0aa\";\n}\n.fa-arrow-circle-down:before {\n  content: \"\\f0ab\";\n}\n.fa-globe:before {\n  content: \"\\f0ac\";\n}\n.fa-wrench:before {\n  content: \"\\f0ad\";\n}\n.fa-tasks:before {\n  content: \"\\f0ae\";\n}\n.fa-filter:before {\n  content: \"\\f0b0\";\n}\n.fa-briefcase:before {\n  content: \"\\f0b1\";\n}\n.fa-arrows-alt:before {\n  content: \"\\f0b2\";\n}\n.fa-group:before,\n.fa-users:before {\n  content: \"\\f0c0\";\n}\n.fa-chain:before,\n.fa-link:before {\n  content: \"\\f0c1\";\n}\n.fa-cloud:before {\n  content: \"\\f0c2\";\n}\n.fa-flask:before {\n  content: \"\\f0c3\";\n}\n.fa-cut:before,\n.fa-scissors:before {\n  content: \"\\f0c4\";\n}\n.fa-copy:before,\n.fa-files-o:before {\n  content: \"\\f0c5\";\n}\n.fa-paperclip:before {\n  content: \"\\f0c6\";\n}\n.fa-save:before,\n.fa-floppy-o:before {\n  content: \"\\f0c7\";\n}\n.fa-square:before {\n  content: \"\\f0c8\";\n}\n.fa-navicon:before,\n.fa-reorder:before,\n.fa-bars:before {\n  content: \"\\f0c9\";\n}\n.fa-list-ul:before {\n  content: \"\\f0ca\";\n}\n.fa-list-ol:before {\n  content: \"\\f0cb\";\n}\n.fa-strikethrough:before {\n  content: \"\\f0cc\";\n}\n.fa-underline:before {\n  content: \"\\f0cd\";\n}\n.fa-table:before {\n  content: \"\\f0ce\";\n}\n.fa-magic:before {\n  content: \"\\f0d0\";\n}\n.fa-truck:before {\n  content: \"\\f0d1\";\n}\n.fa-pinterest:before {\n  content: \"\\f0d2\";\n}\n.fa-pinterest-square:before {\n  content: \"\\f0d3\";\n}\n.fa-google-plus-square:before {\n  content: \"\\f0d4\";\n}\n.fa-google-plus:before {\n  content: \"\\f0d5\";\n}\n.fa-money:before {\n  content: \"\\f0d6\";\n}\n.fa-caret-down:before {\n  content: \"\\f0d7\";\n}\n.fa-caret-up:before {\n  content: \"\\f0d8\";\n}\n.fa-caret-left:before {\n  content: \"\\f0d9\";\n}\n.fa-caret-right:before {\n  content: \"\\f0da\";\n}\n.fa-columns:before {\n  content: \"\\f0db\";\n}\n.fa-unsorted:before,\n.fa-sort:before {\n  content: \"\\f0dc\";\n}\n.fa-sort-down:before,\n.fa-sort-desc:before {\n  content: \"\\f0dd\";\n}\n.fa-sort-up:before,\n.fa-sort-asc:before {\n  content: \"\\f0de\";\n}\n.fa-envelope:before {\n  content: \"\\f0e0\";\n}\n.fa-linkedin:before {\n  content: \"\\f0e1\";\n}\n.fa-rotate-left:before,\n.fa-undo:before {\n  content: \"\\f0e2\";\n}\n.fa-legal:before,\n.fa-gavel:before {\n  content: \"\\f0e3\";\n}\n.fa-dashboard:before,\n.fa-tachometer:before {\n  content: \"\\f0e4\";\n}\n.fa-comment-o:before {\n  content: \"\\f0e5\";\n}\n.fa-comments-o:before {\n  content: \"\\f0e6\";\n}\n.fa-flash:before,\n.fa-bolt:before {\n  content: \"\\f0e7\";\n}\n.fa-sitemap:before {\n  content: \"\\f0e8\";\n}\n.fa-umbrella:before {\n  content: \"\\f0e9\";\n}\n.fa-paste:before,\n.fa-clipboard:before {\n  content: \"\\f0ea\";\n}\n.fa-lightbulb-o:before {\n  content: \"\\f0eb\";\n}\n.fa-exchange:before {\n  content: \"\\f0ec\";\n}\n.fa-cloud-download:before {\n  content: \"\\f0ed\";\n}\n.fa-cloud-upload:before {\n  content: \"\\f0ee\";\n}\n.fa-user-md:before {\n  content: \"\\f0f0\";\n}\n.fa-stethoscope:before {\n  content: \"\\f0f1\";\n}\n.fa-suitcase:before {\n  content: \"\\f0f2\";\n}\n.fa-bell-o:before {\n  content: \"\\f0a2\";\n}\n.fa-coffee:before {\n  content: \"\\f0f4\";\n}\n.fa-cutlery:before {\n  content: \"\\f0f5\";\n}\n.fa-file-text-o:before {\n  content: \"\\f0f6\";\n}\n.fa-building-o:before {\n  content: \"\\f0f7\";\n}\n.fa-hospital-o:before {\n  content: \"\\f0f8\";\n}\n.fa-ambulance:before {\n  content: \"\\f0f9\";\n}\n.fa-medkit:before {\n  content: \"\\f0fa\";\n}\n.fa-fighter-jet:before {\n  content: \"\\f0fb\";\n}\n.fa-beer:before {\n  content: \"\\f0fc\";\n}\n.fa-h-square:before {\n  content: \"\\f0fd\";\n}\n.fa-plus-square:before {\n  content: \"\\f0fe\";\n}\n.fa-angle-double-left:before {\n  content: \"\\f100\";\n}\n.fa-angle-double-right:before {\n  content: \"\\f101\";\n}\n.fa-angle-double-up:before {\n  content: \"\\f102\";\n}\n.fa-angle-double-down:before {\n  content: \"\\f103\";\n}\n.fa-angle-left:before {\n  content: \"\\f104\";\n}\n.fa-angle-right:before {\n  content: \"\\f105\";\n}\n.fa-angle-up:before {\n  content: \"\\f106\";\n}\n.fa-angle-down:before {\n  content: \"\\f107\";\n}\n.fa-desktop:before {\n  content: \"\\f108\";\n}\n.fa-laptop:before {\n  content: \"\\f109\";\n}\n.fa-tablet:before {\n  content: \"\\f10a\";\n}\n.fa-mobile-phone:before,\n.fa-mobile:before {\n  content: \"\\f10b\";\n}\n.fa-circle-o:before {\n  content: \"\\f10c\";\n}\n.fa-quote-left:before {\n  content: \"\\f10d\";\n}\n.fa-quote-right:before {\n  content: \"\\f10e\";\n}\n.fa-spinner:before {\n  content: \"\\f110\";\n}\n.fa-circle:before {\n  content: \"\\f111\";\n}\n.fa-mail-reply:before,\n.fa-reply:before {\n  content: \"\\f112\";\n}\n.fa-github-alt:before {\n  content: \"\\f113\";\n}\n.fa-folder-o:before {\n  content: \"\\f114\";\n}\n.fa-folder-open-o:before {\n  content: \"\\f115\";\n}\n.fa-smile-o:before {\n  content: \"\\f118\";\n}\n.fa-frown-o:before {\n  content: \"\\f119\";\n}\n.fa-meh-o:before {\n  content: \"\\f11a\";\n}\n.fa-gamepad:before {\n  content: \"\\f11b\";\n}\n.fa-keyboard-o:before {\n  content: \"\\f11c\";\n}\n.fa-flag-o:before {\n  content: \"\\f11d\";\n}\n.fa-flag-checkered:before {\n  content: \"\\f11e\";\n}\n.fa-terminal:before {\n  content: \"\\f120\";\n}\n.fa-code:before {\n  content: \"\\f121\";\n}\n.fa-mail-reply-all:before,\n.fa-reply-all:before {\n  content: \"\\f122\";\n}\n.fa-star-half-empty:before,\n.fa-star-half-full:before,\n.fa-star-half-o:before {\n  content: \"\\f123\";\n}\n.fa-location-arrow:before {\n  content: \"\\f124\";\n}\n.fa-crop:before {\n  content: \"\\f125\";\n}\n.fa-code-fork:before {\n  content: \"\\f126\";\n}\n.fa-unlink:before,\n.fa-chain-broken:before {\n  content: \"\\f127\";\n}\n.fa-question:before {\n  content: \"\\f128\";\n}\n.fa-info:before {\n  content: \"\\f129\";\n}\n.fa-exclamation:before {\n  content: \"\\f12a\";\n}\n.fa-superscript:before {\n  content: \"\\f12b\";\n}\n.fa-subscript:before {\n  content: \"\\f12c\";\n}\n.fa-eraser:before {\n  content: \"\\f12d\";\n}\n.fa-puzzle-piece:before {\n  content: \"\\f12e\";\n}\n.fa-microphone:before {\n  content: \"\\f130\";\n}\n.fa-microphone-slash:before {\n  content: \"\\f131\";\n}\n.fa-shield:before {\n  content: \"\\f132\";\n}\n.fa-calendar-o:before {\n  content: \"\\f133\";\n}\n.fa-fire-extinguisher:before {\n  content: \"\\f134\";\n}\n.fa-rocket:before {\n  content: \"\\f135\";\n}\n.fa-maxcdn:before {\n  content: \"\\f136\";\n}\n.fa-chevron-circle-left:before {\n  content: \"\\f137\";\n}\n.fa-chevron-circle-right:before {\n  content: \"\\f138\";\n}\n.fa-chevron-circle-up:before {\n  content: \"\\f139\";\n}\n.fa-chevron-circle-down:before {\n  content: \"\\f13a\";\n}\n.fa-html5:before {\n  content: \"\\f13b\";\n}\n.fa-css3:before {\n  content: \"\\f13c\";\n}\n.fa-anchor:before {\n  content: \"\\f13d\";\n}\n.fa-unlock-alt:before {\n  content: \"\\f13e\";\n}\n.fa-bullseye:before {\n  content: \"\\f140\";\n}\n.fa-ellipsis-h:before {\n  content: \"\\f141\";\n}\n.fa-ellipsis-v:before {\n  content: \"\\f142\";\n}\n.fa-rss-square:before {\n  content: \"\\f143\";\n}\n.fa-play-circle:before {\n  content: \"\\f144\";\n}\n.fa-ticket:before {\n  content: \"\\f145\";\n}\n.fa-minus-square:before {\n  content: \"\\f146\";\n}\n.fa-minus-square-o:before {\n  content: \"\\f147\";\n}\n.fa-level-up:before {\n  content: \"\\f148\";\n}\n.fa-level-down:before {\n  content: \"\\f149\";\n}\n.fa-check-square:before {\n  content: \"\\f14a\";\n}\n.fa-pencil-square:before {\n  content: \"\\f14b\";\n}\n.fa-external-link-square:before {\n  content: \"\\f14c\";\n}\n.fa-share-square:before {\n  content: \"\\f14d\";\n}\n.fa-compass:before {\n  content: \"\\f14e\";\n}\n.fa-toggle-down:before,\n.fa-caret-square-o-down:before {\n  content: \"\\f150\";\n}\n.fa-toggle-up:before,\n.fa-caret-square-o-up:before {\n  content: \"\\f151\";\n}\n.fa-toggle-right:before,\n.fa-caret-square-o-right:before {\n  content: \"\\f152\";\n}\n.fa-euro:before,\n.fa-eur:before {\n  content: \"\\f153\";\n}\n.fa-gbp:before {\n  content: \"\\f154\";\n}\n.fa-dollar:before,\n.fa-usd:before {\n  content: \"\\f155\";\n}\n.fa-rupee:before,\n.fa-inr:before {\n  content: \"\\f156\";\n}\n.fa-cny:before,\n.fa-rmb:before,\n.fa-yen:before,\n.fa-jpy:before {\n  content: \"\\f157\";\n}\n.fa-ruble:before,\n.fa-rouble:before,\n.fa-rub:before {\n  content: \"\\f158\";\n}\n.fa-won:before,\n.fa-krw:before {\n  content: \"\\f159\";\n}\n.fa-bitcoin:before,\n.fa-btc:before {\n  content: \"\\f15a\";\n}\n.fa-file:before {\n  content: \"\\f15b\";\n}\n.fa-file-text:before {\n  content: \"\\f15c\";\n}\n.fa-sort-alpha-asc:before {\n  content: \"\\f15d\";\n}\n.fa-sort-alpha-desc:before {\n  content: \"\\f15e\";\n}\n.fa-sort-amount-asc:before {\n  content: \"\\f160\";\n}\n.fa-sort-amount-desc:before {\n  content: \"\\f161\";\n}\n.fa-sort-numeric-asc:before {\n  content: \"\\f162\";\n}\n.fa-sort-numeric-desc:before {\n  content: \"\\f163\";\n}\n.fa-thumbs-up:before {\n  content: \"\\f164\";\n}\n.fa-thumbs-down:before {\n  content: \"\\f165\";\n}\n.fa-youtube-square:before {\n  content: \"\\f166\";\n}\n.fa-youtube:before {\n  content: \"\\f167\";\n}\n.fa-xing:before {\n  content: \"\\f168\";\n}\n.fa-xing-square:before {\n  content: \"\\f169\";\n}\n.fa-youtube-play:before {\n  content: \"\\f16a\";\n}\n.fa-dropbox:before {\n  content: \"\\f16b\";\n}\n.fa-stack-overflow:before {\n  content: \"\\f16c\";\n}\n.fa-instagram:before {\n  content: \"\\f16d\";\n}\n.fa-flickr:before {\n  content: \"\\f16e\";\n}\n.fa-adn:before {\n  content: \"\\f170\";\n}\n.fa-bitbucket:before {\n  content: \"\\f171\";\n}\n.fa-bitbucket-square:before {\n  content: \"\\f172\";\n}\n.fa-tumblr:before {\n  content: \"\\f173\";\n}\n.fa-tumblr-square:before {\n  content: \"\\f174\";\n}\n.fa-long-arrow-down:before {\n  content: \"\\f175\";\n}\n.fa-long-arrow-up:before {\n  content: \"\\f176\";\n}\n.fa-long-arrow-left:before {\n  content: \"\\f177\";\n}\n.fa-long-arrow-right:before {\n  content: \"\\f178\";\n}\n.fa-apple:before {\n  content: \"\\f179\";\n}\n.fa-windows:before {\n  content: \"\\f17a\";\n}\n.fa-android:before {\n  content: \"\\f17b\";\n}\n.fa-linux:before {\n  content: \"\\f17c\";\n}\n.fa-dribbble:before {\n  content: \"\\f17d\";\n}\n.fa-skype:before {\n  content: \"\\f17e\";\n}\n.fa-foursquare:before {\n  content: \"\\f180\";\n}\n.fa-trello:before {\n  content: \"\\f181\";\n}\n.fa-female:before {\n  content: \"\\f182\";\n}\n.fa-male:before {\n  content: \"\\f183\";\n}\n.fa-gittip:before {\n  content: \"\\f184\";\n}\n.fa-sun-o:before {\n  content: \"\\f185\";\n}\n.fa-moon-o:before {\n  content: \"\\f186\";\n}\n.fa-archive:before {\n  content: \"\\f187\";\n}\n.fa-bug:before {\n  content: \"\\f188\";\n}\n.fa-vk:before {\n  content: \"\\f189\";\n}\n.fa-weibo:before {\n  content: \"\\f18a\";\n}\n.fa-renren:before {\n  content: \"\\f18b\";\n}\n.fa-pagelines:before {\n  content: \"\\f18c\";\n}\n.fa-stack-exchange:before {\n  content: \"\\f18d\";\n}\n.fa-arrow-circle-o-right:before {\n  content: \"\\f18e\";\n}\n.fa-arrow-circle-o-left:before {\n  content: \"\\f190\";\n}\n.fa-toggle-left:before,\n.fa-caret-square-o-left:before {\n  content: \"\\f191\";\n}\n.fa-dot-circle-o:before {\n  content: \"\\f192\";\n}\n.fa-wheelchair:before {\n  content: \"\\f193\";\n}\n.fa-vimeo-square:before {\n  content: \"\\f194\";\n}\n.fa-turkish-lira:before,\n.fa-try:before {\n  content: \"\\f195\";\n}\n.fa-plus-square-o:before {\n  content: \"\\f196\";\n}\n.fa-space-shuttle:before {\n  content: \"\\f197\";\n}\n.fa-slack:before {\n  content: \"\\f198\";\n}\n.fa-envelope-square:before {\n  content: \"\\f199\";\n}\n.fa-wordpress:before {\n  content: \"\\f19a\";\n}\n.fa-openid:before {\n  content: \"\\f19b\";\n}\n.fa-institution:before,\n.fa-bank:before,\n.fa-university:before {\n  content: \"\\f19c\";\n}\n.fa-mortar-board:before,\n.fa-graduation-cap:before {\n  content: \"\\f19d\";\n}\n.fa-yahoo:before {\n  content: \"\\f19e\";\n}\n.fa-google:before {\n  content: \"\\f1a0\";\n}\n.fa-reddit:before {\n  content: \"\\f1a1\";\n}\n.fa-reddit-square:before {\n  content: \"\\f1a2\";\n}\n.fa-stumbleupon-circle:before {\n  content: \"\\f1a3\";\n}\n.fa-stumbleupon:before {\n  content: \"\\f1a4\";\n}\n.fa-delicious:before {\n  content: \"\\f1a5\";\n}\n.fa-digg:before {\n  content: \"\\f1a6\";\n}\n.fa-pied-piper-square:before,\n.fa-pied-piper:before {\n  content: \"\\f1a7\";\n}\n.fa-pied-piper-alt:before {\n  content: \"\\f1a8\";\n}\n.fa-drupal:before {\n  content: \"\\f1a9\";\n}\n.fa-joomla:before {\n  content: \"\\f1aa\";\n}\n.fa-language:before {\n  content: \"\\f1ab\";\n}\n.fa-fax:before {\n  content: \"\\f1ac\";\n}\n.fa-building:before {\n  content: \"\\f1ad\";\n}\n.fa-child:before {\n  content: \"\\f1ae\";\n}\n.fa-paw:before {\n  content: \"\\f1b0\";\n}\n.fa-spoon:before {\n  content: \"\\f1b1\";\n}\n.fa-cube:before {\n  content: \"\\f1b2\";\n}\n.fa-cubes:before {\n  content: \"\\f1b3\";\n}\n.fa-behance:before {\n  content: \"\\f1b4\";\n}\n.fa-behance-square:before {\n  content: \"\\f1b5\";\n}\n.fa-steam:before {\n  content: \"\\f1b6\";\n}\n.fa-steam-square:before {\n  content: \"\\f1b7\";\n}\n.fa-recycle:before {\n  content: \"\\f1b8\";\n}\n.fa-automobile:before,\n.fa-car:before {\n  content: \"\\f1b9\";\n}\n.fa-cab:before,\n.fa-taxi:before {\n  content: \"\\f1ba\";\n}\n.fa-tree:before {\n  content: \"\\f1bb\";\n}\n.fa-spotify:before {\n  content: \"\\f1bc\";\n}\n.fa-deviantart:before {\n  content: \"\\f1bd\";\n}\n.fa-soundcloud:before {\n  content: \"\\f1be\";\n}\n.fa-database:before {\n  content: \"\\f1c0\";\n}\n.fa-file-pdf-o:before {\n  content: \"\\f1c1\";\n}\n.fa-file-word-o:before {\n  content: \"\\f1c2\";\n}\n.fa-file-excel-o:before {\n  content: \"\\f1c3\";\n}\n.fa-file-powerpoint-o:before {\n  content: \"\\f1c4\";\n}\n.fa-file-photo-o:before,\n.fa-file-picture-o:before,\n.fa-file-image-o:before {\n  content: \"\\f1c5\";\n}\n.fa-file-zip-o:before,\n.fa-file-archive-o:before {\n  content: \"\\f1c6\";\n}\n.fa-file-sound-o:before,\n.fa-file-audio-o:before {\n  content: \"\\f1c7\";\n}\n.fa-file-movie-o:before,\n.fa-file-video-o:before {\n  content: \"\\f1c8\";\n}\n.fa-file-code-o:before {\n  content: \"\\f1c9\";\n}\n.fa-vine:before {\n  content: \"\\f1ca\";\n}\n.fa-codepen:before {\n  content: \"\\f1cb\";\n}\n.fa-jsfiddle:before {\n  content: \"\\f1cc\";\n}\n.fa-life-bouy:before,\n.fa-life-saver:before,\n.fa-support:before,\n.fa-life-ring:before {\n  content: \"\\f1cd\";\n}\n.fa-circle-o-notch:before {\n  content: \"\\f1ce\";\n}\n.fa-ra:before,\n.fa-rebel:before {\n  content: \"\\f1d0\";\n}\n.fa-ge:before,\n.fa-empire:before {\n  content: \"\\f1d1\";\n}\n.fa-git-square:before {\n  content: \"\\f1d2\";\n}\n.fa-git:before {\n  content: \"\\f1d3\";\n}\n.fa-hacker-news:before {\n  content: \"\\f1d4\";\n}\n.fa-tencent-weibo:before {\n  content: \"\\f1d5\";\n}\n.fa-qq:before {\n  content: \"\\f1d6\";\n}\n.fa-wechat:before,\n.fa-weixin:before {\n  content: \"\\f1d7\";\n}\n.fa-send:before,\n.fa-paper-plane:before {\n  content: \"\\f1d8\";\n}\n.fa-send-o:before,\n.fa-paper-plane-o:before {\n  content: \"\\f1d9\";\n}\n.fa-history:before {\n  content: \"\\f1da\";\n}\n.fa-circle-thin:before {\n  content: \"\\f1db\";\n}\n.fa-header:before {\n  content: \"\\f1dc\";\n}\n.fa-paragraph:before {\n  content: \"\\f1dd\";\n}\n.fa-sliders:before {\n  content: \"\\f1de\";\n}\n.fa-share-alt:before {\n  content: \"\\f1e0\";\n}\n.fa-share-alt-square:before {\n  content: \"\\f1e1\";\n}\n.fa-bomb:before {\n  content: \"\\f1e2\";\n}\n"
  },
  {
    "path": "doc/css/images.css",
    "content": "/* Copyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0 */\n\nimg.full-width {\n    width: 100%\n}\n\nimg.screenshot {\n    max-width: calc(100% - 2em);\n    border: 3px;\n    border-style: solid;\n    margin-left: 2em;\n    margin-bottom: 2em;\n}\n\nimg.side {\n    float: left;\n    width: 50%;\n}\n"
  },
  {
    "path": "doc/css/layout.css",
    "content": "/* Copyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0 */\n\nhtml {\n    height:100%;\n}\nbody {\n    padding-top: 61px;\n    height: 90%; /* If calc() is not supported */\n    height: calc(100% - 46px); /* Sets the body full height minus the padding for the menu bar */\n}\n@media (max-width: 1050px) {\n    body {\n\tpadding-top: 121px;\n    }\n    div.frontpagehero {\n\tmargin-left: -20px;\n\tmargin-right: -20px;\n\tpadding-left: 20px;\n    }\n}\n.sidebar-nav {\n    padding: 9px 0;\n}\n.section-block {\n    background: #eeeeee;\n    padding: 1em;\n    -webkit-border-radius: 12px;\n    -moz-border-radius: 12px;\n    border-radius: 12px;\n    margin: 0 2em;\n}\n.row-fluid :first-child .section-block {\n    margin-left: 0;\n}\n.row-fluid :last-child .section-block {\n    margin-right: 0;\n}\n.rarr {\n    font-size: 1.5em;\n}\n.darr {\n    font-size: 4em;\n    text-align: center;\n    margin-bottom: 1em;\n}\n:target {\n    padding-top: 61px;\n    margin-top: -61px;\n}\n\n#annotate-notify { position: fixed; right: 40px; top: 3px;  }\n\nfigure {\n    margin-bottom: 20px;\n}\n"
  },
  {
    "path": "doc/css/nav-list.css",
    "content": "/* Copyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0 */\n\n/* NAV LIST\n   -------- */\n\n.nav-list {\n  padding-left: 15px;\n  padding-right: 15px;\n  margin-bottom: 0;\n}\n.nav-list > li > a,\n.nav-list .nav-header {\n  margin-left:  -15px;\n  margin-right: -15px;\n  text-shadow: 0 1px 0 rgba(255,255,255,.5);\n}\n.nav-list > li > a {\n  padding: 3px 15px;\n}\n.nav-list > .active > a,\n.nav-list > .active > a:hover,\n.nav-list > .active > a:focus {\n  color: white;\n  text-shadow: 0 -1px 0 rgba(0,0,0,.2);\n  background-color: rgb(66, 139, 202);\n}\n\n.spaced-out li {\n   padding-bottom: 1em;\n}\n\n.inside-list ul {\n    list-style-position: inside;\n    padding-left: 0;\n}\n"
  },
  {
    "path": "doc/development/CodingStandards.md",
    "content": "[comment]: # (Copyright © The Arvados Authors. All rights reserved.)\n[comment]: # ()\n[comment]: # (SPDX-License-Identifier: CC-BY-SA-3.0)\n\n# Coding Standards\n\nThe rules are always up for debate. However, when debate is needed, it should happen outside the source tree. In other words, if the rules are wrong, first debate the rules at sprint retrospective, then fix the rules, then follow the new rules.\n\n## Git commit messages\n\n- Prefix the summary line with the issue number this addresses.\n- Describe the delta between the old and new tree. If possible, describe the delta in **behavior** rather than the source code itself.\n- Good: “1234: Support use of spaces in filenames.”\n- Good: “1234: Fix crash when user_id is nil.”\n- Less good: “Add some controller methods.” (What do they do?)\n- Less good: “More progress on UI branch.” (What is different?)\n- Less good: “Incorporate Tom’s suggestions.” (Who cares whose suggestions — what changed?)\n\nIf further background or explanation is needed, separate it from the summary with a blank line.\n\n- Example: “Users found it confusing that the boxes had different colors even though they represented the same kinds of things.”\n\n**Every commit** (including merge commits) must have a DCO sign-off. See `CONTRIBUTING.md` for the full terms of what this means.\n\n- Example: `Arvados-DCO-1.1-Signed-off-by: Alex Doe `\n\nFull examples:\n\n    commit 9c6540b9d42adc4a397a28be1ac23f357ba14ab5\n    Author: Tom Clegg \n    Date:   Mon Aug 7 09:58:04 2017 -0400\n\n        12027: Recognize a new \"node failed\" error message.\n\n        \"srun: error: Cannot communicate with node 0.  Aborting job.\"\n\n        Arvados-DCO-1.1-Signed-off-by: Tom Clegg \n\n    commit 0b4800608e6394d66deec9cecea610c5fbbd75ad\n    Merge: 6f2ce94 3a356c4\n    Author: Tom Clegg \n    Date:   Thu Aug 17 13:16:36 2017 -0400\n\n        Merge branch '12081-crunch-job-retry'\n\n        refs #12080\n        refs #12081\n        refs #12108\n\n        Arvados-DCO-1.1-Signed-off-by: Tom Clegg \n\n## Source code formatting\n\nThese are general baseline rules except when a language-specific guide specifies otherwise.\n\nNo TAB characters in source files [except Go](https://golang.org/cmd/gofmt/).\n\n- For Emacs, add `(setq-default indent-tabs-mode nil)` to `~/.emacs`.\n- For Vim, add `:set expandtab` to `~/.vimrc`.\n\nAvoid long (\\>100 column) lines.\n\nNo whitespace at the end of lines unless technically required (like Markdown line breaks).\n\n## What to include\n\nNo commented-out blocks of code that have been replaced or obsoleted.\n\n- It is in the git history if we want it back.\n- If its absence would confuse someone reading the new code (despite never having read the old code), explain its absence in an English comment. If the old code is really still needed to support the English explanation, then go ahead — now we know why it’s there.\n\nNo commented-out debug statements.\n\n- If the debug statements are likely to be needed in the future, use a logging facility that can be enabled at run time. `logger.debug \"foo\"`\n\n## Style mismatch\n\nAdopt indentation style of surrounding lines or (when starting a new file) the nearest existing source code in this tree/language.\n\nIf you fix up existing indentation/formatting, do that in a separate commit.\n\n- If you bundle formatting changes with functional changes, it makes functional changes hard to find in the diff.\n\n## Go\n\nFollow gofmt, golint, etc., and \n\nUse `%w` when wrapping an error with fmt.Errorf(), so errors.As() can access the wrapped error.\n\n```go\nif err != nil {\n        return fmt.Errorf(\"could not swap widgets: %w\", err)\n}\n```\n\nUse `(logrus.FieldLogger)WithError()` (instead of `Logf(\"blah: %s\", err)`) when logging an error.\n\n```go\nif err != nil {\n        logger.WithError(err).Warn(\"error swapping widgets\")\n}\n```\n\n## Ruby\n\nFollow \n\n## Python\n\n### Python code\n\nFor code, follow [PEP 8](https://peps.python.org/pep-0008/).\n\nWhen you add functions, methods, or attributes that SDK users should not use, their name should start with a leading underscore. This is a common convention to signal that an interface is not intended to be public. Anything named this way will be excluded from our SDK web documentation by default.\n\nYou’re encouraged to add type annotations to functions and methods. As of May 2024 these are purely for documentation: we are not type checking any of our Python. Note that your annotations must be understood by the oldest version of Python we currently support (3.10).\n\n### Python docstrings\n\nPublic classes, methods, and functions should all have docstrings. The content of the docstring should follow [PEP 257](https://peps.python.org/pep-0257/).\n\nFormat docstrings with Markdown and follow these style rules:\n\n* Document function argument lists after the high-level description following this format for each argument:\n\n        * name: type --- Description\n\n   Use exactly three minus-hyphens to get an em dash in the web rendering. Provide a helpful type hint whenever practical. The type hint should be written in “modern” style with builtin subscripting and type union syntax, like `list[str | bytes]`.\n\n   Use fully qualified names for custom types. This way pdoc hyperlinks them.\n\n* When something is deprecated, write a `.. WARNING:: Deprecated` admonition immediately after the first line. Its text should explain that the thing is deprecated, and suggest what to use instead. For example:\n\n        def add(a, b):\n            \"\"\"Add two things.\n\n            .. WARNING:: Deprecated\n               This function is deprecated. Use the `+` operator instead.\n\n            …\n            \"\"\"\n\n   You can similarly note private methods with `.. ATTENTION:: Internal`.\n\n* Mark up all identifiers outside the type hint with backticks. When the identifier exists in the current module, use the short name. Otherwise, use the fully-qualified name. Our web documentation will automatically link these identifiers to their corresponding documentation.\n\n* Mark up links using Markdown’s footnote style. For example:\n\n        \"\"\"Python docstring following [PEP 257][pep257].\n\n\n        \"\"\"\n\n   This looks best in plaintext. A descriptive identifier is nice if you can keep it short, but if that’s challenging, plain ordinals are fine too.\n\n* Mark up headers (e.g., in a module docstring) using underline style. For example:\n\n        \"\"\"Generic utility module\n\n        Filesystem functions\n        --------------------\n\n        …\n\n        Regular expressions\n        -------------------\n\n        …\n        \"\"\"\n\n   This looks best in plaintext.\n\nThe goal of these style rules is to provide a readable, consistent appearance whether people read the documentation in plain text (e.g., using `pydoc`) or their browser (as rendered by `pdoc`).\n\n## JavaScript\n\nWe already have 4-space indents everywhere, so do that.\n\nOther than that, follow the [Airbnb Javascript coding style](https://github.com/airbnb/javascript) guide unless otherwise stated.\n\n## Workbench Design Guidelines\n\n### Font Sizes\n\n- Minimum 12pt (16px)\n- Minimum 9 pt (12px) for things like by copyright, footer\n\nThis should be able to be-resized up to 200% without loss of content or functionality.\n\n### Color\n\n- Text and images of text have a color contrast ratio of at least 4.5:1 You can use [this contrast tool](https://snook.ca/technical/colour_contrast/colour.html#fg=1F7EA1,bg=FFFFFF) to check.\n- Non-text icon, controls, etc - 3:1 must have a color contrast ratio of 3:1.\n- Avoid hard-coding colors. Use theme colors. If a new color is needed, add it to the theme.\n- Used defined grays when possible using RGB value and changing the a value to indicate different meanings (i.e. Active icons have an opacity of 87, Inactive icons have an opacity of 60, Disabled icons have an opacity of 38%)\n\n### Icons\n\n#### General\n\n- Interaction target size of at least 44 x 44 pixels\n- Label should be on right, icon on left for maximum readability\n- Use minimum 3:1 color contrast (see Color above)\n- User appropriate concise alt text for people using screen readers\n\n#### Menu/Navigation\n\n- No navigation should only supported via breadcrumbs\n- If less than 5 menu options, consider visible navigation options\n- If more than 5 menu options, consider a combination navigation where some options are visible and some are hidden\n- Use the following menu consistently:\n  - Hamburger (three bars stacked vertically): Used to indicate navigation bar/menu that toggles between being collapsed behind the button or displayed on the screen, often used for global/site-wide/whole application navigation\n  - Döner (three bars that narrow vertically): Indicates a group filtering menu\n  - Bento (3×3 grid of squares): Indicates a menu presenting a grid of options (not currently applicable to WB)\n  - Kebab (three dots stacked vertically): Indicates a smaller inline-menu or an overflow/combination menu\n  - Meatballs (three dots stacked horizontally): Used to indicate a smaller inline-menu. Often used to indicate action on a related item (i.e. item next to the meatball), good for repeated use in tables, or horizontal elements\n- If component is an accordion window, use caret(‸)\n\nPreferred Icon Repositories:\n\n- \n- \n- \n\n### Buttons\n\n- Label button with action for usability/to reduce ambiguity (avoid generic button labels for actions)\n- Buttons vs Links\n  - Buttons should cause change in current context\n  - Links should navigate to a different content or a new resource (e.g. different page)\n- If text on button - color contract should be 4.5 :1 between button and text\n- Button color and background color contrast should be 3:1\n\n### Arvados Specific Components\n\nUse chips for displaying tokenized values/arrays\n\n### Loading Indicators\n\n#### Page Navigation\n\n- Navigation between pages should be indicated using `progressIndicatorActions.START_WORKING` and `progressIndicatorActions.STOP_WORKING` to show the global top-of-page pulser\n- Only the initial load or refresh of the full page (eg. triggered by the upper right refresh button) should use this indicator. Partial refreshes should use a more local indicator.\n  - Refreshes of only one section of a page should only show its own loading indicator in that section\n- Full page refreshes where the location is unchanged should avoid using the initial full-page spinner in favor of the top-of-page spinner, with updated values substituting in the UI when loaded\n\n#### User Actions\n\n- Form submissions or user actions should be indicated by both the `progressIndicatorActions.START_WORKING` and by enabling the spinner on the submit button of the form (if the action takes place through a form AND if the form stays open for the duration of the action in order to show errors). If the form closes immediately then the page spinner is the only indicator.\n- Toasts should not be used to notify the user of an in-progress action but only completion / error\n\n#### Lazy-loaded fields\n\n- Fields that load or update (eg. with extra info) after the main view should wait 3-5 seconds before showing a spinner/pulser icon while loading - if the request for extra data fails, a placeholder icon should show with a hint (text or tooltip) indicating that the data failed to load.\n  - The delayed indicator should be implemented as a reusable component (tbd)\n- Suggested loading indicator for inline fields: https://mhnpd.github.io/react-loader-spinner/docs/components/three-dots)\n\n### References\n\n[WCAG2.1](https://www.w3.org/WAI/WCAG21/Understanding/)\n\n[Sarah’s talk for references](https://docs.google.com/presentation/d/1HNrhvK7zVZ7jgH3ELbX7KB97SdXCZXrvov_I4Oe1l2c/edit?usp=sharing)\n"
  },
  {
    "path": "doc/development/DevelopmentProcess.md",
    "content": "[comment]: # (Copyright © The Arvados Authors. All rights reserved.)\n[comment]: # ()\n[comment]: # (SPDX-License-Identifier: CC-BY-SA-3.0)\n\n# Development Process\n\nThis document is intended for core engineers who work on the `main` branch of Arvados.\n\n## Two Remotes\n\nThis document assumes you have two remotes, where `origin` refers to `git.arvados.org` and `github` refers to `github.com`:\n\n```sh\n$ git remote -v\ngithub\tgit@github.com:arvados/arvados.git (fetch)\ngithub\tgit@github.com:arvados/arvados.git (push)\norigin\tgit@git.arvados.org:arvados.git (fetch)\norigin\tgit@git.arvados.org:arvados.git (push)\n```\n\n## Fetch GitHub Pull Requests as Branches\n\nGitHub tracks pull requests under `refs/pull/`. You can configure Git to map these to your local repository when you fetch GitHub:\n\n```sh\n$ git config set --append remote.github.fetch '+refs/pull/*:refs/remotes/ghpr/*'\n```\n\nNow after you fetch, you can refer to pull request #123 as `ghpr/123/head`.\n\nIf you prefer to fetch pull requests individually, the command to do that is:\n\n```sh\n$ git fetch github \"pull/PRNUM/head:BRANCHNAME\"\n```\n\n## Review a Pull Request\n\nReviewing a pull request is about verifying that the branch follows all our [coding standards](CodingStandards.md). You should be able to verify that the ready-to-merge checklist is complete and accurate: the branch does what it says, tests pass, it follows our style, etc.\n\nIf you notice scale issues, bugs, missing documentation, etc., you can bring that up as part of the review and it should be addressed. However, the *point* of review is *not* to try to find problems. The *point* is to verify that the branch solves a problem and the code is maintainable.\n\n## Merge a Pull Request\n\nWhen a branch passes review, it should be merged to `main`. Core engineers can (and normally do) merge their own branches. Contributions from others need to be merged by a core engineer. Either way, the process is:\n\n```sh\n$ git switch main\n$ git pull --ff-only\n$ git merge --no-ff BRANCHREF\n# Make sure the commit message includes an issue ref and your DCO signoff.\n$ git push origin main\n```\n"
  },
  {
    "path": "doc/development/DistroVersions.md",
    "content": "[comment]: # (Copyright © The Arvados Authors. All rights reserved.)\n[comment]: # ()\n[comment]: # (SPDX-License-Identifier: CC-BY-SA-3.0)\n\n# Distribution dependency versions\n\nFor runtime dependencies that we aim to get from the distribution, this page lists the versions included with each distribution we support.\n\nFor RHEL releases, this table shows the *latest* version available from AppStreams for **both** the earliest point release we support and the latest point release at the time of the last update.\n\n| Distribution              | Release Date  | PostgreSQL | Python  | Ruby  |\n|---------------------------|---------------|------------|---------|-------|\n| Ubuntu 22.04 “jammy”      | April 2022    | 14.17      | 3.10.12 | 3.0.2 |\n| RHEL 8.8 with AppStreams  | May 2023      | 15.6       | 3.11.5  | 3.1.4 |\n| RHEL 9.2 with AppStreams  | May 2023      | 15.6       | 3.11.5  | 3.1.4 |\n| Debian 12 “bookworm”      | June 2023     | 15.13      | 3.11.2  | 3.1.2 |\n| RHEL 8.9 with AppStreams  | November 2023 | 15.6       | 3.11.5  | 3.1.4 |\n| Ubuntu 24.04 “noble”      | April 2024    | 16.2       | 3.12.3  | 3.2.3 |\n| RHEL 9.5 with AppStreams  | November 2024 | 16.8       | 3.12.5  | 3.3.8 |\n| RHEL 10.0 with AppStreams | May 2025      | 16.8       | 3.12.9  | 3.3.8 |\n| Debian 13 “trixie”        | August 2025   | 17.5       | 3.13.3  | 3.3.8 |\n\n## For Arvados 3.1.x\n\n| Distribution             | Release Date | PostgreSQL | Python | Ruby  |\n|--------------------------|--------------|------------|--------|-------|\n| Ubuntu 20.04 “focal”     | April 2020   | 12.22      | 3.8.10 | 2.7.0 |\n| RHEL 8.4 with AppStreams | June 2021    | 13.3       | 3.9.2  | 2.7.4 |\n| Debian 11 “bullseye”     | August 2021  | 13.16      | 3.9.2  | 2.7.4 |\n"
  },
  {
    "path": "doc/development/Prerequisites.md",
    "content": "[comment]: # (Copyright © The Arvados Authors. All rights reserved.)\n[comment]: # ()\n[comment]: # (SPDX-License-Identifier: CC-BY-SA-3.0)\n\n# Hacking prerequisites\n\nThis page describes how to install all the software necessary to develop Arvados and run tests.\n\n## Host options\n\nYou must have a system running a supported distribution. That system can be installed directly on hardware; running on a cloud instance; or in a virtual machine.\n\n### Supported distributions\n\nAs of March 2026/Arvados 3.2, these instructions and the entire test suite are known to work on Debian 12 \"bookworm\" and Debian 13 “trixie.”\n\nYou may try to run these instructions and tests on Ubuntu 22.04 “jammy”/24.04 “noble,” but they have not been tested and you may find some bugs throughout.\n\nThese instructions are not suitable for any Red Hat-based distribution. Our Ansible playbook will refuse to run on them.\n\n### Base configuration\n\nOn your development system, you should have a user account with full permission to use sudo.\n\nYou can run the Ansible playbook to install your development system on a different system. To do this, you must have permission to SSH into your user account from the system running Ansible (the “control node”) to the development system you’re installing (the “target node”).\n\n### Virtual machine requirements\n\nIf you run your development system in a virtual machine, it needs some permissions. Many environments will allow these operations by default, but they could be limited by your virtual machine setup.\n\n- It must be able to create and manage FUSE mounts (`/dev/fuse`)\n- It must be able to create and run Docker containers\n- It must be able to create and run Singularity containers—this requires creating and managing block loopback devices (`/dev/block-loop`)\n- It must have the `fs.inotify.max_user_watches` sysctl set to at least 524288. Our Ansible playbook will try to set this on the managed host, but if it is unable to do so, you may need to set it on the parent host instead.\n\n## Install development environment with Ansible\n\n### Clone Arvados source\n\nYou will need the Arvados source code to follow this process.\n\n```sh\n$ git clone https://github.com/arvados/arvados.git\n```\n\nIf you want to switch to a specific branch or revision like `3.2-release`, do that here.\n\n### Install Ansible\n\nInstall Ansible following the instructions in `arvados/tools/ansible/README.md`. This ensures you get the right versions of everything.\n\n### Write an Arvados database configuration\n\nMake a copy of the default test configuration:\n\n```sh\n$ cp arvados/tools/ansible/files/default-test-config.yml ~/zzzzz-config.yml\n```\n\nYou can copy the file to a different location if you like. This page will use `~/zzzzz-config.yml` as the placeholder path throughout.\n\nEdit this file with the database configuration you’d like to use. The cluster ID **must** be `zzzzz`. You can change the `user`, `password`, and `dbname` settings freely. Our Ansible playbook will configure PostgreSQL so your settings here work.\n\nThe playbook will always install the `postgresql` server package. It will **not** change any PostgreSQL configuration except to add `pg_hba.conf` entries for this user. You should only change `host` and `port` if you need to use a PostgreSQL server that is already installed and running somewhere else.\n\n### Write an Ansible inventory\n\nAn inventory file tells Ansible what host(s) to manage, how to connect to them, and what settings they use. Write an inventory file to `~/zzzzz-inventory.yml` like this:\n\n```yaml\narvados_test_all:\n  # This is the list of host(s) where we're installing the test environment.\n  # This example installs on the same system running Ansible.\n  # If you want to manage remote hosts, you can write your own host list:\n  # \n  hosts:\n    localhost:\n      ansible_connection: local\n  vars:\n    # The path to the Arvados cluster configuration you wrote in the previous section.\n    arvados_config_file: \"{{ lookup('env', 'HOME') }}/zzzzz-config.yml\"\n\n    # The primary user doing Arvados development and tests.\n    # This user will be added to the `docker` group.\n    # It defaults to the name of the user running `ansible-playbook`.\n    # If you want to configure a different user, set that here:\n    #arvados_dev_user: USERNAME\n\n    # By default, the playbook installs old versions of Python and Ruby from source.\n    # This helps you make sure you don't accidentally use too-new features during\n    # development. If you're sure you don't need that—for example, you specifically\n    # want to test a distribution's packaged version—set this flag:\n    #arvados_dev_from_pkgs: true\n```\n\n### Run the playbook\n\nThe basic command to run the playbook is:\n\n```sh\n$ cd arvados/tools/ansible\n$ ansible-playbook -K -i ~/zzzzz-inventory.yml install-dev-tools.yml\n```\n\nWhen you are prompted for the `BECOME password:`, enter the password for your user account on the development host that lets you run `sudo` commands.\n\n`ansible-playbook` has many options to control how it runs that you can add if you like. Refer to [the `ansible-playbook` documentation](https://docs.ansible.com/ansible/latest/cli/ansible-playbook.html) for more information.\n\n## Run Arvados tests\n\nAfter the playbook runs successfully, you should be able to run the Arvados tests from a source checkout on your development host. This document will walk you through setting up and running a single test suite to verify your setup. `cd` to your Arvados checkout and run:\n\n```sh\n$ mkdir -p ~/.cache/arvados-test\n$ build/run-tests.sh --temp ~/.cache/arvados-test --interactive\n```\n\nThis will install baseline prerequisites, then list commands and test targets, then prompt you with:\n\n    What next? install deps\n\nAccept that command. It will install the rest of the dependencies that are necessary for running a test cluster, then report:\n\n    All test suites passed.\n\nAt this stage, this message simply means that the \"install deps\" command has succeeded.\n\nNow we can run a test suite. The controller tests are a good first example, because they interact with a test cluster but not much else. At the `What next?` prompt, enter `test lib/controller`, and you'll see the test cluster start:\n\n    What next? test lib/controller\n    Starting API, controller, keepproxy, keep-web, ws, and nginx ssl proxy...\n\nYou'll see logs from individual services, then, hopefully, the controller tests starting and passing:\n\n    ======= test lib/controller\n    ok  \tgit.arvados.org/arvados.git/lib/controller\t64.679s\tcoverage: 82.0% of statements\n    ======= test lib/controller -- 68s\n    Pass: lib/controller tests (68s)\n    All test suites passed.\n\nRefer to [Running tests](RunningTests.md) for details about running specific test suites, test selection, and other features.\n\n## Troubleshooting\n\nIf the playbook succeeds but you can't get tests running, there might be a disconnect between your shell configuration and what the system expects. This section documents some places you can look.\n\n### Dependencies in `$PATH`\n\nThe playbook will install symlinks for Go, Node, Python, Ruby, Singularity, and Yarn under `/usr/local/bin`. The actual tools are installed under `/opt`. When you run Arvados tests or other development tools, you must ensure `/usr/local/bin` appears in your `$PATH` before any directories with other versions like `/usr/bin`.\n\n### Arvados `$CONFIGSRC`\n\nThe playbook writes the Arvados test cluster's database configuration at `~/.config/arvados/config.yml`, and sets up a hook `/etc/profile.d/arvados-test.sh` to set your `CONFIGSRC` environment variable to that file's base directory. If most tests fail with a database connection error, check that this variable is set:\n\n```sh\n$ echo \"${CONFIGSRC:-UNSET}\"\n/home/you/.config/arvados\n```\n\nIf that reports `UNSET`, first check if you're using a stale shell session started before the Ansible playbook run. You may need to log out of that session and start a new one.\n\nIf that doesn't work, you may add a line to set `CONFIGSRC=\"$HOME/.config/arvados\"` to your shell configuration, or set it manually when you run `run-tests.sh`:\n\n```sh\n$ CONFIGSRC=\"$HOME/.config/arvados\" build/run-tests.sh ...\n```\n"
  },
  {
    "path": "doc/development/RunningTests.md",
    "content": "[comment]: # (Copyright © The Arvados Authors. All rights reserved.)\n[comment]: # ()\n[comment]: # (SPDX-License-Identifier: CC-BY-SA-3.0)\n\n# Running Tests\n\nArvados includes a script at `build/run-tests.sh` which tests (nearly) all of the components in the source tree. This is the script that [Arvados CI tests](https://ci.arvados.org) use, so running it locally is the most consistent entry point to all Arvados tests.\n\nThis document assumes you have [installed a development environment](Prerequisites.md) following that guide.\n\n## Running interactively\n\nMost developers want to run tests with `--temp` and `--interactive`:\n\n```sh\n$ mkdir -p ~/.cache/arvados-test\n$ build/run-tests.sh --temp ~/.cache/arvados-test --interactive\n```\n\nThis will display help with a list of commands and test targets. When you run with a fresh temp directory, the tool will probably prompt you to `install deps`. You should do this to install dependencies to the temp directory.\n\n### Dealing with state\n\nBefore you change `run-tests.sh` itself—including pulling changes from other developers—you should end any interactive test sessions.\n\nIf you make changes to a low-level library or SDK and want to see how it affects dependent tests, `install` your changed component, then `test` the dependents.\n\nIf you make changes to a cluster component and want to see how they affect tests, `reset` the test cluster, then `test` the components you're interested in.\n\nIf you want to clean your `--temp` directory—because you pulled a bad dependency or just want to recover some disk space—it is safe to end any interactive sessions, remove it, then `mkdir` it again.\n\n### Running individual test cases\n\n#### Golang\n\nMost Go packages use gocheck. Use gocheck command line args like `-check.f` to select tests and `-check.v` to show more output.\n\n    What next? test lib/controller/router -check.f=RouterSuite -check.v\n    ======= test lib/controller/router\n    PASS: request_test.go:135: RouterSuite.TestAttrsInBody\t0.000s\n    PASS: request_test.go:164: RouterSuite.TestBoolParam\t0.000s\n    PASS: router_test.go:55: RouterSuite.TestOptions\t0.002s\n    PASS: request_test.go:209: RouterSuite.TestStringOrArrayParam\t0.000s\n    OK: 4 passed\n    PASS\n    ok  \tgit.arvados.org/arvados.git/lib/controller/router\t0.012s\n    ======= test lib/controller/router -- 1s\n\n#### Python\n\nTests for Python components run under pytest. If what you really want to do is to focus on failing or newly-added tests, consider passing the appropriate switches to do that:\n\n      -x, --exitfirst       Exit instantly on first error or failed test\n      --lf, --last-failed   Rerun only the tests that failed at the last run (or\n                            all if none failed)\n      --ff, --failed-first  Run all tests, but run the last failures first. This\n                            may re-order tests and thus lead to repeated fixture\n                            setup/teardown.\n      --nf, --new-first     Run tests from new files first, then the rest of the\n                            tests sorted by file mtime\n\nIf you want to manually select tests:\n\n      FILENAME              Run tests from FILENAME, relative to the source root\n      FILENAME::CLASSNAME   Run tests from CLASSNAME\n      FILENAME::FUNCNAME, FILENAME::CLASSNAME::FUNCNAME\n                            Run only the named test function\n      -k EXPRESSION         Only run tests which match the given substring\n                            expression. An expression is a Python evaluable\n                            expression where all names are substring-matched\n                            against test names and their parent classes.\n                            Example: -k 'test_method or test_other' matches all\n                            test functions and classes whose name contains\n                            'test_method' or 'test_other', while -k 'not\n                            test_method' matches those that don't contain\n                            'test_method' in their names. -k 'not test_method\n                            and not test_other' will eliminate the matches.\n                            Additionally keywords are matched to classes and\n                            functions containing extra names in their\n                            'extra_keyword_matches' set, as well as functions\n                            which have names assigned directly to them. The\n                            matching is case-insensitive.\n      -m MARKEXPR           Only run tests matching given mark expression. For\n                            example: -m 'mark1 and not mark2'.\n\nFor even more options, refer to the [pytest command line reference](https://docs.pytest.org/en/stable/reference/reference.html#command-line-flags).\n\nExample:\n\n    What next? test sdk/python --disable-warnings --tb=no --no-showlocals tests/test_keep_client.py::KeepDiskCacheTestCase\n    ======= test sdk/python\n    […pip output…]\n    ========================================================== test session starts ==========================================================\n    platform linux -- Python 3.10.19, pytest-9.0.2, pluggy-1.6.0\n    rootdir: /home/brett/Curii/arvados/sdk/python\n    configfile: pytest.ini\n    collected 9 items\n\n    tests/test_keep_client.py F........                                                                                               [100%]\n\n    ======================================================== short test summary info ========================================================\n    FAILED tests/test_keep_client.py::KeepDiskCacheTestCase::test_disk_cache_cap - AssertionError: True is not false\n    ====================================================== 1 failed, 8 passed in 0.16s ======================================================\n    ======= sdk/python tests -- FAILED\n    ======= test sdk/python -- 2s\n    Failures (1):\n    Fail: sdk/python tests (2s)\n    What next? test sdk/python --disable-warnings --tb=no --no-showlocals --lf\n    ======= test sdk/python\n    […pip output…]\n    ========================================================== test session starts ==========================================================\n    platform linux -- Python 3.10.19, pytest-9.0.2, pluggy-1.6.0\n    rootdir: /home/brett/Curii/arvados/sdk/python\n    configfile: pytest.ini\n    testpaths: tests\n    collected 964 items / 963 deselected / 1 selected\n    run-last-failure: rerun previous 1 failure\n\n    tests/test_keep_client.py F                                                                                                       [100%]\n\n    ======================================================== short test summary info ========================================================\n    FAILED tests/test_keep_client.py::KeepDiskCacheTestCase::test_disk_cache_cap - AssertionError: True is not false\n    ============================================= 1 failed, 963 deselected, 1 warning in 0.43s ==============================================\n    ======= sdk/python tests -- FAILED\n    ======= test sdk/python -- 2s\n    Failures (1):\n    Fail: sdk/python tests (2s)\n\n#### RailsAPI\n\nRails parses `TESTOPTS` and passes them to the test runner:\n\n    What next? test services/api TESTOPTS=--name=/.*signed.locators.*/\n    [...]\n    # Running:\n\n    ....\n\n    Finished in 1.080084s, 3.7034 runs/s, 461.0751 assertions/s.\n\n##### Controlling Rails test order\n\nRails tests start off with a line like this\n\n    Run options: -v -d --seed 57089\n\nThe seed value determines the order tests are run. To reproduce reproduce an order-dependent test failure, specify the same seed as a previous failed run:\n\n    What next? test services/api TESTOPTS=\"-v -d --seed 57089\"\n\n## Environment variables\n\nThe following variables affect test setup and execution:\n\nVariable    | Value\n------------|-----------------------------------------------------------------\n`CONFIGSRC` | A directory with an Arvados cluster `config.yml`. Tests will read `Clusters.zzzzz.PostgreSQL.Connection` from that file to determine how to connect to the test database. If not set, the tests will use default connection settings.\n`WORKSPACE` | A directory with an Arvados Git checkout. Defaults to what Git reports for `run-tests.sh` itself.\n\n`run-tests.sh` cleans the `ARVADOS_[…]` variables from the environment to help ensure consistent test execution. Sometimes you may want to set these variables nonetheless, but then you must pass them as arguments to `run-tests.sh` rather than export them to the environment directly.\n\nVariable               | Value\n-----------------------|------------------------------------------------------\n`ARVADOS_DEBUG`        | If 1, lots of components will log more information.\n`ARVADOS_TEST_PRIVESC` | A literal string. If `sudo`, various tests that need to perform privileged operations with run with `sudo` to get them. Otherwise, those tests are skipped.\n\n## Scripting run-tests\n\nIf you run `run-tests.sh` without `--interactive`, by default it runs all the tests and reports their results. This is how CI runs. Run the script with `--help` to see the options you can use to control this behavior. Common options include:\n\nOption           | Behavior\n-----------------|------------------------------------------------------------\n`--only`         | Run a single set of tests\n`--skip`         | Skip a set of tests during a full run\n`NAME_test=ARGS` | Pass arguments to a set of tests\n\n## Running Workbench tests in Docker\n\nIf you do not have a full development environment, Workbench tests can be run in Docker. The `services/workbench2` subfolder includes Makefile targets that preinstall the necessary dependencies in a Docker container using Ansible.\n\nWith Docker and Ansible installed (see `arvados/tools/ansible/README.md`), run this command from within the `arvados/services/workbench2` directory:\n\n    make workbench-docker-image\n\nYou can verify the docker image was built by looking for `arvados/workbench` in `docker image ls`\n\nThen, start the interactive tests with this command:\n\n    make interactive-tests-in-docker\n\nNon-interactive (headless) tests can be run with the targets:\n\n    # Both e2e & component tests\n    make tests-in-docker\n\n    # Integration (e2e) only\n    make integration-tests-in-docker\n\n    # Unit (component) only\n    make unit-tests-in-docker\n\n### Troubleshooting\n\n#### Missing X server or `$DISPLAY`\n\nRun:\n\n    xhost +local:root\n\n#### No version of Cypress is installed / other error starting Cypress\n\nRecreate the home volume which re-installs Cypress and other persisted dependencies by running:\n\n    make clean-docker-volume\n    make workbench-docker-volume\n"
  },
  {
    "path": "doc/development/UpdatingDependencies.md",
    "content": "[comment]: # (Copyright © The Arvados Authors. All rights reserved.)\n[comment]: # ()\n[comment]: # (SPDX-License-Identifier: CC-BY-SA-3.0)\n\n# Updating dependencies\n\n## Go\n\n(see also: [real documentation](https://go.dev/doc/modules/managing-dependencies))\n\nUpdate a single dependency:\n\n~/arvados\\$ go get github.com/docker/docker@latest\n\nUpdate all dependencies:\n\n~/arvados\\$ go get -u -t ./…\n\nThen sync:\n\n~/arvados\\$ go mod tidy\n\nThis is a good time to review “replace” directives in source:go.mod and find better solutions to issues that are currently handled by pinning modules to old versions or unmaintained forks.\n"
  },
  {
    "path": "doc/development/git.conf",
    "content": "# Suggested Git configuration for Arvados\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n\n# Arvados standards forbid trailing whitespace.\n# Configure Git to highlight it.\n[color \"diff\"]\nwhitespace = red reverse\n\n[core]\nwhitespace = trailing-space\n\n[merge]\n# Merges to main should never be fast-forward.\n# Easiest to turn it off by default:\nff = false\n\n[user]\n# The Arvados DCO sign-off requires your real name and email.\n# Refer to CONTRIBUTING.md for full details.\n#name = Your Name\n#email = yourmail@example.local\n\n[trailer.arvados]\nkey = Arvados-DCO-1.1-Signed-off-by\nifexists = doNothing\n# If you uncomment cmd, the prepare-commit-msg hook will prepare commit\n# messages with your DCO sign-off, which attests you have permission to\n# contribute the code. Refer to CONTRIBUTING.md for full terms.\n#cmd = echo \\\"$(git config user.name) <$(git config user.email)>\\\"\n"
  },
  {
    "path": "doc/development/prepare-commit-msg.sh",
    "content": "#!/bin/sh\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: Apache-2.0\n#\n# This Git hook adds refs to branch merges and adds the Arvados DCO sign-off\n# if you have configured it to do so.\n\nset -e\nset -u\n\nmsgfile=\"$1\"; shift\ntrailer=\"$(git interpret-trailers --trailer arvados /dev/null | grep @ || :)\"\n\nnew_msg=\"$(mktemp --tmpdir=\"$(dirname \"$msgfile\")\" commit-XXXXXX.txt)\"\ntrap 'rm -f \"$new_msg\"' EXIT INT TERM QUIT\ngawk -f - -v source=\"${1:-}\" -v trailer=\"$trailer\" -- \"$msgfile\" >\"$new_msg\" <<'EOF'\nBEGIN { $0=trailer; trailer_key=$1; }\nfunction write_trailer() {\n  if (trailer) {\n    if (last1 != trailer_key) { print \"\"; }\n    print trailer;\n    trailer=\"\";\n  }\n}\nEND { write_trailer(); }\n($0 == trailer) { trailer=\"\"; }\n((last1 == trailer_key && $1 != trailer_key) ||\n $1 == \"#\" || $1 == \"---\" || $1 == \"diff\") { write_trailer(); }\n(NR == 1 && $1 == \"Merge\" && $(NF - 1) == \"branch\") {\n  match($NF, /[[:punct:]]([0-9]+)[[:punct:]]/, bmatch);\n  sub(/^'.*\\//, \"'\", $NF);\n  print \"Merge branch\", $NF;\n  if (RSTART) {\n    printf(\"%sRefs #%s.%s\", ORS, bmatch[1], ORS);\n  }\n  last1=$1;\n  next;\n}\n{ print; last1=$1; }\nEOF\nmv -f \"$new_msg\" \"$msgfile\"\n"
  },
  {
    "path": "doc/development/release/Checklist.md",
    "content": "[comment]: # (Copyright © The Arvados Authors. All rights reserved.)\n[comment]: # ()\n[comment]: # (SPDX-License-Identifier: CC-BY-SA-3.0)\n\n# Release Checklist\n\nPre-process:\n\n1.  Create an issue for the release.\n2.  Add each of the following steps (starting at step 1) as tasks with the step number in the subject\n3.  Assign each task\n4.  The current task goes into the “In Progress” column\n5.  When the current task is finished, move it to resolved, and move the next task into “In Progress”\n6.  Notify the assignee of the next task that it is ready to begin\n\nMeta-process:\n\n1.  Periodically review this documented process reflects our actual process & update it\n2.  When steps are added/changed/rearranged/removed, be sure to update [`cmd/art/TASKS` in the `arvados-dev` repository](https://dev.arvados.org/projects/arvados/repository/arvados-dev/revisions/main/show/cmd/art).\n\n\n  \n    \n      \n      \n      \n    \n  \n  \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n    \n      \n      \n      \n    \n  \n
StepWhoWhat
0engineeringBuild new features, refine good code into great code
1ops\n        Build a new tordo compute image against the latest development packages.
\n        Update the tordo configuration and test it with a couple representative workflows (at least one bioinformatics workflow and one S3 download workflow).
\n        If everything works well, update version pins based on the versions installed in the new image. Update:\n        
    \n          
  • tools/ansible/roles/arvados_docker/files/arvados-docker.pref
    • \n          
  • tools/ansible/roles/compute_amd_rocm/defaults/main.yml (update arvados_compute_amd_rocm_version)
    • \n          
  • tools/ansible/roles/compute_nvidia/files/arvados-nvidia.pref
    • \n        
\n      
2engineeringPrepare release branch on the arvados and arvados-formula repositories. For major releases, this means branching a new X.Y-staging from main. For minor releases, this means cherry-picking features onto the existing X.Y-staging branch. Ensure that Redmine issues for features or bugfixes that are appearing for the first time in this version are associated with the correct release (for major releases, use art redmine issues find-and-associate).
3engineering\n        Ensure that the release staging branch passes automated tests on Jenkins.\n        \n      
4engineeringReview release branch to make sure all commits that need to be in the release are in the release. If new commits are added, resume checklist from step 3.
5product mgrWrite release notes and publish them on the www-dev site.
6everyoneReview release notes
7product mgrCreate a Redmine release for the next patch release after the current one.
8release engBuild release candidate packages with version X.Y.Z~rcN-1 using the Jenkins job build-and-publish-rc-packages. Add a comment on the release ticket identifying the Git commit hash used for the build, and link to your Jenkins run.
9release engPublish release candidate arvados/jobs Docker image using docker-jobs-image-release
10opsTest installer formula / provision scripts with RC packages. Run the test-provision Jenkins job where git_hash is your X.Y-staging commit and RELEASE is testing.
11ops\n        Update pirca to use the RC packages: build a new compute image, update the Arvados version in Salt and deploy.
\n        After Salt updates the cluster, check that your new version deployed successfully by running arvados-server version and then arvados-server check to verify other running services have the same version.\n      
12bfx\n        Run CWL integration tests and fastq-to-gvcf pipeline on pirca (more about running fastq-to-gvcf).
\n        After the workflow succeeds, check the versions reported at the top of the workflow logs to verify it ran your RC for crunch-run, arv-mount, and a-c-r.\n      
13engineeringPerform final manual testing based on risk assessment, the release notes and manual testing plan. This should involve at least a \"smell check\" to confirm that key features, improvements or bug fixes intended to appear in the release are present and behave as intended.
14product mgrApprove RC for release
15release engPublish Ruby gems using build-publish-packages-python-ruby with only the BUILD_RUBY box checked.
16release eng\n        On the X.Y-staging branch, update these files to refer to the release version:\n        
    \n          
  • doc/admin/upgrading.html.textile.liquid the \"Upgrading Arvados and Release notes\" doc page with the version and date of the release.
    • \n          
  • contrib/arvados-bootstrap/pyproject.toml, update project.version and project.dependencies
    • \n          
  • contrib/R-sdk/DESCRIPTION, update Version:\n          
  • services/api/Gemfile to depend on the newly published Arvados gem and run bundle install to update Gemfile.lock
    • \n          
  • tools/ansible/roles/arvados_apt/defaults/main.yml update arvados_pin_version
    • \n        
\n      
17release engBuild final release packages with version X.Y.Z-1 using the Jenkins job build-and-publish-rc-packages. Add a comment on the release ticket identifying the Git commit hash used for the build, and link to your Jenkins run.
18release engPublish stable release arvados/jobs Docker image using docker-jobs-image-release
19release engPush packages to stable repos using publish-packages-to-stable-repo (more info)
20release engPublish Python packages using build-publish-packages-python-ruby with only the BUILD_PYTHON box checked.
21release engPublish Java package using build-java-sdk and following Releasing Java SDK packages
22release engPublish R package using build-package-r
23release eng\n        Tag the commits in each repo used to build the release in Git. Create an annotated tag (git tag --annotate) with a message like \"Release notes at https://arvados.org/release-notes/X.Y.Z/\" That makes the GitHub releases page look good. See GitHub documentation for more details about how to automate releases.
\n        Create or fast forward the X.Y-release branch to match X.Y-staging.
\n        Cherry-pick the upgrade notes commit (from step 2) onto main.\n      
24release eng\n        Ensure new release is published on https://doc.arvados.org/.
\n        Ensure that release notes & any other materials are pointing to correct version of the docs.
\n        (If anything goes wrong, see https://dev.arvados.org/projects/arvados-private/wiki/Docarvadosorg_deployment)\n      
25opsUpdate pirca and jutro to the new stable release: build new compute images, update the Arvados version in Salt and deploy.
26product mgrMerge release notes (step 6) from \"develop\" branch to \"main\" branch of the arvados-www Git repository and check that the https://arvados.org front page is updated
27product mgrSend out the release notes via MailChimp, tweet from the Arvados account, announce on the Discourse forum, Matrix, etc.
28release eng\n        In Jenkins:\n        
    \n          
  • For each test from step 3, go to \"Job Config History\" and record on the release ticket the timestamp of the configuration used to test the release
    • \n          
  • Go to Manage Jenkins > Clouds > gce2 > Configure and record the VM image tagged \"tests\" used for jenkins workers to run the tests for the release (should be something like jenkins-image-arvados-tests-YYYYMMDDHHMMSS) on the release ticket
    • \n          
  • Go to packer-build-jenkins-image-arvados-tests history and record on the release ticket the Jenkins job used to build the above VM image.
    • \n        
      \n      
29release eng\n        Add the release to doi:10.5281/zenodo.6382942
\n        Updating Zenodo Version of Arvados after Release
\n        https://zenodo.org/record/6382943\n      
\n"
  },
  {
    "path": "doc/development/release/FastqPipeline.md",
    "content": "[comment]: # (Copyright © The Arvados Authors. All rights reserved.)\n[comment]: # ()\n[comment]: # (SPDX-License-Identifier: CC-BY-SA-3.0)\n\n# More about running fastq-to-gVCF\n\nWhen we do releases, we run a test pipeline that is intended to be representative of a bioinformatics workload.\n\n1. Deploy the version of `arvados-cwl-runner` that you want to test and make sure that the corresponding `arvados/jobs` image [has been built and uploaded to docker hub](https://ci.arvados.org/view/Release%20Pipeline/job/docker-jobs-image-release/) or built using the `arvados/build/build-dev-docker-jobs-image.sh` script and uploaded using `arv-keepdocker`.\n2. Clone \n3. Create an Arvados project for the test run\n4. `cd arvados/tutorial/WGS-processing`\n5. Run the following command: `arvados-cwl-runner --no-wait --disable-reuse --project-uuid  cwl/wgs-processing-wf.cwl yml/wgs-processing-wf-chr19.yml`\n6. Monitor this for success. It usually takes about an hour to run.\n\nIf you are running this on `pirca` then all the data should already be present. If you are running it from somewhere else, you may need to do some additional data copying from `pirca` to the other cluster. The input document `yml/wgs-processing-wf-chr19.yml` has the portable data hashes of the collections.\n"
  },
  {
    "path": "doc/development/release/JavaSDK.md",
    "content": "[comment]: # (Copyright © The Arvados Authors. All rights reserved.)\n[comment]: # ()\n[comment]: # (SPDX-License-Identifier: CC-BY-SA-3.0)\n\n# Releasing Java SDK packages\n\nThe Java SDK is distributed on the Sonatype Central Repository. Here are the steps to release a new jar file:\n\n1.  Build and upload package using https://ci.arvados.org/view/All/job/build-java-sdk\n2.  Go to [Sonatype Publishing Settings](https://central.sonatype.com/publishing/deployments) and log in with the appropriate credentials (gopass oss.sonatype.org/curii)\n3.  Make sure you’re on the “Deployments” tab.\n4.  Find the jar that was just uploaded by Jenkins. Click the “Publish” button and wait for the process to finish.\n\nSee [documentation about the publishing API we use](https://central.sonatype.org/publish/publish-portal-ossrh-staging-api/).\n\n## Getting the authentication token for Sonatype\n\n[Log into Sonatype](https://central.sonatype.com/usertoken) and under the account menu select “User Tokens” to review and manage tokens. Our current Jenkins token is stored in gopass as `curii-systems/websites/oss.sonatype.org/jenkins`.\n\n## gradle.properties\n\nTo upload to Sonatype, you need the token (see above) and a secret key. You must upload a GPG-signed package. All these parameters are set in `gradle.properties` which we keep as a Jenkins secret. Note that the property values after the equals sign should not be quoted. I’m not certain if spaces are allowed around the equals sign, but currently it works with no extra spaces.\n\n    ossrhUsername=...\n    ossrhPassword=...\n    signing.keyId=... \n    signing.password= \n    signing.secretKeyRingFile=...-secret-key.gpg \n"
  },
  {
    "path": "doc/development/release/ManualTests.md",
    "content": "[comment]: # (Copyright © The Arvados Authors. All rights reserved.)\n[comment]: # ()\n[comment]: # (SPDX-License-Identifier: CC-BY-SA-3.0)\n\n# Manual testing plan\n\n## Manual testing of SDKs that don’t have good coverage\n\n## Release candidate builds with ~1 ~2\n\n## Workbench2\n\nNeed to go through this whole testing plan with both an admin and non-admin account. Admin only operations are indicated.\n\n### Login\n\n- Test login using username/password\n- Test login using OpenID Connect\n- Test login as federated user\n  - Login to using remote account on centralized federation (LoginCluster)\n  - Login to using remote account on peer federation\n\n### Left side navigation\n\n- Click on each top level icon (home projects, favorites/public favorite, shared, all processes, instance types, shell access, groups, trash) and confirm that the appropriate page loads with no errors\n- Check that the left side panel can be resized\n- Check that that the toggle side panel button works as expected\n- Check that the +NEW button is disabled unless a project is displayed\n\n### Home projects top panel\n\n- Project name should match the logged in user\n- Check for expected toolbar buttons\n  - Details\n  - User account\n  - API Details\n- Check the buttons have the expected behavior\n\n### Project view\n\n#### Top panel\n\nCheck that it shows the project name\n\nCheck that it shows the first line of the project description. Check that there is an arrow that expands to show the full description.\n\nCheck that it shows project properties. Check that there is an arrow that expands to show all the properties if they don’t fit on a single line.\n\nCheck that it renders the toolbar of project operations (listed below).\n\nCheck that the each operation behaves correctly and operates on the project.\n\nCheck that if the window is narrow, the rightmost toolbar icons spill into an overflow menu\n\nCheck that breadcrumbs include the project name and each parent project.\n\n#### Data tab\n\nShould show projects, workflows, and collections (in that order)\n\nClicking on the name rendered in blue text should navigate to the item\n\nClicking anywhere else but the name should toggle between selected and not selected\n\n- Unless clicking on the checkbox, clicking on the row clears any other selected items\n- When a row is selected, the toolbar moves from the top panel to the data table panel\n- Clicking on the check box to the left when a different item is selected selects both items\n  - The toolbar updates to show only the operations that can be applied to both items\n- Clicking “View details” should open the right info panel. Check that it shows details for the currently selected item.\n\nCheck that the toolbar operations are sorted and grouped consistently across different types of items.\n\nCheck that the toolbar operations are appropriate to the type of item selected.\n\nExpected toolbar when project is selected:\n\n- View details\n- Open in new tab\n- Copy link to clipboard\n- Open with 3rd party client\n- API details\n- —-\n- Share\n- New project\n- Edit project\n- Move to\n- Move to trash\n- —-\n- Freeze project\n- Add to favorites\n- Add to public favorites (admin only)\n\nExpected toolbar when workflow is selected:\n\n- View details\n- Open in new tab\n- Copy link to clipboard\n- API details\n- —-\n- Run workflow\n- Delete workflow\n\nExpected toolbar when collection is selected\n\n- View details\n- Open in new tab\n- Copy link to clipboard\n- Open with 3rd party client\n- API details\n- —-\n- Share\n- Edit collection\n- Move to\n- Make a copy\n- Move to trash\n- —-\n- Add to favorites\n- Add to public favorites (admin only)\n\nCheck that all the toolbar operations work as expected.\n\nCheck that right-clicking on a row selects the row and then opens the appropriate context menu.\n\n- Check that the operations apply to the item that was clicked on.\n- Check that the operations in the right-click context menu match the toolbar.\n\nCheck that clicking on each action in the context menu works as expected.\n\nCheck that entering text into the search box refreshes the list with search results\n\nCheck that clicking on the three bars in the upper right opens a menu to select columns\n\nCheck that enabling/disabling data columns works. Check that all columns are filled in appropriately for each item, or blank (“-”) where no such data applies.\n\nCheck that clicking on the “Name” column sorts by name.\n\n- Check sort by “Date created”\n- Check sort by “Last modified”\n- Check sort by “Trash at”\n- Check sort by “Delete at”\n\nCheck that clicking on “Go to the next page” loads the next page of items.\n\nCheck that the getting the number of items doesn’t block loading the table contents.\n\n#### Workflows tab\n\nCheck it shows processes (workflow runs) only.\n\nCheck that it shows the number of completed, failed, queued and running processes, as well as the total, at the top of the data table.\n\nCheck that it shows the name, status, type, runtime and last modified times.\n\nCheck that entering text into the search box refreshes the list with search results\n\nCheck that the toolbar and context menu behave as expected:\n\n- View details\n- Open in new tab\n- Outputs\n- API details\n- —-\n- Edit process\n- Copy and re-run process\n- Remove\n- —-\n- Add to favorites\n- Add to public favorites (admin only)\n\nCheck that selecting more that one item updates the toolbar to “Remove”.\n\nCheck that clicking on “Go to the next page” loads the next page of items.\n\nCheck that the getting the number of items doesn’t block loading the table contents.\n\nCheck that process status is rendered correctly.\n\nCheck that filtering by process status shows only rows with the intended status.\n\nCheck that runtime is calculated/rendered correctly.\n\n### My favorites\n\nCheck that all items marked as “favorite” appear.\n\nCheck that clicking on an item shows the appropriate toolbar.\n\nCheck that clicking “Remove from favorites” on an item refreshes the favorites list and the item is no longer present.\n\n### Public favorites\n\nCheck that all items marked as “public favorite” appear.\n\nCheck that clicking on an item shows the appropriate toolbar.\n\nCheck that clicking “Remove from public favorites” on an item refreshes the public favorites list and the item is no longer present. (admin only)\n\n### Shared with me\n\nCheck that it shows all the things that don’t belong to the current user.\n\nCheck that selecting an item shows the appropriate toolbar.\n\nCheck that clicking on “Go to the next page” loads the next page of items.\n\nCheck that the getting the number of items doesn’t block loading the table contents.\n\n### All processes\n\nCheck that it shows all processes visible to the user, regardless of owner project.\n\nCheck that selecting an item shows the appropriate toolbar.\n\nCheck that clicking on “Go to the next page” loads the next page of items.\n\nCheck that the getting the number of items doesn’t block loading the table contents.\n\n### Instance types\n\nCheck that all the available instance types are listed and formatted properly.\n\n### Shell Access\n\nCheck that shell nodes are listed.\n\nCheck that the ssh command line is valid.\n\nCheck that webshell works properly.\n\n## Groups — standalone and peer federation\n\n1.  Create group\n2.  Log in as non-admin user.\n3.  Log in as a second non-admin user in a private window for testing sharing.\n4.  check that users cannot see one another\n5.  Add user to group\n6.  Check that users can see one another\n\n## Collections\n\n1.  Create a collection & upload a file\n2.  Add a file\n3.  Rename a file\n4.  Remove a file\n5.  Download one of the files\n6.  Make a sharing link to the collection & check usage from private window\n7.  Mark collection as a favorite, check that it shows up in favorites\n8.  Rename collection\n9.  Edit description\n10. Add property\n11. Search for collection by property\n12. Search for collection by name\n13. Search for collection by filename\n14. Search for collection by keyword in description\n15. Trash collection\n16. Check that collection can be found in the trash\n17. Untrash collection\n\n## Projects\n\n1.  Create a project\n2.  Rename a project\n3.  Edit description\n4.  Create a collection inside the project\n5.  Move a collection into the project\n6.  Add read-only sharing permission to the project & check access from other user\n7.  Add read-write sharing permission to project & check access from other user\n8.  Add manage sharing permission to project & check access from other user\n9.  Mark project as favorite, check that it shows up in favorites\n10. Search for project by name\n11. Search for project by keyword in description\n12. Trash project\n13. Check that project can be found in the trash\n14. Untrash project\n\n## Workflows\n\n1.  Upload workflow with arvados-cwl-runnner —create-workflow\n2.  Browse workflow\n3.  Select workflow to run\n4.  Choose input file\n5.  Watch it run\n    1.  Check logging\n    2.  Check live updates\n    3.  Check links to input & output\n6.  Check that it shows up in All Processes\n\n## Federation\n\n### Peer federation\n\n2 or more clusters are configured with a ‘Remoteclusters’ entry in config.yml.\n\n### Login cluster federation\n\n2 or more clusters are configured with a ‘Remoteclusters’ entry in config.yml. One of the clusters is the ‘login cluster’, which means the **other** clusters have a section like this in their config (clsr1 is the login cluster):\n\n    Clusters:\n      clsr2:\n        Login:\n          LoginCluster: clsr1\n\n#### Groups\n\n1.  Login cluster: create group\n2.  Satellite cluster: Log in as non-admin user.\n3.  Satellite cluster: Log in as a second non-admin user in a private window for testing sharing.\n4.  Satellite cluster: check that users cannot see one another\n5.  Login cluster: add both users to group\n6.  Satellite cluster: Check that users can see one another\n7.  Satellite cluster: create group\n8.  Satellite cluster: add both users to group\n9.  Satellite cluster: Check that both users can share with the group created on the satellite cluster\n\n## Misc\n\n1.  As admin, create a “public favorite” and make sure users see it.\n2.  As admin, deactivate a user. Make sure that user can’t log back in\n3.  Add a cluster for multi-site search.\n4.  Upload ssh key & check view\n5.  Create git repo & check view\n6.  As admin, add virtual machine access & check view\n"
  },
  {
    "path": "doc/development/release/Zenodo.md",
    "content": "[comment]: # (Copyright © The Arvados Authors. All rights reserved.)\n[comment]: # ()\n[comment]: # (SPDX-License-Identifier: CC-BY-SA-3.0)\n\n# Updating Zenodo Version of Arvados after Release\n\n1.  Download a `.zip` of your new Arvados release from [GitHub Releases](https://github.com/arvados/arvados/tags)\n2.  Log in to [Zenodo](https://zenodo.org/) using the credentials from `gopass \"curii-systems/zenodo.org/sysadmin+zenodo@curii.com\"`\n3.  Go to the [Arvados record](https://zenodo.org/records/15213491) and press the the New Version button  \n    (Using new versions lets us use the overarching DOI for our `citations.md` and keep all the versions together on Zenodo)\n4.  In the form, update the following:\n    1.  Upload the `.zip` file for this release you downloaded earlier\n    2.  Request a new DOI for this version\n    3.  Update the Publication Date for this release\n    4.  Add any Creators who have worked on Arvados and aren’t listed\n    5.  Under Additional Description, edit the links for Release Notes and (if you’re doing a major release) Documentation\n    6.  Update the Version number with this release\n\nOnce you add a new version, you can’t change its DOI but everything else is editable if you accidentally make a mistake. So, don’t worry :) just edit the new version to fix it.\n"
  },
  {
    "path": "doc/examples/config/zzzzz.yml",
    "content": "AutoReloadConfig: true\nClusters:\n  zzzzz:\n    ManagementToken: e687950a23c3a9bceec28c6223a06c79\n    SystemRootToken: systemusertesttoken1234567890aoeuidhtnsqjkxbmwvzpy\n    API:\n      RequestTimeout: 30s\n    TLS:\n      Insecure: true\n    Collections:\n      BlobSigningKey: zfhgfenhffzltr9dixws36j1yhksjoll2grmku38mi7yxd66h5j4q9w4jzanezacp8s6q0ro3hxakfye02152hncy6zml2ed0uc\n      TrustAllContent: true\n      ForwardSlashNameSubstitution: /\n"
  },
  {
    "path": "doc/index.html.liquid",
    "content": "---\nlayout: default\nno_nav_left: true\nnavsection: top\ntitle: Arvados | Documentation\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n
\n  
\n    
\n      
\n        
ARVADOS
\n        
A free and open source platform for big data science
\n      
\n      
\n        \"Dax\n      
\n    
\n  
\n
\n\n
\n  
\n    
\n      
What is Arvados\n      
Arvados is a platform for managing compute and storage for cloud and HPC clusters. It allows you to track your methods and datasets, share them securely, and easily re-run analyses.  It also make it possible to run analysis across multiple clusters (HPC, cloud, or hybrid) with Federated Multi-Cluster Workflows.\n      
\n\n      \n      
Support and Community
\n\n      
Interact with the Arvados community on the Arvados Forum\n\tand the arvados/community channel at gitter.im.\n      
\n\n      
Curii Corporation provides managed Arvados installations as well as commercial support for Arvados. Please contact info@curii.com for more information.
\n\n      
Contributing
\n      
Please visit the developer documentation. Arvados is 100% free and open source software, check out the code on GitHub.\n\n      
Arvados is under active development, see the recent developer activity.\n      
\n      
License
\n      
Most of Arvados is licensed under the GNU AGPL v3. The SDKs are licensed under the Apache License 2.0 and can be incorporated into proprietary code. See Arvados Free Software Licenses for more information.\n      
\n\n    
\n    
\n      
Sections\n      
\n      
\n        User Guide — How to manage data and do analysis with Arvados.\n      
\n      
\n        SDK Reference — Details about the accessing Arvados from various programming languages.\n      
\n      
\n        Arvados Architecture — Details about the Arvados components and architecture.\n      
\n      
\n        API Reference — Details about the Arvados REST API.\n      
\n      
\n        Admin Guide — Details about administering an Arvados cluster.\n      
\n      
\n        Install Guide — How to install Arvados.\n      
\n    
\n  
\n\n  
\n    
\n      
\n      
\n      The content of the above documentation is licensed under the\n      Creative\n        Commons Attribution-Share Alike 3.0 United States license. Code samples in the above documentation are licensed under the\n      Apache License, Version 2.0.\n      
\n    
\n  
\n
\n"
  },
  {
    "path": "doc/install/arvbox.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Arvados-in-a-box\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nh2. arvbox is unsupported\n\nThe Arvados team does not maintain or support arvbox as of Arvados 3.2.0.\n\nIf you were using arvbox in demo mode, consider installing on a Debian-based virtual machine with our \"single-node Ansible installer\":{{ site.baseurl }}/install/install-single-host.html.\n\nIf you were using arvbox for development, we now provide an Ansible playbook to install development dependencies on a Debian-based system. Our \"Hacking Prerequisites documentation\":https://github.com/arvados/arvados/blob/main/doc/development/Prerequisites.md has instructions for how to use it.\n\nInstalling systems with Ansible requires a little more initial setup, but once you've done that, it's easier to keep a system up-to-date: when you want to update a system, you simply re-run the playbook. We think this trade-off lets us provide a better experience to a wider variety of users.\n"
  },
  {
    "path": "doc/install/config.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Configuration files\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nh2. Arvados /etc/arvados/config.yml\n\nThe configuration file is normally found at @/etc/arvados/config.yml@ and will be referred to as just @config.yml@ in this guide.  This configuration file must be kept in sync across every service node in the cluster, but not shell and compute nodes (which do not require config.yml).\n\nh3. Syntax\n\nThe configuration file is in \"YAML\":https://yaml.org/ format.  This is a block syntax where indentation is significant (similar to Python).  By convention we use two space indent.  The first line of the file is always \"Clusters:\", underneath it at the first indent level is the Cluster ID.  All the actual cluster configuration follows under the Cluster ID.  This means all configuration parameters are indented by at least two levels (four spaces).  Comments start with @#@ .\n\nWe recommend a YAML-syntax plugin for your favorite text editor, such as @yaml-mode@ (Emacs) or @yaml-vim@.\n\nExample file:\n\n
\nClusters:                         # Clusters block, everything else is listed under this\n  abcde:                          # Cluster ID, everything under it is configuration for this cluster\n    ExampleConfigKey: \"fghijk\"    # An example configuration key\n    ExampleConfigGroup:           # A group of keys\n      ExampleDurationConfig: 12s  # Example duration\n      ExampleSizeConfig: 99KiB    # Example with a size suffix\n
\n\nEach configuration group may only appear once.  When a configuration key is within a config group, it will be written with the group name leading, for example @ExampleConfigGroup.ExampleSizeConfig@.\n\nDuration suffixes are s=seconds, m=minutes or h=hours.\n\nSize suffixes are K=10 ^3^, Ki=2 ^10^, M=10 ^6^, Mi=2 ^20^, G=10 ^9^, Gi=2 ^30^, T=10 ^12^, Ti=2 ^40^, P=10 ^15^, Pi=2 ^50^, E=10 ^18^, Ei=2 ^60^.  You can optionally follow with a \"B\" (eg \"MB\" or \"MiB\") for readability (it does not affect the units.)\n\nh3(#empty). Create empty configuration file\n\nChange @webserver-user@ to the user that runs your web server process.  This is @www-data@ on Debian-based systems, and @nginx@ on Red Hat-based systems.\n\n\n
# export ClusterID=xxxxx\n# umask 027\n# mkdir -p /etc/arvados\n# cat > /etc/arvados/config.yml <<EOF\nClusters:\n  ${ClusterID}:\nEOF\n# chgrp webserver-user /etc/arvados /etc/arvados/config.yml\n
\n\n\nh2. Nginx configuration\n\nThis guide will also cover setting up \"Nginx\":https://www.nginx.com/ as a reverse proxy for Arvados services.  Nginx performs two main functions: TLS termination and virtual host routing.  The virtual host configuration for each component will go in its own file in @/etc/nginx/conf.d/@.\n\nh2. Synchronizing config file\n\nThe Arvados configuration file must be kept in sync across every service node in the cluster.  We strongly recommend using a devops configuration management tool such as \"Puppet\":https://puppet.com/open-source/ to synchronize the config file.  Alternately, something like the following script to securely copy the configuration file to each node may be helpful.  Replace the @ssh@ targets with your nodes.\n\n\n
#!/bin/sh\nsudo cat /etc/arvados/config.yml | ssh 10.0.0.2 sudo sh -c \"'cat > /etc/arvados/config.yml'\"\nsudo cat /etc/arvados/config.yml | ssh 10.0.0.3 sudo sh -c \"'cat > /etc/arvados/config.yml'\"\n
\n\n"
  },
  {
    "path": "doc/install/configure-azure-blob-storage.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Configure Azure Blob storage\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nKeepstore can store data in one or more Azure Storage containers.\n\nh2. Set up VMs and Storage Accounts\n\nBefore starting the configuration of individual keepstore servers is good to have an idea of the keepstores servers' final layout. One key decision is the amount of servers and type of VM to run. Azure may change over time the bandwith capacity of each type. After conducting some empirical saturation tests, the conclusion was that the bandwith is proportional to the amount of cores with some exceptions. As a rule of thumb, is better to invest resources in more cores instead of memory or IOps.\n\nAnother decision is how many VMs should be running keepstore. For example there could be 8 VMs with one core each or one machine with 8 cores. Or anything in between. Assuming is the same cost for Cloud resources, there is always the benefit of distributing the risk of faulty VMs. The recommendation is to start with 2 VMs and expand in pairs. Having a minimum of 2 cores each. The total amount of VMs will be a function of the budget and the pipeline traffic to avoid saturation during periods of high usage. Standard D v3 family is a balanced choice, making Standard_D2_v3 the 2-core option\n\nThere are many options for storage accounts. You can read details from Azure on their documentation \"https://docs.microsoft.com/en-us/azure/storage/common/storage-introduction\":https://docs.microsoft.com/en-us/azure/storage/common/storage-introduction. The type of storage and access tier will be a function of the budget and desired responsiveness. A balanced option is to have General-purpose Standard Storage account and use Blob storage, hot access tiers.\n\nKeepstore can be configure to reflect the level of underlaying redundancy the storage will have. This is call data replication option. For example LRS (Locally Redundant Storage) saves 3 copies of the data. There desired redundancy can be chosen at the keepstore layer or at the Storage Accunt layer. The decision where the redundancy will be done and the type of Storage Account data replication (LRS, ZRS, GRS and RA-GRS) has trade-offs. Please read more on \"https://docs.microsoft.com/en-us/azure/storage/common/storage-redundancy\":https://docs.microsoft.com/en-us/azure/storage/common/storage-redundancy and decide what is best for your needs.\n\nh2. Create a storage container\n\nUsing the Azure web portal or command line tool, create or choose a storage account with a suitable redundancy profile and availability region. Use the storage account keys to create a new container.\n\n\n
~$ azure config mode arm\n~$ az login\n~$ az group create exampleGroupName eastus2\n~$ az storage account create --sku Standard_LRS --kind BlobStorage --encryption-services blob --access-tier Hot --https-only true --location eastus2 --resource-group exampleGroupName --name exampleStorageAccountName\n~$ az storage account keys list --resource-group exampleGroupName --account-name exampleStorageAccountName\n[\n  {\n    \"keyName\": \"key1\",\n    \"permissions\": \"Full\",\n    \"value\": \"zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz==\"\n  },\n  {\n    \"keyName\": \"key2\",\n    \"permissions\": \"Full\",\n    \"value\": \"yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy==\"\n  }\n]\n~$ AZURE_STORAGE_ACCOUNT=\"exampleStorageAccountName\" \\\nAZURE_STORAGE_ACCESS_KEY=\"zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz==\" \\\nazure storage container create --name exampleContainerName\n
\n\n\nNote that Keepstore services may be configured to use multiple Azure Storage accounts and multiple containers within a storage account.\n\nh2. Configure keepstore\n\nVolumes are configured in the @Volumes@ section of the cluster configuration file.\n\n{% include 'assign_volume_uuid' %}\n\n
    Volumes:\n      ClusterID-nyw5e-000000000000000:\n        AccessViaHosts:\n          # This section determines which keepstore servers access the\n          # volume. In this example, keep0 has read/write access, and\n          # keep1 has read-only access.\n          #\n          # If the AccessViaHosts section is empty or omitted, all\n          # keepstore servers will have read/write access to the\n          # volume.\n          \"http://keep0.ClusterID.example.com:25107\": {}\n          \"http://keep1.ClusterID.example.com:25107\": {ReadOnly: true}\n\n        Driver: Azure\n        DriverParameters:\n          # Storage account name and secret key, used for\n          # authentication.\n          StorageAccountName: exampleStorageAccountName\n          StorageAccountKey: zzzzzzzzzzzzzzzzzzzzzzzzzz\n\n          # Storage container name.\n          ContainerName: exampleContainerName\n\n          # The cloud environment to use,\n          # e.g. \"core.chinacloudapi.cn\". Defaults to\n          # \"core.windows.net\" if blank or omitted.\n          StorageBaseURL: \"\"\n\n          # Time to wait for an upstream response before failing the\n          # request.\n          RequestTimeout: 10m\n\n          # Time to wait before retrying a failed \"list blobs\" Azure\n          # API call.\n          ListBlobsRetryDelay: 10s\n\n          # Maximum attempts at a \"list blobs\" Azure API call before\n          # giving up.\n          ListBlobsMaxAttempts: 12\n\n          # If non-zero, use multiple concurrent requests (each\n          # requesting MaxGetBytes bytes) when retrieving data. If\n          # zero or omitted, get the entire blob with one request.\n          #\n          # Normally this is zero but if you find that 4 small\n          # requests complete faster than a single large request, for\n          # example, you might set this to 16777216 (64 MiB ÷ 4).\n          MaxGetBytes: 0\n\n          # Time to wait for an unexpectedly empty blob to become\n          # non-empty. Azure's create-and-write operation is not\n          # atomic. The default value typically allows concurrent GET\n          # and PUT requests to succeed despite the race window.\n          WriteRaceInterval: 15s\n\n          # Time to wait between GET attempts while waiting for\n          # WriteRaceInterval to expire.\n          WriteRacePollTime: 1s\n\n        # How much replication is provided by the underlying storage\n        # container.  This is used to inform replication decisions at\n        # the Keep layer.\n        Replication: 3\n\n        # If true, do not accept write or trash operations, even if\n        # AccessViaHosts.*.ReadOnly is false.\n        #\n        # If false or omitted, enable write access (subject to\n        # AccessViaHosts.*.ReadOnly, where applicable).\n        ReadOnly: false\n\n        # Storage classes to associate with this volume.  See \"Storage\n        # classes\" in the \"Admin\" section of doc.arvados.org.\n        StorageClasses: null\n
\n"
  },
  {
    "path": "doc/install/configure-fs-storage.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Configure filesystem storage\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nKeepstore can store data in local and network-attached POSIX filesystems.\n\nh2. Setting up filesystem mounts\n\nVolumes are configured in the @Volumes@ section of the cluster configuration file.  You may provide multiple volumes for a single keepstore process to manage multiple disks.  Keepstore distributes blocks among volumes in round-robin fashion.\n\n{% include 'assign_volume_uuid' %}\n\nNote that each volume entry has an @AccessViaHosts@ section indicating which Keepstore instance(s) will serve that volume.  In this example, keep0 and keep1 each have their own data disk.  The @/mnt/local-disk@ directory on keep0 is volume @ClusterID-nyw5e-000000000000000@, and the @/mnt/local-disk@ directory on keep1 is volume @ClusterID-nyw5e-000000000000001@ .\n\n\n
    Volumes:\n      ClusterID-nyw5e-000000000000000:\n        AccessViaHosts:\n          \"http://keep0.ClusterID.example.com:25107\": {}\n        Driver: Directory\n        DriverParameters:\n          # The directory that will be used as the backing store.\n          Root: /mnt/local-disk\n\n        # How much replication is performed by the underlying\n        # filesystem.  (for example, a network filesystem may provide\n        # its own replication).  This is used to inform replication\n        # decisions at the Keep layer.\n        Replication: 1\n\n        # If true, do not accept write or trash operations, only\n        # reads.\n        ReadOnly: false\n\n        # Storage classes to associate with this volume.\n        StorageClasses: null\n\n      ClusterID-nyw5e-000000000000001:\n        AccessViaHosts:\n          \"http://keep1.ClusterID.example.com:25107\": {}\n        Driver: Directory\n        DriverParameters:\n          Root: /mnt/local-disk\n
\n\nIn the case of a network-attached filesystem, the @AccessViaHosts@ section can have multiple entries. If the filesystem is accessible by all keepstore servers, the AccessViaHosts section can be empty, or omitted entirely.  In this example, the underlying storage system performs replication, so specifying @Replication: 2@ means a block is considered to be stored twice for the purposes of data integrity, while only stored on a single volume from the perspective of Keep.\n\n\n
    Volumes:\n      ClusterID-nyw5e-000000000000002:\n        AccessViaHosts:\n          # This section determines which keepstore servers access the\n          # volume. In this example, keep0 has read/write access, and\n          # keep1 has read-only access.\n          #\n          # If the AccessViaHosts section is empty or omitted, all\n          # keepstore servers will have read/write access to the\n          # volume.\n          \"http://keep0.ClusterID.example.com:25107/\": {}\n          \"http://keep1.ClusterID.example.com:25107/\": {ReadOnly: true}\n        Driver: Directory\n        DriverParameters:\n          Root: /mnt/network-attached-filesystem\n        Replication: 2\n
\n"
  },
  {
    "path": "doc/install/configure-s3-object-storage.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Configure S3 object storage\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nKeepstore can store data in object storage compatible with the S3 API, such as Amazon S3, Google Cloud Storage, Ceph RADOS, NetApp StorageGRID, and others.\n\nVolumes are configured in the @Volumes@ section of the cluster configuration file.\n\n# \"Configuration example\":#example\n# \"IAM Policy\":#IAM\n\nh2(#example). Configuration example\n\n{% include 'assign_volume_uuid' %}\n\n
    Volumes:\n      ClusterID-nyw5e-000000000000000:\n        AccessViaHosts:\n          # This section determines which keepstore servers access the\n          # volume. In this example, keep0 has read/write access, and\n          # keep1 has read-only access.\n          #\n          # If the AccessViaHosts section is empty or omitted, all\n          # keepstore servers will have read/write access to the\n          # volume.\n          \"http://keep0.ClusterID.example.com:25107\": {}\n          \"http://keep1.ClusterID.example.com:25107\": {ReadOnly: true}\n\n        Driver: S3\n        DriverParameters:\n          # Bucket name.\n          Bucket: example-bucket-name\n\n          # Optionally, you can specify S3 access credentials here.\n          # If these are left blank, IAM role credentials will be\n          # retrieved from instance metadata (IMDSv2).\n          AccessKeyID: \"\"\n          SecretAccessKey: \"\"\n\n          # Storage provider region. If Endpoint is specified, the\n          # region determines the request signing method, and defaults\n          # to \"us-east-1\".\n          Region: us-east-1\n\n          # Storage provider endpoint. For Amazon S3, use \"\" or\n          # omit. For Google Cloud Storage, use\n          # \"https://storage.googleapis.com\".\n          Endpoint: \"\"\n\n          # Change to true if the region requires a LocationConstraint\n          # declaration.\n          LocationConstraint: false\n\n          # Use V2 signatures instead of the default V4. Amazon S3\n          # supports V4 signatures in all regions, but this option\n          # might be needed for other S3-compatible services.\n          V2Signature: false\n\n          # Use path-style requests instead of the default\n          # virtual-hosted-style requests.  This might be needed for\n          # S3-compatible services other than AWS.  If using AWS, see\n          # https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#path-style-access\n          # for deprecation information.\n          UsePathStyle: false\n\n          # By default keepstore stores data using the MD5 checksum\n          # (32 hexadecimal characters) as the object name, e.g.,\n          # \"0123456abc...\". Setting PrefixLength to 3 changes this\n          # naming scheme to \"012/0123456abc...\". This can improve\n          # performance, depending on the S3 service being used. For\n          # example, PrefixLength 3 is recommended to avoid AWS\n          # limitations on the number of read/write operations per\n          # second per prefix (see\n          # https://aws.amazon.com/premiumsupport/knowledge-center/s3-request-limit-avoid-throttling/).\n          #\n          # Note that changing PrefixLength on an existing volume is\n          # not currently supported. Once you have started using a\n          # bucket as an Arvados volume, you should not change its\n          # configured PrefixLength, or configure another volume using\n          # the same bucket and a different PrefixLength.\n          PrefixLength: 0\n\n          # Requested page size for \"list bucket contents\" requests.\n          IndexPageSize: 1000\n\n          # Maximum time to wait while making the initial connection\n          # to the backend before failing the request.\n          ConnectTimeout: 1m\n\n          # Maximum time to wait for a complete response from the\n          # backend before failing the request.\n          ReadTimeout: 2m\n\n          # Maximum eventual consistency latency\n          RaceWindow: 24h\n\n        # How much replication is provided by the underlying bucket.\n        # This is used to inform replication decisions at the Keep\n        # layer.\n        Replication: 2\n\n        # If true, do not accept write or trash operations, even if\n        # AccessViaHosts.*.ReadOnly is false.\n        #\n        # If false or omitted, enable write access (subject to\n        # AccessViaHosts.*.ReadOnly, where applicable).\n        ReadOnly: false\n\n        # Storage classes to associate with this volume.  See \"Storage\n        # classes\" in the \"Admin\" section of doc.arvados.org.\n        StorageClasses: null\n
\n\nh2(#IAM). IAM Policy\n\nOn Amazon, VMs which will access the S3 bucket (these include keepstore and compute nodes) will need an IAM policy with \"permission that can read, write, list and delete objects in the bucket\":https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create.html .  Here is an example policy:\n\n\n
\n{\n    \"Id\": \"arvados-keepstore policy\",\n    \"Statement\": [\n        {\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                  \"s3:*\"\n            ],\n            \"Resource\": \"arn:aws:s3:::xarv1-nyw5e-000000000000000-volume\"\n            \"Resource\": \"arn:aws:s3:::xarv1-nyw5e-000000000000000-volume/*\"\n        }\n    ]\n}\n
\n\n"
  },
  {
    "path": "doc/install/container-shell-access.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Configure container shell access\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nArvados can be configured to permit shell access to running containers. This can be handy for debugging, but it could affect reproducability of workflows. This feature can be enabled for admin users, or for all users. By default, it is entirely disabled.\n\nThe relevant configuration section is\n\n\n
    Containers:\n      ShellAccess:\n        # An admin user can use \"arvados-client shell\" to start an\n        # interactive shell (with any user ID) in any running\n        # container.\n        Admin: false\n\n        # Any user can use \"arvados-client shell\" to start an\n        # interactive shell (with any user ID) in any running\n        # container that they started, provided it isn't also\n        # associated with a different user's container request.\n        #\n        # Interactive sessions make it easy to alter the container's\n        # runtime environment in ways that aren't recorded or\n        # reproducible. Consider the implications for automatic\n        # container reuse before enabling and using this feature. In\n        # particular, note that starting an interactive session does\n        # not disqualify a container from being reused by a different\n        # user/workflow in the future.\n        User: false\n
\n\n\nTo enable the feature a firewall change may also be required. This feature requires the opening of tcp connections from @arvados-controller@ to the range specified in the @net.ipv4.ip_local_port_range@ sysctl on compute nodes. If that range is unknown or hard to determine, it will be sufficient to allow tcp connections from @arvados-controller@ to port 1024-65535 on compute nodes, while allowing traffic that is part of existing tcp connections.\n\nAfter changing the configuration, @arvados-controller@ must be restarted for the change to take effect. When enabling, shell access will be enabled for any running containers. When disabling, access is removed immediately for any running containers, as well as any containers started subsequently. Restarting @arvados-controller@ will kill any active connections.\n\nUsage instructions for this feature are available in the \"User guide\":{{site.baseurl}}/user/debugging/container-shell-access.html.\n"
  },
  {
    "path": "doc/install/crunch2/install-compute-node-docker.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Set up a compute node with Docker\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n{% include 'notebox_begin_warning' %}\nThis page describes the requirements for a compute node in a Slurm or LSF cluster that will run containers dispatched by @crunch-dispatch-slurm@ or @arvados-dispatch-lsf@. If you are installing a cloud cluster, refer to \"Build a cloud compute node image\":{{ site.baseurl }}/install/crunch2-cloud/install-compute-node.html.\n{% include 'notebox_end' %}\n\n{% include 'notebox_begin_warning' %}\nThese instructions apply when Containers.RuntimeEngine is set to @docker@, refer to \"Set up a compute node with Singularity\":install-compute-node-singularity.html when running @singularity@.\n{% include 'notebox_end' %}\n\n# \"Introduction\":#introduction\n# \"Set up Docker\":#docker\n# \"Update fuse.conf\":#fuse\n# \"Update docker-cleaner.json\":#docker-cleaner\n# \"Install'python-arvados-fuse and crunch-run and arvados-docker-cleaner\":#install-packages\n\nh2(#introduction). Introduction\n\nThis page describes how to configure a compute node so that it can be used to run containers dispatched by Arvados on a static cluster. These steps must be performed on every compute node.\n\nh2(#docker). Set up Docker\n\nSee \"Set up Docker\":../install-docker.html\n\n{% include 'install_cuda' %}\n\n{% assign arvados_component = 'python-arvados-fuse crunch-run arvados-docker-cleaner' %}\n\n{% include 'install_compute_fuse' %}\n\n{% include 'install_docker_cleaner' %}\n\n{% include 'install_packages' %}\n\n{% assign arvados_component = 'arvados-docker-cleaner' %}\n\n{% include 'start_service' %}\n"
  },
  {
    "path": "doc/install/crunch2/install-compute-node-singularity.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Set up a compute node with Singularity\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n{% include 'notebox_begin_warning' %}\nThis page describes the requirements for a compute node in a Slurm or LSF cluster that will run containers dispatched by @crunch-dispatch-slurm@ or @arvados-dispatch-lsf@. If you are installing a cloud cluster, refer to \"Build a cloud compute node image\":{{ site.baseurl }}/install/crunch2-cloud/install-compute-node.html.\n{% include 'notebox_end' %}\n\n{% include 'notebox_begin_warning' %}\nThese instructions apply when Containers.RuntimeEngine is set to @singularity@, refer to \"Set up a compute node with Docker\":install-compute-node-docker.html when running @docker@.\n{% include 'notebox_end' %}\n\n# \"Introduction\":#introduction\n# \"Install python-arvados-fuse and crunch-run and squashfs-tools\":#install-packages\n# \"Set up Singularity\":#singularity\n# \"Singularity mksquashfs configuration\":#singularity_mksquashfs_configuration\n\nh2(#introduction). Introduction\n\nPlease refer to the \"Singularity\":{{site.baseurl}}/architecture/singularity.html documentation in the Architecture section.\n\nThis page describes how to configure a compute node so that it can be used to run containers dispatched by Arvados on a static cluster. These steps must be performed on every compute node.\n\n{% assign arvados_component = 'python-arvados-fuse crunch-run squashfs-tools' %}\n\n{% include 'install_packages' %}\n\n{% include 'install_cuda' %}\n\nh2(#singularity). Set up Singularity\n\nFollow the \"Singularity installation instructions\":https://sylabs.io/guides/latest/user-guide/quick_start.html. Note that while the latest stable version is normally expected to be compatible, Arvados is currently tested with singularity 3.10.4.\n\nMake sure @singularity@ and @mksquashfs@ are working:\n\n\n
$ singularity version\nsingularity-ce version 3.10.4-dirty\n$ mksquashfs -version\nmksquashfs version 4.4 (2019/08/29)\n[...]\n
\n\n\nThen update @Containers.RuntimeEngine@ in your cluster configuration:\n\n\n
      # Container runtime: \"docker\" (default) or \"singularity\"\n      RuntimeEngine: singularity\n
\n\n\n{% include 'singularity_mksquashfs_configuration' %}\n"
  },
  {
    "path": "doc/install/crunch2-cloud/install-compute-node.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Build a cloud compute node image\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n{% include 'notebox_begin_warning' %}\n@arvados-dispatch-cloud@ is only relevant for cloud installations. Skip this section if you are installing an on premises cluster that will spool jobs to Slurm or LSF.\n{% include 'notebox_end' %}\n\np(#introduction). This page describes how to build a compute node image that can be used to run containers dispatched by Arvados in the cloud.\n\n# \"Prerequisites\":#prerequisites\n## \"Check your distribution\":#check-distro\n## \"Create and configure an SSH keypair\":#sshkeypair\n## \"Get the Arvados source\":#git-clone\n## \"Install Ansible\":#install-ansible\n## \"Install Packer and the Ansible plugin\":#install-packer\n# \"Fully automated build with Packer and Ansible\":#building\n## \"Write Ansible settings for the compute node\":#ansible-variables\n## \"Set up Packer for your cloud\":#packer-variables\n### \"AWS\":#aws-variables\n### \"Azure\":#azure-variables\n## \"Run Packer\":#run-packer\n# \"Partially automated build with Ansible\":#ansible-build\n## \"Write Ansible settings for the compute node\":#ansible-variables-standalone\n## \"Write an Ansible inventory\":#ansible-inventory\n## \"Run Ansible\":#run-ansible\n# \"Manual build\":#requirements\n\n\n\nh2(#prerequisites). Prerequisites\n\nh3(#sshkeypair). Create and configure an SSH keypair\n\n@arvados-dispatch-cloud@ communicates with the compute nodes via SSH. To do this securely, an SSH keypair is needed. The key type must be RSA or ED25519 to work with Amazon EC2. Generate an ED25519 keypair with no passphrase:\n\n\n
~$ ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_dispatcher\nGenerating public/private ed25519 key pair.\nYour identification has been saved in /home/user/.ssh/id_dispatcher.\nYour public key has been saved in /home/user/.ssh/id_dispatcher.pub.\nThe key fingerprint is:\n[...]\n
\n\n\nAfter you do this, the contents of the private key in @~/.ssh/id_dispatcher@ need to be stored in your \"cluster configuration file\":{{ site.baseurl }}/admin/config.html under @Containers.DispatchPrivateKey@.\n\nThe public key at @~/.ssh/id_dispatcher.pub@ will need to be authorized to access instances booted from the image. Keep this file; our Ansible playbook will read it to set this up for you.\n\nh3(#git-clone). Get the Arvados source\n\nCompute node templates are only available in the Arvados source tree. Clone a copy of the Arvados source for the version of Arvados you're using in a directory convenient for you:\n\n{% include 'branchname' %}\n\n
~$ git clone --depth=1 --branch={{ branchname }} https://github.com/arvados/arvados ~/arvados\n
\n\n\nh3(#install-ansible). Install Ansible\n\n{% include 'install_ansible' header_level: 'h4' %}\n\nh3(#install-packer). Install Packer and the Ansible plugin\n\nWe provide Packer templates that can automatically create a compute instance, configure it with Ansible, shut it down, and create a cloud image from the result. \"Install Packer following their instructions.\":https://developer.hashicorp.com/packer/docs/install After you do, install Packer's Ansible provisioner by running:\n\n\n
~$ packer plugins install github.com/hashicorp/ansible\n
\n\n\nh2(#building). Fully automated build with Packer and Ansible\n\nAfter you have both tools installed, you can configure both with information about your Arvados cluster and cloud environment and then run a fully automated build.\n\nh3(#ansible-variables). Write Ansible settings for the compute node\n\nIn the @tools/compute-images@ directory of your Arvados source checkout, copy @host_config.example.yml@ to @host_config.yml@. Edit @host_config.yml@ with information about how your compute nodes should be set up following the instructions in the comments.\n\nh3(#packer-variables). Set up Packer for your cloud\n\nYou need to provide different configuration to Packer depending on which cloud you're deploying Arvados in.\n\nh4(#aws-variables). AWS\n\nInstall Packer's AWS builder by running:\n\n\n
~$ packer plugins install github.com/hashicorp/amazon\n
\n\n\nIn the @tools/compute-images@ directory of your Arvados source checkout, copy @aws_config.example.json@ to @aws_config.json@. Fill in values for the configuration settings as follows:\n\n* If you already have AWS credentials configured that Packer can use to create and manage an EC2 instance, set @aws_profile@ to the name of those credentials in your configuration. Otherwise, set @aws_access_key@ and @aws_secret_key@ with information from an API token with those permissions.\n* Set @aws_region@, @vpc_id@, and @subnet_id@ with identifiers for the network where Packer should create the EC2 instance.\n* Set @aws_source_ami@ to the AMI of the base image that should be booted and used as the base for your compute node image. Set @ssh_user@ to the name of administrator account that is used on that image.\n* Set @aws_volume_gb@ to the size of of the image you want to create in GB. The default 20 should be sufficient for most installs. You may increase this if you're using a custom source AMI with more software pre-installed.\n* Set @arvados_cluster@ to the same five-alphanumeric identifier used under @Clusters@ in your Arvados cluster configuration.\n* If you installed Ansible to a nonstandard location, set @ansible_command@ to the absolute path of @ansible-playbook@. For example, if you installed Ansible in a virtualenv at @~/ansible@, set @ansible_command@ to {% raw %}\"{{env `HOME`}}/ansible/bin/ansible-playbook\"{% endraw %}.\n\nWhen you finish writing your configuration, \"run Packer\":#run-packer.\n\nh4(#azure-variables). Azure\n\n{% comment %}\nFIXME: Incomplete\n{% endcomment %}\n\nInstall Packer's Azure builder by running:\n\n\n
~$ packer plugins install github.com/hashicorp/azure\n
\n\n\nIn the @tools/compute-images@ directory of your Arvados source checkout, copy @azure_config.example.json@ to @azure_config.json@. Fill in values for the configuration settings as follows:\n\n* The settings load credentials from Azure's standard environment variables. As long as you have these environment variables set in the shell before you run Packer, they will be loaded as normal. Alternatively, you can set them directly in the configuration file. These secrets can be generated from the Azure portal, or with the CLI using a command like:
~$ az ad sp create-for-rbac --name Packer --password ...\n
\n* Set @location@ and @resource_group@ with identifiers for where Packer should create the cloud instance.\n* Set @image_sku@ to the identifier of the base image that should be booted and used as the base for your compute node image. Set @ssh_user@ to the name of administrator account you want to use on that image.\n* Set @ssh_private_key_file@ to the path with the private key you generated earlier for the dispatcher to use. For example, {% raw %}\"{{env `HOME`}}/.ssh/id_dispatcher\"{% endraw %}.\n* Set @arvados_cluster@ to the same five-alphanumeric identifier used under @Clusters@ in your Arvados cluster configuration.\n* If you installed Ansible to a nonstandard location, set @ansible_command@ to the absolute path of @ansible-playbook@. For example, if you installed Ansible in a virtualenv at @~/ansible@, set @ansible_command@ to {% raw %}\"{{env `HOME`}}/ansible/bin/ansible-playbook\"{% endraw %}.\n\nWhen you finish writing your configuration, \"run Packer\":#run-packer.\n\nh3(#run-packer). Run Packer\n\nIn the @tools/compute-images@ directory of your Arvados source checkout, run Packer with your configuration and the template appropriate for your cloud. For example, to build an image on AWS, run:\n\n\n
arvados/tools/compute-images$ packer build -var-file=aws_config.json aws_template.json\n
\n\n\nTo build an image on Azure, replace both instances of *@aws@* with *@azure@*, and run that command.\n\n{% include 'notebox_begin_warning' %}\nIf @packer build@ fails early with @ok=0@, @changed=0@, @failed=1@, and a message like this:\n\n\n
TASK [Gathering Facts] *********************************************************\nfatal: [default]: FAILED! => {\"msg\": \"failed to transfer file to /home/you/.ansible/tmp/ansible-local-1821271ym6nh1cw/tmp2kyfkhy4 /home/admin/.ansible/tmp/ansible-tmp-1732380360.0917368-1821275-172216075852170/AnsiballZ_setup.py:\\n\\n\"}\n\nPLAY RECAP *********************************************************************\ndefault : ok=0  changed=0  unreachable=0  failed=1  skipped=0  rescued=0  ignored=0\n
\n\n\nThis might mean the version of @scp@ on your computer is trying to use new protocol features that doesn't work with the older SSH server on the cloud image. You can work around this by running:\n\n\n
$ export ANSIBLE_SCP_EXTRA_ARGS=\"'-O'\"\n
\n\n\nThen rerun your full @packer build@ command from the same shell.\n{% include 'notebox_end' %}\n\nIf the build succeeds, it will report the identifier of your image at the end of the process. For example, when you build an AWS image, it will look like this:\n\n\n
==> Builds finished. The artifacts of successful builds are:\n--> amazon-ebs: AMIs were created:\nus-east-1: ami-012345abcdef56789\n
\n\n\nThat identifier can now be set as @CloudVMs.ImageID@ in your cluster configuration. You do not need to run any other compute node build process on this page; continue to \"installing the cloud dispatcher\":install-dispatch-cloud.html.\n\nh2(#ansible-build). Partially automated build with Ansible\n\nIf Arvados does not include a template for your cloud, or you do not have permission to run Packer, you can run the Ansible playbook by itself. This can set up a base Debian or Ubuntu system with all the software and configuration necessary to do Arvados compute work. After it's done, you can manually snapshot the node and create a cloud image from it.\n\nh3(#ansible-variables-standalone). Write Ansible settings for the compute node\n\nIn the @tools/compute-images@ directory of your Arvados source checkout, copy @host_config.example.yml@ to @host_config.yml@. Edit @host_config.yml@ with information about how your compute nodes should be set up following the instructions in the comments. Note that you *must set* @arvados_cluster_id@ in this file since you are not running Packer.\n\nh3(#ansible-inventory). Write an Ansible inventory\n\nThe compute node playbook runs on a host named @default@. In the @tools/compute-images@ directory of your Arvados source checkout, write a file named @inventory.ini@ with information about how to connect to this node via SSH. It should be one line like this:\n\n\n
# Example inventory.ini for an Arvados compute node\ndefault ansible_host=192.0.2.9 ansible_user=admin\n
\n\n\n* @ansible_host@ can be the running node's hostname or IP address. You need to be able to reach this host from the system where you're running Ansible.\n* @ansible_user@ names the user account that Ansible should use for the SSH connection. It needs to have permission to use @sudo@ on the running node.\n\nYou can add other Ansible configuration options like @ansible_port@ to your inventory if needed. Refer to the \"Ansible inventory documentation\":https://docs.ansible.com/ansible/latest/inventory_guide/intro_inventory.html for details.\n\nh3(#run-ansible). Run Ansible\n\nIf you installed Ansible inside a virtualenv, activate that virtualenv now. Then, in the @tools/compute-images@ directory of your Arvados source checkout, run @ansible-playbook@ with your inventory and configuration:\n\n\n
arvados/tools/compute-images$ ansible-playbook --ask-become-pass --inventory=inventory.ini --extra-vars=@host_config.yml ../ansible/build-compute-image.yml\n
\n\n\nYou'll be prompted with @BECOME password:@. Enter the password for the @ansible_user@ you defined in the inventory to use sudo on the running node.\n\n{% include 'notebox_begin_warning' %}\nIf @ansible-playbook@ fails early with @ok=0@, @changed=0@, @failed=1@, and a message like this:\n\n\n
TASK [Gathering Facts] *********************************************************\nfatal: [default]: FAILED! => {\"msg\": \"failed to transfer file to /home/you/.ansible/tmp/ansible-local-1821271ym6nh1cw/tmp2kyfkhy4 /home/admin/.ansible/tmp/ansible-tmp-1732380360.0917368-1821275-172216075852170/AnsiballZ_setup.py:\\n\\n\"}\n\nPLAY RECAP *********************************************************************\ndefault : ok=0  changed=0  unreachable=0  failed=1  skipped=0  rescued=0  ignored=0\n
\n\n\nThis might mean the version of @scp@ on your computer is trying to use new protocol features that doesn't work with the older SSH server on the cloud image. You can work around this by running:\n\n\n
$ export ANSIBLE_SCP_EXTRA_ARGS=\"'-O'\"\n
\n\n\nThen rerun your full @ansible-playbook@ command from the same shell.\n{% include 'notebox_end' %}\n\nIf it succeeds, Ansible should report a \"PLAY RECAP\" with @failed=0@:\n\n\n
PLAY RECAP *********************************************************************\ndefault : ok=41  changed=37  unreachable=0  failed=0  skipped=5  rescued=0  ignored=0\n
\n\n\nYour node is now ready to run Arvados compute work. You can snapshot the node, create an image from it, and set that image as @CloudVMs.ImageID@ in your Arvados cluster configuration. The details of that process are cloud-specific and out of scope for this documentation. You do not need to run any other compute node build process on this page; continue to \"installing the cloud dispatcher\":install-dispatch-cloud.html.\n\nh2(#requirements). Manual build\n\nIf you cannot run Ansible, you can create a cloud instance, manually set it up to be a compute node, and then create an image from it. The details of this process depend on which distribution you use on the cloud instance and which cloud you use; all these variations are out of scope for this documentation. These are the requirements:\n\n* Except on Azure, the SSH public key you generated previously must be an authorized key for the user that Crunch is configured to use. For example, if your cluster's @CloudVMs.DriverParameters.AdminUsername@ setting is *@crunch@*, then the dispatcher's public key should be listed in ~crunch/.ssh/authorized_keys in the image. This user must also be allowed to use sudo without a password unless the user is @root@.\n  (On Azure, the dispatcher makes additional calls to automatically set up and authorize the user, making these steps unnecessary.)\n* SSH needs to be running and reachable by @arvados-dispatch-cloud@ on the port named by @CloudVMs.SSHPort@ in your cluster's configuration file (default 22).\n* Install the @python3-arvados-fuse@ package. Enable the @user_allow_other@ option in @/etc/fuse.conf@.\n* Install either \"Docker\":https://docs.docker.com/engine/install/ or \"Singularity\":https://docs.sylabs.io/guides/3.0/user-guide/installation.html as appropriate based on the @Containers.RuntimeEngine@ setting in your cluster's configuration file. If you install Docker, you may also want to install and set up the @arvados-docker-cleaner@ package to conserve space on long-running instances, but it's not strictly required.\n* All available scratch space should be made available under @/tmp@.\n"
  },
  {
    "path": "doc/install/crunch2-cloud/install-dispatch-cloud.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Install the cloud dispatcher\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n{% include 'notebox_begin_warning' %}\n@arvados-dispatch-cloud@ is only relevant for cloud installations. Skip this section if you are installing an on premises cluster that will spool jobs to Slurm or LSF.\n{% include 'notebox_end' %}\n\n# \"Introduction\":#introduction\n# \"Create compute node VM image\":#create-image\n# \"Update config.yml\":#update-config\n# \"Install arvados-dispatch-cloud\":#install-packages\n# \"Start the service\":#start-service\n# \"Restart the API server and controller\":#restart-api\n# \"Confirm working installation\":#confirm-working\n\nh2(#introduction). Introduction\n\nThe cloud dispatch service is for running containers on cloud VMs. It works with Microsoft Azure and Amazon EC2; future versions will also support Google Compute Engine.\n\nThe cloud dispatch service can run on any node that can connect to the Arvados API service, the cloud provider's API, and the SSH service on cloud VMs.  It is not resource-intensive, so you can run it on the API server node.\n\nMore detail about the internal operation of the dispatcher can be found in the \"architecture section\":{{site.baseurl}}/architecture/dispatchcloud.html.\n\nh2(#update-config). Update config.yml\n\nh3. Configure CloudVMs\n\nAdd or update the following portions of your cluster configuration file, @config.yml@. Refer to \"config.defaults.yml\":{{site.baseurl}}/admin/config.html for information about additional configuration options. The @DispatchPrivateKey@ should be the *private* key generated in \"Create a SSH keypair\":install-compute-node.html#sshkeypair .\n\n\n
    Services:\n      DispatchCloud:\n        InternalURLs:\n          \"http://localhost:9006\": {}\n    Containers:\n      CloudVMs:\n        # BootProbeCommand is a shell command that succeeds when an instance is ready for service\n        BootProbeCommand: \"sudo systemctl status docker\"\n\n        # --- driver-specific configuration goes here --- see Amazon and Azure examples below ---\n\n      DispatchPrivateKey: |\n        -----BEGIN RSA PRIVATE KEY-----\n        MIIEpQIBAAKCAQEAqXoCzcOBkFQ7w4dvXf9B++1ctgZRqEbgRYL3SstuMV4oawks\n        ttUuxJycDdsPmeYcHsKo8vsEZpN6iYsX6ZZzhkO5nEayUTU8sBjmg1ZCTo4QqKXr\n        FJ+amZ7oYMDof6QEdwl6KNDfIddL+NfBCLQTVInOAaNss7GRrxLTuTV7HcRaIUUI\n        jYg0Ibg8ZZTzQxCvFXXnjseTgmOcTv7CuuGdt91OVdoq8czG/w8TwOhymEb7mQlt\n        lXuucwQvYgfoUgcnTgpJr7j+hafp75g2wlPozp8gJ6WQ2yBWcfqL2aw7m7Ll88Nd\n        [...]\n        oFyAjVoexx0RBcH6BveTfQtJKbktP1qBO4mXo2dP0cacuZEtlAqW9Eb06Pvaw/D9\n        foktmqOY8MyctzFgXBpGTxPliGjqo8OkrOyQP2g+FL7v+Km31Xs61P8=\n        -----END RSA PRIVATE KEY-----\n    InstanceTypes:\n      x1md:\n        ProviderType: x1.medium\n        VCPUs: 8\n        RAM: 64GiB\n        IncludedScratch: 64GB\n        Price: 0.62\n      x1lg:\n        ProviderType: x1.large\n        VCPUs: 16\n        RAM: 128GiB\n        IncludedScratch: 128GB\n        Price: 1.23\n
\n\n\nh3(#GPUsupport). NVIDIA GPU support\n\nTo specify instance types with NVIDIA GPUs, the compute image must be built with CUDA support (this means setting @arvados_compute_nvidia: true@ in @host_config.yml@ when \"building the compute image\":install-compute-node.html).  You must include an additional @GPU@ section for each instance type that includes GPUs:\n\n\n
    InstanceTypes:\n      g4dn:\n        ProviderType: g4dn.xlarge\n        VCPUs: 4\n        RAM: 16GiB\n        IncludedScratch: 125GB\n        Price: 0.56\n        GPU:\n          Stack: \"cuda\"\n          DriverVersion: \"11.4\"\n          HardwareTarget: \"7.5\"\n          DeviceCount: 1\n          VRAM: 16GiB\n
\n\n\nThe @DriverVersion@ is the version of the CUDA toolkit installed in your compute image (in \"X.Y\" format, do not include the patchlevel).\n\nThe @HardwareTarget@ is the \"CUDA compute capability of the GPUs available for this instance type\":https://developer.nvidia.com/cuda-gpus in \"X.Y\" format.\n\nThe @DeviceCount@ is the number of GPU cores available for this instance type.\n\n@VRAM@ is the amount of VRAM available per GPU device.\n\nh3(#ROCmGPUsupport). AMD GPU support\n\nTo specify instance types with AMD GPUs, the compute image must be built with ROCm support (currently, installing ROCm automatically is not supported by the Arvados compute image Ansible playbook, but can be added manually after the fact).  You must include an additional @GPU@ section for each instance type that includes GPUs:\n\n\n
    InstanceTypes:\n      g4dn:\n        ProviderType: g4da.xlarge\n        VCPUs: 4\n        RAM: 16GiB\n        IncludedScratch: 125GB\n        Price: 0.56\n        GPU:\n          Stack: \"rocm\"\n          DriverVersion: \"6.2\"\n          HardwareTarget: \"gfx1100\"\n          DeviceCount: 1\n          VRAM: 16GiB\n
\n\n\n@DriverVersion@ is the version of the ROCm toolkit installed in your compute image (in \"X.Y\" format, do not include the patchlevel).\n\n@HardwareTarget@ (e.g. gfx1100) corresponds to the GPU architecture of the device.  Use @rocminfo@ to determine your hardware target.  See also \"Accelerator and GPU hardware specifications\":https://rocm.docs.amd.com/en/latest/reference/gpu-arch-specs.html (use the column \"LLVM target name\") and \"LLVM AMDGPU backend documentation\":https://llvm.org/docs/AMDGPUUsage.html .\n\n@DeviceCount@ is the number of GPU cores available for this instance type.\n\n@VRAM@ is the amount of VRAM available per GPU device.\n\nh3(#aws-ebs-autoscaler). EBS Autoscale configuration\n\nSee \"Autoscaling compute node scratch space\":install-compute-node.html#aws-ebs-autoscaler for details about compute image configuration.\n\nThe @Containers.InstanceTypes@ list should be modified so that all @AddedScratch@ lines are removed, and the @IncludedScratch@ value should be set to 5 TB. This way, the scratch space requirements will be met by all the defined instance type. For example:\n\n
    InstanceTypes:\n      c5large:\n        ProviderType: c5.large\n        VCPUs: 2\n        RAM: 4GiB\n        IncludedScratch: 5TB\n        Price: 0.085\n      m5large:\n        ProviderType: m5.large\n        VCPUs: 2\n        RAM: 8GiB\n        IncludedScratch: 5TB\n        Price: 0.096\n...\n
\n\nYou will also need to create an IAM role in AWS with these permissions:\n\n
{\n    \"Statement\": [\n        {\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"ec2:AttachVolume\",\n                \"ec2:DescribeVolumeStatus\",\n                \"ec2:DescribeVolumes\",\n                \"ec2:DescribeTags\",\n                \"ec2:ModifyInstanceAttribute\",\n                \"ec2:DescribeVolumeAttribute\",\n                \"ec2:CreateVolume\",\n                \"ec2:DeleteVolume\",\n                \"ec2:CreateTags\"\n            ],\n            \"Resource\": \"*\"\n        }\n    ]\n}\n
\n\nThen set @Containers.CloudVMs.DriverParameters.IAMInstanceProfile@ to the name of the IAM role. This will make @arvados-dispatch-cloud@ pass an IAM instance profile to the compute nodes when they start up, giving them sufficient permissions to attach and grow EBS volumes.\n\nh3. AWS Credentials for Local Keepstore on Compute node\n\nWhen @Containers.LocalKeepBlobBuffersPerVCPU@ is non-zero, the compute node will spin up a local Keepstore service for direct storage access. If Keep is backed by S3, the compute node will need to be able to access the S3 bucket.\n\nIf the AWS credentials for S3 access are configured in @config.yml@ (i.e. @Volumes.DriverParameters.AccessKeyID@ and @Volumes.DriverParameters.SecretAccessKey@), these credentials will be made available to the local Keepstore on the compute node to access S3 directly and no further configuration is necessary.\n\nIf @config.yml@ does not have @Volumes.DriverParameters.AccessKeyID@ and @Volumes.DriverParameters.SecretAccessKey@ defined, Keepstore uses instance metadata to retrieve IAM role credentials. The @CloudVMs.DriverParameters.IAMInstanceProfile@ parameter must be configured with the name of a profile whose IAM role has permission to access the S3 bucket(s). With this setup, @arvados-dispatch-cloud@ will attach the IAM role to the compute node as it is created. The instance profile name is \"often identical to the name of the IAM role\":https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#ec2-instance-profile.\n\n*If you are also using EBS Autoscale feature, the role in @IAMInstanceProfile@ must have both ec2 and s3 permissions.*\n\nh3. Minimal configuration example for Amazon EC2\n\nThe ImageID value is the compute node image that was built in \"the previous section\":install-compute-node.html#aws.\n\n\n
    Containers:\n      CloudVMs:\n        ImageID: ami-01234567890abcdef\n        Driver: ec2\n        DriverParameters:\n          # If you are not using an IAM role for authentication, specify access\n          # credentials here. Otherwise, omit or set AccessKeyID and\n          # SecretAccessKey to an empty value.\n          AccessKeyID: XXXXXXXXXXXXXXXXXXXX\n          SecretAccessKey: YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY\n\n          SecurityGroupIDs:\n          - sg-0123abcd\n          SubnetID: subnet-0123abcd\n          Region: us-east-1\n          EBSVolumeType: gp2\n          AdminUsername: arvados\n
\n\n\nh3(#IAM). Example IAM policy for cloud dispatcher\n\nExample policy for the IAM role used by the cloud dispatcher:\n\n\n
\n{\n    \"Id\": \"arvados-dispatch-cloud policy\",\n    \"Statement\": [\n        {\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                  \"ec2:CreateTags\",\n                  \"ec2:Describe*\",\n                  \"ec2:CreateImage\",\n                  \"ec2:CreateKeyPair\",\n                  \"ec2:ImportKeyPair\",\n                  \"ec2:DeleteKeyPair\",\n                  \"ec2:RunInstances\",\n                  \"ec2:StopInstances\",\n                  \"ec2:TerminateInstances\",\n                  \"ec2:ModifyInstanceAttribute\",\n                  \"ec2:CreateSecurityGroup\",\n                  \"ec2:DeleteSecurityGroup\",\n                  \"iam:PassRole\"\n            ],\n            \"Resource\": \"*\"\n        }\n    ]\n}\n
\n\n\nh3. Minimal configuration example for Azure\n\nUsing managed disks:\n\nThe ImageID value is the compute node image that was built in \"the previous section\":install-compute-node.html#azure.\n\n\n
    Containers:\n      CloudVMs:\n        ImageID: \"zzzzz-compute-v1597349873\"\n        Driver: azure\n        # (azure) managed disks: set MaxConcurrentInstanceCreateOps to 20 to avoid timeouts, cf\n        # https://docs.microsoft.com/en-us/azure/virtual-machines/linux/capture-image\n        MaxConcurrentInstanceCreateOps: 20\n        DriverParameters:\n          # Credentials.\n          SubscriptionID: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX\n          ClientID: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX\n          ClientSecret: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\n          TenantID: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX\n\n          # Data center where VMs will be allocated\n          Location: centralus\n\n          # The resource group where the VM and virtual NIC will be\n          # created.\n          ResourceGroup: zzzzz\n          NetworkResourceGroup: yyyyy   # only if different from ResourceGroup\n          Network: xxxxx\n          Subnet: xxxxx-subnet-private\n\n          # The resource group where the disk image is stored, only needs to\n          # be specified if it is different from ResourceGroup\n          ImageResourceGroup: aaaaa\n\n
\n\n\nAzure recommends using managed images. If you plan to start more than 20 VMs simultaneously, Azure recommends using a shared image gallery instead to avoid slowdowns and timeouts during the creation of the VMs.\n\nUsing an image from a shared image gallery:\n\n\n
    Containers:\n      CloudVMs:\n        ImageID: \"shared_image_gallery_image_definition_name\"\n        Driver: azure\n        DriverParameters:\n          # Credentials.\n          SubscriptionID: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX\n          ClientID: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX\n          ClientSecret: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\n          TenantID: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX\n\n          # Data center where VMs will be allocated\n          Location: centralus\n\n          # The resource group where the VM and virtual NIC will be\n          # created.\n          ResourceGroup: zzzzz\n          NetworkResourceGroup: yyyyy   # only if different from ResourceGroup\n          Network: xxxxx\n          Subnet: xxxxx-subnet-private\n\n          # The resource group where the disk image is stored, only needs to\n          # be specified if it is different from ResourceGroup\n          ImageResourceGroup: aaaaa\n\n          # (azure) shared image gallery: the name of the gallery\n          SharedImageGalleryName: \"shared_image_gallery_1\"\n          # (azure) shared image gallery: the version of the image definition\n          SharedImageGalleryImageVersion: \"0.0.1\"\n\n
\n\n\nUsing unmanaged disks (deprecated):\n\nThe ImageID value is the compute node image that was built in \"the previous section\":install-compute-node.html#azure.\n\n\n
    Containers:\n      CloudVMs:\n        ImageID: \"https://zzzzzzzz.blob.core.windows.net/system/Microsoft.Compute/Images/images/zzzzz-compute-osDisk.55555555-5555-5555-5555-555555555555.vhd\"\n        Driver: azure\n        DriverParameters:\n          # Credentials.\n          SubscriptionID: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX\n          ClientID: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX\n          ClientSecret: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\n          TenantID: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX\n\n          # Data center where VMs will be allocated\n          Location: centralus\n\n          # The resource group where the VM and virtual NIC will be\n          # created.\n          ResourceGroup: zzzzz\n          NetworkResourceGroup: yyyyy   # only if different from ResourceGroup\n          Network: xxxxx\n          Subnet: xxxxx-subnet-private\n\n          # Where to store the VM VHD blobs\n          StorageAccount: example\n          BlobContainer: vhds\n\n
\n\n\nGet the @SubscriptionID@ and @TenantID@:\n\n
\n$ az account list\n[\n  {\n    \"cloudName\": \"AzureCloud\",\n    \"id\": \"XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX\",\n    \"isDefault\": true,\n    \"name\": \"Your Subscription\",\n    \"state\": \"Enabled\",\n    \"tenantId\": \"YYYYYYYY-YYYY-YYYY-YYYYYYYY\",\n    \"user\": {\n      \"name\": \"you@example.com\",\n      \"type\": \"user\"\n    }\n  }\n]\n
\n\nYou will need to create a \"service principal\" to use as a delegated authority for API access.\n\n
$ az ad app create --display-name \"Arvados Dispatch Cloud (ClusterID)\" --homepage \"https://arvados.org\" --identifier-uris \"https://ClusterID.example.com\" --end-date 2299-12-31 --password Your_Password\n$ az ad sp create \"appId\"\n(appId is part of the response of the previous command)\n$ az role assignment create --assignee \"objectId\" --role Owner --scope /subscriptions/{subscriptionId}/\n(objectId is part of the response of the previous command)\n
\n\nNow update your @config.yml@ file:\n\n@ClientID@ is the 'appId' value.\n\n@ClientSecret@ is what was provided as Your_Password.\n\nh3. Test your configuration\n\nRun the @cloudtest@ tool to verify that your configuration works. This creates a new cloud VM, confirms that it boots correctly and accepts your configured SSH private key, and shuts it down.\n\n\n
~$ arvados-server cloudtest && echo \"OK!\"\n
\n\n\nRefer to the \"cloudtest tool documentation\":../../admin/cloudtest.html for more information.\n\n{% assign arvados_component = 'arvados-dispatch-cloud' %}\n\n{% include 'install_packages' %}\n\n{% include 'start_service' %}\n\n{% include 'restart_api' %}\n\nh2(#confirm-working). Confirm working installation\n\nOn the dispatch node, start monitoring the arvados-dispatch-cloud logs:\n\n\n
# journalctl -o cat -fu arvados-dispatch-cloud.service\n
\n\n\nIn another terminal window, use the diagnostics tool to run a simple container.\n\n\n
# arvados-client sudo diagnostics\nINFO       5: running health check (same as `arvados-server check`)\nINFO      10: getting discovery document from https://zzzzz.arvadosapi.com/discovery/v1/apis/arvados/v1/rest\n...\nINFO     160: running a container\nINFO      ... container request submitted, waiting up to 10m for container to run\n
\n\n\nAfter performing a number of other quick tests, this will submit a new container request and wait for it to finish.\n\nWhile the diagnostics tool is waiting, the @arvados-dispatch-cloud@ logs will show details about creating a cloud instance, waiting for it to be ready, and scheduling the new container on it.\n\nYou can also use the \"arvados-dispatch-cloud API\":{{site.baseurl}}/api/dispatch.html to get a list of queued and running jobs and cloud instances. Use your @ManagementToken@ to test the dispatcher's endpoint. For example, when one container is running:\n\n\n
~$ curl -sH \"Authorization: Bearer $token\" http://localhost:9006/arvados/v1/dispatch/containers\n{\n  \"items\": [\n    {\n      \"container\": {\n        \"uuid\": \"zzzzz-dz642-hdp2vpu9nq14tx0\",\n        ...\n        \"state\": \"Running\",\n        \"scheduling_parameters\": {\n          \"partitions\": null,\n          \"preemptible\": false,\n          \"max_run_time\": 0\n        },\n        \"exit_code\": 0,\n        \"runtime_status\": null,\n        \"started_at\": null,\n        \"finished_at\": null\n      },\n      \"instance_type\": {\n        \"Name\": \"Standard_D2s_v3\",\n        \"ProviderType\": \"Standard_D2s_v3\",\n        \"VCPUs\": 2,\n        \"RAM\": 8589934592,\n        \"Scratch\": 16000000000,\n        \"IncludedScratch\": 16000000000,\n        \"AddedScratch\": 0,\n        \"Price\": 0.11,\n        \"Preemptible\": false\n      }\n    }\n  ]\n}\n
\n\n\nA similar request can be made to the @http://localhost:9006/arvados/v1/dispatch/instances@ endpoint.\n\nAfter the container finishes, you can get the container record by UUID *from a shell server* to see its results:\n\n\n
shell:~$ arv get zzzzz-dz642-hdp2vpu9nq14tx0\n{\n ...\n \"exit_code\":0,\n \"log\":\"a01df2f7e5bc1c2ad59c60a837e90dc6+166\",\n \"output\":\"d41d8cd98f00b204e9800998ecf8427e+0\",\n \"state\":\"Complete\",\n ...\n}\n
\n\n\nYou can use standard Keep tools to view the container's output and logs from their corresponding fields.  For example, to see the logs from the collection referenced in the @log@ field:\n\n\n
~$ arv keep ls a01df2f7e5bc1c2ad59c60a837e90dc6+166\n./crunch-run.txt\n./stderr.txt\n./stdout.txt\n~$ arv-get a01df2f7e5bc1c2ad59c60a837e90dc6+166/stdout.txt\n2016-08-05T13:53:06.201011Z Hello, Crunch!\n
\n\n\nIf the container does not dispatch successfully, refer to the @arvados-dispatch-cloud@ logs for information about why it failed.\n"
  },
  {
    "path": "doc/install/crunch2-lsf/install-dispatch.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Install the LSF dispatcher\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n{% include 'notebox_begin_warning' %}\n@arvados-dispatch-lsf@ is only relevant for on premises clusters that will spool jobs to LSF. Skip this section if you use Slurm or if you are installing a cloud cluster.\n{% include 'notebox_end' %}\n\nh2(#overview). Overview\n\nContainers can be dispatched to an LSF cluster.  The dispatcher sends work to the cluster using LSF's @bsub@ command, so it works in a variety of LSF configurations.\n\nIn order to run containers, you must choose a user that has permission to set up FUSE mounts and run Singularity/Docker containers on each compute node.  This install guide refers to this user as the @crunch@ user.  We recommend you create this user on each compute node with the same UID and GID, and add it to the @fuse@ and @docker@ system groups to grant it the necessary permissions.  However, you can run the dispatcher under any account with sufficient permissions across the cluster.\n\nSet up all of your compute nodes with \"Docker\":../crunch2/install-compute-node-singularity.html or \"Singularity\":../crunch2/install-compute-node-docker.html.\n\n*Current limitations*:\n* Arvados container priority is not propagated to LSF job priority. This can cause inefficient use of compute resources, and even deadlock if there are fewer compute nodes than concurrent Arvados workflows.\n* Combining LSF with docker may not work, depending on LSF configuration and user/group IDs (if LSF only sets up the configured user's primary group ID when executing the crunch-run process on a compute node, it may not have permission to connect to the docker daemon).\n\nh2(#update-config). Update config.yml\n\nArvados-dispatch-lsf reads the common configuration file at @/etc/arvados/config.yml@.\n\nAdd a DispatchLSF entry to the Services section, using the hostname where @arvados-dispatch-lsf@ will run, and an available port:\n\n\n
    Services:\n      DispatchLSF:\n        InternalURLs:\n          \"http://hostname.zzzzz.arvadosapi.com:9007\": {}
\n\n\nReview the following configuration parameters and adjust as needed.\n\n{% include 'hpc_max_gateway_tunnels' %}\n\nh3(#BsubSudoUser). Containers.LSF.BsubSudoUser\n\narvados-dispatch-lsf uses @sudo@ to execute @bsub@, for example @sudo -E -u crunch bsub [...]@. This means the @crunch@ account must exist on the hosts where LSF jobs run (\"execution hosts\"), as well as on the host where you are installing the Arvados LSF dispatcher (the \"submission host\"). To use a user account other than @crunch@, configure @BsubSudoUser@:\n\n\n
    Containers:\n      LSF:\n        BsubSudoUser: lsfuser\n
\n\n\nAlternatively, you can arrange for the arvados-dispatch-lsf process to run as an unprivileged user that has a corresponding account on all compute nodes, and disable the use of @sudo@ by specifying an empty string:\n\n\n
    Containers:\n      LSF:\n        # Don't use sudo\n        BsubSudoUser: \"\"\n
\n\n\n\nh3(#BsubArgumentsList). Containers.LSF.BsubArgumentsList\n\nWhen arvados-dispatch-lsf invokes @bsub@, you can add arguments to the command by specifying @BsubArgumentsList@.  You can use this to send the jobs to specific cluster partitions or add resource requests.  Set @BsubArgumentsList@ to an array of strings.\n\nTemplate variables starting with % will be substituted as follows:\n\n%U uuid\n%C number of VCPUs\n%M memory in MB\n%T tmp in MB\n%G number of GPU devices (@runtime_constraints.cuda.device_count@)\n%W maximum job run time in minutes, suitable for use with @-W@ or @-We@ flags (see MaxRunTimeOverhead MaxRunTimeDefault below)\n\nUse %% to express a literal %. The %%J in the default will be changed to %J, which is interpreted by @bsub@ itself.\n\nFor example:\n\n\n
    Containers:\n      LSF:\n        BsubArgumentsList: [\"-o\", \"/tmp/crunch-run.%%J.out\", \"-e\", \"/tmp/crunch-run.%%J.err\", \"-J\", \"%U\", \"-n\", \"%C\", \"-D\", \"%MMB\", \"-R\", \"rusage[mem=%MMB:tmp=%TMB] span[hosts=1]\", \"-R\", \"select[mem>=%MMB]\", \"-R\", \"select[tmp>=%TMB]\", \"-R\", \"select[ncpus>=%C]\", \"-We\", \"%W\"]\n
\n\n\nNote that the default value for @BsubArgumentsList@ uses the @-o@ and @-e@ arguments to write stdout/stderr data to files in @/tmp@ on the compute nodes, which is helpful for troubleshooting installation/configuration problems. Ensure you have something in place to delete old files from @/tmp@, or adjust these arguments accordingly.\n\nh3(#BsubCUDAArguments). Containers.LSF.BsubCUDAArguments\n\nIf the container requests access to GPUs (@runtime_constraints.cuda.device_count@ of the container request is greater than zero), the command line arguments in @BsubCUDAArguments@ will be added to the command line _after_ @BsubArgumentsList@.  This should consist of the additional @bsub@ flags your site requires to schedule the job on a node with GPU support.  Set @BsubCUDAArguments@ to an array of strings.  For example:\n\n\n
    Containers:\n      LSF:\n        BsubCUDAArguments: [\"-gpu\", \"num=%G\"]\n
\n\n\nh3(#MaxRunTimeOverhead). Containers.LSF.MaxRunTimeOverhead\n\nExtra time to add to each container's @scheduling_parameters.max_run_time@ value when substituting for @%W@ in @BsubArgumentsList@, to account for time spent setting up the container image, copying output files, etc.\n\nh3(#MaxRunTimeDefault). Containers.LSF.MaxRunTimeDefault\n\nDefault @max_run_time@ value to use for containers that do not specify one in @scheduling_parameters.max_run_time@. If this is zero, and @BsubArgumentsList@ contains @\"-W\", \"%W\"@ or @\"-We\", \"%W\"@, those arguments will be dropped when submitting containers that do not specify @scheduling_parameters.max_run_time@.\n\nh3(#PollInterval). Containers.PollInterval\n\narvados-dispatch-lsf polls the API server periodically for new containers to run.  The @PollInterval@ option controls how often this poll happens.  Set this to a string of numbers suffixed with one of the time units @s@, @m@, or @h@.  For example:\n\n\n
    Containers:\n      PollInterval: 10s\n
\n\n\n\nh3(#ReserveExtraRAM). Containers.ReserveExtraRAM: Extra RAM for jobs\n\nExtra RAM to reserve (in bytes) on each LSF job submitted by Arvados, which is added to the amount specified in the container's @runtime_constraints@.  If not provided, the default value is zero.\n\nSupports suffixes @KB@, @KiB@, @MB@, @MiB@, @GB@, @GiB@, @TB@, @TiB@, @PB@, @PiB@, @EB@, @EiB@ (where @KB@ is 10[^3^], @KiB@ is 2[^10^], @MB@ is 10[^6^], @MiB@ is 2[^20^] and so forth).\n\n\n
    Containers:\n      ReserveExtraRAM: 256MiB\n
\n\n\n\nh3(#CrunchRunArgumentList). Containers.CrunchRunArgumentList: Using host networking for containers\n\nOlder Linux kernels (prior to 3.18) have bugs in network namespace handling which can lead to compute node lockups.  This by is indicated by blocked kernel tasks in \"Workqueue: netns cleanup_net\".   If you are experiencing this problem, as a workaround you can disable use of network namespaces by Docker across the cluster.  Be aware this reduces container isolation, which may be a security risk.\n\n\n
    Containers:\n      CrunchRunArgumentsList:\n        - \"-container-enable-networking=always\"\n        - \"-container-network-mode=host\"\n
\n\n\n\nh3(#InstanceTypes). InstanceTypes: Avoid submitting jobs with unsatisfiable resource constraints\n\nLSF does not provide feedback when a submitted job's RAM, CPU, or disk space constraints cannot be satisfied by any node: the job will wait in the queue indefinitely with \"pending\" status, reported by Arvados as \"queued\".\n\nAs a workaround, you can configure @InstanceTypes@ with your LSF cluster's compute node sizes. Arvados will use these sizes to determine when a container is impossible to run, and cancel it instead of submitting an LSF job.\n\nApart from detecting non-runnable containers, the configured instance types will not have any effect on scheduling.\n\n\n
    InstanceTypes:\n      most-ram:\n        VCPUs: 8\n        RAM: 640GiB\n        IncludedScratch: 640GB\n      most-cpus:\n        VCPUs: 32\n        RAM: 256GiB\n        IncludedScratch: 640GB\n      gpu:\n        VCPUs: 8\n        RAM: 256GiB\n        IncludedScratch: 640GB\n        CUDA:\n          DriverVersion: \"11.4\"\n          HardwareCapability: \"7.5\"\n          DeviceCount: 1\n
\n\n\n\n{% assign arvados_component = 'arvados-dispatch-lsf' %}\n\n{% include 'install_packages' %}\n\n{% include 'start_service' %}\n\n{% include 'restart_api' %}\n\nh2(#confirm-working). Confirm working installation\n\nOn the dispatch node, start monitoring the arvados-dispatch-lsf logs:\n\n\n
# journalctl -o cat -fu arvados-dispatch-lsf.service\n
\n\n\nIn another terminal window, use the diagnostics tool to run a simple container.\n\n\n
# arvados-client sudo diagnostics\nINFO       5: running health check (same as `arvados-server check`)\nINFO      10: getting discovery document from https://zzzzz.arvadosapi.com/discovery/v1/apis/arvados/v1/rest\n...\nINFO     160: running a container\nINFO      ... container request submitted, waiting up to 10m for container to run\n
\n\n\nAfter performing a number of other quick tests, this will submit a new container request and wait for it to finish.\n\nWhile the diagnostics tool is waiting, the @arvados-dispatch-lsf@ logs will show details about submitting an LSF job to run the container.\n"
  },
  {
    "path": "doc/install/crunch2-slurm/configure-slurm.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Configure Slurm\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n{% include 'notebox_begin_warning' %}\n@crunch-dispatch-slurm@ is only relevant for on premises clusters that will spool jobs to Slurm. Skip this section if you use LSF or if you are installing a cloud cluster.\n{% include 'notebox_end' %}\n\nContainers can be dispatched to a Slurm cluster.  The dispatcher sends work to the cluster using Slurm's @sbatch@ command, so it works in a variety of Slurm configurations.\n\nIn order to run containers, you must run the dispatcher as a user that has permission to set up FUSE mounts and run Docker containers on each compute node.  This install guide refers to this user as the @crunch@ user.  We recommend you create this user on each compute node with the same UID and GID, and add it to the @fuse@ and @docker@ system groups to grant it the necessary permissions.  However, you can run the dispatcher under any account with sufficient permissions across the cluster.\n\nWe will assume that you have Slurm and munge running.\n\nh3. Sample Slurm configuration file\n\nHere's an example @slurm.conf@ for use with Arvados:\n\n\n
\nControlMachine=ClusterID.example.com\nSlurmctldPort=6817\nSlurmdPort=6818\nAuthType=auth/munge\nStateSaveLocation=/tmp\nSlurmdSpoolDir=/tmp/slurmd\nSwitchType=switch/none\nMpiDefault=none\nSlurmctldPidFile=/var/run/slurmctld.pid\nSlurmdPidFile=/var/run/slurmd.pid\nProctrackType=proctrack/pgid\nCacheGroups=0\nReturnToService=2\nTaskPlugin=task/affinity\n#\n# TIMERS\nSlurmctldTimeout=300\nSlurmdTimeout=300\nInactiveLimit=0\nMinJobAge=300\nKillWait=30\nWaittime=0\n#\n# SCHEDULING\nSchedulerType=sched/backfill\nSchedulerPort=7321\nSelectType=select/linear\nFastSchedule=0\n#\n# LOGGING\nSlurmctldDebug=3\n#SlurmctldLogFile=\nSlurmdDebug=3\n#SlurmdLogFile=\nJobCompType=jobcomp/none\n#JobCompLoc=\nJobAcctGatherType=jobacct_gather/none\n#\n# COMPUTE NODES\nNodeName=DEFAULT\nPartitionName=DEFAULT MaxTime=INFINITE State=UP\n\nNodeName=compute[0-255]\nPartitionName=compute Nodes=compute[0-255] Default=YES Shared=YES\n
\n\n\nh3. Slurm configuration essentials\n\nWhenever you change this file, you will need to update the copy _on every compute node_ as well as the controller node, and then run @sudo scontrol reconfigure@.\n\n*@ControlMachine@* should be a DNS name that resolves to the Slurm controller (dispatch/API server). This must resolve correctly on all Slurm worker nodes as well as the controller itself. In general Slurm is very sensitive about all of the nodes being able to communicate with the controller _and one another_, all using the same DNS names.\n\n*@SelectType=select/linear@* is needed on cloud-based installations that update node sizes dynamically, but it can only schedule one container at a time on each node. On a static or homogeneous cluster, use @SelectType=select/cons_res@ with @SelectTypeParameters=CR_CPU_Memory@ instead to enable node sharing.\n\n*@NodeName=compute[0-255]@* establishes that the hostnames of the worker nodes will be compute0, compute1, etc. through compute255.\n* There are several ways to compress sequences of names, like @compute[0-9,80,100-110]@. See the \"hostlist\" discussion in the @slurm.conf(5)@ and @scontrol(1)@ man pages for more information.\n* It is not necessary for all of the nodes listed here to be alive in order for Slurm to work, although you should make sure the DNS entries exist. It is easiest to define lots of hostnames up front, assigning them to real nodes and updating your DNS records as the nodes appear. This minimizes the frequency of @slurm.conf@ updates and use of @scontrol reconfigure@.\n\nEach hostname in @slurm.conf@ must also resolve correctly on all Slurm worker nodes as well as the controller itself. Furthermore, the hostnames used in the configuration file must match the hostnames reported by @hostname@ or @hostname -s@ on the nodes themselves. This applies to the ControlMachine as well as the worker nodes.\n\nFor example:\n* In @slurm.conf@ on control and worker nodes: @ControlMachine=ClusterID.example.com@\n* In @slurm.conf@ on control and worker nodes: @NodeName=compute[0-255]@\n* In @/etc/resolv.conf@ on control and worker nodes: @search ClusterID.example.com@\n* On the control node: @hostname@ reports @ClusterID.example.com@\n* On worker node 123: @hostname@ reports @compute123.ClusterID.example.com@\n\nh3. Automatic hostname assignment\n\nThe API server will choose an unused hostname from the set given in @application.yml@, which defaults to @compute[0-255]@.\n\nIf it is not feasible to give your compute nodes hostnames like compute0, compute1, etc., you can accommodate other naming schemes with a bit of extra configuration.\n\nIf you want Arvados to assign names to your nodes with a different consecutive numeric series like @{worker1-0000, worker1-0001, worker1-0002}@, add an entry to @application.yml@; see @/var/www/arvados-api/current/config/application.default.yml@ for details. Example:\n* In @application.yml@: assign_node_hostname: worker1-%04d\n* In @slurm.conf@: NodeName=worker1-[0000-0255]\n\nIf your worker hostnames are already assigned by other means, and the full set of names is known in advance, have your worker node bootstrapping script send its current hostname, rather than expect Arvados to assign one.\n* In @application.yml@: assign_node_hostname: false\n* In @slurm.conf@: NodeName=alice,bob,clay,darlene\n\nIf your worker hostnames are already assigned by other means, but the full set of names is _not_ known in advance, you can use the @slurm.conf@ and @application.yml@ settings in the previous example, but you must also update @slurm.conf@ (both on the controller and on all worker nodes) and run @sudo scontrol reconfigure@ whenever a new node comes online.\n"
  },
  {
    "path": "doc/install/crunch2-slurm/install-dispatch.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Install the Slurm dispatcher\n\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n{% include 'notebox_begin_warning' %}\n@crunch-dispatch-slurm@ is only relevant for on premises clusters that will spool jobs to Slurm. Skip this section if you use LSF or if you are installing a cloud cluster.\n{% include 'notebox_end' %}\n\n# \"Introduction\":#introduction\n# \"Update config.yml\":#update-config\n# \"Install crunch-dispatch-slurm\":#install-packages\n# \"Start the service\":#start-service\n# \"Restart the API server and controller\":#restart-api\n\nh2(#introduction). Introduction\n\nThis assumes you already have a Slurm cluster, and have set up all of your compute nodes with \"Docker\":../crunch2/install-compute-node-docker.html or \"Singularity\":../crunch2/install-compute-node-singularity.html.  Slurm packages are available on all distributions supported by Arvados. Please see your distribution package repositories. For information on installing Slurm from source, see \"this install guide\":https://slurm.schedmd.com/quickstart_admin.html\n\nThe Arvados Slurm dispatcher can run on any node that can submit requests to both the Arvados API server and the Slurm controller (via @sbatch@).  It is not resource-intensive, so you can run it on the API server node.\n\nh2(#update-config). Update config.yml\n\nCrunch-dispatch-slurm reads the common configuration file at @/etc/arvados/config.yml@.\n\nAdd a DispatchSLURM entry to the Services section, using the hostname where @crunch-dispatch-slurm@ will run, and an available port:\n\n\n
    Services:\n      DispatchSLURM:\n        InternalURLs:\n          \"http://hostname.zzzzz.arvadosapi.com:9007\": {}
\n\n\nThe following configuration parameters are optional.\n\n{% include 'hpc_max_gateway_tunnels' %}\n\nh3(#PollPeriod). Containers.PollInterval\n\ncrunch-dispatch-slurm polls the API server periodically for new containers to run.  The @PollInterval@ option controls how often this poll happens.  Set this to a string of numbers suffixed with one of the time units @ns@, @us@, @ms@, @s@, @m@, or @h@.  For example:\n\n\n
    Containers:\n      PollInterval: 3m30s\n
\n\n\nh3(#ReserveExtraRAM). Containers.ReserveExtraRAM: Extra RAM for jobs\n\nExtra RAM to reserve (in bytes) on each Slurm job submitted by Arvados, which is added to the amount specified in the container's @runtime_constraints@.  If not provided, the default value is zero.  Helpful when using @-cgroup-parent-subsystem@, where @crunch-run@ and @arv-mount@ share the control group memory limit with the user process.  In this situation, at least 256MiB is recommended to accommodate each container's @crunch-run@ and @arv-mount@ processes.\n\nSupports suffixes @KB@, @KiB@, @MB@, @MiB@, @GB@, @GiB@, @TB@, @TiB@, @PB@, @PiB@, @EB@, @EiB@ (where @KB@ is 10[^3^], @KiB@ is 2[^10^], @MB@ is 10[^6^], @MiB@ is 2[^20^] and so forth).\n\n\n
    Containers:\n      ReserveExtraRAM: 256MiB\n
\n\n\nh3(#MinRetryPeriod). Containers.MinRetryPeriod: Rate-limit repeated attempts to start containers\n\nIf Slurm is unable to run a container, the dispatcher will submit it again after the next PollPeriod. If PollPeriod is very short, this can be excessive. If MinRetryPeriod is set, the dispatcher will avoid submitting the same container to Slurm more than once in the given time span.\n\n\n
    Containers:\n      MinRetryPeriod: 30s\n
\n\n\nh3(#KeepServiceURIs). Containers.Slurm.SbatchEnvironmentVariables\n\nSome Arvados installations run a local keepstore on each compute node to handle all Keep traffic.  To override Keep service discovery and access the local keep server instead of the global servers, set ARVADOS_KEEP_SERVICES in SbatchEnvironmentVariables:\n\n\n
    Containers:\n      SLURM:\n        SbatchEnvironmentVariables:\n          ARVADOS_KEEP_SERVICES: \"http://127.0.0.1:25107\"\n
\n\n\nh3(#PrioritySpread). Containers.Slurm.PrioritySpread\n\ncrunch-dispatch-slurm adjusts the \"nice\" values of its Slurm jobs to ensure containers are prioritized correctly relative to one another. This option tunes the adjustment mechanism.\n* If non-Arvados jobs run on your Slurm cluster, and your Arvados containers are waiting too long in the Slurm queue because their \"nice\" values are too high for them to compete with other Slurm jobs, you should use a smaller PrioritySpread value.\n* If you have an older Slurm system that limits nice values to 10000, a smaller @PrioritySpread@ can help avoid reaching that limit.\n* In other cases, a larger value is beneficial because it reduces the total number of adjustments made by executing @scontrol@.\n\nThe smallest usable value is @1@. The default value of @10@ is used if this option is zero or negative. Example:\n\n\n
    Containers:\n      SLURM:\n        PrioritySpread: 1000
\n\n\nh3(#SbatchArguments). Containers.Slurm.SbatchArgumentsList\n\nWhen crunch-dispatch-slurm invokes @sbatch@, you can add arguments to the command by specifying @SbatchArguments@.  You can use this to send the jobs to specific cluster partitions or add resource requests.  Set @SbatchArguments@ to an array of strings.  For example:\n\n\n
    Containers:\n      SLURM:\n        SbatchArgumentsList:\n          - \"--partition=PartitionName\"\n
\n\n\nNote: If an argument is supplied multiple times, @slurm@ uses the value of the last occurrence of the argument on the command line.  Arguments specified through Arvados are added after the arguments listed in SbatchArguments.  This means, for example, an Arvados container with that specifies @partitions@ in @scheduling_parameter@ will override an occurrence of @--partition@ in SbatchArguments.  As a result, for container parameters that can be specified through Arvados, SbatchArguments can be used to specify defaults but not enforce specific policy.\n\nh3(#CrunchRunCommand-cgroups). Containers.CrunchRunArgumentList: Dispatch to Slurm cgroups\n\nIf your Slurm cluster uses the @task/cgroup@ TaskPlugin, you can configure Crunch's Docker containers to be dispatched inside Slurm's cgroups.  This provides consistent enforcement of resource constraints.  To do this, use a crunch-dispatch-slurm configuration like the following:\n\n\n
    Containers:\n      CrunchRunArgumentsList:\n        - \"-cgroup-parent-subsystem=memory\"\n
\n\n\nWhen using cgroups v1, the choice of subsystem (\"memory\" in this example) must correspond to one of the resource types enabled in Slurm's @cgroup.conf@.  The specified subsystem is singled out only to let Crunch determine the name of the cgroup provided by Slurm.  Limits for other resource types will also be respected.\n\nWhen doing this, you should also set \"ReserveExtraRAM\":#ReserveExtraRAM .\n\n{% include 'notebox_begin' %}\n\nSome versions of Docker (at least 1.9), when run under systemd, require the cgroup parent to be specified as a systemd slice.  This causes an error when specifying a cgroup parent created outside systemd, such as those created by Slurm.\n\nYou can work around this issue by disabling the Docker daemon's systemd integration.  This makes it more difficult to manage Docker services with systemd, but Crunch does not require that functionality, and it will be able to use Slurm's cgroups as container parents.  To do this, configure the Docker daemon on all compute nodes to run with the option @--exec-opt native.cgroupdriver=cgroupfs@.\n\n{% include 'notebox_end' %}\n\nh3(#CrunchRunCommand-network). Containers.CrunchRunArgumentList: Using host networking for containers\n\nOlder Linux kernels (prior to 3.18) have bugs in network namespace handling which can lead to compute node lockups.  This by is indicated by blocked kernel tasks in \"Workqueue: netns cleanup_net\".   If you are experiencing this problem, as a workaround you can disable use of network namespaces by Docker across the cluster.  Be aware this reduces container isolation, which may be a security risk.\n\n\n
    Containers:\n      CrunchRunArgumentsList:\n        - \"-container-enable-networking=always\"\n        - \"-container-network-mode=host\"\n
\n\n\n{% assign arvados_component = 'crunch-dispatch-slurm' %}\n\n{% include 'install_packages' %}\n\n{% include 'start_service' %}\n\n{% include 'restart_api' %}\n"
  },
  {
    "path": "doc/install/crunch2-slurm/install-test.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Test Slurm dispatch\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n{% include 'notebox_begin_warning' %}\n@crunch-dispatch-slurm@ is only relevant for on premises clusters that will spool jobs to Slurm. Skip this section if you use LSF or if you are installing a cloud cluster.\n{% include 'notebox_end' %}\n\nh2. Test compute node setup\n\nYou should now be able to submit Slurm jobs that run in Docker containers.  On the node where you're running the dispatcher, you can test this by running:\n\n\n
~$ sudo -u crunch srun -N1 docker run busybox echo OK\n
\n\n\nIf it works, this command should print @OK@ (it may also show some status messages from Slurm and/or Docker).  If it does not print @OK@, double-check your compute node setup, and that the @crunch@ user can submit Slurm jobs.\n\nh2. Test the dispatcher\n\nMake sure all of your compute nodes are set up with \"Docker\":../crunch2/install-compute-node-docker.html or \"Singularity\":../crunch2/install-compute-node-singularity.html.\n\nOn the dispatch node, start monitoring the crunch-dispatch-slurm logs:\n\n\n
# journalctl -o cat -fu crunch-dispatch-slurm.service\n
\n\n\nIn another terminal window, use the diagnostics tool to run a simple container.\n\n\n
# arvados-client sudo diagnostics\nINFO       5: running health check (same as `arvados-server check`)\nINFO      10: getting discovery document from https://zzzzz.arvadosapi.com/discovery/v1/apis/arvados/v1/rest\n...\nINFO     160: running a container\nINFO      ... container request submitted, waiting up to 10m for container to run\n
\n\n\nOnce @crunch-dispatch-slurm@ polls the API server for new containers to run, you should see it dispatch the new container.  It will log messages like:\n\n\n
2016/08/05 13:52:54 Monitoring container zzzzz-dz642-hdp2vpu9nq14tx0 started\n2016/08/05 13:53:04 About to submit queued container zzzzz-dz642-hdp2vpu9nq14tx0\n2016/08/05 13:53:04 sbatch succeeded: Submitted batch job 8102\n
\n\n\nBefore the container finishes, Slurm's @squeue@ command will show the new job in the list of queued and running jobs.  For example, you might see:\n\n\n
~$ squeue --long\nFri Aug  5 13:57:50 2016\n  JOBID PARTITION     NAME     USER    STATE       TIME TIMELIMIT  NODES NODELIST(REASON)\n   8103   compute zzzzz-dz   crunch  RUNNING       1:56 UNLIMITED      1 compute0\n
\n\n\nThe job's name corresponds to the container's UUID.  You can get more information about it by running, e.g., scontrol show job Name=UUID.\n\nWhen the container finishes, the dispatcher will log that, with the final result:\n\n\n
2016/08/05 13:53:14 Container zzzzz-dz642-hdp2vpu9nq14tx0 now in state \"Complete\" with locked_by_uuid \"\"\n2016/08/05 13:53:14 Monitoring container zzzzz-dz642-hdp2vpu9nq14tx0 finished\n
\n\n\nAfter the container finishes, you can get the container record by UUID *from a shell server* to see its results:\n\n\n
shell:~$ arv get zzzzz-dz642-hdp2vpu9nq14tx0\n{\n ...\n \"exit_code\":0,\n \"log\":\"a01df2f7e5bc1c2ad59c60a837e90dc6+166\",\n \"output\":\"d41d8cd98f00b204e9800998ecf8427e+0\",\n \"state\":\"Complete\",\n ...\n}\n
\n\n\nYou can use standard Keep tools to view the container's output and logs from their corresponding fields.  For example, to see the logs from the collection referenced in the @log@ field:\n\n\n
~$ arv keep ls a01df2f7e5bc1c2ad59c60a837e90dc6+166\n./crunch-run.txt\n./stderr.txt\n./stdout.txt\n~$ arv-get a01df2f7e5bc1c2ad59c60a837e90dc6+166/stdout.txt\n2016-08-05T13:53:06.201011Z Hello, Crunch!\n
\n\n\nIf the container does not dispatch successfully, refer to the @crunch-dispatch-slurm@ logs for information about why it failed.\n"
  },
  {
    "path": "doc/install/diagnostics.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Cluster diagnostics tool\n\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThe @diagnostics@ subcommand of @arvados-client@ performs a variety of checks to help confirm that your Arvados installation has been properly configured.  It is extremely helpful to validate that your install is successful.\n\nDepending on where you are running the installer, you need to provide @-internal-client@ or @-external-client@.\n\n* If you are running the diagnostics from one of the Arvados machines inside the private network, you want @-internal-client@.\n* If you running the diagnostics from your workstation outside of the private network, you should use @-external-client@.\n\nHere is an example of it in action:\n\n\n
$ ARVADOS_API_HOST=ClusterID.example.com ARVADOS_API_TOKEN=YourSystemTokenHere arvados-client diagnostics -external-client\nINFO      10: getting discovery document from https://ClusterID.example.com/discovery/v1/apis/arvados/v1/rest\nINFO      20: getting exported config from https://ClusterID.example.com/arvados/v1/config\nINFO      30: getting current user record\nINFO      40: connecting to service endpoint https://keep.ClusterID.example.com/\nINFO      41: connecting to service endpoint https://*.collections.ClusterID.example.com/\nINFO      42: connecting to service endpoint https://download.ClusterID.example.com/\nINFO      43: connecting to service endpoint wss://ws.ClusterID.example.com/websocket\nINFO      44: connecting to service endpoint https://workbench.ClusterID.example.com/\nINFO      45: connecting to service endpoint https://workbench2.ClusterID.example.com/\nINFO      50: checking CORS headers at https://ClusterID.example.com/\nINFO      51: checking CORS headers at https://keep.ClusterID.example.com/d41d8cd98f00b204e9800998ecf8427e+0\nINFO      52: checking CORS headers at https://download.ClusterID.example.com/\nINFO      60: checking internal/external client detection\nINFO      61: reading+writing via keep service at https://keep.ClusterID.example.com:443/\nINFO      80: finding/creating \"scratch area for diagnostics\" project\nINFO      90: creating temporary collection\nINFO     100: uploading file via webdav\nINFO     110: checking WebDAV ExternalURL wildcard (https://*.collections.ClusterID.example.com/)\nINFO     120: downloading from webdav (https://d41d8cd98f00b204e9800998ecf8427e-0.collections.ClusterID.example.com/foo)\nINFO     121: downloading from webdav (https://d41d8cd98f00b204e9800998ecf8427e-0.collections.ClusterID.example.com/sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412.tar)\nINFO     122: downloading from webdav (https://download.ClusterID.example.com/c=d41d8cd98f00b204e9800998ecf8427e+0/_/foo)\nINFO     123: downloading from webdav (https://download.ClusterID.example.com/c=d41d8cd98f00b204e9800998ecf8427e+0/_/sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412.tar)\nINFO     124: downloading from webdav (https://a15a27cbc1c7d2d4a0d9e02529aaec7e-128.collections.ClusterID.example.com/sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412.tar)\nINFO     125: downloading from webdav (https://download.ClusterID.example.com/c=ce8i5-4zz18-bkfvq2skqqf78xd/_/sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412.tar)\nINFO     130: getting list of virtual machines\nINFO     140: getting workbench1 webshell page\nINFO     150: connecting to webshell service\nINFO     160: running a container\nINFO      ... container request submitted, waiting up to 10m for container to run\nINFO    9990: deleting temporary collection\nINFO    --- no errors ---\n
\n\n"
  },
  {
    "path": "doc/install/index.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Installation options\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n{% include 'notebox_begin' %}\nThis section is about installing an Arvados cluster.  If you are just looking to install Arvados client tools and libraries, \"go to the SDK section.\":{{site.baseurl}}/sdk/\n{% include 'notebox_end' %}\n\nArvados components run on supported GNU/Linux distributions. The Arvados elastic compute management layer supports Amazon Web Services (AWS) and Microsoft Azure cloud platforms as well as on-premises installs using SLURM and IBM Spectrum LSF.  The Arvados storage layer supports filesystem storage (including NFS, such as IBM GPFS), Azure blob storage, Amazon S3, and systems that offer an S3-compatible API such as Ceph Object Gateway and NetApp StorageGRID.\n\n\"Arvados is Free Software\":{{site.baseurl}}/user/copying/copying.html and self-install installations are not limited in any way.  Commercial support and development are also available from \"Curii Corporation.\":https://www.curii.com/\n\nArvados components can be installed and configured in a number of different ways.\n\n
\ntable(table table-bordered table-condensed).\n||_. Setup difficulty|_. Arvados Evaluation|_. Development|_. Production Data Management|_. Production Workflows|\n|\"Single-host install\":install-single-host.html|Easy|yes|limited|limited|limited|\n|\"Multi-host install\":install-multi-host.html|Moderate|yes|yes|yes|yes|\n|\"Manual installation\":install-manual-prerequisites.html|Difficult|yes|yes|yes|yes|\n|\"Cluster Operation Subscription supported by Curii\":https://curii.com|N/A ^1^|yes|yes|yes|yes|\n
\n\n^1^ No user installation necessary.  Curii engineers will install and configure Arvados in your own infrastructure.\n"
  },
  {
    "path": "doc/install/install-api-server.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Install API server and Controller\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n# \"Introduction\":#introduction\n# \"Install dependencies\":#dependencies\n# \"Set up database\":#database-setup\n# \"Update config.yml\":#update-config\n# \"Update nginx configuration\":#update-nginx\n# \"Install arvados-api-server and arvados-controller\":#install-packages\n# \"Confirm working installation\":#confirm-working\n\nh2(#introduction). Introduction\n\nThe Arvados core API server consists of four services: PostgreSQL, Arvados Rails API, Arvados Controller, and Nginx.\n\nHere is a simplified diagram showing the relationship between the core services.  Client requests arrive at the public-facing Nginx reverse proxy.  The request is forwarded to Arvados controller.  The controller is able handle some requests itself, the rest are forwarded to the Arvados Rails API.  The Rails API server implements the majority of business logic, communicating with the PostgreSQL database to fetch data and make transactional updates.  All services are stateless, except the PostgreSQL database.  This guide assumes all of these services will be installed on the same node, but it is possible to install these services across multiple nodes.\n\n!(full-width){{site.baseurl}}/images/proxy-chain.svg!\n\nh2(#dependencies). Install dependencies\n\n# \"Install PostgreSQL\":install-postgresql.html\n# \"Install nginx\":nginx.html\n\nh2(#database-setup). Set up database\n\n{% assign service_role = \"arvados\" %}\n{% assign service_database = \"arvados_production\" %}\n{% assign use_contrib = true %}\n{% include 'install_postgres_database' %}\n\nh2(#update-config). Update config.yml\n\nStarting from an \"empty config.yml file,\":config.html#empty add the following configuration keys.\n\nh3. Tokens\n\n\n
    SystemRootToken: \"$system_root_token\"\n    ManagementToken: \"$management_token\"\n    Collections:\n      BlobSigningKey: \"$blob_signing_key\"\n
\n\n\nThese secret tokens are used to authenticate messages between Arvados components.\n* @SystemRootToken@ is used by Arvados system services to authenticate as the system (root) user when communicating with the API server.\n* @ManagementToken@ is used to authenticate access to system metrics.\n* @Collections.BlobSigningKey@ is used to control access to Keep blocks.\n\nEach token should be a string of at least 50 alphanumeric characters. You can generate a suitable token with the following command:\n\n\n
~$ tr -dc 0-9a-zA-Z </dev/urandom | head -c50 ; echo\n
\n\n\nh3. PostgreSQL.Connection\n\n\n
    PostgreSQL:\n      Connection:\n        host: localhost\n        user: arvados\n        password: $postgres_password\n        dbname: arvados_production\n
\n\n\nReplace the @$postgres_password@ placeholder with the password you generated during \"database setup\":#database-setup.\n\nh3. Services\n\n\n
    Services:\n      Controller:\n        ExternalURL: \"https://ClusterID.example.com\"\n        InternalURLs:\n          \"http://localhost:8003\": {}\n      RailsAPI:\n        # Does not have an ExternalURL\n        InternalURLs:\n          \"http://localhost:8004\": {}\n      ContainerWebServices:\n        # Does not have InternalURLs\n        ExternalURL: \"https://*.containers.ClusterID.example.com\"\n
\n\n\nReplace @ClusterID.example.com@ with the hostname that you previously selected for the API server.\n\nThe @Services@ section of the configuration helps Arvados components contact one another (service discovery).  Each service has one or more @InternalURLs@ and an @ExternalURL@.  The @InternalURLs@ describe where the service runs, and how the Nginx reverse proxy will connect to it.  The @ExternalURL@ is how external clients contact the service.\n\nh2(#update-nginx). Update nginx configuration\n\nUse a text editor to create a new file @/etc/nginx/conf.d/arvados-controller.conf@ with the following configuration.  Options that need attention are marked in red.\n\n\n
proxy_http_version 1.1;\n\n# When Keep clients request a list of Keep services from the API\n# server, use the origin IP address to determine if the request came\n# from the internal subnet or it is an external client.  This sets the\n# $external_client variable which in turn is used to set the\n# X-External-Client header.\n#\n# The API server uses this header to choose whether to respond to a\n# \"available keep services\" request with either a list of internal keep\n# servers (0) or with the keepproxy (1).\n#\n# Following the example here, update the 10.20.30.0/24 netmask\n# to match your private subnet.\n# Update 1.2.3.4 and add lines as necessary with the public IP\n# address of all servers that can also access the private network to\n# ensure they are not considered 'external'.\n\ngeo $external_client {\n  default        1;\n  127.0.0.0/24   0;\n  10.20.30.0/24  0;\n  1.2.3.4/32     0;\n}\n\n# This is the port where nginx expects to contact arvados-controller.\nupstream controller {\n  server     localhost:8003  fail_timeout=10s;\n}\n\nserver {\n  # This configures the public https port that clients will actually connect to,\n  # the request is reverse proxied to the upstream 'controller'\n\n  listen       443 ssl;\n  server_name  ClusterID.example.com\n               *.containers.ClusterID.example.com;\n\n  ## If a wildcard name like *.containers.ClusterID.example.com is not\n  ## available, and Services.ContainerWebServices.ExternalPortMin and\n  ## ExternalPortMax are configured instead, then the \"listen\" and\n  ## \"server_name\" directives should be adjusted accordingly.  Example:\n  #\n  # listen       443 ssl;\n  # listen       2000-2999 ssl;\n  # server_name  ClusterID.example.com\n  #              containers.ClusterID.example.com;\n  #\n  ## The number of ports in the range (1000 in this example) should be\n  ## added to the worker_connections setting in the events section of\n  ## your Nginx configuration (default 512).  If the system-supplied\n  ## RLIMIT_NOFILE value is low (some systems default to 1024), the\n  ## worker_rlimit_nofile setting in the main section should also be\n  ## increased by the same amount.\n  #\n  # events { worker_connections: 1512; }\n  # worker_rlimit_nofile: 2024;\n\n  ssl_certificate     /YOUR/PATH/TO/cert.pem;\n  ssl_certificate_key /YOUR/PATH/TO/cert.key;\n\n  # Refer to the comment about this setting in the passenger (arvados\n  # api server) section of your Nginx configuration.\n  client_max_body_size 128m;\n\n  location / {\n    proxy_pass               http://controller;\n    proxy_redirect           off;\n    proxy_connect_timeout    90s;\n    proxy_read_timeout       300s;\n    proxy_max_temp_file_size 0;\n    proxy_request_buffering  off;\n    proxy_buffering          off;\n    proxy_http_version       1.1;\n\n    proxy_set_header      Host              $http_host;\n    proxy_set_header      Upgrade           $http_upgrade;\n    proxy_set_header      Connection        \"upgrade\";\n    proxy_set_header      X-External-Client $external_client;\n    proxy_set_header      X-Forwarded-For   $proxy_add_x_forwarded_for;\n    proxy_set_header      X-Forwarded-Proto https;\n    proxy_set_header      X-Real-IP         $remote_addr;\n  }\n}\n
\n\n\nh2(#install-packages). Install arvados-api-server and arvados-controller\n\nh3. Red Hat, AlmaLinux, and Rocky Linux 8\n\n\n
# dnf install --enablerepo=powertools arvados-api-server arvados-controller\n
\n\n\nh3. Red Hat, AlmaLinux, and Rocky Linux 9 or 10\n\n\n
# dnf install --enablerepo=devel arvados-api-server arvados-controller\n
\n\n\nh3. Debian and Ubuntu\n\n\n
# apt install arvados-api-server arvados-controller\n
\n\n\nh3(#railsapi-config). Configure Rails API server\n\nBy default, the Rails API server is configured to listen on @localhost:8004@, matching the example cluster configuration above. If you need to change this, edit the @arvados-railsapi.service@ definition to redefine the @PASSENGER_ADDRESS@ and @PASSENGER_PORT@ environment variables, like this:\n\n\n
# systemctl edit arvados-railsapi.service\n### Editing /etc/systemd/system/arvados-railsapi.service.d/override.conf\n### Anything between here and the comment below will become the new contents of the file\n[Service]\nEnvironment=PASSENGER_ADDRESS=0.0.0.0\nEnvironment=PASSENGER_PORT=8040\n### Lines below this comment will be discarded\n[...]\n
\n\n\nYou can similarly define other Passenger settings if desired. The \"Passenger Standalone reference\":https://www.phusionpassenger.com/library/config/standalone/reference/ documents all the available settings.\n\n{% assign arvados_component = 'arvados-railsapi arvados-controller' %}\n\n{% include 'start_service' %}\n\nh2(#confirm-working). Confirm working installation\n\nWe recommend using the \"Cluster diagnostics tool.\":diagnostics.html  The first few tests (10, 20, 30) will succeed if you have a working API server and controller.  Of course, tests for services that you have not yet installed and configured will fail.\n\nHere are some other checks you can perform manually.\n\nh3. Confirm working controller\n\n
$ curl https://ClusterID.example.com/arvados/v1/config\n
\n\nh3. Confirm working Rails API server\n\n
$ curl https://ClusterID.example.com/discovery/v1/apis/arvados/v1/rest\n
\n\nh3. Confirm that you can use the system root token to act as the system root user\n\n
$ curl -H \"Authorization: Bearer $system_root_token\" https://ClusterID.example.com/arvados/v1/users/current\n
\n\nh3. Troubleshooting\n\nIf you are getting TLS errors, make sure the @ssl_certificate@ directive in your nginx configuration has the \"full certificate chain\":http://nginx.org/en/docs/http/configuring_https_servers.html#chains.\n\nLogs can be found in @/var/www/arvados-api/current/log/production.log@ and using @journalctl -u arvados-controller@. See also the admin page on \"Logging\":{{site.baseurl}}/admin/logging.html.\n"
  },
  {
    "path": "doc/install/install-docker.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Set up Docker\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n{% include 'install_compute_docker' %}\n"
  },
  {
    "path": "doc/install/install-keep-balance.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Install Keep-balance\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n# \"Introduction\":#introduction\n# \"Update config.yml\":#update-config\n# \"Install keep-balance package\":#install-packages\n# \"Start the service\":#start-service\n\nh2(#introduction). Introduction\n\nKeep-balance deletes unreferenced and overreplicated blocks from Keep servers, makes additional copies of underreplicated blocks, and moves blocks into optimal locations as needed (e.g., after adding new servers). See \"Balancing Keep servers\":{{site.baseurl}}/admin/keep-balance.html for usage details.\n\nKeep-balance can be installed anywhere with network access to Keep services, arvados-controller, and PostgreSQL. Typically it runs on the same host as keepproxy.\n\n*A cluster should have only one instance of keep-balance running at a time.*\n\n{% include 'notebox_begin' %}\n\nIf you are installing keep-balance on an existing system with valuable data, you can run keep-balance in \"dry run\" mode first and review its logs as a precaution. To do this, set the @Collections.BalancePullLimit@ and @Collections.BalanceTrashLimit@ configuration entries to zero.\n\n{% include 'notebox_end' %}\n\nh2(#update-config). Update the cluster config\n\nEdit the cluster config at @config.yml@ and set @Services.Keepbalance.InternalURLs@.  This port is only used to publish metrics.\n\n\n
    Services:\n      Keepbalance:\n        InternalURLs:\n          \"http://keep.ClusterID.example.com:9005/\": {}\n
\n\n\nEnsure your cluster configuration has @Collections.BlobTrash: true@ (this is the default).\n\n\n
# arvados-server config-dump | grep BlobTrash:\n      BlobTrash: true\n
\n\n\nIf BlobTrash is false, unneeded blocks will be counted and logged by keep-balance, but they will not be deleted.\n\n{% assign arvados_component = 'keep-balance' %}\n\n{% include 'install_packages' %}\n\n{% include 'start_service' %}\n"
  },
  {
    "path": "doc/install/install-keep-web.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Install Keep-web server\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n# \"Introduction\":#introduction\n# \"Configure DNS\":#introduction\n# \"Configure anonymous user token\":#update-config\n# \"Update nginx configuration\":#update-nginx\n# \"Install keep-web package\":#install-packages\n# \"Start the service\":#start-service\n# \"Restart the API server and controller\":#restart-api\n# \"Confirm working installation\":#confirm-working\n\nh2(#introduction). Introduction\n\nThe Keep-web server provides read/write access to files stored in Keep using WebDAV and S3 protocols.  This makes it easy to access files in Keep from a browser, or mount Keep as a network folder using WebDAV support in various operating systems. It serves public data to unauthenticated clients, and serves private data to clients that supply Arvados API tokens. It can be installed anywhere with access to Keep services, controller, and the PostgreSQL server. It is typically installed behind a web proxy that provides TLS support. See the \"godoc page\":https://pkg.go.dev/git.arvados.org/arvados.git/services/keep-web for more detail.\n\nh2(#dns). Configure DNS\n\nIt is important to properly configure the keep-web service to so it does not open up cross-site-scripting (XSS) attacks.  A HTML file can be stored in collection.  If an attacker causes a victim to visit that HTML file through Workbench, it will be rendered by the browser.  If all collections are served at the same domain, the browser will consider collections as coming from the same origin and thus have access to the same browsing data (such as API token), enabling malicious Javascript in the HTML file to access Arvados as the victim.\n\nThere are two approaches to mitigate this.\n\n# The service can tell the browser that all files should go to download instead of in-browser preview, except in situations where an attacker is unlikely to be able to gain access to anything they didn't already have access to.\n# Each collection served by @keep-web@ is served on its own virtual host.  This allows for file with executable content to be displayed in-browser securely.  The virtual host embeds the collection uuid or portable data hash in the hostname.  For example, a collection with uuid @xxxxx-4zz18-tci4vn4fa95w0zx@ could be served as @xxxxx-4zz18-tci4vn4fa95w0zx.collections.ClusterID.example.com@ .  The portable data hash @dd755dbc8d49a67f4fe7dc843e4f10a6+54@ could be served at @dd755dbc8d49a67f4fe7dc843e4f10a6-54.collections.ClusterID.example.com@ .  This requires \"wildcard DNS record\":https://en.wikipedia.org/wiki/Wildcard_DNS_record and \"wildcard TLS certificate.\":https://en.wikipedia.org/wiki/Wildcard_certificate\n\nh3. Collections download URL\n\nDownloads links will served from the URL in @Services.WebDAVDownload.ExternalURL@ .  The collection uuid or PDH is put in the URL path.\n\nIf blank, serve links to WebDAV with @disposition=attachment@ query param.  Unlike preview links, browsers do not render attachments, so there is no risk of XSS.\n\nIf @WebDAVDownload@ is blank, and @WebDAV@ has a single origin (not wildcard, see below), then Workbench will show an error page\n\n\n
    Services:\n      WebDAVDownload:\n        ExternalURL: https://download.ClusterID.example.com\n
\n\n\nh3. Collections preview URL\n\nCollections will be served using the URL pattern in @Services.WebDAV.ExternalURL@ .  If blank, use @Services.WebDAVDownload.ExternalURL@ instead, and disable inline preview.  If both are empty, downloading collections from workbench will be impossible.  When wildcard domains configured, credentials are still required to access non-public data.\n\nh4. In their own subdomain\n\nCollections can be served from their own subdomain:\n\n\n
    Services:\n      WebDAV:\n        ExternalURL: https://*.collections.ClusterID.example.com/\n
\n\n\nThis option is preferred if you plan to access Keep using third-party S3 client software, because it accommodates S3 virtual host-style requests and path-style requests without any special client configuration.\n\nh4. Under the main domain\n\nAlternately, they can go under the main domain by including @--@:\n\n\n
    Services:\n      WebDAV:\n        ExternalURL: https://*--collections.ClusterID.example.com/\n
\n\n\nh4. From a single domain\n\nServe preview links from a single domain, setting uuid or pdh in the path (similar to downloads).  This configuration only allows previews of public data (data accessible by the anonymous user) and collection-sharing links (where the token is already embedded in the URL); it will ignore authorization headers, so a request for non-public data may return \"404 Not Found\" even if normally valid credentials were provided.\n\n\n
    Services:\n      WebDAV:\n        ExternalURL: https://collections.ClusterID.example.com/\n
\n\n\nNote the trailing slash.\n\n{% include 'notebox_begin' %}\nWhether you choose to serve collections from their own subdomain or from a single domain, it's important to keep in mind that they should be served from me same _site_ as Workbench for the inline previews to work.\n\nPlease check \"keep-web's URL pattern guide\":../api/keep-web-urls.html#same-site to learn more.\n{% include 'notebox_end' %}\n\nh2. Set InternalURLs\n\n\n
    Services:\n      WebDAV:\n        InternalURLs:\n          http://localhost:9002: {}\n
\n\n\nh2(#update-config). Configure anonymous user token\n\nIf you intend to use Keep-web to serve public data to anonymous clients, configure it with an anonymous token.\n\nGenerate a random string (>= 32 characters long) and put it in the @config.yml@ file, in the @AnonymousUserToken@ field.\n\n\n
    Users:\n      AnonymousUserToken: \"zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz\"\n
\n\n\nSet @Users.AnonymousUserToken: \"\"@ (empty string) or leave it out if you do not want to serve public data.\n\nh3. Update nginx configuration\n\nPut a reverse proxy with SSL support in front of keep-web.  Keep-web itself runs on the port 9002 (or whatever is specified in @Services.WebDAV.InternalURL@) while the reverse proxy runs on port 443 and forwards requests to Keep-web.\n\nUse a text editor to create a new file @/etc/nginx/conf.d/keep-web.conf@ with the following configuration. Options that need attention are marked in red.\n\n
\nupstream keep-web {\n  server                127.0.0.1:9002;\n}\n\nserver {\n  listen                443 ssl;\n  server_name           download.ClusterID.example.com\n                        collections.ClusterID.example.com\n                        *.collections.ClusterID.example.com\n                        ~.*--collections\\.ClusterID\\.example\\.com;\n\n  proxy_connect_timeout 90s;\n  proxy_read_timeout    300s;\n\n  ssl                   on;\n  ssl_certificate       /YOUR/PATH/TO/cert.pem;\n  ssl_certificate_key   /YOUR/PATH/TO/cert.key;\n\n  location / {\n    proxy_pass          http://keep-web;\n    proxy_set_header    Host            $host;\n    proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;\n\n    client_max_body_size    0;\n    proxy_http_version      1.1;\n    proxy_request_buffering off;\n    proxy_max_temp_file_size 0;\n  }\n}\n
\n\n{% include 'notebox_begin' %}\nIf you restrict access to your Arvados services based on network topology -- for example, your proxy server is not reachable from the public internet -- additional proxy configuration might be needed to thwart cross-site scripting attacks that would circumvent your restrictions.\n\nNormally, Keep-web accepts requests for multiple collections using the same host name, provided the client's credentials are not being used. This provides insufficient XSS protection in an installation where the \"anonymously accessible\" data is not truly public, but merely protected by network topology.\n\nIn such cases -- for example, a site which is not reachable from the internet, where some data is world-readable from Arvados's perspective but is intended to be available only to users within the local network -- the downstream proxy should configured to return 401 for all paths beginning with \"/c=\"\n{% include 'notebox_end' %}\n\nh3. Configure filesystem cache size\n\nKeep-web stores copies of recently accessed data blocks in @/var/cache/arvados/keep@. The cache size defaults to 10% of the size of the filesystem where that directory is located (typically @/var@) and can be customized with the @DiskCacheSize@ config entry.\n\n\n
  Collections:\n    WebDAVCache:\n      DiskCacheSize: 20 GiB
\n\n{% assign arvados_component = 'keep-web' %}\n\n{% include 'install_packages' %}\n\n{% include 'start_service' %}\n\n{% include 'restart_api' %}\n\nh2(#confirm-working). Confirm working installation\n\nWe recommend using the \"Cluster diagnostics tool.\":diagnostics.html\n\nHere are some other checks you can perform manually.\n\n\n
$ curl -H \"Authorization: Bearer $system_root_token\" https://download.ClusterID.example.com/c=59389a8f9ee9d399be35462a0f92541c-53/_/hello.txt
\n\n\nIf wildcard collections domains are configured:\n\n\n
$ curl -H \"Authorization: Bearer $system_root_token\" https://59389a8f9ee9d399be35462a0f92541c-53.collections.ClusterID.example.com/hello.txt
\n\n\nIf using a single collections preview domain:\n\n\n
$ curl https://collections.ClusterID.example.com/c=59389a8f9ee9d399be35462a0f92541c-53/t=$system_root_token/_/hello.txt
\n\n"
  },
  {
    "path": "doc/install/install-keepproxy.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Install Keepproxy server\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n# \"Introduction\":#introduction\n# \"Update config.yml\":#update-config\n# \"Update nginx configuration\":#update-nginx\n# \"Install keepproxy package\":#install-packages\n# \"Start the service\":#start-service\n# \"Restart the API server and controller\":#restart-api\n# \"Confirm working installation\":#confirm-working\n\nh2(#introduction). Introduction\n\nThe Keepproxy server is a gateway into your Keep storage. Unlike the Keepstore servers, which are only accessible on the local LAN, Keepproxy is suitable for clients located elsewhere on the internet. Specifically, in contrast to Keepstore:\n* A client writing through Keepproxy sends a single copy of a data block, and Keepproxy distributes copies to the appropriate Keepstore servers.\n* A client can write through Keepproxy without precomputing content hashes.\n* Keepproxy checks API token validity before processing requests. (Clients that can connect directly to Keepstore can use it as scratch space even without a valid API token.)\n\nBy convention, we use the following hostname for the Keepproxy server:\n\n
\ntable(table table-bordered table-condensed).\n|_. Hostname|\n|@keep.ClusterID.example.com@|\n
\n\nThis hostname should resolve from anywhere on the internet.\n\nh2(#update-config). Update config.yml\n\nEdit the cluster config at @config.yml@ and set @Services.Keepproxy.ExternalURL@ and @Services.Keepproxy.InternalURLs@.\n\n\n
    Services:\n      Keepproxy:\n        ExternalURL: https://keep.ClusterID.example.com\n        InternalURLs:\n          \"http://localhost:25107\": {}\n
\n\n\nh2(#update-nginx). Update Nginx configuration\n\nPut a reverse proxy with SSL support in front of Keepproxy. Keepproxy itself runs on the port 25107 (or whatever is specified in @Services.Keepproxy.InternalURL@) while the reverse proxy runs on port 443 and forwards requests to Keepproxy.\n\nUse a text editor to create a new file @/etc/nginx/conf.d/keepproxy.conf@ with the following configuration. Options that need attention are marked in red.\n\n
upstream keepproxy {\n  server                127.0.0.1:25107;\n}\n\nserver {\n  listen                  443 ssl;\n  server_name             keep.ClusterID.example.com;\n\n  proxy_connect_timeout   90s;\n  proxy_read_timeout      300s;\n  proxy_set_header        X-Real-IP $remote_addr;\n  proxy_http_version      1.1;\n  proxy_request_buffering off;\n  proxy_max_temp_file_size 0;\n\n  ssl_certificate     /YOUR/PATH/TO/cert.pem;\n  ssl_certificate_key /YOUR/PATH/TO/cert.key;\n\n  # Clients need to be able to upload blocks of data up to 64MiB in size.\n  client_max_body_size    64m;\n\n  location / {\n    proxy_pass            http://keepproxy;\n  }\n}\n
\n\nNote: if the Web uploader is failing to upload data and there are no logs from keepproxy, be sure to check the nginx proxy logs.  In addition to \"GET\" and \"PUT\", The nginx proxy must pass \"OPTIONS\" requests to keepproxy, which should respond with appropriate Cross-origin resource sharing headers.  If the CORS headers are not present, brower security policy will cause the upload request to silently fail.  The CORS headers are generated by keepproxy and should not be set in nginx.\n\n{% assign arvados_component = 'keepproxy' %}\n\n{% include 'install_packages' %}\n\n{% include 'start_service' %}\n\n{% include 'restart_api' %}\n\nh2(#confirm-working). Confirm working installation\n\nWe recommend using the \"Cluster diagnostics tool.\":diagnostics.html  Because Keepproxy is specifically a gateway used by outside clients, for this test you should run the diagnostics from a client machine outside the Arvados private network, and provide the @-external-client@ parameter.\n\nHere are some other checks you can perform manually.\n\nLog into a host that is on a network external to your private Arvados network.  The host should be able to contact your keepproxy server (eg @keep.ClusterID.example.com@), but not your keepstore servers (eg keep[0-9].ClusterID.example.com).\n\n@ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ must be set in the environment.\n\n@ARVADOS_API_HOST@ should be the hostname of the API server.\n\n@ARVADOS_API_TOKEN@ should be the system root token.\n\nInstall the \"Command line SDK\":{{site.baseurl}}/sdk/cli/install.html\n\nCheck that the keepproxy server is in the @keep_service@ \"accessible\" list:\n\n\n
\n$ arv keep_service accessible\n[...]\n
\n\n\nIf keepstore does not show up in the \"accessible\" list, and you are accessing it from within the private network, check that you have \"properly configured the @geo@ block for the API server\":install-api-server.html#update-nginx .\n\nInstall the \"Python SDK\":{{site.baseurl}}/sdk/python/sdk-python.html\n\nYou should now be able to use @arv-put@ to upload collections and @arv-get@ to fetch collections.  Be sure to execute this from _outside_ the cluster's private network.\n\n{% include 'arv_put_example' %}\n"
  },
  {
    "path": "doc/install/install-keepstore.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Install Keepstore servers\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n# \"Introduction\":#introduction\n# \"Update config.yml\":#update-config\n# \"Install keepstore package\":#install-packages\n# \"Restart the API server and controller\":#restart-api\n# \"Confirm working installation\":#confirm-working\n# \"Note on storage management\":#note\n\nh2. Introduction\n\nKeepstore provides access to underlying storage for reading and writing content-addressed blocks, with enforcement of Arvados permissions.  Keepstore supports a variety of cloud object storage and POSIX filesystems for its backing store.\n\nh3. Plan your storage layout\n\nIn the steps below, you will configure a number of backend storage volumes (like local filesystems and S3 buckets) and specify which keepstore servers have read-only and read-write access to which volumes.\n\nIt is possible to configure arbitrary server/volume layouts. However, in order to provide good performance and efficient use of storage resources, we strongly recommend using one of the following layouts:\n\n# Each volume is writable by exactly one server, and optionally readable by one or more other servers. The total capacity of all writable volumes is the same for each server.\n# Each volume is writable by all servers. Each volume has enough built-in redundancy to satisfy your requirements, i.e., you do not need Arvados to mirror data across multiple volumes.\n\nWe recommend starting off with two Keepstore servers.  Exact server specifications will be site and workload specific, but in general keepstore will be I/O bound and should be set up to maximize aggregate bandwidth with compute nodes.  To increase capacity (either space or throughput) it is straightforward to add additional servers, or (in cloud environments) to increase the machine size of the existing servers.\n\nBy convention, we use the following hostname pattern:\n\n
\ntable(table table-bordered table-condensed).\n|_. Hostname|\n|@keep0.ClusterID.example.com@|\n|@keep1.ClusterID.example.com@|\n
\n\nKeepstore servers should not be directly accessible from the Internet (they are accessed via \"keepproxy\":install-keepproxy.html), so the hostnames only need to resolve on the private network.\n\nh2(#update-config). Update cluster config\n\nh3. Configure storage volumes\n\nFill in the @Volumes@ section of @config.yml@ for each storage volume.  Available storage volume types include POSIX filesystems and cloud object storage.  It is possible to have different volume types in the same cluster.\n\n* To use a POSIX filesystem, including both local filesystems (ext4, xfs) and network file system such as GPFS or Lustre, follow the setup instructions on \"Filesystem storage\":configure-fs-storage.html\n* If you are using S3-compatible object storage (including Amazon S3, Google Cloud Storage, and Ceph RADOS), follow the setup instructions on \"S3 Object Storage\":configure-s3-object-storage.html\n* If you are using Azure Blob Storage, follow the setup instructions on \"Azure Blob Storage\":configure-azure-blob-storage.html\n\nThere are a number of general configuration parameters for Keepstore. They are described in the \"configuration reference\":{{site.baseurl}}/admin/config.html. In particular, you probably want to change @API/MaxKeepBlobBuffers@ to align Keepstore's memory usage with the available memory on the machine that hosts it.\n\nh3. List services\n\nAdd each keepstore server to the @Services.Keepstore@ section of @/etc/arvados/config.yml@ .\n\n\n
    Services:\n      Keepstore:\n        # No ExternalURL because they are only accessed by the internal subnet.\n        InternalURLs:\n          \"http://keep0.ClusterID.example.com:25107\": {}\n          \"http://keep1.ClusterID.example.com:25107\": {}\n          # and so forth\n
\n\n\n{% assign arvados_component = 'keepstore' %}\n\n{% include 'install_packages' %}\n\n{% include 'start_service' %}\n\n{% include 'restart_api' %}\n\nh2(#confirm-working). Confirm working installation\n\nWe recommend using the \"Cluster diagnostics tool.\":diagnostics.html\n\nHere are some other checks you can perform manually.\n\nLog into a host that is on your private Arvados network.  The host should be able to contact your your keepstore servers (eg keep[0-9].ClusterID.example.com).\n\n@ARVADOS_API_HOST@ and @ARVADOS_API_TOKEN@ must be set in the environment.\n\n@ARVADOS_API_HOST@ should be the hostname of the API server.\n\n@ARVADOS_API_TOKEN@ should be the system root token.\n\nInstall the \"Command line SDK\":{{site.baseurl}}/sdk/cli/install.html\n\nCheck that the keepstore server is in the @keep_service@ \"accessible\" list:\n\n\n
$ arv keep_service accessible\n[...]\n
\n\n\nIf keepstore does not show up in the \"accessible\" list, and you are accessing it from within the private network, check that you have \"properly configured the @geo@ block for the API server\":install-api-server.html#update-nginx .\n\nNext, install the \"Python SDK\":{{site.baseurl}}/sdk/python/sdk-python.html\n\nYou should now be able to use @arv-put@ to upload collections and @arv-get@ to fetch collections.  Be sure to execute this from _inside_ the cluster's private network.  You will be able to access keep from _outside_ the private network after setting up \"keepproxy\":install-keepproxy.html .\n\n{% include 'arv_put_example' %}\n\nh2(#note). Note on storage management\n\nOn its own, a keepstore server never deletes data. Instead, the keep-balance service determines which blocks are candidates for deletion and instructs the keepstore to move those blocks to the trash. Please see the \"Balancing Keep servers\":{{site.baseurl}}/admin/keep-balance.html for more details.\n"
  },
  {
    "path": "doc/install/install-manual-prerequisites.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Planning and prerequisites\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nBefore attempting installation, you should begin by reviewing supported platforms, choosing backends for identity, storage, and scheduling, and decide how you will distribute Arvados services onto machines.  You should also choose an Arvados Cluster ID, choose your hostnames, and aquire TLS certificates.  It may be helpful to make notes as you go along using one of these worksheets:  \"New cluster checklist for AWS\":new_cluster_checklist_AWS.xlsx - \"New cluster checklist for Azure\":new_cluster_checklist_Azure.xlsx - \"New cluster checklist for on premises Slurm\":new_cluster_checklist_slurm.xlsx\n\nThe installation guide describes how to set up a basic standalone Arvados instance.  Additional configuration for features including \"federation,\":{{site.baseurl}}/admin/federation.html \"collection versioning,\":{{site.baseurl}}/admin/collection-versioning.html \"managed properties,\":{{site.baseurl}}/admin/collection-managed-properties.html and \"storage classes\":{{site.baseurl}}/admin/collection-managed-properties.html are described in the \"Admin guide.\":{{site.baseurl}}/admin/\n\nThe Arvados storage subsystem is called \"keep\".  The compute subsystem is called \"crunch\".\n\n# \"Supported GNU/Linux distributions\":#supportedlinux\n# \"Choosing which components to install\":#components\n# \"Identity provider\":#identity\n# \"Storage backend (Keep)\":#storage\n# \"Container compute scheduler (Crunch)\":#scheduler\n# \"Hardware or virtual machines\":#machines\n# \"Arvados Cluster ID\":#clusterid\n# \"DNS and TLS\":#dnstls\n\n\nh2(#supportedlinux). Supported GNU/Linux distributions\n\n{% include 'supportedlinux' %}\n\nh2(#components). Choosing which components to install\n\nArvados consists of many components, some of which may be omitted (at the cost of reduced functionality.)  It may also be helpful to review the \"Arvados Architecture\":{{site.baseurl}}/architecture/ to understand how these components interact.\n\ntable(table table-bordered table-condensed).\n|\\3=. *Core*|\n|\"PostgreSQL database\":install-postgresql.html |Stores data for the API server.|Required.|\n|\"API server + Controller\":install-api-server.html |Core Arvados logic for managing users, groups, collections, containers, and enforcing permissions.|Required.|\n|\\3=. *Keep (storage)*|\n|\"Keepstore\":install-keepstore.html |Stores content-addressed blocks in a variety of backends (local filesystem, cloud object storage).|Required.|\n|\"Keepproxy\":install-keepproxy.html |Gateway service to access keep servers from external networks.|Required to be able to use arv-put, arv-get, or arv-mount outside the private Arvados network.|\n|\"Keep-web\":install-keep-web.html |Gateway service providing read/write HTTP and WebDAV support on top of Keep.|Required to access files from Workbench.|\n|\"Keep-balance\":install-keep-balance.html |Storage cluster maintenance daemon responsible for moving blocks to their optimal server location, adjusting block replication levels, and trashing unreferenced blocks.|Required to free deleted data from underlying storage, and to ensure proper replication and block distribution (including support for storage classes).|\n|\\3=. *User interface*|\n|\"Workbench2\":install-workbench2-app.html |Primary graphical user interface for working with file collections and running containers.|Optional.  Depends on API server, keep-web, websockets server.|\n|\\3=. *Additional services*|\n|\"Websockets server\":install-ws.html |Event distribution server.|Required to view streaming container logs in Workbench.|\n|\"Shell server\":install-shell-server.html |Grant Arvados users access to Unix shell accounts on dedicated shell nodes.|Optional.|\n|\\3=. *Crunch (running containers)*|\n|\"arvados-dispatch-cloud\":crunch2-cloud/install-dispatch-cloud.html |Run analysis workflows on cloud by allocating and freeing cloud VM instances on demand.|Optional|\n|\"crunch-dispatch-slurm\":crunch2-slurm/install-dispatch.html |Run analysis workflows distributed across a Slurm cluster.|Optional|\n|\"crunch-dispatch-lsf\":crunch2-lsf/install-dispatch.html |Run analysis workflows distributed across an LSF cluster.|Optional|\n\nh2(#identity). Identity provider\n\nChoose which backend you will use to authenticate users.\n\n* Google login to authenticate users with a Google account.\n* OpenID Connect (OIDC) if you have Single-Sign-On (SSO) service that supports the OpenID Connect standard.\n* LDAP login to authenticate users by username/password using the LDAP protocol, supported by many services such as OpenLDAP and Active Directory.\n* PAM login to authenticate users by username/password according to the PAM configuration on the controller node.\n\nh2(#postgresql). PostgreSQL\n\nArvados works well with a standalone PostgreSQL installation. When deploying on AWS, Aurora RDS also works but Aurora Serverless is not recommended.\n\nh2(#storage). Storage backend\n\nChoose which backend you will use for storing and retrieving content-addressed Keep blocks.\n\n* File systems storage, such as ext4 or xfs, or network file systems such as GPFS or Lustre\n* Amazon S3, or other object storage that supports the S3 API including Google Cloud Storage and Ceph.\n* Azure blob storage\n\nYou should also determine the desired replication factor for your data.  A replication factor of 1 means only a single copy of a given data block is kept.  With a conventional file system backend and a replication factor of 1, a hard drive failure is likely to lose data.  For this reason the default replication factor is 2 (two copies are kept).\n\nA backend may have its own replication factor (such as durability guarantees of cloud buckets) and Arvados will take this into account when writing a new data block.\n\nh2(#scheduler). Container compute scheduler\n\nChoose which backend you will use to schedule computation.\n\n* On AWS EC2 and Azure, you probably want to use @arvados-dispatch-cloud@ to manage the full lifecycle of cloud compute nodes: starting up nodes sized to the container request, executing containers on those nodes, and shutting nodes down when no longer needed.\n* For on-premises HPC clusters using \"slurm\":https://slurm.schedmd.com/ use @crunch-dispatch-slurm@ to execute containers with slurm job submissions.\n* For on-premises HPC clusters using \"LSF\":https://www.ibm.com/products/hpc-workload-management/ use @crunch-dispatch-lsf@ to execute containers with slurm job submissions.\n* For single node demos, use @crunch-dispatch-local@ to execute containers directly.\n\nh2(#machines). Hardware (or virtual machines)\n\nChoose how to allocate Arvados services to machines.  We recommend that each machine start with a clean installation of a supported GNU/Linux distribution.\n\nFor a production installation, this is a reasonable starting point:\n\n
\ntable(table table-bordered table-condensed).\n|_. Function|_. Number of nodes|_. Recommended specs|\n|PostgreSQL database, Arvados API server, Arvados controller, Websockets, Container dispatcher|1|16+ GiB RAM, 4+ cores, fast disk for database|\n|Workbench, Keepproxy, Keep-web, Keep-balance|1|8 GiB RAM, 2+ cores|\n|Keepstore servers ^1^|2+|4 GiB RAM|\n|Compute worker nodes ^1^|0+ |Depends on workload; scaled dynamically in the cloud|\n|User shell nodes ^2^|0+|Depends on workload|\n
\n\n^1^ Should be scaled up as needed\n^2^ Refers to shell nodes managed by Arvados that provide ssh access for users to interact with Arvados at the command line.  Optional.\n\n{% include 'notebox_begin' %}\nFor a small demo installation, it is possible to run all the Arvados services on a single node.  Special considerations for single-node installs will be noted in boxes like this.\n{% include 'notebox_end' %}\n\nh2(#clusterid). Arvados Cluster ID\n\nEach Arvados installation is identified by a cluster identifier, which is a unique 5-character lowercase alphanumeric string. There are 36 5 = 60466176 possible cluster identifiers.\n\n* For automated test purposes, use “z****”\n* For experimental/local-only/private clusters that won’t ever be visible on the public Internet, use “x****”\n* For long-lived clusters, we recommend reserving a cluster id.  Contact \"info@curii.com\":mailto:info@curii.com for more information.\n\nHere is one way to make a random 5-character string:\n\n\n
~$ tr -dc 0-9a-z </dev/urandom | head -c5; echo\n
\n\n\nYou may also use a different method to pick the cluster identifier. The cluster identifier will be part of the hostname of the services in your Arvados cluster. The rest of this documentation will refer to it as your @ClusterID@.  Whenever @ClusterID@ appears in a configuration example, replace it with your five-character cluster identifier.\n\nh2(#dnstls). DNS entries and TLS certificates\n\nThe following services are normally public-facing and require DNS entries and corresponding TLS certificates.  Get certificates from your preferred TLS certificate provider.  We recommend using \"Let's Encrypt\":https://letsencrypt.org/.  You can run several services on the same node, but each distinct DNS name requires a valid, matching TLS certificate.\n\nThis guide uses the following DNS name conventions.  A later part of this guide will describe how to set up Nginx virtual hosts.\nIt is possible to use custom DNS names for the Arvados services.\n\n
\ntable(table table-bordered table-condensed).\n|_. Function|_. DNS name|\n|Arvados API|@ClusterID.example.com@|\n|Arvados Webshell|webshell.@ClusterID.example.com@|\n|Arvados Websockets endpoint|ws.@ClusterID.example.com@|\n|Arvados Workbench|workbench.@ClusterID.example.com@|\n|Arvados Workbench 2|workbench2.@ClusterID.example.com@|\n|Arvados Keepproxy server|keep.@ClusterID.example.com@|\n|Arvados Keep-web server|download.@ClusterID.example.com@\n_and_\n*.collections.@ClusterID.example.com@ _or_\n*--collections.@ClusterID.example.com@ _or_\ncollections.@ClusterID.example.com@ (see the \"keep-web install docs\":install-keep-web.html)|\n|Container web services|*.containers.@ClusterID.example.com@ _or_\n*--containers.@ClusterID.example.com@|\n
\n\nSetting up Arvados is easiest when Wildcard TLS and wildcard DNS are available. It is also possible to set up Arvados without wildcard TLS and DNS, but some functionality will be unavailable:\n* A wildcard for @keep-web@ (e.g., *.collections.@ClusterID.example.com@) is needed to allow users to view Arvados-hosted data in their browsers. More information on this tradeoff caused by the CORS rules applied by modern browsers is available in the \"keep-web URL pattern guide\":../api/keep-web-urls.html.\n* A wildcard for @controller@ (e.g., *.containers.@ClusterID.example.com@) is needed to allow users to connect to Arvados-hosted services in their browsers.\n\nThe table below lists the required TLS certificates and DNS names in each scenario.\n\n
\ntable(table table-bordered table-condensed).\n||_. Wildcard TLS and DNS available|_. Wildcard TLS available|_. Other|\n|TLS|@ClusterID.example.com@\n*.@ClusterID.example.com@\n*.collections.@ClusterID.example.com@\n*.containers.@ClusterID.example.com@|*.@ClusterID.example.com@\n@ClusterID.example.com@|@ClusterID.example.com@\ngit.@ClusterID.example.com@\nwebshell.@ClusterID.example.com@\nws.@ClusterID.example.com@\nworkbench.@ClusterID.example.com@\nworkbench2.@ClusterID.example.com@\nkeep.@ClusterID.example.com@\ndownload.@ClusterID.example.com@\ncollections.@ClusterID.example.com@|\n|DNS|@ClusterID.example.com@\ngit.@ClusterID.example.com@\nwebshell.@ClusterID.example.com@\nws.@ClusterID.example.com@\nworkbench.@ClusterID.example.com@\nworkbench2.@ClusterID.example.com@\nkeep.@ClusterID.example.com@\ndownload.@ClusterID.example.com@\n*.collections.@ClusterID.example.com@\n*.containers.@ClusterID.example.com@|@ClusterID.example.com@\ngit.@ClusterID.example.com@\nwebshell.@ClusterID.example.com@\nws.@ClusterID.example.com@\nworkbench.@ClusterID.example.com@\nworkbench2.@ClusterID.example.com@\nkeep.@ClusterID.example.com@\ndownload.@ClusterID.example.com@\ncollections.@ClusterID.example.com@|@ClusterID.example.com@\ngit.@ClusterID.example.com@\nwebshell.@ClusterID.example.com@\nws.@ClusterID.example.com@\nworkbench.@ClusterID.example.com@\nworkbench2.@ClusterID.example.com@\nkeep.@ClusterID.example.com@\ndownload.@ClusterID.example.com@\ncollections.@ClusterID.example.com@|\n
\n\n{% include 'notebox_begin' %}\nIt is also possible to create your own certificate authority, issue server certificates, and install a custom root certificate in the browser.  This is out of scope for this guide.\n{% include 'notebox_end' %}\n"
  },
  {
    "path": "doc/install/install-multi-host.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Multi-Host Arvados\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n# \"Introduction\":#introduction\n# \"Prerequisites and planning\":#prerequisites\n# \"Download the installer\":#download\n# \"Initialize the installer\":#copy_config\n# \"Set up your infrastructure\":#setup-infra\n## \"Create AWS infrastructure with Terraform\":#terraform\n## \"Create required infrastructure manually\":#inframanual\n# \"Edit local.params* files\":#localparams\n# \"Configure Keep storage\":#keep\n# \"Choose the SSL configuration\":#certificates\n## \"Using a Let's Encrypt certificates\":#lets-encrypt\n## \"Bring your own certificates\":#bring-your-own\n### \"Securing your TLS certificate keys\":#secure-tls-keys\n# \"Create a compute image\":#create_a_compute_image\n# \"Begin installation\":#installation\n# \"Further customization of the installation\":#further_customization\n# \"Confirm the cluster is working\":#test-install\n## \"Debugging issues\":#debugging\n## \"Iterating on config changes\":#iterating\n## \"Common problems and solutions\":#common-problems\n# \"Initial user and login\":#initial_user\n# \"Monitoring and Metrics\":#monitoring\n# \"Load balancing controllers\":#load_balancing\n# \"After the installation\":#post_install\n\nh2(#introduction). Introduction\n\nThis multi host installer is the recommended way to set up a production Arvados cluster.  These instructions include specific details for installing on Amazon Web Services (AWS), which are marked as \"AWS specific\".  However with additional customization the installer can be used as a template for deployment on other cloud provider or HPC systems.\n\nh2(#prerequisites). Prerequisites and planning\n\nh3. Cluster ID and base domain\n\nChoose a 5-character cluster identifier that will represent the cluster.  Here are \"guidelines on choosing a cluster identifier\":../architecture/federation.html#cluster_id .  Only lowercase letters and digits 0-9 are allowed.  Examples will use @xarv1@ or @${CLUSTER}@, you should substitute the cluster id you have selected.\n\nDetermine the base domain for the cluster.  This will be referred to as @${DOMAIN}@.\n\nFor example, if DOMAIN is @xarv1.example.com@, then @controller.${DOMAIN}@ means @controller.xarv1.example.com@.\n\nh3(#DNS). DNS hostnames for each service\n\nYou will need a DNS entry for each service.  When using the \"Terraform script\":#terraform to set up your infrastructure, these domains will be created automatically using AWS Route 53.\n\nIn the default configuration these are:\n\n# @controller.${DOMAIN}@\n# @ws.${DOMAIN}@\n# @keep0.${DOMAIN}@\n# @keep1.${DOMAIN}@\n# @keep.${DOMAIN}@\n# @download.${DOMAIN}@\n# @*.collections.${DOMAIN}@  -- a wildcard DNS resolving to the @keepweb@ service\n# @*.containers.${DOMAIN}@  -- a wildcard DNS resolving to the @controller@ service\n# @workbench.${DOMAIN}@\n# @workbench2.${DOMAIN}@\n# @webshell.${DOMAIN}@\n# @shell.${DOMAIN}@\n# @prometheus.${DOMAIN}@\n# @grafana.${DOMAIN}@\n\nFor more information, see \"DNS entries and TLS certificates\":install-manual-prerequisites.html#dnstls.\n\nh2(#download). Download the installer\n\n{% assign local_params_src = 'multiple_hosts' %}\n{% assign config_examples_src = 'multi_host/aws' %}\n{% assign terraform_src = 'terraform/aws' %}\n{% include 'download_installer' %}\n\nh2(#setup-infra). Set up your infrastructure\n\n## \"Create AWS infrastructure with Terraform\":#terraform\n## \"Create required infrastructure manually\":#inframanual\n\nh3(#terraform). Create AWS infrastructure with Terraform (AWS specific)\n\nWe provide a set of Terraform code files that you can run to create the necessary infrastructure on Amazon Web Services.\n\nThese files are located in the @terraform@ installer directory and are divided in three sections:\n\n# The @terraform/vpc/@ subdirectory controls the network related infrastructure of your cluster, including firewall rules and split-horizon DNS resolution.\n# The @terraform/data-storage/@ subdirectory controls the stateful part of your cluster, currently only sets up the S3 bucket for holding the Keep blocks and in the future it'll also manage the database service.\n# The @terraform/services/@ subdirectory controls the hosts that will run the different services on your cluster, makes sure that they have the required software for the installer to do its job.\n\nh4. Software requirements & considerations\n\n{% include 'notebox_begin' %}\nThe Terraform state files (that keep crucial infrastructure information from the cloud) will be saved inside each subdirectory, under the @terraform.tfstate@ name.  These will be committed to the git repository used to coordinate deployment.  It is very important to keep this git repository secure, only sysadmins that will be responsible for maintaining your Arvados cluster should have access to it.\n{% include 'notebox_end' %}\n\nh4. Terraform code configuration\n\nEach section described above contain a @terraform.tfvars@ file with some configuration values that you should set before applying each configuration. You should at least set the AWS region, cluster prefix and domain name in @terraform/vpc/terraform.tfvars@:\n\n
{% include 'terraform_vpc_tfvars' %}
\n\nIf you don't set the main configuration variables at @vpc/terraform.tfvars@ file, you will be asked to re-enter these parameters every time you run Terraform.\n\nThe @data-storage/terraform.tfvars@ and @services/terraform.tfvars@ let you configure additional details, including the SSH public key for deployment, instance & volume sizes, etc. All these configurations are provided with sensible defaults:\n\n
{% include 'terraform_datastorage_tfvars' %}
\n\n
{% include 'terraform_services_tfvars' %}
\n\nh4. Set credentials\n\nYou will need an AWS access key and secret key to create the infrastructure.\n\n
export AWS_ACCESS_KEY_ID=\"anaccesskey\"\nexport AWS_SECRET_ACCESS_KEY=\"asecretkey\"
\n\nh4. Create the infrastructure\n\nBuild the infrastructure by running @./installer.sh terraform@.  The last stage will output the information needed to set up the cluster's domain and continue with the installer. for example:\n\n
./installer.sh terraform\n...\nApply complete! Resources: 16 added, 0 changed, 0 destroyed.\n\nOutputs:\n\narvados_sg_id = \"sg-02f999a99973999d7\"\narvados_subnet_id = \"subnet-01234567abc\"\ncluster_int_cidr = \"10.1.0.0/16\"\ncluster_name = \"xarv1\"\ncompute_subnet_id = \"subnet-abcdef12345\"\ndeploy_user = \"admin\"\ndomain_name = \"xarv1.example.com\"\nletsencrypt_iam_access_key_id = \"AKAA43MAAAWAKAADAASD\"\nprivate_ip = {\n  \"controller\" = \"10.1.1.1\"\n  \"keep0\" = \"10.1.1.3\"\n  \"keep1\" = \"10.1.1.4\"\n  \"keepproxy\" = \"10.1.1.2\"\n  \"shell\" = \"10.1.1.7\"\n  \"workbench\" = \"10.1.1.5\"\n}\npublic_ip = {\n  \"controller\" = \"18.235.116.23\"\n  \"keep0\" = \"34.202.85.86\"\n  \"keep1\" = \"38.22.123.98\"\n  \"keepproxy\" = \"34.231.9.201\"\n  \"shell\" = \"44.208.155.240\"\n  \"workbench\" = \"52.204.134.136\"\n}\nregion_name = \"us-east-1\"\nroute53_dns_ns = tolist([\n  \"ns-1119.awsdns-11.org\",\n  \"ns-1812.awsdns-34.co.uk\",\n  \"ns-437.awsdns-54.com\",\n  \"ns-809.awsdns-37.net\",\n])\nssl_password_secret_name = \"xarv1-arvados-ssl-privkey-password\"\nvpc_id = \"vpc-0999994998399923a\"\nletsencrypt_iam_secret_access_key = \"XXXXXSECRETACCESSKEYXXXX\"\ndatabase_password = \n
\n\n\nh4. Additional DNS configuration\n\nOnce Terraform has completed, the infrastructure for your Arvados cluster is up and running.  One last piece of DNS configuration is required.\n\nThe domain names for your cluster (e.g.: controller.xarv1.example.com) are managed via \"Route 53\":https://aws.amazon.com/route53/ and the TLS certificates will be issued using \"Let's Encrypt\":https://letsencrypt.org/ .\n\nYou need to configure the parent domain to delegate to the newly created zone.  For example, you need to configure \"example.com\" to delegate the subdomain \"xarv1.example.com\" to the nameservers for the Arvados hostname records created by Terraform.  You do this by creating a @NS@ record on the parent domain that refers to the name servers listed in the Terraform output parameter @route53_dns_ns@.\n\nIf your parent domain is also controlled by Route 53, the process will be like this:\n\n# Log in to the AWS Console and navigate to the service page for *Route 53*\n# Go to the list of *Hosted zones* and click on the zone for the parent domain\n# Click on *Create record*\n# For *Record name* put the cluster id\n# For *Record type* choose @NS - Name servers for a hosted zone@\n# For *Value* add the values from Terraform output parameter @route53_dns_ns@, one hostname per line, with punctuation (quotes and commas) removed.\n# Click *Create records*\n\nIf the parent domain is controlled by some other service, follow the guide for the the appropriate service.\n\nh4. Other important output parameters\n\nThe certificates will be requested from Let's Encrypt when you run the installer.\n\n* @cluster_int_cidr@ will be used to set @CLUSTER_INT_CIDR@\n\n* You'll also need @compute_subnet_id@ and @arvados_sg_id@ to set @COMPUTE_SUBNET@ and @COMPUTE_SG@ in @local.params@ and when you \"create a compute image\":#create_a_compute_image.\n\nYou can now proceed to \"edit local.params* files\":#localparams.\n\nh3(#inframanual). Create required infrastructure manually\n\nIf you will be setting up infrastructure without using the provided Terraform script, here are the recommendations you will need to consider.\n\nh4. Virtual Private Cloud (AWS specific)\n\nWe recommend setting Arvados up in its own \"Virtual Private Cloud (VPC)\":https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html\n\nWhen you do so, you need to configure a couple of additional things:\n\n# \"Create a subnet for the compute nodes\":https://docs.aws.amazon.com/vpc/latest/userguide/configure-subnets.html\n# You should set up a \"security group which allows SSH access (port 22)\":https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html\n# Make sure to add a \"VPC S3 endpoint\":https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html\n\nh4(#keep-bucket). S3 Bucket (AWS specific)\n\nWe recommend \"creating an S3 bucket\":https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html for data storage named @${CLUSTER}-nyw5e-000000000000000-volume@.  We recommend creating an IAM role called @${CLUSTER}-keepstore-00-iam-role@ with a \"policy that can read, write, list and delete objects in the bucket\":configure-s3-object-storage.html#IAM .  With the example cluster id @xarv1@ the bucket would be called @xarv1-nyw5e-000000000000000-volume@ and the role would be called @xarv1-keepstore-00-iam-role@.\n\nThese names are recommended because they are default names used in the configuration template.  If you use different names, you will need to edit the configuration template later.\n\nh4(#hosts). Required hosts\n\nYou will need to allocate several hosts (physical or virtual machines) for the fixed infrastructure of the Arvados cluster.  These machines should have at least 2 cores and 8 GiB of RAM, running a supported Linux distribution.\n\n{% include 'supportedlinux' %}\n\nAllocate the following hosts as appropriate for your site.  On AWS you may choose to do it manually with the AWS console, or using a DevOps tool such as CloudFormation or Terraform.  With the exception of \"keep0\" and \"keep1\", all of these hosts should have external (public) IP addresses if you intend for them to be accessible outside of the private network or VPC.\n\nThe installer will set up the Arvados services on your machines.  Here is the default assignment of services to machines:\n\n# API node\n## postgresql server\n## arvados api server\n## arvados controller  (recommended hostname @controller.${DOMAIN}@ and @*.containers.${DOMAIN}@)\n# KEEPSTORE nodes (at least 1 if using S3 as a Keep backend, else 2)\n## arvados keepstore   (recommended hostname @keep0.${DOMAIN}@ and @keep1.${DOMAIN}@)\n# WORKBENCH node\n## arvados legacy workbench URLs   (recommended hostname @workbench.${DOMAIN}@)\n## arvados workbench2              (recommended hostname @workbench2.${DOMAIN}@)\n## arvados webshell                (recommended hostname @webshell.${DOMAIN}@)\n## arvados websocket               (recommended hostname @ws.${DOMAIN}@)\n## arvados cloud dispatcher\n## arvados keepbalance\n## arvados keepproxy   (recommended hostname @keep.${DOMAIN}@)\n## arvados keepweb     (recommended hostname @download.${DOMAIN}@ and @*.collections.${DOMAIN}@)\n# SHELL node  (optional)\n## arvados shell       (recommended hostname @shell.${DOMAIN}@)\n\nWhen using the database installed by Arvados (and not an \"external database\":#ext-database), the database is stored under @/var/lib/postgresql@.  Arvados logs are also kept in @/var/log@ and @/var/www/arvados-api/shared/log@.  Accordingly, you should ensure that the disk partition containing @/var@ has adequate storage for your planned usage.  We suggest starting with 50GiB of free space on the database host.\n\nh4. Additional prerequisites when preparing machines to run the installer\n\n# From the account where you are performing the install, passwordless @ssh@ to each machine\nThis means the client's public key should added to @~/.ssh/authorized_keys@ on each node.\n# Passwordless @sudo@ access on the account on each machine you will @ssh@ in to\nThis usually means adding the account to the @sudo@ group and having a rule like this in @/etc/sudoers.d/arvados_passwordless@ that allows members of group @sudo@ to execute any command without entering a password.\n
%sudo ALL=(ALL:ALL) NOPASSWD:ALL
\n# @git@ installed on each machine\n# Port 443 reachable by clients\n\n(AWS specific) The machine that runs the arvados cloud dispatcher will need an \"IAM role that allows it to manage EC2 instances.\":{{site.baseurl}}/install/crunch2-cloud/install-dispatch-cloud.html#IAM\n\nIf your infrastructure differs from the setup proposed above (ie, different hostnames), you can still use the installer, but \"additional customization may be necessary\":#further_customization .\n\nh2(#localparams). Edit @local.params*@ files\n\nThe cluster configuration parameters are included in two files: @local.params@ and @local.params.secrets@. These files can be found wherever you choose to initialize the installation files (e.g., @~/setup-arvados-xarv1@ in these examples).\n\nThe @local.params.secrets@ file is intended to store security-sensitive data such as passwords, private keys, tokens, etc. Depending on the security requirements of the cluster deployment, you may wish to store this file in a secrets store like AWS Secrets Manager or Jenkins credentials.\n\nh3. Parameters from @local.params@:\n\n# Set @CLUSTER@ to the 5-character cluster identifier. (e.g. \"xarv1\")\n# Set @DOMAIN@ to the base DNS domain of the environment. (e.g. \"xarv1.example.com\")\n# Set the @*_INT_IP@ variables with the internal (private) IP addresses of each host. Since services share hosts, some hosts are the same.  See \"note about /etc/hosts\":#etchosts\n# Edit @CLUSTER_INT_CIDR@, this should be the CIDR of the private network that Arvados is running on, e.g. the VPC.  If you used terraform, this is emitted as @cluster_int_cidr@.\n_CIDR stands for \"Classless Inter-Domain Routing\" and describes which portion of the IP address that refers to the network.  For example 192.168.3.0/24 means that the first 24 bits are the network (192.168.3) and the last 8 bits are a specific host on that network._\n_AWS Specific: Go to the AWS console and into the VPC service, there is a column in this table view of the VPCs that gives the CIDR for the VPC (IPv4 CIDR)._\n# Set @INITIAL_USER_EMAIL@ to your email address, as you will be the first admin user of the system.\n\nh3. Parameters from @local.params.secrets@:\n\n# Set each @KEY@ / @TOKEN@ / @PASSWORD@ to a random string.  You can use @installer.sh generate-tokens@\n
./installer.sh generate-tokens\nBLOB_SIGNING_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\nMANAGEMENT_TOKEN=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\nSYSTEM_ROOT_TOKEN=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\nANONYMOUS_USER_TOKEN=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\nDATABASE_PASSWORD=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\n
\n# Set @DATABASE_PASSWORD@ to a random string (unless you \"already have a database\":#ext-database then you should set it to that database's password)\n   Important! If this contains any non-alphanumeric characters, in particular ampersand ('&'), it is necessary to add backslash quoting.\n   For example, if the password is @Lq&MZDATABASE_PASSWORD=\"Lq\\&MZ\\
\n# Set @LE_AWS_*@ credentials to allow Let's Encrypt do authentication through Route53\n# Set @DISPATCHER_SSH_PRIVKEY@ to a SSH private key that @arvados-dispatch-cloud@ will use to connect to the compute nodes:\n
DISPATCHER_SSH_PRIVKEY=\"-----BEGIN OPENSSH PRIVATE KEY-----\nb3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn\n...\ns4VY40kNxs6MsAAAAPbHVjYXNAaW5zdGFsbGVyAQIDBA==\n-----END OPENSSH PRIVATE KEY-----\"\n
You can create one by following the steps described on the \"building a compute node documentation\":{{site.baseurl}}/install/crunch2-cloud/install-compute-node.html#sshkeypair page.\n\nh3(#etchosts). Note on @/etc/hosts@\n\nBecause Arvados services are typically accessed by external clients, they are likely to have both a public IP address and a internal IP address.\n\nOn cloud providers such as AWS, sending internal traffic to a service's public IP address can incur egress costs and throttling. Thus it is very important for internal traffic to stay on the internal network. The installer implements this by updating @/etc/hosts@ on each node to associate each service's hostname with the internal IP address, so that when Arvados services communicate with one another, they always use the internal network address. This is NOT a substitute for DNS, you still need to set up DNS names for all of the services that have public IP addresses (it does, however, avoid a complex \"split-horizon\" DNS configuration).\n\nIt is important to be aware of this because if you mistype the IP address for any of the @*_INT_IP@ variables, hosts may unexpectedly fail to be able to communicate with one another. If this happens, check and edit as necessary the file @/etc/hosts@ on the host that is failing to make an outgoing connection.\n\nh2(#keep). Configure Keep storage\n\nThe @multi_host/aws@ template uses S3 for storage. Arvados also supports \"filesystem storage\":configure-fs-storage.html and \"Azure blob storage\":configure-azure-blob-storage.html . Keep storage configuration can be found in in the @arvados.cluster.Volumes@ section of @local_config_dir/pillars/arvados.sls@.\n\nh3. Object storage in S3 (AWS Specific)\n\nIf you \"followed the recommended naming scheme\":#keep-bucket for both the bucket and role (or used the provided Terraform script), you're done.\n\nIf you did not follow the recommended naming scheme for either the bucket or role, you'll need to update these parameters in @local.params@:\n\n# Set @KEEP_AWS_S3_BUCKET@ to the value of \"keepstore bucket you created earlier\":#keep-bucket\n# Set @KEEP_AWS_IAM_ROLE@ to \"keepstore role you created earlier\":#keep-bucket\n\nYou can also configure a specific AWS Region for the S3 bucket by setting @KEEP_AWS_REGION@.\n\n{% include 'ssl_config_multi' %}\n\nh2(#authentication). Configure your authentication provider (optional, recommended)\n\nBy default, the installer will use the \"Test\" provider, which is a list of usernames and cleartext passwords stored in the Arvados config file. *This is low security configuration and you are strongly advised to configure one of the other \"supported authentication methods\":setup-login.html* .\n\nh2(#ext-database). Using an external database (optional)\n\nThe standard behavior of the installer is to install and configure PostgreSQL for use by Arvados. You can optionally configure it to use a separately managed database instead.\n\nArvados requires a database that is compatible with PostgreSQL 9.5 or later. For example, Arvados is known to work with Amazon Aurora (note: even idle, Arvados services will periodically poll the database, so we strongly advise using \"provisioned\" mode).\n\n# In @local.params@, remove 'database' from the list of roles assigned to the controller node:\n
NODES=(\n  [controller.${DOMAIN}]=controller,websocket,dispatcher,keepbalance\n  ...\n)\n
\n# In @local.params@, set @DATABASE_INT_IP@ to empty string and @DATABASE_EXTERNAL_SERVICE_HOST_OR_IP@ to the database endpoint (can be a hostname, does not have to be an IP address).\n
DATABASE_INT_IP=\"\"\n...\nDATABASE_EXTERNAL_SERVICE_HOST_OR_IP=\"arvados.xxxxxxx.eu-east-1.rds.amazonaws.com\"\n
\n# In @local.params.secrets@, set @DATABASE_PASSWORD@ to the correct value. \"See the previous section describing correct quoting\":#localparams\n# In @local.params@ you may need to adjust the database name and user.\n\nh2(#further_customization). Further customization of the installation (optional)\n\nIf you are installing on AWS and have followed all of the naming conventions recommend in this guide, you probably don't need to do any further customization.\n\nIf you are installing on a different cloud provider or on HPC, other changes may require editing the Saltstack pillars and states files found in @local_config_dir@. In particular, @local_config_dir/pillars/arvados.sls@ contains the template (in the @arvados.cluster@ section) used to produce the Arvados configuration file that is distributed to all the nodes. Consult the \"Configuration reference\":config.html for a comprehensive list of configuration keys.\n\nAny extra Salt \"state\" files you add under @local_config_dir/states@ will be added to the Salt run and applied to the hosts.\n\nh2(#create_a_compute_image). Configure compute nodes\n\n{% include 'branchname' %}\n\nIf you will use fixed compute nodes with an HPC scheduler such as SLURM or LSF, you will need to \"Set up your compute nodes with Docker\":{{site.baseurl}}/install/crunch2/install-compute-node-docker.html or \"Set up your compute nodes with Singularity\":{{site.baseurl}}/install/crunch2/install-compute-node-singularity.html.\n\nOn cloud installations, containers are dispatched in Docker daemons running in the _compute instances_, which need some additional setup.\n\nh3. Build the compute image\n\nFollow \"the instructions to build a cloud compute node image\":{{site.baseurl}}/install/crunch2-cloud/install-compute-node.html using the compute image builder script found in @arvados/tools/compute-images@ in your Arvados clone from \"step 3\":#download.\n\nh3. Configure the compute image\n\nOnce the image has been created, open @local.params@ and edit as follows (AWS specific settings described here, you will need to make custom changes for other cloud providers):\n\n# Set @COMPUTE_AMI@ to the AMI produced by Packer\n# Set @COMPUTE_AWS_REGION@ to the appropriate AWS region\n# Set @COMPUTE_USER@ to the admin user account on the image\n# Set the @COMPUTE_SG@ list to the VPC security group which you set up to allow SSH connections to these nodes\n# Set @COMPUTE_SUBNET@ to the value of SubnetId of your VPC\n# Update @arvados.cluster.InstanceTypes@ in @local_config_dir/pillars/arvados.sls@ as necessary. The example instance types are for AWS, other cloud providers will of course have different instance types with different names and specifications.\n(AWS specific) If m5/c5 node types are not available, replace them with m4/c4. You'll need to double check the values for Price and IncludedScratch/AddedScratch for each type that is changed.\n\nh2(#installation). Begin installation\n\nAt this point, you are ready to run the installer script in deploy mode that will conduct all of the Arvados installation.\n\nRun this in the @~/arvados-setup-xarv1@ directory:\n\n
./installer.sh deploy
\n\nThis will install and configure Arvados on all the nodes. It will take a while and produce a lot of logging. If it runs into an error, it will stop.\n\nh2(#test-install). Confirm the cluster is working\n\nWhen everything has finished, you can run the diagnostics. There's a couple ways of doing this listed below.\n\nh3. Running diagnostics from the same system as the installer\n\nThe requirements to run diagnostics are having @arvados-client@ and @docker@ installed. If this is not possible you can run them on your Arvados shell node as explained in the next section.\n\nDepending on where you are running the installer, you need to provide @-internal-client@ or @-external-client@. If you are running the installer from a host connected to the Arvados private network, use @-internal-client@. Otherwise, use @-external-client@.\n\n
./installer.sh diagnostics (-internal-client|-external-client)
\n\nh3. Running diagnostics from a cluster node\n\nYou can run the diagnostics from the cluster's shell node. This has the advantage that you don't need to manage any software on your local system, but might not be a possibility if your Arvados cluster doesn't include a shell node.\n\n
./installer.sh diagnostics-internal
\n\nh3(#debugging). Debugging issues\n\nThe installer records log files for each deployment.\n\nMost service logs go to @/var/log/syslog@.\n\nThe logs for Rails API server can be found in @/var/www/arvados-api/current/log/production.log@ on the appropriate instance(s).\n\nWorkbench 2 is a client-side Javascript application. If you are having trouble loading Workbench 2, check the browser's developer console (this can be found in \"Tools → Developer Tools\").\n\nh3(#iterating). Iterating on config changes\n\nYou can iterate on the config and maintain the cluster by making changes to @local.params@ and @local_config_dir@ and running @installer.sh deploy@ again.\n\nIf you are debugging a configuration issue on a specific node, you can speed up the cycle a bit by deploying just one node:\n\n
./installer.sh deploy keep0.xarv1.example.com
\n\nHowever, once you have a final configuration, you should run a full deploy to ensure that the configuration has been synchronized on all the nodes.\n\nh3(#common-problems). Common problems and solutions\n\nh4. PG::UndefinedTable: ERROR: relation \\\"api_clients\\\" does not exist\n\nThe arvados-api-server package sets up the database as a post-install script. If the database host or password wasn't set correctly (or quoted correctly) at the time that package is installed, it won't be able to set up the database.\n\nThis will manifest as an error like this:\n\n
\n#\n\nIf this happens, you need to\n\n1. correct the database information\n2. run @./installer.sh deploy xarv1.example.com@ to update the configuration on the API/controller node\n3. Log in to the API/controller server node, then run this command to re-run the post-install script, which will set up the database:\n
dpkg-reconfigure arvados-api-server
\n4. Re-run @./installer.sh deploy@ again to synchronize everything, and so that the install steps that need to contact the API server are run successfully.\n\nh4. Missing ENA support (AWS Specific)\n\nIf the AMI wasn't built with ENA (extended networking) support and the instance type requires it, it'll fail to start.  You'll see an error in syslog on the node that runs @arvados-dispatch-cloud@.  The solution is to build a new AMI with --aws-ena-support true\n\nh2(#initial_user). Initial user and login\n\nAt this point you should be able to log into the Arvados cluster. The initial URL will be\n\n@https://workbench.${DOMAIN}@\n\nIf you did *not* \"configure a different authentication provider\":#authentication you will be using the \"Test\" provider, and the provision script creates an initial user for testing purposes. This user is configured as administrator of the newly created cluster.  It uses the values of @INITIAL_USER@ and @INITIAL_USER_PASSWORD@ from the @local.params*@ file.\n\nIf you *did* configure a different authentication provider, the first user to log in will automatically be given Arvados admin privileges.\n\nh2(#monitoring). Monitoring and Metrics\n\nYou can monitor the health and performance of the system using the admin dashboard:\n\n@https://grafana.${DOMAIN}@\n\nTo log in, use username \"admin\" and @${INITIAL_USER_PASSWORD}@ from @local.params.secrets@.\n\nOnce logged in, you will want to add the dashboards to the front page.\n\n# On the left icon bar, click on \"Browse\"\n# You should see a folder called \"Arvados Cluster\", click to open it\n## If you don't see anything, make sure the check box next to \"Starred\" is not selected\n# You should see three dashboards \"Arvados cluster overview\", \"Node exporter\" and \"Postgres exporter\"\n# Visit each dashboard, at the top of the page click on the star next to the title to \"Mark as favorite\"\n# They should now be linked on the front page.\n\nh2(#load_balancing). Load balancing controllers (optional)\n\nIn order to handle high loads and perform rolling upgrades, the controller service can be scaled to a number of hosts and the installer make this implementation a fairly simple task.\n\nFirst, you should take care of the infrastructure deployment: if you use our Terraform code, you will need to set up the @terraform.tfvars@ in @terraform/vpc/@ so that in addition to the node named @controller@ (the load-balancer), a number of @controllerN@ nodes (backends) are defined as needed, and added to the @internal_service_hosts@ list.\n\nWe suggest that the backend nodes just hold the controller service and nothing else, so they can be easily created or destroyed as needed without other service disruption.\n\nThe following is an example @terraform/vpc/terraform.tfvars@ file that describes a cluster with a load-balancer, 2 backend nodes, a separate database node, a shell node, a keepstore node and a workbench node that will also hold other miscelaneous services:\n\n
region_name = \"us-east-1\"\ncluster_name = \"xarv1\"\ndomain_name = \"xarv1.example.com\"\n# Include controller nodes in this list so instances are assigned to the\n# private subnet. Only the balancer node should be connecting to them.\ninternal_service_hosts = [ \"keep0\", \"shell\", \"database\", \"controller1\", \"controller2\" ]\n\n# Assign private IPs for the controller nodes. These will be used to create\n# internal DNS resolutions that will get used by the balancer and database nodes.\nprivate_ip = {\n  controller = \"10.1.1.11\"\n  workbench = \"10.1.1.15\"\n  database = \"10.1.2.12\"\n  controller1 = \"10.1.2.21\"\n  controller2 = \"10.1.2.22\"\n  shell = \"10.1.2.17\"\n  keep0 = \"10.1.2.13\"\n}
\n\nOnce the infrastructure is deployed, you'll then need to define which node will be using the @balancer@ role and which will be the @controller@ nodes in @local.params@, as it's being shown in this partial example:\n\n
NODES=(\n  [controller.${DOMAIN}]=balancer\n  [controller1.${DOMAIN}]=controller\n  [controller2.${DOMAIN}]=controller\n  [database.${DOMAIN}]=database\n  ...\n)\n
\n\nNote that we also set the @database@ role to its own node instead of just leaving it in a shared controller node.\n\nEach time you run @installer.sh deploy@, the system will automatically do rolling upgrades. This means it will make changes to one controller node at a time, after removing it from the balancer so that there's no downtime.\n\nh2(#post_install). After the installation\n\nAs part of the operation of @installer.sh@, it automatically creates a @git@ repository with your configuration templates.  You should retain this repository but *be aware that it contains sensitive information* (passwords and tokens used by the Arvados services as well as cloud credentials if you used Terraform to create the infrastructure).\n\nAs described in \"Iterating on config changes\":#iterating you may use @installer.sh deploy@ to re-run the Salt to deploy configuration changes and upgrades.  However, be aware that the configuration templates created for you by @installer.sh@ are a snapshot which are not automatically kept up to date.\n\nWhen deploying upgrades, consult the \"Arvados upgrade notes\":{{site.baseurl}}/admin/upgrading.html to see if changes need to be made to the configuration file template in @local_config_dir/pillars/arvados.sls@.  To specify the version to upgrade to, set the @VERSION@ parameter in @local.params@.\n\nSee also \"Maintenance and upgrading\":{{site.baseurl}}/admin/maintenance-and-upgrading.html for more information.\n"
  },
  {
    "path": "doc/install/install-postgresql.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Install PostgreSQL 9.4+\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nArvados requires at least version *9.4* of PostgreSQL. We recommend using version 10 or newer.\n\n* \"AWS\":#aws\n* \"Red Hat, AlmaLinux, and Rocky Linux\":#rh8\n* \"Debian or Ubuntu\":#debian\n\nh3(#aws). AWS\n\nWhen deploying on AWS, Arvados can use an Aurora RDS PostgreSQL database. Aurora Serverless is not recommended.\n\nh3(#rh8). Red Hat, AlmaLinux, and Rocky Linux\n\n{% comment %}\nThe default version on RH8 is PostgreSQL 10. You can install up to PostgreSQL 13.\n{% endcomment %}\n\n# Install PostgreSQL\n  
# dnf install postgresql-server postgresql-contrib
\n# Initialize the database\n  
# postgresql-setup initdb
\n# Configure the database to accept password connections from localhost\n  
# sed -ri -e 's/^(host +all +all +(127\\.0\\.0\\.1\\/32|::1\\/128) +)ident$/\\1md5/' /var/lib/pgsql/data/pg_hba.conf
\n# Configure the database to accept password connections from the local network (replace @10.9.8.0/24@ with your private network mask)\n  
# echo 'host all all 10.9.8.0/24 md5' | tee -a /var/lib/pgsql/data/pg_hba.conf
\n# Configure the database to launch at boot and start now\n  
# systemctl enable --now postgresql
\n\nh3(#debian). Debian or Ubuntu\n\nAll supported versions of Debian and Ubuntu include a version of PostgreSQL you can use with Arvados.\n\n# Install PostgreSQL\n
# apt --no-install-recommends install postgresql postgresql-contrib
\n# Configure PostgreSQL to accept password connections from the local network (replace @10.9.8.0/24@ with your private network mask)\n
# echo 'host all all 10.9.8.0/24 md5' | tee -a /etc/postgresql/*/main/pg_hba.conf
\n# Configure the database to launch at boot and start now\n
# systemctl enable --now postgresql
\n"
  },
  {
    "path": "doc/install/install-shell-server.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Set up a shell node\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n# \"Introduction\":#introduction\n# \"Install Dependencies and SDKs\":#dependencies\n# \"Install git and curl\":#install-packages\n# \"Create record for VM\":#vm-record\n# \"Install arvados-login-sync\":#arvados-login-sync\n# \"Confirm working installation\":#confirm-working\n\nh2(#introduction). Introduction\n\nArvados support for shell nodes allows you to use Arvados permissions to grant Linux shell accounts to users.\n\nA shell node runs the @arvados-login-sync@ service to manage user accounts, and typically has Arvados utilities and SDKs pre-installed.  Users are allowed to log in and run arbitrary programs.  For optimal performance, the Arvados shell server should be on the same LAN as the Arvados cluster.\n\nBecause Arvados @config.yml@ _contains secrets_ it should *not* be present on shell nodes.\n\nShell nodes should be separate virtual machines from the VMs running other Arvados services.  You may choose to grant root access to users so that they can customize the node, for example, installing new programs.  This has security considerations depending on whether a shell node is single-user or multi-user.\n\nA single-user shell node should be set up so that it only stores Arvados access tokens that belong to that user.  In that case, that user can be safely granted root access without compromising other Arvados users.\n\nIn the multi-user shell node case, a malicious user with @root@ access could access other user's Arvados tokens.  Users should only be given @root@ access on a multi-user shell node if you would trust them to be Arvados administrators.  Be aware that with access to the @docker@ daemon, it is trival to gain *root* access to any file on the system, so giving users @docker@ access should be considered equivalent to @root@ access.\n\nh2(#dependencies). Install Dependencies and SDKs\n\n# \"Install Ruby and Bundler\":ruby.html\n# \"Install the Python SDK\":{{site.baseurl}}/sdk/python/sdk-python.html\n# \"Install the FUSE driver\":{{site.baseurl}}/sdk/fuse/install.html\n# \"Install the CLI\":{{site.baseurl}}/sdk/cli/install.html\n# \"Install the R SDK\":{{site.baseurl}}/sdk/R/index.html (optional)\n# \"Install Docker\":install-docker.html (optional)\n\n{% assign arvados_component = 'git curl' %}\n\n{% include 'install_packages' %}\n\nh2(#vm-record). Create record for VM\n\nAs an admin, create an Arvados virtual_machine object representing this shell server. This will return a uuid.\n\n\n
\napiserver:~$ arv --format=uuid virtual_machine create --virtual-machine '{\"hostname\":\"shell.ClusterID.example.com\"}'\nzzzzz-2x53u-zzzzzzzzzzzzzzz\n
\n\n\nh2(#arvados-login-sync). Install arvados-login-sync\n\nThe @arvados-login-sync@ service makes it possible for Arvados users to log in to the shell server.  It sets up login accounts, updates group membership, adds each user's SSH public keys to the @~/.ssh/authorized_keys@ file, and adds an Arvados token to @~/.config/arvados/settings.conf@ .\n\nInstall the @arvados-login-sync@ program from RubyGems.\n\n\n
\nshellserver:# gem install arvados-login-sync\n
\n\n\nh2(#arvados-login-sync). Run arvados-login-sync periodically\n\nCreate a cron job to run the @arvados-login-sync@ program every 2 minutes.  This will synchronize user accounts.\n\nIf this is a single-user shell node, then @ARVADOS_API_TOKEN@ should be a token for that user.  See \"Create a token for a user\":{{site.baseurl}}/admin/user-management-cli.html#create-token .\n\nIf this is a multi-user shell node, then @ARVADOS_API_TOKEN@ should be an administrator token such as the @SystemRootToken@.  See discussion in the \"introduction\":#introduction about security on multi-user shell nodes.\n\nSet @ARVADOS_VIRTUAL_MACHINE_UUID@ to the UUID from \"Create record for VM\":#vm-record\n\nh3. Standalone cluster\n\n\n
\nshellserver:# umask 0700; tee /etc/cron.d/arvados-login-sync <<EOF\nARVADOS_API_HOST=\"ClusterID.example.com\"\nARVADOS_API_TOKEN=\"xxxxxxxxxxxxxxxxx\"\nARVADOS_VIRTUAL_MACHINE_UUID=\"zzzzz-2x53u-zzzzzzzzzzzzzzz\"\n*/2 * * * * root arvados-login-sync\nEOF\n
\n\n\nh3. Part of a LoginCluster federation\n\nIf the cluster is part of a \"federation with centralized user management\":../admin/federation.html#LoginCluster , the login sync script needs to be given an admin token from the login cluster.\n\n\n
\nshellserver:# umask 0700; tee /etc/cron.d/arvados-login-sync <<EOF\nARVADOS_API_HOST=\"ClusterID.example.com\"\nARVADOS_API_TOKEN=\"yyyloginclusteradmintokenyyyy\"\nARVADOS_VIRTUAL_MACHINE_UUID=\"zzzzz-2x53u-zzzzzzzzzzzzzzz\"\n*/2 * * * * root arvados-login-sync\nEOF\n
\n\n\n\nh2(#confirm-working). Confirm working installation\n\nA user should be able to log in to the shell server when the following conditions are satisfied:\n\n# As an admin user, you have given the user permission to log in using the Workbench → Admin menu → \"Users\" item → \"Show\" button → \"Admin\" tab → \"Setup account\" button.\n# The cron job has run.\n\nIn order to log in via SSH, the user must also upload an SSH public key.  Alternately, if configured, users can log in using \"Webshell\":install-webshell.html .\n\nSee also \"how to add a VM login permission link at the command line\":../admin/user-management-cli.html\n"
  },
  {
    "path": "doc/install/install-single-host.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Single host Arvados\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n# \"Limitations of the single host install\":#limitations\n# \"Prerequisites and planning\":#prerequisites\n# \"Download the installer\":#download\n# \"Install Ansible\":#install-ansible\n# \"Set up cluster configuration\":#localparams\n# \"Set up cluster inventory\":#inventory\n# \"Run the installer playbook\":#run-playbook\n# \"Test the cluster\":#test-install\n# \"Changing your configuration\":#further_customization\n# \"Upgrading your Arvados cluster\":#post_install\n\nh2(#limitations). Limitations of the single host install\n\n*NOTE: The single host installation is a good choice for evaluating Arvados, but it is not recommended for production use.*\n\nUsing the default configuration, the single host install has scaling limitations compared to a production multi-host install:\n\n* It uses the local @/var@ partition to store all user data and logs.\n* It uses the @crunch-dispatch-local@ dispatcher, which has a limit of eight concurrent jobs.\n* Because jobs and Arvados services all run on the same machine, they will compete for CPU/RAM resources.\n\nh2(#prerequisites). Prerequisites and planning\n\nh3. Cluster ID\n\nChoose a 5-character cluster identifier that will represent the cluster. Refer to \"our guidelines on choosing a cluster identifier\":../architecture/federation.html#cluster_id.  Only lowercase letters and digits 0-9 are allowed.  Our documentation uses @xurid@ throughout. You should replace this each time it appears with your chosen cluster identifier.\n\nh3. Cluster host\n\nYou will need a dedicated (virtual) machine for your Arvados server with at least 2 cores and 8 GiB of RAM (4+ cores / 16+ GiB recommended if you are running workflows) running a supported Linux distribution:\n\n{% include 'supportedlinux' %}\n\nThe single host install stores all user data and logs under @/var@. You should ensure that this partition has adequate storage for your planned usage.  We suggest starting with at least 50GiB of free space.\n\nYou must be able to connect to this host via SSH. Your account must have permission to run arbitrary commands with @sudo@.\n\nh2(#download). Download the installer\n\nThe Ansible installer is only available in the Arvados source tree. Clone a copy of the Arvados source for the version of Arvados you're using in a directory convenient for you:\n\n{% include 'branchname' %}\n\n
~$ git clone --depth=1 --branch={{ branchname }} https://github.com/arvados/arvados ~/arvados\n
\n\n\nh2(#install-ansible). Install Ansible\n\n{% include 'install_ansible' header_level: 'h3' %}\n\nh2(#localparams). Set up cluster configuration\n\nCopy the example cluster configuration from the Arvados source tree to a location outside it. We recommend you use your chosen cluster ID in the filename to help keep it unique. For example:\n\n\n
$ cp arvados/tools/ansible/examples/simple-cluster-config.yml ~/xurid-config.yml\n
\n\n\nOpen the copy you created in your editor, and make changes following the instructions at the top of the file.\n\nh2(#inventory). Set up cluster inventory\n\nCopy the example cluster inventory from the Arvados source tree to a location outside it. We recommend you use your chosen cluster ID in the filename to help keep it unique. For example:\n\n\n
$ cp arvados/tools/ansible/examples/simple-cluster-inventory.yml ~/xurid-inventory.yml\n
\n\n\nOpen the copy you created in your editor and make these changes noted in comments:\n\n* Under @hosts:@, change @hostname.example@ to the hostname or address of your cluster node.\n* Change @arvados_config_file@ to the path of the cluster configuration you created in the previous step.\n* Change @arvados_cluster_id@ to your chosen cluster ID.\n\nYou may make other changes noted in comments, but the changes listed above are required.\n\nh2(#run-playbook). Run the installer playbook\n\nWith your cluster configuration and inventory complete, you can use them to run the installer playbook:\n\n\n
$ cd arvados/tools/ansible\narvados/tools/ansible $ ansible-playbook -Ki ~/xurid-inventory.yml install-arvados-cluster.yml\n
\n\n\nThis will prompt you for a @BECOME password:@. Enter your sudo password on the cluster node. Ansible will use this to perform privileged system configuration. You will see it start to log tasks like:\n\n\n
PLAY [Bootstrap nodes] *********************************************************\n\nTASK [Load Arvados configuration file] *****************************************\nok: [hostname.example -> localhost]\n\nTASK [Load Arvados cluster configuration] **************************************\nok: [hostname.example]\n\nTASK [ansible.builtin.include_role : distro_bootstrap] *************************\n\nTASK [distro_bootstrap : Get distribution IDs] *********************************\nchanged: [hostname.example]\n
\n\n\nIf all goes well, it will log finish with a @PLAY RECAP@ reporting @failed=0@, which indicates all tasks were successful:\n\n\n
PLAY RECAP *********************************************************************\nhostname.example : ok=161  changed=34   unreachable=0    failed=0    skipped=23   rescued=0    ignored=0\n
\n\n\nh3(#playbook-problems). Diagnosing problems with the playbook run\n\nIf the @PLAY RECAP@ indicates that a task failed, that will typically be logged with a message like this:\n\n\n
TASK [arvados_controller : Start and enable arvados-controller.service] ********\nfatal: [hostname.example]: FAILED! => {\"changed\": false, \"msg\": \"Unable to restart service arvados-controller.service: Job for arvados-controller.service failed because the control process exited with error code.\\nSee \\\"systemctl status arvados-controller.service\\\" and \\\"journalctl -xeu arvados-controller.service\\\" for details.\\n\"}\n
\n\n\nThe @TASK@ line gives you some context for what failed. The first part (@arvados_controller@ in this example) describes generally what Arvados service it was configuring. The rest of the line describes the specific step it was taking (starting @arvados-controller.service@ in this example). This context can suggest where you might check your configuration for problems or look on the cluster node for additional information. This example problem was caused by the Controller service in the cluster configuration trying to use an already-claimed port in one of the @InternalURLs@.\n\nh2(#test-install). Test the cluster\n\nh3. Run diagnostics\n\nThe @arvados-client diagnostics@ command can check all services on a cluster to identify problems with inconsistent configuration. *On your cluster node*, install and run it like this:\n\n\n
$ sudo apt install arvados-client\n$ sudo arvados-client sudo diagnostics -internal-client\nINFO       5: running health check (same as `arvados-server check`)\nINFO      10: getting discovery document from https://hostname.example:8443/discovery/v1/apis/arvados/v1/rest\nINFO      20: getting exported config from https://hostname.example:8443/arvados/v1/config\n[…]\nINFO     160: running a container\nINFO      ... container request uuid = xurid-xvhdp-12345abcde67890\nINFO      ... container request submitted, waiting up to 10m for container to run\nINFO    9990: deleting temporary collection\nINFO    --- no errors ---\n
\n\n\nh3. Access Workbench\n\nThe default Ansible inventory deploys Arvados with a self-signed certificate. If you deployed this way, you will have the best Workbench experience if you configure your browser to trust that certificate for it and supporting services. Follow the instructions for your specific browser below.\n\nIf you configured the inventory with a different certificate that is already trusted by your browser, you can skip these steps. You should be able to open the URL from @Services.Workbench2.ExternalURL@ from your cluster configuration in your browser.\n\nh4. Trusting self-signed certificates in Chrome\n\n{% comment %}\nLast updated for Chrome v138\n{% endcomment %}\n\n# Find the @arvados_tls.Default@ setting in your Ansible inventory.\n# If those options specify @remote: true@, copy the @cert@ path from your cluster host to the host where you're running the browser. Note you _only_ need the @cert@ file, not the @key@ file.\n# In the URL bar, enter @chrome://certificate-manager/@ and open that.\n# Under the \"Custom\" header, open \"Installed by you.\"\n# Next to \"Trusted Cerficates,\" press the \"Import\" button.\n# In the file picker dialog, open your copy of the @arvados_tls.Default.cert@ file.\n\nNow you should be able to open the URL from @Services.Workbench2.ExternalURL@ from your cluster configuration in your browser. You can skip the next section unless you also want to set up Firefox.\n\nh4. Trusting self-signed certificates in Firefox\n\n{% comment %}\nLast updated for Firefox 140\n{% endcomment %}\n\n# Open the \"Edit\" menu and select \"Settings.\"\n# Find and press the \"View Certificates…\" button to open the Certificate Manager.\n# Open the \"Servers\" tab.\n# Press the \"Add Exception…\" button.\n# Enter the @ExternalURL@ in your cluster configuration for the @Workbench2@ service.\n# Press the \"Get Certificate\" button.\n# Press the \"Confirm Security Exception\" button.\n# Repeat the process from step 4 with your configured URLs for the @Controller@, @Keepproxy@, @WebDAV@, and @WebDAVDownload@ services.\n\nNow you should be able to open the URL from @Services.Workbench2.ExternalURL@ from your cluster configuration in your browser.\n\nh2(#further_customization). Changing your configuration\n\nIn the future, if you want to make changes to your Arvados cluster or Ansible inventory configuration, simply edit those files and \"run the playbook again\":#run-playbook. The playbook will deploy your changes to all the component services.\n\nh2(#post_install). Upgrading your Arvados cluster\n\nWhen a new version of Arvados is released, the general process to upgrade the cluster is:\n\n# In your Arvados checkout directory, @git fetch@ and then @git switch@ to the branch or tag that corresponds to the release you want to use.\n# Consult the \"Arvados upgrade notes\":{{site.baseurl}}/admin/upgrading.html to see if you need or want to make change to your cluster configuration file.\n# \"Run the playbook again\":#run-playbook with your cluster inventory.\n\nSee also \"Maintenance and upgrading\":{{site.baseurl}}/admin/maintenance-and-upgrading.html for more information.\n"
  },
  {
    "path": "doc/install/install-webshell.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Configure webshell\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n# \"Introduction\":#introduction\n# \"Prerequisites\":#prerequisites\n# \"Update config.yml\":#configure\n# \"Update nginx configuration\":#update-nginx\n# \"Install packages\":#install-packages\n# \"Configure shellinabox\":#config-shellinabox\n# \"Configure pam\":#config-pam\n# \"Confirm working installation\":#confirm-working\n\nh2(#introduction). Introduction\n\nArvados supports @webshell@, which allows ssh access to shell nodes via the browser. This functionality is integrated in @Workbench@.\n\n@Webshell@ is provided by the @shellinabox@ package which runs on each shell node for which webshell is enabled. For authentication, a supported @pam library@ that allows authentication against Arvados is also required. One Nginx (or similar web server) virtualhost is also needed to expose all the @shellinabox@ instances via https.\n\nh2(#prerequisites). Prerequisites\n\n# \"Install Workbench 2\":{{site.baseurl}}/install/install-workbench2-app.html\n# \"Set up a shell node\":{{site.baseurl}}/install/install-shell-server.html\n\nh2(#configure). Update config.yml\n\nEdit the cluster config at @config.yml@ and set @Services.WebShell.ExternalURL@.  Replace @zzzzz@ with your cluster id. Workbench will use this information to activate its support for webshell.\n\n\n
    Services:\n      WebShell:\n        InternalURLs: {}\n        ExternalURL: https://webshell.ClusterID.example.com/\n
\n\n\nh2(#update-nginx). Update Nginx configuration\n\nThe arvados-webshell service will be accessible from anywhere on the internet, so we recommend using SSL for transport encryption. This Nginx virtualhost could live on your Workbench server, or any other server that is reachable by your Workbench users and can access the @shell-in-a-box@ service on the shell node(s) on port 4200.\n\nUse a text editor to create a new file @/etc/nginx/conf.d/arvados-webshell.conf@ with the following configuration.  Options that need attention are marked in red.\n\n
\nupstream arvados-webshell {\n  server                shell.ClusterID.example.com:4200;\n}\n\nserver {\n  listen                443 ssl;\n  server_name           webshell.ClusterID.example.com;\n\n  proxy_connect_timeout 90s;\n  proxy_read_timeout    300s;\n\n  ssl                   on;\n  ssl_certificate       /YOUR/PATH/TO/cert.pem;\n  ssl_certificate_key   /YOUR/PATH/TO/cert.key;\n\n  location /shell.ClusterID {\n    if ($request_method = 'OPTIONS') {\n       add_header 'Access-Control-Allow-Origin' '*';\n       add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';\n       add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';\n       add_header 'Access-Control-Max-Age' 1728000;\n       add_header 'Content-Type' 'text/plain charset=UTF-8';\n       add_header 'Content-Length' 0;\n       return 204;\n    }\n    if ($request_method = 'POST') {\n       add_header 'Access-Control-Allow-Origin' '*';\n       add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';\n       add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';\n    }\n    if ($request_method = 'GET') {\n       add_header 'Access-Control-Allow-Origin' '*';\n       add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';\n       add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';\n    }\n\n    proxy_ssl_session_reuse off;\n    proxy_read_timeout  90;\n    proxy_set_header    X-Forwarded-Proto https;\n    proxy_set_header    Host $http_host;\n    proxy_set_header    X-Real-IP $remote_addr;\n    proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;\n    proxy_pass          http://arvados-webshell;\n  }\n}\n
\n\nNote that the location line in the nginx config matches your shell node hostname *without domain*, because that is how the shell node was defined in the \"Set up a shell node\":{{site.baseurl}}/install/install-shell-server.html#vm-record instructions. It makes for a more user friendly experience in Workbench.\n\nFor additional shell nodes with @shell-in-a-box@, add @location@ and @upstream@ sections as needed.\n\n{% assign arvados_component = 'shellinabox libpam-arvados-go' %}\n\n{% include 'install_packages' %}\n\nh2(#config-shellinabox). Configure shellinabox\n\nh3. Red Hat, AlmaLinux, and Rocky Linux\n\nEdit @/etc/sysconfig/shellinaboxd@:\n\n
\n# TCP port that shellinboxd's webserver listens on\nPORT=4200\n\n# SSL is disabled because it is terminated in Nginx. Adjust as needed.\nOPTS=\"--disable-ssl --no-beep --service=/shell.ClusterID.example.com:AUTH:HOME:SHELL\"\n
\n\n\n
\n# systemctl enable shellinabox\n# systemctl start shellinabox\n
\n\n\nh3. Debian and Ubuntu\n\nEdit @/etc/default/shellinabox@:\n\n
\n# TCP port that shellinboxd's webserver listens on\nSHELLINABOX_PORT=4200\n\n# SSL is disabled because it is terminated in Nginx. Adjust as needed.\nSHELLINABOX_ARGS=\"--disable-ssl --no-beep --service=/shell.ClusterID.example.com:AUTH:HOME:SHELL\"\n
\n\n\n
\n# systemctl enable shellinabox\n# systemctl start shellinabox\n
\n\n\n\nh2(#config-pam). Configure pam\n\nUse a text editor to create a new file @/etc/pam.d/shellinabox@ with the following configuration.  Options that need attention are marked in red.\n\n
\n# This example is a stock debian \"login\" file with pam_arvados\n# replacing pam_unix. It can be installed as /etc/pam.d/shellinabox .\n\nauth       optional   pam_faildelay.so  delay=3000000\nauth [success=ok new_authtok_reqd=ok ignore=ignore user_unknown=bad default=die] pam_securetty.so\nauth       requisite  pam_nologin.so\nsession [success=ok ignore=ignore module_unknown=ignore default=bad] pam_selinux.so close\nsession       required   pam_env.so readenv=1\nsession       required   pam_env.so readenv=1 envfile=/etc/default/locale\n\n# The first argument is the address of the API server.  The second\n# argument is this shell node's hostname.  The hostname must match the\n# \"hostname\" field of the virtual_machine record.\nauth [success=1 default=ignore] /usr/lib/pam_arvados.so ClusterID.example.com shell.ClusterID.example.com\n\nauth    requisite            pam_deny.so\nauth    required            pam_permit.so\n\nauth       optional   pam_group.so\nsession    required   pam_limits.so\nsession    optional   pam_lastlog.so\nsession    optional   pam_motd.so  motd=/run/motd.dynamic\nsession    optional   pam_motd.so\nsession    optional   pam_mail.so standard\n\n@include common-account\n@include common-session\n@include common-password\n\nsession [success=ok ignore=ignore module_unknown=ignore default=bad] pam_selinux.so open\n
\n\nh2(#confirm-working). Confirm working installation\n\nWe recommend using the \"Cluster diagnostics tool.\":diagnostics.html\n\nHere are some other checks you can perform manually.\n\nA user should now be able to log in to the shell server, using webshell via workbench. Please refer to \"Accessing an Arvados VM with Webshell\":{{site.baseurl}}/user/getting_started/vm-login-with-webshell.html\n"
  },
  {
    "path": "doc/install/install-workbench2-app.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Install Workbench 2\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n# \"Update config.yml\":#update-config\n# \"Update Nginx configuration\":#update-nginx\n# \"Install arvados-workbench2\":#install-packages\n# \"Restart the API server and controller\":#restart-api\n# \"Confirm working installation\":#confirm-working\n# \"Trusted client setting\":#trusted_client\n\nWorkbench2 is the web-based user interface for Arvados.\n\n{% include 'notebox_begin' %}\nWorkbench2 is the replacement for Arvados Workbench. Workbench2 is suitable for day-to-day use, but does not yet implement every feature of the traditional Workbench.\n{% include 'notebox_end' %}\n\nh2(#configure). Update config.yml\n\nEdit @config.yml@ to set the keys below.  The full set of configuration options are in the \"Workbench section of config.yml\":{{site.baseurl}}/admin/config.html\n\n\n
    Services:\n      Workbench2:\n        ExternalURL: \"https://workbench2.ClusterID.example.com\"\n
\n\n\nh2(#update-nginx). Update Nginx configuration\n\nWorkbench2 does not require its own database. It is a set of html, javascript and css files that are served as static files from Nginx.\n\nUse a text editor to create a new file @/etc/nginx/conf.d/arvados-workbench2.conf@ with the following configuration.  Options that need attention are marked in red.\n\n\n
server {\n    listen       80;\n    server_name  workbench2.ClusterID.example.com;\n    return 301   https://workbench2.ClusterID.example.com$request_uri;\n}\n\nserver {\n  listen       443 ssl;\n  server_name  workbench2.ClusterID.example.com;\n\n  ssl_certificate     /YOUR/PATH/TO/cert.pem;\n  ssl_certificate_key /YOUR/PATH/TO/cert.key;\n\n  index  index.html;\n\n  # Workbench2 uses a call to /config.json to bootstrap itself\n  # and find out where to contact the API server.\n  location /config.json {\n    return 200 '{ \"API_HOST\": \"ClusterID.example.com\" }';\n  }\n\n  location / {\n    root      /var/www/arvados-workbench2/workbench2;\n    index     index.html;\n    try_files $uri $uri/ /index.html;\n    if (-f $document_root/maintenance.html) {\n      return 503;\n    }\n  }\n}\n
\n\n\nh2. Vocabulary configuration\n\nWorkbench2 will load, if available, a vocabulary definition which lists available metadata properties for groups and collections.  To learn how to configure the property vocabulary definition, please visit the \"Metadata Vocabulary Format\":{{site.baseurl}}/admin/metadata-vocabulary.html page in the Admin section.\n\n{% assign arvados_component = 'arvados-workbench2' %}\n\n{% include 'install_packages' %}\n\n{% include 'restart_api' %}\n\nh2(#confirm-working). Confirm working installation\n\nVisit @https://workbench2.ClusterID.example.com@ in a browser.  You should be able to log in using the login method you configured in the previous step.  If @Users.AutoAdminFirstUser@ is true, you will be an admin user.\n"
  },
  {
    "path": "doc/install/install-ws.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Install the websocket server\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThe arvados-ws server provides event notifications to websocket clients. It can be installed anywhere with access to Postgres database and the Arvados API server, typically behind a web proxy that provides SSL support. See the \"godoc page\":http://godoc.org/github.com/arvados/arvados/services/ws for additional information.\n\n# \"Update config.yml\":#update-config\n# \"Update nginx configuration\":#update-nginx\n# \"Install arvados-ws package\":#install-packages\n# \"Start the service\":#start-service\n# \"Restart the API server and controller\":#restart-api\n# \"Confirm working installation\":#confirm-working\n\nh2(#configure). Update config.yml\n\nEdit the cluster config at @config.yml@ and set @Services.Websocket.ExternalURL@ and @Services.Websocket.InternalURLs@.  Replace @zzzzz@ with your cluster id.\n\n\n
    Services:\n      Websocket:\n        InternalURLs:\n\t  \"http://localhost:8005\": {}\n        ExternalURL: wss://ws.ClusterID.example.com/websocket\n
\n\n\nh2(#update-nginx). Update Nginx configuration\n\nThe arvados-ws service will be accessible from anywhere on the internet, so we recommend using SSL for transport encryption.\n\nUse a text editor to create a new file @/etc/nginx/conf.d/arvados-ws.conf@ with the following configuration.  Options that need attention are marked in red.\n\n
\nupstream arvados-ws {\n  server                127.0.0.1:8005;\n}\n\nserver {\n  listen                443 ssl;\n  server_name           ws.ClusterID.example.com;\n\n  proxy_connect_timeout 90s;\n  proxy_read_timeout    300s;\n\n  ssl                   on;\n  ssl_certificate       /YOUR/PATH/TO/cert.pem;\n  ssl_certificate_key   /YOUR/PATH/TO/cert.key;\n\n  location / {\n    proxy_pass          http://arvados-ws;\n    proxy_set_header    Upgrade         $http_upgrade;\n    proxy_set_header    Connection      \"upgrade\";\n    proxy_set_header    Host            $host;\n    proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;\n  }\n}\n
\n\n{% assign arvados_component = 'arvados-ws' %}\n\n{% include 'install_packages' %}\n\n{% include 'start_service' %}\n\n{% include 'restart_api' %}\n\nh2(#confirm). Confirm working installation\n\nWe recommend using the \"Cluster diagnostics tool.\":diagnostics.html\n\nHere are some other checks you can perform manually.\n\nConfirm the service is listening on its assigned port and responding to requests.\n\n\n
~$ curl https://ws.ClusterID.example.com/websocket\nnot websocket protocol\n
\n\n"
  },
  {
    "path": "doc/install/nginx.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Install Nginx\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nh3. Red Hat, AlmaLinux, and Rocky Linux\n\n\n
# dnf install nginx
\n\n\nh3. Debian and Ubuntu\n\n\n
# apt --no-install-recommends install nginx
\n\n"
  },
  {
    "path": "doc/install/packages.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Arvados package repositories\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nOn any host where you install Arvados software, you'll need to add the Arvados package repository.  They're available for several popular distributions.\n\n* \"Red Hat, AlmaLinux, and Rocky Linux\":#redhat\n* \"Debian and Ubuntu\":#debian\n\n\n\n\n\nh3(#redhat). Red Hat, AlmaLinux, and Rocky Linux\n\nPackages are available for the following Red Hat-based distributions:\n\n* AlmaLinux 10 (since 10.0)\n* AlmaLinux 9 (since 9.2)\n* AlmaLinux 8 (since 8.8)\n* RHEL 10 (since 10.0)\n* RHEL 9 (since 9.2)\n* RHEL 8 (since 8.8)\n* Rocky Linux 10 (since 10.0)\n* Rocky Linux 9 (since 9.2)\n* Rocky Linux 8 (since 8.8)\n\n{% include 'setup_redhat_repo' %}\n\nh3(#debian). Debian and Ubuntu\n\nPackages are available for the following Debian-based distributions:\n\n* Debian 12 (\"bookworm\")\n* Ubuntu 24.04 (\"noble\")\n* Ubuntu 22.04 (\"jammy\")\n\n{% include 'setup_debian_repo' %}\n"
  },
  {
    "path": "doc/install/ruby.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Install Ruby and Bundler\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n{% include 'install_ruby_and_bundler' %}\n"
  },
  {
    "path": "doc/install/salt-multi-host.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Multi-Host Arvados\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n\"This page has moved.\":install-multi-host.html\n"
  },
  {
    "path": "doc/install/salt-single-host.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Single host Arvados\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n\"This page has moved.\":install-single-host.html\n"
  },
  {
    "path": "doc/install/setup-login.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Set up web based login\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nSelect one of the following login mechanisms for your cluster.\n\n# If all users will authenticate with Google, \"configure Google login\":#google.\n# If all users will authenticate with an OpenID Connect provider (other than Google), \"configure OpenID Connect\":#oidc.\n# If all users will authenticate with an existing LDAP service, \"configure LDAP\":#ldap.\n# If all users will authenticate using PAM as configured on your controller node, \"configure PAM\":#pam.\n\nh2(#google). Google login\n\nWith this configuration, users will sign in with their Google accounts.\n\nUse the Google Developers Console to create a set of client credentials.\n# Select or create a project.\n# Click *+ Enable APIs and Services*.\n#* Search for *Google People API* and click *Enable API*.\n#* Navigate back to the main \"APIs & Services\" page.\n# On the sidebar, click *OAuth consent screen*.\n#* On consent screen settings, enter your identifying details.\n#* Under *Branding* → *Authorized domains* add your domain (@example.com@).\n#* Click *Save*.\n# On the sidebar, click *Clients*, then click *+ Create client*, arriving at the *OAuth client ID* setup page.\n# Under *Application type* select *Web application*.\n# Add the JavaScript origin: @https://workbench2.ClusterID.example.com@. This should match the Web origin where you will host Workbench. Note that it can only include the schema, hostname, and port parts; the path, in particular a trailing @/@, is not allowed.\n# Add the Redirect URI: @https://ClusterID.example.com/login@. The host part of this URI should match the @ExternalURL@ of the Arvados controller service as specified in the configuration file @/etc/arvados/config.yml@, including the port if specified.\n# Copy the values of *Client ID* and *Client secret* to the @Login.Google@ section of @/etc/arvados/config.yml@.\n\n{% codeblock as yaml %}\n    Login:\n      Google:\n        Enable: true\n        ClientID: \"0000000000000-zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz.apps.googleusercontent.com\"\n        ClientSecret: \"zzzzzzzzzzzzzzzzzzzzzzzz\"\n{% endcodeblock %}\n\nh2(#oidc). OpenID Connect\n\nWith this configuration, users will sign in with a third-party OpenID Connect provider such as GitHub, Auth0, Okta, or PingFederate.\n\nSimilar to the Google login section above, you will need to register your Arvados cluster with the provider as an application (relying party). When asked for a redirect URL or callback URL, use @https://ClusterID.example.com/login@ (the external URL of your controller service, plus @/login@).\n\nThe provider will supply an issuer URL, client ID, and client secret. Add these to your Arvados configuration.\n\n{% codeblock as yaml %}\n    Login:\n      OpenIDConnect:\n        Enable: true\n        Issuer: https://accounts.example.com/\n        ClientID: \"0123456789abcdef\"\n        ClientSecret: \"zzzzzzzzzzzzzzzzzzzzzzzz\"\n{% endcodeblock %}\n\nh3. Accepting OpenID bearer tokens as Arvados API tokens\n\nArvados can also be configured to accept provider-issued access tokens as Arvados API tokens by setting @Login.OpenIDConnect.AcceptAccessToken@ to @true@. This can be useful for integrating third party applications.\n\n{% codeblock as yaml %}\n    Login:\n      OpenIDConnect:\n        AcceptAccessToken: true\n        AcceptAccessTokenScope: \"arvados\"\n{% endcodeblock %}\n\n# If the provider-issued tokens are JWTs, and @Login.OpenIDConnect.AcceptAccessTokenScope@ is not empty, Arvados will check that the token contains the configured scope, and reject tokens that do not have the configured scope.  This can be used to control which users or applications are permitted to access your Arvados instance.\n# Tokens are validated by presenting them to the UserInfo endpoint advertised by the OIDC provider.\n# Once validated, a token is cached and accepted without re-checking for up to 10 minutes.\n# A token that fails validation is cached and will not be re-checked for up to 5 minutes.\n# Network errors and HTTP 5xx responses from the provider's UserInfo endpoint are not cached.\n# The OIDC token cache size is currently limited to 1000 tokens, if the number of distinct tokens used in a 5 minute period is greater than this, tokens may be checked more frequently.\n\nCheck the OpenIDConnect section in the \"default config file\":{{site.baseurl}}/admin/config.html for more details and configuration options.\n\nh2(#ldap). LDAP\n\nWith this configuration, authentication uses an external LDAP service like OpenLDAP or Active Directory.\n\nEnable LDAP authentication and provide your LDAP server's host, port, and credentials (if needed to search the directory) in @config.yml@:\n\n{% codeblock as yaml %}\n    Login:\n      LDAP:\n        Enable: true\n        URL: ldap://ldap.example.com:389\n        SearchBindUser: cn=lookupuser,dc=example,dc=com\n        SearchBindPassword: xxxxxxxx\n        SearchBase: ou=Users,dc=example,dc=com\n{% endcodeblock %}\n\nThe email address reported by LDAP will be used as primary key for Arvados accounts. This means *users must not be able to edit their own email addresses* in the directory.\n\nAdditional configuration settings are available:\n* @StartTLS@ is enabled by default.\n* @StripDomain@ and @AppendDomain@ modify the username entered by the user before searching for it in the directory.\n* @SearchAttribute@ (default @uid@) is the LDAP attribute used when searching for usernames.\n* @SearchFilters@ accepts LDAP filter expressions to control which users can log in.\n\nCheck the LDAP section in the \"default config file\":{{site.baseurl}}/admin/config.html for more details and configuration options.\n\nh2(#pam). PAM\n\nWith this configuration, authentication is done according to the Linux PAM (\"Pluggable Authentication Modules\") configuration on your controller host.\n\nEnable PAM authentication in @config.yml@:\n\n{% codeblock as yaml %}\n    Login:\n      PAM:\n        Enable: true\n{% endcodeblock %}\n\nCheck the \"default config file\":{{site.baseurl}}/admin/config.html for more PAM configuration options.\n\nThe default PAM configuration on most Linux systems uses the local user/password database in @/etc/passwd@ and @/etc/shadow@ for all logins. In this case, in order to log in to Arvados, users must have a UNIX account and password on the controller host itself. This can be convenient for a single-user or test cluster. Configuring a user account with a shell of @/bin/false@ will enable the user to log into Arvados but not log into shell login on the controller host.\n\nPAM can also be configured to use other authentication systems such such as NIS or Kerberos. In a production environment, PAM configuration should use the service name (\"arvados\" by default) and set a separate policy for Arvados login.  In this case, Arvados users should not have shell accounts on the controller node.\n\nFor information about configuring PAM, refer to the \"PAM System Administrator's Guide\":http://www.linux-pam.org/Linux-PAM-html/Linux-PAM_SAG.html.\n"
  },
  {
    "path": "doc/install/workbench.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: installguide\ntitle: Customizing Workbench\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nh2. Site name\n\nUse the @Workbench.SiteName@ configuration option to set the site name rendered at the top of Workbench.\n\n{% codeblock as yaml %}\n    Workbench:\n      SiteName: Arvados Workbench\n{% endcodeblock %}\n\nh2. Welcome page\n\nUse the @Workbench.WelcomePageHTML@ configuration option to set the text that is rendered when a user arrives at the front page (and has not yet logged in).\n\n{% codeblock as yaml %}\n    Workbench:\n      WelcomePageHTML: |\n        \n        
Please log in.
\n\n        
If you have never used Arvados Workbench before, logging in\n        for the first time will automatically create a new\n        account.
\n\n        Arvados Workbench uses your information only for\n        identification, and does not retrieve any other personal\n        information.\n{% endcodeblock %}\n\nh2. Inactive user page\n\nUse the @Workbench.InactivePageHTML@ configuration option to set the text that is rendered when a user logs in but is inactive.\n\n{% codeblock as yaml %}\n    Workbench:\n      InactivePageHTML: |\n        \n        
Hi! You're logged in, but...
\n        
Your account is inactive.
\n        
An administrator must activate your account before you can get\n        any further.
\n{% endcodeblock %}\n\nh2(#banner). Message banner on login and custom tooltips\n\nSet the @Workbench.BannerUUID@ configuration option to the UUID of a collection.  *This collection should be shared with all users.*\n\n{% codeblock as yaml %}\n    Workbench:\n      BannerUUID: zzzzz-4zz18-0123456789abcde\n{% endcodeblock %}\n\nh3. Banner\n\nYou can have box pop up when users load Workbench to give information such as links to site-specific documentation or notification about anticipated downtime.\n\nThe banner appears when a user loads workbench and have not yet viewed the current banner text.  Users can also view the banner after dismissing it by selecting the *Restore Banner* option from the *Notifications* menu.\n\nThe banner text (HTML formatted) is loaded from the file @banner.html@ in the collection provided in @BannerUUID@.  The banner does _not_ need to be wrapped by *html* or *body* tags (if present, they will be removed).\n\n{% include 'html_tags' %}\n\nh3. Tooltips\n\nYou can provide a custom tooltip overlay to provide site-specific guidance for using workbench.  Users can opt-out by selecting *Disable Tooltips* from the *Notifications* menu.\n\nThe tooltips are loaded from the file @tooltips.json@ in the collection provided in @BannerUUID@.\n\nThe format of this file is a JSON object where the key is a \"CSS selector\":https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors and the value is the text of the tooltip.  Here is an example:\n\n{% codeblock as yaml %}\n{\n    \"[data-cy=side-panel-button]\": \"Click here to create a new project!\",\n    \"[data-cy=project-panel] tbody tr:nth-child(1)\": \"First element in the project list\"\n}\n{% endcodeblock %}\n\nThe first example adds a tooltip displaying \"Click here to create a new project!\" to the HTML node with the attribute @data-cy=\"side-panel-button\"@.\n\nThe second example adds a tooltip displaying \"First element in the project list\" by finding the project panel element, finding the table body element within the project panel, then matching the first table row.\n\nUse the web developer tools offer by your browser to determine what identifiers are available and construct selections that will anchor your tooltips to the desired workbench components.\n"
  },
  {
    "path": "doc/js/bootstrap.js",
    "content": "/*!\n * Bootstrap v3.1.0 (http://getbootstrap.com)\n * Copyright 2011-2014 Twitter, Inc.\n * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)\n */\n\nif (typeof jQuery === 'undefined') { throw new Error('Bootstrap requires jQuery') }\n\n/* ========================================================================\n * Bootstrap: transition.js v3.1.0\n * http://getbootstrap.com/javascript/#transitions\n * ========================================================================\n * Copyright 2011-2014 Twitter, Inc.\n * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)\n * ======================================================================== */\n\n\n+function ($) {\n  'use strict';\n\n  // CSS TRANSITION SUPPORT (Shoutout: http://www.modernizr.com/)\n  // ============================================================\n\n  function transitionEnd() {\n    var el = document.createElement('bootstrap')\n\n    var transEndEventNames = {\n      'WebkitTransition' : 'webkitTransitionEnd',\n      'MozTransition'    : 'transitionend',\n      'OTransition'      : 'oTransitionEnd otransitionend',\n      'transition'       : 'transitionend'\n    }\n\n    for (var name in transEndEventNames) {\n      if (el.style[name] !== undefined) {\n        return { end: transEndEventNames[name] }\n      }\n    }\n\n    return false // explicit for ie8 (  ._.)\n  }\n\n  // http://blog.alexmaccaw.com/css-transitions\n  $.fn.emulateTransitionEnd = function (duration) {\n    var called = false, $el = this\n    $(this).one($.support.transition.end, function () { called = true })\n    var callback = function () { if (!called) $($el).trigger($.support.transition.end) }\n    setTimeout(callback, duration)\n    return this\n  }\n\n  $(function () {\n    $.support.transition = transitionEnd()\n  })\n\n}(jQuery);\n\n/* ========================================================================\n * Bootstrap: alert.js v3.1.0\n * http://getbootstrap.com/javascript/#alerts\n * ========================================================================\n * Copyright 2011-2014 Twitter, Inc.\n * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)\n * ======================================================================== */\n\n\n+function ($) {\n  'use strict';\n\n  // ALERT CLASS DEFINITION\n  // ======================\n\n  var dismiss = '[data-dismiss=\"alert\"]'\n  var Alert   = function (el) {\n    $(el).on('click', dismiss, this.close)\n  }\n\n  Alert.prototype.close = function (e) {\n    var $this    = $(this)\n    var selector = $this.attr('data-target')\n\n    if (!selector) {\n      selector = $this.attr('href')\n      selector = selector && selector.replace(/.*(?=#[^\\s]*$)/, '') // strip for ie7\n    }\n\n    var $parent = $(selector)\n\n    if (e) e.preventDefault()\n\n    if (!$parent.length) {\n      $parent = $this.hasClass('alert') ? $this : $this.parent()\n    }\n\n    $parent.trigger(e = $.Event('close.bs.alert'))\n\n    if (e.isDefaultPrevented()) return\n\n    $parent.removeClass('in')\n\n    function removeElement() {\n      $parent.trigger('closed.bs.alert').remove()\n    }\n\n    $.support.transition && $parent.hasClass('fade') ?\n      $parent\n        .one($.support.transition.end, removeElement)\n        .emulateTransitionEnd(150) :\n      removeElement()\n  }\n\n\n  // ALERT PLUGIN DEFINITION\n  // =======================\n\n  var old = $.fn.alert\n\n  $.fn.alert = function (option) {\n    return this.each(function () {\n      var $this = $(this)\n      var data  = $this.data('bs.alert')\n\n      if (!data) $this.data('bs.alert', (data = new Alert(this)))\n      if (typeof option == 'string') data[option].call($this)\n    })\n  }\n\n  $.fn.alert.Constructor = Alert\n\n\n  // ALERT NO CONFLICT\n  // =================\n\n  $.fn.alert.noConflict = function () {\n    $.fn.alert = old\n    return this\n  }\n\n\n  // ALERT DATA-API\n  // ==============\n\n  $(document).on('click.bs.alert.data-api', dismiss, Alert.prototype.close)\n\n}(jQuery);\n\n/* ========================================================================\n * Bootstrap: button.js v3.1.0\n * http://getbootstrap.com/javascript/#buttons\n * ========================================================================\n * Copyright 2011-2014 Twitter, Inc.\n * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)\n * ======================================================================== */\n\n\n+function ($) {\n  'use strict';\n\n  // BUTTON PUBLIC CLASS DEFINITION\n  // ==============================\n\n  var Button = function (element, options) {\n    this.$element  = $(element)\n    this.options   = $.extend({}, Button.DEFAULTS, options)\n    this.isLoading = false\n  }\n\n  Button.DEFAULTS = {\n    loadingText: 'loading...'\n  }\n\n  Button.prototype.setState = function (state) {\n    var d    = 'disabled'\n    var $el  = this.$element\n    var val  = $el.is('input') ? 'val' : 'html'\n    var data = $el.data()\n\n    state = state + 'Text'\n\n    if (!data.resetText) $el.data('resetText', $el[val]())\n\n    $el[val](data[state] || this.options[state])\n\n    // push to event loop to allow forms to submit\n    setTimeout($.proxy(function () {\n      if (state == 'loadingText') {\n        this.isLoading = true\n        $el.addClass(d).attr(d, d)\n      } else if (this.isLoading) {\n        this.isLoading = false\n        $el.removeClass(d).removeAttr(d)\n      }\n    }, this), 0)\n  }\n\n  Button.prototype.toggle = function () {\n    var changed = true\n    var $parent = this.$element.closest('[data-toggle=\"buttons\"]')\n\n    if ($parent.length) {\n      var $input = this.$element.find('input')\n      if ($input.prop('type') == 'radio') {\n        if ($input.prop('checked') && this.$element.hasClass('active')) changed = false\n        else $parent.find('.active').removeClass('active')\n      }\n      if (changed) $input.prop('checked', !this.$element.hasClass('active')).trigger('change')\n    }\n\n    if (changed) this.$element.toggleClass('active')\n  }\n\n\n  // BUTTON PLUGIN DEFINITION\n  // ========================\n\n  var old = $.fn.button\n\n  $.fn.button = function (option) {\n    return this.each(function () {\n      var $this   = $(this)\n      var data    = $this.data('bs.button')\n      var options = typeof option == 'object' && option\n\n      if (!data) $this.data('bs.button', (data = new Button(this, options)))\n\n      if (option == 'toggle') data.toggle()\n      else if (option) data.setState(option)\n    })\n  }\n\n  $.fn.button.Constructor = Button\n\n\n  // BUTTON NO CONFLICT\n  // ==================\n\n  $.fn.button.noConflict = function () {\n    $.fn.button = old\n    return this\n  }\n\n\n  // BUTTON DATA-API\n  // ===============\n\n  $(document).on('click.bs.button.data-api', '[data-toggle^=button]', function (e) {\n    var $btn = $(e.target)\n    if (!$btn.hasClass('btn')) $btn = $btn.closest('.btn')\n    $btn.button('toggle')\n    e.preventDefault()\n  })\n\n}(jQuery);\n\n/* ========================================================================\n * Bootstrap: carousel.js v3.1.0\n * http://getbootstrap.com/javascript/#carousel\n * ========================================================================\n * Copyright 2011-2014 Twitter, Inc.\n * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)\n * ======================================================================== */\n\n\n+function ($) {\n  'use strict';\n\n  // CAROUSEL CLASS DEFINITION\n  // =========================\n\n  var Carousel = function (element, options) {\n    this.$element    = $(element)\n    this.$indicators = this.$element.find('.carousel-indicators')\n    this.options     = options\n    this.paused      =\n    this.sliding     =\n    this.interval    =\n    this.$active     =\n    this.$items      = null\n\n    this.options.pause == 'hover' && this.$element\n      .on('mouseenter', $.proxy(this.pause, this))\n      .on('mouseleave', $.proxy(this.cycle, this))\n  }\n\n  Carousel.DEFAULTS = {\n    interval: 5000,\n    pause: 'hover',\n    wrap: true\n  }\n\n  Carousel.prototype.cycle =  function (e) {\n    e || (this.paused = false)\n\n    this.interval && clearInterval(this.interval)\n\n    this.options.interval\n      && !this.paused\n      && (this.interval = setInterval($.proxy(this.next, this), this.options.interval))\n\n    return this\n  }\n\n  Carousel.prototype.getActiveIndex = function () {\n    this.$active = this.$element.find('.item.active')\n    this.$items  = this.$active.parent().children()\n\n    return this.$items.index(this.$active)\n  }\n\n  Carousel.prototype.to = function (pos) {\n    var that        = this\n    var activeIndex = this.getActiveIndex()\n\n    if (pos > (this.$items.length - 1) || pos < 0) return\n\n    if (this.sliding)       return this.$element.one('slid.bs.carousel', function () { that.to(pos) })\n    if (activeIndex == pos) return this.pause().cycle()\n\n    return this.slide(pos > activeIndex ? 'next' : 'prev', $(this.$items[pos]))\n  }\n\n  Carousel.prototype.pause = function (e) {\n    e || (this.paused = true)\n\n    if (this.$element.find('.next, .prev').length && $.support.transition) {\n      this.$element.trigger($.support.transition.end)\n      this.cycle(true)\n    }\n\n    this.interval = clearInterval(this.interval)\n\n    return this\n  }\n\n  Carousel.prototype.next = function () {\n    if (this.sliding) return\n    return this.slide('next')\n  }\n\n  Carousel.prototype.prev = function () {\n    if (this.sliding) return\n    return this.slide('prev')\n  }\n\n  Carousel.prototype.slide = function (type, next) {\n    var $active   = this.$element.find('.item.active')\n    var $next     = next || $active[type]()\n    var isCycling = this.interval\n    var direction = type == 'next' ? 'left' : 'right'\n    var fallback  = type == 'next' ? 'first' : 'last'\n    var that      = this\n\n    if (!$next.length) {\n      if (!this.options.wrap) return\n      $next = this.$element.find('.item')[fallback]()\n    }\n\n    if ($next.hasClass('active')) return this.sliding = false\n\n    var e = $.Event('slide.bs.carousel', { relatedTarget: $next[0], direction: direction })\n    this.$element.trigger(e)\n    if (e.isDefaultPrevented()) return\n\n    this.sliding = true\n\n    isCycling && this.pause()\n\n    if (this.$indicators.length) {\n      this.$indicators.find('.active').removeClass('active')\n      this.$element.one('slid.bs.carousel', function () {\n        var $nextIndicator = $(that.$indicators.children()[that.getActiveIndex()])\n        $nextIndicator && $nextIndicator.addClass('active')\n      })\n    }\n\n    if ($.support.transition && this.$element.hasClass('slide')) {\n      $next.addClass(type)\n      $next[0].offsetWidth // force reflow\n      $active.addClass(direction)\n      $next.addClass(direction)\n      $active\n        .one($.support.transition.end, function () {\n          $next.removeClass([type, direction].join(' ')).addClass('active')\n          $active.removeClass(['active', direction].join(' '))\n          that.sliding = false\n          setTimeout(function () { that.$element.trigger('slid.bs.carousel') }, 0)\n        })\n        .emulateTransitionEnd($active.css('transition-duration').slice(0, -1) * 1000)\n    } else {\n      $active.removeClass('active')\n      $next.addClass('active')\n      this.sliding = false\n      this.$element.trigger('slid.bs.carousel')\n    }\n\n    isCycling && this.cycle()\n\n    return this\n  }\n\n\n  // CAROUSEL PLUGIN DEFINITION\n  // ==========================\n\n  var old = $.fn.carousel\n\n  $.fn.carousel = function (option) {\n    return this.each(function () {\n      var $this   = $(this)\n      var data    = $this.data('bs.carousel')\n      var options = $.extend({}, Carousel.DEFAULTS, $this.data(), typeof option == 'object' && option)\n      var action  = typeof option == 'string' ? option : options.slide\n\n      if (!data) $this.data('bs.carousel', (data = new Carousel(this, options)))\n      if (typeof option == 'number') data.to(option)\n      else if (action) data[action]()\n      else if (options.interval) data.pause().cycle()\n    })\n  }\n\n  $.fn.carousel.Constructor = Carousel\n\n\n  // CAROUSEL NO CONFLICT\n  // ====================\n\n  $.fn.carousel.noConflict = function () {\n    $.fn.carousel = old\n    return this\n  }\n\n\n  // CAROUSEL DATA-API\n  // =================\n\n  $(document).on('click.bs.carousel.data-api', '[data-slide], [data-slide-to]', function (e) {\n    var $this   = $(this), href\n    var $target = $($this.attr('data-target') || (href = $this.attr('href')) && href.replace(/.*(?=#[^\\s]+$)/, '')) //strip for ie7\n    var options = $.extend({}, $target.data(), $this.data())\n    var slideIndex = $this.attr('data-slide-to')\n    if (slideIndex) options.interval = false\n\n    $target.carousel(options)\n\n    if (slideIndex = $this.attr('data-slide-to')) {\n      $target.data('bs.carousel').to(slideIndex)\n    }\n\n    e.preventDefault()\n  })\n\n  $(window).on('load', function () {\n    $('[data-ride=\"carousel\"]').each(function () {\n      var $carousel = $(this)\n      $carousel.carousel($carousel.data())\n    })\n  })\n\n}(jQuery);\n\n/* ========================================================================\n * Bootstrap: collapse.js v3.1.0\n * http://getbootstrap.com/javascript/#collapse\n * ========================================================================\n * Copyright 2011-2014 Twitter, Inc.\n * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)\n * ======================================================================== */\n\n\n+function ($) {\n  'use strict';\n\n  // COLLAPSE PUBLIC CLASS DEFINITION\n  // ================================\n\n  var Collapse = function (element, options) {\n    this.$element      = $(element)\n    this.options       = $.extend({}, Collapse.DEFAULTS, options)\n    this.transitioning = null\n\n    if (this.options.parent) this.$parent = $(this.options.parent)\n    if (this.options.toggle) this.toggle()\n  }\n\n  Collapse.DEFAULTS = {\n    toggle: true\n  }\n\n  Collapse.prototype.dimension = function () {\n    var hasWidth = this.$element.hasClass('width')\n    return hasWidth ? 'width' : 'height'\n  }\n\n  Collapse.prototype.show = function () {\n    if (this.transitioning || this.$element.hasClass('in')) return\n\n    var startEvent = $.Event('show.bs.collapse')\n    this.$element.trigger(startEvent)\n    if (startEvent.isDefaultPrevented()) return\n\n    var actives = this.$parent && this.$parent.find('> .panel > .in')\n\n    if (actives && actives.length) {\n      var hasData = actives.data('bs.collapse')\n      if (hasData && hasData.transitioning) return\n      actives.collapse('hide')\n      hasData || actives.data('bs.collapse', null)\n    }\n\n    var dimension = this.dimension()\n\n    this.$element\n      .removeClass('collapse')\n      .addClass('collapsing')\n      [dimension](0)\n\n    this.transitioning = 1\n\n    var complete = function () {\n      this.$element\n        .removeClass('collapsing')\n        .addClass('collapse in')\n        [dimension]('auto')\n      this.transitioning = 0\n      this.$element.trigger('shown.bs.collapse')\n    }\n\n    if (!$.support.transition) return complete.call(this)\n\n    var scrollSize = $.camelCase(['scroll', dimension].join('-'))\n\n    this.$element\n      .one($.support.transition.end, $.proxy(complete, this))\n      .emulateTransitionEnd(350)\n      [dimension](this.$element[0][scrollSize])\n  }\n\n  Collapse.prototype.hide = function () {\n    if (this.transitioning || !this.$element.hasClass('in')) return\n\n    var startEvent = $.Event('hide.bs.collapse')\n    this.$element.trigger(startEvent)\n    if (startEvent.isDefaultPrevented()) return\n\n    var dimension = this.dimension()\n\n    this.$element\n      [dimension](this.$element[dimension]())\n      [0].offsetHeight\n\n    this.$element\n      .addClass('collapsing')\n      .removeClass('collapse')\n      .removeClass('in')\n\n    this.transitioning = 1\n\n    var complete = function () {\n      this.transitioning = 0\n      this.$element\n        .trigger('hidden.bs.collapse')\n        .removeClass('collapsing')\n        .addClass('collapse')\n    }\n\n    if (!$.support.transition) return complete.call(this)\n\n    this.$element\n      [dimension](0)\n      .one($.support.transition.end, $.proxy(complete, this))\n      .emulateTransitionEnd(350)\n  }\n\n  Collapse.prototype.toggle = function () {\n    this[this.$element.hasClass('in') ? 'hide' : 'show']()\n  }\n\n\n  // COLLAPSE PLUGIN DEFINITION\n  // ==========================\n\n  var old = $.fn.collapse\n\n  $.fn.collapse = function (option) {\n    return this.each(function () {\n      var $this   = $(this)\n      var data    = $this.data('bs.collapse')\n      var options = $.extend({}, Collapse.DEFAULTS, $this.data(), typeof option == 'object' && option)\n\n      if (!data && options.toggle && option == 'show') option = !option\n      if (!data) $this.data('bs.collapse', (data = new Collapse(this, options)))\n      if (typeof option == 'string') data[option]()\n    })\n  }\n\n  $.fn.collapse.Constructor = Collapse\n\n\n  // COLLAPSE NO CONFLICT\n  // ====================\n\n  $.fn.collapse.noConflict = function () {\n    $.fn.collapse = old\n    return this\n  }\n\n\n  // COLLAPSE DATA-API\n  // =================\n\n  $(document).on('click.bs.collapse.data-api', '[data-toggle=collapse]', function (e) {\n    var $this   = $(this), href\n    var target  = $this.attr('data-target')\n        || e.preventDefault()\n        || (href = $this.attr('href')) && href.replace(/.*(?=#[^\\s]+$)/, '') //strip for ie7\n    var $target = $(target)\n    var data    = $target.data('bs.collapse')\n    var option  = data ? 'toggle' : $this.data()\n    var parent  = $this.attr('data-parent')\n    var $parent = parent && $(parent)\n\n    if (!data || !data.transitioning) {\n      if ($parent) $parent.find('[data-toggle=collapse][data-parent=\"' + parent + '\"]').not($this).addClass('collapsed')\n      $this[$target.hasClass('in') ? 'addClass' : 'removeClass']('collapsed')\n    }\n\n    $target.collapse(option)\n  })\n\n}(jQuery);\n\n/* ========================================================================\n * Bootstrap: dropdown.js v3.1.0\n * http://getbootstrap.com/javascript/#dropdowns\n * ========================================================================\n * Copyright 2011-2014 Twitter, Inc.\n * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)\n * ======================================================================== */\n\n\n+function ($) {\n  'use strict';\n\n  // DROPDOWN CLASS DEFINITION\n  // =========================\n\n  var backdrop = '.dropdown-backdrop'\n  var toggle   = '[data-toggle=dropdown]'\n  var Dropdown = function (element) {\n    $(element).on('click.bs.dropdown', this.toggle)\n  }\n\n  Dropdown.prototype.toggle = function (e) {\n    var $this = $(this)\n\n    if ($this.is('.disabled, :disabled')) return\n\n    var $parent  = getParent($this)\n    var isActive = $parent.hasClass('open')\n\n    clearMenus()\n\n    if (!isActive) {\n      if ('ontouchstart' in document.documentElement && !$parent.closest('.navbar-nav').length) {\n        // if mobile we use a backdrop because click events don't delegate\n        $('
').insertAfter($(this)).on('click', clearMenus)\n      }\n\n      var relatedTarget = { relatedTarget: this }\n      $parent.trigger(e = $.Event('show.bs.dropdown', relatedTarget))\n\n      if (e.isDefaultPrevented()) return\n\n      $parent\n        .toggleClass('open')\n        .trigger('shown.bs.dropdown', relatedTarget)\n\n      $this.focus()\n    }\n\n    return false\n  }\n\n  Dropdown.prototype.keydown = function (e) {\n    if (!/(38|40|27)/.test(e.keyCode)) return\n\n    var $this = $(this)\n\n    e.preventDefault()\n    e.stopPropagation()\n\n    if ($this.is('.disabled, :disabled')) return\n\n    var $parent  = getParent($this)\n    var isActive = $parent.hasClass('open')\n\n    if (!isActive || (isActive && e.keyCode == 27)) {\n      if (e.which == 27) $parent.find(toggle).focus()\n      return $this.click()\n    }\n\n    var desc = ' li:not(.divider):visible a'\n    var $items = $parent.find('[role=menu]' + desc + ', [role=listbox]' + desc)\n\n    if (!$items.length) return\n\n    var index = $items.index($items.filter(':focus'))\n\n    if (e.keyCode == 38 && index > 0)                 index--                        // up\n    if (e.keyCode == 40 && index < $items.length - 1) index++                        // down\n    if (!~index)                                      index = 0\n\n    $items.eq(index).focus()\n  }\n\n  function clearMenus(e) {\n    $(backdrop).remove()\n    $(toggle).each(function () {\n      var $parent = getParent($(this))\n      var relatedTarget = { relatedTarget: this }\n      if (!$parent.hasClass('open')) return\n      $parent.trigger(e = $.Event('hide.bs.dropdown', relatedTarget))\n      if (e.isDefaultPrevented()) return\n      $parent.removeClass('open').trigger('hidden.bs.dropdown', relatedTarget)\n    })\n  }\n\n  function getParent($this) {\n    var selector = $this.attr('data-target')\n\n    if (!selector) {\n      selector = $this.attr('href')\n      selector = selector && /#[A-Za-z]/.test(selector) && selector.replace(/.*(?=#[^\\s]*$)/, '') //strip for ie7\n    }\n\n    var $parent = selector && $(selector)\n\n    return $parent && $parent.length ? $parent : $this.parent()\n  }\n\n\n  // DROPDOWN PLUGIN DEFINITION\n  // ==========================\n\n  var old = $.fn.dropdown\n\n  $.fn.dropdown = function (option) {\n    return this.each(function () {\n      var $this = $(this)\n      var data  = $this.data('bs.dropdown')\n\n      if (!data) $this.data('bs.dropdown', (data = new Dropdown(this)))\n      if (typeof option == 'string') data[option].call($this)\n    })\n  }\n\n  $.fn.dropdown.Constructor = Dropdown\n\n\n  // DROPDOWN NO CONFLICT\n  // ====================\n\n  $.fn.dropdown.noConflict = function () {\n    $.fn.dropdown = old\n    return this\n  }\n\n\n  // APPLY TO STANDARD DROPDOWN ELEMENTS\n  // ===================================\n\n  $(document)\n    .on('click.bs.dropdown.data-api', clearMenus)\n    .on('click.bs.dropdown.data-api', '.dropdown form', function (e) { e.stopPropagation() })\n    .on('click.bs.dropdown.data-api', toggle, Dropdown.prototype.toggle)\n    .on('keydown.bs.dropdown.data-api', toggle + ', [role=menu], [role=listbox]', Dropdown.prototype.keydown)\n\n}(jQuery);\n\n/* ========================================================================\n * Bootstrap: modal.js v3.1.0\n * http://getbootstrap.com/javascript/#modals\n * ========================================================================\n * Copyright 2011-2014 Twitter, Inc.\n * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)\n * ======================================================================== */\n\n\n+function ($) {\n  'use strict';\n\n  // MODAL CLASS DEFINITION\n  // ======================\n\n  var Modal = function (element, options) {\n    this.options   = options\n    this.$element  = $(element)\n    this.$backdrop =\n    this.isShown   = null\n\n    if (this.options.remote) {\n      this.$element\n        .find('.modal-content')\n        .load(this.options.remote, $.proxy(function () {\n          this.$element.trigger('loaded.bs.modal')\n        }, this))\n    }\n  }\n\n  Modal.DEFAULTS = {\n    backdrop: true,\n    keyboard: true,\n    show: true\n  }\n\n  Modal.prototype.toggle = function (_relatedTarget) {\n    return this[!this.isShown ? 'show' : 'hide'](_relatedTarget)\n  }\n\n  Modal.prototype.show = function (_relatedTarget) {\n    var that = this\n    var e    = $.Event('show.bs.modal', { relatedTarget: _relatedTarget })\n\n    this.$element.trigger(e)\n\n    if (this.isShown || e.isDefaultPrevented()) return\n\n    this.isShown = true\n\n    this.escape()\n\n    this.$element.on('click.dismiss.bs.modal', '[data-dismiss=\"modal\"]', $.proxy(this.hide, this))\n\n    this.backdrop(function () {\n      var transition = $.support.transition && that.$element.hasClass('fade')\n\n      if (!that.$element.parent().length) {\n        that.$element.appendTo(document.body) // don't move modals dom position\n      }\n\n      that.$element\n        .show()\n        .scrollTop(0)\n\n      if (transition) {\n        that.$element[0].offsetWidth // force reflow\n      }\n\n      that.$element\n        .addClass('in')\n        .attr('aria-hidden', false)\n\n      that.enforceFocus()\n\n      var e = $.Event('shown.bs.modal', { relatedTarget: _relatedTarget })\n\n      transition ?\n        that.$element.find('.modal-dialog') // wait for modal to slide in\n          .one($.support.transition.end, function () {\n            that.$element.focus().trigger(e)\n          })\n          .emulateTransitionEnd(300) :\n        that.$element.focus().trigger(e)\n    })\n  }\n\n  Modal.prototype.hide = function (e) {\n    if (e) e.preventDefault()\n\n    e = $.Event('hide.bs.modal')\n\n    this.$element.trigger(e)\n\n    if (!this.isShown || e.isDefaultPrevented()) return\n\n    this.isShown = false\n\n    this.escape()\n\n    $(document).off('focusin.bs.modal')\n\n    this.$element\n      .removeClass('in')\n      .attr('aria-hidden', true)\n      .off('click.dismiss.bs.modal')\n\n    $.support.transition && this.$element.hasClass('fade') ?\n      this.$element\n        .one($.support.transition.end, $.proxy(this.hideModal, this))\n        .emulateTransitionEnd(300) :\n      this.hideModal()\n  }\n\n  Modal.prototype.enforceFocus = function () {\n    $(document)\n      .off('focusin.bs.modal') // guard against infinite focus loop\n      .on('focusin.bs.modal', $.proxy(function (e) {\n        if (this.$element[0] !== e.target && !this.$element.has(e.target).length) {\n          this.$element.focus()\n        }\n      }, this))\n  }\n\n  Modal.prototype.escape = function () {\n    if (this.isShown && this.options.keyboard) {\n      this.$element.on('keyup.dismiss.bs.modal', $.proxy(function (e) {\n        e.which == 27 && this.hide()\n      }, this))\n    } else if (!this.isShown) {\n      this.$element.off('keyup.dismiss.bs.modal')\n    }\n  }\n\n  Modal.prototype.hideModal = function () {\n    var that = this\n    this.$element.hide()\n    this.backdrop(function () {\n      that.removeBackdrop()\n      that.$element.trigger('hidden.bs.modal')\n    })\n  }\n\n  Modal.prototype.removeBackdrop = function () {\n    this.$backdrop && this.$backdrop.remove()\n    this.$backdrop = null\n  }\n\n  Modal.prototype.backdrop = function (callback) {\n    var animate = this.$element.hasClass('fade') ? 'fade' : ''\n\n    if (this.isShown && this.options.backdrop) {\n      var doAnimate = $.support.transition && animate\n\n      this.$backdrop = $('
')\n        .appendTo(document.body)\n\n      this.$element.on('click.dismiss.bs.modal', $.proxy(function (e) {\n        if (e.target !== e.currentTarget) return\n        this.options.backdrop == 'static'\n          ? this.$element[0].focus.call(this.$element[0])\n          : this.hide.call(this)\n      }, this))\n\n      if (doAnimate) this.$backdrop[0].offsetWidth // force reflow\n\n      this.$backdrop.addClass('in')\n\n      if (!callback) return\n\n      doAnimate ?\n        this.$backdrop\n          .one($.support.transition.end, callback)\n          .emulateTransitionEnd(150) :\n        callback()\n\n    } else if (!this.isShown && this.$backdrop) {\n      this.$backdrop.removeClass('in')\n\n      $.support.transition && this.$element.hasClass('fade') ?\n        this.$backdrop\n          .one($.support.transition.end, callback)\n          .emulateTransitionEnd(150) :\n        callback()\n\n    } else if (callback) {\n      callback()\n    }\n  }\n\n\n  // MODAL PLUGIN DEFINITION\n  // =======================\n\n  var old = $.fn.modal\n\n  $.fn.modal = function (option, _relatedTarget) {\n    return this.each(function () {\n      var $this   = $(this)\n      var data    = $this.data('bs.modal')\n      var options = $.extend({}, Modal.DEFAULTS, $this.data(), typeof option == 'object' && option)\n\n      if (!data) $this.data('bs.modal', (data = new Modal(this, options)))\n      if (typeof option == 'string') data[option](_relatedTarget)\n      else if (options.show) data.show(_relatedTarget)\n    })\n  }\n\n  $.fn.modal.Constructor = Modal\n\n\n  // MODAL NO CONFLICT\n  // =================\n\n  $.fn.modal.noConflict = function () {\n    $.fn.modal = old\n    return this\n  }\n\n\n  // MODAL DATA-API\n  // ==============\n\n  $(document).on('click.bs.modal.data-api', '[data-toggle=\"modal\"]', function (e) {\n    var $this   = $(this)\n    var href    = $this.attr('href')\n    var $target = $($this.attr('data-target') || (href && href.replace(/.*(?=#[^\\s]+$)/, ''))) //strip for ie7\n    var option  = $target.data('bs.modal') ? 'toggle' : $.extend({ remote: !/#/.test(href) && href }, $target.data(), $this.data())\n\n    if ($this.is('a')) e.preventDefault()\n\n    $target\n      .modal(option, this)\n      .one('hide', function () {\n        $this.is(':visible') && $this.focus()\n      })\n  })\n\n  $(document)\n    .on('show.bs.modal', '.modal', function () { $(document.body).addClass('modal-open') })\n    .on('hidden.bs.modal', '.modal', function () { $(document.body).removeClass('modal-open') })\n\n}(jQuery);\n\n/* ========================================================================\n * Bootstrap: tooltip.js v3.1.0\n * http://getbootstrap.com/javascript/#tooltip\n * Inspired by the original jQuery.tipsy by Jason Frame\n * ========================================================================\n * Copyright 2011-2014 Twitter, Inc.\n * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)\n * ======================================================================== */\n\n\n+function ($) {\n  'use strict';\n\n  // TOOLTIP PUBLIC CLASS DEFINITION\n  // ===============================\n\n  var Tooltip = function (element, options) {\n    this.type       =\n    this.options    =\n    this.enabled    =\n    this.timeout    =\n    this.hoverState =\n    this.$element   = null\n\n    this.init('tooltip', element, options)\n  }\n\n  Tooltip.DEFAULTS = {\n    animation: true,\n    placement: 'top',\n    selector: false,\n    template: '
',\n    trigger: 'hover focus',\n    title: '',\n    delay: 0,\n    html: false,\n    container: false\n  }\n\n  Tooltip.prototype.init = function (type, element, options) {\n    this.enabled  = true\n    this.type     = type\n    this.$element = $(element)\n    this.options  = this.getOptions(options)\n\n    var triggers = this.options.trigger.split(' ')\n\n    for (var i = triggers.length; i--;) {\n      var trigger = triggers[i]\n\n      if (trigger == 'click') {\n        this.$element.on('click.' + this.type, this.options.selector, $.proxy(this.toggle, this))\n      } else if (trigger != 'manual') {\n        var eventIn  = trigger == 'hover' ? 'mouseenter' : 'focusin'\n        var eventOut = trigger == 'hover' ? 'mouseleave' : 'focusout'\n\n        this.$element.on(eventIn  + '.' + this.type, this.options.selector, $.proxy(this.enter, this))\n        this.$element.on(eventOut + '.' + this.type, this.options.selector, $.proxy(this.leave, this))\n      }\n    }\n\n    this.options.selector ?\n      (this._options = $.extend({}, this.options, { trigger: 'manual', selector: '' })) :\n      this.fixTitle()\n  }\n\n  Tooltip.prototype.getDefaults = function () {\n    return Tooltip.DEFAULTS\n  }\n\n  Tooltip.prototype.getOptions = function (options) {\n    options = $.extend({}, this.getDefaults(), this.$element.data(), options)\n\n    if (options.delay && typeof options.delay == 'number') {\n      options.delay = {\n        show: options.delay,\n        hide: options.delay\n      }\n    }\n\n    return options\n  }\n\n  Tooltip.prototype.getDelegateOptions = function () {\n    var options  = {}\n    var defaults = this.getDefaults()\n\n    this._options && $.each(this._options, function (key, value) {\n      if (defaults[key] != value) options[key] = value\n    })\n\n    return options\n  }\n\n  Tooltip.prototype.enter = function (obj) {\n    var self = obj instanceof this.constructor ?\n      obj : $(obj.currentTarget)[this.type](this.getDelegateOptions()).data('bs.' + this.type)\n\n    clearTimeout(self.timeout)\n\n    self.hoverState = 'in'\n\n    if (!self.options.delay || !self.options.delay.show) return self.show()\n\n    self.timeout = setTimeout(function () {\n      if (self.hoverState == 'in') self.show()\n    }, self.options.delay.show)\n  }\n\n  Tooltip.prototype.leave = function (obj) {\n    var self = obj instanceof this.constructor ?\n      obj : $(obj.currentTarget)[this.type](this.getDelegateOptions()).data('bs.' + this.type)\n\n    clearTimeout(self.timeout)\n\n    self.hoverState = 'out'\n\n    if (!self.options.delay || !self.options.delay.hide) return self.hide()\n\n    self.timeout = setTimeout(function () {\n      if (self.hoverState == 'out') self.hide()\n    }, self.options.delay.hide)\n  }\n\n  Tooltip.prototype.show = function () {\n    var e = $.Event('show.bs.' + this.type)\n\n    if (this.hasContent() && this.enabled) {\n      this.$element.trigger(e)\n\n      if (e.isDefaultPrevented()) return\n      var that = this;\n\n      var $tip = this.tip()\n\n      this.setContent()\n\n      if (this.options.animation) $tip.addClass('fade')\n\n      var placement = typeof this.options.placement == 'function' ?\n        this.options.placement.call(this, $tip[0], this.$element[0]) :\n        this.options.placement\n\n      var autoToken = /\\s?auto?\\s?/i\n      var autoPlace = autoToken.test(placement)\n      if (autoPlace) placement = placement.replace(autoToken, '') || 'top'\n\n      $tip\n        .detach()\n        .css({ top: 0, left: 0, display: 'block' })\n        .addClass(placement)\n\n      this.options.container ? $tip.appendTo(this.options.container) : $tip.insertAfter(this.$element)\n\n      var pos          = this.getPosition()\n      var actualWidth  = $tip[0].offsetWidth\n      var actualHeight = $tip[0].offsetHeight\n\n      if (autoPlace) {\n        var $parent = this.$element.parent()\n\n        var orgPlacement = placement\n        var docScroll    = document.documentElement.scrollTop || document.body.scrollTop\n        var parentWidth  = this.options.container == 'body' ? window.innerWidth  : $parent.outerWidth()\n        var parentHeight = this.options.container == 'body' ? window.innerHeight : $parent.outerHeight()\n        var parentLeft   = this.options.container == 'body' ? 0 : $parent.offset().left\n\n        placement = placement == 'bottom' && pos.top   + pos.height  + actualHeight - docScroll > parentHeight  ? 'top'    :\n                    placement == 'top'    && pos.top   - docScroll   - actualHeight < 0                         ? 'bottom' :\n                    placement == 'right'  && pos.right + actualWidth > parentWidth                              ? 'left'   :\n                    placement == 'left'   && pos.left  - actualWidth < parentLeft                               ? 'right'  :\n                    placement\n\n        $tip\n          .removeClass(orgPlacement)\n          .addClass(placement)\n      }\n\n      var calculatedOffset = this.getCalculatedOffset(placement, pos, actualWidth, actualHeight)\n\n      this.applyPlacement(calculatedOffset, placement)\n      this.hoverState = null\n\n      var complete = function() {\n        that.$element.trigger('shown.bs.' + that.type)\n      }\n\n      $.support.transition && this.$tip.hasClass('fade') ?\n        $tip\n          .one($.support.transition.end, complete)\n          .emulateTransitionEnd(150) :\n        complete()\n    }\n  }\n\n  Tooltip.prototype.applyPlacement = function (offset, placement) {\n    var replace\n    var $tip   = this.tip()\n    var width  = $tip[0].offsetWidth\n    var height = $tip[0].offsetHeight\n\n    // manually read margins because getBoundingClientRect includes difference\n    var marginTop = parseInt($tip.css('margin-top'), 10)\n    var marginLeft = parseInt($tip.css('margin-left'), 10)\n\n    // we must check for NaN for ie 8/9\n    if (isNaN(marginTop))  marginTop  = 0\n    if (isNaN(marginLeft)) marginLeft = 0\n\n    offset.top  = offset.top  + marginTop\n    offset.left = offset.left + marginLeft\n\n    // $.fn.offset doesn't round pixel values\n    // so we use setOffset directly with our own function B-0\n    $.offset.setOffset($tip[0], $.extend({\n      using: function (props) {\n        $tip.css({\n          top: Math.round(props.top),\n          left: Math.round(props.left)\n        })\n      }\n    }, offset), 0)\n\n    $tip.addClass('in')\n\n    // check to see if placing tip in new offset caused the tip to resize itself\n    var actualWidth  = $tip[0].offsetWidth\n    var actualHeight = $tip[0].offsetHeight\n\n    if (placement == 'top' && actualHeight != height) {\n      replace = true\n      offset.top = offset.top + height - actualHeight\n    }\n\n    if (/bottom|top/.test(placement)) {\n      var delta = 0\n\n      if (offset.left < 0) {\n        delta       = offset.left * -2\n        offset.left = 0\n\n        $tip.offset(offset)\n\n        actualWidth  = $tip[0].offsetWidth\n        actualHeight = $tip[0].offsetHeight\n      }\n\n      this.replaceArrow(delta - width + actualWidth, actualWidth, 'left')\n    } else {\n      this.replaceArrow(actualHeight - height, actualHeight, 'top')\n    }\n\n    if (replace) $tip.offset(offset)\n  }\n\n  Tooltip.prototype.replaceArrow = function (delta, dimension, position) {\n    this.arrow().css(position, delta ? (50 * (1 - delta / dimension) + '%') : '')\n  }\n\n  Tooltip.prototype.setContent = function () {\n    var $tip  = this.tip()\n    var title = this.getTitle()\n\n    $tip.find('.tooltip-inner')[this.options.html ? 'html' : 'text'](title)\n    $tip.removeClass('fade in top bottom left right')\n  }\n\n  Tooltip.prototype.hide = function () {\n    var that = this\n    var $tip = this.tip()\n    var e    = $.Event('hide.bs.' + this.type)\n\n    function complete() {\n      if (that.hoverState != 'in') $tip.detach()\n      that.$element.trigger('hidden.bs.' + that.type)\n    }\n\n    this.$element.trigger(e)\n\n    if (e.isDefaultPrevented()) return\n\n    $tip.removeClass('in')\n\n    $.support.transition && this.$tip.hasClass('fade') ?\n      $tip\n        .one($.support.transition.end, complete)\n        .emulateTransitionEnd(150) :\n      complete()\n\n    this.hoverState = null\n\n    return this\n  }\n\n  Tooltip.prototype.fixTitle = function () {\n    var $e = this.$element\n    if ($e.attr('title') || typeof($e.attr('data-original-title')) != 'string') {\n      $e.attr('data-original-title', $e.attr('title') || '').attr('title', '')\n    }\n  }\n\n  Tooltip.prototype.hasContent = function () {\n    return this.getTitle()\n  }\n\n  Tooltip.prototype.getPosition = function () {\n    var el = this.$element[0]\n    return $.extend({}, (typeof el.getBoundingClientRect == 'function') ? el.getBoundingClientRect() : {\n      width: el.offsetWidth,\n      height: el.offsetHeight\n    }, this.$element.offset())\n  }\n\n  Tooltip.prototype.getCalculatedOffset = function (placement, pos, actualWidth, actualHeight) {\n    return placement == 'bottom' ? { top: pos.top + pos.height,   left: pos.left + pos.width / 2 - actualWidth / 2  } :\n           placement == 'top'    ? { top: pos.top - actualHeight, left: pos.left + pos.width / 2 - actualWidth / 2  } :\n           placement == 'left'   ? { top: pos.top + pos.height / 2 - actualHeight / 2, left: pos.left - actualWidth } :\n        /* placement == 'right' */ { top: pos.top + pos.height / 2 - actualHeight / 2, left: pos.left + pos.width   }\n  }\n\n  Tooltip.prototype.getTitle = function () {\n    var title\n    var $e = this.$element\n    var o  = this.options\n\n    title = $e.attr('data-original-title')\n      || (typeof o.title == 'function' ? o.title.call($e[0]) :  o.title)\n\n    return title\n  }\n\n  Tooltip.prototype.tip = function () {\n    return this.$tip = this.$tip || $(this.options.template)\n  }\n\n  Tooltip.prototype.arrow = function () {\n    return this.$arrow = this.$arrow || this.tip().find('.tooltip-arrow')\n  }\n\n  Tooltip.prototype.validate = function () {\n    if (!this.$element[0].parentNode) {\n      this.hide()\n      this.$element = null\n      this.options  = null\n    }\n  }\n\n  Tooltip.prototype.enable = function () {\n    this.enabled = true\n  }\n\n  Tooltip.prototype.disable = function () {\n    this.enabled = false\n  }\n\n  Tooltip.prototype.toggleEnabled = function () {\n    this.enabled = !this.enabled\n  }\n\n  Tooltip.prototype.toggle = function (e) {\n    var self = e ? $(e.currentTarget)[this.type](this.getDelegateOptions()).data('bs.' + this.type) : this\n    self.tip().hasClass('in') ? self.leave(self) : self.enter(self)\n  }\n\n  Tooltip.prototype.destroy = function () {\n    clearTimeout(this.timeout)\n    this.hide().$element.off('.' + this.type).removeData('bs.' + this.type)\n  }\n\n\n  // TOOLTIP PLUGIN DEFINITION\n  // =========================\n\n  var old = $.fn.tooltip\n\n  $.fn.tooltip = function (option) {\n    return this.each(function () {\n      var $this   = $(this)\n      var data    = $this.data('bs.tooltip')\n      var options = typeof option == 'object' && option\n\n      if (!data && option == 'destroy') return\n      if (!data) $this.data('bs.tooltip', (data = new Tooltip(this, options)))\n      if (typeof option == 'string') data[option]()\n    })\n  }\n\n  $.fn.tooltip.Constructor = Tooltip\n\n\n  // TOOLTIP NO CONFLICT\n  // ===================\n\n  $.fn.tooltip.noConflict = function () {\n    $.fn.tooltip = old\n    return this\n  }\n\n}(jQuery);\n\n/* ========================================================================\n * Bootstrap: popover.js v3.1.0\n * http://getbootstrap.com/javascript/#popovers\n * ========================================================================\n * Copyright 2011-2014 Twitter, Inc.\n * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)\n * ======================================================================== */\n\n\n+function ($) {\n  'use strict';\n\n  // POPOVER PUBLIC CLASS DEFINITION\n  // ===============================\n\n  var Popover = function (element, options) {\n    this.init('popover', element, options)\n  }\n\n  if (!$.fn.tooltip) throw new Error('Popover requires tooltip.js')\n\n  Popover.DEFAULTS = $.extend({}, $.fn.tooltip.Constructor.DEFAULTS, {\n    placement: 'right',\n    trigger: 'click',\n    content: '',\n    template: '
'\n  })\n\n\n  // NOTE: POPOVER EXTENDS tooltip.js\n  // ================================\n\n  Popover.prototype = $.extend({}, $.fn.tooltip.Constructor.prototype)\n\n  Popover.prototype.constructor = Popover\n\n  Popover.prototype.getDefaults = function () {\n    return Popover.DEFAULTS\n  }\n\n  Popover.prototype.setContent = function () {\n    var $tip    = this.tip()\n    var title   = this.getTitle()\n    var content = this.getContent()\n\n    $tip.find('.popover-title')[this.options.html ? 'html' : 'text'](title)\n    $tip.find('.popover-content')[ // we use append for html objects to maintain js events\n      this.options.html ? (typeof content == 'string' ? 'html' : 'append') : 'text'\n    ](content)\n\n    $tip.removeClass('fade top bottom left right in')\n\n    // IE8 doesn't accept hiding via the `:empty` pseudo selector, we have to do\n    // this manually by checking the contents.\n    if (!$tip.find('.popover-title').html()) $tip.find('.popover-title').hide()\n  }\n\n  Popover.prototype.hasContent = function () {\n    return this.getTitle() || this.getContent()\n  }\n\n  Popover.prototype.getContent = function () {\n    var $e = this.$element\n    var o  = this.options\n\n    return $e.attr('data-content')\n      || (typeof o.content == 'function' ?\n            o.content.call($e[0]) :\n            o.content)\n  }\n\n  Popover.prototype.arrow = function () {\n    return this.$arrow = this.$arrow || this.tip().find('.arrow')\n  }\n\n  Popover.prototype.tip = function () {\n    if (!this.$tip) this.$tip = $(this.options.template)\n    return this.$tip\n  }\n\n\n  // POPOVER PLUGIN DEFINITION\n  // =========================\n\n  var old = $.fn.popover\n\n  $.fn.popover = function (option) {\n    return this.each(function () {\n      var $this   = $(this)\n      var data    = $this.data('bs.popover')\n      var options = typeof option == 'object' && option\n\n      if (!data && option == 'destroy') return\n      if (!data) $this.data('bs.popover', (data = new Popover(this, options)))\n      if (typeof option == 'string') data[option]()\n    })\n  }\n\n  $.fn.popover.Constructor = Popover\n\n\n  // POPOVER NO CONFLICT\n  // ===================\n\n  $.fn.popover.noConflict = function () {\n    $.fn.popover = old\n    return this\n  }\n\n}(jQuery);\n\n/* ========================================================================\n * Bootstrap: scrollspy.js v3.1.0\n * http://getbootstrap.com/javascript/#scrollspy\n * ========================================================================\n * Copyright 2011-2014 Twitter, Inc.\n * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)\n * ======================================================================== */\n\n\n+function ($) {\n  'use strict';\n\n  // SCROLLSPY CLASS DEFINITION\n  // ==========================\n\n  function ScrollSpy(element, options) {\n    var href\n    var process  = $.proxy(this.process, this)\n\n    this.$element       = $(element).is('body') ? $(window) : $(element)\n    this.$body          = $('body')\n    this.$scrollElement = this.$element.on('scroll.bs.scroll-spy.data-api', process)\n    this.options        = $.extend({}, ScrollSpy.DEFAULTS, options)\n    this.selector       = (this.options.target\n      || ((href = $(element).attr('href')) && href.replace(/.*(?=#[^\\s]+$)/, '')) //strip for ie7\n      || '') + ' .nav li > a'\n    this.offsets        = $([])\n    this.targets        = $([])\n    this.activeTarget   = null\n\n    this.refresh()\n    this.process()\n  }\n\n  ScrollSpy.DEFAULTS = {\n    offset: 10\n  }\n\n  ScrollSpy.prototype.refresh = function () {\n    var offsetMethod = this.$element[0] == window ? 'offset' : 'position'\n\n    this.offsets = $([])\n    this.targets = $([])\n\n    var self     = this\n    var $targets = this.$body\n      .find(this.selector)\n      .map(function () {\n        var $el   = $(this)\n        var href  = $el.data('target') || $el.attr('href')\n        var $href = /^#./.test(href) && $(href)\n\n        return ($href\n          && $href.length\n          && $href.is(':visible')\n          && [[ $href[offsetMethod]().top + (!$.isWindow(self.$scrollElement.get(0)) && self.$scrollElement.scrollTop()), href ]]) || null\n      })\n      .sort(function (a, b) { return a[0] - b[0] })\n      .each(function () {\n        self.offsets.push(this[0])\n        self.targets.push(this[1])\n      })\n  }\n\n  ScrollSpy.prototype.process = function () {\n    var scrollTop    = this.$scrollElement.scrollTop() + this.options.offset\n    var scrollHeight = this.$scrollElement[0].scrollHeight || this.$body[0].scrollHeight\n    var maxScroll    = scrollHeight - this.$scrollElement.height()\n    var offsets      = this.offsets\n    var targets      = this.targets\n    var activeTarget = this.activeTarget\n    var i\n\n    if (scrollTop >= maxScroll) {\n      return activeTarget != (i = targets.last()[0]) && this.activate(i)\n    }\n\n    if (activeTarget && scrollTop <= offsets[0]) {\n      return activeTarget != (i = targets[0]) && this.activate(i)\n    }\n\n    for (i = offsets.length; i--;) {\n      activeTarget != targets[i]\n        && scrollTop >= offsets[i]\n        && (!offsets[i + 1] || scrollTop <= offsets[i + 1])\n        && this.activate( targets[i] )\n    }\n  }\n\n  ScrollSpy.prototype.activate = function (target) {\n    this.activeTarget = target\n\n    $(this.selector)\n      .parentsUntil(this.options.target, '.active')\n      .removeClass('active')\n\n    var selector = this.selector +\n        '[data-target=\"' + target + '\"],' +\n        this.selector + '[href=\"' + target + '\"]'\n\n    var active = $(selector)\n      .parents('li')\n      .addClass('active')\n\n    if (active.parent('.dropdown-menu').length) {\n      active = active\n        .closest('li.dropdown')\n        .addClass('active')\n    }\n\n    active.trigger('activate.bs.scrollspy')\n  }\n\n\n  // SCROLLSPY PLUGIN DEFINITION\n  // ===========================\n\n  var old = $.fn.scrollspy\n\n  $.fn.scrollspy = function (option) {\n    return this.each(function () {\n      var $this   = $(this)\n      var data    = $this.data('bs.scrollspy')\n      var options = typeof option == 'object' && option\n\n      if (!data) $this.data('bs.scrollspy', (data = new ScrollSpy(this, options)))\n      if (typeof option == 'string') data[option]()\n    })\n  }\n\n  $.fn.scrollspy.Constructor = ScrollSpy\n\n\n  // SCROLLSPY NO CONFLICT\n  // =====================\n\n  $.fn.scrollspy.noConflict = function () {\n    $.fn.scrollspy = old\n    return this\n  }\n\n\n  // SCROLLSPY DATA-API\n  // ==================\n\n  $(window).on('load', function () {\n    $('[data-spy=\"scroll\"]').each(function () {\n      var $spy = $(this)\n      $spy.scrollspy($spy.data())\n    })\n  })\n\n}(jQuery);\n\n/* ========================================================================\n * Bootstrap: tab.js v3.1.0\n * http://getbootstrap.com/javascript/#tabs\n * ========================================================================\n * Copyright 2011-2014 Twitter, Inc.\n * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)\n * ======================================================================== */\n\n\n+function ($) {\n  'use strict';\n\n  // TAB CLASS DEFINITION\n  // ====================\n\n  var Tab = function (element) {\n    this.element = $(element)\n  }\n\n  Tab.prototype.show = function () {\n    var $this    = this.element\n    var $ul      = $this.closest('ul:not(.dropdown-menu)')\n    var selector = $this.data('target')\n\n    if (!selector) {\n      selector = $this.attr('href')\n      selector = selector && selector.replace(/.*(?=#[^\\s]*$)/, '') //strip for ie7\n    }\n\n    if ($this.parent('li').hasClass('active')) return\n\n    var previous = $ul.find('.active:last a')[0]\n    var e        = $.Event('show.bs.tab', {\n      relatedTarget: previous\n    })\n\n    $this.trigger(e)\n\n    if (e.isDefaultPrevented()) return\n\n    var $target = $(selector)\n\n    this.activate($this.parent('li'), $ul)\n    this.activate($target, $target.parent(), function () {\n      $this.trigger({\n        type: 'shown.bs.tab',\n        relatedTarget: previous\n      })\n    })\n  }\n\n  Tab.prototype.activate = function (element, container, callback) {\n    var $active    = container.find('> .active')\n    var transition = callback\n      && $.support.transition\n      && $active.hasClass('fade')\n\n    function next() {\n      $active\n        .removeClass('active')\n        .find('> .dropdown-menu > .active')\n        .removeClass('active')\n\n      element.addClass('active')\n\n      if (transition) {\n        element[0].offsetWidth // reflow for transition\n        element.addClass('in')\n      } else {\n        element.removeClass('fade')\n      }\n\n      if (element.parent('.dropdown-menu')) {\n        element.closest('li.dropdown').addClass('active')\n      }\n\n      callback && callback()\n    }\n\n    transition ?\n      $active\n        .one($.support.transition.end, next)\n        .emulateTransitionEnd(150) :\n      next()\n\n    $active.removeClass('in')\n  }\n\n\n  // TAB PLUGIN DEFINITION\n  // =====================\n\n  var old = $.fn.tab\n\n  $.fn.tab = function ( option ) {\n    return this.each(function () {\n      var $this = $(this)\n      var data  = $this.data('bs.tab')\n\n      if (!data) $this.data('bs.tab', (data = new Tab(this)))\n      if (typeof option == 'string') data[option]()\n    })\n  }\n\n  $.fn.tab.Constructor = Tab\n\n\n  // TAB NO CONFLICT\n  // ===============\n\n  $.fn.tab.noConflict = function () {\n    $.fn.tab = old\n    return this\n  }\n\n\n  // TAB DATA-API\n  // ============\n\n  $(document).on('click.bs.tab.data-api', '[data-toggle=\"tab\"], [data-toggle=\"pill\"]', function (e) {\n    e.preventDefault()\n    $(this).tab('show')\n  })\n\n}(jQuery);\n\n/* ========================================================================\n * Bootstrap: affix.js v3.1.0\n * http://getbootstrap.com/javascript/#affix\n * ========================================================================\n * Copyright 2011-2014 Twitter, Inc.\n * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)\n * ======================================================================== */\n\n\n+function ($) {\n  'use strict';\n\n  // AFFIX CLASS DEFINITION\n  // ======================\n\n  var Affix = function (element, options) {\n    this.options = $.extend({}, Affix.DEFAULTS, options)\n    this.$window = $(window)\n      .on('scroll.bs.affix.data-api', $.proxy(this.checkPosition, this))\n      .on('click.bs.affix.data-api',  $.proxy(this.checkPositionWithEventLoop, this))\n\n    this.$element     = $(element)\n    this.affixed      =\n    this.unpin        =\n    this.pinnedOffset = null\n\n    this.checkPosition()\n  }\n\n  Affix.RESET = 'affix affix-top affix-bottom'\n\n  Affix.DEFAULTS = {\n    offset: 0\n  }\n\n  Affix.prototype.getPinnedOffset = function () {\n    if (this.pinnedOffset) return this.pinnedOffset\n    this.$element.removeClass(Affix.RESET).addClass('affix')\n    var scrollTop = this.$window.scrollTop()\n    var position  = this.$element.offset()\n    return (this.pinnedOffset = position.top - scrollTop)\n  }\n\n  Affix.prototype.checkPositionWithEventLoop = function () {\n    setTimeout($.proxy(this.checkPosition, this), 1)\n  }\n\n  Affix.prototype.checkPosition = function () {\n    if (!this.$element.is(':visible')) return\n\n    var scrollHeight = $(document).height()\n    var scrollTop    = this.$window.scrollTop()\n    var position     = this.$element.offset()\n    var offset       = this.options.offset\n    var offsetTop    = offset.top\n    var offsetBottom = offset.bottom\n\n    if (this.affixed == 'top') position.top += scrollTop\n\n    if (typeof offset != 'object')         offsetBottom = offsetTop = offset\n    if (typeof offsetTop == 'function')    offsetTop    = offset.top(this.$element)\n    if (typeof offsetBottom == 'function') offsetBottom = offset.bottom(this.$element)\n\n    var affix = this.unpin   != null && (scrollTop + this.unpin <= position.top) ? false :\n                offsetBottom != null && (position.top + this.$element.height() >= scrollHeight - offsetBottom) ? 'bottom' :\n                offsetTop    != null && (scrollTop <= offsetTop) ? 'top' : false\n\n    if (this.affixed === affix) return\n    if (this.unpin) this.$element.css('top', '')\n\n    var affixType = 'affix' + (affix ? '-' + affix : '')\n    var e         = $.Event(affixType + '.bs.affix')\n\n    this.$element.trigger(e)\n\n    if (e.isDefaultPrevented()) return\n\n    this.affixed = affix\n    this.unpin = affix == 'bottom' ? this.getPinnedOffset() : null\n\n    this.$element\n      .removeClass(Affix.RESET)\n      .addClass(affixType)\n      .trigger($.Event(affixType.replace('affix', 'affixed')))\n\n    if (affix == 'bottom') {\n      this.$element.offset({ top: scrollHeight - offsetBottom - this.$element.height() })\n    }\n  }\n\n\n  // AFFIX PLUGIN DEFINITION\n  // =======================\n\n  var old = $.fn.affix\n\n  $.fn.affix = function (option) {\n    return this.each(function () {\n      var $this   = $(this)\n      var data    = $this.data('bs.affix')\n      var options = typeof option == 'object' && option\n\n      if (!data) $this.data('bs.affix', (data = new Affix(this, options)))\n      if (typeof option == 'string') data[option]()\n    })\n  }\n\n  $.fn.affix.Constructor = Affix\n\n\n  // AFFIX NO CONFLICT\n  // =================\n\n  $.fn.affix.noConflict = function () {\n    $.fn.affix = old\n    return this\n  }\n\n\n  // AFFIX DATA-API\n  // ==============\n\n  $(window).on('load', function () {\n    $('[data-spy=\"affix\"]').each(function () {\n      var $spy = $(this)\n      var data = $spy.data()\n\n      data.offset = data.offset || {}\n\n      if (data.offsetBottom) data.offset.bottom = data.offsetBottom\n      if (data.offsetTop)    data.offset.top    = data.offsetTop\n\n      $spy.affix(data)\n    })\n  })\n\n}(jQuery);\n"
  },
  {
    "path": "doc/pysdk_pdoc.py",
    "content": "#!/usr/bin/env python3\n# Copyright (C) The Arvados Authors. All rights reserved.\n#\n# SPDX-License-Identifier: AGPL-3.0\n\"\"\"pysdk_pdoc.py - Run pdoc with extra rendering options\n\nThis script is a wrapper around the standard `pdoc` tool that enables the\n`admonitions` and `smarty-pants` extras for nicer rendering.\n\nIf run without arguments, it uses arguments to build the Arvados Python SDK\ndocumentation.\n\"\"\"\n\nimport collections\nimport functools\nimport os\nimport sys\n\nfrom pathlib import Path\n\ntry:\n    import pdoc.__main__\n    import pdoc.render_helpers\nexcept ImportError as err:\n    if __name__ == '__main__':\n        _imp_err = err\n    else:\n        raise\nelse:\n    _imp_err = None\n\ntry:\n    import arvados\nexcept ImportError:\n    DEFAULT_ARGLIST = []\nelse:\n    DEFAULT_ARGLIST = [\n        '--output-directory=sdk/python',\n        str(Path(arvados.__file__).parent),\n        # Because the module is prviate, pdoc does not build documentation for any\n        # of it. The exclusion below additionally prevents pdoc from hyperlinking\n        # references under arvados._internal that appear in method signatures, etc.\n        '!arvados._internal',\n    ]\n\nMD_EXTENSIONS = {\n    'admonitions': None,\n    'smarty-pants': None,\n}\n\ndef main(arglist=None):\n    if _imp_err is not None:\n        print(\"error: failed to import pdoc:\", _imp_err, file=sys.stderr)\n        return os.EX_SOFTWARE\n\n    # Configure pdoc to use extras we want.\n    pdoc.render_helpers.markdown_extensions = collections.ChainMap(\n        pdoc.render_helpers.markdown_extensions,\n        MD_EXTENSIONS,\n    )\n    pdoc.__main__.cli(arglist)\n    return os.EX_OK\n\nif __name__ == '__main__':\n    sys.exit(main(sys.argv[1:] or DEFAULT_ARGLIST))\n"
  },
  {
    "path": "doc/sdk/cli/index.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: sdk\nnavmenu: Command line tools (CLI SDK)\ntitle: \"Overview\"\n\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThe @arv@ CLI tool provide provides a convenient interface to manipulate API resources. Additionally, it provides access to a number of subcommands.\n\nh3. Syntax\n\nThe @arv@ command takes the following arguments:\n\n
\nArvados command line client\nUsage: arv [--flags] subcommand|resource [method] [--parameters]\n\nAvailable flags:\n  -n, --dry-run       Don't actually do anything\n  -v, --verbose       Print some things on stderr\n  -f, --format=    Set the output format. Must be one of json (default),\n                      yaml or uuid. (Default: json)\n  -s, --short         Return only UUIDs (equivalent to --format=uuid)\n\nUse 'arv subcommand|resource --help' to get more information about a particular\ncommand or resource.\n\nAvailable subcommands: copy, create, edit, keep, run, tag, ws\n\nAvailable resources: api_client_authorization, api_client,\nauthorized_key, collection, container, container_request,\nuser_agreement, group, keep_service, link, log, user, virtual_machine,\nworkflow\n\nAdditional options:\n  -e, --version       Print version and exit\n  -h, --help          Show this message\n
\n\nh4. Flags: @--format@\n\n- @--format=json@ := Output response as JSON. This is the default format.\n\n- @--format=yaml@ := Output response as YAML\n\n- @--format=uuid@ := Output only the UUIDs of object(s) in the API response, one per line.\n\n\n\nh3. Resources\n\nSee the \"arv reference\":{{site.baseurl}}/sdk/cli/reference.html page.\n\nh3. Subcommands\n\nSee the \"arv subcommands\":{{site.baseurl}}/sdk/cli/subcommands.html page.\n"
  },
  {
    "path": "doc/sdk/cli/install.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: sdk\nnavmenu: Command line tools (CLI SDK)\ntitle: \"Installation\"\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nArvados CLI tools are written in Ruby and Python.  To use the @arv@ command, you can either install the @arvados-cli@ gem via RubyGems or build and install the package from source.  The @arv@ command also relies on other Arvados tools.  To get those, install the @arvados-python-client@ and @arvados-cwl-runner@ packages, either from PyPI or source.\n\nh2. Prerequisites\n\n# \"Install Ruby\":../../install/ruby.html\n# \"Install the Python SDK\":../python/sdk-python.html\n\nThe SDK uses @curl@ which depends on the @libcurl@ C library.  To build the module you may have to install additional packages.  On supported versions of Debian and Ubuntu, run:\n\n\n
\n# apt install build-essential libcurl4-openssl-dev\n
\n\n\nh2. Install from RubyGems\n\n\n
\n# gem install arvados-cli\n
\n\n"
  },
  {
    "path": "doc/sdk/cli/reference.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: sdk\nnavmenu: Command line tools (CLI SDK)\ntitle: \"arv reference\"\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n_In order to use the @arv@ command, make sure that you have a \"working environment.\":{{site.baseurl}}/user/getting_started/check-environment.html_\n\nh3. Usage\n\nSee the \"CLI overview\":{{site.baseurl}}/sdk/cli/index.html page.\n\nh3. Resource types and methods\n\nGet list of resource types\n@arv --help@\n\nGet list of resource methods for the \"user\" resource type\n@arv user --help@\n\n\nh3. Basic examples\n\nGet record for current user\n@arv user current@\n\nGet entire record for some specific user\n@arv user get --uuid 6dnxa-tpzed-iimd25zhzh84gbk@\n\nUpdate user record\n@arv user update --uuid 6dnxa-tpzed-iimd25zhzh84gbk --user '{\"first_name\":\"Bob\"}'@\n\nGet list of groups\n@arv group list@\n\nDelete a group\n@arv group delete --uuid 6dnxa-j7d0g-iw7i6n43d37jtog@\n\nCreate an empty collection\n@arv collection create --collection '{\"name\": \"test collection\"}'@\n\nh3. Common commands\n\nMost @arv@ resources accept the following commands:\n\n* @get@\n* @list@\n* @create@\n* @update@\n* @delete@\n\n\nh4. @list@\n\nArguments accepted by the @list@ subcommand include:\n\n
\n  -l, --limit=        Maximum number of items to return. (Default: 100)\n  -o, --offset=       Number of items to skip before first returned record. (Default: 0)\n  -f, --filters=      Conditions for filtering items.\n  -r, --order=        Order in which to return matching items.\n  -s, --select=       Select which fields to return.\n  -d, --distinct         Return each distinct object.\n  -c, --count=        Type of count to return in items_available ('none' or 'exact'). (Default: exact)\n
\n\nThe @--filters@ option takes a string describing a JSON list of filters on which the returned resources should be returned. Each filter is a three-element list of _[field, operator, value]_, where the _operator_ may be one of @=@, @<@, @<=@, @>@, @>=@, @!=@, @like@, or @ilike@.\n\nExample:\n\n@arv collection list --filters '[[\"name\", \"=\", \"PGP VAR inputs\"], [\"created_at\", \">=\", \"2014-10-01\"]]'@\n\nwill return a list of all collections visible to the current user which are named \"PGP VAR inputs\" and were created on or after October 1, 2014. See the \"Common resource methods\":{{site.baseurl}}/api/methods.html#index page for more details on using @list@ and @--filters@.\n"
  },
  {
    "path": "doc/sdk/cli/subcommands.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: sdk\nnavmenu: Command line tools (CLI SDK)\ntitle: \"arv subcommands\"\n\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\n_In order to use the @arv@ command, make sure that you have a \"working environment.\":{{site.baseurl}}/user/getting_started/check-environment.html_\n\nh3(#arv-create). arv create\n\n@arv create@ can be used to create Arvados objects from the command line. Arv create opens up the editor of your choice (set the EDITOR environment variable) and allows you to type or paste a json or yaml description. When saved the object will be created on the API server, if it passes validation.\n\n\n
\n$ arv create --help\nOptions:\n  --project-uuid, -p <s>:   Project uuid in which to create the object\n              --help, -h:   Show this message\n
\n\n\nh3(#arv-get). arv get\n\n@arv get@ can be used to get a textual representation of Arvados objects from the command line. The output can be limited to a subset of the object's fields. This command can be used with only the knowledge of an object's UUID.\n\n\n
\n$ arv get --help\nUsage: arv [--format json|yaml] get [uuid] [fields...]\n\nFetch the specified Arvados object, select the specified fields,\nand print a text representation.\n
\n\n\nh3(#arv-edit). arv edit\n\n@arv edit@ can be used to edit Arvados objects from the command line. Arv edit opens up the editor of your choice (set the EDITOR environment variable) with the json or yaml description of the object. Saving the file will update the Arvados object on the API server, if it passes validation.\n\n\n
\n$ arv edit --help\nArvados command line client\nUsage: arv edit [uuid] [fields...]\n\nFetch the specified Arvados object, select the specified fields,\nopen an interactive text editor on a text representation (json or\nyaml, use --format) and then update the object.  Will use 'nano'\nby default, customize with the EDITOR or VISUAL environment variable.\n
\n\n\nh3(#arv-copy). arv copy\n\n@arv copy@ can be used to copy a pipeline instance, template or collection from one Arvados instance to another. It takes care of copying the object and all its dependencies.\n\n\n
\n$ arv copy --help\nusage: arv-copy [-h] [--version] [-v] [--progress] [--no-progress] [-f]\n                [--src SOURCE_ARVADOS] [--dst DESTINATION_ARVADOS]\n                [--recursive] [--no-recursive] [--project-uuid PROJECT_UUID]\n                [--replication N] [--storage-classes STORAGE_CLASSES]\n                [--varying-url-params VARYING_URL_PARAMS]\n                [--prefer-cached-downloads] [--retries RETRIES]\n                object_uuid\n\nCopy a workflow, collection or project from one Arvados instance to another.\nOn success, the uuid of the copied object is printed to stdout.\n\npositional arguments:\n  object_uuid           The UUID of the object to be copied.\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --version             Print version and exit.\n  -v, --verbose         Verbose output.\n  --progress            Report progress on copying collections. (default)\n  --no-progress         Do not report progress on copying collections.\n  -f, --force           Perform copy even if the object appears to exist at\n                        the remote destination.\n  --src SOURCE_ARVADOS  Client configuration location for the source Arvados\n                        cluster. May be either a configuration file path, or a\n                        plain identifier like `foo` to search for a\n                        configuration file `foo.conf` under a systemd or XDG\n                        configuration directory. If not provided, will search\n                        for a configuration file named after the cluster ID of\n                        the source object UUID.\n  --dst DESTINATION_ARVADOS\n                        Client configuration location for the destination\n                        Arvados cluster. May be either a configuration file\n                        path, or a plain identifier like `foo` to search for a\n                        configuration file `foo.conf` under a systemd or XDG\n                        configuration directory. If not provided, will use the\n                        default client configuration from the environment or\n                        `settings.conf`.\n  --recursive           Recursively copy any dependencies for this object, and\n                        subprojects. (default)\n  --no-recursive        Do not copy any dependencies or subprojects.\n  --project-uuid PROJECT_UUID\n                        The UUID of the project at the destination to which\n                        the collection or workflow should be copied.\n  --replication N\n                        Number of replicas per storage class for the copied\n                        collections at the destination. If not provided (or if\n                        provided with invalid value), use the destination's\n                        default replication-level setting (if found), or the\n                        fallback value 2.\n  --storage-classes STORAGE_CLASSES\n                        Comma separated list of storage classes to be used\n                        when saving data to the destinaton Arvados instance.\n  --varying-url-params VARYING_URL_PARAMS\n                        A comma separated list of URL query parameters that\n                        should be ignored when storing HTTP URLs in Keep.\n  --prefer-cached-downloads\n                        If a HTTP URL is found in Keep, skip upstream URL\n                        freshness check (will not notice if the upstream has\n                        changed, but also not error if upstream is\n                        unavailable).\n  --retries RETRIES     Maximum number of times to retry server requests that\n                        encounter temporary failures (e.g., server down).\n                        Default 10.\n
\n\n\nh3(#arv-tag). arv tag\n\n@arv tag@ is used to tag Arvados objects.\n\n\n
\n$ arv tag --help\n\nUsage:\narv tag add tag1 [tag2 ...] --object object_uuid1 [object_uuid2...]\narv tag remove tag1 [tag2 ...] --object object_uuid1 [object_uuid2...]\narv tag remove --all\n\n  --dry-run, -n:   Don't actually do anything\n  --verbose, -v:   Print some things on stderr\n     --uuid, -u:   Return the UUIDs of the objects in the response, one per\n                   line (default)\n     --json, -j:   Return the entire response received from the API server, as\n                   a JSON object\n    --human, -h:   Return the response received from the API server, as a JSON\n                   object with whitespace added for human consumption\n   --pretty, -p:   Synonym of --human\n     --yaml, -y:   Return the response received from the API server, in YAML\n                   format\n     --help, -e:   Show this message\n
\n\n\n\nh3(#arv-ws). arv ws\n\nThis is a frontend to @arv-ws@.\n\n@arv ws@ provides access to the websockets event stream.\n\n\n
\n$ arv ws --help\nusage: arv-ws [-h] [-u UUID] [-f FILTERS]\n              [--poll-interval POLL_INTERVAL | --no-poll]\n              [-p PIPELINE | -j JOB]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -u UUID, --uuid UUID  Filter events on object_uuid\n  -f FILTERS, --filters FILTERS\n                        Arvados query filter to apply to log events (JSON\n                        encoded)\n  --poll-interval POLL_INTERVAL\n                        If websockets is not available, specify the polling\n                        interval, default is every 15 seconds\n  --no-poll             Do not poll if websockets are not available, just fail\n  -p PIPELINE, --pipeline PIPELINE\n                        Supply pipeline uuid, print log output from pipeline\n                        and its jobs\n  -j JOB, --job JOB     Supply job uuid, print log output from jobs\n
\n\n\nh3(#arv-keep). arv keep\n\n@arv keep@ commands for accessing the Keep storage service.\n\n\n
\n$ arv keep --help\nUsage: arv keep [method] [--parameters]\nUse 'arv keep [method] --help' to get more information about specific methods.\n\nAvailable methods: ls, get, put, docker\n
\n\n\nh3(#arv-keep-ls). arv keep ls\n\nThis is a frontend to @arv-ls@.\n\n\n
\n$ arv keep ls --help\nusage: arv-ls [-h] [--retries RETRIES] [-s] locator\n\nList contents of a manifest\n\npositional arguments:\n  locator            Collection UUID or locator\n\noptional arguments:\n  -h, --help         show this help message and exit\n  --retries RETRIES  Maximum number of times to retry server requests that\n                     encounter temporary failures (e.g., server down). Default\n                     3.\n  -s                 List file sizes, in KiB.\n
\n\n\nh3(#arv-keep-get). arv keep get\n\nThis is a frontend to @arv-get@.\n\n\n
\n$ arv keep get --help\nusage: arv-get [-h] [--retries RETRIES] [--version]\n               [--progress | --no-progress | --batch-progress]\n               [--hash HASH | --md5sum] [-n] [-r]\n               [-f | -v | --skip-existing | --strip-manifest] [--threads N]\n               locator [destination]\n\nCopy data from Keep to a local file or pipe.\n\npositional arguments:\n  locator            Collection locator, optionally with a file path or\n                     prefix.\n  destination        Local file or directory where the data is to be written.\n                     Default: stdout.\n\noptional arguments:\n  -h, --help         show this help message and exit\n  --retries RETRIES  Maximum number of times to retry server requests that\n                     encounter temporary failures (e.g., server down).\n                     Default 3.\n  --version          Print version and exit.\n  --progress         Display human-readable progress on stderr (bytes and, if\n                     possible, percentage of total data size). This is the\n                     default behavior when it is not expected to interfere\n                     with the output: specifically, stderr is a tty _and_\n                     either stdout is not a tty, or output is being written\n                     to named files rather than stdout.\n  --no-progress      Do not display human-readable progress on stderr.\n  --batch-progress   Display machine-readable progress on stderr (bytes and,\n                     if known, total data size).\n  --hash HASH        Display the hash of each file as it is read from Keep,\n                     using the given hash algorithm. Supported algorithms\n                     include md5, sha1, sha224, sha256, sha384, and sha512.\n  --md5sum           Display the MD5 hash of each file as it is read from\n                     Keep.\n  -n                 Do not write any data -- just read from Keep, and report\n                     md5sums if requested.\n  -r                 Retrieve all files in the specified collection/prefix.\n                     This is the default behavior if the \"locator\" argument\n                     ends with a forward slash.\n  -f                 Overwrite existing files while writing. The default\n                     behavior is to refuse to write *anything* if any of the\n                     output files already exist. As a special case, -f is not\n                     needed to write to stdout.\n  -v                 Once for verbose mode, twice for debug mode.\n  --skip-existing    Skip files that already exist. The default behavior is\n                     to refuse to write *anything* if any files exist that\n                     would have to be overwritten. This option causes even\n                     devices, sockets, and fifos to be skipped.\n  --strip-manifest   When getting a collection manifest, strip its access\n                     tokens before writing it.\n  --threads N        Set the number of download threads to be used. Take into\n                     account that using lots of threads will increase the RAM\n                     requirements. Default is to use 4 threads. On high\n                     latency installations, using a greater number will\n                     improve overall throughput.\n
\n\n\nh3(#arv-keep-put). arv keep put\n\nThis is a frontend to @arv-put@.\n\n\n
\n$ arv keep put --help\nusage: arv-put [-h] [--max-manifest-depth N | --normalize]\n               [--as-stream | --stream | --as-manifest | --in-manifest | --manifest | --as-raw | --raw]\n               [--use-filename FILENAME] [--filename FILENAME]\n               [--portable-data-hash] [--replication N]\n               [--project-uuid UUID] [--name NAME]\n               [--progress | --no-progress | --batch-progress]\n               [--resume | --no-resume] [--retries RETRIES]\n               [path [path ...]]\n\nCopy data from the local filesystem to Keep.\n\npositional arguments:\n  path                  Local file or directory. Default: read from standard\n                        input.\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --max-manifest-depth N\n                        Maximum depth of directory tree to represent in the\n                        manifest structure. A directory structure deeper than\n                        this will be represented as a single stream in the\n                        manifest. If N=0, the manifest will contain a single\n                        stream. Default: -1 (unlimited), i.e., exactly one\n                        manifest stream per filesystem directory that contains\n                        files.\n  --normalize           Normalize the manifest by re-ordering files and\n                        streams after writing data.\n  --as-stream           Synonym for --stream.\n  --stream              Store the file content and display the resulting\n                        manifest on stdout. Do not write the manifest to Keep\n                        or save a Collection object in Arvados.\n  --as-manifest         Synonym for --manifest.\n  --in-manifest         Synonym for --manifest.\n  --manifest            Store the file data and resulting manifest in Keep,\n                        save a Collection object in Arvados, and display the\n                        manifest locator (Collection uuid) on stdout. This is\n                        the default behavior.\n  --as-raw              Synonym for --raw.\n  --raw                 Store the file content and display the data block\n                        locators on stdout, separated by commas, with a\n                        trailing newline. Do not store a manifest.\n  --use-filename FILENAME\n                        Synonym for --filename.\n  --filename FILENAME   Use the given filename in the manifest, instead of the\n                        name of the local file. This is useful when \"-\" or\n                        \"/dev/stdin\" is given as an input file. It can be used\n                        only if there is exactly one path given and it is not\n                        a directory. Implies --manifest.\n  --portable-data-hash  Print the portable data hash instead of the Arvados\n                        UUID for the collection created by the upload.\n  --replication N       Set the replication level for the new collection: how\n                        many different physical storage devices (e.g., disks)\n                        should have a copy of each data block. Default is to\n                        use the server-provided default (if any) or 2.\n  --project-uuid UUID   Store the collection in the specified project, instead\n                        of your Home project.\n  --name NAME           Save the collection with the specified name.\n  --progress            Display human-readable progress on stderr (bytes and,\n                        if possible, percentage of total data size). This is\n                        the default behavior when stderr is a tty.\n  --no-progress         Do not display human-readable progress on stderr, even\n                        if stderr is a tty.\n  --batch-progress      Display machine-readable progress on stderr (bytes\n                        and, if known, total data size).\n  --resume              Continue interrupted uploads from cached state\n                        (default).\n  --no-resume           Do not continue interrupted uploads from cached state.\n  --retries RETRIES     Maximum number of times to retry server requests that\n                        encounter temporary failures (e.g., server down).\n                        Default 3.\n
\n\n"
  },
  {
    "path": "doc/sdk/fuse/install.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: sdk\nnavmenu: FUSE Driver\ntitle: Installing the FUSE Driver\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThe Arvados FUSE driver is a Python utility that allows you to browse Arvados projects and collections in a filesystem, so you can access that data using existing Unix tools.\n\nh2. Installation\n\nIf you are logged in to a managed Arvados VM, the @arv-mount@ utility should already be installed.\n\nTo use the FUSE driver elsewhere, you can install from a distribution package or pip.\n\nh2. Option 1: Install from distribution packages\n\nFirst, \"add the appropriate package repository for your distribution\":{{ site.baseurl }}/install/packages.html.\n\n{% assign arvados_component = 'python3-arvados-fuse' %}\n\n{% include 'install_packages' %}\n\nh2. Option 2: Install with pip\n\nRun @pip install arvados_fuse@ in an appropriate installation environment, such as a virtualenv.\n\nNote: The FUSE driver depends on the @libcurl@ and @libfuse@ C libraries.  To install the module you may need to install development headers from your distribution.  On Debian-based distributions you can install them by running:\n\n\n
# apt install build-essential python3-dev libcurl4-openssl-dev libfuse-dev libssl-dev\n
\n\n\nh2. Usage\n\nFor an introduction of how to mount and navigate data, refer to the \"Access Keep as a GNU/Linux filesystem\":{{site.baseurl}}/user/tutorials/tutorial-keep-mount-gnu-linux.html tutorial.\n"
  },
  {
    "path": "doc/sdk/fuse/options.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: sdk\nnavmenu: FUSE Driver\ntitle: arv-mount options\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThis page documents all available @arv-mount@ options with some usage examples.\n\n# \"Mount contents\":#contents\n# \"Mount custom layout and filtering\":#layout\n## \"@--filters@ usage and limitations\":#filters\n# \"Mount access and permissions\":#access\n# \"Mount lifecycle management\":#lifecycle\n# \"Mount logging and statistics\":#logging\n# \"Mount local cache setup\":#cache\n# \"Mount interactions with Arvados and Linux\":#plumbing\n# \"Examples\":#examples\n## \"Using @--exec@\":#exec\n## \"Running arv-mount as a systemd service\":#systemd\n\nh2(#contents). Mount contents\n\ntable(table table-bordered table-condensed).\n|_. Option(s)|_. Description|\n|@--all@|Mount a subdirectory for each mode: @home@, @shared@, @by_id@, and @by_tag@ (default if no @--mount-*@ options are given)|\n|@--custom@|Mount a subdirectory for each mode specified by a @--mount-*@ option (default if any @--mount-*@ options are given; see \"Mount custom layout and filtering\":#layout section)|\n|@--collection UUID_OR_PDH@|Mount the specified collection|\n|@--home@|Mount your home project|\n|@--project UUID@|Mount the specified project|\n|@--shared@|Mount a subdirectory for each project shared with you|\n|@--by-id@|Mount a magic directory where collections and projects are accessible through subdirectories named after their UUID or portable data hash|\n|@--by-pdh@|Mount a magic directory where collections are accessible through subdirectories named after their portable data hash|\n|@--by-tag@|Mount a subdirectory for each tag attached to a collection or project|\n\nh2(#layout). Mount custom layout and filtering\n\ntable(table table-bordered table-condensed).\n|_. Option(s)|_. Description|\n|@--filters FILTERS@|Filters to apply to all project, shared, and tag directory contents. Pass filters as either a JSON string or a path to a JSON file. The JSON object should be a list of filters in \"Arvados API list filter syntax\":{{ site.baseurl }}/api/methods.html#filters. See the \"example filters\":#filters.|\n|@--mount-home PATH@|Make your home project available under the mount at @PATH@|\n|@--mount-shared PATH@|Make projects shared with you available under the mount at @PATH@|\n|@--mount-tmp PATH@|Make a new temporary writable collection available under the mount at @PATH@. This collection is deleted when the mount is unmounted.|\n|@--mount-by-id PATH@|Make a magic directory available under the mount at @PATH@ where collections and projects are accessible through subdirectories named after their UUID or portable data hash|\n|@--mount-by-pdh PATH@|Make a magic directory available under the mount at @PATH@ where collections are accessible through subdirectories named after portable data hash|\n|@--mount-by-tag PATH@|Make a subdirectory for each tag attached to a collection or project available under the mount at @PATH@|\n\nh3(#filters). @--filters@ usage and limitations\n\nYour argument to @--filters@ should be a JSON list of filters in \"Arvados API list filter syntax\":{{ site.baseurl }}/api/methods.html#filters. If your filter checks any field besides @uuid@, you should prefix it with the @.@ Taken together, here's an example that mounts your home directory excluding filter groups, workflow intermediate output collections, and workflow log collections:\n\n\n
$ arv-mount --home --filters '[[\"groups.group_class\", \"!=\", \"filter\"], [\"collections.properties.type\", \"not in\", [\"intermediate\", \"log\"]]]' ...\n
\n\n\nBecause filters can be awkward to write on the command line, you can also write them in a file, and pass that file path to the @--filters@ option. This example does the same filtering:\n\n\n
$ cat >~/arv-mount-filters.json <<EOF\n[\n  [\n    \"groups.group_class\",\n    \"!=\",\n    \"filter\"\n  ],\n  [\n    \"collections.properties.type\",\n    \"not in\",\n    [\n      \"intermediate\",\n      \"log\"\n    ]\n  ]\n]\nEOF\n$ arv-mount --home --filters ~/arv-mount-filters.json ...\n
\n\n\nThe current implementation of @--filters@ has a few limitations. These may be lifted in a future release:\n\n* You can always access any project or collection by UUID or portable data hash under a magic directory. If you access a project this way, your filters _will_ apply to the project contents.\n* Tag directory listings are generated by querying tags alone. Only filters that apply to @links@ will affect these listings.\n\nh2(#access). Mount access and permissions\n\ntable(table table-bordered table-condensed).\n|_. Option(s)|_. Description|\n|@--allow-other@|Let other users on this system read mounted data (default false)|\n|@--read-only@|Mounted data cannot be modified from the mount (default)|\n|@--read-write@|Mounted data can be modified from the mount|\n\nh2(#lifecycle). Mount lifecycle management\n\ntable(table table-bordered table-condensed).\n|_. Option(s)|_. Description|\n|@--exec ...@|Mount data, run the specified command, then unmount and exit. @--exec@ reads all remaining options as the command to run, so it must be the last option you specify. Either end your command arguments (and other options) with a @--@ argument, or specify @--exec@ after your mount point.|\n|@--foreground@|Run mount process in the foreground instead of daemonizing (default false)|\n|@--subtype SUBTYPE@|Set mounted filesystem type to @fuse.SUBTYPE@ (default is just @fuse@)|\n|@--replace@|If a FUSE mount is already mounted at the given directory, unmount it before mounting the requested data. If @--subtype@ is specified, unmount only if the mount has that subtype. WARNING: This command can affect any kind of FUSE mount, not just arv-mount.|\n|@--unmount@|If a FUSE mount is already mounted at the given directory, unmount it and exit. If @--subtype@ is specified, unmount only if the mount has that subtype. WARNING: This command can affect any kind of FUSE mount, not just arv-mount.|\n|@--unmount-all@|Unmount all FUSE mounts at or below the given directory, then exit. If @--subtype@ is specified, unmount only if the mount has that subtype. WARNING: This command can affect any kind of FUSE mount, not just arv-mount.|\n|@--unmount-timeout SECONDS@|The number of seconds to wait for a clean unmount after an @--exec@ command has exited (default 2.0). After this time, the mount will be forcefully unmounted.|\n\nh2(#logging). Mount logging and statistics\n\ntable(table table-bordered table-condensed).\n|_. Option(s)|_. Description|\n|@--crunchstat-interval SECONDS@|Write stats to stderr every N seconds (default disabled)|\n|@--debug@|Log debug information|\n|@--logfile LOGFILE@|Write debug logs and errors to the specified file (default stderr)|\n\nh2(#cache). Mount local cache setup\n\ntable(table table-bordered table-condensed).\n|_. Option(s)|_. Description|\n|@--disk-cache@|Cache data on the local filesystem (default)|\n|@--ram-cache@|Cache data in memory|\n|@--disk-cache-dir DIRECTORY@|Set custom filesystem cache location|\n|@--directory-cache BYTES@|Size of directory data cache in bytes (default 128 MiB)|\n|@--file-cache BYTES@|Size of file data cache in bytes (default 8 GiB for filesystem cache, 256 MiB for memory cache)|\n\nh2(#plumbing). Mount interactions with Arvados and Linux\n\ntable(table table-bordered table-condensed).\n|_. Option(s)|_. Description|\n|@--disable-event-listening@|Don't subscribe to events on the API server to update mount contents|\n|@--encoding ENCODING@|Filesystem character encoding (default 'utf-8'; specify a name from the \"Python codec registry\":https://docs.python.org/3/library/codecs.html#standard-encodings)|\n|@--retries RETRIES@|Maximum number of times to retry server requests that encounter temporary failures (e.g., server down). Default 10.|\n|@--storage-classes CLASSES@|Comma-separated list of storage classes to request for new collections|\n\nh2(#examples). Examples\n\nh3(#exec). Using @--exec@\n\nThere are a couple of details that are important to understand when you use @--exec@:\n\n* @--exec@ reads all remaining options as the command to run, so it must be the last option you specify. Either end your command arguments (and other options) with a @--@ argument, or specify @--exec@ after your mount point.\n* The command you specify runs from the same directory that you started @arv-mount@ from. To access data inside the mount, you will generally need to pass the path to the mount as an argument.\n\nFor example, this generates a recursive listing of all the projects and collections under your home project:\n\n\n
$ arv-mount --home --exec find -type d ArvadosHome -- ArvadosHome\n
\n\n\nThe first @ArvadosHome@ is a path argument to @find@. The second is the mount point argument to @arv-mount@.\n\nh3(#systemd). Running arv-mount as a systemd service\n\nIf you want to run @arv-mount@ as a long-running service, it's easy to write a systemd service definition for it. We do not publish one because the entire definition tends to be site-specific, but you can start from this template. You must change the @ExecStart@ path. Comments detail other changes you might want to make.\n\n\n
[Unit]\nDescription=Arvados FUSE mount\nDocumentation={{ site.baseurl }}/sdk/fuse/options.html\n\n[Service]\nType=simple\n\n# arv-mount will cache data under a `keep` subdirectory of CacheDirectory.\n# If this is a system service installed under /etc/systemd/system,\n# the cache will be at /var/cache/arvados/keep.\n# The default value of `arvados` lets arv-mount share the cache with other\n# tools.\nCacheDirectory=arvados\n\n# arv-mount will get Arvados API credentials from the `settings.conf` file\n# under ConfigurationDirectory.\n# If this is a system service installed under /etc/systemd/system,\n# the configuration will be read from /etc/arvados/settings.conf.\n# The default value of `arvados` lets arv-mount read configuration from the\n# same location as other tools.\nConfigurationDirectory=arvados\n\n# This unit makes the mount available as `Arvados` under the runtime directory root.\n# If this is a system service installed under /etc/systemd/system,\n# the mount will be at /run/Arvados.\n# If this is a user service installed under ~/.config/systemd/user,\n# the mount will be at $XDG_RUNTIME_DIR/Arvados.\n# If you want to mount at another location on the filesystem, remove RuntimeDirectory\n# and replace both instances of %t/Arvados with your desired path.\nRuntimeDirectory=Arvados\n\n# The arv-mount path must be the absolute path where you installed the command.\n# If you installed from a distribution package, make this /usr/bin/arv-mount.\n# If you installed from pip, replace ... with the path to your virtualenv.\n# You can add options to select what gets mounted, access permissions,\n# cache size, log level, etc.\nExecStart=.../bin/arv-mount --foreground %t/Arvados\nExecStop=/usr/bin/fusermount -u %t/Arvados\n
\n\n"
  },
  {
    "path": "doc/sdk/go/example.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: sdk\nnavmenu: Go\ntitle: Examples\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nSee \"Arvados GoDoc\":https://godoc.org/git.arvados.org/arvados.git/sdk/go for detailed documentation.\n\nIn these examples, the site prefix is @aaaaa@.\n\nh2.  Initialize SDK\n\n{% codeblock as go %}\nimport (\n  \"git.arvados.org/arvados.git/sdk/go/arvados\"\n  \"git.arvados.org/arvados.git/sdk/go/arvadosclient\"\n}\n\nfunc main() {\n  arv, err := arvadosclient.MakeArvadosClient()\n  if err != nil {\n    log.Fatalf(\"Error setting up arvados client %s\", err.Error())\n  }\n}\n{% endcodeblock %}\n\nh2. create\n\n{% codeblock as go %}\n  var collection arvados.Collection\n  err := api.Create(\"collections\", Dict{\"collection\": Dict{\"name\": \"create example\"}}, &collection)\n{% endcodeblock %}\n\nh2. delete\n\n{% codeblock as go %}\n  var collection arvados.Collection\n  err := api.Delete(\"collections\", \"aaaaa-4zz18-ccccccccccccccc\", Dict{}, &collection)\n{% endcodeblock %}\n\nh2. get\n\n{% codeblock as go %}\n  var collection arvados.Collection\n  err := api.Get(\"collections\", \"aaaaa-4zz18-ccccccccccccccc\", Dict{}, &collection)\n{% endcodeblock %}\n\nh2. list\n\n{% codeblock as go %}\n  var collection arvados.Collection\n  err := api.List(\"collections\", Dict{}, &collection)\n{% endcodeblock %}\n\nh2. update\n\n{% codeblock as go %}\n  var collection arvados.Collection\n  err := api.Update(\"collections\", \"aaaaa-4zz18-ccccccccccccccc\", Dict{\"collection\": Dict{\"name\": \"update example\"}}, &collection)\n{% endcodeblock %}\n\nh2. Get current user\n\n{% codeblock as go %}\n  var user arvados.User\n  err := api.Get(\"users\", \"current\", Dict{}, &user)\n{% endcodeblock %}\n\nh2. Example program\n\nYou can save this source as a .go file and run it:\n\n{% code example_sdk_go as go %}\n\nA few more usage examples can be found in the \"services/keepproxy\":https://github.com/arvados/arvados/tree/main/services/keepproxy and \"sdk/go/keepclient\":https://github.com/arvados/arvados/tree/main/sdk/go/keepclient directories in the arvados source tree.\n"
  },
  {
    "path": "doc/sdk/go/index.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: sdk\nnavmenu: Go\ntitle: \"Installation\"\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThe Go (\"Golang\":http://golang.org) SDK provides a generic set of wrappers so you can make API calls easily.\n\nSee \"Arvados GoDoc\":https://godoc.org/git.arvados.org/arvados.git/sdk/go for detailed documentation.\n\nh3. Installation\n\nUse @go get git.arvados.org/arvados.git/sdk/go/arvadosclient@.  The go tools will fetch the relevant code and dependencies for you.\n\n{% codeblock as go %}\nimport (\n\t\"git.arvados.org/arvados.git/sdk/go/arvadosclient\"\n\t\"git.arvados.org/arvados.git/sdk/go/keepclient\"\n)\n{% endcodeblock %}\n\nIf you need pre-release client code, you can use the latest version from the repo by following \"these instructions.\":https://dev.arvados.org/projects/arvados/wiki/Go#Using-Go-with-Arvados\n"
  },
  {
    "path": "doc/sdk/index.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: sdk\ntitle: \"SDK Reference\"\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThis section documents client tools and language bindings for the \"Arvados API\":{{site.baseurl}}/api/index.html and Keep that are available for various programming languages. The most mature, popular packages are:\n\n* \"Python SDK\":{{site.baseurl}}/sdk/python/sdk-python.html (also includes essential command line tools such as @arv-put@ and @arv-get@)\n* \"Command line SDK\":{{site.baseurl}}/sdk/cli/install.html (includes the @arv@ tool)\n\nMany Arvados Workbench pages provide examples of using the Python SDK and command line tools to access a given resource. Open \"API details\" from the action menu and open the tab with the example you're interested in.\n\nWe provide API bindings for several other languages, but these SDKs may be missing some features or documentation:\n\n* \"Go SDK\":{{site.baseurl}}/sdk/go/index.html\n* \"Java SDK\":{{site.baseurl}}/sdk/java-v2/index.html\n* \"R SDK\":{{site.baseurl}}/sdk/R/index.html\n* \"Ruby SDK\":{{site.baseurl}}/sdk/ruby/index.html\n\nConsult the \"Arvados API\":{{site.baseurl}}/api/index.html section for detailed documentation about Arvados API calls available on each resource.\n"
  },
  {
    "path": "doc/sdk/java-v2/example.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: sdk\nnavmenu: Java\ntitle: Examples\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nIn these examples, the site prefix is @aaaaa@.\n\nh2.  Initialize SDK\n\n{% codeblock as java %}\npackage org.arvados.example;\n\nimport java.util.List;\nimport org.arvados.client.config.ConfigProvider;\nimport org.arvados.client.config.ExternalConfigProvider;\nimport org.arvados.client.api.model.CollectionList;\nimport org.arvados.client.api.model.Collection;\nimport org.arvados.client.api.client.CollectionsApiClient;\n\npublic class CollectionExample {\n    public static void main(String[] argv) {\n\tConfigProvider conf = ExternalConfigProvider.builder().\n\t    apiProtocol(\"https\").\n\t    apiHost(\"zzzzz.arvadosapi.com\").\n\t    apiPort(443).\n\t    apiToken(\"...\").\n\t    build();\n\tCollectionsApiClient collectionsApi = new CollectionsApiClient(conf);\n\t/* ... */\n    }\n}\n{% endcodeblock %}\n\nh2. list\n\n{% codeblock as java %}\n\tCollectionList cl = collectionsApi.list();\n\tList items = cl.getItems();\n\tfor (int i = 0; i < items.size(); i++) {\n\t    System.out.println(items.get(i));\n\t}\n{% endcodeblock %}\n"
  },
  {
    "path": "doc/sdk/java-v2/index.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: sdk\nnavmenu: Java\ntitle: \"Installation\"\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nThe Arvados Java SDK v2 provides a high level API for working with Arvados resources.\n\n{% include 'contrib_component' component_name: \"The Java SDK v2\" %}\n\nh2. Using the SDK\n\nThe SDK is packaged as a JAR named @arvados-java-.jar@, which is published to Maven Central and can be included using Maven, Gradle, or by hand.\n\nHere is an example @build.gradle@ file that uses the Arvados java sdk:\n\n
\napply plugin: 'application'\napply plugin: 'java-library'\napply plugin: 'maven'\n\nrepositories {\n    mavenCentral()\n}\n\napplication {\n    mainClassName = \"org.arvados.example.CollectionExample\"\n}\n\ndependencies {\n    api 'org.arvados:arvados-java-sdk:0.1.1'\n}\n
\n\nSee \"Java SDK Examples\":example.html to get started using the SDK.\n\nh3. Logging\n\nThe SDK uses the SLF4J facade library for logging. A concrete logging \"binding\":https://www.slf4j.org/manual.html#swapping (and configuration, if required) must be provided by a client. For small applications, you can use the Simple implementation by adding slf4j-simple-1.8.0-beta4.jar to your classpath.\n\nh3. Configuration\n\n\"TypeSafe Configuration\":https://github.com/lightbend/config is used for configuring this library.\n\nPlease review src/main/resources/reference.conf for default values provided with this library.\n\n* **keepweb-host** - host of your Keep-Web server (default: localhost)\n* **keepweb-port** - port of your Keep-Web server (default: 8000)\n* **host** - host of your Arvados API server\n* **port** - port of your Arvados API server\n* **token** - Arvados token to authenticate registered user, one must provide \"token obtained from Arvados Workbench\":https://doc.arvados.org/user/reference/api-tokens.html\n* **protocol** - don't change to unless really needed (default: https)\n* **host-insecure** - ignores SSL certificate verification if true (default: false Don't change to *true* unless really needed)\n* **split-size** - size of chunk files in megabytes (default: 64)\n* **temp-dir** - temporary chunk files storage\n* **copies** - amount of chunk files duplicates per Keep server\n* **retries** - UNIMPLEMENTED\n\nIn order to override default settings one can create an application.conf file in an application.  Example: src/test/resources/application.conf.\n\nAlternatively @ExternalConfigProvider@ class can be used to pass configuration via code.  @ExternalConfigProvider@ comes with a builder and all of the above values must be provided in order for it to work properly.\n\n@ArvadosFacade@ has two constructors, one without arguments that uses values from application.conf and second one taking @ExternalConfigProvider@ as an argument.\n\nh3. API clients\n\nAll API clients inherit from @BaseStandardApiClient@. This class contains implementation of all common methods as described in \"Arvados Common Resource Methods\":http://doc.arvados.org/api/methods.html.\n\nParameters provided to common or specific methods are String UUID or fields wrapped in Java objects. For example:\n\n{% codeblock as java %}\nString uuid = \"ardev-4zz18-rxcql7qwyakg1r1\";\n\nCollection actual = client.get(uuid);\n{% endcodeblock %}\n\n{% codeblock as java %}\nListArgument listArgument = ListArgument.builder()\n        .filters(Arrays.asList(\n                Filter.of(\"owner_uuid\", Operator.LIKE, \"ardev%\"),\n                Filter.of(\"name\", Operator.LIKE, \"Super%\"),\n                Filter.of(\"portable_data_hash\", Operator.IN, Lists.newArrayList(\"54f6d9f59065d3c009d4306660989379+65\")\n            )))\n        .build();\n\nCollectionList actual = client.list(listArgument);\n{% endcodeblock %}\n\nNon-standard API clients must inherit from BaseApiClient. For example: KeepServerApiClient communicates directly with Keep servers using exclusively non-common methods.\n\nh3. Business logic\n\nMore advanced API data handling could be implemented as *Facade* classes. In current version functionalities provided by SDK are handled by @ArvadosFacade@. They include:\n\n* **downloading single file from collection** - using Keep-Web\n* **downloading whole collection** - using Keep-Web or Keep Server API\n* **listing file info from certain collection** - information is returned as list of *FileTokens* providing file details\n* **uploading single file** - to either new or existing collection\n* **uploading list of files** - to either new or existing collection\n* **creating an empty collection**\n* **getting current user info**\n* **listing current user's collections**\n* **creating new project**\n* **deleting certain collection**\n\nh3. Note regarding Keep-Web\n\nThe Java SDK requires Keep Web (which is part of the standard configuration) as well as the API server and Keep server(s).\n\nh3. Integration tests\n\nIn order to run the integration tests, all fields within following configuration file must be provided: @src/test/resources/integration-test-appliation.conf@\n\n\nThe parameter @integration-tests.project-uuid@ should contain UUID of one project available to user who's token was provided within configuration file.\n\nIntegration tests require connection to a real Arvados server.\n\nh3. Note regarding file naming\n\nWhen uploading via the current implementation of the Java SDK all uploaded files within single collection must have different names. This applies also to uploading files to already existing collection. Renaming files with duplicate names is not currently implemented.\n\nh3. Javadoc\n\nSee \"Javadoc\":javadoc.html\n\nh2. Building the Arvados SDK\n\nDependencies:\n* JDK for Java 8 or later \"https://www.oracle.com/technetwork/java/javase/downloads/index.html\":https://www.oracle.com/technetwork/java/javase/downloads/index.html\n* Gradle \"https://gradle.org/install/\":https://gradle.org/install/\n\n\n\n
\n$ git clone https://github.com/arvados/arvados.git\n$ cd arvados/contrib/java-sdk-v2\n$ gradle test\n$ gradle jar -Pversion=0.1.1\n
\nThis will build the SDK and run all unit tests, then generate an Arvados Java sdk jar file in build/libs/arvados-java-0.1.1.jar\n\n"
  },
  {
    "path": "doc/sdk/java-v2/javadoc.html.textile.liquid",
    "content": "---\nlayout: default\nnavsection: sdk\nnavmenu: Java\ntitle: \"Javadoc Reference\"\n\nno_nav_left: true\n...\n{% comment %}\nCopyright (C) The Arvados Authors. All rights reserved.\n\nSPDX-License-Identifier: CC-BY-SA-3.0\n{% endcomment %}\n\nnotextile.