[ { "path": ".github/workflows/publish-pypi.yml", "content": "name: Publish to PyPI\n\n# Publish to PyPI when a tag is pushed\non:\n push:\n tags:\n - 'ckanapi-**'\n\njobs:\n build:\n if: github.repository == 'ckan/ckanapi'\n name: Build distribution\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v4\n - name: Set up Python\n uses: actions/setup-python@v5\n with:\n python-version: \"3.11\"\n - name: Install pypa/build\n run: python3 -m pip install build --user\n - name: Build a binary wheel and a source tarball\n run: python3 -m build\n - name: Store the distribution packages\n uses: actions/upload-artifact@v4\n with:\n name: python-package-distributions\n path: dist/\n\n publish-to-pypi:\n name: Publish Python distribution on PyPI\n needs:\n - build\n runs-on: ubuntu-latest\n environment:\n name: pypi\n url: https://pypi.org/p/ckanapi\n permissions:\n id-token: write\n steps:\n - name: Download all the dists\n uses: actions/download-artifact@v4\n with:\n name: python-package-distributions\n path: dist/\n - name: Publish distribution to PyPI\n uses: pypa/gh-action-pypi-publish@release/v1\n\n publishSkipped:\n if: github.repository != 'ckan/ckanapi'\n runs-on: ubuntu-latest\n steps:\n - run: |\n echo \"## Skipping PyPI publish on downstream repository\" >> $GITHUB_STEP_SUMMARY\n" }, { "path": ".github/workflows/test.yml", "content": "name: Tests\non: [push, pull_request]\njobs:\n test:\n strategy:\n matrix:\n python-version: [\"3.9\", \"3.10\", \"3.11\", \"3.12\", \"3.13\", \"3.14\"]\n runs-on: ubuntu-latest\n container:\n # INFO: python 2 is no longer supported in\n # actions/setup-python, use python docker image instead\n image: python:${{ matrix.python-version }}\n\n steps:\n - uses: actions/checkout@v3\n - name: Install requirements (py ${{ matrix.python-version }})\n run: |\n pip install -e \".[testing]\"\n - name: Run all tests (py ${{ matrix.python-version }})\n run: python -m unittest discover\n" }, { "path": ".gitignore", "content": "*.pyc\nMANIFEST\nbuild/\ndist/\nckanapi.egg-info/\n" }, { "path": "CHANGELOG.md", "content": "# ckanapi Changelog\n\n\n## v4.11 - 2026-03-20\n\n* Fix Reference Assignment in Dump Things [#227](https://github.com/ckan/ckanapi/pull/227)\n\n## v4.10 - 2026-03-13\n\n* Fix Log File Not Working (Requires Bytes Mode) [#224](https://github.com/ckan/ckanapi/pull/224)\n* Python 3.14 support, Drop Python 2 support, cleanup [#225](https://github.com/ckan/ckanapi/pull/225)\n* Arguments for Dataset Dump Include Private and States [#223](https://github.com/ckan/ckanapi/pull/223)\n* Allow to define a timeout to all requests [#226](https://github.com/ckan/ckanapi/pull/226)\n" }, { "path": "COPYING", "content": "ckanapi - Terms and Conditions of Use\n\nUnless otherwise noted, computer program source code of ckanapi is\ncovered under Crown Copyright, Government of Canada, and is distributed under the MIT License.\n\n\nMIT License\n\nCopyright (c) Her Majesty the Queen in Right of Canada, represented by the President of the Treasury\nBoard, 2013-2018\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this software and\nassociated documentation files (the \"Software\"), to deal in the Software without restriction,\nincluding without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,\nand/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so,\nsubject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all copies or substantial\nportions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT\nNOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.\nIN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,\nWHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE\nSOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n" }, { "path": "COPYING.fr", "content": "ckanapi - Conditions régissant l'utilisation\n\nSauf indication contraire, le code source de la ckanapi\nest protégé par le droit d'auteur de la Couronne du gouvernement du Canada et distribué\nsous la licence MIT.\n\n\nLicence MIT\n\n(c) Droit d'auteur – Sa Majesté la Reine du chef du Canada, représentée par le président du Conseil\ndu Trésor, 2013-2018\n\nLa présente autorise toute personne d'obtenir gratuitement une copie du présent logiciel et des\ndocuments connexes (le « logiciel »), de traiter le logiciel sans restriction, y compris, mais sans\ns'y limiter, les droits d'utiliser, de copier, de modifier, de fusionner, de publier, de distribuer,\nd'accorder une sous licence et de vendre des copies dudit logiciel, et de permettre aux personnes\nauxquelles le logiciel est fourni de le faire, selon les conditions suivantes :\n\nL'avis de droit d'auteur ci dessus et le présent avis de permission seront inclus dans toutes les copies\net les sections importantes du logiciel.\n\nLE LOGICIEL EST FOURNI « TEL QUEL », SANS AUCUNE GARANTIE, EXPRESSE OU IMPLICITE, Y COMPRIS, MAIS SANS\nS'Y LIMITER, LA GARANTIE DE QUALITÉ MARCHANDE, L'ADAPTATION À UN USAGE PARTICULIER ET L'ABSENCE DE\nCONTREFAÇON. EN AUCUN CAS LES AUTEURS OU LES DÉTENTEURS DU DROIT D'AUTEUR NE SERONT TENUS RESPONSABLES\nDE TOUTE DEMANDE, DOMMAGE OU BRIS DE CONTRAT, DÉLIT CIVIL OU TOUT AUTRE MANQUEMENT LIÉ AU LOGICIEL,\nÀ SON UTILISATION OU À D'AUTRES ÉCHANGES LIÉS AU LOGICIEL.\n" }, { "path": "README.md", "content": "## ckanapi\n\nA command line interface and Python module for accessing the\n[CKAN Action API](http://docs.ckan.org/en/latest/api/index.html#action-api-reference)\n\n- [Installation](https://github.com/ckan/ckanapi/blob/master/README.md#installation)\n- [ckanapi CLI](https://github.com/ckan/ckanapi/blob/master/README.md#ckanapi-cli)\n - [Actions](https://github.com/ckan/ckanapi/blob/master/README.md#actions)\n - [Action Arguments](https://github.com/ckan/ckanapi/blob/master/README.md#action-arguments)\n - [Bulk Dumping and Loading](https://github.com/ckan/ckanapi/blob/master/README.md#bulk-dumping-and-loading)\n - [Bulk Delete](https://github.com/ckan/ckanapi/blob/master/README.md#bulk-delete)\n - [Bulk Dataset and Resource Export](https://github.com/ckan/ckanapi/blob/master/README.md#bulk-dataset-and-resource-export---datapackagejson-format)\n - [Batch Actions](https://github.com/ckan/ckanapi/blob/master/README.md#batch-actions)\n - [Shell Pipelines](https://github.com/ckan/ckanapi/blob/master/README.md#shell-pipelines)\n- [ckanapi Python Module](https://github.com/ckan/ckanapi/blob/master/README.md#ckanapi-python-module)\n - [RemoteCKAN](https://github.com/ckan/ckanapi/blob/master/README.md#remoteckan)\n - [Exceptions](https://github.com/ckan/ckanapi/blob/master/README.md#exceptions)\n - [File Uploads](https://github.com/ckan/ckanapi/blob/master/README.md#file-uploads)\n - [Session Control](https://github.com/ckan/ckanapi/blob/master/README.md#session-control)\n - [LocalCKAN](https://github.com/ckan/ckanapi/blob/master/README.md#localckan)\n - [TestAppCKAN](https://github.com/ckan/ckanapi/blob/master/README.md#testappckan)\n- [Tests](https://github.com/ckan/ckanapi/blob/master/README.md#tests)\n- [License](https://github.com/ckan/ckanapi/blob/master/README.md#license)\n\n\n## Installation\n\nInstallation with pip:\n```\npip install ckanapi\n```\n\nInstallation with conda:\n```\nconda install -c conda-forge ckanapi\n```\n\n\n## ckanapi CLI\n\nThe ckanapi command line interface lets you access local and\nremote CKAN instances for bulk operations and simple API actions.\n\n\n### Actions\n\nSimple actions with string parameters may be called directly. The\nresponse is pretty-printed to STDOUT.\n\n#### 🔧 List names of groups on a remote CKAN site\n\n```\n$ ckanapi action group_list -r https://demo.ckan.org --insecure\n[\n \"data-explorer\",\n \"example-group\",\n \"geo-examples\",\n ...\n]\n```\n\nUse -r to specify the remote CKAN instance, and -a to provide an\nAPI KEY. Remote actions connect as an anonymous user by default.\nFor this example, we use --insecure as the CKAN demo uses a\nself-signed certificate.\n\nLocal CKAN actions may be run by specifying the config file with -c.\nIf no remote server or config file is specified, the CLI will look for\na ckan.ini file in the current directory, much like `ckan` commands.\n\nLocal CKAN actions are performed by the site user (default system\nadministrator) when -u is not specified.\n\nTo perform local actions with a less privileged user use\nthe -u option with a user name or a name that doesn't exist. This is\nuseful if you don't want things like deleted datasets or private\ninformation to be returned.\n\nNote that all actions in the [CKAN Action API](http://docs.ckan.org/en/latest/api/index.html#action-api-reference)\nand actions added by CKAN plugins are supported.\n\n\n### Action Arguments\n\nSimple action arguments may be passed in KEY=STRING form for string\nvalues or in KEY:JSON form for JSON values.\n\n#### 🔧 View a dataset using a KEY=STRING parameter\n\n```\n$ ckanapi action package_show id=my-dataset-name\n{\n \"name\": \"my-dataset-name\",\n ...\n}\n\n```\n\n#### 🔧 Get detailed info about a resource in the datastore\n\n```\n$ ckanapi action datastore_info id=my-resource-id-or-alias\n{\n \"meta\": {\n \"aliases\": [\n \"test_alias\"\n ],\n \"count\": 1000,\n ...\n}\n```\n\n#### 🔧 Get the number of datasets for each organization using KEY:JSON parameters\n\n```\n$ ckanapi action package_search facet.field:'[\"organization\"]' rows:0\n{\n \"facets\": {\n \"organization\": {\n \"org1\": 42,\n \"org2\": 21,\n ...\n }\n },\n ...\n}\n```\n\n#### 🔧 Create a resource with a file attached\n\nFiles may be passed for upload using the KEY@FILE form.\n\n```\n$ ckanapi action resource_create package_id=my-dataset-with-files \\\n upload@/path/to/file/to/upload.csv\n```\n\n#### 🔧 Edit a dataset with a text editor\n\n```\n$ ckanapi action package_show id=my-dataset-id > my-dataset.json\n$ nano my-dataset.json\n$ ckanapi action package_update -I my-dataset.json\n$ rm my-dataset.json\n```\n\n#### 🔧 Update a single resource field\n\n```\n$ ckanapi action resource_patch id=my-resource-id size:42000000\n```\n\n### Bulk Dumping and Loading\n\nDatasets, groups, organizations, users and related items may be dumped to\n[JSON lines](http://jsonlines.org)\ntext files and created or updated from JSON lines text files.\n\n`dump` and `load` jobs can be run in parallel with\nmultiple worker processes using the `-p` parameter. The jobs in progress,\nthe rate of job completion and any individual errors are shown on STDERR\nwhile the jobs run.\n\nThere are no parallel limits when running against a CKAN on localhost.\nWhen running against a remote site, there's a default limit of 3 worker processes.\n\nThe environment variables `CKANAPI_MY_SITES` and`CKANAPI_PARALLEL_LIMIT` can be\nused to adjust these limits. `CKANAPI_MY_SITES` (comma-delimited list of CKAN urls)\nwill not have the `PARALLEL_LIMIT` applied.\n\n`dump` and `load` jobs may be resumed from the last completed\nrecord or split across multiple servers by specifying record\nstart and max values.\n\n#### 🔧 Dump datasets from CKAN into a local file with 4 processes\n\n```\n$ ckanapi dump datasets --all -O datasets.jsonl.gz -z -p 4 -r http://localhost\n```\n\n#### 🔧 Export datasets including private ones using search\n\n```\n$ ckanapi search datasets include_private=true -O datasets.jsonl.gz -z \\\n -c /etc/ckan/production.ini\n```\n\n`search` is faster than `dump` because it calls `package_search` to retrieve\nmany records per call, paginating automatically.\n\nYou may add parameters supported by `package_search` to filter the\nrecords returned.\n\n\n#### 🔧 Load/update datasets from a dataset JSON lines file with 3 processes\n\n```\n$ ckanapi load datasets -I datasets.jsonl.gz -z -p 3 -c /etc/ckan/production.ini\n```\n\n\n### Bulk Delete\n\nDatasets, groups, organizations, users and related items may be deleted in\nbulk with the delete command. This command accepts ids or names on the\ncommand line or a number of different formats piped on standard input.\n\n#### 🔧 All datasets (JSON list of \"id\" or \"name\" values)\n```\n$ ckanapi action package_list -j | ckanapi delete datasets\n```\n\n#### 🔧 Selective delete (JSON object with \"results\" list containing \"id\" values)\n```\n$ ckanapi action package_search q=ponies | ckanapi delete datasets\n```\n\n#### 🔧 Processed JSON Lines (JSON objects with \"id\" or \"name\" value, one per line)\n```\n$ ckanapi dump groups --all > groups.jsonl\n$ grep ponies groups.jsonl | ckanapi delete groups\n```\n\n#### 🔧 Text list of \"id\" or \"name\" values (one per line)\n```\n$ cat users_to_remove.txt\nfred\nbill\nlarry\n$ ckanapi delete users < users_to_remove.txt\n```\n\n\n### Bulk Dataset and Resource Export - datapackage.json format\n\nDatasets may be exported to a simplified\n[datapackage.json format](http://dataprotocols.org/data-packages/)\n(which includes the actual resources, where available).\n\nIf the resource url is not available, the resource will be included\nin the datapackage.json file but the actual resource data will not be downloaded.\n\n```\n$ ckanapi dump datasets --all --datapackages=./output_directory/ -r http://sourceckan.example.com\n```\n\n### Batch Actions\n\nRun a set of actions from a JSON lines file. For local actions this is much faster than running\n`ckanapi action ...` in a shell loop because the local start-up time only happens once.\n\nBatch actions can also be run in parallel with multiple processes and errors logged, just like the\ndump and load commands.\n\n#### 🔧 Update a dataset field across a number of datasets\n```\n$ cat update-emails.jsonl\n{\"action\":\"package_patch\",\"data\":{\"id\":\"dataset-1\",\"maintainer_email\":\"new@example.com\"}}\n{\"action\":\"package_patch\",\"data\":{\"id\":\"dataset-2\",\"maintainer_email\":\"new@example.com\"}}\n{\"action\":\"package_patch\",\"data\":{\"id\":\"dataset-3\",\"maintainer_email\":\"new@example.com\"}}\n$ ckanapi batch -I update-emails.jsonl\n```\n\n#### 🔧 Replace a set of uploaded files\n```\n$ cat upload-files.jsonl\n{\"action\":\"resource_patch\",\"data\":{\"id\":\"408e1b1d-d0ca-50ca-9ae6-aedcee37aaa9\"},\"files\":{\"upload\":\"data1.csv\"}}\n{\"action\":\"resource_patch\",\"data\":{\"id\":\"c1eab17f-c2d0-536d-a3f6-41a3dfe6a2c3\"},\"files\":{\"upload\":\"data2.csv\"}}\n{\"action\":\"resource_patch\",\"data\":{\"id\":\"8ed068c2-4d4c-5f20-90db-39d2d596ce1a\"},\"files\":{\"upload\":\"data3.csv\"}}\n$ ckanapi batch -I upload-files.jsonl --local-files\n```\n\nThe `\"files\"` values in the JSON lines file is ignored unless the `--local-files` parameter is passed.\nPaths in the JSON lines file reference files on the local filesystems relative to the current working\ndirectory.\n\n### Shell pipelines\n\nSimple shell pipelines are possible with the CLI.\n\n#### 🔧 Copy the name of a dataset to its title with 'jq'\n```\n$ ckanapi action package_show id=my-dataset \\\n | jq '.+{\"title\":.name}' \\\n | ckanapi action package_update -i\n```\n\n#### 🔧 Mirror all datasets from one CKAN instance to another\n```\n$ ckanapi dump datasets --all -q -r http://sourceckan.example.com \\\n | ckanapi load datasets\n```\n\n\n## ckanapi Python Module\n\nThe ckanapi Python module may be used from within a\n[CKAN extension](http://docs.ckan.org/en/latest/extensions/index.html)\nor in a Python 2 or Python 3 application separate from CKAN.\n\n### RemoteCKAN\n\nMaking a request:\n\n```python\nfrom ckanapi import RemoteCKAN\nua = 'ckanapiexample/1.0 (+http://example.com/my/website)'\n\ndemo = RemoteCKAN('https://demo.ckan.org', user_agent=ua)\ngroups = demo.action.group_list(id='data-explorer')\nprint(groups)\n```\n\nresult:\n\n```\n[u'data-explorer', u'example-group', u'geo-examples', u'skeenawild']\n```\n\nThe example above is using an \"action shortcut\". The `.action` object detects\nthe method name used (\"group_list\" above) and converts it to a normal\n`call_action` call. This is equivalent code without using an action shortcut:\n\n```python\ngroups = demo.call_action('group_list', {'id': 'data-explorer'})\n```\n\nOnce again, all actions in the [CKAN Action API](http://docs.ckan.org/en/latest/api/index.html#action-api-reference)\nand actions added by CKAN plugins are supported by action shortcuts and\n`call_action` calls.\n\nFor example, if the [Showcase](https://github.com/ckan/ckanext-showcase#api) extension is installed:\n\n```python\nfrom ckanapi import RemoteCKAN\nua = 'ckanapiexample/1.0 (+http://example.com/my/website)'\n\ndemo = RemoteCKAN('https://demo.ckan.org', user_agent=ua)\nshowcases= demo.action.ckanext_showcase_list()\nprint(showcases)\n```\n\nCombining query parameters clauses is possible as in the following `package_search` action. This query combines three clauses that are all satisfied by the single [example dataset](https://demo.ckan.org/dataset/sample-dataset-1) in the Demo CKAN site.\n\nMore detailed complex query syntax examples can be found in the [SOLR documentation](https://solr.apache.org/guide/6_6/common-query-parameters.html).\n\n```python\nfrom ckanapi import RemoteCKAN\nua = 'ckanapiexample/1.0 (+http://example.com/my/website)'\n\ndemo = RemoteCKAN('https://demo.ckan.org', user_agent=ua)\npackages = demo.action.package_search(q='+organization:sample-organization +res_format:GeoJSON +tags:geojson')\nprint(packages)\n```\n\nMany CKAN API functions can only be used by authenticated users. Use the\n`apikey` parameter to supply your CKAN API key to `RemoteCKAN`:\n\n demo = RemoteCKAN('https://demo.ckan.org', apikey='MY-SECRET-API-KEY')\n\nAn example of updating a single field in an existing dataset can be seen in the [Examples directory](examples/update_single_field.py)\n\n### Exceptions\n\n* `NotAuthorized` - user unauthorized or accessing a deleted item\n* `NotFound` - name/id not found\n* `ValidationError` - field errors listed in `.error_dict`\n* `SearchQueryError` - error reported from SOLR index\n* `SearchError`\n* `CKANAPIError` - incorrect use of ckanapi or unable to parse response\n* `ServerIncompatibleError` - the remote API is not a CKAN API\n\nWhen using an action shortcut or the `call_action` method\nfailures are raised as exceptions just like when calling `get_action` from a\nCKAN plugin:\n\n```python\nfrom ckanapi import RemoteCKAN, NotAuthorized\nua = 'ckanapiexample/1.0 (+http://example.com/my/website)'\n\ndemo = RemoteCKAN('https://demo.ckan.org', apikey='phony-key', user_agent=ua)\ntry:\n pkg = demo.action.package_create(name='my-dataset', title='not going to work')\nexcept NotAuthorized:\n print('denied')\n```\n\nWhen it is possible to `import ckan` all the ckanapi exception classes are\nreplaced with the CKAN exceptions with the same names.\n\n\n### File Uploads\n\nFile uploads for CKAN 2.2+ are supported by passing file-like objects to action\nshortcut methods:\n\n```python\nfrom ckanapi import RemoteCKAN\nua = 'ckanapiexample/1.0 (+http://example.com/my/website)'\n\nmysite = RemoteCKAN('http://myckan.example.com', apikey='real-key', user_agent=ua)\nmysite.action.resource_create(\n package_id='my-dataset-with-files',\n url='dummy-value', # ignored but required by CKAN<2.6\n upload=open('/path/to/file/to/upload.csv', 'rb'))\n```\n\nWhen using `call_action` you must pass file objects separately:\n\n```python\nmysite.call_action('resource_create',\n {'package_id': 'my-dataset-with-files'},\n files={'upload': open('/path/to/file/to/upload.csv', 'rb')})\n```\n\n### Session Control\n\nAs of ckanapi 4.0 RemoteCKAN will keep your HTTP connection open using a\n[requests session](http://docs.python-requests.org/en/master/user/advanced/).\n\nFor long-running scripts make sure to close your connections by using\nRemoteCKAN as a context manager:\n\n```python\nfrom ckanapi import RemoteCKAN\nua = 'ckanapiexample/1.0 (+http://example.com/my/website)'\n\nwith RemoteCKAN('https://demo.ckan.org', user_agent=ua) as demo:\n groups = demo.action.group_list(id='data-explorer')\nprint(groups)\n```\n\nOr by explicitly calling `RemoteCKAN.close()`.\n\n### LocalCKAN\n\nA similar class is provided for accessing local CKAN instances from a plugin in\nthe same way as remote CKAN instances.\nUnlike [CKAN's get_action](http://docs.ckan.org/en/latest/extensions/plugins-toolkit.html?highlight=get_action#ckan.plugins.toolkit.get_action)\nLocalCKAN prevents data from one action\ncall leaking into the next which can cause issues that are very hard do debug.\n\nThis class defaults to using the site user with full access.\n\n```python\nfrom ckanapi import LocalCKAN, ValidationError\n\nregistry = LocalCKAN()\ntry:\n registry.action.package_create(name='my-dataset', title='this will work fine')\nexcept ValidationError:\n print('unless my-dataset already exists')\n```\n\nFor extra caution pass a blank username to LocalCKAN and only actions allowed\nby anonymous users will be permitted.\n\n```python\nfrom ckanapi import LocalCKAN\n\nanon = LocalCKAN(username='')\nprint(anon.action.status_show())\n```\n\n#### Extra Loggging\n\nTo enable extra info logging for the execution of LocalCKAN ckanapi commands, you can enable the config option in your CKAN INI file.\n\n```\nckanapi.log_local = True\n```\n\nThe output of the log will look like:\n\n```\nINFO [ckan.ckanapi] OS User executed LocalCKAN: ckanapi \n```\n\n### TestAppCKAN\n\nA class is provided for making action requests to a\n[webtest.TestApp](http://webtest.readthedocs.org/en/latest/testapp.html)\ninstance for use in CKAN tests:\n\n```python\nfrom ckanapi import TestAppCKAN\nfrom webtest import TestApp\n\ntest_app = TestApp(...)\ndemo = TestAppCKAN(test_app, apikey='my-test-key')\ngroups = demo.action.group_list(id='data-explorer')\n```\n\n## Timeouts\n\nAll requests performed to CKAN either via the CLI or the Python module can have a timeout defined.\nBy defaults it is not set, but you can define a custom timeout value using environment variables or passing\na value explicitly.\n\nTo use environment variables:\n\n* `CKANAPI_REQUEST_TIMEOUT`: this is the connect timeout (the time waited to connect to the remote server)\n* `CKANAPI_REQUEST_READ_TIMEOUT`: this is the read timeout (the time waited to receive a response)\n\nIf the read timeout is not defined, the connect timeout will be used. Please refer to\nthe [requests library documentation](https://requests.readthedocs.io/en/latest/user/advanced/#timeouts) for more details.\n\nTo pass a timeout on a `call_action` call use the `requests_kwargs` param:\n\n```\n```python\nfrom ckanapi import RemoteCKAN\n\ndemo = RemoteCKAN('https://demo.ckan.org')\ngroups = demo.action.group_list(id='data-explorer', requests_kwargs={\"timeout\": 10})\n\n```\n\n\n## Tests\n\nTo run the tests:\n\n python -m unittest discover\n\n\n## License\n\n🇨🇦 Government of Canada / Gouvernement du Canada\n\nThe project files are covered under Crown Copyright, Government of Canada\nand is distributed under the MIT license. Please see [COPYING](COPYING) /\n[COPYING.fr](COPYING.fr) for full details.\n" }, { "path": "ckanapi/__init__.py", "content": "\"\"\"\nckanapi\n-------\n\nThis module a thin wrapper around the CKAN's action API.\n\"\"\"\n\nfrom ckanapi.errors import (\n CKANAPIError,\n NotAuthorized,\n NotFound,\n ValidationError,\n SearchQueryError,\n SearchError,\n SearchIndexError,\n ServerIncompatibleError,\n )\nfrom ckanapi.localckan import LocalCKAN\nfrom ckanapi.remoteckan import RemoteCKAN\nfrom ckanapi.testappckan import TestAppCKAN\n\n\n\n\n\n\n" }, { "path": "ckanapi/cli/__init__.py", "content": "" }, { "path": "ckanapi/cli/action.py", "content": "\"\"\"\nimplementation of the action cli command\n\"\"\"\n\nimport sys\nimport json\nfrom os.path import expanduser\n\nfrom ckanapi.cli.utils import compact_json, pretty_json\nfrom ckanapi.errors import CLIError\n\n\ndef action(ckan, arguments, stdin=None):\n \"\"\"\n call an action with KEY=STRING, KEY:JSON or JSON args, yield the result\n \"\"\"\n if stdin is None:\n stdin = getattr(sys.stdin, 'buffer', sys.stdin)\n\n file_args = {}\n requests_kwargs = None\n if arguments['--insecure']:\n requests_kwargs = {'verify': False}\n if arguments['--input-json']:\n action_args = json.loads(stdin.read().decode('utf-8'))\n elif arguments['--input']:\n action_args = {}\n with open(expanduser(arguments['--input'])) as in_f:\n action_args = json.loads(\n in_f.read())\n else:\n action_args = {}\n for kv in arguments['KEY=STRING']:\n if hasattr(kv, 'decode'):\n kv = kv.decode('utf-8')\n skey, p, svalue = kv.partition('=')\n jkey, p, jvalue = kv.partition(':')\n fkey, p, fvalue = kv.partition('@')\n if len(jkey) > len(skey) < len(fkey):\n action_args[skey] = svalue\n elif len(skey) > len(jkey) < len(fkey):\n try:\n value = json.loads(jvalue)\n except ValueError:\n raise CLIError(\"KEY:JSON argument %r has invalid JSON \"\n \"value %r\" % (jkey, jvalue))\n action_args[jkey] = value\n elif len(jkey) > len(fkey) < len(skey):\n try:\n f = open(expanduser(fvalue), 'rb')\n except IOError as e:\n raise CLIError(\"Error opening %r: %s\" %\n (expanduser(fvalue), e.args[1]))\n file_args[fkey] = f\n else:\n raise CLIError(\"argument not in the form KEY=STRING, \"\n \"KEY:JSON or KEY@FILE %r\" % kv)\n\n def call():\n return ckan.call_action(arguments['ACTION_NAME'], action_args,\n files=file_args,\n requests_kwargs=requests_kwargs)\n\n if arguments['--profile']:\n from cProfile import Profile\n with Profile() as pr:\n result = call()\n pr.dump_stats(arguments['--profile'])\n else:\n result = call()\n\n if arguments['--output-jsonl']:\n if isinstance(result, list):\n for r in result:\n yield compact_json(r) + b'\\n'\n else:\n yield compact_json(result) + b'\\n'\n elif arguments['--output-json']:\n yield compact_json(result) + b'\\n'\n else:\n yield pretty_json(result) + b'\\n'\n" }, { "path": "ckanapi/cli/batch.py", "content": "\"\"\"\nimplementation of batch cli command\n\"\"\"\n\nimport sys\nimport gzip\nimport json\nfrom datetime import datetime\n\nfrom ckanapi.errors import (NotFound, NotAuthorized, ValidationError,\n SearchIndexError)\nfrom ckanapi.cli import workers\nfrom ckanapi.cli.utils import completion_stats, compact_json, quiet_int_pipe\n\n\ndef batch_actions(ckan, arguments,\n worker_pool=None, stdin=None, stdout=None, stderr=None):\n \"\"\"\n call actions from a jsonl file\n\n The parent process creates a pool of worker processes and hands\n out json lines to each worker as they finish a task. Status of\n last record completed and records being processed is displayed\n on stderr.\n \"\"\"\n if worker_pool is None:\n worker_pool = workers.worker_pool\n if stdin is None:\n stdin = getattr(sys.stdin, 'buffer', sys.stdin)\n if stdout is None:\n stdout = getattr(sys.__stdout__, 'buffer', sys.__stdout__)\n if stderr is None:\n stderr = getattr(sys.stderr, 'buffer', sys.stderr)\n\n if arguments['--worker']:\n return batch_actions_worker(ckan, arguments)\n\n log = None\n if arguments['--log']:\n log = open(arguments['--log'], 'ab')\n\n jsonl_input = stdin\n if arguments['--input']:\n jsonl_input = open(arguments['--input'], 'rb')\n if arguments['--gzip']:\n jsonl_input = gzip.GzipFile(fileobj=jsonl_input)\n\n def line_reader():\n \"\"\"\n handle start-record and max-records options\n \"\"\"\n start_record = int(arguments['--start-record'])\n max_records = arguments['--max-records']\n if max_records is not None:\n max_records = int(max_records)\n for num, line in enumerate(jsonl_input, 1): # records start from 1\n if num < start_record:\n continue\n if max_records is not None and num >= start_record + max_records:\n break\n yield num, line\n\n cmd = _worker_command_line(arguments)\n processes = int(arguments['--processes'])\n if hasattr(ckan, 'parallel_limit'):\n # add your sites to CKANAPI_MY_SITES instead of removing\n processes = min(processes, ckan.parallel_limit)\n stats = completion_stats(processes)\n pool = worker_pool(cmd, processes, line_reader())\n\n with quiet_int_pipe() as errors:\n for job_ids, finished, result in pool:\n if not result:\n # child exited with traceback\n return 1\n timestamp, action, error, response = json.loads(\n result.decode('utf-8'))\n\n if not arguments['--quiet']:\n stderr.write(('%s %s %s %s %s %s\\n' % (\n finished,\n job_ids,\n next(stats),\n action,\n error,\n compact_json(response).decode('utf-8') if response else ''\n )).encode('utf-8'))\n\n if log:\n log.write(compact_json([\n timestamp,\n finished,\n action,\n error,\n response,\n ]) + b'\\n')\n log.flush()\n if 'pipe' in errors:\n return 1\n if 'interrupt' in errors:\n return 2\n\n\ndef batch_actions_worker(ckan, arguments,\n stdin=None, stdout=None):\n \"\"\"\n a process that accepts lines of json on stdin which is parsed and\n passed to action calls. it produces lines of json\n which are the responses from each action call.\n \"\"\"\n if stdin is None:\n stdin = getattr(sys.stdin, 'buffer', sys.stdin)\n # hack so that pdb can be used in extension/ckan\n # code called by this worker\n try:\n sys.stdin = open('/dev/tty', 'rb')\n except IOError:\n pass\n if stdout is None:\n stdout = getattr(sys.__stdout__, 'buffer', sys.__stdout__)\n # hack so that \"print debugging\" can work in extension/ckan\n # code called by this worker\n sys.stdout = sys.stderr\n\n def reply(action, error, response):\n \"\"\"\n format messages to be sent back to parent process\n \"\"\"\n stdout.write(compact_json([\n datetime.now().isoformat(),\n action,\n error,\n response]) + b'\\n')\n stdout.flush()\n\n for line in iter(stdin.readline, b''):\n try:\n obj = json.loads(line.decode('utf-8'))\n except UnicodeDecodeError as e:\n obj = None\n reply('read', 'UnicodeDecodeError', str(e))\n continue\n\n requests_kwargs = None\n if arguments['--insecure']:\n requests_kwargs = {'verify': False}\n\n if obj is not None:\n action = obj['action']\n data = obj.get('data', {})\n files = {}\n if arguments['--local-files']:\n try:\n for fkey, fvalue in obj.get('files', {}).items():\n f = open(fvalue, 'rb')\n files[fkey] = f\n except IOError as e:\n reply('read', 'IOError', {\n 'parameter':fkey,\n 'file_name':fvalue,\n 'error':str(e.args[1]),\n })\n continue\n\n try:\n r = ckan.call_action(action, data, files=files,\n requests_kwargs=requests_kwargs)\n except ValidationError as e:\n reply(action, 'ValidationError', e.error_dict)\n except SearchIndexError as e:\n reply(action, 'SearchIndexError', str(e))\n except NotAuthorized as e:\n reply(action, 'NotAuthorized', str(e))\n except NotFound:\n reply(action, 'NotFound', obj)\n else:\n reply(action, None, r)\n\ndef _worker_command_line(arguments):\n \"\"\"\n Create a worker command line suitable for Popen with only the\n options the worker process requires\n \"\"\"\n def a(name):\n \"options with values\"\n return [name, arguments[name]] * (arguments[name] is not None)\n def b(name):\n \"boolean options\"\n return [name] * bool(arguments[name])\n return (\n ['ckanapi', 'batch', '--worker']\n + a('--config')\n + a('--ckan-user')\n + a('--remote')\n + a('--apikey')\n + b('--local-files')\n + b('--insecure')\n )\n" }, { "path": "ckanapi/cli/ckan_click.py", "content": "import click\n\n@click.command(\n context_settings={'ignore_unknown_options': True},\n short_help='Local API calls with ckanapi tool'\n)\n@click.argument('args', nargs=-1, type=click.UNPROCESSED)\n@click.pass_context\ndef api(context, args):\n from ckanapi.cli.main import main\n import sys\n sys.argv[1:] = args\n context.exit(main(running_with_ckan_command=True) or 0)\n" }, { "path": "ckanapi/cli/delete.py", "content": "\"\"\"\nimplementation of delete cli command\n\"\"\"\n\nimport sys\nimport gzip\nimport json\nfrom datetime import datetime\nfrom itertools import chain\nimport re\nfrom urllib.parse import urlparse\n\nfrom ckanapi.errors import (NotFound, NotAuthorized, ValidationError,\n SearchIndexError)\nfrom ckanapi.cli import workers\nfrom ckanapi.cli.utils import completion_stats, compact_json, quiet_int_pipe\n\n\ndef delete_things(ckan, thing, arguments,\n worker_pool=None, stdin=None, stdout=None, stderr=None):\n \"\"\"\n delete datasets, groups, orgs, users etc,\n\n The parent process creates a pool of worker processes and hands\n out json lines to each worker as they finish a task. Status of\n last record completed and records being processed is displayed\n on stderr.\n \"\"\"\n if worker_pool is None:\n worker_pool = workers.worker_pool\n if stdin is None:\n stdin = getattr(sys.stdin, 'buffer', sys.stdin)\n if stdout is None:\n stdout = getattr(sys.__stdout__, 'buffer', sys.__stdout__)\n if stderr is None:\n stderr = getattr(sys.stderr, 'buffer', sys.stderr)\n\n if arguments['--worker']:\n return delete_things_worker(ckan, thing, arguments)\n\n log = None\n if arguments['--log']:\n log = open(arguments['--log'], 'ab')\n\n jsonl_input = stdin\n if arguments['--input']:\n jsonl_input = open(arguments['--input'], 'rb')\n if arguments['--gzip']:\n jsonl_input = gzip.GzipFile(fileobj=jsonl_input)\n\n def name_reader():\n \"\"\"\n handle start-record and max-records options and extract all\n ids or names from each line (e.g. package_search, package_show\n or package_list output)\n record numbers here correspond to names/ids extracted not lines\n \"\"\"\n start_record = int(arguments['--start-record'])\n max_records = arguments['--max-records']\n if max_records is not None:\n max_records = int(max_records)\n\n for num, name in enumerate(chain.from_iterable(\n extract_ids_or_names(line) for line in jsonl_input), 1):\n if num < start_record:\n continue\n if max_records is not None and num >= start_record + max_records:\n break\n yield num, compact_json(name)\n\n cmd = _worker_command_line(thing, arguments)\n processes = int(arguments['--processes'])\n if hasattr(ckan, 'parallel_limit'):\n # add your sites to CKANAPI_MY_SITES instead of removing\n processes = min(processes, ckan.parallel_limit)\n stats = completion_stats(processes)\n if not arguments['ID_OR_NAME']:\n pool = worker_pool(cmd, processes, name_reader())\n else:\n pool = worker_pool(cmd, processes, enumerate(\n (compact_json(n) + b'\\n' for n in arguments['ID_OR_NAME']), 1))\n\n with quiet_int_pipe() as errors:\n for job_ids, finished, result in pool:\n if not result:\n # child exited with traceback\n return 1\n timestamp, error, response = json.loads(\n result.decode('utf-8'))\n\n if not arguments['--quiet']:\n stderr.write(('%s %s %s %s %s\\n' % (\n finished,\n job_ids,\n next(stats),\n error,\n compact_json(response).decode('utf-8') if response else ''\n )).encode('utf-8'))\n\n if log:\n log.write(compact_json([\n timestamp,\n finished,\n error,\n response,\n ]) + b'\\n')\n log.flush()\n if 'pipe' in errors:\n return 1\n if 'interrupt' in errors:\n return 2\n\n\ndef extract_ids_or_names(line):\n \"\"\"\n Be generous in what we accept:\n\n line may contain\n 1. a JSON object with an \"id\" or \"name\" value (e.g. package_show result)\n 2. a JSON object with a \"results\" value containing a list\n of objects with \"id\" values (e.g. package_search result)\n 3. a JSON string id or name value\n 4. a JSON list of string id or name values (e.g. package_list)\n 5. a simple string id or name value\n\n Returns a list of ids or names found in line\n \"\"\"\n try:\n j = json.loads(line)\n except ValueError:\n return [line.strip()] # 5\n if isinstance(j, list) and all(\n isinstance(e, str) for e in j):\n return j # 4\n elif isinstance(j, str):\n return [j] # 3\n elif isinstance(j, dict):\n if 'id' in j and isinstance(j['id'], str):\n return [j['id']] # 1\n if 'name' in j and isinstance(j['name'], str):\n return [j['name']] # 1 again\n if 'results' in j and isinstance(j['results'], list):\n out = []\n for r in j['results']:\n if (not isinstance(r, dict) or 'id' not in r or\n not isinstance(r['id'], str)):\n break\n out.append(r['id'])\n else:\n return out\n\n # 5 again (e.g. \"true\" or \"null\" or something stranger)\n return [line.strip()]\n\n\ndef delete_things_worker(ckan, thing, arguments,\n stdin=None, stdout=None):\n \"\"\"\n a process that accepts lines of json on stdin which is parsed and\n passed to the {thing}_delete actions. it produces lines of json\n which are the responses from each action call.\n \"\"\"\n if stdin is None:\n stdin = getattr(sys.stdin, 'buffer', sys.stdin)\n # hack so that pdb can be used in extension/ckan\n # code called by this worker\n try:\n sys.stdin = open('/dev/tty', 'rb')\n except IOError:\n pass\n if stdout is None:\n stdout = getattr(sys.__stdout__, 'buffer', sys.__stdout__)\n # hack so that \"print debugging\" can work in extension/ckan\n # code called by this worker\n sys.stdout = sys.stderr\n\n thing_delete = {\n 'datasets': 'package_delete',\n 'groups': 'group_delete',\n 'organizations': 'organization_delete',\n 'users': 'user_delete',\n 'related': 'related_delete',\n }[thing]\n\n def reply(error, response):\n \"\"\"\n format messages to be sent back to parent process\n \"\"\"\n stdout.write(compact_json([\n datetime.now().isoformat(),\n error,\n response]) + b'\\n')\n stdout.flush()\n\n for line in iter(stdin.readline, b''):\n try:\n name = json.loads(line.decode('utf-8'))\n except UnicodeDecodeError as e:\n reply('UnicodeDecodeError', str(e))\n continue\n\n try:\n requests_kwargs = None\n if arguments['--insecure']:\n requests_kwargs = {'verify': False}\n ckan.call_action(thing_delete, {'id': name},\n requests_kwargs=requests_kwargs)\n except NotAuthorized as e:\n reply('NotAuthorized', str(e))\n except NotFound:\n reply('NotFound', name)\n else:\n reply(None, name)\n\ndef _worker_command_line(thing, arguments):\n \"\"\"\n Create a worker command line suitable for Popen with only the\n options the worker process requires\n \"\"\"\n def a(name):\n \"options with values\"\n return [name, arguments[name]] * (arguments[name] is not None)\n return (\n ['ckanapi', 'delete', thing, '--worker']\n + a('--config')\n + a('--ckan-user')\n + a('--remote')\n + a('--apikey')\n )\n" }, { "path": "ckanapi/cli/dump.py", "content": "\"\"\"\nimplementation of dump cli command\n\"\"\"\n\nimport sys\nimport gzip\nimport json\nfrom datetime import datetime\nimport os\n\nfrom ckanapi.errors import (CKANAPIError, NotFound, NotAuthorized, ValidationError,\n SearchIndexError)\nfrom ckanapi.cli import workers\nfrom ckanapi.cli.utils import completion_stats, compact_json, \\\n quiet_int_pipe\nfrom ckanapi.datapackage import create_datapackage, \\\n populate_datastore_res_fields\n\n\ndef dump_things(ckan, thing, arguments,\n worker_pool=None, stdout=None, stderr=None):\n \"\"\"\n dump all datasets, groups, orgs or users accessible by the connected user\n\n The parent process creates a pool of worker processes and hands\n out ids to each worker. Status of last record completed and records\n being processed is displayed on stderr.\n \"\"\"\n if worker_pool is None:\n worker_pool = workers.worker_pool\n if stdout is None:\n stdout = getattr(sys.__stdout__, 'buffer', sys.__stdout__)\n if stderr is None:\n stderr = getattr(sys.stderr, 'buffer', sys.stderr)\n\n if arguments['--worker']:\n return dump_things_worker(ckan, thing, arguments)\n\n log = None\n if arguments['--log']:\n log = open(arguments['--log'], 'ab')\n\n jsonl_output = stdout\n if arguments['--datapackages']: # TODO: do we want to just divert this to devnull?\n jsonl_output = open(os.devnull, 'wb')\n if arguments['--output']:\n jsonl_output = open(arguments['--output'], 'wb')\n if arguments['--gzip']:\n jsonl_output = gzip.GzipFile(fileobj=jsonl_output)\n if arguments['--all']:\n params = None\n get_thing_list = {\n 'datasets': 'package_list',\n 'groups': 'group_list',\n 'organizations': 'organization_list',\n 'users': 'user_list',\n 'related' :'related_list',\n }[thing]\n if get_thing_list == \"user_list\":\n params = dict(\n all_fields=False\n )\n elif get_thing_list == \"package_list\":\n params = dict(\n include_private=arguments['--include-private'] if '--include-private' in arguments else False,\n include_drafts=arguments['--include-drafts'] if '--include-drafts' in arguments else False,\n include_deleted=arguments['--include-deleted'] if '--include-deleted' in arguments else False,\n )\n\n names = ckan.call_action(get_thing_list, params)\n\n else:\n names = arguments['ID_OR_NAME']\n\n if names and isinstance(names[0], dict):\n names = [rec.get('name',rec.get('id')) for rec in names]\n\n if arguments['--datapackages']:\n arguments['--datastore-fields'] = True\n cmd = _worker_command_line(thing, arguments)\n processes = int(arguments['--processes'])\n if hasattr(ckan, 'parallel_limit'):\n # add your sites to CKANAPI_MY_SITES instead of removing\n processes = min(processes, ckan.parallel_limit)\n stats = completion_stats(processes)\n pool = worker_pool(cmd, processes,\n enumerate(compact_json(n) + b'\\n' for n in names))\n\n results = {}\n expecting_number = 0\n with quiet_int_pipe() as errors:\n for job_ids, finished, result in pool:\n if not result:\n # child exited with traceback\n return 1\n timestamp, error, record = json.loads(result.decode('utf-8'))\n results[finished] = record\n\n if not arguments['--quiet']:\n stderr.write('{0} {1} {2} {3} {4}\\n'.format(\n finished,\n job_ids,\n next(stats),\n error,\n record.get('name', '') if record else '',\n ).encode('utf-8'))\n\n if log:\n log.write(compact_json([\n timestamp,\n finished,\n error,\n record.get('name', '') if record else None,\n ]) + b'\\n')\n\n datapackages_path = arguments['--datapackages']\n apikey = arguments['--apikey']\n if datapackages_path:\n create_datapackage(record, datapackages_path, stderr, apikey)\n\n # keep the output in the same order as names\n while expecting_number in results:\n record = results.pop(expecting_number)\n if record:\n # sort keys so we can diff output\n jsonl_output.write(compact_json(record,\n sort_keys=True) + b'\\n')\n expecting_number += 1\n if jsonl_output != stdout:\n jsonl_output.close()\n if 'pipe' in errors:\n return 1\n if 'interrupt' in errors:\n return 2\n\n\ndef dump_things_worker(ckan, thing, arguments,\n stdin=None, stdout=None):\n \"\"\"\n a process that accepts names on stdin which are\n passed to the {thing}_show actions. it produces lines of json\n which are the responses from each action call.\n \"\"\"\n if stdin is None:\n stdin = getattr(sys.stdin, 'buffer', sys.stdin)\n # hack so that pdb can be used in extension/ckan\n # code called by this worker\n try:\n sys.stdin = open('/dev/tty', 'rb')\n except IOError:\n pass\n if stdout is None:\n stdout = getattr(sys.__stdout__, 'buffer', sys.__stdout__)\n # hack so that \"print debugging\" can work in extension/ckan\n # code called by this worker\n sys.stdout = sys.stderr\n\n thing_show = {\n 'datasets': 'package_show',\n 'groups': 'group_show',\n 'organizations': 'organization_show',\n 'users': 'user_show',\n 'related':'related_show'\n }[thing]\n\n def reply(error, record=None):\n \"\"\"\n format messages to be sent back to parent process\n \"\"\"\n stdout.write(compact_json([\n datetime.now().isoformat(),\n error,\n record]) + b'\\n')\n stdout.flush()\n\n for line in iter(stdin.readline, b''):\n try:\n name = json.loads(line.decode('utf-8'))\n except UnicodeDecodeError as e:\n reply('UnicodeDecodeError')\n continue\n\n try:\n requests_kwargs = None\n if arguments['--insecure']:\n requests_kwargs = {'verify': False}\n include_users = False\n if '--include-users' in arguments \\\n and arguments['--include-users']:\n include_users = True\n obj = ckan.call_action(thing_show, {'id': name,\n 'include_datasets': False,\n 'include_password_hash': True,\n 'include_users': include_users,\n }, requests_kwargs=requests_kwargs)\n except NotFound:\n reply('NotFound')\n except NotAuthorized:\n reply('NotAuthorized')\n else:\n if thing == 'datasets' and arguments['--datastore-fields']:\n for res in obj.get('resources', []):\n populate_datastore_res_fields(ckan, res)\n if thing == 'datasets' and arguments['--resource-views']:\n for res in obj.get('resources', []):\n populate_res_views(ckan, res)\n reply(None, obj)\n\ndef _worker_command_line(thing, arguments):\n \"\"\"\n Create a worker command line suitable for Popen with only the\n options the worker process requires\n \"\"\"\n def a(name):\n \"options with values\"\n return [name, arguments[name]] * (arguments[name] is not None)\n def b(name):\n \"boolean options\"\n return [name] * bool(arguments[name])\n return (\n ['ckanapi', 'dump', thing, '--worker']\n + a('--config')\n + a('--ckan-user')\n + a('--remote')\n + a('--apikey')\n + b('--get-request')\n + b('--datastore-fields')\n + b('--resource-views')\n + b('--include-users')\n + ['value-here-to-make-docopt-happy']\n )\n\n\ndef populate_res_views(ckan, res):\n \"\"\"\n update resource dict in-place with resource_view_list values\n in every resource with views using ckan LocalCKAN/RemoteCKAN instance\n \"\"\"\n try:\n views = ckan.call_action('resource_view_list', {\n 'id': res['id'],\n 'limit':0})\n except CKANAPIError:\n return\n except NotFound:\n return # with localckan we'll get the real CKAN exception not a CKANAPIError subclass\n if not views:\n return # return if the resource views list is empty\n res['resource_views'] = views\n\n" }, { "path": "ckanapi/cli/load.py", "content": "\"\"\"\nimplementation of load cli command\n\"\"\"\n\nimport sys\nimport gzip\nimport json\nimport requests\nfrom datetime import datetime\nimport re\nfrom urllib.parse import urlparse\n\nfrom ckanapi.common import REQUEST_TIMEOUT\nfrom ckanapi.errors import (NotFound, NotAuthorized, ValidationError,\n SearchIndexError)\nfrom ckanapi.cli import workers\nfrom ckanapi.cli.utils import completion_stats, compact_json, quiet_int_pipe\n\n\ndef load_things(ckan, thing, arguments,\n worker_pool=None, stdin=None, stdout=None, stderr=None):\n \"\"\"\n create and update datasets, groups, orgs and users\n\n The parent process creates a pool of worker processes and hands\n out json lines to each worker as they finish a task. Status of\n last record completed and records being processed is displayed\n on stderr.\n \"\"\"\n if worker_pool is None:\n worker_pool = workers.worker_pool\n if stdin is None:\n stdin = getattr(sys.stdin, 'buffer', sys.stdin)\n if stdout is None:\n stdout = getattr(sys.__stdout__, 'buffer', sys.__stdout__)\n if stderr is None:\n stderr = getattr(sys.stderr, 'buffer', sys.stderr)\n\n if arguments['--worker']:\n return load_things_worker(ckan, thing, arguments)\n\n log = None\n if arguments['--log']:\n log = open(arguments['--log'], 'ab')\n\n jsonl_input = stdin\n if arguments['--input']:\n jsonl_input = open(arguments['--input'], 'rb')\n if arguments['--gzip']:\n jsonl_input = gzip.GzipFile(fileobj=jsonl_input)\n\n def line_reader():\n \"\"\"\n handle start-record and max-records options\n \"\"\"\n start_record = int(arguments['--start-record'])\n max_records = arguments['--max-records']\n if max_records is not None:\n max_records = int(max_records)\n for num, line in enumerate(jsonl_input, 1): # records start from 1\n if num < start_record:\n continue\n if max_records is not None and num >= start_record + max_records:\n break\n yield num, line\n\n cmd = _worker_command_line(thing, arguments)\n processes = int(arguments['--processes'])\n if hasattr(ckan, 'parallel_limit'):\n # add your sites to CKANAPI_MY_SITES instead of removing\n processes = min(processes, ckan.parallel_limit)\n stats = completion_stats(processes)\n pool = worker_pool(cmd, processes, line_reader())\n\n failures = 0\n with quiet_int_pipe() as errors:\n for job_ids, finished, result in pool:\n if not result:\n # child exited with traceback\n return 1\n timestamp, action, error, response = json.loads(\n result.decode('utf-8'))\n if error:\n failures += 1\n\n if not arguments['--quiet']:\n stderr.write(('%s %s %s %s %s %s\\n' % (\n finished,\n job_ids,\n next(stats),\n action,\n error,\n compact_json(response).decode('utf-8') if response else ''\n )).encode('utf-8'))\n\n if log:\n log.write(compact_json([\n timestamp,\n finished,\n action,\n error,\n response,\n ]) + b'\\n')\n log.flush()\n if 'pipe' in errors:\n return 1\n if 'interrupt' in errors:\n return 2\n if failures:\n return 3\n\n\ndef load_things_worker(ckan, thing, arguments,\n stdin=None, stdout=None):\n \"\"\"\n a process that accepts lines of json on stdin which is parsed and\n passed to the {thing}_create/update actions. it produces lines of json\n which are the responses from each action call.\n \"\"\"\n if stdin is None:\n stdin = getattr(sys.stdin, 'buffer', sys.stdin)\n # hack so that pdb can be used in extension/ckan\n # code called by this worker\n try:\n sys.stdin = open('/dev/tty', 'rb')\n except IOError:\n pass\n if stdout is None:\n stdout = getattr(sys.__stdout__, 'buffer', sys.__stdout__)\n # hack so that \"print debugging\" can work in extension/ckan\n # code called by this worker\n sys.stdout = sys.stderr\n\n thing_show, thing_create, thing_update = {\n 'datasets': (\n 'package_show', 'package_create', 'package_update'),\n 'groups': (\n 'group_show', 'group_create', 'group_update'),\n 'organizations': (\n 'organization_show', 'organization_create', 'organization_update'),\n 'users': (\n 'user_show', 'user_create', 'user_update'),\n 'related':(\n 'related_show','related_create','related_update'),\n }[thing]\n\n def reply(action, error, response):\n \"\"\"\n format messages to be sent back to parent process\n \"\"\"\n stdout.write(compact_json([\n datetime.now().isoformat(),\n action,\n error,\n response]) + b'\\n')\n stdout.flush()\n\n for line in iter(stdin.readline, b''):\n try:\n obj = json.loads(line.decode('utf-8'))\n except UnicodeDecodeError as e:\n obj = None\n reply('read', 'UnicodeDecodeError', str(e))\n continue\n\n requests_kwargs = None\n if arguments['--insecure']:\n requests_kwargs = {'verify': False}\n\n if obj is not None:\n existing = None\n if not arguments['--create-only']:\n # use either id or name to locate existing records\n name = obj.get('id')\n if name:\n try:\n existing = ckan.call_action(thing_show,\n {'id': name,\n 'include_datasets': False,\n 'include_password_hash': True,\n 'include_users': True,\n },\n requests_kwargs=requests_kwargs)\n except NotFound:\n pass\n except NotAuthorized as e:\n reply('show', 'NotAuthorized', str(e))\n continue\n name = obj.get('name')\n if not existing and name:\n try:\n existing = ckan.call_action(thing_show, {'id': name},\n requests_kwargs=requests_kwargs)\n except NotFound:\n pass\n except NotAuthorized as e:\n reply('show', 'NotAuthorized', str(e))\n continue\n\n if existing:\n _copy_from_existing_for_update(obj, existing, thing)\n\n # FIXME: compare and reply when 'unchanged'?\n\n if not existing and arguments['--update-only']:\n reply('show', 'NotFound', [obj.get('id'), obj.get('name')])\n continue\n\n act = 'update' if existing else 'create'\n try:\n if existing:\n r = ckan.call_action(thing_update, obj,\n requests_kwargs=requests_kwargs)\n else:\n r = ckan.call_action(thing_create, obj)\n if thing == 'datasets' and 'resources' in obj:# check if it is needed to upload resources when creating/updating packages\n _upload_resources(ckan,obj,arguments)\n elif thing in ['groups','organizations'] and 'image_display_url' in obj: #load images for groups and organizations\n if arguments['--upload-logo']:\n users = obj['users']\n obj = _upload_logo(ckan,obj)\n obj.pop('image_upload')\n obj['users'] = users\n ckan.call_action(thing_update, obj,\n requests_kwargs=requests_kwargs)\n except ValidationError as e:\n reply(act, 'ValidationError', e.error_dict)\n except SearchIndexError as e:\n reply(act, 'SearchIndexError', str(e))\n except NotAuthorized as e:\n reply(act, 'NotAuthorized', str(e))\n except NotFound:\n reply(act, 'NotFound', obj)\n else:\n reply(act, None, r.get('name',r.get('id')))\n\ndef _worker_command_line(thing, arguments):\n \"\"\"\n Create a worker command line suitable for Popen with only the\n options the worker process requires\n \"\"\"\n def a(name):\n \"options with values\"\n return [name, arguments[name]] * (arguments[name] is not None)\n def b(name):\n \"boolean options\"\n return [name] * bool(arguments[name])\n return (\n ['ckanapi', 'load', thing, '--worker']\n + a('--config')\n + a('--ckan-user')\n + a('--remote')\n + a('--apikey')\n + b('--create-only')\n + b('--update-only')\n + b('--upload-resources')\n + b('--upload-logo')\n )\n\n\ndef _copy_from_existing_for_update(obj, existing, thing):\n \"\"\"\n modifies obj dict in place, copying values from existing.\n\n the id is alwasys copied from existing to make sure update updates\n the correct object.\n\n users lists for groups and orgs are maintained if not present in obj\n \"\"\"\n if 'id' in existing:\n obj['id'] = existing['id']\n\n if thing in ('organizations', 'groups'):\n if 'users' not in obj and 'users' in existing:\n obj['users'] = existing['users']\n\ndef _upload_resources(ckan,obj,arguments):\n resources = obj['resources']\n if not arguments['--upload-resources']:\n return\n requests_kwargs = None\n if arguments['--insecure']:\n requests_kwargs = {'verify': False}\n for resource in resources:\n if resource.get('url_type') != 'upload':\n continue\n\n f = requests.get(resource['url'], stream=True, timeout=REQUEST_TIMEOUT)\n name = resource['url'].rsplit('/',1)[-1]\n ckan.call_action('resource_patch',\n {'id':resource['id']},\n files={'upload':(name, f.raw)},\n requests_kwargs=requests_kwargs)\n\n\ndef _upload_logo(ckan,obj_orig):\n obj = obj_orig.copy()\n for key in obj_orig.keys():\n if isinstance(obj[key],(dict,list)):\n obj.pop(key) #dict/list objects can't be encoded\n if urlparse(obj['image_url']).netloc: # logo is an external link\n obj['clear_upload'] = True\n obj['image_upload'] = obj['image_url']\n else:\n f = requests.get(obj['image_display_url'], stream=True, timeout=REQUEST_TIMEOUT)\n name,ext = obj['image_url'].rsplit('.',1) #reformulate image_url for new site\n new_name = re.sub('[0-9.-]','',name)\n new_url = new_name+'.'+ext\n obj['image_upload'] = (new_url, f.raw)\n ckan.action.group_update(**obj)\n return obj\n" }, { "path": "ckanapi/cli/main.py", "content": "\"\"\"ckanapi command line inter face\n\nUsage:\n ckanapi action ACTION_NAME\n [(KEY=STRING | KEY:JSON | KEY@FILE ) ... | -i | -I JSON_INPUT]\n [-j | -J] [-P PROFILE ]\n [[-c CONFIG] [-u USER] | -r SITE_URL [-a APIKEY] [-g] [--insecure]]\n ckanapi batch [-I JSONL_INPUT] [-s START] [-m MAX] [--local-files]\n [-p PROCESSES] [-l LOG_FILE] [-qwz]\n [[-c CONFIG] [-u USER] | -r SITE_URL [-a APIKEY] [--insecure]]\n ckanapi delete (datasets | groups | organizations | users | related)\n (ID_OR_NAME ... | [-I JSONL_INPUT] [-s START] [-m MAX])\n [-p PROCESSES] [-l LOG_FILE] [-qwz]\n [[-c CONFIG] [-u USER] | -r SITE_URL [-a APIKEY] [--insecure]]\n ckanapi dump (datasets | groups | organizations | users | related)\n (ID_OR_NAME ... | --all) ([-O JSONL_OUTPUT] | [-D DIRECTORY])\n [-p PROCESSES] [-dqwzRU --include-private --include-drafts --include-deleted]\n [[-c CONFIG] [-u USER] | -r SITE_URL [-a APIKEY] [-g] [--insecure]]\n ckanapi load datasets\n [--upload-resources] [-I JSONL_INPUT] [-s START] [-m MAX]\n [-p PROCESSES] [-l LOG_FILE] [-n | -o] [-qwz]\n [[-c CONFIG] [-u USER] | -r SITE_URL [-a APIKEY] [--insecure]]\n ckanapi load (groups | organizations)\n [--upload-logo] [-I JSONL_INPUT] [-s START] [-m MAX]\n [-p PROCESSES] [-l LOG_FILE] [-n | -o] [-qwzU]\n [[-c CONFIG] [-u USER] | -r SITE_URL [-a APIKEY] [--insecure]]\n ckanapi load (users | related)\n [-I JSONL_INPUT] [-s START] [-m MAX] [-p PROCESSES] [-l LOG_FILE]\n [-n | -o] [-qwz]\n [[-c CONFIG] [-u USER] | -r SITE_URL [-a APIKEY] [--insecure]]\n ckanapi search datasets\n [(KEY=STRING | KEY:JSON ) ... | -i | -I JSON_INPUT]\n [-O JSONL_OUTPUT] [-z]\n [[-c CONFIG] [-u USER] | -r SITE_URL [-a APIKEY] [-g] [--insecure]]\n ckanapi (-h | --help)\n ckanapi --version\n\nOptions:\n -h --help show this screen\n --version show version\n -a --apikey=APIKEY API key to use for remote actions\n --all all the things\n -c --config=CONFIG CKAN configuration file for local actions,\n defaults to $CKAN_INI or development.ini\n -d --datastore-fields export datastore field information along with\n resource metadata as datastore_fields lists\n --include-private include private datasets in the dump\n --include-drafts include draft datasets in the dump\n --include-deleted include deleted datasets in the dump\n -D --datapackages=DIR download resources and output as datapackages\n in DIR instead of metadata-only json lines\n -g --get-request use GET instead of POST for API calls\n -i --input-json read json from stdin to send to action\n -I --input=INPUT input json/ json lines from file instead of stdin\n -j --output-json output plain json instead of pretty-printed json\n -J --output-jsonl output list responses as json lines instead of\n pretty-printed json\n --local-files allow batch instructions to reference local files\n for file uploads\n -l --log=LOG_FILE append messages generated to LOG_FILE\n -m --max-records=MAX exit after processing MAX records\n -n --create-only create new records, don't update existing records\n --insecure ignore verifying the SSL certificate for sites\n using https\n -o --update-only update existing records, don't create new records\n -O --output=JSONL_OUTPUT output to json lines file instead of stdout\n -p --processes=PROCESSES set the number of worker processes [default: 1]\n -P --profile=PROFILE run action with cProfile and output to PROFILE\n only local actions (no -r) will show internals\n -q --quiet don't display progress messages\n -r --remote=URL URL of CKAN server for remote actions\n -R --resource-views export resource views information along with\n resource metadata as resource_views lists\n -s --start-record=START start from record number START, where the first\n record is number 1 [default: 1]\n -u --ckan-user=USER perform actions as user with this name, uses the\n site sysadmin user when not specified\n -U --include-users include users of a group/organization\n --upload-logo upload logo image of a group/organization if the\n image is stored in the original server, otherwise\n its image url will be used\n --upload-resources upload resources of a dataset that were uploaded to\n server. Resources originally linked by external\n urls will keep the urls,will not be uploaded\n -w --worker launch worker process - used internally by load,\n dump, delete and batch commands\n -z --gzip read/write gzipped data\n\"\"\"\n\nimport sys\nimport os\nfrom docopt import docopt\nimport subprocess\n\nfrom ckanapi.version import __version__\nfrom ckanapi.remoteckan import RemoteCKAN\nfrom ckanapi.localckan import LocalCKAN\nfrom ckanapi.errors import CLIError\nfrom ckanapi.cli.load import load_things\nfrom ckanapi.cli.dump import dump_things\nfrom ckanapi.cli.delete import delete_things\nfrom ckanapi.cli.action import action\nfrom ckanapi.cli.search import search_datasets\nfrom ckanapi.cli.batch import batch_actions\n\nfrom logging import getLogger\n\n# explicit logger namespace for easy logging handlers\nlog = getLogger('ckan.ckanapi')\n\ndef parse_arguments():\n # docopt is awesome\n return docopt(__doc__, version=__version__)\n\n\ndef main(running_with_ckan_command=False):\n \"\"\"\n ckanapi command line entry point\n \"\"\"\n arguments = parse_arguments()\n\n if not running_with_ckan_command and not arguments['--remote']:\n return _switch_to_ckan_click(arguments)\n\n if arguments['--remote']:\n ckan = RemoteCKAN(arguments['--remote'],\n apikey=arguments['--apikey'],\n user_agent=\"ckanapi-cli/{version} (+{url})\".format(\n version=__version__,\n url='https://github.com/open-data/ckanapi'),\n get_only=arguments['--get-request'],\n )\n else:\n ckan = LocalCKAN(username=arguments['--ckan-user'])\n # log execution of LocalCKAN commands\n from ckan.plugins.toolkit import config, asbool\n if asbool(config.get('ckanapi.log_local')) and len(sys.argv) > 1:\n cmd = ['who', 'am', 'i']\n proc = subprocess.Popen(cmd, stdout=subprocess.PIPE,\n stderr=subprocess.PIPE)\n out, err = proc.communicate()\n if not out or err:\n # fallback to whoami if `who am i` is empty or errored\n cmd = ['whoami']\n proc = subprocess.Popen(cmd, stdout=subprocess.PIPE,\n stderr=subprocess.PIPE)\n out, err = proc.communicate()\n if not out or err:\n # cannot find user\n out = ''\n else:\n # decode and remove line breaks from whoami's\n out = out.decode().replace('\\n', '').replace('\\r', '')\n # split the `who am i`\n out = out.split()[0]\n log.info('OS User %s executed LocalCKAN: ckanapi %s',\n out, ' '.join(sys.argv[1:]))\n\n stdout = getattr(sys.stdout, 'buffer', sys.stdout)\n if arguments['action']:\n try:\n for r in action(ckan, arguments):\n stdout.write(r)\n return\n except CLIError as e:\n sys.stderr.write(e.args[0] + '\\n')\n return 1\n\n things = ['datasets', 'groups', 'organizations', 'users', 'related']\n thing = [x for x in things if arguments[x]]\n if (arguments['load'] or arguments['dump'] or arguments['delete']\n ) and arguments['--processes'] != '1' and os.name == 'nt':\n sys.stderr.write(\n \"multiple worker processes are not supported on windows\\n\")\n arguments['--processes'] = '1'\n\n if arguments['load']:\n return load_things(ckan, thing[0], arguments)\n\n if arguments['dump']:\n return dump_things(ckan, thing[0], arguments)\n\n if arguments['delete']:\n return delete_things(ckan, thing[0], arguments)\n\n if arguments['search']:\n return search_datasets(ckan, arguments)\n\n if arguments['batch']:\n return batch_actions(ckan, arguments)\n\n assert 0, arguments # we shouldn't be here\n\n\ndef _switch_to_ckan_click(arguments):\n \"\"\"\n Local commands must be run through ckan CLI to set up environment\n \"\"\"\n if arguments['--config']:\n # config needs to come before \"api\" for ckan click CLI\n sys.exit(os.execvp(\"ckan\", [\"ckan\", \"-c\", arguments['--config'], \"api\"] + sys.argv[1:]))\n sys.exit(os.execvp(\"ckan\", [\"ckan\", \"api\"] + sys.argv[1:]))\n" }, { "path": "ckanapi/cli/search.py", "content": "\"\"\"\nimplementation of the search datasets cli command\n\"\"\"\n\nimport sys\nimport gzip\nimport json\nfrom os.path import expanduser\n\nfrom ckanapi.cli.utils import compact_json, pretty_json\nfrom ckanapi.errors import CLIError\n\n\nROWS_PER_QUERY = 1000 # match hard limit in some versions of ckan\n\n\ndef search_datasets(ckan, arguments, stdin=None, stdout=None, stderr=None):\n \"\"\"\n call package_search with KEY=STRING, KEY:JSON or JSON args,\n paginate over the results yield the result\n \"\"\"\n if stdin is None:\n stdin = getattr(sys.stdin, 'buffer', sys.stdin)\n if stdout is None:\n stdout = getattr(sys.__stdout__, 'buffer', sys.__stdout__)\n if stderr is None:\n stderr = getattr(sys.stderr, 'buffer', sys.stderr)\n\n requests_kwargs = None\n if arguments['--insecure']:\n requests_kwargs = {'verify': False}\n if arguments['--input-json']:\n action_args = json.loads(stdin.read().decode('utf-8'))\n elif arguments['--input']:\n action_args = {}\n with open(expanduser(arguments['--input'])) as in_f:\n action_args = json.loads(\n in_f.read())\n else:\n action_args = {}\n for kv in arguments['KEY=STRING']:\n if hasattr(kv, 'decode'):\n kv = kv.decode('utf-8')\n skey, p, svalue = kv.partition('=')\n jkey, p, jvalue = kv.partition(':')\n if len(jkey) > len(skey):\n action_args[skey] = svalue\n elif len(skey) > len(jkey):\n try:\n value = json.loads(jvalue)\n except ValueError:\n raise CLIError(\"KEY:JSON argument %r has invalid JSON \"\n \"value %r\" % (jkey, jvalue))\n action_args[jkey] = value\n else:\n raise CLIError(\"argument not in the form KEY=STRING, \"\n \"or KEY:JSON %r\" % kv)\n\n jsonl_output = stdout\n if arguments['--output']:\n jsonl_output = open(arguments['--output'], 'wb')\n if arguments['--gzip']:\n jsonl_output = gzip.GzipFile(fileobj=jsonl_output)\n\n start = int(action_args.get('start', 0))\n while True:\n args = action_args\n if 'rows' not in action_args:\n args = dict(action_args, start=start, rows=ROWS_PER_QUERY)\n result = ckan.call_action(\n 'package_search',\n args,\n requests_kwargs=requests_kwargs\n )\n rows = result['results']\n for r in rows:\n jsonl_output.write(compact_json(r, sort_keys=True) + b'\\n')\n if not rows or 'rows' in action_args:\n break\n\n start += len(rows)\n\n if jsonl_output != stdout:\n jsonl_output.close()\n" }, { "path": "ckanapi/cli/utils.py", "content": "\"\"\"\nuseful bits of code not tied to ckanapi in any way\n\"\"\"\n\nimport time\n\nimport simplejson as json\nfrom contextlib import contextmanager\n\n\ndef completion_stats(window=1):\n \"\"\"\n Generate completions/second reports on each iteration.\n\n window - window size for completion reports\n \"\"\"\n stamps = []\n while True:\n stamps.append(time.time())\n if len(stamps) < window + 1:\n yield '---'\n else:\n yield '%4.2fs' % ((stamps[-1] - stamps[0]) / window)\n stamps = stamps[-window:]\n\n\n@contextmanager\ndef quiet_int_pipe():\n \"\"\"\n let pipe errors and KeyboardIterrupt exceptions cause silent exit\n \"\"\"\n errors = []\n try:\n yield errors\n except KeyboardInterrupt:\n errors.append('interrupt')\n except IOError as e:\n if e.errno != 32:\n raise\n errors.append('pipe')\n\n\ndef compact_json(r, sort_keys=False):\n \"\"\"\n JSON as small as we can make it, with UTF-8\n \"\"\"\n return json.dumps(r, ensure_ascii=False, separators=(',', ':'),\n sort_keys=sort_keys).encode('utf-8')\n\n\ndef pretty_json(r):\n \"\"\"\n legible sorted JSON, with UTF-8\n \"\"\"\n return json.dumps(r, ensure_ascii=False, separators=(',', ': '),\n indent=2, sort_keys=True).encode('utf-8')\n\n" }, { "path": "ckanapi/cli/workers.py", "content": "import select\nimport subprocess\n\ndef worker_pool(popen_arg, num_workers, job_iterable,\n stop_when_jobs_done=True, stop_on_keyboard_interrupt=True,\n popen=None):\n \"\"\"\n Coroutine to manage a pool of workers that accept jobs as single lines\n of input on stdin and produces results as single lines of output.\n\n popen_arg - parameter to pass to subprocess.Popen when creating workers\n num_workers - maximum number of workers to create\n job_iterable - iterable producing (job id, job string) tuples where\n job string should include a single trailing newline\n stop_when_jobs_done - True: generator exits when all jobs are done\n stop_on_keyboard_interrupt - True: generator exits on KeyboardIterrupt\n\n accepted to send(): job iterable or None, when a new job iterable is\n sent it will replace the previous one used for assigning jobs to workers\n\n This generator blocks until there is a result from one of the workers.\n\n yields (currently processing job id list, finished job id, job result)\n tuples as jobs are completed, or (None, None, None) when no jobs remain\n to be completed and stop_when_jobs_done is False.\n\n currently processing job id list will include None if some workers are\n idle. job result will include trailing newline.\n\n when no jobs remain to be completed and stop_when_jobs_done is False a\n new job iterable must be sent to this generator with send().\n \"\"\"\n if popen is None:\n popen = subprocess.Popen\n\n workers = []\n job_ids = []\n worker_fds = {}\n job_iter = iter(job_iterable)\n\n def start_job(worker=None):\n \"\"\"\n assign a job to exiting or newly created worker subprocess.\n\n returns (job_id, worker) or (None, None) when no more jobs\n \"\"\"\n job_id, job_str = next(job_iter, (None, None))\n if job_str is None:\n return None, None\n job_str = job_str.rstrip(b'\\n') + b'\\n'\n if not worker:\n worker = popen(\n popen_arg,\n stdin=subprocess.PIPE,\n stdout=subprocess.PIPE,\n )\n worker.stdin.write(job_str)\n worker.stdin.flush()\n return (job_id, worker)\n\n def assign_jobs():\n \"\"\"\n start as many jobs as possible given maximum/idle workers\n and available jobs\n \"\"\"\n while None in job_ids:\n wnum = job_ids.index(None)\n job_ids[wnum], w = start_job(workers[wnum])\n if w is None:\n return\n\n while len(workers) < num_workers:\n job_id, w = start_job()\n if w is None:\n return\n worker_fds[w.stdout] = len(workers)\n workers.append(w)\n job_ids.append(job_id)\n\n try:\n assign_jobs()\n while True:\n if all(i is None for i in job_ids):\n if stop_when_jobs_done:\n return\n new_jobs = yield (None, None, None)\n # require new jobs to be submitted\n job_iter = iter(new_jobs)\n assign_jobs()\n continue\n\n try:\n readable, _, _ = select.select(worker_fds, [], [])\n except select.error as e:\n if e.args[0] == 10038:\n # XXX: no many-worker support on windows yet\n readable = list(worker_fds)[:1]\n else:\n raise\n except KeyboardInterrupt:\n if stop_on_keyboard_interrupt:\n return\n raise\n\n fd = readable[0]\n wnum = worker_fds[fd]\n w = workers[wnum]\n result = w.stdout.readline()\n finished = job_ids[wnum]\n job_ids[wnum], _ = start_job(w)\n\n new_jobs = yield (job_ids, finished, result)\n if new_jobs:\n job_iter = iter(new_jobs)\n assign_jobs()\n\n finally:\n for w in workers:\n w.stdin.close()\n" }, { "path": "ckanapi/common.py", "content": "\"\"\"\nCode shared by LocalCKAN, RemoteCKAN and TestCKAN\n\"\"\"\n\nimport json\nimport os\n\nfrom ckanapi.errors import (CKANAPIError, NotAuthorized, NotFound,\n ValidationError, SearchQueryError, SearchError, SearchIndexError,\n ServerIncompatibleError)\n\n\nif request_connection_timeout := os.getenv(\"CKANAPI_REQUEST_TIMEOUT\"):\n request_connection_timeout = int(request_connection_timeout)\n request_read_timeout= int(os.getenv(\"CKANAPI_REQUEST_READ_TIMEOUT\", default=request_connection_timeout))\n REQUEST_TIMEOUT = (request_connection_timeout, request_read_timeout)\nelse:\n REQUEST_TIMEOUT = None\n\nclass ActionShortcut(object):\n \"\"\"\n ActionShortcut(foo).bar(baz=2) <=> foo.call_action('bar', {'baz':2})\n\n An instance of this class is used as the .action attribute of\n LocalCKAN and RemoteCKAN instances to provide a short way to call\n actions, e.g::\n\n pkg = demo.action.package_show(id='adur_district_spending')\n\n instead of::\n\n pkg = demo.call_action('package_show', {'id':'adur_district_spending'})\n\n File-like values (objects with a 'read' attribute) are\n sent as file-uploads::\n\n pkg = demo.action.resource_update(package_id='foo', upload=open(..))\n\n becomes::\n\n pkg = demo.call_action('resource_update',\n {'package_id': 'foo'}, files={'upload': open(..)})\n\n \"\"\"\n def __init__(self, ckan):\n self._ckan = ckan\n\n def __getattr__(self, name):\n def action(**kwargs):\n files = {}\n for k, v in kwargs.items():\n if is_file_like(v):\n files[k] = v\n if files:\n nonfiles = dict((k, v) for k, v in kwargs.items()\n if k not in files)\n return self._ckan.call_action(name,\n data_dict=nonfiles,\n files=files)\n return self._ckan.call_action(name, data_dict=kwargs)\n return action\n\n\ndef is_file_like(v):\n \"\"\"\n Return True if this object is file-like or is a tuple in a format\n that the requests library would accept for uploading.\n \"\"\"\n # see http://docs.python-requests.org/en/latest/user/quickstart/#more-complicated-post-requests\n return hasattr(v, 'read') or (\n isinstance(v, tuple) and len(v) >= 2 and hasattr(v[1], 'read'))\n\n\ndef prepare_action(action, data_dict=None, apikey=None, files=None,\n base_url='api/action/'):\n \"\"\"\n Return action_url, data_json, http_headers\n \"\"\"\n if not data_dict:\n data_dict = {}\n headers = {}\n if files:\n # when uploading files all parameters must be strings and\n # no nesting is allowed because request is sent as multipart\n items = data_dict.items()\n data_dict = {}\n for (k, v) in items:\n if v is None:\n continue # assuming missing will work the same as None\n if isinstance(v, (int, float)):\n v = str(v)\n data_dict[k.encode('utf-8')] = v.encode('utf-8')\n else:\n data_dict = json.dumps(data_dict).encode('ascii')\n headers['Content-Type'] = 'application/json'\n if apikey:\n apikey = str(apikey)\n headers['X-CKAN-API-Key'] = apikey\n headers['Authorization'] = apikey\n url = base_url + action\n return url, data_dict, headers\n\n\ndef reverse_apicontroller_action(url, status, response):\n \"\"\"\n Make an API call look like a direct action call by reversing the\n exception -> HTTP response translation that ApiController.action does\n \"\"\"\n try:\n parsed = json.loads(response)\n if parsed.get('success'):\n return parsed['result']\n if hasattr(parsed, 'get'):\n err = parsed.get('error', {})\n else:\n err = {}\n except (AttributeError, ValueError):\n err = {}\n\n if not isinstance(err, dict): # possibly a Socrata API.\n raise ServerIncompatibleError(repr([url, status, response]))\n\n etype = err.get('__type')\n emessage = err.get('message', '')\n if hasattr(emessage, 'split'):\n emessage = emessage.split(': ', 1)[-1]\n if etype == 'Search Query Error':\n # I refuse to eval(emessage), even if it would be more correct\n raise SearchQueryError(emessage)\n elif etype == 'Search Error':\n # I refuse to eval(emessage), even if it would be more correct\n raise SearchError(emessage)\n elif etype == 'Search Index Error':\n raise SearchIndexError(emessage)\n elif etype == 'Validation Error':\n raise ValidationError(err)\n elif etype == 'Not Found Error':\n raise NotFound(emessage)\n elif etype == 'Authorization Error':\n raise NotAuthorized(err)\n\n # don't recognize the error\n raise CKANAPIError(repr([url, status, response]))\n" }, { "path": "ckanapi/datapackage.py", "content": "import os\nimport requests\nimport json\n\nimport slugify\n\nfrom ckanapi.common import REQUEST_TIMEOUT\nfrom ckanapi.cli.utils import pretty_json\nfrom ckanapi.errors import CKANAPIError, NotFound\n\nDL_CHUNK_SIZE = 100 * 1024\nDATAPACKAGE_TYPES = { # map datastore types to datapackage types\n 'text': 'string',\n 'numeric': 'number',\n 'timestamp': 'datetime',\n}\n\n\ndef create_resource(resource, filename, datapackage_dir, stderr, apikey):\n '''Downloads the resource['url'] to disk.\n '''\n path = os.path.join('data', filename)\n headers = {}\n headers['X-CKAN-API-Key'] = apikey\n headers['Authorization'] = apikey\n\n try:\n r = requests.get(resource['url'], headers=headers, stream=True, timeout=REQUEST_TIMEOUT)\n with open(os.path.join(datapackage_dir, path), 'wb') as f:\n for chunk in r.iter_content(chunk_size=DL_CHUNK_SIZE):\n if chunk: # filter out keep-alive new chunks\n f.write(chunk)\n return dict(resource, path=path)\n except requests.ConnectionError:\n stderr.write('URL {url} refused connection. The resource will not be downloaded\\n'.format(url=resource['url']))\n except requests.exceptions.RequestException as e:\n stderr.write(str(e.args[0]) if len(e.args) > 0 else '')\n stderr.write('\\n')\n except Exception as e:\n stderr.write(str(e.args[0]) if len(e.args) > 0 else '')\n return resource\n\n\ndef create_datapackage(record, base_path, stderr, apikey):\n # TODO: how are we going to handle which resources to\n # leave alone? They're very inconsistent in some instances\n # And I can't imagine anyone wants to download a copy\n # of, for example, the API base endpoint\n resource_formats_to_ignore = ['API', 'api']\n dataset_name = record.get('name', '')\n\n datapackage_dir = os.path.join(base_path, dataset_name)\n os.makedirs(os.path.join(datapackage_dir, 'data'))\n\n # filter out some resources\n ckan_resources = []\n for resource in record.get('resources', []):\n if resource['format'] in resource_formats_to_ignore:\n continue\n ckan_resources.append(resource)\n dataset = dict(record, resources=ckan_resources)\n\n # get the datapackage (metadata)\n datapackage = dataset_to_datapackage(dataset)\n\n for cres, dres in zip(ckan_resources, datapackage.get('resources', [])):\n filename = resource_filename(dres)\n\n # download the resource\n cres = \\\n create_resource(resource, filename, datapackage_dir, stderr, apikey)\n dres['path'] = 'data/' + filename\n\n populate_schema_from_datastore(cres, dres)\n\n json_path = os.path.join(datapackage_dir, 'datapackage.json')\n with open(json_path, 'wb') as out:\n out.write(pretty_json(datapackage))\n\n return datapackage_dir, datapackage, json_path\n\n\ndef resource_filename(dres):\n # prefer resource names from datapackage metadata, because those have been\n # made unique\n name = dres['name']\n ext = slugify.slugify(dres['format'])\n if name.endswith(ext):\n name = name[:-len(ext)]\n return name + '.' + ext\n\n\ndef populate_schema_from_datastore(cres, dres):\n \"\"\"\n populate the data schema in a datapackage resource, from the Datastore.\n This info must already be added to the cres using\n 'populate_datastore_res_fields'\n\n :param cres: CKAN resource dict\n :param dres: datapackage.json style resource dict, for the same resource as\n the cres\n \"\"\"\n # convert datastore data dictionary to datapackage schema\n if 'schema' not in dres and 'datastore_fields' in cres:\n fields = []\n for f in cres['datastore_fields']:\n if f['id'] == '_id':\n continue\n df = {'name': f['id']}\n dtyp = DATAPACKAGE_TYPES.get(f['type'])\n if dtyp:\n df['type'] = dtyp\n dtit = f.get('info', {}).get('label', '')\n if dtit:\n df['title'] = dtit\n ddesc = f.get('info', {}).get('notes', '')\n if ddesc:\n df['description'] = ddesc\n fields.append(df)\n dres['schema'] = {'fields': fields}\n\ndef populate_datastore_res_fields(ckan, res):\n \"\"\"\n update resource dict in-place with datastore_fields values\n in every resource with datastore active using ckan\n LocalCKAN/RemoteCKAN instance\n \"\"\"\n if not res.get('datastore_active', False):\n return\n try:\n ds = ckan.call_action('datastore_search', {\n 'resource_id': res['id'],\n 'limit':0})\n except CKANAPIError:\n return\n except NotFound:\n return # with localckan we'll get the real CKAN exception not a CKANAPIError subclass\n res['datastore_fields'] = ds['fields']\n\n\n# functions below are from https://github.com/frictionlessdata/ckan-datapackage-tools\n# commit c87e07d0d0\n# we can't import and use until dependency issue is resolved:\n# https://github.com/frictionlessdata/ckan-datapackage-tools/issues/11\n\ndef _convert_to_datapackage_resource(resource_dict):\n '''Convert a CKAN resource dict into a Data Package resource dict.\n\n from https://github.com/frictionlessdata/ckan-datapackage-tools\n '''\n resource = {}\n\n if resource_dict.get('url'):\n resource['path'] = resource_dict['url']\n # TODO: DataStore only resources?\n\n if resource_dict.get('description'):\n resource['description'] = resource_dict['description']\n\n if resource_dict.get('format'):\n resource['format'] = resource_dict['format']\n\n if resource_dict.get('hash'):\n resource['hash'] = resource_dict['hash']\n\n if resource_dict.get('name'):\n resource['name'] = slugify.slugify(resource_dict['name']).lower()\n resource['title'] = resource_dict['name']\n else:\n resource['name'] = resource_dict['id']\n\n schema = resource_dict.get('schema')\n if isinstance(schema, str):\n try:\n resource['schema'] = json.loads(schema)\n except ValueError:\n # Assume it's a path or URL\n resource['schema'] = schema\n elif isinstance(schema, dict):\n resource['schema'] = schema\n\n return resource\n\n\ndef dataset_to_datapackage(dataset_dict):\n '''Convert the given CKAN dataset dict into a Data Package dict.\n\n :returns: the datapackage dict\n :rtype: dict\n\n '''\n PARSERS = [\n _rename_dict_key('title', 'title'),\n _rename_dict_key('version', 'version'),\n _parse_ckan_url,\n _parse_notes,\n _parse_license,\n _parse_author_and_source,\n _parse_maintainer,\n _parse_tags,\n _parse_extras,\n ]\n dp = {\n 'name': dataset_dict['name']\n }\n\n for parser in PARSERS:\n dp.update(parser(dataset_dict))\n\n resources = dataset_dict.get('resources')\n if resources:\n dp['resources'] = [_convert_to_datapackage_resource(r)\n for r in resources]\n\n # Ensure unique resource names\n names = {}\n for resource in dp.get('resources', []):\n if resource['name'] in names.keys():\n old_resource_name = resource['name']\n resource['name'] = resource['name'] + str(names[old_resource_name])\n names[old_resource_name] += 1\n else:\n names[resource['name']] = 0\n\n return dp\n\n\ndef _rename_dict_key(original_key, destination_key):\n def _parser(the_dict):\n result = {}\n\n if the_dict.get(original_key):\n result[destination_key] = the_dict[original_key]\n\n return result\n return _parser\n\n\ndef _parse_ckan_url(dataset_dict):\n result = {}\n\n if dataset_dict.get('ckan_url'):\n result['homepage'] = dataset_dict['ckan_url']\n\n return result\n\n\ndef _parse_notes(dataset_dict):\n result = {}\n\n if dataset_dict.get('notes'):\n result['description'] = dataset_dict['notes']\n\n return result\n\n\ndef _parse_license(dataset_dict):\n result = {}\n license = {}\n\n if dataset_dict.get('license_id'):\n license['type'] = dataset_dict['license_id']\n if dataset_dict.get('license_title'):\n license['title'] = dataset_dict['license_title']\n if dataset_dict.get('license_url'):\n license['url'] = dataset_dict['license_url']\n\n if license:\n result['license'] = license\n\n return result\n\n\ndef _parse_author_and_source(dataset_dict):\n result = {}\n source = {}\n\n if dataset_dict.get('author'):\n source['name'] = dataset_dict['author']\n if dataset_dict.get('author_email'):\n source['email'] = dataset_dict['author_email']\n if dataset_dict.get('url'):\n source['web'] = dataset_dict['url']\n\n if source:\n result['sources'] = [source]\n\n return result\n\n\ndef _parse_maintainer(dataset_dict):\n result = {}\n author = {}\n\n if dataset_dict.get('maintainer'):\n author['name'] = dataset_dict['maintainer']\n if dataset_dict.get('maintainer_email'):\n author['email'] = dataset_dict['maintainer_email']\n\n if author:\n result['author'] = author\n\n return result\n\n\ndef _parse_tags(dataset_dict):\n result = {}\n\n keywords = [tag['name'] for tag in dataset_dict.get('tags', [])]\n\n if keywords:\n result['keywords'] = keywords\n\n return result\n\n\ndef _parse_extras(dataset_dict):\n result = {}\n\n extras = [[extra['key'], extra['value']] for extra\n in dataset_dict.get('extras', [])]\n\n for extra in extras:\n try:\n extra[1] = json.loads(extra[1])\n except (ValueError, TypeError):\n pass\n\n if extras:\n result['extras'] = dict(extras)\n\n return result\n" }, { "path": "ckanapi/errors.py", "content": "class ServerIncompatibleError(Exception):\n \"\"\"\n The error raised from RemoteCKAN.call_action when the API doesn't behave\n like a CKAN API.\n \"\"\"\n\n\nclass CKANAPIError(Exception):\n \"\"\"\n The error raised from RemoteCKAN.call_action when no other error\n is recognized.\n\n If importing CKAN source fails then new versions of NotAuthorized,\n ValidationError, NotFound, SearchQueryError, SearchError and\n SearchIndexError are created as subclasses of this class so that they\n provide a helpful str() for tracebacks.\n \"\"\"\n\n def __init__(self, extra_msg=None):\n self.extra_msg = extra_msg\n\n def __str__(self):\n return str(self.extra_msg)\n\n\nclass CLIError(Exception):\n pass\n\n\ntry:\n from ckan.logic import (NotAuthorized, NotFound, ValidationError)\n from ckan.lib.search import (SearchQueryError, SearchError,\n SearchIndexError)\n\nexcept ImportError:\n # Implement the minimum to be compatible with existing errors\n # without requiring CKAN\n\n class NotAuthorized(CKANAPIError):\n pass\n\n class ValidationError(CKANAPIError):\n def __init__(self, error_dict):\n self.error_dict = error_dict\n def __str__(self):\n return repr(self.error_dict)\n\n class NotFound(CKANAPIError):\n def __init__(self, extra_msg=None):\n self.extra_msg = extra_msg\n def __str__(self):\n return self.extra_msg\n\n class SearchQueryError(CKANAPIError):\n pass\n\n class SearchError(CKANAPIError):\n pass\n\n class SearchIndexError(CKANAPIError):\n pass\n\n" }, { "path": "ckanapi/localckan.py", "content": "from tempfile import TemporaryFile\n\nfrom ckanapi.errors import CKANAPIError\nfrom ckanapi.common import ActionShortcut\n\nCOPY_CHUNK = 1024*1024\n\nclass LocalCKAN(object):\n \"\"\"\n An interface to calling actions with get_action() for CKAN plugins.\n\n :param username: perform action as this user, defaults to the site user\n and stored as self.username\n :param context: a default context dict to use when calling actions,\n stored as self.context with username added as its 'user'\n value\n \"\"\"\n def __init__(self, username=None, context=None):\n from ckan.logic import get_action\n self._get_action = get_action\n\n if username is None:\n username = self.get_site_username()\n self.username = username\n self.context = dict(context or [], user=self.username)\n self.action = ActionShortcut(self)\n\n def get_site_username(self):\n user = self._get_action('get_site_user')({'ignore_auth': True}, ())\n return user['name']\n\n def call_action(self, action, data_dict=None, context=None, apikey=None,\n files=None, requests_kwargs=None):\n \"\"\"\n :param action: the action name, e.g. 'package_create'\n :param data_dict: the dict to pass to the action, defaults to {}\n :param context: an override for the context to use for this action,\n remember to include a 'user' when necessary\n :param apikey: not supported\n :param files: None or {field-name: file-to-be-sent, ...}\n :param requests_kwargs: ignored for LocalCKAN (requests not used)\n \"\"\"\n # copy dicts because actions may modify the dicts they are passed\n # (CKAN...you so crazy)\n data_dict = dict(data_dict or [])\n context = dict(self.context if context is None else context)\n if apikey:\n # FIXME: allow use of apikey to set a user in context?\n raise CKANAPIError(\"LocalCKAN.call_action does not support \"\n \"use of apikey parameter, use context['user'] instead\")\n\n to_close = []\n try:\n for fieldname in files or []:\n f = files[fieldname]\n if isinstance(f, tuple):\n # requests accepts (filename, file...) tuples\n filename, f = f[:2]\n else:\n filename = f.name\n try:\n f.seek(0)\n except (AttributeError, IOError):\n f = _write_temp_file(f)\n to_close.append(f)\n\n from werkzeug.datastructures import FileStorage\n\n file_storage = FileStorage()\n file_storage.stream = f\n file_storage.filename = filename\n data_dict[fieldname] = file_storage\n\n return self._get_action(action)(context, data_dict)\n finally:\n for f in to_close:\n f.close()\n\n\ndef _write_temp_file(f):\n \"\"\"\n Pull all data from stream f into a temporary file\n\n Caller must close file returned.\n \"\"\"\n out = TemporaryFile()\n while True: # FIXME: check for maximum size?\n chunk = f.read(COPY_CHUNK)\n if not chunk:\n break\n out.write(chunk)\n return out\n" }, { "path": "ckanapi/remoteckan.py", "content": "from urllib.request import Request, urlopen, HTTPError\nfrom urllib.parse import urlparse\n\nfrom ckanapi.errors import CKANAPIError\nfrom ckanapi.common import (ActionShortcut, prepare_action,\n reverse_apicontroller_action, REQUEST_TIMEOUT)\nfrom ckanapi.version import __version__\nimport os\n\n# add your sites to remove parallel limits on ckanapi cli\nMY_SITES = ['localhost', '127.0.0.1', '[::1]']\nCKANAPI_MY_SITES = os.getenv('CKANAPI_MY_SITES')\nif CKANAPI_MY_SITES:\n additional_sites = CKANAPI_MY_SITES.split()\n MY_SITES.extend(additional_sites)\n\n# add your site above instead of changing this\nPARALLEL_LIMIT = os.getenv('CKANAPI_PARALLEL_LIMIT', default = 3)\n\nimport requests\n\n\nclass RemoteCKAN(object):\n \"\"\"\n An interface to the the CKAN API actions on a remote CKAN instance.\n\n :param address: the web address of the CKAN instance, e.g.\n 'http://demo.ckan.org', stored as self.address\n :param apikey: the API key to pass as an 'X-CKAN-API-Key' header\n when actions are called, stored as self.apikey\n :param user_agent: the User-agent to report when making requests\n :param get_only: only use GET requests (default: False)\n :param session: session to use (default: None)\n \"\"\"\n\n base_url = 'api/action/'\n\n def __init__(self, address, apikey=None, user_agent=None, get_only=False, session=None):\n self.address = address\n self.apikey = apikey\n self.get_only = get_only\n self.session = session\n if not user_agent:\n user_agent = \"ckanapi/{version} (+{url})\".format(\n version=__version__,\n url='https://github.com/ckan/ckanapi')\n self.user_agent = user_agent\n self.action = ActionShortcut(self)\n\n net_loc = urlparse(address)\n if ']' in net_loc:\n net_loc = net_loc[:net_loc.index(']') + 1]\n elif ':' in net_loc:\n net_loc = net_loc[:net_loc.index(':')]\n if net_loc not in MY_SITES:\n # add your sites to MY_SITES above instead of removing this\n self.parallel_limit = PARALLEL_LIMIT\n\n def call_action(self, action, data_dict=None, context=None, apikey=None,\n files=None, requests_kwargs=None):\n \"\"\"\n :param action: the action name, e.g. 'package_create'\n :param data_dict: the dict to pass to the action as JSON,\n defaults to {}\n :param context: always set to None for RemoteCKAN\n :param apikey: API key for authentication\n :param files: None or {field-name: file-to-be-sent, ...}\n :param requests_kwargs: kwargs for requests get/post calls\n\n This function parses the response from the server as JSON and\n returns the decoded value. When an error is returned this\n function will convert it back to an exception that matches the\n one the action function itself raised.\n \"\"\"\n if context:\n raise CKANAPIError(\"RemoteCKAN.call_action does not support \"\n \"use of context parameter, use apikey instead\")\n if files and self.get_only:\n raise CKANAPIError(\"RemoteCKAN: files may not be sent when \"\n \"get_only is True\")\n url, data, headers = prepare_action(\n action, data_dict, apikey or self.apikey, files,\n base_url=self.base_url)\n headers['User-Agent'] = self.user_agent\n url = self.address.rstrip('/') + '/' + url\n requests_kwargs = requests_kwargs or {}\n requests_kwargs.setdefault(\"timeout\", REQUEST_TIMEOUT)\n if not self.session:\n self.session = requests.Session()\n if self.get_only:\n status, response = self._request_fn_get(url, data_dict, headers, requests_kwargs)\n else:\n status, response = self._request_fn(url, data, headers, files, requests_kwargs)\n return reverse_apicontroller_action(url, status, response)\n\n def _request_fn(self, url, data, headers, files, requests_kwargs):\n r = self.session.post(url, data=data, headers=headers, files=files,\n allow_redirects=False, **requests_kwargs)\n # allow_redirects=False because: if a post is redirected (e.g. 301 due\n # to a http to https redirect), then the second request is made to the\n # new URL, but *without* the data. This gives a confusing \"No request\n # body data\" error. It is better to just return the 301 to the user, so\n # we disallow redirects.\n return r.status_code, r.text\n\n def _request_fn_get(self, url, data_dict, headers, requests_kwargs):\n r = self.session.get(url, params=data_dict, headers=headers,\n **requests_kwargs)\n return r.status_code, r.text\n\n def close(self):\n \"\"\"Close session\"\"\"\n if self.session:\n self.session.close()\n self.session = None\n\n def __enter__(self):\n return self\n\n def __exit__(self, *args):\n self.close()\n" }, { "path": "ckanapi/testappckan.py", "content": "import os.path\n\nfrom ckanapi.errors import CKANAPIError\nfrom ckanapi.common import (ActionShortcut, prepare_action,\n reverse_apicontroller_action)\n\nclass TestAppCKAN(object):\n \"\"\"\n An interface to the the CKAN API actions on a paste TestApp\n\n :param test_app: the paste.fixture.TestApp instance, stored as\n self.test_app\n :param apikey: the API key to pass as an 'X-CKAN-API-Key' header\n when actions are called, stored as self.apikey\n \"\"\"\n def __init__(self, test_app, apikey=None):\n self.test_app = test_app\n self.apikey = apikey\n self.action = ActionShortcut(self)\n\n def call_action(self, action, data_dict=None, context=None, apikey=None,\n files=None):\n \"\"\"\n :param action: the action name, e.g. 'package_create'\n :param data_dict: the dict to pass to the action as JSON,\n defaults to {}\n :param context: not supported\n :param files: None or {field-name: file-to-be-sent, ...}\n\n This function parses the response from the server as JSON and\n returns the decoded value. When an error is returned this\n function will convert it back to an exception that matches the\n one the action function itself raised.\n \"\"\"\n if context:\n raise CKANAPIError(\"TestAppCKAN.call_action does not support \"\n \"use of context parameter, use apikey instead\")\n url, data, headers = prepare_action(action, data_dict,\n apikey or self.apikey, files)\n\n kwargs = {}\n if files:\n # Convert the list of (fieldname, file_object) tuples into the\n # (fieldname, filename, file_contents) tuples that webtests needs.\n upload_files = []\n for fieldname, file_ in files.items():\n if hasattr(file_, 'name'):\n filename = os.path.split(file_.name)[1]\n else:\n filename = fieldname\n upload_files.append( (fieldname, filename, file_.read()) )\n kwargs['upload_files'] = upload_files\n\n r = self.test_app.post('/' + url, params=data, headers=headers,\n expect_errors=True, **kwargs)\n return reverse_apicontroller_action(url, r.status, r.body)\n" }, { "path": "ckanapi/tests/__init__.py", "content": "" }, { "path": "ckanapi/tests/mock/mock_ckan.py", "content": "import json\nimport csv\nfrom io import StringIO\nfrom werkzeug.formparser import parse_form_data\nfrom wsgiref.simple_server import make_server\n\n\ndef mock_ckan(environ, start_response):\n status = '200 OK'\n headers = [\n ('Content-type', 'application/json;charset=utf-8'),\n ]\n if environ['PATH_INFO'] == '/api/action/site_read':\n start_response(status, headers)\n return [json.dumps(True).encode('utf-8')]\n if environ['PATH_INFO'] == '/api/action/organization_list':\n start_response(status, headers)\n return [json.dumps({\n \"help\": \"none\",\n \"success\": True,\n \"result\": [\"aa\", \"bb\", \"cc\"]\n }).encode('utf-8')]\n if environ['PATH_INFO'] == '/api/action/test_echo_user_agent':\n start_response(status, headers)\n return [json.dumps({\n \"help\": \"none\",\n \"success\": True,\n \"result\": environ['HTTP_USER_AGENT']\n }).encode('utf-8')]\n if environ['PATH_INFO'] == '/api/action/test_echo_content_type':\n start_response(status, headers)\n return [json.dumps({\n \"help\": \"none\",\n \"success\": True,\n \"result\": environ['CONTENT_TYPE']\n }).encode('utf-8')]\n if environ['PATH_INFO'] == '/api/action/test_upload':\n _, form, files = parse_form_data(environ)\n upload_data = files['upload'].stream.read().decode('utf-8').splitlines()\n csv_file = StringIO()\n writer = csv.writer(csv_file)\n for line_data in upload_data:\n row_data = line_data.split(',')\n writer.writerow(row_data)\n csv_file.seek(0)\n records = list(csv.reader(csv_file))\n start_response(status, headers)\n return [json.dumps({\n \"help\": \"none\",\n \"success\": True,\n \"result\": {\n 'option': form['option'],\n 'last_row': records[-1],\n },\n }).encode('utf-8')]\n if environ['PATH_INFO'].startswith('/api/action/'):\n start_response(status, headers)\n return [json.dumps({\n \"help\": \"none\",\n \"success\": False,\n \"error\": {'__type': 'Not Found Error'},\n }).encode('utf-8')]\n start_response('404 Not Found', headers)\n return []\n\nhttpd = make_server('localhost', 8901, mock_ckan)\nhttpd.serve_forever()\n\n" }, { "path": "ckanapi/tests/test_call.py", "content": "import ckanapi\nimport unittest\n\n\nclass TestCallAction(unittest.TestCase):\n def test_local_fail(self):\n try:\n import ckan\n except ImportError:\n raise unittest.SkipTest('ckan not importable')\n self.assertRaises(\n ckanapi.CKANAPIError,\n ckanapi.LocalCKAN('fake').call_action,\n 'fake', {}, {}, 'apikey not allowed')\n\n def test_remote_fail(self):\n self.assertRaises(\n ckanapi.CKANAPIError,\n ckanapi.RemoteCKAN('fake').call_action,\n 'fake', {}, 'context not allowed')\n\n def test_test_fail(self):\n self.assertRaises(\n ckanapi.CKANAPIError,\n ckanapi.TestAppCKAN('fake').call_action,\n 'fake', {}, 'context not allowed')\n" }, { "path": "ckanapi/tests/test_cli_action.py", "content": "from ckanapi.cli.action import action\nfrom ckanapi.errors import CLIError\nimport unittest\n\nfrom io import BytesIO\n\n\nclass MockCKAN(object):\n def __init__(self, expected_name, expected_args, response,\n expected_files=None):\n self._expected_name = expected_name\n self._expected_args = expected_args\n self._expected_files = expected_files or {}\n self._response = response\n\n def call_action(self, name, args, context=None, apikey=None, files=None,\n requests_kwargs=None):\n if name != self._expected_name:\n return [\"wrong name\", name, self._expected_name]\n if args != self._expected_args:\n return [\"wrong args\", args, self._expected_args]\n files = dict((f, v.name) for f,v in files.items())\n if files != self._expected_files:\n return [\"wrong files\", files, self._expected_files]\n return self._response\n\n\nclass TestCLIAction(unittest.TestCase):\n def test_pretty(self):\n ckan = MockCKAN('shake_it', {'who': 'me'}, {\"oh\": [\"right\", \"on\"]})\n rval = action(ckan, {\n 'ACTION_NAME': 'shake_it',\n 'KEY=STRING': ['who=me'],\n '--output-json': False,\n '--output-jsonl': False,\n '--input-json': False,\n '--input': None,\n '--insecure': False,\n '--profile': None,\n })\n self.assertEqual(b''.join(rval), b\"\"\"\n{\n \"oh\": [\n \"right\",\n \"on\"\n ]\n}\n\"\"\".lstrip())\n\n def test_compact(self):\n ckan = MockCKAN('shake_it', {'who': 'me'}, [\"right\", \"on\"])\n rval = action(ckan, {\n 'ACTION_NAME': 'shake_it',\n 'KEY=STRING': ['who=me'],\n '--output-json': True,\n '--output-jsonl': False,\n '--input-json': False,\n '--input': None,\n '--insecure': False,\n '--profile': None,\n })\n self.assertEqual(b''.join(rval), b'[\"right\",\"on\"]\\n')\n\n def test_compact_fallback(self):\n ckan = MockCKAN('shake_it', {'who': 'me'}, {\"oh\": [\"right\", \"on\"]})\n rval = action(ckan, {\n 'ACTION_NAME': 'shake_it',\n 'KEY=STRING': ['who=me'],\n '--output-json': False,\n '--output-jsonl': True,\n '--input-json': False,\n '--input': None,\n '--insecure': False,\n '--profile': None,\n })\n self.assertEqual(b''.join(rval), b'{\"oh\":[\"right\",\"on\"]}\\n')\n\n def test_jsonl(self):\n ckan = MockCKAN('shake_it', {'who': 'me'}, [99,98,97])\n rval = action(ckan, {\n 'ACTION_NAME': 'shake_it',\n 'KEY=STRING': ['who=me'],\n '--output-json': False,\n '--output-jsonl': True,\n '--input-json': False,\n '--input': None,\n '--insecure': False,\n '--profile': None,\n })\n self.assertEqual(b''.join(rval), b'99\\n98\\n97\\n')\n\n def test_stdin_json(self):\n ckan = MockCKAN('shake_it', {'who': ['just', 'me']}, \"yeah\")\n rval = action(ckan, {\n 'ACTION_NAME': 'shake_it',\n 'KEY=STRING': ['who=me'],\n '--output-json': False,\n '--output-jsonl': False,\n '--input-json': True,\n '--input': None,\n '--insecure': False,\n '--profile': None,\n },\n stdin=BytesIO(b'{\"who\":[\"just\",\"me\"]}'),\n )\n self.assertEqual(b''.join(rval), b'\"yeah\"\\n')\n\n def test_key_json(self):\n ckan = MockCKAN('shake_it', {'who': ['just', 'me']}, \"yeah\")\n rval = action(ckan, {\n 'ACTION_NAME': 'shake_it',\n 'KEY=STRING': ['who:[\"just\", \"me\"]'],\n '--output-json': False,\n '--output-jsonl': False,\n '--input-json': False,\n '--input': None,\n '--insecure': False,\n '--profile': None,\n })\n self.assertEqual(b''.join(rval), b'\"yeah\"\\n')\n\n def test_bad_arg(self):\n ckan = MockCKAN('shake_it', {'who': 'me'}, \"yeah\")\n rval = action(ckan, {\n 'ACTION_NAME': 'shake_it',\n 'KEY=STRING': ['who'],\n '--output-json': False,\n '--output-jsonl': False,\n '--input-json': False,\n '--input': None,\n '--insecure': False,\n '--profile': None,\n })\n self.assertRaises(CLIError, list, rval)\n\n def test_bad_key_json(self):\n ckan = MockCKAN('shake_it', {'who': 'me'}, \"yeah\")\n rval = action(ckan, {\n 'ACTION_NAME': 'shake_it',\n 'KEY=STRING': ['who:me'],\n '--output-json': False,\n '--output-jsonl': False,\n '--input-json': False,\n '--input': None,\n '--insecure': False,\n '--profile': None,\n })\n self.assertRaises(CLIError, list, rval)\n\n def test_key_string_or_json(self):\n ckan = MockCKAN('shake_it', {'who': 'me=you'}, \"yeah\")\n rval = action(ckan, {\n 'ACTION_NAME': 'shake_it',\n 'KEY=STRING': ['who:\"me=you\"'],\n '--output-json': False,\n '--output-jsonl': False,\n '--input-json': False,\n '--input': None,\n '--insecure': False,\n '--profile': None,\n })\n self.assertEqual(b''.join(rval), b'\"yeah\"\\n')\n\n def test_key_json_or_string(self):\n ckan = MockCKAN('shake_it', {'who': 'me:you'}, \"yeah\")\n rval = action(ckan, {\n 'ACTION_NAME': 'shake_it',\n 'KEY=STRING': ['who=me:you'],\n '--output-json': False,\n '--output-jsonl': False,\n '--input-json': False,\n '--input': None,\n '--insecure': False,\n '--profile': None,\n })\n self.assertEqual(b''.join(rval), b'\"yeah\"\\n')\n" }, { "path": "ckanapi/tests/test_cli_dump.py", "content": "from ckanapi.cli.dump import dump_things, dump_things_worker\nfrom ckanapi.errors import NotFound\nimport json\nimport tempfile\nimport shutil\nfrom os.path import exists\n\nimport unittest\nfrom io import BytesIO\n\n\nclass MockCKAN(object):\n def call_action(self, name, data_dict, requests_kwargs=None):\n try:\n return {\n 'package_list': {\n None: ['12', '34', 'dp']},\n 'package_show': {\n '12': {\n 'id': '12',\n 'name': 'twelve',\n 'title': \"Twelve\"},\n '34': {\n 'id': '34',\n 'name': 'thirtyfour',\n 'title': \"Thirty-four\"},\n 'dp': {\n 'id': 'dp',\n 'name': 'dp',\n 'title': 'Test for datapackage',\n 'resources': [\n {'name': 'resource1',\n 'id': 'd902fafc-5717-4dd0-87f2-7a6fc96989b7',\n 'format': 'csv',\n 'datastore_active': True,\n 'url': 'https://google.com'}]}},\n 'group_show': {\n 'ab': {'title': \"ABBA\"}},\n 'organization_show': {\n 'cd': {'title': \"Super Trouper\"}},\n 'datastore_search': {\n 'd902fafc-5717-4dd0-87f2-7a6fc96989b7':\n {'fields': [{\n 'id': 'col1',\n 'type': 'text',\n 'info': {\n 'label': 'Column One',\n 'notes': 'Description One',\n }}]}},\n 'resource_view_list': {\n 'd902fafc-5717-4dd0-87f2-7a6fc96989b7': [{\n 'description': 'Test view',\n 'filterable': True,\n 'id': 'd902fafc-5717-4dd0-87f2-7a6fc96989d9',\n 'package_id': 'dp',\n 'resource_id': 'd902fafc-5717-4dd0-87f2-7a6fc96989b7',\n 'responsive': True,\n 'show_fields': ['_id']}]},\n }[name][data_dict.get('id') or data_dict.get('resource_id')]\n except KeyError:\n raise NotFound()\n\n\nclass TestCLIDump(unittest.TestCase):\n def setUp(self):\n self.ckan = MockCKAN()\n self.stdout = BytesIO()\n self.stderr = BytesIO()\n\n def test_worker_one(self):\n rval = dump_things_worker(self.ckan, 'datasets',\n {'--datastore-fields': False,\n '--resource-views': False,\n '--insecure': False},\n stdin=BytesIO(b'\"34\"\\n'), stdout=self.stdout)\n response = self.stdout.getvalue()\n self.assertEqual(response[-1:], b'\\n')\n timstamp, error, data = json.loads(response.decode('UTF-8'))\n self.assertEqual(error, None)\n self.assertEqual(data[\"title\"], \"Thirty-four\")\n\n def test_worker_two(self):\n rval = dump_things_worker(self.ckan, 'datasets',\n {'--datastore-fields': False,\n '--resource-views': False,\n '--insecure': False},\n stdin=BytesIO(b'\"12\"\\n\"34\"\\n'), stdout=self.stdout)\n response = self.stdout.getvalue()\n self.assertEqual(response.count(b'\\n'), 2, response)\n self.assertEqual(response[-1:], b'\\n')\n r1, r2 = response.split(b'\\n', 1)\n timstamp, error, data = json.loads(r1.decode('UTF-8'))\n self.assertEqual(error, None)\n self.assertEqual(data[\"title\"], \"Twelve\")\n timstamp, error, data = json.loads(r2.decode('UTF-8'))\n self.assertEqual(error, None)\n self.assertEqual(data[\"title\"], \"Thirty-four\")\n\n def test_worker_error(self):\n dump_things_worker(self.ckan, 'datasets',\n {'--insecure': False},\n stdin=BytesIO(b'\"99\"\\n'), stdout=self.stdout)\n response = self.stdout.getvalue()\n self.assertEqual(response[-1:], b'\\n')\n timstamp, error, data = json.loads(response.decode('UTF-8'))\n self.assertEqual(error, \"NotFound\")\n self.assertEqual(data, None)\n\n def test_worker_group(self):\n dump_things_worker(self.ckan, 'groups',\n {'--insecure': False},\n stdin=BytesIO(b'\"ab\"\\n'), stdout=self.stdout)\n response = self.stdout.getvalue()\n self.assertEqual(response[-1:], b'\\n')\n timstamp, error, data = json.loads(response.decode('UTF-8'))\n self.assertEqual(error, None)\n self.assertEqual(data, {\"title\":\"ABBA\"})\n\n def test_worker_organization(self):\n dump_things_worker(self.ckan, 'organizations',\n {'--insecure': False},\n stdin=BytesIO(b'\"cd\"\\n'), stdout=self.stdout)\n response = self.stdout.getvalue()\n self.assertEqual(response[-1:], b'\\n')\n timstamp, error, data = json.loads(response.decode('UTF-8'))\n self.assertEqual(error, None)\n self.assertEqual(data, {\"title\":\"Super Trouper\"})\n\n def test_parent_dump_all(self):\n dump_things(self.ckan, 'datasets', {\n '--quiet': False,\n '--ckan-user': None,\n '--config': None,\n '--remote': None,\n '--apikey': None,\n '--worker': False,\n '--log': None,\n '--output': None,\n '--datapackages': None,\n '--gzip': False,\n '--all': True,\n '--processes': '1',\n '--get-request': False,\n '--datastore-fields': False,\n '--resource-views': False,\n '--insecure': False,\n '--include-users': False,\n },\n worker_pool=self._mock_worker_pool,\n stdout=self.stdout,\n stderr=self.stderr)\n self.assertEqual(self.worker_cmd, [\n 'ckanapi', 'dump', 'datasets', '--worker',\n 'value-here-to-make-docopt-happy'])\n self.assertEqual(self.worker_processes, 1)\n self.assertEqual(self.worker_jobs,\n [(0, b'\"12\"\\n'), (1, b'\"34\"\\n'), (2, b'\"dp\"\\n')])\n\n def test_parent_parallel_limit(self):\n self.ckan.parallel_limit = 2\n dump_things(self.ckan, 'datasets', {\n '--quiet': False,\n '--ckan-user': None,\n '--config': None,\n '--remote': None,\n '--apikey': None,\n '--worker': False,\n '--log': None,\n '--output': None,\n '--datapackages': None,\n '--gzip': False,\n '--all': False,\n 'ID_OR_NAME': ['12'],\n '--processes': '5',\n '--get-request': False,\n '--datastore-fields': False,\n '--resource-views': False,\n '--insecure': False,\n '--include-users': False,\n },\n worker_pool=self._mock_worker_pool,\n stdout=self.stdout,\n stderr=self.stderr)\n self.assertEqual(self.worker_cmd, [\n 'ckanapi', 'dump', 'datasets', '--worker',\n 'value-here-to-make-docopt-happy'])\n self.assertEqual(self.worker_processes, 2)\n\n def test_parent_id_argument(self):\n dump_things(self.ckan, 'groups', {\n '--quiet': False,\n '--ckan-user': None,\n '--config': None,\n '--remote': None,\n '--apikey': None,\n '--worker': False,\n '--log': None,\n '--output': None,\n '--datapackages': None,\n '--gzip': False,\n '--all': False,\n 'ID_OR_NAME': ['ab'],\n '--processes': '1',\n '--get-request': False,\n '--datastore-fields': False,\n '--resource-views': False,\n '--insecure': False,\n '--include-users': False,\n },\n\n worker_pool=self._mock_worker_pool,\n stdout=self.stdout,\n stderr=self.stderr)\n self.assertEqual(self.worker_cmd, [\n 'ckanapi', 'dump', 'groups', '--worker',\n 'value-here-to-make-docopt-happy'])\n self.assertEqual(self.worker_processes, 1)\n self.assertEqual(self.worker_jobs, [(0, b'\"ab\"\\n')])\n\n def test_parent_maintain_order(self):\n dump_things(self.ckan, 'organizations', {\n '--quiet': False,\n '--ckan-user': None,\n '--config': None,\n '--remote': None,\n '--apikey': None,\n '--worker': False,\n '--log': None,\n '--output': None,\n '--datapackages': None,\n '--gzip': False,\n '--all': False,\n 'ID_OR_NAME': ['P', 'Q', 'R', 'S'],\n '--processes': '1',\n '--get-request': False,\n '--datastore-fields': False,\n '--resource-views': False,\n '--insecure': False,\n '--include-users': False,\n },\n worker_pool=self._mock_worker_pool_reversed,\n stdout=self.stdout,\n stderr=self.stderr)\n self.assertEqual(self.worker_cmd, [\n 'ckanapi', 'dump', 'organizations', '--worker',\n 'value-here-to-make-docopt-happy'])\n self.assertEqual(self.worker_processes, 1)\n self.assertEqual(self.stdout.getvalue(),\n b'{\"id\":\"P\"}\\n'\n b'{\"id\":\"Q\"}\\n'\n b'{\"id\":\"R\"}\\n'\n b'{\"id\":\"S\"}\\n')\n\n def test_parent_datapackages(self):\n target = tempfile.mkdtemp()\n try:\n dump_things(self.ckan, 'datasets', {\n '--quiet': False,\n '--ckan-user': None,\n '--config': None,\n '--remote': None,\n '--apikey': None,\n '--worker': False,\n '--log': None,\n '--output': None,\n '--datapackages': target,\n '--gzip': False,\n '--all': True,\n '--processes': '1',\n '--get-request': False,\n '--datastore-fields': False,\n '--resource-views': False,\n '--insecure': False,\n '--include-users': False,\n },\n worker_pool=self._worker_pool_with_data,\n stdout=self.stdout,\n stderr=self.stderr)\n assert exists(target + '/twelve/datapackage.json')\n assert exists(target + '/thirtyfour/datapackage.json')\n assert exists(target + '/dp/datapackage.json')\n assert exists(target + '/dp/data/resource1.csv')\n with open(target + '/dp/datapackage.json') as dpf:\n dp = json.load(dpf)\n self.assertEqual(dp, {\n 'name': 'dp',\n 'title': 'Test for datapackage',\n 'resources': [{\n 'name': 'resource1',\n 'format': 'csv',\n 'path': 'data/resource1.csv',\n 'title': 'resource1',\n 'schema': {\n 'fields': [{\n 'name': 'col1',\n 'title': 'Column One',\n 'description': 'Description One',\n 'type': 'string',\n }],\n }\n }]\n })\n finally:\n shutil.rmtree(target)\n\n\n def test_resource_views(self):\n target = tempfile.mkdtemp()\n try:\n dump_things(self.ckan, 'datasets', {\n 'ID_OR_NAME': ['dp'],\n '--quiet': False,\n '--ckan-user': None,\n '--config': None,\n '--remote': None,\n '--apikey': None,\n '--worker': False,\n '--log': None,\n '--output': target + '/dpf.jsonl',\n '--datapackages': None,\n '--gzip': False,\n '--all': False,\n '--processes': '1',\n '--get-request': False,\n '--datastore-fields': False,\n '--resource-views': True,\n '--insecure': False,\n '--include-users': False,\n },\n worker_pool=self._worker_pool_with_resource_views,\n stdout=self.stdout,\n stderr=self.stderr)\n assert exists(target + '/dpf.jsonl')\n with open(target + '/dpf.jsonl') as dpf:\n dp = json.load(dpf)\n self.assertEqual(dp, {\n 'id': 'dp',\n 'name': 'dp',\n 'title': 'Test for datapackage',\n 'resources': [{\n 'name': 'resource1',\n 'format': 'csv',\n 'id': 'd902fafc-5717-4dd0-87f2-7a6fc96989b7',\n 'url': 'https://google.com',\n 'datastore_active': True,\n 'resource_views': [{\n 'description': 'Test view',\n 'filterable': True,\n 'id': 'd902fafc-5717-4dd0-87f2-7a6fc96989d9',\n 'package_id': 'dp',\n 'resource_id': 'd902fafc-5717-4dd0-87f2-7a6fc96989b7',\n 'responsive': True,\n 'show_fields': ['_id']\n }]\n }]\n })\n finally:\n shutil.rmtree(target)\n\n def test_include_params_default(self):\n\n ckan = unittest.mock.MagicMock()\n ckan.parallel_limit = 1\n dump_things(ckan, 'datasets', {\n '--all': True,\n '--quiet': False,\n '--ckan-user': None,\n '--config': None,\n '--remote': None,\n '--apikey': None,\n '--worker': False,\n '--log': None,\n '--output': None,\n '--datapackages': None,\n '--gzip': False,\n '--processes': '1',\n '--get-request': False,\n '--datastore-fields': False,\n '--resource-views': False,\n '--insecure': False,\n '--include-users': False,\n })\n\n action = ckan.method_calls[0].args[0]\n data_dict = ckan.method_calls[0].args[1]\n\n self.assertEqual(action, \"package_list\")\n\n self.assertEqual(data_dict[\"include_private\"], False)\n self.assertEqual(data_dict[\"include_drafts\"], False)\n self.assertEqual(data_dict[\"include_deleted\"], False)\n\n def test_include_params_true(self):\n\n ckan = unittest.mock.MagicMock()\n ckan.parallel_limit = 1\n dump_things(ckan, 'datasets', {\n '--all': True,\n '--quiet': False,\n '--ckan-user': None,\n '--config': None,\n '--remote': None,\n '--apikey': None,\n '--worker': False,\n '--log': None,\n '--output': None,\n '--datapackages': None,\n '--gzip': False,\n '--processes': '1',\n '--get-request': False,\n '--datastore-fields': False,\n '--resource-views': False,\n '--insecure': False,\n '--include-users': False,\n\n '--include-private': True,\n '--include-drafts': True,\n '--include-deleted': True,\n })\n\n action = ckan.method_calls[0].args[0]\n data_dict = ckan.method_calls[0].args[1]\n\n self.assertEqual(action, \"package_list\")\n\n self.assertEqual(data_dict[\"include_private\"], True)\n self.assertEqual(data_dict[\"include_drafts\"], True)\n self.assertEqual(data_dict[\"include_deleted\"], True)\n\n def _mock_worker_pool(self, cmd, processes, job_iter):\n self.worker_cmd = cmd\n self.worker_processes = processes\n self.worker_jobs = list(job_iter)\n for i, j in self.worker_jobs:\n jname = json.loads(j.decode('UTF-8'))\n yield [[], i, json.dumps(['some-date', None, {'id': jname}]\n ).encode('UTF-8') + b'\\n']\n\n def _mock_worker_pool_reversed(self, cmd, processes, job_iter):\n return reversed(list(\n self._mock_worker_pool(cmd, processes, job_iter)))\n\n def _worker_pool_with_data(self, cmd, processes, job_iter):\n worker_stdin = BytesIO(b''.join(v for i, v in job_iter))\n worker_stdout = BytesIO()\n dump_things_worker(self.ckan, 'datasets', {\n '--datastore-fields': True,\n '--resource-views': False,\n '--insecure': False,\n '--include-users': False,},\n stdin=worker_stdin,\n stdout=worker_stdout)\n for i, v in enumerate(worker_stdout.getvalue().strip().split(b'\\n')):\n yield [[], i, v]\n\n\n def _worker_pool_with_resource_views(self, cmd, proccesses, job_iter):\n worker_stdin = BytesIO(b''.join(v for i, v in job_iter))\n worker_stdout = BytesIO()\n dump_things_worker(self.ckan, 'datasets', {\n '--datastore-fields': False,\n '--resource-views': True,\n '--insecure': False,\n '--include-users': False,},\n stdin=worker_stdin,\n stdout=worker_stdout)\n for i, v in enumerate(worker_stdout.getvalue().strip().split(b'\\n')):\n yield [[], i, v]\n" }, { "path": "ckanapi/tests/test_cli_load.py", "content": "from ckanapi.cli.load import load_things, load_things_worker\nfrom ckanapi.errors import NotFound, ValidationError, NotAuthorized\nimport json\n\nimport unittest\nfrom io import BytesIO\n\nclass MockCKAN(object):\n def call_action(self, name, data_dict, requests_kwargs=None):\n if name == 'package_show' and data_dict['id'] == 'seekrit':\n raise NotAuthorized('naughty user')\n if name == 'package_create' and data_dict.get('name') == '34':\n raise ValidationError({'name': 'That URL is already in use.'})\n if name == 'organization_update':\n if data_dict['id'] == 'used' and data_dict.get('users') != [\n 'people']:\n raise ValidationError({'users': 'should be unchanged'})\n if data_dict['id'] == 'unused' and data_dict.get('users') != []:\n raise ValidationError({'users': 'should be cleared'})\n try:\n return {\n 'package_show': {\n '12': {'title': \"Twelve\"},\n '30ish': {'id': '34', 'title': \"Thirty-four\"},\n '34': {'id': '34', 'title': \"Thirty-four\"},\n },\n 'group_show': {\n 'ab': {'title': \"ABBA\"},\n },\n 'organization_show': {\n 'cd': {'id': 'cd', 'title': \"Super Trouper\"},\n 'used': {'users': ['people']},\n 'unused': {'users': ['people']},\n },\n 'package_create': {\n None: {'name': 'something-new'},\n },\n 'package_update': {\n '34': {'name': 'something-updated'},\n },\n 'group_update': {\n 'ab': {'name': 'group-updated'},\n },\n 'organization_update': {\n 'cd': {'name': 'org-updated'},\n 'used': {'name': 'users-unchanged'},\n 'unused': {'name': 'users-cleared'},\n },\n 'organization_create': {\n None: {'name': 'org-created'},\n },\n }[name][data_dict.get('id')]\n except KeyError:\n raise NotFound()\n\n\nclass TestCLILoad(unittest.TestCase):\n def setUp(self):\n self.ckan = MockCKAN()\n self.stdout = BytesIO()\n self.stderr = BytesIO()\n\n def test_create_with_no_resources(self):\n load_things_worker(self.ckan, 'datasets', {\n '--create-only': False,\n '--update-only': False,\n '--upload-resources': False,\n '--insecure': False,\n },\n stdin=BytesIO(b'{\"name\": \"45\",\"title\":\"Forty-five\"}\\n'),\n stdout=self.stdout)\n response = self.stdout.getvalue()\n self.assertEqual(response[-1:], b'\\n')\n timstamp, action, error, data = json.loads(response.decode('UTF-8'))\n self.assertEqual(action, 'create')\n self.assertEqual(error, None)\n self.assertEqual(data, 'something-new')\n\n def test_create_with_corrupted_resources(self):\n load_things_worker(self.ckan, 'datasets', {\n '--create-only': False,\n '--update-only': False,\n '--upload-resources': False,\n '--insecure': False,\n },\n stdin=BytesIO(b'{\"name\": \"45\",\"title\":\"Forty-five\",\"resources\":[{\"id\":\"123\"}]}\\n'),\n stdout=self.stdout)\n response = self.stdout.getvalue()\n self.assertEqual(response[-1:], b'\\n')\n timstamp, action, error, data = json.loads(response.decode('UTF-8'))\n self.assertEqual(action, 'create')\n self.assertEqual(error, None)\n self.assertEqual(data, 'something-new')\n\n def test_create_with_complete_resources(self):\n load_things_worker(self.ckan, 'datasets', {\n '--create-only': False,\n '--update-only': False,\n '--upload-resources': False,\n '--insecure': False,\n },\n stdin=BytesIO(\n b'{\"name\": \"45\",\"title\":\"Forty-five\",'\n b'\"resources\":[{\"id\":\"123\",\"url_type\":\"\",\"url\":\"http://example.com\"}]}\\n'),\n stdout=self.stdout)\n response = self.stdout.getvalue()\n self.assertEqual(response[-1:], b'\\n')\n timstamp, action, error, data = json.loads(response.decode('UTF-8'))\n self.assertEqual(action, 'create')\n self.assertEqual(error, None)\n self.assertEqual(data, 'something-new')\n\n def test_create_only(self):\n load_things_worker(self.ckan, 'datasets', {\n '--create-only': True,\n '--update-only': False,\n '--upload-resources': False,\n '--insecure': False,\n },\n stdin=BytesIO(b'{\"name\": \"45\",\"title\":\"Forty-five\"}\\n'),\n stdout=self.stdout)\n response = self.stdout.getvalue()\n self.assertEqual(response[-1:], b'\\n')\n timstamp, action, error, data = json.loads(response.decode('UTF-8'))\n self.assertEqual(action, 'create')\n self.assertEqual(error, None)\n self.assertEqual(data, 'something-new')\n\n def test_create_empty_dict(self):\n load_things_worker(self.ckan, 'datasets', {\n '--create-only': False,\n '--update-only': False,\n '--upload-resources': False,\n '--insecure': False,\n },\n stdin=BytesIO(b'{}\\n'),\n stdout=self.stdout)\n response = self.stdout.getvalue()\n self.assertEqual(response[-1:], b'\\n')\n timstamp, action, error, data = json.loads(response.decode('UTF-8'))\n self.assertEqual(action, 'create')\n self.assertEqual(error, None)\n self.assertEqual(data, 'something-new')\n\n def test_create_bad_option(self):\n load_things_worker(self.ckan, 'datasets', {\n '--create-only': False,\n '--update-only': True,\n '--insecure': False,\n },\n stdin=BytesIO(b'{\"name\": \"45\",\"title\":\"Forty-five\"}\\n'),\n stdout=self.stdout)\n response = self.stdout.getvalue()\n self.assertEqual(response[-1:], b'\\n')\n timstamp, action, error, data = json.loads(response.decode('UTF-8'))\n self.assertEqual(action, 'show')\n self.assertEqual(error, 'NotFound')\n self.assertEqual(data, [None, '45'])\n\n def test_update_with_no_resources(self):\n load_things_worker(self.ckan, 'datasets', {\n '--create-only': False,\n '--update-only': False,\n '--insecure': False,\n },\n stdin=BytesIO(b'{\"name\": \"30ish\",\"title\":\"3.4 times ten\"}\\n'),\n stdout=self.stdout)\n response = self.stdout.getvalue()\n self.assertEqual(response[-1:], b'\\n')\n timstamp, action, error, data = json.loads(response.decode('UTF-8'))\n self.assertEqual(action, 'update')\n self.assertEqual(error, None)\n self.assertEqual(data, 'something-updated')\n\n def test_update_with_corrupted_resources(self):\n load_things_worker(self.ckan, 'datasets', {\n '--create-only': False,\n '--update-only': False,\n '--upload-resources': False,\n '--insecure': False,\n },\n stdin=BytesIO(b'{\"name\": \"30ish\",\"title\":\"3.4 times ten\",\"resources\":[{\"id\":\"123\"}]}\\n'),\n stdout=self.stdout)\n response = self.stdout.getvalue()\n self.assertEqual(response[-1:], b'\\n')\n timstamp, action, error, data = json.loads(response.decode('UTF-8'))\n self.assertEqual(action, 'update')\n self.assertEqual(error, None)\n self.assertEqual(data, 'something-updated')\n\n def test_update_with_complete_resources(self):\n load_things_worker(self.ckan, 'datasets', {\n '--create-only': False,\n '--update-only': False,\n '--upload-resources': False,\n '--insecure': False,\n },\n stdin=BytesIO(\n b'{\"name\": \"30ish\",\"title\":\"3.4 times ten\",'\n b'\"resources\":[{\"id\":\"123\",\"url_type\":\"\",\"url\":\"http://example.com\"}]}\\n'),\n stdout=self.stdout)\n response = self.stdout.getvalue()\n self.assertEqual(response[-1:], b'\\n')\n timstamp, action, error, data = json.loads(response.decode('UTF-8'))\n self.assertEqual(action, 'update')\n self.assertEqual(error, None)\n self.assertEqual(data, 'something-updated')\n\n def test_update_only(self):\n load_things_worker(self.ckan, 'datasets', {\n '--create-only': False,\n '--update-only': True,\n '--upload-resources': False,\n '--insecure': False,\n },\n stdin=BytesIO(b'{\"name\": \"34\",\"title\":\"3.4 times ten\"}\\n'),\n stdout=self.stdout)\n response = self.stdout.getvalue()\n self.assertEqual(response[-1:], b'\\n')\n timstamp, action, error, data = json.loads(response.decode('UTF-8'))\n self.assertEqual(action, 'update')\n self.assertEqual(error, None)\n self.assertEqual(data, 'something-updated')\n\n def test_update_bad_option(self):\n load_things_worker(self.ckan, 'datasets', {\n '--create-only': True,\n '--update-only': False,\n '--upload-resources': False,\n '--insecure': False,\n },\n stdin=BytesIO(b'{\"name\": \"34\",\"title\":\"3.4 times ten\"}\\n'),\n stdout=self.stdout)\n response = self.stdout.getvalue()\n self.assertEqual(response[-1:], b'\\n')\n timstamp, action, error, data = json.loads(response.decode('UTF-8'))\n self.assertEqual(action, 'create')\n self.assertEqual(error, 'ValidationError')\n self.assertEqual(data, {'name': 'That URL is already in use.'})\n\n def test_update_unauthorized(self):\n load_things_worker(self.ckan, 'datasets', {\n '--create-only': False,\n '--update-only': False,\n '--upload-resources': False,\n '--insecure': False,\n },\n stdin=BytesIO(b'{\"name\": \"seekrit\", \"title\": \"Things\"}\\n'),\n stdout=self.stdout)\n response = self.stdout.getvalue()\n self.assertEqual(response[-1:], b'\\n')\n timstamp, action, error, data = json.loads(response.decode('UTF-8'))\n self.assertEqual(action, 'show')\n self.assertEqual(error, 'NotAuthorized')\n self.assertEqual(data, 'naughty user')\n\n def test_update_group(self):\n load_things_worker(self.ckan, 'groups', {\n '--create-only': False,\n '--update-only': False,\n '--upload-resources': False,\n '--insecure': False,\n },\n stdin=BytesIO(b'{\"id\": \"ab\",\"title\":\"a balloon\"}\\n'),\n stdout=self.stdout)\n response = self.stdout.getvalue()\n self.assertEqual(response[-1:], b'\\n')\n timstamp, action, error, data = json.loads(response.decode('UTF-8'))\n self.assertEqual(action, 'update')\n self.assertEqual(error, None)\n self.assertEqual(data, 'group-updated')\n\n def test_update_organization_two(self):\n load_things_worker(self.ckan, 'organizations', {\n '--create-only': False,\n '--update-only': False,\n '--upload-resources': False,\n '--insecure': False,\n },\n stdin=BytesIO(\n b'{\"name\": \"cd\", \"title\": \"Go\"}\\n'\n b'{\"name\": \"ef\", \"title\": \"Play\"}\\n'),\n stdout=self.stdout)\n response = self.stdout.getvalue()\n self.assertEqual(response.count(b'\\n'), 2, response)\n self.assertEqual(response[-1:], b'\\n')\n r1, r2 = response.split(b'\\n', 1)\n timstamp, action, error, data = json.loads(r1.decode('UTF-8'))\n self.assertEqual(action, 'update')\n self.assertEqual(error, None)\n self.assertEqual(data, 'org-updated')\n timstamp, action, error, data = json.loads(r2.decode('UTF-8'))\n self.assertEqual(action, 'create')\n self.assertEqual(error, None)\n self.assertEqual(data, 'org-created')\n\n def test_update_organization_with_users_unchanged(self):\n load_things_worker(self.ckan, 'organizations', {\n '--create-only': False,\n '--update-only': False,\n '--upload-resources': False,\n '--insecure': False,\n },\n stdin=BytesIO(b'{\"id\": \"used\", \"title\": \"here\"}\\n'),\n stdout=self.stdout)\n response = self.stdout.getvalue()\n self.assertEqual(response[-1:], b'\\n')\n timstamp, action, error, data = json.loads(response.decode('UTF-8'))\n self.assertEqual(action, 'update')\n self.assertEqual(error, None)\n self.assertEqual(data, 'users-unchanged')\n\n def test_update_organization_with_users_cleared(self):\n load_things_worker(self.ckan, 'organizations', {\n '--create-only': False,\n '--update-only': False,\n '--upload-resources': False,\n '--insecure': False,\n },\n stdin=BytesIO(b'{\"id\": \"unused\", \"users\": []}\\n'),\n stdout=self.stdout)\n response = self.stdout.getvalue()\n self.assertEqual(response[-1:], b'\\n')\n timstamp, action, error, data = json.loads(response.decode('UTF-8'))\n self.assertEqual(action, 'update')\n self.assertEqual(error, None)\n self.assertEqual(data, 'users-cleared')\n\n def test_parent_load_two(self):\n load_things(self.ckan, 'datasets', {\n '--quiet': False,\n '--ckan-user': None,\n '--config': None,\n '--remote': None,\n '--apikey': None,\n '--worker': False,\n '--log': None,\n '--gzip': False,\n '--processes': '1',\n '--input': None,\n '--create-only': False,\n '--update-only': False,\n '--start-record': '1',\n '--max-records': None,\n '--upload-resources': False,\n '--upload-logo': False,\n '--insecure': False,\n },\n worker_pool=self._mock_worker_pool,\n stdin=BytesIO(\n b'{\"name\": \"cd\", \"title\": \"Go\"}\\n'\n b'{\"name\": \"ef\", \"title\": \"Play\"}\\n'\n ),\n stdout=self.stdout,\n stderr=self.stderr)\n self.assertEqual(self.worker_cmd, [\n 'ckanapi', 'load', 'datasets', '--worker'])\n self.assertEqual(self.worker_processes, 1)\n self.assertEqual(self.worker_jobs, [\n (1, b'{\"name\": \"cd\", \"title\": \"Go\"}\\n'),\n (2, b'{\"name\": \"ef\", \"title\": \"Play\"}\\n'),\n ])\n\n def test_parent_load_start_max(self):\n load_things(self.ckan, 'groups', {\n '--quiet': False,\n '--ckan-user': None,\n '--config': None,\n '--remote': None,\n '--apikey': None,\n '--worker': False,\n '--log': None,\n '--gzip': False,\n '--processes': '1',\n '--input': None,\n '--create-only': False,\n '--update-only': False,\n '--start-record': '2',\n '--max-records': '2',\n '--upload-resources': False,\n '--upload-logo': False,\n '--insecure': False,\n },\n worker_pool=self._mock_worker_pool,\n stdin=BytesIO(\n b'{\"name\": \"cd\", \"title\": \"Go\"}\\n'\n b'{\"name\": \"ef\", \"title\": \"Play\"}\\n'\n b'{\"name\": \"gh\", \"title\": \"Hotel\"}\\n'\n b'{\"name\": \"ij\", \"title\": \"Ambient\"}\\n'\n ),\n stdout=self.stdout,\n stderr=self.stderr)\n self.assertEqual(self.worker_cmd, [\n 'ckanapi', 'load', 'groups', '--worker'])\n self.assertEqual(self.worker_processes, 1)\n self.assertEqual(self.worker_jobs, [\n (2, b'{\"name\": \"ef\", \"title\": \"Play\"}\\n'),\n (3, b'{\"name\": \"gh\", \"title\": \"Hotel\"}\\n'),\n ])\n\n def test_parent_parallel_limit(self):\n self.ckan.parallel_limit = 2\n load_things(self.ckan, 'datasets', {\n '--quiet': False,\n '--ckan-user': None,\n '--config': None,\n '--remote': None,\n '--apikey': None,\n '--worker': False,\n '--log': None,\n '--gzip': False,\n '--processes': '5',\n '--input': None,\n '--create-only': False,\n '--update-only': False,\n '--start-record': '1',\n '--max-records': None,\n '--upload-resources': False,\n '--upload-logo': False,\n '--insecure': False,\n },\n worker_pool=self._mock_worker_pool,\n stdin=BytesIO(\n b'{\"name\": \"cd\", \"title\": \"Go\"}\\n'\n b'{\"name\": \"ef\", \"title\": \"Play\"}\\n'\n ),\n stdout=self.stdout,\n stderr=self.stderr)\n self.assertEqual(self.worker_cmd, [\n 'ckanapi', 'load', 'datasets', '--worker'])\n self.assertEqual(self.worker_processes, 2)\n\n def _mock_worker_pool(self, cmd, processes, job_iter):\n self.worker_cmd = cmd\n self.worker_processes = processes\n self.worker_jobs = list(job_iter)\n for i, j in self.worker_jobs:\n jname = json.loads(j.decode('UTF-8'))\n yield [[], i, json.dumps(['some-date', None, None, {'id':jname}]\n ).encode('UTF-8') + b'\\n']\n" }, { "path": "ckanapi/tests/test_cli_workers.py", "content": "from ckanapi.cli.workers import worker_pool\nimport os\n\nimport unittest\n\n\nclass _MockPopen(object):\n def __init__(self, popen_args, stdin, stdout):\n read1fd, write1fd = os.pipe()\n read2fd, write2fd = os.pipe()\n self.stdin = os.fdopen(write1fd, 'wb')\n self.stdin_inside = os.fdopen(read1fd, 'rb')\n self.stdout = os.fdopen(read2fd, 'rb')\n self.stdout_inside = os.fdopen(write2fd, 'wb')\n # use popen_args as an after-create callback\n popen_args(self)\n\n def stdout_write(self, data):\n self.stdout_inside.write(data)\n self.stdout_inside.flush()\n\n def stdin_readline(self):\n return self.stdin_inside.readline()\n\n def close_pipes(self):\n for f in (self.stdin, self.stdin_inside, self.stdout, self.stdout_inside):\n f.close()\n\n\nclass TestCLIWorkers(unittest.TestCase):\n def test_one(self):\n children = []\n def child_created(child):\n # need to respond or pool will block test\n child.stdout_write(b'AA\\n')\n children.append(child)\n pool = worker_pool(\n child_created,\n 1,\n enumerate((b\"job1\\n\", b\"job2\\n\")),\n popen=_MockPopen,\n )\n response = next(pool)\n self.assertEqual(len(children), 1)\n c = children[0]\n self.assertEqual(c.stdin_readline(), b'job1\\n')\n self.assertEqual(response, ([1], 0, b'AA\\n'))\n self.assertEqual(c.stdin_readline(), b'job2\\n')\n c.stdout_write(b'BB\\n')\n self.assertEqual(next(pool), ([None], 1, b'BB\\n'))\n self.assertRaises(StopIteration, next, pool)\n for c in children:\n c.close_pipes()\n\n def test_two(self):\n children = []\n def child_created(child):\n # first child responds\n if not children:\n child.stdout_write(b'AA\\n')\n children.append(child)\n pool = worker_pool(\n child_created,\n 2,\n enumerate((b\"job1\\n\", b\"job2\\n\", b\"job3\\n\", b\"job4\\n\")),\n popen=_MockPopen,\n )\n response = next(pool)\n self.assertEqual(len(children), 2)\n c0, c1 = children\n self.assertEqual(c0.stdin_readline(), b'job1\\n')\n self.assertEqual(c1.stdin_readline(), b'job2\\n')\n self.assertEqual(response, ([2, 1], 0, b'AA\\n'))\n self.assertEqual(c0.stdin_readline(), b'job3\\n')\n c1.stdout_write(b'BB\\n')\n self.assertEqual(next(pool), ([2, 3], 1, b'BB\\n'))\n self.assertEqual(c1.stdin_readline(), b'job4\\n')\n c0.stdout_write(b'CC\\n')\n self.assertEqual(next(pool), ([None, 3], 2, b'CC\\n'))\n c1.stdout_write(b'DD\\n')\n self.assertEqual(next(pool), ([None, None], 3, b'DD\\n'))\n self.assertRaises(StopIteration, next, pool)\n for c in children:\n c.close_pipes()\n\n def test_uneven(self):\n children = []\n def child_created(child):\n # second child responds\n if children:\n child.stdout_write(b'AA\\n')\n children.append(child)\n pool = worker_pool(\n child_created,\n 2,\n enumerate((b\"job1\\n\", b\"job2\\n\", b\"job3\\n\", b\"job4\\n\")),\n popen=_MockPopen,\n )\n response = next(pool)\n self.assertEqual(len(children), 2)\n c0, c1 = children\n self.assertEqual(c0.stdin_readline(), b'job1\\n')\n self.assertEqual(c1.stdin_readline(), b'job2\\n')\n self.assertEqual(response, ([0, 2], 1, b'AA\\n'))\n self.assertEqual(c1.stdin_readline(), b'job3\\n')\n c1.stdout_write(b'BB\\n')\n self.assertEqual(next(pool), ([0, 3], 2, b'BB\\n'))\n self.assertEqual(c1.stdin_readline(), b'job4\\n')\n c1.stdout_write(b'CC\\n')\n self.assertEqual(next(pool), ([0, None], 3, b'CC\\n'))\n c0.stdout_write(b'DD\\n')\n self.assertEqual(next(pool), ([None, None], 0, b'DD\\n'))\n self.assertRaises(StopIteration, next, pool)\n for c in children:\n c.close_pipes()\n\n def test_overkill(self):\n children = []\n def child_created(child):\n if not children:\n child.stdout_write(b'AA\\n')\n children.append(child)\n pool = worker_pool(\n child_created,\n 10,\n enumerate((b\"job1\\n\",)),\n popen=_MockPopen,\n )\n response = next(pool)\n self.assertEqual(len(children), 1)\n c = children[0]\n self.assertEqual(c.stdin_readline(), b'job1\\n')\n self.assertEqual(response, ([None], 0, b'AA\\n'))\n self.assertRaises(StopIteration, next, pool)\n for c in children:\n c.close_pipes()\n\n def test_batch(self):\n children = []\n def child_created(child):\n if not children:\n child.stdout_write(b'AA\\n')\n children.append(child)\n pool = worker_pool(\n child_created,\n 2,\n enumerate((b\"job1\\n\",)),\n stop_when_jobs_done=False,\n popen=_MockPopen,\n )\n response = next(pool)\n self.assertEqual(len(children), 1)\n c0 = children[0]\n self.assertEqual(c0.stdin_readline(), b'job1\\n')\n self.assertEqual(response, ([None], 0, b'AA\\n'))\n self.assertEqual(next(pool), (None, None, None))\n # need to write in advance to avoid blocking test\n c0.stdout_write(b'BB\\n')\n response = pool.send(enumerate((b\"job2\\n\", b\"job3\\n\"), 1))\n self.assertEqual(response, ([None, 2], 1, b'BB\\n'))\n for c in children:\n c.close_pipes()\n" }, { "path": "ckanapi/tests/test_datapackage.py", "content": "from ckanapi.datapackage import (\n dataset_to_datapackage, create_resource, create_datapackage,\n resource_filename, populate_schema_from_datastore)\n\nimport unittest\nfrom io import BytesIO\nimport os\nfrom pyfakefs import fake_filesystem_unittest\n\n\nclass TestDatasetToDataPackage(unittest.TestCase):\n def test_simple_dataset(self):\n dataset_dict = {\n u'extras': [{u'key': u'subject', u'value': u'science'}],\n u'name': u'test_dataset_00',\n u'notes': u'Just another test dataset.',\n u'resources': [{\n u'format': u'PNG',\n u'name': u'Image 1',\n u'url': u'http://example.com/image.png',\n }],\n u'tags': [{\n u'display_name': u'science',\n u'id': u'59f9359c-002b-4166-a519-755f89a631da',\n u'name': u'science',\n }],\n u'title': u'Test Dataset',\n u'type': u'dataset',\n }\n\n datapackage = dataset_to_datapackage(dataset_dict)\n\n # code copied from test_package_show_with_full_dataset()\n assert datapackage == {\n u'description': u'Just another test dataset.',\n u'extras': {u'subject': u'science'},\n u'keywords': [u'science'],\n u'name': u'test_dataset_00',\n u'resources': [{u'format': u'PNG',\n u'name': u'image-1',\n u'path': u'http://example.com/image.png',\n u'title': u'Image 1'}],\n u'title': u'Test Dataset'}\n\n def test_full_dataset(self):\n # This sample dataset_dict was generated in CKAN along the lines of\n # ckan/tests/logic/action/test_get.py\n # TestPackageShow.test_package_show_with_full_dataset()\n dataset_dict = {\n u'author': None,\n u'author_email': None,\n u'creator_user_id': u'3267d399-5517-47ef-ac02-13bb29372428',\n u'extras': [{u'key': u'subject', u'value': u'science'}],\n u'groups': [{u'description': u'A test description for this test group.',\n u'display_name': u'Test Group 00',\n u'id': u'cca3543f-0ba0-4194-b2f3-326498eb88b7',\n u'image_display_url': u'',\n u'name': u'test_group_00',\n u'title': u'Test Group 00'}],\n u'id': u'a7165429-dde3-4a5f-ba7d-c690209200cf',\n u'isopen': False,\n u'license_id': None,\n u'license_title': None,\n u'maintainer': None,\n u'maintainer_email': None,\n u'metadata_created': u'2019-05-24T16:30:43.889152',\n u'metadata_modified': u'2019-05-24T16:30:43.889161',\n u'name': u'test_dataset_00',\n u'notes': u'Just another test dataset.',\n u'num_resources': 1,\n u'num_tags': 1,\n u'organization': {\n u'approval_status': u'approved',\n u'created': u'2019-05-24T16:30:43.608032',\n u'description': u'Just another test organization.',\n u'id': u'aa878f8c-1f6e-4e87-b08e-67272d9c3d16',\n u'image_url': u'http://placekitten.com/g/200/100',\n u'is_organization': True,\n u'name': u'test_org_00',\n u'revision_id': u'bb31cfee-aee9-4031-9333-ed922bf3f049',\n u'state': u'active',\n u'title': u'Test Organization',\n u'type': u'organization'},\n u'owner_org': u'aa878f8c-1f6e-4e87-b08e-67272d9c3d16',\n u'private': False,\n u'relationships_as_object': [],\n u'relationships_as_subject': [],\n u'resources': [{\n u'cache_last_updated': None,\n u'cache_url': None,\n u'created': u'2019-05-24T16:30:43.894623',\n u'description': u'',\n u'format': u'PNG',\n u'hash': u'',\n u'id': u'a8e2f627-0450-4728-a0a4-ed3a091c303c',\n u'last_modified': None,\n u'mimetype': None,\n u'mimetype_inner': None,\n u'name': u'Image 1',\n u'package_id': u'a7165429-dde3-4a5f-ba7d-c690209200cf',\n u'position': 0,\n u'resource_type': None,\n u'revision_id': u'990df889-690c-412e-a7ad-f848c9927218',\n u'size': None,\n u'state': u'active',\n u'url': u'http://example.com/image.png',\n u'url_type': None}],\n u'revision_id': u'990df889-690c-412e-a7ad-f848c9927218',\n u'state': u'active',\n u'tags': [{\n u'display_name': u'science',\n u'id': u'59f9359c-002b-4166-a519-755f89a631da',\n u'name': u'science',\n u'state': u'active',\n u'vocabulary_id': None}],\n u'title': u'Test Dataset',\n u'type': u'dataset',\n u'url': None,\n u'version': None\n }\n\n datapackage = dataset_to_datapackage(dataset_dict)\n\n assert datapackage == {\n u'description': u'Just another test dataset.',\n u'extras': {u'subject': u'science'},\n u'keywords': [u'science'],\n u'name': u'test_dataset_00',\n u'resources': [{u'format': u'PNG',\n u'name': u'image-1',\n u'path': u'http://example.com/image.png',\n u'title': u'Image 1'}],\n u'title': u'Test Dataset'}\n\n def test_resource_names_are_unique(self):\n # Somehow these resources got the same name\n dataset_dict = {\n u'name': u'test_dataset_00',\n u'notes': u'Just another test dataset.',\n u'resources': [\n {\n u'format': u'PNG',\n u'name': u'Image',\n u'url': u'http://example.com/imageA.png',\n },\n {\n u'format': u'PNG',\n u'name': u'Image',\n u'url': u'http://example.com/imageB.png',\n },\n {\n u'format': u'PNG',\n u'name': u'Image',\n u'url': u'http://example.com/imageC.png',\n },\n ],\n u'tags': [{\n u'display_name': u'science',\n u'id': u'59f9359c-002b-4166-a519-755f89a631da',\n u'name': u'science',\n }],\n u'title': u'Test Dataset',\n u'type': u'dataset',\n }\n\n datapackage = dataset_to_datapackage(dataset_dict)\n\n assert [res['name'] for res in datapackage['resources']] == \\\n [u'image', u'image0', u'image1']\n\n\nclass TestCreateResource(fake_filesystem_unittest.TestCase):\n def setUp(self):\n self.setUpPyfakefs()\n\n def test_simple(self):\n resource = {\n u'format': u'PNG',\n u'name': u'Image',\n u'url': u'http://example.com/image.png',\n }\n filename = 'image_saved.png'\n os.makedirs('/test/data')\n stderr = BytesIO()\n # TODO mock the HTTP request to example.com\n\n returned_resource = create_resource(\n resource, filename='image_saved.png',\n datapackage_dir='/test', stderr=stderr,\n apikey='')\n\n stderr.seek(0)\n assert not stderr.read()\n assert returned_resource == {\n u'url': u'http://example.com/image.png',\n u'name': u'Image',\n u'format': u'PNG',\n u'path': u'data/image_saved.png',\n }\n\n\nclass TestCreateDataPackage(fake_filesystem_unittest.TestCase):\n def setUp(self):\n self.setUpPyfakefs()\n\n def test_simple(self):\n dataset = {\n u'extras': [{u'key': u'subject', u'value': u'science'}],\n u'name': u'test_dataset_00',\n u'notes': u'Just another test dataset.',\n u'resources': [{\n u'format': u'PNG',\n u'name': u'Image 1',\n u'url': u'http://example.com/image.png',\n }],\n u'tags': [{\n u'display_name': u'science',\n u'id': u'59f9359c-002b-4166-a519-755f89a631da',\n u'name': u'science',\n }],\n u'title': u'Test Dataset',\n u'type': u'dataset',\n }\n stderr = BytesIO()\n os.makedirs('/test/data')\n # TODO mock the HTTP request to example.com\n\n datapackage_dir, datapackage, json_path = \\\n create_datapackage(record=dataset, base_path='/test/',\n stderr=stderr, apikey='')\n\n stderr.seek(0)\n assert not stderr.read()\n assert datapackage_dir == u'/test/test_dataset_00'\n assert datapackage == {\n u'name': u'test_dataset_00',\n u'description': u'Just another test dataset.',\n u'title': u'Test Dataset',\n u'extras': {u'subject': u'science'},\n u'keywords': [u'science'],\n u'resources': [{\n u'path': u'data/image-1.png', # i.e. it was downloaded\n u'title': u'Image 1',\n u'name': u'image-1',\n u'format': u'PNG'}],\n }\n assert json_path == u'/test/test_dataset_00/datapackage.json'\n\n\nclass TestResourceFilename(unittest.TestCase):\n def test_simple(self):\n datapackage_resource = {\n u'title': u'Image 1',\n u'name': u'image-1',\n u'format': u'PNG'\n }\n\n filename = resource_filename(dres=datapackage_resource)\n\n assert filename == u'image-1.png'\n\n\nclass TestPopulateSchemaFromDatastore(unittest.TestCase):\n def test_simple(self):\n ckan_resource = {\n u'format': u'CSV',\n u'name': u'Buildings 1',\n u'url': u'http://example.com/buildings.csv',\n # example datastore fields from:\n # curl 'https://data.boston.gov/api/3/action/datastore_search?resource_id=28ca9f8d-f6ad-4855-bf14-90d2d0bc85ca&limit=0' |jq '.result.fields'\n u'datastore_fields': [\n {\n u'id': u'country',\n u'type': u'int',\n u'info': {\n u'label': u'The country',\n u'notes': u'iso code',\n }\n },\n {\n u'id': u'NUM_FLOORS',\n u'type': u'text',\n u'info': {\n u'type_override': {}\n },\n }\n ]\n }\n datapackage_resource = {\n u'title': u'Buildings 1',\n u'name': u'buildings-1',\n u'format': u'CSV'\n }\n\n populate_schema_from_datastore(cres=ckan_resource,\n dres=datapackage_resource)\n\n assert datapackage_resource == {\n u'title': u'Buildings 1',\n u'name': u'buildings-1',\n u'format': u'CSV',\n u'schema': {'fields': [{'description': u'iso code',\n 'name': u'country',\n 'title': u'The country'},\n {'name': u'NUM_FLOORS',\n 'type': 'string'}]\n }\n }\n" }, { "path": "ckanapi/tests/test_remote.py", "content": "import subprocess\nimport time\nimport os\nimport atexit\nimport socket\nimport requests\nimport json\n\nfrom ckanapi import RemoteCKAN, NotFound\nfrom ckanapi.common import REQUEST_TIMEOUT\nimport unittest\nfrom unittest import mock\nfrom subprocess import DEVNULL\nfrom urllib.request import urlopen, URLError\nfrom io import StringIO\n\nTEST_CKAN = 'http://localhost:8901'\n\nNUMBER_THING_CSV = \"\"\"\nNumber,Thing\n5,sasquach\n\"\"\".lstrip()\n\nclass TestRemoteAction(unittest.TestCase):\n @classmethod\n def setUpClass(cls):\n script = os.path.join(os.path.dirname(__file__), 'mock/mock_ckan.py')\n _mock_ckan = subprocess.Popen(['python', script],\n stdout=DEVNULL, stderr=DEVNULL)\n def kill_child():\n try:\n _mock_ckan.kill()\n _mock_ckan.wait()\n except OSError:\n pass # alread cleaned up from tearDownClass\n atexit.register(kill_child)\n cls._mock_ckan = _mock_ckan\n while True: # wait for the server to start\n try:\n r = urlopen(TEST_CKAN + '/api/action/site_read')\n if r.getcode() == 200:\n break\n except URLError as e:\n pass\n time.sleep(0.1)\n\n def test_good_oldstyle(self):\n ckan = RemoteCKAN(TEST_CKAN)\n self.assertEqual(\n ckan.action.organization_list(),\n ['aa', 'bb', 'cc'])\n ckan.close()\n\n def test_good(self):\n with RemoteCKAN(TEST_CKAN) as ckan:\n self.assertEqual(\n ckan.action.organization_list(),\n ['aa', 'bb', 'cc'])\n\n def test_missing(self):\n with RemoteCKAN(TEST_CKAN) as ckan:\n self.assertRaises(\n NotFound,\n ckan.action.organization_show,\n id='qqq')\n\n def test_default_ua(self):\n with RemoteCKAN(TEST_CKAN) as ckan:\n self.assertTrue(\n ckan.action.test_echo_user_agent().startswith('ckanapi'))\n\n def test_custom_ua(self):\n ua = 'testckanapibot/1.0 (+https://github.com/ckan/ckanapi)'\n with RemoteCKAN(TEST_CKAN, user_agent=ua) as ckan:\n self.assertEqual(ckan.action.test_echo_user_agent(), ua)\n\n def test_default_content_type(self):\n with RemoteCKAN(TEST_CKAN) as ckan:\n self.assertEqual(ckan.action.test_echo_content_type(),\n \"application/json\")\n\n def test_resource_upload(self):\n with RemoteCKAN(TEST_CKAN) as ckan:\n res = ckan.call_action('test_upload',\n {'option': \"42\"},\n files={'upload': StringIO(NUMBER_THING_CSV)})\n self.assertEqual(res.get('last_row'), ['5', 'sasquach'])\n\n def test_resource_upload_extra_param(self):\n with RemoteCKAN(TEST_CKAN) as ckan:\n res = ckan.call_action('test_upload',\n {'option': \"42\"},\n files={'upload': StringIO(NUMBER_THING_CSV)})\n self.assertEqual(res.get('option'), \"42\")\n\n def test_resource_upload_unicode_param(self):\n uname = b't\\xc3\\xab\\xc3\\x9ft resource'.decode('utf-8')\n with RemoteCKAN(TEST_CKAN) as ckan:\n res = ckan.call_action('test_upload',\n {'option': uname},\n files={'upload': StringIO(NUMBER_THING_CSV)})\n self.assertEqual(res.get('option'), uname)\n\n def test_resource_upload_content_type(self):\n with RemoteCKAN(TEST_CKAN) as ckan:\n res = ckan.call_action('test_echo_content_type',\n {'option': \"42\"},\n files={'upload': StringIO(NUMBER_THING_CSV)})\n self.assertEqual(res.split(';')[0], \"multipart/form-data\")\n\n def test_default_timeout(self):\n mock_response = mock.MagicMock()\n mock_response.status_code = 200\n mock_response.text = json.dumps({\"success\": True, \"result\": []})\n\n with mock.patch('requests.Session.post', return_value=mock_response) as mock_post:\n with RemoteCKAN(TEST_CKAN) as ckan:\n ckan.action.organization_list()\n _, kwargs = mock_post.call_args\n self.assertIs(REQUEST_TIMEOUT, None)\n self.assertEqual(kwargs.get('timeout'), REQUEST_TIMEOUT)\n\n def test_custom_timeout(self):\n mock_response = mock.MagicMock()\n mock_response.status_code = 200\n mock_response.text = json.dumps({\"success\": True, \"result\": []})\n\n # We patch at the module level because the env var is read at import time and\n # can't be patched\n with mock.patch(\"ckanapi.remoteckan.REQUEST_TIMEOUT\", (2, 30)):\n with mock.patch('requests.Session.post', return_value=mock_response) as mock_post:\n with RemoteCKAN(TEST_CKAN) as ckan:\n ckan.action.organization_list()\n _, kwargs = mock_post.call_args\n self.assertEqual(kwargs.get('timeout'), (2, 30))\n\n @classmethod\n def tearDownClass(cls):\n cls._mock_ckan.kill()\n cls._mock_ckan.wait()\n" }, { "path": "ckanapi/version.py", "content": "from importlib.metadata import version\n\n__version__ = version(\"ckanapi\")\n" }, { "path": "examples/update_single_field.py", "content": "#!/usr/bin/python3\nfrom ckanapi import RemoteCKAN\nserver_url='https://ckan.my-domain.com'\ntoken = 'very_secret_token'\nselected_id = '0f800659-16d2-449a-923f-a6d04f8edbb9'\nwith RemoteCKAN(server_url, apikey=token) as ckan:\n ckan.action.package_patch(id=selected_id, title='New title')" }, { "path": "pyproject.toml", "content": "[build-system]\nrequires = [\"setuptools>=61.0\"]\nbuild-backend = \"setuptools.build_meta\"\n\n[project]\nname = \"ckanapi\"\nversion = \"4.11\"\ndescription = \"A command line interface and Python module for accessing the CKAN Action API\"\nlicense = {text = \"MIT\"}\nauthors = [\n {name = \"Ian Ward\", email = \"ian@excess.org\"},\n]\nclassifiers = [\n \"Intended Audience :: Developers\",\n \"Development Status :: 5 - Production/Stable\",\n \"License :: OSI Approved :: MIT License\",\n \"Programming Language :: Python :: 3.9\",\n \"Programming Language :: Python :: 3.10\",\n \"Programming Language :: Python :: 3.11\",\n \"Programming Language :: Python :: 3.12\",\n \"Programming Language :: Python :: 3.13\",\n \"Programming Language :: Python :: 3.14\",\n]\nkeywords = [\n \"ckan\",\n \"ckanext\",\n \"API\",\n]\n\nrequires-python = \">=3.9\"\ndependencies = [\n \"setuptools\",\n \"docopt\",\n \"requests\",\n \"python-slugify>=1.0\",\n \"simplejson\",\n]\n\n[project.readme]\nfile = \"README.md\"\ncontent-type = \"text/markdown\"\n\n[project.urls]\nHomepage = \"https://github.com/ckan/ckanapi\"\n\n[project.optional-dependencies]\ntesting = [\n \"pyfakefs==5.10.2\",\n \"werkzeug\",\n]\n\n[project.scripts]\nckanapi = \"ckanapi.cli.main:main\"\n\n[project.entry-points.\"ckan.click_command\"]\napi = \"ckanapi.cli.ckan_click:api\"\n\n[tool.setuptools.packages.find]\ninclude = [\"ckanapi*\"]\n" }, { "path": "requirements.txt", "content": "setuptools\ndocopt\nrequests\nsimplejson\n" } ]