[
  {
    "path": ".gitignore",
    "content": "*.pyc\ntags\ncloudbiolinux.egg-info/\nbuild/\ndoc/build\ndist/\n*.pem\ndpkg.lst\n*.log\nvenv/\nvenv_cbl/\ndeploy/build\ndeploy/keys\ndeploy/settings.yaml\ndeploy/.venv-deploy\ndeploy/.vagrant\n.ropeproject\n.idea"
  },
  {
    "path": ".gitmodules",
    "content": "[submodule \"config/puppet/modules/apache\"]\n\tpath = config/puppet/modules/apache\n\turl = git://github.com/puppetlabs/puppetlabs-apache.git\n[submodule \"config/puppet/modules/concat\"]\n\tpath = config/puppet/modules/concat\n\turl = git://github.com/puppetlabs/puppetlabs-concat.git\n[submodule \"config/puppet/modules/firewall\"]\n\tpath = config/puppet/modules/firewall\n\turl = git://github.com/puppetlabs/puppetlabs-firewall.git\n[submodule \"config/puppet/modules/stdlib\"]\n\tpath = config/puppet/modules/stdlib\n\turl = git://github.com/puppetlabs/puppetlabs-stdlib.git\n[submodule \"config/puppet/modules/vcsrepo\"]\n\tpath = config/puppet/modules/vcsrepo\n\turl = git://github.com/puppetlabs/puppetlabs-vcsrepo.git\n[submodule \"config/puppet/modules/biocloudcentral\"]\n\tpath = config/puppet/modules/biocloudcentral\n\turl = git://github.com/bioconfig/puppet-biocloudcentral.git\n[submodule \"config/puppet/modules/python\"]\n\tpath = config/puppet/modules/python\n\turl = git://github.com/bioconfig/puppet-python.git\n[submodule \"config/chef/cookbooks/globus\"]\n\tpath = config/chef/cookbooks/globus\n\turl = git://github.com/bioconfig/chef-globus.git\n[submodule \"config/puppet/modules/lwr\"]\n\tpath = config/puppet/modules/lwr\n\turl = git://github.com/bioconfig/puppet-lwr.git\n"
  },
  {
    "path": "LICENSE.txt",
    "content": "Copyright (c) 2013 CloudBioLinux contributors\n\nPermission is hereby granted, free of charge, to any person obtaining\na copy of this software and associated documentation files (the\n\"Software\"), to deal in the Software without restriction, including\nwithout limitation the rights to use, copy, modify, merge, publish,\ndistribute, sublicense, and/or sell copies of the Software, and to\npermit persons to whom the Software is furnished to do so, subject to\nthe following conditions:\n\nThe above copyright notice and this permission notice shall be\nincluded in all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND,\nEXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF\nMERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND\nNONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE\nLIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION\nOF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION\nWITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n"
  },
  {
    "path": "MANIFEST.in",
    "content": "include *fabfile.py\ninclude *.md\ninclude *.mkd\ninclude config/*.yaml\ninclude config/fabricrc.txt\ninclude config/README.md\ninclude doc/*.md\ninclude installed_files/*\ninclude utils/*\n"
  },
  {
    "path": "README.rst",
    "content": "CloudBioLinux is a build and deployment system which installs an easily\ncustomizable selection of bioinformatics and machine learning libraries on a\nlinux container, bare virtual machine (VM) image, freshly installed PC, or in\nthe cloud. CloudBioLinux is a curated and community developed set of\ninstructions for tools provided by operating system packages (debs and RPMs),\nexternal packaging efforts (`bioconda <https://bioconda.github.io/>`_ and\n`homebrew-science <https://github.com/Homebrew/homebrew-science>`_)\nand language specific library installers (Python, R, Perl and Ruby).\n\nCloudBioLinux included software packages are fully customizable. In\naddition to the default configuration, we support custom configuration\nbuilds through flavors. Flavors support overriding default package\ninstallations, making it simple to create derived installs for specific\npurposes.\n\nCloudBioLinux is a single install route for `Docker containers <http://www.docker.com/>`_\n,desktop VMs such as `VirtualBox <http://digitizor.com/2011/01/07/virtualbox-4-0-install-ubuntu/>`_,\ncloud providers such as `Amazon EC2 <http://aws.amazon.com/ec2/>`_ or\ndesktop machines. This works equally well for other virtual machines and\nprivate cloud environments, including `XEN <http://xen.org/>`_, Linux\n`KVM <http://www.linux-kvm.org/>`_,\n`Eucalyptus <http://open.eucalyptus.com/>`_ and\n`Openstack <http://www.openstack.org/>`_.\n\nQuick start\n===========\n\n`bcbio <http://bcbio-nextgen.readthedocs.io/en/latest/>`_ uses CloudBioLinux as\nthe basis for tool installation and provides a large set of supported and tested\ntools. If you're looking to bootstrap a system with tools and data for high\nthroughput sequencing analysis, we suggest using the `bcbio installer\n<http://bcbio-nextgen.readthedocs.io/en/latest/contents/installation.html#automated>`_\nwhich fully wraps CloudBioLinux and provides an easy path to customize install\ndirectories, organisms and biological data installed.\n\nTo modify/add data/package recipies to bcbio, edit the below configs:\n\n- bcbio data recipes: https://github.com/chapmanb/cloudbiolinux/tree/master/ggd-recipes\n- bcbio data index1: https://github.com/chapmanb/cloudbiolinux/blob/master/config/biodata.yaml\n- bcbio data index2: https://github.com/bcbio/bcbio-nextgen/tree/master/config/genomes\n- bcbio conda packages index: https://github.com/chapmanb/cloudbiolinux/blob/master/contrib/flavor/ngs_pipeline_minimal/packages-conda.yaml\n\nWe recommend using, or developing, a custom flavor to choose tools of interest\nto install. The amount of bioinformatics software continues to increase -- there are\n`over 1000 recipes in bioconda <https://github.com/bioconda/bioconda-recipes>`_\n-- and it's difficult to come up with a default installation that includes\neverything for everyone. The ``ngs_pipeline_minimal`` flavor has the set of NGS\nanalysis tools installed with bcbio and is a good starting point for\nunderstanding the CloudBioLinux install process. To install inside an isolated\nconda environment on a bare machine do::\n\n    git clone https://github.com/chapmanb/cloudbiolinux.git\n    cd cloudbiolinux\n    wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh\n    bash Miniconda2-latest-Linux-x86_64.sh -b -p ~/cblenv\n    ~/cblenv/bin/conda install -y -c bioconda -c conda-forge pip fabric pyyaml\n    ~/cblenv/bin/fab -f fabfile.py -H localhost install_biolinux:flavor=ngs_pipeline_minimal --set keep_isolated=true\n\nInstalling CloudBioLinux on a local machine\n===========================================\n\nThe install process for CloudBioLinux is fully automated through a `Fabric build\nfile <http://fabfile.org/>`_ written in Python. Everything is fully configurable\nthrough plain text YAML configuration files, and custom build targets allow\ninstallation of a subset of the total available packages.\n\nSetup\n-----\n\nRetrieve the CloudBioLinux code base and install fabric::\n\n    pip install fabric\n    git clone git://github.com/chapmanb/cloudbiolinux.git\n    cd cloudbiolinux\n\nUsage\n-----\n\nThe basic usage specifies the hostname of a machine accessible via ssh or the\nlocal machine::\n\n    fab -f fabfile.py -H localhost install_biolinux\n\nFabric contains some other useful commandline arguments for customizing\nthis to your environments:\n\n-  ``-c your_fabricrc.txt`` -- Specify the path to a fabricrc\n   configuration files. This allows customization of install directories\n   and other server specific details. See the default\n   ``config/fabricrc.txt`` for a full list of options.\n\n-  ``-u username`` -- The username on a remote machine, overriding the\n   default of your current username.\n\nCustomization with flavors\n--------------------------\n\nIn most cases you want to customize a specific set of packages,\nor install into an isolated directory without root access, using flavors::\n\n    fab -f fabfile.py -H localhost install_biolinux:flavor=my_flavor\n\n``my_flavor`` can be the name of an existing flavor in\n``contrib/flavor`` or the path to a directory with customization\ninformation. The files in your flavor directory replace those in the\nstandard ``config`` directory, allowing replacement of any of the\nconfiguration files like ``main.yaml`` with customized copies.\nIf you desire even more control, flavors allow custom python hooks. See\n``doc/hacking.md`` for more details.\n\nThe best place to get started is the `demo flavor\n<https://github.com/chapmanb/cloudbiolinux/tree/master/contrib/flavor/demo>`_\nincluded with CloudBioLinux. This installs a small number of common packages\ninto an isolated directory (``~/tmp/cbl_demo`` by default), without root access.\nRun the example with::\n\n    fab -f fabfile.py -H localhost install_biolinux:flavor=demo\n\nSpecific install targets\n------------------------\n\nYou can substitute ``install_biolinux`` with more specific targets to\nonly build portions of CloudBioLinux:\n\n-  ``install_biolinux:packages`` -- Install all of the defined system\n   packages.\n-  ``install_biolinux:libraries`` -- Install all libraries for various\n   programming languages.\n-  ``install_biolinux:brew`` -- Install homebrew packages only.\n-  ``install_libraries:language`` -- Install libraries for a specific\n   language.\n-  ``install_biolinux:custom`` -- Install all custom programs.\n-  ``install_brew:a_package_name`` -- Install a specific brew package.\n-  ``install_custom:a_package_name`` -- Install a specific custom\n   program.\n\nHomebrew package installation\n-----------------------------\n\n`Homebrew <https://github.com/Homebrew/homebrew>`_ and `Linuxbrew\n<https://github.com/Homebrew/linuxbrew>`_ provide a Ruby-based environment for\ninstalling packages on MacOSX and Linux. The active\n`homebrew-science <https://github.com/Homebrew/homebrew-science>`_ packaging\ncommunity maintains a number of common scientific tools. We also maintain a\n`homebrew-cbl <https://github.com/chapmanb/homebrew-cbl>`_ repository with tools\nnot yet integrated into homebrew-science.\n\nCloudBioLinux manages installation of the Linuxbrew or Homebrew framework and\npulls in the ``homebrew/science`` and ``chapmanb/cbl`` taps, as well as\ninjecting your current compilers into the homebrew build scripts. To install a\n`supported package\n<https://github.com/chapmanb/cloudbiolinux/blob/master/config/packages-homebrew.yaml>`_\nusing CloudBioLinux::\n\n     fab -f fabfile.py -H localhost install_custom:bedtools\n\nSpecific package installation\n-----------------------------\n\nThe custom directory contains installation instructions for programs\nthat are not available from standard package repositories, written in Python\nusing the `Fabric <http://fabfile.org/>`_ remote deployment tool. To install\nindividual `custom packages\n<https://github.com/chapmanb/cloudbiolinux/blob/master/config/custom.yaml>`_::\n\n      fab -f fabfile.py -H localhost install_custom:your_package_name\n\nWe prefer using the Homebrew framework for new packages over writing custom\npackages.\n\nBiological data\n---------------\n\nWe manage a repository of useful public biological data on an `Amazon S3\nbucket <http://s3.amazonaws.com/biodata>`_. Currently this includes\nwhole genomes pre-indexed for a number of popular aligners. Downloading\nand installing these saves a ton of time over running the indexing steps\nyourself, and eases running next-generation analyses on cloud machines.\n\nA Fabric build script is provided to install this data on your local\nmachine. A `biodata configuration file in YAML\nformat <https://github.com/chapmanb/cloudbiolinux/blob/master/config/biodata.yaml>`_,\n``config/biodata.yaml``, specifies the genomes of interest and the\naligner indexes to use. The ``config/fabricrc.txt`` file specifies\ndetails about the system and where to install the data.\n\nThe basic commandline is::\n\n    fab -f data_fabfile.py -H your_machine install_data_s3\n\nand you can pass in custom biodata and fabricrc files with::\n\n    fab -f data_fabfile.py -H your_machine -c your_fabricrc.txt install_data_s3:your_biodata.yaml\n\nIn addition to downloading and preparing the data, the script will\nintegrate these files with a Galaxy instance by updating appropriate\nGalaxy configuration files. This makes it useful for installing data to\na local or\n`cloud-based <https://bitbucket.org/galaxy/galaxy-central/wiki/cloud>`_\nGalaxy server.\n\nNot all of the genomes are hosted on the S3 bucket, but are still supported. If your\ngenome fails to install with install_data_s3, you might be able to download the genome\nfrom Ensembl, etc and prepare it::\n\n\n    fab -f data_fabfile.py -H your_machine -c your_fabricrc.txt install_data:your_biodata.yaml\n\nUsing pre-built cloud images\n============================\n\nAmazon\n------\n\nSee the 'Getting Started with CloudBioLinux' guide on the `CloudBioLinux\nwebsite <http://cloudbiolinux.org/>`_ for a detailed description. The\nshort version for users familiar with Amazon is:\n\n-  Login to the `Amazon EC2\n   console <https://console.aws.amazon.com/ec2/home>`_.\n-  Click Launch Instance, and choose the latest CloudBioLinux AMI from\n   the `website <http://cloudbiolinux.org/>`_ in the community AMI\n   section (search for 'CloudBioLinux').\n-  After launching the instance, find the host details of your running\n   instance from the Instances section.\n-  Connect to your machine via ssh or VNC (using the Amazon PEM keys)\n\nSupported environments\n======================\n\nDocker\n------\n`Docker <http://www.docker.com/>`_ provides lightweight local containers for\nLinux machines, allowing isolation without the associated overhead of full\nvirtual machines. Include any of the standard CloudBioLinux commands inside\na `Dockerfile <http://docs.docker.com/reference/builder/>`_ to use CloudBioLinux\nto build up the set of tools on your instance. See the\n`Dockerfile examples <http://docs.docker.com/installation/#examples>`_ for\ninformation how to write Dockerfiles.\n\nTo use a pre-built Docker image made with CloudBioLinux infrastructure, using\nthis `bcbio-nextgen Dockerfile\n<https://github.com/chapmanb/bcbio-nextgen/blob/master/Dockerfile>`_, you can\nimport the `bcbio-nextgen <https://github.com/chapmanb/bcbio-nextgen>`_\ncontainer into your local docker environment::\n\n    docker import https://s3.amazonaws.com/bcbio_nextgen/bcbio-nextgen-docker-image.gz chapmanb/bcbio-nextgen-cbl\n\nAmazon\n------\n\nA bare Linux image launched in Amazon EC2 is configured from another\nmachine, i.e. your local desktop, using ssh and cloudbiolinux. See the\nInstallation section for installing CloudBioLinux with fabric.\n\nAny cloudbiolinux distribution can be used, including Ubuntu, Debian\nLinux and CentOS. We recommend using m1.medium or better instance for building a\nCloudBioLinux image from scratch, due to resource usage while compiling\nsoftware.\n\n1. Go to the cloudbiolinux source and edit the ``config/fabricrc.txt``,\n   to match the system you plan to install on. Specifically,\n   ``distribution`` and ``dist_name`` parameters specify details about\n   the type of target.\n\n2. Start an Amazon EC2 base instance and retrieve it's DNS hostname:\n\n-  `Alestic Ubuntu images <http://alestic.com/>`_\n-  `Camptocamp Debian\n   images <http://www.camptocamp.com/en/infrastructure-solutions/amazon-images>`_\n\n3. From your local machine, have CloudBioLinux install your Amazon\n   instance:\n\n   ::\n\n       fab -f fabfile.py -H hostname -u username -i private_key_file install_biolinux\n\n4. When finished, use the `Amazon\n   console <https://console.aws.amazon.com/ec2/home>`_ to create an AMI.\n   Thereafter make it public so it can be used by others.\n\nVagrant and VirtualBox\n----------------------\n\nVagrant allows easy deploying and connecting to VirtualBox images. The\nsetup is ideal for running CloudBioLinux on a desktop computer. Install\n`VirtualBox <https://www.virtualbox.org/>`_\nand `vagrant <http://vagrantup.com/>`_.\n\nSee `the VirtualBox and Vagrant documentation\n<https://github.com/chapmanb/cloudbiolinux/blob/master/doc/virtualbox.md>`_ for\ndetails on creating a local virtual machine from scratch with CloudBioLinux.\n\nThrough Vagrant additional facilities are available, such as a shared\nnetwork drive. It is also possible to tweak the image (e.g. RAM/CPU\nsettings, and getting the all important guest additions) by firing up\nvirtualbox itself. For more information, see the\ndocumentation on the `Vagrant website <http://vagrantup.com/>`_.\n\nOpenStack/XEN/KVM/Eucalyptus private Cloud\n------------------------------------------\n\nAs long as there is an 'ssh' entry to an running VM, CloudBioLinux can\ninstall itself.\n\nFor more on private Cloud and CloudBioLinux see ./doc/private\\_cloud.md.\n\nEC2 quickstart\n==============\n\nThis provides a quick cheat sheet of commands for getting up and running\non EC2 using Amazon's command line tools.\n\nInitial set up\n--------------\n\nThe first time using EC2, you'll need to install the toolkit and\ncredentials for connecting on your local machine, following the `getting\nstarted\nguide <http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/>`_.\n\nLogin to your `Amazon EC2 account <http://aws.amazon.com/account/>`_ and\ngo to Security Credentials/X.509. Create a new certificate and download\nthe public ``cert-*.pem`` and ``private pk-*.pem`` files. Put these in\n``~.ec2``.\n\nInstall the `ec2 api\ntools <http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351&categoryID=88>`_,\nwhich require java.\n\nSet up .zshrc/.bashrc:\n\n::\n\n       export EC2_PRIVATE_KEY=~/.ec2/pk-UBH43XTAWVNQMIZRAV3RP5IIBAPBIFVP.pem\n       export EC2_CERT=~/.ec2/cert-UBH43XTAWVNQMIZRAV3RP5IIBAPBIFVP.pem\n       export AWS_ACCESS_KEY_ID=<your access key>\n       export AWS_SECRET_ACCESS_KEY=<your secret access key>\n\nTo test, you should be able to run the command:\n\n::\n\n       % ec2-describe-regions\n\nNow generate a privatekey for logging in:\n\n::\n\n       % ec2-add-keypair yourmachine-keypair\n\nThis will produce an RSA private key. You should copy and paste this to\nyour .ec2 directory for future use:\n\n::\n\n       % vim ~/.ec2/id-yourmachine.keypair\n       % chmod 600 ~/.ec2/id-yourmachine.keypair\n\nAllow ssh and web access to your instances:\n\n::\n\n       % ec2-authorize default -p 22\n       % ec2-authorize default -p 80\n\nStarting an instance\n--------------------\n\nEach time you'd like to use EC2, you need to create a remote instance to\nwork with; the `AWS console <http://alestic.com/>`_ is useful for\nmanaging this process.\n\nWhen building from scratch with Alestic images, you will need to\nincrease the size of the root filesystem to fit all of the CloudBioLinux\ndata and libraries. This is done by starting the instance from the\ncommandline with:\n\n::\n\n       % ec2-run-instances ami-1aad5273 -k kunkel-keypair -t m1.large\n                           -b /dev/sda1=:20\n       % ec2-describe-instances i-0ca39764\n\nOn Ubuntu 10.04, you then need to ssh into the instance and resize the\nfilesystem with:\n\n::\n\n       % sudo resize2fs /dev/sda1\n\nOn 11.04 the resize happens automatically and this is not required.\n\nTesting\n=======\n\nBioLinux comes with an integration testing frame work - currently based\non Vagrant. Try:\n\n::\n\n        cd test\n        ./testing_vagrant --help\n\nTarget VMs can be listed with\n\n::\n\n        ./testing_vagrant --list\n\nBuild a minimal VM\n\n::\n\n        ./testing_vagrant Minimal\n\nDocumentation\n=============\n\nAdditional documentation can be found in the `./doc\ndirectory <https://github.com/chapmanb/cloudbiolinux>`_ in the BioLinux\nsource tree.\n\nLICENSE\n=======\n\nThe code is freely available under the `MIT\nlicense <http://www.opensource.org/licenses/mit-license.html>`_.\n"
  },
  {
    "path": "cloudbio/__init__.py",
    "content": "\"\"\"Module level code supporting CloudBioLinux installations.\n\nThis provides a reusable architecture allowing definitions and helper code to be used \nin other projects.\n\"\"\"\n__import__('pkg_resources').declare_namespace(__name__)\n"
  },
  {
    "path": "cloudbio/biodata/__init__.py",
    "content": "\"\"\"Download, installation and configuration of biological data.\n\"\"\"\n"
  },
  {
    "path": "cloudbio/biodata/galaxy.py",
    "content": "\"\"\"Retrieve indexed genomes using Galaxy's rsync server resources.\n\nhttp://wiki.galaxyproject.org/Admin/Data%20Integration\n\"\"\"\nfrom __future__ import print_function\n\nimport os\nimport shutil\nimport subprocess\nfrom xml.etree import ElementTree\n\nfrom cloudbio.custom import shared\n\n# ## Compatibility definitions\n\nserver = \"rsync://datacache.g2.bx.psu.edu\"\n\nindex_map = {\"bismark\": \"bismark_index\",\n             \"bowtie\": \"bowtie_index\",\n             \"bowtie2\": \"bowtie2_index\",\n             \"bwa\": \"bwa_index\",\n             \"novoalign\": \"novoalign_index\",\n             \"ucsc\": \"seq\",\n             \"seq\": \"sam_index\"}\n\norg_remap = {\"phix\": \"phiX\",\n             \"GRCh37\": \"hg_g1k_v37\",\n             \"araTha_tair9\": \"Arabidopsis_thaliana_TAIR9\",\n             \"araTha_tair10\": \"Arabidopsis_thaliana_TAIR10\",\n             \"WS210\": \"ce10\",\n             \"WS220\": \"ce10\"}\n\ngalaxy_subdirs = [\"\", \"/microbes\"]\n\n# ## Galaxy location files\n\nclass LocCols(object):\n    # Hold all possible .loc file column fields making sure the local\n    # variable names match column names in Galaxy's tool_data_table_conf.xml\n    def __init__(self, config, dbkey, file_path):\n        self.dbkey = dbkey\n        self.path = file_path\n        self.value = config.get(\"value\", dbkey)\n        self.name = config.get(\"name\", dbkey)\n        self.species = config.get('species', '')\n        self.index = config.get('index', 'index')\n        self.formats = config.get('index', 'fastqsanger')\n        self.dbkey1 = config.get('index', dbkey)\n        self.dbkey2 = config.get('index', dbkey)\n\ndef _get_tool_conf(env, tool_name):\n    \"\"\"\n    Parse the tool_data_table_conf.xml from installed_files subfolder and extract\n    values for the 'columns' tag and 'path' parameter for the 'file' tag, returning\n    those as a dict.\n    \"\"\"\n    tool_conf = {}\n    tdtc = ElementTree.parse(env.tool_data_table_conf_file)\n    tables = tdtc.getiterator('table')\n    for t in tables:\n        if tool_name in t.attrib.get('name', ''):\n            tool_conf['columns'] = t.find('columns').text.replace(' ', '').split(',')\n            tool_conf['file'] = t.find('file').attrib.get('path', '')\n    return tool_conf\n\ndef _build_galaxy_loc_line(env, dbkey, file_path, config, prefix, tool_name):\n    \"\"\"Prepare genome information to write to a Galaxy *.loc config file.\n    \"\"\"\n    if tool_name:\n        str_parts = []\n        tool_conf = _get_tool_conf(env, tool_name)\n        loc_cols = LocCols(config, dbkey, file_path)\n        # Compose the .loc file line as str_parts list by looking for column values\n        # from the retrieved tool_conf (as defined in tool_data_table_conf.xml).\n        # Any column values required but missing in the tool_conf are\n        # supplemented by the defaults defined in LocCols class\n        for col in tool_conf.get('columns', []):\n            str_parts.append(config.get(col, getattr(loc_cols, col)))\n    else:\n        str_parts = [dbkey, file_path]\n    if prefix:\n        str_parts.insert(0, prefix)\n    return str_parts\n\ndef update_loc_file(env, ref_file, line_parts):\n    \"\"\"Add a reference to the given genome to the base index file.\n    \"\"\"\n    if getattr(env, \"galaxy_home\", None) is not None:\n        tools_dir = os.path.join(env.galaxy_home, \"tool-data\")\n        if not os.path.exists(tools_dir):\n            subprocess.check_call(\"mkdir -p %s\" % tools_dir, shell=True)\n        dt_file = os.path.join(env.galaxy_home, \"tool_data_table_conf.xml\")\n        if not os.path.exists(dt_file):\n            shutil.copy(env.tool_data_table_conf_file, dt_file)\n        add_str = \"\\t\".join(line_parts)\n        with shared.chdir(tools_dir):\n            if not os.path.exists(ref_file):\n                subprocess.check_call(\"touch %s\" % ref_file, shell=True)\n            has_line = False\n            with open(ref_file) as in_handle:\n                for line in in_handle:\n                    if line.strip() == add_str.strip():\n                        has_line = True\n            if not has_line:\n                with open(ref_file, \"a\") as out_handle:\n                    out_handle.write(add_str + \"\\n\")\n\ndef prep_locs(env, gid, indexes, config):\n    \"\"\"Prepare Galaxy location files for all available indexes.\n    \"\"\"\n    for ref_index_file, cur_index, prefix, tool_name in [\n            (\"alignseq.loc\", indexes.get(\"ucsc\", None), \"seq\", None),\n            (\"bismark_indices.loc\", indexes.get(\"bismark\", None), \"\", 'bismark_indexes'),\n            (\"bowtie2_indices.loc\", indexes.get(\"bowtie2\", None), \"\", 'bowtie2_indexes'),\n            (\"bowtie_indices.loc\", indexes.get(\"bowtie\", None), \"\", 'bowtie_indexes'),\n            (\"bwa_index.loc\", indexes.get(\"bwa\", None), \"\", 'bwa_indexes'),\n            (\"gatk_sorted_picard_index.loc\", indexes.get(\"seq\", None), \"\", \"gatk_picard_indexes\"),\n            (\"mosaik_index.loc\", indexes.get(\"mosaik\", None), \"\", \"mosaik_indexes\"),\n            (\"novoalign_indices.loc\", indexes.get(\"novoalign\", None), \"\", \"novoalign_indexes\"),\n            (\"picard_index.loc\", indexes.get(\"seq\", None), \"\", \"picard_indexes\"),\n            (\"sam_fa_indices.loc\", indexes.get(\"seq\", None), \"\", 'sam_fa_indexes'),\n            (\"twobit.loc\", indexes.get(\"ucsc\", None), \"\", None)]:\n        if cur_index:\n            str_parts = _build_galaxy_loc_line(env, gid, cur_index, config, prefix, tool_name)\n            update_loc_file(env, ref_index_file, str_parts)\n\n# ## Finalize downloads\n\ndef index_picard(ref_file):\n    \"\"\"Provide a Picard style dict index file for a reference genome.\n    \"\"\"\n    index_file = \"%s.dict\" % os.path.splitext(ref_file)[0]\n    if not os.path.exists(index_file):\n        subprocess.check_call(\"picard -Xms500m -Xmx3500m CreateSequenceDictionary REFERENCE={ref} OUTPUT={out}\"\n                              .format(ref=ref_file, out=index_file), shell=True)\n    return index_file\n\ndef _finalize_index_seq(fname):\n    \"\"\"Convert UCSC 2bit file into fasta file.\n    \"\"\"\n    out_fasta = fname + \".fa\"\n    if not os.path.exists(out_fasta):\n        subprocess.check_call(\"twoBitToFa {base}.2bit {out}\".format(\n            base=fname, out=out_fasta), shell=True)\n\nfinalize_fns = {\"ucsc\": _finalize_index_seq,\n                \"seq\": index_picard}\n\ndef _finalize_index(idx, fname):\n    \"\"\"Perform final processing on an rsync'ed index file if necessary.\n    \"\"\"\n    finalize_fn = finalize_fns.get(idx)\n    if finalize_fn:\n        finalize_fn(fname)\n\n# ## Retrieve data from Galaxy\n\ndef rsync_genomes(genome_dir, genomes, genome_indexes):\n    \"\"\"Top level entry point to retrieve rsync'ed indexes from Galaxy.\n    \"\"\"\n    for gid in (x[1] for x in genomes):\n        galaxy_gid = org_remap.get(gid, gid)\n        indexes = _get_galaxy_genomes(galaxy_gid, genome_dir, genomes, genome_indexes)\n        _finalize_index(\"ucsc\", indexes[\"ucsc\"])\n        for idx, fname in indexes.iteritems():\n            _finalize_index(idx, fname)\n        prep_locs(galaxy_gid, indexes, {})\n\ndef _get_galaxy_genomes(gid, genome_dir, genomes, genome_indexes):\n    \"\"\"Retrieve the provided genomes and indexes from Galaxy rsync.\n    \"\"\"\n    out = {}\n    org_dir = os.path.join(genome_dir, gid)\n    if not os.path.exists(org_dir):\n        subprocess.check_call('mkdir -p %s' % org_dir, shell=True)\n    for idx in genome_indexes:\n        galaxy_index_name = index_map.get(idx)\n        index_file = None\n        if galaxy_index_name:\n            index_file = _rsync_genome_index(gid, galaxy_index_name, org_dir)\n        if index_file:\n            out[idx] = index_file\n        else:\n            print(\"Galaxy does not support {0} for {1}\".format(idx, gid))\n    return out\n\ndef _rsync_genome_index(gid, idx, org_dir):\n    \"\"\"Retrieve index for a genome from rsync server, returning path to files.\n    \"\"\"\n    idx_dir = os.path.join(org_dir, idx)\n    if not os.path.exists(idx_dir):\n        org_rsync = None\n        for subdir in galaxy_subdirs:\n            test_rsync = \"{server}/indexes{subdir}/{gid}/{idx}/\".format(\n                server=server, subdir=subdir, gid=gid, idx=idx)\n            try:\n                subprocess.check_output(\"rsync --list-only {server}\".format(server=test_rsync))\n                org_rsync = test_rsync\n            except subprocess.CalledProcessError:\n                pass\n        if org_rsync is None:\n            raise ValueError(\"Could not find genome %s on Galaxy rsync\" % gid)\n        try:\n            subprocess.check_call(\"rsync --list-only {server}\".format(server=org_rsync), shell=True)\n            if not os.path.exists(idx_dir):\n                subprocess.check_call('mkdir -p %s' % idx_dir, shell=True)\n            with cd(idx_dir):\n                subprocess.check_call(\"rsync -avzP {server} {idx_dir}\".format(server=org_rsync,\n                                                                              idx_dir=idx_dir), shell=True)\n        except subprocess.CalledProcessError:\n            pass\n    if os.path.exists(idx_dir):\n        try:\n            subprocess.check_call(\"ls {idx_dir}/{gid}.fa*\".format(idx_dir=idx_dir,\n                                                                  gid=gid), shell=True)\n            ext = \".fa\" if (has_fa_ext.succeeded and idx not in [\"seq\"]) else \"\"\n        except subprocess.CalledProcessError:\n            pass\n        return os.path.join(idx_dir, gid + ext)\n"
  },
  {
    "path": "cloudbio/biodata/genomes.py",
    "content": "\"\"\"Download and install structured genome data and aligner index files.\n\nDownloads prepared FASTA, indexes for aligners like BWA, Bowtie and novoalign\nand other genome data in automated pipelines. Specify the genomes and aligners\nto use in an input biodata.yaml configuration file.\n\nThe main targets are fabric functions:\n\n  - install_data -- Install biological data from scratch, including indexing genomes.\n  - install_data_s3 -- Install biological data, downloading pre-computed indexes from S3.\n  - upload_s3 -- Upload created indexes to biodata S3 bucket.\n\"\"\"\nfrom __future__ import print_function\nimport collections\nimport os\nimport operator\nimport socket\nimport subprocess\nimport sys\nimport traceback\nfrom math import log\n\ntry:\n    import yaml\nexcept ImportError:\n    yaml = None\ntry:\n    import boto\nexcept ImportError:\n    boto = None\n\nfrom cloudbio.biodata import galaxy, ggd, rnaseq\nfrom cloudbio.custom import shared\n\n# -- Configuration for genomes to download and prepare\n\nclass _DownloadHelper:\n    def __init__(self):\n        self.config = {}\n\n    def ucsc_name(self):\n        return None\n\n    def _exists(self, fname, seq_dir):\n        \"\"\"Check if a file exists in either download or final destination.\n        \"\"\"\n        return os.path.exists(fname) or os.path.exists(os.path.join(seq_dir, fname))\n\nclass UCSCGenome(_DownloadHelper):\n    def __init__(self, genome_name, dl_name=None):\n        _DownloadHelper.__init__(self)\n        self.data_source = \"UCSC\"\n        self._name = genome_name\n        self.dl_name = dl_name if dl_name is not None else genome_name\n        self._url = \"ftp://hgdownload.cse.ucsc.edu/goldenPath/%s/bigZips\" % \\\n                genome_name\n\n    def ucsc_name(self):\n        return self._name\n\n    def _karyotype_sort(self, xs):\n        \"\"\"Sort reads in karyotypic order to work with GATK's defaults.\n        \"\"\"\n        def karyotype_keyfn(x):\n            base = os.path.splitext(os.path.basename(x))[0]\n            if base.startswith(\"chr\"):\n                base = base[3:]\n            parts = base.split(\"_\")\n            try:\n                base = int(parts[0])\n            except ValueError:\n                base = sys.maxsize\n            # unplaced at the very end\n            if parts[0] == \"Un\":\n                parts.insert(0, \"z\")\n            # mitochondrial special case -- after X/Y\n            elif parts[0] in [\"M\", \"MT\"]:\n                parts.insert(0, \"x\")\n            # sort random and extra chromosomes after M\n            elif len(parts) > 1:\n                parts.insert(0, \"y\")\n            # standard integers, sort first\n            else:\n                parts.insert(0, \"a\")\n            return [base] + parts\n        return sorted(xs, key=karyotype_keyfn)\n\n    def _split_multifasta(self, fasta_file):\n        chrom = \"\"\n        file_handle = None\n        file_names = []\n        out_dir = os.path.dirname(fasta_file)\n        with open(fasta_file) as in_handle:\n            for line in in_handle:\n                if line.startswith(\">\"):\n                    chrom = line.split(\">\")[1].strip()\n                    file_handle.close() if file_handle else None\n                    file_names.append(chrom + \".fa\")\n                    file_handle = open(os.path.join(out_dir, chrom + \".fa\"), \"w\")\n                    file_handle.write(line)\n                else:\n                    file_handle.write(line)\n        file_handle.close()\n        return file_names\n\n    def download(self, seq_dir):\n        zipped_file = None\n        genome_file = \"%s.fa\" % self._name\n        if not self._exists(genome_file, seq_dir):\n            prep_dir = \"seq_prep\"\n            subprocess.check_call(\"mkdir -p %s\" % prep_dir, shell=True)\n            with shared.chdir(prep_dir):\n                zipped_file = self._download_zip(seq_dir)\n                if zipped_file.endswith(\".tar.gz\"):\n                    subprocess.check_call(\"tar -xzpf %s\" % zipped_file, shell=True)\n                elif zipped_file.endswith(\".zip\"):\n                    subprocess.check_call(\"unzip %s\" % zipped_file, shell=True)\n                elif zipped_file.endswith(\".gz\"):\n                    if not os.path.exists(\"out.fa\"):\n                        subprocess.check_call(\"gunzip -c %s > out.fa\" % zipped_file, shell=True)\n                else:\n                    raise ValueError(\"Do not know how to handle: %s\" % zipped_file)\n                tmp_file = genome_file.replace(\".fa\", \".txt\")\n                result = subprocess.check_output(\"find `pwd` -name '*.fa'\", shell=True).decode()\n                result = [x.strip() for x in result.split(\"\\n\")]\n                if len(result) == 1:\n                    orig_result = result[0]\n                    result = self._split_multifasta(result[0])\n                    subprocess.check_call(\"rm %s\" % orig_result, shell=True)\n                result = self._karyotype_sort(result)\n                subprocess.check_call(\"rm -f inputs.txt\", shell=True)\n                for fname in result:\n                    subprocess.check_output(\"echo '%s' >> inputs.txt\" % fname, shell=True).decode()\n                subprocess.check_call(\"cat `cat inputs.txt` > %s\" % (tmp_file), shell=True)\n                for fname in result:\n                    subprocess.check_output(\"rm -f %s\" % fname, shell=True).decode()\n                subprocess.check_call(\"mv %s %s\" % (tmp_file, genome_file), shell=True)\n                zipped_file = os.path.join(prep_dir, zipped_file)\n                genome_file = os.path.join(prep_dir, genome_file)\n        return genome_file, [zipped_file]\n\n    def _download_zip(self, seq_dir):\n        for zipped_file in [\"chromFa.tar.gz\", \"%s.fa.gz\" % self._name,\n                            \"chromFa.zip\"]:\n            if not self._exists(zipped_file, seq_dir):\n                result = shared._remote_fetch(None, \"%s/%s\" % (self._url, zipped_file), allow_fail=True)\n                if result:\n                    break\n            else:\n                break\n        return zipped_file\n\nclass NCBIRest(_DownloadHelper):\n    \"\"\"Retrieve files using the TogoWS REST server pointed at NCBI.\n    \"\"\"\n    def __init__(self, name, refs, dl_name=None):\n        _DownloadHelper.__init__(self)\n        self.data_source = \"NCBI\"\n        self._name = name\n        self._refs = refs\n        self.dl_name = dl_name if dl_name is not None else name\n        self._base_url = \"http://togows.dbcls.jp/entry/ncbi-nucleotide/%s.fasta\"\n\n    def download(self, seq_dir):\n        genome_file = \"%s.fa\" % self._name\n        if not self._exists(genome_file, seq_dir):\n            for ref in self._refs:\n                shared._remote_fetch(None, self._base_url % ref)\n                subprocess.check_call(\"ls -l\", shell=True)\n                subprocess.check_call(r\"sed -i 's/^>.*$/>%s/' %s.fasta\" % (ref, ref), shell=True)\n            tmp_file = genome_file.replace(\".fa\", \".txt\")\n            subprocess.check_call(\"cat *.fasta > %s\" % tmp_file, shell=True)\n            subprocess.check_call(\"rm -f *.fasta\", shell=True)\n            subprocess.check_call(\"rm -f *.bak\", shell=True)\n            subprocess.check_call(\"mv %s %s\" % (tmp_file, genome_file), shell=True)\n        return genome_file, []\n\nclass VectorBase(_DownloadHelper):\n    \"\"\"Retrieve genomes from VectorBase) \"\"\"\n\n    def __init__(self, name, genus, species, strain, release, assembly_types):\n        _DownloadHelper.__init__(self)\n        self._name = name\n        self.data_source = \"VectorBase\"\n        self._base_url = (\"http://www.vectorbase.org/sites/default/files/ftp/\"\n                          \"downloads/\")\n        _base_file = (\"{genus}-{species}-{strain}_{assembly}\"\n                      \"_{release}.fa.gz\")\n        self._to_get = []\n        for assembly in assembly_types:\n            self._to_get.append(_base_file.format(**locals()))\n\n    def download(self, seq_dir):\n        genome_file = \"%s.fa\" % self._name\n        for fn in self._to_get:\n            url = self._base_url + fn\n            if not self._exists(fn, seq_dir):\n                shared._remote_fetch(None, url)\n                subprocess.check_call(\"gunzip -c %s >> %s\" % (fn, genome_file), shell=True)\n        return genome_file, []\n\nclass EnsemblGenome(_DownloadHelper):\n    \"\"\"Retrieve genome FASTA files from Ensembl.\n    ftp://ftp.ensemblgenomes.org/pub/plants/release-22/fasta/\n    arabidopsis_thaliana/dna/Arabidopsis_thaliana.TAIR10.22.dna.toplevel.fa.gz\n    ftp://ftp.ensembl.org/pub/release-75/fasta/\n    caenorhabditis_elegans/dna/Caenorhabditis_elegans.WBcel235.75.dna.toplevel.fa.gz\n    ftp://ftp.ensemblgenomes.org/pub/bacteria/release-23/bacteria/fasta/\n    bacteria_17_collection/pseudomonas_aeruginosa_ucbpp_pa14/dna/\n    Pseudomonas_aeruginosa_ucbpp_pa14.GCA_000014625.1.23.dna.toplevel.fa.gz\n    \"\"\"\n    def __init__(self, ensembl_section, release, organism, name, subsection=None):\n        _DownloadHelper.__init__(self)\n        self.data_source = \"Ensembl\"\n        if ensembl_section == \"standard\":\n            url = \"ftp://ftp.ensembl.org/pub/\"\n        else:\n            url = \"ftp://ftp.ensemblgenomes.org/pub/%s/\" % ensembl_section\n        url += \"release-%s/fasta/\" % release\n        if subsection:\n            url += \"%s/\" % subsection\n        url += \"%s/dna/\" % organism.lower()\n        self._url = url\n        if ensembl_section == \"standard\":\n            self._get_file = \"%s.%s.dna.toplevel.fa.gz\" % (organism, name)\n        else:\n            self._get_file = \"%s.%s.%s.dna.toplevel.fa.gz\" % (organism, name, release)\n        self._name = name\n        self.dl_name = name\n\n    def download(self, seq_dir):\n        genome_file = \"%s.fa\" % self._name\n        if not self._exists(self._get_file, seq_dir):\n            shared._remote_fetch(None, \"%s%s\" % (self._url, self._get_file))\n        if not self._exists(genome_file, seq_dir):\n            subprocess.check_call(\"gunzip -c %s > %s\" % (self._get_file, genome_file), shell=True)\n        return genome_file, [self._get_file]\n\nclass BroadGenome(_DownloadHelper):\n    \"\"\"Retrieve genomes organized and sorted by Broad for use with GATK.\n\n    Uses the UCSC-name compatible versions of the GATK bundles.\n    \"\"\"\n    def __init__(self, name, target_fasta, dl_name=None):\n        _DownloadHelper.__init__(self)\n        self.data_source = \"UCSC\"\n        self._name = name\n        self.dl_name = dl_name if dl_name is not None else name\n        self._target = target_fasta\n        self._ftp_url = \"ftp://gsapubftp-anonymous:@ftp.broadinstitute.org/bundle/\" + \\\n                        \"{org}/\".format(org=self.dl_name)\n\n    def download(self, seq_dir):\n        org_file = \"%s.fa\" % self._name\n        if not self._exists(org_file, seq_dir):\n            shared._remote_fetch(None, \"%s%s.gz\" % (self._ftp_url, self._target))\n            subprocess.check_call(\"gunzip %s.gz\" % self._target, shell=True)\n            subprocess.check_call(\"mv %s %s\" % (self._target, org_file), shell=True)\n        return org_file, []\n\nclass GGDGenome:\n    \"\"\"Genome with download specified via a GGD recipe.\n    \"\"\"\n    def __init__(self, name):\n        self._name = name\n\nGENOMES_SUPPORTED = [\n           (\"phiX174\", \"phix\", NCBIRest(\"phix\", [\"NC_001422.1\"])),\n           (\"Scerevisiae\", \"sacCer3\", UCSCGenome(\"sacCer3\")),\n           (\"Mmusculus\", \"mm10\", UCSCGenome(\"mm10\")),\n           (\"Mmusculus\", \"mm9\", UCSCGenome(\"mm9\")),\n           (\"Mmusculus\", \"mm8\", UCSCGenome(\"mm8\")),\n           (\"Hsapiens\", \"hg18\", BroadGenome(\"hg18\", \"Homo_sapiens_assembly18.fasta\")),\n           (\"Hsapiens\", \"hg19\", BroadGenome(\"hg19\", \"ucsc.hg19.fasta\")),\n           (\"Hsapiens\", \"GRCh37\", BroadGenome(\"GRCh37\", \"human_g1k_v37.fasta\", \"b37\")),\n           (\"Hsapiens\", \"hg38\", GGDGenome(\"hg38\")),\n           (\"Hsapiens\", \"hg38-noalt\", GGDGenome(\"hg38-noalt\")),\n           (\"Rnorvegicus\", \"rn6\", GGDGenome(\"rn6\")),\n           (\"Rnorvegicus\", \"rn5\", UCSCGenome(\"rn5\")),\n           (\"Rnorvegicus\", \"rn4\", UCSCGenome(\"rn4\")),\n           (\"Xtropicalis\", \"xenTro3\", UCSCGenome(\"xenTro3\")),\n           (\"Athaliana\", \"TAIR10\", EnsemblGenome(\"plants\", \"26\",\n                                                 \"Arabidopsis_thaliana\", \"TAIR10\")),\n           (\"Dmelanogaster\", \"dm3\", UCSCGenome(\"dm3\")),\n           (\"Dmelanogaster\", \"BDGP6\", GGDGenome(\"BDGP6\")),\n           (\"Celegans\", \"WBcel235\", EnsemblGenome(\"standard\", \"80\",\n                                                  \"Caenorhabditis_elegans\", \"WBcel235\")),\n           (\"Mtuberculosis_H37Rv\", \"mycoTube_H37RV\", NCBIRest(\"mycoTube_H37RV\",\n                                                              [\"NC_000962\"])),\n           (\"Msmegmatis\", \"92\", NCBIRest(\"92\", [\"NC_008596.1\"])),\n           (\"Paeruginosa_UCBPP-PA14\", \"pseudomonas_aeruginosa_ucbpp_pa14\",\n            EnsemblGenome(\"bacteria\", \"26\", \"Pseudomonas_aeruginosa_ucbpp_pa14\",\n                          \"GCA_000014625.1\", \"bacteria_17_collection\")),\n           (\"Ecoli\", \"eschColi_K12\", NCBIRest(\"eschColi_K12\", [\"U00096.2\"])),\n           (\"Amellifera_Honeybee\", \"apiMel3\", UCSCGenome(\"apiMel3\")),\n           (\"Cfamiliaris_Dog\", \"canFam3\", UCSCGenome(\"canFam3\")),\n           (\"Cfamiliaris_Dog\", \"canFam2\", UCSCGenome(\"canFam2\")),\n           (\"Drerio_Zebrafish\", \"Zv9\", EnsemblGenome(\"standard\", \"80\", \"Danio_rerio\", \"Zv9\")),\n           (\"Drerio_Zebrafish\", \"GRCz10\", EnsemblGenome(\"standard\", \"81\", \"Danio_rerio\", \"GRCz10\")),\n           (\"Drerio_Zebrafish\", \"GRCz11\", EnsemblGenome(\"standard\", \"92\", \"Danio_rerio\", \"GRCz11\")),\n           (\"Sscrofa\", \"Sscrofa11.1\", EnsemblGenome(\"standard\", \"92\", \"Sus_scrofa\", \"Sscrofa11.1\")),\n           (\"Ecaballus_Horse\", \"equCab2\", UCSCGenome(\"equCab2\")),\n           (\"Fcatus_Cat\", \"felCat3\", UCSCGenome(\"felCat3\")),\n           (\"Ggallus_Chicken\", \"galGal4\", UCSCGenome(\"galGal4\")),\n           (\"Tguttata_Zebra_finch\", \"taeGut1\", UCSCGenome(\"taeGut1\")),\n           (\"Aalbimanus\", \"AalbS1\", VectorBase(\"AalbS1\", \"Anopheles\",\n                                               \"albimanus\", \"STECLA\",\n                                               \"AalbS1\", [\"SCAFFOLDS\"])),\n           (\"Agambiae\", \"AgamP3\", VectorBase(\"AgamP3\", \"Anopheles\",\n                                             \"gambiae\", \"PEST\",\n                                             \"AgamP3\", [\"CHROMOSOMES\"]))]\nGENOME_INDEXES_SUPPORTED = [\"bowtie\", \"bowtie2\", \"bwa\", \"maq\", \"minimap2\", \"novoalign\",\n                            \"novoalign-cs\", \"ucsc\", \"mosaik\", \"snap\", \"star\",\n                            \"rtg\", \"hisat2\", \"bbmap\", \"bismark\"]\nDEFAULT_GENOME_INDEXES = [\"seq\"]\n\n# -- Fabric instructions\n\ndef _check_version(env):\n    version = env.version\n    if int(version.split(\".\")[0]) < 1:\n        raise NotImplementedError(\"Please install fabric version 1 or better\")\n\ndef install_data(config_source, approaches=None):\n    \"\"\"Main entry point for installing useful biological data, back compatible.\n    \"\"\"\n    from fabric.api import env\n    _check_version(env)\n    install_data_local(config_source, env.system_install, env.data_files,\n                       env.galaxy_home, env.tool_data_table_conf_file, env.cores, approaches)\n\ndef install_data_local(config_source, system_installdir, data_filedir,\n                       galaxy_home=None, tool_data_table_conf_file=None,\n                       cores=None, approaches=None):\n    \"\"\"Local installation of biological data, avoiding fabric usage.\n    \"\"\"\n    if not cores:\n        cores = 1\n    PREP_FNS = {\"s3\": _download_s3_index,\n                \"ggd\": _install_with_ggd,\n                \"raw\": _prep_raw_index}\n    if approaches is None: approaches = [\"ggd\", \"s3\", \"raw\"]\n    ready_approaches = []\n    Env = collections.namedtuple(\"Env\", \"system_install, galaxy_home, tool_data_table_conf_file, cores\")\n    env = Env(system_installdir, galaxy_home, tool_data_table_conf_file, cores)\n    for approach in approaches:\n        ready_approaches.append((approach, PREP_FNS[approach]))\n    # Append a potentially custom system install path to PATH so tools are found\n    os.environ[\"PATH\"] = \"%s/bin:%s\" % (os.path.join(system_installdir), os.environ[\"PATH\"])\n    genomes, genome_indexes, config = _get_genomes(config_source)\n    genome_indexes = [x for x in DEFAULT_GENOME_INDEXES if x not in genome_indexes] + genome_indexes\n    _make_genome_directories(genomes, data_filedir)\n    rnaseq.cleanup(genomes, data_filedir)\n    _prep_genomes(env, genomes, genome_indexes, ready_approaches, data_filedir)\n    rnaseq.finalize(genomes, data_filedir)\n\ndef install_data_s3(config_source):\n    \"\"\"Install data using pre-existing genomes present on Amazon s3.\n    \"\"\"\n    from fabric.api import env\n    _check_version(env)\n    genomes, genome_indexes, config = _get_genomes(config_source)\n    genome_indexes += [x for x in DEFAULT_GENOME_INDEXES if x not in genome_indexes]\n    _make_genome_directories(genomes, env.data_files)\n    rnaseq.cleanup(genomes, env.data_files)\n    _download_genomes(env, genomes, genome_indexes)\n    rnaseq.finalize(genomes, env.data_files)\n    _install_additional_data(env, genomes, genome_indexes, config)\n\ndef install_data_rsync(config_source):\n    \"\"\"Install data using pre-existing genomes from Galaxy rsync servers.\n    \"\"\"\n    from fabric.api import env\n    _check_version(env)\n    genomes, genome_indexes, config = _get_genomes(config_source)\n    genome_indexes += [x for x in DEFAULT_GENOME_INDEXES if x not in genome_indexes]\n    # Galaxy stores FASTAs in ucsc format and generates on the fly\n    if \"ucsc\" not in genome_indexes:\n        genome_indexes.append(\"ucsc\")\n    genome_dir = _make_genome_dir(env.data_files)\n    galaxy.rsync_genomes(genome_dir, genomes, genome_indexes)\n\ndef upload_s3(config_source):\n    \"\"\"Upload prepared genome files by identifier to Amazon s3 buckets.\n    \"\"\"\n    from fabric.api import env\n    if boto is None:\n        raise ImportError(\"install boto to upload to Amazon s3\")\n    if env.host != \"localhost\" and not env.host.startswith(socket.gethostname()):\n        raise ValueError(\"Need to run S3 upload on a local machine\")\n    _check_version(env)\n    genomes, genome_indexes, config = _get_genomes(config_source)\n    genome_indexes += [x for x in DEFAULT_GENOME_INDEXES if x not in genome_indexes]\n    _data_ngs_genomes(env, genomes, genome_indexes)\n    _upload_genomes(env, genomes, genome_indexes)\n\n\ndef _install_additional_data(env, genomes, genome_indexes, config):\n    for custom in (config.get(\"custom\") or []):\n        _prep_custom_genome(custom, genomes, genome_indexes, env)\n    if config.get(\"install_liftover\", False):\n        lift_over_genomes = [g.ucsc_name() for (_, _, g) in genomes if g.ucsc_name()]\n        _data_liftover(env, lift_over_genomes)\n    if config.get(\"install_uniref\", False):\n        _data_uniref(env)\n\ndef _get_genomes(config_source):\n    if isinstance(config_source, dict):\n        config = config_source\n    else:\n        if yaml is None:\n            raise ImportError(\"install yaml to read configuration from %s\" % config_source)\n        with open(config_source) as in_handle:\n            config = yaml.load(in_handle)\n    genomes = []\n    genomes_config = config[\"genomes\"] or []\n    print(\"List of genomes to get (from the config file at '{0}'): {1}\"\n          .format(config_source, ', '.join(g.get('name', g[\"dbkey\"]) for g in genomes_config)))\n    for g in genomes_config:\n        ginfo = None\n        for info in GENOMES_SUPPORTED:\n            if info[1] == g[\"dbkey\"]:\n                ginfo = info\n                break\n        assert ginfo is not None, \"Did not find download info for %s\" % g[\"dbkey\"]\n        name, gid, manager = ginfo\n        manager.config = g\n        genomes.append((name, gid, manager))\n    indexes = config[\"genome_indexes\"] or []\n    if \"seq\" in indexes:\n        indexes.remove(\"seq\")\n        indexes.insert(0, \"seq\")\n    return genomes, indexes, config\n\n# ## Decorators and context managers\n\ndef _if_installed(pname):\n    \"\"\"Run if the given program name is installed.\n    \"\"\"\n    def argcatcher(func):\n        def decorator(*args, **kwargs):\n            envs = [x for x in args if hasattr(x, \"system_install\")]\n            env = envs[0] if envs else None\n            if shared.which(pname, env):\n                return func(*args, **kwargs)\n        return decorator\n    return argcatcher\n\n# ## Generic preparation functions\n\ndef _make_genome_dir(data_filedir):\n    genome_dir = os.path.join(data_filedir, \"genomes\")\n    subprocess.check_output(\"mkdir -p %s\" % genome_dir, shell=True).decode()\n    return genome_dir\n\ndef _make_genome_directories(genomes, data_filedir):\n    genome_dir = _make_genome_dir(data_filedir)\n    for (orgname, gid, manager) in genomes:\n        org_dir = os.path.join(genome_dir, orgname, gid)\n        if not os.path.exists(org_dir):\n            subprocess.check_call('mkdir -p %s' % org_dir, shell=True)\n\ndef _prep_genomes(env, genomes, genome_indexes, retrieve_fns, data_filedir):\n    \"\"\"Prepare genomes with the given indexes, supporting multiple retrieval methods.\n    \"\"\"\n    genome_dir = _make_genome_dir(data_filedir)\n    for (orgname, gid, manager) in genomes:\n        org_dir = os.path.join(genome_dir, orgname, gid)\n        if not os.path.exists(org_dir):\n            subprocess.check_call('mkdir -p %s' % org_dir, shell=True)\n        ggd_recipes = manager.config.get(\"annotations\", []) + manager.config.get(\"validation\", [])\n        ggd_recipes += [x for x in manager.config.get(\"indexes\", []) if x in genome_indexes]\n        for idx in genome_indexes + ggd_recipes:\n            with shared.chdir(org_dir):\n                if idx in ggd_recipes or not os.path.exists(idx):\n                    finished = False\n                    last_exc = None\n                    for method, retrieve_fn in retrieve_fns:\n                        try:\n                            retrieve_fn(env, manager, gid, idx)\n                            finished = True\n                            break\n                        except KeyboardInterrupt:\n                            raise\n                        except BaseException as e:\n                            # Fail on incorrect GGD recipes\n                            if idx in ggd_recipes and method == \"ggd\":\n                                raise\n                            else:\n                                last_exc = traceback.format_exc()\n                                print(\"Moving on to next genome prep method after trying {0}\\n{1}\".format(\n                                      method, str(e)))\n                    if not finished:\n                        raise IOError(\"Could not prepare index {0} for {1} by any method\\n{2}\"\n                                      .format(idx, gid, last_exc))\n        ref_file = os.path.join(org_dir, \"seq\", \"%s.fa\" % gid)\n        if not os.path.exists(ref_file):\n            ref_file = os.path.join(org_dir, \"seq\", \"%s.fa\" % manager._name)\n        assert os.path.exists(ref_file), ref_file\n        _index_to_galaxy(env, org_dir, ref_file, gid, genome_indexes, manager.config)\n\n# ## Genomes index for next-gen sequencing tools\n\ndef _get_ref_seq(manager):\n    \"\"\"Check for or retrieve the reference sequence.\n    \"\"\"\n    seq_dir = os.path.join(os.getcwd(), \"seq\")\n    ref_file = os.path.join(seq_dir, \"%s.fa\" % manager._name)\n    if not os.path.exists(ref_file):\n        ref_file, base_zips = manager.download(seq_dir)\n        ref_file = _move_seq_files(ref_file, base_zips, seq_dir)\n    return ref_file\n\ndef _prep_raw_index(env, manager, gid, idx):\n    \"\"\"Prepare genome from raw downloads and indexes.\n    \"\"\"\n    print(\"Preparing genome {0} with index {1}\".format(gid, idx))\n    ref_file = _get_ref_seq(manager)\n    get_index_fn(idx)(env, ref_file)\n\ndef _data_ngs_genomes(env, genomes, genome_indexes):\n    \"\"\"Download and create index files for next generation genomes.\n    \"\"\"\n    genome_dir = _make_genome_dir(env.data_files)\n    for organism, genome, manager in genomes:\n        cur_dir = os.path.join(genome_dir, organism, genome)\n        print(\"Processing genome {0} and putting it to {1}\".format(organism, cur_dir))\n        if not os.path.exists(cur_dir):\n            subprocess.check_call('mkdir -p %s' % cur_dir, shell=True)\n        with shared.chdir(cur_dir):\n            if hasattr(env, \"remove_old_genomes\") and env.remove_old_genomes:\n                _clean_genome_directory()\n            seq_dir = 'seq'\n            ref_file, base_zips = manager.download(seq_dir)\n            ref_file = _move_seq_files(ref_file, base_zips, seq_dir)\n        cur_indexes = manager.config.get(\"indexes\", genome_indexes)\n        _index_to_galaxy(env, cur_dir, ref_file, genome, cur_indexes, manager.config)\n\ndef _index_to_galaxy(env, work_dir, ref_file, gid, genome_indexes, config):\n    \"\"\"Index sequence files and update associated Galaxy loc files.\n    \"\"\"\n    indexes = {}\n    with shared.chdir(work_dir):\n        for idx in genome_indexes:\n            index_file = get_index_fn(idx)(env, ref_file)\n            if index_file:\n                indexes[idx] = os.path.join(work_dir, index_file)\n    galaxy.prep_locs(env, gid, indexes, config)\n\nclass CustomMaskManager:\n    \"\"\"Create a custom genome based on masking an existing genome.\n    \"\"\"\n    def __init__(self, custom, config):\n        assert \"mask\" in custom\n        self._custom = custom\n        self.config = config\n\n    def download(self, seq_dir):\n        base_seq = os.path.join(os.pardir, self._custom[\"base\"],\n                                \"seq\", \"{0}.fa\".format(self._custom[\"base\"]))\n        assert os.path.exists(base_seq)\n        mask_file = os.path.basename(self._custom[\"mask\"])\n        ready_mask = apply(\"{0}-complement{1}\".format, os.path.splitext(mask_file))\n        out_fasta = \"{0}.fa\".format(self._custom[\"dbkey\"])\n        if not os.path.exists(os.path.join(seq_dir, out_fasta)):\n            if not os.path.exists(mask_file):\n                shared._remote_fetch(None, self._custom[\"mask\"])\n            if not os.path.exists(ready_mask):\n                subprocess.check_call(\"bedtools complement -i {i} -g {g}.fai > {o}\".format(\n                    i=mask_file, g=base_seq, o=ready_mask), shell=True)\n            if not os.path.exists(out_fasta):\n                subprocess.check_call(\"bedtools maskfasta -fi {fi} -bed {bed} -fo {fo}\".format(\n                    fi=base_seq, bed=ready_mask, fo=out_fasta), shell=True)\n        return out_fasta, [mask_file, ready_mask]\n\ndef _prep_custom_genome(custom, genomes, genome_indexes, env):\n    \"\"\"Prepare a custom genome derived from existing genome.\n    Allows creation of masked genomes for specific purposes.\n    \"\"\"\n    cur_org = None\n    cur_manager = None\n    for org, gid, manager in genomes:\n        if gid == custom[\"base\"]:\n            cur_org = org\n            cur_manager = manager\n            break\n    assert cur_org is not None\n    _data_ngs_genomes(env, [[cur_org, custom[\"dbkey\"],\n                             CustomMaskManager(custom, cur_manager.config)]],\n                      genome_indexes)\n\ndef _clean_genome_directory():\n    \"\"\"Remove any existing sequence information in the current directory.\n    \"\"\"\n    for dirname in GENOME_INDEXES_SUPPORTED + DEFAULT_GENOME_INDEXES:\n        if os.path.exists(dirname):\n            subprocess.check_call(\"rm -rf %s\" % dirname, shell=True)\n\ndef _move_seq_files(ref_file, base_zips, seq_dir):\n    if not os.path.exists(seq_dir):\n        subprocess.check_call('mkdir %s' % seq_dir, shell=True)\n    for move_file in [ref_file] + base_zips:\n        if os.path.exists(move_file):\n            subprocess.check_call(\"mv %s %s\" % (move_file, seq_dir), shell=True)\n    path, fname = os.path.split(ref_file)\n    moved_ref = os.path.join(path, seq_dir, fname)\n    assert os.path.exists(moved_ref), moved_ref\n    return moved_ref\n\n# ## Indexing for specific aligners\n\ndef _index_w_command(env, dir_name, command, ref_file, pre=None, post=None, ext=None):\n    \"\"\"Low level function to do the indexing and paths with an index command.\n    \"\"\"\n    path_export = _get_path_export(env)\n    index_name = os.path.splitext(os.path.basename(ref_file))[0]\n    if ext is not None: index_name += ext\n    full_ref_path = os.path.join(os.pardir, ref_file)\n    if not os.path.exists(dir_name):\n        subprocess.check_call(\"mkdir %s\" % dir_name, shell=True)\n        with shared.chdir(dir_name):\n            if pre:\n                full_ref_path = pre(full_ref_path)\n            subprocess.check_call(path_export + command.format(ref_file=full_ref_path, index_name=index_name),\n                                  shell=True)\n            if post:\n                post(full_ref_path)\n    return os.path.join(dir_name, index_name)\n\n@_if_installed(\"faToTwoBit\")\ndef _index_twobit(env, ref_file):\n    \"\"\"Index reference files using 2bit for random access.\n    \"\"\"\n    dir_name = \"ucsc\"\n    cmd = \"faToTwoBit {ref_file} {index_name}\"\n    return _index_w_command(env, dir_name, cmd, ref_file)\n\ndef _index_bowtie(env, ref_file):\n    dir_name = \"bowtie\"\n    cmd = \"bowtie-build -f {ref_file} {index_name}\"\n    return _index_w_command(env, dir_name, cmd, ref_file)\n\ndef _index_bowtie2(env, ref_file):\n    dir_name = \"bowtie2\"\n    cmd = \"bowtie2-build {ref_file} {index_name}\"\n    out_suffix = _index_w_command(env, dir_name, cmd, ref_file)\n    bowtie_link = os.path.normpath(os.path.join(os.path.dirname(ref_file), os.path.pardir,\n                                                out_suffix + \".fa\"))\n    relative_ref_file = os.path.relpath(ref_file, os.path.dirname(bowtie_link))\n    if not os.path.exists(bowtie_link):\n        subprocess.check_call(\"ln -sf %s %s\" % (relative_ref_file, bowtie_link), shell=True)\n    return out_suffix\n\ndef _index_bwa(env, ref_file):\n    dir_name = \"bwa\"\n    local_ref = os.path.split(ref_file)[-1]\n    if not os.path.exists(os.path.join(dir_name, \"%s.bwt\" % local_ref)):\n        subprocess.check_call(\"mkdir -p %s\" % dir_name, shell=True)\n        with shared.chdir(dir_name):\n            subprocess.check_call(\"ln -sf %s\" % os.path.join(os.pardir, ref_file), shell=True)\n            try:\n                subprocess.check_call(\"bwa index -a bwtsw %s\" % local_ref, shell=True)\n            except subprocess.CalledProcessError:\n                # work around a bug in bwa indexing for small files\n                subprocess.check_call(\"bwa index %s\" % local_ref, shell=True)\n            subprocess.check_call(\"rm -f %s\" % local_ref, shell=True)\n    return os.path.join(dir_name, local_ref)\n\ndef _index_bbmap(env, ref_file):\n    dir_name = \"bbmap\"\n    try:\n        cores = env.cores\n    except:\n        cores = 1\n    if not os.path.exists(os.path.join(dir_name, \"ref\", \"genome\", \"1\", \"summary.txt\")):\n        subprocess.check_call(\"mkdir -p %s\" % dir_name, shell=True)\n        subprocess.check_call(\"bbmap.sh -Xms%sg -Xmx24g path=%s ref=%s\" %\n                              (cores, dir_name, ref_file), shell=True)\n    return dir_name\n\ndef _index_bismark(env, ref_file):\n    dir_name = \"bismark\"\n    subprocess.check_call(\"mkdir -p %s\" % dir_name, shell=True)\n    out_dir = os.path.join(dir_name, \"Bisulfite_Genome\")\n    if os.path.exists(out_dir):\n        return out_dir\n    with shared.chdir(dir_name):\n        local = os.path.basename(ref_file)\n        subprocess.check_call(\"ln -sf {0} {1}\".format(ref_file, local), shell=True)\n        cmd= f\"bismark_genome_preparation .\"\n        subprocess.check_call(cmd, shell=True)\n    return out_dir\n\ndef _index_maq(env, ref_file):\n    dir_name = \"maq\"\n    cmd = \"maq fasta2bfa {ref_file} {index_name}\"\n    def link_local(ref_file):\n        local = os.path.basename(ref_file)\n        subprocess.check_call(\"ln -sf {0} {1}\".format(ref_file, local), shell=True)\n        return local\n    def rm_local(local_file):\n        subprocess.check_call(\"rm -f {0}\".format(local_file), shell=True)\n    return _index_w_command(env, dir_name, cmd, ref_file, pre=link_local, post=rm_local)\n\ndef _index_minimap2(env, ref_file):\n    dir_name = \"minimap2\"\n    indexes = []\n    for preset in [\"sr\"]:\n        index_name = \"%s-%s.mmi\" % (os.path.splitext(os.path.basename(ref_file))[0], preset)\n        cmd = \"minimap2 -x %s -d %s {ref_file}\" % (preset, index_name)\n        out_basename = _index_w_command(env, dir_name, cmd, ref_file)\n        indexes.append(os.path.join(os.path.dirname(out_basename), index_name))\n    return indexes[0]\n\n@_if_installed(\"novoindex\")\ndef _index_novoalign(env, ref_file):\n    dir_name = \"novoalign\"\n    cmd = \"novoindex {index_name} {ref_file}\"\n    return _index_w_command(env, dir_name, cmd, ref_file)\n\n@_if_installed(\"novoalignCS\")\ndef _index_novoalign_cs(env, ref_file):\n    dir_name = \"novoalign_cs\"\n    cmd = \"novoindex -c {index_name} {ref_file}\"\n    return _index_w_command(env, dir_name, cmd, ref_file)\n\ndef _index_sam(env, ref_file):\n    (ref_dir, local_file) = os.path.split(ref_file)\n    with shared.chdir(ref_dir):\n        if not os.path.exists(\"%s.fai\" % local_file):\n            subprocess.check_call(\"samtools faidx %s\" % local_file, shell=True)\n    galaxy.index_picard(ref_file)\n    return ref_file\n\n@_if_installed(\"STAR\")\ndef _index_star(env, ref_file):\n    (ref_dir, local_file) = os.path.split(ref_file)\n    build = os.path.basename(os.path.splitext(ref_file)[0])\n    dir_name = os.path.normpath(os.path.join(ref_dir, os.pardir, \"star\"))\n    sentinel_file = os.path.join(dir_name, \"SA\")\n    if os.path.exists(sentinel_file):\n        return dir_name \n    if build == \"hg38\":\n        simple_file = os.path.splitext(ref_file)[0] + \"-simple.fa\"\n        print(f\"hg38 detected, building a simple reference with no alts, decoys or HLA from {ref_file} to {simple_file}.\")\n        ref_file = prepare_simple_reference(ref_file, simple_file)\n    GenomeLength = os.path.getsize(ref_file)\n    Nbases = int(round(min(14, log(GenomeLength, 2) / 2 - 2), 0))\n    # if there is a large number of contigs, scale nbits down\n    # https://github.com/alexdobin/STAR/issues/103#issuecomment-173009628\n    # if there is a small genome, scale nbits down\n    # https://groups.google.com/forum/#!topic/rna-star/9g8Uoe1Igho\n    cmd = 'grep \">\" {ref_file} | wc -l'.format(ref_file=ref_file)\n    nrefs = float(subprocess.check_output(cmd, shell=True).decode())\n    nbits = int(round(min(14, log(GenomeLength / nrefs, 2), log(GenomeLength, 2) / 2 - 1)))\n    # first we estimate the number of bits we need to hold the genome and allocate\n    # double that plus some padding to build the index\n    mem = ((GenomeLength + 1) / nbits + 1) * nbits\n    mem = (mem + 10000) * 2\n    mem = mem + mem / 3\n    mem = max(mem, 30000000000)\n    try:\n        cpu = env.cores\n    except:\n        cpu = 1\n    print(f\"Preparing STAR index from {ref_file}.\") \n    cmd = (\"STAR --genomeDir %s --genomeFastaFiles {ref_file} \"\n           \"--runThreadN %s \"\n           \"--limitGenomeGenerateRAM %s \"\n           \"--genomeChrBinNbits %s \"\n           \"--runMode genomeGenerate \"\n           \"--genomeSAindexNbases %s\" % (dir_name, str(cpu), str(mem), Nbases,\n                                         nbits))\n    if not os.path.exists(os.path.join(dir_name, \"SA\")):\n        _index_w_command(env, dir_name, cmd, ref_file)\n    if build == \"hg38\":\n        print(f\"Removing {ref_file}.\")\n        os.remove(ref_file)\n    return dir_name\n\n@_if_installed(\"hisat2-build\")\ndef _index_hisat2(env, ref_file):\n    path_export = _get_path_export(env)\n    build = os.path.splitext(os.path.basename(ref_file))[0]\n    (ref_dir, local_file) = os.path.split(ref_file)\n    gtf_file = os.path.join(ref_dir, os.pardir, \"rnaseq\", \"ref-transcripts.gtf\")\n    dir_name = os.path.normpath(os.path.join(ref_dir, os.pardir, \"hisat2\"))\n    index_prefix = os.path.join(dir_name, build)\n    if os.path.exists(os.path.join(index_prefix + \".1.ht2\")):\n        return dir_name\n    if not os.path.exists(dir_name):\n        subprocess.check_call('mkdir -p %s' % dir_name, shell=True)\n    try:\n        cpu = env.cores\n    except:\n        cpu = 1\n    cmd = \"{path_export}hisat2-build -p {cpu} \"\n\n    exons_file = index_prefix + \".exons\"\n    splicesites_file = index_prefix + \".splicesites\"\n    if os.path.exists(gtf_file):\n        if not os.path.exists(exons_file):\n            with open(exons_file, \"w\") as out_handle:\n                exons_cmd = [\"hisat2_extract_exons.py\", gtf_file]\n                subprocess.check_call(path_export + \" \".join(exons_cmd), stdout=out_handle, shell=True)\n        if not os.path.exists(splicesites_file):\n            with open(splicesites_file, \"w\") as out_handle:\n                splicesites_cmd = [\"hisat2_extract_splice_sites.py\", gtf_file]\n                subprocess.check_call(path_export + \" \".join(splicesites_cmd), stdout=out_handle, shell=True)\n\n        if os.stat(exons_file).st_size > 0 and os.stat(splicesites_file).st_size > 0:\n            cmd += \"--exon {exons_file} --ss {splicesites_file} \"\n    cmd += \"{ref_file} {index_prefix} \"\n    if not os.path.exists(os.path.join(index_prefix + \".1.ht2\")):\n        subprocess.check_call(cmd.format(**locals()), shell=True)\n    return dir_name\n\ndef _index_snap(env, ref_file):\n    \"\"\"Snap indexing is computationally expensive. Requests all cores and 64Gb of memory.\n    \"\"\"\n    dir_name = \"snap\"\n    index_name = os.path.splitext(os.path.basename(ref_file))[0]\n    org_arg = \"-hg19\" if index_name in [\"hg19\", \"GRCh37\"] else \"\"\n    cmd = \"snap-aligner index {ref_file} {dir_name} -bSpace {org_arg}\"\n    if not os.path.exists(os.path.join(dir_name, \"GenomeIndex\")):\n        subprocess.check_call(cmd.format(**locals()), shell=True)\n    return dir_name\n\ndef _get_path_export(env):\n    \"\"\"Ensure PATH points to local install directory.\n    \"\"\"\n    path_export = \"\"\n    if hasattr(env, \"system_install\") and env.system_install:\n        local_bin = os.path.join(env.system_install, 'bin')\n        if os.path.exists(local_bin):\n            path_export = \"export PATH=%s:$PATH && \" % local_bin\n    return path_export\n\ndef _index_rtg(env, ref_file):\n    \"\"\"Perform indexing for use with Real Time Genomics tools.\n\n    https://github.com/RealTimeGenomics/rtg-tools\n    \"\"\"\n    path_export = _get_path_export(env)\n    dir_name = \"rtg\"\n    index_name = \"%s.sdf\" % os.path.splitext(os.path.basename(ref_file))[0]\n    if not os.path.exists(os.path.join(dir_name, index_name, \"done\")):\n        cmd = (\"{path_export}export RTG_JAVA_OPTS='-Xms1g' && export RTG_MEM=2g && \"\n               \"rtg format -o {dir_name}/{index_name} {ref_file}\")\n        subprocess.check_call(cmd.format(**locals()), shell=True)\n    return dir_name\n\n@_if_installed(\"MosaikJump\")\ndef _index_mosaik(env, ref_file):\n    hash_size = 15\n    dir_name = \"mosaik\"\n    cmd = \"MosaikBuild -fr {ref_file} -oa {index_name}\"\n    def create_jumpdb(ref_file):\n        jmp_base = os.path.splitext(os.path.basename(ref_file))[0]\n        dat_file = \"{0}.dat\".format(jmp_base)\n        if not os.path.exists(\"{0}_keys.jmp\".format(jmp_base)):\n            cmd = \"export MOSAIK_TMP=`pwd` && MosaikJump -hs {hash_size} -ia {ref_file} -out {index_name}\".format(\n                hash_size=hash_size, ref_file=dat_file, index_name=jmp_base)\n            subprocess.check_call(cmd, shell=True)\n    return _index_w_command(env, dir_name, cmd, ref_file,\n                            post=create_jumpdb, ext=\".dat\")\n\n# -- Retrieve using GGD recipes\n\ndef _install_with_ggd(env, manager, gid, recipe):\n    recipe_dir = os.path.normpath(os.path.join(os.path.dirname(__file__),\n                                               os.pardir, os.pardir, \"ggd-recipes\"))\n    recipe_file = os.path.join(recipe_dir, gid, \"%s.yaml\" % recipe)\n    if os.path.exists(recipe_file):\n        ggd.install_recipe(os.getcwd(), env.system_install, recipe_file, gid)\n    else:\n        raise NotImplementedError(\"GGD recipe not available for %s %s\" % (gid, recipe))\n\n# -- Genome upload and download to Amazon s3 buckets\n\ndef _download_s3_index(env, manager, gid, idx):\n    print(\"Downloading genome from s3: {0} {1}\".format(gid, idx))\n    url = \"https://s3.amazonaws.com/biodata/genomes/%s-%s.tar.xz\" % (gid, idx)\n    if gid in [\"GRCh37\", \"hg19\", \"mm10\"] and idx in [\"bowtie2\", \"bwa\", \"novoalign\"]:\n        out_file = shared._remote_fetch(env, url, samedir=True)\n        subprocess.check_call(\"xz -dc %s | tar -xvpf -\" % out_file, shell=True)\n        subprocess.check_call(\"rm -f %s\" % out_file, shell=True)\n    else:\n        raise NotImplementedError(\"No pre-computed indices for %s %s\" % (gid, idx))\n\ndef _download_genomes(env, genomes, genome_indexes):\n    \"\"\"Download a group of genomes from Amazon s3 bucket.\n    \"\"\"\n    genome_dir = _make_genome_dir(env.data_files)\n    for (orgname, gid, manager) in genomes:\n        org_dir = os.path.join(genome_dir, orgname, gid)\n        if not os.path.exists(org_dir):\n            subprocess.check_call('mkdir -p %s' % org_dir, shell=True)\n        for idx in genome_indexes:\n            with shared.chdir(org_dir):\n                if not os.path.exists(idx):\n                    _download_s3_index(env, manager, gid, idx)\n        ref_file = os.path.join(org_dir, \"seq\", \"%s.fa\" % gid)\n        if not os.path.exists(ref_file):\n            ref_file = os.path.join(org_dir, \"seq\", \"%s.fa\" % manager._name)\n        assert os.path.exists(ref_file), ref_file\n        cur_indexes = manager.config.get(\"indexes\", genome_indexes)\n        _index_to_galaxy(env, org_dir, ref_file, gid, cur_indexes, manager.config)\n\ndef _upload_genomes(env, genomes, genome_indexes):\n    \"\"\"Upload our configured genomes to Amazon s3 bucket.\n    \"\"\"\n    conn = boto.connect_s3()\n    bucket = conn.create_bucket(\"biodata\")\n    genome_dir = os.path.join(env.data_files, \"genomes\")\n    for (orgname, gid, _) in genomes:\n        cur_dir = os.path.join(genome_dir, orgname, gid)\n        _clean_directory(cur_dir, gid)\n        for idx in genome_indexes:\n            idx_dir = os.path.join(cur_dir, idx)\n            tarball = _tar_directory(idx_dir, \"%s-%s\" % (gid, idx))\n            _upload_to_s3(tarball, bucket)\n    bucket.make_public()\n\ndef _upload_to_s3(tarball, bucket):\n    \"\"\"Upload the genome tarball to s3.\n    \"\"\"\n    upload_script = os.path.join(os.path.dirname(__file__), os.pardir, os.pardir,\n                                 \"utils\", \"s3_multipart_upload.py\")\n    s3_key_name = os.path.join(\"genomes\", os.path.basename(tarball))\n    if not bucket.get_key(s3_key_name):\n        gb_size = int(subprocess.check_output(\"du -sm %s\" % tarball, shell=True).decode().split()[0]) / 1000.0\n        print(\"Uploading %s %.1fGb\" % (s3_key_name, gb_size))\n        cl = [\"python\", upload_script, tarball, bucket.name, s3_key_name, \"--public\"]\n        subprocess.check_call(cl)\n\ndef _tar_directory(dir, tar_name):\n    \"\"\"Create a tarball of the directory.\n    \"\"\"\n    base_dir, tar_dir = os.path.split(dir)\n    tarball = os.path.join(base_dir, \"%s.tar.xz\" % tar_name)\n    if not os.path.exists(tarball):\n        with shared.chdir(base_dir):\n            subprocess.check_call(\"tar -cvpf - %s | xz -zc - > %s\" %\n                                  (tar_dir, os.path.basename(tarball)), shell=True)\n    return tarball\n\ndef _clean_directory(dir, gid):\n    \"\"\"Clean duplicate files from directories before tar and upload.\n    \"\"\"\n    # get rid of softlinks\n    bowtie_ln = os.path.join(dir, \"bowtie\", \"%s.fa\" % gid)\n    maq_ln = os.path.join(dir, \"maq\", \"%s.fa\" % gid)\n    for to_remove in [bowtie_ln, maq_ln]:\n        if os.path.exists(to_remove):\n            subprocess.check_call(\"rm -f %s\" % to_remove, shell=True)\n    # remove any downloaded original sequence files\n    remove_exts = [\"*.gz\", \"*.zip\"]\n    with shared.chdir(os.path.join(dir, \"seq\")):\n        for rext in remove_exts:\n            fnames = subprocess.check_output(\"find . -name '%s'\" % rext, shell=True).decode()\n            for fname in (f.strip() for f in fnames.split(\"\\n\") if f.strip()):\n                subprocess.check_call(\"rm -f %s\" % fname, shell=True)\n\n# == Liftover files\n\ndef _data_liftover(env, lift_over_genomes):\n    \"\"\"Download chain files for running liftOver.\n\n    Does not install liftOver binaries automatically.\n    \"\"\"\n    lo_dir = os.path.join(env.data_files, \"liftOver\")\n    if not os.path.exists(lo_dir):\n        subprocess.check_call(\"mkdir %s\" % lo_dir, shell=True)\n    lo_base_url = \"ftp://hgdownload.cse.ucsc.edu/goldenPath/%s/liftOver/%s\"\n    lo_base_file = \"%sTo%s.over.chain.gz\"\n    for g1 in lift_over_genomes:\n        for g2 in [g for g in lift_over_genomes if g != g1]:\n            g2u = g2[0].upper() + g2[1:]\n            cur_file = lo_base_file % (g1, g2u)\n            non_zip = os.path.splitext(cur_file)[0]\n            worked = False\n            with shared.chdir(lo_dir):\n                if not os.path.exists(non_zip):\n                    result = shared._remote_fetch(env, \"%s\" % (lo_base_url % (g1, cur_file)), allow_fail=True)\n                    # Lift over back and forths don't always exist\n                    # Only move forward if we found the file\n                    if result:\n                        worked = True\n                        subprocess.check_call(\"gunzip %s\" % result, shell=True)\n            if worked:\n                ref_parts = [g1, g2, os.path.join(lo_dir, non_zip)]\n                galaxy.update_loc_file(env, \"liftOver.loc\", ref_parts)\n\n# == UniRef\ndef _data_uniref(env):\n    \"\"\"Retrieve and index UniRef databases for protein searches.\n\n    http://www.ebi.ac.uk/uniref/\n\n    These are currently indexed for FASTA searches. Are other indexes desired?\n    Should this be separated out and organized by program like genome data?\n    This should also check the release note and automatically download and\n    replace older versions.\n    \"\"\"\n    site = \"ftp://ftp.uniprot.org\"\n    base_url = site + \"/pub/databases/uniprot/\" \\\n               \"current_release/uniref/%s/%s\"\n    for uniref_db in [\"uniref50\", \"uniref90\", \"uniref100\"]:\n        work_dir = os.path.join(env.data_files, \"uniref\", uniref_db)\n        if not os.path.exists(work_dir):\n            subprocess.check_call(\"mkdir -p %s\" % work_dir, shell=True)\n        base_work_url = base_url % (uniref_db, uniref_db)\n        fasta_url = base_work_url + \".fasta.gz\"\n        base_file = os.path.splitext(os.path.basename(fasta_url))[0]\n        with shared.chdir(work_dir):\n            if not os.path.exists(base_file):\n                out_file = shared._remote_fetch(env, fasta_url)\n                subprocess.check_call(\"gunzip %s\" % out_file, shell=True)\n                shared._remote_fetch(env, base_work_url + \".release_note\")\n        _index_blast_db(work_dir, base_file, \"prot\")\n\ndef _index_blast_db(work_dir, base_file, db_type):\n    \"\"\"Index a database using blast+ for similary searching.\n    \"\"\"\n    type_to_ext = dict(prot=(\"phr\", \"pal\"), nucl=(\"nhr\", \"nal\"))\n    db_name = os.path.splitext(base_file)[0]\n    with shared.chdir(work_dir):\n        if not reduce(operator.or_,\n                      (os.path.exists(\"%s.%s\" % (db_name, ext)) for ext in type_to_ext[db_type])):\n            subprocess.check_call(\"makeblastdb -in %s -dbtype %s -out %s\" %\n                                  (base_file, db_type, db_name), shell=True)\ndef get_index_fn(index):\n    \"\"\"\n    return the index function for an index, if it is missing return a function\n    that is a no-op\n    \"\"\"\n    def noop(env, ref_file):\n        pass\n    return INDEX_FNS.get(index, noop)\n\ndef prepare_simple_reference(ref_file, out_file):\n    \"\"\"\n    given an hg38 FASTA file, create a FASTA file with no alts, HLA or decoys\n    \"\"\"\n    if os.path.exists(out_file):\n        return out_file\n    with open(ref_file + \".fai\") as in_handle:\n        chroms = [x.split()[0].strip() for x in in_handle]\n    chroms = [x for x in chroms if not (is_alt(x) or is_decoy(x) or is_HLA(x))]\n    cmd = [\"samtools\", \"faidx\", ref_file] + chroms\n    with open(out_file, \"w\") as out_handle:\n        subprocess.check_call(cmd, stdout=out_handle)\n    return out_file\n\ndef is_alt(chrom):\n    return chrom.endswith(\"_alt\")\n\ndef is_decoy(chrom):\n    return chrom.endswith(\"_decoy\")\n\ndef is_HLA(chrom):\n    return chrom.startswith(\"HLA\")\n\nINDEX_FNS = {\n    \"seq\": _index_sam,\n    \"bbmap\": _index_bbmap,\n    \"bismark\": _index_bismark,\n    \"bwa\": _index_bwa,\n    \"bowtie\": _index_bowtie,\n    \"bowtie2\": _index_bowtie2,\n    \"maq\": _index_maq,\n    \"mosaik\": _index_mosaik,\n    \"minimap2\": _index_minimap2,\n    \"novoalign\": _index_novoalign,\n    \"novoalign_cs\": _index_novoalign_cs,\n    \"ucsc\": _index_twobit,\n    \"twobit\": _index_twobit,\n    \"star\": _index_star,\n    \"snap\": _index_snap,\n    \"rtg\": _index_rtg,\n    \"hisat2\": _index_hisat2\n    }\n"
  },
  {
    "path": "cloudbio/biodata/ggd.py",
    "content": "\"\"\"Process GGD (Get Genomics Data) configurations for installation in biodata directories.\n\nBuilds off work done by Aaron Quinlan to define and install genomic data:\n\nhttps://github.com/arq5x/ggd\n\"\"\"\nfrom __future__ import print_function\nimport collections\nimport contextlib\nfrom distutils.version import LooseVersion\nimport os\nimport shutil\nimport subprocess\n\nimport yaml\n\ndef install_recipe(base_dir, system_install, recipe_file, genome_build):\n    \"\"\"Install data in a biodata directory given instructions from GGD YAML recipe.\n    \"\"\"\n    if not os.path.exists(base_dir):\n        os.makedirs(base_dir)\n    recipe = _read_recipe(recipe_file)\n    if not version_uptodate(base_dir, recipe):\n        if _has_required_programs(recipe[\"recipe\"][\"full\"].get(\"required\", [])):\n            with tx_tmpdir(base_dir) as tmpdir:\n                with chdir(tmpdir):\n                    print(\"Running GGD recipe: %s %s %s\" % (genome_build, recipe[\"attributes\"][\"name\"],\n                                                            recipe[\"attributes\"][\"version\"]))\n                    _run_recipe(tmpdir, recipe[\"recipe\"][\"full\"][\"recipe_cmds\"],\n                                recipe[\"recipe\"][\"full\"][\"recipe_type\"], system_install)\n                _move_files(tmpdir, base_dir, recipe[\"recipe\"][\"full\"][\"recipe_outfiles\"])\n            add_version(base_dir, recipe)\n\ndef _has_required_programs(programs):\n    \"\"\"Ensure the provided programs exist somewhere in the current PATH.\n\n    http://stackoverflow.com/questions/377017/test-if-executable-exists-in-python\n    \"\"\"\n    def is_exe(fpath):\n        return os.path.isfile(fpath) and os.access(fpath, os.X_OK)\n\n    for p in programs:\n        found = False\n        for path in os.environ[\"PATH\"].split(os.pathsep):\n            path = path.strip('\"')\n            exe_file = os.path.join(path, p)\n            if is_exe(exe_file):\n                found = True\n                break\n        if not found:\n            return False\n    return True\n\ndef _run_recipe(work_dir, recipe_cmds, recipe_type, system_install):\n    \"\"\"Create a bash script and run the recipe to download data.\n    \"\"\"\n    assert recipe_type == \"bash\", \"Can only currently run bash recipes\"\n    run_file = os.path.join(work_dir, \"ggd-run.sh\")\n    with open(run_file, \"w\") as out_handle:\n        out_handle.write(\"#!/bin/bash\\nset -eu -o pipefail\\nexport PATH=%s/bin:$PATH\\n\" % system_install)\n        out_handle.write(\"\\n\".join(recipe_cmds))\n    subprocess.check_output([\"bash\", run_file])\n\ndef _move_files(tmp_dir, final_dir, targets):\n    for target in targets:\n        if os.path.isdir(os.path.join(tmp_dir, target)):\n            out_files = [os.path.join(target, f) for f in os.listdir(os.path.join(tmp_dir, target))]\n        else:\n            out_files = [target]\n        for out_file in out_files:\n            orig = os.path.join(tmp_dir, out_file)\n            final = os.path.join(final_dir, out_file)\n            assert os.path.exists(orig), (\"Did not find expected output file %s in %s\" %\n                                          (out_file, tmp_dir))\n            cur_dir = os.path.dirname(final)\n            if not os.path.exists(cur_dir):\n                os.makedirs(cur_dir)\n            if os.path.exists(final):\n                os.remove(final)\n            shutil.move(orig, final)\n\ndef _read_recipe(in_file):\n    in_file = os.path.abspath(os.path.expanduser(in_file))\n    with open(in_file) as in_handle:\n        recipe = yaml.safe_load(in_handle)\n    return recipe\n\n# ## Versioning\n\ndef version_uptodate(base_dir, recipe):\n    \"\"\"Check if we have an up to date GGD installation in this directory.\n    \"\"\"\n    versions = _get_versions(base_dir)\n    return (recipe[\"attributes\"][\"name\"] in versions and\n            LooseVersion(versions[recipe[\"attributes\"][\"name\"]]) >=\n            LooseVersion(str(recipe[\"attributes\"][\"version\"])))\n\ndef add_version(base_dir, recipe):\n    versions = _get_versions(base_dir)\n    versions[recipe[\"attributes\"][\"name\"]] = recipe[\"attributes\"][\"version\"]\n    with open(_get_version_file(base_dir), \"w\") as out_handle:\n        for n, v in versions.items():\n            out_handle.write(\"%s,%s\\n\" % (n, v))\n\ndef _get_versions(base_dir):\n    version_file = _get_version_file(base_dir)\n    versions = collections.OrderedDict()\n    if os.path.exists(version_file):\n        with open(version_file) as in_handle:\n            for line in in_handle:\n                name, version = line.strip().split(\",\")\n                versions[name] = version\n    return versions\n\ndef _get_version_file(base_dir):\n    return os.path.join(base_dir, \"versions.csv\")\n\n# ## Transactional utilities\n\n@contextlib.contextmanager\ndef tx_tmpdir(base_dir):\n    \"\"\"Context manager to create and remove a transactional temporary directory.\n    \"\"\"\n    tmp_dir = os.path.join(base_dir, \"txtmp\")\n    if not os.path.exists(tmp_dir):\n        os.makedirs(tmp_dir)\n    yield tmp_dir\n    shutil.rmtree(tmp_dir, ignore_errors=True)\n\n@contextlib.contextmanager\ndef chdir(new_dir):\n    \"\"\"Context manager to temporarily change to a new directory.\n\n    http://lucentbeing.com/blog/context-managers-and-the-with-statement-in-python/\n    \"\"\"\n    cur_dir = os.getcwd()\n    os.chdir(new_dir)\n    try:\n        yield\n    finally:\n        os.chdir(cur_dir)\n"
  },
  {
    "path": "cloudbio/biodata/rnaseq.py",
    "content": "\"\"\"Infrastructure for RNA-seq supporting files.\n\"\"\"\nimport os\nimport subprocess\n\nfrom cloudbio.custom import shared\n\ndef finalize(genomes, data_filedir):\n    \"\"\"Provide symlinks back to reference genomes so tophat avoids generating FASTA genomes.\n    \"\"\"\n    genome_dir = os.path.join(data_filedir, \"genomes\")\n    for (orgname, gid, manager) in genomes:\n        org_dir = os.path.join(genome_dir, orgname)\n        for aligner in [\"bowtie\", \"bowtie2\"]:\n            aligner_dir = os.path.join(org_dir, gid, aligner)\n            if os.path.exists(aligner_dir):\n                with shared.chdir(aligner_dir):\n                    for ext in [\"\", \".fai\"]:\n                        orig_seq = os.path.join(os.pardir, \"seq\", \"%s.fa%s\" % (gid, ext))\n                        if os.path.exists(orig_seq) and not os.path.exists(os.path.basename(orig_seq)):\n                            subprocess.check_call(\"ln -sf %s\" % orig_seq, shell=True)\n\ndef cleanup(genomes, data_filedir):\n    \"\"\"Cleanup for GGD recipe installation, removing old rnaseq symlinks.\n    \"\"\"\n    folder_name = \"rnaseq\"\n    genome_dir = os.path.join(data_filedir, \"genomes\")\n    for (orgname, gid, manager) in genomes:\n        org_dir = os.path.join(genome_dir, orgname)\n        target_dir = os.path.join(org_dir, gid, folder_name)\n        if os.path.lexists(target_dir) and os.path.islink(target_dir):\n            os.remove(target_dir)\n"
  },
  {
    "path": "cloudbio/cloudbiolinux.py",
    "content": "\"\"\"CloudBioLinux specific scripts\n\"\"\"\nimport os\nfrom fabric.api import *\nfrom fabric.contrib.files import *\n\nfrom cloudbio.custom import shared\n\ndef _freenx_scripts(env):\n    \"\"\"Provide graphical access to clients via FreeNX.\n    \"\"\"\n    home_dir = env.safe_run_output(\"echo $HOME\")\n    setup_script = \"setupnx.sh\"\n    bin_dir = shared._get_bin_dir(env)\n    install_file_dir = os.path.join(env.config_dir, os.pardir, \"installed_files\")\n    if not env.safe_exists(os.path.join(bin_dir, setup_script)):\n        env.safe_put(os.path.join(install_file_dir, setup_script),\n                     os.path.join(home_dir, setup_script))\n        env.safe_run(\"chmod 0777 %s\" % os.path.join(home_dir, setup_script))\n        env.safe_sudo(\"mv %s %s\" % (os.path.join(home_dir, setup_script), bin_dir))\n    remote_login = \"configure_freenx.sh\"\n    if not env.safe_exists(os.path.join(home_dir, remote_login)):\n        env.safe_put(os.path.join(install_file_dir, 'bash_login'), os.path.join(home_dir, remote_login))\n        env.safe_run(\"chmod 0777 %s\" % os.path.join(home_dir, remote_login))\n    _configure_gnome(env)\n\ndef _cleanup_space(env):\n    \"\"\"Cleanup to recover space from builds and packages.\n    \"\"\"\n    env.logger.info(\"Cleaning up space from package builds\")\n    with settings(warn_only=True):\n        env.safe_sudo(\"rm -rf .cpanm\")\n        env.safe_sudo(\"rm -f /var/crash/*\")\n        env.safe_run(\"rm -f ~/*.dot\")\n        env.safe_run(\"rm -f ~/*.log\")\n\ndef _configure_gnome(env):\n    \"\"\"Configure NX server to use classic GNOME.\n\n    http://askubuntu.com/questions/50503/why-do-i-get-unity-instead-of-classic-when-using-nx\n    http://notepad2.blogspot.com/2012/04/install-freenx-server-on-ubuntu-1110.html\n    \"\"\"\n    add = 'COMMAND_START_GNOME=\"gnome-session --session gnome-fallback\"'\n    fname = \"/etc/nxserver/node.conf\"\n    if env.safe_exists(\"/etc/nxserver/\"):\n        env.safe_append(fname, add, use_sudo=True)\n"
  },
  {
    "path": "cloudbio/cloudman.py",
    "content": "\"\"\"Build instructions associated with CloudMan.\n\nhttp://wiki.g2.bx.psu.edu/Admin/Cloud\n\nAdapted from Enis Afgan's code: https://bitbucket.org/afgane/mi-deployment\n\"\"\"\n\ncm_upstart = \"\"\"\ndescription     \"Start CloudMan contextualization script\"\n\nstart on runlevel [2345]\n\ntask\nexec python %s 2> %s.log\n\"\"\"\nimport os\n\nfrom fabric.api import sudo, cd, run, put\nfrom fabric.contrib.files import exists, settings\n\nfrom cloudbio.galaxy import _setup_users\nfrom cloudbio.flavor.config import get_config_file\nfrom cloudbio.package.shared import _yaml_to_packages\nfrom cloudbio.custom.shared import (_make_tmp_dir, _write_to_file, _get_install,\n                                    _configure_make, _if_not_installed,\n                                    _setup_conf_file, _add_to_profiles,\n                                    _create_python_virtualenv,\n                                    _setup_simple_service,\n                                    _read_boolean)\nfrom cloudbio.package.deb import (_apt_packages, _setup_apt_automation)\n\nMI_REPO_ROOT_URL = \"https://bitbucket.org/afgane/mi-deployment/raw/tip\"\nCM_REPO_ROOT_URL = \"https://bitbucket.org/galaxy/cloudman/raw/tip\"\n\n\ndef _configure_cloudman(env, use_repo_autorun=False):\n    \"\"\"\n    Configure the machine to be capable of running CloudMan.\n\n    ..Also see: ``custom/cloudman.py``\n    \"\"\"\n    env.logger.debug(\"Configuring CloudMan\")\n    _setup_users(env)\n    _setup_env(env)\n    _configure_logrotate(env)\n    _configure_ec2_autorun(env, use_repo_autorun)\n    _configure_sge(env)\n    _configure_hadoop(env)\n    _configure_nfs(env)\n    _configure_novnc(env)\n    _configure_desktop(env)\n    install_s3fs(env)\n\n\ndef _configure_desktop(env):\n    \"\"\"\n    Configure a desktop manager to work with VNC. Note that `xfce4` (or `jwm`)\n    and `vnc4server` packages need to be installed for this to have effect.\n    \"\"\"\n    if not _read_boolean(env, \"configure_desktop\", False):\n        return\n    # Set nginx PAM module to allow logins for any system user\n    if env.safe_exists(\"/etc/pam.d\"):\n        env.safe_sudo('echo \"@include common-auth\" > /etc/pam.d/nginx')\n    env.safe_sudo('usermod -a -G shadow galaxy')\n    # Create a start script for X\n    _setup_conf_file(env, \"/home/ubuntu/.vnc/xstartup\", \"xstartup\", default_source=\"xstartup\")\n    # Create jwmrc config file (uncomment this if using jwm window manager)\n    # _setup_conf_file(env, \"/home/ubuntu/.jwmrc\", \"jwmrc.xml\",\n    #     default_source=\"jwmrc.xml\", mode=\"0644\")\n    env.logger.info(\"----- Done configuring desktop -----\")\n\n\ndef _configure_novnc(env):\n    if not _read_boolean(env, \"configure_novnc\", False):\n        # Longer term would like this enabled by default. -John\n        return\n    if not \"novnc_install_dir\" in env:\n        env.novnc_install_dir = \"/opt/novnc\"\n    if not \"vnc_password\" in env:\n        env.vnc_password = \"cl0udbi0l1nux\"\n    if not \"vnc_user\" in env:\n        env.vnc_user = env.user\n    if not \"vnc_display\" in env:\n        env.vnc_display = \"1\"\n    if not \"vnc_depth\" in env:\n        env.vnc_depth = \"16\"\n    if not \"vnc_geometry\" in env:\n        env.vnc_geometry = \"1024x768\"\n\n    _configure_vncpasswd(env)\n\n    novnc_dir = env.novnc_install_dir\n    env.safe_sudo(\"mkdir -p '%s'\" % novnc_dir)\n    env.safe_sudo(\"chown %s '%s'\" % (env.user, novnc_dir))\n    clone_cmd = \"NOVNC_DIR='%s'; rm -rf $NOVNC_DIR; git clone https://github.com/kanaka/noVNC.git $NOVNC_DIR\" % novnc_dir\n    run(clone_cmd)\n    ## Move vnc_auto.html which takes vnc_password as query argument\n    ## to index.html and rewrite it so that password is autoset, no\n    ## need to specify via query parameter.\n    run(\"sed s/password\\\\ =\\\\ /password\\\\ =\\\\ \\\\\\'%s\\\\\\'\\\\;\\\\\\\\\\\\\\\\/\\\\\\\\\\\\\\\\// '%s/vnc_auto.html' > '%s/index.html'\" % (env.vnc_password, novnc_dir, novnc_dir))\n\n    _setup_conf_file(env, \"/etc/init.d/novnc\", \"novnc_init\", default_source=\"novnc_init\")\n    _setup_conf_file(env, \"/etc/default/novnc\", \"novnc_default\", default_source=\"novnc_default.template\")\n    _setup_conf_file(env, \"/etc/init.d/vncserver\", \"vncserver_init\", default_source=\"vncserver_init\")\n    _setup_conf_file(env, \"/etc/default/vncserver\", \"vncserver_default\", default_source=\"vncserver_default.template\")\n    _setup_simple_service(\"novnc\")\n    _setup_simple_service(\"vncserver\")\n\n\ndef _configure_vncpasswd(env):\n    with cd(\"~\"):\n        run(\"mkdir -p ~/.vnc\")\n        run(\"rm -rf vncpasswd\")\n        run(\"git clone https://github.com/trinitronx/vncpasswd.py vncpasswd\")\n        run(\"python vncpasswd/vncpasswd.py '%s' -f ~/.vnc/passwd\" % env.vnc_password)\n        run(\"chmod 600 ~/.vnc/passwd\")\n        run(\"rm -rf vncpasswd\")\n\n\ndef _setup_env(env):\n    \"\"\"\n    Setup the system environment required to run CloudMan. This means\n    installing required system-level packages (as defined in CBL's\n    ``packages.yaml``, or a flavor thereof) and Python dependencies\n    (i.e., libraries) as defined in CloudMan's ``requirements.txt`` file.\n    \"\"\"\n    # Get and install required system packages\n    if env.distribution in [\"debian\", \"ubuntu\"]:\n        config_file = get_config_file(env, \"packages.yaml\")\n        (packages, _) = _yaml_to_packages(config_file.base, 'cloudman')\n        # Allow flavors to modify the package list\n        packages = env.flavor.rewrite_config_items(\"packages\", packages)\n        _setup_apt_automation()\n        _apt_packages(pkg_list=packages)\n    elif env.distribution in [\"centos\", \"scientificlinux\"]:\n        env.logger.warn(\"No CloudMan system package dependencies for CentOS\")\n        pass\n    # Get and install required Python libraries\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            url = os.path.join(CM_REPO_ROOT_URL, 'requirements.txt')\n            _create_python_virtualenv(env, 'CM', reqs_url=url)\n    # Add a custom vimrc\n    vimrc_url = os.path.join(MI_REPO_ROOT_URL, 'conf_files', 'vimrc')\n    remote_file = '/etc/vim/vimrc'\n    if env.safe_exists(\"/etc/vim\"):\n        env.safe_sudo(\"wget --output-document=%s %s\" % (remote_file, vimrc_url))\n        env.logger.debug(\"Added a custom vimrc to {0}\".format(remote_file))\n    # Setup profile\n    aliases = ['alias lt=\"ls -ltr\"', 'alias ll=\"ls -l\"']\n    for alias in aliases:\n        _add_to_profiles(alias, ['/etc/bash.bashrc'])\n    env.logger.info(\"Done setting up CloudMan's environment\")\n\n\ndef _configure_logrotate(env):\n    \"\"\"\n    Add logrotate config file, which will automatically rotate CloudMan's log\n    \"\"\"\n    conf_file = \"cloudman.logrotate\"\n    remote = '/etc/logrotate.d/cloudman'\n    url = os.path.join(MI_REPO_ROOT_URL, 'conf_files', conf_file)\n    env.safe_sudo(\"wget --output-document=%s %s\" % (remote, url))\n    env.logger.info(\"----- Added logrotate file to {0} -----\".format(remote))\n\n\ndef _configure_ec2_autorun(env, use_repo_autorun=False):\n    \"\"\"\n    ec2autorun.py is a script that launches CloudMan on instance boot\n    and is thus required on an instance. See the script itself for the\n    details of what it does.\n\n    This script also adds a cloudman service to ``/etc/init``, which\n    actually runs ec2autorun.py as a system-level service at system boot.\n    \"\"\"\n    script = \"ec2autorun.py\"\n    remote = os.path.join(env.install_dir, \"bin\", script)\n    if not env.safe_exists(os.path.dirname(remote)):\n        env.safe_sudo('mkdir -p {0}'.format(os.path.dirname(remote)))\n    if use_repo_autorun:\n        # Is this used, can we eliminate use_repo_autorun?\n        url = os.path.join(MI_REPO_ROOT_URL, script)\n        env.safe_sudo(\"wget --output-document=%s %s\" % (remote, url))\n    else:\n        install_file_dir = os.path.join(env.config_dir, os.pardir, \"installed_files\")\n        tmp_remote = os.path.join(\"/tmp\", os.path.basename(remote))\n        env.safe_put(os.path.join(install_file_dir, script), tmp_remote)\n        env.safe_sudo(\"mv %s %s\" % (tmp_remote, remote))\n        env.safe_sudo(\"chmod 0777 %s\" % remote)\n    # Create upstart configuration file for boot-time script\n    cloudman_boot_file = 'cloudman.conf'\n    remote_file = '/etc/init/%s' % cloudman_boot_file\n    _write_to_file(cm_upstart % (remote, os.path.splitext(remote)[0]), remote_file, mode=\"0644\")\n    # Setup default image user data (if configured by image_user_data_path or\n    # image_user_data_template_path). This specifies defaults for CloudMan when\n    # used with resulting image, normal userdata supplied by user will override\n    # these defaults.\n    image_user_data_path = os.path.join(env.install_dir, \"bin\", \"IMAGE_USER_DATA\")\n    if \"image_user_data_dict\" in env:\n        # Explicit YAML contents defined in env, just dump them as is.\n        import yaml\n        _write_to_file(yaml.dump(env.get(\"image_user_data_dict\")), image_user_data_path, mode=\"0644\")\n    else:\n        # Else use file or template file.\n        _setup_conf_file(env, image_user_data_path, \"image_user_data\", default_source=\"image_user_data\")\n    env.logger.info(\"Done configuring CloudMan's ec2_autorun\")\n\n\ndef _configure_sge(env):\n    \"\"\"\n    This method sets up the environment for SGE w/o\n    actually setting up SGE; it basically makes sure system paths expected\n    by CloudMan exist on the system.\n\n    TODO: Merge this with ``install_sge`` method in ``custom/cloudman.py``.\n    \"\"\"\n    sge_root = '/opt/sge'\n    if not env.safe_exists(sge_root):\n        env.safe_sudo(\"mkdir -p %s\" % sge_root)\n        env.safe_sudo(\"chown sgeadmin:sgeadmin %s\" % sge_root)\n    # Link our installed SGE to CloudMan's expected directory\n    sge_package_dir = \"/opt/galaxy/pkg\"\n    sge_dir = \"ge6.2u5\"\n    if not env.safe_exists(os.path.join(sge_package_dir, sge_dir)):\n        env.safe_sudo(\"mkdir -p %s\" % sge_package_dir)\n    if not env.safe_exists(os.path.join(sge_package_dir, sge_dir)):\n        env.safe_sudo(\"ln --force -s %s/%s %s/%s\" % (env.install_dir, sge_dir, sge_package_dir, sge_dir))\n    env.logger.info(\"Done configuring SGE for CloudMan\")\n\n\ndef _configure_hadoop(env):\n    \"\"\"\n    Grab files required by CloudMan to setup a Hadoop cluster atop SGE.\n    \"\"\"\n    hadoop_root = '/opt/hadoop'\n    url_root = 'https://s3.amazonaws.com/cloudman'\n    hcm_file = 'hadoop.1.0.4__1.0.tar.gz'\n    si_file = 'sge_integration.1.0.tar.gz'\n    # Make sure we're working with a clean hadoop_home dir to avoid any version conflicts\n    env.safe_sudo(\"rm -rf {0}\".format(hadoop_root))\n    env.safe_sudo(\"mkdir -p %s\" % hadoop_root)\n    with cd(hadoop_root):\n        env.safe_sudo(\"wget --output-document={0} {1}/{0}\".format(hcm_file, url_root))\n        env.safe_sudo(\"wget --output-document={0} {1}/{0}\".format(si_file, url_root))\n    env.safe_sudo(\"chown -R {0} {1}\".format(env.user, hadoop_root))\n    env.logger.info(\"Done configuring Hadoop for CloudMan\")\n\n\ndef _configure_nfs(env):\n    \"\"\"\n    Edit ``/etc/exports`` to append paths that are shared over NFS by CloudMan.\n\n    In addition to the hard coded paths listed here, additional paths\n    can be included by setting ``extra_nfs_exports`` in ``fabricrc.txt`` as\n    a comma-separated list of directories.\n    \"\"\"\n    nfs_dir = \"/export/data\"\n    cloudman_dir = \"/mnt/galaxy/export\"\n    if not env.safe_exists(nfs_dir):\n        # For the case of rerunning this script, ensure the nfs_dir does\n        # not exist (exists() method does not recognize it as a file because\n        # by default it points to a non-existing dir/file).\n        with settings(warn_only=True):\n            env.safe_sudo('rm -rf {0}'.format(nfs_dir))\n        env.safe_sudo(\"mkdir -p %s\" % os.path.dirname(nfs_dir))\n        env.safe_sudo(\"ln -s %s %s\" % (cloudman_dir, nfs_dir))\n    env.safe_sudo(\"chown -R %s %s\" % (env.user, os.path.dirname(nfs_dir)))\n    # Setup /etc/exports paths, to be used as NFS mount points\n    # galaxy_data_mount = env.get(\"galaxy_data_mount\", \"/mnt/galaxyData\")\n    # galaxy_indices_mount = env.get(\"galaxy_indices_mount\", \"/mnt/galaxyIndices\")\n    # galaxy_tools_mount = env.get(\"galaxy_tools_mount\", \"/mnt/galaxyTools\")\n    exports = ['/opt/sge           *(rw,sync,no_root_squash,no_subtree_check)',\n               '/opt/hadoop           *(rw,sync,no_root_squash,no_subtree_check)',\n               # '%s    *(rw,sync,no_root_squash,subtree_check,no_wdelay)' % galaxy_data_mount,\n               # '%s *(rw,sync,no_root_squash,no_subtree_check)' % galaxy_indices_mount,\n               # '%s   *(rw,sync,no_root_squash,no_subtree_check)' % galaxy_tools_mount,\n               # '%s       *(rw,sync,no_root_squash,no_subtree_check)' % nfs_dir,\n               # '%s/openmpi         *(rw,sync,no_root_squash,no_subtree_check)' % env.install_dir\n               ]\n    extra_nfs_exports = env.get(\"extra_nfs_exports\", \"\")\n    if extra_nfs_exports:\n        for extra_nfs_export in extra_nfs_exports.split(\",\"):\n            exports.append('%s   *(rw,sync,no_root_squash,no_subtree_check)' % extra_nfs_export)\n    env.safe_append('/etc/exports', exports, use_sudo=True)\n    # Create a symlink for backward compatibility where all of CloudMan's\n    # stuff is expected to be in /opt/galaxy\n    old_dir = '/opt/galaxy'\n    # Because stow is used, the equivalent to CloudMan's expected path\n    # is actually the parent of the install_dir so use it for the symlink\n    new_dir = os.path.dirname(env.install_dir)\n    if not env.safe_exists(old_dir) and exists(new_dir):\n        env.safe_sudo('ln -s {0} {1}'.format(new_dir, old_dir))\n    env.logger.info(\"Done configuring NFS for CloudMan\")\n\n\n@_if_not_installed(\"s3fs\")\ndef install_s3fs(env):\n    \"\"\"\n    Install s3fs, allowing S3 buckets to be mounted as ~POSIX file systems\n    \"\"\"\n    default_version = \"1.61\"\n    version = env.get(\"tool_version\", default_version)\n    url = \"http://s3fs.googlecode.com/files/s3fs-%s.tar.gz\" % version\n    _get_install(url, env, _configure_make)\n\n\ndef _cleanup_ec2(env):\n    \"\"\"\n    Clean up any extra files after building. This method must be called\n    on an instance after being built and before creating a new machine\n    image. *Note* that after this method has run, key-based ssh access\n    to the machine is no longer possible.\n    \"\"\"\n    env.logger.info(\"Cleaning up for EC2 AMI creation\")\n    # Clean up log files and such\n    fnames = [\".bash_history\", \"/var/log/firstboot.done\", \".nx_setup_done\",\n              \"/var/crash/*\", \"%s/ec2autorun.py.log\" % env.install_dir,\n              \"%s/ec2autorun.err\" % env.install_dir, \"%s/ec2autorun.log\" % env.install_dir,\n              \"%s/bin/ec2autorun.log\" % env.install_dir]\n    for fname in fnames:\n        sudo(\"rm -f %s\" % fname)\n    rmdirs = [\"/mnt/galaxyData\", \"/mnt/cm\", \"/tmp/cm\"]\n    for rmdir in rmdirs:\n        sudo(\"rm -rf %s\" % rmdir)\n    # Seed the history with frequently used commands\n    env.logger.debug(\"Setting bash history\")\n    local = os.path.join(env.config_dir, os.pardir, \"installed_files\", \"bash_history\")\n    remote = os.path.join('/home', 'ubuntu', '.bash_history')\n    put(local, remote, mode=\"0660\", use_sudo=True)\n    # Make sure the default config dir is owned by ubuntu\n    sudo(\"chown ubuntu:ubuntu ~/.config\")\n    # Stop Apache from starting automatically at boot (it conflicts with Galaxy's nginx)\n    sudo('/usr/sbin/update-rc.d -f apache2 remove')\n    with settings(warn_only=True):\n        # RabbitMQ fails to start if its database is embedded into the image\n        # because it saves the current IP address or host name so delete it now.\n        # When starting up, RabbitMQ will recreate that directory.\n        sudo('/etc/init.d/rabbitmq-server stop')\n        sudo('service rabbitmq-server stop')\n        # Clean up packages that are causing issues or are unnecessary\n        pkgs_to_remove = ['tntnet', 'tntnet-runtime', 'libtntnet9', 'vsftpd']\n        for ptr in pkgs_to_remove:\n            sudo('apt-get -y --force-yes remove --purge {0}'.format(ptr))\n    sudo('initctl reload-configuration')\n    for db_location in ['/var/lib/rabbitmq/mnesia', '/mnesia']:\n        if exists(db_location):\n            sudo('rm -rf %s' % db_location)\n    # remove existing ssh host key pairs\n    # http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/AESDG-chapter-sharingamis.html\n    sudo(\"rm -f /etc/ssh/ssh_host_*\")\n    sudo(\"rm -f ~/.ssh/authorized_keys*\")\n    sudo(\"rm -f /root/.ssh/authorized_keys*\")\n"
  },
  {
    "path": "cloudbio/config_management/__init__.py",
    "content": "\"\"\" This module contains code related to integrating the configuration\nmanagement tools `chef` and `puppet` into CloudBioLinux. \"\"\"\n"
  },
  {
    "path": "cloudbio/config_management/chef.py",
    "content": "import os\nimport json\n\nfrom fabric.api import cd\nfrom fabric.contrib import files\nfrom fabric.state import _AttributeDict\n\nfrom cloudbio.flavor.config import get_config_file\nfrom utils import build_properties, upload_config, config_dir\n\n\n# Code based heavily on fabric-provision. https://github.com/caffeinehit/fabric-provision\n\nDEFAULTS = dict(\n    path='/var/chef',\n    data_bags=config_dir(os.path.join('chef', 'data_bags')),\n    roles=config_dir(os.path.join('chef', 'roles')),\n    cookbooks=config_dir(os.path.join('chef', 'cookbooks')),\n    log_level='info',\n    recipes=[],\n    run_list=[],\n    json={},\n)\n\nSOLO_RB = \"\"\"\nlog_level            :%(log_level)s\nlog_location         STDOUT\nfile_cache_path      \"%(path)s\"\ndata_bag_path        \"%(path)s/data_bags\"\nrole_path            [ \"%(path)s/roles\" ]\ncookbook_path        [ \"%(path)s/cookbooks\" ]\nChef::Log::Formatter.show_time = true\n\"\"\"\n\n\nclass ChefDict(_AttributeDict):\n    def add_recipe(self, recipe):\n        self.run_list.append('recipe[{0}]'.format(recipe))\n\n    def add_role(self, role):\n        self.run_list.append('role[{0}]'.format(role))\n\n    def _get_json(self):\n        the_json = self['json'].copy()\n        the_json['run_list'] = self['run_list']\n        return the_json\n\n    json = property(fget=_get_json)\n\nchef = ChefDict(DEFAULTS)\n\n\ndef omnibus(env):\n    \"\"\"\n    Install Chef from Opscode's Omnibus installer\n    \"\"\"\n    ctx = {\n        'filename': '%(path)s/install-chef.sh' % chef,\n        'url': 'http://opscode.com/chef/install.sh',\n    }\n    if not files.exists(ctx['filename']):\n        env.safe_sudo('wget -O %(filename)s %(url)s' % ctx)\n        with cd(chef.path):\n            env.safe_sudo('bash install-chef.sh')\n\n\ndef _chef_provision(env, _omnibus=True):\n    env.safe_sudo('mkdir -p %(path)s' % chef)\n\n    omnibus(env)\n\n    config_files = {'node.json': json.dumps(chef.json),\n                    'solo.rb': SOLO_RB % chef}\n    upload_config(chef, config_folder_names=['cookbooks', 'data_bags', 'roles'], config_files=config_files)\n\n    with cd(chef.path):\n        env.safe_sudo('chef-solo -c solo.rb -j node.json')\n\n\ndef _configure_chef(env, chef):\n\n    # Set node json properties\n    node_json_path = get_config_file(env, \"node_extra.json\").base\n    chef.json = _build_chef_properties(env, node_json_path)\n\n    # Set whether to use the Opscode Omnibus Installer to load Chef.\n    use_omnibus_installer_str = env.get(\"use_chef_omnibus_installer\", \"false\")\n    chef.use_omnibus_installer = use_omnibus_installer_str.upper() in [\"TRUE\", \"YES\"]\n\n\ndef _build_chef_properties(env, config_file):\n    \"\"\"\n    Build python object representation of the Chef-solo node.json file from\n    node_extra.json in config dir and the fabric environment.\n    \"\"\"\n\n    json_properties = _parse_json(config_file)\n    return build_properties(env, \"chef\", json_properties)\n\n\ndef _parse_json(filename):\n    \"\"\" Parse a JSON file\n        First remove comments and then use the json module package\n        Comments look like :\n            // ...\n    \"\"\"\n    with open(filename) as f:\n        lines = f.readlines()\n        content = ''.join([line for line in lines if not line.startswith('//')])\n        return json.loads(content)\n"
  },
  {
    "path": "cloudbio/config_management/puppet.py",
    "content": "from fabric.state import _AttributeDict\nfrom fabric.api import cd\n\nfrom utils import upload_config, config_dir, build_properties\nfrom cloudbio.package.deb import _apt_packages\nimport os\n\nDEFAULTS = dict(\n    path='/var/puppet',\n    log_level='info',\n    modules=config_dir(os.path.join('puppet', 'modules'))\n)\n\npuppet = _AttributeDict(DEFAULTS)\n\n\ndef _puppet_provision(env, classes):\n    env.safe_sudo('mkdir -p %(path)s' % puppet)\n    manifest_body = \"node default {\\n%s\\n}\\n\" % _build_node_def_body(env, classes)\n    config_files = {\"manifest.pp\": manifest_body}\n    upload_config(puppet, config_folder_names=[\"modules\"], config_files=config_files)\n    # TODO: Allow yum based install\n    _apt_packages(pkg_list=[\"puppet\"])\n    with cd(puppet.path):\n        env.safe_sudo(\"sudo puppet apply --modulepath=modules manifest.pp\")\n\n\ndef _build_node_def_body(env, classes):\n    contents = \"\"\n    properties = build_properties(env, \"puppet\")\n    contents += \"\\n\".join([\"$%s = '%s'\" % (key, value.replace(\"'\", r\"\\'\")) for key, value in properties.iteritems()])\n    contents += \"\\n\"\n    contents += \"\\n\".join([_build_class_include(env, class_name) for class_name in classes])\n    return contents\n\n\ndef _build_class_include(env, class_name):\n    \"\"\"\n    If parentns::classname is included and fabric\n    properties such as puppet_parentns__classname_prop = val1\n    are set, the class included in puppet will be something like\n\n    class { 'parentns::classname':\n        prop => 'val1',\n    }\n    \"\"\"\n    include_def = \"class { '%s': \\n\" % class_name\n    property_prefix = _property_prefix(class_name)\n    for name, value in env.iteritems():\n        if name.startswith(property_prefix):\n            property_name = name[len(property_prefix):]\n            if not property_name.startswith(\"_\"):  # else subclass property\n                include_def += \"  %s => '%s',\\n\" % (property_name, value)\n    include_def += \"\\n}\"\n    return include_def\n\n\ndef _property_prefix(class_name):\n    return \"puppet_%s_\" % class_name.replace(\"::\", \"__\")\n"
  },
  {
    "path": "cloudbio/config_management/utils.py",
    "content": "from tempfile import mkdtemp\nimport os\nfrom fabric.api import settings, local, put, sudo, cd\nfrom fabric.contrib import files\n\n\ndef config_dir(relative_path):\n    cloudbiolinux_dir = os.path.join(os.path.dirname(__file__), os.pardir, os.pardir)\n    return os.path.join(cloudbiolinux_dir, \"config\", relative_path)\n\n\ndef build_properties(env, prefix, overrides={}):\n    # Prefix will be either chef or puppet\n    prefix = \"%s_\" % prefix\n    # Clone fresh dictonary to modify\n    overrides = dict(overrides)\n\n    # Load fabric environment properties into properties.\n    for key, value in env.iteritems():\n        # Skip invalid properties.\n        if key in overrides or not isinstance(value, str):\n            continue\n\n        if key.startswith(prefix):\n            # If a property starts with chef_ assume it is meant for chef and\n            # add without this prefix. So chef_apache_dir would be available\n            # as apache_dir.\n            overrides[key[len(prefix):]] = value\n        else:\n            # Otherwise, allow chef to access property anyway but prefix with\n            # cloudbiolinux_ so it doesn't clash with anything explicitly\n            # configured for chef.\n            overrides[\"cloudbiolinux_%s\" % key] = value\n    return overrides\n\n\ndef upload_config(config, config_folder_names=[], config_files={}):\n    \"\"\" Common code to upload puppet and chef config files\n    to remote server.\n\n    Heavily based on upload procedure from fabric-provision:\n    https://github.com/caffeinehit/fabric-provision/blob/master/provision/__init__.py\n    \"\"\"\n    names = config_folder_names + config_files.keys()\n    ctx = dict(map(lambda name: (name, '%s/%s' % (config.path, name)), names))\n\n    tmpfolder = mkdtemp()\n\n    listify = lambda what: what if isinstance(what, list) else [what]\n\n    for folder_name in config_folder_names:\n        setattr(config, folder_name, listify(getattr(config, folder_name)))\n\n    for folder_name in config_folder_names:\n        local('mkdir %s/%s' % (tmpfolder, folder_name))\n\n    def copyfolder(folder, what):\n        if not os.path.exists(folder):\n            os.makedirs(folder)\n\n        with settings(warn_only=True):\n            local('cp -r %(folder)s/* %(tmpfolder)s/%(what)s' % dict(\n                    folder=folder,\n                    tmpfolder=tmpfolder,\n                    what=what))\n\n    for what in config_folder_names:\n        map(lambda f: copyfolder(f, what), getattr(config, what))\n\n    folder_paths = \" \".join(map(lambda folder_name: \"./%s\" % folder_name, config_folder_names))\n    local('cd %s && tar -f config_dir.tgz -cz %s' % (tmpfolder, folder_paths))\n\n    # Get rid of old files\n    with settings(warn_only=True):\n        map(lambda what: sudo(\"rm -rf '%s'\" % ctx[what]), ctx.keys())\n\n    # Upload\n    put('%s/config_dir.tgz' % tmpfolder, config.path, use_sudo=True)\n\n    with cd(config.path):\n        sudo('tar -xf config_dir.tgz')\n\n    for file, contents in config_files.iteritems():\n        files.append(ctx[file], contents, use_sudo=True)\n"
  },
  {
    "path": "cloudbio/custom/__init__.py",
    "content": "\"\"\"Fabric sub-modules providing custom installation for non-packaged programs.\n\"\"\"\n"
  },
  {
    "path": "cloudbio/custom/bio_general.py",
    "content": "\"\"\"Custom installs for biological packages.\n\"\"\"\nimport os\n\nfrom fabric.api import *\nfrom fabric.contrib.files import *\n\nfrom cloudbio.custom import shared\nfrom shared import (_if_not_installed, _get_install, _configure_make, _java_install,\n                    _make_tmp_dir)\n\ndef install_anaconda(env):\n    \"\"\"Pre-packaged Anaconda Python installed from Continuum.\n    http://docs.continuum.io/anaconda/index.html\n    \"\"\"\n    version = \"2.0.0\"\n    outdir = os.path.join(env.system_install, \"anaconda\")\n    if env.distribution in [\"ubuntu\", \"centos\", \"scientificlinux\", \"debian\", \"arch\", \"suse\"]:\n        platform = \"Linux\"\n    elif env.distribution in [\"macosx\"]:\n        platform = \"MacOSX\"\n    else:\n        raise ValueError(\"Unexpected distribution: %s\" % env.distribution)\n    url = \"http://09c8d0b2229f813c1b93-c95ac804525aac4b6dba79b00b39d1d3.r79.cf1.rackcdn.com/\" \\\n          \"Anaconda-%s-%s-x86_64.sh\" % (version, platform)\n    if not env.safe_exists(outdir):\n        with _make_tmp_dir() as work_dir:\n            with cd(work_dir):\n                installer = shared._remote_fetch(env, url)\n                env.safe_sed(os.path.basename(url), \"more <<EOF\", \"cat  <<EOF\")\n                env.safe_sudo(\"echo -e '\\nyes\\n%s\\nyes\\n' | bash %s\" % (outdir, installer))\n                env.safe_sudo(\"chown -R %s %s\" % (env.user, outdir))\n                comment_line = \"# added by Ananconda %s installer\" % version\n                if not env.safe_contains(env.shell_config, comment_line):\n                    env.safe_append(env.shell_config, comment_line)\n                    env.safe_append(env.shell_config, \"export PATH=%s/bin:$PATH\" % outdir)\n                # remove curl library with broken certificates\n                env.safe_run(\"%s/bin/conda remove --yes curl\" % outdir)\n                env.safe_run(\"%s/bin/conda install --yes pip\" % outdir)\n\n@_if_not_installed(\"embossversion\")\ndef install_emboss(env):\n    \"\"\"EMBOSS: A high-quality package of free, Open Source software for molecular biology.\n    http://emboss.sourceforge.net/\n    Emboss target for platforms without packages (CentOS -- rpm systems).\n    \"\"\"\n    default_version = \"6.6.0\"\n    version = env.get(\"tool_version\", default_version)\n    url = \"https://science-annex.org/pub/emboss/EMBOSS-%s.tar.gz\" % version\n    #url = \"ftp://emboss.open-bio.org/pub/EMBOSS/EMBOSS-%s.tar.gz\" % version\n    _get_install(url, env, _configure_make)\n\ndef install_pgdspider(env):\n    \"\"\"PGDSpider format conversion for population genetics programs.\n    http://www.cmpg.unibe.ch/software/PGDSpider/\n    \"\"\"\n    if os.path.exists(os.path.join(shared._get_bin_dir(env), \"PGDSpider2.sh\")):\n        return\n    version = \"2.0.2.0\"\n    url = \"http://www.cmpg.unibe.ch/software/PGDSpider/PGDSpider_{v}.zip\".format(\n        v=version)\n    def _install_fn(env, install_dir):\n        env.safe_sudo(\"mv *.jar %s\" % install_dir)\n        bin_dir = shared._get_bin_dir(env)\n        exe_file = \"PGDSpider2.sh\"\n        jar = \"PGDSpider2.jar\"\n        env.safe_sed(exe_file, jar, \"{dir}/{jar}\".format(dir=install_dir, jar=jar))\n        env.safe_run(\"chmod a+x {0}\".format(exe_file))\n        env.safe_sudo(\"mv {exe} {bin}\".format(exe=exe_file, bin=bin_dir))\n    _java_install(\"PGDSpider\", version, url, env, install_fn=_install_fn)\n\ndef install_bio4j(env):\n    \"\"\"Bio4j graph based database built on Neo4j with UniProt, GO, RefSeq and more.\n    http://www.bio4j.com/\n    \"\"\"\n    version = \"0.8\"\n    url = \"https://s3-eu-west-1.amazonaws.com/bio4j-public/releases/\" \\\n          \"{v}/bio4j-{v}.zip\".format(v=version)\n    def _install_fn(env, install_dir):\n        targets = [\"conf\", \"doc\", \"jars\", \"lib\", \"README\"]\n        for x in targets:\n            env.safe_sudo(\"mv {0} {1}\".format(x, install_dir))\n    _java_install(\"bio4j\", version, url, env, install_fn=_install_fn)\n"
  },
  {
    "path": "cloudbio/custom/bio_nextgen.py",
    "content": "\"\"\"Install next gen sequencing analysis tools not currently packaged.\n\"\"\"\nfrom __future__ import print_function\nimport os\nimport re\n\nfrom fabric.api import *\nfrom fabric.contrib.files import *\nimport yaml\n\nfrom shared import (_if_not_installed, _make_tmp_dir,\n                    _get_install, _get_install_local, _make_copy, _configure_make,\n                    _java_install, _python_cmd,\n                    _symlinked_java_version_dir, _fetch_and_unpack, _python_make,\n                    _get_lib_dir, _get_include_dir, _apply_patch)\nfrom cloudbio.custom import shared, versioncheck\n\nfrom cloudbio import libraries\nfrom cloudbio.flavor.config import get_config_file\n\n\n@_if_not_installed([\"twoBitToFa\", \"gtfToGenePred\"])\ndef install_ucsc_tools(env):\n    \"\"\"Useful executables from UCSC.\n\n    todo: install from source to handle 32bit and get more programs\n    http://hgdownload.cse.ucsc.edu/admin/jksrc.zip\n    \"\"\"\n    tools = [\"liftOver\", \"faToTwoBit\", \"bedToBigBed\",\n             \"bigBedInfo\", \"bigBedSummary\", \"bigBedToBed\",\n             \"bedGraphToBigWig\", \"bigWigInfo\", \"bigWigSummary\",\n             \"bigWigToBedGraph\", \"bigWigToWig\",\n             \"fetchChromSizes\", \"wigToBigWig\", \"faSize\", \"twoBitInfo\",\n             \"twoBitToFa\", \"faCount\", \"gtfToGenePred\"]\n    url = \"http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/\"\n    _download_executables(env, url, tools)\n\n\n@_if_not_installed(\"blat\")\ndef install_kent_tools(env):\n    \"\"\"\n\n    Please note that the Blat source and executables are freely available for\n    academic, nonprofit and personal use. Commercial licensing information is\n    available on the Kent Informatics website (http://www.kentinformatics.com/).\n    \"\"\"\n    tools = [\"blat\", \"gfClient\", \"gfServer\"]\n    url = \"http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/blat/\"\n    _download_executables(env, url, tools)\n\n\ndef _download_executables(env, base_url, tools):\n    install_dir = shared._get_bin_dir(env)\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            for tool in tools:\n                final_tool = os.path.join(install_dir, tool)\n                if not env.safe_exists(final_tool) and shared._executable_not_on_path(tool):\n                    shared._remote_fetch(env, \"%s%s\" % (base_url, tool))\n                    env.safe_sudo(\"cp -f %s %s\" % (tool, install_dir))\n                    final_path = os.path.join(install_dir, tool)\n                    env.safe_sudo(\"chmod uga+rx %s\" % final_path)\n\n# --- Alignment tools\ndef install_featurecounts(env):\n    \"\"\"\n    featureCounts from the subread package for counting reads mapping to\n    genomic features\n    \"\"\"\n    default_version = \"1.4.4\"\n    version = env.get(\"tool_version\", default_version)\n    if versioncheck.up_to_date(env, \"featureCounts\", version, stdout_flag=\"Version\"):\n        return\n    platform = \"MacOS\" if env.distribution == \"macosx\" else \"Linux\"\n    url = (\"http://downloads.sourceforge.net/project/subread/\"\n           \"subread-%s/subread-%s-%s-x86_64.tar.gz\"\n           % (version, version, platform))\n    _get_install(url, env, _make_copy(\"find . -type f -perm -100 -name 'featureCounts'\",\n                                      do_make=False))\n\n\n@_if_not_installed(\"bowtie\")\ndef install_bowtie(env):\n    \"\"\"The bowtie short read aligner.\n    http://bowtie-bio.sourceforge.net/index.shtml\n    \"\"\"\n    default_version = \"1.0.0\"\n    version = env.get(\"tool_version\", default_version)\n    url = \"http://downloads.sourceforge.net/project/bowtie-bio/bowtie/%s/\" \\\n          \"bowtie-%s-src.zip\" % (version, version)\n    _get_install(url, env, _make_copy(\"find . -perm -100 -name 'bowtie*'\"))\n\n@_if_not_installed(\"bowtie2\")\ndef install_bowtie2(env):\n    \"\"\"bowtie2 short read aligner, with gap support.\n    http://bowtie-bio.sourceforge.net/bowtie2/index.shtml\n    \"\"\"\n    default_version = \"2.1.0\"\n    version = env.get(\"tool_version\", default_version)\n    url = \"http://downloads.sourceforge.net/project/bowtie-bio/bowtie2/%s/\" \\\n          \"bowtie2-%s-source.zip\" % (version, version)\n    _get_install(url, env, _make_copy(\"find . -perm -100 -name 'bowtie2*'\"))\n\n@_if_not_installed(\"bfast\")\ndef install_bfast(env):\n    \"\"\"BFAST: Blat-like Fast Accurate Search Tool.\n    http://sourceforge.net/apps/mediawiki/bfast/index.php?title=Main_Page\n    \"\"\"\n    default_version = \"0.7.0a\"\n    version = env.get(\"tool_version\", default_version)\n    major_version_regex = \"\\d+\\.\\d+\\.\\d+\"\n    major_version = re.search(major_version_regex, version).group(0)\n    url = \"http://downloads.sourceforge.net/project/bfast/bfast/%s/bfast-%s.tar.gz\"\\\n            % (major_version, version)\n    _get_install(url, env, _configure_make)\n\n@_if_not_installed(\"perm\")\ndef install_perm(env):\n    \"\"\"Efficient mapping of short sequences accomplished with periodic full sensitive spaced seeds.\n    https://code.google.com/p/perm/\n    \"\"\"\n    default_version = \"4\"\n    version = env.get(\"tool_version\", default_version)\n    url = \"http://perm.googlecode.com/files/PerM%sSource.tar.gz\" % version\n    def gcc44_makefile_patch():\n        gcc_cmd = \"g++44\"\n        with settings(hide('warnings', 'running', 'stdout', 'stderr'),\n                      warn_only=True):\n            result = env.safe_run(\"%s -v\" % gcc_cmd)\n        print(result.return_code)\n        if result.return_code == 0:\n            env.safe_sed(\"makefile\", \"g\\+\\+\", gcc_cmd)\n    _get_install(url, env, _make_copy(\"ls -1 perm\", gcc44_makefile_patch))\n\n@_if_not_installed(\"snap\")\ndef install_snap(env):\n    \"\"\"Scalable Nucleotide Alignment Program\n    http://snap.cs.berkeley.edu/\n    \"\"\"\n    version = \"0.15\"\n    url = \"http://github.com/downloads/amplab/snap/\" \\\n          \"snap-%s-linux.tar.gz\" % version\n    _get_install(url, env, _make_copy(\"find . -perm -100 -type f\", do_make=False))\n\ndef install_stampy(env):\n    \"\"\"Stampy: mapping of short reads from illumina sequencing machines onto a reference genome.\n    http://www.well.ox.ac.uk/project-stampy\n    \"\"\"\n    version = \"1.0.21\"\n    #version = base_version\n    #revision = \"1654\"\n    #version = \"{0}r{1}\".format(base_version, revision)\n    #url = \"http://www.well.ox.ac.uk/bioinformatics/Software/\" \\\n    #      \"stampy-%s.tgz\" % (version)\n    # Ugh -- Stampy now uses a 'Stampy-latest' download target\n    url = \"http://www.well.ox.ac.uk/bioinformatics/Software/\" \\\n          \"Stampy-latest.tgz\"\n    def _clean_makefile(env):\n        env.safe_sed(\"makefile\", \" -Wl\", \"\")\n    _get_install_local(url, env, _make_copy(),\n                       dir_name=\"stampy-{0}\".format(version),\n                       post_unpack_fn=_clean_makefile)\n\n@_if_not_installed(\"gmap\")\ndef install_gmap(env):\n    \"\"\"GMAP and GSNAP: A Genomic Mapping and Alignment Program for mRNA EST and short reads.\n    http://research-pub.gene.com/gmap/\n    \"\"\"\n    version = \"2012-11-09\"\n    url = \"http://research-pub.gene.com/gmap/src/gmap-gsnap-%s.tar.gz\" % version\n    _get_install(url, env, _configure_make)\n\ndef _wget_with_cookies(ref_url, dl_url):\n    env.safe_run(\"wget --cookies=on --keep-session-cookies --save-cookies=cookie.txt %s\"\n                 % (ref_url))\n    env.safe_run(\"wget --referer=%s --cookies=on --load-cookies=cookie.txt \"\n                 \"--keep-session-cookies --save-cookies=cookie.txt %s\" %\n                 (ref_url, dl_url))\n\n@_if_not_installed(\"novoalign\")\ndef install_novoalign(env):\n    \"\"\"Novoalign short read aligner using Needleman-Wunsch algorithm with affine gap penalties.\n    http://www.novocraft.com/main/index.php\n    \"\"\"\n    base_version = \"V3.00.02\"\n    cs_version = \"V1.03.02\"\n    _url = \"http://www.novocraft.com/downloads/%s/\" % base_version\n    ref_url = \"http://www.novocraft.com/main/downloadpage.php\"\n    base_url = \"%s/novocraft%s.gcc.tar.gz\" % (_url, base_version)\n    cs_url = \"%s/novoalignCS%s.gcc.tar.gz\" % (_url, cs_version)\n    install_dir = shared._get_bin_dir(env)\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            _wget_with_cookies(ref_url, base_url)\n            env.safe_run(\"tar -xzvpf novocraft%s.gcc.tar.gz\" % base_version)\n            with cd(\"novocraft\"):\n                for fname in [\"isnovoindex\", \"novo2maq\", \"novo2paf\",\n                              \"novo2sam.pl\", \"novoalign\", \"novobarcode\",\n                              \"novoindex\", \"novope2bed.pl\", \"novorun.pl\",\n                              \"novoutil\"]:\n                    env.safe_sudo(\"mv %s %s\" % (fname, install_dir))\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            _wget_with_cookies(ref_url, cs_url)\n            env.safe_run(\"tar -xzvpf novoalignCS%s.gcc.tar.gz\" % cs_version)\n            with cd(\"novoalignCS\"):\n                for fname in [\"novoalignCS\"]:\n                    env.safe_sudo(\"mv %s %s\" % (fname, install_dir))\n\n@_if_not_installed(\"novosort\")\ndef install_novosort(env):\n    \"\"\"Multithreaded sort and merge for BAM files.\n    http://www.novocraft.com/wiki/tiki-index.php?page=Novosort\n    \"\"\"\n    base_version = \"V3.00.02\"\n    version = \"V1.00.02\"\n    url = \"http://www.novocraft.com/downloads/%s/novosort%s.gcc.tar.gz\" % (base_version, version)\n    ref_url = \"http://www.novocraft.com/main/downloadpage.php\"\n    install_dir = shared._get_bin_dir(env)\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            _wget_with_cookies(ref_url, url)\n            env.safe_run(\"tar -xzvpf novosort%s.gcc.tar.gz\" % version)\n            with cd(\"novosort\"):\n                for fname in [\"novosort\"]:\n                    env.safe_sudo(\"mv %s %s\" % (fname, install_dir))\n\n@_if_not_installed(\"lastz\")\ndef install_lastz(env):\n    \"\"\"LASTZ sequence alignment program.\n    http://www.bx.psu.edu/miller_lab/dist/README.lastz-1.02.00/README.lastz-1.02.00a.html\n    \"\"\"\n    default_version = \"1.02.00\"\n    version = env.get(\"tool_version\", default_version)\n    url = \"http://www.bx.psu.edu/miller_lab/dist/\" \\\n          \"lastz-%s.tar.gz\" % version\n    def _remove_werror(env):\n        env.safe_sed(\"src/Makefile\", \" -Werror\", \"\")\n    _get_install(url, env, _make_copy(\"find . -perm -100 -name 'lastz'\"),\n                 post_unpack_fn=_remove_werror)\n\n@_if_not_installed(\"MosaikAligner\")\ndef install_mosaik(env):\n    \"\"\"MOSAIK: reference-guided aligner for next-generation sequencing technologies\n    http://code.google.com/p/mosaik-aligner/\n    \"\"\"\n    version = \"2.2.3\"\n    url = \"https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/mosaik-aligner/\" \\\n          \"MOSAIK-%s-Linux-x64.tar\" % version\n    _get_install(url, env, _make_copy(\"find . -perm -100 -type f\", do_make=False))\n\n# --- Utilities\n\ndef install_samtools(env):\n    \"\"\"SAM Tools provide various utilities for manipulating alignments in the SAM format.\n    http://samtools.sourceforge.net/\n    \"\"\"\n    default_version = \"0.1.19\"\n    version = env.get(\"tool_version\", default_version)\n    if versioncheck.up_to_date(env, \"samtools\", version, stdout_flag=\"Version:\"):\n        env.logger.info(\"samtools version {0} is up to date; not installing\"\n                        .format(version))\n        return\n    url = \"http://downloads.sourceforge.net/project/samtools/samtools/\" \\\n          \"%s/samtools-%s.tar.bz2\" % (version, version)\n    def _safe_ncurses_make(env):\n        \"\"\"Combine samtools, removing ncurses refs if not present on system.\n        \"\"\"\n        with settings(warn_only=True):\n            result = env.safe_run(\"make\")\n        # no ncurses, fix Makefile and rebuild\n        if result.failed:\n            env.safe_sed(\"Makefile\", \"-D_CURSES_LIB=1\", \"-D_CURSES_LIB=0\")\n            env.safe_sed(\"Makefile\", \"-lcurses\", \"# -lcurses\")\n            env.safe_run(\"make clean\")\n            env.safe_run(\"make\")\n        install_dir = shared._get_bin_dir(env)\n        for fname in env.safe_run_output(\"ls -1 samtools bcftools/bcftools bcftools/vcfutils.pl misc/wgsim\").split(\"\\n\"):\n            env.safe_sudo(\"cp -f %s %s\" % (fname.rstrip(\"\\r\"), install_dir))\n    _get_install(url, env, _safe_ncurses_make)\n\n@_if_not_installed(\"vtools\")\ndef install_varianttools(env):\n    \"\"\"Annotation, selection, and analysis of variants in the context of next-gen sequencing analysis.\n    http://varianttools.sourceforge.net/\n    \"\"\"\n    version = \"1.0.6\"\n    url = \"http://downloads.sourceforge.net/project/varianttools/\" \\\n          \"{ver}/variant_tools-{ver}-src.tar.gz\".format(ver=version)\n    _get_install(url, env, _python_make)\n\n@_if_not_installed(\"dwgsim\")\ndef install_dwgsim(env):\n    \"\"\"DWGSIM: simulating NGS data and evaluating mappings and variant calling.\n    http://sourceforge.net/apps/mediawiki/dnaa/index.php?title=Main_Page\n    \"\"\"\n    version = \"0.1.10\"\n    samtools_version = \"0.1.18\"\n    url = \"http://downloads.sourceforge.net/project/dnaa/dwgsim/\" \\\n          \"dwgsim-{0}.tar.gz\".format(version)\n    samtools_url = \"http://downloads.sourceforge.net/project/samtools/samtools/\" \\\n                   \"{ver}/samtools-{ver}.tar.bz2\".format(ver=samtools_version)\n    def _get_samtools(env):\n        shared._remote_fetch(env, samtools_url)\n        env.safe_run(\"tar jxf samtools-{0}.tar.bz2\".format(samtools_version))\n        env.safe_run(\"ln -s samtools-{0} samtools\".format(samtools_version))\n    _get_install(url, env, _make_copy(\"ls -1 dwgsim dwgsim_eval scripts/dwgsim_pileup_eval.pl\"),\n                 post_unpack_fn=_get_samtools)\n\n@_if_not_installed(\"fastq_screen\")\ndef install_fastq_screen(env):\n    \"\"\"A screening application for high througput sequence data.\n    http://www.bioinformatics.babraham.ac.uk/projects/fastq_screen/\n    \"\"\"\n    version = \"0.4\"\n    url = \"http://www.bioinformatics.babraham.ac.uk/projects/fastq_screen/\" \\\n          \"fastq_screen_v%s.tar.gz\" % version\n    install_dir = shared._symlinked_shared_dir(\"fastqc_screen\", version, env)\n    executable = \"fastq_screen\"\n    if install_dir:\n        with _make_tmp_dir() as work_dir:\n            with cd(work_dir):\n                out_file = shared._remote_fetch(env, url)\n                env.safe_run(\"tar -xzvpf %s\" % out_file)\n                with cd(\"fastq_screen_v%s\" % version):\n                    env.safe_sudo(\"mv * %s\" % install_dir)\n                env.safe_sudo(\"ln -s %s/%s %s/bin/%s\" % (install_dir, executable,\n                                                         env.system_install, executable))\n\ndef install_bedtools(env):\n    \"\"\"A flexible suite of utilities for comparing genomic features.\n    https://code.google.com/p/bedtools/\n    \"\"\"\n    version = \"2.17.0\"\n    if versioncheck.up_to_date(env, \"bedtools --version\", version, stdout_flag=\"bedtools\"):\n        return\n    url = \"https://bedtools.googlecode.com/files/\" \\\n          \"BEDTools.v%s.tar.gz\" % version\n    _get_install(url, env, _make_copy(\"ls -1 bin/*\"))\n\n_shrec_run = \"\"\"\n#!/usr/bin/perl\nuse warnings;\nuse strict;\nuse FindBin qw($RealBin);\nuse Getopt::Long;\n\nmy @java_args;\nmy @args;\nforeach (@ARGV) {\n  if (/^\\-X/) {push @java_args,$_;}\n  else {push @args,$_;}}\nsystem(\"java -cp $RealBin @java_args Shrec @args\");\n\"\"\"\n\n@_if_not_installed(\"shrec\")\ndef install_shrec(env):\n    \"\"\"Shrec is a bioinformatics tool for error correction of HTS read data.\n    http://sourceforge.net/projects/shrec-ec/\n    \"\"\"\n    version = \"2.2\"\n    url = \"http://downloads.sourceforge.net/project/shrec-ec/SHREC%%20%s/bin.zip\" % version\n    install_dir = _symlinked_java_version_dir(\"shrec\", version, env)\n    if install_dir:\n        shrec_script = \"%s/shrec\" % install_dir\n        with _make_tmp_dir() as work_dir:\n            with cd(work_dir):\n                out_file = shared._remote_fetch(env, url)\n                env.safe_run(\"unzip %s\" % out_file)\n                env.safe_sudo(\"mv *.class %s\" % install_dir)\n                for line in _shrec_run.split(\"\\n\"):\n                    if line.strip():\n                        env.safe_append(shrec_script, line, use_sudo=env.use_sudo)\n                env.safe_sudo(\"chmod a+rwx %s\" % shrec_script)\n                env.safe_sudo(\"ln -s %s %s/bin/shrec\" % (shrec_script, env.system_install))\n\ndef install_echo(env):\n    \"\"\"ECHO: A reference-free short-read error correction algorithm\n    http://uc-echo.sourceforge.net/\n    \"\"\"\n    version = \"1_12\"\n    url = \"http://downloads.sourceforge.net/project/uc-echo/source%20release/\" \\\n          \"echo_v{0}.tgz\".format(version)\n    _get_install_local(url, env, _make_copy())\n\n# -- Analysis\n\ndef install_picard(env):\n    \"\"\"Command-line utilities that manipulate BAM files with a Java API.\n    http://picard.sourceforge.net/\n    \"\"\"\n    version = \"1.96\"\n    url = \"http://downloads.sourceforge.net/project/picard/\" \\\n          \"picard-tools/%s/picard-tools-%s.zip\" % (version, version)\n    _java_install(\"picard\", version, url, env)\n\ndef install_alientrimmer(env):\n    \"\"\"\n    Adapter removal tool\n    http://www.ncbi.nlm.nih.gov/pubmed/23912058\n    \"\"\"\n    version = \"0.3.2\"\n    url = (\"ftp://ftp.pasteur.fr/pub/gensoft/projects/AlienTrimmer/\"\n           \"AlienTrimmer_%s.tar.gz\" % version)\n    _java_install(\"AlienTrimmer\", version, url, env)\n\ndef install_rnaseqc(env):\n    \"\"\"Quality control metrics for RNA-seq data\n    https://www.broadinstitute.org/cancer/cga/rna-seqc\n    \"\"\"\n    version = \"1.1.7\"\n    url = (\"https://github.com/chapmanb/RNA-SeQC/releases/download/\"\n           \"v%s/RNA-SeQC_v%s.jar\" % (version, version))\n    install_dir = _symlinked_java_version_dir(\"RNA-SeQC\", version, env)\n    if install_dir:\n        with _make_tmp_dir() as work_dir:\n            with cd(work_dir):\n                out_file = shared._remote_fetch(env, url)\n                env.safe_sudo(\"mv %s %s\" % (out_file, install_dir))\n\ndef install_varscan(env):\n    \"\"\"Variant detection in massively parallel sequencing data\n    http://varscan.sourceforge.net/\n    \"\"\"\n    version = \"2.3.7\"\n    url = \"http://downloads.sourceforge.net/project/varscan/VarScan.v%s.jar\" % version\n    install_dir = _symlinked_java_version_dir(\"varscan\", version, env)\n    if install_dir:\n        with _make_tmp_dir() as work_dir:\n            with cd(work_dir):\n                out_file = shared._remote_fetch(env, url)\n                env.safe_sudo(\"mv %s %s\" % (out_file, install_dir))\n\ndef install_mutect(env):\n    version = \"1.1.5\"\n    url = \"https://github.com/broadinstitute/mutect/releases/download/\" \\\n          \"%s/muTect-%s-bin.zip\" % (version, version)\n    install_dir = _symlinked_java_version_dir(\"mutect\", version, env)\n    if install_dir:\n        with _make_tmp_dir() as work_dir:\n            with cd(work_dir):\n                out_file = shared._remote_fetch(env, url)\n                env.safe_run(\"unzip %s\" % out_file)\n                env.safe_sudo(\"mv *.jar version.txt LICENSE* %s\" % install_dir)\n\n@_if_not_installed(\"bam\")\ndef install_bamutil(env):\n    \"\"\"Utilities for working with BAM files, from U of M Center for Statistical Genetics.\n    http://genome.sph.umich.edu/wiki/BamUtil\n    \"\"\"\n    version = \"1.0.7\"\n    url = \"http://genome.sph.umich.edu/w/images/5/5d/BamUtilLibStatGen.%s.tgz\" % version\n    _get_install(url, env, _make_copy(\"ls -1 bamUtil/bin/bam\"),\n                 dir_name=\"bamUtil_%s\" % version)\n\n@_if_not_installed(\"tabix\")\ndef install_tabix(env):\n    \"\"\"Generic indexer for TAB-delimited genome position files\n    http://samtools.sourceforge.net/tabix.shtml\n    \"\"\"\n    version = \"0.2.6\"\n    url = \"http://downloads.sourceforge.net/project/samtools/tabix/tabix-%s.tar.bz2\" % version\n    _get_install(url, env, _make_copy(\"ls -1 tabix bgzip\"))\n\n@_if_not_installed(\"disambiguate.py\")\ndef install_disambiguate(env):\n    \"\"\"a  tool for disambiguating reads aligning to multiple genomes\n    https://github.com:mjafin/disambiguate\n    \"\"\"\n    repository = \"git clone https://github.com/mjafin/disambiguate.git\"\n    _get_install(repository, env, _python_make)\n\ndef install_grabix(env):\n    \"\"\"a wee tool for random access into BGZF files\n    https://github.com/arq5x/grabix\n    \"\"\"\n    version = \"0.1.6\"\n    revision = \"ba792bc872d38d3cb5a69b2de00e39a6ac367d69\"\n    try:\n        uptodate = versioncheck.up_to_date(env, \"grabix\", version, stdout_flag=\"version:\")\n    # Old versions will not have any version information\n    except IOError:\n        uptodate = False\n    if uptodate:\n        return\n    repository = \"git clone https://github.com/arq5x/grabix.git\"\n    _get_install(repository, env, _make_copy(\"ls -1 grabix\"),\n                 revision=revision)\n\n@_if_not_installed(\"pbgzip\")\ndef install_pbgzip(env):\n    \"\"\"Parallel blocked bgzip -- compatible with bgzip but with thread support.\n    https://github.com/nh13/samtools/tree/master/pbgzip\n    \"\"\"\n    repository = \"git clone https://github.com/chapmanb/samtools.git\"\n    revision = \"2cce3ffa97\"\n    def _build(env):\n        with cd(\"pbgzip\"):\n            env.safe_run(\"make\")\n            install_dir = shared._get_bin_dir(env)\n            env.safe_sudo(\"cp -f pbgzip %s\" % (install_dir))\n    _get_install(repository, env, _build, revision=revision)\n\n@_if_not_installed(\"bamtools\")\ndef install_bamtools(env):\n    \"\"\"command-line toolkit for working with BAM data\n    https://github.com/pezmaster31/bamtools\n    \"\"\"\n    version = \"3fe66b9\"\n    repository = \"git clone --recursive https://github.com/pezmaster31/bamtools.git\"\n    def _cmake_bamtools(env):\n        env.safe_run(\"mkdir build\")\n        with cd(\"build\"):\n            env.safe_run(\"cmake ..\")\n            env.safe_run(\"make\")\n        env.safe_sudo(\"cp bin/* %s\" % shared._get_bin_dir(env))\n        env.safe_sudo(\"cp lib/* %s\" % shared._get_lib_dir(env))\n    _get_install(repository, env, _cmake_bamtools,\n                 revision=version)\n\n@_if_not_installed(\"ogap\")\ndef install_ogap(env):\n    \"\"\"gap opening realigner for BAM data streams\n    https://github.com/ekg/ogap\n    \"\"\"\n    version = \"652c525\"\n    repository = \"git clone --recursive https://github.com/ekg/ogap.git\"\n    _get_install(repository, env, _make_copy(\"ls ogap\"),\n                 revision=version)\n\ndef install_tophat(env):\n    \"\"\"TopHat is a fast splice junction mapper for RNA-Seq reads\n    http://ccb.jhu.edu/software/tophat/index.shtml\n    \"\"\"\n    default_version = \"2.0.9\"\n    version = env.get(\"tool_version\", default_version)\n    if versioncheck.is_version(env, \"tophat\", version, args=\"--version\", stdout_flag=\"TopHat\"):\n        env.logger.info(\"tophat version {0} is up to date; not installing\"\n            .format(version))\n        return\n    platform = \"OSX\" if env.distribution == \"macosx\" else \"Linux\"\n    url = \"http://ccb.jhu.edu/software/tophat/downloads/\" \\\n          \"tophat-%s.%s_x86_64.tar.gz\" % (version, platform)\n\n    _get_install(url, env,\n                 _make_copy(\"find . -perm -100 -type f\", do_make=False))\n\ninstall_tophat2 = install_tophat\n\n# --- Assembly\n\n@_if_not_installed(\"ABYSS\")\ndef install_abyss(env):\n    \"\"\"Assembly By Short Sequences - a de novo, parallel, paired-end sequence assembler.\n    http://www.bcgsc.ca/platform/bioinfo/software/abyss\n    \"\"\"\n    # XXX check for no sparehash on non-ubuntu systems\n    default_version = \"2.0.2\"\n    version = env.get(\"tool_version\", default_version)\n    url = \"http://www.bcgsc.ca/platform/bioinfo/software/abyss/releases/%s/abyss-%s.tar.gz\" % (version, version)\n    def _remove_werror_get_boost(env):\n        env.safe_sed(\"configure\", \" -Werror\", \"\")\n        # http://osdir.com/ml/abyss-users-science/2011-10/msg00108.html\n        url = \"http://downloads.sourceforge.net/project/boost/boost/1.63.0/boost_1_63_0.tar.bz2\"\n        dl_file = shared._remote_fetch(env, url)\n        env.safe_run(\"tar jxf %s\" % dl_file)\n        env.safe_run(\"ln -s boost_1_63_0/boost boost\")\n    _get_install(url, env, _configure_make, post_unpack_fn=_remove_werror_get_boost)\n\ndef install_transabyss(env):\n    \"\"\"Analyze ABySS multi-k-assembled shotgun transcriptome data.\n    http://www.bcgsc.ca/platform/bioinfo/software/trans-abyss\n    \"\"\"\n    version = \"1.4.4\"\n    url = \"http://www.bcgsc.ca/platform/bioinfo/software/trans-abyss/\" \\\n          \"releases/%s/trans-ABySS-v%s.tar.gz\" % (version, version)\n    _get_install_local(url, env, _make_copy(do_make=False))\n\n@_if_not_installed(\"velvetg\")\ndef install_velvet(env):\n    \"\"\"Sequence assembler for very short reads.\n    http://www.ebi.ac.uk/~zerbino/velvet/\n    \"\"\"\n    default_version = \"1.2.10\"\n    version = env.get(\"tool_version\", default_version)\n    url = \"http://www.ebi.ac.uk/~zerbino/velvet/velvet_%s.tgz\" % version\n    def _fix_library_order(env):\n        \"\"\"Fix library order problem in recent gcc versions\n        http://biostar.stackexchange.com/questions/13713/\n        error-installing-velvet-assembler-1-1-06-on-ubuntu-server\n        \"\"\"\n        env.safe_sed(\"Makefile\", \"Z_LIB_FILES=-lz\", \"Z_LIB_FILES=-lz -lm\")\n    _get_install(url, env, _make_copy(\"find . -perm -100 -name 'velvet*'\"),\n                 post_unpack_fn=_fix_library_order)\n\n@_if_not_installed(\"Ray\")\ndef install_ray(env):\n    \"\"\"Ray -- Parallel genome assemblies for parallel DNA sequencing\n    http://denovoassembler.sourceforge.net/\n    \"\"\"\n    default_version = \"2.2.0\"\n    version = env.get(\"tool_version\", default_version)\n    url = \"http://downloads.sourceforge.net/project/denovoassembler/Ray-v%s.tar.bz2\" % version\n    def _ray_do_nothing(env):\n        return\n    _get_install(url, env, _make_copy(\"find . -name Ray\"),\n                 post_unpack_fn=_ray_do_nothing)\n\ndef install_trinity(env):\n    \"\"\"Efficient and robust de novo reconstruction of transcriptomes from RNA-seq data.\n    http://trinityrnaseq.github.io/\n    \"\"\"\n    version = \"2.3.2\"\n    url = \"https://github.com/trinityrnaseq/trinityrnaseq/archive/\" \\\n          \"Trinity-v%s.tar.gz\" % version\n    dir_name = \"trinityrnaseq-%s\" % version\n    _get_install_local(url, env, _make_copy(),\n                       dir_name=dir_name)\n\ndef install_cortex_var(env):\n    \"\"\"De novo genome assembly and variation analysis from sequence data.\n    http://cortexassembler.sourceforge.net/index_cortex_var.html\n    \"\"\"\n    version = \"1.0.5.21\"\n    url = \"http://downloads.sourceforge.net/project/cortexassembler/cortex_var/\" \\\n          \"latest/CORTEX_release_v{0}.tgz\".format(version)\n    def _cortex_build(env):\n        env.safe_sed(\"Makefile\", \"\\-L/full/path/\\S*\",\n                     \"-L{0}/lib -L/usr/lib -L/usr/local/lib\".format(env.system_install))\n        env.safe_sed(\"Makefile\", \"^IDIR_GSL =.*$\",\n                     \"IDIR_GSL={0}/include -I/usr/include -I/usr/local/include\".format(env.system_install))\n        env.safe_sed(\"Makefile\", \"^IDIR_GSL_ALSO =.*$\",\n                     \"IDIR_GSL_ALSO={0}/include/gsl -I/usr/include/gsl -I/usr/local/include/gsl\".format(\n                         env.system_install))\n        with cd(\"libs/gsl-1.15\"):\n            env.safe_run(\"make clean\")\n        with cd(\"libs/htslib\"):\n            env.safe_run(\"make clean\")\n            env.safe_run(\"make\")\n        for cols in [\"1\", \"2\", \"3\", \"4\", \"5\"]:\n            for kmer in [\"31\", \"63\", \"95\"]:\n                env.safe_run(\"make MAXK={0} NUM_COLS={1} cortex_var\".format(kmer, cols))\n        with cd(\"scripts/analyse_variants/needleman_wunsch\"):\n            env.safe_sed(\"Makefile\", \"string_buffer.c\", \"string_buffer.c -lz\")\n            # Fix incompatibilities with gzfile struct in zlib 1.2.6+\n            for fix_gz in [\"libs/string_buffer/string_buffer.c\", \"libs/bioinf/bioinf.c\",\n                           \"libs/string_buffer/string_buffer.h\", \"libs/bioinf/bioinf.h\"]:\n                env.safe_sed(fix_gz, \"gzFile \\*\", \"gzFile \")\n                env.safe_sed(fix_gz, \"gzFile\\*\", \"gzFile\")\n            env.safe_run(\"make\")\n    _get_install_local(url, env, _cortex_build)\n\ndef install_bcbio_variation(env):\n    \"\"\"Toolkit to analyze genomic variation data with comparison and ensemble approaches.\n    https://github.com/chapmanb/bcbio.variation\n    \"\"\"\n    version = \"0.2.6\"\n    url = \"https://github.com/chapmanb/bcbio.variation/releases/download/\" \\\n          \"v%s/bcbio.variation-%s-standalone.jar\" % (version, version)\n    install_dir = _symlinked_java_version_dir(\"bcbio_variation\", version, env)\n    if install_dir:\n        with _make_tmp_dir() as work_dir:\n            with cd(work_dir):\n                jar_file = shared._remote_fetch(env, url)\n                env.safe_sudo(\"mv %s %s\" % (jar_file, install_dir))\n\n# --- ChIP-seq\n\n@_if_not_installed(\"macs14\")\ndef install_macs(env):\n    \"\"\"Model-based Analysis for ChIP-Seq.\n    http://liulab.dfci.harvard.edu/MACS/\n    \"\"\"\n    default_version = \"1.4.2\"\n    version = env.get(\"tool_version\", default_version)\n    url = \"https://github.com/downloads/taoliu/MACS/\" \\\n          \"MACS-%s.tar.gz\" % version\n    _get_install(url, env, _python_make)\n\n# --- Structural variation\n@_if_not_installed(\"hydra\")\ndef install_hydra(env):\n    \"\"\"Hydra detects structural variation breakpoints in both unique and duplicated genomic regions.\n    https://code.google.com/p/hydra-sv/\n    \"\"\"\n    version = \"0.5.3\"\n    url = \"https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/hydra-sv/Hydra.v{0}.tar.gz\".format(version)\n    def clean_libs(env):\n        env.safe_run(\"make clean\")\n    _get_install(url, env, _make_copy(\"ls -1 bin/* scripts/*\"),\n                 post_unpack_fn=clean_libs)\n\ndef install_freec(env):\n    \"\"\"Control-FREEC: a tool for detection of copy number changes and allelic imbalances.\n    http://bioinfo-out.curie.fr/projects/freec/\n    \"\"\"\n    version = \"6.4\"\n    if env.distribution in [\"ubuntu\", \"debian\"]:\n        if env.is_64bit:\n            url = \"http://bioinfo-out.curie.fr/projects/freec/src/FREEC_Linux64.tar.gz\"\n        else:\n            url = \"http://bioinfo-out.curie.fr/projects/freec/src/FREEC_LINUX32.tar.gz\"\n\n        if not versioncheck.up_to_date(env, \"freec\", version, stdout_index=1):\n            _get_install(url, env, _make_copy(\"find . -name 'freec'\"), dir_name=\".\")\n\n@_if_not_installed(\"CRISP.py\")\ndef install_crisp(env):\n    \"\"\"Detect SNPs and short indels from pooled sequencing data.\n    https://sites.google.com/site/vibansal/software/crisp/\n    \"\"\"\n    version = \"5\"\n    url = \"https://sites.google.com/site/vibansal/software/crisp/\" \\\n          \"CRISP-linux-v{0}.tar.gz\".format(version)\n    def _make_executable():\n        env.safe_run(\"chmod a+x *.py\")\n    _get_install(url, env, _make_copy(\"ls -1 CRISP.py crisp_to_vcf.py\",\n                                      premake_cmd=_make_executable,\n                                      do_make=False))\n\n@_if_not_installed(\"run_pipeline.pl\")\ndef install_tassel(env):\n    \"\"\"TASSEL: evaluate traits associations, evolutionary patterns, and linkage disequilibrium.\n    http://www.maizegenetics.net/index.php?option=com_content&task=view&id=89&/Itemid=119\n    \"\"\"\n    version = \"5\"\n    build_id = \"1140d3fceb75\"\n    url = \"https://bitbucket.org/tasseladmin/tassel-{0}-standalone/get/{1}.zip\".format(version, build_id)\n    executables = [\"start_tassel.pl\", \"run_pipeline.pl\"]\n    install_dir = _symlinked_java_version_dir(\"tassel\", version, env)\n    if install_dir:\n        with _make_tmp_dir() as work_dir:\n            with cd(work_dir):\n                dl_file = shared._remote_fetch(env, url)\n                env.safe_run(\"unzip %s\" % dl_file)\n                with cd(\"tasseladmin-tassel-{0}-standalone-{1}\".format(version, build_id)):\n                    for x in executables:\n                        env.safe_sed(x, \"^my \\$top.*;\",\n                                     \"use FindBin qw($RealBin); my $top = $RealBin;\")\n                        env.safe_sudo(\"chmod a+rwx %s\" % x)\n                    env.safe_sudo(\"mv * %s\" % install_dir)\n                for x in executables:\n                    env.safe_sudo(\"ln -s %s/%s %s/bin/%s\" % (install_dir, x,\n                                                             env.system_install, x))\n\n@_if_not_installed(\"ustacks\")\ndef install_stacks(env):\n    \"\"\"Stacks: build loci out of a set of short-read sequenced samples.\n    http://creskolab.uoregon.edu/stacks/\n    \"\"\"\n    version = \"0.9999\"\n    url = \"http://creskolab.uoregon.edu/stacks/source/\" \\\n          \"stacks-{0}.tar.gz\".format(version)\n    _get_install(url, env, _configure_make)\n\n@_if_not_installed(\"seqlogo\")\ndef install_weblogo(env):\n    \"\"\"Weblogo\n    http://weblogo.berkeley.edu/\n    \"\"\"\n    version = \"2.8.2\"\n    url = \"http://weblogo.berkeley.edu/release/weblogo.%s.tar.gz\" % version\n    _get_install(url, env, _make_copy(\"find . -perm -100 -type f\", do_make=False))\n    def _cp_pm(env):\n        for perl_module in [\"template.pm\", \"logo.pm\", \"template.eps\"]:\n            env.safe_sudo(\"cp %s %s/lib/perl5\" % (perl_module, env.system_install))\n    _get_install(url, env, _cp_pm(env))\n"
  },
  {
    "path": "cloudbio/custom/bio_proteomics.py",
    "content": "\"\"\"Install proteomics tools not currently packaged.\n\"\"\"\n\nimport os\nimport re\n\nfrom fabric.api import cd\nfrom fabric.context_managers import prefix\n\nfrom shared import (_if_not_installed, _make_tmp_dir,\n                    _get_install, _make_copy,\n                    _java_install, _symlinked_java_version_dir,\n                    _get_bin_dir, _get_install_subdir,\n                    _fetch_and_unpack,\n                    _create_python_virtualenv,\n                    _get_bitbucket_download_url,\n                    _write_to_file)\nfrom cloudbio.galaxy.utils import _chown_galaxy\n\n# Tools from Tabb lab are only available via TeamCity builds that\n# and the artifacts eventually are deleted (I think), storing versions\n# for CloudBioLinux at getgalaxyp.msi.umn.edu for safe keeping.\nPROTEOMICS_APP_ARCHIVE_URL = \"http://getgalaxyp.msi.umn.edu/downloads\"\n\n\n# TODO: Define TPP install root\n@_if_not_installed(\"xinteract\")\ndef install_transproteomic_pipeline(env):\n    \"\"\"\n    \"\"\"\n    ## version should be of form X.X.X-codename\n    default_version = \"4.6.1-occupy\"\n    version = env.get(\"tool_version\", default_version)\n    version_parts = re.match(\"(\\d\\.\\d)\\.(\\d)-(.*)\", version)\n    major_version = version_parts.group(1)\n    revision = version_parts.group(2)\n    codename = version_parts.group(3)\n    if revision == \"0\":\n        download_rev = \"\"\n    else:\n        download_rev = \".%s\" % revision\n    download_version = (\"%s%s\" % (major_version, download_rev))\n    url_pieces = (major_version, codename, revision, download_version)\n    url = 'http://sourceforge.net/projects/sashimi/files/Trans-Proteomic Pipeline (TPP)/TPP v%s (%s) rev %s/TPP-%s.tgz' % url_pieces\n\n    def _chdir_src(work_cmd):\n        def do_work(env):\n            src_dir = \"trans_proteomic_pipeline/src\" if version == \"4.6.1-occupy\" else \"src\"\n            with cd(src_dir):\n                env.safe_append(\"Makefile.config.incl\", \"TPP_ROOT=%s/\" % env[\"system_install\"])\n                env.safe_append(\"Makefile.config.incl\", \"TPP_WEB=/tpp/\")\n                env.safe_append(\"Makefile.config.incl\", \"XSLT_PROC=/usr/bin/xsltproc\")\n                env.safe_append(\"Makefile.config.incl\", \"CGI_USERS_DIR=${TPP_ROOT}cgi-bin\")\n                work_cmd(env)\n        return do_work\n\n    def _make(env):\n        env.safe_run(\"make\")\n        env.safe_sudo(\"make install\")\n    _get_install(url, env, _chdir_src(_make))\n\n\n@_if_not_installed(\"omssacl\")\ndef install_omssa(env):\n    default_version = \"2.1.9\"\n    version = env.get(\"tool_version\", default_version)\n    url = 'ftp://ftp.ncbi.nih.gov/pub/lewisg/omssa/%s/omssa-%s.linux.tar.gz' % (version, version)\n    env.safe_sudo(\"mkdir -p '%s'\" % env[\"system_install\"])\n    ## OMSSA really wants mods.xml, usermods.xml, etc... in the same directory\n    ## so just copying everything there.\n    _get_install(url, env, _make_copy(find_cmd=\"ls -1\", do_make=False))\n\n\n@_if_not_installed(\"OpenMSInfo\")\ndef install_openms(env):\n    \"\"\"\n    See comments above, working on getting this to compile from source. In\n    the meantime installing from deb will have to do.\n    \"\"\"\n    default_version = \"1.10.0\"\n    version = env.get(\"tool_version\", default_version)\n    dot_version = version[0:version.rindex('.')]\n    url = 'http://downloads.sourceforge.net/project/open-ms/OpenMS/OpenMS-%s/OpenMS-%s.tar.gz' % (dot_version, version)\n\n    def _make(env):\n        with cd(\"contrib\"):\n            env.safe_run(\"cmake -DINSTALL_PREFIX=%s .\" % env.get('system_install'))\n            env.safe_run(\"make\")\n        env.safe_run(\"cmake -DINSTALL_PREFIX=%s .\" % env.get('system_install'))\n        env.safe_run(\"make\")\n        env.safe_sudo(\"make install\")\n    _get_install(url, env, _make)\n\n\n@_if_not_installed(\"LTQ-iQuant\")\ndef install_tint_proteomics_scripts(env):\n    default_version = \"1.19.19\"\n    version = env.get(\"tool_version\", default_version)\n    url = \"http://artifactory.msi.umn.edu/simple/ext-release-local/msi/umn/edu/tint-proteomics-scripts/%s/tint-proteomics-scripts-%s.zip\" % (version, version)\n\n    def install_fn(env, install_dir):\n        env.safe_sudo(\"mv * '%s'\" % install_dir)\n        bin_dir = _get_bin_dir(env)\n        for script in [\"ITraqScanSummarizer\", \"LTQ-iQuant\", \"LTQ-iQuant-cli\", \"MgfFormatter\"]:\n            env.safe_sudo(\"ln -s '%s' %s\" % (os.path.join(install_dir, script), bin_dir))\n        env.safe_sudo(\"chmod +x '%s'/*\" % bin_dir)\n\n    _java_install(\"tint-proteomics-scripts\", version, url, env, install_fn)\n\n\n@_if_not_installed(\"ms2preproc\")\ndef install_ms2preproc(env):\n    default_version = \"2009\"\n    version = env.get(\"tool_version\", default_version)\n    get_cmd = 'wget \"http://software.steenlab.org/ms2preproc/ms2preproc.zip\" -O ms2preproc.zip'\n\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            env.safe_run(get_cmd)\n            env.safe_run(\"unzip ms2preproc.zip\")\n            with cd(\"ms2preproc\"):\n                env.safe_run(\"mv ms2preproc-r2821-x86_64 ms2preproc-x86_64\")\n                env.safe_run(\"chmod +x ms2preproc-x86_64\")\n                install_dir = _get_bin_dir(env)\n                env.safe_sudo(\"mv ms2preproc-x86_64 '%s'/ms2preproc\" % install_dir)\n\n\n@_if_not_installed(\"MZmine\")\ndef install_mzmine(env):\n    default_version = \"2.10\"\n    version = env.get(\"tool_version\", default_version)\n    url = \"http://downloads.sourceforge.net/project/mzmine/mzmine2/%s/MZmine-%s.zip\" % (version, version)\n\n    def install_fn(env, install_dir):\n        ## Enhanced MZmine startup script that works when used a symbolic link and tailored for CloudBioLinux.\n        _get_gist_script(env, \"https://gist.github.com/jmchilton/5474421/raw/15f3b817fa82d5f5e2143ee08bd248efee951d6a/MZmine\")\n        # Hack for multi-user environment.\n        env.safe_sudo(\"chmod -R o+w conf\")\n        env.safe_sudo(\"mv * '%s'\" % install_dir)\n        bin_dir = os.path.join(env.get(\"system_install\"), \"bin\")\n        env.safe_sudo(\"mkdir -p '%s'\" % bin_dir)\n        env.safe_sudo(\"ln -s '%s' %s\" % (os.path.join(install_dir, \"MZmine\"), os.path.join(bin_dir, \"MZmine\")))\n\n    _java_install(\"mzmine2\", version, url, env, install_fn)\n\n\n@_if_not_installed(\"SearchGUI\")\ndef install_searchgui(env):\n    default_version = \"1.13.1\"\n    version = env.get(\"tool_version\", default_version)\n    url = \"http://searchgui.googlecode.com/files/SearchGUI-%s_mac_and_linux.zip\" % version\n\n    def install_fn(env, install_dir):\n        dir_name = \"SearchGUI-%s_mac_and_linux\" % version\n        env.safe_sudo(\"tar -xf %s.tar\" % dir_name)\n        with cd(dir_name):\n            _get_gist_script(env, \"https://gist.github.com/jmchilton/5002161/raw/dc9fa36dd0e6eddcdf43cd2b659e4ecee5ad29df/SearchGUI\")\n            _get_gist_script(env, \"https://gist.github.com/jmchilton/5002161/raw/b97fb4d9fe9927de1cfc5433dd1702252e9c0348/SearchCLI\")\n            # Fix known bug with SearchGUI version 1.12.2\n            env.safe_sudo(\"find -iname \\\"*.exe\\\" -exec rename s/.exe// {} \\;\")\n            # Hack for multi-user environment.\n            env.safe_sudo(\"chmod -R o+w resources\")\n            env.safe_sudo(\"mv * '%s'\" % install_dir)\n            bin_dir = os.path.join(env.get(\"system_install\"), \"bin\")\n            env.safe_sudo(\"mkdir -p '%s'\" % bin_dir)\n            env.safe_sudo(\"ln -s '%s' %s\" % (os.path.join(install_dir, \"SearchGUI\"), os.path.join(bin_dir, \"SearchGUI\")))\n            env.safe_sudo(\"ln -s '%s' %s\" % (os.path.join(install_dir, \"SearchCLI\"), os.path.join(bin_dir, \"SearchCLI\")))\n\n    _unzip_install(\"SearchGUI\", version, url, env, install_fn)\n\n\n@_if_not_installed(\"psm_eval\")\ndef install_psm_eval(env):\n    default_version = \"0.1.0\"\n    version = env.get(\"tool_version\", default_version)\n    url = \"git clone https://github.com/jmchilton/psm-eval.git\"\n\n    def install_fn(env, install_dir):\n        env.safe_sudo(\"cp -r psm-eval/* '%s'\" % install_dir)\n        _create_python_virtualenv(env, \"psme\", \"%s/requirements.txt\" % install_dir)\n        bin_dir = os.path.join(env.get(\"system_install\"), \"bin\")\n        env.safe_sudo(\"mkdir -p '%s'\" % bin_dir)\n        env.safe_sudo(\"ln -s '%s' %s\" % (os.path.join(install_dir, \"psm_eval\"), os.path.join(bin_dir, \"psm_eval\")))\n\n    _unzip_install(\"psm_eval\", version, url, env, install_fn)\n\n\n@_if_not_installed(\"PeptideShaker\")\ndef install_peptide_shaker(env):\n    default_version = \"0.20.1\"\n    version = env.get(\"tool_version\", default_version)\n    url = \"http://peptide-shaker.googlecode.com/files/PeptideShaker-%s.zip\" % version\n\n    def install_fn(env, install_dir):\n        _get_gist_script(env, \"https://gist.github.com/jmchilton/5002161/raw/f1fe76d6e6eed99a768ed0b9f41c2d0a6a4b24b7/PeptideShaker\")\n        _get_gist_script(env, \"https://gist.github.com/jmchilton/5002161/raw/8a17d5fb589984365284e55a98a455c2b47da54f/PeptideShakerCLI\")\n        # Hack for multi-user environment.\n        env.safe_sudo(\"chmod -R o+w resources\")\n        env.safe_sudo(\"mv * '%s'\" % install_dir)\n        bin_dir = os.path.join(env.get(\"system_install\"), \"bin\")\n        env.safe_sudo(\"mkdir -p '%s'\" % bin_dir)\n        env.safe_sudo(\"ln -s '%s' %s\" % (os.path.join(install_dir, \"PeptideShaker\"), os.path.join(bin_dir, \"PeptideShaker\")))\n        env.safe_sudo(\"ln -s '%s' %s\" % (os.path.join(install_dir, \"PeptideShakerCLI\"), os.path.join(bin_dir, \"PeptideShakerCLI\")))\n\n    _java_install(\"PeptideShaker\", version, url, env, install_fn)\n\n\ndef _get_gist_script(env, url):\n    name = url.split(\"/\")[-1]\n    env.safe_sudo(\"wget '%s'\" % url)\n    env.safe_sudo(\"chmod +x '%s'\" % name)\n\n\n@_if_not_installed(\"Mayu\")\ndef install_mayu(env):\n    default_version = \"1.06\"\n    version = env.get(\"tool_version\", default_version)\n    url = \"http://proteomics.ethz.ch/muellelu/web/LukasReiter/Mayu/package/Mayu.zip\"\n\n    def install_fn(env, install_dir):\n        share_dir = _get_install_subdir(env, \"share\")\n        env.safe_sudo(\"mv Mayu '%s'\" % share_dir)\n        bin_dir = _get_bin_dir(env)\n        executable = \"%s/Mayu\" % bin_dir\n        env.safe_sudo(\"\"\"echo '#!/bin/bash\\ncd %s/Mayu; perl Mayu.pl \\\"$@\\\"' > %s \"\"\" % (share_dir, executable))\n        env.safe_sudo(\"chmod +x '%s'\" % executable)\n\n    _unzip_install(\"mayu\", version, url, env, install_fn)\n\n\ndef install_pride_inspector(env):\n    default_version = \"1.3.0\"\n    version = env.get(\"tool_version\", default_version)\n    url = \"http://pride-toolsuite.googlecode.com/files/pride-inspector-%s.zip\" % version\n\n    def install_fn(env, install_dir):\n        _get_gist_script(env, \"https://gist.github.com/jmchilton/5474788/raw/6bcffd8680ec0e0301af44961184529a1f76dd3b/pride-inspector\")\n        # Hack for multi-user environment.\n        env.safe_sudo(\"chmod -R o+w log config\")\n        env.safe_sudo(\"mv * '%s'\" % install_dir)\n        bin_dir = os.path.join(env.get(\"system_install\"), \"bin\")\n        env.safe_sudo(\"mkdir -p '%s'\" % bin_dir)\n        env.safe_sudo(\"ln -s '%s' %s\" % (os.path.join(install_dir, \"pride-inspector\"), os.path.join(bin_dir, \"pride-inspector\")))\n\n    _unzip_install(\"pride_inspector\", version, url, env, install_fn, \"PRIDE_Inspector\")\n\n\ndef install_pride_converter2(env):\n    default_version = \"2.0.17\"\n    version = env.get(\"tool_version\", default_version)\n    url = \"http://pride-converter-2.googlecode.com/files/pride-converter-%s-bin.zip\" % version\n\n    def install_fn(env, install_dir):\n        _get_gist_script(env, \"https://gist.github.com/jmchilton/5475119/raw/4e9135ada5114ba149f3ebc8965aee242bfc776f/pride-converter\")\n        # Hack for multi-user environment.\n        env.safe_sudo(\"mkdir log; chmod o+w log\")\n        env.safe_sudo(\"mv * '%s'\" % install_dir)\n        bin_dir = os.path.join(env.get(\"system_install\"), \"bin\")\n        env.safe_sudo(\"mkdir -p '%s'\" % bin_dir)\n        env.safe_sudo(\"ln -s '%s' %s\" % (os.path.join(install_dir, \"pride-converter\"), os.path.join(bin_dir, \"pride-converter\")))\n\n    _unzip_install(\"pride_converter2\", version, url, env, install_fn, \".\")\n\n\ndef _unzip_install(pname, version, url, env, install_fn, dir_name=\".\"):\n    install_dir = _symlinked_java_version_dir(pname, version, env)\n    if install_dir:\n        with _make_tmp_dir() as work_dir:\n            with cd(work_dir):\n                _fetch_and_unpack(url, need_dir=False)\n                with cd(dir_name):\n                    install_fn(env, install_dir)\n\n\n@_if_not_installed(\"SuperHirnv03\")\ndef install_superhirn(env):\n    default_version = \"0.03\"\n    version = env.get(\"tool_version\", default_version)\n    url = \"https://github.com/jmchilton/SuperHirn/zipball/%s/SuperHirn.zip\" % version\n\n    def _chdir(work_cmd):\n        def do_work(env):\n            with cd(\"SuperHirnv03/make\"):\n                work_cmd(env)\n        return do_work\n\n    _get_install(url, env, _chdir(_make_copy(find_cmd=\"find -perm -100 -name 'SuperHirn*'\")))\n\n\n@_if_not_installed(\"percolator\")\ndef install_percolator(env):\n    default_version = \"2_04\"\n    version = env.get(\"tool_version\", default_version)\n    url = \"https://github.com/downloads/percolator/percolator/percolator_%s_full_src.tar.gz\" % version\n\n    def make(env):\n        with cd(\"..\"):\n            env.safe_run(\"env\")\n            env.safe_run(\"cmake -DCMAKE_INSTALL_PREFIX='%s' . \" % env.system_install)\n            env.safe_run(\"make -j8\")\n            env.safe_sudo(\"make install\")\n\n    _get_install(url, env, make)\n\n\n@_if_not_installed(\"PepNovo\")\ndef install_pepnovo(env):\n    default_version = \"20120423\"\n    version = env.get(\"tool_version\", default_version)\n    url = \"http://proteomics.ucsd.edu/Downloads/PepNovo.%s.zip\" % version\n\n    def install_fn(env, install_dir):\n        with cd(\"src\"):\n            env.safe_run(\"make\")\n            env.safe_sudo(\"mkdir -p '%s/bin'\" % env.system_install)\n            env.safe_sudo(\"mkdir -p '%s/share/pepnovo'\" % env.system_install)\n            env.safe_sudo(\"mv PepNovo_bin '%s/bin/PepNovo'\" % env.system_install)\n            env.safe_sudo(\"cp -r '../Models' '%s/share/pepnovo'\" % env.system_install)\n\n    _unzip_install(\"pepnovo\", version, url, env, install_fn)\n\n\n@_if_not_installed(\"crux\")\ndef install_crux(env):\n    default_version = \"1.39\"\n    version = env.get(\"tool_version\", default_version)\n    url = \"http://noble.gs.washington.edu/proj/crux/download/crux_%s-x86_64-Linux.zip\" % version\n\n    def _move(env):\n        bin_dir = _get_bin_dir(env)\n        env.safe_sudo(\"mv bin/* '%s'\" % (bin_dir))\n\n    _get_install(url, env, _move)\n\n\n@_if_not_installed(\"Fido\")\ndef install_fido(env):\n    version = \"2011\"\n    url = 'http://noble.gs.washington.edu/proj/fido/fido.tar.gz'\n\n    # Adapted from Jorrit Boekel's mi-deployment fork\n    # https://bitbucket.org/glormph/mi-deployment-protoeimcs\n    def _chdir_src(work_cmd):\n        def do_work(env):\n            with cd(\"src/cpp\"):\n                env.safe_append('tmpmake', 'SHELL=/bin/bash')\n                env.safe_append('tmpmake', 'prefix=%s' % env.get(\"system_install\"))\n                env.safe_append('tmpmake', 'CPPFLAGS=-Wall -ffast-math -march=x86-64 -pipe -O4 -g')\n                env.safe_run('cat makefile |grep BINPATH -A 9999 >> tmpmake')\n                env.safe_run('cp tmpmake makefile')\n                work_cmd(env)\n        return do_work\n\n    _get_install(url, env, _chdir_src(_make_copy(find_cmd=\"find ../../bin -perm -100 -name 'Fido*'\")))\n\n\ndef install_ipig(env):\n    \"\"\" This tool is installed in Galaxy's jars dir \"\"\"\n    # This galaxy specific download probable doesn't belong in this file.\n    default_version = \"r5\"\n    version = env.get(\"tool_version\", default_version)\n    url = 'http://downloads.sourceforge.net/project/ipig/ipig_%s.zip' % version\n    pkg_name = 'ipig'\n    install_dir = os.path.join(env.galaxy_jars_dir, pkg_name)\n    install_cmd = env.safe_sudo if env.use_sudo else env.safe_run\n    install_cmd(\"mkdir -p %s\" % install_dir)\n    with cd(install_dir):\n        install_cmd(\"wget %s -O %s\" % (url, os.path.split(url)[-1]))\n        install_cmd(\"unzip -u %s\" % (os.path.split(url)[-1]))\n        install_cmd(\"rm %s\" % (os.path.split(url)[-1]))\n        install_cmd('chown --recursive %s:%s %s' % (env.galaxy_user, env.galaxy_user, install_dir))\n\n\ndef install_peptide_to_gff(env):\n    default_version = \"master\"\n    version = env.get(\"tool_version\", default_version)\n    repository = \"hg clone https://jmchilton@bitbucket.org/galaxyp/peptide_to_gff\"\n\n    def install_fn(env, install_dir):\n        env.safe_sudo(\"cp -r peptide_to_gff/* '%s'\" % install_dir)\n        _create_python_virtualenv(env, \"peptide_to_gff\", \"%s/requirements.txt\" % install_dir)\n        bin_dir = os.path.join(env.get(\"system_install\"), \"bin\")\n        env.safe_sudo(\"mkdir -p '%s'\" % bin_dir)\n        env.safe_sudo(\"ln -s '%s' '%s'\" % (os.path.join(install_dir, \"peptide_to_gff\"), os.path.join(bin_dir, \"peptide_to_gff\")))\n\n    _unzip_install(\"peptide_to_gff\", version, repository, env, install_fn)\n\n\ndef install_galaxy_protk(env):\n    \"\"\"This method installs Ira Cooke's ProtK framework. Very galaxy specific,\n    can only be installed in context of custom Galaxy tool.\n\n\n    By default this will install ProtK from rubygems server, but if\n    env.protk_version is set to <version>@<url> (e.g.\n    1.1.5@https://bitbucket.org/iracooke/protk-working) the\n    gem will be cloned with hg and installed from source.\n    \"\"\"\n    if not env.get('galaxy_tool_install', False):\n        from cloudbio.custom.galaxy import _prep_galaxy\n        _prep_galaxy(env)\n    default_version = \"1.2.2\"\n    version = env.get(\"tool_version\", default_version)\n    version_and_revision = version\n    install_from_source = version_and_revision.find(\"@\") > 0\n    # e.g. protk_version = 1.1.5@https://bitbucket.org/iracooke/protk-working\n    if install_from_source:\n        (version, revision) = version_and_revision.split(\"@\")\n        url = _get_bitbucket_download_url(revision, \"https://bitbucket.org/iracooke/protk\")\n    else:\n        version = version_and_revision\n\n    ruby_version = \"1.9.3\"\n    force_rvm_install = False\n    with prefix(\"HOME=~%s\" % env.galaxy_user):\n        def rvm_exec(env, cmd=\"\", rvm_cmd=\"use\", with_gemset=False):\n            target = ruby_version if not with_gemset else \"%s@%s\" % (ruby_version, \"protk-%s\" % version)\n            prefix = \". $HOME/.rvm/scripts/rvm; rvm %s %s; \" % (rvm_cmd, target)\n            env.safe_sudo(\"%s %s\" % (prefix, cmd), user=env.galaxy_user)\n        if not env.safe_exists(\"$HOME/.rvm\") or force_rvm_install:\n            env.safe_sudo(\"curl -L get.rvm.io | bash -s stable; source ~%s/.rvm/scripts/rvm\" % (env.galaxy_user), user=env.galaxy_user)\n            rvm_exec(env, rvm_cmd=\"install\")\n            rvm_exec(env, cmd=\"rvm gemset create protk-%s\" % version)\n        if not install_from_source:\n            # Typical rubygem install\n            rvm_exec(env, \"gem install  --no-ri --no-rdoc protk -v %s\" % version, with_gemset=True)\n        else:\n            with cd(\"~%s\" % env.galaxy_user):\n                env.safe_sudo(\"rm -rf protk_source; hg clone '%s' protk_source\" % url, user=env.galaxy_user)\n                rvm_exec(env, \"cd protk_source; gem build protk.gemspec; gem install protk\", with_gemset=True)\n\n        protk_properties = {}\n        ## ProtK can set these up itself, should make that an option.\n        protk_properties[\"tpp_root\"] = os.path.join(env.galaxy_tools_dir, \"transproteomic_pipeline\", \"default\")\n        protk_properties['openms_root'] = \"/usr\"  # os.path.join(env.galaxy_tools_dir, \"openms\", \"default\", \"bin\")\n        ### Assumes omssa, blast, and transproteomic_pipeline CBL galaxy installs.\n        protk_properties['omssa_root'] = os.path.join(env.galaxy_tools_dir, \"omssa\", \"default\", \"bin\")\n        protk_properties['blast_root'] = os.path.join(env.galaxy_tools_dir, \"blast+\", \"default\")\n        protk_properties['pwiz_root'] = os.path.join(env.galaxy_tools_dir, \"transproteomic_pipeline\", \"default\", \"bin\")\n        # Other properties: log_file, blast_root\n        env.safe_sudo(\"mkdir -p \\\"$HOME/.protk\\\"\", user=env.galaxy_user)\n        env.safe_sudo(\"mkdir -p \\\"$HOME/.protk/Databases\\\"\", user=env.galaxy_user)\n        import  yaml\n        _write_to_file(yaml.dump(protk_properties), \"/home/%s/.protk/config.yml\" % env.galaxy_user, \"0755\")\n\n        rvm_exec(env, \"protk_setup.rb galaxyenv\", with_gemset=True)\n\n        install_dir = os.path.join(env.galaxy_tools_dir, \"galaxy_protk\", version)\n        env.safe_sudo(\"mkdir -p '%s'\" % install_dir)\n        _chown_galaxy(env, install_dir)\n        env.safe_sudo('ln -s -f \"$HOME/.protk/galaxy/env.sh\" \"%s/env.sh\"' % install_dir, user=env.galaxy_user)\n        with cd(install_dir):\n            with cd(\"..\"):\n                env.safe_sudo(\"ln -s -f '%s' default\" % version)\n\n\n@_if_not_installed(\"myrimatch\")\ndef install_myrimatch(env):\n    default_version = \"2.1.131\"\n    _install_tabb_tool(env, default_version, \"myrimatch-bin-linux-x86_64-gcc41-release\", [\"myrimatch\"])\n\n\n@_if_not_installed(\"pepitome\")\ndef install_pepitome(env):\n    default_version = \"1.0.45\"\n    _install_tabb_tool(env, default_version, \"pepitome-bin-linux-x86_64-gcc41-release\", [\"pepitome\"])\n\n\n@_if_not_installed(\"directag\")\ndef install_directag(env):\n    default_version = \"1.3.62\"\n    _install_tabb_tool(env, default_version, \"directag-bin-linux-x86_64-gcc41-release\", [\"adjustScanRankerScoreByGroup\", \"directag\"])\n\n\n@_if_not_installed(\"tagrecon\")\ndef install_tagrecon(env):\n    default_version = \"1.4.63\"\n    # TODO: Should consider a better way to handle the unimod xml and blosum matrix.\n    _install_tabb_tool(env, default_version, \"tagrecon-bin-linux-x86_64-gcc41-release\", [\"tagrecon\", \"unimod.xml\", \"blosum62.fas\"])\n\n\n@_if_not_installed(\"idpQonvert\")\ndef install_idpqonvert(env):\n    default_version = \"3.0.475\"\n    version = env.get(\"tool_version\", default_version)\n    url = \"%s/idpQonvert_%s\" % (PROTEOMICS_APP_ARCHIVE_URL, version)\n    env.safe_run(\"wget --no-check-certificate -O %s '%s'\" % (\"idpQonvert\", url))\n    env.safe_run(\"chmod 755 idpQonvert\")\n    env.safe_sudo(\"mkdir -p '%s/bin'\" % env[\"system_install\"])\n    env.safe_sudo(\"mv %s '%s/bin'\" % (\"idpQonvert\", env[\"system_install\"]))\n    env.safe_sudo(\"chmod +x '%s/bin/idpQonvert'\" % env[\"system_install\"])\n\n\ndef _install_tabb_tool(env, default_version, download_name, exec_names):\n    version = env.get(\"tool_version\", default_version)\n    url = \"%s/%s-%s.tar.bz2\" \\\n        % (PROTEOMICS_APP_ARCHIVE_URL, download_name, version.replace(\".\", \"_\"))\n    _fetch_and_unpack(url, False)\n    env.safe_sudo(\"mkdir -p '%s/bin'\" % env[\"system_install\"])\n    for exec_name in exec_names:\n        env.safe_sudo(\"mv %s '%s/bin'\" % (exec_name, env[\"system_install\"]))\n"
  },
  {
    "path": "cloudbio/custom/bio_proteomics_wine.py",
    "content": "from fabric.api import cd\n\nfrom shared import (_make_tmp_dir, _fetch_and_unpack, _write_to_file, _get_bin_dir)\n\nimport os\n\n\ndef install_proteomics_wine_env(env):\n    script_src = env.get(\"setup_proteomics_wine_env_script\")\n    script_dest = \"%s/bin/setup_proteomics_wine_env.sh\" % env.get(\"system_install\")\n    if not env.safe_exists(script_dest):\n        env.safe_put(script_src, script_dest, mode=\"0755\", use_sudo=True)\n\n\ndef install_multiplierz(env):\n    \"\"\"\n    Assumes your wine environment contains an install Python 2.6\n    in C:\\Python26.\n    \"\"\"\n    wine_user = _get_wine_user(env)\n\n    install_proteomics_wine_env(env)\n    env.safe_sudo(\"setup_proteomics_wine_env.sh\", user=wine_user)\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            _fetch_and_unpack(\"hg clone http://multiplierz.hg.sourceforge.net:8000/hgroot/multiplierz/multiplierz\")\n            with cd(\"multiplierz\"):\n                wine_prefix = _get_wine_prefix(env)\n                env.safe_sudo(\"%s; wine %s/drive_c/Python26/python.exe setup.py install\" % (_conf_wine(env), wine_prefix), user=wine_user)\n\n\ndef install_proteowizard(env):\n    build_id = \"85131\"\n    version = \"3_0_4624\"\n    url = \"http://teamcity.labkey.org:8080/repository/download/bt36/%s:id/pwiz-bin-windows-x86-vc100-release-%s.tar.bz2?guest=1\" % (build_id, version)\n    install_dir = env.get(\"install_dir\")\n    share_dir = \"%s/share/proteowizard\" % install_dir\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            _fetch_and_unpack(url, need_dir=False)\n            env.safe_sudo(\"cp -r . '%s'\" % share_dir)\n    proteowizard_apps = [\"msconvert\", \"msaccess\", \"chainsaw\", \"msdiff\", \"mspicture\", \"mscat\", \"txt2mzml\", \"MSConvertGUI\", \"Skyline\", \"Topograph\", \"SeeMS\"]\n    for app in proteowizard_apps:\n        setup_wine_wrapper(env, \"%s/%s\" % (share_dir, app))\n\n\ndef install_morpheus(env):\n    url = \"http://www.chem.wisc.edu/~coon/Downloads/Morpheus/latest/Morpheus.zip\"  # TODO:\n    install_dir = env.get(\"install_dir\")\n    share_dir = \"%s/share/morpheus\" % install_dir\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            _fetch_and_unpack(url, need_dir=False)\n            env.safe_sudo(\"cp -r Morpheus '%s'\" % share_dir)\n    morpheus_exes = [\"morpheus_cl.exe\", \"Morpheus.exe\"]\n    for app in morpheus_exes:\n        setup_wine_wrapper(env, \"%s/%s\" % (share_dir, app))\n\n\ndef setup_wine_wrapper(env, to):\n    basename = os.path.basename(to)\n    contents = \"\"\"#!/bin/bash\nsetup_proteomics_wine_env.sh\nexport WINEPREFIX=$HOME/.wine-proteomics\nwine %s \"$@\"\n\"\"\" % to\n    bin_dir = _get_bin_dir(env)\n    dest = \"%s/%s\" % (bin_dir, basename)\n    _write_to_file(contents, dest, '0755')\n\n\ndef _conf_wine(env):\n    return \"export WINEPREFIX=%s\" % _get_wine_prefix(env)\n\n\ndef _get_wine_prefix(env):\n    wine_user = _get_wine_user(env)\n    return \"~%s/.wine-proteomics\" % wine_user\n\n\ndef _get_wine_user(env):\n    return env.get(\"wine_user\", env.get(\"user\"))\n"
  },
  {
    "path": "cloudbio/custom/cloudman.py",
    "content": "\"\"\"Custom install scripts for CloudMan environment.\n\nFrom Enis Afgan: https://bitbucket.org/afgane/mi-deployment\n\"\"\"\nimport os\nimport contextlib\n\nfrom fabric.api import cd\nfrom fabric.contrib.files import settings, hide\n\nfrom cloudbio.custom.shared import (_make_tmp_dir, _setup_conf_file)\nfrom cloudbio.cloudman import (_configure_cloudman, _configure_novnc,\n    _configure_desktop, _configure_ec2_autorun)\nfrom cloudbio.galaxy import _install_nginx\n\nCDN_ROOT_URL = \"http://linuxcourse.rutgers.edu/rate/Clusters/download\"\nREPO_ROOT_URL = \"https://bitbucket.org/afgane/mi-deployment/raw/tip\"\n\n\ndef install_cloudman(env):\n    \"\"\" A meta method for installing all of CloudMan components.\n        Allows CloudMan and all of its dependencies to be installed via:\n        fab -f fabfile.py -i <key> -H ubuntu@<IP> install_custom:cloudman\n    \"\"\"\n    env.logger.debug(\"Installing CloudMan\")\n    _configure_cloudman(env, use_repo_autorun=False)\n    install_nginx(env)\n    install_proftpd(env)\n    install_sge(env)\n    install_novnc(env)\n\n\ndef install_ec2_autorun(env):\n    _configure_ec2_autorun(env)\n\n\ndef install_novnc(env):\n    _configure_novnc(env)\n    _configure_desktop(env)\n\n\ndef install_nginx(env):\n    _install_nginx(env)\n\n\ndef install_proftpd(env):\n    \"\"\"Highly configurable GPL-licensed FTP server software.\n    http://proftpd.org/\n    \"\"\"\n    version = \"1.3.4c\"\n    postgres_ver = \"9.1\"\n    url = \"ftp://ftp.tpnet.pl/pub/linux/proftpd/distrib/source/proftpd-%s.tar.gz\" % version\n    modules = \"mod_sql:mod_sql_postgres:mod_sql_passwd\"\n    extra_modules = env.get(\"extra_proftp_modules\", \"\")  # Comma separated list of extra modules\n    if extra_modules:\n        modules = \"%s:%s\" % (modules, extra_modules.replace(\",\", \":\"))\n    install_dir = os.path.join(env.install_dir, 'proftpd')\n    remote_conf_dir = os.path.join(install_dir, \"etc\")\n    # Skip install if already available\n    if env.safe_exists(remote_conf_dir):\n        env.logger.debug(\"ProFTPd seems to already be installed in {0}\".format(install_dir))\n        return\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            env.safe_run(\"wget %s\" % url)\n            with settings(hide('stdout')):\n                env.safe_run(\"tar xvzf %s\" % os.path.split(url)[1])\n            with cd(\"proftpd-%s\" % version):\n                env.safe_run(\"CFLAGS='-I/usr/include/postgresql' ./configure --prefix=%s \"\n                    \"--disable-auth-file --disable-ncurses --disable-ident --disable-shadow \"\n                    \"--enable-openssl --with-modules=%s \"\n                    \"--with-libraries=/usr/lib/postgresql/%s/lib\" % (install_dir, modules, postgres_ver))\n                env.safe_sudo(\"make\")\n                env.safe_sudo(\"make install\")\n                env.safe_sudo(\"make clean\")\n    # Get the init.d startup script\n    initd_script = 'proftpd.initd'\n    initd_url = os.path.join(REPO_ROOT_URL, 'conf_files', initd_script)\n    remote_file = \"/etc/init.d/proftpd\"\n    env.safe_sudo(\"wget --output-document=%s %s\" % (remote_file, initd_url))\n    env.safe_sed(remote_file, 'REPLACE_THIS_WITH_CUSTOM_INSTALL_DIR', install_dir, use_sudo=True)\n    env.safe_sudo(\"chmod 755 %s\" % remote_file)\n    # Set the configuration file\n    conf_file = 'proftpd.conf'\n    remote_file = os.path.join(remote_conf_dir, conf_file)\n    if \"postgres_port\" not in env:\n        env.postgres_port = '5910'\n    if \"galaxy_ftp_user_password\" not in env:\n        env.galaxy_ftp_user_password = 'fu5yOj2sn'\n    proftpd_conf = {'galaxy_uid': env.safe_run('id -u galaxy'),\n                    'galaxy_fs': '/mnt/galaxy',  # Should be a var but uncertain how to get it\n                    'install_dir': install_dir}\n    _setup_conf_file(env, remote_file, conf_file, overrides=proftpd_conf,\n        default_source=\"proftpd.conf.template\")\n    # Get the custom welcome msg file\n    welcome_msg_file = 'welcome_msg.txt'\n    welcome_url = os.path.join(REPO_ROOT_URL, 'conf_files', welcome_msg_file)\n    env.safe_sudo(\"wget --output-document=%s %s\" %\n       (os.path.join(remote_conf_dir, welcome_msg_file), welcome_url))\n    # Stow\n    env.safe_sudo(\"cd %s; stow proftpd\" % env.install_dir)\n    env.logger.debug(\"----- ProFTPd %s installed to %s -----\" % (version, install_dir))\n\n\ndef install_sge(env):\n    \"\"\"Sun Grid Engine.\n    \"\"\"\n    out_dir = \"ge6.2u5\"\n    url = \"%s/ge62u5_lx24-amd64.tar.gz\" % CDN_ROOT_URL\n    install_dir = env.install_dir\n    if env.safe_exists(os.path.join(install_dir, out_dir)):\n        return\n    with _make_tmp_dir() as work_dir:\n        with contextlib.nested(cd(work_dir), settings(hide('stdout'))):\n            env.safe_run(\"wget %s\" % url)\n            env.safe_sudo(\"chown %s %s\" % (env.user, install_dir))\n            env.safe_run(\"tar -C %s -xvzf %s\" % (install_dir, os.path.split(url)[1]))\n    env.logger.debug(\"SGE setup\")\n"
  },
  {
    "path": "cloudbio/custom/distributed.py",
    "content": "\"\"\"Install instructions for distributed MapReduce style programs.\n\"\"\"\nimport os\n\nfrom fabric.api import *\nfrom fabric.contrib.files import *\n\nfrom shared import (_if_not_python_lib, _pip_cmd, _is_anaconda)\n\n@_if_not_python_lib(\"pydoop\")\ndef install_pydoop(env):\n    \"\"\"pydoop; provides Hadoop access for Python.\n    http://pydoop.sourceforge.net/docs/\n    \"\"\"\n    java_home = env.java_home if \"java_home\" in env else os.environ[\"JAVA_HOME\"]\n    export_str = \"export JAVA_HOME=%s\" % (java_home)\n    cmd = env.safe_run if _is_anaconda(env) else env.safe_sudo\n    cmd(\"%s && %s install pydoop\" % (export_str, _pip_cmd(env)))\n\n@_if_not_python_lib(\"bl.mr.seq.seqal\")\ndef install_seal(env):\n    \"\"\"Install seal: process high-throughput sequencing with Hadoop.\n\n    http://biodoop-seal.sf.net/\n    \"\"\"\n    install_pydoop(env)\n\n    java_home = env.java_home if \"java_home\" in env else os.environ[\"JAVA_HOME\"]\n    export_str = \"export JAVA_HOME=%s\" % (java_home)\n    cmd = env.safe_run if _is_anaconda(env) else env.safe_sudo\n    cmd(\"%s && %s install --pre seal\" % (export_str, _pip_cmd(env)))\n"
  },
  {
    "path": "cloudbio/custom/galaxy.py",
    "content": "\"\"\"\nInstall any components that fall under 'galaxy' directive in main.yaml\n\"\"\"\nfrom cloudbio.galaxy import _setup_users\nfrom cloudbio.galaxy import _setup_galaxy_env_defaults\nfrom cloudbio.galaxy import _install_galaxy\nfrom cloudbio.galaxy import _configure_galaxy_options\n\n\ndef install_galaxy_webapp(env):\n    _prep_galaxy(env)\n    _install_galaxy(env)\n    _configure_galaxy_options(env)\n\n\ndef _prep_galaxy(env):\n    _setup_users(env)\n    _setup_galaxy_env_defaults(env)\n"
  },
  {
    "path": "cloudbio/custom/galaxy_tools.py",
    "content": "\"\"\"\nInstall any components that fall under 'galaxy_tools' directive in main.yaml\n\"\"\"\nfrom cloudbio.galaxy.tools import _install_tools\nfrom cloudbio.custom.galaxy import _prep_galaxy\n\n\ndef install_cbl_galaxy_tools(env):\n    _prep_galaxy(env)\n    _install_tools(env)\n"
  },
  {
    "path": "cloudbio/custom/galaxyp.py",
    "content": "\"\"\"\n\"\"\"\n\nfrom cloudbio.galaxy.utils import _chown_galaxy\n\nfrom fabric.contrib.files import *\n\nfrom shared import _write_to_file\n\n\ndef install_protvis(env):\n    \"\"\" Installs Andrew Brock's proteomics visualize tool.\n    https://bitbucket.org/Andrew_Brock/proteomics-visualise/\n    \"\"\"\n    _setup_protvis_env(env)\n    protvis_home = env[\"protvis_home\"]\n    env.safe_sudo(\"sudo apt-get -y --force-yes install libxml2-dev libxslt-dev\")\n\n    run(\"rm -rf protvis\")\n    run(\"git clone -b lorikeet https://github.com/jmchilton/protvis.git\")\n    with cd(\"protvis\"):\n        run(\"git submodule init\")\n        run(\"git submodule update\")\n        env.safe_sudo(\"rsync -avur --delete-after . %s\" % (protvis_home))\n        _chown_galaxy(env, protvis_home)\n        with cd(protvis_home):\n            env.safe_sudo(\"./setup.sh\", user=env.get(\"galaxy_user\", \"galaxy\"))\n\n    #default_revision = \"8cc6af1c492c\"\n    #\n    #revision = env.get(\"protvis_revision\", default_revision)\n    #url = _get_bitbucket_download_url(revision, \"https://bitbucket.org/Andrew_Brock/proteomics-visualise\")\n    #def _make(env):\n    #_get_install(url, env, _make)\n\n    galaxy_data_dir = env.get('galaxy_data_dir', \"/mnt/galaxyData/\")\n    protvis_converted_files_dir = env.get('protvis_converted_files_dir')\n    _write_to_file('''GALAXY_ROOT = \"%s\"\nPATH_WHITELIST = [\"%s/files/\", \"%s\"]\nCONVERTED_FILES = \"%s\"\n''' % (env.galaxy_home, galaxy_data_dir, protvis_converted_files_dir, protvis_converted_files_dir), \"%s/conf.py\" % protvis_home, \"0755\")\n    _setup_protvis_service(env)\n\n\ndef _setup_protvis_env(env):\n    if not \"protvis_home\" in env:\n        env[\"protvis_home\"] = \"%s/%s\" % (env.galaxy_tools_dir, \"protvis\")\n    if not \"protvis_user\" in env:\n        env[\"protvis_user\"] = \"galaxy\"\n    if not \"protvis_port\" in env:\n        env[\"protvis_port\"] = \"8500\"\n    if not \"protvis_converted_files_dir\" in env:\n        galaxy_data_dir = env.get('galaxy_data_dir', \"/mnt/galaxyData/\")\n        env['protvis_converted_files_dir'] = \"%s/tmp/protvis\" % galaxy_data_dir\n\n\ndef _setup_protvis_service(env):\n    _setup_conf_file(env, os.path.join(\"/etc/init.d/protvis\"), \"protvis_init\", default_source=\"protvis_init\")\n    _setup_conf_file(env, os.path.join(\"/etc/default/protvis\"), \"protvis_default\")\n    _setup_simple_service(\"protvis\")\n"
  },
  {
    "path": "cloudbio/custom/java.py",
    "content": "\"\"\"Install instructions for non-packaged java programs.\n\"\"\"\nimport os\n\nfrom fabric.api import *\nfrom fabric.contrib.files import *\n\nfrom shared import (_if_not_installed, _make_tmp_dir)\nfrom cloudbio.custom import shared\n\n@_if_not_installed(\"lein -v\")\ndef install_leiningen(env):\n    \"\"\"Clojure tool for project configuration and automation.\n    http://github.com/technomancy/leiningen\n    \"\"\"\n    bin_dir = os.path.join(env.system_install, \"bin\")\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            shared._remote_fetch(env, \"https://raw.github.com/technomancy/leiningen/stable/bin/lein\")\n            env.safe_run(\"chmod a+rwx lein\")\n            env.safe_sudo(\"mv lein %s\" % bin_dir)\n            env.safe_run(\"%s/lein\" % bin_dir)\n"
  },
  {
    "path": "cloudbio/custom/millstone.py",
    "content": "\"\"\"Install instructions for non-packaged programs required by Millstone.\n\"\"\"\n\nfrom fabric.api import cd\n\nfrom cloudbio.custom.shared import _make_tmp_dir\n\n\ndef install_unafold(env):\n    \"\"\"Required by optmage.\n    \"\"\"\n    # Since unafold is distributed as an .rpm, we need the program alien to\n    # convert it into a .deb that can be installed on this system.\n    env.safe_sudo(\"apt-get install -y alien\")\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            env.safe_run(\"wget http://dinamelt.rit.albany.edu/download/unafold-3.8-1.x86_64.rpm\")\n            env.safe_sudo(\"alien -i unafold-3.8-1.x86_64.rpm\")\n"
  },
  {
    "path": "cloudbio/custom/phylogeny.py",
    "content": "\"\"\"Install instructions for non-packaged phyologeny programs.\n\"\"\"\nimport os\n\nfrom fabric.api import *\nfrom fabric.contrib.files import *\n\nfrom cloudbio.custom.shared import _if_not_installed, _make_tmp_dir\n\ndef install_tracer(env):\n    \"\"\"A program for analysing results from Bayesian MCMC programs such as BEAST & MrBayes.\n    http://tree.bio.ed.ac.uk/software/tracer/\n    \"\"\"\n    version = \"1.5\"\n    install_dir = os.path.join(env.system_install, \"bioinf\")\n    final_exe = os.path.join(env.system_install, \"bin\", \"tracer\")\n    if env.safe_exists(final_exe):\n        return\n    if not env.safe_exists(final_exe):\n        with _make_tmp_dir() as work_dir:\n            with cd(work_dir):\n                env.safe_run(\"wget -O Tracer_v{0}.tgz 'http://tree.bio.ed.ac.uk/download.php?id=80&num=3'\".format(\n                    version))\n                env.safe_run(\"tar xvzf Tracer_v{0}.tgz\".format(version))\n                env.safe_run(\"chmod a+x Tracer_v{0}/bin/tracer\".format(version))\n                env.safe_sudo(\"mkdir -p %s\" % install_dir)\n                env.safe_sudo(\"rm -rvf %s/tracer\" % install_dir)\n                env.safe_sudo(\"mv -f Tracer_v%s %s/tracer\" % (version, install_dir))\n                env.safe_sudo(\"ln -sf %s/tracer/bin/tracer %s\" % (install_dir, final_exe))\n\n@_if_not_installed(\"beast -help\")\ndef install_beast(env):\n    \"\"\"BEAST: Bayesian MCMC analysis of molecular sequences.\n    http://beast.bio.ed.ac.uk\n    \"\"\"\n    version = \"1.7.4\"\n    install_dir = os.path.join(env.system_install, \"bioinf\")\n    final_exe = os.path.join(env.system_install, \"bin\", \"beast\")\n    if not env.safe_exists(final_exe):\n        with _make_tmp_dir() as work_dir:\n            with cd(work_dir):\n                env.safe_run(\"wget -c http://beast-mcmc.googlecode.com/files/BEASTv%s.tgz\" % version)\n                env.safe_run(\"tar xvzf BEASTv%s.tgz\" % version)\n                env.safe_sudo(\"mkdir -p %s\" % install_dir)\n                env.safe_sudo(\"rm -rvf %s/beast\" % install_dir)\n                env.safe_sudo(\"mv -f BEASTv%s %s/beast\" % (version, install_dir))\n                for l in [\"beast\",\"beauti\",\"loganalyser\",\"logcombiner\",\"treeannotator\",\"treestat\"]:\n                    env.safe_sudo(\"ln -sf %s/beast/bin/%s %s/bin/%s\" % (install_dir, l,\n                                                                        env.system_install, l))\n\n"
  },
  {
    "path": "cloudbio/custom/python.py",
    "content": "\"\"\"Install instructions for python libraries not ready for easy_install.\n\"\"\"\nimport os\n\nfrom fabric.api import *\nfrom fabric.contrib.files import *\n\nfrom shared import (_if_not_python_lib, _get_install, _python_make, _pip_cmd,\n                    _is_anaconda)\n\n@_if_not_python_lib(\"bx\")\ndef install_bx_python(env):\n    \"\"\"Tools for manipulating biological data, particularly multiple sequence alignments\n    https://bitbucket.org/james_taylor/bx-python/wiki/Home\n    \"\"\"\n    version = \"bitbucket\"\n    url = \"https://bitbucket.org/james_taylor/bx-python/get/tip.tar.bz2\"\n    cmd = env.safe_run if _is_anaconda(env) else env.safe_sudo\n    if not _is_anaconda(env):\n        cmd(\"%s install --upgrade distribute\" % _pip_cmd(env))\n    cmd(\"%s install --upgrade %s\" % (_pip_cmd(env), url))\n\n@_if_not_python_lib(\"rpy\")\ndef install_rpy(env):\n    \"\"\"RPy is a very simple, yet robust, Python interface to the R Programming Language.\n    http://rpy.sourceforge.net/\n    \"\"\"\n    version = \"1.0.3\"\n    ext = \"a\"\n    url = \"http://downloads.sourceforge.net/project/rpy/rpy/\" \\\n          \"%s/rpy-%s%s.zip\" % (version, version, ext)\n    def _fix_libraries(env):\n        env.safe_run(\"\"\"sed -i.bak -r -e \"s/,'Rlapack'//g\" setup.py\"\"\")\n    with settings(hide('warnings', 'running', 'stdout', 'stderr'),\n                  warn_only=True):\n        result = env.safe_run(\"R --version\")\n        if result.failed:\n            return\n    _get_install(url, env, _python_make, post_unpack_fn=_fix_libraries)\n\n@_if_not_python_lib(\"netsa\")\ndef install_netsa_python(env):\n    \"\"\"A suite of open source tools for monitoring large-scale networks using flow data.\n    http://tools.netsa.cert.org/index.html\n    \"\"\"\n    version = \"1.3\"\n    url = \"http://tools.netsa.cert.org/releases/netsa-python-%s.tar.gz\" % version\n    cmd = env.safe_run if _is_anaconda(env) else env.safe_sudo\n    cmd(\"%s install %s\" % (_pip_cmd(env), url))\n"
  },
  {
    "path": "cloudbio/custom/shared.py",
    "content": "\"\"\"Reusable decorators and functions for custom installations.\n\"\"\"\nfrom __future__ import print_function\nfrom contextlib import contextmanager\nimport functools\nimport os\nimport socket\nfrom string import Template\nimport sys\nimport tempfile\nfrom tempfile import NamedTemporaryFile\nimport urllib\nimport uuid\nimport shutil\nimport subprocess\nimport time\n\n# Optional fabric imports, for back compatibility\ntry:\n    from fabric.api import *\n    from fabric.contrib.files import *\n    from cloudbio.fabutils import quiet, warn_only\nexcept ImportError:\n    pass\n\nCBL_REPO_ROOT_URL = \"https://raw.github.com/chapmanb/cloudbiolinux/master/\"\n\n# -- decorators and context managers\n\n@contextmanager\ndef chdir(new_dir):\n    \"\"\"Context manager to temporarily change to a new directory.\n\n    http://lucentbeing.com/blog/context-managers-and-the-with-statement-in-python/\n    \"\"\"\n    # On busy filesystems can have issues accessing main directory. Allow retries\n    num_tries = 0\n    max_tries = 5\n    cur_dir = None\n    while cur_dir is None:\n        try:\n            cur_dir = os.getcwd()\n        except OSError:\n            if num_tries > max_tries:\n                raise\n            num_tries += 1\n            time.sleep(2)\n    safe_makedir(new_dir)\n    os.chdir(new_dir)\n    try:\n        yield\n    finally:\n        os.chdir(cur_dir)\n\ndef safe_makedir(dname):\n    \"\"\"Make a directory if it doesn't exist, handling concurrent race conditions.\n    \"\"\"\n    if not dname:\n        return dname\n    num_tries = 0\n    max_tries = 5\n    while not os.path.exists(dname):\n        # we could get an error here if multiple processes are creating\n        # the directory at the same time. Grr, concurrency.\n        try:\n            os.makedirs(dname)\n        except OSError:\n            if num_tries > max_tries:\n                raise\n            num_tries += 1\n            time.sleep(2)\n    return dname\n\ndef which(program, env=None):\n    \"\"\" returns the path to an executable or None if it can't be found\"\"\"\n    paths = os.environ[\"PATH\"].split(os.pathsep)\n    if env and hasattr(env, \"system_install\"):\n        paths += [env.system_install, os.path.join(env.system_install, \"anaconda\")]\n\n    def is_exe(fpath):\n        return os.path.isfile(fpath) and os.access(fpath, os.X_OK)\n\n    fpath, fname = os.path.split(program)\n    if fpath:\n        if is_exe(program):\n            return program\n    else:\n        for path in paths:\n            exe_file = os.path.join(path, program)\n            if is_exe(exe_file):\n                return exe_file\n    return None\n\ndef _if_not_installed(pname):\n    \"\"\"Decorator that checks if a callable program is installed.\n    \"\"\"\n    def argcatcher(func):\n        functools.wraps(func)\n\n        def decorator(*args, **kwargs):\n            if _galaxy_tool_install(args):\n                run_function = not _galaxy_tool_present(args)\n            elif isinstance(pname, list):\n                run_function = any([_executable_not_on_path(x) for x in pname])\n            else:\n                run_function = _executable_not_on_path(pname)\n            if run_function:\n                return func(*args, **kwargs)\n        return decorator\n    return argcatcher\n\ndef _all_cbl_paths(env, ext):\n    \"\"\"Add paths to other non-system directories installed by CloudBioLinux.\n    \"\"\"\n    return \":\".join(\"%s/%s\" % (p, ext) for p in [env.system_install,\n                                                 os.path.join(env.system_install, \"anaconda\")])\ndef _executable_not_on_path(pname):\n    with settings(hide('warnings', 'running', 'stdout', 'stderr'),\n                  warn_only=True):\n        result = env.safe_run(\"export PATH=%s:$PATH && \"\n                              \"export LD_LIBRARY_PATH=%s:$LD_LIBRARY_PATH && %s\" %\n                              (_all_cbl_paths(env, \"bin\"), _all_cbl_paths(env, \"lib\"), pname))\n    return result.return_code == 127\n\n\ndef _galaxy_tool_install(args):\n    try:\n        return args[0][\"galaxy_tool_install\"]\n    except:\n        return False\n\n\ndef _galaxy_tool_present(args):\n    return env.safe_exists(os.path.join(args[0][\"system_install\"], \"env.sh\"))\n\n\ndef _if_not_python_lib(library):\n    \"\"\"Decorator that checks if a python library is installed.\n    \"\"\"\n    def argcatcher(func):\n        functools.wraps(func)\n\n        def decorator(*args, **kwargs):\n            with settings(warn_only=True):\n                errcount = int(env.safe_run_output(\"%s -c 'import %s' 2>&1 | grep -c ImportError | cat\" % (_python_cmd(env), library)))\n                result = 0 if errcount >= 1 else 1\n            if result == 0:\n                return func(*args, **kwargs)\n            else:\n                return result\n        return decorator\n    return argcatcher\n\n@contextmanager\ndef make_tmp_dir_local(ext, work_dir):\n    if ext:\n        work_dir += ext\n    safe_makedir(work_dir)\n    yield work_dir\n    shutil.rmtree(work_dir)\n\n@contextmanager\ndef _make_tmp_dir(ext=None, work_dir=None):\n    \"\"\"\n    Setup a temporary working directory for building custom software. First checks\n    fabric environment for a `work_dir` path, if that is not set it will use the\n    remote path $TMPDIR/cloudbiolinux if $TMPDIR is defined remotely, finally falling\n    back on remote $HOME/cloudbiolinux otherwise.\n    `ext` allows creation of tool specific temporary directories to avoid conflicts\n    using CloudBioLinux inside of CloudBioLinux.\n    \"\"\"\n    if not work_dir:\n        work_dir = __work_dir()\n    if ext:\n        work_dir += ext\n    use_sudo = False\n    if not env.safe_exists(work_dir):\n        with settings(warn_only=True):\n            # Try to create this directory without using sudo, but\n            # if needed fallback.\n            result = env.safe_run(\"mkdir -p '%s'\" % work_dir)\n            if result.return_code != 0:\n                use_sudo = True\n        if use_sudo:\n            env.safe_sudo(\"mkdir -p '%s'\" % work_dir)\n            env.safe_sudo(\"chown -R %s '%s'\" % (env.user, work_dir))\n    yield work_dir\n    if env.safe_exists(work_dir):\n        run_func = env.safe_sudo if use_sudo else env.safe_run\n        run_func(\"rm -rf %s\" % work_dir)\n\n\ndef __work_dir():\n    work_dir = env.get(\"work_dir\", None)\n    if not work_dir:\n        with quiet():\n            tmp_dir = env.safe_run_output(\"echo $TMPDIR\")\n        if tmp_dir.failed or not tmp_dir.strip():\n            home_dir = env.safe_run_output(\"echo $HOME\")\n            tmp_dir = os.path.join(home_dir, \"tmp\")\n        work_dir = os.path.join(tmp_dir.strip(), \"cloudbiolinux\")\n    return work_dir\n\n\n# -- Standard build utility simplifiers\n\n\ndef _get_expected_file(url, dir_name=None, safe_tar=False, tar_file_name=None):\n    if tar_file_name:\n        tar_file = tar_file_name\n    else:\n        tar_file = os.path.split(url.split(\"?\")[0])[-1]\n    safe_tar = \"--pax-option='delete=SCHILY.*,delete=LIBARCHIVE.*'\" if safe_tar else \"\"\n    exts = {(\".tar.gz\", \".tgz\"): \"tar %s -xzpf\" % safe_tar,\n            (\".tar\",): \"tar %s -xpf\" % safe_tar,\n            (\".tar.bz2\",): \"tar %s -xjpf\" % safe_tar,\n            (\".zip\",): \"unzip\"}\n    for ext_choices, tar_cmd in exts.iteritems():\n        for ext in ext_choices:\n            if tar_file.endswith(ext):\n                if dir_name is None:\n                    dir_name = tar_file[:-len(ext)]\n                return tar_file, dir_name, tar_cmd\n    raise ValueError(\"Did not find extract command for %s\" % url)\n\n\ndef _safe_dir_name(dir_name, need_dir=True):\n    replace_try = [\"\", \"-src\", \"_core\"]\n    for replace in replace_try:\n        check = dir_name.replace(replace, \"\")\n        if env.safe_exists(check):\n            return check\n    # still couldn't find it, it's a nasty one\n    for check_part in (dir_name.split(\"-\")[0].split(\"_\")[0],\n                       dir_name.split(\"-\")[-1].split(\"_\")[-1],\n                       dir_name.split(\".\")[0],\n                       dir_name.lower().split(\".\")[0]):\n        with settings(hide('warnings', 'running', 'stdout', 'stderr'),\n                      warn_only=True):\n            dirs = env.safe_run_output(\"ls -d1 *%s*/\" % check_part).split(\"\\n\")\n            dirs = [x for x in dirs if \"cannot access\" not in x and \"No such\" not in x]\n        if len(dirs) == 1 and dirs[0]:\n            return dirs[0]\n    dirs = env.safe_run_output(\"find * -type d -maxdepth 0\").split(\"\\n\")\n    if len(dirs) == 1 and dirs[0]:\n        return dirs[0]\n    if need_dir:\n        raise ValueError(\"Could not find directory %s\" % dir_name)\n\ndef _remote_fetch(env, url, out_file=None, allow_fail=False, fix_fn=None, samedir=False):\n    \"\"\"Retrieve url using wget, performing download in a temporary directory.\n\n    Provides a central location to handle retrieval issues and avoid\n    using interrupted downloads.\n    \"\"\"\n    if out_file is None:\n        out_file = os.path.basename(url)\n    if not os.path.exists(out_file):\n        if samedir and os.path.isabs(out_file):\n            orig_dir = os.path.dirname(out_file)\n            out_file = os.path.basename(out_file)\n        else:\n            orig_dir = os.getcwd()\n        temp_ext = \"/%s\" % uuid.uuid3(uuid.NAMESPACE_URL,\n                                      str(\"file://%s/%s/%s\" %\n                                          (\"localhost\", socket.gethostname(), out_file)))\n        with make_tmp_dir_local(ext=temp_ext, work_dir=orig_dir) as tmp_dir:\n            with chdir(tmp_dir):\n                try:\n                    subprocess.check_call(\"wget --continue --no-check-certificate -O %s '%s'\"\n                                          % (out_file, url), shell=True)\n                    if fix_fn:\n                        out_file = fix_fn(env, out_file)\n                    subprocess.check_call(\"mv %s %s\" % (out_file, orig_dir), shell=True)\n                except subprocess.CalledProcessError:\n                    if allow_fail:\n                        out_file = None\n                    else:\n                        raise IOError(\"Failure to retrieve remote file: %s\" % url)\n        if samedir and out_file:\n            out_file = os.path.join(orig_dir, out_file)\n    return out_file\n\ndef _fetch_and_unpack(url, need_dir=True, dir_name=None, revision=None,\n                      safe_tar=False, tar_file_name=None):\n    if url.startswith((\"git\", \"svn\", \"hg\", \"cvs\")):\n        base = os.path.splitext(os.path.basename(url.split()[-1]))[0]\n        if env.safe_exists(base):\n            env.safe_sudo(\"rm -rf {0}\".format(base))\n        env.safe_run(url)\n        if revision:\n            if url.startswith(\"git\"):\n                env.safe_run(\"cd %s && git checkout %s\" % (base, revision))\n            else:\n                raise ValueError(\"Need to implement revision retrieval for %s\" % url.split()[0])\n        return base\n    else:\n        # If tar_file_name is provided, use it instead of the inferred one\n        tar_file, dir_name, tar_cmd = _get_expected_file(url, dir_name, safe_tar, tar_file_name=tar_file_name)\n        tar_file = _remote_fetch(env, url, tar_file)\n        env.safe_run(\"%s %s\" % (tar_cmd, tar_file))\n        return _safe_dir_name(dir_name, need_dir)\n\n\ndef _configure_make(env):\n    env.safe_run(\"export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:%s/lib/pkgconfig && \" \\\n                 \"./configure --disable-werror --prefix=%s \" %\n                 (env.system_install, env.system_install))\n    lib_export = \"export LD_LIBRARY_PATH=%s/lib:$LD_LIBRARY_PATH\" % env.system_install\n    env.safe_run(\"%s && make\" % lib_export)\n    env.safe_sudo(\"%s && make install\" % lib_export)\n\ndef _ac_configure_make(env):\n    env.safe_run(\"autoreconf -i -f\")\n    _configure_make(env)\n\ndef _make_copy(find_cmd=None, premake_cmd=None, do_make=True):\n    def _do_work(env):\n        if premake_cmd:\n            premake_cmd()\n        if do_make:\n            env.safe_run(\"make\")\n        if find_cmd:\n            install_dir = _get_bin_dir(env)\n            for fname in env.safe_run_output(find_cmd).split(\"\\n\"):\n                env.safe_sudo(\"cp -rf %s %s\" % (fname.rstrip(\"\\r\"), install_dir))\n    return _do_work\n\n\ndef _get_install(url, env, make_command, post_unpack_fn=None, revision=None, dir_name=None,\n                 safe_tar=False, tar_file_name=None):\n    \"\"\"Retrieve source from a URL and install in our system directory.\n    \"\"\"\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            dir_name = _fetch_and_unpack(url, revision=revision, dir_name=dir_name,\n                                         safe_tar=safe_tar, tar_file_name=tar_file_name)\n        with cd(os.path.join(work_dir, dir_name)):\n            if post_unpack_fn:\n                post_unpack_fn(env)\n            make_command(env)\n\ndef _apply_patch(env, url):\n    patch = os.path.basename(url)\n    cmd = \"wget {url}; patch -p0 < {patch}\".format(url=url, patch=patch)\n    env.safe_run(cmd)\n\ndef _get_install_local(url, env, make_command, dir_name=None,\n                       post_unpack_fn=None, safe_tar=False, tar_file_name=None):\n    \"\"\"Build and install in a local directory.\n    \"\"\"\n    (_, test_name, _) = _get_expected_file(url, safe_tar=safe_tar, tar_file_name=tar_file_name)\n    test1 = os.path.join(env.local_install, test_name)\n    if dir_name is not None:\n        test2 = os.path.join(env.local_install, dir_name)\n    elif \"-\" in test1:\n        test2, _ = test1.rsplit(\"-\", 1)\n    else:\n        test2 = os.path.join(env.local_install, test_name.split(\"_\")[0])\n    if not env.safe_exists(test1) and not env.safe_exists(test2):\n        with _make_tmp_dir() as work_dir:\n            with cd(work_dir):\n                dir_name = _fetch_and_unpack(url, dir_name=dir_name, safe_tar=safe_tar,\n                    tar_file_name=tar_file_name)\n                if not env.safe_exists(os.path.join(env.local_install, dir_name)):\n                    with cd(dir_name):\n                        if post_unpack_fn:\n                            post_unpack_fn(env)\n                        make_command(env)\n                    # Copy instead of move because GNU mv does not have --parents flag.\n                    # The source dir will get cleaned up anyhow so just leave it.\n                    destination_dir = env.local_install\n                    env.safe_sudo(\"mkdir -p '%s'\" % destination_dir)\n                    env.safe_sudo(\"cp --recursive %s %s\" % (dir_name, destination_dir))\n\n# --- Language specific utilities\n\ndef _symlinked_install_dir(pname, version, env, extra_dir=None):\n    if extra_dir:\n        base_dir = os.path.join(env.system_install, \"share\", extra_dir, pname)\n    else:\n        base_dir = os.path.join(env.system_install, \"share\", pname)\n    return base_dir, \"%s-%s\" % (base_dir, version)\n\ndef _symlinked_dir_exists(pname, version, env, extra_dir=None):\n    \"\"\"Check if a symlinked directory exists and is non-empty.\n    \"\"\"\n    _, install_dir = _symlinked_install_dir(pname, version, env, extra_dir)\n    if env.safe_exists(install_dir):\n        items = env.safe_run_output(\"ls %s\" % install_dir)\n        if items.strip() != \"\":\n            return True\n    return False\n\ndef _symlinked_shared_dir(pname, version, env, extra_dir=None):\n    \"\"\"Create a symlinked directory of files inside the shared environment.\n    \"\"\"\n    base_dir, install_dir = _symlinked_install_dir(pname, version, env, extra_dir)\n    relative_install_dir = os.path.relpath(install_dir, os.path.dirname(base_dir))\n    # Does not exist, change symlink to new directory\n    if not env.safe_exists(install_dir):\n        env.safe_sudo(\"mkdir -p %s\" % install_dir)\n        if env.safe_exists(base_dir):\n            env.safe_sudo(\"rm -f %s\" % base_dir)\n        env.safe_sudo(\"ln -sf %s %s\" % (relative_install_dir, base_dir))\n        return install_dir\n    items = env.safe_run_output(\"ls %s\" % install_dir)\n    # empty directory, change symlink and re-download\n    if items.strip() == \"\":\n        if env.safe_exists(base_dir):\n            env.safe_sudo(\"rm -f %s\" % base_dir)\n        env.safe_sudo(\"ln -sf %s %s\" % (relative_install_dir, base_dir))\n        return install_dir\n    # Create symlink if previously deleted\n    if not env.safe_exists(base_dir):\n        env.safe_sudo(\"ln -sf %s %s\" % (relative_install_dir, base_dir))\n    return None\n\ndef _symlinked_java_version_dir(pname, version, env):\n    return _symlinked_shared_dir(pname, version, env, extra_dir=\"java\")\n\n\ndef _java_install(pname, version, url, env, install_fn=None,\n                  pre_fetch_fn=None):\n    \"\"\"Download java jars into versioned input directories.\n\n    pre_fetch_fn runs before URL retrieval, allowing insertion of\n    manual steps like restricted downloads.\n    \"\"\"\n    install_dir = _symlinked_java_version_dir(pname, version, env)\n    if install_dir:\n        with _make_tmp_dir() as work_dir:\n            with cd(work_dir):\n                if pre_fetch_fn:\n                    out = pre_fetch_fn(env)\n                    if out is None:\n                        return\n                dir_name = _fetch_and_unpack(url)\n                with cd(dir_name):\n                    if install_fn is not None:\n                        install_fn(env, install_dir)\n                    else:\n                        env.safe_sudo(\"mv *.jar %s\" % install_dir)\n\n\ndef _python_cmd(env):\n    \"\"\"Retrieve python command, handling tricky situations on CentOS.\n    \"\"\"\n    anaconda_py = os.path.join(env.system_install, \"anaconda\", \"bin\", \"python\")\n    if env.safe_exists(anaconda_py):\n        return anaconda_py\n    if \"python_version_ext\" in env and env.python_version_ext:\n        major, minor = env.safe_run(\"python --version\").split()[-1].split(\".\")[:2]\n        check_major, check_minor = env.python_version_ext.split(\".\")[:2]\n        if major != check_major or int(check_minor) > int(minor):\n            return \"python%s\" % env.python_version_ext\n        else:\n            return \"python\"\n    else:\n        return \"python\"\n\ndef _pip_cmd(env):\n    \"\"\"Retrieve pip command for installing python packages, allowing configuration.\n    \"\"\"\n    anaconda_pip = os.path.join(env.system_install, \"anaconda\", \"bin\", \"pip\")\n    if env.safe_exists(anaconda_pip):\n        to_check = [anaconda_pip]\n    else:\n        to_check = [\"pip\"]\n    if \"pip_cmd\" in env and env.pip_cmd:\n        to_check.append(env.pip_cmd)\n    if not env.use_sudo:\n        to_check.append(os.path.join(env.system_install, \"bin\", \"pip\"))\n    if \"python_version_ext\" in env and env.python_version_ext:\n        to_check.append(\"pip-{0}\".format(env.python_version_ext))\n    for cmd in to_check:\n        with quiet():\n            pip_version = env.safe_run(\"%s --version\" % cmd)\n        if pip_version.succeeded:\n            return cmd\n    raise ValueError(\"Could not find pip installer from: %s\" % to_check)\n\ndef _conda_cmd(env):\n    if hasattr(env, \"conda_cmd\") and env.conda_cmd:\n        return env.conda_cmd\n    to_check = []\n    if env.hosts == [\"localhost\"]:\n        to_check.append(os.path.join(os.path.dirname(os.path.realpath(sys.executable)), \"conda\"))\n    to_check.extend([os.path.join(env.system_install, \"anaconda\", \"bin\", \"conda\"), \"conda\"])\n    for cmd in to_check:\n        with quiet():\n            test = env.safe_run(\"%s --version\" % cmd)\n        if test.succeeded:\n            return cmd\n    return None\n\ndef _is_anaconda(env):\n    \"\"\"Check if we have a conda command or are in an anaconda subdirectory.\n    \"\"\"\n    with quiet():\n        conda = _conda_cmd(env)\n        has_conda = conda and env.safe_run_output(\"%s -h\" % conda).startswith(\"usage: conda\")\n    with quiet():\n        try:\n            full_pip = env.safe_run_output(\"which %s\" % _pip_cmd(env))\n        except ValueError:\n            full_pip = None\n    in_anaconda_dir = full_pip and full_pip.succeeded and \"/anaconda/\" in full_pip\n    return has_conda or in_anaconda_dir\n\ndef _python_make(env):\n    run_cmd = env.safe_run if _is_anaconda(env) else env.safe_sudo\n    # Clean up previously failed builds\n    env.safe_sudo(\"rm -rf /tmp/pip-build-%s\" % env.user)\n    env.safe_sudo(\"rm -rf /tmp/pip-*-build\")\n    run_cmd(\"%s install --upgrade .\" % _pip_cmd(env))\n    for clean in [\"dist\", \"build\", \"lib/*.egg-info\"]:\n        env.safe_sudo(\"rm -rf %s\" % clean)\n\n\ndef _get_installed_file(env, local_file):\n    installed_files_dir = \\\n        os.path.join(os.path.dirname(os.path.abspath(__file__)), \"..\", \"..\", \"installed_files\")\n    path = os.path.join(installed_files_dir, local_file)\n    if not os.path.exists(path):\n        # If using cloudbiolinux as a library, this won't be available,\n        # download the file from github instead\n        f = NamedTemporaryFile(delete=False)\n        cloudbiolinx_repo_url = env.get(\"cloudbiolinux_repo_url\", CBL_REPO_ROOT_URL)\n        url = os.path.join(cloudbiolinx_repo_url, 'installed_files', local_file)\n        urllib.urlretrieve(url, f.name)\n        path = f.name\n    return path\n\n\ndef _get_installed_file_contents(env, local_file):\n    return open(_get_installed_file(env, local_file), \"r\").read()\n\n\ndef _write_to_file(contents, path, mode):\n    \"\"\"\n    Use fabric to write string contents to remote file specified by path.\n    \"\"\"\n    fd, local_path = tempfile.mkstemp()\n    try:\n        os.write(fd, contents)\n        tmp_path = os.path.join(\"/tmp\", os.path.basename(path))\n        env.safe_put(local_path, tmp_path)\n        env.safe_sudo(\"mv %s %s\" % (tmp_path, path))\n        env.safe_sudo(\"chmod %s %s\" % (mode, path))\n        os.close(fd)\n    finally:\n        os.unlink(local_path)\n\n\ndef _get_bin_dir(env):\n    \"\"\"\n    When env.system_install is /usr this exists, but in the Galaxy\n    it may not already exist.\n    \"\"\"\n    return _get_install_subdir(env, \"bin\")\n\n\ndef _get_include_dir(env):\n    return _get_install_subdir(env, \"include\")\n\n\ndef _get_lib_dir(env):\n    return _get_install_subdir(env, \"lib\")\n\n\ndef _get_install_subdir(env, subdir):\n    path = os.path.join(env.system_install, subdir)\n    if not env.safe_exists(path):\n        env.safe_sudo(\"mkdir -p '%s'\" % path)\n    return path\n\n\ndef _set_default_config(env, install_dir, sym_dir_name=\"default\"):\n    \"\"\"\n    Sets up default galaxy config directory symbolic link (if needed). Needed\n    when it doesn't exists or when installing a new version of software.\n    \"\"\"\n    version = env[\"tool_version\"]\n    if env.safe_exists(install_dir):\n        install_dir_root = \"%s/..\" % install_dir\n        sym_dir = \"%s/%s\" % (install_dir_root, sym_dir_name)\n        replace_default = False\n        if not env.safe_exists(sym_dir):\n            replace_default = True\n        if not replace_default:\n            default_version = env.safe_sudo(\"basename `readlink -f %s`\" % sym_dir)\n            if version > default_version:  # Bug: Wouldn't work for 1.9 < 1.10\n                print(\"default version %s is older than version %s just installed, replacing...\" % (default_version, version))\n                replace_default = True\n        if replace_default:\n            env.safe_sudo(\"rm -rf %s; ln -f -s %s %s\" % (sym_dir, install_dir, sym_dir))\n\n\ndef _setup_simple_service(service_name):\n    \"\"\"\n    Very Ubuntu/Debian specific, will need to be modified if used on other\n    archs.\n    \"\"\"\n    sudo(\"ln -f -s /etc/init.d/%s /etc/rc0.d/K01%s\" % (service_name, service_name))\n    sudo(\"ln -f -s /etc/init.d/%s /etc/rc1.d/K01%s\" % (service_name, service_name))\n    sudo(\"ln -f -s /etc/init.d/%s /etc/rc2.d/S99%s\" % (service_name, service_name))\n    sudo(\"ln -f -s /etc/init.d/%s /etc/rc3.d/S99%s\" % (service_name, service_name))\n    sudo(\"ln -f -s /etc/init.d/%s /etc/rc4.d/S99%s\" % (service_name, service_name))\n    sudo(\"ln -f -s /etc/init.d/%s /etc/rc5.d/S99%s\" % (service_name, service_name))\n    sudo(\"ln -f -s /etc/init.d/%s /etc/rc6.d/K01%s\" % (service_name, service_name))\n\n\ndef _render_config_file_template(env, name, defaults={}, overrides={}, default_source=None):\n    \"\"\"\n    If ``name` is say ``nginx.conf``, check fabric environment for\n    ``nginx_conf_path`` and then ``nginx_conf_template_path``. If\n    ``nginx_conf_path`` is set, return the contents of that file. If\n    nginx_conf_template_path is set, return the contents of that file\n    but with variable interpolation performed. Variable interpolation\n    is performed using a derivative of the fabric environment defined\n    using the supplied ``defaults`` and ``overrides`` using the\n    ``_extend_env`` function below.\n\n    Finally, if neither ``nginx_conf_path`` or\n    ``nginx_conf_template_path`` are set, check the\n    ``installed_files`` directory for ``nginx.conf`` and finally\n    ``nginx.conf.template``.\n    \"\"\"\n    param_prefix = name.replace(\".\", \"_\")\n    # Deployer can specify absolute path for config file, check this first\n    path_key_name = \"%s_path\" % param_prefix\n    template_key_name = \"%s_template_path\" % param_prefix\n    if env.get(path_key_name, None):\n        source_path = env[path_key_name]\n        source_template = False\n    elif env.get(template_key_name, None):\n        source_path = env[template_key_name]\n        source_template = True\n    elif default_source:\n        source_path = _get_installed_file(env, default_source)\n        source_template = source_path.endswith(\".template\")\n    else:\n        default_template_name = \"%s.template\" % name\n        source_path = _get_installed_file(env, default_template_name)\n        source_template = True\n\n    if source_template:\n        template = Template(open(source_path, \"r\").read())\n        template_params = _extend_env(env, defaults=defaults, overrides=overrides)\n        contents = template.substitute(template_params)\n    else:\n        contents = open(source_path, \"r\").read()\n    return contents\n\n\ndef _extend_env(env, defaults={}, overrides={}):\n    \"\"\"\n    Create a new ``dict`` from fabric's ``env``, first adding defaults\n    specified via ``defaults`` (if available). Finally, override\n    anything in env, with values specified by ``overrides``.\n    \"\"\"\n    new_env = {}\n    for key, value in defaults.iteritems():\n        new_env[key] = value\n    for key, value in env.iteritems():\n        new_env[key] = value\n    for key, value in overrides.iteritems():\n        new_env[key] = value\n    return new_env\n\n\ndef _setup_conf_file(env, dest, name, defaults={}, overrides={}, default_source=None, mode=\"0755\"):\n    conf_file_contents = _render_config_file_template(env, name, defaults, overrides, default_source)\n    _write_to_file(conf_file_contents, dest, mode=mode)\n\n\ndef _add_to_profiles(line, profiles=[], use_sudo=True):\n    \"\"\"\n    If it's not already there, append ``line`` to shell profiles files.\n    By default, these are ``/etc/profile`` and ``/etc/bash.bashrc`` but can be\n    overridden by providing a list of file paths to the ``profiles`` argument.\n    \"\"\"\n    if not profiles:\n        profiles = ['/etc/bash.bashrc', '/etc/profile']\n    for profile in profiles:\n        if not env.safe_contains(profile, line):\n            env.safe_append(profile, line, use_sudo=use_sudo)\n\n\ndef install_venvburrito():\n    \"\"\"\n    If not already installed, install virtualenv-burrito\n    (https://github.com/brainsik/virtualenv-burrito) as a convenient\n    method for installing and managing Python virtualenvs.\n    \"\"\"\n    url = \"https://raw.github.com/brainsik/virtualenv-burrito/master/virtualenv-burrito.sh\"\n    if not env.safe_exists(\"$HOME/.venvburrito/startup.sh\"):\n        env.safe_run(\"curl -sL {0} | $SHELL\".format(url))\n        # Add the startup script into the ubuntu user's bashrc\n        _add_to_profiles(\". $HOME/.venvburrito/startup.sh\", [env.shell_config], use_sudo=False)\n\ndef _create_python_virtualenv(env, venv_name, reqs_file=None, reqs_url=None):\n    \"\"\"\n    Using virtual-burrito, create a new Python virtualenv named ``venv_name``.\n    Do so only if the virtualenv of the given name does not already exist.\n    virtual-burrito installs virtualenvs in ``$HOME/.virtualenvs``.\n\n    By default, an empty virtualenv is created. Python libraries can be\n    installed into the virutalenv at the time of creation by providing a path\n    to the requirements.txt file (``reqs_file``). Instead of providing the file,\n    a url to the file can be provided via ``reqs_url``, in which case the\n    requirements file will first be downloaded. Note that if the ``reqs_url``\n    is provided, the downloaded file will take precedence over ``reqs_file``.\n    \"\"\"\n    # First make sure virtualenv-burrito is installed\n    install_venvburrito()\n    activate_vburrito = \". $HOME/.venvburrito/startup.sh\"\n\n    def create():\n        if \"venv_directory\" not in env:\n            _create_global_python_virtualenv(env, venv_name, reqs_file, reqs_url)\n        else:\n            _create_local_python_virtualenv(env, venv_name, reqs_file, reqs_url)\n\n    # TODO: Terrible hack here, figure it out and fix it.\n    #   prefix or vburrito does not work with is_local or at least deployer+is_local\n    if env.is_local:\n        create()\n    else:\n        with prefix(activate_vburrito):\n            create()\n\n\ndef _create_local_python_virtualenv(env, venv_name, reqs_file, reqs_url):\n    \"\"\"\n    Use virtualenv directly to setup virtualenv in specified directory.\n    \"\"\"\n    venv_directory = env.get(\"venv_directory\")\n    if not env.safe_exists(venv_directory):\n        if reqs_url:\n            _remote_fetch(env, reqs_url, reqs_file)\n        env.logger.debug(\"Creating virtualenv in directory %s\" % venv_directory)\n        env.safe_sudo(\"virtualenv --no-site-packages '%s'\" % venv_directory)\n        env.logger.debug(\"Activating\")\n        env.safe_sudo(\". %s/bin/activate; pip install -r '%s'\" % (venv_directory, reqs_file))\n\n\ndef _create_global_python_virtualenv(env, venv_name, reqs_file, reqs_url):\n    \"\"\"\n    Use mkvirtualenv to setup this virtualenv globally for user.\n    \"\"\"\n    if venv_name in env.safe_run_output(\"bash -l -c lsvirtualenv | grep {0} || true\"\n        .format(venv_name)):\n        env.logger.info(\"Virtualenv {0} already exists\".format(venv_name))\n    else:\n        with _make_tmp_dir():\n            if reqs_file or reqs_url:\n                if not reqs_file:\n                    # This mean the url only is provided so 'standardize ' the file name\n                    reqs_file = 'requirements.txt'\n                cmd = \"bash -l -c 'mkvirtualenv -r {0} {1}'\".format(reqs_file, venv_name)\n            else:\n                cmd = \"bash -l -c 'mkvirtualenv {0}'\".format(venv_name)\n            if reqs_url:\n                _remote_fetch(env, reqs_url, reqs_file)\n            env.safe_run(cmd)\n            env.logger.info(\"Finished installing virtualenv {0}\".format(venv_name))\n\n\ndef _get_bitbucket_download_url(revision, default_repo):\n    if revision.startswith(\"http\"):\n        url = revision\n    else:\n        url = \"%s/get/%s.tar.gz\" % (default_repo, revision)\n    return url\n\n\ndef _read_boolean(env, name, default):\n    property_str = env.get(name, str(default))\n    return property_str.upper() in [\"TRUE\", \"YES\"]\n"
  },
  {
    "path": "cloudbio/custom/system.py",
    "content": "\"\"\"\nInstall system programs not available from packages.\n\"\"\"\nimport os\n\nfrom fabric.api import cd\n\nfrom cloudbio.custom import shared\nfrom cloudbio.custom.shared import _if_not_installed, _get_install, _configure_make\nfrom cloudbio.fabutils import quiet\n\ndef install_homebrew(env):\n    \"\"\"Homebrew package manager for OSX and Linuxbrew for linux systems.\n\n    https://github.com/mxcl/homebrew\n    https://github.com/Homebrew/linuxbrew\n    \"\"\"\n    if env.distribution == \"macosx\":\n        with quiet():\n            test_brewcmd = env.safe_run(\"brew --version\")\n        if not test_brewcmd.succeeded:\n            env.safe_run('ruby -e \"$(curl -fsSL https://raw.github.com/mxcl/homebrew/go/install)\"')\n    else:\n        brewcmd = os.path.join(env.system_install, \"bin\", \"brew\")\n        with quiet():\n            test_brewcmd = env.safe_run(\"%s --version\" % brewcmd)\n        if not test_brewcmd.succeeded or _linuxbrew_origin_problem(brewcmd):\n            with shared._make_tmp_dir() as tmp_dir:\n                with cd(tmp_dir):\n                    if env.safe_exists(\"linuxbrew\"):\n                        env.safe_run(\"rm -rf linuxbrew\")\n                    for cleandir in [\"Library\", \".git\"]:\n                        if env.safe_exists(\"%s/%s\" % (env.system_install, cleandir)):\n                            env.safe_run(\"rm -rf %s/%s\" % (env.system_install, cleandir))\n                    env.safe_run(\"git clone https://github.com/Homebrew/linuxbrew.git\")\n                    with cd(\"linuxbrew\"):\n                        if not env.safe_exists(env.system_install):\n                            env.safe_sudo(\"mkdir -p %s\" % env.system_install)\n                        env.safe_sudo(\"chown %s %s\" % (env.user, env.system_install))\n                        paths = [\"bin\", \"etc\", \"include\", \"lib\", \"lib/pkgconfig\", \"Library\",\n                                 \"sbin\", \"share\", \"var\", \"var/log\", \"share/java\", \"share/locale\",\n                                 \"share/man\", \"share/man/man1\", \"share/man/man2\",\n                                 \"share/man/man3\", \"share/man/man4\", \"share/man/man5\",\n                                 \"share/man/man6\", \"share/man/man7\", \"share/man/man8\",\n                                 \"share/info\", \"share/doc\", \"share/aclocal\",\n                                 \"lib/python2.7/site-packages\", \"lib/python2.6/site-packages\",\n                                 \"lib/python3.2/site-packages\", \"lib/python3.3/site-packages\",\n                                 \"lib/perl5\", \"lib/perl5/site_perl\"]\n                        if not env.safe_exists(\"%s/bin\" % env.system_install):\n                            env.safe_sudo(\"mkdir -p %s/bin\" % env.system_install)\n                        for path in paths:\n                            if env.safe_exists(\"%s/%s\" % (env.system_install, path)):\n                                env.safe_sudo(\"chown %s %s/%s\" % (env.user, env.system_install, path))\n                        if not env.safe_exists(\"%s/Library\" % env.system_install):\n                            env.safe_run(\"mv Library %s\" % env.system_install)\n                        if not env.safe_exists(\"%s/.git\" % env.system_install):\n                            env.safe_run(\"mv .git %s\" % env.system_install)\n                        man_dir = \"share/man/man1\"\n                        if not env.safe_exists(\"%s/%s\" % (env.system_install, man_dir)):\n                            env.safe_run(\"mkdir -p %s/%s\" % (env.system_install, man_dir))\n                        env.safe_run(\"mv -f %s/brew.1 %s/%s\" % (man_dir, env.system_install, man_dir))\n                        env.safe_run(\"mv -f bin/brew %s/bin\" % env.system_install)\n\ndef _linuxbrew_origin_problem(brewcmd):\n    \"\"\"Check for linuxbrew origins which point to Homebrew instead of Linuxbrew.\n    \"\"\"\n    config_file = os.path.abspath(os.path.normpath((os.path.expanduser(\n        os.path.join(os.path.dirname(brewcmd), os.pardir, \".git\", \"config\")))))\n    if not os.path.exists(config_file):\n        return True\n    with open(config_file) as in_handle:\n        return \"linuxbrew\" not in in_handle.read()\n\n@_if_not_installed(\"s3fs\")\ndef install_s3fs(env):\n    \"\"\"FUSE-based file system backed by Amazon S3.\n    https://code.google.com/p/s3fs/\n    \"\"\"\n    version = \"1.61\"\n    url = \"http://s3fs.googlecode.com/files/s3fs-{0}.tar.gz\".format(version)\n    _get_install(url, env, _configure_make)\n"
  },
  {
    "path": "cloudbio/custom/vcr.py",
    "content": "#\n# vcr.py\n#  - Configures the environment for running the Viral Assembly (viral_assembly_pipeline.py) and VIGOR (VIGOR3.pl) pipelines (creating directory structure and installs software). \n#\n\nfrom __future__ import print_function\nimport os.path, re, mmap\nfrom fabric.api import cd, env, hide, local, run, settings, sudo, task\nfrom fabric.network import disconnect_all\n\n# Common variables\ndependency_URL = \"http://s3.amazonaws.com/VIGOR-GSC\"\n\n\n# Galaxy VCR\ngalaxy_central = \"/mnt/galaxyTools/galaxy-central\"\ngalaxy_VCR_path = \"/%s/tools/viral_assembly_annotation\" % galaxy_central\n\n# Galaxy VCR - install method\ndef install_galaxy_vcr(env):\n\twith cd(\"~\"):\n\t\tprint(\"Installing galaxy VCR tools (python and xml scripts).\")\n\t\tsudo(\"git clone git://github.com/JCVI-Cloud/galaxy-tools-vcr.git\")\n\t\tsudo(\"cp -R galaxy-tools-vcr/tools/viral_assembly_annotation %s\" % galaxy_VCR_path)\n\t\tsudo(\"chown -R galaxy:galaxy %s\" % galaxy_VCR_path)\n\twith cd(galaxy_central):\n\t\tprint(\"Adding VCR to tool_conf.xml.\")\n\t\ttcx_file = \"tool_conf.xml\"\n\t\t_set_pre_VCR(tcx_file,\"galaxy\",\"galaxy\")\n\t\ttcx_string = _get_file_string(tcx_file,galaxy_central)\n\t\t\n\t\tvcr_header = \"<section name=\\\"Viral Assembly and Annotation\\\" id=\\\"viral_assembly_annotation\\\">\\\"\"\n\t\tif (tcx_string.find(vcr_header) != -1):\n\t\t\tprint(\"Galaxy VCR tools already included in tools_conf.xml!\")\n\t\telse:\n\t\t\tsudo(\"sed -i '$d' %s/%s\" % (galaxy_central,tcx_file))\n\t\t\tsudo(\"echo -e '  <section name=\\\"Viral Assembly and Annotation\\\" id=\\\"viral_assembly_annotation\\\">' >> %s\" % tcx_file)\n\t\t\tsudo(\"echo -e '    <tool file=\\\"viral_assembly_annotation/viral_assembly.xml\\\" />' >> %s\" % tcx_file)\n\t\t\tsudo(\"echo -e '    <tool file=\\\"viral_assembly_annotation/VIGOR.xml\\\" />' >> %s\" % tcx_file)\n\t\t\tsudo(\"echo -e '  </section>' >> %s\" % tcx_file)\n\t\t\tsudo(\"echo -e '</toolbox>' >> %s\" % tcx_file)\n\t\t\n\t\tprint(\"Adding 'sanitize_all_html = False' to universe_wsgi.ini to enable JBrowse for VICVB.\")\n\t\tuwi_file = \"universe_wsgi.ini\"\n\t\t_set_pre_VCR(uwi_file,\"galaxy\",\"galaxy\")\n\t\tuwi_string = _get_file_string(uwi_file,galaxy_central)\n\t\t\n\t\tif (uwi_string.find(\"sanitize_all_html\") != -1):\n\t\t\tprint(\"Setting sanitize_all_html in %s to False.\" % uwi_file)\n\t\t\tsudo(\"sed -i '/^sanitize_all_html/c\\sanitize_all_html = False' %s\" % uwi_file)\n\t\telse:\n\t\t\tprint(\"No sanitize_all_html present! Adding...\")\n\t\t\tsudo(\"sed -i '/^\\[app:main\\]/a\\\\\\nsanitize_all_html = False' %s\" % uwi_file)\n\n\n# Viral Assembly\nviral_dirs = {}\nviral_urls = {}\nviral_tars = {}\n\n# Viral Assembly - install methods\ndef install_viralassembly(env):\n\ttry:\n\t\t_initialize_area_viral()\n\t\t_add_tools_viral()\n\t\t_add_refs()\n\t\tsudo(\"chmod -R 755 %(VIRAL_ROOT_DIR)s\" % env)\n\tfinally:\n\t\tdisconnect_all()\n\ndef install_viralassembly_cleanall(env):\n\ttry:\n\t\t_initialize_env(\"viral\")\n\t\t_remove_dir(\"%(VIRAL_ROOT_DIR)s\" % env)\n\t\tprint(\"Viral Assembly Removed\\n\")\n\tfinally:\n\t\tdisconnect_all()\n\n# Viral Assembly - utility methods\n\ndef _initialize_area_viral():\n\t_initialize_env(\"viral\")\n\t\n\tenv.VIRAL_SCRIPT = \"%s/viral_assembly_pipeline.py\" % dependency_URL\n\t\n\tviral_dirs[\"PROJECT_DIR\"] = \"%(VIRAL_ROOT_DIR)s/project\" % env\n\tviral_dirs[\"REF_DIR\"] = \"%(VIRAL_ROOT_DIR)s/references\"  % env\n\tviral_dirs[\"TOOLS_DIR\"] = \"%(VIRAL_ROOT_DIR)s/tools\"     % env\n\tviral_dirs[\"TOOLS_BINARIES_DIR\"] = \"%s/BINARIES\"         % viral_dirs[\"TOOLS_DIR\"]\n\tviral_dirs[\"TOOLS_PERL_DIR\"] = \"%s/PERL\"                 % viral_dirs[\"TOOLS_DIR\"]\n\t\n\tenv.VIRAL_REF_FILES = \"corona_virus,hadv,influenza_a_virus,jev,mpv,norv,rota_virus,rsv,veev,vzv,yfv\"\n\t\n\tviral_urls[\"BIO_LINUX_URL\"] = \"http://nebc.nerc.ac.uk/bio-linux/\"\n\t\n\tviral_tars[\"BINARIES_TARBALL\"] = \"BINARIES.tgz\"\n\tviral_tars[\"PERL_TARBALL\"] = \"PERL.tgz\"\n\t\n\tprint(\"user:   %(user)s\" % env)\n\tprint(\"host:   %(host)s\" % env)\n\tprint(\"ROOT DIR:   %(VIRAL_ROOT_DIR)s\" % env)\n\tprint(\"VIRAL ASSEMBLY SCRIPT:   %(VIRAL_SCRIPT)s\" % env)\n\tfor name in sorted(viral_dirs.keys()):\n\t\tif not _path_is_dir(viral_dirs[name]):\n\t\t\tsudo(\"mkdir -p %s\" % viral_dirs[name])\n\t\tprint(\"%s:   %s\" % (name,viral_dirs[name]))\n\tprint(\"VIRAL ASSEMBLY REFS FILES: %(VIRAL_REF_FILES)s\" % env)\n\tfor name in sorted(viral_urls.keys()):\n\t\tprint(\"%s:   %s\" % (name,viral_urls[name]))\n\tfor name in sorted(viral_tars.keys()):\n\t\tprint(\"%s:   %s\" % (name,viral_tars[name]))\n\ndef _add_tools_viral():\n\twith cd(\"/home/ubuntu/\"):\n\t\tbashrc_file = \".bashrc\"\n\t\t_set_pre_VCR(bashrc_file,\"ubuntu\",\"ubuntu\")\n\t\tbashrc_string = _get_file_string(bashrc_file,\"/home/ubuntu/\")\n\t\t\n\t\tif (bashrc_string.find(\"DEBIAN_FRONTEND\") != -1):\n\t\t\tprint(\"Setting DEBIAN_FRONTEND in %s to noninteractive.\" % bashrc_file)\n\t\t\tsudo(\"sed -i \\\"/DEBIAN_FRONTEND/c\\DEBIAN_FRONTEND=noninteractive\\\" %s/%s\" % (\"/home/ubuntu\",bashrc_file))\n\t\telse:\n\t\t\tprint(\"No DEBIAN_FRONTEND present! Adding...\")\n\t\t\tsudo(\"echo -e \\\"DEBIAN_FRONTEND=noninteractive\\\" >> %s\" % bashrc_file)\n\t\n\tsudo(\"wget --no-check-certificate -O %s/viral_assembly_pipeline.py %s\" % (env.VIRAL_ROOT_DIR,env.VIRAL_SCRIPT))\n\t_add_package(dependency_URL,viral_tars[\"BINARIES_TARBALL\"],viral_dirs[\"TOOLS_BINARIES_DIR\"],\"tar\")\n\t_add_package(dependency_URL,viral_tars[\"PERL_TARBALL\"],viral_dirs[\"TOOLS_PERL_DIR\"],\"tar\")\n\t_apt_get_install(\"csh\")\n\t_apt_get_install(\"gawk\")\n\t_initialize_bio_linux()\n\ndef _add_refs():\n\tfiles = (env.VIRAL_REF_FILES).split(\",\")\n\tfor file in files:\n\t\t_add_package(dependency_URL,\"%s.tgz\" % file,viral_dirs[\"REF_DIR\"],\"tar\")\n\ndef _initialize_bio_linux():\n\tsudo(\"echo -e \\\"deb %s unstable bio-linux\\\" >> /etc/apt/sources.list\" % viral_urls[\"BIO_LINUX_URL\"])\n\tsudo(\"sudo apt-get update\")\n\t_apt_get_install(\"bio-linux-keyring\")\n\t_apt_get_install(\"bwa\")\n\t_apt_get_install(\"samtools\")\n\t_apt_get_install(\"bio-linux-cap3\")\n\t_apt_get_install(\"emboss\")\n\n\n# VIGOR\n\nvigor_dirs = {}\nvigor_urls = {}\nvigor_tars = {}\nvigor_names = {}\n\n# VIGOR - install methods\n\ndef install_viralvigor(env):\n\ttry:\n\t\t_initialize_area_vigor()\n\t\t_initialize_host()\n\t\t_add_vigor()\n\t\t_add_tools_vigor()\n\tfinally:\n\t\tdisconnect_all()\n\ndef install_viralvigor_test(env):\n\ttry:\n\t\t_initialize_area_vigor()\n\t\tcmd = (\"\"\"%s/VIGOR3.pl \\\n\t\t\t\t-D yfv \\\n\t\t\t\t-i %s/westnile.fasta \\\n\t\t\t\t-O %s/westnile \\\n\t\t\t\t> %s/westnile_test_run.log 2>&1 \\\n\t\t\t\t\"\"\") % (vigor_dirs[\"VIGOR_RUNTIME_DIR\"],vigor_dirs[\"VIGOR_SAMPLE_DATA_DIR\"],vigor_dirs[\"VIGOR_TEST_OUTPUT_DIR\"],env.VIGOR_SCRATCH_DIR)\n\t\tprint(\"DEBUG: cmd[%s]\" % cmd)\n\t\trun(cmd)\n\tfinally:\n\t\tdisconnect_all()\n\ndef install_viralvigor_validate(env):\n\ttry:\n\t\t_initialize_area_vigor()\n\t\tsudo(\"rm -f %s/westnile.rpt\" % vigor_dirs[\"VIGOR_TEST_OUTPUT_DIR\"])\n\t\tsudo(\"rm -f %s/westnile.rpt\" % vigor_dirs[\"VIGOR_SAMPLE_DATA_DIR\"])\n\t\twith settings(hide(\"running\",\"stdout\")):\n\t\t\tresults = run(\"\"\"diff -Bwr %s %s || echo 'VALIDATION FAILED'\"\"\" % (vigor_dirs[\"VIGOR_SAMPLE_DATA_DIR\"],vigor_dirs[\"VIGOR_TEST_OUTPUT_DIR\"]))\n\t\tif results:\n\t\t\tprint(\"\\n\\nValidation Failed:\\n\\n%s\\n\" % results)\n\tfinally:\n\t\tdisconnect_all()\n\ndef install_viralvigor_cleanall(env):\n\ttry:\n\t\t_initialize_env(\"vigor\")\n\t\t_remove_dir(env.VIGOR_ROOT_DIR)\n\t\t_remove_dir(env.VIGOR_SCRATCH_DIR)\n\t\tprint(\"Vigor Removed\\n\")\n\tfinally:\n\t\tdisconnect_all()\n\n# VIGOR - utility methods\n\ndef _initialize_area_vigor():\n\tmachine = run(\"uname -m\")\n\t\n\tif machine.find('64')>0:\n\t\tenv.ARCH = 'x64-linux'\n\telse:\n\t\tenv.ARCH = 'ia32-linux'\n\t\n\t_initialize_env(\"vigor\")\n\t\n\tvigor_dirs[\"TOOLS_DIR\"] = \"%s/tools\"                                     % env.VIGOR_ROOT_DIR\n\tvigor_dirs[\"VIGOR_STORED_DIR\"] = \"%s/vigor\"                              % vigor_dirs[\"TOOLS_DIR\"]\n\tvigor_dirs[\"VIGOR_RUNTIME_DIR\"] = \"%s/prod3\"                             % vigor_dirs[\"VIGOR_STORED_DIR\"]\n\tvigor_dirs[\"VIGOR_TEMPSPACE_DIR\"] = \"%s/tempspace\"                       % env.VIGOR_SCRATCH_DIR\n\tvigor_dirs[\"VIGOR_SAMPLE_DATA_DIR\"] = \"%s/samples\"                       % vigor_dirs[\"VIGOR_STORED_DIR\"]\n\tvigor_dirs[\"VIGOR_TEST_OUTPUT_DIR\"] = \"%s/test\"                          % vigor_dirs[\"VIGOR_STORED_DIR\"]\n\tvigor_dirs[\"BLAST_DIR\"] = \"%s/blast\"                                     % vigor_dirs[\"TOOLS_DIR\"]\n\tvigor_dirs[\"CLUSTALW_DIR\"] = \"%s/clustalw\"                               % vigor_dirs[\"TOOLS_DIR\"]\n\tvigor_dirs[\"EXE_DIR\"] = vigor_dirs[\"VIGOR_RUNTIME_DIR\"]\n\t\n\tvigor_names[\"BLAST_NAME\"] = 'blast-2.2.15'\n\tvigor_names[\"CLUSTALW_NAME\"] = 'clustalw-1.83'\n\tvigor_names[\"VIGOR_NAME\"] = 'vigor-GSCcloud'\n\t\n\tvigor_tars[\"VIGOR_TAR_FILENAME\"] = \"%s.tgz\"                              % vigor_names[\"VIGOR_NAME\"]\n\tvigor_tars[\"BLAST_TAR_FILENAME\"] = \"%s-%s.tar.gz\"                        % (vigor_names[\"BLAST_NAME\"],env.ARCH)\n\tvigor_tars[\"CLUSTALW_TAR_FILENAME\"] = \"%s-%s.deb\"                        % (vigor_names[\"CLUSTALW_NAME\"],env.ARCH)\n\t\n\tprint(\"user:   %(user)s\" % env)\n\tprint(\"host:   %(host)s\" % env)\n\tprint(\"ARCH:   %(ARCH)s\" % env)\n\tprint(\"ROOT DIR:   %(VIGOR_ROOT_DIR)s\" % env)\n\tprint(\"SCRATCH DIR:   %(VIGOR_SCRATCH_DIR)s\" % env)\n\tfor name in sorted(vigor_dirs.keys()):\n\t\tprint(\"%s:   %s\" % (name,vigor_dirs[name]))\n\tfor name in sorted(vigor_urls.keys()):\n\t\tprint(\"%s:   %s\" % (name,vigor_urls[name]))\n\tprint(\"BLAST_NAME:   %s\" % vigor_names[\"BLAST_NAME\"])\n\tprint(\"CLUSTALW_NAME:   %s\" % vigor_names[\"CLUSTALW_NAME\"])\n\tprint(\"VIGOR_NAME:   %s\" % vigor_names[\"VIGOR_NAME\"])\n\tfor name in sorted(vigor_tars.keys()):\n\t\tprint(\"%s:   %s\" % (name,vigor_tars[name]))\n\ndef _initialize_host():\n\tlocal(\"ssh-keygen -R %(host)s\" % env)\n\t_fix_etc_hosts()\n\t_create_vigor_scratch_dir()\n\ndef _add_vigor():\n\tprint(\"Installing VIGOR...\")\n\t_create_vigor_tempspace_dir()\n\t_create_vigor_scratch_dir()\n\t_add_package(dependency_URL, vigor_tars[\"VIGOR_TAR_FILENAME\"], vigor_dirs[\"VIGOR_STORED_DIR\"], \"tar\")\n\tsudo(\"chmod 755 %s\" % os.path.join(vigor_dirs[\"VIGOR_RUNTIME_DIR\"], \"*.pl\"))\n\tif not _path_exists(os.path.join(vigor_dirs[\"EXE_DIR\"], \"perl\")):\n\t\tsudo(\"ln -sf %s %s\" % (\"/usr/bin/perl\", vigor_dirs[\"EXE_DIR\"]))\n\t\tsudo(\"ln -sf %s %s\" % (\"/usr/bin/perl\", \"/usr/local/bin\"))\n\tif not _path_exists(os.path.join(vigor_dirs[\"EXE_DIR\"], \"vigorscratch\")):\n\t\tsudo(\"ln -sf %s %s/vigorscratch\" % (vigor_dirs[\"VIGOR_TEMPSPACE_DIR\"], vigor_dirs[\"EXE_DIR\"]))\n\ndef _add_tools_vigor():\n\tprint(\"Install tools...\")\n\t_create_tools_dir()\n\t_add_blast()\n\t_add_clustalw()\n\t_apt_get_install(\"libapache-dbi-perl\")\n\t_apt_get_install(\"libclass-dbi-sqlite-perl\")\n\ndef _fix_etc_hosts():\n\tinternal_ip = sudo(\"hostname\")\n\tprint(\"internal_ip[%s]\" % internal_ip)\n\tfilespec = \"/etc/hosts\"\n\tsudo(\"echo '127.0.0.1 %s' >> %s\" % (internal_ip, filespec))\n\ndef _create_vigor_tempspace_dir():\n\tif not _path_is_dir(vigor_dirs[\"VIGOR_TEMPSPACE_DIR\"]):\n\t\tsudo(\"mkdir -p %s\" % vigor_dirs[\"VIGOR_TEMPSPACE_DIR\"])\n\t\tsudo(\"chown -R %s:%s %s\" % (env.user, env.user, vigor_dirs[\"VIGOR_TEMPSPACE_DIR\"]))\n\t\tsudo(\"find %s -type d -exec chmod 777 {} \\;\" % vigor_dirs[\"VIGOR_TEMPSPACE_DIR\"])\n\ndef _create_vigor_scratch_dir():\n\tif not _path_is_dir(env.VIGOR_SCRATCH_DIR):\n\t\tsudo(\"mkdir -p %s\" % env.VIGOR_SCRATCH_DIR)\n\tsudo(\"find %s -type f -exec chmod 666 {} \\;\" % env.VIGOR_SCRATCH_DIR)\n\tsudo(\"find %s -type d -exec chmod 777 {} \\;\" % env.VIGOR_SCRATCH_DIR)\n\ndef _create_tools_dir():\n\tif not _path_is_dir(vigor_dirs[\"TOOLS_DIR\"]):\n\t\tsudo(\"mkdir -p %s\" % vigor_dirs[\"TOOLS_DIR\"])\n\tsudo(\"chown -R %s:%s %s\" % (env.user,env.user,vigor_dirs[\"TOOLS_DIR\"]))\n\ndef _add_blast():\n\tprint(\"    Installing blast...\")\n\t_create_tools_dir()\n\t_add_package(dependency_URL, vigor_tars[\"BLAST_TAR_FILENAME\"], vigor_dirs[\"BLAST_DIR\"], \"tar\")\n\tif not _path_exists(os.path.join(vigor_dirs[\"EXE_DIR\"], \"blastall\")):\n\t\tsudo(\"ln -sf %s %s\" % (os.path.join(vigor_dirs[\"BLAST_DIR\"], vigor_names[\"BLAST_NAME\"], \"bin\", \"bl2seq\"), vigor_dirs[\"EXE_DIR\"]))\n\t\tsudo(\"ln -sf %s %s\" % (os.path.join(vigor_dirs[\"BLAST_DIR\"], vigor_names[\"BLAST_NAME\"], \"bin\", \"blastall\"), vigor_dirs[\"EXE_DIR\"]))\n\t\tsudo(\"ln -sf %s %s\" % (os.path.join(vigor_dirs[\"BLAST_DIR\"], vigor_names[\"BLAST_NAME\"], \"bin\", \"fastacmd\"), vigor_dirs[\"EXE_DIR\"]))\n\t\tsudo(\"ln -sf %s %s\" % (os.path.join(vigor_dirs[\"BLAST_DIR\"], vigor_names[\"BLAST_NAME\"], \"bin\", \"formatdb\"), vigor_dirs[\"EXE_DIR\"]))\n\ndef _add_clustalw():\n\tprint(\"    Installing clustalw...\")\n\t_create_tools_dir()\n\t_add_package(dependency_URL, vigor_tars[\"CLUSTALW_TAR_FILENAME\"], vigor_dirs[\"CLUSTALW_DIR\"], \"deb\")\n\tif not _path_exists(os.path.join(vigor_dirs[\"EXE_DIR\"], \"clustalw\")):\n\t\tsudo(\"ln -sf %s %s\" % (os.path.join(vigor_dirs[\"CLUSTALW_DIR\"], vigor_names[\"CLUSTALW_NAME\"], \"clustalw\"), vigor_dirs[\"EXE_DIR\"]))\n\n\n# VICVB - install methods\n\ndef install_vicvb(env):\n\ttry:\n\t\t_initialize_env(\"vicvb\")\n\t\t\n\t\t_apt_get_install(\"libperlio-gzip-perl\")\n\t\t_apt_get_install(\"liblocal-lib-perl\")\n\t\t\n\t\ttbl2asn_download_dir = \"/usr/local/tbl2asn_download\"\n\t\ttbl2asn_dir = \"/usr/local/bin\"\n\t\tif _path_exists(os.path.join(tbl2asn_dir,\"tbl2asn\")):\n\t\t\tsudo(\"mv %s/tbl2asn %s/tbl2asn_pre_VCR\" % (tbl2asn_dir,tbl2asn_dir))\n\t\t_add_package(\"ftp://ftp.ncbi.nih.gov/toolbox/ncbi_tools/converters/by_program/tbl2asn\",\"linux64.tbl2asn.gz\",tbl2asn_download_dir,\"gzip\")\n\t\tsudo(\"chmod 777 %s/linux64.tbl2asn\" % tbl2asn_download_dir)\n\t\tsudo(\"mv %s/linux64.tbl2asn %s/tbl2asn\" % (tbl2asn_download_dir,tbl2asn_dir))\n\t\t_remove_dir(tbl2asn_download_dir)\n\t\t\n\t\twith cd(\"~\"):\n\t\t\tsudo(\"git clone git://github.com/JCVI-Cloud/VICVB.git\")\n\t\twith cd(\"~/VICVB\"):\n\t\t\tsudo(\"lib/VICVB/data/install/install_to_dir_full.sh %s /mnt/galaxyTools/galaxy-central /\" % (env.VICVB_LOCAL_DIR))\n\tfinally:\n\t\tdisconnect_all()\n\ndef install_vicvb_cleanall(env):\n\ttry:\n\t\t_initialize_env(\"vicvb\")\n\t\t_remove_dir(env.VICVB_LOCAL_DIR)\n\t\t_remove_dir(env.VICVB_GALAXY_DIR)\n\t\twith cd (\"~\"):\n\t\t\tsudo(\"rm -fr ~/VICVB\")\n\t\tprint(\"VICVB Removed\\n\")\n\tfinally:\n\t\tdisconnect_all()\n\n\n# Common methods\n\ndef _initialize_env(pipeline):\n\tif pipeline == \"viral\":\n\t\tenv.VIRAL_ROOT_DIR = \"/usr/local/VHTNGS\"\n\t\tif not _path_exists(env.VIRAL_ROOT_DIR):\n\t\t\tsudo(\"mkdir -p %s\" % env.VIRAL_ROOT_DIR)\n\telif pipeline == \"vigor\":\n\t\tenv.VIGOR_ROOT_DIR = \"/usr/local/VIGOR\"\n\t\tif not _path_exists(env.VIGOR_ROOT_DIR):\n\t\t\tsudo(\"mkdir -p %s\" % env.VIGOR_ROOT_DIR)\n\t\tenv.VIGOR_SCRATCH_DIR = \"/usr/local/scratch/vigor\"\n\t\tif not _path_exists(env.VIGOR_SCRATCH_DIR):\n\t\t\tsudo(\"mkdir -p %s\" % env.VIGOR_SCRATCH_DIR)\n\t\t\tsudo(\"find %s -type f -exec chmod 666 {} \\;\" % env.VIGOR_SCRATCH_DIR)\n\t\t\tsudo(\"find %s -type d -exec chmod 777 {} \\;\" % env.VIGOR_SCRATCH_DIR)\n\telse:\n\t\tenv.VICVB_LOCAL_DIR = \"/usr/local/VICVB\";\n\t\tenv.VICVB_GALAXY_DIR = \"/mnt/galaxyTools/galaxy-central/static/vicvb\";\n\ndef _add_package(download_url, filename, install_dir, type):\n\tif not _path_is_dir(install_dir):\n\t\tsudo(\"mkdir -p %s\" % install_dir)\n\twith cd(install_dir):\n\t\tif not _path_exists(os.path.join(install_dir, filename)):\n\t\t\tsudo(\"\"\"wget --no-host-directories --cut-dirs=1 --directory-prefix=%s %s/%s\"\"\" % (install_dir, download_url, filename))\n\t\t\tif type == \"tar\":\n\t\t\t\tsudo(\"tar xvfz %s\" % filename)\n\t\t\telif type == \"bz2\":\n\t\t\t\tsudo(\"tar xfj %s\" % filename)\n\t\t\telif type == \"gzip\":\n\t\t\t\tsudo(\"gunzip %s\" % filename)\n\t\t\telse: \n\t\t\t\tsudo(\"dpkg -x %s %s\" % (filename,install_dir))\n\t\t\t\tsudo(\"mkdir %s/%s\" % (install_dir, vigor_names[\"CLUSTALW_NAME\"]))\n\t\t\t\tsudo(\"cp %s/usr/bin/* %s/%s\" % (install_dir,install_dir,vigor_names[\"CLUSTALW_NAME\"]))\n\tsudo(\"chown -R %s:%s %s\" % (env.user, env.user, install_dir))\n\tsudo(\"find %s -type d -exec chmod 755 {} \\;\" % install_dir)\n\ndef _remove_dir(dirspec):\n\tif _path_is_dir(dirspec):\n\t\t_unlock_dir(dirspec)\n\t\tsudo(\"rm -rf %s\" % dirspec)\n\telse:\n\t\tprint(\"DEBUG: _remove_dir[%s] -- NOT FOUND\" % dirspec)\n\ndef _unlock_dir(dirspec):\n\twith settings(hide(\"running\",\"stdout\")):\n\t\tsudo(\"find %s -type d -exec chmod 755 {} \\;\" % dirspec)\n\t\tsudo(\"find %s -type d -exec chmod g+s {} \\;\" % dirspec)\n\t\tsudo(\"find %s -type f -exec chmod 644 {} \\;\" % dirspec)\n\ndef _apt_get_install(tool):\n\tsudo(\"apt-get -q -y --force-yes install %s\" % tool)\n\ndef _path_exists(path):\n\tfound = False\n\twith settings(hide(\"running\",\"stdout\")):\n\t\tresult = sudo(\"test -e '%s' || echo 'FALSE'\" % path)\n\tif result != \"FALSE\": found = True\n\treturn found\n\ndef _path_is_dir(path):\n\tfound = False\n\twith settings(hide(\"running\",\"stdout\")):\n\t\tresult = sudo(\"test -d '%s' || echo 'FALSE'\" % path)\n\tif result != \"FALSE\": found = True\n\treturn found\n\ndef _set_pre_VCR(filename,user,group):\n\tsudo(\"cp %s %s_pre_VCR\" %(filename,filename))\n\tsudo(\"chown %s:%s %s_pre_VCR\" % (user,group,filename))\n\ndef _get_file_string(filename,directory):\n\tfh = open(\"%s/%s\" % (directory,filename))\n\tstring = mmap.mmap(fh.fileno(),0,access=mmap.ACCESS_READ)\n\tfh.close()\n\treturn string\n"
  },
  {
    "path": "cloudbio/custom/versioncheck.py",
    "content": "\"\"\"Tool specific version checking to identify out of date dependencies.\n\nThis provides infrastructure to check version strings against installed\ntools, enabling re-installation if a version doesn't match. This is a\nlightweight way to avoid out of date dependencies.\n\"\"\"\nfrom __future__ import print_function\nfrom distutils.version import LooseVersion\n\nfrom cloudbio.custom import shared\nfrom cloudbio.fabutils import quiet\n\ndef _parse_from_stdoutflag(out, flag, stdout_index=-1):\n    \"\"\"Extract version information from a flag in verbose stdout.\n\n    flag -- text information to identify the line we should split for a version\n    stdout_index -- Position of the version information in the split line. Defaults\n    to the last item.\n    \"\"\"\n    for line in out.split(\"\\n\") + out.stderr.split(\"\\n\"):\n        if line.find(flag) >= 0:\n            parts = line.split()\n            return parts[stdout_index].strip()\n    print(\"Did not find version information with flag %s from: \\n %s\" % (flag, out))\n    return \"\"\n\ndef _clean_version(x):\n    if x.startswith(\"upstream/\"):\n        x = x.replace(\"upstream/\", \"\")\n    if x.startswith(\"(\"):\n        x = x[1:].strip()\n    if x.endswith(\")\"):\n        x = x[:-1].strip()\n    if x.startswith(\"v\"):\n        x = x[1:].strip()\n    return x\n\ndef up_to_date(env, cmd, version, args=None, stdout_flag=None,\n               stdout_index=-1):\n    iversion = get_installed_version(env, cmd, version, args, stdout_flag,\n                                     stdout_index)\n    if not iversion:\n        return False\n    else:\n        return LooseVersion(iversion) >= LooseVersion(version)\n\ndef is_version(env, cmd, version, args=None, stdout_flag=None,\n               stdout_index=-1):\n    iversion = get_installed_version(env, cmd, version, args, stdout_flag,\n                                     stdout_index)\n    if not iversion:\n        return False\n    else:\n        return LooseVersion(iversion) == LooseVersion(version)\n\ndef get_installed_version(env, cmd, version, args=None, stdout_flag=None,\n                          stdout_index=-1):\n    \"\"\"Check if the given command is up to date with the provided version.\n    \"\"\"\n    if shared._executable_not_on_path(cmd):\n        return False\n    if args:\n        cmd = cmd + \" \" + \" \".join(args)\n    with quiet():\n        path_safe = (\"export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:{s}/lib/pkgconfig && \"\n                     \"export PATH=$PATH:{s}/bin && \"\n                     \"export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:{s}/lib && \".format(s=env.system_install))\n        out = env.safe_run_output(path_safe + cmd)\n    if stdout_flag:\n        iversion = _parse_from_stdoutflag(out, stdout_flag, stdout_index)\n    else:\n        iversion = out.strip()\n    iversion = _clean_version(iversion)\n    if \" not found in the pkg-config search path\" in iversion:\n        return False\n    return iversion\n"
  },
  {
    "path": "cloudbio/deploy/__init__.py",
    "content": "#!/usr/bin/env python\nfrom __future__ import print_function\nimport os\n\nfrom tempfile import tempdir\nfrom subprocess import call\nfrom inspect import getargspec\n\nfrom cloudbio.utils import _setup_logging, _configure_fabric_environment, _parse_fabricrc\nfrom cloudbio.biodata.genomes import install_data, install_data_s3, install_data_rsync\nfrom cloudbio.galaxy import _setup_galaxy_env_defaults\nfrom cloudbio.galaxy.utils import _chown_galaxy\nfrom cloudbio.galaxy.tools import _install_tools\n\nfrom fabfile import _perform_install, _install_custom\n\nfrom .util import eval_template\n\nimport cloudbio.deploy.plugins\n\nfrom fabric.main import load_settings\nfrom fabric.api import put, run, env, settings, sudo\n\ntry:\n    from .vmlauncher.transfer import FileTransferManager\n    from .vmlauncher import build_vm_launcher\nexcept ImportError:\n    build_vm_launcher = None\n    FileTransferManager = None\n\nDEFAULT_CLOUDBIOLINUX_TARGET = None\nDEFAULT_CLOUDBIOLINUX_FLAVOR = None\n\n\ndef deploy(options):\n    _setup_logging(env)\n\n    actions = _expand_actions(options.get(\"actions\"))\n    if options[\"vm_provider\"] == \"novm\":\n        vm_launcher = LocalVmLauncher(options)\n    else:\n        if not build_vm_launcher:\n            raise ImportError(\"Require vmlauncher: https://github.com/jmchilton/vm-launcher\")\n        vm_launcher = build_vm_launcher(options)\n\n    if _do_perform_action(\"list\", actions):\n        for node in vm_launcher.list():\n            print(\"Active node with uuid %s <%s>\" % (node.uuid, node))\n\n    if _do_perform_action(\"destroy\", actions):\n        target_name = options[\"hostname\"]\n        for node in vm_launcher.list():\n            node_name = node.name\n            if node_name == target_name:\n                vm_launcher.destroy(node)\n\n    __invoke_plugin_actions(env, actions, \"local_actions\", [vm_launcher, options])\n\n    # Do we have remaining actions requiring an vm?\n    if len(actions) > 0:\n        print('Setting up virtual machine')\n        vm_launcher.boot_and_connect()\n        _setup_vm(options, vm_launcher, actions)\n\n\nclass LocalVmLauncher:\n    \"\"\"Provide a lightweight real machine, non-vm class for launching.\n    \"\"\"\n    def __init__(self, options):\n        self.options = options\n\n    def get_ip(self):\n        specified_hostname = self.options.get(\"hostname\", None)\n        hostname = specified_hostname or \"localhost\"\n        return hostname\n\n    def get_key_file(self):\n        return None\n\n    def boot_and_connect(self):\n        pass\n\n    def destroy(self):\n        pass\n\n    def get_user(self):\n        return env.user\n\n    def list(self):\n        return []\n\n\ndef _setup_vm(options, vm_launcher, actions):\n    destroy_on_complete = get_boolean_option(options, 'destroy_on_complete', False)\n    try:\n        ip = vm_launcher.get_ip()\n        _setup_fabric(vm_launcher, ip, options)\n        with settings(host_string=ip):\n            _setup_cloudbiolinux(options)\n            if 'attach_volumes' in actions:\n                from .volume import attach_volumes\n                attach_volumes(vm_launcher, options)\n            if 'max_lifetime' in options:\n                seconds = options['max_lifetime']\n                # Unclear why the sleep is needed, but seems to be otherwise\n                # this doesn't work.\n                run(\"bash -c 'nohup sudo shutdown -h %d &'; sleep 2\" % seconds)\n\n            configure_instance(options, actions)\n\n            if 'transfer' in actions:\n                transfer_files(options)\n\n            __invoke_plugin_actions(env, actions, \"ready_actions\", [vm_launcher, options])\n\n            if 'ssh' in actions:\n                _interactive_ssh(vm_launcher)\n            if 'attach_ip' in actions:\n                vm_launcher.attach_public_ip()\n            if 'snapshot_volumes' in actions:\n                from .volume import make_snapshots\n                make_snapshots(vm_launcher, options)\n            if 'detach_volumes' in actions:\n                from .volume import detach_volumes\n                detach_volumes(vm_launcher, options)\n            if 'package' in actions:\n                name_template = vm_launcher.package_image_name()\n                name = eval_template(env, name_template)\n                vm_launcher.package(name=name)\n            if not destroy_on_complete and hasattr(vm_launcher, \"uuid\"):\n                print('Your instance (%s) is waiting at http://%s' % (vm_launcher.uuid, ip))\n    finally:\n        if destroy_on_complete:\n            vm_launcher.destroy()\n\n\ndef _expand_actions(actions):\n    unique_actions = set()\n    for simple_action in _possible_actions():\n        if simple_action in actions:\n            unique_actions.add(simple_action)\n    compound_actions = __get_plugin_actions(env, \"compound_actions\")\n    for compound_action in compound_actions.keys():\n        if compound_action in actions:\n            for compound_action_part in compound_actions[compound_action]:\n                unique_actions.add(compound_action_part)\n    return unique_actions\n\n\ndef _possible_actions():\n    possible_actions = [ \"list\",\n                         \"destroy\",\n                         \"transfer\",\n                         \"purge_tools\",\n                         \"setup_tools\",\n                         \"setup_biodata\",\n                         \"setup_ssh_key\",\n                         \"package\",\n                         \"setup_image\",\n                         \"launch\",  # Dummy action justs launches image\n                         \"install_biolinux\",\n                         \"install_custom\",\n                         \"ssh\",\n                         \"attach_ip\",\n                         \"snapshot_volumes\",\n                         \"attach_volumes\",\n                         \"detach_volumes\",\n                        ]\n    for action_type in [\"local_actions\", \"configure_actions\", \"ready_action\"]:\n        for action in  __get_plugin_actions(env, action_type):\n            possible_actions.append(action)\n    return possible_actions\n\n\ndef _do_perform_action(action, action_list):\n    do_perform = action in action_list\n    if do_perform:\n        action_list.remove(action)\n    return do_perform\n\n\ndef _setup_fabric(vm_launcher, ip, options):\n    env.user = vm_launcher.get_user()\n    env.hosts = [ip]\n    env.key_filename = vm_launcher.get_key_file()\n    env.disable_known_hosts = True\n\n\ndef _setup_cloudbiolinux(options):\n    def fabricrc_loader(env):\n        _setup_cloudbiolinux_fabric_properties(env, options)\n\n    flavor = get_main_options_string(options, \"flavor\", DEFAULT_CLOUDBIOLINUX_FLAVOR)\n    need_distcheck = options.get(\"fabricrc_overrides\", {}).get(\"use_sudo\")\n    _configure_fabric_environment(env, flavor, fabricrc_loader=fabricrc_loader,\n                                  ignore_distcheck=not need_distcheck)\n    _setup_image_user_data(env, options)\n\n\ndef _setup_cloudbiolinux_fabric_properties(env, options):\n    fabricrc_file = get_main_options_string(options, \"fabricrc_file\", None)\n    env.config_dir = os.path.join(os.path.dirname(__file__), \"..\", \"..\", \"config\")\n    env.tool_data_table_conf_file = os.path.join(env.config_dir, \"..\",\n                                                 \"installed_files\",\n                                                 \"tool_data_table_conf.xml\")\n    if fabricrc_file:\n        env.update(load_settings(fabricrc_file))\n    else:\n        # Let cloudbiolinux find out default file based on flavor, dist, etc...\n        _parse_fabricrc(env)\n    overrides = options.get(\"fabricrc_overrides\", {})\n    for key, value in overrides.iteritems():\n        # yaml parses bools, wouldn't be expected coming out of a fabricrc\n        # file so replace everything with a string.\n        if isinstance(value, bool):\n            overrides[key] = str(value)\n    env.update(overrides)\n    _setup_galaxy_env_defaults(env)\n\n\ndef _setup_image_user_data(env, options):\n    if \"image_user_data\" in options:\n        env[\"image_user_data_dict\"] = options[\"image_user_data\"]\n\n\ndef purge_genomes():\n    sudo(\"rm -rf %s\" % env.data_files)\n\n\ndef configure_ssh_key(options):\n    if \"galaxy_ssh_key\" in options:\n        key_file = options[\"galaxy_ssh_key\"]\n        sudo(\"mkdir -p /home/%s/.ssh\" % (env.galaxy_user))\n        sudo(\"chmod 700 /home/%s/.ssh\" % (env.galaxy_user))\n        put(local_path=key_file,\n            remote_path=\"/home/%s/.ssh/%s\" % (env.galaxy_user, os.path.basename(key_file)),\n            use_sudo=True,\n            mode=\"0600\")\n        _chown_galaxy(env, \"/home/%s/.ssh\" % env.galaxy_user)\n\n\ndef setup_biodata(options):\n    install_proc = install_data\n    genome_source = options.get(\"genome_source\", \"default\")\n    install_proc = {\n        \"default\": install_data,\n        \"S3\": install_data_s3,\n        \"rsync\": install_data_rsync,\n    }[genome_source]\n    if genome_source == \"default\":\n        install_proc(options[\"genomes\"], [\"ggd\", \"s3\", \"raw\"])\n    else:\n        install_proc(options[\"genomes\"])\n\n\ndef configure_instance(options, actions):\n    if \"install_biolinux\" in actions:\n        install_biolinux(options)\n    if \"install_custom\" in actions:\n        install_custom(options)\n    if \"purge_tools\" in actions:\n        purge_tools()\n\n    __invoke_plugin_actions(env, actions, \"configure_actions\", [options])\n\n    if \"setup_tools\" in actions:\n        install_tools(options[\"tools\"])\n    if \"setup_biodata\" in actions:\n        setup_biodata(options)\n    if \"setup_ssh_key\" in actions:\n        configure_ssh_key(options)\n\n\ndef install_custom(options):\n    package = options.get(\"package\")\n    _install_custom(package)\n\n\ndef install_biolinux(options):\n    flavor = options.get(\"flavor\", DEFAULT_CLOUDBIOLINUX_FLAVOR)\n    target = options.get(\"target\", DEFAULT_CLOUDBIOLINUX_TARGET)\n    _perform_install(target=target, flavor=flavor, more_custom_add=options.get(\"custom_add\", None))\n\n\ndef _interactive_ssh(vm_launcher):\n    \"\"\" Launch an interactive SSH session to host described by vm_launcher object.\n    \"\"\"\n    host = vm_launcher.get_ip()\n    user = vm_launcher.get_user()\n    key_file = vm_launcher.get_key_file()\n    cmd = \"ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i '%s' -l '%s' '%s'\" % (key_file, user, host)\n    call(cmd, shell=True)\n\n\ndef transfer_files(options):\n    transfer_options = _build_transfer_options(options, \"/mnt/uploaded_data\", \"galaxy\")\n    _do_transfer(transfer_options, options.get(\"files\", []), options.get(\"compressed_files\", []))\n\n\ndef _build_transfer_options(options, destination, user):\n    transfer_options = {}\n    transfer_options['compress'] = get_boolean_option(options, 'compress_transfers', True)\n    transfer_options['num_compress_threads'] = int(get_main_options_string(options, 'num_compress_threads', '1'))\n    transfer_options['num_transfer_threads'] = int(get_main_options_string(options, 'num_transfer_threads', '1'))\n    transfer_options['num_decompress_threads'] = int(get_main_options_string(options, 'num_decompress_threads', '1'))\n    transfer_options['chunk_size'] = int(get_main_options_string(options, 'transfer_chunk_size', '0'))\n    transfer_options['transfer_retries'] = int(get_main_options_string(options, 'transfer_retries', '3'))\n    transfer_options['local_temp'] = get_main_options_string(options, 'local_temp_dir', tempdir)\n    transfer_options['destination'] = destination\n    transfer_options['transfer_as'] = user\n    return transfer_options\n\n\ndef _do_transfer(transfer_options, files, compressed_files=[]):\n    if not FileTransferManager:\n        raise ImportError(\"Require vmlauncher: https://github.com/jmchilton/vm-launcher\")\n    FileTransferManager(**transfer_options).transfer_files(files, compressed_files)\n\n\ndef purge_tools():\n    env.safe_sudo(\"rm -rf %s\" % env.install_dir)\n\n\ndef install_tools(tools_conf):\n    \"\"\"\n    \"\"\"\n    _install_tools(env, tools_conf)\n\n\ndef get_boolean_option(options, name, default=False):\n    if name not in options:\n        return default\n    else:\n        return options[name]\n\n\ndef get_main_options_string(options, key, default=''):\n    value = default\n    if key in options:\n        value = options[key]\n    return value\n\n\ndef __invoke_plugin_actions(env, actions, action_type, provided_args):\n    possible_actions = __get_plugin_actions(env, action_type)\n    for action in list(actions):\n        if action in possible_actions:\n            __invoke_plugin_action(env, possible_actions[action], provided_args)\n            actions.remove(action)\n\n\ndef __invoke_plugin_action(env, action_function, provided_args):\n    arg_spec = getargspec(action_function).args\n    args = [] if not arg_spec else provided_args\n    action_function(*args)\n\n\ndef __get_plugin_actions(env, action_type):\n    actions = {}\n    for plugin_module in __get_plugin_modules(env):\n        if hasattr(plugin_module, action_type):\n            for action_name, action_function in getattr(plugin_module, action_type).iteritems():\n                actions[action_name] = action_function\n    return actions\n\n\ndef __get_plugin_modules(env):\n    if not \"plugin_modules\" in env:\n        unsorted_module_names = __get_plugin_module_names( )\n        ## Load modules in reverse order to allow hierarchical overrides\n        module_names = sorted(unsorted_module_names, reverse=True)\n        modules = []\n        for plugin_module_name in module_names:\n            try:\n                module = __import__(plugin_module_name)\n                for comp in plugin_module_name.split(\".\")[1:]:\n                    module = getattr(module, comp)\n                modules.append(module)\n            except BaseException as exception:\n                exception_str = str(exception)\n                message = \"%s rule module could not be loaded: %s\" % (plugin_module_name, exception_str)\n                env.logger.warn(message)\n                continue\n        env.plugin_modules = modules\n    return env.plugin_modules\n\n\ndef __get_plugin_module_names():\n    plugin_module_dir = cloudbio.deploy.plugins.__path__[0]\n    names = []\n    for fname in os.listdir(plugin_module_dir):\n        if not(fname.startswith(\"_\")) and fname.endswith(\".py\"):\n            rule_module_name = \"cloudbio.deploy.plugins.%s\" % fname[:-len(\".py\")]\n            names.append( rule_module_name )\n    return names\n"
  },
  {
    "path": "cloudbio/deploy/config.py",
    "content": "import inspect\nimport os\nimport yaml\n\n\ndef parse_settings(name=\"deploy/settings.yaml\"):\n    return _read_yaml(_path_from_root(name))\n\n\ndef _path_from_root(name):\n    root_path = os.path.join(os.path.dirname(inspect.getfile(inspect.currentframe())), \"..\", \"..\")\n    file_path = os.path.join(root_path, name)\n    return file_path\n\n\ndef _read_yaml(yaml_file):\n    with open(yaml_file) as in_handle:\n        return yaml.safe_load(in_handle)\n"
  },
  {
    "path": "cloudbio/deploy/main.py",
    "content": "from argparse import ArgumentParser\nimport yaml\n\nfrom cloudbio.deploy import deploy\n\nDESC = \"Creates an on-demand cloud instance, sets up applications, and transfer files to it.\"\n\n## Properties that may be specified as args or in settings file,\n## argument takes precedence.\nARG_PROPERTIES = [\n  # VM launcher options\n  \"files\",\n  \"compressed_files\",\n  \"actions\",\n  \"runtime_properties\",\n  \"vm_provider\",\n  \"hostname\",\n\n  # CloudBioLinux options\n  \"target\",\n  \"flavor\",\n  \"package\",\n\n  # CloudMan options\n  \"target_bucket\",\n\n  # Galaxy options\n  \"galaxy_tool_version\",\n  \"galaxy_tool_name\",\n  \"galaxy_tool_dir\",\n]\n\n\ndef main():\n    args = parse_args()\n    options = parse_settings(args.settings)\n\n    for property in ARG_PROPERTIES:\n        _copy_arg_to_options(options, args, property)\n\n    for fabric_property, fabric_value in zip(args.fabric_properties, args.fabric_values):\n        if \"fabricrc_overrides\" not in options:\n            options[\"fabricrc_overrides\"] = {}\n        options[\"fabricrc_overrides\"][fabric_property] = fabric_value\n\n    deploy(options)\n\n\ndef _copy_arg_to_options(options, args, property):\n    arg_property = getattr(args, property)\n    if arg_property or not property in options:\n        options[property] = arg_property\n\n\ndef parse_args():\n    parser = ArgumentParser(DESC)\n    parser.add_argument(\"--settings\", dest=\"settings\", default=\"settings.yaml\")\n    parser.add_argument('--action', dest=\"actions\", action=\"append\", default=[])\n    parser.add_argument('--runtime_property', dest=\"runtime_properties\", action=\"append\", default=[])\n    parser.add_argument('--compressed_file', dest=\"compressed_files\", action=\"append\", default=[], help=\"file to transfer to new instance and decompress\")\n    parser.add_argument('--file', dest=\"files\", action=\"append\", default=[], help=\"file to transfer to new instance\")\n    parser.add_argument(\"--vm_provider\", dest=\"vm_provider\", default=None, help=\"libcloud driver to use (or vagrant) (e.g. aws, openstack)\")\n    parser.add_argument(\"--hostname\", dest=\"hostname\", default=None, help=\"Newly created nodes are created with this specified hostname.\")\n\n    # CloudBioLinux options\n    parser.add_argument(\"--target\", dest=\"target\", default=None, help=\"Specify a CloudBioLinux target, used with action install_biolinux action\")\n    parser.add_argument(\"--flavor\", dest=\"flavor\", default=None, help=\"Specify a CloudBioLinux flavor, used with action install_biolinux action\")\n    parser.add_argument(\"--package\", dest=\"package\", default=None, help=\"Specify a CloudBioLinux package, used with action install_custom\")\n\n    # CloudMan related options\n    parser.add_argument(\"--target_bucket\", dest=\"target_bucket\", default=None, help=\"Specify a target bucket for CloudMan bucket related actions.\")\n\n    # Galaxy options\n    parser.add_argument(\"--galaxy_tool_version\", dest=\"galaxy_tool_version\")\n    parser.add_argument(\"--galaxy_tool_name\", dest=\"galaxy_tool_name\")\n    parser.add_argument(\"--galaxy_tool_dir\", dest=\"galaxy_tool_dir\")\n\n    parser.add_argument('--fabric_property', dest=\"fabric_properties\", action=\"append\", default=[])\n    parser.add_argument('--fabric_value', dest=\"fabric_values\", action=\"append\", default=[])\n\n    args = parser.parse_args()\n    if len(args.actions) == 0:\n        args.actions = [\"transfer\"]\n    return args\n\n\ndef parse_settings(name):\n    if not name == \"__none__\":\n        # Rather just die if settings.yaml does not exist or is not set, but would also\n        # like to support pure command-line driven mode so make settings.yaml if\n        # --settings=__none__ is passed to application.\n        return _read_yaml(name)\n    else:\n        return {}\n\n\ndef _read_yaml(yaml_file):\n    with open(yaml_file) as in_handle:\n        return yaml.safe_load(in_handle)\n\n\nif __name__ == \"__main__\":\n    main()\n"
  },
  {
    "path": "cloudbio/deploy/plugins/__init__.py",
    "content": "\"\"\"\nDeploy plugins.\n\nlocal_actions:\n  These actions occur before a VM has been created, if only local actions are specified, no VM is launched.\n\nconfigure_actions:\n  Take in only options and use fabric to configure instance in some way.\n\nready_actions:\n  Once VM is configured, these actions are executed.\n\ncompound_actions:\n  Dictionary of list values. Key represents an short-cut action that is expanded to each action specified\n  in corresponding value (a list of simple actions - standard or defined in the plugins).\n\"\"\"\n"
  },
  {
    "path": "cloudbio/deploy/plugins/cloudman.py",
    "content": "from __future__ import print_function\nfrom datetime import datetime\nfrom os.path import exists, join\nfrom os import listdir\nfrom tempfile import mkdtemp\n\nfrom cloudbio.deploy.util import eval_template\n\nfrom boto.exception import S3ResponseError\nfrom boto.s3.key import Key\n\nimport yaml\n\nfrom fabric.api import local, lcd, env\n\nDEFAULT_BUCKET_NAME = 'cloudman'\n\nDEFAULT_CLOUDMAN_PASSWORD = 'adminpass'\nDEFAULT_CLOUDMAN_CLUSTER_NAME = 'cloudman'\n\n\ndef bundle_cloudman(vm_launcher, options):\n    cloudman_options = options.get('cloudman')\n    cloudman_repository_path = cloudman_options['cloudman_repository']\n    delete_repository = False\n    bucket_source = cloudman_options.get(\"bucket_source\")\n    if cloudman_repository_path.startswith(\"http\"):\n        # Not a local path, lets clone it out of a remote repostiroy,\n        temp_directory = mkdtemp()\n        if cloudman_repository_path.endswith(\".git\"):\n            branch_opts = \"\"\n            repository_branch = cloudman_options.get('repository_branch', None)\n            if repository_branch:\n                branch_opts = \"-b '%s'\" % repository_branch\n            clone_command = \"git clone \" + branch_opts + \" '%s' '%s'\"\n        else:\n            clone_command = \"hg clone '%s' '%s'\"\n        local(clone_command % (cloudman_repository_path, temp_directory))\n        cloudman_repository_path = temp_directory\n        delete_repository = True\n    try:\n        with lcd(cloudman_repository_path):\n            try:\n                local(\"tar czvf cm.tar.gz *\")\n                local(\"mv cm.tar.gz '%s'\" % bucket_source)\n            finally:\n                local(\"rm -f cm.tar.gz\")\n    finally:\n        if delete_repository:\n            local(\"rm -rf '%s'\" % cloudman_repository_path)\n\n\ndef cloudman_launch(vm_launcher, options):\n    cloudman_options = options.get('cloudman')\n    image_id = cloudman_options.get('image_id', None)\n    if str(image_id).lower() == \"__use_snaps__\":\n        # TODO: Make more flexible\n        bucket_source = cloudman_options.get(\"bucket_source\")\n        snaps_path = join(bucket_source, \"snaps.yaml\")\n        if not exists(snaps_path):\n            raise Exception(\"CloudMan AMI set to __use_snaps__ but now snaps.yaml file could be found with path %s\" % snaps_path)\n        snaps = {}\n        with open(snaps_path, \"r\") as in_handle:\n            snaps = yaml.safe_load(in_handle)\n        clouds = snaps[\"clouds\"]\n        if len(clouds) != 1:\n            raise Exception(\"Exactly one cloud must be defined snaps.yaml for the deployer's CloudMan launch to work.\")\n        regions = clouds[0][\"regions\"]\n        if len(regions) != 1:\n            raise Exception(\"Exactly one region must be defined snaps.yaml for the deployer's CloudMan launch to work.\")\n        deployments = regions[0][\"deployments\"]\n        if len(deployments) != 1:\n            raise Exception(\"Exactly one deployment must be defined snaps.yaml for the deployer's CloudMan launch to work.\")\n        image_id = deployments[0][\"default_mi\"]\n\n    size_id = cloudman_options.get('size_id', None)\n    user_data = _prepare_user_data(vm_launcher, cloudman_options)\n    vm_launcher.create_node('cloudman',\n                            image_id=image_id,\n                            size_id=size_id,\n                            ex_userdata=user_data)\n\n\ndef sync_cloudman_bucket(vm_launcher, options):\n    bucket = options.get(\"target_bucket\", None)\n    if not bucket:\n        bucket = __get_bucket_default(options)\n    bucket_source = options.get(\"cloudman\", {}).get(\"bucket_source\", None)\n    if not bucket or not bucket_source:\n        print(\"Warning: Failed to sync cloud bucket, bucket or bucket_source is undefined.\")\n        return\n    conn = vm_launcher.boto_s3_connection()\n    for file_name in listdir(bucket_source):\n        _save_file_to_bucket(conn, bucket, file_name, join(bucket_source, file_name))\n\n\ndef _save_file_to_bucket(conn, bucket_name, remote_filename, local_file, **kwargs):\n    \"\"\" Save the local_file to bucket_name as remote_filename. Also, any additional\n    arguments passed as key-value pairs, are stored as file's metadata on S3.\"\"\"\n    # print \"Establishing handle with bucket '%s'...\" % bucket_name\n    b = _get_bucket(conn, bucket_name)\n    if b is not None:\n        # print \"Establishing handle with key object '%s'...\" % remote_filename\n        k = Key( b, remote_filename )\n        print(\"Attempting to save file '%s' to bucket '%s'...\" % (remote_filename, bucket_name))\n        try:\n            # Store some metadata (key-value pairs) about the contents of the file being uploaded\n            # Note that the metadata must be set *before* writing the file\n            k.set_metadata('date_uploaded', str(datetime.utcnow()))\n            for args_key in kwargs:\n                print(\"Adding metadata to file '%s': %s=%s\" % (remote_filename, args_key, kwargs[args_key]))\n                k.set_metadata(args_key, kwargs[args_key])\n            print(\"Saving file '%s'\" % local_file)\n            k.set_contents_from_filename(local_file)\n            print(\"Successfully added file '%s' to bucket '%s'.\" % (remote_filename, bucket_name))\n            make_public = True\n            if make_public:\n                k.make_public()\n        except S3ResponseError as e:\n            print(\"Failed to save file local file '%s' to bucket '%s' as file '%s': %s\" % ( local_file, bucket_name, remote_filename, e ))\n            return False\n        return True\n    else:\n        return False\n\n\ndef __get_bucket_default(options):\n    cloudman_options = options.get(\"cloudman\", {})\n    user_data = cloudman_options = cloudman_options.get('user_data', None) or {}\n    bucket = user_data.get(\"bucket_default\", None)\n    return bucket\n\n\ndef _prepare_user_data(vm_launcher, cloudman_options):\n    cloudman_user_data = cloudman_options.get('user_data', None) or {}\n    cluster_name = \\\n        cloudman_options.get('cluster_name', DEFAULT_CLOUDMAN_CLUSTER_NAME)\n    password = cloudman_options.get('password', DEFAULT_CLOUDMAN_PASSWORD)\n    access_key = vm_launcher.access_id()\n    secret_key = vm_launcher.secret_key()\n\n    _set_property_if_needed(cloudman_user_data, 'access_key', access_key)\n    _set_property_if_needed(cloudman_user_data, 'secret_key', secret_key)\n    cluster_name = eval_template(env, cluster_name)\n    _set_property_if_needed(cloudman_user_data, 'cluster_name', cluster_name)\n    _set_property_if_needed(cloudman_user_data, 'password', password)\n\n    return yaml.dump(cloudman_user_data)\n\n\ndef _set_property_if_needed(user_data, property, value):\n    if property not in user_data:\n        user_data[property] = value\n\n\ndef _get_bucket(s3_conn, bucket_name):\n    b = None\n    for i in range(0, 5):\n        try:\n            b = s3_conn.get_bucket(bucket_name)\n            break\n        except S3ResponseError:\n            print(\"Bucket '%s' not found, attempt %s/5\" % (bucket_name, i))\n            return None\n    return b\n\n\nlocal_actions = {\n    \"cloudman_launch\": cloudman_launch,\n    \"sync_cloudman_bucket\": sync_cloudman_bucket,\n    \"bundle_cloudman\": bundle_cloudman,\n}\n"
  },
  {
    "path": "cloudbio/deploy/plugins/galaxy.py",
    "content": "from cloudbio.galaxy.tools import _install_application\n\n\ndef install_tool(options):\n    version = options.get(\"galaxy_tool_version\")\n    name = options.get(\"galaxy_tool_name\")\n    install_dir = options.get(\"galaxy_tool_dir\", None)\n    _install_application(name, version, tool_install_dir=install_dir)\n\n\nconfigure_actions = {\n    \"install_galaxy_tool\": install_tool,\n}\n"
  },
  {
    "path": "cloudbio/deploy/plugins/gvl.py",
    "content": "\"\"\"\nDeployer plugin containing actions related to older galaxy-vm-launcher functionality.\n\"\"\"\nfrom __future__ import print_function\n\nimport os\nimport time\n\nfrom cloudbio.biodata.genomes import install_data, install_data_s3\nfrom cloudbio.deploy import get_main_options_string, _build_transfer_options, _do_transfer, transfer_files, get_boolean_option\nfrom cloudbio.deploy.util import wget, start_service, ensure_can_sudo_into, sudoers_append\nfrom cloudbio.galaxy.utils import _chown_galaxy\nfrom cloudbio.galaxy.tools import _setup_install_dir\nfrom cloudbio.custom.galaxy import install_galaxy_webapp\nfrom cloudbio.galaxy import _setup_users, _setup_xvfb, _install_nginx_standalone, _setup_postgresql\nfrom cloudbio.package import _configure_and_install_native_packages\nfrom cloudbio.package.deb import _apt_packages\n\n\nfrom fabric.api import put, run, env, sudo, get, cd\nfrom fabric.context_managers import prefix\nfrom fabric.contrib.files import append, contains, exists\n\n\n## Deprecated galaxy-vm-launcher way of setting up biodata.\ndef setup_genomes(options):\n    install_proc = install_data\n    sudo(\"mkdir -p %s\" % env.data_files)\n    sudo(\"chown -R %s:%s %s\" % (env.user, env.user, env.data_files))\n    put(\"config/tool_data_table_conf.xml\", \"%s/tool_data_table_conf.xml\" % env.galaxy_home)\n    indexing_packages = [\"bowtie\", \"bwa\", \"samtools\"]\n    path_extensions = \":\".join(map(lambda package: \"/opt/galaxyTools/tools/%s/default\" % package, indexing_packages))\n    with prefix(\"PATH=$PATH:%s\" % path_extensions):\n        if 'S3' == options['genome_source']:\n            install_proc = install_data_s3\n        install_proc(options[\"genomes\"])\n    if options.get(\"setup_taxonomy_data\", False):\n        setup_taxonomy_data()\n    stash_genomes_where = get_main_options_string(options, \"stash_genomes\")\n    if stash_genomes_where:\n        stash_genomes(stash_genomes_where)\n\n\ndef setup_taxonomy_data():\n    \"\"\"\n    Setup up taxonomy data required by Galaxy. Need to find another place to put\n    this, it is useful.\n    \"\"\"\n    taxonomy_directory = os.path.join(env.data_files, \"taxonomy\")\n    env.safe_sudo(\"mkdir -p '%s'\" % taxonomy_directory, user=env.user)\n    with cd(taxonomy_directory):\n        taxonomy_url = \"ftp://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz\"\n        gi_taxid_nucl = \"ftp://ftp.ncbi.nih.gov/pub/taxonomy/gi_taxid_nucl.dmp.gz\"\n        gi_taxid_prot = \"ftp://ftp.ncbi.nih.gov/pub/taxonomy/gi_taxid_prot.dmp.gz\"\n        wget(taxonomy_url)\n        wget(gi_taxid_nucl)\n        wget(gi_taxid_prot)\n        run(\"gunzip -c taxdump.tar.gz | tar xvf -\")\n        run(\"gunzip gi_taxid_nucl.dmp.gz\")\n        run(\"gunzip gi_taxid_prot.dmp.gz\")\n        run(\"cat gi_taxid_nucl.dmp gi_taxid_prot.dmp > gi_taxid_all.dmp\")\n        run(\"sort -n -k 1 gi_taxid_all.dmp > gi_taxid_sorted.txt\")\n        run(\"rm gi_taxid_nucl.dmp gi_taxid_prot.dmp gi_taxid_all.dmp\")\n        run(\"cat names.dmp | sed s/[\\\\(\\\\)\\\\'\\\\\\\"]/_/g > names.temporary\")\n        run(\"mv names.dmp names.dmp.orig\")\n        run(\"mv names.temporary names.dmp\")\n\n\ndef stash_genomes(where):\n    with _cd_indices_parent():\n        sudo(\"chown %s:%s .\" % (env.user, env.user))\n        indices_dir_name = _indices_dir_name()\n        remote_compressed_indices = \"%s.tar.gz\" % indices_dir_name\n        run(\"tar czvf %s %s\" % (remote_compressed_indices, indices_dir_name))\n        if where == 'download':\n            get(remote_path=remote_compressed_indices,\n                local_path=\"compressed_genomes.tar.gz\")\n        elif where == 'opt':\n            sudo(\"cp %s /opt/compressed_genomes.tar.gz\" % remote_compressed_indices)\n        else:\n            print(\"Invalid option specified for stash_genomes [%s] - valid values include download and opt.\" % where)\n\n\ndef upload_genomes(options):\n    with _cd_indices_parent():\n        sudo(\"chown %s:%s .\" % (env.user, env.user))\n        indices_dir_name = _indices_dir_name()\n        _transfer_genomes(options)\n        run(\"rm -rf %s\" % indices_dir_name)\n        run(\"tar xzvfm compressed_genomes.tar.gz\")\n        sudo(\"/etc/init.d/galaxy restart\")\n\n\ndef purge_genomes():\n    sudo(\"rm -rf %s\" % env.data_files)\n\n\ndef _cd_indices_parent():\n    return cd(_indices_parent())\n\n\ndef _indices_parent():\n    parent_dir = os.path.abspath(os.path.join(env.data_files, \"..\"))\n    return parent_dir\n\n\ndef _indices_dir_name():\n    indices_dir = env.data_files\n    if indices_dir.endswith(\"/\"):\n        indices_dir = indices_dir[0:(len(indices_dir) - 1)]\n    indices_dir_name = os.path.basename(indices_dir)\n    return indices_dir_name\n\n\ndef galaxy_transfer(vm_launcher, options):\n    transfer_files(options)\n    # Upload local compressed genomes to the cloud image, obsecure option.\n    do_upload_genomes = get_boolean_option(options, 'upload_genomes', False)\n    if do_upload_genomes:\n        upload_genomes(options)\n    if not _seed_at_configure_time(options):\n        seed_database()\n        seed_workflows(options)\n    wait_for_galaxy()\n    create_data_library_for_uploads(options)\n\n\ndef create_data_library_for_uploads(options):\n    with cd(os.path.join(env.galaxy_home, \"scripts\", \"api\")):\n        db_key_arg = get_main_options_string(options, 'db_key')\n        transfer_history_name = get_main_options_string(options, 'transfer_history_name')\n        transfer_history_api_key = get_main_options_string(options, 'transfer_history_api_key')\n        cmd_template = 'python handle_uploads.py --api_key=\"%s\" --db_key=\"%s\" --history=\"%s\" --history_api_key=\"%s\" '\n        galaxy_data = options[\"galaxy\"]\n        admin_user_api_key = galaxy_data[\"users\"][0][\"api_key\"]\n        cmd = cmd_template % (admin_user_api_key, db_key_arg, transfer_history_name, transfer_history_api_key)\n        sudo(\"bash -c 'export PYTHON_EGG_CACHE=eggs; %s'\" % cmd, user=\"galaxy\")\n\n\ndef _seed_at_configure_time(options):\n    if 'seed_galaxy' in options:\n        return options['seed_galaxy'] == 'configure'\n    else:\n        return True\n\n\ndef copy_runtime_properties(vm_launcher, options):\n    fqdn = vm_launcher.get_ip()\n    runtime_properties_raw = options.get(\"runtime_properties\", {})\n    runtime_properties = {\"FQDN\": fqdn}\n    for runtime_property_raw in runtime_properties_raw:\n        (name, value) = runtime_property_raw.split(\":\")\n        runtime_properties[name] = value\n    export_file = \"\"\n    for (name, value) in runtime_properties.iteritems():\n        export_file = \"export %s=%s\\n%s\" % (name, value, export_file)\n    sudo('mkdir -p %s' % env.galaxy_home)\n    _chown_galaxy(env, env.galaxy_home)\n    sudo(\"echo '%s' > %s/runtime_properties\" % (export_file, env.galaxy_home), user=env.galaxy_user)\n\n\ndef _transfer_genomes(options):\n    # Use just transfer settings in YAML\n    options = options['transfer']\n    transfer_options = _build_transfer_options(options, _indices_parent(), env.user)\n    transfer_options[\"compress\"] = False\n    _do_transfer(transfer_options, [\"compressed_genomes.tar.gz\"])\n\n\ndef wait_for_galaxy():\n\n    while not \"8080\" in run(\"netstat -lant\"):\n        # Check if galaxy has started\n        print(\"Waiting for galaxy to start.\")\n        time.sleep(10)\n\n\ndef purge_galaxy():\n    sudo(\"/etc/init.d/galaxy stop\")\n    sudo(\"rm -rf %s\" % env.galaxy_home)\n    init_script = \"postgresql\"\n    # if env.postgres_version[0] < '9':\n    #    # Postgres 8.4 had different name for script\n    #    init_script = \"postgresql-%s\" % env.postgres_version\n    sudo(\"/etc/init.d/%s restart\" % init_script)\n    sudo('psql  -c \"drop database galaxy;\"', user=\"postgres\")\n    sudo('psql  -c \"create database galaxy;\"', user=\"postgres\")\n\n\ndef setup_galaxy(options):\n    seed = _seed_at_configure_time(options)\n    setup_galaxy(options, seed=seed)\n    if seed:\n        seed_workflows(options)\n\n\ndef _setup_galaxy(options, seed=True):\n    \"\"\"Deploy a Galaxy server along with some tools.\n    \"\"\"\n    _setup_install_dir(env)  # Still needed? -John\n    install_galaxy_webapp(env)\n    #_fix_galaxy_permissions()\n    _setup_shed_tools_dir()\n    _setup_galaxy_log_dir()\n    _migrate_galaxy_database()\n    if seed:\n        seed_database(options[\"galaxy\"])\n    _start_galaxy()\n\n\ndef _migrate_galaxy_database():\n    with cd(env.galaxy_home):\n        sudo(\"bash -c 'export PYTHON_EGG_CACHE=eggs; python ./scripts/build_universe_config.py conf.d; python -ES ./scripts/fetch_eggs.py; ./create_db.sh'\", user=\"galaxy\")\n\n\ndef seed_database(galaxy_data):\n    with cd(env.galaxy_home):\n        sudo(\"rm -f seed.py\")\n        _setup_database_seed_file(galaxy_data)\n        sudo(\"bash -c 'export PYTHON_EGG_CACHE=eggs; python ./scripts/build_universe_config.py conf.d; python -ES ./scripts/fetch_eggs.py; python seed.py'\", user=\"galaxy\")\n\n\ndef seed_workflows(options):\n    wait_for_galaxy()\n    galaxy_data = options[\"galaxy\"]\n    with cd(os.path.join(env.galaxy_home, \"workflows\")):\n        for user in galaxy_data[\"users\"]:\n            api_key = user[\"api_key\"]\n            workflows = None\n            if \"workflows\" in user:\n                workflows = user[\"workflows\"]\n            if not workflows:\n                continue\n            for workflow in workflows:\n                sudo(\"bash -c 'export PYTHON_EGG_CACHE=eggs; bash import_all.sh %s %s'\" % (api_key, workflow), user=env.galaxy_user)\n\n\ndef _setup_database_seed_file(galaxy_data):\n    _seed_append(\"\"\"from scripts.db_shell import *\nfrom galaxy.util.bunch import Bunch\nfrom galaxy.security import GalaxyRBACAgent\nbunch = Bunch( **globals() )\nbunch.engine = engine\n# model.flush() has been removed.\nbunch.session = db_session\n# For backward compatibility with \"model.context.current\"\nbunch.context = db_session\nsecurity_agent = GalaxyRBACAgent( bunch )\nsecurity_agent.sa_session = sa_session\n\ndef add_user(email, password, key=None):\n    query = sa_session.query( User ).filter_by( email=email )\n    if query.count() > 0:\n        return query.first()\n    else:\n        user = User(email)\n        user.set_password_cleartext(password)\n        sa_session.add(user)\n        sa_session.flush()\n\n        security_agent.create_private_user_role( user )\n        if not user.default_permissions:\n            security_agent.user_set_default_permissions( user, history=True, dataset=True )\n\n        if key is not None:\n            api_key = APIKeys()\n            api_key.user_id = user.id\n            api_key.key = key\n            sa_session.add(api_key)\n            sa_session.flush()\n        return user\n\ndef add_history(user, name):\n    query = sa_session.query( History ).filter_by( user=user ).filter_by( name=name )\n    if query.count() == 0:\n        history = History(user=user, name=name)\n        sa_session.add(history)\n        sa_session.flush()\n        return history\n    else:\n        return query.first()\n\n\"\"\")\n    i = 0\n    for user in galaxy_data[\"users\"]:\n        username = user[\"username\"]\n        password = user[\"password\"]\n        api_key = user[\"api_key\"]\n        histories = None\n        if \"histories\" in user:\n            histories = user[\"histories\"]\n        user_object = \"user_%d\" % i\n        _seed_append(\"\"\"%s = add_user(\"%s\", \"%s\", \"%s\")\"\"\" % (user_object, username, password, api_key))\n        _import_histories(user_object, histories)\n        i = i + 1\n\n\ndef _import_histories(user_object, histories):\n    if not histories:\n        return\n    for history_name in histories:\n        _import_history(user_object, history_name)\n\n\ndef _import_history(user_object, history_name):\n    history_name_stripped = history_name.strip()\n    if history_name_stripped:\n        _seed_append(\"\"\"add_history(%s, \"%s\")\"\"\" % (user_object, history_name_stripped))\n\n\ndef _seed_append(text):\n    append(\"%s/seed.py\" % env.galaxy_home, text, use_sudo=True)\n\n\ndef _start_galaxy():\n    # Create directory to store galaxy service's pid file.\n    _make_dir_for_galaxy(\"/var/lib/galaxy\")\n    start_service(\"galaxy\")\n\n\ndef refresh_galaxy(target_galaxy_repo):\n    _update_galaxy(target_galaxy_repo)\n    sudo(\"/etc/init.d/galaxy restart\", pty=False)\n\n\ndef _setup_galaxy_log_dir():\n    _make_dir_for_galaxy(\"/var/log/galaxy\")\n\n\ndef _setup_shed_tools_dir():\n    _make_dir_for_galaxy(\"%s/../shed_tools\" % env.galaxy_home)\n\n\ndef _make_dir_for_galaxy(path):\n    sudo(\"mkdir -p '%s'\" % path)\n    _chown_galaxy(env, path)\n\n\ndef _update_galaxy(target_galaxy_repo):\n    # Need to merge? -John\n    hg_command = \"hg pull %s; hg update\" % target_galaxy_repo\n    with cd(env.galaxy_home):\n        sudo(hg_command, user=env.galaxy_user)\n\n\ndef refresh_galaxy_action(vm_launcher, options):\n    refresh_galaxy(env.galaxy_repository)\n\n\ndef setup_image(options):\n    _configure_package_holds(options)\n    configure_MI(env)\n    configure_smtp(options)\n    configure_sudoers(options)\n\n\ndef _configure_package_holds(options):\n    # No longer respected. TODO: Implement.\n    if 'package_holds' in options:\n        env.package_holds = options['package_holds']\n    else:\n        env.package_holds = None\n\n\ndef configure_smtp(options):\n    if 'smtp_server' in options:\n        smtp_server = options['smtp_server']\n        username = options['smtp_user']\n        password = options['smtp_password']\n        conf_file_contents = \"\"\"mailhub=%s\nUseSTARTTLS=YES\nAuthUser=%s\nAuthPass=%s\nFromLineOverride=YES\n\"\"\" % (smtp_server, username, password)\n        _apt_packages(pkg_list=[\"ssmtp\"])\n        sudo(\"\"\"echo \"%s\" > /etc/ssmtp/ssmtp.conf\"\"\" % conf_file_contents)\n        aliases = \"\"\"root:%s:%s\ngalaxy:%s:%s\n%s:%s:%s\"\"\" % (username, smtp_server, username, smtp_server, env.user, username, smtp_server)\n        sudo(\"\"\"echo \"%s\" > /etc/ssmtp/revaliases\"\"\" % aliases)\n\n\ndef configure_sudoers(options):\n    if \"sudoers_additions\" in options:\n        for addition in options[\"sudoers_additions\"]:\n            sudoers_append(addition)\n\n\ndef configure_MI(env):\n    # Clean this next line up.\n    _configure_and_install_native_packages(env, [\"minimal\", \"cloudman\", \"galaxy\"])\n    # _update_system()\n    _setup_users(env)\n    _setup_xvfb(env)\n    _required_programs(env)\n\n\n# == required programs\ndef _required_programs(env):\n    \"\"\" Install required programs \"\"\"\n    if not exists(env.install_dir):\n        sudo(\"mkdir -p %s\" % env.install_dir)\n        sudo(\"chown %s %s\" % (env.user, env.install_dir))\n\n    # Setup global environment for all users\n    install_dir = os.path.split(env.install_dir)[0]\n    exports = [\"export PATH=%s/bin:%s/sbin:$PATH\" % (install_dir, install_dir),\n               \"export LD_LIBRARY_PATH=%s/lib\" % install_dir]\n    for e in exports:\n        _ensure_export(e)\n    # Install required programs\n    _install_nginx_standalone(env)\n    _start_nginx(env)\n    _deploy_setup_postgresql(env)\n\n    # Verify this is not needed.\n    # _install_samtools()\n\n\ndef _ensure_export(command):\n    if not contains('/etc/bash.bashrc', command):\n        append('/etc/bash.bashrc', command, use_sudo=True)\n\n\ndef _start_nginx(env):\n    galaxy_data = env.galaxy_data_mount\n    env.safe_sudo(\"mkdir -p '%s'\" % env.galaxy_data)\n    _chown_galaxy(env, galaxy_data)\n    start_service(\"nginx\")\n\n\ndef _deploy_setup_postgresql(env):\n    ensure_can_sudo_into(\"postgres\")\n    _setup_postgresql(env)\n\n\nconfigure_actions = {\"setup_image\": setup_image,\n                     \"setup_genomes\": setup_genomes,\n                     \"purge_genomes\": purge_genomes,\n                     \"setup_galaxy\": setup_galaxy,\n                     \"purge_galaxy\": purge_galaxy,\n                    }\n\nready_actions = {\"galaxy_transfer\": galaxy_transfer,\n                 \"refresh_galaxy\": refresh_galaxy_action,\n                 \"copy_runtime_properties\": copy_runtime_properties,\n}\n\ncompound_actions = {\"configure\": [\"setup_image\", \"setup_tools\", \"setup_genomes\", \"setup_galaxy\", \"setup_ssh_key\"],\n                    \"reinstall_galaxy\": [\"purge_galaxy\", \"setup_galaxy\"],\n                    \"reinstall_genomes\": [\"purge_genomes\", \"setup_genomes\"],\n                    \"reinstall_tools\": [\"purge_tools\", \"setup_tools\"]\n}\n"
  },
  {
    "path": "cloudbio/deploy/util.py",
    "content": "from string import Template\nfrom time import strftime\nimport os\n\nfrom fabric.api import local, sudo, env, put, get\nfrom fabric.contrib.files import exists, append\n\n\ndef setup_install_dir():\n    \"\"\"Sets up install dir and ensures its owned by Galaxy\"\"\"\n    if not exists(env.install_dir):\n        sudo(\"mkdir -p %s\" % env.install_dir)\n    if not exists(env.jars_dir):\n        sudo(\"mkdir -p %s\" % env.jars_dir)\n    # TODO: Fix bug here\n    chown_galaxy(os.path.split(env.install_dir)[0])\n\n\ndef eval_template(env, template_str):\n    props = {\n        \"env\": env,\n        \"the_date\": strftime('%Y%m%d'),\n        \"the_date_with_time\": strftime('%Y%m%d_%H%M%S'),\n    }\n    return Template(template_str).safe_substitute(props)\n\n\ndef ensure_can_sudo_into(user):\n    sudoers_append(\"%admin  ALL = (\" + user + \") NOPASSWD: ALL\")\n\n\ndef sudoers_append(line):\n    append(\"/etc/sudoers\", line, use_sudo=True)\n\n\ndef start_service(service_name):\n    # For reasons I don't understand this doesn't work for galaxy init\n    # script unless pty=False\n    sudo(\"/etc/init.d/%s start\" % service_name, pty=False)\n\n\ndef wget(url, install_command=sudo, file_name=None):\n    if not file_name:\n        file_name = os.path.split(url)[-1]\n        if '?' in file_name:\n            file_name = file_name[0:file_name.index('?')]\n    if (\"cache_source_downloads\" in env) and (not env.cache_source_downloads):\n        install_command(\"wget %s -O %s\" % (url, file_name))\n    else:\n        cache_dir = env.source_cache_dir\n        if not cache_dir:\n            cache_dir = \".downloads\"\n        cached_file = os.path.join(cache_dir, file_name)\n        if os.path.exists(cached_file):\n            put(cached_file, file_name)\n        else:\n            install_command(\"wget %s -O %s\" % (url, file_name))\n            local(\"mkdir -p '%s'\" % cache_dir)\n            get(file_name, cached_file)\n"
  },
  {
    "path": "cloudbio/deploy/vmlauncher/__init__.py",
    "content": "from __future__ import print_function\nimport os\nimport time\n\nfrom libcloud.compute.ssh import SSHClient\nfrom libcloud.compute.base import NodeImage, NodeSize\nfrom libcloud.compute.types import Provider\nfrom libcloud.compute.providers import get_driver\nimport six\n\n# Ubuntu 10.04 LTS (Lucid Lynx) Daily Build [20120302]\nDEFAULT_AWS_IMAGE_ID = \"ami-0bf6af4e\"\nDEFAULT_AWS_SIZE_ID = \"m1.large\"\nDEFAULT_AWS_AVAILABILITY_ZONE = \"us-west-1\"\n\nfrom fabric.api import local, env, sudo, put, run\n\n\nclass VmLauncher:\n\n    def __init__(self, driver_options_key, options):\n        self.driver_options_key = driver_options_key\n        self.options = options\n        self.__set_and_verify_key()\n\n    def __set_and_verify_key(self):\n        key_file = self.options.get('key_file', None)\n        if not key_file:\n            key_file = self._driver_options()['key_file']\n        # Expand tildes in path\n        self.key_file = os.path.expanduser(key_file)\n        if not os.path.exists(self.key_file):\n            raise Exception(\"Invalid or unspecified key_file option: %s\" % self.key_file)\n\n    def _get_driver_options(self, driver_option_keys):\n        driver_options = {}\n        for key in driver_option_keys:\n            if key in self._driver_options():\n                driver_options[key] = self._driver_options()[key]\n        return driver_options\n\n    def _driver_options(self):\n        return self.options[self.driver_options_key]\n\n    def get_key_file(self):\n        return self.key_file\n\n    def boot_and_connect(self):\n        conn = self._connect_driver()\n        node = self._boot()  # Subclasses should implement this, and return libcloud node like object\n        self.conn = conn\n        self.node = node\n        self.uuid = node.uuid\n        self.connect(conn)\n\n    def _connect_driver(self):\n        if not getattr(self, 'conn', None):\n            self.conn = self._get_connection()\n        return self.conn\n\n    def _wait_for_node_info(self, f):\n        initial_value = f(self.node)\n        if initial_value:\n            return self._parse_node_info(initial_value)\n        while True:\n            time.sleep(10)\n            refreshed_node = self._find_node()\n            refreshed_value = f(refreshed_node)\n            if refreshed_value and not refreshed_value == []:\n                return self._parse_node_info(refreshed_value)\n\n    def _parse_node_info(self, value):\n        if isinstance(value, six.string_types):\n            return value\n        else:\n            return value[0]\n\n    def _find_node(self):\n        nodes = self.conn.list_nodes()\n        node_uuid = self.node.uuid\n        for node in nodes:\n            if node.uuid == node_uuid:\n                return node\n\n    def destroy(self, node=None):\n        self._connect_driver()\n        if node == None:\n            node = self.node\n        self.conn.destroy_node(node)\n\n    def __get_ssh_client(self):\n        ip = self.get_ip()  # Subclasses should implement this\n        key_file = self.get_key_file()\n        ssh_client = SSHClient(hostname=ip,\n                               port=self.get_ssh_port(),\n                               username=self.get_user(),\n                               key=key_file)\n        return ssh_client\n\n    def get_user(self):\n        return \"ubuntu\"\n\n    def get_ssh_port(self):\n        return 22\n\n    def connect(self, conn, tries=5):\n        i = 0\n        while i < tries:\n            try:\n                ssh_client = self.__get_ssh_client()\n                conn._ssh_client_connect(ssh_client=ssh_client, timeout=60)\n                return\n            except:\n                i = i + 1\n\n    def list(self):\n        self._connect_driver()\n        return self.conn.list_nodes()\n\n    def _boot(self):\n        conn = self.conn\n        boot_new = True\n        last_instance_path = None\n        if 'use_existing_instance' in self._driver_options():\n            boot_new = False\n            instance_id = self._driver_options()['use_existing_instance']\n            if instance_id == \"__auto__\":\n                last_instance_path = \".vmlauncher_last_instance_%s\" % self.driver_options_key\n                if not os.path.exists(last_instance_path):\n                    boot_new = True\n                else:\n                    instance_id = open(last_instance_path, \"r\").read()\n            if not boot_new:\n                nodes = conn.list_nodes()\n                nodes_with_id = [node for node in nodes if node.uuid == instance_id]\n                if not nodes_with_id:\n                    err_msg_template = \"Specified use_existing_instance with instance id %s, but no such instance found.\"\n                    raise Exception(err_msg_template % instance_id)\n                node = nodes_with_id[0]\n        if boot_new:\n            node = self._boot_new(conn)\n            if last_instance_path:\n                open(last_instance_path, \"w\").write(node.uuid)\n        return node\n\n    def _image_from_id(self, image_id=None):\n        image = NodeImage(id=image_id, name=\"\", driver=\"\")\n        return image\n\n    def _get_image_id(self, image_id=None):\n        if not image_id:\n            if 'image_id' in self._driver_options():\n                image_id = self._driver_options()['image_id']\n            else:\n                image_id = self._default_image_id()\n        return image_id\n\n    def _default_image_id(self):\n        return None\n\n    def _get_default_size_id(self):\n        return None\n\n    def _get_size_id_option(self):\n        return \"size_id\"\n\n    def _size_from_id(self, size_id):\n        size = NodeSize(id=size_id, name=\"\", ram=None, disk=None, bandwidth=None, price=None, driver=\"\")\n        return size\n\n    def _get_size_id(self, size_id=None):\n        if not size_id:\n            size_id_option = self._get_size_id_option()\n            if size_id_option in self._driver_options():\n                size_id = self._driver_options()[size_id_option]\n            else:\n                size_id = self._get_default_size_id()\n        return size_id\n\n    def _boot_new(self, conn):\n        hostname = self.options.get(\"hostname\", \"vm_launcher_instance\")\n        node = self.create_node(hostname)\n        return node\n\n    def access_id(self):\n        return self._driver_options()[\"access_id\"]\n\n    def secret_key(self):\n        return self._driver_options()[\"secret_key\"]\n\n    def package_image_name(self):\n        name = self._driver_options()[\"package_image_name\"] or \"cloudbiolinux\"\n        return name\n\n    def package_image_description(self, default=\"\"):\n        description = self._driver_options().get(\"package_image_description\", default)\n        return description\n\n\nclass VagrantConnection:\n    \"\"\"'Fake' connection type to mimic libcloud's but for Vagrant\"\"\"\n\n    def _ssh_client_connect(self, ssh_client):\n        pass\n\n    def destroy_node(self, node=None):\n        local(\"vagrant halt\")\n\n    def list_nodes(self):\n        return [VagrantNode()]\n\n\nclass VagrantNode:\n\n    def __init__(self):\n        self.name = \"vagrant\"\n        self.uuid = \"vagrant\"\n\n\nclass VagrantVmLauncher(VmLauncher):\n    \"\"\"Launches vagrant VMs.\"\"\"\n\n    def _get_connection():\n        return VagrantConnection()\n\n    def __init__(self, driver_options_key, options):\n        if not 'key_file' in options:\n            options['key_file'] = os.path.join(os.environ[\"HOME\"], \".vagrant.d\", \"insecure_private_key\")\n        VmLauncher.__init__(self, driver_options_key, options)\n        self.uuid = \"test\"\n\n    def _boot(self):\n        local(\"vagrant up\")\n        return VagrantNode()\n\n    def get_ip(self):\n        return \"33.33.33.11\"\n\n    def get_user(self):\n        return \"vagrant\"\n\n    def package(self, **kwds):\n        local(\"vagrant package\")\n\n\nclass OpenstackVmLauncher(VmLauncher):\n    \"\"\" Wrapper around libcloud's openstack API. \"\"\"\n\n    def get_ip(self):\n        return self._wait_for_node_info(lambda node: node.public_ips + node.private_ips)\n\n    def _get_size_id_option(self):\n        return \"flavor_id\"\n\n    def create_node(self, hostname, image_id=None, size_id=None, **kwds):\n        image_id = self._get_image_id()\n        image = self._image_from_id(image_id)\n        size_id = self._get_size_id()\n        size = self._size_from_id(size_id)\n        if 'ex_keyname' not in kwds:\n            kwds['ex_keyname'] = self._driver_options()['ex_keyname']\n\n        node = self.conn.create_node(name=hostname,\n                                     image=image,\n                                     size=size,\n                                     **kwds)\n        return node\n\n    def _get_connection(self):\n        driver = get_driver(Provider.OPENSTACK)\n        openstack_username = self._driver_options()['username']\n        openstack_api_key = self._driver_options()['password']\n\n        driver_option_keys = ['host',\n                              'secure',\n                              'port',\n                              'ex_force_auth_url',\n                              'ex_force_auth_version',\n                              'ex_force_base_url',\n                              'ex_tenant_name']\n\n        driver_options = self._get_driver_options(driver_option_keys)\n        conn = driver(openstack_username,\n                      openstack_api_key,\n                      **driver_options)\n        return conn\n\n    def package(self, **kwds):\n        name = kwds.get(\"name\", self.package_image_name())\n        self.conn.ex_save_image(self.node, name)\n\n    def attach_public_ip(self, public_ip=None):\n        if not public_ip:\n            public_ip = self._driver_options()[\"public_ip\"]\n        self.conn._node_action(self.node, \"addFloatingIp\", address=public_ip)\n\n\nclass EucalyptusVmLauncher(VmLauncher):\n\n    def get_ip(self):\n        return self._wait_for_node_info(lambda node: node.public_ips)\n\n    def _get_connection(self):\n        driver = get_driver(Provider.EUCALYPTUS)\n        driver_option_keys = ['secret',\n                              'secure',\n                              'port',\n                              'host',\n                              'path']\n\n        driver_options = self._get_driver_options(driver_option_keys)\n        ec2_access_id = self.access_id()\n        conn = driver(ec2_access_id, **driver_options)\n        return conn\n\n    def create_node(self, hostname, image_id=None, size_id=None, **kwds):\n        image_id = self._get_image_id()\n        image = self._image_from_id(image_id)\n        size_id = self._get_size_id()\n        size = self._size_from_id(size_id)\n        if 'ex_keyname' not in kwds:\n            kwds['ex_keyname'] = self._driver_options()[\"keypair_name\"]\n        node = self.conn.create_node(name=hostname,\n                                     image=image,\n                                     size=size,\n                                     **kwds)\n        return node\n\n\nclass Ec2VmLauncher(VmLauncher):\n\n    def get_ip(self):\n        return self._wait_for_node_info(lambda node: node.extra['dns_name'])\n\n    def boto_connection(self):\n        \"\"\"\n        Establish a boto library connection (for functionality not available in libcloud).\n        \"\"\"\n        import boto.ec2\n        region = boto.ec2.get_region(self._availability_zone())\n        ec2_access_id = self.access_id()\n        ec2_secret_key = self.secret_key()\n        return region.connect(aws_access_key_id=ec2_access_id, aws_secret_access_key=ec2_secret_key)\n\n    def boto_s3_connection(self):\n        from boto.s3.connection import S3Connection\n        ec2_access_id = self.access_id()\n        ec2_secret_key = self.secret_key()\n        return S3Connection(ec2_access_id, ec2_secret_key)\n\n    def _default_image_id(self):\n        return DEFAULT_AWS_IMAGE_ID\n\n    def package(self, **kwds):\n        package_type = self._driver_options().get('package_type', 'default')\n        if package_type == \"create_image\":\n            self._create_image(**kwds)\n        else:\n            self._default_package(**kwds)\n\n    def _create_image(self, **kwds):\n        ec2_conn = self.boto_connection()\n        instance_id = run(\"curl --silent http://169.254.169.254/latest/meta-data/instance-id\")\n\n        if \"name\" not in kwds:\n            name = self.package_image_name()\n        else:\n            name = kwds[\"name\"]\n\n        if \"description\" not in kwds:\n            description = self.package_image_description(default=\"\")\n        else:\n            description = kwds[\"description\"]\n\n        image_id = ec2_conn.create_image(instance_id, name=name, description=description)\n        if self._driver_options().get(\"make_public\", False):\n            ec2_conn.modify_image_attribute(image_id, attribute='launchPermission', operation='add', groups=['all'])\n\n    def _default_package(self, **kwds):\n        env.packaging_dir = \"/mnt/packaging\"\n        sudo(\"mkdir -p %s\" % env.packaging_dir)\n        self._copy_keys()\n        self._install_ec2_tools()\n        self._install_packaging_scripts()\n\n    def _install_ec2_tools(self):\n        sudo(\"apt-add-repository ppa:awstools-dev/awstools\")\n        sudo(\"apt-get update\")\n        sudo('export DEBIAN_FRONTEND=noninteractive; sudo -E apt-get install ec2-api-tools ec2-ami-tools -y --force-yes')\n\n    def _install_packaging_scripts(self):\n        user_id = self._driver_options()[\"user_id\"]\n        bundle_cmd = \"sudo ec2-bundle-vol -k %s/ec2_key -c%s/ec2_cert -u %s\" % \\\n            (env.packaging_dir, env.packaging_dir, user_id)\n        self._write_script(\"%s/bundle_image.sh\" % env.packaging_dir, bundle_cmd)\n\n        bucket = self._driver_options()[\"package_bucket\"]\n        upload_cmd = \"sudo ec2-upload-bundle -b %s -m /tmp/image.manifest.xml -a %s -s %s\" % \\\n            (bucket,  self.access_id(), self.secret_key())\n        self._write_script(\"%s/upload_bundle.sh\" % env.packaging_dir, upload_cmd)\n\n        name = self.package_image_name()\n\n        manifest = \"image.manifest.xml\"\n        register_cmd = \"sudo ec2-register -K %s/ec2_key -C %s/ec2_cert %s/%s -n %s\" % (env.packaging_dir, env.packaging_dir, bucket, manifest, name)\n        self._write_script(\"%s/register_bundle.sh\" % env.packaging_dir, register_cmd)\n\n    def _write_script(self, path, contents):\n        full_contents = \"#!/bin/bash\\n%s\" % contents\n        sudo(\"echo '%s' > %s\" % (full_contents, path))\n        sudo(\"chmod +x %s\" % path)\n\n    def _copy_keys(self):\n        ec2_key_path = self._driver_options()[\"x509_key\"]\n        ec2_cert_path = self._driver_options()[\"x509_cert\"]\n        put(ec2_key_path, \"%s/ec2_key\" % env.packaging_dir, use_sudo=True)\n        put(ec2_cert_path, \"%s/ec2_cert\" % env.packaging_dir, use_sudo=True)\n\n    def _availability_zone(self):\n        if \"availability_zone\" in self._driver_options():\n            availability_zone = self._driver_options()[\"availability_zone\"]\n        else:\n            availability_zone = DEFAULT_AWS_AVAILABILITY_ZONE\n        return availability_zone\n\n    def _get_default_size_id(self):\n        return DEFAULT_AWS_SIZE_ID\n\n    def _get_location(self):\n        availability_zone = self._availability_zone()\n        locations = self.conn.list_locations()\n        for location in locations:\n            if location.availability_zone.name == availability_zone:\n                break\n        return location\n\n    def create_node(self, hostname, image_id=None, size_id=None, location=None, **kwds):\n        self._connect_driver()\n        image_id = self._get_image_id(image_id)\n        image = self._image_from_id(image_id)\n\n        size_id = self._get_size_id(size_id)\n        size = self._size_from_id(size_id)\n\n        if not location:\n            location = self._get_location()\n\n        if not \"ex_keyname\" in kwds:\n            keyname = self._driver_options()[\"keypair_name\"]\n            kwds[\"ex_keyname\"] = keyname\n\n        node = self.conn.create_node(name=hostname,\n                                     image=image,\n                                     size=size,\n                                     location=location,\n                                     **kwds)\n        return node\n\n    def attach_public_ip(self, public_ip=None):\n        if not public_ip:\n            public_ip = self._driver_options()[\"public_ip\"]\n        self.conn.ex_associate_addresses(self.node, public_ip)\n\n    def _get_connection(self):\n        driver = get_driver(Provider.EC2)\n        ec2_access_id = self.access_id()\n        ec2_secret_key = self.secret_key()\n        conn = driver(ec2_access_id, ec2_secret_key)\n        return conn\n\n\ndef build_vm_launcher(options):\n    provider_option_key = 'vm_provider'\n    # HACK to maintain backward compatibity on vm_host option\n    if not 'vm_provider' in options and 'vm_host' in options:\n        print(\"Using deprecated 'vm_host' setting, please change this to 'vm_provider'\")\n        provider_option_key = 'vm_host'\n    driver = options.get(provider_option_key, 'aws')   # Will just fall back on EC2\n    driver_options_key = driver\n    if driver in options:\n        # Allow multiple sections or providers per driver (e.g. aws-project-1).\n        # Assume the driver is just the provider name unless the provider\n        # section sets an explict driver option. In above example,\n        # the aws-project-1 would have to have a \"driver: 'aws'\" option\n        # set.\n        provider_options = options.get(driver)\n        driver = provider_options.get('driver', driver)\n    driver_classes = {'openstack': OpenstackVmLauncher,\n                      'vagrant': VagrantVmLauncher,\n                      'eucalyptus': EucalyptusVmLauncher}\n    driver_class = driver_classes.get(driver, Ec2VmLauncher)\n    vm_launcher = driver_class(driver_options_key, options)\n    return vm_launcher\n"
  },
  {
    "path": "cloudbio/deploy/vmlauncher/config.md",
    "content": "# Configuring Cloud Parameters\n\nCurrently four different virtual machine providers are implemented: `aws`\n(default), `openstack`, `eucalyptus` (partial support), and `vagrant`. Request\nfor supporting additional cloud infrastructures can be created here\nhttps://github.com/jmchilton/vm-launcher/issues/new or pull requests are\nalways welcome. The `vm-launcher` project is built heavily on Apache\n[libcloud], so support should be implemented at that level first, though\ndozens of cloud providers are currently implemented.\n\n## aws \n\nThis cloud supports the following compute parameters: `access_id`,\n`secret_key`, `size_id`, `image_id`, `availability_zone`.\n\nThe aws driver supports two packaging modes, this is the code that is called\nwhen the `package` action is executed. By default, package will cause some\nscripts to be created on the remote server to aid in packaging, if however\n`package_type` parameter is set to `create_image`, the Amazon EC2 CreateImage\n(http://support.smartbear.com/viewarticle/22739/) operation will be used to\nautomatically package the target instance. `create_image` mode can only be\nused for EBS backed instances, which is why the other more complex mode is the\ndefault.\n\nWhen `create_image` is enabled, the additional packaging parameters include\n`package_image_name`, `package_image_description`, and `make_public`.\n\nFor the default packaging mode, many additional parameters related to S3 must\nbe set including `x509_cert`, `x509_key`, `user_id`, and `package_bucket`.\n\n## openstack\n\nOpenStack may be targetted using either the native OpenStack APIs or using the\nEC2 compatibility layer (e.g. what boto does). To target the EC2 compatibility\nuse the `eucalyptus` driver, this `openstack` driver targets the native API.\n\nThis driver allows the following parameters `username`, `password`, `host`,\n`secure` (boolean), `port`, `ex_force_uth_url`, `ex_force_base_url`,\n`ex_force_auth_version`, `ex_tenant_name`, `flavor_id`, `image_id`,\n`keypair_name`, and `package_image_name`.\n\n## eucalyptus\n\nSupport for the eucalyptus driver is somewhat experimental at this time and automated packaging is not available. This driver can be configured via the following options:: `secret`, `secure`, `port`, `host`, `path`, `size_id`.\n\n## vagrant\n\nThe vagrant driver supports no additional parameters, a precise64 box\nshould be configured though this can be tweaks by adjusting the file\n`Vagrantfile`.\n\n[libcloud]: http://libcloud.apache.org/\n\n\n\n"
  },
  {
    "path": "cloudbio/deploy/vmlauncher/transfer.py",
    "content": "from __future__ import print_function\nimport os\nimport gzip\n\nfrom operator import itemgetter\nfrom sys import exit\nfrom threading import Thread\nfrom threading import Condition\nfrom Queue import Queue\n\nfrom fabric.api import local, put, sudo, cd\nfrom fabric.colors import red\n\n\n\nclass FileSplitter:\n    \"\"\"\n    Works like the UNIX split command break up a file into parts like:\n        filename_aaaaaaaaa\n        filename_aaaaaaaab\n        etc...\n    \"\"\"\n\n    def __init__(self, chunk_size, destination_directory, callback):\n        self.chunk_size = chunk_size * 1024 * 1024\n        self.destination_directory = destination_directory\n        self.chunk_callback = callback\n\n    def split_file(self, path, compress, transfer_target):\n        basename = os.path.basename(path)\n        file_size = os.path.getsize(path)\n        total_bytes = 0\n        chunk_num = 0\n        suffix = ''\n        if compress:\n            suffix = '.gz'\n\n        input = open(path, 'rb')\n        while True:\n            chunk_name = \"%s_part%08d%s\" % (basename, chunk_num, suffix)\n            chunk_path = os.path.join(self.destination_directory, chunk_name)\n            this_chunk_size = min(self.chunk_size, file_size - total_bytes)\n            if this_chunk_size <= 0:\n                break\n\n            chunk = input.read(this_chunk_size)\n            total_bytes += len(chunk)\n            if compress:\n                chunk_output = gzip.open(chunk_path, 'wb')\n            else:\n                chunk_output = file(chunk_path, 'wb')\n            chunk_output.write(chunk)\n            chunk_output.close()\n\n            self.chunk_callback.handle_chunk(chunk_path, transfer_target)\n            chunk_num += 1\n\n\nclass TransferTarget:\n\n    def __init__(self, file, precompressed, transfer_manager):\n        self.file = file\n        self.precompressed = precompressed\n        self.do_compress = transfer_manager.compress\n        self.do_split = transfer_manager.chunk_size > 0\n        self.local_temp = transfer_manager.local_temp\n        basename = os.path.basename(file)\n        if len(basename) < 1:\n            print(red(Exception(\"Invalid file specified - %s\" % file)))\n            exit(-1)\n        self.basename = basename\n\n    def should_compress(self):\n        return not self.precompressed and self.do_compress\n\n    def split_up(self):\n        return self.do_split\n\n    def clean(self):\n        if self.should_compress():\n            local(\"rm -rf '%s'\" % self.compressed_file())\n\n    def compressed_basename(self):\n        if not self.precompressed:\n            compressed_basename = \"%s.gz\" % self.basename\n        else:\n            compressed_basename = self.basename\n        return compressed_basename\n\n    def decompressed_basename(self):\n        basename = self.basename\n        if basename.endswith(\".gz\"):\n            decompressed_basename = basename[:-len(\".gz\")]\n        else:\n            decompressed_basename = basename\n        return decompressed_basename\n\n    def compressed_file(self):\n        compressed_file = \"%s/%s.gz\" % (self.local_temp, self.basename)\n        return compressed_file\n\n    def build_simple_chunk(self):\n        if self.should_compress():\n            compressed_file = self.compressed_file()\n            local(\"gzip -f -9 '%s' -c > '%s'\" % (self.file, compressed_file))\n            return TransferChunk(compressed_file, self)\n        else:\n            return TransferChunk(self.file, self)\n\n\nclass TransferChunk:\n\n    def __init__(self, chunk_path, transfer_target):\n        self.chunk_path = chunk_path\n        self.transfer_target = transfer_target\n\n    def clean_up(self):\n        was_split = self.transfer_target.split_up()\n        was_compressed = self.transfer_target.should_compress()\n        if was_split or was_compressed:\n            local(\"rm '%s'\" % self.chunk_path)\n\n\nclass FileTransferManager:\n\n    def __init__(self,\n                 compress=True,\n                 num_compress_threads=1,\n                 num_transfer_threads=1,\n                 num_decompress_threads=1,\n                 chunk_size=0,\n                 transfer_retries=3,\n                 destination=\"/tmp\",\n                 transfer_as=\"root\",\n                 local_temp=None):\n        self.compress = compress\n        self.num_compress_threads = num_compress_threads\n        self.num_transfer_threads = num_transfer_threads\n        self.num_decompress_threads = num_decompress_threads\n        self.chunk_size = chunk_size\n        self.transfer_retries = transfer_retries\n        self.destination = destination\n        self.transfer_as = transfer_as\n        self.local_temp = local_temp\n\n        if not self.local_temp:\n            self.local_temp = \"/tmp\"\n\n        local(\"mkdir -p '%s'\" % self.local_temp)\n        self.file_splitter = FileSplitter(self.chunk_size, self.local_temp, self)\n\n    def handle_chunk(self, chunk, transfer_target):\n        self._enqueue_chunk(TransferChunk(chunk, transfer_target))\n\n    def transfer_files(self, files=[], compressed_files=[]):\n        self.transfer_complete = False\n        self.transfer_complete_condition = Condition()\n\n        self._setup_destination_directory()\n\n        self._setup_workers()\n\n        self._enqueue_files(files, compressed_files)\n\n        self._wait_for_completion()\n\n    def _setup_workers(self):\n        self._setup_compress_threads()\n        self._setup_transfer_threads()\n        self._setup_decompress_threads()\n\n    def _setup_destination_directory(self):\n        sudo(\"mkdir -p %s\" % self.destination)\n        self._chown(self.destination)\n\n    def _setup_compress_threads(self):\n        self.compress_queue = Queue()\n        self._launch_threads(self.num_compress_threads, self._compress_files)\n\n    def _setup_decompress_threads(self):\n        self.decompress_queue = Queue()\n        self._launch_threads(self.num_decompress_threads, self._decompress_files)\n\n    def _setup_transfer_threads(self):\n        self.transfer_queue = Queue()  # For now just transfer one file at a time\n        self._launch_threads(self.num_transfer_threads, self._put_files)\n\n    def _launch_threads(self, num_threads, func):\n        for thread_index in range(num_threads):\n            t = Thread(target=func)\n            t.daemon = True\n            t.start()\n\n    def _enqueue_files(self, files, compressed_files):\n        transfer_targets = []\n\n        for file in files:\n            transfer_target = TransferTarget(file, False, self)\n            transfer_targets.append(transfer_target)\n\n        for compressed_file in compressed_files:\n            transfer_target = TransferTarget(compressed_file, True, self)\n            transfer_targets.append(transfer_target)\n\n        transfer_targets = self._sort_transfer_targets(transfer_targets)\n        for transfer_target in transfer_targets:\n            self.compress_queue.put(transfer_target)\n\n    def _sort_transfer_targets(self, transfer_targets):\n        for i in range(len(transfer_targets)):\n            transfer_target = transfer_targets[i]\n            transfer_targets[i] = transfer_target, os.stat(transfer_target.file).st_size\n        transfer_targets.sort(key=itemgetter(1), reverse=True)\n        return  [transfer_target[0] for transfer_target in transfer_targets]\n\n    def _wait_for_completion(self):\n        self.compress_queue.join()\n        self.transfer_queue.join()\n        self.transfer_complete_condition.acquire()\n        self.transfer_complete = True\n        self.transfer_complete_condition.notifyAll()\n        self.transfer_complete_condition.release()\n        self.decompress_queue.join()\n\n    def _compress_files(self):\n        while True:\n            try:\n                transfer_target = self.compress_queue.get()\n                file = transfer_target.file\n                if self.chunk_size > 0:\n                    should_compress = transfer_target.should_compress()\n                    self.file_splitter.split_file(file, should_compress, transfer_target)\n                    self.decompress_queue.put(transfer_target)\n                else:\n                    simple_chunk = transfer_target.build_simple_chunk()\n                    self._enqueue_chunk(simple_chunk)\n            except Exception as e:\n                print(red(\"Failed to compress a file to transfer\"))\n                print(red(e))\n            finally:\n                self.compress_queue.task_done()\n\n    def _decompress_files(self):\n        if self.chunk_size > 0:\n            self.transfer_complete_condition.acquire()\n            while not self.transfer_complete:\n                self.transfer_complete_condition.wait()\n            self.transfer_complete_condition.release()\n        while True:\n            try:\n                transfer_target = self.decompress_queue.get()\n                basename = transfer_target.basename\n                chunked = transfer_target.split_up()\n                compressed = transfer_target.do_compress or transfer_target.precompressed\n                with cd(self.destination):\n                    if compressed and chunked:\n                        destination = transfer_target.decompressed_basename()\n                        if transfer_target.precompressed:\n                            sudo(\"cat '%s_part'* | gunzip -c > %s\" % (basename, destination), user=self.transfer_as)\n                        else:\n                            sudo(\"zcat '%s_part'* > %s\" % (basename, destination), user=self.transfer_as)\n                        sudo(\"rm '%s_part'*\" % (basename), user=self.transfer_as)\n                    elif compressed:\n                        sudo(\"gunzip -f '%s'\" % transfer_target.compressed_basename(), user=self.transfer_as)\n                    elif chunked:\n                        sudo(\"cat '%s'_part* > '%s'\" % (basename, basename), user=self.transfer_as)\n                        sudo(\"rm '%s_part'*\" % (basename), user=self.transfer_as)\n            except Exception as e:\n                print(red(\"Failed to decompress or unsplit a transfered file.\"))\n                print(red(e))\n            finally:\n                self.decompress_queue.task_done()\n\n    def _put_files(self):\n        while True:\n            try:\n                transfer_chunk = self.transfer_queue.get()\n                transfer_target = transfer_chunk.transfer_target\n                compressed_file = transfer_chunk.chunk_path\n                basename = os.path.basename(compressed_file)\n                self._put_as_user(compressed_file, \"%s/%s\" % (self.destination, basename))\n                if not transfer_target.split_up():\n                    self.decompress_queue.put(transfer_target)\n            except Exception as e:\n                print(red(\"Failed to upload a file.\"))\n                print(red(e))\n            finally:\n                transfer_chunk.clean_up()\n                self.transfer_queue.task_done()\n\n    def _chown(self, destination):\n        sudo(\"chown %s:%s '%s'\" % (self.transfer_as, self.transfer_as, destination))\n\n    def _put_as_user(self, source, destination):\n        for attempt in range(self.transfer_retries):\n            retry = False\n            try:\n                put(source, destination, use_sudo=True)\n                self._chown(destination)\n            except BaseException as e:\n                retry = True\n                print(red(e))\n                print(red(\"Failed to upload %s on attempt %d\" % (source, attempt + 1)))\n            except:\n                # Should never get here, delete this block when more confident\n                retry = True\n                print(red(\"Failed to upload %s on attempt %d\" % (source, attempt + 1)))\n            finally:\n                if not retry:\n                    return\n        print(red(\"Failed to transfer file %s, exiting...\" % source))\n        exit(-1)\n\n    def _enqueue_chunk(self, transfer_chunk):\n        self.transfer_queue.put(transfer_chunk)\n"
  },
  {
    "path": "cloudbio/deploy/volume.py",
    "content": "from __future__ import print_function\nfrom fabric.api import run, env\nfrom time import sleep\nfrom boto.exception import EC2ResponseError\nfrom .util import eval_template\n\n\ndef attach_volumes(vm_launcher, options, format=False):\n    \"\"\"\n    \"\"\"\n    volumes = options.get(\"volumes\", [])\n    if not volumes:\n        return\n    boto_connection = vm_launcher.boto_connection()\n    instance_id = run(\"curl --silent http://169.254.169.254/latest/meta-data/instance-id\")\n    for volume in volumes:\n        volume_id = volume['id']\n        device_id = volume['device']\n        if not _get_attached(boto_connection, instance_id, device_id, valid_states=[\"attached\", \"attaching\"]):\n            boto_connection.attach_volume(volume_id, instance_id, device_id)\n    for volume in volumes:\n        volume_id = volume['id']\n        device_id = volume['device']\n        path = volume.get(\"path\")\n\n        while True:\n            if _get_attached(boto_connection, instance_id, device_id):\n                break\n\n            sleep(5)\n            print(\"Waiting for volume corresponding to device %s to attach\" % device_id)\n            break\n\n        # Don't mount if already mounted\n        if _find_mounted_device_id(path):\n            continue\n\n        format = str(volume.get('format', \"False\")).lower()\n        if format == \"true\":\n            _format_device(device_id)\n        env.safe_sudo(\"mkdir -p '%s'\" % path)\n        try:\n            _mount(device_id, path)\n        except:\n            if format == \"__auto__\":\n                print(\"Failed to mount device. format is set to __auto__ so will now format device and retry mount\")\n                _format_device(device_id)\n                _mount(device_id, path)\n            else:\n                raise\n\n\ndef _mount(device_id, path):\n    env.safe_sudo(\"mount '%s' '%s'\" % (device_id, path))\n\n\ndef _format_device(device_id):\n    env.safe_sudo(\"mkfs -t ext3 %s\" % device_id)\n\n\ndef detach_volumes(vm_launcher, options):\n    volumes = options.get(\"volumes\", [])\n    if not volumes:\n        return\n\n    boto_connection = vm_launcher.boto_connection()\n    instance_id = run(\"curl --silent http://169.254.169.254/latest/meta-data/instance-id\")\n    for volume in volumes:\n        volume_id = volume['id']\n        path = volume.get(\"path\")\n        env.safe_sudo(\"umount '%s'\" % path)\n        _detach(boto_connection, instance_id, volume_id)\n\n\ndef make_snapshots(vm_launcher, options):\n    volumes = options.get(\"volumes\", [])\n    for volume in volumes:\n        path = volume.get(\"path\")\n        desc = volume.get(\"description\", \"Snapshot of path %s\" % path)\n        desc = eval_template(env, desc)\n        # Allow volume to specify it should not be snapshotted, e.g. if\n        # piggy backing on core teams snapshots for galaxyIndicies for instance.\n        snapshot = volume.get(\"snapshot\", True)\n        if snapshot:\n            _make_snapshot(vm_launcher, path, desc)\n\n\ndef _get_attached(conn, instance_id, device_id, valid_states=['attached']):\n    vol_list = conn.get_all_volumes()\n    fs_vol = None\n    for vol in vol_list:\n        if vol.attach_data.instance_id == instance_id and vol.attach_data.device == device_id:\n            if vol.attach_data.status in valid_states:\n                fs_vol = vol\n                break\n    return fs_vol\n\n\ndef _make_snapshot(vm_launcher, fs_path, desc):\n    \"\"\" Create a snapshot of an existing volume that is currently attached to an\n    instance, taking care of the unmounting and detaching. If you specify the\n    optional argument (:galaxy), the script will pull the latest Galaxy code\n    from bitbucket and perform an update before snapshotting. Else, the script\n    will prompt for the file system path to be snapshoted.\n\n    In order for this to work, an instance on EC2 needs to be running with a\n    volume that wants to be snapshoted attached and mounted. The script will\n    unmount the volume, create a snaphost and offer to reattach and mount the\n    volume or create a new one from the freshly created snapshot.\n\n    Except for potentially Galaxy, MAKE SURE there are no running processes\n    using the volume and that no one is logged into the instance and sitting\n    in the given directory.\n    \"\"\"\n    instance_id = run(\"curl --silent http://169.254.169.254/latest/meta-data/instance-id\")\n    availability_zone = run(\"curl --silent http://169.254.169.254/latest/meta-data/placement/availability-zone\")\n    instance_region = availability_zone[:-1]  # Truncate zone letter to get region name\n    # Find the device where the file system is mounted to\n    # Find the EBS volume where the file system resides\n    device_id = _find_mounted_device_id(fs_path)\n    ec2_conn = vm_launcher.boto_connection()\n    fs_vol = _get_attached(ec2_conn, instance_id, device_id)\n    if fs_vol:\n        env.safe_sudo(\"umount %s\" % fs_path)\n        _detach(ec2_conn, instance_id, fs_vol.id)\n        snap_id = _create_snapshot(ec2_conn, fs_vol.id, desc)\n        # TODO: Auto Update snaps?\n        make_public = True\n        if make_public:  # Make option\n            ec2_conn.modify_snapshot_attribute(snap_id, attribute='createVolumePermission', operation='add', groups=['all'])\n        reattach = True\n        if reattach:\n            _attach(ec2_conn, instance_id, fs_vol.id, device_id)\n            env.safe_sudo(\"mount %s %s\" % (device_id, fs_path))\n        delete_old_volume = False\n        if delete_old_volume:\n            _delete_volume(ec2_conn, fs_vol.id)\n        print(\"----- Done snapshoting volume '%s' for file system '%s' -----\" % (fs_vol.id, fs_path))\n    else:\n        print(\"ERROR: Failed to find require file system, is boto installed? Is it not actually mounted?\")\n\n\ndef _find_mounted_device_id(path):\n    # Adding dollar sign to grep to distinguish between /mnt/galaxy and /mnt/galaxyIndices\n    device_id = env.safe_sudo(\"df | grep '%s$' | awk '{print $1}'\" % path)\n    return device_id\n\n\ndef _attach(ec2_conn, instance_id, volume_id, device):\n    \"\"\"\n    Attach EBS volume to the given device (using boto).\n    Try it for some time.\n    \"\"\"\n    try:\n        print(\"Attaching volume '%s' to instance '%s' as device '%s'\" % (volume_id, instance_id, device))\n        volumestatus = ec2_conn.attach_volume(volume_id, instance_id, device)\n    except EC2ResponseError as e:\n        print(\"Attaching volume '%s' to instance '%s' as device '%s' failed. Exception: %s\" % (volume_id, instance_id, device, e))\n        return False\n\n    for counter in range(30):\n        print(\"Attach attempt %s, volume status: %s\" % (counter, volumestatus))\n        if volumestatus == 'attached':\n            print(\"Volume '%s' attached to instance '%s' as device '%s'\" % (volume_id, instance_id, device))\n            break\n        if counter == 29:\n            print(\"Volume '%s' FAILED to attach to instance '%s' as device '%s'. Aborting.\" % (volume_id, instance_id, device))\n            return False\n        volumes = ec2_conn.get_all_volumes([volume_id])\n        volumestatus = volumes[0].attachment_state()\n        sleep(3)\n    return True\n\n\ndef _detach(ec2_conn, instance_id, volume_id):\n    \"\"\"\n    Detach EBS volume from the given instance (using boto).\n    Try it for some time.\n    \"\"\"\n    try:\n        volumestatus = ec2_conn.detach_volume( volume_id, instance_id, force=True )\n    except EC2ResponseError as e:\n        print(\"Detaching volume '%s' from instance '%s' failed. Exception: %s\" % ( volume_id, instance_id, str(e) ))\n        return False\n\n    for counter in range( 30 ):\n        print(\"Volume '%s' status '%s'\" % ( volume_id, volumestatus ))\n        if volumestatus == 'available':\n            print(\"Volume '%s' successfully detached from instance '%s'.\" % ( volume_id, instance_id ))\n            break\n        if counter == 29:\n            print(\"Volume '%s' FAILED to detach to instance '%s'.\" % ( volume_id, instance_id ))\n        sleep(3)\n        volumes = ec2_conn.get_all_volumes( [volume_id] )\n        volumestatus = volumes[0].status\n\n\ndef _delete_volume(ec2_conn, vol_id):\n    try:\n        ec2_conn.delete_volume(vol_id)\n        print(\"Deleted volume '%s'\" % vol_id)\n    except EC2ResponseError as e:\n        print(\"ERROR deleting volume '%s': %s\" % (vol_id, e))\n\n\ndef _create_snapshot(ec2_conn, volume_id, description=None):\n    \"\"\"\n    Create a snapshot of the EBS volume with the provided volume_id.\n    Wait until the snapshot process is complete (note that this may take quite a while)\n    \"\"\"\n    snapshot = ec2_conn.create_snapshot(volume_id, description=description)\n    if snapshot:\n        while snapshot.status != 'completed':\n            sleep(6)\n            snapshot.update()\n        print(\"Creation of snapshot for volume '%s' completed: '%s'\" % (volume_id, snapshot))\n        return snapshot.id\n    else:\n        print(\"Could not create snapshot from volume with ID '%s'\" % volume_id)\n        return False\n"
  },
  {
    "path": "cloudbio/distribution.py",
    "content": "\"\"\"Configuration details for specific server types.\n\nThis module contains functions that help with initializing a Fabric environment\nfor standard server types.\n\"\"\"\nimport os\nimport subprocess\n\nfrom fabric.api import env\n\nfrom cloudbio.fabutils import quiet\nfrom cloudbio.fabutils import configure_runsudo\nfrom cloudbio.custom import system\n\ndef _setup_distribution_environment(ignore_distcheck=False):\n    \"\"\"Setup distribution environment.\n\n    In low-level terms, this method attempts to populate various values in the fabric\n    env data structure for use other places in CloudBioLinux.\n    \"\"\"\n    if \"distribution\" not in env:\n        env.distribution = \"__auto__\"\n    if \"dist_name\" not in env:\n        env.dist_name = \"__auto__\"\n    env.logger.info(\"Distribution %s\" % env.distribution)\n\n    if env.hosts == [\"vagrant\"]:\n        _setup_vagrant_environment()\n    elif env.hosts == [\"localhost\"]:\n        _setup_local_environment()\n    configure_runsudo(env)\n    if env.distribution == \"__auto__\":\n        env.distribution = _determine_distribution(env)\n    if env.distribution == \"ubuntu\":\n        ## TODO: Determine if dist_name check works with debian.\n        if env.dist_name == \"__auto__\":\n            env.dist_name = _ubuntu_dist_name(env)\n        _setup_ubuntu()\n    elif env.distribution == \"centos\":\n        _setup_centos()\n    elif env.distribution == \"scientificlinux\":\n        _setup_scientificlinux()\n    elif env.distribution == \"debian\":\n        if env.dist_name == \"__auto__\":\n            env.dist_name = _debian_dist_name(env)\n        _setup_debian()\n    elif env.distribution == \"arch\":\n        pass  # No package support for Arch yet\n    elif env.distribution == \"suse\":\n        pass  # No package support for SUSE yet\n    elif env.distribution == \"macosx\":\n        _setup_macosx(env)\n        ignore_distcheck = True\n    else:\n        raise ValueError(\"Unexpected distribution %s\" % env.distribution)\n    if not ignore_distcheck:\n        _validate_target_distribution(env.distribution, env.get('dist_name', None))\n    _cloudman_compatibility(env)\n    _setup_nixpkgs()\n    _setup_fullpaths(env)\n    # allow us to check for packages only available on 64bit machines\n    machine = env.safe_run_output(\"uname -m\")\n    env.is_64bit = machine.find(\"_64\") > 0\n\n\ndef _setup_fullpaths(env):\n    home_dir = env.safe_run_output(\"echo $HOME\")\n    for attr in [\"data_files\", \"galaxy_home\", \"local_install\"]:\n        if hasattr(env, attr):\n            x = getattr(env, attr)\n            if x.startswith(\"~\"):\n                x = x.replace(\"~\", home_dir)\n                setattr(env, attr, x)\n\n\ndef _cloudman_compatibility(env):\n    \"\"\"Environmental variable naming for compatibility with CloudMan.\n    \"\"\"\n    env.install_dir = env.system_install\n\n\ndef _validate_target_distribution(dist, dist_name=None):\n    \"\"\"Check target matches environment setting (for sanity)\n\n    Throws exception on error\n    \"\"\"\n    env.logger.debug(\"Checking target distribution \" + env.distribution)\n    if dist in [\"debian\", \"ubuntu\"]:\n        tag = env.safe_run_output(\"cat /proc/version\")\n        if tag.lower().find(dist) == -1:\n           # hmmm, test issue file\n            tag2 = env.safe_run_output(\"cat /etc/issue\")\n            if tag2.lower().find(dist) == -1:\n                raise ValueError(\"Distribution does not match machine; are you using correct fabconfig for \" + dist)\n        if not dist_name:\n            raise ValueError(\"Must specify a dist_name property when working with distribution %s\" % dist)\n        # Does this new method work with CentOS, do we need this.\n        if dist == 'debian':\n            actual_dist_name = _debian_dist_name(env)\n        else:\n            actual_dist_name = _ubuntu_dist_name(env)\n        if actual_dist_name != dist_name:\n            raise ValueError(\"Distribution does not match machine; are you using correct fabconfig <\"+actual_dist_name+\"> for \" + dist)\n    else:\n        env.logger.debug(\"Unknown target distro\")\n\n\ndef _setup_ubuntu():\n    env.logger.info(\"Ubuntu setup\")\n    shared_sources = _setup_deb_general()\n    # package information. This is ubuntu/debian based and could be generalized.\n    sources = [\n      \"deb http://us.archive.ubuntu.com/ubuntu/ %s universe\",  # unsupported repos\n      \"deb http://us.archive.ubuntu.com/ubuntu/ %s multiverse\",\n      \"deb http://us.archive.ubuntu.com/ubuntu/ %s-updates universe\",\n      \"deb http://us.archive.ubuntu.com/ubuntu/ %s-updates multiverse\",\n      \"deb http://archive.canonical.com/ubuntu %s partner\",  # partner repositories\n      \"deb http://cran.fhcrc.org/bin/linux/ubuntu %s/\",  # lastest R versions\n      \"deb http://archive.canonical.com/ubuntu %s partner\",  # sun-java\n      \"deb http://ppa.launchpad.net/nebc/bio-linux/ubuntu trusty main\",  # Bio-Linux\n      \"deb [arch=amd64 trusted=yes] http://research.cs.wisc.edu/htcondor/debian/stable/ squeeze contrib\",  # HTCondor\n    ] + shared_sources\n    env.std_sources = _add_source_versions(env.dist_name, sources)\n\n\ndef _setup_debian():\n    env.logger.info(\"Debian setup\")\n    unstable_remap = {\"sid\": \"squeeze\"}\n    shared_sources = _setup_deb_general()\n    sources = [\n        \"deb http://cran.fhcrc.org/bin/linux/debian %s-cran/\",  # lastest R versions\n        \"deb http://nebc.nerc.ac.uk/bio-linux/ unstable bio-linux\",  # Bio-Linux\n        ] + shared_sources\n    # fill in %s\n    dist_name = unstable_remap.get(env.dist_name, env.dist_name)\n    env.std_sources = _add_source_versions(dist_name, sources)\n\n\ndef _setup_deb_general():\n    \"\"\"Shared settings for different debian based/derived distributions.\n    \"\"\"\n    env.logger.debug(\"Debian-shared setup\")\n    env.sources_file = \"/etc/apt/sources.list.d/cloudbiolinux.list\"\n    env.global_sources_file = \"/etc/apt/sources.list\"\n    env.apt_preferences_file = \"/etc/apt/preferences\"\n    if not hasattr(env, \"python_version_ext\"):\n        env.python_version_ext = \"\"\n    if not hasattr(env, \"ruby_version_ext\"):\n        env.ruby_version_ext = \"1.9.1\"\n    if not env.has_key(\"java_home\"):\n        # Try to determine java location from update-alternatives\n        java_home = \"/usr/lib/jvm/java-7-openjdk-amd64\"\n        with quiet():\n            java_info = env.safe_run_output(\"update-alternatives --display java\")\n        for line in java_info.split(\"\\n\"):\n            if line.strip().startswith(\"link currently points to\"):\n                java_home = line.split()[-1].strip()\n                java_home = java_home.replace(\"/jre/bin/java\", \"\")\n        env.java_home = java_home\n    shared_sources = [\n        \"deb http://download.virtualbox.org/virtualbox/debian %s contrib\",  # virtualbox\n    ]\n    return shared_sources\n\n\ndef _setup_centos():\n    env.logger.info(\"CentOS setup\")\n    if not hasattr(env, \"python_version_ext\"):\n        # use installed anaconda version instead of package 2.6\n        #env.python_version_ext = \"2.6\"\n        env.python_version_ext = \"\"\n    #env.pip_cmd = \"pip-python\"\n    if not hasattr(env, \"ruby_version_ext\"):\n        env.ruby_version_ext = \"\"\n    if not env.has_key(\"java_home\"):\n        env.java_home = \"/etc/alternatives/java_sdk\"\n\n\ndef _setup_scientificlinux():\n    env.logger.info(\"ScientificLinux setup\")\n    if not hasattr(env, \"python_version_ext\"):\n        env.python_version_ext = \"\"\n    env.pip_cmd = \"pip-python\"\n    if not env.has_key(\"java_home\"):\n        env.java_home = \"/etc/alternatives/java_sdk\"\n\ndef _setup_macosx(env):\n    # XXX Only framework in place; needs testing\n    env.logger.info(\"MacOSX setup\")\n    # XXX Ensure XCode is installed and provide useful directions if not\n    system.install_homebrew(env)\n    # XXX find java correctly\n    env.java_home = \"\"\n\ndef _setup_nixpkgs():\n    # for now, Nix packages are only supported in Debian - it can\n    # easily be done for others - just get Nix installed from the .rpm\n    nixpkgs = False\n    if env.has_key(\"nixpkgs\"):\n        if env.distribution in [\"debian\", \"ubuntu\"]:\n            if env.nixpkgs == \"True\":\n                nixpkgs = True\n            else:\n                nixpkgs = False\n        else:\n            env.logger.warn(\"NixPkgs are currently not supported for \" + env.distribution)\n    if nixpkgs:\n        env.logger.info(\"NixPkgs: supported\")\n    else:\n        env.logger.debug(\"NixPkgs: Ignored\")\n    env.nixpkgs = nixpkgs\n\n\ndef _setup_local_environment():\n    \"\"\"Setup a localhost environment based on system variables.\n    \"\"\"\n    env.logger.info(\"Get local environment\")\n    if not env.has_key(\"user\"):\n        env.user = os.environ[\"USER\"]\n\n\ndef _setup_vagrant_environment():\n    \"\"\"Use vagrant commands to get connection information.\n    https://gist.github.com/1d4f7c3e98efdf860b7e\n    \"\"\"\n    env.logger.info(\"Get vagrant environment\")\n    raw_ssh_config = subprocess.Popen([\"vagrant\", \"ssh-config\"],\n                                      stdout=subprocess.PIPE).communicate()[0]\n    env.logger.info(raw_ssh_config)\n    ssh_config = dict([l.strip().split() for l in raw_ssh_config.split(\"\\n\") if l])\n    env.user = ssh_config[\"User\"]\n    env.hosts = [ssh_config[\"HostName\"]]\n    env.port = ssh_config[\"Port\"]\n    env.host_string = \"%s@%s:%s\" % (env.user, env.hosts[0], env.port)\n    env.key_filename = ssh_config[\"IdentityFile\"].replace('\"', '')\n    env.logger.debug(\"ssh %s\" % env.host_string)\n\n\ndef _add_source_versions(version, sources):\n    \"\"\"Patch package source strings for version, e.g. Debian 'stable'\n    \"\"\"\n    name = version\n    env.logger.debug(\"Source=%s\" % name)\n    final = []\n    for s in sources:\n        if s.find(\"%s\") > 0:\n            s = s % name\n        final.append(s)\n    return final\n\n\ndef _ubuntu_dist_name(env):\n    \"\"\"\n    Determine Ubuntu dist name (e.g. precise or quantal).\n    \"\"\"\n    return env.safe_run_output(\"cat /etc/*release | grep DISTRIB_CODENAME | cut -f 2 -d =\")\n\n\ndef _debian_dist_name(env):\n    \"\"\"\n    Determine Debian dist name (e.g. squeeze).\n    \"\"\"\n    return env.safe_run_output(\"lsb_release -a | grep Codename | cut -f 2\")\n\n\ndef _determine_distribution(env):\n    \"\"\"\n    Attempt to automatically determine the distribution of the target machine.\n\n    Currently works for Ubuntu, CentOS, Debian, Scientific Linux and Mac OS X.\n    \"\"\"\n    with quiet():\n        output = env.safe_run_output(\"cat /etc/*release\").lower()\n    if output.find(\"id=ubuntu\") >= 0:\n        return \"ubuntu\"\n    elif output.find(\"centos release\") >= 0:\n        return \"centos\"\n    elif output.find(\"centos linux release\") >= 0:\n        return \"centos\"\n    elif output.find(\"red hat enterprise linux\") >= 0:\n        return \"centos\"\n    elif output.find(\"fedora\") >= 0:\n        return \"centos\"\n    # Amazon AMIs are Red-Hat based\n    elif output.find(\"amzn\") >= 0 or output.find(\"amazon\") >= 0:\n        return \"centos\"\n    elif output.find(\"suse linux\") >= 0:\n        return \"suse\"\n    elif output.find(\"opensuse\") >= 0:\n        return \"suse\"\n    elif output.find(\"scientific linux\") >= 0:\n        return \"scientificlinux\"\n    elif env.safe_exists(\"/etc/debian_version\"):\n        return \"debian\"\n    elif output.find(\"id=arch\") >= 0 or output.find('id_like=\"arch\"') >=0:\n        return \"arch\"\n    elif output.find(\"antergos\") >= 0:\n        return \"arch\"\n    # check for file used by Python's platform.mac_ver\n    elif env.safe_exists(\"/System/Library/CoreServices/SystemVersion.plist\"):\n        return \"macosx\"\n    else:\n        raise Exception(\"Attempt to automatically determine Linux distribution of target machine failed:\\n%s\" % output)\n"
  },
  {
    "path": "cloudbio/fabutils.py",
    "content": "\"\"\"Utilities to generalize usage of fabric for local and remote builds.\n\nHandles:\n  - Providing a local equivalent of standard functions that avoid\n    the need to ssh to a local machine.\n    Adds generalized targets to the global `env` object which cleanly\n    handle local and remote execution:\n    - safe_run: Run a command\n    - safe_run_output: Run a command, capturing the output\n    - safe_sudo: Run a command as sudo user\n    - safe_exists: Check for existence of a file.\n    - safe_sed: Run sed command.\n\"\"\"\nimport hashlib\nimport os\nimport re\nimport shutil\n\nfrom fabric.api import env, run, sudo, local, settings, hide, put\nfrom fabric.contrib.files import exists, sed, contains, append, comment\n\nimport six\n\n\nSUDO_ENV_KEEPS = []  # Environment variables passed through to sudo environment when using local sudo.\nSUDO_ENV_KEEPS += [\"http_proxy\", \"https_proxy\"]  # Required for local sudo to work behind a proxy.\n\n\n# ## Local non-ssh access\ndef local_exists(path, use_sudo=False):\n    func = env.safe_sudo if use_sudo else env.safe_run\n    cmd = 'test -e \"$(echo %s)\"' % path\n    cmd_symbolic = 'test -h \"$(echo %s)\"' % path\n    with settings(hide('everything'), warn_only=True):\n        env.lcwd = env.cwd\n        # We do not use cmd_symbolic so we avoid rescuing broken symlinks\n        return not func(cmd).failed\n\ndef run_local(use_sudo=False, capture=False):\n    def _run(command, *args, **kwags):\n        if use_sudo:\n            sudo_env = \" \".join([\"%s=$%s\" % (keep, keep) for keep in SUDO_ENV_KEEPS])\n            sudo_to = \"\"\n            if \"user\" in kwags:\n                sudo_to = \"su - %s\" % kwags[\"user\"]\n            sudo_prefix = \"sudo %s %s bash -c \" % (sudo_env, sudo_to)\n\n            command = sudo_prefix + '\"%s\"' % command.replace('\"', '\\\\\"')\n        env.lcwd = env.cwd\n        return local(command, capture=capture)\n    return _run\n\ndef local_put(orig_file, new_file):\n    shutil.copyfile(orig_file, new_file)\n\ndef local_sed(filename, before, after, limit='', use_sudo=False, backup='.bak',\n              flags='', shell=False):\n    \"\"\" Run a search-and-replace on ``filename`` with given regex patterns.\n\n    From main fabric contrib, modified to handle local.\n    \"\"\"\n    func = env.safe_sudo if use_sudo else env.safe_run\n    # Characters to be escaped in both\n    for char in \"/'\":\n        before = before.replace(char, r'\\%s' % char)\n        after = after.replace(char, r'\\%s' % char)\n    # Characters to be escaped in replacement only (they're useful in regexen\n    # in the 'before' part)\n    for char in \"()\":\n        after = after.replace(char, r'\\%s' % char)\n    if limit:\n        limit = r'/%s/ ' % limit\n    context = {\n        'script': r\"'%ss/%s/%s/%sg'\" % (limit, before, after, flags),\n        'filename': '\"$(echo %s)\"' % filename,\n        'backup': backup\n    }\n    # Test the OS because of differences between sed versions\n\n    with hide('running', 'stdout'):\n        platform = env.safe_run(\"uname\")\n    if platform in ('NetBSD', 'OpenBSD', 'QNX'):\n        # Attempt to protect against failures/collisions\n        hasher = hashlib.sha1()\n        hasher.update(env.host_string)\n        hasher.update(filename)\n        context['tmp'] = \"/tmp/%s\" % hasher.hexdigest()\n        # Use temp file to work around lack of -i\n        expr = r\"\"\"cp -p %(filename)s %(tmp)s \\\n&& sed -r -e %(script)s %(filename)s > %(tmp)s \\\n&& cp -p %(filename)s %(filename)s%(backup)s \\\n&& mv %(tmp)s %(filename)s\"\"\"\n    else:\n        context['extended_regex'] = '-E' if platform == 'Darwin' else '-r'\n        expr = r\"sed -i%(backup)s %(extended_regex)s -e %(script)s %(filename)s\"\n    command = expr % context\n    return func(command, shell=shell)\n\ndef local_comment(filename, regex, use_sudo=False, char='#', backup='.bak', shell=False):\n    carot, dollar = '', ''\n    if regex.startswith('^'):\n        carot = '^'\n        regex = regex[1:]\n    if regex.endswith('$'):\n        dollar = '$'\n        regex = regex[:-1]\n    regex = \"%s(%s)%s\" % (carot, regex, dollar)\n    return local_sed(\n        filename,\n        before=regex,\n        after=r'%s\\1' % char,\n        use_sudo=use_sudo,\n        backup=backup,\n        shell=shell\n    )\n\ndef _escape_for_regex(text):\n    \"\"\"Escape ``text`` to allow literal matching using egrep\"\"\"\n    regex = re.escape(text)\n    # Seems like double escaping is needed for \\\n    regex = regex.replace('\\\\\\\\', '\\\\\\\\\\\\')\n    # Triple-escaping seems to be required for $ signs\n    regex = regex.replace(r'\\$', r'\\\\\\$')\n    # Whereas single quotes should not be escaped\n    regex = regex.replace(r\"\\'\", \"'\")\n    return regex\n\ndef _expand_path(path):\n    return '\"$(echo %s)\"' % path\n\ndef local_contains(filename, text, exact=False, use_sudo=False, escape=True,\n    shell=False):\n    func = use_sudo and env.safe_sudo or env.safe_run\n    if escape:\n        text = _escape_for_regex(text)\n        if exact:\n            text = \"^%s$\" % text\n    with settings(hide('everything'), warn_only=True):\n        egrep_cmd = 'egrep \"%s\" %s' % (text, _expand_path(filename))\n        return func(egrep_cmd, shell=shell).succeeded\n\ndef local_append(filename, text, use_sudo=False, partial=False, escape=True, shell=False):\n    func = use_sudo and env.safe_sudo or env.safe_run\n    # Normalize non-list input to be a list\n    if isinstance(text, six.string_types):\n        text = [text]\n    for line in text:\n        regex = '^' + _escape_for_regex(line)  + ('' if partial else '$')\n        if (env.safe_exists(filename, use_sudo=use_sudo) and line\n            and env.safe_contains(filename, regex, use_sudo=use_sudo, escape=False,\n                                  shell=shell)):\n            continue\n        line = line.replace(\"'\", r\"'\\\\''\") if escape else line\n        func(\"echo '%s' >> %s\" % (line, _expand_path(filename)))\n\ndef run_output(*args, **kwargs):\n    if not 'shell' in kwargs:\n        kwargs['shell'] = False\n    return run(*args, **kwargs)\n\ndef configure_runsudo(env):\n    \"\"\"Setup env variable with safe_sudo and safe_run,\n    supporting non-privileged users and local execution.\n    \"\"\"\n    env.is_local = env.hosts == [\"localhost\"]\n    if env.is_local:\n        env.safe_put = local_put\n        env.safe_sed = local_sed\n        env.safe_comment = local_comment\n        env.safe_contains = local_contains\n        env.safe_append = local_append\n        env.safe_exists = local_exists\n        env.safe_run = run_local()\n        env.safe_run_output = run_local(capture=True)\n    else:\n        env.safe_put = put\n        env.safe_sed = sed\n        env.safe_comment = comment\n        env.safe_contains = contains\n        env.safe_append = append\n        env.safe_exists = exists\n        env.safe_run = run\n        env.safe_run_output = run_output\n    if isinstance(getattr(env, \"use_sudo\", \"true\"), six.string_types):\n        if getattr(env, \"use_sudo\", \"true\").lower() in [\"true\", \"yes\"]:\n            env.use_sudo = True\n            if env.is_local:\n                env.safe_sudo = run_local(True)\n            else:\n                env.safe_sudo = sudo\n        else:\n            env.use_sudo = False\n            if env.is_local:\n                env.safe_sudo = run_local()\n            else:\n                env.safe_sudo = run\n\ndef find_cmd(env, cmd, args):\n    \"\"\"Retrieve location of a command, checking in installation directory.\n    \"\"\"\n    local_cmd = os.path.join(env.system_install, \"bin\", cmd)\n    for cmd in [local_cmd, cmd]:\n        with quiet():\n            test_version = env.safe_run(\"%s %s\" % (cmd, args))\n        if test_version.succeeded:\n            return cmd\n    return None\n\ntry:\n    from fabric.api import quiet\nexcept ImportError:\n    def quiet():\n        return settings(hide('warnings', 'running', 'stdout', 'stderr'), warn_only=True)\n\ntry:\n    from fabric.api import warn_only\nexcept ImportError:\n    def warn_only():\n        return settings(warn_only=True)\n"
  },
  {
    "path": "cloudbio/flavor/__init__.py",
    "content": "\"\"\"A Flavor reflects a specialization of a base install, the default being BioLinux.\n\nIf you want to create a new specialization (say for your own server), the\nrecommended procedure is to choose an existing base install (Edition) and write\na Flavor. When your Flavor is of interest to other users, it may be a good idea\nto commit it to the main project (in ./contrib/flavor).\n\nOther (main) flavors can be found in this directory and in ./contrib/flavors\n\"\"\"\n\nclass Flavor:\n    \"\"\"Base class. Every flavor derives from this\n    \"\"\"\n    def __init__(self, env):\n        self.name = \"Base Flavor - no overrides\"  # should override this\n        self.short_name = \"\"\n        self.env = env\n        self.check_distribution()\n\n    def rewrite_config_items(self, name, items):\n        \"\"\"Generic hook to rewrite a list of configured items.\n\n        Can define custom dispatches based on name: packages, custom,\n        python, ruby, perl\n        \"\"\"\n        return items\n\n    def check_distribution(self):\n        \"\"\"Ensure the distribution matches an expected type for this edition.\n\n        Base supports multiple distributions.\n        \"\"\"\n        pass\n\n    def check_packages_source(self):\n        \"\"\"Override for check package definition file before updating\n        \"\"\"\n        pass\n\n    def rewrite_apt_sources_list(self, sources):\n        \"\"\"Allows editions to modify the sources list\n        \"\"\"\n        return sources\n\n    def rewrite_apt_preferences(self, preferences):\n        \"\"\"Allows editions to modify the apt preferences policy file\n        \"\"\"\n        return preferences\n\n    def rewrite_apt_automation(self, package_info):\n        \"\"\"Allows editions to modify the apt automation list\n        \"\"\"\n        return package_info\n\n    def rewrite_apt_keys(self, standalone, keyserver):\n        \"\"\"Allows editions to modify key list\"\"\"\n        return standalone, keyserver\n\n    def apt_upgrade_system(self, env=None):\n        \"\"\"Upgrade system through apt - so this behaviour can be overridden\n        \"\"\"\n        env.safe_sudo(\"apt-get -y --force-yes upgrade\")\n\n    def post_install(self):\n        \"\"\"Add scripts for starting FreeNX and CloudMan.\n        \"\"\"\n        pass\n\nclass Minimal(Flavor):\n\n    def __init__(self, env):\n        Flavor.__init__(self, env)\n        self.name = \"Minimal Flavor\"\n        self.short_name = \"minimal\"\n\n    def rewrite_config_items(self, name, items):\n        \"\"\"Generic hook to rewrite a list of configured items.\n\n        Can define custom dispatches based on name: packages, custom,\n        python, ruby, perl\n        \"\"\"\n        return items\n\n    def post_install(self, pkg_install=None):\n        \"\"\"Add scripts for starting FreeNX and CloudMan.\n        \"\"\"\n        pass\n"
  },
  {
    "path": "cloudbio/flavor/config.py",
    "content": "\"\"\"\nHandle alternative configuration file locations for flavor customizations.\n\"\"\"\nimport os\nimport collections\n\ndef _find_fname(env, fname):\n    for dirname in [env.get(\"flavor_dir\", None), env.config_dir]:\n        if dirname:\n            full_path = os.path.join(dirname, fname)\n            if os.path.exists(full_path):\n                return full_path\n    return None\n\ndef get_config_file(env, fname):\n    \"\"\"\n    Retrieve YAML configuration file from the default config directory or flavor directory.\n\n    This combines all options for getting distribution or flavor specific customizations.\n    \"\"\"\n    base, ext = os.path.splitext(fname)\n    distribution_fname = \"{0}-{1}{2}\".format(base, env.get(\"distribution\", \"notspecified\"), ext)\n    Config = collections.namedtuple(\"Config\", \"base dist\")\n    out = Config(base=_find_fname(env, fname), dist=_find_fname(env, distribution_fname))\n    env.logger.debug(\"Using config file {0}\".format(out.base))\n    return out\n"
  },
  {
    "path": "cloudbio/galaxy/__init__.py",
    "content": "\"\"\"\nAdapted from Enis Afgan's mi-deployment code:\nhttps://bitbucket.org/afgane/mi-deployment\n\"\"\"\nfrom __future__ import print_function\nimport os\nimport contextlib\n\nfrom fabric.api import sudo, cd, settings, hide\nfrom fabric.colors import red\n\nfrom cloudbio.custom.shared import (_write_to_file, _setup_conf_file,\n                                    _setup_simple_service, _make_tmp_dir,\n                                    _add_to_profiles)\nfrom cloudbio.galaxy.tools import _install_tools\nfrom cloudbio.galaxy.utils import _chown_galaxy, _read_boolean, _dir_is_empty\n\n\n# -- Adjust this link if using content from another location\nCDN_ROOT_URL = \"http://userwww.service.emory.edu/~eafgan/content\"\nREPO_ROOT_URL = \"https://bitbucket.org/afgane/mi-deployment/raw/tip\"\nCM_REPO_ROOT_URL = \"https://bitbucket.org/galaxy/cloudman/raw/tip/\"\n\n\ndef _setup_users(env):\n    def _add_user(username, uid=None):\n        \"\"\" Add user with username to the system \"\"\"\n        if not env.safe_contains('/etc/passwd', \"%s:\" % username):\n            uid_str = \"--uid %s\" % uid if uid else \"\"\n            env.safe_sudo('useradd -d /home/%s --create-home --shell /bin/bash '\n                          '-c\"Galaxy-required user\" %s --user-group %s' %\n                          (username, uid_str, username))\n    _add_user('galaxy')\n    _add_user('sgeadmin')\n    _add_user('postgres')\n    env.logger.debug(\"Done setting up Galaxy/CloudMan users\")\n\n\ndef _setup_galaxy_env_defaults(env):\n    if \"galaxy_user\" not in env:\n        env.galaxy_user = \"galaxy\"\n    if \"galaxy_home\" not in env:\n        env.galaxy_home = \"/mnt/galaxy/galaxy-app\"\n    if \"galaxy_tools_dir\" not in env:\n        # Was called install_dir in tools_fabfile.py\n        env.galaxy_tools_dir = \"/mnt/galaxy/tools\"\n    if \"galaxy_loc_files\" not in env:\n        indicies_dir = env.get(\"data_files\", \"/mnt/galaxyIndices\")\n        env.galaxy_loc_files = os.path.join(indicies_dir, \"galaxy\", \"galaxy-data\")\n    if \"galaxy_jars_dir\" not in env:\n        env.galaxy_jars_dir = os.path.join(env.galaxy_home, \"tool-data\", \"shared\", \"jars\")\n    if \"galaxy_update_default\" not in env:\n        env.galaxy_update_default = True\n    if \"python_version\" not in env:\n        env.python_version = \"2.7\"  # Override in fabricrc if this is not the case.\n    if \"galaxy_indices_mount\" not in env:\n        indicies_dir = env.get(\"data_files\", \"/mnt/galaxyIndices\")\n        env.galaxy_indices_mount = indicies_dir\n    if \"galaxy_data_mount\" not in env:\n        env.galaxy_data_mount = \"/mnt/galaxyData\"\n    if \"galaxy_init_database\" not in env:\n        env.galaxy_init_database = False\n    if \"galaxy_len_files\" not in env:\n        env.galaxy_len_files = \"/mnt/galaxy/configuration_data/len\"\n\n\ndef _install_galaxy(env):\n    \"\"\"\n    Used to install Galaxy and setup its environment, including tools.\n    This method is somewhat targeted for the cloud deployment so some\n    tweaking of the code may be desirable. This method cannot be used\n    to update an existing Galaxy.\n    \"\"\"\n    _clone_galaxy_repo(env)\n    _chown_galaxy(env, env.galaxy_home)  # Make sure env.galaxy_user owns env.galaxy_home\n    env.safe_sudo(\"chmod 755 %s\" % os.path.split(env.galaxy_home)[0])\n    setup_db = _read_boolean(env, \"galaxy_setup_database\", False)\n    if setup_db:\n        _setup_galaxy_db(env)\n    setup_service = _read_boolean(env, \"galaxy_setup_service\", False)\n    if setup_service:\n        _setup_service(env)\n    _setup_trackster(env)\n    _setup_shed_tools_dir(env)\n    _install_tools(env)\n    setup_xvfb = _read_boolean(env, \"galaxy_setup_xvfb\", False)\n    if setup_xvfb:\n        _setup_xvfb(env)\n    _chown_galaxy(env, env.galaxy_home)  # Make sure env.galaxy_user owns env.galaxy_home\n    _chown_galaxy(env, env.galaxy_loc_files)  # Make sure env.galaxy_user owns env.galaxy_loc_files\n    return True\n\n\ndef _clone_galaxy_repo(env):\n    \"\"\"\n    Clone Galaxy source code repository from ``env.galaxy_repository`` to\n    ``env.galaxy_home``, setting the directory ownership to ``env.galaxy_user``\n\n    This method cannot be used to update an existing Galaxy installation.\n    \"\"\"\n    # Make sure ``env.galaxy_home`` dir exists but without Galaxy in it\n    galaxy_exists = False\n    if env.safe_exists(env.galaxy_home):\n        if env.safe_exists(os.path.join(env.galaxy_home, '.hg')):\n            env.logger.warning(\"Galaxy install dir {0} exists and seems to have \"\n                \"a Mercurial repository already there. Galaxy already installed?\"\n                .format(env.galaxy_home))\n            galaxy_exists = True\n    else:\n        env.safe_sudo(\"mkdir -p '%s'\" % env.galaxy_home)\n    if not galaxy_exists:\n        with cd(env.galaxy_home):\n            # Needs to be done as non galaxy user, otherwise we have a\n            # permissions problem.\n            galaxy_repository = env.get(\"galaxy_repository\", 'https://bitbucket.org/galaxy/galaxy-central/')\n            env.safe_sudo('hg clone %s .' % galaxy_repository)\n    # Make sure ``env.galaxy_home`` is owned by ``env.galaxy_user``\n    _chown_galaxy(env, env.galaxy_home)\n    # Make sure env.galaxy_home root dir is also owned by env.galaxy_user so Galaxy\n    # process can create necessary dirs (e.g., shed_tools, tmp)\n    env.safe_sudo(\"chown {0}:{0} {1}\".format(env.galaxy_user, os.path.split(env.galaxy_home)[0]))\n    # If needed, custom-configure this freshly cloned Galaxy\n    preconfigured = _read_boolean(env, \"galaxy_preconfigured_repository\", False)\n    if not preconfigured:\n        _configure_galaxy_repository(env)\n\n\ndef _setup_galaxy_db(env):\n    \"\"\"\n    Create (if one already does not exist) and initialize a database for use\n    by Galaxy.\n    \"\"\"\n    if not _galaxy_db_exists(env):\n        _create_galaxy_db(env)\n    _init_galaxy_db(env)\n\n\ndef _get_galaxy_db_configs(env):\n    \"\"\"\n    Extract configuration options for Galaxy database and return those as a dictionary\n    \"\"\"\n    config = {}\n    config['psql_data_dir'] = env.get('galaxy_database_path', '/mnt/galaxy/db')\n    config['psql_conf_file'] = os.path.join(config['psql_data_dir'], 'postgresql.conf')\n    config['psql_bin_dir'] = env.get('postgres_bin_dir', '/usr/lib/postgresql/9.1/bin')\n    config['psql_user'] = env.get('postrges_user', 'postgres')\n    config['psql_port'] = env.get('postgres_port', '5910')\n    config['psql_log'] = '/tmp/pSQL.log'\n    config['galaxy_db_name'] = env.get('galaxy_db_name', 'galaxy')\n    config['galaxy_ftp_user_pwd'] = env.get('galaxy_ftp_user_password', 'fu5yOj2sn')\n    # And a couple of useful command shortcuts\n    config['pg_ctl_cmd'] = \"{0} -D {2}\".format(os.path.join(config['psql_bin_dir'], 'pg_ctl'),\n        config['psql_port'], config['psql_data_dir'])\n    config['pg_start_cmd'] = \"{0} -w -l {1} start\".format(config['pg_ctl_cmd'], config['psql_log'])\n    config['pg_stop_cmd'] = \"{0} -w stop\".format(config['pg_ctl_cmd'])\n    config['psql_cmd'] = \"{0} -p {1}\".format(os.path.join(config['psql_bin_dir'], 'psql'),\n        config['psql_port'])\n    return config\n\n\ndef _galaxy_db_exists(env):\n    \"\"\"\n    Check if galaxy database already exists. Return ``True`` if it does,\n    ``False`` otherwise.\n\n    Note that this method does a best-effort attempt at starting the DB server\n    if one is not already running to do a thorough test. It shuts the server\n    down upon completion, but only it if also started it.\n    \"\"\"\n    db_exists = False\n    started = False\n    c = _get_galaxy_db_configs(env)\n    if env.safe_exists(c['psql_data_dir']) and not _dir_is_empty(c['psql_data_dir']):\n        env.safe_sudo(\"chown --recursive {0}:{0} {1}\".format(c['psql_user'], c['psql_data_dir']))\n        env.logger.debug(\"Galaxy database directory {0} already exists.\".format(c['psql_data_dir']))\n        # Check if PostgreSQL is already running and try to start the DB if not\n        if not _postgres_running(env):\n            with settings(warn_only=True):\n                env.logger.debug(\"Trying to start DB server in {0}\".format(c['psql_data_dir']))\n                env.safe_sudo(\"{0}\".format(c['pg_start_cmd']), user=c['psql_user'])\n                started = True\n        # Check if galaxy DB already exists\n        if 'galaxy' in sudo(\"{0} -P pager --list | grep {1} || true\".format(c['psql_cmd'],\n           c['galaxy_db_name']), user=c['psql_user']):\n            env.logger.warning(\"Galaxy database {0} already exists in {1}! Not creating it.\"\n                .format(c['galaxy_db_name'], c['psql_data_dir']))\n            db_exists = True\n        if started:\n            with settings(warn_only=True):\n                env.safe_sudo(\"{0}\".format(c['pg_stop_cmd']), user=c['psql_user'])\n    return db_exists\n\n\ndef _create_galaxy_db(env):\n    \"\"\"\n    Create a new PostgreSQL database for use by Galaxy\n    \"\"\"\n    c = _get_galaxy_db_configs(env)\n    if not env.safe_exists(c['psql_data_dir']):\n        env.safe_sudo(\"mkdir -p {0}\".format(c['psql_data_dir']))\n    env.safe_sudo(\"chown --recursive {0}:{0} {1}\".format(c['psql_user'], c['psql_data_dir']))\n    # Initialize a new database for Galaxy in ``psql_data_dir``\n    if _dir_is_empty(c['psql_data_dir']):\n        env.safe_sudo(\"{0} -D {1}\".format(os.path.join(c['psql_bin_dir'], 'initdb'), c['psql_data_dir']),\n            user=c['psql_user'])\n    # Set port for the database server\n    env.safe_sed(c['psql_conf_file'], '#port = 5432', 'port = {0}'.format(c['psql_port']), use_sudo=True)\n    env.safe_sudo(\"chown {0}:{0} {1}\".format(c['psql_user'], c['psql_conf_file']))\n    # Start PostgreSQL server so a role for Galaxy user can be created\n    if not _postgres_running(env):\n        env.safe_sudo(c['pg_start_cmd'], user=c['psql_user'])\n        started = True\n    else:\n        # Restart is required so port setting takes effect\n        env.safe_sudo(\"{0} -D {1} -w -l {2} restart\".format(c['pg_ctl_cmd']), c['psql_data_dir'],\n            c['psql_log'], user=c['psql_user'])\n        started = False\n    # Create a role for env.galaxy_user\n    env.safe_sudo('{0} -c\"CREATE ROLE {1} LOGIN CREATEDB\"'.format(c['psql_cmd'], env.galaxy_user),\n        user=c['psql_user'])\n    # Create a Galaxy database\n    env.safe_sudo('{0} -p {1} {2}'.format(os.path.join(c['psql_bin_dir'], 'createdb'),\n        c['psql_port'], c['galaxy_db_name']), user=env.galaxy_user)\n    # Create a role for 'galaxyftp' user\n    env.safe_sudo('{0} -c\"CREATE ROLE galaxyftp LOGIN PASSWORD \\'{1}\\'\"'.format(c['psql_cmd'],\n        c['galaxy_ftp_user_pwd']), user=c['psql_user'])\n    if started:\n        with settings(warn_only=True):\n            env.safe_sudo(\"{0}\".format(c['pg_stop_cmd']), user=c['psql_user'])\n    exp = \"export PATH={0}:$PATH\".format(c['psql_bin_dir'])\n    _add_to_profiles(exp)\n\n\ndef _init_galaxy_db(env):\n    \"\"\"\n    Initialize Galaxy's database with the tables and apply the migrations,\n    fetching Galaxy eggs in the process.\n    \"\"\"\n    with cd(env.galaxy_home):\n        universe_wsgi_url = env.get('galaxy_universe_wsgi_url',\n            os.path.join(CM_REPO_ROOT_URL, 'universe_wsgi.ini.cloud'))\n        env.safe_sudo(\"wget --output-document=universe_wsgi.ini {0}\".format(universe_wsgi_url))\n        started = False\n        if not _postgres_running(env):\n            c = _get_galaxy_db_configs(env)\n            env.safe_sudo(c['pg_start_cmd'], user=c['psql_user'])\n            started = True\n        sudo(\"bash -c 'export PYTHON_EGG_CACHE=eggs; python -ES ./scripts/fetch_eggs.py; ./create_db.sh'\",\n            user=env.galaxy_user)\n        if started:\n            with settings(warn_only=True):\n                env.safe_sudo(\"{0}\".format(c['pg_stop_cmd']), user=c['psql_user'])\n\n\ndef _configure_galaxy_options(env, option_dict=None, prefix=\"galaxy_universe_\"):\n    \"\"\"\n    Read through fabric's environment and make sure any property of\n    the form galaxy_universe_XXX=YYY lands up in Galaxy's universe_wsgi.ini\n    options as XXX=YYY using Galaxy configuration directory:\n    \"\"\"\n    galaxy_conf_directory = env.get(\"galaxy_conf_directory\", None)\n    if not galaxy_conf_directory:\n        return False\n    # By default just read the options from env (i.e. from fabricrc), but\n    # allow override so the options can come from a YAML file (such as done\n    # with galaxy-vm-launcher.)\n    if option_dict is None:\n        option_dict = env\n\n    option_priority = env.get(\"galaxy_conf_priority\", \"200\")\n    for key, value in option_dict.iteritems():\n        if key.startswith(prefix):\n            key = key[len(prefix):]\n            conf_file_name = \"%s_override_%s.ini\" % (option_priority, key)\n            conf_file = os.path.join(galaxy_conf_directory, conf_file_name)\n            contents = \"[app:main]\\n%s=%s\" % (key, value)\n            _write_to_file(contents, conf_file, \"0700\")\n            _chown_galaxy(env, conf_file)\n\n\ndef _setup_shed_tools_dir(env):\n    ts_dir = \"%s/../shed_tools\" % env.galaxy_home\n    if not env.safe_exists(ts_dir):\n        _make_dir_for_galaxy(env, ts_dir)\n        env.logger.info(\"Setup Tool Shed directory {0}\".format(ts_dir))\n\n\ndef _setup_trackster(env):\n    \"\"\"\n    Download .len files required by Trackster:\n    http://wiki.galaxyproject.org/Learn/Visualization#Setup_for_Local_Instances\n    \"\"\"\n    if not env.safe_exists(env.galaxy_len_files):\n        env.safe_sudo(\"mkdir -p {0}\".format(env.galaxy_len_files))\n    with cd(env.galaxy_len_files):\n        if not env.safe_exists(os.path.join(env.galaxy_len_files, \"hg19.len\")):\n            local_fn = \"len-files.tar.gz\"\n            env.safe_sudo('wget --output-document={0} '\n                 'https://s3.amazonaws.com/usegalaxy/len-files.tar.gz'.format(local_fn))\n            env.safe_sudo(\"tar xzf {0}\".format(local_fn))\n            env.safe_sudo(\"rm {0}\".format(local_fn))\n    _chown_galaxy(env, env.galaxy_len_files)\n\n\ndef _configure_galaxy_repository(env):\n    \"\"\"\n    Custom-configure Galaxy repository. This is primarily targeted at a cloud\n    deployment.\n\n    mi-deployment would always edit the repository in this way, but\n    galaxy-vm-launcher requires the configured Galaxy repository to pull\n    in the changesets from https://bitbucket.org/jmchilton/cloud-galaxy-dist\n    which prebakes these modifications in.\n    \"\"\"\n    _chown_galaxy(env, env.galaxy_home)  # Make sure env.galaxy_user owns env.galaxy_home\n    with cd(env.galaxy_home):  # and settings(warn_only=True):\n        # Make sure Galaxy runs in a new shell and does not\n        # inherit the environment by adding the '-ES' flag\n        # to all invocations of python within run.sh\n        env.safe_sudo(\"sed -i 's/python .\\//python -ES .\\//g' %s/run.sh\" % (env.galaxy_home), user=env.galaxy_user)\n        if _read_boolean(env, \"galaxy_cloud\", True):\n            # Append DRMAA_LIBRARY_PATH in run.sh as well (this file will exist\n            # once SGE is installed - which happens at instance contextualization)\n            env.safe_sudo(\"grep -q 'export DRMAA_LIBRARY_PATH=/opt/sge/lib/lx24-amd64/libdrmaa.so.1.0' %s/run.sh; if [ $? -eq 1 ]; then sed -i '2 a export DRMAA_LIBRARY_PATH=/opt/sge/lib/lx24-amd64/libdrmaa.so.1.0' run.sh; fi\" % (env.galaxy_home), user=env.galaxy_user)\n            # Upload the custom cloud welcome screen files\n            if not env.safe_exists(\"%s/static/images/cloud.gif\" % env.galaxy_home):\n                env.safe_sudo(\"wget --output-document=%s/static/images/cloud.gif %s/cloud.gif\" % (env.galaxy_home, CDN_ROOT_URL), user=env.galaxy_user)\n            if not env.safe_exists(\"%s/static/images/cloud_txt.png\" % env.galaxy_home):\n                env.safe_sudo(\"wget --output-document=%s/static/images/cloud_text.png %s/cloud_text.png\" % (env.galaxy_home, CDN_ROOT_URL), user=env.galaxy_user)\n            env.safe_sudo(\"wget --output-document=%s/static/welcome.html %s/welcome.html\" % (env.galaxy_home, CDN_ROOT_URL), user=env.galaxy_user)\n        # Set up the symlink for SAMTOOLS (remove this code once SAMTOOLS is converted to data tables)\n        if env.safe_exists(\"%s/tool-data/sam_fa_indices.loc\" % env.galaxy_home):\n            env.safe_sudo(\"rm %s/tool-data/sam_fa_indices.loc\" % env.galaxy_home, user=env.galaxy_user)\n        # TODO: Is this really necessary here? Shouldn't the tools do this themselves?\n        # set up the jars directory for Java tools\n        if not env.safe_exists(env.galaxy_jars_dir):\n            env.safe_sudo(\"mkdir -p %s\" % env.galaxy_jars_dir, user=env.galaxy_user)\n        srma_dir = os.path.join(env.galaxy_tools_dir, 'srma', 'default')\n        haploview_dir = os.path.join(env.galaxy_tools_dir, 'haploview', 'default')\n        picard_dir = os.path.join(env.galaxy_tools_dir, 'picard', 'default')\n        env.safe_sudo('ln -s -f %s/srma.jar %s' % (srma_dir, env.galaxy_jars_dir), user=env.galaxy_user)\n        env.safe_sudo('ln -s -f %s/haploview.jar %s' % (haploview_dir, env.galaxy_jars_dir), user=env.galaxy_user)\n        env.safe_sudo('ln -s -f %s/*.jar %s' % (picard_dir, env.galaxy_jars_dir), user=env.galaxy_user)\n    return True\n\n\ndef _setup_service(env):\n    _setup_conf_file(env, \"/etc/init.d/galaxy\", \"galaxy_init\", default_source=\"galaxy_init\")\n    _setup_conf_file(env, \"/etc/default/galaxy\", \"galaxy_default\")\n    _setup_simple_service(\"galaxy\")\n\n\ndef _setup_xvfb(env):\n    _setup_conf_file(env, \"/etc/init.d/xvfb\", \"xvfb_init\", default_source=\"xvfb_init\")\n    _setup_conf_file(env, \"/etc/default/xvfb\", \"xvfb_default\", default_source=\"xvfb_default\")\n    _setup_simple_service(\"xvfb\")\n    env.safe_sudo(\"mkdir /var/lib/xvfb; chown root:root /var/lib/xvfb; chmod 0755 /var/lib/xvfb\")\n    display_export = \"-v DISPLAY=:42\"\n    _add_to_profiles(display_export, profiles=[\"/home/%s/.sge_request\" % env.get(\"galaxy_user\", \"galaxy\")])\n\n\ndef _setup_nginx_service(env):\n    # Setup system service for nginx, not needed for CloudMan but it is\n    # useful if CloudMan is not being used (such as galaxy-vm-launcher work).\n    _setup_conf_file(env, \"/etc/init.d/nginx\", \"nginx_init\", default_source=\"nginx_init\")\n    _setup_simple_service(\"nginx\")\n\n\ndef _install_nginx_standalone(env):\n    _install_nginx(env)\n    _setup_nginx_service(env)\n\n\ndef _install_nginx_package(env):\n    \"\"\"\n    Install nginx from a custom package; also see\n    https://github.com/afgane/gvl_flavor/tree/master/playbook\n    \"\"\"\n    version = \"1.4.7-gvl20140507b01-precise\"\n    package_url = \"http://cloudman-dev.s3.amazonaws.com/gvl-nginx-%s-amd64.deb\" % version\n\n    with _make_tmp_dir() as work_dir:\n        with contextlib.nested(cd(work_dir), settings(hide('stdout'))):\n            env.safe_run(\"wget %s\" % package_url)\n            env.safe_sudo(\"dpkg -i %s\" % os.path.split(package_url)[1])\n\n    # The path for `nginx.conf` depends on how the package was built and\n    # what's set in `nginx_remote_conf_path` must match that!\n    defaults = {\"galaxy_home\": \"/mnt/galaxy/galaxy-app\"}\n    _setup_conf_file(env, env.nginx_remote_conf_path, \"nginx.conf\", defaults=defaults)\n    env.logger.debug(\"Nginx v{0} installed from package {1}\".format(version, package_url))\n\n\ndef _install_nginx(env):\n    \"\"\"Nginx open source web server.\n    http://www.nginx.org/\n    \"\"\"\n    if \"use_nginx_package\" in env and env.use_nginx_package.upper() in [\"TRUE\", \"YES\"]:\n        _install_nginx_package(env)\n        return\n\n    # Install nginx from directly\n    version = \"1.3.8\"\n    url = \"http://nginx.org/download/nginx-%s.tar.gz\" % version\n\n    install_dir = os.path.join(env.install_dir, \"nginx\")\n    remote_conf_dir = os.path.join(install_dir, \"conf\")\n\n    # Skip install if already present\n    if env.safe_exists(remote_conf_dir) and env.safe_contains(os.path.join(remote_conf_dir, \"nginx.conf\"), \"/cloud\"):\n        env.logger.debug(\"Nginx already installed; not installing it again.\")\n        return\n\n    with _make_tmp_dir() as work_dir:\n        with contextlib.nested(cd(work_dir), settings(hide('stdout'))):\n            modules = _get_nginx_modules(env)\n            module_flags = \" \".join([\"--add-module=../%s\" % x for x in modules])\n            env.safe_run(\"wget %s\" % url)\n            env.safe_run(\"tar xvzf %s\" % os.path.split(url)[1])\n            with cd(\"nginx-%s\" % version):\n                env.safe_run(\"./configure --prefix=%s --with-ipv6 %s \"\n                    \"--user=galaxy --group=galaxy --with-debug \"\n                    \"--with-http_ssl_module --with-http_gzip_static_module \" %\n                    (install_dir, module_flags))\n                env.safe_sed(\"objs/Makefile\", \"-Werror\", \"\")\n                env.safe_run(\"make\")\n                env.safe_sudo(\"make install\")\n                env.safe_sudo(\"cd %s; stow nginx\" % env.install_dir)\n\n    defaults = {\"galaxy_home\": \"/mnt/galaxy/galaxy-app\"}\n    _setup_conf_file(env, os.path.join(remote_conf_dir, \"nginx.conf\"), \"nginx.conf\", defaults=defaults)\n\n    nginx_errdoc_file = 'nginx_errdoc.tar.gz'\n    url = os.path.join(REPO_ROOT_URL, nginx_errdoc_file)\n    remote_errdoc_dir = os.path.join(install_dir, \"html\")\n    with cd(remote_errdoc_dir):\n        env.safe_sudo(\"wget --output-document=%s/%s %s\" % (remote_errdoc_dir, nginx_errdoc_file, url))\n        env.safe_sudo('tar xvzf %s' % nginx_errdoc_file)\n\n    env.safe_sudo(\"mkdir -p %s\" % env.install_dir)\n    if not env.safe_exists(\"%s/nginx\" % env.install_dir):\n        env.safe_sudo(\"ln -s %s/sbin/nginx %s/nginx\" % (install_dir, env.install_dir))\n    # If the guessed symlinking did not work, force it now\n    cloudman_default_dir = \"/opt/galaxy/sbin\"\n    if not env.safe_exists(cloudman_default_dir):\n        env.safe_sudo(\"mkdir -p %s\" % cloudman_default_dir)\n    if not env.safe_exists(os.path.join(cloudman_default_dir, \"nginx\")):\n        env.safe_sudo(\"ln -s %s/sbin/nginx %s/nginx\" % (install_dir, cloudman_default_dir))\n    env.logger.debug(\"Nginx {0} installed to {1}\".format(version, install_dir))\n\n\ndef _get_nginx_modules(env):\n    \"\"\"Retrieve add-on modules compiled along with nginx.\n    \"\"\"\n    modules = {\n        \"upload\": True,\n        \"chunk\": True,\n        \"ldap\": False\n    }\n\n    module_dirs = []\n\n    for module, enabled_by_default in modules.iteritems():\n        enabled = _read_boolean(env, \"nginx_enable_module_%s\" % module, enabled_by_default)\n        if enabled:\n            module_dirs.append(eval(\"_get_nginx_module_%s\" % module)(env))\n\n    return module_dirs\n\n\ndef _get_nginx_module_upload(env):\n    upload_module_version = \"2.2.0\"\n    upload_url = \"http://www.grid.net.ru/nginx/download/\" \\\n                 \"nginx_upload_module-%s.tar.gz\" % upload_module_version\n    env.safe_run(\"wget %s\" % upload_url)\n    upload_fname = os.path.split(upload_url)[1]\n    env.safe_run(\"tar -xvzpf %s\" % upload_fname)\n    return upload_fname.rsplit(\".\", 2)[0]\n\n\ndef _get_nginx_module_chunk(env):\n    chunk_module_version = \"0.22\"\n    chunk_git_version = \"b46dd27\"\n\n    chunk_url = \"https://github.com/agentzh/chunkin-nginx-module/tarball/v%s\" % chunk_module_version\n    chunk_fname = \"agentzh-chunkin-nginx-module-%s.tar.gz\" % (chunk_git_version)\n    env.safe_run(\"wget -O %s %s\" % (chunk_fname, chunk_url))\n    env.safe_run(\"tar -xvzpf %s\" % chunk_fname)\n    return chunk_fname.rsplit(\".\", 2)[0]\n\n\ndef _get_nginx_module_ldap(env):\n    env.safe_run(\"rm -rf nginx-auth-ldap\")  # Delete it if its there or git won't clone\n    env.safe_run(\"git clone https://github.com/kvspb/nginx-auth-ldap\")\n    return \"nginx-auth-ldap\"\n\n\ndef _setup_postgresql(env):\n    # Handled by CloudMan, but if configuring standalone galaxy, this\n    # will need to be executed to create a postgres user for Galaxy.\n    _configure_postgresql(env)\n    _init_postgresql_data(env)\n\n\ndef _configure_postgresql(env, delete_main_dbcluster=False):\n    \"\"\" This method is intended for cleaning up the installation when\n    PostgreSQL is installed from a package. Basically, when PostgreSQL\n    is installed from a package, it creates a default database cluster\n    and splits the config file away from the data.\n    This method can delete the default database cluster that was automatically\n    created when the package is installed. Deleting the main database cluster\n    also has the effect of stopping the auto-start of the postmaster server at\n    machine boot. The method adds all of the PostgreSQL commands to the PATH.\n    \"\"\"\n    pg_ver = sudo(\"dpkg -s postgresql | grep Version | cut -f2 -d':'\")\n    pg_ver = pg_ver.strip()[:3]  # Get first 3 chars of the version since that's all that's used for dir name\n    got_ver = False\n    while(not got_ver):\n        try:\n            pg_ver = float(pg_ver)\n            got_ver = True\n        except Exception:\n            print(red(\"Problems trying to figure out PostgreSQL version.\"))\n            pg_ver = raw_input(red(\"Enter the correct one (eg, 9.1; not 9.1.3): \"))\n    if delete_main_dbcluster:\n        env.safe_sudo('pg_dropcluster --stop %s main' % pg_ver, user='postgres')\n    exp = \"export PATH=/usr/lib/postgresql/%s/bin:$PATH\" % pg_ver\n    if not env.safe_contains('/etc/bash.bashrc', exp):\n        env.safe_append('/etc/bash.bashrc', exp, use_sudo=True)\n\n\ndef _init_postgresql_data(env):\n    if \"galaxy\" not in env.safe_sudo(\"psql -P pager --list | grep galaxy || true\", user=\"postgres\"):\n        env.safe_sudo(\"createdb galaxy\", user=\"postgres\")\n        env.safe_sudo(\"psql -c 'create user galaxy; grant all privileges on database galaxy to galaxy;'\", user=\"postgres\")\n\n\ndef _postgres_running(env):\n    \"\"\"\n    Return ``True`` if PostgreSQL is running, ``False`` otherwise.\n    \"\"\"\n    c = _get_galaxy_db_configs(env)\n    if 'no server running' in sudo(\"{0} status || true\".format(c['pg_ctl_cmd']), user=c['psql_user']):\n        return False\n    return True\n\n\ndef _make_dir_for_galaxy(env, path):\n    env.safe_sudo(\"mkdir -p '%s'\" % path)\n    _chown_galaxy(env, path)\n"
  },
  {
    "path": "cloudbio/galaxy/applications.py",
    "content": "\"\"\"\nThis file is largely derived from a similar file in mi-deployment written Dr.\nEnis Afgan.\n\nhttps://bitbucket.org/afgane/mi-deployment/src/8cba95baf98f/tools_fabfile.py\n\nLong term it will be best to install these packages for Galaxy via the Tool\nShed, however many of these tools are not yet in the tool shed and the tool\nshed installation is not currently available via the Galaxy API. Until such a\ntime as that is available, Galaxy dependencies may be installed via these\nfunctions.\n\nI have taken a first crack at harmonizing this with the rest of CloudBioLinux.\nWasn't able to reuse fastx_toolkit, tophat, cufflinks.\n\n\"\"\"\nimport os\n\nfrom fabric.api import cd\n\nfrom cloudbio.custom.shared import _make_tmp_dir, _if_not_installed, _set_default_config\nfrom cloudbio.custom.shared import _get_install, _configure_make, _fetch_and_unpack, _get_bin_dir\n\n\n@_if_not_installed(None)\ndef install_fastx_toolkit(env):\n    version = env.tool_version\n    gtext_version = \"0.6.1\"\n    url_base = \"http://hannonlab.cshl.edu/fastx_toolkit/\"\n    fastx_url = \"%sfastx_toolkit-%s.tar.bz2\" % (url_base, version)\n    gtext_url = \"%slibgtextutils-%s.tar.bz2\" % (url_base, gtext_version)\n    pkg_name = 'fastx_toolkit'\n    install_dir = os.path.join(env.galaxy_tools_dir, pkg_name, version)\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            env.safe_run(\"wget %s\" % gtext_url)\n            env.safe_run(\"tar -xjvpf %s\" % (os.path.split(gtext_url)[-1]))\n            install_cmd = env.safe_sudo if env.use_sudo else env.safe_run\n            with cd(\"libgtextutils-%s\" % gtext_version):\n                env.safe_run(\"./configure --prefix=%s\" % (install_dir))\n                env.safe_run(\"make\")\n                install_cmd(\"make install\")\n            env.safe_run(\"wget %s\" % fastx_url)\n            env.safe_run(\"tar -xjvpf %s\" % os.path.split(fastx_url)[-1])\n            with cd(\"fastx_toolkit-%s\" % version):\n                env.safe_run(\"export PKG_CONFIG_PATH=%s/lib/pkgconfig; ./configure --prefix=%s\" % (install_dir, install_dir))\n                env.safe_run(\"make\")\n                install_cmd(\"make install\")\n\n\n## TODO: Rework to use more of custom enhancements\n@_if_not_installed(\"maq\")\ndef install_maq(env):\n    version = env[\"tool_version\"]\n    url = \"http://downloads.sourceforge.net/project/maq/maq/%s/maq-%s.tar.bz2\" \\\n            % (version, version)\n    _get_install(url, env, _configure_make)\n\n\n@_if_not_installed(\"macs14\")\ndef install_macs(env):\n    from cloudbio.custom.bio_nextgen  import install_macs as cbl_install_macs\n    install_dir = env.system_install\n    cbl_install_macs(env)\n    env.safe_sudo(\"echo 'PATH=%s/bin:$PATH' > %s/env.sh\" % (install_dir, install_dir))\n    env.safe_sudo(\"echo 'PYTHONPATH=%s/lib/python%s/site-packages:$PYTHONPATH' >> %s/env.sh\" % (env.python_version, install_dir, install_dir))\n    _update_default(env, install_dir)\n\n\n@_if_not_installed(\"megablast\")\ndef install_megablast(env):\n    version = env.tool_version\n    url = 'ftp://ftp.ncbi.nlm.nih.gov/blast/executables/release/%s/blast-%s-x64-linux.tar.gz' % (version, version)\n    install_dir = env.system_install\n    install_cmd = env.safe_sudo if env.use_sudo else env.safe_run\n    if not env.safe_exists(install_dir):\n        install_cmd(\"mkdir -p %s\" % install_dir)\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            env.safe_run(\"wget %s\" % url)\n            env.safe_run(\"tar -xvzf %s\" % os.path.split(url)[-1])\n            with cd('blast-%s/bin' % version):\n                    install_cmd(\"mv * %s\" % install_dir)\n\n\n@_if_not_installed(\"blastn\")\ndef install_blast(env):\n    version = env.tool_version\n    url = 'ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/%s/ncbi-blast-%s-x64-linux.tar.gz' % (version[:-1], version)\n    install_dir = env.system_install\n    install_cmd = env.safe_sudo if env.use_sudo else env.safe_run\n    if not env.safe_exists(install_dir):\n        install_cmd(\"mkdir -p %s\" % install_dir)\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            env.safe_run(\"wget %s\" % url)\n            env.safe_run(\"tar -xvzf %s\" % os.path.split(url)[-1])\n            with cd('ncbi-blast-%s/bin' % version):\n                bin_dir = _get_bin_dir(env)\n                install_cmd(\"mv * '%s'\" % bin_dir)\n\n\n@_if_not_installed(\"sputnik\")\ndef install_sputnik(env):\n    version = env.tool_version\n    url = 'http://bitbucket.org/natefoo/sputnik-mononucleotide/downloads/sputnik_%s_linux2.6_x86_64' % version\n    install_dir = env.system_install\n    install_cmd = env.safe_sudo if env.use_sudo else env.safe_run\n    if not env.safe_exists(install_dir):\n        install_cmd(\"mkdir -p %s\" % install_dir)\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            env.safe_run(\"wget -O sputnik %s\" % url)\n            install_cmd(\"mv sputnik %s\" % install_dir)\n\n\n@_if_not_installed(\"taxonomy2tree\")\ndef install_taxonomy(env):\n    version = env.tool_version\n    url = 'http://bitbucket.org/natefoo/taxonomy/downloads/taxonomy_%s_linux2.6_x86_64.tar.gz' % version\n    install_dir = env.system_install\n    install_cmd = env.safe_sudo if env.use_sudo else env.safe_run\n    if not env.safe_exists(install_dir):\n        install_cmd(\"mkdir -p %s\" % install_dir)\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            env.safe_run(\"wget %s\" % url)\n            env.safe_run(\"tar -xvzf %s\" % os.path.split(url)[-1])\n            with cd(os.path.split(url)[-1].split('.tar.gz')[0]):\n                install_cmd(\"mv * %s\" % install_dir)\n\n\n@_if_not_installed(\"add_scores\")\ndef install_add_scores(env):\n    version = env.tool_version\n    url = 'http://bitbucket.org/natefoo/add_scores/downloads/add_scores_%s_linux2.6_x86_64' % version\n    install_dir = env.system_install\n    install_cmd = env.safe_sudo if env.use_sudo else env.safe_run\n    if not env.safe_exists(install_dir):\n        install_cmd(\"mkdir -p %s\" % install_dir)\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            env.safe_run(\"wget -O add_scores %s\" % url)\n            install_cmd(\"mv add_scores %s\" % install_dir)\n\n\n@_if_not_installed(\"HYPHY\")\ndef install_hyphy(env):\n    version = env.tool_version\n    url = 'http://www.datam0nk3y.org/svn/hyphy'\n    install_dir = env.system_install\n    install_cmd = env.safe_sudo if env.use_sudo else env.safe_run\n    if not env.safe_exists(install_dir):\n        install_cmd(\"mkdir -p %s\" % install_dir)\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            env.safe_run(\"svn co -r %s %s src\" % (version, url))\n            env.safe_run(\"mkdir -p build/Source/Link\")\n            env.safe_run(\"mkdir build/Source/SQLite\")\n            env.safe_run(\"cp src/trunk/Core/*.{h,cp,cpp} build/Source\")\n            env.safe_run(\"cp src/trunk/HeadlessLink/*.{h,cpp} build/Source/SQLite\")\n            env.safe_run(\"cp src/trunk/NewerFunctionality/*.{h,cpp} build/Source/\")\n            env.safe_run(\"cp src/SQLite/trunk/*.{c,h} build/Source/SQLite/\")\n            env.safe_run(\"cp src/trunk/Scripts/*.sh build/\")\n            env.safe_run(\"cp src/trunk/Mains/main-unix.cpp build/Source/main-unix.cxx\")\n            env.safe_run(\"cp src/trunk/Mains/hyphyunixutils.cpp build/Source/hyphyunixutils.cpp\")\n            env.safe_run(\"cp -R src/trunk/{ChartAddIns,DatapanelAddIns,GeneticCodes,Help,SubstitutionClasses,SubstitutionModels,TemplateBatchFiles,TopologyInference,TreeAddIns,UserAddins} build\")\n            env.safe_run(\"rm build/Source/preferences.cpp\")\n            with cd(\"build\"):\n                env.safe_run(\"bash build.sh SP\")\n            install_cmd(\"mv build/* %s\" % install_dir)\n    _update_default(env, install_dir)\n\n\n@_if_not_installed(None)\ndef install_gatk(env):\n    version = env.tool_version\n    url = 'ftp://ftp.broadinstitute.org/pub/gsa/GenomeAnalysisTK/GenomeAnalysisTK-%s.tar.bz2' % version\n    pkg_name = 'gatk'\n    install_dir = os.path.join(env.galaxy_tools_dir, pkg_name, version)\n    install_cmd = env.safe_sudo if env.use_sudo else env.safe_run\n    if not env.safe_exists(install_dir):\n        install_cmd(\"mkdir -p %s\" % install_dir)\n        install_cmd(\"mkdir -p %s/bin\" % install_dir)\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            env.safe_run(\"wget -O gatk.tar.bz2 %s\" % url)\n            env.safe_run(\"tar -xjf gatk.tar.bz2\")\n            install_cmd(\"cp GenomeAnalysisTK-%s/*.jar %s/bin\" % (version, install_dir))\n    # Create shell script to wrap jar\n    env.safe_sudo(\"echo '#!/bin/sh' > %s/bin/gatk\" % (install_dir))\n    env.safe_sudo(\"echo 'java -jar %s/bin/GenomeAnalysisTK.jar $@' >> %s/bin/gatk\" % (install_dir, install_dir))\n    env.safe_sudo(\"chmod +x %s/bin/gatk\" % install_dir)\n    # env file\n    env.safe_sudo(\"echo 'PATH=%s/bin:$PATH' > %s/env.sh\" % (install_dir, install_dir))\n    _update_default(env, install_dir)\n    # Link jar to Galaxy's jar dir\n    jar_dir = os.path.join(env.galaxy_jars_dir, pkg_name)\n    if not env.safe_exists(jar_dir):\n        install_cmd(\"mkdir -p %s\" % jar_dir)\n    tool_dir = os.path.join(env.galaxy_tools_dir, pkg_name, 'default', 'bin')\n    install_cmd('ln --force --symbolic %s/*.jar %s/.' % (tool_dir, jar_dir))\n    install_cmd('chown --recursive %s:%s %s' % (env.galaxy_user, env.galaxy_user, jar_dir))\n\n\n@_if_not_installed(\"srma.jar\")\ndef install_srma(env):\n    version = env.tool_version\n    mirror_info = \"?use_mirror=voxel\"\n    url = 'http://downloads.sourceforge.net/project/srma/srma/%s/srma-%s.jar' \\\n            % (version[:3], version)\n    install_dir = env.system_install\n    install_cmd = env.safe_sudo if env.use_sudo else env.safe_run\n    if not env.safe_exists(install_dir):\n        install_cmd(\"mkdir -p %s\" % install_dir)\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            env.safe_run(\"wget %s%s -O %s\" % (url, mirror_info, os.path.split(url)[-1]))\n            install_cmd(\"mv srma-%s.jar %s\" % (version, install_dir))\n            install_cmd(\"ln -f -s srma-%s.jar %s/srma.jar\" % (version, install_dir))\n    env.safe_sudo(\"touch %s/env.sh\" % install_dir)\n    _update_default(env, install_dir)\n\n\n@_if_not_installed(\"BEAM2\")\ndef install_beam(env):\n    url = 'http://www.stat.psu.edu/~yuzhang/software/beam2.tar'\n    install_dir = env.system_install\n    install_cmd = env.safe_sudo if env.use_sudo else env.safe_run\n    if not env.safe_exists(install_dir):\n        install_cmd(\"mkdir -p %s\" % install_dir)\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            env.safe_run(\"wget %s -O %s\" % (url, os.path.split(url)[-1]))\n            env.safe_run(\"tar xf %s\" % (os.path.split(url)[-1]))\n            install_cmd(\"mv BEAM2 %s\" % install_dir)\n    env.safe_sudo(\"echo 'PATH=%s:$PATH' > %s/env.sh\" % (install_dir, install_dir))\n    _update_default(env, install_dir)\n\n\n@_if_not_installed(\"pass2\")\ndef install_pass(env):\n    url = 'http://www.stat.psu.edu/~yuzhang/software/pass2.tar'\n    install_dir = env.system_install\n    install_cmd = env.safe_sudo if env.use_sudo else env.safe_run\n    if not env.safe_exists(install_dir):\n        install_cmd(\"mkdir -p %s\" % install_dir)\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            env.safe_run(\"wget %s -O %s\" % (url, os.path.split(url)[-1]))\n            env.safe_run(\"tar xf %s\" % (os.path.split(url)[-1]))\n            install_cmd(\"mv pass2 %s\" % install_dir)\n    env.safe_sudo(\"echo 'PATH=%s:$PATH' > %s/env.sh\" % (install_dir, install_dir))\n    _update_default(env, install_dir)\n\n\n@_if_not_installed(\"lps_tool\")\ndef install_lps_tool(env):\n    version = env.tool_version\n    url = 'http://www.bx.psu.edu/miller_lab/dist/lps_tool.%s.tar.gz' % version\n    install_dir = env.system_install\n    install_cmd = env.safe_sudo if env.use_sudo else env.safe_run\n    if not env.safe_exists(install_dir):\n        install_cmd(\"mkdir -p %s\" % install_dir)\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            env.safe_run(\"wget %s -O %s\" % (url, os.path.split(url)[-1]))\n            env.safe_run(\"tar zxf %s\" % (os.path.split(url)[-1]))\n            install_cmd(\"./lps_tool.%s/MCRInstaller.bin -P bean421.installLocation=\\\"%s/MCR\\\" -silent\" % (version, install_dir))\n            install_cmd(\"mv lps_tool.%s/lps_tool %s\" % (version, install_dir))\n    env.safe_sudo(\"echo 'PATH=%s:$PATH' > %s/env.sh\" % (install_dir, install_dir))\n    env.safe_sudo(\"echo 'MCRROOT=%s/MCR/v711; export MCRROOT' >> %s/env.sh\" % (install_dir, install_dir))\n    _update_default(env, install_dir)\n\n\n@_if_not_installed(\"plink\")\ndef install_plink(env):\n    version = env.tool_version\n    url = 'http://pngu.mgh.harvard.edu/~purcell/plink/dist/plink-%s-x86_64.zip' % version\n    install_dir = env.system_install\n    install_cmd = env.safe_sudo if env.use_sudo else env.safe_run\n    if not env.safe_exists(install_dir):\n        install_cmd(\"mkdir -p %s\" % install_dir)\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            env.safe_run(\"wget %s -O %s\" % (url, os.path.split(url)[-1]))\n            env.safe_run(\"unzip %s\" % (os.path.split(url)[-1]))\n            install_cmd(\"mv plink-%s-x86_64/plink %s\" % (version, install_dir))\n    env.safe_sudo(\"echo 'PATH=%s:$PATH' > %s/env.sh\" % (install_dir, install_dir))\n    _update_default(env, install_dir)\n\n\n@_if_not_installed(None)\ndef install_fbat(env):\n    version = env.tool_version\n    url = 'http://www.biostat.harvard.edu/~fbat/software/fbat%s_linux64.tar.gz' % version.replace('.', '')\n    install_dir = env.system_install\n    install_cmd = env.safe_sudo if env.use_sudo else env.safe_run\n    if not env.safe_exists(install_dir):\n        install_cmd(\"mkdir -p %s\" % install_dir)\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            env.safe_run(\"wget %s -O %s\" % (url, os.path.split(url)[-1]))\n            env.safe_run(\"tar zxf %s\" % (os.path.split(url)[-1]))\n            install_cmd(\"mv fbat %s\" % install_dir)\n    env.safe_sudo(\"echo 'PATH=%s:$PATH' > %s/env.sh\" % (install_dir, install_dir))\n    _update_default(env, install_dir)\n\n\n@_if_not_installed(\"Haploview_beta.jar\")\ndef install_haploview(env):\n    url = 'http://www.broadinstitute.org/ftp/pub/mpg/haploview/Haploview_beta.jar'\n    install_dir = env.system_install\n    install_cmd = env.safe_sudo if env.use_sudo else env.safe_run\n    if not env.safe_exists(install_dir):\n        install_cmd(\"mkdir -p %s\" % install_dir)\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            env.safe_run(\"wget %s -O %s\" % (url, os.path.split(url)[-1]))\n            install_cmd(\"mv %s %s\" % (os.path.split(url)[-1], install_dir))\n            install_cmd(\"ln -s %s %s/haploview.jar\" % (os.path.split(url)[-1], install_dir))\n    _update_default(env, install_dir)\n\n\n@_if_not_installed(\"eigenstrat\")\ndef install_eigenstrat(env):\n    version = env.tool_version\n    url = 'http://www.hsph.harvard.edu/faculty/alkes-price/files/EIG%s.tar.gz' % version\n    install_dir = env.system_install\n    install_cmd = env.safe_sudo if env.use_sudo else env.safe_run\n    if not env.safe_exists(install_dir):\n        install_cmd(\"mkdir -p %s\" % install_dir)\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            env.safe_run(\"wget %s -O %s\" % (url, os.path.split(url)[-1]))\n            env.safe_run(\"tar zxf %s\" % (os.path.split(url)[-1]))\n            install_cmd(\"mv bin %s\" % install_dir)\n    env.safe_sudo(\"echo 'PATH=%s/bin:$PATH' > %s/env.sh\" % (install_dir, install_dir))\n    _update_default(env, install_dir)\n\n\n@_if_not_installed(\"augustus\")\ndef install_augustus(env):\n    default_version = \"2.7\"\n    version = env.get('tool_version', default_version)\n    url = \"http://bioinf.uni-greifswald.de/augustus/binaries/augustus.%s.tar.gz\" % version\n    install_dir = env.system_install\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            _fetch_and_unpack(url, need_dir=False)\n            env.safe_sudo(\"mkdir -p '%s'\" % install_dir)\n            env.safe_sudo(\"mv augustus.%s/* '%s'\" % (version, install_dir))\n\n\n@_if_not_installed(\"SortSam.jar\")\ndef install_picard(env):\n    version = env.tool_version\n    mirror_info = \"?use_mirror=voxel\"\n    url = 'http://downloads.sourceforge.net/project/picard/picard-tools/%s/picard-tools-%s.zip' % (version, version)\n    pkg_name = 'picard'\n    install_dir = env.system_install\n    install_cmd = env.safe_sudo if env.use_sudo else env.safe_run\n    if not env.safe_exists(install_dir):\n        install_cmd(\"mkdir -p %s\" % install_dir)\n    with _make_tmp_dir() as work_dir:\n        with cd(work_dir):\n            env.safe_run(\"wget %s%s -O %s\" % (url, mirror_info, os.path.split(url)[-1]))\n            env.safe_run(\"unzip %s\" % (os.path.split(url)[-1]))\n            install_cmd(\"mv picard-tools-%s/*.jar %s\" % (version, install_dir))\n    _update_default(env, install_dir)\n    # set up the jars directory\n    jar_dir = os.path.join(env.galaxy_jars_dir, 'picard')\n    if not env.safe_exists(jar_dir):\n        install_cmd(\"mkdir -p %s\" % jar_dir)\n    tool_dir = os.path.join(env.galaxy_tools_dir, pkg_name, 'default')\n    install_cmd('ln --force --symbolic %s/*.jar %s/.' % (tool_dir, jar_dir))\n    install_cmd('chown --recursive %s:%s %s' % (env.galaxy_user, env.galaxy_user, jar_dir))\n\n\n@_if_not_installed(\"fastqc\")\ndef install_fastqc(env):\n    \"\"\" This tool is installed in Galaxy's jars dir \"\"\"\n    version = env.tool_version\n    url = 'http://www.bioinformatics.bbsrc.ac.uk/projects/fastqc/fastqc_v%s.zip' % version\n    pkg_name = 'FastQC'\n    install_dir = os.path.join(env.galaxy_jars_dir)\n    install_cmd = env.safe_sudo if env.use_sudo else env.safe_run\n    if not env.safe_exists(install_dir):\n        install_cmd(\"mkdir -p %s\" % install_dir)\n    with cd(install_dir):\n        install_cmd(\"wget %s -O %s\" % (url, os.path.split(url)[-1]))\n        install_cmd(\"unzip -u %s\" % (os.path.split(url)[-1]))\n        install_cmd(\"rm %s\" % (os.path.split(url)[-1]))\n        with cd(pkg_name):\n            install_cmd('chmod 755 fastqc')\n        install_cmd('chown --recursive %s:%s %s' % (env.galaxy_user, env.galaxy_user, pkg_name))\n\n\ndef _update_default(env, install_dir):\n    env.safe_sudo(\"touch %s/env.sh\" % install_dir)\n    env.safe_sudo(\"chmod +x %s/env.sh\" % install_dir)\n    _set_default_config(env, install_dir)\n\n#@if_tool_not_found()\n#def install_emboss(env):\n#    version = env.tool_version\n#    url = 'ftp://emboss.open-bio.org/pub/EMBOSS/old/%s/EMBOSS-%s.tar.gz' % (version, version)\n#    pkg_name = 'emboss'\n#    install_dir = os.path.join(env.galaxy_tools_dir, pkg_name, version)\n#    install_cmd = env.safe_sudo if env.use_sudo else env.safe_run\n#    if not env.safe_exists(install_dir):\n#        install_cmd(\"mkdir -p %s\" % install_dir)\n#    with _make_tmp_dir() as work_dir:\n#        with cd(work_dir):\n#            env.safe_run(\"wget %s\" % url)\n#            env.safe_run(\"tar -xvzf %s\" % os.path.split(url)[-1])\n#            with cd(os.path.split(url)[-1].split('.tar.gz')[0]):\n#                env.safe_run(\"./configure --prefix=%s\" % install_dir)\n#                env.safe_run(\"make\")\n#                install_cmd(\"make install\")\n#    phylip_version = '3.6b'\n#    url = 'ftp://emboss.open-bio.org/pub/EMBOSS/old/%s/PHYLIP-%s.tar.gz' % (version, phylip_version)\n#    with _make_tmp_dir() as work_dir:\n#        with cd(work_dir):\n#            env.safe_run(\"wget %s\" % url)\n#            env.safe_run(\"tar -xvzf %s\" % os.path.split(url)[-1])\n#            with cd(os.path.split(url)[-1].split('.tar.gz')[0]):\n#                env.safe_run(\"./configure --prefix=%s\" % install_dir)\n#                env.safe_run(\"make\")\n#                install_cmd(\"make install\")\n\n"
  },
  {
    "path": "cloudbio/galaxy/r.py",
    "content": "import os\nimport tempfile\n\n\nfrom cloudbio.custom.shared import _make_tmp_dir\nfrom fabric.api import sudo, put, cd\n\nr_packages_template = \"\"\"\nr <- getOption(\"repos\");\nr[\"CRAN\"] <- \"http://watson.nci.nih.gov/cran_mirror\";\noptions(repos=r);\ninstall.packages( c( %s ), dependencies = TRUE);\nsource(\"http://bioconductor.org/biocLite.R\");\nbiocLite( c( %s ) );\n\"\"\"\n\n\ndef _install_r_packages(tools_conf):\n    f = tempfile.NamedTemporaryFile()\n    r_packages = tools_conf[\"r_packages\"]\n    bioconductor_packages = tools_conf[\"bioconductor_packages\"]\n    if not r_packages and not bioconductor_packages:\n        return\n    r_cmd = r_packages_template % (_concat_strings(r_packages), _concat_strings(bioconductor_packages))\n    f.write(r_cmd)\n    f.flush()\n    with _make_tmp_dir() as work_dir:\n        put(f.name, os.path.join(work_dir, 'install_packages.r'))\n        with cd(work_dir):\n            sudo(\"R --vanilla --slave < install_packages.r\")\n    f.close()\n\n\ndef _concat_strings(strings):\n    if strings:\n        return \", \".join(map(lambda x: '\"%s\"' % x, strings))\n    else:\n        return \"\"\n"
  },
  {
    "path": "cloudbio/galaxy/tools.py",
    "content": "import os\nfrom string import Template\n\nimport six\nimport yaml\n\nfrom cloudbio.custom.bio_general import *\nfrom cloudbio.custom.bio_nextgen import *\nfrom cloudbio.custom.bio_proteomics import *\nfrom cloudbio.custom.shared import _set_default_config, _add_to_profiles\nfrom cloudbio.galaxy.applications import *\nfrom cloudbio.galaxy.r import _install_r_packages\nfrom cloudbio.galaxy.utils import _chown_galaxy, _read_boolean\n\nFAILED_INSTALL_MESSAGE = \\\n    \"Failed to install application %s as a Galaxy application. This may be a transient problem (e.g. mirror for download is currently unavailable) or misconfiguration. The contents of the CloudBioLinux temporary working directory may need to be deleted.\"\n\n\ndef _install_tools(env, tools_conf=None):\n    \"\"\"\n    Install tools needed for Galaxy along with tool configuration\n    directories needed by Galaxy.\n    \"\"\"\n\n    if not tools_conf:\n        tools_conf = _load_tools_conf(env)\n\n    if _read_boolean(env, \"galaxy_install_dependencies\", False):\n       # Need to ensure the install dir exists and is owned by env.galaxy_user\n        _setup_install_dir(env)\n        _install_configured_applications(env, tools_conf)\n        _chown_galaxy(env, env.galaxy_tools_dir)\n        _chown_galaxy(env, env.galaxy_jars_dir)\n\n    if _read_boolean(env, \"galaxy_install_r_packages\", False):\n        _install_r_packages(tools_conf)\n\n\ndef _tools_conf_path(env):\n    \"\"\"\n    Load path to galaxy_tools_conf file from env, allowing expansion of $__contrib_dir__.\n    Default to $__contrib_dir__/flavor/cloudman/tools.yaml.\n    \"\"\"\n    contrib_dir = os.path.join(env.config_dir, os.pardir, \"contrib\")\n    default_tools_conf_path = os.path.join(contrib_dir, \"flavor\", \"cloudman\", \"tools.yaml\")\n    tools_conf_path = env.get(\"galaxy_tools_conf\", default_tools_conf_path)\n    ## Allow expansion of __config_dir__ in galaxy_tools_conf property.\n    return Template(tools_conf_path).safe_substitute({\"__contrib_dir__\": contrib_dir})\n\n\ndef _load_tools_conf(env):\n    with open(_tools_conf_path(env)) as in_handle:\n        full_data = yaml.safe_load(in_handle)\n    return full_data\n\n\ndef _setup_install_dir(env):\n    \"\"\"Sets up install dir and ensures its owned by Galaxy\"\"\"\n    if not env.safe_exists(env.galaxy_tools_dir):\n        env.safe_sudo(\"mkdir -p %s\" % env.galaxy_tools_dir)\n        _chown_galaxy(env, env.galaxy_tools_dir)\n    # Create a general-purpose ``bin`` directory under the galaxy_tools_dir\n    # and put it on the PATH so users can more easily add custom tools\n    bin_dir = os.path.join(env.galaxy_tools_dir, 'bin')\n    if not env.safe_exists(bin_dir):\n        env.safe_sudo(\"mkdir -p %s\" % bin_dir)\n        _chown_galaxy(env, bin_dir)\n        line = \"export PATH={0}:$PATH\".format(bin_dir)\n        _add_to_profiles(line)\n    if not env.safe_exists(env.galaxy_jars_dir):\n        env.safe_sudo(\"mkdir -p %s\" % env.galaxy_jars_dir)\n        _chown_galaxy(env, env.galaxy_jars_dir)\n\n\ndef _install_configured_applications(env, tools_conf):\n    \"\"\"\n    Install external tools defined by YAML or dictionary data structure.  Instead of\n    installing in system_install (e.g. /usr), these custom tools will be installed as\n    Galaxy dependency applications.\n    \"\"\"\n    applications = tools_conf[\"applications\"] or {}\n    # Changing the default behavior here to install all tools and\n    # just record exceptions as they occur, but wait until the end\n    # raise an exception out of this block. Disable this behavior\n    # by setting galaxay_tool_defer_errors to False.\n    defer_errors = env.get(\"galaxy_tool_defer_errors\", True)\n    exceptions = {}\n    for (name, tool_conf) in applications.iteritems():\n        if not __check_conditional(tool_conf):\n            continue\n\n        try:\n            _install_application(name, tool_conf)\n        except BaseException as e:\n            exceptions[name] = e\n            if not defer_errors:\n                break\n\n    if exceptions:\n        for name, exception in exceptions.iteritems():\n            env.logger.warn(FAILED_INSTALL_MESSAGE % name)\n        first_exception = list(exceptions.values())[0]\n        raise first_exception\n\n\ndef _install_application(name, versions, tool_install_dir=None):\n    \"\"\"\n    Install single custom tool as Galaxy dependency application.\n\n    TODO: Rename versions and document options.\n    \"\"\"\n    if type(versions) is str:\n        versions = [versions]\n    for version_info in versions:\n        if type(version_info) is str:\n            _install_tool(env, name, version=version_info, requirement_name=name, tool_install_dir=tool_install_dir)\n        else:\n            version = version_info[\"version\"]\n            bin_dirs = version_info.get(\"bin_dirs\", [\"bin\"])\n            env_vars = version_info.get(\"env_vars\", {})\n            provides = version_info.get(\"provides\", [])\n            if isinstance(provides, (str, unicode, six.string_types)):\n                provides = [provides]\n            for provide_conf in provides[:]:\n                if isinstance(provide_conf, dict):\n                    provides.remove(provide_conf)\n                    if __check_conditional(provide_conf):\n                        provies.append(provide_conf[\"name\"])\n\n            # Some requirements (e.g. blast+) maybe not have valid python\n            # identifiers as name. Use install_blast to setup but override\n            # requirement directory name with requirement_name field.\n            requirement_name = version_info.get(\"requirement_name\", name)\n            tool_env = _install_tool(env, name, version, bin_dirs=bin_dirs, env_vars=env_vars, requirement_name=requirement_name, tool_install_dir=tool_install_dir)\n            symlink_versions = version_info.get(\"symlink_versions\", [])\n            if type(symlink_versions) is str:\n                symlink_versions = [symlink_versions]\n            for symlink_version in symlink_versions:\n                _set_default_config(tool_env, tool_env[\"system_install\"], symlink_version)\n\n            if provides:\n                install_dir = tool_env[\"system_install\"]\n                ## Create additional symlinked packages from this one.\n                tool_dir = \"%s/..\" % install_dir\n                tools_dir = \"%s/..\" % tool_dir\n                for package in provides:\n                    link_dir = \"%s/%s\" % (tools_dir, package)\n                    env.safe_sudo(\"ln -f -s '%s' '%s'\" % (requirement_name, link_dir))\n\n\ndef _install_tool(env, name, version, requirement_name, bin_dirs=[\"bin\"], env_vars={}, tool_install_dir=None):\n    tool_env = _build_tool_env(env, requirement_name, version, tool_install_dir)\n    env.logger.debug(\"Installing a Galaxy tool via 'install_%s'\" % name)\n    eval(\"install_%s\" % name)(tool_env)\n    _install_galaxy_config(tool_env, bin_dirs, env_vars=env_vars)\n    return tool_env\n\n\ndef _build_tool_env(env, name, version, tool_install_dir):\n    \"\"\" Build new env to have tool installed for Galaxy instead of into /usr. \"\"\"\n    tool_env = {\"tool_version\": version,\n                \"galaxy_tool_install\": True}\n    for key, value in env.iteritems():\n        tool_env[key] = value\n    if not tool_install_dir:\n        tool_install_dir = os.path.join(env.galaxy_tools_dir, name, version)\n    tool_env[\"system_install\"] = tool_install_dir\n    tool_env[\"local_install\"] = tool_install_dir\n    tool_env[\"venv_directory\"] = \"%s/%s\" % (tool_env[\"system_install\"], \"venv\")\n    return AttributeDict(tool_env)\n\n\ndef __check_conditional(conf_dict):\n    passes = True\n    try:\n        if \"if\" in conf_dict:\n            value = conf_dict[\"if\"]\n            passes = _read_boolean(env, value, False)\n        elif \"unless\" in conf_dict:\n            value = conf_dict[\"unless\"]\n            passes = not _read_boolean(env, value, False)\n    except TypeError:\n        # configuration is not a dictionary, default to True\n        pass\n    return passes\n\n\nclass AttributeDict(dict):\n    \"\"\"\n    Dictionary that allows attribute access to values.\n\n    This is needed because cloudbio.custom.* accesses env extensively via\n    attributes (e.g. env.system_install).\n\n    http://stackoverflow.com/questions/4984647/accessing-dict-keys-like-an-attribute-in-python\n    \"\"\"\n    __getattr__ = dict.__getitem__\n    __setattr__ = dict.__setitem__\n\n\ndef _install_galaxy_config(tool_env, bin_dirs, env_vars):\n    \"\"\"\n    Setup galaxy tool config files (env.sh-es) and default version\n    symbolic links.\n    \"\"\"\n    install_dir = tool_env[\"system_install\"]\n    env_path = os.path.join(install_dir, \"env.sh\")\n    bin_paths = [os.path.join(install_dir, bin_dir) for bin_dir in bin_dirs]\n    path_pieces = [bin_path for bin_path in bin_paths if env.safe_exists(bin_path)]\n    if len(path_pieces) > 0 and not env.safe_exists(env_path):\n        path_addtion = \":\".join(path_pieces)\n        # Standard bin install, just add it to path\n        env.safe_sudo(\"echo 'PATH=%s:$PATH' > %s\" % (path_addtion, env_path))\n        venv_path = \"%s/%s\" % (install_dir, \"venv\")\n        if env.safe_exists(venv_path):\n            #  Have env.sh activate virtualdirectory\n            env.safe_sudo(\"echo '. %s/bin/activate' >> %s\" % (venv_path, env_path))\n        env.safe_sudo(\"chmod +x %s\" % env_path)\n        for env_var, env_var_value in env_vars.iteritems():\n            env_var_template = Template(env_var_value)\n            expanded_env_var_value = env_var_template.substitute(tool_env)\n            env.safe_sudo(\"echo 'export %s=%s' >> %s\" % (env_var, expanded_env_var_value, env_path))\n        env.logger.debug(\"Added Galaxy env.sh file: %s\" % env_path)\n\n    # TODO: If a direct install (i.e. tool_install_dir specified instead of galaxy_tools_dir)\n    # default is still setup. This is not really desired.\n    _set_default_config(tool_env, install_dir)\n    if _read_boolean(tool_env, \"autoload_galaxy_tools\", False) and env.safe_exists(env_path):\n        # In this case, the web user (e.g. ubuntu) should auto-load all of\n        # galaxy's default env.sh files so they are available for direct use\n        # as well.\n        _add_to_profiles(\". %s\" % env_path, profiles=[\"~/.bashrc\"])\n"
  },
  {
    "path": "cloudbio/galaxy/utils.py",
    "content": "from fabric.api import sudo\nfrom fabric.contrib.files import exists\n\n\ndef _read_boolean(env, name, default):\n    ## TODO: Replace calls to this with calls to cloudbio.custom.shared version\n    property_str = env.get(name, str(default))\n    return property_str.upper() in [\"TRUE\", \"YES\"]\n\n\ndef _chown_galaxy(env, path):\n    \"\"\"\n    Recursively change ownership of ``path``, first checking if ``path`` exists.\n    \"\"\"\n    chown_command = \"chown --recursive %s:%s %s\"\n    galaxy_user = env.get(\"galaxy_user\", \"galaxy\")\n    if env.safe_exists(path):\n        env.safe_sudo(chown_command % (galaxy_user, galaxy_user, path))\n\n\ndef _dir_is_empty(path):\n    \"\"\"\n    Return ``True`` is ``path`` directory has no files or folders in it.\n    Return ``False`` otherwise.\n    \"\"\"\n    if \"empty\" in sudo('[ \"$(ls -A {0})\" ] || echo \"empty\"'.format(path)):\n        return True\n    return False\n"
  },
  {
    "path": "cloudbio/libraries.py",
    "content": "\"\"\"Installers for programming language specific libraries.\n\"\"\"\nimport os\n\nfrom fabric.api import env, cd, settings\nfrom cloudbio import fabutils\nfrom cloudbio.custom import shared\n\ndef r_library_installer(config):\n    \"\"\"Install R libraries using CRAN and Bioconductor.\n    \"\"\"\n    if config.get(\"cran\") or config.get(\"bioc\") or config.get(\"github\"):\n        with shared._make_tmp_dir() as tmp_dir:\n            with cd(tmp_dir):\n                # Create an Rscript file with install details.\n                out_file = os.path.join(tmp_dir, \"install_packages.R\")\n                _make_install_script(out_file, config)\n                # run the script and then get rid of it\n                # try using either\n                rlib_installed = False\n                rscripts = []\n                conda_bin = shared._conda_cmd(env)\n                if conda_bin:\n                    rscripts.append(fabutils.find_cmd(env, os.path.join(os.path.dirname(conda_bin), \"Rscript\"),\n                                                    \"--version\"))\n                rscripts.append(fabutils.find_cmd(env, \"Rscript\", \"--version\"))\n                for rscript in rscripts:\n                    if rscript:\n                        env.safe_run(\"%s %s\" % (rscript, out_file))\n                        rlib_installed = True\n                        break\n                if not rlib_installed:\n                    env.logger.warn(\"Rscript not found; skipping install of R libraries.\")\n                env.safe_run(\"rm -f %s\" % out_file)\n\ndef _make_install_script(out_file, config):\n    if env.safe_exists(out_file):\n        env.safe_run(\"rm -f %s\" % out_file)\n    env.safe_run(\"touch %s\" % out_file)\n    lib_loc = os.path.join(env.system_install, \"lib\", \"R\", \"site-library\")\n    env.safe_sudo(\"mkdir -p %s\" % lib_loc)\n    with settings(warn_only=True):\n        env.safe_sudo(\"chown -R %s %s\" % (env.user, lib_loc))\n    repo_info = \"\"\"\n    .libPaths(c(\"%s\"))\n    library(methods)\n    cran.repos <- getOption(\"repos\")\n    cran.repos[\"CRAN\" ] <- \"%s\"\n    options(repos=cran.repos)\n    \"\"\" % (lib_loc, config[\"cranrepo\"])\n    if config.get(\"biocrepo\"):\n        repo_info += \"\"\"\\nsource(\"%s\")\\n\"\"\" % config[\"biocrepo\"]\n    env.safe_append(out_file, repo_info)\n    install_fn = \"\"\"\n    repo.installer <- function(repos, install.fn, pkg_name_fn) {\n      %s\n      maybe.install <- function(pname) {\n        if (!is.null(pkg_name_fn)) {\n           pinfo <- pkg_name_fn(pname)\n           ipkgs <- installed.packages()[,3][pinfo[\"pkg\"]]\n           if (is.na(ipkgs[pinfo[\"pkg\"]]) || pinfo[\"version\"] != ipkgs[pinfo[\"pkg\"]])\n             try(install.fn(pinfo[\"pname\"]))\n        }\n        else if (!(is.element(pname, installed.packages()[,1])))\n           install.fn(pname)\n      }\n\n    }\n    \"\"\"\n    if config.get(\"update_packages\", True):\n        update_str = \"\"\"\n        update.packages(lib.loc=\"%s\", repos=repos, ask=FALSE)\n        \"\"\" % lib_loc\n    else:\n        update_str = \"\\n\"\n    env.safe_append(out_file, install_fn % update_str)\n    if len(config.get(\"cran\") or []) > 0:\n        std_install = \"\"\"\n        std.pkgs <- c(%s)\n        std.installer = repo.installer(cran.repos, install.packages, NULL)\n        lapply(std.pkgs, std.installer)\n        \"\"\" % (\", \".join('\"%s\"' % p for p in config['cran']))\n        env.safe_append(out_file, std_install)\n    if len(config.get(\"bioc\") or []) > 0:\n        bioc_install = \"\"\"\n        bioc.pkgs <- c(%s)\n        bioc.installer = repo.installer(biocinstallRepos(), biocLite, NULL)\n        lapply(bioc.pkgs, bioc.installer)\n        \"\"\" % (\", \".join('\"%s\"' % p for p in config['bioc']))\n        env.safe_append(out_file, bioc_install)\n    if config.get(\"cran-after-bioc\"):\n        std2_install = \"\"\"\n        std2.pkgs <- c(%s)\n        lapply(std2.pkgs, std.installer)\n        \"\"\" % (\", \".join('\"%s\"' % p for p in config['cran-after-bioc']))\n        env.safe_append(out_file, std2_install)\n    if config.get(\"github\"):\n        dev_install = \"\"\"\n        library(devtools)\n        github.pkgs <- c(%s)\n        get_pkg_name <- function(orig) {\n          c(pkg=unlist(strsplit(unlist(strsplit(orig, \"/\"))[2], \"@\"))[1],\n            version=unlist(strsplit(orig, \";\"))[2],\n            pname=unlist(strsplit(orig, \";\"))[1])\n        }\n        gh_install <- function(name) {\n          install_github(name, upgrade_dependencies=FALSE)\n        }\n        github_installer = repo.installer(NULL, gh_install, get_pkg_name)\n        lapply(github.pkgs, github_installer)\n        \"\"\" % (\", \".join('\"%s\"' % p for p in config['github']))\n        env.safe_append(out_file, dev_install)"
  },
  {
    "path": "cloudbio/manifest.py",
    "content": "\"\"\"Provide dump of software and libraries installed on CloudBioLinux image.\n\nThis provides an output YAML file with package details, providing a complete\ndump of installed software and packages. The YAML output feeds into a BioGems\nstyle webpage that provides a more human friendly view of installed packages.\nThe version information provides a reproducible dump of software on a system.\n\"\"\"\nimport os\nimport collections\nimport json\nimport inspect\nimport subprocess\nimport sys\n\nfrom six.moves import urllib\n\nimport yaml\ntry:\n    import yolk.yolklib\n    import yolk.metadata\nexcept ImportError:\n    yolk = None\n\ndef create(out_dir, tooldir=\"/usr/local\", fetch_remote=False):\n    \"\"\"Create a manifest in the output directory with installed packages.\n    \"\"\"\n    if not os.path.exists(out_dir):\n        os.makedirs(out_dir)\n    #write_debian_pkg_info(out_dir, fetch_remote)\n    write_python_pkg_info(out_dir)\n    write_r_pkg_info(out_dir)\n    #write_brew_pkg_info(out_dir, tooldir)\n    #write_custom_pkg_info(out_dir, tooldir)\n\n# ## Custom packages\n\ndef _get_custom_pkg_info(name, fn):\n    \"\"\"Retrieve information about the installed package from the install function.\n    \"\"\"\n    vals = dict((k, v) for k, v in inspect.getmembers(fn))\n    code = inspect.getsourcelines(fn)\n    if vals[\"__name__\"] == \"decorator\":\n        fn = [x for x in fn.func_closure if not isinstance(x.cell_contents, str)][0].cell_contents\n        vals = dict((k, v) for k, v in inspect.getmembers(fn))\n        code = inspect.getsourcelines(fn)\n    version = \"\"\n    for line in (l.strip() for l in code[0]):\n        if line.find(\"version\") >= 0 and line.find(\" =\") > 0:\n            version = line.split()[-1].replace('\"', '').replace(\"'\", \"\")\n        if version:\n            break\n    doc = vals.get(\"func_doc\", \"\")\n    descr, homepage = \"\", \"\"\n    if doc is not None:\n        descr = doc.split(\"\\n\")[0]\n        for line in doc.split(\"\\n\"):\n            if line.strip().startswith(\"http\"):\n                homepage = line.strip()\n    return {\"name\": name.replace(\"install_\", \"\"),\n            \"description\": descr,\n            \"homepage_uri\": homepage,\n            \"version\": version}\n\ndef write_custom_pkg_info(out_dir, tooldir):\n    custom_names = [\"bio_general\", \"bio_nextgen\", \"cloudman\", \"distributed\",\n                    \"java\", \"python\", \"phylogeny\", \"system\"]\n    out_file = os.path.join(out_dir, \"custom-packages.yaml\")\n    if not os.path.exists(out_file):\n        out = {}\n        for modname in custom_names:\n            try:\n                mod = getattr(__import__(\"cloudbio.custom\", globals(), locals(),\n                                         [modname], -1),\n                              modname)\n            except ImportError as msg:\n                # Skip fabric import errors as we transition away from it\n                if \"fabric\" in str(msg):\n                    mod = None\n                else:\n                    raise\n            except ValueError:\n                mod = None\n            if mod:\n                for prog in [x for x in dir(mod) if x.startswith(\"install\")]:\n                    pkg = _get_custom_pkg_info(prog, getattr(mod, prog))\n                    out[pkg[\"name\"]] = pkg\n        with open(out_file, \"w\") as out_handle:\n            yaml.safe_dump(out, out_handle, default_flow_style=False, allow_unicode=False)\n    return out_file\n\n# ## Homebrew/Linuxbrew packages\n\ndef write_brew_pkg_info(out_dir, tooldir):\n    \"\"\"Extract information for packages installed by homebrew/linuxbrew.\n    \"\"\"\n    out_file = os.path.join(out_dir, \"brew-packages.yaml\")\n    if not os.path.exists(out_file):\n        brew_cmd = os.path.join(tooldir, \"bin\", \"brew\") if tooldir else None\n        if not brew_cmd or not os.path.exists(brew_cmd):\n            brew_cmd = \"brew\"\n        try:\n            vout = subprocess.check_output([brew_cmd, \"list\", \"--versions\"]).decode()\n        except (OSError, subprocess.CalledProcessError):  # brew not installed/used\n            vout = \"\"\n        out = {}\n        for vstr in vout.split(\"\\n\"):\n            if vstr.strip():\n                parts = vstr.rstrip().split()\n                name = parts[0]\n                v = parts[-1]\n                # remove any revisions from the version\n                v = v.rsplit(\"_\", 1)[0]\n                out[name] = {\"name\": name, \"version\": v}\n        with open(out_file, \"w\") as out_handle:\n            yaml.safe_dump(out, out_handle, default_flow_style=False, allow_unicode=False)\n    return out_file\n\n# ## R packages\n\ndef get_r_pkg_info():\n    r_command = (\"options(width=10000); subset(installed.packages(fields=c('Title', 'URL')), \"\n                 \"select=c('Version', 'Title','URL'))\")\n    try:\n        out = subprocess.check_output([\"Rscript\", \"-e\", r_command]).decode()\n    except (subprocess.CalledProcessError, OSError):\n        out = \"\"\n    pkg_raw_list = []\n    for line in out.split(\"\\n\")[1:]:\n        pkg_raw_list.append(list(filter(None, [entry.strip(' ') for entry in line.split('\"')])))\n    for pkg in pkg_raw_list:\n        if len(pkg) > 2:\n            yield {\"name\": pkg[0], \"version\": pkg[1],\n                   \"description\": pkg[2],\n                   \"homepage_uri\": (pkg[3], '')[pkg[3] == 'NA'] if len(pkg) > 3 else \"\"}\n\ndef write_r_pkg_info(out_dir):\n    out_file = os.path.join(out_dir, \"r-packages.yaml\")\n    if not os.path.exists(out_file):\n        out = {}\n        for pkg in get_r_pkg_info():\n            out[pkg[\"name\"]] = pkg\n        with open(out_file, \"w\") as out_handle:\n            yaml.safe_dump(out, out_handle, default_flow_style=False, allow_unicode=False)\n    return out_file\n\n# ## Python packages\n\ndef _get_conda_envs(conda_bin):\n    info = json.loads(subprocess.check_output(\"{conda_bin} info --envs --json\".format(**locals()), shell=True))\n    prefix = info[\"conda_prefix\"] + \"/envs/\"\n    return [e.replace(prefix, \"\") for e in info[\"envs\"] if e.startswith(prefix)]\n\ndef get_python_pkg_info():\n    if yolk:\n        for dist in yolk.yolklib.Distributions().get_packages(\"all\"):\n            md = yolk.metadata.get_metadata(dist)\n            yield {\"name\": md[\"Name\"].lower(), \"version\": md[\"Version\"],\n                   \"description\": md.get(\"Summary\", \"\"),\n                   \"homepage_uri\": md.get(\"Home-page\", \"\")}\n    else:\n        base_dir = os.path.dirname(os.path.realpath(sys.executable))\n        conda_bin = os.path.join(base_dir, \"conda\")\n        if os.path.exists(conda_bin):\n            for line in subprocess.check_output([conda_bin, \"list\"]).decode().split(\"\\n\"):\n                if line.strip() and not line.startswith(\"#\"):\n                    name, version = line.split()[:2]\n                    yield {\"name\": name.lower(), \"version\": version}\n            for env in _get_conda_envs(conda_bin):\n                for line in subprocess.check_output([conda_bin, \"list\", \"-n\", env]).decode().split(\"\\n\"):\n                    if line.strip() and not line.startswith(\"#\"):\n                        name, version = line.split()[:2]\n                    yield {\"name\": name.lower(), \"version\": version}\n\n        else:\n            for line in subprocess.check_output([os.path.join(base_dir, \"pip\"), \"list\"]).decode().split(\"\\n\"):\n                if line.strip() and not line.startswith(\"#\"):\n                    name, version = line.split()[:2]\n                    yield {\"name\": name.lower(), \"version\": version[1:-1]}\n\ndef _resolve_latest_pkg(pkgs):\n    if len(pkgs) == 1 or not yolk:\n        return pkgs[0]\n    else:\n        latest_version = yolk.yolklib.Distributions().get_highest_installed(pkgs[0][\"name\"])\n        return [x for x in pkgs if x[\"version\"] == latest_version][0]\n\ndef write_python_pkg_info(out_dir):\n    out_file = os.path.join(out_dir, \"python-packages.yaml\")\n    if not os.path.exists(out_file):\n        pkgs_by_name = collections.defaultdict(list)\n        for pkg in get_python_pkg_info():\n            pkgs_by_name[pkg[\"name\"]].append(pkg)\n        out = {}\n        for name in sorted(pkgs_by_name.keys()):\n            out[name] = _resolve_latest_pkg(pkgs_by_name[name])\n        with open(out_file, \"w\") as out_handle:\n            yaml.safe_dump(out, out_handle, default_flow_style=False, allow_unicode=False)\n    return out_file\n\n# ## Debian packages\n\ndef _get_pkg_popcon():\n    \"\"\"Retrieve popularity information for debian packages.\n    \"\"\"\n    url = \"http://popcon.debian.org/by_vote\"\n    popcon = {}\n    for line in (l for l in urllib.request.urlopen(url) if not l.startswith((\"#\", \"--\"))):\n        parts = line.split()\n        popcon[parts[1]] = int(parts[3])\n    return popcon\n\ndef get_debian_pkg_info(fetch_remote=False):\n    pkg_popcon = _get_pkg_popcon() if fetch_remote else {}\n    cmd = (\"dpkg-query --show --showformat \"\n           \"'${Status}\\t${Package}\\t${Version}\\t${Section}\\t${Homepage}\\t${binary:Summary}\\n'\")\n    for pkg_line in [l for l in subprocess.check_output(cmd, shell=True).decode(errors=\"replace\").split(\"\\n\")\n                     if l.startswith(\"install ok\")]:\n        parts = pkg_line.rstrip(\"\\n\").split(\"\\t\")\n        if len(parts) > 5:\n            pkg = {\"name\": parts[1], \"version\": parts[2],\n                   \"section\": parts[3], \"homepage_uri\": parts[4],\n                   \"description\": parts[5]}\n            if pkg_popcon.get(pkg[\"name\"]):\n                pkg[\"downloads\"] = pkg_popcon.get(pkg[\"name\"], 0)\n            yield pkg\n\ndef write_debian_pkg_info(out_dir, fetch_remote=False):\n    base_sections = set([\"gnome\", \"admin\", \"utils\", \"web\", \"games\",\n                         \"sound\", \"devel\", \"kde\", \"x11\", \"net\", \"text\",\n                         \"graphics\", \"misc\", \"editors\", \"fonts\", \"doc\",\n                         \"mail\", \"otherosfs\", \"video\", \"kernel\",\n                         \"libs\", \"libdevel\", \"comm\", \"metapackages\", \"tex\",\n                         \"introspection\"])\n    for s in list(base_sections):\n        base_sections.add(\"universe/%s\" % s)\n        base_sections.add(\"partner/%s\" % s)\n    out_file = os.path.join(out_dir, \"debian-packages.yaml\")\n    out_base_file = os.path.join(out_dir, \"debian-base-packages.yaml\")\n    try:\n        subprocess.check_call([\"dpkg\", \"--help\"], stdout=subprocess.PIPE)\n        has_dpkg = True\n    except (subprocess.CalledProcessError, OSError):\n        has_dpkg = False\n    if has_dpkg and (not os.path.exists(out_file) or not os.path.exists(out_base_file)):\n        out = {}\n        out_base = {}\n        for pkg in get_debian_pkg_info(fetch_remote):\n            if pkg.get(\"section\") in base_sections:\n                out_base[pkg[\"name\"]] = pkg\n            else:\n                out[pkg[\"name\"]] = pkg\n        with open(out_file, \"w\") as out_handle:\n            yaml.safe_dump(out, out_handle, default_flow_style=False, allow_unicode=False)\n        with open(out_base_file, \"w\") as out_handle:\n            yaml.safe_dump(out_base, out_handle, default_flow_style=False, allow_unicode=False)\n    return out_file\n"
  },
  {
    "path": "cloudbio/package/__init__.py",
    "content": "\"\"\"Install software and configure package managers.\n\"\"\"\nfrom __future__ import print_function\nimport os\n\n\ndef _configure_and_install_native_packages(env, pkg_install):\n    \"\"\"\n    Setups up native package repositories, determines list\n    of native packages to install, and installs them.\n    \"\"\"\n    from fabric.api import env\n    from cloudbio.package import brew\n    from cloudbio.package.deb import (_apt_packages, _add_apt_gpg_keys,\n                                      _setup_apt_automation, _setup_apt_sources)\n    from cloudbio.package.rpm import (_yum_packages, _setup_yum_bashrc,\n                                      _setup_yum_sources)\n\n    home_dir = env.safe_run(\"echo $HOME\")\n    if home_dir:\n        if env.shell_config.startswith(\"~\"):\n            nonhome = env.shell_config.split(\"~/\", 1)[-1]\n            env.shell_config = os.path.join(home_dir, nonhome)\n    if env.distribution in [\"debian\", \"ubuntu\"]:\n        _setup_apt_sources()\n        _setup_apt_automation()\n        _add_apt_gpg_keys()\n        _apt_packages(pkg_install)\n    elif env.distribution in [\"centos\", \"scientificlinux\"]:\n        _setup_yum_sources()\n        _yum_packages(pkg_install)\n        _setup_yum_bashrc()\n    elif env.distribution in [\"arch\", \"suse\"]:\n        pass  # No package support for Arch, SUSE yet\n    elif env.distribution == \"macosx\":\n        brew.install_packages(env, pkg_install)\n    else:\n        raise NotImplementedError(\"Unknown target distribution\")\n\ndef _connect_native_packages(env, pkg_install, lib_install):\n    \"\"\"Connect native installed packages to local versions.\n\n    This helps setup a non-sudo environment to handle software\n    that needs a local version in our non-root directory tree.\n    \"\"\"\n    from fabric.api import env\n    bin_dir = os.path.join(env.system_install, \"bin\")\n    exports = _get_shell_exports(env)\n    path = env.safe_run_output(\"echo $PATH\")\n    comment_line = \"# CloudBioLinux PATH updates\"\n    if not env.safe_contains(env.shell_config, comment_line):\n        env.safe_append(env.shell_config, \"\\n\" + comment_line)\n    if bin_dir not in path and env.safe_exists(env.shell_config):\n        if not env.safe_contains(env.shell_config, exports[\"path\"]):\n            env.safe_append(env.shell_config, exports[\"path\"])\n    if \"python\" in pkg_install and \"python\" in lib_install:\n        _create_local_virtualenv(env.system_install)\n\ndef _get_shell_exports(env):\n    bin_dir = os.path.join(env.system_install, \"bin\")\n    return {\"path\": \"export PATH=%s:$PATH\" % bin_dir}\n\ndef _print_shell_exports(env):\n    \"\"\"Print a set of exports to add to shell in isolated installations.\n    \"\"\"\n    exports = _get_shell_exports(env)\n    print(\"\\nIsolated installation: add the following to your shell to include installed files:\")\n    for e in [\"path\"]:\n        print(exports[e])\n\ndef _create_local_virtualenv(target_dir):\n    \"\"\"Create virtualenv in target directory for non-sudo installs.\n    \"\"\"\n    from fabric.api import cd\n    from cloudbio.custom.shared import _make_tmp_dir\n\n    url = \"https://raw.github.com/pypa/virtualenv/master/virtualenv.py\"\n    if not os.path.exists(os.path.join(target_dir, \"bin\", \"python\")):\n        with _make_tmp_dir() as work_dir:\n            with cd(work_dir):\n                env.safe_run(\"wget --no-check-certificate %s\" % url)\n                env.safe_run(\"python virtualenv.py %s\" % target_dir)\n"
  },
  {
    "path": "cloudbio/package/brew.py",
    "content": "\"\"\"Install packages via the MacOSX Homebrew and Linux Linuxbrew package manager.\nhttps://github.com/mxcl/homebrew\nhttps://github.com/Homebrew/linuxbrew\n\"\"\"\nfrom __future__ import print_function\nimport contextlib\nfrom distutils.version import LooseVersion\nimport os\nimport sys\n\nimport yaml\n\nfrom cloudbio.custom import system, shared\nfrom cloudbio.flavor.config import get_config_file\nfrom cloudbio.fabutils import quiet, find_cmd\nfrom cloudbio.package import cpan\nfrom cloudbio.package.shared import _yaml_to_packages\n\nfrom fabric.api import cd, settings\n\nBOTTLE_URL = \"https://s3.amazonaws.com/cloudbiolinux/brew_bottles/{pkg}-{version}.x86_64-linux.bottle.tar.gz\"\nBOTTLE_SUPPORTED = set([\"isaac-aligner\", \"isaac-variant-caller\", \"cmake\"])\n\ndef install_packages(env, to_install=None, packages=None):\n    \"\"\"Install packages using the home brew package manager.\n\n    Handles upgrading brew, tapping required repositories and installing or upgrading\n    packages as appropriate.\n\n    `to_install` is a CloudBioLinux compatible set of top level items to add,\n    alternatively `packages` is a list of raw package names.\n    \"\"\"\n    config_file = get_config_file(env, \"packages-homebrew.yaml\")\n    if to_install:\n        (packages, _) = _yaml_to_packages(config_file.base, to_install, config_file.dist)\n    # if we have no packages to install, do not try to install or update brew\n    if len(packages) == 0:\n        _remove_old(env, config_file.base)\n        return\n    system.install_homebrew(env)\n    brew_cmd = _brew_cmd(env)\n    formula_repos = [\"homebrew/science\", \"chapmanb/cbl\", \"homebrew/dupes\"]\n    current_taps = set([x.strip() for x in env.safe_run_output(\"%s tap\" % brew_cmd).split()])\n    _safe_update(env, brew_cmd, formula_repos, current_taps)\n    current_taps = set([x.strip() for x in env.safe_run_output(\"%s tap\" % brew_cmd).split()])\n    for repo in formula_repos:\n        if repo not in current_taps:\n            env.safe_run(\"%s tap %s\" % (brew_cmd, repo))\n    env.safe_run(\"%s tap --repair\" % brew_cmd)\n    ipkgs = {\"outdated\": set([x.strip() for x in env.safe_run_output(\"%s outdated\" % brew_cmd).split()]),\n             \"current\": _get_current_pkgs(env, brew_cmd)}\n    _install_brew_baseline(env, brew_cmd, ipkgs, packages)\n    ipkgs = {\"outdated\": set([x.strip() for x in env.safe_run_output(\"%s outdated\" % brew_cmd).split()]),\n             \"current\": _get_current_pkgs(env, brew_cmd)}\n    for pkg_str in packages:\n        _install_pkg(env, pkg_str, brew_cmd, ipkgs)\n    for pkg_str in [\"pkg-config\", \"openssl\", \"cmake\", \"unzip\"]:\n        _safe_unlink_pkg(env, pkg_str, brew_cmd)\n    with open(config_file.base) as in_handle:\n        to_remove = yaml.safe_load(in_handle).get(\"to_remove\", [])\n    for pkg_str in [\"curl\"] + to_remove:\n        _safe_uninstall_pkg(env, pkg_str, brew_cmd)\n\ndef _remove_old(env, config_file):\n    \"\"\"Temporary approach to remove an old brew install migrated to conda packages.\n    \"\"\"\n    brew_cmd = os.path.join(env.system_install, \"bin\", \"brew\")\n    if env.safe_exists(brew_cmd):\n        baseline = [\"pkg-config\", \"openssl\", \"cmake\", \"unzip\", \"curl\"]\n        with open(config_file) as in_handle:\n            to_remove = yaml.safe_load(in_handle).get(\"to_remove\", [])\n        for pkg_str in baseline + to_remove:\n            _safe_uninstall_pkg(env, pkg_str, brew_cmd)\n\ndef _safe_update(env, brew_cmd, formula_repos, cur_taps):\n    \"\"\"Revert any taps if we fail to update due to local changes.\n    \"\"\"\n    with _git_stash(env, brew_cmd):\n        with quiet():\n            with settings(warn_only=True):\n                out = env.safe_run(\"%s update\" % brew_cmd)\n        if out.failed:\n            for repo in formula_repos:\n                if repo in cur_taps:\n                    env.safe_run(\"%s untap %s\" % (brew_cmd, repo))\n            with settings(warn_only=True):\n                out = env.safe_run(\"%s update\" % brew_cmd)\n            if out.failed:\n                print(\"\\n\\nHomebrew update failed.\")\n                print(\"You might need to upgrade git by installing inside bcbio with:\")\n                print(\"'brew install git --env=inherit --ignore-dependences'\\n\\n\")\n\n@contextlib.contextmanager\ndef _git_stash(env, brew_cmd):\n    \"\"\"Perform a safe git stash around an update.\n\n    This circumvents brews internal stash approach which doesn't work on older versions\n    of git and is sensitive to missing config.emails.\n    \"\"\"\n    brew_prefix = env.safe_run_output(\"{brew_cmd} --prefix\".format(**locals()))\n    with cd(brew_prefix):\n        with quiet():\n            with settings(warn_only=True):\n                env.safe_run(\"git config user.email 'stash@brew.sh'\")\n                check_diff = env.safe_run(\"git diff --quiet\")\n                git_version = env.safe_run_output(\"git --version\").strip().split()[-1]\n    if git_version and LooseVersion(git_version) < LooseVersion(\"1.7\"):\n        if check_diff.return_code > 0:\n            with cd(brew_prefix):\n                with settings(warn_only=True):\n                    env.safe_run(\"git stash --quiet\")\n        try:\n            yield None\n        finally:\n            if check_diff.return_code > 0:\n                with cd(brew_prefix):\n                    with settings(warn_only=True):\n                        env.safe_run(\"git stash pop --quiet\")\n    else:\n        yield None\n\ndef _get_current_pkgs(env, brew_cmd):\n    out = {}\n    with quiet():\n        which_out = env.safe_run_output(\"{brew_cmd} list --versions\".format(**locals()))\n    for line in which_out.split(\"\\n\"):\n        if line:\n            parts = line.rstrip().split()\n            if len(parts) == 2:\n                pkg, version = line.rstrip().split()\n                if pkg.endswith(\":\"):\n                    pkg = pkg[:-1]\n                out[pkg] = version\n    return out\n\ndef _safe_unlink_pkg(env, pkg_str, brew_cmd):\n    \"\"\"Unlink packages which can cause issues with a Linux system.\n    \"\"\"\n    with settings(warn_only=True):\n        with quiet():\n            env.safe_run(\"{brew_cmd} unlink {pkg_str}\".format(**locals()))\n\ndef _safe_link_pkg(env, pkg_str, brew_cmd):\n    \"\"\"Link packages required for builds, but not necessarily installed\n    \"\"\"\n    with settings(warn_only=True):\n        with quiet():\n            env.safe_run(\"{brew_cmd} link --overwrite {pkg_str}\".format(**locals()))\n\ndef _safe_uninstall_pkg(env, pkg_str, brew_cmd):\n    \"\"\"Uninstall packages which get pulled in even when unlinked by brew.\n    \"\"\"\n    with settings(warn_only=True):\n        with quiet():\n            env.safe_run(\"{brew_cmd} uninstall {pkg_str}\".format(**locals()))\n\ndef _install_pkg(env, pkg_str, brew_cmd, ipkgs):\n    \"\"\"Install a specific brew package, handling versioning and existing packages.\n    \"\"\"\n    pkg, version, args = _get_pkg_version_args(pkg_str)\n    installed = False\n    if version:\n        _install_pkg_version(env, pkg, args, version, brew_cmd, ipkgs)\n        installed = True\n    if pkg in BOTTLE_SUPPORTED and not env.distribution == \"macosx\":\n        installed = _install_bottle(env, brew_cmd, pkg, ipkgs)\n    if not installed:\n        _install_pkg_latest(env, pkg, args, brew_cmd, ipkgs)\n\ndef _install_pkg_version(env, pkg, args, version, brew_cmd, ipkgs):\n    \"\"\"Install a specific version of a package by retrieving from git history.\n    https://gist.github.com/gcatlin/1847248\n    Handles both global packages and those installed via specific taps.\n    \"\"\"\n    if ipkgs[\"current\"].get(pkg.split(\"/\")[-1]) == version:\n        return\n    if version == \"HEAD\":\n        args = \" \".join(args)\n        brew_install = _get_brew_install_cmd(brew_cmd, env, pkg)\n        env.safe_run(\"{brew_install} {args} --HEAD {pkg}\".format(**locals()))\n    else:\n        raise ValueError(\"Cannot currently handle installing brew packages by version.\")\n        with _git_pkg_version(env, brew_cmd, pkg, version):\n            if pkg.split(\"/\")[-1] in ipkgs[\"current\"]:\n                with settings(warn_only=True):\n                    env.safe_run(\"{brew_cmd} unlink {pkg}\".format(\n                        brew_cmd=brew_cmd, pkg=pkg.split(\"/\")[-1]))\n            # if we have a more recent version, uninstall that first\n            cur_version_parts = env.safe_run_output(\"{brew_cmd} list --versions {pkg}\".format(\n                brew_cmd=brew_cmd, pkg=pkg.split(\"/\")[-1])).strip().split()\n            if len(cur_version_parts) > 1 and LooseVersion(cur_version_parts[1]) > LooseVersion(version):\n                with settings(warn_only=True):\n                    env.safe_run(\"{brew_cmd} uninstall {pkg}\".format(**locals()))\n            env.safe_run(\"{brew_cmd} install {pkg}\".format(**locals()))\n            with settings(warn_only=True):\n                env.safe_run(\"{brew_cmd} switch {pkg} {version}\".format(**locals()))\n    env.safe_run(\"%s link --overwrite %s\" % (brew_cmd, pkg))\n\n@contextlib.contextmanager\ndef _git_pkg_version(env, brew_cmd, pkg, version):\n    \"\"\"Convert homebrew Git to previous revision to install a specific package version.\n    \"\"\"\n    git_cmd = _git_cmd_for_pkg_version(env, brew_cmd, pkg, version)\n    git_fname = git_cmd.split()[-1]\n    brew_prefix = env.safe_run_output(\"{brew_cmd} --prefix\".format(**locals()))\n    if git_fname.startswith(\"{brew_prefix}/Library/Taps/\".format(**locals())):\n        brew_prefix = os.path.dirname(git_fname)\n    try:\n        with cd(brew_prefix):\n            if version != \"HEAD\":\n                env.safe_run(git_cmd)\n        yield\n    finally:\n        # reset Git back to latest\n        with cd(brew_prefix):\n            if version != \"HEAD\":\n                cmd_parts = git_cmd.split()\n                env.safe_run(\"%s reset HEAD %s\" % (cmd_parts[0], cmd_parts[-1]))\n                cmd_parts[2] = \"--\"\n                env.safe_run(\" \".join(cmd_parts))\n\ndef _git_cmd_for_pkg_version(env, brew_cmd, pkg, version):\n    \"\"\"Retrieve git command to check out a specific version from homebrew.\n    \"\"\"\n    git_cmd = None\n    for git_line in env.safe_run_output(\"{brew_cmd} versions {pkg}\".format(**locals())).split(\"\\n\"):\n        if git_line.startswith(version):\n            git_cmd = \" \".join(git_line.rstrip().split()[1:])\n            break\n    if git_cmd is None:\n        raise ValueError(\"Did not find version %s for %s\" % (version, pkg))\n    return git_cmd\n\ndef _latest_pkg_version(env, brew_cmd, pkg, devel=False):\n    \"\"\"Retrieve the latest available version of a package and if it is linked.\n    \"\"\"\n    i = 0\n    version, is_linked = None, False\n    with settings(warn_only=True):\n        info_str = env.safe_run_output(\"{brew_cmd} info {pkg}\".format(**locals()))\n    for i, git_line in enumerate(info_str.split(\"\\n\")):\n        if git_line.strip():\n            if i == 0:\n                _, version_str = git_line.split(\":\")\n                versions = version_str.split(\",\")\n                if devel:\n                    dev_strs = [x for x in versions if x.strip().startswith(\"devel\")]\n                    version = dev_strs[0].split()[-1].strip()\n                else:\n                    version = versions[0].replace(\"(bottled)\", \"\").split()[-1].strip()\n            elif git_line.find(\"Cellar/%s\" % pkg) > 0 and git_line.find(\" files,\") > 0:\n                is_linked = git_line.strip().split()[-1] == \"*\"\n    return version, is_linked\n\ndef _get_brew_install_cmd(brew_cmd, env, pkg):\n    perl_setup = \"export PERL5LIB=%s/lib/perl5:${PERL5LIB}\" % env.system_install\n    compiler_setup = \"export CC=${CC:-`which gcc`} && export CXX=${CXX:-`which g++`}\"\n    shell_setup = \"export SHELL=${SHELL:-/bin/bash}\"\n    extra_args = \"\"\n    if pkg in [\"cmake\"]:\n        extra_args += \" --without-docs\"\n    if pkg in [\"lumpy-sv\", \"bamtools\", \"freebayes\", \"git\"]:\n        extra_args += \" --ignore-dependencies\"\n    return \"%s && %s && %s && %s install --env=inherit %s\" % (compiler_setup, shell_setup, perl_setup,\n                                                              brew_cmd, extra_args)\n\ndef _install_pkg_latest(env, pkg, args, brew_cmd, ipkgs):\n    \"\"\"Install the latest version of the given package.\n    \"\"\"\n    short_pkg = pkg.split(\"/\")[-1]\n    do_install = True\n    is_linked = True\n    remove_old = False\n    if pkg in ipkgs[\"outdated\"] or short_pkg in ipkgs[\"outdated\"]:\n        remove_old = True\n    elif pkg in ipkgs[\"current\"] or short_pkg in ipkgs[\"current\"]:\n        do_install = False\n        pkg_version, is_linked = _latest_pkg_version(env, brew_cmd, pkg, devel=\"--devel\" in args)\n        cur_version = ipkgs[\"current\"].get(pkg, ipkgs[\"current\"][short_pkg])\n        if cur_version != pkg_version and cur_version.split(\"_\")[0] != pkg_version:\n            remove_old = True\n            do_install = True\n    if do_install:\n        if remove_old:\n            env.safe_run(\"{brew_cmd} remove --force {short_pkg}\".format(**locals()))\n        flags = \" \".join(args)\n        with settings(warn_only=True):\n            cmd = \"%s %s %s\" % (_get_brew_install_cmd(brew_cmd, env, pkg), flags, pkg)\n            with _custom_unlink(env, brew_cmd, pkg):\n                result = env.safe_run_output(cmd)\n            if result.failed and not result.find(\"Could not symlink\") > 0:\n                sys.tracebacklimit = 1\n                raise ValueError(\"Failed to install brew formula: %s\\n\" % pkg +\n                                 \"To debug, please try re-running the install command with verbose output:\\n\" +\n                                 cmd.replace(\"brew install\", \"brew install -v\"))\n        env.safe_run(\"%s link --overwrite %s\" % (brew_cmd, pkg))\n    # installed but not linked\n    elif not is_linked:\n        env.safe_run(\"%s link --overwrite %s\" % (brew_cmd, pkg))\n\n@contextlib.contextmanager\ndef _custom_unlink(env, brew_cmd, pkg):\n    \"\"\"Handle custom unlinking of packages that can break builds of others.\n\n    Does a temporary unlink and relink of packages while building.\n    \"\"\"\n    unlinks = {\"lumpy-sv\": [\"bamtools\"]}\n    for upkg in unlinks.get(pkg, []):\n        _safe_unlink_pkg(env, upkg, brew_cmd)\n    try:\n        yield None\n    finally:\n        for upkg in unlinks.get(pkg, []):\n            with settings(warn_only=True):\n                with quiet():\n                    env.safe_run(\"%s link --overwrite %s\" % (brew_cmd, upkg))\n\ndef _get_pkg_version_args(pkg_str):\n    \"\"\"Uses Python style package==0.1 version specifications and args separated with ';'\n    \"\"\"\n    arg_parts = pkg_str.split(\";\")\n    if len(arg_parts) == 1:\n        args = []\n    else:\n        pkg_str = arg_parts[0]\n        args = arg_parts[1:]\n    parts = pkg_str.split(\"==\")\n    if len(parts) == 1:\n        return parts[0], None, args\n    else:\n        assert len(parts) == 2\n        name, version = parts\n        return name, version, args\n\ndef _install_bottle(env, brew_cmd, pkg, ipkgs):\n    \"\"\"Install Linux bottles for brew packages that can be tricky to build.\n    \"\"\"\n    if env.distribution == \"macosx\":  # Only Linux bottles, build away on Mac\n        return False\n    pkg_version, is_linked = _latest_pkg_version(env, brew_cmd, pkg)\n    install_version = ipkgs[\"current\"].get(pkg)\n    if pkg_version == install_version:  # Up to date\n        if not is_linked:\n            env.safe_run(\"%s link --overwrite %s\" % (brew_cmd, pkg))\n        return True\n    elif install_version or pkg in ipkgs[\"outdated\"]:\n        env.safe_run(\"{brew_cmd} remove --force {pkg}\".format(**locals()))\n    url = BOTTLE_URL.format(pkg=pkg, version=pkg_version)\n    brew_cachedir = env.safe_run_output(\"%s --cache\" % brew_cmd)\n    brew_cellar = os.path.join(env.safe_run_output(\"%s --prefix\" % brew_cmd), \"Cellar\")\n    with quiet():\n        env.safe_run(\"mkdir -p %s\" % brew_cellar)\n    out_file = os.path.join(brew_cachedir, os.path.basename(url))\n    if env.safe_exists(out_file):\n        env.safe_run(\"rm -f %s\" % out_file)\n    bottle_file = shared._remote_fetch(env, url, out_file=out_file,\n                                       allow_fail=True, samedir=True)\n    if bottle_file:\n        with cd(brew_cellar):\n            env.safe_run(\"tar -xf %s\" % bottle_file)\n        env.safe_run(\"%s link --overwrite %s\" % (brew_cmd, pkg))\n        return True\n    else:\n        return False\n\ndef _install_brew_baseline(env, brew_cmd, ipkgs, packages):\n    \"\"\"Install baseline brew components not handled by dependency system.\n\n    - Installation of required Perl libraries.\n    - Upgrades any package dependencies\n    \"\"\"\n    for dep in [\"openssl\"]:\n        _safe_link_pkg(env, dep, brew_cmd)\n    for dep in [\"expat\", \"pkg-config\", \"xz\", \"unzip\"]:\n        _install_pkg(env, dep, brew_cmd, ipkgs)\n    # check if we have an older git and need to install it from brew\n    git_version = None\n    with quiet():\n        with settings(warn_only=True):\n            git_version = env.safe_run_output(\"git --version\").strip().split()[-1]\n    if git_version and LooseVersion(git_version) < LooseVersion(\"1.7\"):\n        _install_pkg(env, \"git\", brew_cmd, ipkgs)\n    for dep in [\"sambamba\"]:  # Avoid conflict with homebrew-science sambamba\n        env.safe_run(\"{brew_cmd} remove --force {dep}\".format(**locals()))\n    for dependency in [\"htslib\"]:\n        if dependency in packages:\n            if (dependency in ipkgs[\"outdated\"] or \"chapmanb/cbl/%s\" % dependency in ipkgs[\"outdated\"]\n                  or dependency not in ipkgs[\"current\"]):\n                _install_pkg_latest(env, dependency, [], brew_cmd, ipkgs)\n    if \"cpanminus\" in packages:\n        _install_pkg_latest(env, \"cpanminus\", [], brew_cmd, ipkgs)\n        _install_pkg_latest(env, \"samtools-library-0.1\", [], brew_cmd, ipkgs)\n        cpan.install_packages(env)\n    # Ensure paths we may have missed on install are accessible to regular user\n    if env.use_sudo:\n        paths = [\"share\", \"share/java\"]\n        for path in paths:\n            with quiet():\n                test_access = env.safe_run(\"test -d %s/%s && test -O %s/%s\" % (env.system_install, path,\n                                                                               env.system_install, path))\n            if test_access.failed and env.safe_exists(\"%s/%s\" % (env.system_install, path)):\n                env.safe_sudo(\"chown %s %s/%s\" % (env.user, env.system_install, path))\n\ndef _brew_cmd(env):\n    \"\"\"Retrieve brew command for installing homebrew packages.\n    \"\"\"\n    cmd = find_cmd(env, \"brew\", \"--version\")\n    if cmd is None:\n        raise ValueError(\"Did not find working installation of Linuxbrew/Homebrew. \"\n                         \"Please check if you have ruby available.\")\n    else:\n        return cmd\n"
  },
  {
    "path": "cloudbio/package/conda.py",
    "content": "\"\"\"Install packages via the Conda package manager: http://conda.pydata.org/\n\"\"\"\nfrom __future__ import print_function\nimport collections\nimport json\nimport os\nimport shutil\nimport subprocess\nimport yaml\n\nfrom cloudbio.package.shared import _yaml_to_packages\n\nENV_PY_VERSIONS = collections.defaultdict(lambda: \"python=3\")\nENV_PY_VERSIONS[None] = \"python=3\"\nENV_PY_VERSIONS[\"python2\"] = \"python=2\"\nENV_PY_VERSIONS[\"python3.6\"] = \"python=3.6\"\nENV_PY_VERSIONS[\"dv\"] = \"python=3.6\"\nENV_PY_VERSIONS[\"samtools0\"] = \"python=3\"\nENV_PY_VERSIONS[\"r35\"] = \"python=3\"\nENV_PY_VERSIONS[\"rbcbiornaseq\"] = \"python=3\"\nENV_PY_VERSIONS[\"htslib1.9\"] = \"python=3\"\nENV_PY_VERSIONS[\"htslib1.10\"] = \"python=3\"\nENV_PY_VERSIONS[\"htslib1.11\"] = \"python=3\"\nENV_PY_VERSIONS[\"htslib1.12\"] = \"python=3\"\nENV_PY_VERSIONS[\"htslib1.12_py3.9\"] = \"python=3.9\"\nENV_PY_VERSIONS[\"java\"] = \"python=3\"\nENV_PY_VERSIONS[\"bwakit\"] = \"python=3\"\n\ndef install_packages(env, to_install=None, packages=None):\n    \"\"\"Old installation, based on pre-configured fabric inputs.\n    \"\"\"\n    from cloudbio.flavor.config import get_config_file\n    from cloudbio.custom import shared\n\n    if shared._is_anaconda(env):\n        conda_bin = shared._conda_cmd(env)\n        if hasattr(env, \"conda_yaml\"):\n            Config = collections.namedtuple(\"Config\", \"base dist\")\n            config_file = Config(base=env.conda_yaml, dist=None)\n        else:\n            config_file = get_config_file(env, \"packages-conda.yaml\")\n        install_in(conda_bin, env.system_install, config_file.base, packages)\n\ndef _install_env_pkgs(env_name, env_packages, conda_bin, conda_envs, channels):\n    \"\"\"Install packages into the given environment.\n\n    Uses mamba for initial install for speed, following by conda for completeness.\n\n    TODO: currently duplicates mamba base code in _initial_base_install to make it\n    easy to remove or roll back general mamba usage. We can refactor _initial_base_install\n    in favor of this after further testing.\n\n    conda_bin could refer to mamba\n    \"\"\"\n    mamba_bin = os.path.join(os.path.dirname(conda_bin), \"mamba\")\n    conda_bin = os.path.join(os.path.dirname(mamba_bin), \"conda\")\n    if env_name:\n        assert env_name in conda_envs, (env_name, conda_envs)\n        env_str = \"-n %s\" % env_name\n    else:\n        env_str = \"\"\n    pkgs_str = \" \".join([\"'%s'\" % x for x in sorted(env_packages)])\n    py_version = ENV_PY_VERSIONS[env_name]\n    if \"deepvariant\" in env_packages:\n        # Ignore /etc/boto.cfg which creates conflicts with conda gsutils\n        # https://github.com/GoogleCloudPlatform/gsutil/issues/516\n        exports = \"export BOTO_CONFIG=/ignoreglobal && \"\n    else:\n        exports = \"\"\n    if os.path.exists(mamba_bin):\n        try:\n            subprocess.check_call(\"{mamba_bin} install -q -y {env_str} {channels} \"\n                                  \"{py_version} {pkgs_str}\".format(**locals()), shell=True)\n        except subprocess.CalledProcessError:\n            # Fall back to standard conda install when we have system specific issues\n            # https://github.com/bcbio/bcbio-nextgen/issues/2871\n            subprocess.check_call(\"{exports}{conda_bin} install -q -y {env_str} {channels} \"\n                                \"{py_version} {pkgs_str}\".format(**locals()), shell=True)\n    conda_pkg_list = json.loads(subprocess.check_output(\n        \"{conda_bin} list --json {env_str} -q\".format(**locals()), shell=True))\n    return conda_pkg_list\n\ndef install_in(conda_bin, system_installdir, config_file=None, packages=None):\n    \"\"\"Install packages inside a given anaconda directory.\n\n    New approach, local only and not dependent on fabric.\n    conda_bin could refer to mamba\n    \"\"\"\n    if config_file is None and packages is None:\n        packages = []\n        check_channels = []\n    else:\n        (packages, _) = _yaml_to_packages(config_file)\n        with open(config_file) as in_handle:\n            check_channels = yaml.safe_load(in_handle).get(\"channels\", [])\n    channels = \" \".join([\"-c %s\" % x for x in check_channels])\n    conda_envs = _create_environments(conda_bin, packages)\n    for env_dir in conda_envs.values():\n        _clean_environment(env_dir)\n    conda_info = json.loads(subprocess.check_output(\"{conda_bin} info --json -q\".format(**locals()), shell=True))\n    # Uninstall old R packages that clash with updated versions\n    # Temporary fix to allow upgrades from older versions that have migrated\n    # r-tximport is now bioconductor-tximport\n    # py2cairo is incompatible with r 3.4.1+\n    problems = [\"r-tximport\", \"py2cairo\"]\n    # Add packages migrated into separate environments, like python2\n    for env_name, env_packages in _split_by_condaenv(packages):\n        if env_name:\n            problems += env_packages\n    if problems:\n        print(\"Checking for problematic or migrated packages in default environment\")\n        cur_packages = [x[\"name\"] for x in\n                        json.loads(subprocess.check_output(\"%s list --json -q\" % (conda_bin), shell=True))\n                        if x[\"name\"] in problems and x[\"channel\"] in check_channels]\n        if cur_packages:\n            print(\"Found packages that moved from default environment: %s\" % \", \".join(cur_packages))\n            problems = \" \".join(cur_packages)\n            subprocess.check_call(\"{conda_bin} remove {channels} -y {problems}\".format(**locals()), shell=True)\n    _initial_base_install(conda_bin, [ps for (n, ps) in _split_by_condaenv(packages) if n is None][0],\n                          check_channels)\n    # install our customized packages\n    # don't link samtools from python3.6, use samtools from anaconda/bin\n    # don't link tabix and bgzip\n    dont_link = [\"samtools\", \"htslib\"]\n    if len(packages) > 0:\n        for env_name, env_packages in _split_by_condaenv(packages):\n            print(\"# Installing into conda environment %s: %s\" % (env_name or \"default\", \", \".join(env_packages)))\n            conda_pkg_list = _install_env_pkgs(env_name, env_packages, conda_bin, conda_envs, channels)\n            for package in env_packages:\n                package_base = package.split(\"=\")[0].split(\">\")[0]\n                if not package_base in dont_link:\n                    _link_bin(package, system_installdir, conda_info, conda_bin, conda_pkg_list,\n                              conda_envdir=conda_envs.get(env_name))\n    conda_pkg_list = json.loads(subprocess.check_output(\"{conda_bin} list --json -q\".format(**locals()), shell=True))\n    for pkg in [\"python\", \"conda\", \"pip\"]:\n        _link_bin(pkg, system_installdir, conda_info, conda_bin, conda_pkg_list, files=[pkg], prefix=\"bcbio_\")\n        \n    # link packages which avoided _link_bin\n    exception_packages = {\"python2\": \"python2\",\n                          \"vt\": \"python3.6\"}\n    for package in exception_packages:\n        env_name = exception_packages[package]\n        source_file = os.path.realpath(os.path.join(conda_envs.get(env_name), \"bin\", package))\n        # not realpath, because realpath would go to the source_file if exists\n        dest_file = os.path.join(system_installdir, \"bin\", package)\n        _do_link(source_file, dest_file)\n\ndef _initial_base_install(conda_bin, env_packages, check_channels):\n    \"\"\"Provide a faster initial installation of base packages, avoiding dependency issues.\n\n    Uses mamba (https://github.com/QuantStack/mamba) to provide quicker package resolution\n    and avoid dependency conflicts with base install environment. Bootstraps the initial\n    installation of all tools when key inputs that cause conflicts are missing.\n\n    TODO: we could remove mamba package running code here in favor of _install_env_pkgs general\n    mamba usage once that is further tested.\n    \"\"\"\n    initial_package_targets = {None: [\"r-base\"]}\n    env_name = None\n    env_str = \"\"\n    channels = \" \".join([\"-c %s\" % x for x in check_channels])\n    cur_ps = [x[\"name\"] for x in\n              json.loads(subprocess.check_output(\"{conda_bin} list --json {env_str} -q\".format(**locals()), shell=True))\n              if x[\"channel\"] in check_channels]\n    have_package_targets = env_name in initial_package_targets and any([p for p in cur_ps\n                                                                        if p in initial_package_targets[env_name]])\n    if not have_package_targets:\n        print(\"Initalling initial set of packages for %s environment with mamba\" % (env_name or \"default\"))\n        py_version = ENV_PY_VERSIONS[env_name]\n        pkgs_str = \" \".join([\"'%s'\" % x for x in sorted(env_packages)])\n        if \"mamba\" not in cur_ps:\n            subprocess.check_call(\"{conda_bin} install -y {env_str} {channels} \"\n                                  \"{py_version} mamba\".format(**locals()), shell=True)\n        mamba_bin = os.path.join(os.path.dirname(conda_bin), \"mamba\")\n        pkgs_str = \" \".join([\"'%s'\" % x for x in sorted(env_packages)])\n        # Skip in favor of _install_env_pkgs, can be removed later after testing\n        if False:\n            try:\n                subprocess.check_call(\"{mamba_bin} install -y {env_str} {channels} \"\n                                      \"{py_version} {pkgs_str}\".format(**locals()), shell=True)\n            except subprocess.CalledProcessError:\n                # Fall back to standard conda install when we have system specific issues\n                # https://github.com/bcbio/bcbio-nextgen/issues/2871\n                pass\n\ndef _link_bin(package, system_installdir, conda_info, conda_bin, conda_pkg_list, files=None,\n              prefix=\"\", conda_env=None, conda_envdir=None):\n    \"\"\"Link files installed in the bin directory into the install directory.\n\n    This is imperfect but we're trying not to require injecting everything in the anaconda\n    directory into a user's path.\n    \"\"\"\n    package = package.split(\"=\")[0].split(\">\")[0]\n    final_bindir = os.path.join(system_installdir, \"bin\")\n    if conda_envdir:\n        base_bindir = os.path.join(conda_envdir, \"bin\")\n    else:\n        base_bindir = os.path.dirname(conda_bin)\n    # resolve any symlinks in the final and base heirarchies\n    final_bindir = subprocess.check_output(\"cd %s && pwd -P\" % final_bindir, shell=True).decode().strip()\n    base_bindir = subprocess.check_output(\"cd %s && pwd -P\" % base_bindir, shell=True).decode().strip()\n    for pkg_subdir in [x for x in conda_pkg_list if x[\"name\"] == package]:\n        pkg_subdir = pkg_subdir[\"dist_name\"].split(\"::\")[-1]\n        for pkg_dir in conda_info[\"pkgs_dirs\"]:\n            pkg_bindir = os.path.join(os.path.realpath(pkg_dir), pkg_subdir, \"bin\")\n            python_bindir = os.path.join(os.path.dirname(pkg_bindir), \"python-scripts\")\n            if (os.path.commonprefix([pkg_bindir, base_bindir]).find(\"anaconda\") > 0 and\n                    (os.path.exists(python_bindir) or os.path.exists(pkg_bindir))):\n                if not files:\n                    if not os.path.exists(python_bindir):\n                        python_bindir = \"\"\n                    if not os.path.exists(pkg_bindir):\n                        pkg_bindir = \"\"\n                    files = subprocess.check_output(\"ls -1 {pkg_bindir} {python_bindir}\"\n                                                    .format(**locals()), shell=True).decode().split()\n                for fname in files:\n                    # symlink to the original file in the /anaconda/bin directory\n                    # this could be a hard or soft link\n                    base_fname = os.path.join(base_bindir, fname)\n                    if os.path.exists(base_fname) and os.path.lexists(base_fname):\n                        _do_link(base_fname,\n                                 os.path.join(final_bindir, \"%s%s\" % (prefix, fname)))\n\ndef _do_link(orig_file, final_file):\n    \"\"\"Perform a soft link of the original file into the final location.\n\n    We need the symlink to point to /anaconda/bin directory, not the real location\n    in the pkgs directory so conda can resolve LD_LIBRARY_PATH and the interpreters.\n    \"\"\"\n    needs_link = True\n    # working symlink, check if already in the right place or remove it\n    if os.path.exists(final_file):\n        if (os.path.realpath(final_file) == os.path.realpath(orig_file) and\n              orig_file == os.path.normpath(os.path.join(os.path.dirname(final_file), os.readlink(final_file)))):\n            needs_link = False\n        else:\n            os.remove(final_file)\n    # broken symlink\n    elif os.path.lexists(final_file):\n        os.unlink(final_file)\n    if needs_link:\n        os.symlink(os.path.relpath(orig_file, os.path.dirname(final_file)), final_file)\n\ndef _split_by_condaenv(packages):\n    \"\"\"Split packages into those requiring special conda environments.\n    \"\"\"\n    out = collections.defaultdict(list)\n    envs = set([])\n    for p in packages:\n        parts = p.split(\";\")\n        name = parts[0]\n        metadata = parts[1:]\n        condaenv = None\n        for k, v in [x.split(\"=\") for x in metadata]:\n            if k == \"env\":\n                condaenv = v\n        envs.add(condaenv)\n        out[condaenv].append(name)\n    envs = [None] + sorted([x for x in list(envs) if x])\n    return [(e, out[e]) for e in envs]\n\ndef _get_conda_envs(conda_bin):\n    info = json.loads(subprocess.check_output(\"{conda_bin} info --envs --json -q\".format(**locals()), shell=True))\n    return [e for e in info[\"envs\"] if e.startswith(info[\"conda_prefix\"])]\n\ndef _create_environments(conda_bin, packages):\n    \"\"\"Creates custom local environments that conflict with global dependencies.\n\n    Available environments:\n\n    - python2 -- tools that require python 2 and are not compatible with python3.\n      The goal is to move all other installs into a default python 3 base environment.\n    - python3 -- support tools that require python 3. This will get deprecated but for\n      and removed as we move to an all python3 install. For now it collects tools that\n      require 3 or some other specific requirements.\n    - samtools0 -- For tools that require older samtools 0.1.19\n    - dv -- DeepVariant, which requires a specific version of numpy and tensorflow\n    - r36 -- R3.6 for PureCN\n    - htslib1.10 -- htslib 1.10 for mosdepth and other packages that require it until bioconda\n      switches off of 1.9.\n    - htslib1.11 -- htslib 1.11 for scramble until bioconda switched off of 1.9\n    - bwakit -- requires a specific (old) version of samblaster\n    \"\"\"\n    env_names = set([e for e, ps in _split_by_condaenv(packages) if e])\n    out = {}\n    conda_envs = _get_conda_envs(conda_bin)\n    for addenv in [\"python3.6\", \"samtools0\", \"dv\", \"python2\", \"r35\", \n                   \"htslib1.9\", \"htslib1.10\", \"htslib1.11\", \"htslib1.12\", \"htslib1.12_py3.9\", \n                   \"bwakit\", \"java\", \"rbcbiornaseq\"]:\n        if addenv in env_names:\n            if not any(x.endswith(\"/%s\" % addenv) for x in conda_envs):\n                print(\"Creating conda environment: %s\" % addenv)\n                py_version = ENV_PY_VERSIONS[addenv]\n                subprocess.check_call(\"{conda_bin} create -q --no-default-packages -y --name {addenv} {py_version} nomkl\"\n                                      .format(**locals()), shell=True)\n                conda_envs = _get_conda_envs(conda_bin)\n            out[addenv] = [x for x in conda_envs if x.endswith(\"/%s\" % addenv)][0]\n    return out\n\ndef _clean_environment(env_dir):\n    \"\"\"Remove problem elements in environmental directories.\n\n    - Get rid of old history comment lines that cause parsing failures:\n      https://github.com/bcbio/bcbio-nextgen/issues/2431\n    \"\"\"\n    history_file = os.path.join(env_dir, \"conda-meta\", \"history\")\n    if os.path.exists(history_file):\n        has_problem = False\n        cleaned_lines = []\n        with open(history_file) as in_handle:\n            for line in in_handle:\n                # Remove lines like `# create specs:` which have no information after colon\n                if line.startswith(\"#\") and len([x for x in line.strip().split(\":\") if x]) == 1:\n                    has_problem = True\n                else:\n                    cleaned_lines.append(line)\n        if has_problem:\n            shutil.copy(history_file, history_file + \".orig\")\n            with open(history_file, \"w\") as out_handle:\n                for line in cleaned_lines:\n                    out_handle.write(line)\n"
  },
  {
    "path": "cloudbio/package/cpan.py",
    "content": "\"\"\"Install perl packages using CPAN and cpanminus (cpanm).\n\"\"\"\nimport os\n\nfrom fabric.api import cd, settings\n\nfrom cloudbio.flavor.config import get_config_file\nfrom cloudbio.fabutils import find_cmd\nfrom cloudbio.package.shared import _yaml_to_packages\nfrom cloudbio.custom import shared as cshared\n\ndef install_packages(env):\n    config_file = get_config_file(env, \"perl-libs.yaml\")\n    (packages, _) = _yaml_to_packages(config_file.base, subs_yaml_file=config_file.dist, namesort=False)\n    cpanm_cmd = find_cmd(env, \"cpanm\", \"--version\")\n    for package in packages:\n        if package.count(\"==\") > 1:\n            _install_from_url(env, cpanm_cmd, package)\n        else:\n            _install_from_cpan(env, cpanm_cmd, package)\n\ndef _install_from_cpan(env, cpanm_cmd, package):\n    \"\"\"Install from CPAN using cpanm, handling special arguments.\n\n    The simplest input string is just a package to install (like XML::Simple) but\n    users can also specify build arguments and exports as additional items separated\n    by ';'\n    \"\"\"\n    parts = package.split(\";\")\n    if len(parts) == 1:\n        perl_lib = parts[0]\n        args = \"\"\n        exports = []\n    elif len(parts) == 2:\n        perl_lib, args = parts\n        exports = []\n    else:\n        perl_lib, args = parts[:2]\n        exports = parts[2:]\n    export_strs = []\n    for export in exports:\n        export_strs.append(\"export \" + export.format(system_install=env.system_install))\n    export = \" && \".join(export_strs) + \" && \" if export_strs else \"\"\n    build_args = (\"--build-args='%s'\" % args) if args else \"\"\n    env.safe_run(\"%s %s -i --notest --local-lib=%s %s '%s'\" % (export, cpanm_cmd, env.system_install,\n                                                                build_args, perl_lib))\n\ndef _install_from_url(env, cpanm_cmd, package):\n    \"\"\"Check version of a dependency and download and install with cpanm if not up to date.\n\n    Packages installed via URL have the package name, target version and URL separated\n    with '=='. They can also optionally have a build directory or dependency to remove.\n    \"\"\"\n    parts = package.split(\"==\")\n    package, target_version, url = parts[:3]\n    args = {}\n    if len(parts) > 3:\n        for key, value in (x.split(\"=\") for x in parts[3:]):\n            args[key] = value\n    with settings(warn_only=True):\n        cur_version = env.safe_run_output(\"export PERL5LIB=%s/lib/perl5:${PERL5LIB} && \" % env.system_install +\n                                          \"\"\"perl -le 'eval \"require $ARGV[0]\" and print $ARGV[0]->VERSION' %s\"\"\"\n                                          % package)\n    if cur_version != target_version:\n        with cshared._make_tmp_dir() as work_dir:\n            with cd(work_dir):\n                dl_dir = cshared._fetch_and_unpack(url)\n                if args.get(\"build\"):\n                    dl_dir = os.path.join(dl_dir, args[\"build\"])\n                with cd(dl_dir):\n                    if args.get(\"depremove\"):\n                        for fname in [\"Makefile.PL\", \"MYMETA.json\", \"MYMETA.yml\"]:\n                            env.safe_run(r\"\"\"sed -i.bak -e '/^.*%s.*/s/^/#/' %s\"\"\" % (args[\"depremove\"], fname))\n                    env.safe_run(\"%s -i --notest --local-lib=%s .\" % (cpanm_cmd, env.system_install))\n"
  },
  {
    "path": "cloudbio/package/deb.py",
    "content": "\"\"\"\nAutomated installation on debian package systems with apt.\n\"\"\"\nfrom fabric.api import *\nfrom fabric.contrib.files import *\n\nfrom cloudbio.package.shared import _yaml_to_packages\nfrom cloudbio.flavor.config import get_config_file\n\n\ndef _apt_packages(to_install=None, pkg_list=None):\n    \"\"\"\n    Install packages available via apt-get.\n    Note that ``to_install`` and ``pkg_list`` arguments cannot be used simultaneously.\n\n    :type to_install:  list\n    :param to_install: A list of strings (ie, groups) present in the ``main.yaml``\n                       config file that will be used to filter out the specific\n                       packages to be installed.\n\n    :type pkg_list:  list\n    :param pkg_list: An explicit list of packages to install. No other files,\n                     flavors are considered.\n    \"\"\"\n    if \"minimal\" not in env.flavor.short_name:\n        env.logger.info(\"Update the system\")\n        with settings(warn_only=True):\n            env.safe_sudo(\"apt-get update\")\n    if to_install is not None:\n        config_file = get_config_file(env, \"packages.yaml\")\n        if \"minimal\" not in env.flavor.name and \"minimal\" not in env.flavor.short_name:\n            env.flavor.apt_upgrade_system(env=env)\n        (packages, _) = _yaml_to_packages(config_file.base, to_install, config_file.dist)\n        packages = env.flavor.rewrite_config_items(\"packages\", packages)\n    elif pkg_list is not None:\n        env.logger.info(\"Will install specific packages: {0}\".format(pkg_list))\n        packages = pkg_list\n    else:\n        raise ValueError(\"Need a file with packages or a list of packages\")\n    # A single line install is much faster - note that there is a max\n    # for the command line size, so we do 30 at a time\n    group_size = 30\n    i = 0\n    env.logger.info(\"Installing %i packages\" % len(packages))\n    while i < len(packages):\n        env.logger.info(\"Package install progress: {0}/{1}\".format(i, len(packages)))\n        env.safe_sudo(\"apt-get -y --force-yes install %s\" % \" \".join(packages[i:i + group_size]))\n        i += group_size\n    env.safe_sudo(\"apt-get clean\")\n\ndef _add_apt_gpg_keys():\n    \"\"\"Adds GPG keys from all repositories\n    \"\"\"\n    env.logger.info(\"Update GPG keys for repositories\")\n    standalone = [\n        \"http://archive.cloudera.com/debian/archive.key\",\n        'http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc',\n        'https://www.postgresql.org/media/keys/ACCC4CF8.asc',\n    ]\n    keyserver = [\n            (\"keyserver.ubuntu.com\", \"7F0CEB10\"),\n            (\"keyserver.ubuntu.com\", \"E084DAB9\"),\n            (\"keyserver.ubuntu.com\", \"D018A4CE\"),\n            (\"keyserver.ubuntu.com\", \"D67FC6EAE2A11821\"),\n        ]\n    standalone, keyserver = env.flavor.rewrite_apt_keys(standalone, keyserver)\n    for key in standalone:\n        with settings(warn_only=True):\n            env.safe_sudo(\"wget -q -O- %s | apt-key add -\" % key)\n    for url, key in keyserver:\n        with settings(warn_only=True):\n            env.safe_sudo(\"apt-key adv --keyserver %s --recv %s\" % (url, key))\n    with settings(warn_only=True):\n        env.safe_sudo(\"apt-get update\")\n        env.safe_sudo(\"sudo apt-get install -y --force-yes bio-linux-keyring\")\n\ndef _setup_apt_automation():\n    \"\"\"Setup the environment to be fully automated for tricky installs.\n\n    Sun Java license acceptance:\n    http://www.davidpashley.com/blog/debian/java-license\n\n    MySQL root password questions; install with empty root password:\n    http://snowulf.com/archives/540-Truly-non-interactive-unattended-apt-get-install.html\n\n    Postfix, setup for no configuration. See more on issues here:\n    http://www.uluga.ubuntuforums.org/showthread.php?p=9120196\n    \"\"\"\n    interactive_cmd = \"export DEBIAN_FRONTEND=noninteractive\"\n    if not env.safe_contains(env.shell_config, interactive_cmd):\n        env.safe_append(env.shell_config, interactive_cmd)\n    # Remove interactive checks in .bashrc which prevent\n    # bash customizations\n    env.safe_comment(env.shell_config, \"^[ ]+\\*\\) return;;$\")\n    package_info = [\n            \"postfix postfix/not_configured boolean true\",\n            \"postfix postfix/main_mailer_type select 'No configuration'\",\n            \"mysql-server-5.1 mysql-server/root_password string '(password omitted)'\",\n            \"mysql-server-5.1 mysql-server/root_password_again string '(password omitted)'\",\n            \"mysql-server-5.5 mysql-server/root_password string '(password omitted)'\",\n            \"mysql-server-5.5 mysql-server/root_password_again string '(password omitted)'\",\n            \"sun-java6-jdk shared/accepted-sun-dlj-v1-1 select true\",\n            \"sun-java6-jre shared/accepted-sun-dlj-v1-1 select true\",\n            \"sun-java6-bin shared/accepted-sun-dlj-v1-1 select true\",\n            \"grub-pc grub2/linux_cmdline string ''\",\n            \"grub-pc grub-pc/install_devices_empty boolean true\",\n            \"acroread acroread/default-viewer boolean false\",\n            \"rabbitmq-server rabbitmq-server/upgrade_previous note\",\n            \"condor condor/wantdebconf boolean false\",\n            \"ttf-mscorefonts-installer msttcorefonts/accepted-mscorefonts-eula boolean true\",\n            \"ttf-mscorefonts-installer msttcorefonts/present-mscorefonts-eula note\",\n            \"gdm shared/default-x-display-manager select gdm\",\n            \"lightdm shared/default-x-display-manager select gdm\",\n            \"postfix postfix/mailname string notusedexample.org\",\n            # Work harder to avoid gdm dialogs\n            # https://bugs.launchpad.net/ubuntu/+source/gdm/+bug/1020770\n            \"debconf debconf/priority select critical\"\n            ]\n    package_info = env.flavor.rewrite_apt_automation(package_info)\n    cmd = \"\"\n    for l in package_info:\n        cmd += 'echo \"%s\" | /usr/bin/debconf-set-selections;' % l\n    env.safe_sudo(cmd)\n\ndef _setup_apt_sources():\n    \"\"\"Add sources for retrieving library packages.\n       Using add-apt-repository allows processing PPAs (on Ubuntu)\n\n       This method modifies the apt sources file.\n\n       Uses python-software-properties, which provides an abstraction of apt repositories\n    \"\"\"\n\n    # It may be sudo is not installed - which has fab fail - therefor\n    # we'll try to install it by default, assuming we have root access\n    # already (e.g. on EC2). Fab will fail anyway, otherwise.\n    if not env.safe_exists('/usr/bin/sudo') or not env.safe_exists('/usr/bin/curl'):\n        env.safe_sudo('apt-get update')\n        env.safe_sudo('apt-get -y --force-yes install sudo curl')\n\n    env.logger.debug(\"_setup_apt_sources \" + env.sources_file)\n    env.flavor.check_packages_source()\n    comment = \"# This file was modified for \" + env.flavor.name\n    # Setup apt download policy (default is None)\n    # (see also https://help.ubuntu.com/community/PinningHowto)\n    preferences = env.flavor.rewrite_apt_preferences([])\n    if len(preferences):\n        # make sure it exists, and is empty\n        env.safe_sudo(\"rm -f %s\" % env.apt_preferences_file)\n        env.safe_sudo(\"touch %s\" % env.apt_preferences_file)\n        env.safe_append(env.apt_preferences_file, comment, use_sudo=True)\n        lines = \"\\n\".join(preferences)\n        env.logger.debug(\"Policy %s\" % lines)\n        # append won't duplicate, so we use echo\n        env.safe_sudo(\"/bin/echo -e \\\"%s\\\" >> %s\" % (lines, env.apt_preferences_file))\n        # check there is no error parsing the file\n        env.logger.debug(env.safe_sudo(\"apt-cache policy\"))\n\n    # Make sure a source file exists\n    if not env.safe_exists(env.sources_file):\n        env.safe_sudo(\"touch %s\" % env.sources_file)\n    # Add a comment\n    if not env.safe_contains(env.sources_file, comment):\n        env.safe_append(env.sources_file, comment, use_sudo=True)\n    for source in env.flavor.rewrite_apt_sources_list(env.std_sources):\n        env.logger.debug(\"Source %s\" % source)\n        if source.startswith(\"ppa:\"):\n            env.safe_sudo(\"apt-get install -y --force-yes python-software-properties\")\n            env.safe_sudo(\"add-apt-repository '%s'\" % source)\n        elif (not env.safe_contains(env.sources_file, source) and\n              not env.safe_contains(env.global_sources_file, source)):\n            env.safe_append(env.sources_file, source, use_sudo=True)\n"
  },
  {
    "path": "cloudbio/package/nix.py",
    "content": "\"\"\"Install software with the Nix package manager.\n\"\"\"\nfrom fabric.api import *\nfrom fabric.contrib.files import *\n\nfrom cloudbio.package.shared import _yaml_to_packages\nfrom cloudbio.flavor.config import get_config_file\n\ndef _setup_nix_sources():\n    if env.nixpkgs:\n        target_info = run(\"uname -a\")\n        env.logger.info(\"Target: \"+target_info)\n        # find the target architecture, if not preset\n        if not env.has_key(\"arch\"):\n          env.arch = run(\"uname -m\")\n\n     # first override the path\n        append(\"/root/.bashrc\", \"export PATH=$HOME/.nix-profile/bin:$PATH\", use_sudo=True)\n        env.logger.info(\"Checking NixPkgs\")\n        if not exists(\"/nix/store\"):\n            # first time installation\n            if not exists(\"/usr/bin/nix-env\"):\n               # install Nix (standard Debian release)\n               nix_deb = \"nix_0.16-1_\"+env.arch+\".deb\"\n               if not exists(nix_deb):\n                   # run(\"wget http://hydra.nixos.org/build/565031/download/1/nix_0.16-1_i386.deb\")\n                   run(\"wget http://hydra.nixos.org/build/565048/download/1/\"+nix_deb)\n                   sudo(\"dpkg -i \"+nix_deb)\n        run(\"nix-channel --list\")\n        if run(\"nix-channel --list\") == \"\":\n            # Setup channel\n            sudo(\"nix-channel --add http://nixos.org/releases/nixpkgs/channels/nixpkgs-unstable\")\n        sudo(\"nix-channel --update\")\n        # upgrade Nix to latest (and remove the older version, as it is much slower)\n        sudo(\"nix-env -b -i nix\")\n        if exists(\"/usr/bin/nix-env\"):\n            env.logger.info(\"uninstall older Nix (Debian release)\")\n            sudo(\"dpkg -r nix\")\n\ndef _nix_packages(to_install):\n    \"\"\"Install packages available via nixpkgs (optional)\n    \"\"\"\n    if env.nixpkgs:\n        env.logger.info(\"Update and install NixPkgs packages\")\n        pkg_config_file = get_config_file(env, \"packages-nix.yaml\").base\n        sudo(\"nix-channel --update\")\n        # Retrieve final package names\n        (packages, _) = _yaml_to_packages(pkg_config_file, to_install)\n        packages = env.flavor.rewrite_config_items(\"packages\", packages)\n        for p in packages:\n            sudo(\"nix-env -b -i %s\" % p)\n"
  },
  {
    "path": "cloudbio/package/rpm.py",
    "content": "\"\"\"Automated installation on RPM systems with the yum package manager.\n\"\"\"\nimport itertools\nfrom fabric.api import *\nfrom fabric.contrib.files import *\n\nfrom cloudbio.package.shared import _yaml_to_packages\nfrom cloudbio.flavor.config import get_config_file\n\ndef _yum_packages(to_install):\n    \"\"\"Install rpm packages available via yum.\n    \"\"\"\n    if env.distribution == \"scientificlinux\":\n        package_file = \"packages-scientificlinux.yaml\"\n    else:\n        package_file = \"packages-yum.yaml\"\n    pkg_config = get_config_file(env, package_file).base\n    with settings(warn_only=True):\n        env.safe_sudo(\"yum -y update\")\n        env.safe_sudo(\"yum clean all\")\n        env.safe_sudo(\"yum makecache\")\n        env.safe_sudo(\"yum check-update\")\n    # Retrieve packages to get and install each of them\n    (packages, _) = _yaml_to_packages(pkg_config, to_install)\n    # At this point allow the Flavor to rewrite the package list\n    packages = env.flavor.rewrite_config_items(\"packages\", packages)\n    for package_group in _partition_all(20, packages):\n        env.safe_sudo(\"yum -y install %s\" % \" \".join(package_group))\n\ndef _partition_all(n, iterable):\n    \"\"\"http://stackoverflow.com/questions/5129102/python-equivalent-to-clojures-partition-all\n    \"\"\"\n    it = iter(iterable)\n    while True:\n        chunk = list(itertools.islice(it, n))\n        if not chunk:\n            break\n        yield chunk\n\ndef _setup_yum_bashrc():\n    \"\"\"Fix the user bashrc to update compilers.\n    \"\"\"\n    if env.distribution in [\"centos\"]:\n        to_include = [\"export PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:/usr/lib/pkgconfig\"]\n        # gcc fixes no longer necessary on recent CentOS versions\n        #\"export CC=gcc44\", \"export CXX=g++44\", \"export FC=gfortran44\",\n        fname = env.safe_run_output(\"ls %s\" % env.shell_config)\n        for line in to_include:\n            if not env.safe_contains(fname, line.split(\"=\")[0]):\n                env.safe_append(fname, line)\n\ndef _setup_yum_sources():\n    \"\"\"Add additional useful yum repositories.\n    \"\"\"\n    repos = [\n      \"http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm\",\n      \"http://archive.cloudera.com/redhat/6/x86_64/cdh/cdh3-repository-1.0-1.noarch.rpm\"\n    ]\n    for repo in repos:\n        with settings(warn_only=True):\n            env.safe_sudo(\"rpm -Uvh %s\" % repo)\n"
  },
  {
    "path": "cloudbio/package/shared.py",
    "content": "\"\"\"Shared functionality useful for multiple package managers.\n\"\"\"\nfrom __future__ import print_function\nimport yaml\n\ndef _yaml_to_packages(yaml_file, to_install=None, subs_yaml_file=None, namesort=True, env=None):\n    \"\"\"Read a list of packages from a nested YAML configuration file.\n    \"\"\"\n    print(\"Reading packages from %s\" % yaml_file)\n    with open(yaml_file) as in_handle:\n        full_data = yaml.safe_load(in_handle)\n        if full_data is None:\n            full_data = {}\n    if subs_yaml_file is not None:\n        with open(subs_yaml_file) as in_handle:\n            subs = yaml.safe_load(in_handle)\n    else:\n        subs = {}\n    # filter the data based on what we have configured to install\n    data = [(k, v) for (k, v) in full_data.items()\n            if (to_install is None or k in to_install) and k not in [\"channels\"]]\n    data.sort()\n    packages = []\n    pkg_to_group = dict()\n    while len(data) > 0:\n        cur_key, cur_info = data.pop(0)\n        if cur_info:\n            if isinstance(cur_info, (list, tuple)):\n                packages.extend(_filter_subs_packages(cur_info, subs, namesort))\n                for p in cur_info:\n                    pkg_to_group[p] = cur_key\n            elif isinstance(cur_info, dict):\n                for key, val in cur_info.items():\n                    # if we are okay, propagate with the top level key\n                    if env and key == 'needs_64bit':\n                        if env.is_64bit:\n                            data.insert(0, (cur_key, val))\n                    elif env and key.startswith(env.distribution):\n                        if key.endswith(env.dist_name):\n                            data.insert(0, (cur_key, val))\n                    else:\n                        data.insert(0, (cur_key, val))\n            else:\n                raise ValueError(cur_info)\n    return packages, pkg_to_group\n\ndef _filter_subs_packages(initial, subs, namesort=True):\n    \"\"\"Rename and filter package list with subsitutions; for similar systems.\n    \"\"\"\n    final = []\n    for p in initial:\n        try:\n            new_p = subs[p]\n        except KeyError:\n            new_p = p\n        if new_p:\n            final.append(new_p)\n    if namesort:\n        final.sort()\n    return final\n"
  },
  {
    "path": "cloudbio/utils.py",
    "content": "\"\"\"Utilities for logging and progress tracking.\n\"\"\"\nimport logging\nimport os\nimport sys\n\nfrom fabric.main import load_settings\nfrom fabric.colors import yellow, red, green, magenta\nfrom fabric.api import settings, hide, cd, run\nfrom fabric.contrib.files import exists\n\nfrom cloudbio.distribution import _setup_distribution_environment\nfrom cloudbio.flavor import Flavor\nfrom cloudbio.flavor.config import get_config_file\n\n\nclass ColorFormatter(logging.Formatter):\n    \"\"\" Format log message based on the message level\n        http://stackoverflow.com/questions/1343227/can-pythons-logging-format-be-modified-depending-on-the-message-log-level\n    \"\"\"\n    # Setup formatters for each of the levels\n    err_fmt  = red(\"ERR [%(filename)s(%(lineno)d)] %(msg)s\")\n    warn_fmt  = magenta(\"WARN [%(filename)s(%(lineno)d)]: %(msg)s\")\n    dbg_fmt  = yellow(\"DBG [%(filename)s]: %(msg)s\")\n    info_fmt = green(\"INFO: %(msg)s\")\n\n    def __init__(self, fmt=\"%(name)s %(levelname)s: %(msg)s\"):\n        logging.Formatter.__init__(self, fmt)\n\n    def format(self, record):\n        # Save the original format configured by the user\n        # when the logger formatter was instantiated\n        format_orig = self._fmt\n        # Replace the original format with one customized by logging level\n        if record.levelno == 10:   # DEBUG\n            self._fmt = ColorFormatter.dbg_fmt\n        elif record.levelno == 20: # INFO\n            self._fmt = ColorFormatter.info_fmt\n        elif record.levelno == 30: # WARN\n            self._fmt = ColorFormatter.warn_fmt\n        elif record.levelno == 40: # ERROR\n            self._fmt = ColorFormatter.err_fmt\n        # Call the original formatter class to do the grunt work\n        result = logging.Formatter.format(self, record)\n        # Restore the original format configured by the user\n        self._fmt = format_orig\n        return result\n\ndef _setup_logging(env):\n    env.logger = logging.getLogger(\"cloudbiolinux\")\n    env.logger.setLevel(logging.DEBUG)\n\n    ch = logging.StreamHandler()\n    ch.setLevel(logging.DEBUG)\n    # Use custom formatter\n    ch.setFormatter(ColorFormatter())\n    env.logger.addHandler(ch)\n\ndef _update_biolinux_log(env, target, flavor):\n    \"\"\"Updates the VM so it contains information on the latest BioLinux\n       update in /var/log/biolinux.log.\n\n       The latest information is appended to the file and can be used to see if\n       an installation/update has completed (see also ./test/test_vagrant).\n    \"\"\"\n    if not target:\n        target = env.get(\"target\", None)\n        if not target:\n            target = \"unknown\"\n        else:\n            target = target.name\n    if not flavor:\n        flavor = env.get(\"flavor\", None)\n        if not flavor:\n            flavor = \"unknown\"\n        else:\n            flavor = flavor.name\n    logfn = \"/var/log/biolinux.log\"\n    info = \"Target=\"+target+\"; Flavor=\"+flavor\n    env.logger.info(info)\n    if env.use_sudo:\n        env.safe_sudo(\"date +\\\"%D %T - Updated \"+info+\"\\\" >> \"+logfn)\n\n\ndef _configure_fabric_environment(env, flavor=None, fabricrc_loader=None,\n                                  ignore_distcheck=False):\n    if not fabricrc_loader:\n        fabricrc_loader = _parse_fabricrc\n\n    _setup_flavor(env, flavor)\n    fabricrc_loader(env)\n    # get parameters for distro, packages etc.\n    _setup_distribution_environment(ignore_distcheck=ignore_distcheck)\n    _create_local_paths(env)\n\ndef _setup_flavor(env, flavor):\n    \"\"\"Setup a flavor, providing customization hooks to modify CloudBioLinux installs.\n\n    Specify flavor as a name, in which case we look it up in the standard\n    flavor directory (contrib/flavor/your_flavor), or as an absolute path to a\n    flavor directory outside of cloudbiolinux.\n    \"\"\"\n    env.flavor = Flavor(env)\n    env.flavor_dir = None\n    if flavor:\n        # set the directory for flavor customizations\n        if os.path.isabs(flavor) or os.path.exists(flavor):\n            flavor_dir = flavor\n        else:\n            flavor_dir = os.path.join(os.path.dirname(__file__), '..', 'contrib', 'flavor', flavor)\n        assert os.path.exists(flavor_dir), \\\n            \"Did not find directory {0} for flavor {1}\".format(flavor_dir, flavor)\n        env.flavor_dir = flavor_dir\n        flavor_name = os.path.split(flavor_dir)[-1]\n        # Reinstantiate class if custom defined\n        import cloudbio.flavor\n        try:\n            env.flavor = getattr(cloudbio.flavor, flavor_name.capitalize())(env)\n        except AttributeError:\n            pass\n        env.flavor.name = flavor_name\n        # Load python customizations to base configuration if present\n        for ext in [\"\", \"flavor\"]:\n            py_flavor = os.path.split(os.path.realpath(flavor_dir))[1] + ext\n            flavor_custom_py = os.path.join(flavor_dir, \"{0}.py\".format(py_flavor))\n            if os.path.exists(flavor_custom_py):\n                sys.path.append(flavor_dir)\n                mod = __import__(py_flavor, fromlist=[py_flavor])\n        env.logger.info(env.flavor)\n        env.logger.info(\"This is a %s flavor\" % env.flavor.name)\n\ndef _parse_fabricrc(env):\n    \"\"\"Defaults from fabricrc.txt file; loaded if not specified at commandline.\n    \"\"\"\n    env.config_dir = os.path.join(os.path.dirname(__file__), \"..\", \"config\")\n    env.tool_data_table_conf_file = os.path.join(env.config_dir, \"..\",\n                                                 \"installed_files\",\n                                                 \"tool_data_table_conf.xml\")\n    if not env.has_key(\"distribution\") and not env.has_key(\"system_install\"):\n        env.logger.info(\"Reading default fabricrc.txt\")\n        env.update(load_settings(get_config_file(env, \"fabricrc.txt\").base))\n    if \"shell_config\" not in env:\n        env.shell_config = \"~/.bashrc\"\n    if \"shell\" not in env:\n        env.shell = \"/bin/bash -i -c\"\n\ndef _create_local_paths(env):\n    \"\"\"Expand any paths defined in terms of shell shortcuts (like ~).\n    \"\"\"\n    with settings(hide('warnings', 'running', 'stdout', 'stderr'),\n                  warn_only=True):\n        # This is the first point we call into a remote host - make sure\n        # it does not fail silently by calling a dummy run\n        env.logger.info(\"Now, testing connection to host...\")\n        test = env.safe_run(\"pwd\")\n        # If there is a connection failure, the rest of the code is (sometimes) not\n        # reached - for example with Vagrant the program just stops after above run\n        # command.\n        if test != None:\n            env.logger.info(\"Connection to host appears to work!\")\n        else:\n            raise NotImplementedError(\"Connection to host failed\")\n        env.logger.debug(\"Expand paths\")\n        if \"local_install\" in env:\n            if not env.safe_exists(env.local_install):\n                env.safe_sudo(\"mkdir -p %s\" % env.local_install)\n                user = env.safe_run_output(\"echo $USER\")\n                env.safe_sudo(\"chown -R %s %s\" % (user, env.local_install))\n            with cd(env.local_install):\n                result = env.safe_run_output(\"pwd\")\n                env.local_install = result\n"
  },
  {
    "path": "config/README.md",
    "content": "This directory contains details of the software installed with\n[CloudBioLinux][1]. This is the right place to dig around if you are interested\nin adding packages to the image, or would like to get an overview of what is\ninstalled. The configuration files are written in easily readable [YAML format][2].\n\n* [main.yaml][4] --  High level category view of packages and libraries that are\n  installed with CloudBioLinux.\n\n* [packages.yaml][5] -- A full list of operating system packages that are included,\n  organized by category. The names are standard [Ubuntu APT package names][3].\n\n* [custom.yaml][6] -- Installed software that is not included in the standard\n  package repository. These are often specialized biological packages that have\n  not yet been cleanly packaged. Actual installation code is in the `custom`\n  sub-directory.\n\n* [python-libs.yaml][7], [r-libs.yaml][8], [perl-libs.yaml][9],\n  [ruby-libs.yaml][10],-- Libraries installed for a number of programming\n  languages. These are installed by the language specific library managers\n  (easy\\_install for Python, cpan for Perl, gem for Ruby).\n\n\n[1]: http://cloudbiolinux.com/\n[2]: http://en.wikipedia.org/wiki/YAML\n[3]: https://help.ubuntu.com/community/AptGet/Howto\n[4]: https://github.com/chapmanb/cloudbiolinux/blob/master/config/main.yaml\n[5]: https://github.com/chapmanb/cloudbiolinux/blob/master/config/packages.yaml\n[6]: https://github.com/chapmanb/cloudbiolinux/blob/master/config/custom.yaml\n[7]: https://github.com/chapmanb/cloudbiolinux/blob/master/config/python-libs.yaml\n[8]: https://github.com/chapmanb/cloudbiolinux/blob/master/config/r-libs.yaml\n[9]: https://github.com/chapmanb/cloudbiolinux/blob/master/config/perl-libs.yaml\n[10]: https://github.com/chapmanb/cloudbiolinux/blob/master/config/ruby-libs.yaml\n\n"
  },
  {
    "path": "config/biodata.yaml",
    "content": "---\n# Configuration file defining biological data to retrieve and install.\n# These are stored in an Amazon S3 buckets:\n# https://s3.amazonaws.com/biodata\n# https://s3.amazonaws.com/bcbio-nextgen\n# and retrieved using the data_fabfile Fabric script.\n\n# ## Genome data -- Next generation sequencing and Galaxy\n\n# Details about the genomes you want to include.\n#  Required genome fields (corresponding to Galaxy's tool_data_table_conf.xml columns):\n#    dbkey   - globally unique identifier for the genome (e.g., hg19)\n#    name    - descriptive name for the given genome (to be displayed in Galaxy, e.g., Hsapiens)\n#  Optional genome fields (corresponding to Galaxy's tool_data_table_conf.xml columns):\n#    formats, species, dbkey1, dbkey2, value, path, index\n#  Additional genome fields specific to data deployment:\n#    genome_indexes - list of tool indexes specific to the associated genome (overrides global 'genome_indexes')\ngenomes:\n  - dbkey: phix\n    name: phiX174\n  - dbkey: hg19\n    name: Human (hg19)\n    indexes: [seq, twobit]\n    annotations: [GA4GH_problem_regions, capture_regions, rmsk,\n                  MIG, prioritize, dbsnp, hapmap, 1000g_omni_snps, 1000g_snps,\n                  mills_indels, cosmic, ancestral, clinvar, qsignature, ACMG56_genes, transcripts, RADAR, mirbase,\n                  genesplicer, effects_transcripts, vcfanno, viral, exac, gnomad_exome, esp, 1000g, varpon, topmed,\n                  genotype2phenotype, fusion-blacklist, simple_repeat, purecn_mappability, af_only_gnomad]\n    annotations_available: [battenberg, dbnsfp, dbscsnv, ericscript, gnomad, topmed]\n    validation: [giab-NA12878, platinum-genome-NA12878, giab-NA24385, giab-NA24631, giab-NA24143, giab-NA24149]\n  - dbkey: GRCh37\n    name: Human (GRCh37)\n    indexes: [seq, twobit]\n    annotations: [GA4GH_problem_regions, capture_regions,\n                  MIG, prioritize, dbsnp, hapmap, 1000g_omni_snps, 1000g_snps,\n                  mills_indels, cosmic, ancestral, clinvar, qsignature, ACMG56_genes, transcripts, RADAR, mirbase,\n                  genesplicer, effects_transcripts, vcfanno, viral, exac, gnomad_exome, esp, 1000g, varpon, topmed,\n                  genotype2phenotype, fusion-blacklist]\n    annotations_available: [battenberg, dbnsfp, dbscsnv, ericscript, gnomad, topmed]\n    validation: [giab-NA12878, giab-NA24385, giab-NA24631, dream-syn3, dream-syn4, giab-NA12878-NA24385-somatic,\n                 giab-NA24143, giab-NA24149, giab-NA24694, giab-NA24695]\n  - dbkey: hg38\n    name: Human (hg38) full\n    indexes: [seq, twobit, bwa, hisat2]\n    annotations: [ccds, coverage, capture_regions, rmsk, prioritize, dbsnp, hapmap_snps,\n                  1000g_omni_snps, 1000g_snps, 1000g_indels, mills_indels, clinvar, qsignature,\n                  ACMG56_genes, transcripts, genesplicer, effects_transcripts, vcfanno, esp,\n                  exac, gnomad_exome, viral, RADAR, mirbase, varpon, topmed, genotype2phenotype,\n                  salmon-decoys, fusion-blacklist, purecn_mappability, simple_repeat, af_only_gnomad]\n    annotations_available: [dbnsfp, dbscsnv, ericscript, gnomad, topmed]\n    validation: [giab-NA12878, giab-NA24385, giab-NA24631,\n                 platinum-genome-NA12878, giab-NA12878-remap, giab-NA12878-crossmap,\n                 dream-syn4-crossmap, dream-syn3-crossmap, giab-NA12878-NA24385-somatic,\n                 giab-NA24143, giab-NA24149, giab-NA24694, giab-NA24695]\n  - dbkey: hg38-noalt\n    name: Human (hg38) without alternative alleles\n    annotations: [coverage, dbsnp, hapmap_snps, 1000g_omni_snps, 1000g_snps,\n                  1000g_indels, mills_indels, clinvar, transcripts, mirbase, genotype2phenotype]\n    annotations_available: [dbnsfp, dbscsnv]\n  - dbkey: mm9\n    name: Mouse (mm9)\n  - dbkey: mm10\n    name: Mouse (mm10)\n    indexes: [seq, twobit]\n    annotations: [problem_regions, dbsnp, transcripts, mirbase, rmsk, vcfanno, prioritize]\n  - dbkey: rn5\n    name: Rat (rn5)\n  - dbkey: rn6\n    name: Rat (rn6)\n    indexes: [seq, twobit]\n    annotations: [transcripts, mirbase]\n  - dbkey: canFam3\n    name: Dog (canFam3)\n    indexes: [twobit]\n    annotations: [dbsnp, transcripts, mirbase]\n  - dbkey: galGal4\n    name: Chicken (galGal4)\n  - dbkey: Sscrofa11.1\n    name: Pig (Sscrofa11.1)\n    indexes: [seq, twobit]\n    annotations: [transcripts]\n  - dbkey: dm3\n    name: D melangogaster (dm3)\n  - dbkey: BDGP6\n    name: D melangogaster (BDGP6)\n    indexes: [seq]\n    annotations: [transcripts, mirbase]\n  - dbkey: TAIR10\n    name: Arabidopsis thaliana (TAIR10)\n    annotations: [mirbase]\n  - dbkey: xenTro3\n    name: X tropicalis (xenTro3)\n  - dbkey: GRCz11\n    name: Zebrafish (GRCz11)\n    indexes: [seq, twobit]\n    annotations: [transcripts]\n  - dbkey: GRCz10\n    name: Zebrafish (GRCz10)\n  - dbkey: Zv9\n    name: Zebrafish (Zv9)\n  - dbkey: sacCer3\n    indexes: [seq]\n    annotations: [transcripts]\n    name: S cerevisiae (sacCer3)\n  - dbkey: WBcel235\n    name: C elegans (WBcel235)\n  - dbkey: pseudomonas_aeruginosa_ucbpp_pa14\n    name: Pseudomonas aeruginosa UCBPP-PA14\n\n# High level targets for specifying annotations\nannotation_groups:\n  variation: [ccds, problem_regions, GA4GH_problem_regions, capture_regions, MIG, coverage, prioritize, dbsnp,\n              hapmap, hapmap_snps, 1000g_omni_snps, ACMG56_genes,\n              1000g_snps, mills_indels, 1000g_indels, clinvar, cosmic, ancestral, qsignature,\n              genesplicer, effects_transcripts, varpon, vcfanno, viral, purecn_mappability, simple_repeat,\n              af_only_gnomad]\n  rnaseq: [transcripts, RADAR, rmsk, salmon-decoys, fusion-blacklist]\n  smallrna: [mirbase]\n  gemini: [esp, exac, gnomad_exome, 1000g]\n\n# Global set of indexes to include for each genome.\n# Available choices are in GENOME_INDEXES_SUPPORTED in cloudbio/biodata/genomes.py\ngenome_indexes:\n  - bwa\n  - twobit\n\n# Additional data targets\ninstall_liftover: false\ninstall_uniref: false\n"
  },
  {
    "path": "config/chef/cookbooks/.gitkeep",
    "content": ""
  },
  {
    "path": "config/chef_recipes.yaml",
    "content": "minimal:\nglobus:\n  - globus::client-tools\n  - globus::myproxy\n  - globus::gridftp-default\n"
  },
  {
    "path": "config/custom.yaml",
    "content": "---\n# List of custom programs to install that are not packaged. These each require\n# a install_name function in the appropriate place in the 'custom' directory to\n# define the steps for downloading, building, and installing the package.\nbio_general:\n #- bio4j # no easy download for 0.8\n - anaconda\n - emboss\n - pgdspider\nbio_nextgen:\n  alignment:\n    - bowtie\n    - gmap\n    - lastz\n    - mosaik\n    # - perm # difficult build process with directory names\n    - snap\n    #- stampy # no version names on tarballs\n  utilities:\n    - dwgsim\n    - fastq_screen\n    - ogap\n    - rnaseqc\n    - varianttools\n    - disambiguate\n    - alientrimmer\n  analysis:\n    # - echo # build problems on ubuntu 12.10\n    - picard\n    - shrec\n    - tophat\n    - featurecounts\n  assembly:\n    - abyss\n    - cortex_var\n    - ray\n    - transabyss\n    - trinity\n    - velvet\n  sv:\n    - hydra\n    - freec\n  variant:\n    - bcbio_variation\n    - crisp\n    - grabix\n    - mutect\n    - tassel\n    - stacks\n    - varscan\n  chip:\n    - macs\n  needs_64bit:\n    - ucsc_tools\n    - bfast\nbio_proteomics:\n  - transproteomic_pipeline\n  - omssa\n  - mzmine\n  - myrimatch\n  - directag\n  - tagrecon\n  - idpqonvert\n  - pepitome\n  - percolator\nbio_proteomics_wine:\n  - proteomics_wine_env\n  #- multiplierz\n  - proteowizard\n  - morpheus\ncloudman:\n  - cloudman\n  - novnc\ngalaxy:\n  - galaxy_webapp\ngalaxy_tools:\n  - cbl_galaxy_tools\ngalaxyp:\n  - galaxy_protk\n  - protvis\n# Pydoop and seal need update and should only be installed on cloud systems\n#distributed:\n#  - pydoop\n#  - seal\nsystem:\n  - homebrew\n  - s3fs\npython:\n  - bx-python\n  - netsa-python\n  - rpy\njava:\n  - leiningen\nphylogeny:\n  - tracer\n  - beast\n#Viral Cloud Resource (VCR) - JCVI's Viral Genomic Pipelines on the cloud\nvcr:\n  - galaxy_vcr\n  - viralassembly\n  - viralassembly_cleanall\n  - viralvigor\n  - viralvigor_test\n  - viralvigor_validate\n  - viralvigor_cleanall\n  - vicvb\n  - vicvb_cleanall\n# ToDo -- test on 64bit with bfast\n#- dnaa\n#- srma\n"
  },
  {
    "path": "config/fabricrc.txt",
    "content": "# Configuration file for Fabric specifying filesystem information\n# This can be adjusted and passed to fab on the commandline as '-c'\n#\n# For example\n#\n#   fab -f data_fabfile.py -H your_machine -c your_fabricrc.txt install_data_s3:your_biodata.yaml\n\n# -- Details about the operating system\n\n## If target machine is Ubuntu or CentOS, the following defaults should allow CloudBioLinux to \n## automatically determine the correct settings. For other operating systems (e.g. Debian or ScientificLinux)\n## please override distribution (and dist_name in case of Debian).\ndistribution = __auto__\ndist_name = __auto__\n\n## Ubuntu\n#distribution = ubuntu\n#dist_name = quantal\n## Centos\n#distribution = centos\n## Debian\n#distribution = debian\n#dist_name = squeeze\n\n# -- Installation details\n\n# Whether to clean the system for EC2 AMI preparation;\n# do not set on vagrant or local machines\nis_ec2_image = false\n\n# Global installation directory for packages and standard programs\nsystem_install = /usr/local\n\n# Local install directory for versioned software that will not\n# be included in the path by default\nlocal_install = /usr/local/share\n\n# Shell to be used by CBL scripts during runtime\nshell_config = ~/.bashrc\nshell = /bin/bash -i -c\n\n# Global setting for using sudo; allows installation of custom packages\n# by non-privileged users.\n# *Note*: ``system_install`` needs to point to a user-writeable directory if\n# ``use_sudo`` is set to ``False``\nuse_sudo = True\n\n# -- Details about reference data installation\n\n# Path where biological reference data files should be retrieved to\ndata_files = /mnt/biodata\n\n# --  Details about installing Galaxy and its dependencies. Values behind the\n#     comments are the defaults.\n\n# Specify as which user should Galaxy be installed\n#galaxy_user = galaxy\n\n# Path where Galaxy application is installed\n#galaxy_home = /mnt/galaxyTools/galaxy-central\n\n# URL for the Galaxy application source code repository\n#galaxy_repository = https://bitbucket.org/galaxy/galaxy-central/\n\n# A path to the binaries for interacting with PostgreSQL\n#postgres_bin_dir = /usr/lib/postgresql/9.1/bin\n#postrges_user = postgres\n#postgres_port = 5840\n\n# If ``True``, setup and initialize Postgres database for use by Galaxy\n# according to the following options\n#galaxy_setup_database = False\n#galaxy_db_name = galaxy\n#galaxy_database_path = /mnt/galaxy/db\n#galaxy_ftp_user_password = fu5yOj2sn\n#galaxy_universe_wsgi_url = https://bitbucket.org/galaxy/cloudman/raw/tip/universe_wsgi.ini.cloud\n\n# If ``True``, Galaxy repository will not be custom-configured after being cloned\n# The customization details are primarily targeted at a cloud deployment; for\n# details, see ``cbl.galaxy.__init__ -> _configure_galaxy_repository``\n#galaxy_preconfigured_repository = False\n\n# If ``True``, setup and initialize Postgres database for use by Galaxy\n#galaxy_setup_database = False\n\n# If ``True``, setup Galaxy as a system-level service controlled via init.d\n#galaxy_setup_service = False\n\n# If ``True``, setup Xvfb as a system-level service via init.d\n#galaxy_setup_xvfb = False\n\n# Location of Galaxy's .loc files\n#galaxy_loc_files = /mnt/biodata/galaxy/galaxy-data\n\n# Location of files required by Galaxy Trackster\n# This path needs to match what will be set in Galaxy's universe_wsgi.ini\n#galaxy_len_files = /mnt/galaxy/configuration_data/len\n\n# Galaxy requires Java tools to have their jar files places in a specific\n# directory, so specify it here.\n#galaxy_jars_dir = /mnt/galaxyTools/galaxy-central/tool-data/shared/jars\n\n# If ``True``, install a set of tools considered as Galaxy dependencies\n#galaxy_install_dependencies = False\n\n# Specify a custom location to a YAML file containing a list of tools to\n# install as Galaxy dependencies\n#galaxy_tools_conf = $__contrib_dir__/cloudman/tools.yaml\n\n# Location where (some?!) Galaxy dependency tools should be installed\n#galaxy_tools_dir = /mnt/galaxyTools/tools\n\n# If ``True``, install R packages listed in contrib/cloudman/tools.yaml\n#galaxy_install_r_packages = False\n\n# Paths for respective file systems that will be added to ``/etc/exports``\n# and thus exported via NFS when the operating system boots\n#galaxy_data_mount = /mnt/galaxyData\n#galaxy_tools_mount = /mnt/galaxyTools\n#galaxy_indices_mount = /mnt/biodata\n#extra_nfs_exports = comma,separate,desired,paths\n\n## Defaults for CloudMan LWR\npuppet_lwr_user = galaxy\npuppet_lwr_destination = /mnt/galaxy/lwr\n\n# -- Tool-specific configurations\n\n# A location of a custom nginx.conf or a comparable template, respectively.\n#nginx_conf_template_path = installed_files/nginx.conf.template\n#nginx_conf_path = path/to/custom/nginx.conf\n\n# Picard home\npicard_home = /usr/share/java/picard\n\n# -- novnc configuration options\n# Configure novnc along with CloudMan, disabled\n# by default for now.\n#configure_novnc = False\n# Where to install novnc\n#novnc_install_dir = /opt/novnc\n# Password used by novnc to connect to vncserver.\n#vnc_password = cl0udbi0l1nux\n# Set vnc_user if different than env.user\n#vnc_user=ubuntu\n#vnc_display=1\n#vnc_depth=16\n#vnc_geometry=1024x768\n"
  },
  {
    "path": "config/haskell-libs.yaml",
    "content": "---\n# Define haskell libraries to be installed via cabal.\ncabal:\n  # base\n  - network\n  - cabal-install\n  - cabal-dev\n  # libraries\n  - mtl\n  - parsec2\n  - parsec3\n  - quickcheck\n  - tagsoup\n  - utf8-string\n  - x11\n  - x11-xft\n  # bioinformatics\n  - bio\n  - flower\n  # FIXME: Fails in Lucid, if the source is downloaded separately and installed according to the README the issue won't appread\n  - flowsim\n  #- samtools\n"
  },
  {
    "path": "config/main.yaml",
    "content": "---\n# Top level configuration file that specifies which groups of programs\n# should be installed. New sections that are added to individual config\n# files should go here. Comment out any groups you don't want to have\n# installed.\npackages:\n  - minimal\n  - desktop\n  - programming\n  - distributed\n  - documentation\n  - libraries\n  - amazon\n  - python\n  - r\n  - ruby\n  - perl\n  - java\n  - java_extras\n  - erlang\n  - haskell\n  - databases\n  - math\n  - viz\n  - web\n  - bio_general\n  - bio_search\n  - bio_alignment\n  - bio_nextgen\n  - bio_sequencing\n  - bio_annotation\n  - bio_microarray\n  - bio_visualization\n  - bio_utils\n  - phylogeny\n# Do not install by default to avoid errors with local installs\n# Injected by setting is_ec2_image in fabricrc.txt\n#  - galaxy\n#  - galaxy_tools\n#  - cloudman\nlibraries:\n  #- haskell-libs\n  - perl-libs\n  - python-libs\n  - r-libs\n  - ruby-libs\n"
  },
  {
    "path": "config/node_extra.json",
    "content": "// Properties used by chef to configure node. Merged with fabric\n// properties to produce final node.json file used by Chef.\n{\n  \"globus\": {\n    \"repository_deb_url\" : \"http://www.globus.org/ftppub/gt5/5.2/5.2.4/installers/repo/globus-repository-5.2-stable-quantal_0.0.3_all.deb\"\n  }\n}"
  },
  {
    "path": "config/packages-debian.yaml",
    "content": "---\n# Debian packages use the standard packages.yaml list with naming substitutions\n# specified in this file.\nclojure: \"\"\nec2-ami-tools: \"\"\nec2-api-tools: \"\"\nfirefox: iceweasel\nfreenx: \"\"\nfreenx-server: \"\"\ngdm: gdm3\nmongodb-10gen: mongodb\npython-vm-builder-ec2: \"\"\nubuntu-desktop: \"\"\nugene: \"\"\nunity-2d: \"\"\n# Problem Bio-Linux builds on debian \nbio-linux-arb: \"\"\nbio-linux-assembly-conversion-tools: \"\"\nbio-linux-emboss: \"\"\n# emboss-data is broken in Squeeze\nemboss: \"\"\nemboss-data: \"\"\nemboss-lib: \"\"\n\n"
  },
  {
    "path": "config/packages-homebrew.yaml",
    "content": "# Packages available in the Homebrew and Linuxbrew package manager\n---\nbio_nextgen:\n  alignment:\n    - bwa\n    - bowtie2\n    - novoalign\n    - rna-star\n  utilities:\n    - bamtools\n    - bedtools\n    - cramtools\n    - biobambam\n    - fastqc\n    - fastx_toolkit\n    - qualimap\n    - sambamba-binary\n    - staden_io_lib\n  analysis:\n    - cufflinks-binary\n    - samtools\n    - htslib\n    - bcftools\n    #- tophat\n  variant:\n    - delly;--with-binary\n    - freebayes\n    - gatk-framework\n    - lofreq\n    - lumpy-sv\n    - pindel\n    - platypus-variant\n    - rtg-tools\n    - scalpel\n    - snpeff\n    - theta2\n    - vardict\n    - vcflib\n    - vcftools\n    - vep\n    - wham\n    # - yaha  The Principal Investigator has moved; the software does not download ATM\n  integration:\n    - biointerchange\ndistributed:\n  - parallel\n"
  },
  {
    "path": "config/packages-nix.yaml",
    "content": "# These are packages defined in Nix packages. Nix is an independent and\n# distribution agnostic packaging system, with many interesting features.\n#\n# For more information see http://nixos.org/nixpkgs/\n\nminimal:\n  - nix\neditors:\n  - vim-7.3 # a recent vim, just for the test\nphylogeny:\n  - paml\n"
  },
  {
    "path": "config/packages-scientificlinux.yaml",
    "content": "minimal:\n  version_control:\n    - git-core\n    - subversion\n    - mercurial\n  build:\n    - cmake\n    - gcc\n    - gcc-c++\n    - gcc-gfortran\n    - make\n  util:\n    - screen\n    - unzip\n    - tar\n    - less\n  network:\n    - curl\n    - wget\nlibraries:\n  - atlas\n  - atlas-devel\n  - blas\n  - blas-devel\n  - bzip2-devel\n  - freetype-devel\n  - gsl-devel\n  - libpng-devel\n  - openssl-devel\n  - ncurses-devel\ndocumentation:\nprogramming:\n  - protobuf-c\n  - protobuf-compiler\n  - protobuf-python\njava:\n  - ant\n  - ant-junit\n  - ant-nodeps\n  - java-1.7.0-openjdk\n  - java-1.7.0-openjdk-devel\n  - junit4\npython:\n  - python\n  - python-devel\n  - boost-python-devel\n  - numpy\n  - python-pip\nr:\n  - R\n  - R-core\n  - libRmath\n  - libRmath-devel\ndistributed:\n  - hadoop-0.20\n  - hadoop-0.20-native\n  - hadoop-0.20-pipes\n  - hadoop-0.20-source\n  - hadoop-0.20-libhdfs\n  - mpich2\n  - mpich2-devel\ngalaxy:\n  all:\n    - postgresql\n    - gcc-gfortran\n    - java-1.7.0-openjdk\n    - xorg-x11-server-Xvfb\n    - cmake\n    - zlib-devel\n    - ncurses-devel\n    - xz-devel\n    - tcl-devel\n    - tk-devel    \n    - ImageMagick\n    #  - scipy  ## Available in CentOS but not SL, install some prequirements instead including numpy (likely many tools just require this.)\n    - numpy\n    - numpy-f2py\n    - blas-devel \n    - atlas-devel\n\n    # Following requrie EPEL repository (setup by CloudBioLinux)    \n    - pdfjam\n    - sparsehash-devel\n\n    # cran packages for CentOS are not maintained, do what we can with EPEL.\n    - R\n    - R-devel\n    # Missing relative to ubuntu install: All missing in CentOS (e.g. axel, python-rpy, many R modules)\n    # Missing relative to CentOS install: scipy.\n    #   Information on setting up scipy and rpy can be found at http://getgalaxyp.org/install.html\nbio_proteomics:\n  # Needed for TPP\n  - libxslt\n  - perl-libxml-perl\n  - libXpm-devel\n  - bzip2-devel\n  - gnuplot\n  # Needed for percolator\n  - xerces-c-devel\n  - xerces-c\n  - xsd\n  - sqlite-devel\n  # - libboost-serialization-dev, libboost-system-dev, libboost-filesystem-dev Installed by boost-devel\n  - zlib-devel\n  - boost-devel\n\n  # needed for building OpenMS\n  - qt-devel\n  - qtwebkit-devel\n\n  # Needed for PSM-eval\n  - libyaml-devel\n\n  # Needed for RVM (in turn required for ProtK)\n  - bison\n  - libffi-devel\n  - libtool\n  - libyaml-devel\n  - openssl-devel\n  - patch\n  - readline-devel\ngalaxyp:\n  # Needed for TPP\n  - libxslt\n  - perl-libxml-perl\n  - libXpm-devel\n  - bzip2-devel\n  - gnuplot\n  # Needed for percolator\n  - xerces-c-devel\n  - xerces-c\n  - xsd\n  - sqlite-devel\n  # - libboost-serialization-dev, libboost-system-dev, libboost-filesystem-dev Installed by boost-devel\n  - zlib-devel\n  - boost-devel\n\n  # Needed for nbic galaxy database utilities\n  - perl-Log-Log4perl\n  # - libswiss-perl NOT AVAILABLE, ONE TOOL WON'T WORK.\n\n  # needed for building OpenMS\n  - qt-devel\n  - qtwebkit-devel\n\n  # Needed for PSM-eval\n  - libyaml-devel\n\n  # Needed for RVM (in turn required for ProtK)\n  - bison\n  - libffi-devel\n  - libtool\n  - libyaml-devel\n  - openssl-devel\n  - patch\n  - readline-devel\n"
  },
  {
    "path": "config/packages-yum.yaml",
    "content": "---\n# List of packages to be installed via the yum package manager.\nminimal:\n  version_control:\n    - git-core\n    - subversion\n    - mercurial\n  build:\n    - cmake\n    - gcc\n    - gcc-c++\n    - gcc-gfortran\n    - make\n    - patch\n  util:\n    - screen\n    - unzip\n    - tar\n    - less\n  network:\n    - curl\n    - wget\nlibraries:\n  - atlas\n  - atlas-devel\n  - blas\n  - blas-devel\n  - bzip2\n  - bzip2-devel\n  - freetype-devel\n  - gsl-devel\n  - lapack\n  - lapack-devel\n  - libgfortran\n  - libpng-devel\n  - openssl-devel\n  - ncurses-devel\n  - libxml2-devel\n  - zlib-devel\ndocumentation:\ndesktop:\n  - gnome-panel\n  - freenx\n  - gnome-terminal\n  - firefox\n  - evince\n  - xorg-x11-server-Xvfb\nprogramming:\n  editors:\n    - emacs\n    - emacs-common\n    - emacs-el\n    - vim-X11\n    - ctags\n  version_control:\n    - cvs\n    - bzr\n  build:\n    - swig\n    - curl-devel\n    - libxslt\n    - libxslt-devel\n    - xfsprogs\n    - glibc\n    - glibc-devel\n    - zip\n    - zziplib-devel\n    - xz\n    - xz-devel\n    - gtk2\n    - gtk2-devel\n    - globus-openssl-module\n    - globus-openssl-module-devel\n    - pcre\n    - pcre-devel\n    - readline\n    - readline-devel\n    - rsync\n    - librsync\n    - librsync-devel\n  libraries:\n    - libuuid-devel\n    - tcl-devel\n    - tk-devel\n    - ImageMagick\n    - boost\n    - boost-devel\n    - openmpi\n    - openmpi-devel\n    - gd\n    - gd-devel\n    - libgnomecanvas\n    - libgnomecanvas-devel\n    - gnome-vfs2\n    - gnome-vfs2-devel\n    - libgnomeui\n    - libgnomeui-devel\n    - pango\n    - pango-devel\n  virtual:\n  util:\n    - tree\n  lang:\n    - aspell\ndistributed:\namazon:\npython:\n  - python\n  - python-devel\n  - boost-python-devel\n  - numpy\nruby:\n  - ruby\n  - ruby-devel\n  - rubygems\nr:\n  - R\n  - R-core\n  - R-core-devel\n  - R-java\n  - libRmath\n  - libRmath-devel\nperl: \n  - perl\n  - perl-devel\n  - perl-core\njava:\n  - ant\n  - java-1.7.0-openjdk\n  - java-1.7.0-openjdk-devel\nerlang:\n  - erlang\n  # - erlang-devel  ## Not available, erlang seems to install quite a bit.\nhaskell:\n  - ghc\ndatabases:\n  mysql:\n    - mysql\n    - mysql-devel\n    - mysql-server\n  postgres:\n    - postgresql\n    - postgresql-devel\n    - postgresql-server\n  sqlite:\n    - sqlite\n    - sqlite-devel\n  document:\n    - mongodb\n    - mongodb-devel\n    - couchdb\n    - tokyocabinet\n    - tokyocabinet-devel\n  hdf:\n    - hdf5\n    #- hdf5-devel\nmath:\n  # - prover9  # No longer available? Cannot find.\n  - octave\n  - octave-devel\nviz:\n  - gnuplot\n  - ImageMagick\n  - ImageMagick-devel\n  - graphviz\n  - graphviz-devel\nweb:\n  - httpd\n  - httpd-devel\n  - nginx\nbio_general:\nbio_search:\nbio_alignment:\nbio_nextgen:\nbio_sequencing:\nbio_annotation:\nbio_microarray:\nbio_visualization:\nbio_utils:\nphylogeny:\n  all:\n  needs_64bit:\ngalaxy:\n  all:\n    - postgresql\n    - gcc-gfortran\n    - java-1.7.0-openjdk\n    - xorg-x11-server-Xvfb\n    - cmake\n    - zlib-devel\n    - ncurses-devel\n    - xz-devel\n    - tcl-devel\n    - tk-devel    \n    - ImageMagick\n    - scipy\n    # Following requrie EPEL repository (setup by CloudBioLinux)    \n    - pdfjam\n    - sparsehash-devel\n\n    # cran packages for CentOS are not maintained, do what we can with EPEL.\n    - R\n    - R-devel\n    # Missing relative to ubuntu install: axel, python-rpy, all R modules\n    #   Information on setting up rpy can be found at http://getgalaxyp.org/install.html\nbio_proteomics:\n  # Needed for TPP\n  - libxslt\n  - perl-libxml-perl\n  - libXpm-devel\n  - bzip2-devel\n  - gnuplot\n  # Needed for percolator\n  - xerces-c-devel\n  - xerces-c\n  - xsd\n  - sqlite-devel\n  # - libboost-serialization-dev, libboost-system-dev, libboost-filesystem-dev Installed by boost-devel\n  - zlib-devel\n  - boost-devel\n\n  # needed for building OpenMS\n  - qt-devel\n  - qtwebkit-devel\n\n  # Needed for PSM-eval\n  - libyaml-devel\n\n  # Needed for RVM (in turn required for ProtK)\n  - bison\n  - libffi-devel\n  - libtool\n  - libyaml-devel\n  - openssl-devel\n  - patch\n  - readline-devel\ngalaxyp:\n  # Needed for TPP\n  - libxslt\n  - perl-libxml-perl\n  - libXpm-devel\n  - bzip2-devel\n  - gnuplot\n  # Needed for percolator\n  - xerces-c-devel\n  - xerces-c\n  - xsd\n  - sqlite-devel\n  # - libboost-serialization-dev, libboost-system-dev, libboost-filesystem-dev Installed by boost-devel\n  - zlib-devel\n  - boost-devel\n\n  # Needed for nbic galaxy database utilities\n  - perl-Log-Log4perl\n  # - libswiss-perl NOT AVAILABLE, ONE TOOL WON'T WORK.\n\n  # needed for building OpenMS\n  - qt-devel\n  - qtwebkit-devel\n\n  # Needed for PSM-eval\n  - libyaml-devel\n\n  # Needed for RVM (in turn required for ProtK)\n  - bison\n  - libffi-devel\n  - libtool\n  - libyaml-devel\n  - openssl-devel\n  - patch\n  - readline-devel\n"
  },
  {
    "path": "config/packages.yaml",
    "content": "---\n# List of packages to be installed via the package manager.\n#\n# This is derived from the Infochimps machetEC2 package:\n#\n# http://github.com/infochimps/machetec2\n#\n# and merged with packages from bioperl-max:\n#\n# http://fortinbras.us/bioperl-max/\n#\n# and biolinux:\n#\n# http://www.jcvi.org/cms/research/projects/jcvi-cloud-biolinux/included-software/\n#\n# Package names are the terminal symbols in the tree (the text on any\n# line which begins with whitespace followed by a hypen and a space).\n#\n# The package list is organized taxonomically so that parts of it can\n# be selectively installed/ignored. See main.yaml for top level configuration.\nminimal:\n  #Packages to support a minimal installation (no X!)\n  administration:\n    - sudo\n    - htop\n  editors:\n    - vim\n    # don't put in emacs, as it pulls in X\n  version_control:\n    - git-core\n    - cvs\n    - subversion\n    - mercurial\n    - bzr\n    #- darcs # pulls in postfix mail configuration.\n  build:\n    - bsdmainutils\n    - clang\n    - cmake\n    - gcc\n    - g++\n    - gfortran\n    - libtool\n    - make\n    - patch\n    - pkg-config\n  util:\n    - axel\n    - gawk\n    - openssh-server\n    - pigz\n    - rsync\n    - screen\n    - unzip\n    - less\n  network:\n    - curl\n    - wget\nlibraries:\n  - libatlas-base-dev\n  #- libatlas-dev\n  - libblas-dev\n  - liblapack-dev\n  - libeigen3-dev\n  - libc6-dev\n  - libzip-dev\n  - libbz2-dev\n  - zlib1g-dev\n  - libcurl4-openssl-dev\n  - libgsl-dev\n  - libgsl2\n  - gsl-bin\n  - mime-support\n  - libncurses5-dev\n  - libpcre3-dev\n  - libpng12-dev\n  - libreadline-dev\n  - libssl-dev\n  - uuid\n  - uuid-dev\n  - libxml2\n  - libxml2-dev\n  - libxslt1-dev\n  - libxslt1.1\n  - xz-utils\n  - liblzma-dev\n  - zlib1g-dev\n  - libnetcdf-dev\ndocumentation:\n  - texlive-latex-base\n  - texlive-latex-recommended\n  - texlive-latex-extra\ndesktop:\n  #Packages to support a remote desktop\n  all:\n    - gnome-panel\n    - gnome-system-tools\n    - synaptic\n    - software-center\n    #- freenx-server  # Gone on 14.04\n    - gnome-backgrounds\n    - gnome-terminal\n    - firefox\n    - evince\n    - ubuntu-desktop\n    - xvfb\n    - gdm\n  # NX compatible window managers on Ubuntu 11.10+\n  ubuntu_oneiric:\n    - gnome-session-fallback\n    - unity-2d\n  ubuntu_lucid:\n    - rubygems\ndesktop-cloud:\n  # some special purpose VMs, such as BOINC, only require a minimal desktop\n  - lxde\n  - vnc4server\nprogramming:\n  editors:\n    - emacs\n    - emacs-goodies-el\n    - emacsen-common\n    - exuberant-ctags\n    - gedit\n  build:\n    - libgtk2.0-dev\n    - protobuf-compiler\n    - python-software-properties\n    - stow\n    - swig\n    - rsync\n    - xfsprogs\n  filesystem:\n    - libfuse-dev\n  libraries:\n    - tcl-dev\n    - tk-dev\n    - imagemagick\n    - pdfjam\n    - libsparsehash-dev\n    - xvfb\n    - libboost-dev\n    - libboost-python-dev\n    - libboost-thread-dev\n    - libgd2-xpm-dev\n    - libgnomecanvas2-0\n    - libgnomecanvas2-dev\n    - libgnomevfs2-0\n    - libgnomevfs2-dev\n    - libgnomeui-0\n    - libgnomeui-dev\n    - libpango1.0-0\n    - libpango1.0-dev\n    - libx11-dev\n  packaging:\n    - build-essential\n    - devscripts\n    - debhelper\n    - dh-make\n    - pbuilder\n    - unzip\n  virtual:\n    #- virtualbox-4.2\n    - dkms\n  util:\n    - mc\n    - tree\n    - screen\n    - time\n    - fdupes\n  network:\n    - lynx\n    - traceroute\n  lang:\n    - aspell\n    - dictionaries-common\n    - libaspell-dev\ndistributed:\n  all:\n    - rabbitmq-server\n    - nfs-kernel-server\n    - nfs-common\n    - libopenmpi-dev\n    - openmpi-bin\n    - libzmq-dev\n    - libfuse-dev\n  # only for oneiric 11.10\n  ubuntu_oneiric:\n    - libmpich2-3\n    - libmpich2-dev\n    - mpich2\namazon:\n  - s3cmd\n  - euca2ools\npython:\n  - python\n  - python-dev\n  - python-distutils-extra\n  - python-pip\n  - python-setuptools\n  - python-virtualenv\n  - python3\n  - python3-dev\n  - python-rpy\nruby:\n  - ruby1.9.1 # default Ruby install\n  - ruby1.9.1-dev\nr:\n  - r-base\n  - r-base-core\n  - r-base-dev\nperl:\n  - perl\n  - libperl-dev\n  - perl-modules\n  - perl-tk\n  - parrot\n  - parrot-devel\n  - cpanminus\n  - libextutils-autoinstall-perl\njava:\n  # - sun-java6-bin # Sun Java removed from Ubuntu package repositories\n  # - sun-java6-jre # Package 'sun-java6-jre' has no installation candidate\n  # - sun-java6-jdk Package 'sun-java6-jdk' has no installation candidate\n  - openjdk-7-jdk\n  - openjdk-7-jre\njava_extras:\n  - openjdk-6-jdk\n  - openjdk-6-jre\n  - ant\n  - libbiojava-java\n  - libbiojava-java-demos\n  #- eclipse\n  - maven\n  - libjline-java\n  - libjhdf5-java\n  - jruby\n  - jython\n  - scala\n  - scala-library\nerlang:\n  - erlang\n  - erlang-base\n  - erlang-dev\n  - erlang-examples\n  - erlang-manpages\n  - erlang-mode\nhaskell:\n  - ghc\n  - cabal-install\n  - c2hs\ndatabases:\n  mysql:\n    - mysql-client\n    - mysql-common\n    - mysql-server\n    - libmysqlclient-dev\n  postgres:\n    - postgresql\n    - postgresql-client\n    - postgresql-plpython\n    - postgresql-plperl\n    - libpq-dev\n  sqlite:\n    - libsqlite0\n    - libsqlite0-dev\n    - libsqlite3-0\n    - libsqlite3-dev\n    - sqlite3\n  document:\n    - libdb-dev\n    - mongodb\n    - couchdb\n    - tokyocabinet-bin\n    - libtokyocabinet-dev\n  hdf:\n    - libhdf5-serial-dev\nmath:\n  - prover9\n  - octave\nviz:\n  - imagemagick\n  - x11-apps\n  - mayavi2\n  - mtasc # for modest maps\n  - gnuplot\n  - graphviz\n  - libgraphviz-dev\n  - libgl1-mesa-dev\n  - libgl1-mesa-dri\n  - libgl1-mesa-glx\n  - libglu1-mesa\n  - libglu1-mesa-dev\nweb:\n  - apache2\nboinc:\n  - boinc-client\n  - boinc-manager\nbio_general:\n  - primer3\n  - ugene\nbio_search:\n  - blast2\n  - ncbi-tools-bin\n  - cdbfasta\nbio_alignment:\n  - exonerate\n  - mafft\n  - muscle\n  - probcons\n  - wise\nbio_nextgen:\n  - fastx-toolkit\n  - maq\n  - plink\n  - last-align\n  - tabix\n  # - samtools # only on 0.18, prefer custom build\nbio_sequencing:\nbio_annotation:\n  - mcl\n  - predictprotein\n  - tigr-glimmer\nbio_markergenes:\n  - ampliconnoise\n  - mothur\n  - rdp-classifier\n  - qiime\nbio_microarray:\nbio_visualization:\n  - rasmol\nbio_proteomics:\n  all:\n    # Required for protk\n    - libxml2-dev\n    - libreadline-dev\n    - libxslt-dev\n\n    # Required for protvis\n    - python-virtualenv\n    - python-setuptools\n    - libpng12-dev\n    - python-dev\n    - libexpat1-dev\n    - libboost-dev\n\n    # Needed for TPP\n    - xsltproc\n    - libxml-sax-expat-perl\n    - libgd2-xpm-dev\n    - libbz2-dev\n    - gnuplot\n\n    # Needed for percolator\n    - libxerces-c3.1\n    - libxerces-c-dev\n    - xsdcxx\n    - libboost-dev\n    - libboost-filesystem-dev\n    - libboost-system-dev\n    - libsqlite3-0\n    - libboost-serialization-dev\n    - zlib1g-dev\n    - libboost-dev\n\n    # needed for 32bit execs distributed with searchgui.\n    # - ia32-libs ## Doesn't play well with dependencies of gnuplot, TODO: Find more specific lib.\n\n    # needed for building OpenMS\n    - libtool\n    - qt4-dev-tools\n    - libqtwebkit-dev\n    - libqtwebkit4\n\n     # Needed for psm-eval\n    - libyaml-dev\n\n    # Needed for RVM\n    - libbison-dev\n    - ncurses-dev\n    - automake\n    - libtool\n    - bison\n    - subversion\n    - libgdbm-dev\n    - libffi-dev\n    - libsqlite3-dev\n    - sqlite3\n\nbio_proteomics_wine:\n  - wine\nbio_utils:\nphylogeny_testing:\n  # new packages in testing (Debian 'wheezy')\n  all:\n    - debian-archive-keyring\n    - raxml\n    # - mrbayes-mpi broken, should eventually replace biolinux-mrbayes-multi\nphylogeny_experimental:\n  # new packages in experimental\n  all:\n    - paml        # should replace biolinux-paml\nphylogeny:\n  all:\n    - phyml\n    - mrbayes\n    - njplot\n    - tree-puzzle\n    - treeviewx\n  needs_64bit:\n    - fasttree\n## Package lists imported from mi-deployment.\ncloudman:\n  all:\n    # Required to install CloudMan-required Python libraries\n    - python-pip\n    # Required for communication between instances\n    - rabbitmq-server\n    # Required to handle data volumes (including LVM support)\n    - lvm2\n    - dmsetup\n    # - mdadm\n    - reiserfsprogs\n    - xfsprogs\n    # Required to compile nginx\n    - build-essential\n    #- gcc ## included in minimal\n    - libpcre3-dev\n    - libssl-dev\n    - stow\n    # Required for cluster-wide NFS file system\n    - nfs-kernel-server\n    - nfs-common\n    # Enable OpenMPI\n    - libopenmpi-dev\n    - libopenmpi1.3\n    - openmpi-bin\n    - openmpi-common\n    # Needed to enable updates for Galaxy\n    # - mercurial ## Included in minimal\n    # Required for compiling ProFTPd (must match installed PostgreSQL version!)\n    - postgresql-server-dev-9.5\n    # Actually not sure what of the following is really needed\n    - libboost-all-dev\n    - libreadline6-dev\n    # Required for s3fs\n    - libfuse-dev\n    - libfuse2\n    - pkg-config\n    - libcurl4-openssl-dev\n    - libxml2-dev\n    - mime-support\n    # Required for novnc\n    - vnc4server\n    - glusterfs-client\n  ubuntu_quantal:\n    # Required to get HTCondor, not available on 12.04 (latest LTS release).\n    - condor\ngalaxy:\n  all:\n    - postgresql\n    - gfortran\n    - default-jdk\n    - python-rpy\n    - xvfb\n    # Required by Freebayes\n    - cmake\n    # Required by BWA\n    - zlib1g-dev\n    # Required by Samtools\n    - libncurses5-dev\n    # Required by Compute q-values\n    - liblzma-dev\n    # Parallel file download (used by the Galaxy ObjectStore)\n    - axel\n    # Required by various R modules\n    - tcl-dev\n    # Required by various R modules\n    - tk-dev\n    # Required by RGalaxy\n    - imagemagick\n    - pdfjam\n    - python-scipy\n    - libsparsehash-dev\n\n    # Latest python-rpy not currently compatible with R 3.0.1 at least on precise (presumably\n    # quantal as well), hack below is to specify an older version of R compabile with python-rpy.\n    #- r-base\n    #- r-base-core\n    #- r-base-dev\n    - r-cran-lattice\n    - r-cran-qvalue\n    # Required by HVIS\n    - r-bioc-hilbertvis\n  ubuntu_quantal:\n    - \"r-base-core=3.0.0-2quantal\"\n    - \"r-base-dev=3.0.0-2quantal\"\n  ubuntu_precise:\n    - \"r-base-core=3.0.0-2precise\"\n    - \"r-base-dev=3.0.0-2precise\"\ngalaxyp:\n  all:\n     # Required for protk\n     - libxml2-dev\n     - libreadline-dev\n     - libxslt-dev\n\n     # Required for protvis\n     - python-virtualenv\n     - python-setuptools\n     - libpng12-dev\n     - python-dev\n     - libexpat1-dev\n     - libboost-dev\n\n     # Needed for TPP\n     - xsltproc\n     - libxml-sax-expat-perl\n     - libgd2-xpm-dev\n     - libbz2-dev\n     - gnuplot\n\n     # Needed for percolator\n     - libxerces-c3.1\n     - libxerces-c-dev\n     - xsdcxx\n     - libboost-dev\n     - libboost-filesystem-dev\n     - libboost-system-dev\n     - libsqlite3-0\n     - libboost-serialization-dev\n     - zlib1g-dev\n     - libboost-dev\n\n     # needed for 32bit execs distributed with searchgui.\n     # - ia32-libs ## Doesn't play well with dependencies of gnuplot, TODO: Find more specific lib.\n\n     # needed for building OpenMS\n     - libtool\n     - qt4-dev-tools\n     - libqtwebkit-dev\n     - libqtwebkit4\n\n     # Needed for nbic galaxy database utilities\n     - liblog-log4perl-perl\n     - libswiss-perl\n\n     # Needed  for psm-eval\n     - libyaml-dev\n\n     # Needed for RVM\n     - libbison-dev\n     - ncurses-dev\n     - automake\n     - libtool\n     - bison\n     - subversion\n     - libgdbm-dev\n     - libffi-dev\n     - libsqlite3-dev\n     - sqlite3\n\nnginx:\n  - nginx\n\nrabbitmq:\n  - rabbitmq-server\n\nsupervisor:\n  - supervisor\n\ncoffeescript:\n  - coffeescript\n"
  },
  {
    "path": "config/perl-libs.yaml",
    "content": "---\n# Configuration file defining perl libraries to install via CPAN\ncpan:\n# build\n  - 'Locale::Maketext::Lexicon'\n  - 'Module::Signature'\n  - 'Build'\n# dependencies\n  - 'Algorithm::C3'\n  - 'Algorithm::Diff'\n  - 'Algorithm::Munkres'\n  - 'Array::Compare'\n  - 'Bit::Vector'\n  - 'Bundle::LWP'\n  - 'Cache'\n  - 'Cairo'\n  - 'Carp::Assert'\n  - 'Carp::Clan'\n  - 'CGI::Application'\n  - 'CGI::Application::Plugin::Authentication'\n  - 'CGI::Application::Plugin::Authorization'\n  - 'CGI::Application::Plugin::TT'\n  - 'CGI::Application::Plugin::HTDot'\n  - 'CGI::Session'\n  - 'Class::Accessor'\n  - 'Class::C3'\n  - 'Class::C3::XS'\n  - 'Class::Data::Inheritable'\n  - 'Class::Inspector'\n  - 'Class::ISA'\n  - 'Class::MOP'\n  - 'Clone'\n  - 'Config::Auto'\n  - 'Config::IniFiles'\n  - 'Config::Simple'\n  - 'Convert::Binary::C'\n  - 'Crypt::SSLeay'\n  - 'Data::FormValidator'\n  - 'Data::OptList'\n  - 'Data::Stag'\n  - 'Date::Calc'\n  - 'Date::Manip'\n  - 'DateTime::TimeZone'\n  - 'DateTime::Locale'\n  - 'DateTime'\n  - 'Devel::GlobalDestruction'\n  - 'Devel::StackTrace'\n  - 'Digest::MD5'\n  - 'Digest::HMAC'\n  - 'Digest::SHA1'\n  - 'Email::Valid'\n  - 'Error'\n  - 'Exception::Class'\n  - 'Exception::Class::TryCatch'\n  - 'ExtUtils::Depends'\n  - 'ExtUtils::Manifest'\n  - 'ExtUtils::ModuleMaker'\n  - 'ExtUtils::PkgConfig'\n  - 'FCGI'\n  - 'File::Copy::Recursive'\n  - 'File::Listing'\n  - 'File::MMagic'\n  - 'File::Slurp'\n  - 'File::Which'\n  - 'Font::AFM'\n  - 'FreezeThaw'\n  - 'GD::Graph'\n  - 'GD::Text'\n  - 'Glib'\n  - 'Gtk2'\n  - 'Gnome2::Canvas'\n  - 'Gnome2::VFS'\n  - 'Gnome2'\n  - 'Graph'\n  - 'GraphViz'\n  - 'Heap'\n  - 'HTML::Entities'\n  - 'HTML::FillInForm'\n  - 'HTML::FormatData'\n  - 'HTML::HeadParser'\n  - 'HTML::Parser'\n  - 'HTML::Tagset'\n  - 'HTML::Template'\n  - 'HTML::Tree'\n  - 'Image::Size'\n  - 'Inline'\n  - 'IO::Compress::Base'\n  - 'IO::Compress::Bzip2'\n  - 'IO::Compress::Lzf'\n  - 'IO::Pty'\n  - 'IO::Socket::SSL'\n  - 'IO::String'\n  - 'IO::Stringy'\n  - 'IO::Unread'\n  - 'IPC::Run'\n  - 'IPC::ShareLite'\n  - 'IPC::Signal'\n  - 'Jcode'\n  - 'List::MoreUtils'\n  - 'Locale::gettext'\n  - 'Log::Dispatch'\n  - 'LWP::Simple'\n  - 'Mail::Internet'\n  - 'Math::Derivative'\n  - 'Math::Random'\n  - 'Math::Spline'\n  - 'MIME::Tools'\n  - 'MIME::Types'\n  - 'MLDBM'\n  - 'Module::Build'\n  - 'MRO::Compat'\n  - 'Net::Daemon'\n  - 'Net::DBus'\n  - 'Net::DNS'\n  - 'Net::Domain::TLD'\n  - 'Net::IP'\n  - 'Net::LibIDN'\n  - 'Net::SSLeay'\n  - 'OLE::Storage'\n  - 'Pango'\n  - 'Params::Util'\n  - 'Params::Validate'\n  - 'Parse::RecDescent'\n  - 'Perl6::Junction'\n  - 'Pod::Xhtml'\n  - 'PostScript::File'\n  - 'PostScript::TextBlock'\n  - 'Proc::InvokeEditor'\n  - 'Proc::WaitStat'\n  - 'Regexp::Common'\n  - 'RPC::XML'\n  - 'Set::Scalar'\n  - 'Scalar::Util'\n  - 'Scope::Guard'\n  - 'SOAP::Lite'\n  - 'Sort::Naturally'\n  - 'Spreadsheet::ParseExcel'\n  - 'Spreadsheet::ParseExcel::Simple'\n  - 'Spreadsheet::WriteExcel'\n  - 'Storable'\n  - 'SQL::Statement'\n  - 'Statistics::Descriptive'\n  - 'Sub::Exporter'\n  - 'Sub::Install'\n  - 'Sub::Name'\n  - 'Sub::Uplevel'\n  - 'SVG::Graph'\n  - 'SVG'\n  - 'Syntax::Highlight::Perl::Improved'\n  - 'Task::Weaken'\n  - 'Term::ReadKey'\n  - 'Test::Exception'\n  - 'Text::CharWidth'\n  - 'Text::CSV'\n  - 'Text::CSV_XS'\n  - 'Text::Iconv'\n  - 'Text::ParseWords'\n  - 'Test::Harness'\n  - 'Test::More'\n  - 'Tie::IxHash'\n  - 'Tree::DAG_Node'\n  - 'Try::Tiny'\n  - 'Unicode::Map'\n  - 'UNIVERSAL::require'\n  - 'URI::Escape'\n  - 'UUID'\n  - 'WWW::RobotRules'\n  - 'XML::DOM'\n  - 'XML::DOM::XPath'\n  - 'XML::Filter::BufferText'\n  - 'XML::LibXML '\n  - 'XML::NamespaceSupport'\n  - 'XML::Parser'\n  - 'XML::Parser::PerlSAX'\n  - 'XML::RegExp'\n  - 'XML::SAX'\n  - 'XML::SAX::Expat'\n  - 'XML::SAX::Writer'\n  - 'XML::Simple'\n  - 'XML::Twig'\n  - 'XML::Writer'\n  - 'XML::XPathEngine'\n  - 'XML::XPath'\n  - 'YAML'\n  - 'YAML::Syck'\n  # Database drivers\n  - 'DB_File'\n  - 'DBI'\n  - 'DBD::AnyData'\n  - 'DBD::CSV'\n  - 'DBD::mysql'\n  - 'DBD::Pg'\n  - 'DBD::SQLite'\n  # Biome : these will definitely be added to over time\n  - 'Moose'\n  - 'MooseX::ClassAttribute'\n  - 'MooseX::Types'\n  - 'MooseX::Role::Parameterized'\n  - 'MooseX::Aliases'\n  # Biology\n  - 'Ace'\n  - 'Bio::ASN1::EntrezGene'\n  - 'Bio::Graphics'\n  - 'Bio::Phylo'\n  - 'http://www.cpan.org/authors/id/C/CJ/CJFIELDS/BioPerl-1.6.901.tar.gz'\n  - 'Bio::Das'\n  - 'Bio::GFF3::LowLevel'\n# Currently have problems building on CentOS\n#  - 'AnyData'\n#  - 'Convert::BinHex'\n#  - 'XML'\n#  - 'XML::LibXSLT'\n#  - 'Modern::Perl'\n#  These need samtools and Jim Kent's source trees so need to be done by hand\n#  - 'Bio::DB::Sam'\n#  - 'Bio::DB::BigFile'\n"
  },
  {
    "path": "config/puppet/modules/.gitkeep",
    "content": ""
  },
  {
    "path": "config/puppet_classes.yaml",
    "content": "minimal:\nbiocloudcentral:\n  # Idea is still being developed, I wonder if actually defining a puppet class\n  # like cloudbiolinux::biocloudcentral that in turn brings in these classes might\n  # be a better idea.\n  - \"concat::setup\" # Required for apache\n  - \"apache\"\n  - \"apache::mod::wsgi\"\n  - \"biocloudcentral\"\n  - \"biocloudcentral::apache\"\n\n## Uncomment following lines to add LWR to CloudMan meta-package.\n#cloudmn:\n#  - lwr\n"
  },
  {
    "path": "config/python-libs.yaml",
    "content": "---\n# Configuration file defining python specific libraries on pypi to install\n# using easy install\npypi:\n  # infrastructure\n  - setuptools\n  - amqplib\n  - BeautifulSoup\n  - fabric\n  - Logbook\n  - lxml\n  - mako\n  - paste\n  - pastescript\n  - pycrypto\n  - python-cjson\n  - pyOpenSSL\n  - routes\n  - simplejson\n  - sphinx\n  - StarCluster \n  - webhelpers\n  - webob\n  - xlrd\n  # build\n  - Pyrex-real\n  - cython  # needed for installing pybedtools\n# concurrency/web\n  - Twisted\n  # science\n  - cutadapt\n  - ruffus\n  - DendroPy\n  - HTSeq\n  - pybedtools\n  - pygr\n  - PyVCF\n  - rpy2\n  - ramp\n  # databases\n  - CouchDB\n  - MySQL-python\n  - pymongo\n  - pysqlite\n  - psycopg2\n  # visualization\n  - gnuplot-py\n  - PIL\n  - pypdf\n  - reportlab\n  # cloudman\n  - oca\n  - hoover\n  - Mako\nconda:\n  - boto\n  - biopython\n  - cython\n  - h5py\n  - ipython\n  - matplotlib\n  - networkx\n  - nose\n  - numpy\n  - pandas\n  - pysam\n  - pyyaml\n  - pyzmq\n  - scikit-learn\n  - scipy\n  - SQLAlchemy\n  - statsmodels\n  - tornado\n  \n## compile problems\n  #- cogent\n  #- tables\n  #- pygraphviz\n  #- pysparse\n  #- sympy\n  #- pydot # references python 3-only pyparsing\n  #- pygooglechart\n  #- rpy2-bioconductor-extensions\n"
  },
  {
    "path": "config/r-libs.yaml",
    "content": "---\n# Configuration file defining R specific libraries that are installed \n# via CRAN and Bioconductor.\ncranrepo: http://cran.fhcrc.org/\ncran:\n - abind\n - ape\n - aplpack\n - biglm\n - BiodiversityR\n - bitops\n - boot\n - car\n - caret\n - caTools\n - chron\n - class\n - cluster\n - codetools\n - coin\n - colorspace\n - date\n - DBI\n - devtools\n - digest\n - effects\n - FactoMineR\n - flashClust\n - foreign\n - gdata\n - gee\n - graph\n - ggplot2\n - gplots\n - gsalib\n - gtools\n - hdf5\n - hexbin\n - hmisc\n - HH\n - hwriter\n - kernlab\n - kernsmooth\n - knitr\n - lattice\n - latticeExtra\n - leaps\n - lmtest\n - maptree\n - markdown\n - mass\n - matrix\n - mclust\n - mgcv\n - modeltools\n - multcomp\n - multicore\n - nlme\n - NMF\n - nnet\n - pamr\n - pi0\n - plotrix\n - plyr\n - proto\n - qvalue\n - rcmdr\n - relimp\n - reshape\n - reshape2\n - rgl\n - RColorBrewer\n - RCurl\n - rpart\n - rjson\n - Rmpi\n - RMySQL\n - RSQLite\n - sandwich\n - sm\n - snowfall\n - spatial\n - sqldf\n - statmod\n - stringr\n - strucchange\n - survival\n - testthat\n - vegan\n - vgam\n - VennDiagram \n - vr\n - XML\n - xtable\n - yacca\n - yaml\n - zoo\n\nbiocrepo: http://bioconductor.org/biocLite.R\nbioc:\n - affxparser\n - affy\n - affydata\n - affyio\n - affyPLM\n - affyQCReport\n - annaffy\n - annotate\n - AnnotationDbi\n - aroma.light\n - ArrayExpress\n - arrayQualityMetrics\n - ArrayTools\n - BayesPeak\n - baySeq\n - beadarray\n - Biobase\n - BiocInstaller\n - bioDist\n - biomaRt\n - Biostrings\n - BSgenome\n #- BSgenome.Athaliana.TAIR.TAIR9\n #- BSgenome.Celegans.UCSC.ce6\n #- BSgenome.Dmelanogaster.UCSC.dm3\n #- BSgenome.Hsapiens.UCSC.hg19\n #- BSgenome.Mmusculus.UCSC.mm9\n #- BSgenome.Scerevisiae.UCSC.sacCer3\n - ChIPpeakAnno\n - chipseq\n - ChIPseqR\n - ChIPsim\n - CNAnorm\n - cn.mops\n - CSAR\n - cummeRbund\n - DEGseq\n - DESeq\n - DEXSeq\n - DiffBind\n - DNAcopy\n - DynDoc\n - edgeR\n - EDASeq\n - exomeCopy\n - gage\n - gcrma\n - genefilter\n - geneplotter\n - GenomeGraphs\n - genomeIntervals\n - GenomicFeatures\n - GenomicRanges\n - Genominator \n - GEOquery\n - GGBase\n - GGtools\n - ggbio\n - girafe\n - globaltest\n #- GO.db\n - GOFunction\n - goseq\n - GOstats\n - GSEABase\n - GWASTools\n - hgu95av2.db\n - HilbertVis\n - htSeqTools\n - impute\n - IRanges\n - KEGG.db\n - limma\n - lumi\n - makecdfenv\n - marray\n - MEDIPS\n - multtest\n - MotIV\n - mzR\n - oneChannelGUI\n #- org.At.tair.db\n #- org.Ce.eg.db\n #- org.Dm.eg.db\n #- org.Hs.eg.db\n #- org.Mm.eg.db\n #- org.Sc.sgd.db\n #- org.Xl.eg.db\n - PAnnBuilder\n - preprocessCore\n - PICS\n - qpgraph\n - qrqc\n - rGADEM\n - R453Plus1Toolbox\n - RBGL\n - Repitools\n - rGADEM\n - Rgraphviz\n - Ringo\n - rnaSeqMap\n - Rolexa\n - Rsamtools\n - Rsubread\n - rtracklayer\n - segmentSeq\n - seqbias\n - seqLogo\n - ShortRead\n - siggenes\n - simpleaffy\n - snpMatrix\n - snpStats\n - splots\n - SRAdb\n - tkWidgets\n - VariantAnnotation\n - vsn\n - widgetTools\n - xcms\n - zlibbioc\n"
  },
  {
    "path": "config/ruby-libs.yaml",
    "content": "---\n# Configuration file defining ruby libraries to install\n# using gem install (avoiding the distribution package manager)\n#\n# Gems are installed for Ruby1.9\ngems:\n  - addressable\n  - cake\n  - cheat\n  - fastercsv\n  - hpricot\n  - htmlentities\n  - json\n  - kwalify\n  - libxml-ruby\n  - uuidtools\n  - xml-simple\n  # science\n  - bio\n  - bio-bwa\n  - bio-gff3\n  - biomart\n  #- bio-biolinux # error: uninitialized constant Syck::Syck\n  # databases\n  - couchdb-ruby\n  - data_objects\n  - dm-adjust\n  - dm-aggregates\n  - dm-ar-finders\n  - dm-cli\n  - dm-constraints\n  - dm-core\n  - dm-is-list\n  - dm-is-nested_set\n  - dm-is-remixable\n  - dm-is-searchable\n  - dm-is-state_machine\n  - dm-is-tree\n  - dm-is-versioned\n  - dm-migrations\n  - dm-observer\n  - dm-tags\n  - dm-timestamps\n  - dm-validations\n  - do_sqlite3\n  - datamapper\n  - dm-rest-adapter\n  - dm-serializer\n  - dm-types\n  # Require up to date ruby. Fail on Centos default\n  - vagrant\n  - mysql\n  - do_mysql\n  # sqlite3-ruby is renamed with sqlite3\n  - sqlite3\n  # Problems building on 1.9.1\n  #- dm-ferret-adapter\n  #- dm-sweatshop\n  #- bio-blastxmlparser\n  #- ruby-ensembl-api\n"
  },
  {
    "path": "contrib/__init__.py",
    "content": "\"\"\"Module containing user contributions\n\"\"\"\n"
  },
  {
    "path": "contrib/flavor/__init__.py",
    "content": "\"\"\"BioLinux flavors\"\"\"\n"
  },
  {
    "path": "contrib/flavor/biocloudcentral/main.yaml",
    "content": "packages:\n - minimal\n - biocloudcentral\n"
  },
  {
    "path": "contrib/flavor/biopython/custom.yaml",
    "content": "---\nbio_general:\n  - anaconda\nbio_nextgen:\n"
  },
  {
    "path": "contrib/flavor/biopython/fabricrc.txt",
    "content": "# Configuration file for Fabric specifying filesystem information\n# This can be adjusted and passed to fab on the commandline as '-c'\n#\n# For example\n#\n#   fab -f data_fabfile.py -H your_machine -c your_fabricrc.txt install_data_s3:your_biodata.yaml\n\n# -- Details about the operating system\ndistribution = __auto__\ndist_name = __auto__\n\n# -- Installation details\n\n# Global installation directory for packages and standard programs\nsystem_install = ~/biopython\n# Local install directory for versioned software that will not\n# be included in the path by default\nlocal_install = ~/biopython/share\n\n# Global setting for using sudo; allows installation of custom packages\n# by non-privileged users.\n# *Note*: ``system_install`` needs to point to a user-writeable directory if\n# ``use_sudo`` is set to ``False``\nuse_sudo = True\nkeep_isolated = True\n\n# Set to get CloudBioLinux customizations like setup scripts;\n# or set to different edition to get specific customizations\n#edition = cloudbiolinux\n\n# Whether to clean the system for EC2 AMI preparation;\n# do not set on vagrant or local machines\nis_ec2_image = false\n\n# Shell to be used by CBL scripts during runtime\nshell_config = ~/.bashrc\nshell = /bin/bash -i -c\n"
  },
  {
    "path": "contrib/flavor/biopython/main.yaml",
    "content": "---\n# Flavor containing with minimal instructions to install tools for\n# running next-generation sequencing pipelines.\npackages:\n  - minimal\n  - libraries\n  - python\n  - ruby\n  - bio_general\n  - bio_alignment\n  - bio_search\n  - bio_nextgen\nlibraries:\n  - python-libs"
  },
  {
    "path": "contrib/flavor/biopython/packages-homebrew.yaml",
    "content": "# Packages available in the Homebrew and Linuxbrew package manager\n---\nbio_alignment:\n  - clustal-w\n  - clustal-omega\n  - mafft\nbio_nextgen:\n  utilities:\n    - bedtools\n  analysis:\n    - samtools"
  },
  {
    "path": "contrib/flavor/biopython/packages.yaml",
    "content": "---\nminimal:\n  build:\n    - cmake\n    - gcc\n    - g++\n    - gfortran\n    - libtool\n    - make\n    - patch\n    - pkg-config\n  util:\n    - unzip\n  network:\n    - curl\n    - wget\nlibraries:\n  - libzip-dev\n  - libbz2-dev\n  - zlib1g-dev\n  - libcurl4-openssl-dev\n  - libgsl0-dev\n  - libgsl0ldbl\n  - gsl-bin\n  - libncurses5-dev\n  - libpcre3-dev\n  - libpng12-dev\n  - libreadline-dev\n  - libssl-dev\n  - libxml2\n  - libxml2-dev\n  - zlib1g-dev\npython:\nruby:\n  - ruby1.9.1 # default Ruby install\n  - ruby1.9.1-dev\nbio_general:\n  - primer3\n  - emboss # Broken on 13.10\n  - emboss-data # Broken on 13.10\n  - emboss-lib # broken on 13.10\nbio_search:\n  - ncbi-blast+\nbio_alignment:\nbio_nextgen:"
  },
  {
    "path": "contrib/flavor/biopython/python-libs.yaml",
    "content": "---\npypi:\n  - MySQL-python\n  - psycopg2\n  - rdflib\n  - reportlab\nconda:\n  - ipython\n  - matplotlib\n  - numpy\n  - pandas"
  },
  {
    "path": "contrib/flavor/boinc/__init__.py",
    "content": "\"\"\"Boinc flavor\n\n   Copyright (C) 2011 Pjotr Prins <pjotr.prins@thebird.nl> and Steffen Moeller <moeller@debian.org>\n\n\"\"\"\n\n"
  },
  {
    "path": "contrib/flavor/boinc/boincflavor.py",
    "content": "from fabric.api import *\nfrom fabric.contrib.files import *\n\nfrom cloudbio.flavor import Flavor\n\nfrom cloudbio.custom.shared import (_fetch_and_unpack)\n\nclass BoincFlavor(Flavor):\n    \"\"\"A VM flavor for running Boinc\n    \"\"\"\n    def __init__(self, env):\n        Flavor.__init__(self,env)\n        self.name = \"Boinc Flavor\"\n\n    def rewrite_config_items(self, name, packages):\n        if name == 'packages':\n          packages += [ 'openssh-server', 'unzip', 'tar', 'sudo' ]\n        for package in packages:\n          env.logger.info(\"Selected: \"+name+\" \"+package)\n        return packages\n\n    def post_install(self):\n        env.logger.info(\"Starting post-install\")\n        pass\n\nenv.flavor = BoincFlavor(env)\n"
  },
  {
    "path": "contrib/flavor/boinc/fabricrc_debian.txt",
    "content": "edition = minimal\nedition_version = 0.10\nflavor_path = boinc.boincflavor\ndistribution = debian\ndist_name = squeeze\n\n# Update to the target's nearest repository:\ndebian_repository=http://ftp.nl.debian.org/debian/\n\n# Global installation directory for packages and standard programs\nsystem_install = /usr\nlocal_install = ~/install\nshell_config = ~/.bashrc\nshell = /bin/bash -l -c\n# data_files = /mnt/biodata\n\n"
  },
  {
    "path": "contrib/flavor/boinc/main.yaml",
    "content": "---\npackages:\n  # - minimal : skip, as we don't need the build system\n  - boinc\n  - desktop-cloud\nlibraries:\n"
  },
  {
    "path": "contrib/flavor/cloudman/README.md",
    "content": "# CloudMan Flavors\n\nThis document briefly describes the CloudMan/[Galaxy][10] flavors of\nCloudBioLinux available and how to configure them. For quick, small\nmodifications to CloudMan your best bet is probably to modify an existing\ninstance, documentation on how to do this can be found [here][9].\nDocumentation for building a CloudMan image and corresponding volumes and\nbucket from scratch using the CloudBioLinux deployer can be found [here][8].\n\n## Core Flavors\n\nThis directory contains package lists used to setup [CloudMan][1] and/or CloudMan\nand [Galaxy][2]. A minimal CloudMan machine can be configured with the following command:\n\n         fab -f fabfile.py -i <key> -H ubuntu@<IP> install_custom:cloudman\n\nA slightly more flushed out instance can be installed via the command:\n\n        fab -f fabfile.py -i <key> -H ubuntu@<IP> install_biolinux:flavor=cloudman/cloudman\n\nFinally, CloudMan and Galaxy can be installed together via the command:\n\n        fab -f fabfile.py -i <key> -H ubuntu@<IP> install_biolinux:flavor=cloudman/cloudman_and_galaxy\n\nYou can additionally configure your CloudMan and Galaxy instance by specifying\na configuration file: \"-c <your_fabricrc.txt>\" in the above command.\n\nA subset of the parameters you may override via this mechanism includes (see\n``config/fabricrc.txt`` for a full list):\n\n* `galaxy_user` (default `galaxy`): User of Galaxy webapp\n* `galaxy_home` (default `/mnt/galaxyTools/galaxy-central`): Galaxy installation directory\n* `galaxy_tools_dir` (default `/mnt/galaxyTools/tools`): Galaxy tool's directory, formally called `install_dir` in mi-deployment's `tools_fabfile.py` file.\n* `galaxy_loc_files` (default `/mnt/galaxyIndices`): Directory for installation of Galaxy loc files.\n* `galaxy_repository` (default `https://bitbucket.org/galaxy/galaxy-central/`): This is the mercurial repository of Galaxy to install into.\n* `galaxy_preconfigured_repository` (default `False`): If this is `True`, CloudBioLinux will not tweak the repository pulled in via mercurial for CloudMan. This is for applications which prepackage the needed changes to Galaxy directly into the Mercurial repository.\n* `galaxy_cloud` (default `True`): If this is `True`, the Galaxy installation will be preconfigured for SGE and contain CloudMan branding.\n\n## Galaxy Extensions\n\n### Galaxy-P Flavors\n\nThere are two additional flavors assembled for the [Galaxy-P][4] project. \n\nBelow are some recommended fabricrc.txt overrides for Galaxy-P.\n  \n* `dist_name = precise`\n* `galaxy_repository = https://bitbucket.org/galaxyp/galaxyp-central/`\n* `galaxy_tool_conf = /path/to/cloudbiolinux/contrib/flavor/proteomics/galaxyp/galaxy_tools.yaml`\n\nSide Note: Galaxy-P is actively developed, tested, and deployed with Ubuntu 12.04\nLTS, so this is what to target for best results. Feel free to let me\n(jmchilton@gmail.com) know if there are issues or if you would like\nhelp deploying in other environments and I will attempt to help in\nwhatever way I can.\n\n\n#### The Galaxy-P Server Flavor\n\nThis flavor is used to build the internal and [public][5] Galaxy-P servers\nhosted on the OpenStack cloud at the Minnesota Supercomputing Institute,\nthough can of course be used to build your own Galaxy-P environment. This\nflavor is a sort of stripped down proteomics environment for Galaxy.\n\n        fab -f fabfile.py -i <key> -H ubuntu@<IP> install_biolinux:flavor=cloudman/cloudman_and_galaxyp\n\nThis flavor additionally requires a wine environment to be packaged using the\n[proteomics-wine-env][7] project and made available to CloudBioLinux via the\n`setup_proteomics_wine_env_script` fabricrc property.\n\n#### The Galaxy-P Desktop Flavor\n\nThe Galaxy-P desktop flavor builds on the server flavor with additional\ndesktop tools and programming libraries designed to make it a broadly useful\nenvironment for mass spec data analysis even outside the context of Galaxy\nwhile also stripping out the wine related tools to avoid potential legal\nconcerns associated with redistributing such tools.\n\nAn Amazon AMI of this flavor will be available for launch on MSI's\n[BioCloudCentral instance][6].\n\n        fab -f fabfile.py -i <key> -H ubuntu@<IP> install_biolinux:flavor=cloudman/cloudman_desktop_and_galaxyp\n\n\n[1]: http://usecloudman.org/\n[2]: http://usegalaxy.org/\n[3]: http://cloudbiolinux.org/\n[4]: http://getgalaxyp.org/\n[5]: https://usegalaxyp.org/\n[6]: https://biocloudcentral.msi.umn.edu/\n[7]: https://github.com/jmchilton/proteomics-wine-env\n[8]: https://github.com/chapmanb/cloudbiolinux/blob/master/deploy/cloudman.md\n[9]: http://wiki.galaxyproject.org/CloudMan/CustomizeGalaxyCloud\n[10]: http://galaxyproject.org\n"
  },
  {
    "path": "contrib/flavor/cloudman/cloudman/main.yaml",
    "content": "---\npackages:\n  - minimal\n  - cloudman\nlibraries:\n"
  },
  {
    "path": "contrib/flavor/cloudman/cloudman_and_galaxy/main.yaml",
    "content": "---\npackages:\n  - minimal\n  - cloudman\n  - galaxy\nlibraries:\n"
  },
  {
    "path": "contrib/flavor/cloudman/cloudman_and_galaxyp/main.yaml",
    "content": "---\npackages:\n  - minimal\n  - cloudman\n  - galaxy\n  - galaxyp\n  - bio_proteomics_wine\nlibraries:\n"
  },
  {
    "path": "contrib/flavor/cloudman/cloudman_desktop_and_galaxyp/main.yaml",
    "content": "---\npackages:\n  - minimal\n  - ruby\n#  - python\n#  - databases\n  - cloudman\n  - galaxy\n  - galaxyp\n  - java\n  - desktop\nlibraries:\n  - ruby-libs\n  - r-libs"
  },
  {
    "path": "contrib/flavor/cloudman/cloudman_desktop_and_galaxyp/ruby-libs.yaml",
    "content": "---\n# Configuration file defining ruby libraries to install\n# using gem install (avoiding the distribution package manager)\n#\n# Gems are installed for Ruby1.9\ngems:\n  # Library for working with mass spectrometry data\n  - mspire  \n"
  },
  {
    "path": "contrib/flavor/cloudman/migration_checklist.md",
    "content": "This document is meant to layout work that is to be done and/or that\nhas been done in merging mi-deployment and galaxy-vm-launcher into\ncloudbiolinux.\n\n## Concrete TODO List:\n\n### mi-deployment migration:\n\n* *TODETERMINE*: Is SGE configuration in cloudbiolinux up-to-date with mi-deployment?\n* *TODETERMINE*: Is setuptools install in mi-deployment needed with cloudbiolinux? (Seems no. -John)\n* *TODO*: Port mi-deployment s3fs install functionality to cloudbiolinux.\n* *TODETERMINE*: Is _configure_postgresql functionality needed? Seems to work without it.\n* *TODO*: Port mi-deployment _configure_galaxy_env functionality to cloudbiolinux.\n* *TODO*: Port mi-deployment _save_image_conf_support functionality to cloudbiolinx\n* *TODO*: Move required files for _configure_logrotate to cloudbiolinux installed_files and update setup procedure accordingly.\n* *TODO*: Move required files for proftpd to cloudbiolinux installed_files and update setup procedure accordingly.\n* *TODO*: Move nginx_errdoc.tar.gz to cloudbiolinux installed_files and update setup procedure accordingly.\n* *TODO*: Move required files for vimrc to cloudbiolinux installed_files and update setup procedure accordingly.\n* *TODO*: Port mi-deployment volume_manipulations_fab.py functionality to cloudbiolinux (if makes sense ).\n* *TODO*: Port mi-deployment instance-to-ebs-ami.sh functionality to cloudbiolinux (if makes sense ).\n* *TODO*: Port mi-deployment copy_snap/local_to_ebs_fab.py functionality to cloudbiolinux (if makes sense ).\n* *TODO*: Port mi-deployment wrf, graphlab, and tools/* functionality to cloudbiolinux (if makes sense ).\n\n### galaxy-vm-launcher migration:\n\n* *TODO*: Move these procedures into cloudbiolinux - start_nginx (all disabled by default to ensure cloudman compat.).\n* *TODO*: Refactor deploy.py to use cloudbio/galaxy/tools instead of gvl/lib/tools.py\n* *TODO*: Refactor galaxy.py to install galaxy via cloudbiolinux methods.\n* *TODO*: Move this functionality into cloudbiolinx - galaxy database migrations (disabled by default to ensure cloudman compat.)\n* *TODETERMINE*: Could we move setup_taxonomy_data from gvl/lib/deploy.py into cloudbio/biodata somewhere? How has cloudman been configuring this data?\n\n### Completed TODOs from original checklist\n\n* *TODO*: Migrate _init_postgresql_data\n* *TODO*: Migrate _configure_nginx_service\n* *TODETERMINE*: Is install_nginx in cloudbiolinux up-to-date with mi-deployment?\n* *TODO*: Refactor install_nginx in cloudbio.custom.cloudman so it can be used by \ngalaxy-vm-launcher\n* *TODO*: Determine and implement good way to make cloudman specific parts of nginx.conf optional.\n* *TODO*: Port mi-deployment _configure_xvfb functionality to cloudbiolinx\n* *TODO*: Move this functionality into cloudbiolinx - setting up galaxy init service, log (disabled by default to ensure cloudman compat.)\n\n## mi-deployment\n\n* data_fabfile.py - Seems all the work was already been done.\n\n* ec2autorun.py - Ported with the same default behavior as of 8/12/12,\n  but optional user data extensions to handle more galaxy-vm-launcher \n  style use cases.\n\n* instance-to-ebs-ami.sh - No work done?\n\n* mi-fabile.py - \n\n  * apt package installations - Ported over and update to date as of\n    8/12/12.\n\n  * setting up users - Ported over and update to date as of 8/12/12.\n\n  * install nginx - Ported over at some point. TODO: Determine if the\n    cloudbiolinux stuff is up-to-date. \n    \n    Extension: The cloudbiolinux version has been extended to allow\n    parameterization of the Galaxy webapp installation directory.\n\n  * Configure SGE - Ported over. TODO: Determine if the cloudbiolinux\n    stuff is up-to-date.\n\n  * Install setuptools - This doesn't seem to be needed with\n    cloudbiolinux, at least with Ubuntu 12.04.\n\n  * Install s3fs - TODO: Port this functionality.\n \n  * _configure_postgresql - Is this really needed? Have the same code in \n    galaxy-vm-launcher, but Cloudman seems to working without this cleanup. \n\n  * _install_proftpd - Ported over, up-to-date as of 8/12/12. Should move \n    configuration files to installed_files instead of mi-deployment.\n\n  * _configure_ec2_autorun - Ported over and up-to-date as of 8/12/12.\n\n  * _configure_sge - Ported over and up-to-date as of 8/12/12.\n\n  * _configure_galaxy_env - TODO: Port this functionality.\n\n  * _configure_nfs - Ported over and seems functionally equivalent as of 8/12/12.\n\n  * _configure_logrotate - Files should be moved into cloudbiolinux/installed_files,\n    but this is functionally equivalent as of 8/12/12.\n\n  * _save_image_conf_support - TODO: Port this functionality.\n  \n  * _configure_xvfb - TODO: Port this functionality. \n\n  * _configure_bash - Ported over as part of cloudbio.cloudman's _setup_env method\n    and up-to-date \n\n* nginx.conf - Ported over (8/12/12), extended to allow \n  parameterization of galaxy_home path.\n* nginx_errdoc.tar.gz - Right now this is fetched from mi-deployment, \n  should be stored in cloudbiolinux.\n* tools_fabfile.py - Ported over and up-to-date as of 8/12/12.\n  Functionality split into installing Galaxy and installing actual tool dependencies.\n\n  To install Galaxy the way tools_fabfile does, install cloudman via:\n  fab -f fabfile.py -i <key> -H ubuntu@<IP> install_biolinux:packagelist=./contrib/cloudman/cloudman_and_galaxy.yaml  \n  instead of :\n  fab -f fabfile.py -i <key> -H ubuntu@<IP> install_custom:cloudman\n  \n  Several extension points on top of mi-deployment have been added\n  (configured via fabricrc options), but the default behavior should\n  be the same. These extensions include, allowing deployer to set:\n\n  * galaxy_repository: Which repo to point at.\n  * galaxy_preconfigured_repository: Override default of true to skip the \n    tweaking of the installed galaxy repository that tools_fabfile does. The\n    galaxy-vm-launcher approach is to include these changesets \n    https://bitbucket.org/jmchilton/cloud-galaxy-dist\n    in the repository Galaxy being installed from.\n  * galaxy_conf_directory: Use this work \n    https://bitbucket.org/galaxy/galaxy-central/pull-request/44/allow-usage-of-directory-of-configuration\n    to allow overridding specific galaxy universe_wsgi.ini properties at image configuration time.\n    Any fabric environment properties of the form: galaxy_universe_XXXX=YYYY will end up at runtime\n    as Galaxy universe_wsgi.ini properties of the form XXXX=YYYY inside [app:main]. My extensions to CloudMan also allow setting properties via this mechanism from user data at startup time. By default, cloudbiolinux properties will have a priority of 200 and cloudman properties a priority of 400, so if there are conflicts, CloudMan's startup time properties will override those of CloudBioLinux's. \n  \n  The remaining functionality of installing Galaxy application and\n  R/Bioconductor dependencies is available, but turned off by\n  default. To enable these, update the fabric enviornment to set\n  galaxy_install_dependencies=true and galaxy_install_r_packages. This\n  functionality has been extended to dynamically read which packages\n  and versions to install from the file: contrib/cloudman/tools.yaml.\n  Multiple versions of the same tool can be installed via this\n  mechanism. I have updated various software versions and tweaked the\n  install methods to work under Ubuntu 12.04 for all tools.\n\n* volume_manipulations_fab.py - No work has been done on this. This\n  file has not been updated in a while, is it still useful?\n\n* xvfb_default and xvfb_init - TODO: Port these files and mi-deployments' \n  configure_xvfb functionality over.\n\n* conf_files/proftp.conf, conf_files/proftp.initd - Right now this is fetched from mi-  deployment, should be stored in cloudbiolinux.\n\n* conf_files/vimrc - Right now this is fetched from mi-deployment, \n  should be stored in cloudbiolinux.\n\n* conf_files/apps.yaml, conf_files/config.yaml - The cloudman and\n  galaxy dependencies have been ported over and will be installed\n  using cloudbiolinux's similar package management configuration\n  mechanisms. The extensions (wrf, graphlab) have not been.\n\n* conf_files/cloudman.logrotate  - File still be fetched from mi-deployment, \n  this should be read from installed_files instead.\n\n* copy_snap/local_to_ebs_fab.py - No work has been done on this. This\n  file has not been updated in a while, is it still useful?\n\n* tools/ - No work on the custom tools has been done. Are there more\n  cloudbiolinux-y ways to handle these?\n\n## galaxy-vm-launcher\n\n* lib/genomes.py - Uses cloudbiolinux alread. TODO: Just delete out of \n  galaxy-vm-launcher.\n\n* lib/image.py - \n\n  Setting up users and installing packages is good to go.\n\n  install_nginx in cloudbio/custom/cloudman should be refactored to a common shared place, but still invoked in install_cloudman. A seperate nginx conf file should be optional for galaxy-vm-launcher or we should find a good way to make the cloudman \n  paths optional (and enabled by default).\n\n  _configure_postgresql - I don't think this is needed except maybe allowing \n  overridding of postgresql.conf etc file.\n\n  _init_postgresql_data, _configure_nginx_service, start_nginx - This should all\n  be ported over but disabled by default (for cloudman compatiablity).\n\n  configure_xvfb - TODO: Configure this.\n\n* lib/tools.py - Mostly merged already. galaxy-vm-launcher should be refactored \n  to use cloudbio/galaxy/tools.\n\n* lib/galaxy.py - \n\n  * Setting up galaxy and options (functionality mostly available in cloudbiolinux now gvl needs to be refactored to use it.)\n\n  * Setting up galaxy service, log, and database migrations. Functionality could be move to cloudbio/galaxy and disabled by default (not compatiable with cloudman).\n\n  * Seeding galaxy with data. Functionality should remain in the galaxy-vm-launcher, \n  not really compatiable with cloudman. galaxy-vm-launcher should be refactored to\n  use blend though and possibily some fork of blend with additional methods that \n  directly interact with the database the way galaxy-vm-launcher does.\n\n* lib/deploy.py - File should largely remain as is.\n  \n  setup_taxonomy_data - Could be moved into cloudbio/biodata. Does cloudman use \n  the metagenomics tools? How is this currently being configured?\n\n"
  },
  {
    "path": "contrib/flavor/cloudman/tools.yaml",
    "content": "---\napplications:\n  ## Coming from bio_nextgen.py\n  # bwa: \"0.5.9\" # ToolShed installed\n  ucsc_tools: \"default\"\n  # bowtie: \"0.12.7\" # ToolShed installed\n  samtools:\n    - version: \"0.1.16\"\n  # abyss: \"1.3.3\" # Compile error\n  bfast: \"0.7.0a\"\n  velvet: \"1.2.07\"\n  # perm: \"3.6\" # Compile error\n  lastz: \"1.02.00\"\n  mosaik: \"default\" # Version ignored now, CBL pulls from trunk\n  # freebayes: \"default\" # ToolShed installed\n  macs: \"1.4.2-1\"\n\n  ## Coming from bio_general.py\n  emboss:\n    - version: \"6.5.7\"\n      symlink_versions: \"5.0.0\"  # Setup a symbolic link, so when Galaxy looks for 5.0.0 it finds newest version\n\n  ## Coming from applications.py\n\n  ## Tried with bio_nextgen didn't work. [ubuntu@10.1.0.3] out: No package 'gtextutils' found\n  fastx_toolkit: \"0.0.13.2\"\n  ## Tried with bio_nextgen Eventually got bam to work, not eigen dependencies...\n  ## switching to version 2.0.0 didn't help\n  cufflinks:\n    - \"1.3.0\"\n    - \"2.0.2\"\n  tophat:\n    - \"1.3.3\"\n    - \"2.0.6\"\n\n  picard: \"1.56\"\n  fastqc: \"0.10.0\"\n  gatk: \"2.3-9-gdcdccbb\"\n\n  # maq: \"0.7.1\" # Compile error\n  megablast: \"2.2.22\"\n  blast: \"2.2.25+\"\n  sputnik: \"r1\"\n  taxonomy: \"r2\"\n  add_scores: \"r1\"\n  # hyphy: \"r418\" # Compile error\n  srma: \"0.1.15\"\n  beam: \"2\"\n  pass: \"2.0\"\n  lps_tool: \"2010.09.30\"\n  plink: \"1.07\"\n  haploview: \"4.2b\"\n  # eigenstrat: \"3.0\" # Compile error\n\nr_packages:\n  - DBI\n  - RColorBrewer\n  - RCurl\n  - RSQLite\n  - XML\n  - biglm\n  - bitops\n  - digest\n  - ggplot2\n  - graph\n  - hexbin\n  - hwriter\n  - kernlab\n  - latticeExtra\n  - leaps\n  - pamr\n  - plyr\n  - proto\n  - qvalue\n  - reshape\n  - statmod\n  - xtable\n  - yacca\n\nbioconductor_packages:\n  - AnnotationDbi\n  - ArrayExpress\n  - ArrayTools\n  - Biobase\n  - Biostrings\n  - DynDoc\n  - GEOquery\n  - GGBase\n  - GGtools\n  - GSEABase\n  - IRanges\n  - affy\n  - affyPLM\n  - affyQCReport\n  - affydata\n  - affyio\n  - annaffy\n  - annotate\n  - arrayQualityMetrics\n  - beadarray\n  - biomaRt\n  - gcrma\n  - genefilter\n  - geneplotter\n  - globaltest\n  - hgu95av2.db\n  - limma\n  - lumi\n  - makecdfenv\n  - marray\n  - preprocessCore\n  - ShortRead\n  - siggenes\n  - simpleaffy\n  - snpMatrix\n  - vsn\n"
  },
  {
    "path": "contrib/flavor/cwl_dockers/packages-bcbio-alignment.yaml",
    "content": "bcbio-process_alignment:\n- bwa\n- bwakit\n- novoalign\n- snap-aligner\n- samtools\n- sambamba\n- fgbio\n- umis\n- biobambam\nchannels:\n- bioconda\n- conda-forge\n\n"
  },
  {
    "path": "contrib/flavor/demo/README.md",
    "content": "Small example flavor to demonstrate ability to provide local installations\n"
  },
  {
    "path": "contrib/flavor/demo/custom.yaml",
    "content": "---\nbio_nextgen:"
  },
  {
    "path": "contrib/flavor/demo/fabricrc.txt",
    "content": "# -- Installation details\n\n# Global installation directory for packages and standard programs\nsystem_install = ~/tmp/cbl_demo\n\n# Local install directory for versioned software that will not\n# be included in the path by default\nlocal_install = ~/tmp/cbl_demo/share\n\n# Global setting for using sudo; allows installation of custom packages\n# by non-privileged users.\n# *Note*: ``system_install`` needs to point to a user-writeable directory if\n# ``use_sudo`` is set to ``False``\nuse_sudo = False\nkeep_isolated = True\n\n# -- Details about reference data installation\n\n# Path where biological reference data files should be retrieved to\ndata_files = ~/tmp/cbl_demo/biodata\n"
  },
  {
    "path": "contrib/flavor/demo/main.yaml",
    "content": "---\n# Demonstration flavor for learning use\npackages:\n  - minimal\n  - libraries\n  - python\n  - java\n  - ruby\n  - bio_nextgen"
  },
  {
    "path": "contrib/flavor/demo/packages-homebrew.yaml",
    "content": "# Packages available in the Homebrew and Linuxbrew package manager\n---\nbio_nextgen:\n  alignment:\n    - bwa\n  utilities:\n    - bedtools\n  analysis:\n    - samtools\n  variant:\n    - gatk-framework"
  },
  {
    "path": "contrib/flavor/edx_course/custom.yaml",
    "content": "---\nbio_nextgen:\n  - picard"
  },
  {
    "path": "contrib/flavor/edx_course/edx_setup.sh",
    "content": "#!/bin/bash\nset -eu -o pipefail\n\n# Vagrant installation script for creating VM to use in the\n# edX variant analysis course, PH525.6x:\n# https://www.edx.org/course/case-study-variant-discovery-genotyping-harvardx-ph525-6x\n\n# Base system\nsudo apt-get update\nsudo apt-get install -y build-essential zlib1g-dev wget curl python-setuptools git\nsudo apt-get install -y openjdk-7-jdk openjdk-7-jre ruby libncurses5-dev libcurl4-openssl-dev libbz2-dev unzip pigz bsdmainutils\nsudo apt-get install -y python-pip python-dev\nsudo pip install fabric\n\n# CloudBioLinux\ncd /vagrant\nsudo chown -R vagrant:vagrant /usr/local\ngit clone https://github.com/chapmanb/cloudbiolinux.git\nfab -H localhost install_biolinux:flavor=edx_course\n# Need to hack cmake (to build 32bit): brew uninstall --force cmake && brew install cmake --without-docs\n#\n# GEMINI\nwget https://raw.github.com/arq5x/gemini/master/gemini/scripts/gemini_install.py\npython gemini_install.py /usr/local /usr/local/share/gemini --nosudo --nodata --notools\n\n# Copy snpEff hg19 file from existing bcbio installation\n\n# cleanup\nrm -rf ~/.cache/Homebrew\nrm -rf /usr/local/share/gemini/gemini\n/usr/local/share/gemini/anaconda/bin/conda clean --yes --tarballs\nsudo apt-get clean\nsudo rm -rf /var/lib/apt/lists/*\n\n# On VM -- Compact space: http://andrewdeponte.com/2013/10/29/shrinking-vagrant-linux-boxes.html\nsudo dd if=/dev/zero of=wipefile bs=1024x1024; rm wipefile\n# Afterwards, clone and export:\n# vagrant halt\n# virtualbox\n#  - Select vagrant image\n#  - Machine -> Clone to new VirtualMachine\n#  - File -> Export as OVA\n"
  },
  {
    "path": "contrib/flavor/edx_course/fabricrc.txt",
    "content": "# Configuration file for Fabric specifying filesystem information\n# This can be adjusted and passed to fab on the commandline as '-c'\n#\n# For example\n#\n#   fab -f data_fabfile.py -H your_machine -c your_fabricrc.txt install_data_s3:your_biodata.yaml\n\n# -- Details about the operating system\n\n## If target machine is Ubuntu or CentOS, the following defaults should allow CloudBioLinux to \n## automatically determine the correct settings. For other operating systems (e.g. Debian or ScientificLinux)\n## please override distribution (and dist_name in case of Debian).\ndistribution = __auto__\ndist_name = __auto__\n\n# -- Installation details\n\n# Whether to clean the system for EC2 AMI preparation;\n# do not set on vagrant or local machines\nis_ec2_image = false\n\n# Global installation directory for packages and standard programs\nsystem_install = /usr/local\n\n# Local install directory for versioned software that will not\n# be included in the path by default\nlocal_install = /usr/local/share\n\n# Shell to be used by CBL scripts during runtime\nshell_config = ~/.bashrc\nshell = /bin/bash -i -c\n\nuse_sudo = False\nkeep_isolated = True\n"
  },
  {
    "path": "contrib/flavor/edx_course/main.yaml",
    "content": "---\n# Flavor containing with minimal instructions to install tools for\n# running next-generation sequencing pipelines.\npackages:\n  - minimal\n  - libraries\n  - perl\n  - python\n  - java\n  - r\n  - ruby\n  - bio_nextgen\nlibraries:\n  - python-libs"
  },
  {
    "path": "contrib/flavor/edx_course/packages-homebrew.yaml",
    "content": "# Packages available in the Homebrew and Linuxbrew package manager\n---\nminimal:\n  - cmake\nperl:\nbio_nextgen:\n  alignment:\n    - bwa\n  utilities:\n    - bedtools\n    - fastqc\n    - grabix\n    - sambamba-binary\n    - samblaster\n  analysis:\n    - samtools;--without-curses\n    - htslib\n    - bcftools\n  variant:\n    - freebayes\n    - gatk-framework\n    - metal-gwas\n    - plink\n    - snpeff\n    - vcflib"
  },
  {
    "path": "contrib/flavor/edx_course/python-libs.yaml",
    "content": "---\npypi:\n  - cutadapt"
  },
  {
    "path": "contrib/flavor/globus/main.yaml",
    "content": "---\npackages:\n  - minimal\n  - globus\nlibraries:"
  },
  {
    "path": "contrib/flavor/millstone/README.md",
    "content": "The Millstone flavor of cloudbiolinux eases deploying the Church Lab's\n[Millstone](http://churchlab.github.io/millstone/) software,\na platform for genome design and analysis.\n\nThis document is intended for developers. Most users will want to clone an\nexisting, pre-packaged Amazon Machine Image (AMI), Those instructions are\navailable at the main Millstone webpage here: <http://churchlab.github.io/millstone/>.\n\nThe following instructions provided steps for deploying a (possibly) modified\nversion of Millstone to AWS. The instructions are specific to Amazon, but the\nsteps can be modified for development in different VM environments.\n\nThis document describes how to deploy an instance and create an AMI.\nSee the main cloudbiolinux README.md for general usage instructions for\ncloudbiolinux.\n\n### Usage\n\nThe following steps provide instructions for configuring a master, worker, or\nhybrid instance of Millstone. The MASTER instance runs the main Django web\nserver as well as the Postgresql database and hosts the central RabbitMQ\nmessaging system for distributing computation. A WORKER instance runs a minimal\nweb server to allow the user to connect it to the master instance, and then\nrun asynchronous tasks as they become available.\n\nWhen executing a fabric install, be sure to set either `MASTER=1` or `WORKER=1`, or both in environment variables to specify the role (Master or Worker) of the VM. Setting both runs the web server and background workers on the same machine. See below for example commands.\n\n*NOTE*: The split master/worker implementation is currenlty under development. Only the combined setup has been tested.\n\n#### Steps\n\n##### Local requirements\n\nOn your local machine, install Python-Fabric (and its dep PyYAML), which are\nused by cloudbiolinux to manage deploying to a remote machine.\n\n    pip install fabric\n    pip install pyyaml\n\n##### Commands to load VMs\n\nThe following commands are run from the top-level cloudbiolinux directory\non your local machine and convert a clean EC2 instance to running a combined,\nmaster, or worker configuration of Millstone, respectively.\n\n*NOTE*: Remember to replace both `SSH_KEY_FILE` and `ec2-xx-xx-xx-xx.compute-1.amazonaws.com` with your ec2 private key and the correct instance url.\n\n###### Master/Worker combined VM:\n\n    MASTER=1 WORKER=1 fab -i SSH_KEY_FILE -u ubuntu -H ec2-xx-xx-xx-xx.compute-1.amazonaws.com install_biolinux:flavor=millstone\n\n###### Master VM only (IN PROGRESS)\n\n    MASTER=1 fab -i SSH_KEY_FILE -u ubuntu -H ec2-xx-xx-xx-xx.compute-1.amazonaws.com install_biolinux:flavor=millstone\n\n###### Worker VM (IN PROGRESS)\n\n    WORKER=1 fab -i SSH_KEY_FILE -u ubuntu -H ec2-xx-xx-xx-xx.compute-1.amazonaws.com install_biolinux:flavor=millstone\n\n*DO NOT RESTART* if you want to package an Amazon Machine Image (AMI).\nRestarting will bootstrap the software and data, resulting in potentially\ninsecure/malfunctional configurations. Immediately follow the steps in the\nsection on Building an AMI below.\n\n##### Connecting Workers to Master\n\n*NOTE*: Not yet supported. Draft:\n\nIf you configured separate master and worker instances, you will need to tell\nthe worker instances about the master instance.\n\nThe master info is located on the master instance at:\n\n    http://ec2-xx-xx-xx-xx.compute-1.amazonaws.com/ec2/info\n\nThis shows its hostname and password, two important information that need to\nput into the worker instance.\n\nOn worker instance, add the above information here:\n\n    http://ec2-xx-xx-xx-xx.compute-1.amazonaws.com/worker\n\nYou need to update hostname and password from the master instance, so that the\nworker processes can connect properly. No need to change other fields. That\npage then allows you to test the connection.\n\nThe master will automatically put async tasks on its RabbitMQ and registered\nworkers will consume them. Workers will write back to the master's Postgresql\ndatabase.\n\n##### Building an Amazon Machine Image (AMI)\n\nFirst, clean up remaining SSH keys and history files.\n\n    fab -c contrib/flavor/millstone/ec2.txt -i SSH_KEY_FILE -u ubuntu -H ec2-xx-xx-xx-xx.compute-1.amazonaws.com install_biolinux:flavor=millstone,target=clean\n\nGo to EC2 Dashboard inside AWS Console <<a href=\"https://console.aws.amazon.com/\">console.aws.amazon.com</a>>. Find the instance on which we just installed our software, and select \"Create Image\" in \"Actions\" tab.\n\n##### Creating VM from AMI\n\nInside EC2 Dashboard, select create instance and choose AMI built in previous section. Initial bootstrap will take about five minutes for master VM and one minute for worker VM, before the VM becomes responsive.\n\nIf you instantiate separate master and worker instances, follow the directions\nabove for connecting workers to the master.\n\n### How it works\n\nCloudbiolinux executes a sequence of installation targets listed in `fabfile.py`. Specifically:\n\nTarget `packages` installs system-wide dependencies, defined in `main.yaml` in this directory.\n\nTarget `libraries` installs packages for different languages. We only use it for python packages in `python-libs.yaml`.\n\nTarget `post_install` invokes `installer.py`, which defines installation procedures for millstone. The script clones the millstone repository from Github, sets up jbrowse, and writes a one-time bootstrap script `bootstrap.sh` located in clone directory set by `INSTALLATION_PATH`.\n\n`bootstrap.sh` will not run until first reboot, and will run on startup after an instance is created from AMI. It generates a new password for PostgreSQL and RabbitMQ if running on master VM. This password is saved to `password.txt` along with `bootstrap.sh`.\n\nTo reconfigure, remove `BOOTSTRAPPED` file and re-run `bootstrap.sh`. Note this will generate a new password.\n\n### Thanks\n\nBig thanks to Changping Chen (https://github.com/ccp0101) for designing and implementing the Millstone integration with cloudbiolinux.\n"
  },
  {
    "path": "contrib/flavor/millstone/__init__.py",
    "content": ""
  },
  {
    "path": "contrib/flavor/millstone/custom.yaml",
    "content": "---\njava: []\n\npython: []\n\nmillstone:\n  - unafold\n"
  },
  {
    "path": "contrib/flavor/millstone/installer.py",
    "content": "from fabric.api import *\nfrom fabric.contrib.files import *\nimport os.path\nimport os\n\nINSTALLATION_PATH = \"$HOME/millstone\"\n\nREPO_URL = \"git@github.com:churchlab/millstone.git\"\nREPO_STABLE_COMMIT = \"634db63de2fb275719868839bf44bd9b4b9f016e\"\n\nBOOTSTRAP_SCRIPT = \"\"\"#!/bin/bash\nset -x\nexport RUN_MASTER=%d\nexport RUN_WORKER=%d\nexport PROJECT_DIR=\"%s\"\nexport MILLSTONE_PATH=\"%s\"\nexport CELERY_MANAGER_PATH=\"%s\"\nexport JBROWSE_PATH=\"%s\"\nexport TEMP_FILE_PATH=\"%s\"\nexport PASSWORD_FILE_PATH=\"%s\"\nexport BOOTSTRAP_FINISH_PATH=\"%s\"\nexport RUN_AS_USER=\"%s\"\nexport EC2_HOSTNAME=$(curl http://169.254.169.254/latest/meta-data/public-hostname)\n\nexport MILLSTONE_WEB_PORT=8000\nexport CELERY_MANAGER_WEB_PORT=8001\n\ngenpw() {\n    python -c \"import string,random;print ''.join(random.choice(string.letters + string.digits) for x in range(10))\"\n}\n\ncpucount() {\n    grep -c ^processor /proc/cpuinfo\n}\n\nexport NUM_CPU=$(cpucount)\n\nexport TIMEOUT=3600\n\nif [ \"$RUN_MASTER\" == \"1\" ];\nthen\n    echo \"Configuring master...\"\n\n    export PASSWORD=$(genpw)\n    export RABBITMQ_USER=\"millstone\"\n    export POSTGRES_DB=\"millstone\"\n    export POSTGRES_USER=\"millstone\"\n\n    echo ${PASSWORD} > ${PASSWORD_FILE_PATH}\n    echo \"Using '${PASSWORD}' as password for PostgresSQL and RabbitMQ\"\n\n    /etc/init.d/rabbitmq-server stop\n    cat > /etc/rabbitmq/rabbitmq-env.conf << EOF\nNODE_IP_ADDRESS=0.0.0.0\nNODE_PORT=5672\nNODENAME=\"rabbit@localhost\"\nEOF\n\n    /etc/init.d/rabbitmq-server start\n\n    # Setup user in RabbitMQ and make it public.\n    rabbitmqctl change_password guest $(genpw)\n    RABBITMQ_USERS=$(sudo rabbitmqctl list_users -q)\n    if [[ \"${RABBITMQ_USERS}\" =~ \"${RABBITMQ_USER}\" ]]\n    then\n        echo \"Deleting existing ${RABBITMQ_USER} user in RabbitMQ.\"\n        rabbitmqctl delete_user ${RABBITMQ_USER}\n    fi\n    rabbitmqctl add_user ${RABBITMQ_USER} ${PASSWORD}\n    rabbitmqctl set_permissions -p / ${RABBITMQ_USER} \".*\" \".*\" \".*\"\n\n    /etc/init.d/rabbitmq-server restart\n\n    # Setup user and database in PostgresSQL\n    sudo -u postgres psql -U postgres -d postgres -c \"DROP DATABASE IF EXISTS ${POSTGRES_DB};\"\n    sudo -u postgres psql -U postgres -d postgres -c \"DROP USER IF EXISTS ${POSTGRES_USER};\"\n    sudo -u postgres psql -U postgres -d postgres -c \"CREATE USER ${POSTGRES_USER} WITH PASSWORD '${PASSWORD}';\"\n    sudo -u postgres psql -U postgres -d postgres -c \"CREATE DATABASE ${POSTGRES_DB};\"\n    sudo -u postgres psql -U postgres -d postgres -c \"GRANT ALL PRIVILEGES ON DATABASE ${POSTGRES_DB} to ${POSTGRES_USER};\"\n    sudo -u postgres psql -U postgres -d postgres -c \"ALTER USER ${POSTGRES_USER} CREATEDB;\"\n\n    POSTGRES_CONF=$(find /etc/postgresql -name \"postgresql.conf\" | head -1)\n    echo \"listen_addresses='*'\" >> ${POSTGRES_CONF}\n\n    PG_HDA_CONF=$(find /etc/postgresql -name \"pg_hba.conf\" | head -1)\n    echo \"host all all  0.0.0.0/0 md5\" >> ${PG_HDA_CONF}\n\n    /etc/init.d/postgresql restart\n\n    LOCAL_SETTINGS_PATH=\"${MILLSTONE_PATH}/conf/local_settings.py\"\n\n    if [[ ! -e ${LOCAL_SETTINGS_PATH} ]]\n    then\n        echo \"${LOCAL_SETTINGS_PATH} does not exist!\"\n    fi\n\n    cat > ${LOCAL_SETTINGS_PATH} << EOF\nDATABASES = {\n    'default': {\n        'ENGINE': 'django.db.backends.postgresql_psycopg2',\n        'NAME': '${POSTGRES_DB}',\n        'USER': '${POSTGRES_USER}',\n        'PASSWORD': '${PASSWORD}',\n        'HOST': '127.0.0.1',\n        'PORT': '5432',\n        'OS_USER': 'postgres',\n    }\n}\nBROKER_URL = 'amqp://${RABBITMQ_USER}:${PASSWORD}@127.0.0.1:5672//'\n\nEOF\n    chown ${RUN_AS_USER} ${LOCAL_SETTINGS_PATH}\n\n    echo \"New local settings in ${LOCAL_SETTINGS_PATH}:\"\n    cat ${LOCAL_SETTINGS_PATH}\nfi\n\nchown -R ${RUN_AS_USER} ${PROJECT_DIR}\n\necho \"Reconfiguring nginx and supervisor...\"\n/etc/init.d/supervisor stop\n/etc/init.d/nginx stop\n\nrm -f /etc/nginx/sites-enabled/default\nrm -f /etc/nginx/sites-available/millstone\nrm -f /etc/nginx/sites-enabled/millstone\nrm -f /etc/supervisor/supervisord.conf\n\ncat > /etc/nginx/sites-available/millstone << EOF\nserver {\n  server_name localhost;\n\n  location /jbrowse {\n    alias ${JBROWSE_PATH};\n  }\n\n  location /tmp {\n    alias ${TEMP_FILE_PATH};\n  }\n\n  location /static {\n    alias ${MILLSTONE_PATH}/main/static;\n  }\n\n  location / {\n    proxy_pass http://127.0.0.1:${MILLSTONE_WEB_PORT}/;\n  }\n\n  location /worker/ {\n    proxy_pass http://127.0.0.1:${CELERY_MANAGER_WEB_PORT}/;\n  }\n\n  location /worker/static {\n    alias ${CELERY_MANAGER_PATH}/celery_manager/static;\n  }\n\n  # Override timeouts. Especially relevant to upload requests.\n  proxy_connect_timeout ${TIMEOUT};\n  proxy_read_timeout ${TIMEOUT};\n\n  # No limit on upload size.\n  client_max_body_size 0;\n}\nEOF\n\nif [ \"$RUN_WORKER\" == \"1\" ];\nthen\n    echo \"Configuring worker...\"\nfi\n\n\nln -s /etc/nginx/sites-available/millstone /etc/nginx/sites-enabled/millstone\n/etc/init.d/nginx start\n\ncat > \"/etc/supervisor/supervisord.conf\" << EOF\n[unix_http_server]\nfile=/var/run//supervisor.sock   ; (the path to the socket file)\nchmod=0700                       ; sockef file mode (default 0700)\n\n[inet_http_server]\nport=127.0.0.1:9001\n\n[supervisord]\nlogfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)\npidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)\nchildlogdir=/var/log/supervisor            ; ('AUTO' child log dir, default $TEMP)\n\n[rpcinterface:supervisor]\nsupervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface\n\n[supervisorctl]\nserverurl=http://127.0.0.1:9001\n\n[program:millstone]\ncommand=gunicorn_django -b 127.0.0.1:${MILLSTONE_WEB_PORT} --workers=$(expr ${NUM_CPU} + 1) --timeout=${TIMEOUT}\ndirectory=${MILLSTONE_PATH}\nautostart=${RUN_MASTER}\nautorestart=True\nredirect_stderr=True\nuser=${RUN_AS_USER}\n\n[program:celery_manager]\ncommand=gunicorn_django -b 127.0.0.1:${CELERY_MANAGER_WEB_PORT} --workers=$(expr ${NUM_CPU} + 1) --timeout=${TIMEOUT}\ndirectory=${CELERY_MANAGER_PATH}\nautostart=${RUN_WORKER}\nautorestart=True\nredirect_stderr=True\nuser=${RUN_AS_USER}\n\n[program:celery]\ncommand=python manage.py celery worker --loglevel=info\ndirectory=${MILLSTONE_PATH}\nautostart=${RUN_WORKER}\nautorestart=True\nredirect_stderr=True\nuser=${RUN_AS_USER}\n\nEOF\n\n/etc/init.d/supervisor start\n\nupdate-rc.d nginx defaults\nupdate-rc.d supervisor defaults\n\nif [ \"$RUN_MASTER\" == \"1\" ];\nthen\n    pushd ${MILLSTONE_PATH}\n    sudo -u ${RUN_AS_USER} python manage.py syncdb --noinput\n    sudo -u ${RUN_AS_USER} python manage.py migrate\n    echo \"y\" | sudo -u ${RUN_AS_USER} python scripts/bootstrap_data.py\n    popd\nfi\n\ntouch ${BOOTSTRAP_FINISH_PATH}\n\necho \"Bootstrap finished!\"\n\"\"\"\n\nBOOTSTRAP_INVOKER_SCRIPT = \"\"\"#!/bin/bash\n### BEGIN INIT INFO\n# Provides:          millstone_setup\n# Required-Start:    $all\n# Required-Stop:     $remote_fs $syslog\n# Default-Start:     2 3 4 5\n# Default-Stop:      0 1 6\n### END INIT INFO\n\nexport PROJECT_DIR=\"%s\"\nexport BOOTSTRAP_FINISH_PATH=\"%s\"\nexport BOOTSTRAP_SCRIPT_PATH=\"%s\"\n\n# Carry out specific functions when asked to by the system\ncase \"$1\" in\n  start)\n    if [[ ! -e \"${BOOTSTRAP_FINISH_PATH}\" ]];\n      then\n        /bin/bash ${BOOTSTRAP_SCRIPT_PATH} 2>&1 > \"${PROJECT_DIR}/bootstrap.log\"\n      fi\n    ;;\n  stop)\n    echo \"/etc/init.d/millstone_setup stop\"\n    ;;\n  *)\n    echo \"Usage: /etc/init.d/millstone_setup {start|stop}\"\n    exit 1\n    ;;\nesac\n\nexit 0\n\n\"\"\"\n\n\ndef install_millstone(env):\n    current_user = env.safe_run(\"echo $USER\").strip()\n    home_dir = env.safe_run(\"echo $HOME\").strip()\n    installation_dir = env.safe_run(\"echo %s\" % INSTALLATION_PATH).strip()\n\n    VM_MODE = {\n        'MASTER': os.getenv(\"MASTER\") is not None,\n        'WORKER': os.getenv(\"WORKER\") is not None,\n    }\n    env.logger.info(\"VM_MODE: %r\" % VM_MODE)\n\n    if env.safe_exists(installation_dir):\n        env.logger.warning(\"%s already exists. Removing the directory. \" % installation_dir)\n        with cd(installation_dir):\n            env.safe_sudo(\"rm -rf %s\" % installation_dir)\n\n    env.safe_run(\"mkdir -p %s\" % installation_dir)\n    env.logger.info(\"Installing Millstone to %s\" % installation_dir)\n\n    env.logger.debug(\"Configure SSH to ignore host checking for Github...\")\n    env.safe_run(\"mkdir -p %s\" % os.path.join(home_dir, \".ssh\"))\n    env.safe_run(\"chmod 700 ~/.ssh\")\n    env.safe_run(\"chmod 600 ~/.ssh/*\")\n    append(\"~/.ssh/config\", \"Host github.com\\n\\tStrictHostKeyChecking no\\n\")\n\n    with cd(installation_dir):\n        env.safe_run(\"git clone %s %s\" % (REPO_URL, installation_dir))\n        env.safe_run(\"git reset --hard %s\" % REPO_STABLE_COMMIT)\n\n        project_dir = installation_dir\n        jbrowse_dir = os.path.join(project_dir, \"jbrowse\")\n        genome_designer_dir = os.path.join(project_dir, \"genome_designer\")\n        temp_file_dir = os.path.join(genome_designer_dir, \"temp_data/tmp\")\n        celery_manager_dir = os.path.join(project_dir, \"celery_manager\")\n        config_dir = os.path.join(project_dir, \"config\")\n        assert env.safe_exists(project_dir)\n        assert env.safe_exists(jbrowse_dir)\n        assert env.safe_exists(genome_designer_dir)\n        assert env.safe_exists(celery_manager_dir)\n        assert env.safe_exists(config_dir)\n\n    with cd(project_dir):\n        # clone JBrowse\n        env.safe_run(\"git submodule update --init --recursive\")\n\n    with cd(jbrowse_dir):\n        # Setup JBrowse.\n        env.safe_run(\"./setup.sh\")\n\n    # Install millstone python requirements.\n    python_requirements_file = os.path.join(installation_dir,\n            'requirements', 'deploy.txt')\n    env.safe_sudo(\"pip install -r %s\" % python_requirements_file)\n\n    with cd(genome_designer_dir):\n        env.safe_run(\"ln -s ../jbrowse jbrowse\")\n        env.safe_run(\"./millstone_setup.py\")\n\n    run_worker = 1 if VM_MODE['WORKER'] else 0\n    run_master = 1 if VM_MODE['MASTER'] else 0\n\n    bootstrap_script_path = os.path.join(project_dir, \"bootstrap.sh\")\n    bootstrap_finish_path = os.path.join(project_dir, \"BOOTSTRAPPED\")\n    bootstrap_script = BOOTSTRAP_SCRIPT % (run_master, run_worker,\n        project_dir, genome_designer_dir, celery_manager_dir, jbrowse_dir,\n        temp_file_dir, os.path.join(project_dir, \"password.txt\"),\n        bootstrap_finish_path, current_user)\n\n    append(bootstrap_script_path, bootstrap_script)\n    env.safe_run(\"chmod +x %s\" % bootstrap_script_path)\n\n    \"\"\"\n    /etc/init.d/millstone_setup will check if bootstrap script has\n    run before, and execute bootstrap if not.\n    \"\"\"\n    append(\"/etc/init.d/millstone_setup\", BOOTSTRAP_INVOKER_SCRIPT % (project_dir,\n        bootstrap_finish_path, bootstrap_script_path), use_sudo=True)\n    env.safe_sudo(\"sudo chmod +x /etc/init.d/millstone_setup\")\n    env.safe_sudo(\"sudo update-rc.d millstone_setup defaults\")\n"
  },
  {
    "path": "contrib/flavor/millstone/main.yaml",
    "content": "---\n# NOTE: rabbitmq and postgres installed manually.\npackages:\n  - minimal\n  - libraries\n  - amazon\n  - python\n  - ruby\n  - perl\n  - java\n  - nginx\n  - supervisor\n  - coffeescript\n  - millstone\nlibraries:\n  - python-libs\n"
  },
  {
    "path": "contrib/flavor/millstone/millstoneflavor.py",
    "content": "\"\"\"\nMillstone flavor.\n\"\"\"\n\nfrom fabric.api import *\nfrom fabric.contrib.files import *\n\nfrom cloudbio.custom import shared\nfrom cloudbio.flavor import Flavor\n\nfrom installer import install_millstone\n\n\nclass MillstoneFlavor(Flavor):\n    def __init__(self, env):\n        Flavor.__init__(self, env)\n        self.name = \"Millstone Flavor\"\n\n    def rewrite_config_items(self, name, packages):\n        return packages\n\n    def post_install(self):\n        # Manually install logrotate, postgresql-9.3, rabbitmq-server.\n        # postgresql-9.3 and rabbitmq-server require incompatible versions of\n        # logrotate by default, so we force install the working version.\n        env.safe_sudo(\"apt-get -y --force-yes install logrotate=3.7.8-6ubuntu5\")\n        env.safe_sudo(\"apt-get -y --force-yes install postgresql-9.3\")\n        env.safe_sudo(\"apt-get -y --force-yes install libpq-dev\")\n        env.safe_sudo(\"apt-get -y --force-yes install pgdg-keyring\")\n        env.safe_sudo(\"apt-get -y --force-yes install rabbitmq-server\")\n\n        # Now install psycogpg2, which requires postgresql-9.3 and related\n        # to have been installed first.\n        env.safe_sudo(\"{0} install --upgrade {1}\".format(shared._pip_cmd(env), 'psycopg2'))\n\n        # Finally, install Millstone.\n        install_millstone(self.env)\n\nenv.flavor = MillstoneFlavor(env)\n"
  },
  {
    "path": "contrib/flavor/millstone/python-libs.yaml",
    "content": "---\n# Millstone libaries installed from millstone's requirements/deploy.txt\npypi: []\n"
  },
  {
    "path": "contrib/flavor/minimal/fabricrc_debian.txt",
    "content": "# BioNode configuration file for Fabric specifying filesystem information\n#\n# For example, to configure a minimal install with vagrant (for VirtualBox),\n# assuming you have a configured ./Vagrantfile\n#\n#  fab -H vagrant -f $source/fabfile.py -c $source/contrib/flavor/minimal/fabricrc.txt install_biolinux:packagelist=$source/contrib/flavor/minimal/main.yaml\n#\n#  Updated by Pjotr Prins <pjotr.debian@thebird.nl>\n#\n\nedition_version = 0.93\ndistribution = __auto__\ndist_name = __auto__\nnixpkgs = False\n\n# Update to the target's nearest repository:\ndebian_repository=http://ftp.nl.debian.org/debian/\n\n# Global installation directory for packages and standard programs\nsystem_install = /usr/local\nlocal_install = /usr/local/share\nshell_config = ~/.bashrc\nshell = /bin/bash -i -c\nuse_sudo = True\nis_ec2_image = false\n\n# data_files = /mnt/biodata\n"
  },
  {
    "path": "contrib/flavor/minimal/main.yaml",
    "content": "---\n# Top level configuration file that specifies which groups of programs\n# should be installed. New sections that are added to individual config\n# files should go here. Comment out any groups you don't want to have \n# installed.\n#\n# This is the minimal version to kick-off development, including editors,\n# version control, and basic build support (GNU tools)\npackages:\n  - minimal\n  - ruby\nlibraries:\n"
  },
  {
    "path": "contrib/flavor/neuro/__init__.py",
    "content": "\"\"\"Neuroinformatics flavor\n\"\"\"\n"
  },
  {
    "path": "contrib/flavor/neuro/custom.yaml",
    "content": "---\nbio_nextgen:\n"
  },
  {
    "path": "contrib/flavor/neuro/fabricrc.txt",
    "content": "edition = minimal\nflavor_path = neuro.neuro\ndistribution = ubuntu\nuser = vagrant \n\n# Global installation directory for packages and standard programs\nsystem_install = /usr/local\nlocal_install = ~/install\nshell_config = ~/.bashrc\nshell = /bin/bash -l -c\n# data_files = /mnt/biodata\n"
  },
  {
    "path": "contrib/flavor/neuro/main.yaml",
    "content": "---\npackages:\n    - neuro\nlibraries:\n"
  },
  {
    "path": "contrib/flavor/neuro/neuro.py",
    "content": "from fabric.api import *\nfrom fabric.contrib.files import *\n\nfrom cloudbio.flavor import Flavor\n\n# This flavour installs several neuroinformatics software from NeuroDebian\n#\n#   http://neuro.debian.net/\n#\n# Author:  Roman Valls Guimera <roman@incf.org>\n\nclass NeuroFlavor(Flavor):\n    \"\"\" A flavour for installing NeuroDebian. A debian neuroinformatics repository\n        of software and datasets.\n    \"\"\"\n\n    def __init__(self, env):\n        self.name = \"Neuroinformatics Flavor\"\n\n    def rewrite_config_items(self, name, packages):\n        if name == 'packages':\n            packages.extend([\n                \"git\"\n            ])\n        return packages\n\n\n    def rewrite_apt_sources_list(self, name, sources):\n        sources = [\n            'deb http://neurodeb.pirsquared.org data main contrib non-free',\n            '#deb-src http://neurodeb.pirsquared.org data main contrib non-free',\n            'deb http://neurodeb.pirsquared.org saucy main contrib non-free',\n            '#deb-src http://neurodeb.pirsquared.org saucy main contrib non-free'\n        ]\n\n#wget -O- http://neuro.debian.net/lists/saucy.us-ca.full | sudo tee /etc/apt/sources.list.d/neurodebian.sources.list\n#sudo apt-key adv --recv-keys --keyserver pgp.mit.edu 2649A5A9\n\n#env.flavor = NeuroFlavor(env)\n"
  },
  {
    "path": "contrib/flavor/neuro/packages.yaml",
    "content": "---\nneuro:\n  - afni\n  - afni-atlases\n  - afni-common\n  - afni-dbg\n  - afni-dev\n  - aghermann\n  - ants\n  - arno-iptables-firewall\n  - autotools-dev\n  - biosig-tools\n  - caret\n  - caret-data\n  - cde\n  - cgroup-bin\n  - cmtk\n  - cnrun\n  - condor\n  - condor-dbg\n  - condor-dev\n  - condor-doc\n  - connectomeviewer\n  - coop-computing-tools\n  - coop-computing-tools-dev\n  - coop-computing-tools-doc\n  - cython\n  - cython-dbg \n  - debhelper\n  - debian-handbook\n  - debootstrap\n  - debootstrap-udeb\n  - debruijn\n  - dh-autoreconf\n  - dh-python\n  - dh-systemd\n  - dicomnifti\n  - dmtcp\n  - dmtcp-dbg\n  - eatmydata\n  - edac-utils\n  - eegdev-plugins-free\n  - eeglab11-sampledata\n  - eegview\n  - fail2ban\n  - fis-gtm\n  - fis-gtm-6.0-003\n  - freeipmi\n  - freeipmi-bmc-watchdog\n  - freeipmi-common\n  - freeipmi-ipmidetect\n  - freeipmi-tools\n  - freenect\n  - fsl\n  - fsl-5.0\n  - fsl-5.0-complete\n  - fsl-5.0-core\n  - fsl-5.0-doc\n  - fsl-5.0-doc-wikiattachments\n  - fsl-5.0-gpu\n  - fsl-atlases\n  - fsl-bangor-cerebellar-atlas\n  - fsl-complete\n  - fsl-core\n  - fsl-doc-5.0\n  - fsl-feeds\n  - fsl-first-data\n  - fsl-harvard-oxford-atlases\n  - fsl-harvard-oxford-cortical-lateralized-atlas\n  - fsl-jhu-dti-whitematter-atlas\n  - fsl-juelich-histological-atlas\n  - fsl-mni-structural-atlas\n  - fsl-mni152-templates\n  - fsl-neurosynth-atlas\n  - fsl-neurosynth-top100-atlas\n  - fsl-oxford-striatal-atlas\n  - fsl-oxford-thalamic-connectivity-atlas\n  - fsl-possum-data\n  - fsl-subthalamic-nucleus-atlas\n  - fsl-talairach-daemon-atlas\n  - fslview\n  - fslview-doc\n  - gdf-tools\n  - gifti-bin\n  - glew-utils\n  - gmsl\n  - guacamole\n  - guacamole-tomcat\n  - guacd\n  - haxby2001-faceobject\n  - haxby2001-faceobject-subject1\n  - haxby2001-faceobject-subject2\n  - haxby2001-faceobject-subject3\n  - haxby2001-faceobject-subject4\n  - haxby2001-faceobject-subject5\n  - haxby2001-faceobject-subject6\n  - impressive\n  - incf-nidash-oneclick-clients\n  - init-system-helpers\n  - insighttoolkit4-examples\n  - ipython01x\n  - ipython01x-doc\n  - ipython01x-notebook\n  - ipython01x-parallel\n  - ipython01x-qtconsole\n  - ipython1x\n  - ipython1x-doc\n  - ipython1x-notebook\n  - ipython1x-parallel\n  - ipython1x-qtconsole\n  - isis-utils\n  - itksnap\n  - jist\n  - kbibtex\n  - klustakwik\n  - libbiosig-dev\n  - libbiosig1\n  - libbiosig1-dbg\n  - libcgroup-dev\n  - libcgroup1\n  - libclassad-dev\n  - libclassad3\n  - libdmtcpaware-dev\n  - libdmtcpaware1\n  - libdouble-conversion-dbg\n  - libdouble-conversion-dev\n  - libdouble-conversion1\n  - libdrawtk-dev\n  - libdrawtk0\n  - libdrawtk0-dbg\n  - libedac-dev\n  - libedac1\n  - libedac1-dbg\n  - libeegdev-dev\n  - libeegdev0\n  - libeegdev0-dbg\n  - libeigen3-dev\n  - libeigen3-doc\n  - libfreeipmi-dev\n  - libfreeipmi12\n  - libfreenect-bin\n  - libfreenect-demos\n  - libfreenect-dev\n  - libfreenect-doc\n  - libfreenect0.1\n  - libgdf-dev\n  - libgdf0\n  - libgdf0-dbg\n  - libgiftiio-dev\n  - libgiftiio0\n  - libglew1.9\n  - libglew1.9-dbg\n  - libglew1.9-dev\n  - libglewmx1.9\n  - libglewmx1.9-dbg\n  - libglewmx1.9-dev\n  - libgoogle-glog-dev\n  - libgoogle-glog-doc\n  - libgoogle-glog0\n  - libguac-client-rdp0\n  - libguac-client-ssh0\n  - libguac-client-vnc0\n  - libguac-dev\n  - libguac3\n  - libguac5\n  - libinsighttoolkit4-dev\n  - libinsighttoolkit4.2\n  - libinsighttoolkit4.5\n  - libipmiconsole-dev\n  - libipmiconsole2\n  - libipmidetect-dev\n  - libipmidetect0\n  - libipmimonitoring-dev\n  - libipmimonitoring5\n  - libisis-core-dev\n  - libisis-core0\n  - libisis-ioplugins-common\n  - libisis-ioplugins-dicom\n  - libisis-qt4-0\n  - libisis-qt4-dev\n  - libjs-underscore\n  - libmcpanel-dev\n  - libmcpanel0\n  - libmcpanel0-dbg\n  - libmia-2.0-8\n  - libmia-2.0-8-dbg\n  - libmia-2.0-dev\n  - libmia-2.0-doc\n  - libmialm-dev\n  - libmialm-doc\n  - libmialm3\n  - libmialm3-dbg\n  - libmtcp-dev\n  - libmtcp1\n  - libnifti-dev\n  - libnifti-doc\n  - libnifti2\n  - libnlopt-dev\n  - libnlopt-guile0\n  - libnlopt0\n  - libodin-dev\n  - libopenmeeg-dev\n  - libopenmeeg1\n  - libopenwalnut1\n  - libopenwalnut1-dev\n  - libopenwalnut1-doc\n  - liborthancclient-dev\n  - liborthancclient-doc\n  - liborthancclient0.7\n  - libpam-cgroup\n  - libpgm-5.1-0\n  - libpgm-dbg\n  - libpgm-dev\n  - librtfilter-dev\n  - librtfilter1\n  - librtfilter1-dbg\n  - libshogun-dev\n  - libshogun11\n  - libslicer3\n  - libslicer3-dev\n  - libsvm-dev\n  - libsvm-java\n  - libsvm-tools\n  - libsvm3\n  - libsvm3-java\n  - libusb-1.0-0\n  - libusb-1.0-0-dev\n  - libusb-1.0-0-udeb\n  - libvia-dev\n  - libvia-doc\n  - libvia2\n  - libvistaio-dev\n  - libvistaio14\n  - libvistaio14-dbg\n  - libvrpn-dev\n  - libvrpn0\n  - libvrpnserver0\n  - libvtk-java\n  - libvtk5-dev\n  - libvtk5-qt4-dev\n  - libvtk5.8\n  - libvtk5.8-qt4\n  - libvw-dev\n  - libvw0\n  - libxdffileio-dev\n  - libxdffileio0\n  - libxdffileio0-dbg\n  - libzmq-dbg\n  - libzmq-dev\n  - libzmq1\n  - lipsia\n  - lipsia-doc\n  - matlab-eeglab11\n  - matlab-gdf\n  - matlab-psychtoolbox-3\n  - matlab-psychtoolbox-3-nonfree\n  - matlab-spm8\n  - matlab-support\n  - matlab-support-dev\n  - mclaren-rhesus-macaque-atlas\n  - mia-tools\n  - mia-tools-dbg\n  - mia-tools-doc\n  - mialmpick\n  - mialmpick-dbg\n  - mipav\n  - mitools\n  - mni-colin27-minc\n  - mni-colin27-nifti\n  - mni-icbm152-nlin-2009a\n  - mni-icbm152-nlin-2009b\n  - mni-icbm152-nlin-2009c\n  - mriconvert\n  - mricron\n  - mricron-data\n  - mricron-doc\n  - mrtrix\n  - mrtrix-doc\n  - mwrap\n  - neurodebian-desktop\n  - neurodebian-dev\n  - neurodebian-guest-additions\n  - neurodebian-keyring\n  - neurodebian-popularity-contest\n  - neurosynth-dataset\n  - nifti-bin\n  - nifti2dicom\n  - nifti2dicom-data\n  - nipy-suite\n  - nipy-suite-doc\n  - nuitka\n  - numdiff\n  - octave-biosig\n  - octave-gdf\n  - octave-nlopt\n  - octave-psychtoolbox-3\n  - octave-psychtoolbox-3-nonfree\n  - odin\n  - openelectrophy\n  - openmeeg-tools\n  - opensesame\n  - openvibe-bin\n  - openvibe-data\n  - openvibe-dev\n  - openvibe-libs\n  - openvibe-plugins\n  - openwalnut-modules\n  - openwalnut-qt4\n  - orthanc\n  - packaging-tutorial\n  - psychopy\n  - psychtoolbox-3-common\n  - psychtoolbox-3-dbg\n  - psychtoolbox-3-lib\n  - python-biosig\n  - python-brian\n  - python-brian-doc\n  - python-brian-lib\n  - python-cfflib\n  - python-dicom\n  - python-dipy\n  - python-dipy-doc\n  - python-dipy-lib\n  - python-expyriment\n  - python-freenect\n  - python-isis\n  - python-joblib\n  - python-lazyarray\n  - python-libsvm\n  - python-mdp\n  - python-mlpy\n  - python-mlpy-doc\n  - python-mlpy-lib\n  - python-mne\n  - python-mpi4py\n  - python-mpi4py-dbg\n  - python-mpi4py-doc\n  - python-mvpa\n  - python-mvpa-doc\n  - python-mvpa-lib\n  - python-mvpa2\n  - python-mvpa2-doc\n  - python-mvpa2-lib\n  - python-mvpa2-tutorialdata\n  - python-neo\n  - python-networkx\n  - python-networkx-doc\n  - python-neuroshare\n  - python-neurosynth\n  - python-nibabel\n  - python-nibabel-doc\n  - python-nibabel-snapshot\n  - python-nifti\n  - python-nipy\n  - python-nipy-doc\n  - python-nipy-lib\n  - python-nipy-lib-dbg\n  - python-nipype\n  - python-nipype-doc\n  - python-nitime\n  - python-nitime-doc\n  - python-nlopt\n  - python-numexpr\n  - python-numpydoc\n  - python-openmeeg\n  - python-openopt\n  - python-openpyxl\n  - python-pandas\n  - python-pandas-lib\n  - python-patsy\n  - python-patsy-doc\n  - python-pp\n  - python-pprocess\n  - python-pyentropy\n  - python-pyepl\n  - python-pyepl-common\n  - python-pyglet\n  - python-pymc\n  - python-pymc-doc\n  - python-pynn\n  - python-pyo\n  - python-pyoptical\n  - python-pypsignifit\n  - python-pyssdh\n  - python-pyxid\n  - python-pyxnat\n  - python-quantities\n  - python-scikits-learn\n  - python-scikits-statsmodels\n  - python-scikits-statsmodels-doc\n  - python-scikits.statsmodels\n  - python-scipy\n  - python-scipy-dbg\n  - python-simplegeneric\n  - python-skimage\n  - python-skimage-doc\n  - python-skimage-lib\n  - python-sklearn\n  - python-sklearn-doc\n  - python-sklearn-lib\n  - python-sphinx\n  - python-spyderlib\n  - python-spykeutils\n  - python-statsmodels\n  - python-statsmodels-doc\n  - python-statsmodels-lib\n  - python-stfio\n  - python-surfer\n  - python-sympy\n  - python-tornado\n  - python-traits4\n  - python-tz\n  - python-visionegg\n  - python-vtk\n  - python-workqueue\n  - python-zmq\n  - python-zmq-dbg\n  - python3-dateutil\n  - python3-mdp\n  - python3-mpi4py\n  - python3-mpi4py-dbg\n  - python3-pandas\n  - python3-pandas-lib\n  - python3-patsy\n  - python3-skimage\n  - python3-skimage-lib\n  - python3-tz\n  - qlandkarte\n  - qlandkartegt\n  - qlandkartegt-garmin\n  - qnifti2dicom\n  - rdfind\n  - remake\n  - rorden-mri-tutorial\n  - rorden-mri-tutorial-data\n  - shogun-cmdline-static\n  - shogun-csharp-modular\n  - shogun-dbg\n  - shogun-doc-cn\n  - shogun-doc-en\n  - shogun-elwms-static\n  - shogun-java-modular\n  - shogun-lua-modular\n  - shogun-python-modular\n  - shogun-python-static\n  - shogun-r-static\n  - shogun-ruby-modular\n  - sigviewer\n  - slicer\n  - slicer-data\n  - spm8-common\n  - spm8-data\n  - spm8-doc\n  - spyder\n  - spykeviewer\n  - sri24-atlas\n  - stabilitycalc\n  - stimfit\n  - stimfit-dbg\n  - svgtune\n  - tcl-vtk\n  - testkraut\n  - tigervnc-common\n  - tigervnc-scraping-server\n  - tigervnc-standalone-server\n  - tigervnc-viewer\n  - tigervnc-xorg-extension\n  - ubuntu-keyring\n  - ubuntu-keyring-udeb\n  - via-bin\n  - virtual-mri-nonfree\n  - vowpal-wabbit\n  - vowpal-wabbit-dbg\n  - vowpal-wabbit-doc\n  - voxbo\n  - vrpn\n  - vrpn-dbg\n  - vtk-doc\n  - vtk-examples\n  - xmhtml1\n  - xmhtml1-dev\n  - xppaut\nlibraries:\n"
  },
  {
    "path": "contrib/flavor/ngs_pipeline_minimal/custom.yaml",
    "content": "---\nbio_nextgen:\n  alignment:\n  utilities:\n  analysis:\n  variant:\n"
  },
  {
    "path": "contrib/flavor/ngs_pipeline_minimal/main.yaml",
    "content": "---\n# Flavor containing with minimal instructions to install tools for\n# running next-generation sequencing pipelines.\npackages:\n  - minimal\n  - libraries\n  - perl\n  - python\n  - java\n  - r\n  - ruby\n  - bio_nextgen\nlibraries:\n  - r-libs"
  },
  {
    "path": "contrib/flavor/ngs_pipeline_minimal/packages-conda.yaml",
    "content": "---\n# some ideas how to debug solving issues\n# not pinning is best - easier to solve, assuming latest possible versions\n# sometimes pinning means the latest version, say we update the old one by pinning package>=2.0\n# in 3 years that pinning pushes the installation down (it is still calculated as lower boundary)\n# don't duplicate packages - if a package is in bcbio bioconda recipe, delete it from here\n# mind r-base / python / htslib / java differences - their versions create separate solution spaces\n# the goal is to make one big main enviroment and isolate problematic packages\nchannels:\n  - conda-forge\n  - bioconda\nminimal:\n  - anaconda-client\n  - awscli\n  - bzip2\n  - ncurses\n  - nodejs\n  - p7zip\n  - readline\n  - s3gof3r\n  - xz\n# currently not included to reduce total size of Docker images\n# - pandoc\nbio_nextgen:\n  # default: python3.7.10  htslib1.13\n  # htslib1.14 is too fresh - solving issues\n  - age-metasv\n  - atropos\n  - bamtools\n  - bamutil\n  # installs samtools, a lot of xorg, openjdk11.0.9.1 - a candidate for isolation\n  - bbmap\n  # wants to update htslib if unrestricted\n  # installs perl 5.32.1\n  - bcftools=1.13\n  - bedops\n  # installed by bcbio bioconda recipe\n  # 2.30.0 + pybedtools\n  # - bedtools\n  # installs boost, curl, libmaus\n  # see issue 3650\n  - biobambam=2.0.87\n  - bowtie\n  - break-point-inspector\n  - bwa\n  # installs ruby\n  - bio-vcf\n  - cage\n  # downgrades joblib\n  - cnvkit\n  - coincbc\n  - cramtools\n  # cyvcf2 is installed from bcbio-nextgen.yaml\n  # debugpy=1.4.1 may cause solve issues\n  # - debugpy=1.4.0=py37he226ad3_0\n  - deeptools\n  - express\n  - fastp\n  # tmp pin adapt the wrapper to the new version\n  - fastqc=0.11.9\n  - geneimpacts\n  # installs libgcc 7.20\n  - genesplicer\n  - gffcompare\n  - goleft\n  - grabix\n  - gsort\n  - gsutil\n  - gvcfgenotyper\n  - h5py=3.3\n  - hdf5=1.10\n  - hmmlearn\n  - hisat2\n  # installed by bcbio bioconda recipe\n  # - htslib=1.13 \n  - htseq\n  - impute2\n  # kallisto 0.46 downgrades h5py to 3.3.0, to hdf5 1.10.6, kallisto is not working with 3.4.0/1.12.1\n  - kallisto=0.46\n  # downgrades perl to 5.26.2\n  - kraken\n  - ldc\n  # installed by bcbio bioconda recipe = 7.79.1 required by pysam and mamba\n  # - libcurl\n  - macs2\n  - maxentscan\n  - mbuffer\n  - minimap2\n  - mintmap\n  - mirdeep2\n  - mirtop\n  - moreutils\n  - multiqc\n  - multiqc-bcbio\n  - ngs-disambiguate\n  - novoalign\n  - oncofuse\n  - pandoc\n  - parallel\n  - pbgzip\n  - peddy\n  # installed by bcbio conda recipe\n  #- pysam>=0.14.0\n  - pizzly\n  # TODO: re-enable preseq but it pulls in openblas which causes problems with other blas libraries\n  # - preseq\n  - pythonpy\n  - qsignature\n  - rapmap\n  # installed by optitype\n  # - razers3\n  - rtg-tools\n  - sailfish\n  - salmon\n  - samblaster\n  - samtools=1.13\n  - scalpel\n  # old versions numbered as 2015.x new versions - 1.3\n  - seq2c<2016\n  - seqbuster\n  - seqcluster\n  - seqtk\n  - sickle-trim\n  - simple_sv_annotation\n  - singlecell-barcodes\n  - snap-aligner=1.0dev.97\n  - snpeff=5.0\n  - solvebio\n  # downgrades openmp\n  - spades\n  # installed\n  #- staden_io_lib\n  - star=2.6.1d\n  - stringtie\n  - subread\n  - survivor\n  - tophat-recondition\n  - tdrmapper\n  - trim-galore\n  - ucsc-bedtobigbed\n  - ucsc-bigbedinfo\n  - ucsc-bigbedsummary\n  - ucsc-bigbedtobed\n  - ucsc-bedgraphtobigwig\n  - ucsc-bigwiginfo\n  - ucsc-bigwigsummary\n  - ucsc-bigwigtobedgraph\n  - ucsc-bigwigtowig\n  - ucsc-fatotwobit\n  - ucsc-gtftogenepred\n  - ucsc-liftover\n  - ucsc-wigtobigwig\n  - umis\n  - wham\n  # old versions numbered as 2015.x new versions - 1.8.2\n  # ok with r installed, without r installs r3.2.2\n  - vardict<=2015\n  - vardict-java\n  - variantbam\n  - varscan\n  - vcfanno\n  - viennarna\n  # a lot of dependencies including QT - separate env?\n  - vqsr_cnn\n# corrects bcbio install, lift up to bcbio conda recipe\nipython:\n  # noarch\n  - ipython-cluster-helper=0.6.4=py_0\n  - ipython_genutils=0.2.0=py37_0\n  # 5x fails\n  - traitlets=4.3.3\n  # 7x has different interface\n  - ipyparallel=6.3.0\n  # <6.0.0 pin in requirements isn't applied\n  - ipython=7.29.0\n#installs many more perl dependencies\nperl:\n  - perl-app-cpanminus\n  - perl-archive-extract\n  - perl-archive-zip\n  # perl-bio-db-sam is deprecated\n  - perl-bio-samtools\n  - perl-cgi\n  - perl-dbi\n  - perl-encode-locale\n  - perl-file-fetch\n  - perl-file-sharedir\n  - perl-file-sharedir-install\n  - perl-ipc-system-simple\n  - perl-lwp-simple\n  - perl-lwp-protocol-https\n  # removed due to the perl upgrade\n  # needs to be upgraded from https://github.com/cancerit/cgpBattenberg\n  # perl-sanger-cgp-battenberg\n  - perl-statistics-descriptive\n  - perl-time-hires\n  - perl-vcftools-vcf\nr:\n  # default r=4.1 SI=1.22 VA=1.38 htslib1.13\n  # r4.0 pushes htslibto 1.10\n  - r=4.1=r41hd8ed1ab_1004\n  # yes r is two digit, rbase is three\n  # another 4.1.1 rbase pushed down htslib\n  - r-base=4.1.1=hb67fd72_0\n  - xorg-libxt\n  - r-chbutils\n  - r-devtools\n  - r-dplyr\n  - r-dt\n  - r-janitor\n  - r-knitr\n  - r-ggdendro\n  - r-ggplot2\n  - r-ggrepel\n  - r-gplots\n  - r-gsalib\n  - r-pheatmap\n  - r-plyr\n  - r-pscbs\n  - r-reshape\n  - r-rmarkdown\n  - r-rsqlite\n  - r-sleuth\n  - r-snow\n  - r-stringi\n  - r-viridis\n  - r-wasabi\n  - bioconductor-annotate\n  - bioconductor-apeglm\n  - bioconductor-biocgenerics\n  - bioconductor-biocinstaller\n  - bioconductor-biocstyle\n  - bioconductor-biostrings\n  # 1.22.0 requires r4.1 which gives more solve issues\n  # 1.20.0 = r40_0\n  - bioconductor-summarizedexperiment\n  - bioconductor-biovizbase\n  - bioconductor-bsgenome.hsapiens.ucsc.hg19\n  - bioconductor-bsgenome.hsapiens.ucsc.hg38\n  - bioconductor-bubbletree\n  - bioconductor-cn.mops\n  - bioconductor-copynumber\n  - bioconductor-dnacopy\n  - bioconductor-degreport\n  - bioconductor-deseq2\n  - bioconductor-dexseq\n  - bioconductor-genomeinfodbdata\n  - bioconductor-genomicranges\n  - bioconductor-iranges\n  - bioconductor-limma\n  # 2.38.0 breaks wasabi!\n  - bioconductor-rhdf5\n  - bioconductor-rtracklayer\n  # no bioconductor-snpchip in bioconda anymore\n  # bioconductor-snpchip needed for the TitanCNA wrapper but not listed as a depedency in the formula\n  - bioconductor-titancna\n  - bioconductor-vsn\n  #- r-optparse=1.7.1=r40hc72bb7e_0\n  - r-optparse\n  - bioconductor-purecn>=2.0.1\n  - bioconductor-txdb.hsapiens.ucsc.hg38.knowngene\n  - bioconductor-org.hs.eg.db\n  - bioconductor-genomeinfodb\n  - bioconductor-genomeinfodbdata\n  - bioconductor-rtracklayer\n  - bioconductor-txdb.hsapiens.ucsc.hg19.knowngene\n  - r-tidyverse\n  - bioconductor-tximport\n  - r-deconstructsigs\nr35:\n  # for arriba, old r dependencies htslib1.9,\n  # libtiff -> libdeflate nightmare\n  - r=3.5.1;env=r35\n  - r-base=3.5.1;env=r35\n  - arriba=1.2.0;env=r35\nrbcbiornaseq:\n  - r-bcbiornaseq>=0.5.1;env=rbcbiornaseq\n# openjdk 8 - waiting for gatk 4.4.0 release in conda to switch to java17\njava:\n  - fgbio;env=java\n  - gatk;env=java\n  # fixes log4j\n  - gatk4=4.2.5.0;env=java\n  - picard=2.27.5;env=java\n  # lots of dependencies\n  # try removing gridss and hmftools\n  # - gridss;env=java\n  # - hmftools-amber;env=java\n  # - hmftools-cobalt;env=java\n  # - hmftools-purple;env=java\n  - qualimap;env=java\n  - bcbio-prioritize;env=java\n  - bcbio-variation;env=java\n  # does not work with java11\n  - bcbio-variation-recall;env=java\nhtslib1.12:\n  - htslib=1.12;env=htslib1.12\n  - octopus>=0.7.4;env=htslib1.12\nhtslib1.11:\n  - htslib=1.11;env=htslib1.11\n  - scramble;env=htslib1.11\nhtslib1.10:\n  - htslib=1.10;env=htslib1.10\n  - ataqv;env=htslib1.10\n  - mosdepth;env=htslib1.10\n  - hts-nim-tools;env=htslib1.10\n  - pindel;env=htslib1.10\n  - ensembl-vep=104;env=htslib1.10\n  - perl-bio-db-hts>=3.01;env=htslib1.10\n  - cancerit-allelecount;env=htslib1.10\n  - delly;env=htslib1.10\n  - freebayes;env=htslib1.10\n  - vcflib;env=htslib1.10\n  - verifybamid2;env=htslib1.10\n  # this isn't working for vt normalize\n  #- vt;env=htslib1.10\nhtslib1.12_py3.9:\n  - htslib=1.12;env=htslib1.12_py3.9\n  - lofreq;env=htslib1.12_py3.9\n  - sambamba;env=htslib1.12_py3.9\n  - chipseq-greylist;env=htslib1.12_py3.9\nhtslib1.9:\n  - htslib=1.9;env=htslib1.9\n  - duphold;env=htslib1.9\n  - extract-sv-reads;env=htslib1.9\n# stall packages\npython3.6:\n  # pushes down mamba, libtiff, but not hts\n  - bowtie2;env=python3.6\n  # uses specific pysam \n  - crossmap;env=python3.6\n  - cufflinks;env=python3.6\n  - pyloh;env=python3.6\n  # vt is tricky, the only combination that works for vt normalize is\n  - samtools=1.7;env=python3.6\n  - htslib=1.7;env=python3.6\n  - vt=2015.11.10=he941832_3;env=python3.6\n  # not working in the base py3.7\n  - optitype;env=python3.6\n  - razers3;env=python3.6\n  - pysam=0.15.4;env=python3.6\n  - pandas=0.25.3;env=python3.6\n#htslib/samtools 1.10\npython2:\n  - bismark;env=python2\n  - cpat;env=python2\n  - cutadapt=1.16;env=python2\n  # by default 0.30.11 is installed - causes issues with blob gts fields in vcf2db.py/gemini\n  - cyvcf2=0.10.0;env=python2\n  - dkfz-bias-filter;env=python2\n  - gemini;env=python2\n  - gvcf-regions;env=python2\n  - hap.py;env=python2\n  - htslib=1.10;env=python2\n  - lumpy-sv;env=python2\n  - manta;env=python2\n  - metasv;env=python2\n  - mirge;env=python2\n  - phylowgs;env=python2\n  - platypus-variant;env=python2\n  - samtools=1.10;env=python2\n  - sentieon;env=python2\n  - smcounter2;env=python2\n  - smoove;env=python2\n  - strelka;env=python2\n  - svtools;env=python2\n  - svtyper;env=python2\n  - theta2;env=python2\n  - tophat;env=python2\n  - vawk;env=python2\n  - vcf2db;env=python2\n  - numpy=1.16;env=python2\n  # for dkfz filter\n  # downgrades metasv0.4.0 and libdeflate1.3\n  - pysam=0.15.3;env=python2\nsamtools0:\n  # rbase4.1.1 ncurses5.9. samtools0.1.19\n  - ericscript;env=samtools0\n# samtools 1.7\nbwakit:\n  - bwakit;env=bwakit\n# python3.6\ndv:\n  - deepvariant;env=dv\n"
  },
  {
    "path": "contrib/flavor/ngs_pipeline_minimal/packages-homebrew.yaml",
    "content": "# Packages available in the Homebrew and Linuxbrew package manager\n---\n# Migrated to use conda/bioconda builds\nto_remove:\n  - age-metasv\n  - bamtools\n  - bcbio-variation-recall\n  - bcftools\n  - bedtools\n  - biobambam\n  - bowtie2\n  - bwa\n  - bwakit\n  - cancerit-allelecount\n  - cramtools\n  - cufflinks-binary\n  - delly\n  - express-binary\n  - fastqc\n  - freebayes\n  - gatk-framework\n  - grabix\n  - hall-lab-sv-tools\n  - htslib\n  - impute2\n  - kraken\n  - lofreq\n  - lumpy-sv\n  - manta\n  - novoalign\n  - oncofuse\n  - p7zip\n  - picard-tools\n  - pindel\n  - platypus-variant\n  - qualimap\n  - sambamba-binary\n  - samtools-library-0.1\n  - samblaster\n  - samtools\n  - sickle\n  - snpeff\n  - spades-binary\n  - theta2\n  - tophat-binary\n  - qsignature\n  - rtg-tools\n  - scalpel\n  - seqbuster\n  - seqtk\n  - staden_io_lib\n  - wham\n  - vardict\n  - vardict-java\n  - vcflib\n  - vep\n  - vt\nminimal:\nperl:\nbio_nextgen:\nold-unused:\n  # - staden_io_lib # Migrated to samtools indexing\n  # - express-binary # No longer used with Sailfish\n  # - glia  # Needs updates to fsom\n  # - qsnp # Issues with download and reliability of qcmg site\n  # - vcftools # Build problems with default libz on Ubuntu 12.04\n"
  },
  {
    "path": "contrib/flavor/ngs_pipeline_minimal/perl-libs.yaml",
    "content": "---\ncpan:\n  # mirdeep2\n  # - PDF::API2\n  # ## VEP dependencies\n  # don't need with conda perl\n  # - parent\n  # - Archive::Tar\n  # - Module::Load::Conditional\n  # - IPC::Cmd\n  # - Params::Check\n  # ## Migrated to conda\n  # - Encode::Locale\n  # - Archive::Extract\n  # - Archive::Zip\n  # - Statistics::Descriptive\n  # - Compress::Raw::Zlib\n  # - DBI\n  # - LWP::Simple\n  # - LWP::Protocol::https\n  # - File::Fetch\n  # - File::ShareDir\n  # - File::ShareDir::Install\n  # - Time::HiRes\n  # - IPC::System::Simple\n  # ## Sanger Cancer library dependencies\n  # - Bio::DB::Sam;--config lddlflags=-shared;SAMTOOLS={system_install}/share/samtools-0.1\n  # - Vcf==0.953==https://github.com/chapmanb/vcftools-cpan/archive/v0.953.tar.gz\n\n  # Sanger Cancer library dependencies -- to migrate\n  #- Sanger::CGP::Vcf==1.2.3==https://github.com/cancerit/cgpVcf/archive/v1.2.3.tar.gz\n  #- Sanger::CGP::AlleleCount==2.1.0==https://github.com/cancerit/alleleCount/archive/v2.1.0.tar.gz==build=perl\n  #- PCAP==1.8.1==https://github.com/ICGC-TCGA-PanCancer/PCAP-core/archive/v1.8.1.tar.gz==depremove=GD\n  #- Sanger::CGP::Battenberg==1.3.2_07==https://github.com/cancerit/cgpBattenberg/archive/92cd3a2.tar.gz==build=perl\n"
  },
  {
    "path": "contrib/flavor/ngs_pipeline_minimal/r-libs.yaml",
    "content": "---\ncranrepo: http://cran.fhcrc.org/\ncran:\ncran-after-bioc:\n#biocrepo: http://bioconductor.org/biocLite.R\nbioc:\ngithub:\n#github:\n#  - genome/bmm@b49c57a\n#  - chapmanb/sciClone@e8ed9d0\n"
  },
  {
    "path": "contrib/flavor/phylogeny/__init__.py",
    "content": "\"\"\"Phylogeny flavor\n\n   Copyright (C) 2011 Pjotr Prins <pjotr.prins@thebird.nl>\n\n\"\"\"\n\n"
  },
  {
    "path": "contrib/flavor/phylogeny/fabricrc_debian.txt",
    "content": "# Flavor targetting runningy Phylogeny programs in the Cloud,\n# such as recent edititions of mrbayes-mpi, raxml, BEAST and\n# tracer.\n#\n# This VM is used for a practical Phylogeny course.\n#\n# Copyright (C) 2011 Pjotr Prins <pjotr.prins@thebird.nl>\n#\nedition = bionode\nedition_version = 0.93\nflavor_path = phylogeny.phylogenyflavor\ndistribution = debian\ndist_name = squeeze\nnixpkgs = True\n\n# Update to the target's nearest repository:\ndebian_repository=http://ftp.nl.debian.org/debian/\n\n# Global installation directory for packages and standard programs\nsystem_install = /usr\nlocal_install = ~/install\nshell_config = ~/.bashrc\nshell = /bin/bash -l -c\n# data_files = /mnt/biodata\n\n"
  },
  {
    "path": "contrib/flavor/phylogeny/fabricrc_ubuntu.txt",
    "content": "# Flavor targetting runningy Phylogeny programs in the Cloud,\n# such as recent edititions of mrbayes-mpi, raxml, BEAST and\n# tracer.\n#\n# This VM is used for a practical Phylogeny course.\n#\n# Copyright (C) 2011 Pjotr Prins <pjotr.prins@thebird.nl>\n#\nedition = bionode\nedition_version = 0.93\nflavor_path = phylogeny.phylogenyflavor\ndistribution = ubuntu\ndist_name = natty\n# nixpkgs = True\n\n# Update to the target's nearest repository:\n# debian_repository=http://ftp.nl.debian.org/debian/\n\n# Global installation directory for packages and standard programs\nsystem_install = /usr\nlocal_install = ~/install\nshell_config = ~/.bashrc\nshell = /bin/bash -l -c\n# data_files = /mnt/biodata\n\n"
  },
  {
    "path": "contrib/flavor/phylogeny/install_debian.sh",
    "content": "#! /bin/sh \n#\n# Install the biolinux-phylogeny on a host\n#\n# Usage:\n#\n#   ./contrib/flavor/phylogeny/install_host.sh user@hostname\n#\n# where\n#\n#   user:      the password-less login name (see doc/private_cloud.md)\n#   hostname:  the hostname, or IP\n#\n#\n# Copyright (C) 2011 Pjotr Prins <pjotr.prins@thebird.nl>\n\nhost=$1\nif [ ! -e fabfile.py -o -z $host ]; then\n  echo Usage:\n  echo\n  echo   ./contrib/flavor/phylogeny/install_host.sh user@hostname\nfi\nsource=`pwd`\nfabricrc=$source/contrib/flavor/phylogeny/fabricrc_debian.txt\npackagelist=$source/contrib/flavor/phylogeny/main.yaml\nfab -f $source/fabfile.py -H $host -c $fabricrc install_biolinux:packagelist=$packagelist\n"
  },
  {
    "path": "contrib/flavor/phylogeny/install_ubuntu.sh",
    "content": "#! /bin/sh \n#\n# Install the biolinux-phylogeny on a host\n#\n# Usage:\n#\n#   ./contrib/flavor/phylogeny/install_host.sh user@hostname\n#\n# where\n#\n#   user:      the password-less login name (see doc/private_cloud.md)\n#   hostname:  the hostname, or IP\n#\n#\n# Copyright (C) 2011 Pjotr Prins <pjotr.prins@thebird.nl>\n\nhost=$1\nif [ ! -e fabfile.py -o -z $host ]; then\n  echo Usage:\n  echo\n  echo   ./contrib/flavor/phylogeny/install_host.sh user@hostname\nfi\nsource=`pwd`\nfabricrc=$source/contrib/flavor/phylogeny/fabricrc_ubuntu.txt\npackagelist=$source/contrib/flavor/phylogeny/main.yaml\nfab -f $source/fabfile.py -H $host -c $fabricrc install_biolinux:packagelist=$packagelist\n"
  },
  {
    "path": "contrib/flavor/phylogeny/main.yaml",
    "content": "---\npackages:\n  - minimal\n  - desktop-cloud\n  - editors\n  - phylogeny\n  # - phylogeny_testing - Debian only\n  - r\nlibraries:\n"
  },
  {
    "path": "contrib/flavor/phylogeny/phylogenyflavor.py",
    "content": "from fabric.api import *\nfrom fabric.contrib.files import *\nfrom fabfile import _freenx_scripts\n\nfrom cloudbio.flavor import Flavor\n\nfrom cloudbio.custom.shared import (_fetch_and_unpack)\n\nclass PhylogenyFlavor(Flavor):\n    \"\"\"A VM flavor for running Phylogeny\n    \"\"\"\n    def __init__(self, env):\n        Flavor.__init__(self,env)\n        self.name = \"Phylogeny Flavor\"\n\n    def rewrite_config_items(self, name, packages):\n        if name == 'packages':\n          packages += [ 'openssh-server', 'unzip', 'tar', 'sudo', 'openjdk-6-jre']\n          packages += [ 'openmpi-bin' ]  # required for MrBayes-MPI\n          # if 'bio-linux-mrbayes-multi' in packages:\n          #   (Debian version is still not OK)\n          #   packages.remove('bio-linux-mrbayes-multi')\n\n        for package in packages:\n          env.logger.info(\"Selected: \"+name+\" \"+package)\n        return packages\n\n    def post_install(self):\n        env.logger.info(\"Starting post-install\")\n        _freenx_scripts()\n        pass\n\nenv.flavor = PhylogenyFlavor(env)\n"
  },
  {
    "path": "contrib/flavor/phylogeny/virtualbox.md",
    "content": "= Running the Phylogeny VM in Virtualbox\n\n== Install Virtualbox\n\nDownload and install Virtualbox from https://www.virtualbox.org/\n\nStart Virtualbox.\n\n== Download and install the BioLinux image\n\nFetch a BioLinux for virtualbox using the provided link, e.g.\n\n        wget http://hostname/biolinux-phylogeny-virtualbox.vmdk\n\nAdd this file to VirtualBox by selecting 'New', choose a name and\nselect Debian Linux. After setting RAM, select 'Use existing hard disk' and\nselect the downloaded .vmdk file.\n\n== Start up the image\n\nSelect the image and press 'Start'.\n\n== Using the VM\n\nAfter starting the VM you get a desktop. The user name is 'vagrant', as well\nas the password. You can get root with the same password. Open a terminal\nwith LXterminal (from the menu). Run tools from the command line, e.g.\n\n       raxmlHPC\n       mb-mpi\n\nyou can have root, so installing new software is possible using apt-get.\n\n== Modifying the virtual hardware\n\nYou can configure your image to run on multiple CPUs. Right-click on the image\nicon in virtualbox and 'Settings'. Configure the hardware.\n\n== Use ssh/scp\n\nOpenssh-server should be running on the VM\n\n        apt-get install openssh-server\n\nThe VM is visible on a local network inside your PC. The VM network card is\neth1. Start a terminal as root and initialize\n\n        dhcpclient -v eth1\n        /sbin/ifconfig\n\nfrom your PC use ssh to the listed IP address. \n\nWhen DHCP fails, reconfigure networking in VirtualBox. Right-click on the image\nicon of the VM, click 'Settings', and configure the network to use 'bridge'.\nAnd again run DHCP.\n\n== Copying files to the image\n\nWith scp files can be copied to the image. An alternative approach, since the\nVM has network, is to provide files through a webserver, or Amazon S3. E.g.\n\n         wget http://webserver/file.tgz\n\n"
  },
  {
    "path": "contrib/flavor/pjotrp/__init__.py",
    "content": "\"\"\"Pjotr's flavors\"\"\"\n"
  },
  {
    "path": "contrib/flavor/pjotrp/biotest/__init__.py",
    "content": "\"\"\"BioTest flavor\n\n   Copyright (C) 2011 Pjotr Prins <pjotr.prins@thebird.nl>\n\"\"\"\n"
  },
  {
    "path": "contrib/flavor/pjotrp/biotest/biotestflavor.py",
    "content": "from fabric.api import *\nfrom fabric.contrib.files import *\n\nfrom cloudbio.flavor import Flavor\n\nfrom cloudbio.custom.shared import (_fetch_and_unpack)\n\nclass BioTestFlavor(Flavor):\n    \"\"\"A Flavor for cross Bio* tests\n    \"\"\"\n    def __init__(self, env):\n        Flavor.__init__(self,env)\n        self.name = \"Bio* cross-lang flavor\"\n\n    def rewrite_config_items(self, name, items):\n        if name == \"packages\":\n            # list.remove('screen')\n            # list.append('test')\n            return items\n        elif name == \"python\":\n            return [ 'biopython' ]\n        elif name == \"perl\":\n            return [ 'bioperl' ]\n        elif name == \"ruby\":\n            return [ 'bio' ]\n        elif name == \"custom\":\n            return []\n        else:\n            return items\n\n    def post_install(self):\n        env.logger.info(\"Starting post-install\")\n        env.logger.info(\"Load Scalability tests\")\n        if exists('Scalability'):\n            with cd('Scalability'):\n               run('git pull')\n        else:\n           _fetch_and_unpack(\"git clone git://github.com/pjotrp/Scalability.git\")\n        # Now run a post installation routine (for the heck of it)\n        run('./Scalability/scripts/hello.sh')\n\n        env.logger.info(\"Load Cross-language tests\")\n        if exists('Cross-language-interfacing'):\n            with cd('Cross-language-interfacing'):\n               run('git pull')\n        else:\n           _fetch_and_unpack(\"git clone git://github.com/pjotrp/Cross-language-interfacing.git\")\n        # Special installs for the tests\n        with cd('Cross-language-interfacing'):\n            sudo('./scripts/install-packages-root.sh ')\n            run('./scripts/install-packages.sh')\n            run('./scripts/create_test_files.rb')\n\n\nenv.flavor = BioTestFlavor(env)\n"
  },
  {
    "path": "contrib/flavor/pjotrp/biotest/fabricrc_debian.txt",
    "content": "edition = minimal\nedition_version = 0.10\nflavor_path = pjotrp.biotest.biotestflavor\ndistribution = debian\ndist_name = squeeze\n\n# Update to the target's nearest repository:\ndebian_repository=http://ftp.nl.debian.org/debian/\n\n# Global installation directory for packages and standard programs\nsystem_install = /usr\nlocal_install = ~/install\nshell_config = ~/.bashrc\nshell = /bin/bash -l -c\n# data_files = /mnt/biodata\n"
  },
  {
    "path": "contrib/flavor/pjotrp/biotest/main.yaml",
    "content": "---\n# Top level configuration file that specifies which groups of programs\n# should be installed. New sections that are added to individual config\n# files should go here. Comment out any groups you don't want to have \n# installed.\n#\n# This is the minimal version to kick-off development, including editors,\n# version control, and basic build support (GNU tools)\npackages:\n  - minimal\n  - ruby\n  - r\n  - python\nlibraries:\n  - ruby-libs\n"
  },
  {
    "path": "contrib/flavor/proteomics/galaxyp/README.md",
    "content": "This directory contains a stand-alone (no CloudMan) Galaxy-P flavor\nfor installing Galaxy-P in desktop or cluster environments. If\nconfiguring Galaxy-P for cloud based enviornments, the\ncloudman/cloudman_and_galaxyp flavor is likely a more appropriate target.\n\nFor more information on using CloudBioLinux to install Galaxy-P, please see\nhttp://getgalaxyp.org/install.html.\n"
  },
  {
    "path": "contrib/flavor/proteomics/galaxyp/main.yaml",
    "content": "---\npackages:\n  - minimal\n  - galaxy\n  - galaxyp\nlibraries:\n"
  },
  {
    "path": "contrib/flavor/proteomics/galaxyp/settings-sample-galaxyp.yaml",
    "content": "---                                                                                                                                                                          \nvm_provider: novm                                                                                                                                                           \n\nfabricrc_overrides:\n  # Directory to install Galaxy dependencies into. Be sure to update Galaxy's universe_wsgi.ini file\n  # so that tool_dependency_dir in the app:main section points to this same directory.\n  galaxy_tools_dir: /opt/galaxy/tools\n\n  # System user Galaxy will run as.\n  galaxy_user: galaxy\n\n  galaxy_tools_conf: $__contrib_dir__/flavor/proteomics/galaxyp/tools.yaml\n\n  galaxy_install_dependencies: True\n"
  },
  {
    "path": "contrib/flavor/proteomics/galaxyp/tools.yaml",
    "content": "--- \n\n# This file contains the install procedures for Galaxy tools used by\n# the stock Galaxy-P configuration (hosted on usegalaxyp.org).\n\n# Override the property `galaxy_tools_conf` in fabricrc to reference this \n# file and set the property `install_galaxy_dependencies` to True and \n# install with cbl_galaxy_tools custom install function directly using\n# \n#  % fab -f fabfile.py -i <key> -H ubuntu@<IP> install_custom:galaxy_tools\n#\n# Or as part of a full flavor install that include galaxy, for instance using\n# the cloudman/cloudman_and_galaxyp flavor.\n#\n#  % fab -f fabfile.py -i <key> -H ubuntu@<IP> install_biolinux:flavor=cloudman/cloudman_and_galaxyp\n#\n\napplications:\n  galaxy_protk: \"1.2.2\"\n  ## Bioinformatics and NGS Applications\n  emboss: \n    - version: \"6.6.0\"\n      symlink_versions: \"5.0.0\"\n  augustus: \n    - version: \"2.7\"\n      bin_dirs: \n        - bin\n        - scripts\n      env_vars:      \n        AUGUSTUS_CONFIG_PATH: $system_install/config\n  fastx_toolkit: \"0.0.13.2\"\n  ucsc_tools: \"latest\"\n  kent_tools: \"latest\"\n  blast: \n    - version: \"2.2.26+\"\n      requirement_name: \"blast+\"\n    - version: \"2.2.28+\"\n      requirement_name: \"blast+\"  \n  bowtie: \"0.12.9\"\n  tophat:\n    - version: \"1.3.3\"\n      bin_dirs: # tophat requires bowtie so make sure to include it in it's env.sh\n        - bin\n        - ../../bowtie/0.12.9/bin\n  bowtie2: \"2.1.0\"\n  tophat2:\n    - version: \"2.0.9\"\n      bin_dirs:\n        - bin\n        - ../../bowtie2/2.1.0/bin\n  trinity:\n    - version: \"r2012-10-05\"\n      bin_dirs:\n        - .\n  cufflinks: \"2.1.1\"\n  samtools: \"0.1.16\" \n\n  bwa: \"0.5.9\"\n  gatk: \"1.4-9-g1f1233b\"\n  fastqc: \"0.10.0\"\n  picard: \"1.56\"\n\n\n  ## Mass Spec Applications:\n\n  peptide_to_gff: \"master\"\n  transproteomic_pipeline: \n    - version: \"4.6.1-occupy\"\n      bin_dirs:\n        - bin\n        - cgi-bin\n      provides:\n        - \"xtandem\"\n        - \"trans_proteomic_pipeline\"\n        - name: \"proteowizard\"        \n          unless: \"using_wine\"\n  openms: \n    - version: \"1.10.0\"\n      env_vars:\n        LD_LIBRARY_PATH: \"${system_install}/lib:$$LD_LIBRARY_PATH\"\n  omssa: \"2.1.9\"\n  tint_proteomics_scripts:\n    - version: \"1.19.20\"\n      symlink_versions: \"1.19.19\"\n  superhirn: \"0.03\"\n  mzmine: \"2.10\"\n  percolator: \"2_04\"\n  myrimatch: \"2.1.131\"\n  directag: \"1.3.62\"\n  idpqonvert: \"3.0.475\"\n  pepitome: \"1.0.45\"\n  tagrecon: \"1.4.63\"\n  crux: \"1.39\"\n  fido: \"2011\"\n  pride_inspector: \"1.3.0\"\n  pride_converter2: \"2.0.17\"\n  ms2preproc: \"2009\"\n  mayu: \"1.06\"\n  pepnovo: \"20120423\"\n  ipig: \"r5\"\n  searchgui: \"1.14.1\"\n  peptide_shaker: \n    - version: \"0.20.1\"\n      env_vars:\n        # If DISPLAY set keep it the same, otherwise use XVFB\n        DISPLAY: \"$${DISPLAY:-:42}\"\n  psm_eval: \"0.1.0\"\n"
  },
  {
    "path": "contrib/flavor/proteomics/swift/custom.yaml",
    "content": "---\nbio_proteomics_wine:\n  - proteomics_wine_env\n  - proteowizard\nbio_proteomics:\n  - idpqonvert\n  - myrimatch\n#  - transproteomic_pipeline\n#  - omssa\n#  - mzmine\n#  - directag\n#  - tagrecon\n#  - pepitome\n#  - percolator\n"
  },
  {
    "path": "contrib/flavor/proteomics/swift/main.yaml",
    "content": "---\npackages:\n  # Editors, utils, build systems, system administration, etc..\n  - minimal\n  # Brings in JDK and maven\n  - java\n  # R packages determined in r-libs.yaml in this directory.\n  - r\n  # See custom.yaml for what subsets of these packages are imported.\n  - bio_proteomics_wine\n  - bio_proteomics\n"
  },
  {
    "path": "contrib/flavor/proteomics/swift/r-libs.yaml",
    "content": "---\n# Configuration file defining R specific libraries that are installed \n# via CRAN and Bioconductor.\ncranrepo: http://watson.nci.nih.gov/cran_mirror/\ncran:\n - caTools\n - gam\n - png\n - bitops\n - akima\n"
  },
  {
    "path": "contrib/flavor/seal/__init__.py",
    "content": "\"\"\"Seal flavor\n\"\"\"\n"
  },
  {
    "path": "contrib/flavor/seal/fabricrc_sl.txt",
    "content": "edition = minimal\n#edition_version = 0.10\nflavor_path = seal.sealflavor\ndistribution = scientificlinux\n#dist_name = \n\n# Global installation directory for packages and standard programs\nsystem_install = /usr\nlocal_install = ~/install\nshell_config = ~/.bashrc\nshell = /bin/bash -l -c\n# data_files = /mnt/biodata\n# galaxy_base = /mnt/biodata/galaxy\n\n"
  },
  {
    "path": "contrib/flavor/seal/main.yaml",
    "content": "---\npackages:\n  - distributed\n  - java\n  - programming\n  - python\nlibraries:\n"
  },
  {
    "path": "contrib/flavor/seal/sealflavor.py",
    "content": "from fabric.api import *\nfrom fabric.contrib.files import *\n\nfrom cloudbio.flavor import Flavor\n\nfrom cloudbio.custom.shared import (_fetch_and_unpack)\n\nimport sys\n\n# This flavour installs the Seal toolkit for processing high-throughput\n# sequencing data on Hadoop.\n#   http://biodoop-seal.sf.net/\n#\n# It pulls in quite a few dependencies, including Hadoop itself and\n# Pydoop (http://pydoop.sf.net/).\n#\n# The dependencies it pulls into Cloudbiolinux are structured as follows:\n#\n# contrib/flavor/seal/main.yaml\n#   sealdist\n#   customsealdist\n#\n# config/packages-yum.yaml\n#   sealdist (metapackage)\n# config/custom.yaml\n#   customsealdist (metapackage)\n#     - pydoop\n#     - seal\n#\n# The components of the customsealdist metapackage are installed through\n# the functions in cloudbio/custom/customsealdist.py\n#\n#\n# This flavour has only been installed on Scientific Linux and has not\n# yet been well tested.\n#\n# To try installing it run the following:\n#   cd <your cloudbiolinux directory>\n#   fab -f ./fabfile.py -H root@<your host> -c ./contrib/flavor/seal/fabricrc_sl.txt  install_biolinux:packagelist=contrib/flavor/seal/main.yaml\n#\n# Authors:  Roman Valls Guimera <roman.valls.guimera@scilifelab.se>\n#           Luca Pireddu <luca.pireddu@crs4.it>\n\nclass SealFlavor(Flavor):\n\t\"\"\"A flavour for installing Seal\n\t\"\"\"\n\tdef __init__(self, env):\n\t\tFlavor.__init__(self,env)\n\t\tself.name = \"Seal Flavor\"\n\n\tdef rewrite_config_items(self, name, packages):\n\t\tif name == 'packages':\n\t\t\tif sys.version_info < (2,7):\n\t\t\t\t# for versions of Python prior to 2.7 we need to add importlib\n\t\t\t\t# and argparse\n\t\t\t\tpackages.extend([ \n\t\t\t\t\t\"python-importlib\",\n\t\t\t\t\t\"python-argparse\"\n\t\t\t\t])\n\t\treturn packages\n\n\n\tdef post_install(self):\n\t\tenv.logger.info(\"Starting post-install\")\n\t\tpass\n\nenv.flavor = SealFlavor(env)\n"
  },
  {
    "path": "contrib/flavor/variantviz/custom.yaml",
    "content": "---\n# List of custom programs to install that are not packaged. These each require \n# a install_name function in the appropriate place in the 'custom' directory to\n# define the steps for downloading, building, and installing the package.\nbio_nextgen:\n  - vep\njava:\n  - leinengin\n"
  },
  {
    "path": "contrib/flavor/variantviz/fabricrc.txt",
    "content": "# Configuration file for Fabric specifying filesystem information\n# This can be adjusted and passed to fab on the commandline as '-c'\n#\n# For example\n#\n#   fab -f data_fabfile.py -H your_machine -c your_fabricrc.txt install_data_s3:your_biodata.yaml\n\n# details about the operating system\n## Ubuntu\n#distribution = ubuntu\n#dist_name = precise\n## Centos\ndistribution = centos\n## Debian\n#distribution = debian\n#dist_name = squeeze\n\n# Unset to avoid CloudBioLinux customizations like setup scripts;\n# or set to different edition to get specific customizations\n#edition = cloudbiolinux\n\n# Whether to clean the system for EC2 AMI preparation;\n# do not set on vagrant or local machines\nis_ec2_image = false\n\n# Global installation directory for packages and standard programs\nsystem_install = /usr\n\n# Local install directory for versioned software that will not\n# be included in the path by default\nlocal_install = ~/install\nshell_config = ~/.bashrc\nshell = /bin/bash -l -c\n# Global setting for using sudo; allows installation of custom packages\n# by non-privileged users.\n# system_install needs to point to a user writeable directory if set to False\nuse_sudo = true\n\n# details for biological data files and Galaxy\ndata_files = /mnt/biodata\ngalaxy_home = /mnt/biodata/galaxy\n\n# Picard home\npicard_home = /usr/share/java/picard\n"
  },
  {
    "path": "contrib/flavor/variantviz/main.yaml",
    "content": "---\n# Top level configuration file that specifies which groups of programs\n# should be installed. New sections that are added to individual config\n# files should go here. Comment out any groups you don't want to have \n# installed.\npackages:\n  - programming\n  - python\n  - perl\n  - ruby\n  - java\n  - web\n  - bio_nextgen\nlibraries:\n"
  },
  {
    "path": "contrib/flavor/variantviz/packages-yum.yaml",
    "content": "---\n# List of packages to be installed via the yum package manager.\nprogramming:\n  build:\n    - gcc\n    - gcc-c++\n    - gcc-gfortran\n  libraries:\n    - atlas-devel\n    - blas\n    - blas-devel\n    - lapack\n    - lapack-devel\npython:\n  - python\n  - python-devel\n  - python-distribute\n  - python-pip\n  - scipy\n  - numpy\nperl: \n  - perl\n  - perl-DBI\n  - perl-DBD-MySQL\nruby:\n  - ruby\n  - ruby-devel\n  - rubygems\njava:\n  - java-1.7.0-openjdk\n  - java-1.7.0-openjdk-devel\n  - ant\nweb:\n  - nginx\nbio_nextgen:\n"
  },
  {
    "path": "contributors.mkd",
    "content": "## Contributors\n\nAn open source project is nothing without the great people who generously\ncontribute time and expertise.\n\n- Enis Afgan\n- Sébastien Boisvert\n- Tim Booth\n- Brad Chapman\n- John Chilton\n- Michael Heuer\n- Richard Holland\n- Mark Jensen\n- Justin Johnson\n- Ntino Krampis\n- Tristan J. Lubinski\n- Hervé Ménager\n- Dave Messina\n- Steffen Möller\n- Pjotr Prins\n- Mitch Skinner\n- Bela Tiwari\n- Roman Valls\n- Luca Pireddu\n"
  },
  {
    "path": "data_fabfile.py",
    "content": "\"\"\"Fabric deployment file to install genomic data on remote instances.\n\nDesigned to automatically download and manage biologically associated\ndata on cloud instances like Amazon EC2.\n\nFabric (http://docs.fabfile.org) manages automation of remote servers.\n\nUsage:\n    fab -i key_file -H servername -f data_fabfile.py install_data\n\"\"\"\nimport os\nimport sys\n\nfrom fabric.main import load_settings\nfrom fabric.api import *\nfrom fabric.contrib.files import *\nfrom fabric.context_managers import path\ntry:\n    import boto\nexcept ImportError:\n    boto = None\n\n# preferentially use local cloudbio directory\nfor to_remove in [p for p in sys.path if p.find(\"cloudbiolinux-\") > 0]:\n    sys.path.remove(to_remove)\nsys.path.append(os.path.dirname(__file__))\n\nfrom cloudbio.utils import _setup_logging, _configure_fabric_environment\nfrom cloudbio.biodata import genomes\n\n# -- Host specific setup\n\nenv.remove_old_genomes = False\n\ndef setup_environment():\n    \"\"\"Setup environment with required data file locations.\n    \"\"\"\n    _setup_logging(env)\n    _add_defaults()\n    _configure_fabric_environment(env, ignore_distcheck=True)\n\ndef _add_defaults():\n    \"\"\"Defaults from fabricrc.txt file; loaded if not specified at commandline.\n    \"\"\"\n    env.config_dir = os.path.join(os.path.dirname(__file__), \"config\")\n    conf_file = \"tool_data_table_conf.xml\"\n    env.tool_data_table_conf_file = os.path.join(os.path.dirname(__file__),\n                                                 \"installed_files\", conf_file)\n    if not env.has_key(\"distribution\"):\n        config_file = os.path.join(env.config_dir, \"fabricrc.txt\")\n        if os.path.exists(config_file):\n            env.update(load_settings(config_file))\n\nCONFIG_FILE = os.path.join(os.path.dirname(__file__), \"config\", \"biodata.yaml\")\n\ndef install_data(config_source=CONFIG_FILE):\n    \"\"\"Main entry point for installing useful biological data.\n    \"\"\"\n    setup_environment()\n    genomes.install_data(config_source)\n\ndef install_data_raw(config_source=CONFIG_FILE):\n    \"\"\"Installing useful biological data building from scratch. Useful for debugging.\n    \"\"\"\n    setup_environment()\n    genomes.install_data(config_source, approaches=[\"raw\"])\n\ndef install_data_s3(config_source=CONFIG_FILE, do_setup_environment=True):\n    \"\"\"Install data using pre-existing genomes present on Amazon s3.\n    \"\"\"\n    setup_environment()\n    genomes.install_data_s3(config_source)\n\ndef install_data_rsync(config_source=CONFIG_FILE):\n    \"\"\"Install data using Galaxy rsync data servers.\n    \"\"\"\n    setup_environment()\n    genomes.install_data_rsync(config_source)\n\ndef install_data_ggd(recipe, organism):\n    \"\"\"Install data using Get Genomics Data (GGD) recipes.\n    \"\"\"\n    setup_environment()\n    from cloudbio.biodata import ggd, genomes\n    genome_dir = os.path.join(genomes._make_genome_dir(), organism)\n    recipe_file = os.path.join(os.path.dirname(__file__), \"ggd-recipes\", organism, \"%s.yaml\" % recipe)\n    ggd.install_recipe(genome_dir, env, recipe_file, organism)\n\ndef upload_s3(config_source=CONFIG_FILE):\n    \"\"\"Upload prepared genome files by identifier to Amazon s3 buckets.\n    \"\"\"\n    setup_environment()\n    genomes.upload_s3(config_source)\n"
  },
  {
    "path": "deploy/README.md",
    "content": "# CloudBioLinux Deployer\n\nThis CloudBioLinux deployer has grown out of the galaxy-vm-launcher and\ncan be used to launch cloud virtual machines, configure them with\nGalaxy, and seed it with input data, genomes, workflows, etc.... More\nrecently, actions for installing CloudBioLinux and launching CloudMan\nhave been added.\n\n## Prerequisites\n\nThe `deploy.sh` script should install the needed dependencies in a Python\nvirtual environment using venvburrito and doesn't require special permissions\nas long as `python`, `easy_install`, and `git` are available.\n\n## Specify settings\n\nAll deploy actions first require the existence of a setting file. \n\n    cp settings-sample-oldgalaxyvmlauncher.yaml settings.yaml\n\nThis file has numerous settings to customize how the deployer acts. Be\ndefault, the deployer will target Amazon Web Services and `key_file`,\n`access_id`, and `private_key` in the aws section of of this file must\nbe specified.\n\nThe argument `--settings=/path/to/custom_settings.yaml` may be passed\nto `deploy.sh` to specify a custom path for this settings file.\n\n## Configuring Galaxy\n\n    ./deploy.sh --action=configure --action=transfer file1 file2 file3\n\nWhen called this way, deploy.sh will launch a VM, configure Galaxy,\ntools, and genomes. Once Galaxy is ready, it will transfer each of\nprovided input files to the newly launched VM, and use the Galaxy REST\nAPI to add them to a Galaxy data library (and optionally a\nhistory). Once all of that is complete, it will print a URL to the\nscreen telling the operator where to find the new Galaxy instance.\n\nThis does not install CloudMan, Galaxy is configured to run at startup\nby an init script. A more traditional CloudMan workflow can be\nachieved using the `install_biolinux` action described next.\n\n## Installing CloudBioLinux\n\n    ./deploy.sh --action=install_biolinux --action=package\n\nThis mode will launch an instance, install CloudBioLinux (a flavor can\nbe specified in setting.yaml), and `package` (see settings.yaml for\nmore details) the resulting virtual image.\n\n## Additional Actions\n\nThe actions show above can be combined in different manners, for\ninstance `configure` and `package` can be used to configure a Galaxy\ninstance and package so that later `transfer` can be used without\nrequiring a full configure. Alternatively, `install_biolinux` can be\nfollowed up with `transfer` to install CloudBioLinux and start\nanalyzing data without requiring Galaxy (be sure to set `use_galaxy:\nFalse` in settings.yaml in this case).\n\nYou can see running instances on target cloud with this command: \n\n    ./deploy.sh --action=list\n\nYou can also destroy all running instances with this command:\n    \n    ./deploy.sh --action=destroy\n\nIf an existing CloudBioLinux image bundled with CloudMan has been\ncreated and its image id set as `image_id` in the `cloudman` section\nof `setting.yaml`, then this image can be launched for testing with:\n\n    ./deploy.sh --action=launch_cloudman\n\nThe full list of actions can be found in `cloudbio/deploy/__init__.py`\nand includes:\n\n* `list`\n* `destroy`\n* `transfer`\n* `destroy`\n* `transfer`,\n* `purge_galaxy`\n* `setup_galaxy`\n* `purge_tools`\n* `setup_tools`\n* `purge_genomes`\n* `setup_genomes`\n* `setup_ssh_key`\n* `package`\n* `setup_image`\n* `launch` - Dummy action justs launches instance\n* `install_biolinux`\n* `cloudman_launch`\n\nAdditional composite actions are shortcuts for multiple actions - these include:\n\n* `configure` - `setup_image`, `setup_tools`, `setup_genomes`, `setup_ssh_key`\n* `reinstall_galaxy` - `purge_galaxy` and `setup_galaxy`\n* `reinstall_genomes` - `purge_genomes` and `setup_genomes`\n* `reinstall_tools` - `purge_tools` and `setup_tools`\n\n## Configuring Cloud Provider\n\nCloud interactions are managed via the [vm-launcher] project, full\ninformation on configuring different cloud providers can be found\n[here][vm-launcher-config]\n\nIn brief, there are few different options for where to create the\nVMs. Amazon EC2 is the default target, but it can also target\nEucalyptus or OpenStack based clouds. The ruby package `vagrant` can\nbe used to target virtual instances on your own machine.\n\n[vm-launcher-config]: https://github.com/jmchilton/vm-launcher/blob/master/config.md\n"
  },
  {
    "path": "deploy/TODO",
    "content": "TODO: Refactor use_existing_instance out of cloud specific config, make command-line option\nTODO: Allow specification of clouds by name instead of type.\nTODO: Implement alternate transfer mechanisms (fabric-based transfer has not proven very robust). \nTODO: Improve documentation.\n\n"
  },
  {
    "path": "deploy/Vagrantfile",
    "content": "# -*- mode: ruby -*-\n# vi: set ft=ruby :\n\n# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!\nVAGRANTFILE_API_VERSION = \"2\"\n\nVagrant.configure(VAGRANTFILE_API_VERSION) do |config|\n  # All Vagrant configuration is done here. The most common configuration\n  # options are documented and commented below. For a complete reference,\n  # please see the online documentation at vagrantup.com.\n\n  # Every Vagrant virtual environment requires a box to build off of.\n  config.vm.box = \"base\"\n\n  # Disable automatic box update checking. If you disable this, then\n  # boxes will only be checked for updates when the user runs\n  # `vagrant box outdated`. This is not recommended.\n  # config.vm.box_check_update = false\n\n  # Create a forwarded port mapping which allows access to a specific port\n  # within the machine from a port on the host machine. In the example below,\n  # accessing \"localhost:8080\" will access port 80 on the guest machine.\n  # config.vm.network \"forwarded_port\", guest: 80, host: 8080\n\n  # Create a private network, which allows host-only access to the machine\n  # using a specific IP.\n  # config.vm.network \"private_network\", ip: \"192.168.33.10\"\n\n  # Create a public network, which generally matched to bridged network.\n  # Bridged networks make the machine appear as another physical device on\n  # your network.\n  # config.vm.network \"public_network\"\n\n  # If true, then any SSH connections made will enable agent forwarding.\n  # Default value: false\n  # config.ssh.forward_agent = true\n\n  # Share an additional folder to the guest VM. The first argument is\n  # the path on the host to the actual folder. The second argument is\n  # the path on the guest to mount the folder. And the optional third\n  # argument is a set of non-required options.\n  # config.vm.synced_folder \"../data\", \"/vagrant_data\"\n\n  # Provider-specific configuration so you can fine-tune various\n  # backing providers for Vagrant. These expose provider-specific options.\n  # Example for VirtualBox:\n  #\n  # config.vm.provider \"virtualbox\" do |vb|\n  #   # Don't boot with headless mode\n  #   vb.gui = true\n  #\n  #   # Use VBoxManage to customize the VM. For example to change memory:\n  #   vb.customize [\"modifyvm\", :id, \"--memory\", \"1024\"]\n  # end\n  #\n  # View the documentation for the provider you're using for more\n  # information on available options.\n\n  # Enable provisioning with CFEngine. CFEngine Community packages are\n  # automatically installed. For example, configure the host as a\n  # policy server and optionally a policy file to run:\n  #\n  # config.vm.provision \"cfengine\" do |cf|\n  #   cf.am_policy_hub = true\n  #   # cf.run_file = \"motd.cf\"\n  # end\n  #\n  # You can also configure and bootstrap a client to an existing\n  # policy server:\n  #\n  # config.vm.provision \"cfengine\" do |cf|\n  #   cf.policy_server_address = \"10.0.2.15\"\n  # end\n\n  # Enable provisioning with Puppet stand alone.  Puppet manifests\n  # are contained in a directory path relative to this Vagrantfile.\n  # You will need to create the manifests directory and a manifest in\n  # the file default.pp in the manifests_path directory.\n  #\n  # config.vm.provision \"puppet\" do |puppet|\n  #   puppet.manifests_path = \"manifests\"\n  #   puppet.manifest_file  = \"site.pp\"\n  # end\n\n  # Enable provisioning with chef solo, specifying a cookbooks path, roles\n  # path, and data_bags path (all relative to this Vagrantfile), and adding\n  # some recipes and/or roles.\n  #\n  # config.vm.provision \"chef_solo\" do |chef|\n  #   chef.cookbooks_path = \"../my-recipes/cookbooks\"\n  #   chef.roles_path = \"../my-recipes/roles\"\n  #   chef.data_bags_path = \"../my-recipes/data_bags\"\n  #   chef.add_recipe \"mysql\"\n  #   chef.add_role \"web\"\n  #\n  #   # You may also specify custom JSON attributes:\n  #   chef.json = { mysql_password: \"foo\" }\n  # end\n\n  # Enable provisioning with chef server, specifying the chef server URL,\n  # and the path to the validation key (relative to this Vagrantfile).\n  #\n  # The Opscode Platform uses HTTPS. Substitute your organization for\n  # ORGNAME in the URL and validation key.\n  #\n  # If you have your own Chef Server, use the appropriate URL, which may be\n  # HTTP instead of HTTPS depending on your configuration. Also change the\n  # validation key to validation.pem.\n  #\n  # config.vm.provision \"chef_client\" do |chef|\n  #   chef.chef_server_url = \"https://api.opscode.com/organizations/ORGNAME\"\n  #   chef.validation_key_path = \"ORGNAME-validator.pem\"\n  # end\n  #\n  # If you're using the Opscode platform, your validator client is\n  # ORGNAME-validator, replacing ORGNAME with your organization name.\n  #\n  # If you have your own Chef Server, the default validation client name is\n  # chef-validator, unless you changed the configuration.\n  #\n  #   chef.validation_client_name = \"ORGNAME-validator\"\nend\n"
  },
  {
    "path": "deploy/cloudman.html",
    "content": "<h1>CloudBioLinux Deployer CloudMan QuickStart</h1>\n\n<p>As far as I can determine there is no current documentation on how to build\nCloudMan instances from scratch. Thus I am collecting my unofficial notes on\nhow to do this here - specifically using the CloudBioLinux deployer.</p>\n\n<p>You will need to navigate the AWS management console and obtain the following\ninformation.</p>\n\n<ul>\n<li><code>access_id</code></li>\n<li><code>secret_key</code></li>\n<li>Ubuntu EBS-backed AMI ID to target. This writeup was tested with ami-9b85eef2 (12.04.2 (64-bit) in us-east-1)</li>\n<li>Image size to use (e.g. m1-small)</li>\n<li>Availability zone (e.g. us-east-1)</li>\n<li>You will need to setup a bucket to store your snaps file, here you will need the bucket name.</li>\n<li>You will need to setup two volumes in your target availability zone, one for\nGalaxy tools and data (perhaps 20Gb for testing) and one for galaxyIndices. Here you will need the volume ids.</li>\n<li>Generate a private a key (e.g. galaxy1.pem) and copy it into keys directory (or anywhere really), \nalso note the keypair_name corresponding to the key.</li>\n</ul>\n\n<p>Create a directory (e.g. <code>/home/mary/marys_cloudman_bucket_contents</code>). Copy\nthe files from an existing CloudMan bucket here (e.g. http://s3.amazonaws.com\n/cloudman-dev).</p>\n\n<p>It is not really important how you download these files, but one quick option\nis to use <code>s3cmd</code> tool:</p>\n\n<pre><code>% sudo apt-get install s3cmd  # Or your OS's package manager\n% mkdir /home/mary/marys_cloudman_bucket_contents\n% s3cmd -r get s3://cloudman-dev marys_cloudman_bucket_contents\n</code></pre>\n\n<p>Here you can update cm.tar.gz or any of these files to match the customized\nsetup you would like. In particular you are going to want to create a custom\nsnaps.yaml file. Here is a simple outline that we will fill out as we good.</p>\n\n<pre><code>version: 1\nclouds:\n  - name: amazon\n    regions:\n    - deployments:\n      - name: GalaxyCloud\n        filesystems:\n        - name: galaxy\n          roles: galaxyTools,galaxyData\n          snap_id: snap-XXXXXXXXXXX\n        - name: galaxyIndices\n          roles: galaxyIndices\n          snap_id: snap-XXXXXXXXXXXX\n        default_mi: ami-XXXXXXXXXXXXX\n        bucket: marys_cloudman_bucket\n      name: us-east-1\n</code></pre>\n\n<p>Immediately this template can be updated to reflect the bucket created above\nand the availability zone you are targetting. We can update the snap<em>id's and\nthe default</em>mi after creating them.</p>\n\n<p>Copy and modify <code>settings.yaml</code>:</p>\n\n<pre><code>% cp settings-sample-cm.yaml settings.yaml\n% vim settings.yaml # or your favorite editor\n</code></pre>\n\n<p>Carefully scan through that file and change the properties marked as requiring\nchange. The keyword <code>UPDATE</code> indicates properties of special interest that\neither don't have reasonable defaults or have reasonable defaults but that I\nhave deemed highly likely to be overridden.</p>\n\n<p>Now you can use the CloudBioLinux deployer to launch an image, attach volumes,\ninstall biolinux, take needed snapshots, and package the whole thing up:</p>\n\n<pre><code>% ./deploy.sh --action=launch\n% ./deploy.sh --action=attach_volumes\n% ./deploy.sh --action=install_biolinux --flavor=cloudman/cloudman_and_galaxy\n% ./deploy.sh --action=snapshot_volumes\n% ./deploy.sh --action=detach_volumes\n% ./deploy.sh --action=package\n</code></pre>\n\n<p>Finally, update snaps.yaml in your bucket directory (e.g.\n<code>/home/mary/marys_cloudman_bucket_contents</code>) to reflect the snap_ids and AMI's\njust created. These should all be available via the AWS console or by\nreviewing the output of the steps above.</p>\n\n<p>Finally, you can upload your new bucket and launch a test CloudMan instance:</p>\n\n<pre><code>% ./deploy.sh --action=sync_cloudman_bucket\n% ./deploy.sh --action=cloudman_launch\n</code></pre>\n"
  },
  {
    "path": "deploy/cloudman.md",
    "content": "# CloudBioLinux Deployer CloudMan QuickStart\n\nAs far as I can determine there is no current documentation on how to build\nCloudMan instances from scratch. Thus I am collecting my unofficial notes on\nhow to do this here - specifically using the CloudBioLinux deployer.\n\nYou will need to navigate the AWS management console and obtain the following\ninformation.\n\n* Your AWS Access ID and secret key (`access_id`, `secret_key`)\n* Ubuntu EBS-backed AMI ID to target. This writeup was tested with ami-9b85eef2 (12.04.2 (64-bit) in us-east-1)\n* Image size to use (e.g. m1-small)\n* Availability zone (e.g. us-east-1)\n* You will need to setup a bucket to store your snaps file, here you will need the bucket name.\n* You will need to setup two volumes in your target availability zone, one for\n  Galaxy tools and data (perhaps 20Gb for testing) and one for galaxyIndices. Here you will need the volume ids.\n* Generate a private a key (e.g. galaxy1.pem) and copy it into keys directory (or anywhere really), \n  also note the keypair_name corresponding to the key.\n\nCreate a directory (e.g. `/home/mary/marys_cloudman_bucket_contents`). Copy\nthe files from an existing CloudMan bucket here (e.g. http://s3.amazonaws.com\n/cloudman-dev).\n\nIt is not really important how you download these files, but one quick option\nis to use `s3cmd` tool:\n\n    % sudo apt-get install s3cmd  # Or your OS's package manager\n    % mkdir /home/mary/marys_cloudman_bucket_contents\n    % s3cmd -r get s3://cloudman-dev /home/mary/marys_cloudman_bucket_contents\n\nHere you can replace the CloudMan source (i.e. `cm.tar.gz`) or any of these\nfiles to match the customized setup you would like. In particular you are\ngoing to want to create a custom snaps.yaml file. Here is a simple outline\nthat we will fill out as we good.\n\n    version: 1\n    clouds:\n      - name: amazon\n        regions:\n        - deployments:\n          - name: GalaxyCloud\n            filesystems:\n            - name: galaxy\n              roles: galaxyTools,galaxyData\n              snap_id: snap-XXXXXXXXXXX\n              mount_point: /mnt/galaxy\n            - name: galaxyIndices\n              roles: galaxyIndices\n              snap_id: snap-XXXXXXXXXXXX\n              mount_point: /mnt/galaxyIndices\n            default_mi: ami-XXXXXXXXXXXXX\n            bucket: marys_cloudman_bucket\n          name: us-east-1\n\nImmediately this template can be updated to reflect the bucket created above\nand the availability zone you are targetting. We can update the snap_id's and\nthe default_mi after creating them.\n\nCopy and modify `settings-sample-cm.yaml` to `settings.yaml`:\n\n    % git clone git://github.com/chapmanb/cloudbiolinux.git\n    % cd cloudbiolinux/deploy\n    % cp settings-sample-cm.yaml settings.yaml\n    % vim settings.yaml # or your favorite editor\n\nCarefully scan through `settings.yaml` and change the properties marked as requiring\nchange. The word `UPDATE` in the comments indicates properties of special\ninterest that either don't have reasonable defaults or have reasonable\ndefaults but that I have deemed highly likely to be overridden.\n\nNow you can use the CloudBioLinux deployer to launch an image, attach volumes,\ninstall biolinux, take needed snapshots, and package the whole thing up:\n\n    % ./deploy.sh --action=launch\n    % ./deploy.sh --action=attach_volumes\n    % ./deploy.sh --action=install_biolinux --flavor=cloudman/cloudman_and_galaxy\n    % ./deploy.sh --action=snapshot_volumes\n    % ./deploy.sh --action=detach_volumes\n    % ./deploy.sh --action=package\n\nIf at any point in the above process you need to interactively inspect the\nstate of the instance being configured you can do this via the following command:\n\n    % ./deploy.sh --action=ssh\n\nOnce a CloudMan AMI has been created, update `snaps.yaml` in your bucket\ndirectory (e.g. `/home/mary/marys_cloudman_bucket_contents`) to reflect the\n`snap_id`s and AMI created. These should all be available via the AWS\nmanagement console or by reviewing the output of the steps above.\n\nFinally, you can upload your new bucket and launch a test CloudMan instance:\n\n    % ./deploy.sh --action=sync_cloudman_bucket\n    % ./deploy.sh --action=cloudman_launch\n\n## Customizing\n\nThe above example uses the `cloudman/cloudman_and_galaxy` CloudBioLinux\nflavor, but there are additional flavors of CloudBioLinux available. Please\nconsult [this page][1] \nand choose the most appropriate flavor:\n\n### Customizing Galaxy\n\nInstalling a customized Galaxy is as simple as overriding the\n`galaxy_repository` variable in the `fabricrc_overrides` section of the\n`settings.yaml`.\n\n### Customizing Tools\n\nOut of the box, CloudBioLinux can be configured to install dozens of\nbioinformatic packages out of the box and adding additional packages is fairly\nstraight forward. One simply need to create a CloudBioLinux flavor that\nconfigures which such packages are installed and specify that flavor (either\nin the command-line as shown above or in `settings.yaml`).\n\nYour custom flavor should include the `cloudman` packages. If your flavor\nadditionally includes `galaxy` (as the flavor `cloudman_and_galaxy` shown\nabove) packages and `install_tool_dependencies` is set to `True` in\n`settings.yaml` - CloudBioLinux will setup a tool dependencies directory for\nGalaxy. This allows multiple versions of an application to be installed in\nisolation.\n\nWhen enabled, the list of tools and versions that is installed can be found in\n``cloudbiolinux/contrib/flavor/cloudman/tools.yaml <https://github.com/chapmanb/cloudbiolinux/blob/master/contrib/flavor/cloudman/tools.yaml>``. One can\nmodify that file directly or specify an entirely new file by setting the\n``galaxy_tools_conf`` property in the `fabric_overrides` section of `settings.yaml`.\n\n### Customizing CloudMan\n\nCloudMan is downloaded from the bucket you specify and installed at system\nstartup. Hence one can simply place a customized version of CloudMan (tarred\nup and named `cm.tar.gz`) in the bucket.\n\nIf `cloudman_repository`, `bucket_source`, and `bucket_default` are set in the\n`cloudman` section of `settings.yaml`, then one can execute the following\ncommand to quickly tar up the local copy of CloudMan (in\n`cloudman_repository`) and update your target bucket.\n\n    % ./deploy.sh --action=bundle_cloudman --action=sync_cloudman_bucket\n\n[1]: https://github.com/chapmanb/cloudbiolinux/tree/master/contrib/flavor/cloudman\n"
  },
  {
    "path": "deploy/config/tool_data_table_conf.xml",
    "content": "<!-- Use the file tool_data_table_conf.xml.oldlocstyle if you don't want to update your loc files as changed in revision 4550:535d276c92bc-->\n<tables>\n    <!-- Locations of all fasta files under genome directory -->\n    <table name=\"all_fasta\" comment_char=\"#\">\n        <columns>value, dbkey, name, path</columns>\n        <file path=\"/mnt/galaxyIndices/galaxy/tool-data/all_fasta.loc\" />\n    </table>\n    <!-- Locations of indexes in the BFAST mapper format -->\n    <table name=\"bfast_indexes\" comment_char=\"#\">\n        <columns>value, dbkey, formats, name, path</columns>\n        <file path=\"/mnt/galaxyIndices/galaxy/tool-data/bfast_indexes.loc\" />\n    </table>\n    <!-- Locations of nucleotide (mega)blast databases -->\n    <table name=\"blastdb\" comment_char=\"#\">\n        <columns>value, name, path</columns>\n        <file path=\"/mnt/galaxyIndices/galaxy/tool-data/blastdb.loc\" />\n    </table>\n    <!-- Locations of protein (mega)blast databases -->\n    <table name=\"blastdb_p\" comment_char=\"#\">\n        <columns>value, name, path</columns>\n        <file path=\"/mnt/galaxyIndices/galaxy/tool-data/blastdb_p.loc\" />\n    </table>\n    <!-- Locations of indexes in the Bowtie mapper format -->\n    <table name=\"bowtie_indexes\" comment_char=\"#\">\n        <columns>value, dbkey, name, path</columns>\n        <file path=\"/mnt/galaxyIndices/galaxy/tool-data/bowtie_indices.loc\" />\n    </table>\n    <!-- Locations of indexes in the Bowtie color-space mapper format -->\n    <table name=\"bowtie_indexes_color\" comment_char=\"#\">\n        <columns>value, dbkey, name, path</columns>\n        <file path=\"/mnt/galaxyIndices/galaxy/tool-data/bowtie_indices_color.loc\" />\n    </table>\n    <!-- Locations of indexes in the BWA mapper format -->\n    <table name=\"bwa_indexes\" comment_char=\"#\">\n        <columns>value, dbkey, name, path</columns>\n        <file path=\"/mnt/galaxyIndices/galaxy/tool-data/bwa_index.loc\" />\n    </table>\n    <!-- Locations of indexes in the BWA color-space mapper format -->\n    <table name=\"bwa_indexes_color\" comment_char=\"#\">\n        <columns>value, dbkey, name, path</columns>\n        <file path=\"/mnt/galaxyIndices/galaxy/tool-data/bwa_index_color.loc\" />\n    </table>\n    <!-- Locations of MAF files that have been indexed with bx-python -->\n    <table name=\"indexed_maf_files\">\n        <columns>name, value, dbkey, species</columns>\n        <file path=\"/mnt/galaxyIndices/galaxy/tool-data/maf_index.loc\" />\n    </table>\n    <!-- Locations of fasta files appropriate for NGS simulation -->\n    <table name=\"ngs_sim_fasta\" comment_char=\"#\">\n        <columns>value, dbkey, name, path</columns>\n        <file path=\"/mnt/galaxyIndices/galaxy/tool-data/ngs_sim_fasta.loc\" />\n    </table>\n    <!-- Locations of 2bit sequence files for use in Lastz -->\n    <table name=\"lastz_seqs\" comment_char=\"#\">\n        <columns>value, name, path</columns>\n        <file path=\"/mnt/galaxyIndices/galaxy/tool-data/lastz_seqs.loc\" />\n    </table>\n    <!-- Locations of PerM base index files -->\n    <table name=\"perm_base_indexes\" comment_char=\"#\">\n        <columns>value, name, path</columns>\n        <file path=\"/mnt/galaxyIndices/galaxy/tool-data/perm_base_index.loc\" />\n    </table>\n    <!-- Locations of PerM color-space index files -->\n    <table name=\"perm_color_indexes\" comment_char=\"#\">\n        <columns>value, name, path</columns>\n        <file path=\"/mnt/galaxyIndices/galaxy/tool-data/perm_color_index.loc\" />\n    </table>\n    <!-- Location of SAMTools indexes and other files -->\n    <table name=\"sam_fa_indexes\" comment_char=\"#\">\n        <columns>line_type, value, path</columns>\n        <file path=\"/mnt/galaxyIndices/galaxy/tool-data/sam_fa_indices.loc\" />\n    </table>\n    <!-- Location of Picard dict file and other files -->\n    <table name=\"picard_indexes\" comment_char=\"#\">\n        <columns>value, dbkey, name, path</columns>\n        <file path=\"/mnt/galaxyIndices/galaxy/tool-data/picard_index.loc\" />\n    </table>\n    <!-- Location of Picard dict files valid for GATK -->\n    <table name=\"gatk_picard_indexes\" comment_char=\"#\">\n        <columns>value, dbkey, name, path</columns>\n        <file path=\"/mnt/galaxyIndices/galaxy/tool-data/picard_index.loc\" />\n    </table>\n    <!-- Available of GATK references -->\n    <table name=\"gatk_annotations\" comment_char=\"#\">\n        <columns>value, name, gatk_value, tools_valid_for</columns>\n        <file path=\"/opt/galaxyTools/galaxy-central/tool-data/gatk_annotations.txt\" />\n    </table>\n    <!-- Location of SRMA dict file and other files -->\n    <table name=\"srma_indexes\" comment_char=\"#\">\n        <columns>value, dbkey, name, path</columns>\n        <file path=\"/mnt/galaxyIndices/galaxy/tool-data/picard_index.loc\" />\n    </table>\n    <!-- Locations of indexes in the Bowtie mapper format for TopHat to use -->\n    <table name=\"tophat_indexes\" comment_char=\"#\">\n        <columns>value, dbkey, name, path</columns>\n        <file path=\"/mnt/galaxyIndices/galaxy/tool-data/bowtie_indices.loc\" />\n    </table>\n    <!-- Locations of indexes in the Bowtie color-space mapper format for TopHat to use -->\n    <table name=\"tophat_indexes_color\" comment_char=\"#\">\n        <columns>value, dbkey, name, path</columns>\n        <file path=\"/mnt/galaxyIndices/galaxy/tool-data/bowtie_indices_color.loc\" />\n    </table>\n    <!-- Locations of configurations in the CCAT peak/region caller format -->\n    <table name=\"ccat_configurations\" comment_char=\"#\">\n        <columns>value, name, path</columns>\n        <file path=\"/mnt/galaxyIndices/galaxy/tool-data/ccat_configurations.loc\" />\n    </table>\n    <!-- Location of Mosaik files -->\n    <table name=\"mosaik_indexes\" comment_char=\"#\">\n        <columns>value, dbkey, name, path</columns>\n        <file path=\"/mnt/galaxyIndices/galaxy/tool-data/mosaik_index.loc\" />\n    </table>\n</tables>\n"
  },
  {
    "path": "deploy/deploy.sh",
    "content": "#!/bin/bash\n\n# Name of virtualenv to create using virtualenvwrapper\nVIRTUALENV_NAME=cbl_deploy\n\n# Ensure working directory is cloudbiolinux/deploy. \nPROJECT_DIRECTORY=\"$( cd \"$( dirname \"${BASH_SOURCE[0]}\" )\" && pwd )\"\ncd $PROJECT_DIRECTORY\n\n# Ensure virtualenv-burrito has been installed.\nif [ ! -e $HOME/.venvburrito/startup.sh ];\nthen\n    wget -qO- https://raw.github.com/brainsik/virtualenv-burrito/master/virtualenv-burrito.sh | $SHELL\nfi\n\n# Configure virtualenv and virtualenvwrapper with virtualenv-burrito\n. $HOME/.venvburrito/startup.sh\n\n# If no cbl_deploy virtualenv exists, create it and populate\n# requirements.\nif [ ! `lsvirtualenv | grep $VIRTUALENV_NAME` ];\nthen\n    mkvirtualenv -r requirements.txt $VIRTUALENV_NAME\nfi\n\n# Use cbl_deploy virtualenv\nworkon $VIRTUALENV_NAME\n\nsh $PROJECT_DIRECTORY/deploy_no_deps.sh \"$@\"\n"
  },
  {
    "path": "deploy/deploy_bourne.sh",
    "content": "#!/bin/sh\n\nVIRTUALENV_VERSION=${VIRTUALENV_VERSION:-1.10.1}\n\ncd `dirname $0`\nPROJECT_DIRECTORY=${PROJECT_DIRECTORY:-`pwd`}\nVENV_DIRECTORY=$PROJECT_DIRECTORY/.venv\n\nif [ ! -e $VENV_DIRECTORY ];\nthen\n    VIRTUALENV_URL=\"https://pypi.python.org/packages/source/v/virtualenv/virtualenv-${VIRTUALENV_VERSION}.tar.gz\"\n    DOWNLOAD_TAR_BALL=\"$PROJECT_DIRECTORY/virtualenv.tar.gz\"\n    PYTHON_VERSION=${PYTHON_VERSION:-`python -c \"import sys; rev = sys.version_info; str = '%d.%d' % (rev[0], rev[1]); print str\"`}\n    LOCAL_PYTHON=$PROJECT_DIRECTORY/.virtualenv\n\n    VIRTUALENV_PACKAGES_DIR=\"$LOCAL_PYTHON/lib/python${PYTHON_VERSION}/site-packages\"\n    VIRTUALENV_SOURCE_DIR=$PROJECT_DIRECTORY/.virtualenv_source\n    \n    wget -O \"$DOWNLOAD_TAR_BALL\" ${VIRTUALENV_URL}\n    tar xzvf \"$DOWNLOAD_TAR_BALL\"\n    mv virtualenv-${VIRTUALENV_VERSION} $VIRTUALENV_SOURCE_DIR\n    mkdir -p $VIRTUALENV_PACKAGES_DIR\n\n    export PYTHONPATH=$VIRTUALENV_PACKAGES_DIR:$PYTHONPATH\n    export PATH=$LOCAL_PYTHON/bin:$PATH\n\n    cd $VIRTUALENV_SOURCE_DIR\n    python setup.py install --prefix=\"$LOCAL_PYTHON\"\n    cd $PROJECT_DIRECTORY\n    virtualenv --no-site-packages $VENV_DIRECTORY\n    . $VENV_DIRECTORY/bin/activate\n    pip install -r $PROJECT_DIRECTORY/requirements.txt\nfi\n\nsh $PROJECT_DIRECTORY/deploy_no_deps.sh \"$@\"\n"
  },
  {
    "path": "deploy/deploy_no_deps.sh",
    "content": "#!/bin/sh\n\nexport PROJECT_DIRECTORY=\".\"\n\n# Add cloudbiolinux to python path and run deployment.\nexport PYTHONPATH=..:$PYTHONPATH\npython $PROJECT_DIRECTORY/../cloudbio/deploy/main.py \"$@\"\n"
  },
  {
    "path": "deploy/requirements.txt",
    "content": "fabric\nparamiko==2.10.1\nargparse\npyyaml\napache-libcloud==0.11.3\nboto\n"
  },
  {
    "path": "deploy/settings-sample-cm.yaml",
    "content": "---                                                                                                                                                                          \nvm_provider: aws                                                                                                                                                             \n                                                                                                                                                                             \nhostname: cloudman                                                                                                                                                                \n\nfabricrc_overrides:\n  ## CloudBioLinux fabric property overrides\n\n  ## Mercurial repository for Galaxy (default is https://bitbucket.org/galaxy/galaxy-central/)\n  ## e.g. for galaxyp\n  #galaxy_repository: https://bitbucket.org/galaxyp/galaxyp-central\n\n  # Location to install galaxy\n  galaxy_home: /mnt/galaxy/galaxy-app\n  # Location to setup galaxy tool dependencies\n  galaxy_tools_dir: /mnt/galaxy/tools\n  galaxy_jars_dir: /mnt/galaxy/jars\n\n  # Configure biodata location\n  data_files: \"/mnt/galaxyIndices\"\n  galaxy_base: \"/mnt/galaxyIndices/galaxy\"\n\n  galaxy_data_mount: \"/mnt/galaxy\"\n  galaxy_indices_mount: \"/mnt/galaxyIndices\"\n\n  galaxy_install_dependencies: True\n\n  # Setup an init service for Galaxy and virtual frame buffer\n  galaxy_setup_xvfb: True\n\n  distribution: ubuntu\n  # UPDATE: Change this if you are targetting quantal or even newer Ubuntu.\n  dist_name: precise\n\n  ## UPDATE: Override tools that galaxy will install, defaults to *cloudbiolinux*/contrib/flavor/cloudman/tools.yaml\n  # galaxy_tools_conf: /home/gxadmin/cloud_manage/tools.yaml\n\naws:\n  # UPDATE: Path to your galaxy keys file\n  key_file: keys/galaxy1.pem\n  # UPDATE: Set this to the name of the keypair corresponding to the above key_file.\n  keypair_name: \"galaxy1\"\n  # UPDATE: Set this to you access_id\n  access_id: \"XXXXXXXXXXXXXXXXXXXXXX\"\n  # UPDATE: Set this to your secret key.\n  secret_key: \"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n  # UPDATE: Set this to the size you want.\n  size_id: \"m1.small\"\n  # UPDATE: set this to the Ubuntu AMI you would like to target.\n  image_id: ami-9b85eef2 # EBS backed 12.04.2 (64-bit)\n  # UPDATE: Set this to the availability zone to target.\n  availability_zone: us-east-1\n\n  # Create a new instnace on first use, then reuses same instance for subsequent calls.\n  use_existing_instance: __auto__\n\n  ## Packing Options  \n\n  # Use Amazon's create image API, only works with EBS volumes and AWS.\n  package_type: create_image\n\n  # Make new AMI public\n  make_public: True\n\n  # UPDATE: Name to give AMI image.\n  package_image_name: \"marys_cloudman_${the_date_with_time}\"\n\nvolumes:\n  \n  - id: vol-XXXXXXX  # UPDATE with volume id created via AWS console.\n    path: /mnt/galaxyIndices\n    device: /dev/xvdj\n    description: \"Galaxy Indices [${the_date_with_time}]\"\n    format: __auto__  # Attempt to auto-format new partitions if needed, deactivate by setting to False.\n\n  - id: vol-XXXXXX  # UPDATE with volume id created via AWS console.\n    path: /mnt/galaxy\n    device: /dev/xvdk\n    description: \"Galaxy Tools and Data [${the_date_with_time}]\"\n    format: __auto__  # Attempt to auto-format new partitions if needed, deactivate by setting to False.\n\n\n## CloudMan Options (mostly used after image is created for launching new CloudMan instance)\ncloudman:\n  bucket_source: /home/mary/marys_cloudman_bucket_contents\n\n  ## Id of image containing cloudman to launch\n  ## Can set an AMI here, but __use_snaps__ will look in the local bucket\n  ## contents of bucket_source for a snaps.yaml file and load that AMI instead.\n  image_id: __use_snaps__\n\n  ## Size or flavor id of machine to launch. Defaults to \n  ## cloud-specific option above.\n  # size_id:\n\n  ## access_key & secret_key to for use by CloudMan. Defaults \n  ## to access_id & secret_key specified in cloud-spefic options above.\n  # access_key:\n  # secret_key: \n\n  ## Name of cloudman cluster to create.\n  cluster_name: \"cloud_test_${the_date_with_time}\"\n\n  ## Password for CloudMan admin interface.\n  # password: adminpass\n\n  ## CloudMan source (local_path, mercurial repo (e.g. https://bitbucket.org/galaxy/cloudman), \n  ## or git repo (https://github.com/jmchilton/cloudman.git).\n  ## It can be packaged and inserted into bucket_source using: \n  ##   ./deploy.sh --action=bundle_cloudman \n  # cloudman_repository: /path/to/cloudman_source\n\n  user_data:\n    # Update bucket to target.\n    bucket_default: marys_cloudman_bucket\n\n## Uncomment and update this to specify default user data properties for this\n## AMI.\n## I suppose all custom CloudMan AMIs should at least specify a new bucket \n## default right? -John\n#image_user_data:\n#  bucket_default: marys_cloudman_bucket\n\n\n# Advanced option sections ignored for CloudMan workflow outlined in cloudman.md\ngalaxy:\n\ngenomes:\n\ntools:\n"
  },
  {
    "path": "deploy/settings-sample-minimal.yaml",
    "content": "  ---                                                                                                                                                                          \nvm_provider: aws                                                                                                                                                             \n                                                                                                                                                                             \nhostname: minimal\n\nfabricrc_overrides:\n  distribution: ubuntu\n  # UPDATE: Change this if you are targetting quantal or even newer Ubuntu.\n  dist_name: precise\n\n\naws:\n  # UPDATE: Path to your galaxy keys file\n  key_file: keys/galaxy1.pem\n  # UPDATE: Set this to the name of the keypair corresponding to the above key_file.\n  keypair_name: \"galaxy1\"\n  # UPDATE: Set this to you access_id\n  access_id: \"XXXXXXXXXXXXXXXXXXXXXX\"\n  # UPDATE: Set this to your secret key.\n  secret_key: \"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n  # UPDATE: Set this to the size you want.\n  size_id: \"m1.small\"\n  # UPDATE: set this to the Ubuntu AMI you would like to target.\n  image_id: ami-9b85eef2 # EBS backed 12.04.2 (64-bit)\n  # UPDATE: Set this to the availability zone to target.\n  availability_zone: us-east-1\n\n  # Create a new instnace on first use, then reuses same instance for subsequent calls.\n  use_existing_instance: __auto__\n\n  ## Packing Options  \n\n  # Use Amazon's create image API, only works with EBS volumes and AWS.\n  package_type: create_image\n\n  # Make new AMI public\n  make_public: True\n\n  # UPDATE: Name to give AMI image.\n  package_image_name: \"marys_minimal_cbl_${the_date_with_time}\"\n\n# Advanced option sections ignored for CloudMan workflow outlined in cloudman.md\ngalaxy:\n\ngenomes:\n\ntools:\n"
  },
  {
    "path": "deploy/settings-sample-oldgalaxyvmlauncher.yaml",
    "content": "---\n\n## Set VM provider, default is aws. Other options include\n## vagrant, openstack, and eucalyptus. Specify connection \n## details in corresponding sections below.\n# vm_provider: aws\n\nhostname: galaxy\n\n## Uncomment to destroy VM after the script is complete (for testing).\n#destroy_on_complete: True\n\n## Pull in galaxy update and restart galaxy service at runtime.\nrefresh_galaxy: False\n## Seed galaxy with API users, API keys, workflows etc... at 'configure' time or 'runtime'\nseed_galaxy: configure\n\ngenome_source: default # can also set to 'S3' to download pre-existing genomes\ntool_data_table_conf_file: config/tool_data_table_conf.xml\n\n## Uncomment to compress and download genomes locally during the\n## configure or setup_genomes step.\n# stash_genomes: download\n\n## Uncomment to compress genomes and store in /opt of the VM during\n## the configure or setup_genomes_step, when galaxy boots in a fresh\n## VM this file will be decompressed into /mnt if needed.\n# stash_genomes: opt\n\n## Uncomment to upload compressed genomes previously downloaded during\n## a configure step.\n# upload_genomes: True\n\n## Add sudoers_addition lines to add options to the /etc/sudoers\n## file. Use with caution!\nsudoers_additions:\n  # This line is needed to implement max_lifetime option below.\n  - \"galaxy       ALL=NOPASSWD:   /sbin/shutdown -h now,/sbin/reboot,/sbin/halt\"\n\n## Configure max lifetime for VM in minutes.\n# max_lifetime: 10200\n\n## Configure ssmtp (right now only gmail is really supported).\n# smtp_server: smtp.gmail.com:587\n# smtp_user: <gmail username>@gmail.com\n# smtp_password: <gmail password>\n\n## Configure genome for initially uploaded files.\n# db_key: hg19\n\n## Configure history name for initially uploaded files. This history\n## name and corresponding user api key must match data found in\n## galaxy section below.\n# transfer_history_name:\n# transfer_history_api_key: \n\n## Uncomment to setup taxonomy data required by Galaxy metagenomics\n## tools during the setup_genomes action.\n# setup_taxonomy_data: True\n\n## Path to ssh key to install for galaxy user (optional). (We\n## use this to automate transfers of results back to home base \n## from the Cloud. -John)\n#galaxy_ssh_key: /path/to/ssh_private_key\n\n## Specify flavor and target for CloudBioLinux to install with \n## when configuring environmet and install (if install_biolinux\n## action is used).\n# flavor: cloudman/cloudman_and_galaxy\n# target: \n\n## Default CloudBioLinux properties to load, by default CloudBioLinux\n## will just its own defaults (either ../config/fabrircr.txt or whatever\n## is specified by the CloudBioLinux flavor).\n# fabricrc_file: ../config/fabricrc.txt\n\n## CloudMan Options (does not work with Vagrant driver)\ncloudman:\n  ## Id of image containing cloudman to launch\n  image_id: XXXXXXXXXX\n\n  ## Size or flavor id of machine to launch. Defaults to \n  ## cloud-specific option below.\n  # size_id:\n\n  ## access_key & secret_key to for use by CloudMan. Defaults \n  ## to access_id & secret_key specified in cloud-spefic options below.\n  # access_key:\n  # secret_key: \n\n  ## Name of cloudman cluster to create.\n  # cluster_name: cloudman\n\n  ## Password for CloudMan admin interface.\n  # password: adminpass\n\n\n## Set fabric env properties once virtual machine has been created, \n## these are mostly used to modify CloudBioLinux behavior and will\n## override the properties loaded by fabricrc_file defind above.\nfabricrc_overrides:\n\n  ## Mercurial repository for galaxy.  If use_galaxy is True above,\n  ## this should be fork of or contain changesets in\n  ## https://bitbucket.org/jmchilton/cloud-galaxy-dist. For a more\n  ## traditional CloudMan workflow this can be any galaxy repository.\n  galaxy_repository: https://bitbucket.org/jmchilton/cloud-galaxy-dist\n\n  ## Set to true because above galaxy repo preconfigures Galaxy\n  ## properly. Comment this out for a more traditional CloudMan \n  ## workflow.\n  galaxy_preconfigured_repository: True\n\n  ## These defaults reflect installing galaxy and tools on instance\n  ## disk. These defaults should be changed for more traditional\n  ## CloudMan workflow to reflect defaults in config/fabricrc.txt.\n\n  # Location to install galaxy\n  galaxy_home: /opt/galaxy/web\n  # Location to setup galaxy tool dependencies\n  galaxy_tools_dir: /opt/galaxy/tools\n  galaxy_jars_dir: /opt/galaxy/jars\n\n  # Configure biodata location\n  data_files: \"/mnt/galaxyIndices\"\n  galaxy_base: \"/mnt/galaxyIndices/galaxy\"\n\n\n  ## Setup Galaxy init service. For more traditional cloudman workflow,\n  ## this should be set to False.\n  galaxy_setup_service: True\n  galaxy_setup_xvfb: True\n\n  postgres_version: 9.1\n\n  ## Transferring large files is tough, have fabric try multiple\n  ## times if needed and lengthen timeouts.\n  keepalive: 30\n  timeout: 60\n  connection_attempts: 5\n\n  tool_data_table_conf_file: config/tool_data_table_conf.xml\n\n## Define Amazon connection information. This is used if `vm_provider: aws`\n## (the default) is set can also specify `eucalytus` or `openstack` sections\n## for other cloud environments.\naws:\n  # Path to private key used to connect to machine image\n  key_file: XXXXXXXXXXXXX\n\n  access_id: XXXXXXXXXXXXX\n  secret_key: XXXXXXXXXXXX\n  keypair_name: galaxy_key\n  size_id: m1.medium\n  ## Fresh Ubuntu Instance\n  image_id: ami-5c9b4935\n  availability_zone: us-east-1\n\n  ## Set this to a specific instance UUID to have deploy.sh commands use an\n  ## existing VM instead launching a new one\n  # use_existing_instance: XXXXXXXXXXXXXXXXXXXXXXXXXXXX\n\n\n  ## If package_type is default, manual packaging scripts are setup on the VM \n  ## that are appropriate for instance or EBS backed instances. If package_type\n  ## is create_image, Amazon's CreateImage instance command is used, this  is\n  ## only appropriate for EBS backed instances (requires boto).\n  # package_type: default\n\n  ## Packaging related options, all optional unless --action=package is\n  ## specified.\n\n\n  ## AWS -> Your Account -> Security Credentials -> X.509 Certificates \n  # x509_cert: /path/to/cert-XXXXXXXXXXXXXXXX.pem\n  # x509_key: /path/to/pk-XXXXXXXXXXXXXX.pem\n  ## AWS -> Your Account -> Security Credentials -> Account Identifiers -> AWS Account ID\n  # user_id: 123456789\n  ## S3 bucket to store your image into\n  # package_bucket: test1\n  ## Name of packaged instance\n  # package_image_name: Test1\n\n  ## Make the resulting image available to the public (currently only available \n  ## for package_type: create_image).\n  # make_public: False\n\n# Vagrant VM provider options, these are used if vm_provider: vagrant is set. \nvagrant:\n  key_file: ~/.vagrant.d/insecure_private_key\n\ngalaxy:\n  ## In order to create data libraries. First user should be admin@localhost\n  ## and an API key must be specified, be sure to change API keys and passwords\n  users:\n    - username: admin@localhost\n      password: adminpass\n      api_key: 1234556789\n    - username: user1@example.com\n      password: pass1\n      api_key: 987654321\n      ## Histories to create for this user\n      histories:\n      - ExampleHistory\n      ## Workflows to automatically import for this user.\n      workflows:\n      # - /path/to/workflow_file\n\n## We have had all sorts of trouble getting fabric to reliably SFTP\n## really large files, so we have made all sorts of options available\n## to tweak this process.\ntransfer:\n  ## Override what local temp directory is used on this machine if\n  ## chunking files or compression is used in transfer\n  # local_temp_dir: /tmp/\n  ## Compress transferred files (default is False)\n  # compress_transfers: True\n  ## If the following parameter is set, files will be split into\n  ## chunks of this size (in Mb) and recombined on remote host.\n  # transfer_chunk_size: 1\n  \ngenomes:\n  # Details about the genomes you want to include.\n  #  Required genome fields (corresponding to Galaxy's tool_data_table_conf.xml columns):\n  #    dbkey   - globally unique identifier for the genome (e.g., hg19)\n  #    name    - descriptive name for the given genome (to be displayed in Galaxy, e.g., Hsapiens)\n  #  Optional genome fields (corresponding to Galaxy's tool_data_table_conf.xml columns):\n  #    formats, species, dbkey1, dbkey2, value, path, index\n  #  Additional genome fields specific to data deployment:\n  #    genome_indexes - list of tool indexes specific to the associated genome (overrides global 'genome_indexes')\n  genomes:\n    - dbkey: phix\n      name: phiX174\n\n    # - dbkey: hg19\n    #   name: Human (hg19)\n    #   dbsnp: false\n    #   rnaseq: false\n\n  # Global set of indexes to include for each genome.\n  # Available choices are in GENOMES_INDEXES_SUPPORTED in cloudbiolinux/biodata/genomes.py\n  genome_indexes:\n    - bwa\n\n  # Additional data targets\n  install_liftover: false\n  install_uniref: false\n\n## This tool selection was up-to-date for the galaxy-vm-launcher, but\n## this probably should be updated based on the latest contents of\n## contrib/flavor/cloudman/tools.yaml.\ntools:\n  applications:\n    bwa: \"0.5.9\"\n    ucsc_tools: \"default\"\n    bowtie: \"0.12.7\" \n    samtools: \n      - \"0.1.16\"\n      - \"0.1.18\"\n    abyss: \"1.3.3\"\n    bfast: \"0.7.0a\"\n    velvet: \"1.2.07\"\n    perm: \"3.6\"\n    lastz: \"1.02.00\"\n    mosaik: \"default\"\n    freebayes: \"default\"\n    macs: \"1.4.2-1\"\n    emboss:\n      - version: \"6.5.7\"\n        symlink_versions: \"5.0.0\"  # Setup a symbolic link, so when Galaxy looks for 5.0.0 it finds newest version\n    fastx_toolkit: \"0.0.13.2\"\n    cufflinks: \"1.3.0\"\n    tophat: \"1.3.3\"\n    picard: \"1.56\"\n    fastqc: \"0.10.0\"  \n    gatk: \"1.4-9-g1f1233b\"  \n    maq: \"0.7.1\"\n    megablast: \"2.2.22\"\n    blast: \"2.2.25+\"\n    sputnik: \"r1\"\n    taxonomy: \"r2\"\n    add_scores: \"r1\"\n    hyphy: \"r418\"\n    srma: \"0.1.15\"\n    beam: \"2\"\n    pass: \"2.0\"\n    lps_tool: \"2010.09.30\"\n    plink: \"1.07\"\n    haploview: \"4.2b\"\n    eigenstrat: \"3.0\"\n\n  r_packages:\n    - DBI\n    - RColorBrewer\n    - RCurl\n    - RSQLite\n    - XML\n    - biglm\n    - bitops\n    - digest\n    - ggplot2\n    - graph\n    - hexbin\n    - hwriter\n    - kernlab\n    - latticeExtra\n    - leaps\n    - pamr\n    - plyr\n    - proto\n    - qvalue\n    - reshape\n    - statmod\n    - xtable\n    - yacca\n\n  bioconductor_packages:\n    - AnnotationDbi \n    - ArrayExpress \n    - ArrayTools \n    - Biobase\n    - Biostrings \n    - DynDoc \n    - GEOquery \n    - GGBase \n    - GGtools \n    - GSEABase\n    - IRanges \n    - affy \n    - affyPLM \n    - affyQCReport \n    - affydata \n    - affyio\n    - annaffy \n    - annotate \n    - arrayQualityMetrics \n    - beadarray \n    - biomaRt \n    - gcrma \n    - genefilter \n    - geneplotter \n    - globaltest \n    - hgu95av2.db \n    - limma   \n    - lumi \n    - makecdfenv \n    - marray \n    - preprocessCore \n    - ShortRead \n    - siggenes\n    - simpleaffy \n    - snpMatrix \n    - vsn \n"
  },
  {
    "path": "deploy/test_install_galaxy_tool.py",
    "content": "\"\"\"\nTest script for building Python API for installing Galaxy tools using\nCBL without any dependencies (i.e. it clones down CBL and utilizes it\nlike bcbio-nextgen's installer).\n\nGoal is to ultimately fold something like this to Galaxy tool shed\nclient code to provide high-level support for easy CloudBioLinux based\ntool installations as @chapmanb described at the 2013 BOSC Codefest.\n\n  <action type=\"cloudbiolinux_install\"\n          [cbl_revision=\"<cbl_git_changeset(default=master)>\"]\n          [cbl_url=\"<cbl_repo_url(default=https://github.com/chapmanb/cloudbiolinux)>\"]\n          [tool_name=\"<tool_name(default=use dependency package name)>\"]\n          [tool_version=\"<tool_version(default=use dependency package version)>\"]\n          />\n\n\"\"\"\n\nimport os\nfrom subprocess import check_call\nfrom tempfile import mkdtemp\nfrom getpass import getuser\n\n\nDEFAULT_CBL_URL = \"https://github.com/chapmanb/cloudbiolinux.git\"\n\n\ndef __clone_cloudbiolinux(cbl_config):\n    \"\"\"Clone CloudBioLinux to a temporary directory.\n\n    TODO: Support particular revision.\n    \"\"\"\n    cbl_url = cbl_config.get(\"repository\", DEFAULT_CBL_URL)\n    cbl_dir = mkdtemp(suffix=\"cbl\")\n    check_call([\"git\", \"clone\", cbl_url, cbl_dir])\n\n    revision = cbl_config.get(\"revision\", None)\n    if revision:\n        git_dir = os.path.join(cbl_dir, \".git\")\n        check_call([\"git\", \"--work-tree\", cbl_dir, \"--git-dir\", git_dir, \"checkout\", revision])\n    return cbl_dir\n\n\ndef install_cbl_tool(tool_name, tool_version, install_dir, cbl_config={}):\n    cbl_dir = __clone_cloudbiolinux(cbl_config)\n    cbl_install_command = [os.path.join(cbl_dir, \"deploy\", \"deploy.sh\"), \"--action\", \"install_galaxy_tool\"]\n    deployer_args = {\"vm_provider\": \"novm\",\n                     \"galaxy_tool_name\": tool_name,\n                     \"galaxy_tool_version\": tool_version,\n                     \"galaxy_tool_dir\": install_dir,\n                     \"settings\": \"__none__\"}\n    for prop, val in deployer_args.iteritems():\n        cbl_install_command.append(\"--%s\" % prop)\n        cbl_install_command.append(val)\n\n    fabric_properties = {\"use_sudo\": \"False\",\n                         \"galaxy_user\": getuser()}\n    for prop, val in fabric_properties.iteritems():\n        cbl_install_command.append(\"--fabric_property\")\n        cbl_install_command.append(prop)\n        cbl_install_command.append(\"--fabric_value\")\n        cbl_install_command.append(val)\n    check_call(cbl_install_command)\n\ncbl_config = {\"repository\": \"https://github.com/jmchilton/cloudbiolinux.git\"}\ninstall_cbl_tool(\"tint_proteomics_scripts\", \"1.19.20\", os.path.abspath(\"test_tool_dir\"), cbl_config)\n"
  },
  {
    "path": "deploy/update_dependencies.sh",
    "content": "#!/bin/bash\n\n# Name of virtualenv to create using virtualenvwrapper \nVIRTUALENV_NAME=cbl_deploy\n\n# Configure virtualenv and virtualenvwrapper with virtualenv-burrito \n. $HOME/.venvburrito/startup.sh\n\n# Upgrade dependencies\nmkvirtualenv -r requirements.txt $VIRTUALENV_NAME\n\n"
  },
  {
    "path": "doc/Makefile",
    "content": "# Makefile for Sphinx documentation\n#\n\n# You can set these variables from the command line.\nSPHINXOPTS    =\nSPHINXBUILD   = sphinx-build\nPAPER         =\nBUILDDIR      = build\n\n# Internal variables.\nPAPEROPT_a4     = -D latex_paper_size=a4\nPAPEROPT_letter = -D latex_paper_size=letter\nALLSPHINXOPTS   = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source\n# the i18n builder cannot share the environment and doctrees with the others\nI18NSPHINXOPTS  = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source\n\n.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext\n\nhelp:\n\t@echo \"Please use \\`make <target>' where <target> is one of\"\n\t@echo \"  html       to make standalone HTML files\"\n\t@echo \"  dirhtml    to make HTML files named index.html in directories\"\n\t@echo \"  singlehtml to make a single large HTML file\"\n\t@echo \"  pickle     to make pickle files\"\n\t@echo \"  json       to make JSON files\"\n\t@echo \"  htmlhelp   to make HTML files and a HTML help project\"\n\t@echo \"  qthelp     to make HTML files and a qthelp project\"\n\t@echo \"  devhelp    to make HTML files and a Devhelp project\"\n\t@echo \"  epub       to make an epub\"\n\t@echo \"  latex      to make LaTeX files, you can set PAPER=a4 or PAPER=letter\"\n\t@echo \"  latexpdf   to make LaTeX files and run them through pdflatex\"\n\t@echo \"  text       to make text files\"\n\t@echo \"  man        to make manual pages\"\n\t@echo \"  texinfo    to make Texinfo files\"\n\t@echo \"  info       to make Texinfo files and run them through makeinfo\"\n\t@echo \"  gettext    to make PO message catalogs\"\n\t@echo \"  changes    to make an overview of all changed/added/deprecated items\"\n\t@echo \"  linkcheck  to check all external links for integrity\"\n\t@echo \"  doctest    to run all doctests embedded in the documentation (if enabled)\"\n\nclean:\n\t-rm -rf $(BUILDDIR)/*\n\nhtml:\n\t$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html\n\t@echo\n\t@echo \"Build finished. The HTML pages are in $(BUILDDIR)/html.\"\n\ndirhtml:\n\t$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml\n\t@echo\n\t@echo \"Build finished. The HTML pages are in $(BUILDDIR)/dirhtml.\"\n\nsinglehtml:\n\t$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml\n\t@echo\n\t@echo \"Build finished. The HTML page is in $(BUILDDIR)/singlehtml.\"\n\npickle:\n\t$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle\n\t@echo\n\t@echo \"Build finished; now you can process the pickle files.\"\n\njson:\n\t$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json\n\t@echo\n\t@echo \"Build finished; now you can process the JSON files.\"\n\nhtmlhelp:\n\t$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp\n\t@echo\n\t@echo \"Build finished; now you can run HTML Help Workshop with the\" \\\n\t      \".hhp project file in $(BUILDDIR)/htmlhelp.\"\n\nqthelp:\n\t$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp\n\t@echo\n\t@echo \"Build finished; now you can run \"qcollectiongenerator\" with the\" \\\n\t      \".qhcp project file in $(BUILDDIR)/qthelp, like this:\"\n\t@echo \"# qcollectiongenerator $(BUILDDIR)/qthelp/CloudBioLinux.qhcp\"\n\t@echo \"To view the help file:\"\n\t@echo \"# assistant -collectionFile $(BUILDDIR)/qthelp/CloudBioLinux.qhc\"\n\ndevhelp:\n\t$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp\n\t@echo\n\t@echo \"Build finished.\"\n\t@echo \"To view the help file:\"\n\t@echo \"# mkdir -p $$HOME/.local/share/devhelp/CloudBioLinux\"\n\t@echo \"# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/CloudBioLinux\"\n\t@echo \"# devhelp\"\n\nepub:\n\t$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub\n\t@echo\n\t@echo \"Build finished. The epub file is in $(BUILDDIR)/epub.\"\n\nlatex:\n\t$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex\n\t@echo\n\t@echo \"Build finished; the LaTeX files are in $(BUILDDIR)/latex.\"\n\t@echo \"Run \\`make' in that directory to run these through (pdf)latex\" \\\n\t      \"(use \\`make latexpdf' here to do that automatically).\"\n\nlatexpdf:\n\t$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex\n\t@echo \"Running LaTeX files through pdflatex...\"\n\t$(MAKE) -C $(BUILDDIR)/latex all-pdf\n\t@echo \"pdflatex finished; the PDF files are in $(BUILDDIR)/latex.\"\n\ntext:\n\t$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text\n\t@echo\n\t@echo \"Build finished. The text files are in $(BUILDDIR)/text.\"\n\nman:\n\t$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man\n\t@echo\n\t@echo \"Build finished. The manual pages are in $(BUILDDIR)/man.\"\n\ntexinfo:\n\t$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo\n\t@echo\n\t@echo \"Build finished. The Texinfo files are in $(BUILDDIR)/texinfo.\"\n\t@echo \"Run \\`make' in that directory to run these through makeinfo\" \\\n\t      \"(use \\`make info' here to do that automatically).\"\n\ninfo:\n\t$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo\n\t@echo \"Running Texinfo files through makeinfo...\"\n\tmake -C $(BUILDDIR)/texinfo info\n\t@echo \"makeinfo finished; the Info files are in $(BUILDDIR)/texinfo.\"\n\ngettext:\n\t$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale\n\t@echo\n\t@echo \"Build finished. The message catalogs are in $(BUILDDIR)/locale.\"\n\nchanges:\n\t$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes\n\t@echo\n\t@echo \"The overview file is in $(BUILDDIR)/changes.\"\n\nlinkcheck:\n\t$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck\n\t@echo\n\t@echo \"Link check complete; look for any errors in the above output \" \\\n\t      \"or in $(BUILDDIR)/linkcheck/output.txt.\"\n\ndoctest:\n\t$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest\n\t@echo \"Testing of doctests in the sources finished, look at the \" \\\n\t      \"results in $(BUILDDIR)/doctest/output.txt.\"\n"
  },
  {
    "path": "doc/hacking.md",
    "content": "# Hacking BioLinux tips and tricks\n\nThe BioLinux tools allow building a full environment for Bioinformatics. The\ndesign allows for flexible targets (Flavors).\n\nVirtualBox + Vagrant make an ideal toying environment for building and testing\ntargets. The BioLinux regression test system (in ./test/) uses that combo too.\nPlease read the README and ./doc/vagrant documentation that come with the\nBioLinux source tree first (see http://github.com/chapmanb/cloudbiolinux).\n\n## Start with the Minimal \n\nMinimal is the smallest common denominator of all, as it\ninstalls the minimum of packages to bootstrap a full install. Once the\nvagrant box is up and running, Minimal is invoked from the desktop by\n\n          fab -f $source/fabfile.py -H target_hostname -c $source/contrib/flavor/minimal/fabricrc_debian.txt install_biolinux:flavor=$source/contrib/flavor/minimal\n\nwhere $source points to your biolinux source tree (replace 'target_hostname'\nwith 'vagrant' when using that). In fact, the testing script in\n./test/test_vagrant does exactly this! Try:\n\n          cd $my_vms\n          $source/test/test_vagrant --help\n\nand the actual run:\n\n          $source/test/test_vagrant\n\n(also read $source/test/README)\n\nThe main.yaml file ascertains the major editors are included, as well remote\naccess, version control, and the basic build system (gcc and friends). Note the\nMinimal overwrites the (apt) sources file to make sure there are no\nconflicts with user settings.\n\n## Adding install packages\n\nTo expand on the package list you can define your own main.yaml, and pass that\nin. In your main.yaml file add the meta-packages listed in\nconfig/packages.yaml. Invoke your new package list with\n\n          fab -f $source/fabfile.py -H target_hostname -c $source/contrib/flavor/minimal/fabricrc_debian.txt install_biolinux:flavor=/path/to/myproject\n\nwhere the `myproject` directory contains your main.yaml. It is that simple!\n\nIf packages.yaml is not complete, you may suggest changing its contents in the\nmain repository. The alternative is to create your own flavor, which we will do\nin a minute. The same strategy holds for the other definitions in the ./config\ndirectory, such as for Ruby gems, Python eggs, Perl CPAN, R-CRAN etc.\n\n## Define a Flavor\n\nFor a cross language Bio* project performance test I needed to create a special\nversion of BioLinux that would pull in a list of scripts and some additional\npackages.  Starting from an existing edition (in this case the Minimum edition,\nbut it also works on top of BioNode and BioLinux editions), I created a new\nflavor in ./contrib/flavor/pjotrp/biotest/biotestflavor.py, named BioTestFlavor\n(note you also need an empty __init__.py file).  A Flavor class overrides the\nFlavor methods defined in ./cloudbio/flavor/__init__.py, in particular\nrewrite_config_items, a generic hook to rewrite a list of configured items (the\npackage lists), and post_install, a post installation hook. To see what packages\nyour Flavor wants to install, simply override rewrite_config_items, and add a print\nstatement. For example:\n\n    def rewrite_config_items(self, name, packages):\n        for package in packages:\n          env.logger.info(\"Selected: \"+name+\" \"+package)\n        return packages\n\nThe flavor is itself is found through a fabricrc.txt file. The main package\nlist may be in a new main.yaml file.  Kicking it into submission:\n\n          fab -f $source/fabfile.py -H target_hostname -c $source/contrib/flavor/pjotrp/biotest/fabricrc_debian.txt install_biolinux:flavor=$source/contrib/flavor/pjotrp/biotest\n\nThe flavor module itself sets env.flavor on loading the module (this can only\nhappen once). For more examples see the files in ./contrib/flavor.\n\n## Distribute a VirtualBox\n\nWith vagrant a box can be exported with\n\n          vagrant package\n\nTo extract the contained VirtualBox vmdk:\n\n          tar xvf package.box\n\n## Flavor: change default sources (apt, yum, rpm)\n\nNote: NYI\n\nBioLinux creates a default list of package sources. These\nsources can be overridden by the Flavor.rewrite_apt_sources_list method - which\nshould return a new list.\n\n## Flavor: install additional packages\n\nThe primary way of adding new packages is by creating a new main.yaml file, as\ndiscussed above in ''Define a flavor''. In addition a flavor can define a\nmethod: BioLinux creates a list of packages. These\nsources can be overridden by the Flavor.rewrite_config_list method - which\nshould return a new list.\n\n## Flavor: filter packages\n\nTo filter/remove packages from the default list, use rewrite_config_list to\nfilter existing meta packages.\n\n## Flavor: rewrite Ruby gem, Perl CPAN, Python egg, R CRAN lists\n\nThe function rewrite_config_list also allows rewriting package lists for Ruby,\nPython, R, Perl etc. The general idea is that they define the\ninclusion of the main languages, and pull in Bio* related packages. To override\nthis behaviour use the rewrite functions, e.g.\n\n    def rewrite_config_list(self, name, list):\n      if name == 'ruby':\n        return [ 'bio' ]\n      return list\n\nonly allows the BioRuby 'bio' gem to be installed. This happens at the time\nyour meta main.yaml reads\n\n    libraries:\n      - ruby-libs\n\nand pulls in ruby-libs.yaml. One ruby-libs.yaml is shared to make sure all\neditions are up-to-date. Likewise for all the other yaml files. The\nconfiguration options are at the main.yaml (meta-package) level, and by using\nrewrite methods at Edition and Flavor levels.\n\n## Flavor: install special software\n\nBioLinux comes with a bag of tricks to install special software outside the\nmain package system. There are methods for checking out source repositories,\nand building software. There are methods for accessing public data resources\n(such as Amazon S3). These are so called custom installs which are defined in\ncustom.yaml. Each of these can be pulled in and are configured by code in the\n./cloudbio/custom/ directory.  The method fetches names from custom.yaml that\ndelegate to a method in the custom/name.py program.  These mechanisms are\nshared between BioLinux editions.\n\nBut, importantly, it is easy to role your own custom methods using a Flavor!\nThis mechanism can also be used to automatically run post-install software,\nsuch as puppet, cfruby and chef.\n\nFor example, you can tell your flavor to clone a git repository, and execute\na script by adding a post_install method to your flavor. E.g.\n\n            def post_install(self):\n                env.logger.info(\"Starting post-install\")\n                if exists('Scalability'):\n                    with cd('Scalability'):\n                       run('git pull')\n                else:\n                   _fetch_and_unpack(\"git clone git://github.com/pjotrp/Scalability.git\")\n                # Now run a post installation routine\n                run('./Scalability/scripts/hello.sh')\n\nYou can run post_install on its own (convenient for testing!) using the finalize\ntarget, e.g.\n\n         fab -H hostname -f $source/fabfile.py -c  $flavor/fabricrc_debian.txt install_biolinux:flavor=$flavor,target=finalize\n\n(Note: finalize may become post-install in the future)\n\nFor a full Flavor example see\n\n    https://github.com/pjotrp/cloudbiolinux/blob/master/contrib/flavor/pjotrp/biotest/biotestflavor.py\n\n## Individualize a Flavor with env.environment\n\nSometimes it may be useful to have post-install one-offs, for\nindividual purposes (say you want to define a user account for\nyourself). Rather than create a full Flavor for every possibility, you\ncould add a parameter to the fabricrc file. Even better, add a command\nline parameter named 'environment' to the install_biolinux parameter\nlist. E.g.\n\n         fab -H hostname -f $source/fabfile.py -c  $flavor/fabricrc_debian.txt install_biolinux:flavor=$flavor,environment=special\n\nwhich automatically becomes part of the Flavor environment state as\n'env.environment'. Use this parameter to distinguish between targets.\nWe use it for distinguishing dev,test and production environments.\n\n# More tips and tricks\n\n## Using VNC\n\nTo have a remote desktop, login with ssh to the VM and start VNC\n\n        vnc4server\n\nEnter a password. Note the output pointing to the VNC viewer (IP:1).\n\non the client (your desktop) use\n\n        vncviewer IP:1\n\n## Tip for checking BioLinux installation effects\n\nTo see the what a BioLinux install does to your system, store the settings of\nthe original (untouched) state of a VM:\n\n1. Make a dump of the current installed package list\n\n                 dpkg -l > dpkg-original-list.txt\n\n2. Store the /etc tree - one way is to use git in /etc\n\nAfter running BioLinux you can see what has been done to your system by diffing\nagainst the package list, and checking /etc.\n\n## Use the testing framework to create new Flavors\n\nBioLinux comes with a testing framework in ./test. The frame work\ncreates a new VM on a local machine. You can add tests, to check if a\nVM is complete. See the main README file for more information.\n\n## Testing a ggd recipe\n\nreference=hg38, or GRCh37, or another.\nsystem_install=/path/bcbio without bin\n\n```\nmkdir recipe_test\ncp recipe.yaml recipe_test\ncd recipe_test\ngit clone https://github.com/chapmanb/cloudbiolinux\nln -s /path/bcbio/genomes/Hsapiens/[reference]/seq seq\nexport PYTHONPATH=/path/cloudbiolinux:$PYTHONPATH\npython -c 'from cloudbio.biodata.ggd import install_recipe; install_recipe(\"/path/where_to_install\", \"/path/bcbio\", \"recipe.yaml\", \"[reference]\")'\n```\n\nIn bcbio the alternative option is to:\n- start bcbio_nextge.py upgrade [options]\n- interrupt it\n- modify the recipe in tmpbcbio-install/cloudbiolinux/ggd-recipes (not in ggd-run.sh!) \n- rerun bcbio_nextgen.py upgrade [options].\n\n## Testing conda environments with bcbio before pushing a PR:\n\nWhen bcbio is doing an update \n`bcbio_nextgen.py upgrade -u development --tools`, it pulls cloudbiolinux from github.\nTo use your copy of cloudbiolinux with modified conda yaml:\n\nBefore running bcbio upgrade do\n```\nmkdir tmpbcbio-install\nln -s /your/test/copy/of/cloudbiolinux tmpbcbio-install/cloudbiolinux\n```\nand bcbio will skip that step and use the link instead, so you can make the change locally and test it out before pushing it.\n"
  },
  {
    "path": "doc/intro/FAQ.tex",
    "content": "\n\n\n\nAre there online or local user groups for Amazon Web Services?\n\n\n\n\nUser groups are listed on the \\href{https://aws.amazon.com/about-aws/events/}{AWS Events page}.\n"
  },
  {
    "path": "doc/intro/README",
    "content": "These files make up an introductory tutorial about running CloudBioLinux on Amazon EC2. \n\nThe controlling file is called gettingStarted_Cloud-Bio-Linux.tex and mostly just has include statements for the other tex files. The pdf was made using pdflatex and calling it on gettingStarted_Cloud-Bio-Linux.tex \n\n \n"
  },
  {
    "path": "doc/intro/basicTerminology.aux",
    "content": "\\relax \n\\@writefile{toc}{\\contentsline {section}{\\numberline {A.2}Basic terminology}{29}{section.A.2}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{29}{section*.80}}\n\\@writefile{toc}{\\contentsline {paragraph}{Amazon Machine Image}{29}{section*.81}}\n\\@writefile{toc}{\\contentsline {paragraph}{Amazon Simple Storage Service}{29}{section*.82}}\n\\@writefile{toc}{\\contentsline {paragraph}{AMI}{29}{section*.83}}\n\\@writefile{toc}{\\contentsline {paragraph}{Availability Regions and Zones}{29}{section*.84}}\n\\@writefile{toc}{\\contentsline {paragraph}{AWS}{29}{section*.85}}\n\\@writefile{toc}{\\contentsline {paragraph}{AWS console}{29}{section*.86}}\n\\@writefile{toc}{\\contentsline {paragraph}{EBS}{29}{section*.87}}\n\\@writefile{toc}{\\contentsline {paragraph}{EC2}{29}{section*.88}}\n\\@writefile{toc}{\\contentsline {paragraph}{Elastic Block Storage}{29}{section*.89}}\n\\@writefile{toc}{\\contentsline {paragraph}{Elastic Compute Cloud}{30}{section*.90}}\n\\@writefile{toc}{\\contentsline {paragraph}{Image}{30}{section*.91}}\n\\@writefile{toc}{\\contentsline {paragraph}{Instance}{30}{section*.92}}\n\\@writefile{toc}{\\contentsline {paragraph}{NX}{30}{section*.93}}\n\\@writefile{toc}{\\contentsline {paragraph}{Putty}{30}{section*.94}}\n\\@writefile{toc}{\\contentsline {paragraph}{S3}{30}{section*.95}}\n\\@writefile{toc}{\\contentsline {paragraph}{SSH}{30}{section*.96}}\n\\@writefile{toc}{\\contentsline {paragraph}{Snapshot}{30}{section*.97}}\n\\@writefile{toc}{\\contentsline {paragraph}{Volume}{30}{section*.98}}\n\\@setckpt{basicTerminology}{\n\\setcounter{page}{31}\n\\setcounter{equation}{0}\n\\setcounter{enumi}{4}\n\\setcounter{enumii}{0}\n\\setcounter{enumiii}{0}\n\\setcounter{enumiv}{0}\n\\setcounter{footnote}{0}\n\\setcounter{mpfootnote}{0}\n\\setcounter{part}{0}\n\\setcounter{chapter}{1}\n\\setcounter{section}{2}\n\\setcounter{subsection}{0}\n\\setcounter{subsubsection}{0}\n\\setcounter{paragraph}{0}\n\\setcounter{subparagraph}{0}\n\\setcounter{figure}{3}\n\\setcounter{table}{0}\n\\setcounter{SC@C}{5}\n\\setcounter{Item}{32}\n\\setcounter{Hfootnote}{9}\n\\setcounter{section@level}{4}\n}\n"
  },
  {
    "path": "doc/intro/basicTerminology.tex",
    "content": "\\section{Basic terminology}\n\\paragraph{}One of the most daunting parts of starting to work on the cloud is the abundance of new words and acronyms that litter the documentation. Here are a few of the most common terms you'll come across, and what they mean when used in the cloud computing context.\n\n\\paragraph{Amazon Machine Image} - Pre-configured machine images that you can start up and work on. Most commonly referred to as AMI, or just \"image\" in documentation.\n\n\\paragraph{Amazon Simple Storage Service} - More commonly known as \"S3\", this is a storage type offered as a web service. This document is primarily concerned with EBS Volumes for storage, rather than S3. However, if you take a snapshot of a volume, it is stored in S3. There are many \\href{http://www.cloudiquity.com/2009/03/differences-between-s3-and-ebs/}{differences between EBS and S3}, which you should find out about if you plan to work on the cloud seriously.\n\n\\paragraph{AMI} - see \"Amazon Machine Image\".\n\n\\paragraph{Availability Regions and Zones} - Amazon have a number of data centres around the world. A Region in this context described centres in different geographic areas (e.g. the U.S. and Europe). An Availability Zone describes distinct locations within a region. For a new user, the most likely thing to look out for is that you create volumes in the same region and zone as your running instance(s). For more detailed information about regions and zones, check out the \\href{http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/concepts-regions-availability-zones.html}{EC2 Userguide}, including the \\href{http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/index.html?FAQ_Regions_Availability_Zones.html}{Region and Availability Zone FAQ}.  \n\n\\paragraph{AWS} - Stands for \"Amazon Web Services\". An umbrella name for the myriad of web services offered by Amazon. These services include the provision of compute power and storage, which are the focus of this document.\n\n\\paragraph{AWS console} - \\href{http://aws.amazon.com/console/}{A graphical web interface to your AWS account}. From this interface, you can create, manage and delete Amazon resources such as system instances and data volumes.\n\n\\paragraph{EBS} - See \"Elastic Block Storage\".\n\n\\paragraph{EC2} - See \"Elastic Compute Cloud\"  \n\n\\paragraph{Elastic Block Storage} - Also referred to as \"volumes\". A type of storage that was designed for Amazon EC2 instances. You can create EBS volumes and mount these as devices, as you might with a hard drive on a standard system. EBS volumes are particularly useful if you plan to store data for multiple uses.\n\n\\paragraph{Elastic Compute Cloud} - Also known as EC2, this is an Amazon web service that makes computational capacity available as a virtual computing environment. When you start up an image, you do so from within an EC2 account, from the EC2 section of the AWS Console. \n\n\\paragraph{Image} - See \"Amazon Machine Image\"\n\n\\paragraph{Instance} - See \"Amazon Machine Image\"\n\n\\paragraph{NX} - At a user level, NX is a method allowing you to experience a full, graphical desktop session on a machine you are logged into remotely. \\href{http://en.wikipedia.org/wiki/NX_technology}{Documentation outlining how this is achieved is available}.\n\n\\paragraph{Putty} - In the context of this document, Putty is a program that can be used on Windows as an SSH client.\n\n\\paragraph{S3} - See \"Amazon Simple Storage Service\"   \n\n\\paragraph{SSH} - Stands for Secure Shell. A network protocol that allows data to be exchanged using a secure channel. Information sent over an SSH connection is encrypted. The standard port for SSH connections is port 22.\n\n\\paragraph{Snapshot} - A snapshot is essentially a copy of a volume. Taking a snapshot of an EBS volume means that you save the state of your volume, as a snapshot, in Amazon S3. This snapshot is then replicated across multiple Availability Zones. This means that the information on your EBS volume is saved in a durable manner. Snapshots can then be used as the starting point for new EBS volumes. For example, if you select a snapshot in the AWS Console, you can choose to make a volume of it. Once you have attached and mounted that volume to a running image, you have access to all the information you saved when you took the snapshot. Snapshots can also be used when sharing data or images. See the official documentation for more. \n\n\\paragraph{Volume} - See \"Elastic Block Storage\".\n\n\n\n\n"
  },
  {
    "path": "doc/intro/cloudbl_desktopIntro.aux",
    "content": "\\relax \n\\@writefile{toc}{\\contentsline {section}{\\numberline {A.1}The CloudBioLinux Desktop}{27}{section.A.1}}\n\\newlabel{section:cloudblDesktop}{{A.1}{27}{The CloudBioLinux Desktop\\relax }{section.A.1}{}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{27}{section*.75}}\n\\@writefile{toc}{\\contentsline {subsection}{\\numberline {A.1.1}Bioinformatics documentation}{27}{subsection.A.1.1}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{27}{section*.76}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{27}{section*.77}}\n\\@writefile{lof}{\\contentsline {figure}{\\numberline {A.1}{\\ignorespaces Bioinformatics docs}}{27}{figure.A.1}}\n\\newlabel{fig:bldocs}{{A.1}{27}{Bioinformatics docs\\relax }{figure.A.1}{}}\n\\newlabel{SC@4}{{A.1.2}{28}{\\relax }{section*.78}{}}\n\\@writefile{lof}{\\contentsline {figure}{\\numberline {A.2}{\\ignorespaces Bioinformatics menu}}{28}{figure.A.2}}\n\\newlabel{fig:bioinfmenu}{{A.2}{28}{\\SC@OPTCAPtext \\relax }{figure.A.2}{}}\n\\newlabel{SC@5}{{A.1.3}{28}{\\relax }{section*.79}{}}\n\\@writefile{lof}{\\contentsline {figure}{\\numberline {A.3}{\\ignorespaces Open terminal}}{28}{figure.A.3}}\n\\newlabel{fig:NXterminal}{{A.3}{28}{\\SC@OPTCAPtext \\relax }{figure.A.3}{}}\n\\@writefile{toc}{\\contentsline {subsection}{\\numberline {A.1.2}The bioinformatics menu}{28}{subsection.A.1.2}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{28}{section*.78}}\n\\@writefile{toc}{\\contentsline {subsection}{\\numberline {A.1.3}Opening a terminal}{28}{subsection.A.1.3}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{28}{section*.79}}\n\\@setckpt{cloudbl_desktopIntro}{\n\\setcounter{page}{29}\n\\setcounter{equation}{0}\n\\setcounter{enumi}{4}\n\\setcounter{enumii}{0}\n\\setcounter{enumiii}{0}\n\\setcounter{enumiv}{0}\n\\setcounter{footnote}{0}\n\\setcounter{mpfootnote}{0}\n\\setcounter{part}{0}\n\\setcounter{chapter}{1}\n\\setcounter{section}{1}\n\\setcounter{subsection}{3}\n\\setcounter{subsubsection}{0}\n\\setcounter{paragraph}{0}\n\\setcounter{subparagraph}{0}\n\\setcounter{figure}{3}\n\\setcounter{table}{0}\n\\setcounter{SC@C}{5}\n\\setcounter{Item}{32}\n\\setcounter{Hfootnote}{9}\n\\setcounter{section@level}{4}\n}\n"
  },
  {
    "path": "doc/intro/cloudbl_desktopIntro.tex",
    "content": "\\section{The CloudBioLinux Desktop}\n\\label{section:cloudblDesktop}\n\\paragraph{}This section provides only a few tips about the CloudBioLinux desktop. As CloudBioLinux is based on Ubuntu, we recommend referring to online Ubuntu documentation for further details about the system in general. \n\n\\subsection{Bioinformatics documentation}\n\n\\paragraph{}CloudBioLinux comes with a categorised bioinformatics documentation system, which provides information about the bioinformatics software on the system. This is the easiest place to find out about what bioinformatics programs are available, and where to find out more about how to use them. \n\n\\paragraph{}At the moment, the easiest way to get to the installed documents is to open up a web browser (just click on the little firefox icon in the top taskbar), and then enter the address:  \\textbf{file:///var/www/bioinformatics/index.html}. See figure \\ref{fig:bldocs}.\n\n\\begin{figure}[!hd]\n\t\\fbox\n\t{\n\t\t\\begin{minipage}{13cm}\n\\includegraphics[width=\\maxwidth]{\"images/blbioinfDocumentation\"}\n\\caption[Bioinformatics docs]{\\label{fig:bldocs}A categorised bioinformatics documentation system. Perusing the information in these pages is probably the easiest way to start finding out what is available on your CloudBioLinux image.}\n\t\t\\end{minipage}\n\t}\n\\end{figure}\n\n\n\\subsection{The bioinformatics menu}\n\n\\paragraph{}Some of the bioinformatics software on the system can be accessed via the graphical bioinformatics menu. See figure \\ref{fig:bioinfmenu}. \n\n\\begin{SCfigure}[][t]\n\\includegraphics[width=40mm]{\"images/bioinfMenuNX2\"}\n\\caption[Bioinformatics menu]{\\label{fig:bioinfmenu}Some bioinformatics programs can be launched from the Bioinformatics menu. For a more comprehensive listing of the bioinformatics software on the system, check out the installed bioinformatics documentation pages. See figure \\ref{fig:bldocs}.}\n\\end{SCfigure}\n\n\n\\subsection{Opening a terminal}\n\n\\paragraph{}Much of the bioinformatics software on CloudBioLinux systems can only be launched from the command line. This includes command line tools, and also some graphical software. In addition, much of the power of Linux comes through using the command line. You can easily open a terminal, in which you can type commands, when logged into a full desktop session. Just go to the \\textbf{Accessories submenu} under the Applications menu in the top taskbar, and choose \\textbf{Terminal}. See figure \\ref{fig:NXterminal}. \n\n\n\\begin{SCfigure}[][t]\n\\includegraphics[width=40mm]{\"images/openTerminalNX\"}\n\\caption[Open terminal]{\\label{fig:NXterminal}You can open a command line terminal from under the Accessories submenu of the Applications menu in the top taskbar.}\n\\end{SCfigure}\n"
  },
  {
    "path": "doc/intro/getReady.aux",
    "content": "\\relax \n\\newlabel{SC@1}{{1.1}{3}{\\relax }{section*.2}{}}\n\\@writefile{lof}{\\contentsline {figure}{\\numberline {1.1}{\\ignorespaces Sign up for AWS}}{3}{figure.1.1}}\n\\newlabel{fig:aws_signup}{{1.1}{3}{\\SC@OPTCAPtext \\relax }{figure.1.1}{}}\n\\@writefile{toc}{\\contentsline {section}{\\numberline {1.1}Get an Amazon AWS account}{3}{section.1.1}}\n\\newlabel{section:getaccount}{{1.1}{3}{Get an Amazon AWS account\\relax }{section.1.1}{}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{3}{section*.2}}\n\\@writefile{toc}{\\contentsline {paragraph}{The rest of this document assumes you have an AWS account and you are logged into it.}{3}{section*.3}}\n\\@writefile{toc}{\\contentsline {section}{\\numberline {1.2}Get an Amazon EC2 Account}{3}{section.1.2}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{3}{section*.4}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{3}{section*.5}}\n\\@writefile{toc}{\\contentsline {paragraph}{To get your EC2 account}{3}{section*.6}}\n\\@writefile{toc}{\\contentsline {section}{\\numberline {1.3}Get an EC2 key pair}{3}{section.1.3}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{3}{section*.7}}\n\\@writefile{lof}{\\contentsline {figure}{\\numberline {1.2}{\\ignorespaces EC2 Homepage}}{4}{figure.1.2}}\n\\newlabel{fig:ec2homepage}{{1.2}{4}{EC2 Homepage\\relax }{figure.1.2}{}}\n\\@writefile{toc}{\\contentsline {paragraph}{To create a key pair:}{4}{section*.8}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{4}{section*.9}}\n\\newlabel{SC@2}{{1.3}{5}{\\relax }{figure.1.2}{}}\n\\@writefile{lof}{\\contentsline {figure}{\\numberline {1.3}{\\ignorespaces Keypair creation}}{5}{figure.1.3}}\n\\newlabel{fig:keypaircreate}{{1.3}{5}{\\SC@OPTCAPtext \\relax }{figure.1.3}{}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{5}{section*.10}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{5}{section*.11}}\n\\@writefile{toc}{\\contentsline {paragraph}{How the EC2 key pairs work}{5}{section*.12}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{5}{section*.13}}\n\\@writefile{toc}{\\contentsline {section}{\\numberline {1.4}Install an NX Client on your local machine}{5}{section.1.4}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{5}{section*.14}}\n\\@writefile{toc}{\\contentsline {paragraph}{You only need to install an NX Client on your local system if you wish to run a full graphical desktop session for working on CloudBioLinux.}{5}{section*.15}}\n\\@setckpt{getReady}{\n\\setcounter{page}{6}\n\\setcounter{equation}{0}\n\\setcounter{enumi}{5}\n\\setcounter{enumii}{0}\n\\setcounter{enumiii}{0}\n\\setcounter{enumiv}{0}\n\\setcounter{footnote}{2}\n\\setcounter{mpfootnote}{0}\n\\setcounter{part}{0}\n\\setcounter{chapter}{1}\n\\setcounter{section}{4}\n\\setcounter{subsection}{0}\n\\setcounter{subsubsection}{0}\n\\setcounter{paragraph}{0}\n\\setcounter{subparagraph}{0}\n\\setcounter{figure}{3}\n\\setcounter{table}{0}\n\\setcounter{SC@C}{2}\n\\setcounter{Item}{9}\n\\setcounter{Hfootnote}{2}\n\\setcounter{section@level}{4}\n}\n"
  },
  {
    "path": "doc/intro/getReady.tex",
    "content": "\\section{Get an Amazon AWS account}\n\\label{section:getaccount}\n\\paragraph{}Anyone can set up an account with Amazon to access their computer cloud. Just go to \\href{http://aws.amazon.com}{http://aws.amazon.com} and sign up for an account.\n\n\\begin{SCfigure}[][t]\n\\includegraphics[width=40mm]{\"images/aws_signup_button\"}\n\\caption[Sign up for AWS]{\\label{fig:aws_signup}Signing up for an AWS account starts with the click of a button.}\n\\end{SCfigure}\n\n\\paragraph{The rest of this document assumes you have an AWS account and you are logged into it.}\n\n\\section{Get an Amazon EC2 Account}\n\n\\paragraph{}There are various ways you can access the power of the Amazon cloud. In this document, we describe using EC2. \n\n\\paragraph{}If you do not already have one, you need to sign up for an EC2 account. This is in addition to the general Amazon aws account you have if you followed the instructions in section \\ref{section:getaccount}. \n\n\\paragraph{To get your EC2 account}, \n\\begin{enumerate}\n\\item click on the Products tab on the \\href{http://aws.amazon.com}{Amazon aws page},\n\\item click on the \\href{http://aws.amazon.com/ec2/}{Amazon Elastic Compute Cloud (EC2) link} that appears in the Compute section of the listing, and\n\\item click on the button in the right hand pane that says Sign up for Amazon EC2.  \n\\item Complete the registration process\\footnote{Signing up for Amazon EC2 also automatically signs you up for Amazon Simple Storage Service and Amazon Virtual Private Cloud. You will not be charged for any service unless you use it.}.\n\\end{enumerate}\n\n\\section{Get an EC2 key pair}\n\n\\paragraph{}After you've signed up for your account, Amazon will send you an email with a link in it to the Access Identifiers section of your account. Amazon provides a list of \\href{http://docs.amazonwebservices.com/AWSSecurityCredentials/1.0/AboutAWSCredentials.html#EC2Credentials}{which credentials you need to do particular tasks}.  If all you will be doing is starting up Bio-Linux using the Amazon (graphical) console, then you only need an \\href{http://docs.amazonwebservices.com/AWSSecurityCredentials/1.0/AboutAWSCredentials.html#EC2KeyPairs}{Amazon EC2 Key Pair}. \n\n\\begin{figure}[!hd]\n\t\\fbox\n\t{\n\t\t\\begin{minipage}{13cm}\n%the maxwidth variable is defined in the summary page: gettingStarted_CloudBioLinux.tex\n\\includegraphics[width=\\maxwidth]{\"images/EC2Homepage2\"}\n\\caption[EC2 Homepage]{\\label{fig:ec2homepage}Your AWS home are will look something like this. Here, we are looking at the information under the EC2 tab (Red Circle). Blue Circle: your Key Pairs - if you don't have any, click on the link to create some. Purple Circle: You can easily launch any publicly available EC2 image from Amazon by clicking on this button. Green Circle: The number of running instances you have. If you have any, you can find out more about them by clicking on this link. Pink Circle: If you add Key Pairs, or start up instances, you may need to hit the refresh button to see changes on your EC2 Dashboard.}\n\t\t\\end{minipage}\n\t}\n\\end{figure}\n\n\n\\begin{SCfigure}[][t]\n%the maxwidth variable is defined in the summary page: gettingStarted_CloudBioLinux.tex\n\\includegraphics[width=60mm]{\"images/keypairCreateButton\"}\n\\caption[Keypair creation]{\\label{fig:keypaircreate}Keypair creation is simple - just click on the button and follow the instructions on screen.}\n\\end{SCfigure}\n\n\\paragraph{To create a key pair:}\n\\begin{enumerate}\n\\item Go to the the \\href{https://console.aws.amazon.com/ec2/home}{EC2 area on Amazon https://console.aws.amazon.com/ec2/home}.\n\\item Click on the \\emph{Key Pairs} link under My Resources in the right hand area of the window. See the blue circle in figure~\\ref{fig:ec2homepage}.\n\\item Click on the \\emph{Create Key Pair} button near the top of the Key Pairs section of the window. See figure \\ref{fig:keypaircreate} on page \\pageref{fig:keypaircreate}.\n\\item Give your key pair a memorable name when prompted. Save your private key to a safe location. See the further information below about this. \n\\item Click on the link in the left hand pane to go back to the \\emph{EC2 Dashboard} and then click on the \\emph{Refresh button} at the far right hand side of the window (see pink circle in figure~\\ref{fig:ec2homepage}). \n\\end{enumerate}\n\n\\paragraph{}You should now see that you have a key pair registered in the \\emph{My Resources} section. \n\n\\paragraph{}Each EC2 key pair includes a private key file and a public key file. \\textbf{Save your private key to a secure and memorable location}. Don't lose it or share it. (Amazon does not make a copy of it.) If you're working on Linux, adjust the permissions on your key file so it is readable only by you. \n\n\\paragraph{}If you plan to use the command line tools to start up an instance, you will also need to get your \\href{http://docs.amazonwebservices.com/AWSSecurityCredentials/1.0/AboutAWSCredentials.html#X509Credentials}{X509 certificates}. This document assumes that you will only be using the graphical console, so this is not covered further here. \n\n\\paragraph{How the EC2 key pairs work}\n\n\\paragraph{}When you launch a CloudBioLinux instance from  Amazon, you will specify a particular EC2 key pair. The Amazon system puts a copy of your public key, which it has a record of, on the instance. You, as the (only!) holder of the private key will be the only one able to access the CloudBioLinux instance you just started up. \n\n\n\\section{Install an NX Client on your local machine}\n\n\\paragraph{}To run a full desktop session, you need have NX client software on the machine you will be connecting to the Amazon Cloud from. If you don't already have NX Client software installed, we recommend the free clients from \\href{http://www.nomachine.com/download.php}{NoMachine}\\footnote{CloudBioLinux already has the FreeNXServer software installed}.\n%put that server line in a footnote, because for anyone not familiar with \"server\" and \"client\", it might be easy to get confused. \n\n\\paragraph{You only need to install an NX Client on your local system if you wish to run a full graphical desktop session for working on CloudBioLinux.}  \n\n\n"
  },
  {
    "path": "doc/intro/gettingStarted_CloudBioLinux.aux",
    "content": "\\relax \n\\ifx\\hyper@anchor\\@undefined\n\\global \\let \\oldcontentsline\\contentsline\n\\gdef \\contentsline#1#2#3#4{\\oldcontentsline{#1}{#2}{#3}}\n\\global \\let \\oldnewlabel\\newlabel\n\\gdef \\newlabel#1#2{\\newlabelxx{#1}#2}\n\\gdef \\newlabelxx#1#2#3#4#5#6{\\oldnewlabel{#1}{{#2}{#3}}}\n\\AtEndDocument{\\let \\contentsline\\oldcontentsline\n\\let \\newlabel\\oldnewlabel}\n\\else\n\\global \\let \\hyper@last\\relax \n\\fi\n\n\\@writefile{toc}{\\contentsline {chapter}{\\numberline {1}Preparing to work on the Amazon Cloud}{2}{chapter.1}}\n\\@writefile{lof}{\\addvspace {10\\p@ }}\n\\@writefile{lot}{\\addvspace {10\\p@ }}\n\\@input{getReady.aux}\n\\@writefile{toc}{\\contentsline {chapter}{\\numberline {2}Running a CloudBioLinux instance}{6}{chapter.2}}\n\\@writefile{lof}{\\addvspace {10\\p@ }}\n\\@writefile{lot}{\\addvspace {10\\p@ }}\n\\@input{workingOnCloudBL.aux}\n\\@writefile{toc}{\\contentsline {chapter}{\\numberline {3}Working with data on the cloud}{17}{chapter.3}}\n\\@writefile{lof}{\\addvspace {10\\p@ }}\n\\@writefile{lot}{\\addvspace {10\\p@ }}\n\\@input{workingWithData.aux}\n\\@writefile{toc}{\\contentsline {chapter}{\\numberline {A}Appendices}{26}{appendix.A}}\n\\@writefile{lof}{\\addvspace {10\\p@ }}\n\\@writefile{lot}{\\addvspace {10\\p@ }}\n\\@input{cloudbl_desktopIntro.aux}\n\\@input{basicTerminology.aux}\n"
  },
  {
    "path": "doc/intro/gettingStarted_CloudBioLinux.out",
    "content": "\\BOOKMARK [0][-]{chapter.1}{Preparing to work on the Amazon Cloud}{}\n\\BOOKMARK [1][-]{section.1.1}{Get an Amazon AWS account}{chapter.1}\n\\BOOKMARK [1][-]{section.1.2}{Get an Amazon EC2 Account}{chapter.1}\n\\BOOKMARK [1][-]{section.1.3}{Get an EC2 key pair}{chapter.1}\n\\BOOKMARK [1][-]{section.1.4}{Install an NX Client on your local machine}{chapter.1}\n\\BOOKMARK [0][-]{chapter.2}{Running a CloudBioLinux instance}{}\n\\BOOKMARK [1][-]{section.2.1}{The process in a nutshell}{chapter.2}\n\\BOOKMARK [1][-]{section.2.2}{Starting up a CloudBioLinux instance}{chapter.2}\n\\BOOKMARK [1][-]{section.2.3}{Connecting to and logging into your CloudBioLinux instance}{chapter.2}\n\\BOOKMARK [2][-]{subsection.2.3.1}{Graphical, or command line?}{section.2.3}\n\\BOOKMARK [2][-]{subsection.2.3.2}{Find out the address of your instance}{section.2.3}\n\\BOOKMARK [2][-]{subsection.2.3.3}{Logging into a terminal using ssh}{section.2.3}\n\\BOOKMARK [2][-]{subsection.2.3.4}{Logging into graphical desktop using NX}{section.2.3}\n\\BOOKMARK [2][-]{subsection.2.3.5}{The logic of the NX setup}{section.2.3}\n\\BOOKMARK [1][-]{section.2.4}{Logging out of your CloudBioLinux instance}{chapter.2}\n\\BOOKMARK [1][-]{section.2.5}{Terminating your CloudBioLinux instance}{chapter.2}\n\\BOOKMARK [0][-]{chapter.3}{Working with data on the cloud}{}\n\\BOOKMARK [1][-]{section.3.1}{Introduction}{chapter.3}\n\\BOOKMARK [1][-]{section.3.2}{Copying data onto your CloudBioLinux instance}{chapter.3}\n\\BOOKMARK [1][-]{section.3.3}{Using EBS volumes for data}{chapter.3}\n\\BOOKMARK [1][-]{section.3.4}{Accesing public datasets on Amazon}{chapter.3}\n\\BOOKMARK [0][-]{appendix.A}{Appendices}{}\n\\BOOKMARK [1][-]{section.A.1}{The CloudBioLinux Desktop}{appendix.A}\n\\BOOKMARK [2][-]{subsection.A.1.1}{Bioinformatics documentation}{section.A.1}\n\\BOOKMARK [2][-]{subsection.A.1.2}{The bioinformatics menu}{section.A.1}\n\\BOOKMARK [2][-]{subsection.A.1.3}{Opening a terminal}{section.A.1}\n\\BOOKMARK [1][-]{section.A.2}{Basic terminology}{appendix.A}\n"
  },
  {
    "path": "doc/intro/gettingStarted_CloudBioLinux.tex",
    "content": "\\documentclass[pdftex]{report}\n\\usepackage[pdftex]{graphicx}\n%\\usepackage{graphicx}\n%\\usepackage{floatflt}\n\\usepackage{sidecap}\n\\usepackage{hyperref}\n\n\\begin{document}\n\\title{Getting started with CloudBioLinux}\n\\author{Konstantinos Krampis, Bela Tiwari}\n\\maketitle\n\n\\makeatletter\n\t\\def\\maxwidth\n\t{\n\t\\ifdim\\Gin@nat@width>\\linewidth\n\t\t\\linewidth\n\t\\else\n\t\t\\Gin@nat@width\n\t\\fi\n\t}\n\\makeatother\n\\tableofcontents\n\\chapter{Preparing to work on the Amazon Cloud}\n\\include{getReady}\n\n\n\\chapter{Running a CloudBioLinux instance}\n\\include{workingOnCloudBL}\n\n\\chapter{Working with data on the cloud}\n\n\\include{workingWithData}\n\n\\appendix\n\\chapter{Appendices}\n\\include{cloudbl_desktopIntro}\n\\include{basicTerminology}\n\n%\\include{FAQ}\n\n\n\\end{document}\n"
  },
  {
    "path": "doc/intro/gettingStarted_CloudBioLinux.toc",
    "content": "\\contentsline {chapter}{\\numberline {1}Preparing to work on the Amazon Cloud}{2}{chapter.1}\n\\contentsline {section}{\\numberline {1.1}Get an Amazon AWS account}{3}{section.1.1}\n\\contentsline {paragraph}{}{3}{section*.2}\n\\contentsline {paragraph}{The rest of this document assumes you have an AWS account and you are logged into it.}{3}{section*.3}\n\\contentsline {section}{\\numberline {1.2}Get an Amazon EC2 Account}{3}{section.1.2}\n\\contentsline {paragraph}{}{3}{section*.4}\n\\contentsline {paragraph}{}{3}{section*.5}\n\\contentsline {paragraph}{To get your EC2 account}{3}{section*.6}\n\\contentsline {section}{\\numberline {1.3}Get an EC2 key pair}{3}{section.1.3}\n\\contentsline {paragraph}{}{3}{section*.7}\n\\contentsline {paragraph}{To create a key pair:}{4}{section*.8}\n\\contentsline {paragraph}{}{4}{section*.9}\n\\contentsline {paragraph}{}{5}{section*.10}\n\\contentsline {paragraph}{}{5}{section*.11}\n\\contentsline {paragraph}{How the EC2 key pairs work}{5}{section*.12}\n\\contentsline {paragraph}{}{5}{section*.13}\n\\contentsline {section}{\\numberline {1.4}Install an NX Client on your local machine}{5}{section.1.4}\n\\contentsline {paragraph}{}{5}{section*.14}\n\\contentsline {paragraph}{You only need to install an NX Client on your local system if you wish to run a full graphical desktop session for working on CloudBioLinux.}{5}{section*.15}\n\\contentsline {chapter}{\\numberline {2}Running a CloudBioLinux instance}{6}{chapter.2}\n\\contentsline {section}{\\numberline {2.1}The process in a nutshell}{7}{section.2.1}\n\\contentsline {paragraph}{}{7}{section*.16}\n\\contentsline {paragraph}{}{8}{section*.17}\n\\contentsline {paragraph}{A note about charging:}{8}{section*.18}\n\\contentsline {paragraph}{}{8}{section*.19}\n\\contentsline {paragraph}{A couple of things to note when starting out:}{8}{section*.20}\n\\contentsline {section}{\\numberline {2.2}Starting up a CloudBioLinux instance}{8}{section.2.2}\n\\contentsline {paragraph}{This document focusses on using the \\href {http://console.aws.amazon.com/ec2/home}{AWS Management Console}, a web-interface, for managing Amazon Web Services resources.}{8}{figure.2.2}\n\\contentsline {paragraph}{}{10}{section*.22}\n\\contentsline {section}{\\numberline {2.3}Connecting to and logging into your CloudBioLinux instance}{10}{section.2.3}\n\\contentsline {subsection}{\\numberline {2.3.1}Graphical, or command line?}{10}{subsection.2.3.1}\n\\contentsline {paragraph}{For those wanting to work in a graphical computing environment}{10}{section*.23}\n\\contentsline {paragraph}{For those comfortable in text-only environments, including Linux users who wish to run graphical programs, without a full desktop,}{10}{section*.24}\n\\contentsline {paragraph}{}{10}{section*.25}\n\\contentsline {subsection}{\\numberline {2.3.2}Find out the address of your instance}{10}{subsection.2.3.2}\n\\contentsline {paragraph}{}{10}{section*.26}\n\\contentsline {subsection}{\\numberline {2.3.3}Logging into a terminal using ssh}{11}{subsection.2.3.3}\n\\contentsline {paragraph}{}{11}{section*.27}\n\\contentsline {paragraph}{ssh -i mykey.pem root@ec2-184-72-144-209.compute-1.amazonaws.com}{11}{section*.28}\n\\contentsline {paragraph}{}{11}{section*.29}\n\\contentsline {paragraph}{ssh -i mykey.pem ubuntu@ec2-184-72-144-209.compute-1.amazonaws.com}{11}{section*.30}\n\\contentsline {paragraph}{}{11}{section*.31}\n\\contentsline {paragraph}{Note:}{12}{section*.32}\n\\contentsline {paragraph}{ssh -i /home/mydirectory/keys/mykey.pem ubuntu@ec2-184-72-144-209.compute-1.amazonaws.com}{12}{section*.33}\n\\contentsline {paragraph}{}{12}{section*.34}\n\\contentsline {paragraph}{}{12}{section*.35}\n\\contentsline {subsection}{\\numberline {2.3.4}Logging into graphical desktop using NX}{12}{subsection.2.3.4}\n\\contentsline {paragraph}{}{12}{section*.36}\n\\contentsline {paragraph}{}{12}{section*.37}\n\\contentsline {paragraph}{}{12}{section*.38}\n\\contentsline {subsection}{\\numberline {2.3.5}The logic of the NX setup}{12}{subsection.2.3.5}\n\\contentsline {paragraph}{}{12}{section*.39}\n\\contentsline {paragraph}{}{13}{section*.40}\n\\contentsline {section}{\\numberline {2.4}Logging out of your CloudBioLinux instance}{13}{section.2.4}\n\\contentsline {paragraph}{From an NX connection}{13}{section*.41}\n\\contentsline {paragraph}{From an ssh command line (or Putty) connection}{13}{section*.42}\n\\contentsline {section}{\\numberline {2.5}Terminating your CloudBioLinux instance}{14}{section.2.5}\n\\contentsline {paragraph}{}{14}{section*.43}\n\\contentsline {paragraph}{}{15}{section*.44}\n\\contentsline {chapter}{\\numberline {3}Working with data on the cloud}{17}{chapter.3}\n\\contentsline {section}{\\numberline {3.1}Introduction}{18}{section.3.1}\n\\contentsline {paragraph}{}{18}{section*.45}\n\\contentsline {paragraph}{}{18}{section*.46}\n\\contentsline {section}{\\numberline {3.2}Copying data onto your CloudBioLinux instance}{18}{section.3.2}\n\\contentsline {paragraph}{}{18}{section*.47}\n\\contentsline {paragraph}{}{18}{section*.48}\n\\contentsline {paragraph}{}{19}{section*.49}\n\\contentsline {paragraph}{}{19}{section*.50}\n\\contentsline {paragraph}{}{19}{section*.51}\n\\contentsline {section}{\\numberline {3.3}Using EBS volumes for data}{20}{section.3.3}\n\\contentsline {paragraph}{}{20}{section*.52}\n\\contentsline {paragraph}{}{20}{section*.53}\n\\contentsline {paragraph}{A note on charging:}{20}{section*.54}\n\\contentsline {paragraph}{Creating your volume}{20}{figure.3.2}\n\\contentsline {paragraph}{}{20}{section*.56}\n\\contentsline {paragraph}{}{20}{section*.57}\n\\contentsline {paragraph}{}{21}{section*.58}\n\\contentsline {paragraph}{}{22}{section*.59}\n\\contentsline {paragraph}{Accessing your volume}{22}{section*.60}\n\\contentsline {paragraph}{}{22}{section*.61}\n\\contentsline {paragraph}{}{23}{section*.62}\n\\contentsline {paragraph}{Unmounting your volume}{24}{figure.3.6}\n\\contentsline {paragraph}{This is a simple but vital step to avoid the possibility of data corruption.}{24}{section*.64}\n\\contentsline {paragraph}{}{24}{section*.65}\n\\contentsline {paragraph}{Detaching your volume}{24}{section*.66}\n\\contentsline {paragraph}{}{24}{section*.67}\n\\contentsline {paragraph}{Backing up or sharing your volume}{25}{section*.68}\n\\contentsline {paragraph}{}{25}{section*.69}\n\\contentsline {paragraph}{Deleting your volume}{25}{section*.70}\n\\contentsline {paragraph}{}{25}{section*.71}\n\\contentsline {section}{\\numberline {3.4}Accesing public datasets on Amazon}{25}{section.3.4}\n\\contentsline {paragraph}{}{25}{section*.72}\n\\contentsline {paragraph}{}{25}{section*.73}\n\\contentsline {paragraph}{}{25}{section*.74}\n\\contentsline {chapter}{\\numberline {A}Appendices}{26}{appendix.A}\n\\contentsline {section}{\\numberline {A.1}The CloudBioLinux Desktop}{27}{section.A.1}\n\\contentsline {paragraph}{}{27}{section*.75}\n\\contentsline {subsection}{\\numberline {A.1.1}Bioinformatics documentation}{27}{subsection.A.1.1}\n\\contentsline {paragraph}{}{27}{section*.76}\n\\contentsline {paragraph}{}{27}{section*.77}\n\\contentsline {subsection}{\\numberline {A.1.2}The bioinformatics menu}{28}{subsection.A.1.2}\n\\contentsline {paragraph}{}{28}{section*.78}\n\\contentsline {subsection}{\\numberline {A.1.3}Opening a terminal}{28}{subsection.A.1.3}\n\\contentsline {paragraph}{}{28}{section*.79}\n\\contentsline {section}{\\numberline {A.2}Basic terminology}{29}{section.A.2}\n\\contentsline {paragraph}{}{29}{section*.80}\n\\contentsline {paragraph}{Amazon Machine Image}{29}{section*.81}\n\\contentsline {paragraph}{Amazon Simple Storage Service}{29}{section*.82}\n\\contentsline {paragraph}{AMI}{29}{section*.83}\n\\contentsline {paragraph}{Availability Regions and Zones}{29}{section*.84}\n\\contentsline {paragraph}{AWS}{29}{section*.85}\n\\contentsline {paragraph}{AWS console}{29}{section*.86}\n\\contentsline {paragraph}{EBS}{29}{section*.87}\n\\contentsline {paragraph}{EC2}{29}{section*.88}\n\\contentsline {paragraph}{Elastic Block Strorage}{29}{section*.89}\n\\contentsline {paragraph}{Elastic Compute Cloud}{30}{section*.90}\n\\contentsline {paragraph}{Image}{30}{section*.91}\n\\contentsline {paragraph}{Instance}{30}{section*.92}\n\\contentsline {paragraph}{NX}{30}{section*.93}\n\\contentsline {paragraph}{Putty}{30}{section*.94}\n\\contentsline {paragraph}{S3}{30}{section*.95}\n\\contentsline {paragraph}{SSH}{30}{section*.96}\n\\contentsline {paragraph}{Snapshot}{30}{section*.97}\n\\contentsline {paragraph}{Volume}{30}{section*.98}\n"
  },
  {
    "path": "doc/intro/tips.tex",
    "content": "\\section{Tips}\n\n\\paragraph{This page contains links and tips that I found useful when starting out.}\n\nThe \\href{https://aws.amazon.com/about-aws/events/}{AWS Events} page contains a listing of things like webinars, conferences and user groups - both online and real world.\n"
  },
  {
    "path": "doc/intro/usefulLinks.tex",
    "content": "\\section{Useful Links}\n\nThe \\href{http://docs.amazonwebservices.com}{main Amazon documentation webpage} contains links to oodles of documentation for the Amazon web services.\n\n\\href{http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/}{Amazon User Guide}\n\n\\href{http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/}{EC2 Userguide}. \n\nThe \\href{https://aws.amazon.com/about-aws/events/}{AWS Events} page contains a listing of things like webinars, conferences and user groups - both online and real world.\n"
  },
  {
    "path": "doc/intro/workingOnCloudBL.aux",
    "content": "\\relax \n\\@writefile{toc}{\\contentsline {section}{\\numberline {2.1}The process in a nutshell}{7}{section.2.1}}\n\\@writefile{lof}{\\contentsline {figure}{\\numberline {2.1}{\\ignorespaces Start an Instance}}{7}{figure.2.1}}\n\\newlabel{fig:nutshell}{{2.1}{7}{Start an Instance\\relax }{figure.2.1}{}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{7}{section*.16}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{8}{section*.17}}\n\\@writefile{toc}{\\contentsline {paragraph}{A note about charging:}{8}{section*.18}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{8}{section*.19}}\n\\@writefile{toc}{\\contentsline {paragraph}{A couple of things to note when starting out:}{8}{section*.20}}\n\\@writefile{toc}{\\contentsline {section}{\\numberline {2.2}Starting up a CloudBioLinux instance}{8}{section.2.2}}\n\\@writefile{toc}{\\contentsline {paragraph}{This document focusses on using the \\href  {http://console.aws.amazon.com/ec2/home}{AWS Management Console}, a web-interface, for managing Amazon Web Services resources.}{8}{figure.2.2}}\n\\@writefile{lof}{\\contentsline {figure}{\\numberline {2.2}{\\ignorespaces Start an Instance}}{9}{figure.2.2}}\n\\newlabel{fig:requestInstance}{{2.2}{9}{Start an Instance\\relax }{figure.2.2}{}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{10}{section*.22}}\n\\@writefile{toc}{\\contentsline {section}{\\numberline {2.3}Connecting to and logging into your CloudBioLinux instance}{10}{section.2.3}}\n\\@writefile{toc}{\\contentsline {subsection}{\\numberline {2.3.1}Graphical, or command line?}{10}{subsection.2.3.1}}\n\\@writefile{toc}{\\contentsline {paragraph}{For those wanting to work in a graphical computing environment}{10}{section*.23}}\n\\@writefile{toc}{\\contentsline {paragraph}{For those comfortable in text-only environments, including Linux users who wish to run graphical programs, without a full desktop,}{10}{section*.24}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{10}{section*.25}}\n\\@writefile{toc}{\\contentsline {subsection}{\\numberline {2.3.2}Find out the address of your instance}{10}{subsection.2.3.2}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{10}{section*.26}}\n\\@writefile{lof}{\\contentsline {figure}{\\numberline {2.3}{\\ignorespaces Start an Instance}}{11}{figure.2.3}}\n\\newlabel{fig:instancesOptions}{{2.3}{11}{Start an Instance\\relax }{figure.2.3}{}}\n\\@writefile{toc}{\\contentsline {subsection}{\\numberline {2.3.3}Logging into a terminal using ssh}{11}{subsection.2.3.3}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{11}{section*.27}}\n\\@writefile{toc}{\\contentsline {paragraph}{ssh -i mykey.pem root@ec2-184-72-144-209.compute-1.amazonaws.com}{11}{section*.28}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{11}{section*.29}}\n\\@writefile{toc}{\\contentsline {paragraph}{ssh -i mykey.pem ubuntu@ec2-184-72-144-209.compute-1.amazonaws.com}{11}{section*.30}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{11}{section*.31}}\n\\@writefile{toc}{\\contentsline {paragraph}{Note:}{12}{section*.32}}\n\\@writefile{toc}{\\contentsline {paragraph}{ssh -i /home/mydirectory/keys/mykey.pem ubuntu@ec2-184-72-144-209.compute-1.amazonaws.com}{12}{section*.33}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{12}{section*.34}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{12}{section*.35}}\n\\newlabel{section:nx}{{2.3.3}{12}{\\relax }{section*.35}{}}\n\\@writefile{toc}{\\contentsline {subsection}{\\numberline {2.3.4}Logging into graphical desktop using NX}{12}{subsection.2.3.4}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{12}{section*.36}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{12}{section*.37}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{12}{section*.38}}\n\\@writefile{toc}{\\contentsline {subsection}{\\numberline {2.3.5}The logic of the NX setup}{12}{subsection.2.3.5}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{12}{section*.39}}\n\\@writefile{lof}{\\contentsline {figure}{\\numberline {2.4}{\\ignorespaces NX Client start}}{13}{figure.2.4}}\n\\newlabel{fig:NX-menuOnUbuntu}{{2.4}{13}{NX Client start\\relax }{figure.2.4}{}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{13}{section*.40}}\n\\@writefile{toc}{\\contentsline {section}{\\numberline {2.4}Logging out of your CloudBioLinux instance}{13}{section.2.4}}\n\\@writefile{toc}{\\contentsline {paragraph}{From an NX connection}{13}{section*.41}}\n\\@writefile{toc}{\\contentsline {paragraph}{From an ssh command line (or Putty) connection}{13}{section*.42}}\n\\@writefile{lof}{\\contentsline {figure}{\\numberline {2.5}{\\ignorespaces NX Client start}}{14}{figure.2.5}}\n\\newlabel{fig:NXConnectionWizard}{{2.5}{14}{NX Client start\\relax }{figure.2.5}{}}\n\\@writefile{toc}{\\contentsline {section}{\\numberline {2.5}Terminating your CloudBioLinux instance}{14}{section.2.5}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{14}{section*.43}}\n\\@writefile{lof}{\\contentsline {figure}{\\numberline {2.6}{\\ignorespaces CloudBL desktop}}{15}{figure.2.6}}\n\\newlabel{fig:bldesktop}{{2.6}{15}{CloudBL desktop\\relax }{figure.2.6}{}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{15}{section*.44}}\n\\newlabel{section:cloudblDesktop}{{2.5}{15}{\\relax }{section*.44}{}}\n\\newlabel{SC@3}{{2.5}{16}{\\relax }{section*.44}{}}\n\\@writefile{lof}{\\contentsline {figure}{\\numberline {2.7}{\\ignorespaces Logging out of NX}}{16}{figure.2.7}}\n\\newlabel{fig:nxshutdown}{{2.7}{16}{\\SC@OPTCAPtext \\relax }{figure.2.7}{}}\n\\@setckpt{workingOnCloudBL}{\n\\setcounter{page}{17}\n\\setcounter{equation}{0}\n\\setcounter{enumi}{11}\n\\setcounter{enumii}{0}\n\\setcounter{enumiii}{0}\n\\setcounter{enumiv}{0}\n\\setcounter{footnote}{5}\n\\setcounter{mpfootnote}{0}\n\\setcounter{part}{0}\n\\setcounter{chapter}{2}\n\\setcounter{section}{5}\n\\setcounter{subsection}{0}\n\\setcounter{subsubsection}{0}\n\\setcounter{paragraph}{0}\n\\setcounter{subparagraph}{0}\n\\setcounter{figure}{7}\n\\setcounter{table}{0}\n\\setcounter{SC@C}{3}\n\\setcounter{Item}{25}\n\\setcounter{Hfootnote}{7}\n\\setcounter{section@level}{4}\n}\n"
  },
  {
    "path": "doc/intro/workingOnCloudBL.tex",
    "content": "\\section{The process in a nutshell}\n\n\\begin{figure}[!hd]\n\t\\fbox\n\t{\n\t\t\\begin{minipage}{13cm}\n%the maxwidth variable is defined in the summary page: gettingStarted_CloudBioLinux.tex\n\\includegraphics[width=\\maxwidth]{\"images/nutshell\"}\n\\caption[Start an Instance]{\\label{fig:nutshell}Schematic of the basic process of working on CloudBioLinux on Amazon EC2. You can log in and out of your instance as often as you like; you continue to be charged whether you are logged in or not. You can also stop and re-start instances (not shown here). When you are finished with an instance, you should terminate it. Once terminated, the system and all data on it are deleted. This is the simplest setup. There are additional easy steps you can take to store data and even whole systems, at a fraction of the price of a running image.}\n\t\t\\end{minipage}\n\t}\n\\end{figure}\n\n\\paragraph{}The general process you will follow when working with CloudBioLinux is outlined in figure~\\ref{fig:nutshell}:\n\\begin{enumerate}\n\\item \\textbf{Start up} a CloudBioLinux instance\n\\item \\textbf{Log into} your CloudBioLinux instance\n\\item \\textbf{Log out of} the  CloudBioLinux instance\n\\item Still want to work on this instance? You can log into it and out of it as often as you like, or you can stop and start the instance, which can work out slightly cheaper.\n\\item When you're really finished, and don't need the CloudBioLinux instance anymore, \\textbf{terminate} the instance. You will stop being charged for this instance when it is terminated.\\footnote{\\href{http://support.rightscale.com/06-FAQs/FAQ\\_0149\\_-_What\\%27s_the_difference_between_Terminating_and_Stopping_an_EC2_Instance\\%3F}{Stopping and terminating are different.}}\n\\end{enumerate}\n\n\\paragraph{}This chapter focusses on starting and logging into a full graphical Bio-Linux desktop on the cloud. Of course, there are other things that you may wish to do, like save your system and data for use again later, or share it with others. These things and more are covered in the {http://aws.amazon.com/ebs/}{official documentation for Elastic Block Storage}.\n%It would be good to write a basic intro, as the page linked here is hardly an easy newbie read. However, this is \n%lower priority than getting the rest of this guide done.\n\n\\paragraph{A note about charging:} \n\\paragraph{}The charging structure for Amazon EC2 is well defined and quite detailed. It is important to understand what you are being charged for, so you can make good decisions about when using the cloud is a cost effective option, and when it is not. You will be charged for running instances, and also for things like bandwidth when transferring data on and off Amazon systems, and data volumes you wish to use later. Please read the \\href{http://aws.amazon.com/ec2/pricing/}{Amazon pricing documentation} so you don't get surprised when you next see your credit card bill. \n\\paragraph{A couple of things to note when starting out:}\n\\begin{itemize}\n\\item \\textbf{You will be charged for the time your instance is running.} It's not about when you're logged into it that counts. Charging for the instance terminates when you terminate the instance. You can just transfer your files off the system onto your local machine - but be aware that you may be charged for the bandwidth you use. Alternatively, you could consider using \\href{http://aws.amazon.com/ebs/}{Elastic Block Storage}. \n\\item \\textbf{You are charged by the time-hour.} This means that if you start up an instance at 1:55pm and use it until 2:05pm, you are charged for two hours - because your instance was running in two different hours of the clock. \n\n\\end{itemize}\n\n\\section{Starting up a CloudBioLinux instance}\n\\paragraph{This document focusses on using the \\href{http://console.aws.amazon.com/ec2/home}{AWS Management Console}, a web-interface, for managing Amazon Web Services resources.}\n\n\\begin{figure}[!hd]\t\n\t\\fbox\n\t{\n\t\t\\begin{minipage}{13cm}\n%the maxwidth variable is defined in the summary page: gettingStarted_CloudBioLinux.tex\n\\includegraphics[width=\\maxwidth]{\"images/requestInstance\"}\n\\caption[Start an Instance]{\\label{fig:requestInstance}Search for the term \"biolinux\" in the Community instances. You are likely to find a number of different images available. Those listed here were available on July 20, 2010. We recommend you pick the one with the system you want (e.g. 32 bit or 64 bit) that has \\textbf{the most recent date in the text within the Manifest column}.}\n\t\t\\end{minipage}\n\t}\n\\end{figure}\n\n%\\begin{figure}[!hd]\n%%the maxwidth variable is defined in the summary page: gettingStarted_CloudBioLinux.tex\n%\\includegraphics[width=\\maxwidth]{\"images/openssh\"}\n%\\caption[Start an Instance]{\\label{fig:openssh}XXX text here}\n%\\end{figure}\n\n\\begin{enumerate}\n\\item Go to the \\href{http://console.aws.amazon.com/ec2/home}{EC2 Management Console URL: http://console.aws.amazon.com/ec2/home}\n\\item You should see a button saying \\textbf{Launch instance}. Click on this. \n\\item You are presented with a window called \\textbf{Request Instances Wizard}. \n\\item To start up CloudBioLinux, go to the \\textbf{Community AMIs tab} and search All Images for the term \\textbf{biolinux}. This will bring up a list of available CloudBioLinux images. See figure~\\ref{fig:requestInstance}.\n\\item Click on the image you wish to run to highlight it. Then click on the Select button on the right hand side. \n\\item Now click on the \\textbf{Launch Instances} in the next window presented to you.\n\\item Click on the Continue Button at the bottom of the next window. \n\\item Leave the Advanced options on the next page as they are. \\emph{Note that this would be the time to alter these settings if you wanted to do so; you cannot change them in a running instance.}\n%The above item needs editing once the NX version is available, as you will likely have to provide \n%user data to get that to work.\n\\item In the next window, you'll need to provide the name of your Key Pair. \\emph{If you created a key pair earlier, but are not offered the option of using it, and if you created your keys in the same session you are currently logged into, try logging out of Amazon and logging back in again.}\n\\item Once you have provided a key pair name, the next window will ask about your preferred security settings. This is analogous to setting up a machine firewall. \\textbf{At a minimum you will need to enable ssh access;} ssh is the protocol you need to use to connect to your instance, whether you do so via the command line or via a graphical NX connection. If you want to access web pages provided by your instance, then you also need to open a port for http. You will want to do this if, say, you wish to refer to the Bio-Linux documentation pages on your instance. If you will be running MySQL or postgreSQL for example, you'll need to enable access to these also.\n\\item Once you've done all this, you should be able to review the information you've provided, and if you're happy click on the \\textbf{Launch instance} button.\n\\end{enumerate}\n\n\\paragraph{}If you go back to your \\href{http://console.aws.amazon.com/ec2/home}{Amazon EC2 home area} and click on the Instances link in the left hand pane, you should see your CloudBioLinux instance starting up. When you see a green icon with the word running beside it, your instance is ready to log into.\n\n\\section{Connecting to and logging into your CloudBioLinux instance}\n\n\\subsection{Graphical, or command line?}\n\n\\paragraph{For those wanting to work in a graphical computing environment}, as opposed to working from the command line, we recommend that you set up an NX connection. This provides you with a full graphical CloudBioLinux desktop. For a given instance that you have launched, you must go through the steps in the following two sections once. After that, you will be able to connect to a graphical desktop session for your launched instance as often as you like.\n\n\\paragraph{For those comfortable in text-only environments, including Linux users who wish to run graphical programs, without a full desktop,} you need only follow the instructions in the next section one time. Then using the instructions in the following section, you can log into a terminal using ssh as often as you like. \n\n\\paragraph{}For Windows users who wish to have access to graphical programs, it is easiest to run an NX connection.  \n\n\\subsection{Find out the address of your instance}\n\n\\paragraph{}You need to know the address that's been assigned to your image, so that you can tell ssh or NX which machine you are trying to connect to.\n\n\\begin{figure}[!hd]\n\t\\fbox\n\t{\n\t\t\\begin{minipage}{13cm}\n\\includegraphics[width=\\maxwidth]{\"images/instancesOptions\"}\n\\caption[Start an Instance]{\\label{fig:instancesOptions}Click the Instance Actions button (pink circle) to bring up a menu with options including connecting to an instance you have already started up. This menu is also used to terminate a running instance.}\n\t\t\\end{minipage}\n\t}\n\\end{figure}\n\n\n\\begin{itemize}\n\\item Assuming you have already clicked on the Instances link on the left side of your \\href{https://console.aws.amazon.com/ec2/home}{EC2 Dashboard}, click on the \\textbf{Instance Actions} button near the top of the Instances page. See figure~\\ref{fig:instancesOptions}.\n\\item Choose \\textbf{Connect}. A window will open containing directions about how to connect to your CloudBioLinux instance using ssh. \\emph{You need to make a couple of changes to the suggested connection instructions}, described below.\n\\end{itemize}\n\n\n\\subsection{Logging into a terminal using ssh}\n\n\\paragraph{}After you clicked on the Connect option to connect to your instance, you should have seen a small window pop up. The instructions in that window should have text similar to the following in it: \n\n\\paragraph{ssh -i mykey.pem root@ec2-184-72-144-209.compute-1.amazonaws.com}\n\n\\paragraph{}The text after the @ symbol is the address of your running instance. If you are working on Linux, or you have an ssh program on Windows with a terminal, the information in the window is \\emph{similar to} the command you could use to connect to your instance \\footnote{If you are logging in using Putty on Windows, you will need to \\href{http://www.ualberta.ca/CNS/RESEARCH/LinuxClusters/pka-putty.html}{enter the relevant information into the Putty system} in order to connect.}. CloudBioLinux is based on Ubuntu. \\textbf{To log into the instance, you need to use the \\emph{ubuntu} user}, not the root user\\footnote{The default for most systems on Amazon EC2 is to log in as the root user.}. So, an example command you might run in a terminal to connect to your instance is:\n\n\\paragraph{ssh -i mykey.pem ubuntu@ec2-184-72-144-209.compute-1.amazonaws.com}\n\n\\paragraph{}where you have used the ubuntu username (instead of root), and you include your machine address after the @ symbol.\n\n\\paragraph{Note:} If you get a warning when you try to connect that suggests that your key cannot be found, it may mean that you have saved your key to a non-standard location and/or given it a non-standard name. In this case add \\href{http://nebc.nerc.ac.uk/tools/bio-linux/bio-linux-faq\\#path}{path} information for your key to the command line so that the private key can be found from where you run the ssh connection command. For example, if your key is stored in a subdirectory of your home directory called \\emph{keys}, and you want to log in as the \\emph{ubuntu} user, you could log in using ssh and the command, you need to\n\n\\paragraph{ssh -i /home/mydirectory/keys/mykey.pem ubuntu@ec2-184-72-144-209.compute-1.amazonaws.com} \n\n\\paragraph{}The first time you connect to your running CloudBioLinux instance, you should be offered the opportunity to set up NX on your instance. At this point, you can also provide a username other than \"ubuntu\", and this new user will be created for you. \n\n\\paragraph{}You can continue at this point to run programs on the command line, or if you are working on a Linux system, you can launch graphical applications using the command line. However, we generally recommend connecting to your CloudBioLinux instance using an NX client instead of a text-based ssh client. This is because many people find a graphical desktop environment easier to work on, and the menus and desktop links help people take full advantage of the facilities of the CloudBioLinux system. To log in via NX, just follow the instructions in the next section. You may wish to log out of your current ssh session (although this is not necessary).\n\n\\label{section:nx}\n\\subsection{Logging into graphical desktop using NX}\n\n\\paragraph{}Start up your NX client software\\footnote{Images shown here refer to the Nomachine NX client for Linux, running on a standard NEBC Bio-Linux machine, but the process should be similar no matter what type of system you are working on (Linux, Windows, Mac).}\n\n\\begin{figure}[!hd]\n\t\\fbox\n\t{\n\t\t\\begin{minipage}{13cm}\n\\includegraphics[width=\\maxwidth]{\"images/NX-menuOnUbuntu\"}\n\\caption[NX Client start]{\\label{fig:NX-menuOnUbuntu}If an NX client is already installed, you should be able launch it from under your Applications menu on Ubuntu-based machines, or from your programs listing on Windows machines.}\n\t\t\\end{minipage}\n\t}\n\\end{figure}\n\n\\paragraph{}If you are using a Nomachine NX client, you should now see an NX connection wizard. Here, you need to enter the address of your launched instance (the same address you used to log into the terminal earlier), and you should change the desktop type to Gnome. See figure \\ref{fig:NXConnectionWizard}. \n\n\\begin{figure}[!hd]\n\t\\fbox\n\t{\n\t\t\\begin{minipage}{13cm}\n\\includegraphics[width=\\maxwidth]{\"images/NXConnectionWizardandGnome\"}\n\\caption[NX Client start]{\\label{fig:NXConnectionWizard}Enter the same machine address you used for your terminal login earlier. Remember to use the same username you provided earlier - this may be the default username, \"ubuntu\", or it may be something else that you chose when you set the NX password. Please choose the gnome desktop type.}\n\t\t\\end{minipage}\n\t}\n\\end{figure}\n\n\\paragraph{}When you've logged in, you should see a desktop similar to that in figure \\ref{fig:bldesktop}.\n\n\\subsection{The logic of the NX setup}\n\n\\paragraph{}It may initially seem strange that you need to log into a command terminal before you can log into your EC2 instance using an NX client. The reason that this needs to be done is that key auth support, which is used for the ssh connections into the EC2 instances, is not supported by NX. Thus, for any given instance that is started, \\emph{you need a password to be able to log in using NX}. \n\n\\paragraph{}When you set a password during your terminal session, you are ensuring that only you have that password. You also have the opportunity to create another user on the system, just by providing a username other than \"ubuntu\" when prompted.\n\n\\section{Logging out of your CloudBioLinux instance}\n\n\\paragraph{From an NX connection} you need to go to the System menu and choose the option \\textbf{Shut Down...}. See figure~\\ref{fig:nxshutdown}.\n\n\\paragraph{From an ssh command line (or Putty) connection} you need to type \\textbf{exit} at the command prompt.\n\n\\section{Terminating your CloudBioLinux instance}\n\n\\paragraph{}Highlight the instance you wish to terminate in the list on your Instances page. Click on the \\textbf{Instance Actions} button (see figure~\\ref{fig:instancesOptions}) and choose \\textbf{Terminate} under the Instance Action section of the menu. In basic terms, terminating results in the system and all the files and data on it being deleted \\footnote{You will still be charged a fee if you have only stopped your instance, as opposed to terminating it, and \n\\href{http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/index.html?Concepts\\_BootFromEBS.html\\#Stop\\_Start} {your data may still be deleted depending on how you have set things up}. \\href{http://support.rightscale.com/06-FAQs/FAQ\\_0149\\_-_What\\%27s\\_the\\_difference\\_between\\_Terminating\\_and\\_Stopping\\_an\\_EC2\\_Instance\\%3F}{Stopping and terminating are different.}}. If you have work you wish to save before terminating, or if you wish to keep a copy of this image so that you can use it later, without paying as much as you would for a running instance, please check out the Amazon documentation on EBS Volumes and taking snapshots of instances.\n\n\n\\begin{figure}[!hd]\n\t\\fbox\n\t{\n\t\t\\begin{minipage}{13cm}\n\\includegraphics[width=\\maxwidth]{\"images/initalNXdesktop\"}\n\\caption[CloudBL desktop]{\\label{fig:bldesktop}The CloudBioLinux desktop. It includes a number of menus, as well as icons for folders containing sample data relevant for the bioinformatics software installed, and a link to this document.}\n\t\t\\end{minipage}\n\t}\n\\end{figure}\n\n\\paragraph{}We provide a short introduction to the Cloud Bio-Linux desktop in Appendix A.2 on page \\label{section:cloudblDesktop}. \n\n\\begin{SCfigure}[][t]\n\\includegraphics[width=40mm]{\"images/nxshutdown\"}\n\\caption[Logging out of NX]{\\label{fig:nxshutdown}Choosing the \\textbf{Shut Down...} option in an NX session logs you out. Logging out is not the same as terminating your CloudBioLinux instance. You will still be charged while the instance is running - whether you are logged into it or not.}\n\\end{SCfigure}\n\n"
  },
  {
    "path": "doc/intro/workingWithData.aux",
    "content": "\\relax \n\\@writefile{toc}{\\contentsline {section}{\\numberline {3.1}Introduction}{18}{section.3.1}}\n\\newlabel{section:data}{{3.1}{18}{Introduction\\relax }{section.3.1}{}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{18}{section*.45}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{18}{section*.46}}\n\\@writefile{toc}{\\contentsline {section}{\\numberline {3.2}Copying data onto your CloudBioLinux instance}{18}{section.3.2}}\n\\newlabel{section:directcopy}{{3.2}{18}{Copying data onto your CloudBioLinux instance\\relax }{section.3.2}{}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{18}{section*.47}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{18}{section*.48}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{19}{section*.49}}\n\\@writefile{lof}{\\contentsline {figure}{\\numberline {3.1}{\\ignorespaces Drag and drop from remote machine}}{19}{figure.3.1}}\n\\newlabel{fig:graphicalftp}{{3.1}{19}{Drag and drop from remote machine\\relax }{figure.3.1}{}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{19}{section*.50}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{19}{section*.51}}\n\\@writefile{toc}{\\contentsline {section}{\\numberline {3.3}Using EBS volumes for data}{20}{section.3.3}}\n\\newlabel{section:ebscopy}{{3.3}{20}{Using EBS volumes for data\\relax }{section.3.3}{}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{20}{section*.52}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{20}{section*.53}}\n\\@writefile{toc}{\\contentsline {paragraph}{A note on charging:}{20}{section*.54}}\n\\@writefile{toc}{\\contentsline {paragraph}{Creating your volume}{20}{figure.3.2}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{20}{section*.56}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{20}{section*.57}}\n\\@writefile{lof}{\\contentsline {figure}{\\numberline {3.2}{\\ignorespaces Attaching a volume}}{21}{figure.3.2}}\n\\newlabel{fig:createandmountvolume}{{3.2}{21}{Attaching a volume\\relax }{figure.3.2}{}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{21}{section*.58}}\n\\@writefile{lof}{\\contentsline {figure}{\\numberline {3.3}{\\ignorespaces Create volume in console}}{22}{figure.3.3}}\n\\newlabel{fig:createvolume}{{3.3}{22}{Create volume in console\\relax }{figure.3.3}{}}\n\\@writefile{lof}{\\contentsline {figure}{\\numberline {3.4}{\\ignorespaces Attaching a volume}}{22}{figure.3.4}}\n\\newlabel{fig:attachvolume1}{{3.4}{22}{Attaching a volume\\relax }{figure.3.4}{}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{22}{section*.59}}\n\\@writefile{toc}{\\contentsline {paragraph}{Accessing your volume}{22}{section*.60}}\n\\newlabel{text:mounting}{{3.3}{22}{\\relax }{section*.61}{}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{22}{section*.61}}\n\\@writefile{lof}{\\contentsline {figure}{\\numberline {3.5}{\\ignorespaces Attaching a volume}}{23}{figure.3.5}}\n\\newlabel{fig:attachvolume2}{{3.5}{23}{Attaching a volume\\relax }{figure.3.5}{}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{23}{section*.62}}\n\\@writefile{lof}{\\contentsline {figure}{\\numberline {3.6}{\\ignorespaces Attaching a volume}}{24}{figure.3.6}}\n\\newlabel{fig:unmountdetach}{{3.6}{24}{Attaching a volume\\relax }{figure.3.6}{}}\n\\@writefile{toc}{\\contentsline {paragraph}{Unmounting your volume}{24}{figure.3.6}}\n\\@writefile{toc}{\\contentsline {paragraph}{This is a simple but vital step to avoid the possibility of data corruption.}{24}{section*.64}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{24}{section*.65}}\n\\newlabel{text:unmounting}{{3.3}{24}{\\relax }{section*.65}{}}\n\\@writefile{toc}{\\contentsline {paragraph}{Detaching your volume}{24}{section*.66}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{24}{section*.67}}\n\\@writefile{toc}{\\contentsline {paragraph}{Backing up or sharing your volume}{25}{section*.68}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{25}{section*.69}}\n\\@writefile{toc}{\\contentsline {paragraph}{Deleting your volume}{25}{section*.70}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{25}{section*.71}}\n\\@writefile{toc}{\\contentsline {section}{\\numberline {3.4}Accesing public datasets on Amazon}{25}{section.3.4}}\n\\newlabel{section:publiccopy}{{3.4}{25}{Accesing public datasets on Amazon\\relax }{section.3.4}{}}\n\\@writefile{lof}{\\contentsline {figure}{\\numberline {3.7}{\\ignorespaces List public data}}{25}{figure.3.7}}\n\\newlabel{fig:ensembl}{{3.7}{25}{List public data\\relax }{figure.3.7}{}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{25}{section*.72}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{25}{section*.73}}\n\\@writefile{toc}{\\contentsline {paragraph}{}{25}{section*.74}}\n\\@setckpt{workingWithData}{\n\\setcounter{page}{26}\n\\setcounter{equation}{0}\n\\setcounter{enumi}{4}\n\\setcounter{enumii}{0}\n\\setcounter{enumiii}{0}\n\\setcounter{enumiv}{0}\n\\setcounter{footnote}{2}\n\\setcounter{mpfootnote}{0}\n\\setcounter{part}{0}\n\\setcounter{chapter}{3}\n\\setcounter{section}{4}\n\\setcounter{subsection}{0}\n\\setcounter{subsubsection}{0}\n\\setcounter{paragraph}{0}\n\\setcounter{subparagraph}{0}\n\\setcounter{figure}{7}\n\\setcounter{table}{0}\n\\setcounter{SC@C}{3}\n\\setcounter{Item}{32}\n\\setcounter{Hfootnote}{9}\n\\setcounter{section@level}{4}\n}\n"
  },
  {
    "path": "doc/intro/workingWithData.tex",
    "content": "\\section{Introduction}\\label{section:data}\n\n\\paragraph{}For many bioinformatics tasks, you will want to work on your own data and files – for example, perhaps your own sequence data and blast databases. To do this, you will need to \\emph{upload your files onto a machine that your CloudBioLinux instance can access}. Three options are covered in this chapter:\n\\begin{enumerate}\n\\item Copy your data directly onto the CloudBioLinux instance you are running. See section \\ref{section:directcopy} This would be alright if you were going to use this data only on this running instance and you're happy for it to be deleted when you terminate the instance.\n\\item Copy your date onto a separate EBS volume. This would be useful if you wish to store your files for use in other sessions, but you do not plan to keep the same running instance. (EBS Volumes are cheaper than running instances.) See section \\ref{section:ebscopy}\n\\item If the data you want to use is already available on Amazon EBS volumes (for example, ENSEMBL data), you can access this easily, with no data transfer costs.  \\ref{section:publiccopy}. \n\\end{enumerate}\n\n\\paragraph{}\\href{http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/creating-snapshot-ebs.html}{Snapshots} are not discussed in this document, but are an important topic to understand if you plan to use the Amazon Cloud, especially if you will be working with your own data over a prolonged period or making customisations you do not wish to lose. \n\n\\section{Copying data onto your CloudBioLinux instance}\n\\label{section:directcopy}\n\\paragraph{}If you only need your data for a single CloudBioLinux instance, then you can just copy your data onto that instance directly. Once you are logged into your CloudBioLinux instance, there are a number of ways to do this. For example, there are command line tools like \\textbf{scp}, for copying files from a machine you have an account on, or \\textbf{wget} to bring in data from public websites or ftp sites. \n\\paragraph{}Alternatively, if you are logged into the full graphical desktop using NX (information on page \\pageref{section:nx}), you can use the file browser to connect to a remote site and \\textbf{drag and drop} your files to your running Bio-Linux instance. This is the method we focus on here.\n\\begin{itemize}\n\n\\item Go to the \\textbf{Places} menu in the top taskbar and open up a file browser, for example by clicking on your Home Folder. \n\\item Now go to the \\textbf{Go} menu and click on Location... (or just type Ctrl-L). \n\\item If you are going to copy files from a machine that you have login permissions on, then in the box next to the word \\emph{Location} that appears in your file browser, type:\n\\textbf{ssh://your.machine.com}, replacing your.machine.com with the address of the machine your files are on. Alternatively, if you wanted to copy files from a public ftp server, say, then you would enter something like the following in the Location box: \n\\\\\\textbf{ftp://ftp.someother.database.site}\n\\end{itemize}\n\n\\paragraph{}As a specific example, if I want to copy fasta files from the EMBL database sections, I would type the following into the Location box: \n\\\\\\textbf{ftp://ftp.ebi.ac.uk/pub/databases/fastafiles/emblrelease}\n\n\\begin{figure}[!hd]\n\t\\fbox\n\t{\n\t\t\\begin{minipage}{13cm}\n\\includegraphics[width=\\maxwidth]{\"images/graphicalFTP_full\"}\n\\caption[Drag and drop from remote machine]{\\label{fig:graphicalftp}Copying files from remote machine is easy using the graphical File Browser, which can be launched from under the \\textbf{Places} menu in the top taskbar. Choosing the Locations option under the Go menu of the file browser will allow you to type in a protocol (e.g. ftp, ssh) and a location. Here, a section of EMBL from the EBI is copied to my system using drag and drop between the two file browser windows.}\n\t\t\\end{minipage}\n\t}\n\\end{figure}\n\n\\paragraph{}Now open another file browser by going to the \\textbf{Places} menu. Navigate to the folder you wish to store the files in. Now you can just drag and drop your files from the remote machine onto your CloudBioLinux instance. See figure \\ref{fig:graphicalftp} on page \\pageref{fig:graphicalftp}.\n\n\n\n\n\\paragraph{}This process is simple, and for one-off jobs, is perfectly adequate. Note that you will generally pay for the network traffic you generate in transferring the data\\footnote{Until November 1, 2010, data transfer onto Amazon is free. The first Gb per month of transfer off is also free. (Information taken from the \\href{http://aws.amazon.com/ec2/pricing/}{Amazon website} on July 21, 2010, with no guarantees to be correct at the time you are reading this document. Please check the \\href{http://aws.amazon.com/ec2/}{official pricing list}.} So if you are going to use the same dataset numerous times, it is worth considering setting up an EBS volume rather than transferring data onto new instances. Even if this transfer is free, it will still generally take more time than mounting an EBS volume that already has your data on it.\n\n\\section{Using EBS volumes for data}\n\\label{section:ebscopy}\n\\paragraph{}An Amazon EBS volume is what you need if \n\\begin{itemize}\n\\item you are going to use a dataset a number of times, with gaps in time between uses, or\n\\item you want to store your data such that you can connect to it from different CloudBioLinux (or other Amazon EC2) images, or\n\\item if you wish to share your data with other people also working on Amazon EC2 systems.\n\\end{itemize}\n\n\\paragraph{}This guide presents only a small part of what is possible with EBS volumes. Please check out the \\href{http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/ebs-api-overview.html}{EBS volume documentation on the Amazon website} for further information.\n\n\\paragraph{A note on charging:}You will be \\href{http://aws.amazon.com/ec2/}{charged for your Amazon EBS volume} as long as it is in existence, and you will be \\emph{charged for the space you request}, not the space you are really using. So if you ask for 1Gb, you are paying for 1Gb, even if you only use 100Kb. \n\n\\paragraph{Creating your volume}\n\n\\begin{figure}[!ht]\n%\t\\centering\n\t\\fbox\n\t{\n\t\t\\begin{minipage}{13cm}\n\t\t\t\\includegraphics[width=\\maxwidth]{\"images/createAndMountVol-1\"}\n\t\t\t\\caption[Attaching a volume]{\\label{fig:createandmountvolume}To create a new volume and get access to it from your running instance involves 4 steps. Here we suggest that the first two are done using the AWS Console. The second two steps are done within your running instance. If you want to work with a volume you (or someone else) has created earlier, then you need only carry out the steps in the grey boxes. The commands for carrying out the two steps in your running instance are provided in the text on page \\pageref{text:mounting}}\n\t\t\\end{minipage}\n\t}\n\\end{figure}\n\n\n\\paragraph{}The instructions in this section assume you have already started up and have logged into an EC2 instance. \n\\paragraph{}The steps involved in creating and mounting a volume are illustrated in figure~\\ref{fig:createandmountvolume}.\n\n\\begin{figure}[!hd]\n\t\\fbox\n\t{\n\t\t\\begin{minipage}{13cm}\n\\includegraphics[width=\\maxwidth]{\"images/createVolume-1\"}\n\\caption[Create volume in console]{\\label{fig:createvolume}The volumes pane of the AWS Console is brought up by clicking on the Volumes link in the Navigation pane (Green oval).  Red circle: Just click the Create Volume button - it does what it says. Purple circle: the status of your volume creation is reported to you. When the circle turns blue and the word says \\emph{available}, you can proceed to attach and mount your volume.}\n\t\t\\end{minipage}\n\t}\n\\end{figure}\n\n\\begin{itemize}\n\\item In the Navigation pane (left side) of the AWS Management Console, go to the Elastic Block Store area and choose \\textbf{Volumes}. See figure~\\ref{fig:createvolume}.\n\\item Click on the \\textbf{Create Volume} button.\n\\item Choose the same availability zone as the images you plan to use this volume with.\n\\item After changing any other settings in this window, press the \\textbf{Create} button.\n\\item Wait until the yellow circle beside the word \\emph{creating} is replaced by a blue circle beside the word \\emph{available}.\n\\end{itemize}\n\n\\paragraph{}You now have an EBS volume attached to your running instance. This is analagous to plugging in a new piece of hardware to a server. This means that the volume \\textbf{is not yet ready for copying data onto}. You first must mount the volume on your instance - this makes the volume accessible to you when you are logged into your instance \\footnote{For those used to user-friendly plug 'n play systems, where you just plug a pre-formatted hard drive into a machine and copy to it directly, the steps outlined here might come as a bit of long-winded shock. In fact, all you are doing is carrying out the steps that, between your hardware vendor and your computer operating system, are often done for you. In particular, the equivalent to a disk drive here is the EBS volume, which is initally unformatted and needs to be formatted. To do this, you attach it to your running instance, then you format it. Now, to work with the filesystem on your volume, you need to mount in onto your image file system, so the whole thing - image and volume, can be interacted with as one. Until you unmount the volume of course.}. \n\n\\begin{figure}[!hd]\n\t\\fbox\n\t{\n\t\t\\begin{minipage}{13cm}\n\\includegraphics[width=\\maxwidth]{\"images/attachVolume-1\"}\n\\caption[Attaching a volume]{\\label{fig:attachvolume1}Click on the Attach Volume button (Orange circle) to Attach a volume to an instance.}\n\t\t\\end{minipage}\n\t}\t\n\\end{figure}\n\n\n\\begin{figure}[!hd]\n\t\\fbox\n\t{\n\t\t\\begin{minipage}{13cm}\n\\includegraphics[width=\\maxwidth]{\"images/attachVolume-2\"}\n\\caption[Attaching a volume]{\\label{fig:attachvolume2}Fill in the information requested. Note that both the instance and the EBS volume are in the same region. You just need to provide an unused device to attach your volume to. If you don't know what this means, and are starting up a CloudBioLinux instance, then just use any of the suggested locations: /dev/sdf, /dev/sdg, /dev/sdh....up to /dev/sdp for now and check out \\href{http://it.toolbox.com/wiki/index.php/Mount_point}{some documentation to find out what it means later}.}\n\t\t\\end{minipage}\n\t}\n\\end{figure}\n\n\n\\begin{itemize}\n\\item Click on the \\textbf{Attach Volume} button on the Volumes page. See figure~\\ref{fig:attachvolume1}. \n\\item To attach a volume using the AWS Console interface, you just need to fill in the requested information. See figure~\\ref{fig:attachvolume2}.\n\\end{itemize}\n\\paragraph{}Note that selecting any of your volumes in the AWS Management Console will bring up details of that volume at the bottom of the page. \n\n\n\\paragraph{Accessing your volume}\n\\paragraph{}\\label{text:mounting}This is where it gets a bit ugly, as you need to log into your machine and use the command line for the next couple of steps. The first of these steps, \\textbf{formatting your disk}, only needs to be done \\emph{the first time} you use a particular volume. The second step, \\textbf{mounting the volume}, needs to be done  \\emph{each time} you want to access data on your volume from a new instance. There are ways to automate mounting a volume onto new instances, but these are not covered here. \n\n\\begin{enumerate}\n\\item Log into your instance. If you are logged in using NX, start up a terminal window.\n\\item \\textbf{The first time you mount a volume for use:} Type the following command to create an ext3 filesystem on your volume. Here I assume you have mounted it to /dev/sdf. See figure \\ref{fig:attachvolume2} on page \\pageref{fig:attachvolume2}.\n\\\\\\textbf{sudo mkfs -t ext3 /dev/sdf }\n\\item Now make an empty directory. This will act as a place where you can mount your new filesystem. By convention, mount points tend to be put in directories such as /mnt or /media. For example, the command below creates a directory called /mnt/datasets:\n\\\\\\textbf{sudo mkdir /mnt/datasets}\n\\item Now you can mount your volume onto this:\n\\\\\\textbf{sudo mount /dev/sdf /mnt/datasets}\n\\end{enumerate}\n\n\\paragraph{}You will now be able to put data under the folder /mnt/datasets. All files under that directory are on your EBS volume and will not be lost when your instance terminates. Ensure you read the section on page \\pageref{text:unmounting} on \\textbf{unmounting your volume} as failure to unmount before detaching your volume (or terminating your instance) could lead to data corruption. \n\n\\paragraph{Unmounting your volume}\n\n\\begin{figure}[!hd]\n\t\\fbox\n\t{\n\t\t\\begin{minipage}{13cm}\n\\includegraphics[width=\\maxwidth]{\"images/unmountDetach-1\"}\n\\caption[Attaching a volume]{\\label{fig:unmountdetach}You must unmount a volume before detaching it. Not doing so may result in data corruption on the volume. You can then either detach the volume from your instance by using the \\textbf{Detach Volume} button on the AWS Console, or the volume will be detached for you when you terminate your instance. The commands for unmounting a volume from within your running instance are provided in the text on page \\pageref{text:unmounting}}\n\t\t\\end{minipage}\n\t}\n\\end{figure}\n\n\\paragraph{This is a simple but vital step to avoid the possibility of data corruption.} Do this \\emph{before you detach} your volume or terminate your instance.\n\n\\paragraph{}If you had attached your device to /dev/sdf, then you simply need to type:\n\\\\\\textbf{umount -d /dev/sdf}\n\\label{text:unmounting}\n\\paragraph{Detaching your volume}\n\\paragraph{}You can detach your volume from your instance using the AWS Management Console using the Detach Volume button on the console Navigation pane. Alternatively, your volumes will be detached automatically when you terminate your instance.\n\n\\paragraph{Backing up or sharing your volume}\n\\paragraph{}Check out the Userguide information on \\href{http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/creating-snapshot-ebs.html}{creating snapshots} and on \\href{http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/modifying-snapshot-permissions-ebs.html}{modifying permisisons on snapshots}.\n\n\n\\paragraph{Deleting your volume}\n\\paragraph{}You can delete your volume using the Delete button on the console Navigation pane. \n\n\\section{Accesing public datasets on Amazon}\n\\label{section:publiccopy}\n\\begin{figure}[!hd]\n\t\\fbox\n\t{\n\t\t\\begin{minipage}{13cm}\n\\includegraphics[width=\\maxwidth]{\"images/public-ensembl\"}\n\\caption[List public data]{\\label{fig:ensembl}Click the Snapshot link in the Navigation pane. Then select Public Snapshots in the Viewing menu. Searching for a term such as \\emph{ensembl} brings up all the public snapshots that contain ensembl in their description. }\n\t\t\\end{minipage}\n\t}\n\\end{figure}\n\n\\paragraph{}Amazon makes some \\href{http://aws.amazon.com/publicdatasets/}{public data sets available as snapshots}. You can just attach and mount these (see grey boxes in figure~\\ref{fig:createandmountvolume}. No data transfer is necessary. Finding datasets is easy: just search through the public snapshots for relevant terms. See figure~\\ref{fig:ensembl}. Check out the \\href{http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=243}{full public data set listing}. \n\n\\paragraph{}Amazon provides \\href{http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/using-public-data-sets.html}{documentation on how to make use of these public data resources}.\n\n\n\\paragraph{}\n\n\n\n\n"
  },
  {
    "path": "doc/linux_kvm.md",
    "content": "# CloudBioLinux and Linux KVM\n\nThis document gives some additional information on using BioLinux on Linux KVM.\nKVM is a great virtualization environment, it is part of the Linux effort, will\nwork with the default Linux kernel. A running 64-bit Linux kernel can run both\n32-bit and 64-bit VMs.  With most Linux distributions KVM comes out of the\nbox...\n\nTogether with the BioLinux fabric environment, any KMV VM can be\nbootstrapped. Here we start from a bare Debian/Ubuntu\ninstallation. With Debian you can install from a fresh download of\n[stable](http://www.debian.org/releases/stable/) version burning it on\na CDROM.  The default install will do on, say, a 10GB partition - so\nthe rest can be used for LVM partitioning. Select ssh and the standard\nsystem utilities.\n\n# Install KVM\n\nThere are many web resources for installing KVM. On Debian derived systems:\n\n      apt-get install kvm libvirt-bin virtinst bridge-utils \\\n                      qemu-kvm virt-manager libvirt-bin\n\nand add your user to the kvm group. E.g.\n\n    adduser username kvm\n\nFor non-Debian systems see, for example, [OpenSuse](http://doc.opensuse.org/documentation/html/openSUSE/opensuse-kvm/cha.kvm.requires.html).\n\nNote that not all machines support virtualization, and if they do you\nmay need to switch it on in the BIOS (especially true on older\nhardware). Check for CPU support with:\n\n    egrep --color '(vmx|svm)' /proc/cpuinfo\n\nAlso note that some older Linux installations may need a kernel\nupgrade. Start KVM with\n\n    /etc/init.d/qemu-kvm start\n\nor with later versions\n\n    /etc/init.d/qemu-system-x86 start\n\n# Create a bare VM\n\nDownload a live installation image file. For example fetch a standard\nor network image from [[http://www.debian.org/][Debian]].\n\nReserve space on the disk partition - this should be enough for a\nDebian install and updates.\n\n      qemu-img create hda.img -opreallocation=metadata -ocluster_size=2M -f qcow2 10G\n\n(settings suggested by Red Hat) and fire up the VM\n\n      kvm debian-live-$(VER).img -hda hda.img -curses -no-reboot -serial pty\n\nAlternatively use the netinstall. The CloudBioLinux integration test\nsystem does something similar, starting from the smaller net install\nof Debian Linux:\n\n      qemu-system-x86_64 -enable-kvm -cdrom debian-$(VER)-amd64-netinst.iso -hda hda.img\n\nhit ESC and optionally type 'install fb=false' to disable the frame buffer.\nFire up the installer. Note that the file system of the installer can be slow,\nthat speed is not representative for an installed VM later (with -enable-kvm).\n\nWith the base install, boot the new system\n\n      qemu-system-x86_64 -enable-kvm -redir tcp:2222::22 -hda hda.img\n\nand install ssh on the VM (it comes already on netinst)\n\n      apt-get install openssh-server\n\nso that ssh login works\n\n      ssh -p 2222 biolinux@localhost\n\non user biolinux without a password (preferably using a key with\nempty password). And give that user 'sudo bash'. This ssh and sudo\nconfiguration is described in ./doc/private_cloud.md. After generating\nthe key \n\n      ssh -i ~/.ssh/biolinux -p 2222 biolinux@localhost\n\ntest run sudo without a password\n\n      sudo bash\n\nNot much to installing Linux with KVM! From this point onwards you can install\nCloudBioLinux using the fabric file. Make a copy of the hda.img file,\nso you can have the same starting point every time\n\n      cp hda.img kvm_with_biolinux_login.img\n\nThe CloudBioLinux test script also starts from here.\nTry the ./test/test_biolinux script to test drive the VM. test_biolinux\nwill install a CloudBioLinux flavor, and check whether the installation is\ncomplete. Essentially with a running instance:\n\n      ./test/test_biolinux -p 2222 -u biolinux -i ~/.ssh/biolinux 127.0.0.1\n\n(note the use of 127.0.0.1 over localhost - this is because of a bug\nin fabric - is this still true?).\n\n# KVM tips\n\nKVM is nice and powerful. It is used in many Cloud service providers.\n\nFor fast performance, it pays to install on a raw (LVM) partition,\nget bridging sorted, and make sure hardware acceleration is in place.\nInteresting goodies are the monitor (Crtl-Alt-2), virtsh, etc. See also\n[http://www.linux-kvm.org/page/FAQ][kvm tips].\n\n[kvm tips]: http://www.linux-kvm.org/page/FAQ\n"
  },
  {
    "path": "doc/private_cloud.md",
    "content": "# Private Cloud and CloudBioLinux\n\nCloudBioLinux can be used to create a private Cloud for\nBioinformatics. Essentially, all you need is ssh access to a VM\nrunning somewhere. This VM should be a clean install of Linux.  With\nCloudBioLinux Debian and Ubuntu distributions are supported best.\n\n## Start a VM\n\nStart a VM and make sure there is a network defined, and ssh running.\nOn the VM\n\n       dhclient -v\n       ifconfig\n       ps xa|grep ssh\n       ssh localhost\n\nCheck the network (e.g. with Debian)\n\n       apt-get update\n       apt-get install vim\n\nYou should be able to use the IP address to login from your desktop\n\n       ssh biolinux@VM_IP_address\n\n## Get password free ssh access\n\nThe CloudBioLinux fabric tools work best when you have password free\nlogin. If you can login to the remote with\n\n       ssh biolinux@VM_IP_address\n\nyou are set. Otherwise, create a password free ssh key. To achieve this, see\nthe many Internet resources, e.g.  http://www.mtu.net/~engstrom/ssh-agent.php.\nCombine username, key, hostname and port in ./ssh/config as\n\n    Host biolinux\n      hostname localhost\n      user biolinux\n      port 2222\n      IdentityFile ~/.ssh/biolinux\n\nso you can login with\n\n    ssh biolinux\n\nIf this works it is possible cloudbiolinux comes with a fab error saying\n\n    Fatal error: Low level socket error connecting to host localhost on port 2222\n\nOne possibility is that it tries IPv6 to connect to localhost. You may have\nto comment out the line '::1 localhost ' in /etc/hosts to run fab (it has bitten\nme several times).\n\n## Install sudo without password\n\nInstall the sudo program. Next, edit /etc/sudoers with the 'visudo'\ncommand, and add the line\n\n       biolinux ALL=NOPASSWD: /bin/bash\n\nwhere biolinux is your VM user login name. Alternatively add biolinux to the sudo\ngroup.\n\nNow try:\n\n       sudo bash\n\nand you should be root, without a password.\n\n## Install CloudBioLinux\n\nSee the README for installing CloudBioLinux and fabric.\n\n## Run fabric\n\nNow you should be set! To install BioLinux\n\n       fab -f $source/fabfile.py -H biolinux@$VM_IP_address -c $fabricrc install_biolinux:packagelist=$packagelist\n\nWhere source points to the checked out source tree, e.g.\n\n       export source=$HOME/izip/git/opensource/debian/biolinux\n\nFor example, to install the Minimal flavor on Debian stable on a VM\nrunning on IP 192.168.64.105:\n\n       fab -f $source/fabfile.py -H biolinux@192.168.64.105 \\\n       -c $source/contrib/flavor/minimal/fabricrc_debian.txt \\\n       install_biolinux:packagelist=$source/contrib/flavor/minimal/main.yaml\n\nCloudBioLinux shows the following output. First it sets up the\nenvironment\n\n        [192.168.64.105] Executing task 'install_biolinux'\n        cloudbiolinux WARNING: Skipping fabricrc.txt as distribution is already defined\n        cloudbiolinux DEBUG: Minimal Edition 1.0.1\n        cloudbiolinux INFO: This is a minimal\n        cloudbiolinux INFO: This is a Base Flavor - no overrides\n        cloudbiolinux INFO: Distribution debian\n        cloudbiolinux INFO: Debian setup\n        cloudbiolinux DEBUG: Debian-shared setup\n        cloudbiolinux DEBUG: Source=squeeze\n        cloudbiolinux DEBUG: Checking target distribution debian\n        [192.168.64.105] run: cat /proc/version\n        [192.168.64.105] out: Linux version 2.6.32-5-amd64 (Debian 2.6.32-31) (ben@decadent.org.uk) (gcc version 4.3.5 (Debian 4.3.5-4) ) #1 SMP Mon Mar 7 21:35:22 UTC [192.168.64.105] out:\n        [192.168.64.105] out:\n        cloudbiolinux INFO: Now, testing connection to host...\n        cloudbiolinux INFO: Connection to host appears to work!\n        cloudbiolinux DEBUG: Expand paths\n        cloudbiolinux INFO: packagelist=/home/biolinux/izip/git/opensource/debian/biolinux/contrib/flavor/minimal/main.yaml\n        cloudbiolinux INFO: Meta-package information\n        cloudbiolinux INFO: minimal,ruby\n        cloudbiolinux INFO:\n        cloudbiolinux INFO: Target=None\n\nHere it modifies the source file for apt-get, as well as keys:\n\n        cloudbiolinux DEBUG: _setup_apt_sources /etc/apt/sources.list.d/cloudbiolinux.list Minimal Edition\n        [192.168.64.105] sudo: touch /etc/apt/sources.list.d/cloudbiolinux.list\n        [192.168.64.105] sudo: echo '# This file was modified for Minimal Edition' >> /etc/apt/sources.list.d/cloudbiolinux.list\n        cloudbiolinux DEBUG: Source deb http://ftp.nl.debian.org/debian/ squeeze main contrib non-free\n        [192.168.64.105] sudo: echo 'deb http://ftp.nl.debian.org/debian/ squeeze main contrib non-free' >> /etc/apt/sources.list.d/cloudbiolinux.list\n        cloudbiolinux DEBUG: Source deb http://ftp.nl.debian.org/debian/ squeeze-updates main contrib non-free\n        [192.168.64.105] sudo: echo 'deb http://ftp.nl.debian.org/debian/ squeeze-updates main contrib non-free' >> /etc/apt/sources.list.d/cloudbiolinux.list\n        [192.168.64.105] sudo:\n        cloudbiolinux INFO: Update GPG keys for repositories\n        cloudbiolinux INFO: Update and install all packages\n        [192.168.64.105] sudo: apt-get update\n        [192.168.64.105] out: Hit http://ftp.nl.debian.org squeeze Release.gpg\n\nand starts installing packages\n\n        cloudbiolinux INFO: Updating 26 packages\n        [192.168.64.105] sudo: apt-get -y --force-yes install ruby1.8 ruby1.8-dev ruby1.9.1 ruby1.9.1-dev axel less openssh-server rsync screen sudo tar unzip bzr cvs darcs git-core mercurial subversion vim cmake g++ gcc gfortran make patch swig\n        [192.168.64.105] out: Reading package lists... Done\n        [192.168.64.105] out: Building dependency tree\n        [192.168.64.105] out: Reading state information... Done\n        [192.168.64.105] out: gcc is already the newest version.\n        [192.168.64.105] out: gcc set to manually installed.\n        [192.168.64.105] out: less is already the newest version.\n        [192.168.64.105] out: less set to manually installed.\n        [192.168.64.105] out: make is already the newest version.\n        (etc, etc)\n\nFinally some clean ups\n\n        [192.168.64.105] sudo: apt-get clean\n        cloudbiolinux INFO: Target=unknown; Edition=Minimal Edition; Flavor=Base Flavor - no overrides\n\nwrite an entry in the log file\n\n        [192.168.64.105] sudo: date +\"%D %T - Updated Target=unknown; Edition=Minimal Edition; Flavor=Base Flavor - no overrides\" >> /var/log/biolinux.log\n        [192.168.64.105] run: uname -m\n        [192.168.64.105] out: x86_64\n        [192.168.64.105] out:\n        cloudbiolinux INFO: Reading /home/biolinux/izip/git/opensource/debian/biolinux/config/custom.yaml\n        cloudbiolinux DEBUG: Packages:\n        cloudbiolinux DEBUG:\n        cloudbiolinux INFO: Cleaning up space from package builds\n        [192.168.64.105] sudo: rm -rf .cpanm\n        [192.168.64.105] sudo: rm -f /var/crash/*\n\nAnd it is done. Minimal has no post-installation configuration, but\nthat is easy to add.\n\n"
  },
  {
    "path": "doc/remote_gui.md",
    "content": "# BioLinux Remote X access\n\nBioLinux supports both VNC and freenx GUI X-windows access to a remote\nVM. And you can use X programs through ssh, naturally.\n\n## VNC\n\nVNC is a ubiquitous remote access tool - always there, and easy to install/use.\n\nIn a nutshell:\n\nMake sure vnc4server is installed on the VM.  Enable ports 5900, 5901 and 5902\non the VM. Run the server\n\n        vnc4server -depth 24\n           (set password)\n\nRun the client on your desktop\n\n       vncviewer -FullColor=1 HostIP:1\n\nWhere HostIP is the reachable host IP address or DNS name. Next, it \nmay be necessary to start an X desktop, such as LXDE:\n\n       startlxde\n\n### Amazon EC2 ports\n\nCreate a security group for your instance that allows at least ports \n22,5900,5901 and 5902.\n\n### Vagrant ports\n\nYou may need to add port forwarding to vagrant - as the testing system\ndoes. I.e. add to the Vagrantfile:\n\n       config.vm.forward_port('vnc0', 5900, 5900)\n       config.vm.forward_port('vnc1', 5901, 5901)\n       config.vm.forward_port('vnc2', 5902, 5902)\n\nThis is for testing, mostly. You do not need VNC on Vagrant/VirtualBox. Fire up\nthe GUI directly!\n\n### VNC Security\n\nPlease note that VNC is not very secure - it has no proper key protection. You\ncan tunnel over ssh for improved security. Or use freenx instead.\n\n## FreeNX\n\nFreeNX is a fast version of the X protocol.\n\nMake sure freenx is installed on the VM. CloudBioLinux comes with scripts\nfor setting up freenx.\n\n(to be filled in)\n\n## X over ssh\n\nNormally you have ssh access to the remote VM. You can use X-windows programs\nremotely, provided you have a local X server (always on Linux and OSX). Just\nlogin with the -X switch\n\n      ssh -X user@$hostIP\n\nin the terminal type an X program, e.g.\n\n      firefox\n\nand the program should display locally (running remotely).\n\n"
  },
  {
    "path": "doc/source/conf.py",
    "content": "# -*- coding: utf-8 -*-\n#\n# CloudBioLinux documentation build configuration file, created by\n# sphinx-quickstart on Wed Jul 17 09:14:27 2013.\n#\n# This file is execfile()d with the current directory set to its containing dir.\n#\n# Note that not all possible configuration values are present in this\n# autogenerated file.\n#\n# All configuration values have a default; values that are commented out\n# serve to show the default.\n\nimport sys, os\n\n# If extensions (or modules to document with autodoc) are in another directory,\n# add these directories to sys.path here. If the directory is relative to the\n# documentation root, use os.path.abspath to make it absolute, like shown here.\n#sys.path.insert(0, os.path.abspath('.'))\n\n# -- General configuration -----------------------------------------------------\n\n# If your documentation needs a minimal Sphinx version, state it here.\n#needs_sphinx = '1.0'\n\n# Add any Sphinx extension module names here, as strings. They can be extensions\n# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.\nextensions = ['sphinx.ext.autodoc', 'sphinx.ext.todo', 'sphinx.ext.viewcode']\n\n# Add any paths that contain templates here, relative to this directory.\ntemplates_path = ['_templates']\n\n# The suffix of source filenames.\nsource_suffix = '.rst'\n\n# The encoding of source files.\n#source_encoding = 'utf-8-sig'\n\n# The master toctree document.\nmaster_doc = 'index'\n\n# General information about the project.\nproject = u'CloudBioLinux'\ncopyright = u'2013, CloudBioLinux contributors'\n\n# The version info for the project you're documenting, acts as replacement for\n# |version| and |release|, also used in various other places throughout the\n# built documents.\n#\n# The short X.Y version.\nversion = '1.0'\n# The full version, including alpha/beta/rc tags.\nrelease = '1.0'\n\n# The language for content autogenerated by Sphinx. Refer to documentation\n# for a list of supported languages.\n#language = None\n\n# There are two options for replacing |today|: either, you set today to some\n# non-false value, then it is used:\n#today = ''\n# Else, today_fmt is used as the format for a strftime call.\n#today_fmt = '%B %d, %Y'\n\n# List of patterns, relative to source directory, that match files and\n# directories to ignore when looking for source files.\nexclude_patterns = []\n\n# The reST default role (used for this markup: `text`) to use for all documents.\n#default_role = None\n\n# If true, '()' will be appended to :func: etc. cross-reference text.\n#add_function_parentheses = True\n\n# If true, the current module name will be prepended to all description\n# unit titles (such as .. function::).\n#add_module_names = True\n\n# If true, sectionauthor and moduleauthor directives will be shown in the\n# output. They are ignored by default.\n#show_authors = False\n\n# The name of the Pygments (syntax highlighting) style to use.\npygments_style = 'sphinx'\n\n# A list of ignored prefixes for module index sorting.\n#modindex_common_prefix = []\n\n\n# -- Options for HTML output ---------------------------------------------------\n\n# The theme to use for HTML and HTML Help pages.  See the documentation for\n# a list of builtin themes.\nhtml_theme = 'default'\n\n# Theme options are theme-specific and customize the look and feel of a theme\n# further.  For a list of options available for each theme, see the\n# documentation.\n#html_theme_options = {}\n\n# Add any paths that contain custom themes here, relative to this directory.\n#html_theme_path = []\n\n# The name for this set of Sphinx documents.  If None, it defaults to\n# \"<project> v<release> documentation\".\n#html_title = None\n\n# A shorter title for the navigation bar.  Default is the same as html_title.\n#html_short_title = None\n\n# The name of an image file (relative to this directory) to place at the top\n# of the sidebar.\n#html_logo = None\n\n# The name of an image file (within the static path) to use as favicon of the\n# docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32\n# pixels large.\n#html_favicon = None\n\n# Add any paths that contain custom static files (such as style sheets) here,\n# relative to this directory. They are copied after the builtin static files,\n# so a file named \"default.css\" will overwrite the builtin \"default.css\".\nhtml_static_path = ['_static']\n\n# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,\n# using the given strftime format.\n#html_last_updated_fmt = '%b %d, %Y'\n\n# If true, SmartyPants will be used to convert quotes and dashes to\n# typographically correct entities.\n#html_use_smartypants = True\n\n# Custom sidebar templates, maps document names to template names.\n#html_sidebars = {}\n\n# Additional templates that should be rendered to pages, maps page names to\n# template names.\n#html_additional_pages = {}\n\n# If false, no module index is generated.\n#html_domain_indices = True\n\n# If false, no index is generated.\n#html_use_index = True\n\n# If true, the index is split into individual pages for each letter.\n#html_split_index = False\n\n# If true, links to the reST sources are added to the pages.\n#html_show_sourcelink = True\n\n# If true, \"Created using Sphinx\" is shown in the HTML footer. Default is True.\n#html_show_sphinx = True\n\n# If true, \"(C) Copyright ...\" is shown in the HTML footer. Default is True.\n#html_show_copyright = True\n\n# If true, an OpenSearch description file will be output, and all pages will\n# contain a <link> tag referring to it.  The value of this option must be the\n# base URL from which the finished HTML is served.\n#html_use_opensearch = ''\n\n# This is the file name suffix for HTML files (e.g. \".xhtml\").\n#html_file_suffix = None\n\n# Output file base name for HTML help builder.\nhtmlhelp_basename = 'CloudBioLinuxdoc'\n\n\n# -- Options for LaTeX output --------------------------------------------------\n\nlatex_elements = {\n# The paper size ('letterpaper' or 'a4paper').\n#'papersize': 'letterpaper',\n\n# The font size ('10pt', '11pt' or '12pt').\n#'pointsize': '10pt',\n\n# Additional stuff for the LaTeX preamble.\n#'preamble': '',\n}\n\n# Grouping the document tree into LaTeX files. List of tuples\n# (source start file, target name, title, author, documentclass [howto/manual]).\nlatex_documents = [\n  ('index', 'CloudBioLinux.tex', u'CloudBioLinux Documentation',\n   u'CloudBioLinux contributors', 'manual'),\n]\n\n# The name of an image file (relative to this directory) to place at the top of\n# the title page.\n#latex_logo = None\n\n# For \"manual\" documents, if this is true, then toplevel headings are parts,\n# not chapters.\n#latex_use_parts = False\n\n# If true, show page references after internal links.\n#latex_show_pagerefs = False\n\n# If true, show URL addresses after external links.\n#latex_show_urls = False\n\n# Documents to append as an appendix to all manuals.\n#latex_appendices = []\n\n# If false, no module index is generated.\n#latex_domain_indices = True\n\n\n# -- Options for manual page output --------------------------------------------\n\n# One entry per manual page. List of tuples\n# (source start file, name, description, authors, manual section).\nman_pages = [\n    ('index', 'cloudbiolinux', u'CloudBioLinux Documentation',\n     [u'CloudBioLinux contributors'], 1)\n]\n\n# If true, show URL addresses after external links.\n#man_show_urls = False\n\n\n# -- Options for Texinfo output ------------------------------------------------\n\n# Grouping the document tree into Texinfo files. List of tuples\n# (source start file, target name, title, author,\n#  dir menu entry, description, category)\ntexinfo_documents = [\n  ('index', 'CloudBioLinux', u'CloudBioLinux Documentation',\n   u'CloudBioLinux contributors', 'CloudBioLinux', 'One line description of project.',\n   'Miscellaneous'),\n]\n\n# Documents to append as an appendix to all manuals.\n#texinfo_appendices = []\n\n# If false, no module index is generated.\n#texinfo_domain_indices = True\n\n# How to display URL addresses: 'footnote', 'no', or 'inline'.\n#texinfo_show_urls = 'footnote'\n"
  },
  {
    "path": "doc/source/framework.rst",
    "content": "\nUsing the CloudBioLinux Build Framework\n---------------------------------------\n\n-----------------------\nObtaining CloudBioLinux\n-----------------------\n\nCloudBioLinux can be obtained using `git <http://git-scm.com/>`_.\n\n::\n\n    % git clone https://github.com/chapmanb/cloudbiolinux.git\n    % cd cloudbiolinux\n\n-------------------------\nOverview of the Framework\n-------------------------\n\n\n------------------------\nBuilding Cloud Instances\n------------------------\n\nWhen building CloudBioLinux instances for the cloud, ``deploy/deploy.sh``\ncontains a script to automate cloud interactions and the installation of\nCloudBioLinux. This script will require a system Python be available on your\nsystem, but should otherwise install its own dependencies.\n\nBefore using the deployer you will need to create a settings file describing\nyour cloud credentials and connection information as well as any tweaks you\nwould like to make to the:\n\n::\n\n    % cd deploy\n    % cp settings-sample-minimal.yaml settings.yaml\n\nBefore updating settings.yaml you will need to navigate the AWS management\nconsole and obtain the following information.\n\n* Your AWS Access ID and secret key (`access_id`, `secret_key`)\n* Ubuntu EBS-backed AMI ID to target. This writeup was tested with \n  `ami-9b85eef2` (12.04.2 (64-bit) in us-east-1).\n* Image size to use (e.g. m1-small)\n* Availability zone (e.g. us-east-1)\n\nCarefully scan through `settings.yaml` and change the properties marked as\nrequiring change. The word `UPDATE` in the comments indicates properties of\nspecial interest that either don't have reasonable defaults or have reasonable\ndefaults but that I have deemed highly likely to be overridden. For this simple\nexample the only changes you will need to make are in the ``aws`` section.\n\nOnce you have updated ``settings.yaml``, launch and cloud instance and\nconfigure it with the following command:\n\n::\n\n    % ./deploy.sh --action=install_biolinux --flavor=minimal\n\nThis command will configure CloudBioLinux with a minimal set of CloudBioLinux\npackages. The set of packages that is installed is controlled by the\n``--flavor`` command. More sophisticated setups that require using Amazon EBS\nvolumes and S3 buckets such as CloudMan clusters require additional\nconfiguration as outlined below.\n\nYou can SSH into the newly created cloud instance with the command:\n\n::\n\n    % ./deploy.sh --action=ssh\n\n-----------------------------------\nBuilding CloudMan Enabled Instances\n-----------------------------------\n\nBefore continuing, delete your previous instance and the file\n``.vmlauncher_last_instance_aws``. TODO: Add action for this.\n\nBuilding a more sophisticated CloudBioLinux image integrating tools such as\nCloudMan requires additional settings. Please start by copying ``settings-\nsample-cm.yaml`` to ``settings.yaml`` and repopulating the ``aws`` section\noptions. \n\nTo fill out the remaining options found in this file, you will need to return\nthe AWS management console and do the following:\n\n* You will need to setup a bucket to store your snaps file, here you will need the bucket name.\n* You will need to setup two volumes in your target availability zone, one for\n  Galaxy tools and data (perhaps 20Gb for testing) and one for galaxyIndices. Here you will need the volume ids.\n* Generate a private a key (e.g. galaxy1.pem) and copy it into keys directory (or anywhere really), \n  also note the keypair_name corresponding to the key.\n\nNext you will want to setup a directory to contain the S3 bucket contents that\nwill eventually be used by CloudMan to configure your cluster. Create a\ndirectory (e.g. `/home/mary/marys_cloudman_bucket_contents`). Copy the files\nfrom an existing CloudMan bucket here (e.g. http://s3.amazonaws.com /cloudman-\ndev).\n\nIt is not really important how you download these files, but one quick option\nis to use `s3cmd` tool:\n\n::\n\n    % sudo apt-get install s3cmd  # Or your OS's package manager\n    % mkdir /home/mary/marys_cloudman_bucket_contents\n    % s3cmd -r get s3://cloudman-dev /home/mary/marys_cloudman_bucket_contents\n\nHere you can replace the CloudMan source (i.e. `cm.tar.gz`) or any of these\nfiles to match the customized setup you would like. In particular you are\ngoing to want to create a custom snaps.yaml file. Here is a simple outline\nthat we will fill out as we good.\n\n::\n\n    version: 1\n    clouds:\n      - name: amazon\n        regions:\n        - deployments:\n          - name: GalaxyCloud\n            filesystems:\n            - name: galaxy\n              roles: galaxyTools,galaxyData\n              snap_id: snap-XXXXXXXXXXX\n              mount_point: /mnt/galaxy\n            - name: galaxyIndices\n              roles: galaxyIndices\n              snap_id: snap-XXXXXXXXXXXX\n              mount_point: /mnt/galaxyIndices\n            default_mi: ami-XXXXXXXXXXXXX\n            bucket: marys_cloudman_bucket\n          name: us-east-1\n\nImmediately this template can be updated to reflect the bucket created above\nand the availability zone you are targetting. We can update the snap_id's and\nthe default_mi after creating them.\n\nReopen ``settings.yaml`` and fill out the remaining properties, including the\nvolume ids you just created and the name of the bucket you used.\n\nThe following set of commands will now launch a new cloud instance, attach\nand format tool and data volumes for CloudMan, build CloudBioLinux, snapshot\nthese volumes, and package the image.\n\n::\n\n    % ./deploy.sh --action=launch\n    % ./deploy.sh --action=attach_volumes\n    % ./deploy.sh --action=install_biolinux --flavor=cloudman/cloudman_and_galaxy\n    % ./deploy.sh --action=snapshot_volumes\n    % ./deploy.sh --action=detach_volumes\n    % ./deploy.sh --action=package\n\nOnce a CloudMan AMI has been created, update `snaps.yaml` in your bucket\ndirectory (e.g. `/home/mary/marys_cloudman_bucket_contents`) to reflect the\n`snap_id`s and AMI created. These should all be available via the AWS\nmanagement console or by reviewing the output of the steps above.\n\nFinally, you can upload your new bucket and launch a test CloudMan instance:\n\n::\n\n    % ./deploy.sh --action=sync_cloudman_bucket\n    % ./deploy.sh --action=cloudman_launch\n\nThis last action (``cloudman_launch``) requires uncommenting the following\nlines and updating the bucket name:\n\n::\n\n    #image_user_data:\n    #  bucket_default: marys_cloudman_bucket\n\n"
  },
  {
    "path": "doc/source/index.rst",
    "content": ".. CloudBioLinux documentation master file, created by\n   sphinx-quickstart on Wed Jul 17 09:14:27 2013.\n   You can adapt this file completely to your liking, but it should at least\n   contain the root `toctree` directive.\n\nWelcome to CloudBioLinux's documentation!\n=========================================\n\n.. include:: ../../README.rst\n.. include:: framework.rst\n\n\nIndices and tables\n==================\n\n* :ref:`genindex`\n* :ref:`modindex`\n* :ref:`search`\n\n"
  },
  {
    "path": "doc/virtualbox.md",
    "content": "# CloudBioLinux, VirtualBox and Vagrant\n\nThis document gives some additional information on using Vagrant with BioLinux.\n[Vagrant][v1] is a convenient command line manager for VirtualBox. In conjunction\nwith the BioLinux fabric environment, any VirtualBox VM can be bootstrapped.\n\nNote the current version of vagrant needs at least VirtualBox version 4.1.x.\n\n## VirtualBox with vagrant\n\nAdd a base image to vagrant, and boot it up; community Vagrant boxes are available from\n[http://vagrantbox.es][v3] and [BioLinux flavors][v4]:\n\n        vagrant box add box_name http://path_to_the_image.box\n        mkdir tmp/biolinux\n        cd tmp/biolinux\n        vagrant init box_name\n        vagrant up\n\nRun the fabfile, building CloudBioLinux:\n\n        fab -H vagrant -f /path/to/cloudbiolinux/fabfile.py install_biolinux\n\nThen build the box, renaming package.box to `cloudbiolinux_date` and\nmove it to a public webserver, such as Amazon S3:\n\n        vagrant package\n        mv package.box biolinux_20110122.box\n        s3cmd put --acl-public --guess-mime-type biolinux_20110122.box\n              s3://chapmanb/biolinux_20110122.box\n\n[v3]: http://vagrantbox.es/\n[v4]: http://biobeat.org/bionode\n\n# Rolling your own\n\n## Start from a BioLinux box\n\nSee the main README file for firing up a pre-installed BioLinux box.\n\n## Start from scratch\n\nDespite the extra work, starting from scratch may have advantages. For\none you have more control of the base install. Say for a different\nversion of Linux, a BSD kernel, or for install less software (do you\nreally need X/KDE/Gnome?), so you do not end up with an 8 GB VM, or for more\nsoftware and/or data pre-installed on a VM. \n\nThe BioLinux setup is designed to be modular, to support multiple \nflavors (see the main README for an explanation of terms).\n\nStart with a standard downloadable prepared Vagrant box. For example a Debian\n32-bits box prepared for Vagrant, or create one from scratch as is explained on\nthe [Vagrant web site][v1].\n\nNext add the virtualbox to vagrant using a URL, or box file:\n\n          vagrant box add debian_squeeze_32 debian_squeeze_32.box\n\n(boxes are available form [http://vagrantbox.es][v3] and\n[http://biobeat.org/bionode][BioLinux flavors]) and create your own version\n\n          mkdir myflavor\n          cd myflavor\n\nCreates a ./Vagrantfile describing the VM.\n\n          vagrant init debian_squeeze_32\n\nHave a look inside the Vagrantfile. The default should be fine now.\n\nStart the VM (which gets copied the first time, which may take a while):\n\n          vagrant up\n\nand login\n\n          vagrant ssh  # no password needed\n\nmake sure you have enough disk space (twice the box size) for the dir\n~/VirtualBox\\ VMs and ~/.vagrant, as this is where VMs are copied from the\noriginal box file.\n\nAt this point a bare VM is running that will accept BioLinux installations. The\nnext step is to pull the BioLinux tree on your local system, and to run fab using the\nvagrant host, using a minimal install target. E.g.\n\n          export source=/path/to/cloudbiolinux\n\nand\n\n          fab -f $source/fabfile.py -H vagrant  -c $source/contrib/flavor/minimal/fabricrc_debian.txt install_biolinux:packagelist=$source/contrib/flavor/minimal/main.yaml\n\nwhich uses the information from the local ./Vagrantfile. \n\nThe first time the minimal fabfile is run it updates the /etc/apt/sources (on\nDebian-based systems), and a number of basic packages, including sudo, python,\nchef. It may be the Linux kernel and support libraries get upgraded, if they\nare in the dependency tree. Starting from a minimalistic Debian Vagrant box, the\nBioLinux minimal install has an unpacked size under 1Gb. E.g.\n\n        vagrant@vagrant-debian-squeeze:~$ df -h\n        Filesystem            Size  Used Avail Use% Mounted on\n        /dev/sda1              39G  804M   36G   3% /\n        tmpfs                 188M     0  188M   0% /lib/init/rw\n        udev                  184M  116K  184M   1% /dev\n        tmpfs                 188M     0  188M   0% /dev/shm\n        v-root                 51G   24G   27G  47% /vagrant\n\nDespite the fact that running fabfile.py is destructive, i.e. it overwrites the\ncurrent install, it is reasonably safe as it mostly uses the underlying package\nmanagement system and dependency resolution. Rerunning a BioLinux fabfile can\nbe fast.  Minimal runs the second time in under 20 seconds on a\nbasic laptop, as we do with a 'Minimal' install:\n\n         ./test/test_vagrant --continue\n\nFor completeness, after a minimal install you can still install a full BioLinux\nexecute\n\n        fab -H vagrant -f $source/fabfile.py install_biolinux\n\nOnce you have a working Virtual Box VM with vagrant, you can package it with\n\n        vagrant package\n\nand make the resulting .box file available for others to use.\n\nRead the README for further information.\n\n[v1]: http://vagrantup.com/docs/base_boxes.html\n\n## Trouble shooting\n\n### Guest additions\n\nYou may see an error\n\n  [default] The guest additions on this VM do not match the install version of\n  VirtualBox! This may cause things such as forwarded ports, shared\n  folders, and more to not work properly. If any of those things fail on\n  this machine, please update the guest additions and repackage the\n  box.\n\n  Guest Additions Version: 4.0.4\n  VirtualBox Version: 4.1.0\n\nthis error may actually be caused by the Vbox Linux kernel drivers not having\nbeen loaded! Fix\n\n       modprobe vboxdrv\n\n# Converting Vagrant images to VirtualBox and Eucalyptus images\n\n(protocol steps tested in Ubuntu Natty)\n\n## software pre-requisite\n\n    sudo gem install vagrant\n    sudo apt-get install cloud-utils\n\n## Importing cloud biolinux VM to your system\n\n    vagrant box add base \n    https://s3.amazonaws.com/cloudbiolinux/cbl_ubuntu_11_4_32_20110628.box\n    vagrant init base\n    vagrant up\n\n## adding some missing components to the vagrant VMs \n\n    vagrant ssh\n    sudo apt-get install gdm cloud-utils openssh\n    sudo useradd -d /home/ubuntu -m ubuntusudo passwd ubuntu\n    sudo shutdown -r now\n\nin the graphical login after reboot get in with user:ubuntu / pass:ubuntu\ngo to System--->Administration--->Login Window to enable autologin\n\n## VirtualBox Appliance\n\nVirtual Appliances are pre-assemblied VM images configured for various purposes.\n\nOpen the Virtualbox GUI, you should see the VM added by vagrant - you can \nrename it to \"Cloud BioLinux 32\"\n\n  File->Export Appliance\n\nand distribute the .ova.\n\nAnyone in any OS running Virtualbox can import the .ova with File->Import\nAppliance.\n\n# Making a Eucalyptus image from VirtualBox\n\nStart with the Cloud BioLinux Virtualbox .vmdk (its location is in the VM\nproperties from the Virtualbox GUI). Resize the vmdk, since the size may be\n40G, and the Eucalyptus image will have that size. \n\nAccording to http://mtnbike.org/blog/?p=29 and the same here:\nhttp://www.my-guides.net/en/content/view/122/26/\n\nconvert to raw .img \n\n    qemu-img convert -O raw CloudBioLinux-32bit-disk1.vmdk \n    CloudBioLinux-32bit-disk1.img\n\ndeploy to Eucalyptus via\n\n    uec-publish-img CloudBioLinux-32bit-disk1.img\n\n# VirtualBox, KVM or XEN?\n\nThere are more ways than one to virtualize machines on Linux.\n\nDespite the attractions of vagrant and Virtualbox, as displayed here, we note\nthat Linux KVM may be a better choice for virtualization and testing of\nCloudBioLinux, as Linux distributions support KVM out of the box, and KVM has\nmore Unix-like control.  See also the information for using KVM in\n./doc/linux_kvm.md. \n\nFor production environments check out XEN virtualization (XEN runs Amazon EC2).\n\n\n"
  },
  {
    "path": "fabfile.py",
    "content": "\"\"\"Main Fabric deployment file for CloudBioLinux distribution.\n\nThis installs a standard set of useful biological applications on a remote\nserver. It is designed for bootstrapping a machine from scratch, as with new\nAmazon EC2 instances.\n\nUsage:\n\n    fab -H hostname -i private_key_file install_biolinux\n\nwhich will call into the 'install_biolinux' method below. See the README for\nmore examples. hostname can be a named host in ~/.ssh/config\n\nRequires:\n    Fabric http://docs.fabfile.org\n    PyYAML http://pyyaml.org/wiki/PyYAMLDocumentation\n\"\"\"\nimport os\nimport sys\nfrom datetime import datetime\n\nfrom fabric.api import *\nfrom fabric.contrib.files import *\nimport yaml\n\n# use local cloudbio directory\nfor to_remove in [p for p in sys.path if p.find(\"cloudbiolinux-\") > 0]:\n    sys.path.remove(to_remove)\nsys.path.append(os.path.dirname(__file__))\nimport cloudbio\n\nfrom cloudbio import libraries\nfrom cloudbio.utils import _setup_logging, _configure_fabric_environment\nfrom cloudbio.cloudman import _cleanup_ec2, _configure_cloudman\nfrom cloudbio.cloudbiolinux import _cleanup_space, _freenx_scripts\nfrom cloudbio.custom import shared\nfrom cloudbio.package.shared import _yaml_to_packages\nfrom cloudbio.package import brew, conda\nfrom cloudbio.package import (_configure_and_install_native_packages,\n                              _connect_native_packages, _print_shell_exports)\nfrom cloudbio.package.nix import _setup_nix_sources, _nix_packages\nfrom cloudbio.flavor.config import get_config_file\nfrom cloudbio.config_management.puppet import _puppet_provision\nfrom cloudbio.config_management.chef import _chef_provision, chef, _configure_chef\n\n# ### Shared installation targets for all platforms\n\ndef install_biolinux(target=None, flavor=None):\n    \"\"\"Main entry point for installing BioLinux on a remote server.\n\n    `flavor` allows customization of CloudBioLinux behavior. It can either\n    be a flavor name that maps to a corresponding directory in contrib/flavor\n    or the path to a custom directory. This can contain:\n\n      - alternative package lists (main.yaml, packages.yaml, custom.yaml)\n      - custom python code (nameflavor.py) that hooks into the build machinery\n\n    `target` allows running only particular parts of the build process. Valid choices are:\n\n      - packages     Install distro packages\n      - custom       Install custom packages\n      - chef_recipes Provision chef recipes\n      - libraries    Install programming language libraries\n      - post_install Setup CloudMan, FreeNX and other system services\n      - cleanup      Remove downloaded files and prepare images for AMI builds\n    \"\"\"\n    _setup_logging(env)\n    time_start = _print_time_stats(\"Config\", \"start\")\n    _check_fabric_version()\n    if env.ssh_config_path and os.path.isfile(os.path.expanduser(env.ssh_config_path)):\n        env.use_ssh_config = True\n    _configure_fabric_environment(env, flavor,\n                                  ignore_distcheck=(target is not None\n                                                    and target in [\"libraries\", \"custom\"]))\n    env.logger.debug(\"Target is '%s'\" % target)\n    env.logger.debug(\"Flavor is '%s'\" % flavor)\n    _perform_install(target, flavor)\n    _print_time_stats(\"Config\", \"end\", time_start)\n    if hasattr(env, \"keep_isolated\") and env.keep_isolated:\n        _print_shell_exports(env)\n\ndef _perform_install(target=None, flavor=None, more_custom_add=None):\n    \"\"\"\n    Once CBL/fabric environment is setup, this method actually\n    runs the required installation procedures.\n\n    See `install_biolinux` for full details on arguments\n    `target` and `flavor`.\n    \"\"\"\n    pkg_install, lib_install, custom_ignore, custom_add = _read_main_config()\n    if more_custom_add:\n        if custom_add is None:\n            custom_add = {}\n        for k, vs in more_custom_add.iteritems():\n            if k in custom_add:\n                custom_add[k].extend(vs)\n            else:\n                custom_add[k] = vs\n    if target is None or target == \"packages\":\n        env.keep_isolated = getattr(env, \"keep_isolated\", \"false\").lower() in [\"true\", \"yes\"]\n        # Only touch system information if we're not an isolated installation\n        if not env.keep_isolated:\n            # can only install native packages if we have sudo access or are root\n            if env.use_sudo or env.safe_run_output(\"whoami\").strip() == \"root\":\n                _configure_and_install_native_packages(env, pkg_install)\n            else:\n                _connect_native_packages(env, pkg_install, lib_install)\n        if env.nixpkgs:  # ./doc/nixpkgs.md\n            _setup_nix_sources()\n            _nix_packages(pkg_install)\n    if target is None or target == \"custom\":\n        _custom_installs(pkg_install, custom_ignore, custom_add)\n    if target is None or target == \"chef_recipes\":\n        _provision_chef_recipes(pkg_install, custom_ignore)\n    if target is None or target == \"puppet_classes\":\n        _provision_puppet_classes(pkg_install, custom_ignore)\n    if target is None or target == \"brew\":\n        install_brew(flavor=flavor, automated=True)\n    if target is None or target == \"conda\":\n        install_conda(flavor=flavor, automated=True)\n    if target is None or target == \"libraries\":\n        _do_library_installs(lib_install)\n    if target is None or target == \"post_install\":\n        env.flavor.post_install()\n        if \"is_ec2_image\" in env and env.is_ec2_image.upper() in [\"TRUE\", \"YES\"]:\n            _freenx_scripts(env)\n            if pkg_install is not None and 'cloudman' in pkg_install:\n                _configure_cloudman(env)\n    if target is None or target == \"cleanup\":\n        if env.use_sudo:\n            _cleanup_space(env)\n        if \"is_ec2_image\" in env and env.is_ec2_image.upper() in [\"TRUE\", \"YES\"]:\n            _cleanup_ec2(env)\n\ndef _print_time_stats(action, event, prev_time=None):\n    \"\"\" A convenience method for displaying time event during configuration.\n\n    :type action: string\n    :param action: Indicates type of action (eg, Config, Lib install, Pkg install)\n\n    :type event: string\n    :param event: The monitoring event (eg, start, stop)\n\n    :type prev_time: datetime\n    :param prev_time: A timeststamp of a previous event. If provided, duration between\n                      the time the method is called and the time stamp is included in\n                      the printout\n\n    :rtype: datetime\n    :return: A datetime timestamp of when the method was called\n    \"\"\"\n    time = datetime.utcnow()\n    s = \"{0} {1} time: {2}\".format(action, event, time)\n    if prev_time: s += \"; duration: {0}\".format(str(time-prev_time))\n    env.logger.info(s)\n    return time\n\ndef _check_fabric_version():\n    \"\"\"Checks for fabric version installed\n    \"\"\"\n    version = env.version\n    if int(version.split(\".\")[0]) < 1:\n        raise NotImplementedError(\"Please install fabric version 1 or higher\")\n\ndef _custom_installs(to_install, ignore=None, add=None):\n    if not env.safe_exists(env.local_install) and env.local_install:\n        env.safe_run(\"mkdir -p %s\" % env.local_install)\n    pkg_config = get_config_file(env, \"custom.yaml\").base\n    packages, pkg_to_group = _yaml_to_packages(pkg_config, to_install)\n    packages = [p for p in packages if ignore is None or p not in ignore]\n    if add is not None:\n        for key, vals in add.iteritems():\n            for v in vals:\n                pkg_to_group[v] = key\n                packages.append(v)\n    for p in env.flavor.rewrite_config_items(\"custom\", packages):\n        install_custom(p, True, pkg_to_group)\n\n\ndef _provision_chef_recipes(to_install, ignore=None):\n    \"\"\"\n    Much like _custom_installs, read config file, determine what to install,\n    and install it.\n    \"\"\"\n    pkg_config = get_config_file(env, \"chef_recipes.yaml\").base\n    packages, _ = _yaml_to_packages(pkg_config, to_install)\n    packages = [p for p in packages if ignore is None or p not in ignore]\n    recipes = [recipe for recipe in env.flavor.rewrite_config_items(\"chef_recipes\", packages)]\n    if recipes:  # Don't bother running chef if nothing to configure\n        install_chef_recipe(recipes, True)\n\n\ndef _provision_puppet_classes(to_install, ignore=None):\n    \"\"\"\n    Much like _custom_installs, read config file, determine what to install,\n    and install it.\n    \"\"\"\n    pkg_config = get_config_file(env, \"puppet_classes.yaml\").base\n    packages, _ = _yaml_to_packages(pkg_config, to_install)\n    packages = [p for p in packages if ignore is None or p not in ignore]\n    classes = [recipe for recipe in env.flavor.rewrite_config_items(\"puppet_classes\", packages)]\n    if classes:  # Don't bother running chef if nothing to configure\n        install_puppet_class(classes, True)\n\n\ndef install_chef_recipe(recipe, automated=False, flavor=None):\n    \"\"\"Install one or more chef recipes by name.\n\n    Usage: fab [-i key] [-u user] -H host install_chef_recipe:recipe\n\n    :type recipe:  string or list\n    :param recipe: TODO\n\n    :type automated:  bool\n    :param automated: If set to True, the environment is not loaded.\n    \"\"\"\n    _setup_logging(env)\n    if not automated:\n        _configure_fabric_environment(env, flavor)\n\n    time_start = _print_time_stats(\"Chef provision for recipe(s) '{0}'\".format(recipe), \"start\")\n    _configure_chef(env, chef)\n    recipes = recipe if isinstance(recipe, list) else [recipe]\n    for recipe_to_add in recipes:\n        chef.add_recipe(recipe_to_add)\n    _chef_provision(env, recipes)\n    _print_time_stats(\"Chef provision for recipe(s) '%s'\" % recipe, \"end\", time_start)\n\n\ndef install_puppet_class(classes, automated=False, flavor=None):\n    \"\"\"Install one or more puppet classes by name.\n\n    Usage: fab [-i key] [-u user] -H host install_puppet_class:class\n\n    :type classes:  string or list\n    :param classes: TODO\n\n    :type automated:  bool\n    :param automated: If set to True, the environment is not loaded.\n    \"\"\"\n    _setup_logging(env)\n    if not automated:\n        _configure_fabric_environment(env, flavor)\n\n    time_start = _print_time_stats(\"Puppet provision for class(es) '{0}'\".format(classes), \"start\")\n    classes = classes if isinstance(classes, list) else [classes]\n    _puppet_provision(env, classes)\n    _print_time_stats(\"Puppet provision for classes(s) '%s'\" % classes, \"end\", time_start)\n\n\ndef install_custom(p, automated=False, pkg_to_group=None, flavor=None):\n    \"\"\"\n    Install a single custom program or package by name.\n\n    This method fetches program name from ``config/custom.yaml`` and delegates\n    to a method in ``custom/*name*.py`` to proceed with the installation.\n    Alternatively, if a program install method is defined in the appropriate\n    package, it will be called directly (see param ``p``).\n\n    Usage: fab [-i key] [-u user] -H host install_custom:program_name\n\n    :type p:  string\n    :param p: A name of the custom program to install. This has to be either a name\n              that is listed in ``custom.yaml`` as a subordinate to a group name or a\n              program name whose install method is defined in either ``cloudbio`` or\n              ``custom`` packages\n              (e.g., ``cloudbio/custom/cloudman.py -> install_cloudman``).\n\n    :type automated:  bool\n    :param automated: If set to True, the environment is not loaded and reading of\n                      the ``custom.yaml`` is skipped.\n    \"\"\"\n    p = p.lower() # All packages listed in custom.yaml are in lower case\n    if not automated:\n        _setup_logging(env)\n        _configure_fabric_environment(env, flavor, ignore_distcheck=True)\n        pkg_config = get_config_file(env, \"custom.yaml\").base\n        packages, pkg_to_group = _yaml_to_packages(pkg_config, None)\n    time_start = _print_time_stats(\"Custom install for '{0}'\".format(p), \"start\")\n    fn = _custom_install_function(env, p, pkg_to_group)\n    fn(env)\n    ## TODO: Replace the previous 4 lines with the following one, barring\n    ## objections. Slightly different behavior because pkg_to_group will be\n    ## loaded regardless of automated if it is None, but IMO this shouldn't\n    ## matter because the following steps look like they would fail if\n    ## automated is True and pkg_to_group is None.\n    # _install_custom(p, pkg_to_group)\n    _print_time_stats(\"Custom install for '%s'\" % p, \"end\", time_start)\n\n\ndef _install_custom(p, pkg_to_group=None):\n    if pkg_to_group is None:\n        pkg_config = get_config_file(env, \"custom.yaml\").base\n        packages, pkg_to_group = _yaml_to_packages(pkg_config, None)\n    fn = _custom_install_function(env, p, pkg_to_group)\n    fn(env)\n\ndef install_brew(p=None, version=None, flavor=None, automated=False):\n    \"\"\"Top level access to homebrew/linuxbrew packages.\n    p is a package name to install, or all configured packages if not specified.\n    \"\"\"\n    if not automated:\n        _setup_logging(env)\n        _configure_fabric_environment(env, flavor, ignore_distcheck=True)\n    if p is not None:\n        if version:\n            p = \"%s==%s\" % (p, version)\n        brew.install_packages(env, packages=[p])\n    else:\n        pkg_install = _read_main_config()[0]\n        brew.install_packages(env, to_install=pkg_install)\n\ndef install_conda(p=None, flavor=None, automated=False):\n    if not automated:\n        _setup_logging(env)\n        _configure_fabric_environment(env, flavor, ignore_distcheck=True)\n    if p is not None:\n        conda.install_packages(env, packages=[p])\n    else:\n        pkg_install = _read_main_config()[0]\n        conda.install_packages(env, to_install=pkg_install)\n\ndef _custom_install_function(env, p, pkg_to_group):\n    \"\"\"\n    Find custom install function to execute based on package name to\n    pkg_to_group dict.\n    \"\"\"\n    try:\n        # Allow direct calling of a program install method, even if the program\n        # is not listed in the custom list (ie, not contained as a key value in\n        # pkg_to_group). For an example, see 'install_cloudman' or use p=cloudman.\n        mod_name = pkg_to_group[p] if p in pkg_to_group else p\n        env.logger.debug(\"Importing module cloudbio.custom.%s\" % mod_name)\n        mod = __import__(\"cloudbio.custom.%s\" % mod_name,\n                         fromlist=[\"cloudbio\", \"custom\"])\n    except ImportError:\n        raise ImportError(\"Need to write module cloudbio.custom.%s\" %\n                pkg_to_group[p])\n    replace_chars = [\"-\"]\n    try:\n        for to_replace in replace_chars:\n            p = p.replace(to_replace, \"_\")\n        env.logger.debug(\"Looking for custom install function %s.install_%s\"\n            % (mod.__name__, p))\n        fn = getattr(mod, \"install_%s\" % p)\n    except AttributeError:\n        raise ImportError(\"Need to write a install_%s function in custom.%s\"\n                % (p, pkg_to_group[p]))\n    return fn\n\n\ndef _read_main_config():\n    \"\"\"Pull a list of groups to install based on our main configuration YAML.\n\n    Reads 'main.yaml' and returns packages and libraries\n    \"\"\"\n    yaml_file = get_config_file(env, \"main.yaml\").base\n    with open(yaml_file) as in_handle:\n        full_data = yaml.safe_load(in_handle)\n    packages = full_data.get('packages', [])\n    packages = env.flavor.rewrite_config_items(\"main_packages\", packages)\n    libraries = full_data.get('libraries', [])\n    custom_ignore = full_data.get('custom_ignore', [])\n    custom_add = full_data.get(\"custom_additional\")\n    if packages is None: packages = []\n    if libraries is None: libraries = []\n    if custom_ignore is None: custom_ignore = []\n    env.logger.info(\"Meta-package information from {2}\\n- Packages: {0}\\n- Libraries: \"\n            \"{1}\".format(\",\".join(packages), \",\".join(libraries), yaml_file))\n    return packages, sorted(libraries), custom_ignore, custom_add\n\n# ### Library specific installation code\n\ndef _python_library_installer(config):\n    \"\"\"Install python specific libraries using pip, conda and easy_install.\n    Handles using isolated anaconda environments.\n    \"\"\"\n    if shared._is_anaconda(env):\n        conda_bin = shared._conda_cmd(env)\n        for pname in env.flavor.rewrite_config_items(\"python\", config.get(\"conda\", [])):\n            env.safe_run(\"{0} install --yes {1}\".format(conda_bin, pname))\n        cmd = env.safe_run\n        with settings(warn_only=True):\n            cmd(\"%s -U distribute\" % os.path.join(os.path.dirname(conda_bin), \"easy_install\"))\n    else:\n        pip_bin = shared._pip_cmd(env)\n        ei_bin = pip_bin.replace(\"pip\", \"easy_install\")\n        env.safe_sudo(\"%s -U pip\" % ei_bin)\n        with settings(warn_only=True):\n            env.safe_sudo(\"%s -U distribute\" % ei_bin)\n        cmd = env.safe_sudo\n    for pname in env.flavor.rewrite_config_items(\"python\", config['pypi']):\n        cmd(\"{0} install --upgrade {1} --allow-unverified {1} --allow-external {1}\".format(shared._pip_cmd(env), pname)) # fixes problem with packages not being in pypi\n\ndef _ruby_library_installer(config):\n    \"\"\"Install ruby specific gems.\n    \"\"\"\n    gem_ext = getattr(env, \"ruby_version_ext\", \"\")\n    def _cur_gems():\n        with settings(\n                hide('warnings', 'running', 'stdout', 'stderr')):\n            gem_info = env.safe_run_output(\"gem%s list --no-versions\" % gem_ext)\n        return [l.rstrip(\"\\r\") for l in gem_info.split(\"\\n\") if l.rstrip(\"\\r\")]\n    installed = _cur_gems()\n    for gem in env.flavor.rewrite_config_items(\"ruby\", config['gems']):\n        # update current gems only to check for new installs\n        if gem not in installed:\n            installed = _cur_gems()\n        if gem in installed:\n            env.safe_sudo(\"gem%s update %s\" % (gem_ext, gem))\n        else:\n            env.safe_sudo(\"gem%s install %s\" % (gem_ext, gem))\n\ndef _perl_library_installer(config):\n    \"\"\"Install perl libraries from CPAN with cpanminus.\n    \"\"\"\n    with shared._make_tmp_dir() as tmp_dir:\n        with cd(tmp_dir):\n            env.safe_run(\"wget --no-check-certificate -O cpanm \"\n                         \"https://raw.github.com/miyagawa/cpanminus/master/cpanm\")\n            env.safe_run(\"chmod a+rwx cpanm\")\n            env.safe_sudo(\"mv cpanm %s/bin\" % env.system_install)\n    sudo_str = \"--sudo\" if env.use_sudo else \"\"\n    for lib in env.flavor.rewrite_config_items(\"perl\", config['cpan']):\n        # Need to hack stdin because of some problem with cpanminus script that\n        # causes fabric to hang\n        # http://agiletesting.blogspot.com/2010/03/getting-past-hung-remote-processes-in.html\n        env.safe_run(\"cpanm %s --skip-installed --notest %s < /dev/null\" % (sudo_str, lib))\n\ndef _haskell_library_installer(config):\n    \"\"\"Install haskell libraries using cabal.\n    \"\"\"\n    run(\"cabal update\")\n    for lib in config[\"cabal\"]:\n        sudo_str = \"--root-cmd=sudo\" if env.use_sudo else \"\"\n        env.safe_run(\"cabal install %s --global %s\" % (sudo_str, lib))\n\nlib_installers = {\n    \"r-libs\" : libraries.r_library_installer,\n    \"python-libs\" : _python_library_installer,\n    \"ruby-libs\" : _ruby_library_installer,\n    \"perl-libs\" : _perl_library_installer,\n    \"haskell-libs\": _haskell_library_installer,\n    }\n\ndef install_libraries(language):\n    \"\"\"High level target to install libraries for a specific language.\n    \"\"\"\n    _setup_logging(env)\n    _check_fabric_version()\n    _configure_fabric_environment(env, ignore_distcheck=True)\n    _do_library_installs([\"%s-libs\" % language])\n\ndef _do_library_installs(to_install):\n    for iname in to_install:\n        yaml_file = get_config_file(env, \"%s.yaml\" % iname).base\n        with open(yaml_file) as in_handle:\n            config = yaml.safe_load(in_handle)\n        lib_installers[iname](config)\n"
  },
  {
    "path": "ggd-recipes/BDGP6/gtf.yaml",
    "content": "# Ensembl GTF file distributed by Ensembl for BDGP6\n---\nattributes:\n  name: gtf\n  version: 95\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=http://ftp.ensembl.org/pub/release-95/gtf/drosophila_melanogaster/Drosophila_melanogaster.BDGP6.95.gtf.gz\n        mkdir -p rnaseq\n        wget --no-check-certificate -qO- $url | gunzip -c | rnaseq/BDGP6.gtf\n    recipe_outfiles:\n    - rnaseq/BDGP6.gtf\n"
  },
  {
    "path": "ggd-recipes/BDGP6/mirbase.yaml",
    "content": "# small RNA seq annotation from different sources\n---\nattributes:\n  name: srnaseq\n  version: 20200302\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p srnaseq\n        cd srnaseq\n        wget --random-wait --retry-connrefused -nv -c -O tmp.gtf.gz ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/001/215/GCF_000001215.4_Release_6_plus_ISO1_MT/GCF_000001215.4_Release_6_plus_ISO1_MT_genomic.gff.gz\n        zgrep -v exon tmp.gtf.gz | grep -v region | sed 's/Name/name/g' | sed -e 's/NC_004353.4/chr4/g; s/NC_004354.4/chrX/g; s/NC_024511.2/chrM/g; s/NC_024512.1/chrY/g; s/NT_033777.3/chr3R/g; s/NT_033778.4/chr2R/g; s/NT_033779.5/chr2L/g; s/NT_037436.4/chr3L/g; s/NW_007931121.1/CP007120.1/g' | sed 's/=/ /g' > srna-transcripts.gtf\n        # mirbase\n        wget --random-wait --retry-connrefused -nv -c -O hairpin.t.fa https://mirbase.org/download/CURRENT/hairpin.fa\n        cat hairpin.t.fa | awk '{if ($0~/>dme/){name=$0; print name} else if ($0~/^>/){name=0};if (name!=0 && $0!~/^>/){print $0;}}' | sed 's/U/T/g'  > hairpin.fa\n        rm hairpin.t.fa\n        wget --random-wait --retry-connrefused -nv -c -O mature.t.fa https://mirbase.org/download/CURRENT/mature.fa\n        cat mature.t.fa | awk '{if ($0~/>dme/){name=$0; print name} else if ($0~/^>/){name=0};if (name!=0 && $0!~/^>/){print $0;}}' | sed 's/U/T/g'  > mature.fa\n        rm mature.t.fa\n        wget --random-wait --retry-connrefused -nv -c -O miRNA.t.str https://mirbase.org/download/CURRENT/miRNA.dat\n        cat miRNA.t.str | awk '{if ($0~/dme/)print $0}' > miRNA.str\n        rm miRNA.t.str\n        wget --random-wait --retry-connrefused -nv -c -O mirbase.gff3 https://mirbase.org/download/dme.gff3\n        wget --no-check-certificate -c -O Rfam_for_miRDeep.fa.gz https://github.com/lpantano/mirdeep2_core/raw/data/Rfam_for_miRDeep.fa.gz && gunzip -f Rfam_for_miRDeep.fa.gz\n        # targetscan analysis\n        wget  --random-wait --retry-connrefused --no-check-certificate -nv -c -O Summary_Counts.txt.zip http://www.targetscan.org/fly_72/fly_72_data_download/Summary_Counts.all_predictions.txt.zip && unzip Summary_Counts.txt.zip\n        wget  --random-wait --retry-connrefused --no-check-certificate -nv -c -O miR_Family_Info.txt.zip http://www.targetscan.org/fly_72/fly_72_data_download/miR_Family_Info.txt.zip && unzip miR_Family_Info.txt.zip\n        wget  --random-wait --retry-connrefused --no-check-certificate -nv -N -c ftp://mirbase.org/pub/mirbase/22/database_files/mirna_mature.txt.gz\n    recipe_outfiles:\n        - srnaseq/srna-transcripts.gtf\n        - srnaseq/hairpin.fa\n        - srnaseq/mature.fa\n        - srnaseq/miRNA.str\n        - srnaseq/mirbase.gff3\n        - srnaseq/Rfam_for_miRDeep.fa\n        - srnaseq/Summary_Counts.all_predictions.txt\n        - srnaseq/miR_Family_Info.txt\n        - srnaseq/mirna_mature.txt.gz\n"
  },
  {
    "path": "ggd-recipes/BDGP6/seq.yaml",
    "content": "# Drosophila BDGP6 genome distributed by Ensembl\n---\nattributes:\n  name: seq\n  version: BDGP6\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p seq\n        url=http://ftp.ensembl.org/pub/release-95/fasta/drosophila_melanogaster/dna/Drosophila_melanogaster.BDGP6.dna.toplevel.fa.gz\n        wget -c -O seq/BDGP6.fa.gz ${url}\n        gunzip -c seq/BDGP6.fa.gz > seq/BDGP6.fa\n    recipe_outfiles:\n      - seq/BDGP6.fa\n      - seq/BDGP6.fa.gz\n"
  },
  {
    "path": "ggd-recipes/BDGP6/transcripts.yaml",
    "content": "# Transcript, annotation and indices for RNA-seq analysis\n# Prepared from Ensembl transcripts using: \n# https://github.com/chapmanb/cloudbiolinux/blob/master/utils/prepare_tx_gff.py\n# Prepared against Ensembl gene build 95\n---\nattributes:\n  name: transcripts\n  version: 2019-03-26_95\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/biodata/annotation/BDGP6-rnaseq-2019-03-26_95.tar.xz\n        wget -c -N --no-check-certificate $baseurl\n        xz -dc *-rnaseq-*.tar.xz | tar -xpf -\n        mv */rnaseq-* rnaseq\n    recipe_outfiles:\n      - rnaseq/ref-transcripts.bed\n      - rnaseq/ref-transcripts.dexseq.gff3\n      - rnaseq/ref-transcripts.fa\n      - rnaseq/ref-transcripts.genePred\n      - rnaseq/ref-transcripts.gtf\n      - rnaseq/ref-transcripts.gtf.db\n      - rnaseq/ref-transcripts-mask.gtf\n      - rnaseq/ref-transcripts.refFlat\n      - rnaseq/ref-transcripts-splicesites.txt\n      - rnaseq/rRNA.gtf\n      - rnaseq/rRNA.gtf.db\n      - rnaseq/rRNA.interval_list\n      - rnaseq/tx2gene.csv\n      - rnaseq/version.txt\n"
  },
  {
    "path": "ggd-recipes/GRCh37/1000g.yaml",
    "content": "# 1000 genomes\n---\nattributes:\n  name: 1000g\n  version: phase3_shapeit2_mvncall_integrated_v5a.20130502\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/gemini-annotations\n        url=$baseurl/ALL.wgs.phase3_shapeit2_mvncall_integrated_v5a.20130502.sites.tidy.vcf.gz\n        mkdir -p variation\n        wget -c -O variation/1000g.vcf.gz $url\n        wget -c -O variation/1000g.vcf.gz.tbi $url.tbi\n    recipe_outfiles:\n      - variation/1000g.vcf.gz\n      - variation/1000g.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/GRCh37/1000g_omni_snps.yaml",
    "content": "---\nattributes:\n  name: 1000g_omni_snps\n  version: 2.5\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=ftp://gsapubftp-anonymous:none@ftp.broadinstitute.org/bundle/b37/1000G_omni2.5.b37.vcf.gz\n        mkdir -p variation\n        cd variation\n        wget -O - $baseurl | gunzip -c | bgzip -c > 1000G_omni2.5.vcf.gz\n        tabix -f -p vcf  1000G_omni2.5.vcf.gz\n    recipe_outfiles:\n      - variation/1000G_omni2.5.vcf.gz\n      - variation/1000G_omni2.5.vcf.gz.tbi\n \n"
  },
  {
    "path": "ggd-recipes/GRCh37/1000g_snps.yaml",
    "content": "---\nattributes:\n  name: 1000g_snps\n  version: 2.8\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=ftp://gsapubftp-anonymous:none@ftp.broadinstitute.org/bundle/b37/1000G_phase1.snps.high_confidence.b37.vcf.gz\n        mkdir -p variation\n        cd variation\n        wget -O - $baseurl | gunzip -c | bgzip -c > 1000G_phase1.snps.high_confidence.vcf.gz\n        tabix -f -p vcf 1000G_phase1.snps.high_confidence.vcf.gz\n    recipe_outfiles:\n      - variation/1000G_phase1.snps.high_confidence.vcf.gz\n      - variation/1000G_phase1.snps.high_confidence.vcf.gz.tbi\n \n"
  },
  {
    "path": "ggd-recipes/GRCh37/ACMG56_genes.yaml",
    "content": "# ACMG regions: http://iobio.io/2016/03/29/acmg56/\n# https://github.com/lpantano/custom_genome_annotation\n---\nattributes:\n  name: ACMG56_genes\n  version: 20160810\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://raw.githubusercontent.com/lpantano/custom_genome_annotation/master/acmg56.GRCh37.bed\n        dir=coverage/prioritize/actionable\n        mkdir -p $dir\n        cd $dir\n        wget --no-check-certificate -c -O ACMG56.bed $baseurl\n    recipe_outfiles:\n      - coverage/prioritize/actionable/ACMG56.bed\n"
  },
  {
    "path": "ggd-recipes/GRCh37/GA4GH_problem_regions.yaml",
    "content": "# GA4GH problematic region files: https://drive.google.com/folderview?id=0B7Ao1qqJJDHQUjVIN3liUUZNWjg&usp=sharing#list\n# Cached mappability regions\n# wget --no-check-certificate -O - http://hgdownload.cse.ucsc.edu/goldenPath/hg19/encodeDCC/wgEncodeMapability/wgEncodeDacMapabilityConsensusExcludable.bed.gz | gunzip -c | sed \"s/^chrM/MT/g\" | sed \"s/^chr//g\" | bgzip -c > $encode/wgEncodeDacMapabilityConsensusExcludable.bed.gz\n# tabix -p bed -f $encode/wgEncodeDacMapabilityConsensusExcludable.bed.gz\n# GC regions for Hartwig tools: https://resources.hartwigmedicalfoundation.nl/\n---\nattributes:\n  name: GA4GH_problem_regions\n  version: 20181016\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=http://bcbio_nextgen.s3.amazonaws.com/GA4GH_problem_regions.zip\n        mkdir -p coverage/problem_regions/GA4GH\n        cd coverage/problem_regions/GA4GH\n        wget --no-check-certificate -c -O GA4GH_problem_regions.zip $baseurl\n        unzip -o GA4GH_problem_regions.zip\n        for file in *.bed\n        do\n          sed 's/^chr//g'  $file > $file.tmp\n          mv $file.tmp $file\n          bgzip $file\n          tabix -p bed $file.gz\n        done\n        cd ../../..\n        encode=coverage/problem_regions/ENCODE\n        mkdir -p $encode\n        cd $encode\n        wget --no-check-certificate http://s3.amazonaws.com/biodata/coverage/problem_regions/ENCODE/GRCh37/wgEncodeDacMapabilityConsensusExcludable.bed.gz\n        wget --no-check-certificate http://s3.amazonaws.com/biodata/coverage/problem_regions/ENCODE/GRCh37/wgEncodeDacMapabilityConsensusExcludable.bed.gz.tbi\n        cd ../../..\n        repeats=coverage/problem_regions/repeats\n        mkdir -p $repeats\n        wget --no-check-certificate -O - https://raw.githubusercontent.com/chapmanb/delly/master/human.hg19.excl.tsv | grep -v ^chr > $repeats/sv_repeat_telomere_centromere.bed\n        wget --no-check-certificate -O - https://github.com/lh3/varcmp/raw/master/scripts/LCR-hs37d5.bed.gz | gunzip -c  | bgzip -c > $repeats/LCR.bed.gz\n        tabix -p bed -f $repeats/LCR.bed.gz\n        wget --no-check-certificate -O $repeats/polyx.bed.gz http://s3.amazonaws.com/biodata/coverage/problem_regions/polyx/polyX-GRCh37.bed.gz\n        wget --no-check-certificate -O $repeats/polyx.bed.gz.tbi http://s3.amazonaws.com/biodata/coverage/problem_regions/polyx/polyX-GRCh37.bed.gz.tbi\n        gc=coverage/gc\n        mkdir -p $gc\n        wget --no-check-certificate -O - https://s3.amazonaws.com/biodata/collections/GRCh37/coverage/gc/GC_profile.1000bp.cnp > $gc/GC_profile.1000bp.cnp\n    recipe_outfiles:\n      - coverage/problem_regions/GA4GH/README.md\n      - coverage/problem_regions/GA4GH/bad_promoter.bed.gz\n      - coverage/problem_regions/GA4GH/bad_promoter.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/gc15.bed.gz\n      - coverage/problem_regions/GA4GH/gc15.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/gc15to20.bed.gz\n      - coverage/problem_regions/GA4GH/gc15to20.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/gc20to25.bed.gz\n      - coverage/problem_regions/GA4GH/gc20to25.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/gc25to30.bed.gz\n      - coverage/problem_regions/GA4GH/gc25to30.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/gc65to70.bed.gz\n      - coverage/problem_regions/GA4GH/gc65to70.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/gc70to75.bed.gz\n      - coverage/problem_regions/GA4GH/gc70to75.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/gc75to80.bed.gz\n      - coverage/problem_regions/GA4GH/gc75to80.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/gc80to85.bed.gz\n      - coverage/problem_regions/GA4GH/gc80to85.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/gc85.bed.gz\n      - coverage/problem_regions/GA4GH/gc85.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/heng_um75-hs37d5.bed.gz\n      - coverage/problem_regions/GA4GH/heng_um75-hs37d5.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/low_complexity_51to200bp.bed.gz\n      - coverage/problem_regions/GA4GH/low_complexity_51to200bp.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/low_complexity_gt200bp.bed.gz\n      - coverage/problem_regions/GA4GH/low_complexity_gt200bp.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/self_chain.bed.gz\n      - coverage/problem_regions/GA4GH/self_chain.bed.gz.tbi\n      - coverage/problem_regions/ENCODE/wgEncodeDacMapabilityConsensusExcludable.bed.gz\n      - coverage/problem_regions/ENCODE/wgEncodeDacMapabilityConsensusExcludable.bed.gz.tbi\n      - coverage/problem_regions/repeats/sv_repeat_telomere_centromere.bed\n      - coverage/problem_regions/repeats/LCR.bed.gz\n      - coverage/problem_regions/repeats/LCR.bed.gz.tbi\n      - coverage/problem_regions/repeats/polyx.bed.gz\n      - coverage/problem_regions/repeats/polyx.bed.gz.tbi\n      - coverage/gc/GC_profile.1000bp.cnp\n"
  },
  {
    "path": "ggd-recipes/GRCh37/GRCh37_NCBI2ensembl.txt",
    "content": "NC_000001.10\t1\nNC_000002.11\t2\nNC_000003.11\t3\nNC_000004.11\t4\nNC_000005.9\t5\nNC_000006.11\t6\nNC_000007.13\t7\nNC_000008.10\t8\nNC_000009.11\t9\nNC_000010.10\t10\nNC_000011.9\t11\nNC_000012.11\t12\nNC_000013.10\t13\nNC_000014.8\t14\nNC_000015.9\t15\nNC_000016.9\t16\nNC_000017.10\t17\nNC_000018.9\t18\nNC_000019.9\t19\nNC_000020.10\t20\nNC_000021.8\t21\nNC_000022.10\t22\nNC_000023.10\tX\nNC_000024.9\tY\nNC_012920.1\tMT\n"
  },
  {
    "path": "ggd-recipes/GRCh37/MIG.yaml",
    "content": "# Medically Interpretable Genes, compiled by Personalis: http://www.genomemedicine.com/content/pdf/s13073-015-0197-4.pdf\n---\nattributes:\n  name: MIG\n  version: 20150730\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=http://bcbio_nextgen.s3.amazonaws.com/MIG.zip\n        mkdir -p prioritization\n        cd prioritization\n        wget -c -O MIG.zip $baseurl\n        unzip MIG.zip\n        file=MIG.bed\n        sed 's/^chr//g' $file > $file.tmp\n        mv $file.tmp $file\n        cd ..\n    recipe_outfiles:\n      - prioritization/MIG.bed\n"
  },
  {
    "path": "ggd-recipes/GRCh37/RADAR.yaml",
    "content": "# RADAR RNA editing events:\n#\t(http://lilab.stanford.edu/GokulR/database/Human_AG_all_hg19_v2.txt)\n# installs a BED file of the RADAR events, converted to GRCh37 chromosome\n# coordinates\n---\nattributes:\n  name: RADAR\n  version: v2-20180202\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=https://s3.amazonaws.com/biodata/annotation/RADAR/GRCh37/RADAR.bed.gz\n        mkdir -p editing\n        cd editing\n        wget --no-check-certificate -q $url\n        wget --no-check-certificate -q $url.tbi\n    recipe_outfiles:\n      - editing/RADAR.bed.gz\n      - editing/RADAR.bed.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/GRCh37/af_only_gnomad.yaml",
    "content": "# af_only_gnomad file for t-only variant calling with mutect2\n# From the Broad bundle:\n# ftp://gsapubftp-anonymous@ftp.broadinstitute.org/bundle/Mutect2/af-only-gnomad.raw.sites.b37.vcf.gz\n---\nattributes:\n  name: af_only_gnomad\n  version: 20170602\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=ftp://gsapubftp-anonymous@ftp.broadinstitute.org/bundle/Mutect2/\n        base=af-only-gnomad.raw.sites.b37\n        new=af_only_gnomad\n        mkdir -p variation\n        for suffix in .vcf.gz .vcf.gz.tbi\n        do\n          [[ -f variation/$new$suffix ]] || wget --no-check-certificate -c -O variation/$new$suffix $url/$base$suffix\n        done\n    recipe_outfiles:\n      - variation/af_only_gnomad.vcf.gz\n      - variation/af_only_gnomad.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/GRCh37/ancestral.yaml",
    "content": "---\nattributes:\n  name: ancestral\n  version: 20141010\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/bcbio_nextgen/human_ancestor.fa.gz\n        mkdir -p variation\n        cd variation\n        wget --no-check-certificate -c $baseurl\n        wget --no-check-certificate -c $baseurl.fai\n        wget --no-check-certificate -c $baseurl.gzi\n    recipe_outfiles:\n        - variation/human_ancestor.fa.gz\n        - variation/human_ancestor.fa.gz.fai\n        - variation/human_ancestor.fa.gz.gzi\n \n"
  },
  {
    "path": "ggd-recipes/GRCh37/battenberg.yaml",
    "content": "# Battenberg CNV caller input data:\n# https://github.com/cancerit/cgpBattenberg\n---\nattributes:\n  name: battenberg\n  version: v3-20150610\nrecipe:\n  full:\n    required:\n      - download_generate_bberg_ref_files.pl\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        proburl=https://github.com/cancerit/cgpBattenberg/raw/dev/perl/share/battenberg/probloci.txt.gz\n        impute_info=battenberg/impute/impute_info.txt\n        mkdir -p battenberg\n        [[ -f $impute_info ]] || download_generate_bberg_ref_files.pl -c -o `pwd`/battenberg\n        # Change references to point to final directory\n        sed -i 's/txtmp\\///g' $impute_info\n\n        wget --no-check-certificate -c -O battenberg/probloci.txt.gz $proburl\n        gunzip battenberg/probloci.txt.gz\n    recipe_outfiles:\n      - battenberg/probloci.txt\n      - battenberg/1000genomesloci\n      - battenberg/impute\n"
  },
  {
    "path": "ggd-recipes/GRCh37/capture_regions.yaml",
    "content": "# Common sets of capture region BED files for different platforms\n---\nattributes:\n  name: capture_regions\n  version: 20161202\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://raw.githubusercontent.com/AstraZeneca-NGS/reference_data/master/GRCh37/bed\n        files=\"Exome-Agilent_V5_Clinical.bed Exome-Agilent_V6.bed Exome-IDT_V1.bed\n               Exome-MedExome.bed Exome-NGv3.bed\"\n        mkdir -p coverage/capture_regions\n        cd coverage/capture_regions\n        for fname in $files\n        do\n          wget --no-check-certificate -c $baseurl/$fname\n        done\n    recipe_outfiles:\n      - coverage/capture_regions/Exome-Agilent_V5_Clinical.bed\n      - coverage/capture_regions/Exome-Agilent_V6.bed\n      - coverage/capture_regions/Exome-IDT_V1.bed\n      - coverage/capture_regions/Exome-MedExome.bed\n      - coverage/capture_regions/Exome-NGv3.bed\n"
  },
  {
    "path": "ggd-recipes/GRCh37/clinvar.yaml",
    "content": "# ClinVar: http://www.clinvar.com/\n# UCSFify name sed magic from: https://github.com/mmarchin/utilities/blob/master/ucscify.sh\n---\nattributes:\n  name: clinvar\n  version: 20210110\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        release=20210110\n        # the latest version\n        baseurl=https://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh37/clinvar.vcf.gz\n        mkdir -p variation\n        wget --no-check-certificate -c -O variation/clinvar-orig.vcf.gz $baseurl\n        # remove old files\n        rm -f variation/clinvar.vcf.gz\n        rm -f variation/clinvar.vcf.gz.tbi\n        zcat variation/clinvar-orig.vcf.gz | bgzip -c > variation/clinvar.vcf.gz\n        tabix -f -p vcf variation/clinvar.vcf.gz\n    recipe_outfiles:\n      - variation/clinvar.vcf.gz\n      - variation/clinvar.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/GRCh37/cosmic.yaml",
    "content": "---\nattributes:\n  name: cosmic\n  version: 68-20180114\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/biodata/variants/cosmic-v68-GRCh37.vcf.gz\n        mkdir -p variation\n        cd variation\n        wget --no-check-certificate -c -O cosmic.vcf.gz $baseurl\n        wget --no-check-certificate -c -O cosmic.vcf.gz.tbi $baseurl.tbi\n    recipe_outfiles:\n        - variation/cosmic.vcf.gz\n        - variation/cosmic.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/GRCh37/dbnsfp.yaml",
    "content": "# google drive downloads are unreliable - too many requests, try again in 24h\n# using ftp ftp://dbnsfp:dbnsfp@dbnsfp.softgenetics.com/dbNSFP4.1a.zip\n# bumped version because 3.5a is not supported anymore - 25G instead of 16G\n---\nattributes:\n  name: dbnsfp\n  version: 4.1a\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p variation\n        cd variation\n        wget -c ftp://dbnsfp:dbnsfp@dbnsfp.softgenetics.com/dbNSFP4.1a.zip\n        UNPACK_DIR=`pwd`/tmpunpack\n        if [ ! -f dbNSFP.txt.gz ]; then\n          mkdir -p $UNPACK_DIR\n          7z e dbNSFP*.zip \"dbNSFP*_variant.chrM.gz\"\n          gunzip dbNSFP*_variant.chrM.gz\n          head -n1 dbNSFP*_variant.chrM > $UNPACK_DIR/header.txt\n          rm dbNSFP*_variant.chrM\n          # unzip only files with chromosomal info, eg. skip genes and readme.\n          # [awk] check if hg19 position exists, if so, move data to col 1 and 2, then print\n          # and sort on first and second column\n          unzip -p dbNSFP*.zip \"dbNSFP*_variant.chr*\" | gunzip -c | grep -v '^#chr' | awk -F $'\\t' 'BEGIN {OFS = FS} {if ($8 != \".\" && $9 != \".\") {$1=$8;$2=$9;print}}' | sort -T $UNPACK_DIR -k1,1 -k2,2n | cat $UNPACK_DIR/header.txt - | bgzip -c > dbNSFP.txt.gz\n          #extract readme file, used by VEP plugin to add vcf header info\n          unzip -p dbNSFP*.zip \"*readme.txt\" > dbNSFP.readme.txt\n        fi\n        # index in position 1 and 2\n        tabix -f -s 1 -b 2 -e 2 -c '#' dbNSFP.txt.gz\n        rm -f $UNPACK_DIR/* && rmdir $UNPACK_DIR\n        rm -f dbNSFP*.zip\n    recipe_outfiles:\n      - variation/dbNSFP.txt.gz\n      - variation/dbNSFP.txt.gz.tbi\n      - variation/dbNSFP.readme.txt\n"
  },
  {
    "path": "ggd-recipes/GRCh37/dbscsnv.yaml",
    "content": "---\nattributes:\n  name: dbscsnv\n  version: 1.1\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=ftp://dbnsfp:dbnsfp@dbnsfp.softgenetics.com/dbscSNV1.1.zip\n        mkdir -p variation\n        cd variation\n        wget -c -N $baseurl\n        if [ ! -f dbscSNV.txt.gz ]; then\n          UNPACK_DIR=`pwd`/tmp-unpack\n          mkdir -p $UNPACK_DIR\n          unzip -p dbscSNV*.zip \"dbscSNV*.chr*\" | \\\n          awk -F $'\\t' '\n            BEGIN {OFS = FS} \n            (FNR==1){print} \n            ($1~/^chr/){next} \n            {print | \"sort -T \"UNPACK_DIR\" -k1,1 -k2,2n\"}\n          ' \"UNPACK_DIR=$UNPACK_DIR\" | \\\n            bgzip -c > dbscSNV.txt.gz\n        fi\n        tabix -s 1 -b 2 -e 2 -S 1 dbscSNV.txt.gz\n        rm -f $UNPACK_DIR/* && rmdir $UNPACK_DIR\n        rm -f dbscSNV*.zip\n    recipe_outfiles:\n      - variation/dbscSNV.txt.gz\n      - variation/dbscSNV.txt.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/GRCh37/dbsnp.yaml",
    "content": "# UCSFify name sed magic from: https://github.com/mmarchin/utilities/blob/master/ucscify.sh\n# https://github.com/dpryan79/ChromosomeMappings/\n---\nattributes:\n  name: dbsnp\n  version: 156-20231016\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        build=156\n        version=GCF_000001405.25\n        url=https://ftp.ncbi.nih.gov/snp/archive/b$build/VCF/$version.gz\n        mkdir -p variation\n        cd variation\n        wget -c -O dbsnp-${build}-orig.vcf.gz $url\n        wget -c -O dbsnp-${build}-orig.vcf.gz.tbi $url.tbi\n        remap_url=https://raw.githubusercontent.com/chapmanb/cloudbiolinux/master/ggd-recipes/GRCh37/GRCh37_NCBI2ensembl.txt\n        wget --no-check-certificate -q -O remap.tsv $remap_url\n        export TMPDIR=`pwd`\n        # remove NW_ and NT_ contigs absent in hg19 and in GRCh37\n        gunzip -c dbsnp-${build}-orig.vcf.gz | grep -v \"^NW_\" | grep -v \"^NT_\" | bgzip -c > dbsnp-${build}.noNW.vcf.gz\n        tabix dbsnp-${build}.noNW.vcf.gz\n        bcftools annotate -Ou --rename-chrs remap.tsv dbsnp-${build}.noNW.vcf.gz | \\\n            bcftools sort -m 1G -Oz -T . -o dbsnp-${build}.vcf.gz\n        tabix -f -p vcf -C dbsnp-${build}.vcf.gz\n        tabix -f -p vcf dbsnp-${build}.vcf.gz\n        ln -sf dbsnp-${build}.vcf.gz dbsnp.vcf.gz\n        ln -sf dbsnp-${build}.vcf.gz.tbi dbsnp.vcf.gz.tbi\n        ln -sf dbsnp-${build}.vcf.gz.csi dbsnp.vcf.gz.csi\n        cd ..\n    recipe_outfiles:\n      - variation/dbsnp.vcf.gz\n      - variation/dbsnp.vcf.gz.csi\n      - variation/dbsnp.vcf.gz.tbi\n      - variation/dbsnp-156.vcf.gz\n      - variation/dbsnp-156.vcf.gz.csi\n      - variation/dbsnp-156.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/GRCh37/dream-syn3.yaml",
    "content": "# Truth sets from the DREAM genomic challenge 3\n# https://www.synapse.org/#!Synapse:syn312572/wiki/62018\n#\n# Originally retrieved from https://www.synapse.org/#!Synapse:syn2177211\n# and prepared with this script:\n# https://github.com/chapmanb/bcbio-nextgen/blob/master/scripts/utils/format_dream_truthset.py\n---\nattributes:\n  name: dream-syn3\n  version: 2014-08-04\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/dream-syn3\n        orig=synthetic_challenge_set3_tumor_20pctmasked_truth\n        mkdir -p $dir\n        wget --no-check-certificate https://s3.amazonaws.com/bcbio_nextgen/dream/synthetic_challenge_set3_tumor_20pctmasked_truth.tar.gz\n        tar -xzvpf synthetic_challenge_set3_tumor_20pctmasked_truth.tar.gz\n        cp ${orig}.vcf.gz $dir/truth_small_variants.vcf.gz\n        cp ${orig}.vcf.gz.tbi $dir/truth_small_variants.vcf.gz.tbi\n        cp ${orig}_regions.bed $dir/truth_regions.bed\n        cp ${orig}_sv_DEL.bed $dir/truth_DEL.bed \n        cp ${orig}_sv_DUP.bed $dir/truth_DUP.bed\n        cp ${orig}_sv_INS.bed $dir/truth_INS.bed\n        cp ${orig}_sv_INV.bed $dir/truth_INV.bed\n    recipe_outfiles:\n      - validation/dream-syn3/truth_small_variants.vcf.gz\n      - validation/dream-syn3/truth_small_variants.vcf.gz.tbi\n      - validation/dream-syn3/truth_regions.bed\n      - validation/dream-syn3/truth_DEL.bed\n      - validation/dream-syn3/truth_DUP.bed\n      - validation/dream-syn3/truth_INS.bed\n      - validation/dream-syn3/truth_INV.bed\n"
  },
  {
    "path": "ggd-recipes/GRCh37/dream-syn4.yaml",
    "content": "# Truth sets from the DREAM genomic challenge 4\n# https://www.synapse.org/#!Synapse:syn312572/wiki/62018\n#\n# Originally retrieved from https://www.synapse.org/#!Synapse:syn2177211\n# and prepared with this script:\n# https://github.com/chapmanb/bcbio-nextgen/blob/master/scripts/utils/format_dream_truthset.py\n---\nattributes:\n  name: dream-syn4\n  version: 2016-06-11\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/dream-syn4\n        orig=synthetic_challenge_set4_tumour_25pctmasked_truth\n        mkdir -p $dir\n        wget --no-check-certificate -c https://s3.amazonaws.com/bcbio_nextgen/dream/synthetic_challenge_set4_tumour_25pctmasked_truth.tar.gz\n        tar -xzvpf ${orig}.tar.gz\n        cp ${orig}.vcf.gz $dir/truth_small_variants.vcf.gz\n        cp ${orig}.vcf.gz.tbi $dir/truth_small_variants.vcf.gz.tbi\n        cp ${orig}_regions.bed $dir/truth_regions.bed\n        cp ${orig}_sv_DEL.bed $dir/truth_DEL.bed \n        cp ${orig}_sv_DUP.bed $dir/truth_DUP.bed\n        cp ${orig}_sv_INV.bed $dir/truth_INV.bed\n    recipe_outfiles:\n      - validation/dream-syn4/truth_small_variants.vcf.gz\n      - validation/dream-syn4/truth_small_variants.vcf.gz.tbi\n      - validation/dream-syn4/truth_regions.bed\n      - validation/dream-syn4/truth_DEL.bed\n      - validation/dream-syn4/truth_DUP.bed\n      - validation/dream-syn4/truth_INV.bed\n"
  },
  {
    "path": "ggd-recipes/GRCh37/ericscript.yaml",
    "content": "---\nattributes:\n  name: ericscript\n  version: 73\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        # Uses Google Drive for faster download with tricks from\n        # http://stackoverflow.com/a/38937732/252589\n        ggID='0B9s__vuJPvIibDRIb0RFdHFlQmM'\n        ggURL='https://drive.google.com/uc?export=download'\n        mkdir -p rnaseq/ericscript\n        cd rnaseq/ericscript\n        filename=\"$(curl -k -sc tmp-gcokie \"${ggURL}&id=${ggID}\" | grep -o '=\"uc-name.*</span>' | sed 's/.*\">//;s/<.a> .*//')\"\n        getcode=\"$(awk '/_warning_/ {print $NF}' tmp-gcokie)\"\n        curl -k -Lb tmp-gcokie \"${ggURL}&confirm=${getcode}&id=${ggID}\" -o \"${filename}\" -C -\n        tar -xjvpf ericscript_db_homosapiens_ensembl73.tar.bz2\n    recipe_outfiles:\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl73/data/homo_sapiens/EnsemblGene.Reference.fa.amb\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl73/data/homo_sapiens/allseq.fa.pac\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl73/data/homo_sapiens/allseq.fa.bwt\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl73/data/homo_sapiens/allseq.fa\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl73/data/homo_sapiens/EnsemblGene.Sequences.RData\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl73/data/homo_sapiens/EnsemblGene.GeneInfo.RData\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl73/data/homo_sapiens/EnsemblGene.Reference.fa\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl73/data/homo_sapiens/EnsemblGene.GenePosition.RData\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl73/data/homo_sapiens/allseq.fa.sa\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl73/data/homo_sapiens/allseq.fa.amb\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl73/data/homo_sapiens/EnsemblGene.Paralogs.RData\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl73/data/homo_sapiens/EnsemblGene.GeneNames.RData\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl73/data/homo_sapiens/allseq.fa.ann\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl73/data/homo_sapiens/EnsemblGene.Reference.fa.bwt\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl73/data/homo_sapiens/EnsemblGene.Reference.fa.sa\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl73/data/homo_sapiens/EnsemblGene.Transcripts.RData\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl73/data/homo_sapiens/EnsemblGene.Reference.fa.ann\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl73/data/homo_sapiens/EnsemblGene.Reference.fa.pac\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl73/data/homo_sapiens/EnsemblGene.Structures.RData\n"
  },
  {
    "path": "ggd-recipes/GRCh37/esp.yaml",
    "content": "# Exome Sequencing Project (ESP): http://evs.gs.washington.edu/EVS/\n---\nattributes:\n  name: esp\n  version: ESP6500SI-V2\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/gemini-annotations\n        url=$baseurl/ESP6500SI.all.snps_indels.tidy.v2.vcf.gz\n        mkdir -p variation\n        wget -c -O variation/esp.vcf.gz $url\n        wget -c -O variation/esp.vcf.gz.tbi $url.tbi\n    recipe_outfiles:\n      - variation/esp.vcf.gz\n      - variation/esp.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/GRCh37/exac.yaml",
    "content": "# ExAC VCF sites: http://exac.broadinstitute.org/\n---\nattributes:\n  name: exac\n  version: 0.3\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/gemini-annotations\n        url=$baseurl/ExAC.r0.3.sites.vep.tidy.vcf.gz\n        mkdir -p variation\n        wget -c -O variation/exac.vcf.gz $url\n        wget -c -O variation/exac.vcf.gz.tbi $url.tbi\n    recipe_outfiles:\n      - variation/exac.vcf.gz\n      - variation/exac.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/GRCh37/fusion-blacklist.yaml",
    "content": "# fusion blacklist regions from arriba\n---\nattributes:\n  name: fusion-blacklist\n  version: 2\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        version=v1.1.0\n        url=https://github.com/suhrig/arriba/releases/download/${version}/arriba_${version}.tar.gz\n        arriba_tarball=arriba_${version}.tar.gz\n        outdir=rnaseq/fusion-blacklist\n        mkdir -p ${outdir}\n        wget --no-check-certificate -c -O ${arriba_tarball} ${url}\n        tar zxvf ${arriba_tarball}\n        mv arriba_${version}/database/blacklist_hg19_hs37d5_GRCh37_2018-11-04.tsv.gz ${outdir}/arriba-blacklist.tsv.gz\n        mv arriba_${version}/database/cytobands_hg19_hs37d5_GRCh37_2018-02-23.tsv ${outdir}/arriba-cytobands.tsv\n        mv arriba_${version}/database/protein_domains_hg19_hs37d5_GRCh37_2018-03-06.gff3 ${outdir}/arriba-protein-domains.gff3\n        mv arriba_${version}/database/protein_domains_hg19_hs37d5_GRCh37_2018-03-06.gff3.idx ${outdir}/arriba-protein-domains.gff3.idx\n        rm -r arriba_${version}\n        rm ${arriba_tarball}\n    recipe_outfiles:\n      - rnaseq/fusion-blacklist/arriba-blacklist.tsv.gz\n      - rnaseq/fusion-blacklist/arriba-cytobands.tsv\n      - rnaseq/fusion-blacklist/arriba-protein-domains.gff3\n      - rnaseq/fusion-blacklist/arriba-protein-domains.gff3.idx\n"
  },
  {
    "path": "ggd-recipes/GRCh37/genesplicer.yaml",
    "content": "---\nattributes:\n  name: genesplicer\n  version: 2004.04.03\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p variation/genesplicer\n        wget ftp://ftp.ccb.jhu.edu/pub/software/genesplicer/GeneSplicer.tar.gz\n        tar -C variation/genesplicer --strip-components=2 -xvf GeneSplicer.tar.gz GeneSplicer/human\n        rm GeneSplicer.tar.gz\n    recipe_outfiles:\n      - variation/genesplicer/acc1.mar\n      - variation/genesplicer/acc14\n      - variation/genesplicer/acc20\n      - variation/genesplicer/acc26\n      - variation/genesplicer/acc33\n      - variation/genesplicer/acc7\n      - variation/genesplicer/chgd.fasta\n      - variation/genesplicer/don1.mar\n      - variation/genesplicer/don15\n      - variation/genesplicer/don19\n      - variation/genesplicer/don20\n      - variation/genesplicer/don24\n      - variation/genesplicer/embls.seq\n      - variation/genesplicer/exons.list\n      - variation/genesplicer/f.don\n      - variation/genesplicer/outin\n      - variation/genesplicer/score_ex.don\n      - variation/genesplicer/score_in.don\n      - variation/genesplicer/acc13\n      - variation/genesplicer/acc19\n      - variation/genesplicer/acc25\n      - variation/genesplicer/acc32\n      - variation/genesplicer/acc34\n      - variation/genesplicer/acc8\n      - variation/genesplicer/config_file\n      - variation/genesplicer/don10\n      - variation/genesplicer/don16\n      - variation/genesplicer/don2\n      - variation/genesplicer/don23\n      - variation/genesplicer/don9\n      - variation/genesplicer/exons.dat\n      - variation/genesplicer/f.acc\n      - variation/genesplicer/outex\n      - variation/genesplicer/score_ex.acc\n      - variation/genesplicer/score_in.acc\n      - variation/genesplicer/seqs\n"
  },
  {
    "path": "ggd-recipes/GRCh37/giab-NA12878-NA24385-somatic.yaml",
    "content": "# Truth set for NA12878/NA24385 somatic mixture\n# http://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/use_cases/mixtures/UMCUTRECHT_NA12878_NA24385_mixture_10052016/\n---\nattributes:\n  name: giab-NA12878-NA24385-somatic\n  version: v3_3_2-2018-07-05\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/giab-NA12878-NA24385-somatic\n        url=https://s3.amazonaws.com/bcbio/giab/NA12878-NA24385/2018-07-05\n        calls=na12878-na24385-somatic-GRCh37-truth.vcf.gz\n        regions=na12878-na24385-somatic-GRCh37-truth-regions.bed\n        mkdir -p $dir\n        wget -c $url/$calls -O $dir/truth_small_variants.vcf.gz\n        wget -c $url/$calls.tbi -O $dir/truth_small_variants.vcf.gz.tbi\n        wget -c $url/$regions -O $dir/truth_regions.bed\n    recipe_outfiles:\n      - validation/giab-NA12878-NA24385-somatic/truth_small_variants.vcf.gz\n      - validation/giab-NA12878-NA24385-somatic/truth_small_variants.vcf.gz.tbi\n      - validation/giab-NA12878-NA24385-somatic/truth_regions.bed\n"
  },
  {
    "path": "ggd-recipes/GRCh37/giab-NA12878.yaml",
    "content": "# Truth sets from Genome in a Bottle for NA12878 -- http://genomeinabottle.org/\n---\nattributes:\n  name: giab-NA12878\n  version: v3_3_2\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/giab-NA12878\n        url=http://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/NA12878_HG001/NISTv3.3.2/GRCh37/\n        calls=HG001_GRCh37_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X-SOLID_CHROM1-X_v.3.3.2_highconf_PGandRTGphasetransfer.vcf.gz\n        regions=HG001_GRCh37_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X-SOLID_CHROM1-X_v.3.3.2_highconf_nosomaticdel.bed\n        mkdir -p $dir\n        wget -c -O $dir/truth_small_variants.vcf.gz $url/$calls\n        tabix -f -p vcf $dir/truth_small_variants.vcf.gz\n        wget -c -O $dir/truth_regions.bed $url/$regions\n        wget -O - http://ftp.ncbi.nih.gov/giab/ftp/technical/svclassify_Manuscript/Supplementary_Information/Personalis_1000_Genomes_deduplicated_deletions.bed | grep -v ^Chr > $dir/truth_DEL.bed\n        wget -O - http://ftp.ncbi.nih.gov/giab/ftp/technical/svclassify_Manuscript/Supplementary_Information/Spiral_Genetics_insertions.bed | grep -v ^Chr > $dir/truth_INS.bed\n    recipe_outfiles:\n      - validation/giab-NA12878/truth_small_variants.vcf.gz\n      - validation/giab-NA12878/truth_small_variants.vcf.gz.tbi\n      - validation/giab-NA12878/truth_regions.bed\n      - validation/giab-NA12878/truth_DEL.bed\n      - validation/giab-NA12878/truth_INS.bed\n"
  },
  {
    "path": "ggd-recipes/GRCh37/giab-NA24143.yaml",
    "content": "# Truth sets from Genome in a Bottle for NA24143 -- http://genomeinabottle.org/\n---\nattributes:\n  name: giab-NA24143\n  version: v3_3_2\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/giab-NA24143\n        url=https://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/AshkenazimTrio/HG004_NA24143_mother/NISTv3.3.2/GRCh37/\n        calls=HG004_GRCh37_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X_CHROM1-22_v.3.3.2_highconf.vcf.gz \n        regions=HG004_GRCh37_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X_CHROM1-22_v.3.3.2_highconf_noinconsistent.bed\n        mkdir -p $dir\n        wget -c -O $dir/truth_small_variants.vcf.gz $url/$calls\n        tabix -f -p vcf $dir/truth_small_variants.vcf.gz\n        wget -c -O $dir/truth_regions.bed $url/$regions\n    recipe_outfiles:\n      - validation/giab-NA24143/truth_small_variants.vcf.gz\n      - validation/giab-NA24143/truth_small_variants.vcf.gz.tbi\n      - validation/giab-NA24143/truth_regions.bed\n"
  },
  {
    "path": "ggd-recipes/GRCh37/giab-NA24149.yaml",
    "content": "# Truth sets from Genome in a Bottle for NA24149 -- http://genomeinabottle.org/\n---\nattributes:\n  name: giab-NA24149\n  version: v3_3_2\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/giab-NA24149\n        url=https://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/AshkenazimTrio/HG003_NA24149_father/NISTv3.3.2/GRCh37/\n        calls=HG003_GRCh37_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X_CHROM1-22_v.3.3.2_highconf.vcf.gz \n        regions=HG003_GRCh37_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X_CHROM1-22_v.3.3.2_highconf_noinconsistent.bed\n        mkdir -p $dir\n        wget -c -O $dir/truth_small_variants.vcf.gz $url/$calls\n        tabix -f -p vcf $dir/truth_small_variants.vcf.gz\n        wget -c -O $dir/truth_regions.bed $url/$regions\n    recipe_outfiles:\n      - validation/giab-NA24149/truth_small_variants.vcf.gz\n      - validation/giab-NA24149/truth_small_variants.vcf.gz.tbi\n      - validation/giab-NA24149/truth_regions.bed\n"
  },
  {
    "path": "ggd-recipes/GRCh37/giab-NA24385.yaml",
    "content": "# Truth sets from Genome in a Bottle for NA24385 -- http://genomeinabottle.org/\n# Structural variants from\n# ftp://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/data/AshkenazimTrio/analysis/NIST_UnionSVs_05092017/Preliminary_Integrations_v0.4.0/\n---\nattributes:\n  name: giab-NA24385\n  version: v3_3_2-sv_v0.6\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/giab-NA24385\n        url=https://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/AshkenazimTrio/HG002_NA24385_son/NISTv3.3.2/GRCh37/\n        calls=HG002_GRCh37_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X-SOLID_CHROM1-22_v.3.3.2_highconf_triophased.vcf.gz\n        regions=HG002_GRCh37_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X-SOLID_CHROM1-22_v.3.3.2_highconf_noinconsistent.bed\n        mkdir -p $dir\n        wget -c -O $dir/truth_small_variants.vcf.gz $url/$calls\n        tabix -f -p vcf $dir/truth_small_variants.vcf.gz\n        wget -c -O $dir/truth_regions.bed $url/$regions\n        delurl=https://s3.amazonaws.com/bcbio/giab/NA24385/NA24385-GIAB-2tech-dels-v0_1_8-GRCh37.bed.gz\n        wget -c -O $dir/truth_DEL_giab.bed.gz $delurl\n        wget -c -O $dir/truth_DEL_giab.bed.gz.tbi $delurl.tbi\n        cnvurl=https://s3.amazonaws.com/bcbio/giab/NA24385/NA24385-crowd-dels-GRCh37.bed.gz\n        wget -c -O $dir/truth_DEL_crowd.bed.gz $cnvurl\n        wget -c -O $dir/truth_DEL_crowd.bed.gz.tbi $cnvurl.tbi\n        svurl=https://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/data/AshkenazimTrio/analysis/NIST_SVs_Integration_v0.6/HG002_SVs_Tier1_v0.6.vcf.gz\n        svbedurl=https://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/data/AshkenazimTrio/analysis/NIST_SVs_Integration_v0.6/HG002_SVs_Tier1_v0.6.bed\n        wget -c -O $dir/truth_sv.vcf.gz $svurl\n        wget -c -O $dir/truth_sv.vcf.gz.tbi $svurl.tbi\n        wget -c -O $dir/truth_sv_regions.bed $svbedurl\n    recipe_outfiles:\n      - validation/giab-NA24385/truth_small_variants.vcf.gz\n      - validation/giab-NA24385/truth_small_variants.vcf.gz.tbi\n      - validation/giab-NA24385/truth_regions.bed\n      - validation/giab-NA24385/truth_DEL_giab.bed.gz\n      - validation/giab-NA24385/truth_DEL_giab.bed.gz.tbi\n      - validation/giab-NA24385/truth_DEL_crowd.bed.gz\n      - validation/giab-NA24385/truth_DEL_crowd.bed.gz.tbi\n      - validation/giab-NA24385/truth_sv.vcf.gz\n      - validation/giab-NA24385/truth_sv.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/GRCh37/giab-NA24631.yaml",
    "content": "# Truth sets from Genome in a Bottle for NA24631 -- http://genomeinabottle.org/\n---\nattributes:\n  name: giab-NA24631\n  version: v3_3_2\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/giab-NA24631\n        url=http://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/ChineseTrio/HG005_NA24631_son/NISTv3.3.2/GRCh37/\n        calls=HG005_GRCh37_highconf_CG-IllFB-IllGATKHC-Ion-SOLID_CHROM1-22_v.3.3.2_highconf.vcf.gz\n        regions=HG005_GRCh37_highconf_CG-IllFB-IllGATKHC-Ion-SOLID_CHROM1-22_v.3.3.2_highconf_noMetaSV.bed\n        mkdir -p $dir\n        wget -c -O $dir/truth_small_variants.vcf.gz $url/$calls\n        tabix -f -p vcf $dir/truth_small_variants.vcf.gz\n        wget -c -O $dir/truth_regions.bed $url/$regions\n    recipe_outfiles:\n      - validation/giab-NA24631/truth_small_variants.vcf.gz\n      - validation/giab-NA24631/truth_small_variants.vcf.gz.tbi\n      - validation/giab-NA24631/truth_regions.bed\n"
  },
  {
    "path": "ggd-recipes/GRCh37/giab-NA24694.yaml",
    "content": "# Truth sets from Genome in a Bottle for NA24694 -- http://genomeinabottle.org/\n---\nattributes:\n  name: giab-NA24694\n  version: v3_3_2\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/giab-NA24694\n        url=https://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/ChineseTrio/HG006_NA24694_father/NISTv3.3.2/GRCh37/\n        calls=HG006_GIAB_GRCh37_highconf_CG-IllFB-IllSNT-10X_CHROM1-22_v.3.3.2_highconf.vcf.gz\n        regions=HG006_GIAB_GRCh37_highconf_CG-IllFB-IllSNT-10X_CHROM1-22_v.3.3.2_highconf_noinconsistent.bed\n        mkdir -p $dir\n        wget -c $url/$calls -O $dir/truth_small_variants.vcf.gz\n        tabix -f -p vcf $dir/truth_small_variants.vcf.gz\n        wget -c $url/$regions -O $dir/truth_regions.bed\n    recipe_outfiles:\n      - validation/giab-NA24694/truth_small_variants.vcf.gz\n      - validation/giab-NA24694/truth_small_variants.vcf.gz.tbi\n      - validation/giab-NA24694/truth_regions.bed\n"
  },
  {
    "path": "ggd-recipes/GRCh37/giab-NA24695.yaml",
    "content": "# Truth sets from Genome in a Bottle for NA24695 -- http://genomeinabottle.org/\n---\nattributes:\n  name: giab-NA24695\n  version: v3_3_2\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/giab-NA24695\n        url=https://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/ChineseTrio/HG007_NA24695_mother/NISTv3.3.2/GRCh37/\n        calls=HG007_GIAB_GRCh37_highconf_CG-IllFB-IllSNT-10X_CHROM1-22_v.3.3.2_highconf.vcf.gz\n        regions=HG007_GIAB_GRCh37_highconf_CG-IllFB-IllSNT-10X_CHROM1-22_v.3.3.2_highconf_noinconsistent.bed\n        mkdir -p $dir\n        wget -c $url/$calls -O $dir/truth_small_variants.vcf.gz\n        tabix -f -p vcf $dir/truth_small_variants.vcf.gz\n        wget -c $url/$regions -O $dir/truth_regions.bed\n    recipe_outfiles:\n      - validation/giab-NA24695/truth_small_variants.vcf.gz\n      - validation/giab-NA24695/truth_small_variants.vcf.gz.tbi\n      - validation/giab-NA24695/truth_regions.bed\n"
  },
  {
    "path": "ggd-recipes/GRCh37/gnomad.yaml",
    "content": "# GnomAD VCF sites: \n# - https://gnomad.broadinstitute.org/downloads\n# - https://macarthurlab.org/2018/10/17/gnomad-v2-1/\n# - http://ftp.ensemblorg.ebi.ac.uk/pub/data_files/homo_sapiens/GRCh37/variation_genotype/gnomad/r2.1/genomes/\n# Script \n# - sorts according to the reference file\n# - no need in decomposing (it is done in gnomad genome 2.1), but normalization and uniq are needed\n# - filters only PASS variants (segdup, decoy are retained)\n# - removes many INFO fields to reduce file size: controls_, hist, non_, using bcftools annotate instead of vt rminfo, because it failed randomly on chr1\n# - merges all chr into a single file\n# wget is separated from the processing because wget in the pipe fails randomly (on some systems?): it comes out when downloading many files.\n---\nattributes:\n  name: gnomad\n  version: 2.1.1\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url_prefix=https://storage.googleapis.com/gnomad-public/release/2.1.1/vcf/genomes/\n        vcf_file=gnomad.genomes.r2.1.1.sites.vcf.bgz\n        ref=../seq/GRCh37.fa\n        mkdir -p variation\n        export TMPDIR=`pwd`\n\n        gnomad_fields_to_keep_url=https://gist.githubusercontent.com/naumenko-sa/d20db928b915a87bba4012ba1b89d924/raw/cf343b105cb3347e966cc95d049e364528c86880/gnomad_fields_to_keep.txt\n        wget --no-check-certificate -c $gnomad_fields_to_keep_url\n\n\n        vcf=${url_prefix}${vcf_file}\n        wget --no-check-certificate -c $vcf\n        wget --no-check-certificate -c $vcf.tbi\n        fields_to_keep=\"INFO/\"$(cat gnomad_fields_to_keep.txt | paste -s | sed s/\"\\t\"/\",INFO\\/\"/g)\n        bcftools view -f PASS $vcf_file | bcftools annotate -x \"^$fields_to_keep\" -Ov | vt decompose -s - | vt normalize -r $ref -n - | vt uniq - | bgzip -c > variation/gnomad_genome.vcf.gz\n        tabix -f -p vcf variation/gnomad_genome.vcf.gz\n        tabix -f -p vcf --csi variation/gnomad_genome.vcf.gz\n        rm $vcf_file $vcf_file.tbi\n    recipe_outfiles:\n      - variation/gnomad_genome.vcf.gz\n      - variation/gnomad_genome.vcf.gz.tbi\n      - variation/gnomad_genome.vcf.gz.csi\n"
  },
  {
    "path": "ggd-recipes/GRCh37/gnomad_exome.yaml",
    "content": "# GnomAD VCF sites: \n# - https://gnomad.broadinstitute.org/downloads\n# - https://macarthurlab.org/2018/10/17/gnomad-v2-1/\n# Script\n# - no need in decomposing, normalizing,sort, and uniq anymore - it is done in gnomad 2.1.1 exomes\n# - filters only PASS variants (segdup, decoy are retained)\n# - removes many INFO fields to reduce file size: controls_, hist, non_, using bcftools annotate instead of vt rminfo, because it failed randomly on chr1\n# recipe test: https://github.com/chapmanb/cloudbiolinux/blob/master/doc/hacking.md#testing-a-ggd-recipe \n# much faster from gnomad in google cloud - downloads in 20-25 min on O2\n# vt is 3x faster than bcftools for removing INFO tags: in a test with 4444 variants t(bcftools -x) = t(bcftools -x \"^\") = 27s, time(vt rminfo) = 8s,\n# but vt rminfo is failing after 148M, using bcftools\n# overall processing is still slow: 87G to G in \n---\nattributes:\n  name: gnomad_exome\n  version: 2.1.1\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        vcf_prefix=https://storage.googleapis.com/gnomad-public/release/2.1.1/vcf/exomes/\n        vcf_file=gnomad.exomes.r2.1.1.sites.vcf.bgz\n        ref=../seq/GRCh37.fa\n        mkdir -p variation\n        export TMPDIR=`pwd`\n        gnomad_fields_to_keep_url=https://gist.githubusercontent.com/naumenko-sa/d20db928b915a87bba4012ba1b89d924/raw/cf343b105cb3347e966cc95d049e364528c86880/gnomad_fields_to_keep.txt        \n        wget --no-check-certificate -c $gnomad_fields_to_keep_url\n        vcf=${vcf_prefix}${vcf_file}\n        wget --no-check-certificate -c $vcf\n        wget --no-check-certificate -c $vcf.tbi\n        fields_to_keep=\"INFO/\"$(cat gnomad_fields_to_keep.txt | paste -s | sed s/\"\\t\"/\",INFO\\/\"/g)\n        bcftools view -f PASS $vcf_file | bcftools annotate -x \"^$fields_to_keep\" -Ov | bgzip -c > variation/gnomad_exome.vcf.gz\n        tabix -f -p vcf variation/gnomad_exome.vcf.gz\n        tabix -f -p vcf --csi variation/gnomad_exome.vcf.gz\n        rm $vcf_file $vcf_file.tbi\n    recipe_outfiles:\n      - variation/gnomad_exome.vcf.gz\n      - variation/gnomad_exome.vcf.gz.tbi\n      - variation/gnomad_exome.vcf.gz.csi\n"
  },
  {
    "path": "ggd-recipes/GRCh37/gnomad_sv.yaml",
    "content": "---\nattributes:\n  name: gnomad_sv\n  version: 2\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://storage.googleapis.com/gnomad-public/papers/2019-sv/gnomad_v2_sv.sites\n        mkdir -p variation\n        cd variation\n        wget -O - $baseurl.vcf.gz | gunzip -c | bgzip -c > gnomad_sv.vcf.gz\n        tabix -f -p vcf gnomad_sv.vcf.gz\n        wget -O - $baseurl.bed.gz | gunzip -c | bgzip -c > gnomad_sv.bed.gz\n        tabix -f -p bed gnomad_sv.bed.gz\n    recipe_outfiles:\n      - variation/gnomad_sv.vcf.gz\n      - variation/gnomad_sv.vcf.gz.tbi\n      - variation/gnomad_sv.bed.gz\n      - variation/gnomad_sv.bed.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/GRCh37/hapmap.yaml",
    "content": "---\nattributes:\n  name: hapmap\n  version: 3.3\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=ftp://gsapubftp-anonymous:none@ftp.broadinstitute.org/bundle/b37/hapmap_3.3.b37.vcf.gz\n        mkdir -p variation\n        cd variation\n        wget -O - $baseurl | gunzip -c | bgzip -c > hapmap_3.3.vcf.gz\n        tabix -f -p vcf hapmap_3.3.vcf.gz\n    recipe_outfiles:\n      - variation/hapmap_3.3.vcf.gz\n      - variation/hapmap_3.3.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/GRCh37/mills_indels.yaml",
    "content": "---\nattributes:\n  name: mills_indels\n  version: 2.8\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=ftp://gsapubftp-anonymous:none@ftp.broadinstitute.org/bundle/b37/Mills_and_1000G_gold_standard.indels.b37.vcf.gz\n        mkdir -p variation\n        cd variation\n        wget -O - $baseurl | gunzip -c | bgzip -c > Mills_and_1000G_gold_standard.indels.vcf.gz\n        tabix -f -p vcf Mills_and_1000G_gold_standard.indels.vcf.gz\n    recipe_outfiles:\n      - variation/Mills_and_1000G_gold_standard.indels.vcf.gz\n      - variation/Mills_and_1000G_gold_standard.indels.vcf.gz.tbi\n \n"
  },
  {
    "path": "ggd-recipes/GRCh37/prioritize.yaml",
    "content": "# Regions for prioritization:\n# cancer\n#  - civic -- known cancer genes from the CIViC database\n#  - az300 -- list of 300 cancer genes of interest from AstraZeneca Oncology\n#  - az300-with-fusion -- az300 list plus known interacting fusions\n#      from FusionCatcher https://github.com/ndaniel/fusioncatcher/blob/master/bin/generate_known.py#L102\n#  - az-cancer-panel -- az300 genes plus additional useful panel genes\n---\nattributes:\n  name: prioritize\n  version: 20181227\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/biodata/coverage/prioritize/prioritize-cancer-GRCh37-20181227.tar.gz\n        outdir=coverage/prioritize\n        mkdir -p $outdir\n        cd $outdir\n        wget --no-check-certificate -c -O cancer.tar.gz $baseurl\n        tar -xzvpf cancer.tar.gz\n    recipe_outfiles:\n      - coverage/prioritize/cancer/az300.bed.gz\n      - coverage/prioritize/cancer/az300.bed.gz.tbi\n      - coverage/prioritize/cancer/az300-with-fusion.bed.gz\n      - coverage/prioritize/cancer/az300-with-fusion.bed.gz.tbi\n      - coverage/prioritize/cancer/az-cancer-panel.bed.gz\n      - coverage/prioritize/cancer/az-cancer-panel.bed.gz.tbi\n      - coverage/prioritize/cancer/civic-2018-12-27.bed.gz\n      - coverage/prioritize/cancer/civic-2018-12-27.bed.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/GRCh37/qsignature.yaml",
    "content": "---\nattributes:\n  name: qsignature\n  version: 20140703\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=http://downloads.sourceforge.net/project/adamajava/qsignature.tar.bz2\n        mkdir -p variation\n        cd variation\n        wget -N -c $baseurl\n        tar -xjf qsignature.tar.bz2 qsignature_positions.txt\n        mv qsignature_positions.txt qsignature.vcf\n    recipe_outfiles:\n        - variation/qsignature.vcf\n \n"
  },
  {
    "path": "ggd-recipes/GRCh37/seq.yaml",
    "content": "# GRCh37 reference genome from Broad bundle\n# ftp://gsapubftp-anonymous:none@ftp.broadinstitute.org/bundle/b37\n# Include bgzipped fasta file for Ensembl and other tools that support it\n---\nattributes:\n  name: seq\n  version: broad-20120813\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        wget --no-check-certificate -c https://s3.amazonaws.com/biodata/genomes/GRCh37-seq.tar.gz\n        tar -xzvpf GRCh37-seq.tar.gz\n        gunzip -c seq/GRCh37.fa.gz > seq/GRCh37.fa\n        touch seq/GRCh37.fa.fai\n        touch seq/GRCh37.dict\n    recipe_outfiles:\n      - seq/GRCh37.fa\n      - seq/GRCh37.fa.fai\n      - seq/GRCh37.fa.gz\n      - seq/GRCh37.fa.gz.fai\n      - seq/GRCh37.fa.gz.gzi\n      - seq/GRCh37.dict\n"
  },
  {
    "path": "ggd-recipes/GRCh37/topmed.yaml",
    "content": "# Topmed WGS allele frequencies: https://bravo.sph.umich.edu/freeze3a/hg19/\n#url=ftp://ftp.ensembl.org/pub/data_files/homo_sapiens/GRCh37/variation_genotype/TOPMED_GRCh37.vcf.gz\n#ref=GRCh37.fa\n#mkdir -p variation\n#export TMPDIR=`pwd`\n#gunzip -c TOPMED_GRCh37.vcf.gz | sed s/Number=\\./Number=A/g | vt normalize -r $ref -n - | vt uniq - | gsort -m 3000 /dev/stdin $ref.fai | bgzip -c > variation/topmed-freeze3a.vcf.gz\n#tabix -f -p vcf variation/topmed-freeze3a.vcf.gz\n---\nattributes:\n  name: topmed\n  version: freeze3a\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=http://s3.amazonaws.com/bcbio-nextgen/genomes/Hsapiens/GRCh37/variation/topmed-freeze3a.vcf.gz\n        mkdir -p variation\n        wget -c -O - ${url} > variation/topmed-freeze3a.vcf.gz\n        url=http://s3.amazonaws.com/bcbio-nextgen/genomes/Hsapiens/GRCh37/variation/topmed-freeze3a.vcf.gz.tbi\n        wget -c -O - ${url} > variation/topmed-freeze3a.vcf.gz.tbi\n        rm -f variation/topmed.vcf.gz\n        rm -f variation/topmed.vcf.gz.tbi\n        cd variation\n        ln -s topmed-freeze3a.vcf.gz topmed.vcf.gz\n        ln -s topmed-freeze3a.vcf.gz.tbi topmed.vcf.gz.tbi\n    recipe_outfiles:\n      - variation/topmed.vcf.gz\n      - variation/topmed.vcf.gz.tbi\n      - variation/topmed-freeze3a.vcf.gz\n      - variation/topmed-freeze3a.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/GRCh37/transcripts.yaml",
    "content": "# Transcript, annotation and indices for RNA-seq analysis\n# Prepared from Ensembl transcripts using: \n# https://github.com/chapmanb/cloudbiolinux/blob/master/utils/prepare_tx_gff.py\n---\nattributes:\n  name: transcripts\n  version: 2015-12-01\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/biodata/annotation/GRCh37-rnaseq-2015-12-01.tar.xz\n        wget -c -N --random-wait --retry-connrefused --no-check-certificate $baseurl\n        xz -dc *-rnaseq-*.tar.xz | tar --no-same-owner -xpf -\n        mv */rnaseq-* rnaseq\n    recipe_outfiles:\n      - rnaseq/kallisto\n      - rnaseq/tophat\n      - rnaseq/ref-transcripts.bed\n      - rnaseq/ref-transcripts.dexseq.gff3\n      - rnaseq/ref-transcripts.fa\n      - rnaseq/ref-transcripts.genePred\n      - rnaseq/ref-transcripts.gtf\n      - rnaseq/ref-transcripts.gtf.db\n      - rnaseq/ref-transcripts-mask.gtf\n      - rnaseq/ref-transcripts.refFlat\n      - rnaseq/ref-transcripts-splicesites.txt\n      - rnaseq/rRNA.gtf\n      - rnaseq/rRNA.gtf.db\n      - rnaseq/rRNA.interval_list\n"
  },
  {
    "path": "ggd-recipes/GRCh37/twobit.yaml",
    "content": "# GRCh37 UCSC 2bit reference file\n---\nattributes:\n  name: twobit\n  version: broad-20120813\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p ucsc\n        cd ucsc\n        faToTwoBit ../../seq/GRCh37.fa GRCh37.2bit\n    recipe_outfiles:\n      - ucsc/GRCh37.2bit\n\n"
  },
  {
    "path": "ggd-recipes/GRCh37/varpon.yaml",
    "content": "# Panels of normals (PONs) for variant calling\n# germline_het_pon -- Locations of likely germline heterogygote sites\n---\nattributes:\n  name: varpon\n  version: 20181105\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p variation\n        wget --no-check-certificate -O - 'https://s3.amazonaws.com/biodata/collections/GRCh37/variation/germline_het_pon.bed.gz' > variation/germline_het_pon.bed.gz\n        wget --no-check-certificate -O - 'https://s3.amazonaws.com/biodata/collections/GRCh37/variation/germline_het_pon.bed.gz.tbi' > variation/germline_het_pon.bed.gz.tbi\n    recipe_outfiles:\n      - variation/germline_het_pon.bed.gz\n      - variation/germline_het_pon.bed.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/GRCh37/vcfanno.yaml",
    "content": "# Annotation files for vcfanno variant annotation\n# Feeds into vcf2db GEMINI database creation and other annotation tasks\n---\nattributes:\n  name: vcfanno\n  version: 20210204\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p config/vcfanno\n        cd config/vcfanno\n        bcbio_baseurl=https://raw.githubusercontent.com/bcbio/bcbio-nextgen/master/config/vcfanno\n        wget --no-check-certificate -O gemini.conf -c $bcbio_baseurl/GRCh37-gemini.conf\n        wget --no-check-certificate -O gemini.lua -c $bcbio_baseurl/GRCh37-gemini.lua\n        wget --no-check-certificate -O rnaedit.conf -c $bcbio_baseurl/rnaedit.conf\n        wget --no-check-certificate -O rnaedit.lua -c $bcbio_baseurl/rnaedit.lua\n        wget --no-check-certificate -O dbnsfp.conf -c $bcbio_baseurl/GRCh37-dbnsfp.conf\n        wget --no-check-certificate -O dbscsnv.conf -c $bcbio_baseurl/GRCh37-dbscsnv.conf\n        wget --no-check-certificate -O gnomad_genome.conf -c $bcbio_baseurl/GRCh37-gnomad_genome.conf\n        wget --no-check-certificate -O somatic.conf -c $bcbio_baseurl/somatic.conf\n\n    recipe_outfiles:\n      - config/vcfanno/gemini.lua\n      - config/vcfanno/gemini.conf\n      - config/vcfanno/rnaedit.lua\n      - config/vcfanno/rnaedit.conf\n      - config/vcfanno/dbnsfp.conf\n      - config/vcfanno/dbscsnv.conf\n      - config/vcfanno/gnomad_genome.conf\n      - config/vcfanno/somatic.conf\n"
  },
  {
    "path": "ggd-recipes/GRCh37/viral.yaml",
    "content": "# Virus reference genomes, prepared for bwa alignment\n# Taken from TCGA GDC reference genome\n# https://gdc.cancer.gov/about-data/data-harmonization-and-generation/gdc-reference-files\n---\nattributes:\n  name: viral\n  version: 2019.12.03\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=http://s3.amazonaws.com/biodata/viral/GRCh37-gdc-viral.tar.gz\n        mkdir -p viral\n        wget --no-check-certificate -c $url\n        tar -xzvpf *.tar.gz\n        mv *-gdc-viral/* viral\n    recipe_outfiles:\n      - viral/gdc-viral.fa\n      - viral/gdc-viral.dict\n      - viral/gdc-viral.fa.ann\n      - viral/gdc-viral.fa.fai\n      - viral/gdc-viral.fa.pac\n      - viral/gdc-viral.fa.bwt\n      - viral/gdc-viral.fa.amb\n      - viral/gdc-viral.fa.sa\n"
  },
  {
    "path": "ggd-recipes/GRCz11/seq.yaml",
    "content": "---\nattributes:\n  name: seq\n  version: 92\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        BUILD=GRCz11\n        url=http://ftp.ensembl.org/pub/release-92/fasta/danio_rerio/dna/Danio_rerio.GRCz11.dna.toplevel.fa.gz\n        mkdir -p seq\n        wget --no-check-certificate -O - $url | gunzip -c > seq/$BUILD.fa\n        samtools faidx seq/$BUILD.fa\n        picard -Xms1g -Xmx3g CreateSequenceDictionary REFERENCE=seq/$BUILD.fa OUTPUT=seq/$BUILD.dict SPECIES=$BUILD\n    recipe_outfiles:\n      - seq/GRCz11.fa\n      - seq/GRCz11.fa.fai\n      - seq/GRCz11.dict\n"
  },
  {
    "path": "ggd-recipes/GRCz11/transcripts.yaml",
    "content": "# Transcript, annotation and indices for RNA-seq analysis\n# Prepared from Ensembl transcripts using:\n# https://github.com/chapmanb/cloudbiolinux/blob/master/utils/prepare_tx_gff.py\n---\nattributes:\n  name: transcripts\n  version: 2018-05-15_92\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/biodata/annotation/GRCz11-rnaseq-2018-05-15_92.tar.xz\n        wget -c -N --no-check-certificate $baseurl\n        xz -dc *-rnaseq-*.tar.xz | tar -xpf -\n        mv */rnaseq-* rnaseq\n    recipe_outfiles:\n      - rnaseq/ref-transcripts.bed\n      - rnaseq/ref-transcripts.dexseq.gff3\n      - rnaseq/ref-transcripts.fa\n      - rnaseq/ref-transcripts.genePred\n      - rnaseq/ref-transcripts.gtf\n      - rnaseq/ref-transcripts.gtf.db\n      - rnaseq/ref-transcripts-mask.gtf\n      - rnaseq/ref-transcripts.refFlat\n      - rnaseq/ref-transcripts-splicesites.txt\n      - rnaseq/rRNA.gtf\n      - rnaseq/rRNA.gtf.db\n      - rnaseq/rRNA.interval_list\n      - rnaseq/tx2gene.csv\n      - rnaseq/version.txt\n"
  },
  {
    "path": "ggd-recipes/GRCz11/twobit.yaml",
    "content": "---\nattributes:\n  name: twobit\n  version: 92\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p ucsc\n        cd ucsc\n        faToTwoBit -long ../../seq/GRCz11.fa GRCz11.2bit\n    recipe_outfiles:\n      - ucsc/GRCz11.2bit\n"
  },
  {
    "path": "ggd-recipes/README.md",
    "content": "Recipes for downloading data\n\nIdea developed from Aaron Quinlan's GGD (Get Genomic Data)\n\nhttps://github.com/arq5x/ggd\nhttps://github.com/arq5x/ggd-recipes\n"
  },
  {
    "path": "ggd-recipes/Sscrofa11.1/seq.yaml",
    "content": "---\nattributes:\n  name: seq\n  version: 92\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        BUILD=Sscrofa11.1\n        url=http://ftp.ensembl.org/pub/release-92/fasta/sus_scrofa/dna/Sus_scrofa.Sscrofa11.1.dna_rm.toplevel.fa.gz\n        mkdir -p seq\n        wget --no-check-certificate -O - $url | gunzip -c > seq/$BUILD.fa\n        samtools faidx seq/$BUILD.fa\n        picard -Xms1g -Xmx3g CreateSequenceDictionary REFERENCE=seq/$BUILD.fa OUTPUT=seq/$BUILD.dict SPECIES=$BUILD\n    recipe_outfiles:\n      - seq/Sscrofa11.1.fa\n      - seq/Sscrofa11.1.fa.fai\n      - seq/Sscrofa11.1.dict\n"
  },
  {
    "path": "ggd-recipes/Sscrofa11.1/transcripts.yaml",
    "content": "# Transcript, annotation and indices for RNA-seq analysis\n# Prepared from Ensembl transcripts using:\n# https://github.com/chapmanb/cloudbiolinux/blob/master/utils/prepare_tx_gff.py\n---\nattributes:\n  name: transcripts\n  version: 2018-05-14_92\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/biodata/annotation/Sscrofa11.1-rnaseq-2018-05-14_92.tar.xz\n        wget -c -N --no-check-certificate $baseurl\n        xz -dc *-rnaseq-*.tar.xz | tar -xpf -\n        mv */rnaseq-* rnaseq\n    recipe_outfiles:\n      - rnaseq/ref-transcripts.bed\n      - rnaseq/ref-transcripts.dexseq.gff3\n      - rnaseq/ref-transcripts.fa\n      - rnaseq/ref-transcripts.genePred\n      - rnaseq/ref-transcripts.gtf\n      - rnaseq/ref-transcripts.gtf.db\n      - rnaseq/ref-transcripts-mask.gtf\n      - rnaseq/ref-transcripts.refFlat\n      - rnaseq/ref-transcripts-splicesites.txt\n      - rnaseq/rRNA.gtf\n      - rnaseq/rRNA.gtf.db\n      - rnaseq/rRNA.interval_list\n      - rnaseq/tx2gene.csv\n      - rnaseq/version.txt\n"
  },
  {
    "path": "ggd-recipes/Sscrofa11.1/twobit.yaml",
    "content": "---\nattributes:\n  name: twobit\n  version: 92\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p ucsc\n        cd ucsc\n        faToTwoBit ../../seq/Sscrofa11.1.fa Sscrofa11.1.2bit\n    recipe_outfiles:\n      - ucsc/Sscrofa11.1.2bit\n"
  },
  {
    "path": "ggd-recipes/TAIR10/mirbase.yaml",
    "content": "# small RNA seq annotation from different sources\n---\nattributes:\n  name: srnaseq\n  version: 20160603\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p srnaseq\n        cd srnaseq\n\n        wget -c -O tmp.gtf.gz ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/001/735/GCF_000001735.3_TAIR10/GCF_000001735.3_TAIR10_genomic.gff.gz\n        zgrep -v exon tmp.gtf.gz | grep -v region | sed 's/Name/name/g' | sed 's/=/ /g' > srna-transcripts.gtf\n        wget -c -O hairpin.t.fa.gz ftp://mirbase.org/pub/mirbase/21/hairpin.fa.gz && gunzip -f hairpin.t.fa.gz\n        cat hairpin.t.fa | awk '{if ($0~/>ath/){name=$0; print name} else if ($0~/^>/){name=0};if (name!=0 && $0!~/^>/){print $0;}}' | sed 's/U/T/g'  > hairpin.fa\n        wget -c -O mature.t.fa.gz ftp://mirbase.org/pub/mirbase/21/mature.fa.gz && gunzip -f mature.t.fa.gz\n        cat mature.t.fa | awk '{if ($0~/>ath/){name=$0; print name} else if ($0~/^>/){name=0};if (name!=0 && $0!~/^>/){print $0;}}' | sed 's/U/T/g'  > mature.fa\n        wget -c -O miRNA.t.str.gz ftp://mirbase.org/pub/mirbase/21/miRNA.str.gz && gunzip -f miRNA.t.str.gz\n        cat miRNA.t.str | awk '{if ($0~/ath/)print $0}' > miRNA.str\n        wget --no-check-certificate -c -O Rfam_for_miRDeep.fa.gz https://github.com/lpantano/mirdeep2_core/raw/data/Rfam_for_miRDeep.fa.gz && gunzip -f Rfam_for_miRDeep.fa.gz\n    recipe_outfiles:\n        - srnaseq/srna-transcripts.gtf\n        - srnaseq/hairpin.fa\n        - srnaseq/mature.fa\n        - srnaseq/miRNA.str\n        - srnaseq/Rfam_for_miRDeep.fa\n"
  },
  {
    "path": "ggd-recipes/canFam3/dbsnp.yaml",
    "content": "---\nattributes:\n  name: dbsnp\n  version: 20140510\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/biodata/variants/canFam3-dbSNP-2014-05-10.vcf.gz\n        mkdir -p variation\n        cd variation\n        wget -c -N $baseurl\n        wget -c -N $baseurl.tbi\n    recipe_outfiles:\n      - variation/canFam3-dbSNP-2014-05-10.vcf.gz\n      - variation/canFam3-dbSNP-2014-05-10.vcf.gz.tbi"
  },
  {
    "path": "ggd-recipes/canFam3/mirbase.yaml",
    "content": "# small RNA seq annotation from different sources\n---\nattributes:\n  name: srnaseq\n  version: 20181024\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p srnaseq\n        cd srnaseq\n        wget --random-wait --retry-connrefused -q -c -O cfa.gff3 https://mirbase.org/download/cfa.gff3\n        awk '$3==\"miRNA\"' cfa.gff3 | sed 's/=/ /g' > srna-transcripts.gtf\n        wget  --random-wait --retry-connrefused -q -c -O rmsk.txt.gz http://hgdownload.soe.ucsc.edu/goldenPath/canFam3/database/rmsk.txt.gz\n        zcat rmsk.txt.gz | awk '{print $6\"\\t.\\trepeat\\t\"$7+1\"\\t\"$8+1\"\\t.\\t\"$10\"\\t.\\tname \"$12\";\"}' >> srna-transcripts.gtf\n        wget  --random-wait --retry-connrefused  -q -c -O refGene.txt.gz http://hgdownload.soe.ucsc.edu/goldenPath/canFam3/database/refGene.txt.gz\n        zcat refGene.txt.gz | awk '{print $3\"\\t.\\tgene\\t\"$5\"\\t\"$6\"\\t.\\t\"$4\"\\t.\\tname \"$13\";\"}' >> srna-transcripts.gtf\n        # mirbase\n        wget  --random-wait --retry-connrefused  -q -c -O hairpin.fa.raw https://mirbase.org/download/CURRENT/hairpin.fa\n        cat hairpin.fa.raw |  awk '{if ($0~/>cfa/){name=$0; print name} else if ($0~/^>/){name=0};if (name!=0 && $0!~/^>/){print $0;}}' | sed 's/U/T/g'  > hairpin.fa\n        rm hairpin.fa.raw\n        wget  --random-wait --retry-connrefused  -q -c -O mature.fa.raw https://mirbase.org/download/CURRENT/mature.fa\n        cat mature.fa.raw |  awk '{if ($0~/>cfa/){name=$0; print name} else if ($0~/^>/){name=0};if (name!=0 && $0!~/^>/){print $0;}}' | sed 's/U/T/g'  > mature.fa\n        rm mature.fa.raw\n        wget  --random-wait --retry-connrefused  -q -c -O miRNA.dat.raw https://mirbase.org/download/CURRENT/miRNA.dat\n        cat miRNA.dat.raw | awk '{if ($0~/cfa/)print $0}' > miRNA.str\n        rm miRNA.dat.raw\n        wget --random-wait --retry-connrefused -q -c -O mirbase.gff3 https://mirbase.org/download/cfa.gff3\n        # mintmap\n        # mirdeep2\n        wget  --random-wait --retry-connrefused --no-check-certificate -q -c -O Rfam_for_miRDeep.fa.gz https://github.com/lpantano/mirdeep2_core/raw/data/Rfam_for_miRDeep.fa.gz && gunzip Rfam_for_miRDeep.fa.gz\n        # targetscan analysis\n    recipe_outfiles:\n        - srnaseq/srna-transcripts.gtf\n        - srnaseq/hairpin.fa\n        - srnaseq/mature.fa\n        - srnaseq/miRNA.str\n        - srnaseq/mirbase.gff3\n        - srnaseq/Rfam_for_miRDeep.fa\n"
  },
  {
    "path": "ggd-recipes/canFam3/transcripts.yaml",
    "content": "# Transcript, annotation and indices for RNA-seq analysis\n# Prepared from Ensembl transcripts using: \n# https://github.com/chapmanb/cloudbiolinux/blob/master/utils/prepare_tx_gff.py\n---\nattributes:\n  name: transcripts\n  version: 2018-03-22_91\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/biodata/annotation/canFam3-rnaseq-2018-03-22_91.tar.xz\n        wget -c -N --no-check-certificate $baseurl\n        xz -dc *-rnaseq-*.tar.xz | tar -xpf -\n        mv */rnaseq-* rnaseq\n    recipe_outfiles:\n      - rnaseq/ref-transcripts.bed\n      - rnaseq/ref-transcripts.dexseq.gff3\n      - rnaseq/ref-transcripts.fa\n      - rnaseq/ref-transcripts.genePred\n      - rnaseq/ref-transcripts.gtf\n      - rnaseq/ref-transcripts.gtf.db\n      - rnaseq/ref-transcripts-mask.gtf\n      - rnaseq/ref-transcripts.refFlat\n      - rnaseq/ref-transcripts-splicesites.txt\n      - rnaseq/rRNA.gtf\n      - rnaseq/rRNA.gtf.db\n      - rnaseq/rRNA.interval_list\n      - rnaseq/tx2gene.csv\n      - rnaseq/version.txt\n"
  },
  {
    "path": "ggd-recipes/canFam3/twobit.yaml",
    "content": "# UCSC twobit reference\n---\nattributes:\n  name: twobit\n  version: canFam3\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p ucsc\n        cd ucsc\n        faToTwoBit ../../seq/canFam3.fa canFam3.2bit\n    recipe_outfiles:\n      - ucsc/canFam3.2bit\n"
  },
  {
    "path": "ggd-recipes/hg19/1000g.yaml",
    "content": "# 1000 genomes\n---\nattributes:\n  name: 1000g\n  version: phase3_shapeit2_mvncall_integrated_v5a.20130502\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/gemini-annotations\n        url=$baseurl/ALL.wgs.phase3_shapeit2_mvncall_integrated_v5a.20130502.sites.tidy.vcf.gz\n        remap_url=https://raw.githubusercontent.com/dpryan79/ChromosomeMappings/master/GRCh37_ensembl2UCSC.txt\n        mkdir -p variation\n        wget --no-check-certificate -qO- $remap_url | awk '{if($1!=$2) print \"s/^\"$1\"/\"$2\"/g\"}' > remap.sed\n        export TMPDIR=`pwd`\n        wget -c -O - $url | gunzip -c | sed -f remap.sed | bgzip -c > variation/1000g.vcf.gz\n        tabix -f -p vcf variation/1000g.vcf.gz\n    recipe_outfiles:\n      - variation/1000g.vcf.gz\n      - variation/1000g.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg19/1000g_omni_snps.yaml",
    "content": "---\nattributes:\n  name: 1000g_omni_snps\n  version: 2.5\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=ftp://gsapubftp-anonymous:none@ftp.broadinstitute.org/bundle/hg19/1000G_omni2.5.hg19.sites.vcf.gz\n        mkdir -p variation\n        cd variation\n        wget -O - $baseurl | gunzip -c | bgzip -c > 1000G_omni2.5.vcf.gz\n        tabix -f -p vcf  1000G_omni2.5.vcf.gz\n    recipe_outfiles:\n      - variation/1000G_omni2.5.vcf.gz\n      - variation/1000G_omni2.5.vcf.gz.tbi\n \n"
  },
  {
    "path": "ggd-recipes/hg19/1000g_snps.yaml",
    "content": "---\nattributes:\n  name: 1000g_snps\n  version: 2.8\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=ftp://gsapubftp-anonymous:none@ftp.broadinstitute.org/bundle/hg19/1000G_phase1.snps.high_confidence.hg19.sites.vcf.gz\n        mkdir -p variation\n        cd variation\n        wget -O - $baseurl | gunzip -c | bgzip -c > 1000G_phase1.snps.high_confidence.vcf.gz\n        tabix -f -p vcf 1000G_phase1.snps.high_confidence.vcf.gz\n    recipe_outfiles:\n      - variation/1000G_phase1.snps.high_confidence.vcf.gz\n      - variation/1000G_phase1.snps.high_confidence.vcf.gz.tbi\n \n"
  },
  {
    "path": "ggd-recipes/hg19/ACMG56_genes.yaml",
    "content": "# ACMG regions: http://iobio.io/2016/03/29/acmg56/\n# https://github.com/lpantano/custom_genome_annotation\n---\nattributes:\n  name: ACMG56_genes\n  version: 20160629\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://raw.githubusercontent.com/lpantano/custom_genome_annotation/master/acmg56.ucsc.bed\n        dir=coverage/prioritize/actionable\n        mkdir -p $dir\n        cd $dir\n        wget --no-check-certificate -c -O ACMG56.bed $baseurl\n    recipe_outfiles:\n      - coverage/prioritize/actionable/ACMG56.bed\n"
  },
  {
    "path": "ggd-recipes/hg19/GA4GH_problem_regions.yaml",
    "content": "# GA4GH problematic region files: https://drive.google.com/folderview?id=0B7Ao1qqJJDHQUjVIN3liUUZNWjg&usp=sharing#list\n# Cached mappability regions\n# wget --no-check-certificate -O - http://hgdownload.cse.ucsc.edu/goldenPath/hg19/encodeDCC/wgEncodeMapability/wgEncodeDacMapabilityConsensusExcludable.bed.gz | gunzip -c | bgzip -c > $encode/wgEncodeDacMapabilityConsensusExcludable.bed.gz\n# tabix -p bed -f $encode/wgEncodeDacMapabilityConsensusExcludable.bed.gz\n---\nattributes:\n  name: GA4GH_problem_regions\n  version: 20181016\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=http://bcbio_nextgen.s3.amazonaws.com/GA4GH_problem_regions.zip\n        mkdir -p coverage/problem_regions/GA4GH\n        cd coverage/problem_regions/GA4GH\n        wget --no-check-certificate -c -O GA4GH_problem_regions.zip $baseurl\n        unzip -o GA4GH_problem_regions.zip\n        for file in *.bed\n        do\n          bgzip $file\n          tabix -p bed $file.gz\n        done\n        cd ../../..\n        encode=coverage/problem_regions/ENCODE\n        mkdir -p $encode\n        cd $encode\n        wget --no-check-certificate http://s3.amazonaws.com/biodata/coverage/problem_regions/ENCODE/hg19/wgEncodeDacMapabilityConsensusExcludable.bed.gz\n        wget --no-check-certificate http://s3.amazonaws.com/biodata/coverage/problem_regions/ENCODE/hg19/wgEncodeDacMapabilityConsensusExcludable.bed.gz.tbi\n        cd ../../..\n        repeats=coverage/problem_regions/repeats\n        mkdir -p $repeats\n        wget --no-check-certificate -O - https://raw.githubusercontent.com/chapmanb/delly/master/human.hg19.excl.tsv | grep ^chr > $repeats/sv_repeat_telomere_centromere.bed\n        wget --no-check-certificate -O - https://github.com/lh3/varcmp/raw/master/scripts/LCR-hs37d5.bed.gz | gunzip -c | grep -v ^GL | grep -v ^NC | grep -v ^hs | sed 's/^/chr/' | bgzip -c > $repeats/LCR.bed.gz\n        tabix -p bed -f $repeats/LCR.bed.gz\n        wget --no-check-certificate -O $repeats/polyx.bed.gz http://s3.amazonaws.com/biodata/coverage/problem_regions/polyx/polyX-hg19.bed.gz\n        wget --no-check-certificate -O $repeats/polyx.bed.gz.tbi http://s3.amazonaws.com/biodata/coverage/problem_regions/polyx/polyX-hg19.bed.gz.tbi\n        gc=coverage/gc\n        mkdir -p $gc\n        wget --no-check-certificate -O - https://s3.amazonaws.com/biodata/collections/hg19/coverage/gc/GC_profile.1000bp.cnp > $gc/GC_profile.1000bp.cnp\n    recipe_outfiles:\n      - coverage/problem_regions/GA4GH/README.md\n      - coverage/problem_regions/GA4GH/bad_promoter.bed.gz\n      - coverage/problem_regions/GA4GH/bad_promoter.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/gc15.bed.gz\n      - coverage/problem_regions/GA4GH/gc15.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/gc15to20.bed.gz\n      - coverage/problem_regions/GA4GH/gc15to20.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/gc20to25.bed.gz\n      - coverage/problem_regions/GA4GH/gc20to25.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/gc25to30.bed.gz\n      - coverage/problem_regions/GA4GH/gc25to30.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/gc65to70.bed.gz\n      - coverage/problem_regions/GA4GH/gc65to70.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/gc70to75.bed.gz\n      - coverage/problem_regions/GA4GH/gc70to75.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/gc75to80.bed.gz\n      - coverage/problem_regions/GA4GH/gc75to80.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/gc80to85.bed.gz\n      - coverage/problem_regions/GA4GH/gc80to85.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/gc85.bed.gz\n      - coverage/problem_regions/GA4GH/gc85.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/heng_um75-hs37d5.bed.gz\n      - coverage/problem_regions/GA4GH/heng_um75-hs37d5.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/low_complexity_51to200bp.bed.gz\n      - coverage/problem_regions/GA4GH/low_complexity_51to200bp.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/low_complexity_gt200bp.bed.gz\n      - coverage/problem_regions/GA4GH/low_complexity_gt200bp.bed.gz.tbi\n      - coverage/problem_regions/GA4GH/self_chain.bed.gz\n      - coverage/problem_regions/GA4GH/self_chain.bed.gz.tbi\n      - coverage/problem_regions/ENCODE/wgEncodeDacMapabilityConsensusExcludable.bed.gz\n      - coverage/problem_regions/ENCODE/wgEncodeDacMapabilityConsensusExcludable.bed.gz.tbi\n      - coverage/problem_regions/repeats/sv_repeat_telomere_centromere.bed\n      - coverage/problem_regions/repeats/LCR.bed.gz\n      - coverage/problem_regions/repeats/LCR.bed.gz.tbi\n      - coverage/problem_regions/repeats/polyx.bed.gz\n      - coverage/problem_regions/repeats/polyx.bed.gz.tbi\n      - coverage/gc/GC_profile.1000bp.cnp\n"
  },
  {
    "path": "ggd-recipes/hg19/MIG.yaml",
    "content": "# Medically Interpretable Genes, compiled by Personalis: http://www.genomemedicine.com/content/pdf/s13073-015-0197-4.pdf\n---\nattributes:\n  name: MIG\n  version: 20150730\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=http://bcbio_nextgen.s3.amazonaws.com/MIG.zip\n        mkdir -p prioritization\n        cd prioritization\n        wget -c -O MIG.zip $baseurl\n        unzip MIG.zip\n    recipe_outfiles:\n      - prioritization/MIG.bed\n"
  },
  {
    "path": "ggd-recipes/hg19/RADAR.yaml",
    "content": "# RADAR RNA editing events: (http://www.stanford.edu/~gokulr/database/Human_AG_all_hg19_v2.txt)\n# installs a BED file of the RADAR events\n---\nattributes:\n  name: RADAR\n  version: v2-20180202\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=https://s3.amazonaws.com/biodata/annotation/RADAR/hg19/RADAR.bed.gz\n        mkdir -p editing\n        cd editing\n        wget --no-check-certificate -q $url\n        wget --no-check-certificate -q $url.tbi\n    recipe_outfiles:\n      - editing/RADAR.bed.gz\n      - editing/RADAR.bed.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg19/af_only_gnomad.yaml",
    "content": "# af_only_gnomad file for t-only variant calling with mutect2\n# From the Broad bundle:\n# ftp://gsapubftp-anonymous@ftp.broadinstitute.org/bundle/Mutect2/af-only-gnomad.raw.sites.b37.vcf.gz\n---\nattributes:\n  name: af_only_gnomad\n  version: 20170602\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=ftp://gsapubftp-anonymous@ftp.broadinstitute.org/bundle/Mutect2/\n        base=af-only-gnomad.raw.sites.b37\n        mkdir -p variation\n        for suffix in .vcf.gz .vcf.gz.tbi\n        do\n          wget --no-check-certificate -c ${url}/${base}${suffix}\n        done\n        remap_url=https://raw.githubusercontent.com/dpryan79/ChromosomeMappings/master/GRCh37_ensembl2UCSC.txt\n        ref=../seq/hg19.fa\n        export TMPDIR=`pwd`\n        wget --no-check-certificate -qO- $remap_url | awk '{if($1!=$2) print \"s/^\"$1\"/\"$2\"/g\"}' > remap.sed\n        # remove grch37 contig names in the header\n        bcftools view -f PASS $base.vcf.gz | sed -f remap.sed | grep -v \"##contig=\" | vt decompose -s - | vt normalize -r $ref -n - | vt uniq - | bgzip -c > variation/af_only_gnomad.vcf.gz\n        tabix -f -p vcf variation/af_only_gnomad.vcf.gz\n        tabix -f -p vcf --csi variation/af_only_gnomad.vcf.gz\n        rm $base.vcf.gz $base.vcf.gz.tbi\n    recipe_outfiles:\n      - variation/af_only_gnomad.vcf.gz\n      - variation/af_only_gnomad.vcf.gz.tbi\n      - variation/af_only_gnomad.vcf.gz.csi\n"
  },
  {
    "path": "ggd-recipes/hg19/battenberg.yaml",
    "content": "# Battenberg CNV caller input data:\n# https://github.com/cancerit/cgpBattenberg\n---\nattributes:\n  name: battenberg\n  version: v3-20150610\nrecipe:\n  full:\n    required:\n      - download_generate_bberg_ref_files.pl\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        proburl=https://github.com/cancerit/cgpBattenberg/raw/dev/perl/share/battenberg/probloci.txt.gz\n        impute_info=battenberg/impute/impute_info.txt\n        mkdir -p battenberg\n        [[ -f $impute_info ]] || download_generate_bberg_ref_files.pl -c -o `pwd`/battenberg\n        # Change references to point to final directory\n        sed -i 's/txtmp\\///g' $impute_info\n\n        wget --no-check-certificate -c -O battenberg/probloci.txt.gz $proburl\n        gunzip battenberg/probloci.txt.gz\n\n        # Convert chromosomes to hg19 names\n        sed -i \"s/^\\([0-9]\\+\\)\\t/chr\\1\\t/g\" battenberg/probloci.txt\n        sed -i \"s/^X/chrX/g\" battenberg/probloci.txt\n        sed -i \"s/^\\([0-9]\\+\\)\\t/chr\\1\\t/g\" $impute_info\n        sed -i \"s/^X/chrX/g\" $impute_info\n        sed -i \"s/^\\([0-9]\\+\\)\\t/chr\\1\\t/g\" battenberg/1000genomesloci/1000genomesloci2012_chr*\n        sed -i \"s/^X/chrX/g\" battenberg/1000genomesloci/1000genomesloci2012_chr23.txt\n    recipe_outfiles:\n      - battenberg/probloci.txt\n      - battenberg/1000genomesloci\n      - battenberg/impute\n"
  },
  {
    "path": "ggd-recipes/hg19/capture_regions.yaml",
    "content": "# Common sets of capture region BED files for different platforms\n---\nattributes:\n  name: capture_regions\n  version: 20161202\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://raw.githubusercontent.com/AstraZeneca-NGS/reference_data/master/hg19/bed\n        files=\"Exome-Agilent_V5_Clinical.bed Exome-Agilent_V6.bed Exome-AZ_V2.bed Exome-IDT_V1.bed\n               Exome-MedExome.bed Exome-NGv3.bed\"\n        mkdir -p coverage/capture_regions\n        cd coverage/capture_regions\n        for fname in $files\n        do\n          wget --no-check-certificate -c $baseurl/$fname\n        done\n    recipe_outfiles:\n      - coverage/capture_regions/Exome-Agilent_V5_Clinical.bed\n      - coverage/capture_regions/Exome-Agilent_V6.bed\n      - coverage/capture_regions/Exome-AZ_V2.bed\n      - coverage/capture_regions/Exome-IDT_V1.bed\n      - coverage/capture_regions/Exome-MedExome.bed\n      - coverage/capture_regions/Exome-NGv3.bed\n"
  },
  {
    "path": "ggd-recipes/hg19/clinvar.yaml",
    "content": "# ClinVar: http://www.clinvar.com/\n# UCSFify name sed magic from: https://github.com/mmarchin/utilities/blob/master/ucscify.sh\n---\nattributes:\n  name: clinvar\n  version: 20210110\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        release=20210110\n        # clinvar.vcf.gz is the latest one\n        baseurl=https://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh37/clinvar.vcf.gz\n        mkdir -p variation\n        wget --no-check-certificate -c -O variation/clinvar-orig.vcf.gz $baseurl\n        rm -f variation/clinvar.vcf.gz\n        rm -f variation/clinvar.vcf.gz.tbi\n        zcat variation/clinvar-orig.vcf.gz | sed \"s/^\\([0-9]\\+\\)\\t/chr\\1\\t/g\" | sed \"s/^MT/chrM/g\" | sed \"s/^X/chrX/g\" | sed \"s/^Y/chrY/g\" | bgzip -c > variation/clinvar.vcf.gz\n        tabix -f -p vcf variation/clinvar.vcf.gz\n    recipe_outfiles:\n      - variation/clinvar.vcf.gz\n      - variation/clinvar.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg19/cosmic.yaml",
    "content": "---\nattributes:\n  name: cosmic\n  version: 68-20180114\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/biodata/variants/cosmic-v68-hg19.vcf.gz\n        mkdir -p variation\n        cd variation\n        wget --no-check-certificate -c -O cosmic.vcf.gz $baseurl\n        wget --no-check-certificate -c -O cosmic.vcf.gz.tbi $baseurl.tbi\n    recipe_outfiles:\n        - variation/cosmic.vcf.gz\n        - variation/cosmic.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg19/dbsnp.yaml",
    "content": "# recipe_outfiles: move symlinks first, not otherwise\n---\nattributes:\n  name: dbsnp\n  version: 156-20230320\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        build=156\n        version=GCF_000001405.25\n        url=https://ftp.ncbi.nih.gov/snp/archive/b$build/VCF/$version.gz\n        mkdir -p variation\n        cd variation\n        wget -c -O dbsnp-${build}-orig.vcf.gz $url\n        wget -c -O dbsnp-${build}-orig.vcf.gz.tbi $url.tbi\n        remap_url=https://raw.githubusercontent.com/dpryan79/ChromosomeMappings/master/GRCh37_NCBI2UCSC.txt\n        # remove chromosomes not included in hg19 from remap\n        wget --no-check-certificate -qO- $remap_url | sed 's/\\r//' | awk '{if(NF==2) print $0}'> remap.tsv\n        export TMPDIR=`pwd`\n        # remove NW_ contigs absent in hg19\n        gunzip -c dbsnp-${build}-orig.vcf.gz | grep -v \"NW_\" | bgzip -c > dbsnp-${build}.noNW.vcf.gz\n        tabix dbsnp-${build}.noNW.vcf.gz\n        bcftools annotate -Ou --rename-chrs remap.tsv dbsnp-${build}.noNW.vcf.gz | \\\n        bcftools sort -m 1G -Oz -T . -o dbsnp-${build}.vcf.gz\n        tabix -f -p vcf dbsnp-${build}.vcf.gz\n        tabix -f -p vcf -C dbsnp-${build}.vcf.gz\n        ln -sf dbsnp-${build}.vcf.gz dbsnp.vcf.gz\n        ln -sf dbsnp-${build}.vcf.gz.tbi dbsnp.vcf.gz.tbi\n        ln -sf dbsnp-${build}.vcf.gz.csi dbsnp.vcf.gz.csi\n        cd ..\n    recipe_outfiles:\n      - variation/dbsnp.vcf.gz\n      - variation/dbsnp.vcf.gz.csi\n      - variation/dbsnp.vcf.gz.tbi\n      - variation/dbsnp-156.vcf.gz\n      - variation/dbsnp-156.vcf.gz.csi\n      - variation/dbsnp-156.vcf.gz.tbi \n"
  },
  {
    "path": "ggd-recipes/hg19/effects_transcripts.yaml",
    "content": "# Custom transcripts for effects prediction\n# Tweaks canonical transcripts in effects predictors like snpEff.\n---\nattributes:\n  name: effects_transcript\n  version: 2017-03-16\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        COMMIT=93d7d3794e6afe39d92e1abc0c56406373f98501\n        mkdir -p snpeff/transcripts\n        cd snpeff/transcripts\n        wget --no-check-certificate -O canonical_cancer.txt https://raw.githubusercontent.com/AstraZeneca-NGS/reference_data/$COMMIT/transcripts/canon_cancer_replacement.txt\n    recipe_outfiles:\n      - snpeff/transcripts/canonical_cancer.txt\n"
  },
  {
    "path": "ggd-recipes/hg19/esp.yaml",
    "content": "# Exome Sequencing Project (ESP): http://evs.gs.washington.edu/EVS/\n---\nattributes:\n  name: esp\n  version: ESP6500SI-V2\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/gemini-annotations\n        url=$baseurl/ESP6500SI.all.snps_indels.tidy.v2.vcf.gz\n        remap_url=https://raw.githubusercontent.com/dpryan79/ChromosomeMappings/master/GRCh37_ensembl2UCSC.txt\n        mkdir -p variation\n        wget --no-check-certificate -qO- $remap_url | awk '{if($1!=$2) print \"s/^\"$1\"/\"$2\"/g\"}' > remap.sed\n        export TMPDIR=`pwd`\n        wget -c -O - $url | gunzip -c | sed -f remap.sed | bgzip -c > variation/esp.vcf.gz\n        tabix -f -p vcf variation/esp.vcf.gz\n    recipe_outfiles:\n      - variation/esp.vcf.gz\n      - variation/esp.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg19/exac.yaml",
    "content": "# ExAC VCF sites: http://exac.broadinstitute.org/\n# Script cleans up to use chr-style naming and sort by ref file\n---\nattributes:\n  name: exac\n  version: 0.3\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/gemini-annotations\n        url=$baseurl/ExAC.r0.3.sites.vep.tidy.vcf.gz\n        remap_url=https://raw.githubusercontent.com/dpryan79/ChromosomeMappings/master/GRCh37_ensembl2UCSC.txt\n        mkdir -p variation\n        wget --no-check-certificate -qO- $remap_url | awk '{if($1!=$2) print \"s/^\"$1\"/\"$2\"/g\"}' > remap.sed\n        export TMPDIR=`pwd`\n        wget -c -O - $url | gunzip -c | sed -f remap.sed | bgzip -c > variation/exac.vcf.gz\n        tabix -f -p vcf variation/exac.vcf.gz\n    recipe_outfiles:\n      - variation/exac.vcf.gz\n      - variation/exac.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg19/fusion-blacklist.yaml",
    "content": "# fusion blacklist regions from arriba\n---\nattributes:\n  name: fusion-blacklist\n  version: 2\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        version=v1.1.0\n        url=https://github.com/suhrig/arriba/releases/download/${version}/arriba_${version}.tar.gz\n        arriba_tarball=arriba_${version}.tar.gz\n        outdir=rnaseq/fusion-blacklist\n        mkdir -p ${outdir}\n        wget --no-check-certificate -c -O ${arriba_tarball} ${url}\n        tar zxvf ${arriba_tarball}\n        mv arriba_${version}/database/blacklist_hg19_hs37d5_GRCh37_2018-11-04.tsv.gz ${outdir}/arriba-blacklist.tsv.gz\n        mv arriba_${version}/database/cytobands_hg19_hs37d5_GRCh37_2018-02-23.tsv ${outdir}/arriba-cytobands.tsv\n        mv arriba_${version}/database/protein_domains_hg19_hs37d5_GRCh37_2018-03-06.gff3 ${outdir}/arriba-protein-domains.gff3\n        mv arriba_${version}/database/protein_domains_hg19_hs37d5_GRCh37_2018-03-06.gff3.idx ${outdir}/arriba-protein-domains.gff3.idx\n        rm -r arriba_${version}\n        rm ${arriba_tarball}\n    recipe_outfiles:\n      - rnaseq/fusion-blacklist/arriba-blacklist.tsv.gz\n      - rnaseq/fusion-blacklist/arriba-cytobands.tsv\n      - rnaseq/fusion-blacklist/arriba-protein-domains.gff3\n      - rnaseq/fusion-blacklist/arriba-protein-domains.gff3.idx\n"
  },
  {
    "path": "ggd-recipes/hg19/giab-NA12878.yaml",
    "content": "# Truth sets from Genome in a Bottle for NA12878 -- http://genomeinabottle.org/\n# Converts coordinates to UCSC 'chr' style naming with sed scripts\n# Removes mitochondrial chromosome to avoid comparisons there\n---\nattributes:\n  name: giab-NA12878\n  version: v3_3_2\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/giab-NA12878\n        remapurl=https://raw.githubusercontent.com/dpryan79/ChromosomeMappings/master/GRCh37_ensembl2UCSC.txt\n        url=http://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/NA12878_HG001/NISTv3.3.2/GRCh37/\n        calls=HG001_GRCh37_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X-SOLID_CHROM1-X_v.3.3.2_highconf_PGandRTGphasetransfer.vcf.gz\n        regions=HG001_GRCh37_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X-SOLID_CHROM1-X_v.3.3.2_highconf_nosomaticdel.bed\n        mkdir -p $dir\n        wget -qO- $remapurl | awk '{if($1!=$2) print \"s/^\"$1\"/\"$2\"/g\"}' > remap.sed\n        wget -c -O - $url/$calls | gunzip -c | sed -f remap.sed | grep -v \"##contig=\" | bgzip -c > $dir/truth_small_variants.vcf.gz\n        tabix -f -p vcf $dir/truth_small_variants.vcf.gz\n        wget -c -O - $url/$regions | sed -f remap.sed > $dir/truth_regions.bed\n        wget -O - http://ftp.ncbi.nih.gov/giab/ftp/technical/svclassify_Manuscript/Supplementary_Information/Personalis_1000_Genomes_deduplicated_deletions.bed | grep -v ^Chr | sed \"s/^\\([0-9]\\+\\)\\t/chr\\1\\t/g\" | sed \"s/^MT/chrM/g\" | sed \"s/^X/chrX/g\" | sed \"s/^Y/chrY/g\" > $dir/truth_DEL.bed\n        wget -O -  http://ftp.ncbi.nih.gov/giab/ftp/technical/svclassify_Manuscript/Supplementary_Information/Spiral_Genetics_insertions.bed | grep -v ^Chr | sed \"s/^\\([0-9]\\+\\)\\t/chr\\1\\t/g\" | sed \"s/^MT/chrM/g\" | sed \"s/^X/chrX/g\" | sed \"s/^Y/chrY/g\" > $dir/truth_INS.bed\n    recipe_outfiles:\n      - validation/giab-NA12878/truth_small_variants.vcf.gz\n      - validation/giab-NA12878/truth_small_variants.vcf.gz.tbi\n      - validation/giab-NA12878/truth_regions.bed\n      - validation/giab-NA12878/truth_DEL.bed\n      - validation/giab-NA12878/truth_INS.bed\n"
  },
  {
    "path": "ggd-recipes/hg19/giab-NA24143.yaml",
    "content": "# Truth sets from Genome in a Bottle for NA24143 -- http://genomeinabottle.org/\n---\nattributes:\n  name: giab-NA24143\n  version: v3_3_2\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/giab-NA24143\n        url=https://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/AshkenazimTrio/HG004_NA24143_mother/NISTv3.3.2/GRCh37/\n        calls=HG004_GRCh37_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X_CHROM1-22_v.3.3.2_highconf.vcf.gz \n        regions=HG004_GRCh37_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X_CHROM1-22_v.3.3.2_highconf_noinconsistent.bed\n        remapurl=https://raw.githubusercontent.com/dpryan79/ChromosomeMappings/master/GRCh37_ensembl2UCSC.txt\n        mkdir -p $dir\n        wget -qO- $remapurl | awk '{if($1!=$2) print \"s/^\"$1\"/\"$2\"/g\"}' > remap.sed\n        wget -c -O - $url/$calls | gunzip -c | sed -f remap.sed | grep -v \"##contig=\" | bgzip -c > $dir/truth_small_variants.vcf.gz\n        tabix -f -p vcf $dir/truth_small_variants.vcf.gz\n        wget -c -O - $url/$regions | sed -f remap.sed > $dir/truth_regions.bed\n    recipe_outfiles:\n      - validation/giab-NA24143/truth_small_variants.vcf.gz\n      - validation/giab-NA24143/truth_small_variants.vcf.gz.tbi\n      - validation/giab-NA24143/truth_regions.bed\n"
  },
  {
    "path": "ggd-recipes/hg19/giab-NA24149.yaml",
    "content": "# Truth sets from Genome in a Bottle for NA24149 -- http://genomeinabottle.org/\n---\nattributes:\n  name: giab-NA24149\n  version: v3_3_2\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/giab-NA24149\n        url=https://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/AshkenazimTrio/HG003_NA24149_father/NISTv3.3.2/GRCh37/\n        calls=HG003_GRCh37_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X_CHROM1-22_v.3.3.2_highconf.vcf.gz \n        regions=HG003_GRCh37_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X_CHROM1-22_v.3.3.2_highconf_noinconsistent.bed\n        remapurl=https://raw.githubusercontent.com/dpryan79/ChromosomeMappings/master/GRCh37_ensembl2UCSC.txt\n        mkdir -p $dir\n        wget -qO- $remapurl | awk '{if($1!=$2) print \"s/^\"$1\"/\"$2\"/g\"}' > remap.sed\n        wget -c -O - $url/$calls | gunzip -c | sed -f remap.sed | grep -v \"##contig=\" | bgzip -c > $dir/truth_small_variants.vcf.gz\n        tabix -f -p vcf $dir/truth_small_variants.vcf.gz\n        wget -c -O - $url/$regions | sed -f remap.sed > $dir/truth_regions.bed\n    recipe_outfiles:\n      - validation/giab-NA24149/truth_small_variants.vcf.gz\n      - validation/giab-NA24149/truth_small_variants.vcf.gz.tbi\n      - validation/giab-NA24149/truth_regions.bed\n"
  },
  {
    "path": "ggd-recipes/hg19/giab-NA24385.yaml",
    "content": "# Truth sets from Genome in a Bottle for NA24385 -- http://genomeinabottle.org/\n---\nattributes:\n  name: giab-NA24385\n  version: v3_3_2-sv_v0.6\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/giab-NA24385\n        url=http://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/AshkenazimTrio/HG002_NA24385_son/NISTv3.3.2/GRCh37/\n        calls=HG002_GRCh37_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X-SOLID_CHROM1-22_v.3.3.2_highconf_triophased.vcf.gz\n        regions=HG002_GRCh37_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X-SOLID_CHROM1-22_v.3.3.2_highconf_noinconsistent.bed\n        remapurl=https://raw.githubusercontent.com/dpryan79/ChromosomeMappings/master/GRCh37_ensembl2UCSC.txt\n        mkdir -p $dir\n        wget -qO- $remapurl | awk '{if($1!=$2) print \"s/^\"$1\"/\"$2\"/g\"}' > remap.sed\n        wget -c -O - $url/$calls | gunzip -c | sed -f remap.sed | grep -v \"##contig=\" | bgzip -c > $dir/truth_small_variants.vcf.gz\n        tabix -f -p vcf $dir/truth_small_variants.vcf.gz\n        wget -c -O - $url/$regions | sed -f remap.sed > $dir/truth_regions.bed\n        delurl=https://s3.amazonaws.com/bcbio/giab/NA24385/NA24385-GIAB-2tech-dels-v0_1_8-hg19.bed.gz\n        wget -c -O $dir/truth_DEL_giab.bed.gz $delurl\n        wget -c -O $dir/truth_DEL_giab.bed.gz.tbi $delurl.tbi\n        cnvurl=https://s3.amazonaws.com/bcbio/giab/NA24385/NA24385-crowd-dels-hg19.bed.gz\n        wget -c -O $dir/truth_DEL_crowd.bed.gz $cnvurl\n        wget -c -O $dir/truth_DEL_crowd.bed.gz.tbi $cnvurl.tbi\n        svurl=ftp://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/data/AshkenazimTrio/analysis/NIST_SVs_Integration_v0.6/HG002_SVs_Tier1_v0.6.vcf.gz\n        svbedurl=ftp://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/data/AshkenazimTrio/analysis/NIST_SVs_Integration_v0.6/HG002_SVs_Tier1_v0.6.bed\n        wget -c -O - $svurl | gunzip -c | sed -f remap.sed | grep -v \"##contig=\" | bgzip -c > $dir/truth_sv.vcf.gz\n        tabix -f -p vcf $dir/truth_sv.vcf.gz\n        wget -c -O - $svbedurl | sed -f remap.sed > $dir/truth_sv_regions.bed\n    recipe_outfiles:\n      - validation/giab-NA24385/truth_small_variants.vcf.gz\n      - validation/giab-NA24385/truth_small_variants.vcf.gz.tbi\n      - validation/giab-NA24385/truth_regions.bed\n      - validation/giab-NA24385/truth_DEL_giab.bed.gz\n      - validation/giab-NA24385/truth_DEL_giab.bed.gz.tbi\n      - validation/giab-NA24385/truth_DEL_crowd.bed.gz\n      - validation/giab-NA24385/truth_DEL_crowd.bed.gz.tbi\n      - validation/giab-NA24385/truth_sv.vcf.gz\n      - validation/giab-NA24385/truth_sv.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg19/giab-NA24631.yaml",
    "content": "# Truth sets from Genome in a Bottle for NA24631 -- http://genomeinabottle.org/\n---\nattributes:\n  name: giab-NA24631\n  version: v3_3_2\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/giab-NA24631\n        url=http://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/ChineseTrio/HG005_NA24631_son/NISTv3.3.2/GRCh37/\n        calls=HG005_GRCh37_highconf_CG-IllFB-IllGATKHC-Ion-SOLID_CHROM1-22_v.3.3.2_highconf.vcf.gz\n        regions=HG005_GRCh37_highconf_CG-IllFB-IllGATKHC-Ion-SOLID_CHROM1-22_v.3.3.2_highconf_noMetaSV.bed\n        remapurl=https://raw.githubusercontent.com/dpryan79/ChromosomeMappings/master/GRCh37_ensembl2UCSC.txt\n        mkdir -p $dir\n        wget -qO- $remapurl | awk '{if($1!=$2) print \"s/^\"$1\"/\"$2\"/g\"}' > remap.sed\n        wget -c -O - $url/$calls | gunzip -c | sed -f remap.sed | grep -v \"##contig=\" | bgzip -c > $dir/truth_small_variants.vcf.gz\n        tabix -f -p vcf $dir/truth_small_variants.vcf.gz\n        wget -c -O - $url/$regions | sed -f remap.sed > $dir/truth_regions.bed\n    recipe_outfiles:\n      - validation/giab-NA24631/truth_small_variants.vcf.gz\n      - validation/giab-NA24631/truth_small_variants.vcf.gz.tbi\n      - validation/giab-NA24631/truth_regions.bed\n"
  },
  {
    "path": "ggd-recipes/hg19/gnomad.yaml",
    "content": "# GnomAD VCF sites: \n# - https://gnomad.broadinstitute.org/downloads\n# - https://macarthurlab.org/2018/10/17/gnomad-v2-1/\n# - http://ftp.ensemblorg.ebi.ac.uk/pub/data_files/homo_sapiens/GRCh37/variation_genotype/gnomad/r2.1/genomes/\n# Script \n# - cleans up to use chr-style naming,\n# - sorts according to the reference file\n# - normalizes (input is decomposed but not normalized in gnomad genome 2.1)\n# - filters only PASS variants (segdup, decoy are retained)\n# - removes many INFO fields to reduce file size: controls_, hist, non_, using bcftools annotate instead of vt rminfo, because it failed randomly on chr1\n# - merges all chr into a single file\n# wget is separated from the processing because wget in the pipe fails randomly (on some systems?): it comes out when downloading many files.\n# recipe test: https://github.com/chapmanb/cloudbiolinux/blob/master/doc/hacking.md#testing-a-ggd-recipe \n---\nattributes:\n  name: gnomad\n  version: 2.1.1\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url_prefix=https://storage.googleapis.com/gnomad-public/release/2.1.1/vcf/genomes/\n        vcf_file=gnomad.genomes.r2.1.1.sites.vcf.bgz\n        remap_url=https://raw.githubusercontent.com/dpryan79/ChromosomeMappings/master/GRCh37_ensembl2UCSC.txt\n        ref=../seq/hg19.fa\n        mkdir -p variation\n        export TMPDIR=`pwd`\n\n        wget --no-check-certificate -qO- $remap_url | awk '{if($1!=$2) print \"s/^\"$1\"/\"$2\"/g\"}' > remap.sed\n\n        gnomad_fields_to_keep_url=https://gist.githubusercontent.com/naumenko-sa/d20db928b915a87bba4012ba1b89d924/raw/cf343b105cb3347e966cc95d049e364528c86880/gnomad_fields_to_keep.txt\n        wget --no-check-certificate -c $gnomad_fields_to_keep_url\n\n        vcf=${url_prefix}${vcf_file}\n        wget --no-check-certificate -c $vcf\n        wget --no-check-certificate -c $vcf.tbi\n\n        fields_to_keep=\"INFO/\"$(cat gnomad_fields_to_keep.txt | paste -s | sed s/\"\\t\"/\",INFO\\/\"/g)\n        bcftools view -f PASS $vcf_file | bcftools annotate -x \"^$fields_to_keep\" -Ov | sed -f remap.sed | grep -v \"##config=\" | vt decompose -s - | vt normalize -r $ref -n - | vt uniq - | bgzip -c > variation/gnomad_genome.vcf.gz\n        tabix -f -p vcf variation/gnomad_genome.vcf.gz\n        tabix -f -p vcf --csi variation/gnomad_genome.vcf.gz\n        rm $vcf_file $vcf_file.tbi\n    recipe_outfiles:\n      - variation/gnomad_genome.vcf.gz\n      - variation/gnomad_genome.vcf.gz.tbi\n      - variation/gnomad_genome.vcf.gz.csi\n"
  },
  {
    "path": "ggd-recipes/hg19/gnomad_exome.yaml",
    "content": "# GnomAD VCF sites: \n# - https://gnomad.broadinstitute.org/downloads\n# - https://macarthurlab.org/2018/10/17/gnomad-v2-1/\n# Script\n# - no need in decomposing, normalizing,sort, and uniq anymore - it is done in gnomad 2.1.1 exomes\n# - filters only PASS variants (segdup, decoy are retained)\n# - removes many INFO fields to reduce file size: controls_, hist, non_, using bcftools annotate instead of vt rminfo, because it failed randomly on chr1\n# recipe test: https://github.com/chapmanb/cloudbiolinux/blob/master/doc/hacking.md#testing-a-ggd-recipe \n# much faster from gnomad in google cloud - downloads in 20-25 min on O2\n# vt is 3x faster than bcftools for removing INFO tags: in a test with 4444 variants t(bcftools -x) = t(bcftools -x \"^\") = 27s, time(vt rminfo) = 8s,\n# but vt rminfo is failing after 148M, using bcftools\n# overall processing is still slow: 87G to G in \n---\nattributes:\n  name: gnomad_exome\n  version: 2.1.1\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |        \n        remap_url=https://raw.githubusercontent.com/dpryan79/ChromosomeMappings/master/GRCh37_ensembl2UCSC.txt\n        ref=../seq/hg19.fa\n        mkdir -p variation\n        export TMPDIR=`pwd`\n        wget --no-check-certificate -qO- $remap_url | awk '{if($1!=$2) print \"s/^\"$1\"/\"$2\"/g\"}' > remap.sed\n\n        vcf_prefix=https://storage.googleapis.com/gnomad-public/release/2.1.1/vcf/exomes/\n        vcf_file=gnomad.exomes.r2.1.1.sites.vcf.bgz\n        ref=../seq/GRCh37.fa\n        mkdir -p variation\n        export TMPDIR=`pwd`\n        gnomad_fields_to_keep_url=https://gist.githubusercontent.com/naumenko-sa/d20db928b915a87bba4012ba1b89d924/raw/cf343b105cb3347e966cc95d049e364528c86880/gnomad_fields_to_keep.txt        \n        wget --no-check-certificate -c $gnomad_fields_to_keep_url\n        vcf=${vcf_prefix}${vcf_file}\n        wget --no-check-certificate -c $vcf\n        wget --no-check-certificate -c $vcf.tbi\n        fields_to_keep=\"INFO/\"$(cat gnomad_fields_to_keep.txt | paste -s | sed s/\"\\t\"/\",INFO\\/\"/g)\n        bcftools view -f PASS $vcf_file | bcftools annotate -x \"^$fields_to_keep\" -Ov | sed -f remap.sed | grep -v \"##contig=\" | bgzip -c > variation/gnomad_exome.vcf.gz\n        tabix -f -p vcf variation/gnomad_exome.vcf.gz\n        tabix -f -p vcf --csi variation/gnomad_exome.vcf.gz\n        rm $vcf_file $vcf_file.tbi\n    recipe_outfiles:\n      - variation/gnomad_exome.vcf.gz\n      - variation/gnomad_exome.vcf.gz.tbi\n      - variation/gnomad_exome.vcf.gz.csi\n"
  },
  {
    "path": "ggd-recipes/hg19/gnomad_genome.grch37_to_hg19.sh",
    "content": "#!/bin/bash\n\n#PBS -l walltime=230:00:00,nodes=1:ppn=1\n#PBS -joe .\n#PBS -d .\n#PBS -l vmem=10g,mem=10g\n\ndate\nexport PATH=/hpf/largeprojects/ccmbio/naumenko/tools/bcbio_1.1.5/bin:$PATH\nremap_url=https://raw.githubusercontent.com/dpryan79/ChromosomeMappings/master/GRCh37_ensembl2UCSC.txt\nref=seq/hg19.fa\nexport TMPDIR=`pwd`\n#wget --no-check-certificate -qO- $remap_url | awk '{if($1!=$2) print \"s/^\"$1\"/\"$2\"/g\"}' > remap.sed\ngunzip -c gnomad_genome.grch37.vcf.gz | sed -f remap.sed | gsort -m 3000 /dev/stdin $ref.fai | vt normalize -r $ref -n - | vt uniq - | bgzip -c >  gnomad_genome.vcf.gz\ntabix -f -p vcf gnomad_genome.vcf.gz\ntabix -f -p vcf --csi gnomad_genome.vcf.gz\ndate\n"
  },
  {
    "path": "ggd-recipes/hg19/gtf.yaml",
    "content": "---\nattributes:\n  name: gtf\n  version: 87\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=ftp://ftp.ensembl.org/pub/grch37/current/gtf/homo_sapiens/Homo_sapiens.GRCh37.87.gtf.gz\n        mkdir -p rnaseq\n        remap_url=http://raw.githubusercontent.com/dpryan79/ChromosomeMappings/master/GRCh37_ensembl2UCSC.txt\n        wget --no-check-certificate -qO- $remap_url | awk '{if($1!=$2) print \"s/^\"$1\"/\"$2\"/g\"}' > remap.sed\n        wget --no-check-certificate -qO- $url | gunzip -c > hg19.gtf\n        sed -f remap.sed hg19.gtf | grep -v _hap > rnaseq/hg19.gtf\n        rm remap.sed\n        rm hg19.gtf\n    recipe_outfiles:\n    - rnaseq/hg19.gtf\n"
  },
  {
    "path": "ggd-recipes/hg19/hapmap.yaml",
    "content": "---\nattributes:\n  name: hapmap\n  version: 3.3\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=ftp://gsapubftp-anonymous:none@ftp.broadinstitute.org/bundle/hg19/hapmap_3.3.hg19.sites.vcf.gz\n        mkdir -p variation\n        cd variation\n        wget -O - $baseurl | gunzip -c | bgzip -c > hapmap_3.3.vcf.gz\n        tabix -f -p vcf hapmap_3.3.vcf.gz\n    recipe_outfiles:\n      - variation/hapmap_3.3.vcf.gz\n      - variation/hapmap_3.3.vcf.gz.tbi\n \n"
  },
  {
    "path": "ggd-recipes/hg19/mills_indels.yaml",
    "content": "---\nattributes:\n  name: mills_indels\n  version: 2.8\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=ftp://gsapubftp-anonymous:none@ftp.broadinstitute.org/bundle/hg19/Mills_and_1000G_gold_standard.indels.hg19.sites.vcf.gz\n        mkdir -p variation\n        cd variation\n        wget -O - $baseurl | gunzip -c | bgzip -c > Mills_and_1000G_gold_standard.indels.vcf.gz\n        tabix -f -p vcf Mills_and_1000G_gold_standard.indels.vcf.gz\n    recipe_outfiles:\n      - variation/Mills_and_1000G_gold_standard.indels.vcf.gz\n      - variation/Mills_and_1000G_gold_standard.indels.vcf.gz.tbi\n \n"
  },
  {
    "path": "ggd-recipes/hg19/mirbase.yaml",
    "content": "# small RNA seq annotation from different sources\n---\nattributes:\n  name: srnaseq\n  version: 20211104\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p srnaseq\n        cd srnaseq\n        wget --random-wait --retry-connrefused -nv -c -O hsahg19.gff3 https://www.mirbase.org/ftp/20/genomes/hsa.gff3\n        awk '$3==\"miRNA\"' hsahg19.gff3 | sed 's/=/ /g' > srna-transcripts.gtf\n        wget  --random-wait --retry-connrefused -nv -c -O wgRna.txt.gz http://hgdownload.soe.ucsc.edu/goldenPath/hg19/database/wgRna.txt.gz\n        zgrep -v 'hsa-' wgRna.txt.gz | awk '{print $2\"\\t.\\tncrna\\t\"$3\"\\t\"$4\"\\t.\\t\"$7\"\\t.\\tname \"$5\";\"}' >> srna-transcripts.gtf\n        wget  --random-wait --retry-connrefused -nv -c -O tRNAs.txt.gz http://hgdownload.soe.ucsc.edu/goldenPath/hg19/database/tRNAs.txt.gz\n        zcat tRNAs.txt.gz | awk '{print $2\"\\t.\\ttRNA\\t\"$3\"\\t\"$4\"\\t.\\t\"$7\"\\t.\\tname \"$5\";\"}' >> srna-transcripts.gtf\n        wget  --random-wait --retry-connrefused -nv -c -O rmsk.txt.gz http://hgdownload.soe.ucsc.edu/goldenPath/hg19/database/rmsk.txt.gz\n        zcat rmsk.txt.gz | awk '{print $6\"\\t.\\trepeat\\t\"$7+1\"\\t\"$8+1\"\\t.\\t\"$10\"\\t.\\tname \"$12\";\"}' >> srna-transcripts.gtf\n        wget  --random-wait --retry-connrefused  -nv -c -O refGene.txt.gz http://hgdownload.soe.ucsc.edu/goldenPath/hg19/database/refGene.txt.gz\n        zcat refGene.txt.gz | awk '{print $3\"\\t.\\tgene\\t\"$5\"\\t\"$6\"\\t.\\t\"$4\"\\t.\\tname \"$13\";\"}' >> srna-transcripts.gtf\n        # wget --random-wait --retry-connrefused  -nv -c -O piR_hg19_v1.0.bed.gz http://www.regulatoryrna.org/database/piRNA/download/archive/v1.0/bed/piR_hg19_v1.0.bed.gz\n        # zcat piR_hg19_v1.0.bed.gz | awk '{print $1\"\\t.\\tpiRNA\\t\"$2\"\\t\"$3\"\\t.\\t\"$6\"\\t.\\tname \"$4\";\"}' >> srna-transcripts.gtf\n        # mirbase\n        wget  --random-wait --retry-connrefused  -nv -c -O hairpin.fa.raw https://mirbase.org/download/CURRENT/hairpin.fa\n        cat hairpin.fa.raw |  awk '{if ($0~/>hsa/){name=$0; print name} else if ($0~/^>/){name=0};if (name!=0 && $0!~/^>/){print $0;}}' | sed 's/U/T/g'  > hairpin.fa\n        rm hairpin.fa.raw\n        wget  --random-wait --retry-connrefused  -nv -c -O mature.fa.raw https://mirbase.org/download/CURRENT/mature.fa\n        cat mature.fa.raw |  awk '{if ($0~/>hsa/){name=$0; print name} else if ($0~/^>/){name=0};if (name!=0 && $0!~/^>/){print $0;}}' | sed 's/U/T/g'  > mature.fa\n        rm mature.fa.raw\n        wget  --random-wait --retry-connrefused  -nv -c -O miRNA.dat.raw https://mirbase.org/download/CURRENT/miRNA.dat\n        cat miRNA.dat.raw | awk '{if ($0~/hsa/)print $0}' > miRNA.str\n        rm miRNA.dat.raw\n        wget --random-wait --retry-connrefused -nv -c -O mirbase.gff3 https://mirbase.org/download/hsa.gff3\n        # tdrmapper\n        wget   --random-wait --retry-connrefused  --no-check-certificate -nv -c -O trna_mature_pre.fa https://github.com/sararselitsky/tDRmapper/raw/master/hg19_mature_and_pre.fa\n        # mintmap\n        wget --random-wait --retry-connrefused  -nv -c https://github.com/TJU-CMC-Org/MINTmap/archive/refs/tags/MINTmap_Standard_v1.0.tar.gz\n        tar xzf MINTmap_Standard_v1.0.tar.gz MINTmap-MINTmap_Standard_v1.0/LookupTable.tRFs.MINTmap_v1.txt --strip-components=1\n        tar xzf MINTmap_Standard_v1.0.tar.gz MINTmap-MINTmap_Standard_v1.0/tRNAspace.Spliced.Sequences.MINTmap_v1.fa --strip-components=1\n        tar xzf MINTmap_Standard_v1.0.tar.gz MINTmap-MINTmap_Standard_v1.0/OtherAnnotations.MINTmap_v1.txt --strip-components=1\n        # mirdeep2\n        wget  --random-wait --retry-connrefused --no-check-certificate -nv -c -O Rfam_for_miRDeep.fa.gz https://github.com/lpantano/mirdeep2_core/raw/data/Rfam_for_miRDeep.fa.gz && gunzip Rfam_for_miRDeep.fa.gz\n        # targetscan analysis\n        wget  --random-wait --retry-connrefused --no-check-certificate -nv -c -O Summary_Counts.txt.zip http://www.targetscan.org/vert_71/vert_71_data_download/Summary_Counts.all_predictions.txt.zip && unzip Summary_Counts.txt.zip\n        wget  --random-wait --retry-connrefused --no-check-certificate -nv -c -O miR_Family_Info.txt.zip http://www.targetscan.org/vert_71/vert_71_data_download/miR_Family_Info.txt.zip && unzip miR_Family_Info.txt.zip\n        wget  --random-wait --retry-connrefused --no-check-certificate -nv -N -c https://www.mirbase.org/ftp/21/database_files/mirna_mature.txt.gz\n    recipe_outfiles:\n        - srnaseq/srna-transcripts.gtf\n        - srnaseq/hairpin.fa\n        - srnaseq/mature.fa\n        - srnaseq/miRNA.str\n        - srnaseq/mirbase.gff3\n        - srnaseq/trna_mature_pre.fa\n        - srnaseq/LookupTable.tRFs.MINTmap_v1.txt\n        - srnaseq/tRNAspace.Spliced.Sequences.MINTmap_v1.fa\n        - srnaseq/OtherAnnotations.MINTmap_v1.txt\n        - srnaseq/Rfam_for_miRDeep.fa\n        - srnaseq/Summary_Counts.all_predictions.txt\n        - srnaseq/miR_Family_Info.txt\n        - srnaseq/mirna_mature.txt.gz\n"
  },
  {
    "path": "ggd-recipes/hg19/platinum-genome-NA12878.yaml",
    "content": "# Truth sets from Illumina Platinum Genomes project\n# http://www.illumina.com/platinumgenomes/\n---\nattributes:\n  name: platinum-genome-NA12878\n  version: v8_0_1\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        version=v8_0_1\n        dir=validation/platinum-genome-NA12878\n        mkdir -p $dir\n        wget --no-check-certificate -c -O $dir/truth_small_variants.vcf.gz https://s3.amazonaws.com/cloudbiolinux/cache/platinum-genome/platinum-genome-NA12878-hg19-$version.vcf.gz\n        wget --no-check-certificate -c -O $dir/truth_small_variants.vcf.gz.tbi https://s3.amazonaws.com/cloudbiolinux/cache/platinum-genome/platinum-genome-NA12878-hg19-$version.vcf.gz.tbi\n        wget --no-check-certificate -c -O - https://s3.amazonaws.com/cloudbiolinux/cache/platinum-genome/platinum-genome-NA12878-hg19-regions-$version.bed.gz | gunzip -c > $dir/truth_regions.bed\n    recipe_outfiles:\n      - validation/platinum-genome-NA12878/truth_small_variants.vcf.gz\n      - validation/platinum-genome-NA12878/truth_small_variants.vcf.gz.tbi\n      - validation/platinum-genome-NA12878/truth_regions.bed\n"
  },
  {
    "path": "ggd-recipes/hg19/prioritize.yaml",
    "content": "# Regions for prioritization:\n# cancer\n#  - civic -- known cancer genes from the CIViC database\n#  - az300 -- list of 300 cancer genes of interest from AstraZeneca Oncology\n#  - az300-with-fusion -- az300 list plus known interacting fusions\n#      from FusionCatcher https://github.com/ndaniel/fusioncatcher/blob/master/bin/generate_known.py#L102\n#  - az-cancer-panel -- az300 genes plus additional useful panel genes\n---\nattributes:\n  name: prioritize\n  version: 20181227\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/biodata/coverage/prioritize/prioritize-cancer-hg19-20181227.tar.gz\n        outdir=coverage/prioritize\n        mkdir -p $outdir\n        cd $outdir\n        wget --no-check-certificate -c -O cancer.tar.gz $baseurl\n        tar -xzvpf cancer.tar.gz\n    recipe_outfiles:\n      - coverage/prioritize/cancer/az300.bed.gz\n      - coverage/prioritize/cancer/az300.bed.gz.tbi\n      - coverage/prioritize/cancer/az300-with-fusion.bed.gz\n      - coverage/prioritize/cancer/az300-with-fusion.bed.gz.tbi\n      - coverage/prioritize/cancer/az-cancer-panel.bed.gz\n      - coverage/prioritize/cancer/az-cancer-panel.bed.gz.tbi\n      - coverage/prioritize/cancer/civic-2018-12-27.bed.gz\n      - coverage/prioritize/cancer/civic-2018-12-27.bed.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg19/purecn_mappability.yaml",
    "content": "# mappability regions for purecn analyses:\n# http://bioconductor.org/packages/release/bioc/vignettes/PureCN/inst/doc/Quick.html\n# file size: 99M\n---\nattributes:\n  name: purecn_mappability\n  version: 20200928\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p coverage/mappability\n        cd coverage/mappability\n        wget --no-check-certificate -c http://hgdownload.cse.ucsc.edu/goldenPath/hg19/encodeDCC/wgEncodeMapability/wgEncodeCrgMapabilityAlign100mer.bigWig\n        cd ..\n    recipe_outfiles:\n      - coverage/mappability/wgEncodeCrgMapabilityAlign100mer.bigWig\n"
  },
  {
    "path": "ggd-recipes/hg19/rmsk.yaml",
    "content": "# repeat elements location on the genome\n---\nattributes:\n  name: rmsk\n  version: 20180319\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p coverage/problem_regions/repeats\n        cd coverage/problem_regions/repeats\n        wget  --random-wait --retry-connrefused -q -c -O rmsk.txt.gz http://hgdownload.soe.ucsc.edu/goldenPath/hg19/database/rmsk.txt.gz\n        zcat rmsk.txt.gz | awk '{print $6\"\\t.\\trepeat\\t\"$7+1\"\\t\"$8+1\"\\t.\\t\"$10\"\\t.\\tname \"$12\";\"}' | gzip > rmsk.gtf.gz\n    recipe_outfiles:\n      - coverage/problem_regions/repeats/rmsk.gtf.gz\n"
  },
  {
    "path": "ggd-recipes/hg19/seq.yaml",
    "content": "# hg19 reference genome from Broad bundle\n# ftp://gsapubftp-anonymous:none@ftp.broadinstitute.org/bundle/hg19\n# Include bgzipped fasta file for Ensembl and other tools that support it\n---\nattributes:\n  name: seq\n  version: broad-20120813\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        wget --no-check-certificate -c https://s3.amazonaws.com/biodata/genomes/hg19-seq.tar.gz\n        tar -xzvpf hg19-seq.tar.gz\n        gunzip -c seq/hg19.fa.gz > seq/hg19.fa\n        touch seq/hg19.fa.fai\n        touch seq/hg19.dict\n    recipe_outfiles:\n      - seq/hg19.fa\n      - seq/hg19.fa.fai\n      - seq/hg19.fa.gz\n      - seq/hg19.fa.gz.fai\n      - seq/hg19.fa.gz.gzi\n      - seq/hg19.dict\n"
  },
  {
    "path": "ggd-recipes/hg19/simple_repeat.yaml",
    "content": "# UCSC repeats track for purecn variant filtration:\n---\nattributes:\n  name: simple_repeat\n  version: 20200928\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p coverage/problem_regions/repeats\n        cd coverage/problem_regions/repeats\n        wget --random-wait --retry-connrefused -q -c -O simple_repeat.txt.gz http://hgdownload.soe.ucsc.edu/goldenPath/hg19/database/simpleRepeat.txt.gz\n        zcat simple_repeat.txt.gz | awk -F '\\t' '{print $2\"\\t\"$3\"\\t\"$4\"\\t\"$5\"\\t\"$11\"\\t.\"}' > simple_repeat.bed\n    recipe_outfiles:\n      - coverage/problem_regions/repeats/simple_repeat.bed\n"
  },
  {
    "path": "ggd-recipes/hg19/topmed.yaml",
    "content": "# Topmed WGS allele frequencies: https://bravo.sph.umich.edu/freeze3a/hg19/\n---\nattributes:\n  name: topmed\n  version: freeze3a\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=ftp://ftp.ensembl.org/pub/data_files/homo_sapiens/GRCh37/variation_genotype/TOPMED_GRCh37.vcf.gz \n        remap_url=https://raw.githubusercontent.com/dpryan79/ChromosomeMappings/master/GRCh37_ensembl2UCSC.txt\n        ref=../seq/hg19.fa\n        mkdir -p variation\n        export TMPDIR=`pwd`\n        wget --no-check-certificate -qO- $remap_url | awk '{if($1!=$2) print \"s/^\"$1\"/\"$2\"/g\"}' > remap.sed\n        wget -c -O - $url | gunzip -c | sed -f remap.sed | gsort -m 3000 /dev/stdin $ref.fai | vt decompose -s - | vt normalize -r $ref -n - | vt uniq - |  bgzip -c > variation/topmed.vcf.gz\n        tabix -f -p vcf variation/topmed.vcf.gz\n    recipe_outfiles:\n      - variation/topmed.vcf.gz\n      - variation/topmed.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg19/transcripts.yaml",
    "content": "# Transcript, annotation and indices for RNA-seq analysis\n# Prepared from Ensembl transcripts using: \n# https://github.com/chapmanb/cloudbiolinux/blob/master/utils/prepare_tx_gff.py\n# Built from ftp://ftp.ensembl.org/pub/release-67/gtf/homo_sapiens/Homo_sapiens.GRCh37.67.gtf.gz\n# translated to UCSC chromosomes, with genes in haplotype patches discarded.\n# See gtf.yaml for details.\n---\nattributes:\n  name: transcripts\n  version: 2019-03-01_87\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/biodata/annotation/hg19-rnaseq-2019-03-01_75.tar.xz\n        wget -c -N --no-check-certificate $baseurl\n        xz -dc *-rnaseq-*.tar.xz | tar -xpf -\n        mv */rnaseq-* rnaseq\n    recipe_outfiles:\n      - rnaseq/ref-transcripts.bed\n      - rnaseq/ref-transcripts.fa\n      - rnaseq/ref-transcripts.genePred\n      - rnaseq/ref-transcripts.gtf\n      - rnaseq/ref-transcripts.gtf.db\n      - rnaseq/ref-transcripts-mask.gtf\n      - rnaseq/ref-transcripts.refFlat\n      - rnaseq/ref-transcripts-splicesites.txt\n      - rnaseq/rRNA.gtf\n      - rnaseq/rRNA.gtf.db\n      - rnaseq/rRNA.interval_list\n      - rnaseq/tx2gene.csv\n"
  },
  {
    "path": "ggd-recipes/hg19/twobit.yaml",
    "content": "# UCSC twobit reference\n---\nattributes:\n  name: twobit\n  version: broad-20120813\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p ucsc\n        cd ucsc\n        faToTwoBit ../../seq/hg19.fa hg19.2bit\n    recipe_outfiles:\n      - ucsc/hg19.2bit\n"
  },
  {
    "path": "ggd-recipes/hg19/varpon.yaml",
    "content": "# Panels of normals (PONs) for variant calling\n# germline_het_pon -- Locations of likely germline heterogygote sites\n---\nattributes:\n  name: varpon\n  version: 20181105\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p variation\n        wget --no-check-certificate -O - 'https://s3.amazonaws.com/biodata/collections/hg19/variation/germline_het_pon.bed.gz' > variation/germline_het_pon.bed.gz\n        wget --no-check-certificate -O - 'https://s3.amazonaws.com/biodata/collections/hg19/variation/germline_het_pon.bed.gz.tbi' > variation/germline_het_pon.bed.gz.tbi\n    recipe_outfiles:\n      - variation/germline_het_pon.bed.gz\n      - variation/germline_het_pon.bed.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg19/viral.yaml",
    "content": "# Virus reference genomes, prepared for bwa alignment\n# Taken from TCGA GDC reference genome\n# https://gdc.cancer.gov/about-data/data-harmonization-and-generation/gdc-reference-files\n---\nattributes:\n  name: viral\n  version: 2019.12.03\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=http://s3.amazonaws.com/biodata/viral/GRCh37-gdc-viral.tar.gz\n        mkdir -p viral\n        wget --no-check-certificate -c $url\n        tar -xzvpf *.tar.gz\n        mv *-gdc-viral/* viral\n    recipe_outfiles:\n      - viral/gdc-viral.fa\n      - viral/gdc-viral.dict\n      - viral/gdc-viral.fa.ann\n      - viral/gdc-viral.fa.fai\n      - viral/gdc-viral.fa.pac\n      - viral/gdc-viral.fa.bwt\n      - viral/gdc-viral.fa.amb\n      - viral/gdc-viral.fa.sa\n"
  },
  {
    "path": "ggd-recipes/hg38/1000g_indels.yaml",
    "content": "# 1000 genomes phase 1 indel calls\n#\n# From the Broad v2.8 resource bundle\n# https://www.broadinstitute.org/gatk/guide/article.php?id=1213\n# lifted over to hg38 coordinates by Alison Meynert\n# https://github.com/chapmanb/bcbio-nextgen/issues/817\n---\nattributes:\n  name: 1000g_indels\n  version: 2.8_hg38_20150522\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=https://s3.amazonaws.com/biodata/hg38_bundle\n        base=1000G_phase1.indels.b38.primary_assembly\n        new=1000G_phase1.indels\n        mkdir -p variation\n        for suffix in .vcf.gz .vcf.gz.tbi\n        do\n          [[ -f variation/$new$suffix ]] || wget -c -O variation/$new$suffix $url/$base$suffix\n        done\n    recipe_outfiles:\n      - variation/1000G_phase1.indels.vcf.gz\n      - variation/1000G_phase1.indels.vcf.gz.tbi"
  },
  {
    "path": "ggd-recipes/hg38/1000g_omni_snps.yaml",
    "content": "# 1000 genomes OMNI 2.5 array calls\n# http://www.1000genomes.org/category/frequently-asked-questions/omni\n#\n# From the Broad hg38 bundle:\n# ftp://gsapubftp-anonymous:none@ftp.broadinstitute.org/bundle/\n# https://www.broadinstitute.org/gatk/guide/article.php?id=1213\n---\nattributes:\n  name: 1000g_omni_snps\n  version: 20160105\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=ftp://gsapubftp-anonymous:none@ftp.broadinstitute.org/bundle/hg38/\n        new=1000G_omni2.5\n        base=$new.hg38\n        mkdir -p variation\n        for suffix in .vcf.gz .vcf.gz.tbi\n        do\n          [[ -f variation/$new$suffix ]] || wget --no-check-certificate -c -O variation/$new$suffix $url/$base$suffix\n        done\n    recipe_outfiles:\n      - variation/1000G_omni2.5.vcf.gz\n      - variation/1000G_omni2.5.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg38/1000g_snps.yaml",
    "content": "# 1000 genomes phase 1 SNP calls\n#\n# From the Broad hg38 bundle:\n# ftp://gsapubftp-anonymous:none@ftp.broadinstitute.org/bundle/\n# https://www.broadinstitute.org/gatk/guide/article.php?id=1213\n---\nattributes:\n  name: 1000g_snps\n  version: 20160105\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=ftp://gsapubftp-anonymous:none@ftp.broadinstitute.org/bundle/hg38/\n        new=1000G_phase1.snps.high_confidence\n        base=$new.hg38\n        mkdir -p variation\n        for suffix in .vcf.gz .vcf.gz.tbi\n        do\n          [[ -f variation/$new$suffix ]] || wget --no-check-certificate -c -O variation/$new$suffix $url/$base$suffix\n        done\n    recipe_outfiles:\n      - variation/1000G_phase1.snps.high_confidence.vcf.gz\n      - variation/1000G_phase1.snps.high_confidence.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg38/ACMG56_genes.yaml",
    "content": "# ACMG regions: http://iobio.io/2016/03/29/acmg56/\n# https://github.com/lpantano/custom_genome_annotation\n---\nattributes:\n  name: ACMG56_genes\n  version: 20160726\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://raw.githubusercontent.com/lpantano/custom_genome_annotation/master/acmg56.hg38.bed\n        dir=coverage/prioritize/actionable\n        mkdir -p $dir\n        cd $dir\n        wget --no-check-certificate -c -O ACMG56.bed $baseurl\n    recipe_outfiles:\n      - coverage/prioritize/actionable/ACMG56.bed\n"
  },
  {
    "path": "ggd-recipes/hg38/RADAR.yaml",
    "content": "# RADAR RNA editing events: (http://www.stanford.edu/~gokulr/database/Human_AG_all_hg19_v2.txt)\n# installs a BED file of the RADAR events, preconverted to hg38\n# how this file was created:\n#\n# 1. Download the hg19 RADAR file and converted to BED format.\n# url=http://www.stanford.edu/~gokulr/database/Human_AG_all_hg19_v2.txt\n# mkdir -p editing\n# cd editing\n# wget -qO- $url | awk 'BEGIN{OFS=\"\\t\"} {print $1,$2,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11}' | sed \"s/position  position/start  end/\" > RADAR-hg19.bed\n# cd ../\n# 2. Convert RADAR BED file to BED3 format.\n# cut -f1,2,3 RADAR-hg19.bed > RADAR-hg19.tmp.bed\n# 3. Get the Crossmap hg19 to hg38 chain file:\n# wget http://hgdownload.soe.ucsc.edu/goldenPath/hg19/liftOver/hg19ToHg38.over.chain.gz\n# 4. Run Crossmap:\n# Crossmap.py bed hg19ToHg38.over.chain.gz RADAR-hg19.tmp.bed > conversions.bed\n# 5. Merge original RADAR file with the Crossmap output\n# wget https://raw.githubusercontent.com/roryk/seqscripts/master/radar-convert-annotations.py\n# python radar-convert-annotations.py RADAR-hg19.bed conversions.bed RADAR-hg38.bed.gz\n\n---\nattributes:\n  name: RADAR\n  version: v2-20180202\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=https://s3.amazonaws.com/biodata/annotation/RADAR/hg38/RADAR.bed.gz\n        mkdir -p editing\n        cd editing\n        wget --no-check-certificate -q $url\n        wget --no-check-certificate -q $url.tbi\n    recipe_outfiles:\n      - editing/RADAR.bed.gz\n      - editing/RADAR.bed.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg38/README.md",
    "content": "Full hg38/GRCh38 reference genome distributed by 1000 genomes\nDerived from NCBI set with HLA and decoy alternative alleles\nhttp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/GRCh38_reference_genome/\n"
  },
  {
    "path": "ggd-recipes/hg38/af_only_gnomad.yaml",
    "content": "# af_only_gnomad file for t-only variant calling with mutect2\n# From the Broad hg38 bundle:\n# ftp://gsapubftp-anonymous@ftp.broadinstitute.org/bundle/Mutect2/af-only-gnomad.hg38.vcf.gz\n---\nattributes:\n  name: af_only_gnomad\n  version: 20180129\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=ftp://gsapubftp-anonymous@ftp.broadinstitute.org/bundle/Mutect2/\n        base=af-only-gnomad.hg38\n        new=af_only_gnomad\n        mkdir -p variation\n        for suffix in .vcf.gz .vcf.gz.tbi\n        do\n          [[ -f variation/$new$suffix ]] || wget --no-check-certificate -c -O variation/$new$suffix $url/$base$suffix\n        done\n    recipe_outfiles:\n      - variation/af_only_gnomad.vcf.gz\n      - variation/af_only_gnomad.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg38/bwa.yaml",
    "content": "# Full hg38/GRCh38 reference genome distributed by 1000 genomes\n# Derived from NCBI set with HLA and decoy alternative alleles\n# http://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/GRCh38_reference_genome/\n# Pre-build bwa indices\n---\nattributes:\n  name: bwa\n  version: 1000g-20150219 \nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/GRCh38_reference_genome\n        base=GRCh38_full_analysis_set_plus_decoy_hla.fa\n        new=hg38.fa\n        mkdir -p bwa\n        for suffix in .bwt .amb .ann .pac .sa .alt\n        do\n          [[ -f bwa/$new$suffix ]] || wget -c -O bwa/$new$suffix $url/$base$suffix\n        done\n    recipe_outfiles:\n      - bwa\n"
  },
  {
    "path": "ggd-recipes/hg38/canonical_cancer_99.txt",
    "content": "ENSG00000007312\tENST00000006750\nENSG00000012048\tENST00000357654\nENSG00000049618\tENST00000636930\nENSG00000066468\tENST00000358487\nENSG00000068078\tENST00000440486\nENSG00000077782\tENST00000447712\nENSG00000085999\tENST00000371975\nENSG00000087460\tENST00000371085\nENSG00000091831\tENST00000206249\nENSG00000105976\tENST00000397752\nENSG00000115392\tENST00000233741\nENSG00000139144\tENST00000538779\nENSG00000141510\tENST00000269305\nENSG00000142208\tENST00000649815\nENSG00000147889\tENST00000304494\nENSG00000149554\tENST00000438015\nENSG00000168769\tENST00000380013\nENSG00000172936\tENST00000650905\nENSG00000183765\tENST00000404276\nENSG00000185379\tENST00000345365\nENSG00000221914\tENST00000380737\nENSG00000245848\tENST00000498907\n"
  },
  {
    "path": "ggd-recipes/hg38/capture_regions.yaml",
    "content": "# Common sets of capture region BED files for different platforms\n---\nattributes:\n  name: capture_regions\n  version: 20161202\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://raw.githubusercontent.com/AstraZeneca-NGS/reference_data/master/hg38/bed\n        files=\"Exome-Agilent_V5_Clinical.bed Exome-Agilent_V6.bed Exome-AZ_V2.bed Exome-IDT_V1.bed\n               Exome-MedExome.bed Exome-NGv3.bed\"\n        mkdir -p coverage/capture_regions\n        cd coverage/capture_regions\n        for fname in $files\n        do\n          wget --no-check-certificate -c $baseurl/$fname\n        done\n    recipe_outfiles:\n      - coverage/capture_regions/Exome-Agilent_V5_Clinical.bed\n      - coverage/capture_regions/Exome-Agilent_V6.bed\n      - coverage/capture_regions/Exome-AZ_V2.bed\n      - coverage/capture_regions/Exome-IDT_V1.bed\n      - coverage/capture_regions/Exome-MedExome.bed\n      - coverage/capture_regions/Exome-NGv3.bed\n"
  },
  {
    "path": "ggd-recipes/hg38/ccds.yaml",
    "content": "# Consensus CDS (CCDS https://www.ncbi.nlm.nih.gov/projects/CCDS/CcdsBrowse.cgi) regions with two adjustments:\n# \n# - 2 bps added to internal introns to capture canonical splice acceptor/donor sites\n# - Multiple transcripts from a single gene are merged into a single all inclusive gene entry.\n---\nattributes:\n  name: ccds\n  version: r20\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/biodata/coverage/ccds-hg38-r20.bed.gz\n        outdir=coverage/coding\n        mkdir -p $outdir\n        cd $outdir\n        wget --no-check-certificate -c -O ccds.bed.gz $baseurl\n        wget --no-check-certificate -c -O ccds.bed.gz.tbi $baseurl.tbi\n    recipe_outfiles:\n      - coverage/coding/ccds.bed.gz\n      - coverage/coding/ccds.bed.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg38/clinvar.yaml",
    "content": "# ClinVar: http://www.clinvar.com/\n# UCSFify name sed magic from: https://github.com/mmarchin/utilities/blob/master/ucscify.sh\n---\nattributes:\n  name: clinvar\n  version: 20210110\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        release=20210110\n        # the latest release\n        baseurl=https://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/clinvar.vcf.gz\n        mkdir -p variation\n        wget --no-check-certificate -c -O variation/clinvar-orig.vcf.gz $baseurl\n        rm -f variation/clinvar.vcf.gz\n        rm -f variation/clinvar.vcf.gz.tbi\n        zcat variation/clinvar-orig.vcf.gz | sed \"s/^\\([0-9]\\+\\)\\t/chr\\1\\t/g\" | sed \"s/^MT/chrM/g\" | sed \"s/^X/chrX/g\" | sed \"s/^Y/chrY/g\" | bgzip -c > variation/clinvar.vcf.gz\n        tabix -f -p vcf variation/clinvar.vcf.gz\n    recipe_outfiles:\n      - variation/clinvar.vcf.gz\n      - variation/clinvar.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg38/coverage.yaml",
    "content": "# Problematic and other coverage regions of interest\n#\n# Low complexity regions from supplemental data of Heng Li's work on\n# reducing variant calling artifacts:\n#\n# http://bioinformatics.oxfordjournals.org/content/30/20/2843\n# https://github.com/lh3/varcmp/tree/master/scripts\n#\n# Centromere and other exclusion regions for SV calling from:\n# https://github.com/tobiasrausch/delly/blob/master/excludeTemplates/human.hg38.excl.tsv\n---\nattributes:\n  name: coverage\n  version: 2018-10-16\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        repeats=coverage/problem_regions/repeats\n        mkdir -p $repeats\n        url=https://gist.githubusercontent.com/chapmanb/4c40f961b3ac0a4a22fd/raw/2025f3912a477edc597e61d911bd1044dc943440/sv_repeat_telomere_centromere.bed\n        wget --no-check-certificate -O - $url > $repeats/sv_repeat_telomere_centromere.bed\n        url=https://github.com/lh3/varcmp/raw/bb5b616526c5c3ecb46abfd9877e1bd6d50d1802/scripts/LCR-hs38.bed.gz\n        out=$repeats/LCR.bed.gz\n        wget --no-check-certificate -O $out -c $url\n        tabix -f -p bed $out\n        wget --no-check-certificate -O $repeats/polyx.bed.gz http://s3.amazonaws.com/biodata/coverage/problem_regions/polyx/polyX-hg38.bed.gz\n        wget --no-check-certificate -O $repeats/polyx.bed.gz.tbi http://s3.amazonaws.com/biodata/coverage/problem_regions/polyx/polyX-hg38.bed.gz.tbi\n        gc=coverage/gc\n        mkdir -p $gc\n        wget --no-check-certificate -O - https://s3.amazonaws.com/biodata/collections/hg38/coverage/gc/GC_profile.1000bp.cnp > $gc/GC_profile.1000bp.cnp\n    recipe_outfiles:\n      - coverage/problem_regions/repeats/sv_repeat_telomere_centromere.bed\n      - coverage/problem_regions/repeats/LCR.bed.gz\n      - coverage/problem_regions/repeats/LCR.bed.gz.tbi\n      - coverage/problem_regions/repeats/polyx.bed.gz\n      - coverage/problem_regions/repeats/polyx.bed.gz.tbi\n      - coverage/gc/GC_profile.1000bp.cnp\n"
  },
  {
    "path": "ggd-recipes/hg38/dbnsfp.yaml",
    "content": "# google drive downloads are unreliable - too many requests, try again in 24h\n# using ftp ftp://dbnsfp:dbnsfp@dbnsfp.softgenetics.com/dbNSFP4.1a.zip\n# bumped version because 3.5a is not supported anymore - 25G download instead of 16G\n---\nattributes:\n  name: dbnsfp\n  version: 4.1a\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p variation\n        cd variation\n        wget -c ftp://dbnsfp:dbnsfp@dbnsfp.softgenetics.com/dbNSFP4.1a.zip\n        if [ ! -f dbNSFP.txt.gz ]; then\n          UNPACK_DIR=`pwd`/tmpunpack\n          mkdir -p $UNPACK_DIR\n          7z e dbNSFP*.zip \"dbNSFP*_variant.chrM.gz\"\n          gunzip dbNSFP*_variant.chrM.gz\n          head -n1 dbNSFP*_variant.chrM > $UNPACK_DIR/header.txt\n          rm dbNSFP*_variant.chrM\n          # unzip only files with chromosomal info, eg. skip genes and readme.\n          cat $UNPACK_DIR/header.txt > dbNSFP.txt\n          7z e dbNSFP*.zip \"dbNSFP*_variant.chr*.gz\" -so | gunzip -c | grep -v '^#chr' | sort -T $UNPACK_DIR -k1,1 -k2,2n >> dbNSFP.txt\n          bgzip dbNSFP.txt\n          #extract readme file, used by VEP plugin to add vcf header info\n          7z e dbNSFP*.zip \"*readme.txt\" -so > dbNSFP.readme.txt\n        fi\n        # index in position 1 and 2\n        tabix -f -s 1 -b 2 -e 2 -c '#' dbNSFP.txt.gz\n        rm -f ${UNPACK_DIR}/* && rmdir $UNPACK_DIR\n        rm -f dbNSFP*.zip\n    recipe_outfiles:\n      - variation/dbNSFP.txt.gz\n      - variation/dbNSFP.txt.gz.tbi\n      - variation/dbNSFP.readme.txt\n"
  },
  {
    "path": "ggd-recipes/hg38/dbscsnv.yaml",
    "content": "---\nattributes:\n  name: dbscsnv\n  version: 1.1\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=ftp://dbnsfp:dbnsfp@dbnsfp.softgenetics.com/dbscSNV1.1.zip\n        mkdir -p variation\n        cd variation\n        UNPACK_DIR=`pwd`/tmp-unpack\n        mkdir -p $UNPACK_DIR\n        if [ ! -f dbscSNV.txt.gz.tbi ]; then\n          wget -c -N $baseurl\n          # [awk] check if hg38 position exists, if so, move data to col 1 and 2, then print\n          # and sort on first and second column\n          unzip -p dbscSNV*.zip \"dbscSNV*.chr*\" | awk -F $'\\t' 'BEGIN {OFS = FS} (FNR==1){print} ($1~/^chr/){next} ($5 != \".\" && $6 != \".\"){$1=$5;$2=$6;print | \"sort -T \"UNPACK_DIR\" -k1,1 -k2,2n\"}' \"UNPACK_DIR=$UNPACK_DIR\" | bgzip -c > dbscSNV.txt.gz\n          tabix -s 1 -b 2 -e 2 -S 1 dbscSNV.txt.gz\n        fi\n        rm -f $UNPACK_DIR/* && rm -rf $UNPACK_DIR\n        rm -f dbscSNV*.zip\n    recipe_outfiles:\n      - variation/dbscSNV.txt.gz\n      - variation/dbscSNV.txt.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg38/dbsnp.yaml",
    "content": "# install time: 1h20min\n# install size: 16G\n# order of files in recipe_outfiles is important\n# first come symlinks, than real files.\n# otherwise real files get copied first, and symlinks become corrupt\n---\nattributes:\n  name: dbsnp\n  version: 156-20231017\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        build=156\n        version=GCF_000001405.40\n        url=http://ftp.ncbi.nih.gov/snp/archive/b$build/VCF/$version.gz\n        remap_url=https://gist.githubusercontent.com/matthdsm/f833aedd2d67e28013ff1d171c70f4ee/raw/442a45ed3ddc6e85c66c5e58e0fa78e16a0821c8/refseq2ucsc.tsv\n        mkdir -p variation\n        cd variation\n        wget -c -O dbsnp-${build}-orig.vcf.gz $url\n        wget -c -O dbsnp-${build}-orig.vcf.gz.tbi $url.tbi\n        [[ -f dbsnp-$build.vcf.gz ]] || bcftools annotate -Ou --rename-chrs $remap_url dbsnp-$build-orig.vcf.gz |\\\n        bcftools sort -m 1G -Oz -T . -o dbsnp-$build.vcf.gz && \\\n        tabix -f -p vcf -C dbsnp-${build}.vcf.gz\n        tabix -f -p vcf dbsnp-${build}.vcf.gz        \n        ln -sf dbsnp-${build}.vcf.gz dbsnp.vcf.gz\n        ln -sf dbsnp-${build}.vcf.gz.tbi dbsnp.vcf.gz.tbi\n        ln -sf dbsnp-${build}.vcf.gz.csi dbsnp.vcf.gz.csi\n        cd ..\n    recipe_outfiles:\n      - variation/dbsnp.vcf.gz\n      - variation/dbsnp.vcf.gz.csi\n      - variation/dbsnp.vcf.gz.tbi\n      - variation/dbsnp-156.vcf.gz\n      - variation/dbsnp-156.vcf.gz.csi\n      - variation/dbsnp-156.vcf.gz.tbi \n"
  },
  {
    "path": "ggd-recipes/hg38/dream-syn3-crossmap.yaml",
    "content": "# Truth sets from the DREAM genomic challenge 3\n# https://www.synapse.org/#!Synapse:syn312572/wiki/62018\n#\n# Originally retrieved from https://www.synapse.org/#!Synapse:syn2177211\n# and prepared with this script:\n# https://github.com/chapmanb/bcbio-nextgen/blob/master/scripts/utils/format_dream_truthset.py\n#\n# Converted to hg38 coordinates with CrossMap:\n# https://github.com/hbc/giab_remap_38\n---\nattributes:\n  name: dream-syn3-crossmap\n  version: 2014-08-04\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/dream-syn3-crossmap\n        orig=synthetic_challenge_set3_tumor_20pctmasked_truth-crossmap-hg38\n        mkdir -p $dir\n        wget --no-check-certificate -c https://s3.amazonaws.com/bcbio_nextgen/dream/$orig.tar.gz\n        tar -xzvpf ${orig}.tar.gz\n        cp $orig/${orig}.vcf.gz $dir/truth_small_variants.vcf.gz\n        cp $orig/${orig}.vcf.gz.tbi $dir/truth_small_variants.vcf.gz.tbi\n        cp $orig/${orig}_regions.bed $dir/truth_regions.bed\n        cp $orig/${orig}_sv_DEL.bed $dir/truth_DEL.bed \n        cp $orig/${orig}_sv_DUP.bed $dir/truth_DUP.bed\n        cp $orig/${orig}_sv_INV.bed $dir/truth_INV.bed\n        cp $orig/${orig}_sv_INV.bed $dir/truth_INS.bed\n    recipe_outfiles:\n      - validation/dream-syn3-crossmap/truth_small_variants.vcf.gz\n      - validation/dream-syn3-crossmap/truth_small_variants.vcf.gz.tbi\n      - validation/dream-syn3-crossmap/truth_regions.bed\n      - validation/dream-syn3-crossmap/truth_DEL.bed\n      - validation/dream-syn3-crossmap/truth_DUP.bed\n      - validation/dream-syn3-crossmap/truth_INV.bed\n      - validation/dream-syn3-crossmap/truth_INS.bed\n"
  },
  {
    "path": "ggd-recipes/hg38/dream-syn4-crossmap.yaml",
    "content": "# Truth sets from the DREAM genomic challenge 4\n# https://www.synapse.org/#!Synapse:syn312572/wiki/62018\n#\n# Originally retrieved from https://www.synapse.org/#!Synapse:syn2177211\n# and prepared with this script:\n# https://github.com/chapmanb/bcbio-nextgen/blob/master/scripts/utils/format_dream_truthset.py\n#\n# Converted to hg38 coordinates with CrossMap:\n# https://github.com/hbc/giab_remap_38\n---\nattributes:\n  name: dream-syn4-crossmap\n  version: 2014-08-09\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/dream-syn4-crossmap\n        orig=synthetic_challenge_set4_tumour_25pctmasked_truth-crossmap-hg38\n        mkdir -p $dir\n        wget --no-check-certificate -c https://s3.amazonaws.com/bcbio_nextgen/dream/synthetic_challenge_set4_tumour_25pctmasked_truth-crossmap-hg38.tar.gz\n        tar -xzvpf ${orig}.tar.gz\n        cp $orig/${orig}.vcf.gz $dir/truth_small_variants.vcf.gz\n        cp $orig/${orig}.vcf.gz.tbi $dir/truth_small_variants.vcf.gz.tbi\n        cp $orig/${orig}_regions.bed $dir/truth_regions.bed\n        cp $orig/${orig}_sv_DEL.bed $dir/truth_DEL.bed \n        cp $orig/${orig}_sv_DUP.bed $dir/truth_DUP.bed\n        cp $orig/${orig}_sv_INV.bed $dir/truth_INV.bed\n    recipe_outfiles:\n      - validation/dream-syn4-crossmap/truth_small_variants.vcf.gz\n      - validation/dream-syn4-crossmap/truth_small_variants.vcf.gz.tbi\n      - validation/dream-syn4-crossmap/truth_regions.bed\n      - validation/dream-syn4-crossmap/truth_DEL.bed\n      - validation/dream-syn4-crossmap/truth_DUP.bed\n      - validation/dream-syn4-crossmap/truth_INV.bed"
  },
  {
    "path": "ggd-recipes/hg38/effects_transcripts.yaml",
    "content": "# Custom transcripts for effects prediction\n# Tweaks canonical transcripts in effects predictors like snpEff.\n# the new table was created by extracted canonical trascripts from biomaRt:\n# https://github.com/naumenko-sa/bioscripts/blob/master/gene_panels/genes.R\n---\nattributes:\n  name: effects_transcript\n  version: 2021-12-08\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p snpeff/transcripts\n        cd snpeff/transcripts\n        wget --no-check-certificate -O canonical_cancer.txt https://raw.githubusercontent.com/chapmanb/cloudbiolinux/master/ggd-recipes/hg38/canonical_cancer_99.txt\n    recipe_outfiles:\n      - snpeff/transcripts/canonical_cancer.txt\n"
  },
  {
    "path": "ggd-recipes/hg38/ericscript.yaml",
    "content": "---\nattributes:\n  name: ericscript\n  version: 84\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        # Uses Google Drive for faster download with tricks from\n        # http://stackoverflow.com/a/38937732/252589\n        ggID='0B9s__vuJPvIiUGt1SnFMZFg4TlE'\n        ggURL='https://drive.google.com/uc?export=download'\n        mkdir -p rnaseq/ericscript\n        cd rnaseq/ericscript\n        filename=\"$(curl -k -sc tmp-gcokie \"${ggURL}&id=${ggID}\" | grep -o '=\"uc-name.*</span>' | sed 's/.*\">//;s/<.a> .*//')\"\n        getcode=\"$(awk '/_warning_/ {print $NF}' tmp-gcokie)\"\n        curl -k -Lb tmp-gcokie \"${ggURL}&confirm=${getcode}&id=${ggID}\" -o \"${filename}\" -C -\n        tar -xjvpf ericscript_db_homosapiens_ensembl84.tar.bz2\n    recipe_outfiles:\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl84/data/homo_sapiens/EnsemblGene.Reference.fa.amb\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl84/data/homo_sapiens/allseq.fa.pac\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl84/data/homo_sapiens/allseq.fa.bwt\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl84/data/homo_sapiens/allseq.fa\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl84/data/homo_sapiens/EnsemblGene.Sequences.RData\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl84/data/homo_sapiens/EnsemblGene.GeneInfo.RData\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl84/data/homo_sapiens/EnsemblGene.Reference.fa\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl84/data/homo_sapiens/EnsemblGene.GenePosition.RData\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl84/data/homo_sapiens/allseq.fa.sa\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl84/data/homo_sapiens/allseq.fa.amb\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl84/data/homo_sapiens/EnsemblGene.Paralogs.RData\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl84/data/homo_sapiens/EnsemblGene.GeneNames.RData\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl84/data/homo_sapiens/allseq.fa.ann\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl84/data/homo_sapiens/EnsemblGene.Reference.fa.bwt\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl84/data/homo_sapiens/EnsemblGene.Reference.fa.sa\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl84/data/homo_sapiens/EnsemblGene.Transcripts.RData\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl84/data/homo_sapiens/EnsemblGene.Reference.fa.ann\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl84/data/homo_sapiens/EnsemblGene.Reference.fa.pac\n      - rnaseq/ericscript/ericscript_db_homosapiens_ensembl84/data/homo_sapiens/EnsemblGene.Structures.RData\n"
  },
  {
    "path": "ggd-recipes/hg38/esp.yaml",
    "content": "# Exome Sequencing Project (ESP): http://evs.gs.washington.edu/EVS/\n# http://evs.gs.washington.edu/evs_bulk_data/ESP6500SI-V2-SSA137.GRCh38-liftover.snps_indels.vcf.tar.gz\n# Lifted over to hg38 and prepped with utils/prep_esp_hg38.py\n---\nattributes:\n  name: esp\n  version: ESP6500SI-V2-fix\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=https://s3.amazonaws.com/biodata/hg38_bundle/ESP6500SI-V2-hg38.vcf.gz\n        mkdir -p variation\n        wget -c -O variation/esp.vcf.gz $url\n        wget -c -O variation/esp.vcf.gz.tbi $url.tbi\n    recipe_outfiles:\n      - variation/esp.vcf.gz\n      - variation/esp.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg38/exac.yaml",
    "content": "# ExAC VCF sites: http://exac.broadinstitute.org/\n# Lifted over to hg38 by the Ensembl variation team\n# Script cleans up to use chr-style naming and sort by ref file\n---\nattributes:\n  name: exac\n  version: 0.3\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=http://ftp.ensembl.org/pub/data_files/homo_sapiens/GRCh38/variation_genotype/ExAC.0.3.GRCh38.vcf.gz\n        remap_url=http://raw.githubusercontent.com/dpryan79/ChromosomeMappings/master/GRCh38_ensembl2UCSC.txt\n        ref=../seq/hg38.fa\n        mkdir -p variation\n        wget --no-check-certificate -qO- $remap_url | awk '{if($1!=$2) print \"s/^\"$1\"/\"$2\"/g\"}' > remap.sed\n        export TMPDIR=`pwd`\n        wget -c -O - $url | gunzip -c | sed -f remap.sed | grep -v \"##contig=\" | gsort -m 500 /dev/stdin $ref.fai | vt decompose -s - | vt normalize -r $ref -n - | vt uniq - | bgzip -c > variation/exac.vcf.gz\n        tabix -f -p vcf variation/exac.vcf.gz\n    recipe_outfiles:\n      - variation/exac.vcf.gz\n      - variation/exac.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg38/fusion-blacklist.yaml",
    "content": "# fusion blacklist regions from arriba\n---\nattributes:\n  name: fusion-blacklist\n  version: 2\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        version=v1.1.0\n        url=https://github.com/suhrig/arriba/releases/download/${version}/arriba_${version}.tar.gz\n        arriba_tarball=arriba_${version}.tar.gz\n        outdir=rnaseq/fusion-blacklist\n        mkdir -p ${outdir}\n        wget --no-check-certificate -c -O ${arriba_tarball} ${url}\n        tar zxvf ${arriba_tarball}\n        mv arriba_${version}/database/blacklist_hg38_GRCh38_2018-11-04.tsv.gz ${outdir}/arriba-blacklist.tsv.gz\n        mv arriba_${version}/database/cytobands_hg38_GRCh38_2018-02-23.tsv ${outdir}/arriba-cytobands.tsv\n        mv arriba_${version}/database/protein_domains_hg38_GRCh38_2018-03-06.gff3 ${outdir}/arriba-protein-domains.gff3\n        mv arriba_${version}/database/protein_domains_hg38_GRCh38_2018-03-06.gff3.idx ${outdir}/arriba-protein-domains.gff3.idx\n        rm -r arriba_${version}\n        rm ${arriba_tarball}\n    recipe_outfiles:\n      - rnaseq/fusion-blacklist/arriba-blacklist.tsv.gz\n      - rnaseq/fusion-blacklist/arriba-cytobands.tsv\n      - rnaseq/fusion-blacklist/arriba-protein-domains.gff3\n      - rnaseq/fusion-blacklist/arriba-protein-domains.gff3.idx\n"
  },
  {
    "path": "ggd-recipes/hg38/genesplicer.yaml",
    "content": "---\nattributes:\n  name: genesplicer\n  version: 2004.04.03\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p variation/genesplicer\n        wget ftp://ftp.ccb.jhu.edu/pub/software/genesplicer/GeneSplicer.tar.gz\n        tar -C variation/genesplicer --strip-components=2 -xvf GeneSplicer.tar.gz GeneSplicer/human\n        rm GeneSplicer.tar.gz\n    recipe_outfiles:\n      - variation/genesplicer/acc1.mar\n      - variation/genesplicer/acc14\n      - variation/genesplicer/acc20\n      - variation/genesplicer/acc26\n      - variation/genesplicer/acc33\n      - variation/genesplicer/acc7\n      - variation/genesplicer/chgd.fasta\n      - variation/genesplicer/don1.mar\n      - variation/genesplicer/don15\n      - variation/genesplicer/don19\n      - variation/genesplicer/don20\n      - variation/genesplicer/don24\n      - variation/genesplicer/embls.seq\n      - variation/genesplicer/exons.list\n      - variation/genesplicer/f.don\n      - variation/genesplicer/outin\n      - variation/genesplicer/score_ex.don\n      - variation/genesplicer/score_in.don\n      - variation/genesplicer/acc13\n      - variation/genesplicer/acc19\n      - variation/genesplicer/acc25\n      - variation/genesplicer/acc32\n      - variation/genesplicer/acc34\n      - variation/genesplicer/acc8\n      - variation/genesplicer/config_file\n      - variation/genesplicer/don10\n      - variation/genesplicer/don16\n      - variation/genesplicer/don2\n      - variation/genesplicer/don23\n      - variation/genesplicer/don9\n      - variation/genesplicer/exons.dat\n      - variation/genesplicer/f.acc\n      - variation/genesplicer/outex\n      - variation/genesplicer/score_ex.acc\n      - variation/genesplicer/score_in.acc\n      - variation/genesplicer/seqs\n"
  },
  {
    "path": "ggd-recipes/hg38/genotype2phenotype.yaml",
    "content": "# G2P is a publicly-accessible online system designed to facilitate the development, validation,\n# curation and distribution of large-scale, evidence-based datasets for use in diagnostic variant filtering.\n# Each G2P entry associates an allelic requirement and a mutational consequence at a defined locus with a disease entity.\n# A confidence level and evidence link are assigned to each entry.\n# https://www.ebi.ac.uk/gene2phenotype/about\n---\nattributes:\n  name: genotype2phenotype\n  version: 20180802\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        cancer_gene_url=\"https://www.ebi.ac.uk/gene2phenotype/downloads/CancerG2P.csv.gz\"\n        DD_gene_url=\"https://www.ebi.ac.uk/gene2phenotype/downloads/DDG2P.csv.gz\"\n\n        wget $cancer_gene_url\n        wget $cancer_gene_url\n        zcat *csv.gz > G2P.csv\n\n    recipe_outfiles:\n      - variation/G2P.csv\n"
  },
  {
    "path": "ggd-recipes/hg38/giab-NA12878-NA24385-somatic.yaml",
    "content": "# Truth set for NA12878/NA24385 somatic mixture\n# http://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/use_cases/mixtures/UMCUTRECHT_NA12878_NA24385_mixture_10052016/\n---\nattributes:\n  name: giab-NA12878-NA24385-somatic\n  version: v3_3_2-2018-07-05\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/giab-NA12878-NA24385-somatic\n        url=https://s3.amazonaws.com/bcbio/giab/NA12878-NA24385/2018-07-05\n        calls=na12878-na24385-somatic-hg38-truth.vcf.gz\n        regions=na12878-na24385-somatic-hg38-truth-regions.bed\n        mkdir -p $dir\n        wget -c $url/$calls -O $dir/truth_small_variants.vcf.gz\n        wget -c $url/$calls.tbi -O $dir/truth_small_variants.vcf.gz.tbi\n        wget -c $url/$regions -O $dir/truth_regions.bed\n    recipe_outfiles:\n      - validation/giab-NA12878-NA24385-somatic/truth_small_variants.vcf.gz\n      - validation/giab-NA12878-NA24385-somatic/truth_small_variants.vcf.gz.tbi\n      - validation/giab-NA12878-NA24385-somatic/truth_regions.bed\n"
  },
  {
    "path": "ggd-recipes/hg38/giab-NA12878-crossmap.yaml",
    "content": "# Truth sets from the Genome in a Bottle project (https://sites.stanford.edu/abms/giab)\n# Converted to hg38 coordinates using scripts from:\n# https://github.com/hbc/giab_remap_38\n# These are created using CrossMap with UCSC chain files. \n---\nattributes:\n  name: giab-NA12878-crossmap\n  version: v2_19_1\n\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/giab-NA12878-crossmap\n        mkdir -p $dir\n        wget --no-check-certificate -c -O $dir/truth_small_variants.vcf.gz https://biodata.s3.amazonaws.com/giab_hg38_remap/GiaB_v2_19-38_crossmap.vcf.gz\n        wget --no-check-certificate -c -O $dir/truth_small_variants.vcf.gz.tbi https://biodata.s3.amazonaws.com/giab_hg38_remap/GiaB_v2_19-38_crossmap.vcf.gz.tbi\n        wget --no-check-certificate -c -O - https://biodata.s3.amazonaws.com/giab_hg38_remap/GiaB_v2_19-38_crossmap-regions.bed | grep -v ^chrM > $dir/truth_regions.bed\n        wget --no-check-certificate -c -O - https://biodata.s3.amazonaws.com/giab_hg38_remap/giab-svclassify-deletions-2015-05-22-crossmap-hg38.bed.gz | gunzip -c > $dir/truth_DEL.bed\n    recipe_outfiles:\n      - validation/giab-NA12878-crossmap/truth_small_variants.vcf.gz\n      - validation/giab-NA12878-crossmap/truth_small_variants.vcf.gz.tbi\n      - validation/giab-NA12878-crossmap/truth_regions.bed\n      - validation/giab-NA12878-crossmap/truth_DEL.bed\n"
  },
  {
    "path": "ggd-recipes/hg38/giab-NA12878-remap.yaml",
    "content": "# Truth sets from the Genome in a Bottle project (https://sites.stanford.edu/abms/giab)\n# Converted to hg38 coordinates using scripts from:\n# https://github.com/hbc/giab_remap_38\n# These are created using NCBI's remap\n---\nattributes:\n  name: giab-NA12878-remap\n  version: v2_19_1\n\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/giab-NA12878-remap\n        mkdir -p $dir\n        wget --no-check-certificate -c -O $dir/truth_small_variants.vcf.gz https://biodata.s3.amazonaws.com/giab_hg38_remap/GiaB_v2_19-38_remap.vcf.gz\n        wget --no-check-certificate -c -O $dir/truth_small_variants.vcf.gz.tbi https://biodata.s3.amazonaws.com/giab_hg38_remap/GiaB_v2_19-38_remap.vcf.gz.tbi\n        wget --no-check-certificate -c -O - https://biodata.s3.amazonaws.com/giab_hg38_remap/GiaB_v2_19-38_remap-regions.bed | grep -v ^chrM > $dir/truth_regions.bed\n    recipe_outfiles:\n      - validation/giab-NA12878-remap/truth_small_variants.vcf.gz\n      - validation/giab-NA12878-remap/truth_small_variants.vcf.gz.tbi\n      - validation/giab-NA12878-remap/truth_regions.bed\n"
  },
  {
    "path": "ggd-recipes/hg38/giab-NA12878.yaml",
    "content": "# Truth sets from Genome in a Bottle for NA12878 -- http://genomeinabottle.org/\n---\nattributes:\n  name: giab-NA12878\n  version: v3_3_2\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/giab-NA12878\n        url=http://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/NA12878_HG001/NISTv3.3.2/GRCh38/\n        calls=HG001_GRCh38_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X-SOLID_CHROM1-X_v.3.3.2_highconf_PGandRTGphasetransfer.vcf.gz\n        regions=HG001_GRCh38_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X-SOLID_CHROM1-X_v.3.3.2_highconf_nosomaticdel_noCENorHET7.bed\n        mkdir -p $dir\n        wget -c $url/$calls -O $dir/truth_small_variants.vcf.gz\n        tabix -f -p vcf $dir/truth_small_variants.vcf.gz\n        wget -c $url/$regions -O $dir/truth_regions.bed\n    recipe_outfiles:\n      - validation/giab-NA12878/truth_small_variants.vcf.gz\n      - validation/giab-NA12878/truth_small_variants.vcf.gz.tbi\n      - validation/giab-NA12878/truth_regions.bed\n"
  },
  {
    "path": "ggd-recipes/hg38/giab-NA24143.yaml",
    "content": "# Truth sets from Genome in a Bottle for NA24143 -- http://genomeinabottle.org/\n---\nattributes:\n  name: giab-NA24143\n  version: v3_3_2\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/giab-NA24143\n        url=https://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/AshkenazimTrio/HG004_NA24143_mother/NISTv3.3.2/GRCh38/\n        calls=HG004_GRCh38_GIAB_highconf_CG-Illfb-IllsentieonHC-Ion-10XsentieonHC_CHROM1-22_v.3.3.2_highconf.vcf.gz\n        regions=HG004_GRCh38_GIAB_highconf_CG-Illfb-IllsentieonHC-Ion-10XsentieonHC_CHROM1-22_v.3.3.2_highconf_noinconsistent.bed\n        mkdir -p $dir\n        wget -c $url/$calls -O $dir/truth_small_variants.vcf.gz\n        tabix -f -p vcf $dir/truth_small_variants.vcf.gz\n        wget -c $url/$regions -O $dir/truth_regions.bed\n    recipe_outfiles:\n      - validation/giab-NA24143/truth_small_variants.vcf.gz\n      - validation/giab-NA24143/truth_small_variants.vcf.gz.tbi\n      - validation/giab-NA24143/truth_regions.bed\n"
  },
  {
    "path": "ggd-recipes/hg38/giab-NA24149.yaml",
    "content": "# Truth sets from Genome in a Bottle for NA24149 -- http://genomeinabottle.org/\n---\nattributes:\n  name: giab-NA24149\n  version: v3_3_2\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/giab-NA24149\n        url=https://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/AshkenazimTrio/HG003_NA24149_father/NISTv3.3.2/GRCh38/\n        calls=HG003_GRCh38_GIAB_highconf_CG-Illfb-IllsentieonHC-Ion-10XsentieonHC_CHROM1-22_v.3.3.2_highconf.vcf.gz\n        regions=HG003_GRCh38_GIAB_highconf_CG-Illfb-IllsentieonHC-Ion-10XsentieonHC_CHROM1-22_v.3.3.2_highconf_noinconsistent.bed\n        mkdir -p $dir\n        wget -c $url/$calls -O $dir/truth_small_variants.vcf.gz\n        tabix -f -p vcf $dir/truth_small_variants.vcf.gz\n        wget -c $url/$regions -O $dir/truth_regions.bed\n    recipe_outfiles:\n      - validation/giab-NA24149/truth_small_variants.vcf.gz\n      - validation/giab-NA24149/truth_small_variants.vcf.gz.tbi\n      - validation/giab-NA24149/truth_regions.bed\n"
  },
  {
    "path": "ggd-recipes/hg38/giab-NA24385.yaml",
    "content": "# Truth sets from Genome in a Bottle for NA24385 -- http://genomeinabottle.org/\n---\nattributes:\n  name: giab-NA24385\n  version: v3_3_2\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/giab-NA24385\n        url=http://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/AshkenazimTrio/HG002_NA24385_son/NISTv3.3.2/GRCh38/\n        calls=HG002_GRCh38_GIAB_highconf_CG-Illfb-IllsentieonHC-Ion-10XsentieonHC-SOLIDgatkHC_CHROM1-22_v.3.3.2_highconf_triophased.vcf.gz\n        regions=HG002_GRCh38_GIAB_highconf_CG-Illfb-IllsentieonHC-Ion-10XsentieonHC-SOLIDgatkHC_CHROM1-22_v.3.3.2_highconf_noinconsistent.bed\n        mkdir -p $dir\n        wget -c $url/$calls -O $dir/truth_small_variants.vcf.gz\n        tabix -f -p vcf $dir/truth_small_variants.vcf.gz\n        wget -c $url/$regions -O $dir/truth_regions.bed\n    recipe_outfiles:\n      - validation/giab-NA24385/truth_small_variants.vcf.gz\n      - validation/giab-NA24385/truth_small_variants.vcf.gz.tbi\n      - validation/giab-NA24385/truth_regions.bed\n"
  },
  {
    "path": "ggd-recipes/hg38/giab-NA24631.yaml",
    "content": "# Truth sets from Genome in a Bottle for NA24631 -- http://genomeinabottle.org/\n---\nattributes:\n  name: giab-NA24631\n  version: v3_3_2\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/giab-NA24631\n        url=http://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/ChineseTrio/HG005_NA24631_son/NISTv3.3.2/GRCh38/\n        calls=HG005_GRCh38_GIAB_highconf_CG-Illfb-IllsentieonHC-Ion-10XsentieonHC-SOLIDgatkHC_CHROM1-22_v.3.3.2_highconf.vcf.gz\n        regions=HG005_GRCh38_GIAB_highconf_CG-Illfb-IllsentieonHC-Ion-10XsentieonHC-SOLIDgatkHC_CHROM1-22_v.3.3.2_highconf.bed\n        mkdir -p $dir\n        wget -c $url/$calls -O $dir/truth_small_variants.vcf.gz\n        tabix -f -p vcf $dir/truth_small_variants.vcf.gz\n        wget -c $url/$regions -O $dir/truth_regions.bed\n    recipe_outfiles:\n      - validation/giab-NA24631/truth_small_variants.vcf.gz\n      - validation/giab-NA24631/truth_small_variants.vcf.gz.tbi\n      - validation/giab-NA24631/truth_regions.bed\n"
  },
  {
    "path": "ggd-recipes/hg38/giab-NA24694.yaml",
    "content": "# Truth sets from Genome in a Bottle for NA24631 -- http://genomeinabottle.org/\n---\nattributes:\n  name: giab-NA24694\n  version: v3_3_2\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/giab-NA24694\n        url=https://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/ChineseTrio/HG006_NA24694_father/NISTv3.3.2/GRCh38/\n        calls=HG006_GIAB_GRCh38_highconf_CG-IllFB-IllSNT-10X_CHROM1-22_v.3.3.2_highconf.vcf.gz\n        regions=HG006_GIAB_GRCh38_highconf_CG-IllFB-IllSNT-10X_CHROM1-22_v.3.3.2_highconf_noinconsistent.bed\n        mkdir -p $dir\n        wget -c $url/$calls -O $dir/truth_small_variants.vcf.gz\n        tabix -f -p vcf $dir/truth_small_variants.vcf.gz\n        wget -c $url/$regions -O $dir/truth_regions.bed\n    recipe_outfiles:\n      - validation/giab-NA24694/truth_small_variants.vcf.gz\n      - validation/giab-NA24694/truth_small_variants.vcf.gz.tbi\n      - validation/giab-NA24694/truth_regions.bed\n"
  },
  {
    "path": "ggd-recipes/hg38/giab-NA24695.yaml",
    "content": "# Truth sets from Genome in a Bottle for NA24631 -- http://genomeinabottle.org/\n---\nattributes:\n  name: giab-NA24695\n  version: v3_3_2\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        dir=validation/giab-NA24695\n        url=https://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/ChineseTrio/HG007_NA24695_mother/NISTv3.3.2/GRCh38/\n        calls=HG007_GIAB_GRCh38_highconf_CG-IllFB-IllSNT-10X_CHROM1-22_v.3.3.2_highconf.vcf.gz\n        regions=HG007_GIAB_GRCh38_highconf_CG-IllFB-IllSNT-10X_CHROM1-22_v.3.3.2_highconf_noinconsistent.bed\n        mkdir -p $dir\n        wget -c $url/$calls -O $dir/truth_small_variants.vcf.gz\n        tabix -f -p vcf $dir/truth_small_variants.vcf.gz\n        wget -c $url/$regions -O $dir/truth_regions.bed\n    recipe_outfiles:\n      - validation/giab-NA24695/truth_small_variants.vcf.gz\n      - validation/giab-NA24695/truth_small_variants.vcf.gz.tbi\n      - validation/giab-NA24695/truth_regions.bed\n"
  },
  {
    "path": "ggd-recipes/hg38/gnomad.yaml",
    "content": "# GnomAD VCF sites:\n# https://gnomad.broadinstitute.org/downloads\n# Using only PASS variants, removing some fields to make the vcf smaller\n# recipe test: https://github.com/chapmanb/cloudbiolinux/blob/master/doc/hacking.md#testing-a-ggd-recipe\n# using gnomad v3 which has WGS only bug aligned to hg38\n# gnomad v2 is WES aligned to hg19 + liftover - not using it\n# download from google cloud with gsutil\n# processes 241G vcf.gz to 132G vcf.gz in < 6 days - old estimate\n# gnomad fields changes in 2023 - no version update to avoid triggering gnomad reinstalls\n---\nattributes:\n  name: gnomad\n  version: 4\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        vcf_prefix=gnomad.genomes.v3.1.sites\n        mkdir -p variation\n        export TMPDIR=`pwd`        \n        gnomad_fields_to_keep_url=https://gist.githubusercontent.com/naumenko-sa/7b83cb718ffeadb75ae61676f2158de9/raw/e6d4e39fa2744e1271d3fce0613808d167c158e4/gnomad_fields_to_keep_2023.txt\n        wget $gnomad_fields_to_keep_url\n        gsutil cp gs://gcp-public-data--gnomad/release/3.1/vcf/genomes/*sites.chr*.vcf.bgz .\n        gsutil cp gs://gcp-public-data--gnomad/release/3.1/vcf/genomes/*sites.chr*.vcf.bgz.tbi .\n        fields_to_keep=\"INFO/\"$(cat gnomad_fields_to_keep_2023.txt | paste -s | sed s/\"\\t\"/\",INFO\\/\"/g)\n        ref=../seq/hg38.fa\n        for f in `seq 1 22` X Y M\n        do\n            vcf_file=${vcf_prefix}.chr${f}.vcf.bgz\n            bcftools view -f PASS $vcf_file | bcftools annotate -x \"^$fields_to_keep\" -Ov | vt decompose -s - | vt normalize -r $ref -n - | vt uniq - | bgzip -c > gg.chr${f}.vcf.gz\n            tabix gg.chr${f}.vcf.gz\n            echo gg.chr${f}.vcf.gz >> gnomad_to_merge.txt\n        done\n        bcftools merge -l gnomad_to_merge.txt -Oz > variation/gnomad_genome.vcf.gz\n        tabix -f -p vcf variation/gnomad_genome.vcf.gz\n        tabix -f -p vcf --csi variation/gnomad_genome.vcf.gz\n        rm ${vcf_prefix}*.vcf.bgz\n        rm ${vcf_prefix}*.vcf.bgz.tbi\n        rm gg.chr*.vcf.gz\n        rm gg.chr*.vcf.gz.tbi\n    recipe_outfiles:\n      - variation/gnomad_genome.vcf.gz\n      - variation/gnomad_genome.vcf.gz.tbi\n      - variation/gnomad_genome.vcf.gz.csi\n"
  },
  {
    "path": "ggd-recipes/hg38/gnomad_exome.yaml",
    "content": "# GnomAD VCF sites:\n# - https://gnomad.broadinstitute.org/downloads\n# - https://macarthurlab.org/2018/10/17/gnomad-v2-1/\n# Lifted over to hg38 by Gnomad team\n# Script\n# - filters only PASS variants (segdup, decoy are retained)\n# - removes many INFO fields to reduce file size: controls_, hist, non_, using bcftools annotate instead of vt rminfo, because it failed randomly on chr1\n# recipe test: https://github.com/chapmanb/cloudbiolinux/blob/master/doc/hacking.md#testing-a-ggd-recipe\n# much faster from gnomad in the google cloud - download in 20-25 min on O2\n# vt is 3x faster than bcftools for removing INFO tags: in a test with 4444 variants t(bcftools -x) = t(bcftools -x \"^\") = 27s, time(vt rminfo) = 8s,\n# but vt rminfo is failing after 148M, using bcftools\n# processing time: 87G to 4G in 7h\n# there are no native hg38 exome calls in gnomad.v3 - only for genomes, so exome variants are still lifted over from grch37,\n# and they need processing because liftover creates duplicated variants\n# receipe version is the same - the old installations won't be reinstalled\n---\nattributes:\n  name: gnomad_exome\n  version: 2.1.2\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        gsutil cp gs://gcp-public-data--gnomad/release/2.1.1/liftover_grch38/vcf/exomes/gnomad.exomes.r2.1.1.sites.liftover_grch38.vcf.bgz .\n        gsutil cp gs://gcp-public-data--gnomad/release/2.1.1/liftover_grch38/vcf/exomes/gnomad.exomes.r2.1.1.sites.liftover_grch38.vcf.bgz.tbi .\n        vcf_file=gnomad.exomes.r2.1.1.sites.liftover_grch38.vcf.bgz\n        mkdir -p variation\n        export TMPDIR=`pwd`\n        gnomad_fields_to_keep_url=https://raw.githubusercontent.com/chapmanb/cloudbiolinux/master/ggd-recipes/hg38/gnomad_fields_to_keep.txt\n        wget --no-check-certificate -c $gnomad_fields_to_keep_url\n        fields_to_keep=\"INFO/\"$(cat gnomad_fields_to_keep.txt | paste -s | sed s/\"\\t\"/\",INFO\\/\"/g)\n        ref=../seq/hg38.fa\n        bcftools view -f PASS $vcf_file | bcftools annotate -x \"^$fields_to_keep\" -Ov | vt decompose -s - | vt normalize -r $ref -n - | vt uniq - | bgzip -c > variation/gnomad_exome.vcf.gz\n        tabix -f -p vcf variation/gnomad_exome.vcf.gz\n        tabix -f -p vcf --csi variation/gnomad_exome.vcf.gz\n        rm $vcf_file $vcf_file.tbi\n    recipe_outfiles:\n      - variation/gnomad_exome.vcf.gz\n      - variation/gnomad_exome.vcf.gz.tbi\n      - variation/gnomad_exome.vcf.gz.csi\n"
  },
  {
    "path": "ggd-recipes/hg38/gnomad_fields_to_keep.txt",
    "content": "AC\nAN\nAF\nFS\nInbreedingCoeff\nMQ\nMQRankSum\nQD\nReadPosRankSum\nSOR\nBaseQRankSum\nClippingRankSum\nDP\nvariant_type\nallele_type\nn_alt_alleles\nAC_nfe_seu\nAN_nfe_seu\nAF_nfe_seu\nnhomalt_nfe_seu\nAC_raw\nAN_raw\nAF_raw\nnhomalt_raw\nAC_fin_female\nAN_fin_female\nAF_fin_female\nnhomalt_fin_female\nAC_nfe_bgr\nAN_nfe_bgr\nAF_nfe_bgr\nnhomalt_nfe_bgr\nAC_sas_male\nAN_sas_male\nAF_sas_male\nnhomalt_sas_male\nAC_afr_male\nAN_afr_male\nAF_afr_male\nnhomalt_afr_male\nAC_afr\nAN_afr\nAF_afr\nnhomalt_afr\nAC_eas_female\nAN_eas_female\nAF_eas_female\nnhomalt_eas_female\nAC_afr_female\nAN_afr_female\nAF_afr_female\nnhomalt_afr_female\nAC_sas\nAN_sas\nAF_sas\nnhomalt_sas\nAC_nfe_onf\nAN_nfe_onf\nAF_nfe_onf\nnhomalt_nfe_onf\nAC_fin_male\nAN_fin_male\nAF_fin_male\nnhomalt_fin_male\nAC_nfe_female\nAN_nfe_female\nAF_nfe_female\nnhomalt_nfe_female\nAC_amr\nAN_amr\nAF_amr\nnhomalt_amr\nAC_eas\nAN_eas\nAF_eas\nnhomalt_eas\nnhomalt\nAC_asj_male\nAN_asj_male\nAF_asj_male\nnhomalt_asj_male\nAC_oth_female\nAN_oth_female\nAF_oth_female\nnhomalt_oth_female\nAC_nfe_swe\nAN_nfe_swe\nAF_nfe_swe\nnhomalt_nfe_swe\nAC_nfe_nwe\nAN_nfe_nwe\nAF_nfe_nwe\nnhomalt_nfe_nwe\nAC_eas_jpn\nAN_eas_jpn\nAF_eas_jpn\nnhomalt_eas_jpn\nAC_female\nAN_female\nAF_female\nnhomalt_female\nAC_eas_kor\nAN_eas_kor\nAF_eas_kor\nnhomalt_eas_kor\nAC_eas_oea\nAN_eas_oea\nAF_eas_oea\nnhomalt_eas_oea\nAC_nfe_est\nAN_nfe_est\nAF_nfe_est\nnhomalt_nfe_est\nAC_eas_male\nAN_eas_male\nAF_eas_male\nnhomalt_eas_male\nAC_nfe\nAN_nfe\nAF_nfe\nnhomalt_nfe\nAC_fin\nAN_fin\nAF_fin\nnhomalt_fin\nAC_nfe_male\nAN_nfe_male\nAF_nfe_male\nnhomalt_nfe_male\nAC_sas_female\nAN_sas_female\nAF_sas_female\nnhomalt_sas_female\nAC_asj_female\nAN_asj_female\nAF_asj_female\nnhomalt_asj_female\nAC_asj\nAN_asj\nAF_asj\nnhomalt_asj\nAC_oth\nAN_oth\nAF_oth\nnhomalt_oth\nAC_male\nAN_male\nAF_male\nnhomalt_male\nAC_amr_male\nAN_amr_male\nAF_amr_male\nnhomalt_amr_male\nAC_amr_female\nAN_amr_female\nAF_amr_female\nnhomalt_amr_female\nAC_oth_male\nAN_oth_male\nAF_oth_male\nnhomalt_oth_male\nfaf95\nfaf99\npopmax\nAC_popmax\nAN_popmax\nAF_popmax\nnhomalt_popmax\n"
  },
  {
    "path": "ggd-recipes/hg38/gtf.yaml",
    "content": "# Ensembl GTF file distributed by Ensembl for hg38\n# Cleans GTF file by converting chromosome names to standard names\n# Uses https://github.com/dpryan79/ChromosomeMappings to remap the chromosome names\n---\nattributes:\n  name: gtf\n  version: 78\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=http://ftp.ensembl.org/pub/release-78/gtf/homo_sapiens/Homo_sapiens.GRCh38.78.gtf.gz\n        mkdir -p rnaseq\n        remap_url=http://raw.githubusercontent.com/dpryan79/ChromosomeMappings/master/GRCh38_ensembl2UCSC.txt\n        wget --no-check-certificate -qO- $remap_url | awk '{if($1!=$2) print \"s/^\"$1\"/\"$2\"/g\"}' > remap.sed\n        wget --no-check-certificate -qO- $url | gunzip | sed -f remap.sed | grep -v \"*_*_alt\" > rnaseq/hg38.gtf\n        rm remap.sed\n    recipe_outfiles:\n    - rnaseq/hg38.gtf\n"
  },
  {
    "path": "ggd-recipes/hg38/hapmap_snps.yaml",
    "content": "# HapMap v3.3 SNP calls\n#\n# From the Broad hg38 bundle:\n# ftp://gsapubftp-anonymous:none@ftp.broadinstitute.org/bundle/\n# https://www.broadinstitute.org/gatk/guide/article.php?id=1213\n---\nattributes:\n  name: hapmap_snps\n  version: 20160105\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=ftp://gsapubftp-anonymous:none@ftp.broadinstitute.org/bundle/hg38/\n        new=hapmap_3.3\n        base=$new.hg38\n        mkdir -p variation\n        for suffix in .vcf.gz .vcf.gz.tbi\n        do\n          [[ -f variation/$new$suffix ]] || wget --no-check-certificate -c -O variation/$new$suffix $url/$base$suffix\n        done\n    recipe_outfiles:\n      - variation/hapmap_3.3.vcf.gz\n      - variation/hapmap_3.3.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg38/hisat2.yaml",
    "content": "# hisat2 (v2.0.1) reference with Ensembl release 78 exon and splicesite annotations\n# Derived from NCBI set with HLA and decoy alternative alleles\n# http://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/GRCh38_reference_genome/\n# README.md has an in depth description of how this was created\n---\nattributes:\n  name: hisat2\n  version: 12-07-2015\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=https://s3.amazonaws.com/biodata/genomes/hg38-hisat2-12-07-2015.tar.xz\n        wget -qO- $url > hg38-hisat2-tar.xz\n        tar Jxvf hg38-hisat2-tar.xz\n        rm hg38-hisat2-tar.xz\n    recipe_outfiles:\n      - hisat2/hg38.1.ht2\n      - hisat2/hg38.2.ht2\n      - hisat2/hg38.3.ht2\n      - hisat2/hg38.4.ht2\n      - hisat2/hg38.5.ht2\n      - hisat2/hg38.6.ht2\n      - hisat2/hg38.7.ht2\n      - hisat2/hg38.8.ht2\n      - hisat2/README.md\n"
  },
  {
    "path": "ggd-recipes/hg38/mills_indels.yaml",
    "content": "# Curated training indels from Mills et al:\n#\n# http://genome.cshlp.org/content/21/6/830.full\n#\n# From the Broad hg38 bundle:\n# ftp://gsapubftp-anonymous:none@ftp.broadinstitute.org/bundle/\n# https://www.broadinstitute.org/gatk/guide/article.php?id=1213\n---\nattributes:\n  name: mills_indels\n  version: 20160105\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=ftp://gsapubftp-anonymous:none@ftp.broadinstitute.org/bundle/hg38/\n        base=Mills_and_1000G_gold_standard.indels.hg38\n        new=Mills_and_1000G_gold_standard.indels\n        mkdir -p variation\n        for suffix in .vcf.gz .vcf.gz.tbi\n        do\n          [[ -f variation/$new$suffix ]] || wget --no-check-certificate -c -O variation/$new$suffix $url/$base$suffix\n        done\n    recipe_outfiles:\n      - variation/Mills_and_1000G_gold_standard.indels.vcf.gz\n      - variation/Mills_and_1000G_gold_standard.indels.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg38/mirbase.yaml",
    "content": "# small RNA seq annotation from different sources\n---\nattributes:\n  name: srnaseq\n  version: 20211104\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p srnaseq\n        cd srnaseq\n        wget --random-wait --retry-connrefused -q -c -O hsa.gff3 https://mirbase.org/download/hsa.gff3\n        awk '$3==\"miRNA\"' hsa.gff3 | sed 's/=/ /g' > srna-transcripts.gtf\n        wget  --random-wait --retry-connrefused -q -c -O wgRna.txt.gz http://hgdownload.soe.ucsc.edu/goldenPath/hg38/database/wgRna.txt.gz\n        zgrep -v 'hsa-' wgRna.txt.gz | awk '{print $2\"\\t.\\tncrna\\t\"$3\"\\t\"$4\"\\t.\\t\"$7\"\\t.\\tname \"$5\";\"}' >> srna-transcripts.gtf\n        wget  --random-wait --retry-connrefused -q -c -O tRNAs.txt.gz http://hgdownload.soe.ucsc.edu/goldenPath/hg38/database/tRNAs.txt.gz\n        zcat tRNAs.txt.gz | awk '{print $2\"\\t.\\ttRNA\\t\"$3\"\\t\"$4\"\\t.\\t\"$7\"\\t.\\tname \"$5\";\"}' >> srna-transcripts.gtf\n        wget  --random-wait --retry-connrefused -q -c -O rmsk.txt.gz http://hgdownload.soe.ucsc.edu/goldenPath/hg38/database/rmsk.txt.gz\n        zcat rmsk.txt.gz | awk '{print $6\"\\t.\\trepeat\\t\"$7+1\"\\t\"$8+1\"\\t.\\t\"$10\"\\t.\\tname \"$12\";\"}' >> srna-transcripts.gtf\n        wget  --random-wait --retry-connrefused  -q -c -O refGene.txt.gz http://hgdownload.soe.ucsc.edu/goldenPath/hg38/database/refGene.txt.gz\n        zcat refGene.txt.gz | awk '{print $3\"\\t.\\tgene\\t\"$5\"\\t\"$6\"\\t.\\t\"$4\"\\t.\\tname \"$13\";\"}' >> srna-transcripts.gtf\n        sed -i 's/\\t0\\t/\\t1\\t/' srna-transcripts.gtf\n        # wget http://www.regulatoryrna.org/database/piRNA/download/archive/v1.0/bed/piR_hg19_v1.0.bed.gz\n        # zcat piR_hg19_v1.0.bed.gz | awk '{print $1\"\\t.\\tpiRNA\\t\"$2\"\\t\"$3\"\\t.\\t\"$6\"\\t.\\tname \"$4\";\"}' >> srna-transcripts.gtf\n        # mirbase\n        wget  --random-wait --retry-connrefused  -q -c -O hairpin.fa.raw https://mirbase.org/download/CURRENT/hairpin.fa\n        cat hairpin.fa.raw |  awk '{if ($0~/>hsa/){name=$0; print name} else if ($0~/^>/){name=0};if (name!=0 && $0!~/^>/){print $0;}}' | sed 's/U/T/g'  > hairpin.fa\n        rm hairpin.fa.raw\n        wget  --random-wait --retry-connrefused  -q -c -O mature.fa.raw https://mirbase.org/download/CURRENT/mature.fa\n        cat mature.fa.raw |  awk '{if ($0~/>hsa/){name=$0; print name} else if ($0~/^>/){name=0};if (name!=0 && $0!~/^>/){print $0;}}' | sed 's/U/T/g'  > mature.fa\n        rm mature.fa.raw\n        wget  --random-wait --retry-connrefused  -q -c -O miRNA.dat.raw https://mirbase.org/download/CURRENT/miRNA.dat\n        cat miRNA.dat.raw | awk '{if ($0~/hsa/)print $0}' > miRNA.str\n        rm miRNA.dat.raw\n        wget --random-wait --retry-connrefused -q -c -O mirbase.gff3 https://mirbase.org/download/hsa.gff3\n        #tdrmapper\n        wget   --random-wait --retry-connrefused  --no-check-certificate -q -c -O trna_mature_pre.fa https://github.com/sararselitsky/tDRmapper/raw/master/hg19_mature_and_pre.fa\n        # mintmap\n        wget --random-wait --retry-connrefused  -nv -c https://github.com/TJU-CMC-Org/MINTmap/archive/refs/tags/MINTmap_Standard_v1.0.tar.gz\n        tar xzf MINTmap_Standard_v1.0.tar.gz MINTmap-MINTmap_Standard_v1.0/LookupTable.tRFs.MINTmap_v1.txt --strip-components=1\n        tar xzf MINTmap_Standard_v1.0.tar.gz MINTmap-MINTmap_Standard_v1.0/tRNAspace.Spliced.Sequences.MINTmap_v1.fa --strip-components=1\n        tar xzf MINTmap_Standard_v1.0.tar.gz MINTmap-MINTmap_Standard_v1.0/OtherAnnotations.MINTmap_v1.txt --strip-components=1\n        # mirdeep2\n        wget  --random-wait --retry-connrefused --no-check-certificate -q -c -O Rfam_for_miRDeep.fa.gz https://github.com/lpantano/mirdeep2_core/raw/data/Rfam_for_miRDeep.fa.gz && gunzip Rfam_for_miRDeep.fa.gz\n        # targetscan analysis\n    recipe_outfiles:\n        - srnaseq/srna-transcripts.gtf\n        - srnaseq/hairpin.fa\n        - srnaseq/mature.fa\n        - srnaseq/miRNA.str\n        - srnaseq/mirbase.gff3\n        - srnaseq/trna_mature_pre.fa\n        - srnaseq/LookupTable.tRFs.MINTmap_v1.txt\n        - srnaseq/tRNAspace.Spliced.Sequences.MINTmap_v1.fa\n        - srnaseq/OtherAnnotations.MINTmap_v1.txt\n        - srnaseq/Rfam_for_miRDeep.fa\n"
  },
  {
    "path": "ggd-recipes/hg38/platinum-genome-NA12878.yaml",
    "content": "# Truth sets from Illumina Platinum Genomes project\n# http://www.illumina.com/platinumgenomes/\n---\nattributes:\n  name: platinum-genome-NA12878\n  version: v2_0_1\n\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        version=v2_0_1\n        dir=validation/platinum-genome-NA12878\n        mkdir -p $dir\n        wget --no-check-certificate -c -O $dir/truth_small_variants.vcf.gz https://s3.amazonaws.com/cloudbiolinux/cache/platinum-genome/platinum-genome-NA12878-hg38-$version.vcf.gz\n        wget --no-check-certificate -c -O $dir/truth_small_variants.vcf.gz.tbi https://s3.amazonaws.com/cloudbiolinux/cache/platinum-genome/platinum-genome-NA12878-hg38-$version.vcf.gz.tbi\n        wget --no-check-certificate -c -O - https://s3.amazonaws.com/cloudbiolinux/cache/platinum-genome/platinum-genome-NA12878-hg38-regions-$version.bed.gz | gunzip -c > $dir/truth_regions.bed\n    recipe_outfiles:\n      - validation/platinum-genome-NA12878/truth_small_variants.vcf.gz\n      - validation/platinum-genome-NA12878/truth_small_variants.vcf.gz.tbi\n      - validation/platinum-genome-NA12878/truth_regions.bed\n"
  },
  {
    "path": "ggd-recipes/hg38/prioritize.yaml",
    "content": "# Regions for prioritization:\n# cancer\n#  - civic -- known cancer genes from the CIViC database\n#  - az300 -- list of 300 cancer genes of interest from AstraZeneca Oncology\n#  - az300-with-fusion -- az300 list plus known interacting fusions\n#      from FusionCatcher https://github.com/ndaniel/fusioncatcher/blob/master/bin/generate_known.py#L102\n#  - az-cancer-panel -- az300 genes plus additional useful panel genes\n---\nattributes:\n  name: prioritize\n  version: 20181227\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/biodata/coverage/prioritize/prioritize-cancer-hg38-20181227.tar.gz\n        outdir=coverage/prioritize\n        mkdir -p $outdir\n        cd $outdir\n        wget --no-check-certificate -c -O cancer.tar.gz $baseurl\n        tar -xzvpf cancer.tar.gz\n    recipe_outfiles:\n      - coverage/prioritize/cancer/az300.bed.gz\n      - coverage/prioritize/cancer/az300.bed.gz.tbi\n      - coverage/prioritize/cancer/az300-with-fusion.bed.gz\n      - coverage/prioritize/cancer/az300-with-fusion.bed.gz.tbi\n      - coverage/prioritize/cancer/az-cancer-panel.bed.gz\n      - coverage/prioritize/cancer/az-cancer-panel.bed.gz.tbi\n      - coverage/prioritize/cancer/civic-2018-12-27.bed.gz\n      - coverage/prioritize/cancer/civic-2018-12-27.bed.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg38/purecn_mappability.yaml",
    "content": "# mappability regions for purecn analyses:\n# http://bioconductor.org/packages/release/bioc/vignettes/PureCN/inst/doc/Quick.html\n# file size: 1.5G\n---\nattributes:\n  name: purecn_mappability\n  version: 20200928\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p coverage/mappability\n        cd coverage/mappability\n        wget --no-check-certificate -c https://s3.amazonaws.com/purecn/GCA_000001405.15_GRCh38_no_alt_analysis_set_100.bw\n        cd ..\n    recipe_outfiles:\n      - coverage/mappability/GCA_000001405.15_GRCh38_no_alt_analysis_set_100.bw\n"
  },
  {
    "path": "ggd-recipes/hg38/qsignature.yaml",
    "content": "---\nattributes:\n  name: qsignature\n  version: 20160526\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=http://github.com/lpantano/custom_genome_annotation/raw/master/qsignature.hg38.tbz\n        mkdir -p variation\n        cd variation\n        wget -N -c --no-check-certificate $baseurl\n        tar xjvf qsignature.hg38.tbz\n        mv qsignature.hg38.txt qsignature.txt\n    recipe_outfiles:\n        - variation/qsignature.txt\n \n"
  },
  {
    "path": "ggd-recipes/hg38/rmsk.yaml",
    "content": "# repeat elements location on the genome\n---\nattributes:\n  name: rmsk\n  version: 20180319\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p coverage/problem_regions/repeats\n        cd coverage/problem_regions/repeats\n        wget  --random-wait --retry-connrefused -q -c -O rmsk.txt.gz http://hgdownload.soe.ucsc.edu/goldenPath/hg38/database/rmsk.txt.gz\n        zcat rmsk.txt.gz | awk '{print $6\"\\t.\\trepeat\\t\"$7+1\"\\t\"$8+1\"\\t.\\t\"$10\"\\t.\\tname \"$12\";\"}' | bgzip > rmsk.gtf.gz\n        tabix rmsk.gtf.gz\n    recipe_outfiles:\n      - coverage/problem_regions/repeats/rmsk.gtf.gz\n      - coverage/problem_regions/repeats/rmsk.gtf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg38/salmon-decoys.yaml",
    "content": "# Decoys for SA mode of Salmon\n# built against hg38 with the alt alleles\n# Ensembl gene build 94\n---\nattributes:\n  name: salmon-decoys\n  version: 94\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=https://s3.amazonaws.com/biodata/annotation/hg38-build94-salmon-decoys.tar.gz\n        outdir=rnaseq/salmon-decoys\n        mkdir -p ${outdir}\n        wget -c -O decoys.tar.gz ${url}\n        tar zxvf decoys.tar.gz\n        mv decoys.fa ${outdir}/.\n        mv decoys.txt ${outdir}/.\n        rm decoys.tar.gz\n    recipe_outfiles:\n      - rnaseq/salmon-decoys/decoys.fa\n      - rnaseq/salmon-decoys/decoys.txt\n"
  },
  {
    "path": "ggd-recipes/hg38/seq.yaml",
    "content": "# Full hg38/GRCh38 reference genome distributed by 1000 genomes\n# Derived from NCBI set with HLA and decoy alternative alleles\n# ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/GRCh38_reference_genome/\n---\nattributes:\n  name: seq\n  version: 1000g-20150219_1\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/GRCh38_reference_genome\n        base=GRCh38_full_analysis_set_plus_decoy_hla\n        new=hg38\n        mkdir -p seq\n        for suffix in .dict .fa.fai\n        do\n          [[ -f seq/$new$suffix ]] || wget -c -O seq/$new$suffix $url/$base$suffix\n        done\n        # gzipped references for use with ensembl-vep HGVS\n        urlgz=https://s3.amazonaws.com/biodata/hg38_bundle\n        for suffix in .fa.gz .fa.gz.fai .fa.gz.gzi\n        do\n          [[ -f seq/$new$suffix ]] || wget --no-check-certificate -c -O seq/$new$suffix $urlgz/$new$suffix\n        done\n        gunzip -c seq/hg38.fa.gz > seq/hg38.fa\n        touch seq/hg38.fa.fai\n        touch seq/hg38.dict\n    recipe_outfiles:\n      - seq/hg38.fa\n      - seq/hg38.fa.fai\n      - seq/hg38.fa.gz\n      - seq/hg38.fa.gz.fai\n      - seq/hg38.fa.gz.gzi\n      - seq/hg38.dict\n"
  },
  {
    "path": "ggd-recipes/hg38/simple_repeat.yaml",
    "content": "# UCSC repeats track for purecn variant filtration:\n---\nattributes:\n  name: simple_repeat\n  version: 20200928\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p coverage/problem_regions/repeats\n        cd coverage/problem_regions/repeats\n        wget --random-wait --retry-connrefused -q -c -O simple_repeat.txt.gz http://hgdownload.soe.ucsc.edu/goldenPath/hg38/database/simpleRepeat.txt.gz\n        zcat simple_repeat.txt.gz | awk -F '\\t' '{print $2\"\\t\"$3\"\\t\"$4\"\\t\"$5\"\\t\"$11\"\\t.\"}' > simple_repeat.bed\n    recipe_outfiles:\n      - coverage/problem_regions/repeats/simple_repeat.bed\n"
  },
  {
    "path": "ggd-recipes/hg38/topmed.yaml",
    "content": "# Topmed WGS allele frequencies: https://bravo.sph.umich.edu/freeze5/hg38/\n# topmed vcf from ensembl does not have contig names\n# \n# made via:\n# url=ftp://ftp.ensembl.org/pub/data_files/homo_sapiens/GRCh38/variation_genotype/TOPMED_GRCh38_20180418.vcf.gz\n# remap_url=http://raw.githubusercontent.com/dpryan79/ChromosomeMappings/master/GRCh38_ensembl2UCSC.txt\n# ref=hg38.fa\n# wget --no-check-certificate -qO- $remap_url | awk '{ print length, $0 }' | sort -n -s -r | cut -d\" \" -f2- | awk '{if(!$2) print \"/^\"$1\"/d\"; else if($1!=$2) print \"s/^\"$1\"/\"$2\"/g\";}' > remap.sed\n# mkdir -p variation\n# export TMPDIR=`pwd`\n# wget --no-check-certificate -qO- ${url} > topmed.vcf.gz\n# gunzip -c topmed.vcf.gz | sed -f remap.sed | sed s/Number=\\./Number=A/g | vt normalize -r $ref -n - | vt uniq - | gsort -m 5000  /dev/stdin $ref.fai | bgzip -c > variation/topmed.vcf.gz\n# tabix -f -p vcf variation/topmed.vcf.gz\n---\nattributes:\n  name: topmed\n  version: freeze5\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=http://s3.amazonaws.com/bcbio-nextgen/genomes/Hsapiens/hg38/variation/topmed-freeze5.vcf.gz\n        mkdir -p variation\n        wget -c -O - ${url} > variation/topmed-freeze5.vcf.gz\n        url=http://s3.amazonaws.com/bcbio-nextgen/genomes/Hsapiens/hg38/variation/topmed-freeze5.vcf.gz.tbi\n        wget -c -O - ${url} > variation/topmed-freeze5.vcf.gz.tbi\n        rm -f variation/topmed.vcf.gz\n        rm -f variation/topmed.vcf.gz.tbi\n        cd variation\n        ln -s topmed-freeze5.vcf.gz topmed.vcf.gz\n        ln -s topmed-freeze5.vcf.gz.tbi topmed.vcf.gz.tbi\n    recipe_outfiles:\n        - variation/topmed.vcf.gz\n        - variation/topmed.vcf.gz.tbi\n        - variation/topmed-freeze5.vcf.gz\n        - variation/topmed-freeze5.vcf.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg38/transcripts.yaml",
    "content": "# Transcript, annotation and indices for RNA-seq analysis\n# Prepared from Ensembl transcripts using: \n# https://github.com/chapmanb/cloudbiolinux/blob/master/utils/prepare_tx_gff.py\n# Prepared against Ensembl gene build 94\n---\nattributes:\n  name: transcripts\n  version: 2018-10-10_92\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/biodata/annotation/hg38-rnaseq-2018-10-10_92.tar.xz\n        wget -c -N --no-check-certificate $baseurl\n        xz -dc *-rnaseq-*.tar.xz | tar -xpf -\n        mv */rnaseq-* rnaseq\n    recipe_outfiles:\n      - rnaseq/ref-transcripts.bed\n      - rnaseq/ref-transcripts.dexseq.gff3\n      - rnaseq/ref-transcripts.fa\n      - rnaseq/ref-transcripts.genePred\n      - rnaseq/ref-transcripts.gtf\n      - rnaseq/ref-transcripts.gtf.db\n      - rnaseq/ref-transcripts-mask.gtf\n      - rnaseq/ref-transcripts.refFlat\n      - rnaseq/ref-transcripts-splicesites.txt\n      - rnaseq/rRNA.gtf\n      - rnaseq/rRNA.gtf.db\n      - rnaseq/rRNA.interval_list\n      - rnaseq/tx2gene.csv\n      - rnaseq/version.txt\n"
  },
  {
    "path": "ggd-recipes/hg38/twobit.yaml",
    "content": "# UCSC 2bit reference\n---\nattributes:\n  name: twobit\n  version: 1000g-20150219_1\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p ucsc\n        cd ucsc\n        faToTwoBit ../../seq/hg38.fa hg38.2bit\n    recipe_outfiles:\n      - ucsc/hg38.2bit\n"
  },
  {
    "path": "ggd-recipes/hg38/varpon.yaml",
    "content": "# Panels of normals (PONs) for variant calling\n# germline_het_pon -- Locations of likely germline heterogygote sites\n---\nattributes:\n  name: varpon\n  version: 20181105\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        ref=../seq/hg38.fa\n        mkdir -p variation\n        wget --no-check-certificate -O - 'https://s3.amazonaws.com/biodata/collections/hg38/variation/germline_het_pon.bed.gz' > variation/germline_het_pon.bed.gz\n        wget --no-check-certificate -O - 'https://s3.amazonaws.com/biodata/collections/hg38/variation/germline_het_pon.bed.gz.tbi' > variation/germline_het_pon.bed.gz.tbi\n    recipe_outfiles:\n      - variation/germline_het_pon.bed.gz\n      - variation/germline_het_pon.bed.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/hg38/vcfanno.yaml",
    "content": "# Annotation files for vcfanno variant annotation\n# Feeds into vcf2db GEMINI database creation and other annotation tasks\n---\nattributes:\n  name: vcfanno\n  version: 20210204\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p config/vcfanno\n        cd config/vcfanno\n        bcbio_baseurl=https://raw.githubusercontent.com/bcbio/bcbio-nextgen/master/config/vcfanno\n        wget --no-check-certificate -O gemini.conf -c $bcbio_baseurl/hg38-gemini.conf\n        wget --no-check-certificate -O gemini.lua -c $bcbio_baseurl/hg38-gemini.lua\n        wget --no-check-certificate -O dbnsfp.conf -c $bcbio_baseurl/hg38-dbnsfp.conf\n        wget --no-check-certificate -O dbscsnv.conf -c $bcbio_baseurl/hg38-dbscsnv.conf\n        wget --no-check-certificate -O gnomad_genome.conf -c $bcbio_baseurl/hg38-gnomad_genome.conf\n        wget --no-check-certificate -O rnaedit.conf -c $bcbio_baseurl/rnaedit.conf\n        wget --no-check-certificate -O rnaedit.lua -c $bcbio_baseurl/rnaedit.lua\n        wget --no-check-certificate -O somatic.conf -c $bcbio_baseurl/somatic.conf\n\n    recipe_outfiles:\n      - config/vcfanno/gemini.lua\n      - config/vcfanno/gemini.conf\n      - config/vcfanno/dbnsfp.conf\n      - config/vcfanno/dbscsnv.conf\n      - config/vcfanno/gnomad_genome.conf\n      - config/vcfanno/rnaedit.lua\n      - config/vcfanno/rnaedit.conf\n      - config/vcfanno/somatic.conf\n"
  },
  {
    "path": "ggd-recipes/hg38/viral.yaml",
    "content": "# Virus reference genomes, prepared for bwa alignment\n# Taken from TCGA GDC reference genome\n# https://gdc.cancer.gov/about-data/data-harmonization-and-generation/gdc-reference-files\n---\nattributes:\n  name: viral\n  version: 2017.02.04\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=http://s3.amazonaws.com/biodata/viral/hg38-gdc-viral.tar.gz\n        mkdir -p viral\n        wget --no-check-certificate -c $url\n        tar -xzvpf *.tar.gz\n        mv *-gdc-viral/* viral\n    recipe_outfiles:\n      - viral/gdc-viral.fa\n      - viral/gdc-viral.dict\n      - viral/gdc-viral.fa.ann\n      - viral/gdc-viral.fa.fai\n      - viral/gdc-viral.fa.pac\n      - viral/gdc-viral.fa.bwt\n      - viral/gdc-viral.fa.amb\n      - viral/gdc-viral.fa.sa\n"
  },
  {
    "path": "ggd-recipes/hg38-noalt/RADAR.yaml",
    "content": "# RADAR RNA editing events: (http://www.stanford.edu/~gokulr/database/Human_AG_all_hg19_v2.txt)\n# installs a BED file of the RADAR events, preconverted to hg38\n# how this file was created:\n#\n# 1. Download the hg19 RADAR file and converted to BED format.\n# url=http://www.stanford.edu/~gokulr/database/Human_AG_all_hg19_v2.txt\n# mkdir -p editing\n# cd editing\n# wget -qO- $url | awk 'BEGIN{OFS=\"\\t\"} {print $1,$2,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11}' | sed \"s/position  position/start  end/\" > RADAR-hg19.bed\n# cd ../\n# 2. Convert RADAR BED file to BED3 format.\n# cut -f1,2,3 RADAR-hg19.bed > RADAR-hg19.tmp.bed\n# 3. Get the Crossmap hg19 to hg38 chain file:\n# wget http://hgdownload.soe.ucsc.edu/goldenPath/hg19/liftOver/hg19ToHg38.over.chain.gz\n# 4. Run Crossmap:\n# Crossmap.py bed hg19ToHg38.over.chain.gz RADAR-hg19.tmp.bed > conversions.bed\n# 5. Merge original RADAR file with the Crossmap output\n# wget https://raw.githubusercontent.com/roryk/seqscripts/master/radar-convert-annotations.py\n# python radar-convert-annotations.py RADAR-hg19.bed conversions.bed RADAR-hg38.bed.gz\n\n---\nattributes:\n  name: RADAR-hg38\n  version: 1\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=https://biodata.s3.amazonaws.com/annotation/RADAR-hg38.bed.gz\n        mkdir -p editing\n        cd editing\n      \twget -qO- $url > RADAR-hg38.bed.gz\n       \tcd ../\n    recipe_outfiles:\n      - editing/RADAR-hg38.bed.gz\n"
  },
  {
    "path": "ggd-recipes/hg38-noalt/README.md",
    "content": "Human reference genome: GRCh38/hg38 without any alternative reference contigs\n\nhttp://ftp.ncbi.nlm.nih.gov/genomes/genbank/vertebrate_mammalian/Homo_sapiens/all_assembly_versions/GCA_000001405.15_GRCh38/seqs_for_alignment_pipelines.ucsc_ids/\n"
  },
  {
    "path": "ggd-recipes/hg38-noalt/bowtie2.yaml",
    "content": "# pre-built bowtie2 indices\n---\nattributes:\n  name: bowtie2\n  version: 0.1\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        base=GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.bowtie_index\n        new=hg38-noalt.fa\n        ncbiurl=http://ftp.ncbi.nlm.nih.gov/genomes/genbank/vertebrate_mammalian/Homo_sapiens/all_assembly_versions/GCA_000001405.15_GRCh38/seqs_for_alignment_pipelines.ucsc_ids\n        wget -c $ncbiurl/$base.tar.gz\n        [[ -f $base.1.bt2 ]] || tar -xzvpf $base.tar.gz\n        mkdir -p bowtie2\n        for suffix in .1.bt2 .2.bt2 .3.bt2 .4.bt2 .rev.1.bt2 .rev.2.bt2\n        do\n          [[ -f bowtie2/$new$suffix ]] || mv -f $base$suffix bowtie2/$new$suffix\n        done\n    recipe_outfiles:\n      - bowtie2\n"
  },
  {
    "path": "ggd-recipes/hg38-noalt/bwa.yaml",
    "content": "# pre-built bwa indices from NCBIs hg38 distribution with no alternative alleles\n---\nattributes:\n  name: bwa\n  version: 0.1\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        base=GCA_000001405.15_GRCh38_no_alt_analysis_set.fna\n        new=hg38-noalt.fa\n        ncbiurl=http://ftp.ncbi.nlm.nih.gov/genomes/genbank/vertebrate_mammalian/Homo_sapiens/all_assembly_versions/GCA_000001405.15_GRCh38/seqs_for_alignment_pipelines.ucsc_ids\n        wget -c $ncbiurl/$base.bwa_index.tar.gz\n        [[ -f $base.fna.bwt ]] || tar -xzvpf $base.bwa_index.tar.gz\n        mkdir -p bwa\n        for suffix in .bwt .amb .ann .pac .sa\n        do\n          [[ -f bwa/$new$suffix ]] || mv -f $base$suffix bwa/$new$suffix\n        done\n    recipe_outfiles:\n      - bwa"
  },
  {
    "path": "ggd-recipes/hg38-noalt/gtf.yaml",
    "content": "# Ensembl GTF file distributed by Ensembl for hg38-noalt\n# Cleans GTF file by converting chromosome names to standard names \n# Uses https://github.com/dpryan79/ChromosomeMappings to remap the chromosome names\n---\nattributes:\n  name: gtf\n  version: 78\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=http://ftp.ensembl.org/pub/release-78/gtf/homo_sapiens/Homo_sapiens.GRCh38.78.gtf.gz\n        mkdir -p rnaseq\n        remap_url=http://raw.githubusercontent.com/dpryan79/ChromosomeMappings/master/GRCh38_ensembl2UCSC.txt\n        wget -qO- $remap_url | awk '{if($1!=$2) print \"s/^\"$1\"/\"$2\"/g\"}' > remap.sed\n        wget -qO- $url | gunzip | sed -f remap.sed > rnaseq/hg38-noalt.gtf\n        rm remap.sed\n    recipe_outfiles:\n    - rnaseq/hg38-noalt.gtf\n"
  },
  {
    "path": "ggd-recipes/hg38-noalt/mirbase.yaml",
    "content": "# small RNA seq annotation from different sources\n---\nattributes:\n  name: srnaseq\n  version: 20211104\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p srnaseq\n        cd srnaseq\n        wget --random-wait --retry-connrefused -q -c -O hsa.gff3 https://mirbase.org/download/hsa.gff3\n        awk '$3==\"miRNA\"' hsa.gff3 | sed 's/=/ /g' > srna-transcripts.gtf\n        wget  --random-wait --retry-connrefused -q -c -O wgRna.txt.gz http://hgdownload.soe.ucsc.edu/goldenPath/hg38/database/wgRna.txt.gz\n        zgrep -v 'hsa-' wgRna.txt.gz | awk '{print $2\"\\t.\\tncrna\\t\"$3\"\\t\"$4\"\\t.\\t\"$7\"\\t.\\tname \"$5\";\"}' >> srna-transcripts.gtf\n        wget  --random-wait --retry-connrefused -q -c -O tRNAs.txt.gz http://hgdownload.soe.ucsc.edu/goldenPath/hg38/database/tRNAs.txt.gz\n        zcat tRNAs.txt.gz | awk '{print $2\"\\t.\\ttRNA\\t\"$3\"\\t\"$4\"\\t.\\t\"$7\"\\t.\\tname \"$5\";\"}' >> srna-transcripts.gtf\n        wget  --random-wait --retry-connrefused -q -c -O rmsk.txt.gz http://hgdownload.soe.ucsc.edu/goldenPath/hg38/database/rmsk.txt.gz\n        zcat rmsk.txt.gz | awk '{print $6\"\\t.\\trepeat\\t\"$7+1\"\\t\"$8+1\"\\t.\\t\"$10\"\\t.\\tname \"$12\";\"}' >> srna-transcripts.gtf\n        wget  --random-wait --retry-connrefused  -q -c -O refGene.txt.gz http://hgdownload.soe.ucsc.edu/goldenPath/hg38/database/refGene.txt.gz\n        zcat refGene.txt.gz | awk '{print $3\"\\t.\\tgene\\t\"$5\"\\t\"$6\"\\t.\\t\"$4\"\\t.\\tname \"$13\";\"}' >> srna-transcripts.gtf\n        sed -i 's/\\t0\\t/\\t1\\t/' srna-transcripts.gtf\n        # wget http://www.regulatoryrna.org/database/piRNA/download/archive/v1.0/bed/piR_hg19_v1.0.bed.gz\n        # zcat piR_hg19_v1.0.bed.gz | awk '{print $1\"\\t.\\tpiRNA\\t\"$2\"\\t\"$3\"\\t.\\t\"$6\"\\t.\\tname \"$4\";\"}' >> srna-transcripts.gtf\n        # mirbase\n        wget  --random-wait --retry-connrefused  -q -c -O hairpin.fa.raw https://mirbase.org/download/CURRENT/hairpin.fa\n        cat hairpin.fa.raw |  awk '{if ($0~/>hsa/){name=$0; print name} else if ($0~/^>/){name=0};if (name!=0 && $0!~/^>/){print $0;}}' | sed 's/U/T/g'  > hairpin.fa\n        rm hairpin.fa.raw\n        wget  --random-wait --retry-connrefused  -q -c -O mature.fa.raw https://mirbase.org/download/CURRENT/mature.fa\n        cat mature.fa.raw|  awk '{if ($0~/>hsa/){name=$0; print name} else if ($0~/^>/){name=0};if (name!=0 && $0!~/^>/){print $0;}}' | sed 's/U/T/g'  > mature.fa\n        rm mature.fa.raw\n        wget  --random-wait --retry-connrefused  -q -c -O miRNA.dat.raw https://mirbase.org/download/CURRENT/miRNA.dat\n        cat miRNA.dat.raw | awk '{if ($0~/hsa/)print $0}' > miRNA.str\n        rm miRNA.dat.raw\n        wget --random-wait --retry-connrefused -q -c -O mirbase.gff3 https://mirbase.org/download/hsa.gff3\n        #tdrmapper\n        wget   --random-wait --retry-connrefused  --no-check-certificate -q -c -O trna_mature_pre.fa https://github.com/sararselitsky/tDRmapper/raw/master/hg19_mature_and_pre.fa\n        # mintmap\n        wget --random-wait --retry-connrefused  -nv -c https://github.com/TJU-CMC-Org/MINTmap/archive/refs/tags/MINTmap_Standard_v1.0.tar.gz\n        tar xzf MINTmap_Standard_v1.0.tar.gz MINTmap-MINTmap_Standard_v1.0/LookupTable.tRFs.MINTmap_v1.txt --strip-components=1\n        tar xzf MINTmap_Standard_v1.0.tar.gz MINTmap-MINTmap_Standard_v1.0/tRNAspace.Spliced.Sequences.MINTmap_v1.fa --strip-components=1\n        tar xzf MINTmap_Standard_v1.0.tar.gz MINTmap-MINTmap_Standard_v1.0/OtherAnnotations.MINTmap_v1.txt --strip-components=1\n        # mirdeep2\n        wget  --random-wait --retry-connrefused --no-check-certificate -q -c -O Rfam_for_miRDeep.fa.gz https://github.com/lpantano/mirdeep2_core/raw/data/Rfam_for_miRDeep.fa.gz && gunzip Rfam_for_miRDeep.fa.gz\n        # targetscan analysis\n    recipe_outfiles:\n        - srnaseq/srna-transcripts.gtf\n        - srnaseq/hairpin.fa\n        - srnaseq/mature.fa\n        - srnaseq/miRNA.str\n        - srnaseq/mirbase.gff3\n        - srnaseq/trna_mature_pre.fa\n        - srnaseq/LookupTable.tRFs.MINTmap_v1.txt\n        - srnaseq/tRNAspace.Spliced.Sequences.MINTmap_v1.fa\n        - srnaseq/OtherAnnotations.MINTmap_v1.txt\n        - srnaseq/Rfam_for_miRDeep.fa\n"
  },
  {
    "path": "ggd-recipes/hg38-noalt/seq.yaml",
    "content": "# Reference genome distributed by NCBI for GRCh38, without alternative reference contigs\n---\nattributes:\n  name: seq\n  version: 0.1\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p seq\n        ncbidir=ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/001/405/GCA_000001405.15_GRCh38/seqs_for_alignment_pipelines.ucsc_ids\n        filebase=GCA_000001405.15_GRCh38_no_alt_analysis_set.fna\n        wget -c -O - $ncbidir/${filebase}.gz | gunzip -c > seq/hg38-noalt.fa\n      - wget -c -O seq/hg38-noalt.fa.fai $ncbidir/${filebase}.fai\n      - >\n        [[ -f seq/hg38-noalt.dict ]] ||\n        picard CreateSequenceDictionary REFERENCE=seq/hg38-noalt.fa OUTPUT=seq/hg38-noalt.dict SPECIES=hg38-noalt\n    recipe_outfiles:\n      - seq/hg38-noalt.fa\n      - seq/hg38-noalt.fa.fai\n      - seq/hg38-noalt.dict\n"
  },
  {
    "path": "ggd-recipes/hg38-noalt/transcripts.yaml",
    "content": "# Transcript, annotation and indices for RNA-seq analysis\n# Prepared from Ensembl transcripts using: \n# https://github.com/chapmanb/cloudbiolinux/blob/master/utils/prepare_tx_gff.py\n---\nattributes:\n  name: transcripts\n  version: 2015-11-22\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/biodata/annotation/hg38-noalt-rnaseq-2015-11-22.tar.xz\n        wget -c -N --no-check-certificate $baseurl\n        xz -dc *-rnaseq-*.tar.xz | tar -xpf -\n        mv */rnaseq-* rnaseq\n    recipe_outfiles:\n      - rnaseq/tophat\n      - rnaseq/ref-transcripts.bed\n      - rnaseq/ref-transcripts.dexseq.gff3\n      - rnaseq/ref-transcripts.fa\n      - rnaseq/ref-transcripts.genePred\n      - rnaseq/ref-transcripts.gtf\n      - rnaseq/ref-transcripts.gtf.db\n      - rnaseq/ref-transcripts-mask.gtf\n      - rnaseq/ref-transcripts.refFlat\n      - rnaseq/ref-transcripts-splicesites.txt\n      - rnaseq/rRNA.gtf\n      - rnaseq/rRNA.gtf.db\n      - rnaseq/rRNA.interval_list\n"
  },
  {
    "path": "ggd-recipes/mm10/dbsnp.yaml",
    "content": "# dbsnp for mouse was discontinued in 2017, we are using the latest release\n# https://www.animalgenome.org/community/angenmap/hmail/archive/5936.html\n---\nattributes:\n  name: dbsnp\n  version: 20130912\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/biodata/variants/mm10-dbSNP-2013-09-12.vcf.gz\n        mkdir -p variation\n        cd variation\n        wget -c -N $baseurl\n        wget -c -N $baseurl.tbi\n    recipe_outfiles:\n      - variation/mm10-dbSNP-2013-09-12.vcf.gz\n      - variation/mm10-dbSNP-2013-09-12.vcf.gz.tbi\n \n"
  },
  {
    "path": "ggd-recipes/mm10/mirbase.yaml",
    "content": "# small RNA seq annotation from different sources\n---\nattributes:\n  name: srnaseq\n  version: 20211203\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p srnaseq\n        cd srnaseq\n        wget  --random-wait --retry-connrefused  -q -N -c http://hgdownload.soe.ucsc.edu/goldenPath/mm10/database/wgEncodeGencodeBasicVM11.txt.gz\n        zcat  wgEncodeGencodeBasicVM11.txt.gz | awk '{print $3\"\\t.\\tencode\\t\"$5+1\"\\t\"$6\"\\t.\\t\"$4\"\\t.\\tname \"$13\";\"}' | awk '$5-$4 < 500' > srna-transcripts.gtf\n        wget  --random-wait --retry-connrefused -q -N -c http://hgdownload.soe.ucsc.edu/goldenPath/mm10/database/tRNAs.txt.gz\n        zcat tRNAs.txt.gz | awk '{print $2\"\\t.\\ttRNA\\t\"$3+1\"\\t\"$4\"\\t.\\t\"$7\"\\t.\\tname \"$5\";\"}' >> srna-transcripts.gtf\n        wget  --random-wait --retry-connrefused -q -N -c http://hgdownload.soe.ucsc.edu/goldenPath/mm10/database/rmsk.txt.gz\n        zcat rmsk.txt.gz | awk '{print $6\"\\t.\\trepeat\\t\"$7+1\"\\t\"$8+1\"\\t.\\t\"$10\"\\t.\\tname \"$12\";\"}' >> srna-transcripts.gtf\n        wget  --random-wait --retry-connrefused -q -N -c https://www.mirbase.org/ftp/20/genomes/mmu.gff3\n        awk '$3==\"miRNA\"' mmu.gff3 | sed 's/=/ /g' >> srna-transcripts.gtf\n        wget  --random-wait --retry-connrefused -q -c -O hairpin.fa.raw https://mirbase.org/download/CURRENT/hairpin.fa\n        cat hairpin.fa.raw |  awk '{if ($0~/>mmu/){name=$0; print name} else if ($0~/^>/){name=0};if (name!=0 && $0!~/^>/){print $0;}}' | sed 's/U/T/g'  > hairpin.fa\n        rm hairpin.fa.raw\n        wget  --random-wait --retry-connrefused -q -c -O mature.fa.raw https://mirbase.org/download/CURRENT/mature.fa\n        cat mature.fa.raw |  awk '{if ($0~/>mmu/){name=$0; print name} else if ($0~/^>/){name=0};if (name!=0 && $0!~/^>/){print $0;}}' | sed 's/U/T/g'  > mature.fa\n        rm mature.fa.raw\n        wget --random-wait --retry-connrefused  -q -c -O miRNA.dat.raw https://mirbase.org/download/CURRENT/miRNA.dat\n        cat miRNA.dat.raw | awk '{if ($0~/mmu/)print $0}' > miRNA.str\n        rm miRNA.dat.raw\n        wget --random-wait --retry-connrefused -nv -c -O mirbase.gff3 https://mirbase.org/download/mmu.gff3\n        wget --random-wait --retry-connrefused  --no-check-certificate -q -c -O Rfam_for_miRDeep.fa.gz https://github.com/lpantano/mirdeep2_core/raw/data/Rfam_for_miRDeep.fa.gz && gunzip Rfam_for_miRDeep.fa.gz\n        # targetscan analysis\n        # wget --no-check-certificate -q -c -O Summary_Counts.txt.zip http://www.targetscan.org/mmu_71/mmu_71_data_download/Summary_Counts.all_predictions.txt.zip && unzip Summary_Counts.txt.zip\n        # wget --no-check-certificate -q -c -O miR_Family_Info.txt.zip http://www.targetscan.org/mmu_71/mmu_71_data_download/miR_Family_Info.txt.zip && unzip miR_Family_Info.txt.zip\n        # wget --no-check-certificate -q -N -c ftp://mirbase.org/pub/mirbase/21/database_files/mirna_mature.txt.gz\n        # tdrmapper\n        wget  --random-wait --retry-connrefused --no-check-certificate -c -O trna_mature_pre.fa https://github.com/sararselitsky/tDRmapper/raw/master/mm10_mature_pre_for_tdrMapper.fa\n    recipe_outfiles:\n        - srnaseq/srna-transcripts.gtf\n        - srnaseq/hairpin.fa\n        - srnaseq/mature.fa\n        - srnaseq/mirbase.gff3\n        - srnaseq/miRNA.str\n        - srnaseq/Rfam_for_miRDeep.fa\n        - srnaseq/trna_mature_pre.fa\n"
  },
  {
    "path": "ggd-recipes/mm10/prioritize.yaml",
    "content": "# Regions for prioritization:\n# cancer\n#  - az-cancer-panel -- az300 genes plus additional useful panel genes\n# mm10 version contributed by Ramiya Gurrapu\n---\nattributes:\n  name: prioritize\n  version: 20210329\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/bcbio-nextgen/coverage/prioritize/prioritize-cancer-mm10-20210329.tar.gz\n        outdir=coverage/prioritize\n        mkdir -p $outdir\n        cd $outdir\n        wget --no-check-certificate -c -O cancer.tar.gz $baseurl\n        tar -xzvpf cancer.tar.gz\n    recipe_outfiles:\n      - coverage/prioritize/cancer/az-cancer-panel.bed.gz\n      - coverage/prioritize/cancer/az-cancer-panel.bed.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/mm10/problem_regions.yaml",
    "content": "# Problematic genomic regions for annotations and improving variant call comparisons\n# mm10 LCRs from Brent Pedersen: http://figshare.com/articles/LCR_mm10_bed_gz/1180124\n---\nattributes:\n  name: problem_regions\n  version: 20151028\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        repeats=coverage/problem_regions/repeats\n        mkdir -p $repeats\n        wget --no-check-certificate -O - http://files.figshare.com/1688228/LCR_mm10.bed.gz | gunzip -c | bgzip -c > $repeats/LCR.bed.gz\n        tabix -p vcf -f $repeats/LCR.bed.gz\n    recipe_outfiles:\n      - coverage/problem_regions/repeats/LCR.bed.gz\n      - coverage/problem_regions/repeats/LCR.bed.gz.tbi\n"
  },
  {
    "path": "ggd-recipes/mm10/rmsk.yaml",
    "content": "# repeat elements location on the genome\n---\nattributes:\n  name: rmsk\n  version: 20180319\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p coverage/problem_regions/repeats\n        cd coverage/problem_regions/repeats\n        wget  --random-wait --retry-connrefused -q -c -O rmsk.txt.gz http://hgdownload.soe.ucsc.edu/goldenPath/mm10/database/rmsk.txt.gz\n        zcat rmsk.txt.gz | awk '{print $6\"\\t.\\trepeat\\t\"$7+1\"\\t\"$8+1\"\\t.\\t\"$10\"\\t.\\tname \"$12\";\"}' | gzip > rmsk.gtf.gz\n    recipe_outfiles:\n      - coverage/problem_regions/repeats/rmsk.gtf.gz\n"
  },
  {
    "path": "ggd-recipes/mm10/seq.yaml",
    "content": "---\nattributes:\n  name: seq\n  version: ucsc-201112\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        wget --no-check-certificate -c https://s3.amazonaws.com/biodata/genomes/mm10-seq.tar.xz\n        tar -xJvpf mm10-seq.tar.xz\n    recipe_outfiles:\n      - seq/mm10.fa\n      - seq/mm10.fa.fai\n      - seq/mm10.dict\n"
  },
  {
    "path": "ggd-recipes/mm10/transcripts.yaml",
    "content": "# Transcript, annotation and indices for RNA-seq analysis\n# Prepared from Ensembl transcripts using: \n# https://github.com/chapmanb/cloudbiolinux/blob/master/utils/prepare_tx_gff.py\n# Ensembl genebuild 94\n---\nattributes:\n  name: transcripts\n  version: 2018-10-10_92\n\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/biodata/annotation/mm10-rnaseq-2018-10-10_92.tar.xz\n        wget -c -N --no-check-certificate $baseurl\n        xz -dc *-rnaseq-*.tar.xz | tar -xpf -\n        mv */rnaseq-* rnaseq\n    recipe_outfiles:\n      - rnaseq/ref-transcripts.bed\n      - rnaseq/ref-transcripts.dexseq.gff3\n      - rnaseq/ref-transcripts.fa\n      - rnaseq/ref-transcripts.genePred\n      - rnaseq/ref-transcripts.gtf\n      - rnaseq/ref-transcripts.gtf.db\n      - rnaseq/ref-transcripts-mask.gtf\n      - rnaseq/ref-transcripts.refFlat\n      - rnaseq/ref-transcripts-splicesites.txt\n      - rnaseq/rRNA.gtf\n      - rnaseq/rRNA.gtf.db\n      - rnaseq/rRNA.interval_list\n      - rnaseq/tx2gene.csv\n      - rnaseq/version.txt\n"
  },
  {
    "path": "ggd-recipes/mm10/twobit.yaml",
    "content": "# UCSC twobit reference\n---\nattributes:\n  name: twobit\n  version: mm10\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p ucsc\n        cd ucsc\n        faToTwoBit ../../seq/mm10.fa mm10.2bit\n    recipe_outfiles:\n      - ucsc/mm10.2bit\n"
  },
  {
    "path": "ggd-recipes/mm10/vcfanno.yaml",
    "content": "# Annotation files for vcfanno variant annotation\n---\nattributes:\n  name: vcfanno\n  version: 20180427\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        githash=b9bc5cc34dbedaf49ff41ca6249d49b3bb74e67a\n        mkdir -p config/vcfanno\n        cd config/vcfanno\n        bcbio_baseurl=https://github.com/chapmanb/bcbio-nextgen/raw/$githash/config/vcfanno/\n        wget --no-check-certificate -O gemini.conf -c $bcbio_baseurl/mm10-gemini.conf\n    recipe_outfiles:\n      - config/vcfanno/gemini.conf\n"
  },
  {
    "path": "ggd-recipes/rn6/mirbase.yaml",
    "content": "# small RNA seq annotation from different sources\n---\nattributes:\n  name: srnaseq\n  version: 20161103\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p srnaseq\n        cd srnaseq\n        wget -q -N -c ftp://ftp.ensembl.org/pub/release-86/gtf/rattus_norvegicus/Rattus_norvegicus.Rnor_6.0.86.gtf.gz\n        zcat  Rattus_norvegicus.Rnor_6.0.86.gtf.gz | awk '$5-$4 < 500' | grep -v \"^#\"> srna-transcripts.gtf\n        wget -q -N -c http://hgdownload.soe.ucsc.edu/goldenPath/rn6/database/rmsk.txt.gz\n        zcat rmsk.txt.gz | awk '{print $6\"\\t.\\trepeat\\t\"$7+1\"\\t\"$8+1\"\\t.\\t\"$10\"\\t.\\tname \"$12\";\"}' | sed 's/^chr//' | sed 's/^M/MT/' >> srna-transcripts.gtf\n        # wget -q -N -c ftp://mirbase.org/pub/mirbase/20/genomes/rno.gff3\n        # awk '$3==\"miRNA\"' mmu.gff3 | sed 's/=/ /g' >> srna-transcripts.gtf\n        wget -q -c -O hairpin.fa.gz ftp://mirbase.org/pub/mirbase/21/hairpin.fa.gz\n        zcat hairpin.fa.gz |  awk '{if ($0~/>rno/){name=$0; print name} else if ($0~/^>/){name=0};if (name!=0 && $0!~/^>/){print $0;}}' | sed 's/U/T/g'  > hairpin.fa\n        wget -q -c -O mature.fa.gz ftp://mirbase.org/pub/mirbase/21/mature.fa.gz\n        zcat mature.fa.gz |  awk '{if ($0~/>rno/){name=$0; print name} else if ($0~/^>/){name=0};if (name!=0 && $0!~/^>/){print $0;}}' | sed 's/U/T/g'  > mature.fa\n        wget -q -c -O miRNA.str.gz ftp://mirbase.org/pub/mirbase/21/miRNA.str.gz\n        zcat miRNA.str.gz | awk '{if ($0~/rno/)print $0}' > miRNA.str\n        wget --no-check-certificate -q -c -O Rfam_for_miRDeep.fa.gz https://github.com/lpantano/mirdeep2_core/raw/data/Rfam_for_miRDeep.fa.gz && gunzip Rfam_for_miRDeep.fa.gz\n        # targetscan analysis\n        # tdrmapper\n    recipe_outfiles:\n        - srnaseq/srna-transcripts.gtf\n        - srnaseq/hairpin.fa\n        - srnaseq/mature.fa\n        - srnaseq/miRNA.str\n        - srnaseq/Rfam_for_miRDeep.fa\n"
  },
  {
    "path": "ggd-recipes/rn6/seq.yaml",
    "content": "# Full hg38/GRCh38 reference genome distributed by 1000 genomes\n# Derived from NCBI set with HLA and decoy alternative alleles\n# http://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/GRCh38_reference_genome/\n---\nattributes:\n  name: seq\n  version: 85\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        url=http://ftp.ensembl.org/pub/release-85/fasta/rattus_norvegicus/dna/Rattus_norvegicus.Rnor_6.0.dna.toplevel.fa.gz\n        mkdir -p seq\n        wget --no-check-certificate -O - $url | gunzip -c > seq/rn6.fa\n    recipe_outfiles:\n      - seq/rn6.fa\n"
  },
  {
    "path": "ggd-recipes/rn6/transcripts.yaml",
    "content": "# Transcript, annotation and indices for RNA-seq analysis\n# Prepared from Ensembl transcripts using:\n# https://github.com/chapmanb/cloudbiolinux/blob/master/utils/prepare_tx_gff.py\n---\nattributes:\n  name: transcripts\n  version: 2018-03-22_91\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/biodata/annotation/rn6-rnaseq-2018-03-22_91.tar.xz\n        wget -c -N --no-check-certificate $baseurl\n        xz -dc *-rnaseq-*.tar.xz | tar -xpf -\n        mv */rnaseq-* rnaseq\n    recipe_outfiles:\n      - rnaseq/ref-transcripts.bed\n      - rnaseq/ref-transcripts.dexseq.gff3\n      - rnaseq/ref-transcripts.fa\n      - rnaseq/ref-transcripts.genePred\n      - rnaseq/ref-transcripts.gtf\n      - rnaseq/ref-transcripts.gtf.db\n      - rnaseq/ref-transcripts-mask.gtf\n      - rnaseq/ref-transcripts.refFlat\n      - rnaseq/ref-transcripts-splicesites.txt\n      - rnaseq/rRNA.gtf\n      - rnaseq/rRNA.gtf.db\n      - rnaseq/rRNA.interval_list\n      - rnaseq/tx2gene.csv\n      - rnaseq/version.txt\n"
  },
  {
    "path": "ggd-recipes/rn6/twobit.yaml",
    "content": "# UCSC twobit reference\n---\nattributes:\n  name: twobit\n  version: 85\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        mkdir -p ucsc\n        cd ucsc\n        faToTwoBit ../../seq/rn6.fa rn6.2bit\n    recipe_outfiles:\n      - ucsc/rn6.2bit\n"
  },
  {
    "path": "ggd-recipes/sacCer3/seq.yaml",
    "content": "---\nattributes:\n  name: seq\n  version: 94\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        RELEASE=94\n        BUILD=sacCer3\n        url=ftp://ftp.ensembl.org/pub/release-$RELEASE/fasta/saccharomyces_cerevisiae/dna/Saccharomyces_cerevisiae.R64-1-1.dna.toplevel.fa.gz\n        mkdir -p seq\n        wget -O - --no-check-certificate https://gist.githubusercontent.com/chapmanb/a973558c2ec67a57eb4592c709ec0d09/raw/81d398fd7f3c901f89753d4f9f39c5a87d05a7fa/sacCer3.chrom.sizes | cut -f 1 > sacCer3.chrom.sizes\n        wget -c --no-check-certificate https://raw.githubusercontent.com/kwongj/sort-contigs/master/sort-contigs.py\n        wget --no-check-certificate -O - $url | gunzip -c | python -c 'import sys; [sys.stdout.write(\">chr%s\\n\" % (l[1:].split()[0].replace(\"Mito\", \"M\")) if l.startswith(\">\") else l) for l in sys.stdin]' > seq/$BUILD.fa.orig\n        python2 sort-contigs.py seq/$BUILD.fa.orig --out seq/$BUILD.fa --fmt fasta --order sacCer3.chrom.sizes\n        samtools faidx seq/$BUILD.fa\n        picard -Xms1g -Xmx3g CreateSequenceDictionary REFERENCE=seq/$BUILD.fa OUTPUT=seq/$BUILD.dict SPECIES=$BUILD\n    recipe_outfiles:\n      - seq/sacCer3.fa\n      - seq/sacCer3.fa.fai\n      - seq/sacCer3.dict\n"
  },
  {
    "path": "ggd-recipes/sacCer3/transcripts.yaml",
    "content": "# Transcript, annotation and indices for RNA-seq analysis\n# Prepared from Ensembl transcripts using: \n# https://github.com/chapmanb/cloudbiolinux/blob/master/utils/prepare_tx_gff.py\n---\nattributes:\n  name: transcripts\n  version: 2018-11-20_94\nrecipe:\n  full:\n    recipe_type: bash\n    recipe_cmds:\n      - |\n        baseurl=https://s3.amazonaws.com/biodata/annotation/sacCer3-rnaseq-2018-11-20_94.tar.xz\n        wget -c -N --no-check-certificate $baseurl\n        xz -dc *-rnaseq-*.tar.xz | tar -xpf -\n        mv */rnaseq-* rnaseq\n    recipe_outfiles:\n      - rnaseq/ref-transcripts.bed\n      - rnaseq/ref-transcripts.dexseq.gff3\n      - rnaseq/ref-transcripts.fa\n      - rnaseq/ref-transcripts.genePred\n      - rnaseq/ref-transcripts.gtf\n      - rnaseq/ref-transcripts.gtf.db\n      - rnaseq/ref-transcripts-mask.gtf\n      - rnaseq/ref-transcripts.refFlat\n      - rnaseq/ref-transcripts-splicesites.txt\n      - rnaseq/rRNA.gtf\n      - rnaseq/rRNA.gtf.db\n      - rnaseq/rRNA.interval_list\n      - rnaseq/tx2gene.csv\n      - rnaseq/version.txt\n"
  },
  {
    "path": "installed_files/bash_history",
    "content": "sudo -i\nsudo su galaxy\ntail -444f /mnt/cm/paster.log\n"
  },
  {
    "path": "installed_files/bash_login",
    "content": "#!/bin/bash\n\n# Here is a suitable bash_login to drop into /home/ubuntu/.bash_login\n# Ensures this is an interactive ssh login, then asks the user if they\n# want to enable NX functionality.\n\nflagfile=~/.nx_setup_done\nsetupscript=setupnx.sh\ncont=yes\n\n# Do I go ahead?\n[ -e \"$flagfile\" ] && cont=no\n[ -n \"$SSH_CONNECTION\" ] || cont=no\ntty -s || cont=no\n\nif [ \"$cont\" = yes ] ; then\ncat <<BLURB \n\nCloudBioLinux comes with FreeNX server to provide a fast remote\ndesktop on your cloud server instance.  Do you wish to configure this now?\n\n  y = Yes\n  n = No, but ask me again next time.\n  x = No, and do not ask again.\nBLURB\n\nread -n 1 -p $'\\n [ ]\\b\\b' answer\nanswer=`echo \"$answer\" | tr 'YNX' 'ynx'`\necho\n\nif [ \"$answer\" = y ] ; then\n    #Setupscript needs to be executable already.\n    sudo \"$setupscript\" $USER && touch \"$flagfile\"\nelif [ \"$answer\" = x ] ; then\n    touch \"$flagfile\"\n    echo \"To initiate setup manually, type 'sudo $setupscript <username>'.\"\nelse\n    echo \"Quitting.\"   \nfi\nfi\n"
  },
  {
    "path": "installed_files/ec2autorun.py",
    "content": "#!/usr/bin/env python\n\"\"\"\nThis is a contextualization script required by CloudMan; it is automatically run\nat instance startup (via an upstart job).\n\nRequires:\n    PyYAML http://pyyaml.org/wiki/PyYAMLDocumentation (easy_install pyyaml)\n    boto http://code.google.com/p/boto/ (easy_install boto)\n\nAssumptions:\n    DEFAULT_BUCKET_NAME and DEFAULT_BOOT_SCRIPT_NAME are publicly accessible and\n    do not require any form of authentication\n\"\"\"\n\nimport os, sys, yaml, urllib2, logging, hashlib, time, subprocess, random\nfrom urlparse import urlparse\n\nfrom boto.s3.key import Key\nfrom boto.s3.connection import S3Connection\nfrom boto.exception import S3ResponseError\nfrom boto.s3.connection import OrdinaryCallingFormat\n\nlogging.getLogger('boto').setLevel(logging.INFO) # Only log boto messages >=INFO\nlog = None\n\nUSER_DATA_URL = 'http://169.254.169.254/latest/user-data'\n# USER_DATA_URL = 'http://userwww.service.emory.edu/~eafgan/content/userData.yaml.sample' # used for testing\n# USER_DATA_URL = 'http://userwww.service.emory.edu/~eafgan/content/url_ud.txt' # used for testing\nLOCAL_PATH = '/tmp/cm' # Local path destination used for storing/reading any files created by this script\nUSER_DATA_FILE_NAME = 'userData.yaml' # Local file with user data formatted by this script\nUSER_DATA_FILE = os.path.join(LOCAL_PATH, USER_DATA_FILE_NAME) # The final/processed UD file\n# Local file containing UD in its original format\nUSER_DATA_ORIG = os.path.join(LOCAL_PATH, 'original_%s' % USER_DATA_FILE_NAME)\nSERVICE_ROOT = 'http://s3.amazonaws.com/' # Obviously, customized for Amazon's S3\nDEFAULT_BUCKET_NAME = 'cloudman' # Ensure this bucket is accessible to anyone!\nDEFAULT_BOOT_SCRIPT_NAME = 'cm_boot.py' # Ensure this file is accessible to anyone in the public bucket!\nCLOUDMAN_HOME = '/mnt/cm'\n\n# ====================== Utility methods ======================\n\ndef _setup_logging():\n    # Logging setup\n    formatter = logging.Formatter(\"[%(levelname)s] %(module)s:%(lineno)d %(asctime)s: %(message)s\")\n    console = logging.StreamHandler() # log to console - used during testing\n    # console.setLevel(logging.INFO) # accepts >INFO levels\n    console.setFormatter(formatter)\n    # log_file = logging.FileHandler(os.path.join(LOCAL_PATH, \"%s.log\" % os.path.splitext(sys.argv[0])[0]), 'w')\n    # log_file.setLevel(logging.DEBUG) # accepts all levels\n    # log_file.setFormatter(formatter)\n    log = logging.root\n    log.addHandler(console)\n    # log.addHandler(log_file)\n    log.setLevel(logging.DEBUG)\n    return log\n\ndef _get_user_data():\n    ud = ''\n    for i in range(0, 5):\n          try:\n              log.info(\"Getting user data from '%s', attempt %s\" % (USER_DATA_URL, i))\n              fp = urllib2.urlopen(USER_DATA_URL)\n              ud = fp.read()\n              fp.close()\n              log.debug(\"Saving user data in its original format to file '%s'\" % USER_DATA_ORIG)\n              with open(USER_DATA_ORIG, 'w') as ud_orig:\n                  ud_orig.write(ud)\n              if ud:\n                  log.debug(\"Got user data\")\n                  return ud\n          except IOError:\n              log.info(\"User data not found. Setting it to empty.\")\n              return ''\n    # Used for testing\n    # return 'http://s3.amazonaws.com/cloudman/cm_boot'\n    # return ''\n    # return \"gc_dev1|<account_key>|<secret_key>|somePWD\"\n    # with open('sample.yaml') as ud_yaml:\n    #     ud = ud_yaml.read()\n    if ud == '':\n        log.debug(\"Received empty/no user data\")\n    return ud\n\ndef _get_bucket_name(cluster_name, access_key):\n    \"\"\"Compose bucket name based on the user-provided cluster name and user access key\"\"\"\n    m = hashlib.md5()\n    m.update( cluster_name + access_key )\n    return \"cm-\" + m.hexdigest()\n\ndef _isurl(path):\n    \"\"\"Test if path is a net location. Tests the scheme and netloc.\"\"\"\n    # BUG : URLs require a scheme string ('http://') to be used.\n    #       www.google.com will fail.\n    #       Should we prepend the scheme for those that don't have it and\n    #       test that also?\n    scheme, netloc, upath, uparams, uquery, ufrag = urlparse(path)\n    return bool(scheme and netloc)\n\ndef _get_s3_conn(ud):\n    try:\n        if 'cloud_type' in ud and ud['cloud_type'] != 'ec2':\n            # If the user has specified a cloud type other than EC2,\n            # create an s3 connection using the info from their user data\n            log.debug('Establishing boto S3 connection to a custom Object Store')\n            try:\n                s3_conn = S3Connection(aws_access_key_id=ud['access_key'],\n                        aws_secret_access_key=ud['secret_key'],\n                        is_secure=ud.get('is_secure', True),\n                        host=ud.get('s3_host', ''),\n                        port=ud.get('s3_port', 8888),\n                        calling_format=OrdinaryCallingFormat(),\n                        path=ud.get('s3_conn_path', '/'))\n            except S3ResponseError, e:\n                log.error(\"Trouble connecting to a custom Object Store. User data: {0}; Exception: {1}\"\\\n                    .format(ud, e))\n        else:\n            # Use the default Amazon S3 connection\n            log.debug('Establishing boto S3 connection to Amazon')\n            s3_conn = S3Connection(ud['access_key'], ud['secret_key'])\n    except Exception, e:\n        log.error(\"Exception getting S3 connection: %s\" % e)\n        return None\n    return s3_conn\n\n\ndef _bucket_exists(s3_conn, bucket_name):\n    bucket = None\n    for i in range(1, 6):\n        try:\n            # log.debug(\"Looking for bucket '%s'\" % bucket_name)\n            bucket = s3_conn.lookup(bucket_name)\n            break\n        except S3ResponseError:\n            log.error (\"Bucket '%s' not found, attempt %s/5\" % (bucket_name, i+1))\n            time.sleep(2)\n\n    if bucket is not None:\n        log.debug(\"Cluster bucket '%s' found.\" % bucket_name)\n        return True\n    else:\n        log.debug(\"Cluster bucket '%s' not found.\" % bucket_name)\n        return False\n\ndef _remote_file_exists(s3_conn, bucket_name, remote_filename):\n    b = None\n    for i in range(0, 5):\n        try:\n            b = s3_conn.get_bucket(bucket_name)\n            break\n        except S3ResponseError:\n            log.error (\"Problem connecting to bucket '%s', attempt %s/5\" % (bucket_name, i))\n            time.sleep(2)\n\n    if b is not None:\n        k = Key(b, remote_filename)\n        if k.exists():\n            return True\n    return False\n\ndef _save_file_to_bucket(s3_conn, bucket_name, remote_filename, local_file, force=False):\n    local_file = os.path.join(LOCAL_PATH, local_file)\n    # log.debug( \"Establishing handle with bucket '%s'...\" % bucket_name)\n    b = None\n    for i in range(0, 5):\n        try:\n            b = s3_conn.get_bucket(bucket_name)\n            break\n        except S3ResponseError, e:\n            log.error (\"Problem connecting to bucket '%s', attempt %s/5\" % (bucket_name, i))\n            time.sleep(2)\n\n    if b is not None:\n        # log.debug(\"Establishing handle with key object '%s'...\" % remote_filename)\n        k = Key(b, remote_filename)\n        if k.exists() and not force:\n            log.debug(\"Remote file '%s' already exists. Not overwriting it.\" % remote_filename)\n            return True\n        log.debug( \"Attempting to save local file '%s' to bucket '%s' as '%s'\"\n            % (local_file, bucket_name, remote_filename))\n        try:\n            k.set_contents_from_filename(local_file)\n            log.info( \"Successfully saved file '%s' to bucket '%s'.\" % (remote_filename, bucket_name))\n            return True\n        except S3ResponseError, e:\n             log.error(\"Failed to save file local file '%s' to bucket '%s' as file '%s': %s\"\n                % (local_file, bucket_name, remote_filename, e))\n             return False\n    else:\n        return False\n\ndef _get_file_from_bucket(s3_conn, bucket_name, remote_filename, local_filename):\n    local_filename = os.path.join(LOCAL_PATH, local_filename)\n    try:\n        # log.debug(\"Establishing handle with bucket '%s'\" % bucket_name)\n        b = s3_conn.get_bucket(bucket_name)\n\n        # log.debug(\"Establishing handle with file object '%s'\" % remote_filename)\n        k = Key(b, remote_filename)\n\n        log.debug(\"Attempting to retrieve file '%s' from bucket '%s'\" % (remote_filename, bucket_name))\n        if k.exists():\n            k.get_contents_to_filename(local_filename)\n            log.info(\"Successfully retrieved file '%s' from bucket '%s' to '%s'.\"\n                % (remote_filename, bucket_name, local_filename))\n            return True\n        else:\n            log.error(\"File '%s' in bucket '%s' not found.\" % (remote_filename, bucket_name))\n            return False\n    except S3ResponseError, e:\n        log.error(\"Failed to get file '%s' from bucket '%s': %s\" % (remote_filename, bucket_name, e))\n        return False\n\ndef _get_file_from_url(url):\n    local_filename = os.path.join(LOCAL_PATH, os.path.split(url)[1])\n    log.info(\"Getting boot script from '%s' and saving it locally to '%s'\" % (url, local_filename))\n    try:\n        f = urllib2.urlopen(url)\n        with open(local_filename, 'w') as local_file:\n            local_file.write(f.read())\n        os.chmod(local_filename, 0744)\n        if f:\n            log.debug(\"Got boot script from '%s'\" % url)\n            return True\n        return False\n    except IOError:\n        log.error(\"Boot script at '%s' not found.\" % url)\n        return False\n\ndef _get_boot_script(ud):\n    # Test if cluster bucket exists; if it does not, resort to the default\n    # bucket for downloading the boot script\n    use_default_bucket = ud.get(\"use_default_bucket\", False)\n    if ud.has_key('bucket_default'):\n        default_bucket_name = ud['bucket_default']\n    else:\n        default_bucket_name = DEFAULT_BUCKET_NAME\n    if not use_default_bucket and ud.has_key('bucket_cluster') and ud['access_key'] is not None and ud['secret_key'] is not None:\n        s3_conn = _get_s3_conn(ud)\n        # Check if cluster bucket exists or use the default one\n        if not _bucket_exists(s3_conn, ud['bucket_cluster']) or \\\n           not _remote_file_exists(s3_conn, ud['bucket_cluster'], ud['boot_script_name']):\n            log.debug(\"Using default bucket '%s'\" % default_bucket_name)\n            use_default_bucket = True\n        else:\n            log.debug(\"Using cluster bucket '%s'\" % ud['bucket_cluster'])\n            use_default_bucket = False\n    else:\n        log.debug(\"bucket_clutser not specified or no credentials provided; defaulting to bucket '%s'\"\n            % default_bucket_name)\n        use_default_bucket = True\n\n    # If using cluster bucket, use credentials because the boot script may not be accessible to everyone\n    got_boot_script = False\n    if use_default_bucket is False:\n        log.debug(\"Trying to get boot script '%s' from cluster bucket '%s'\"\n            % (ud['boot_script_name'], ud.get('bucket_cluster', None)))\n        got_boot_script = _get_file_from_bucket(s3_conn, ud['bucket_cluster'], ud['boot_script_name'],\n            DEFAULT_BOOT_SCRIPT_NAME)\n        if got_boot_script:\n            os.chmod(os.path.join(LOCAL_PATH, DEFAULT_BOOT_SCRIPT_NAME), 0744)\n    # If did not get the boot script, fall back on the publicly available one\n    if not got_boot_script or use_default_bucket:\n        boot_script_url = os.path.join(_get_default_bucket_url(ud), ud.get('boot_script_name',\n            DEFAULT_BOOT_SCRIPT_NAME))\n        log.debug(\"Could not get boot script '%s' from cluster bucket '%s'; \"\n            \"retrieving the public one from bucket url '%s'\" \\\n            % (ud['boot_script_name'], ud.get('bucket_cluster', None), boot_script_url))\n        got_boot_script = _get_file_from_url(boot_script_url)\n    if got_boot_script:\n        log.debug(\"Saved boot script to '%s'\" % os.path.join(LOCAL_PATH, DEFAULT_BOOT_SCRIPT_NAME))\n        # Save the downloaded boot script to cluster bucket for future invocations\n        use_object_store = ud.get(\"use_object_store\", True)\n        if use_object_store and ud.has_key('bucket_cluster') and ud['bucket_cluster']:\n            s3_conn = _get_s3_conn(ud)\n            if _bucket_exists(s3_conn, ud['bucket_cluster']) and \\\n               not _remote_file_exists(s3_conn, ud['bucket_cluster'], ud['boot_script_name']):\n                _save_file_to_bucket(s3_conn, ud['bucket_cluster'], ud['boot_script_name'], \\\n                    DEFAULT_BOOT_SCRIPT_NAME)\n        return True\n    log.debug(\"**Could not get the boot script**\")\n    return False\n\ndef _run_boot_script(boot_script_name):\n    script = os.path.join(LOCAL_PATH, boot_script_name)\n    log.info(\"Running boot script '%s'\" % script)\n    process = subprocess.Popen(script, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n    stdout, stderr = process.communicate()\n    if process.returncode == 0:\n        log.debug(\"Successfully ran boot script '%s'\" % script)\n        return True\n    else:\n        log.error(\"Error running boot script '%s'. Process returned code '%s' and following stderr: %s\"\n            % (script, process.returncode, stderr))\n        return False\n\ndef _create_basic_user_data_file():\n    # Create a basic YAML file that is expected by CloudMan\n    with open(USER_DATA_FILE, 'w') as ud_file:\n        ud_formatted = {'access_key': None,\n                        'boot_script_name': DEFAULT_BOOT_SCRIPT_NAME,\n                        'boot_script_path': LOCAL_PATH,\n                        'bucket_default': DEFAULT_BUCKET_NAME,\n                        'bucket_cluster': None,\n                        'cloudman_home': CLOUDMAN_HOME,\n                        'cluster_name': 'aGalaxyCloudManCluster_%s' % random.randrange(1, 9999999),\n                        'role': 'master',\n                        'secret_key': None}\n        yaml.dump(ud_formatted, ud_file, default_flow_style=False)\n    return ud_formatted\n\ndef _get_default_bucket_url(ud=None):\n    if ud and ud.has_key('bucket_default'):\n        default_bucket_name = ud['bucket_default']\n    else:\n        default_bucket_name = DEFAULT_BUCKET_NAME\n    # TODO: Check if th bucket 'default_bucket_name' is accessible to everyone\n    # because it is being accessed as a URL\n    if ud:\n        bucket_url = ud.get(\"default_bucket_url\", None)\n    else:\n        bucket_url = None\n    if not bucket_url:\n        bucket_url = os.path.join(SERVICE_ROOT, default_bucket_name)\n    log.debug(\"Default bucket url: %s\" % bucket_url)\n    return bucket_url\n\ndef _user_exists(username):\n    \"\"\" Check if the given username exists as a system user\n    \"\"\"\n    with open('/etc/passwd', 'r') as f:\n        ep = f.read()\n    return ep.find(username) > 0\n\ndef _allow_password_logins(passwd):\n    for user in [\"ubuntu\", \"galaxy\"]:\n        if _user_exists(user):\n            log.info(\"Setting up password-based login for user '{0}'\".format(user))\n            p1 = subprocess.Popen([\"echo\", \"%s:%s\" % (user, passwd)], stdout=subprocess.PIPE)\n            p2 = subprocess.Popen([\"chpasswd\"], stdin=p1.stdout, stdout=subprocess.PIPE)\n            p1.stdout.close()\n            p2.communicate()[0]\n            cl = [\"sed\", \"-i\", \"s/^PasswordAuthentication .*/PasswordAuthentication yes/\",\n                  \"/etc/ssh/sshd_config\"]\n            subprocess.check_call(cl)\n            cl = [\"/usr/sbin/service\", \"ssh\", \"reload\"]\n            subprocess.check_call(cl)\n\ndef _handle_freenx(passwd):\n    # Check if FreeNX is installed on the image before trying to configure it\n    cl = \"/usr/bin/dpkg --get-selections | /bin/grep freenx\"\n    retcode = subprocess.call(cl, shell=True)\n    if retcode == 0:\n        log.info(\"Setting up FreeNX\")\n        cl = [\"dpkg-reconfigure\", \"-pcritical\", \"freenx-server\"]\n        # On slower/small instance types, there can be a conflict when running\n        # debconf so try this a few times\n        for i in range(5):\n            retcode = subprocess.call(cl)\n            if retcode == 0:\n                break\n            else:\n                time.sleep(5)\n    else:\n        log.info(\"freenx-server is not installed; not configuring it\")\n\n# ====================== Actions methods ======================\n\ndef _handle_empty():\n    log.info(\"Received empty user data; assuming default contextualization\")\n    _create_basic_user_data_file() # This file is expected by CloudMan\n    # Get & run boot script\n    file_url = os.path.join(_get_default_bucket_url(), DEFAULT_BOOT_SCRIPT_NAME)\n    log.debug(\"Resorting to the default bucket to get the boot script: %s\" % file_url)\n    _get_file_from_url(file_url)\n    _run_boot_script(DEFAULT_BOOT_SCRIPT_NAME)\n\ndef _handle_url(url):\n    log.info(\"Handling user data provided URL: '%s'\" % url)\n    _get_file_from_url(url)\n    boot_script_name = os.path.split(url)[1]\n    _run_boot_script(boot_script_name)\n\n\n#http://stackoverflow.com/questions/823196/yaml-merge-in-python\ndef _merge(specific, default):\n    \"\"\"\n    Recursively merges two yaml produced data structures,\n    a more specific input (`specific`) and defaults\n    (`default`).\n    \"\"\"\n    if isinstance(specific, dict) and isinstance(default, dict):\n        for k, v in default.iteritems():\n            if k not in specific:\n                specific[k] = v\n            else:\n                specific[k] = _merge(specific[k], v)\n    return specific\n\ndef _load_user_data(user_data):\n    \"\"\" Loads user data into dict (using pyyaml). If machine image\n    contains default data this is loaded and populated in resulting\n    data structure as well. These separate options are merged using\n    the `_merge` function above and priority is always given to\n    user supplied options.\n    \"\"\"\n    ud = yaml.safe_load(user_data)\n    if ud == user_data:\n        # Bad user data, cannot merge default\n        return ud\n    default_user_data_path = \\\n        os.path.join(os.path.dirname(os.path.abspath(__file__)), 'IMAGE_USER_DATA')\n    if os.path.exists(default_user_data_path):\n        image_ud = yaml.safe_load(open(default_user_data_path, 'r').read())\n        if image_ud:\n            ud = _merge(ud, image_ud)\n    return ud\n\ndef _handle_yaml(user_data):\n    \"\"\" Process user data in YAML format\"\"\"\n    log.info(\"Handling user data in YAML format.\")\n    ud = _load_user_data(user_data)\n    # Handle bad user data as a string\n    if ud == user_data:\n        return _handle_empty()\n    # Allow password based logins. Do so also in case only NX is being setup.\n    if \"freenxpass\" in ud or \"password\" in ud:\n        passwd = ud.get(\"freenxpass\", None) or ud.get(\"password\", None)\n        _allow_password_logins(passwd)\n    # Handle freenx passwords and the case with only a NX password sent\n    if \"freenxpass\" in ud:\n        _handle_freenx(ud[\"freenxpass\"])\n        if len(ud) == 1:\n            return _handle_empty()\n    # Create a YAML file from user data and store it as USER_DATA_FILE\n    # This code simply ensures fields required by CloudMan are in the\n    # created file. Any other fields that might be included as user data\n    # are also included in the created USER_DATA_FILE\n    if ud.get('no_start', None) is not None:\n        log.info(\"Received 'no_start' user data option. Not doing anything else.\")\n        return\n    if not ud.has_key('cluster_name'):\n        log.warning(\"The provided user data should contain cluster_name field.\")\n        ud['cluster_name'] = 'aCloudManCluster_%s' % random.randrange(1, 9999999)\n    elif ud['cluster_name'] == '':\n        log.warning(\"The cluster_name field of user data should not be empty.\")\n        ud['cluster_name'] = 'aCloudManCluster_%s' % random.randrange(1, 9999999)\n\n    if not ud.has_key('access_key'):\n        log.info(\"The provided user data does not contain access_key field; setting it to None..\")\n        ud['access_key'] = None\n    elif ud['access_key'] == '' or ud['access_key'] is None:\n        log.warning(\"The access_key field of user data should not be empty; setting it to None.\")\n        ud['access_key'] = None\n\n    if not ud.has_key('secret_key'):\n        log.info(\"The provided user data does not contain secret_key field; setting it to None.\")\n        ud['secret_key'] = None\n    elif ud['secret_key'] == '' or ud['secret_key'] is None:\n        log.warning(\"The secret_key field of user data should not be empty; setting it to None.\")\n        ud['secret_key'] = None\n\n    if not ud.has_key('password'):\n        log.warning(\"The provided user data should contain password field.\")\n    elif ud['password'] == '':\n        log.warning(\"The password field of user data should not be empty.\")\n    else: # ensure the password is a string\n        ud['password'] = str(ud['password'])\n\n    if not ud.has_key('bucket_default'):\n        log.debug(\"The provided user data does not contain bucket_default field; setting it to '%s'.\"\n            % DEFAULT_BUCKET_NAME)\n        ud['bucket_default'] = DEFAULT_BUCKET_NAME\n    elif ud['bucket_default'] == '':\n        log.warning(\"The bucket_default field of user data was empty; setting it to '%s'.\"\n            % DEFAULT_BUCKET_NAME)\n        ud['bucket_default'] = DEFAULT_BUCKET_NAME\n\n    if not ud.has_key('bucket_cluster'):\n        if ud['access_key'] is not None and ud['secret_key'] is not None:\n            ud['bucket_cluster'] = _get_bucket_name(ud['cluster_name'], ud['access_key'])\n\n    if not ud.has_key('role'):\n        ud['role'] = 'master'\n\n    if not ud.has_key('cloudman_home'):\n        ud['cloudman_home'] = CLOUDMAN_HOME\n\n    if not ud.has_key('boot_script_name'):\n        ud['boot_script_name'] = DEFAULT_BOOT_SCRIPT_NAME\n    ud['boot_script_path'] = LOCAL_PATH # Marks where boot script was saved\n\n    log.debug(\"Composed user data: %s\" % ud)\n    with open(USER_DATA_FILE, 'w') as ud_yaml:\n        yaml.dump(ud, ud_yaml, default_flow_style=False)\n\n    # Get & run boot script\n    if _get_boot_script(ud):\n        _run_boot_script(DEFAULT_BOOT_SCRIPT_NAME)\n\n# ====================== Driver code ======================\n\ndef _parse_user_data(ud):\n    if ud == '':\n        _handle_empty()\n    elif _isurl(ud):\n        _handle_url(ud)\n    else: # default to yaml\n        _handle_yaml(ud)\n\ndef main():\n    if not os.path.exists(LOCAL_PATH):\n        os.mkdir(LOCAL_PATH)\n    global log\n    log = _setup_logging()\n    ud = _get_user_data()\n    _parse_user_data(ud)\n    log.info(\"---> %s done <---\" % sys.argv[0])\n\nif __name__ == \"__main__\":\n    main()\n"
  },
  {
    "path": "installed_files/galaxy_default.template",
    "content": "export GALAXY_USER=galaxy\nexport GALAXY_HOME=$galaxy_home\nexport PIDFILE=/var/lib/galaxy/galaxy.pid\nexport GALAXY_INDICES=$galaxy_indices_mount\nexport GALAXY_DATA=$galaxy_data_mount"
  },
  {
    "path": "installed_files/galaxy_init",
    "content": "#!/bin/sh\n\n### BEGIN INIT INFO\n# Provides:        galaxy\n# Required-Start:  \n# Required-Stop:   \n# Default-Start:   2 3 4 5\n# Default-Stop:    0 1 6\n# Short-Description: Start Galaxy daemon\n### END INIT INFO\n\nPATH=/sbin:/bin:/usr/sbin:/usr/bin\n\n. /lib/lsb/init-functions\n. /etc/default/galaxy\n\nNAME=galaxy\n\ncase $1 in\n        start)\n                if [ ! -d $GALAXY_DATA ];\n                then\n                    log_daemon_msg \"Creating directory for galaxy data\"\n                    mkdir -p $GALAXY_DATA\n                    chown $GALAXY_USER:$GALAXY_USER $GALAXY_DATA\n                fi\n                if [ ! -d $GALAXY_INDICES -a -f /opt/compressed_genomes.tar.gz ];\n                then\n                    # Galaxy VM launcher has option to stash compressed genomes right\n                    # in image in compressed state. Look for such a file in /opt/compressed_genomes.tar.gz\n                    # and decompress in appropriate directory if needed.\n                    log_daemon_msg \"Decompressing genomes for galaxy\"\n                    cp /opt/compressed_genomes.tar.gz $GALAXY_INDICES/..\n                    tar xzvf $GALAXY_INDICES/../compressed_genomes.tar.gz -C $GALAXY_INDICES/..\n                fi\n\n                log_daemon_msg \"Starting Galaxy\" \"Galaxy\"\n                su - $GALAXY_USER /bin/sh -c \"$GALAXY_HOME/run.sh --pid-file=$PIDFILE --log-file /var/log/galaxy/galaxy.log  --daemon\"\n                status=$?\n                log_end_msg $status\n                ;;\n        stop)\n                log_daemon_msg \"Stopping Galaxy\" \"Galaxy\"\n                su - $GALAXY_USER /bin/sh -c \"$GALAXY_HOME/run.sh --pid-file=$PIDFILE --stop-daemon\"\n                log_end_msg $?\n                rm -f $PIDFILE\n                ;;\n        restart|force-reload)\n                $0 stop && sleep 2 && $0 start\n                ;;\n        try-restart)\n                if $0 status >/dev/null; then\n                        $0 restart\n                else\n                        exit 0\n                fi\n                ;;\n        reload)\n                exit 3\n                ;;\n        *)\n                echo \"Usage: $0 {start|stop|restart|try-restart|force-reload}\"\n                exit 2\n                ;;\nesac\n"
  },
  {
    "path": "installed_files/galaxyp_nginx.conf.template",
    "content": "worker_processes  2;\n\nevents {\n    worker_connections  1024;\n}\n\nhttp {\n    include       mime.types;\n    default_type  application/octet-stream;\n    sendfile        on;\n    keepalive_timeout  65;\n\n    gzip  on;\n    gzip_http_version 1.1;\n    gzip_vary on;\n    gzip_comp_level 4;\n    gzip_proxied any;\n    gzip_types text/plain text/css application/x-javascript text/xml application/xml text/javascript application/json;\n    gzip_buffers 16 8k;\n    gzip_disable \"MSIE [1-6].(?!.*SV1)\";\n\n    upstream galaxy_app {\n        server localhost:8080;\n    }\n\n    upstream cm_app {\n        server localhost:42284;\n    }\n\n    upstream protvis-site {\n        server 127.0.0.1:$protvis_port;\n    }\n\n    server {\n        listen       80;\n        client_max_body_size 2048m;\n        server_name  localhost;\n                proxy_read_timeout 600;\n\n        location /cloud {\n            proxy_pass  http://cm_app;\n            proxy_set_header   X-Forwarded-Host $$host;\n            proxy_set_header   X-Forwarded-For  $$proxy_add_x_forwarded_for;\n            error_page   502    /errdoc/cm_502.html;\n        }\n\n        location /cloud/static {\n            alias /mnt/cm/static;\n            expires 24h;\n        }\n\n        location /cloud/static/style {\n            alias /mnt/cm/static/style;\n            expires 24h;\n        }\n\n        location /cloud/static/scripts {\n            alias /mnt/cm/static/scripts;\n            expires 24h;\n        }\n\n        location / {\n            proxy_pass  http://galaxy_app;\n            proxy_set_header   X-Forwarded-Host $$host;\n            proxy_set_header   X-Forwarded-For  $$proxy_add_x_forwarded_for;\n        }\n\n        location /static {\n            alias $galaxy_home/static;\n            expires 24h;\n        }\n\n        location /static/style {\n            alias $galaxy_home/static/june_2007_style/blue;\n            expires 24h;\n        }\n\n        location /static/scripts {\n            alias $galaxy_home/static/scripts/packed;\n            expires 24h;\n        }\n\n        location /robots.txt {\n            alias $galaxy_home/static/robots.txt;\n        }\n\n        location /favicon.ico {\n            alias $galaxy_home/static/favicon.ico;\n        }\n\n        location /_x_accel_redirect/ {\n            internal;\n            alias /;\n        }\n\n        location /_upload {\n            upload_store /mnt/galaxyData/upload_store;\n            upload_pass_form_field \"\";\n            upload_set_form_field \"__$${upload_field_name}__is_composite\" \"true\";\n            upload_set_form_field \"__$${upload_field_name}__keys\" \"name path\";\n            upload_set_form_field \"$${upload_field_name}_name\" \"$$upload_file_name\";\n            upload_set_form_field \"$${upload_field_name}_path\" \"$$upload_tmp_path\";\n            upload_pass_args on;\n            upload_pass /_upload_done;\n        }\n\n        location /_upload_done {\n            set $$dst /tool_runner/index;\n            if ($$args ~ nginx_redir=([^&]+)) {\n                set $$dst $$1;\n            }\n            rewrite \"\" $$dst;\n        }\n\n        error_page   502    /errdoc/502.html;\n        location /errdoc {\n            root   html;\n        }\n\n        location /protvis {\n            rewrite                 ^/protvis/(.*) /\\$1 break;\n            proxy_set_header        Host $host;\n            proxy_set_header        X-Real-IP \\$remote_addr;\n            proxy_set_header        X-Forwarded-For \\$proxy_add_x_forwarded_for;\n            proxy_set_header        X-Forwarded-Proto \\$scheme;\n\n            client_max_body_size    16m;\n            client_body_buffer_size 128k;\n            proxy_connect_timeout   60s;\n            proxy_send_timeout      90s;\n            proxy_read_timeout      90s;\n            proxy_buffering         off;\n            proxy_temp_file_write_size 64k;\n            proxy_pass http://protvis-site;\n            proxy_redirect          off;\n        }\n\n        location /protvis/res/ {\n             alias $protvis_home/res/;\n             expires                 30d;\n             add_header              Cache-Control public;\n             access_log              off;\n        }\n\n    }\n\n}\n"
  },
  {
    "path": "installed_files/image_user_data",
    "content": "# YAML properties placed in this file will act as defaults\n# for CloudMan user data. User supplied data at launch time \n# will override these properties.\n"
  },
  {
    "path": "installed_files/ipython_config.py",
    "content": "# Configuration file for ipython.\n\nc = get_config()\n\nc.InteractiveShell.autoindent = True\nc.InteractiveShell.colors = 'Linux'\nc.InteractiveShell.confirm_exit = False\nc.AliasManager.user_aliases = [\n ('ll', 'ls -l'),\n ('lt', 'ls -ltr'),\n]\n\n#------------------------------------------------------------------------------\n# InteractiveShellApp configuration\n#------------------------------------------------------------------------------\n\n# A Mixin for applications that start InteractiveShell instances.\n#\n# Provides configurables for loading extensions and executing files as part of\n# configuring a Shell environment.\n#\n# Provides init_extensions() and init_code() methods, to be called after\n# init_shell(), which must be implemented by subclasses.\n\n# Execute the given command string.\n# c.InteractiveShellApp.code_to_run = ''\n\n# lines of code to run at IPython startup.\n# c.InteractiveShellApp.exec_lines = []\n\n# If true, an 'import *' is done from numpy and pylab, when using pylab\n# c.InteractiveShellApp.pylab_import_all = True\n\n# A list of dotted module names of IPython extensions to load.\n# c.InteractiveShellApp.extensions = []\n\n# dotted module name of an IPython extension to load.\n# c.InteractiveShellApp.extra_extension = ''\n\n# List of files to run at IPython startup.\n# c.InteractiveShellApp.exec_files = []\n\n# A file to be run\n# c.InteractiveShellApp.file_to_run = ''\n\n#------------------------------------------------------------------------------\n# TerminalIPythonApp configuration\n#------------------------------------------------------------------------------\n\n# TerminalIPythonApp will inherit config from: BaseIPythonApplication,\n# Application, InteractiveShellApp\n\n# Execute the given command string.\n# c.TerminalIPythonApp.code_to_run = ''\n\n# The IPython profile to use.\n# c.TerminalIPythonApp.profile = u'default'\n\n# Set the log level by value or name.\n# c.TerminalIPythonApp.log_level = 30\n\n# lines of code to run at IPython startup.\n# c.TerminalIPythonApp.exec_lines = []\n\n# Enable GUI event loop integration ('qt', 'wx', 'gtk', 'glut', 'pyglet').\n# c.TerminalIPythonApp.gui = None\n\n# Pre-load matplotlib and numpy for interactive use, selecting a particular\n# matplotlib backend and loop integration.\n# c.TerminalIPythonApp.pylab = None\n\n# Suppress warning messages about legacy config files\n# c.TerminalIPythonApp.ignore_old_config = False\n\n# Create a massive crash report when IPython enconters what may be an internal\n# error.  The default is to append a short message to the usual traceback\n# c.TerminalIPythonApp.verbose_crash = False\n\n# If a command or file is given via the command-line, e.g. 'ipython foo.py\n# c.TerminalIPythonApp.force_interact = False\n\n# If true, an 'import *' is done from numpy and pylab, when using pylab\n# c.TerminalIPythonApp.pylab_import_all = True\n\n# The name of the IPython directory. This directory is used for logging\n# configuration (through profiles), history storage, etc. The default is usually\n# $HOME/.ipython. This options can also be specified through the environment\n# variable IPYTHON_DIR.\n# c.TerminalIPythonApp.ipython_dir = u'/home/ubuntu/.ipython'\n\n# Whether to display a banner upon starting IPython.\n# c.TerminalIPythonApp.display_banner = True\n\n# Start IPython quickly by skipping the loading of config files.\n# c.TerminalIPythonApp.quick = False\n\n# A list of dotted module names of IPython extensions to load.\n# c.TerminalIPythonApp.extensions = []\n\n# Whether to install the default config files into the profile dir. If a new\n# profile is being created, and IPython contains config files for that profile,\n# then they will be staged into the new directory.  Otherwise, default config\n# files will be automatically generated.\n# c.TerminalIPythonApp.copy_config_files = False\n\n# dotted module name of an IPython extension to load.\n# c.TerminalIPythonApp.extra_extension = ''\n\n# List of files to run at IPython startup.\n# c.TerminalIPythonApp.exec_files = []\n\n# Whether to overwrite existing config files when copying\n# c.TerminalIPythonApp.overwrite = False\n\n# A file to be run\n# c.TerminalIPythonApp.file_to_run = ''\n\n#------------------------------------------------------------------------------\n# TerminalInteractiveShell configuration\n#------------------------------------------------------------------------------\n\n# TerminalInteractiveShell will inherit config from: InteractiveShell\n\n# auto editing of files with syntax errors.\n# c.TerminalInteractiveShell.autoedit_syntax = False\n\n# Use colors for displaying information about objects. Because this information\n# is passed through a pager (like 'less'), and some pagers get confused with\n# color codes, this capability can be turned off.\n# c.TerminalInteractiveShell.color_info = True\n\n#\n# c.TerminalInteractiveShell.history_length = 10000\n\n# Don't call post-execute functions that have failed in the past.\n# c.TerminalInteractiveShell.disable_failing_post_execute = False\n\n# Show rewritten input, e.g. for autocall.\n# c.TerminalInteractiveShell.show_rewritten_input = True\n\n# Set the color scheme (NoColor, Linux, or LightBG).\n# c.TerminalInteractiveShell.colors = 'LightBG'\n\n# Autoindent IPython code entered interactively.\n# c.TerminalInteractiveShell.autoindent = True\n\n#\n# c.TerminalInteractiveShell.separate_in = '\\n'\n\n# Deprecated, use PromptManager.in2_template\n# c.TerminalInteractiveShell.prompt_in2 = '   .\\\\D.: '\n\n#\n# c.TerminalInteractiveShell.separate_out = ''\n\n# Deprecated, use PromptManager.in_template\n# c.TerminalInteractiveShell.prompt_in1 = 'In [\\\\#]: '\n\n# Enable deep (recursive) reloading by default. IPython can use the deep_reload\n# module which reloads changes in modules recursively (it replaces the reload()\n# function, so you don't need to change anything to use it). deep_reload()\n# forces a full reload of modules whose code may have changed, which the default\n# reload() function does not.  When deep_reload is off, IPython will use the\n# normal reload(), but deep_reload will still be available as dreload().\n# c.TerminalInteractiveShell.deep_reload = False\n\n# Make IPython automatically call any callable object even if you didn't type\n# explicit parentheses. For example, 'str 43' becomes 'str(43)' automatically.\n# The value can be '0' to disable the feature, '1' for 'smart' autocall, where\n# it is not applied if there are no more arguments on the line, and '2' for\n# 'full' autocall, where all callable objects are automatically called (even if\n# no arguments are present).\n# c.TerminalInteractiveShell.autocall = 0\n\n# Number of lines of your screen, used to control printing of very long strings.\n# Strings longer than this number of lines will be sent through a pager instead\n# of directly printed.  The default value for this is 0, which means IPython\n# will auto-detect your screen size every time it needs to print certain\n# potentially long strings (this doesn't change the behavior of the 'print'\n# keyword, it's only triggered internally). If for some reason this isn't\n# working well (it needs curses support), specify it yourself. Otherwise don't\n# change the default.\n# c.TerminalInteractiveShell.screen_length = 0\n\n# Set the editor used by IPython (default to $EDITOR/vi/notepad).\n# c.TerminalInteractiveShell.editor = 'vi'\n\n# Deprecated, use PromptManager.justify\n# c.TerminalInteractiveShell.prompts_pad_left = True\n\n# The part of the banner to be printed before the profile\n# c.TerminalInteractiveShell.banner1 = 'Python 2.7.1 (r271:86832, Jun 25 2011, 05:09:01) \\nType \"copyright\", \"credits\" or \"license\" for more information.\\n\\nIPython 0.12 -- An enhanced Interactive Python.\\n?         -> Introduction and overview of IPython\\'s features.\\n%quickref -> Quick reference.\\nhelp      -> Python\\'s own help system.\\nobject?   -> Details about \\'object\\', use \\'object??\\' for extra details.\\n'\n\n#\n# c.TerminalInteractiveShell.readline_parse_and_bind = ['tab: complete', '\"\\\\C-l\": clear-screen', 'set show-all-if-ambiguous on', '\"\\\\C-o\": tab-insert', '\"\\\\C-r\": reverse-search-history', '\"\\\\C-s\": forward-search-history', '\"\\\\C-p\": history-search-backward', '\"\\\\C-n\": history-search-forward', '\"\\\\e[A\": history-search-backward', '\"\\\\e[B\": history-search-forward', '\"\\\\C-k\": kill-line', '\"\\\\C-u\": unix-line-discard']\n\n# The part of the banner to be printed after the profile\n# c.TerminalInteractiveShell.banner2 = ''\n\n#\n# c.TerminalInteractiveShell.separate_out2 = ''\n\n#\n# c.TerminalInteractiveShell.wildcards_case_sensitive = True\n\n#\n# c.TerminalInteractiveShell.debug = False\n\n# Set to confirm when you try to exit IPython with an EOF (Control-D in Unix,\n# Control-Z/Enter in Windows). By typing 'exit' or 'quit', you can force a\n# direct exit without any confirmation.\n# c.TerminalInteractiveShell.confirm_exit = True\n\n#\n# c.TerminalInteractiveShell.ipython_dir = ''\n\n#\n# c.TerminalInteractiveShell.readline_remove_delims = '-/~'\n\n# Start logging to the default log file.\n# c.TerminalInteractiveShell.logstart = False\n\n# The name of the logfile to use.\n# c.TerminalInteractiveShell.logfile = ''\n\n# The shell program to be used for paging.\n# c.TerminalInteractiveShell.pager = 'less'\n\n# Enable magic commands to be called without the leading %.\n# c.TerminalInteractiveShell.automagic = True\n\n# Save multi-line entries as one entry in readline history\n# c.TerminalInteractiveShell.multiline_history = True\n\n#\n# c.TerminalInteractiveShell.readline_use = True\n\n# Start logging to the given file in append mode.\n# c.TerminalInteractiveShell.logappend = ''\n\n#\n# c.TerminalInteractiveShell.xmode = 'Context'\n\n#\n# c.TerminalInteractiveShell.quiet = False\n\n# Enable auto setting the terminal title.\n# c.TerminalInteractiveShell.term_title = False\n\n#\n# c.TerminalInteractiveShell.object_info_string_level = 0\n\n# Deprecated, use PromptManager.out_template\n# c.TerminalInteractiveShell.prompt_out = 'Out[\\\\#]: '\n\n# Set the size of the output cache.  The default is 1000, you can change it\n# permanently in your config file.  Setting it to 0 completely disables the\n# caching system, and the minimum value accepted is 20 (if you provide a value\n# less than 20, it is reset to 0 and a warning is issued).  This limit is\n# defined because otherwise you'll spend more time re-flushing a too small cache\n# than working\n# c.TerminalInteractiveShell.cache_size = 1000\n\n# Automatically call the pdb debugger after every exception.\n# c.TerminalInteractiveShell.pdb = False\n\n#------------------------------------------------------------------------------\n# PromptManager configuration\n#------------------------------------------------------------------------------\n\n# This is the primary interface for producing IPython's prompts.\n\n# Output prompt. '\\#' will be transformed to the prompt number\n# c.PromptManager.out_template = 'Out[\\\\#]: '\n\n# Continuation prompt.\n# c.PromptManager.in2_template = '   .\\\\D.: '\n\n# If True (default), each prompt will be right-aligned with the preceding one.\n# c.PromptManager.justify = True\n\n# Input prompt.  '\\#' will be transformed to the prompt number\n# c.PromptManager.in_template = 'In [\\\\#]: '\n\n#\n# c.PromptManager.color_scheme = 'Linux'\n\n#------------------------------------------------------------------------------\n# ProfileDir configuration\n#------------------------------------------------------------------------------\n\n# An object to manage the profile directory and its resources.\n#\n# The profile directory is used by all IPython applications, to manage\n# configuration, logging and security.\n#\n# This object knows how to find, create and manage these directories. This\n# should be used by any code that wants to handle profiles.\n\n# Set the profile location directly. This overrides the logic used by the\n# `profile` option.\n# c.ProfileDir.location = u''\n\n#------------------------------------------------------------------------------\n# PlainTextFormatter configuration\n#------------------------------------------------------------------------------\n\n# The default pretty-printer.\n#\n# This uses :mod:`IPython.external.pretty` to compute the format data of the\n# object. If the object cannot be pretty printed, :func:`repr` is used. See the\n# documentation of :mod:`IPython.external.pretty` for details on how to write\n# pretty printers.  Here is a simple example::\n#\n#     def dtype_pprinter(obj, p, cycle):\n#         if cycle:\n#             return p.text('dtype(...)')\n#         if hasattr(obj, 'fields'):\n#             if obj.fields is None:\n#                 p.text(repr(obj))\n#             else:\n#                 p.begin_group(7, 'dtype([')\n#                 for i, field in enumerate(obj.descr):\n#                     if i > 0:\n#                         p.text(',')\n#                         p.breakable()\n#                     p.pretty(field)\n#                 p.end_group(7, '])')\n\n# PlainTextFormatter will inherit config from: BaseFormatter\n\n#\n# c.PlainTextFormatter.type_printers = {}\n\n#\n# c.PlainTextFormatter.newline = '\\n'\n\n#\n# c.PlainTextFormatter.float_precision = ''\n\n#\n# c.PlainTextFormatter.verbose = False\n\n#\n# c.PlainTextFormatter.deferred_printers = {}\n\n#\n# c.PlainTextFormatter.pprint = True\n\n#\n# c.PlainTextFormatter.max_width = 79\n\n#\n# c.PlainTextFormatter.singleton_printers = {}\n\n#------------------------------------------------------------------------------\n# IPCompleter configuration\n#------------------------------------------------------------------------------\n\n# Extension of the completer class with IPython-specific features\n\n# IPCompleter will inherit config from: Completer\n\n# Instruct the completer to omit private method names\n#\n# Specifically, when completing on ``object.<tab>``.\n#\n# When 2 [default]: all names that start with '_' will be excluded.\n#\n# When 1: all 'magic' names (``__foo__``) will be excluded.\n#\n# When 0: nothing will be excluded.\n# c.IPCompleter.omit__names = 2\n\n# Whether to merge completion results into a single list\n#\n# If False, only the completion results from the first non-empty completer will\n# be returned.\n# c.IPCompleter.merge_completions = True\n\n# Activate greedy completion\n#\n# This will enable completion on elements of lists, results of function calls,\n# etc., but can be unsafe because the code is actually evaluated on TAB.\n# c.IPCompleter.greedy = False\n"
  },
  {
    "path": "installed_files/jwmrc.xml",
    "content": "<?xml version=\"1.0\"?>\n\n<JWM>\n\n   <!-- The root menu, if this is undefined you will not get a menu. -->\n   <!-- Additional RootMenu attributes: onroot, labeled, label -->\n   <RootMenu height=\"15\" onroot=\"12\">\n      <Program icon=\"terminal.png\" label=\"Terminal\">x-terminal-emulator</Program>\n      <Program icon=\"firefox.png\" label=\"Www Browser\">x-www-broser</Program>\n\n<!-- #DEBIAN\n      <Menu icon=\"folder.png\" label=\"Applications\">\n         <Program icon=\"editor.png\" label=\"Dia\">dia</Program>\n         <Program icon=\"www.png\" label=\"Firefox\">firefox</Program>\n         <Program icon=\"gimp.png\" label=\"Gimp\">gimp</Program>\n         <Program icon=\"mail.png\" label=\"Mutt\">\n            xterm -e mutt\n         </Program>\n         <Program icon=\"word-processor.png\" label=\"Open Office\">  \n            ooffice\n         </Program>\n         <Program icon=\"chat.png\" label=\"Pidgin\">pidgin</Program>\n         <Program icon=\"music.png\" label=\"Rhythmbox\">rhythmbox</Program>\n         <Program icon=\"development.png\" label=\"Xilinx ISE\">\n            /usr/local/xilinx/bin/ise\n         </Program>\n         <Program icon=\"video.png\" label=\"Xine\">gxine</Program>\n      </Menu>\n      <Menu icon=\"folder.png\" label=\"Utilities\">\n         <Program icon=\"calculator.png\" label=\"Calculator\">xcalc</Program>\n         <Program icon=\"font.png\" label=\"Fonts\">xfontsel</Program>\n         <Program icon=\"viewer.png\" label=\"Magnify\">xmag</Program>\n         <Program icon=\"programs.png\" label=\"Synaptic\">\n            gksudo synaptic\n         </Program>\n         <Program icon=\"window.png\" label=\"Window Properties\">\n            xprop | xmessage -file -\n         </Program>\n      </Menu>\n-->\n      <Separator/>\n      <Program icon=\"lock.png\" label=\"Lock\">\n         xscreensaver-command -activate\n      </Program>\n      <Separator/>\n      <Restart label=\"Restart\" icon=\"restart.png\"/>\n      <Exit label=\"Exit\" confirm=\"true\" icon=\"quit.png\"/>\n   </RootMenu>\n\n   <Group>\n      <Class>Pidgin</Class>\n      <Option>sticky</Option>\n   </Group>\n\n   <Group>\n      <Name>gkrellm</Name>\n      <Option>nolist</Option>\n      <Option>sticky</Option>\n   </Group>\n\n   <Group>\n      <Name>xterm</Name>\n      <Option>vmax</Option>\n      <Option>icon:terminal.png</Option>\n   </Group>\n\n   <!-- Additional tray attributes: autohide, width, border, layer, layout -->\n   <Tray  x=\"0\" y=\"-1\" height=\"32\" autohide=\"false\">\n\n      <!-- Additional TrayButton attribute: label -->\n      <TrayButton label=\"JWM\">root:1</TrayButton>\n\n      <TrayButton label=\"_\">showdesktop</TrayButton>\n\n      <!-- Additional Pager attribute: labeled -->\n      <Pager labeled=\"true\"/>\n\n      <TaskList maxwidth=\"256\"/>\n\n      <Dock/>\n\n      <!-- Additional Swallow attribute: height -->\n      <Swallow name=\"xload\" width=\"64\">\n         xload -nolabel -bg black -fg red -hl white\n      </Swallow>\n\n      <Clock format=\"%H:%M\">xclock</Clock>\n\n   </Tray>\n\n   <!-- Visual Styles -->\n\n   <WindowStyle>\n\n      <Font>-*-fixed-*-r-*-*-10-*-*-*-*-*-*-*</Font>\n      <Width>4</Width>\n      <Height>20</Height>\n\n      <Active>\n         <Text>white</Text>\n<!-- #DEBIAN\n         <Title>#70849d:#2e3a67</Title>\n-->\n         <Title>gray30:gray60</Title>\n         <Outline>black</Outline>\n         <Opacity>1.0</Opacity>\n      </Active>\n\n      <Inactive>\n         <Text>#aaaaaa</Text>\n         <Title>#808488:#303438</Title>\n         <Outline>black</Outline>\n         <Opacity>0.5:0.9:0.1</Opacity>\n      </Inactive>\n\n   </WindowStyle>\n\n   <TaskListStyle>\n      <Font>-*-fixed-*-r-*-*-10-*-*-*-*-*-*-*</Font>\n      <ActiveForeground>black</ActiveForeground>\n      <ActiveBackground>gray80:gray90</ActiveBackground>\n      <Foreground>black</Foreground>\n      <Background>gray90:gray80</Background>\n   </TaskListStyle>\n\n   <!-- Additional TrayStyle attribute: insert -->\n   <TrayStyle>\n      <Font>-*-fixed-*-r-*-*-10-*-*-*-*-*-*-*</Font>\n      <Background>gray90</Background>\n      <Foreground>black</Foreground>\n      <Opacity>0.75</Opacity>\n   </TrayStyle>\n\n   <PagerStyle>\n      <Outline>black</Outline>\n      <Foreground>gray90</Foreground>\n      <Background>#808488</Background>\n      <ActiveForeground>#70849d</ActiveForeground>\n      <ActiveBackground>#2e3a67</ActiveBackground>\n   </PagerStyle>\n\n   <MenuStyle>\n      <Font>-*-fixed-*-r-*-*-10-*-*-*-*-*-*-*</Font>\n      <Foreground>black</Foreground>\n      <Background>gray90</Background>\n      <ActiveForeground>white</ActiveForeground>\n      <ActiveBackground>#70849d:#2e3a67</ActiveBackground>\n      <Opacity>0.85</Opacity>\n   </MenuStyle>\n\n   <PopupStyle>\n      <Font>-*-fixed-*-r-*-*-10-*-*-*-*-*-*-*</Font>\n      <Outline>black</Outline>\n      <Foreground>black</Foreground>\n      <Background>yellow</Background>\n   </PopupStyle>\n\n   <IconPath>\n      /usr/share/icons/wm-icons/32x32-gant\n   </IconPath>\n\n   <!-- Virtual Desktops -->\n   <!-- Desktop tags can be contained within Desktops for desktop names. -->\n   <Desktops width=\"4\" height=\"2\">\n      <!-- Default background. Note that a Background tag can be\n           contained within a Desktop tag to give a specific background\n           for that desktop.\n       -->\n      <Background type=\"image\">/export0/images/formulae.jpg</Background>\n   </Desktops>\n\n   <!-- Double click speed (in milliseconds) -->\n   <DoubleClickSpeed>400</DoubleClickSpeed>\n\n   <!-- Double click delta (in pixels) -->\n   <DoubleClickDelta>2</DoubleClickDelta>\n\n   <!-- The focus model (sloppy or click) -->\n   <FocusModel>sloppy</FocusModel>\n\n   <!-- The snap mode (none, screen, or border) -->\n   <SnapMode distance=\"10\">border</SnapMode>\n\n   <!-- The move mode (outline or opaque) -->\n   <MoveMode>opaque</MoveMode>\n\n   <!-- The resize mode (outline or opaque) -->\n   <ResizeMode>opaque</ResizeMode>\n\n   <StartupCommand>gkrellm</StartupCommand>\n\n   <!-- Key bindings -->\n   <Key key=\"Up\">up</Key>\n   <Key key=\"Down\">down</Key>\n   <Key key=\"Right\">right</Key>\n   <Key key=\"Left\">left</Key>\n   <Key key=\"h\">left</Key>\n   <Key key=\"j\">down</Key>\n   <Key key=\"k\">up</Key>\n   <Key key=\"l\">right</Key>\n   <Key key=\"Return\">select</Key>\n   <Key key=\"Escape\">escape</Key>\n\n   <Key mask=\"A\" key=\"Tab\">next</Key>\n   <Key mask=\"A\" key=\"F4\">close</Key>\n   <Key mask=\"A\" key=\"#\">desktop#</Key>\n   <Key mask=\"A\" key=\"F1\">root:1</Key>\n   <Key mask=\"A\" key=\"F2\">window</Key>\n   <Key mask=\"A\" key=\"F10\">maximize</Key>\n   <Key mask=\"A\" key=\"Right\">rdesktop</Key>\n   <Key mask=\"A\" key=\"Left\">ldesktop</Key>\n   <Key mask=\"A\" key=\"Up\">udesktop</Key>\n   <Key mask=\"A\" key=\"Down\">ddesktop</Key>\n\n</JWM>\n\n"
  },
  {
    "path": "installed_files/nginx.conf.template",
    "content": "worker_processes  2;\n\nevents {\n    worker_connections  1024;\n}\n\nhttp {\n    include       mime.types;\n    default_type  application/octet-stream;\n    sendfile        on;\n    keepalive_timeout  65;\n\n    gzip  on;\n    gzip_http_version 1.1;\n    gzip_vary on;\n    gzip_comp_level 4;\n    gzip_proxied any;\n    gzip_types text/plain text/css application/x-javascript text/xml application/xml text/javascript application/json;\n    gzip_buffers 16 8k;\n    gzip_disable \"MSIE [1-6].(?!.*SV1)\";\n\n    upstream galaxy_app {\n        server localhost:8080;\n    }\n\n    upstream cm_app {\n        server localhost:42284;\n    }\n\n    upstream galaxy_reports_app {\n        server localhost:9001;\n    }\n\n    server {\n        listen       80;\n        client_max_body_size 2048m;\n        server_name  localhost;\n                proxy_read_timeout 600;\n\n        location /cloud {\n            proxy_pass  http://cm_app;\n            proxy_set_header   X-Forwarded-Host $$host;\n            proxy_set_header   X-Forwarded-For  $$proxy_add_x_forwarded_for;\n            error_page   502    /errdoc/cm_502.html;\n        }\n\n        location /cloud/static {\n            alias /mnt/cm/static;\n            expires 24h;\n        }\n\n        location /cloud/static/style {\n            alias /mnt/cm/static/style;\n            expires 24h;\n        }\n\n        location /cloud/static/scripts {\n            alias /mnt/cm/static/scripts;\n            expires 24h;\n        }\n\n        location /reports {\n            rewrite ^/reports/(.*) /$$1 break;\n            proxy_pass http://galaxy_reports_app;\n            proxy_set_header   X-Forwarded-Host $$host;\n            proxy_set_header   X-Forwarded-For  $$proxy_add_x_forwarded_for;\n        }\n\n        location / {\n            proxy_pass  http://galaxy_app;\n            proxy_set_header   X-Forwarded-Host $$host;\n            proxy_set_header   X-Forwarded-For  $$proxy_add_x_forwarded_for;\n        }\n\n        location /static {\n            alias $galaxy_home/static;\n            expires 24h;\n        }\n\n        location /static/style {\n            alias $galaxy_home/static/june_2007_style/blue;\n            expires 24h;\n        }\n\n        location /static/scripts {\n            alias $galaxy_home/static/scripts/packed;\n            expires 24h;\n        }\n\n        location /robots.txt {\n            alias $galaxy_home/static/robots.txt;\n        }\n\n        location /favicon.ico {\n            alias $galaxy_home/static/favicon.ico;\n        }\n\n        location /_x_accel_redirect/ {\n            internal;\n            alias /;\n        }\n\n        location /_upload {\n            upload_store /mnt/galaxyData/upload_store;\n            upload_pass_form_field \"\";\n            upload_set_form_field \"__$${upload_field_name}__is_composite\" \"true\";\n            upload_set_form_field \"__$${upload_field_name}__keys\" \"name path\";\n            upload_set_form_field \"$${upload_field_name}_name\" \"$$upload_file_name\";\n            upload_set_form_field \"$${upload_field_name}_path\" \"$$upload_tmp_path\";\n            upload_pass_args on;\n            upload_pass /_upload_done;\n        }\n\n        location /_upload_done {\n            set $$dst /tool_runner/index;\n            if ($$args ~ nginx_redir=([^&]+)) {\n                set $$dst $$1;\n            }\n            rewrite \"\" $$dst;\n        }\n\n        error_page   502    /errdoc/502.html;\n        location /errdoc {\n            root   html;\n        }\n\n    }\n\n}\n"
  },
  {
    "path": "installed_files/nginx_init",
    "content": "#! /bin/sh\n\n### BEGIN INIT INFO\n# Provides:          nginx\n# Required-Start:    $local_fs $remote_fs $network $syslog\n# Required-Stop:     $local_fs $remote_fs $network $syslog\n# Default-Start:     2 3 4 5\n# Default-Stop:      0 1 6\n# Short-Description: starts the nginx web server\n# Description:       starts nginx using start-stop-daemon\n### END INIT INFO\n\nPATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/galaxy/sbin/\nDAEMON=/opt/galaxy/sbin/nginx\nNAME=nginx\nDESC=nginx\n\ntest -x $DAEMON || exit 0\n\n# Include nginx defaults if available\nif [ -f /etc/default/nginx ] ; then\n   . /etc/default/nginx\nfi\n\nset -e\n\n. /lib/lsb/init-functions\n\ntest_nginx_config() {\n  if nginx -t $DAEMON_OPTS\n  then\n    return 0\n  else\n    return $?\n  fi\n}\n\ncase \"$1\" in\n  start)\n  echo -n \"Starting $DESC: \"\n        test_nginx_config\n        start-stop-daemon --start --quiet --pidfile /var/run/$NAME.pid \\\n                          --exec $DAEMON -- $DAEMON_OPTS || true\n                          echo \"$NAME.\"\n                          ;;\n  stop)\n  echo -n \"Stopping $DESC: \"\n  start-stop-daemon --stop --quiet --pidfile /var/run/$NAME.pid \\\n                    --exec $DAEMON || true\n                    echo \"$NAME.\"\n                    ;;\n  restart|force-reload)\n  echo -n \"Restarting $DESC: \"\n  start-stop-daemon --stop --quiet --pidfile \\\n                    /var/run/$NAME.pid --exec $DAEMON || true\n                    sleep 1\n        test_nginx_config\n        start-stop-daemon --start --quiet --pidfile \\\n                          /var/run/$NAME.pid --exec $DAEMON -- $DAEMON_OPTS || true\n                          echo \"$NAME.\"\n                          ;;\n  reload)\n        echo -n \"Reloading $DESC configuration: \"\n        test_nginx_config\n        start-stop-daemon --stop --signal HUP --quiet --pidfile /var/run/$NAME.pid \\\n            --exec $DAEMON || true\n        echo \"$NAME.\"\n        ;;\n  configtest)\n        echo -n \"Testing $DESC configuration: \"\n        if test_nginx_config\n        then\n          echo \"$NAME.\"\n        else\n          exit $?\n        fi\n        ;;\n  status)\n  status_of_proc -p /var/run/$NAME.pid \"$DAEMON\" nginx && exit 0 || exit $?\n  ;;\n  *)\n  echo \"Usage: $NAME {start|stop|restart|reload|force-reload|status|configtest}\" >&2\n  exit 1\n  ;;\nesac\n\nexit 0\n"
  },
  {
    "path": "installed_files/novnc_default.template",
    "content": "NOVNC_USER=$vnc_user\nNOVNC_HOME=$novnc_install_dir\nPORT=\"0.0.0.0:6080\"\nVNC_DEST=\"localhost:590$vnc_display\"\n"
  },
  {
    "path": "installed_files/novnc_init",
    "content": "#! /bin/sh\n### BEGIN INIT INFO\n# Provides:          novnc\n# Required-Start:    $remote_fs $syslog\n# Required-Stop:     $remote_fs $syslog\n# Default-Start:     2 3 4 5\n# Default-Stop:      0 1 6\n# Short-Description: novnc init script.\n# Description:       novnc init script.\n### END INIT INFO\n\n# Author: John Chilton <jmchilton@gmail.com>\n#\n# TODO: This script doesn't work to stop/kill novnc,\n# this should be fixed.\n\nPATH=/sbin:/usr/sbin:/bin:/usr/bin\nDESC=\"novnc proxy web server\"\nNAME=novnc\n\n# Read configuration variable file if it is present\n[ -r /etc/default/$NAME ] && . /etc/default/$NAME\n\nDAEMON=/usr/bin/env\nDAEMON_ARGS=\"python $NOVNC_HOME/utils/wsproxy.py --web $NOVNC_HOME $PORT $VNC_DEST\"\nPIDFILE=/var/run/$NAME.pid\nSCRIPTNAME=/etc/init.d/$NAME\n\n# Exit if the package is not installed\n[ -x \"$DAEMON\" ] || exit 0\n\n# Load the VERBOSE setting and other rcS variables\n. /lib/init/vars.sh\n\n# Define LSB log_* functions.\n# Depend on lsb-base (>= 3.2-14) to ensure that this file is present\n# and status_of_proc is working.\n. /lib/lsb/init-functions\n\n#\n# Function that starts the daemon/service\n#\ndo_start()\n{\n\t# Return\n\t#   0 if daemon has been started\n\t#   1 if daemon was already running\n\t#   2 if daemon could not be started\n\tstart-stop-daemon --start --quiet --pidfile $PIDFILE --user $NOVNC_USER --chuid $NOVNC_USER --exec $DAEMON --test > /dev/null \\\n\t\t|| return 1\n\tstart-stop-daemon --start --quiet --make-pidfile --pidfile $PIDFILE --user $NOVNC_USER --chuid $NOVNC_USER --background --exec $DAEMON -- \\\n\t\t$DAEMON_ARGS \\\n\t\t|| return 2\n\t# Add code here, if necessary, that waits for the process to be ready\n\t# to handle requests from services started subsequently which depend\n\t# on this one.  As a last resort, sleep for some time.\n}\n\n#\n# Function that stops the daemon/service\n#\ndo_stop()\n{\n\t# Return\n\t#   0 if daemon has been stopped\n\t#   1 if daemon was already stopped\n\t#   2 if daemon could not be stopped\n\t#   other if a failure occurred\n\tstart-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME\n\tRETVAL=\"$?\"\n\t[ \"$RETVAL\" = 2 ] && return 2\n\t# Wait for children to finish too if this is a daemon that forks\n\t# and if the daemon is only ever run from this initscript.\n\n\n\t# If the above conditions are not satisfied then add some other code\n\t# that waits for the process to drop all resources that could be\n\t# needed by services started subsequently.  A last resort is to\n\t# sleep for some time.\n\tstart-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON\n\t[ \"$?\" = 2 ] && return 2\n\t# Many daemons don't delete their pidfiles when they exit.\n\trm -f $PIDFILE\n\treturn \"$RETVAL\"\n}\n\n#\n# Function that sends a SIGHUP to the daemon/service\n#\ndo_reload() {\n\t#\n\t# If the daemon can reload its configuration without\n\t# restarting (for example, when it is sent a SIGHUP),\n\t# then implement that here.\n\t#\n\tstart-stop-daemon --stop --signal 1 --quiet --pidfile $PIDFILE --name $NAME\n\treturn 0\n}\n\ncase \"$1\" in\n  start)\n\t[ \"$VERBOSE\" != no ] && log_daemon_msg \"Starting $DESC\" \"$NAME\"\n\tdo_start\n\tcase \"$?\" in\n\t\t0|1) [ \"$VERBOSE\" != no ] && log_end_msg 0 ;;\n\t\t2) [ \"$VERBOSE\" != no ] && log_end_msg 1 ;;\n\tesac\n\t;;\n  stop)\n\t[ \"$VERBOSE\" != no ] && log_daemon_msg \"Stopping $DESC\" \"$NAME\"\n\tdo_stop\n\tcase \"$?\" in\n\t\t0|1) [ \"$VERBOSE\" != no ] && log_end_msg 0 ;;\n\t\t2) [ \"$VERBOSE\" != no ] && log_end_msg 1 ;;\n\tesac\n\t;;\n  status)\n       status_of_proc \"$DAEMON\" \"$NAME\" && exit 0 || exit $?\n       ;;\n  #reload|force-reload)\n\t#\n\t# If do_reload() is not implemented then leave this commented out\n\t# and leave 'force-reload' as an alias for 'restart'.\n\t#\n\t#log_daemon_msg \"Reloading $DESC\" \"$NAME\"\n\t#do_reload\n\t#log_end_msg $?\n\t#;;\n  restart|force-reload)\n\t#\n\t# If the \"reload\" option is implemented then remove the\n\t# 'force-reload' alias\n\t#\n\tlog_daemon_msg \"Restarting $DESC\" \"$NAME\"\n\tdo_stop\n\tcase \"$?\" in\n\t  0|1)\n\t\tdo_start\n\t\tcase \"$?\" in\n\t\t\t0) log_end_msg 0 ;;\n\t\t\t1) log_end_msg 1 ;; # Old process is still running\n\t\t\t*) log_end_msg 1 ;; # Failed to start\n\t\tesac\n\t\t;;\n\t  *)\n\t  \t# Failed to stop\n\t\tlog_end_msg 1\n\t\t;;\n\tesac\n\t;;\n  *)\n\t#echo \"Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}\" >&2\n\techo \"Usage: $SCRIPTNAME {start|stop|status|restart|force-reload}\" >&2\n\texit 3\n\t;;\nesac\n\n:\n"
  },
  {
    "path": "installed_files/pg_ctl",
    "content": "#!/bin/bash\n#\n# This script is required because of CloudMan - CloudMan's path resolution\n# code would result in using the latest version of PostgreSQL installed via\n# a package manager. However, if a user switches images, the database data\n# may not match the version of PostgreSQL being invoked so this script\n# is used to differentiate which PostgreSQL version is actually invoked.\n# This had to be done here because it was the only variable for old\n# clusters being invoked with a new image. Also note that this script\n# is image dependent (because PostgreSQL v9.1 will be the default version).\n\n# Recompose args so pg_ctl will work\nargs=\"\"\n\n# If an argument contains whitespace, add double quotes around it\nwhitespace=\"[[:space:]]\"\nfor arg in \"$@\"; do\n    if [[ $arg =~ $whitespace ]]; then\n        arg=\\\"$arg\\\"\n    fi\n    args=\"$args $arg\"\ndone\n\n# If the -D option is specified (path to data dir), extract PG_VERSION from the data dir\nwhile getopts \":D:\" opt; do\n  case $opt in\n    D)\n      # Get current Postgres DB version\n      PG_VERSION=`cat \"$OPTARG\"/PG_VERSION`\n      ;;\n  esac\ndone\n\nif [ \"$PG_VERSION\" == \"8.4\" ]; then\n    CMD=\"/usr/lib/postgresql/8.4/bin/pg_ctl $args\"\nelse\n    CMD=\"/usr/lib/postgresql/9.1/bin/pg_ctl_orig $args\"\nfi\n# Must eval here; otherwise quotes in the args don’t get passed through properly\neval $CMD\n"
  },
  {
    "path": "installed_files/proftpd.conf.template",
    "content": "ServerName                      \"Galaxy CloudMan FTP\"\nServerType                      standalone\nDefaultServer                   on\n\n# Port 21 is the standard FTP port.\nPort                            21\n\n# Don't use IPv6 support by default.\nUseIPv6                         on\n\n# Umask 022 is a good standard umask to prevent new dirs and files\n# from being group and world writable.\nUmask                           022\n\nSyslogFacility                  DAEMON\nSyslogLevel                     debug\nPidFile                         /var/run/proftpd.pid\n\n# To prevent DoS attacks, set the maximum number of child processes\n# to 30.  If you need to allow more than 30 concurrent connections\n# at once, simply increase this value.  Note that this ONLY works\n# in standalone mode, in inetd mode you should use an inetd server\n# that allows you to limit maximum number of processes per service\n# (such as xinetd).\nMaxInstances                    30\n\n# Set the user and group under which the server will run.\nUser                            nobody\nGroup                           nogroup\n\nPassivePorts                    30000 30100\n\nDisplayConnect REPLACE_THIS_WITH_CUSTOM_INSTALL_DIR/etc/welcome_msg.txt\n\n# To cause every FTP user to be \"jailed\" (chrooted) into their home\n# directory, uncomment this line.\nDefaultRoot                     ~\n\n# Normally, we want files to be overwriteable.\nAllowOverwrite                  on\n\nAllowStoreRestart               on\n\n# Bar use of SITE CHMOD by default\n<Limit SITE_CHMOD>\n  DenyAll\n</Limit>\n\n<Limit RETR>\n  DenyAll\n</Limit>\n\n<IfModule mod_facts.c>\n  FactsAdvertise off\n</IfModule>\n\n<Limit MFMT MFCT MFF SITE_UTIME>\n  DenyAll\n</Limit>\n\n# AuthPAM off\n\nCreateHome on dirmode 700\n\nSQLPasswordEngine on\nSQLPasswordEncoding hex\n\nSQLEngine               on\nSQLLogFile              /var/log/proftpd-sql.log\nSQLBackend              postgres\nSQLConnectInfo          galaxy@localhost:$postgres_port galaxyftp $galaxy_ftp_user_password\nSQLAuthTypes            SHA1\nSQLAuthenticate         users\nSQLDefaultUID           $galaxy_uid\nSQLDefaultGID           $galaxy_uid\nSQLDefaultHomedir       $install_dir/var\nSQLUserInfo             custom:/LookupGalaxyUser\nSQLNamedQuery           LookupGalaxyUser SELECT \"email,password,'$galaxy_uid','$galaxy_uid','$galaxy_fs/tmp/ftp/%U','/bin/bash' from galaxy_user where email='%U'\"\n"
  },
  {
    "path": "installed_files/protvis_default.template",
    "content": "export TMPDIR=$protvis_converted_files_dir\nexport PORT=$protvis_port\nexport ADDRESS=\"0.0.0.0\"\nexport PROTVIS_HOME=$protvis_home\nexport PROTVIS_USER=$protvis_user\n"
  },
  {
    "path": "installed_files/protvis_init",
    "content": "#!/bin/sh\n\n### BEGIN INIT INFO\n# Provides:        protvis\n# Required-Start:  \n# Required-Stop:   \n# Default-Start:   2 3 4 5\n# Default-Stop:    0 1 6\n# Short-Description: Start protvis daemon\n### END INIT INFO\n\nPATH=/sbin:/bin:/usr/sbin:/usr/bin\n\nNAME=\"protvis\"\n. /lib/lsb/init-functions\n. /etc/default/$NAME\n\ncase $1 in\n    start)\n\tDAEMON_ARGS=\"--daemon --pid-file=$PROTVIS_HOME/daemon.pid $PROTVIS_HOME/production.ini http_port=$PORT http_address=$ADDRESS\"\n\tif [ ! -f \"$PROTVIS_HOME/production.ini\" ]; then\n\t    log_daemon_msg \"Copying over default settings for protvis.\"\n\t    su - $PROTVIS_USER /bin/bash -c \"cd $PROTVIS_HOME; cp production.ini.sample production.ini\"\n\tfi\n\tlog_daemon_msg \"Starting $NAME\" \"$NAME\"\n\tsu - $PROTVIS_USER /bin/bash -c \"export TMPDIR=$TMPDIR; cd $PROTVIS_HOME; env/bin/pserve $DAEMON_ARGS\"\n        log_end_msg $?\n\t;;\n    stop)\n\tlog_daemon_msg \"Stopping $NAME\" \"$NAME\"\n\tDAEMON_ARGS=\"--stop-daemon --pid-file=$PROTVIS_HOME/daemon.pid\"\n\tsu - $PROTVIS_USER /bin/bash -c \"cd $PROTVIS_HOME; env/bin/pserve $DAEMON_ARGS\"\n\tlog_end_msg $?\n\t;;\n    restart)\n\t$0 stop && sleep 2 && $0 start\n\t;;\n    *)\n        echo \"Usage: $0 {start|stop|restart}\"\n        exit 2\n        ;;\n\nesac\n\n"
  },
  {
    "path": "installed_files/psql",
    "content": "#!/bin/bash\n#\n# This script is required because of CloudMan - CloudMan's path resolution\n# code would result in using the latest version of PostgreSQL installed via\n# a package manager. However, if a user switches images, the database data\n# may not match the version of PostgreSQL being invoked so this script\n# is used to differentiate which PostgreSQL version is actually invoked.\n# This had to be done here because it was the only variable for old\n# clusters being invoked with a new image. Also note that this script\n# is image dependent (because PostgreSQL v9.1 will be the default version).\n\nversions=( \"9.1\" \"8.4\" )\n\nargs=\"\"\nfor arg in \"$@\"; do\n    # If there are spaces in an argument, surround it with quotes or psql won’t work\n    [[ \"$arg\" = \"${arg% *}\" ]] && args=\"$args $arg\" || args=\"$args \\\"$arg\\\"\"\ndone\n\nfor ver in \"${versions[@]}\"; do\n    if [ \"$ver\" == \"9.1\" ]; then\n       cmd=\"/usr/lib/postgresql/$ver/bin/psql_orig $args\"\n    else\n       cmd=\"/usr/lib/postgresql/$ver/bin/psql $args\"\n    fi\n    echo “Running command: $cmd”\n    # Must eval here; otherwise quotes in the args don’t get passed through properly\n    eval $cmd\n    ec=$?\n    if [ $ec -eq 0 ]; then\n        echo \"Version $ver seems to work\"\n        exit $ec\n    fi\ndone\n# Didn’t find a working solution it seems\nexit 1\n"
  },
  {
    "path": "installed_files/setupnx.sh",
    "content": "#!/bin/bash\n\n# Set up NX to work with password access.\n# This can be run manually or triggered from ~/.bash_login\n\nif [ `id -u` != 0 ] ; then\n    echo \"This script must be run as root.\"\n    exit 1\nfi\n\nCURRENTUSER=${1:-$USER}\n\ncat <<BLURB\nThe NX remote desktop software needs password-based authentication enabled.\nThis functionality will now be activated.  While you can still log in using your private\nkey file, please note that password authentication will now be possible both via NX and \nregular ssh for any user account which has a password set.\n\nYou can choose to set a password for $CURRENTUSER or give a new user name to create a\nnew non-privileged account.\n\nBLURB\n\nread -p \"User to set password for [$CURRENTUSER]: \" USERTOMAKE\nif [ -z \"$USERTOMAKE\" ] ; then USERTOMAKE=\"$CURRENTUSER\" ; fi\nif getent passwd \"$USERTOMAKE\" > /dev/null ; then\n    passwd \"$USERTOMAKE\"\nelse\n    useradd -m -s /bin/bash \"$USERTOMAKE\"\n    passwd \"$USERTOMAKE\"\nfi\n\n#force SSH to allow password logins\necho \"Reconfiguring SSH to allow password login.\"\nsed -i 's/^PasswordAuthentication .*/PasswordAuthentication yes/' /etc/ssh/sshd_config\n/etc/init.d/ssh reload\n\necho \"Setting up NX.\"\n#force freenx-server to fix itself in any case\nif ! dpkg -s freenx-server >&/dev/null ; then\n    apt-get -y install freenx-server\nfi\ndpkg-reconfigure -pcritical freenx-server >&/dev/null\n\necho \"DONE!  You can find clients for NX here: \"\necho \"               http://www.nomachine.com/download.php\"\n\n"
  },
  {
    "path": "installed_files/tool_data_table_conf.xml",
    "content": "<tables>\n    <!-- Locations of MAF files that have been indexed with bx-python -->\n    <table name=\"indexed_maf_files\">\n        <columns>name, value, dbkey, species</columns>\n        <file path=\"tool-data/maf_index.loc\" />\n    </table>\n    <!-- Locations of indexes in the BWA mapper format -->\n    <table name=\"bwa_indexes\" comment_char=\"#\">\n        <columns>value, dbkey, name, path</columns>\n        <file path=\"tool-data/bwa_index.loc\" />\n    </table>\n    <!-- Locations of indexes in the Bowtie mapper format -->\n    <table name=\"bowtie_indexes\" comment_char=\"#\">\n        <columns>value, dbkey, name, path</columns>\n        <file path=\"tool-data/bowtie_indices.loc\" />\n    </table>\n    <!-- Locations of indexes in the Bowtie2 mapper format -->\n    <table name=\"bowtie2_indexes\" comment_char=\"#\">\n        <columns>value, dbkey, name, path</columns>\n        <file path=\"tool-data/bowtie2_indices.loc\" />\n    </table>\n    <!-- Locations of indexes in the Bismark mapper format -->\n    <table name=\"bismark_indexes\" comment_char=\"#\">\n        <columns>value, dbkey, name, path</columns>\n        <file path=\"tool-data/bismark_indices.loc\" />\n    </table>\n    <!-- Locations of indexes in the Bowtie mapper format for TopHat to use -->\n    <table name=\"tophat_indexes\" comment_char=\"#\">\n        <columns>value, dbkey, name, path</columns>\n        <file path=\"tool-data/bowtie_indices.loc\" />\n    </table>\n    <!-- Locations of indexes in the Bowtie2 mapper format for TopHat2 to use -->\n    <table name=\"tophat2_indexes\" comment_char=\"#\">\n        <columns>value, dbkey, name, path</columns>\n        <file path=\"tool-data/bowtie2_indices.loc\" />\n    </table>\n    <!-- Location of SAMTools indexes and other files -->\n    <table name=\"sam_fa_indexes\" comment_char=\"#\">\n        <columns>index, value, path</columns>\n        <file path=\"tool-data/sam_fa_indices.loc\" />\n    </table>\n    <!-- Location of Picard dict file and other files -->\n    <table name=\"picard_indexes\" comment_char=\"#\">\n        <columns>value, dbkey, name, path</columns>\n        <file path=\"tool-data/picard_index.loc\" />\n    </table>\n    <!-- Location of Picard dict files valid for GATK -->\n    <table name=\"gatk_picard_indexes\" comment_char=\"#\">\n        <columns>value, dbkey, name, path</columns>\n        <file path=\"tool-data/gatk_sorted_picard_index.loc\" />\n    </table>\n    <!-- Alignment and sequence data -->\n    <table name=\"alignseq\">\n       <!-- XXX This has two different types of columns -->\n        <columns>index, dbkey, value</columns>\n        <file path=\"tool-data/alignseq.loc\" />\n    </table>\n    <!-- UCSC twoBit indexed files -->\n    <table name=\"twobit\">\n        <columns>dbkey, value</columns>\n        <file path=\"tool-data/twobit.loc\" />\n    </table>\n    <!-- liftOver files -->\n    <table name=\"lift_over\">\n        <columns>dbkey1, dbkey2, value</columns>\n        <file path=\"tool-data/liftOver.loc\" />\n    </table>\n    <!-- Location of Mosaik files -->\n    <table name=\"mosaik_indexes\" comment_char=\"#\">\n        <columns>value, dbkey, name, path</columns>\n        <file path=\"tool-data/mosaik_index.loc\" />\n    </table>\n    <!-- Novoalign indexes, not part of default Galaxy configurations -->\n    <table name=\"novoalign_indexes\" comment_char=\"#\">\n        <columns>value, dbkey, name, path</columns>\n        <file path=\"tool-data/novoalign_indices.loc\" />\n    </table>\n</tables>\n"
  },
  {
    "path": "installed_files/vncserver_default.template",
    "content": "# The Username:Group that will run VNC\nexport USER=\"$vnc_user\"\n\n# The display that VNC will use\nexport DISPLAY=\"$vnc_display\"\n\n# Color depth (between 8 and 32)\nexport DEPTH=\"$vnc_depth\"\n\n# The Desktop geometry to use.\nexport GEOMETRY=\"$vnc_geometry\"\n\n# The name that the VNC Desktop will have.\nexport DESKTOP_NAME=\"cloudbiolinux-vnc-server\""
  },
  {
    "path": "installed_files/vncserver_init",
    "content": "#!/bin/sh -e\n### BEGIN INIT INFO\n# Provides:          vncserver\n# Required-Start:    networking\n# Default-Start:     3 4 5\n# Default-Stop:      0 6\n### END INIT INFO\nPATH=\"$PATH:/usr/X11R6/bin/\"\nNAME=\"vncserver\"\n\n# Read configuration variable file if it is present\n[ -r /etc/default/$NAME ] && . /etc/default/$NAME\n\nOPTIONS=\"-name ${DESKTOP_NAME} -depth ${DEPTH} -geometry ${GEOMETRY} :${DISPLAY}\"\n\n. /lib/lsb/init-functions\n\ncase \"$1\" in\nstart)\nlog_action_begin_msg \"Starting vncserver for user '${USER}' on localhost:${DISPLAY}\"\nsu ${USER} -c \"/usr/bin/vncserver ${OPTIONS}\"\n;;\n\nstop)\nlog_action_begin_msg \"Stoping vncserver for user '${USER}' on localhost:${DISPLAY}\"\nsu ${USER} -c \"/usr/bin/vncserver -kill :${DISPLAY}\"\n;;\n\nrestart)\n$0 stop\n$0 start\n;;\nesac\n\nexit 0"
  },
  {
    "path": "installed_files/xstartup",
    "content": "[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup\n[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources\nxsetroot -solid grey\nvncconfig -iconic &\nx-terminal-emulator -geometry 80x24+10+10 -ls -title \"$VNCDESKTOP Desktop\" &\n# jwm &\nstartxfce4 &\n"
  },
  {
    "path": "installed_files/xvfb_default",
    "content": "XVFB_OPTS=\":42 -auth /var/lib/xvfb/auth -ac -nolisten tcp -shmem -screen 0 800x600x24\"\n\n"
  },
  {
    "path": "installed_files/xvfb_init",
    "content": "#!/bin/sh\n\n### BEGIN INIT INFO\n# Provides:        xvfb\n# Required-Start:  $syslog\n# Required-Stop:   $syslog\n# Default-Start:   2 3 4 5\n# Default-Stop:    0 1 6\n# Short-Description: Start Xvfb daemon\n### END INIT INFO\n\nPATH=/sbin:/bin:/usr/sbin:/usr/bin\n\n. /lib/lsb/init-functions\n\nNAME=xvfb\nDAEMON=/usr/bin/Xvfb\nPIDFILE=/var/run/Xvfb.pid\n\ntest -x $DAEMON || exit 5\n\nif [ -r /etc/default/$NAME ]; then\n        . /etc/default/$NAME\nfi\n\ncase $1 in\n        start)\n                log_daemon_msg \"Starting Virtual Framebuffer\" \"Xvfb\"\n                start-stop-daemon --start --quiet --background --make-pidfile --pidfile $PIDFILE --startas $DAEMON -- $XVFB_OPTS\n                status=$?\n                log_end_msg $status\n                ;;\n        stop)\n                log_daemon_msg \"Stopping Virtual Framebuffer\" \"Xvfb\"\n                start-stop-daemon --stop --quiet --pidfile $PIDFILE\n                log_end_msg $?\n                rm -f $PIDFILE\n                ;;\n        restart|force-reload)\n                $0 stop && sleep 2 && $0 start\n                ;;\n        try-restart)\n                if $0 status >/dev/null; then\n                        $0 restart\n                else\n                        exit 0\n                fi\n                ;;\n        reload)\n                exit 3\n                ;;\n        status)\n                pidofproc -p $PIDFILE $DAEMON >/dev/null\n                status=$?\n                if [ $status -eq 0 ]; then\n                        log_success_msg \"Xvfb server is running.\"\n                else\n                        log_failure_msg \"Xvfb server is not running.\"\n                fi\n                exit $status\n                ;;\n        *)\n                echo \"Usage: $0 {start|stop|restart|try-restart|force-reload|status}\"\n                exit 2\n                ;;\nesac\n\n"
  },
  {
    "path": "manifest/custom-packages.yaml",
    "content": "abyss:\n  description: Assembly By Short Sequences - a de novo, parallel, paired-end sequence\n    assembler.\n  homepage_uri: http://www.bcgsc.ca/platform/bioinfo/software/abyss\n  name: abyss\n  version: 1.3.3\nbeast:\n  description: 'BEAST: Bayesian MCMC analysis of molecular sequences.'\n  homepage_uri: http://beast.bio.ed.ac.uk/main_page\n  name: beast\n  version: ''\nbedtools:\n  description: A flexible suite of utilities for comparing genomic features.\n  homepage_uri: https://code.google.com/p/bedtools/\n  name: bedtools\n  version: github\nbfast:\n  description: 'BFAST: Blat-like Fast Accurate Search Tool.'\n  homepage_uri: http://sourceforge.net/apps/mediawiki/bfast/index.php?title=Main_Page\n  name: bfast\n  version: 0.7.0\nbio4j:\n  description: Bio4j graph based database built on Neo4j with UniProt, GO, RefSeq\n    and more.\n  homepage_uri: http://www.bio4j.com/\n  name: bio4j\n  version: '0.7'\nbowtie:\n  description: The bowtie short read aligner.\n  homepage_uri: http://bowtie-bio.sourceforge.net/index.shtml\n  name: bowtie\n  version: 0.12.7\nbowtie2:\n  description: bowtie2 short read aligner, with gap support.\n  homepage_uri: http://bowtie-bio.sourceforge.net/bowtie2/index.shtml\n  name: bowtie2\n  version: 2.0.0-beta6\nbwa:\n  description: 'BWA:  aligns short nucleotide sequences against a long reference sequence.'\n  homepage_uri: http://bio-bwa.sourceforge.net/\n  name: bwa\n  version: 0.5.9\nbx_python:\n  description: Tools for manipulating biological data, particularly multiple sequence\n    alignments\n  homepage_uri: https://bitbucket.org/james_taylor/bx-python/wiki/Home\n  name: bx_python\n  version: bitbucket\ncljr:\n  description: Clojure package manager, cljr.\n  homepage_uri: http://github.com/liebke/cljr\n  name: cljr\n  version: ''\ncloudman:\n  description: ' A meta method for installing all of CloudMan components.'\n  homepage_uri: ''\n  name: cloudman\n  version: ''\ncrisp:\n  description: Detect SNPs and short indels from pooled sequencing data.\n  homepage_uri: https://sites.google.com/site/vibansal/software/crisp/\n  name: crisp\n  version: '5'\ncufflinks:\n  description: Cufflinks assembles transcripts, estimates their abundances, and tests\n    for differential expression and regulation in RNA-Seq samples.\n  homepage_uri: http://cufflinks.cbcb.umd.edu/\n  name: cufflinks\n  version: 2.0.0\ndwgsim:\n  description: 'DWGSIM: simulating NGS data and evaluating mappings and variant calling.'\n  homepage_uri: http://sourceforge.net/apps/mediawiki/dnaa/index.php?title=Main_Page\n  name: dwgsim\n  version: 0.1.8\necho:\n  description: 'ECHO: A reference-free short-read error correction algorithm'\n  homepage_uri: http://uc-echo.sourceforge.net/\n  name: echo\n  version: '1_11'\nemboss:\n  description: 'EMBOSS: A high-quality package of free, Open Source software for molecular\n    biology.'\n  homepage_uri: http://emboss.sourceforge.net/\n  name: emboss\n  version: 6.3.1\nfastqc:\n  description: A quality control tool for high throughput sequence data.\n  homepage_uri: http://www.bioinformatics.babraham.ac.uk/projects/fastqc/\n  name: fastqc\n  version: 0.10.1\nfastx_toolkit:\n  description: 'FASTX-Toolkit: collection of command line tools for Short-Reads FASTA/FASTQ\n    files preprocessing.'\n  homepage_uri: http://hannonlab.cshl.edu/fastx_toolkit/\n  name: fastx_toolkit\n  version: 0.0.13\nfreebayes:\n  description: Bayesian haplotype-based polymorphism discovery and genotyping.\n  homepage_uri: https://github.com/ekg/freebayes\n  name: freebayes\n  version: github\ngatk:\n  description: 'GATK: library for writing efficient analysis tools using next-generation\n    sequencing data '\n  homepage_uri: http://www.broadinstitute.org/gsa/wiki/index.php/Home_Page\n  name: gatk\n  version: 1.6-11-g3b2fab9\ngatk_queue:\n  description: Command-line scripting framework for defining multi-stage genomic analysis\n    pipelines.\n  homepage_uri: http://www.broadinstitute.org/gsa/wiki/index.php/GATK-Queue\n  name: gatk_queue\n  version: 1.6-11-g3b2fab9\ngmap:\n  description: 'GMAP and GSNAP: A Genomic Mapping and Alignment Program for mRNA EST\n    and short reads.'\n  homepage_uri: http://research-pub.gene.com/gmap/\n  name: gmap\n  version: '2011-11-12'\nhydra:\n  description: Hydra detects structural variation breakpoints in both unique and duplicated\n    genomic regions.\n  homepage_uri: https://code.google.com/p/hydra-sv/\n  name: hydra\n  version: 0.5.3\nkraken:\n  description: kraken\n  homepage_uri: 'http://ccb.jhu.edu/software/kraken/'\n  name: kraken\n  version: 0.10.3-beta\nlastz:\n  description: LASTZ sequence alignment program.\n  homepage_uri: http://www.bx.psu.edu/miller_lab/dist/README.lastz-1.02.00/README.lastz-1.02.00a.html\n  name: lastz\n  version: 1.02.00\nleinengin:\n  description: Clojure tool for project configuration and automation.\n  homepage_uri: http://github.com/technomancy/leiningen\n  name: leinengin\n  version: ''\nmacs:\n  description: Model-based Analysis for ChIP-Seq.\n  homepage_uri: http://liulab.dfci.harvard.edu/MACS/\n  name: macs\n  version: 1.4.2\nmatplotlib:\n  description: matplotlib is a python 2D plotting library which produces publication\n    quality figures\n  homepage_uri: http://matplotlib.sourceforge.net/\n  name: matplotlib\n  version: 1.0.1\nmosaik:\n  description: 'MOSAIK: reference-guided aligner for next-generation sequencing technologies'\n  homepage_uri: http://code.google.com/p/mosaik-aligner/\n  name: mosaik\n  version: github\nnginx:\n  description: Nginx open source web server.\n  homepage_uri: http://www.nginx.org/\n  name: nginx\n  version: 1.2.0\nnovoalign:\n  description: Novoalign short read aligner using Needleman-Wunsch algorithm with\n    affine gap penalties.\n  homepage_uri: http://www.novocraft.com/main/index.php\n  name: novoalign\n  version: V2.08.01\nperm:\n  description: Efficient mapping of short sequences accomplished with periodic full\n    sensitive spaced seeds.\n  homepage_uri: https://code.google.com/p/perm/\n  name: perm\n  version: '3.6'\npgdspider:\n  description: PGDSpider format conversion for population genetics programs.\n  homepage_uri: http://www.cmpg.unibe.ch/software/PGDSpider/\n  name: pgdspider\n  version: 2.0.1.2\npicard:\n  description: Command-line utilities that manipulate BAM files with a Java API.\n  homepage_uri: http://picard.sourceforge.net/\n  name: picard\n  version: '1.68'\nplink_seq:\n  description: A toolset for working with human genetic variation data.\n  homepage_uri: http://atgu.mgh.harvard.edu/plinkseq/\n  name: plink_seq\n  version: '0.08'\nproftpd:\n  description: Highly configurable GPL-licensed FTP server software.\n  homepage_uri: http://proftpd.org/\n  name: proftpd\n  version: 1.3.4a\npydoop:\n  description: pydoop; provides Hadoop access for Python.\n  homepage_uri: http://pydoop.sourceforge.net/docs/\n  name: pydoop\n  version: ''\nrpy:\n  description: RPy is a very simple, yet robust, Python interface to the R Programming\n    Language.\n  homepage_uri: http://rpy.sourceforge.net/\n  name: rpy\n  version: 1.0.3\ns3fs:\n  description: FUSE-based file system backed by Amazon S3.\n  homepage_uri: https://code.google.com/p/s3fs/\n  name: s3fs\n  version: '1.61'\nsamtools:\n  description: SAM Tools provide various utilities for manipulating alignments in\n    the SAM format.\n  homepage_uri: http://samtools.sourceforge.net/\n  name: samtools\n  version: 0.1.18\nseal:\n  description: 'Install seal: process high-throughput sequencing with Hadoop.'\n  homepage_uri: http://biodoop-seal.sf.net/\n  name: seal\n  version: ''\nsge:\n  description: Sun Grid Engine.\n  homepage_uri: ''\n  name: sge\n  version: ''\nshrec:\n  description: Shrec is a bioinformatics tool for error correction of HTS read data.\n  homepage_uri: http://sourceforge.net/projects/shrec-ec/\n  name: shrec\n  version: '2.2'\nsnpeff:\n  description: Variant annotation and effect prediction tool.\n  homepage_uri: http://snpeff.sourceforge.net/\n  name: snpeff\n  version: '2_0_5'\nsolexaqa:\n  description: SolexaQA creates visual representations of data quality from FASTQ\n    files.\n  homepage_uri: http://solexaqa.sourceforge.net/\n  name: solexaqa\n  version: '1.4'\nstacks:\n  description: 'Stacks: build loci out of a set of short-read sequenced samples.'\n  homepage_uri: http://creskolab.uoregon.edu/stacks/\n  name: stacks\n  version: '0.998'\nstampy:\n  description: 'Stampy: mapping of short reads from illumina sequencing machines onto\n    a reference genome.'\n  homepage_uri: http://www.well.ox.ac.uk/project-stampy\n  name: stampy\n  version: 1.0.15\ntassel:\n  description: 'TASSEL: evaluate traits associations, evolutionary patterns, and linkage\n    disequilibrium.'\n  homepage_uri: http://www.maizegenetics.net/index.php?option=com_content&task=view&id=89&Itemid=119\n  name: tassel\n  version: '3.0'\ntophat:\n  description: TopHat is a fast splice junction mapper for RNA-Seq reads\n  homepage_uri: http://tophat.cbcb.umd.edu/\n  name: tophat\n  version: 2.0.0\ntracer:\n  description: A program for analysing results from Bayesian MCMC programs such as\n    BEAST & MrBayes.\n  homepage_uri: http://beast.bio.ed.ac.uk/main_page\n  name: tracer\n  version: ''\ntransabyss:\n  description: Analyze ABySS multi-k-assembled shotgun transcriptome data.\n  homepage_uri: http://www.bcgsc.ca/platform/bioinfo/software/trans-abyss\n  name: transabyss\n  version: 1.3.2\ntrinity:\n  description: Efficient and robust de novo reconstruction of transcriptomes from\n    RNA-seq data.\n  homepage_uri: http://trinityrnaseq.sourceforge.net/\n  name: trinity\n  version: r2012-05-18\nucsc_tools:\n  description: Useful executables from UCSC.\n  homepage_uri: http://hgdownload.cse.ucsc.edu/admin/jksrc.zip\n  name: ucsc_tools\n  version: ''\nvarianttools:\n  description: Annotation, selection, and analysis of variants in the context of next-gen\n    sequencing analysis.\n  homepage_uri: http://varianttools.sourceforge.net/\n  name: varianttools\n  version: 1.0.1\nvcftools:\n  description: Work with VCF files, such as those generated by the 1000 Genomes Project.\n  homepage_uri: http://vcftools.sourceforge.net/\n  name: vcftools\n  version: 0.1.7\nvelvet:\n  description: Sequence assembler for very short reads.\n  homepage_uri: http://www.ebi.ac.uk/~zerbino/velvet/\n  name: velvet\n  version: 1.2.05\n"
  },
  {
    "path": "manifest/debian-packages.yaml",
    "content": "accountsservice:\n  description: query and manipulate user account information\n  downloads: 8774\n  homepage_uri: http://cgit.freedesktop.org/accountsservice/\n  name: accountsservice\n  section: gnome\n  version: 0.6.15-2ubuntu9\nacl:\n  description: Access control list utilities\n  downloads: 12836\n  homepage_uri: http://savannah.nongnu.org/projects/acl/\n  name: acl\n  section: utils\n  version: 2.2.51-5ubuntu1\nacpi-support:\n  description: scripts for handling many ACPI events\n  downloads: 131\n  homepage_uri: ''\n  name: acpi-support\n  section: admin\n  version: '0.140'\nacpid:\n  description: Advanced Configuration and Power Interface event daemon\n  downloads: 65108\n  homepage_uri: http://www.tedfelix.com/linux/acpid-netlink.html\n  name: acpid\n  section: admin\n  version: 1:2.0.10-1ubuntu3\nactivity-log-manager-common:\n  description: blacklist configuration for Zeitgeist (assets)\n  downloads: 0\n  homepage_uri: https://launchpad.net/activity-log-manager\n  name: activity-log-manager-common\n  section: utils\n  version: 0.9.4-0ubuntu3\nactivity-log-manager-control-center:\n  description: blacklist configuration for Zeitgeist (control center integration)\n  downloads: 0\n  homepage_uri: https://launchpad.net/activity-log-manager\n  name: activity-log-manager-control-center\n  section: utils\n  version: 0.9.4-0ubuntu3\nadduser:\n  description: add and remove users and groups\n  downloads: 108947\n  homepage_uri: http://alioth.debian.org/projects/adduser/\n  name: adduser\n  section: admin\n  version: 3.113ubuntu2\nadium-theme-ubuntu:\n  description: Adium message style for Ubuntu\n  downloads: 1\n  homepage_uri: http://launchpad.net/adium-theme-ubuntu\n  name: adium-theme-ubuntu\n  section: gnome\n  version: 0.3.2-0ubuntu1\naisleriot:\n  description: Solitaire card games\n  downloads: 751\n  homepage_uri: https://live.gnome.org/Aisleriot\n  name: aisleriot\n  section: games\n  version: 1:3.2.3.2-0ubuntu1\nalacarte:\n  description: easy GNOME menu editing tool\n  downloads: 4661\n  homepage_uri: ''\n  name: alacarte\n  section: universe/utils\n  version: 0.13.2-2ubuntu4\nalsa-base:\n  description: ALSA driver configuration files\n  downloads: 2273\n  homepage_uri: http://www.alsa-project.org/\n  name: alsa-base\n  section: sound\n  version: 1.0.25+dfsg-0ubuntu1\nalsa-utils:\n  description: Utilities for configuring and using ALSA\n  downloads: 43508\n  homepage_uri: http://www.alsa-project.org/\n  name: alsa-utils\n  section: sound\n  version: 1.0.25-1ubuntu5\nanacron:\n  description: cron-like program that doesn't go by time\n  downloads: 52784\n  homepage_uri: ''\n  name: anacron\n  section: admin\n  version: 2.3-14ubuntu1\nant:\n  description: Java based build tool like make\n  downloads: 3787\n  homepage_uri: http://ant.apache.org/\n  name: ant\n  section: java\n  version: 1.8.2-4build1\nant-optional:\n  description: Java based build tool like make - optional libraries\n  downloads: 1806\n  homepage_uri: http://ant.apache.org/\n  name: ant-optional\n  section: java\n  version: 1.8.2-4build1\nantlr3:\n  description: language tool for constructing recognizers, compilers etc\n  downloads: 845\n  homepage_uri: ''\n  name: antlr3\n  section: universe/devel\n  version: 3.2.is.3.2-0ubuntu1\napache2:\n  description: Apache HTTP Server metapackage\n  downloads: 22309\n  homepage_uri: http://httpd.apache.org/\n  name: apache2\n  section: web\n  version: 2.2.22-1ubuntu1\napache2-mpm-worker:\n  description: Apache HTTP Server - high speed threaded model\n  downloads: 7746\n  homepage_uri: http://httpd.apache.org/\n  name: apache2-mpm-worker\n  section: net\n  version: 2.2.22-1ubuntu1\napache2-utils:\n  description: utility programs for webservers\n  downloads: 52222\n  homepage_uri: http://httpd.apache.org/\n  name: apache2-utils\n  section: net\n  version: 2.2.22-1ubuntu1\napache2.2-bin:\n  description: Apache HTTP Server common binary files\n  downloads: 54017\n  homepage_uri: http://httpd.apache.org/\n  name: apache2.2-bin\n  section: httpd\n  version: 2.2.22-1ubuntu1\napache2.2-common:\n  description: Apache HTTP Server common files\n  downloads: 52101\n  homepage_uri: http://httpd.apache.org/\n  name: apache2.2-common\n  section: web\n  version: 2.2.22-1ubuntu1\napg:\n  description: Automated Password Generator - Standalone version\n  downloads: 2576\n  homepage_uri: ''\n  name: apg\n  section: admin\n  version: 2.2.3.dfsg.1-2\napp-install-data:\n  description: Ubuntu applications (data files)\n  downloads: 11372\n  homepage_uri: ''\n  name: app-install-data\n  section: gnome\n  version: 0.12.04.4\napp-install-data-partner:\n  description: Application Installer (data files for partner applications/repositories)\n  downloads: 0\n  homepage_uri: ''\n  name: app-install-data-partner\n  section: x11\n  version: 12.12.04.1\napparmor:\n  description: User-space parser utility for AppArmor\n  downloads: 80\n  homepage_uri: http://apparmor.net/\n  name: apparmor\n  section: admin\n  version: 2.7.102-0ubuntu3.1\nappmenu-gtk:\n  description: Export GTK menus over DBus\n  downloads: 3\n  homepage_uri: https://launchpad.net/appmenu-gtk\n  name: appmenu-gtk\n  section: libs\n  version: 0.3.92-0ubuntu1\nappmenu-gtk3:\n  description: Export GTK menus over DBus\n  downloads: 2\n  homepage_uri: https://launchpad.net/appmenu-gtk\n  name: appmenu-gtk3\n  section: libs\n  version: 0.3.92-0ubuntu1\nappmenu-qt:\n  description: appmenu support for Qt\n  downloads: 3\n  homepage_uri: https://launchpad.net/appmenu-qt\n  name: appmenu-qt\n  section: x11\n  version: 0.2.6-0ubuntu1\napport:\n  description: automatically generate crash reports for debugging\n  downloads: 18\n  homepage_uri: https://wiki.ubuntu.com/Apport\n  name: apport\n  section: utils\n  version: 2.0.1-0ubuntu8\napport-gtk:\n  description: GTK+ frontend for the apport crash report system\n  downloads: 0\n  homepage_uri: https://wiki.ubuntu.com/Apport\n  name: apport-gtk\n  section: gnome\n  version: 2.0.1-0ubuntu8\napport-symptoms:\n  description: symptom scripts for apport\n  downloads: 0\n  homepage_uri: https://wiki.ubuntu.com/Apport\n  name: apport-symptoms\n  section: utils\n  version: '0.16'\napt:\n  description: commandline package manager\n  downloads: 109951\n  homepage_uri: ''\n  name: apt\n  section: admin\n  version: 0.8.16~exp12ubuntu10.2\napt-transport-https:\n  description: https download transport for APT\n  downloads: 140\n  homepage_uri: ''\n  name: apt-transport-https\n  section: admin\n  version: 0.8.16~exp12ubuntu10.2\napt-utils:\n  description: package managment related utility programs\n  downloads: 51631\n  homepage_uri: ''\n  name: apt-utils\n  section: admin\n  version: 0.8.16~exp12ubuntu10.2\napt-xapian-index:\n  description: maintenance and search tools for a Xapian index of Debian packages\n  downloads: 46570\n  homepage_uri: http://www.enricozini.org/sw/apt-xapian-index/\n  name: apt-xapian-index\n  section: admin\n  version: 0.44ubuntu5\naptdaemon:\n  description: transaction based package management service\n  downloads: 20343\n  homepage_uri: https://launchpad.net/aptdaemon\n  name: aptdaemon\n  section: admin\n  version: 0.43+bzr805-0ubuntu1\naptdaemon-data:\n  description: data files for clients\n  downloads: 844\n  homepage_uri: https://launchpad.net/aptdaemon\n  name: aptdaemon-data\n  section: admin\n  version: 0.43+bzr805-0ubuntu1\naptitude:\n  description: terminal-based package manager (terminal interface only)\n  downloads: 28616\n  homepage_uri: http://aptitude.alioth.debian.org/\n  name: aptitude\n  section: admin\n  version: 0.6.6-1ubuntu1\napturl:\n  description: install packages using the apt protocol - GTK+ frontend\n  downloads: 16\n  homepage_uri: ''\n  name: apturl\n  section: admin\n  version: 0.5.1ubuntu3\napturl-common:\n  description: install packages using the apt protocol - common data\n  downloads: 14\n  homepage_uri: ''\n  name: apturl-common\n  section: admin\n  version: 0.5.1ubuntu3\naspell:\n  description: GNU Aspell spell-checker\n  downloads: 18130\n  homepage_uri: http://aspell.net/\n  name: aspell\n  section: text\n  version: 0.60.7~20110707-1\naspell-en:\n  description: English dictionary for GNU Aspell\n  downloads: 10099\n  homepage_uri: http://aspell.net/\n  name: aspell-en\n  section: text\n  version: 6.0-0-6ubuntu2\nat:\n  description: Delayed job execution and batch processing\n  downloads: 61794\n  homepage_uri: ''\n  name: at\n  section: admin\n  version: 3.1.13-1ubuntu1\nat-spi2-core:\n  description: Assistive Technology Service Provider Interface (dbus core)\n  downloads: 352\n  homepage_uri: http://live.gnome.org/Accessibility/GNOME3\n  name: at-spi2-core\n  section: misc\n  version: 2.4.2-0ubuntu0.1\nautoconf:\n  description: automatic configure script builder\n  downloads: 4443\n  homepage_uri: http://www.gnu.org/software/autoconf\n  name: autoconf\n  section: devel\n  version: 2.68-1ubuntu2\nautomake:\n  description: Tool for generating GNU Standards-compliant Makefiles\n  downloads: 3850\n  homepage_uri: http://www.gnu.org/software/automake/\n  name: automake\n  section: devel\n  version: 1:1.11.3-1ubuntu2\nautotools-dev:\n  description: Update infrastructure for config.{guess,sub} files\n  downloads: 4445\n  homepage_uri: http://savannah.gnu.org/projects/config/\n  name: autotools-dev\n  section: devel\n  version: 20120210.1ubuntu1\navahi-autoipd:\n  description: Avahi IPv4LL network address configuration daemon\n  downloads: 5901\n  homepage_uri: http://avahi.org/\n  name: avahi-autoipd\n  section: net\n  version: 0.6.30-5ubuntu2\navahi-daemon:\n  description: Avahi mDNS/DNS-SD daemon\n  downloads: 48215\n  homepage_uri: http://avahi.org/\n  name: avahi-daemon\n  section: net\n  version: 0.6.30-5ubuntu2\navahi-utils:\n  description: Avahi browsing, publishing and discovery utilities\n  downloads: 632\n  homepage_uri: http://avahi.org/\n  name: avahi-utils\n  section: net\n  version: 0.6.30-5ubuntu2\naxel:\n  description: light download accelerator - console version\n  downloads: 200\n  homepage_uri: http://axel.alioth.debian.org/\n  name: axel\n  section: universe/web\n  version: 2.4-1\nbamfdaemon:\n  description: Window matching library - daemon\n  downloads: 18\n  homepage_uri: https://launchpad.net/bamf\n  name: bamfdaemon\n  section: libs\n  version: 0.2.118-0ubuntu0.1\nbaobab:\n  description: GNOME disk usage analyzer\n  downloads: 4281\n  homepage_uri: http://live.gnome.org/GnomeUtils\n  name: baobab\n  section: gnome\n  version: 3.4.1-0ubuntu1\nbase-files:\n  description: Debian base system miscellaneous files\n  downloads: 110597\n  homepage_uri: ''\n  name: base-files\n  section: admin\n  version: 6.5ubuntu6\nbase-passwd:\n  description: Debian base system master password and group files\n  downloads: 110291\n  homepage_uri: ''\n  name: base-passwd\n  section: admin\n  version: 3.5.24\nbash:\n  description: GNU Bourne Again SHell\n  downloads: 110784\n  homepage_uri: http://tiswww.case.edu/php/chet/bash/bashtop.html\n  name: bash\n  section: shells\n  version: 4.2-2ubuntu2\nbash-completion:\n  description: programmable completion for the bash shell\n  downloads: 6280\n  homepage_uri: http://bash-completion.alioth.debian.org\n  name: bash-completion\n  section: shells\n  version: 1:1.3-1ubuntu8\nbc:\n  description: The GNU bc arbitrary precision calculator language\n  downloads: 18413\n  homepage_uri: ''\n  name: bc\n  section: math\n  version: 1.06.95-2\nbind9-host:\n  description: Version of 'host' bundled with BIND 9.X\n  downloads: 53926\n  homepage_uri: ''\n  name: bind9-host\n  section: net\n  version: 1:9.8.1.dfsg.P1-4ubuntu0.1\nbinfmt-support:\n  description: Support for extra binary formats\n  downloads: 18676\n  homepage_uri: ''\n  name: binfmt-support\n  section: admin\n  version: 2.0.8\nbinutils:\n  description: GNU assembler, linker and binary utilities\n  downloads: 69573\n  homepage_uri: ''\n  name: binutils\n  section: devel\n  version: 2.22-6ubuntu1\nbio-linux-assembly-conversion-tools:\n  description: Conversion tools for handling 454 assemblies.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-assembly-conversion-tools\n  section: science\n  version: 0.01-9\nbio-linux-base-directories:\n  description: Essential Bio-Linux configuration files.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-base-directories\n  section: science\n  version: 1.0-37\nbio-linux-big-blast:\n  description: The big-blast script for annotation of long sequences\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-big-blast\n  section: ''\n  version: 1.0.3-4\nbio-linux-blast:\n  description: Rapid searching of nucleotide and protein databases.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-blast\n  section: science\n  version: 1:1bl-6\nbio-linux-blast+:\n  description: Blast+ contains new builds of the popular blast family of programs\n    from the NCBI.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-blast+\n  section: science\n  version: 1:1bl1\nbio-linux-bldp-files:\n  description: Documentation for bioinformatics software on Bio-Linux.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-bldp-files\n  section: science\n  version: 1.1-96\nbio-linux-blixem:\n  description: Interactive browser for viewing pairwise Blast results.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-blixem\n  section: science\n  version: 3.1.1-1\nbio-linux-cap3:\n  description: A base calling system\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-cap3\n  section: science\n  version: 20080520-1\nbio-linux-cd-hit:\n  description: a suite of programs designed to quickly group sequences\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-cd-hit\n  section: science\n  version: 2009-1020-1\nbio-linux-clcsequenceviewer:\n  description: CLC Free Sequence Viewer\n  downloads: 1\n  homepage_uri: ''\n  name: bio-linux-clcsequenceviewer\n  section: science\n  version: 6.4-1\nbio-linux-clustal:\n  description: Installs clustalw and clustalx for multiple sequence and profile alignments.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-clustal\n  section: science\n  version: 1:1bl-6\nbio-linux-coalesce:\n  description: Population estimate modelling.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-coalesce\n  section: science\n  version: 1.5b-1\nbio-linux-das-prep:\n  description: Sets up dependencies necessary for running the DAS Proserver on Bio-Linux\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-das-prep\n  section: science\n  version: 0.1-4\nbio-linux-dendroscope:\n  description: Interactive viewer for large phylogenetic trees.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-dendroscope\n  section: science\n  version: 1.4.0-5\nbio-linux-dotter:\n  description: Graphical dotplot program for detailed comparison of two sequences\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-dotter\n  section: science\n  version: 3.1-1\nbio-linux-dust:\n  description: Dust - repetitive sequence masker\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-dust\n  section: science\n  version: 1+0.1+20020708-4\nbio-linux-estscan:\n  description: Program for detecting coding regions of DNA sequences.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-estscan\n  section: science\n  version: 2.1-11\nbio-linux-exchanger:\n  description: Exchanger lets you store, annotate and share files via an omixed server.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-exchanger\n  section: science\n  version: 1.0-4\nbio-linux-fasta:\n  description: Collection of programs for searching DNA and protein databases.\n  downloads: 1\n  homepage_uri: ''\n  name: bio-linux-fasta\n  section: science\n  version: 36t2d5-0\nbio-linux-fastdnaml:\n  description: fastDNAml is a program for estimating maximum likelihood phylogenetic\n    trees from nucleotide sequences.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-fastdnaml\n  section: science\n  version: 1:1bl-0\nbio-linux-fasttree:\n  description: FastTree infers approximately-maximum-likelihood phylogenetic trees\n    from alignments of nucleotide or protein sequences. FastTree can handle alignments\n    with up to a million of sequences in a reasonable amount of time and memory.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-fasttree\n  section: science\n  version: 2.1.3-0\nbio-linux-fluctuate:\n  description: Population modelling software.\n  downloads: 1\n  homepage_uri: ''\n  name: bio-linux-fluctuate\n  section: science\n  version: 1.4-0\nbio-linux-forester:\n  description: ATV (A Tree Viewer) is a Java tool for the visualization of annotated\n    phylogenetic trees.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-forester\n  section: science\n  version: 1.92-10\nbio-linux-gap2caf:\n  description: Conversion tools for handling conversion of gap format assemblies to\n    caf format.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-gap2caf\n  section: science\n  version: 0.20100408-1\nbio-linux-genquery:\n  description: GenQuery is a set of Perl libraries for managing SQL query templates\n    and making web-based query forms.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-genquery\n  section: science\n  version: 2.2.4-3\nbio-linux-glimmer3:\n  description: Glimmer is a system for finding genes in microbial DNA.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-glimmer3\n  section: science\n  version: 3.02-0\nbio-linux-happy:\n  description: 'HAPPY: a software package for Multipoint QTL Mapping in Genetically\n    Heterogeneous Animals'\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-happy\n  section: science\n  version: 1.2-1\nbio-linux-jalview:\n  description: Jalview is a Java multiple alignment editor\n  downloads: 1\n  homepage_uri: ''\n  name: bio-linux-jalview\n  section: science\n  version: 1:2.6.1-1\nbio-linux-keyring:\n  description: Public key package for the Bio-Linux package repository\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-keyring\n  section: misc\n  version: '4'\nbio-linux-lucy:\n  description: Preparation of raw DNA sequence fragments for sequence assembly.\n  downloads: 1\n  homepage_uri: ''\n  name: bio-linux-lucy\n  section: science\n  version: 1.20-0\nbio-linux-maxd:\n  description: maxd is a data warehouse and visualisation environment for genomic\n    expression data.\n  downloads: 1\n  homepage_uri: ''\n  name: bio-linux-maxd\n  section: science\n  version: 2.3.0-5\nbio-linux-mesquite:\n  description: Mesquite is software for evolutionary biology.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-mesquite\n  section: science\n  version: 2.74-1\nbio-linux-migrate:\n  description: Estimates effective population sizes and past migration rates between\n    n populations.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-migrate\n  section: science\n  version: 3.1.2-1\nbio-linux-mrbayes-multi:\n  description: Multiprocessor version of MrBayes\n  downloads: 2\n  homepage_uri: ''\n  name: bio-linux-mrbayes-multi\n  section: science\n  version: 3.1.2-4\nbio-linux-mspcrunch:\n  description: MSPcrunch is a BLAST post-processing filter.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-mspcrunch\n  section: science\n  version: 2.5-0\nbio-linux-mview:\n  description: Multiple alignment viewer for sequence database search results.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-mview\n  section: science\n  version: 1.51-2\nbio-linux-ncbi-tools-x11:\n  description: NCBI libraries for biology applications (X-based utilities)\n  downloads: 2\n  homepage_uri: ''\n  name: bio-linux-ncbi-tools-x11\n  section: science\n  version: 1:1bl-2\nbio-linux-nrdb:\n  description: nrdb can be used to generate quasi-nonredundant protein and nucleotide\n    sequence databases.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-nrdb\n  section: science\n  version: 2-0\nbio-linux-ocount:\n  description: Ocount - oligonucleotide frequency counter.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-ocount\n  section: science\n  version: 0.5-0\nbio-linux-oligoarray:\n  description: Computation of gene-specific oligonucleotides for genome-scale microarray\n    construction.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-oligoarray\n  section: science\n  version: 2.1-8\nbio-linux-oligoarrayaux:\n  description: OligoArrayAux 3.7 is free software that is required for the OligoArray2.1\n    software.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-oligoarrayaux\n  section: science\n  version: 3.7-0\nbio-linux-omegamap:\n  description: OmegaMap is a program for detecting natural selection and recombination\n    in DNA or RNA sequences.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-omegamap\n  section: science\n  version: 0.5-0\nbio-linux-paml:\n  description: Phylogenetic analysis tool for DNA and protein sequences using maximum\n    likelihood.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-paml\n  section: science\n  version: 4.4c-1\nbio-linux-partigene:\n  description: Cluster and anotation tool for sequence data.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-partigene\n  section: science\n  version: 3.05-8\nbio-linux-pfaat:\n  description: Sequence alignment editor designed for protein family analysis.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-pfaat\n  section: science\n  version: 1.0.034-5\nbio-linux-prank:\n  description: Multiple alignment software.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-prank\n  section: science\n  version: 20100318-0\nbio-linux-prevent-upgrade:\n  description: Stop Bio-Linux from upgrading on release of 12.04 LTS\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-prevent-upgrade\n  section: science\n  version: '0.2'\nbio-linux-prot4est:\n  description: Translation of error-prone nucleotide sequences producing robust polypeptide\n    predictions\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-prot4est\n  section: science\n  version: 2.2-10\nbio-linux-qtlcart:\n  description: Suite of programs to map quantitative traits using a map of molecular\n    markers.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-qtlcart\n  section: science\n  version: 1.17f-0\nbio-linux-rbs-finder:\n  description: A program to find Ribosomal binding sites.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-rbs-finder\n  section: ''\n  version: 1.0-3\nbio-linux-recombine:\n  description: Population modelling software.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-recombine\n  section: science\n  version: 1.41-0\nbio-linux-sampledata:\n  description: Sample data for Bio-Linux packages\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-sampledata\n  section: science\n  version: 20041011-9\nbio-linux-seaview:\n  description: Multiple sequence alignment editor.\n  downloads: 1\n  homepage_uri: ''\n  name: bio-linux-seaview\n  section: science\n  version: 4.2.8-2\nbio-linux-sequin:\n  description: Transitional package that provides bio-linux-ncbi-tools-x11\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-sequin\n  section: science\n  version: 1:1bl-0\nbio-linux-splitstree:\n  description: Analysis and visualisation tool for distance data from biological sequences.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-splitstree\n  section: science\n  version: 4.0-11\nbio-linux-squint:\n  description: Squint is a java-based multiple alignment program and editor\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-squint\n  section: science\n  version: 1.02-2\nbio-linux-staden:\n  description: A suite of tools for sequence analysis.\n  downloads: 1\n  homepage_uri: ''\n  name: bio-linux-staden\n  section: science\n  version: 2006.1.7.0+1.11.6-5\nbio-linux-stars:\n  description: STARS is an alternative interface to Staden.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-stars\n  section: science\n  version: 1.2a-11\nbio-linux-taxinspector:\n  description: TaxInspector is a browser for entries in the NCBI taxonomy database.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-taxinspector\n  section: science\n  version: 1.0b3-10\nbio-linux-tetra:\n  description: TETRA - tetranucleotide frequency calculator with GUI.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-tetra\n  section: science\n  version: 1.0b3\nbio-linux-themes-v5:\n  description: Graphics, icons, KDE splash screen and a GDM theme for Bio-Linux 5.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-themes-v5\n  section: science\n  version: '1.2'\nbio-linux-transterm-hp:\n  description: Transterm finds rho-independent transcription terminators in bacterial\n    genomes.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-transterm-hp\n  section: science\n  version: 2.07-1\nbio-linux-treeview:\n  description: TreeView is a simple program for displaying phylogenies.\n  downloads: 1\n  homepage_uri: ''\n  name: bio-linux-treeview\n  section: science\n  version: 1:1bl-0\nbio-linux-trnascan:\n  description: tRNAscan-SE searches for tRNA genes in genomic sequences.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-trnascan\n  section: science\n  version: 1.23-0\nbio-linux-tutorials:\n  description: Tutorials and documentation for Bio-Linux users.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-tutorials\n  section: science\n  version: 1-18\nbio-linux-wise2:\n  description: Tool for comparison of biopolymers, DNA, and protein sequence.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-wise2\n  section: science\n  version: 2.1.20-0\nbio-linux-xcut:\n  description: A re-write of the GNU cut command with many new features.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-xcut\n  section: science\n  version: 1.1-6\nbio-linux-zsh:\n  description: Z-Shell wrapper package for Bio-Linux.\n  downloads: 0\n  homepage_uri: ''\n  name: bio-linux-zsh\n  section: science\n  version: 1-18\nblast2:\n  description: Basic Local Alignment Search Tool\n  downloads: 64\n  homepage_uri: http://www.ncbi.nih.gov/BLAST/\n  name: blast2\n  section: science\n  version: 1:2.2.25.20110713-3ubuntu2\nblcr-util:\n  description: Userspace tools to Checkpoint and Restart Linux processes\n  downloads: 214\n  homepage_uri: https://ftg.lbl.gov/CheckpointRestart/CheckpointRestart.shtml\n  name: blcr-util\n  section: universe/misc\n  version: 0.8.2-15ubuntu2.1\nbluez:\n  description: Bluetooth tools and daemons\n  downloads: 36308\n  homepage_uri: http://www.bluez.org\n  name: bluez\n  section: admin\n  version: 4.98-2ubuntu7\nbluez-alsa:\n  description: Bluetooth ALSA support\n  downloads: 197\n  homepage_uri: http://www.bluez.org\n  name: bluez-alsa\n  section: admin\n  version: 4.98-2ubuntu7\nbluez-alsa:i386:\n  description: Bluetooth ALSA support\n  downloads: 0\n  homepage_uri: http://www.bluez.org\n  name: bluez-alsa:i386\n  section: admin\n  version: 4.98-2ubuntu7\nbluez-cups:\n  description: Bluetooth printer driver for CUPS\n  downloads: 2273\n  homepage_uri: http://www.bluez.org\n  name: bluez-cups\n  section: admin\n  version: 4.98-2ubuntu7\nbluez-gstreamer:\n  description: Bluetooth GStreamer support\n  downloads: 5056\n  homepage_uri: http://www.bluez.org\n  name: bluez-gstreamer\n  section: admin\n  version: 4.98-2ubuntu7\nbranding-ubuntu:\n  description: Replacement artwork with Ubuntu branding\n  downloads: 0\n  homepage_uri: https://wiki.ubuntu.com/branding\n  name: branding-ubuntu\n  section: graphics\n  version: '0.7'\nbrasero:\n  description: CD/DVD burning application for GNOME\n  downloads: 26866\n  homepage_uri: http://www.gnome.org/projects/brasero/\n  name: brasero\n  section: gnome\n  version: 3.4.1-0ubuntu1\nbrasero-cdrkit:\n  description: cdrkit extensions for the Brasero burning application\n  downloads: 624\n  homepage_uri: http://www.gnome.org/projects/brasero/\n  name: brasero-cdrkit\n  section: gnome\n  version: 3.4.1-0ubuntu1\nbrasero-common:\n  description: Common files for the Brasero CD burning application and library\n  downloads: 12343\n  homepage_uri: http://www.gnome.org/projects/brasero/\n  name: brasero-common\n  section: gnome\n  version: 3.4.1-0ubuntu1\nbridge-utils:\n  description: Utilities for configuring the Linux Ethernet bridge\n  downloads: 6694\n  homepage_uri: ''\n  name: bridge-utils\n  section: net\n  version: 1.5-2ubuntu6\nbrltty:\n  description: Access software for a blind person using a braille display\n  downloads: 2336\n  homepage_uri: http://mielke.cc/brltty/\n  name: brltty\n  section: admin\n  version: 4.3-1ubuntu5\nbsdmainutils:\n  description: collection of more utilities from FreeBSD\n  downloads: 105954\n  homepage_uri: ''\n  name: bsdmainutils\n  section: utils\n  version: 8.2.3ubuntu1\nbsdutils:\n  description: Basic utilities from 4.4BSD-Lite\n  downloads: 100492\n  homepage_uri: http://userweb.kernel.org/~kzak/util-linux/\n  name: bsdutils\n  section: utils\n  version: 1:2.20.1-1ubuntu3\nbsh:\n  description: Java scripting environment (BeanShell) Version 2\n  downloads: 2545\n  homepage_uri: http://www.beanshell.org/\n  name: bsh\n  section: devel\n  version: 2.0b4-12build1\nbsh-gcj:\n  description: Java scripting environment (BeanShell) Version 2 (native code)\n  downloads: 642\n  homepage_uri: http://www.beanshell.org/\n  name: bsh-gcj\n  section: libs\n  version: 2.0b4-12build1\nbuild-essential:\n  description: Informational list of build-essential packages\n  downloads: 1485\n  homepage_uri: ''\n  name: build-essential\n  section: devel\n  version: 11.5ubuntu2\nbusybox-initramfs:\n  description: Standalone shell setup for initramfs\n  downloads: 29\n  homepage_uri: http://www.busybox.net\n  name: busybox-initramfs\n  section: shells\n  version: 1:1.18.5-1ubuntu4\nbusybox-static:\n  description: Standalone rescue shell with tons of builtin utilities\n  downloads: 119\n  homepage_uri: http://www.busybox.net\n  name: busybox-static\n  section: shells\n  version: 1:1.18.5-1ubuntu4\nbyobu:\n  description: powerful, text based window manager and shell multiplexer\n  downloads: 475\n  homepage_uri: http://launchpad.net/byobu\n  name: byobu\n  section: misc\n  version: 5.17-0ubuntu1\nbzip2:\n  description: high-quality block-sorting file compressor - utilities\n  downloads: 61069\n  homepage_uri: http://www.bzip.org/\n  name: bzip2\n  section: utils\n  version: 1.0.6-1\nbzr:\n  description: easy to use distributed version control system\n  downloads: 1101\n  homepage_uri: http://bazaar-vcs.org\n  name: bzr\n  section: devel\n  version: 2.5.0-2ubuntu2\nc2hs:\n  description: C->Haskell Interface Generator\n  downloads: 36\n  homepage_uri: http://hackage.haskell.org/package/c2hs\n  name: c2hs\n  section: universe/devel\n  version: 0.16.3-2\nca-certificates:\n  description: Common CA certificates\n  downloads: 32260\n  homepage_uri: ''\n  name: ca-certificates\n  section: misc\n  version: '20111211'\nca-certificates-java:\n  description: Common CA certificates (JKS keystore)\n  downloads: 12396\n  homepage_uri: ''\n  name: ca-certificates-java\n  section: misc\n  version: 20110912ubuntu6\ncabal-install:\n  description: command-line interface for Cabal and Hackage\n  downloads: 145\n  homepage_uri: http://www.haskell.org/cabal/\n  name: cabal-install\n  section: universe/haskell\n  version: 0.14.0-2\ncdbs:\n  description: common build system for Debian packages\n  downloads: 620\n  homepage_uri: ''\n  name: cdbs\n  section: devel\n  version: 0.4.100ubuntu2\ncgroup-lite:\n  description: Light-weight package to set up cgroups at system boot\n  downloads: 0\n  homepage_uri: ''\n  name: cgroup-lite\n  section: admin\n  version: '1.1'\ncheckbox:\n  description: System testing application\n  downloads: 0\n  homepage_uri: ''\n  name: checkbox\n  section: python\n  version: 0.13.7\ncheckbox-qt:\n  description: QT4 interface for checkbox\n  downloads: 0\n  homepage_uri: ''\n  name: checkbox-qt\n  section: utils\n  version: 0.13.7\nclojure:\n  description: a Lisp dialect for the JVM\n  downloads: 20\n  homepage_uri: http://clojure.org\n  name: clojure\n  section: universe/devel\n  version: 1.1.0+dfsg-3\ncloud-init:\n  description: Init scripts for cloud instances\n  downloads: 1\n  homepage_uri: ''\n  name: cloud-init\n  section: admin\n  version: 0.6.3-0ubuntu1\ncloud-initramfs-growroot:\n  description: automatically resize the root partition on first boot\n  downloads: 0\n  homepage_uri: ''\n  name: cloud-initramfs-growroot\n  section: admin\n  version: 0.4ubuntu1\ncloud-initramfs-rescuevol:\n  description: boot off a rescue volume rather than root filesystem\n  downloads: 0\n  homepage_uri: ''\n  name: cloud-initramfs-rescuevol\n  section: admin\n  version: 0.4ubuntu1\ncloud-utils:\n  description: cloud image management utilities\n  downloads: 1\n  homepage_uri: ''\n  name: cloud-utils\n  section: admin\n  version: 0.25-0ubuntu5\ncloudbl-desktop:\n  description: Adds useful links to the remote desktop for Cloud BL\n  downloads: 0\n  homepage_uri: ''\n  name: cloudbl-desktop\n  section: science\n  version: '1.4'\nclustalw:\n  description: global multiple nucleotide or peptide sequence alignment\n  downloads: 40\n  homepage_uri: http://www.clustal.org/\n  name: clustalw\n  section: non-free/science\n  version: 2.1+lgpl-2\nclustalx:\n  description: Multiple alignment of nucleic acid and protein sequences (graphical\n    interface)\n  downloads: 96\n  homepage_uri: ftp://ftp.ebi.ac.uk/pub/software/unix/clustalx/\n  name: clustalx\n  section: non-free/science\n  version: 2.1+lgpl-2\ncmake:\n  description: cross-platform, open-source make system\n  downloads: 1799\n  homepage_uri: http://cmake.org/\n  name: cmake\n  section: devel\n  version: 2.8.7-0ubuntu4\ncmake-data:\n  description: CMake data files (modules, templates and documentation)\n  downloads: 1153\n  homepage_uri: http://cmake.org/\n  name: cmake-data\n  section: devel\n  version: 2.8.7-0ubuntu4\ncmap-adobe-japan1:\n  description: CMaps for Adobe-Japan1\n  downloads: 1870\n  homepage_uri: http://opensource.adobe.com/wiki/display/cmap/CMap+Resources\n  name: cmap-adobe-japan1\n  section: misc\n  version: 0+20090930-2\ncmap-adobe-japan2:\n  description: CMaps for Adobe-Japan2\n  downloads: 0\n  homepage_uri: http://opensource.adobe.com/wiki/display/cmap/CMap+Resources\n  name: cmap-adobe-japan2\n  section: misc\n  version: 0+20090930-2\ncolord:\n  description: system service to manage device colour profiles -- system daemon\n  downloads: 12797\n  homepage_uri: http://www.freedesktop.org/software/colord/\n  name: colord\n  section: graphics\n  version: 0.1.16-2\ncomerr-dev:\n  description: common error description library - headers and static libraries\n  downloads: 12043\n  homepage_uri: http://e2fsprogs.sourceforge.net\n  name: comerr-dev\n  section: libdevel\n  version: 2.1-1.42-1ubuntu2\ncommand-not-found:\n  description: Suggest installation of packages in interactive bash sessions\n  downloads: 640\n  homepage_uri: ''\n  name: command-not-found\n  section: admin\n  version: 0.2.46ubuntu6\ncommand-not-found-data:\n  description: Set of data files for command-not-found.\n  downloads: 0\n  homepage_uri: ''\n  name: command-not-found-data\n  section: admin\n  version: 0.2.46ubuntu6\ncompiz:\n  description: OpenGL window and compositing manager\n  downloads: 1\n  homepage_uri: ''\n  name: compiz\n  section: x11\n  version: 1:0.9.7.8-0ubuntu1\ncompiz-core:\n  description: OpenGL window and compositing manager\n  downloads: 2258\n  homepage_uri: ''\n  name: compiz-core\n  section: x11\n  version: 1:0.9.7.8-0ubuntu1\ncompiz-gnome:\n  description: OpenGL window and compositing manager - GNOME window decorator\n  downloads: 465\n  homepage_uri: ''\n  name: compiz-gnome\n  section: x11\n  version: 1:0.9.7.8-0ubuntu1\ncompiz-plugins-default:\n  description: OpenGL window and compositing manager - default plugins\n  downloads: 1\n  homepage_uri: ''\n  name: compiz-plugins-default\n  section: x11\n  version: 1:0.9.7.8-0ubuntu1\ncompiz-plugins-main-default:\n  description: Compiz plugins - main default collection\n  downloads: 1\n  homepage_uri: ''\n  name: compiz-plugins-main-default\n  section: x11\n  version: 1:0.9.7.0~bzr19-0ubuntu10\ncompizconfig-backend-gconf:\n  description: Compiz Fusion configuration system - gconf backend\n  downloads: 617\n  homepage_uri: ''\n  name: compizconfig-backend-gconf\n  section: libs\n  version: 0.9.5.92-0ubuntu5\nconsole-setup:\n  description: console font and keymap setup program\n  downloads: 49561\n  homepage_uri: ''\n  name: console-setup\n  section: utils\n  version: 1.70ubuntu5\nconsolekit:\n  description: framework for defining and tracking users, sessions and seats\n  downloads: 52291\n  homepage_uri: http://www.freedesktop.org/wiki/Software/ConsoleKit\n  name: consolekit\n  section: admin\n  version: 0.4.5-2\ncoreutils:\n  description: GNU core utilities\n  downloads: 110992\n  homepage_uri: http://gnu.org/software/coreutils\n  name: coreutils\n  section: utils\n  version: 8.13-3ubuntu3\ncouchdb:\n  description: RESTful document oriented database, system DB\n  downloads: 207\n  homepage_uri: http://couchdb.apache.org/\n  name: couchdb\n  section: universe/misc\n  version: 1.0.1-0ubuntu18\ncouchdb-bin:\n  description: RESTful document oriented database, programs\n  downloads: 8\n  homepage_uri: http://couchdb.apache.org/\n  name: couchdb-bin\n  section: universe/misc\n  version: 1.0.1-0ubuntu18\ncpio:\n  description: GNU cpio -- a program to manage archives of files\n  downloads: 51114\n  homepage_uri: http://www.gnu.org/software/cpio/\n  name: cpio\n  section: utils\n  version: 2.11-7ubuntu3\ncpp:\n  description: GNU C preprocessor (cpp)\n  downloads: 48670\n  homepage_uri: ''\n  name: cpp\n  section: interpreters\n  version: 4:4.6.3-1ubuntu5\ncpp-4.6:\n  description: GNU C preprocessor\n  downloads: 13427\n  homepage_uri: http://gcc.gnu.org/\n  name: cpp-4.6\n  section: interpreters\n  version: 4.6.3-1ubuntu5\ncpu-checker:\n  description: tools to help evaluate certain CPU (or BIOS) features\n  downloads: 22\n  homepage_uri: https://launchpad.net/cpu-checker\n  name: cpu-checker\n  section: utils\n  version: 0.7-0ubuntu1\ncrda:\n  description: wireless Central Regulatory Domain Agent\n  downloads: 9531\n  homepage_uri: http://wireless.kernel.org/en/developers/Regulatory/#CRDA\n  name: crda\n  section: net\n  version: 1.1.2-1ubuntu1\ncron:\n  description: process scheduling daemon\n  downloads: 110008\n  homepage_uri: http://ftp.isc.org/isc/cron/\n  name: cron\n  section: admin\n  version: 3.0pl1-120ubuntu3\ncryptsetup-bin:\n  description: disk encryption support - command line tools\n  downloads: 2319\n  homepage_uri: http://code.google.com/p/cryptsetup/\n  name: cryptsetup-bin\n  section: admin\n  version: 2:1.4.1-2ubuntu4\ncups:\n  description: Common UNIX Printing System(tm) - server\n  downloads: 44997\n  homepage_uri: http://www.cups.org\n  name: cups\n  section: net\n  version: 1.5.3-0ubuntu1\ncups-bsd:\n  description: Common UNIX Printing System(tm) - BSD commands\n  downloads: 9338\n  homepage_uri: http://www.cups.org\n  name: cups-bsd\n  section: net\n  version: 1.5.3-0ubuntu1\ncups-client:\n  description: Common UNIX Printing System(tm) - client programs (SysV)\n  downloads: 18378\n  homepage_uri: http://www.cups.org\n  name: cups-client\n  section: net\n  version: 1.5.3-0ubuntu1\ncups-common:\n  description: Common UNIX Printing System(tm) - common files\n  downloads: 17924\n  homepage_uri: http://www.cups.org\n  name: cups-common\n  section: net\n  version: 1.5.3-0ubuntu1\ncups-filters:\n  description: OpenPrinting CUPS Filters\n  downloads: 5104\n  homepage_uri: http://www.openprinting.org/\n  name: cups-filters\n  section: net\n  version: 1.0.18-0ubuntu0.1\ncups-pk-helper:\n  description: PolicyKit helper to configure cups with fine-grained privileges\n  downloads: 3113\n  homepage_uri: http://www.freedesktop.org/software/cups-pk-helper/\n  name: cups-pk-helper\n  section: universe/gnome\n  version: 0.2.1.2-1\ncups-ppdc:\n  description: Common UNIX Printing System(tm) - PPD manipulation utilities\n  downloads: 7732\n  homepage_uri: http://www.cups.org\n  name: cups-ppdc\n  section: net\n  version: 1.5.3-0ubuntu1\ncurl:\n  description: Get a file from an HTTP, HTTPS or FTP server\n  downloads: 10344\n  homepage_uri: http://curl.haxx.se\n  name: curl\n  section: web\n  version: 7.22.0-3ubuntu4\ncvs:\n  description: Concurrent Versions System\n  downloads: 2449\n  homepage_uri: http://www.nongnu.org/cvs/\n  name: cvs\n  section: devel\n  version: 2:1.12.13+real-8\ndarcs:\n  description: distributed, interactive, smart revision control system\n  downloads: 243\n  homepage_uri: http://darcs.net/\n  name: darcs\n  section: universe/devel\n  version: 2.7.99.1-1\ndash:\n  description: POSIX-compliant shell\n  downloads: 92934\n  homepage_uri: http://gondor.apana.org.au/~herbert/dash/\n  name: dash\n  section: shells\n  version: 0.5.7-2ubuntu2\ndbus:\n  description: simple interprocess messaging system (daemon and utilities)\n  downloads: 66602\n  homepage_uri: http://dbus.freedesktop.org/\n  name: dbus\n  section: devel\n  version: 1.4.18-1ubuntu1\ndbus-x11:\n  description: simple interprocess messaging system (X11 deps)\n  downloads: 48441\n  homepage_uri: http://dbus.freedesktop.org/\n  name: dbus-x11\n  section: x11\n  version: 1.4.18-1ubuntu1\ndc:\n  description: The GNU dc arbitrary precision reverse-polish calculator\n  downloads: 14520\n  homepage_uri: ''\n  name: dc\n  section: math\n  version: 1.06.95-2\ndconf-gsettings-backend:\n  description: simple configuration storage system - GSettings back-end\n  downloads: 14720\n  homepage_uri: http://live.gnome.org/dconf\n  name: dconf-gsettings-backend\n  section: libs\n  version: 0.12.0-0ubuntu1\ndconf-service:\n  description: simple configuration storage system - D-Bus service\n  downloads: 13671\n  homepage_uri: http://live.gnome.org/dconf\n  name: dconf-service\n  section: libs\n  version: 0.12.0-0ubuntu1\ndctrl-tools:\n  description: Command-line tools to process Debian package information\n  downloads: 4785\n  homepage_uri: ''\n  name: dctrl-tools\n  section: utils\n  version: 2.18ubuntu1\ndebconf:\n  description: Debian configuration management system\n  downloads: 111132\n  homepage_uri: ''\n  name: debconf\n  section: admin\n  version: 1.5.42ubuntu1\ndebconf-i18n:\n  description: full internationalization support for debconf\n  downloads: 864\n  homepage_uri: ''\n  name: debconf-i18n\n  section: admin\n  version: 1.5.42ubuntu1\ndebhelper:\n  description: helper programs for debian/rules\n  downloads: 5177\n  homepage_uri: http://kitenet.net/~joey/code/debhelper/\n  name: debhelper\n  section: devel\n  version: 9.20120115ubuntu3\ndebianutils:\n  description: Miscellaneous utilities specific to Debian\n  downloads: 116783\n  homepage_uri: ''\n  name: debianutils\n  section: utils\n  version: 4.2.1ubuntu2\ndebootstrap:\n  description: Bootstrap a basic Debian system\n  downloads: 1603\n  homepage_uri: ''\n  name: debootstrap\n  section: admin\n  version: 1.0.39ubuntu0.1\ndefault-jdk:\n  description: Standard Java or Java compatible Development Kit\n  downloads: 45\n  homepage_uri: ''\n  name: default-jdk\n  section: devel\n  version: 1:1.6-43ubuntu2\ndefault-jre:\n  description: Standard Java or Java compatible Runtime\n  downloads: 12183\n  homepage_uri: ''\n  name: default-jre\n  section: interpreters\n  version: 1:1.6-43ubuntu2\ndefault-jre-headless:\n  description: Standard Java or Java compatible Runtime (headless)\n  downloads: 13149\n  homepage_uri: ''\n  name: default-jre-headless\n  section: interpreters\n  version: 1:1.6-43ubuntu2\ndeja-dup:\n  description: Back up your files\n  downloads: 345\n  homepage_uri: https://launchpad.net/deja-dup\n  name: deja-dup\n  section: utils\n  version: 22.0-0ubuntu2\ndesktop-file-utils:\n  description: Utilities for .desktop files\n  downloads: 37457\n  homepage_uri: ''\n  name: desktop-file-utils\n  section: devel\n  version: 0.20-0ubuntu2\ndevscripts:\n  description: scripts to make the life of a Debian Package maintainer easier\n  downloads: 1921\n  homepage_uri: ''\n  name: devscripts\n  section: devel\n  version: 2.11.6ubuntu1.2\ndh-apparmor:\n  description: AppArmor debhelper routines\n  downloads: 1\n  homepage_uri: http://apparmor.net/\n  name: dh-apparmor\n  section: devel\n  version: 2.7.102-0ubuntu3.1\ndh-make:\n  description: tool that converts source archives into Debian package source\n  downloads: 520\n  homepage_uri: ''\n  name: dh-make\n  section: devel\n  version: 0.59ubuntu1\ndh-translations:\n  description: debhelper extension for translation support\n  downloads: 1\n  homepage_uri: ''\n  name: dh-translations\n  section: devel\n  version: '116'\ndialign:\n  description: Segment-based multiple sequence alignment\n  downloads: 31\n  homepage_uri: http://dialign.gobics.de/\n  name: dialign\n  section: science\n  version: 2.2.1-5\ndict:\n  description: dictionary client\n  downloads: 892\n  homepage_uri: http://sourceforge.net/projects/dict/\n  name: dict\n  section: text\n  version: 1.12.0+dfsg-4build1\ndictionaries-common:\n  description: Common utilities for spelling dictionary tools\n  downloads: 18912\n  homepage_uri: http://dict-common.alioth.debian.org\n  name: dictionaries-common\n  section: text\n  version: 1.12.1ubuntu2\ndiffstat:\n  description: produces graph of changes introduced by a diff file\n  downloads: 2366\n  homepage_uri: http://invisible-island.net/diffstat/\n  name: diffstat\n  section: devel\n  version: 1.54-1\ndiffutils:\n  description: File comparison utilities\n  downloads: 86627\n  homepage_uri: http://www.gnu.org/software/diffutils/\n  name: diffutils\n  section: utils\n  version: 1:3.2-1ubuntu1\ndkms:\n  description: Dynamic Kernel Module Support Framework\n  downloads: 9246\n  homepage_uri: http://linux.dell.com/dkms\n  name: dkms\n  section: admin\n  version: 2.2.0.3-1ubuntu3\ndmidecode:\n  description: SMBIOS/DMI table decoder\n  downloads: 14886\n  homepage_uri: http://dmidecode.nongnu.org/\n  name: dmidecode\n  section: utils\n  version: 2.11-4\ndmsetup:\n  description: The Linux Kernel Device Mapper userspace library\n  downloads: 63207\n  homepage_uri: http://sources.redhat.com/lvm2/\n  name: dmsetup\n  section: admin\n  version: 2:1.02.48-4ubuntu7.1\ndmz-cursor-theme:\n  description: Style neutral, scalable cursor theme\n  downloads: 0\n  homepage_uri: ''\n  name: dmz-cursor-theme\n  section: gnome\n  version: 0.4.3\ndnsmasq-base:\n  description: Small caching DNS proxy and DHCP/TFTP server\n  downloads: 3857\n  homepage_uri: ''\n  name: dnsmasq-base\n  section: net\n  version: 2.59-4\ndnsutils:\n  description: Clients provided with BIND\n  downloads: 14967\n  homepage_uri: ''\n  name: dnsutils\n  section: net\n  version: 1:9.8.1.dfsg.P1-4ubuntu0.1\ndoc-base:\n  description: utilities to manage online documentation\n  downloads: 7963\n  homepage_uri: ''\n  name: doc-base\n  section: doc\n  version: 0.10.3\ndocbook-xml:\n  description: standard XML documentation system for software and systems\n  downloads: 679\n  homepage_uri: http://docbook.org/schemas/4x.html\n  name: docbook-xml\n  section: text\n  version: 4.5-7ubuntu1\ndosfstools:\n  description: utilities for making and checking MS-DOS FAT filesystems\n  downloads: 5523\n  homepage_uri: http://www.daniel-baumann.ch/software/dosfstools/\n  name: dosfstools\n  section: otherosfs\n  version: 3.0.12-1ubuntu1\ndpatch:\n  description: patch maintenance system for Debian source packages\n  downloads: 869\n  homepage_uri: ''\n  name: dpatch\n  section: devel\n  version: 2.0.34ubuntu1\ndpkg:\n  description: Debian package management system\n  downloads: 116788\n  homepage_uri: http://wiki.debian.org/Teams/Dpkg\n  name: dpkg\n  section: admin\n  version: 1.16.1.2ubuntu7\ndpkg-dev:\n  description: Debian package development tools\n  downloads: 7479\n  homepage_uri: http://wiki.debian.org/Teams/Dpkg\n  name: dpkg-dev\n  section: utils\n  version: 1.16.1.2ubuntu7\ndput:\n  description: Debian package upload tool\n  downloads: 1149\n  homepage_uri: ''\n  name: dput\n  section: devel\n  version: 0.9.6.2ubuntu1\nduplicity:\n  description: encrypted bandwidth-efficient backup\n  downloads: 907\n  homepage_uri: http://duplicity.nongnu.org/\n  name: duplicity\n  section: utils\n  version: 0.6.18-0ubuntu3\ndvd+rw-tools:\n  description: DVD+-RW/R tools\n  downloads: 12960\n  homepage_uri: http://fy.chalmers.se/~appro/linux/DVD+RW/\n  name: dvd+rw-tools\n  section: utils\n  version: 7.1-10\ne2fslibs:\n  description: ext2/ext3/ext4 file system libraries\n  downloads: 19465\n  homepage_uri: http://e2fsprogs.sourceforge.net\n  name: e2fslibs\n  section: libs\n  version: 1.42-1ubuntu2\ne2fsprogs:\n  description: ext2/ext3/ext4 file system utilities\n  downloads: 73393\n  homepage_uri: http://e2fsprogs.sourceforge.net\n  name: e2fsprogs\n  section: admin\n  version: 1.42-1ubuntu2\nebtables:\n  description: Ethernet bridge frame table administration\n  downloads: 3094\n  homepage_uri: http://ebtables.sourceforge.net\n  name: ebtables\n  section: net\n  version: 2.0.9.2-2ubuntu2\nec2-ami-tools:\n  description: Amazon EC2 AMI tools\n  downloads: 2\n  homepage_uri: http://aws.amazon.com/developertools/368\n  name: ec2-ami-tools\n  section: multiverse/admin\n  version: 1.4.0.2-0ubuntu1\nec2-api-tools:\n  description: Amazon EC2 API tools\n  downloads: 12\n  homepage_uri: http://aws.amazon.com/developertools/351\n  name: ec2-api-tools\n  section: multiverse/admin\n  version: 1.5.0.0-0ubuntu1\neclipse:\n  description: Extensible Tool Platform and Java IDE\n  downloads: 15\n  homepage_uri: http://www.eclipse.org/\n  name: eclipse\n  section: universe/devel\n  version: 3.7.2-1\neclipse-jdt:\n  description: Eclipse Java Development Tools (JDT)\n  downloads: 967\n  homepage_uri: http://www.eclipse.org/\n  name: eclipse-jdt\n  section: universe/devel\n  version: 3.7.2-1\neclipse-pde:\n  description: Eclipse Plug-in Development Environment (PDE)\n  downloads: 985\n  homepage_uri: http://www.eclipse.org/\n  name: eclipse-pde\n  section: universe/devel\n  version: 3.7.2-1\neclipse-platform:\n  description: Eclipse platform without development plug-ins\n  downloads: 1691\n  homepage_uri: http://www.eclipse.org/\n  name: eclipse-platform\n  section: universe/devel\n  version: 3.7.2-1\neclipse-platform-data:\n  description: Eclipse platform without development plug-ins (data)\n  downloads: 1681\n  homepage_uri: http://www.eclipse.org/\n  name: eclipse-platform-data\n  section: universe/java\n  version: 3.7.2-1\neclipse-rcp:\n  description: Eclipse Rich Client Platform (RCP)\n  downloads: 1651\n  homepage_uri: http://www.eclipse.org/\n  name: eclipse-rcp\n  section: universe/devel\n  version: 3.7.2-1\ned:\n  description: classic UNIX line editor\n  downloads: 8485\n  homepage_uri: http://www.gnu.org/software/ed/\n  name: ed\n  section: editors\n  version: 1.5-3\neject:\n  description: ejects CDs and operates CD-Changers under Linux\n  downloads: 19162\n  homepage_uri: http://www.pobox.com/~tranter/eject.html\n  name: eject\n  section: utils\n  version: 2.1.5+deb1+cvs20081104-9\nemacs:\n  description: The GNU Emacs editor (metapackage)\n  downloads: 205\n  homepage_uri: http://www.gnu.org/software/emacs/\n  name: emacs\n  section: editors\n  version: 23.3+1-1ubuntu9\nemacs-goodies-el:\n  description: Miscellaneous add-ons for Emacs\n  downloads: 319\n  homepage_uri: ''\n  name: emacs-goodies-el\n  section: editors\n  version: 35.2ubuntu1.1\nemacs23:\n  description: The GNU Emacs editor (with GTK+ user interface)\n  downloads: 6479\n  homepage_uri: http://www.gnu.org/software/emacs/\n  name: emacs23\n  section: editors\n  version: 23.3+1-1ubuntu9\nemacs23-bin-common:\n  description: The GNU Emacs editor's shared, architecture dependent files\n  downloads: 7524\n  homepage_uri: http://www.gnu.org/software/emacs/\n  name: emacs23-bin-common\n  section: editors\n  version: 23.3+1-1ubuntu9\nemacs23-common:\n  description: The GNU Emacs editor's shared, architecture independent infrastructure\n  downloads: 2045\n  homepage_uri: http://www.gnu.org/software/emacs/\n  name: emacs23-common\n  section: editors\n  version: 23.3+1-1ubuntu9\nemacsen-common:\n  description: Common facilities for all emacsen\n  downloads: 6504\n  homepage_uri: ''\n  name: emacsen-common\n  section: editors\n  version: 1.4.22ubuntu1\nemboss:\n  description: the european molecular biology open software suite\n  downloads: 45\n  homepage_uri: http://emboss.sourceforge.net\n  name: emboss\n  section: science\n  version: 6.3.1-6ubuntu3\nemboss-data:\n  description: data files for the EMBOSS package\n  downloads: 1\n  homepage_uri: http://emboss.sourceforge.net\n  name: emboss-data\n  section: science\n  version: 6.3.1-6ubuntu3\nemboss-lib:\n  description: EMBOSS Libraries\n  downloads: 46\n  homepage_uri: http://emboss.sourceforge.net\n  name: emboss-lib\n  section: libs\n  version: 6.3.1-6ubuntu3\nempathy:\n  description: GNOME multi-protocol chat and call client\n  downloads: 4159\n  homepage_uri: http://live.gnome.org/Empathy\n  name: empathy\n  section: gnome\n  version: 3.4.2-0ubuntu1\nempathy-common:\n  description: GNOME multi-protocol chat and call client (common files)\n  downloads: 0\n  homepage_uri: http://live.gnome.org/Empathy\n  name: empathy-common\n  section: gnome\n  version: 3.4.2-0ubuntu1\nenchant:\n  description: Wrapper for various spell checker engines (binary programs)\n  downloads: 1688\n  homepage_uri: http://www.abisource.com/projects/enchant/\n  name: enchant\n  section: text\n  version: 1.6.0-7\neog:\n  description: Eye of GNOME graphics viewer program\n  downloads: 17636\n  homepage_uri: http://projects.gnome.org/eog\n  name: eog\n  section: gnome\n  version: 3.4.1-0ubuntu1\nerlang:\n  description: Concurrent, real-time, distributed functional language\n  downloads: 0\n  homepage_uri: http://www.erlang.org/\n  name: erlang\n  section: universe/interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-appmon:\n  description: Erlang/OTP application monitor\n  downloads: 36\n  homepage_uri: http://www.erlang.org/\n  name: erlang-appmon\n  section: universe/interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-asn1:\n  description: Erlang/OTP modules for ASN.1 support\n  downloads: 494\n  homepage_uri: http://www.erlang.org/\n  name: erlang-asn1\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-base:\n  description: Erlang/OTP virtual machine and base applications\n  downloads: 1098\n  homepage_uri: http://www.erlang.org/\n  name: erlang-base\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-common-test:\n  description: Erlang/OTP application for automated testing\n  downloads: 71\n  homepage_uri: http://www.erlang.org/\n  name: erlang-common-test\n  section: universe/interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-corba:\n  description: Erlang/OTP applications for CORBA support\n  downloads: 95\n  homepage_uri: http://www.erlang.org/\n  name: erlang-corba\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-crypto:\n  description: Erlang/OTP cryptographic modules\n  downloads: 714\n  homepage_uri: http://www.erlang.org/\n  name: erlang-crypto\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-debugger:\n  description: Erlang/OTP application for debugging and testing\n  downloads: 31\n  homepage_uri: http://www.erlang.org/\n  name: erlang-debugger\n  section: universe/interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-dev:\n  description: Erlang/OTP development libraries and headers\n  downloads: 136\n  homepage_uri: http://www.erlang.org/\n  name: erlang-dev\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-dialyzer:\n  description: Erlang/OTP discrepancy analyzer application\n  downloads: 72\n  homepage_uri: http://www.erlang.org/\n  name: erlang-dialyzer\n  section: universe/interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-diameter:\n  description: Erlang/OTP implementation of RFC 3588 protocol\n  downloads: 61\n  homepage_uri: http://www.erlang.org/\n  name: erlang-diameter\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-docbuilder:\n  description: Erlang/OTP application for building HTML documentation\n  downloads: 32\n  homepage_uri: http://www.erlang.org/\n  name: erlang-docbuilder\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-edoc:\n  description: Erlang/OTP module for generating documentation\n  downloads: 103\n  homepage_uri: http://www.erlang.org/\n  name: erlang-edoc\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-erl-docgen:\n  description: Erlang/OTP documentation stylesheets\n  downloads: 95\n  homepage_uri: http://www.erlang.org/\n  name: erlang-erl-docgen\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-et:\n  description: Erlang/OTP event tracer application\n  downloads: 32\n  homepage_uri: http://www.erlang.org/\n  name: erlang-et\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-eunit:\n  description: Erlang/OTP module for unit testing\n  downloads: 120\n  homepage_uri: http://www.erlang.org/\n  name: erlang-eunit\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-examples:\n  description: Erlang/OTP application examples\n  downloads: 27\n  homepage_uri: http://www.erlang.org/\n  name: erlang-examples\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-gs:\n  description: Erlang/OTP graphics system\n  downloads: 41\n  homepage_uri: http://www.erlang.org/\n  name: erlang-gs\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-ic:\n  description: Erlang/OTP IDL compiler\n  downloads: 94\n  homepage_uri: http://www.erlang.org/\n  name: erlang-ic\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-ic-java:\n  description: Erlang/OTP IDL compiler (Java classes)\n  downloads: 22\n  homepage_uri: http://www.erlang.org/\n  name: erlang-ic-java\n  section: universe/interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-inets:\n  description: Erlang/OTP Internet clients and servers\n  downloads: 426\n  homepage_uri: http://www.erlang.org/\n  name: erlang-inets\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-inviso:\n  description: Erlang/OTP trace tool\n  downloads: 92\n  homepage_uri: http://www.erlang.org/\n  name: erlang-inviso\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-jinterface:\n  description: Java communication tool to Erlang\n  downloads: 25\n  homepage_uri: http://www.erlang.org/\n  name: erlang-jinterface\n  section: universe/interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-manpages:\n  description: Erlang/OTP manual pages\n  downloads: 0\n  homepage_uri: http://www.erlang.org/\n  name: erlang-manpages\n  section: universe/interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-megaco:\n  description: Erlang/OTP implementation of Megaco/H.248 protocol\n  downloads: 32\n  homepage_uri: http://www.erlang.org/\n  name: erlang-megaco\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-mnesia:\n  description: Erlang/OTP distributed relational/object hybrid database\n  downloads: 530\n  homepage_uri: http://www.erlang.org/\n  name: erlang-mnesia\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-mode:\n  description: Erlang major editing mode for Emacs\n  downloads: 60\n  homepage_uri: http://www.erlang.org/\n  name: erlang-mode\n  section: universe/interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-nox:\n  description: Erlang/OTP applications that don't require X Window System\n  downloads: 213\n  homepage_uri: http://www.erlang.org/\n  name: erlang-nox\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-observer:\n  description: Erlang/OTP application for investigating distributed systems\n  downloads: 34\n  homepage_uri: http://www.erlang.org/\n  name: erlang-observer\n  section: universe/interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-odbc:\n  description: Erlang/OTP interface to SQL databases\n  downloads: 131\n  homepage_uri: http://www.erlang.org/\n  name: erlang-odbc\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-os-mon:\n  description: Erlang/OTP operating system monitor\n  downloads: 246\n  homepage_uri: http://www.erlang.org/\n  name: erlang-os-mon\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-parsetools:\n  description: Erlang/OTP parsing tools\n  downloads: 94\n  homepage_uri: http://www.erlang.org/\n  name: erlang-parsetools\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-percept:\n  description: Erlang/OTP concurrency profiling tool\n  downloads: 92\n  homepage_uri: http://www.erlang.org/\n  name: erlang-percept\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-pman:\n  description: Erlang/OTP process manager\n  downloads: 32\n  homepage_uri: http://www.erlang.org/\n  name: erlang-pman\n  section: universe/interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-public-key:\n  description: Erlang/OTP public key infrastructure\n  downloads: 528\n  homepage_uri: http://www.erlang.org/\n  name: erlang-public-key\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-reltool:\n  description: Erlang/OTP release management tool\n  downloads: 31\n  homepage_uri: http://www.erlang.org/\n  name: erlang-reltool\n  section: universe/interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-runtime-tools:\n  description: Erlang/OTP runtime tracing/debugging tools\n  downloads: 176\n  homepage_uri: http://www.erlang.org/\n  name: erlang-runtime-tools\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-snmp:\n  description: Erlang/OTP SNMP applications\n  downloads: 221\n  homepage_uri: http://www.erlang.org/\n  name: erlang-snmp\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-src:\n  description: Erlang/OTP applications sources\n  downloads: 36\n  homepage_uri: http://www.erlang.org/\n  name: erlang-src\n  section: universe/interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-ssh:\n  description: Erlang/OTP implementation of SSH protocol\n  downloads: 93\n  homepage_uri: http://www.erlang.org/\n  name: erlang-ssh\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-ssl:\n  description: Erlang/OTP implementation of SSL\n  downloads: 462\n  homepage_uri: http://www.erlang.org/\n  name: erlang-ssl\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-syntax-tools:\n  description: Erlang/OTP modules for handling abstract Erlang syntax trees\n  downloads: 204\n  homepage_uri: http://www.erlang.org/\n  name: erlang-syntax-tools\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-test-server:\n  description: Erlang/OTP server for automated application testing\n  downloads: 32\n  homepage_uri: http://www.erlang.org/\n  name: erlang-test-server\n  section: universe/interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-toolbar:\n  description: Erlang/OTP graphical toolbar\n  downloads: 31\n  homepage_uri: http://www.erlang.org/\n  name: erlang-toolbar\n  section: universe/interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-tools:\n  description: Erlang/OTP various tools\n  downloads: 232\n  homepage_uri: http://www.erlang.org/\n  name: erlang-tools\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-tv:\n  description: Erlang/OTP table viewer\n  downloads: 32\n  homepage_uri: http://www.erlang.org/\n  name: erlang-tv\n  section: universe/interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-typer:\n  description: Erlang/OTP code type annotator\n  downloads: 66\n  homepage_uri: http://www.erlang.org/\n  name: erlang-typer\n  section: universe/interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-webtool:\n  description: Erlang/OTP helper for web-based tools\n  downloads: 161\n  homepage_uri: http://www.erlang.org/\n  name: erlang-webtool\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nerlang-xmerl:\n  description: Erlang/OTP XML tools\n  downloads: 236\n  homepage_uri: http://www.erlang.org/\n  name: erlang-xmerl\n  section: interpreters\n  version: 1:14.b.4-dfsg-1ubuntu1\nesound-common:\n  description: Enlightened Sound Daemon - Common files\n  downloads: 8826\n  homepage_uri: ''\n  name: esound-common\n  section: sound\n  version: 0.2.41-10build3\nespeak:\n  description: Multi-lingual software speech synthesizer\n  downloads: 2513\n  homepage_uri: http://espeak.sourceforge.net/\n  name: espeak\n  section: sound\n  version: 1.46.02-0ubuntu1\nespeak-data:\n  description: 'Multi-lingual software speech synthesizer: speech data files'\n  downloads: 46\n  homepage_uri: http://espeak.sourceforge.net/\n  name: espeak-data\n  section: sound\n  version: 1.46.02-0ubuntu1\neuca2ools:\n  description: managing cloud instances for Eucalyptus\n  downloads: 17\n  homepage_uri: http://open.eucalyptus.com/open/\n  name: euca2ools\n  section: utils\n  version: 2.0.0~bzr516-0ubuntu3.1\nevince:\n  description: Document (PostScript, PDF) viewer\n  downloads: 33662\n  homepage_uri: http://www.gnome.org/projects/evince/\n  name: evince\n  section: gnome\n  version: 3.4.0-0ubuntu1\nevince-common:\n  description: Document (PostScript, PDF) viewer - common files\n  downloads: 0\n  homepage_uri: http://www.gnome.org/projects/evince/\n  name: evince-common\n  section: gnome\n  version: 3.4.0-0ubuntu1\nevolution-data-server:\n  description: evolution database backend server\n  downloads: 21994\n  homepage_uri: http://www.gnome.org/projects/evolution/\n  name: evolution-data-server\n  section: gnome\n  version: 3.2.3-0ubuntu7\nevolution-data-server-common:\n  description: architecture independent files for Evolution Data Server\n  downloads: 72\n  homepage_uri: http://www.gnome.org/projects/evolution/\n  name: evolution-data-server-common\n  section: gnome\n  version: 3.2.3-0ubuntu7\nexample-content:\n  description: Ubuntu example content\n  downloads: 0\n  homepage_uri: https://launchpad.net/example-content\n  name: example-content\n  section: x11\n  version: '46'\nexonerate:\n  description: generic tool for pairwise sequence comparison\n  downloads: 27\n  homepage_uri: http://www.ebi.ac.uk/~guy/exonerate/\n  name: exonerate\n  section: universe/science\n  version: 2.2.0-6\nexpect:\n  description: Automates interactive applications\n  downloads: 4238\n  homepage_uri: http://sourceforge.net/projects/expect/\n  name: expect\n  section: interpreters\n  version: 5.45-2\nexuberant-ctags:\n  description: build tag file indexes of source code definitions\n  downloads: 1238\n  homepage_uri: http://ctags.sourceforge.net/\n  name: exuberant-ctags\n  section: editors\n  version: 1:5.9~svn20110310-3\nfakeroot:\n  description: tool for simulating superuser privileges\n  downloads: 6533\n  homepage_uri: ''\n  name: fakeroot\n  section: utils\n  version: 1.18.2-1\nfastdnaml:\n  description: Tool for construction of phylogenetic trees of DNA sequences\n  downloads: 20\n  homepage_uri: http://web.archive.org/web/20061017161001/http://geta.life.uiuc.edu/~gary/programs/fastDNAml.html\n  name: fastdnaml\n  section: science\n  version: 1.2.2-10\nfastjar:\n  description: Jar creation utility\n  downloads: 1799\n  homepage_uri: ''\n  name: fastjar\n  section: misc\n  version: 2:0.98-3\nfastx-toolkit:\n  description: FASTQ/A short nucleotide reads pre-processing tools\n  downloads: 9\n  homepage_uri: http://hannonlab.cshl.edu/fastx_toolkit/\n  name: fastx-toolkit\n  section: universe/science\n  version: 0.0.13.1-1\nfdupes:\n  description: identifies duplicate files within given directories\n  downloads: 309\n  homepage_uri: http://code.google.com/p/fdupes/\n  name: fdupes\n  section: utils\n  version: 1.50-PR2-3\nfile:\n  description: Determines file type using \"magic\" numbers\n  downloads: 34695\n  homepage_uri: http://www.darwinsys.com/file/\n  name: file\n  section: utils\n  version: 5.09-2\nfile-roller:\n  description: archive manager for GNOME\n  downloads: 29747\n  homepage_uri: http://fileroller.sourceforge.net/\n  name: file-roller\n  section: gnome\n  version: 3.4.1-0ubuntu1\nfindutils:\n  description: utilities for finding files--find, xargs\n  downloads: 110950\n  homepage_uri: http://savannah.gnu.org/projects/findutils/\n  name: findutils\n  section: utils\n  version: 4.4.2-4ubuntu1\nfirefox:\n  description: Safe and easy web browser from Mozilla\n  downloads: 393\n  homepage_uri: ''\n  name: firefox\n  section: web\n  version: 13.0+build1-0ubuntu0.12.04.1\nfirefox-globalmenu:\n  description: Unity appmenu integration for Firefox\n  downloads: 4\n  homepage_uri: ''\n  name: firefox-globalmenu\n  section: web\n  version: 13.0+build1-0ubuntu0.12.04.1\nfirefox-gnome-support:\n  description: Safe and easy web browser from Mozilla - GNOME support\n  downloads: 1\n  homepage_uri: ''\n  name: firefox-gnome-support\n  section: gnome\n  version: 13.0.1+build1-0ubuntu0.12.04.1\nfolks-common:\n  description: library to aggregates people into metacontacts (common files)\n  downloads: 0\n  homepage_uri: http://telepathy.freedesktop.org/wiki/Folks\n  name: folks-common\n  section: libs\n  version: 0.6.8-2\nfontconfig:\n  description: generic font configuration library - support binaries\n  downloads: 55585\n  homepage_uri: ''\n  name: fontconfig\n  section: utils\n  version: 2.8.0-3ubuntu9\nfontconfig-config:\n  description: generic font configuration library - configuration\n  downloads: 54331\n  homepage_uri: ''\n  name: fontconfig-config\n  section: libs\n  version: 2.8.0-3ubuntu9\nfonts-kacst:\n  description: KACST free TrueType Arabic fonts\n  downloads: 0\n  homepage_uri: ''\n  name: fonts-kacst\n  section: fonts\n  version: 2.01+mry-3\nfonts-kacst-one:\n  description: TrueType font designed for Arabic language\n  downloads: 0\n  homepage_uri: http://projects.arabeyes.org\n  name: fonts-kacst-one\n  section: fonts\n  version: 5.0+svn11846-2\nfonts-khmeros-core:\n  description: KhmerOS Unicode fonts for the Khmer language of Cambodia\n  downloads: 0\n  homepage_uri: http://www.khmeros.info\n  name: fonts-khmeros-core\n  section: fonts\n  version: 5.0-5ubuntu1\nfonts-lao:\n  description: TrueType font for Lao language\n  downloads: 0\n  homepage_uri: ''\n  name: fonts-lao\n  section: fonts\n  version: 0.0.20060226-8\nfonts-liberation:\n  description: Fonts with the same metrics as Times, Arial and Courier\n  downloads: 708\n  homepage_uri: https://fedorahosted.org/liberation-fonts/\n  name: fonts-liberation\n  section: fonts\n  version: 1.07.0-2\nfonts-nanum:\n  description: Nanum Korean fonts\n  downloads: 0\n  homepage_uri: http://hangeul.naver.com/nanum.nhn\n  name: fonts-nanum\n  section: fonts\n  version: 3.010-2\nfonts-opensymbol:\n  description: OpenSymbol TrueType font\n  downloads: 0\n  homepage_uri: http://www.libreoffice.org\n  name: fonts-opensymbol\n  section: fonts\n  version: 2:102.2+LibO3.5.3-0ubuntu1\nfonts-takao-pgothic:\n  description: Japanese TrueType font set, Takao P Gothic Fonts\n  downloads: 0\n  homepage_uri: https://launchpad.net/takao-fonts\n  name: fonts-takao-pgothic\n  section: fonts\n  version: 003.02.01-5ubuntu1\nfonts-thai-tlwg:\n  description: Thai fonts maintained by TLWG (meta package)\n  downloads: 52\n  homepage_uri: http://linux.thai.net/projects/thaifonts-scalable\n  name: fonts-thai-tlwg\n  section: fonts\n  version: 1:0.4.17-1ubuntu1\nfonts-tlwg-garuda:\n  description: Thai Garuda font\n  downloads: 0\n  homepage_uri: http://linux.thai.net/projects/thaifonts-scalable\n  name: fonts-tlwg-garuda\n  section: fonts\n  version: 1:0.4.17-1ubuntu1\nfonts-tlwg-kinnari:\n  description: Thai Kinnari font\n  downloads: 0\n  homepage_uri: http://linux.thai.net/projects/thaifonts-scalable\n  name: fonts-tlwg-kinnari\n  section: fonts\n  version: 1:0.4.17-1ubuntu1\nfonts-tlwg-loma:\n  description: Thai Loma font\n  downloads: 0\n  homepage_uri: http://linux.thai.net/projects/thaifonts-scalable\n  name: fonts-tlwg-loma\n  section: fonts\n  version: 1:0.4.17-1ubuntu1\nfonts-tlwg-mono:\n  description: Thai TlwgMono font\n  downloads: 0\n  homepage_uri: http://linux.thai.net/projects/thaifonts-scalable\n  name: fonts-tlwg-mono\n  section: fonts\n  version: 1:0.4.17-1ubuntu1\nfonts-tlwg-norasi:\n  description: Thai Norasi font\n  downloads: 0\n  homepage_uri: http://linux.thai.net/projects/thaifonts-scalable\n  name: fonts-tlwg-norasi\n  section: fonts\n  version: 1:0.4.17-1ubuntu1\nfonts-tlwg-purisa:\n  description: Thai Purisa font\n  downloads: 0\n  homepage_uri: http://linux.thai.net/projects/thaifonts-scalable\n  name: fonts-tlwg-purisa\n  section: fonts\n  version: 1:0.4.17-1ubuntu1\nfonts-tlwg-sawasdee:\n  description: Thai Sawasdee font\n  downloads: 0\n  homepage_uri: http://linux.thai.net/projects/thaifonts-scalable\n  name: fonts-tlwg-sawasdee\n  section: fonts\n  version: 1:0.4.17-1ubuntu1\nfonts-tlwg-typewriter:\n  description: Thai TlwgTypewriter font\n  downloads: 0\n  homepage_uri: http://linux.thai.net/projects/thaifonts-scalable\n  name: fonts-tlwg-typewriter\n  section: fonts\n  version: 1:0.4.17-1ubuntu1\nfonts-tlwg-typist:\n  description: Thai TlwgTypist font\n  downloads: 0\n  homepage_uri: http://linux.thai.net/projects/thaifonts-scalable\n  name: fonts-tlwg-typist\n  section: fonts\n  version: 1:0.4.17-1ubuntu1\nfonts-tlwg-typo:\n  description: Thai TlwgTypo font\n  downloads: 0\n  homepage_uri: http://linux.thai.net/projects/thaifonts-scalable\n  name: fonts-tlwg-typo\n  section: fonts\n  version: 1:0.4.17-1ubuntu1\nfonts-tlwg-umpush:\n  description: Thai Umpush font\n  downloads: 0\n  homepage_uri: http://linux.thai.net/projects/thaifonts-scalable\n  name: fonts-tlwg-umpush\n  section: fonts\n  version: 1:0.4.17-1ubuntu1\nfonts-tlwg-waree:\n  description: Thai Waree font\n  downloads: 0\n  homepage_uri: http://linux.thai.net/projects/thaifonts-scalable\n  name: fonts-tlwg-waree\n  section: fonts\n  version: 1:0.4.17-1ubuntu1\nfonts-ubuntu-font-family-console:\n  description: Ubuntu Font Family Linux console fonts, sans-serif monospace\n  downloads: 0\n  homepage_uri: http://font.ubuntu.com/\n  name: fonts-ubuntu-font-family-console\n  section: fonts\n  version: 0.80-0ubuntu2\nfoomatic-db-compressed-ppds:\n  description: OpenPrinting printer support - Compressed PPDs derived from the database\n  downloads: 4921\n  homepage_uri: http://www.openprinting.org/\n  name: foomatic-db-compressed-ppds\n  section: text\n  version: 20120322-0ubuntu1\nfoomatic-db-engine:\n  description: OpenPrinting printer support - programs\n  downloads: 11594\n  homepage_uri: http://www.openprinting.org/\n  name: foomatic-db-engine\n  section: text\n  version: 4.0.8-2ubuntu1\nfoomatic-filters:\n  description: OpenPrinting printer support - filters\n  downloads: 16910\n  homepage_uri: http://www.openprinting.org/\n  name: foomatic-filters\n  section: text\n  version: 4.0.15-0ubuntu1\nfop:\n  description: XML formatter driven by XSL Formatting Objects (XSL-FO.)\n  downloads: 805\n  homepage_uri: http://xmlgraphics.apache.org/fop/\n  name: fop\n  section: text\n  version: 1:1.0.dfsg2-6\nfreenx-server:\n  description: Remote desktop/application/thin-client server core features\n  downloads: 113\n  homepage_uri: ''\n  name: freenx-server\n  section: x11\n  version: 0.7.3.zgit.120322.977c28d-0~ppa11\nfriendly-recovery:\n  description: Make recovery more user-friendly\n  downloads: 1\n  homepage_uri: ''\n  name: friendly-recovery\n  section: admin\n  version: 0.2.25\nftp:\n  description: classical file transfer client\n  downloads: 9907\n  homepage_uri: ''\n  name: ftp\n  section: net\n  version: 0.17-25\nfuse:\n  description: Filesystem in Userspace\n  downloads: 10674\n  homepage_uri: http://fuse.sourceforge.net/\n  name: fuse\n  section: utils\n  version: 2.8.6-2ubuntu2\nfuse-utils:\n  description: Filesystem in Userspace (transitional package)\n  downloads: 9727\n  homepage_uri: http://fuse.sourceforge.net/\n  name: fuse-utils\n  section: utils\n  version: 2.8.6-2ubuntu2\ng++:\n  description: GNU C++ compiler\n  downloads: 10176\n  homepage_uri: ''\n  name: g++\n  section: devel\n  version: 4:4.6.3-1ubuntu5\ng++-4.6:\n  description: GNU C++ compiler\n  downloads: 5740\n  homepage_uri: http://gcc.gnu.org/\n  name: g++-4.6\n  section: devel\n  version: 4.6.3-1ubuntu5\ngawk:\n  description: GNU awk, a pattern scanning and processing language\n  downloads: 35792\n  homepage_uri: http://www.gnu.org/software/gawk/\n  name: gawk\n  section: interpreters\n  version: 1:3.1.8+dfsg-0.1ubuntu1\ngcalctool:\n  description: GNOME desktop calculator\n  downloads: 11016\n  homepage_uri: http://live.gnome.org/Gcalctool\n  name: gcalctool\n  section: math\n  version: 6.4.1.1-0ubuntu2\ngcc:\n  description: GNU C compiler\n  downloads: 23226\n  homepage_uri: ''\n  name: gcc\n  section: devel\n  version: 4:4.6.3-1ubuntu5\ngcc-4.6:\n  description: GNU C compiler\n  downloads: 10350\n  homepage_uri: http://gcc.gnu.org/\n  name: gcc-4.6\n  section: devel\n  version: 4.6.3-1ubuntu5\ngcc-4.6-base:\n  description: GCC, the GNU Compiler Collection (base package)\n  downloads: 13433\n  homepage_uri: http://gcc.gnu.org/\n  name: gcc-4.6-base\n  section: libs\n  version: 4.6.3-1ubuntu5\ngcc-4.6-base:i386:\n  description: GCC, the GNU Compiler Collection (base package)\n  downloads: 0\n  homepage_uri: http://gcc.gnu.org/\n  name: gcc-4.6-base:i386\n  section: libs\n  version: 4.6.3-1ubuntu5\ngccxml:\n  description: XML output extension to GCC\n  downloads: 341\n  homepage_uri: http://www.gccxml.org/\n  name: gccxml\n  section: devel\n  version: 0.9.0+cvs20111013-1\ngcj-4.6-base:\n  description: GCC, the GNU Compiler Collection (gcj base package)\n  downloads: 3645\n  homepage_uri: http://gcc.gnu.org/\n  name: gcj-4.6-base\n  section: libs\n  version: 4.6.3-1ubuntu2\ngcj-4.6-jre-lib:\n  description: Java runtime library for use with gcj (jar files)\n  downloads: 205\n  homepage_uri: http://gcc.gnu.org/\n  name: gcj-4.6-jre-lib\n  section: java\n  version: 4.6.3-1ubuntu2\ngconf-service:\n  description: GNOME configuration database system (D-Bus service)\n  downloads: 15018\n  homepage_uri: http://projects.gnome.org/gconf/\n  name: gconf-service\n  section: libs\n  version: 3.2.5-0ubuntu2\ngconf-service-backend:\n  description: GNOME configuration database system (D-Bus service)\n  downloads: 1\n  homepage_uri: http://projects.gnome.org/gconf/\n  name: gconf-service-backend\n  section: libs\n  version: 3.2.5-0ubuntu2\ngconf2:\n  description: GNOME configuration database system (support tools)\n  downloads: 45482\n  homepage_uri: http://projects.gnome.org/gconf/\n  name: gconf2\n  section: libs\n  version: 3.2.5-0ubuntu2\ngconf2-common:\n  description: GNOME configuration database system (common files)\n  downloads: 15781\n  homepage_uri: http://projects.gnome.org/gconf/\n  name: gconf2-common\n  section: libs\n  version: 3.2.5-0ubuntu2\ngdb:\n  description: The GNU Debugger\n  downloads: 5081\n  homepage_uri: ''\n  name: gdb\n  section: devel\n  version: 7.4-2012.04-0ubuntu2\ngdm:\n  description: GNOME Display Manager\n  downloads: 7248\n  homepage_uri: ''\n  name: gdm\n  section: universe/gnome\n  version: 3.0.4-0ubuntu15\ngedit:\n  description: official text editor of the GNOME desktop environment\n  downloads: 19078\n  homepage_uri: http://www.gnome.org/projects/gedit/\n  name: gedit\n  section: gnome\n  version: 3.4.1-0ubuntu1\ngedit-common:\n  description: official text editor of the GNOME desktop environment (support files)\n  downloads: 0\n  homepage_uri: http://www.gnome.org/projects/gedit/\n  name: gedit-common\n  section: gnome\n  version: 3.4.1-0ubuntu1\ngenisoimage:\n  description: Creates ISO-9660 CD-ROM filesystem images\n  downloads: 15424\n  homepage_uri: ''\n  name: genisoimage\n  section: otherosfs\n  version: 9:1.1.11-2ubuntu2\ngeoclue:\n  description: Geographic information framework\n  downloads: 944\n  homepage_uri: http://www.freedesktop.org/wiki/Software/GeoClue\n  name: geoclue\n  section: utils\n  version: 0.12.0-1ubuntu12\ngeoclue-ubuntu-geoip:\n  description: Provide positioning for GeoClue via Ubuntu GeoIP services\n  downloads: 0\n  homepage_uri: https://launchpad.net/ubuntu-geoip\n  name: geoclue-ubuntu-geoip\n  section: gnome\n  version: 0.0.2-0ubuntu6\ngeoip-database:\n  description: IP lookup command line tools that use the GeoIP library (country database)\n  downloads: 908\n  homepage_uri: http://www.maxmind.com/\n  name: geoip-database\n  section: net\n  version: 20111220-1\ngettext:\n  description: GNU Internationalization utilities\n  downloads: 8557\n  homepage_uri: http://www.gnu.org/software/gettext/\n  name: gettext\n  section: devel\n  version: 0.18.1.1-5ubuntu3\ngettext-base:\n  description: GNU Internationalization utilities for the base system\n  downloads: 26115\n  homepage_uri: http://www.gnu.org/software/gettext/\n  name: gettext-base\n  section: utils\n  version: 0.18.1.1-5ubuntu3\ngfortran:\n  description: GNU Fortran 95 compiler\n  downloads: 1171\n  homepage_uri: ''\n  name: gfortran\n  section: devel\n  version: 4:4.6.3-1ubuntu5\ngfortran-4.6:\n  description: GNU Fortran compiler\n  downloads: 480\n  homepage_uri: http://gcc.gnu.org/\n  name: gfortran-4.6\n  section: devel\n  version: 4.6.3-1ubuntu5\nghc:\n  description: The Glasgow Haskell Compilation system\n  downloads: 420\n  homepage_uri: http://haskell.org/ghc/\n  name: ghc\n  section: universe/haskell\n  version: 7.4.1-1ubuntu2\nghc6:\n  description: transitional dummy package\n  downloads: 338\n  homepage_uri: http://haskell.org/ghc/\n  name: ghc6\n  section: universe/devel\n  version: 7.4.1-1ubuntu2\nghostscript:\n  description: interpreter for the PostScript language and for PDF\n  downloads: 25415\n  homepage_uri: http://www.ghostscript.com/\n  name: ghostscript\n  section: text\n  version: 9.05~dfsg-0ubuntu4\nghostscript-cups:\n  description: interpreter for the PostScript language and for PDF - CUPS filters\n  downloads: 6081\n  homepage_uri: http://www.ghostscript.com/\n  name: ghostscript-cups\n  section: text\n  version: 9.05~dfsg-0ubuntu4\nghostscript-x:\n  description: interpreter for the PostScript language and for PDF - X11 support\n  downloads: 8197\n  homepage_uri: http://www.ghostscript.com/\n  name: ghostscript-x\n  section: text\n  version: 9.05~dfsg-0ubuntu4\nginn:\n  description: 'Gesture Injector: No-GEIS, No-Toolkits'\n  downloads: 0\n  homepage_uri: https://launchpad.net/ginn\n  name: ginn\n  section: misc\n  version: 0.2.4-0ubuntu1\ngir1.2-appindicator3-0.1:\n  description: Typelib files for libappindicator3-1.\n  downloads: 3\n  homepage_uri: https://launchpad.net/libappindicator\n  name: gir1.2-appindicator3-0.1\n  section: libs\n  version: 0.4.92-0ubuntu1\ngir1.2-atk-1.0:\n  description: ATK accessibility toolkit (GObject introspection)\n  downloads: 11420\n  homepage_uri: http://www.gtk.org/\n  name: gir1.2-atk-1.0\n  section: libs\n  version: 2.4.0-0ubuntu1\ngir1.2-atspi-2.0:\n  description: Assistive Technology Service Provider (GObject introspection)\n  downloads: 267\n  homepage_uri: http://live.gnome.org/Accessibility/GNOME3\n  name: gir1.2-atspi-2.0\n  section: misc\n  version: 2.4.2-0ubuntu0.1\ngir1.2-dbusmenu-glib-0.4:\n  description: typelib file for libdbusmenu-glib4\n  downloads: 9\n  homepage_uri: https://launchpad.net/dbusmenu\n  name: gir1.2-dbusmenu-glib-0.4\n  section: libs\n  version: 0.6.1-0ubuntu3\ngir1.2-dbusmenu-gtk-0.4:\n  description: typelib file for libdbusmenu-gtk4\n  downloads: 4\n  homepage_uri: https://launchpad.net/dbusmenu\n  name: gir1.2-dbusmenu-gtk-0.4\n  section: libs\n  version: 0.6.1-0ubuntu3\ngir1.2-dee-1.0:\n  description: GObject introspection data for the Dee library\n  downloads: 1\n  homepage_uri: https://launchpad.net/dee\n  name: gir1.2-dee-1.0\n  section: libs\n  version: 1.0.10-0ubuntu1\ngir1.2-freedesktop:\n  description: Introspection data for some FreeDesktop components\n  downloads: 10913\n  homepage_uri: http://live.gnome.org/GObjectIntrospection\n  name: gir1.2-freedesktop\n  section: libs\n  version: 1.32.0-1\ngir1.2-gconf-2.0:\n  description: GNOME configuration database system (GObject-Introspection)\n  downloads: 3376\n  homepage_uri: http://projects.gnome.org/gconf/\n  name: gir1.2-gconf-2.0\n  section: libs\n  version: 3.2.5-0ubuntu2\ngir1.2-gdkpixbuf-2.0:\n  description: GDK Pixbuf library - GObject-Introspection\n  downloads: 11411\n  homepage_uri: http://www.gtk.org/\n  name: gir1.2-gdkpixbuf-2.0\n  section: libs\n  version: 2.26.1-1\ngir1.2-glib-2.0:\n  description: Introspection data for GLib, GObject, Gio and GModule\n  downloads: 13556\n  homepage_uri: http://live.gnome.org/GObjectIntrospection\n  name: gir1.2-glib-2.0\n  section: libs\n  version: 1.32.0-1\ngir1.2-gmenu-3.0:\n  description: GObject introspection data for the GNOME menu library\n  downloads: 7880\n  homepage_uri: ''\n  name: gir1.2-gmenu-3.0\n  section: libs\n  version: 3.4.0-0ubuntu1\ngir1.2-gnomebluetooth-1.0:\n  description: Introspection data for GnomeBluetooth\n  downloads: 9491\n  homepage_uri: http://live.gnome.org/GnomeBluetooth\n  name: gir1.2-gnomebluetooth-1.0\n  section: gnome\n  version: 3.2.2-0ubuntu5\ngir1.2-gnomekeyring-1.0:\n  description: GNOME keyring services library - introspection data\n  downloads: 422\n  homepage_uri: http://live.gnome.org/GnomeKeyring\n  name: gir1.2-gnomekeyring-1.0\n  section: introspection\n  version: 3.2.2-2\ngir1.2-gst-plugins-base-0.10:\n  description: 'Description: GObject introspection data for the GStreamer Plugins\n    Base library'\n  downloads: 2071\n  homepage_uri: http://gstreamer.freedesktop.org\n  name: gir1.2-gst-plugins-base-0.10\n  section: libs\n  version: 0.10.36-1\ngir1.2-gstreamer-0.10:\n  description: 'Description: GObject introspection data for the GStreamer library'\n  downloads: 2489\n  homepage_uri: http://gstreamer.freedesktop.org\n  name: gir1.2-gstreamer-0.10\n  section: libs\n  version: 0.10.36-1ubuntu1\ngir1.2-gtk-2.0:\n  description: GTK+ graphical user interface library -- gir bindings\n  downloads: 737\n  homepage_uri: http://www.gtk.org/\n  name: gir1.2-gtk-2.0\n  section: libs\n  version: 2.24.10-0ubuntu6\ngir1.2-gtk-3.0:\n  description: GTK+ graphical user interface library -- gir bindings\n  downloads: 10691\n  homepage_uri: http://www.gtk.org/\n  name: gir1.2-gtk-3.0\n  section: libs\n  version: 3.4.2-0ubuntu0.2\ngir1.2-gtksource-3.0:\n  description: gir files for the GTK+ syntax highlighting widget\n  downloads: 5493\n  homepage_uri: http://projects.gnome.org/gtksourceview/\n  name: gir1.2-gtksource-3.0\n  section: libs\n  version: 3.4.2-0ubuntu1\ngir1.2-gudev-1.0:\n  description: libgudev-1.0 introspection data\n  downloads: 1\n  homepage_uri: http://www.kernel.org/pub/linux/utils/kernel/hotplug/udev.html\n  name: gir1.2-gudev-1.0\n  section: libs\n  version: 175-0ubuntu9\ngir1.2-indicate-0.7:\n  description: Typelib file for libindicate5\n  downloads: 3\n  homepage_uri: https://launchpad.net/libindicate\n  name: gir1.2-indicate-0.7\n  section: libs\n  version: 0.6.92-0ubuntu1\ngir1.2-javascriptcoregtk-3.0:\n  description: GObject introspection data for the GTK+-based JavaScriptCore library\n  downloads: 1241\n  homepage_uri: http://webkitgtk.org/\n  name: gir1.2-javascriptcoregtk-3.0\n  section: libs\n  version: 1.8.0-0ubuntu2\ngir1.2-launchpad-integration-3.0:\n  description: library for launchpad integration (gir files)\n  downloads: 0\n  homepage_uri: ''\n  name: gir1.2-launchpad-integration-3.0\n  section: libs\n  version: 0.1.56\ngir1.2-notify-0.7:\n  description: sends desktop notifications to a notification daemon (Introspection\n    files)\n  downloads: 484\n  homepage_uri: ''\n  name: gir1.2-notify-0.7\n  section: libs\n  version: 0.7.5-1\ngir1.2-panelapplet-4.0:\n  description: GObject introspection for the GNOME Panel Applet library\n  downloads: 197\n  homepage_uri: ''\n  name: gir1.2-panelapplet-4.0\n  section: universe/libs\n  version: 1:3.4.1-0ubuntu1\ngir1.2-pango-1.0:\n  description: Layout and rendering of internationalized text - gir bindings\n  downloads: 11358\n  homepage_uri: ''\n  name: gir1.2-pango-1.0\n  section: libs\n  version: 1.30.0-0ubuntu3\ngir1.2-peas-1.0:\n  description: Application plugin library (introspection files)\n  downloads: 8059\n  homepage_uri: http://live.gnome.org/Libpeas\n  name: gir1.2-peas-1.0\n  section: libs\n  version: 1.2.0-1ubuntu1\ngir1.2-rb-3.0:\n  description: GObject introspection data for the rhythmbox music player\n  downloads: 1405\n  homepage_uri: http://projects.gnome.org/rhythmbox/\n  name: gir1.2-rb-3.0\n  section: libs\n  version: 2.96-0ubuntu4\ngir1.2-soup-2.4:\n  description: GObject introspection data for the libsoup HTTP library\n  downloads: 8431\n  homepage_uri: ''\n  name: gir1.2-soup-2.4\n  section: libs\n  version: 2.38.1-1\ngir1.2-totem-1.0:\n  description: GObject introspection data for Totem media player\n  downloads: 4634\n  homepage_uri: http://www.gnome.org/projects/totem/\n  name: gir1.2-totem-1.0\n  section: libs\n  version: 3.0.1-0ubuntu21\ngir1.2-totem-plparser-1.0:\n  description: GObject introspection data for the Totem Playlist Parser library\n  downloads: 4566\n  homepage_uri: ''\n  name: gir1.2-totem-plparser-1.0\n  section: libs\n  version: 3.4.1-1\ngir1.2-ubuntuoneui-3.0:\n  description: Ubuntu One widget library\n  downloads: 1\n  homepage_uri: https://launchpad.net/libubuntuone\n  name: gir1.2-ubuntuoneui-3.0\n  section: introspection\n  version: 3.0.1-0ubuntu1\ngir1.2-unity-5.0:\n  description: GObject introspection data for the Unity library\n  downloads: 0\n  homepage_uri: https://launchpad.net/libunity\n  name: gir1.2-unity-5.0\n  section: libs\n  version: 5.12.0-0ubuntu1\ngir1.2-vte-2.90:\n  description: GObject introspection data for the VTE library\n  downloads: 4537\n  homepage_uri: ''\n  name: gir1.2-vte-2.90\n  section: libs\n  version: 1:0.32.1-0ubuntu1\ngir1.2-webkit-3.0:\n  description: GObject introspection data for the WebKit library\n  downloads: 1920\n  homepage_uri: http://webkitgtk.org/\n  name: gir1.2-webkit-3.0\n  section: libs\n  version: 1.8.0-0ubuntu2\ngir1.2-wnck-3.0:\n  description: GObject introspection data for the WNCK library\n  downloads: 461\n  homepage_uri: ''\n  name: gir1.2-wnck-3.0\n  section: libs\n  version: 3.4.0-0ubuntu1\ngit:\n  description: fast, scalable, distributed revision control system\n  downloads: 13988\n  homepage_uri: http://git-scm.com/\n  name: git\n  section: vcs\n  version: 1:1.7.9.5-1\ngit-core:\n  description: fast, scalable, distributed revision control system (obsolete)\n  downloads: 1481\n  homepage_uri: http://git-scm.com/\n  name: git-core\n  section: devel\n  version: 1:1.7.9.5-1\ngit-man:\n  description: fast, scalable, distributed revision control system (manual pages)\n  downloads: 0\n  homepage_uri: http://git-scm.com/\n  name: git-man\n  section: vcs\n  version: 1:1.7.9.5-1\ngksu:\n  description: graphical frontend to su\n  downloads: 29768\n  homepage_uri: http://www.nongnu.org/gksu\n  name: gksu\n  section: admin\n  version: 2.0.2-6ubuntu1\nglib-networking:\n  description: network-related giomodules for GLib\n  downloads: 2203\n  homepage_uri: ''\n  name: glib-networking\n  section: libs\n  version: 2.32.1-1ubuntu1\nglib-networking-common:\n  description: network-related giomodules for GLib - data files\n  downloads: 2362\n  homepage_uri: ''\n  name: glib-networking-common\n  section: libs\n  version: 2.32.1-1ubuntu1\nglib-networking-services:\n  description: network-related giomodules for GLib - D-Bus services\n  downloads: 2361\n  homepage_uri: ''\n  name: glib-networking-services\n  section: libs\n  version: 2.32.1-1ubuntu1\nglib-networking:i386:\n  description: network-related giomodules for GLib\n  downloads: 0\n  homepage_uri: ''\n  name: glib-networking:i386\n  section: libs\n  version: 2.32.1-1ubuntu1\ngnome-accessibility-themes:\n  description: accessibility themes for the GNOME desktop\n  downloads: 0\n  homepage_uri: ''\n  name: gnome-accessibility-themes\n  section: gnome\n  version: 3.4.1-0ubuntu1\ngnome-applets:\n  description: Various applets for the GNOME panel - binary files\n  downloads: 21647\n  homepage_uri: ''\n  name: gnome-applets\n  section: universe/gnome\n  version: 3.4.1-0ubuntu1\ngnome-applets-data:\n  description: Various applets for the GNOME panel - data files\n  downloads: 22572\n  homepage_uri: ''\n  name: gnome-applets-data\n  section: universe/gnome\n  version: 3.4.1-0ubuntu1\ngnome-backgrounds:\n  description: Set of backgrounds packaged with the GNOME desktop\n  downloads: 0\n  homepage_uri: ''\n  name: gnome-backgrounds\n  section: universe/gnome\n  version: 3.4.1-1\ngnome-bluetooth:\n  description: GNOME Bluetooth tools\n  downloads: 28843\n  homepage_uri: http://live.gnome.org/GnomeBluetooth\n  name: gnome-bluetooth\n  section: gnome\n  version: 3.2.2-0ubuntu5\ngnome-control-center:\n  description: utilities to configure the GNOME desktop\n  downloads: 17783\n  homepage_uri: ''\n  name: gnome-control-center\n  section: gnome\n  version: 1:3.4.2-0ubuntu0.2\ngnome-control-center-data:\n  description: configuration applets for GNOME - data files\n  downloads: 0\n  homepage_uri: ''\n  name: gnome-control-center-data\n  section: gnome\n  version: 1:3.4.2-0ubuntu0.2\ngnome-desktop3-data:\n  description: Common files for GNOME desktop apps\n  downloads: 0\n  homepage_uri: ''\n  name: gnome-desktop3-data\n  section: gnome\n  version: 3.4.1-0ubuntu1\ngnome-disk-utility:\n  description: manage and configure disk drives and media\n  downloads: 26660\n  homepage_uri: http://git.gnome.org/cgit/gnome-disk-utility/\n  name: gnome-disk-utility\n  section: admin\n  version: 3.0.2-2ubuntu7\ngnome-font-viewer:\n  description: font viewer for GNOME\n  downloads: 1378\n  homepage_uri: http://live.gnome.org/GnomeUtils\n  name: gnome-font-viewer\n  section: gnome\n  version: 3.4.0-1\ngnome-games-data:\n  description: data files for the GNOME games\n  downloads: 1226\n  homepage_uri: http://live.gnome.org/GnomeGames\n  name: gnome-games-data\n  section: gnome\n  version: 1:3.4.1-0ubuntu2\ngnome-icon-theme:\n  description: GNOME Desktop icon theme (small subset)\n  downloads: 36553\n  homepage_uri: ''\n  name: gnome-icon-theme\n  section: gnome\n  version: 3.4.0-0ubuntu1\ngnome-icon-theme-symbolic:\n  description: GNOME desktop icon theme (symbolic icons)\n  downloads: 8456\n  homepage_uri: ''\n  name: gnome-icon-theme-symbolic\n  section: gnome\n  version: 3.4.0-1\ngnome-keyring:\n  description: GNOME keyring services (daemon and tools)\n  downloads: 37949\n  homepage_uri: http://live.gnome.org/GnomeKeyring\n  name: gnome-keyring\n  section: gnome\n  version: 3.2.2-2ubuntu4\ngnome-media:\n  description: GNOME media utilities\n  downloads: 8792\n  homepage_uri: http://live.gnome.org/GnomeMedia\n  name: gnome-media\n  section: gnome\n  version: 3.4.0-0ubuntu2\ngnome-menus:\n  description: GNOME implementation of the freedesktop menu specification\n  downloads: 22903\n  homepage_uri: ''\n  name: gnome-menus\n  section: gnome\n  version: 3.4.0-0ubuntu1\ngnome-nettool:\n  description: network information tool for GNOME\n  downloads: 3412\n  homepage_uri: http://projects.gnome.org/gnome-network/\n  name: gnome-nettool\n  section: gnome\n  version: 3.2.0-0ubuntu1\ngnome-online-accounts:\n  description: GNOME Online Accounts\n  downloads: 1009\n  homepage_uri: https://live.gnome.org/OnlineAccounts\n  name: gnome-online-accounts\n  section: gnome\n  version: 3.4.0-0ubuntu1\ngnome-orca:\n  description: Scriptable screen reader\n  downloads: 2177\n  homepage_uri: http://live.gnome.org/Orca\n  name: gnome-orca\n  section: gnome\n  version: 3.4.2-0ubuntu0.1\ngnome-panel:\n  description: launcher and docking facility for GNOME\n  downloads: 25163\n  homepage_uri: ''\n  name: gnome-panel\n  section: universe/gnome\n  version: 1:3.4.1-0ubuntu1\ngnome-panel-data:\n  description: common files for the GNOME Panel\n  downloads: 22503\n  homepage_uri: ''\n  name: gnome-panel-data\n  section: universe/gnome\n  version: 1:3.4.1-0ubuntu1\ngnome-power-manager:\n  description: power management tool for the GNOME desktop\n  downloads: 24959\n  homepage_uri: http://www.gnome.org/projects/gnome-power-manager/\n  name: gnome-power-manager\n  section: gnome\n  version: 3.4.0-0ubuntu1\ngnome-screensaver:\n  description: GNOME screen saver and locker\n  downloads: 30960\n  homepage_uri: http://live.gnome.org/GnomeScreensaver\n  name: gnome-screensaver\n  section: gnome\n  version: 3.4.1-0ubuntu1\ngnome-screenshot:\n  description: screenshot application for GNOME\n  downloads: 8739\n  homepage_uri: http://live.gnome.org/GnomeUtils\n  name: gnome-screenshot\n  section: gnome\n  version: 3.4.1-0ubuntu1\ngnome-session:\n  description: GNOME Session Manager - GNOME 3 session\n  downloads: 20285\n  homepage_uri: ''\n  name: gnome-session\n  section: gnome\n  version: 3.2.1-0ubuntu8\ngnome-session-bin:\n  description: GNOME Session Manager - Minimal runtime\n  downloads: 32359\n  homepage_uri: ''\n  name: gnome-session-bin\n  section: gnome\n  version: 3.2.1-0ubuntu8\ngnome-session-canberra:\n  description: GNOME session log in and log out sound events\n  downloads: 2866\n  homepage_uri: http://0pointer.de/lennart/projects/libcanberra/\n  name: gnome-session-canberra\n  section: libs\n  version: 0.28-3ubuntu3\ngnome-session-common:\n  description: GNOME Session Manager - common files\n  downloads: 22194\n  homepage_uri: ''\n  name: gnome-session-common\n  section: gnome\n  version: 3.2.1-0ubuntu8\ngnome-session-fallback:\n  description: GNOME Session Manager - GNOME fallback session\n  downloads: 3923\n  homepage_uri: ''\n  name: gnome-session-fallback\n  section: universe/gnome\n  version: 3.2.1-0ubuntu8\ngnome-settings-daemon:\n  description: daemon handling the GNOME session settings\n  downloads: 33233\n  homepage_uri: ''\n  name: gnome-settings-daemon\n  section: gnome\n  version: 3.4.1-0ubuntu1.1\ngnome-sudoku:\n  description: Sudoku puzzle game for GNOME\n  downloads: 480\n  homepage_uri: http://live.gnome.org/GnomeGames\n  name: gnome-sudoku\n  section: games\n  version: 1:3.4.1-0ubuntu2\ngnome-system-log:\n  description: system log viewer for GNOME\n  downloads: 2978\n  homepage_uri: http://live.gnome.org/GnomeUtils\n  name: gnome-system-log\n  section: gnome\n  version: 3.4.1-0ubuntu1\ngnome-system-monitor:\n  description: Process viewer and system resource monitor for GNOME\n  downloads: 8633\n  homepage_uri: ''\n  name: gnome-system-monitor\n  section: gnome\n  version: 3.4.1-0ubuntu1\ngnome-system-tools:\n  description: Cross-platform configuration utilities for GNOME\n  downloads: 6907\n  homepage_uri: ''\n  name: gnome-system-tools\n  section: universe/gnome\n  version: 3.0.0-2ubuntu1\ngnome-terminal:\n  description: GNOME terminal emulator application\n  downloads: 25270\n  homepage_uri: ''\n  name: gnome-terminal\n  section: gnome\n  version: 3.4.1.1-0ubuntu1\ngnome-terminal-data:\n  description: Data files for the GNOME terminal emulator\n  downloads: 1753\n  homepage_uri: ''\n  name: gnome-terminal-data\n  section: gnome\n  version: 3.4.1.1-0ubuntu1\ngnome-user-guide:\n  description: GNOME user's guide\n  downloads: 1\n  homepage_uri: ''\n  name: gnome-user-guide\n  section: gnome\n  version: 3.4.1-1\ngnome-user-share:\n  description: User level public file sharing via WebDAV or ObexFTP\n  downloads: 24693\n  homepage_uri: ''\n  name: gnome-user-share\n  section: gnome\n  version: 3.0.2-0ubuntu1\ngnomine:\n  description: popular minesweeper puzzle game for GNOME\n  downloads: 403\n  homepage_uri: http://live.gnome.org/GnomeGames\n  name: gnomine\n  section: games\n  version: 1:3.4.1-0ubuntu2\ngnupg:\n  description: GNU privacy guard - a free PGP replacement\n  downloads: 49882\n  homepage_uri: http://www.gnupg.org\n  name: gnupg\n  section: utils\n  version: 1.4.11-3ubuntu2\ngnuplot:\n  description: A command-line driven interactive plotting program\n  downloads: 213\n  homepage_uri: http://gnuplot.sourceforge.net/\n  name: gnuplot\n  section: universe/math\n  version: 4.4.3-0ubuntu3\ngnuplot-nox:\n  description: A command-line driven interactive plotting program\n  downloads: 1961\n  homepage_uri: http://gnuplot.sourceforge.net/\n  name: gnuplot-nox\n  section: universe/math\n  version: 4.4.3-0ubuntu3\ngnuplot-x11:\n  description: A command-line driven interactive plotting program\n  downloads: 2498\n  homepage_uri: http://gnuplot.sourceforge.net/\n  name: gnuplot-x11\n  section: universe/math\n  version: 4.4.3-0ubuntu3\ngpgv:\n  description: GNU privacy guard - signature verification tool\n  downloads: 68978\n  homepage_uri: http://www.gnupg.org\n  name: gpgv\n  section: utils\n  version: 1.4.11-3ubuntu2\ngraphviz:\n  description: rich set of graph drawing tools\n  downloads: 2333\n  homepage_uri: http://www.graphviz.org/\n  name: graphviz\n  section: graphics\n  version: 2.26.3-10ubuntu1\ngrep:\n  description: GNU grep, egrep and fgrep\n  downloads: 110992\n  homepage_uri: http://www.gnu.org/software/grep/\n  name: grep\n  section: utils\n  version: 2.10-1\ngroff:\n  description: GNU troff text-formatting system\n  downloads: 1090\n  homepage_uri: http://groff.ffii.org/\n  name: groff\n  section: text\n  version: 1.21-7\ngroff-base:\n  description: GNU troff text-formatting system (base system components)\n  downloads: 105546\n  homepage_uri: http://groff.ffii.org/\n  name: groff-base\n  section: text\n  version: 1.21-7\ngrowisofs:\n  description: DVD+-RW/R recorder\n  downloads: 4117\n  homepage_uri: http://fy.chalmers.se/~appro/linux/DVD+RW/\n  name: growisofs\n  section: video\n  version: 7.1-10\ngrub-common:\n  description: GRand Unified Bootloader (common files)\n  downloads: 17653\n  homepage_uri: http://www.gnu.org/software/grub/\n  name: grub-common\n  section: admin\n  version: 1.99-21ubuntu3.1\ngrub-gfxpayload-lists:\n  description: GRUB gfxpayload blacklist\n  downloads: 1\n  homepage_uri: ''\n  name: grub-gfxpayload-lists\n  section: admin\n  version: '0.6'\ngrub-legacy-ec2:\n  description: Handles update-grub for ec2 instances\n  downloads: 1\n  homepage_uri: ''\n  name: grub-legacy-ec2\n  section: admin\n  version: 0.6.3-0ubuntu1\ngrub-pc:\n  description: GRand Unified Bootloader, version 2 (PC/BIOS version)\n  downloads: 10228\n  homepage_uri: http://www.gnu.org/software/grub/\n  name: grub-pc\n  section: admin\n  version: 1.99-21ubuntu3.1\ngrub-pc-bin:\n  description: GRand Unified Bootloader, version 2 (PC/BIOS binaries)\n  downloads: 1709\n  homepage_uri: http://www.gnu.org/software/grub/\n  name: grub-pc-bin\n  section: admin\n  version: 1.99-21ubuntu3.1\ngrub2-common:\n  description: GRand Unified Bootloader (common files for version 2)\n  downloads: 6525\n  homepage_uri: http://www.gnu.org/software/grub/\n  name: grub2-common\n  section: admin\n  version: 1.99-21ubuntu3.1\ngs-cjk-resource:\n  description: Resource files for gs-cjk, ghostscript CJK-TrueType extension\n  downloads: 3135\n  homepage_uri: ''\n  name: gs-cjk-resource\n  section: text\n  version: 1.20100103-3\ngsettings-desktop-schemas:\n  description: GSettings deskop-wide schemas\n  downloads: 10179\n  homepage_uri: http://www.gnome.org/\n  name: gsettings-desktop-schemas\n  section: libs\n  version: 3.4.1-0ubuntu1\ngsfonts:\n  description: Fonts for the Ghostscript interpreter(s)\n  downloads: 25634\n  homepage_uri: http://www.ghostscript.com/\n  name: gsfonts\n  section: text\n  version: 1:8.11+urwcyr1.0.7~pre44-4.2ubuntu1\ngsl-bin:\n  description: GNU Scientific Library (GSL) -- binary package\n  downloads: 108\n  homepage_uri: http://www.gnu.org/software/gsl\n  name: gsl-bin\n  section: universe/math\n  version: 1.15+dfsg-1build1\ngstreamer0.10-alsa:\n  description: GStreamer plugin for ALSA\n  downloads: 24770\n  homepage_uri: http://gstreamer.freedesktop.org\n  name: gstreamer0.10-alsa\n  section: libs\n  version: 0.10.36-1\ngstreamer0.10-gconf:\n  description: GStreamer plugin for getting the sink/source information from GConf\n  downloads: 2698\n  homepage_uri: ''\n  name: gstreamer0.10-gconf\n  section: sound\n  version: 0.10.31-1ubuntu1\ngstreamer0.10-nice:\n  description: ICE library (GStreamer plugin)\n  downloads: 4907\n  homepage_uri: http://nice.freedesktop.org/\n  name: gstreamer0.10-nice\n  section: net\n  version: 0.1.1-2ubuntu1\ngstreamer0.10-plugins-base:\n  description: GStreamer plugins from the \"base\" set\n  downloads: 34634\n  homepage_uri: http://gstreamer.freedesktop.org\n  name: gstreamer0.10-plugins-base\n  section: libs\n  version: 0.10.36-1\ngstreamer0.10-plugins-base-apps:\n  description: GStreamer helper programs from the \"base\" set\n  downloads: 170\n  homepage_uri: http://gstreamer.freedesktop.org\n  name: gstreamer0.10-plugins-base-apps\n  section: utils\n  version: 0.10.36-1\ngstreamer0.10-plugins-base:i386:\n  description: GStreamer plugins from the \"base\" set\n  downloads: 0\n  homepage_uri: http://gstreamer.freedesktop.org\n  name: gstreamer0.10-plugins-base:i386\n  section: libs\n  version: 0.10.36-1\ngstreamer0.10-plugins-good:\n  description: GStreamer plugins from the \"good\" set\n  downloads: 33874\n  homepage_uri: ''\n  name: gstreamer0.10-plugins-good\n  section: libs\n  version: 0.10.31-1ubuntu1\ngstreamer0.10-plugins-good:i386:\n  description: GStreamer plugins from the \"good\" set\n  downloads: 0\n  homepage_uri: ''\n  name: gstreamer0.10-plugins-good:i386\n  section: libs\n  version: 0.10.31-1ubuntu1\ngstreamer0.10-pulseaudio:\n  description: GStreamer plugin for PulseAudio\n  downloads: 5700\n  homepage_uri: ''\n  name: gstreamer0.10-pulseaudio\n  section: sound\n  version: 0.10.31-1ubuntu1\ngstreamer0.10-tools:\n  description: Tools for use with GStreamer\n  downloads: 3295\n  homepage_uri: http://gstreamer.freedesktop.org\n  name: gstreamer0.10-tools\n  section: utils\n  version: 0.10.36-1ubuntu1\ngstreamer0.10-x:\n  description: GStreamer plugins for X11 and Pango\n  downloads: 30682\n  homepage_uri: http://gstreamer.freedesktop.org\n  name: gstreamer0.10-x\n  section: libs\n  version: 0.10.36-1\ngstreamer0.10-x:i386:\n  description: GStreamer plugins for X11 and Pango\n  downloads: 0\n  homepage_uri: http://gstreamer.freedesktop.org\n  name: gstreamer0.10-x:i386\n  section: libs\n  version: 0.10.36-1\ngtk2-engines:\n  description: theme engines for GTK+ 2.x\n  downloads: 27638\n  homepage_uri: ''\n  name: gtk2-engines\n  section: gnome\n  version: 1:2.20.2-1ubuntu1\ngtk2-engines-murrine:\n  description: cairo-based gtk+-2.0 theme engine\n  downloads: 963\n  homepage_uri: ''\n  name: gtk2-engines-murrine\n  section: x11\n  version: 0.98.2-0ubuntu1\ngtk2-engines-murrine:i386:\n  description: cairo-based gtk+-2.0 theme engine\n  downloads: 0\n  homepage_uri: ''\n  name: gtk2-engines-murrine:i386\n  section: x11\n  version: 0.98.2-0ubuntu1\ngtk2-engines-oxygen:i386:\n  description: Oxygen widget theme for GTK+-based applications\n  downloads: 0\n  homepage_uri: https://projects.kde.org/projects/playground/artwork/oxygen-gtk\n  name: gtk2-engines-oxygen:i386\n  section: x11\n  version: 1.2.2-0ubuntu1.1\ngtk2-engines-pixbuf:i386:\n  description: pixbuf-based theme for GTK+ 2.x\n  downloads: 0\n  homepage_uri: http://www.gtk.org/\n  name: gtk2-engines-pixbuf:i386\n  section: universe/graphics\n  version: 2.24.10-0ubuntu6\ngtk2-engines:i386:\n  description: theme engines for GTK+ 2.x\n  downloads: 0\n  homepage_uri: ''\n  name: gtk2-engines:i386\n  section: gnome\n  version: 1:2.20.2-1ubuntu1\ngtk3-engines-unico:\n  description: Unico Gtk+ 3 theme engine\n  downloads: 55\n  homepage_uri: https://launchpad.net/unico\n  name: gtk3-engines-unico\n  section: x11\n  version: 1.0.2-0ubuntu1\ngucharmap:\n  description: Unicode character picker and font browser\n  downloads: 3104\n  homepage_uri: http://live.gnome.org/Gucharmap\n  name: gucharmap\n  section: x11\n  version: 1:3.4.1.1-0ubuntu1\nguile-1.8-libs:\n  description: Core Guile libraries\n  downloads: 2341\n  homepage_uri: http://www.gnu.org/software/guile/\n  name: guile-1.8-libs\n  section: libs\n  version: 1.8.8+1-6ubuntu2\ngvfs:\n  description: userspace virtual filesystem - GIO module\n  downloads: 38669\n  homepage_uri: ''\n  name: gvfs\n  section: libs\n  version: 1.12.1-0ubuntu1\ngvfs-backends:\n  description: userspace virtual filesystem - backends\n  downloads: 33369\n  homepage_uri: ''\n  name: gvfs-backends\n  section: libs\n  version: 1.12.1-0ubuntu1\ngvfs-bin:\n  description: userspace virtual filesystem - binaries\n  downloads: 11948\n  homepage_uri: ''\n  name: gvfs-bin\n  section: libs\n  version: 1.12.1-0ubuntu1\ngvfs-common:\n  description: userspace virtual filesystem - common data files\n  downloads: 14841\n  homepage_uri: ''\n  name: gvfs-common\n  section: libs\n  version: 1.12.1-0ubuntu1\ngvfs-daemons:\n  description: userspace virtual filesystem - servers\n  downloads: 14827\n  homepage_uri: ''\n  name: gvfs-daemons\n  section: libs\n  version: 1.12.1-0ubuntu1\ngvfs-fuse:\n  description: userspace virtual filesystem - fuse server\n  downloads: 3866\n  homepage_uri: ''\n  name: gvfs-fuse\n  section: libs\n  version: 1.12.1-0ubuntu1\ngvfs-libs:\n  description: userspace virtual filesystem - private libraries\n  downloads: 14834\n  homepage_uri: ''\n  name: gvfs-libs\n  section: libs\n  version: 1.12.1-0ubuntu1\ngvfs-libs:i386:\n  description: userspace virtual filesystem - private libraries\n  downloads: 0\n  homepage_uri: ''\n  name: gvfs-libs:i386\n  section: libs\n  version: 1.12.1-0ubuntu1\ngvfs:i386:\n  description: userspace virtual filesystem - GIO module\n  downloads: 0\n  homepage_uri: ''\n  name: gvfs:i386\n  section: libs\n  version: 1.12.1-0ubuntu1\ngwibber:\n  description: Open source social networking client for GNOME\n  downloads: 131\n  homepage_uri: https://launchpad.net/gwibber\n  name: gwibber\n  section: misc\n  version: 3.4.1-0ubuntu1\ngwibber-service:\n  description: Open source social networking client for GNOME\n  downloads: 149\n  homepage_uri: https://launchpad.net/gwibber\n  name: gwibber-service\n  section: misc\n  version: 3.4.1-0ubuntu1\ngwibber-service-facebook:\n  description: Facebook plugin for Gwibber\n  downloads: 0\n  homepage_uri: https://launchpad.net/gwibber\n  name: gwibber-service-facebook\n  section: misc\n  version: 3.4.1-0ubuntu1\ngwibber-service-identica:\n  description: Identi.ca plugin for Gwibber\n  downloads: 0\n  homepage_uri: https://launchpad.net/gwibber\n  name: gwibber-service-identica\n  section: misc\n  version: 3.4.1-0ubuntu1\ngwibber-service-twitter:\n  description: Twitter plugin for Gwibber\n  downloads: 0\n  homepage_uri: https://launchpad.net/gwibber\n  name: gwibber-service-twitter\n  section: misc\n  version: 3.4.1-0ubuntu1\ngzip:\n  description: GNU compression utilities\n  downloads: 110479\n  homepage_uri: ''\n  name: gzip\n  section: utils\n  version: 1.4-1ubuntu2\nhadoop-0.20:\n  description: A software platform for processing vast amounts of data\n  downloads: 32\n  homepage_uri: http://hadoop.apache.org/core/\n  name: hadoop-0.20\n  section: misc\n  version: 0.20.2+923.256-1~maverick-cdh3\nhadoop-0.20-pipes:\n  description: Interface to author Hadoop MapReduce jobs in C++\n  downloads: 0\n  homepage_uri: http://hadoop.apache.org/core/\n  name: hadoop-0.20-pipes\n  section: misc\n  version: 0.20.2+923.256-1~maverick-cdh3\nhadoop-0.20-source:\n  description: Source code for Hadoop\n  downloads: 0\n  homepage_uri: http://hadoop.apache.org/core/\n  name: hadoop-0.20-source\n  section: misc\n  version: 0.20.2+923.256-1~maverick-cdh3\nhadoop-hive:\n  description: A data warehouse infrastructure built on top of Hadoop\n  downloads: 2\n  homepage_uri: http://hadoop.apache.org/hive/\n  name: hadoop-hive\n  section: misc\n  version: 0.7.1+42.43-2~maverick-cdh3\nhadoop-pig:\n  description: A platform for analyzing large data sets using Hadoop\n  downloads: 3\n  homepage_uri: http://hadoop.apache.org/pig/\n  name: hadoop-pig\n  section: misc\n  version: 0.8.1+28.32-1~maverick-cdh3\nhdparm:\n  description: tune hard disk parameters for high performance\n  downloads: 23393\n  homepage_uri: http://sourceforge.net/projects/hdparm/\n  name: hdparm\n  section: admin\n  version: 9.37-0ubuntu3.1\nhicolor-icon-theme:\n  description: default fallback theme for FreeDesktop.org icon themes\n  downloads: 33911\n  homepage_uri: ''\n  name: hicolor-icon-theme\n  section: misc\n  version: 0.12-1ubuntu2\nhmmer:\n  description: profile hidden Markov models for protein sequence analysis\n  downloads: 30\n  homepage_uri: http://hmmer.janelia.org/\n  name: hmmer\n  section: science\n  version: 3.0-3\nhostname:\n  description: utility to set/show the host name or domain name\n  downloads: 73725\n  homepage_uri: ''\n  name: hostname\n  section: admin\n  version: 3.06ubuntu1\nhplip:\n  description: HP Linux Printing and Imaging System (HPLIP)\n  downloads: 17838\n  homepage_uri: http://hplipopensource.com/hplip-web/index.html\n  name: hplip\n  section: utils\n  version: 3.12.2-1ubuntu3\nhplip-data:\n  description: HP Linux Printing and Imaging - data files\n  downloads: 140\n  homepage_uri: http://hplipopensource.com/hplip-web/index.html\n  name: hplip-data\n  section: utils\n  version: 3.12.2-1ubuntu3\nhtml2text:\n  description: advanced HTML to text converter\n  downloads: 15442\n  homepage_uri: http://www.mbayer.de/html2text/\n  name: html2text\n  section: web\n  version: 1.3.2a-15\nhtop:\n  description: interactive processes viewer\n  downloads: 8845\n  homepage_uri: http://htop.sourceforge.net\n  name: htop\n  section: universe/utils\n  version: 1.0.1-1\nhumanity-icon-theme:\n  description: Humanity Icon theme\n  downloads: 40\n  homepage_uri: ''\n  name: humanity-icon-theme\n  section: gnome\n  version: 0.5.3.11\nhunspell-en-us:\n  description: English_american dictionary for hunspell\n  downloads: 0\n  homepage_uri: ''\n  name: hunspell-en-us\n  section: text\n  version: 20070829-4ubuntu3\nhwdata:\n  description: hardware identification / configuration data\n  downloads: 3\n  homepage_uri: http://git.fedorahosted.org/git/hwdata.git\n  name: hwdata\n  section: misc\n  version: 0.233-1ubuntu1\nia32-libs:\n  description: ia32 shared libraries - transitional package\n  downloads: 701\n  homepage_uri: ''\n  name: ia32-libs\n  section: universe/libs\n  version: 20090808ubuntu36\nia32-libs-multiarch:i386:\n  description: Multi-arch versions of former ia32-libraries\n  downloads: 0\n  homepage_uri: ''\n  name: ia32-libs-multiarch:i386\n  section: universe/libs\n  version: 20090808ubuntu36\nibus:\n  description: Intelligent Input Bus - core\n  downloads: 679\n  homepage_uri: http://code.google.com/p/ibus/\n  name: ibus\n  section: utils\n  version: 1.4.1-3ubuntu1\nibus-gtk:\n  description: Intelligent Input Bus - GTK+2 support\n  downloads: 452\n  homepage_uri: http://code.google.com/p/ibus/\n  name: ibus-gtk\n  section: utils\n  version: 1.4.1-3ubuntu1\nibus-gtk3:\n  description: Intelligent Input Bus - GTK+3 support\n  downloads: 48\n  homepage_uri: http://code.google.com/p/ibus/\n  name: ibus-gtk3\n  section: utils\n  version: 1.4.1-3ubuntu1\nibus-gtk:i386:\n  description: Intelligent Input Bus - GTK+2 support\n  downloads: 0\n  homepage_uri: http://code.google.com/p/ibus/\n  name: ibus-gtk:i386\n  section: utils\n  version: 1.4.1-3ubuntu1\nibus-pinyin:\n  description: Pinyin engine for IBus\n  downloads: 178\n  homepage_uri: http://code.google.com/p/ibus\n  name: ibus-pinyin\n  section: utils\n  version: 1.4.0-1\nibus-pinyin-db-android:\n  description: Pinyin engine for IBus - Android database\n  downloads: 0\n  homepage_uri: http://code.google.com/p/ibus\n  name: ibus-pinyin-db-android\n  section: utils\n  version: 1.4.0-1\nibus-pinyin-db-open-phrase:\n  description: Pinyin engine for IBus - open-phrase database\n  downloads: 173\n  homepage_uri: http://code.google.com/p/ibus\n  name: ibus-pinyin-db-open-phrase\n  section: universe/utils\n  version: 1.4.0-1\nibus-table:\n  description: table engine for IBus\n  downloads: 119\n  homepage_uri: http://code.google.com/p/ibus\n  name: ibus-table\n  section: utils\n  version: 1.3.9.20110827-1ubuntu1\nicedtea-6-jre-cacao:\n  description: Alternative JVM for OpenJDK, using Cacao\n  downloads: 91\n  homepage_uri: http://openjdk.java.net/\n  name: icedtea-6-jre-cacao\n  section: interpreters\n  version: 6b24-1.11.1-4ubuntu3\nicedtea-6-jre-jamvm:\n  description: Alternative JVM for OpenJDK, using JamVM\n  downloads: 33\n  homepage_uri: http://openjdk.java.net/\n  name: icedtea-6-jre-jamvm\n  section: java\n  version: 6b24-1.11.1-4ubuntu3\nicedtea-7-jre-cacao:\n  description: Alternative JVM for OpenJDK, using Cacao\n  downloads: 2\n  homepage_uri: http://openjdk.java.net/\n  name: icedtea-7-jre-cacao\n  section: universe/java\n  version: 7~u3-2.1.1~pre1-1ubuntu3\nicedtea-7-jre-jamvm:\n  description: Alternative JVM for OpenJDK, using JamVM\n  downloads: 1\n  homepage_uri: http://openjdk.java.net/\n  name: icedtea-7-jre-jamvm\n  section: universe/java\n  version: 7~u3-2.1.1~pre1-1ubuntu3\nicedtea-netx:\n  description: NetX - implementation of the Java Network Launching Protocol (JNLP)\n  downloads: 343\n  homepage_uri: http://icedtea.classpath.org/wiki/IcedTea-Web\n  name: icedtea-netx\n  section: java\n  version: 1.2-2ubuntu1\nicedtea-netx-common:\n  description: NetX - implementation of the Java Network Launching Protocol (JNLP)\n  downloads: 427\n  homepage_uri: http://icedtea.classpath.org/wiki/IcedTea-Web\n  name: icedtea-netx-common\n  section: java\n  version: 1.2-2ubuntu1\nifupdown:\n  description: high level tools to configure network interfaces\n  downloads: 71985\n  homepage_uri: ''\n  name: ifupdown\n  section: admin\n  version: 0.7~beta2ubuntu8\nim-switch:\n  description: Input method switch framework\n  downloads: 191\n  homepage_uri: ''\n  name: im-switch\n  section: x11\n  version: 1.20ubuntu5\nimagemagick:\n  description: image manipulation programs\n  downloads: 23864\n  homepage_uri: http://www.imagemagick.org/\n  name: imagemagick\n  section: graphics\n  version: 8:6.6.9.7-5ubuntu3.1\nimagemagick-common:\n  description: image manipulation programs -- infrastructure\n  downloads: 0\n  homepage_uri: http://www.imagemagick.org/\n  name: imagemagick-common\n  section: graphics\n  version: 8:6.6.9.7-5ubuntu3.1\nindicator-applet-complete:\n  description: Clone of the GNOME panel indicator applet\n  downloads: 95\n  homepage_uri: https://launchpad.net/indicator-applet\n  name: indicator-applet-complete\n  section: universe/gnome\n  version: 0.5.0-0ubuntu1\nindicator-application:\n  description: Application Indicators\n  downloads: 39\n  homepage_uri: https://launchpad.net/indicator-application\n  name: indicator-application\n  section: gnome\n  version: 0.5.0-0ubuntu1\nindicator-appmenu:\n  description: Indicator for application menus.\n  downloads: 0\n  homepage_uri: http://launchpad.net/indicator-appmenu\n  name: indicator-appmenu\n  section: gnome\n  version: 0.3.97-0ubuntu1\nindicator-datetime:\n  description: Simple clock\n  downloads: 0\n  homepage_uri: https://launchpad.net/indicator-datetime\n  name: indicator-datetime\n  section: misc\n  version: 0.3.94-0ubuntu2\nindicator-messages:\n  description: indicator that collects messages that need a response\n  downloads: 132\n  homepage_uri: https://launchpad.net/indicator-messages\n  name: indicator-messages\n  section: gnome\n  version: 0.6.0-0ubuntu1\nindicator-power:\n  description: Indicator showing power state.\n  downloads: 0\n  homepage_uri: https://launchpad.net/indicator-power\n  name: indicator-power\n  section: gnome\n  version: 2.0-0ubuntu1\nindicator-printers:\n  description: indicator showing active print jobs\n  downloads: 0\n  homepage_uri: https://launchpad.net/indicator-printers\n  name: indicator-printers\n  section: gnome\n  version: 0.1.6-0ubuntu1\nindicator-session:\n  description: indicator showing session management, status and user switching\n  downloads: 45\n  homepage_uri: https://launchpad.net/indicator-session\n  name: indicator-session\n  section: gnome\n  version: 0.3.96-0ubuntu1\nindicator-sound:\n  description: System sound indicator.\n  downloads: 19\n  homepage_uri: https://launchpad.net/indicator-sound\n  name: indicator-sound\n  section: sound\n  version: 0.8.5.0-0ubuntu2.1\nindicator-status-provider-mc5:\n  description: indicator-messages status provider for telepathy mission-control-5\n  downloads: 6\n  homepage_uri: https://launchpad.net/indicator-messages\n  name: indicator-status-provider-mc5\n  section: gnome\n  version: 0.6.0-0ubuntu1\ninfo:\n  description: Standalone GNU Info documentation browser\n  downloads: 10379\n  homepage_uri: ''\n  name: info\n  section: doc\n  version: 4.13a.dfsg.1-8ubuntu2\ninitramfs-tools:\n  description: tools for generating an initramfs\n  downloads: 52983\n  homepage_uri: ''\n  name: initramfs-tools\n  section: utils\n  version: 0.99ubuntu13\ninitramfs-tools-bin:\n  description: binaries used by initramfs-tools\n  downloads: 28\n  homepage_uri: ''\n  name: initramfs-tools-bin\n  section: utils\n  version: 0.99ubuntu13\ninitscripts:\n  description: scripts for initializing and shutting down the system\n  downloads: 110492\n  homepage_uri: http://savannah.nongnu.org/projects/sysvinit\n  name: initscripts\n  section: admin\n  version: 2.88dsf-13.10ubuntu11\ninputattach:\n  description: utility to connect serial-attached peripherals to the input subsystem\n  downloads: 366\n  homepage_uri: https://sourceforge.net/projects/linuxconsole/\n  name: inputattach\n  section: utils\n  version: 1:1.4.2-1\ninsserv:\n  description: Tool to organize boot sequence using LSB init.d script dependencies\n  downloads: 85368\n  homepage_uri: http://savannah.nongnu.org/projects/sysvinit\n  name: insserv\n  section: misc\n  version: 1.14.0-2.1ubuntu2\ninstall-info:\n  description: Manage installed documentation in info format\n  downloads: 34204\n  homepage_uri: ''\n  name: install-info\n  section: doc\n  version: 4.13a.dfsg.1-8ubuntu2\nintel-gpu-tools:\n  description: tools for debugging the Intel graphics driver\n  downloads: 58\n  homepage_uri: http://www.intellinuxgraphics.org/\n  name: intel-gpu-tools\n  section: x11\n  version: 1.2-1\nintltool:\n  description: Utility scripts for internationalizing XML\n  downloads: 1275\n  homepage_uri: https://launchpad.net/intltool\n  name: intltool\n  section: devel\n  version: 0.50.2-2\nintltool-debian:\n  description: Help i18n of RFC822 compliant config files\n  downloads: 6883\n  homepage_uri: ''\n  name: intltool-debian\n  section: devel\n  version: 0.35.0+20060710.1\niproute:\n  description: networking and traffic control tools\n  downloads: 79306\n  homepage_uri: http://www.linux-foundation.org/en/Net:Iproute2\n  name: iproute\n  section: net\n  version: 20111117-1ubuntu2\niptables:\n  description: administration tools for packet filtering and NAT\n  downloads: 31867\n  homepage_uri: http://www.netfilter.org/\n  name: iptables\n  section: net\n  version: 1.4.12-1ubuntu4\niputils-arping:\n  description: Tool to send ICMP echo requests to an ARP address\n  downloads: 561\n  homepage_uri: ''\n  name: iputils-arping\n  section: net\n  version: 3:20101006-1ubuntu1\niputils-ping:\n  description: Tools to test the reachability of network hosts\n  downloads: 44649\n  homepage_uri: ''\n  name: iputils-ping\n  section: net\n  version: 3:20101006-1ubuntu1\niputils-tracepath:\n  description: Tools to trace the network path to a remote host\n  downloads: 1408\n  homepage_uri: ''\n  name: iputils-tracepath\n  section: net\n  version: 3:20101006-1ubuntu1\nirqbalance:\n  description: Daemon to balance interrupts for SMP systems\n  downloads: 832\n  homepage_uri: http://irqbalance.org/\n  name: irqbalance\n  section: utils\n  version: 0.56-1ubuntu4\nisc-dhcp-client:\n  description: ISC DHCP client\n  downloads: 52500\n  homepage_uri: ''\n  name: isc-dhcp-client\n  section: net\n  version: 4.1.ESV-R4-0ubuntu5.1\nisc-dhcp-common:\n  description: common files used by all the isc-dhcp* packages\n  downloads: 48869\n  homepage_uri: ''\n  name: isc-dhcp-common\n  section: net\n  version: 4.1.ESV-R4-0ubuntu5.1\niso-codes:\n  description: ISO language, territory, currency, script codes and their translations\n  downloads: 7814\n  homepage_uri: http://pkg-isocodes.alioth.debian.org/\n  name: iso-codes\n  section: libs\n  version: 3.31-1\niw:\n  description: tool for configuring Linux wireless devices\n  downloads: 962\n  homepage_uri: http://wireless.kernel.org/en/users/Documentation/iw\n  name: iw\n  section: net\n  version: 3.2-1\njarwrapper:\n  description: Run executable Java .jar files\n  downloads: 1853\n  homepage_uri: ''\n  name: jarwrapper\n  section: universe/libs\n  version: 0.40ubuntu1\njava-common:\n  description: Base of all Java packages\n  downloads: 24761\n  homepage_uri: ''\n  name: java-common\n  section: misc\n  version: 0.43ubuntu2\njava-wrappers:\n  description: wrappers for java executables\n  downloads: 2660\n  homepage_uri: ''\n  name: java-wrappers\n  section: misc\n  version: 0.1.24\njockey-common:\n  description: user interface and desktop integration for driver management\n  downloads: 7\n  homepage_uri: ''\n  name: jockey-common\n  section: admin\n  version: 0.9.7-0ubuntu7\njockey-gtk:\n  description: GNOME user interface and desktop integration for driver management\n  downloads: 5\n  homepage_uri: ''\n  name: jockey-gtk\n  section: admin\n  version: 0.9.7-0ubuntu7\njruby:\n  description: 100% pure-Java implementation of Ruby\n  downloads: 21\n  homepage_uri: http://jruby.org\n  name: jruby\n  section: universe/ruby\n  version: 1.5.6-2\njunit:\n  description: Automated testing framework for Java\n  downloads: 1687\n  homepage_uri: http://www.junit.org\n  name: junit\n  section: devel\n  version: 3.8.2-8\njunit4:\n  description: JUnit regression test framework for Java\n  downloads: 1460\n  homepage_uri: http://www.junit.org/\n  name: junit4\n  section: devel\n  version: 4.8.2-2\njython:\n  description: Python seamlessly integrated with Java\n  downloads: 96\n  homepage_uri: http://www.jython.org/\n  name: jython\n  section: universe/python\n  version: 2.5.1-2ubuntu2\nkbd:\n  description: Linux console font and keytable utilities\n  downloads: 47118\n  homepage_uri: ''\n  name: kbd\n  section: utils\n  version: 1.15.2-3ubuntu4\nkerneloops-daemon:\n  description: kernel oops tracker\n  downloads: 7220\n  homepage_uri: http://www.kerneloops.org/\n  name: kerneloops-daemon\n  section: utils\n  version: 0.12+git20090217-1ubuntu19\nkeyboard-configuration:\n  description: system-wide keyboard preferences\n  downloads: 48584\n  homepage_uri: ''\n  name: keyboard-configuration\n  section: utils\n  version: 1.70ubuntu5\nklibc-utils:\n  description: small utilities built with klibc for early boot\n  downloads: 32706\n  homepage_uri: ''\n  name: klibc-utils\n  section: libs\n  version: 1.5.25-1ubuntu2\nkpartx:\n  description: create device mappings for partitions\n  downloads: 1378\n  homepage_uri: http://christophe.varoqui.free.fr/\n  name: kpartx\n  section: admin\n  version: 0.4.9-3ubuntu5\nkrb5-locales:\n  description: Internationalization support for MIT Kerberos\n  downloads: 1\n  homepage_uri: http://web.mit.edu/kerberos/\n  name: krb5-locales\n  section: localization\n  version: 1.10+dfsg~beta1-2ubuntu0.1\nkrb5-multidev:\n  description: Development files for MIT Kerberos without Heimdal conflict\n  downloads: 11860\n  homepage_uri: http://web.mit.edu/kerberos/\n  name: krb5-multidev\n  section: libdevel\n  version: 1.10+dfsg~beta1-2ubuntu0.1\nkvm-ipxe:\n  description: PXE ROM's for KVM\n  downloads: 0\n  homepage_uri: http://ipxe.org/\n  name: kvm-ipxe\n  section: admin\n  version: 1.0.0+git-3.55f6c88-0ubuntu1\nlacheck:\n  description: Simple syntax checker for LaTeX\n  downloads: 1365\n  homepage_uri: http://www.ctan.org/tex-archive/support/lacheck/\n  name: lacheck\n  section: tex\n  version: 1.26-14\nlandscape-client:\n  description: The Landscape administration system client\n  downloads: 1\n  homepage_uri: ''\n  name: landscape-client\n  section: admin\n  version: 12.04.3-0ubuntu1\nlandscape-client-ui-install:\n  description: The Landscape administration system client - UI installer\n  downloads: 0\n  homepage_uri: ''\n  name: landscape-client-ui-install\n  section: admin\n  version: 12.04.3-0ubuntu1\nlandscape-common:\n  description: The Landscape administration system client - Common files\n  downloads: 3\n  homepage_uri: ''\n  name: landscape-common\n  section: admin\n  version: 12.04.3-0ubuntu1\nlanguage-selector-common:\n  description: Language selector for Ubuntu\n  downloads: 6\n  homepage_uri: ''\n  name: language-selector-common\n  section: admin\n  version: '0.79'\nlanguage-selector-gnome:\n  description: Language selector for Ubuntu\n  downloads: 0\n  homepage_uri: ''\n  name: language-selector-gnome\n  section: admin\n  version: '0.79'\nlaptop-detect:\n  description: attempt to detect a laptop\n  downloads: 7447\n  homepage_uri: ''\n  name: laptop-detect\n  section: utils\n  version: 0.13.7ubuntu2\nlast-align:\n  description: genome-scale comparison of biological sequences\n  downloads: 20\n  homepage_uri: http://last.cbrc.jp/\n  name: last-align\n  section: universe/science\n  version: 189-1\nlatex-beamer:\n  description: LaTeX class to produce presentations\n  downloads: 0\n  homepage_uri: ''\n  name: latex-beamer\n  section: tex\n  version: 3.10-1\nlatex-xcolor:\n  description: Easy driver-independent TeX class for color\n  downloads: 106\n  homepage_uri: ''\n  name: latex-xcolor\n  section: tex\n  version: 2.11-1\nlaunchpad-integration:\n  description: launchpad integration\n  downloads: 6\n  homepage_uri: ''\n  name: launchpad-integration\n  section: libs\n  version: 0.1.56\nless:\n  description: pager program similar to more\n  downloads: 54882\n  homepage_uri: http://www.greenwoodsoftware.com/less/\n  name: less\n  section: text\n  version: 444-1ubuntu1\nlesstif2:\n  description: OSF/Motif 2.1 implementation released under LGPL\n  downloads: 6241\n  homepage_uri: http://www.lesstif.org/\n  name: lesstif2\n  section: universe/libs\n  version: 1:0.95.2-1\nlibaa1:\n  description: ASCII art library\n  downloads: 38610\n  homepage_uri: http://aa-project.sourceforge.net/aalib/\n  name: libaa1\n  section: libs\n  version: 1.4p5-39ubuntu1\nlibaa1:i386:\n  description: ASCII art library\n  downloads: 0\n  homepage_uri: http://aa-project.sourceforge.net/aalib/\n  name: libaa1:i386\n  section: libs\n  version: 1.4p5-39ubuntu1\nlibaccountsservice0:\n  description: query and manipulate user account information - shared libraries\n  downloads: 8882\n  homepage_uri: http://cgit.freedesktop.org/accountsservice/\n  name: libaccountsservice0\n  section: libs\n  version: 0.6.15-2ubuntu9\nlibacl1:\n  description: Access control list shared library\n  downloads: 64410\n  homepage_uri: http://savannah.nongnu.org/projects/acl/\n  name: libacl1\n  section: libs\n  version: 2.2.51-5ubuntu1\nlibacl1:i386:\n  description: Access control list shared library\n  downloads: 0\n  homepage_uri: http://savannah.nongnu.org/projects/acl/\n  name: libacl1:i386\n  section: libs\n  version: 2.2.51-5ubuntu1\nlibaio1:\n  description: Linux kernel AIO access library - shared library\n  downloads: 8133\n  homepage_uri: http://www.kernel.org/pub/linux/libs/aio/\n  name: libaio1\n  section: libs\n  version: 0.3.109-2ubuntu1\nlibaio1:i386:\n  description: Linux kernel AIO access library - shared library\n  downloads: 0\n  homepage_uri: http://www.kernel.org/pub/linux/libs/aio/\n  name: libaio1:i386\n  section: libs\n  version: 0.3.109-2ubuntu1\nlibalgorithm-diff-perl:\n  description: module to find differences between files\n  downloads: 2116\n  homepage_uri: http://search.cpan.org/dist/Algorithm-Diff/\n  name: libalgorithm-diff-perl\n  section: perl\n  version: 1.19.02-2\nlibalgorithm-diff-xs-perl:\n  description: module to find differences between files (XS accelerated)\n  downloads: 1639\n  homepage_uri: http://search.cpan.org/dist/Algorithm-Diff-XS/\n  name: libalgorithm-diff-xs-perl\n  section: perl\n  version: 0.04-2build2\nlibalgorithm-merge-perl:\n  description: Perl module for three-way merge of textual data\n  downloads: 1540\n  homepage_uri: http://search.cpan.org/dist/Algorithm-Merge/\n  name: libalgorithm-merge-perl\n  section: perl\n  version: 0.08-2\nlibamd2.2.0:\n  description: approximate minimum degree ordering library for sparse matrices\n  downloads: 1188\n  homepage_uri: http://www.cise.ufl.edu/research/sparse/SuiteSparse/\n  name: libamd2.2.0\n  section: universe/libs\n  version: 1:3.4.0-2ubuntu3\nlibantlr-java:\n  description: language tool for constructing recognizers, compilers etc (java library)\n  downloads: 1621\n  homepage_uri: http://www.antlr2.org/\n  name: libantlr-java\n  section: devel\n  version: 2.7.7+dfsg-3\nlibao-common:\n  description: Cross Platform Audio Output Library (Common files)\n  downloads: 20028\n  homepage_uri: http://www.xiph.org/ao/\n  name: libao-common\n  section: libs\n  version: 1.1.0-1ubuntu2\nlibao4:i386:\n  description: Cross Platform Audio Output Library\n  downloads: 0\n  homepage_uri: http://www.xiph.org/ao/\n  name: libao4:i386\n  section: libs\n  version: 1.1.0-1ubuntu2\nlibapache-pom-java:\n  description: Maven metadata for all Apache Software projects\n  downloads: 414\n  homepage_uri: http://maven.apache.org/pom/asf/\n  name: libapache-pom-java\n  section: java\n  version: 10-2\nlibapparmor1:\n  description: changehat AppArmor library\n  downloads: 23\n  homepage_uri: http://apparmor.net/\n  name: libapparmor1\n  section: libs\n  version: 2.7.102-0ubuntu3.1\nlibappindicator1:\n  description: Application Indicators\n  downloads: 92\n  homepage_uri: https://launchpad.net/libappindicator\n  name: libappindicator1\n  section: libs\n  version: 0.4.92-0ubuntu1\nlibappindicator3-1:\n  description: Application Indicators\n  downloads: 23\n  homepage_uri: https://launchpad.net/libappindicator\n  name: libappindicator3-1\n  section: libs\n  version: 0.4.92-0ubuntu1\nlibapr1:\n  description: Apache Portable Runtime Library\n  downloads: 69515\n  homepage_uri: http://apr.apache.org/\n  name: libapr1\n  section: libs\n  version: 1.4.6-1\nlibaprutil1:\n  description: Apache Portable Runtime Utility Library\n  downloads: 63932\n  homepage_uri: http://apr.apache.org/\n  name: libaprutil1\n  section: libs\n  version: 1.3.12+dfsg-3\nlibaprutil1-dbd-sqlite3:\n  description: Apache Portable Runtime Utility Library - SQLite3 Driver\n  downloads: 50226\n  homepage_uri: http://apr.apache.org/\n  name: libaprutil1-dbd-sqlite3\n  section: libs\n  version: 1.3.12+dfsg-3\nlibaprutil1-ldap:\n  description: Apache Portable Runtime Utility Library - LDAP Driver\n  downloads: 51271\n  homepage_uri: http://apr.apache.org/\n  name: libaprutil1-ldap\n  section: libs\n  version: 1.3.12+dfsg-3\nlibapt-inst1.4:\n  description: deb package format runtime library\n  downloads: 3\n  homepage_uri: ''\n  name: libapt-inst1.4\n  section: admin\n  version: 0.8.16~exp12ubuntu10.2\nlibapt-pkg-perl:\n  description: Perl interface to libapt-pkg\n  downloads: 12808\n  homepage_uri: ''\n  name: libapt-pkg-perl\n  section: perl\n  version: 0.1.25build2\nlibapt-pkg4.12:\n  description: package managment runtime library\n  downloads: 15503\n  homepage_uri: ''\n  name: libapt-pkg4.12\n  section: admin\n  version: 0.8.16~exp12ubuntu10.2\nlibarchive12:\n  description: Multi-format archive and compression library (shared library)\n  downloads: 12762\n  homepage_uri: http://libarchive.github.com/\n  name: libarchive12\n  section: libs\n  version: 3.0.3-6ubuntu1\nlibarpack2:\n  description: Fortran77 subroutines to solve large scale eigenvalue problems\n  downloads: 53\n  homepage_uri: http://forge.scilab.org/index.php/p/arpack-ng/\n  name: libarpack2\n  section: universe/libs\n  version: 3.0.2-3\nlibart-2.0-2:\n  description: Library of functions for 2D graphics - runtime files\n  downloads: 35944\n  homepage_uri: ''\n  name: libart-2.0-2\n  section: libs\n  version: 2.3.21-1\nlibart-2.0-dev:\n  description: Library of functions for 2D graphics - development files\n  downloads: 403\n  homepage_uri: ''\n  name: libart-2.0-dev\n  section: libdevel\n  version: 2.3.21-1\nlibasm3-java:\n  description: Java bytecode manipulation framework\n  downloads: 1474\n  homepage_uri: http://asm.objectweb.org/\n  name: libasm3-java\n  section: devel\n  version: 3.3.2-1\nlibasn1-8-heimdal:\n  description: Heimdal Kerberos - ASN.1 library\n  downloads: 470\n  homepage_uri: http://www.h5l.org/\n  name: libasn1-8-heimdal\n  section: libs\n  version: 1.6~git20120311.dfsg.1-2\nlibasn1-8-heimdal:i386:\n  description: Heimdal Kerberos - ASN.1 library\n  downloads: 0\n  homepage_uri: http://www.h5l.org/\n  name: libasn1-8-heimdal:i386\n  section: libs\n  version: 1.6~git20120311.dfsg.1-2\nlibasound2:\n  description: shared library for ALSA applications\n  downloads: 53087\n  homepage_uri: http://www.alsa-project.org/\n  name: libasound2\n  section: libs\n  version: 1.0.25-1ubuntu10.1\nlibasound2-plugins:\n  description: ALSA library additional plugins\n  downloads: 7008\n  homepage_uri: http://www.alsa-project.org/\n  name: libasound2-plugins\n  section: libs\n  version: 1.0.25-1ubuntu1\nlibasound2-plugins:i386:\n  description: ALSA library additional plugins\n  downloads: 0\n  homepage_uri: http://www.alsa-project.org/\n  name: libasound2-plugins:i386\n  section: libs\n  version: 1.0.25-1ubuntu1\nlibasound2:i386:\n  description: shared library for ALSA applications\n  downloads: 0\n  homepage_uri: http://www.alsa-project.org/\n  name: libasound2:i386\n  section: libs\n  version: 1.0.25-1ubuntu10.1\nlibaspell-dev:\n  description: Development files for applications with GNU Aspell support\n  downloads: 124\n  homepage_uri: http://aspell.net/\n  name: libaspell-dev\n  section: libdevel\n  version: 0.60.7~20110707-1\nlibaspell15:\n  description: GNU Aspell spell-checker runtime library\n  downloads: 43298\n  homepage_uri: http://aspell.net/\n  name: libaspell15\n  section: libs\n  version: 0.60.7~20110707-1\nlibasyncns0:\n  description: Asynchronous name service query library\n  downloads: 30522\n  homepage_uri: http://0pointer.de/lennart/projects/libasyncns/\n  name: libasyncns0\n  section: libs\n  version: 0.8-4\nlibasyncns0:i386:\n  description: Asynchronous name service query library\n  downloads: 0\n  homepage_uri: http://0pointer.de/lennart/projects/libasyncns/\n  name: libasyncns0:i386\n  section: libs\n  version: 0.8-4\nlibatasmart4:\n  description: ATA S.M.A.R.T. reading and parsing library\n  downloads: 46706\n  homepage_uri: http://0pointer.de/blog/projects/being-smart.html\n  name: libatasmart4\n  section: libs\n  version: 0.18-3\nlibatk-adaptor:\n  description: at-spi2-atk\n  downloads: 490\n  homepage_uri: http://live.gnome.org/Accessibility/GNOME3\n  name: libatk-adaptor\n  section: misc\n  version: 2.4.0-1ubuntu2\nlibatk-adaptor-schemas:\n  description: at-spi2-atk GSettings schemas\n  downloads: 0\n  homepage_uri: http://live.gnome.org/Accessibility/GNOME3\n  name: libatk-adaptor-schemas\n  section: misc\n  version: 2.4.0-1ubuntu2\nlibatk-wrapper-java:\n  description: An ATK implementation for Java using JNI\n  downloads: 693\n  homepage_uri: http://ftp.gnome.org/pub/GNOME/sources/java-atk-wrapper/\n  name: libatk-wrapper-java\n  section: java\n  version: 0.30.4-0ubuntu2\nlibatk-wrapper-java-jni:\n  description: An ATK implementation for Java using JNI (jni bindings)\n  downloads: 692\n  homepage_uri: http://ftp.gnome.org/pub/GNOME/sources/java-atk-wrapper/\n  name: libatk-wrapper-java-jni\n  section: java\n  version: 0.30.4-0ubuntu2\nlibatk1.0-0:\n  description: ATK accessibility toolkit\n  downloads: 53350\n  homepage_uri: http://www.gtk.org/\n  name: libatk1.0-0\n  section: libs\n  version: 2.4.0-0ubuntu1\nlibatk1.0-0:i386:\n  description: ATK accessibility toolkit\n  downloads: 0\n  homepage_uri: http://www.gtk.org/\n  name: libatk1.0-0:i386\n  section: libs\n  version: 2.4.0-0ubuntu1\nlibatk1.0-data:\n  description: Common files for the ATK accessibility toolkit\n  downloads: 45828\n  homepage_uri: http://www.gtk.org/\n  name: libatk1.0-data\n  section: misc\n  version: 2.4.0-0ubuntu1\nlibatk1.0-dev:\n  description: Development files for the ATK accessibility toolkit\n  downloads: 3161\n  homepage_uri: http://www.gtk.org/\n  name: libatk1.0-dev\n  section: libdevel\n  version: 2.4.0-0ubuntu1\nlibatkmm-1.6-1:\n  description: C++ wrappers for ATK accessibility toolkit (shared libraries)\n  downloads: 6986\n  homepage_uri: http://www.gtkmm.org/\n  name: libatkmm-1.6-1\n  section: libs\n  version: 2.22.6-1ubuntu1\nlibatlas-base-dev:\n  description: Automatically Tuned Linear Algebra Software, generic static\n  downloads: 737\n  homepage_uri: http://math-atlas.sourceforge.net/\n  name: libatlas-base-dev\n  section: universe/devel\n  version: 3.8.4-3build1\nlibatlas-dev:\n  description: Automatically Tuned Linear Algebra Software, C header files\n  downloads: 713\n  homepage_uri: http://math-atlas.sourceforge.net/\n  name: libatlas-dev\n  section: universe/libdevel\n  version: 3.8.4-3build1\nlibatlas3gf-base:\n  description: Automatically Tuned Linear Algebra Software, generic shared\n  downloads: 2645\n  homepage_uri: http://math-atlas.sourceforge.net/\n  name: libatlas3gf-base\n  section: universe/libs\n  version: 3.8.4-3build1\nlibatspi2.0-0:\n  description: Assistive Technology Service Provider Interface - shared library\n  downloads: 654\n  homepage_uri: http://live.gnome.org/Accessibility/GNOME3\n  name: libatspi2.0-0\n  section: misc\n  version: 2.4.2-0ubuntu0.1\nlibattr1:\n  description: Extended attribute shared library\n  downloads: 65923\n  homepage_uri: http://savannah.nongnu.org/projects/attr/\n  name: libattr1\n  section: libs\n  version: 1:2.4.46-5ubuntu1\nlibattr1:i386:\n  description: Extended attribute shared library\n  downloads: 0\n  homepage_uri: http://savannah.nongnu.org/projects/attr/\n  name: libattr1:i386\n  section: libs\n  version: 1:2.4.46-5ubuntu1\nlibaudio2:\n  description: Network Audio System - shared libraries\n  downloads: 23866\n  homepage_uri: ''\n  name: libaudio2\n  section: libs\n  version: 1.9.3-4\nlibaudio2:i386:\n  description: Network Audio System - shared libraries\n  downloads: 0\n  homepage_uri: ''\n  name: libaudio2:i386\n  section: libs\n  version: 1.9.3-4\nlibaudiofile1:i386:\n  description: Open-source version of SGI's audiofile library\n  downloads: 0\n  homepage_uri: http://audiofile.68k.org/\n  name: libaudiofile1:i386\n  section: libs\n  version: 0.3.3-2\nlibavahi-client-dev:\n  description: Development files for the Avahi client library\n  downloads: 1442\n  homepage_uri: http://avahi.org/\n  name: libavahi-client-dev\n  section: net\n  version: 0.6.30-5ubuntu2\nlibavahi-client3:\n  description: Avahi client library\n  downloads: 52737\n  homepage_uri: http://avahi.org/\n  name: libavahi-client3\n  section: libs\n  version: 0.6.30-5ubuntu2\nlibavahi-client3:i386:\n  description: Avahi client library\n  downloads: 0\n  homepage_uri: http://avahi.org/\n  name: libavahi-client3:i386\n  section: libs\n  version: 0.6.30-5ubuntu2\nlibavahi-common-data:\n  description: Avahi common data files\n  downloads: 34905\n  homepage_uri: http://avahi.org/\n  name: libavahi-common-data\n  section: libs\n  version: 0.6.30-5ubuntu2\nlibavahi-common-data:i386:\n  description: Avahi common data files\n  downloads: 0\n  homepage_uri: http://avahi.org/\n  name: libavahi-common-data:i386\n  section: libs\n  version: 0.6.30-5ubuntu2\nlibavahi-common-dev:\n  description: Development files for the Avahi common library\n  downloads: 1321\n  homepage_uri: http://avahi.org/\n  name: libavahi-common-dev\n  section: net\n  version: 0.6.30-5ubuntu2\nlibavahi-common3:\n  description: Avahi common library\n  downloads: 55373\n  homepage_uri: http://avahi.org/\n  name: libavahi-common3\n  section: libs\n  version: 0.6.30-5ubuntu2\nlibavahi-common3:i386:\n  description: Avahi common library\n  downloads: 0\n  homepage_uri: http://avahi.org/\n  name: libavahi-common3:i386\n  section: libs\n  version: 0.6.30-5ubuntu2\nlibavahi-core7:\n  description: Avahi's embeddable mDNS/DNS-SD library\n  downloads: 43012\n  homepage_uri: http://avahi.org/\n  name: libavahi-core7\n  section: libs\n  version: 0.6.30-5ubuntu2\nlibavahi-glib-dev:\n  description: Development headers for the Avahi glib integration library\n  downloads: 371\n  homepage_uri: http://avahi.org/\n  name: libavahi-glib-dev\n  section: net\n  version: 0.6.30-5ubuntu2\nlibavahi-glib1:\n  description: Avahi glib integration library\n  downloads: 40876\n  homepage_uri: http://avahi.org/\n  name: libavahi-glib1\n  section: libs\n  version: 0.6.30-5ubuntu2\nlibavahi-gobject0:\n  description: Avahi GObject library\n  downloads: 16653\n  homepage_uri: http://avahi.org/\n  name: libavahi-gobject0\n  section: libs\n  version: 0.6.30-5ubuntu2\nlibavahi-ui-gtk3-0:\n  description: Avahi GTK+ User interface library for GTK3\n  downloads: 7410\n  homepage_uri: http://avahi.org/\n  name: libavahi-ui-gtk3-0\n  section: libs\n  version: 0.6.30-5ubuntu2\nlibavalon-framework-java:\n  description: Common framework for Java server applications\n  downloads: 669\n  homepage_uri: http://excalibur.apache.org/\n  name: libavalon-framework-java\n  section: libs\n  version: 4.2.0-8\nlibavc1394-0:\n  description: control IEEE 1394 audio/video devices\n  downloads: 33336\n  homepage_uri: ''\n  name: libavc1394-0\n  section: libs\n  version: 0.5.3-1ubuntu2\nlibavc1394-0:i386:\n  description: control IEEE 1394 audio/video devices\n  downloads: 0\n  homepage_uri: ''\n  name: libavc1394-0:i386\n  section: libs\n  version: 0.5.3-1ubuntu2\nlibavcodec53:\n  description: Libav codec library\n  downloads: 14915\n  homepage_uri: http://libav.org/\n  name: libavcodec53\n  section: libs\n  version: 4:0.8.1-0ubuntu1\nlibavformat53:\n  description: Libav file format library\n  downloads: 3444\n  homepage_uri: http://libav.org/\n  name: libavformat53\n  section: libs\n  version: 4:0.8.1-0ubuntu1\nlibavutil51:\n  description: Libav utility library\n  downloads: 10200\n  homepage_uri: http://libav.org/\n  name: libavutil51\n  section: libs\n  version: 4:0.8.1-0ubuntu1\nlibbackport-util-concurrent-java:\n  description: backport of java.util.concurrent to Java 1.4\n  downloads: 246\n  homepage_uri: http://backport-jsr166.sourceforge.net/\n  name: libbackport-util-concurrent-java\n  section: libs\n  version: 3.1-3\nlibbamf0:\n  description: Window matching library - shared library\n  downloads: 1\n  homepage_uri: https://launchpad.net/bamf\n  name: libbamf0\n  section: libs\n  version: 0.2.118-0ubuntu0.1\nlibbamf3-0:\n  description: Window matching library - shared library\n  downloads: 35\n  homepage_uri: https://launchpad.net/bamf\n  name: libbamf3-0\n  section: libs\n  version: 0.2.118-0ubuntu0.1\nlibbatik-java:\n  description: xml.apache.org SVG Library\n  downloads: 692\n  homepage_uri: http://xml.apache.org/batik/\n  name: libbatik-java\n  section: libs\n  version: 1.7.ubuntu-8ubuntu1\nlibbind9-80:\n  description: BIND9 Shared Library used by BIND\n  downloads: 15273\n  homepage_uri: ''\n  name: libbind9-80\n  section: libs\n  version: 1:9.8.1.dfsg.P1-4ubuntu0.1\nlibbiojava-java:\n  description: Java API to biological data and applications\n  downloads: 19\n  homepage_uri: http://www.biojava.org\n  name: libbiojava-java\n  section: universe/science\n  version: 1:1.7.1-1\nlibbiojava-java-demos:\n  description: Example programs for BioJava\n  downloads: 0\n  homepage_uri: http://www.biojava.org\n  name: libbiojava-java-demos\n  section: universe/science\n  version: 1:1.7.1-1\nlibbiojava1.7-java:\n  description: Java API to biological data and applications\n  downloads: 4\n  homepage_uri: http://www.biojava.org\n  name: libbiojava1.7-java\n  section: universe/science\n  version: 1:1.7.1-1\nlibblas-dev:\n  description: Basic Linear Algebra Subroutines 3, static library\n  downloads: 1678\n  homepage_uri: http://www.netlib.org/blas/\n  name: libblas-dev\n  section: libdevel\n  version: 1.2.20110419-2ubuntu1\nlibblas3gf:\n  description: Basic Linear Algebra Reference implementations, shared library\n  downloads: 31268\n  homepage_uri: http://www.netlib.org/blas/\n  name: libblas3gf\n  section: libs\n  version: 1.2.20110419-2ubuntu1\nlibblkid1:\n  description: block device id library\n  downloads: 39432\n  homepage_uri: http://userweb.kernel.org/~kzak/util-linux/\n  name: libblkid1\n  section: libs\n  version: 2.20.1-1ubuntu3\nlibbluetooth3:\n  description: Library to use the BlueZ Linux Bluetooth stack\n  downloads: 37668\n  homepage_uri: http://www.bluez.org\n  name: libbluetooth3\n  section: libs\n  version: 4.98-2ubuntu7\nlibbonobo2-0:\n  description: Bonobo CORBA interfaces library\n  downloads: 35551\n  homepage_uri: ''\n  name: libbonobo2-0\n  section: libs\n  version: 2.32.1-0ubuntu1\nlibbonobo2-common:\n  description: Bonobo CORBA interfaces library -- support files\n  downloads: 33480\n  homepage_uri: ''\n  name: libbonobo2-common\n  section: devel\n  version: 2.32.1-0ubuntu1\nlibbonobo2-dev:\n  description: Bonobo CORBA interfaces library -- development files\n  downloads: 196\n  homepage_uri: ''\n  name: libbonobo2-dev\n  section: libdevel\n  version: 2.32.1-0ubuntu1\nlibbonoboui2-0:\n  description: The Bonobo UI library\n  downloads: 30699\n  homepage_uri: ''\n  name: libbonoboui2-0\n  section: libs\n  version: 2.24.5-0ubuntu1\nlibbonoboui2-common:\n  description: The Bonobo UI library -- common files\n  downloads: 10\n  homepage_uri: ''\n  name: libbonoboui2-common\n  section: libs\n  version: 2.24.5-0ubuntu1\nlibbonoboui2-dev:\n  description: The Bonobo UI library - development files\n  downloads: 279\n  homepage_uri: ''\n  name: libbonoboui2-dev\n  section: libdevel\n  version: 2.24.5-0ubuntu1\nlibboost-all-dev:\n  description: Boost C++ Libraries development files (ALL, default version)\n  downloads: 0\n  homepage_uri: http://www.boost.org/\n  name: libboost-all-dev\n  section: universe/libdevel\n  version: 1.48.0.2\nlibboost-date-time-dev:\n  description: date-time libraries based on generic programming (default version)\n  downloads: 31\n  homepage_uri: http://www.boost.org/libs/date_time/\n  name: libboost-date-time-dev\n  section: universe/libdevel\n  version: 1.48.0.2\nlibboost-date-time1.46-dev:\n  description: set of date-time libraries based on generic programming concepts\n  downloads: 7\n  homepage_uri: http://www.boost.org/libs/date_time/\n  name: libboost-date-time1.46-dev\n  section: universe/libdevel\n  version: 1.46.1-7ubuntu3\nlibboost-date-time1.46.1:\n  description: set of date-time libraries based on generic programming concepts\n  downloads: 0\n  homepage_uri: http://www.boost.org/libs/date_time/\n  name: libboost-date-time1.46.1\n  section: universe/libs\n  version: 1.46.1-7ubuntu3\nlibboost-dev:\n  description: Boost C++ Libraries development files (default version)\n  downloads: 109\n  homepage_uri: http://www.boost.org/\n  name: libboost-dev\n  section: libdevel\n  version: 1.48.0.2\nlibboost-filesystem-dev:\n  description: filesystem operations in C++ (default version)\n  downloads: 22\n  homepage_uri: http://boost.org/libs/filesystem/\n  name: libboost-filesystem-dev\n  section: universe/libdevel\n  version: 1.48.0.2\nlibboost-filesystem1.46-dev:\n  description: filesystem operations (portable paths, iteration over directories,\n    etc) in C++\n  downloads: 5\n  homepage_uri: http://boost.org/libs/filesystem/\n  name: libboost-filesystem1.46-dev\n  section: universe/libdevel\n  version: 1.46.1-7ubuntu3\nlibboost-filesystem1.46.1:\n  description: filesystem operations (portable paths, iteration over directories,\n    etc) in C++\n  downloads: 14\n  homepage_uri: http://boost.org/libs/filesystem/\n  name: libboost-filesystem1.46.1\n  section: universe/libs\n  version: 1.46.1-7ubuntu3\nlibboost-graph-dev:\n  description: generic graph components and algorithms in C++ (default version)\n  downloads: 8\n  homepage_uri: http://www.boost.org/libs/graph/\n  name: libboost-graph-dev\n  section: universe/libdevel\n  version: 1.48.0.2\nlibboost-graph-parallel-dev:\n  description: generic graph components and algorithms in C++ (default version)\n  downloads: 0\n  homepage_uri: http://www.boost.org/libs/graph_parallel/\n  name: libboost-graph-parallel-dev\n  section: universe/libdevel\n  version: 1.48.0.2\nlibboost-graph-parallel1.46-dev:\n  description: generic graph components and algorithms in C++\n  downloads: 4\n  homepage_uri: http://www.boost.org/libs/graph_parallel/\n  name: libboost-graph-parallel1.46-dev\n  section: universe/libdevel\n  version: 1.46.1-7ubuntu3\nlibboost-graph-parallel1.46.1:\n  description: generic graph components and algorithms in C++\n  downloads: 0\n  homepage_uri: http://www.boost.org/libs/graph_parallel/\n  name: libboost-graph-parallel1.46.1\n  section: universe/libs\n  version: 1.46.1-7ubuntu3\nlibboost-graph1.46-dev:\n  description: generic graph components and algorithms in C++\n  downloads: 6\n  homepage_uri: http://www.boost.org/libs/graph/\n  name: libboost-graph1.46-dev\n  section: libdevel\n  version: 1.46.1-7ubuntu3\nlibboost-graph1.46.1:\n  description: generic graph components and algorithms in C++\n  downloads: 0\n  homepage_uri: http://www.boost.org/libs/graph/\n  name: libboost-graph1.46.1\n  section: libs\n  version: 1.46.1-7ubuntu3\nlibboost-iostreams-dev:\n  description: Boost.Iostreams Library development files (default version)\n  downloads: 12\n  homepage_uri: http://www.boost.org/libs/iostreams/\n  name: libboost-iostreams-dev\n  section: libdevel\n  version: 1.48.0.2\nlibboost-iostreams1.46-dev:\n  description: Boost.Iostreams Library development files\n  downloads: 5\n  homepage_uri: http://www.boost.org/libs/iostreams/\n  name: libboost-iostreams1.46-dev\n  section: libdevel\n  version: 1.46.1-7ubuntu3\nlibboost-iostreams1.46.1:\n  description: Boost.Iostreams Library\n  downloads: 0\n  homepage_uri: http://www.boost.org/libs/iostreams/\n  name: libboost-iostreams1.46.1\n  section: libs\n  version: 1.46.1-7ubuntu3\nlibboost-math-dev:\n  description: Boost.Math Library development files (default version)\n  downloads: 1\n  homepage_uri: http://www.boost.org/libs/math/\n  name: libboost-math-dev\n  section: universe/libdevel\n  version: 1.48.0.2\nlibboost-math1.46-dev:\n  description: Boost.Math Library development files\n  downloads: 5\n  homepage_uri: http://www.boost.org/libs/math/\n  name: libboost-math1.46-dev\n  section: universe/libdevel\n  version: 1.46.1-7ubuntu3\nlibboost-math1.46.1:\n  description: Boost.Math Library\n  downloads: 0\n  homepage_uri: http://www.boost.org/libs/math/\n  name: libboost-math1.46.1\n  section: universe/libs\n  version: 1.46.1-7ubuntu3\nlibboost-mpi-dev:\n  description: C++ interface to the Message Passing Interface (MPI) (default version)\n  downloads: 1\n  homepage_uri: http://www.boost.org/doc/html/mpi.html\n  name: libboost-mpi-dev\n  section: universe/libdevel\n  version: 1.48.0.2\nlibboost-mpi-python-dev:\n  description: Python interface to the Message Passing Interface (MPI) (default version)\n  downloads: 0\n  homepage_uri: http://www.boost.org/doc/html/mpi.html\n  name: libboost-mpi-python-dev\n  section: universe/libdevel\n  version: 1.48.0.2\nlibboost-mpi1.46-dev:\n  description: C++ interface to the Message Passing Interface (MPI)\n  downloads: 5\n  homepage_uri: http://www.boost.org/doc/html/mpi.html\n  name: libboost-mpi1.46-dev\n  section: universe/libdevel\n  version: 1.46.1-7ubuntu3\nlibboost-mpi1.46.1:\n  description: C++ interface to the Message Passing Interface (MPI)\n  downloads: 0\n  homepage_uri: http://www.boost.org/doc/html/mpi.html\n  name: libboost-mpi1.46.1\n  section: universe/libs\n  version: 1.46.1-7ubuntu3\nlibboost-program-options-dev:\n  description: program options library for C++ (default version)\n  downloads: 56\n  homepage_uri: http://www.boost.org/libs/program_options/\n  name: libboost-program-options-dev\n  section: libdevel\n  version: 1.48.0.2\nlibboost-program-options1.46-dev:\n  description: program options library for C++\n  downloads: 5\n  homepage_uri: http://www.boost.org/libs/program_options/\n  name: libboost-program-options1.46-dev\n  section: libdevel\n  version: 1.46.1-7ubuntu3\nlibboost-program-options1.46.1:\n  description: program options library for C++\n  downloads: 0\n  homepage_uri: http://www.boost.org/libs/program_options/\n  name: libboost-program-options1.46.1\n  section: libs\n  version: 1.46.1-7ubuntu3\nlibboost-python-dev:\n  description: Boost.Python Library development files (default version)\n  downloads: 17\n  homepage_uri: http://www.boost.org/libs/python/\n  name: libboost-python-dev\n  section: libdevel\n  version: 1.48.0.2\nlibboost-python1.46-dev:\n  description: Boost.Python Library development files\n  downloads: 22\n  homepage_uri: http://www.boost.org/libs/python/\n  name: libboost-python1.46-dev\n  section: libdevel\n  version: 1.46.1-7ubuntu3\nlibboost-python1.46.1:\n  description: Boost.Python Library\n  downloads: 0\n  homepage_uri: http://www.boost.org/libs/python/\n  name: libboost-python1.46.1\n  section: python\n  version: 1.46.1-7ubuntu3\nlibboost-regex-dev:\n  description: regular expression library for C++ (default version)\n  downloads: 15\n  homepage_uri: http://www.boost.org/libs/regex/\n  name: libboost-regex-dev\n  section: universe/libdevel\n  version: 1.48.0.2\nlibboost-regex1.46-dev:\n  description: regular expression library for C++\n  downloads: 5\n  homepage_uri: http://www.boost.org/libs/regex/\n  name: libboost-regex1.46-dev\n  section: libdevel\n  version: 1.46.1-7ubuntu3\nlibboost-regex1.46.1:\n  description: regular expression library for C++\n  downloads: 0\n  homepage_uri: http://www.boost.org/libs/regex/\n  name: libboost-regex1.46.1\n  section: libs\n  version: 1.46.1-7ubuntu3\nlibboost-serialization-dev:\n  description: serialization library for C++ (default version)\n  downloads: 14\n  homepage_uri: http://www.boost.org/libs/serialization/\n  name: libboost-serialization-dev\n  section: libdevel\n  version: 1.48.0.2\nlibboost-serialization1.46-dev:\n  description: serialization library for C++\n  downloads: 9\n  homepage_uri: http://www.boost.org/libs/serialization/\n  name: libboost-serialization1.46-dev\n  section: libdevel\n  version: 1.46.1-7ubuntu3\nlibboost-serialization1.46.1:\n  description: serialization library for C++\n  downloads: 0\n  homepage_uri: http://www.boost.org/libs/serialization/\n  name: libboost-serialization1.46.1\n  section: libs\n  version: 1.46.1-7ubuntu3\nlibboost-signals-dev:\n  description: managed signals and slots library for C++ (default version)\n  downloads: 12\n  homepage_uri: http://www.boost.org/libs/signals/\n  name: libboost-signals-dev\n  section: universe/libdevel\n  version: 1.48.0.2\nlibboost-signals1.46-dev:\n  description: managed signals and slots library for C++\n  downloads: 6\n  homepage_uri: http://www.boost.org/libs/signals/\n  name: libboost-signals1.46-dev\n  section: universe/libdevel\n  version: 1.46.1-7ubuntu3\nlibboost-signals1.46.1:\n  description: managed signals and slots library for C++\n  downloads: 0\n  homepage_uri: http://www.boost.org/libs/signals/\n  name: libboost-signals1.46.1\n  section: universe/libs\n  version: 1.46.1-7ubuntu3\nlibboost-system-dev:\n  description: Operating system (e.g. diagnostics support) library (default version)\n  downloads: 6\n  homepage_uri: http://www.boost.org/libs/system/\n  name: libboost-system-dev\n  section: universe/libdevel\n  version: 1.48.0.2\nlibboost-system1.46-dev:\n  description: Operating system (e.g. diagnostics support) library\n  downloads: 6\n  homepage_uri: http://www.boost.org/libs/system/\n  name: libboost-system1.46-dev\n  section: universe/libdevel\n  version: 1.46.1-7ubuntu3\nlibboost-system1.46.1:\n  description: Operating system (e.g. diagnostics support) library\n  downloads: 14\n  homepage_uri: http://www.boost.org/libs/system/\n  name: libboost-system1.46.1\n  section: universe/libs\n  version: 1.46.1-7ubuntu3\nlibboost-test-dev:\n  description: components for writing and executing test suites (default version)\n  downloads: 7\n  homepage_uri: http://www.boost.org/libs/test/\n  name: libboost-test-dev\n  section: libdevel\n  version: 1.48.0.2\nlibboost-test1.46-dev:\n  description: components for writing and executing test suites\n  downloads: 7\n  homepage_uri: http://www.boost.org/libs/test/\n  name: libboost-test1.46-dev\n  section: libdevel\n  version: 1.46.1-7ubuntu3\nlibboost-test1.46.1:\n  description: components for writing and executing test suites\n  downloads: 0\n  homepage_uri: http://www.boost.org/libs/test/\n  name: libboost-test1.46.1\n  section: libs\n  version: 1.46.1-7ubuntu3\nlibboost-thread-dev:\n  description: portable C++ multi-threading (default version)\n  downloads: 82\n  homepage_uri: http://www.boost.org/libs/thread/\n  name: libboost-thread-dev\n  section: universe/libdevel\n  version: 1.48.0.2\nlibboost-thread1.46-dev:\n  description: portable C++ multi-threading\n  downloads: 7\n  homepage_uri: http://www.boost.org/libs/thread/\n  name: libboost-thread1.46-dev\n  section: universe/libdevel\n  version: 1.46.1-7ubuntu3\nlibboost-thread1.46.1:\n  description: portable C++ multi-threading\n  downloads: 1\n  homepage_uri: http://www.boost.org/libs/thread/\n  name: libboost-thread1.46.1\n  section: universe/libs\n  version: 1.46.1-7ubuntu3\nlibboost-wave-dev:\n  description: C99/C++ preprocessor library (default version)\n  downloads: 7\n  homepage_uri: http://www.boost.org/libs/wave/\n  name: libboost-wave-dev\n  section: universe/libdevel\n  version: 1.48.0.2\nlibboost-wave1.46-dev:\n  description: C99/C++ preprocessor library\n  downloads: 5\n  homepage_uri: http://www.boost.org/libs/wave/\n  name: libboost-wave1.46-dev\n  section: universe/libdevel\n  version: 1.46.1-7ubuntu3\nlibboost-wave1.46.1:\n  description: C99/C++ preprocessor library\n  downloads: 0\n  homepage_uri: http://www.boost.org/libs/wave/\n  name: libboost-wave1.46.1\n  section: universe/libs\n  version: 1.46.1-7ubuntu3\nlibboost1.46-dev:\n  description: Boost C++ Libraries development files\n  downloads: 39\n  homepage_uri: http://www.boost.org/\n  name: libboost1.46-dev\n  section: libdevel\n  version: 1.46.1-7ubuntu3\nlibbrasero-media3-1:\n  description: CD/DVD burning library for GNOME - runtime\n  downloads: 6990\n  homepage_uri: http://www.gnome.org/projects/brasero/\n  name: libbrasero-media3-1\n  section: libs\n  version: 3.4.1-0ubuntu1\nlibbrlapi0.5:\n  description: braille display access via BRLTTY - shared library\n  downloads: 9379\n  homepage_uri: http://mielke.cc/brltty/\n  name: libbrlapi0.5\n  section: libs\n  version: 4.3-1ubuntu5\nlibbsd-dev:\n  description: utility functions from BSD systems - development files\n  downloads: 391\n  homepage_uri: http://libbsd.freedesktop.org/\n  name: libbsd-dev\n  section: libdevel\n  version: 0.3.0-2\nlibbsd0:\n  description: utility functions from BSD systems - shared library\n  downloads: 39621\n  homepage_uri: http://libbsd.freedesktop.org/\n  name: libbsd0\n  section: libs\n  version: 0.3.0-2\nlibbsf-java:\n  description: Bean Scripting Framework to support scripting languages in Java\n  downloads: 725\n  homepage_uri: http://jakarta.apache.org/bsf/\n  name: libbsf-java\n  section: libs\n  version: 1:2.4.0-5\nlibburn4:\n  description: library to provide CD/DVD writing functions\n  downloads: 12911\n  homepage_uri: http://libburnia-project.org\n  name: libburn4\n  section: libs\n  version: 1.1.8-1\nlibbytecode-java:\n  description: Java bytecode manipulation library\n  downloads: 19\n  homepage_uri: ''\n  name: libbytecode-java\n  section: universe/devel\n  version: 0.92.svn.20090106-1\nlibbz2-1.0:\n  description: high-quality block-sorting file compressor library - runtime\n  downloads: 104583\n  homepage_uri: http://www.bzip.org/\n  name: libbz2-1.0\n  section: libs\n  version: 1.0.6-1\nlibbz2-1.0:i386:\n  description: high-quality block-sorting file compressor library - runtime\n  downloads: 0\n  homepage_uri: http://www.bzip.org/\n  name: libbz2-1.0:i386\n  section: libs\n  version: 1.0.6-1\nlibbz2-dev:\n  description: high-quality block-sorting file compressor library - development\n  downloads: 1473\n  homepage_uri: http://www.bzip.org/\n  name: libbz2-dev\n  section: libdevel\n  version: 1.0.6-1\nlibc-bin:\n  description: 'Embedded GNU C Library: Binaries'\n  downloads: 93979\n  homepage_uri: http://www.eglibc.org\n  name: libc-bin\n  section: libs\n  version: 2.15-0ubuntu10\nlibc-dev-bin:\n  description: 'Embedded GNU C Library: Development binaries'\n  downloads: 33548\n  homepage_uri: http://www.eglibc.org\n  name: libc-dev-bin\n  section: libdevel\n  version: 2.15-0ubuntu10\nlibc6:\n  description: 'Embedded GNU C Library: Shared libraries'\n  downloads: 117876\n  homepage_uri: http://www.eglibc.org\n  name: libc6\n  section: libs\n  version: 2.15-0ubuntu10\nlibc6-dev:\n  description: 'Embedded GNU C Library: Development Libraries and Header Files'\n  downloads: 52134\n  homepage_uri: http://www.eglibc.org\n  name: libc6-dev\n  section: libdevel\n  version: 2.15-0ubuntu10\nlibc6:i386:\n  description: 'Embedded GNU C Library: Shared libraries'\n  downloads: 0\n  homepage_uri: http://www.eglibc.org\n  name: libc6:i386\n  section: libs\n  version: 2.15-0ubuntu10\nlibcaca0:\n  description: colour ASCII art library\n  downloads: 32988\n  homepage_uri: http://caca.zoy.org/wiki/libcaca\n  name: libcaca0\n  section: libs\n  version: 0.99.beta17-2.1ubuntu2\nlibcaca0:i386:\n  description: colour ASCII art library\n  downloads: 0\n  homepage_uri: http://caca.zoy.org/wiki/libcaca\n  name: libcaca0:i386\n  section: libs\n  version: 0.99.beta17-2.1ubuntu2\nlibcache-cache-perl:\n  description: Managed caches of persistent information\n  downloads: 700\n  homepage_uri: http://search.cpan.org/dist/Cache-Cache/\n  name: libcache-cache-perl\n  section: universe/perl\n  version: 1.06-2\nlibcairo-gobject2:\n  description: The Cairo 2D vector graphics library (GObject library)\n  downloads: 16269\n  homepage_uri: http://cairographics.org/\n  name: libcairo-gobject2\n  section: libs\n  version: 1.10.2-6.1ubuntu3\nlibcairo-gobject2:i386:\n  description: The Cairo 2D vector graphics library (GObject library)\n  downloads: 0\n  homepage_uri: http://cairographics.org/\n  name: libcairo-gobject2:i386\n  section: libs\n  version: 1.10.2-6.1ubuntu3\nlibcairo-perl:\n  description: Perl interface to the Cairo graphics library\n  downloads: 17618\n  homepage_uri: http://gtk2-perl.sourceforge.net/\n  name: libcairo-perl\n  section: perl\n  version: 1.081-1build2\nlibcairo-script-interpreter2:\n  description: The Cairo 2D vector graphics library (script interpreter)\n  downloads: 1611\n  homepage_uri: http://cairographics.org/\n  name: libcairo-script-interpreter2\n  section: libs\n  version: 1.10.2-6.1ubuntu3\nlibcairo2:\n  description: The Cairo 2D vector graphics library\n  downloads: 59418\n  homepage_uri: http://cairographics.org/\n  name: libcairo2\n  section: libs\n  version: 1.10.2-6.1ubuntu3\nlibcairo2-dev:\n  description: Development files for the Cairo 2D graphics library\n  downloads: 3292\n  homepage_uri: http://cairographics.org/\n  name: libcairo2-dev\n  section: libdevel\n  version: 1.10.2-6.1ubuntu3\nlibcairo2:i386:\n  description: The Cairo 2D vector graphics library\n  downloads: 0\n  homepage_uri: http://cairographics.org/\n  name: libcairo2:i386\n  section: libs\n  version: 1.10.2-6.1ubuntu3\nlibcairomm-1.0-1:\n  description: C++ wrappers for Cairo (shared libraries)\n  downloads: 18118\n  homepage_uri: http://cairographics.org/cairomm/\n  name: libcairomm-1.0-1\n  section: libs\n  version: 1.10.0-1ubuntu1\nlibcamel-1.2-29:\n  description: Evolution MIME message handling library\n  downloads: 0\n  homepage_uri: http://www.gnome.org/projects/evolution/\n  name: libcamel-1.2-29\n  section: libs\n  version: 3.2.3-0ubuntu7\nlibcanberra-dev:\n  description: simple abstract interface for playing event sounds\n  downloads: 200\n  homepage_uri: http://0pointer.de/lennart/projects/libcanberra/\n  name: libcanberra-dev\n  section: libdevel\n  version: 0.28-3ubuntu3\nlibcanberra-gtk-module:\n  description: translates GTK+ widgets signals to event sounds\n  downloads: 26234\n  homepage_uri: http://0pointer.de/lennart/projects/libcanberra/\n  name: libcanberra-gtk-module\n  section: libs\n  version: 0.28-3ubuntu3\nlibcanberra-gtk-module:i386:\n  description: translates GTK+ widgets signals to event sounds\n  downloads: 0\n  homepage_uri: http://0pointer.de/lennart/projects/libcanberra/\n  name: libcanberra-gtk-module:i386\n  section: libs\n  version: 0.28-3ubuntu3\nlibcanberra-gtk0:\n  description: GTK+ helper for playing widget event sounds with libcanberra\n  downloads: 36287\n  homepage_uri: http://0pointer.de/lennart/projects/libcanberra/\n  name: libcanberra-gtk0\n  section: libs\n  version: 0.28-3ubuntu3\nlibcanberra-gtk0:i386:\n  description: GTK+ helper for playing widget event sounds with libcanberra\n  downloads: 0\n  homepage_uri: http://0pointer.de/lennart/projects/libcanberra/\n  name: libcanberra-gtk0:i386\n  section: libs\n  version: 0.28-3ubuntu3\nlibcanberra-gtk3-0:\n  description: GTK+ 3.0 helper for playing widget event sounds with libcanberra\n  downloads: 11593\n  homepage_uri: http://0pointer.de/lennart/projects/libcanberra/\n  name: libcanberra-gtk3-0\n  section: libs\n  version: 0.28-3ubuntu3\nlibcanberra-gtk3-module:\n  description: translates GTK3 widgets signals to event sounds\n  downloads: 2502\n  homepage_uri: http://0pointer.de/lennart/projects/libcanberra/\n  name: libcanberra-gtk3-module\n  section: debug\n  version: 0.28-3ubuntu3\nlibcanberra-pulse:\n  description: PulseAudio backend for libcanberra\n  downloads: 2027\n  homepage_uri: http://0pointer.de/lennart/projects/libcanberra/\n  name: libcanberra-pulse\n  section: libdevel\n  version: 0.28-3ubuntu3\nlibcanberra0:\n  description: simple abstract interface for playing event sounds\n  downloads: 41526\n  homepage_uri: http://0pointer.de/lennart/projects/libcanberra/\n  name: libcanberra0\n  section: libs\n  version: 0.28-3ubuntu3\nlibcanberra0:i386:\n  description: simple abstract interface for playing event sounds\n  downloads: 0\n  homepage_uri: http://0pointer.de/lennart/projects/libcanberra/\n  name: libcanberra0:i386\n  section: libs\n  version: 0.28-3ubuntu3\nlibcap-ng0:\n  description: An alternate POSIX capabilities library\n  downloads: 43995\n  homepage_uri: http://people.redhat.com/sgrubb/libcap-ng\n  name: libcap-ng0\n  section: libs\n  version: 0.6.6-1ubuntu1\nlibcap2:\n  description: support for getting/setting POSIX.1e capabilities\n  downloads: 87126\n  homepage_uri: http://sites.google.com/site/fullycapable/\n  name: libcap2\n  section: libs\n  version: 1:2.22-1ubuntu3\nlibcap2-bin:\n  description: basic utility programs for using capabilities\n  downloads: 21861\n  homepage_uri: http://sites.google.com/site/fullycapable/\n  name: libcap2-bin\n  section: utils\n  version: 1:2.22-1ubuntu3\nlibcap2:i386:\n  description: support for getting/setting POSIX.1e capabilities\n  downloads: 0\n  homepage_uri: http://sites.google.com/site/fullycapable/\n  name: libcap2:i386\n  section: libs\n  version: 1:2.22-1ubuntu3\nlibcapi20-3:i386:\n  description: libraries for CAPI support\n  downloads: 0\n  homepage_uri: ''\n  name: libcapi20-3:i386\n  section: universe/libs\n  version: 1:3.12.20071127-0ubuntu11\nlibcbf0:\n  description: shared library supporting CBFlib\n  downloads: 284\n  homepage_uri: http://www.bernstein-plus-sons.com/software/CBF/\n  name: libcbf0\n  section: universe/libs\n  version: 0.7.9.1-3build1\nlibccolamd2.7.1:\n  description: constrained column approximate library for sparse matrices\n  downloads: 0\n  homepage_uri: http://www.cise.ufl.edu/research/sparse/SuiteSparse/\n  name: libccolamd2.7.1\n  section: universe/libs\n  version: 1:3.4.0-2ubuntu3\nlibcdio-cdda1:\n  description: library to read and control digital audio CDs\n  downloads: 6433\n  homepage_uri: http://www.gnu.org/software/libcdio/\n  name: libcdio-cdda1\n  section: libs\n  version: 0.83-1\nlibcdio-paranoia1:\n  description: library to read digital audio CDs with error correction\n  downloads: 6427\n  homepage_uri: http://www.gnu.org/software/libcdio/\n  name: libcdio-paranoia1\n  section: libs\n  version: 0.83-1\nlibcdio13:\n  description: library to read and control CD-ROM\n  downloads: 6890\n  homepage_uri: http://www.gnu.org/software/libcdio/\n  name: libcdio13\n  section: libs\n  version: 0.83-1\nlibcdparanoia0:\n  description: audio extraction tool for sampling CDs (library)\n  downloads: 27393\n  homepage_uri: http://www.xiph.org/paranoia/\n  name: libcdparanoia0\n  section: sound\n  version: 3.10.2+debian-10ubuntu1\nlibcdparanoia0:i386:\n  description: audio extraction tool for sampling CDs (library)\n  downloads: 0\n  homepage_uri: http://www.xiph.org/paranoia/\n  name: libcdparanoia0:i386\n  section: sound\n  version: 3.10.2+debian-10ubuntu1\nlibcdt4:\n  description: rich set of graph drawing tools - cdt library\n  downloads: 3832\n  homepage_uri: http://www.graphviz.org/\n  name: libcdt4\n  section: libs\n  version: 2.26.3-10ubuntu1\nlibcgi-application-perl:\n  description: framework for building reusable web-applications\n  downloads: 37\n  homepage_uri: http://search.cpan.org/dist/CGI-Application/\n  name: libcgi-application-perl\n  section: universe/perl\n  version: 4.50-1\nlibcgraph5:\n  description: rich set of graph drawing tools - cgraph library\n  downloads: 2227\n  homepage_uri: http://www.graphviz.org/\n  name: libcgraph5\n  section: libs\n  version: 2.26.3-10ubuntu1\nlibcholmod1.7.1:\n  description: sparse Cholesky factorization library for sparse matrices\n  downloads: 9\n  homepage_uri: http://www.cise.ufl.edu/research/sparse/SuiteSparse/\n  name: libcholmod1.7.1\n  section: universe/libs\n  version: 1:3.4.0-2ubuntu3\nlibck-connector0:\n  description: ConsoleKit libraries\n  downloads: 52459\n  homepage_uri: http://www.freedesktop.org/wiki/Software/ConsoleKit\n  name: libck-connector0\n  section: libs\n  version: 0.4.5-2\nlibclass-accessor-perl:\n  description: Perl module that automatically generates accessors\n  downloads: 3682\n  homepage_uri: http://search.cpan.org/dist/Class-Accessor/\n  name: libclass-accessor-perl\n  section: perl\n  version: 0.34-1\nlibclass-inspector-perl:\n  description: Perl module that provides information about classes\n  downloads: 1859\n  homepage_uri: http://search.cpan.org/dist/Class-Inspector/\n  name: libclass-inspector-perl\n  section: perl\n  version: 1.25-1\nlibclass-isa-perl:\n  description: report the search path for a class's ISA tree\n  downloads: 16211\n  homepage_uri: http://search.cpan.org/dist/Class-ISA/\n  name: libclass-isa-perl\n  section: perl\n  version: 0.36-3\nlibclassworlds-java:\n  description: framework for container developers requiring manipulation of ClassLoaders\n  downloads: 93\n  homepage_uri: http://classworlds.codehaus.org/\n  name: libclassworlds-java\n  section: universe/libs\n  version: 1.1-final-5\nlibclone-perl:\n  description: recursively copy Perl datatypes\n  downloads: 1427\n  homepage_uri: http://search.cpan.org/dist/Clone/\n  name: libclone-perl\n  section: perl\n  version: 0.31-1build3\nlibcmis-0.2-0:\n  description: CMIS protocol client library\n  downloads: 3\n  homepage_uri: ''\n  name: libcmis-0.2-0\n  section: libs\n  version: 0.1.0-1\nlibcolamd2.7.1:\n  description: column approximate minimum degree ordering library for sparse matrices\n  downloads: 4010\n  homepage_uri: http://www.cise.ufl.edu/research/sparse/SuiteSparse/\n  name: libcolamd2.7.1\n  section: universe/libs\n  version: 1:3.4.0-2ubuntu3\nlibcolord1:\n  description: system service to manage device colour profiles -- runtime\n  downloads: 15526\n  homepage_uri: http://www.freedesktop.org/software/colord/\n  name: libcolord1\n  section: libs\n  version: 0.1.16-2\nlibcomerr2:\n  description: common error description library\n  downloads: 104955\n  homepage_uri: http://e2fsprogs.sourceforge.net\n  name: libcomerr2\n  section: libs\n  version: 1.42-1ubuntu2\nlibcomerr2:i386:\n  description: common error description library\n  downloads: 0\n  homepage_uri: http://e2fsprogs.sourceforge.net\n  name: libcomerr2:i386\n  section: libs\n  version: 1.42-1ubuntu2\nlibcommon-sense-perl:\n  description: module that implements some sane defaults for Perl programs\n  downloads: 2068\n  homepage_uri: http://search.cpan.org/dist/common-sense/\n  name: libcommon-sense-perl\n  section: perl\n  version: 3.4-1\nlibcommons-beanutils-java:\n  description: utility for manipulating JavaBeans\n  downloads: 92\n  homepage_uri: http://commons.apache.org/beanutils/\n  name: libcommons-beanutils-java\n  section: universe/libs\n  version: 1.8.3-2\nlibcommons-cli-java:\n  description: API for working with the command line arguments and options\n  downloads: 2015\n  homepage_uri: http://commons.apache.org/cli/\n  name: libcommons-cli-java\n  section: libs\n  version: 1.2-3\nlibcommons-codec-java:\n  description: encoder and decoders such as Base64 and hexadecimal codec\n  downloads: 2317\n  homepage_uri: http://commons.apache.org/codec/\n  name: libcommons-codec-java\n  section: libs\n  version: 1.5-1\nlibcommons-collections-java:\n  description: set of abstract data type interfaces and implementations\n  downloads: 92\n  homepage_uri: http://commons.apache.org/collections/\n  name: libcommons-collections-java\n  section: libs\n  version: 2.1.1-10\nlibcommons-collections3-java:\n  description: A set of abstract data type interfaces and implementations\n  downloads: 67\n  homepage_uri: http://commons.apache.org/collections/\n  name: libcommons-collections3-java\n  section: libs\n  version: 3.2.1-5\nlibcommons-compress-java:\n  description: Java API for working with tar, zip and bzip2 files\n  downloads: 94\n  homepage_uri: http://commons.apache.org/compress/\n  name: libcommons-compress-java\n  section: libs\n  version: 1.2-1ubuntu1\nlibcommons-configuration-java:\n  description: Java based library providing a generic configuration interface\n  downloads: 28\n  homepage_uri: http://commons.apache.org/configuration/\n  name: libcommons-configuration-java\n  section: universe/libs\n  version: 1.7-1\nlibcommons-dbcp-java:\n  description: Database Connection Pooling Services\n  downloads: 778\n  homepage_uri: http://commons.apache.org/dbcp/\n  name: libcommons-dbcp-java\n  section: libs\n  version: 1.4-1ubuntu1\nlibcommons-digester-java:\n  description: Rule based XML Java object mapping tool\n  downloads: 19\n  homepage_uri: http://commons.apache.org/digester/\n  name: libcommons-digester-java\n  section: universe/libs\n  version: 1.8.1-3\nlibcommons-el-java:\n  description: Implementation of the JSP2.0 Expression Language interpreter\n  downloads: 9\n  homepage_uri: http://commons.apache.org/el/\n  name: libcommons-el-java\n  section: universe/libs\n  version: 1.0-7\nlibcommons-httpclient-java:\n  description: A Java(TM) library for creating HTTP clients\n  downloads: 2067\n  homepage_uri: http://hc.apache.org/httpclient-3.x\n  name: libcommons-httpclient-java\n  section: libs\n  version: 3.1-10\nlibcommons-io-java:\n  description: Common useful IO related classes\n  downloads: 708\n  homepage_uri: http://commons.apache.org/io/\n  name: libcommons-io-java\n  section: libs\n  version: 1.4-4\nlibcommons-jexl-java:\n  description: expression language engine\n  downloads: 15\n  homepage_uri: http://commons.apache.org/jexl\n  name: libcommons-jexl-java\n  section: universe/libs\n  version: 1.1-3\nlibcommons-jxpath-java:\n  description: manipulate javabean using XPath syntax\n  downloads: 0\n  homepage_uri: http://commons.apache.org/jxpath/\n  name: libcommons-jxpath-java\n  section: universe/libs\n  version: 1.3-5\nlibcommons-lang-java:\n  description: Extension of the java.lang package\n  downloads: 1741\n  homepage_uri: http://commons.apache.org/lang/\n  name: libcommons-lang-java\n  section: libs\n  version: 2.6-3ubuntu1\nlibcommons-logging-java:\n  description: commmon wrapper interface for several logging APIs\n  downloads: 3368\n  homepage_uri: http://commons.apache.org/logging\n  name: libcommons-logging-java\n  section: libs\n  version: 1.1.1-9\nlibcommons-net2-java:\n  description: internet protocol suite Java library\n  downloads: 23\n  homepage_uri: http://commons.apache.org/net/\n  name: libcommons-net2-java\n  section: java\n  version: 2.2-1ubuntu1\nlibcommons-parent-java:\n  description: Maven metadata for Apache Commons project\n  downloads: 1527\n  homepage_uri: http://commons.apache.org/\n  name: libcommons-parent-java\n  section: java\n  version: 22-2\nlibcommons-pool-java:\n  description: pooling implementation for Java objects\n  downloads: 779\n  homepage_uri: http://commons.apache.org/pool/\n  name: libcommons-pool-java\n  section: libs\n  version: 1.5.6-1\nlibcommons-vfs-java:\n  description: Java API for accessing various filesystems\n  downloads: 0\n  homepage_uri: http://commons.apache.org/vfs/\n  name: libcommons-vfs-java\n  section: libs\n  version: 2.0-1ubuntu1\nlibcompizconfig0:\n  description: Settings library for plugins - OpenCompositing Project\n  downloads: 2041\n  homepage_uri: ''\n  name: libcompizconfig0\n  section: libs\n  version: 0.9.7.0~bzr428-0ubuntu6\nlibconfig-inifiles-perl:\n  description: Read .ini-style configuration files\n  downloads: 1804\n  homepage_uri: http://search.cpan.org/dist/Config-IniFiles/\n  name: libconfig-inifiles-perl\n  section: perl\n  version: 2.68-1\nlibconstantine-java:\n  description: platform constants for Java\n  downloads: 87\n  homepage_uri: http://github.com/wmeissner/jnr-constants/\n  name: libconstantine-java\n  section: universe/java\n  version: 0.7-3\nlibconvert-binhex-perl:\n  description: Perl5 module for extracting data from macintosh BinHex files\n  downloads: 3721\n  homepage_uri: http://search.cpan.org/dist/Convert-BinHex/\n  name: libconvert-binhex-perl\n  section: perl\n  version: 1.119+pristine-3\nlibcr0:\n  description: Libraries to Checkpoint and Restart Linux processes\n  downloads: 248\n  homepage_uri: https://ftg.lbl.gov/CheckpointRestart/CheckpointRestart.shtml\n  name: libcr0\n  section: universe/libs\n  version: 0.8.2-15ubuntu2.1\nlibcroco3:\n  description: Cascading Style Sheet (CSS) parsing and manipulation toolkit\n  downloads: 28507\n  homepage_uri: ''\n  name: libcroco3\n  section: libs\n  version: 0.6.5-1\nlibcroco3:i386:\n  description: Cascading Style Sheet (CSS) parsing and manipulation toolkit\n  downloads: 0\n  homepage_uri: ''\n  name: libcroco3:i386\n  section: libs\n  version: 0.6.5-1\nlibcrypt-passwdmd5-perl:\n  description: interoperable MD5-based crypt() for perl\n  downloads: 1454\n  homepage_uri: http://search.cpan.org/dist/Crypt-PasswdMD5/\n  name: libcrypt-passwdmd5-perl\n  section: perl\n  version: 1.3-10\nlibcrypt-ssleay-perl:\n  description: Support for https protocol in LWP\n  downloads: 3613\n  homepage_uri: http://search.cpan.org/dist/Crypt-SSLeay/\n  name: libcrypt-ssleay-perl\n  section: universe/perl\n  version: 0.57-2ubuntu1\nlibcryptsetup4:\n  description: disk encryption support - shared library\n  downloads: 2655\n  homepage_uri: http://code.google.com/p/cryptsetup/\n  name: libcryptsetup4\n  section: libs\n  version: 2:1.4.1-2ubuntu4\nlibcups2:\n  description: Common UNIX Printing System(tm) - Core library\n  downloads: 56561\n  homepage_uri: http://www.cups.org\n  name: libcups2\n  section: libs\n  version: 1.5.3-0ubuntu1\nlibcups2:i386:\n  description: Common UNIX Printing System(tm) - Core library\n  downloads: 0\n  homepage_uri: http://www.cups.org\n  name: libcups2:i386\n  section: libs\n  version: 1.5.3-0ubuntu1\nlibcupscgi1:\n  description: Common UNIX Printing System(tm) - CGI library\n  downloads: 41402\n  homepage_uri: http://www.cups.org\n  name: libcupscgi1\n  section: libs\n  version: 1.5.3-0ubuntu1\nlibcupsdriver1:\n  description: Common UNIX Printing System(tm) - Driver library\n  downloads: 255\n  homepage_uri: http://www.cups.org\n  name: libcupsdriver1\n  section: libs\n  version: 1.5.3-0ubuntu1\nlibcupsfilters1:\n  description: OpenPrinting CUPS Filters - Shared library\n  downloads: 5114\n  homepage_uri: http://www.openprinting.org/\n  name: libcupsfilters1\n  section: libs\n  version: 1.0.18-0ubuntu0.1\nlibcupsimage2:\n  description: Common UNIX Printing System(tm) - Raster image library\n  downloads: 45416\n  homepage_uri: http://www.cups.org\n  name: libcupsimage2\n  section: libs\n  version: 1.5.3-0ubuntu1\nlibcupsimage2:i386:\n  description: Common UNIX Printing System(tm) - Raster image library\n  downloads: 0\n  homepage_uri: http://www.cups.org\n  name: libcupsimage2:i386\n  section: libs\n  version: 1.5.3-0ubuntu1\nlibcupsmime1:\n  description: Common UNIX Printing System(tm) - MIME library\n  downloads: 41415\n  homepage_uri: http://www.cups.org\n  name: libcupsmime1\n  section: libs\n  version: 1.5.3-0ubuntu1\nlibcupsppdc1:\n  description: Common UNIX Printing System(tm) - PPD manipulation library\n  downloads: 41493\n  homepage_uri: http://www.cups.org\n  name: libcupsppdc1\n  section: libs\n  version: 1.5.3-0ubuntu1\nlibcurl3:\n  description: Multi-protocol file transfer library (OpenSSL)\n  downloads: 27070\n  homepage_uri: http://curl.haxx.se\n  name: libcurl3\n  section: libs\n  version: 7.22.0-3ubuntu4\nlibcurl3-gnutls:\n  description: Multi-protocol file transfer library (GnuTLS)\n  downloads: 46882\n  homepage_uri: http://curl.haxx.se\n  name: libcurl3-gnutls\n  section: libs\n  version: 7.22.0-3ubuntu4\nlibcurl3-nss:\n  description: Multi-protocol file transfer library (NSS)\n  downloads: 1287\n  homepage_uri: http://curl.haxx.se\n  name: libcurl3-nss\n  section: libs\n  version: 7.22.0-3ubuntu4\nlibcurl3:i386:\n  description: Multi-protocol file transfer library (OpenSSL)\n  downloads: 0\n  homepage_uri: http://curl.haxx.se\n  name: libcurl3:i386\n  section: libs\n  version: 7.22.0-3ubuntu4\nlibcurl4-openssl-dev:\n  description: Development files and documentation for libcurl (OpenSSL)\n  downloads: 887\n  homepage_uri: http://curl.haxx.se\n  name: libcurl4-openssl-dev\n  section: libdevel\n  version: 7.22.0-3ubuntu4\nlibcvector2:\n  description: ANSI C implementation of dynamic arrays\n  downloads: 281\n  homepage_uri: http://sourceforge.net/projects/cvector/\n  name: libcvector2\n  section: universe/libs\n  version: 1.0.3-1\nlibcwidget3:\n  description: high-level terminal interface library for C++ (runtime files)\n  downloads: 28117\n  homepage_uri: http://cwidget.alioth.debian.org\n  name: libcwidget3\n  section: libs\n  version: 0.5.16-3.1ubuntu1\nlibcxsparse2.2.3:\n  description: concise sparse matrix library (complex, int and long int support)\n  downloads: 0\n  homepage_uri: http://www.cise.ufl.edu/research/sparse/SuiteSparse/\n  name: libcxsparse2.2.3\n  section: universe/libs\n  version: 1:3.4.0-2ubuntu3\nlibcxxtools7:\n  description: library of unrelated but useful C++ classes\n  downloads: 0\n  homepage_uri: http://www.tntnet.org/cxxtools.html\n  name: libcxxtools7\n  section: universe/libs\n  version: 2.0-4ubuntu2\nlibdaemon0:\n  description: lightweight C library for daemons - runtime library\n  downloads: 48519\n  homepage_uri: http://0pointer.de/lennart/projects/libdaemon/\n  name: libdaemon0\n  section: libs\n  version: 0.14-2\nlibdatrie1:\n  description: Double-array trie library\n  downloads: 41416\n  homepage_uri: http://linux.thai.net/projects/libthai\n  name: libdatrie1\n  section: libs\n  version: 0.2.5-3\nlibdatrie1:i386:\n  description: Double-array trie library\n  downloads: 0\n  homepage_uri: http://linux.thai.net/projects/libthai\n  name: libdatrie1:i386\n  section: libs\n  version: 0.2.5-3\nlibdb-dev:\n  description: Berkeley Database Libraries [development]\n  downloads: 98\n  homepage_uri: ''\n  name: libdb-dev\n  section: libdevel\n  version: 5.1.4ubuntu1\nlibdb-java:\n  description: Berkeley Database Libraries for Java\n  downloads: 0\n  homepage_uri: ''\n  name: libdb-java\n  section: universe/java\n  version: 5.1.4ubuntu1\nlibdb-je-java:\n  description: Oracle Berkeley Database Java Edition\n  downloads: 12\n  homepage_uri: http://www.oracle.com/database/berkeley-db/je/index.html\n  name: libdb-je-java\n  section: universe/libs\n  version: 3.3.98-1\nlibdb4.8:\n  description: Berkeley v4.8 Database Libraries [runtime]\n  downloads: 6016\n  homepage_uri: http://www.oracle.com/technology/software/products/berkeley-db/index.html\n  name: libdb4.8\n  section: libs\n  version: 4.8.30-11ubuntu1\nlibdb5.1:\n  description: Berkeley v5.1 Database Libraries [runtime]\n  downloads: 21287\n  homepage_uri: http://www.oracle.com/technology/software/products/berkeley-db/index.html\n  name: libdb5.1\n  section: libs\n  version: 5.1.25-11build1\nlibdb5.1-dev:\n  description: Berkeley v5.1 Database Libraries [development]\n  downloads: 216\n  homepage_uri: http://www.oracle.com/technology/software/products/berkeley-db/index.html\n  name: libdb5.1-dev\n  section: libdevel\n  version: 5.1.25-11build1\nlibdb5.1-java:\n  description: Berkeley v5.1 Database Libraries for Java\n  downloads: 161\n  homepage_uri: http://www.oracle.com/technology/software/products/berkeley-db/index.html\n  name: libdb5.1-java\n  section: java\n  version: 5.1.25-11build1\nlibdb5.1-java-gcj:\n  description: Berkeley v5.1 Database Libraries for Java (native code)\n  downloads: 137\n  homepage_uri: http://www.oracle.com/technology/software/products/berkeley-db/index.html\n  name: libdb5.1-java-gcj\n  section: java\n  version: 5.1.25-11build1\nlibdb5.1:i386:\n  description: Berkeley v5.1 Database Libraries [runtime]\n  downloads: 0\n  homepage_uri: http://www.oracle.com/technology/software/products/berkeley-db/index.html\n  name: libdb5.1:i386\n  section: libs\n  version: 5.1.25-11build1\nlibdbd-mysql-perl:\n  description: Perl5 database interface to the MySQL database\n  downloads: 30179\n  homepage_uri: http://search.cpan.org/dist/DBD-mysql/\n  name: libdbd-mysql-perl\n  section: perl\n  version: 4.020-1build2\nlibdbd-pg-perl:\n  description: Perl DBI driver for the PostgreSQL database server\n  downloads: 1756\n  homepage_uri: http://search.cpan.org/dist/DBD-Pg/\n  name: libdbd-pg-perl\n  section: universe/perl\n  version: 2.19.0-1\nlibdbi-perl:\n  description: Perl Database Interface (DBI)\n  downloads: 32158\n  homepage_uri: http://dbi.perl.org/\n  name: libdbi-perl\n  section: perl\n  version: 1.616-1build2\nlibdbus-1-3:\n  description: simple interprocess messaging system (library)\n  downloads: 71064\n  homepage_uri: http://dbus.freedesktop.org/\n  name: libdbus-1-3\n  section: libs\n  version: 1.4.18-1ubuntu1\nlibdbus-1-3:i386:\n  description: simple interprocess messaging system (library)\n  downloads: 0\n  homepage_uri: http://dbus.freedesktop.org/\n  name: libdbus-1-3:i386\n  section: libs\n  version: 1.4.18-1ubuntu1\nlibdbus-1-dev:\n  description: simple interprocess messaging system (development headers)\n  downloads: 2521\n  homepage_uri: http://dbus.freedesktop.org/\n  name: libdbus-1-dev\n  section: libdevel\n  version: 1.4.18-1ubuntu1\nlibdbus-glib-1-2:\n  description: simple interprocess messaging system (GLib-based shared library)\n  downloads: 58809\n  homepage_uri: http://www.freedesktop.org/wiki/Software/DBusBindings\n  name: libdbus-glib-1-2\n  section: libs\n  version: 0.98-1ubuntu1\nlibdbus-glib-1-2:i386:\n  description: simple interprocess messaging system (GLib-based shared library)\n  downloads: 0\n  homepage_uri: http://www.freedesktop.org/wiki/Software/DBusBindings\n  name: libdbus-glib-1-2:i386\n  section: libs\n  version: 0.98-1ubuntu1\nlibdbusmenu-glib4:\n  description: library for passing menus over DBus\n  downloads: 583\n  homepage_uri: https://launchpad.net/dbusmenu\n  name: libdbusmenu-glib4\n  section: libs\n  version: 0.6.1-0ubuntu3\nlibdbusmenu-gtk3-4:\n  description: library for passing menus over DBus - GTK+ version\n  downloads: 228\n  homepage_uri: https://launchpad.net/dbusmenu\n  name: libdbusmenu-gtk3-4\n  section: libs\n  version: 0.6.1-0ubuntu3\nlibdbusmenu-gtk4:\n  description: library for passing menus over DBus - GTK+ version\n  downloads: 80\n  homepage_uri: https://launchpad.net/dbusmenu\n  name: libdbusmenu-gtk4\n  section: libs\n  version: 0.6.1-0ubuntu3\nlibdbusmenu-qt2:\n  description: Qt implementation of the DBusMenu protocol\n  downloads: 1797\n  homepage_uri: https://launchpad.net/libdbusmenu-qt\n  name: libdbusmenu-qt2\n  section: libs\n  version: 0.9.2-0ubuntu1\nlibdconf-dbus-1-0:\n  description: simple configuration storage system - D-Bus library\n  downloads: 0\n  homepage_uri: http://live.gnome.org/dconf\n  name: libdconf-dbus-1-0\n  section: libs\n  version: 0.12.0-0ubuntu1\nlibdconf-qt0:\n  description: dconf Qt bindings (library)\n  downloads: 0\n  homepage_uri: https://gitorious.org/dconf-qt\n  name: libdconf-qt0\n  section: libs\n  version: 0.0.0.110722-0ubuntu4\nlibdconf0:\n  description: simple configuration storage system - runtime library\n  downloads: 13821\n  homepage_uri: http://live.gnome.org/dconf\n  name: libdconf0\n  section: libs\n  version: 0.12.0-0ubuntu1\nlibdecoration0:\n  description: Compiz window decoration library\n  downloads: 2134\n  homepage_uri: ''\n  name: libdecoration0\n  section: x11\n  version: 1:0.9.7.8-0ubuntu1\nlibdee-1.0-4:\n  description: model to synchronize mutiple instances over DBus - shared lib\n  downloads: 338\n  homepage_uri: https://launchpad.net/dee\n  name: libdee-1.0-4\n  section: libs\n  version: 1.0.10-0ubuntu1\nlibdevmapper-event1.02.1:\n  description: The Linux Kernel Device Mapper userspace library\n  downloads: 3829\n  homepage_uri: http://sources.redhat.com/lvm2/\n  name: libdevmapper-event1.02.1\n  section: libs\n  version: 2:1.02.48-4ubuntu7.1\nlibdevmapper1.02.1:\n  description: The Linux Kernel Device Mapper userspace library\n  downloads: 70680\n  homepage_uri: http://sources.redhat.com/lvm2/\n  name: libdevmapper1.02.1\n  section: libs\n  version: 2:1.02.48-4ubuntu7.1\nlibdigest-hmac-perl:\n  description: module for creating standard message integrity checks\n  downloads: 9900\n  homepage_uri: http://search.cpan.org/dist/Digest-HMAC/\n  name: libdigest-hmac-perl\n  section: perl\n  version: 1.03+dfsg-1\nlibdiscid0:\n  description: Library for creating MusicBrainz DiscIDs\n  downloads: 19854\n  homepage_uri: http://musicbrainz.org/doc/libdiscid\n  name: libdiscid0\n  section: libs\n  version: 0.2.2-3\nlibdjvulibre-text:\n  description: Linguistic support files for libdjvulibre\n  downloads: 0\n  homepage_uri: http://djvu.sourceforge.net/\n  name: libdjvulibre-text\n  section: libs\n  version: 3.5.24-9\nlibdjvulibre21:\n  description: Runtime support for the DjVu image format\n  downloads: 30294\n  homepage_uri: http://djvu.sourceforge.net/\n  name: libdjvulibre21\n  section: libs\n  version: 3.5.24-9\nlibdmapsharing-3.0-2:\n  description: DMAP client and server library - runtime\n  downloads: 4900\n  homepage_uri: http://www.flyn.org/projects/libdmapsharing/\n  name: libdmapsharing-3.0-2\n  section: libs\n  version: 2.9.14-1\nlibdns81:\n  description: DNS Shared Library used by BIND\n  downloads: 1908\n  homepage_uri: ''\n  name: libdns81\n  section: libs\n  version: 1:9.8.1.dfsg.P1-4ubuntu0.1\nlibdom4j-java:\n  description: flexible XML framework for Java\n  downloads: 229\n  homepage_uri: http://sourceforge.net/projects/dom4j/\n  name: libdom4j-java\n  section: libs\n  version: 1.6.1+dfsg.2-5\nlibdotconf1.0:\n  description: Configuration file parser library - runtime files\n  downloads: 1656\n  homepage_uri: ''\n  name: libdotconf1.0\n  section: libs\n  version: 1.0.13-3\nlibdoxia-java:\n  description: powerful content generation framework\n  downloads: 93\n  homepage_uri: http://maven.apache.org/doxia/\n  name: libdoxia-java\n  section: universe/devel\n  version: 1.1.4-1ubuntu3\nlibdpkg-perl:\n  description: Dpkg perl modules\n  downloads: 7768\n  homepage_uri: http://wiki.debian.org/Teams/Dpkg\n  name: libdpkg-perl\n  section: perl\n  version: 1.16.1.2ubuntu7\nlibdrm-dev:\n  description: Userspace interface to kernel DRM services -- development files\n  downloads: 294\n  homepage_uri: ''\n  name: libdrm-dev\n  section: libdevel\n  version: 2.4.32-1ubuntu1\nlibdrm-intel1:\n  description: Userspace interface to intel-specific kernel DRM services -- runtime\n  downloads: 29506\n  homepage_uri: ''\n  name: libdrm-intel1\n  section: libs\n  version: 2.4.32-1ubuntu1\nlibdrm-intel1:i386:\n  description: Userspace interface to intel-specific kernel DRM services -- runtime\n  downloads: 0\n  homepage_uri: ''\n  name: libdrm-intel1:i386\n  section: libs\n  version: 2.4.32-1ubuntu1\nlibdrm-nouveau1a:\n  description: Userspace interface to nouveau-specific kernel DRM services -- runtime\n  downloads: 3676\n  homepage_uri: ''\n  name: libdrm-nouveau1a\n  section: libs\n  version: 2.4.32-1ubuntu1\nlibdrm-nouveau1a:i386:\n  description: Userspace interface to nouveau-specific kernel DRM services -- runtime\n  downloads: 0\n  homepage_uri: ''\n  name: libdrm-nouveau1a:i386\n  section: libs\n  version: 2.4.32-1ubuntu1\nlibdrm-radeon1:\n  description: Userspace interface to radeon-specific kernel DRM services -- runtime\n  downloads: 27911\n  homepage_uri: ''\n  name: libdrm-radeon1\n  section: libs\n  version: 2.4.32-1ubuntu1\nlibdrm-radeon1:i386:\n  description: Userspace interface to radeon-specific kernel DRM services -- runtime\n  downloads: 0\n  homepage_uri: ''\n  name: libdrm-radeon1:i386\n  section: libs\n  version: 2.4.32-1ubuntu1\nlibdrm2:\n  description: Userspace interface to kernel DRM services -- runtime\n  downloads: 44737\n  homepage_uri: ''\n  name: libdrm2\n  section: libs\n  version: 2.4.32-1ubuntu1\nlibdrm2:i386:\n  description: Userspace interface to kernel DRM services -- runtime\n  downloads: 0\n  homepage_uri: ''\n  name: libdrm2:i386\n  section: libs\n  version: 2.4.32-1ubuntu1\nlibdv4:\n  description: software library for DV format digital video (runtime lib)\n  downloads: 27909\n  homepage_uri: http://libdv.sourceforge.net/\n  name: libdv4\n  section: libs\n  version: 1.0.0-3ubuntu1\nlibdv4:i386:\n  description: software library for DV format digital video (runtime lib)\n  downloads: 0\n  homepage_uri: http://libdv.sourceforge.net/\n  name: libdv4:i386\n  section: libs\n  version: 1.0.0-3ubuntu1\nlibeasymock-java:\n  description: Java library to generate Mock Objects for given interfaces\n  downloads: 95\n  homepage_uri: http://www.easymock.org\n  name: libeasymock-java\n  section: libs\n  version: 2.4+ds1-6\nlibebackend-1.2-1:\n  description: Utility library for evolution data servers\n  downloads: 0\n  homepage_uri: http://www.gnome.org/projects/evolution/\n  name: libebackend-1.2-1\n  section: libs\n  version: 3.2.3-0ubuntu7\nlibebook-1.2-12:\n  description: Client library for evolution address books\n  downloads: 0\n  homepage_uri: http://www.gnome.org/projects/evolution/\n  name: libebook-1.2-12\n  section: libs\n  version: 3.2.3-0ubuntu7\nlibecal-1.2-10:\n  description: Client library for evolution calendars\n  downloads: 0\n  homepage_uri: http://www.gnome.org/projects/evolution/\n  name: libecal-1.2-10\n  section: libs\n  version: 3.2.3-0ubuntu7\nlibecj-java:\n  description: Eclipse Java compiler (library)\n  downloads: 1270\n  homepage_uri: ''\n  name: libecj-java\n  section: libs\n  version: 3.5.1-3\nlibedata-book-1.2-11:\n  description: Backend library for evolution address books\n  downloads: 0\n  homepage_uri: http://www.gnome.org/projects/evolution/\n  name: libedata-book-1.2-11\n  section: libs\n  version: 3.2.3-0ubuntu7\nlibedata-cal-1.2-13:\n  description: Backend library for evolution calendars\n  downloads: 0\n  homepage_uri: http://www.gnome.org/projects/evolution/\n  name: libedata-cal-1.2-13\n  section: libs\n  version: 3.2.3-0ubuntu7\nlibedataserver-1.2-15:\n  description: Utility library for evolution data servers\n  downloads: 0\n  homepage_uri: http://www.gnome.org/projects/evolution/\n  name: libedataserver-1.2-15\n  section: libs\n  version: 3.2.3-0ubuntu7\nlibedataserverui-3.0-1:\n  description: GUI utility library for evolution data servers\n  downloads: 9215\n  homepage_uri: http://www.gnome.org/projects/evolution/\n  name: libedataserverui-3.0-1\n  section: libs\n  version: 3.2.3-0ubuntu7\nlibedit2:\n  description: BSD editline and history libraries\n  downloads: 74951\n  homepage_uri: http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/lib/libedit/\n  name: libedit2\n  section: libs\n  version: 2.11-20080614-3ubuntu2\nlibeigen3-dev:\n  description: lightweight C++ template library for linear algebra\n  downloads: 46\n  homepage_uri: http://eigen.tuxfamily.org\n  name: libeigen3-dev\n  section: universe/libdevel\n  version: 3.0.5-1\nlibelf1:\n  description: library to read and write ELF files\n  downloads: 15227\n  homepage_uri: https://fedorahosted.org/elfutils/\n  name: libelf1\n  section: libs\n  version: 0.152-1ubuntu3\nlibemail-valid-perl:\n  description: Perl module for checking the validity of Internet email addresses\n  downloads: 1365\n  homepage_uri: http://search.cpan.org/dist/Email-Valid/\n  name: libemail-valid-perl\n  section: perl\n  version: 0.185-1\nlibenchant1c2a:\n  description: Wrapper library for various spell checker engines (runtime libs)\n  downloads: 40880\n  homepage_uri: http://www.abisource.com/projects/enchant/\n  name: libenchant1c2a\n  section: libs\n  version: 1.6.0-7\nlibencode-locale-perl:\n  description: utility to determine the locale encoding\n  downloads: 5317\n  homepage_uri: http://search.cpan.org/dist/Encode-Locale/\n  name: libencode-locale-perl\n  section: perl\n  version: 1.02-2\nlibept1.4.12:\n  description: High-level library for managing Debian package information\n  downloads: 9660\n  homepage_uri: ''\n  name: libept1.4.12\n  section: libs\n  version: 1.0.6~exp1ubuntu1\nlibequinox-osgi-java:\n  description: Equinox OSGi framework\n  downloads: 1625\n  homepage_uri: http://www.eclipse.org/\n  name: libequinox-osgi-java\n  section: universe/java\n  version: 3.7.2-1\nliberror-perl:\n  description: Perl module for error/exception handling in an OO-ish way\n  downloads: 8317\n  homepage_uri: ''\n  name: liberror-perl\n  section: perl\n  version: 0.17-1\nlibesd0:i386:\n  description: Enlightened Sound Daemon - Shared libraries\n  downloads: 0\n  homepage_uri: ''\n  name: libesd0:i386\n  section: libs\n  version: 0.2.41-10build3\nlibespeak1:\n  description: 'Multi-lingual software speech synthesizer: shared library'\n  downloads: 22011\n  homepage_uri: http://espeak.sourceforge.net/\n  name: libespeak1\n  section: libs\n  version: 1.46.02-0ubuntu1\nlibevent-2.0-5:\n  description: Asynchronous event notification library\n  downloads: 14054\n  homepage_uri: http://www.monkey.org/~provos/libevent/\n  name: libevent-2.0-5\n  section: libs\n  version: 2.0.16-stable-1\nlibevince3-3:\n  description: Document (PostScript, PDF) rendering library\n  downloads: 2194\n  homepage_uri: http://www.gnome.org/projects/evince/\n  name: libevince3-3\n  section: libs\n  version: 3.4.0-0ubuntu1\nlibexempi3:\n  description: library to parse XMP metadata (Library)\n  downloads: 32109\n  homepage_uri: http://libopenraw.freedesktop.org/wiki/Exempi\n  name: libexempi3\n  section: libs\n  version: 2.2.0-1\nlibexif12:\n  description: library to parse EXIF files\n  downloads: 42348\n  homepage_uri: http://libexif.sourceforge.net/\n  name: libexif12\n  section: libs\n  version: 0.6.20-2\nlibexif12:i386:\n  description: library to parse EXIF files\n  downloads: 0\n  homepage_uri: http://libexif.sourceforge.net/\n  name: libexif12:i386\n  section: libs\n  version: 0.6.20-2\nlibexiv2-11:\n  description: EXIF/IPTC metadata manipulation library\n  downloads: 0\n  homepage_uri: http://www.exiv2.org\n  name: libexiv2-11\n  section: libs\n  version: 0.22-2\nlibexpat1:\n  description: XML parsing C library - runtime library\n  downloads: 87687\n  homepage_uri: http://expat.sourceforge.net\n  name: libexpat1\n  section: libs\n  version: 2.0.1-7.2ubuntu1\nlibexpat1-dev:\n  description: XML parsing C library - development kit\n  downloads: 5258\n  homepage_uri: http://expat.sourceforge.net\n  name: libexpat1-dev\n  section: libdevel\n  version: 2.0.1-7.2ubuntu1\nlibexpat1:i386:\n  description: XML parsing C library - runtime library\n  downloads: 0\n  homepage_uri: http://expat.sourceforge.net\n  name: libexpat1:i386\n  section: libs\n  version: 2.0.1-7.2ubuntu1\nlibexporter-lite-perl:\n  description: lightweight subset of Exporter\n  downloads: 794\n  homepage_uri: http://search.cpan.org/dist/Exporter-Lite/\n  name: libexporter-lite-perl\n  section: perl\n  version: 0.02-2\nlibexttextcat-data:\n  description: Language detection library - data files\n  downloads: 7\n  homepage_uri: http://software.wise-guys.nl/libtextcat/\n  name: libexttextcat-data\n  section: text\n  version: 3.2.0-1ubuntu1\nlibexttextcat0:\n  description: Language detection library\n  downloads: 7\n  homepage_uri: http://software.wise-guys.nl/libtextcat/\n  name: libexttextcat0\n  section: libs\n  version: 3.2.0-1ubuntu1\nlibfarstream-0.1-0:\n  description: 'Audio/Video communications framework: core library'\n  downloads: 2769\n  homepage_uri: http://www.freedesktop.org/software/farstream/releases/farstream/\n  name: libfarstream-0.1-0\n  section: libs\n  version: 0.1.2-0ubuntu1\nlibfcgi-perl:\n  description: helper module for FastCGI\n  downloads: 2141\n  homepage_uri: http://search.cpan.org/dist/FCGI/\n  name: libfcgi-perl\n  section: universe/perl\n  version: 0.74-1build1\nlibffi-dev:\n  description: Foreign Function Interface library (development files)\n  downloads: 1263\n  homepage_uri: ''\n  name: libffi-dev\n  section: libdevel\n  version: 3.0.11~rc1-5\nlibffi6:\n  description: Foreign Function Interface library runtime\n  downloads: 7\n  homepage_uri: ''\n  name: libffi6\n  section: libs\n  version: 3.0.11~rc1-5\nlibffi6:i386:\n  description: Foreign Function Interface library runtime\n  downloads: 0\n  homepage_uri: ''\n  name: libffi6:i386\n  section: libs\n  version: 3.0.11~rc1-5\nlibfftw3-3:\n  description: library for computing Fast Fourier Transforms\n  downloads: 18319\n  homepage_uri: http://fftw.org\n  name: libfftw3-3\n  section: libs\n  version: 3.3-1ubuntu1\nlibfile-basedir-perl:\n  description: Perl module to use the freedesktop basedir specification\n  downloads: 2537\n  homepage_uri: http://search.cpan.org/dist/File-BaseDir/\n  name: libfile-basedir-perl\n  section: perl\n  version: 0.03-1fakesync1\nlibfile-copy-recursive-perl:\n  description: Perl extension for recursively copying files and directories\n  downloads: 17283\n  homepage_uri: http://search.cpan.org/dist/File-Copy-Recursive/\n  name: libfile-copy-recursive-perl\n  section: perl\n  version: 0.38-1\nlibfile-desktopentry-perl:\n  description: Perl module to handle freedesktop .desktop files\n  downloads: 2479\n  homepage_uri: http://search.cpan.org/dist/File-DesktopEntry/\n  name: libfile-desktopentry-perl\n  section: perl\n  version: 0.04-3\nlibfile-listing-perl:\n  description: module to parse directory listings\n  downloads: 5271\n  homepage_uri: http://search.cpan.org/dist/File-Listing/\n  name: libfile-listing-perl\n  section: perl\n  version: 6.03-1\nlibfile-mimeinfo-perl:\n  description: Perl module to determine file types\n  downloads: 2474\n  homepage_uri: http://search.cpan.org/dist/File-MimeInfo/\n  name: libfile-mimeinfo-perl\n  section: perl\n  version: 0.15-2\nlibflac8:\n  description: Free Lossless Audio Codec - runtime C library\n  downloads: 40618\n  homepage_uri: http://flac.sourceforge.net/\n  name: libflac8\n  section: libs\n  version: 1.2.1-6\nlibflac8:i386:\n  description: Free Lossless Audio Codec - runtime C library\n  downloads: 0\n  homepage_uri: http://flac.sourceforge.net/\n  name: libflac8:i386\n  section: libs\n  version: 1.2.1-6\nlibfltk1.1:\n  description: Fast Light Toolkit - shared libraries\n  downloads: 1663\n  homepage_uri: http://www.fltk.org/\n  name: libfltk1.1\n  section: libs\n  version: 1.1.10-10\nlibfolks-eds25:\n  description: Evolution-data-server backend for libfolks\n  downloads: 5262\n  homepage_uri: http://telepathy.freedesktop.org/wiki/Folks\n  name: libfolks-eds25\n  section: libs\n  version: 0.6.8-2\nlibfolks-telepathy25:\n  description: Telepathy backend for libfolks\n  downloads: 5215\n  homepage_uri: http://telepathy.freedesktop.org/wiki/Folks\n  name: libfolks-telepathy25\n  section: libs\n  version: 0.6.8-2\nlibfolks25:\n  description: library to aggregates people into metacontacts\n  downloads: 7939\n  homepage_uri: http://telepathy.freedesktop.org/wiki/Folks\n  name: libfolks25\n  section: libs\n  version: 0.6.8-2\nlibfont-afm-perl:\n  description: Font::AFM - Interface to Adobe Font Metrics files\n  downloads: 2415\n  homepage_uri: http://search.cpan.org/dist/Font-AFM/\n  name: libfont-afm-perl\n  section: perl\n  version: 1.20-1\nlibfontconfig1:\n  description: generic font configuration library - runtime\n  downloads: 66704\n  homepage_uri: ''\n  name: libfontconfig1\n  section: libs\n  version: 2.8.0-3ubuntu9\nlibfontconfig1-dev:\n  description: generic font configuration library - development\n  downloads: 4320\n  homepage_uri: ''\n  name: libfontconfig1-dev\n  section: libdevel\n  version: 2.8.0-3ubuntu9\nlibfontconfig1:i386:\n  description: generic font configuration library - runtime\n  downloads: 0\n  homepage_uri: ''\n  name: libfontconfig1:i386\n  section: libs\n  version: 2.8.0-3ubuntu9\nlibfontenc1:\n  description: X11 font encoding library\n  downloads: 36105\n  homepage_uri: ''\n  name: libfontenc1\n  section: x11\n  version: 1:1.1.0-1\nlibfop-java:\n  description: XML formatter driven by XSL Formatting Objects (XSL-FO.)\n  downloads: 499\n  homepage_uri: http://xmlgraphics.apache.org/fop/\n  name: libfop-java\n  section: java\n  version: 1:1.0.dfsg2-6\nlibfreerdp-plugins-standard:\n  description: RDP client for Windows Terminal Services (plugins)\n  downloads: 2465\n  homepage_uri: http://www.freerdp.com/\n  name: libfreerdp-plugins-standard\n  section: x11\n  version: 1.0.1-1ubuntu2\nlibfreerdp1:\n  description: RDP client for Windows Terminal Services (library)\n  downloads: 665\n  homepage_uri: http://www.freerdp.com/\n  name: libfreerdp1\n  section: libs\n  version: 1.0.1-1ubuntu2\nlibfreetype6:\n  description: FreeType 2 font engine, shared library files\n  downloads: 77130\n  homepage_uri: http://www.freetype.org\n  name: libfreetype6\n  section: libs\n  version: 2.4.8-1ubuntu2\nlibfreetype6-dev:\n  description: FreeType 2 font engine, development files\n  downloads: 5105\n  homepage_uri: http://www.freetype.org\n  name: libfreetype6-dev\n  section: libdevel\n  version: 2.4.8-1ubuntu2\nlibfreetype6:i386:\n  description: FreeType 2 font engine, shared library files\n  downloads: 0\n  homepage_uri: http://www.freetype.org\n  name: libfreetype6:i386\n  section: libs\n  version: 2.4.8-1ubuntu2\nlibfribidi0:\n  description: Free Implementation of the Unicode BiDi algorithm\n  downloads: 11609\n  homepage_uri: http://www.fribidi.org/\n  name: libfribidi0\n  section: libs\n  version: 0.19.2-1\nlibfs6:\n  description: X11 Font Services library\n  downloads: 5546\n  homepage_uri: ''\n  name: libfs6\n  section: libs\n  version: 2:1.0.3-1\nlibfuse-dev:\n  description: Filesystem in Userspace (development)\n  downloads: 138\n  homepage_uri: http://fuse.sourceforge.net/\n  name: libfuse-dev\n  section: libdevel\n  version: 2.8.6-2ubuntu2\nlibfuse2:\n  description: Filesystem in Userspace (library)\n  downloads: 23958\n  homepage_uri: http://fuse.sourceforge.net/\n  name: libfuse2\n  section: libs\n  version: 2.8.6-2ubuntu2\nlibgail-3-0:\n  description: GNOME Accessibility Implementation Library -- shared libraries\n  downloads: 11162\n  homepage_uri: http://www.gtk.org/\n  name: libgail-3-0\n  section: libs\n  version: 3.4.2-0ubuntu0.2\nlibgail-common:\n  description: GNOME Accessibility Implementation Library -- common modules\n  downloads: 18796\n  homepage_uri: http://www.gtk.org/\n  name: libgail-common\n  section: libs\n  version: 2.24.10-0ubuntu6\nlibgail-common:i386:\n  description: GNOME Accessibility Implementation Library -- common modules\n  downloads: 0\n  homepage_uri: http://www.gtk.org/\n  name: libgail-common:i386\n  section: libs\n  version: 2.24.10-0ubuntu6\nlibgail-dev:\n  description: GNOME Accessibility Implementation Library -- development files\n  downloads: 311\n  homepage_uri: http://www.gtk.org/\n  name: libgail-dev\n  section: libdevel\n  version: 2.24.10-0ubuntu6\nlibgail18:\n  description: GNOME Accessibility Implementation Library -- shared libraries\n  downloads: 27619\n  homepage_uri: http://www.gtk.org/\n  name: libgail18\n  section: libs\n  version: 2.24.10-0ubuntu6\nlibgail18:i386:\n  description: GNOME Accessibility Implementation Library -- shared libraries\n  downloads: 0\n  homepage_uri: http://www.gtk.org/\n  name: libgail18:i386\n  section: libs\n  version: 2.24.10-0ubuntu6\nlibganymed-ssh2-java:\n  description: pure Java implementation of the SSH-2 protocol\n  downloads: 2\n  homepage_uri: http://www.cleondris.ch/ssh2/\n  name: libganymed-ssh2-java\n  section: universe/libs\n  version: 250-2\nlibgc1c2:\n  description: conservative garbage collector for C and C++\n  downloads: 29447\n  homepage_uri: http://www.hpl.hp.com/personal/Hans_Boehm/gc/\n  name: libgc1c2\n  section: libs\n  version: 1:7.1-8build1\nlibgcc1:\n  description: GCC support library\n  downloads: 118025\n  homepage_uri: http://gcc.gnu.org/\n  name: libgcc1\n  section: libs\n  version: 1:4.6.3-1ubuntu5\nlibgcc1:i386:\n  description: GCC support library\n  downloads: 0\n  homepage_uri: http://gcc.gnu.org/\n  name: libgcc1:i386\n  section: libs\n  version: 1:4.6.3-1ubuntu5\nlibgcj-bc:\n  description: Link time only library for use with gcj\n  downloads: 579\n  homepage_uri: ''\n  name: libgcj-bc\n  section: java\n  version: 4.6.3-1ubuntu5\nlibgcj-common:\n  description: Java runtime library (common files)\n  downloads: 13885\n  homepage_uri: ''\n  name: libgcj-common\n  section: libs\n  version: 1:4.6.3-1ubuntu5\nlibgcj12:\n  description: Java runtime library for use with gcj\n  downloads: 3640\n  homepage_uri: http://gcc.gnu.org/\n  name: libgcj12\n  section: libs\n  version: 4.6.3-1ubuntu2\nlibgck-1-0:\n  description: Glib wrapper library for PKCS#11 - runtime\n  downloads: 12242\n  homepage_uri: http://live.gnome.org/GnomeKeyring\n  name: libgck-1-0\n  section: libs\n  version: 3.2.2-2ubuntu4\nlibgconf-2-4:\n  description: GNOME configuration database system (shared libraries)\n  downloads: 15055\n  homepage_uri: http://projects.gnome.org/gconf/\n  name: libgconf-2-4\n  section: libs\n  version: 3.2.5-0ubuntu2\nlibgconf-2-4:i386:\n  description: GNOME configuration database system (shared libraries)\n  downloads: 0\n  homepage_uri: http://projects.gnome.org/gconf/\n  name: libgconf-2-4:i386\n  section: libs\n  version: 3.2.5-0ubuntu2\nlibgconf2-4:\n  description: GNOME configuration database system (dummy package)\n  downloads: 43270\n  homepage_uri: http://projects.gnome.org/gconf/\n  name: libgconf2-4\n  section: libs\n  version: 3.2.5-0ubuntu2\nlibgconf2-dev:\n  description: GNOME configuration database system (development)\n  downloads: 819\n  homepage_uri: http://projects.gnome.org/gconf/\n  name: libgconf2-dev\n  section: libdevel\n  version: 3.2.5-0ubuntu2\nlibgconf2-doc:\n  description: GNOME configuration database system (API reference)\n  downloads: 0\n  homepage_uri: http://projects.gnome.org/gconf/\n  name: libgconf2-doc\n  section: doc\n  version: 3.2.5-0ubuntu2\nlibgcr-3-1:\n  description: Library for Crypto UI related task - runtime\n  downloads: 12252\n  homepage_uri: http://live.gnome.org/GnomeKeyring\n  name: libgcr-3-1\n  section: libs\n  version: 3.2.2-2ubuntu4\nlibgcr-3-common:\n  description: Library for Crypto UI related task - common files\n  downloads: 11822\n  homepage_uri: http://live.gnome.org/GnomeKeyring\n  name: libgcr-3-common\n  section: libs\n  version: 3.2.2-2ubuntu4\nlibgcrypt11:\n  description: LGPL Crypto library - runtime library\n  downloads: 85078\n  homepage_uri: http://directory.fsf.org/project/libgcrypt/\n  name: libgcrypt11\n  section: libs\n  version: 1.5.0-3ubuntu0.1\nlibgcrypt11-dev:\n  description: LGPL Crypto library - development files\n  downloads: 3027\n  homepage_uri: http://directory.fsf.org/project/libgcrypt/\n  name: libgcrypt11-dev\n  section: libdevel\n  version: 1.5.0-3ubuntu0.1\nlibgcrypt11:i386:\n  description: LGPL Crypto library - runtime library\n  downloads: 0\n  homepage_uri: http://directory.fsf.org/project/libgcrypt/\n  name: libgcrypt11:i386\n  section: libs\n  version: 1.5.0-3ubuntu0.1\nlibgd-gd2-perl:\n  description: Perl module wrapper for libgd - gd2 variant\n  downloads: 965\n  homepage_uri: http://search.cpan.org/dist/GD/\n  name: libgd-gd2-perl\n  section: perl\n  version: 1:2.46-3build1\nlibgd-graph-perl:\n  description: Graph Plotting Module for Perl 5\n  downloads: 595\n  homepage_uri: http://search.cpan.org/dist/GDGraph/\n  name: libgd-graph-perl\n  section: perl\n  version: 1.44-6\nlibgd-text-perl:\n  description: Text utilities for use with GD\n  downloads: 603\n  homepage_uri: http://search.cpan.org/dist/GDTextUtil/\n  name: libgd-text-perl\n  section: perl\n  version: 0.86-8\nlibgd2-xpm:\n  description: GD Graphics Library version 2\n  downloads: 39833\n  homepage_uri: http://www.libgd.org/\n  name: libgd2-xpm\n  section: libs\n  version: 2.0.36~rc1~dfsg-6ubuntu2\nlibgd2-xpm-dev:\n  description: GD Graphics Library version 2 (development version)\n  downloads: 225\n  homepage_uri: http://www.libgd.org/\n  name: libgd2-xpm-dev\n  section: libdevel\n  version: 2.0.36~rc1~dfsg-6ubuntu2\nlibgd2-xpm:i386:\n  description: GD Graphics Library version 2\n  downloads: 0\n  homepage_uri: http://www.libgd.org/\n  name: libgd2-xpm:i386\n  section: libs\n  version: 2.0.36~rc1~dfsg-6ubuntu2\nlibgdata-common:\n  description: Library for accessing GData webservices - common data files\n  downloads: 97\n  homepage_uri: http://live.gnome.org/libgdata\n  name: libgdata-common\n  section: libs\n  version: 0.12.0-1\nlibgdata13:\n  description: Library for accessing GData webservices - shared libraries\n  downloads: 7312\n  homepage_uri: http://live.gnome.org/libgdata\n  name: libgdata13\n  section: libs\n  version: 0.12.0-1\nlibgdbm3:\n  description: GNU dbm database routines (runtime version)\n  downloads: 109796\n  homepage_uri: http://directory.fsf.org/project/gdbm/\n  name: libgdbm3\n  section: libs\n  version: 1.8.3-10\nlibgdbm3:i386:\n  description: GNU dbm database routines (runtime version)\n  downloads: 0\n  homepage_uri: http://directory.fsf.org/project/gdbm/\n  name: libgdbm3:i386\n  section: libs\n  version: 1.8.3-10\nlibgdk-pixbuf2.0-0:\n  description: GDK Pixbuf library\n  downloads: 17771\n  homepage_uri: http://www.gtk.org/\n  name: libgdk-pixbuf2.0-0\n  section: libs\n  version: 2.26.1-1\nlibgdk-pixbuf2.0-0:i386:\n  description: GDK Pixbuf library\n  downloads: 0\n  homepage_uri: http://www.gtk.org/\n  name: libgdk-pixbuf2.0-0:i386\n  section: libs\n  version: 2.26.1-1\nlibgdk-pixbuf2.0-common:\n  description: GDK Pixbuf library - data files\n  downloads: 15887\n  homepage_uri: http://www.gtk.org/\n  name: libgdk-pixbuf2.0-common\n  section: libs\n  version: 2.26.1-1\nlibgdk-pixbuf2.0-dev:\n  description: GDK Pixbuf library (development files)\n  downloads: 2027\n  homepage_uri: http://www.gtk.org/\n  name: libgdk-pixbuf2.0-dev\n  section: libdevel\n  version: 2.26.1-1\nlibgdu-gtk0:\n  description: GTK+ standard dialog library for libgdu\n  downloads: 26659\n  homepage_uri: http://git.gnome.org/cgit/gnome-disk-utility/\n  name: libgdu-gtk0\n  section: libs\n  version: 3.0.2-2ubuntu7\nlibgdu0:\n  description: GObject based Disk Utility Library\n  downloads: 34679\n  homepage_uri: http://git.gnome.org/cgit/gnome-disk-utility/\n  name: libgdu0\n  section: libs\n  version: 3.0.2-2ubuntu7\nlibgee2:\n  description: GObject based collection library\n  downloads: 13375\n  homepage_uri: http://live.gnome.org/Libgee\n  name: libgee2\n  section: libs\n  version: 0.6.4-1\nlibgeoclue0:\n  description: C API for GeoClue\n  downloads: 24563\n  homepage_uri: http://www.freedesktop.org/wiki/Software/GeoClue\n  name: libgeoclue0\n  section: libs\n  version: 0.12.0-1ubuntu12\nlibgeoip1:\n  description: non-DNS IP-to-country resolver library\n  downloads: 6506\n  homepage_uri: http://www.maxmind.com/\n  name: libgeoip1\n  section: libs\n  version: 1.4.8+dfsg-2\nlibgettextpo0:\n  description: GNU Internationalization library\n  downloads: 2211\n  homepage_uri: http://www.gnu.org/software/gettext/\n  name: libgettextpo0\n  section: devel\n  version: 0.18.1.1-5ubuntu3\nlibgettextpo0:i386:\n  description: GNU Internationalization library\n  downloads: 0\n  homepage_uri: http://www.gnu.org/software/gettext/\n  name: libgettextpo0:i386\n  section: devel\n  version: 0.18.1.1-5ubuntu3\nlibgexiv2-1:\n  description: GObject-based wrapper around the Exiv2 library\n  downloads: 1286\n  homepage_uri: http://redmine.yorba.org/projects/gexiv2/wiki\n  name: libgexiv2-1\n  section: libs\n  version: 0.4.1-1build1\nlibgfortran3:\n  description: Runtime library for GNU Fortran applications\n  downloads: 3850\n  homepage_uri: http://gcc.gnu.org/\n  name: libgfortran3\n  section: libs\n  version: 4.6.3-1ubuntu5\nlibgif4:\n  description: library for GIF images (library)\n  downloads: 22730\n  homepage_uri: http://giflib.sourceforge.net/\n  name: libgif4\n  section: libs\n  version: 4.1.6-9ubuntu1\nlibgirepository-1.0-1:\n  description: Library for handling GObject introspection data (runtime library)\n  downloads: 13188\n  homepage_uri: http://live.gnome.org/GObjectIntrospection\n  name: libgirepository-1.0-1\n  section: libs\n  version: 1.32.0-1\nlibgksu2-0:\n  description: library providing su and sudo functionality\n  downloads: 29787\n  homepage_uri: http://www.nongnu.org/gksu\n  name: libgksu2-0\n  section: libs\n  version: 2.0.13~pre1-5ubuntu2\nlibgl1-mesa-dev:\n  description: free implementation of the OpenGL API -- GLX development files\n  downloads: 2647\n  homepage_uri: http://mesa3d.sourceforge.net/\n  name: libgl1-mesa-dev\n  section: libdevel\n  version: 8.0.2-0ubuntu3.1\nlibgl1-mesa-dri:\n  description: free implementation of the OpenGL API -- DRI modules\n  downloads: 26165\n  homepage_uri: http://mesa3d.sourceforge.net/\n  name: libgl1-mesa-dri\n  section: libs\n  version: 8.0.2-0ubuntu3.1\nlibgl1-mesa-dri:i386:\n  description: free implementation of the OpenGL API -- DRI modules\n  downloads: 0\n  homepage_uri: http://mesa3d.sourceforge.net/\n  name: libgl1-mesa-dri:i386\n  section: libs\n  version: 8.0.2-0ubuntu3.1\nlibgl1-mesa-glx:\n  description: free implementation of the OpenGL API -- GLX runtime\n  downloads: 44268\n  homepage_uri: http://mesa3d.sourceforge.net/\n  name: libgl1-mesa-glx\n  section: libs\n  version: 8.0.2-0ubuntu3.1\nlibgl1-mesa-glx:i386:\n  description: free implementation of the OpenGL API -- GLX runtime\n  downloads: 0\n  homepage_uri: http://mesa3d.sourceforge.net/\n  name: libgl1-mesa-glx:i386\n  section: libs\n  version: 8.0.2-0ubuntu3.1\nlibgl2ps0:\n  description: Lib providing high quality vector output for OpenGL application\n  downloads: 72\n  homepage_uri: http://geuz.org/gl2ps/\n  name: libgl2ps0\n  section: universe/libs\n  version: 1.3.6-1\nlibglade2-0:\n  description: library to load .glade files at runtime\n  downloads: 36287\n  homepage_uri: ''\n  name: libglade2-0\n  section: libs\n  version: 1:2.6.4-1ubuntu1.1\nlibglapi-mesa:\n  description: free implementation of the GL API -- shared library\n  downloads: 2749\n  homepage_uri: http://mesa3d.sourceforge.net/\n  name: libglapi-mesa\n  section: libs\n  version: 8.0.2-0ubuntu3.1\nlibglapi-mesa:i386:\n  description: free implementation of the GL API -- shared library\n  downloads: 0\n  homepage_uri: http://mesa3d.sourceforge.net/\n  name: libglapi-mesa:i386\n  section: libs\n  version: 8.0.2-0ubuntu3.1\nlibglew1.6:\n  description: OpenGL Extension Wrangler - runtime environment\n  downloads: 120\n  homepage_uri: http://glew.sourceforge.net/\n  name: libglew1.6\n  section: libs\n  version: 1.6.0-4\nlibglewmx1.6:\n  description: OpenGL Extension Wrangler - runtime environment\n  downloads: 10\n  homepage_uri: http://glew.sourceforge.net/\n  name: libglewmx1.6\n  section: libs\n  version: 1.6.0-4\nlibglib-perl:\n  description: interface to the GLib and GObject libraries\n  downloads: 17559\n  homepage_uri: http://gtk2-perl.sourceforge.net/\n  name: libglib-perl\n  section: perl\n  version: 2:1.241-1\nlibglib2.0-0:\n  description: GLib library of C routines\n  downloads: 75544\n  homepage_uri: http://www.gtk.org/\n  name: libglib2.0-0\n  section: libs\n  version: 2.32.3-0ubuntu1\nlibglib2.0-0:i386:\n  description: GLib library of C routines\n  downloads: 0\n  homepage_uri: http://www.gtk.org/\n  name: libglib2.0-0:i386\n  section: libs\n  version: 2.32.3-0ubuntu1\nlibglib2.0-bin:\n  description: Programs for the GLib library\n  downloads: 13149\n  homepage_uri: http://www.gtk.org/\n  name: libglib2.0-bin\n  section: misc\n  version: 2.32.3-0ubuntu1\nlibglib2.0-data:\n  description: Common files for GLib library\n  downloads: 33562\n  homepage_uri: http://www.gtk.org/\n  name: libglib2.0-data\n  section: misc\n  version: 2.32.3-0ubuntu1\nlibglib2.0-dev:\n  description: Development files for the GLib library\n  downloads: 4917\n  homepage_uri: http://www.gtk.org/\n  name: libglib2.0-dev\n  section: libdevel\n  version: 2.32.3-0ubuntu1\nlibglibmm-2.4-1c2a:\n  description: C++ wrapper for the GLib toolkit (shared libraries)\n  downloads: 815\n  homepage_uri: http://www.gtkmm.org/\n  name: libglibmm-2.4-1c2a\n  section: libs\n  version: 2.32.0-0ubuntu1\nlibglpk0:\n  description: linear programming kit with integer (MIP) support\n  downloads: 155\n  homepage_uri: http://www.gnu.org/software/glpk/glpk.html\n  name: libglpk0\n  section: universe/math\n  version: 4.45-1build1\nlibglu1-mesa:\n  description: Mesa OpenGL utility library (GLU)\n  downloads: 13895\n  homepage_uri: http://mesa3d.sourceforge.net/\n  name: libglu1-mesa\n  section: libs\n  version: 8.0.2-0ubuntu3.1\nlibglu1-mesa-dev:\n  description: Mesa OpenGL utility library -- development files\n  downloads: 2574\n  homepage_uri: http://mesa3d.sourceforge.net/\n  name: libglu1-mesa-dev\n  section: libdevel\n  version: 8.0.2-0ubuntu3.1\nlibglu1-mesa:i386:\n  description: Mesa OpenGL utility library (GLU)\n  downloads: 0\n  homepage_uri: http://mesa3d.sourceforge.net/\n  name: libglu1-mesa:i386\n  section: libs\n  version: 8.0.2-0ubuntu3.1\nlibgmime-2.6-0:\n  description: MIME message parser and creator library - runtime\n  downloads: 2714\n  homepage_uri: http://spruce.sourceforge.net/gmime/\n  name: libgmime-2.6-0\n  section: libs\n  version: 2.6.7-1\nlibgmp-dev:\n  description: Multiprecision arithmetic library developers tools\n  downloads: 937\n  homepage_uri: http://gmplib.org/\n  name: libgmp-dev\n  section: libdevel\n  version: 2:5.0.2+dfsg-2ubuntu1\nlibgmp10:\n  description: Multiprecision arithmetic library\n  downloads: 17100\n  homepage_uri: http://gmplib.org/\n  name: libgmp10\n  section: libs\n  version: 2:5.0.2+dfsg-2ubuntu1\nlibgmpxx4ldbl:\n  description: Multiprecision arithmetic library (C++ bindings)\n  downloads: 246\n  homepage_uri: http://gmplib.org/\n  name: libgmpxx4ldbl\n  section: libs\n  version: 2:5.0.2+dfsg-2ubuntu1\nlibgnome-bluetooth8:\n  description: GNOME Bluetooth tools - support library\n  downloads: 0\n  homepage_uri: http://live.gnome.org/GnomeBluetooth\n  name: libgnome-bluetooth8\n  section: libs\n  version: 3.2.2-0ubuntu5\nlibgnome-control-center1:\n  description: utilities to configure the GNOME desktop\n  downloads: 0\n  homepage_uri: ''\n  name: libgnome-control-center1\n  section: libs\n  version: 1:3.4.2-0ubuntu0.2\nlibgnome-desktop-3-2:\n  description: Utility library for loading .desktop files - runtime files\n  downloads: 10541\n  homepage_uri: ''\n  name: libgnome-desktop-3-2\n  section: libs\n  version: 3.4.1-0ubuntu1\nlibgnome-keyring-common:\n  description: GNOME keyring services library - data files\n  downloads: 9471\n  homepage_uri: http://live.gnome.org/GnomeKeyring\n  name: libgnome-keyring-common\n  section: libs\n  version: 3.2.2-2\nlibgnome-keyring-dev:\n  description: Development files for GNOME keyring service\n  downloads: 359\n  homepage_uri: http://live.gnome.org/GnomeKeyring\n  name: libgnome-keyring-dev\n  section: libdevel\n  version: 3.2.2-2\nlibgnome-keyring0:\n  description: GNOME keyring services library\n  downloads: 41606\n  homepage_uri: http://live.gnome.org/GnomeKeyring\n  name: libgnome-keyring0\n  section: libs\n  version: 3.2.2-2\nlibgnome-keyring0:i386:\n  description: GNOME keyring services library\n  downloads: 0\n  homepage_uri: http://live.gnome.org/GnomeKeyring\n  name: libgnome-keyring0:i386\n  section: libs\n  version: 3.2.2-2\nlibgnome-media-profiles-3.0-0:\n  description: GNOME Media Profiles library\n  downloads: 1362\n  homepage_uri: ''\n  name: libgnome-media-profiles-3.0-0\n  section: libs\n  version: 3.0.0-1\nlibgnome-menu-3-0:\n  description: GNOME implementation of the freedesktop menu specification\n  downloads: 9037\n  homepage_uri: ''\n  name: libgnome-menu-3-0\n  section: gnome\n  version: 3.4.0-0ubuntu1\nlibgnome-menu2:\n  description: GNOME implementation of the freedesktop menu specification\n  downloads: 4142\n  homepage_uri: ''\n  name: libgnome-menu2\n  section: gnome\n  version: 3.0.1-0ubuntu7\nlibgnome2-0:\n  description: The GNOME library - runtime files\n  downloads: 34342\n  homepage_uri: ''\n  name: libgnome2-0\n  section: libs\n  version: 2.32.1-2ubuntu1\nlibgnome2-common:\n  description: The GNOME library - common files\n  downloads: 12\n  homepage_uri: ''\n  name: libgnome2-common\n  section: libs\n  version: 2.32.1-2ubuntu1\nlibgnome2-dev:\n  description: The GNOME library - development files\n  downloads: 173\n  homepage_uri: ''\n  name: libgnome2-dev\n  section: libdevel\n  version: 2.32.1-2ubuntu1\nlibgnomecanvas2-0:\n  description: powerful object-oriented display engine - runtime files\n  downloads: 29139\n  homepage_uri: ''\n  name: libgnomecanvas2-0\n  section: libs\n  version: 2.30.3-1ubuntu1\nlibgnomecanvas2-common:\n  description: powerful object-oriented display engine - common files\n  downloads: 3\n  homepage_uri: ''\n  name: libgnomecanvas2-common\n  section: x11\n  version: 2.30.3-1ubuntu1\nlibgnomecanvas2-dev:\n  description: powerful object-oriented display engine - development files\n  downloads: 195\n  homepage_uri: ''\n  name: libgnomecanvas2-dev\n  section: libdevel\n  version: 2.30.3-1ubuntu1\nlibgnomekbd-common:\n  description: GNOME library to manage keyboard configuration - common files\n  downloads: 7367\n  homepage_uri: ''\n  name: libgnomekbd-common\n  section: libs\n  version: 3.4.0.2-1\nlibgnomekbd7:\n  description: GNOME library to manage keyboard configuration - shared library\n  downloads: 10484\n  homepage_uri: ''\n  name: libgnomekbd7\n  section: libs\n  version: 3.4.0.2-1\nlibgnomeui-0:\n  description: GNOME user interface library - runtime files\n  downloads: 28160\n  homepage_uri: ''\n  name: libgnomeui-0\n  section: libs\n  version: 2.24.5-2ubuntu2\nlibgnomeui-common:\n  description: GNOME user interface library - common files\n  downloads: 5\n  homepage_uri: ''\n  name: libgnomeui-common\n  section: libs\n  version: 2.24.5-2ubuntu2\nlibgnomeui-dev:\n  description: GNOME user interface library - development files\n  downloads: 155\n  homepage_uri: ''\n  name: libgnomeui-dev\n  section: libdevel\n  version: 2.24.5-2ubuntu2\nlibgnomevfs2-0:\n  description: GNOME Virtual File System (runtime libraries)\n  downloads: 37623\n  homepage_uri: ''\n  name: libgnomevfs2-0\n  section: libs\n  version: 1:2.24.4-1ubuntu2\nlibgnomevfs2-common:\n  description: GNOME Virtual File System (common files)\n  downloads: 16\n  homepage_uri: ''\n  name: libgnomevfs2-common\n  section: devel\n  version: 1:2.24.4-1ubuntu2\nlibgnomevfs2-dev:\n  description: GNOME Virtual File System library (development files)\n  downloads: 262\n  homepage_uri: ''\n  name: libgnomevfs2-dev\n  section: libdevel\n  version: 1:2.24.4-1ubuntu2\nlibgnutls-dev:\n  description: GNU TLS library - development files\n  downloads: 2912\n  homepage_uri: http://www.gnutls.org/\n  name: libgnutls-dev\n  section: libdevel\n  version: 2.12.14-5ubuntu3\nlibgnutls-openssl27:\n  description: GNU TLS library - OpenSSL wrapper\n  downloads: 2421\n  homepage_uri: http://www.gnutls.org/\n  name: libgnutls-openssl27\n  section: libs\n  version: 2.12.14-5ubuntu3\nlibgnutls26:\n  description: GNU TLS library - runtime library\n  downloads: 94484\n  homepage_uri: http://www.gnutls.org/\n  name: libgnutls26\n  section: libs\n  version: 2.12.14-5ubuntu3\nlibgnutls26:i386:\n  description: GNU TLS library - runtime library\n  downloads: 0\n  homepage_uri: http://www.gnutls.org/\n  name: libgnutls26:i386\n  section: libs\n  version: 2.12.14-5ubuntu3\nlibgnutlsxx27:\n  description: GNU TLS library - C++ runtime library\n  downloads: 1575\n  homepage_uri: http://www.gnutls.org/\n  name: libgnutlsxx27\n  section: libs\n  version: 2.12.14-5ubuntu3\nlibgoa-1.0-0:\n  description: library for GNOME Online Accounts\n  downloads: 8078\n  homepage_uri: https://live.gnome.org/OnlineAccounts\n  name: libgoa-1.0-0\n  section: libs\n  version: 3.4.0-0ubuntu1\nlibgoa-1.0-common:\n  description: library for GNOME Online Accounts - common files\n  downloads: 7857\n  homepage_uri: https://live.gnome.org/OnlineAccounts\n  name: libgoa-1.0-common\n  section: libs\n  version: 3.4.0-0ubuntu1\nlibgomp1:\n  description: GCC OpenMP (GOMP) support library\n  downloads: 37166\n  homepage_uri: http://gcc.gnu.org/\n  name: libgomp1\n  section: libs\n  version: 4.6.3-1ubuntu5\nlibgomp1:i386:\n  description: GCC OpenMP (GOMP) support library\n  downloads: 0\n  homepage_uri: http://gcc.gnu.org/\n  name: libgomp1:i386\n  section: libs\n  version: 4.6.3-1ubuntu5\nlibgoogle-collections-java:\n  description: suite of collections and related goodies for Java 5.0\n  downloads: 25\n  homepage_uri: http://code.google.com/p/google-collections/\n  name: libgoogle-collections-java\n  section: universe/libs\n  version: 1.0-2\nlibgpg-error-dev:\n  description: library for common error values and messages in GnuPG components (development)\n  downloads: 3104\n  homepage_uri: http://www.gnupg.org/related_software/libgpg-error/\n  name: libgpg-error-dev\n  section: libdevel\n  version: 1.10-2ubuntu1\nlibgpg-error0:\n  description: library for common error values and messages in GnuPG components\n  downloads: 82490\n  homepage_uri: http://www.gnupg.org/related_software/libgpg-error/\n  name: libgpg-error0\n  section: libs\n  version: 1.10-2ubuntu1\nlibgpg-error0:i386:\n  description: library for common error values and messages in GnuPG components\n  downloads: 0\n  homepage_uri: http://www.gnupg.org/related_software/libgpg-error/\n  name: libgpg-error0:i386\n  section: libs\n  version: 1.10-2ubuntu1\nlibgpgme11:\n  description: GPGME - GnuPG Made Easy\n  downloads: 39396\n  homepage_uri: ''\n  name: libgpgme11\n  section: libs\n  version: 1.2.0-1.4ubuntu2\nlibgphoto2-2:\n  description: gphoto2 digital camera library\n  downloads: 40261\n  homepage_uri: http://www.gphoto.org/proj/libgphoto2/\n  name: libgphoto2-2\n  section: libs\n  version: 2.4.13-1ubuntu1\nlibgphoto2-2:i386:\n  description: gphoto2 digital camera library\n  downloads: 0\n  homepage_uri: http://www.gphoto.org/proj/libgphoto2/\n  name: libgphoto2-2:i386\n  section: libs\n  version: 2.4.13-1ubuntu1\nlibgphoto2-l10n:\n  description: gphoto2 digital camera library - localized messages\n  downloads: 1\n  homepage_uri: http://www.gphoto.org/proj/libgphoto2/\n  name: libgphoto2-l10n\n  section: localization\n  version: 2.4.13-1ubuntu1\nlibgphoto2-port0:\n  description: gphoto2 digital camera port library\n  downloads: 40174\n  homepage_uri: http://www.gphoto.org/proj/libgphoto2/\n  name: libgphoto2-port0\n  section: libs\n  version: 2.4.13-1ubuntu1\nlibgphoto2-port0:i386:\n  description: gphoto2 digital camera port library\n  downloads: 0\n  homepage_uri: http://www.gphoto.org/proj/libgphoto2/\n  name: libgphoto2-port0:i386\n  section: libs\n  version: 2.4.13-1ubuntu1\nlibgpm2:\n  description: General Purpose Mouse - shared library\n  downloads: 37225\n  homepage_uri: http://unix.schottelius.org/gpm/\n  name: libgpm2\n  section: libs\n  version: 1.20.4-4\nlibgpm2:i386:\n  description: General Purpose Mouse - shared library\n  downloads: 0\n  homepage_uri: http://unix.schottelius.org/gpm/\n  name: libgpm2:i386\n  section: libs\n  version: 1.20.4-4\nlibgpod-common:\n  description: common files for libgpod\n  downloads: 14508\n  homepage_uri: http://www.gtkpod.org/wiki/Libgpod\n  name: libgpod-common\n  section: libs\n  version: 0.8.2-4\nlibgpod4:\n  description: library to read and write songs and artwork to an iPod\n  downloads: 20074\n  homepage_uri: http://www.gtkpod.org/wiki/Libgpod\n  name: libgpod4\n  section: libs\n  version: 0.8.2-4\nlibgraph4:\n  description: rich set of graph drawing tools - graph library\n  downloads: 3762\n  homepage_uri: http://www.graphviz.org/\n  name: libgraph4\n  section: libs\n  version: 2.26.3-10ubuntu1\nlibgraphicsmagick++3:\n  description: format-independent image processing - C++ shared library\n  downloads: 64\n  homepage_uri: ''\n  name: libgraphicsmagick++3\n  section: universe/libs\n  version: 1.3.12-1.1build1\nlibgraphicsmagick3:\n  description: format-independent image processing - C shared library\n  downloads: 6346\n  homepage_uri: ''\n  name: libgraphicsmagick3\n  section: universe/libs\n  version: 1.3.12-1.1build1\nlibgraphviz-dev:\n  description: graphviz libs and headers against which to build applications\n  downloads: 259\n  homepage_uri: http://www.graphviz.org/\n  name: libgraphviz-dev\n  section: libdevel\n  version: 2.26.3-10ubuntu1\nlibgrip0:\n  description: Shared library providing multitouch gestures to GTK+ apps.\n  downloads: 0\n  homepage_uri: https://launchpad.net/libgrip\n  name: libgrip0\n  section: libs\n  version: 0.3.4-0ubuntu2~ubuntu12.04.1\nlibgs9:\n  description: interpreter for the PostScript language and for PDF - Library\n  downloads: 12326\n  homepage_uri: http://www.ghostscript.com/\n  name: libgs9\n  section: libs\n  version: 9.05~dfsg-0ubuntu4\nlibgs9-common:\n  description: interpreter for the PostScript language and for PDF - common files\n  downloads: 0\n  homepage_uri: http://www.ghostscript.com/\n  name: libgs9-common\n  section: libs\n  version: 9.05~dfsg-0ubuntu4\nlibgsl0-dev:\n  description: GNU Scientific Library (GSL) -- development package\n  downloads: 496\n  homepage_uri: http://www.gnu.org/software/gsl\n  name: libgsl0-dev\n  section: libdevel\n  version: 1.15+dfsg-1build1\nlibgsl0ldbl:\n  description: GNU Scientific Library (GSL) -- library package\n  downloads: 5499\n  homepage_uri: http://www.gnu.org/software/gsl\n  name: libgsl0ldbl\n  section: math\n  version: 1.15+dfsg-1build1\nlibgsm1:\n  description: Shared libraries for GSM speech compressor\n  downloads: 9419\n  homepage_uri: ''\n  name: libgsm1\n  section: libs\n  version: 1.0.13-3\nlibgssapi-krb5-2:\n  description: MIT Kerberos runtime libraries - krb5 GSS-API Mechanism\n  downloads: 83523\n  homepage_uri: http://web.mit.edu/kerberos/\n  name: libgssapi-krb5-2\n  section: libs\n  version: 1.10+dfsg~beta1-2ubuntu0.1\nlibgssapi-krb5-2:i386:\n  description: MIT Kerberos runtime libraries - krb5 GSS-API Mechanism\n  downloads: 0\n  homepage_uri: http://web.mit.edu/kerberos/\n  name: libgssapi-krb5-2:i386\n  section: libs\n  version: 1.10+dfsg~beta1-2ubuntu0.1\nlibgssapi3-heimdal:\n  description: Heimdal Kerberos - GSSAPI support library\n  downloads: 135\n  homepage_uri: http://www.h5l.org/\n  name: libgssapi3-heimdal\n  section: libs\n  version: 1.6~git20120311.dfsg.1-2\nlibgssapi3-heimdal:i386:\n  description: Heimdal Kerberos - GSSAPI support library\n  downloads: 0\n  homepage_uri: http://www.h5l.org/\n  name: libgssapi3-heimdal:i386\n  section: libs\n  version: 1.6~git20120311.dfsg.1-2\nlibgssdp-1.0-3:\n  description: GObject-based library for SSDP\n  downloads: 3568\n  homepage_uri: http://www.gupnp.org\n  name: libgssdp-1.0-3\n  section: libs\n  version: 0.12.1-2\nlibgssglue1:\n  description: mechanism-switch gssapi library\n  downloads: 13251\n  homepage_uri: http://www.citi.umich.edu/projects/nfsv4/linux/\n  name: libgssglue1\n  section: libs\n  version: 0.3-4\nlibgssrpc4:\n  description: MIT Kerberos runtime libraries - GSS enabled ONCRPC\n  downloads: 11945\n  homepage_uri: http://web.mit.edu/kerberos/\n  name: libgssrpc4\n  section: libs\n  version: 1.10+dfsg~beta1-2ubuntu0.1\nlibgstreamer-plugins-base0.10-0:\n  description: GStreamer libraries from the \"base\" set\n  downloads: 44173\n  homepage_uri: http://gstreamer.freedesktop.org\n  name: libgstreamer-plugins-base0.10-0\n  section: libs\n  version: 0.10.36-1\nlibgstreamer-plugins-base0.10-0:i386:\n  description: GStreamer libraries from the \"base\" set\n  downloads: 0\n  homepage_uri: http://gstreamer.freedesktop.org\n  name: libgstreamer-plugins-base0.10-0:i386\n  section: libs\n  version: 0.10.36-1\nlibgstreamer0.10-0:\n  description: Core GStreamer libraries and elements\n  downloads: 45345\n  homepage_uri: http://gstreamer.freedesktop.org\n  name: libgstreamer0.10-0\n  section: libs\n  version: 0.10.36-1ubuntu1\nlibgstreamer0.10-0:i386:\n  description: Core GStreamer libraries and elements\n  downloads: 0\n  homepage_uri: http://gstreamer.freedesktop.org\n  name: libgstreamer0.10-0:i386\n  section: libs\n  version: 0.10.36-1ubuntu1\nlibgtextutils0:\n  description: Gordon Text_utils library\n  downloads: 1\n  homepage_uri: http://hannonlab.cshl.edu/fastx_toolkit/\n  name: libgtextutils0\n  section: universe/libs\n  version: 0.6.2-1\nlibgtk-3-0:\n  description: GTK+ graphical user interface library\n  downloads: 15534\n  homepage_uri: http://www.gtk.org/\n  name: libgtk-3-0\n  section: libs\n  version: 3.4.2-0ubuntu0.2\nlibgtk-3-bin:\n  description: programs for the GTK+ graphical user interface library\n  downloads: 13404\n  homepage_uri: http://www.gtk.org/\n  name: libgtk-3-bin\n  section: misc\n  version: 3.4.2-0ubuntu0.2\nlibgtk-3-common:\n  description: common files for the GTK+ graphical user interface library\n  downloads: 15279\n  homepage_uri: http://www.gtk.org/\n  name: libgtk-3-common\n  section: misc\n  version: 3.4.2-0ubuntu0.2\nlibgtk2-perl:\n  description: Perl interface to the 2.x series of the Gimp Toolkit library\n  downloads: 17546\n  homepage_uri: http://gtk2-perl.sourceforge.net/\n  name: libgtk2-perl\n  section: perl\n  version: 2:1.223-1build3\nlibgtk2.0-0:\n  description: GTK+ graphical user interface library\n  downloads: 52473\n  homepage_uri: http://www.gtk.org/\n  name: libgtk2.0-0\n  section: libs\n  version: 2.24.10-0ubuntu6\nlibgtk2.0-0:i386:\n  description: GTK+ graphical user interface library\n  downloads: 0\n  homepage_uri: http://www.gtk.org/\n  name: libgtk2.0-0:i386\n  section: libs\n  version: 2.24.10-0ubuntu6\nlibgtk2.0-bin:\n  description: programs for the GTK+ graphical user interface library\n  downloads: 31537\n  homepage_uri: http://www.gtk.org/\n  name: libgtk2.0-bin\n  section: misc\n  version: 2.24.10-0ubuntu6\nlibgtk2.0-common:\n  description: common files for the GTK+ graphical user interface library\n  downloads: 51408\n  homepage_uri: http://www.gtk.org/\n  name: libgtk2.0-common\n  section: misc\n  version: 2.24.10-0ubuntu6\nlibgtk2.0-dev:\n  description: development files for the GTK+ library\n  downloads: 2991\n  homepage_uri: http://www.gtk.org/\n  name: libgtk2.0-dev\n  section: libdevel\n  version: 2.24.10-0ubuntu6\nlibgtkmm-3.0-1:\n  description: C++ wrappers for GTK+ (shared libraries)\n  downloads: 173\n  homepage_uri: http://www.gtkmm.org/\n  name: libgtkmm-3.0-1\n  section: libs\n  version: 3.4.0-0ubuntu1\nlibgtksourceview-3.0-0:\n  description: shared libraries for the GTK+ syntax highlighting widget\n  downloads: 5166\n  homepage_uri: http://projects.gnome.org/gtksourceview/\n  name: libgtksourceview-3.0-0\n  section: libs\n  version: 3.4.2-0ubuntu1\nlibgtksourceview-3.0-common:\n  description: common files for the GTK+ syntax highlighting widget\n  downloads: 4194\n  homepage_uri: http://projects.gnome.org/gtksourceview/\n  name: libgtksourceview-3.0-common\n  section: libs\n  version: 3.4.2-0ubuntu1\nlibgtkspell-3-0:\n  description: spell-checking addon for GTK's TextView widget\n  downloads: 52\n  homepage_uri: http://gtkspell.sourceforge.net/\n  name: libgtkspell-3-0\n  section: libs\n  version: 3.0.0~hg20110814-1\nlibgtop2-7:\n  description: gtop system monitoring library (shared)\n  downloads: 35432\n  homepage_uri: ''\n  name: libgtop2-7\n  section: libs\n  version: 2.28.4-2\nlibgtop2-common:\n  description: gtop system monitoring library (common)\n  downloads: 29978\n  homepage_uri: ''\n  name: libgtop2-common\n  section: libs\n  version: 2.28.4-2\nlibgucharmap-2-90-7:\n  description: Unicode browser widget library (shared library)\n  downloads: 3886\n  homepage_uri: http://live.gnome.org/Gucharmap\n  name: libgucharmap-2-90-7\n  section: libs\n  version: 1:3.4.1.1-0ubuntu1\nlibgudev-1.0-0:\n  description: GObject-based wrapper library for libudev\n  downloads: 44090\n  homepage_uri: http://www.kernel.org/pub/linux/utils/kernel/hotplug/udev.html\n  name: libgudev-1.0-0\n  section: libs\n  version: 1:175-0ubuntu9\nlibgudev-1.0-0:i386:\n  description: GObject-based wrapper library for libudev\n  downloads: 0\n  homepage_uri: http://www.kernel.org/pub/linux/utils/kernel/hotplug/udev.html\n  name: libgudev-1.0-0:i386\n  section: libs\n  version: 1:175-0ubuntu9\nlibgupnp-1.0-4:\n  description: GObject-based library for UPnP\n  downloads: 3564\n  homepage_uri: http://www.gupnp.org\n  name: libgupnp-1.0-4\n  section: libs\n  version: 0.18.1-2\nlibgupnp-igd-1.0-4:\n  description: library to handle UPnP IGD port mapping\n  downloads: 3098\n  homepage_uri: http://www.gupnp.org/\n  name: libgupnp-igd-1.0-4\n  section: libs\n  version: 0.2.1-2\nlibgutenprint2:\n  description: runtime for the Gutenprint printer driver library\n  downloads: 11599\n  homepage_uri: ''\n  name: libgutenprint2\n  section: libs\n  version: 5.2.8~pre1-0ubuntu2\nlibgvc5:\n  description: rich set of graph drawing tools - gvc library\n  downloads: 3789\n  homepage_uri: http://www.graphviz.org/\n  name: libgvc5\n  section: libs\n  version: 2.26.3-10ubuntu1\nlibgvpr1:\n  description: rich set of graph drawing tools - gvpr library\n  downloads: 2163\n  homepage_uri: http://www.graphviz.org/\n  name: libgvpr1\n  section: libs\n  version: 2.26.3-10ubuntu1\nlibgweather-3-0:\n  description: GWeather shared library\n  downloads: 7369\n  homepage_uri: ''\n  name: libgweather-3-0\n  section: libs\n  version: 3.4.1-0ubuntu1\nlibgweather-common:\n  description: GWeather common files\n  downloads: 6027\n  homepage_uri: ''\n  name: libgweather-common\n  section: libs\n  version: 3.4.1-0ubuntu1\nlibgwibber-gtk2:\n  description: Gwibber GTK Widgets\n  downloads: 0\n  homepage_uri: https://launchpad.net/gwibber\n  name: libgwibber-gtk2\n  section: libs\n  version: 3.4.1-0ubuntu1\nlibgwibber2:\n  description: Gwibber - shared library\n  downloads: 0\n  homepage_uri: https://launchpad.net/gwibber\n  name: libgwibber2\n  section: libs\n  version: 3.4.1-0ubuntu1\nlibhal1:\n  description: Hardware Abstraction Layer - shared library\n  downloads: 24663\n  homepage_uri: http://hal.freedesktop.org/\n  name: libhal1\n  section: universe/libs\n  version: 0.5.14-8\nlibhamcrest-java:\n  description: library of matchers for building test expressions\n  downloads: 1000\n  homepage_uri: http://code.google.com/p/hamcrest/\n  name: libhamcrest-java\n  section: libs\n  version: 1.1-8\nlibhawtjni-runtime-java:\n  description: Java library that provide JNI code generation\n  downloads: 145\n  homepage_uri: http://fusesource.com/forge/sites/hawtjni/\n  name: libhawtjni-runtime-java\n  section: universe/java\n  version: 1.0~+git0c502e20c4-3\nlibhcrypto4-heimdal:\n  description: Heimdal Kerberos - crypto library\n  downloads: 156\n  homepage_uri: http://www.h5l.org/\n  name: libhcrypto4-heimdal\n  section: libs\n  version: 1.6~git20120311.dfsg.1-2\nlibhcrypto4-heimdal:i386:\n  description: Heimdal Kerberos - crypto library\n  downloads: 0\n  homepage_uri: http://www.h5l.org/\n  name: libhcrypto4-heimdal:i386\n  section: libs\n  version: 1.6~git20120311.dfsg.1-2\nlibhdf5-serial-1.8.4:\n  description: Hierarchical Data Format 5 (HDF5) - runtime files - serial version\n  downloads: 0\n  homepage_uri: http://hdfgroup.org/HDF5/\n  name: libhdf5-serial-1.8.4\n  section: universe/libs\n  version: 1.8.4-patch1-3ubuntu2\nlibhdf5-serial-dev:\n  description: Hierarchical Data Format 5 (HDF5) - development files - serial version\n  downloads: 123\n  homepage_uri: http://hdfgroup.org/HDF5/\n  name: libhdf5-serial-dev\n  section: universe/libdevel\n  version: 1.8.4-patch1-3ubuntu2\nlibhdfs0:\n  description: JNI Bindings to access Hadoop HDFS from C\n  downloads: 0\n  homepage_uri: http://hadoop.apache.org/core/\n  name: libhdfs0\n  section: misc\n  version: 0.20.2+923.256-1~maverick-cdh3\nlibhdfs0-dev:\n  description: Development support for libhdfs0\n  downloads: 0\n  homepage_uri: http://hadoop.apache.org/core/\n  name: libhdfs0-dev\n  section: libdevel\n  version: 0.20.2+923.256-1~maverick-cdh3\nlibheimbase1-heimdal:\n  description: Heimdal Kerberos - Base library\n  downloads: 124\n  homepage_uri: http://www.h5l.org/\n  name: libheimbase1-heimdal\n  section: libs\n  version: 1.6~git20120311.dfsg.1-2\nlibheimbase1-heimdal:i386:\n  description: Heimdal Kerberos - Base library\n  downloads: 0\n  homepage_uri: http://www.h5l.org/\n  name: libheimbase1-heimdal:i386\n  section: libs\n  version: 1.6~git20120311.dfsg.1-2\nlibheimntlm0-heimdal:\n  description: Heimdal Kerberos - NTLM support library\n  downloads: 159\n  homepage_uri: http://www.h5l.org/\n  name: libheimntlm0-heimdal\n  section: libs\n  version: 1.6~git20120311.dfsg.1-2\nlibheimntlm0-heimdal:i386:\n  description: Heimdal Kerberos - NTLM support library\n  downloads: 0\n  homepage_uri: http://www.h5l.org/\n  name: libheimntlm0-heimdal:i386\n  section: libs\n  version: 1.6~git20120311.dfsg.1-2\nlibhpdf-2.2.1:\n  description: C library for generating pdf files\n  downloads: 30\n  homepage_uri: http://www.libharu.org\n  name: libhpdf-2.2.1\n  section: universe/libs\n  version: 2.2.1-1\nlibhpmud0:\n  description: HP Multi-Point Transport Driver (hpmud) run-time libraries\n  downloads: 22962\n  homepage_uri: http://hplipopensource.com/hplip-web/index.html\n  name: libhpmud0\n  section: libs\n  version: 3.12.2-1ubuntu3\nlibhsqldb-java:\n  description: Java SQL database engine\n  downloads: 769\n  homepage_uri: ''\n  name: libhsqldb-java\n  section: libs\n  version: 1.8.0.10-9ubuntu2\nlibhtml-form-perl:\n  description: module that represents an HTML form element\n  downloads: 869\n  homepage_uri: http://search.cpan.org/dist/HTML-Form/\n  name: libhtml-form-perl\n  section: perl\n  version: 6.00-1\nlibhtml-format-perl:\n  description: module for transforming HTML into various formats\n  downloads: 2414\n  homepage_uri: http://search.cpan.org/dist/HTML-Format/\n  name: libhtml-format-perl\n  section: perl\n  version: 2.10-1\nlibhtml-parser-perl:\n  description: collection of modules that parse HTML text documents\n  downloads: 25434\n  homepage_uri: http://search.cpan.org/dist/HTML-Parser/\n  name: libhtml-parser-perl\n  section: perl\n  version: 3.69-1build1\nlibhtml-tagset-perl:\n  description: Data tables pertaining to HTML\n  downloads: 25059\n  homepage_uri: http://search.cpan.org/dist/HTML-Tagset/\n  name: libhtml-tagset-perl\n  section: perl\n  version: 3.20-2\nlibhtml-template-perl:\n  description: module for using HTML Templates with Perl\n  downloads: 5445\n  homepage_uri: http://search.cpan.org/dist/HTML-Template/\n  name: libhtml-template-perl\n  section: perl\n  version: 2.10-1\nlibhtml-tree-perl:\n  description: Perl module to represent and create HTML syntax trees\n  downloads: 22482\n  homepage_uri: http://search.cpan.org/dist/HTML-Tree/\n  name: libhtml-tree-perl\n  section: perl\n  version: 4.2-1\nlibhttp-cookies-perl:\n  description: HTTP cookie jars\n  downloads: 5256\n  homepage_uri: http://search.cpan.org/dist/HTTP-Cookies/\n  name: libhttp-cookies-perl\n  section: perl\n  version: 6.00-2\nlibhttp-daemon-perl:\n  description: simple http server class\n  downloads: 1195\n  homepage_uri: http://search.cpan.org/dist/HTTP-Daemon/\n  name: libhttp-daemon-perl\n  section: perl\n  version: 6.00-1\nlibhttp-date-perl:\n  description: module of date conversion routines\n  downloads: 5326\n  homepage_uri: http://search.cpan.org/dist/HTTP-Date/\n  name: libhttp-date-perl\n  section: perl\n  version: 6.00-1\nlibhttp-message-perl:\n  description: perl interface to HTTP style messages\n  downloads: 5299\n  homepage_uri: http://search.cpan.org/dist/HTTP-Message/\n  name: libhttp-message-perl\n  section: perl\n  version: 6.01-1\nlibhttp-negotiate-perl:\n  description: implementation of content negotiation\n  downloads: 5255\n  homepage_uri: http://search.cpan.org/dist/HTTP-Negotiate/\n  name: libhttp-negotiate-perl\n  section: perl\n  version: 6.00-2\nlibhttpclient-java:\n  description: HTTP/1.1 compliant HTTP agent implementation\n  downloads: 116\n  homepage_uri: http://hc.apache.org/httpcomponents-client/index.html\n  name: libhttpclient-java\n  section: universe/java\n  version: 4.1.1-1\nlibhttpcore-java:\n  description: set of low level HTTP transport components for Java\n  downloads: 93\n  homepage_uri: http://hc.apache.org/httpcomponents-core/index.html\n  name: libhttpcore-java\n  section: universe/java\n  version: 4.1.4-1\nlibhunspell-1.3-0:\n  description: spell checker and morphological analyzer (shared library)\n  downloads: 3059\n  homepage_uri: http://hunspell.sourceforge.net/\n  name: libhunspell-1.3-0\n  section: libs\n  version: 1.3.2-4\nlibhx509-5-heimdal:\n  description: Heimdal Kerberos - X509 support library\n  downloads: 339\n  homepage_uri: http://www.h5l.org/\n  name: libhx509-5-heimdal\n  section: libs\n  version: 1.6~git20120311.dfsg.1-2\nlibhx509-5-heimdal:i386:\n  description: Heimdal Kerberos - X509 support library\n  downloads: 0\n  homepage_uri: http://www.h5l.org/\n  name: libhx509-5-heimdal:i386\n  section: libs\n  version: 1.6~git20120311.dfsg.1-2\nlibhyphen0:\n  description: ALTLinux hyphenation library - shared library\n  downloads: 41\n  homepage_uri: ''\n  name: libhyphen0\n  section: libs\n  version: 2.8.3-1\nlibibus-1.0-0:\n  description: Intelligent Input Bus - shared library\n  downloads: 401\n  homepage_uri: http://code.google.com/p/ibus/\n  name: libibus-1.0-0\n  section: libs\n  version: 1.4.1-3ubuntu1\nlibibus-1.0-0:i386:\n  description: Intelligent Input Bus - shared library\n  downloads: 0\n  homepage_uri: http://code.google.com/p/ibus/\n  name: libibus-1.0-0:i386\n  section: libs\n  version: 1.4.1-3ubuntu1\nlibibverbs-dev:\n  description: Development files for the libibverbs library\n  downloads: 1254\n  homepage_uri: http://www.openfabrics.org/\n  name: libibverbs-dev\n  section: libdevel\n  version: 1.1.5-1ubuntu1\nlibibverbs1:\n  description: Library for direct userspace use of RDMA (InfiniBand/iWARP)\n  downloads: 1349\n  homepage_uri: http://www.openfabrics.org/\n  name: libibverbs1\n  section: libs\n  version: 1.1.5-1ubuntu1\nlibical0:\n  description: iCalendar library implementation in C (runtime)\n  downloads: 30310\n  homepage_uri: http://freeassociation.sourceforge.net\n  name: libical0\n  section: libs\n  version: 0.48-1ubuntu3\nlibice-dev:\n  description: X11 Inter-Client Exchange library (development headers)\n  downloads: 3671\n  homepage_uri: ''\n  name: libice-dev\n  section: libdevel\n  version: 2:1.0.7-2build1\nlibice6:\n  description: X11 Inter-Client Exchange library\n  downloads: 56047\n  homepage_uri: ''\n  name: libice6\n  section: libs\n  version: 2:1.0.7-2build1\nlibice6:i386:\n  description: X11 Inter-Client Exchange library\n  downloads: 0\n  homepage_uri: ''\n  name: libice6:i386\n  section: libs\n  version: 2:1.0.7-2build1\nlibicu-dev:\n  description: Development files for International Components for Unicode\n  downloads: 762\n  homepage_uri: http://www.icu-project.org\n  name: libicu-dev\n  section: libdevel\n  version: 4.8.1.1-3\nlibicu48:\n  description: International Components for Unicode\n  downloads: 3750\n  homepage_uri: http://www.icu-project.org\n  name: libicu48\n  section: libs\n  version: 4.8.1.1-3\nlibicu4j-4.4-java:\n  description: Library for Unicode support and internalisation\n  downloads: 691\n  homepage_uri: http://www.icu-project.org/\n  name: libicu4j-4.4-java\n  section: universe/java\n  version: 4.4.2.2-1\nlibicu4j-java:\n  description: Library for unicode support and internalisation\n  downloads: 0\n  homepage_uri: http://www.icu-project.org/\n  name: libicu4j-java\n  section: universe/devel\n  version: 4.2.1.1-1\nlibidl-common:\n  description: library for parsing CORBA IDL files (common files)\n  downloads: 0\n  homepage_uri: ''\n  name: libidl-common\n  section: libs\n  version: 0.8.14-0.2ubuntu2\nlibidl-dev:\n  description: development files for programs that use libIDL\n  downloads: 531\n  homepage_uri: ''\n  name: libidl-dev\n  section: libdevel\n  version: 0.8.14-0.2ubuntu2\nlibidl0:\n  description: library for parsing CORBA IDL files\n  downloads: 26535\n  homepage_uri: ''\n  name: libidl0\n  section: libs\n  version: 0.8.14-0.2ubuntu2\nlibidn11:\n  description: GNU Libidn library, implementation of IETF IDN specifications\n  downloads: 62674\n  homepage_uri: http://www.gnu.org/software/libidn/\n  name: libidn11\n  section: libs\n  version: 1.23-2\nlibidn11-dev:\n  description: Development files for GNU Libidn, an IDN library\n  downloads: 2178\n  homepage_uri: http://www.gnu.org/software/libidn/\n  name: libidn11-dev\n  section: libdevel\n  version: 1.23-2\nlibidn11:i386:\n  description: GNU Libidn library, implementation of IETF IDN specifications\n  downloads: 0\n  homepage_uri: http://www.gnu.org/software/libidn/\n  name: libidn11:i386\n  section: libs\n  version: 1.23-2\nlibido3-0.1-0:\n  description: Shared library providing extra gtk menu items for display in\n  downloads: 141\n  homepage_uri: https://launchpad.net/ido\n  name: libido3-0.1-0\n  section: libs\n  version: 0.3.4-0ubuntu1\nlibiec61883-0:\n  description: an partial implementation of IEC 61883\n  downloads: 61\n  homepage_uri: ''\n  name: libiec61883-0\n  section: libs\n  version: 1.2.0-0.1ubuntu1\nlibiec61883-0:i386:\n  description: an partial implementation of IEC 61883\n  downloads: 0\n  homepage_uri: ''\n  name: libiec61883-0:i386\n  section: libs\n  version: 1.2.0-0.1ubuntu1\nlibieee1284-3:\n  description: cross-platform library for parallel port access\n  downloads: 23399\n  homepage_uri: ''\n  name: libieee1284-3\n  section: libs\n  version: 0.2.11-10build1\nlibieee1284-3:i386:\n  description: cross-platform library for parallel port access\n  downloads: 0\n  homepage_uri: ''\n  name: libieee1284-3:i386\n  section: libs\n  version: 0.2.11-10build1\nlibijs-0.35:\n  description: 'IJS raster image transport protocol: shared library'\n  downloads: 8343\n  homepage_uri: http://www.linuxprinting.org/ijs/\n  name: libijs-0.35\n  section: libs\n  version: 0.35-8\nlibilmbase6:\n  description: several utility libraries from ILM used by OpenEXR\n  downloads: 288\n  homepage_uri: http://www.openexr.com\n  name: libilmbase6\n  section: libs\n  version: 1.0.1-3build2\nlibimobiledevice2:\n  description: Library for communicating with the iPhone and iPod Touch\n  downloads: 14843\n  homepage_uri: http://libimobiledevice.org/\n  name: libimobiledevice2\n  section: libs\n  version: 1.1.1-4\nlibindicate-gtk3:\n  description: library for raising indicators via DBus - GTK+ bindings\n  downloads: 284\n  homepage_uri: https://launchpad.net/libindicate\n  name: libindicate-gtk3\n  section: libs\n  version: 0.6.92-0ubuntu1\nlibindicate5:\n  description: library for raising indicators via DBus\n  downloads: 452\n  homepage_uri: https://launchpad.net/libindicate\n  name: libindicate5\n  section: libs\n  version: 0.6.92-0ubuntu1\nlibindicator-messages-status-provider1:\n  description: indicator status provider - shared library\n  downloads: 81\n  homepage_uri: https://launchpad.net/indicator-messages\n  name: libindicator-messages-status-provider1\n  section: libs\n  version: 0.6.0-0ubuntu1\nlibindicator3-7:\n  description: panel indicator applet - shared library\n  downloads: 219\n  homepage_uri: https://launchpad.net/libindicator\n  name: libindicator3-7\n  section: libs\n  version: 0.5.0-0ubuntu1\nlibindicator7:\n  description: panel indicator applet - shared library\n  downloads: 70\n  homepage_uri: https://launchpad.net/libindicator\n  name: libindicator7\n  section: libs\n  version: 0.5.0-0ubuntu1\nlibio-pty-perl:\n  description: Perl module for pseudo tty IO\n  downloads: 4862\n  homepage_uri: ''\n  name: libio-pty-perl\n  section: perl\n  version: 1:1.08-1build2\nlibio-socket-inet6-perl:\n  description: object interface for AF_INET6 domain sockets\n  downloads: 9137\n  homepage_uri: http://search.cpan.org/dist/IO-Socket-INET6/\n  name: libio-socket-inet6-perl\n  section: perl\n  version: 2.69-2\nlibio-socket-ssl-perl:\n  description: Perl module implementing object oriented interface to SSL sockets\n  downloads: 5741\n  homepage_uri: http://search.cpan.org/dist/IO-Socket-SSL/\n  name: libio-socket-ssl-perl\n  section: perl\n  version: 1.53-1\nlibio-string-perl:\n  description: Emulate IO::File interface for in-core strings\n  downloads: 3238\n  homepage_uri: ''\n  name: libio-string-perl\n  section: perl\n  version: 1.08-2\nlibio-stringy-perl:\n  description: Perl modules for IO from scalars and arrays\n  downloads: 6820\n  homepage_uri: http://search.cpan.org/dist/IO-stringy/\n  name: libio-stringy-perl\n  section: perl\n  version: 2.110-5\nlibipc-run-perl:\n  description: Perl module for running processes\n  downloads: 1669\n  homepage_uri: http://search.cpan.org/dist/IPC-Run/\n  name: libipc-run-perl\n  section: perl\n  version: 0.90-1\nlibipc-sharelite-perl:\n  description: Perl module that provides a simple interface to shared memory\n  downloads: 639\n  homepage_uri: http://search.cpan.org/dist/IPC-ShareLite/\n  name: libipc-sharelite-perl\n  section: perl\n  version: 0.17-1+build4\nlibisc83:\n  description: ISC Shared Library used by BIND\n  downloads: 1908\n  homepage_uri: ''\n  name: libisc83\n  section: libs\n  version: 1:9.8.1.dfsg.P1-4ubuntu0.1\nlibisccc80:\n  description: Command Channel Library used by BIND\n  downloads: 1200\n  homepage_uri: ''\n  name: libisccc80\n  section: libs\n  version: 1:9.8.1.dfsg.P1-4ubuntu0.1\nlibisccfg82:\n  description: Config File Handling Library used by BIND\n  downloads: 1907\n  homepage_uri: ''\n  name: libisccfg82\n  section: libs\n  version: 1:9.8.1.dfsg.P1-4ubuntu0.1\nlibisofs6:\n  description: library to create ISO9660 images\n  downloads: 12871\n  homepage_uri: http://libburnia-project.org\n  name: libisofs6\n  section: libs\n  version: 1.1.6-1ubuntu1\nlibitext1-java:\n  description: Java Library to generate PDF on the Fly\n  downloads: 8\n  homepage_uri: http://www.lowagie.com/iText/\n  name: libitext1-java\n  section: universe/java\n  version: 1.4-5\nlibiw30:\n  description: Wireless tools - library\n  downloads: 10789\n  homepage_uri: http://www.hpl.hp.com/personal/Jean_Tourrilhes/Linux/Tools.html\n  name: libiw30\n  section: libs\n  version: 30~pre9-5ubuntu2\nlibjack-jackd2-0:\n  description: JACK Audio Connection Kit (libraries)\n  downloads: 5632\n  homepage_uri: http://jackaudio.org/\n  name: libjack-jackd2-0\n  section: libs\n  version: 1.9.8~dfsg.1-1ubuntu1\nlibjack-jackd2-0:i386:\n  description: JACK Audio Connection Kit (libraries)\n  downloads: 0\n  homepage_uri: http://jackaudio.org/\n  name: libjack-jackd2-0:i386\n  section: libs\n  version: 1.9.8~dfsg.1-1ubuntu1\nlibjansi-java:\n  description: Java library for generating and interpreting ANSI escape sequences\n  downloads: 310\n  homepage_uri: http://jansi.fusesource.org/\n  name: libjansi-java\n  section: universe/java\n  version: 1.4-3\nlibjansi-native-java:\n  description: Java library that allows to use ANSI escape sequences - Native\n  downloads: 145\n  homepage_uri: http://jansi.fusesource.org/\n  name: libjansi-native-java\n  section: universe/java\n  version: 1.0-3\nlibjasper-java:\n  description: Implementation of the JSP Container\n  downloads: 0\n  homepage_uri: http://tomcat.apache.org\n  name: libjasper-java\n  section: universe/java\n  version: 5.5.33-2\nlibjasper1:\n  description: JasPer JPEG-2000 runtime library\n  downloads: 26321\n  homepage_uri: ''\n  name: libjasper1\n  section: libs\n  version: 1.900.1-13\nlibjasper1:i386:\n  description: JasPer JPEG-2000 runtime library\n  downloads: 0\n  homepage_uri: ''\n  name: libjasper1:i386\n  section: libs\n  version: 1.900.1-13\nlibjavascriptcoregtk-1.0-0:\n  description: Javascript engine library for GTK+\n  downloads: 2263\n  homepage_uri: http://webkitgtk.org/\n  name: libjavascriptcoregtk-1.0-0\n  section: libs\n  version: 1.8.0-0ubuntu2\nlibjavascriptcoregtk-3.0-0:\n  description: Javascript engine library for GTK+\n  downloads: 10097\n  homepage_uri: http://webkitgtk.org/\n  name: libjavascriptcoregtk-3.0-0\n  section: libs\n  version: 1.8.0-0ubuntu2\nlibjaxen-java:\n  description: Java XPath engine\n  downloads: 256\n  homepage_uri: http://jaxen.org\n  name: libjaxen-java\n  section: libs\n  version: 1.1.3-1\nlibjaxme-java:\n  description: implementation of the JAXB specification for Java/XML binding\n  downloads: 215\n  homepage_uri: http://ws.apache.org/jaxme/\n  name: libjaxme-java\n  section: libs\n  version: 0.5.2+dfsg-6\nlibjaxp1.3-java:\n  description: Java XML parser and transformer APIs (DOM, SAX, JAXP, TrAX)\n  downloads: 8312\n  homepage_uri: http://java.sun.com/xml/jaxp/index.jsp\n  name: libjaxp1.3-java\n  section: libs\n  version: 1.3.05-2ubuntu2\nlibjbig2dec0:\n  description: JBIG2 decoder library - shared libraries\n  downloads: 3554\n  homepage_uri: http://jbig2dec.sourceforge.net/\n  name: libjbig2dec0\n  section: libs\n  version: 0.11-1ubuntu1\nlibjdom1-java:\n  description: lightweight and fast library using XML\n  downloads: 239\n  homepage_uri: http://jdom.org/\n  name: libjdom1-java\n  section: libs\n  version: 1.1.2+dfsg-2\nlibjetty-java:\n  description: Java servlet engine and webserver -- core libraries\n  downloads: 356\n  homepage_uri: http://jetty.mortbay.com/\n  name: libjetty-java\n  section: universe/java\n  version: 6.1.24-6ubuntu0.12.04.1\nlibjffi-java:\n  description: Java Foreign Function Interface\n  downloads: 38\n  homepage_uri: http://github.com/wmeissner/jffi\n  name: libjffi-java\n  section: universe/java\n  version: 1.0.2-8ubuntu1\nlibjffi-jni:\n  description: Java Foreign Function Interface (JNI library)\n  downloads: 50\n  homepage_uri: http://github.com/wmeissner/jffi\n  name: libjffi-jni\n  section: universe/java\n  version: 1.0.2-8ubuntu1\nlibjhdf5-java:\n  description: Java HDF5 Object Package\n  downloads: 39\n  homepage_uri: http://www.hdfgroup.org/hdf-java-html/\n  name: libjhdf5-java\n  section: universe/java\n  version: 2.6.1-2build1\nlibjhdf5-jni:\n  description: Java HDF5 Object Package (Java JNI library)\n  downloads: 69\n  homepage_uri: http://www.hdfgroup.org/hdf-java-html/\n  name: libjhdf5-jni\n  section: universe/libs\n  version: 2.6.1-2build1\nlibjline-java:\n  description: Java library for handling console input\n  downloads: 3048\n  homepage_uri: http://jline.sourceforge.net/\n  name: libjline-java\n  section: libs\n  version: 1.0-1\nlibjna-java:\n  description: Dynamic access of native libraries from Java without JNI\n  downloads: 573\n  homepage_uri: http://jna.dev.java.net/\n  name: libjna-java\n  section: universe/libs\n  version: 3.2.7-4\nlibjna-posix-java:\n  description: basic POSIX-like functions for Java\n  downloads: 12\n  homepage_uri: http://kenai.com/projects/jna-posix\n  name: libjna-posix-java\n  section: universe/java\n  version: 1.0.1-1\nlibjpeg-dev:\n  description: Independent JPEG Group's JPEG runtime library (dependency package)\n  downloads: 2\n  homepage_uri: ''\n  name: libjpeg-dev\n  section: libdevel\n  version: 8c-2ubuntu7\nlibjpeg-turbo8:\n  description: IJG JPEG compliant runtime library.\n  downloads: 1\n  homepage_uri: http://libjpeg-turbo.virtualgl.org/\n  name: libjpeg-turbo8\n  section: libs\n  version: 1.1.90+svn733-0ubuntu4\nlibjpeg-turbo8-dev:\n  description: Development files for the IJG JPEG library\n  downloads: 1\n  homepage_uri: http://libjpeg-turbo.virtualgl.org/\n  name: libjpeg-turbo8-dev\n  section: libdevel\n  version: 1.1.90+svn733-0ubuntu4\nlibjpeg-turbo8:i386:\n  description: IJG JPEG compliant runtime library.\n  downloads: 0\n  homepage_uri: http://libjpeg-turbo.virtualgl.org/\n  name: libjpeg-turbo8:i386\n  section: libs\n  version: 1.1.90+svn733-0ubuntu4\nlibjpeg8:\n  description: Independent JPEG Group's JPEG runtime library (dependency package)\n  downloads: 30858\n  homepage_uri: ''\n  name: libjpeg8\n  section: libs\n  version: 8c-2ubuntu7\nlibjpeg8-dev:\n  description: Independent JPEG Group's JPEG runtime library (dependency package)\n  downloads: 1545\n  homepage_uri: ''\n  name: libjpeg8-dev\n  section: libdevel\n  version: 8c-2ubuntu7\nlibjpeg8:i386:\n  description: Independent JPEG Group's JPEG runtime library (dependency package)\n  downloads: 0\n  homepage_uri: ''\n  name: libjpeg8:i386\n  section: libs\n  version: 8c-2ubuntu7\nlibjs-jquery:\n  description: JavaScript library for dynamic web applications\n  downloads: 3979\n  homepage_uri: http://jquery.com/\n  name: libjs-jquery\n  section: web\n  version: 1.7.1-1ubuntu1\nlibjsch-java:\n  description: pure Java implementation of the SSH2 protocol\n  downloads: 1770\n  homepage_uri: http://www.jcraft.com/jsch/\n  name: libjsch-java\n  section: devel\n  version: 0.1.42-2fakesync1\nlibjson-glib-1.0-0:\n  description: GLib JSON manipulation library\n  downloads: 25681\n  homepage_uri: http://live.gnome.org/JsonGlib\n  name: libjson-glib-1.0-0\n  section: libs\n  version: 0.14.2-1\nlibjson-perl:\n  description: module for manipulating JSON-formatted data\n  downloads: 1823\n  homepage_uri: http://search.cpan.org/dist/JSON/\n  name: libjson-perl\n  section: perl\n  version: 2.53-1\nlibjson-xs-perl:\n  description: module for manipulating JSON-formatted data (C/XS-accelerated)\n  downloads: 2339\n  homepage_uri: http://search.cpan.org/dist/JSON-XS/\n  name: libjson-xs-perl\n  section: perl\n  version: 2.320-1build1\nlibjson0:\n  description: JSON manipulation library - shared library\n  downloads: 575\n  homepage_uri: http://oss.metaparadigm.com/json-c/\n  name: libjson0\n  section: libs\n  version: 0.9-1ubuntu1\nlibjson0:i386:\n  description: JSON manipulation library - shared library\n  downloads: 0\n  homepage_uri: http://oss.metaparadigm.com/json-c/\n  name: libjson0:i386\n  section: libs\n  version: 0.9-1ubuntu1\nlibjsoup-java:\n  description: Java HTML parser that makes sense of real-world HTML soup\n  downloads: 9\n  homepage_uri: http://jsoup.org/\n  name: libjsoup-java\n  section: universe/java\n  version: 1.6.1-2\nlibjsr305-java:\n  description: Java library that provides annotations for software defect detection\n  downloads: 3\n  homepage_uri: http://code.google.com/p/jsr-305/\n  name: libjsr305-java\n  section: universe/java\n  version: 0.1~+svn49-4\nlibjte1:\n  description: Jigdo Template Export - runtime library\n  downloads: 138\n  homepage_uri: http://www.einval.com/~steve/software/JTE/\n  name: libjte1\n  section: libs\n  version: 1.19-1\nlibjtidy-java:\n  description: JTidy\n  downloads: 26\n  homepage_uri: http://jtidy.sourceforge.net/\n  name: libjtidy-java\n  section: universe/libs\n  version: 7+svn20110807-3\nlibk5crypto3:\n  description: MIT Kerberos runtime libraries - Crypto Library\n  downloads: 77684\n  homepage_uri: http://web.mit.edu/kerberos/\n  name: libk5crypto3\n  section: libs\n  version: 1.10+dfsg~beta1-2ubuntu0.1\nlibk5crypto3:i386:\n  description: MIT Kerberos runtime libraries - Crypto Library\n  downloads: 0\n  homepage_uri: http://web.mit.edu/kerberos/\n  name: libk5crypto3:i386\n  section: libs\n  version: 1.10+dfsg~beta1-2ubuntu0.1\nlibkadm5clnt-mit8:\n  description: MIT Kerberos runtime libraries - Administration Clients\n  downloads: 1838\n  homepage_uri: http://web.mit.edu/kerberos/\n  name: libkadm5clnt-mit8\n  section: libs\n  version: 1.10+dfsg~beta1-2ubuntu0.1\nlibkadm5srv-mit8:\n  description: MIT Kerberos runtime libraries - KDC and Admin Server\n  downloads: 1835\n  homepage_uri: http://web.mit.edu/kerberos/\n  name: libkadm5srv-mit8\n  section: libs\n  version: 1.10+dfsg~beta1-2ubuntu0.1\nlibkdb5-6:\n  description: MIT Kerberos runtime libraries - Kerberos database\n  downloads: 1388\n  homepage_uri: http://web.mit.edu/kerberos/\n  name: libkdb5-6\n  section: libs\n  version: 1.10+dfsg~beta1-2ubuntu0.1\nlibkeyutils1:\n  description: Linux Key Management Utilities (library)\n  downloads: 73600\n  homepage_uri: http://people.redhat.com/~dhowells/keyutils/\n  name: libkeyutils1\n  section: misc\n  version: 1.5.2-2\nlibkeyutils1:i386:\n  description: Linux Key Management Utilities (library)\n  downloads: 0\n  homepage_uri: http://people.redhat.com/~dhowells/keyutils/\n  name: libkeyutils1:i386\n  section: misc\n  version: 1.5.2-2\nlibklibc:\n  description: minimal libc subset for use with initramfs\n  downloads: 25583\n  homepage_uri: ''\n  name: libklibc\n  section: libs\n  version: 1.5.25-1ubuntu2\nlibkms1:\n  description: Userspace interface to kernel DRM buffer management\n  downloads: 315\n  homepage_uri: ''\n  name: libkms1\n  section: libs\n  version: 2.4.32-1ubuntu1\nlibkpathsea5:\n  description: 'TeX Live: path search library for TeX (runtime part)'\n  downloads: 9\n  homepage_uri: http://www.tug.org/texlive\n  name: libkpathsea5\n  section: libs\n  version: 2009-11ubuntu2\nlibkrb5-26-heimdal:\n  description: Heimdal Kerberos - libraries\n  downloads: 438\n  homepage_uri: http://www.h5l.org/\n  name: libkrb5-26-heimdal\n  section: libs\n  version: 1.6~git20120311.dfsg.1-2\nlibkrb5-26-heimdal:i386:\n  description: Heimdal Kerberos - libraries\n  downloads: 0\n  homepage_uri: http://www.h5l.org/\n  name: libkrb5-26-heimdal:i386\n  section: libs\n  version: 1.6~git20120311.dfsg.1-2\nlibkrb5-3:\n  description: MIT Kerberos runtime libraries\n  downloads: 83479\n  homepage_uri: http://web.mit.edu/kerberos/\n  name: libkrb5-3\n  section: libs\n  version: 1.10+dfsg~beta1-2ubuntu0.1\nlibkrb5-3:i386:\n  description: MIT Kerberos runtime libraries\n  downloads: 0\n  homepage_uri: http://web.mit.edu/kerberos/\n  name: libkrb5-3:i386\n  section: libs\n  version: 1.10+dfsg~beta1-2ubuntu0.1\nlibkrb5-dev:\n  description: Headers and development libraries for MIT Kerberos\n  downloads: 3792\n  homepage_uri: http://web.mit.edu/kerberos/\n  name: libkrb5-dev\n  section: libdevel\n  version: 1.10+dfsg~beta1-2ubuntu0.1\nlibkrb5support0:\n  description: MIT Kerberos runtime libraries - Support library\n  downloads: 72904\n  homepage_uri: http://web.mit.edu/kerberos/\n  name: libkrb5support0\n  section: libs\n  version: 1.10+dfsg~beta1-2ubuntu0.1\nlibkrb5support0:i386:\n  description: MIT Kerberos runtime libraries - Support library\n  downloads: 0\n  homepage_uri: http://web.mit.edu/kerberos/\n  name: libkrb5support0:i386\n  section: libs\n  version: 1.10+dfsg~beta1-2ubuntu0.1\nlibladr4:\n  description: the LADR deduction library\n  downloads: 4\n  homepage_uri: http://www.cs.unm.edu/~mccune/mace4/\n  name: libladr4\n  section: universe/libs\n  version: 0.0.200902a-2\nliblapack-dev:\n  description: library of linear algebra routines 3 - static version\n  downloads: 816\n  homepage_uri: http://www.netlib.org/lapack/\n  name: liblapack-dev\n  section: devel\n  version: 3.3.1-1\nliblapack3gf:\n  description: library of linear algebra routines 3 - shared version\n  downloads: 4966\n  homepage_uri: http://www.netlib.org/lapack/\n  name: liblapack3gf\n  section: libs\n  version: 3.3.1-1\nliblaunchpad-integration-3.0-1:\n  description: library for launchpad integration\n  downloads: 0\n  homepage_uri: ''\n  name: liblaunchpad-integration-3.0-1\n  section: libs\n  version: 0.1.56\nliblaunchpad-integration-common:\n  description: library for launchpad integration common data\n  downloads: 0\n  homepage_uri: ''\n  name: liblaunchpad-integration-common\n  section: libs\n  version: 0.1.56\nliblcms1:\n  description: Little CMS color management library\n  downloads: 48150\n  homepage_uri: ''\n  name: liblcms1\n  section: libs\n  version: 1.19.dfsg-1ubuntu3\nliblcms1:i386:\n  description: Little CMS color management library\n  downloads: 0\n  homepage_uri: ''\n  name: liblcms1:i386\n  section: libs\n  version: 1.19.dfsg-1ubuntu3\nliblcms2-2:\n  description: Little CMS 2 color management library\n  downloads: 15581\n  homepage_uri: http://www.littlecms.com/\n  name: liblcms2-2\n  section: libs\n  version: 2.2+git20110628-2ubuntu3\nlibldap-2.4-2:\n  description: OpenLDAP libraries\n  downloads: 82688\n  homepage_uri: http://www.openldap.org/\n  name: libldap-2.4-2\n  section: libs\n  version: 2.4.28-1.1ubuntu4\nlibldap-2.4-2:i386:\n  description: OpenLDAP libraries\n  downloads: 0\n  homepage_uri: http://www.openldap.org/\n  name: libldap-2.4-2:i386\n  section: libs\n  version: 2.4.28-1.1ubuntu4\nlibldap2-dev:\n  description: OpenLDAP development libraries\n  downloads: 2501\n  homepage_uri: http://www.openldap.org/\n  name: libldap2-dev\n  section: libdevel\n  version: 2.4.28-1.1ubuntu4\nliblightdm-gobject-1-0:\n  description: LightDM GObject client library\n  downloads: 1061\n  homepage_uri: https://launchpad.net/lightdm\n  name: liblightdm-gobject-1-0\n  section: x11\n  version: 1.2.1-0ubuntu1\nliblircclient0:\n  description: infra-red remote control support - client library\n  downloads: 26561\n  homepage_uri: http://lirc.org\n  name: liblircclient0\n  section: libs\n  version: 0.9.0-0ubuntu1\nlibllvm3.0:\n  description: Low-Level Virtual Machine (LLVM), runtime library\n  downloads: 35\n  homepage_uri: http://www.llvm.org/\n  name: libllvm3.0\n  section: libs\n  version: 3.0-4ubuntu1\nlibllvm3.0:i386:\n  description: Low-Level Virtual Machine (LLVM), runtime library\n  downloads: 0\n  homepage_uri: http://www.llvm.org/\n  name: libllvm3.0:i386\n  section: libs\n  version: 3.0-4ubuntu1\nliblocale-gettext-perl:\n  description: module using libc functions for internationalization in Perl\n  downloads: 84667\n  homepage_uri: http://search.cpan.org/dist/gettext/gettext.pm\n  name: liblocale-gettext-perl\n  section: perl\n  version: 1.05-7build1\nliblockfile-bin:\n  description: support binaries for and cli utilities based on liblockfile\n  downloads: 10314\n  homepage_uri: ''\n  name: liblockfile-bin\n  section: utils\n  version: 1.09-3\nliblockfile1:\n  description: NFS-safe locking library\n  downloads: 59233\n  homepage_uri: ''\n  name: liblockfile1\n  section: libs\n  version: 1.09-3\nliblog4j1.2-java:\n  description: Logging library for java\n  downloads: 1166\n  homepage_uri: http://logging.apache.org/log4j/1.2/\n  name: liblog4j1.2-java\n  section: libs\n  version: 1.2.16-3ubuntu1\nliblouis-data:\n  description: Braille translation library - data\n  downloads: 1065\n  homepage_uri: http://code.google.com/p/liblouis/\n  name: liblouis-data\n  section: text\n  version: 2.3.0-3\nliblouis2:\n  description: Braille translation library - shared libs\n  downloads: 1323\n  homepage_uri: http://code.google.com/p/liblouis/\n  name: liblouis2\n  section: libs\n  version: 2.3.0-3\nliblqr-1-0:\n  description: converts plain array images into multi-size representation\n  downloads: 110\n  homepage_uri: http://liblqr.wikidot.com/\n  name: liblqr-1-0\n  section: libs\n  version: 0.4.1-1.1\nlibltdl7:\n  description: A system independent dlopen wrapper for GNU libtool\n  downloads: 56813\n  homepage_uri: http://www.gnu.org/software/libtool/\n  name: libltdl7\n  section: libs\n  version: 2.4.2-1ubuntu1\nlibltdl7:i386:\n  description: A system independent dlopen wrapper for GNU libtool\n  downloads: 0\n  homepage_uri: http://www.gnu.org/software/libtool/\n  name: libltdl7:i386\n  section: libs\n  version: 2.4.2-1ubuntu1\nliblua5.1-0:\n  description: Shared library for the Lua interpreter version 5.1\n  downloads: 7810\n  homepage_uri: http://www.lua.org\n  name: liblua5.1-0\n  section: libs\n  version: 5.1.4-12ubuntu1\nliblucene2-java:\n  description: Full-text search engine library for Java(TM)\n  downloads: 5270\n  homepage_uri: http://lucene.apache.org/java/\n  name: liblucene2-java\n  section: universe/text\n  version: 2.9.4+ds1-4\nliblvm2app2.2:\n  description: LVM2 application library\n  downloads: 28\n  homepage_uri: http://sources.redhat.com/lvm2/\n  name: liblvm2app2.2\n  section: libs\n  version: 2.02.66-4ubuntu7.1\nliblwp-mediatypes-perl:\n  description: module to guess media type for a file or a URL\n  downloads: 5298\n  homepage_uri: http://search.cpan.org/dist/LWP-MediaTypes/\n  name: liblwp-mediatypes-perl\n  section: perl\n  version: 6.01-1\nliblwp-protocol-https-perl:\n  description: https driver for LWP::UserAgent\n  downloads: 5216\n  homepage_uri: http://search.cpan.org/dist/LWP-Protocol-https/\n  name: liblwp-protocol-https-perl\n  section: perl\n  version: 6.02-1\nliblwres80:\n  description: Lightweight Resolver Library used by BIND\n  downloads: 1\n  homepage_uri: ''\n  name: liblwres80\n  section: libs\n  version: 1:9.8.1.dfsg.P1-4ubuntu0.1\nliblzma-dev:\n  description: XZ-format compression library - development files\n  downloads: 103\n  homepage_uri: http://tukaani.org/xz/\n  name: liblzma-dev\n  section: libdevel\n  version: 5.1.1alpha+20110809-3\nliblzma5:\n  description: XZ-format compression library\n  downloads: 18267\n  homepage_uri: http://tukaani.org/xz/\n  name: liblzma5\n  section: libs\n  version: 5.1.1alpha+20110809-3\nlibm17n-0:\n  description: multilingual text processing library - runtime\n  downloads: 1253\n  homepage_uri: http://www.m17n.org/\n  name: libm17n-0\n  section: libs\n  version: 1.6.3-1\nlibmaa3:\n  description: maa programming library\n  downloads: 779\n  homepage_uri: http://sourceforge.net/projects/dict/\n  name: libmaa3\n  section: libs\n  version: 1.3.1-1\nlibmad0:i386:\n  description: MPEG audio decoder library\n  downloads: 0\n  homepage_uri: ''\n  name: libmad0:i386\n  section: libs\n  version: 0.15.1b-7ubuntu1\nlibmagic1:\n  description: File type determination library using \"magic\" numbers\n  downloads: 54754\n  homepage_uri: http://www.darwinsys.com/file/\n  name: libmagic1\n  section: libs\n  version: 5.09-2\nlibmagickcore4:\n  description: low-level image manipulation library\n  downloads: 318\n  homepage_uri: http://www.imagemagick.org/\n  name: libmagickcore4\n  section: libs\n  version: 8:6.6.9.7-5ubuntu3.1\nlibmagickcore4-extra:\n  description: low-level image manipulation library - extra codecs\n  downloads: 37\n  homepage_uri: http://www.imagemagick.org/\n  name: libmagickcore4-extra\n  section: libs\n  version: 8:6.6.9.7-5ubuntu3.1\nlibmagickwand4:\n  description: image manipulation library\n  downloads: 0\n  homepage_uri: http://www.imagemagick.org/\n  name: libmagickwand4\n  section: libs\n  version: 8:6.6.9.7-5ubuntu3.1\nlibmail-sendmail-perl:\n  description: Send email from a perl script\n  downloads: 1915\n  homepage_uri: http://search.cpan.org/dist/Mail-Sendmail/\n  name: libmail-sendmail-perl\n  section: perl\n  version: 0.79.16-1\nlibmailtools-perl:\n  description: Manipulate email in perl programs\n  downloads: 7806\n  homepage_uri: http://search.cpan.org/dist/MailTools/\n  name: libmailtools-perl\n  section: perl\n  version: 2.08-1\nlibmaven-archiver-java:\n  description: Archiver component for Maven\n  downloads: 1\n  homepage_uri: http://maven.apache.org/shared/maven-archiver/\n  name: libmaven-archiver-java\n  section: universe/devel\n  version: 2.4-4\nlibmaven-clean-plugin-java:\n  description: Maven clean plugin\n  downloads: 10\n  homepage_uri: http://maven.apache.org/plugins/maven-clean-plugin/\n  name: libmaven-clean-plugin-java\n  section: universe/java\n  version: 2.3-5\nlibmaven-compiler-plugin-java:\n  description: Maven Compiler plugin\n  downloads: 10\n  homepage_uri: http://maven.apache.org/plugins/maven-compiler-plugin/\n  name: libmaven-compiler-plugin-java\n  section: universe/java\n  version: 2.0.2-6\nlibmaven-dependency-tree-java:\n  description: Maven Dependency Tree\n  downloads: 0\n  homepage_uri: http://maven.apache.org\n  name: libmaven-dependency-tree-java\n  section: universe/java\n  version: 1.2-4\nlibmaven-file-management-java:\n  description: Maven File Management API\n  downloads: 0\n  homepage_uri: http://maven.apache.org/shared/file-management/\n  name: libmaven-file-management-java\n  section: universe/devel\n  version: 1.2.1-3\nlibmaven-filtering-java:\n  description: Maven Filtering\n  downloads: 0\n  homepage_uri: http://maven.apache.org/shared/maven-filtering/\n  name: libmaven-filtering-java\n  section: universe/devel\n  version: 1.0~beta-2-5\nlibmaven-install-plugin-java:\n  description: Maven install plugin\n  downloads: 0\n  homepage_uri: http://maven.apache.org/plugins/maven-install-plugin/\n  name: libmaven-install-plugin-java\n  section: universe/java\n  version: 2.3-4\nlibmaven-jar-plugin-java:\n  description: Maven Jar Plugin\n  downloads: 10\n  homepage_uri: http://maven.apache.org/plugins/maven-jar-plugin/\n  name: libmaven-jar-plugin-java\n  section: universe/java\n  version: 2.2-6\nlibmaven-resources-plugin-java:\n  description: Maven resources plugin\n  downloads: 10\n  homepage_uri: http://maven.apache.org/plugins/maven-resources-plugin/\n  name: libmaven-resources-plugin-java\n  section: universe/java\n  version: 2.3-7\nlibmaven-scm-java:\n  description: Maven SCM provides a common API for doing SCM operations\n  downloads: 10\n  homepage_uri: http://maven.apache.org/scm/\n  name: libmaven-scm-java\n  section: universe/java\n  version: 1.3-4\nlibmaven-shade-plugin-java:\n  description: Maven shade plugin\n  downloads: 0\n  homepage_uri: http://maven.apache.org/plugins/maven-shade-plugin/\n  name: libmaven-shade-plugin-java\n  section: universe/java\n  version: 1.2.1-4\nlibmaven-shared-io-java:\n  description: Maven API for I/O support\n  downloads: 0\n  homepage_uri: http://maven.apache.org/shared/maven-shared-io/\n  name: libmaven-shared-io-java\n  section: universe/devel\n  version: 1.1-4\nlibmaven2-core-java:\n  description: Core libraries for Maven2\n  downloads: 363\n  homepage_uri: http://maven.apache.org/\n  name: libmaven2-core-java\n  section: universe/java\n  version: 2.2.1-8\nlibmeanwhile1:\n  description: open implementation of the Lotus Sametime Community Client protocol\n  downloads: 5641\n  homepage_uri: http://meanwhile.sourceforge.net/\n  name: libmeanwhile1\n  section: libs\n  version: 1.0.2-4ubuntu1\nlibmetacity-private0:\n  description: library for the Metacity window manager\n  downloads: 20\n  homepage_uri: ''\n  name: libmetacity-private0\n  section: libs\n  version: 1:2.34.1-1ubuntu11\nlibmhash2:\n  description: Library for cryptographic hashing and message authentication\n  downloads: 768\n  homepage_uri: ''\n  name: libmhash2\n  section: libs\n  version: 0.9.9.9-1\nlibmikmod2:i386:\n  description: Portable sound library\n  downloads: 0\n  homepage_uri: http://mikmod.raphnet.net/\n  name: libmikmod2:i386\n  section: universe/libs\n  version: 3.1.12-2\nlibmime-tools-perl:\n  description: Perl5 modules for MIME-compliant messages\n  downloads: 5595\n  homepage_uri: http://search.cpan.org/dist/MIME-tools/\n  name: libmime-tools-perl\n  section: perl\n  version: 5.502-1\nlibminiupnpc8:\n  description: UPnP IGD client lightweight library\n  downloads: 0\n  homepage_uri: http://miniupnp.free.fr/\n  name: libminiupnpc8\n  section: net\n  version: 1.6-3ubuntu1\nlibmission-control-plugins0:\n  description: management daemon for Telepathy (library for plugins)\n  downloads: 7784\n  homepage_uri: http://telepathy.freedesktop.org/wiki/\n  name: libmission-control-plugins0\n  section: libs\n  version: 1:5.12.0-0ubuntu2\nlibmng1:\n  description: Multiple-image Network Graphics library\n  downloads: 19461\n  homepage_uri: http://www.libmng.com\n  name: libmng1\n  section: libs\n  version: 1.0.10-3\nlibmng1:i386:\n  description: Multiple-image Network Graphics library\n  downloads: 0\n  homepage_uri: http://www.libmng.com\n  name: libmng1:i386\n  section: libs\n  version: 1.0.10-3\nlibmodello-java:\n  description: a Data Model toolkit in use by the Maven 2 Project\n  downloads: 93\n  homepage_uri: http://modello.codehaus.org\n  name: libmodello-java\n  section: universe/libs\n  version: 1.1-2\nlibmount1:\n  description: block device id library\n  downloads: 3160\n  homepage_uri: http://userweb.kernel.org/~kzak/util-linux/\n  name: libmount1\n  section: libs\n  version: 2.20.1-1ubuntu3\nlibmozjs185-1.0:\n  description: Spidermonkey javascript engine\n  downloads: 7970\n  homepage_uri: http://www.mozilla.org/js/spidermonkey/\n  name: libmozjs185-1.0\n  section: universe/libs\n  version: 1.8.5-1.0.0-0ubuntu8\nlibmpc2:\n  description: multiple precision complex floating-point library\n  downloads: 16388\n  homepage_uri: http://www.multiprecision.org/mpc/\n  name: libmpc2\n  section: libs\n  version: 0.9-4\nlibmpfr4:\n  description: multiple precision floating-point computation\n  downloads: 64013\n  homepage_uri: http://www.mpfr.org/\n  name: libmpfr4\n  section: libs\n  version: 3.1.0-3ubuntu2\nlibmpg123-0:i386:\n  description: MPEG layer 1/2/3 audio decoder -- runtime library\n  downloads: 0\n  homepage_uri: ''\n  name: libmpg123-0:i386\n  section: universe/libs\n  version: 1.12.1-3.2ubuntu1\nlibmtdev1:\n  description: Multitouch Protocol Translation Library - shared library\n  downloads: 5764\n  homepage_uri: http://bitmath.org/code/mtdev/\n  name: libmtdev1\n  section: libs\n  version: 1.1.0-2ubuntu1\nlibmtp-common:\n  description: Media Transfer Protocol (MTP) common files\n  downloads: 12505\n  homepage_uri: http://libmtp.sourceforge.net/\n  name: libmtp-common\n  section: libs\n  version: 1.1.3-1\nlibmtp-runtime:\n  description: Media Transfer Protocol (MTP) runtime tools\n  downloads: 10087\n  homepage_uri: http://libmtp.sourceforge.net/\n  name: libmtp-runtime\n  section: libs\n  version: 1.1.3-1\nlibmtp9:\n  description: Media Transfer Protocol (MTP) library\n  downloads: 10955\n  homepage_uri: http://libmtp.sourceforge.net/\n  name: libmtp9\n  section: libs\n  version: 1.1.3-1\nlibmusicbrainz3-6:\n  description: library to access the MusicBrainz.org database\n  downloads: 2658\n  homepage_uri: ''\n  name: libmusicbrainz3-6\n  section: libs\n  version: 3.0.2-2.1\nlibmysqlclient-dev:\n  description: MySQL database development files\n  downloads: 1256\n  homepage_uri: http://dev.mysql.com/\n  name: libmysqlclient-dev\n  section: libdevel\n  version: 5.5.24-0ubuntu0.12.04.1\nlibmysqlclient18:\n  description: MySQL database client library\n  downloads: 3752\n  homepage_uri: http://dev.mysql.com/\n  name: libmysqlclient18\n  section: libs\n  version: 5.5.24-0ubuntu0.12.04.1\nlibmysqlclient18:i386:\n  description: MySQL database client library\n  downloads: 0\n  homepage_uri: http://dev.mysql.com/\n  name: libmysqlclient18:i386\n  section: libs\n  version: 5.5.24-0ubuntu0.12.04.1\nlibmythes-1.2-0:\n  description: simple thesaurus library\n  downloads: 9\n  homepage_uri: ''\n  name: libmythes-1.2-0\n  section: libs\n  version: 2:1.2.2-1\nlibnautilus-extension1a:\n  description: libraries for nautilus components - runtime version\n  downloads: 9552\n  homepage_uri: http://www.gnome.org/projects/nautilus/\n  name: libnautilus-extension1a\n  section: libs\n  version: 1:3.4.2-0ubuntu2\nlibncbi6:\n  description: NCBI libraries for biology applications\n  downloads: 162\n  homepage_uri: http://www.ncbi.nlm.nih.gov/IEB/ToolBox/\n  name: libncbi6\n  section: libs\n  version: 6.1.20110713-3ubuntu2\nlibncurses5:\n  description: shared libraries for terminal handling\n  downloads: 107369\n  homepage_uri: http://invisible-island.net/ncurses/\n  name: libncurses5\n  section: libs\n  version: 5.9-4\nlibncurses5-dev:\n  description: developer's libraries for ncurses\n  downloads: 3263\n  homepage_uri: http://invisible-island.net/ncurses/\n  name: libncurses5-dev\n  section: libdevel\n  version: 5.9-4\nlibncurses5:i386:\n  description: shared libraries for terminal handling\n  downloads: 0\n  homepage_uri: http://invisible-island.net/ncurses/\n  name: libncurses5:i386\n  section: libs\n  version: 5.9-4\nlibncursesw5:\n  description: shared libraries for terminal handling (wide character support)\n  downloads: 103990\n  homepage_uri: http://invisible-island.net/ncurses/\n  name: libncursesw5\n  section: libs\n  version: 5.9-4\nlibncursesw5:i386:\n  description: shared libraries for terminal handling (wide character support)\n  downloads: 0\n  homepage_uri: http://invisible-island.net/ncurses/\n  name: libncursesw5:i386\n  section: libs\n  version: 5.9-4\nlibneon27-gnutls:\n  description: HTTP and WebDAV client library (GnuTLS enabled)\n  downloads: 2323\n  homepage_uri: http://www.webdav.org/neon/\n  name: libneon27-gnutls\n  section: libs\n  version: 0.29.6-1\nlibnet-daemon-perl:\n  description: Perl module for building portable Perl daemons easily\n  downloads: 3747\n  homepage_uri: http://search.cpan.org/dist/Net-Daemon/\n  name: libnet-daemon-perl\n  section: perl\n  version: 0.48-1\nlibnet-dbus-perl:\n  description: Extension for the DBus bindings\n  downloads: 9344\n  homepage_uri: http://search.cpan.org/dist/Net-DBus/\n  name: libnet-dbus-perl\n  section: universe/perl\n  version: 1.0.0-1build1\nlibnet-dns-perl:\n  description: Perform DNS queries from a Perl script\n  downloads: 9096\n  homepage_uri: http://www.net-dns.org/\n  name: libnet-dns-perl\n  section: perl\n  version: 0.66-2ubuntu3\nlibnet-domain-tld-perl:\n  description: Perl module for retrieving a list of currently available TLDs\n  downloads: 1063\n  homepage_uri: http://search.cpan.org/dist/Net-Domain-TLD/\n  name: libnet-domain-tld-perl\n  section: perl\n  version: 1.69-1\nlibnet-http-perl:\n  description: module providing low-level HTTP connection client\n  downloads: 5071\n  homepage_uri: http://search.cpan.org/dist/Net-HTTP/\n  name: libnet-http-perl\n  section: perl\n  version: 6.02-1\nlibnet-ip-perl:\n  description: Perl extension for manipulating IPv4/IPv6 addresses\n  downloads: 10893\n  homepage_uri: http://search.cpan.org/dist/Net-IP/\n  name: libnet-ip-perl\n  section: perl\n  version: 1.25-3\nlibnet-ssleay-perl:\n  description: Perl module for Secure Sockets Layer (SSL)\n  downloads: 9569\n  homepage_uri: http://search.cpan.org/dist/Net-SSLeay/\n  name: libnet-ssleay-perl\n  section: perl\n  version: 1.42-1build1\nlibnetbeans-cvsclient-java:\n  description: NetBeans CVS Client library\n  downloads: 76\n  homepage_uri: http://javacvs.netbeans.org/library/\n  name: libnetbeans-cvsclient-java\n  section: universe/java\n  version: 6.5-2\nlibnetcdf6:\n  description: An interface for scientific data access to large binary data\n  downloads: 0\n  homepage_uri: http://www.unidata.ucar.edu/software/netcdf/\n  name: libnetcdf6\n  section: universe/libs\n  version: 1:4.1.1-6\nlibnetfilter-conntrack3:\n  description: Netfilter netlink-conntrack library\n  downloads: 1549\n  homepage_uri: http://www.netfilter.org/projects/libnetfilter_conntrack/\n  name: libnetfilter-conntrack3\n  section: libs\n  version: 0.9.1-1ubuntu1\nlibnetpbm10:\n  description: Graphics conversion tools shared libraries\n  downloads: 5717\n  homepage_uri: http://netpbm.alioth.debian.org\n  name: libnetpbm10\n  section: libs\n  version: 2:10.0-15\nlibnettle4:\n  description: low level cryptographic library (symmetric and one-way cryptos)\n  downloads: 2272\n  homepage_uri: http://www.lysator.liu.se/~nisse/nettle/\n  name: libnettle4\n  section: libs\n  version: 2.4-1\nlibnewt0.52:\n  description: Not Erik's Windowing Toolkit - text mode windowing with slang\n  downloads: 26597\n  homepage_uri: https://fedorahosted.org/newt/\n  name: libnewt0.52\n  section: libs\n  version: 0.52.11-2ubuntu10\nlibnfnetlink0:\n  description: Netfilter netlink library\n  downloads: 25624\n  homepage_uri: ''\n  name: libnfnetlink0\n  section: libs\n  version: 1.0.0-1\nlibnfsidmap2:\n  description: NFS idmapping library\n  downloads: 9998\n  homepage_uri: http://www.citi.umich.edu/projects/nfsv4/linux/\n  name: libnfsidmap2\n  section: libs\n  version: 0.25-1ubuntu2\nlibnice10:\n  description: ICE library (shared library)\n  downloads: 3184\n  homepage_uri: http://nice.freedesktop.org/\n  name: libnice10\n  section: libs\n  version: 0.1.1-2ubuntu1\nlibnih-dbus1:\n  description: NIH D-Bus Bindings Library\n  downloads: 131\n  homepage_uri: https://launchpad.net/libnih\n  name: libnih-dbus1\n  section: libs\n  version: 1.0.3-4ubuntu9\nlibnih1:\n  description: NIH Utility Library\n  downloads: 132\n  homepage_uri: https://launchpad.net/libnih\n  name: libnih1\n  section: libs\n  version: 1.0.3-4ubuntu9\nlibnl-3-200:\n  description: library for dealing with netlink sockets\n  downloads: 15141\n  homepage_uri: http://people.suug.ch/~tgr/libnl/\n  name: libnl-3-200\n  section: libs\n  version: 3.2.3-2ubuntu2\nlibnl-genl-3-200:\n  description: library for dealing with netlink sockets - generic netlink\n  downloads: 14658\n  homepage_uri: http://people.suug.ch/~tgr/libnl/\n  name: libnl-genl-3-200\n  section: libs\n  version: 3.2.3-2ubuntu2\nlibnl-route-3-200:\n  description: library for dealing with netlink sockets - route interface\n  downloads: 12982\n  homepage_uri: http://people.suug.ch/~tgr/libnl/\n  name: libnl-route-3-200\n  section: libs\n  version: 3.2.3-2ubuntu2\nlibnm-glib-vpn1:\n  description: network management framework (GLib VPN shared library)\n  downloads: 26545\n  homepage_uri: http://www.gnome.org/projects/NetworkManager/\n  name: libnm-glib-vpn1\n  section: libs\n  version: 0.9.4.0-0ubuntu4.1\nlibnm-glib4:\n  description: network management framework (GLib shared library)\n  downloads: 13402\n  homepage_uri: http://www.gnome.org/projects/NetworkManager/\n  name: libnm-glib4\n  section: libs\n  version: 0.9.4.0-0ubuntu4.1\nlibnm-gtk-common:\n  description: network management framework (common files for wifi and mobile)\n  downloads: 7534\n  homepage_uri: http://www.gnome.org/projects/NetworkManager/\n  name: libnm-gtk-common\n  section: gnome\n  version: 0.9.4.1-0ubuntu2\nlibnm-gtk0:\n  description: network management framework (GNOME dialogs for wifi and mobile)\n  downloads: 9228\n  homepage_uri: http://www.gnome.org/projects/NetworkManager/\n  name: libnm-gtk0\n  section: libs\n  version: 0.9.4.1-0ubuntu2\nlibnm-util2:\n  description: network management framework (shared library)\n  downloads: 13493\n  homepage_uri: http://www.gnome.org/projects/NetworkManager/\n  name: libnm-util2\n  section: libs\n  version: 0.9.4.0-0ubuntu4.1\nlibnotify-bin:\n  description: sends desktop notifications to a notification daemon (Utilities)\n  downloads: 1371\n  homepage_uri: ''\n  name: libnotify-bin\n  section: utils\n  version: 0.7.5-1\nlibnotify4:\n  description: sends desktop notifications to a notification daemon\n  downloads: 15455\n  homepage_uri: ''\n  name: libnotify4\n  section: libs\n  version: 0.7.5-1\nlibnspr4:\n  description: NetScape Portable Runtime Library\n  downloads: 12211\n  homepage_uri: http://www.mozilla.org/projects/nspr/\n  name: libnspr4\n  section: libs\n  version: 4.8.9-1ubuntu2\nlibnspr4:i386:\n  description: NetScape Portable Runtime Library\n  downloads: 0\n  homepage_uri: http://www.mozilla.org/projects/nspr/\n  name: libnspr4:i386\n  section: libs\n  version: 4.8.9-1ubuntu2\nlibnss-mdns:\n  description: NSS module for Multicast DNS name resolution\n  downloads: 259\n  homepage_uri: ''\n  name: libnss-mdns\n  section: admin\n  version: 0.10-3.2\nlibnss3:\n  description: Network Security Service libraries\n  downloads: 4646\n  homepage_uri: http://www.mozilla.org/projects/security/pki/nss/\n  name: libnss3\n  section: libs\n  version: 3.13.1.with.ckbi.1.88-1ubuntu6\nlibnss3-1d:\n  description: Network Security Service libraries\n  downloads: 31079\n  homepage_uri: http://www.mozilla.org/projects/security/pki/nss/\n  name: libnss3-1d\n  section: libs\n  version: 3.13.1.with.ckbi.1.88-1ubuntu6\nlibnss3:i386:\n  description: Network Security Service libraries\n  downloads: 0\n  homepage_uri: http://www.mozilla.org/projects/security/pki/nss/\n  name: libnss3:i386\n  section: libs\n  version: 3.13.1.with.ckbi.1.88-1ubuntu6\nlibnuma1:\n  description: Libraries for controlling NUMA policy\n  downloads: 120\n  homepage_uri: ''\n  name: libnuma1\n  section: libs\n  version: 2.0.8~rc3-1\nlibnux-2.0-0:\n  description: Visual rendering toolkit for real-time applications - shared lib\n  downloads: 0\n  homepage_uri: http://launchpad.net/nux\n  name: libnux-2.0-0\n  section: libs\n  version: 2.12.0-0ubuntu1\nlibnux-2.0-common:\n  description: Visual rendering toolkit for real-time applications - common files\n  downloads: 0\n  homepage_uri: http://launchpad.net/nux\n  name: libnux-2.0-common\n  section: libs\n  version: 2.12.0-0ubuntu1\nlibnx-xorg:\n  description: NX support libraries\n  downloads: 0\n  homepage_uri: ''\n  name: libnx-xorg\n  section: x11\n  version: 1:3.5.0-9-2-1-2-0~ppa1~precise2\nliboauth0:\n  description: C library for implementing OAuth 1.0\n  downloads: 3467\n  homepage_uri: http://liboauth.sourceforge.net/\n  name: liboauth0\n  section: libs\n  version: 0.9.4-3\nlibodbc1:\n  description: ODBC library for Unix\n  downloads: 4181\n  homepage_uri: http://www.unixodbc.org/\n  name: libodbc1\n  section: libs\n  version: 2.2.14p2-5ubuntu3\nlibodbc1:i386:\n  description: ODBC library for Unix\n  downloads: 0\n  homepage_uri: http://www.unixodbc.org/\n  name: libodbc1:i386\n  section: libs\n  version: 2.2.14p2-5ubuntu3\nlibogg0:\n  description: Ogg bitstream library\n  downloads: 44811\n  homepage_uri: http://xiph.org/ogg/\n  name: libogg0\n  section: libs\n  version: 1.2.2~dfsg-1ubuntu1\nlibogg0:i386:\n  description: Ogg bitstream library\n  downloads: 0\n  homepage_uri: http://xiph.org/ogg/\n  name: libogg0:i386\n  section: libs\n  version: 1.2.2~dfsg-1ubuntu1\nliboobs-1-5:\n  description: GObject based interface to system-tools-backends - shared library\n  downloads: 964\n  homepage_uri: ''\n  name: liboobs-1-5\n  section: universe/libs\n  version: 3.0.0-1\nlibopenal-data:\n  description: Software implementation of the OpenAL API (data files)\n  downloads: 719\n  homepage_uri: http://kcat.strangesoft.net/openal.html\n  name: libopenal-data\n  section: libs\n  version: 1:1.13-4ubuntu3\nlibopenal1:i386:\n  description: Software implementation of the OpenAL API (shared library)\n  downloads: 0\n  homepage_uri: http://kcat.strangesoft.net/openal.html\n  name: libopenal1:i386\n  section: libs\n  version: 1:1.13-4ubuntu3\nlibopencc1:\n  description: simplified-traditional chinese conversion library - runtime\n  downloads: 30\n  homepage_uri: http://code.google.com/p/opencc\n  name: libopencc1\n  section: libs\n  version: 0.3.0-1\nlibopenexr6:\n  description: runtime files for the OpenEXR image library\n  downloads: 256\n  homepage_uri: http://www.openexr.com\n  name: libopenexr6\n  section: libs\n  version: 1.6.1-4.1\nlibopenmpi-dev:\n  description: high performance message passing library -- header files\n  downloads: 1188\n  homepage_uri: http://www.open-mpi.org/\n  name: libopenmpi-dev\n  section: universe/libdevel\n  version: 1.4.3-2.1ubuntu3\nlibopenmpi1.3:\n  description: high performance message passing library -- shared library\n  downloads: 1715\n  homepage_uri: http://www.open-mpi.org/\n  name: libopenmpi1.3\n  section: universe/libs\n  version: 1.4.3-2.1ubuntu3\nlibopenobex1:\n  description: OBEX protocol library\n  downloads: 260\n  homepage_uri: http://www.openobex.org\n  name: libopenobex1\n  section: libs\n  version: 1.5-2build1\nliborbit2:\n  description: libraries for ORBit2 - a CORBA ORB\n  downloads: 33873\n  homepage_uri: ''\n  name: liborbit2\n  section: libs\n  version: 1:2.14.19-0.1ubuntu1\nliborbit2-dev:\n  description: development files for ORBit2 - a CORBA ORB\n  downloads: 543\n  homepage_uri: ''\n  name: liborbit2-dev\n  section: libdevel\n  version: 1:2.14.19-0.1ubuntu1\nliborc-0.4-0:\n  description: Library of Optimized Inner Loops Runtime Compiler\n  downloads: 37029\n  homepage_uri: http://code.entropywave.com/projects/orc/\n  name: liborc-0.4-0\n  section: libs\n  version: 1:0.4.16-1ubuntu2\nliborc-0.4-0:i386:\n  description: Library of Optimized Inner Loops Runtime Compiler\n  downloads: 0\n  homepage_uri: http://code.entropywave.com/projects/orc/\n  name: liborc-0.4-0:i386\n  section: libs\n  version: 1:0.4.16-1ubuntu2\nlibossp-uuid-perl:\n  description: perl OSSP::UUID - OSSP uuid Perl Binding\n  downloads: 1796\n  homepage_uri: http://www.ossp.org/pkg/lib/uuid/\n  name: libossp-uuid-perl\n  section: universe/libs\n  version: 1.6.2-1.1build3\nlibossp-uuid16:\n  description: OSSP uuid ISO-C and C++ - shared library\n  downloads: 1299\n  homepage_uri: http://www.ossp.org/pkg/lib/uuid/\n  name: libossp-uuid16\n  section: libs\n  version: 1.6.2-1.1build3\nlibotf0:\n  description: A Library for handling OpenType Font - runtime\n  downloads: 66\n  homepage_uri: http://www.m17n.org/\n  name: libotf0\n  section: libs\n  version: 0.9.12-1\nliboverlay-scrollbar-0.2-0:\n  description: Scrollbar overlayed widget - shared lib\n  downloads: 0\n  homepage_uri: http://launchpad.net/ayatana-scrollbar\n  name: liboverlay-scrollbar-0.2-0\n  section: libs\n  version: 0.2.16-0ubuntu1\nliboverlay-scrollbar3-0.2-0:\n  description: Scrollbar overlayed widget - shared lib\n  downloads: 0\n  homepage_uri: http://launchpad.net/ayatana-scrollbar\n  name: liboverlay-scrollbar3-0.2-0\n  section: libs\n  version: 0.2.16-0ubuntu1\nlibp11-kit-dev:\n  description: Library for loading and coordinating access to PKCS#11 modules - development\n  downloads: 1565\n  homepage_uri: http://p11-glue.freedesktop.org/p11-kit.html\n  name: libp11-kit-dev\n  section: libdevel\n  version: 0.12-2ubuntu1\nlibp11-kit0:\n  description: Library for loading and coordinating access to PKCS#11 modules - runtime\n  downloads: 19102\n  homepage_uri: http://p11-glue.freedesktop.org/p11-kit.html\n  name: libp11-kit0\n  section: libs\n  version: 0.12-2ubuntu1\nlibp11-kit0:i386:\n  description: Library for loading and coordinating access to PKCS#11 modules - runtime\n  downloads: 0\n  homepage_uri: http://p11-glue.freedesktop.org/p11-kit.html\n  name: libp11-kit0:i386\n  section: libs\n  version: 0.12-2ubuntu1\nlibpackagekit-glib2-14:\n  description: Library for accessing PackageKit using GLib\n  downloads: 10162\n  homepage_uri: http://www.packagekit.org\n  name: libpackagekit-glib2-14\n  section: libs\n  version: 0.7.2-4ubuntu3\nlibpam-cap:\n  description: PAM module for implementing capabilities\n  downloads: 59\n  homepage_uri: http://sites.google.com/site/fullycapable/\n  name: libpam-cap\n  section: libs\n  version: 1:2.22-1ubuntu3\nlibpam-ck-connector:\n  description: ConsoleKit PAM module\n  downloads: 35908\n  homepage_uri: http://www.freedesktop.org/wiki/Software/ConsoleKit\n  name: libpam-ck-connector\n  section: admin\n  version: 0.4.5-2\nlibpam-gnome-keyring:\n  description: PAM module to unlock the GNOME keyring upon login\n  downloads: 46702\n  homepage_uri: http://live.gnome.org/GnomeKeyring\n  name: libpam-gnome-keyring\n  section: admin\n  version: 3.2.2-2ubuntu4\nlibpam-modules:\n  description: Pluggable Authentication Modules for PAM\n  downloads: 108815\n  homepage_uri: http://pam.sourceforge.net/\n  name: libpam-modules\n  section: admin\n  version: 1.1.3-7ubuntu2\nlibpam-modules-bin:\n  description: Pluggable Authentication Modules for PAM - helper binaries\n  downloads: 10192\n  homepage_uri: http://pam.sourceforge.net/\n  name: libpam-modules-bin\n  section: admin\n  version: 1.1.3-7ubuntu2\nlibpam-runtime:\n  description: Runtime support for the PAM library\n  downloads: 86700\n  homepage_uri: http://pam.sourceforge.net/\n  name: libpam-runtime\n  section: admin\n  version: 1.1.3-7ubuntu2\nlibpam0g:\n  description: Pluggable Authentication Modules library\n  downloads: 107338\n  homepage_uri: http://pam.sourceforge.net/\n  name: libpam0g\n  section: libs\n  version: 1.1.3-7ubuntu2\nlibpanel-applet-4-0:\n  description: library for GNOME Panel applets\n  downloads: 4761\n  homepage_uri: ''\n  name: libpanel-applet-4-0\n  section: universe/libs\n  version: 1:3.4.1-0ubuntu1\nlibpango-perl:\n  description: Perl module to layout and render international text\n  downloads: 17349\n  homepage_uri: http://gtk2-perl.sourceforge.net/\n  name: libpango-perl\n  section: perl\n  version: 1.222-1build1\nlibpango1.0-0:\n  description: Layout and rendering of internationalized text\n  downloads: 58689\n  homepage_uri: ''\n  name: libpango1.0-0\n  section: libs\n  version: 1.30.0-0ubuntu3\nlibpango1.0-0:i386:\n  description: Layout and rendering of internationalized text\n  downloads: 0\n  homepage_uri: ''\n  name: libpango1.0-0:i386\n  section: libs\n  version: 1.30.0-0ubuntu3\nlibpango1.0-dev:\n  description: Development files for the Pango\n  downloads: 3243\n  homepage_uri: ''\n  name: libpango1.0-dev\n  section: libdevel\n  version: 1.30.0-0ubuntu3\nlibpangomm-1.4-1:\n  description: C++ Wrapper for pango (shared libraries)\n  downloads: 219\n  homepage_uri: http://gtkmm.org\n  name: libpangomm-1.4-1\n  section: libs\n  version: 2.28.4-1ubuntu1\nlibpaper-utils:\n  description: library for handling paper characteristics (utilities)\n  downloads: 27405\n  homepage_uri: ''\n  name: libpaper-utils\n  section: utils\n  version: 1.1.24+nmu1build1\nlibpaper1:\n  description: library for handling paper characteristics\n  downloads: 28396\n  homepage_uri: ''\n  name: libpaper1\n  section: libs\n  version: 1.1.24+nmu1build1\nlibparams-validate-perl:\n  description: Perl module to validate parameters to Perl method/function calls\n  downloads: 2612\n  homepage_uri: http://search.cpan.org/dist/Params-Validate/\n  name: libparams-validate-perl\n  section: perl\n  version: 0.97-1build2\nlibparpack2:\n  description: Parallel subroutines to solve large scale eigenvalue problems\n  downloads: 15\n  homepage_uri: http://forge.scilab.org/index.php/p/arpack-ng/\n  name: libparpack2\n  section: universe/libs\n  version: 3.0.2-3\nlibparrot-dev:\n  description: Parrot shared library development files\n  downloads: 18\n  homepage_uri: http://parrot.org\n  name: libparrot-dev\n  section: universe/libdevel\n  version: 3.6.0-1\nlibparrot3.6.0:\n  description: Parrot shared library\n  downloads: 0\n  homepage_uri: http://parrot.org\n  name: libparrot3.6.0\n  section: universe/libs\n  version: 3.6.0-1\nlibparse-debcontrol-perl:\n  description: parser for debian control-like files\n  downloads: 766\n  homepage_uri: http://search.cpan.org/dist/Parse-DebControl/\n  name: libparse-debcontrol-perl\n  section: perl\n  version: 2.005-3\nlibparse-debianchangelog-perl:\n  description: parse Debian changelogs and output them in other formats\n  downloads: 1924\n  homepage_uri: ''\n  name: libparse-debianchangelog-perl\n  section: perl\n  version: 1.2.0-1ubuntu1\nlibparted0debian1:\n  description: disk partition manipulator - shared library\n  downloads: 4406\n  homepage_uri: http://www.gnu.org/software/parted\n  name: libparted0debian1\n  section: libs\n  version: 2.3-8ubuntu5\nlibpathplan4:\n  description: rich set of graph drawing tools - pathplan library\n  downloads: 3685\n  homepage_uri: http://www.graphviz.org/\n  name: libpathplan4\n  section: libs\n  version: 2.26.3-10ubuntu1\nlibpcap0.8:\n  description: system interface for user-level packet capture\n  downloads: 14933\n  homepage_uri: http://www.tcpdump.org/\n  name: libpcap0.8\n  section: libs\n  version: 1.1.1-10\nlibpci3:\n  description: Linux PCI Utilities (shared library)\n  downloads: 23202\n  homepage_uri: http://atrey.karlin.mff.cuni.cz/~mj/pciutils.shtml\n  name: libpci3\n  section: libs\n  version: 1:3.1.8-2ubuntu5\nlibpciaccess0:\n  description: Generic PCI access library for X\n  downloads: 27790\n  homepage_uri: ''\n  name: libpciaccess0\n  section: libs\n  version: 0.12.902-1\nlibpciaccess0:i386:\n  description: Generic PCI access library for X\n  downloads: 0\n  homepage_uri: ''\n  name: libpciaccess0:i386\n  section: libs\n  version: 0.12.902-1\nlibpcre3:\n  description: Perl 5 Compatible Regular Expression Library - runtime files\n  downloads: 102180\n  homepage_uri: ''\n  name: libpcre3\n  section: libs\n  version: 8.12-4\nlibpcre3-dev:\n  description: Perl 5 Compatible Regular Expression Library - development files\n  downloads: 4209\n  homepage_uri: ''\n  name: libpcre3-dev\n  section: libdevel\n  version: 8.12-4\nlibpcre3:i386:\n  description: Perl 5 Compatible Regular Expression Library - runtime files\n  downloads: 0\n  homepage_uri: ''\n  name: libpcre3:i386\n  section: libs\n  version: 8.12-4\nlibpcrecpp0:\n  description: Perl 5 Compatible Regular Expression Library - C++ runtime files\n  downloads: 206\n  homepage_uri: ''\n  name: libpcrecpp0\n  section: libs\n  version: 8.12-4\nlibpcsclite1:\n  description: Middleware to access a smart card using PC/SC (library)\n  downloads: 40799\n  homepage_uri: http://pcsclite.alioth.debian.org/\n  name: libpcsclite1\n  section: libs\n  version: 1.7.4-2ubuntu2\nlibpeas-1.0-0:\n  description: Application plugin library\n  downloads: 9305\n  homepage_uri: http://live.gnome.org/Libpeas\n  name: libpeas-1.0-0\n  section: libs\n  version: 1.2.0-1ubuntu1\nlibpeas-common:\n  description: Application plugin library (common files)\n  downloads: 8459\n  homepage_uri: http://live.gnome.org/Libpeas\n  name: libpeas-common\n  section: libs\n  version: 1.2.0-1ubuntu1\nlibperl-dev:\n  description: 'Perl library: development files'\n  downloads: 365\n  homepage_uri: ''\n  name: libperl-dev\n  section: libdevel\n  version: 5.14.2-6ubuntu2\nlibperl5.14:\n  description: shared Perl library\n  downloads: 9519\n  homepage_uri: ''\n  name: libperl5.14\n  section: libs\n  version: 5.14.2-6ubuntu2\nlibpgm-5.1-0:\n  description: OpenPGM shared library\n  downloads: 2\n  homepage_uri: http://code.google.com/p/openpgm/\n  name: libpgm-5.1-0\n  section: universe/libs\n  version: 5.1.116~dfsg-2\nlibpipeline1:\n  description: pipeline manipulation library\n  downloads: 20210\n  homepage_uri: http://libpipeline.nongnu.org/\n  name: libpipeline1\n  section: libs\n  version: 1.2.1-1\nlibpixman-1-0:\n  description: pixel-manipulation library for X and cairo\n  downloads: 54407\n  homepage_uri: ''\n  name: libpixman-1-0\n  section: libs\n  version: 0.24.4-1\nlibpixman-1-0:i386:\n  description: pixel-manipulation library for X and cairo\n  downloads: 0\n  homepage_uri: ''\n  name: libpixman-1-0:i386\n  section: libs\n  version: 0.24.4-1\nlibpixman-1-dev:\n  description: pixel-manipulation library for X and cairo (development files)\n  downloads: 2924\n  homepage_uri: ''\n  name: libpixman-1-dev\n  section: libdevel\n  version: 0.24.4-1\nlibplexus-ant-factory-java:\n  description: Plexus Ant Factory\n  downloads: 93\n  homepage_uri: http://plexus.codehaus.org/\n  name: libplexus-ant-factory-java\n  section: universe/java\n  version: 1.0~alpha2.1-3\nlibplexus-archiver-java:\n  description: The archiver plugin for the Plexus compiler system\n  downloads: 93\n  homepage_uri: http://plexus.codehaus.org/plexus-components/plexus-archiver/\n  name: libplexus-archiver-java\n  section: universe/libs\n  version: 1.0~alpha12-3\nlibplexus-bsh-factory-java:\n  description: Plexus Beanshell Factory\n  downloads: 93\n  homepage_uri: http://plexus.codehaus.org/\n  name: libplexus-bsh-factory-java\n  section: universe/java\n  version: 1.0~alpha7-3\nlibplexus-build-api-java:\n  description: Incremental build API for Plexus components.\n  downloads: 20\n  homepage_uri: https://github.com/sonatype/sisu-build-api\n  name: libplexus-build-api-java\n  section: universe/java\n  version: 0.0.4-4\nlibplexus-cipher-java:\n  description: Plexus Cipher Component used by Maven\n  downloads: 5\n  homepage_uri: https://github.com/sonatype/plexus-cipher\n  name: libplexus-cipher-java\n  section: universe/java\n  version: 1.5-2\nlibplexus-classworlds-java:\n  description: Class loading utilities for the Plexus framework\n  downloads: 39\n  homepage_uri: http://plexus.codehaus.org/\n  name: libplexus-classworlds-java\n  section: universe/libs\n  version: 1.5.0-3\nlibplexus-compiler-api-java:\n  description: API for the Plexus compiler system (dummy package)\n  downloads: 0\n  homepage_uri: http://plexus.codehaus.org/plexus-components/plexus-compiler/\n  name: libplexus-compiler-api-java\n  section: universe/libs\n  version: 1.6-2\nlibplexus-compiler-java:\n  description: API for the Plexus compiler system\n  downloads: 0\n  homepage_uri: http://plexus.codehaus.org/plexus-components/plexus-compiler/\n  name: libplexus-compiler-java\n  section: universe/java\n  version: 1.6-2\nlibplexus-compiler-javac-java:\n  description: javac plugin for the Plexus compiler system (dummy package)\n  downloads: 0\n  homepage_uri: http://plexus.codehaus.org/plexus-components/plexus-compiler/\n  name: libplexus-compiler-javac-java\n  section: universe/libs\n  version: 1.6-2\nlibplexus-compiler-manager-java:\n  description: manager for the Plexus compiler system (dummy package)\n  downloads: 0\n  homepage_uri: http://plexus.codehaus.org/plexus-components/plexus-compiler/\n  name: libplexus-compiler-manager-java\n  section: universe/libs\n  version: 1.6-2\nlibplexus-container-default-java:\n  description: utilities for the Plexus framework\n  downloads: 93\n  homepage_uri: http://plexus.codehaus.org/\n  name: libplexus-container-default-java\n  section: universe/libs\n  version: 1.0-alpha-9-stable-1-6\nlibplexus-containers-java:\n  description: utilities for the Plexus framework\n  downloads: 33\n  homepage_uri: http://plexus.codehaus.org\n  name: libplexus-containers-java\n  section: universe/libs\n  version: 1.0~beta3.0.7-5\nlibplexus-digest-java:\n  description: Digest utilities for the Plexus system\n  downloads: 0\n  homepage_uri: http://plexus.codehaus.org/plexus-components/plexus-digest/\n  name: libplexus-digest-java\n  section: universe/java\n  version: 1.1-3\nlibplexus-i18n-java:\n  description: a component to support internationalization of applications using Plexus\n  downloads: 9\n  homepage_uri: http://plexus.codehaus.org/\n  name: libplexus-i18n-java\n  section: universe/libs\n  version: 1.0-beta-10-3\nlibplexus-interactivity-api-java:\n  description: interactivity API for the Plexus framework\n  downloads: 93\n  homepage_uri: http://plexus.codehaus.org/plexus-components/plexus-interactivity/\n  name: libplexus-interactivity-api-java\n  section: universe/libs\n  version: 1.0-alpha-6-6\nlibplexus-interpolation-java:\n  description: Plexus Interpolation API\n  downloads: 93\n  homepage_uri: http://plexus.codehaus.org/plexus-components/plexus-interpolation/\n  name: libplexus-interpolation-java\n  section: universe/libs\n  version: 1.11-3\nlibplexus-io-java:\n  description: Plexus IO Components\n  downloads: 1\n  homepage_uri: http://plexus.codehaus.org/plexus-components/plexus-io/\n  name: libplexus-io-java\n  section: universe/libs\n  version: 1.0~alpha5-2\nlibplexus-sec-dispatcher-java:\n  description: Plexus Security Dispatcher Component used by Maven\n  downloads: 93\n  homepage_uri: https://github.com/sonatype/plexus-sec-dispatcher\n  name: libplexus-sec-dispatcher-java\n  section: universe/java\n  version: 1.3.1-5\nlibplexus-utils-java:\n  description: utilities for the Plexus framework\n  downloads: 98\n  homepage_uri: http://plexus.codehaus.org/\n  name: libplexus-utils-java\n  section: universe/libs\n  version: 1:1.5.15-4\nlibplist1:\n  description: Library for handling Apple binary and XML property lists\n  downloads: 38812\n  homepage_uri: http://www.libimobiledevice.org/\n  name: libplist1\n  section: libs\n  version: 1.8-1\nlibplrpc-perl:\n  description: Perl extensions for writing PlRPC servers and clients\n  downloads: 23806\n  homepage_uri: http://search.cpan.org/search?query=PlRPC&mode=module\n  name: libplrpc-perl\n  section: perl\n  version: 0.2020-2\nlibplymouth2:\n  description: graphical boot animation and logger - shared libraries\n  downloads: 0\n  homepage_uri: ''\n  name: libplymouth2\n  section: libs\n  version: 0.8.2-2ubuntu30\nlibpng12-0:\n  description: PNG library - runtime\n  downloads: 74882\n  homepage_uri: http://libpng.org/pub/png/libpng.html\n  name: libpng12-0\n  section: libs\n  version: 1.2.46-3ubuntu4\nlibpng12-0:i386:\n  description: PNG library - runtime\n  downloads: 0\n  homepage_uri: http://libpng.org/pub/png/libpng.html\n  name: libpng12-0:i386\n  section: libs\n  version: 1.2.46-3ubuntu4\nlibpng12-dev:\n  description: PNG library - development\n  downloads: 5284\n  homepage_uri: http://libpng.org/pub/png/libpng.html\n  name: libpng12-dev\n  section: libdevel\n  version: 1.2.46-3ubuntu4\nlibpolkit-agent-1-0:\n  description: PolicyKit Authentication Agent API\n  downloads: 41384\n  homepage_uri: http://hal.freedesktop.org/docs/PolicyKit/\n  name: libpolkit-agent-1-0\n  section: libs\n  version: 0.104-1ubuntu1\nlibpolkit-backend-1-0:\n  description: PolicyKit backend API\n  downloads: 40928\n  homepage_uri: http://hal.freedesktop.org/docs/PolicyKit/\n  name: libpolkit-backend-1-0\n  section: libs\n  version: 0.104-1ubuntu1\nlibpolkit-gobject-1-0:\n  description: PolicyKit Authorization API\n  downloads: 51739\n  homepage_uri: http://hal.freedesktop.org/docs/PolicyKit/\n  name: libpolkit-gobject-1-0\n  section: libs\n  version: 0.104-1ubuntu1\nlibpoppler-glib8:\n  description: PDF rendering library (GLib-based shared library)\n  downloads: 7196\n  homepage_uri: http://poppler.freedesktop.org/\n  name: libpoppler-glib8\n  section: libs\n  version: 0.18.4-1ubuntu2\nlibpoppler19:\n  description: PDF rendering library\n  downloads: 214\n  homepage_uri: http://poppler.freedesktop.org/\n  name: libpoppler19\n  section: libs\n  version: 0.18.4-1ubuntu2\nlibpopt-dev:\n  description: lib for parsing cmdline parameters - development files\n  downloads: 759\n  homepage_uri: http://rpm5.org/\n  name: libpopt-dev\n  section: libdevel\n  version: 1.16-3ubuntu1\nlibpopt0:\n  description: lib for parsing cmdline parameters\n  downloads: 110129\n  homepage_uri: http://rpm5.org/\n  name: libpopt0\n  section: libs\n  version: 1.16-3ubuntu1\nlibportaudio2:\n  description: Portable audio I/O - shared library\n  downloads: 2126\n  homepage_uri: http://www.portaudio.com/\n  name: libportaudio2\n  section: libs\n  version: 19+svn20111121-1\nlibpq-dev:\n  description: header files for libpq5 (PostgreSQL library)\n  downloads: 1258\n  homepage_uri: http://www.postgresql.org/\n  name: libpq-dev\n  section: libdevel\n  version: 9.1.4-0ubuntu12.04\nlibpq5:\n  description: PostgreSQL C client library\n  downloads: 16324\n  homepage_uri: http://www.postgresql.org/\n  name: libpq5\n  section: libs\n  version: 9.1.4-0ubuntu12.04\nlibprotobuf7:\n  description: protocol buffers C++ library\n  downloads: 52\n  homepage_uri: http://code.google.com/p/protobuf/\n  name: libprotobuf7\n  section: devel\n  version: 2.4.1-1ubuntu2\nlibprotoc7:\n  description: protocol buffers compiler library\n  downloads: 3\n  homepage_uri: http://code.google.com/p/protobuf/\n  name: libprotoc7\n  section: devel\n  version: 2.4.1-1ubuntu2\nlibproxy1:\n  description: automatic proxy configuration management library (shared)\n  downloads: 1\n  homepage_uri: http://code.google.com/p/libproxy/\n  name: libproxy1\n  section: libs\n  version: 0.4.7-0ubuntu4\nlibproxy1-plugin-gsettings:\n  description: automatic proxy configuration management library (GSettings plugin)\n  downloads: 0\n  homepage_uri: http://code.google.com/p/libproxy/\n  name: libproxy1-plugin-gsettings\n  section: libs\n  version: 0.4.7-0ubuntu4\nlibproxy1-plugin-networkmanager:\n  description: automatic proxy configuration management library (Network Manager plugin)\n  downloads: 0\n  homepage_uri: http://code.google.com/p/libproxy/\n  name: libproxy1-plugin-networkmanager\n  section: libs\n  version: 0.4.7-0ubuntu4\nlibproxy1:i386:\n  description: automatic proxy configuration management library (shared)\n  downloads: 0\n  homepage_uri: http://code.google.com/p/libproxy/\n  name: libproxy1:i386\n  section: libs\n  version: 0.4.7-0ubuntu4\nlibpth20:\n  description: The GNU Portable Threads\n  downloads: 39911\n  homepage_uri: http://www.gnu.org/software/pth/\n  name: libpth20\n  section: libs\n  version: 2.0.7-16ubuntu3\nlibpthread-stubs0:\n  description: pthread stubs not provided by native libc\n  downloads: 969\n  homepage_uri: ''\n  name: libpthread-stubs0\n  section: libs\n  version: 0.3-3\nlibpthread-stubs0-dev:\n  description: pthread stubs not provided by native libc, development files\n  downloads: 4850\n  homepage_uri: ''\n  name: libpthread-stubs0-dev\n  section: libdevel\n  version: 0.3-3\nlibpulse-mainloop-glib0:\n  description: PulseAudio client libraries (glib support)\n  downloads: 17470\n  homepage_uri: http://www.pulseaudio.org\n  name: libpulse-mainloop-glib0\n  section: sound\n  version: 1:1.1-0ubuntu15.1\nlibpulse-mainloop-glib0:i386:\n  description: PulseAudio client libraries (glib support)\n  downloads: 0\n  homepage_uri: http://www.pulseaudio.org\n  name: libpulse-mainloop-glib0:i386\n  section: sound\n  version: 1:1.1-0ubuntu15.1\nlibpulse0:\n  description: PulseAudio client libraries\n  downloads: 32941\n  homepage_uri: http://www.pulseaudio.org\n  name: libpulse0\n  section: libs\n  version: 1:1.1-0ubuntu15.1\nlibpulse0:i386:\n  description: PulseAudio client libraries\n  downloads: 0\n  homepage_uri: http://www.pulseaudio.org\n  name: libpulse0:i386\n  section: libs\n  version: 1:1.1-0ubuntu15.1\nlibpulsedsp:\n  description: PulseAudio OSS pre-load library\n  downloads: 0\n  homepage_uri: http://www.pulseaudio.org\n  name: libpulsedsp\n  section: sound\n  version: 1:1.1-0ubuntu15.1\nlibpulsedsp:i386:\n  description: PulseAudio OSS pre-load library\n  downloads: 0\n  homepage_uri: http://www.pulseaudio.org\n  name: libpulsedsp:i386\n  section: sound\n  version: 1:1.1-0ubuntu15.1\nlibpurple-bin:\n  description: multi-protocol instant messaging library - extra utilities\n  downloads: 1289\n  homepage_uri: http://www.pidgin.im\n  name: libpurple-bin\n  section: net\n  version: 1:2.10.3-0ubuntu1\nlibpurple0:\n  description: multi-protocol instant messaging library\n  downloads: 6069\n  homepage_uri: http://www.pidgin.im\n  name: libpurple0\n  section: net\n  version: 1:2.10.3-0ubuntu1\nlibpvm3:\n  description: Parallel Virtual Machine - shared libraries\n  downloads: 261\n  homepage_uri: ''\n  name: libpvm3\n  section: universe/libs\n  version: 3.4.5-12.3\nlibpython2.7:\n  description: Shared Python runtime library (version 2.7)\n  downloads: 13671\n  homepage_uri: ''\n  name: libpython2.7\n  section: libs\n  version: 2.7.3-0ubuntu3\nlibpython3.2:\n  description: Shared Python runtime library (version 3.2)\n  downloads: 97\n  homepage_uri: ''\n  name: libpython3.2\n  section: libs\n  version: 3.2.3-0ubuntu3\nlibqhull5:\n  description: calculate convex hulls and related structures (shared library)\n  downloads: 428\n  homepage_uri: http://www.qhull.org\n  name: libqhull5\n  section: universe/libs\n  version: 2009.1-3ubuntu1\nlibqrupdate1:\n  description: Fast updates of QR and Cholesky decompositions\n  downloads: 2\n  homepage_uri: http://qrupdate.sf.net\n  name: libqrupdate1\n  section: universe/libs\n  version: 1.1.1-1\nlibqt4-dbus:\n  description: Qt 4 D-Bus module\n  downloads: 6741\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-dbus\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-dbus:i386:\n  description: Qt 4 D-Bus module\n  downloads: 0\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-dbus:i386\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-declarative:\n  description: Qt 4 Declarative module\n  downloads: 448\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-declarative\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-declarative:i386:\n  description: Qt 4 Declarative module\n  downloads: 0\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-declarative:i386\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-designer:\n  description: Qt 4 designer module\n  downloads: 301\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-designer\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-designer:i386:\n  description: Qt 4 designer module\n  downloads: 0\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-designer:i386\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-help:\n  description: Qt 4 help module\n  downloads: 524\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-help\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-network:\n  description: Qt 4 network module\n  downloads: 1418\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-network\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-network:i386:\n  description: Qt 4 network module\n  downloads: 0\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-network:i386\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-opengl:\n  description: Qt 4 OpenGL module\n  downloads: 1235\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-opengl\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-opengl:i386:\n  description: Qt 4 OpenGL module\n  downloads: 0\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-opengl:i386\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-qt3support:i386:\n  description: Qt 3 compatibility library for Qt 4\n  downloads: 0\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-qt3support:i386\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-script:\n  description: Qt 4 script module\n  downloads: 1282\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-script\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-script:i386:\n  description: Qt 4 script module\n  downloads: 0\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-script:i386\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-scripttools:\n  description: Qt 4 script tools module\n  downloads: 78\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-scripttools\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-scripttools:i386:\n  description: Qt 4 script tools module\n  downloads: 0\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-scripttools:i386\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-sql:\n  description: Qt 4 SQL module\n  downloads: 659\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-sql\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-sql-mysql:\n  description: Qt 4 MySQL database driver\n  downloads: 3031\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-sql-mysql\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-sql-mysql:i386:\n  description: Qt 4 MySQL database driver\n  downloads: 0\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-sql-mysql:i386\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-sql-sqlite:\n  description: Qt 4 SQLite 3 database driver\n  downloads: 4678\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-sql-sqlite\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-sql:i386:\n  description: Qt 4 SQL module\n  downloads: 0\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-sql:i386\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-svg:\n  description: Qt 4 SVG module\n  downloads: 14224\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-svg\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-svg:i386:\n  description: Qt 4 SVG module\n  downloads: 0\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-svg:i386\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-test:\n  description: Qt 4 test module\n  downloads: 334\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-test\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-test:i386:\n  description: Qt 4 test module\n  downloads: 0\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-test:i386\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-xml:\n  description: Qt 4 XML module\n  downloads: 920\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-xml\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-xml:i386:\n  description: Qt 4 XML module\n  downloads: 0\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-xml:i386\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-xmlpatterns:\n  description: Qt 4 XML patterns module\n  downloads: 438\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-xmlpatterns\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqt4-xmlpatterns:i386:\n  description: Qt 4 XML patterns module\n  downloads: 0\n  homepage_uri: http://qt.nokia.com/\n  name: libqt4-xmlpatterns:i386\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqtassistantclient4:\n  description: Qt Assistant client library (runtime)\n  downloads: 43\n  homepage_uri: http://doc.qt.nokia.com/4.6/qassistantclient.html\n  name: libqtassistantclient4\n  section: libs\n  version: 4.6.3-3ubuntu2\nlibqtbamf1:\n  description: Qt binding and QML plugin for bamf - shared library\n  downloads: 0\n  homepage_uri: ''\n  name: libqtbamf1\n  section: libs\n  version: 0.2.4-0ubuntu1\nlibqtcore4:\n  description: Qt 4 core module\n  downloads: 6540\n  homepage_uri: http://qt.nokia.com/\n  name: libqtcore4\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqtcore4:i386:\n  description: Qt 4 core module\n  downloads: 0\n  homepage_uri: http://qt.nokia.com/\n  name: libqtcore4:i386\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqtdee2:\n  description: Qt binding and QML plugin for Dee - shared library\n  downloads: 0\n  homepage_uri: ''\n  name: libqtdee2\n  section: libs\n  version: 0.2.4-0ubuntu1\nlibqtgconf1:\n  description: Qt binding and QML plugin for GConf - shared library\n  downloads: 0\n  homepage_uri: ''\n  name: libqtgconf1\n  section: libs\n  version: 0.1-0ubuntu5\nlibqtgui4:\n  description: Qt 4 GUI module\n  downloads: 17821\n  homepage_uri: http://qt.nokia.com/\n  name: libqtgui4\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqtgui4:i386:\n  description: Qt 4 GUI module\n  downloads: 0\n  homepage_uri: http://qt.nokia.com/\n  name: libqtgui4:i386\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nlibqtwebkit4:\n  description: Web content engine library for Qt\n  downloads: 363\n  homepage_uri: http://trac.webkit.org/wiki/QtWebKit\n  name: libqtwebkit4\n  section: libs\n  version: 2.2.1-1ubuntu4\nlibqtwebkit4:i386:\n  description: Web content engine library for Qt\n  downloads: 0\n  homepage_uri: http://trac.webkit.org/wiki/QtWebKit\n  name: libqtwebkit4:i386\n  section: libs\n  version: 2.2.1-1ubuntu4\nlibquadmath0:\n  description: GCC Quad-Precision Math Library\n  downloads: 1010\n  homepage_uri: http://gcc.gnu.org/\n  name: libquadmath0\n  section: libs\n  version: 4.6.3-1ubuntu5\nlibquvi-scripts:\n  description: library for parsing video download links (Lua scripts)\n  downloads: 1740\n  homepage_uri: http://quvi.sourceforge.net\n  name: libquvi-scripts\n  section: libs\n  version: 0.4.2-1\nlibquvi7:\n  description: library for parsing video download links (runtime libraries)\n  downloads: 2780\n  homepage_uri: http://quvi.sourceforge.net\n  name: libquvi7\n  section: libs\n  version: 0.4.0-1\nlibrados2:\n  description: RADOS distributed object store client library\n  downloads: 2776\n  homepage_uri: http://ceph.newdream.net/\n  name: librados2\n  section: libs\n  version: 0.41-1ubuntu2\nlibraptor2-0:\n  description: Raptor 2 RDF syntax library\n  downloads: 172\n  homepage_uri: http://librdf.org/raptor/\n  name: libraptor2-0\n  section: libs\n  version: 2.0.6-1\nlibrarian0:\n  description: Documentation meta-data library (library package)\n  downloads: 4\n  homepage_uri: http://rarian.freedesktop.org/\n  name: librarian0\n  section: libs\n  version: 0.8.1-5\nlibrasqal3:\n  description: Rasqal RDF query library\n  downloads: 96\n  homepage_uri: http://librdf.org/rasqal/\n  name: librasqal3\n  section: libs\n  version: 0.9.28-1\nlibraw1394-11:\n  description: library for direct access to IEEE 1394 bus (aka FireWire)\n  downloads: 32778\n  homepage_uri: https://ieee1394.wiki.kernel.org/\n  name: libraw1394-11\n  section: libs\n  version: 2.0.7-1ubuntu1\nlibraw1394-11:i386:\n  description: library for direct access to IEEE 1394 bus (aka FireWire)\n  downloads: 0\n  homepage_uri: https://ieee1394.wiki.kernel.org/\n  name: libraw1394-11:i386\n  section: libs\n  version: 2.0.7-1ubuntu1\nlibraw5:\n  description: raw image decoder library\n  downloads: 1309\n  homepage_uri: http://www.libraw.org/\n  name: libraw5\n  section: libs\n  version: 0.14.4-0ubuntu2\nlibrbd1:\n  description: RADOS block device client library\n  downloads: 2913\n  homepage_uri: http://ceph.newdream.net/\n  name: librbd1\n  section: libs\n  version: 0.41-1ubuntu2\nlibrdf0:\n  description: Redland Resource Description Framework (RDF) library\n  downloads: 468\n  homepage_uri: http://librdf.org/\n  name: librdf0\n  section: libs\n  version: 1.0.14-1\nlibreadline-dev:\n  description: GNU readline and history libraries, development files\n  downloads: 629\n  homepage_uri: ''\n  name: libreadline-dev\n  section: libdevel\n  version: 6.2-8\nlibreadline-java:\n  description: GNU readline and BSD editline wrappers for Java\n  downloads: 102\n  homepage_uri: http://java-readline.sourceforge.net/\n  name: libreadline-java\n  section: universe/libs\n  version: 0.8.0.1+dfsg-2\nlibreadline5:\n  description: GNU readline and history libraries, run-time libraries\n  downloads: 12279\n  homepage_uri: ''\n  name: libreadline5\n  section: libs\n  version: 5.2-11\nlibreadline6:\n  description: GNU readline and history libraries, run-time libraries\n  downloads: 72242\n  homepage_uri: ''\n  name: libreadline6\n  section: libs\n  version: 6.2-8\nlibreadline6-dev:\n  description: GNU readline and history libraries, development files\n  downloads: 1582\n  homepage_uri: ''\n  name: libreadline6-dev\n  section: libdevel\n  version: 6.2-8\nlibrecode0:\n  description: Shared library on which recode is based\n  downloads: 5741\n  homepage_uri: ''\n  name: librecode0\n  section: libs\n  version: 3.6-18\nlibregexp-java:\n  description: Regular expression library for Java\n  downloads: 456\n  homepage_uri: http://jakarta.apache.org/regexp/\n  name: libregexp-java\n  section: libs\n  version: 1.5-3\nlibreoffice-base-core:\n  description: office productivity suite -- shared library\n  downloads: 1442\n  homepage_uri: http://www.libreoffice.org\n  name: libreoffice-base-core\n  section: editors\n  version: 1:3.5.3-0ubuntu1\nlibreoffice-calc:\n  description: office productivity suite -- spreadsheet\n  downloads: 14504\n  homepage_uri: http://www.libreoffice.org\n  name: libreoffice-calc\n  section: editors\n  version: 1:3.5.3-0ubuntu1\nlibreoffice-common:\n  description: office productivity suite -- arch-independent files\n  downloads: 15266\n  homepage_uri: http://www.libreoffice.org\n  name: libreoffice-common\n  section: editors\n  version: 1:3.5.3-0ubuntu1\nlibreoffice-core:\n  description: office productivity suite -- arch-dependent files\n  downloads: 14658\n  homepage_uri: http://www.libreoffice.org\n  name: libreoffice-core\n  section: editors\n  version: 1:3.5.3-0ubuntu1\nlibreoffice-draw:\n  description: office productivity suite -- drawing\n  downloads: 14326\n  homepage_uri: http://www.libreoffice.org\n  name: libreoffice-draw\n  section: editors\n  version: 1:3.5.3-0ubuntu1\nlibreoffice-emailmerge:\n  description: office productivity suite -- email mail merge\n  downloads: 281\n  homepage_uri: http://www.libreoffice.org\n  name: libreoffice-emailmerge\n  section: editors\n  version: 1:3.5.3-0ubuntu1\nlibreoffice-gnome:\n  description: office productivity suite -- GNOME integration\n  downloads: 4712\n  homepage_uri: http://www.libreoffice.org\n  name: libreoffice-gnome\n  section: gnome\n  version: 1:3.5.3-0ubuntu1\nlibreoffice-gtk:\n  description: office productivity suite -- GTK+ integration\n  downloads: 5457\n  homepage_uri: http://www.libreoffice.org\n  name: libreoffice-gtk\n  section: gnome\n  version: 1:3.5.3-0ubuntu1\nlibreoffice-help-en-us:\n  description: office productivity suite -- English_american help\n  downloads: 0\n  homepage_uri: http://www.libreoffice.org\n  name: libreoffice-help-en-us\n  section: doc\n  version: 1:3.5.3-0ubuntu1\nlibreoffice-impress:\n  description: office productivity suite -- presentation\n  downloads: 14275\n  homepage_uri: http://www.libreoffice.org\n  name: libreoffice-impress\n  section: editors\n  version: 1:3.5.3-0ubuntu1\nlibreoffice-math:\n  description: office productivity suite -- equation editor\n  downloads: 14474\n  homepage_uri: http://www.libreoffice.org\n  name: libreoffice-math\n  section: editors\n  version: 1:3.5.3-0ubuntu1\nlibreoffice-style-human:\n  description: office productivity suite -- Human symbol style\n  downloads: 0\n  homepage_uri: http://www.libreoffice.org\n  name: libreoffice-style-human\n  section: editors\n  version: 1:3.5.3-0ubuntu1\nlibreoffice-style-tango:\n  description: office productivity suite -- Tango symbol style\n  downloads: 0\n  homepage_uri: http://www.libreoffice.org\n  name: libreoffice-style-tango\n  section: editors\n  version: 1:3.5.3-0ubuntu1\nlibreoffice-writer:\n  description: office productivity suite -- word processor\n  downloads: 14740\n  homepage_uri: http://www.libreoffice.org\n  name: libreoffice-writer\n  section: editors\n  version: 1:3.5.3-0ubuntu1\nlibrest-0.7-0:\n  description: REST service access library\n  downloads: 7833\n  homepage_uri: http://www.gnome.org/\n  name: librest-0.7-0\n  section: libs\n  version: 0.7.12-1ubuntu2\nlibrhino-java:\n  description: Libraries for rhino Java Script Engine\n  downloads: 569\n  homepage_uri: http://www.mozilla.org/rhino/\n  name: librhino-java\n  section: java\n  version: 1.7R3-5\nlibrhythmbox-core5:\n  description: support library for the rhythmbox music player\n  downloads: 0\n  homepage_uri: http://projects.gnome.org/rhythmbox/\n  name: librhythmbox-core5\n  section: libs\n  version: 2.96-0ubuntu4\nlibroken18-heimdal:\n  description: Heimdal Kerberos - roken support library\n  downloads: 462\n  homepage_uri: http://www.h5l.org/\n  name: libroken18-heimdal\n  section: libs\n  version: 1.6~git20120311.dfsg.1-2\nlibroken18-heimdal:i386:\n  description: Heimdal Kerberos - roken support library\n  downloads: 0\n  homepage_uri: http://www.h5l.org/\n  name: libroken18-heimdal:i386\n  section: libs\n  version: 1.6~git20120311.dfsg.1-2\nlibrsvg2-2:\n  description: SAX-based renderer library for SVG files (runtime)\n  downloads: 42164\n  homepage_uri: http://librsvg.sourceforge.net/\n  name: librsvg2-2\n  section: libs\n  version: 2.36.1-0ubuntu1\nlibrsvg2-2:i386:\n  description: SAX-based renderer library for SVG files (runtime)\n  downloads: 0\n  homepage_uri: http://librsvg.sourceforge.net/\n  name: librsvg2-2:i386\n  section: libs\n  version: 2.36.1-0ubuntu1\nlibrsvg2-common:\n  description: SAX-based renderer library for SVG files (extra runtime)\n  downloads: 44086\n  homepage_uri: http://librsvg.sourceforge.net/\n  name: librsvg2-common\n  section: libs\n  version: 2.36.1-0ubuntu1\nlibrsvg2-common:i386:\n  description: SAX-based renderer library for SVG files (extra runtime)\n  downloads: 0\n  homepage_uri: http://librsvg.sourceforge.net/\n  name: librsvg2-common:i386\n  section: libs\n  version: 2.36.1-0ubuntu1\nlibrsync1:\n  description: rsync remote-delta algorithm library\n  downloads: 2601\n  homepage_uri: http://librsync.sourceforge.net/\n  name: librsync1\n  section: libs\n  version: 0.9.7-8build1\nlibrtmp-dev:\n  description: toolkit for RTMP streams (development files)\n  downloads: 1148\n  homepage_uri: http://rtmpdump.mplayerhq.hu/\n  name: librtmp-dev\n  section: libdevel\n  version: 2.4~20110711.gitc28f1bab-1\nlibrtmp0:\n  description: toolkit for RTMP streams (shared library)\n  downloads: 23234\n  homepage_uri: http://rtmpdump.mplayerhq.hu/\n  name: librtmp0\n  section: libs\n  version: 2.4~20110711.gitc28f1bab-1\nlibrtmp0:i386:\n  description: toolkit for RTMP streams (shared library)\n  downloads: 0\n  homepage_uri: http://rtmpdump.mplayerhq.hu/\n  name: librtmp0:i386\n  section: libs\n  version: 2.4~20110711.gitc28f1bab-1\nlibruby:\n  description: Transitional package for libruby1.8\n  downloads: 92\n  homepage_uri: http://www.ruby-lang.org/\n  name: libruby\n  section: libs\n  version: '4.8'\nlibruby1.8:\n  description: Libraries necessary to run Ruby 1.8\n  downloads: 11284\n  homepage_uri: http://www.ruby-lang.org/\n  name: libruby1.8\n  section: libs\n  version: 1.8.7.352-2ubuntu1\nlibruby1.9.1:\n  description: Libraries necessary to run Ruby 1.9.1\n  downloads: 1918\n  homepage_uri: http://www.ruby-lang.org/\n  name: libruby1.9.1\n  section: libs\n  version: 1.9.3.0-1ubuntu1\nlibsamplerate0:\n  description: Audio sample rate conversion library\n  downloads: 29750\n  homepage_uri: http://www.mega-nerd.com/SRC/\n  name: libsamplerate0\n  section: libs\n  version: 0.1.8-4\nlibsamplerate0:i386:\n  description: Audio sample rate conversion library\n  downloads: 0\n  homepage_uri: http://www.mega-nerd.com/SRC/\n  name: libsamplerate0:i386\n  section: libs\n  version: 0.1.8-4\nlibsane:\n  description: API library for scanners\n  downloads: 23907\n  homepage_uri: http://www.sane-project.org\n  name: libsane\n  section: libs\n  version: 1.0.22-7ubuntu1\nlibsane-common:\n  description: API library for scanners -- documentation and support files\n  downloads: 13894\n  homepage_uri: http://www.sane-project.org\n  name: libsane-common\n  section: libs\n  version: 1.0.22-7ubuntu1\nlibsane-hpaio:\n  description: HP SANE backend for multi-function peripherals\n  downloads: 13127\n  homepage_uri: http://hplipopensource.com/hplip-web/index.html\n  name: libsane-hpaio\n  section: libs\n  version: 3.12.2-1ubuntu3\nlibsane:i386:\n  description: API library for scanners\n  downloads: 0\n  homepage_uri: http://www.sane-project.org\n  name: libsane:i386\n  section: libs\n  version: 1.0.22-7ubuntu1\nlibsasl2-2:\n  description: Cyrus SASL - authentication abstraction library\n  downloads: 70343\n  homepage_uri: http://www.cyrusimap.org/\n  name: libsasl2-2\n  section: libs\n  version: 2.1.25.dfsg1-3ubuntu0.1\nlibsasl2-2:i386:\n  description: Cyrus SASL - authentication abstraction library\n  downloads: 0\n  homepage_uri: http://www.cyrusimap.org/\n  name: libsasl2-2:i386\n  section: libs\n  version: 2.1.25.dfsg1-3ubuntu0.1\nlibsasl2-modules:\n  description: Cyrus SASL - pluggable authentication modules\n  downloads: 16494\n  homepage_uri: http://www.cyrusimap.org/\n  name: libsasl2-modules\n  section: devel\n  version: 2.1.25.dfsg1-3ubuntu0.1\nlibsasl2-modules:i386:\n  description: Cyrus SASL - pluggable authentication modules\n  downloads: 0\n  homepage_uri: http://www.cyrusimap.org/\n  name: libsasl2-modules:i386\n  section: devel\n  version: 2.1.25.dfsg1-3ubuntu0.1\nlibsaxon-java:\n  description: Saxon XSLT Processor\n  downloads: 501\n  homepage_uri: http://saxon.sourceforge.net/\n  name: libsaxon-java\n  section: libs\n  version: 1:6.5.5-8\nlibschroedinger-1.0-0:\n  description: library for encoding/decoding of Dirac video streams\n  downloads: 1181\n  homepage_uri: ''\n  name: libschroedinger-1.0-0\n  section: libs\n  version: 1.0.11-1\nlibsctp1:\n  description: user-space access to Linux Kernel SCTP - shared library\n  downloads: 294\n  homepage_uri: http://lksctp.sf.net/\n  name: libsctp1\n  section: libs\n  version: 1.0.11+dfsg-2\nlibsdl-image1.2:i386:\n  description: image loading library for Simple DirectMedia Layer 1.2\n  downloads: 0\n  homepage_uri: http://www.libsdl.org/projects/SDL_image/\n  name: libsdl-image1.2:i386\n  section: universe/libs\n  version: 1.2.10-3\nlibsdl-mixer1.2:i386:\n  description: Mixer library for Simple DirectMedia Layer 1.2, libraries\n  downloads: 0\n  homepage_uri: http://www.libsdl.org/projects/SDL_mixer/\n  name: libsdl-mixer1.2:i386\n  section: universe/libs\n  version: 1.2.11-7\nlibsdl-net1.2:i386:\n  description: Network library for Simple DirectMedia Layer 1.2, libraries\n  downloads: 0\n  homepage_uri: http://www.libsdl.org/projects/SDL_net/\n  name: libsdl-net1.2:i386\n  section: universe/libs\n  version: 1.2.7-5\nlibsdl-ttf2.0-0:\n  description: ttf library for Simple DirectMedia Layer with FreeType 2 support\n  downloads: 3157\n  homepage_uri: ''\n  name: libsdl-ttf2.0-0\n  section: universe/libs\n  version: 2.0.9-1.1ubuntu1\nlibsdl-ttf2.0-0:i386:\n  description: ttf library for Simple DirectMedia Layer with FreeType 2 support\n  downloads: 0\n  homepage_uri: ''\n  name: libsdl-ttf2.0-0:i386\n  section: universe/libs\n  version: 2.0.9-1.1ubuntu1\nlibsdl1.2debian:\n  description: Simple DirectMedia Layer\n  downloads: 12691\n  homepage_uri: http://www.libsdl.org/\n  name: libsdl1.2debian\n  section: libs\n  version: 1.2.14-6.4ubuntu3\nlibsdl1.2debian:i386:\n  description: Simple DirectMedia Layer\n  downloads: 0\n  homepage_uri: http://www.libsdl.org/\n  name: libsdl1.2debian:i386\n  section: libs\n  version: 1.2.14-6.4ubuntu3\nlibselinux1:\n  description: SELinux runtime shared libraries\n  downloads: 110664\n  homepage_uri: ''\n  name: libselinux1\n  section: libs\n  version: 2.1.0-4.1ubuntu1\nlibselinux1-dev:\n  description: SELinux development headers\n  downloads: 706\n  homepage_uri: ''\n  name: libselinux1-dev\n  section: libs\n  version: 2.1.0-4.1ubuntu1\nlibselinux1:i386:\n  description: SELinux runtime shared libraries\n  downloads: 0\n  homepage_uri: ''\n  name: libselinux1:i386\n  section: libs\n  version: 2.1.0-4.1ubuntu1\nlibsensors4:\n  description: library to read temperature/voltage/fan sensors\n  downloads: 48392\n  homepage_uri: http://www.lm-sensors.org\n  name: libsensors4\n  section: libs\n  version: 1:3.3.1-2ubuntu1\nlibsepol1:\n  description: SELinux library for manipulating binary security policies\n  downloads: 108108\n  homepage_uri: ''\n  name: libsepol1\n  section: libs\n  version: 2.1.0-1.2\nlibsepol1-dev:\n  description: SELinux binary policy maniulation library and development files\n  downloads: 711\n  homepage_uri: ''\n  name: libsepol1-dev\n  section: libdevel\n  version: 2.1.0-1.2\nlibservlet2.4-java:\n  description: Servlet 2.4 and JSP 2.0 Java library\n  downloads: 15\n  homepage_uri: http://jakarta.apache.org/tomcat/index.html\n  name: libservlet2.4-java\n  section: universe/libs\n  version: 5.5.33-1\nlibservlet2.5-java:\n  description: Servlet 2.5 and JSP 2.1 Java API classes\n  downloads: 1271\n  homepage_uri: http://tomcat.apache.org\n  name: libservlet2.5-java\n  section: web\n  version: 6.0.35-1ubuntu3\nlibsgutils2-2:\n  description: utilities for devices using the SCSI command set (shared libraries)\n  downloads: 14688\n  homepage_uri: http://sg.danny.cz/sg/\n  name: libsgutils2-2\n  section: libs\n  version: 1.33-1\nlibshout3:\n  description: MP3/Ogg Vorbis broadcast streaming library\n  downloads: 1266\n  homepage_uri: http://www.icecast.org/\n  name: libshout3\n  section: libs\n  version: 2.2.2-7ubuntu1\nlibshout3:i386:\n  description: MP3/Ogg Vorbis broadcast streaming library\n  downloads: 0\n  homepage_uri: http://www.icecast.org/\n  name: libshout3:i386\n  section: libs\n  version: 2.2.2-7ubuntu1\nlibsigc++-2.0-0c2a:\n  description: type-safe Signal Framework for C++ - runtime\n  downloads: 339\n  homepage_uri: ''\n  name: libsigc++-2.0-0c2a\n  section: libs\n  version: 2.2.10-0ubuntu2\nlibsigsegv2:\n  description: Library for handling page faults in a portable way\n  downloads: 10017\n  homepage_uri: http://libsigsegv.sourceforge.net/\n  name: libsigsegv2\n  section: libs\n  version: 2.9-4ubuntu2\nlibslang2:\n  description: S-Lang programming library - runtime version\n  downloads: 40802\n  homepage_uri: ''\n  name: libslang2\n  section: libs\n  version: 2.2.4-3ubuntu1\nlibslang2:i386:\n  description: S-Lang programming library - runtime version\n  downloads: 0\n  homepage_uri: ''\n  name: libslang2:i386\n  section: libs\n  version: 2.2.4-3ubuntu1\nlibslf4j-java:\n  description: Simple Logging Facade for Java\n  downloads: 400\n  homepage_uri: http://www.slf4j.org/\n  name: libslf4j-java\n  section: universe/libs\n  version: 1.6.4-1\nlibslp1:\n  description: OpenSLP libraries\n  downloads: 3525\n  homepage_uri: http://www.openslp.org/\n  name: libslp1\n  section: libs\n  version: 1.2.1-7.8ubuntu1\nlibsm-dev:\n  description: X11 Session Management library (development headers)\n  downloads: 4318\n  homepage_uri: ''\n  name: libsm-dev\n  section: libdevel\n  version: 2:1.2.0-2build1\nlibsm6:\n  description: X11 Session Management library\n  downloads: 11640\n  homepage_uri: ''\n  name: libsm6\n  section: libs\n  version: 2:1.2.0-2build1\nlibsm6:i386:\n  description: X11 Session Management library\n  downloads: 0\n  homepage_uri: ''\n  name: libsm6:i386\n  section: libs\n  version: 2:1.2.0-2build1\nlibsmbclient:\n  description: shared library for communication with SMB/CIFS servers\n  downloads: 33987\n  homepage_uri: http://www.samba.org\n  name: libsmbclient\n  section: libs\n  version: 2:3.6.3-2ubuntu2.2\nlibsndfile1:\n  description: Library for reading/writing audio files\n  downloads: 11542\n  homepage_uri: http://www.mega-nerd.com/libsndfile/\n  name: libsndfile1\n  section: libs\n  version: 1.0.25-4\nlibsndfile1:i386:\n  description: Library for reading/writing audio files\n  downloads: 0\n  homepage_uri: http://www.mega-nerd.com/libsndfile/\n  name: libsndfile1:i386\n  section: libs\n  version: 1.0.25-4\nlibsnmp-base:\n  description: SNMP (Simple Network Management Protocol) MIBs and documentation\n  downloads: 18974\n  homepage_uri: http://net-snmp.sourceforge.net/\n  name: libsnmp-base\n  section: libs\n  version: 5.4.3~dfsg-2.4ubuntu1.1\nlibsnmp15:\n  description: SNMP (Simple Network Management Protocol) library\n  downloads: 27082\n  homepage_uri: http://net-snmp.sourceforge.net/\n  name: libsnmp15\n  section: libs\n  version: 5.4.3~dfsg-2.4ubuntu1.1\nlibsoap-lite-perl:\n  description: Perl implementation of a SOAP client and server\n  downloads: 1802\n  homepage_uri: http://sourceforge.net/projects/soaplite/\n  name: libsoap-lite-perl\n  section: universe/perl\n  version: 0.714-1\nlibsocket6-perl:\n  description: Perl extensions for IPv6\n  downloads: 11051\n  homepage_uri: ''\n  name: libsocket6-perl\n  section: perl\n  version: 0.23-1build2\nlibsonic0:\n  description: Simple library to speed up or slow down speech\n  downloads: 132\n  homepage_uri: http://dev.vinux-project.org/sonic\n  name: libsonic0\n  section: libs\n  version: 0.1.17-1.1\nlibsoup-gnome2.4-1:\n  description: HTTP library implementation in C -- GNOME support library\n  downloads: 26762\n  homepage_uri: ''\n  name: libsoup-gnome2.4-1\n  section: libs\n  version: 2.38.1-1\nlibsoup-gnome2.4-1:i386:\n  description: HTTP library implementation in C -- GNOME support library\n  downloads: 0\n  homepage_uri: ''\n  name: libsoup-gnome2.4-1:i386\n  section: libs\n  version: 2.38.1-1\nlibsoup2.4-1:\n  description: HTTP library implementation in C -- Shared library\n  downloads: 35798\n  homepage_uri: ''\n  name: libsoup2.4-1\n  section: libs\n  version: 2.38.1-1\nlibsoup2.4-1:i386:\n  description: HTTP library implementation in C -- Shared library\n  downloads: 0\n  homepage_uri: ''\n  name: libsoup2.4-1:i386\n  section: libs\n  version: 2.38.1-1\nlibsparsehash-dev:\n  description: Google's extremely memory-efficient C++ hash_map implementation\n  downloads: 10\n  homepage_uri: http://code.google.com/p/google-sparsehash\n  name: libsparsehash-dev\n  section: universe/libdevel\n  version: 1.10-1build1\nlibspectre1:\n  description: Library for rendering PostScript documents\n  downloads: 51\n  homepage_uri: http://libspectre.freedesktop.org\n  name: libspectre1\n  section: libs\n  version: 0.2.6-1build1\nlibspeechd2:\n  description: 'Speech Dispatcher: Shared libraries'\n  downloads: 1663\n  homepage_uri: http://devel.freebsoft.org/speechd\n  name: libspeechd2\n  section: libs\n  version: 0.7.1-6ubuntu3\nlibspeex1:\n  description: The Speex codec runtime library\n  downloads: 21855\n  homepage_uri: http://www.speex.org/\n  name: libspeex1\n  section: libs\n  version: 1.2~rc1-3ubuntu2\nlibspeex1:i386:\n  description: The Speex codec runtime library\n  downloads: 0\n  homepage_uri: http://www.speex.org/\n  name: libspeex1:i386\n  section: libs\n  version: 1.2~rc1-3ubuntu2\nlibspeexdsp1:\n  description: The Speex extended runtime library\n  downloads: 5580\n  homepage_uri: http://www.speex.org/\n  name: libspeexdsp1\n  section: libs\n  version: 1.2~rc1-3ubuntu2\nlibspeexdsp1:i386:\n  description: The Speex extended runtime library\n  downloads: 0\n  homepage_uri: http://www.speex.org/\n  name: libspeexdsp1:i386\n  section: libs\n  version: 1.2~rc1-3ubuntu2\nlibsprng2:\n  description: SPRNG Scalable Parallel RNG library -- library package\n  downloads: 71\n  homepage_uri: ''\n  name: libsprng2\n  section: universe/math\n  version: 2.0a-8ubuntu1\nlibsqlite0:\n  description: SQLite shared library\n  downloads: 1414\n  homepage_uri: ''\n  name: libsqlite0\n  section: universe/libs\n  version: 2.8.17-7fakesync1build1\nlibsqlite0-dev:\n  description: SQLite development files\n  downloads: 142\n  homepage_uri: ''\n  name: libsqlite0-dev\n  section: universe/libdevel\n  version: 2.8.17-7fakesync1build1\nlibsqlite3-0:\n  description: SQLite 3 shared library\n  downloads: 81378\n  homepage_uri: ''\n  name: libsqlite3-0\n  section: libs\n  version: 3.7.9-2ubuntu1\nlibsqlite3-0:i386:\n  description: SQLite 3 shared library\n  downloads: 0\n  homepage_uri: ''\n  name: libsqlite3-0:i386\n  section: libs\n  version: 3.7.9-2ubuntu1\nlibsqlite3-dev:\n  description: SQLite 3 development files\n  downloads: 1843\n  homepage_uri: ''\n  name: libsqlite3-dev\n  section: libdevel\n  version: 3.7.9-2ubuntu1\nlibss2:\n  description: command-line interface parsing library\n  downloads: 1052\n  homepage_uri: http://e2fsprogs.sourceforge.net\n  name: libss2\n  section: libs\n  version: 1.42-1ubuntu2\nlibssh-4:\n  description: tiny C SSH library\n  downloads: 3905\n  homepage_uri: http://www.libssh.org/\n  name: libssh-4\n  section: libs\n  version: 0.5.2-1\nlibssl-dev:\n  description: SSL development libraries, header files and documentation\n  downloads: 5911\n  homepage_uri: ''\n  name: libssl-dev\n  section: libdevel\n  version: 1.0.1-4ubuntu5.2\nlibssl-doc:\n  description: SSL development documentation documentation\n  downloads: 101\n  homepage_uri: ''\n  name: libssl-doc\n  section: doc\n  version: 1.0.1-4ubuntu5.2\nlibssl0.9.8:i386:\n  description: SSL shared libraries\n  downloads: 0\n  homepage_uri: ''\n  name: libssl0.9.8:i386\n  section: universe/libs\n  version: 0.9.8o-7ubuntu3.1\nlibssl1.0.0:\n  description: SSL shared libraries\n  downloads: 22162\n  homepage_uri: ''\n  name: libssl1.0.0\n  section: libs\n  version: 1.0.1-4ubuntu5.2\nlibssl1.0.0:i386:\n  description: SSL shared libraries\n  downloads: 0\n  homepage_uri: ''\n  name: libssl1.0.0:i386\n  section: libs\n  version: 1.0.1-4ubuntu5.2\nlibstartup-notification0:\n  description: library for program launch feedback (shared library)\n  downloads: 36041\n  homepage_uri: ''\n  name: libstartup-notification0\n  section: libs\n  version: 0.12-1ubuntu1\nlibstdc++5:i386:\n  description: The GNU Standard C++ Library v3\n  downloads: 0\n  homepage_uri: ''\n  name: libstdc++5:i386\n  section: universe/libs\n  version: 1:3.3.6-25ubuntu1\nlibstdc++6:\n  description: GNU Standard C++ Library v3\n  downloads: 73251\n  homepage_uri: http://gcc.gnu.org/\n  name: libstdc++6\n  section: libs\n  version: 4.6.3-1ubuntu5\nlibstdc++6-4.6-dev:\n  description: GNU Standard C++ Library v3 (development files)\n  downloads: 5740\n  homepage_uri: http://gcc.gnu.org/\n  name: libstdc++6-4.6-dev\n  section: libdevel\n  version: 4.6.3-1ubuntu5\nlibstdc++6:i386:\n  description: GNU Standard C++ Library v3\n  downloads: 0\n  homepage_uri: http://gcc.gnu.org/\n  name: libstdc++6:i386\n  section: libs\n  version: 4.6.3-1ubuntu5\nlibstringtemplate-java:\n  description: StringTemplate templating engine for Java\n  downloads: 842\n  homepage_uri: http://www.stringtemplate.org/\n  name: libstringtemplate-java\n  section: universe/devel\n  version: 3.2.1-1\nlibsub-name-perl:\n  description: module for assigning a new name to referenced sub\n  downloads: 2975\n  homepage_uri: http://search.cpan.org/dist/Sub-Name/\n  name: libsub-name-perl\n  section: perl\n  version: 0.05-1build2\nlibsvn1:\n  description: Shared libraries used by Subversion\n  downloads: 12515\n  homepage_uri: http://subversion.apache.org/\n  name: libsvn1\n  section: libs\n  version: 1.6.17dfsg-3ubuntu3\nlibswitch-perl:\n  description: switch statement for Perl\n  downloads: 16209\n  homepage_uri: http://search.cpan.org/dist/Switch/\n  name: libswitch-perl\n  section: perl\n  version: 2.16-2\nlibswscale2:\n  description: Libav video scaling library\n  downloads: 2242\n  homepage_uri: http://libav.org/\n  name: libswscale2\n  section: libs\n  version: 4:0.8.1-0ubuntu1\nlibswt-cairo-gtk-3-jni:\n  description: Standard Widget Toolkit for GTK+ Cairo JNI library\n  downloads: 927\n  homepage_uri: http://www.eclipse.org/swt/\n  name: libswt-cairo-gtk-3-jni\n  section: universe/libs\n  version: 3.7.2-2\nlibswt-glx-gtk-3-jni:\n  description: Standard Widget Toolkit for GTK+ GLX JNI library\n  downloads: 48\n  homepage_uri: http://www.eclipse.org/swt/\n  name: libswt-glx-gtk-3-jni\n  section: universe/libs\n  version: 3.7.2-2\nlibswt-gnome-gtk-3-jni:\n  description: Standard Widget Toolkit for GTK+ GNOME JNI library\n  downloads: 244\n  homepage_uri: http://www.eclipse.org/swt/\n  name: libswt-gnome-gtk-3-jni\n  section: universe/libs\n  version: 3.7.2-2\nlibswt-gtk-3-java:\n  description: Standard Widget Toolkit for GTK+ Java library\n  downloads: 971\n  homepage_uri: http://www.eclipse.org/swt/\n  name: libswt-gtk-3-java\n  section: universe/java\n  version: 3.7.2-2\nlibswt-gtk-3-jni:\n  description: Standard Widget Toolkit for GTK+ JNI library\n  downloads: 942\n  homepage_uri: http://www.eclipse.org/swt/\n  name: libswt-gtk-3-jni\n  section: universe/libs\n  version: 3.7.2-2\nlibswt-webkit-gtk-3-jni:\n  description: Standard Widget Toolkit for GTK+ WebKit JNI library\n  downloads: 916\n  homepage_uri: http://www.eclipse.org/swt/\n  name: libswt-webkit-gtk-3-jni\n  section: universe/libs\n  version: 3.7.2-2\nlibsyncdaemon-1.0-1:\n  description: Ubuntu One synchronization daemon library\n  downloads: 0\n  homepage_uri: https://one.ubuntu.com\n  name: libsyncdaemon-1.0-1\n  section: libs\n  version: 3.0.1-0ubuntu1.0.1\nlibsys-hostname-long-perl:\n  description: Figure out the long (fully-qualified) hostname\n  downloads: 7705\n  homepage_uri: http://search.cpan.org/dist/Sys-Hostname-Long/\n  name: libsys-hostname-long-perl\n  section: perl\n  version: 1.4-2\nlibsysfs2:\n  description: interface library to sysfs\n  downloads: 1267\n  homepage_uri: ''\n  name: libsysfs2\n  section: libs\n  version: 2.1.0+repack-1\nlibt1-5:\n  description: Type 1 font rasterizer library - runtime\n  downloads: 328\n  homepage_uri: ftp://sunsite.unc.edu/pub/Linux/libs/graphics/\n  name: libt1-5\n  section: libs\n  version: 5.1.2-3.4ubuntu1\nlibtag1-vanilla:\n  description: audio meta-data library - vanilla flavour\n  downloads: 3317\n  homepage_uri: http://developer.kde.org/~wheeler/taglib.html\n  name: libtag1-vanilla\n  section: libs\n  version: 1.7-1ubuntu5\nlibtag1-vanilla:i386:\n  description: audio meta-data library - vanilla flavour\n  downloads: 0\n  homepage_uri: http://developer.kde.org/~wheeler/taglib.html\n  name: libtag1-vanilla:i386\n  section: libs\n  version: 1.7-1ubuntu5\nlibtag1c2a:\n  description: audio meta-data library\n  downloads: 2589\n  homepage_uri: http://developer.kde.org/~wheeler/taglib.html\n  name: libtag1c2a\n  section: libs\n  version: 1.7-1ubuntu5\nlibtag1c2a:i386:\n  description: audio meta-data library\n  downloads: 0\n  homepage_uri: http://developer.kde.org/~wheeler/taglib.html\n  name: libtag1c2a:i386\n  section: libs\n  version: 1.7-1ubuntu5\nlibtalloc2:\n  description: hierarchical pool based memory allocator\n  downloads: 42627\n  homepage_uri: http://talloc.samba.org/\n  name: libtalloc2\n  section: libs\n  version: 2.0.7-3\nlibtask-weaken-perl:\n  description: Ensure that a platform has weaken support\n  downloads: 1721\n  homepage_uri: http://search.cpan.org/dist/Task-Weaken/\n  name: libtask-weaken-perl\n  section: perl\n  version: 1.03-1\nlibtasn1-3:\n  description: Manage ASN.1 structures (runtime)\n  downloads: 63506\n  homepage_uri: http://www.gnu.org/software/libtasn1/\n  name: libtasn1-3\n  section: libs\n  version: 2.10-1ubuntu1.1\nlibtasn1-3-dev:\n  description: Manage ASN.1 structures (development)\n  downloads: 2279\n  homepage_uri: http://www.gnu.org/software/libtasn1/\n  name: libtasn1-3-dev\n  section: libdevel\n  version: 2.10-1ubuntu1.1\nlibtasn1-3:i386:\n  description: Manage ASN.1 structures (runtime)\n  downloads: 0\n  homepage_uri: http://www.gnu.org/software/libtasn1/\n  name: libtasn1-3:i386\n  section: libs\n  version: 2.10-1ubuntu1.1\nlibtdb1:\n  description: Trivial Database - shared library\n  downloads: 43875\n  homepage_uri: http://tdb.samba.org/\n  name: libtdb1\n  section: libs\n  version: 1.2.9-4\nlibtdb1:i386:\n  description: Trivial Database - shared library\n  downloads: 0\n  homepage_uri: http://tdb.samba.org/\n  name: libtdb1:i386\n  section: libs\n  version: 1.2.9-4\nlibtelepathy-farstream2:\n  description: Glue library between telepathy and farstream\n  downloads: 16\n  homepage_uri: http://telepathy.freedesktop.org/\n  name: libtelepathy-farstream2\n  section: libs\n  version: 0.4.0-0ubuntu1\nlibtelepathy-glib0:\n  description: Telepathy framework - GLib library\n  downloads: 12519\n  homepage_uri: http://telepathy.freedesktop.org/wiki/\n  name: libtelepathy-glib0\n  section: libs\n  version: 0.18.0-1ubuntu1\nlibtelepathy-logger2:\n  description: Telepathy logger service - utility library\n  downloads: 8063\n  homepage_uri: http://telepathy.freedesktop.org/wiki/\n  name: libtelepathy-logger2\n  section: libs\n  version: 0.4.0-0ubuntu1\nlibterm-readkey-perl:\n  description: A perl module for simple terminal control\n  downloads: 5229\n  homepage_uri: http://search.cpan.org/dist/TermReadKey/\n  name: libterm-readkey-perl\n  section: perl\n  version: 2.30-4build3\nlibtext-charwidth-perl:\n  description: get display widths of characters on the terminal\n  downloads: 56874\n  homepage_uri: http://search.cpan.org/search?module=Text::CharWidth\n  name: libtext-charwidth-perl\n  section: perl\n  version: 0.04-7build1\nlibtext-csv-perl:\n  description: comma-separated values manipulator (using XS or PurePerl)\n  downloads: 379\n  homepage_uri: http://search.cpan.org/dist/Text-CSV/\n  name: libtext-csv-perl\n  section: perl\n  version: 1.21-1\nlibtext-csv-xs-perl:\n  description: Perl C/XS module to process Comma-Separated Value files\n  downloads: 547\n  homepage_uri: http://search.cpan.org/dist/Text-CSV_XS/\n  name: libtext-csv-xs-perl\n  section: perl\n  version: 0.85-1build1\nlibtext-iconv-perl:\n  description: converts between character sets in Perl\n  downloads: 55885\n  homepage_uri: http://search.cpan.org/search?module=Text::Iconv\n  name: libtext-iconv-perl\n  section: perl\n  version: 1.7-5\nlibtext-wrapi18n-perl:\n  description: internationalized substitute of Text::Wrap\n  downloads: 56781\n  homepage_uri: http://search.cpan.org/search?module=Text::WrapI18N\n  name: libtext-wrapi18n-perl\n  section: perl\n  version: 0.06-7\nlibthai-data:\n  description: Data files for Thai language support library\n  downloads: 44645\n  homepage_uri: http://linux.thai.net/projects/libthai\n  name: libthai-data\n  section: libs\n  version: 0.1.16-3\nlibthai0:\n  description: Thai language support library\n  downloads: 53443\n  homepage_uri: http://linux.thai.net/projects/libthai\n  name: libthai0\n  section: libs\n  version: 0.1.16-3\nlibthai0:i386:\n  description: Thai language support library\n  downloads: 0\n  homepage_uri: http://linux.thai.net/projects/libthai\n  name: libthai0:i386\n  section: libs\n  version: 0.1.16-3\nlibtheora0:\n  description: The Theora Video Compression Codec\n  downloads: 2207\n  homepage_uri: http://www.theora.org/\n  name: libtheora0\n  section: libs\n  version: 1.1.1+dfsg.1-3ubuntu2\nlibtheora0:i386:\n  description: The Theora Video Compression Codec\n  downloads: 0\n  homepage_uri: http://www.theora.org/\n  name: libtheora0:i386\n  section: libs\n  version: 1.1.1+dfsg.1-3ubuntu2\nlibtie-ixhash-perl:\n  description: ordered associative arrays for Perl\n  downloads: 1420\n  homepage_uri: ''\n  name: libtie-ixhash-perl\n  section: perl\n  version: 1.21-2\nlibtiff4:\n  description: Tag Image File Format (TIFF) library\n  downloads: 39340\n  homepage_uri: http://libtiff.maptools.org\n  name: libtiff4\n  section: libs\n  version: 3.9.5-2ubuntu1\nlibtiff4:i386:\n  description: Tag Image File Format (TIFF) library\n  downloads: 0\n  homepage_uri: http://libtiff.maptools.org\n  name: libtiff4:i386\n  section: libs\n  version: 3.9.5-2ubuntu1\nlibtimedate-perl:\n  description: collection of modules to manipulate date/time information\n  downloads: 15372\n  homepage_uri: http://search.cpan.org/dist/TimeDate/\n  name: libtimedate-perl\n  section: perl\n  version: 1.2000-1\nlibtimezonemap1:\n  description: GTK+3 timezone map widget\n  downloads: 0\n  homepage_uri: ''\n  name: libtimezonemap1\n  section: libs\n  version: 0.3.2\nlibtinfo-dev:\n  description: developer's library for the low-level terminfo library\n  downloads: 2346\n  homepage_uri: http://invisible-island.net/ncurses/\n  name: libtinfo-dev\n  section: libdevel\n  version: 5.9-4\nlibtinfo5:\n  description: shared low-level terminfo library for terminal handling\n  downloads: 21285\n  homepage_uri: http://invisible-island.net/ncurses/\n  name: libtinfo5\n  section: libs\n  version: 5.9-4\nlibtinfo5:i386:\n  description: shared low-level terminfo library for terminal handling\n  downloads: 0\n  homepage_uri: http://invisible-island.net/ncurses/\n  name: libtinfo5:i386\n  section: libs\n  version: 5.9-4\nlibtirpc1:\n  description: transport-independent RPC library\n  downloads: 13634\n  homepage_uri: ''\n  name: libtirpc1\n  section: libs\n  version: 0.2.2-5\nlibtntnet9:\n  description: Tntnet libraries\n  downloads: 0\n  homepage_uri: http://www.tntnet.org/\n  name: libtntnet9\n  section: universe/libs\n  version: 2.0+dfsg1-2\nlibtokyocabinet-dev:\n  description: Tokyo Cabinet Database Libraries [development]\n  downloads: 38\n  homepage_uri: ''\n  name: libtokyocabinet-dev\n  section: libdevel\n  version: 1.4.37-6.1build1\nlibtokyocabinet8:\n  description: Tokyo Cabinet Database Libraries [runtime]\n  downloads: 6\n  homepage_uri: ''\n  name: libtokyocabinet8\n  section: libs\n  version: 1.4.37-6.1build1\nlibtorque2:\n  description: shared library for Torque client and server\n  downloads: 77\n  homepage_uri: http://www.clusterresources.com/pages/products/torque/\n  name: libtorque2\n  section: universe/libs\n  version: 2.4.16+dfsg-1build1\nlibtotem-plparser17:\n  description: Totem Playlist Parser library - runtime files\n  downloads: 30386\n  homepage_uri: ''\n  name: libtotem-plparser17\n  section: libs\n  version: 3.4.1-1\nlibtotem0:\n  description: Main library for the Totem media player\n  downloads: 8049\n  homepage_uri: http://www.gnome.org/projects/totem/\n  name: libtotem0\n  section: video\n  version: 3.0.1-0ubuntu21\nlibubuntuoneui-3.0-1:\n  description: Ubuntu One widget library\n  downloads: 0\n  homepage_uri: https://launchpad.net/libubuntuone\n  name: libubuntuoneui-3.0-1\n  section: libs\n  version: 3.0.1-0ubuntu1\nlibudev0:\n  description: udev library\n  downloads: 72455\n  homepage_uri: http://www.kernel.org/pub/linux/utils/kernel/hotplug/udev.html\n  name: libudev0\n  section: admin\n  version: 175-0ubuntu9\nlibudev0:i386:\n  description: udev library\n  downloads: 0\n  homepage_uri: http://www.kernel.org/pub/linux/utils/kernel/hotplug/udev.html\n  name: libudev0:i386\n  section: admin\n  version: 175-0ubuntu9\nlibumfpack5.4.0:\n  description: sparse LU factorization library\n  downloads: 0\n  homepage_uri: http://www.cise.ufl.edu/research/sparse/SuiteSparse/\n  name: libumfpack5.4.0\n  section: universe/libs\n  version: 1:3.4.0-2ubuntu3\nlibunique-3.0-0:\n  description: Library for writing single instance applications - shared libraries\n  downloads: 7505\n  homepage_uri: http://live.gnome.org/LibUnique\n  name: libunique-3.0-0\n  section: libs\n  version: 3.0.2-1\nlibunistring0:\n  description: Unicode string library for C\n  downloads: 14108\n  homepage_uri: http://www.gnu.org/software/libunistring/\n  name: libunistring0\n  section: libs\n  version: 0.9.3-5\nlibunistring0:i386:\n  description: Unicode string library for C\n  downloads: 0\n  homepage_uri: http://www.gnu.org/software/libunistring/\n  name: libunistring0:i386\n  section: libs\n  version: 0.9.3-5\nlibunity-2d-private0:\n  description: Unity 2D shared library\n  downloads: 0\n  homepage_uri: ''\n  name: libunity-2d-private0\n  section: libs\n  version: 5.12.0-0ubuntu1.1\nlibunity-core-5.0-5:\n  description: Core library for the Unity interface.\n  downloads: 0\n  homepage_uri: https://launchpad.net/unity\n  name: libunity-core-5.0-5\n  section: libs\n  version: 5.12-0ubuntu1.1\nlibunity-misc4:\n  description: Miscellaneous functions for Unity - shared library\n  downloads: 0\n  homepage_uri: https://launchpad.net/libunity-misc\n  name: libunity-misc4\n  section: libs\n  version: 4.0.4-0ubuntu2\nlibunity9:\n  description: binding to get places into the launcher - shared library\n  downloads: 0\n  homepage_uri: https://launchpad.net/libunity\n  name: libunity9\n  section: libs\n  version: 5.12.0-0ubuntu1\nlibupower-glib1:\n  description: abstraction for power management - shared library\n  downloads: 35721\n  homepage_uri: http://upower.freedesktop.org/\n  name: libupower-glib1\n  section: libs\n  version: 0.9.15-3git1\nliburi-perl:\n  description: module to manipulate and access URI strings\n  downloads: 40880\n  homepage_uri: http://search.cpan.org/dist/URI/\n  name: liburi-perl\n  section: perl\n  version: 1.59-1\nlibusb-0.1-4:\n  description: userspace USB programming library\n  downloads: 59696\n  homepage_uri: http://www.linux-usb.org/\n  name: libusb-0.1-4\n  section: libs\n  version: 2:0.1.12-20\nlibusb-0.1-4:i386:\n  description: userspace USB programming library\n  downloads: 0\n  homepage_uri: http://www.linux-usb.org/\n  name: libusb-0.1-4:i386\n  section: libs\n  version: 2:0.1.12-20\nlibusb-1.0-0:\n  description: userspace USB programming library\n  downloads: 43040\n  homepage_uri: http://www.linux-usb.org/\n  name: libusb-1.0-0\n  section: libs\n  version: 2:1.0.9~rc3-2ubuntu1\nlibusbmuxd1:\n  description: USB multiplexor daemon for iPhone and iPod Touch devices - library\n  downloads: 14591\n  homepage_uri: http://marcansoft.com/blog/iphonelinux/usbmuxd/\n  name: libusbmuxd1\n  section: libs\n  version: 1.0.7-2\nlibutempter0:\n  description: A privileged helper for utmp/wtmp updates (runtime)\n  downloads: 13636\n  homepage_uri: http://freshmeat.net/projects/libutempter\n  name: libutempter0\n  section: libs\n  version: 1.1.5-4\nlibutouch-evemu1:\n  description: KernelInput Event Device Emulation Library\n  downloads: 0\n  homepage_uri: https://launchpad.net/utouch-evemu\n  name: libutouch-evemu1\n  section: libs\n  version: 1.0.9-0ubuntu1\nlibutouch-frame1:\n  description: Touch Frame Library\n  downloads: 0\n  homepage_uri: https://launchpad.net/utouch-frame\n  name: libutouch-frame1\n  section: libs\n  version: 2.2.3-0ubuntu1\nlibutouch-geis1:\n  description: Gesture engine interface support\n  downloads: 0\n  homepage_uri: https://launchpad.net/utouch-geis\n  name: libutouch-geis1\n  section: libs\n  version: 2.2.9-0ubuntu3\nlibutouch-grail1:\n  description: Gesture Recognition And Instantiation Library\n  downloads: 0\n  homepage_uri: https://launchpad.net/utouch-grail\n  name: libutouch-grail1\n  section: libs\n  version: 3.0.5-0ubuntu1\nlibuuid-perl:\n  description: Perl extension for using UUID interfaces as defined in e2fsprogs\n  downloads: 37705\n  homepage_uri: http://search.cpan.org/dist/UUID/\n  name: libuuid-perl\n  section: perl\n  version: 0.02-4ubuntu1\nlibuuid1:\n  description: Universally Unique ID library\n  downloads: 73488\n  homepage_uri: http://userweb.kernel.org/~kzak/util-linux/\n  name: libuuid1\n  section: libs\n  version: 2.20.1-1ubuntu3\nlibuuid1:i386:\n  description: Universally Unique ID library\n  downloads: 0\n  homepage_uri: http://userweb.kernel.org/~kzak/util-linux/\n  name: libuuid1:i386\n  section: libs\n  version: 2.20.1-1ubuntu3\nlibv4l-0:\n  description: Collection of video4linux support libraries\n  downloads: 4781\n  homepage_uri: http://linuxtv.org/downloads/v4l-utils/\n  name: libv4l-0\n  section: libs\n  version: 0.8.6-1ubuntu2\nlibv4l-0:i386:\n  description: Collection of video4linux support libraries\n  downloads: 0\n  homepage_uri: http://linuxtv.org/downloads/v4l-utils/\n  name: libv4l-0:i386\n  section: libs\n  version: 0.8.6-1ubuntu2\nlibv4lconvert0:\n  description: Video4linux frame format conversion library\n  downloads: 3177\n  homepage_uri: http://linuxtv.org/downloads/v4l-utils/\n  name: libv4lconvert0\n  section: libs\n  version: 0.8.6-1ubuntu2\nlibv4lconvert0:i386:\n  description: Video4linux frame format conversion library\n  downloads: 0\n  homepage_uri: http://linuxtv.org/downloads/v4l-utils/\n  name: libv4lconvert0:i386\n  section: libs\n  version: 0.8.6-1ubuntu2\nlibva1:\n  description: Video Acceleration (VA) API for Linux -- runtime\n  downloads: 2614\n  homepage_uri: http://www.freedesktop.org/wiki/Software/vaapi\n  name: libva1\n  section: libs\n  version: 1.0.15-4\nlibvibrant6a:\n  description: NCBI libraries for graphic biology applications\n  downloads: 120\n  homepage_uri: http://www.ncbi.nlm.nih.gov/IEB/ToolBox/\n  name: libvibrant6a\n  section: libs\n  version: 6.1.20110713-3ubuntu2\nlibvirt-bin:\n  description: programs for the libvirt library\n  downloads: 3171\n  homepage_uri: http://libvirt.org\n  name: libvirt-bin\n  section: devel\n  version: 0.9.8-2ubuntu17.1\nlibvirt0:\n  description: library for interfacing with different virtualization systems\n  downloads: 532\n  homepage_uri: http://libvirt.org\n  name: libvirt0\n  section: devel\n  version: 0.9.8-2ubuntu17.1\nlibvisual-0.4-0:\n  description: Audio visualization framework\n  downloads: 335\n  homepage_uri: ''\n  name: libvisual-0.4-0\n  section: libs\n  version: 0.4.0-4\nlibvisual-0.4-0:i386:\n  description: Audio visualization framework\n  downloads: 0\n  homepage_uri: ''\n  name: libvisual-0.4-0:i386\n  section: libs\n  version: 0.4.0-4\nlibvisual-0.4-plugins:\n  description: Audio visualization framework plugins\n  downloads: 2341\n  homepage_uri: ''\n  name: libvisual-0.4-plugins\n  section: sound\n  version: 0.4.0.dfsg.1-7\nlibvisual-0.4-plugins:i386:\n  description: Audio visualization framework plugins\n  downloads: 0\n  homepage_uri: ''\n  name: libvisual-0.4-plugins:i386\n  section: sound\n  version: 0.4.0.dfsg.1-7\nlibvncserver0:\n  description: API to write one's own vnc server\n  downloads: 1953\n  homepage_uri: http://libvncserver.sourceforge.net\n  name: libvncserver0\n  section: libs\n  version: 0.9.8.2-2ubuntu1\nlibvorbis0a:\n  description: The Vorbis General Audio Compression Codec (Decoder library)\n  downloads: 42690\n  homepage_uri: ''\n  name: libvorbis0a\n  section: libs\n  version: 1.3.2-1ubuntu3\nlibvorbis0a:i386:\n  description: The Vorbis General Audio Compression Codec (Decoder library)\n  downloads: 0\n  homepage_uri: ''\n  name: libvorbis0a:i386\n  section: libs\n  version: 1.3.2-1ubuntu3\nlibvorbisenc2:\n  description: The Vorbis General Audio Compression Codec (Encoder library)\n  downloads: 15005\n  homepage_uri: ''\n  name: libvorbisenc2\n  section: libs\n  version: 1.3.2-1ubuntu3\nlibvorbisenc2:i386:\n  description: The Vorbis General Audio Compression Codec (Encoder library)\n  downloads: 0\n  homepage_uri: ''\n  name: libvorbisenc2:i386\n  section: libs\n  version: 1.3.2-1ubuntu3\nlibvorbisfile3:\n  description: The Vorbis General Audio Compression Codec (High Level API)\n  downloads: 41932\n  homepage_uri: ''\n  name: libvorbisfile3\n  section: libs\n  version: 1.3.2-1ubuntu3\nlibvorbisfile3:i386:\n  description: The Vorbis General Audio Compression Codec (High Level API)\n  downloads: 0\n  homepage_uri: ''\n  name: libvorbisfile3:i386\n  section: libs\n  version: 1.3.2-1ubuntu3\nlibvpx1:\n  description: VP8 video codec (shared library)\n  downloads: 2840\n  homepage_uri: http://www.webmproject.org\n  name: libvpx1\n  section: libs\n  version: 1.0.0-1\nlibvte-2.90-9:\n  description: Terminal emulator widget for GTK+ 3.0 - runtime files\n  downloads: 6293\n  homepage_uri: ''\n  name: libvte-2.90-9\n  section: libs\n  version: 1:0.32.1-0ubuntu1\nlibvte-2.90-common:\n  description: Terminal emulator widget for GTK+ 3.0 - common files\n  downloads: 1568\n  homepage_uri: ''\n  name: libvte-2.90-common\n  section: libs\n  version: 1:0.32.1-0ubuntu1\nlibvte-common:\n  description: Terminal emulator widget for GTK+ 2.x - common files\n  downloads: 23330\n  homepage_uri: ''\n  name: libvte-common\n  section: libs\n  version: 1:0.28.2-3ubuntu2\nlibvte9:\n  description: Terminal emulator widget for GTK+ 2.0 - runtime files\n  downloads: 27115\n  homepage_uri: ''\n  name: libvte9\n  section: libs\n  version: 1:0.28.2-3ubuntu2\nlibvtk5.8:\n  description: Visualization Toolkit - A high level 3D visualization library - runtime\n  downloads: 116\n  homepage_uri: http://www.vtk.org/\n  name: libvtk5.8\n  section: universe/libs\n  version: 5.8.0-5\nlibwacom-common:\n  description: Wacom model feature query library (common files)\n  downloads: 5066\n  homepage_uri: ''\n  name: libwacom-common\n  section: libs\n  version: 0.4-1ubuntu1\nlibwacom2:\n  description: Wacom model feature query library\n  downloads: 5120\n  homepage_uri: ''\n  name: libwacom2\n  section: libs\n  version: 0.4-1ubuntu1\nlibwagon-java:\n  description: tools to manage Maven artifacts and deployment\n  downloads: 22\n  homepage_uri: http://maven.apache.org/wagon/\n  name: libwagon-java\n  section: universe/libs\n  version: 1.0.0-2ubuntu2\nlibwavpack1:\n  description: audio codec (lossy and lossless) - library\n  downloads: 8327\n  homepage_uri: http://www.wavpack.com\n  name: libwavpack1\n  section: libs\n  version: 4.60.1-2\nlibwavpack1:i386:\n  description: audio codec (lossy and lossless) - library\n  downloads: 0\n  homepage_uri: http://www.wavpack.com\n  name: libwavpack1:i386\n  section: libs\n  version: 4.60.1-2\nlibwbclient0:\n  description: Samba winbind client library\n  downloads: 43887\n  homepage_uri: http://www.samba.org\n  name: libwbclient0\n  section: libs\n  version: 2:3.6.3-2ubuntu2.2\nlibwebkitgtk-1.0-0:\n  description: Web content engine library for GTK+\n  downloads: 6789\n  homepage_uri: http://webkitgtk.org/\n  name: libwebkitgtk-1.0-0\n  section: libs\n  version: 1.8.0-0ubuntu2\nlibwebkitgtk-1.0-common:\n  description: Web content engine library for GTK+ - data files\n  downloads: 0\n  homepage_uri: http://webkitgtk.org/\n  name: libwebkitgtk-1.0-common\n  section: libs\n  version: 1.8.0-0ubuntu2\nlibwebkitgtk-3.0-0:\n  description: Web content engine library for GTK+\n  downloads: 9072\n  homepage_uri: http://webkitgtk.org/\n  name: libwebkitgtk-3.0-0\n  section: libs\n  version: 1.8.0-0ubuntu2\nlibwebkitgtk-3.0-common:\n  description: Web content engine library for GTK+ - data files\n  downloads: 0\n  homepage_uri: http://webkitgtk.org/\n  name: libwebkitgtk-3.0-common\n  section: libs\n  version: 1.8.0-0ubuntu2\nlibwind0-heimdal:\n  description: Heimdal Kerberos - stringprep implementation\n  downloads: 367\n  homepage_uri: http://www.h5l.org/\n  name: libwind0-heimdal\n  section: libs\n  version: 1.6~git20120311.dfsg.1-2\nlibwind0-heimdal:i386:\n  description: Heimdal Kerberos - stringprep implementation\n  downloads: 0\n  homepage_uri: http://www.h5l.org/\n  name: libwind0-heimdal:i386\n  section: libs\n  version: 1.6~git20120311.dfsg.1-2\nlibwmf0.2-7:\n  description: Windows metafile conversion library\n  downloads: 37381\n  homepage_uri: ''\n  name: libwmf0.2-7\n  section: libs\n  version: 0.2.8.4-10ubuntu1\nlibwmf0.2-7-gtk:\n  description: Windows metafile conversion library\n  downloads: 21\n  homepage_uri: ''\n  name: libwmf0.2-7-gtk\n  section: libs\n  version: 0.2.8.4-10ubuntu1\nlibwnck-3-0:\n  description: Window Navigator Construction Kit - runtime files\n  downloads: 729\n  homepage_uri: ''\n  name: libwnck-3-0\n  section: libs\n  version: 3.4.0-0ubuntu1\nlibwnck-3-common:\n  description: Window Navigator Construction Kit - common files\n  downloads: 74\n  homepage_uri: ''\n  name: libwnck-3-common\n  section: libs\n  version: 3.4.0-0ubuntu1\nlibwnck-common:\n  description: Window Navigator Construction Kit - common files\n  downloads: 5349\n  homepage_uri: ''\n  name: libwnck-common\n  section: libs\n  version: 1:2.30.7-0ubuntu1\nlibwnck22:\n  description: Window Navigator Construction Kit - runtime files\n  downloads: 8940\n  homepage_uri: ''\n  name: libwnck22\n  section: libs\n  version: 1:2.30.7-0ubuntu1\nlibwpd-0.9-9:\n  description: Library for handling WordPerfect documents (shared library)\n  downloads: 17\n  homepage_uri: ''\n  name: libwpd-0.9-9\n  section: libs\n  version: 0.9.4-1\nlibwpg-0.2-2:\n  description: WordPerfect graphics import/convert library (shared library)\n  downloads: 19\n  homepage_uri: ''\n  name: libwpg-0.2-2\n  section: libs\n  version: 0.2.1-1\nlibwps-0.2-2:\n  description: Works text file format import filter library (shared library)\n  downloads: 6\n  homepage_uri: ''\n  name: libwps-0.2-2\n  section: libs\n  version: 0.2.4-1\nlibwrap0:\n  description: Wietse Venema's TCP wrappers library\n  downloads: 99797\n  homepage_uri: ''\n  name: libwrap0\n  section: libs\n  version: 7.6.q-21\nlibwrap0:i386:\n  description: Wietse Venema's TCP wrappers library\n  downloads: 0\n  homepage_uri: ''\n  name: libwrap0:i386\n  section: libs\n  version: 7.6.q-21\nlibwww-perl:\n  description: simple and consistent interface to the world-wide web\n  downloads: 39604\n  homepage_uri: http://search.cpan.org/dist/libwww-perl/\n  name: libwww-perl\n  section: perl\n  version: 6.03-1\nlibwww-robotrules-perl:\n  description: database of robots.txt-derived permissions\n  downloads: 5254\n  homepage_uri: http://search.cpan.org/dist/WWW-RobotRules/\n  name: libwww-robotrules-perl\n  section: perl\n  version: 6.01-1\nlibwxbase2.8-0:\n  description: wxBase library (runtime) - non-GUI support classes of wxWidgets toolkit\n  downloads: 652\n  homepage_uri: http://www.wxwidgets.org/\n  name: libwxbase2.8-0\n  section: universe/libs\n  version: 2.8.12.1-6ubuntu2\nlibwxgtk2.8-0:\n  description: wxWidgets Cross-platform C++ GUI toolkit (GTK+ runtime)\n  downloads: 759\n  homepage_uri: http://www.wxwidgets.org/\n  name: libwxgtk2.8-0\n  section: universe/libs\n  version: 2.8.12.1-6ubuntu2\nlibx11-6:\n  description: X11 client-side library\n  downloads: 80016\n  homepage_uri: ''\n  name: libx11-6\n  section: libs\n  version: 2:1.4.99.1-0ubuntu2\nlibx11-6:i386:\n  description: X11 client-side library\n  downloads: 0\n  homepage_uri: ''\n  name: libx11-6:i386\n  section: libs\n  version: 2:1.4.99.1-0ubuntu2\nlibx11-data:\n  description: X11 client-side library\n  downloads: 76574\n  homepage_uri: ''\n  name: libx11-data\n  section: x11\n  version: 2:1.4.99.1-0ubuntu2\nlibx11-dev:\n  description: X11 client-side library (development headers)\n  downloads: 6788\n  homepage_uri: ''\n  name: libx11-dev\n  section: libdevel\n  version: 2:1.4.99.1-0ubuntu2\nlibx11-doc:\n  description: X11 client-side library (development documentation)\n  downloads: 0\n  homepage_uri: ''\n  name: libx11-doc\n  section: libdevel\n  version: 2:1.4.99.1-0ubuntu2\nlibx11-xcb1:\n  description: Xlib/XCB interface library\n  downloads: 33058\n  homepage_uri: ''\n  name: libx11-xcb1\n  section: libs\n  version: 2:1.4.99.1-0ubuntu2\nlibx11-xcb1:i386:\n  description: Xlib/XCB interface library\n  downloads: 0\n  homepage_uri: ''\n  name: libx11-xcb1:i386\n  section: libs\n  version: 2:1.4.99.1-0ubuntu2\nlibx86-1:\n  description: x86 real-mode library\n  downloads: 6098\n  homepage_uri: http://www.codon.org.uk/~mjg59/libx86/\n  name: libx86-1\n  section: libs\n  version: 1.1+ds1-7ubuntu1\nlibxalan2-java:\n  description: XSL Transformations (XSLT) processor in Java\n  downloads: 7172\n  homepage_uri: http://xml.apache.org/xalan-j/\n  name: libxalan2-java\n  section: libs\n  version: 2.7.1-7\nlibxapian22:\n  description: Search engine library\n  downloads: 24429\n  homepage_uri: http://xapian.org/\n  name: libxapian22\n  section: libs\n  version: 1.2.8-1\nlibxatracker1:\n  description: X acceleration library -- runtime\n  downloads: 0\n  homepage_uri: http://mesa3d.sourceforge.net/\n  name: libxatracker1\n  section: libs\n  version: 8.0.2-0ubuntu3.1\nlibxau-dev:\n  description: X11 authorisation library (development headers)\n  downloads: 4925\n  homepage_uri: ''\n  name: libxau-dev\n  section: libdevel\n  version: 1:1.0.6-4\nlibxau6:\n  description: X11 authorisation library\n  downloads: 60200\n  homepage_uri: ''\n  name: libxau6\n  section: libs\n  version: 1:1.0.6-4\nlibxau6:i386:\n  description: X11 authorisation library\n  downloads: 0\n  homepage_uri: ''\n  name: libxau6:i386\n  section: libs\n  version: 1:1.0.6-4\nlibxaw7:\n  description: X11 Athena Widget library\n  downloads: 47931\n  homepage_uri: ''\n  name: libxaw7\n  section: libs\n  version: 2:1.0.9-3ubuntu1\nlibxaw7:i386:\n  description: X11 Athena Widget library\n  downloads: 0\n  homepage_uri: ''\n  name: libxaw7:i386\n  section: libs\n  version: 2:1.0.9-3ubuntu1\nlibxbean-java:\n  description: plugin based Java application server\n  downloads: 16\n  homepage_uri: http://geronimo.apache.org/xbean/\n  name: libxbean-java\n  section: universe/libs\n  version: 3.7-4\nlibxcb-dri2-0:\n  description: X C Binding, dri2 extension\n  downloads: 14482\n  homepage_uri: http://xcb.freedesktop.org\n  name: libxcb-dri2-0\n  section: libs\n  version: 1.8.1-1\nlibxcb-glx0:\n  description: X C Binding, glx extension\n  downloads: 295\n  homepage_uri: http://xcb.freedesktop.org\n  name: libxcb-glx0\n  section: libs\n  version: 1.8.1-1\nlibxcb-glx0:i386:\n  description: X C Binding, glx extension\n  downloads: 0\n  homepage_uri: http://xcb.freedesktop.org\n  name: libxcb-glx0:i386\n  section: libs\n  version: 1.8.1-1\nlibxcb-render0:\n  description: X C Binding, render extension\n  downloads: 54187\n  homepage_uri: http://xcb.freedesktop.org\n  name: libxcb-render0\n  section: libs\n  version: 1.8.1-1\nlibxcb-render0-dev:\n  description: X C Binding, render extension, development files\n  downloads: 2736\n  homepage_uri: http://xcb.freedesktop.org\n  name: libxcb-render0-dev\n  section: libdevel\n  version: 1.8.1-1\nlibxcb-render0:i386:\n  description: X C Binding, render extension\n  downloads: 0\n  homepage_uri: http://xcb.freedesktop.org\n  name: libxcb-render0:i386\n  section: libs\n  version: 1.8.1-1\nlibxcb-shape0:\n  description: X C Binding, shape extension\n  downloads: 23089\n  homepage_uri: http://xcb.freedesktop.org\n  name: libxcb-shape0\n  section: libs\n  version: 1.8.1-1\nlibxcb-shm0:\n  description: X C Binding, shm extension\n  downloads: 37273\n  homepage_uri: http://xcb.freedesktop.org\n  name: libxcb-shm0\n  section: libs\n  version: 1.8.1-1\nlibxcb-shm0-dev:\n  description: X C Binding, shm extension, development files\n  downloads: 2045\n  homepage_uri: http://xcb.freedesktop.org\n  name: libxcb-shm0-dev\n  section: libdevel\n  version: 1.8.1-1\nlibxcb-shm0:i386:\n  description: X C Binding, shm extension\n  downloads: 0\n  homepage_uri: http://xcb.freedesktop.org\n  name: libxcb-shm0:i386\n  section: libs\n  version: 1.8.1-1\nlibxcb-util0:\n  description: utility libraries for X C Binding -- atom, aux and event\n  downloads: 8018\n  homepage_uri: http://xcb.freedesktop.org\n  name: libxcb-util0\n  section: libs\n  version: 0.3.8-2\nlibxcb1:\n  description: X C Binding\n  downloads: 75811\n  homepage_uri: http://xcb.freedesktop.org\n  name: libxcb1\n  section: libs\n  version: 1.8.1-1\nlibxcb1-dev:\n  description: X C Binding, development files\n  downloads: 4930\n  homepage_uri: http://xcb.freedesktop.org\n  name: libxcb1-dev\n  section: libdevel\n  version: 1.8.1-1\nlibxcb1:i386:\n  description: X C Binding\n  downloads: 0\n  homepage_uri: http://xcb.freedesktop.org\n  name: libxcb1:i386\n  section: libs\n  version: 1.8.1-1\nlibxcomp3:\n  description: NX X compression library\n  downloads: 79\n  homepage_uri: ''\n  name: libxcomp3\n  section: universe/x11\n  version: 3.5.0-2-0ubuntu1~precise\nlibxcompext3:\n  description: NX X11 protocol compression extensions library\n  downloads: 55\n  homepage_uri: ''\n  name: libxcompext3\n  section: x11\n  version: 1:3.5.0-9-2-1-2-0~ppa1~precise2\nlibxcomposite-dev:\n  description: X11 Composite extension library (development headers)\n  downloads: 2348\n  homepage_uri: ''\n  name: libxcomposite-dev\n  section: libdevel\n  version: 1:0.4.3-2build1\nlibxcomposite1:\n  description: X11 Composite extension library\n  downloads: 51862\n  homepage_uri: ''\n  name: libxcomposite1\n  section: libs\n  version: 1:0.4.3-2build1\nlibxcomposite1:i386:\n  description: X11 Composite extension library\n  downloads: 0\n  homepage_uri: ''\n  name: libxcomposite1:i386\n  section: libs\n  version: 1:0.4.3-2build1\nlibxcompshad3:\n  description: NX shadowing library\n  downloads: 55\n  homepage_uri: ''\n  name: libxcompshad3\n  section: universe/x11\n  version: 1:3.5.0-9-2-1-2-0~ppa1~precise2\nlibxcursor-dev:\n  description: X cursor management library (development files)\n  downloads: 2576\n  homepage_uri: ''\n  name: libxcursor-dev\n  section: libdevel\n  version: 1:1.1.12-1\nlibxcursor1:\n  description: X cursor management library\n  downloads: 33654\n  homepage_uri: ''\n  name: libxcursor1\n  section: libs\n  version: 1:1.1.12-1\nlibxcursor1:i386:\n  description: X cursor management library\n  downloads: 0\n  homepage_uri: ''\n  name: libxcursor1:i386\n  section: libs\n  version: 1:1.1.12-1\nlibxdamage-dev:\n  description: X11 damaged region extension library (development headers)\n  downloads: 2256\n  homepage_uri: ''\n  name: libxdamage-dev\n  section: libdevel\n  version: 1:1.1.3-2build1\nlibxdamage1:\n  description: X11 damaged region extension library\n  downloads: 7362\n  homepage_uri: ''\n  name: libxdamage1\n  section: libs\n  version: 1:1.1.3-2build1\nlibxdamage1:i386:\n  description: X11 damaged region extension library\n  downloads: 0\n  homepage_uri: ''\n  name: libxdamage1:i386\n  section: libs\n  version: 1:1.1.3-2build1\nlibxdmcp-dev:\n  description: X11 authorisation library (development headers)\n  downloads: 4435\n  homepage_uri: ''\n  name: libxdmcp-dev\n  section: libdevel\n  version: 1:1.1.0-4\nlibxdmcp6:\n  description: X11 Display Manager Control Protocol library\n  downloads: 39868\n  homepage_uri: ''\n  name: libxdmcp6\n  section: libs\n  version: 1:1.1.0-4\nlibxdmcp6:i386:\n  description: X11 Display Manager Control Protocol library\n  downloads: 0\n  homepage_uri: ''\n  name: libxdmcp6:i386\n  section: libs\n  version: 1:1.1.0-4\nlibxdot4:\n  description: rich set of graph drawing tools - xdot library\n  downloads: 3684\n  homepage_uri: http://www.graphviz.org/\n  name: libxdot4\n  section: libs\n  version: 2.26.3-10ubuntu1\nlibxenstore3.0:\n  description: Xenstore communications library for Xen\n  downloads: 4494\n  homepage_uri: ''\n  name: libxenstore3.0\n  section: libs\n  version: 4.1.2-2ubuntu2\nlibxerces2-java:\n  description: Validating XML parser for Java with DOM level 3 support\n  downloads: 9370\n  homepage_uri: http://xerces.apache.org/xerces2-j/\n  name: libxerces2-java\n  section: libs\n  version: 2.11.0-4\nlibxext-dev:\n  description: X11 miscellaneous extensions library (development headers)\n  downloads: 4591\n  homepage_uri: ''\n  name: libxext-dev\n  section: libdevel\n  version: 2:1.3.0-3build1\nlibxext6:\n  description: X11 miscellaneous extension library\n  downloads: 56111\n  homepage_uri: ''\n  name: libxext6\n  section: libs\n  version: 2:1.3.0-3build1\nlibxext6:i386:\n  description: X11 miscellaneous extension library\n  downloads: 0\n  homepage_uri: ''\n  name: libxext6:i386\n  section: libs\n  version: 2:1.3.0-3build1\nlibxfixes-dev:\n  description: X11 miscellaneous 'fixes' extension library (development headers)\n  downloads: 2659\n  homepage_uri: ''\n  name: libxfixes-dev\n  section: libdevel\n  version: 1:5.0-4ubuntu4\nlibxfixes3:\n  description: X11 miscellaneous 'fixes' extension library\n  downloads: 36634\n  homepage_uri: ''\n  name: libxfixes3\n  section: libs\n  version: 1:5.0-4ubuntu4\nlibxfixes3:i386:\n  description: X11 miscellaneous 'fixes' extension library\n  downloads: 0\n  homepage_uri: ''\n  name: libxfixes3:i386\n  section: libs\n  version: 1:5.0-4ubuntu4\nlibxfont1:\n  description: X11 font rasterisation library\n  downloads: 18569\n  homepage_uri: ''\n  name: libxfont1\n  section: libs\n  version: 1:1.4.4-1\nlibxft-dev:\n  description: FreeType-based font drawing library for X (development files)\n  downloads: 3744\n  homepage_uri: ''\n  name: libxft-dev\n  section: libdevel\n  version: 2.2.0-3ubuntu2\nlibxft2:\n  description: FreeType-based font drawing library for X\n  downloads: 54321\n  homepage_uri: ''\n  name: libxft2\n  section: libs\n  version: 2.2.0-3ubuntu2\nlibxft2:i386:\n  description: FreeType-based font drawing library for X\n  downloads: 0\n  homepage_uri: ''\n  name: libxft2:i386\n  section: libs\n  version: 2.2.0-3ubuntu2\nlibxi-dev:\n  description: X11 Input extension library (development headers)\n  downloads: 2720\n  homepage_uri: ''\n  name: libxi-dev\n  section: libdevel\n  version: 2:1.6.0-0ubuntu2\nlibxi6:\n  description: X11 Input extension library\n  downloads: 27373\n  homepage_uri: ''\n  name: libxi6\n  section: libs\n  version: 2:1.6.0-0ubuntu2\nlibxi6:i386:\n  description: X11 Input extension library\n  downloads: 0\n  homepage_uri: ''\n  name: libxi6:i386\n  section: libs\n  version: 2:1.6.0-0ubuntu2\nlibxinerama-dev:\n  description: X11 Xinerama extension library (development headers)\n  downloads: 2958\n  homepage_uri: ''\n  name: libxinerama-dev\n  section: libdevel\n  version: 2:1.1.1-3build1\nlibxinerama1:\n  description: X11 Xinerama extension library\n  downloads: 16279\n  homepage_uri: ''\n  name: libxinerama1\n  section: libs\n  version: 2:1.1.1-3build1\nlibxinerama1:i386:\n  description: X11 Xinerama extension library\n  downloads: 0\n  homepage_uri: ''\n  name: libxinerama1:i386\n  section: libs\n  version: 2:1.1.1-3build1\nlibxkbfile1:\n  description: X11 keyboard file manipulation library\n  downloads: 46629\n  homepage_uri: ''\n  name: libxkbfile1\n  section: libs\n  version: 1:1.0.7-1\nlibxklavier16:\n  description: X Keyboard Extension high-level API\n  downloads: 30355\n  homepage_uri: http://www.freedesktop.org/wiki/Software/LibXklavier\n  name: libxklavier16\n  section: libs\n  version: 5.2.1-1ubuntu1\nlibxml-commons-external-java:\n  description: XML Commons external code - DOM, SAX, and JAXP, etc\n  downloads: 1552\n  homepage_uri: http://xml.apache.org/commons/\n  name: libxml-commons-external-java\n  section: libs\n  version: 1.4.01-2\nlibxml-commons-resolver1.1-java:\n  description: XML entity and URI resolver library\n  downloads: 1269\n  homepage_uri: http://xml.apache.org/commons/\n  name: libxml-commons-resolver1.1-java\n  section: libs\n  version: 1.2-7\nlibxml-namespacesupport-perl:\n  description: Perl module for supporting simple generic namespaces\n  downloads: 7761\n  homepage_uri: ''\n  name: libxml-namespacesupport-perl\n  section: perl\n  version: 1.09-3\nlibxml-parser-perl:\n  description: Perl module for parsing XML files\n  downloads: 13643\n  homepage_uri: http://www.libexpat.org/\n  name: libxml-parser-perl\n  section: perl\n  version: 2.41-1build1\nlibxml-sax-base-perl:\n  description: base class for SAX drivers and filters\n  downloads: 3649\n  homepage_uri: http://search.cpan.org/dist/XML-SAX-Base/\n  name: libxml-sax-base-perl\n  section: perl\n  version: 1.07-1\nlibxml-sax-expat-perl:\n  description: Perl module for a SAX2 driver for Expat (XML::Parser)\n  downloads: 3318\n  homepage_uri: http://search.cpan.org/dist/XML-SAX-Expat/\n  name: libxml-sax-expat-perl\n  section: perl\n  version: 0.40-2\nlibxml-sax-perl:\n  description: Perl module for using and building Perl SAX2 XML processors\n  downloads: 6399\n  homepage_uri: http://search.cpan.org/dist/XML-SAX/\n  name: libxml-sax-perl\n  section: perl\n  version: 0.99+dfsg-1\nlibxml-simple-perl:\n  description: Perl module for reading and writing XML\n  downloads: 4698\n  homepage_uri: http://search.cpan.org/dist/XML-Simple/\n  name: libxml-simple-perl\n  section: perl\n  version: 2.18-3\nlibxml-twig-perl:\n  description: Perl module for processing huge XML documents in tree mode\n  downloads: 10124\n  homepage_uri: http://www.xmltwig.com/\n  name: libxml-twig-perl\n  section: perl\n  version: 1:3.39-1ubuntu1\nlibxml-writer-perl:\n  description: Perl module for writing XML documents\n  downloads: 671\n  homepage_uri: http://search.cpan.org/dist/XML-Writer/\n  name: libxml-writer-perl\n  section: universe/perl\n  version: 0.612-1\nlibxml-xpath-perl:\n  description: Perl module for processing XPath\n  downloads: 579\n  homepage_uri: http://search.cpan.org/dist/XML-XPath/\n  name: libxml-xpath-perl\n  section: perl\n  version: 1.13-7\nlibxml2:\n  description: GNOME XML library\n  downloads: 82994\n  homepage_uri: http://xmlsoft.org/\n  name: libxml2\n  section: libs\n  version: 2.7.8.dfsg-5.1ubuntu4.1\nlibxml2-dev:\n  description: Development files for the GNOME XML library\n  downloads: 3497\n  homepage_uri: http://xmlsoft.org/\n  name: libxml2-dev\n  section: libdevel\n  version: 2.7.8.dfsg-5.1ubuntu4.1\nlibxml2-utils:\n  description: XML utilities\n  downloads: 4698\n  homepage_uri: http://xmlsoft.org/\n  name: libxml2-utils\n  section: text\n  version: 2.7.8.dfsg-5.1ubuntu4.1\nlibxml2:i386:\n  description: GNOME XML library\n  downloads: 0\n  homepage_uri: http://xmlsoft.org/\n  name: libxml2:i386\n  section: libs\n  version: 2.7.8.dfsg-5.1ubuntu4.1\nlibxmlgraphics-commons-java:\n  description: reusable components used by Batik and FOP\n  downloads: 45\n  homepage_uri: http://xmlgraphics.apache.org/commons/\n  name: libxmlgraphics-commons-java\n  section: libs\n  version: 1.4.dfsg-4ubuntu1\nlibxmlrpc-core-c3:\n  description: lightweight RPC library based on XML and HTTP [C runtime libraries]\n  downloads: 32\n  homepage_uri: http://xmlrpc-c.sourceforge.net\n  name: libxmlrpc-core-c3\n  section: libs\n  version: 1.16.33-3.1ubuntu5\nlibxmu6:\n  description: X11 miscellaneous utility library\n  downloads: 36776\n  homepage_uri: ''\n  name: libxmu6\n  section: libs\n  version: 2:1.1.0-3\nlibxmu6:i386:\n  description: X11 miscellaneous utility library\n  downloads: 0\n  homepage_uri: ''\n  name: libxmu6:i386\n  section: libs\n  version: 2:1.1.0-3\nlibxmuu1:\n  description: X11 miscellaneous micro-utility library\n  downloads: 39923\n  homepage_uri: ''\n  name: libxmuu1\n  section: libs\n  version: 2:1.1.0-3\nlibxom-java:\n  description: A new XML object model for Java\n  downloads: 49\n  homepage_uri: http://www.xom.nu/\n  name: libxom-java\n  section: libs\n  version: 1.2.1-3\nlibxp6:\n  description: X Printing Extension (Xprint) client library\n  downloads: 617\n  homepage_uri: ''\n  name: libxp6\n  section: x11\n  version: 1:1.0.1-2\nlibxp6:i386:\n  description: X Printing Extension (Xprint) client library\n  downloads: 0\n  homepage_uri: ''\n  name: libxp6:i386\n  section: x11\n  version: 1:1.0.1-2\nlibxpm-dev:\n  description: X11 pixmap library (development headers)\n  downloads: 1066\n  homepage_uri: ''\n  name: libxpm-dev\n  section: libdevel\n  version: 1:3.5.9-4\nlibxpm4:\n  description: X11 pixmap library\n  downloads: 57681\n  homepage_uri: ''\n  name: libxpm4\n  section: libs\n  version: 1:3.5.9-4\nlibxpm4:i386:\n  description: X11 pixmap library\n  downloads: 0\n  homepage_uri: ''\n  name: libxpm4:i386\n  section: libs\n  version: 1:3.5.9-4\nlibxpp2-java:\n  description: XML pull parser library for java V2\n  downloads: 214\n  homepage_uri: http://www.extreme.indiana.edu/xgws/xsoap/xpp/xpp2/\n  name: libxpp2-java\n  section: libs\n  version: 2.1.10-7\nlibxpp3-java:\n  description: XML pull parser library for java\n  downloads: 605\n  homepage_uri: http://www.extreme.indiana.edu/xgws/xsoap/xpp\n  name: libxpp3-java\n  section: libs\n  version: 1.1.4c-2\nlibxrandr-dev:\n  description: X11 RandR extension library (development headers)\n  downloads: 2706\n  homepage_uri: ''\n  name: libxrandr-dev\n  section: libdevel\n  version: 2:1.3.2-2\nlibxrandr2:\n  description: X11 RandR extension library\n  downloads: 38647\n  homepage_uri: ''\n  name: libxrandr2\n  section: libs\n  version: 2:1.3.2-2\nlibxrandr2:i386:\n  description: X11 RandR extension library\n  downloads: 0\n  homepage_uri: ''\n  name: libxrandr2:i386\n  section: libs\n  version: 2:1.3.2-2\nlibxrender-dev:\n  description: X Rendering Extension client library (development files)\n  downloads: 4211\n  homepage_uri: ''\n  name: libxrender-dev\n  section: libdevel\n  version: 1:0.9.6-2build1\nlibxrender1:\n  description: X Rendering Extension client library\n  downloads: 56401\n  homepage_uri: ''\n  name: libxrender1\n  section: libs\n  version: 1:0.9.6-2build1\nlibxrender1:i386:\n  description: X Rendering Extension client library\n  downloads: 0\n  homepage_uri: ''\n  name: libxrender1:i386\n  section: libs\n  version: 1:0.9.6-2build1\nlibxres1:\n  description: X11 Resource extension library\n  downloads: 5965\n  homepage_uri: ''\n  name: libxres1\n  section: libs\n  version: 2:1.0.5-1\nlibxslt1-dev:\n  description: XSLT 1.0 processing library - development kit\n  downloads: 968\n  homepage_uri: http://xmlsoft.org/xslt/\n  name: libxslt1-dev\n  section: libdevel\n  version: 1.1.26-8ubuntu1\nlibxslt1.1:\n  description: XSLT 1.0 processing library - runtime library\n  downloads: 17000\n  homepage_uri: http://xmlsoft.org/xslt/\n  name: libxslt1.1\n  section: libs\n  version: 1.1.26-8ubuntu1\nlibxslt1.1:i386:\n  description: XSLT 1.0 processing library - runtime library\n  downloads: 0\n  homepage_uri: http://xmlsoft.org/xslt/\n  name: libxslt1.1:i386\n  section: libs\n  version: 1.1.26-8ubuntu1\nlibxss-dev:\n  description: X11 Screen Saver extension library (development headers)\n  downloads: 798\n  homepage_uri: ''\n  name: libxss-dev\n  section: libdevel\n  version: 1:1.2.1-2\nlibxss1:\n  description: X11 Screen Saver extension library\n  downloads: 10752\n  homepage_uri: ''\n  name: libxss1\n  section: libs\n  version: 1:1.2.1-2\nlibxss1:i386:\n  description: X11 Screen Saver extension library\n  downloads: 0\n  homepage_uri: ''\n  name: libxss1:i386\n  section: libs\n  version: 1:1.2.1-2\nlibxt-dev:\n  description: X11 toolkit intrinsics library (development headers)\n  downloads: 3984\n  homepage_uri: ''\n  name: libxt-dev\n  section: libdevel\n  version: 1:1.1.1-2build1\nlibxt6:\n  description: X11 toolkit intrinsics library\n  downloads: 48065\n  homepage_uri: ''\n  name: libxt6\n  section: libs\n  version: 1:1.1.1-2build1\nlibxt6:i386:\n  description: X11 toolkit intrinsics library\n  downloads: 0\n  homepage_uri: ''\n  name: libxt6:i386\n  section: libs\n  version: 1:1.1.1-2build1\nlibxtst6:\n  description: X11 Testing -- Record extension library\n  downloads: 37093\n  homepage_uri: ''\n  name: libxtst6\n  section: libs\n  version: 2:1.2.0-4\nlibxtst6:i386:\n  description: X11 Testing -- Record extension library\n  downloads: 0\n  homepage_uri: ''\n  name: libxtst6:i386\n  section: libs\n  version: 2:1.2.0-4\nlibxv1:\n  description: X11 Video extension library\n  downloads: 30977\n  homepage_uri: ''\n  name: libxv1\n  section: libs\n  version: 2:1.0.6-2build1\nlibxv1:i386:\n  description: X11 Video extension library\n  downloads: 0\n  homepage_uri: ''\n  name: libxv1:i386\n  section: libs\n  version: 2:1.0.6-2build1\nlibxvmc1:\n  description: X11 Video extension library\n  downloads: 20191\n  homepage_uri: ''\n  name: libxvmc1\n  section: libs\n  version: 2:1.0.6-1ubuntu2\nlibxxf86dga1:\n  description: X11 Direct Graphics Access extension library\n  downloads: 884\n  homepage_uri: ''\n  name: libxxf86dga1\n  section: libs\n  version: 2:1.1.2-1\nlibxxf86vm1:\n  description: X11 XFree86 video mode extension library\n  downloads: 36992\n  homepage_uri: ''\n  name: libxxf86vm1\n  section: libs\n  version: 1:1.1.1-2build1\nlibxxf86vm1:i386:\n  description: X11 XFree86 video mode extension library\n  downloads: 0\n  homepage_uri: ''\n  name: libxxf86vm1:i386\n  section: libs\n  version: 1:1.1.1-2build1\nlibyajl1:\n  description: Yet Another JSON Library\n  downloads: 432\n  homepage_uri: http://lloyd.github.com/yajl/\n  name: libyajl1\n  section: libs\n  version: 1.0.12-2\nlibyaml-0-2:\n  description: Fast YAML 1.1 parser and emitter library\n  downloads: 2551\n  homepage_uri: http://pyyaml.org/wiki/LibYAML\n  name: libyaml-0-2\n  section: libs\n  version: 0.1.4-2\nlibyaml-libyaml-perl:\n  description: Perl interface to libyaml, a YAML implementation\n  downloads: 175\n  homepage_uri: http://search.cpan.org/dist/YAML-LibYAML/\n  name: libyaml-libyaml-perl\n  section: perl\n  version: 0.38-2\nlibyaml-perl:\n  description: YAML Ain't Markup Language\n  downloads: 1151\n  homepage_uri: http://search.cpan.org/dist/YAML/\n  name: libyaml-perl\n  section: perl\n  version: 0.77-1\nlibyaml-tiny-perl:\n  description: Perl module for reading and writing YAML files\n  downloads: 3633\n  homepage_uri: http://search.cpan.org/~adamk/YAML-Tiny/\n  name: libyaml-tiny-perl\n  section: perl\n  version: 1.50-1\nlibyelp0:\n  description: Library for the GNOME help browser\n  downloads: 3015\n  homepage_uri: http://live.gnome.org/Yelp\n  name: libyelp0\n  section: libs\n  version: 3.4.1-0ubuntu1\nlibzeitgeist-1.0-1:\n  description: library to access Zeitgeist - shared library\n  downloads: 737\n  homepage_uri: https://launchpad.net/libzeitgeist\n  name: libzeitgeist-1.0-1\n  section: libs\n  version: 0.3.18-1ubuntu1\nlibzephyr4:\n  description: Project Athena's notification service - non-Kerberos libraries\n  downloads: 104\n  homepage_uri: ''\n  name: libzephyr4\n  section: libs\n  version: 3.0.1-1\nlibzip-dev:\n  description: library for reading, creating, and modifying zip archives (development)\n  downloads: 74\n  homepage_uri: http://www.nih.at/libzip/\n  name: libzip-dev\n  section: libdevel\n  version: 0.10-1ubuntu1\nlibzip2:\n  description: library for reading, creating, and modifying zip archives (runtime)\n  downloads: 482\n  homepage_uri: http://www.nih.at/libzip/\n  name: libzip2\n  section: libs\n  version: 0.10-1ubuntu1\nlibzmq-dev:\n  description: ZeroMQ lightweight messaging kernel (development libraries and header\n    files)\n  downloads: 36\n  homepage_uri: http://www.zeromq.org/\n  name: libzmq-dev\n  section: universe/libdevel\n  version: 2.1.11-1ubuntu1\nlibzmq1:\n  description: ZeroMQ lightweight messaging kernel (shared library)\n  downloads: 192\n  homepage_uri: http://www.zeromq.org/\n  name: libzmq1\n  section: universe/libs\n  version: 2.1.11-1ubuntu1\nlight-themes:\n  description: Light Themes (Ambiance and Radiance)\n  downloads: 0\n  homepage_uri: https://launchpad.net/light-themes\n  name: light-themes\n  section: gnome\n  version: 0.1.9.1-0ubuntu1\nlightdm:\n  description: Display Manager\n  downloads: 1096\n  homepage_uri: https://launchpad.net/lightdm\n  name: lightdm\n  section: x11\n  version: 1.2.1-0ubuntu1\nlintian:\n  description: Debian package checker\n  downloads: 1270\n  homepage_uri: ''\n  name: lintian\n  section: devel\n  version: 2.5.6ubuntu0.1\nlinux-firmware:\n  description: Firmware for Linux kernel drivers\n  downloads: 22\n  homepage_uri: ''\n  name: linux-firmware\n  section: misc\n  version: '1.79'\nlinux-headers-3.2.0-25:\n  description: Header files related to Linux kernel version 3.2.0\n  downloads: 0\n  homepage_uri: ''\n  name: linux-headers-3.2.0-25\n  section: devel\n  version: 3.2.0-25.40\nlinux-headers-3.2.0-25-generic:\n  description: Linux kernel headers for version 3.2.0 on 64 bit x86 SMP\n  downloads: 0\n  homepage_uri: ''\n  name: linux-headers-3.2.0-25-generic\n  section: devel\n  version: 3.2.0-25.40\nlinux-headers-generic:\n  description: Generic Linux kernel headers\n  downloads: 0\n  homepage_uri: ''\n  name: linux-headers-generic\n  section: devel\n  version: 3.2.0.25.27\nlinux-image-3.2.0-23-virtual:\n  description: Linux kernel image for version 3.2.0 on 64 bit x86 Virtual Guests\n  downloads: 0\n  homepage_uri: ''\n  name: linux-image-3.2.0-23-virtual\n  section: kernel\n  version: 3.2.0-23.36\nlinux-image-virtual:\n  description: Linux kernel image for virtual machines\n  downloads: 0\n  homepage_uri: ''\n  name: linux-image-virtual\n  section: metapackages\n  version: 3.2.0.23.25\nlinux-libc-dev:\n  description: Linux Kernel Headers for development\n  downloads: 34885\n  homepage_uri: ''\n  name: linux-libc-dev\n  section: devel\n  version: 3.2.0-25.40\nlinux-sound-base:\n  description: base package for ALSA and OSS sound systems\n  downloads: 37280\n  homepage_uri: http://www.alsa-project.org/\n  name: linux-sound-base\n  section: sound\n  version: 1.0.25+dfsg-0ubuntu1\nlinux-virtual:\n  description: Complete Linux kernel for virtual machines\n  downloads: 0\n  homepage_uri: ''\n  name: linux-virtual\n  section: metapackages\n  version: 3.2.0.23.25\nlksctp-tools:\n  description: user-space access to Linux Kernel SCTP - commandline tools\n  downloads: 267\n  homepage_uri: http://lksctp.sf.net/\n  name: lksctp-tools\n  section: net\n  version: 1.0.11+dfsg-2\nlmodern:\n  description: scalable PostScript and OpenType fonts based on Computer Modern\n  downloads: 1972\n  homepage_uri: http://www.gust.org.pl/projects/e-foundry/latin-modern/\n  name: lmodern\n  section: tex\n  version: 2.004.1-3.1ubuntu1\nlocales:\n  description: common files for locale support\n  downloads: 28005\n  homepage_uri: ''\n  name: locales\n  section: libs\n  version: 2.13+git20120306-3\nlockfile-progs:\n  description: Programs for locking and unlocking files and mailboxes\n  downloads: 44611\n  homepage_uri: ''\n  name: lockfile-progs\n  section: misc\n  version: 0.1.16\nlogin:\n  description: system login tools\n  downloads: 110986\n  homepage_uri: http://pkg-shadow.alioth.debian.org/\n  name: login\n  section: admin\n  version: 1:4.1.4.2+svn3283-3ubuntu5\nlogrotate:\n  description: Log rotation utility\n  downloads: 110453\n  homepage_uri: ''\n  name: logrotate\n  section: admin\n  version: 3.7.8-6ubuntu5\nlsb-base:\n  description: Linux Standard Base 4.0 init script functionality\n  downloads: 110574\n  homepage_uri: http://www.linux-foundation.org/en/LSB\n  name: lsb-base\n  section: misc\n  version: 4.0-0ubuntu20\nlsb-release:\n  description: Linux Standard Base version reporting utility\n  downloads: 55189\n  homepage_uri: http://www.linux-foundation.org/en/LSB\n  name: lsb-release\n  section: misc\n  version: 4.0-0ubuntu20\nlshw:\n  description: information about hardware configuration\n  downloads: 1433\n  homepage_uri: ''\n  name: lshw\n  section: utils\n  version: 02.15-2\nlsof:\n  description: List open files\n  downloads: 19260\n  homepage_uri: ''\n  name: lsof\n  section: utils\n  version: 4.81.dfsg.1-1build1\nltrace:\n  description: Tracks runtime library calls in dynamically linked programs\n  downloads: 399\n  homepage_uri: ''\n  name: ltrace\n  section: utils\n  version: 0.5.3-2.1ubuntu2\nluatex:\n  description: next generation TeX engine\n  downloads: 9397\n  homepage_uri: http://www.luatex.org\n  name: luatex\n  section: tex\n  version: 0.70.1-1ubuntu1\nlvm2:\n  description: The Linux Logical Volume Manager\n  downloads: 17090\n  homepage_uri: http://sources.redhat.com/lvm2/\n  name: lvm2\n  section: admin\n  version: 2.02.66-4ubuntu7.1\nlynx:\n  description: Text-mode WWW Browser (transitional package)\n  downloads: 530\n  homepage_uri: http://lynx.isc.org/\n  name: lynx\n  section: web\n  version: 2.8.8dev.9-2\nlynx-cur:\n  description: Text-mode WWW Browser with NLS support (development version)\n  downloads: 8607\n  homepage_uri: http://lynx.isc.org/\n  name: lynx-cur\n  section: web\n  version: 2.8.8dev.9-2\nm17n-contrib:\n  description: multilingual text processing library - contributed database\n  downloads: 0\n  homepage_uri: http://www.m17n.org/\n  name: m17n-contrib\n  section: text\n  version: 1.1.13-1\nm17n-db:\n  description: multilingual text processing library - database\n  downloads: 1137\n  homepage_uri: http://www.m17n.org/\n  name: m17n-db\n  section: text\n  version: 1.6.3-1\nm4:\n  description: a macro processing language\n  downloads: 12727\n  homepage_uri: http://www.gnu.org/software/m4/\n  name: m4\n  section: interpreters\n  version: 1.4.16-2ubuntu1\nmafft:\n  description: Multiple alignment program for amino acid or nucleotide sequences\n  downloads: 31\n  homepage_uri: http://align.bmr.kyushu-u.ac.jp/mafft/software/\n  name: mafft\n  section: universe/science\n  version: 6.850-1\nmahjongg:\n  description: classic Eastern tile game for GNOME\n  downloads: 480\n  homepage_uri: http://live.gnome.org/GnomeGames\n  name: mahjongg\n  section: games\n  version: 1:3.4.1-0ubuntu2\nmake:\n  description: An utility for Directing compilation.\n  downloads: 24471\n  homepage_uri: http://www.gnu.org/software/make/\n  name: make\n  section: devel\n  version: 3.81-8.1ubuntu1\nmakedev:\n  description: creates device files in /dev\n  downloads: 4954\n  homepage_uri: ''\n  name: makedev\n  section: admin\n  version: 2.3.1-89ubuntu2\nman-db:\n  description: on-line manual pager\n  downloads: 105783\n  homepage_uri: http://man-db.nongnu.org/\n  name: man-db\n  section: doc\n  version: 2.6.1-2\nmanpages:\n  description: Manual pages about using a GNU/Linux system\n  downloads: 1\n  homepage_uri: http://www.kernel.org/doc/man-pages/\n  name: manpages\n  section: doc\n  version: 3.35-0.1ubuntu1\nmanpages-dev:\n  description: Manual pages about using GNU/Linux for development\n  downloads: 1\n  homepage_uri: http://www.kernel.org/doc/man-pages/\n  name: manpages-dev\n  section: doc\n  version: 3.35-0.1ubuntu1\nmaq:\n  description: maps short fixed-length polymorphic DNA sequence reads to reference\n    sequences\n  downloads: 21\n  homepage_uri: http://maq.sourceforge.net/\n  name: maq\n  section: science\n  version: 0.7.1-4\nmaven2:\n  description: Java software project management and comprehension tool\n  downloads: 369\n  homepage_uri: http://maven.apache.org/\n  name: maven2\n  section: universe/devel\n  version: 2.2.1-10\nmawk:\n  description: a pattern scanning and text processing language\n  downloads: 78500\n  homepage_uri: ''\n  name: mawk\n  section: utils\n  version: 1.3.3-17\nmayavi2:\n  description: scientific visualization package for 2-D and 3-D data\n  downloads: 103\n  homepage_uri: http://code.enthought.com/projects/mayavi/\n  name: mayavi2\n  section: universe/science\n  version: 4.0.0-3build1\nmc:\n  description: Midnight Commander - a powerful file manager\n  downloads: 12038\n  homepage_uri: http://www.midnight-commander.org\n  name: mc\n  section: universe/utils\n  version: 3:4.8.1-2ubuntu1\nmc-data:\n  description: Midnight Commander - a powerful file manager -- data files\n  downloads: 1928\n  homepage_uri: http://www.midnight-commander.org\n  name: mc-data\n  section: universe/utils\n  version: 3:4.8.1-2ubuntu1\nmcl:\n  description: Markov Cluster algorithm\n  downloads: 17\n  homepage_uri: http://micans.org/mcl/\n  name: mcl\n  section: math\n  version: 1:11-294-1\nmdadm:\n  description: tool to administer Linux MD arrays (software RAID)\n  downloads: 12014\n  homepage_uri: http://neil.brown.name/blog/mdadm\n  name: mdadm\n  section: admin\n  version: 3.2.3-2ubuntu1\nmedia-player-info:\n  description: Media player identification files\n  downloads: 10387\n  homepage_uri: http://www.freedesktop.org/software/media-player-info/\n  name: media-player-info\n  section: admin\n  version: 16-1\nmemtest86+:\n  description: thorough real-mode memory tester\n  downloads: 479\n  homepage_uri: http://www.memtest.org/\n  name: memtest86+\n  section: misc\n  version: 4.20-1.1ubuntu1\nmenu:\n  description: generates programs menu for all menu-aware applications\n  downloads: 34177\n  homepage_uri: ''\n  name: menu\n  section: universe/admin\n  version: 2.1.46ubuntu1\nmercurial:\n  description: easy-to-use, scalable distributed version control system\n  downloads: 2401\n  homepage_uri: http://mercurial.selenic.com/\n  name: mercurial\n  section: universe/devel\n  version: 2.0.2-1ubuntu1\nmercurial-common:\n  description: easy-to-use, scalable distributed version control system (common files)\n  downloads: 2451\n  homepage_uri: http://mercurial.selenic.com/\n  name: mercurial-common\n  section: universe/devel\n  version: 2.0.2-1ubuntu1\nmesa-common-dev:\n  description: Developer documentation for Mesa\n  downloads: 2670\n  homepage_uri: http://mesa3d.sourceforge.net/\n  name: mesa-common-dev\n  section: devel\n  version: 8.0.2-0ubuntu3.1\nmetacity:\n  description: lightweight GTK+ window manager\n  downloads: 32103\n  homepage_uri: ''\n  name: metacity\n  section: x11\n  version: 1:2.34.1-1ubuntu11\nmetacity-common:\n  description: shared files for the Metacity window manager\n  downloads: 10\n  homepage_uri: ''\n  name: metacity-common\n  section: misc\n  version: 1:2.34.1-1ubuntu11\nmime-support:\n  description: MIME files 'mime.types' & 'mailcap', and support programs\n  downloads: 92780\n  homepage_uri: ''\n  name: mime-support\n  section: net\n  version: 3.51-1ubuntu1\nmlocate:\n  description: quickly find files on the filesystem based on their name\n  downloads: 88285\n  homepage_uri: http://carolina.mff.cuni.cz/~trmac/blog/mlocate\n  name: mlocate\n  section: utils\n  version: 0.23.1-1ubuntu2\nmobile-broadband-provider-info:\n  description: database of mobile broadband service providers\n  downloads: 2\n  homepage_uri: http://live.gnome.org/NetworkManager/MobileBroadband/ServiceProviders\n  name: mobile-broadband-provider-info\n  section: admin\n  version: 20120410-0ubuntu1\nmodemmanager:\n  description: D-Bus service for managing modems\n  downloads: 32673\n  homepage_uri: ''\n  name: modemmanager\n  section: net\n  version: 0.5.2.0-0ubuntu2\nmodule-init-tools:\n  description: tools for managing Linux kernel modules\n  downloads: 88323\n  homepage_uri: http://www.kerneltools.org/\n  name: module-init-tools\n  section: admin\n  version: 3.16-1ubuntu2\nmongodb-10gen:\n  description: An object/document-oriented database\n  downloads: 136\n  homepage_uri: http://www.mongodb.org\n  name: mongodb-10gen\n  section: devel\n  version: 2.0.6\nmount:\n  description: Tools for mounting and manipulating filesystems\n  downloads: 110417\n  homepage_uri: http://userweb.kernel.org/~kzak/util-linux/\n  name: mount\n  section: admin\n  version: 2.20.1-1ubuntu3\nmountall:\n  description: filesystem mounting tool\n  downloads: 2\n  homepage_uri: ''\n  name: mountall\n  section: admin\n  version: '2.36'\nmousetweaks:\n  description: mouse accessibility enhancements for the GNOME desktop\n  downloads: 19462\n  homepage_uri: http://live.gnome.org/Mousetweaks/Home\n  name: mousetweaks\n  section: gnome\n  version: 3.4.1-0ubuntu1\nmpi-default-dev:\n  description: Standard MPI development files (metapackage)\n  downloads: 204\n  homepage_uri: ''\n  name: mpi-default-dev\n  section: universe/libdevel\n  version: 1.0.1\nmrbayes:\n  description: Bayesian Inference of Phylogeny\n  downloads: 8\n  homepage_uri: http://mrbayes.csit.fsu.edu/\n  name: mrbayes\n  section: math\n  version: 3.1.2-2\nmscompress:\n  description: Microsoft \"compress.exe/expand.exe\" compatible (de)compressor\n  downloads: 1442\n  homepage_uri: ''\n  name: mscompress\n  section: otherosfs\n  version: 0.3-3.1\nmsr-tools:\n  description: Utilities for modifying MSRs from userspace\n  downloads: 7\n  homepage_uri: ''\n  name: msr-tools\n  section: admin\n  version: 1.2-3\nmtasc:\n  description: ActionScript 2 to Flash (SWF) compiler\n  downloads: 19\n  homepage_uri: http://www.mtasc.org\n  name: mtasc\n  section: universe/devel\n  version: 1.14-2build2\nmtools:\n  description: Tools for manipulating MSDOS files\n  downloads: 6177\n  homepage_uri: http://www.mtools.linux.lu/\n  name: mtools\n  section: otherosfs\n  version: 4.0.12-1\nmtr-tiny:\n  description: Full screen ncurses traceroute tool\n  downloads: 1966\n  homepage_uri: http://www.bitwizard.nl/mtr/\n  name: mtr-tiny\n  section: net\n  version: 0.80-1ubuntu1\nmultiarch-support:\n  description: Transitional package to ensure multiarch compatibility\n  downloads: 1\n  homepage_uri: http://www.eglibc.org\n  name: multiarch-support\n  section: libs\n  version: 2.15-0ubuntu10\nmummer:\n  description: Efficient sequence alignment of full genomes\n  downloads: 26\n  homepage_uri: http://mummer.sourceforge.net/\n  name: mummer\n  section: science\n  version: 3.22~dfsg-2\nmuscle:\n  description: Multiple alignment program of protein sequences\n  downloads: 37\n  homepage_uri: http://www.drive5.com/muscle/\n  name: muscle\n  section: science\n  version: 1:3.8.31-1\nmysql-client:\n  description: MySQL database client (metapackage depending on the latest version)\n  downloads: 1336\n  homepage_uri: http://dev.mysql.com/\n  name: mysql-client\n  section: database\n  version: 5.5.24-0ubuntu0.12.04.1\nmysql-client-5.5:\n  description: MySQL database client binaries\n  downloads: 4207\n  homepage_uri: http://dev.mysql.com/\n  name: mysql-client-5.5\n  section: database\n  version: 5.5.24-0ubuntu0.12.04.1\nmysql-client-core-5.5:\n  description: MySQL database core client binaries\n  downloads: 33\n  homepage_uri: http://dev.mysql.com/\n  name: mysql-client-core-5.5\n  section: database\n  version: 5.5.24-0ubuntu0.12.04.1\nmysql-common:\n  description: MySQL database common files, e.g. /etc/mysql/my.cnf\n  downloads: 27558\n  homepage_uri: http://dev.mysql.com/\n  name: mysql-common\n  section: database\n  version: 5.5.24-0ubuntu0.12.04.1\nmysql-server:\n  description: MySQL database server (metapackage depending on the latest version)\n  downloads: 42\n  homepage_uri: http://dev.mysql.com/\n  name: mysql-server\n  section: database\n  version: 5.5.24-0ubuntu0.12.04.1\nmysql-server-5.5:\n  description: MySQL database server binaries and system database setup\n  downloads: 2790\n  homepage_uri: http://dev.mysql.com/\n  name: mysql-server-5.5\n  section: database\n  version: 5.5.24-0ubuntu0.12.04.1\nmysql-server-core-5.5:\n  description: MySQL database server binaries\n  downloads: 3847\n  homepage_uri: http://dev.mysql.com/\n  name: mysql-server-core-5.5\n  section: database\n  version: 5.5.24-0ubuntu0.12.04.1\nnano:\n  description: small, friendly text editor inspired by Pico\n  downloads: 23959\n  homepage_uri: http://www.nano-editor.org/\n  name: nano\n  section: editors\n  version: 2.2.6-1\nnautilus:\n  description: file manager and graphical shell for GNOME\n  downloads: 30390\n  homepage_uri: http://www.gnome.org/projects/nautilus/\n  name: nautilus\n  section: gnome\n  version: 1:3.4.2-0ubuntu2\nnautilus-data:\n  description: data files for nautilus\n  downloads: 32249\n  homepage_uri: http://www.gnome.org/projects/nautilus/\n  name: nautilus-data\n  section: gnome\n  version: 1:3.4.2-0ubuntu2\nnautilus-sendto:\n  description: integrates Evolution and Pidgin into the Nautilus file manager\n  downloads: 19647\n  homepage_uri: ''\n  name: nautilus-sendto\n  section: gnome\n  version: 3.0.1-2ubuntu2\nnautilus-sendto-empathy:\n  description: GNOME multi-protocol chat and call client (nautilus-sendto plugin)\n  downloads: 1583\n  homepage_uri: http://live.gnome.org/Empathy\n  name: nautilus-sendto-empathy\n  section: gnome\n  version: 3.4.2-0ubuntu1\nnautilus-share:\n  description: Nautilus extension to share folder using Samba\n  downloads: 1213\n  homepage_uri: http://gentoo.ovibes.net/nautilus-share/\n  name: nautilus-share\n  section: gnome\n  version: 0.7.3-1ubuntu2\nncbi-blast+:\n  description: next generation suite of BLAST sequence search tools\n  downloads: 22\n  homepage_uri: http://www.ncbi.nlm.nih.gov/IEB/ToolBox/CPP_DOC/\n  name: ncbi-blast+\n  section: science\n  version: 2.2.25-7\nncbi-blast+-legacy:\n  description: NCBI Blast legacy call script\n  downloads: 2\n  homepage_uri: http://www.ncbi.nlm.nih.gov/IEB/ToolBox/CPP_DOC/\n  name: ncbi-blast+-legacy\n  section: science\n  version: 2.2.25-7\nncbi-data:\n  description: Platform-independent data for the NCBI toolkit\n  downloads: 154\n  homepage_uri: http://www.ncbi.nlm.nih.gov/IEB/ToolBox/\n  name: ncbi-data\n  section: libs\n  version: 6.1.20110713-3ubuntu2\nncbi-tools-bin:\n  description: NCBI libraries for biology applications (text-based utilities)\n  downloads: 26\n  homepage_uri: http://www.ncbi.nlm.nih.gov/IEB/ToolBox/\n  name: ncbi-tools-bin\n  section: science\n  version: 6.1.20110713-3ubuntu2\nncbi-tools-x11:\n  description: NCBI libraries for biology applications (X-based utilities)\n  downloads: 120\n  homepage_uri: http://www.ncbi.nlm.nih.gov/IEB/ToolBox/\n  name: ncbi-tools-x11\n  section: science\n  version: 6.1.20110713-3ubuntu2\nncftp:\n  description: User-friendly and well-featured FTP client\n  downloads: 1288\n  homepage_uri: http://www.ncftpd.com/ncftp/\n  name: ncftp\n  section: universe/net\n  version: 2:3.2.5-1\nncurses-base:\n  description: basic terminal type definitions\n  downloads: 90113\n  homepage_uri: http://invisible-island.net/ncurses/\n  name: ncurses-base\n  section: utils\n  version: 5.9-4\nncurses-bin:\n  description: terminal-related programs and man pages\n  downloads: 108834\n  homepage_uri: http://invisible-island.net/ncurses/\n  name: ncurses-bin\n  section: utils\n  version: 5.9-4\nnet-tools:\n  description: The NET-3 networking toolkit\n  downloads: 86718\n  homepage_uri: http://net-tools.berlios.de/\n  name: net-tools\n  section: net\n  version: 1.60-24.1ubuntu2\nnetbase:\n  description: Basic TCP/IP networking system\n  downloads: 90312\n  homepage_uri: ''\n  name: netbase\n  section: admin\n  version: 4.47ubuntu1\nnetcat-openbsd:\n  description: TCP/IP swiss army knife\n  downloads: 1157\n  homepage_uri: ''\n  name: netcat-openbsd\n  section: net\n  version: 1.89-4ubuntu1\nnetpbm:\n  description: Graphics conversion tools between image formats\n  downloads: 5843\n  homepage_uri: http://netpbm.alioth.debian.org\n  name: netpbm\n  section: graphics\n  version: 2:10.0-15\nnetwork-manager:\n  description: network management framework (daemon and userspace tools)\n  downloads: 37981\n  homepage_uri: http://www.gnome.org/projects/NetworkManager/\n  name: network-manager\n  section: net\n  version: 0.9.4.0-0ubuntu4.1\nnetwork-manager-gnome:\n  description: network management framework (GNOME frontend)\n  downloads: 27132\n  homepage_uri: http://www.gnome.org/projects/NetworkManager/\n  name: network-manager-gnome\n  section: net\n  version: 0.9.4.1-0ubuntu2\nnetwork-manager-pptp:\n  description: network management framework (PPTP plugin core)\n  downloads: 894\n  homepage_uri: http://www.gnome.org/projects/NetworkManager/\n  name: network-manager-pptp\n  section: net\n  version: 0.9.4.0-0ubuntu1\nnetwork-manager-pptp-gnome:\n  description: network management framework (PPTP plugin GNOME GUI)\n  downloads: 650\n  homepage_uri: http://www.gnome.org/projects/NetworkManager/\n  name: network-manager-pptp-gnome\n  section: net\n  version: 0.9.4.0-0ubuntu1\nnfs-common:\n  description: NFS support files common to client and server\n  downloads: 59352\n  homepage_uri: http://nfs.sourceforge.net/\n  name: nfs-common\n  section: net\n  version: 1:1.2.5-3ubuntu3\nnfs-kernel-server:\n  description: support for NFS kernel server\n  downloads: 12883\n  homepage_uri: http://nfs.sourceforge.net/\n  name: nfs-kernel-server\n  section: net\n  version: 1:1.2.5-3ubuntu3\nnjplot:\n  description: phylogenetic tree drawing program\n  downloads: 120\n  homepage_uri: http://pbil.univ-lyon1.fr/software/njplot.html\n  name: njplot\n  section: science\n  version: 2.3-4\nnotification-daemon:\n  description: daemon for displaying passive pop-up notifications\n  downloads: 32826\n  homepage_uri: http://www.galago-project.org/specs/notification/index.php\n  name: notification-daemon\n  section: x11\n  version: 0.7.3-1\nnotify-osd:\n  description: daemon that displays passive pop-up notifications\n  downloads: 300\n  homepage_uri: https://launchpad.net/notify-osd\n  name: notify-osd\n  section: x11\n  version: 0.9.34-0ubuntu2\nnotify-osd-icons:\n  description: Notify-OSD icons\n  downloads: 0\n  homepage_uri: ''\n  name: notify-osd-icons\n  section: x11\n  version: '0.7'\nntfs-3g:\n  description: read/write NTFS driver for FUSE\n  downloads: 18895\n  homepage_uri: http://www.tuxera.com/community/ntfs-3g-advanced/\n  name: ntfs-3g\n  section: otherosfs\n  version: 1:2012.1.15AR.1-1ubuntu1\nntpdate:\n  description: client for setting system time from NTP servers\n  downloads: 23778\n  homepage_uri: http://support.ntp.org/\n  name: ntpdate\n  section: net\n  version: 1:4.2.6.p3+dfsg-1ubuntu3.1\nnux-tools:\n  description: Visual rendering toolkit for real-time applications - tools\n  downloads: 0\n  homepage_uri: http://launchpad.net/nux\n  name: nux-tools\n  section: x11\n  version: 2.12.0-0ubuntu1\nnvidia-common:\n  description: Find obsolete NVIDIA drivers\n  downloads: 19\n  homepage_uri: ''\n  name: nvidia-common\n  section: x11\n  version: 1:0.2.44\nnx-common:\n  description: NX server common package.\n  downloads: 0\n  homepage_uri: ''\n  name: nx-common\n  section: x11\n  version: 0.7.3.zgit.120322.977c28d-0~ppa11\nnxagent:\n  description: X server for remote access\n  downloads: 173\n  homepage_uri: ''\n  name: nxagent\n  section: x11\n  version: 1:3.5.0-9-2-1-2-0~ppa1~precise2\nobex-data-server:\n  description: D-Bus service for OBEX client and server side functionality\n  downloads: 29398\n  homepage_uri: http://wiki.muiline.com/obex-data-server\n  name: obex-data-server\n  section: admin\n  version: 0.4.6-0ubuntu1\nobexd-client:\n  description: D-Bus OBEX client\n  downloads: 27934\n  homepage_uri: http://www.bluez.org\n  name: obexd-client\n  section: admin\n  version: 0.44-0ubuntu1\noctave3.2:\n  description: GNU Octave language for numerical computations (3.2 branch)\n  downloads: 664\n  homepage_uri: http://www.octave.org/\n  name: octave3.2\n  section: universe/math\n  version: 3.2.4-12\noctave3.2-common:\n  description: architecture-independent files for octave3.2\n  downloads: 0\n  homepage_uri: http://www.octave.org/\n  name: octave3.2-common\n  section: universe/devel\n  version: 3.2.4-12\nodbcinst:\n  description: Helper program for accessing odbc ini files\n  downloads: 17237\n  homepage_uri: http://www.unixodbc.org/\n  name: odbcinst\n  section: libs\n  version: 2.2.14p2-5ubuntu3\nodbcinst1debian2:\n  description: Support library for accessing odbc ini files\n  downloads: 17275\n  homepage_uri: http://www.unixodbc.org/\n  name: odbcinst1debian2\n  section: libs\n  version: 2.2.14p2-5ubuntu3\nodbcinst1debian2:i386:\n  description: Support library for accessing odbc ini files\n  downloads: 0\n  homepage_uri: http://www.unixodbc.org/\n  name: odbcinst1debian2:i386\n  section: libs\n  version: 2.2.14p2-5ubuntu3\nonboard:\n  description: Simple On-screen Keyboard\n  downloads: 6\n  homepage_uri: https://launchpad.net/onboard\n  name: onboard\n  section: gnome\n  version: 0.97.0-0ubuntu3\noneconf:\n  description: synchronize your configuration data over the network\n  downloads: 0\n  homepage_uri: ''\n  name: oneconf\n  section: python\n  version: 0.2.8.1\nopenjdk-6-jdk:\n  description: OpenJDK Development Kit (JDK)\n  downloads: 1485\n  homepage_uri: http://openjdk.java.net/\n  name: openjdk-6-jdk\n  section: devel\n  version: 6b24-1.11.1-4ubuntu3\nopenjdk-6-jre:\n  description: OpenJDK Java runtime, using Hotspot JIT\n  downloads: 9734\n  homepage_uri: http://openjdk.java.net/\n  name: openjdk-6-jre\n  section: interpreters\n  version: 6b24-1.11.1-4ubuntu3\nopenjdk-6-jre-headless:\n  description: OpenJDK Java runtime, using Hotspot JIT (headless)\n  downloads: 14227\n  homepage_uri: http://openjdk.java.net/\n  name: openjdk-6-jre-headless\n  section: interpreters\n  version: 6b24-1.11.1-4ubuntu3\nopenjdk-6-jre-lib:\n  description: OpenJDK Java runtime (architecture independent libraries)\n  downloads: 12439\n  homepage_uri: http://openjdk.java.net/\n  name: openjdk-6-jre-lib\n  section: interpreters\n  version: 6b24-1.11.1-4ubuntu3\nopenjdk-7-jdk:\n  description: OpenJDK Development Kit (JDK)\n  downloads: 59\n  homepage_uri: http://openjdk.java.net/\n  name: openjdk-7-jdk\n  section: universe/java\n  version: 7~u3-2.1.1~pre1-1ubuntu3\nopenjdk-7-jre:\n  description: OpenJDK Java runtime, using Hotspot JIT\n  downloads: 307\n  homepage_uri: http://openjdk.java.net/\n  name: openjdk-7-jre\n  section: universe/java\n  version: 7~u3-2.1.1~pre1-1ubuntu3\nopenjdk-7-jre-headless:\n  description: OpenJDK Java runtime, using Hotspot JIT (headless)\n  downloads: 373\n  homepage_uri: http://openjdk.java.net/\n  name: openjdk-7-jre-headless\n  section: universe/java\n  version: 7~u3-2.1.1~pre1-1ubuntu3\nopenjdk-7-jre-lib:\n  description: OpenJDK Java runtime (architecture independent libraries)\n  downloads: 369\n  homepage_uri: http://openjdk.java.net/\n  name: openjdk-7-jre-lib\n  section: universe/java\n  version: 7~u3-2.1.1~pre1-1ubuntu3\nopenmpi-bin:\n  description: high performance message passing library -- binaries\n  downloads: 789\n  homepage_uri: http://www.open-mpi.org/\n  name: openmpi-bin\n  section: universe/net\n  version: 1.4.3-2.1ubuntu3\nopenmpi-checkpoint:\n  description: high performance message passing library -- checkpoint support\n  downloads: 130\n  homepage_uri: http://www.open-mpi.org/\n  name: openmpi-checkpoint\n  section: universe/net\n  version: 1.4.3-2.1ubuntu3\nopenmpi-common:\n  description: high performance message passing library -- common files\n  downloads: 1252\n  homepage_uri: http://www.open-mpi.org/\n  name: openmpi-common\n  section: universe/net\n  version: 1.4.3-2.1ubuntu3\nopenprinting-ppds:\n  description: OpenPrinting printer support - PostScript PPD files\n  downloads: 2364\n  homepage_uri: http://www.openprinting.org/\n  name: openprinting-ppds\n  section: text\n  version: 20120322-0ubuntu1\nopenssh-client:\n  description: secure shell (SSH) client, for secure access to remote machines\n  downloads: 67813\n  homepage_uri: http://www.openssh.org/\n  name: openssh-client\n  section: net\n  version: 1:5.9p1-5ubuntu1\nopenssh-server:\n  description: secure shell (SSH) server, for secure access from remote machines\n  downloads: 82562\n  homepage_uri: http://www.openssh.org/\n  name: openssh-server\n  section: net\n  version: 1:5.9p1-5ubuntu1\nopenssl:\n  description: Secure Socket Layer (SSL) binary and related cryptographic tools\n  downloads: 90115\n  homepage_uri: ''\n  name: openssl\n  section: utils\n  version: 1.0.1-4ubuntu5.2\norbit2:\n  description: a CORBA ORB\n  downloads: 367\n  homepage_uri: ''\n  name: orbit2\n  section: devel\n  version: 1:2.14.19-0.1ubuntu1\nos-prober:\n  description: utility to detect other OSes on a set of drives\n  downloads: 18333\n  homepage_uri: ''\n  name: os-prober\n  section: utils\n  version: 1.51ubuntu3\noss-compat:\n  description: Open Sound System (OSS) compatibility package\n  downloads: 12880\n  homepage_uri: ''\n  name: oss-compat\n  section: universe/sound\n  version: '1'\noverlay-scrollbar:\n  description: Scrollbar overlayed widget\n  downloads: 0\n  homepage_uri: http://launchpad.net/ayatana-scrollbar\n  name: overlay-scrollbar\n  section: libs\n  version: 0.2.16-0ubuntu1\nparrot:\n  description: virtual machine for dynamic languages\n  downloads: 23\n  homepage_uri: http://parrot.org\n  name: parrot\n  section: universe/interpreters\n  version: 3.6.0-1\nparrot-devel:\n  description: Parrot development tools\n  downloads: 18\n  homepage_uri: http://parrot.org\n  name: parrot-devel\n  section: universe/devel\n  version: 3.6.0-1\nparrot-minimal:\n  description: minimal install of the Parrot VM\n  downloads: 30\n  homepage_uri: http://parrot.org\n  name: parrot-minimal\n  section: universe/interpreters\n  version: 3.6.0-1\nparted:\n  description: disk partition manipulator\n  downloads: 1565\n  homepage_uri: http://www.gnu.org/software/parted\n  name: parted\n  section: admin\n  version: 2.3-8ubuntu5\npasswd:\n  description: change and administer password and group data\n  downloads: 95504\n  homepage_uri: http://pkg-shadow.alioth.debian.org/\n  name: passwd\n  section: admin\n  version: 1:4.1.4.2+svn3283-3ubuntu5\npatch:\n  description: Apply a diff file to an original\n  downloads: 16773\n  homepage_uri: ''\n  name: patch\n  section: utils\n  version: 2.6.1-3\npatchutils:\n  description: Utilities to work with patches\n  downloads: 1938\n  homepage_uri: http://cyberelk.net/tim/patchutils/index.html\n  name: patchutils\n  section: text\n  version: 0.3.2-1.1\npbuilder:\n  description: personal package builder for Debian packages\n  downloads: 635\n  homepage_uri: ''\n  name: pbuilder\n  section: devel\n  version: 0.208ubuntu1\npciutils:\n  description: Linux PCI Utilities\n  downloads: 19276\n  homepage_uri: http://atrey.karlin.mff.cuni.cz/~mj/pciutils.shtml\n  name: pciutils\n  section: admin\n  version: 1:3.1.8-2ubuntu5\npcmciautils:\n  description: PCMCIA utilities for Linux 2.6\n  downloads: 5559\n  homepage_uri: http://www.kernel.org/pub/linux/utils/kernel/pcmcia/pcmcia.html\n  name: pcmciautils\n  section: admin\n  version: 018-6\npdfjam:\n  description: collection of PDF document handling utilities\n  downloads: 171\n  homepage_uri: ''\n  name: pdfjam\n  section: universe/text\n  version: 2.05-2\nperl:\n  description: Larry Wall's Practical Extraction and Report Language\n  downloads: 96842\n  homepage_uri: ''\n  name: perl\n  section: perl\n  version: 5.14.2-6ubuntu2\nperl-base:\n  description: minimal Perl system\n  downloads: 118139\n  homepage_uri: ''\n  name: perl-base\n  section: perl\n  version: 5.14.2-6ubuntu2\nperl-doc:\n  description: Perl documentation\n  downloads: 1353\n  homepage_uri: ''\n  name: perl-doc\n  section: doc\n  version: 5.14.2-6ubuntu2\nperl-modules:\n  description: Core Perl modules\n  downloads: 92959\n  homepage_uri: ''\n  name: perl-modules\n  section: perl\n  version: 5.14.2-6ubuntu2\nperl-tk:\n  description: Perl module providing the Tk graphics library\n  downloads: 1051\n  homepage_uri: http://www.perltk.org/\n  name: perl-tk\n  section: universe/perl\n  version: 1:804.029-1.1ubuntu2\npgf:\n  description: TeX Portable Graphic Format\n  downloads: 104\n  homepage_uri: ''\n  name: pgf\n  section: tex\n  version: 2.10-1\nphylip:\n  description: package of programs for inferring phylogenies\n  downloads: 12\n  homepage_uri: http://evolution.genetics.washington.edu/phylip.html\n  name: phylip\n  section: non-free/science\n  version: 1:3.69-1\nphyml:\n  description: Phylogenetic estimation using Maximum Likelihood\n  downloads: 32\n  homepage_uri: http://www.atgc-montpellier.fr/phyml\n  name: phyml\n  section: science\n  version: 2:20110919-1\npinyin-database:\n  description: PinYin database used by ibus-pinyin\n  downloads: 5\n  homepage_uri: http://code.google.com/p/ibus\n  name: pinyin-database\n  section: universe/utils\n  version: 1.2.99-3\npkg-config:\n  description: manage compile and link flags for libraries\n  downloads: 11087\n  homepage_uri: http://pkg-config.freedesktop.org\n  name: pkg-config\n  section: devel\n  version: 0.26-1ubuntu1\nplink:\n  description: whole-genome association analysis toolset\n  downloads: 22\n  homepage_uri: http://pngu.mgh.harvard.edu/~purcell/plink/\n  name: plink\n  section: universe/science\n  version: 1.07-1\nplymouth:\n  description: graphical boot animation and logger - main package\n  downloads: 1105\n  homepage_uri: ''\n  name: plymouth\n  section: x11\n  version: 0.8.2-2ubuntu30\nplymouth-label:\n  description: graphical boot animation and logger - label control\n  downloads: 20\n  homepage_uri: ''\n  name: plymouth-label\n  section: x11\n  version: 0.8.2-2ubuntu30\nplymouth-theme-ubuntu-logo:\n  description: graphical boot animation and logger - ubuntu-logo theme\n  downloads: 19\n  homepage_uri: ''\n  name: plymouth-theme-ubuntu-logo\n  section: x11\n  version: 0.8.2-2ubuntu30\nplymouth-theme-ubuntu-text:\n  description: graphical boot animation and logger - ubuntu-logo theme\n  downloads: 2\n  homepage_uri: ''\n  name: plymouth-theme-ubuntu-text\n  section: x11\n  version: 0.8.2-2ubuntu30\npm-utils:\n  description: utilities and scripts for power management\n  downloads: 45158\n  homepage_uri: http://pm-utils.freedesktop.org/\n  name: pm-utils\n  section: admin\n  version: 1.4.1-9\npo-debconf:\n  description: tool for managing templates file translations with gettext\n  downloads: 7107\n  homepage_uri: ''\n  name: po-debconf\n  section: devel\n  version: 1.0.16+nmu2ubuntu1\npolicykit-1:\n  description: framework for managing administrative policies and privileges\n  downloads: 41365\n  homepage_uri: http://hal.freedesktop.org/docs/PolicyKit/\n  name: policykit-1\n  section: admin\n  version: 0.104-1ubuntu1\npolicykit-1-gnome:\n  description: GNOME authentication agent for PolicyKit-1\n  downloads: 32793\n  homepage_uri: http://hal.freedesktop.org/docs/PolicyKit-gnome/\n  name: policykit-1-gnome\n  section: gnome\n  version: 0.105-1ubuntu3\npolicykit-desktop-privileges:\n  description: run common desktop actions without password\n  downloads: 0\n  homepage_uri: ''\n  name: policykit-desktop-privileges\n  section: admin\n  version: '0.10'\npoppler-utils:\n  description: PDF utilities (based on Poppler)\n  downloads: 9981\n  homepage_uri: http://poppler.freedesktop.org/\n  name: poppler-utils\n  section: utils\n  version: 0.18.4-1ubuntu2\npopularity-contest:\n  description: Vote for your favourite packages automatically\n  downloads: 110453\n  homepage_uri: http://popcon.ubuntu.com/\n  name: popularity-contest\n  section: misc\n  version: 1.53ubuntu1\npostfix:\n  description: High-performance mail transport agent\n  downloads: 21151\n  homepage_uri: http://www.postfix.org\n  name: postfix\n  section: mail\n  version: 2.9.1-5\npostgresql:\n  description: object-relational SQL database (supported version)\n  downloads: 88\n  homepage_uri: ''\n  name: postgresql\n  section: database\n  version: 9.1+129\npostgresql-8.4:\n  description: object-relational SQL database, version 8.4 server\n  downloads: 6325\n  homepage_uri: http://www.postgresql.org/\n  name: postgresql-8.4\n  section: universe/database\n  version: 8.4.11-1\npostgresql-9.1:\n  description: object-relational SQL database, version 9.1 server\n  downloads: 2092\n  homepage_uri: http://www.postgresql.org/\n  name: postgresql-9.1\n  section: database\n  version: 9.1.4-0ubuntu12.04\npostgresql-client:\n  description: front-end programs for PostgreSQL (supported version)\n  downloads: 280\n  homepage_uri: ''\n  name: postgresql-client\n  section: database\n  version: 9.1+129\npostgresql-client-8.4:\n  description: front-end programs for PostgreSQL 8.4\n  downloads: 7112\n  homepage_uri: http://www.postgresql.org/\n  name: postgresql-client-8.4\n  section: universe/database\n  version: 8.4.11-1\npostgresql-client-9.1:\n  description: front-end programs for PostgreSQL 9.1\n  downloads: 2248\n  homepage_uri: http://www.postgresql.org/\n  name: postgresql-client-9.1\n  section: database\n  version: 9.1.4-0ubuntu12.04\npostgresql-client-common:\n  description: manager for multiple PostgreSQL client versions\n  downloads: 13611\n  homepage_uri: ''\n  name: postgresql-client-common\n  section: misc\n  version: '129'\npostgresql-common:\n  description: PostgreSQL database-cluster manager\n  downloads: 12529\n  homepage_uri: ''\n  name: postgresql-common\n  section: misc\n  version: '129'\npostgresql-plperl-8.4:\n  description: PL/Perl procedural language for PostgreSQL 8.4\n  downloads: 27\n  homepage_uri: http://www.postgresql.org/\n  name: postgresql-plperl-8.4\n  section: universe/database\n  version: 8.4.11-1\npostgresql-plpython-8.4:\n  description: PL/Python procedural language for PostgreSQL 8.4\n  downloads: 20\n  homepage_uri: http://www.postgresql.org/\n  name: postgresql-plpython-8.4\n  section: universe/database\n  version: 8.4.11-1\npostgresql-server-dev-9.1:\n  description: development files for PostgreSQL 9.1 server-side programming\n  downloads: 36\n  homepage_uri: http://www.postgresql.org/\n  name: postgresql-server-dev-9.1\n  section: libdevel\n  version: 9.1.4-0ubuntu12.04\npowermgmt-base:\n  description: Common utils and configs for power management\n  downloads: 57470\n  homepage_uri: ''\n  name: powermgmt-base\n  section: utils\n  version: '1.31'\nppp:\n  description: Point-to-Point Protocol (PPP) - daemon\n  downloads: 8073\n  homepage_uri: ''\n  name: ppp\n  section: admin\n  version: 2.4.5-5ubuntu1\npppconfig:\n  description: A text menu based utility for configuring ppp\n  downloads: 184\n  homepage_uri: ''\n  name: pppconfig\n  section: admin\n  version: 2.3.18+nmu3ubuntu1\npppoeconf:\n  description: configures PPPoE/ADSL connections\n  downloads: 315\n  homepage_uri: ''\n  name: pppoeconf\n  section: net\n  version: 1.20ubuntu1\npptp-linux:\n  description: Point-to-Point Tunneling Protocol (PPTP) Client\n  downloads: 1254\n  homepage_uri: ''\n  name: pptp-linux\n  section: net\n  version: 1.7.2-6\npreview-latex-style:\n  description: extraction of elements from LaTeX documents as graphics\n  downloads: 3559\n  homepage_uri: http://www.gnu.org/software/auctex/preview-latex.html\n  name: preview-latex-style\n  section: tex\n  version: 11.86-2ubuntu1\nprimer3:\n  description: Tool to design flanking oligo nucleotides for DNA amplification\n  downloads: 35\n  homepage_uri: http://primer3.sourceforge.net\n  name: primer3\n  section: science\n  version: 2.2.3-1\nprinter-driver-c2esp:\n  description: printer driver for Kodak ESP AiO color inkjet Series\n  downloads: 195\n  homepage_uri: http://cupsdriverkodak.sf.net/\n  name: printer-driver-c2esp\n  section: text\n  version: 23-1\nprinter-driver-foo2zjs:\n  description: printer driver for ZjStream-based printers\n  downloads: 9013\n  homepage_uri: http://foo2zjs.rkkda.com/\n  name: printer-driver-foo2zjs\n  section: text\n  version: 20111202dfsg0-1ubuntu1\nprinter-driver-gutenprint:\n  description: printer drivers for CUPS\n  downloads: 7282\n  homepage_uri: ''\n  name: printer-driver-gutenprint\n  section: graphics\n  version: 5.2.8~pre1-0ubuntu2\nprinter-driver-hpcups:\n  description: HP Linux Printing and Imaging - CUPS Raster driver (hpcups)\n  downloads: 1067\n  homepage_uri: http://hplipopensource.com/hplip-web/index.html\n  name: printer-driver-hpcups\n  section: text\n  version: 3.12.2-1ubuntu3\nprinter-driver-hpijs:\n  description: HP Linux Printing and Imaging - gs IJS driver (hpijs)\n  downloads: 1096\n  homepage_uri: http://hplipopensource.com/hplip-web/index.html\n  name: printer-driver-hpijs\n  section: text\n  version: 3.12.2-1ubuntu3\nprinter-driver-min12xxw:\n  description: printer driver for KonicaMinolta PagePro 1[234]xxW\n  downloads: 1376\n  homepage_uri: http://www.hinterbergen.de/mala/min12xxw/\n  name: printer-driver-min12xxw\n  section: text\n  version: 0.0.9-6ubuntu1\nprinter-driver-pnm2ppa:\n  description: printer driver for HP-GDI printers\n  downloads: 1341\n  homepage_uri: http://pnm2ppa.sf.net/\n  name: printer-driver-pnm2ppa\n  section: text\n  version: 1.13+nondbs-0ubuntu1\nprinter-driver-ptouch:\n  description: printer driver Brother P-touch label printers\n  downloads: 5686\n  homepage_uri: http://www.diku.dk/hjemmesider/ansatte/panic/P-touch/\n  name: printer-driver-ptouch\n  section: text\n  version: 1.3-3ubuntu0.1\nprinter-driver-pxljr:\n  description: printer driver for HP Color LaserJet 35xx/36xx\n  downloads: 5858\n  homepage_uri: http://sourceforge.net/projects/hp-pxl-jetready/\n  name: printer-driver-pxljr\n  section: text\n  version: 1.3+repack0-2\nprinter-driver-sag-gdi:\n  description: printer driver for Ricoh Aficio SP 1000s/SP 1100s\n  downloads: 190\n  homepage_uri: http://www.openprinting.org/driver/rastertosag-gdi/\n  name: printer-driver-sag-gdi\n  section: text\n  version: 0.1-3\nprinter-driver-splix:\n  description: Driver for Samsung and Xerox SPL2 and SPLc laser printers\n  downloads: 329\n  homepage_uri: http://splix.ap2c.org/\n  name: printer-driver-splix\n  section: text\n  version: 2.0.0+svn300-1.1ubuntu2\nprobcons:\n  description: PROBabilistic CONSistency-based multiple sequence alignment\n  downloads: 25\n  homepage_uri: http://probcons.stanford.edu/\n  name: probcons\n  section: universe/science\n  version: 1.12-8\nprocps:\n  description: /proc file system utilities\n  downloads: 102900\n  homepage_uri: http://procps.sf.net/\n  name: procps\n  section: admin\n  version: 1:3.2.8-11ubuntu6\nprosper:\n  description: LaTeX class for writing transparencies\n  downloads: 0\n  homepage_uri: ''\n  name: prosper\n  section: tex\n  version: 1.00.4+cvs.2007.05.01-4\nprotobuf-compiler:\n  description: compiler for protocol buffer definition files\n  downloads: 142\n  homepage_uri: http://code.google.com/p/protobuf/\n  name: protobuf-compiler\n  section: devel\n  version: 2.4.1-1ubuntu2\nprover9:\n  description: theorem prover and countermodel generator\n  downloads: 5\n  homepage_uri: http://www.cs.unm.edu/~mccune/mace4/\n  name: prover9\n  section: universe/math\n  version: 0.0.200902a-2\nps2eps:\n  description: convert PostScript to EPS (Encapsulated PostScript) files\n  downloads: 1481\n  homepage_uri: http://tm.uka.de/~bless/ps2eps\n  name: ps2eps\n  section: text\n  version: 1.68-1\npsmisc:\n  description: utilities that use the proc file system\n  downloads: 82235\n  homepage_uri: http://psmisc.sf.net/\n  name: psmisc\n  section: admin\n  version: 22.15-2ubuntu1\npsutils:\n  description: PostScript document handling utilities\n  downloads: 2573\n  homepage_uri: http://www.tardis.ed.ac.uk/~ajcd/psutils/\n  name: psutils\n  section: text\n  version: 1.17-31\npulseaudio:\n  description: PulseAudio sound server\n  downloads: 16800\n  homepage_uri: http://www.pulseaudio.org\n  name: pulseaudio\n  section: sound\n  version: 1:1.1-0ubuntu15.1\npulseaudio-module-bluetooth:\n  description: Bluetooth module for PulseAudio sound server\n  downloads: 881\n  homepage_uri: http://www.pulseaudio.org\n  name: pulseaudio-module-bluetooth\n  section: sound\n  version: 1:1.1-0ubuntu15.1\npulseaudio-module-gconf:\n  description: GConf module for PulseAudio sound server\n  downloads: 2195\n  homepage_uri: http://www.pulseaudio.org\n  name: pulseaudio-module-gconf\n  section: sound\n  version: 1:1.1-0ubuntu15.1\npulseaudio-module-x11:\n  description: X11 module for PulseAudio sound server\n  downloads: 13497\n  homepage_uri: http://www.pulseaudio.org\n  name: pulseaudio-module-x11\n  section: sound\n  version: 1:1.1-0ubuntu15.1\npulseaudio-utils:\n  description: Command line tools for the PulseAudio sound server\n  downloads: 14931\n  homepage_uri: http://www.pulseaudio.org\n  name: pulseaudio-utils\n  section: sound\n  version: 1:1.1-0ubuntu15.1\npvm:\n  description: Parallel Virtual Machine - binaries\n  downloads: 261\n  homepage_uri: ''\n  name: pvm\n  section: universe/devel\n  version: 3.4.5-12.3\npython:\n  description: interactive high-level object-oriented language (default version)\n  downloads: 85476\n  homepage_uri: http://www.python.org/\n  name: python\n  section: python\n  version: 2.7.3-0ubuntu2\npython-appindicator:\n  description: Python bindings for libappindicator\n  downloads: 22\n  homepage_uri: https://launchpad.net/libappindicator\n  name: python-appindicator\n  section: python\n  version: 0.4.92-0ubuntu1\npython-apport:\n  description: apport crash report handling library\n  downloads: 35\n  homepage_uri: https://wiki.ubuntu.com/Apport\n  name: python-apport\n  section: python\n  version: 2.0.1-0ubuntu8\npython-apptools:\n  description: ETS Application Tools\n  downloads: 193\n  homepage_uri: http://pypi.python.org/pypi/apptools\n  name: python-apptools\n  section: universe/python\n  version: 4.0.0-1build1\npython-apt:\n  description: Python interface to libapt-pkg\n  downloads: 65417\n  homepage_uri: ''\n  name: python-apt\n  section: python\n  version: 0.8.3ubuntu7\npython-apt-common:\n  description: Python interface to libapt-pkg (locales)\n  downloads: 1\n  homepage_uri: ''\n  name: python-apt-common\n  section: python\n  version: 0.8.3ubuntu7\npython-aptdaemon:\n  description: Python module for the server and client of aptdaemon\n  downloads: 20325\n  homepage_uri: https://launchpad.net/aptdaemon\n  name: python-aptdaemon\n  section: python\n  version: 0.43+bzr805-0ubuntu1\npython-aptdaemon.gtk3widgets:\n  description: Python GTK+ 3 widgets to run an aptdaemon client\n  downloads: 1585\n  homepage_uri: https://launchpad.net/aptdaemon\n  name: python-aptdaemon.gtk3widgets\n  section: python\n  version: 0.43+bzr805-0ubuntu1\npython-aptdaemon.pkcompat:\n  description: PackageKit compatibilty for AptDaemon\n  downloads: 5\n  homepage_uri: https://launchpad.net/aptdaemon\n  name: python-aptdaemon.pkcompat\n  section: python\n  version: 0.43+bzr805-0ubuntu1\npython-boto:\n  description: Python interface to Amazon's Web Services\n  downloads: 102\n  homepage_uri: http://code.google.com/p/boto/\n  name: python-boto\n  section: python\n  version: 2.2.2-0ubuntu2\npython-brlapi:\n  description: Python bindings for BrlAPI\n  downloads: 2122\n  homepage_uri: http://mielke.cc/brltty/\n  name: python-brlapi\n  section: python\n  version: 4.3-1ubuntu5\npython-bzrlib:\n  description: distributed version control system - python library\n  downloads: 547\n  homepage_uri: http://bazaar-vcs.org\n  name: python-bzrlib\n  section: python\n  version: 2.5.0-2ubuntu2\npython-cairo:\n  description: Python bindings for the Cairo vector graphics library\n  downloads: 38358\n  homepage_uri: http://cairographics.org/pycairo/\n  name: python-cairo\n  section: python\n  version: 1.8.8-1ubuntu3\npython-central:\n  description: register and build utility for Python packages\n  downloads: 45028\n  homepage_uri: ''\n  name: python-central\n  section: universe/python\n  version: 0.6.17ubuntu2\npython-chardet:\n  description: universal character encoding detector\n  downloads: 28442\n  homepage_uri: http://chardet.feedparser.org/\n  name: python-chardet\n  section: python\n  version: 2.0.1-2build1\npython-cheetah:\n  description: text-based template engine and Python code generator\n  downloads: 698\n  homepage_uri: http://www.cheetahtemplate.org/\n  name: python-cheetah\n  section: text\n  version: 2.4.4-2ubuntu3\npython-configglue:\n  description: Glues together optparse.OptionParser and ConfigParser.ConfigParser\n  downloads: 13\n  homepage_uri: ''\n  name: python-configglue\n  section: python\n  version: 1.0-1build1\npython-configobj:\n  description: simple but powerful config file reader and writer for Python\n  downloads: 12308\n  homepage_uri: http://www.voidspace.org.uk/python/configobj.html\n  name: python-configobj\n  section: python\n  version: 4.7.2+ds-3build1\npython-crypto:\n  description: cryptographic algorithms and protocols for Python\n  downloads: 5955\n  homepage_uri: http://www.pycrypto.org/\n  name: python-crypto\n  section: python\n  version: 2.4.1-1\npython-cups:\n  description: Python bindings for CUPS\n  downloads: 28915\n  homepage_uri: ''\n  name: python-cups\n  section: python\n  version: 1.9.61-0ubuntu2\npython-cupshelpers:\n  description: Python modules for printer configuration with CUPS\n  downloads: 26679\n  homepage_uri: ''\n  name: python-cupshelpers\n  section: python\n  version: 1.3.8+20120201-0ubuntu8.1\npython-dateutil:\n  description: powerful extensions to the standard datetime module\n  downloads: 2412\n  homepage_uri: ''\n  name: python-dateutil\n  section: python\n  version: 1.5-1\npython-dbus:\n  description: simple interprocess messaging system (Python interface)\n  downloads: 37449\n  homepage_uri: http://www.freedesktop.org/wiki/Software/DBusBindings#Python\n  name: python-dbus\n  section: python\n  version: 1.0.0-1ubuntu1\npython-dbus-dev:\n  description: main loop integration development files for python-dbus\n  downloads: 8985\n  homepage_uri: http://www.freedesktop.org/wiki/Software/DBusBindings#Python\n  name: python-dbus-dev\n  section: python\n  version: 1.0.0-1ubuntu1\npython-debian:\n  description: Python modules to work with Debian-related data formats\n  downloads: 47296\n  homepage_uri: ''\n  name: python-debian\n  section: devel\n  version: 0.1.21ubuntu1\npython-debtagshw:\n  description: 'Match debtags hardware:: tags against the actual hardware'\n  downloads: 2\n  homepage_uri: http://wiki.debian.org/Debtags\n  name: python-debtagshw\n  section: python\n  version: 1.9+git20120320\npython-defer:\n  description: Small framework for asynchronous programming\n  downloads: 3413\n  homepage_uri: https://launchpad.net/python-defer\n  name: python-defer\n  section: python\n  version: 1.0.2+bzr481-1\npython-dev:\n  description: header files and a static library for Python (default)\n  downloads: 1433\n  homepage_uri: http://www.python.org/\n  name: python-dev\n  section: python\n  version: 2.7.3-0ubuntu2\npython-dirspec:\n  description: Python User Folders Specification Library\n  downloads: 1\n  homepage_uri: ''\n  name: python-dirspec\n  section: python\n  version: 3.0.0-0ubuntu1\npython-distutils-extra:\n  description: enhancements to the Python build system\n  downloads: 93\n  homepage_uri: ''\n  name: python-distutils-extra\n  section: python\n  version: 2.32-2\npython-egenix-mxdatetime:\n  description: date and time handling routines for Python\n  downloads: 1759\n  homepage_uri: ''\n  name: python-egenix-mxdatetime\n  section: python\n  version: 3.2.1-1ubuntu1\npython-egenix-mxtools:\n  description: collection of additional builtins for Python\n  downloads: 1669\n  homepage_uri: ''\n  name: python-egenix-mxtools\n  section: python\n  version: 3.2.1-1ubuntu1\npython-envisage:\n  description: Extensible Application Framework\n  downloads: 35\n  homepage_uri: http://pypi.python.org/pypi/envisage\n  name: python-envisage\n  section: universe/python\n  version: 4.0.0-1build1\npython-gconf:\n  description: Python bindings for the GConf configuration database system\n  downloads: 26968\n  homepage_uri: http://www.pygtk.org/\n  name: python-gconf\n  section: python\n  version: 2.28.1+dfsg-1\npython-gdbm:\n  description: GNU dbm database support for Python\n  downloads: 20429\n  homepage_uri: ''\n  name: python-gdbm\n  section: python\n  version: 2.7.3-1ubuntu1\npython-gi:\n  description: Python 2.x bindings for gobject-introspection libraries\n  downloads: 12254\n  homepage_uri: ''\n  name: python-gi\n  section: python\n  version: 3.2.2-1~precise\npython-gi-cairo:\n  description: Python Cairo bindings for the GObject library\n  downloads: 2665\n  homepage_uri: ''\n  name: python-gi-cairo\n  section: python\n  version: 3.2.2-1~precise\npython-gmenu:\n  description: GNOME implementation of the freedesktop menu specification\n  downloads: 2743\n  homepage_uri: ''\n  name: python-gmenu\n  section: gnome\n  version: 3.0.1-0ubuntu7\npython-gnomekeyring:\n  description: Python bindings for the GNOME keyring library\n  downloads: 21841\n  homepage_uri: ''\n  name: python-gnomekeyring\n  section: python\n  version: 2.32.0+dfsg-1\npython-gnupginterface:\n  description: Python interface to GnuPG (GPG)\n  downloads: 2296\n  homepage_uri: ''\n  name: python-gnupginterface\n  section: python\n  version: 0.3.2-9.1ubuntu3\npython-gobject:\n  description: Python 2.x bindings for GObject - transitional package\n  downloads: 43657\n  homepage_uri: ''\n  name: python-gobject\n  section: python\n  version: 3.2.2-1~precise\npython-gobject-2:\n  description: deprecated static Python bindings for the GObject library\n  downloads: 17303\n  homepage_uri: ''\n  name: python-gobject-2\n  section: oldlibs\n  version: 2.28.6-10ubuntu1\npython-gst0.10:\n  description: generic media-playing framework (Python bindings)\n  downloads: 38116\n  homepage_uri: http://gstreamer.freedesktop.org\n  name: python-gst0.10\n  section: python\n  version: 0.10.22-3\npython-gtk2:\n  description: Python bindings for the GTK+ widget set\n  downloads: 39940\n  homepage_uri: http://www.pygtk.org/\n  name: python-gtk2\n  section: python\n  version: 2.24.0-3\npython-httplib2:\n  description: comprehensive HTTP client library written for Python\n  downloads: 16810\n  homepage_uri: http://code.google.com/p/httplib2/\n  name: python-httplib2\n  section: libs\n  version: 0.7.2-1ubuntu2\npython-ibus:\n  description: Intelligent Input Bus - Python support\n  downloads: 678\n  homepage_uri: http://code.google.com/p/ibus/\n  name: python-ibus\n  section: python\n  version: 1.4.1-3ubuntu1\npython-imaging:\n  description: Python Imaging Library\n  downloads: 45621\n  homepage_uri: ''\n  name: python-imaging\n  section: python\n  version: 1.1.7-4\npython-keyring:\n  description: store and access your passwords safely\n  downloads: 386\n  homepage_uri: http://home.python-keyring.org/\n  name: python-keyring\n  section: python\n  version: 0.7.1-1fakesync1\npython-launchpadlib:\n  description: Launchpad web services client library\n  downloads: 320\n  homepage_uri: https://launchpad.net/launchpadlib\n  name: python-launchpadlib\n  section: python\n  version: 1.9.12-1\npython-lazr.restfulclient:\n  description: client for lazr.restful-based web services\n  downloads: 3852\n  homepage_uri: https://launchpad.net/lazr.restfulclient\n  name: python-lazr.restfulclient\n  section: python\n  version: 0.12.0-1ubuntu1\npython-lazr.uri:\n  description: library for parsing, manipulating, and generating URIs\n  downloads: 3856\n  homepage_uri: https://launchpad.net/lazr.uri\n  name: python-lazr.uri\n  section: python\n  version: 1.0.3-1\npython-libproxy:\n  description: automatic proxy configuration management library (python)\n  downloads: 120\n  homepage_uri: http://code.google.com/p/libproxy/\n  name: python-libproxy\n  section: python\n  version: 0.4.7-0ubuntu4\npython-libvirt:\n  description: libvirt Python bindings\n  downloads: 1251\n  homepage_uri: http://libvirt.org\n  name: python-libvirt\n  section: devel\n  version: 0.9.8-2ubuntu17.1\npython-libxml2:\n  description: Python bindings for the GNOME XML library\n  downloads: 42392\n  homepage_uri: http://xmlsoft.org/\n  name: python-libxml2\n  section: python\n  version: 2.7.8.dfsg-5.1ubuntu4.1\npython-louis:\n  description: Python bindings for liblouis\n  downloads: 2247\n  homepage_uri: http://code.google.com/p/liblouis/\n  name: python-louis\n  section: python\n  version: 2.3.0-3\npython-m2crypto:\n  description: a crypto and SSL toolkit for Python\n  downloads: 469\n  homepage_uri: http://chandlerproject.org/Projects/MeTooCrypto\n  name: python-m2crypto\n  section: libs\n  version: 0.21.1-2ubuntu2\npython-magic:\n  description: File type determination library using \"magic\" numbers (Python bindings)\n  downloads: 433\n  homepage_uri: http://www.darwinsys.com/file/\n  name: python-magic\n  section: python\n  version: 5.09-2\npython-mako:\n  description: fast and lightweight templating for the Python platform\n  downloads: 4062\n  homepage_uri: http://www.makotemplates.org/\n  name: python-mako\n  section: python\n  version: 0.5.0-1\npython-markupsafe:\n  description: XML/HTML/XHTML Markup safe string for Python\n  downloads: 19492\n  homepage_uri: http://pypi.python.org/pypi/MarkupSafe\n  name: python-markupsafe\n  section: python\n  version: 0.15-1\npython-minimal:\n  description: minimal subset of the Python language (default version)\n  downloads: 85145\n  homepage_uri: http://www.python.org/\n  name: python-minimal\n  section: python\n  version: 2.7.3-0ubuntu2\npython-newt:\n  description: A NEWT module for Python\n  downloads: 1059\n  homepage_uri: https://fedorahosted.org/newt/\n  name: python-newt\n  section: python\n  version: 0.52.11-2ubuntu10\npython-notify:\n  description: Python bindings for libnotify\n  downloads: 32518\n  homepage_uri: http://www.galago-project.org\n  name: python-notify\n  section: python\n  version: 0.1.1-3\npython-numpy:\n  description: Numerical Python adds a fast array facility to the Python language\n  downloads: 38123\n  homepage_uri: http://numpy.scipy.org/\n  name: python-numpy\n  section: python\n  version: 1:1.6.1-6ubuntu1\npython-oauth:\n  description: Python library implementing of the OAuth protocol\n  downloads: 4026\n  homepage_uri: http://code.google.com/p/oauth\n  name: python-oauth\n  section: python\n  version: 1.0.1-3build1\npython-openssl:\n  description: Python wrapper around the OpenSSL library\n  downloads: 6669\n  homepage_uri: http://launchpad.net/pyopenssl\n  name: python-openssl\n  section: python\n  version: 0.12-1ubuntu2\npython-packagekit:\n  description: PackageKit Python bindings\n  downloads: 6292\n  homepage_uri: http://www.packagekit.org\n  name: python-packagekit\n  section: python\n  version: 0.7.2-4ubuntu3\npython-pam:\n  description: A Python interface to the PAM library\n  downloads: 930\n  homepage_uri: ''\n  name: python-pam\n  section: python\n  version: 0.4.2-12.2ubuntu4\npython-paramiko:\n  description: Make ssh v2 connections with Python\n  downloads: 493\n  homepage_uri: http://www.lag.net/paramiko/\n  name: python-paramiko\n  section: python\n  version: 1.7.7.1-2\npython-pexpect:\n  description: Python module for automating interactive applications\n  downloads: 1651\n  homepage_uri: ''\n  name: python-pexpect\n  section: python\n  version: 2.3-1ubuntu2\npython-pip:\n  description: alternative Python package installer\n  downloads: 546\n  homepage_uri: http://pip.openplans.org/\n  name: python-pip\n  section: universe/python\n  version: 1.0-1build1\npython-piston-mini-client:\n  description: library for writing clients for Django's Piston REST APIs\n  downloads: 0\n  homepage_uri: https://launchpad.net/piston-mini-client\n  name: python-piston-mini-client\n  section: python\n  version: 0.7.2-0ubuntu1\npython-pkg-resources:\n  description: Package Discovery and Resource Access using pkg_resources\n  downloads: 31151\n  homepage_uri: http://packages.python.org/distribute\n  name: python-pkg-resources\n  section: python\n  version: 0.6.24-1ubuntu1\npython-problem-report:\n  description: Python library to handle problem reports\n  downloads: 15\n  homepage_uri: https://wiki.ubuntu.com/Apport\n  name: python-problem-report\n  section: python\n  version: 2.0.1-0ubuntu8\npython-protobuf:\n  description: Python bindings for protocol buffers\n  downloads: 12\n  homepage_uri: http://code.google.com/p/protobuf/\n  name: python-protobuf\n  section: python\n  version: 2.4.1-1ubuntu2\npython-pyatspi2:\n  description: Assistive Technology Service Provider Interface - Python bindings\n  downloads: 413\n  homepage_uri: http://live.gnome.org/Accessibility/GNOME3\n  name: python-pyatspi2\n  section: python\n  version: 2.4.0+dfsg-0ubuntu3\npython-pycurl:\n  description: Python bindings to libcurl\n  downloads: 10373\n  homepage_uri: http://pycurl.sourceforge.net\n  name: python-pycurl\n  section: python\n  version: 7.19.0-4ubuntu3\npython-pyface:\n  description: traits-capable windowing framework\n  downloads: 96\n  homepage_uri: http://pypi.python.org/pypi/pyface\n  name: python-pyface\n  section: universe/python\n  version: 4.0.0-1build1\npython-pyinotify:\n  description: simple Linux inotify Python bindings\n  downloads: 285\n  homepage_uri: https://github.com/seb-m/pyinotify\n  name: python-pyinotify\n  section: python\n  version: 0.9.2-1\npython-qt4:\n  description: Python bindings for Qt4\n  downloads: 6076\n  homepage_uri: http://www.riverbankcomputing.co.uk/software/pyqt/\n  name: python-qt4\n  section: python\n  version: 4.9.1-2ubuntu1\npython-renderpm:\n  description: python low level render interface\n  downloads: 366\n  homepage_uri: ''\n  name: python-renderpm\n  section: python\n  version: 2.5-1.1build1\npython-reportlab:\n  description: ReportLab library to create PDF documents using Python\n  downloads: 1997\n  homepage_uri: ''\n  name: python-reportlab\n  section: python\n  version: 2.5-1.1build1\npython-reportlab-accel:\n  description: C coded extension accelerator for the ReportLab Toolkit\n  downloads: 1559\n  homepage_uri: ''\n  name: python-reportlab-accel\n  section: python\n  version: 2.5-1.1build1\npython-rpy:\n  description: Python interface to the GNU R language and environment\n  downloads: 29\n  homepage_uri: ''\n  name: python-rpy\n  section: python\n  version: 1.0.3-21precise0\npython-scour:\n  description: SVG scrubber and optimizer\n  downloads: 11\n  homepage_uri: http://codedread.com/scour/\n  name: python-scour\n  section: python\n  version: 0.26-3\npython-serial:\n  description: pyserial - module encapsulating access for the serial port\n  downloads: 612\n  homepage_uri: ''\n  name: python-serial\n  section: python\n  version: 2.5-2.1build1\npython-setuptools:\n  description: Python Distutils Enhancements (setuptools compatibility)\n  downloads: 8371\n  homepage_uri: http://packages.python.org/distribute\n  name: python-setuptools\n  section: python\n  version: 0.6.24-1ubuntu1\npython-simplejson:\n  description: simple, fast, extensible JSON encoder/decoder for Python\n  downloads: 5862\n  homepage_uri: http://undefined.org/python/#simplejson\n  name: python-simplejson\n  section: python\n  version: 2.3.2-1\npython-sip:\n  description: Python/C++ bindings generator runtime library\n  downloads: 4615\n  homepage_uri: http://www.riverbankcomputing.co.uk/software/sip/\n  name: python-sip\n  section: python\n  version: 4.13.2-1\npython-smbc:\n  description: Python bindings for Samba clients (libsmbclient)\n  downloads: 53\n  homepage_uri: http://cyberelk.net/tim/software/pysmbc/\n  name: python-smbc\n  section: python\n  version: 1.0.13-0ubuntu1\npython-software-properties:\n  description: manage the repositories that you install software from\n  downloads: 18287\n  homepage_uri: ''\n  name: python-software-properties\n  section: gnome\n  version: 0.82.7.1\npython-speechd:\n  description: Python interface to Speech Dispatcher\n  downloads: 546\n  homepage_uri: http://devel.freebsoft.org/speechd\n  name: python-speechd\n  section: python\n  version: 0.7.1-6ubuntu3\npython-support:\n  description: automated rebuilding support for Python modules\n  downloads: 80317\n  homepage_uri: ''\n  name: python-support\n  section: universe/python\n  version: 1.0.14ubuntu2\npython-traits:\n  description: Manifest typing and reactive programming for Python\n  downloads: 176\n  homepage_uri: http://pypi.python.org/pypi/traits\n  name: python-traits\n  section: universe/python\n  version: 4.0.0-1build1\npython-traitsui:\n  description: traits-capable user interfaces\n  downloads: 111\n  homepage_uri: http://pypi.python.org/pypi/traitsui\n  name: python-traitsui\n  section: universe/python\n  version: 4.0.1-1build1\npython-twisted-bin:\n  description: Event-based framework for internet applications\n  downloads: 6286\n  homepage_uri: ''\n  name: python-twisted-bin\n  section: python\n  version: 11.1.0-1ubuntu2\npython-twisted-core:\n  description: Event-based framework for internet applications\n  downloads: 6118\n  homepage_uri: ''\n  name: python-twisted-core\n  section: python\n  version: 11.1.0-1ubuntu2\npython-twisted-names:\n  description: DNS protocol implementation with client and server\n  downloads: 297\n  homepage_uri: ''\n  name: python-twisted-names\n  section: python\n  version: 11.1.0-1\npython-twisted-web:\n  description: HTTP protocol implementation together with clients and servers\n  downloads: 4233\n  homepage_uri: ''\n  name: python-twisted-web\n  section: python\n  version: 11.1.0-1\npython-ubuntu-sso-client:\n  description: Ubuntu Single Sign-On client - Python library\n  downloads: 10\n  homepage_uri: ''\n  name: python-ubuntu-sso-client\n  section: python\n  version: 3.0.0-0ubuntu2\npython-ubuntuone-client:\n  description: Ubuntu One client Python libraries\n  downloads: 17\n  homepage_uri: https://one.ubuntu.com\n  name: python-ubuntuone-client\n  section: python\n  version: 3.0.1-0ubuntu1.0.1\npython-ubuntuone-control-panel:\n  description: Ubuntu One Control Panel - Python Libraries\n  downloads: 6\n  homepage_uri: ''\n  name: python-ubuntuone-control-panel\n  section: python\n  version: 3.0.1-0ubuntu1\npython-ubuntuone-storageprotocol:\n  description: Python library for Ubuntu One file storage and sharing service\n  downloads: 18\n  homepage_uri: https://one.ubuntu.com\n  name: python-ubuntuone-storageprotocol\n  section: python\n  version: 3.0.0-0ubuntu1.1\npython-uno:\n  description: Python-UNO bridge\n  downloads: 9177\n  homepage_uri: http://udk.openoffice.org/python/python-bridge.html\n  name: python-uno\n  section: python\n  version: 1:3.5.3-0ubuntu1\npython-virtkey:\n  description: Library to emulate keyboard keypresses.\n  downloads: 4\n  homepage_uri: https://launchpad.net/python-virtkey\n  name: python-virtkey\n  section: python\n  version: 0.60.0-0ubuntu5\npython-vm-builder:\n  description: VM builder\n  downloads: 0\n  homepage_uri: ''\n  name: python-vm-builder\n  section: universe/utils\n  version: 0.12.4+bzr477-0ubuntu1\npython-vm-builder-ec2:\n  description: EC2 Ubuntu VM builder\n  downloads: 0\n  homepage_uri: ''\n  name: python-vm-builder-ec2\n  section: universe/utils\n  version: 0.12.4+bzr477-0ubuntu1\npython-vtk:\n  description: Python bindings for VTK\n  downloads: 170\n  homepage_uri: http://www.vtk.org/\n  name: python-vtk\n  section: universe/python\n  version: 5.8.0-5\npython-wadllib:\n  description: Python library for navigating WADL files\n  downloads: 3825\n  homepage_uri: https://launchpad.net/wadllib\n  name: python-wadllib\n  section: python\n  version: 1.3.0-2\npython-wxgtk2.8:\n  description: wxWidgets Cross-platform C++ GUI toolkit (wxPython binding)\n  downloads: 9076\n  homepage_uri: http://www.wxpython.org/\n  name: python-wxgtk2.8\n  section: universe/python\n  version: 2.8.12.1-6ubuntu2\npython-wxversion:\n  description: wxWidgets Cross-platform C++ GUI toolkit (wxPython version selector)\n  downloads: 10382\n  homepage_uri: http://www.wxpython.org/\n  name: python-wxversion\n  section: universe/python\n  version: 2.8.12.1-6ubuntu2\npython-xapian:\n  description: Xapian search engine interface for Python\n  downloads: 46065\n  homepage_uri: http://xapian.org/\n  name: python-xapian\n  section: python\n  version: 1.2.8-1\npython-xdg:\n  description: Python library to access freedesktop.org standards\n  downloads: 26576\n  homepage_uri: http://www.freedesktop.org/wiki/Software/pyxdg\n  name: python-xdg\n  section: python\n  version: 0.19-3ubuntu2\npython-xkit:\n  description: library for the manipulation of the xorg.conf\n  downloads: 0\n  homepage_uri: ''\n  name: python-xkit\n  section: python\n  version: 0.4.2.3build1\npython-yaml:\n  description: YAML parser and emitter for Python\n  downloads: 794\n  homepage_uri: http://pyyaml.org/\n  name: python-yaml\n  section: python\n  version: 3.10-2\npython-zeitgeist:\n  description: event logging framework - Python bindings\n  downloads: 431\n  homepage_uri: http://zeitgeist-project.com/\n  name: python-zeitgeist\n  section: python\n  version: 0.9.0-1ubuntu1\npython-zope.interface:\n  description: Interfaces for Python\n  downloads: 19516\n  homepage_uri: http://pypi.python.org/pypi/zope.interface\n  name: python-zope.interface\n  section: python\n  version: 3.6.1-1ubuntu3\npython2.7:\n  description: Interactive high-level object-oriented language (version 2.7)\n  downloads: 19008\n  homepage_uri: ''\n  name: python2.7\n  section: python\n  version: 2.7.3-0ubuntu3\npython2.7-dev:\n  description: Header files and a static library for Python (v2.7)\n  downloads: 1098\n  homepage_uri: ''\n  name: python2.7-dev\n  section: python\n  version: 2.7.3-0ubuntu3\npython2.7-minimal:\n  description: Minimal subset of the Python language (version 2.7)\n  downloads: 19404\n  homepage_uri: ''\n  name: python2.7-minimal\n  section: python\n  version: 2.7.3-0ubuntu3\npython3:\n  description: interactive high-level object-oriented language (default python3 version)\n  downloads: 1037\n  homepage_uri: http://www.python.org/\n  name: python3\n  section: python\n  version: 3.2.3-0ubuntu1\npython3-dev:\n  description: header files and a static library for Python (default)\n  downloads: 55\n  homepage_uri: http://www.python.org/\n  name: python3-dev\n  section: python\n  version: 3.2.3-0ubuntu1\npython3-minimal:\n  description: minimal subset of the Python language (default python3 version)\n  downloads: 1202\n  homepage_uri: http://www.python.org/\n  name: python3-minimal\n  section: python\n  version: 3.2.3-0ubuntu1\npython3.2:\n  description: Interactive high-level object-oriented language (version 3.2)\n  downloads: 1125\n  homepage_uri: ''\n  name: python3.2\n  section: python\n  version: 3.2.3-0ubuntu3\npython3.2-dev:\n  description: Header files and a static library for Python (v3.2)\n  downloads: 65\n  homepage_uri: ''\n  name: python3.2-dev\n  section: python\n  version: 3.2.3-0ubuntu3\npython3.2-minimal:\n  description: Minimal subset of the Python language (version 3.2)\n  downloads: 1324\n  homepage_uri: ''\n  name: python3.2-minimal\n  section: python\n  version: 3.2.3-0ubuntu3\nqdbus:\n  description: Qt 4 D-Bus tool\n  downloads: 5325\n  homepage_uri: http://qt.nokia.com/\n  name: qdbus\n  section: libs\n  version: 4:4.8.1-0ubuntu4.1\nqemu-common:\n  description: qemu common functionality (bios, documentation, etc)\n  downloads: 0\n  homepage_uri: http://www.linux-kvm.org\n  name: qemu-common\n  section: misc\n  version: 1.0+noroms-0ubuntu13\nqemu-kvm:\n  description: Full virtualization on i386 and amd64 hardware\n  downloads: 3566\n  homepage_uri: http://www.linux-kvm.org\n  name: qemu-kvm\n  section: misc\n  version: 1.0+noroms-0ubuntu13\nqemu-utils:\n  description: qemu utilities\n  downloads: 1069\n  homepage_uri: http://www.linux-kvm.org\n  name: qemu-utils\n  section: misc\n  version: 1.0+noroms-0ubuntu13\nqt-at-spi:\n  description: accessibility plugin for Qt\n  downloads: 0\n  homepage_uri: ''\n  name: qt-at-spi\n  section: misc\n  version: 0.2.0+git20120411-0ubuntu1\nr-base:\n  description: GNU R statistical computation and graphics system\n  downloads: 1\n  homepage_uri: http://www.r-project.org/\n  name: r-base\n  section: universe/math\n  version: 2.15.0-1precise0\nr-base-core:\n  description: GNU R core of statistical computation and graphics system\n  downloads: 1501\n  homepage_uri: http://www.r-project.org/\n  name: r-base-core\n  section: universe/math\n  version: 2.15.0-1precise0\nr-base-dev:\n  description: GNU R installation of auxiliary GNU R packages\n  downloads: 1\n  homepage_uri: http://www.r-project.org/\n  name: r-base-dev\n  section: universe/devel\n  version: 2.15.0-1precise0\nr-base-html:\n  description: GNU R html docs for statistical computing system functions\n  downloads: 77\n  homepage_uri: http://www.r-project.org/\n  name: r-base-html\n  section: universe/math\n  version: 2.15.0-1precise0\nr-cran-boot:\n  description: GNU R package for bootstrapping functions from Davison and Hinkley\n  downloads: 319\n  homepage_uri: ''\n  name: r-cran-boot\n  section: universe/math\n  version: 1.3-4-1precise0\nr-cran-class:\n  description: GNU R package for classification\n  downloads: 323\n  homepage_uri: http://cran.r-project.org/web/packages/class/index.html\n  name: r-cran-class\n  section: universe/gnu-r\n  version: 7.3-3-1precise0\nr-cran-cluster:\n  description: GNU R package for cluster analysis by Rousseeuw et al\n  downloads: 544\n  homepage_uri: ''\n  name: r-cran-cluster\n  section: universe/math\n  version: 1.14.2-1precise0\nr-cran-codetools:\n  description: GNU R package providing code analysis tools\n  downloads: 345\n  homepage_uri: ''\n  name: r-cran-codetools\n  section: universe/math\n  version: 0.2-8-1precise0\nr-cran-foreign:\n  description: GNU R package to read/write data from other stat. systems\n  downloads: 368\n  homepage_uri: ''\n  name: r-cran-foreign\n  section: universe/math\n  version: 0.8.50-1precise0\nr-cran-kernsmooth:\n  description: GNU R package for kernel smoothing and density estimation\n  downloads: 329\n  homepage_uri: ''\n  name: r-cran-kernsmooth\n  section: universe/math\n  version: 2.23-7-1precise0\nr-cran-lattice:\n  description: GNU R package for 'Trellis' graphics\n  downloads: 636\n  homepage_uri: ''\n  name: r-cran-lattice\n  section: universe/math\n  version: 0.20-6-1precise0\nr-cran-mass:\n  description: GNU R package of Venables and Ripley's MASS\n  downloads: 421\n  homepage_uri: http://cran.r-project.org/web/packages/MASS/index.html\n  name: r-cran-mass\n  section: universe/gnu-r\n  version: 7.3-18-1precise0\nr-cran-matrix:\n  description: GNU R package of classes for dense and sparse matrices\n  downloads: 585\n  homepage_uri: ''\n  name: r-cran-matrix\n  section: universe/math\n  version: 1.0-6-1precise0\nr-cran-mgcv:\n  description: GNU R package for multiple parameter smoothing estimation\n  downloads: 599\n  homepage_uri: ''\n  name: r-cran-mgcv\n  section: universe/math\n  version: 1.7-18-1precise0\nr-cran-nlme:\n  description: GNU R package for (non-)linear mixed effects models\n  downloads: 595\n  homepage_uri: ''\n  name: r-cran-nlme\n  section: universe/math\n  version: 3.1.104-1precise0\nr-cran-nnet:\n  description: GNU R package for feed-forward neural networks\n  downloads: 574\n  homepage_uri: http://cran.r-project.org/web/packages/nnet/index.html\n  name: r-cran-nnet\n  section: universe/gnu-r\n  version: 7.3-1-2precise0\nr-cran-rpart:\n  description: GNU R package for recursive partitioning and regression trees\n  downloads: 317\n  homepage_uri: ''\n  name: r-cran-rpart\n  section: universe/math\n  version: 3.1.52-1precise0\nr-cran-spatial:\n  description: GNU R package for spatial statistics\n  downloads: 312\n  homepage_uri: http://cran.r-project.org/web/packages/spatial/index.html\n  name: r-cran-spatial\n  section: universe/gnu-r\n  version: 7.3-3-1precise0\nr-cran-survival:\n  description: GNU R package for survival analysis\n  downloads: 608\n  homepage_uri: ''\n  name: r-cran-survival\n  section: universe/math\n  version: 2.36-14-1precise0\nr-doc-html:\n  description: GNU R html manuals for statistical computing system\n  downloads: 0\n  homepage_uri: http://www.r-project.org/\n  name: r-doc-html\n  section: universe/doc\n  version: 2.15.0-1precise0\nr-recommended:\n  description: GNU R collection of recommended packages [metapackage]\n  downloads: 33\n  homepage_uri: http://www.r-project.org/\n  name: r-recommended\n  section: universe/math\n  version: 2.15.0-1precise0\nrabbitmq-server:\n  description: An AMQP server written in Erlang\n  downloads: 197\n  homepage_uri: http://www.rabbitmq.com/\n  name: rabbitmq-server\n  section: net\n  version: 2.7.1-0ubuntu4\nradeontool:\n  description: utility to control ATI Radeon backlight functions on laptops\n  downloads: 1434\n  homepage_uri: ''\n  name: radeontool\n  section: utils\n  version: 1.6.2-1.1\nrarian-compat:\n  description: Documentation meta-data library (compatibility tools)\n  downloads: 11693\n  homepage_uri: http://rarian.freedesktop.org/\n  name: rarian-compat\n  section: doc\n  version: 0.8.1-5\nrasmol:\n  description: Visualize biological macromolecules\n  downloads: 284\n  homepage_uri: http://rasmol.org\n  name: rasmol\n  section: universe/science\n  version: 2.7.5-3\nreadline-common:\n  description: GNU readline and history libraries, common files\n  downloads: 18065\n  homepage_uri: ''\n  name: readline-common\n  section: utils\n  version: 6.2-8\nreadseq:\n  description: Conversion between sequence formats\n  downloads: 22\n  homepage_uri: http://iubio.bio.indiana.edu/soft/molbio/readseq/\n  name: readseq\n  section: science\n  version: 1-9\nrecode:\n  description: Character set conversion utility\n  downloads: 1243\n  homepage_uri: ''\n  name: recode\n  section: text\n  version: 3.6-18\nreiserfsprogs:\n  description: User-level tools for ReiserFS filesystems\n  downloads: 1437\n  homepage_uri: ''\n  name: reiserfsprogs\n  section: admin\n  version: 1:3.6.21-1build1\nremmina:\n  description: remote desktop client for GNOME desktop environment\n  downloads: 3739\n  homepage_uri: http://remmina.sourceforge.net/\n  name: remmina\n  section: gnome\n  version: 1.0.0-1ubuntu6.1\nremmina-common:\n  description: common files for remmina remote desktop client\n  downloads: 0\n  homepage_uri: http://remmina.sourceforge.net/\n  name: remmina-common\n  section: gnome\n  version: 1.0.0-1ubuntu6.1\nremmina-plugin-rdp:\n  description: RDP plugin for remmina remote desktop client\n  downloads: 2412\n  homepage_uri: http://remmina.sourceforge.net/\n  name: remmina-plugin-rdp\n  section: gnome\n  version: 1.0.0-1ubuntu6.1\nremmina-plugin-vnc:\n  description: VNC plugin for remmina remote desktop client\n  downloads: 2396\n  homepage_uri: http://remmina.sourceforge.net/\n  name: remmina-plugin-vnc\n  section: gnome\n  version: 1.0.0-1ubuntu6.1\nresolvconf:\n  description: name server information handler\n  downloads: 4595\n  homepage_uri: http://alioth.debian.org/projects/resolvconf/\n  name: resolvconf\n  section: net\n  version: 1.63ubuntu14\nrfkill:\n  description: tool for enabling and disabling wireless devices\n  downloads: 398\n  homepage_uri: http://wireless.kernel.org/en/users/Documentation/rfkill\n  name: rfkill\n  section: utils\n  version: 0.4-1ubuntu2\nrhino:\n  description: JavaScript engine written in Java\n  downloads: 1015\n  homepage_uri: http://www.mozilla.org/rhino/\n  name: rhino\n  section: interpreters\n  version: 1.7R3-5\nrhythmbox:\n  description: music player and organizer for GNOME\n  downloads: 15854\n  homepage_uri: http://projects.gnome.org/rhythmbox/\n  name: rhythmbox\n  section: gnome\n  version: 2.96-0ubuntu4\nrhythmbox-data:\n  description: data files for rhythmbox\n  downloads: 0\n  homepage_uri: http://projects.gnome.org/rhythmbox/\n  name: rhythmbox-data\n  section: gnome\n  version: 2.96-0ubuntu4\nrhythmbox-mozilla:\n  description: Rhythmbox Mozilla plugin\n  downloads: 0\n  homepage_uri: http://projects.gnome.org/rhythmbox/\n  name: rhythmbox-mozilla\n  section: web\n  version: 2.96-0ubuntu4\nrhythmbox-plugin-cdrecorder:\n  description: burning plugin for rhythmbox music player\n  downloads: 4072\n  homepage_uri: http://projects.gnome.org/rhythmbox/\n  name: rhythmbox-plugin-cdrecorder\n  section: gnome\n  version: 2.96-0ubuntu4\nrhythmbox-plugin-magnatune:\n  description: Magnatune plugin for rhythmbox music player\n  downloads: 0\n  homepage_uri: http://projects.gnome.org/rhythmbox/\n  name: rhythmbox-plugin-magnatune\n  section: gnome\n  version: 2.96-0ubuntu4\nrhythmbox-plugin-zeitgeist:\n  description: zeitgeist plugin for rhythmbox music player\n  downloads: 0\n  homepage_uri: http://projects.gnome.org/rhythmbox/\n  name: rhythmbox-plugin-zeitgeist\n  section: gnome\n  version: 2.96-0ubuntu4\nrhythmbox-plugins:\n  description: plugins for rhythmbox music player\n  downloads: 18580\n  homepage_uri: http://projects.gnome.org/rhythmbox/\n  name: rhythmbox-plugins\n  section: gnome\n  version: 2.96-0ubuntu4\nrhythmbox-ubuntuone:\n  description: Ubuntu One Rhythmbox plugin\n  downloads: 1\n  homepage_uri: https://launchpad.net/rhythmbox-ubuntuone\n  name: rhythmbox-ubuntuone\n  section: gnome\n  version: 3.0.0-0ubuntu1\nri1.8:\n  description: Ruby Interactive reference (for Ruby 1.8)\n  downloads: 100\n  homepage_uri: http://www.ruby-lang.org/\n  name: ri1.8\n  section: universe/interpreters\n  version: 1.8.7.352-2ubuntu1\nrpcbind:\n  description: converts RPC program numbers into universal addresses\n  downloads: 13892\n  homepage_uri: http://sourceforge.net/projects/rpcbind/\n  name: rpcbind\n  section: net\n  version: 0.2.0-7ubuntu1.1\nrsync:\n  description: fast, versatile, remote (and local) file-copying tool\n  downloads: 23606\n  homepage_uri: http://rsync.samba.org/\n  name: rsync\n  section: net\n  version: 3.0.9-1ubuntu1\nrsyslog:\n  description: reliable system and kernel logging daemon\n  downloads: 72075\n  homepage_uri: http://www.rsyslog.com/\n  name: rsyslog\n  section: admin\n  version: 5.8.6-1ubuntu8\nrtkit:\n  description: Realtime Policy and Watchdog Daemon\n  downloads: 11470\n  homepage_uri: http://0pointer.de/public/\n  name: rtkit\n  section: admin\n  version: 0.10-2\nruby:\n  description: Transitional package for ruby1.8\n  downloads: 8172\n  homepage_uri: http://www.ruby-lang.org/\n  name: ruby\n  section: interpreters\n  version: '4.8'\nruby1.8:\n  description: Interpreter of object-oriented scripting language Ruby 1.8\n  downloads: 10045\n  homepage_uri: http://www.ruby-lang.org/\n  name: ruby1.8\n  section: interpreters\n  version: 1.8.7.352-2ubuntu1\nruby1.8-dev:\n  description: Header files for compiling extension modules for the Ruby 1.8\n  downloads: 524\n  homepage_uri: http://www.ruby-lang.org/\n  name: ruby1.8-dev\n  section: devel\n  version: 1.8.7.352-2ubuntu1\nruby1.9.1:\n  description: Interpreter of object-oriented scripting language Ruby\n  downloads: 2200\n  homepage_uri: http://www.ruby-lang.org/\n  name: ruby1.9.1\n  section: ruby\n  version: 1.9.3.0-1ubuntu1\nruby1.9.1-dev:\n  description: Header files for compiling extension modules for the Ruby 1.9.1\n  downloads: 189\n  homepage_uri: http://www.ruby-lang.org/\n  name: ruby1.9.1-dev\n  section: ruby\n  version: 1.9.3.0-1ubuntu1\ns3cmd:\n  description: command-line Amazon S3 client\n  downloads: 101\n  homepage_uri: http://s3tools.logix.cz/s3cmd\n  name: s3cmd\n  section: universe/admin\n  version: 1.0.0-1\nsamba-common:\n  description: common files used by both the Samba server and client\n  downloads: 25486\n  homepage_uri: http://www.samba.org\n  name: samba-common\n  section: net\n  version: 2:3.6.3-2ubuntu2.2\nsamba-common-bin:\n  description: common files used by both the Samba server and client\n  downloads: 18675\n  homepage_uri: http://www.samba.org\n  name: samba-common-bin\n  section: net\n  version: 2:3.6.3-2ubuntu2.2\nsane-utils:\n  description: API library for scanners -- utilities\n  downloads: 5068\n  homepage_uri: http://www.sane-project.org\n  name: sane-utils\n  section: graphics\n  version: 1.0.22-7ubuntu1\nsat4j:\n  description: Efficient library of SAT solvers in Java\n  downloads: 1737\n  homepage_uri: http://www.sat4j.org\n  name: sat4j\n  section: universe/science\n  version: 2.3.1-1\nscala:\n  description: Scala programming language\n  downloads: 91\n  homepage_uri: http://www.scala-lang.org/\n  name: scala\n  section: universe/devel\n  version: 2.9.1.dfsg-3\nscala-library:\n  description: Scala standard library\n  downloads: 91\n  homepage_uri: http://www.scala-lang.org/\n  name: scala-library\n  section: universe/devel\n  version: 2.9.1.dfsg-3\nscreen:\n  description: terminal multiplexor with VT100/ANSI terminal emulation\n  downloads: 11357\n  homepage_uri: http://savannah.gnu.org/projects/screen\n  name: screen\n  section: misc\n  version: 4.0.3-14ubuntu8\nseabios:\n  description: legacy BIOS implementation which can be used as a coreboot payload\n  downloads: 0\n  homepage_uri: http://www.coreboot.org/SeaBIOS\n  name: seabios\n  section: misc\n  version: 0.6.2-0ubuntu2.1\nseahorse:\n  description: GNOME front end for GnuPG\n  downloads: 21816\n  homepage_uri: http://live.gnome.org/Seahorse\n  name: seahorse\n  section: gnome\n  version: 3.2.2-0ubuntu2\nsed:\n  description: The GNU sed stream editor\n  downloads: 110703\n  homepage_uri: http://www.gnu.org/software/sed/\n  name: sed\n  section: utils\n  version: 4.2.1-9\nsensible-utils:\n  description: Utilities for sensible alternative selection\n  downloads: 91557\n  homepage_uri: ''\n  name: sensible-utils\n  section: utils\n  version: 0.0.6ubuntu2\nsessioninstaller:\n  description: APT based installer using PackageKit's session DBus API\n  downloads: 570\n  homepage_uri: http://launchpad.net/sessioninstaller\n  name: sessioninstaller\n  section: gnome\n  version: 0.20+bzr128-0ubuntu1\nsgml-base:\n  description: SGML infrastructure and SGML catalog file support\n  downloads: 12945\n  homepage_uri: ''\n  name: sgml-base\n  section: text\n  version: 1.26+nmu1ubuntu1\nsgml-data:\n  description: common SGML and XML data\n  downloads: 418\n  homepage_uri: ''\n  name: sgml-data\n  section: text\n  version: 2.0.6\nshared-mime-info:\n  description: FreeDesktop.org shared MIME database and spec\n  downloads: 42034\n  homepage_uri: http://freedesktop.org/wiki/Software/shared-mime-info\n  name: shared-mime-info\n  section: misc\n  version: 1.0-0ubuntu4.1\nshotwell:\n  description: digital photo organizer\n  downloads: 4973\n  homepage_uri: http://yorba.org/shotwell/\n  name: shotwell\n  section: gnome\n  version: 0.12.3-0ubuntu0.1\nsimple-scan:\n  description: Simple Scanning Utility\n  downloads: 3048\n  homepage_uri: https://launchpad.net/simple-scan\n  name: simple-scan\n  section: gnome\n  version: 3.4.1-0ubuntu1.1\nsmbclient:\n  description: command-line SMB/CIFS clients for Unix\n  downloads: 6314\n  homepage_uri: http://www.samba.org\n  name: smbclient\n  section: net\n  version: 2:3.6.3-2ubuntu2.2\nsni-qt:\n  description: indicator support for Qt\n  downloads: 0\n  homepage_uri: https://launchpad.net/sni-qt\n  name: sni-qt\n  section: x11\n  version: 0.2.5-0ubuntu3\nsoftware-center:\n  description: Utility for browsing, installing, and removing software\n  downloads: 6211\n  homepage_uri: https://launchpad.net/software-center\n  name: software-center\n  section: gnome\n  version: 5.2.2.2\nsoftware-center-aptdaemon-plugins:\n  description: The aptdaemon plugins for software-center\n  downloads: 0\n  homepage_uri: ''\n  name: software-center-aptdaemon-plugins\n  section: gnome\n  version: 0.1.2\nsoftware-properties-common:\n  description: manage the repositories that you install software from (common)\n  downloads: 1119\n  homepage_uri: ''\n  name: software-properties-common\n  section: admin\n  version: 0.82.7.1\nsoftware-properties-gtk:\n  description: manage the repositories that you install software from (gtk)\n  downloads: 5491\n  homepage_uri: ''\n  name: software-properties-gtk\n  section: gnome\n  version: 0.82.7.1\nsound-theme-freedesktop:\n  description: freedesktop.org sound theme\n  downloads: 0\n  homepage_uri: http://www.freedesktop.org/\n  name: sound-theme-freedesktop\n  section: sound\n  version: 0.7.pristine-2\nspeech-dispatcher:\n  description: Common interface to speech synthesizers\n  downloads: 1624\n  homepage_uri: http://devel.freebsoft.org/speechd\n  name: speech-dispatcher\n  section: sound\n  version: 0.7.1-6ubuntu3\nsqlite3:\n  description: Command line interface for SQLite 3\n  downloads: 1855\n  homepage_uri: ''\n  name: sqlite3\n  section: misc\n  version: 3.7.9-2ubuntu1\nssh-askpass-gnome:\n  description: interactive X program to prompt users for a passphrase for ssh-add\n  downloads: 290\n  homepage_uri: http://www.openssh.org/\n  name: ssh-askpass-gnome\n  section: gnome\n  version: 1:5.9p1-5ubuntu1\nssh-import-id:\n  description: securely retrieve an SSH public key and install it locally\n  downloads: 1\n  homepage_uri: http://launchpad.net/ssh-import-id\n  name: ssh-import-id\n  section: misc\n  version: 2.10-0ubuntu1\nssl-cert:\n  description: simple debconf wrapper for OpenSSL\n  downloads: 32921\n  homepage_uri: ''\n  name: ssl-cert\n  section: utils\n  version: 1.0.28ubuntu0.1\nstow:\n  description: Organizer for /usr/local software packages\n  downloads: 122\n  homepage_uri: http://www.gnu.org/software/stow/\n  name: stow\n  section: universe/admin\n  version: 1.3.3-3.2\nstrace:\n  description: A system call tracer\n  downloads: 5412\n  homepage_uri: http://sourceforge.net/projects/strace/\n  name: strace\n  section: utils\n  version: 4.5.20-2.3ubuntu1\nsubversion:\n  description: Advanced version control system\n  downloads: 10196\n  homepage_uri: http://subversion.apache.org/\n  name: subversion\n  section: devel\n  version: 1.6.17dfsg-3ubuntu3\nsudo:\n  description: Provide limited super user privileges to specific users\n  downloads: 53108\n  homepage_uri: ''\n  name: sudo\n  section: admin\n  version: 1.8.3p1-1ubuntu3.3\nswig:\n  description: Generate scripting interfaces to C/C++ code\n  downloads: 365\n  homepage_uri: http://www.swig.org/\n  name: swig\n  section: interpreters\n  version: 2.0.4+really2.0.4-4ubuntu2\nswig2.0:\n  description: Generate scripting interfaces to C/C++ code\n  downloads: 188\n  homepage_uri: http://www.swig.org/\n  name: swig2.0\n  section: interpreters\n  version: 2.0.4+really2.0.4-4ubuntu2\nsynaptic:\n  description: Graphical package manager\n  downloads: 13131\n  homepage_uri: ''\n  name: synaptic\n  section: universe/admin\n  version: 0.75.9ubuntu1\nsyslinux:\n  description: collection of boot loaders\n  downloads: 1623\n  homepage_uri: http://syslinux.zytor.com/\n  name: syslinux\n  section: utils\n  version: 2:4.05+dfsg-2\nsyslinux-common:\n  description: collection of boot loaders (common files)\n  downloads: 2487\n  homepage_uri: http://syslinux.zytor.com/\n  name: syslinux-common\n  section: utils\n  version: 2:4.05+dfsg-2\nsyslinux-legacy:\n  description: Bootloader for Linux/i386 using MS-DOS floppies\n  downloads: 0\n  homepage_uri: http://syslinux.zytor.com/\n  name: syslinux-legacy\n  section: utils\n  version: 2:3.63+dfsg-2ubuntu5\nsystem-config-printer-common:\n  description: Printer configuration GUI\n  downloads: 0\n  homepage_uri: ''\n  name: system-config-printer-common\n  section: python\n  version: 1.3.8+20120201-0ubuntu8.1\nsystem-config-printer-gnome:\n  description: Printer configuration GUI\n  downloads: 23\n  homepage_uri: ''\n  name: system-config-printer-gnome\n  section: python\n  version: 1.3.8+20120201-0ubuntu8.1\nsystem-config-printer-udev:\n  description: Printer auto-configuration facility based on udev\n  downloads: 30742\n  homepage_uri: ''\n  name: system-config-printer-udev\n  section: python\n  version: 1.3.8+20120201-0ubuntu8.1\nsystem-tools-backends:\n  description: System Tools to manage computer configuration -- scripts\n  downloads: 8524\n  homepage_uri: http://system-tools-backends.freedesktop.org/\n  name: system-tools-backends\n  section: universe/gnome\n  version: 2.10.2-1ubuntu1\nsysv-rc:\n  description: System-V-like runlevel change mechanism\n  downloads: 110528\n  homepage_uri: http://savannah.nongnu.org/projects/sysvinit\n  name: sysv-rc\n  section: admin\n  version: 2.88dsf-13.10ubuntu11\nsysvinit-utils:\n  description: System-V-like utilities\n  downloads: 110539\n  homepage_uri: http://savannah.nongnu.org/projects/sysvinit\n  name: sysvinit-utils\n  section: admin\n  version: 2.88dsf-13.10ubuntu11\nt-coffee:\n  description: Multiple Sequence Alignment\n  downloads: 28\n  homepage_uri: http://www.tcoffee.org/Projects_home_page/t_coffee_home_page.html\n  name: t-coffee\n  section: science\n  version: 8.99-1\ntabix:\n  description: generic indexer for TAB-delimited genome position files\n  downloads: 6\n  homepage_uri: http://samtools.sourceforge.net/tabix.shtml\n  name: tabix\n  section: universe/science\n  version: 0.2.5-1ubuntu1\ntar:\n  description: GNU version of the tar archiving utility\n  downloads: 68906\n  homepage_uri: ''\n  name: tar\n  section: utils\n  version: 1.26-4ubuntu1\ntasksel:\n  description: Tool for selecting tasks for installation on Debian systems\n  downloads: 7430\n  homepage_uri: ''\n  name: tasksel\n  section: admin\n  version: 2.88ubuntu9\ntasksel-data:\n  description: Official tasks used for installation of Debian systems\n  downloads: 7286\n  homepage_uri: ''\n  name: tasksel-data\n  section: admin\n  version: 2.88ubuntu9\ntcl:\n  description: The Tool Command Language (default version) - run-time files\n  downloads: 7759\n  homepage_uri: ''\n  name: tcl\n  section: interpreters\n  version: 8.5.0-2\ntcl-dev:\n  description: The Tool Command Language (default version) - development files\n  downloads: 137\n  homepage_uri: ''\n  name: tcl-dev\n  section: devel\n  version: 8.5.0-2\ntcl-vtk:\n  description: Tcl bindings for VTK\n  downloads: 99\n  homepage_uri: http://www.vtk.org/\n  name: tcl-vtk\n  section: universe/interpreters\n  version: 5.8.0-5\ntcl8.5:\n  description: Tcl (the Tool Command Language) v8.5 - run-time files\n  downloads: 8983\n  homepage_uri: http://www.tcl.tk/\n  name: tcl8.5\n  section: interpreters\n  version: 8.5.11-1ubuntu1\ntcl8.5-dev:\n  description: Tcl (the Tool Command Language) v8.5 - development files\n  downloads: 397\n  homepage_uri: http://www.tcl.tk/\n  name: tcl8.5-dev\n  section: devel\n  version: 8.5.11-1ubuntu1\ntcpd:\n  description: Wietse Venema's TCP wrapper utilities\n  downloads: 27483\n  homepage_uri: ''\n  name: tcpd\n  section: net\n  version: 7.6.q-21\ntcpdump:\n  description: command-line network traffic analyzer\n  downloads: 3856\n  homepage_uri: http://www.tcpdump.org/\n  name: tcpdump\n  section: net\n  version: 4.2.1-1ubuntu2\ntcsh:\n  description: TENEX C Shell, an enhanced version of Berkeley csh\n  downloads: 3973\n  homepage_uri: http://www.tcsh.org/\n  name: tcsh\n  section: universe/shells\n  version: 6.17.06-2\ntelepathy-gabble:\n  description: Jabber/XMPP connection manager\n  downloads: 2206\n  homepage_uri: http://telepathy.freedesktop.org/wiki/\n  name: telepathy-gabble\n  section: net\n  version: 0.16.0-0ubuntu1\ntelepathy-haze:\n  description: Telepathy connection manager that uses libpurple\n  downloads: 1443\n  homepage_uri: ''\n  name: telepathy-haze\n  section: net\n  version: 0.6.0-0ubuntu1\ntelepathy-idle:\n  description: IRC connection manager for Telepathy\n  downloads: 248\n  homepage_uri: http://sourceforge.net/projects/telepathy-idle\n  name: telepathy-idle\n  section: net\n  version: 0.1.11-2\ntelepathy-indicator:\n  description: Desktop service to integrate Telepathy with the messaging menu.\n  downloads: 0\n  homepage_uri: https://launchpad.net/telepathy-indicator\n  name: telepathy-indicator\n  section: gnome\n  version: 0.2.1-0ubuntu1\ntelepathy-logger:\n  description: Telepathy logger service - Daemon\n  downloads: 2667\n  homepage_uri: http://telepathy.freedesktop.org/wiki/\n  name: telepathy-logger\n  section: libs\n  version: 0.4.0-0ubuntu1\ntelepathy-mission-control-5:\n  description: management daemon for Telepathy real-time communication framework\n  downloads: 11686\n  homepage_uri: http://telepathy.freedesktop.org/wiki/\n  name: telepathy-mission-control-5\n  section: net\n  version: 1:5.12.0-0ubuntu2\ntelepathy-salut:\n  description: Link-local XMPP connection manager for the Telepathy framework\n  downloads: 1766\n  homepage_uri: http://telepathy.freedesktop.org/wiki/\n  name: telepathy-salut\n  section: net\n  version: 0.8.0-0ubuntu1\ntelnet:\n  description: The telnet client\n  downloads: 14806\n  homepage_uri: ''\n  name: telnet\n  section: net\n  version: 0.17-36build1\ntex-common:\n  description: common infrastructure for building and installing TeX\n  downloads: 11089\n  homepage_uri: ''\n  name: tex-common\n  section: tex\n  version: '2.10'\ntexinfo:\n  description: Documentation system for on-line information and printed output\n  downloads: 9214\n  homepage_uri: ''\n  name: texinfo\n  section: text\n  version: 4.13a.dfsg.1-8ubuntu2\ntexlive-base:\n  description: 'TeX Live: Essential programs and files'\n  downloads: 10342\n  homepage_uri: http://www.tug.org/texlive/\n  name: texlive-base\n  section: tex\n  version: 2009-15\ntexlive-binaries:\n  description: Binaries for TeX Live\n  downloads: 10483\n  homepage_uri: http://www.tug.org/texlive\n  name: texlive-binaries\n  section: tex\n  version: 2009-11ubuntu2\ntexlive-common:\n  description: 'TeX Live: Base component'\n  downloads: 10417\n  homepage_uri: http://www.tug.org/texlive/\n  name: texlive-common\n  section: tex\n  version: 2009-15\ntexlive-doc-base:\n  description: 'TeX Live: TeX Live documentation'\n  downloads: 9632\n  homepage_uri: http://www.tug.org/texlive\n  name: texlive-doc-base\n  section: tex\n  version: 2009-2\ntexlive-extra-utils:\n  description: 'TeX Live: TeX auxiliary programs'\n  downloads: 1688\n  homepage_uri: http://www.tug.org/texlive/\n  name: texlive-extra-utils\n  section: tex\n  version: 2009-10ubuntu1\ntexlive-font-utils:\n  description: 'TeX Live: TeX and Outline font utilities'\n  downloads: 1727\n  homepage_uri: http://www.tug.org/texlive/\n  name: texlive-font-utils\n  section: tex\n  version: 2009-10ubuntu1\ntexlive-fonts-recommended:\n  description: 'TeX Live: Recommended fonts'\n  downloads: 3743\n  homepage_uri: http://www.tug.org/texlive/\n  name: texlive-fonts-recommended\n  section: tex\n  version: 2009-15\ntexlive-fonts-recommended-doc:\n  description: 'TeX Live: Documentation files for texlive-fonts-recommended'\n  downloads: 0\n  homepage_uri: http://www.tug.org/texlive/\n  name: texlive-fonts-recommended-doc\n  section: doc\n  version: 2009-15\ntexlive-generic-recommended:\n  description: 'TeX Live: Recommended generic packages'\n  downloads: 1684\n  homepage_uri: http://www.tug.org/texlive/\n  name: texlive-generic-recommended\n  section: tex\n  version: 2009-15\ntexlive-latex-base:\n  description: 'TeX Live: Basic LaTeX packages'\n  downloads: 7374\n  homepage_uri: http://www.tug.org/texlive/\n  name: texlive-latex-base\n  section: tex\n  version: 2009-15\ntexlive-latex-base-doc:\n  description: 'TeX Live: Documentation files for texlive-latex-base'\n  downloads: 0\n  homepage_uri: http://www.tug.org/texlive/\n  name: texlive-latex-base-doc\n  section: doc\n  version: 2009-15\ntexlive-latex-extra:\n  description: 'TeX Live: LaTeX supplementary packages'\n  downloads: 1889\n  homepage_uri: http://www.tug.org/texlive/\n  name: texlive-latex-extra\n  section: tex\n  version: 2009-10ubuntu1\ntexlive-latex-extra-doc:\n  description: 'TeX Live: Documentation files for texlive-latex-extra'\n  downloads: 0\n  homepage_uri: http://www.tug.org/texlive/\n  name: texlive-latex-extra-doc\n  section: doc\n  version: 2009-10ubuntu1\ntexlive-latex-recommended:\n  description: 'TeX Live: LaTeX recommended packages'\n  downloads: 2444\n  homepage_uri: http://www.tug.org/texlive/\n  name: texlive-latex-recommended\n  section: tex\n  version: 2009-15\ntexlive-latex-recommended-doc:\n  description: 'TeX Live: Documentation files for texlive-latex-recommended'\n  downloads: 0\n  homepage_uri: http://www.tug.org/texlive/\n  name: texlive-latex-recommended-doc\n  section: doc\n  version: 2009-15\ntexlive-luatex:\n  description: 'TeX Live: LuaTeX packages'\n  downloads: 319\n  homepage_uri: http://www.tug.org/texlive/\n  name: texlive-luatex\n  section: tex\n  version: 2009-15\ntexlive-pictures:\n  description: 'TeX Live: Graphics packages and programs'\n  downloads: 1387\n  homepage_uri: http://www.tug.org/texlive/\n  name: texlive-pictures\n  section: tex\n  version: 2009-15\ntexlive-pictures-doc:\n  description: 'TeX Live: Documentation files for texlive-pictures'\n  downloads: 0\n  homepage_uri: http://www.tug.org/texlive/\n  name: texlive-pictures-doc\n  section: doc\n  version: 2009-15\ntexlive-pstricks:\n  description: 'TeX Live: PSTricks packages'\n  downloads: 1134\n  homepage_uri: http://www.tug.org/texlive/\n  name: texlive-pstricks\n  section: tex\n  version: 2009-10ubuntu1\ntexlive-pstricks-doc:\n  description: 'TeX Live: Documentation files for texlive-pstricks'\n  downloads: 0\n  homepage_uri: http://www.tug.org/texlive/\n  name: texlive-pstricks-doc\n  section: doc\n  version: 2009-10ubuntu1\nthunderbird:\n  description: Email, RSS and newsgroup client with integrated spam filter\n  downloads: 181\n  homepage_uri: ''\n  name: thunderbird\n  section: mail\n  version: 13.0.1+build1-0ubuntu0.12.04.1\nthunderbird-globalmenu:\n  description: Unity appmenu integration for Thunderbird\n  downloads: 7\n  homepage_uri: ''\n  name: thunderbird-globalmenu\n  section: web\n  version: 13.0.1+build1-0ubuntu0.12.04.1\nthunderbird-gnome-support:\n  description: Email, RSS and newsgroup client - GNOME support\n  downloads: 0\n  homepage_uri: ''\n  name: thunderbird-gnome-support\n  section: mail\n  version: 13.0.1+build1-0ubuntu0.12.04.1\ntigr-glimmer:\n  description: Gene detection in archea and bacteria\n  downloads: 24\n  homepage_uri: http://www.cbcb.umd.edu/software/glimmer\n  name: tigr-glimmer\n  section: universe/science\n  version: 3.02-2build1\ntime:\n  description: The GNU time program for measuring cpu resource usage\n  downloads: 8630\n  homepage_uri: ''\n  name: time\n  section: utils\n  version: 1.7-23.1\ntipa:\n  description: system for processing phonetic symbols in LaTeX\n  downloads: 3099\n  homepage_uri: http://www.ctan.org/tex-archive/fonts/tipa/\n  name: tipa\n  section: tex\n  version: 2:1.3-15\ntk:\n  description: The Tk toolkit for Tcl and X11 (default version) - run-time files\n  downloads: 2931\n  homepage_uri: ''\n  name: tk\n  section: libs\n  version: 8.5.0-2\ntk-dev:\n  description: The Tk toolkit for Tcl and X11 (default version) - development files\n  downloads: 35\n  homepage_uri: ''\n  name: tk-dev\n  section: devel\n  version: 8.5.0-2\ntk8.5:\n  description: Tk toolkit for Tcl and X11, v8.5 - run-time files\n  downloads: 5598\n  homepage_uri: http://www.tcl.tk/\n  name: tk8.5\n  section: libs\n  version: 8.5.11-1\ntk8.5-dev:\n  description: Tk toolkit for Tcl and X11, v8.5 - development files\n  downloads: 276\n  homepage_uri: http://www.tcl.tk/\n  name: tk8.5-dev\n  section: devel\n  version: 8.5.11-1\ntmux:\n  description: terminal multiplexer\n  downloads: 1403\n  homepage_uri: http://tmux.sourceforge.net/\n  name: tmux\n  section: admin\n  version: 1.6-1ubuntu1\ntntnet:\n  description: modular, multithreaded web application server for C++\n  downloads: 25\n  homepage_uri: http://www.tntnet.org/\n  name: tntnet\n  section: universe/web\n  version: 2.0+dfsg1-2\ntntnet-runtime:\n  description: Tntnet runtime system\n  downloads: 15\n  homepage_uri: http://www.tntnet.org/\n  name: tntnet-runtime\n  section: universe/web\n  version: 2.0+dfsg1-2\ntokyocabinet-bin:\n  description: Tokyo Cabinet Database Utilities\n  downloads: 11\n  homepage_uri: ''\n  name: tokyocabinet-bin\n  section: universe/utils\n  version: 1.4.37-6.1build1\ntoshset:\n  description: Access much of the Toshiba laptop hardware interface\n  downloads: 144\n  homepage_uri: http://www.schwieters.org/toshset/\n  name: toshset\n  section: utils\n  version: 1.76-2\ntotem:\n  description: Simple media player for the GNOME desktop based on GStreamer\n  downloads: 28899\n  homepage_uri: http://www.gnome.org/projects/totem/\n  name: totem\n  section: gnome\n  version: 3.0.1-0ubuntu21\ntotem-common:\n  description: Data files for the Totem media player\n  downloads: 28547\n  homepage_uri: http://www.gnome.org/projects/totem/\n  name: totem-common\n  section: gnome\n  version: 3.0.1-0ubuntu21\ntotem-mozilla:\n  description: Totem Mozilla plugin\n  downloads: 19391\n  homepage_uri: http://www.gnome.org/projects/totem/\n  name: totem-mozilla\n  section: web\n  version: 3.0.1-0ubuntu21\ntotem-plugins:\n  description: Plugins for the Totem media player\n  downloads: 14520\n  homepage_uri: http://www.gnome.org/projects/totem/\n  name: totem-plugins\n  section: gnome\n  version: 3.0.1-0ubuntu21\ntraceroute:\n  description: Traces the route taken by packets over an IPv4/IPv6 network\n  downloads: 13456\n  homepage_uri: http://traceroute.sourceforge.net/\n  name: traceroute\n  section: universe/net\n  version: 1:2.0.18-1\ntransmission-common:\n  description: lightweight BitTorrent client (common files)\n  downloads: 1\n  homepage_uri: http://www.transmissionbt.com/\n  name: transmission-common\n  section: net\n  version: 2.51-0ubuntu1\ntransmission-gtk:\n  description: lightweight BitTorrent client (GTK interface)\n  downloads: 16110\n  homepage_uri: http://www.transmissionbt.com/\n  name: transmission-gtk\n  section: net\n  version: 2.51-0ubuntu1\ntree:\n  description: displays directory tree, in color\n  downloads: 1337\n  homepage_uri: http://mama.indstate.edu/users/ice/tree/\n  name: tree\n  section: universe/utils\n  version: 1.5.3-2\ntree-puzzle:\n  description: Reconstruction of phylogenetic trees by maximum likelihood\n  downloads: 20\n  homepage_uri: http://www.tree-puzzle.de\n  name: tree-puzzle\n  section: science\n  version: 5.2-7\ntree-puzzle-doc:\n  description: Reconstruction of phylogenetic trees by maximum likelihood\n  downloads: 4\n  homepage_uri: http://www.tree-puzzle.de\n  name: tree-puzzle-doc\n  section: universe/science\n  version: 5.2-7\ntreeviewx:\n  description: Displays and prints phylogenetic trees\n  downloads: 126\n  homepage_uri: http://darwin.zoology.gla.ac.uk/~rpage/treeviewx/\n  name: treeviewx\n  section: science\n  version: 0.5.1+20100823-1\nttf-dejavu-core:\n  description: Vera font family derivate with additional characters\n  downloads: 11606\n  homepage_uri: http://dejavu-fonts.org/\n  name: ttf-dejavu-core\n  section: x11\n  version: 2.33-2ubuntu1\nttf-dejavu-extra:\n  description: Vera font family derivate with additional characters\n  downloads: 1008\n  homepage_uri: http://dejavu-fonts.org/\n  name: ttf-dejavu-extra\n  section: x11\n  version: 2.33-2ubuntu1\nttf-freefont:\n  description: Freefont Serif, Sans and Mono Truetype fonts\n  downloads: 3434\n  homepage_uri: http://savannah.gnu.org/projects/freefont/\n  name: ttf-freefont\n  section: x11\n  version: 20100919-1\nttf-indic-fonts-core:\n  description: Core collection of free fonts for languages of India\n  downloads: 0\n  homepage_uri: http://debian-in.alioth.debian.org/\n  name: ttf-indic-fonts-core\n  section: x11\n  version: 1:0.5.11ubuntu1\nttf-liberation:\n  description: transitional dummy package\n  downloads: 1088\n  homepage_uri: https://fedorahosted.org/liberation-fonts/\n  name: ttf-liberation\n  section: fonts\n  version: 1.07.0-2\nttf-punjabi-fonts:\n  description: Free TrueType fonts for the Punjabi language\n  downloads: 1\n  homepage_uri: http://debian-in.alioth.debian.org/\n  name: ttf-punjabi-fonts\n  section: x11\n  version: 1:0.5.11ubuntu1\nttf-ubuntu-font-family:\n  description: Ubuntu Font Family, sans-serif typeface hinted for clarity\n  downloads: 0\n  homepage_uri: http://font.ubuntu.com/\n  name: ttf-ubuntu-font-family\n  section: x11\n  version: 0.80-0ubuntu2\nttf-wqy-microhei:\n  description: A droid derived Sans-Seri style CJK font\n  downloads: 0\n  homepage_uri: http://wqy.sourceforge.net/\n  name: ttf-wqy-microhei\n  section: x11\n  version: 0.2.0-beta-1ubuntu1\ntzdata:\n  description: time zone and daylight-saving time data\n  downloads: 110141\n  homepage_uri: http://www.iana.org/time-zones\n  name: tzdata\n  section: libs\n  version: 2012b-1\ntzdata-java:\n  description: time zone and daylight-saving time data for use by java runtimes\n  downloads: 12801\n  homepage_uri: http://www.iana.org/time-zones\n  name: tzdata-java\n  section: libs\n  version: 2012b-1\nubuntu-artwork:\n  description: Ubuntu themes and artwork\n  downloads: 0\n  homepage_uri: ''\n  name: ubuntu-artwork\n  section: gnome\n  version: '57'\nubuntu-desktop:\n  description: The Ubuntu desktop system\n  downloads: 0\n  homepage_uri: ''\n  name: ubuntu-desktop\n  section: metapackages\n  version: '1.267'\nubuntu-docs:\n  description: Ubuntu Desktop Guide\n  downloads: 0\n  homepage_uri: https://launchpad.net/ubuntu-doc\n  name: ubuntu-docs\n  section: text\n  version: 12.04.5\nubuntu-extras-keyring:\n  description: GnuPG keys of the Ubuntu extras archive\n  downloads: 0\n  homepage_uri: ''\n  name: ubuntu-extras-keyring\n  section: misc\n  version: 2010.09.27\nubuntu-keyring:\n  description: GnuPG keys of the Ubuntu archive\n  downloads: 0\n  homepage_uri: ''\n  name: ubuntu-keyring\n  section: misc\n  version: 2011.11.21\nubuntu-minimal:\n  description: Minimal core of Ubuntu\n  downloads: 0\n  homepage_uri: ''\n  name: ubuntu-minimal\n  section: metapackages\n  version: '1.267'\nubuntu-mono:\n  description: Ubuntu Mono Icon theme\n  downloads: 0\n  homepage_uri: ''\n  name: ubuntu-mono\n  section: gnome\n  version: 0.0.40\nubuntu-sounds:\n  description: Ubuntu's GNOME audio theme\n  downloads: 0\n  homepage_uri: ''\n  name: ubuntu-sounds\n  section: gnome\n  version: '0.13'\nubuntu-sso-client:\n  description: Ubuntu Single Sign-On client\n  downloads: 13\n  homepage_uri: ''\n  name: ubuntu-sso-client\n  section: python\n  version: 3.0.0-0ubuntu2\nubuntu-sso-client-gtk:\n  description: Ubuntu Single Sign-On client - GTK+ frontend\n  downloads: 0\n  homepage_uri: ''\n  name: ubuntu-sso-client-gtk\n  section: python\n  version: 3.0.0-0ubuntu2\nubuntu-standard:\n  description: The Ubuntu standard system\n  downloads: 0\n  homepage_uri: ''\n  name: ubuntu-standard\n  section: metapackages\n  version: '1.267'\nubuntu-system-service:\n  description: Dbus service to set various system-wide configurations\n  downloads: 4\n  homepage_uri: ''\n  name: ubuntu-system-service\n  section: admin\n  version: 0.2.2\nubuntu-wallpapers:\n  description: Ubuntu Wallpapers\n  downloads: 0\n  homepage_uri: ''\n  name: ubuntu-wallpapers\n  section: metapackages\n  version: 0.34.1\nubuntu-wallpapers-precise:\n  description: Ubuntu 12.04 Wallpapers\n  downloads: 0\n  homepage_uri: ''\n  name: ubuntu-wallpapers-precise\n  section: x11\n  version: 0.34.1\nubuntuone-client:\n  description: Ubuntu One client\n  downloads: 14\n  homepage_uri: https://one.ubuntu.com\n  name: ubuntuone-client\n  section: net\n  version: 3.0.1-0ubuntu1.0.1\nubuntuone-client-gnome:\n  description: Ubuntu One client GNOME integration\n  downloads: 4\n  homepage_uri: https://one.ubuntu.com\n  name: ubuntuone-client-gnome\n  section: gnome\n  version: 3.0.1-0ubuntu1\nubuntuone-control-panel:\n  description: Ubuntu One Control Panel\n  downloads: 0\n  homepage_uri: ''\n  name: ubuntuone-control-panel\n  section: python\n  version: 3.0.1-0ubuntu1\nubuntuone-couch:\n  description: Ubuntu One CouchDB\n  downloads: 2\n  homepage_uri: http://launchpad.net/ubuntuone-couch\n  name: ubuntuone-couch\n  section: python\n  version: 0.3.0-0ubuntu4\nubuntuone-installer:\n  description: Ubuntu One Installer\n  downloads: 4\n  homepage_uri: ''\n  name: ubuntuone-installer\n  section: python\n  version: 3.0.0-0ubuntu1\nucf:\n  description: 'Update Configuration File: preserve user changes to config files.'\n  downloads: 88721\n  homepage_uri: ''\n  name: ucf\n  section: utils\n  version: 3.0025+nmu2ubuntu1\nudev:\n  description: rule-based device node and kernel event manager\n  downloads: 84923\n  homepage_uri: http://www.kernel.org/pub/linux/utils/kernel/hotplug/udev.html\n  name: udev\n  section: admin\n  version: 175-0ubuntu9\nudisks:\n  description: storage media interface\n  downloads: 46689\n  homepage_uri: http://www.freedesktop.org/wiki/Software/udisks\n  name: udisks\n  section: admin\n  version: 1.0.4-5ubuntu2\nufw:\n  description: program for managing a Netfilter firewall\n  downloads: 1095\n  homepage_uri: https://launchpad.net/ufw\n  name: ufw\n  section: admin\n  version: 0.31.1-1\nugene:\n  description: integrated bioinformatics toolkit\n  downloads: 1\n  homepage_uri: http://ugene.unipro.ru\n  name: ugene\n  section: universe/science\n  version: 1.9.8+repack-0ubuntu3\nugene-data:\n  description: required data for UGENE - integrated bioinformatics toolkit\n  downloads: 0\n  homepage_uri: http://ugene.unipro.ru\n  name: ugene-data\n  section: universe/science\n  version: 1.9.8+repack-0ubuntu3\nunattended-upgrades:\n  description: automatic installation of security upgrades\n  downloads: 6200\n  homepage_uri: ''\n  name: unattended-upgrades\n  section: admin\n  version: '0.76'\nunity:\n  description: Interface designed for efficiency of space and interaction.\n  downloads: 0\n  homepage_uri: https://launchpad.net/unity\n  name: unity\n  section: gnome\n  version: 5.12-0ubuntu1.1\nunity-2d:\n  description: Unity interface for non-accelerated graphics cards\n  downloads: 0\n  homepage_uri: ''\n  name: unity-2d\n  section: x11\n  version: 5.12.0-0ubuntu1.1\nunity-2d-common:\n  description: Common files for Unity 2D Shell\n  downloads: 0\n  homepage_uri: ''\n  name: unity-2d-common\n  section: x11\n  version: 5.12.0-0ubuntu1.1\nunity-2d-panel:\n  description: Unity 2D Panel\n  downloads: 0\n  homepage_uri: ''\n  name: unity-2d-panel\n  section: x11\n  version: 5.12.0-0ubuntu1.1\nunity-2d-shell:\n  description: Dash and Launcher for the Unity 2D environment\n  downloads: 0\n  homepage_uri: ''\n  name: unity-2d-shell\n  section: x11\n  version: 5.12.0-0ubuntu1.1\nunity-2d-spread:\n  description: Unity 2D Spread\n  downloads: 0\n  homepage_uri: ''\n  name: unity-2d-spread\n  section: x11\n  version: 5.12.0-0ubuntu1.1\nunity-asset-pool:\n  description: Unity Assets Pool\n  downloads: 0\n  homepage_uri: https://launchpad.net/unity-asset-pool\n  name: unity-asset-pool\n  section: gnome\n  version: 0.8.23-0ubuntu1\nunity-common:\n  description: Common files for the Unity interface.\n  downloads: 0\n  homepage_uri: https://launchpad.net/unity\n  name: unity-common\n  section: gnome\n  version: 5.12-0ubuntu1.1\nunity-greeter:\n  description: Unity Greeter\n  downloads: 1\n  homepage_uri: https://launchpad.net/unity-greeter\n  name: unity-greeter\n  section: x11\n  version: 0.2.8-0ubuntu1.1\nunity-lens-applications:\n  description: Application lens for unity\n  downloads: 0\n  homepage_uri: https://launchpad.net/unity-lens-applications\n  name: unity-lens-applications\n  section: gnome\n  version: 5.12.0-0ubuntu1\nunity-lens-files:\n  description: File lens for unity\n  downloads: 0\n  homepage_uri: https://launchpad.net/unity-lens-files\n  name: unity-lens-files\n  section: gnome\n  version: 5.10.0-0ubuntu1\nunity-lens-music:\n  description: Music lens for unity\n  downloads: 0\n  homepage_uri: https://launchpad.net/unity-lens-music\n  name: unity-lens-music\n  section: gnome\n  version: 5.12.0-0ubuntu1\nunity-lens-video:\n  description: Unity Video lens\n  downloads: 0\n  homepage_uri: https://launchpad.net/unity-lens-videos\n  name: unity-lens-video\n  section: gnome\n  version: 0.3.5-0ubuntu1\nunity-scope-musicstores:\n  description: Store music lens for unity\n  downloads: 0\n  homepage_uri: https://launchpad.net/unity-lens-music\n  name: unity-scope-musicstores\n  section: gnome\n  version: 5.12.0-0ubuntu1\nunity-scope-video-remote:\n  description: Remote videos engine\n  downloads: 0\n  homepage_uri: https://launchpad.net/unity-lens-videos\n  name: unity-scope-video-remote\n  section: gnome\n  version: 0.3.5-0ubuntu2\nunity-services:\n  description: Services for the Unity interface\n  downloads: 0\n  homepage_uri: https://launchpad.net/unity\n  name: unity-services\n  section: gnome\n  version: 5.12-0ubuntu1.1\nuno-libs3:\n  description: LibreOffice UNO runtime environment -- public shared libraries\n  downloads: 26384\n  homepage_uri: http://www.libreoffice.org\n  name: uno-libs3\n  section: libs\n  version: 3.5.3-0ubuntu1\nunzip:\n  description: De-archiver for .zip files\n  downloads: 31921\n  homepage_uri: http://www.info-zip.org/UnZip.html\n  name: unzip\n  section: utils\n  version: 6.0-4ubuntu1\nupdate-inetd:\n  description: inetd configuration file updater\n  downloads: 39700\n  homepage_uri: ''\n  name: update-inetd\n  section: admin\n  version: '4.41'\nupdate-manager:\n  description: GNOME application that manages apt updates\n  downloads: 396\n  homepage_uri: ''\n  name: update-manager\n  section: gnome\n  version: 1:0.156.14.5\nupdate-manager-core:\n  description: manage release upgrades\n  downloads: 12973\n  homepage_uri: ''\n  name: update-manager-core\n  section: admin\n  version: 1:0.156.14.5\nupdate-notifier:\n  description: Daemon which notifies about package updates\n  downloads: 21493\n  homepage_uri: ''\n  name: update-notifier\n  section: gnome\n  version: 0.119ubuntu8.4\nupdate-notifier-common:\n  description: Files shared between update-notifier and other packages\n  downloads: 27885\n  homepage_uri: ''\n  name: update-notifier-common\n  section: gnome\n  version: 0.119ubuntu8.4\nupower:\n  description: abstraction for power management\n  downloads: 35238\n  homepage_uri: http://upower.freedesktop.org/\n  name: upower\n  section: admin\n  version: 0.9.15-3git1\nupstart:\n  description: event-based init daemon\n  downloads: 131\n  homepage_uri: http://upstart.ubuntu.com/\n  name: upstart\n  section: admin\n  version: 1.5-0ubuntu7\nure:\n  description: LibreOffice UNO runtime environment\n  downloads: 27417\n  homepage_uri: http://www.libreoffice.org\n  name: ure\n  section: libs\n  version: 3.5.3-0ubuntu1\nureadahead:\n  description: Read required files in advance\n  downloads: 21\n  homepage_uri: ''\n  name: ureadahead\n  section: admin\n  version: 0.100.0-12\nusb-creator-common:\n  description: create a startup disk using a CD or disc image (common files)\n  downloads: 3\n  homepage_uri: ''\n  name: usb-creator-common\n  section: admin\n  version: 0.2.38\nusb-creator-gtk:\n  description: create a startup disk using a CD or disc image (for GNOME)\n  downloads: 17\n  homepage_uri: ''\n  name: usb-creator-gtk\n  section: admin\n  version: 0.2.38\nusb-modeswitch:\n  description: mode switching tool for controlling \"flip flop\" USB devices\n  downloads: 8917\n  homepage_uri: http://www.draisberghof.de/usb_modeswitch/\n  name: usb-modeswitch\n  section: comm\n  version: 1.2.3+repack0-1ubuntu2\nusb-modeswitch-data:\n  description: mode switching data for usb-modeswitch\n  downloads: 12921\n  homepage_uri: http://www.draisberghof.de/usb_modeswitch/\n  name: usb-modeswitch-data\n  section: comm\n  version: 20120120-0ubuntu1\nusbmuxd:\n  description: USB multiplexor daemon for iPhone and iPod Touch devices\n  downloads: 22142\n  homepage_uri: http://marcansoft.com/blog/iphonelinux/usbmuxd/\n  name: usbmuxd\n  section: utils\n  version: 1.0.7-2\nusbutils:\n  description: Linux USB utilities\n  downloads: 33299\n  homepage_uri: ''\n  name: usbutils\n  section: utils\n  version: 1:005-1\nutil-linux:\n  description: Miscellaneous system utilities\n  downloads: 110495\n  homepage_uri: http://userweb.kernel.org/~kzak/util-linux/\n  name: util-linux\n  section: utils\n  version: 2.20.1-1ubuntu3\nuuid-runtime:\n  description: runtime components for the Universally Unique ID library\n  downloads: 1620\n  homepage_uri: http://userweb.kernel.org/~kzak/util-linux/\n  name: uuid-runtime\n  section: libs\n  version: 2.20.1-1ubuntu3\nvbetool:\n  description: run real-mode video BIOS code to alter hardware state\n  downloads: 4680\n  homepage_uri: http://www.codon.org.uk/~mjg59/vbetool/\n  name: vbetool\n  section: utils\n  version: 1.1-2ubuntu1\nvgabios:\n  description: VGA BIOS software for the Bochs and Qemu emulated VGA card\n  downloads: 0\n  homepage_uri: http://savannah.nongnu.org/projects/vgabios/\n  name: vgabios\n  section: misc\n  version: 0.6c-2ubuntu3\nvim:\n  description: Vi IMproved - enhanced vi editor\n  downloads: 20043\n  homepage_uri: http://www.vim.org/\n  name: vim\n  section: editors\n  version: 2:7.3.429-2ubuntu2.1\nvim-common:\n  description: Vi IMproved - Common files\n  downloads: 45172\n  homepage_uri: http://www.vim.org/\n  name: vim-common\n  section: editors\n  version: 2:7.3.429-2ubuntu2.1\nvim-runtime:\n  description: Vi IMproved - Runtime files\n  downloads: 3771\n  homepage_uri: http://www.vim.org/\n  name: vim-runtime\n  section: editors\n  version: 2:7.3.429-2ubuntu2.1\nvim-tiny:\n  description: Vi IMproved - enhanced vi editor - compact version\n  downloads: 13838\n  homepage_uri: http://www.vim.org/\n  name: vim-tiny\n  section: editors\n  version: 2:7.3.429-2ubuntu2.1\nvino:\n  description: VNC server for GNOME\n  downloads: 7110\n  homepage_uri: http://live.gnome.org/Vino\n  name: vino\n  section: gnome\n  version: 3.4.2-0ubuntu1\nvirtualbox-4.1:\n  description: Oracle VM VirtualBox\n  downloads: 3593\n  homepage_uri: ''\n  name: virtualbox-4.1\n  section: contrib/misc\n  version: 4.1.16-78094~Ubuntu~precise\nw3m:\n  description: WWW browsable pager with excellent tables/frames support\n  downloads: 29419\n  homepage_uri: http://sourceforge.net/projects/w3m\n  name: w3m\n  section: text\n  version: 0.5.3-5ubuntu1\nwatershed:\n  description: reduce superfluous executions of idempotent command\n  downloads: 4\n  homepage_uri: ''\n  name: watershed\n  section: admin\n  version: '6'\nwdiff:\n  description: Compares two files word by word\n  downloads: 1669\n  homepage_uri: http://www.gnu.org/software/wdiff/\n  name: wdiff\n  section: text\n  version: 0.6.5-1\nwget:\n  description: retrieves files from the web\n  downloads: 33615\n  homepage_uri: http://www.gnu.org/software/wget/\n  name: wget\n  section: web\n  version: 1.13.4-2ubuntu1\nwhiptail:\n  description: Displays user-friendly dialog boxes from shell scripts\n  downloads: 51026\n  homepage_uri: https://fedorahosted.org/newt/\n  name: whiptail\n  section: utils\n  version: 0.52.11-2ubuntu10\nwhois:\n  description: intelligent WHOIS client\n  downloads: 12086\n  homepage_uri: ''\n  name: whois\n  section: net\n  version: 5.0.15ubuntu2\nwhoopsie:\n  description: Ubuntu crash database submission daemon\n  downloads: 1\n  homepage_uri: http://wiki.ubuntu.com/ErrorTracker\n  name: whoopsie\n  section: utils\n  version: 0.1.32\nwireless-regdb:\n  description: wireless regulatory database\n  downloads: 9537\n  homepage_uri: http://wireless.kernel.org/en/developers/Regulatory/#Theregulatorydatabase\n  name: wireless-regdb\n  section: net\n  version: 2011.04.28-1ubuntu3\nwireless-tools:\n  description: Tools for manipulating Linux Wireless Extensions\n  downloads: 9059\n  homepage_uri: http://www.hpl.hp.com/personal/Jean_Tourrilhes/Linux/Tools.html\n  name: wireless-tools\n  section: net\n  version: 30~pre9-5ubuntu2\nwodim:\n  description: command line CD/DVD writing tool\n  downloads: 4799\n  homepage_uri: ''\n  name: wodim\n  section: otherosfs\n  version: 9:1.1.11-2ubuntu2\nwpasupplicant:\n  description: client support for WPA and WPA2 (IEEE 802.11i)\n  downloads: 42188\n  homepage_uri: http://w1.fi/wpa_supplicant/\n  name: wpasupplicant\n  section: net\n  version: 0.7.3-6ubuntu2\nx11-apps:\n  description: X applications\n  downloads: 6228\n  homepage_uri: ''\n  name: x11-apps\n  section: x11\n  version: 7.6+5ubuntu1\nx11-common:\n  description: X Window System (X.Org) infrastructure\n  downloads: 54261\n  homepage_uri: ''\n  name: x11-common\n  section: x11\n  version: 1:7.6+12ubuntu1\nx11-session-utils:\n  description: X session utilities\n  downloads: 4761\n  homepage_uri: ''\n  name: x11-session-utils\n  section: x11\n  version: 7.6+2\nx11-utils:\n  description: X11 utilities\n  downloads: 34280\n  homepage_uri: ''\n  name: x11-utils\n  section: x11\n  version: 7.6+4\nx11-xfs-utils:\n  description: X font server utilities\n  downloads: 4727\n  homepage_uri: ''\n  name: x11-xfs-utils\n  section: x11\n  version: 7.6+1\nx11-xkb-utils:\n  description: X11 XKB utilities\n  downloads: 46803\n  homepage_uri: ''\n  name: x11-xkb-utils\n  section: x11\n  version: 7.6+4\nx11-xserver-utils:\n  description: X server utilities\n  downloads: 44459\n  homepage_uri: ''\n  name: x11-xserver-utils\n  section: x11\n  version: 7.6+3\nx11proto-composite-dev:\n  description: X11 Composite extension wire protocol\n  downloads: 1215\n  homepage_uri: ''\n  name: x11proto-composite-dev\n  section: x11\n  version: 1:0.4.2-2\nx11proto-core-dev:\n  description: X11 core wire protocol and auxiliary headers\n  downloads: 3904\n  homepage_uri: ''\n  name: x11proto-core-dev\n  section: x11\n  version: 7.0.22-1\nx11proto-damage-dev:\n  description: X11 Damage extension wire protocol\n  downloads: 1098\n  homepage_uri: ''\n  name: x11proto-damage-dev\n  section: x11\n  version: 1:1.2.1-2\nx11proto-fixes-dev:\n  description: X11 Fixes extension wire protocol\n  downloads: 1363\n  homepage_uri: ''\n  name: x11proto-fixes-dev\n  section: x11\n  version: 1:5.0-2ubuntu1\nx11proto-input-dev:\n  description: X11 Input extension wire protocol\n  downloads: 2528\n  homepage_uri: ''\n  name: x11proto-input-dev\n  section: x11\n  version: 2.1.99.6-1\nx11proto-kb-dev:\n  description: X11 XKB extension wire protocol\n  downloads: 2051\n  homepage_uri: ''\n  name: x11proto-kb-dev\n  section: x11\n  version: 1.0.5-2\nx11proto-randr-dev:\n  description: X11 RandR extension wire protocol\n  downloads: 1617\n  homepage_uri: ''\n  name: x11proto-randr-dev\n  section: x11\n  version: 1.4.0+git20101207.0d32bb07-0ubuntu2\nx11proto-render-dev:\n  description: X11 Render extension wire protocol\n  downloads: 1875\n  homepage_uri: ''\n  name: x11proto-render-dev\n  section: x11\n  version: 2:0.11.1-2\nx11proto-scrnsaver-dev:\n  description: X11 Screen Saver extension wire protocol\n  downloads: 356\n  homepage_uri: ''\n  name: x11proto-scrnsaver-dev\n  section: x11\n  version: 1.2.1-2\nx11proto-xext-dev:\n  description: X11 various extension wire protocol\n  downloads: 2139\n  homepage_uri: ''\n  name: x11proto-xext-dev\n  section: x11\n  version: 7.2.0-3\nx11proto-xinerama-dev:\n  description: X11 Xinerama extension wire protocol\n  downloads: 1109\n  homepage_uri: ''\n  name: x11proto-xinerama-dev\n  section: x11\n  version: 1.2.1-2\nxauth:\n  description: X authentication utility\n  downloads: 38431\n  homepage_uri: ''\n  name: xauth\n  section: x11\n  version: 1:1.0.6-1\nxaw3dg:i386:\n  description: Xaw3d widget set\n  downloads: 0\n  homepage_uri: ''\n  name: xaw3dg:i386\n  section: x11\n  version: 1.5+E-18.1ubuntu1\nxbitmaps:\n  description: Base X bitmaps\n  downloads: 13830\n  homepage_uri: ''\n  name: xbitmaps\n  section: x11\n  version: 1.1.1-1\nxcursor-themes:\n  description: Base X cursor themes\n  downloads: 0\n  homepage_uri: ''\n  name: xcursor-themes\n  section: x11\n  version: 1.0.3-1\nxdg-user-dirs:\n  description: tool to manage well known user directories\n  downloads: 31305\n  homepage_uri: http://www.freedesktop.org/wiki/Software/xdg-user-dirs\n  name: xdg-user-dirs\n  section: utils\n  version: 0.14-0ubuntu2\nxdg-user-dirs-gtk:\n  description: tool to manage well known user directories (Gtk extension)\n  downloads: 22352\n  homepage_uri: http://www.freedesktop.org/wiki/Software/xdg-user-dirs\n  name: xdg-user-dirs-gtk\n  section: utils\n  version: 0.9-0ubuntu1\nxdg-utils:\n  description: desktop integration utilities from freedesktop.org\n  downloads: 29617\n  homepage_uri: http://portland.freedesktop.org/\n  name: xdg-utils\n  section: utils\n  version: 1.1.0~rc1-2ubuntu6\nxdiagnose:\n  description: X.org diagnosis tool\n  downloads: 0\n  homepage_uri: http://wiki.ubuntu.com/X\n  name: xdiagnose\n  section: python\n  version: '2.5'\nxfonts-base:\n  description: standard fonts for X\n  downloads: 2\n  homepage_uri: ''\n  name: xfonts-base\n  section: x11\n  version: 1:1.0.3\nxfonts-encodings:\n  description: Encodings for X.Org fonts\n  downloads: 4\n  homepage_uri: ''\n  name: xfonts-encodings\n  section: x11\n  version: 1:1.0.4-1ubuntu1\nxfonts-mathml:\n  description: Type1 Symbol font for MathML\n  downloads: 2\n  homepage_uri: ''\n  name: xfonts-mathml\n  section: x11\n  version: 4ubuntu1\nxfonts-scalable:\n  description: scalable fonts for X\n  downloads: 2\n  homepage_uri: ''\n  name: xfonts-scalable\n  section: x11\n  version: 1:1.0.3-1\nxfonts-utils:\n  description: X Window System font utility programs\n  downloads: 18611\n  homepage_uri: ''\n  name: xfonts-utils\n  section: x11\n  version: 1:7.6+1\nxfsprogs:\n  description: Utilities for managing the XFS filesystem\n  downloads: 3177\n  homepage_uri: http://oss.sgi.com/projects/xfs/\n  name: xfsprogs\n  section: admin\n  version: 3.1.7\nxinit:\n  description: X server initialisation tool\n  downloads: 10127\n  homepage_uri: ''\n  name: xinit\n  section: x11\n  version: 1.3.1-1\nxinput:\n  description: Runtime configuration and test of XInput devices\n  downloads: 2092\n  homepage_uri: ''\n  name: xinput\n  section: x11\n  version: 1.5.99.1-0ubuntu2\nxkb-data:\n  description: X Keyboard Extension (XKB) configuration data\n  downloads: 55581\n  homepage_uri: http://www.freedesktop.org/Software/XKeyboardConfig\n  name: xkb-data\n  section: x11\n  version: 2.5-1ubuntu1\nxml-core:\n  description: XML infrastructure and XML catalog file support\n  downloads: 12520\n  homepage_uri: ''\n  name: xml-core\n  section: text\n  version: '0.13'\nxorg:\n  description: X.Org X Window System\n  downloads: 0\n  homepage_uri: ''\n  name: xorg\n  section: x11\n  version: 1:7.6+12ubuntu1\nxorg-docs-core:\n  description: Core documentation for the X.org X Window System\n  downloads: 0\n  homepage_uri: ''\n  name: xorg-docs-core\n  section: x11\n  version: 1:1.6-1ubuntu2\nxorg-sgml-doctools:\n  description: Common tools for building X.Org SGML documentation\n  downloads: 1751\n  homepage_uri: ''\n  name: xorg-sgml-doctools\n  section: x11\n  version: 1:1.10-1\nxserver-common:\n  description: common files used by various X servers\n  downloads: 42289\n  homepage_uri: ''\n  name: xserver-common\n  section: x11\n  version: 2:1.11.4-0ubuntu10.2\nxserver-xorg:\n  description: X.Org X server\n  downloads: 20883\n  homepage_uri: ''\n  name: xserver-xorg\n  section: x11\n  version: 1:7.6+12ubuntu1\nxserver-xorg-core:\n  description: Xorg X server - core server\n  downloads: 45750\n  homepage_uri: ''\n  name: xserver-xorg-core\n  section: x11\n  version: 2:1.11.4-0ubuntu10.2\nxserver-xorg-input-all:\n  description: X.Org X server -- input driver metapackage\n  downloads: 1\n  homepage_uri: ''\n  name: xserver-xorg-input-all\n  section: x11\n  version: 1:7.6+12ubuntu1\nxserver-xorg-input-evdev:\n  description: X.Org X server -- evdev input driver\n  downloads: 41801\n  homepage_uri: ''\n  name: xserver-xorg-input-evdev\n  section: x11\n  version: 1:2.7.0-0ubuntu1\nxserver-xorg-input-mouse:\n  description: X.Org X server -- mouse input driver\n  downloads: 3523\n  homepage_uri: ''\n  name: xserver-xorg-input-mouse\n  section: x11\n  version: 1:1.7.1-1build3\nxserver-xorg-input-synaptics:\n  description: Synaptics TouchPad driver for X.Org server\n  downloads: 16932\n  homepage_uri: ''\n  name: xserver-xorg-input-synaptics\n  section: x11\n  version: 1.6.0-0ubuntu1~precise1\nxserver-xorg-input-vmmouse:\n  description: X.Org X server -- VMMouse input driver to use with VMWare\n  downloads: 331\n  homepage_uri: ''\n  name: xserver-xorg-input-vmmouse\n  section: x11\n  version: 1:12.8.0-1\nxserver-xorg-input-wacom:\n  description: X.Org X server -- Wacom input driver\n  downloads: 18339\n  homepage_uri: http://linuxwacom.sf.net\n  name: xserver-xorg-input-wacom\n  section: x11\n  version: 1:0.14.0-0ubuntu2\nxserver-xorg-video-all:\n  description: X.Org X server -- output driver metapackage\n  downloads: 0\n  homepage_uri: ''\n  name: xserver-xorg-video-all\n  section: x11\n  version: 1:7.6+12ubuntu1\nxserver-xorg-video-ati:\n  description: X.Org X server -- AMD/ATI display driver wrapper\n  downloads: 8047\n  homepage_uri: ''\n  name: xserver-xorg-video-ati\n  section: x11\n  version: 1:6.14.99~git20111219.aacbd629-0ubuntu2\nxserver-xorg-video-cirrus:\n  description: X.Org X server -- Cirrus display driver\n  downloads: 1448\n  homepage_uri: ''\n  name: xserver-xorg-video-cirrus\n  section: x11\n  version: 1:1.3.2-4build1\nxserver-xorg-video-fbdev:\n  description: X.Org X server -- fbdev display driver\n  downloads: 29638\n  homepage_uri: ''\n  name: xserver-xorg-video-fbdev\n  section: x11\n  version: 1:0.4.2-4ubuntu2\nxserver-xorg-video-intel:\n  description: X.Org X server -- Intel i8xx, i9xx display driver\n  downloads: 15444\n  homepage_uri: ''\n  name: xserver-xorg-video-intel\n  section: x11\n  version: 2:2.17.0-1ubuntu4\nxserver-xorg-video-mach64:\n  description: X.Org X server -- ATI Mach64 display driver\n  downloads: 1647\n  homepage_uri: ''\n  name: xserver-xorg-video-mach64\n  section: x11\n  version: 6.9.0-1build2\nxserver-xorg-video-mga:\n  description: X.Org X server -- MGA display driver\n  downloads: 1721\n  homepage_uri: ''\n  name: xserver-xorg-video-mga\n  section: x11\n  version: 1:1.4.13.dfsg-4build2\nxserver-xorg-video-neomagic:\n  description: X.Org X server -- Neomagic display driver\n  downloads: 1294\n  homepage_uri: ''\n  name: xserver-xorg-video-neomagic\n  section: x11\n  version: 1:1.2.5-2build2\nxserver-xorg-video-nouveau:\n  description: X.Org X server -- Nouveau display driver\n  downloads: 6573\n  homepage_uri: http://nouveau.freedesktop.org/wiki/\n  name: xserver-xorg-video-nouveau\n  section: x11\n  version: 1:0.0.16+git20111201+b5534a1-1build2\nxserver-xorg-video-openchrome:\n  description: X.Org X server -- VIA display driver\n  downloads: 1779\n  homepage_uri: http://www.openchrome.org\n  name: xserver-xorg-video-openchrome\n  section: x11\n  version: 1:0.2.904+svn1050-1\nxserver-xorg-video-qxl:\n  description: X.Org X server -- QXL display driver\n  downloads: 12\n  homepage_uri: http://spice-space.org/\n  name: xserver-xorg-video-qxl\n  section: x11\n  version: 0.0.16-2\nxserver-xorg-video-r128:\n  description: X.Org X server -- ATI r128 display driver\n  downloads: 1402\n  homepage_uri: ''\n  name: xserver-xorg-video-r128\n  section: x11\n  version: 6.8.1-5build2\nxserver-xorg-video-radeon:\n  description: X.Org X server -- AMD/ATI Radeon display driver\n  downloads: 9028\n  homepage_uri: ''\n  name: xserver-xorg-video-radeon\n  section: x11\n  version: 1:6.14.99~git20111219.aacbd629-0ubuntu2\nxserver-xorg-video-s3:\n  description: X.Org X server -- legacy S3 display driver\n  downloads: 1269\n  homepage_uri: ''\n  name: xserver-xorg-video-s3\n  section: x11\n  version: 1:0.6.3-4build2\nxserver-xorg-video-savage:\n  description: X.Org X server -- Savage display driver\n  downloads: 1489\n  homepage_uri: ''\n  name: xserver-xorg-video-savage\n  section: x11\n  version: 1:2.3.3-1ubuntu1\nxserver-xorg-video-siliconmotion:\n  description: X.Org X server -- SiliconMotion display driver\n  downloads: 1257\n  homepage_uri: ''\n  name: xserver-xorg-video-siliconmotion\n  section: x11\n  version: 1:1.7.5-1build2\nxserver-xorg-video-sis:\n  description: X.Org X server -- SiS display driver\n  downloads: 1660\n  homepage_uri: ''\n  name: xserver-xorg-video-sis\n  section: x11\n  version: 1:0.10.3-3build2\nxserver-xorg-video-sisusb:\n  description: X.Org X server -- SiS USB display driver\n  downloads: 1278\n  homepage_uri: ''\n  name: xserver-xorg-video-sisusb\n  section: x11\n  version: 1:0.9.4-2build2\nxserver-xorg-video-tdfx:\n  description: X.Org X server -- tdfx display driver\n  downloads: 1252\n  homepage_uri: ''\n  name: xserver-xorg-video-tdfx\n  section: x11\n  version: 1:1.4.3-4build2\nxserver-xorg-video-trident:\n  description: X.Org X server -- Trident display driver\n  downloads: 1297\n  homepage_uri: ''\n  name: xserver-xorg-video-trident\n  section: x11\n  version: 1:1.3.4-2build2\nxserver-xorg-video-vesa:\n  description: X.Org X server -- VESA display driver\n  downloads: 29945\n  homepage_uri: ''\n  name: xserver-xorg-video-vesa\n  section: x11\n  version: 1:2.3.0-7build2\nxserver-xorg-video-vmware:\n  description: X.Org X server -- VMware display driver\n  downloads: 3528\n  homepage_uri: ''\n  name: xserver-xorg-video-vmware\n  section: x11\n  version: 1:12.0.1-1ubuntu1.1\nxterm:\n  description: X terminal emulator\n  downloads: 14144\n  homepage_uri: http://invisible-island.net/xterm/xterm.html\n  name: xterm\n  section: x11\n  version: 271-1ubuntu2\nxtrans-dev:\n  description: X transport library (development files)\n  downloads: 1210\n  homepage_uri: ''\n  name: xtrans-dev\n  section: x11\n  version: 1.2.6-2\nxul-ext-ubufox:\n  description: Ubuntu-specific configuration defaults and apt support for Firefox\n  downloads: 0\n  homepage_uri: https://launchpad.net/ubufox\n  name: xul-ext-ubufox\n  section: web\n  version: 2.0.3-0ubuntu1\nxvfb:\n  description: Virtual Framebuffer 'fake' X server\n  downloads: 479\n  homepage_uri: ''\n  name: xvfb\n  section: x11\n  version: 2:1.11.4-0ubuntu10.2\nxz-lzma:\n  description: XZ-format compression utilities - compatibility commands\n  downloads: 4344\n  homepage_uri: http://tukaani.org/xz/\n  name: xz-lzma\n  section: utils\n  version: 5.1.1alpha+20110809-3\nxz-utils:\n  description: XZ-format compression utilities\n  downloads: 21528\n  homepage_uri: http://tukaani.org/xz/\n  name: xz-utils\n  section: utils\n  version: 5.1.1alpha+20110809-3\nyelp:\n  description: Help browser for GNOME\n  downloads: 8904\n  homepage_uri: http://live.gnome.org/Yelp\n  name: yelp\n  section: gnome\n  version: 3.4.1-0ubuntu1\nyelp-xsl:\n  description: XSL stylesheets for the yelp help browser\n  downloads: 3019\n  homepage_uri: ''\n  name: yelp-xsl\n  section: gnome\n  version: 3.4.1-1\nzeitgeist:\n  description: event logging framework\n  downloads: 87\n  homepage_uri: http://zeitgeist-project.com/\n  name: zeitgeist\n  section: utils\n  version: 0.9.0-1ubuntu1\nzeitgeist-core:\n  description: event logging framework - engine\n  downloads: 1488\n  homepage_uri: http://zeitgeist-project.com/\n  name: zeitgeist-core\n  section: utils\n  version: 0.9.0-1ubuntu1\nzeitgeist-datahub:\n  description: event logging framework - passive logging daemon\n  downloads: 642\n  homepage_uri: https://launchpad.net/zeitgeist-datahub\n  name: zeitgeist-datahub\n  section: utils\n  version: 0.8.2-1ubuntu2\nzenity:\n  description: Display graphical dialog boxes from shell scripts\n  downloads: 15112\n  homepage_uri: http://live.gnome.org/Zenity\n  name: zenity\n  section: gnome\n  version: 3.4.0-0ubuntu4\nzenity-common:\n  description: Display graphical dialog boxes from shell scripts (common files)\n  downloads: 4902\n  homepage_uri: http://live.gnome.org/Zenity\n  name: zenity-common\n  section: gnome\n  version: 3.4.0-0ubuntu4\nzip:\n  description: Archiver for .zip files\n  downloads: 7573\n  homepage_uri: http://www.info-zip.org/Zip.html\n  name: zip\n  section: utils\n  version: 3.0-4\nzlib1g:\n  description: compression library - runtime\n  downloads: 110281\n  homepage_uri: http://zlib.net/\n  name: zlib1g\n  section: libs\n  version: 1:1.2.3.4.dfsg-3ubuntu4\nzlib1g-dev:\n  description: compression library - development\n  downloads: 9744\n  homepage_uri: http://zlib.net/\n  name: zlib1g-dev\n  section: libdevel\n  version: 1:1.2.3.4.dfsg-3ubuntu4\nzlib1g:i386:\n  description: compression library - runtime\n  downloads: 0\n  homepage_uri: http://zlib.net/\n  name: zlib1g:i386\n  section: libs\n  version: 1:1.2.3.4.dfsg-3ubuntu4\nzsh:\n  description: shell with lots of features\n  downloads: 3830\n  homepage_uri: http://www.zsh.org/\n  name: zsh\n  section: shells\n  version: 4.3.17-1ubuntu1\n"
  },
  {
    "path": "manifest/python-packages.yaml",
    "content": "BeautifulSoup:\n  description: HTML/XML parser for quick-turnaround applications like screen-scraping.\n  homepage_uri: http://www.crummy.com/software/BeautifulSoup/\n  name: BeautifulSoup\n  version: 3.2.1\nBrlapi:\n  description: Python BrlAPI\n  homepage_uri: http://mielke.cc/brltty/\n  name: Brlapi\n  version: 0.5.6\nCheetah:\n  description: Cheetah is a template engine and code generation tool.\n  homepage_uri: http://www.cheetahtemplate.org/\n  name: Cheetah\n  version: 2.4.4\nCouchDB:\n  description: Python library for working with CouchDB\n  homepage_uri: http://code.google.com/p/couchdb-python/\n  name: CouchDB\n  version: '0.8'\nCython:\n  description: The Cython compiler for writing C extensions for the Python language.\n  homepage_uri: http://www.cython.org\n  name: Cython\n  version: '0.16'\nDendroPy:\n  description: 'A Python library for phylogenetics and phylogenetic computing: reading,\n    writing, simulation, processing and manipulation of phylogenetic trees (phylogenies)\n    and characters.'\n  homepage_uri: http://packages.python.org/DendroPy/\n  name: DendroPy\n  version: 3.11.0\nFabric:\n  description: Fabric is a simple, Pythonic tool for remote execution and deployment.\n  homepage_uri: http://fabfile.org\n  name: Fabric\n  version: 1.4.2\nGnuPGInterface:\n  description: GnuPG interactions with file handles\n  homepage_uri: http://py-gnupg.sourceforge.net/\n  name: GnuPGInterface\n  version: 0.3.2\nHTSeq:\n  description: A framework to process and analyze data from high-throughput sequencing\n    (HTS) assays\n  homepage_uri: http://www-huber.embl.de/users/anders/HTSeq/\n  name: HTSeq\n  version: 0.5.3p7\nJinja2:\n  description: A small but fast and easy to use stand-alone template engine written\n    in pure python.\n  homepage_uri: http://jinja.pocoo.org/\n  name: Jinja2\n  version: '2.6'\nLandscape Client:\n  description: Landscape Client\n  homepage_uri: http://landscape.canonical.com\n  name: Landscape Client\n  version: 12.04.2\nLogbook:\n  description: A logging replacement for Python\n  homepage_uri: http://logbook.pocoo.org/\n  name: Logbook\n  version: '0.3'\nM2Crypto:\n  description: 'M2Crypto: A Python crypto and SSL toolkit'\n  homepage_uri: http://chandlerproject.org/Projects/MeTooCrypto\n  name: M2Crypto\n  version: 0.21.1\nMACS:\n  description: Model Based Analysis for ChIP-Seq data\n  homepage_uri: http://liulab.dfci.harvard.edu/MACS/\n  name: MACS\n  version: 1.4.2\nMagic file extensions:\n  description: libmagic Python bindings\n  homepage_uri: UNKNOWN\n  name: Magic file extensions\n  version: '0.2'\nMako:\n  description: A super-fast templating language that borrows the  best ideas from\n    the existing templating languages.\n  homepage_uri: http://www.makotemplates.org/\n  name: Mako\n  version: 0.7.0\nMarkupSafe:\n  description: Implements a XML/HTML/XHTML Markup safe string for Python\n  homepage_uri: http://dev.pocoo.org/\n  name: MarkupSafe\n  version: '0.15'\nMySQL-python:\n  description: Python interface to MySQL\n  homepage_uri: http://sourceforge.net/projects/mysql-python\n  name: MySQL-python\n  version: 1.2.3\nPAM:\n  description: Python bindings for PAM\n  homepage_uri: UNKNOWN\n  name: PAM\n  version: 0.4.2\nPIL:\n  description: Python Imaging Library\n  homepage_uri: http://www.pythonware.com/products/pil\n  name: PIL\n  version: 1.1.7\nPaste:\n  description: Tools for using a Web Server Gateway Interface stack\n  homepage_uri: http://pythonpaste.org\n  name: Paste\n  version: 1.7.5.1\nPasteDeploy:\n  description: Load, configure, and compose WSGI applications and servers\n  homepage_uri: http://pythonpaste.org/deploy/\n  name: PasteDeploy\n  version: 1.5.0\nPasteScript:\n  description: A pluggable command-line frontend, including commands to setup package\n    file layouts\n  homepage_uri: http://pythonpaste.org/script/\n  name: PasteScript\n  version: 1.7.5\nPyVCF:\n  description: Variant Call Format (VCF) parser for Python\n  homepage_uri: https://github.com/jamescasbon/PyVCF\n  name: PyVCF\n  version: 0.4.6\nPyYAML:\n  description: YAML parser and emitter for Python\n  homepage_uri: http://pyyaml.org/wiki/PyYAML\n  name: PyYAML\n  version: '3.10'\nPygments:\n  description: Pygments is a syntax highlighting package written in Python.\n  homepage_uri: http://pygments.org/\n  name: Pygments\n  version: '1.5'\nPyrex-real:\n  description: UNKNOWN\n  homepage_uri: http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/\n  name: Pyrex\n  version: 0.9.9\nPyste:\n  description: Pyste - Python Semi-Automatic Exporter\n  homepage_uri: http://www.boost.org/libs/python/pyste/index.html\n  name: Pyste\n  version: 0.9.10\nPython:\n  description: A high-level object-oriented programming language\n  homepage_uri: http://www.python.org/2.7\n  name: Python\n  version: 2.7.3\nRoutes:\n  description: Routing Recognition and Generation Tools\n  homepage_uri: http://routes.groovie.org/\n  name: Routes\n  version: 1.12.3\nSQLAlchemy:\n  description: Database Abstraction Library\n  homepage_uri: http://www.sqlalchemy.org\n  name: SQLAlchemy\n  version: 0.7.8\nSphinx:\n  description: Python documentation generator\n  homepage_uri: http://sphinx.pocoo.org/\n  name: Sphinx\n  version: 1.1.3\nStarCluster:\n  description: StarCluster is a utility for creating and managing computing clusters\n    hosted on Amazon's Elastic Compute Cloud (EC2).\n  homepage_uri: http://web.mit.edu/starcluster\n  name: StarCluster\n  version: 0.93.3\nTwisted:\n  description: An asynchronous networking framework written in Python\n  homepage_uri: http://twistedmatrix.com/\n  name: Twisted\n  version: 12.1.0\nTwisted Core:\n  description: The core parts of the Twisted networking framework\n  homepage_uri: http://twistedmatrix.com/\n  name: Twisted Core\n  version: 11.1.0\nTwisted Names:\n  description: A Twisted DNS implementation.\n  homepage_uri: http://twistedmatrix.com/trac/wiki/TwistedNames\n  name: Twisted Names\n  version: 11.1.0\nTwisted Web:\n  description: Twisted web server, programmable in Python.\n  homepage_uri: http://twistedmatrix.com/trac/wiki/TwistedWeb\n  name: Twisted Web\n  version: 11.1.0\nVTK:\n  description: The Visualization Toolkit\n  homepage_uri: http://www.vtk.org/\n  name: VTK\n  version: 5.8.0\nWebHelpers:\n  description: Web Helpers\n  homepage_uri: http://webhelpers.groovie.org/\n  name: WebHelpers\n  version: '1.3'\nWebOb:\n  description: WSGI request and response object\n  homepage_uri: http://webob.org/\n  name: WebOb\n  version: 1.2b2\nadium-theme-ubuntu:\n  description: UNKNOWN\n  homepage_uri: UNKNOWN\n  name: adium-theme-ubuntu\n  version: 0.3.2\namqplib:\n  description: AMQP Client Library\n  homepage_uri: http://code.google.com/p/py-amqplib/\n  name: amqplib\n  version: 1.0.2\nanyjson:\n  description: Wraps the best available JSON implementation available in a common\n    interface\n  homepage_uri: http://bitbucket.org/runeh/anyjson\n  name: anyjson\n  version: 0.3.1\napptools:\n  description: application tools\n  homepage_uri: http://code.enthought.com/projects/app_tools.php\n  name: apptools\n  version: 4.0.0\napt-xapian-index:\n  description: Xapian index of Debian packages\n  homepage_uri: http://www.enricozini.org/sw/apt-xapian-index/\n  name: apt-xapian-index\n  version: '0.44'\napturl:\n  description: UNKNOWN\n  homepage_uri: UNKNOWN\n  name: apturl\n  version: 0.5.1ubuntu3\nargparse:\n  description: Python command-line parsing library\n  homepage_uri: ''\n  name: argparse\n  version: 1.2.1\nbcbio-nextgen:\n  description: Automated, distributed next-gen sequencing pipeline; includes Galaxy\n    interaction\n  homepage_uri: https://github.com/chapmanb/bcbb/tree/master/nextgen\n  name: bcbio-nextgen\n  version: '0.2'\nbiopython:\n  description: Freely available tools for computational molecular biology.\n  homepage_uri: http://www.biopython.org/\n  name: biopython\n  version: '1.59'\nboto:\n  description: Amazon Web Services Library\n  homepage_uri: https://github.com/boto/boto/\n  name: boto\n  version: 2.5.1\nbx-python:\n  description: Tools for manipulating biological data, particularly multiple sequence\n    alignments\n  homepage_uri: http://bitbucket.org/james_taylor/bx-python/wiki/Home\n  name: bx-python\n  version: 0.7.1\nbzr:\n  description: Friendly distributed version control system\n  homepage_uri: http://bazaar.canonical.com/\n  name: bzr\n  version: 2.5.0\ncelery:\n  description: Distributed Task Queue\n  homepage_uri: http://celeryproject.org\n  name: celery\n  version: 2.5.5\nchardet:\n  description: Universal encoding detector\n  homepage_uri: http://chardet.feedparser.org/\n  name: chardet\n  version: 2.0.1\ncloud-init:\n  description: EC2 initialisation magic\n  homepage_uri: http://launchpad.net/cloud-init/\n  name: cloud-init\n  version: 0.6.3\ncommand-not-found:\n  description: UNKNOWN\n  homepage_uri: UNKNOWN\n  name: command-not-found\n  version: 0.2.44\nconfigglue:\n  description: Glue to stick OptionParser and ConfigParser together\n  homepage_uri: https://launchpad.net/configglue\n  name: configglue\n  version: '1.0'\nconfigobj:\n  description: Config file reading, writing and validation.\n  homepage_uri: http://www.voidspace.org.uk/python/configobj.html\n  name: configobj\n  version: 4.7.2\ndebtagshw:\n  description: UNKNOWN\n  homepage_uri: UNKNOWN\n  name: debtagshw\n  version: '0.1'\ndecorator:\n  description: Better living through Python with decorators\n  homepage_uri: http://pypi.python.org/pypi/decorator\n  name: decorator\n  version: 3.3.1\ndefer:\n  description: Simple framework for asynchronous programming\n  homepage_uri: http://launchpad.net/python-defer\n  name: defer\n  version: 1.0.2\ndevscripts:\n  description: UNKNOWN\n  homepage_uri: UNKNOWN\n  name: devscripts\n  version: 2.11.6ubuntu1.2\ndirspec:\n  description: XDG Base and User directories implementation\n  homepage_uri: https://launchpad.net/dirspec\n  name: dirspec\n  version: 3.0.0\ndistribute:\n  description: Easily download, build, install, upgrade, and uninstall Python packages\n  homepage_uri: http://packages.python.org/distribute\n  name: distribute\n  version: 0.6.19\ndocutils:\n  description: Docutils -- Python Documentation Utilities\n  homepage_uri: http://docutils.sourceforge.net/\n  name: docutils\n  version: 0.9.1\nduplicity:\n  description: Encrypted backup using rsync algorithm\n  homepage_uri: http://duplicity.nongnu.org/index.html\n  name: duplicity\n  version: 0.6.18\nenvisage:\n  description: extensible application framework\n  homepage_uri: http://code.enthought.com/projects/envisage/\n  name: envisage\n  version: 4.0.0\neuca2ools:\n  description: Elastic Utility Computing Architecture Command Line Tools\n  homepage_uri: http://open.eucalyptus.com\n  name: euca2ools\n  version: 2.0.0\ngnuplot-py:\n  description: A Python interface to the gnuplot plotting program.\n  homepage_uri: http://gnuplot-py.sourceforge.net\n  name: gnuplot-py\n  version: '1.8'\nh5py:\n  description: Read and write HDF5 files from Python\n  homepage_uri: http://h5py.alfven.org\n  name: h5py\n  version: 2.0.1\nhttplib2:\n  description: A comprehensive HTTP client library.\n  homepage_uri: http://code.google.com/p/httplib2/\n  name: httplib2\n  version: 0.7.2\nipython:\n  description: 'IPython: Productive Interactive Computing'\n  homepage_uri: http://ipython.org\n  name: ipython\n  version: 0.12.1\njockey:\n  description: UI for managing third-party and non-free drivers\n  homepage_uri: https://launchpad.net/jockey\n  name: jockey\n  version: 0.9.7\nkeyring:\n  description: Store and access your passwords safely.\n  homepage_uri: http://bitbucket.org/kang/python-keyring-lib\n  name: keyring\n  version: 0.7.1\nkombu:\n  description: Messaging Framework for Python\n  homepage_uri: http://github.com/ask/kombu/\n  name: kombu\n  version: 2.1.8\nlanguage-selector:\n  description: UNKNOWN\n  homepage_uri: UNKNOWN\n  name: language-selector\n  version: '0.1'\nlaunchpadlib:\n  description: Script Launchpad through its web services interfaces.  Officially supported.\n  homepage_uri: https://help.launchpad.net/API/launchpadlib\n  name: launchpadlib\n  version: 1.9.12\nlazr.restfulclient:\n  description: A programmable client library that takes advantage of the commonalities\n    among\n  homepage_uri: https://launchpad.net/lazr.restfulclient\n  name: lazr.restfulclient\n  version: 0.12.0\nlazr.uri:\n  description: A self-contained, easily reusable library for parsing, manipulating,\n  homepage_uri: https://launchpad.net/lazr.uri\n  name: lazr.uri\n  version: 1.0.3\nlouis:\n  description: Python bindings for liblouis\n  homepage_uri: ''\n  name: louis\n  version: 2.3.0\nlxml:\n  description: Powerful and Pythonic XML processing library combining libxml2/libxslt\n    with the ElementTree API.\n  homepage_uri: http://lxml.de/\n  name: lxml\n  version: 2.3.4\nmatplotlib:\n  description: Python plotting package\n  homepage_uri: http://matplotlib.sourceforge.net\n  name: matplotlib\n  version: 1.0.1\nmayavi:\n  description: The Mayavi scientific data 3-dimensional visualizer.\n  homepage_uri: http://code.enthought.com/projects/mayavi/\n  name: mayavi\n  version: 4.0.0\nmercurial:\n  description: Scalable distributed SCM\n  homepage_uri: http://mercurial.selenic.com/\n  name: mercurial\n  version: 2.0.2\nnetworkx:\n  description: Python package for creating and manipulating graphs and networks\n  homepage_uri: http://networkx.lanl.gov/\n  name: networkx\n  version: 1.7rc1\nnose:\n  description: nose extends unittest to make testing easier\n  homepage_uri: http://readthedocs.org/docs/nose/\n  name: nose\n  version: 1.1.2\nnumpy:\n  description: 'NumPy: array processing for numbers, strings, records, and objects.'\n  homepage_uri: http://numpy.scipy.org\n  name: numpy\n  version: 1.6.2\nnvidia-common:\n  description: Find obsolete NVIDIA drivers\n  homepage_uri: http://www.albertomilone.com\n  name: nvidia-common\n  version: 0.0.0\noauth:\n  description: Library for OAuth version 1.0a.\n  homepage_uri: http://code.google.com/p/oauth\n  name: oauth\n  version: 1.0.1\noca:\n  description: Bindings for XMLRPC OpenNebula Cloud API\n  homepage_uri: https://github.com/lukaszo/python-oca\n  name: oca\n  version: 0.2.3\nonboard:\n  description: Simple On-screen Keyboard\n  homepage_uri: http://launchpad.net/onboard/\n  name: onboard\n  version: 0.97.1\noneconf:\n  description: UNKNOWN\n  homepage_uri: UNKNOWN\n  name: oneconf\n  version: 0.2.8.1\npandas:\n  description: Powerful data structures for data analysis and statistics\n  homepage_uri: http://pandas.pydata.org\n  name: pandas\n  version: 0.7.3\nparamiko:\n  description: SSH2 protocol library\n  homepage_uri: http://www.lag.net/paramiko/\n  name: paramiko\n  version: 1.7.7.1\npexpect:\n  description: Pexpect is a pure Python Expect. It allows easy control of other applications.\n  homepage_uri: http://pexpect.sourceforge.net/\n  name: pexpect\n  version: '2.3'\npip:\n  description: pip installs packages. Python packages. An easy_install replacement\n  homepage_uri: http://www.pip-installer.org\n  name: pip\n  version: '1.1'\npiston-mini-client:\n  description: A package to consume Django-Piston web services\n  homepage_uri: https://launchpad.net/piston-mini-client\n  name: piston-mini-client\n  version: 0.7.2\nprotobuf:\n  description: Protocol Buffers\n  homepage_uri: http://code.google.com/p/protobuf/\n  name: protobuf\n  version: 2.4.1\npsycopg2:\n  description: Python-PostgreSQL Database Adapter\n  homepage_uri: http://initd.org/psycopg/\n  name: psycopg2\n  version: 2.4.5\npyOpenSSL:\n  description: Python wrapper module around the OpenSSL library\n  homepage_uri: http://pyopenssl.sourceforge.net/\n  name: pyOpenSSL\n  version: '0.13'\npyasn1:\n  description: ASN.1 types and codecs\n  homepage_uri: http://sourceforge.net/projects/pyasn1/\n  name: pyasn1\n  version: 0.0.13b\npybedtools:\n  description: Wrapper around BEDTools for bioinformatics work\n  homepage_uri: none\n  name: pybedtools\n  version: 0.6.1\npycrypto:\n  description: Cryptographic modules for Python.\n  homepage_uri: http://www.pycrypto.org/\n  name: pycrypto\n  version: '2.6'\npycups:\n  description: Python bindings for libcups\n  homepage_uri: http://cyberelk.net/tim/software/pycups/\n  name: pycups\n  version: 1.9.61\npycurl:\n  description: PycURL -- cURL library module for Python\n  homepage_uri: http://pycurl.sourceforge.net/\n  name: pycurl\n  version: 7.19.0\npydoop:\n  description: 'Pydoop: a Python MapReduce and HDFS API for Hadoop'\n  homepage_uri: http://pydoop.sourceforge.net\n  name: pydoop\n  version: 0.6.0\npydot:\n  description: Python interface to Graphviz's Dot\n  homepage_uri: http://code.google.com/p/pydot/\n  name: pydot\n  version: 1.0.28\npyface:\n  description: traits-capable windowing framework\n  homepage_uri: http://code.enthought.com/projects/traits_gui\n  name: pyface\n  version: 4.0.0\npygooglechart:\n  description: A complete Python wrapper for the Google Chart API\n  homepage_uri: http://pygooglechart.slowchop.com/\n  name: pygooglechart\n  version: 0.3.0\npygr:\n  description: Pygr, a Python graph-database toolkit oriented primarily on bioinformatics\n  homepage_uri: http://code.google.com/p/pygr/\n  name: pygr\n  version: 0.8.2\npyinotify:\n  description: Linux filesystem events monitoring\n  homepage_uri: http://github.com/seb-m/pyinotify\n  name: pyinotify\n  version: 0.9.2\npymongo:\n  description: Python driver for MongoDB <http://www.mongodb.org>\n  homepage_uri: http://github.com/mongodb/mongo-python-driver\n  name: pymongo\n  version: '2.2'\npyparsing:\n  description: Python parsing module\n  homepage_uri: http://pyparsing.wikispaces.com/\n  name: pyparsing\n  version: 1.5.6\npysam:\n  description: pysam\n  homepage_uri: http://code.google.com/p/pysam/\n  name: pysam\n  version: '0.6'\npyserial:\n  description: Python Serial Port Extension\n  homepage_uri: http://pyserial.sourceforge.net/\n  name: pyserial\n  version: '2.5'\npysmbc:\n  description: Python bindings for libsmbclient\n  homepage_uri: http://cyberelk.net/tim/software/pysmbc/\n  name: pysmbc\n  version: 1.0.13\npysqlite:\n  description: DB-API 2.0 interface for SQLite 3.x\n  homepage_uri: http://pysqlite.googlecode.com/\n  name: pysqlite\n  version: 2.6.3\npython-apt:\n  description: Python bindings for APT\n  homepage_uri: UNKNOWN\n  name: python-apt\n  version: 0.8.3ubuntu7\npython-cjson:\n  description: Fast JSON encoder/decoder for Python\n  homepage_uri: http://ag-projects.com/\n  name: python-cjson\n  version: 1.0.5\npython-dateutil:\n  description: Extensions to the standard python 2.3+ datetime module\n  homepage_uri: http://labix.org/python-dateutil\n  name: python-dateutil\n  version: '1.5'\npython-debian:\n  description: Debian package related modules\n  homepage_uri: http://packages.debian.org/sid/python-debian\n  name: python-debian\n  version: 0.1.21ubuntu1\npython-distutils-extra:\n  description: Add support for i18n, documentation and icons to distutils\n  homepage_uri: UNKNOWN\n  name: python-distutils-extra\n  version: '2.32'\npython-virtkey:\n  description: python extension for emulating keypresses and getting keyboard layout\n  homepage_uri: https://launchpad.net/python-virtkey/\n  name: python-virtkey\n  version: 0.60.0\npyxdg:\n  description: PyXDG contains implementations of freedesktop.org standards in python.\n  homepage_uri: http://cvs.freedesktop.org/cgi-bin/viewcvs.cgi/pyxdg/\n  name: pyxdg\n  version: '0.19'\npyzmq:\n  description: Python bindings for 0MQ.\n  homepage_uri: http://github.com/zeromq/pyzmq\n  name: pyzmq\n  version: 2.2.0\nreportlab:\n  description: The Reportlab Toolkit\n  homepage_uri: http://www.reportlab.com/\n  name: reportlab\n  version: '2.5'\nrepoze.lru:\n  description: A tiny LRU cache implementation and decorator\n  homepage_uri: http://www.repoze.org\n  name: repoze.lru\n  version: '0.5'\nrhythmbox-ubuntuone:\n  description: Ubuntu One Rhythmbox plug-in\n  homepage_uri: https://launchpad.net/rhythmbox-ubuntuone\n  name: rhythmbox-ubuntuone\n  version: 3.0.0\nrpy:\n  description: Python interface to the R language\n  homepage_uri: http://rpy.sourceforge.net\n  name: rpy\n  version: 1.0.3\nrpy2:\n  description: ''\n  homepage_uri: ''\n  name: rpy2\n  version: 2.2.6dev-20120618\nrpy2-bioconductor-extensions:\n  description: Bioconductor-specific extensions for rpy2\n  homepage_uri: no url yet\n  name: rpy2-bioconductor-extensions\n  version: 0.2.5-dev\nruffus:\n  description: Light-weight Python Computational Pipeline Management\n  homepage_uri: http://ruffus.googlecode.com\n  name: ruffus\n  version: '2.2'\nscikits.learn:\n  description: A set of python modules for machine learning and data mining\n  homepage_uri: http://scikit-learn.sourceforge.net\n  name: scikits.learn\n  version: 0.8.1\nscikits.statsmodels:\n  description: Statistical computations and models for use with SciPy\n  homepage_uri: http://statsmodels.sourceforge.net/\n  name: scikits.statsmodels\n  version: 0.3.1\nscipy:\n  description: 'SciPy: Scientific Library for Python'\n  homepage_uri: http://www.scipy.org\n  name: scipy\n  version: 0.10.1\nseal:\n  description: 'Seal: Sequence Alignment on Hadoop.'\n  homepage_uri: http://www.crs4.it\n  name: seal\n  version: devel-20120618\nsessioninstaller:\n  description: APT based installer using PackageKit session DBus API\n  homepage_uri: UNKNOWN\n  name: sessioninstaller\n  version: 0.0.0\nsetuptools:\n  description: xxxx\n  homepage_uri: xxx\n  name: setuptools\n  version: 0.6c11\nsimplejson:\n  description: Simple, fast, extensible JSON encoder/decoder for Python\n  homepage_uri: http://github.com/simplejson/simplejson\n  name: simplejson\n  version: 2.5.2\nsoftware-center-aptd-plugins:\n  description: UNKNOWN\n  homepage_uri: UNKNOWN\n  name: software-center-aptd-plugins\n  version: 0.0.0\nssh:\n  description: SSH2 protocol library\n  homepage_uri: UNKNOWN\n  name: ssh\n  version: 1.7.14\nsystem-service:\n  description: UNKNOWN\n  homepage_uri: UNKNOWN\n  name: system-service\n  version: 0.1.6\ntornado:\n  description: Tornado is an open source version of the scalable, non-blocking web\n    server and and tools that power FriendFeed\n  homepage_uri: http://www.tornadoweb.org/\n  name: tornado\n  version: '2.3'\ntraits:\n  description: explicitly typed attributes for Python\n  homepage_uri: http://code.enthought.com/projects/traits\n  name: traits\n  version: 4.0.0\ntraitsui:\n  description: 'traitsui: traits-capable user interfaces'\n  homepage_uri: https://github.com/enthought/traitsui\n  name: traitsui\n  version: 4.0.1\nubuntuone-couch:\n  description: Ubuntu One CouchDB\n  homepage_uri: https://launchpad.net/ubuntuone-couch\n  name: ubuntuone-couch\n  version: 0.3.0\nubuntuone-installer:\n  description: Ubuntu One Installer\n  homepage_uri: https://launchpad.net/ubuntuone-installer\n  name: ubuntuone-installer\n  version: 3.0.0\nubuntuone-storage-protocol:\n  description: UNKNOWN\n  homepage_uri: UNKNOWN\n  name: ubuntuone-storage-protocol\n  version: 3.0.0\nufw:\n  description: front-end for Linux firewalling\n  homepage_uri: https://launchpad.net/ufw\n  name: ufw\n  version: 0.31.1-1\nunattended-upgrades:\n  description: UNKNOWN\n  homepage_uri: UNKNOWN\n  name: unattended-upgrades\n  version: '0.1'\nunity-lens-video:\n  description: UNKNOWN\n  homepage_uri: http://launchpad.net/~davidc3\n  name: unity-lens-video\n  version: 0.3.5\nunity-scope-video-remote:\n  description: UNKNOWN\n  homepage_uri: http://launchpad.net/onehundredscopes\n  name: unity-scope-video-remote\n  version: 0.3.5\nusb-creator:\n  description: Ubuntu startup disk creator\n  homepage_uri: UNKNOWN\n  name: usb-creator\n  version: 0.2.23\nvariant_tools:\n  description: 'Variant tools: an integrated annotation and analysis package for next-gen\n    sequencing data'\n  homepage_uri: http://varianttools.sourceforge.net\n  name: variant_tools\n  version: 1.0.1a\nvboxapi:\n  description: Python interface to VirtualBox\n  homepage_uri: http://www.virtualbox.org\n  name: vboxapi\n  version: '1.0'\nvirtualenv:\n  description: Virtual Python Environment builder\n  homepage_uri: http://www.virtualenv.org\n  name: virtualenv\n  version: 1.7.1.2\nwadllib:\n  description: Navigate HTTP resources using WADL files as guides.\n  homepage_uri: https://launchpad.net/wadllib\n  name: wadllib\n  version: 1.3.0\nworkerpool:\n  description: Module for distributing jobs to a pool of worker threads.\n  homepage_uri: http://code.google.com/p/workerpool/\n  name: workerpool\n  version: 0.9.2\nwsgiref:\n  description: WSGI (PEP 333) Reference Library\n  homepage_uri: ''\n  name: wsgiref\n  version: 0.1.2\nwxPython:\n  description: Cross platform GUI toolkit for Python\n  homepage_uri: http://wxPython.org/\n  name: wxPython\n  version: 2.8.12.1\nwxPython-common:\n  description: Cross platform GUI toolkit for Python\n  homepage_uri: http://wxPython.org/\n  name: wxPython-common\n  version: 2.8.12.1\nxdiagnose:\n  description: Analysis tools for troubleshooting X.org problems\n  homepage_uri: http://launchpad.net/xdiagnose\n  name: xdiagnose\n  version: '2.5'\nxkit:\n  description: library for the manipulation of the xorg.conf\n  homepage_uri: https://launchpad.net/x-kit\n  name: xkit\n  version: 0.0.0\nxlrd:\n  description: Library for developers to extract data from Microsoft Excel (tm) spreadsheet\n    files\n  homepage_uri: http://www.lexicon.net/sjmachin/xlrd.htm\n  name: xlrd\n  version: 0.7.8\nyolk:\n  description: Command-line tool for querying PyPI and Python packages installed on\n    your system.\n  homepage_uri: https://github.com/cakebread/yolk\n  name: yolk\n  version: 0.4.3\nzope.interface:\n  description: Interfaces for Python\n  homepage_uri: http://pypi.python.org/pypi/zope.interface\n  name: zope.interface\n  version: 3.6.1\n"
  },
  {
    "path": "manifest/r-packages.yaml",
    "content": "AnnotationDbi:\n  description: Annotation Database Interface\n  homepage_uri: ''\n  name: AnnotationDbi\n  version: 1.18.1\nArrayExpress:\n  description: 'Access the ArrayExpress Microarray Database at EBI and build\\nBioconductor\n    data structures: ExpressionSet, AffyBatch,\\nNChannelSet'\n  homepage_uri: ''\n  name: ArrayExpress\n  version: 1.16.0\nArrayTools:\n  description: geneChip Analysis Package\n  homepage_uri: ''\n  name: ArrayTools\n  version: 1.16.0\nBSgenome:\n  description: Infrastructure for Biostrings-based genome data packages\n  homepage_uri: ''\n  name: BSgenome\n  version: 1.24.0\nBSgenome.Ecoli.NCBI.20080805:\n  description: Escherichia coli full genomes\n  homepage_uri: ''\n  name: BSgenome.Ecoli.NCBI.20080805\n  version: 1.3.17\nBSgenome.Scerevisiae.UCSC.sacCer3:\n  description: Saccharomyces cerevisiae (Yeast) full genome (UCSC version sacCer3)\n  homepage_uri: ''\n  name: BSgenome.Scerevisiae.UCSC.sacCer3\n  version: 1.3.17\nBayesPeak:\n  description: Bayesian Analysis of ChIP-seq Data\n  homepage_uri: ''\n  name: BayesPeak\n  version: 1.8.0\nBeadDataPackR:\n  description: Compression of Illumina BeadArray data\n  homepage_uri: ''\n  name: BeadDataPackR\n  version: 1.8.0\nBiasedUrn:\n  description: Biased Urn model distributions\n  homepage_uri: http://www.agner.org/random/ http://www.r-project.org\n  name: BiasedUrn\n  version: '1.04'\nBiobase:\n  description: 'Biobase: Base functions for Bioconductor'\n  homepage_uri: ''\n  name: Biobase\n  version: 2.16.0\nBiocGenerics:\n  description: Generic functions for Bioconductor\n  homepage_uri: ''\n  name: BiocGenerics\n  version: 0.2.0\nBiocInstaller:\n  description: Install/Update Bioconductor and CRAN Packages\n  homepage_uri: ''\n  name: BiocInstaller\n  version: 1.4.6\nBiodiversityR:\n  description: GUI for biodiversity and community ecology analysis\n  homepage_uri: http://www.r-project.org,\\nhttp://www.worldagroforestry.org/resources/databases/tree-diversity-analysis\n  name: BiodiversityR\n  version: '1.6'\nBiostrings:\n  description: String objects representing biological sequences, and matching\\nalgorithms\n  homepage_uri: ''\n  name: Biostrings\n  version: 2.24.1\nCNAnorm:\n  description: A normalization method for Copy Number Aberration in cancer\\nsamples\n  homepage_uri: http://www.r-project.org,\n  name: CNAnorm\n  version: 1.2.0\nCSAR:\n  description: Statistical tools for the analysis of ChIP-seq data\n  homepage_uri: ''\n  name: CSAR\n  version: 1.8.0\nCairo:\n  description: R graphics device using cairo graphics library for creating\\nhigh-quality\n    bitmap (PNG, JPEG, TIFF), vector (PDF, SVG,\\nPostScript) and display (X11 and\n    Win32) output.\n  homepage_uri: http://www.rforge.net/Cairo/\n  name: Cairo\n  version: 1.5-1\nCategory:\n  description: Category Analysis\n  homepage_uri: ''\n  name: Category\n  version: 2.22.0\nChIPpeakAnno:\n  description: Batch annotation of the peaks identified from either ChIP-seq,\\nChIP-chip\n    experiments or any experiments resulted in large\\nnumber of chromosome ranges.\n  homepage_uri: ''\n  name: ChIPpeakAnno\n  version: 2.4.0\nChIPseqR:\n  description: Identifying Protein Binding Sites in High-Throughput Sequencing\\nData\n  homepage_uri: ''\n  name: ChIPseqR\n  version: 1.10.0\nChIPsim:\n  description: Simulation of ChIP-seq experiments\n  homepage_uri: ''\n  name: ChIPsim\n  version: 1.10.0\nDBI:\n  description: R Database Interface\n  homepage_uri: ''\n  name: DBI\n  version: 0.2-5\nDEGseq:\n  description: Identify Differentially Expressed Genes from RNA-seq data\n  homepage_uri: ''\n  name: DEGseq\n  version: 1.10.0\nDESeq:\n  description: Differential gene expression analysis based on the negative\\nbinomial\n    distribution\n  homepage_uri: http://www-huber.embl.de/users/anders/DESeq\n  name: DESeq\n  version: 1.8.3\nDEXSeq:\n  description: Inference of differential exon usage in RNA-Seq\n  homepage_uri: ''\n  name: DEXSeq\n  version: 1.2.0\nDNAcopy:\n  description: DNA copy number data analysis\n  homepage_uri: ''\n  name: DNAcopy\n  version: 1.30.0\nDiffBind:\n  description: Differential Binding Analysis of ChIP-Seq peak data\n  homepage_uri: ''\n  name: DiffBind\n  version: 1.2.0\nDynDoc:\n  description: Dynamic document tools\n  homepage_uri: ''\n  name: DynDoc\n  version: 1.34.0\nEDASeq:\n  description: Exploratory Data Analysis and Normalization for RNA-Seq\n  homepage_uri: ''\n  name: EDASeq\n  version: 1.2.0\nGEOquery:\n  description: Get data from NCBI Gene Expression Omnibus (GEO)\n  homepage_uri: http://watson.nci.nih.gov/~sdavis\n  name: GEOquery\n  version: 2.23.5\nGGBase:\n  description: GGBase infrastructure for genetics of gene expression package\\nGGtools\n  homepage_uri: ''\n  name: GGBase\n  version: 3.18.0\nGGtools:\n  description: software and data for analyses in genetics of gene expression\n  homepage_uri: ''\n  name: GGtools\n  version: 4.4.0\nGO.db:\n  description: A set of annotation maps describing the entire Gene Ontology\n  homepage_uri: ''\n  name: GO.db\n  version: 2.7.1\nGOFunction:\n  description: 'GO-function: deriving biologcially relevant functions from\\nstatistically\n    significant functions'\n  homepage_uri: ''\n  name: GOFunction\n  version: 1.2.0\nGOstats:\n  description: Tools for manipulating GO and microarrays.\n  homepage_uri: ''\n  name: GOstats\n  version: 2.22.0\nGSEABase:\n  description: Gene set enrichment data structures and methods\n  homepage_uri: ''\n  name: GSEABase\n  version: 1.18.0\nGWASExactHW:\n  description: Exact Hardy-Weinburg testing for Genome Wide Association Studies\n  homepage_uri: ''\n  name: GWASExactHW\n  version: '1.0'\nGenomicFeatures:\n  description: Tools for making and manipulating transcript centric annotations\n  homepage_uri: ''\n  name: GenomicFeatures\n  version: 1.8.1\nGenomicRanges:\n  description: Representation and manipulation of genomic intervals\n  homepage_uri: ''\n  name: GenomicRanges\n  version: 1.8.6\nHH:\n  description: 'Statistical Analysis and Data Display: Heiberger and Holland'\n  homepage_uri: ''\n  name: HH\n  version: 2.3-15\nHilbertVis:\n  description: Hilbert curve visualization\n  homepage_uri: http://www.ebi.ac.uk/~anders/hilbert\n  name: HilbertVis\n  version: 1.14.0\nHmisc:\n  description: Harrell Miscellaneous\n  homepage_uri: http://biostat.mc.vanderbilt.edu/wiki/Main/Hmisc,\\nhttp://biostat.mc.vanderbilt.edu/trac/Hmisc\n  name: Hmisc\n  version: 3.9-3\nIRanges:\n  description: Infrastructure for manipulating intervals on sequences\n  homepage_uri: ''\n  name: IRanges\n  version: 1.14.3\nIso:\n  description: Functions to perform isotonic regression.\n  homepage_uri: http://www.math.unb.ca/~rolf/\n  name: Iso\n  version: 0.0-10\nKEGG.db:\n  description: A set of annotation maps for KEGG\n  homepage_uri: ''\n  name: KEGG.db\n  version: 2.7.1\nKernSmooth:\n  description: Functions for kernel smoothing for Wand & Jones (1995)\n  homepage_uri: ''\n  name: KernSmooth\n  version: 2.23-7\nLowRankQP:\n  description: Low Rank Quadratic Programming\n  homepage_uri: ''\n  name: LowRankQP\n  version: 1.0.1\nMASS:\n  description: Support Functions and Datasets for Venables and Ripley's MASS\n  homepage_uri: http://www.stats.ox.ac.uk/pub/MASS4/\n  name: MASS\n  version: 7.3-18\nMEDIPS:\n  description: MeDIP-Seq data analysis\n  homepage_uri: ''\n  name: MEDIPS\n  version: 1.6.0\nMatrix:\n  description: Sparse and Dense Matrix Classes and Methods\n  homepage_uri: http://Matrix.R-forge.R-project.org/\n  name: Matrix\n  version: 1.0-7\nMotIV:\n  description: Motif Identification and Validation\n  homepage_uri: ''\n  name: MotIV\n  version: 1.10.0\nNMF:\n  description: Algorithms and framework for Nonnegative Matrix Factorization\\n(NMF).\n  homepage_uri: http://nmf.r-forge.r-project.org\n  name: NMF\n  version: 0.5.06\nPICS:\n  description: Probabilistic inference of ChIP-seq\n  homepage_uri: ''\n  name: PICS\n  version: 1.10.0\nR.methodsS3:\n  description: Utility function for defining S3 methods\n  homepage_uri: http://www.braju.com/R/\n  name: R.methodsS3\n  version: 1.2.2\nR2HTML:\n  description: HTML exportation for R objects\n  homepage_uri: http://www.r-project.org\n  name: R2HTML\n  version: '2.2'\nR453Plus1Toolbox:\n  description: A package for importing and analyzing data from Roche's Genome\\nSequencer\n    System.\n  homepage_uri: ''\n  name: R453Plus1Toolbox\n  version: 1.6.1\nRBGL:\n  description: An interface to the BOOST graph library\n  homepage_uri: http://www.bioconductor.org\n  name: RBGL\n  version: 1.32.0\nRColorBrewer:\n  description: ColorBrewer palettes\n  homepage_uri: ''\n  name: RColorBrewer\n  version: 1.0-5\nRCurl:\n  description: General network (HTTP/FTP/...) client interface for R\n  homepage_uri: http://www.omegahat.org/RCurl\n  name: RCurl\n  version: 1.91-1\nRMySQL:\n  description: R interface to the MySQL database\n  homepage_uri: http://biostat.mc.vanderbilt.edu/RMySQL,\\nhttps://github.com/jeffreyhorner/RMySQL\n  name: RMySQL\n  version: 0.9-3\nRSQLite:\n  description: SQLite interface for R\n  homepage_uri: ''\n  name: RSQLite\n  version: 0.11.1\nRSQLite.extfuns:\n  description: Math and String Extension Functions for RSQLite\n  homepage_uri: http://www.sqlite.org/contrib/\n  name: RSQLite.extfuns\n  version: 0.0.1\nRcpp:\n  description: Seamless R and C++ Integration\n  homepage_uri: http://dirk.eddelbuettel.com/code/rcpp.html,\\nhttp://romainfrancois.blog.free.fr/index.php?category/R-package/Rcpp\n  name: Rcpp\n  version: 0.9.10\nRgraphviz:\n  description: Provides plotting capabilities for R graph objects\n  homepage_uri: ''\n  name: Rgraphviz\n  version: 1.34.1\nRmpi:\n  description: Interface (Wrapper) to MPI (Message-Passing Interface)\n  homepage_uri: http://www.stats.uwo.ca/faculty/yu/Rmpi\n  name: Rmpi\n  version: 0.5-9\nRolexa:\n  description: Statistical analysis of Solexa sequencing data\n  homepage_uri: ''\n  name: Rolexa\n  version: 1.12.0\nRsamtools:\n  description: Binary alignment (BAM), variant call (BCF), or tabix file import\n  homepage_uri: http://bioconductor.org/packages/release/bioc/html/Rsamtools.html\n  name: Rsamtools\n  version: 1.8.5\nSRAdb:\n  description: A compilation of metadata from NCBI SRA and tools\n  homepage_uri: http://watson.nci.nih.gov/\n  name: SRAdb\n  version: 1.10.0\nSVGAnnotation:\n  description: Tools for post-processing SVG plots created in R\n  homepage_uri: ''\n  name: SVGAnnotation\n  version: 0.9-0\nShortRead:\n  description: Classes and methods for high-throughput short-read sequencing\\ndata.\n  homepage_uri: ''\n  name: ShortRead\n  version: 1.14.4\nSparseM:\n  description: Sparse Linear Algebra\n  homepage_uri: http://www.econ.uiuc.edu/~roger/research/sparse/sparse.html\n  name: SparseM\n  version: '0.96'\nTeachingDemos:\n  description: Demonstrations for teaching and learning\n  homepage_uri: ''\n  name: TeachingDemos\n  version: '2.8'\nVariantAnnotation:\n  description: Annotation of Genetic Variants\n  homepage_uri: ''\n  name: VariantAnnotation\n  version: 1.2.8\nXML:\n  description: Tools for parsing and generating XML within R and S-Plus.\n  homepage_uri: http://www.omegahat.org/RSXML\n  name: XML\n  version: 3.9-4\nabind:\n  description: Combine multi-dimensional arrays\n  homepage_uri: ''\n  name: abind\n  version: 1.4-0\naffy:\n  description: Methods for Affymetrix Oligonucleotide Arrays\n  homepage_uri: ''\n  name: affy\n  version: 1.34.0\naffyPLM:\n  description: Methods for fitting probe-level models\n  homepage_uri: http://bmbolstad.com\n  name: affyPLM\n  version: 1.32.0\naffyQCReport:\n  description: QC Report Generation for affyBatch objects\n  homepage_uri: ''\n  name: affyQCReport\n  version: 1.34.0\naffydata:\n  description: Affymetrix Data for Demonstration Purpose\n  homepage_uri: ''\n  name: affydata\n  version: 1.11.16\naffyio:\n  description: Tools for parsing Affymetrix data files\n  homepage_uri: ''\n  name: affyio\n  version: 1.24.0\naffylmGUI:\n  description: GUI for affy analysis using limma package\n  homepage_uri: http://bioinf.wehi.edu.au/affylmGUI/\n  name: affylmGUI\n  version: 1.30.0\nakima:\n  description: Interpolation of irregularly spaced data\n  homepage_uri: ''\n  name: akima\n  version: 0.5-7\namap:\n  description: Another Multidimensional Analysis Package\n  homepage_uri: http://mulcyber.toulouse.inra.fr/projects/amap/\n  name: amap\n  version: 0.8-7\nannaffy:\n  description: Annotation tools for Affymetrix biological metadata\n  homepage_uri: ''\n  name: annaffy\n  version: 1.28.0\nannotate:\n  description: Annotation for microarrays\n  homepage_uri: ''\n  name: annotate\n  version: 1.34.0\nape:\n  description: Analyses of Phylogenetics and Evolution\n  homepage_uri: http://ape.mpl.ird.fr/\n  name: ape\n  version: 3.0-4\naplpack:\n  description: 'Another Plot PACKage: stem.leaf, bagplot, faces, spin3R, and\\nsome\n    slider functions'\n  homepage_uri: http://www.wiwi.uni-bielefeld.de/com/wolf/software/aplpack.html\n  name: aplpack\n  version: 1.2.6\naroma.light:\n  description: Light-weight methods for normalization and visualization of\\nmicroarray\n    data using only basic R data types\n  homepage_uri: http://www.aroma-project.org/\n  name: aroma.light\n  version: 1.24.0\narrayQualityMetrics:\n  description: Quality metrics on microarray data sets\n  homepage_uri: ''\n  name: arrayQualityMetrics\n  version: 3.12.0\nash:\n  description: David Scott's ASH routines\n  homepage_uri: ''\n  name: ash\n  version: 1.0-13\nbase:\n  description: The R Base Package\n  homepage_uri: ''\n  name: base\n  version: 2.15.0\nbaySeq:\n  description: Empirical Bayesian analysis of patterns of differential\\nexpression\n    in count data\n  homepage_uri: ''\n  name: baySeq\n  version: 1.10.0\nbeadarray:\n  description: Quality assessment and low-level analysis for Illumina BeadArray\\ndata\n  homepage_uri: ''\n  name: beadarray\n  version: 2.6.0\nbiglm:\n  description: bounded memory linear and generalized linear models\n  homepage_uri: ''\n  name: biglm\n  version: '0.8'\nbigmemory:\n  description: Manage massive matrices with shared memory and memory-mapped\\nfiles\n  homepage_uri: http://www.bigmemory.org\n  name: bigmemory\n  version: 4.2.11\nbioDist:\n  description: Different distance measures\n  homepage_uri: ''\n  name: bioDist\n  version: 1.28.0\nbiomaRt:\n  description: Interface to BioMart databases (e.g. Ensembl, COSMIC ,Wormbase\\nand\n    Gramene)\n  homepage_uri: ''\n  name: biomaRt\n  version: 2.12.0\nbit:\n  description: A class for vectors of 1-bit booleans\n  homepage_uri: http://ff.r-forge.r-project.org/\n  name: bit\n  version: 1.1-8\nbitops:\n  description: Functions for Bitwise operations\n  homepage_uri: ''\n  name: bitops\n  version: 1.0-4.1\nboot:\n  description: Bootstrap Functions (originally by Angelo Canty for S)\n  homepage_uri: ''\n  name: boot\n  version: 1.3-4\ncaTools:\n  description: 'Tools: moving window statistics, GIF, Base64, ROC AUC, etc.'\n  homepage_uri: ''\n  name: caTools\n  version: '1.13'\ncar:\n  description: Companion to Applied Regression\n  homepage_uri: https://r-forge.r-project.org/projects/car/,\\nhttp://CRAN.R-project.org/package=car,\\nhttp://socserv.socsci.mcmaster.ca/jfox/Books/Companion/index.html\n  name: car\n  version: 2.0-12\nchipseq:\n  description: 'chipseq: A package for analyzing chipseq data'\n  homepage_uri: ''\n  name: chipseq\n  version: 1.6.0\nchron:\n  description: Chronological objects which can handle dates and times\n  homepage_uri: ''\n  name: chron\n  version: 2.3-42\nclass:\n  description: Functions for Classification\n  homepage_uri: http://www.stats.ox.ac.uk/pub/MASS4/\n  name: class\n  version: 7.3-3\ncluster:\n  description: Cluster Analysis Extended Rousseeuw et al.\n  homepage_uri: ''\n  name: cluster\n  version: 1.14.2\ncn.mops:\n  description: cn.mops - Mixture of Poissons for CNV detection in NGS data\n  homepage_uri: http://www.bioinf.jku.at/software/cnmops/cnmops.html\n  name: cn.mops\n  version: 1.2.5\ncodetools:\n  description: Code Analysis Tools for R\n  homepage_uri: ''\n  name: codetools\n  version: 0.2-8\ncoin:\n  description: Conditional Inference Procedures in a Permutation Test Framework\n  homepage_uri: ''\n  name: coin\n  version: 1.0-21\ncolorspace:\n  description: Color Space Manipulation\n  homepage_uri: ''\n  name: colorspace\n  version: 1.1-1\ncompiler:\n  description: The R Compiler Package\n  homepage_uri: ''\n  name: compiler\n  version: 2.15.0\ncummeRbund:\n  description: Analysis, exploration, manipulation, and visualization of\\nCufflinks\n    high-throughput sequencing data.\n  homepage_uri: ''\n  name: cummeRbund\n  version: 1.2.0\ndatasets:\n  description: The R Datasets Package\n  homepage_uri: ''\n  name: datasets\n  version: 2.15.0\ndate:\n  description: Functions for handling dates\n  homepage_uri: ''\n  name: date\n  version: 1.2-32\ndevtools:\n  description: Tools to make developing R code easier\n  homepage_uri: ''\n  name: devtools\n  version: '0.6'\ndichromat:\n  description: Color schemes for dichromats\n  homepage_uri: ''\n  name: dichromat\n  version: 1.2-4\ndigest:\n  description: Create cryptographic hash digests of R objects\n  homepage_uri: http://dirk.eddelbuettel.com/code/digest.html\n  name: digest\n  version: 0.5.2\nedgeR:\n  description: Empirical analysis of digital gene expression data in R\n  homepage_uri: ''\n  name: edgeR\n  version: 2.6.7\neffects:\n  description: Effect Displays for Linear, Generalized Linear,\\nMultinomial-Logit,\n    Proportional-Odds Logit Models and\\nMixed-Effects Models\n  homepage_uri: http://www.r-project.org, http://socserv.socsci.mcmaster.ca/jfox/\n  name: effects\n  version: 2.1-1\nevaluate:\n  description: Parsing and evaluation tools that provide more details than the\\ndefault.\n  homepage_uri: ''\n  name: evaluate\n  version: 0.4.2\nexomeCopy:\n  description: CNV detection from exome sequencing read depth\n  homepage_uri: ''\n  name: exomeCopy\n  version: 1.2.0\nfBasics:\n  description: Rmetrics - Markets and Basic Statistics\n  homepage_uri: http://www.rmetrics.org\n  name: fBasics\n  version: '2160.81'\nff:\n  description: memory-efficient storage of large data on disk and fast access\\nfunctions\n  homepage_uri: http://ff.r-forge.r-project.org/\n  name: ff\n  version: 2.2-7\nflashClust:\n  description: Implementation of optimal hierarchical clustering\n  homepage_uri: ''\n  name: flashClust\n  version: 1.01-1\nforeign:\n  description: Read Data Stored by Minitab, S, SAS, SPSS, Stata, Systat, dBase,\\n...\n  homepage_uri: ''\n  name: foreign\n  version: 0.8-50\nformatR:\n  description: Format R Code Automatically\n  homepage_uri: https://github.com/yihui/formatR/wiki\n  name: formatR\n  version: '0.4'\ngage:\n  description: Generally Applicable Gene-set Enrichment for Pathway Analysis\n  homepage_uri: http://www.biomedcentral.com/1471-2105/10/161\n  name: gage\n  version: 2.6.0\ngcrma:\n  description: Background Adjustment Using Sequence Information\n  homepage_uri: ''\n  name: gcrma\n  version: 2.28.0\ngdata:\n  description: Various R programming tools for data manipulation\n  homepage_uri: ''\n  name: gdata\n  version: 2.10.6\ngee:\n  description: Generalized Estimation Equation solver\n  homepage_uri: ''\n  name: gee\n  version: 4.13-18\ngeneLenDataBase:\n  description: Lengths of mRNA transcripts for a number of genomes\n  homepage_uri: ''\n  name: geneLenDataBase\n  version: 0.99.9\ngenefilter:\n  description: 'genefilter: methods for filtering genes from microarray\\nexperiments'\n  homepage_uri: ''\n  name: genefilter\n  version: 1.38.0\ngeneplotter:\n  description: Graphics related functions for Bioconductor\n  homepage_uri: ''\n  name: geneplotter\n  version: 1.34.0\ngenomeIntervals:\n  description: Operations on genomic intervals\n  homepage_uri: ''\n  name: genomeIntervals\n  version: 1.12.0\ngenoset:\n  description: Provides classes similar to ExpressionSet for copy number\\nanalysis\n  homepage_uri: ''\n  name: genoset\n  version: 1.6.0\nggplot2:\n  description: An implementation of the Grammar of Graphics\n  homepage_uri: http://had.co.nz/ggplot2/\n  name: ggplot2\n  version: 0.9.1\ngirafe:\n  description: Genome Intervals and Read Alignments for Functional Exploration\n  homepage_uri: ''\n  name: girafe\n  version: 1.8.0\nglobaltest:\n  description: Testing groups of covariates/features for association with a\\nresponse\n    variable, with applications to gene set testing\n  homepage_uri: http://www.msbi.nl/goeman\n  name: globaltest\n  version: 5.10.0\ngoseq:\n  description: Gene Ontology analyser for RNA-seq and other length biased data\n  homepage_uri: ''\n  name: goseq\n  version: 1.8.0\ngplots:\n  description: Various R programming tools for plotting data\n  homepage_uri: ''\n  name: gplots\n  version: 2.11.0\ngrDevices:\n  description: The R Graphics Devices and Support for Colours and Fonts\n  homepage_uri: ''\n  name: grDevices\n  version: 2.15.0\ngraph:\n  description: 'graph: A package to handle graph data structures'\n  homepage_uri: ''\n  name: graph\n  version: 1.34.0\ngraphics:\n  description: The R Graphics Package\n  homepage_uri: ''\n  name: graphics\n  version: 2.15.0\ngrid:\n  description: The Grid Graphics Package\n  homepage_uri: ''\n  name: grid\n  version: 2.15.0\ngsubfn:\n  description: Utilities for strings and function arguments.\n  homepage_uri: http://gsubfn.googlecode.com\n  name: gsubfn\n  version: 0.6-3\ngtools:\n  description: Various R programming tools\n  homepage_uri: ''\n  name: gtools\n  version: 2.6.2\nhdf5:\n  description: HDF5\n  homepage_uri: ''\n  name: hdf5\n  version: 1.6.9\nhdrcde:\n  description: Highest density regions and conditional density estimation\n  homepage_uri: http://www.robjhyndman.com/software/hdrcde\n  name: hdrcde\n  version: '2.16'\nhexbin:\n  description: Hexagonal Binning Routines\n  homepage_uri: ''\n  name: hexbin\n  version: 1.26.0\nhgu95av2.db:\n  description: Affymetrix Human Genome U95 Set annotation data (chip hgu95av2)\n  homepage_uri: ''\n  name: hgu95av2.db\n  version: 2.7.1\nhighlight:\n  description: Syntax highlighter\n  homepage_uri: http://r-forge.r-project.org/projects/highlight/,\\nhttp://romainfrancois.blog.free.fr/index.php?category/R-package/highlight\n  name: highlight\n  version: 0.3.1\nhtSeqTools:\n  description: Quality Control, Visualization and Processing for\\nHigh-Throughput\n    Sequencing data\n  homepage_uri: ''\n  name: htSeqTools\n  version: 1.2.0\nhwriter:\n  description: HTML Writer - Outputs R objects in HTML format\n  homepage_uri: http://www.embl.de/~gpau/hwriter/\n  name: hwriter\n  version: '1.3'\nimpute:\n  description: 'impute: Imputation for microarray data'\n  homepage_uri: ''\n  name: impute\n  version: 1.30.0\nintervals:\n  description: Tools for working with points and intervals\n  homepage_uri: ''\n  name: intervals\n  version: 0.13.3\nkernlab:\n  description: Kernel-based Machine Learning Lab\n  homepage_uri: ''\n  name: kernlab\n  version: 0.9-14\nknitr:\n  description: A general-purpose package for dynamic report generation in R\n  homepage_uri: http://yihui.name/knitr/\n  name: knitr\n  version: '0.6'\nks:\n  description: Kernel smoothing\n  homepage_uri: http://www.mvstat.net/tduong\n  name: ks\n  version: 1.8.8\nlabeling:\n  description: Axis Labeling\n  homepage_uri: ''\n  name: labeling\n  version: '0.1'\nlattice:\n  description: Lattice Graphics\n  homepage_uri: http://lattice.r-forge.r-project.org/\n  name: lattice\n  version: 0.20-6\nlatticeExtra:\n  description: Extra Graphical Utilities Based on Lattice\n  homepage_uri: http://latticeextra.r-forge.r-project.org/\n  name: latticeExtra\n  version: 0.6-19\nleaps:\n  description: regression subset selection\n  homepage_uri: ''\n  name: leaps\n  version: '2.9'\nlimSolve:\n  description: Solving Linear Inverse Models\n  homepage_uri: ''\n  name: limSolve\n  version: 1.5.3\nlimma:\n  description: Linear Models for Microarray Data\n  homepage_uri: http://bioinf.wehi.edu.au/limma\n  name: limma\n  version: 3.12.1\nlmtest:\n  description: Testing Linear Regression Models\n  homepage_uri: ''\n  name: lmtest\n  version: 0.9-30\nlocfit:\n  description: Local Regression, Likelihood and Density Estimation.\n  homepage_uri: ''\n  name: locfit\n  version: 1.5-8\nlpSolve:\n  description: Interface to Lp_solve v. 5.5 to solve linear/integer programs\n  homepage_uri: ''\n  name: lpSolve\n  version: 5.6.6\nlumi:\n  description: BeadArray Specific Methods for Illumina Methylation and\\nExpression\n    Microarrays\n  homepage_uri: ''\n  name: lumi\n  version: 2.8.0\nmakecdfenv:\n  description: CDF Environment Maker\n  homepage_uri: ''\n  name: makecdfenv\n  version: 1.34.0\nmaptree:\n  description: Mapping, pruning, and graphing tree models\n  homepage_uri: http://www.epa.gov/wed/pages/staff/white/\n  name: maptree\n  version: 1.4-6\nmarkdown:\n  description: Markdown rendering for R\n  homepage_uri: ''\n  name: markdown\n  version: 0.5.1\nmarray:\n  description: Exploratory analysis for two-color spotted microarray data\n  homepage_uri: http://www.maths.usyd.edu.au/u/jeany/\n  name: marray\n  version: 1.34.0\nmatrixStats:\n  description: Methods that apply to rows and columns of a matrix\n  homepage_uri: ''\n  name: matrixStats\n  version: 0.5.0\nmclust:\n  description: Model-Based Clustering / Normal Mixture Modeling\n  homepage_uri: http://www.stat.washington.edu/research/reports/2006/tr504.pdf\n  name: mclust\n  version: 3.4.11\nmemoise:\n  description: Memoise functions\n  homepage_uri: ''\n  name: memoise\n  version: '0.1'\nmethods:\n  description: Formal Methods and Classes\n  homepage_uri: ''\n  name: methods\n  version: 2.15.0\nmethylumi:\n  description: Handle Illumina methylation data\n  homepage_uri: ''\n  name: methylumi\n  version: 2.2.0\nmgcv:\n  description: Mixed GAM Computation Vehicle with GCV/AIC/REML smoothness\\nestimation\n  homepage_uri: ''\n  name: mgcv\n  version: 1.7-18\nmisc3d:\n  description: Miscellaneous 3D Plots\n  homepage_uri: ''\n  name: misc3d\n  version: 0.8-2\nmodeltools:\n  description: Tools and Classes for Statistical Models\n  homepage_uri: ''\n  name: modeltools\n  version: 0.2-19\nmultcomp:\n  description: Simultaneous Inference in General Parametric Models\n  homepage_uri: The publishers web page is\\nhttp://www.crcpress.com/product/isbn/9781584885740\n  name: multcomp\n  version: 1.2-12\nmulticore:\n  description: Parallel processing of R code on machines with multiple cores or\\nCPUs\n  homepage_uri: http://www.rforge.net/multicore/\n  name: multicore\n  version: 0.1-7\nmulttest:\n  description: Resampling-based multiple hypothesis testing\n  homepage_uri: ''\n  name: multtest\n  version: 2.12.0\nmunsell:\n  description: Munsell colour system\n  homepage_uri: ''\n  name: munsell\n  version: '0.3'\nmvtnorm:\n  description: Multivariate Normal and t Distributions\n  homepage_uri: ''\n  name: mvtnorm\n  version: 0.9-9992\nnleqslv:\n  description: Solve systems of non linear equations\n  homepage_uri: ''\n  name: nleqslv\n  version: 1.9.3\nnlme:\n  description: Linear and Nonlinear Mixed Effects Models\n  homepage_uri: ''\n  name: nlme\n  version: 3.1-104\nnnet:\n  description: Feed-forward Neural Networks and Multinomial Log-Linear Models\n  homepage_uri: http://www.stats.ox.ac.uk/pub/MASS4/\n  name: nnet\n  version: 7.3-1\nnumDeriv:\n  description: Accurate Numerical Derivatives\n  homepage_uri: http:http://optimizer.r-forge.r-project.org/\n  name: numDeriv\n  version: 2012.3-1\noneChannelGUI:\n  description: A graphical interface designed to facilitate analysis of\\nmicroarrays\n    and miRNA/RNA-seq data on laptops.\n  homepage_uri: http://www.bioinformatica.unito.it/oneChannelGUI/\n  name: oneChannelGUI\n  version: 1.22.7\norg.Hs.eg.db:\n  description: Genome wide annotation for Human\n  homepage_uri: ''\n  name: org.Hs.eg.db\n  version: 2.7.1\npamr:\n  description: 'Pam: prediction analysis for microarrays'\n  homepage_uri: ''\n  name: pamr\n  version: '1.54'\nparallel:\n  description: Support for Parallel computation in R\n  homepage_uri: ''\n  name: parallel\n  version: 2.15.0\nparser:\n  description: Detailed R source code parser\n  homepage_uri: http://www.r-enthusiasts.com\n  name: parser\n  version: 0.0-14\npermute:\n  description: Functions for generating restricted permutations of data\n  homepage_uri: http://vegan.r-forge.r-project.org/\n  name: permute\n  version: 0.7-0\npi0:\n  description: Estimating the proportion of true null hypotheses for FDR\n  homepage_uri: ''\n  name: pi0\n  version: 1.3-200\nplotrix:\n  description: Various plotting functions\n  homepage_uri: ''\n  name: plotrix\n  version: 3.4-1\nplyr:\n  description: Tools for splitting, applying and combining data\n  homepage_uri: http://had.co.nz/plyr\n  name: plyr\n  version: 1.7.1\npreprocessCore:\n  description: A collection of pre-processing functions\n  homepage_uri: ''\n  name: preprocessCore\n  version: 1.18.0\nproto:\n  description: Prototype object-based programming\n  homepage_uri: http://r-proto.googlecode.com\n  name: proto\n  version: 0.3-9.2\nquadprog:\n  description: Functions to solve Quadratic Programming Problems.\n  homepage_uri: ''\n  name: quadprog\n  version: 1.5-4\nqvalue:\n  description: Q-value estimation for false discovery rate control\n  homepage_uri: ''\n  name: qvalue\n  version: 1.30.0\nrGADEM:\n  description: de novo motif discovery\n  homepage_uri: ''\n  name: rGADEM\n  version: 2.4.0\nrelimp:\n  description: Relative Contribution of Effects in a Regression Model\n  homepage_uri: http://go.warwick.ac.uk/relimp, http://go.warwick.ac.uk/dfirth\n  name: relimp\n  version: 1.0-3\nreshape:\n  description: Flexibly reshape data.\n  homepage_uri: http://had.co.nz/reshape\n  name: reshape\n  version: 0.8.4\nreshape2:\n  description: 'Flexibly reshape data: a reboot of the reshape package.'\n  homepage_uri: http://had.co.nz/reshape\n  name: reshape2\n  version: 1.2.1\nrgl:\n  description: 3D visualization device system (OpenGL)\n  homepage_uri: http://rgl.neoscientists.org\n  name: rgl\n  version: 0.92.880\nrjson:\n  description: JSON for R\n  homepage_uri: ''\n  name: rjson\n  version: 0.2.8\nrnaSeqMap:\n  description: rnaSeq secondary analyses\n  homepage_uri: ''\n  name: rnaSeqMap\n  version: 2.10.0\nrpart:\n  description: Recursive Partitioning\n  homepage_uri: \\nhttp://mayoresearch.mayo.edu/mayo/research/biostat/splusfunctions.cfm\n  name: rpart\n  version: 3.1-53\nrtracklayer:\n  description: R interface to genome browsers and their annotation tracks\n  homepage_uri: ''\n  name: rtracklayer\n  version: 1.16.1\nsamr:\n  description: 'SAM: Significance Analysis of Microarrays'\n  homepage_uri: http://www-stat.stanford.edu/~tibs/SAM\n  name: samr\n  version: '2.0'\nsandwich:\n  description: Robust Covariance Matrix Estimators\n  homepage_uri: ''\n  name: sandwich\n  version: 2.2-9\nscales:\n  description: Scale functions for graphics.\n  homepage_uri: ''\n  name: scales\n  version: 0.2.1\nscatterplot3d:\n  description: 3D Scatter Plot\n  homepage_uri: ''\n  name: scatterplot3d\n  version: 0.3-33\nsegmentSeq:\n  description: Methods for identifying small RNA loci from high-throughput\\nsequencing\n    data\n  homepage_uri: ''\n  name: segmentSeq\n  version: 1.8.0\nseqLogo:\n  description: Sequence logos for DNA sequence alignments\n  homepage_uri: ''\n  name: seqLogo\n  version: 1.22.0\nsetRNG:\n  description: Set (Normal) Random Number Generator and Seed\n  homepage_uri: http://distr.r-forge.r-project.org/\n  name: setRNG\n  version: 2011.11-2\nsiggenes:\n  description: Multiple testing using SAM and Efron's empirical Bayes\\napproaches\n  homepage_uri: ''\n  name: siggenes\n  version: 1.30.0\nsimpleaffy:\n  description: Very simple high level analysis of Affymetrix data\n  homepage_uri: http://www.bioconductor.org,\\nhttp://bioinformatics.picr.man.ac.uk/simpleaffy/\n  name: simpleaffy\n  version: 2.32.0\nsm:\n  description: Smoothing methods for nonparametric regression and density\\nestimation\n  homepage_uri: http://www.stats.gla.ac.uk/~adrian/sm\n  name: sm\n  version: 2.2-4.1\nsnow:\n  description: Simple Network of Workstations\n  homepage_uri: ''\n  name: snow\n  version: 0.3-9\nsnowfall:\n  description: Easier cluster computing (based on snow).\n  homepage_uri: ''\n  name: snowfall\n  version: '1.84'\nsnpStats:\n  description: SnpMatrix and XSnpMatrix classes and methods\n  homepage_uri: http://www-gene.cimr.cam.ac.uk/clayton\n  name: snpStats\n  version: 1.6.0\nspatial:\n  description: Functions for Kriging and Point Pattern Analysis\n  homepage_uri: http://www.stats.ox.ac.uk/pub/MASS4/\n  name: spatial\n  version: 7.3-3\nsplines:\n  description: Regression Spline Functions and Classes\n  homepage_uri: ''\n  name: splines\n  version: 2.15.0\nsplots:\n  description: Visualization of high-throughput assays in microtitre plate or\\nslide\n    format\n  homepage_uri: ''\n  name: splots\n  version: 1.22.0\nsqldf:\n  description: Perform SQL Selects on R Data Frames\n  homepage_uri: http://sqldf.googlecode.com\n  name: sqldf\n  version: 0.4-6.4\nstabledist:\n  description: Stable Distribution Functions\n  homepage_uri: http://www.rmetrics.org\n  name: stabledist\n  version: 0.6-4\nstatmod:\n  description: Statistical Modeling\n  homepage_uri: http://www.statsci.org/r\n  name: statmod\n  version: 1.4.14\nstats:\n  description: The R Stats Package\n  homepage_uri: ''\n  name: stats\n  version: 2.15.0\nstats4:\n  description: Statistical Functions using S4 Classes\n  homepage_uri: ''\n  name: stats4\n  version: 2.15.0\nstringr:\n  description: Make it easier to work with strings.\n  homepage_uri: ''\n  name: stringr\n  version: '0.6'\nstrucchange:\n  description: Testing, Monitoring, and Dating Structural Changes\n  homepage_uri: ''\n  name: strucchange\n  version: 1.4-7\nsurvival:\n  description: Survival analysis, including penalised likelihood.\n  homepage_uri: http://r-forge.r-project.org\n  name: survival\n  version: 2.36-14\ntcltk:\n  description: Tcl/Tk Interface\n  homepage_uri: ''\n  name: tcltk\n  version: 2.15.0\ntimeDate:\n  description: Rmetrics - Chronological and Calendarical Objects\n  homepage_uri: http://www.rmetrics.org\n  name: timeDate\n  version: '2160.95'\ntimeSeries:\n  description: Rmetrics - Financial Time Series Objects\n  homepage_uri: http://www.rmetrics.org\n  name: timeSeries\n  version: '2160.94'\ntimsac:\n  description: TIMe Series Analysis and Control package\n  homepage_uri: ''\n  name: timsac\n  version: 1.2.7\ntkWidgets:\n  description: R based tk widgets\n  homepage_uri: ''\n  name: tkWidgets\n  version: 1.34.0\ntools:\n  description: Tools for Package Development\n  homepage_uri: ''\n  name: tools\n  version: 2.15.0\nutils:\n  description: The R Utils Package\n  homepage_uri: ''\n  name: utils\n  version: 2.15.0\nvegan:\n  description: Community Ecology Package\n  homepage_uri: http://cran.r-project.org, http://vegan.r-forge.r-project.org/\n  name: vegan\n  version: 2.0-3\nvsn:\n  description: Variance stabilization and calibration for microarray data\n  homepage_uri: http://www.r-project.org, http://www.ebi.ac.uk/huber\n  name: vsn\n  version: 3.24.0\nwidgetTools:\n  description: Creates an interactive tcltk widget\n  homepage_uri: ''\n  name: widgetTools\n  version: 1.34.0\nxmapcore:\n  description: Core access to the xmap database (installed separately)\n  homepage_uri: http://xmap.picr.man.ac.uk, http://www.bioconductor.org\n  name: xmapcore\n  version: 1.10.0\nxtable:\n  description: Export tables to LaTeX or HTML\n  homepage_uri: http://xtable.r-forge.r-project.org/\n  name: xtable\n  version: 1.7-0\nyacca:\n  description: Yet Another Canonical Correlation Analysis Package\n  homepage_uri: ''\n  name: yacca\n  version: '1.1'\nyaml:\n  description: Methods to convert R data to YAML and back\n  homepage_uri: ''\n  name: yaml\n  version: 2.1.4\nzlibbioc:\n  description: An R packaged zlib-1.2.5\n  homepage_uri: http://bioconductor.org/packages/release/bioc/html/Zlibbioc.html\n  name: zlibbioc\n  version: 1.2.0\nzoo:\n  description: S3 Infrastructure for Regular and Irregular Time Series (Z's\\nordered\n    observations)\n  homepage_uri: http://zoo.R-Forge.R-project.org/\n  name: zoo\n  version: 1.7-7\n"
  },
  {
    "path": "setup.py",
    "content": "#!/usr/bin/env python\n\"\"\"Install cloudbiolinux install libraries for external use.\n\nThis is not needed for running fabric files, but is useful for\nreusing code.\n\"\"\"\nfrom setuptools import setup, find_packages\n\nsetup(name = \"cloudbiolinux\",\n      version = \"0.3a\",\n      author = \"Brad Chapman\",\n      author_email = \"chapmanb@50mail.com\",\n      description = \"configure virtual (or real) machines with tools for biological analyses\",\n      license = \"MIT\",\n      url = \"http://cloudbiolinux.org\",\n      packages = find_packages(),\n      scripts = [],\n      install_requires = [\n          \"PyYAML >= 3.09\",\n          \"fabric >= 1.8.0\"]\n      )\n"
  },
  {
    "path": "test/README",
    "content": "Integration test for CloudBioLinux.\n\nWhen you have a base install running, you only need to supply the\n(local) IP address of the VM, and the password-less key for the \ninstall user (who has sudo bash). The tests get invoked with\n\n  ./test_biolinux [-i key.pub] [-u user] IPaddr [Flavor]\n\nFor further options see\n\n  ./test_biolinux --help\n\nAlternatively try the self running test using Virtualbox and Vagrant\nwith:\n\n  ./test_vagrant --help\n\n== Testing issues with Vagrant ==\n\nThe test system depends on a stack of tools, including virtualbox,\nvagrant, ssh, ruby and related libraries. Things can go wrong. Here we\nlist a number of common problems.\n\n=== Error with ssh config ===\n\nThe test system appends login code to enter a running vagrant\nvirtualbox without asking for a password. Sometimes the contents of\n$HOME/.ssh/config get corrupted.  Check the contents out, and remove\nall wrong information with\n\n  vi ~/.ssh/config\n\nand rerun the test.\n\n=== Error suggests 'vagrant up' ===\n\nIf something went wrong with the install, Vagrant may complain that\nyou need to run 'vagrant up' first. In practise, this may mean:\n\n* Remove other running vagrant instances (vagrant halt)\n* Remove an existing (faulty) instance using virtualbox\n* Check ~/.ssh/config contains no faulty lines\n* Rerun the tests from scratch, i.e. without the '--continue' switch\n\n=== Fatal error: Name lookup failed for vagrant\n\nProbably there is another VirtualBox + vagrant running somewhere. This\nconflicts with the information in ./ssh/config. Easy solution: remove\nand kill the running instance first.\n\n=== Error: ffi.rb:106:in `call_and_check': Error in API call to get_state ===\n\nThis is Ruby trying into the vagrant API. You need to reinstall the\nvagrant and virtualbox gems to match your version of Ruby.\n\n=== Grub error\n\nOn a kernel upgrade, the test scripts may want to install grub, which is\nthe boot loader. Vagrant does not need that, so don't try to install (select \nignore).\n"
  },
  {
    "path": "test/test_biolinux",
    "content": "#! /usr/bin/env ruby\n#\n# Integration testing of BioLinux targets\n#\n#   Tests a Biolinux flavor on a running VM. First fabric is invoked\n#   with the flavor. Integration tests for the flavor follow.\n#\n# The integration test aims to catch inconsistencies in the installation,\n# and can be run to test changes in the source tree.\n#\n# Requirements:\n#\n#   gem install net-ssh\n#\n# For more information see:\n#\n#   ./test/test_biolinux --help\n#   ./test/test_biolinux --list\n#\n# Easiest is to create a Host entry in ./ssh/config. See also the hacking\n# documents in ./doc/\n\nrequire 'rubygems'\nrequire 'yaml'\nrequire 'optparse'\n\nbegin\n  require 'net/ssh'\n  # require 'net/scp'\nrescue LoadError\n  $stderr.print \"Please install net-ssh with 'gem install net-ssh'\"\n  exit 1\nend\n\n# Available CloudBioLinux testing Flavors - this could be read from a YAML file.\nCONFIG = <<EOCONFIG\n---\nMinimal:\n  :fabricrc: contrib/flavor/minimal/fabricrc_debian.txt\n  :packagelist: contrib/flavor/minimal\nBoinc:     \n  # Boinc is a Minimal flavor for running Boinc clients\n  :fabricrc:    contrib/flavor/boinc/fabricrc_debian.txt\n  :packagelist: contrib/flavor/boinc\nPhylogeny:  \n  # Phylogeny is a Minimal flavor for running Phylogenetic software in the Cloud\n  :fabricrc:    contrib/flavor/phylogeny/fabricrc_debian.txt\n  :packagelist: contrib/flavor/phylogeny\n  :call_test:   test_phylogeny_flavor\nBioNode: \n  :fabricrc:    contrib/bionode/fabricrc_bionode.txt\n  :packagelist: contrib/bionode\nEOCONFIG\n\nconfig = YAML::load(CONFIG)\n\nprint \"Integration test for CloudBioLinux (by Pjotr Prins and Brad Chapman, 2012)\\n\"\nprint \"BioLinux test frame work running on Ruby \"+RUBY_VERSION+\"\\n\"\n\n# find location of tests\npath = File.dirname(__FILE__)\n$: << path+'/testlib'\n\n# load libraries\nrequire 'test_support'\nrequire 'test_biolinux'\n\ndef list_boxes config\n  print \"BioLinux Flavors available for testing:\\n\\n\"\n  config.each do | flavor, properties |\n    print \"* \", flavor,\"\\n\"\n  end\nend\n\nif ARGV.index('--list')\n  list_boxes config\n  exit 0\nend\n\nif ARGV.index('-h') or ARGV.index('--help')\n  print <<USAGE\n\n    Note this testing framework is under development, and is meant to\n    work on Debian-type systems.\n\n  Usage:\n\n    test_biolinux [-i key] [-u user] hostname [flavor]\n\n  Where hostname can be a (local) IP address or an entry in ~/.ssh/config.\n\n  Options:\n\n      -i            ssh key (passwordless)\n      -p            ssh port (default 22)\n      --password    use password\n      -u            ssh user login (user should have sudo bash)\n      --list        List Flavors available for testing\n      --help        Print this info\n      flavor        The test Flavor name (default is 'Minimal')\n\n  Example:\n\n     cd to VM directory\n\n       cd ~/my_test_vms/\n\n     test Minimal flavor - this will pull the remote box to local disk once\n\n       $cloudbiolinux/test/test_biolinux -u biolinux 10.0.2.1 Minimal\n\n     after a fix, we can continue where we left off\n\n       $cloudbiolinux/test/test_biolinux -u biolinux 10.0.2.1 Minimal\n\n     or, with a few more options\n\n       $cloudbiolinux/test/test_biolinux -p 2222 -u biolinux -i ~/.ssh/biolinux.pub localhost Minimal\n\n  Available flavors are:\n\nUSAGE\n  list_boxes config\n  exit 0\nend\n\nOptionParser.new do |o|\n  o.on('-i sshkey') { |sshkey| $sshkey = sshkey }\n  o.on('-p port') { |port| $port = port }\n  o.on('-u user') { |user| $user = user }\n  o.on('--password pwd') { |pwd| $pwd = pwd }\n  o.on('--continue') { |continue| $continue = continue }\n  o.parse!\nend\n\nif ARGV.size == 0\n  $hostname = 'biolinux'\nelse\n  $hostname = ARGV.shift\n  $flavor = ARGV.shift\nend\n# default values\n$hostname = '127.0.0.1' if not $hostname\n$flavor = 'Minimal' if not $flavor\n\nprint \"Testing #{$flavor} (#{$user}@#{$hostname})\\n\"\n\n# Fire up the CloudBioLinux fab file - this installs the software\n# on the VM\nsrcpath = File.join(File.dirname(__FILE__),\"..\")\nfabricrc = srcpath + '/' + config[$flavor][:fabricrc]\npackagelist = srcpath + '/' + config[$flavor][:packagelist]\n\nfab_opts = \"\"\nfab_opts += \" -p #{$pwd}\" if $pwd\nfab_opts += \" -u #{$user}\" if $user\nfab_opts += \" -i #{$sshkey}\" if $sshkey\nfab_opts += \" -H #{$hostname}\"\nfab_opts += \":#{$port}\" if $port\n\n$stderr.print \"Logging in with fab through ssh hostname <#{$hostname}>, port <#{$port}>, user <#{$user}>, passwd <#{$pwd}>\\n\"\n\nrun \"fab #{fab_opts} -f #{srcpath}/fabfile.py -c #{fabricrc} install_biolinux:flavor=#{packagelist}\"\n\nssh_opts = {}\nssh_opts[:password] = $pwd if $pwd\nssh_opts[:port] = $port if $port\nNet::SSH.start($hostname, $user, ssh_opts) do |ssh|\n  @ssh = ssh\n  @prompt = $hostname\n  @prompt = \"img\" if $hostname == '127.0.0.1'\n  def exec cmd\n    $stderr.print \"#{@prompt}> \",cmd,\"\\n\"\n    @ssh.exec!(cmd)\n  end\n  $stderr.print \"logged in!\\n\"\n  dpkg = exec(\"dpkg -l\")\n  File.open(\"dpkg.lst\",\"w\").write(dpkg)\n  test_for_match(ssh,\"/usr/bin/ruby -v\", \"ruby\")\n  test_in_path(ssh,\"gem\")\n\n  # test for completed install\n  # lastline = ssh.exec!(\"tail -1 /var/log/biolinux.log\")\n  # print lastline\n  log = ssh.exec!(\"cat /var/log/biolinux.log\")\n  File.open(\"biolinux.log\",\"w\").write(log)\n  # if lastline =~ /#{$flavor}/\n    print <<COMPLETE\n\n  SUCCESS!\n\n  According to the tests, the CloudBioLinux integration test for #{$flavor}\n  completed successfully!  You can check the running VM by logging in with:\n\n  Enjoy!\nCOMPLETE\n    exit 0\n  # end\n  print <<INCOMPLETE\n\n  FAILURE!\n\n  Somehow the test did not complete.\n\nINCOMPLETE\n  exit 1\nend\n\n\n"
  },
  {
    "path": "test/test_vagrant",
    "content": "#! /usr/bin/env ruby\n#\n# Integration testing of BioLinux targets\n#\n#   test_vagrant makes use of virtualbox + vagrant, installing a Biolinux\n#   flavor on a bare VM, starting from an image download. The script wraps up\n#   with some tests.\n#\n# You may want to use the ./test/test_biolinux instead - which will test\n# an installed VM (flavor).\n#\n# For more information on test_vagrant see:\n#\n#   ./test/test_vagrant --help\n#   ./test/test_vagrant --list\n#\n# See also the document in ./doc/virtualbox.md\n\nrequire 'rubygems'\n\n# Available testing targets - this could be read from a YAML file.\nBOX = {\n        'Minimal' =>\n           { :url         => 'http://bio4.dnsalias.net/download/biolinux/biolinux-minimal-0.91.box',\n             :md5         => '64cdea1f76028971e53e2cd2f19a490a',\n             :fabricrc    => 'contrib/flavor/minimal/fabricrc_debian.txt',\n             :packagelist => 'contrib/flavor/minimal'\n           },\n        'Boinc' =>  # Boinc is a Minimal flavor for running Boinc clients\n           { :url         => 'http://bio4.dnsalias.net/download/biolinux/biolinux-minimal-0.91.box',\n             :md5         => '64cdea1f76028971e53e2cd2f19a490a',\n             :fabricrc    => 'contrib/flavor/boinc/fabricrc_debian.txt',\n             :packagelist => 'contrib/flavor/boinc'\n           },\n        'Phylogeny' =>  # Phylogeny is a Minimal flavor for running Phylogenetic software in the Cloud\n           { :url         => 'http://bio4.dnsalias.net/download/biolinux/biolinux-phylogeny-debian-0.92-i386.box',\n             :md5         => 'e7481104d24a7896ea1151a14e05d497',\n             :fabricrc    => 'contrib/flavor/phylogeny/fabricrc_debian.txt',\n             :packagelist => 'contrib/flavor/phylogeny',\n             :call_test   => 'test_phylogeny_flavor'\n           },\n        'BioNode' =>\n           { :url         => 'http://bio4.dnsalias.net/download/biolinux/biolinux-minimal-0.91.box',\n             :md5         => '64cdea1f76028971e53e2cd2f19a490a',\n             :fabricrc    => 'contrib/bionode/fabricrc_bionode.txt',\n             :packagelist => 'contrib/bionode'\n           },\n}\n\nprint \"Integration test scripts for CloudBioLinux (by Pjotr Prins and Brad Chapman, 2011)\\n\"\nprint \"BioLinux test frame work running on Ruby \"+RUBY_VERSION+\"\\n\"\n\n# find location of test files\npath = File.dirname(__FILE__)\n$: << path+'/testlib'\nrequire 'test_biolinux'\n\ndef run cmd\n  print \"--> \",cmd\n  system(cmd)  # don't use backticks for fab!\nend\n\ndef error msg\n  $stderr.print \"FATAL ERROR: \",msg\n  exit 1\nend\n\ndef list_boxes\n  print \"BioLinux VMs available for testing:\\n\\n\"\n  BOX.each do | edition, properties |\n    print \"* \", edition, \" (\",properties[:url],\")\\n\"\n  end\nend\n\nif ARGV.index('--list')\n  list_boxes\n  exit 0\nend\n\nif ARGV.index('-h') or ARGV.index('--help')\n  print <<USAGE\n\n    Note this testing framework is under development, and is meant to\n    work on Debian-type systems. It requires vagrant, virtualbox, wget,\n    md5sum. \n\n  Usage:\n\n    test_vagrant [option] [name]\n\n  Options:\n\n      --continue    Continue install on an already running VM (no VM setup)\n      --list        List VMs available for testing\n      --help        Print this info\n      name          The target test VM name (default is 'Minimal')\n\n  Example:\n\n     cd to VM directory\n\n       cd ~/my_test_vms/\n\n     test Minimal - this will pull the remote box to local disk once\n\n       $cloudbiolinux/test/test_vagrant Minimal\n\n     this should pass, but if not (or at interrupt with ^C), start fixing\n     with, for example\n\n       cd vm_name\n       vagrant ssh\n\n     after a fix, continue where we left off\n\n       $cloudbiolinux/test/test_vagrant Minimal --continue\n\n  The base install vm_name.box will be pulled over the Internet, unless it\n  exist in the local directory. On my system the first install run takes\n  approx. 10 minutes (including the original image download at 10Mbs), the\n  second fresh install run, using a local box, takes 5 minutes, and the\n  --continue switch less than half a minute.\n\n  For more information see ./doc/hacking.md\n\n  Available boxes are:\n\nUSAGE\n  list_boxes\n  exit 0\nend\n\nvagrant_hostname = \"default\"\nsrcpath = File.dirname(File.dirname(__FILE__))\nprint srcpath, \"\\n\"\n\nbox_name = ARGV.shift\nuse_running_vm = false\nif box_name == '--continue'\n  use_running_vm = true\n  box_name = ARGV.shift\nend\nbox_name = 'Minimal' if !box_name # Default\n# flavor_name  = ARGV.shift\n# flavor_name  = 'Base' if !flavor_name # not used\n\nprint \"Testing box/VM named '#{box_name}'\\n\"\nraise \"Unknown box for #{box_name}\" if !BOX[box_name]\nremotebox    = BOX[box_name][:url]\nlocalbox     = File.basename(remotebox)\nmd5box       = BOX[box_name][:md5]\ncall_test    = BOX[box_name][:call_test]\nfabricrc     = srcpath + '/' + BOX[box_name][:fabricrc]\npackagelist  = srcpath + '/' + BOX[box_name][:packagelist]\n\n# find or install vagrant\nerror \"Vagrant not installed. Use: gem install vagrant\" if `which vagrant` !~ /vagrant/\n\nrequire 'vagrant'\n\n# Name a temporary dir\ntestname = box_name+\"_BioLinux_\"+File.basename(localbox,\".box\")\n\nif use_running_vm\n  print \"Continuing on an already running VM #{testname}\\n\"\nelse\n  # Remove old box, if there\n  if File.directory?(testname)\n    Dir.chdir(testname) do\n      run \"vagrant destroy\"\n    end\n  end\n  run \"vagrant box remove #{testname}\"\n\n  if not File.exist?(localbox)\n    print \"Fetching #{remotebox}\\n\"\n    run \"wget #{remotebox}\"\n  end\n  error \"Can not download #{remotebox} - is wget installed?\" if not File.exist?(localbox)\n  `md5sum #{localbox}` =~ /(\\w+)/\n  error \"MD5 sum of #{localbox} does not match expected value #{md5box}\" if md5box != $1\n  print \"MD5 sum of #{localbox} is correct\\n\"\n  print \"Starting from local #{localbox}\\n\"\n  run \"vagrant box add #{testname} #{localbox}\"\n\n  Dir.mkdir(testname) if !File.directory?(testname)\n  # Bring VM up\n  Dir.chdir(testname) do\n    print `ls -l`\n    run \"vagrant init #{testname}\"\n  end\nend\n\nvagrant_init = testname + '/Vagrantfile'\nvagrant_init_orig = testname + '/Vagrantfile.original'\nif not File.exist?(vagrant_init)\n  error \"Vagrant VM #{testname} appears not to be installed\"\nend\n\nif not File.exist?(vagrant_init_orig)\n  # modify Vagrant file\n  print `mv #{vagrant_init} #{vagrant_init_orig}`\n  print `sed '$d' < #{vagrant_init_orig} > #{vagrant_init}` # remove last line\n  INJECT =<<EOI\n  # Expand RAM\n  config.vm.customize do |vm|\n    vm.memory_size = 512    # default to 512 Mb RAM\n    # vm.cpu_count = 2      # default to 2 CPUs -- not yet in all Vagrants\n  end\n  # Forward guest port 5900 to host port 5900 and name the mapping \"vnc\"\n  config.vm.forward_port('vnc0', 5900, 5900)\n  config.vm.forward_port('vnc1', 5901, 5901)\nend\nEOI\n  print `echo \"#{INJECT}\" >> #{vagrant_init}`\nend\n\nDir.chdir(testname) do\n  # Always try to start the VM\n  run \"vagrant up\"\nend\n\n# Using the VM\nDir.chdir(testname) do\n  # ssh info\n  ssh_config = `vagrant ssh-config`\n  ssh_config =~ /Host (\\S+)/\n  vagrant_hostname = $1\n  print \"Hostname #{vagrant_hostname}\\n\"\n\n  # Fetch the vagrant ssh key, and store it locally\n  # First see if we can copy a file - if we can there is no need to update .ssh/config\n  File.unlink('remote_hosts') if File.exists?('remote_hosts')\n  run \"scp #{vagrant_hostname}:/etc/hosts remote_hosts\"\n  if !File.exist?('remote_hosts')\n    ssh_config = run \"vagrant ssh-config >> ~/.ssh/config\"\n    run \"scp #{vagrant_hostname}:/etc/hosts remote_hosts\"\n  end\n  raise \"Problem connecting to remote hosts (using scp)\" if !File.exist?('remote_hosts')\n  File.unlink('remote_hosts')\n  # run \"vagrant status\"\n  print \"We can use #{testname}!\\n\"\n\n  # Fire up the BioLinux fab file - this installs the VM\n  run \"fab -f #{srcpath}/fabfile.py -H vagrant -c #{fabricrc} install_biolinux:flavor=#{packagelist},environment=biolinux-test\"\n\n  # Now set up Ruby's vagrant access for testing\n  vagrant = Vagrant::Environment.new\n  raise \"Must run `vagrant up`\" if !vagrant.primary_vm.created?\n  raise \"Must be running!\" if !vagrant.primary_vm.vm.running?\n  vagrant.primary_vm.ssh.execute do |ssh|\n    # copy package list\n    File.open(\"dpkg.lst\",\"w\").write(ssh.exec!(\"dpkg -l\"))\n    # Do we have ruby and tools?\n    test_for_match(ssh,\"/usr/bin/ruby -v\", \"ruby\")\n    test_for_match(ssh,\"/usr/bin/rdoc -v\", \"RDoc\")\n    test_in_path(ssh,\"gem\")\n    # Make sure Ruby 1.9.1 is in Minimal\n    # test_for_match(ssh,\"/usr/bin/ruby1.9.1 -v\", \"1.9\")\n    # Run flavor specific tests\n    eval(call_test+'(vagrant)') if call_test\n    if test_for_completed_install(ssh,vagrant_hostname,box_name)\n      print <<COMPLETE\n\n  SUCCESS!\n\n  The BioLinux installation test of '#{box_name}' completed successfully!\n  You can check the running VM by logging in with:\n\n    cd ./#{testname}\n    vagrant ssh\n\n  Enjoy!\nCOMPLETE\n    exit 0\n    end\n  end\n  print <<INCOMPLETE\n\n  FAILURE!\n\n  Somehow the test did not complete.\n\nINCOMPLETE\n  exit 1\nend\n\n\n"
  },
  {
    "path": "test/testlib/test_biolinux.rb",
    "content": "# Biolinux shared tests\n\ndef test_for_completed_install(ssh, hostname, box_name)\n  # test for completion of the fabfile\n  run \"scp #{hostname}:/var/log/biolinux.log biolinux.log\"\n  lastline = ssh.exec!(\"tail -1 /var/log/biolinux.log\")\n  print lastline\n  error \"Installation looks incomplete \"+lastline if lastline !~ /#{box_name}/\n  true\nend\n\n# Simple comparison using a regex\ndef test_for_match(ssh, cmd, regex)\n  res = ssh.exec!(cmd)\n  if res !~ /#{regex}/\n    error(regex+\" (\"+cmd+\") does not match\")\n  end\nend\n\ndef test_in_path(ssh, program)\n  test_for_match(ssh,\"which \"+program, program)\nend\n\ndef test_phylogeny_flavor(vagrant)\n  vagrant.primary_vm.ssh.execute do |ssh|\n    # test for command line tools\n    test_for_match(ssh,\"beast -help\", \"BEAST\")\n    test_in_path(ssh,\"mb\")\n    test_in_path(ssh,\"mpirun.openmpi\")\n    test_in_path(ssh,\"mrbayes-multi\")\n    # test_for_match(ssh,\"mb --help\", \"MrBayes\")\n    # test_for_match(ssh,\"mpirun.openmpi -np 4 mrbayes-multi -h\", \"MrBayes\")\n    test_for_match(ssh,\"raxmlHPC -v\", \"RAxML\")\n    test_in_path(ssh,\"codeml\")\n  end \nend\n"
  },
  {
    "path": "test/testlib/test_support.rb",
    "content": "# Run a command line command, without intercepting stdout and stderr\ndef run cmd\n  print \"--> \",cmd\n  res = system(cmd)  # don't use backticks for fab!\n  if res == nil\n    error \"Command failed!\"\n  end\n  res\nend\n\n# Error handler\ndef error msg\n  $stderr.print \"\\nFATAL ERROR: \",msg,\"\\n\"\n  exit 1\nend\n\n\n"
  },
  {
    "path": "utils/bootstrap.sh",
    "content": "#!/bin/bash\n\n#  See this blog:\n#  http://alestic.com/2009/08/runurl\n#\n#  The basic Alestic images support execution of the user data, and also\n#  come with \"runurl\" pre-installed.  Therefore we should be able to simply stick the\n#  following into user data:\n#  \n#  #!/bin/bash -ex\n#  runurl cloudbiolinux.com/bootstrap\n#\n#  Of to support non-alestic images:\n#  \n#  #!/bin/bash -ex\n#  which runurl || wget -O- run.alestic.com/install/runurl | bash\n#  runurl cloudbiolinux.com/bootstrap\n\n#  Now the URL given would actually be a redirect to:\n#  http://github.com/chapmanb/bcbb/raw/master/ec2/biolinux/utils/bootstrap.sh\n\nif [ `id -un` != root ] ; then\n\techo \"This script needs to be run as root.\"\n        exit 1\nfi\nHOME=~root; USER=root\n\n#  So, here we go...\ntmpdir=/tmp/bio-linux-bootstrap\n#Skip this bit if the directory was found - eases debugging\nif [ ! -e \"$tmpdir\" ] ; then\n\n\t#  Install git and fabric and python stuff and ssh\n\texport DEBIAN_FRONTEND=noninteractive\n\tapt-get update\n\tapt-get -y install git-core fabric python-setuptools python-yaml openssh-server\n\n\t# Make a directory to work in\n\tmkdir $tmpdir ; cd $tmpdir\n\n\t# Pull the GIT stuff - this is wasteful but I don't know a better way.\n\tgit clone --depth=0 http://github.com/chapmanb/bcbb.git .\n\tls -A | grep -v ec2 | xargs rm -r\n\n\t#Some fixups to the fabfile - these can probably be changed in the upstream but\n        #I want my script to work now...\n\tif ! id -un ubuntu ; then\n\t\techo \"Removing env.user=ubuntu from fabfile\"\n\t\tsed -i -e '/env.user = \"ubuntu\"/d' ec2/biolinux/fabfile.py\n\tfi\nfi\n\ncd \"$tmpdir\"/ec2/biolinux/\n\n# Fabric relies on an ssh connection, but this script is local and running as root,\n# so I need to allow root logins with SSH and get myself a password-less login.\n# What a faff\nssh-keygen -t rsa -N '' -C 'localhost passwordless login key' -f ~/.ssh/id_local_rsa\n( echo -n 'from=\"localhost,::1,127.0.0.0/24\" ' ; cat ~/.ssh/id_local_rsa.pub ) >> ~/.ssh/authorized_keys2\n/usr/sbin/sshd -o PermitRootLogin=without-password -o ListenAddress=localhost:2232 -o AllowGroups=root -o PidFile=/var/run/sshd_local.pid -p 2232\n\n# Run Fabric on localhost\n#fab -i ~/.ssh/id_local_rsa -H localhost:2232 uname_a\nfab -i ~/.ssh/id_local_rsa -H localhost:2232 install_biolinux\n\n# Clean up keys and sshd\nrm ~/.ssh/id_local_rsa.pub ~/.ssh/id_local_rsa ~/.ssh/authorized_keys2\nxargs kill -TERM < /var/run/sshd_local.pid\n\n# Say what we did\necho \"All finished.  You probably want to reboot now.\"\n"
  },
  {
    "path": "utils/cbl_exome_setup.py",
    "content": "#!/usr/bin/env python\n\"\"\"Automate the final steps of configuring the CloudBioLinux exome example.\n\n- Updates configuration file with server details.\n- Provides links to custom Galaxy instance\n- Install latest processing pipeline\n- Start toplevel processing server to run exome analysis.\n- Adds RabbitMQ user and virtual host with correct permissions.\n\nRun the script with sudo or the root user.\n\"\"\"\nimport os\nimport time\nimport shutil\nimport socket\nimport subprocess\nimport contextlib\nimport ConfigParser\n\nimport yaml\n\ndef main():\n    config_dir = \"/export/data/galaxy\"\n    work_dir = \"/export/data/work\"\n    work_user = \"galaxy\"\n    amqp_config = os.path.join(config_dir, \"universe_wsgi.ini\")\n    pp_config = os.path.join(config_dir, \"post_process.yaml\")\n    wait_until_mounted(amqp_config)\n    update_amqp_config(amqp_config, socket.getfqdn())\n    amqp_user, amqp_pass = read_ampq_config(amqp_config)\n    amqp_vhost = read_pp_config(pp_config)\n    install_latest_pipeline()\n    setup_custom_galaxy()\n    run_nextgen_analysis_server(pp_config, work_dir, work_user)\n    setup_rabbitmq(amqp_vhost, amqp_user, amqp_pass)\n\ndef setup_custom_galaxy():\n    \"\"\"Provide links to custom Galaxy instance on shared volume.\n    \"\"\"\n    galaxy_path = \"/mnt/galaxyTools/galaxy-central\"\n    custom_galaxy_path = \"/mnt/galaxyData/galaxy-central-hbc\"\n    storage_dir = \"/export/data/upload/storage\"\n    if not os.path.exists(galaxy_path):\n        subprocess.check_call([\"mkdir\", \"-p\", os.path.split(galaxy_path)[0]])\n        subprocess.check_call([\"ln\", \"-s\", custom_galaxy_path, galaxy_path])\n        subprocess.check_call([\"chown\", \"-R\", \"galaxy:galaxy\",\n                               os.path.split(galaxy_path)[0]])\n    subprocess.check_call([\"chmod\", \"a+rwx\", storage_dir])\n\ndef install_latest_pipeline():\n    url = \"git://github.com/chapmanb/bcbb.git\"\n    tmpdir = \"/tmp\"\n    with chdir(tmpdir):\n        subprocess.check_call([\"git\", \"clone\", url])\n        with chdir(os.path.join(tmpdir, \"bcbb\", \"nextgen\")):\n            subprocess.check_call([\"python\", \"setup.py\", \"install\"])\n        shutil.rmtree(\"bcbb\")\n\nUPSTART_SCRIPT = \"\"\"\ndescription   \"Nextgen sequencing analysis server\"\n\nstart on runlevel [2345]\n\npre-start script\n    mkdir -p {work_dir}\n    chown -R {user} {work_dir}\nend script\n\nexec su -l -c 'nextgen_analysis_server.py -q toplevel {config_file} --basedir={work_dir}' {user}\n\"\"\"\n\n\ndef run_nextgen_analysis_server(pp_config, work_dir, work_user):\n    \"\"\"Run a nextgen sequencing server using Ubuntu upstart.\n    \"\"\"\n    upstart_file = \"/etc/init/nextgen-analysis.conf\"\n    with open(upstart_file, \"w\") as out_handle:\n        out_handle.write(UPSTART_SCRIPT.format(config_file=pp_config,\n                                               work_dir=work_dir,\n                                               user=work_user))\n    subprocess.check_call([\"service\", \"nextgen-analysis\", \"start\"])\n\ndef setup_rabbitmq(vhost, user, passwd):\n    \"\"\"Add virtual host, user and password to RabbitMQ.\n    \"\"\"\n    base_cl = [\"rabbitmqctl\"]\n    subprocess.check_call(base_cl + [\"add_user\", user, passwd])\n    subprocess.check_call(base_cl + [\"add_vhost\", vhost])\n    subprocess.check_call(base_cl + [\"set_permissions\", \"-p\", vhost,\n                                     user, '.*', '.*', '.*'])\n\ndef read_pp_config(fname):\n    \"\"\"Read AMQP vhost from YAML configuration file.\n    \"\"\"\n    with open(fname) as in_handle:\n        config = yaml.safe_load(in_handle)\n    return config[\"distributed\"][\"rabbitmq_vhost\"]\n\ndef read_ampq_config(fname):\n    \"\"\"Get AMQP username and password from configuration file\n    \"\"\"\n    config = ConfigParser.ConfigParser()\n    config.read(fname)\n    return (config.get(\"galaxy_amqp\", \"userid\"),\n            config.get(\"galaxy_amqp\", \"password\"))\n\ndef update_amqp_config(fname, hostname):\n    \"\"\"Update AMQP configuration with internal hostname.\n    \"\"\"\n    orig_stat = os.stat(fname)\n    backup_file = \"{0}.bak\".format(fname)\n    shutil.move(fname, backup_file)\n    with open(backup_file) as in_handle:\n        with open(fname, \"w\") as out_handle:\n            in_amqp = False\n            for line in in_handle:\n                if line.startswith(\"[galaxy_amqp]\"):\n                    in_amqp = True\n                if in_amqp and line.startswith(\"host =\"):\n                    line = \"host = {0}\\n\".format(hostname)\n                    in_amqp = False\n                out_handle.write(line)\n    # make updated file readable by initial user\n    os.chown(fname, orig_stat.st_uid, orig_stat.st_gid)\n\n# ## Utilities\n\ndef wait_until_mounted(fname):\n    \"\"\"Wait up to 3 minutes for mounted directory with file.\n    \"\"\"\n    max_tries = 36\n    wait_sec = 5\n    num_tries = 0\n    while 1:\n        if os.path.exists(fname):\n            break\n        elif num_tries > max_tries:\n            raise ValueError(\"Did not find {f} after {s} seconds.\".format(\n                f=fname, s=max_tries*wait_sec))\n        time.sleep(wait_sec)\n        num_tries += 1\n\n@contextlib.contextmanager\ndef chdir(new_dir):\n    \"\"\"Context manager to temporarily change to a new directory.\n    http://lucentbeing.com/blog/context-managers-and-the-with-statement-in-python/\n    \"\"\"\n    cur_dir = os.getcwd()\n    os.chdir(new_dir)\n    try:\n        yield\n    finally:\n        os.chdir(cur_dir)\n \nif __name__ == \"__main__\":\n    main()\n"
  },
  {
    "path": "utils/cbl_installed_software.py",
    "content": "#!/usr/bin/env python\n\"\"\"Provide dump of software and libraries installed on CloudBioLinux image.\n\nRun from the top level of the cloudbiolinux source directory:\n    python utils/cbl_installed_software.py\n\"\"\"\nimport os\n\nfrom cloudbio import manifest\n\ndef main():\n    out_dir = os.path.join(os.getcwd(), \"manifest\")\n    manifest.create(out_dir)\n\nif __name__ == \"__main__\":\n    main()\n"
  },
  {
    "path": "utils/convert_to_xz.py",
    "content": "#!/usr/bin/env python\n\"\"\"Convert gzipped files on s3 biodata to xz compression format.\n\nThis conversion is designed to save time and space for download.\n\nSome download utilities to speed things up:\naxel, aria2, lftp\n\"\"\"\nimport os\nimport sys\nimport socket\nimport subprocess\n\nimport boto\nimport fabric.api as fabric\n\ndef main(bucket_name):\n    conn = boto.connect_s3()\n    bucket = conn.get_bucket(\"biodata\")\n    for s3_item in bucket.list(\"genomes/\"):\n        if s3_item.name.endswith(\".gz\"):\n            print \"xzipping\", s3_item.name\n            local_file = os.path.basename(s3_item.name)\n            local_xz = \"%s.xz\" % os.path.splitext(local_file)[0]\n            if not os.path.exists(local_xz):\n                if not os.path.exists(local_file):\n                    download_parallel(s3_item.generate_url(7200))\n                    #s3_item.get_contents_to_filename(local_file)\n                local_xz = gzip_to_xz(local_file)\n            swap_s3_item(local_xz, bucket, s3_item)\n            os.remove(local_xz)\n\ndef download_parallel(url):\n    host = socket.gethostbyaddr(socket.gethostname())[0]\n    user = os.environ[\"USER\"]\n    with fabric.settings(host_string=\"%s@%s\" % (user, host)):\n        ncores = fabric.run(\"cat /proc/cpuinfo | grep processor | wc -l\")\n        with fabric.cd(os.getcwd()):\n            fabric.run(\"axel -a -n %s '%s'\" % (ncores, url), shell=False)\n            #fabric.run(\"aria2c -j %s -s %s '%s'\" % (ncores, ncores, url),\n            #           shell=False)\n\ndef swap_s3_item(xz_file, bucket, orig_s3_item):\n    print \" Uploading to S3\"\n    assert os.path.exists(xz_file)\n    new_name = orig_s3_item.name.replace(\".gz\", \".xz\")\n    upload_script = os.path.join(os.path.dirname(__file__), \"s3_multipart_upload.py\")\n    cl = [\"python2.6\", upload_script, xz_file, bucket.name, new_name]\n    subprocess.check_call(cl)\n    orig_s3_item.delete()\n\ndef gzip_to_xz(local_file):\n    cl = [\"gunzip\", local_file]\n    subprocess.check_call(cl)\n    tar_file, _ = os.path.splitext(local_file)\n    cl = [\"xz\", \"-z\", tar_file]\n    subprocess.check_call(cl)\n    return \"%s.xz\" % tar_file\n\nif __name__ == \"__main__\":\n    bucket_name = \"biodata\"\n    main(bucket_name)\n"
  },
  {
    "path": "utils/cwl2yaml_packages.py",
    "content": "#!/usr/bin/env python\n# Parses CWL's \"SoftwareRequirement\" hints section and dumps a cbl compatible yaml file.\n# The purpose with this script is to create smaller composable docker containers for bcbio-nextgen.\n#\n# Usage: cwl2yaml_packages.py test_bcbio_cwl/run_info-cwl-workflow/steps/process_alignment.cwl > cloudbiolinux/contrib/flavor/cwl_dockers/packages-bcbio-alignment.yaml\nimport os\nimport sys\nimport yaml\n\nCWL_STEPS=sys.argv[1]\ncwl_pkgs=yaml.safe_load(open(CWL_STEPS,'r'))\ncbl_yml=dict()\ncbl_pkgs=[]\n\n# take the filename as the flavor/dockerfile name\ncbl_flavor=\"bcbio-\"+os.path.splitext(os.path.basename(sys.argv[1]))[0]\n\nfor pkg in cwl_pkgs['hints'][1]['packages']:\n    cbl_pkgs.append(pkg['package'])\n\ncbl_yml['channels']=['bioconda', 'conda-forge']\ncbl_yml[cbl_flavor]=cbl_pkgs\n\n#print cbl_yml\n\nprint yaml.safe_dump(cbl_yml, default_flow_style=False, indent=4)"
  },
  {
    "path": "utils/get_biolinux_packages.py",
    "content": "\"\"\"Scrape the Biolinux website to retrieve a list of packages they install.\n\nhttp://www.jcvi.org/cms/research/projects/jcvi-cloud-biolinux/included-software\n\nThis needs to run on a machine with an apt system to check for the existance of\npackage names.\n\"\"\"\nimport sys\nimport urllib2\nimport re\nimport subprocess\nimport StringIO\n\nfrom BeautifulSoup import BeautifulSoup\n\ndef main():\n    url = \"http://www.jcvi.org/cms/research/projects/jcvi-cloud-biolinux/included-software\"\n    in_handle = urllib2.urlopen(url)\n    soup = BeautifulSoup(in_handle)\n    tables = soup.findAll(\"table\", {\"class\": \"contenttable\"})\n    to_check = []\n    for t in tables:\n        for row in soup.findAll(\"tr\", {\"class\" : re.compile(\"tableRow.*\")}):\n            for i, item in enumerate(row.findAll(\"p\", {\"class\": \"bodytext\"})):\n                if i == 0:\n                    to_check.append(str(item.contents[0]))\n    to_check = list(set(to_check))\n    packages = [get_package(n) for n in to_check]\n    not_ported = [to_check[i] for i, p in enumerate(packages) if p is None]\n    packages = [p for p in packages if p]\n    print len(to_check), len(packages)\n    with open(\"biolinux-packages.txt\", \"w\") as out_handle:\n        out_handle.write(\"\\n\".join(sorted(packages)))\n    with open(\"biolinux-missing.txt\", \"w\") as out_handle:\n        out_handle.write(\"\\n\".join(sorted(not_ported)))\n\ndef get_package(pname):\n    \"\"\"Try and retrieve a standard or biolinux package for the package name.\n    \"\"\"\n    # custom hacking for painfully general names that take forever\n    if pname in [\"act\", \"documentation\"]:\n        pname = \"bio-linux-%s\" % pname\n    print 'In', pname\n    cl = subprocess.Popen([\"apt-cache\", \"search\", pname], stdout=subprocess.PIPE)\n    cl.wait()\n    for line in cl.stdout.read().split():\n        package = line.split()[0]\n        if package == pname or package == \"bio-linux-%s\" % pname:\n            print 'Out', package\n            return package\n    return None\n\nif __name__ == \"__main__\":\n    main(*sys.argv[1:])\n"
  },
  {
    "path": "utils/get_yum_packages.py",
    "content": "\"\"\"Convert list of apt packages to matching yum packages.\n\nThis needs to run on a machine with yum to check for the existance of\npackage names.\n\"\"\"\nimport os\nimport re\nimport sys\nimport subprocess\nimport platform\nfrom contextlib import nested\nimport StringIO\n\ndef main(orig_file):\n    new_file = \"%s-yum%s\" % os.path.splitext(orig_file)\n    with nested(open(orig_file), open(new_file, \"w\")) as \\\n               (orig_handle, new_handle):\n        for line in orig_handle:\n            if line.lstrip().startswith(\"- \"):\n                base, orig_package = line.split(\"- \")\n                yum_package = get_yum_package(orig_package.strip())\n                if yum_package:\n                    new_handle.write(\"%s- %s\\n\" % (base, yum_package))\n            else:\n                new_handle.write(line)\n\ndef get_yum_package(pname):\n    print 'In', pname\n    # hacks for package names that cause it to hang\n    if pname in [\"ri\"]:\n        return None\n    elif pname in [\"perl\"]:\n        return pname\n    cl = subprocess.Popen([\"yum\", \"search\", pname], stdout=subprocess.PIPE)\n    cl.wait()\n    arch_pname = \"%s.%s\" % (pname, platform.machine())\n    for line in cl.stdout.read().split(\"\\n\"):\n        if line.startswith(arch_pname):\n            return pname\n    return None\n\nif __name__ == \"__main__\":\n    main(*sys.argv[1:])\n"
  },
  {
    "path": "utils/images_and_snapshots.py",
    "content": "import boto\nimport collections\n\nOWNER = '678711657553' # Brad's owner ID\n\ndef images_and_snapshots(owner):\n    \"\"\"Retrieve Biolinux image and snapshot information.\n    \"\"\"\n    conn = boto.connect_ec2()\n    images = conn.get_all_images(owners=[owner])\n    images32 = _sorted_images(images, \"CloudBioLinux 32\")\n    images64 = _sorted_images(images, \"CloudBioLinux 64\")\n    datalibs = _data_libraries(conn, owner)\n    print images32\n    print images64\n    print datalibs\n\ndef _data_libraries(conn, owner):\n    library_types = collections.defaultdict(list)\n    snaps = conn.get_all_snapshots(owner=owner)\n    for snap in snaps:\n        if snap.description.startswith(\"CloudBioLinux Data\"):\n            # the type is everything except the start and date\n            data_type = \" \".join(snap.description.split()[2:-1])\n            library_types[data_type].append(snap)\n    final = dict()\n    for name, snaps in library_types.iteritems():\n        snaps = [(s.description, s) for s in snaps]\n        snaps.sort(reverse=True)\n        final[name] = [(s.id, d) for (d, s) in snaps]\n    return final\n\ndef _sorted_images(images, start_name):\n    \"\"\"Retrieve a sorted list of images with most recent first.\n    \"\"\"\n    images = [(i.name, i) for i in images if i.name.startswith(start_name)]\n    images.sort(reverse=True)\n    return [(i.id, name) for (name, i) in images]\n\nimages_and_snapshots(OWNER)\n"
  },
  {
    "path": "utils/prep_esp_hg38.py",
    "content": "#!/usr/bin/env python\n\"\"\"Prepare hg38 compatible ESP from original download.\n\nhttp://evs.gs.washington.edu/EVS/\n\nThe Download is a tarball of individual VCF files with hg38 coordinates in\nan INFO key. Extract these into a full VCF, then sort, normalize and bgzip.\n\"\"\"\nimport glob\nimport subprocess\n\nfrom bcbio.bam import ref\nfrom bcbio.variation import vcfutils\nfrom bcbio.heterogeneity import chromhacks\n\ndef main():\n    url = \"http://evs.gs.washington.edu/evs_bulk_data/ESP6500SI-V2-SSA137.GRCh38-liftover.snps_indels.vcf.tar.gz\"\n    ref_file = \"../seq/hg38.fa\"\n    subprocess.check_call(\"wget -c -O esp-orig.tar.gz {url}\".format(**locals()), shell=True)\n    subprocess.check_call(\"tar -xzvpf esp-orig.tar.gz\", shell=True)\n    raw_file = \"esp-raw.vcf\"\n    with open(raw_file, \"w\") as out_handle:\n        for i, chrom in enumerate(list(range(1, 23)) + [\"X\", \"Y\"]):\n            fnames = glob.glob(\"*chr%s.snps_indels.vcf\" % chrom)\n            assert len(fnames) == 1, (chrom, fnames)\n            with open(fnames[0]) as in_handle:\n                for line in in_handle:\n                    if line.startswith(\"#\"):\n                        if i == 0:\n                            if line.startswith(\"#CHROM\"):\n                                _add_contigs(out_handle, ref_file)\n                            out_handle.write(line)\n                    else:\n                        parts = line.strip().split(\"\\t\")\n                        key, val = parts[-1].split(\";\")[-1].split(\"=\")\n                        assert key == \"GRCh38_POSITION\"\n                        if val != \"-1\":\n                            new_chrom, new_pos = val.split(\":\")\n                            if chromhacks.is_autosomal_or_sex(new_chrom):\n                                parts[0] = \"chr%s\" % new_chrom\n                                parts[1] = new_pos\n                                out_handle.write(\"\\t\".join(parts) + \"\\n\")\n    out_file = \"ESP6500SI-V2-hg38.vcf.gz\"\n    subprocess.check_call((\"vt sort {raw_file} | vt decompose -s - | \"\n                           \"vt normalize -n -r {ref_file} - | bgzip -c > {out_file}\").format(**locals()),\n                          shell=True)\n    vcfutils.bgzip_and_index(out_file)\n\ndef _add_contigs(out_handle, ref_file):\n    for contig in ref.file_contigs(ref_file):\n        if chromhacks.is_autosomal_or_sex(contig.name):\n            out_handle.write(\"##contig=<ID=%s,length=%s>\\n\" % (contig.name, contig.size))\n\nif __name__ == \"__main__\":\n    main()\n"
  },
  {
    "path": "utils/prepare_cosmic.py",
    "content": "#!/usr/bin/env python\n\"\"\"Prepare combined VCF files of COSMIC resource for cancer variant calling.\n\nThis prepares the specified version and copies the updates into your bcbio installation\nin the correct locations in the variation directory.\n\nBecause of licensing restrictions, download from COSMIC requires registration:\n\nhttps://cancer.sanger.ac.uk/cosmic/download\n\nUsage:\n\n  export COSMIC_USER=\"your@registered.email.edu\"\n  export COSMIC_PASS=\"cosmic_password\"\n  bcbio_python prepare_cosmic.py <cosmic_version> </path/to/bcbio>\n\nReferences:\nhttp://gatkforums.broadinstitute.org/discussion/2226/cosmic-and-dbsnp-files-for-mutect\n\"\"\"\nimport gzip\nimport logging\nimport os\nimport re\nimport requests\nimport subprocess\nimport sys\nimport tempfile\nimport shutil\nfrom argparse import ArgumentParser\n\nfrom bcbio import utils\nfrom bcbio.variation import vcfutils\n\nlogging.basicConfig(format='%(asctime)s [%(levelname).1s] %(message)s', level=logging.INFO)\n\n\ndef main(cosmic_version, bcbio_genome_dir, overwrite=False, clean=False):\n    work_dir = utils.safe_makedir(os.path.join(os.getcwd(), \"cosmic-prep\"))\n    os.chdir(work_dir)\n\n    for genome_build, bcbio_build, add_chr in [(\"GRCh37\", \"GRCh37\", False), (\"GRCh38\", \"hg38\", True)]:\n        bcbio_base = os.path.join(bcbio_genome_dir, \"genomes\", \"Hsapiens\", bcbio_build)\n        installed_file = os.path.join(bcbio_base, \"variation\", f\"cosmic-v{cosmic_version}.vcf.gz\")\n        installed_link = os.path.join(bcbio_base, \"variation\", \"cosmic.vcf.gz\")\n        logging.info(f\"Beginning COSMIC v{cosmic_version} prep for {genome_build}.\")\n        if not os.path.exists(bcbio_base):\n            continue\n        if os.path.exists(installed_file):\n            if not overwrite:\n                logging.info(f\"{installed_file} exists, please use the --overwrite flag to overwrite the existing files if you want to reinstall.\")\n                continue\n            else:\n                logging.info(f\"{installed_file} exists, removing.\")\n                remove_installed(installed_file, installed_link)\n        bcbio_ref = os.path.join(bcbio_base, \"seq\", f\"{bcbio_build}.fa\")\n        cosmic_vcf_files = get_cosmic_vcf_files(genome_build, cosmic_version, clean)\n        sorted_inputs = []\n        for fname in cosmic_vcf_files:\n            sorted_inputs.append(sort_to_ref(fname, bcbio_ref, add_chr=add_chr))\n        out_dir = utils.safe_makedir(os.path.join(f\"v{cosmic_version}\", \"bcbio_ready\", bcbio_build))\n        out_file = os.path.join(out_dir, \"cosmic.vcf.gz\")\n        ready_cosmic = combine_cosmic(sorted_inputs, bcbio_ref, out_file)\n        variation_dir = utils.safe_makedir(os.path.join(bcbio_base, \"variation\"))\n        utils.copy_plus(ready_cosmic, installed_file)\n        logging.info(f\"Created COSMIC v{cosmic_version} resource in {installed_file}.\")\n        logging.info(f\"Linking {installed_file} as {installed_link}.\")\n        make_links(installed_file, installed_link)\n        update_version_file(bcbio_base, cosmic_version)\n        logging.info(f\"Finished COSMIC v{cosmic_version} prep for {genome_build}.\")\n        # prepare hg19 from the GRCh37 file\n        if bcbio_build == \"GRCh37\":\n            genome_build = \"hg19\"\n            logging.info(f\"Prepping COSMIC v{cosmic_version} for {genome_build} from the GRCh37 preparation.\")\n            bcbio_base = os.path.join(bcbio_genome_dir, \"genomes\", \"Hsapiens\", genome_build)\n            if not os.path.exists(bcbio_base):\n                continue\n            if os.path.exists(installed_file):\n                installed_file = os.path.join(bcbio_base, \"variation\", f\"cosmic-v{cosmic_version}.vcf.gz\")\n                installed_link = os.path.join(bcbio_base, \"variation\", \"cosmic.vcf.gz\")\n                if not overwrite:\n                    logging.info(f\"{installed_file} exists, please use the --overwrite flag to overwrite the existing files if you want to reinstall.\")\n                    continue\n                else:\n                    logging.info(f\"{installed_file} exists, removing.\")\n                    remove_installed(installed_file, installed_link)\n            out_dir = utils.safe_makedir(os.path.join(f\"v{cosmic_version}\", \"bcbio_ready\", genome_build))\n            out_file = os.path.join(out_dir, f\"cosmic-v{cosmic_version}.vcf.gz\")\n            logging.info(f\"Translating GRCh37 chromosome names to hg19 chromosome names.\")\n            hg19_cosmic = map_coords_to_ucsc(ready_cosmic, bcbio_ref, out_file)\n            variation_dir = utils.safe_makedir(os.path.join(bcbio_base, \"variation\"))\n            utils.copy_plus(hg19_cosmic, installed_file)\n            logging.info(f\"Created COSMIC v{cosmic_version} resource in {installed_file}.\")\n            logging.info(f\"Linking {installed_file} as {installed_link}.\")\n            make_links(installed_file, installed_link)\n            update_version_file(bcbio_base, cosmic_version)\n            logging.info(f\"Finished COSMIC v{cosmic_version} prep for {genome_build}.\")\n\n\ndef remove_installed(installed_file, installed_link):\n    logging.info(f\"Removing {installed_file}.\")\n    if os.path.exists(installed_file):\n        os.remove(installed_file)\n    installed_index = installed_file + \".tbi\"\n    logging.info(f\"Removing {installed_index}.\")\n    if os.path.exists(installed_index):\n        os.remove(installed_index)\n    logging.info(f\"Removing {installed_link}.\")\n    if os.path.lexists(installed_link):\n        os.remove(installed_link)\n    installed_index = installed_link + \".tbi\"\n    logging.info(f\"Removing {installed_index}.\")\n    if os.path.lexists(installed_index):\n        os.remove(installed_index)\n\n\ndef make_links(installed_file, installed_link):\n    if os.path.islink(installed_link):\n        os.remove(installed_link)\n        os.remove(installed_link + \".tbi\")\n    os.symlink(os.path.basename(installed_file), installed_link)\n    os.symlink(os.path.basename(installed_file + \".tbi\"), installed_link + \".tbi\")\n\n\ndef map_coords_to_ucsc(grc_cosmic, ref_file, out_file):\n    hg19_ref_file = ref_file.replace(\"GRCh37\", \"hg19\")\n    if not os.path.exists(out_file):\n        contig_cl = vcfutils.add_contig_to_header_cl(hg19_ref_file, out_file)\n        cmd = (\"zcat {grc_cosmic} | \"\n               r'sed \"s/^\\([0-9]\\+\\)\\t/chr\\1\\t/g\" | sed \"s/^MT/chrM/g\" | sed \"s/^X/chrX/g\" | sed \"s/^Y/chrY/g\" '\n               \"| {contig_cl} \"\n               \"| bgzip -c > {out_file}\")\n        subprocess.check_call(cmd.format(**locals()), shell=True)\n    if os.path.exists(\"%s-header.txt\" % utils.splitext_plus(out_file)[0]):\n        os.remove(\"%s-header.txt\" % utils.splitext_plus(out_file)[0])\n    return vcfutils.bgzip_and_index(out_file, {})\n\n\ndef _rename_to_ucsc(line):\n    chrom, rest = line.split(\"\\t\", 1)\n    if chrom == \"MT\":\n        new_chrom = \"chrM\"\n    else:\n        new_chrom = \"chr%s\" % chrom\n    return \"%s\\t%s\" % (new_chrom, rest)\n\n\ndef combine_cosmic(fnames, ref_file, out_file):\n    logging.info(f\"Combining COSMIC files to {out_file}.\")\n    if not os.path.exists(out_file):\n        cmd = [\"picard\", \"MergeVcfs\", \"O=%s\" % out_file, \"D=%s\" % ref_file.replace(\".fa\", \".dict\")] + \\\n              [\"I=%s\" % x for x in fnames] + \\\n              [\"USE_JDK_DEFLATER=true\", \"USE_JDK_INFLATER=true\", \"CREATE_INDEX=false\"]\n        subprocess.check_call(cmd)\n    return vcfutils.bgzip_and_index(out_file, {})\n\n\ndef sort_to_ref(fname, ref_file, add_chr):\n    \"\"\"Match reference genome ordering.\n    \"\"\"\n    logging.info(f\"Sorting {fname} to match the order of {ref_file}.\")\n    out_file = \"%s-prep.vcf.gz\" % (fname.replace(\".vcf.gz\", \"\"))\n    if not os.path.exists(out_file):\n        if add_chr:\n            fix_chrom = r'| sed \"s/^\\([0-9]\\+\\)\\t/chr\\1\\t/g\" | sed \"s/^MT/chrM/g\" | sed \"s/^X/chrX/g\" | sed \"s/^Y/chrY/g\" '\n        else:\n            fix_chrom = ''\n        contig_cl = vcfutils.add_contig_to_header_cl(ref_file, out_file)\n        cmd = (\"gunzip -c {fname} {fix_chrom} | \"\n               \"bcftools norm --check-ref s --do-not-normalize -f {ref_file} |\"\n               \"bcftools view -e 'SNP=1' |\"\n               \"gsort /dev/stdin {ref_file}.fai | {contig_cl} | \"\n               \"bgzip -c > {out_file}\")\n        subprocess.check_call(cmd.format(**locals()), shell=True)\n    logging.info(f\"bgzipping and indexing {out_file}.\")\n    return vcfutils.bgzip_and_index(out_file, {})\n\n\ndef get_cosmic_vcf_files(genome_build, cosmic_version, clean):\n    \"\"\"Retrieve using new authentication based download approach.\n\n    GRCh38/cosmic/v85/VCF/CosmicCodingMuts.vcf.gz\n    GRCh38/cosmic/v85/VCF/CosmicNonCodingVariants.vcf.gz\n    \"\"\"\n    vdir = os.path.join(\"v%s\" % cosmic_version, genome_build)\n    if os.path.exists(vdir):\n        if not clean:\n            logging.info(f\"{vdir} files exist, please use the --clean flag to overwrite the existing files if you want to reinstall.\")\n        else:\n            logging.info(f\"{vdir} exists, removing.\")\n            remove_cosmic_directory(vdir)\n    logging.info(\"Downloading COSMIC VCF files.\")\n    url = \"https://cancer.sanger.ac.uk/cosmic/file_download/\"\n    out_dir = utils.safe_makedir(os.path.join(\"v%s\" % cosmic_version, genome_build))\n    fnames = []\n    for ctype in [\"CosmicCodingMuts\", \"CosmicNonCodingVariants\"]:\n        filename = os.path.join(out_dir, \"%s.vcf.gz\" % ctype)\n        if not os.path.exists(filename):\n            filepath = \"%s/cosmic/v%s/VCF/%s.vcf.gz\" % (genome_build, cosmic_version, ctype)\n            logging.info(\"Downloading %s\" % (url + filepath))\n            try:\n                r = requests.get(url + filepath, auth=(os.environ[\"COSMIC_USER\"], os.environ[\"COSMIC_PASS\"]))\n            except KeyError as e:\n                print(\"KeyError: {} not found. Be sure to export your COSMIC_USER and COSMIC_PASS before running in order to download the files\".format(e))\n                raise e\n            download_url = r.json()[\"url\"]\n            r = requests.get(download_url)\n            with open(filename, \"wb\") as f:\n                f.write(r.content)\n        fnames.append(filename)\n    return fnames\n\n\ndef remove_cosmic_directory(installed_directory):\n    logging.info(f\"Removing {installed_directory}.\")\n    shutil.rmtree(installed_directory)\n\ndef update_version_file(bcbio_base, version):\n    \"\"\"\n    update the version of cosmic used in the versions.csv file, adding it if it does not exist\n    \"\"\"\n    versionfile = os.path.join(bcbio_base, \"versions.csv\")\n    updatedfile = os.path.join(bcbio_base, \"versions.csv-tmp\")\n    logging.info(f\"Updating {versionfile}.\")\n    found = False\n    with open(versionfile) as in_handle, open(updatedfile, \"w\") as out_handle:\n        for line in in_handle:\n            tokens = line.split(\",\")\n            if tokens[0] != \"cosmic\":\n                out_handle.write(line)\n            else:\n                # only write it once\n                if not found:\n                    out_handle.write(f\"cosmic,{version}\\n\")\n                found = True\n        if not found:\n            out_handle.write(f\"cosmic,{version}\\n\")\n    shutil.move(updatedfile, versionfile)\n\nif __name__ == \"__main__\":\n    parser = ArgumentParser()\n    parser.add_argument(\"cosmic_version\", help=\"COSMIC version to install.\", default=\"89\")\n    parser.add_argument(\"bcbio_directory\", help=\"Path to bcbio installation. Should contain the 'genomes' directory.\")\n    parser.add_argument(\"--overwrite\", action=\"store_true\", default=False, help=\"Overwrite existing cosmic.vcf.gz files.\")\n    parser.add_argument(\"--clean\", action=\"store_true\", default=False, help=\"Remove pre-downloaded files, if available.\")\n    args = parser.parse_args()\n    main(args.cosmic_version, args.bcbio_directory, args.overwrite, args.clean)\n"
  },
  {
    "path": "utils/prepare_dbsnp.py",
    "content": "\"\"\"Prepare sorted and consolidated dbSNP resources for mouse mm10/GRCh38 in VCF format.\n\"\"\"\nimport datetime\nimport ftplib\nimport gzip\nimport os\nimport subprocess\nfrom argparse import ArgumentParser\nimport re\nimport shutil\n\nFTP = \"ftp.ncbi.nih.gov\"\n\nREMOTES = {\"mm10\": \"snp/organisms/mouse_10090/VCF\",\n           \"canFam3\": \"snp/organisms/dog_9615/VCF/\"}\n\ndef main(org):\n    work_dir = \"tmp-dbsnp-%s\" % org\n    if not os.path.exists(work_dir):\n        os.makedirs(work_dir)\n    conn = ftplib.FTP(FTP, \"anonymous\", \"me@example.com\")\n    conn.cwd(REMOTES[org])\n\n    os.chdir(work_dir)\n    files = []\n    def add_files(x):\n        if x.endswith(\"vcf.gz\"):\n            files.append(get_file(x, REMOTES[org], conn))\n    conn.retrlines(\"NLST\", add_files)\n    out_file = \"%s-dbSNP-%s.vcf\" % (org, datetime.datetime.now().strftime(\"%Y-%m-%d\"))\n    with open(out_file, \"w\") as out_handle:\n        for i, f in enumerate(karyotype_sort(files)):\n            with gzip.open(f) as in_handle:\n                for line in in_handle:\n                    if line.startswith(\"#\"):\n                        if i == 0:\n                            out_handle.write(line)\n                    else:\n                        out_handle.write(\"\\t\".join(fix_info(fix_chrom(line.rstrip().split(\"\\t\")))) + \"\\n\")\n    subprocess.check_call([\"bgzip\", out_file])\n    shutil.move(out_file + \".gz\", os.path.join(os.pardir, out_file + \".gz\"))\n    os.chdir(os.pardir)\n    subprocess.check_call([\"tabix\", \"-p\", \"vcf\", out_file + \".gz\"])\n    shutil.rmtree(work_dir)\n\nmulti_whitespace = re.compile(r\"\\s+\")\n\ndef fix_info(parts):\n    \"\"\"Fix the INFO file to remove whitespace.\n    \"\"\"\n    parts[7] = multi_whitespace.sub(\"_\", parts[7])\n    return parts\n\ndef fix_chrom(parts):\n    MAX_CHROMOSOMES = 50\n    if parts[0] in [str(x) for x in range(1, MAX_CHROMOSOMES)] + [\"X\", \"Y\"]:\n        new_chrom = \"chr%s\" % parts[0]\n    elif parts[0] == \"MT\":\n        new_chrom = \"chrM\"\n    else:\n        raise NotImplementedError(parts)\n    parts[0] = new_chrom\n    return parts\n\ndef get_file(x, ftp_dir, conn):\n    if not os.path.exists(x):\n        print \"Retrieving %s\" % x\n        with open(x, \"wb\") as out_handle:\n            conn = ftplib.FTP(FTP, \"anonymous\", \"me@example.com\")\n            conn.cwd(ftp_dir)\n            conn.retrbinary(\"RETR %s\" % x, out_handle.write)\n    return x\n\ndef karyotype_sort(xs):\n    \"\"\"Sort in karyotypic order to work with GATK's defaults.\n    \"\"\"\n    def karyotype_keyfn(x):\n        for suffix in [\".gz\"]:\n            if x.endswith(suffix):\n                x = x[:-len(suffix)]\n        base = os.path.splitext(os.path.basename(x))[0]\n        for prefix in [\"chr\", \"vcf_chr_\"]:\n            if base.startswith(prefix):\n                base = base[len(prefix):]\n        parts = base.split(\"_\")\n        try:\n            parts[0] =  int(parts[0])\n        except ValueError:\n            pass\n        # unplaced at the very end\n        if isinstance(parts[0], basestring) and parts[0].startswith((\"Un\", \"Alt\", \"Multi\", \"NotOn\")):\n            parts.insert(0, \"z\")\n        # mitochondrial special case -- after X/Y\n        elif parts[0] in [\"M\", \"MT\"]:\n            parts.insert(0, \"x\")\n        # sort random and extra chromosomes after M\n        elif len(parts) > 1:\n            parts.insert(0, \"y\")\n        return parts\n    return sorted(xs, key=karyotype_keyfn)\n\nif __name__ == \"__main__\":\n    parser = ArgumentParser(description=\"Prepare a dbSNP file from NCBI.\")\n    parser.add_argument(\"org_build\", choices=REMOTES.keys(),\n                        help=\"genome build\")\n    args = parser.parse_args()\n    main(args.org_build)\n"
  },
  {
    "path": "utils/prepare_tx_gff.py",
    "content": "#!/usr/bin/env python\n\"\"\"Prepare GFF transcript files for use as input to RNA-seq pipelines\n\nUsage, from within the main genome directory of your organism:\n  prepare_tx_gff.py <organism> <org_build>\n\nrequires these python and external packages which come pre-installed\nwith bcbio using bioconda:\n\nmysql-python\ngffutils\nrequests\npicard\nkallisto\n\"\"\"\nfrom __future__ import print_function\nimport csv\nimport gzip\nimport os\nimport sys\nimport shutil\nimport collections\nimport datetime\nimport subprocess\nimport tempfile\nimport glob\nfrom argparse import ArgumentParser\n\nimport gffutils\nimport requests\n\ntry:\n    import MySQLdb\nexcept:\n    MySQLdb = None\n\n\nfrom bcbio.utils import chdir, safe_makedir, file_exists, get_program_python\nfrom bcbio.rnaseq.gtf import gtf_to_fasta\n\n# ##  Version and retrieval details for Ensembl and UCSC\nensembl_release = \"95\"\nbase_ftp = \"ftp://ftp.ensembl.org/pub/release-{release}/gtf\"\nsupported_oldbuilds = {\"GRCh37\": \"75\", \"hg19\": \"75\"}\nbuild_subsets = {\"hg38-noalt\": \"hg38\"}\n\nucsc_db = \"genome-mysql.cse.ucsc.edu\"\nucsc_user = \"genome\"\n\n# Chromosome name remappings thanks to Devon Ryan\n# https://github.com/dpryan79/ChromosomeMappings\nmanual_remaps = {\"hg38\":\n                 \"https://raw.githubusercontent.com/dpryan79/ChromosomeMappings/master/GRCh38_ensembl2UCSC.txt\"}\n\ndef which(program):\n    def is_exe(fpath):\n        return os.path.isfile(fpath) and os.access(fpath, os.X_OK)\n\n    fpath, fname = os.path.split(program)\n    if fpath:\n        if is_exe(program):\n            return program\n    else:\n        for path in os.environ[\"PATH\"].split(os.pathsep):\n            path = path.strip('\"')\n            exe_file = os.path.join(path, program)\n            if is_exe(exe_file):\n                return exe_file\n    return None\n\ndef manual_ucsc_ensembl_map(org_build):\n    org_build = build_subsets.get(org_build, org_build)\n    requests.packages.urllib3.disable_warnings()\n    r = requests.get(manual_remaps[org_build], verify=False)\n    out = {}\n    for line in r.text.split(\"\\n\"):\n        try:\n            ensembl, ucsc = line.split()\n            out[ensembl] = ucsc\n        except ValueError:\n            pass\n    return out\n\ndef ucsc_ensembl_map_via_download(org_build):\n    \"\"\"Compare .dict files by md5, then length to compare two builds.\n    \"\"\"\n    ensembl_dict_file = get_ensembl_dict(org_build)\n    ucsc_dict_file = get_ucsc_dict(org_build)\n    ensembl_dict = parse_sequence_dict(ensembl_dict_file)\n    ucsc_dict = parse_sequence_dict(ucsc_dict_file)\n    return ensembl_to_ucsc(ensembl_dict, ucsc_dict, org_build)\n\ndef ensembl_to_ucsc(ensembl_dict, ucsc_dict, org_build):\n    name_map = {}\n    for md5, name in ensembl_dict.items():\n        if ucsc_dict.get(md5):\n            name_map[name] = ucsc_dict[md5]\n    map_file = \"%s-map.csv\" % (org_build)\n    with open(map_file, \"w\") as out_handle:\n        writer = csv.writer(out_handle)\n        writer.writerow([\"ensembl\", \"ucsc\"])\n        for md5, name in ensembl_dict.items():\n            ucsc = ucsc_dict.get(md5)\n            if ucsc is not None:\n                writer.writerow([name, ucsc])\n    return name_map\n\ndef ucsc_ensembl_map_via_query(org_build):\n    \"\"\"Retrieve UCSC to Ensembl name mappings from UCSC MySQL database.\n    \"\"\"\n    org_build = build_subsets.get(org_build, org_build)\n    # if MySQLdb is not installed, figure it out via download\n    if not MySQLdb:\n        return ucsc_ensembl_map_via_download(org_build)\n\n    db = MySQLdb.connect(host=ucsc_db, user=ucsc_user, db=org_build)\n    cursor = db.cursor()\n    cursor.execute(\"select * from ucscToEnsembl\")\n    ucsc_map = {}\n    for fields in cursor.fetchall():\n        ucsc = fields[0]\n        ensembl = fields[-1]\n        # workaround for GRCh37/hg19 additional haplotype contigs.\n        # Coordinates differ between builds so do not include these regions.\n        if org_build == \"hg19\" and \"hap\" in ucsc:\n            continue\n        else:\n            ucsc_map[ensembl] = ucsc\n    return ucsc_map\n\n# taxname:\n# biomart_name: name of ensembl gene_id on biomart\n# ucsc_map:\n# fbase: the base filename for ensembl files using this genome\n\nBuild = collections.namedtuple(\"Build\", [\"taxname\", \"biomart_name\",\n                                         \"ucsc_map\", \"fbase\"])\n\nbuild_info = {\n    \"hg19\": Build(\"homo_sapiens\", \"hsapiens_gene_ensembl\",\n                  ucsc_ensembl_map_via_query,\n                  \"Homo_sapiens.GRCh37.\" + supported_oldbuilds[\"GRCh37\"]),\n    \"GRCh37\": Build(\"homo_sapiens\", \"hsapiens_gene_ensembl\",\n                    None,\n                    \"Homo_sapiens.GRCh37.\" + supported_oldbuilds[\"hg19\"]),\n    \"mm9\": Build(\"mus_musculus\", \"mmusculus_gene_ensembl\",\n                 ucsc_ensembl_map_via_query,\n                 \"Mus_musculus.NCBIM37.67\"),\n    \"mm10\": Build(\"mus_musculus\", \"mmusculus_gene_ensembl\",\n                  ucsc_ensembl_map_via_query,\n                  \"Mus_musculus.GRCm38.\" + ensembl_release),\n    \"rn5\": Build(\"rattus_norvegicus\", None,\n                 ucsc_ensembl_map_via_download,\n                 \"Rattus_norvegicus.Rnor_5.0.\" + ensembl_release),\n    \"rn6\": Build(\"rattus_norvegicus\", None,\n                 ucsc_ensembl_map_via_download,\n                 \"Rattus_norvegicus.Rnor_6.0.\" + ensembl_release),\n    \"hg38\": Build(\"homo_sapiens\", \"hsapiens_gene_ensembl\",\n                  manual_ucsc_ensembl_map,\n                  \"Homo_sapiens.GRCh38.\" + ensembl_release),\n    \"hg38-noalt\": Build(\"homo_sapiens\", \"hsapiens_gene_ensembl\",\n                        manual_ucsc_ensembl_map,\n                        \"Homo_sapiens.GRCh38.\" + ensembl_release),\n    \"canFam3\": Build(\"canis_familiaris\", None,\n                     ucsc_ensembl_map_via_download,\n                     \"Canis_familiaris.CanFam3.1.\" + ensembl_release),\n    \"sacCer3\": Build(\"saccharomyces_cerevisiae\", None,\n                     ucsc_ensembl_map_via_download,\n\n                     \"Saccharomyces_cerevisiae.R64-1-1.\" + ensembl_release),\n    \"WBcel235\": Build(\"caenorhabditis_elegans\", None,\n                      ucsc_ensembl_map_via_download,\n                      \"Caenorhabditis_elegans.WBcel235.\" + ensembl_release),\n    \"dm3\": Build(\"drosophila_melanogaster\", None,\n                 ucsc_ensembl_map_via_download,\n                 \"Drosophila_melanogaster.BDGP5.\" + ensembl_release),\n    \"Zv9\": Build(\"danio_rerio\", None,\n                 ucsc_ensembl_map_via_download,\n                 \"Danio_rerio.Zv9.\" + ensembl_release),\n    \"GRCz11\": Build(\"danio_rerio\", None, None,\n                    \"Danio_rerio.GRCz11.\" + ensembl_release),\n    \"xenTro3\": Build(\"xenopus_tropicalis\", None,\n                     ucsc_ensembl_map_via_download,\n                     \"Xenopus_tropicalis.JGI_4.2.\" + ensembl_release),\n    \"Sscrofa11.1\": Build(\"sus_scrofa\", None, None,\n                         \"Sus_scrofa.Sscrofa11.1.\" + ensembl_release),\n}\n\n\ndef parse_sequence_dict(fasta_dict):\n    def _tuples_from_line(line):\n        attrs = {}\n        for tag, val in [x.split(\":\", 1) for x in line.strip().split(\"\\t\")[1:]]:\n            attrs[tag] = val\n        return attrs[\"SN\"], attrs[\"LN\"], attrs[\"M5\"]\n    out = {}\n    with open(fasta_dict) as dict_handle:\n        for name, length, md5 in [_tuples_from_line(x) for x in dict_handle if x.startswith(\"@SQ\")]:\n            out[md5] = name\n    return out\n\nclass SequenceDictParser(object):\n\n    def __init__(self, fname):\n        self.fname = fname\n\n    def _get_sequences_in_genome_dict(self):\n        with open(self.fname) as genome_handle:\n            sequences = [self._sequence_from_line(x) for x in genome_handle if \"@SQ\" in x]\n        return sequences\n\n    def _sequence_from_line(self, line):\n        name = line.split(\"\\t\")[1].split(\":\")[1]\n        md5 = line.split(\"\\t\")[4].split(\":\")[1]\n        return md5, name\n\ndef get_ensembl_dict(org_build):\n    genome_dict = org_build + \".dict\"\n    if not os.path.exists(genome_dict):\n        org_fa = org_build + \".fa.gz\"\n        if not os.path.exists(org_fa):\n            genome = _download_ensembl_genome(org_build)\n            shutil.move(genome, org_fa)\n        genome_dict = make_fasta_dict(org_fa)\n    return genome_dict\n\ndef get_ucsc_dict(org_build):\n    fa_dict = os.path.join(os.getcwd(), os.pardir, \"seq\", org_build + \".dict\")\n    if not file_exists(fa_dict):\n        fa_file = os.path.splitext(fa_dict)[0] + \".fa\"\n        fa_dict = make_fasta_dict(fa_file)\n    return fa_dict\n\ndef make_fasta_dict(fasta_file):\n    dict_file = os.path.splitext(fasta_file.replace(\".fa.gz\", \".fa\"))[0] + \".dict\"\n    if not os.path.exists(dict_file):\n        subprocess.check_call(\"picard -Xms1g -Xmx3g CreateSequenceDictionary R={fasta_file} \"\n                              \"O={dict_file}\".format(**locals()), shell=True)\n    return dict_file\n\ndef _download_ensembl_genome(org_build):\n    build = build_info[org_build]\n    # reference files do not use the ensembl_release version so split it off\n    fname = os.path.splitext(build.fbase)[0] + \".dna_sm.toplevel.fa.gz\"\n    dl_url = (\"ftp://ftp.ensembl.org/pub/release-{release}/\"\n              \"fasta/{taxname}/dna/{fname}\").format(release=ensembl_release,\n                                                    taxname=build.taxname,\n                                                    fname=fname)\n    out_file = os.path.basename(dl_url)\n    if not os.path.exists(out_file):\n        subprocess.check_call([\"wget\", \"-c\", dl_url])\n    return out_file\n\ndef write_version(build=None, gtf_file=None, build_version=None):\n    gtf_file = build.fbase if build else gtf_file\n    gtf_file = os.path.abspath(gtf_file)\n    gtf_file = build_version if build_version else gtf_file\n    version_file = \"version.txt\"\n    with open(version_file, \"w\") as out_handle:\n        out_handle.write(\"Created from: %s\" % gtf_file)\n    return version_file\n\n# ## Main driver functions\n\ndef main(org_build, gtf_file, genome_fasta, genome_dir, cores, args):\n    genome_dir = genome_dir if genome_dir else os.curdir\n    build_dir = os.path.abspath(os.path.join(genome_dir, org_build))\n    work_dir = os.path.join(build_dir, \"tmpcbl\")\n    safe_makedir(work_dir)\n    ens_version = supported_oldbuilds.get(org_build, ensembl_release)\n    out_dir = os.path.join(build_dir,\n                           \"rnaseq-%s_%s\" % (datetime.datetime.now().strftime(\"%Y-%m-%d\"), ens_version))\n    tophat_dir = os.path.join(out_dir, \"tophat\")\n    gtf_file = os.path.abspath(gtf_file) if gtf_file else gtf_file\n\n    if genome_fasta:\n        genome_fasta = os.path.abspath(genome_fasta)\n        work_fasta = os.path.join(work_dir, os.path.basename(genome_fasta))\n        if not os.path.exists(work_fasta):\n            shutil.copy(genome_fasta, work_fasta)\n        genome_fasta = work_fasta\n\n    with chdir(work_dir):\n        if not genome_fasta:\n            genome_fasta = get_genome_fasta(org_build)\n        if not gtf_file:\n            write_version(build=build_info[org_build])\n            build = build_info[org_build]\n            gtf_file = prepare_tx_gff(build, org_build)\n        else:\n            write_version(gtf_file=gtf_file, build_version=args.buildversion)\n            work_gtf = os.path.join(work_dir, \"ref-transcripts.gtf\")\n            if not os.path.exists(work_gtf):\n                shutil.copy(gtf_file, work_gtf)\n            gtf_file = work_gtf\n        gtf_file = clean_gtf(gtf_file, genome_fasta)\n        db = _get_gtf_db(gtf_file)\n        os.remove(gtf_file)\n        gtf_file = db_to_gtf(db, gtf_file)\n        gtf_to_refflat(gtf_file)\n        gtf_to_bed(gtf_file)\n        prepare_tx2gene(gtf_file)\n        prepare_dexseq(gtf_file)\n        mask_gff = prepare_mask_gtf(gtf_file)\n        rrna_gtf = prepare_rrna_gtf(gtf_file)\n        if file_exists(rrna_gtf):\n            gtf_to_interval(rrna_gtf, genome_fasta)\n        if args.tophat:\n            prepare_tophat_index(gtf_file, org_build, genome_fasta)\n        transcriptome_fasta = make_transcriptome_fasta(gtf_file, genome_fasta)\n        if args.kallisto:\n            prepare_kallisto_index(transcriptome_fasta, org_build)\n        make_hisat2_splicesites(gtf_file)\n        cleanup(work_dir, out_dir, org_build)\n        rnaseq_dir = os.path.join(build_dir, \"rnaseq\")\n        if os.path.exists(rnaseq_dir):\n            if os.path.islink(rnaseq_dir):\n                os.unlink(rnaseq_dir)\n            else:\n                shutil.rmtree(rnaseq_dir)\n        os.symlink(out_dir, rnaseq_dir)\n\n    tar_dirs = [os.path.relpath(out_dir)]\n    tarball = create_tarball(tar_dirs, org_build)\n\ndef make_hisat2_splicesites(gtf_file):\n    base, _ = os.path.splitext(gtf_file)\n    out_file = os.path.join(base + \"-splicesites.txt\")\n    executable = get_program_python(\"hisat2\")\n    hisat2_script = os.path.join(os.path.dirname(executable),\n                                 \"hisat2_extract_splice_sites.py\")\n    cmd = \"{executable} {hisat2_script} {gtf_file} > {out_file}\"\n    if file_exists(out_file):\n        return out_file\n    if not file_exists(hisat2_script):\n        return None\n    subprocess.check_call(cmd.format(**locals()), shell=True)\n    return out_file\n\ndef make_transcriptome_fasta(gtf_file, genome_fasta):\n    base, _ = os.path.splitext(gtf_file)\n    out_file = os.path.join(base + \".fa\")\n    out_file = gtf_to_fasta(gtf_file, genome_fasta, out_file=out_file)\n    return out_file\n\ndef clean_gtf(gtf_file, genome_fasta):\n    \"\"\"\n    remove transcripts that have the following properties\n    1) don't have a corresponding ID in the reference\n    2) gencode Selenocysteine features which break many downstream tools\n    3) are not associated with a gene (no gene_id field)\n    \"\"\"\n    temp_gtf = tempfile.NamedTemporaryFile(suffix=\".gtf\").name\n    fa_names = get_fasta_names(genome_fasta)\n    with open(gtf_file) as in_gtf, open(temp_gtf, \"w\") as out_gtf:\n        for line in in_gtf:\n            if line.startswith(\"#\"):\n                continue\n            # these cause problems with downstream tools and we don't use them\n            if \"Selenocysteine\" in line:\n                continue\n            if line.split()[0].strip() not in fa_names:\n                continue\n            if 'gene_id' not in line:\n                continue\n            out_gtf.write(line)\n    # shutil.move breaks on some clusters when /tmp and target dir are on different filesystems\n    shutil.copy(temp_gtf, gtf_file)\n    os.remove(temp_gtf)\n    return gtf_file\n\ndef get_genome_fasta(org_build):\n    fa_path = os.path.abspath(os.path.join(os.curdir, os.pardir, \"seq\",\n                                           org_build + \".fa\"))\n    return fa_path\n\ndef get_fasta_names(genome_fasta):\n    fa_dict = genome_fasta + \".fai\"\n    if not os.path.exists(fa_dict):\n        subprocess.check_call(\"samtools faidx %s\" % genome_fasta, shell=True)\n    with open(fa_dict) as in_handle:\n        return [line.split(\"\\t\")[0] for line in in_handle]\n\ndef cleanup(work_dir, out_dir, org_build):\n    for fname in [os.path.join(work_dir, org_build + \".dict\"),\n                  os.path.join(work_dir, org_build + \".fa\"),\n                  os.path.join(work_dir, org_build + \".fa.gz\"),\n                  os.path.join(work_dir, org_build + \"-map.csv\")]:\n        if os.path.exists(fname):\n            os.remove(fname)\n    if os.path.exists(os.path.join(work_dir, \"bcbiotx\")):\n        shutil.rmtree(os.path.join(work_dir, \"bcbiotx\"))\n    shutil.move(work_dir, out_dir)\n\ndef create_tarball(tar_dirs, org_build):\n    str_tar_dirs = \" \".join(tar_dirs)\n    tarball = \"{org}-{dir}.tar.xz\".format(org=org_build, dir=os.path.basename(tar_dirs[0]))\n    if not os.path.exists(tarball):\n        subprocess.check_call(\"tar -cvpf - {out_dir} | xz -zc - > {tarball}\".format(\n            out_dir=str_tar_dirs, tarball=tarball), shell=True)\n    return tarball\n\ndef upload_to_s3(tarball):\n    upload_script = os.path.join(os.path.dirname(__file__), \"s3_multipart_upload.py\")\n    subprocess.check_call([sys.executable, upload_script, tarball, \"biodata\",\n                           os.path.join(\"annotation\", os.path.basename(tarball)),\n                           \"--public\"])\n\ndef genepred_to_UCSC_table(genepred):\n    header = [\"#bin\", \"name\", \"chrom\", \"strand\",\n              \"txStart\", \"txEnd\", \"cdsStart\", \"cdsEnd\",\n              \"exonCount\", \"exonStarts\", \"exonEnds\", \"score\",\n              \"name2\", \"cdsStartStat\", \"cdsEndStat\",\n              \"exonFrames\"]\n    out_file = os.path.splitext(genepred)[0] + \".UCSCTable\"\n    if file_exists(out_file):\n        return out_file\n    with open(genepred) as in_handle, open(out_file, \"w\") as out_handle:\n        counter = -1\n        current_item = None\n        out_handle.write(\"\\t\".join(header) + \"\\n\")\n        for l in in_handle:\n            item = l.split(\"\\t\")[0]\n            if current_item != item:\n                current_item = item\n                counter = counter + 1\n            out_handle.write(\"\\t\".join([str(counter), l]))\n    return out_file\n\ndef gtf_to_genepred(gtf):\n    out_file = os.path.splitext(gtf)[0] + \".genePred\"\n    if file_exists(out_file):\n        return out_file\n    cmd = \"gtfToGenePred -allErrors -ignoreGroupsWithoutExons -genePredExt {gtf} {out_file}\"\n    subprocess.check_call(cmd.format(**locals()), shell=True)\n    return out_file\n\ndef gtf_to_refflat(gtf):\n    out_file = os.path.splitext(gtf)[0] + \".refFlat\"\n    if file_exists(out_file):\n        return out_file\n\n    genepred = gtf_to_genepred(gtf)\n    with open(genepred) as in_handle, open(out_file, \"w\") as out_handle:\n        for l in in_handle:\n            first = l.split(\"\\t\")[0]\n            out_handle.write(\"\\t\".join([first, l]))\n\n    return out_file\n\ndef gtf_to_bed(gtf):\n    db = _get_gtf_db(gtf)\n    out_file = os.path.splitext(gtf)[0] + \".bed\"\n    if file_exists(out_file):\n        return out_file\n    with open(out_file, \"w\") as out_handle:\n        for feature in db.features_of_type('transcript'):\n            chrom = feature.chrom\n            start = feature.start\n            end = feature.end\n            attributes = feature.attributes.keys()\n            strand = feature.strand\n            name = (feature['gene_name'][0] if 'gene_name' in attributes else\n                    feature['gene_id'][0])\n            line = \"\\t\".join(map(str, [chrom, start, end, name, \".\", strand]))\n            out_handle.write(line + \"\\n\")\n    return out_file\n\ndef _is_selenocysteine(feature):\n    if feature.featuretype == \"Selenocysteine\":\n        return True\n    return False\n\ndef db_to_gtf(db, out_file):\n    if file_exists(out_file):\n        return out_file\n    print(\"Writing out merged GTF file to %s.\" % out_file)\n    with open(out_file, \"w\") as out_handle:\n        for feature in db.all_features():\n            if _is_selenocysteine(feature):\n                continue\n            out_handle.write(str(feature) + \"\\n\")\n    return out_file\n\ndef make_miso_events(gtf, org_build):\n    genepred = gtf_to_genepred(gtf)\n    genepred = genepred_to_UCSC_table(genepred)\n    pred_dir = tempfile.mkdtemp()\n    miso_dir = os.path.join(os.path.dirname(gtf), \"miso\")\n    tmp_pred = os.path.join(pred_dir, \"ensGene.txt\")\n    os.symlink(os.path.abspath(genepred), tmp_pred)\n    make_miso_annotation(pred_dir, miso_dir, org_build)\n\n    gff_files = glob.glob(os.path.join(miso_dir, \"commonshortest\", \"*.gff3\"))\n\n    cmd = \"index_gff --index {f} {prefix}\"\n\n    for f in gff_files:\n        prefix = f.split(\".\")[0] + \"_indexed\"\n        if not file_exists(prefix):\n            subprocess.check_call(cmd.format(**locals()), shell=True)\n\ndef prepare_bowtie_index(genome_fasta, bowtie_dir):\n    if os.path.exists(bowtie_dir + \".1.bt2\"):\n        return bowtie_dir\n    safe_makedir(bowtie_dir)\n    cmd = \"bowtie2-build {genome_fasta} {bowtie_dir}\"\n    subprocess.check_call(cmd.format(**locals()), shell=True)\n    return bowtie_dir\n\ndef prepare_tophat_index(gtf, org_build, genome_fasta):\n    tophat_dir = os.path.abspath(os.path.join(os.path.dirname(gtf), \"tophat\",\n                                              org_build + \"_transcriptome\"))\n    bowtie_dir = os.path.abspath(os.path.join(os.path.dirname(gtf),\n                                              os.path.pardir, \"bowtie2\",\n                                              org_build))\n    bowtie_dir = prepare_bowtie_index(genome_fasta, bowtie_dir)\n    out_dir = tempfile.mkdtemp()\n    fastq = _create_dummy_fastq()\n    cmd = (\"tophat --transcriptome-index {tophat_dir} -G {gtf} \"\n           \"-o {out_dir} {bowtie_dir} {fastq}\")\n    subprocess.check_call(cmd.format(**locals()), shell=True)\n    shutil.rmtree(out_dir)\n    os.remove(fastq)\n\ndef prepare_kallisto_index(transcriptome_fasta, org_build):\n    kallisto = which(\"kallisto\")\n    if not kallisto:\n        return None\n    base_dir = os.path.abspath(os.path.dirname(transcriptome_fasta))\n    kallisto_dir = os.path.join(base_dir, \"kallisto\")\n    safe_makedir(kallisto_dir)\n    kallisto_index = os.path.join(kallisto_dir, org_build)\n    if not os.path.exists(kallisto_index):\n        cmd = (\"kallisto index -i {kallisto_index} {transcriptome_fasta}\")\n        subprocess.check_call(cmd.format(**locals()), shell=True)\n    return kallisto_index\n\ndef prepare_sailfish_index(transcriptome_fasta, org_build):\n    sailfish = which(\"sailfish\")\n    if not sailfish:\n        return None\n    base_dir = os.path.abspath(os.path.dirname(transcriptome_fasta))\n    sailfish_dir = os.path.join(base_dir, \"sailfish\")\n    safe_makedir(sailfish_dir)\n    sailfish_index = os.path.join(sailfish_dir, org_build)\n    cmd = (\"sailfish index -t {sailfish_index} -o {sailfish_index}\")\n    subprocess.check_call(cmd.format(**locals()), shell=True)\n    return sailfish_index\n\ndef _create_dummy_fastq():\n    read = (\"@HWI-ST333_0178_FC:5:1101:1107:2112#ATCTCG/1\\n\"\n            \"GGNCTTTCCTGCTTCTATGTCTTGATCGCCTGTAGGCAGG\\n\"\n            \"+HWI-ST333_0178_FC:5:1101:1107:2112#ATCTCG/1\\n\"\n           \"[[BS\\\\a`ceeagfhhhhhaefhcdfhcf`efeg[cg_b__\\n\")\n    fn = \"dummy.fq\"\n    with open(fn, \"w\") as out_handle:\n        out_handle.write(read)\n    return fn\n\ndef gtf_to_interval(gtf, genome_fasta):\n    fa_dict = make_fasta_dict(genome_fasta)\n    db = _get_gtf_db(gtf)\n    out_file = os.path.splitext(gtf)[0] + \".interval_list\"\n    if file_exists(out_file):\n        return out_file\n\n    with open(out_file, \"w\") as out_handle:\n        with open(fa_dict) as in_handle:\n            for l in in_handle:\n                out_handle.write(l)\n\n        for l in db.all_features():\n            out_handle.write(\"\\t\".join([str(l.seqid), str(l.start),\n                                        str(l.end), str(l.strand),\n                                        str(l.attributes.get(\"transcript_id\",\n                                                             [\".\"])[0])]) + \"\\n\")\n    return out_file\n\ndef prepare_mask_gtf(gtf):\n    \"\"\"\n    make a mask file of usually-masked RNA biotypes\n    \"\"\"\n\n    mask_biotype = [\"rRNA\", \"Mt_rRNA\", \"misc_RNA\", \"snRNA\", \"snoRNA\",\n                    \"tRNA\", \"Mt_tRNA\"]\n    mask_chrom = [\"MT\"]\n    out_file = os.path.join(os.path.dirname(gtf), \"ref-transcripts-mask.gtf\")\n    if file_exists(out_file):\n        return out_file\n    biotype_lookup = _biotype_lookup_fn(gtf)\n    # if we can't find a biotype column, skip this\n    if not biotype_lookup:\n        return None\n    db = _get_gtf_db(gtf)\n    with open(out_file, \"w\") as out_handle:\n        for g in db.all_features():\n            biotype = biotype_lookup(g)\n            if (biotype in mask_biotype) or (g.chrom in mask_chrom):\n                out_handle.write(str(g) + \"\\n\")\n    return out_file\n\ndef prepare_rrna_gtf(gtf):\n    \"\"\"\n    extract out just the rRNA biotypes, for assessing rRNA contamination\n    \"\"\"\n    mask_biotype = [\"rRNA\", \"Mt_rRNA\", \"tRNA\", \"MT_tRNA\"]\n\n    out_file = os.path.join(os.path.dirname(gtf), \"rRNA.gtf\")\n    if os.path.exists(out_file):\n        return out_file\n\n    db = _get_gtf_db(gtf)\n    biotype_lookup = _biotype_lookup_fn(gtf)\n    # if we can't find a biotype column, skip this\n    if not biotype_lookup:\n        return None\n    with open(out_file, \"w\") as out_handle:\n        for feature in db.all_features():\n            biotype = biotype_lookup(feature)\n            if biotype in mask_biotype:\n                out_handle.write(str(feature) + \"\\n\")\n    return out_file\n\ndef prepare_tx2gene(gtf):\n    \"\"\"\n    prepare a file mapping transcripts to genes\n    \"\"\"\n    db = _get_gtf_db(gtf)\n    out_file = os.path.join(os.path.dirname(gtf), \"tx2gene.csv\")\n    if file_exists(out_file):\n        return out_file\n    with open(out_file, \"w\") as out_handle:\n        for transcript in db.features_of_type('transcript'):\n            gene_id = transcript['gene_id'][0]\n            transcript_id = transcript['transcript_id'][0]\n            out_handle.write(\",\".join([transcript_id, gene_id]) + \"\\n\")\n    return out_file\n\ndef _biotype_lookup_fn(gtf):\n    \"\"\"\n    return a function that will look up the biotype of a feature\n    this checks for either gene_biotype or biotype being set or for the source\n    column to have biotype information\n    \"\"\"\n    db = _get_gtf_db(gtf)\n    sources = set([feature.source for feature in db.all_features()])\n    gene_biotypes = set([feature.attributes.get(\"gene_biotype\", [None])[0]\n                         for feature in db.all_features()])\n    biotypes = set([feature.attributes.get(\"biotype\", [None])[0]\n                    for feature in db.all_features()])\n    if \"protein_coding\" in sources:\n        return lambda feature: feature.source\n    elif \"protein_coding\" in biotypes:\n        return lambda feature: feature.attributes.get(\"biotype\", [None])[0]\n    elif \"protein_coding\" in gene_biotypes:\n        return lambda feature: feature.attributes.get(\"gene_biotype\", [None])[0]\n    else:\n        return None\n\ndef prepare_tx_gff(build, org_name):\n    \"\"\"Prepare UCSC ready transcript file given build information.\n    \"\"\"\n    ensembl_gff = _download_ensembl_gff(build, org_name)\n    # if we need to do the name remapping\n    if build.ucsc_map:\n        ucsc_name_map = build.ucsc_map(org_name)\n        tx_gff = _remap_gff(ensembl_gff, ucsc_name_map)\n        os.remove(ensembl_gff)\n    else:\n        tx_gff = \"ref-transcripts.gtf\"\n        os.rename(ensembl_gff, tx_gff)\n    return tx_gff\n\ndef _remap_gff(base_gff, name_map):\n    \"\"\"Remap chromosome names to UCSC instead of Ensembl\n    \"\"\"\n    out_file = \"ref-transcripts.gtf\"\n    wrote_missing = set([])\n    if not os.path.exists(out_file):\n        with open(out_file, \"w\") as out_handle, \\\n             open(base_gff) as in_handle:\n            for line in in_handle:\n                parts = line.split(\"\\t\")\n                ucsc_name = name_map.get(parts[0], None)\n                if ucsc_name:\n                    out_handle.write(\"\\t\".join([ucsc_name] + parts[1:]))\n                elif parts[0] not in wrote_missing and not line.startswith(\"#\"):\n                    print(\"Missing\", parts[0])\n                    wrote_missing.add(parts[0])\n    return out_file\n\ndef _download_ensembl_gff(build, org_name):\n    \"\"\"Given build details, download and extract the relevant ensembl GFF.\n    \"\"\"\n    fname = build.fbase + \".gtf.gz\"\n    dl_url = \"/\".join([base_ftp, build.taxname, fname]).format(\n        release=supported_oldbuilds.get(org_name, ensembl_release))\n    out_file = os.path.splitext(os.path.basename(dl_url))[0]\n    if not os.path.exists(out_file):\n        subprocess.check_call([\"wget\", dl_url])\n        subprocess.check_call([\"gunzip\", os.path.basename(dl_url)])\n    return out_file\n\ndef _create_tiny_gffutils_db(gtf_file):\n    _, ext = os.path.splitext(gtf_file)\n    tmp_out = tempfile.NamedTemporaryFile(suffix=\".gtf\", delete=False).name\n    with open(tmp_out, \"w\") as out_handle:\n        count = 0\n        in_handle = open(gtf_file) if ext != \".gz\" else gzip.open(gtf_file)\n        for line in in_handle:\n            if count > 1000:\n                break\n            out_handle.write(line)\n            count += 1\n        in_handle.close()\n    db = gffutils.create_db(tmp_out, dbfn=\":memory:\",\n                            disable_infer_genes=True,\n                            disable_infer_transcripts=True,\n                            merge_strategy=\"warning\")\n    os.remove(tmp_out)\n    return db\n\n\ndef subfeature_handler(f):\n    \"\"\"\n    Given a gffutils.Feature object (which does not yet have its ID assigned),\n    figure out what its ID should be.\n    This is intended to be used for CDS, UTR, start_codon, and stop_codon\n    features in the Ensembl release 81 GTF files.  I figured a reasonable\n    unique ID would consist of the parent transcript and the feature type,\n    followed by an autoincrementing number.\n    See https://pythonhosted.org/gffutils/database-ids.html#id-spec for\n    details and other options.\n    Grabbed from Ryan Dale: https://www.biostars.org/p/152517/\n    \"\"\"\n    return ''.join(\n        ['autoincrement:',\n         f.attributes['transcript_id'][0],\n         '_',\n         f.featuretype])\n\ndef guess_disable_infer_extent(gtf_file):\n    \"\"\"\n    guess if we need to use disable the infer gene or transcript extent option\n    when making a gffutils database by making a tiny database of 1000 lines\n    from the original GTF and looking for all of the features\n    \"\"\"\n    db = _create_tiny_gffutils_db(gtf_file)\n    features = [x for x in db.featuretypes()]\n    disable_infer_transcript = \"transcript\" in features\n    disable_infer_gene = \"gene\" in features\n    return disable_infer_transcript, disable_infer_gene\n\ndef guess_id_spec(gtf_file):\n    \"\"\"\n    guess at the id spec in a GTF file by examining the first 1000 lines\n    assigns unique ids to features that may not have them\n    \"\"\"\n    db = _create_tiny_gffutils_db(gtf_file)\n    id_spec = {}\n    attributes = set()\n    for f in db.all_features():\n        attributes.update(f.attributes)\n    if \"gene_id\" in attributes:\n        id_spec[\"gene\"] = \"gene_id\"\n        attributes.remove(\"gene_id\")\n    if \"transcript_id\" in attributes:\n        id_spec[\"transcript\"] = \"transcript_id\"\n        attributes.remove(\"transcript_id\")\n    return id_spec\n\ndef _get_gtf_db(gtf):\n    db_file = gtf + \".db\"\n    if not file_exists(db_file):\n        print(\"Creating gffutils database for %s.\" % (gtf))\n        disable_infer_transcripts, disable_infer_genes = guess_disable_infer_extent(gtf)\n        if not disable_infer_transcripts or not disable_infer_genes:\n            print(\"'transcript' or 'gene' entries not found, so inferring \"\n                  \"their extent. This can be very slow.\")\n        id_spec = guess_id_spec(gtf)\n        gffutils.create_db(gtf, dbfn=db_file,\n                           disable_infer_genes=disable_infer_genes,\n                           disable_infer_transcripts=disable_infer_transcripts,\n                           id_spec=id_spec,\n                           merge_strategy=\"create_unique\",\n                           keep_order=True,\n                           verbose=True)\n    return gffutils.FeatureDB(db_file)\n\ndef _dexseq_preparation_path():\n    PREP_FILE = \"python_scripts/dexseq_prepare_annotation.py\"\n    try:\n        cmd = \"%s/Rscript -e 'find.package(\\\"DEXSeq\\\")'\" % os.path.dirname(os.path.realpath(sys.executable))\n        output = subprocess.check_output(cmd, shell=True)\n    except subprocess.CalledProcessError:\n        return None\n    for line in output.decode().split(\"\\n\"):\n        if line.startswith(\"[\"):\n            dirname = line.split(\"[1]\")[1].replace(\"\\\"\", \"\").strip()\n            path = os.path.join(dirname, PREP_FILE)\n            if os.path.exists(path):\n                return path\n    return None\n\ndef prepare_dexseq(gtf):\n    out_file = os.path.splitext(gtf)[0] + \".dexseq.gff3\"\n    if file_exists(out_file):\n        return out_file\n\n    dexseq_path = _dexseq_preparation_path()\n    if not dexseq_path:\n        return None\n    executable = get_program_python(\"htseq-count\")\n    cmd = \"{executable} {dexseq_path} {gtf} {out_file}\"\n    subprocess.check_call(cmd.format(**locals()), shell=True)\n    return out_file\n\nif __name__ == \"__main__\":\n    parser = ArgumentParser(description=\"Prepare the transcriptome files for an \"\n                            \"organism.\")\n    parser.add_argument(\"-c\", \"--cores\", default=1,\n                        help=\"number of cores to use\")\n    parser.add_argument(\"--gtf\",\n                        help=\"Optional GTF file (instead of downloading from Ensembl)\",\n                        default=None),\n    parser.add_argument(\"--fasta\",\n                        help=\"Optional genomic FASTA file (instead of downloading from Ensembl)\",\n                        default=None),\n    parser.add_argument(\"--genome-dir\",\n                        help=(\"Optional location of the root genome directory. \"\n                              \"For example --genome-dir=/foo will install the files \"\n                              \"for a Hsapiens hg19 genome to /foo/Hsapiens/hg19.\"))\n    parser.add_argument(\"--tophat\", help=\"Build TopHat indices\",\n                        default=False, action=\"store_true\")\n    parser.add_argument(\"--kallisto\", help=\"Build Kallisto indices\",\n                        default=False, action=\"store_true\")\n    parser.add_argument(\"--buildversion\", help=(\"Store build information. String should be source_genomebuild.\" \n                        \" Examples: Ensembl_94, EnsemblMetazoa_94, FlyBase_23, etc\"),\n                        default=None)\n    parser.add_argument(\"organism\", help=\"Short name of organism (for example Hsapiens)\")\n    parser.add_argument(\"org_build\", help=\"Build of organism to run.\")\n    args = parser.parse_args()\n    if args.genome_dir:\n        genome_dir = os.path.join(args.genome_dir, args.organism)\n    else:\n        genome_dir = os.curdir\n    main(args.org_build, args.gtf, args.fasta, genome_dir, args.cores, args)\n"
  },
  {
    "path": "utils/prioritize/AZ300.txt",
    "content": "ABL1\nACVR1B\nACVR2A\nACVRL1\nAGTR2\nAJUBA\nAKT1\nAKT1S1\nAKT2\nAKT3\nALK\nAMER1\nAPC\nAR\nARAF\nARHGAP35\nARID1A\nARID1B\nARID2\nARID5B\nASXL1\nATM\nATR\nATRX\nAXIN2\nB4GALT3\nBAP1\nBCL2L1\nBCL9\nBRAF\nBRCA1\nBRCA2\nBRD2\nBRD3\nBRD4\nBRD9\nBRIP1\nCBFB\nCBL\nCCND1\nCCND2\nCCND3\nCCNE1\nCDH1\nCDK12\nCDK4\nCDK6\nCDK8\nCDK9\nCDKN1A\nCDKN1B\nCDKN2A\nCDKN2B\nCDKN2C\nCEBPA\nCHEK1\nCHEK2\nCRIPAK\nCRKL\nCSNK2A1\nCTCF\nCTNNB1\nCYP2D6\nDCUN1D1\nDDR1\nDDR2\nDEPTOR\nDNMT3A\nEGFR\nEGR3\nEIF4A2\nELF3\nEP300\nEPHA3\nEPHB6\nEPPK1\nERAS\nERBB2\nERBB3\nERBB4\nERCC1\nERCC2\nERRFI1\nESR1\nEZH2\nFAM175A\nFBXW7\nFGF1\nFGF10\nFGF12\nFGF14\nFGF19\nFGF2\nFGF23\nFGF3\nFGF4\nFGF5\nFGF6\nFGF7\nFGF8\nFGF9\nFGFR1\nFGFR2\nFGFR3\nFGFR4\nFKBP1A\nFLT3\nFOXA1\nFOXA2\nFOXC1\nFOXL2\nFRS2\nGAS6\nGATA3\nGNA11\nGNA13\nGNAQ\nGNAS\nH3F3C\nHGF\nHIST1H1C\nHIST1H2BD\nHNF1A\nHRAS\nIDH1\nIDH2\nIGF1R\nIL6\nINPP4A\nINPP4B\nIRAK4\nJAK1\nJAK2\nJAK3\nKDM5C\nKDM6A\nKDR\nKEAP1\nKIT\nKMT2B\nKMT2C\nKMT2D\nKRAS\nLAMP1\nLIFR\nLRRK2\nMALAT1\nMAP2K1\nMAP2K2\nMAP2K4\nMAP3K1\nMAP3K13\nMAP3K8\nMAPK1\nMAPK3\nMAPK8IP1\nMAPKAP1\nMCL1\nMDM2\nMDM4\nMECOM\nMED12\nMET\nMGA\nMIR142\nMITF\nMLH1\nMLH3\nMLST8\nMPL\nMRAS\nMSH2\nMSH3\nMSH6\nMTOR\nMUTYH\nMYC\nMYCL\nMYCN\nMYH9\nNAV3\nNCOR1\nNCOR2\nNF1\nNF2\nNFE2L2\nNFE2L3\nNKX2-1\nNKX2-8\nNOTCH1\nNPM1\nNRAS\nNRG1\nNSD1\nPAK1\nPALB2\nPBRM1\nPCBP1\nPDGFRA\nPDPK1\nPHF6\nPHLPP1\nPHLPP2\nPIK3C2B\nPIK3C2G\nPIK3C3\nPIK3CA\nPIK3CB\nPIK3CD\nPIK3CG\nPIK3R1\nPIK3R2\nPIM1\nPIM2\nPIM3\nPMS1\nPMS2\nPOLD1\nPOLE\nPOLQ\nPPP2R1A\nPPP2R2A\nPRKAA1\nPRKAA2\nPRKAB1\nPRX\nPTCH1\nPTEN\nPTPN11\nPTPN6\nRAC1\nRAD21\nRAD51B\nRAD51C\nRAD51D\nRAF1\nRASA1\nRB1\nRBM10\nRELN\nRET\nRHEB\nRHOA\nRICTOR\nRIT1\nROS1\nRPL22\nRPL5\nRPS6\nRPS6KB1\nRPS6KB2\nRPTOR\nRUNX1\nSETBP1\nSETD2\nSF3B1\nSGK1\nSGK2\nSGK3\nSHOC2\nSIN3A\nSMAD2\nSMAD4\nSMARCA4\nSMARCB1\nSMC1A\nSMC3\nSMO\nSOS1\nSOX17\nSOX2\nSOX9\nSPOP\nSPRED1\nSRY\nSTAG2\nSTAT3\nSTK11\nTAF1\nTBL1XR1\nTBX3\nTERT\nTET2\nTFRC\nTGFBR2\nTIAF1\nTLR4\nTOP2A\nTP53\nTSC1\nTSC2\nTSHZ2\nTSHZ3\nU2AF1\nUSP9X\nVEZF1\nVHL\nWEE1\nWT1\nZNF217\n"
  },
  {
    "path": "utils/prioritize/AZ300_with_known.txt",
    "content": "ABL1\nABL2\nAC006372.4\nAC016582.2\nACER3\nACVR2A\nADAMTS19\nADAMTSL2\nADCY3\nADRBK2\nAES\nAF1Q\nAFF1\nAFF3\nAGGF1\nAGK\nAGMO\nAGTRAP\nAHCYL1\nAKAP12\nAKAP13\nAKAP6\nAKAP8L\nAKAP9\nAKT3\nALK\nALO17\nALOXE3\nAML1\nANK1\nANKRD11\nANKRD28\nANXA4\nAP3B1\nARHGAP30\nARHH\nARL6IP4\nARNT\nART3\nASIC2\nASXL1\nATAD1\nATE1\nATF1\nATF7IP\nATG7\nATIC\nATP1B3\nATP8A2\nATRNL1\nATXN7\nB4GALT2\nBAG4\nBAIAP2L1\nBAZ1A\nBAZ2A\nBBS9\nBCAM\nBCAS3\nBCL2\nBCL2L11\nBCL3\nBCL5\nBCL6\nBCL7A\nBCL8\nBCL9\nBCR\nBEND5\nBFSP2\nBICC1\nBIRC6\nBMP8B\nBMPR1B\nBPIL3\nBRD3\nBRD4\nBRIP1\nBRK1\nBSG\nBTG1\nBUB1B\nC10ORF54\nC11ORF48\nC12ORF49\nC14ORF166\nC15ORF21\nC15ORF55\nC1ORF150\nC1S\nC20ORF112\nC2ORF22\nC2ORF44\nC4ORF28\nC6ORF211\nC6ORF97\nC8ORF34\nCACNB1\nCADM2\nCAMKK1\nCAMSAP1\nCAMTA1\nCAND1\nCAPZA2\nCARM1\nCARS\nCASC3\nCASC8\nCASP7\nCASP8\nCASZ1\nCBFA2T1\nCBFA2T2\nCBFA2T3\nCBFB\nCCAR2\nCCDC12\nCCDC127\nCCDC170\nCCDC50\nCCDC6\nCCDC88A\nCCNB1\nCCND1\nCCND2\nCCNF\nCD37\nCD4\nCD44\nCD74\nCD79B\nCD9\nCDC27\nCDC42EP1\nCDC6\nCDH15\nCDH17\nCDK4\nCDK5RAP2\nCDK6\nCDKAL1\nCDX2\nCEP1\nCEP110\nCEP170\nCEP55\nCEP85L\nCEP89\nCFB\nCHCHD7\nCHD1\nCHD7\nCHDC2\nCHEK2\nCHGA\nCHGB\nCHIC2\nCIRH1A\nCIT\nCLCA2\nCLCN4\nCLCN6\nCLIC5\nCLIP1\nCLTC\nCLTCL1\nCLU\nCMAS\nCNIH4\nCNTRL\nCOL14A1\nCOL17A1\nCOX4I2\nCPD\nCPLX2\nCPM\nCPNE8\nCPSF4L\nCPSF6\nCREB5\nCREBBP\nCRKL\nCSGALNACT2\nCSMD3\nCTB-151G24.1\nCTLC\nCTNNB1\nCTTN\nCUL1\nCUX1\nCXCL11\nD10S170\nDAB2\nDAZL\nDBR1\nDCC\nDCTN1\nDDR1\nDDX39B\nDDX42\nDDX5\nDEK\nDEM1\nDERL1\nDGKD\nDHX37\nDIAPH3\nDIP2C\nDNAH9\nDNAJC7\nDNMT1\nDNMT3B\nDOCK8\nDSTYK\nDYX1C1\nE2F7\nEBF1\nEFCAB3\nEFNA3\nEFTUD1\nEGFR\nEIF2AK1\nEIF4A2\nELAVL3\nELKS\nEML1\nEML4\nENO1\nEPHA5\nEPHB1\nEPOR\nERBB\nERC1\nERG\nERLIN2\nERP44\nERVK3-1\nESRP1\nETO\nETV1\nETV6\nEVI1\nEVX1\nEWS\nEWSR1\nEXOC6\nEXOSC2\nEZR\nFACL6\nFAM114A2\nFAM119B\nFAM120AOS\nFAM131B\nFAM133B\nFAM155A\nFAM230A\nFAM70B\nFAM84B\nFANCC\nFBXL20\nFBXO2\nFCHSD1\nFER1L6\nFGD4\nFGF11\nFGFR1\nFGFR1OP\nFGFR1OP1\nFGFR1OP2\nFGFR2\nFGFR3\nFIG\nFIP1L1\nFKBP10\nFKBP15\nFKBP9L\nFMO4\nFN1\nFNDC3B\nFOP\nFOXJ3\nFOXO1\nFOXP1\nFOXP2\nFRMD3\nFRMD4B\nFSIP1\nFSTL3\nFXR1\nGABBR2\nGAP43\nGASP\nGATAD1\nGATM\nGFAP\nGLTP\nGNAI1\nGNS\nGOLGA5\nGOPC\nGOSR1\nGPATCH8\nGPR124\nGRB7\nGRHL2\nGRIA4\nGRM1\nGTF2E2\nGTF2F2\nGTF2IRD1\nGXYLT1\nH4\nHACL1\nHDAC7\nHELQ\nHEPACAM2\nHERPUD1\nHERVK\nHIP1\nHIVEP3\nHJURP\nHLA-A\nHLXB9\nHM13\nHMCES\nHMGA2\nHMGN2P46\nHMHA1\nHOIP\nHOOK3\nHOX11\nHOXA9\nIFLTD1\nIFT20\nIFT80\nIGF2R\nIGFBP5\nIGH@\nIGHG1\nIGK@\nIGL@\nIKZF3\nIL1RAP\nIL27RA\nIL2RB\nINPP5D\nINSR\nIRF4\nIRTA1\nIRX2\nITM2B\nJAK2\nJAKMIP1\nKAZN\nKCNAB1\nKCNJ4\nKCNMA1\nKCNMB4\nKCTD1\nKCTD21\nKDELR2\nKDM4B\nKDM7A\nKIAA0573\nKIAA0833\nKIAA1432\nKIAA1468\nKIAA1549\nKIAA1549L\nKIAA1598\nKIAA1618\nKIAA1967\nKIF5A\nKIF5B\nKLC1\nKLC3\nKLHL22\nKMT2A\nKRT20\nKRT39\nKRT86\nKRTAP1-4\nKTN1\nLAF4\nLAMC1\nLANCL2\nLASP1\nLATS1\nLCLAT1\nLIFR\nLINC01153\nLMNA\nLMTK3\nLNP1\nLNX1\nLPCAT3\nLPXN\nLRIG3\nLRRC37B\nLRRFIP1\nLRRFIP2\nLSM14A\nLTBP4\nLYZ\nLYZL2\nMACF1\nMACROD1\nMAGI3\nMAN2B2\nMAP4K3\nMAPK1\nMAPK14\nMAPK8IP3\nMAPKBP1\nMARK3\nMARK4\nMAST2\nMAST3\nMBD1\nMBNL1\nMCCC1\nMCF2L\nMCM9\nMDM2\nMDS1\nMDS2\nMECOM\nMED1\nMETAP2\nMETTL12\nMETTL21B\nMFGE8\nMGEA5\nMICALCL\nMIDN\nMIEN1\nMIR143\nMIR143HG\nMKRN1\nMLF1\nMLL\nMLL3\nMLLT10\nMN1\nMOB3A\nMPG\nMPRIP\nMROH2B\nMRPL13\nMSH6\nMSL1\nMSN\nMTSS1\nMUC6\nMYB\nMYC\nMYH11\nMYH3\nMYH9\nMYO18A\nMYO18B\nMYO3B\nMYO5A\nMYO7A\nMYST3\nNAA30\nNARS2\nNCOA3\nNCOA4\nNEK4\nNFE2\nNFIB\nNKIRAS1\nNLE1\nNLK\nNLRP8\nNME9\nNOS2\nNOTCH3\nNPM\nNPM1\nNRF1\nNRG1\nNRIP1\nNRROS\nNSMCE4A\nNSUN2\nNTM\nNTRK1\nNUMA1\nNUP107\nNUP214\nNUP358\nNUP98\nNUT\nNUTM1\nODZ4\nOFD1\nORC2L\nOXR1\nPAK1\nPAK3\nPAK4\nPAN3\nPAPD7\nPAPSS1\nPARN\nPAX5\nPCCB\nPCDH11X\nPCM1\nPDE1C\nPDGFC\nPDGFD\nPDK3\nPDSS2\nPER1\nPGC\nPIGR\nPIK3CA\nPIM3\nPIP4K2B\nPIP5K2B\nPIP5KL1\nPITPNA\nPKM\nPLAG1\nPLAT\nPLD5\nPLEKHA6\nPLS1\nPOLD1\nPOLDIP2\nPOLR2J3\nPOU6F2\nPPAPDC1A\nPPARGC1A\nPPFIBP1\nPPHLN1\nPPM1H\nPPP1CA\nPPP1R1B\nPPP2R2A\nPPP2RA\nPPP4R1\nPPT1\nPRAME\nPRDM10\nPRDM12\nPRDM16\nPRDM7\nPRDX4\nPRKAG2\nPRKAR1A\nPRKCZ\nPRKD1\nPRODH\nPRR12\nPRRC1\nPSMB3\nPSMD11\nPSMD9\nPSPH\nPTC\nPTC3\nPTC7\nPTCH1\nPTK2B\nPTPN3\nPTPRZ\nPTPRZ1\nPVT1\nPVT1_1\nPWWP2A\nQKI\nRAB2\nRAB6IP2\nRAB8A\nRANBP2\nRARA\nRASA3\nRAX\nRB1CC1\nRBM14\nRBM35A\nRBM4\nRBPMS\nRCSD1\nRERE\nRERG\nRET\nRFX1\nRFX3\nRGPD2\nRGS17\nRHOT1\nRIMS3\nRLF\nRNF130\nRNF213\nRNF31\nRNPEP\nROCK1\nROS1\nRPL22\nRPL22P1\nRPN1\nRPRD1A\nRPS6KA1\nRPS6KA3\nRPS6KA6\nRPSA\nRUFY2\nRUNX1\nRUNX1T1\nRUNX2\nRUNXBP2\nSCAF11\nSCARB1\nSCO2\nSDC4\nSDF4\nSDK1\nSDK2\nSEC16A\nSEC31A\nSEC31L1\nSEC61G\nSEPT14\nSEPT9\nSFPQ\nSH3BP2\nSH3D19\nSHANK2\nSHBG\nSLC15A2\nSLC34A2\nSLC35A1\nSLC3A2\nSLC45A3\nSLC4A4\nSMAD3\nSMARCD2\nSMCHD1\nSMEK2\nSMG6\nSMIM18\nSND1\nSNF8\nSNHG7\nSNRPB\nSNX19\nSNX2\nSOX5\nSPAG9\nSPECC1L\nSPINT2\nSPTBN2\nSQSTM1\nSRCIN1\nSRGAP3\nSRRM5\nSSBP2\nSSR2\nSSU72\nST3GAL2\nSTIM1\nSTK4\nSTK40\nSTL\nSTRN\nSTRN3\nSTXBP6\nSUDS3\nSUMO1\nSUPT3H\nSV2B\nSYK\nTACC1\nTACC2\nTACC3\nTADA2A\nTAF1D\nTAF4\nTAL1\nTATDN1\nTAX1BP1\nTBC1D5\nTBL1XR1\nTCP1\nTCRB@\nTEL\nTENC1\nTERF2\nTERT\nTEX14\nTFE3\nTFEB\nTFG\nTHADA\nTHOC5\nTIAM2\nTIF1\nTLX3\nTMEM131\nTMEM165\nTMEM40\nTMEM49\nTMP3\nTMP4\nTOLLIP\nTOP3B\nTP53BP1\nTP63\nTPM1\nTPM3\nTPM4\nTPP2\nTPPP\nTPR\nTPRG1\nTRA@\nTRABD\nTRAF1\nTRAK1\nTRB@\nTRIM24\nTRIM27\nTRIM33\nTRIM4\nTRIO\nTRIP4\nTRMT61B\nTROP2\nTRPC6\nTRPS1\nTSC22D1\nTSFM\nTSPAN31\nTSPEAR\nTSPO\nTTBK2\nTTC13\nTTC18\nTTL\nTUBD1\nTULP4\nTWSG1\nTXLNA\nTYK2\nUBA5\nUBC\nUBE2L3\nUBR2\nUBR4\nULK4\nUSP13\nUSP24\nUSP42\nUSP6\nUSP8\nUTP23\nUTRN\nUTS2B\nUVRAG\nVAMP2\nVCL\nVKORC1\nVMP1\nVOPP1\nVSTM2A\nVTCN1\nVTI1A\nWDR11\nWDR82\nWDR93\nWHSC1\nWHSC1L1\nWRN\nWWTR1\nXBP1\nXIST\nXPC\nYAP1\nYIF1B\nYTHDF2\nZBTB5\nZC3H7B\nZC3HAV1\nZCCHC7\nZCCHC8\nZDHHC1\nZEB2\nZFC3H1\nZFPM2\nZMIZ1\nZMYM2\nZNF198\nZNF251\nZNF384\nZNF576\nZNF577\nZNF643\nZNF684\nZNF687\nZNF703\nZNF791\nZNF804A\nZSCAN30\n"
  },
  {
    "path": "utils/prioritize/az-cancer-panel.txt",
    "content": "ABL1\nACVR1B\nACVR2A\nACVRL1\nAGTR2\nAJUBA\nAKT1\nAKT1S1\nAKT2\nAKT3\nALK\nAMER1\nAPC\nAR\nARAF\nARHGAP35\nARID1A\nARID1B\nARID2\nARID5B\nASXL1\nATM\nATR\nATRX\nAXIN2\nB4GALT3\nBAP1\nBCL2L1\nBCL9\nBRAF\nBRCA1\nBRCA2\nBRD2\nBRD3\nBRD4\nBRD9\nBRIP1\nCBFB\nCBL\nCCND1\nCCND2\nCCND3\nCCNE1\nCDH1\nCDK12\nCDK4\nCDK6\nCDK8\nCDK9\nCDKN1A\nCDKN1B\nCDKN2A\nCDKN2B\nCDKN2C\nCEBPA\nCHEK1\nCHEK2\nCRIPAK\nCRKL\nCSNK2A1\nCTCF\nCTNNB1\nCYP2D6\nDCUN1D1\nDDR1\nDDR2\nDEPTOR\nDNMT3A\nEGFR\nEGR3\nEIF4A2\nELF3\nEP300\nEPHA3\nEPHB6\nEPPK1\nERAS\nERBB2\nERBB3\nERBB4\nERCC1\nERCC2\nERRFI1\nESR1\nEZH2\nFAM175A\nFBXW7\nFGF1\nFGF10\nFGF12\nFGF14\nFGF19\nFGF2\nFGF23\nFGF3\nFGF4\nFGF5\nFGF6\nFGF7\nFGF8\nFGF9\nFGFR1\nFGFR2\nFGFR3\nFGFR4\nFKBP1A\nFLT3\nFOXA1\nFOXA2\nFOXC1\nFOXL2\nFRS2\nGAS6\nGATA3\nGNA11\nGNA13\nGNAQ\nGNAS\nH3F3C\nHGF\nHIST1H1C\nHIST1H2BD\nHNF1A\nHRAS\nIDH1\nIDH2\nIGF1R\nIL6\nINPP4A\nINPP4B\nIRAK4\nJAK1\nJAK2\nJAK3\nKDM5C\nKDM6A\nKDR\nKEAP1\nKIT\nKMT2B\nKMT2C\nKMT2D\nKRAS\nLAMP1\nLIFR\nLRRK2\nMALAT1\nMAP2K1\nMAP2K2\nMAP2K4\nMAP3K1\nMAP3K13\nMAP3K8\nMAPK1\nMAPK3\nMAPK8IP1\nMAPKAP1\nMCL1\nMDM2\nMDM4\nMECOM\nMED12\nMET\nMGA\nMIR142\nMITF\nMLH1\nMLH3\nMLST8\nMPL\nMRAS\nMSH2\nMSH3\nMSH6\nMTOR\nMUTYH\nMYC\nMYCL\nMYCN\nMYH9\nNAV3\nNCOR1\nNCOR2\nNF1\nNF2\nNFE2L2\nNFE2L3\nNKX2-1\nNKX2-8\nNOTCH1\nNPM1\nNRAS\nNRG1\nNSD1\nPAK1\nPALB2\nPBRM1\nPCBP1\nPDGFRA\nPDPK1\nPHF6\nPHLPP1\nPHLPP2\nPIK3C2B\nPIK3C2G\nPIK3C3\nPIK3CA\nPIK3CB\nPIK3CD\nPIK3CG\nPIK3R1\nPIK3R2\nPIM1\nPIM2\nPIM3\nPMS1\nPMS2\nPOLD1\nPOLE\nPOLQ\nPPP2R1A\nPPP2R2A\nPRKAA1\nPRKAA2\nPRKAB1\nPRX\nPTCH1\nPTEN\nPTPN11\nPTPN6\nRAC1\nRAD21\nRAD51B\nRAD51C\nRAD51D\nRAF1\nRASA1\nRB1\nRBM10\nRELN\nRET\nRHEB\nRHOA\nRICTOR\nRIT1\nROS1\nRPL22\nRPL5\nRPS6\nRPS6KB1\nRPS6KB2\nRPTOR\nRUNX1\nSETBP1\nSETD2\nSF3B1\nSGK1\nSGK2\nSGK3\nSHOC2\nSIN3A\nSMAD2\nSMAD4\nSMARCA4\nSMARCB1\nSMC1A\nSMC3\nSMO\nSOS1\nSOX17\nSOX2\nSOX9\nSPOP\nSPRED1\nSRY\nSTAG2\nSTAT3\nSTK11\nTAF1\nTBL1XR1\nTBX3\nTERT\nTET2\nTFRC\nTGFBR2\nTIAF1\nTLR4\nTOP2A\nTP53\nTSC1\nTSC2\nTSHZ2\nTSHZ3\nU2AF1\nUSP9X\nVEZF1\nVHL\nWEE1\nWT1\nZNF217\nETV1\nETV4\nETV5\nETV6\nEWSR1\nKMT2A\nPDGFRB\nRARA\nTMPRSS2\nERG\nMYB\nNOTCH2\nNTRK1\nNTRK2\nRSPO2\n"
  },
  {
    "path": "utils/prioritize/az300_to_bed.py",
    "content": "#!/usr/bin/env python\n\"\"\"Map AZ300 gene name list to transcript regions.\n\nThis requires a 3 pass approach to get gene names to coordinates:\n\n- it first tries against ensemble BED coordinates in a transcript file with\n  the supplied name\n- it then remaps the name with gene.info to alternative symbols and tries to\n  find those in the transcript file.\n- Finally, it uses coordinates from gene.info if those exist.\n\"\"\"\nimport os\nimport sys\n\nimport requests\n\nref_dir = \"/human\"\nin_file = sys.argv[1]\n\ndef read_targets(in_file):\n    targets = set([])\n    with open(in_file) as in_handle:\n        for line in in_handle:\n            cur_symbol = (line.strip())\n            targets.add(cur_symbol)\n    return targets\n\ndef get_gene_info(cur_symbol):\n    chroms = [str(x) for x in range(1, 23)] + [\"X\", \"Y\"]\n    fields = \"symbol,genomic_pos_hg19\"\n    url = \"http://mygene.info/v2/query?q=%s&species=human&fields=%s\" % (cur_symbol, fields)\n    info = requests.get(url).json()\n    hits = info[\"hits\"]\n    symbols = [x[\"symbol\"] for x in hits]\n    pos = []\n    for ps in [x[\"genomic_pos_hg19\"] for x in hits if \"genomic_pos_hg19\" in x]:\n        if not isinstance(ps, (list, tuple)):\n            ps = [ps]\n        for p in ps:\n            if p[\"chr\"] in chroms:\n                pos.append(p)\n    return symbols, pos\n\ndef find_missing_targets(missing, in_file, genome):\n    missing_file = \"%s-%s-missingsymbols.txt\" % (os.path.splitext(in_file)[0], genome)\n    out = set([])\n    if os.path.exists(missing_file):\n        with open(missing_file) as in_handle:\n            for line in in_handle:\n                out.add(line.strip())\n    else:\n        with open(missing_file, \"w\") as out_handle:\n            for cur_symbol in missing:\n                symbols, pos = get_gene_info(cur_symbol)\n                if cur_symbol not in symbols and len(symbols) > 0:\n                    cur_symbol = symbols[0]\n                out_handle.write(\"%s\\n\" % cur_symbol)\n                out.add(cur_symbol)\n    return out\n\ndef write_from_transcript_file(targets, ref_dir, genome, out_handle):\n    ref_file = os.path.join(ref_dir, genome, \"rnaseq\", \"ref-transcripts.bed\")\n    found = set([])\n    with open(ref_file) as in_handle:\n        for line in in_handle:\n            name = line.split()[3]\n            if name in targets:\n                found.add(name)\n                out_handle.write(line)\n    return targets - found\n\ndef write_from_remap_names(targets, ref_dir, genome, out_handle, in_file):\n    ref_file = os.path.join(ref_dir, genome, \"rnaseq\", \"ref-transcripts.bed\")\n    targets = find_missing_targets(targets, in_file, genome)\n    found = set([])\n    with open(ref_file) as in_handle:\n        for line in in_handle:\n            name = line.split()[3]\n            if name in targets:\n                found.add(name)\n                out_handle.write(line)\n    return targets - found\n\ndef write_from_gene_info(targets, genome, out_handle):\n    missing = []\n    for target in sorted(targets):\n        symbols, pos = get_gene_info(target)\n        if pos:\n            assert isinstance(pos, (list, tuple))\n            if symbols:\n                target = symbols[0]\n            for p in pos:\n                chrom = \"%s%s\" % (\"chr\" if genome == \"hg19\" else \"\", p[\"chr\"])\n                out_handle.write(\"\\t\".join([chrom, str(p[\"start\"]), str(p[\"end\"]), target]) + \"\\n\")\n        else:\n            missing.append(target)\n    return missing\n\nfor genome in [\"hg19\", \"GRCh37\", \"hg38\"]:\n    out_file = \"%s-%s.bed\" % (os.path.splitext(in_file)[0], genome)\n    with open(out_file, \"w\") as out_handle:\n        targets = read_targets(in_file)\n        print \"total\", len(targets)\n        targets = write_from_transcript_file(targets, ref_dir, genome, out_handle)\n        print(\"after first name pass\", len(targets))\n        targets = write_from_remap_names(targets, ref_dir, genome, out_handle, in_file)\n        print(\"after rename name pass\", len(targets))\n        targets = write_from_gene_info(targets, genome, out_handle)\n        print(\"after coordinate retrieval\", targets)\n"
  },
  {
    "path": "utils/prioritize/prep_ccds_genes.py",
    "content": "#!/usr/bin/env python\n\"\"\"Prepare adjusted CCDS BED file.\n\nConsensus CDS (CCDS https://www.ncbi.nlm.nih.gov/projects/CCDS/CcdsBrowse.cgi) regions with two adjustments:\n\n- 2 bps added to internal introns to capture canonical splice acceptor/donor sites\n- Multiple transcripts from a single gene are merged into a single all inclusive gene entry.\n\"\"\"\nimport subprocess\nimport sys\n\ndef main(in_file, out_name, fai_file):\n    out_file  = \"%s.bed\" % out_name\n    with open(in_file) as in_handle:\n        with open(out_file, \"w\") as out_handle:\n            for line in in_handle:\n                gene, chrom, coords, _ = line.split()\n                chrom = get_chrom(chrom)\n                for start, end in split_coords(coords):\n                    out_handle.write(\"\\t\".join([chrom, start, end, gene]) + \"\\n\")\n    cmd = \"gsort {out_file} {fai_file} | bgzip -c > {out_file}.gz\"\n    subprocess.check_call(cmd.format(**locals()), shell=True)\n\ndef get_chrom(chrom):\n    assert chrom in set([str(x) for x in range(1, 23)] + [\"X\", \"Y\"]), chrom\n    return \"chr%s\" % chrom\n\ndef split_coords(coords):\n    assert coords[0] == \"(\"\n    assert coords[-1] == \")\", coords\n    out = []\n    for c in coords[1:-1].split(\",\"):\n        start, end = c.split(\"..\")\n        out.append((start, end))\n    return out\n\nif __name__ == \"__main__\":\n    main(*sys.argv[1:])\n"
  },
  {
    "path": "utils/prioritize/prep_prioritize_downloads.sh",
    "content": "#!/bin/bash\nset -eu -o pipefail\nversion=20181227\ncivicv=2018-12-27\nREF=/mnt/work/bcbio/genomes/Hsapiens\n\nmkdir -p hg19/cancer\nmkdir -p GRCh37/cancer\nmkdir -p hg38/cancer\n\n# prep work to do\n# bcbio-prioritize create-civic -b GRCh37\n# bcbio-prioritize create-civic -b GRCh38\n# python az300_to_bed.py AZ300.txt\n# python az300_to_bed.py AZ300_with_known.txt\n# python az300_to_bed.py az-cancer-panel.txt\n\n# CIViC\nzcat civic-GRCh37-$civicv.bed.gz | gsort - $REF/GRCh37/seq/GRCh37.fa.fai | bgzip -c > GRCh37/cancer/civic-$civicv.bed.gz\ntabix -f -p bed GRCh37/cancer/civic-$civicv.bed.gz\nzcat civic-GRCh37-$civicv.bed.gz | sed 's/^/chr/' | gsort - $REF/hg19/seq/hg19.fa.fai | bgzip -c > hg19/cancer/civic-$civicv.bed.gz\ntabix -f -p bed hg19/cancer/civic-$civicv.bed.gz\nzcat civic-GRCh38-$civicv.bed.gz | sed 's/^/chr/' | gsort - $REF/hg38/seq/hg38.fa.fai | bgzip -c > hg38/cancer/civic-$civicv.bed.gz\ntabix -f -p bed hg38/cancer/civic-$civicv.bed.gz\n\n# az300\ncat AZ300-hg19.bed | cut -f 1-4 | sort -V -k1,1 -k2,2n | bedtools merge -i - -c 4 -o distinct | bgzip -c > hg19/cancer/az300.bed.gz\ntabix -f -p bed hg19/cancer/az300.bed.gz\ncat AZ300-GRCh37.bed | cut -f 1-4 | sort -V -k1,1 -k2,2n | bedtools merge -i - -c 4 -o distinct | bgzip -c > GRCh37/cancer/az300.bed.gz\ntabix -f -p bed GRCh37/cancer/az300.bed.gz\ncat AZ300-hg38.bed | cut -f 1-4 | sort -V -k1,1 -k2,2n | bedtools merge -i - -c 4 -o distinct | bgzip -c > hg38/cancer/az300.bed.gz\ntabix -f -p bed hg38/cancer/az300.bed.gz\n\n# az300 with fusions\ncat AZ300_with_known-hg19.bed | cut -f 1-4 | sort -V -k1,1 -k2,2n | bedtools merge -i - -c 4 -o distinct | bgzip -c > hg19/cancer/az300-with-fusion.bed.gz\ntabix -f -p bed hg19/cancer/az300-with-fusion.bed.gz\ncat AZ300_with_known-GRCh37.bed | cut -f 1-4 | sort -V -k1,1 -k2,2n | bedtools merge -i - -c 4 -o distinct | bgzip -c > GRCh37/cancer/az300-with-fusion.bed.gz\ntabix -f -p bed GRCh37/cancer/az300-with-fusion.bed.gz\ncat AZ300_with_known-hg38.bed | cut -f 1-4 | sort -V -k1,1 -k2,2n | bedtools merge -i - -c 4 -o distinct | bgzip -c > hg38/cancer/az300-with-fusion.bed.gz\ntabix -f -p bed hg38/cancer/az300-with-fusion.bed.gz\n\n# az-cancer-panel\ncat az-cancer-panel-hg19.bed | cut -f 1-4 | sort -V -k1,1 -k2,2n | bedtools merge -i - -c 4 -o distinct | bgzip -c > hg19/cancer/az-cancer-panel.bed.gz\ntabix -f -p bed hg19/cancer/az-cancer-panel.bed.gz\ncat az-cancer-panel-GRCh37.bed | cut -f 1-4 | sort -V -k1,1 -k2,2n | bedtools merge -i - -c 4 -o distinct | bgzip -c > GRCh37/cancer/az-cancer-panel.bed.gz\ntabix -f -p bed GRCh37/cancer/az-cancer-panel.bed.gz\ncat az-cancer-panel-hg38.bed | cut -f 1-4 | sort -V -k1,1 -k2,2n | bedtools merge -i - -c 4 -o distinct | bgzip -c > hg38/cancer/az-cancer-panel.bed.gz\ntabix -f -p bed hg38/cancer/az-cancer-panel.bed.gz\n\n# tar up downloads\ncd hg19\ntar -czvpf ../prioritize-cancer-hg19-$version.tar.gz cancer\ncd ../GRCh37\ntar -czvpf ../prioritize-cancer-GRCh37-$version.tar.gz cancer\ncd ../hg38\ntar -czvpf ../prioritize-cancer-hg38-$version.tar.gz cancer\ncd ..\n\naws s3 cp prioritize-cancer-hg19-$version.tar.gz s3://biodata/coverage/prioritize/ --acl public-read\naws s3 cp prioritize-cancer-GRCh37-$version.tar.gz s3://biodata/coverage/prioritize/ --acl public-read\naws s3 cp prioritize-cancer-hg38-$version.tar.gz s3://biodata/coverage/prioritize/ --acl public-read\n"
  },
  {
    "path": "utils/query_conda_deps.py",
    "content": "#!/usr/bin/env python\n\"\"\"Query conda dependencies, summarizing for environment stratification.\n\nHelps identify grouping of packages for different sub-environments.\n\"\"\"\nimport sys\n\nimport requests\nimport yaml\n\ndef main(config_file):\n    with open(config_file) as in_handle:\n        config = yaml.safe_load(in_handle)\n    channels = config[\"channels\"]\n    channels.reverse()\n    for p in sorted(config[\"bio_nextgen\"]):\n        p = p.split(\";\")[0]\n        env = \"default\"\n        for c in channels:\n            deps = get_dependencies(c, p)\n            if deps is not None:\n                if \"python\" in deps and deps.get(\"python\"):\n                    if not any([s[1].startswith(\"3.\") for s in deps[\"python\"]]):\n                        env = \"python2\"\n                break\n        print(p, env)\n\ndef get_dependencies(channel, package):\n    url = \"https://api.anaconda.org/package/%s/%s/files\" % (channel, package)\n    out = requests.get(url).json()\n    if isinstance(out, dict) and out.get(\"error\"):\n        return None\n    else:\n        deps = {\"python\": set([])}\n        for f in out:\n            for d in f[\"dependencies\"][\"depends\"]:\n                for k in deps.keys():\n                    if d[\"name\"] == k:\n                        for s in d[\"specs\"]:\n                            deps[k].add(tuple(s))\n        return deps\n\nif __name__ == \"__main__\":\n    main(sys.argv[1])\n"
  },
  {
    "path": "utils/s3_multipart_upload.py",
    "content": "#!/usr/bin/env python\n\"\"\"Split large file into multiple pieces for upload to S3.\n\nS3 only supports 5Gb files for uploading directly, so for larger CloudBioLinux\nbox images we need to use boto's multipart file support.\n\nThis parallelizes the task over available cores using multiprocessing.\n\nIt checks for an up to date version of the file remotely, skipping transfer\nif found.\n\nNote: by default this will look for your default AWS Access Key ID and AWS Secret Access Key\n you setup via 'aws configure'.  You can store additional profiles using \n 'aws configure --profile <some_profile_name>'\n\nUsage:\n  s3_multipart_upload.py <file_to_transfer> <bucket_name> [<s3_key_name>]\n    if <s3_key_name> is not specified, the filename will be used.\n\n    --norr -- Do not use reduced redundancy storage.\n    --public -- Make uploaded files public.\n    --cores=n -- Number of cores to use for upload\n    --profile -- The alternate AWS profile to use for your keys located in ~/.aws/config\n\n    Files are stored at cheaper reduced redundancy storage by default.\n\"\"\"\nimport os\nimport sys\nimport glob\nimport subprocess\nimport contextlib\nimport functools\nimport multiprocessing\nfrom multiprocessing.pool import IMapIterator\nfrom optparse import OptionParser\nimport rfc822\n\nimport boto\n\ndef main(transfer_file, bucket_name, s3_key_name=None, use_rr=True,\n         make_public=True, cores=None, profile=None):\n    if s3_key_name is None:\n        s3_key_name = os.path.basename(transfer_file)\n    if profile is None:\n\t\tconn = boto.connect_s3()\n    else:\n\t\tconn = boto.connect_s3(profile_name=profile)\n    bucket = conn.lookup(bucket_name)\n    if bucket is None:\n        bucket = conn.create_bucket(bucket_name)\n    if s3_has_uptodate_file(bucket, transfer_file, s3_key_name):\n        print \"S3 has up to date version of %s in %s. Not transferring.\" % \\\n            (s3_key_name, bucket.name)\n        return\n    mb_size = os.path.getsize(transfer_file) / 1e6\n    if mb_size < 50:\n        _standard_transfer(bucket, s3_key_name, transfer_file, use_rr)\n    else:\n        _multipart_upload(bucket, s3_key_name, transfer_file, mb_size, use_rr,\n                          cores, profile)\n    s3_key = bucket.get_key(s3_key_name)\n    if make_public:\n        s3_key.set_acl(\"public-read\")\n\ndef s3_has_uptodate_file(bucket, transfer_file, s3_key_name):\n    \"\"\"Check if S3 has an existing, up to date version of this file.\n    \"\"\"\n    s3_key = bucket.get_key(s3_key_name)\n    if s3_key:\n        s3_size = s3_key.size\n        local_size = os.path.getsize(transfer_file)\n        s3_time = rfc822.mktime_tz(rfc822.parsedate_tz(s3_key.last_modified))\n        local_time = os.path.getmtime(transfer_file)\n        return s3_size == local_size and s3_time >= local_time\n    return False\n\ndef upload_cb(complete, total):\n    sys.stdout.write(\".\")\n    sys.stdout.flush()\n\ndef _standard_transfer(bucket, s3_key_name, transfer_file, use_rr):\n    print \" Upload with standard transfer, not multipart\",\n    new_s3_item = bucket.new_key(s3_key_name)\n    new_s3_item.set_contents_from_filename(transfer_file, reduced_redundancy=use_rr,\n                                           cb=upload_cb, num_cb=10)\n    print\n\ndef map_wrap(f):\n    @functools.wraps(f)\n    def wrapper(*args, **kwargs):\n        return apply(f, *args, **kwargs)\n    return wrapper\n\ndef mp_from_ids(mp_id, mp_keyname, mp_bucketname, profile=None):\n    \"\"\"Get the multipart upload from the bucket and multipart IDs.\n\n    This allows us to reconstitute a connection to the upload\n    from within multiprocessing functions.\n    \"\"\"\n    if profile is None:\n\t\tconn = boto.connect_s3()\n    else:\n\t\tconn = boto.connect_s3(profile_name=profile)\n    bucket = conn.lookup(mp_bucketname)\n    mp = boto.s3.multipart.MultiPartUpload(bucket)\n    mp.key_name = mp_keyname\n    mp.id = mp_id\n    return mp\n\n@map_wrap\ndef transfer_part(mp_id, mp_keyname, mp_bucketname, i, part, profile):\n    \"\"\"Transfer a part of a multipart upload. Designed to be run in parallel.\n    \"\"\"\n    mp = mp_from_ids(mp_id, mp_keyname, mp_bucketname, profile)\n    print \" Transferring\", i, part\n    with open(part) as t_handle:\n        mp.upload_part_from_file(t_handle, i+1)\n    os.remove(part)\n\ndef _multipart_upload(bucket, s3_key_name, tarball, mb_size, use_rr=True,\n                      cores=None, profile=None):\n    \"\"\"Upload large files using Amazon's multipart upload functionality.\n    \"\"\"\n    def split_file(in_file, mb_size, split_num=5):\n        prefix = os.path.join(os.path.dirname(in_file),\n                              \"%sS3PART\" % (os.path.basename(s3_key_name)))\n        # require a split size between 5Mb (AWS minimum) and 250Mb\n        split_size = int(max(min(mb_size / (split_num * 2.0), 250), 5))\n        if not os.path.exists(\"%saa\" % prefix):\n            cl = [\"split\", \"-b%sm\" % split_size, in_file, prefix]\n            subprocess.check_call(cl)\n        return sorted(glob.glob(\"%s*\" % prefix))\n\n    mp = bucket.initiate_multipart_upload(s3_key_name, reduced_redundancy=use_rr)\n    with multimap(cores) as pmap:\n        for _ in pmap(transfer_part, ((mp.id, mp.key_name, mp.bucket_name, i, part, profile)\n                                      for (i, part) in\n                                      enumerate(split_file(tarball, mb_size, cores)))):\n            pass\n    mp.complete_upload()\n\n@contextlib.contextmanager\ndef multimap(cores=None):\n    \"\"\"Provide multiprocessing imap like function.\n\n    The context manager handles setting up the pool, worked around interrupt issues\n    and terminating the pool on completion.\n    \"\"\"\n    if cores is None:\n        cores = max(multiprocessing.cpu_count() - 1, 1)\n    def wrapper(func):\n        def wrap(self, timeout=None):\n            return func(self, timeout=timeout if timeout is not None else 1e100)\n        return wrap\n    IMapIterator.next = wrapper(IMapIterator.next)\n    pool = multiprocessing.Pool(cores)\n    yield pool.imap\n    pool.terminate()\n\nif __name__ == \"__main__\":\n    parser = OptionParser()\n    parser.add_option(\"-r\", \"--norr\", dest=\"use_rr\",\n                      action=\"store_false\", default=True)\n    parser.add_option(\"-p\", \"--public\", dest=\"make_public\",\n                      action=\"store_true\", default=False)\n    parser.add_option(\"-c\", \"--cores\", dest=\"cores\",\n                      default=multiprocessing.cpu_count())\n    parser.add_option(\"--profile\", dest=\"profile\")\n    (options, args) = parser.parse_args()\n    if len(args) < 2:\n        print __doc__\n        sys.exit()\n    kwargs = dict(use_rr=options.use_rr, make_public=options.make_public,\n                  cores=int(options.cores), profile=options.profile)\n    main(*args, **kwargs)\n"
  },
  {
    "path": "utils/sv/NA24385_crowd_dels.py",
    "content": "#!/usr/bin/env python\n\"\"\"Prepare truth set of crowd sourced CNVs from GiaB samples.\n\nhttp://biorxiv.org/content/early/2016/12/13/093526\n\"\"\"\nimport requests\n\nfrom bcbio.variation import vcfutils\n\nurl = \"http://biorxiv.org/content/biorxiv/suppl/2016/12/13/093526.DC1/093526-3.txt\"\n\nout_base = \"NA24385-crowd-dels-%s.bed\"\n\nr = requests.get(url, stream=True)\nheader = None\ncalls = []\nfor line in r.iter_lines():\n    if not header:\n        header = line.rstrip().split()\n    else:\n        cur = dict(zip(header, line.rstrip().split()))\n        pos = (int(cur[\"chrom\"].replace(\"chr\", \"\")), int(cur[\"start\"]))\n        probs = sorted([(float(cur[\"%s_prob\" % c]), c) for c in [\"CN0\", \"CN1\", \"CN2\"]], reverse=True)\n        cn = probs[0][-1]\n        if probs[0][0] > 0.8 and cn != \"CN2\":\n            line = \"\\t\".join([cur[\"chrom\"], cur[\"start\"], cur[\"end\"], cur[\"site_id\"], cn]) + \"\\n\"\n            calls.append((pos, line))\ncalls.sort()\n\nhg19_bed = out_base % \"hg19\"\ngrch37_bed = out_base % \"GRCh37\"\nwith open(hg19_bed, \"w\") as hg19_handle:\n    with open(grch37_bed, \"w\") as grch37_handle:\n        for _, line in calls:\n            hg19_handle.write(line)\n            grch37_handle.write(line.replace(\"chr\", \"\"))\n\nvcfutils.bgzip_and_index(hg19_bed, {})\nvcfutils.bgzip_and_index(grch37_bed, {})\n"
  },
  {
    "path": "utils/sv/NA24385_giab_dels.py",
    "content": "#!/usr/bin/env python\n\"\"\"Prepare SV truth set from NIST integrated calls for NA24385\n\"\"\"\nimport os\nimport subprocess\n\nfrom bcbio.variation import vcfutils\n\nurl = \"ftp://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/data/AshkenazimTrio/analysis/NIST_DraftIntegratedDeletionsgt19bp_v0.1.8/\"\nbase = \"GIAB_HG002_del_2tech_filt_%s_noTRs_v0.1.8.bed\"\nsizes = [\"50to100bp\", \"100to1000bp\", \"1000to3000bp\", \"gt3000bp\"]\nfai_file = \"/human/GRCh37/seq/GRCh37.fa.fai\"\nout_file = \"NA24385-GIAB-2tech-dels-v0_1_8-GRCh37.bed\"\n\nfor size in sizes:\n    cur_url = url + base % size\n    subprocess.check_call([\"wget\", \"-c\", cur_url])\n\nwith open(\"header.txt\", \"w\") as out_handle:\n    with open(os.path.basename(cur_url)) as in_handle:\n        out_handle.write(\"# %s\" % in_handle.readline())\nsubprocess.check_call((\"cat GIAB_HG002* | grep -v ^X. | gsort /dev/stdin {fai_file} \"\n                       \"| cat header.txt - > {out_file}\".format(**locals())), shell=True)\nout_file_hg19 = out_file.replace(\"GRCh37\", \"hg19\")\nwith open(out_file) as in_handle:\n    with open(out_file_hg19, \"w\") as out_handle:\n        for line in in_handle:\n            if not line.startswith(\"#\"):\n                line = \"chr\" + line\n            out_handle.write(line)\nvcfutils.bgzip_and_index(out_file, {})\nvcfutils.bgzip_and_index(out_file_hg19, {})\n"
  }
]