[
{
"path": ".gitpod.yml",
"content": "image: gitpod/workspace-base\n\ntasks:\n- name: install mamba and earlgrey\n init: |\n curl -L -O \"https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-$(uname)-$(uname -m).sh\"\n bash Mambaforge-$(uname)-$(uname -m).sh -b -p /workspace/conda && rm Mambaforge-$(uname)-$(uname -m).sh\n /workspace/conda/bin/mamba init bash\n\n source ${HOME}/.bashrc\n\n mamba create -n earlgrey -c conda-forge -c bioconda \\\n earlgrey -y\n\n mamba clean --all -y\n\n mamba activate earlGrey\n\n cd /workspace\n wget https://sk13.cog.sanger.ac.uk/NC_045808_EarlWorkshop.fasta\n\n command: |\n /workspace/conda/bin/mamba init bash\n source ~/.bashrc\n mamba activate earlgrey\n clear\n\ngithub:\n prebuilds:\n # enable for the master/default branch (defaults to true)\n master: true\n # enable for all branches in this repo (defaults to false)\n branches: true\n # enable for pull requests coming from this repo (defaults to true)\n pullRequests: true\n # enable for pull requests coming from forks (defaults to false)\n pullRequestsFromForks: true\n # add a \"Review in Gitpod\" button as a comment to pull requests (defaults to true)\n addComment: true\n # add a \"Review in Gitpod\" button to pull requests (defaults to false)\n addBadge: false\n # add a label once the prebuild is ready to pull requests (defaults to false)\n addLabel: prebuilt-in-gitpod\n\nvscode:\n extensions:\n - anwar.papyrus-pdf\n\nworkspaceLocation: \"/workspace\"\n"
},
{
"path": "CITATION.cff",
"content": "# This CITATION.cff file was generated with cffinit.\n# Visit https://bit.ly/cffinit to generate yours today!\n\ncff-version: 1.2.0\ntitle: Earl Grey\nmessage: >-\n If you use this software, please cite it using the\n metadata from this file.\ntype: software\nversion: \"7.2.4\"\nauthors:\n - given-names: Tobias\n family-names: Baril\n email: toby.baril@icloud.com\n affiliation: Université de Neuchâtel\n orcid: 'https://orcid.org/0000-0002-5936-7531'\n - given-names: James\n family-names: Galbraith\n affiliation: University of Exeter\n orcid: 'https://orcid.org/0000-0002-1871-2108'\n - given-names: Alex\n family-names: Hayward\n affiliation: University of Exeter\n orcid: 'https://orcid.org/0000-0001-7413-718X'\n"
},
{
"path": "CODE_OF_CONDUCT.md",
"content": "# Contributor Covenant Code of Conduct\n\n## Our Pledge\n\nWe as members, contributors, and leaders pledge to make participation in our\ncommunity a harassment-free experience for everyone, regardless of age, body\nsize, visible or invisible disability, ethnicity, sex characteristics, gender\nidentity and expression, level of experience, education, socio-economic status,\nnationality, personal appearance, race, religion, or sexual identity\nand orientation.\n\nWe pledge to act and interact in ways that contribute to an open, welcoming,\ndiverse, inclusive, and healthy community.\n\n## Our Standards\n\nExamples of behavior that contributes to a positive environment for our\ncommunity include:\n\n* Demonstrating empathy and kindness toward other people\n* Being respectful of differing opinions, viewpoints, and experiences\n* Giving and gracefully accepting constructive feedback\n* Accepting responsibility and apologizing to those affected by our mistakes,\n and learning from the experience\n* Focusing on what is best not just for us as individuals, but for the\n overall community\n\nExamples of unacceptable behavior include:\n\n* The use of sexualized language or imagery, and sexual attention or\n advances of any kind\n* Trolling, insulting or derogatory comments, and personal or political attacks\n* Public or private harassment\n* Publishing others' private information, such as a physical or email\n address, without their explicit permission\n* Other conduct which could reasonably be considered inappropriate in a\n professional setting\n\n## Enforcement Responsibilities\n\nCommunity leaders are responsible for clarifying and enforcing our standards of\nacceptable behavior and will take appropriate and fair corrective action in\nresponse to any behavior that they deem inappropriate, threatening, offensive,\nor harmful.\n\nCommunity leaders have the right and responsibility to remove, edit, or reject\ncomments, commits, code, wiki edits, issues, and other contributions that are\nnot aligned to this Code of Conduct, and will communicate reasons for moderation\ndecisions when appropriate.\n\n## Scope\n\nThis Code of Conduct applies within all community spaces, and also applies when\nan individual is officially representing the community in public spaces.\nExamples of representing our community include using an official e-mail address,\nposting via an official social media account, or acting as an appointed\nrepresentative at an online or offline event.\n\n## Enforcement\n\nInstances of abusive, harassing, or otherwise unacceptable behavior may be\nreported to the community leaders responsible for enforcement.\n\nAll complaints will be reviewed and investigated promptly and fairly.\n\nAll community leaders are obligated to respect the privacy and security of the\nreporter of any incident.\n\n## Enforcement Guidelines\n\nCommunity leaders will follow these Community Impact Guidelines in determining\nthe consequences for any action they deem in violation of this Code of Conduct:\n\n### 1. Correction\n\n**Community Impact**: Use of inappropriate language or other behavior deemed\nunprofessional or unwelcome in the community.\n\n**Consequence**: A private, written warning from community leaders, providing\nclarity around the nature of the violation and an explanation of why the\nbehavior was inappropriate. A public apology may be requested.\n\n### 2. Warning\n\n**Community Impact**: A violation through a single incident or series\nof actions.\n\n**Consequence**: A warning with consequences for continued behavior. No\ninteraction with the people involved, including unsolicited interaction with\nthose enforcing the Code of Conduct, for a specified period of time. This\nincludes avoiding interactions in community spaces as well as external channels\nlike social media. Violating these terms may lead to a temporary or\npermanent ban.\n\n### 3. Temporary Ban\n\n**Community Impact**: A serious violation of community standards, including\nsustained inappropriate behavior.\n\n**Consequence**: A temporary ban from any sort of interaction or public\ncommunication with the community for a specified period of time. No public or\nprivate interaction with the people involved, including unsolicited interaction\nwith those enforcing the Code of Conduct, is allowed during this period.\nViolating these terms may lead to a permanent ban.\n\n### 4. Permanent Ban\n\n**Community Impact**: Demonstrating a pattern of violation of community\nstandards, including sustained inappropriate behavior, harassment of an\nindividual, or aggression toward or disparagement of classes of individuals.\n\n**Consequence**: A permanent ban from any sort of public interaction within\nthe community.\n\n## Attribution\n\nThis Code of Conduct is adapted from the [Contributor Covenant][homepage],\nversion 2.0, available at\nhttps://www.contributor-covenant.org/version/2/0/code_of_conduct.html.\n\nCommunity Impact Guidelines were inspired by [Mozilla's code of conduct\nenforcement ladder](https://github.com/mozilla/diversity).\n\n[homepage]: https://www.contributor-covenant.org\n\nFor answers to common questions about this code of conduct, see the FAQ at\nhttps://www.contributor-covenant.org/faq. Translations are available at\nhttps://www.contributor-covenant.org/translations.\n"
},
{
"path": "Docker/Dockerfile",
"content": "# Based on Dockerfile from Dfam TETools https://github.com/Dfam-consortium/TETools, with added database configuration, conda configuration, and Earl Grey installation\n# and configuration\n\nFROM debian:latest AS builder\n\nRUN apt-get -y update && apt-get -y install \\\n curl gcc g++ make zlib1g-dev libgomp1 \\\n perl \\\n python3-h5py \\\n libfile-which-perl \\\n libtext-soundex-perl \\\n libjson-perl liburi-perl libwww-perl\n\n# Install dependencies and some basic utilities\nRUN apt-get -y update \\\n && apt-get -y install \\\n aptitude \\\n libgomp1 \\\n perl \\\n python3-h5py \\\n libfile-which-perl \\\n libtext-soundex-perl \\\n libjson-perl liburi-perl libwww-perl \\\n libdevel-size-perl \\\n && aptitude install -y ~pstandard ~prequired \\\n curl wget \\\n vim nano \\\n procps strace \\\n libpam-systemd-\n\nCOPY src/* /opt/src/\nWORKDIR /opt/src\n\n# Extract RMBlast\nRUN cd /opt \\\n && mkdir rmblast \\\n && tar --strip-components=1 -x -f src/rmblast-2.13.0+-x64-linux.tar.gz -C rmblast \\\n && rm src/rmblast-2.13.0+-x64-linux.tar.gz\n\n# Compile HMMER\nRUN tar -x -f hmmer-3.3.2.tar.gz \\\n && cd hmmer-3.3.2 \\\n && ./configure --prefix=/opt/hmmer && make && make install \\\n && make clean\n\n# Compile TRF\nRUN tar -x -f trf-4.09.1.tar.gz \\\n && cd TRF-4.09.1 \\\n && mkdir build && cd build \\\n && ../configure && make && cp ./src/trf /opt/trf \\\n && cd .. && rm -r build\n\n# Compile RepeatScout\nRUN tar -x -f RepeatScout-1.0.6.tar.gz \\\n && cd RepeatScout-1.0.6 \\\n && sed -i 's#^INSTDIR =.*#INSTDIR = /opt/RepeatScout#' Makefile \\\n && make && make install\n\n# Compile and configure RECON\nRUN tar -x -f RECON-1.08.tar.gz \\\n && mv RECON-1.08 ../RECON \\\n && cd ../RECON \\\n && make -C src && make -C src install \\\n && cp 00README bin/ \\\n && sed -i 's#^\\$path =.*#$path = \"/opt/RECON/bin\";#' scripts/recon.pl\n\n# Compile cd-hit\nRUN tar -x -f cd-hit-v4.8.1-2019-0228.tar.gz \\\n && cd cd-hit-v4.8.1-2019-0228 \\\n && make && mkdir /opt/cd-hit && PREFIX=/opt/cd-hit make install\n\n# Compile genometools (for ltrharvest)\nRUN tar -x -f gt-1.6.2.tar.gz \\\n && cd genometools-1.6.2 \\\n && make -i -j4 cairo=no && make -i cairo=no prefix=/opt/genometools install \\\n && make cleanup\n\n# Configure LTR_retriever\nRUN cd /opt \\\n && tar -x -f src/LTR_retriever-2.9.0.tar.gz \\\n && mv LTR_retriever-2.9.0 LTR_retriever \\\n && cd LTR_retriever \\\n && sed -i \\\n -e 's#BLAST+=#BLAST+=/opt/rmblast/bin#' \\\n -e 's#RepeatMasker=#RepeatMasker=/opt/RepeatMasker#' \\\n -e 's#HMMER=#HMMER=/opt/hmmer/bin#' \\\n -e 's#CDHIT=#CDHIT=/opt/cd-hit#' \\\n paths\n\n# Compile MAFFT\nRUN tar -x -f mafft-7.505-without-extensions-src.tgz \\\n && cd mafft-7.505-without-extensions/core \\\n && sed -i 's#^PREFIX =.*#PREFIX = /opt/mafft#' Makefile \\\n && make clean && make && make install \\\n && make clean\n\n# Compile NINJA\nRUN cd /opt \\\n && mkdir NINJA \\\n && tar --strip-components=1 -x -f src/NINJA-cluster.tar.gz -C NINJA \\\n && cd NINJA/NINJA \\\n && make clean && make all\n\n# Move UCSC tools\nRUN mkdir /opt/ucsc_tools \\\n && mv faToTwoBit twoBitInfo twoBitToFa /opt/ucsc_tools \\\n && chmod +x /opt/ucsc_tools/*\n#COPY LICENSE.ucsc /opt/ucsc_tools/LICENSE\n\n# Compile and configure coseg\nRUN cd /opt \\\n && tar -x -f src/coseg-0.2.2.tar.gz \\\n && cd coseg \\\n && sed -i 's@#!.*perl@#!/usr/bin/perl@' preprocessAlignments.pl runcoseg.pl refineConsSeqs.pl \\\n && sed -i 's#use lib \"/usr/local/RepeatMasker\";#use lib \"/opt/RepeatMasker\";#' preprocessAlignments.pl \\\n && make -i\n\n# Configure RepeatMasker\nRUN cd /opt \\\n && tar -x -f src/RepeatMasker-4.1.4.tar.gz \\\n && chmod a+w RepeatMasker/Libraries \\\n && cd RepeatMasker/Libraries \\\n && cp /opt/src/Dfam.h5.gz . \\\n && gunzip -f Dfam.h5.gz \\\n && cd /opt/RepeatMasker \\\n #&& cp /opt/src/RepBaseRepeatMaskerEdition-20181026.tar.gz . \\\n #&& tar -zxf RepBaseRepeatMaskerEdition-20181026.tar.gz \\\n #&& wget https://github.com/rmhubley/RepeatMasker/blob/development/Libraries/RMRBMeta.embl \\\n #&& cd /opt/RepeatMasker \\\n && mv famdb.py famdb.py.bak \\\n && cp /opt/src/famdb.py . \\\n && chmod 755 famdb.py \\ \n && perl configure \\\n -hmmer_dir=/opt/hmmer/bin \\\n -rmblast_dir=/opt/rmblast/bin \\\n -libdir=/opt/RepeatMasker/Libraries \\\n -trf_prgm=/opt/trf \\\n -default_search_engine=rmblast \\\n && echo \"RepeatMasker configured...\" && cd .. && rm src/RepeatMasker-4.1.4.tar.gz\n\n# Configure RepeatModeler\nRUN cd /opt \\\n && tar -x -f src/RepeatModeler-2.0.4.tar.gz \\\n && mv RepeatModeler-2.0.4 RepeatModeler \\\n && cd RepeatModeler \\\n && perl configure \\\n -cdhit_dir=/opt/cd-hit -genometools_dir=/opt/genometools/bin \\\n -ltr_retriever_dir=/opt/LTR_retriever -mafft_dir=/opt/mafft/bin \\\n -ninja_dir=/opt/NINJA/NINJA -recon_dir=/opt/RECON/bin \\\n -repeatmasker_dir=/opt/RepeatMasker \\\n -rmblast_dir=/opt/rmblast/bin -rscout_dir=/opt/RepeatScout \\\n -trf_dir=/opt/ \\\n -ucsctools_dir=/opt/ucsc_tools\n\nFROM debian:latest\n\n# Install dependencies and some basic utilities\nRUN apt update && apt-get -y install git\nRUN apt-get -y update \\\n && apt-get -y install \\\n aptitude \\\n libgomp1 \\\n perl \\\n python3-h5py \\\n libfile-which-perl \\\n libtext-soundex-perl \\\n libjson-perl liburi-perl libwww-perl \\\n libdevel-size-perl \\\n && aptitude install -y ~pstandard ~prequired \\\n curl wget \\\n vim nano \\\n procps strace \\\n libpam-systemd-\nRUN cd /opt/ \\\n && curl https://repo.anaconda.com/archive/Anaconda3-2022.10-Linux-x86_64.sh --output anaconda.sh \\\n && bash ./anaconda.sh -b -p /anaconda3 \\\n && eval \"$(/anaconda3/bin/conda shell.bash hook)\"\n\n\nENV PATH=$PATH:/opt/RepeatMasker:/opt/RepeatMasker/util:/opt/RepeatModeler:/opt/RepeatModeler/util:/opt/coseg:/opt/ucsc_tools:/opt:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/cd-hit/\n\nCOPY --from=builder /opt /opt\nRUN echo \"PS1='(earlgrey \\$(pwd))\\\\\\$ '\" >> /etc/bash.bashrc\nENV LANG=C\nENV PYTHONIOENCODING=utf8\n\nRUN apt update && apt install build-essential -y --no-install-recommends\nRUN apt update && apt install libcurl4-openssl-dev -y && apt-get install libbz2-dev liblzma-dev libtiff-dev libfreetype6-dev pkgconf libfontconfig1-dev libharfbuzz-dev libfribidi-dev bc -y\n\nARG USER_ID\nARG GROUP_ID\n\nRUN addgroup --gid $GROUP_ID user\nRUN adduser --disabled-password --gecos '' --uid $USER_ID --gid $GROUP_ID --home /home/user user\nRUN chmod a+rwx /opt/*\nRUN chmod a+rwx /home/user\nRUN chown -R user /home/user\nRUN cd /home/user \\\n && git clone https://github.com/ridgelab/SA-SSR \\\n && cd SA-SSR/ \\\n && sed -i \"s|PREFIX=/usr/local/bin|PREFIX=/home/user/SA-SSR/|g\" Makefile && make && make install\nRUN chmod a+rwx /home/user/SA-SSR/bin/* /home/user/SA-SSR/*\nRUN chown -R user /home/user/SA-SSR/\n\nUSER user\nWORKDIR /home/user\n\nRUN git clone https://github.com/TobyBaril/EarlGrey \\\n && cd EarlGrey \\\n && chmod +x ./configureForDocker \\\n && eval \"$(/anaconda3/bin/conda shell.bash hook)\" \\\n && ./configureForDocker \n\nENV PATH=$PATH:/home/user/EarlGrey/:/home/user/SA-SSR/bin/\n\n#RUN echo \"#!/bin/bash\" > /work/start.sh \\\n# && echo 'eval \"$(/anaconda3/bin/conda shell.bash hook)\"' >> /work/start.sh \\\n# && echo \"conda activate earlGrey\" >> /work/start.sh \\\n# && chmod +x /work/start.sh\n"
},
{
"path": "Docker/README.md",
"content": "# Docker Container \n\nA Docker container has been generated with none of Dfam 3.9, but with script generation to source required partitions\n\nI try to keep an up-to-date container in docker hub, but this might not always be the case depending on if I have had time to build and upload a new image. Currently, the recommended image ready for use is `-nodfam` version. Upon running the container interactively and running the command `earlGrey`, instructions will print to `stdout` and a script that you can use will be placed in `/usr/local/share/RepeatMasker/Libraries/famdb/` when the container is running.\n\n```\n# Interactive mode\n# Version 7.2.4 with no preconfigured partitions (RECOMMENDED!) - bind a directory, in my case the current directory using pwd\ndocker run -it -v 'pwd':/data/ tobybaril/earlgrey:latest-nodfam\n# change to library directory\ncd /data/\n# run earlGrey to make the configuration script\nearlGrey\n\n# then alter script with required partitions and run the configuration script\n# change 0-16 to whichever you require, but at least 0. This relates to the partitions of Dfam 3.9 (https://www.dfam.org/releases/Dfam_3.9/families/FamDB/)\n##### e.g. for 0-5:\nsed -i '/^curl/ s/0-16/0-5/g' configure_dfam39.sh\n##### e.g for 1,3,5:\nsed -i '/^curl/ s/0-16/1,3,5/g' configure_dfam39.sh\n\n# run the configuration script\nbash configure_dfam39.sh\n\n# return to your data directory\ncd /data/\n```\n\n"
},
{
"path": "Docker/configInstructions.txt",
"content": "# Get Earl Grey to run in Docker\n\n# Download dependencies\n./getFiles.sh\n\n# NOTE: please make sure that Dfam.h5.gz and RepBase libraries tar.gz are inside the ./src directory after running ./getsrc.sh . \n# If you need to download these,\n# Dfam.h5.gz can be downloaded by running:\nwget https://www.dfam.org/releases/Dfam_3.7/families/Dfam.h5.gz # THIS IS A BIG FILE!\n# Repbase is now behind a paywall, if you do not have access, please comment out the following lines in the Dockerfile (lines 105-107)\n# && cd /opt/RepeatMasker \\\n# && cp /opt/src/RepBaseRepeatMaskerEdition-20181026.tar.gz . \\\n# && tar -zxf RepBaseRepeatMaskerEdition-20181026.tar.gz \\\n\n\n# Build a docker container (run from inside the directory where the Dockerfile for EarlGrey is stored)\ndocker build . -t earlgrey\n\n# start the docker container\n\ndocker run -it --rm --init --mount type=bind,source=\"$(pwd)\",target=/work --user \"$(id -u):$(id -g)\" --workdir \"/work\" --env \"HOME=/work\" earlgrey \"$@\"\n\n# IMPORTANT - once the docker container has started, run these commands to activate the conda environment for EarlGrey\n# we strongly recommend upgrading the RepeatMasker contained within Docker with the Dfam libraries and RepBase, information on this is found: https://github.com/Dfam-consortium/TETools\n# note there are only ~6000 seqs in the basic RepeatMasker library in this container, real Dfam is much larger!\n\neval \"$(/anaconda3/bin/conda shell.bash hook)\"\nconda activate earlGrey\n"
},
{
"path": "Docker/getFiles.sh",
"content": "#!/bin/sh\n\nset -eu\n\n# download src name\ndownload() {\n src=\"$1\"\n shift\n\n if [ $# -ge 1 ]; then\n name=\"$1\"\n else\n name=\"${src##*/}\"\n fi\n\n dest=\"src/$name\"\n\n if [ -n \"${ALWAYS-}\" ] || ! [ -f \"$dest\" ]; then\n echo \"Downloading $src to $dest\"\n curl -sSL \"$src\" > \"$dest\"\n fi\n}\n\nmkdir -p src\n\ndownload https://www.repeatmasker.org/rmblast/rmblast-2.13.0+-x64-linux.tar.gz\ndownload http://eddylab.org/software/hmmer/hmmer-3.3.2.tar.gz\ndownload https://github.com/Benson-Genomics-Lab/TRF/archive/v4.09.1.tar.gz trf-4.09.1.tar.gz\ndownload https://www.repeatmasker.org/RepeatScout-1.0.6.tar.gz\ndownload https://www.repeatmasker.org/RepeatModeler/RECON-1.08.tar.gz\ndownload https://github.com/weizhongli/cdhit/releases/download/V4.8.1/cd-hit-v4.8.1-2019-0228.tar.gz\ndownload https://github.com/genometools/genometools/archive/v1.6.2.tar.gz gt-1.6.2.tar.gz\ndownload https://github.com/oushujun/LTR_retriever/archive/v2.9.0.tar.gz LTR_retriever-2.9.0.tar.gz\ndownload https://mafft.cbrc.jp/alignment/software/mafft-7.505-without-extensions-src.tgz\ndownload https://github.com/TravisWheelerLab/NINJA/archive/0.97-cluster_only.tar.gz NINJA-cluster.tar.gz\ndownload https://www.repeatmasker.org/coseg-0.2.2.tar.gz\ndownload https://www.repeatmasker.org/RepeatMasker/RepeatMasker-4.1.4.tar.gz\ndownload https://github.com/Dfam-consortium/RepeatModeler/archive/2.0.4.tar.gz RepeatModeler-2.0.4.tar.gz\ndownload https://github.com/Dfam-consortium/FamDB/raw/master/famdb.py famdb.py\ndownload https://www.dfam.org/releases/Dfam_3.7/families/Dfam_curatedonly.h5.gz Dfam.h5.gz\n\n# TODO: /exe/ only includes binaries of the \"latest\" version at the time of download.\n# The version listed in README.md is obtained by running 'strings src/faToTwoBit | grep kent'\n# On whatever was downloaded.\n# Consider building these tools from source instead.\nfor tool in faToTwoBit twoBitInfo twoBitToFa; do\n download https://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64/\"$tool\"\ndone\n"
},
{
"path": "LICENSE",
"content": "Open Software License v. 2.1\n============================\nThis Open Software License (the \"License\") applies to any original work\nof authorship (the \"Original Work\") whose owner (the \"Licensor\") has\nplaced the following notice immediately following the copyright notice\nfor the Original Work: Licensed under the Open Software License version 2.1 \n\n1) Grant of Copyright License. Licensor hereby grants You a world-wide,\nroyalty-free, non-exclusive, perpetual, sublicenseable license to do\nthe following:\n\nto reproduce the Original Work in copies; \nto prepare derivative works (\"Derivative Works\") based upon the Original Work; \nto distribute copies of the Original Work and Derivative Works to the \n public, with the proviso that copies of Original Work or Derivative Works\n that You distribute shall be licensed under the Open Software License; \nto perform the Original Work publicly; and \nto display the Original Work publicly. \n\n2) Grant of Patent License. Licensor hereby grants You a world-wide,\nroyalty-free, non-exclusive, perpetual, sublicenseable license, under\npatent claims owned or controlled by the Licensor that are embodied in\nthe Original Work as furnished by the Licensor, to make, use, sell and\noffer for sale the Original Work and Derivative Works.\n\n3) Grant of Source Code License. The term \"Source Code\" means the\npreferred form of the Original Work for making modifications to it\nand all available documentation describing how to modify the Original\nWork. Licensor hereby agrees to provide a machine-readable copy of the\nSource Code of the Original Work along with each copy of the Original\nWork that Licensor distributes. Licensor reserves the right to satisfy\nthis obligation by placing a machine-readable copy of the Source Code in\nan information repository reasonably calculated to permit inexpensive and\nconvenient access by You for as long as Licensor continues to distribute\nthe Original Work, and by publishing the address of that information\nrepository in a notice immediately following the copyright notice that\napplies to the Original Work.\n\n4) Exclusions From License Grant. Neither the names of Licensor, nor\nthe names of any contributors to the Original Work, nor any of their\ntrademarks or service marks, may be used to endorse or promote products\nderived from this Original Work without express prior written permission\nof the Licensor. Nothing in this License shall be deemed to grant any\nrights to trademarks, copyrights, patents, trade secrets or any other\nintellectual property of Licensor except as expressly stated herein. No\npatent license is granted to make, use, sell or offer to sell embodiments\nof any patent claims other than the licensed claims defined in Section\n2. No right is granted to the trademarks of Licensor even if such marks\nare included in the Original Work. Nothing in this License shall be\ninterpreted to prohibit Licensor from licensing under different terms\nfrom this License any Original Work that Licensor otherwise would have\na right to license.\n\n5) External Deployment. The term \"External Deployment\" means the use or\ndistribution of the Original Work or Derivative Works in any way such that\nthe Original Work or Derivative Works may be used by anyone other than\nYou, whether the Original Work or Derivative Works are distributed to\nthose persons or made available as an application intended for use over\na computer network. As an express condition for the grants of license\nhereunder, You agree that any External Deployment by You of a Derivative\nWork shall be deemed a distribution and shall be licensed to all under\nthe terms of this License, as prescribed in section 1(c) herein.\n\n6) Attribution Rights. You must retain, in the Source Code of any\nDerivative Works that You create, all copyright, patent or trademark\nnotices from the Source Code of the Original Work, as well as any\nnotices of licensing and any descriptive text identified therein as an\n\"Attribution Notice.\" You must cause the Source Code for any Derivative\nWorks that You create to carry a prominent Attribution Notice reasonably\ncalculated to inform recipients that You have modified the Original Work.\n\n7) Warranty of Provenance and Disclaimer of Warranty. Licensor\nwarrants that the copyright in and to the Original Work and the patent\nrights granted herein by Licensor are owned by the Licensor or are\nsublicensed to You under the terms of this License with the permission\nof the contributor(s) of those copyrights and patent rights. Except as\nexpressly stated in the immediately proceeding sentence, the Original\nWork is provided under this License on an \"AS IS\" BASIS and WITHOUT\nWARRANTY, either express or implied, including, without limitation,\nthe warranties of NON-INFRINGEMENT, MERCHANTABILITY or FITNESS FOR A\nPARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY OF THE ORIGINAL\nWORK IS WITH YOU. This DISCLAIMER OF WARRANTY constitutes an essential\npart of this License. No license to Original Work is granted hereunder\nexcept under this disclaimer.\n\n8) Limitation of Liability. Under no circumstances and under no legal\ntheory, whether in tort (including negligence), contract, or otherwise,\nshall the Licensor be liable to any person for any direct, indirect,\nspecial, incidental, or consequential damages of any character arising\nas a result of this License or the use of the Original Work including,\nwithout limitation, damages for loss of goodwill, work stoppage, computer\nfailure or malfunction, or any and all other commercial damages or\nlosses. This limitation of liability shall not apply to liability for\ndeath or personal injury resulting from Licensor's negligence to the\nextent applicable law prohibits such limitation. Some jurisdictions do\nnot allow the exclusion or limitation of incidental or consequential\ndamages, so this exclusion and limitation may not apply to You.\n\n9) Acceptance and Termination. If You distribute copies of the Original\nWork or a Derivative Work, You must make a reasonable effort under the\ncircumstances to obtain the express assent of recipients to the terms of\nthis License. Nothing else but this License (or another written agreement\nbetween Licensor and You) grants You permission to create Derivative Works\nbased upon the Original Work or to exercise any of the rights granted\nin Section 1 herein, and any attempt to do so except under the terms of\nthis License (or another written agreement between Licensor and You)\nis expressly prohibited by U.S. copyright law, the equivalent laws of\nother countries, and by international treaty. Therefore, by exercising\nany of the rights granted to You in Section 1 herein, You indicate Your\nacceptance of this License and all of its terms and conditions. This\nLicense shall terminate immediately and you may no longer exercise any\nof the rights granted to You by this License upon Your failure to honor\nthe proviso in Section 1(c) herein.\n\n10) Termination for Patent Action. This License shall terminate\nautomatically and You may no longer exercise any of the rights granted\nto You by this License as of the date You commence an action, including\na cross-claim or counterclaim, against Licensor or any licensee alleging\nthat the Original Work infringes a patent. This termination provision\nshall not apply for an action alleging patent infringement by combinations\nof the Original Work with other software or hardware.\n\n11) Jurisdiction, Venue and Governing Law. Any action or suit relating to this License may be brought only in the courts of a jurisdiction wherein the Licensor resides or in which Licensor conducts its primary business, and under the laws of that jurisdiction excluding its conflict-of-law provisions. The application of the United Nations Convention on Contracts for the International Sale of Goods is expressly excluded. Any use of the Original Work outside the scope of this License or after its termination shall be subject to the requirements and penalties of the U.S. Copyright Act, 17 U.S.C. § 101 et seq., the equivalent laws of other countries, and international treaty. This section shall survive the termination of this License. \n\n12) Attorneys Fees. In any action to enforce the terms of this License or\nseeking damages relating thereto, the prevailing party shall be entitled\nto recover its costs and expenses, including, without limitation,\nreasonable attorneys' fees and costs incurred in connection with such\naction, including any appeal of such action. This section shall survive\nthe termination of this License.\n\n13) Miscellaneous. This License represents the complete agreement\nconcerning the subject matter hereof. If any provision of this License\nis held to be unenforceable, such provision shall be reformed only to\nthe extent necessary to make it enforceable.\n\n14) Definition of \"You\" in This License. \"You\" throughout this License,\nwhether in upper or lower case, means an individual or a legal entity\nexercising rights under, and complying with all of the terms of, this\nLicense. For legal entities, \"You\" includes any entity that controls,\nis controlled by, or is under common control with you. For purposes of\nthis definition, \"control\" means (i) the power, direct or indirect, to\ncause the direction or management of such entity, whether by contract\nor otherwise, or (ii) ownership of fifty percent (50%) or more of the\noutstanding shares, or (iii) beneficial ownership of such entity.\n\n15) Right to Use. You may use the Original Work in all ways not otherwise\nrestricted or conditioned by this License or by law, and Licensor promises\nnot to interfere with or be responsible for such uses by You.\n\nThis license is Copyright (C) 2003-2004 Lawrence E. Rosen. All rights\nreserved. Permission is hereby granted to copy and distribute this\nlicense without modification. This license may not be modified without\nthe express written permission of its copyright owner.\n"
},
{
"path": "README.md",
"content": "\n\n[](https://zenodo.org/badge/latestdoi/412126708) [](https://anaconda.org/bioconda/earlgrey) [](\nhttps://anaconda.org/bioconda/earlgrey) [](https://anaconda.org/bioconda/earlgrey) [](https://anaconda.org/bioconda/earlgrey)\n\n# Earl Grey\n\nEarl Grey is a full-automated transposable element (TE) annotation pipeline, leveraging the most widely-used tools and combining these with a consensus elongation process to better define _de novo_ consensus sequences when annotating new genome assemblies.\n\n# Contents\n\n[Changes in Latest Release](#changes-in-latest-release)\n\n[Example Run](#example)\n\n[References and Acknowledgements](#references-and-acknowledgements)\n\n[Usage Without Installation](#usage-without-installation)\n\n[Recommended Installation](#recommended-installation-with-conda-or-mamba)\n\n[Docker Container](#docker-container)\n\n\n\n# Important Considerations\n\nEarl Grey version 6 uses Dfam 3.9. After installation, you MUST configure Dfam partitions as needed. Earl Grey will generate the script to do this and provide guidance when you run it for the first time. You need to specify which partitions of Dfam and/or RepBase to configure Earl Grey with. Choose partitions carefully as the combination will highly influence your results, especially if you want to pre-mask your input genome. Please make use of issues and discussions tabs if you have questions about this, we are always happy to help!\n\n# Notes / Updates\n\nWe often get questions related to runtime. TE curation and annotation remains resource and time intensive. Fast is not necessarily better, and runtime is highly dependent on genome size, complexity, and repeat content. Runs will likely take longer than you might expect, and be very RAM-hungry. As some generic benchmarks, a 40Mb genome can take anywhere from a few hours to a day, 400Mb up to around 4-5 days, a 3Gb genome ~a week, and a 25Gb genome several weeks! Things will be running even if it doesn't look like they are. Each step checkpoints, so if you have server limits, you can resubmit the same script with the same parameters, and Earl Grey will skip completed steps. `TEstrainer` and the final `divergence calculator` use a lot of memory. Check carefully for OOM errors in the logs! As a rule of thumb, you need _at least_ 3GB of RAM _per thread_, with more being better. Therefore, 16 threads requires at least 48GB of RAM depending on repeat complexity of the input genome.\n\nWe have been made aware of some instability in repeat annotation percentages when high numbers of CPUs are employed in certain server environments. Please be sure to check logs carefully for instances of interruption. Known cases so far will show the following message:\n\n```\nOpenBLAS blas_thread_init: pthread_create failed for thread X of X: Resource temporarily unavailable\nOpenBLAS blas_thread_init: ensure that your address space and process count limits are big enough (ulimit -a)\nOpenBLAS blas_thread_init: or set a smaller OPENBLAS_NUM_THREADS to fit into what you have available\n```\n\nIf you see this message, re-run analysis with less threads. Alternatively, you can modify your instance of the TEstrainer script `initial_mafft_setup.py` to add the following after `import os`:\n\n```\nos.environ['OPENBLAS_NUM_THREADS'] = '1'\n```\n\n# Changes in Latest Release\nEarl Grey v7.2.4 improves pipeline robustness and RepeatModeler handling for small or fragmented genomes:\n\n- **Proactive RepeatModeler sampling cap** (`earlGrey`, `earlGreyLibConstruct`): the `deNovo1` function previously used a nested retry loop that ran RepeatModeler up to three times, falling back to progressively smaller `-genomeSampleSizeMax` values after each failure. It is now replaced by a single proactive run. The samplable genome size (sum of contigs ≥ 40 kb, the threshold below which RepeatModeler discards contigs) is computed upfront using `awk`, and the appropriate `-genomeSampleSizeMax` cap is derived from the cumulative RECON round thresholds (none for ≥ 363 Mb; 81 Mb for ≥ 120 Mb; 27 Mb for ≥ 39 Mb; 9 Mb for ≥ 12 Mb; 3 Mb for < 12 Mb). RepeatModeler runs exactly once with the correct flag, eliminating wasted compute on small or fragmented genomes.\n- **`set -eo pipefail`** added to `earlGrey`, `earlGreyLibConstruct`, and `earlGreyAnnotationOnly`: any command that fails mid-pipeline now immediately aborts execution, preventing downstream stages from running with missing or corrupt intermediate files.\n- **`expr` replaced with `$(( ))`** throughout all three scripts: `expr N / 4` returns exit code 1 when the result is 0 (i.e. fewer than 4 threads requested), which would abort the pipeline under `set -e` before any tool was invoked. All thread-division expressions now use bash arithmetic `$(( ProcNum / 4 ))`.\n- **`ls | head -n 1` pipelines guarded with `|| true`**: under `pipefail`, `ls` receives SIGPIPE (exit 141) when `head` exits after reading one line. All affected pipelines now append `|| true` to suppress this expected signal.\n\n### Previous Changes\nEarl Grey v7.2.3 fixes five bugs reported by users:\n\n\n- **Off-by-one chunking in divergence_calc.py** ([#290](https://github.com/TobyBaril/EarlGrey/issues/290)): The previous floor-division chunking could produce one extra chunk when the row count was not evenly divisible by the thread count, causing the final chunk to run serially after all parallel workers had finished (effectively doubling wall-clock time in the worst case). Chunking now uses `np.array_split`, which always produces exactly `num_processes` chunks and distributes any remainder one row at a time.\n- **OSError: AF_UNIX path too long in divergence_calc.py** ([#294](https://github.com/TobyBaril/EarlGrey/issues/294)): `pybedtools.set_tempdir()` sets Python's `tempfile.tempdir` globally. The `forkserver` start method creates its Unix domain socket under `tempfile.tempdir`, so deeply-nested output paths could exceed the 108-character AF_UNIX socket limit. The divergence calculator is now passed a short per-species path in `/tmp` via `-tmp`, keeping the socket path well within the limit regardless of output directory depth.\n- **Crash when no nested TEs are found** ([#292](https://github.com/TobyBaril/EarlGrey/issues/292)): `filteringOverlappingRepeats.R` crashed with `object 'nested' not found` on genomes with low TE content where no fully-nested elements were detected. When the nesting-detection loop exits without finding any nesting, `bind_rows()` produces a zero-row data frame that lacks the `nested` and `nesting_round` columns, causing the downstream `mutate()` to fail. The fix guards the mutate block with an `nrow()` check — if no nested TEs are found the unnested output is written directly.\n- **Temp file path errors in divergence_calc.py under Singularity and other environments** ([#289](https://github.com/TobyBaril/EarlGrey/issues/289)): All path constructions in `divergence_calc.py` used string concatenation (e.g. `temp_dir + \"/qseqs\"`), which produces double slashes when `temp_dir` already has a trailing slash. This is particularly common in Singularity where bound paths are often passed with trailing slashes. All 13 affected path constructions have been replaced with `os.path.join()`, which correctly handles trailing slashes and redundant separators in all environments.\n- **TEstrainer parallel progress bar cluttering batch job logs** ([#293](https://github.com/TobyBaril/EarlGrey/issues/293)): A new `-q yes` flag on `earlGrey` (and `-q` on `TEstrainer_for_earlGrey.sh` directly) suppresses the GNU `parallel --bar` progress bar. This is particularly useful for `sbatch`/HPC log files where the animated bar produces thousands of redundant lines.\n\n### Previous Changes\nEarl Grey v7.2.1 patches a bug where the curated library directory was not created when an existing library was supplied via `-l` (without `-r`). In this case, `earlGreyAnnotationOnly` could fail when attempting to change into the directory during the final masking step. The directory is now created with `mkdir -p` before use, matching the fix already applied to the main `earlGrey` script.\n\n### Older Changes\nEarl Grey v7.2.0 significantly reduces peak RAM usage in two RAM-intensive components: TEstrainer and divergence_calc.py. These changes prevent OOM kills when running with large thread counts on memory-constrained compute nodes, with no change to output.\n\n**TEstrainer / TEstrainer_for_earlGrey.sh:**\n- The default `MEM_FREE` threshold raised from `200M` to `1G`. The previous value was lower than the startup cost of a single Python interpreter with heavy scientific libraries, making the guard ineffective.\n- All GNU `parallel` calls in the BEAT curation loop (trf, initial_mafft_setup, mafft, TEtrim) now carry `--memfree ${MEM_FREE}`, throttling job dispatch when free RAM drops below the threshold.\n- A RAM-cap guard is applied at startup: the requested thread count is capped based on available RAM (`free -m`) at an estimate of 800 MB per concurrent job, with a warning printed if a cap is applied.\n\n**divergence_calc.py:**\n- Switched from the default `fork` multiprocessing start method to `forkserver`. On Linux, `fork` duplicates the full parent address space (including the GFF DataFrame) into every worker; `forkserver` workers start clean and receive only a file path, eliminating N-fold GFF copies in RAM.\n- GFF chunks are now serialised to temp TSV files on disk before the pool is created. The parent DataFrame is freed before any workers are launched, reducing parent RSS during the pool run.\n- `pool.imap_unordered` replaces `pool.map`, allowing workers to be retired as they finish rather than all buffering results simultaneously.\n- `maxtasksperchild=1` forces worker process restart after each chunk, releasing accumulated pybedtools handles and BioPython caches between chunks.\n- Periodic `pybedtools.cleanup()` every 500 rows prevents temp-file accumulation in long-running workers.\n- A pre-existing bug was fixed: `os.remove(query_path)` was called unconditionally in both cleanup branches even when the file was never created (e.g. when `pybedtools.sequence()` raised a samtools exception). Both removal calls are now guarded with `if exists(query_path)`.\n\nBenchmarked results confirm peak RSS for `divergence_calc.py` is flat at ~76 MB across 1–16 threads (previously scaling linearly), and TEstrainer peak RSS is ~877 MB at both 4 and 32 threads (previously unbounded with thread count).\n\nEarl Grey v7.1.1 patches a small bug in TEstrainer where consensus sequences comprised of tandem repeats were triggering a warning due to the change to `pandas >2.0`. The output results were not affected. The codebase has now been updated to handle these cases without throwing warnings, with the same expected behaviour and handling of tandem repeats as before.\n\nEarl Grey v7.1.0 removes the dependency on Python 3.9, which is no longer supported. Earl Grey can now be built and run with Python 3.10 and above. This change was made to ensure that Earl Grey remains compatible with newer versions of Python and to remove the reliance on an unsupported version.\n\nEarl Grey v7.0.3 fixes an issue with final calculation tables which did not count _other_ repeats towards total repeat content.\n\nEarl Grey v7.0.2 adds RepeatLandscapes for _Penelope_-like elements and SINEs. Importantly, the `-norna` option in RepeatMasker is no longer invoked as default behaviour, which will improver the detection and masking of small tRNA-derived SINEs.\n\nEarl Grey v7.0.1 patches the summary table generation, where LINEs and Penelopes were being counted in both categories for nested repeats only.\n\n☕ Earl Grey v7.0.0 is here!\n\nSome long-awaited changes in this release—thank you for your patience while I found the time to properly test and implement them.\n\n🐞 RepeatCraft fixes\nFirst, I’ve fixed an issue in RepeatCraft where distal TEs could be grouped erroneously. This occurred in rare edge cases where the internal counter failed to iterate when new distal copies of an existing TE were detected on the same contig. This should now be fully resolved.\n\n🧬 Fully nested TE handling (major update)\nI’ve also completely revamped the Earl Grey post-processing and summary steps to properly deal with Fully Nested TEs.\nPartially overlapping TEs are handled as before, following the approach described in the original Earl Grey paper.\nFully nested TEs are now identified using a new iterative process:\nThe GFF is scanned for nested TEs\nThese are labelled and stored in a separate file\nThe nested TE is removed and the GFF is re-scanned to detect deeper (multi-level) nesting\nThis continues until no new nested TEs are found\n\n📊 Changes to coverage and summaries (important)\nNested TEs are no longer included in the TE coverage calculation used to generate the pie chart in the `summaryFiles` directory.\n\nInstead:\nSummary tables now include new categories showing how many base pairs are comprised of nested TEs\nThese base pairs are not counted toward Total Interspersed Repeats, as doing so would result in double-counting genomic space.\n\n⚠️ This represents a substantial change from previous versions, so please be aware of this difference when upgrading to v7.\n\nThe output table summary now has the following format:\n\n```\n|TE Classification | Coverage (bp)|Copy Number | % Genome Coverage| Genome Size| TE Family Count|\n|:-------------------------------------------------|-------------:|:-------------|-----------------:|-----------:|---------------:|\n|DNA | 80886|326 | 0.7607795| 10631990| 326|\n|DNA-nested | 1449|29 | 0.0136287| 10631990| 29|\n|Rolling Circle | 8022|31 | 0.0754515| 10631990| 31|\n|Rolling Circle-nested | 0|0 | 0.0000000| 10631990| NA|\n|Penelope | 112822|812 | 1.0611560| 10631990| 812|\n|Penelope-nested | 1268|13 | 0.0119263| 10631990| 13|\n|LINE | 132133|426 | 1.2427871| 10631990| 426|\n|LINE-nested | 1268|13 | 0.0119263| 10631990| 4|\n|SINE | 0|0 | 0.0000000| 10631990| NA|\n|SINE-nested | 0|0 | 0.0000000| 10631990| NA|\n|LTR | 0|0 | 0.0000000| 10631990| NA|\n|LTR-nested | 0|0 | 0.0000000| 10631990| NA|\n|Other (Simple Repeat, Microsatellite, RNA) | 189999|4241 | 1.7870502| 10631990| 4241|\n|Other (Simple Repeat, Microsatellite, RNA)-nested | 1225|34 | 0.0115218| 10631990| 34|\n|Unclassified | 274026|1291 | 2.5773726| 10631990| 1291|\n|Unclassified-nested | 0|0 | 0.0000000| 10631990| NA|\n|Total Interspersed Repeat | 607889|2886 | 5.7175468| 10631990| NA|\n|Non-Repeat | 10024101|notApplicable | 94.2824532| 10631990| NA|\n```\n\nIn the final GFF output, nested TEs are clearly labelled with a `nested=FULLY_NESTED` attribute in column 9, enabling quick identification and downstream filtering.\n\n```\nNC_045808.1\tEarl_Grey\tSimple_repeat\t51840\t51871\t14\t+\t.\tID=(CGCA)N_33;Name=(CGCA)N;tstart=1;tend=31;shortte=F;nested=FULLY_NESTED-ROUND1\n```\n\nAs always, thank you to the TE community for your enthusiasm in using Earl Grey, and for your invaluable feedback and bug reports. I’ll continue to incorporate improvements and fixes as quickly—and carefully—as possible.\n\nHappy New Year! 🎉\n\nEarl Grey v6.3.6 patches a RepeatCraft bug that arises extremely rarely in specific genomes, linked to dictionary initialisation. \n\nEarl Grey v6.3.5 patches the annotation only pipeline to use the correct divergence calculator when a custom library is used without a RepBase term.\n\nEarl Grey v6.3.4 adds small patches to correct TE family count table, which was previously using ID rather than Name. Also, a new awk one-liner has been added to clean the final GFF to make all attributes compliant with genometools, geneious etc.\n\nEarl Grey v6.3.3 adds small updates to improve user-friendliness when installing and running for the first time.\n\nEarl Grey v6.3.0 makes some important changes. Firstly, RepeatMasker now runs without checking for IS element contamination `-no_is`. In final GFF files, each insertion is now given a unique ID with `ID=`. The TE family is designated with `Name=`. This should enable parsing with Geneious or other visualisers. Other script changes are related to this change in field designation.\n\nEarl Grey v6.2.0 now includes better checkpoints for TEstrainer. In the event of an interrupted run, resubmit your `earlGrey` command with _exactly_ the same parameters and Earl Grey will skip successfully completed steps. With this new update, TEstrainer intermediate files will no longer be deleted and a new run started from scratch. Now, TEstrainer can be recovered mid-run to save time and resources.\n\nEarl Grey v6.1.1 patches a bug where threads set to <4 caused TEstrainer to crash (only present in 6.1.0 and not earlier versions)\n\nEarl Grey v6.1.0 reintroduces the `--curated` flag when known elements are used to pre-mask your input genome. As usual, unless a good quality TE library already exists for your species of interest, or a _very_ closely related one, we do not recommend pre-masking your input genome with known repeats. This reduces the amount of information available for _de novo_ annotation, and can lead to overestimations of TE divergence and lower-quality consensus sequences. If in doubt, leave this option out!\n\nEarl Grey v6.0.3 reduces CPU usage for TEstrainer to reduce memory pressure.\n\nEarl Grey v6.0.2 patches an issue where the use of existing libraries did not work with the new `famdb` formats. \n\nEarl Grey v6.0.1 contains small bug fixes to verify installed RepeatMasker Libraries correctly. There is now a Docker container for Earl Grey v6.0.1 that contains all partitions of Dfam (so it is BIG!).\n\n*Earl Grey v6.0.0 is here!* \n\nThere are some relatively large changes in this release, resulting in the jump to v6.0.0. \n\nImportantly, Earl Grey has been updated to use *Dfam version 3.9*, with RepeatMasker 4.1.8 and famdb 2.0.1. This means that there is some extra configuration required to get the pipeline running. Upon first installation and running of Earl Grey, the pipeline will check whether RepeatMasker has been configured with the correct Dfam database partitions. If not, it will warn you, generate a script that you can modify and run to configure RepeatMasker, and provide instructions to `stdout` if you want to do this yourself.\n\nPlease take care to configure Earl Grey v6 with ALL required partitions. More information on the partitioning can be found at [Dfam.org](https://dfam.org/releases/current/families/FamDB/README.txt).\n\nEarl Grey v5.1.1 will continue to work for those who are happy with Dfam v3.7, but we recommend upgrading to v6.0.0 to keep up to date with the latest improvements to the database.\n\nEarl Grey v5.1.1 contains very small patches to improve compatibility with publicly available genome sequencing data. In rare instances, strange characters in fasta headers were causing issues preventing the pipeline from running. These have been resolved in the preparation step. \n\nIn addition, new _pretty_ tables are now generated in the `summaryFiles` directory and at the end of a successful run. These contain the same information as the `txt` tables, but in the familiar pipe format for readability. These can be added to markdown files if required. One is produced for the high level count as well as for the family level count. Below is an example of the table that is printed at the end of Earl Grey runs as of `v5.1.1`:\n\n```\n|TE Classification | Coverage (bp)| Copy Number| % Genome Coverage| Genome Size| TE Family Count|\n|:------------------------------------------|-------------:|-----------:|-----------------:|-----------:|---------------:|\n|DNA | 69284| 190| 0.6516560| 10631990| 9|\n|Rolling Circle | 71788| 316| 0.6752076| 10631990| 12|\n|Penelope | 139334| 869| 1.3105167| 10631990| 6|\n|LINE | 141841| 491| 1.3340964| 10631990| 22|\n|SINE | 44137| 191| 0.4151339| 10631990| 1|\n|Other (Simple Repeat, Microsatellite, RNA) | 191645| 4235| 1.8025318| 10631990| 940|\n|Unclassified | 155836| 905| 1.4657275| 10631990| 35|\n|Non-Repeat | 9818125| NA| 92.3451301| 10631990| NA|\n```\n\nEarl Grey v5.1.0 contains small changes that drastically improve memory usage in the divergence calculator. We have replaced the use of EMBOSS `water` with EMBOSS `matcher`, which reduces memory consumption on large alignments whilst remaining rigorous. For more information, please see the notes section in the [EMBOSS Manual](https://www.bioinformatics.nl/cgi-bin/emboss/help/matcher). This should prevent jobs running out of memory, particularly when using queuing systems and shared resources. \n\nBig changes in the latest release!\n\n*Earl Grey v5.0.0 is here!* \n\nThis release incorporates the incremental improvements made throughout the life of version 4. \n\nIt is now possible to run some subroutines in Earl Grey (run either of these new commands with `-h` to see a list of options):\n- `earlGreyLibConstruct` can be used to run Earl Grey for _de novo_ TE detection, consensus generation, and improvement through the BEAT process. The output will be the strained TE consensus sequences, which can then be used for subsequent annotation. This is useful when you want to make a combined library from the libraries of several different genomes, where it is no longer required to waste time running annotations. Once the libraries are generated and you have curated them, you can then run the next step in isolation (next point!).\n- `earlGreyAnnotationOnly` can be used to run the final annotation and defragmentation steps in Earl Grey. This is useful if you have already run the BEAT process and have a library of _de novo_ TE consensus sequences that you would like to use to annotate a given genome. This script is also compatible with the `-r` flag to take known repeats from the databases used to configure RepeatMasker in addition to the custom repeat library.\n- *EXPERIMENTAL FEATURE:* I have also added an option to run [HELIANO](https://github.com/Zhenlisme/heliano) for improved detection of Helitrons, which are notoriously difficult to detect and classify using homology methods. This can be implemented by adding `-e yes` to the command line options after upgrading to v5.0.0. Currently, HELIANO annotations replace those which they overlap following the RepeatMasker run, which is performed during defragmentation (in a similar way to full-length LTRs being dealt with in `RepeatCraft`). Feedback is welcomed on this implementation, and I am continuing to test and improve the implementation of HELIANO within Earl Grey.\n- The settings used for HELIANO are: `--nearest -dn 6000 -flank_sim 0.5 -w 10000`. These can be modified in the earlGrey script of your specific installation.\n\nThank you for your continued support and enthusiasm for Earl Grey!\n\n# Example\n\nGiven an input genome, Earl Grey will run through numerous steps to identify, curate, and annotate transposable elements (TEs). We recommend running earlGrey within a tmux or screen session, so that you can log off and leave Earl Grey running.\n\nThere are several required and optional parameters for your Earl Grey run:\n```\nRequired Parameters:\n\t\t-g == genome.fasta\n\t\t-s == species name\n\t\t-o == output directory\n\n\tOptional Parameters:\n\t\t-t == Number of Threads (DO NOT specify more than are available)\n\t\t-r == RepeatMasker search term (e.g arthropoda/eukarya)\n\t\t-l == Starting consensus library for an initial mask (in fasta format)\n\t\t-i == Number of Iterations to BLAST, Extract, Extend (Default: 10)\n\t\t-f == Number flanking basepairs to extract (Default: 1000)\n\t\t-c == Cluster TE library to reduce redundancy? (yes/no, Default: no)\n\t\t-m == Remove putative spurious TE annotations <100bp? (yes/no, Default: no)\n\t\t-d == Create soft-masked genome at the end? (yes/no, Default: no)\n\t\t-n == Max number of sequences used to generate consensus sequences (Default: 20)\n\t\t-a == minimum number of sequences required to build a consensus sequence (Default: 3)\n -e == Optional: Run HELIANO for detection of Helitrons (yes/no, Default: no)\n\t\t-h == Show help\n```\n\n```\n# remember to activate the conda environment before running (NOTE: depending on your install, the environment name might vary)\n\nconda activate earlgrey\n\n# run earl grey with minimum command options\n\nearlGrey -g [genome.fasta] -s [speciesName] -o [outputDirectory]\n\n# e.g\n\nearlGrey -g myzusPersicae.fasta -s myzusPersicae -o ./earlGreyOutputs\n\n```\n\nFollowing this, Earl Grey will run through several processes depending on the options selected. The general pipeline is illustrated below:\n\n\n\nFor a more in-depth description of Earl Grey's steps, please refer to the implementation section in the [manuscript](https://academic.oup.com/mbe/article/41/4/msae068/7635926).\n\nThe runtime of Earl Grey will depend on the repeat content of your input genome. Once finished, you will notice that a number of directories have been created by Earl Grey. The most important results are found within the \"summaryFiles\" folder, however intermediate results are kept in case you wish to use alignments for further manual curation or investigation, for example. NOTE: RepeatModeler2 remains the rate-limiting step, with runtimes leading into several days, or even weeks, with large repeat-rich genomes. This is normal. \n\nDirectories created by earl grey:\n```\n[speciesName]EarlGrey/\n |\n |---[speciesName]_RepeatMasker/ [OPTIONAL: only created with -r or -l flags]\n |\t\t+ Results of the initial RepeatMasker run used to pre-mask previously characterised TEs\n |---[speciesName]_Database/\n |\t\t+ BLAST database built from the (optionally pre-masked) genome, used by RepeatModeler\n |\t\t+ RepeatModeler2 consensus output ([speciesName]-families.fa, [speciesName]-families.stk, [speciesName]-rmod.log)\n |---[speciesName]_RepeatModeler/\n |\t\t+ RepeatModeler2 working directory (RM_* subdirectory with per-round outputs and intermediate consensi)\n |---[speciesName]_strainer/\n |\t\t+ TEstrainer working directory (TS_* subdirectory) containing BEAT curation intermediate files\n |\t\t+ Strained consensus library ([speciesName]-families.fa.strained)\n |---[speciesName]_Curated_Library/ [OPTIONAL: only created with -r or -l flags]\n |\t\t+ Known repeat library extracted from the configured database ([speciesName].RepeatMasker.lib)\n |\t\t+ Combined _de novo_ and known repeat library ([speciesName]_combined_library.fasta)\n |---[speciesName]_heliano/ [OPTIONAL: only created with -e yes flag]\n |\t\t+ HELIANO Helitron detection results (HEL_* subdirectory)\n |---[speciesName]_RepeatMasker_Against_Custom_Library/\n |\t\t+ Results of RepeatMasker run against the final curated repeat library (.out, .tbl, .cat.gz, .masked)\n |---[speciesName]_RepeatLandscape/\n |\t\t+ Divergence-annotated GFF ([speciesName].filteredRepeats.withDivergence.gff)\n |\t\t+ Repeat landscape PDFs (classification landscape, split-class landscape, superfamily divergence plot)\n |\t\t+ Divergence summary table ([speciesName]_summary_table.tsv)\n |---[speciesName]_mergedRepeats/\n |\t\t+ looseMerge/ subdirectory containing:\n |\t\t - LTR_FINDER full-length LTR results (ltrfinder GFF and working subdirectory)\n |\t\t - RepeatCraft TE defragmentation output (RepeatCraft working subdirectory, merged GFFs and BEDs)\n |\t\t - Final filtered and merged repeat annotations (filteredRepeats.bed, filteredRepeats.gff, filteredRepeats.summary)\n |---[speciesName]_summaryFiles/\n \t\t+ Final TE annotations in GFF3 format ([speciesName].filteredRepeats.gff)\n \t\t (fully nested TEs labelled with 'nested=FULLY_NESTED' attribute in column 9)\n \t\t+ Final TE annotations in BED format ([speciesName].filteredRepeats.bed)\n \t\t+ High-level TE quantification table (tab-delimited .txt and markdown-formatted .kable)\n \t\t+ Family-level TE quantification table (tab-delimited .txt and markdown-formatted .kable)\n \t\t+ Divergence summary table ([speciesName]_divergence_summary_table.tsv)\n \t\t+ Pie chart of genome repeat content ([speciesName].summaryPie.pdf)\n \t\t+ Repeat landscape PDFs (classification landscape, split-class landscape, superfamily divergence plot)\n \t\t+ _de novo_ strained TE library ([speciesName]-families.fa.strained)\n \t\t+ Combined repeat library ([speciesName]_combined_library.fasta) [OPTIONAL: only with -r or -l flags]\n \t\t+ Soft-masked genome FASTA ([speciesName].softmasked.fasta) [OPTIONAL: only with --softMask yes]\n```\n\nAs of `v4.4.5`, there is an option to generate _de novo_ TE libraries without running subsequent annotation. To run this option, use `earlGreyLibConstruct` with the same command-line options as `earlGrey`. This will run everything up to the end of TEstrainer, and leave you a `families-fa.strained` library file in the `summaryFiles` directory, which you can then use for manual curation, or for pangenome studies.\n\n### Example Outputs (NOTE: example data has been used here):\n\n- Pie chart summarising TE content in input genome\n\n\n\n- RepeatLandscapes summarising relative TE activity using Kimura 2-Parameter Divergence (recent activity towards the RHS)\n\n![\"Screenshot](\"https://github.com/user-attachments/assets/5e1b18b6-a84a-47e9-bc6d-6f341a2297ec\")
\n\n- TE annotations - These are in standard genomic information formats to be compatible with downstream analyses. NOTE: TE divergence calculated using Kimura 2-Parameter distance is now supplied for each insertion in column 9 of the GFF3 file:\n\n```\n# BED format\nNC45808.1 4964941 4965925 LINE/Penelope 5073 +\nNC45808.1 7291353 7291525 LINE/L2 1279 +\nNC_045808.1 8922477 8923791 DNA/TcMar-Tc1 11957 +\n\n\n# GFF3 format\nscaffold_1\tEarl_Grey\tDNA/Mariner\t71618\t71814\t892\t+\t.\tTSTART=13;TEND=233;ID=RND-1_FAMILY-1;SHORTTE=F;KIMURA80=0.2141\nscaffold_1\tEarl_Grey\tUnknown\t81757\t81927\t785\t-\t.\tTSTART=16;TEND=194;ID=RND-1_FAMILY-0;SHORTTE=F;KIMURA80=0.2037\n```\n\n# References and Acknowledgements\n\nThis pipeline has been designed to be used and shared openly by the community.\n\n### When using Earl Grey, please cite:\n\nBaril, T., Galbraith, J.G., and Hayward, A., Earl Grey: A Fully Automated User-Friendly Transposable Element Annotation and Analysis Pipeline, Molecular Biology and Evolution, Volume 41, Issue 4, April 2024, msae068 [doi:10.1093/molbev/msae068](https://doi.org/10.1093/molbev/msae068)\n\nBaril, Tobias., Galbraith, James., and Hayward, Alexander. (2023) Earl Grey. Zenodo [doi:10.5281/zenodo.5654615](https://doi.org/10.5281/zenodo.5654615)\n\n### This pipeline makes use of scripts from:\n\n[RepeatCraft](https://github.com/niccw/repeatcraftp) - Wong WY, Simakov O. RepeatCraft: a meta-pipeline for repetitive element de-fragmentation and annotation. Bioinformatics 2018;35:1051–2. https://doi.org/10.1093/bioinformatics/bty745.\n\n### The following open source software are utilised by this pipeline:\n\nSmit AFA, Hubley RR, Green PR. RepeatMasker Open-4.0. Http://RepeatmaskerOrg 2013.\n\nFlynn, J.M., Hubley, R., Goubert, C., Rosen, J., Clark, A.G., Feschotte, C. and Smit, A.F. RepeatModeler2 for automated genomic discovery of transposable element families. Proceedings of the National Academy of Sciences 2020;17;9451-9457. https://doi.org/10.1073/pnas.1921046117\n\nBao Z, Eddy SR. Automated De Novo Identification of Repeat Sequence Families in Sequenced Genomes. Genome Res 2002;12:1269–76. https://doi.org/10.1101/gr.88502.\n\nPrice AL, Jones NC, Pevzner PA. De novo identification of repeat families in large genomes. Bioinformatics 2005;21:i351–8.\n\nCamacho C, Coulouris G, Avagyan V, Ma N, Papadopoulos J, Bealer K, et al. BLAST+: Architecture and applications. BMC Bioinformatics 2009;10: 1–9. https://doi.org/10.1186/1471-2105-10-421.\n\nKatoh K, Standley DM. MAFFT multiple sequence alignment software version 7: Improvements in performance and usability. Mol Biol Evol 2013;30:772–80. https://doi.org/10.1093/molbev/mst010.\n\nCapella-Gutiérrez S, Silla-Martínez JM, Gabaldón T. trimAl: A tool for automated alignment trimming in large-scale phylogenetic analyses. Bioinformatics 2009;25:1972–3. https://doi.org/10.1093/bioinformatics/btp348.\n\nRice P, Longden L, Bleasby A. EMBOSS: The European Molecular Biology Open Software Suite. Trends Genet 2000;16:276–7. https://doi.org/10.1016/S0168-9525(00)02024-2.\n\nXu Z, Wang H. LTR_FINDER: an efficient tool for the prediction of full-length LTR retrotransposons. Nucleic Acids Res 2007;35:W265–8. https://doi.org/10.1093/nar/gkm286.\n\nOu S, Jiang N. LTR_FINDER_parallel: parallelization of LTR_FINDER enabling rapid identification of long terminal repeat retrotransposons. BioRxiv 2019:2–6.\n\n# Usage without installation\nIf you would like to try Earl Grey, or prefer to use it in a browser, you can do this through [gitpod](https://gitpod.io). You can get 50 hours free per month and use Earl Grey within a preconfigured environment. Simply select this repository by pasting the repository URL, and the system will be automatically configured for you. You can then upload your genome of interest, and run Earl Grey as you would on the command line. \n\n![\"Screenshot](\"https://github.com/TobyBaril/EarlGrey/assets/46785187/7dd2f2de-3c13-4553-b13a-007fdd8d94d6\")
\n\n# Recommended Installation with Conda or Mamba\n\nEarl Grey version 6 uses Dfam 3.9. After installation, you MUST configure Dfam partitions as needed. Earl Grey will generate the script to do this and provide guidance when you run it for the first time. You need to specify which partitions of Dfam and/or RepBase to configure Earl Grey with. Choose partitions carefully as the combination will highly influence your results, especially if you want to pre-mask your input genome.\n\nEarl Grey version 7.2.4 (latest stable release) with all required and configured dependencies is found in the `biooconda` conda channel. To install, simply run the following depending on your installation:\n```\n# With conda\nconda create -n earlgrey -c conda-forge -c bioconda earlgrey=7.2.4\n\n# With mamba\nmamba create -n earlgrey -c conda-forge -c bioconda earlgrey=7.2.4\n\n# Then run\nearlGrey\n\n# a script will be output to stdout and generated in the current directory to aid in setup\n```\n\n# Recommended Installation on ARM-based Mac Systems (M chips) Using Docker\n\nIt is possible to install x86-based Docker environments on M-chip Mac systems using Rosetta. The below form a guide that should work, but please reach out if you have any trouble!\n\nPlease first follow the Docker on Mac installation [instructions here](https://docs.docker.com/desktop/setup/install/mac-install/). Ensure you have installed Rosetta2, as this is required to get Earl Grey to behave as expected.\n\nNext, we will create aliases to switch between arm and rosetta:\n\nWe want a simple way to activate the x86 architecture. We can do this by adding the following to `~/.zshrc`:\n```\nalias arm=\"env /usr/bin/arch -arm64 /bin/zsh --login\"\nalias intel=\"env /usr/bin/arch -x86_64 /bin/zsh --login\"\n```\n\nThen close the terminal and open a new one.\n\nTo activate the intel/rosetta environment, run the following in a new terminal:\n```\nintel\n```\n\nNext, get the Docker installation and run in an interactive terminal following the instructions in the next section below. You will only need to pull the container once, then can use it for all your Earl Grey needs.\n\nAfter this, you are ready to go! Just remember to activate the _intel_ terminal before starting the interactive container and running Earl Grey. \n\n# Docker Container \n\nA Docker container has been generated with none of Dfam 3.9, but with script generation to source required partitions\n\nI try to keep an up-to-date container in docker hub, but this might not always be the case depending on if I have had time to build and upload a new image. Currently, the recommended image ready for use is `-nodfam` version. Upon running the container interactively and running the command `earlGrey`, instructions will print to `stdout` and a script that you can use will be placed in your current working directory. After an initial setup and configuration in an interative version of the container, you can commit the changes (i.e. the Dfam configuration) using `docker commit [container_ID] yourdockerusername/earlgrey:version7.2.4-configured`. Then, you can run this container interactively, or non-interatively, to annotate focal genomes.\n\n```\n# Interactive mode\n# Version 7.2.4 with no preconfigured partitions (RECOMMENDED!) - bind a directory, in my case the current directory using pwd\ndocker run -it -v 'pwd':/data/ tobybaril/earlgrey:latest-nodfam\n# change to library directory\ncd /data/\n# run earlGrey to make the configuration script\nearlGrey\n\n# then alter script with required partitions and run the configuration script\n# change 0-16 to whichever you require, but at least 0. This relates to the partitions of Dfam 3.9 (https://www.dfam.org/releases/Dfam_3.9/families/FamDB/)\n##### e.g. for 0-5:\nsed -i '/^curl/ s/0-16/0-5/g' configure_dfam39.sh\n##### e.g for 1,3,5:\nsed -i '/^curl/ s/0-16/1,3,5/g' configure_dfam39.sh\n\n# run the configuration script\nbash configure_dfam39.sh\n\n# return to your data directory\ncd /data/\n\n# to quit the container and leave it running so you can commit the configured changes\n# ctrl + p; ctrl + q\n\n# get container ID to commit (not necessarily required if you leave the container running. If you kill the container and start again, you would need to reconfigure the DFAM libraries, so this is recommended. However, the container will be BIG depending on which parts of DFAM you pull.\ndocker ps -a\n\n# commit the modified container so you can use at will (replace yourdockerusername with your docker username)\ndocker commit [container_ID] yourdockerusername/earlgrey:version7.2.4-configured\n\n# you can then run non-interatively if required:\ndocker run -v 'pwd':/data/ yourdockerusername/earlgrey:version7.2.4-configured earlGrey -g /data/GENOME.fasta -s nonInteractiveTest -o /data/ -t 8\n\n# alternatively you can still run interactive sessions\ndocker run -it -v 'pwd':/data/ yourdockerusername/earlgrey:version7.2.4-configured\n``` \n\n"
},
{
"path": "Singularity/README.md",
"content": "# Singularity image for Earl Grey\n\nA Singularity image can be built from the Docker image hosted on Docker Hub. The image contains no pre-configured Dfam partitions, but will generate a configuration script on first run of `earlGrey`. You will need to configure the Dfam libraries once, then you can use the configured sandbox for all subsequent analyses.\n\nWherever you start this is where your `/data` directory will point to inside the container. Make sure your genome assembly is found here, or change `$(pwd)` in the below commands to the directory where you want to perform your analysis.\n\n**Write access is required inside the container to download and configure the Dfam partitions.** The recommended approach is to build a writable sandbox, configure Dfam, and then optionally convert the sandbox to a portable SIF file.\n\n## First-time setup: building and configuring the sandbox\n\n```bash\n# Build a writable sandbox from the Docker image\nsingularity build --sandbox earlgrey_sandbox/ docker://tobybaril/earlgrey:latest-nodfam\n\n# Run the sandbox interactively with write access, binding your working directory to /data\nsingularity shell -C --bind $(pwd):/data --writable earlgrey_sandbox/\n\n# Inside the container: change to the data directory and run earlGrey to generate the configuration script\ncd /data/\nearlGrey\n\n# Modify the configuration script to select the required Dfam 3.9 partitions\n# (https://www.dfam.org/releases/Dfam_3.9/families/FamDB/)\n# At minimum, partition 0 is required. Change 0-16 to whichever partitions you need.\n##### e.g. for partitions 0-5:\nsed -i '/^curl/ s/0-16/0-5/g' configure_dfam39.sh\n##### e.g. for specific partitions 1, 3, and 5:\nsed -i '/^curl/ s/0-16/1,3,5/g' configure_dfam39.sh\n\n# Run the configuration script to download and install the selected Dfam partitions\nbash configure_dfam39.sh\n\n# Return to your data directory\ncd /data/\n\n# Exit the sandbox\nexit\n```\n\n## Running Earl Grey after configuration\n\n```bash\n# Run the configured sandbox interactively\nsingularity shell -C --bind $(pwd):/data --writable earlgrey_sandbox/\n\n# Or run non-interactively\nsingularity exec -C --bind $(pwd):/data --writable earlgrey_sandbox/ earlGrey -g /data/GENOME.fasta -s mySpecies -o /data/ -t 8\n```\n\n## Optional: convert the configured sandbox to a portable SIF\n\nOnce Dfam is configured, you can convert the sandbox to a read-only SIF file for easier distribution and deployment. Note that the resulting SIF will be large depending on which Dfam partitions were downloaded.\n\n```bash\nsingularity build earlgrey_configured.sif earlgrey_sandbox/\n\n# Run non-interactively with the configured SIF\nsingularity exec -C --bind $(pwd):/data earlgrey_configured.sif earlGrey -g /data/GENOME.fasta -s mySpecies -o /data/ -t 8\n```\n"
},
{
"path": "conda/build.sh",
"content": "#!/bin/bash\n#Based on https://github.com/TobyBaril/EarlGrey/blob/main/configure\nset -x\n\n# Define paths\nPACKAGE_HOME=${PREFIX}/share/${PKG_NAME}-${PKG_VERSION}-${PKG_BUILDNUM}\nSCRIPT_DIR=\"${PACKAGE_HOME}/scripts/\"\n\n# Create directories\nmkdir -p ${PREFIX}/bin\nmkdir -p ${PACKAGE_HOME}\n\n# Put package in share directory\ncp -rf * ${PACKAGE_HOME}/\n\n# Install SA-SSR (has to be done here because SA-SSR is an ancient repository without releases)\ngit clone --depth 1 https://github.com/TobyBaril/SA-SSR\ncd SA-SSR\nmake -j\"${CPU_COUNT}\"\ninstall -v -m 0755 bin/sa-ssr \"${PREFIX}/bin\"\ncd ../ && rm -rf SA-SSR/\n\n# Fixes to earlGrey executable\nsed -i.bak \"/CONDA_DEFAULT_ENV/,+4d\" ${PACKAGE_HOME}/earlGrey #remove check that conda environment has a specific name\nsed -i.bak \"/CONDA_DEFAULT_ENV/,+4d\" ${PACKAGE_HOME}/earlGreyLibConstruct #remove check that conda environment has a specific name\nsed -i.bak \"/CONDA_DEFAULT_ENV/,+4d\" ${PACKAGE_HOME}/earlGreyAnnotationOnly #remove check that conda environment has a specific name\n\n# Fixes sed command for executables so that it works on both linux and macos\nsed -i.bak \"s|sed -i |sed -i.bak |g\" ${PACKAGE_HOME}/earlGrey ${PACKAGE_HOME}/earlGreyLibConstruct ${PACKAGE_HOME}/earlGreyAnnotationOnly ${SCRIPT_DIR}/rcMergeRepeat* ${SCRIPT_DIR}/TEstrainer/TEstrainer_for_earlGrey.sh\n\n# Remove -pa from RepeatClassifier\nsed -i.bak 's/RepeatClassifier -pa ${THREADS} /RepeatClassifier /' ${SCRIPT_DIR}/TEstrainer/TEstrainer\n\n# Remove -t parameter from sa-ssr (since multithreading doesn't work on OSX)\nsed -i.bak 's/-t ${THREADS} / /' ${SCRIPT_DIR}/TEstrainer/TEstrainer_for_earlGrey.sh\nsed -i.bak 's/-t ${THREADS} / /' ${SCRIPT_DIR}/TEstrainer/TEstrainer\n\n# Add SCRIPT_DIR to correct path\nsed -i.bak \"s|SCRIPT_DIR=.*|SCRIPT_DIR=${SCRIPT_DIR}|g\" ${PACKAGE_HOME}/earlGrey\nsed -i.bak \"s|SCRIPT_DIR=.*|SCRIPT_DIR=${SCRIPT_DIR}|g\" ${PACKAGE_HOME}/earlGreyLibConstruct\nsed -i.bak \"s|SCRIPT_DIR=.*|SCRIPT_DIR=${SCRIPT_DIR}|g\" ${PACKAGE_HOME}/earlGreyAnnotationOnly\nsed -i.bak \"s|SCRIPT_DIR=.*|SCRIPT_DIR=${SCRIPT_DIR}|g\" ${SCRIPT_DIR}/rcMergeRepeat*\nsed -i.bak \"s|SCRIPT_DIR=.*|SCRIPT_DIR=${SCRIPT_DIR}|g\" ${SCRIPT_DIR}/headSwap.sh\nsed -i.bak \"s|SCRIPT_DIR=.*|SCRIPT_DIR=${SCRIPT_DIR}|g\" ${SCRIPT_DIR}/autoPie.sh\nsed -i.bak \"s|STRAIN_SCRIPTS=.*|STRAIN_SCRIPTS=${SCRIPT_DIR}/TEstrainer/scripts/|g\" ${SCRIPT_DIR}/TEstrainer/TEstrainer_for_earlGrey.sh\n\n# Set permissions to files\nchmod +x ${PACKAGE_HOME}/earlGrey\nchmod +x ${PACKAGE_HOME}/earlGreyLibConstruct\nchmod +x ${PACKAGE_HOME}/earlGreyAnnotationOnly\nchmod +x ${SCRIPT_DIR}/TEstrainer/TEstrainer_for_earlGrey.sh\nchmod +x ${SCRIPT_DIR}/* > /dev/null 2>&1\nchmod +x ${SCRIPT_DIR}/bin/LTR_FINDER.x86_64-1.0.7/ltr_finder\nchmod a+w ${SCRIPT_DIR}/repeatCraft/example\n\n# Extract tRNAdb\ntar -zxf ${SCRIPT_DIR}/bin/LTR_FINDER.x86_64-1.0.7/tRNAdb.tar.gz --directory ${SCRIPT_DIR}/bin/LTR_FINDER.x86_64-1.0.7 && rm -r ${SCRIPT_DIR}/bin/LTR_FINDER.x86_64-1.0.7/tRNAdb.tar.gz\n\n# test for conda\ndf -h\n\n# Set PERL5LIB upon activate/deactivate\nfor CHANGE in \"activate\" \"deactivate\";\ndo\n mkdir -p \"${PREFIX}/etc/conda/${CHANGE}.d\"\ndone\necho \"#!/bin/sh\" > \"${PREFIX}/etc/conda/activate.d/${PKG_NAME}_activate.sh\"\necho \"export PERL5LIB=${PREFIX}/share/RepeatMasker/:${PREFIX}/share/RepeatModeler/\" >> \"${PREFIX}/etc/conda/activate.d/${PKG_NAME}_activate.sh\"\necho \"#!/bin/sh\" > \"${PREFIX}/etc/conda/deactivate.d/${PKG_NAME}_deactivate.sh\"\necho \"unset PERL5LIB\" >> \"${PREFIX}/etc/conda/deactivate.d/${PKG_NAME}_deactivate.sh\"\n\n# Put earlGrey executable in bin\ncd ${PREFIX}/bin\nln -sf ${PACKAGE_HOME}/earlGrey .\nln -sf ${PACKAGE_HOME}/earlGreyLibConstruct .\nln -sf ${PACKAGE_HOME}/earlGreyAnnotationOnly ."
},
{
"path": "conda/developmentNotes.md",
"content": "# Updating Earl Grey to remove Python 3.9 dependency\n\nBjorn wants me to update Earl Grey to remove the Python 3.9 dependency. This is because Python 3.9 is no longer supported and we want to ensure that Earl Grey can run on newer versions of Python.\n\nI will build Earl Grey with the meta dependency removed and test it to ensure that it still works correctly. I will also update the documentation to reflect the change in dependencies.\n\nBuild the conda package:\n\n```bash\nconda build .\n```\n\nMake a new conda environment and install the package:\n\n```bash\nconda create -n earlgrey_python_test -c conda-forge -c bioconda earlgrey --use-local\nconda activate earlgrey_python_test\n```\n\nThis still builds with 3.9. I will force it to build with 3.10 by adding `python >=3.10` to the `run` dependencies in `meta.yaml`. After making this change, I will rebuild the package and test it again. I also removed the dependency on `ncls` version 0.0.64 and replaced it with `ncls` without a specific version to allow for newer versions of `ncls` to be used.\n\n```bash\nconda build .\nconda create -n earlgrey_python_test -c conda-forge -c bioconda earlgrey --use-local\nconda activate earlgrey_python_test\n```\n\nLink the dfam libraries and configure RepeatMasker:\n\n```bash\nln -sf ln -sf /data/toby/tools/earlgrey_databases/Libraries/famdb/* \\\n /data/toby/miniforge3/envs/earlgrey_python_test/share/RepeatMasker/Libraries/famdb/\n\ncd /data/toby/miniforge3/envs/earlgrey_python_test/share/RepeatMasker\nperl configure\n```\n\nRun a test command to ensure that Earl Grey is working correctly:\n\n```bash\ncd /data/toby/testDIR/\nearlGrey -g /data/toby/tools/test.fasta -s test_python312 -o . -t 16\n```\n\nThis passed with a small fix to the `backSwap.py` script where I changed the regular expression for splitting the input file from `'\\s+'` to `r'\\s+'` to ensure that it correctly handles whitespace. After making this change, I will test the script again to ensure that it works correctly with the new dependencies.\n\n```bash\ncd /data/toby/testDIR/\nearlGrey -g /data/toby/testDIR/saturationTests/1_4genomesNoRepMasker/1A5.fa -s test_python312_2 -o . -t 32\n```\n\n---\n\n# Reducing peak RAM usage in TEstrainer and divergence_calc.py\n\n## Background\n\nTwo components were identified as disproportionately RAM-intensive when run with large thread counts, leading to OOM (out-of-memory) kills on compute nodes with limited RAM:\n\n- **TEstrainer** / **TEstrainer_for_earlGrey.sh** — the GNU parallel jobs dispatched during the BEET curation loop (trf, initial_mafft_setup.py, mafft, TEtrim.py) had no memory back-pressure. With 32–64 threads all jobs fired simultaneously, each spawning a Python interpreter that imports `pandas`, `numpy`, `pyranges` and `BioPython` (~300–600 MB each at startup), along with concurrent `--localpair` mafft alignments.\n- **divergence_calc.py** — used `multiprocessing.Pool` with the default `fork` start method. On Linux this fork-copies the entire parent address space (including the full GFF DataFrame, which can be multiple GB for large genomes) into every worker process. At 32 cores this means 32 nearly-identical copies of the GFF in RAM simultaneously, even though each worker only needs its small chunk.\n\n## Changes made\n\n### `scripts/TEstrainer/TEstrainer` and `scripts/TEstrainer/TEstrainer_for_earlGrey.sh`\n\n**1. Default `MEM_FREE` raised from `200M` to `1G`**\n\nThe previous default of 200 MB was smaller than the memory footprint of a single Python interpreter after importing the heavy scientific libraries used by `initial_mafft_setup.py` and `TEtrim.py`. This meant `--memfree 200M` gave essentially no throttling protection. 1 GB is a conservative lower bound that reflects real startup costs.\n\n**2. RAM-cap guard added after argument parsing**\n\nA block querying `/proc/meminfo` via `free -m` is inserted immediately after `MAFFT_THREADS` is calculated. If the requested thread count exceeds what is safely supportable (at an estimate of 800 MB per concurrent job), `THREADS` (and `MAFFT_THREADS`) are silently capped with a warning message to stderr. The 800 MB estimate is conservative; real usage per job is closer to 300–500 MB for typical inputs but spikes during mafft's `--localpair` DP phase.\n\n```bash\nAVAIL_MEM_MB=$(free -m | awk '/^Mem:/{print $7}')\nMEM_PER_JOB_MB=800\nMAX_SAFE_THREADS=$(( AVAIL_MEM_MB / MEM_PER_JOB_MB ))\nif [[ $MAX_SAFE_THREADS -gt 0 && $THREADS -gt $MAX_SAFE_THREADS ]]; then\n echo \"Warning: capping threads from ${THREADS} to ${MAX_SAFE_THREADS} based on available RAM (${AVAIL_MEM_MB} MB free)\"\n THREADS=$MAX_SAFE_THREADS\n if [[ $THREADS -gt 4 ]]; then MAFFT_THREADS=$(($(($THREADS / 4)))); else MAFFT_THREADS=1; fi\nfi\n```\n\n**3. `--memfree ${MEM_FREE}` added to all GNU parallel calls that were missing it**\n\nIn the standalone `TEstrainer`, the four `parallel` calls inside the BEET curation loop (trf, initial_mafft_setup, mafft, TEtrim) were missing `--memfree`. In `TEstrainer_for_earlGrey.sh`, the mafft and TEtrim calls were missing it (the others already had it). GNU parallel's `--memfree` halts job dispatch when free system RAM drops below the threshold, queuing further jobs until a running job finishes and memory is reclaimed. This is the most direct mechanism for preventing simultaneous over-subscription.\n\n### `scripts/divergenceCalc/divergence_calc.py`\n\n**1. `forkserver` multiprocessing start method**\n\n```python\nmultiprocessing.set_start_method('forkserver')\n```\n\nOn Linux the default is `fork`, which duplicates the entire parent address space into every worker via `clone()`. With a multi-GB GFF DataFrame in memory this creates N near-identical copies. `forkserver` instead spawns a minimal server process at startup; each worker is forked from that clean server and receives only the data it actually needs (the chunk file path string) via IPC. This is the primary RAM reduction mechanism.\n\n**2. GFF DataFrame serialised to per-chunk TSV files before pool creation**\n\nPreviously the chunk DataFrames were passed to `pool.map()` directly — they were pickled in the parent and unpickled in each worker. With `forkserver` this already avoids the fork-copy problem, but the parent still holds the full `in_gff` DataFrame in memory throughout the entire pool run. The new approach serialises each chunk to a temp TSV on disk, then deletes `in_gff` and `chunks` from the parent before the pool is even created:\n\n```python\nchunk_files = []\nfor i, chunk in enumerate(chunks):\n chunk_path = os.path.join(args.temp_dir, f\"chunk_{i}.tsv\")\n chunk.to_csv(chunk_path, sep=\"\\t\", index=True)\n chunk_files.append(chunk_path)\n\ndel chunks\ndel in_gff\n```\n\nWorkers then receive a file path string (not a DataFrame), read the chunk at the start of `outer_func`, and immediately delete the file. Parent peak RAM during the pool run is reduced to `simple_gff + other_gff + Python overhead` rather than `full_gff + simple_gff + other_gff`.\n\n**3. `outer_func` signature updated and `pybedtools.set_tempdir` moved inside worker**\n\nThe first argument to `outer_func` is now the chunk file path. `pybedtools.set_tempdir()` is now called at the top of `outer_func` rather than only in the parent. This is required for `forkserver` workers, which do not inherit the parent's in-memory pybedtools state.\n\n**4. Periodic `pybedtools.cleanup()` inside `outer_func`**\n\n```python\nif row_counter % 500 == 0:\n pybedtools.cleanup(remove_all=False)\n```\n\nEach row creates a pybedtools temp file. In a long-running worker processing thousands of rows, these accumulate in the temp directory and inflate virtual memory. Calling `cleanup(remove_all=False)` every 500 rows removes files that are no longer referenced by any active Python object, releasing both disk and virtual-memory pressure without affecting open handles.\n\n**5. `imap_unordered` instead of `pool.map`**\n\n```python\nresults = list(pool.imap_unordered(func, chunk_files))\n```\n\n`pool.map` buffers all results in memory until every worker finishes. `imap_unordered` yields results as each worker completes. Since `outer_func` returns only a file path string this has negligible direct memory impact, but it allows the Python runtime to close completed worker processes and release their memory sooner when combined with `maxtasksperchild`.\n\n**6. `maxtasksperchild=1`**\n\n```python\npool = multiprocessing.Pool(processes=num_processes, maxtasksperchild=1)\n```\n\nForces each worker process to exit and be replaced after processing one chunk. This ensures that any memory accumulated during a chunk run (pybedtools internal state, BioPython caches, subprocess residuals) is released to the OS at chunk boundaries rather than accumulating across the lifetime of the pool.\n\n---\n\n## Tests\n\nThe following tests should be run to verify correctness and confirm the memory ceiling has been reduced. Run them from the root of the EarlGrey repo unless stated otherwise.\n\n### Dependencies for memory monitoring\n\n```bash\n# Install memory_profiler if not already present\npip install memory_profiler psutil\n\n# /usr/bin/time -v is available on Linux via the 'time' package\nwhich /usr/bin/time || sudo apt-get install -y time\n```\n\n---\n\n### Test 1 — Static audit: verify `--memfree` presence in all parallel calls\n\nConfirms no `parallel` call in either TEstrainer script is dispatching jobs without a memory guard.\n\n```bash\necho \"\"\necho \"=== TEstrainer_for_earlGrey.sh ===\"\ngrep -n 'parallel' scripts/TEstrainer/TEstrainer_for_earlGrey.sh | grep -v '#'\n```\n\n**Expected:** Every line containing `parallel --bar` should also contain `--memfree`.\n\n---\n\n### Test 2 — Static audit: verify MEM_FREE defaults\n\n```bash\ngrep 'MEM_FREE=' scripts/TEstrainer/TEstrainer_for_earlGrey.sh\n```\n\n**Expected:** Both lines should read `MEM_FREE=\"1G\"`.\n\n---\n\n### Test 3 — Static audit: verify forkserver and chunk-file pattern in divergence_calc.py\n\n```bash\ngrep -n 'set_start_method\\|chunk_path\\|chunk_files\\|imap_unordered\\|maxtasksperchild\\|pybedtools.cleanup' \\\n scripts/divergenceCalc/divergence_calc.py\n```\n\n**Expected output should contain all of:**\n- `set_start_method('forkserver')`\n- `chunk_path` (argument to `outer_func`)\n- `chunk_files` (list built in `__main__`)\n- `imap_unordered`\n- `maxtasksperchild=1`\n- `pybedtools.cleanup`\n\n---\n\n### Test 4 — RAM-cap guard: unit test on the bash logic\n\n```bash\n# Simulate a machine with 4 GB available and a 64-thread request.\n# The guard should cap THREADS to (4096 / 800) = 5.\nbash -c '\n THREADS=64\n AVAIL_MEM_MB=4096\n MEM_PER_JOB_MB=800\n MAX_SAFE_THREADS=$(( AVAIL_MEM_MB / MEM_PER_JOB_MB ))\n if [[ $MAX_SAFE_THREADS -gt 0 && $THREADS -gt $MAX_SAFE_THREADS ]]; then\n echo \"CAPPED: THREADS set to ${MAX_SAFE_THREADS}\"\n THREADS=$MAX_SAFE_THREADS\n else\n echo \"NOT CAPPED: THREADS remains ${THREADS}\"\n fi\n echo \"Final THREADS: ${THREADS}\"\n'\n```\n\n**Expected:**\n```\nCAPPED: THREADS set to 5\nFinal THREADS: 5\n```\n\n---\n\n### Test 5 — Correctness: divergence_calc.py output matches reference\n\nGenerate a reference output using the old code path on a small test GFF, then confirm the new code produces identical output. A pre-existing completed EarlGrey run in `testDIR` can supply the required files.\n\n```bash\n# Adjust paths to a completed EarlGrey run with a .gff and matching library\nTESTDIR=/data/toby/testDIR/condaPull\nGENOME=/data/toby/testDIR/test.fasta\nGFF=/data/toby/testDIR/condaPull/genome1_EarlGrey/genome1_mergedRepeats/looseMerge/genome1.filteredRepeats.gff\nLIB=/data/toby/testDIR/condaPull/genome1_EarlGrey/genome1_strainer/genome1-families.fa.strained\n\n# Run with the updated script\npython scripts/divergenceCalc/divergence_calc.py \\\n -l \"${LIB}\" \\\n -i \"${GFF}\" \\\n -g \"${GENOME}\" \\\n -o /tmp/divergence_new.gff \\\n -tmp /tmp/divtest_new/ \\\n -t 4\n\necho \"Exit code: $?\"\necho \"Output lines: $(wc -l < /tmp/divergence_new.gff)\"\n```\n\n**Expected:** Zero exit code, output line count matches the input GFF line count, all lines end with `;KIMURA80=`.\n\n```bash\n# Verify all non-simple-repeat lines carry a KIMURA80 tag\ngrep -v 'Simple_repeat\\|Satellite\\|Low_complexity' /tmp/divergence_new.gff | \\\n grep -v 'KIMURA80' | wc -l\n```\n\n**Expected:** 0 for a pure RepeatMasker/Earl Grey GFF. Annotations from other tools (e.g. HELIANO, which uses a different `tool` column value) are passed through unmodified by `parse_gff()` and will not carry a `KIMURA80` tag — one such line per non-RM/EG annotation is acceptable and expected.\n\n---\n\n### Test 6 — Memory ceiling: peak RSS comparison for divergence_calc.py\n\nMeasure peak RSS with different thread counts to confirm sub-linear scaling (the old `fork` behaviour would show near-linear scaling of the peak).\n\n```bash\nTESTDIR=/data/toby/testDIR/condaPull\nGENOME=/data/toby/testDIR/test.fasta\nGFF=/data/toby/testDIR/condaPull/genome1_EarlGrey/genome1_mergedRepeats/looseMerge/genome1.filteredRepeats.gff\nLIB=/data/toby/testDIR/condaPull/genome1_EarlGrey/genome1_strainer/genome1-families.fa.strained\n\nfor T in 1 4 8 16; do\n rm -rf /tmp/divmem_t${T}\n echo -n \"Threads=${T}: \"\n /usr/bin/time -v python scripts/divergenceCalc/divergence_calc.py \\\n -l \"${LIB}\" -i \"${GFF}\" -g \"${GENOME}\" \\\n -o /tmp/divmem_t${T}.gff -tmp /tmp/divmem_t${T}/ -t ${T} \\\n 2>&1 | grep 'Maximum resident'\ndone\n```\n\n**Expected:** Peak RSS should not scale linearly with thread count. With the old `fork` code, doubling the thread count roughly doubled peak RSS. With the new code, peak RSS should plateau once per-worker overheads are comparable to the chunk size — a significant reduction at 8+ threads on large GFFs.\n\nAs a rough guide, if the single-thread peak RSS is R MB, you should aim to see the 16-thread run use no more than ~2–3× R (library imports + 16 small chunks), compared to potentially 16× R with the old fork-based approach.\n\n**Observed results (test genome, condaPull/genome1):**\n\n| Threads | Peak RSS (kB) |\n|---------|--------------|\n| 1 | 75,992 |\n| 4 | 75,940 |\n| 8 | 75,920 |\n| 16 | 76,092 |\n\nPeak RSS is effectively flat across all thread counts (~76 MB). This confirms that `forkserver` workers start from a clean process and do not inherit the parent GFF DataFrame — the old `fork` implementation would have shown peak RSS scaling approximately linearly with thread count (potentially ~16× higher at 16 threads for a large GFF).\n\n---\n\n### Test 7 — Memory ceiling: peak RSS comparison for TEstrainer\n\n```bash\nTESTLIB=/data/toby/testDIR/condaPull/genome1_EarlGrey/genome1_strainer/genome1-families.fa.strained # use a small library for speed\nGENOME=/data/toby/testDIR/test.fasta\n\n# Baseline: 4 threads\n/usr/bin/time -v bash scripts/TEstrainer/TEstrainer_for_earlGrey.sh \\\n -l ${TESTLIB} -g ${GENOME} -t 4 -r 1 -d /tmp/ts_mem_t4 \\\n 2>&1 | grep 'Maximum resident'\n\n# Higher thread count: confirm MEM_FREE throttles correctly\n/usr/bin/time -v bash scripts/TEstrainer/TEstrainer_for_earlGrey.sh \\\n -l ${TESTLIB} -g ${GENOME} -t 32 -r 1 -d /tmp/ts_mem_t32 \\\n 2>&1 | grep 'Maximum resident'\n```\n\n**Expected:** The 32-thread run should emit a `Warning: capping threads` message if available RAM is insufficient, and peak RSS should not grow proportionally with the requested thread count.\n\n**Observed results (test genome, condaPull/genome1 library):**\n\n| Threads | Peak RSS (kB) |\n|---------|--------------|\n| 4 | 897,860 |\n| 32 | 891,420 |\n\nPeak RSS is effectively identical between 4 and 32 threads (~877 MB). Without `--memfree` throttling, the 32-thread run would dispatch all jobs simultaneously and peak RSS would scale with the number of concurrent Python interpreter startups. The flat result confirms that GNU parallel is correctly queuing jobs when free RAM drops below the `MEM_FREE` threshold, keeping concurrent memory usage stable regardless of the requested thread count.\n\n---\n\n### Test 8 — `--memfree` throttling behaviour\n\nVerify that GNU parallel actually pauses job dispatch when free RAM is low, rather than launching all jobs at once.\n\n```bash\n# Install stress-ng to consume a controllable amount of RAM\nwhich stress-ng || sudo apt-get install -y stress-ng\n\n# Consume ~80% of available RAM, then run TEstrainer with many threads.\n# parallel should pause on subsequent jobs when memfree drops below MEM_FREE.\nAVAIL=$(free -m | awk '/^Mem:/{print $7}')\nCONSUME_MB=$(( AVAIL * 80 / 100 ))\n\nstress-ng --vm 1 --vm-bytes ${CONSUME_MB}M --vm-keep &\nSTRESS_PID=$!\nsleep 3\n\n# Now attempt to run parallel with --memfree 1G and many small jobs.\n# Observe that parallel waits rather than dispatching all at once.\nseq 1 20 | parallel --jobs 8 --memfree 1G 'sleep 0.5 && echo job {} done'\n\nkill $STRESS_PID\n```\n\n**Expected:** The 20 sleep jobs complete without error. If free RAM is below 1 GB after stress-ng starts, parallel will hold back some jobs rather than launching all 20 simultaneously, and the output will appear staggered as memory is reclaimed.\n\n---\n\n### Test 9 — Worker isolation: confirm forkserver workers do not inherit parent GFF\n\nThis Python script directly tests that a worker process does not see the parent's `in_gff` variable, confirming the forkserver isolation is working.\n\n```python\n# save as /tmp/test_forkserver_isolation.py and run with: python /tmp/test_forkserver_isolation.py\nimport multiprocessing\nimport os\nimport pandas as pd\n\ndef worker_mem_check(_):\n \"\"\"Return RSS of this worker process in MB.\"\"\"\n import psutil\n proc = psutil.Process(os.getpid())\n return proc.memory_info().rss / 1024 / 1024\n\nif __name__ == \"__main__\":\n # Allocate a large DataFrame in the parent to simulate a big GFF.\n big_df = pd.DataFrame({'a': range(5_000_000), 'b': range(5_000_000)})\n parent_rss_mb = __import__('psutil').Process(os.getpid()).memory_info().rss / 1024 / 1024\n print(f\"Parent RSS after allocating large DataFrame: {parent_rss_mb:.0f} MB\")\n\n multiprocessing.set_start_method('forkserver')\n with multiprocessing.Pool(processes=4) as pool:\n worker_rss_values = pool.map(worker_mem_check, range(4))\n\n for i, rss in enumerate(worker_rss_values):\n print(f\"Worker {i} RSS: {rss:.0f} MB\")\n\n print(f\"\\nParent RSS: {parent_rss_mb:.0f} MB\")\n print(f\"Mean worker RSS: {sum(worker_rss_values)/len(worker_rss_values):.0f} MB\")\n print(f\"Ratio (parent/worker): {parent_rss_mb / (sum(worker_rss_values)/len(worker_rss_values)):.1f}x\")\n```\n\n**Expected:** Worker RSS values should be substantially smaller than the parent RSS (typically 5–20× smaller), confirming that workers do not inherit the large `big_df` from the parent. With the old `fork` start method, worker RSS would be approximately equal to parent RSS.\n\nTo contrast, change `set_start_method('forkserver')` to `set_start_method('fork')` and re-run — worker RSS values should roughly match the parent.\n\n**Observed results:**\n\n```\nParent RSS after allocating large DataFrame: 135 MB\nWorker 0 RSS: 54 MB\nWorker 1 RSS: 54 MB\nWorker 2 RSS: 54 MB\nWorker 3 RSS: 54 MB\n\nParent RSS: 135 MB\nMean worker RSS: 54 MB\nRatio (parent/worker): 2.5x\n```\n\nWorkers consume 54 MB (Python + library imports only) versus 135 MB in the parent (which includes the 5M-row DataFrame). The workers carry no copy of the parent's data, confirming `forkserver` isolation is working correctly. With the old `fork` start method all four workers would each have shown ~135 MB.\n\n---\n\n### Test 10 — End-to-end regression: full EarlGrey run\n\nA complete run to confirm that all components still produce correct outputs after the changes.\n\n```bash\ncd /data/toby/testDIR/\nearlGrey -g test.fasta -s mem_regression_test -o . -t 16\n```\n\n**Expected:** Run completes without error. Output directory `mem_regression_test_EarlGrey/` is populated with the expected files including `*-families.fa`, `*.gff`, and `*.divergence.pdf`. Compare output against a previous known-good run to confirm no regressions in repeat library composition or divergence values.\n\n**Observed:** Run completed without error (exit code 0). All memory-reduction changes are confirmed compatible with a full end-to-end EarlGrey pipeline run.\n\nThis also passed successfully, confirming that Earl Grey is now compatible with Python 3.10 and does not have a dependency on Python 3.9. I will now update the documentation to reflect these changes and ensure that users are aware of the new dependencies when installing Earl Grey.\n\n---\n\n# Version 7.2.3 — bug fixes, robustness improvements, and `-q` quiet-bar flag\n\n## Background\n\nSeveral bugs were found and fixed after the v7.2.0 memory-refinement release. The changes also add a new `-q` flag to `earlGrey` to suppress the GNU parallel progress bar, which is useful when running jobs through batch schedulers (SLURM `sbatch`, PBS, etc.) where the ANSI escape codes written by `--bar` corrupt log files.\n\n## Changes made\n\n### `conda/meta.yaml`\n\n- Bumped version to `7.2.3`.\n- Added `numpy` to the `run` dependencies. `divergence_calc.py` now imports `numpy` directly (for `np.array_split`), so it must be declared explicitly.\n- Changed `source` from a remote tarball + SHA256 to `path: ..` to allow local conda builds during development without needing a published release tarball.\n\n### `earlGrey` (main pipeline script)\n\n**1. New `-q` flag to suppress the TEstrainer parallel progress bar**\n\n```bash\nearlGrey -g genome.fa -s myspecies -o outdir -t 16 -q yes\n```\n\nWhen `-q yes` is passed, `earlGrey` appends `-q` to the `TEstrainer_for_earlGrey.sh` invocation. This propagates to every GNU `parallel --bar` call inside TEstrainer, replacing `--bar` with nothing so the progress bar is omitted. Useful for `sbatch` jobs where `--bar` writes ANSI codes into log files.\n\n**2. Fixed `AF_UNIX` socket path-too-long crash in `divergence_calc.py`**\n\n`pybedtools.set_tempdir()` sets `tempfile.tempdir` globally, which is also used by `forkserver` as the base directory for its `AF_UNIX` socket. If the EarlGrey output path is deeply nested, the socket path can exceed the 108-character kernel limit, causing an `OSError: AF_UNIX path too long`. The fix creates a short per-run temp directory under `/tmp`:\n\n```bash\ndivcalc_tmp=\"/tmp/egdiv_${species}\"\nmkdir -p \"${divcalc_tmp}\"\npython .../divergence_calc.py ... -tmp \"${divcalc_tmp}\"\n# ...\nrm -rf \"${divcalc_tmp}\"\n```\n\n### `scripts/TEstrainer/TEstrainer_for_earlGrey.sh`\n\nAdded a `-q` flag that sets `BAR_FLAG=\"\"` (default `BAR_FLAG=\"--bar\"`). The variable is substituted into every `parallel` call so a single flag controls all progress bars in the script.\n\n### `scripts/divergenceCalc/divergence_calc.py`\n\n**1. Added `import numpy as np`** — used for `np.array_split` (see below).\n\n**2. Replaced manual chunk splitting with `np.array_split`**\n\nThe old code computed `chunk_size = int(rows / num_processes)` and stepped through the DataFrame with a fixed stride. For inputs where `rows` is not evenly divisible by `num_processes` this could silently drop up to `num_processes - 1` rows. `np.array_split` distributes any remainder evenly:\n\n```python\nchunks = [in_gff.iloc[idx] for idx in np.array_split(range(len(in_gff)), num_processes)]\n```\n\n**3. Replaced all path string concatenation with `os.path.join`**\n\nAll occurrences of `temp_dir + \"/subdir/\" + name` have been replaced with `os.path.join(temp_dir, \"subdir\", name)`. This fixes `OSError`s that occurred when `args.temp_dir` was passed with a trailing slash (e.g. `/tmp/egdiv_myspecies/`), which caused double-slash paths like `//pybedtools` that could fail on some systems.\n\n### `scripts/filteringOverlappingRepeats.R`\n\nFixed a crash that occurred when no nested TEs were found in the genome. The previous code unconditionally called `mutate()` and `bind_rows()` on `nested_only`, which threw an error when `nested_only` was an empty data frame. The fix wraps the nested-TE processing in an `if (nrow(nested_only) > 0)` guard and falls through to writing `unnested_only` directly when there are no nested elements:\n\n```r\nif (nrow(nested_only) > 0) {\n nested_only <- nested_only %>%\n mutate(attributes = paste0(attributes, \";nested=\", nested, \"-round\", nesting_round)) %>%\n select(-c(nested, nesting_round))\n output.gff <- bind_rows(nested_only, unnested_only) %>% arrange(seqid, start)\n} else {\n output.gff <- unnested_only %>% arrange(seqid, start)\n}\n```\n\n### `scripts/divergenceCalc/divergence_plot.R`\n\nFixed crashes in the per-superfamily Kimura divergence plots when a TE class (DNA, LINE, LTR, SINE, PLE, RC) is absent from the genome. The previous code built each plot unconditionally and then caught errors with `try(ggplot_build(...))`. The new approach checks `!is.null(...) && nrow(...) > 0` before constructing the ggplot object:\n\n```r\nif (!is.null(divergence_eg_tes_rounded_for_superfamily_plot[[\"DNA\"]]) &&\n nrow(divergence_eg_tes_rounded_for_superfamily_plot[[\"DNA\"]]) > 0) {\n kimura_superfamily_plot_1 <- ggplot(...) + ...\n} else {\n kimura_superfamily_plot_1 <- NULL\n}\n```\n\nThe axis-flipping logic was also updated to skip `NULL` plots rather than attempting to add `scale_x_continuous()` to them (which would previously crash).\n\n## Build and test environment\n\nBuild the conda package from the local source tree:\n\n```bash\ncd /data/toby/EarlGrey/conda\nconda build .\n```\n\nCreate a fresh environment and install:\n\n```bash\nconda create -n earlgrey_723_test -c conda-forge -c bioconda earlgrey --use-local\nconda activate earlgrey_723_test\n```\n\nLink the Dfam libraries and configure RepeatMasker:\n\n```bash\nln -sf /data/toby/tools/earlgrey_databases/Libraries/famdb/* \\\n /data/toby/miniforge3/envs/earlgrey_723_test/share/RepeatMasker/Libraries/famdb/\n\ncd /data/toby/miniforge3/envs/earlgrey_723_test/share/RepeatMasker\nperl configure\n```\n\n### Static checks\n\n```bash\ncd /data/toby/EarlGrey\n\n# 1. Confirm numpy is in meta.yaml\ngrep 'numpy' conda/meta.yaml\n\n# 2. Confirm -q flag is wired up in earlGrey\ngrep -n 'quietBar\\|BAR_FLAG\\|\\-q' earlGrey | head -20\n\n# 3. Confirm BAR_FLAG is used in TEstrainer_for_earlGrey.sh\ngrep -n 'BAR_FLAG\\|bar' scripts/TEstrainer/TEstrainer_for_earlGrey.sh | grep -v '#'\n\n# 4. Confirm os.path.join used throughout divergence_calc.py (no string concat with temp_dir)\ngrep 'temp_dir *+' scripts/divergenceCalc/divergence_calc.py # should return nothing\n\n# 5. Confirm np.array_split is used for chunking\ngrep 'array_split' scripts/divergenceCalc/divergence_calc.py\n\n# 6. Confirm nrow guard in filteringOverlappingRepeats.R\ngrep -n 'nrow(nested_only)' scripts/filteringOverlappingRepeats.R\n```\n\n### Functional tests\n\n```bash\ncd /data/toby/testDIR/\n\n# Basic run — confirm pipeline completes with -q yes\nearlGrey -g test.fasta -s test_723_quiet -o . -t 16 -q yes\n\n# Confirm no ANSI bar codes in the log (stdout should have no ESC sequences)\nearlGrey -g test.fasta -s test_723_quiet2 -o . -t 16 -q yes 2>&1 | cat | grep -c $'\\033' || echo \"No ANSI codes — good\"\n\n# Confirm pipeline still works without -q (bar shown as before)\nearlGrey -g test.fasta -s test_723_noq -o . -t 16\n```\n\n---\n\n# Version 7.2.4 — pipeline error handling, samplable genome size for RepeatModeler, and bash safety\n\n## Background\n\nTwo complementary robustness improvements were identified and implemented across the three main Earl Grey pipeline scripts (`earlGrey`, `earlGreyLibConstruct`, `earlGreyAnnotationOnly`):\n\n1. The `deNovo1()` function in `earlGrey` and `earlGreyLibConstruct` used a deeply nested retry loop to work around RepeatModeler failures on small or fragmented genomes. This approach was reactive: RepeatModeler would run, fail, be retried with a smaller `-genomeSampleSizeMax`, fail again, and be retried once more — wasting potentially hours of compute and producing confusing log output. The ParTEA (pangenome) version of Earl Grey had already solved this more elegantly by pre-computing the samplable genome size and choosing the correct flag upfront.\n\n2. All three scripts lacked `set -eo pipefail`, meaning that any command that exited non-zero in the middle of a function body (outside an explicit `if` check) would silently be ignored and execution would continue into the next stage with corrupt or missing intermediate files. Additionally, two common shell idioms in the scripts would produce false failures under strict error handling: `expr N / 4` (returns exit code 1 when the result is 0) and `ls ... | head -n 1` pipelines (SIGPIPE exit 141 from `ls` when `head` exits early under `pipefail`).\n\n## Changes made\n\n### `earlGrey` and `earlGreyLibConstruct` — `deNovo1()` redesign\n\nThe previous implementation:\n\n```bash\ndeNovo1() {\n RepeatModeler -threads ${ProcNum} -database ...\n if [ ! -e ...-families.fa ]; then\n echo \"ERROR: Retrying with limit set as Round 5\"\n RepeatModeler ... -genomeSampleSizeMax 81000000\n if [ ! -e ...-families.fa ]; then\n echo \"ERROR: Retrying with limit set as Round 4\"\n RepeatModeler ... -genomeSampleSizeMax 27000000\n if [ ! -e ...-families.fa ]; then\n echo \"ERROR: RepeatModeler Failed\" && exit 2\n fi\n fi\n fi\n}\n```\n\nThis ran RepeatModeler up to three times, each time with a smaller sampling cap, without any principled basis for which cap to choose. It also only covered two fallback tiers (rounds 5 and 4), leaving genomes smaller than 12 Mb (rounds 1–2) unhandled.\n\nThe new implementation mirrors the approach used in ParTEA. Before invoking RepeatModeler, the samplable genome size is computed as the sum of all contigs ≥ 40 kb (the threshold below which RepeatModeler discards contigs during sampling). The appropriate `-genomeSampleSizeMax` cap is then derived from the cumulative RECON round thresholds and set in `SAMPLE_FLAG`. RepeatModeler is invoked exactly once with the correct flag, and a single post-run existence check is retained as a safety net.\n\n**Cumulative RECON round thresholds:**\n\n| Rounds supported | Cumulative threshold | Cap applied |\n|-----------------|---------------------|-------------|\n| All 6 rounds | ≥ 363 Mb (3+9+27+81+243) | none (default) |\n| Rounds 1–5 | ≥ 120 Mb (3+9+27+81) | `-genomeSampleSizeMax 81000000` |\n| Rounds 1–4 | ≥ 39 Mb (3+9+27) | `-genomeSampleSizeMax 27000000` |\n| Rounds 1–3 | ≥ 12 Mb (3+9) | `-genomeSampleSizeMax 9000000` |\n| Rounds 1–2 | < 12 Mb | `-genomeSampleSizeMax 3000000` |\n\nThe samplable genome size is computed using `awk` directly on `$genome` (the `.prep` file, which is already available at the `deNovo1` call site):\n\n```bash\nGENOME_SIZE=$(awk '/^>/{if(len>=40000)sum+=len; len=0; next}{len+=length($0)} END{if(len>=40000)sum+=len; print sum+0}' \"${genome}\")\n```\n\n### `earlGrey`, `earlGreyLibConstruct`, `earlGreyAnnotationOnly` — `set -eo pipefail`\n\n`set -eo pipefail` was added immediately after `#!/bin/bash` in all three scripts.\n\n- `-e`: exits the script immediately if any simple command returns a non-zero exit code and is not part of a condition test.\n- `-o pipefail`: extends `-e` to pipelines — a pipeline fails if any stage within it returns non-zero, not just the last stage.\n\nTogether these ensure that a failure in any tool invocation within a function body (RepeatMasker, BuildDatabase, TEstrainer, rcMergeRepeats, R scripts, etc.) will abort the pipeline at the point of failure rather than silently continuing to the next stage.\n\n### `earlGrey`, `earlGreyLibConstruct`, `earlGreyAnnotationOnly` — `expr` replaced with `$(( ))`\n\nAll occurrences of `rmthreads=$(expr $ProcNum / 4)` and `strainthreads=$(expr $ProcNum / 4)` were replaced with the equivalent `$(( ProcNum / 4 ))`. The `expr` utility returns exit code 1 when the arithmetic result is 0, which occurs when `ProcNum < 4`. Under `set -e` this would abort the script before any tool was invoked if the user specified fewer than 4 threads.\n\n### `earlGrey`, `earlGreyLibConstruct`, `earlGreyAnnotationOnly` — `ls | head -n 1` pipeline fix\n\nAll `ls -td ... | head -n 1` pipeline expressions used to retrieve the most recently created directory (TEstrainer output directories, HELIANO output directories) were appended with `|| true`:\n\n```bash\nlatestFile=\"$(realpath $(ls -td -- ${OUTDIR}/${species}_strainer/*/ | head -n 1) || true)/...\"\n```\n\nUnder `pipefail`, when `head` exits after reading one line, `ls` receives SIGPIPE (exit code 141), causing the pipeline to return non-zero. This is expected behaviour — `ls` has more output but `head` is done — and `|| true` suppresses this signal without masking genuine `ls` errors (which would produce a different non-zero exit and an error message to stderr).\n\n## Static verification\n\n```bash\ncd /data/toby/EarlGrey\n\n# 1. Confirm set -eo pipefail is present in all three scripts\ngrep -n 'set -eo pipefail' earlGrey earlGreyLibConstruct earlGreyAnnotationOnly\n\n# 2. Confirm no remaining expr calls\ngrep -n 'expr' earlGrey earlGreyLibConstruct earlGreyAnnotationOnly\n\n# 3. Confirm ls|head pipelines are guarded\ngrep -n 'head -n 1' earlGrey earlGreyLibConstruct earlGreyAnnotationOnly\n\n# 4. Confirm GENOME_SIZE and SAMPLE_FLAG logic is present in deNovo1\ngrep -n 'GENOME_SIZE\\|SAMPLE_FLAG' earlGrey earlGreyLibConstruct\n```\n\n**Expected for check 1:** Three lines, one per script, each showing `set -eo pipefail` at line 2.\n\n**Expected for check 2:** No output (all `expr` calls replaced).\n\n**Expected for check 3:** All matching lines should contain `|| true`.\n\n**Expected for check 4:** Both scripts should show `GENOME_SIZE` and `SAMPLE_FLAG` variable assignments and the five-tier `if/elif/else` block inside `deNovo1`.\n\n"
},
{
"path": "conda/meta.yaml",
"content": "{% set name = \"EarlGrey\" %}\n{% set version = \"7.2.4\" %}\n\npackage:\n name: {{ name|lower }}\n version: {{ version }}\n\nsource:\n path: ..\n\nbuild:\n number: 0\n skip: True # [osx]\n run_exports:\n - {{ pin_subpackage('earlgrey', max_pin=\"x\") }}\n\nrequirements:\n build:\n - make\n - {{ compiler('cxx') }}\n run:\n - python >=3.10\n - hmmer\n - trf\n - cd-hit\n - genometools-genometools\n - numpy\n - pandas\n - ncls\n - pyfaidx\n - pyranges\n - parallel\n - repeatmasker >=4.2.3\n - ltr_retriever\n - mafft\n - mreps\n - ninja-nj\n - repeatscout\n - recon\n - repeatmodeler >=2.0.4\n - bioconductor-genomeinfodb\n - bioconductor-genomeinfodbdata\n - bioconductor-bsgenome\n - bioconductor-plyranges\n - r-ape\n - r-optparse\n - r-tidyverse\n - r-plyr\n - r-viridis\n - r-cowplot\n - r-ggtext\n - r-data.table\n - r-magrittr\n - bedtools\n - emboss\n - pybedtools\n - samtools\n - heliano\n - r-kableextra\n\ntest:\n commands:\n - df -h\n - earlGrey -h\n\nabout:\n home: \"https://github.com/TobyBaril/EarlGrey\"\n dev_url: \"https://github.com/TobyBaril/EarlGrey\"\n doc_url: \"https://github.com/TobyBaril/EarlGrey/blob/v{{ version }}/README.md\"\n license: \"OSL-2.1\"\n license_file: LICENSE\n summary: \"Earl Grey: A fully automated TE curation and annotation pipeline.\"\n description: |\n Earl Grey is a full-automated transposable element (TE) annotation pipeline,\n leveraging the most widely-used tools and combining these with a consensus\n elongation process (BEAT) to better define de novo consensus sequences when\n annotating new genome assemblies.\n\nextra:\n identifiers:\n - doi:10.1093/molbev/msae068\n"
},
{
"path": "earlGrey",
"content": "#!/bin/bash\nset -eo pipefail\n\nusage()\n{\n\techo \"\t#############################\n\tearlGrey version 7.2.4\n\tRequired Parameters:\n\t\t-g == genome.fasta\n\t\t-s == species name\n\t\t-o == output directory\n\n\tOptional Parameters:\n\t\t-t == Number of Threads (DO NOT specify more than are available)\n\t\t-r == RepeatMasker search term (e.g arthropoda/eukarya)\n\t\t-l == Starting consensus library for an initial mask (in fasta format)\n\t\t-i == Number of Iterations to BLAST, Extract, Extend (Default: 10)\n\t\t-f == Number flanking basepairs to extract (Default: 1000)\n\t\t-c == Cluster TE library to reduce redundancy? (yes/no, Default: no)\n\t\t-m == Remove putative spurious TE annotations <100bp? (yes/no, Default: no)\n\t\t-d == Create soft-masked genome at the end? (yes/no, Default: no)\n\t\t-n == Max number of sequences used to generate consensus sequences (Default: 20)\n\t\t-a == minimum number of sequences required to build a consensus sequence (Default: 3)\n\t\t-e == Run HELIANO as an optional step to detect Helitrons (yes/no, Default: no)\n\t\t-q == Suppress TEstrainer parallel progress bar (yes/no, Default: no, useful for batch/sbatch jobs)\n\t\t-h == Show help\n\n\tExample Usage:\n\n\tearlGrey -g bombyxMori.fasta -s bombyxMori -o /home/toby/bombyxMori/repeatAnnotation/ -t 16\n\n\tQueries can be sent to:\n\ttobias.baril[at]unine.ch\n\n\tPlease make use of the GitHub Issues and Discussion Tabs at: https://github.com/TobyBaril/EarlGrey\n\t#############################\"\n}\n\n# Subprocess Make Directories #\nmakeDirectory()\n{\n\tdirectory=$(realpath ${directory})\n\tmkdir -p ${directory}/${species}_EarlGrey/ && OUTDIR=${directory}/${species}_EarlGrey\n\tif [ ! -z \"$RepSpec\" ] || [ ! -z \"$startCust\" ] ; then\n\t\tmkdir -p $OUTDIR/${species}_RepeatMasker/\n\tfi\n\tmkdir -p $OUTDIR/${species}_Database/\n\tmkdir -p $OUTDIR/${species}_RepeatModeler/\n\tmkdir -p $OUTDIR/${species}_strainer/\n\tmkdir -p $OUTDIR/${species}_RepeatMasker_Against_Custom_Library/\n\tmkdir -p $OUTDIR/${species}_RepeatLandscape/\n\tmkdir -p $OUTDIR/${species}_mergedRepeats/\n\tmkdir -p ${OUTDIR}/${species}_summaryFiles/\n\tif [ ! -z \"$heli\" ]; then\n\t\tmkdir -p ${OUTDIR}/${species}_heliano/\n\tfi\n}\n\n# Subprocess PrepGenome #\nprepGenome()\n{\n\tif [ ! -L ${genome} ]; then\n\t\tgenome=$(realpath ${genome})\n\tfi\n\tif [ -L $genome ]; then\n\t\tgenome=$(realpath -s ${genome})\n\tfi\n\tif [ ! -f ${genome}.prep ] || [ ! -f ${genome}.dict ]; then\n\t\tcp ${genome} ${genome}.bak && gzip -f ${genome}.bak \n\t\tsed '/>/ s/[[:space:]].*//g; /^$/d' ${genome} > ${genome}.tmp\n\t\t${SCRIPT_DIR}/headSwap.sh -i ${genome}.tmp -o ${genome}.prep && rm ${genome}.tmp\n\t\tmv ${genome}.tmp.dict ${genome}.dict\n\t\tsed -i '/^>/! s/[DVHBPE]/N/g' ${genome}.prep\n\t\tdict=${genome}.dict\n\t\tgenOrig=${genome}\n\t\tgenome=${genome}.prep\n\telse\n\t\tdict=${genome}.dict\n\t\tgenOrig=${genome}\n\t\tgenome=${genome}.prep\n\tfi\n}\n\n# Subprocess getRepeatMaskerFasta\n# Generate a copy of the RepeatMasker library subset used for the initial TE mask in FASTA format (only used if a RepeatMasker run is specified)\ngetRepeatMaskerFasta()\n{\n\tif [[ $RepSpec = *\" \"* ]]; then\n\t\techo \"ERROR: You have entered a species name that contains a space, please use the NCBI TaxID rather than name. E.G In place of \\\"Homo sapiens\\\" use \\\"9606\\\"\"\n\t\texit 2\n\tfi\n\tif [[ $(which RepeatMasker) == *\"bin\"* ]]; then\n\t\tlibpath=\"$(which RepeatMasker | sed 's|bin/RepeatMasker|share/RepeatMasker/Libraries/famdb/|')\"\n\telse\n\t\tlibpath=\"$(which RepeatMasker | sed 's|/[^/]*$||g')/Libraries/famdb/\"\n\tfi\n\tif [[ $(which RepeatMasker) == *\"bin\"* ]]; then\n\t\tPATH=$PATH:\"$(which RepeatMasker | sed 's|bin/RepeatMasker|share/RepeatMasker/|')\"\n\tfi\n\tmkdir -p $OUTDIR/${species}_Curated_Library/\n\tfamdb.py -i $libpath families -f fasta_name --include-class-in-name -a -d --curated $RepSpec > ${OUTDIR}/${species}_Curated_Library/${RepSpec}.RepeatMasker.lib\n\tRepSub=${OUTDIR}/${species}_Curated_Library/${RepSpec}.RepeatMasker.lib\n}\n\n# Subprocess firstMask\n# Run the initial RepeatMasker run with the specified species subset (only used if a RepeatMasker run is specified)\nfirstMask()\n{\n\tcd ${OUTDIR}/${species}_RepeatMasker\n\trmthreads=$(( ProcNum / 4 ))\n\tRepeatMasker -species $RepSpec -no_is -lcambig -s -a -pa $rmthreads -dir $OUTDIR/${species}_RepeatMasker $genome\n\tif [ ! -f ${OUTDIR}/${species}_RepeatMasker/*.masked ]; then\n\t\techo \"ERROR: RepeatMasker failed, please check logs. This is likely because of an invalid species search term, if issue persists please use NCBI Taxids (E.G Drosophila is replaced with 7125)\"; exit 2\n\tfi\n}\n\n# Subprocess firstMaskCustomLib\n# Run the initial RepeatMasker run with the specific fasta consensus library\nfirstMaskCustomLib()\n{\n\tcd ${OUTDIR}/${species}_RepeatMasker\n\trmthreads=$(( ProcNum / 4 ))\n\tRepeatMasker -lib ${startCust} -no_is -lcambig -s -a -pa $rmthreads -dir $OUTDIR/${species}_RepeatMasker $genome\n\tRepSub=${startCust}\n\tif [ ! -f ${OUTDIR}/${species}_RepeatMasker/*.masked ]; then\n\t\techo \"ERROR: RepeatMasker failed, please check logs. This is likely because of an invalid custom consensus file, check the fasta file looks as expected\"; exit 2\n\tfi\n}\n\n\n# Subprocess buildDB \n# if masked genome exists, build database from this. If not, build database from original input genome\nbuildDB()\n{\n\tif [ -f ${OUTDIR}/${species}_RepeatMasker/*.masked ]; then\n\t\tcd ${OUTDIR}/${species}_Database\n\t\tBuildDatabase -name ${species} ${OUTDIR}/${species}_RepeatMasker/*.masked\n\telse\n\t\tcd ${OUTDIR}/${species}_Database\n\t\tBuildDatabase -name ${species} $genome\n\tfi\n}\n\n# Subprocess deNovo1\n# Run the initial de novo annotation run with RepeatModeler2.\n# The genomeSampleSizeMax flag is set based on the samplable genome size (contigs >= 40 kb)\n# to avoid RepeatModeler failures on small or fragmented genomes.\ndeNovo1()\n{\n\tcd ${OUTDIR}/${species}_RepeatModeler\n\n\t# Compute samplable genome size: sum of contigs >= 40 kb only.\n\t# RepeatModeler discards contigs shorter than 40 kb during sampling, so using\n\t# the total genome size would overestimate the sequence available per round.\n\tGENOME_SIZE=$(awk '/^>/{if(len>=40000)sum+=len; len=0; next}{len+=length($0)} END{if(len>=40000)sum+=len; print sum+0}' \"${genome}\")\n\techo \"Samplable genome size (contigs >= 40 kb): ${GENOME_SIZE} bp\"\n\n\t# Set -genomeSampleSizeMax to the highest RECON round threshold the genome\n\t# can support. Thresholds are cumulative across rounds (r2=3M, r3=9M, r4=27M,\n\t# r5=81M, r6=243M), so the samplable size must cover all rounds up to the cap:\n\t# >= 363M (3+9+27+81+243): no cap -> all 6 rounds\n\t# >= 120M (3+9+27+81): cap 81M -> rounds 1-5\n\t# >= 39M (3+9+27): cap 27M -> rounds 1-4\n\t# >= 12M (3+9): cap 9M -> rounds 1-3\n\t# < 12M: cap 3M -> rounds 1-2\n\tSAMPLE_FLAG=\"\"\n\tif [ \"${GENOME_SIZE}\" -ge 363000000 ] 2>/dev/null; then\n\t\t: # genome large enough for all rounds; use RepeatModeler default\n\telif [ \"${GENOME_SIZE}\" -ge 120000000 ] 2>/dev/null; then\n\t\techo \"Capping at round 5 (-genomeSampleSizeMax 81000000)\"\n\t\tSAMPLE_FLAG=\"-genomeSampleSizeMax 81000000\"\n\telif [ \"${GENOME_SIZE}\" -ge 39000000 ] 2>/dev/null; then\n\t\techo \"Capping at round 4 (-genomeSampleSizeMax 27000000)\"\n\t\tSAMPLE_FLAG=\"-genomeSampleSizeMax 27000000\"\n\telif [ \"${GENOME_SIZE}\" -ge 12000000 ] 2>/dev/null; then\n\t\techo \"Capping at round 3 (-genomeSampleSizeMax 9000000)\"\n\t\tSAMPLE_FLAG=\"-genomeSampleSizeMax 9000000\"\n\telse\n\t\techo \"Capping at round 2 (-genomeSampleSizeMax 3000000)\"\n\t\tSAMPLE_FLAG=\"-genomeSampleSizeMax 3000000\"\n\tfi\n\n\tRepeatModeler -threads ${ProcNum} -database ${OUTDIR}/${species}_Database/${species} ${SAMPLE_FLAG}\n\n\tif [ ! -e ${OUTDIR}/${species}_Database/${species}-families.fa ]; then\n\t\techo \"ERROR: RepeatModeler Failed\"\n\t\texit 2\n\tfi\n}\n\n# Subprocess strainer \n# contains the BLAST, Extract, Extend, Trim pipeline from James Galbraith\nstrainer()\n{\n\tcd ${OUTDIR}/${species}_strainer/\n\tif [ \"$ProcNum\" -ge 4 ]; then\n\t\tstrainthreads=$(( ProcNum / 4 ))\n\telse\n\t\tstrainthreads=$ProcNum\n\tfi\n\t${SCRIPT_DIR}/TEstrainer/TEstrainer_for_earlGrey.sh -g $genome -l ${OUTDIR}/${species}_Database/${species}-families.fa -t ${strainthreads} -f $Flank -r $num -n $no_seq -m $min_seq$([ \"$quietBar\" == \"yes\" ] && echo \" -q\")\n\tlatestFile=\"$(realpath $(ls -td -- ${OUTDIR}/${species}_strainer/*/ | head -n 1) || true)/${species}-families.fa.strained\"\n\tcp $latestFile ${OUTDIR}/${species}_strainer/\n}\n\n# Subprocess strainerResume\n# resumes interrupted TEstrainer runs\nstrainerResume() {\n\tcd ${OUTDIR}/${species}_strainer/\n\tstrainDataDir=$(find . -maxdepth 1 -type d -name \"TS_${species}-families.fa_*\")\n\tif [ ! $(echo \"$strainDataDir\" | wc -l) -eq 1 ]; then\n\t\techo \"ERROR: There are more TEstrainer directories than expected present\"\n\t\texit 2\n\tfi\n\tif [ \"$ProcNum\" -ge 4 ]; then\n\t\tstrainthreads=$(( ProcNum / 4 ))\n\telse\n\t\tstrainthreads=$ProcNum\n\tfi\n\t${SCRIPT_DIR}/TEstrainer/TEstrainer_for_earlGrey.sh -g $genome -l ${OUTDIR}/${species}_Database/${species}-families.fa -t ${strainthreads} -f $Flank -r $num -n $no_seq -m $min_seq$([ \"$quietBar\" == \"yes\" ] && echo \" -q\") -d ${strainDataDir} -R\n\tlatestFile=\"$(realpath $(ls -td -- ${OUTDIR}/${species}_strainer/*/ | head -n 1) || true)/${species}-families.fa.strained\"\n\tcp $latestFile ${OUTDIR}/${species}_strainer/\n}\n\n# Subprocess clust\n# optional wicker clustering of TE library consensus sequences to reduce redundancy\nclust()\n{\n\tcd ${OUTDIR}/${species}_strainer/\n\tcd-hit-est -d 0 -aS 0.8 -c 0.8 -G 0 -g 1 -b 500 -r 1 -i $latestFile -o ${latestFile}.clstrd.fa\n\tlatestFile=$latestFile.clstrd.fa\n}\n\n\n# Subprocess novoMask\n# Run RepeatMasker with the final processed library\nnovoMask()\n{\n\tif [ -z \"$RepSpec\" ] && [ -z \"$startCust\" ]; then\n\t\tcd ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/\n\t\trmthreads=$(( ProcNum / 4 ))\n\t\tRepeatMasker -lib $latestFile -no_is -lcambig -s -a -pa $rmthreads -dir ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/ $genome\n\telse\n\t\tmkdir -p $OUTDIR/${species}_Curated_Library/\n\t\tcd ${OUTDIR}/${species}_Curated_Library/\n\t\tcat $latestFile $RepSub > ${species}_combined_library.fasta\n\t\tlatestFile=$(realpath ${species}_combined_library.fasta)\n\t\tcd ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/\n\t\trmthreads=$(( ProcNum / 4 ))\n\t\tRepeatMasker -lib $latestFile -no_is -lcambig -s -a -pa $rmthreads -dir ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/ $genome\n\tfi\n}\n\n# Subprocess heliano\n# Run HELIANO as an optional step, then replace overlapping repeats in the merged output with HELIANO outputs\n### TODO:\nheliano_optional()\n{\n\tcd ${OUTDIR}/${species}_heliano/\n\theliano -g $genome --nearest -dn 6000 -flank_sim 0.5 -o ${OUTDIR}/${species}_heliano/HEL_${timestamp} -w 10000 -n $ProcNum\n\tawk '{OFS=\"\\t\"}{print $1, \"HELIANO\", \"RC/Helitron\", $2+1, $3, $5, $6, \".\", \"ID=\"$9\"_\"$11\";shortTE=F\"}' ${OUTDIR}/${species}_heliano/HEL_${timestamp}/RC.representative.bed > ${OUTDIR}/${species}_heliano/HEL_${timestamp}/RC.representative.gff\n\thelitron_gff=${OUTDIR}/${species}_heliano/HEL_${timestamp}/RC.representative.gff\n}\n\n# Subprocess rcMergeRepeats\n# Defragment repeat sequences to adjust for insertion times\nmergeRep()\n{\n\tmkdir ${OUTDIR}/${species}_mergedRepeats/looseMerge\n\tif [ -s \"$helitron_gff\" ]; then\n\t\techo \"Running loose merge with HELIANO output\"\n\t\t${SCRIPT_DIR}/rcMergeRepeatsLoose -f $genome -s $species -d ${OUTDIR}/${species}_mergedRepeats/looseMerge -u ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).out -q ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).tbl -t $ProcNum -b ${dict} -m $margin -e $helitron_gff\n\telse\n\t\t${SCRIPT_DIR}/rcMergeRepeatsLoose -f $genome -s $species -d ${OUTDIR}/${species}_mergedRepeats/looseMerge -u ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).out -q ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).tbl -t $ProcNum -b ${dict} -m $margin\n\tfi\n\n\tif [ -f \"${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed\" ]; then\n\t\tawk '{OFS=\"\\t\"}{print $1, $2, $3, $4, $5, $6, $7, $8, toupper($9)}' ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff > ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff.1 && mv ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff{.1,}\n\tfi\n\n\tif [ ! -f \"${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed\" ]; then\n\t\techo \"ERROR: loose merge defragmentation failed, trying strict merge...\"\n\t\tcd ${OUTDIR}/${species}_mergedRepeats/\n\t\tif [ -s \"$helitron_gff\" ]; then\n\t\t\t${SCRIPT_DIR}/rcMergeRepeats -f $genome -s $species -d ${OUTDIR}/${species}_mergedRepeats/ -u ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).out -q ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).tbl -t $ProcNum -b ${dict} -m $margin -e $helitron_gff\n\t\telse\n\t\t\t${SCRIPT_DIR}/rcMergeRepeats -f $genome -s $species -d ${OUTDIR}/${species}_mergedRepeats/ -u ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).out -q ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).tbl -t $ProcNum -b ${dict} -m $margin\n\t\tfi\n\t\tif [ ! -f \"${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.bed\" ]; then\n\t\t\techo \"ERROR: strict merge also failed, check ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).out looks as expected\"\n\t\t\texit 2\n\t\tfi\n\tfi\n}\n\n# Subprocess pieChart\n# Generate a pie chart summary of TE content\ncharts()\n{\n\tcd ${OUTDIR}/${species}_summaryFiles/\n\tif [ -f \"${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed\" ]; then\n\t\t${SCRIPT_DIR}/autoPie.sh -i ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.summary -t ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).tbl -p ${OUTDIR}/${species}_summaryFiles/${species}.summaryPie.pdf -o ${OUTDIR}/${species}_summaryFiles/${species}.highLevelCount.txt\n\telse\n\t\t${SCRIPT_DIR}/autoPie.sh -i ${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.summary -t ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).tbl -p ${OUTDIR}/${species}_summaryFiles/${species}.summaryPie.pdf -o ${OUTDIR}/${species}_summaryFiles/${species}.highLevelCount.txt\n\tfi\n}\n\n# Subprocess calcDivRL\n# Calculate divergence estimates\ncalcDivRL()\n{\n\tcd ${OUTDIR}/${species}_RepeatLandscape\n\t# divergence_calc.py calls pybedtools.set_tempdir() which sets tempfile.tempdir\n\t# globally. forkserver creates its AF_UNIX socket under tempfile.tempdir, so a\n\t# long output path can exceed the 108-char socket limit (OSError: AF_UNIX path\n\t# too long). Pass a short per-species path in /tmp to avoid this.\n\tdivcalc_tmp=\"/tmp/egdiv_${species}\"\n\tmkdir -p \"${divcalc_tmp}\"\n\tif [ -z \"$RepSpec\" ] && [ -z \"$startCust\" ]; then\n\t\tpython ${SCRIPT_DIR}/divergenceCalc/divergence_calc.py -l $latestFile -g $genOrig -i ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff -o ${OUTDIR}/${species}_RepeatLandscape/${species}.filteredRepeats.withDivergence.gff -t $ProcNum -tmp \"${divcalc_tmp}\"\n\t\tRscript ${SCRIPT_DIR}/divergenceCalc/divergence_plot.R -s $species -g ${OUTDIR}/${species}_RepeatLandscape/${species}.filteredRepeats.withDivergence.gff -o ${OUTDIR}/${species}_RepeatLandscape/ && \\\n\t\tawk -F'\\t' 'BEGIN{OFS=\"\\t\"} {gsub(\"NAME=\",\"Name=\",$9); gsub(\"TSTART=\",\"tstart=\",$9); gsub(\"TEND=\",\"tend=\",$9); gsub(\"SHORTTE=\",\"shortte=\",$9); gsub(\"TEGROUP=\",\"tegroup=\",$9); gsub(\"KIMURA80=\",\"kimura80=\",$9); gsub(\"NESTED=\", \"nested=\", $9); print}' ${OUTDIR}/${species}_RepeatLandscape/${species}.filteredRepeats.withDivergence.gff > ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff\n\t\trm -rf \"${divcalc_tmp}\"\n\t\tcp ${OUTDIR}/${species}_RepeatLandscape/*.pdf ${OUTDIR}/${species}_summaryFiles/\n\t\tcp ${OUTDIR}/${species}_RepeatLandscape/*_summary_table.tsv ${OUTDIR}/${species}_summaryFiles/${species}_divergence_summary_table.tsv\n\telse\n\t\tpython ${SCRIPT_DIR}/divergenceCalc/divergence_calc.py -l ${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta -g $genOrig -i ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff -o ${OUTDIR}/${species}_RepeatLandscape/${species}.filteredRepeats.withDivergence.gff -t $ProcNum -tmp \"${divcalc_tmp}\"\n\t\tRscript ${SCRIPT_DIR}/divergenceCalc/divergence_plot.R -s $species -g ${OUTDIR}/${species}_RepeatLandscape/${species}.filteredRepeats.withDivergence.gff -o ${OUTDIR}/${species}_RepeatLandscape/ && \\\n\t\tawk -F'\\t' 'BEGIN{OFS=\"\\t\"} {gsub(\"NAME=\",\"Name=\",$9); gsub(\"TSTART=\",\"tstart=\",$9); gsub(\"TEND=\",\"tend=\",$9); gsub(\"SHORTTE=\",\"shortte=\",$9); gsub(\"TEGROUP=\",\"tegroup=\",$9); gsub(\"KIMURA80=\",\"kimura80=\",$9); gsub(\"NESTED=\", \"nested=\", $9); print}' ${OUTDIR}/${species}_RepeatLandscape/${species}.filteredRepeats.withDivergence.gff > ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff\n\t\trm -rf \"${divcalc_tmp}\"\n\t\tcp ${OUTDIR}/${species}_RepeatLandscape/*.pdf ${OUTDIR}/${species}_summaryFiles/\n\t\tcp ${OUTDIR}/${species}_RepeatLandscape/*_summary_table.tsv ${OUTDIR}/${species}_summaryFiles/${species}_divergence_summary_table.tsv\n\tfi\n}\n\n# Subprocess sweepUp\n# Puts required files into a summary folder\nsweepUp()\n{\n\tcd ${OUTDIR}/${species}_summaryFiles/\n\tif [ -f \"${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed\" ] && [ ! -f \"${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta\" ]; then\n\t\tcp ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff $latestFile ${OUTDIR}/${species}_summaryFiles/\n\telif [ -f \"${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed\" ] && [ -f \"${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta\" ]; then\n\t\tcp ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff ${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta $latestFile ${OUTDIR}/${species}_summaryFiles/\n\telif [ ! -f \"${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed\" ] && [ ! -f \"${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta\" ]; then\n\t\tcp ${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.bed ${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.gff $latestFile ${OUTDIR}/${species}_summaryFiles/\n\telif [ ! -f \"${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed\" ] && [ -f \"${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta\" ]; then\n\t\tcp ${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.bed ${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.gff ${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta $latestFile ${OUTDIR}/${species}_summaryFiles/\n\telse\n\t\techo \"Error: Cannot find required summary files, please check one of the following is present: ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed and/or ${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.bed\"\n\tfi\n}\n\n# Subprocess runningTea \nrunningTea()\n{\n\techo \" \n ) (\n\t ( ) )\n\t ) ( (\n\t _______)_\n\t.-'---------| \n ( C|/\\/\\/\\/\\/|\n\t'-./\\/\\/\\/\\/|\n\t '_________'\n\t '-------'\n\t<<< $stage >>>\"\n}\n\n# Subprocess GetTime\nconvertsecs() \n{\n\th=$(bc <<< \"${1}/3600\")\n\tm=$(bc <<< \"(${1}%3600)/60\")\n\ts=$(bc <<< \"${1}%60\")\n\tprintf \"%02d:%02d:%05.2f\\n\" $h $m $s\n}\n\n# Subprocess Checks\nChecks()\n{\n\tif [ -z \"$genome\" ] || [ -z \"$species\" ] || [ -z \"$directory\" ] ; then\n\t\tusage; exit 1\n\tfi\n\n\tif [ -z \"$ProcNum\" ] ; then\n\t\tProcNum=1; echo \"$ProcNum Cores Will Be Used\"\n\telse\n\t\techo \"$ProcNum Cores Will Be Used\"\n\tfi\n\n\tif [ -z \"$RepSpec\" ] && [ -z \"$startCust\" ] ; then\n\t\techo \"RepeatMasker species not specified, running Earl Grey without an initial mask with known repeats\"\n\telse\n\t\techo \"Running Earl Grey with an intial mask with known repeats\"\n\tfi\n\n\tif [ -z \"$num\" ] ; then\n\t\tnum=10; echo \"De Novo Sequences Will Be Extended Through a Maximum of $num Iterations\"\n\telse\n\t\techo \"De Novo Sequences Will Be Extended Through a Maximum of $num Iterations\"\n\tfi\n\n\tif [ -z \"$no_seq\" ] ; then\n\t\tno_seq=20; echo \"$no_seq sequences will be used in BEAT consensus generation\"\n\telse\n\t\techo \"$no_seq sequences will be used in BEAT consensus generation\"\n\tfi\n\t\n\tif [ -z \"$cluster\" ] || [ \"$cluster\" == \"no\" ] ; then\n\t\tcluster=no; echo \"TE library consensus sequences will not be clustered\"\n\telif [ \"$cluster\" == \"yes\" ]; then\n\t\tcluster=yes; echo \"TE library consensus sequences will be clustered, be aware of the impact this can have on subfamilies and chimeric TEs!\"\n\telse\n\t\tcluster=no; echo \"Clustering not specified using (yes/no). Using default parameter (no).\"\n\tfi\n\n\tif [ -z \"$softMask\" ] || [ \"$softMask\" == \"no\" ] ; then\n\t\tsoftMask=no; echo \"Softmasked genome will not be generated\"\n\telif [ \"$softMask\" == \"yes\" ]; then\n\t\tsoftMask=yes; echo \"Softmasked genome will be generated\"\n\telse\n\t\tsoftMask=no; echo \"Softmask not specified using (yes/no). Using default parameter (no).\"\n\tfi\n\n\tif [ \"$margin\" == \"yes\" ] ; then\n\t\tmargin=yes; echo \"Short TE sequences <100bp will be removed from annotation\"\n\telif [ -z \"$margin\" ] || [ \"$margin\" == \"no\" ]; then\n\t\tmargin=no; echo \"Short TE sequences <100bp will not be removed from annotation\"\n\telse\n\t\tmargin=no; echo \"Margin not specified using (yes/no). Using default parameter (no).\"\n\tfi\n\n\tif [ -z \"$Flank\" ]; then\n\t\tFlank=1000; echo \"Blast, Extend, Align, Trim Process Will Add 1000bp to Each End in Each Iteration\"\n\telse\n\t\techo \"Blast, Extract, Extend, Trim Process Will Add $Flank to Each End in Each Iteration\"\n\tfi\n\n\tif [ -z \"$min_seq\" ]; then\n\t\tmin_seq=3; echo \"Blast, Extend, Align, Trim Process Will Require 3 Sequences to Generate a New Consensus Sequence\"\n\telse\n\t\techo \"Blast, Extend, Align, Trim Process Will Require $min_seq Sequences to Generate a New Consensus Sequence\"\n\tfi\n\n\tif [ ! -d \"$SCRIPT_DIR\" ]; then\n\t\techo \"ERROR: Script directory variable not set, please run the configure script in the Earl Grey directory before attempting to run Earl Grey\"; exit 1\n\tfi\n\n\tif [ ! -d \"$SCRIPT_DIR/TEstrainer/\" ]; then\n\t\techo \"ERROR: teStrainer module not found, please check all modules are present and run the configure script in the Earl Grey directory before attempting to run Earl Grey\"; exit 1\n\tfi\n\n\tif [ \"$heli\" == \"yes\" ] ; then\n\t\theli=yes; echo \"HELITRON detection will be run using HELIANO\"\n\telif [ -z \"$heli\" ] || [ \"$heli\" == \"no\" ]; then\n\t\theli=no; echo \"HELITRON detection will not be run\"\n\telse\n\t\theli=no; echo \"HELITRON detection not specified using (yes/no). Using default parameter (no).\"\n\tfi\n\n\tif [ \"$quietBar\" == \"yes\" ] ; then\n\t\tquietBar=yes; echo \"TEstrainer parallel progress bar will be suppressed\"\n\telif [ -z \"$quietBar\" ] || [ \"$quietBar\" == \"no\" ]; then\n\t\tquietBar=no; echo \"TEstrainer parallel progress bar will be shown\"\n\telse\n\t\tquietBar=no; echo \"quietBar not specified using (yes/no). Using default parameter (no).\"\n\tfi\n\n\techo \"Please cite the following paper when using this software:\"\n\techo \"Baril, T., Galbraith, J. and Hayward, A., 2024. Earl Grey: a fully automated user-friendly transposable element annotation and analysis pipeline. Molecular Biology and Evolution, 41(4), p.msae068.\"\n}\n\n# Subprocess Dfam Checks\ndfamCheck()\n{\n\t# Dfam 3.9 checks\n\tlibrary_path=$(which RepeatMasker | sed 's|/bin/RepeatMasker|/share/RepeatMasker/Libraries/famdb|g')\n\texpected_file=\"${library_path}/dfam39_full.0.h5\"\n\tfile_count=$(find \"$library_path\" -maxdepth 1 -type f | wc -l)\n\tif [ $file_count -eq 2 ] && [ -f $expected_file ] && [ -f ${library_path}/rmlib.config ] ; then\n \t\techo \"WARNING: Earl Grey v6.1.0 has updated to Dfam v3.9.\" \n \t\techo \"Before using Earl Grey, you MUST download the required partitions from Dfam (https://dfam.org/releases/current/families/FamDB/)\"\n \t\techo \"These must be located at ${library_path}/. Then, you must reconfigure RepeatMasker before using Earl Grey.\"\n \t\techo \"I have written the required commands for you to faciliate the process.\"\n \t\techo \"Please follow these carefully to ensure successful configuration of RepeatMasker with Dfam 3.9.\"\n \t\techo \"As a note, these DB paritions are BIG, so only download the ones you need for your studies (or, if you do need all of them, make sure you have the space for them)\"\n \t\techo \"########################################################\"\n \t\techo \"A script for modification and automation of these steps has been generated in $(pwd)/configure_dfam39.sh\"\n \t\techo \"Please open the script in your preferred text editor (e.g. nano or vim) and change the partition numbers in [] to those that you would like to download\"\n \t\techo \"then, make the script executable (chmod +x $(pwd)/configure_dfam39.sh) and run it: ./configure_dfam39.sh\"\n \t\techo \"Alternatively, you can run the steps below one-by-one to achieve the same outcome:\"\n \t\techo \"########################################################\"\n \t\techo \"\"\n \t\tif [ -f \"$(pwd)/configure_dfam39.sh\" ]; then\n \t\trm $(pwd)/configure_dfam39.sh\n \t\tfi\n \t\techo \"# first, change directory to the famdb library location\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"cd ${library_path}/\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"# download the partitions you require from Dfam 3.9. In the below, change the numbers or range inside the square brackets to choose your subsets.\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"# e.g. to download partitions 0 to 10: [0-10]; or to download partitions 3,5, and 7: [3,5,7]; [0-16] is ALL PARTITIONS\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"curl -o 'dfam39_full.#1.h5.gz' 'https://dfam.org/releases/current/families/FamDB/dfam39_full.[0-16].h5.gz'\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"# decompress Dfam 3.9 paritions\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"gunzip *.gz\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"# move up to RepeatMasker main directory\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"cd $(which RepeatMasker | sed 's|/bin/RepeatMasker|/share/RepeatMasker/|g')\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"\" | tee -a $(pwd)/configure_dfam39.sh\n \t\t#echo \"# Replace famdb with version 2.0.1 (required for the Dfam 3.9)\" | tee -a $(pwd)/configure_dfam39.sh\n \t\t#echo \"wget https://github.com/Dfam-consortium/FamDB/archive/refs/tags/2.0.1.tar.gz && tar --wildcards -zxvf 2.0.1.tar.gz FamDB-2.0.1/famdb.py FamDB-2.0.1/famdb_*.py\" | tee -a $(pwd)/configure_dfam39.sh\n \t\t#echo \"mv FamDB-2.0.1/*.py . && rm -rf 2.0.1.tar.gz FamDB-2.0.1/\" | tee -a $(pwd)/configure_dfam39.sh\n \t\t#echo \"\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"# save the min_init partition as a backup, just in case!\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"mv ${library_path}/min_init.0.h5 ${library_path}/min_init.0.h5.bak\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"# Rerun RepeatMasker configuration\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"perl ./configure -libdir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/share/RepeatMasker/Libraries/|g') \\\n \t\t-trf_prgm $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin/trf|g') \\\n \t\t\t-rmblast_dir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin|g') \\\n \t\t-hmmer_dir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin|g') \\\n \t\t-abblast_dir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin|g') \\\n \t\t-crossmatch_dir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin|g') \\\n \t\t-default_search_engine rmblast\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"\" | tee -a $(pwd)/configure_dfam39.sh\n \t\tsed -i '/^perl/ s/\\s\\s*/ /g' $(pwd)/configure_dfam39.sh\n \t\techo \"########################################################\"\n \t\techo \"\"\n \t\techo \"Following these steps should enable you to use Dfam 3.9 with RepeatMasker and Earl Grey\"\n \t\techo \"You will not see this message again if configuration is successful\"\n\t\t\techo \"If you only want partition 0, you do not have to do anything except rerun your Earl Grey command\"\n \t\techo \"If you are still having trouble, please open an issue on the Earl Grey github and we will try to help!\"\n\t\t\ttouch ${library_path}/.earlgrey.config.complete\n \t\texit 2\n\tfi\n}\n\n# Main #\n\nwhile getopts g:s:o:t:f:i:r:c:l:m:d:n:a:e:q:h option\ndo\n\tcase \"${option}\" \n\t\tin\n\t\tg) genome=${OPTARG};;\n\t\ts) species=${OPTARG};;\n\t\to) directory=${OPTARG};;\n\t\tt) ProcNum=${OPTARG};;\n\t\tf) Flank=${OPTARG};;\n\t\ti) num=${OPTARG};;\n\t\tr) RepSpec=${OPTARG};;\n\t\tl) startCust=${OPTARG};;\n\t\tc) cluster=${OPTARG};;\n\t\tm) margin=${OPTARG};;\n\t\td) softMask=${OPTARG};;\n\t\tn) no_seq=${OPTARG};;\n\t\ta) min_seq=${OPTARG};;\n\t\te) heli=${OPTARG};;\n\t\tq) quietBar=${OPTARG};;\n\t\th) usage; exit 0;; \n\tesac\ndone\n\nSECONDS=0\n\n# Step 1 - set up the directories and make sure all modules are present\n\nstage=\"Checking Parameters\" && runningTea\nSCRIPT_DIR=/data/toby/EarlGrey/scripts/\ndfamCheck\nChecks\nstage=\"Making Directories\" && runningTea\nmakeDirectory\n\n# Start Logs - only bother starting if everything is present and correct\nexec > >(tee \"${OUTDIR}/${species}EarlGrey.log\") 2>&1\nstage=\"Cleaning Genome\" && runningTea\nprepGenome\nsleep 1\n\n# Step 2 - initial annotation, either de novo, or if required an initial RepeatMasker followed by a de novo run\nif [ -z \"$RepSpec\" ]; then\n\tsleep 1\nelse\n\tif [ ! -f ${OUTDIR}/${species}_RepeatMasker/*.masked ]; then\n\t\tstage=\"Getting RepeatMasker Sequences for $RepSpec and Saving as Fasta\" && runningTea\n\t\tgetRepeatMaskerFasta\n\t\tstage=\"Running Initial Mask with Known Repeats\" && runningTea\n\t\tfirstMask\n\telse\n\t\tstage=\"Genome has already been masked, skipping...\" && runningTea\n\t\tgetRepeatMaskerFasta\n\tfi\nfi\n\nif [ -z \"$startCust\" ]; then\n\tsleep 1\nelse\n\tif [ ! -f ${OUTDIR}/${species}_RepeatMasker/*.masked ]; then\n\t\tstage=\"Running Initial Mask with Known Repeats in Custom Library\" && runningTea\n\t\tfirstMaskCustomLib\n\telse\n\t\tstage=\"Genome has already been masked, skipping...\" && runningTea\n\tfi\nfi\n\n# Step 3 - make a database from the genome, and then run a de novo TE annotation using RepeatModeler2\nif [ ! -f ${OUTDIR}/${species}_Database/${species}-families.fa ]; then\n\tstage=\"Detecting Novel Repeats\" && runningTea\n\tbuildDB\n\tdeNovo1\n\tsleep 1\nelse\n\tstage=\"De novo repeats have already been detected, skipping...\" && runningTea\n\tsleep 1\nfi\n\n# Step 4 - run TEstrainer, which runs a BLAST, extract, extent, trim, on the the de novo repeat library to refine consensus sequences\nif [ ! -s ${OUTDIR}/${species}_strainer/${species}-families.fa.strained ]; then\n\tstage=\"Straining TEs and Refining de novo Consensus Sequences\" && runningTea\n\tif [ -f ${OUTDIR}/${species}_strainer/${species}-families.fa.strained ]; then\n\t\trm -r ${OUTDIR}/${species}_strainer/${species}-families.fa.strained ${OUTDIR}/${species}_strainer/TS*\n\t\tstrainer\n\telif [ -d ${OUTDIR}/${species}_strainer/TS_${species}-families.fa_* ]; then\n\t\tstage=\"Resuming TEstrainer\" && runningTea\n\t\tstrainerResume\n\telse\n\t\tstrainer\n\tfi\n\tif [ ! -s ${OUTDIR}/${species}_strainer/${species}-families.fa.strained ]; then\n\t\techo \"WARNING: TEstrainer failed to produce a strain file, please check the log file for more information. If you have run an intial mask with known repeats, this could be due RepeatModeler2 failing to identify any new repeats. Please check if this is expected.\"\n\tfi\n\tsleep 1\nelse\n\tstage=\"TEs already strained, skipping...\" && runningTea\n\tlatestFile=${OUTDIR}/${species}_strainer/${species}-families.fa.strained\n\tsleep 1\nfi\n\n# Step 4.5 - cluster TE library IF REQUIRED\n\nif [ \"$cluster\" == \"yes\" ]; then\n\tstage=\"Clustering TE sequences to Wicker TE Family Level (80-80-80 rule)\" && runningTea\n\tclust\n\tsleep 1\nfi\n\n# Stage 5 - run the final RepeatMasker annotation using the refined TE consensus library\nif [ ! -f ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/*.tbl ]; then\n\tstage=\"Identifying Repeats Using Species-Specific Library\" && runningTea\n\tnovoMask\n\tif [ ! -f ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/*.tbl ]; then\n\t\techo \"ERROR: RepeatMasker failed, please check logs\" && exit 2\n\tfi\n\tsleep 1\nelse\n\tstage=\"Final masking already complete, skipping...\" && runningTea\n\tsleep 1\nfi\n\n# Stage 5.5 - Run HELIANO as an optional step\n\nif [ \"$heli\" == \"yes\" ]; then\n\tif [ ! -s ${OUTDIR}/${species}_heliano/*/RC.representative.gff ]; then\n\t\tstage=\"Running HELIANO to Detect Helitrons\" && runningTea\n\t\ttimestamp=$(date +\"%Y%m%d_%H%M\")\n\t\theliano_optional\n\telse\n\t\tstage=\"HELITRON detection already complete, skipping...\" && runningTea\n\t\thelitron_gff=\"$(realpath $(ls -td -- ${OUTDIR}/${species}_heliano/*/ | head -n 1) || true)/RC.representative.gff\"\n\t\techo \"HELITRON GFF: $helitron_gff\"\n\tfi\n\tsleep 1\nfi\n\n# Stage 6\nif [ ! -f ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed ] ; then\n\tstage=\"Defragmenting Repeats\" && runningTea\n\tmergeRep\n\tsleep 1\nelse\n\tstage=\"Repeats already defragmented, skipping...\" && runningTea\n\tsleep 1\nfi\n\n# Stage 7\nstage=\"Generating Summary Plots\" && runningTea\ncharts\ncalcDivRL\nsleep 1\n\n# Stage 8\nstage=\"Tidying Directories and Organising Important Files\" && runningTea\nsweepUp\nsleep 1\n\n# Stage 8.5 Optional Softmask\nif [ \"$softMask\" == \"yes\" ]; then\n\tstage=\"Generating Softmasked Genome\" && runningTea\n\tgunzip ${genome%.prep}.bak.gz && bedtools maskfasta -fi ${genome%.prep}.bak -bed ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed -fo ${OUTDIR}/${species}_summaryFiles/${species}.softmasked.fasta -soft && gzip ${genome%.prep}.bak\n\tsleep 1\nfi\n\n# Stage 9\ntime=$(convertsecs $SECONDS)\nstage=\"Done in $time\" && runningTea\nsleep 5\n\n# Stage 10\nstage=\"TE library, Summary Figures, and TE Quantifications in Standard Formats Can Be Found in ${OUTDIR}/${species}_summaryFiles/\" && runningTea\ncat ${OUTDIR}/${species}_summaryFiles/${species}.highLevelCount.kable\nsleep 5\n"
},
{
"path": "earlGreyAnnotationOnly",
"content": "#!/bin/bash\nset -eo pipefail\n\nusage()\n{\n\techo \"\t#############################\n\tearlGrey version 7.2.4 (AnnotationOnly)\n\tRequired Parameters:\n\t\t-g == genome.fasta\n\t\t-s == species name\n\t\t-o == output directory\n -l == Starting consensus library for annotation (in fasta format)\n\n\tOptional Parameters:\n\t\t-t == Number of Threads (DO NOT specify more than are available)\n \t-r == RepeatMasker species for addition to custom library (Default: None)\n\t\t-m == Remove putative spurious TE annotations <100bp? (yes/no, Default: no)\n\t\t-d == Create soft-masked genome at the end? (yes/no, Default: no)\n\t\t-e == Run HELIANO as an optional step to detect Helitrons (yes/no, Default: no)\n\t\t-h == Show help\n\n\tExample Usage:\n\n\tearlGreyAnnotationOnly -g bombyxMori.fasta -s bombyxMori -o /home/toby/bombyxMori/repeatAnnotation/ -l bombyx-families.fa.strained -t 16\n\n\tQueries can be sent to:\n\ttobias.baril[at]unine.ch\n\n\tPlease make use of the GitHub Issues and Discussion Tabs at: https://github.com/TobyBaril/EarlGrey\n\t#############################\"\n}\n\n# Subprocess Make Directories #\nmakeDirectory()\n{\n\tdirectory=$(realpath ${directory})\n\tmkdir -p ${directory}/${species}_EarlGrey/ && OUTDIR=${directory}/${species}_EarlGrey\n \tmkdir -p ${OUTDIR}/${species}_Curated_Library/\n\tmkdir -p $OUTDIR/${species}_RepeatMasker_Against_Custom_Library/\n\tmkdir -p $OUTDIR/${species}_RepeatLandscape/\n\tmkdir -p $OUTDIR/${species}_mergedRepeats/\n\tmkdir -p ${OUTDIR}/${species}_summaryFiles/\n\tif [ ! -z \"$heli\" ]; then\n\t\tmkdir -p ${OUTDIR}/${species}_heliano/\n\tfi\n}\n\n# Subprocess PrepGenome #\nprepGenome()\n{\n\tif [ ! -L ${genome} ]; then\n\t\tgenome=$(realpath ${genome})\n\tfi\n\tif [ -L $genome ]; then\n\t\tgenome=$(realpath -s ${genome})\n\tfi\n\tif [ ! -f ${genome}.prep ] || [ ! -f ${genome}.dict ]; then\n\t\tcp ${genome} ${genome}.bak && gzip -f ${genome}.bak \n\t\tsed '/>/ s/[[:space:]].*//g; /^$/d' ${genome} > ${genome}.tmp\n\t\t${SCRIPT_DIR}/headSwap.sh -i ${genome}.tmp -o ${genome}.prep && rm ${genome}.tmp\n\t\tmv ${genome}.tmp.dict ${genome}.dict\n\t\tsed -i '/^>/! s/[DVHBPE]/N/g' ${genome}.prep\n\t\tdict=${genome}.dict\n\t\tgenOrig=${genome}\n\t\tgenome=${genome}.prep\n\telse\n\t\tdict=${genome}.dict\n\t\tgenOrig=${genome}\n\t\tgenome=${genome}.prep\n\tfi\n}\n\n# Subprocess novoMask\n# Run RepeatMasker with the final processed library\nnovoMask()\n{\n\tif [ -z \"$RepSpec\" ]; then\n\t\tcd ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/\n\t\trmthreads=$(( ProcNum / 4 ))\n\t\tRepeatMasker -lib $latestFile -no_is -lcambig -s -a -pa $rmthreads -dir ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/ $genome\n\telse\n\t\tmkdir -p $OUTDIR/${species}_Curated_Library/\n\t\tcd ${OUTDIR}/${species}_Curated_Library/\n\t\tcat $latestFile $RepSub > ${species}_combined_library.fasta\n\t\tlatestFile=$(realpath ${species}_combined_library.fasta)\n\t\tcd ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/\n\t\trmthreads=$(( ProcNum / 4 ))\n\t\tRepeatMasker -lib $latestFile -no_is -lcambig -s -a -pa $rmthreads -dir ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/ $genome\n\tfi\n}\n\n# Subprocess heliano\n# Run HELIANO as an optional step, then replace overlapping repeats in the merged output with HELIANO outputs\n### TODO:\nheliano_optional()\n{\n\tcd ${OUTDIR}/${species}_heliano/\n\theliano -g $genome --nearest -dn 6000 -flank_sim 0.5 -o ${OUTDIR}/${species}_heliano/HEL_${timestamp} -w 10000 -n $ProcNum\n\tawk '{OFS=\"\\t\"}{print $1, \"HELIANO\", \"RC/Helitron\", $2+1, $3, $5, $6, \".\", \"ID=\"$9\"_\"$11\";shortTE=F\"}' ${OUTDIR}/${species}_heliano/HEL_${timestamp}/RC.representative.bed > ${OUTDIR}/${species}_heliano/HEL_${timestamp}/RC.representative.gff\n\thelitron_gff=${OUTDIR}/${species}_heliano/HEL_${timestamp}/RC.representative.gff\n}\n\n# Subprocess rcMergeRepeats\n# Defragment repeat sequences to adjust for insertion times\nmergeRep()\n{\n\tmkdir ${OUTDIR}/${species}_mergedRepeats/looseMerge\n\tif [ -s \"$helitron_gff\" ]; then\n\t\techo \"Running loose merge with HELIANO output\"\n\t\t${SCRIPT_DIR}/rcMergeRepeatsLoose -f $genome -s $species -d ${OUTDIR}/${species}_mergedRepeats/looseMerge -u ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).out -q ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).tbl -t $ProcNum -b ${dict} -m $margin -e $helitron_gff\n\telse\n\t\t${SCRIPT_DIR}/rcMergeRepeatsLoose -f $genome -s $species -d ${OUTDIR}/${species}_mergedRepeats/looseMerge -u ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).out -q ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).tbl -t $ProcNum -b ${dict} -m $margin\n\tfi\n\n\tif [ -f \"${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed\" ]; then\n\t\tawk '{OFS=\"\\t\"}{print $1, $2, $3, $4, $5, $6, $7, $8, toupper($9)}' ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff > ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff.1 && mv ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff{.1,}\n\tfi\n\n\tif [ ! -f \"${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed\" ]; then\n\t\techo \"ERROR: loose merge defragmentation failed, trying strict merge...\"\n\t\tcd ${OUTDIR}/${species}_mergedRepeats/\n\t\tif [ -s \"$helitron_gff\" ]; then\n\t\t\t${SCRIPT_DIR}/rcMergeRepeats -f $genome -s $species -d ${OUTDIR}/${species}_mergedRepeats/ -u ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).out -q ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).tbl -t $ProcNum -b ${dict} -m $margin -e $helitron_gff\n\t\telse\n\t\t\t${SCRIPT_DIR}/rcMergeRepeats -f $genome -s $species -d ${OUTDIR}/${species}_mergedRepeats/ -u ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).out -q ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).tbl -t $ProcNum -b ${dict} -m $margin\n\t\tfi\n\t\tif [ ! -f \"${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.bed\" ]; then\n\t\t\techo \"ERROR: strict merge also failed, check ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).out looks as expected\"\n\t\t\texit 2\n\t\tfi\n\tfi\n}\n\n# Subprocess pieChart\n# Generate a pie chart summary of TE content\ncharts()\n{\n\tcd ${OUTDIR}/${species}_summaryFiles/\n\tif [ -f \"${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed\" ]; then\n\t\t${SCRIPT_DIR}/autoPie.sh -i ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.summary -t ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).tbl -p ${OUTDIR}/${species}_summaryFiles/${species}.summaryPie.pdf -o ${OUTDIR}/${species}_summaryFiles/${species}.highLevelCount.txt\n\telse\n\t\t${SCRIPT_DIR}/autoPie.sh -i ${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.summary -t ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).tbl -p ${OUTDIR}/${species}_summaryFiles/${species}.summaryPie.pdf -o ${OUTDIR}/${species}_summaryFiles/${species}.highLevelCount.txt\n\tfi\n}\n\n# Subprocess calcDivRL\n# Calculate divergence estimates\ncalcDivRL()\n{\n\tcd ${OUTDIR}/${species}_RepeatLandscape\n\tif [ -z \"$RepSpec\" ]; then\n\t\tpython ${SCRIPT_DIR}/divergenceCalc/divergence_calc.py -l $latestFile -g $genOrig -i ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff -o ${OUTDIR}/${species}_RepeatLandscape/${species}.filteredRepeats.withDivergence.gff -t $ProcNum\n\t\tRscript ${SCRIPT_DIR}/divergenceCalc/divergence_plot.R -s $species -g ${OUTDIR}/${species}_RepeatLandscape/${species}.filteredRepeats.withDivergence.gff -o ${OUTDIR}/${species}_RepeatLandscape/ && \\\n\t\tawk -F'\\t' 'BEGIN{OFS=\"\\t\"} {gsub(\"NAME=\",\"Name=\",$9); gsub(\"TSTART=\",\"tstart=\",$9); gsub(\"TEND=\",\"tend=\",$9); gsub(\"SHORTTE=\",\"shortte=\",$9); gsub(\"TEGROUP=\",\"tegroup=\",$9); gsub(\"KIMURA80=\",\"kimura80=\",$9); gsub(\"NESTED=\", \"nested=\", $9); print}' ${OUTDIR}/${species}_RepeatLandscape/${species}.filteredRepeats.withDivergence.gff > ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff\n\t\trm -rf ${OUTDIR}/${species}_RepeatLandscape/tmp/\n\t\tcp ${OUTDIR}/${species}_RepeatLandscape/*.pdf ${OUTDIR}/${species}_summaryFiles/\n\t\tcp ${OUTDIR}/${species}_RepeatLandscape/*_summary_table.tsv ${OUTDIR}/${species}_summaryFiles/${species}_divergence_summary_table.tsv\n\telse\n\t\tpython ${SCRIPT_DIR}/divergenceCalc/divergence_calc.py -l ${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta -g $genOrig -i ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff -o ${OUTDIR}/${species}_RepeatLandscape/${species}.filteredRepeats.withDivergence.gff -t $ProcNum\n\t\tRscript ${SCRIPT_DIR}/divergenceCalc/divergence_plot.R -s $species -g ${OUTDIR}/${species}_RepeatLandscape/${species}.filteredRepeats.withDivergence.gff -o ${OUTDIR}/${species}_RepeatLandscape/ && \\\n\t\tawk -F'\\t' 'BEGIN{OFS=\"\\t\"} {gsub(\"NAME=\",\"Name=\",$9); gsub(\"TSTART=\",\"tstart=\",$9); gsub(\"TEND=\",\"tend=\",$9); gsub(\"SHORTTE=\",\"shortte=\",$9); gsub(\"TEGROUP=\",\"tegroup=\",$9); gsub(\"KIMURA80=\",\"kimura80=\",$9); gsub(\"NESTED=\", \"nested=\", $9); print}' ${OUTDIR}/${species}_RepeatLandscape/${species}.filteredRepeats.withDivergence.gff > ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff\n\t\trm -rf ${OUTDIR}/${species}_RepeatLandscape/tmp/\n\t\tcp ${OUTDIR}/${species}_RepeatLandscape/*.pdf ${OUTDIR}/${species}_summaryFiles/\n\t\tcp ${OUTDIR}/${species}_RepeatLandscape/*_summary_table.tsv ${OUTDIR}/${species}_summaryFiles/${species}_divergence_summary_table.tsv\n\tfi\n}\n\n# Subprocess sweepUp\n# Puts required files into a summary folder\nsweepUp()\n{\n\tcd ${OUTDIR}/${species}_summaryFiles/\n\tif [ -f \"${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed\" ] && [ ! -f \"${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta\" ]; then\n\t\tcp ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff $latestFile ${OUTDIR}/${species}_summaryFiles/\n\telif [ -f \"${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed\" ] && [ -f \"${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta\" ]; then\n\t\tcp ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff ${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta $latestFile ${OUTDIR}/${species}_summaryFiles/\n\telif [ ! -f \"${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed\" ] && [ ! -f \"${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta\" ]; then\n\t\tcp ${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.bed ${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.gff $latestFile ${OUTDIR}/${species}_summaryFiles/\n\telif [ ! -f \"${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed\" ] && [ -f \"${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta\" ]; then\n\t\tcp ${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.bed ${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.gff ${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta $latestFile ${OUTDIR}/${species}_summaryFiles/\n\telse\n\t\techo \"Error: Cannot find required summary files, please check one of the following is present: ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed and/or ${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.bed\"\n\tfi\n}\n\n# Subprocess runningTea \nrunningTea()\n{\n\techo \" \n ) (\n\t ( ) )\n\t ) ( (\n\t _______)_\n\t.-'---------| \n ( C|/\\/\\/\\/\\/|\n\t'-./\\/\\/\\/\\/|\n\t '_________'\n\t '-------'\n\t<<< $stage >>>\"\n}\n\n# Subprocess GetTime\nconvertsecs() \n{\n\th=$(bc <<< \"${1}/3600\")\n\tm=$(bc <<< \"(${1}%3600)/60\")\n\ts=$(bc <<< \"${1}%60\")\n\tprintf \"%02d:%02d:%05.2f\\n\" $h $m $s\n}\n\n# Subprocess Checks\nChecks()\n{\n\tif [ -z \"$genome\" ] || [ -z \"$species\" ] || [ -z \"$directory\" ] || [ -z \"$startCust\" ] ; then\n\t\tusage; exit 1\n\tfi\n\n\tif [ -z \"$ProcNum\" ] ; then\n\t\tProcNum=1; echo \"$ProcNum Cores Will Be Used\"\n\telse\n\t\techo \"$ProcNum Cores Will Be Used\"\n\tfi\n\n\tif [ -z \"$RepSpec\" ]; then\n\t\techo \"RepeatMasker species not specified, running Earl Grey without known repeats\"\n\telse\n\t\techo \"Running Earl Grey with sequences from configured databases in addition to custom library\"\n\tfi\n\t\n\tif [ -z \"$softMask\" ] || [ \"$softMask\" == \"no\" ] ; then\n\t\tsoftMask=no; echo \"Softmasked genome will not be generated\"\n\telif [ \"$softMask\" == \"yes\" ]; then\n\t\tsoftMask=yes; echo \"Softmasked genome will be generated\"\n\telse\n\t\tsoftMask=no; echo \"Softmask not specified using (yes/no). Using default parameter (no).\"\n\tfi\n\n\tif [ \"$margin\" == \"yes\" ] ; then\n\t\tmargin=yes; echo \"Short TE sequences <100bp will be removed from annotation\"\n\telif [ -z \"$margin\" ] || [ \"$margin\" == \"no\" ]; then\n\t\tmargin=no; echo \"Short TE sequences <100bp will not be removed from annotation\"\n\telse\n\t\tmargin=no; echo \"Margin not specified using (yes/no). Using default parameter (no).\"\n\tfi\n\n\tif [ ! -d \"$SCRIPT_DIR\" ]; then\n\t\techo \"ERROR: Script directory variable not set, please run the configure script in the Earl Grey directory before attempting to run Earl Grey\"; exit 1\n\tfi\n\n\tif [ ! -d \"$SCRIPT_DIR/TEstrainer/\" ]; then\n\t\techo \"ERROR: teStrainer module not found, please check all modules are present and run the configure script in the Earl Grey directory before attempting to run Earl Grey\"; exit 1\n\tfi\n\n\tif [ \"$heli\" == \"yes\" ] ; then\n\t\theli=yes; echo \"HELITRON detection will be run using HELIANO\"\n\telif [ -z \"$heli\" ] || [ \"$heli\" == \"no\" ]; then\n\t\theli=no; echo \"HELITRON detection will not be run\"\n\telse\n\t\theli=no; echo \"HELITRON detection not specified using (yes/no). Using default parameter (no).\"\n\tfi\n\n\techo \"Please cite the following paper when using this software:\"\n\techo \"Baril, T., Galbraith, J. and Hayward, A., 2024. Earl Grey: a fully automated user-friendly transposable element annotation and analysis pipeline. Molecular Biology and Evolution, 41(4), p.msae068.\"\n\n\t# Dfam 3.9 checks\n\tlibrary_path=$(which RepeatMasker | sed 's|/bin/RepeatMasker|/share/RepeatMasker/Libraries/famdb|g')\n\texpected_file=\"${library_path}/dfam39_full.0.h5\"\n\tfile_count=$(find \"$library_path\" -maxdepth 1 -type f | wc -l)\n\tif [ $file_count -eq 2 ] && [ -f $expected_file ] && [ -f ${library_path}/rmlib.config ] ; then\n \t\techo \"WARNING: Earl Grey v6.1.0 has updated to Dfam v3.9.\" \n \t\techo \"Before using Earl Grey, you MUST download the required partitions from Dfam (https://dfam.org/releases/current/families/FamDB/)\"\n \t\techo \"These must be located at ${library_path}/. Then, you must reconfigure RepeatMasker before using Earl Grey.\"\n \t\techo \"I have written the required commands for you to faciliate the process.\"\n \t\techo \"Please follow these carefully to ensure successful configuration of RepeatMasker with Dfam 3.9.\"\n \t\techo \"As a note, these DB paritions are BIG, so only download the ones you need for your studies (or, if you do need all of them, make sure you have the space for them)\"\n \t\techo \"########################################################\"\n \t\techo \"A script for modification and automation of these steps has been generated in $(pwd)/configure_dfam39.sh\"\n \t\techo \"Please open the script in your preferred text editor (e.g. nano or vim) and change the partition numbers in [] to those that you would like to download\"\n \t\techo \"then, make the script executable (chmod +x $(pwd)/configure_dfam39.sh) and run it: ./configure_dfam39.sh\"\n \t\techo \"Alternatively, you can run the steps below one-by-one to achieve the same outcome:\"\n \t\techo \"########################################################\"\n \t\techo \"\"\n \t\tif [ -f \"$(pwd)/configure_dfam39.sh\" ]; then\n \t\trm $(pwd)/configure_dfam39.sh\n \t\tfi\n \t\techo \"# first, change directory to the famdb library location\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"cd ${library_path}/\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"# download the partitions you require from Dfam 3.9. In the below, change the numbers or range inside the square brackets to choose your subsets.\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"# e.g. to download partitions 0 to 10: [0-10]; or to download partitions 3,5, and 7: [3,5,7]; [0-16] is ALL PARTITIONS\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"curl -o 'dfam39_full.#1.h5.gz' 'https://dfam.org/releases/current/families/FamDB/dfam39_full.[0-16].h5.gz'\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"# decompress Dfam 3.9 paritions\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"gunzip *.gz\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"# move up to RepeatMasker main directory\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"cd $(which RepeatMasker | sed 's|/bin/RepeatMasker|/share/RepeatMasker/|g')\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"\" | tee -a $(pwd)/configure_dfam39.sh\n \t\t#echo \"# Replace famdb with version 2.0.1 (required for the Dfam 3.9)\" | tee -a $(pwd)/configure_dfam39.sh\n \t\t#echo \"wget https://github.com/Dfam-consortium/FamDB/archive/refs/tags/2.0.1.tar.gz && tar --wildcards -zxvf 2.0.1.tar.gz FamDB-2.0.1/famdb.py FamDB-2.0.1/famdb_*.py\" | tee -a $(pwd)/configure_dfam39.sh\n \t\t#echo \"mv FamDB-2.0.1/*.py . && rm -rf 2.0.1.tar.gz FamDB-2.0.1/\" | tee -a $(pwd)/configure_dfam39.sh\n \t\t#echo \"\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"# save the min_init partition as a backup, just in case!\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"mv ${library_path}/min_init.0.h5 ${library_path}/min_init.0.h5.bak\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"# Rerun RepeatMasker configuration\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"perl ./configure -libdir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/share/RepeatMasker/Libraries/|g') \\\n \t\t-trf_prgm $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin/trf|g') \\\n \t\t-rmblast_dir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin|g') \\\n \t\t-hmmer_dir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin|g') \\\n \t\t-abblast_dir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin|g') \\\n \t\t-crossmatch_dir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin|g') \\\n \t\t-default_search_engine rmblast\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"\" | tee -a $(pwd)/configure_dfam39.sh\n \t\tsed -i '/^perl/ s/\\s\\s*/ /g' $(pwd)/configure_dfam39.sh\n \t\techo \"########################################################\"\n \t\techo \"\"\n \t\techo \"Following these steps should enable you to use Dfam 3.9 with RepeatMasker and Earl Grey\"\n \t\techo \"You will not see this message again if configuration is successful\"\n\t\t\techo \"If you only want partition 0, you do not have to do anything except rerun your Earl Grey command\"\n \t\techo \"If you are still having trouble, please open an issue on the Earl Grey github and we will try to help!\"\n\t\t\ttouch ${library_path}/.earlgrey.config.complete\n \t\texit 2\n\tfi\n}\n\n# Main #\n\nwhile getopts g:s:o:t:r:l:m:d:e:h option\ndo\n\tcase \"${option}\" \n\t\tin\n\t\tg) genome=${OPTARG};;\n\t\ts) species=${OPTARG};;\n\t\to) directory=${OPTARG};;\n\t\tt) ProcNum=${OPTARG};;\n\t\tf) Flank=${OPTARG};;\n\t\ti) num=${OPTARG};;\n\t\tr) RepSpec=${OPTARG};;\n\t\tl) startCust=${OPTARG};;\n\t\tc) cluster=${OPTARG};;\n\t\tm) margin=${OPTARG};;\n\t\td) softMask=${OPTARG};;\n\t\tn) no_seq=${OPTARG};;\n\t\ta) min_seq=${OPTARG};;\n\t\te) heli=${OPTARG};;\n\t\th) usage; exit 0;; \n\tesac\ndone\n\nSECONDS=0\n\n# Step 1 - set up the directories and make sure all modules are present\n\nstage=\"Checking Parameters\" && runningTea\nSCRIPT_DIR=/data/toby/EarlGrey/scripts/\nChecks\nstage=\"Making Directories\" && runningTea\nmakeDirectory\n\n# Start Logs - only bother starting if everything is present and correct\nexec > >(tee \"${OUTDIR}/${species}EarlGrey.log\") 2>&1\nstage=\"Cleaning Genome\" && runningTea\nprepGenome\nsleep 1\n\n# Stage 2 - Assign libraries to variables\nlatestFile=$(realpath $startCust)\nif [ ! -z \"$RepSpec\" ]; then\n if [[ $(which RepeatMasker) == *\"bin\"* ]]; then\n\t\tlibpath=\"$(which RepeatMasker | sed 's|bin/RepeatMasker|share/RepeatMasker/Libraries/famdb/|')\"\n\telse\n\t\tlibpath=\"$(which RepeatMasker | sed 's|/[^/]*$||g')/Libraries/famdb/\"\n\tfi\n\tif [[ $(which RepeatMasker) == *\"bin\"* ]]; then\n\t\tPATH=$PATH:\"$(which RepeatMasker | sed 's|bin/RepeatMasker|share/RepeatMasker/|g')\"\n\tfi\n famdb.py -i $libpath families -f fasta_name --include-class-in-name -a -d --curated $RepSpec > ${OUTDIR}/${species}_Curated_Library/${RepSpec}.RepeatMasker.lib\n\tRepSub=${OUTDIR}/${species}_Curated_Library/${RepSpec}.RepeatMasker.lib\nfi\n\n# Stage 5 - run the final RepeatMasker annotation using the refined TE consensus library\nif [ ! -f ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/*.tbl ]; then\n\tstage=\"Identifying Repeats Using Species-Specific Library\" && runningTea\n\tnovoMask\n\tif [ ! -f ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/*.tbl ]; then\n\t\techo \"ERROR: RepeatMasker failed, please check logs\" && exit 2\n\tfi\n\tsleep 1\nelse\n\tstage=\"Final masking already complete, skipping...\" && runningTea\n\tsleep 1\nfi\n\n# Stage 5.5 - Run HELIANO as an optional step\n\nif [ \"$heli\" == \"yes\" ]; then\n\tif [ ! -s ${OUTDIR}/${species}_heliano/*/RC.representative.gff ]; then\n\t\tstage=\"Running HELIANO to Detect Helitrons\" && runningTea\n\t\ttimestamp=$(date +\"%Y%m%d_%H%M\")\n\t\theliano_optional\n\telse\n\t\tstage=\"HELITRON detection already complete, skipping...\" && runningTea\n\t\thelitron_gff=\"$(realpath $(ls -td -- ${OUTDIR}/${species}_heliano/*/ | head -n 1) || true)/RC.representative.gff\"\n\t\techo \"HELITRON GFF: $helitron_gff\"\n\tfi\n\tsleep 1\nfi\n\n# Stage 6\nif [ ! -f ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed ] ; then\n\tstage=\"Defragmenting Repeats\" && runningTea\n\tmergeRep\n\tsleep 1\nelse\n\tstage=\"Repeats already defragmented, skipping...\" && runningTea\n\tsleep 1\nfi\n\n# Stage 7\nstage=\"Generating Summary Plots\" && runningTea\ncharts\ncalcDivRL\nsleep 1\n\n# Stage 8\nstage=\"Tidying Directories and Organising Important Files\" && runningTea\nsweepUp\nsleep 1\n\n# Stage 8.5 Optional Softmask\nif [ \"$softMask\" == \"yes\" ]; then\n\tstage=\"Generating Softmasked Genome\" && runningTea\n\tgunzip ${genome%.prep}.bak.gz && bedtools maskfasta -fi ${genome%.prep}.bak -bed ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed -fo ${OUTDIR}/${species}_summaryFiles/${species}.softmasked.fasta -soft && gzip ${genome%.prep}.bak\n\tsleep 1\nfi\n\n# Stage 9\ntime=$(convertsecs $SECONDS)\nstage=\"Done in $time\" && runningTea\nsleep 5\n\n# Stage 10\nstage=\"TE library, Summary Figures, and TE Quantifications in Standard Formats Can Be Found in ${OUTDIR}/${species}_summaryFiles/\" && runningTea\ncat ${OUTDIR}/${species}_summaryFiles/${species}.highLevelCount.kable\nsleep 5\n"
},
{
"path": "earlGreyLibConstruct",
"content": "#!/bin/bash\nset -eo pipefail\n\nusage()\n{\n echo \" #############################\n earlGrey version 7.2.4 (Library Construction Only)\n Required Parameters:\n -g == genome.fasta\n -s == species name\n -o == output directory\n\n Optional Parameters:\n -t == Number of Threads (DO NOT specify more than are available)\n -r == RepeatMasker search term (e.g arthropoda/eukarya)\n -l == Starting consensus library for an initial mask (in fasta format)\n -i == Number of Iterations to BLAST, Extract, Extend (Default: 10)\n -f == Number flanking basepairs to extract (Default: 1000)\n -n == Max number of sequences used to generate consensus sequences (Default: 20)\n -a == minimum number of sequences required to build a consensus sequence (Default: 3)\n -h == Show help\n\n Example Usage:\n\n earlGrey -g bombyxMori.fasta -s bombyxMori -o /home/toby/bombyxMori/repeatAnnotation/ -t 16\n\n Queries can be sent to:\n tobias.baril[at]unine.ch\n\n Please make use of the GitHub Issues and Discussion Tabs at: https://github.com/TobyBaril/EarlGrey\n #############################\"\n}\n\n# Subprocess Make Directories #\nmakeDirectory()\n{\n directory=$(realpath ${directory})\n mkdir -p ${directory}/${species}_EarlGrey/ && OUTDIR=${directory}/${species}_EarlGrey\n if [ ! -z \"$RepSpec\" ] || [ ! -z \"$startCust\" ] ; then\n mkdir -p $OUTDIR/${species}_RepeatMasker/\n fi\n mkdir -p $OUTDIR/${species}_Database/\n mkdir -p $OUTDIR/${species}_RepeatModeler/\n mkdir -p $OUTDIR/${species}_strainer/\n mkdir -p ${OUTDIR}/${species}_summaryFiles/\n}\n\n# Subprocess PrepGenome #\nprepGenome()\n{\n if [ ! -L ${genome} ]; then\n genome=$(realpath ${genome})\n fi\n if [ -L $genome ]; then\n genome=$(realpath -s ${genome})\n fi\n if [ ! -f ${genome}.prep ] || [ ! -f ${genome}.dict ]; then\n cp ${genome} ${genome}.bak && gzip -f ${genome}.bak \n sed '/>/ s/[[:space:]].*//g; /^$/d' ${genome} > ${genome}.tmp\n ${SCRIPT_DIR}/headSwap.sh -i ${genome}.tmp -o ${genome}.prep && rm ${genome}.tmp\n mv ${genome}.tmp.dict ${genome}.dict\n sed -i.bak '/^>/! s/[DVHBPE]/N/g' ${genome}.prep\n dict=${genome}.dict\n genOrig=${genome}\n genome=${genome}.prep\n else\n dict=${genome}.dict\n genOrig=${genome}\n genome=${genome}.prep\n fi\n}\n\n# Subprocess getRepeatMaskerFasta\n# Generate a copy of the RepeatMasker library subset used for the initial TE mask in FASTA format (only used if a RepeatMasker run is specified)\ngetRepeatMaskerFasta()\n{\n\tif [[ $RepSpec = *\" \"* ]]; then\n\t\techo \"ERROR: You have entered a species name that contains a space, please use the NCBI TaxID rather than name. E.G In place of \\\"Homo sapiens\\\" use \\\"9606\\\"\"\n\t\texit 2\n\tfi\n\tif [[ $(which RepeatMasker) == *\"bin\"* ]]; then\n\t\tlibpath=\"$(which RepeatMasker | sed 's|bin/RepeatMasker|share/RepeatMasker/Libraries/famdb/|')\"\n\telse\n\t\tlibpath=\"$(which RepeatMasker | sed 's|/[^/]*$||g')/Libraries/famdb/\"\n\tfi\n\tif [[ $(which RepeatMasker) == *\"bin\"* ]]; then\n\t\tPATH=$PATH:\"$(which RepeatMasker | sed 's|bin/RepeatMasker|share/RepeatMasker/|')\"\n\tfi\n\tmkdir -p $OUTDIR/${species}_Curated_Library/\n\tfamdb.py -i $libpath families -f fasta_name --include-class-in-name -a -d --curated $RepSpec > ${OUTDIR}/${species}_Curated_Library/${RepSpec}.RepeatMasker.lib\n\tRepSub=${OUTDIR}/${species}_Curated_Library/${RepSpec}.RepeatMasker.lib\n}\n\n# Subprocess firstMask\n# Run the initial RepeatMasker run with the specified species subset (only used if a RepeatMasker run is specified)\nfirstMask()\n{\n cd ${OUTDIR}/${species}_RepeatMasker\n rmthreads=$(( ProcNum / 4 ))\n RepeatMasker -species $RepSpec -no_is -lcambig -s -a -pa $rmthreads -dir $OUTDIR/${species}_RepeatMasker $genome\n if [ ! -f ${OUTDIR}/${species}_RepeatMasker/*.masked ]; then\n echo \"ERROR: RepeatMasker failed, please check logs. This is likely because of an invalid species search term, if issue persists please use NCBI Taxids (E.G Drosophila is replaced with 7125)\"; exit 2\n fi\n}\n\n# Subprocess firstMaskCustomLib\n# Run the initial RepeatMasker run with the specific fasta consensus library\nfirstMaskCustomLib()\n{\n cd ${OUTDIR}/${species}_RepeatMasker\n rmthreads=$(( ProcNum / 4 ))\n RepeatMasker -lib ${startCust} -no_is -lcambig -s -a -pa $rmthreads -dir $OUTDIR/${species}_RepeatMasker $genome\n RepSub=${startCust}\n if [ ! -f ${OUTDIR}/${species}_RepeatMasker/*.masked ]; then\n echo \"ERROR: RepeatMasker failed, please check logs. This is likely because of an invalid custom consensus file, check the fasta file looks as expected\"; exit 2\n fi\n}\n\n\n# Subprocess buildDB \n# if masked genome exists, build database from this. If not, build database from original input genome\nbuildDB()\n{\n if [ -f ${OUTDIR}/${species}_RepeatMasker/*.masked ]; then\n cd ${OUTDIR}/${species}_Database\n BuildDatabase -name ${species} ${OUTDIR}/${species}_RepeatMasker/*.masked\n else\n cd ${OUTDIR}/${species}_Database\n BuildDatabase -name ${species} $genome\n fi\n}\n\n# Subprocess deNovo1\n# Run the initial de novo annotation run with RepeatModeler2.\n# The genomeSampleSizeMax flag is set based on the samplable genome size (contigs >= 40 kb)\n# to avoid RepeatModeler failures on small or fragmented genomes.\ndeNovo1()\n{\n cd ${OUTDIR}/${species}_RepeatModeler\n\n # Compute samplable genome size: sum of contigs >= 40 kb only.\n # RepeatModeler discards contigs shorter than 40 kb during sampling, so using\n # the total genome size would overestimate the sequence available per round.\n GENOME_SIZE=$(awk '/^>/{if(len>=40000)sum+=len; len=0; next}{len+=length($0)} END{if(len>=40000)sum+=len; print sum+0}' \"${genome}\")\n echo \"Samplable genome size (contigs >= 40 kb): ${GENOME_SIZE} bp\"\n\n # Set -genomeSampleSizeMax to the highest RECON round threshold the genome\n # can support. Thresholds are cumulative across rounds (r2=3M, r3=9M, r4=27M,\n # r5=81M, r6=243M), so the samplable size must cover all rounds up to the cap:\n # >= 363M (3+9+27+81+243): no cap -> all 6 rounds\n # >= 120M (3+9+27+81): cap 81M -> rounds 1-5\n # >= 39M (3+9+27): cap 27M -> rounds 1-4\n # >= 12M (3+9): cap 9M -> rounds 1-3\n # < 12M: cap 3M -> rounds 1-2\n SAMPLE_FLAG=\"\"\n if [ \"${GENOME_SIZE}\" -ge 363000000 ] 2>/dev/null; then\n : # genome large enough for all rounds; use RepeatModeler default\n elif [ \"${GENOME_SIZE}\" -ge 120000000 ] 2>/dev/null; then\n echo \"Capping at round 5 (-genomeSampleSizeMax 81000000)\"\n SAMPLE_FLAG=\"-genomeSampleSizeMax 81000000\"\n elif [ \"${GENOME_SIZE}\" -ge 39000000 ] 2>/dev/null; then\n echo \"Capping at round 4 (-genomeSampleSizeMax 27000000)\"\n SAMPLE_FLAG=\"-genomeSampleSizeMax 27000000\"\n elif [ \"${GENOME_SIZE}\" -ge 12000000 ] 2>/dev/null; then\n echo \"Capping at round 3 (-genomeSampleSizeMax 9000000)\"\n SAMPLE_FLAG=\"-genomeSampleSizeMax 9000000\"\n else\n echo \"Capping at round 2 (-genomeSampleSizeMax 3000000)\"\n SAMPLE_FLAG=\"-genomeSampleSizeMax 3000000\"\n fi\n\n RepeatModeler -threads ${ProcNum} -database ${OUTDIR}/${species}_Database/${species} ${SAMPLE_FLAG}\n\n if [ ! -e ${OUTDIR}/${species}_Database/${species}-families.fa ]; then\n echo \"ERROR: RepeatModeler Failed\"\n exit 2\n fi\n}\n\n# Subprocess strainer # CHECK FILE STRUCTURE FOR EARL GREY RUN\n# contains the BLAST, Extract, Extend, Trim pipeline from James Galbraith\nstrainer()\n{\n cd ${OUTDIR}/${species}_strainer/\n\tif [ \"$ProcNum\" -ge 4 ]; then\n\t\tstrainthreads=$(( ProcNum / 4 ))\n\telse\n\t\tstrainthreads=$ProcNum\n\tfi\n ${SCRIPT_DIR}/TEstrainer/TEstrainer_for_earlGrey.sh -g $genome -l ${OUTDIR}/${species}_Database/${species}-families.fa -t ${strainthreads} -f $Flank -r $num -n $no_seq -m $min_seq\n latestFile=\"$(realpath $(ls -td -- ${OUTDIR}/${species}_strainer/*/ | head -n 1) || true)/${species}-families.fa.strained\"\n cp $latestFile ${OUTDIR}/${species}_strainer/\n}\n\n# Subprocess strainerResume\n# resumes interrupted TEstrainer runs\nstrainerResume() {\n cd ${OUTDIR}/${species}_strainer/\n strainDataDir=$(find . -maxdepth 1 -type d -name \"TS_${species}-families.fa_*\")\n if [ ! $(echo \"$strainDataDir\" | wc -l) -eq 1 ]; then\n echo \"ERROR: There are more TEstrainer directories than expected present\"\n exit 2\n fi\n if [ \"$ProcNum\" -ge 4 ]; then\n strainthreads=$(( ProcNum / 4 ))\n else\n strainthreads=$ProcNum\n fi\n ${SCRIPT_DIR}/TEstrainer/TEstrainer_for_earlGrey.sh -g $genome -l ${OUTDIR}/${species}_Database/${species}-families.fa -t ${strainthreads} -f $Flank -r $num -n $no_seq -m $min_seq -d ${strainDataDir} -R\n latestFile=\"$(realpath $(ls -td -- ${OUTDIR}/${species}_strainer/*/ | head -n 1) || true)/${species}-families.fa.strained\"\n cp $latestFile ${OUTDIR}/${species}_strainer/\n}\n\n# Subprocess sweepUp\n# Puts required files into a summary folder\nsweepUp()\n{\n cd ${OUTDIR}/${species}_summaryFiles/\n if [ -s ${latestFile} ]; then\n cp $latestFile ${OUTDIR}/${species}_summaryFiles/ \n else\n echo \"ERROR: TEstrainer failed to generate a TE library, please check logs\" && exit 1\n fi\n}\n\n# Subprocess runningTea \nrunningTea()\n{\n echo \" \n ) (\n ( ) )\n ) ( (\n _______)_\n .-'---------| \n ( C|/\\/\\/\\/\\/|\n '-./\\/\\/\\/\\/|\n '_________'\n '-------'\n <<< $stage >>>\"\n}\n\n# Subprocess GetTime\nconvertsecs() \n{\n h=$(bc <<< \"${1}/3600\")\n m=$(bc <<< \"(${1}%3600)/60\")\n s=$(bc <<< \"${1}%60\")\n printf \"%02d:%02d:%05.2f\\n\" $h $m $s\n}\n\n# Subprocess Checks\nChecks()\n{\n if [ -z \"$genome\" ] || [ -z \"$species\" ] || [ -z \"$directory\" ] ; then\n usage; exit 1\n fi\n\n if [ -z \"$ProcNum\" ] ; then\n ProcNum=1; echo \"$ProcNum Cores Will Be Used\"\n else\n echo \"$ProcNum Cores Will Be Used\"\n fi\n\n if [ -z \"$RepSpec\" ] && [ -z \"$startCust\" ] ; then\n echo \"RepeatMasker species not specified, running Earl Grey without an initial mask with known repeats\"\n else\n echo \"Running Earl Grey with an intial mask with known repeats\"\n fi\n\n if [ -z \"$num\" ] ; then\n num=10; echo \"De Novo Sequences Will Be Extended Through a Maximum of $num Iterations\"\n else\n echo \"De Novo Sequences Will Be Extended Through a Maximum of $num Iterations\"\n fi\n\n if [ -z \"$no_seq\" ] ; then\n no_seq=20; echo \"$no_seq sequences will be used in BEAT consensus generation\"\n else\n echo \"$no_seq sequences will be used in BEAT consensus generation\"\n fi\n\n if [ -z \"$Flank\" ]; then\n Flank=1000; echo \"Blast, Extend, Align, Trim Process Will Add 1000bp to Each End in Each Iteration\"\n else\n echo \"Blast, Extract, Extend, Trim Process Will Add $Flank to Each End in Each Iteration\"\n fi\n\n if [ -z \"$min_seq\" ]; then\n min_seq=3; echo \"Blast, Extend, Align, Trim Process Will Require 3 Sequences to Generate a New Consensus Sequence\"\n else\n echo \"Blast, Extend, Align, Trim Process Will Require $min_seq Sequences to Generate a New Consensus Sequence\"\n fi\n\n if [ ! -d \"$SCRIPT_DIR\" ]; then\n echo \"ERROR: Script directory variable not set, please run the configure script in the Earl Grey directory before attempting to run Earl Grey\"; exit 1\n fi\n\n if [ ! -d \"$SCRIPT_DIR/TEstrainer/\" ]; then\n echo \"ERROR: teStrainer module not found, please check all modules are present and run the configure script in the Earl Grey directory before attempting to run Earl Grey\"; exit 1\n fi\n\n\techo \"Please cite the following paper when using this software:\"\n\techo \"Baril, T., Galbraith, J. and Hayward, A., 2024. Earl Grey: a fully automated user-friendly transposable element annotation and analysis pipeline. Molecular Biology and Evolution, 41(4), p.msae068.\"\n}\n\n# Subprocess Dfam Checks\ndfamCheck()\n{\n\t# Dfam 3.9 checks\n\tlibrary_path=$(which RepeatMasker | sed 's|/bin/RepeatMasker|/share/RepeatMasker/Libraries/famdb|g')\n\texpected_file=\"${library_path}/dfam39_full.0.h5\"\n\tfile_count=$(find \"$library_path\" -maxdepth 1 -type f | wc -l)\n\tif [ $file_count -eq 2 ] && [ -f $expected_file ] && [ -f ${library_path}/rmlib.config ] ; then\n \t\techo \"WARNING: Earl Grey v6.1.0 has updated to Dfam v3.9.\" \n \t\techo \"Before using Earl Grey, you MUST download the required partitions from Dfam (https://dfam.org/releases/current/families/FamDB/)\"\n \t\techo \"These must be located at ${library_path}/. Then, you must reconfigure RepeatMasker before using Earl Grey.\"\n \t\techo \"I have written the required commands for you to faciliate the process.\"\n \t\techo \"Please follow these carefully to ensure successful configuration of RepeatMasker with Dfam 3.9.\"\n \t\techo \"As a note, these DB paritions are BIG, so only download the ones you need for your studies (or, if you do need all of them, make sure you have the space for them)\"\n \t\techo \"########################################################\"\n \t\techo \"A script for modification and automation of these steps has been generated in $(pwd)/configure_dfam39.sh\"\n \t\techo \"Please open the script in your preferred text editor (e.g. nano or vim) and change the partition numbers in [] to those that you would like to download\"\n \t\techo \"then, make the script executable (chmod +x $(pwd)/configure_dfam39.sh) and run it: ./configure_dfam39.sh\"\n \t\techo \"Alternatively, you can run the steps below one-by-one to achieve the same outcome:\"\n \t\techo \"########################################################\"\n \t\techo \"\"\n \t\tif [ -f \"$(pwd)/configure_dfam39.sh\" ]; then\n \t\trm $(pwd)/configure_dfam39.sh\n \t\tfi\n \t\techo \"# first, change directory to the famdb library location\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"cd ${library_path}/\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"# download the partitions you require from Dfam 3.9. In the below, change the numbers or range inside the square brackets to choose your subsets.\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"# e.g. to download partitions 0 to 10: [0-10]; or to download partitions 3,5, and 7: [3,5,7]; [0-16] is ALL PARTITIONS\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"curl -o 'dfam39_full.#1.h5.gz' 'https://dfam.org/releases/current/families/FamDB/dfam39_full.[0-16].h5.gz'\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"# decompress Dfam 3.9 paritions\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"gunzip *.gz\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"# move up to RepeatMasker main directory\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"cd $(which RepeatMasker | sed 's|/bin/RepeatMasker|/share/RepeatMasker/|g')\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"\" | tee -a $(pwd)/configure_dfam39.sh\n \t\t#echo \"# Replace famdb with version 2.0.1 (required for the Dfam 3.9)\" | tee -a $(pwd)/configure_dfam39.sh\n \t\t#echo \"wget https://github.com/Dfam-consortium/FamDB/archive/refs/tags/2.0.1.tar.gz && tar --wildcards -zxvf 2.0.1.tar.gz FamDB-2.0.1/famdb.py FamDB-2.0.1/famdb_*.py\" | tee -a $(pwd)/configure_dfam39.sh\n \t\t#echo \"mv FamDB-2.0.1/*.py . && rm -rf 2.0.1.tar.gz FamDB-2.0.1/\" | tee -a $(pwd)/configure_dfam39.sh\n \t\t#echo \"\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"# save the min_init partition as a backup, just in case!\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"mv ${library_path}/min_init.0.h5 ${library_path}/min_init.0.h5.bak\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"# Rerun RepeatMasker configuration\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"perl ./configure -libdir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/share/RepeatMasker/Libraries/|g') \\\n \t\t-trf_prgm $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin/trf|g') \\\n \t\t\t-rmblast_dir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin|g') \\\n \t\t-hmmer_dir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin|g') \\\n \t\t-abblast_dir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin|g') \\\n \t\t-crossmatch_dir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin|g') \\\n \t\t-default_search_engine rmblast\" | tee -a $(pwd)/configure_dfam39.sh\n \t\techo \"\" | tee -a $(pwd)/configure_dfam39.sh\n \t\tsed -i '/^perl/ s/\\s\\s*/ /g' $(pwd)/configure_dfam39.sh\n \t\techo \"########################################################\"\n \t\techo \"\"\n \t\techo \"Following these steps should enable you to use Dfam 3.9 with RepeatMasker and Earl Grey\"\n \t\techo \"You will not see this message again if configuration is successful\"\n\t\t\techo \"If you only want partition 0, you do not have to do anything except rerun your Earl Grey command\"\n \t\techo \"If you are still having trouble, please open an issue on the Earl Grey github and we will try to help!\"\n\t\t\ttouch ${library_path}/.earlgrey.config.complete\n \t\texit 2\n\tfi\n}\n\n# Main #\n\nwhile getopts g:s:o:t:f:i:r:l:n:a:h option\ndo\n case \"${option}\" \n in\n g) genome=${OPTARG};;\n s) species=${OPTARG};;\n o) directory=${OPTARG};;\n t) ProcNum=${OPTARG};;\n f) Flank=${OPTARG};;\n i) num=${OPTARG};;\n r) RepSpec=${OPTARG};;\n l) startCust=${OPTARG};;\n n) no_seq=${OPTARG};;\n a) min_seq=${OPTARG};;\n h) usage; exit 0;; \n esac\ndone\n\nSECONDS=0\n\n# Step 1 - set up the directories and make sure all modules are present\n\nstage=\"Checking Parameters\" && runningTea\nSCRIPT_DIR=/data/toby/mambaforge/envs/earlgrey_440/share/earlgrey-4.4.0-0/scripts/\ndfamCheck\nChecks\nstage=\"Making Directories\" && runningTea\nmakeDirectory\n\n# Start Logs - only bother starting if everything is present and correct\nexec > >(tee \"${OUTDIR}/${species}EarlGrey.log\") 2>&1\nstage=\"Cleaning Genome\" && runningTea\nprepGenome\nsleep 1\n\n# Step 2 - initial annotation, either de novo, or if required an initial RepeatMasker followed by a de novo run\nif [ -z \"$RepSpec\" ]; then\n sleep 1\nelse\n if [ ! -f ${OUTDIR}/${species}_RepeatMasker/*.masked ]; then\n stage=\"Getting RepeatMasker Sequences for $RepSpec and Saving as Fasta\" && runningTea\n getRepeatMaskerFasta\n stage=\"Running Initial Mask with Known Repeats\" && runningTea\n firstMask\n else\n stage=\"Genome has already been masked, skipping...\" && runningTea\n getRepeatMaskerFasta\n fi\nfi\n\nif [ -z \"$startCust\" ]; then\n sleep 1\nelse\n if [ ! -f ${OUTDIR}/${species}_RepeatMasker/*.masked ]; then\n stage=\"Running Initial Mask with Known Repeats in Custom Library\" && runningTea\n firstMaskCustomLib\n else\n stage=\"Genome has already been masked, skipping...\" && runningTea\n fi\nfi\n\n# Step 3 - make a database from the genome, and then run a de novo TE annotation using RepeatModeler2\nif [ ! -f ${OUTDIR}/${species}_Database/${species}-families.fa ]; then\n stage=\"Detecting Novel Repeats\" && runningTea\n buildDB\n deNovo1\n sleep 1\nelse\n stage=\"De novo repeats have already been detected, skipping...\" && runningTea\n sleep 1\nfi\n\n# Step 4 - run TEstrainer, which runs a BLAST, extract, extent, trim, on the the de novo repeat library to refine consensus sequences\nif [ ! -s ${OUTDIR}/${species}_strainer/${species}-families.fa.strained ]; then\n stage=\"Straining TEs and Refining de novo Consensus Sequences\" && runningTea\n if [ -f ${OUTDIR}/${species}_strainer/${species}-families.fa.strained ]; then\n rm -r ${OUTDIR}/${species}_strainer/${species}-families.fa.strained ${OUTDIR}/${species}_strainer/TS*\n strainer\n elif [ -d ${OUTDIR}/${species}_strainer/TS_${species}-families.fa_* ]; then\n stage=\"Resuming TEstrainer\" && runningTea\n strainerResume\n else\n strainer\n fi\n if [ ! -s ${OUTDIR}/${species}_strainer/${species}-families.fa.strained ]; then\n echo \"WARNING: TEstrainer failed to produce a strain file, please check the log file for more information. If you have run an intial mask with known repeats, this could be due RepeatModeler2 failing to identify any new repeats. Please check if this is expected.\"\n fi\n sleep 1\nelse\n stage=\"TEs already strained, skipping...\" && runningTea\n latestFile=${OUTDIR}/${species}_strainer/${species}-families.fa.strained\n cp ${latestFile} ${OUTDIR}/${species}_summaryFiles/\n sleep 1\nfi\n\n# Stage 5 - Tidy up!\nstage=\"Tidying Directories and Organising Important Files\" && runningTea\nsweepUp\nsleep 1\n\n# Stage 6\ntime=$(convertsecs $SECONDS)\nstage=\"Done in $time\" && runningTea\nsleep 5\n\n# Stage 10\nstage=\"TE library in Standard Format Can Be Found in ${OUTDIR}/${species}_summaryFiles/\" && runningTea\nsleep 5\n"
},
{
"path": "legacyDocs.md",
"content": "# This file contains installation instructions for legacy versions of Earl Grey\n\n## We recommend installing via conda or docker for the latest stable releases of Earl Grey, the below are retained as a reference\n\n#==============================================================================================================================================================================#\n\n## Earl Grey Installation and Configuration (If you already have RepeatMasker and RepeatModeler)\n\nIf you do not currently have RepeatMasker and RepeatModeler installed, the instructions are provided further down this page. If you do have them installed, **ensure the executables are in your PATH environment, including the RepeatMasker/util/ directory!**\n\nAll of the scripts and associated modules are contained within this github repository. Earl Grey runs inside an anaconda environment to ensure all required packages are present and are the correct version. Therefore to run Earl Grey, you will require anaconda to be installed on your system. \n\n**If anaconda is NOT installed on your system, please install it following these instructions:**\n\n```\n# Change to /tmp directory as we won't need the script after running it\ncd /tmp\n\n# Download the anaconda installation script\ncurl https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh --output anaconda.sh\n\n# Run the script to install anaconda\nbash anaconda.sh\n# answer yes when asked, and install anaconda3 in the specified location (recommended) unless you want it to be installed elsewhere.\n# When asked \"Do you wish the installer to initialize Anaconda3 by running conda init?\", answer \"yes\" for ease of use.\n# Activate conda by refreshing terminal session\n\nsource ~/.bashrc\n\n# If successful, you should now see (base) on the left of your username on the command line\n```\n\n**Now that Anaconda is installed, we can install Earl Grey**\n\nClone the Earl Grey github repo\n\n```\n# Clone into a home directory, or somewhere you want to install Earl Grey\ngit clone https://github.com/TobyBaril/EarlGrey\n```\n\nEnter the Earl Grey directory and configure the program\n\n```\ncd ./EarlGrey\nchmod +x ./configure\n./configure\n```\n\nOnce this is complete, remember to activate the earlGrey conda environment before attempting to run the Earl Grey pipeline\n\n```\nconda activate earlGrey\n\nearlGrey -g genome.fasta -s speciesName -o outputDirectory -t threads\n```\n\nFor suggestions or questions, please use the discussion and issues functions in github.\n\nThank you for trying Earl Grey!\n\n#==============================================================================================================================================================================#\n\n## Earl Grey Installation and Configuration (If you DO NOT have RepeatMasker and RepeatModeler) - WITH SUDO PRIVILEGES\n\nThese instructions will guide you through configuring all required programs and scripts to run Earl Grey.\n\nAll of the scripts and associated modules are contained within this github repository. Earl Grey runs inside an anaconda environment to ensure all required packages are present and are the correct version. Therefore to run Earl Grey, you will require anaconda to be installed on your system. \n\n**If anaconda is NOT installed on your system, please install it following these instructions:**\n\n```\n# Change to /tmp directory as we won't need the script after running it\ncd /tmp\n\n# Download the anaconda installation script\ncurl https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh --output anaconda.sh\n\n# Run the script to install anaconda\nbash anaconda.sh\n# answer yes when asked, and install anaconda3 in the specified location (recommended) unless you want it to be installed elsewhere.\n# When asked \"Do you wish the installer to initialize Anaconda3 by running conda init?\", answer \"yes\" for ease of use.\n# Activate conda by refreshing terminal session\n\nsource ~/.bashrc\n\n# If successful, you should now see (base) on the left of your username on the command line\n```\n\n**Now that Anaconda is installed, we can install Earl Grey**\n\n### To install RepeatMasker\n\nRepeatMasker can be downloaded from: http://www.repeatmasker.org/RepeatMasker/. Installation instructions can be found on the website. Alternatively, please use the code below:\n\nYou will need to download and install a couple of programs for RepeatMasker to work. \n\nDownload rmblast\n\n```wget https://www.repeatmasker.org/rmblast/rmblast-2.13.0+-x64-linux.tar.gz```\n\nExtract the rmblast package\n\n```tar -zxvf rmblast-2.13.0+-x64-linux.tar.gz```\n\nMake a note of the full path to rmblast-2.13.0/bin/ as you will need this in the RepeatMasker configuration\nIf you are not certain of the full path, please run the following command\n\n```realpath ./rmblast-2.13.0/bin/```\n\nDownload RepeatMasker (this will download it to the current directory). NOTE some extra steps are required for Dfam 3.8, please refer to repeatmasker.org/RepeatMasker.\n\n```wget https://www.repeatmasker.org/RepeatMasker/RepeatMasker-4.1.6.tar.gz```\n\nCopy the RepeatMasker package to /usr/local/, or somewhere that all users will be able to access the installation. \nCopying to /usr/local/ might require sudo privileges\n\n```sudo cp RepeatMasker-4.1.6.tar.gz /usr/local/```\n\nChange directory to /usr/local, and extract the RepeatMasker package. This might require sudo privileges.\n\n```cd /usr/local/```\n\n```sudo tar -zxvf RepeatMasker-4.1.6.tar.gz```\n\nInstall the required RepeatMasker libraries - Earl Grey has been tested with Dfam 3.4 onwards and RepBase. \nUnfortunately, RepBase is now behind a paywall, but to ensure Earl Grey remains open it does not rely on RepBase, although inclusion of RepBase can improve classification of repeats by RepeatModeler. If you have access to this database, please include it in your configuration of RepeatMasker. \nWe recommend that you download Dfam 3.8 before using Earl Grey, and this is required for RepeatMasker 4.1.6. The Dfam library is large - this could take a while!\n\nWe recommend downloading Dfam into your home directory (~/) or a subdirectory of home\n\nChange directory to home\n\n```cd ~/```\n\nDownload lastest Dfam release - This may take a while\n*** NOTE: There are many partitions for Dfam, each containing curated and uncurated TE consensi split by taxonomy. As a minimum, you need to configure RepeatMasker with the root partition, but you can also include others. Note: There may be erroneous annotations in the uncurated Dfam database, use at your own risk***\n\nFor whole Dfam library run all of the below, or run each part as required (minimum the root (0) partition):\n```\nwget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.0.h5.gz\nwget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.1.h5.gz\nwget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.2.h5.gz\nwget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.3.h5.gz\nwget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.4.h5.gz\nwget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.5.h5.gz\nwget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.6.h5.gz\nwget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.7.h5.gz\nwget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.8.h5.gz\n```\n\nUnzip the Dfam release - This may take a while with no indication that anything is happening, please be patient!\n\nFor whole Dfam library (including uncurated elements):\n```gunzip *.h5.gz```\n\nMove the Dfam library to the RepeatMasker library folder\nNOTE, a warning might come up that this will overwrite the existing file, allow this by pressing \"y\" then Enter\n\n```mv *.h5 /usr/local/RepeatMasker/Libraries/famdb/```\n\nDO NOT configure RepeatMasker just yet...\n\nAdd RepeatMasker and util directory to your path environment\n\n```echo 'export PATH=$PATH:/usr/local/RepeatMasker:/usr/local/RepeatMasker/util/' >> ~/.bashrc```\n\n### To install RepeatModeler\n\nRepeatModeler can be downloaded from: http://www.repeatmasker.org/RepeatModeler/. Installation instructions can be found on the website. Alternatively, please use the code below:\n\nYou will need to download and install a couple of programs for RepeatModeler to work. \n\nDownload RECON and RepeatScout\n\n```\nwget http://www.repeatmasker.org/RepeatModeler/RECON-1.08.tar.gz\nwget http://www.repeatmasker.org/RepeatScout-1.0.6.tar.gz\n```\n\nExtract the RECON and RepeatScout packages\n\n```\ntar -zxvf RECON-1.08.tar.gz\ntar -zxvf RepeatScout-1.0.6.tar.gz\n```\n\nEnter the RECON directory and make from source\n\n```\ncd ./RECON-1.08/src/\nmake\nmake install\n```\n\nEnter the RepeatScout directory and make from source\n\n```\ncd ../../RepeatScout-1.0.6/\nmake\n```\n\nMake a note of the full path to RECON-1.08/bin/ and RepeatScout-1.0.6/ as you will need these in the RepeatModeler configuration\nIf you are not certain of the full paths, please run the following commands\n\n```\nrealpath ./RECON-1.08/bin/\nrealpath ./RepeatScout-1.0.6/\n```\n\nInstall UCSC TwoBit Tools. Make a note of the full path to the directory where these have been installed as you will need these in the RepeatModeler configuration\n\n```\nmkdir ~/ucscTwoBitTools/\ncd ~/ucscTwoBitTools/\nwget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/faToTwoBit\nwget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/twoBitToFa\nwget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/twoBitInfo\nwget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/twoBitMask\nwget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/twoBitDup\n\n# make scripts executable\nchmod +x ~/ucscTwoBitTools/*\n```\n\nFor Earl Grey, the LTR structural pipeline of RepeatModeler is not required, as this is run as part of Earl Grey's defragmentation process.\n\nDownload RepeatModeler\n\n```wget http://www.repeatmasker.org/RepeatModeler/RepeatModeler-2.0.4.tar.gz```\n\nCopy the RepeatModeler package to /usr/local/, or somewhere that all users will be able to access the installation. \nCopying to /usr/local/ might require sudo privileges\n\n```sudo cp RepeatModeler-2.0.4.tar.gz /usr/local/```\n\nChange directory to /usr/local, and extract the RepeatModeler package. This might require sudo privileges.\n\n```cd /usr/local/```\n\n```sudo tar -zxvf RepeatModeler-2.0.4.tar.gz```\n\nDO NOT configure RepeatModeler just yet...\n\nAdd RepeatModeler directory to your path environment\n\n```echo 'export PATH=$PATH:/usr/local/RepeatModeler-2.0.4/' >> ~/.bashrc```\n\n### To Install Earl Grey and Configure all Programs\n\nClone the Earl Grey github repo\n\n```\n# Clone into a home directory, or somewhere you want to install Earl Grey\ngit clone https://github.com/TobyBaril/EarlGrey\n```\n\nEnter the Earl Grey directory and configure the program\n\n```\ncd ./EarlGrey\nchmod +x ./configure\n./configure\n```\n\nActivate the conda environment\n\n```\nconda activate earlGrey\n```\n\nFor configuring RepeatMasker with sudo, h5py is required in the root path. The easiest way to do this is to run the following two lines:\n\n```\nsudo apt install python3-pip\nsudo pip install h5py\n```\n\nThe linux64 version of TRF is included in the modules directory of Earl Grey. Make a note of the full path to this file as this will be needed in the RepeatMasker configuration.\n\n```realpath ./modules/trf409*```\n\nNOTE: For the latest version of RepeatModeler, this must be called trf. We can make a symlinked version: ```ln -s ./modules/trf409* ./modules/trf```\n\nNow to finally configure RepeatMasker! - Run these lines and then follow the on-screen prompts from RepeatMasker. Again, this might require sudo privileges.\n\n```\ncd /usr/local/RepeatMasker\n# check which perl interpreter you should use \nwhich perl\n# replace [perl] in the below command with the path printed from the command above and use this as your perl interpreter (this should be the anaconda perl interpreter)\nsudo [perl] ./configure\n```\n\nTime to configure RepeatModeler! You need to enter the paths to lots of the directories of programs we have installed, please note them down before running the configuration script!! \n\n\n```\ncd /usr/local/RepeatModeler-2.0.4/\n# check which perl interpreter you should use - COPY THIS\nwhich perl\n# replace perl in the below command with the path printed from the command above and use this as your perl interpreter\nsudo perl ./configure\n```\n\nOnce this is complete, remember to activate the earlGrey conda environment before attempting to run the Earl Grey pipeline\n\n```\nconda activate earlGrey\n\nearlGrey -g genome.fasta -s speciesName -o outputDirectory -t threads\n```\n\n#==============================================================================================================================================================================#\n\n## Earl Grey Installation and Configuration (If you DO NOT have RepeatMasker and RepeatModeler) - WITHOUT SUDO PRIVILEGES\n\nThese instructions will guide you through configuring all required programs and scripts to run Earl Grey.\n\nAll of the scripts and associated modules are contained within this github repository. Earl Grey runs inside an anaconda environment to ensure all required packages are present and are the correct version. Therefore to run Earl Grey, you will require anaconda to be installed on your system. \n\n**If anaconda is NOT installed on your system, please install it following these instructions:**\n\n```\n# Change to /tmp directory as we won't need the script after running it\ncd /tmp\n\n# Download the anaconda installation script\ncurl https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh --output anaconda.sh\n\n# Run the script to install anaconda\nbash anaconda.sh\n# answer yes when asked, and install anaconda3 in the specified location (recommended) unless you want it to be installed elsewhere.\n# When asked \"Do you wish the installer to initialize Anaconda3 by running conda init?\", answer \"yes\" for ease of use.\n# Activate conda by refreshing terminal session\n\nsource ~/.bashrc\n\n# If successful, you should now see (base) on the left of your username on the command line\n```\n\n**Now that Anaconda is installed, we can install Earl Grey**\n\n### To install RepeatMasker\n\nRepeatMasker can be downloaded from: http://www.repeatmasker.org/RepeatMasker/. Installation instructions can be found on the website. Alternatively, please use the code below:\n\nYou will need to download and install a couple of programs for RepeatMasker to work. \n\nDownload rmblast\n\n```wget https://www.repeatmasker.org/rmblast/rmblast-2.13.0+-x64-linux.tar.gz```\n\nExtract the rmblast package\n\n```tar -zxvf rmblast-2.13.0+-x64-linux.tar.gz```\n\nMake a note of the full path to rmblast-2.13.0/bin/ as you will need this in the RepeatMasker configuration\nIf you are not certain of the full path, please run the following command\n\n```realpath ./rmblast-2.13.0/bin/```\n\nDownload RepeatMasker (this will download it to the current directory). NOTE some extra steps are required if you wish to use Dfam 3.7, please refer to repeatmasker.org/RepeatMasker for these extra steps if required.\n\n```wget https://www.repeatmasker.org/RepeatMasker/RepeatMasker-4.1.6.tar.gz```\n\nExtract the RepeatMasker package.\n\n```\ntar -zxvf RepeatMasker-4.1.6.tar.gz\n```\n\nInstall the required RepeatMasker libraries - Earl Grey has been tested with Dfam 3.3 and RepBase. \nUnfortunately, RepBase is now behind a paywall, but to ensure Earl Grey remains open it does not rely on RepBase, although inclusion of RepBase can improve classification of repeats by RepeatModeler. If you have access to this database, please include it in your configuration of RepeatMasker. \nWe recommend that you download Dfam 3.8 before using Earl Grey, and this is required for RepeatMasker 4.1.6. The Dfam library is large - this could take a while!\n\nChange directory to RepeatMasker Libraries\n\n```cd ./RepeatMasker/Libraries/famdb/```\n\nDownload lastest Dfam release - This may take a while\n*** NOTE: There are many partitions for Dfam, each containing curated and uncurated TE consensi split by taxonomy. As a minimum, you need to configure RepeatMasker with the root partition, but you can also include others. Note: There may be erroneous annotations in the uncurated Dfam database, use at your own risk***\n\nFor whole Dfam library run all of the below, or run each part as required (minimum the root (0) partition):\n```\nwget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.0.h5.gz\nwget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.1.h5.gz\nwget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.2.h5.gz\nwget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.3.h5.gz\nwget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.4.h5.gz\nwget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.5.h5.gz\nwget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.6.h5.gz\nwget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.7.h5.gz\nwget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.8.h5.gz\n```\n\nUnzip the Dfam release - This may take a while with no indication that anything is happening, please be patient!\n\nFor whole Dfam library (including uncurated elements):\n```gunzip *.h5.gz```\n\nDO NOT configure RepeatMasker just yet...\n\nAdd RepeatMasker and util directory to your path environment (Replace /path/to/ with the full path to your installation directory)\n\n```echo 'export PATH=$PATH:/path/to/RepeatMasker/:/path/to/RepeatMasker/util/' >> ~/.bashrc```\n\n### To install RepeatModeler\n\nRepeatModeler can be downloaded from: http://www.repeatmasker.org/RepeatModeler/. Installation instructions can be found on the website. Alternatively, please use the code below:\n\nYou will need to download and install a couple of programs for RepeatModeler to work. \n\nDownload RECON and RepeatScout (Make sure you are no longer inside your RepeatMasker directory!)\n\n```\nwget http://www.repeatmasker.org/RepeatModeler/RECON-1.08.tar.gz\nwget http://www.repeatmasker.org/RepeatScout-1.0.6.tar.gz\n```\n\nExtract the RECON and RepeatScout packages\n\n```\ntar -zxvf RECON-1.08.tar.gz\ntar -zxvf RepeatScout-1.0.6.tar.gz\n```\n\nEnter the RECON directory and make from source\n\n```\ncd ./RECON-1.08/src/\nmake\nmake install\n```\n\nEnter the RepeatScout directory and make from source\n\n```\ncd ../../RepeatScout-1.0.6/\nmake\n```\n\nMake a note of the full path to RECON-1.08/bin/ and RepeatScout-1.0.6/ as you will need these in the RepeatModeler configuration\nIf you are not certain of the full paths, please run the following commands\n\n```\nrealpath ./RECON-1.08/bin/\nrealpath ./RepeatScout-1.0.6/\n```\n\nInstall UCSC TwoBit Tools. Make a note of the full path to the directory where these have been installed as you will need these in the RepeatModeler configuration\n\n```\nmkdir ./ucscTwoBitTools/\ncd ./ucscTwoBitTools/\nwget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/faToTwoBit\nwget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/twoBitToFa\nwget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/twoBitInfo\nwget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/twoBitMask\nwget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/twoBitDup\n\n# make scripts executable\nchmod +x ./ucscTwoBitTools/*\n```\n\nFor Earl Grey, the LTR structural pipeline of RepeatModeler is not required, as this is run as part of Earl Grey's defragmentation process.\n\nDownload RepeatModeler\n\n```wget http://www.repeatmasker.org/RepeatModeler/RepeatModeler-2.0.4.tar.gz```\n\nUnpack RepeatModeler\n\n```sudo tar -zxvf RepeatModeler-2.0.4.tar.gz```\n\nDO NOT configure RepeatModeler just yet...\n\nAdd RepeatModeler directory to your path environment (Replace /path/to/ with the full path to your installation directory)\n\n\n```echo 'export PATH=$PATH:/path/to/RepeatModeler-2.0.4/' >> ~/.bashrc```\n\n### To Install Earl Grey and Configure all Programs\n\nClone the Earl Grey github repo\n\n```\n# Clone into a home directory, or somewhere you want to install Earl Grey\ngit clone https://github.com/TobyBaril/EarlGrey\n```\n\nEnter the Earl Grey directory and configure the program\n\n```\ncd ./EarlGrey\nchmod +x ./configure\n./configure\n```\n\nActivate the conda environment\n\n```\nconda activate earlGrey\n```\n\nThe linux64 version of TRF is included in the modules directory of Earl Grey. Make a note of the full path to this file as this will be needed in the RepeatMasker configuration.\n\n```realpath ./modules/trf409*```\n\nNOTE: For the latest version of RepeatModeler, this must be called trf. We can make a symlinked version: ```ln -s ./modules/trf409* ./modules/trf```\n\nNow to finally configure RepeatMasker! - Run these lines and then follow the on-screen prompts from RepeatMasker. Again, this might require sudo privileges.\n\n```\ncd /path/to/RepeatMasker\nperl ./configure\n```\n\nTime to configure RepeatModeler! You need to enter the paths to lots of the directories of programs we have installed, please note them down before running the configuration script!! \n\n\n```\ncd /path/to/RepeatModeler-2.0.4/\nperl ./configure\n```\n\nOnce this is complete, remember to activate the earlGrey conda environment before attempting to run the Earl Grey pipeline\n\n```\nconda activate earlGrey\n\nearlGrey -g genome.fasta -s speciesName -o outputDirectory -t threads\n```\n\n# Alternative installation methods\n\nBefore using Earl Grey, please ensure RepeatMasker (>=version 4.1.4) and RepeatModeler (>=version 2.0.4) are installed and configured. If these are not, please follow the instructions below to install these before continuing with Earl Grey Installation. \nNOTE: These instructions are provided to install RepeatMasker, RepeatModeler and related programs with sudo privileges. If you are working on a shared cluster, please request installation of RepeatMasker and RepeatModeler by your sysadmin before working with Earl Grey. Earl Grey will function with RepeatMasker and RepeatModeler installed in the local path environment, or when the modules are loaded on a HPC cluster.\n\n\n#==============================================================================================================================================================================#\n\n## Earl Grey is available in a [Docker container](./Docker) preconfigured with Dfam version 3.9 or empty, but with the option to install required partitions\n\nTo use this container, please make sure Docker is installed and configured on your system. All instructions relating to the docker installation are found within the [Docker directory](./Docker) in this repository. Please consult the README in the Docker directory for installation instructions.\n\n#==============================================================================================================================================================================#\n\n## Earl Grey is available for [Singularity](./Singularity) preconfigured with Dfam version 3.9 or empty, but with the option to install required partitions\n\nTo use this container, please make sure Singularity is installed and configured on your system. All instructions relating to the singularity installation are found within the [Singularity directory](./Singularity) in this repository. Please consult the README in the Singularity directory for installation instructions.\n"
},
{
"path": "scripts/LTR_FINDER_parallel",
"content": "#!/usr/bin/perl -w\nuse strict;\nuse threads;\nuse Thread::Queue;\nuse File::Basename;\nuse FindBin;\n\n# customized parameters\nmy $seq_path = \"\"; #specify the sequence file\nmy $finder_para = \"-w 2 -C -D 15000 -d 1000 -L 7000 -l 100 -p 20 -M 0.85\"; #specify ltr_finder parameters. The \"-w 2\" parameter is required.\nmy $harvest_format = 0; #0, print out format is in LTR_finder table output format (default); 1, LTRharvest screen output format\nmy $size_of_each_piece = 5000000; #5 Mb per piece\nmy $timeout = 1500; #set maximum time for a thread to run. After $timeout seconds, the child thread is killed.\nmy $try1 = 1; #1, further split to 50 Kb regions if thread killed by timeout. 0, no further split.\nmy $next = 0; #0, run LTR_FINDER. 1, only summarize results if the .finder folder is retained.\nmy $verbose = 0; #0, remove the .finder folder after finished; 1, retain the folder for later use.\nmy $threads = 4; #specify thread number to run ltr_finder\nmy $annotator = \"LTR_FINDER_parallel\"; #the annotator name used in gff output\n\n# dependencies\n#my $script_path = dirname(__FILE__);\nmy $script_path = $FindBin::Bin;\nmy $cut = \"$script_path/bin/cut.pl\"; #the program to cut sequence\nmy $ltr_finder = \"$script_path/bin/LTR_FINDER.x86_64-1.0.7/ltr_finder\"; #the path to ltr_finder\n\nmy $usage = \"\n~ ~ ~ Run LTR_FINDER in parallel ~ ~ ~\n\nAuthor: Shujun Ou (shujun.ou.1\\@gmail.com)\nDate: 09/19/2018\nUpdate: 09/27/2019\n\nUsage: perl LTR_FINDER_parallel -seq [file] -size [int] -threads [int]\nOptions:\t-seq\t[file]\tSpecify the sequence file.\n\t\t-size\t[int]\tSpecify the size you want to split the genome sequence.\n\t\t\t\tPlease make it large enough to avoid spliting too many LTR elements. Default 5000000 (bp)\n\t\t-time\t[int]\tSpecify the maximum time to run a subregion (a thread).\n\t\t\t\tThis helps to skip simple repeat regions that take a substantial of time to run. Default: 1500 (seconds).\n\t\t\t\tSuggestion: 300 for -size 1000000. Increase -time when -size increased.\n\t\t-try1\t[0|1]\tIf a region requires more time than the specified -time (timeout), decide:\n\t\t\t\t\t0, discard the entire region.\n\t\t\t\t\t1, further split to 50 Kb regions to salvage LTR candidates (default);\n\t\t-harvest_out\tOutput LTRharvest format if specified. Default: output LTR_FINDER table format.\n\t\t-next\t\tOnly summarize the results for previous jobs without rerunning LTR_FINDER (for -v).\n\t\t-verbose|-v\tRetain LTR_FINDER outputs for each sequence piece.\n\t\t-finder\t[file]\tThe path to the program LTR_FINDER (default v1.0.7, included in this package).\n\t\t-threads|-t\t[int]\tIndicate how many CPU/threads you want to run LTR_FINDER.\n\t\t-help|-h\tDisplay this help information.\n\\n\";\n\n\n# read user parameters\nmy $i=0;\nforeach (@ARGV){\n\t$seq_path = $ARGV[$i+1] if /^-seq$/i;\n\t$size_of_each_piece = $ARGV[$i+1] if /^-size$/i;\n\t$timeout = $ARGV[$i+1] if /^-time$/i;\n\t$try1 = $ARGV[$i+1] if /^-try1$/i;\n\t$harvest_format = 1 if /^-harvest_out$/i;\n\t$next = 1 if /^-next$/i;\n\t$verbose = 1 if /^-verbose$|^-v$/i;\n\t$cut = $ARGV[$i+1] if /^-cut$/i;\n\t$ltr_finder = $ARGV[$i+1] if /^-finder$/i;\n\t$threads = $ARGV[$i+1] if /^-threads$|^-t$/i;\n\tdie $usage if /^-help$|^-h$/i;\n\t$i++;\n\t}\n\n# make a softlink to the genome file\ndie $usage unless -s $seq_path;\nmy $seq_file = basename($seq_path);\n`ln -s $seq_path $seq_file` unless -s $seq_file;\n\n# read genome in memory\nopen SEQ, \"<$seq_file\" or die $usage;\nopen GFF, \">$seq_file.finder.combine.gff3\" or die $!;\nprint GFF \"##gff-version 3\\n\";\nmy %seq; #a hash to store seq name and length info\nmy %order; #store seq order in genome\nmy $chr_info; #store seq id in genome\n$i=0;\n$/ = \"\\n>\";\nwhile (){\n\ts/>//g;\n\tmy ($id, $seq) = (split /\\n/, $_, 2);\n\t$seq =~ s/\\s+//g;\n\tmy $len = length $seq;\n\tprint GFF \"##sequence-region $id 1 $len\\n\";\n\t$chr_info.=\"#$id\\n\";\n\t$seq{$id} = $len;\n\t$order{$id} = $i;\n\t$i++;\n\t}\nprint GFF \"$chr_info\";\nclose SEQ;\n$/=\"\\n\";\n\ngoto Next if $next == 1; #run the next step if all LTR_finder processes are done\n\n`mkdir $seq_file.finder` unless -d \"$seq_file.finder\";\nchdir \"$seq_file.finder\";\n`perl $cut ../$seq_file -s -l $size_of_each_piece`;\n\n##multi-threading using queue, create a worker module for parallel computation\nmy $process_q=Thread::Queue->new();\nsub worker {\n\twhile (my $seq = $process_q -> dequeue()){\n\t\tchomp $seq;\n\t\t$seq =~ s/\\s+//;\n\t\tprint localtime() .\" CPU\".threads -> self() -> tid().\": running on $seq\\n\";\n\t\t`timeout $timeout $ltr_finder $finder_para $seq > $seq.finder.scn`; #set the max running time for a thread as $timeout seconds\n\t\tif ($? ne 0 and $try1 ne 0){\n\t\t\tprint localtime() .\" CPU\".threads -> self() -> tid().\": $seq timeout, process it with the salvage mode\\n\";\n\t\t\tmy $in=`perl $script_path/LTR_FINDER_parallel -seq $seq -size 50000 -time 10 -try1 0 -threads 1 -cut $cut -finder $ltr_finder`;\n\t\t\t`mv $seq.finder.combine.scn $seq.finder.scn`;\n\t\t\t`rm -rf ${seq}.finder`; #remove?\n\t\t\t}\n\t\t}\n\t}\n\n#insert seq names into the worker queue\nopen List, \"<../$seq_file.list\" or die $!;\n$process_q -> enqueue ();\n$process_q -> end(); #stop adding items to the queue\nclose List;\n\n#work and finish\nfor (1..$threads){\n\tthreads -> create ( \\&worker );\n\t}\nforeach my $thr (threads -> list()){\n\t$thr -> join();\n\t}\n\nchdir \"../\";\n\nNext:\n#combine split ltr_finder results\nopen Out, \">$seq_file.finder.combine.scn\" or die $!;\n\n#print out headers\nif ($harvest_format == 0){\n\tprint Out \"index\tSeqID\tLocation\tLTR len\tInserted element len\tTSR\tPBS\tPPT\tRT\tIN (core)\tIN (c-term)\tRH\tStrand\tScore\tSharpness\tSimilarity\\n\";\n\t} else {\n\tprint Out \"#LTR_FINDER_parallel -seq $seq_file -size $size_of_each_piece -time $timeout -try1 $try1 -harvest_out -threads $threads -cut $cut -finder $ltr_finder\n# LTR_FINDER args=$finder_para\n# predictions are reported in the following way\n# s(ret) e(ret) l(ret) s(lLTR) e(lLTR) l(lLTR) s(rLTR) e(rLTR) l(rLTR) sim(LTRs) seq-nr chr\n# where:\n# s = starting position\n# e = ending position\n# l = length\n# ret = LTR-retrotransposon\n# lLTR = left LTR\n# rLTR = right LTR\n# sim = similarity\n# seq-nr = sequence order\\n\";\n\t}\n\n\nopen List, \"<$seq_file.list\" or die $!;\nmy $count = 1; #count repeats in gff\nforeach my $seq (){\n\t$seq =~ s/\\s+//;\n\tmy ($base, $order) = ($1, $2) if $seq =~ /(.*)_sub([0-9]+)$/;\n\tmy $coord_adj = ($order - 1) * $size_of_each_piece;\n\tnext unless -e \"$seq_file.finder/$seq.finder.scn\";\n\topen Scn, \"<$seq_file.finder/$seq.finder.scn\" or die $!;\n\twhile (){\n\t\tnext unless /^\\[/;\n\t\ts/^\\[\\s+[0-9]+\\]/\\[NA\\]/;\n\t\tmy ($index, $id, $loc, $ltr_len, $ele_len, $TSR, $PBS, $PPT, $RT, $IN_core, $IN_cterm, $RH, $strand, $score, $sharpness, $sim) = (split);\n\n\t\t#convert coordinates back to the genome scale\n\t\tmy @coord = ($loc, $PBS, $PPT, $RT, $IN_core, $IN_cterm, $RH);\n\t\tmy $i = -1;\n\t\tforeach (@coord) {\n\t\t\t$i++;\n\t\t\tnext if /^N-N$/;\n\t\t\tmy ($start, $end) = ($1+$coord_adj, $2+$coord_adj) if /([0-9]+)\\-([0-9]+)/;\n\t\t\t$coord[$i] = \"$start-$end\";\n\t\t\t}\n\t\tmy ($start, $end) = ($1, $2) if $coord[0] =~ /([0-9]+)\\-([0-9]+)/;\n\t\tmy ($lltr, $rltr) = ($1, $2) if $ltr_len=~/([0-9]+),([0-9]+)/; \n\t\tmy ($lltr_e, $rltr_s) = ($start+$lltr-1, $end-$rltr+1);\n\n\t\t#output LTRharvest or LTR_FINDER (-w 2) format\n\t\tif ($harvest_format == 0){\n\t\t\tprint Out \"[NA]\\t$base\\t$coord[0]\\t$ltr_len\\t$ele_len\\t$TSR\\t$coord[1]\\t$coord[2]\\t$coord[3]\\t$coord[4]\\t$coord[5]\\t$coord[6]\\t$strand\\t$score\\t$sharpness\\t$sim\\n\";\n\t\t\t} else {\n\t\t\t$sim*=100;\n\t\t\tprint Out \"$start $end $ele_len $start $lltr_e $lltr $rltr_s $end $rltr $sim $order{$base} $base\\n\";\n\t\t\t}\n\n\t\t#print GFF format\n\t\tmy $chr = $base;\n\t\tprint GFF \"$chr\\t$annotator\\trepeat_region\\t$start\\t$end\\t.\\t$strand\\t.\\tID=repeat_region$count\\n\";\n\t\t#print GFF \"$chr\\t$annotator\\ttarget_site_duplication\\t$lTSD\\t.\\t$strand\\t.\\tParent=repeat_region$count\\n\" unless $TSD eq \"NA\";\n\t\tprint GFF \"$chr\\t$annotator\\tLTR_retrotransposon\\t$start\\t$end\\t.\\t$strand\\t.\\tID=LTR_retrotransposon$count;Parent=repeat_region$count;tsd=$TSR;ltr_identity=$sim;seq_number=$order{$chr}\\n\";\n\t\tprint GFF \"$chr\\t$annotator\\tlong_terminal_repeat\\t$start\\t$lltr_e\\t.\\t$strand\\t.\\tParent=LTR_retrotransposon$count\\n\";\n\t\tprint GFF \"$chr\\t$annotator\\tlong_terminal_repeat\\t$rltr_s\\t$end\\t.\\t$strand\\t.\\tParent=LTR_retrotransposon$count\\n\";\n\t\t#print GFF \"$chr\\t$annotator\\ttarget_site_duplication\\t$rTSD\\t.\\t$strand\\t.\\tParent=repeat_region$count\\n\" unless $TSD eq \"NA\";\n\t\tprint GFF \"###\\n\";\n\n\t\t}\n\t$count++;\n\t}\nclose List;\nclose Scn;\nclose GFF;\n`rm -rf ./$seq_file.finder` if $verbose eq 0;\nprint localtime() .\" Job finished! Check out $seq_file.finder.combine.scn\\n\";\n"
},
{
"path": "scripts/TEstrainer/README.md",
"content": "# TEstrainer\nA pipeline to dramatically improve repeat libraries before genome annotation through curation of each repeat, better identification of satellite repeats, removal of multi-copy genes and reclassification of cleaned libraries.\n\nTEstrainer can be used on both de novo libraries (generated by tools such as RepeatModeler, REPET, EDTA, etc) or on libraries from databases (e.g. Repbase, Dfam).\n\nThe iterative curation of repeats is for use on de novo libraries, and will lengthen repeat consensus sequences as close as possible to the full length. This lengthening is accomplished by repeating the following procedure (developed from ):\n\nExample command\n```\n./TEstrainer -l seq/Camellia sinensis-families.fa -g seq/ASM1731120v1.fasta -t 64 -r 12 -C\n```\n\n1) Using BLAST to identify copies of the repeats in the source genome\n2) Extending their coordinates, extracting the extended coordinates\n3) Local aligning the each extracted sequence to each other to identify regions of the flanks which do not align\n4) Trimming off these flanks\n5) Creating a multiple sequence alignment (MSA) of the trimmed sequences\n6) Creating a consensus sequence from the MSA, ensuring coverage of at least three nucleotides along the length of the sequences\n\nIf a repeat's new consensus is more than 25% longer than its previously consensus sequence and aligns to the starting consensus sequence it this cycle is repeated.\n\n\n## Required packages and databases\n\nBLAST+ - Altschul, S.F. et al. (1997). https://doi.org/10.1089/10665270050081478.\n\nMAFFT - Katoh, K and Standley, D.M. (2013). https://doi.org/10.1093/molbev/mst010\n\nmreps - Kolpakov, R et al. (2003). https://doi.org/10.1093/nar/gkg617\n\nsa-ssr - Pickett, B.D. (2016). https://doi.org/10.1093/bioinformatics/btw298\n\nGNU Parallel - Tange, O. (2011) The USENIX Magazine, February 2011:42-47.\n\ncd-hit (Optional, used for clustering) - Li, W (2006) https://doi.org/10.1093/bioinformatics/btl158\n\nRepeatModeler (Optional, used for reclassifying) - Flynn, J.M. (2020). https://doi.org/10.1073/pnas.1921046117\n\n### Python3 packages\n\npyranges - Stovner, E.B and Sætrom, P. (2020). https://doi.org/10.1093/bioinformatics/btz615\n\npyfaidx - Shirley MD, et al. (2015). https://doi.org/10.7287/peerj.preprints.970v1\n\nbiopython - Cock, P.A. et al. (2009). https://doi.org/10.1093/bioinformatics/btp163\n\npandas - McKinney, W. (2010). https://doi.org/10.25080/Majora-92bf1922-00a \n\nnumpy - Harris, C.R. et al. (2020). https://doi.org/10.1038/s41586-020-2649-2. \n\n\n### R packages\n\noptparse\n\ntidyverse - Wickham, H. et al. (2019). https://doi.org/doi:10.21105/joss.01686. \n\nBSgenome - Pagès H (2022). https://bioconductor.org/packages/BSgenome.\n\nplyranges - Lee, S. et al. (2019). https://doi.org/10.1186/s13059-018-1597-8.\n\n\n### Database\nNCBI CDD - Lu, S. et al (2020). https://doi.org/10.1093/nar/gkz991 \n"
},
{
"path": "scripts/TEstrainer/TEstrainer",
"content": "#!/bin/bash\n\nusage() { echo \"Usage: [-l Repeat library] [-g Genome ] [-t Threads (default 4) ] [-f Flank (default 1000) ] [-r Iterations of BEET to run (default 0) ] [-d Out directory, if not specified wil be created ] [-c Set if clustering ] [-C Set if reclassifying ] [-s Set if sorting (only for RepeatModeler output) ] [-D Set if printing all messages ] [-h Print this help] [-M Ammount of memory TEstrainer needs to keep free\" 1>&2; exit 1; }\n\n# default values\nFLANK=1000\nTHREADS=4\nRUNS=0\nCLASSIFY=FALSE\nSORT=FALSE\nDEBUG=FALSE\n# DFAM=FALSE\n# for potential folder name\nTIME=$(date +\"%s\")\nTIME=${TIME: -4}\nMEM_FREE=\"1G\"\n\n# parsing\nwhile getopts l:g:t:f:r:d:M:CsDh flag; do\n case \"${flag}\" in\n l) RM_LIBRARY_PATH=${OPTARG};;\n g) GENOME=${OPTARG};;\n t) THREADS=${OPTARG};;\n f) FLANK=${OPTARG};;\n r) RUNS=${OPTARG};;\n d) DATA_DIR=${OPTARG};;\n M) MEM_FREE=${OPTARG};;\n C) CLASSIFY=TRUE;;\n s) SORT=TRUE ;;\n D) DEBUG=TRUE ;;\n h | *)\n print_usage\n exit_script\n esac\ndone\necho \"Running using\"$GENOME\" and \"$RM_LIBRARY_PATH\n# determine further variables and check files exist\nif [ ! -f ${RM_LIBRARY_PATH} ]; then echo \"Library not found\"; usage; fi\nif [ -z ${RM_LIBRARY_PATH} ]; then echo \"Library must be supplied\"; usage; else RM_LIBRARY=$(echo $RM_LIBRARY_PATH | sed 's/.*\\///'); fi\nif [[ $RUNS -gt 0 ]]; then \n if [ -z ${GENOME} ]; then echo \"If refining genome must be supplied\"; usage; fi\n if [ ! -f ${GENOME} ]; then echo \"Refining genome not found\"; usage; fi\nfi\nif [ -z \"$DATA_DIR\" ]; then DATA_DIR=$(echo \"TS_\"${RM_LIBRARY}\"_\"${TIME}); fi\nif [ \"$DEBUG\" == TRUE ]; then echo \"Files all exist\" ;fi\n# create data dir if missing\nif [ ! -d ${DATA_DIR} ] \nthen\n mkdir ${DATA_DIR}\nfi\n\nif [[ $THREADS -gt 4 ]]; then MAFFT_THREADS=$(($(($THREADS / 4)))); else MAFFT_THREADS=1; fi\n\n# Cap threads based on available RAM to prevent OOM errors\nAVAIL_MEM_MB=$(free -m | awk '/^Mem:/{print $7}')\nMEM_PER_JOB_MB=800\nMAX_SAFE_THREADS=$(( AVAIL_MEM_MB / MEM_PER_JOB_MB ))\nif [[ $MAX_SAFE_THREADS -gt 0 && $THREADS -gt $MAX_SAFE_THREADS ]]; then\n echo \"Warning: capping threads from ${THREADS} to ${MAX_SAFE_THREADS} based on available RAM (${AVAIL_MEM_MB} MB free)\"\n THREADS=$MAX_SAFE_THREADS\n if [[ $THREADS -gt 4 ]]; then MAFFT_THREADS=$(($(($THREADS / 4)))); else MAFFT_THREADS=1; fi\nfi\n\n# make directories\nmkdir -p ${DATA_DIR}/run_0/\n\n# # dfam extract\n# if [ \"$DFAM\" == TRUE ]; then python scripts/Dfam_extractor.py -l ${RM_LIBRARY_PATH} -d ${DATA_DIR} ; else cp ${RM_LIBRARY_PATH} ${DATA_DIR}/${RM_LIBRARY};fi\n\n# initial copy\nif [ \"$DEBUG\" == TRUE ]; then echo \"Copying library to data dir\" ;fi\ncp ${RM_LIBRARY_PATH} ${DATA_DIR}/run_0/further_${RM_LIBRARY}\n\n# setup for BEET runs\nif [[ $RUNS -gt 0 ]]; then\n \n # index genome and make blast db\n if [ \"$DEBUG\" == TRUE ]; then echo \"Indexing genome\" ;fi\n python scripts/indexer.py -g ${GENOME}\n if [ ! -f \"${GENOME}\".nsq ]; then\n makeblastdb -in ${GENOME} -dbtype nucl -out ${GENOME} # makeblastb if needed\n fi\n \n # create og reference\n python scripts/splitter.py -i ${DATA_DIR}/run_0/further_${RM_LIBRARY} -o ${DATA_DIR}/run_0/og\n\n RUN_NO=1\n # BEET runs\n while [ $RUN_NO -le $RUNS ]\n do\n if [ \"$DEBUG\" == TRUE ]; then echo \"Making \"${RUN_NO}\" directories\"; fi\n # make directories\n mkdir -p ${DATA_DIR}/run_${RUN_NO}/raw \\\n ${DATA_DIR}/run_${RUN_NO}/TEtrim_complete \\\n ${DATA_DIR}/run_${RUN_NO}/initial_blast \\\n ${DATA_DIR}/run_${RUN_NO}/self_search \\\n ${DATA_DIR}/run_${RUN_NO}/to_align \\\n ${DATA_DIR}/run_${RUN_NO}/mafft \\\n ${DATA_DIR}/run_${RUN_NO}/TEtrim_con \\\n ${DATA_DIR}/run_${RUN_NO}/TEtrim_unaln \\\n ${DATA_DIR}/run_${RUN_NO}/TEtrim_blast \\\n ${DATA_DIR}/run_${RUN_NO}/TEtrim_mafft \\\n ${DATA_DIR}/run_${RUN_NO}/TEtrim_further \\\n ${DATA_DIR}/run_${RUN_NO}/TEtrim_bp\n \n # split\n cp ${DATA_DIR}/run_$(expr $RUN_NO - 1)/further_${RM_LIBRARY} ${DATA_DIR}/run_${RUN_NO}/${RM_LIBRARY}\n echo \"Splitting run \"${RUN_NO}\n python scripts/splitter.py -i ${DATA_DIR}/run_${RUN_NO}/${RM_LIBRARY} -o ${DATA_DIR}/run_${RUN_NO}/raw\n\n # run trf to determine if sequence is tandem repeat\n echo \"Initial trf check for \"${RUN_NO}\n parallel --bar --jobs ${THREADS} --memfree ${MEM_FREE} -a ${DATA_DIR}/run_${RUN_NO}/raw/${RM_LIBRARY}_split.txt trf ${DATA_DIR}/run_${RUN_NO}/raw/{} 2 7 7 80 10 50 500 -d -h -ngs \">\" ${DATA_DIR}/run_${RUN_NO}/raw/{}.trf\n echo \"Initial blast and preparation for MSA \"${RUN_NO}\n # initial blast and extention\n parallel --bar --jobs ${THREADS} --memfree ${MEM_FREE} -a ${DATA_DIR}/run_${RUN_NO}/raw/${RM_LIBRARY}_split.txt python3 scripts/initial_mafft_setup.py -d ${DATA_DIR} -r ${RUN_NO} -s {} -g ${GENOME} -f ${FLANK} -D\n \n ## first mafft alignment\n find ${DATA_DIR}/run_${RUN_NO}/to_align -type f | sed 's/.*\\///' > ${DATA_DIR}/run_${RUN_NO}/to_align.txt\n echo \"Primary alignment run \"${RUN_NO}\n parallel --bar --jobs $MAFFT_THREADS --memfree ${MEM_FREE} -a ${DATA_DIR}/run_${RUN_NO}/to_align.txt mafft --thread 4 --quiet --localpair --adjustdirectionaccurately ${DATA_DIR}/run_${RUN_NO}/to_align/{} \">\" ${DATA_DIR}/run_${RUN_NO}/mafft/{}\n\n # trim\n echo \"Trimming run \"${RUN_NO}\n parallel --bar --jobs $MAFFT_THREADS --memfree ${MEM_FREE} -a ${DATA_DIR}/run_${RUN_NO}/to_align.txt python scripts/TEtrim.py -i ${DATA_DIR}/run_${RUN_NO}/mafft/{} -t 4 -f ${FLANK} -n ${RUN_NO} -d ${DATA_DIR}\n \n # compile completed curations\n if [ -n \"$(ls -A ${DATA_DIR}/run_${RUN_NO}/TEtrim_complete/ 2>/dev/null)\" ]; then\n cat ${DATA_DIR}/run_${RUN_NO}/TEtrim_complete/*fasta > ${DATA_DIR}/run_${RUN_NO}/complete_${RM_LIBRARY}\n fi\n \n # Either compile consensuses for further curation\n if [ -n \"$(ls -A ${DATA_DIR}/run_${RUN_NO}/TEtrim_further/ 2>/dev/null)\" ]; then\n cat ${DATA_DIR}/run_${RUN_NO}/TEtrim_further/*fasta > ${DATA_DIR}/run_${RUN_NO}/further_${RM_LIBRARY}\n echo \"Ready for \" $(( RUN_NO++ ))\n else\n # or exit loop if no more curation needed\n echo \"Finished extension\"\n break\n fi\n\n done\n\n # Compile all completed\n cat ${DATA_DIR}/run_*/complete_${RM_LIBRARY} > ${DATA_DIR}/${RM_LIBRARY}\n # if more could have been extended append to compilation\n if [ -s ${DATA_DIR}/run_${RUNS}/further_${RM_LIBRARY} ]; then\n cat ${DATA_DIR}/run_${RUNS}/further_${RM_LIBRARY} >> ${DATA_DIR}/${RM_LIBRARY}\n fi\n \n # add any families which went missing along the way due to alignment issues\n find ${DATA_DIR}/run_*/mafft/ -empty | sed 's/mafft/raw/' > ${DATA_DIR}/missing_consensi.txt\n if [[ -s ${DATA_DIR}/missing_consensi.txt ]]; then\n while read file_name; do\n cat $file_name >> ${DATA_DIR}/${RM_LIBRARY}\n done < ${DATA_DIR}/missing_consensi.txt\n fi\n \nelse\n\necho \"No curation\"\ncp ${RM_LIBRARY_PATH} ${DATA_DIR}/${RM_LIBRARY}\n\nfi\n\n# Identify simple repeats and satellites, trim ends of LINEs/SINEs\necho \"Splitting for simple/satellite packages\"\nmkdir -p ${DATA_DIR}/trf/split\npython scripts/splitter.py -i ${DATA_DIR}/${RM_LIBRARY} -o ${DATA_DIR}/trf/split\ncp ${DATA_DIR}/${RM_LIBRARY} ${DATA_DIR}/trf/\n# Run and parse TRF\necho \"Running TRF\"\nparallel --bar --jobs ${THREADS} --memfree ${MEM_FREE} -a ${DATA_DIR}/trf/split/${RM_LIBRARY}_split.txt trf ${DATA_DIR}/trf/split/{} 2 7 7 80 10 50 500 -d -h -ngs \">\" ${DATA_DIR}/trf/split/{}.trf\nparallel --bar --jobs ${THREADS} --memfree ${MEM_FREE} -a ${DATA_DIR}/trf/split/${RM_LIBRARY}_split.txt python3 scripts/trf_parser.py --trf ${DATA_DIR}/trf/split/{}.trf --out ${DATA_DIR}/trf/split/{}.trf.tsv\nfind ./${DATA_DIR}/trf/split/ -type f -name \"*trf.tsv\" -exec cat {} + | cat > ${DATA_DIR}/trf/${RM_LIBRARY}.trf\n# Run SA-SSR\necho \"Running SA-SSR\"\nsa-ssr -e -l 20 -L 50000 -m 1 -M 5000 -t ${THREADS} ${DATA_DIR}/${RM_LIBRARY} ${DATA_DIR}/trf/${RM_LIBRARY}.sassr\n# Run and compile mreps\necho \"Running mreps\"\nparallel --bar --jobs ${THREADS} --memfree ${MEM_FREE} -a ${DATA_DIR}/trf/split/${RM_LIBRARY}_split.txt bash scripts/mreps_parser.sh -i ${DATA_DIR}/trf/split/{} \"2>\"/dev/null\nfind ./${DATA_DIR}/trf/split/ -type f -name \"*mreps\" -exec cat {} + | cat > ${DATA_DIR}/trf/${RM_LIBRARY}.mreps\n# Interpret mreps, TRF and SA-SSR\necho \"Trimming and sorting based on mreps, TRF, SA-SSR\"\nRscript scripts/simple_repeat_filter_trim.R -i ${DATA_DIR}/${RM_LIBRARY} -d ${DATA_DIR}\n\n# Identify and trim chimeric repeats from non-satellite repeats, remove proteins\nmkdir -p ${DATA_DIR}/chimeras/split/\ncp ${DATA_DIR}/trf/${RM_LIBRARY}.nonsatellite ${DATA_DIR}/chimeras/${RM_LIBRARY}\necho \"Split ready for rpstblastn\"\npython scripts/splitter.py -i ${DATA_DIR}/chimeras/${RM_LIBRARY} -o ${DATA_DIR}/chimeras/split/\necho \"Run rpstblastn\"\nparallel --bar --jobs $THREADS --memfree ${MEM_FREE} -a ${DATA_DIR}/chimeras/split/${RM_LIBRARY}_split.txt rpstblastn -query ${DATA_DIR}/chimeras/split/{} -db ~/Databases/cdd/Cdd -out ${DATA_DIR}/chimeras/split/{}.out -outfmt \\\"6 qseqid qstart qend qlen sseqid sstart send slen pident length mismatch gapopen evalue bitscore qcovs stitle\\\" -evalue 0.01 -num_threads 1\nfind ./${DATA_DIR}/chimeras/split/ -type f -name \"*.out\" -exec cat {} + | cat > ${DATA_DIR}/chimeras/${RM_LIBRARY}.rps.out\necho \"Running strainer.R\"\nRscript scripts/strainer.R --in_seq ${DATA_DIR}/chimeras/${RM_LIBRARY} --directory ${DATA_DIR}\ncat ${DATA_DIR}/chimeras/clean_${RM_LIBRARY} ${DATA_DIR}/chimeras/chimeric_${RM_LIBRARY} > ${DATA_DIR}/chimeras/rinsed_${RM_LIBRARY}.nonsatellite\n\n# Delete temp files\nif [ \"$DEBUG\" == FALSE ]; then\n echo \"Removing temporary files\"\n rm -r ${DATA_DIR}/*/split/\n find ${DATA_DIR}/ -mindepth 1 -name \"run_*\" -exec rm -r {} +\nfi\nif [[ $RUNS -gt 0 ]]; then\n rm ${GENOME}.n*\nfi\n\n\n# Classify ?\nif [ \"$CLASSIFY\" == TRUE ]; then\n # Classify improved consensi using RepeatModeler's RepeatClassifier\n echo \"Reclassifying repeats\"\n mkdir -p ${DATA_DIR}/classify/\n cp ${DATA_DIR}/chimeras/rinsed_${RM_LIBRARY}.nonsatellite ${DATA_DIR}/classify/\n cd ${DATA_DIR}/classify/\n RepeatClassifier -pa ${THREADS} -consensi rinsed_${RM_LIBRARY}.nonsatellite\n cd -\n cp ${DATA_DIR}/classify/rinsed_${RM_LIBRARY}.nonsatellite.classified ${DATA_DIR}/${RM_LIBRARY}\n echo \"Compiling library\"\n cat ${DATA_DIR}/trf/${RM_LIBRARY}.satellites >> ${DATA_DIR}/${RM_LIBRARY}\nelse\n echo \"Compiling library\"\n cat ${DATA_DIR}/chimeras/rinsed_${RM_LIBRARY}.nonsatellite ${DATA_DIR}/trf/${RM_LIBRARY}.satellites >> ${DATA_DIR}/${RM_LIBRARY}\nfi\n\n# Sort? (for RepeatModeler output)\nif [ \"$SORT\" == TRUE ]; then\n echo \"Sorting library\"\n Rscript scripts/final_sorter.R -i ${DATA_DIR}/${RM_LIBRARY}\nfi\n"
},
{
"path": "scripts/TEstrainer/TEstrainer.Rproj",
"content": "Version: 1.0\n\nRestoreWorkspace: Default\nSaveWorkspace: Default\nAlwaysSaveHistory: Default\n\nEnableCodeIndexing: Yes\nUseSpacesForTab: Yes\nNumSpacesForTab: 2\nEncoding: UTF-8\n\nRnwWeave: Sweave\nLaTeX: pdfLaTeX\n"
},
{
"path": "scripts/TEstrainer/TEstrainer_for_earlGrey.sh",
"content": "#!/bin/bash\n\nusage() { echo \"Usage: [-l Repeat library] [-g Genome ] [-t Threads (default 4) ] [-f Flank (default 1000) ] [-m Minimum number of sequences required in multiple sequence alignments during BEAT] [-r Number of iterations of BEAT to run (deafult 10)] [-d Out directory, if not specified wil be created ] [-h Print this help] [-M Amount of memory TEstrainer needs to keep free] [-R Resume interrupted run in directory specified by -d] [-q Suppress parallel progress bar (useful for batch/sbatch jobs)]\" 1>&2; exit 1; }\n\n# default values\nSTRAIN_SCRIPTS=/data/toby/EarlGrey/scripts/TEstrainer/scripts/\nFLANK=1000\nTHREADS=4\nRUNS=10\nNO_SEQ=20\n# for potential folder name\nTIME=$(date +\"%s\")\nTIME=${TIME: -4}\nMEM_FREE=\"1G\"\nMIN_SEQ=3\nBAR_FLAG=\"--bar\"\n\n# parsing\nwhile getopts l:g:t:f:m:r:d:h:n:M:Rq flag; do\n case \"${flag}\" in\n l) RM_LIBRARY_PATH=${OPTARG};;\n g) GENOME=${OPTARG};;\n t) THREADS=${OPTARG};;\n f) FLANK=${OPTARG};;\n m) MIN_SEQ=${OPTARG};;\n r) RUNS=${OPTARG};;\n d) DATA_DIR=${OPTARG};;\n n) NO_SEQ=${OPTARG};;\n M) MEM_FREE=${OPTARG};;\n R) RESUME=true;;\n q) BAR_FLAG=\"\";;\n h | *)\n print_usage\n exit_script\n esac\ndone\n\n# determine further variables and check files exist\nif [ ! -f ${RM_LIBRARY_PATH} ]; then echo \"Library not found\"; usage; fi\nif [ -z ${RM_LIBRARY_PATH} ]; then echo \"Library must be supplied\"; usage; else RM_LIBRARY=$(echo $RM_LIBRARY_PATH | sed 's/.*\\///'); fi\nif [[ $RUNS -gt 0 ]]; then \n if [ -z ${GENOME} ]; then echo \"If refining genome must be supplied\"; usage; fi\n if [ ! -f ${GENOME} ]; then echo \"Refining genome not found\"; usage; fi\nfi\n\nif [[ $THREADS -gt 4 ]]; then MAFFT_THREADS=$(($(($THREADS / 4)))); else MAFFT_THREADS=1; fi\n\n# Cap threads based on available RAM to prevent OOM errors\nAVAIL_MEM_MB=$(free -m | awk '/^Mem:/{print $7}')\nMEM_PER_JOB_MB=800\nMAX_SAFE_THREADS=$(( AVAIL_MEM_MB / MEM_PER_JOB_MB ))\nif [[ $MAX_SAFE_THREADS -gt 0 && $THREADS -gt $MAX_SAFE_THREADS ]]; then\n echo \"Warning: capping threads from ${THREADS} to ${MAX_SAFE_THREADS} based on available RAM (${AVAIL_MEM_MB} MB free)\"\n THREADS=$MAX_SAFE_THREADS\n if [[ $THREADS -gt 4 ]]; then MAFFT_THREADS=$(($(($THREADS / 4)))); else MAFFT_THREADS=1; fi\nfi\n\nif [ -n \"$RESUME\" ]\nthen\n if [ -z \"$DATA_DIR\" ]; then echo \"When resuming, out directory (-d) must be specified\"; usage; fi\n RUN_NO=$(ls ${DATA_DIR} | grep -P '^run_\\d+$' | sed 's/run_//' | sort -n | tail -n 1)\n if [ -z \"$RUN_NO\" ] || [ $RUN_NO -eq 0 ]\n then\n echo \"Attempting to resume a run that was interrupted before iterations started\"\n unset RESUME\n if [ -d ${DATA_DIR}/run_0/ ]; then rm -r ${DATA_DIR}/run_0/ ; fi\n if [ -d ${DATA_DIR}/tmp/ ]; then rm -r ${DATA_DIR}/tmp/ ; fi\n elif [ $RUN_NO -gt $RUNS ]\n then\n echo \"Trying to resume, but more runs than expected have already been completed\"\n usage\n elif [ $RUN_NO -eq $RUNS ] && [ -f ${DATA_DIR}/${RM_LIBRARY} ]\n then\n RUN_NO=$RUN_NO+1\n else\n rm -r ${DATA_DIR}/run_${RUN_NO}\n fi\nfi\n\nif [ -z \"$RESUME\" ]\nthen\n if [ -z \"$DATA_DIR\" ]; then DATA_DIR=$(echo \"TS_\"${RM_LIBRARY}\"_\"${TIME}); fi\n\n # create data dir if missing\n if [ ! -d ${DATA_DIR} ]\n then\n mkdir ${DATA_DIR}\n fi\n\n # make directories\n mkdir -p ${DATA_DIR}/run_0/\n mkdir -p ${DATA_DIR}/tmp/\n\n # initial copy\n cp ${RM_LIBRARY_PATH} ${DATA_DIR}/${RM_LIBRARY}\n\n # cp starting seq to starting directory\n mkdir -p ${DATA_DIR}/run_0/og\n cp ${RM_LIBRARY_PATH} ${DATA_DIR}/run_0/further_${RM_LIBRARY}\n # create reference of original sequences\n python ${STRAIN_SCRIPTS}/splitter.py -i ${DATA_DIR}/run_0/further_${RM_LIBRARY} -o ${DATA_DIR}/run_0/og\n\n # runs\n python ${STRAIN_SCRIPTS}/indexer.py -g ${GENOME}\n if [ ! -f \"${GENOME}\".nsq ]; then\n makeblastdb -in ${GENOME} -dbtype nucl -out ${GENOME} # makeblastb if needed\n fi\n\n # curation\n RUN_NO=1\nfi\n\nwhile [ $RUN_NO -le $RUNS ]\ndo\n\n # make directories\n mkdir -p ${DATA_DIR}/run_${RUN_NO}/raw \\\n ${DATA_DIR}/run_${RUN_NO}/TEtrim_complete \\\n ${DATA_DIR}/run_${RUN_NO}/initial_blast \\\n ${DATA_DIR}/run_${RUN_NO}/self_search \\\n ${DATA_DIR}/run_${RUN_NO}/to_align \\\n ${DATA_DIR}/run_${RUN_NO}/mafft \\\n ${DATA_DIR}/run_${RUN_NO}/TEtrim_con \\\n ${DATA_DIR}/run_${RUN_NO}/TEtrim_unaln \\\n ${DATA_DIR}/run_${RUN_NO}/TEtrim_blast \\\n ${DATA_DIR}/run_${RUN_NO}/TEtrim_mafft \\\n ${DATA_DIR}/run_${RUN_NO}/TEtrim_further \\\n ${DATA_DIR}/run_${RUN_NO}/TEtrim_bp\n \n # split\n cp ${DATA_DIR}/run_$(expr $RUN_NO - 1)/further_${RM_LIBRARY} ${DATA_DIR}/run_${RUN_NO}/${RM_LIBRARY}\n echo \"Splitting run \"${RUN_NO}\n python ${STRAIN_SCRIPTS}/splitter.py -i ${DATA_DIR}/run_${RUN_NO}/${RM_LIBRARY} -o ${DATA_DIR}/run_${RUN_NO}/raw\n\n # run trf to determine if sequence is tandem repeat\n echo \"Initial trf check for \"${RUN_NO}\n parallel ${BAR_FLAG} --tmpdir ${DATA_DIR}/tmp/ --jobs ${THREADS} --memfree ${MEM_FREE} -a ${DATA_DIR}/run_${RUN_NO}/raw/${RM_LIBRARY}_split.txt trf ${DATA_DIR}/run_${RUN_NO}/raw/{} 2 7 7 80 10 50 500 -d -h -ngs \">\" ${DATA_DIR}/run_${RUN_NO}/raw/{}.trf\n echo \"Initial blast and preparation for MSA \"${RUN_NO}\n # initial blast and extention\n parallel ${BAR_FLAG} --tmpdir ${DATA_DIR}/tmp/ --jobs ${THREADS} --memfree ${MEM_FREE} -a ${DATA_DIR}/run_${RUN_NO}/raw/${RM_LIBRARY}_split.txt python3 ${STRAIN_SCRIPTS}/initial_mafft_setup.py -d ${DATA_DIR} -r ${RUN_NO} -s {} -g ${GENOME} -f ${FLANK} -D -n ${NO_SEQ}\n \n ## first mafft alignment\n find ${DATA_DIR}/run_${RUN_NO}/to_align -type f | sed 's/.*\\///' > ${DATA_DIR}/run_${RUN_NO}/to_align.txt\n echo \"Primary alignment run \"${RUN_NO}\n parallel ${BAR_FLAG} --tmpdir ${DATA_DIR}/tmp/ --jobs $MAFFT_THREADS --memfree ${MEM_FREE} -a ${DATA_DIR}/run_${RUN_NO}/to_align.txt mafft --thread 4 --quiet --localpair --adjustdirectionaccurately ${DATA_DIR}/run_${RUN_NO}/to_align/{} \">\" ${DATA_DIR}/run_${RUN_NO}/mafft/{}\n\n # trim\n echo \"Trimming run \"${RUN_NO}\n parallel ${BAR_FLAG} --tmpdir ${DATA_DIR}/tmp/ --jobs $MAFFT_THREADS --memfree ${MEM_FREE} -a ${DATA_DIR}/run_${RUN_NO}/to_align.txt python ${STRAIN_SCRIPTS}/TEtrim.py -i ${DATA_DIR}/run_${RUN_NO}/mafft/{} -t 4 -f ${FLANK} -n ${RUN_NO} -d ${DATA_DIR} -m ${MIN_SEQ}\n \n # compile completed curations\n if [ -n \"$(ls -A ${DATA_DIR}/run_${RUN_NO}/TEtrim_complete/ 2>/dev/null)\" ]; then\n cat ${DATA_DIR}/run_${RUN_NO}/TEtrim_complete/*fasta > ${DATA_DIR}/run_${RUN_NO}/complete_${RM_LIBRARY}\n fi\n \n # Either compile consensuses for further curation\n if [ -n \"$(ls -A ${DATA_DIR}/run_${RUN_NO}/TEtrim_further/ 2>/dev/null)\" ]; then\n cat ${DATA_DIR}/run_${RUN_NO}/TEtrim_further/*fasta > ${DATA_DIR}/run_${RUN_NO}/further_${RM_LIBRARY}\n echo \"Ready for \" $(( RUN_NO++ ))\n else\n # or exit loop if no more curation needed\n echo \"Finished extension\"\n break\n fi\n\ndone\n\n# Compile all completed\ncat ${DATA_DIR}/run_*/complete_${RM_LIBRARY} > ${DATA_DIR}/${RM_LIBRARY}\n# if more could have been extended append to compilation\nif [ -s ${DATA_DIR}/run_${RUNS}/further_${RM_LIBRARY} ]; then\n cat ${DATA_DIR}/run_${RUNS}/further_${RM_LIBRARY} >> ${DATA_DIR}/${RM_LIBRARY}\nfi\n\n# add any families which went missing along the way due to alignment issues\nfind ${DATA_DIR}/run_*/mafft/ -empty -type f | sed 's/mafft/raw/' > ${DATA_DIR}/missing_consensi.txt\nif [[ -s ${DATA_DIR}/missing_consensi.txt ]]; then\n while read file_name; do\n cat $file_name >> ${DATA_DIR}/${RM_LIBRARY}\n done < ${DATA_DIR}/missing_consensi.txt\nfi\n \nsed -i 's/ .*//' ${DATA_DIR}/${RM_LIBRARY}\n\n# Identify simple repeats and satellites, trim ends of LINEs/SINEs\necho \"Splitting for simple/satellite packages\"\nmkdir -p ${DATA_DIR}/trf/split\npython ${STRAIN_SCRIPTS}/splitter.py -i ${DATA_DIR}/${RM_LIBRARY} -o ${DATA_DIR}/trf/split\ncp ${DATA_DIR}/${RM_LIBRARY} ${DATA_DIR}/trf/\n# Run and parse TRF\necho \"Running TRF\"\nparallel ${BAR_FLAG} --tmpdir ${DATA_DIR}/tmp/ --jobs ${THREADS} --memfree ${MEM_FREE} -a ${DATA_DIR}/trf/split/${RM_LIBRARY}_split.txt trf ${DATA_DIR}/trf/split/{} 2 7 7 80 10 50 500 -d -h -ngs \">\" ${DATA_DIR}/trf/split/{}.trf\nparallel ${BAR_FLAG} --tmpdir ${DATA_DIR}/tmp/ --jobs ${THREADS} --memfree ${MEM_FREE} -a ${DATA_DIR}/trf/split/${RM_LIBRARY}_split.txt python3 ${STRAIN_SCRIPTS}/trf_parser.py --trf ${DATA_DIR}/trf/split/{}.trf --out ${DATA_DIR}/trf/split/{}.trf.tsv\nfind ${DATA_DIR}/trf/split/ -type f -name \"*trf.tsv\" -exec cat {} + | cat > ${DATA_DIR}/trf/${RM_LIBRARY}.trf\n# Run SA-SSR\necho \"Running SA-SSR\"\nsa-ssr -e -l 20 -L 50000 -m 1 -M 5000 -t ${THREADS} ${DATA_DIR}/${RM_LIBRARY} ${DATA_DIR}/trf/${RM_LIBRARY}.sassr\n# Run and compile mreps\necho \"Running mreps\"\nparallel ${BAR_FLAG} --tmpdir ${DATA_DIR}/tmp/ --jobs ${THREADS} --memfree ${MEM_FREE} -a ${DATA_DIR}/trf/split/${RM_LIBRARY}_split.txt bash ${STRAIN_SCRIPTS}/mreps_parser.sh -i ${DATA_DIR}/trf/split/{} \"2>\"/dev/null\nfind ${DATA_DIR}/trf/split/ -type f -name \"*mreps\" -exec cat {} + | cat > ${DATA_DIR}/trf/${RM_LIBRARY}.mreps\n# Interpret mreps, TRF and SA-SSR\necho \"Trimming and sorting based on mreps, TRF, SA-SSR\"\nif [ ! -f ${DATA_DIR}/trf/${RM_LIBRARY}.sassr ]; then\n touch ${DATA_DIR}/trf/${RM_LIBRARY}.sassr\nfi\nRscript ${STRAIN_SCRIPTS}/simple_repeat_filter_trim.R -i ${DATA_DIR}/${RM_LIBRARY} -d ${DATA_DIR} -p 50\n\n# Delete temp files\necho \"Removing temporary files\"\nrm -r ${DATA_DIR}/*/split/\nrm -r ${DATA_DIR}/tmp/\nfind ${DATA_DIR}/ -mindepth 1 -name \"run_*\" -exec rm -r {} +\nif [[ $RUNS -gt 0 ]]; then rm ${GENOME}.n*; fi\n\n# Classify improved consensi using RepeatModeler's RepeatClassifier\necho \"Reclassifying repeats\"\nmkdir -p ${DATA_DIR}/classify/\ncp ${DATA_DIR}/trf/${RM_LIBRARY}.nonsatellite ${DATA_DIR}/classify/\ncd ${DATA_DIR}/classify/\nif [ -s ${RM_LIBRARY}.nonsatellite ]; then\n RepeatClassifier -threads ${THREADS} -consensi ${RM_LIBRARY}.nonsatellite\nfi\n# legacy command for older installations - DEPRECATED\n# RepeatClassifier -pa ${THREADS} -consensi ${RM_LIBRARY}.nonsatellite\ncd -\n# Compile classified files\nif [ -f ${DATA_DIR}/classify/${RM_LIBRARY}.nonsatellite.classified ]; then\n cp ${DATA_DIR}/classify/${RM_LIBRARY}.nonsatellite.classified ${DATA_DIR}/${RM_LIBRARY}.strained\nelse\n touch ${DATA_DIR}/${RM_LIBRARY}.strained\nfi\necho \"Compiling library\"\nif [ -f ${DATA_DIR}/trf/${RM_LIBRARY}.satellites ]; then\n cat ${DATA_DIR}/trf/${RM_LIBRARY}.satellites >> ${DATA_DIR}/${RM_LIBRARY}.strained\nfi\n"
},
{
"path": "scripts/TEstrainer/data/acceptable_domains.tsv",
"content": "ref\tabbrev\npfam17917\tRT_RNaseH\ncd09274\tRNase_HI_RT_Ty3\ncd01647\tRT_LTR\npfam17919\tRT_RNaseH_2\npfam00078\tRVT_1\npfam17921\tIntegrase_H2C2\ncd03715\tRT_ZFREV_like\npfam00665\trve\ncd01645\tRT_Rtv\ncd00303\tretropepsin_like\ncd03714\tRT_DIRS1\ncd09275\tRNase_HI_RT_DIRS1\ncd00304\tRT_like\npfam13975\tgag-asp_proteas\npfam03732\tRetrotrans_gag\npfam13650\tAsp_protease_2\nNF033577\ttranspos_IS481\ncd06222\tRNase_H_like\ncd05484\tretropepsin_like_LTR_2\ncd05483\tretropepsin_like_bacteria\npfam08284\tRVP_2\ncd01644\tRT_pepA17\npfam09337\tzf-H2C2\ncd00024\tCD_CSD\npfam00385\tChromo\ncd05481\tretropepsin_like_LTR_1\nsmart00298\tCHROMO\npfam19259\tTy3_capsid\ncd18975\tCD_MarY1_POL_like\ncd18978\tCD_DDE_transposase_like\ncd18973\tCD_Tf2-1_POL_like\ncd18631\tCD_HP1_like\npfam00098\tzf-CCHC\nsmart00343\tZnF_C2HC\ncd18634\tCD_CDY\ncd18635\tCD_CMT3_like\npfam13456\tRVT_3\ncd18644\tCD_polycomb\ncd06095\tRP_RTVL_H_like\ncd18980\tCD_NC-like\ncd18650\tCD_HP1beta_Cbx1\ncd18979\tCD_POL_like\ncd09279\tRNase_HI_like\ncd18630\tCD_Rhino\ncd18653\tCD_HP1a_insect\ncd05479\tRP_DDI\ncd18627\tCD_polycomb_like\ncd18651\tCD_HP1alpha_Cbx5\ncd18659\tCD2_tandem\ncd18647\tCD_Cbx2\ncd18633\tCD_MMP8\ncd18638\tCD_EhHp1_like\nCOG0328\tRnhA\ncd06094\tRP_Saci_like\ncd18652\tCD_HP1gamma_Cbx3\ncd18632\tCD_Clr4_like\ncd18968\tchromodomain\ncd18639\tCD_SUV39H1_like\ncd18645\tCD_Cbx4\npfam16297\tDUF4939\ncd18637\tCD_Swi6_like\ncd18646\tCD_Cbx7\ncd18649\tCD_Cbx8\npfam13683\trve_3\ncd18966\tchromodomain\nPRK07708\tPRK07708\nPTZ00368\tPTZ00368\ncd18648\tCD_Cbx6\ncd18962\tCD_MT_like\ncd18970\tCD_POL_like\ncd18640\tCD_Chro-like\nPRK13907\trnhA\ncd09272\tRNase_HI_RT_Ty1\npfam07727\tRVT_2\npfam13976\tgag_pre-integrs\npfam14223\tRetrotran_gag_2\npfam14244\tRetrotran_gag_3\npfam13961\tDUF4219\nCOG2801\tTra5\ncd01650\tRT_nLTR_like\ncd09076\tL1-EN\ncd08372\tEEP\npfam03372\tExo_endo_phos\ncd01651\tRT_G2_intron\ncd09073\tExoIII_AP-endo\npfam13966\tzf-RVT\ncd09087\tApe1-like_AP-endo\ncd09086\tExoIII-like_AP-endo\nTIGR00633\tDNA-apurinic_or_apyrimidinic_site_lyase\ncd09085\tMth212-like_AP-endo\ncd10281\tNape_like_AP-endo\npfam14529\tExo_endo_phos_2\nCOG0708\tXthA\ncd09077\tR1-I-EN\ncd09088\tApe2-like_AP-endo\ncd03487\tRT_Bac_retron_II\npfam02994\tTransposase_22\npfam14111\tDUF4283\ncd09080\tTDP2\nTIGR00195\tExodeoxyribonuclease_III\npfam17490\tTnp_22_dsRBD\ncd09084\tEEP-2\nPRK13911\tPRK13911\nPRK11756\tPRK11756\npfam14392\tzf-CCHC_4\ncd09280\tRNase_HI_eukaryote_like\ncd09276\tRnase_HI_RT_non_LTR\npfam00075\tRNase_H\nCOG3568\tElsH\ncd09278\tRNase_HI_prokaryote_like\npfam05699\tDimer_Tnp_hAT\nsmart00614\tZnF_BED\npfam14291\tDUF4371\npfam14372\tDUF4413\npfam04937\tDUF659\npfam02892\tzf-BED\nsmart00597\tZnF_TTF\npfam18658\tzf-C2H2_12\nNF033545\ttranspos_IS630\npfam13358\tDDE_3\npfam03184\tDDE_1\npfam13565\tHTH_32\npfam01498\tHTH_Tnp_Tc3_2\nCOG3415\tCOG3415\npfam03221\tHTH_Tnp_Tc5\nsmart00674\tCENPB\npfam01359\tTransposase_1\npfam17906\tHTH_48\npfam13384\tHTH_23\npfam13518\tHTH_28\nCOG3335\tCOG3335\npfam04218\tCENP-B_N\npfam13936\tHTH_38\npfam05225\tHTH_psq\npfam16087\tDUF4817\npfam13412\tHTH_24\nsmart00351\tPAX\npfam18701\tDUF5641\npfam05380\tPeptidase_A17\npfam03564\tDUF1759\npfam05585\tDUF1758\ncd15489\tPHD_SF\ncd15522\tPHD_TAF3\ncd15517\tPHD_TCF19_like\nsmart00249\tPHD\ncd15560\tPHD2_3_BPTF\ncd15552\tPHD_PHF3_like\ncd15610\tPHD3_KDM5A_like\ncd15553\tPHD_Cfp1\ncd15556\tPHD_MMD1_like\ncd16039\tPHD_SPP1\ncd15529\tPHD2_PHF10\ncd15554\tPHD_PHF2_like\ncd15550\tPHD_MLL5\ncd15518\tPHD_Ecm5p_Lid2p_like\ncd15505\tPHD_ING\ncd15640\tPHD_KDM7\ncd15551\tPHD_PYGO\ncd15525\tPHD_UHRF1_2\ncd15638\tPHD_PHF3\ncd15510\tPHD2_KMT2C_like\ncd15520\tPHD3_Lid2p_like\ncd15613\tPHD_AL_plant\ncd15587\tPHD_Yng1p_like\ncd15585\tPHD_ING3\ncd15627\tPHD_BAZ1A\ncd15687\tPHD3_KDM5B\ncd15570\tPHD_Bye1p_SIZ1_like\ncd15563\tPHD3_PHF14\ncd15609\tPHD_TCF19\ncd15639\tPHD_DIDO1_like\ncd15543\tPHD_RSF1\ncd15544\tPHD_BAZ1A_like\ncd15594\tPHD2_KMT2C\ncd15508\tPHD3_KMT2A_like\ncd15495\tPHD_ATX3_4_5_like\ncd15506\tPHD1_KMT2A_like\ncd15595\tPHD2_KMT2D\npfam00628\tPHD\ncd15545\tPHD_BAZ2A_like\ncd15519\tPHD1_Lid2p_like\ncd15586\tPHD_ING4_5\ncd15681\tPHD_JADE3\ncd15535\tPHD1_Rco1\ncd15632\tPHD_PHF13\ncd15679\tPHD_JADE1\ncd01646\tRT_Bac_retron_I\nTIGR04416\thypothetical_protein\ncd00229\tSGNH_hydrolase\npfam13472\tLipase_GDSL_2\ncd01828\tsialate_O-acetylesterase_like2\nCOG3344\tCOG3344\ncd04501\tSGNH_hydrolase_like_4\ncd01832\tSGNH_hydrolase_like_1\ncd04502\tSGNH_hydrolase_like_7\ncd15636\tPHD_PYGO2\nCOG2755\tTesA\ncd01841\tNnaC_like\ncd15635\tPHD_PYGO1\ncd01833\tXynB_like\ncd01648\tTERT\ncd01836\tFeeA_FeeB_like\ncd01825\tSGNH_hydrolase_peri1\ncd15637\tPHD_dPYGO\npfam10551\tMULE\nsmart00575\tZnF_PMZ\npfam04434\tSWIM\npfam03108\tDBD_Tnp_Mut\npfam03101\tFAR1\nPLN03097\tFHY3\npfam09331\tDUF1985\npfam02902\tPeptidase_C48\npfam10536\tPMD\npfam00872\tTransposase_mut\nCOG3328\tCOG3328\ncd22744\tOTU\ncd00799\tINT_Cre_C\npfam11560\tLAP2alpha\ncd00397\tDNA_BRE_C\npfam00589\tPhage_integrase\npfam05869\tDam\npfam13495\tPhage_int_SAM_4\npfam02899\tPhage_int_SAM_1\ncd01194\tINT_C_like_4\ncd01188\tINT_RitA_C_like\ncd00796\tINT_Rci_Hp1_C\npfam02992\tTransposase_21\npfam13963\tTranspos_assoc\npfam13960\tDUF4218\npfam13952\tDUF4216\npfam03004\tTransposase_24\npfam03017\tTransposase_23\npfam20049\tDUF6451\ncd10442\tGIY-YIG_PLEs\ncd10443\tGIY-YIG_HE_Tlr8p_PBC-V_like\ncd00719\tGIY-YIG_SF\ncd10437\tGIY-YIG_HE_I-TevI_like\ncd09277\tRNase_HI_bacteria_like\ncd09273\tRNase_HI_RT_Bel\nPRK00203\trnhA\ncd09081\tCdtB\npfam13359\tDDE_Tnp_4\npfam04827\tPlant_tran\npfam14303\tNAM-associated\nNF033581\ttranspos_IS5_4\npfam13837\tMyb_DNA-bind_4\ncd12203\tGT1\npfam01609\tDDE_Tnp_1\ncd10522\tSET_LegAS4-like\ncd19174\tSET_ASH1L\ncd19183\tSET_SpSET3-like\ncd20071\tSET_SMYD\ncd10529\tSET_SETD5-like\ncd10531\tSET_SETD2-like\ncd10533\tSET_EHMT2\ncd10535\tSET_EHMT1\ncd10538\tSET_SETDB-like\ncd19172\tSET_SETD2\ncd19175\tSET_ASHR3-like\ncd19211\tSET_NSD2\ncd20072\tSET_SET1\npfam10545\tMADF_DNA_bdg\npfam18697\tMLVIN_C\npfam00077\tRVP\ncd09950\tENVV1-like_HR1-HR2\ncd09948\tEbola_RSV-like_HR1-HR2\npfam02093\tGag_p30\ncd09951\tHERV-Rb-like_HR1-HR2\ncd09851\tHTLV-1-like_HR1-HR2\ncd09850\tEbola-like_HR1-HR2\ncd09947\tEbola_HIV-1-like_HR1-HR2\ncd13934\tRNase_H_Dikarya_like\npfam00429\tTLV_coat\npfam01140\tGag_MA\ncd05482\tHIV_retropepsin_like\ncd09949\tRSV-like_HR1-HR2\nPRK08719\tPRK08719\npfam16721\tzf-H3C2\nPRK06548\tPRK06548\ncd18037\tDEXSc_Pif1_like\npfam05970\tPIF1\npfam14214\tHelitron_like_N\ncd17933\tDEXSc_RecD-like\ncd18809\tSF1_C_RecD\npfam13604\tAAA_30\ncd18786\tSF1_C\npfam13245\tAAA_19\ncd04480\tRPA1_DBD_A_like\nCOG0507\tRecD\ncd04476\tRPA1_DBD_C\ncd04481\tRPA1_DBD_B_like\npfam02721\tDUF223\npfam13401\tAAA_22\npfam13538\tUvrD_C_2\npfam04843\tHerpes_teg_N\ncd22758\tOTU_232R-like\nsmart00382\tAAA\nTIGR01448\trecD_rel\ncd22748\tOTU_OTUD6-like\ncd22755\tOTU_CeDUB-like\ncd22756\tOTU_OTUD3-like\ncd22747\tOTU_OTUD1\ncd22752\tOTU_OTUD5-like\ncd22753\tOTU_ALG13-like\ncd22771\tOTU_plant_OTU7-like\ncd22751\tOTU_plant_OTU9-like\ncd22757\tOTU_P87_VP80-like\ncd22762\tOTU_fungi_OTU2-like\ncd22794\tOTU_OTUD4\ncd22796\tOTU_plant_OTU6-like\ncd09083\tEEP-1\npfam07530\tPRE_C2HC\npfam00607\tGag_p24\npfam06817\tRVT_thumb\npfam19317\tGag_p24_C\npfam02022\tIntegrase_Zn\npfam00552\tIN_DBD_C\npfam00692\tdUTPase\npfam14787\tzf-CCHC_5\ncd07557\ttrimeric_dUTPase\nPRK00601\tdut\npfam02337\tGag_p10\nTIGR00576\tDeoxyuridine_5'-triphosphate_nucleotidohydrolase\nCOG0756\tDut\nPLN02547\tPLN02547\nPHA02703\tPHA02703\nPHA03094\tPHA03094\nsmart00443\tG_patch\npfam01585\tG-patch\npfam00517\tGP41\ncd09909\tHIV-1-like_HR1-HR2\nPTZ00143\tPTZ00143\npfam04160\tBorrelia_orfX\nPHA02517\tPHA02517\npfam15695\tHERV-K_REC\npfam12656\tG-patch_2\npfam13804\tHERV-K_env_2\nNF033594\ttranspos_ISNCY_2\nPHA03131\tPHA03131\npfam02813\tRetro_M\nCOG5082\tAIR1\npfam13843\tDDE_Tnp_1_7\ncd22343\tPDDEXK_lambda_exonuclease-like\ncd00180\tPKc\ncd05122\tPKc_STE\ncd06917\tSTKc_NAK1_like\ncd07832\tSTKc_CCRK\ncd07847\tSTKc_CDKL1_4\ncd13982\tSTKc_IRE1\ncd14014\tSTKc_PknB_like\ncd14133\tPKc_DYRK_like\ncd14229\tSTKc_HIPK3\npfam13842\tTnp_zf-ribbon_2\npfam01107\tMP\npfam02160\tPeptidase_A3\ncd17920\tDEXHc_RecQ\nsmart00490\tHELICc\ncd18018\tDEXHc_RecQ4-like\npfam00271\tHelicase_C\ncd18794\tSF2_C_RecQ\ncd18017\tDEXHc_RecQ3\npfam00270\tDEAD\ncd18016\tDEXHc_RecQ2_BLM\nsmart00487\tDEXDc\ncd18015\tDEXHc_RecQ1\ncd18014\tDEXHc_RecQ5\nCOG0514\tRecQ\ncd00046\tSF2-N\ncd17921\tDEXHc_Ski2\nPRK11057\tPRK11057\nTIGR00614\tWerner_syndrome_ATP-dependent_helicase\ncd18785\tSF2_C\ncd18787\tSF2_C_DEAD\ncd17923\tDEXHc_Hrq1-like\ncd18797\tSF2_C_Hrq\ncd18793\tSF2_C_SNF\ncd18796\tSF2_C_LHR\ncd17927\tDEXHc_RIG-I\nCOG0513\tSrmB\npfam04851\tResIII\ncd18028\tDEXHc_archSki2\ncd18802\tSF2_C_dicer\ncd17926\tDEXHc_RE\ncd18023\tDEXHc_HFM1\ncd18795\tSF2_C_Ski2\ncd09639\tCas3_I\ncd17922\tDEXHc_LHR-like\ncd18811\tSF2_C_RecG\ncd01820\tPAF_acetylesterase_like\npfam03175\tDNA_pol_B_2\npfam08398\tParvo_coat_N\nPHA02563\tPHA02563\npfam04665\tPox_A32\nNF033516\ttranspos_IS3\ncd18974\tCD_POL_like\npfam02229\tPC4\ncd18965\tchromodomain\npfam19263\tDUF5906\ncd00652\tTBP_TLF\ncd04516\tTBP_eukaryotes\ncd04517\tTLF\ncd04518\tTBP_archaea\ncd18976\tCD_POL_like\nCOG2101\tSPT15\npfam00352\tTBP\npfam02945\tEndonuclease_7\nPLN00062\tPLN00062\nPRK00394\tPRK00394\npfam05272\tVirE\npfam18930\tDUF5679\ncd00009\tAAA\ncd00022\tBIR\ncd00094\tHX\ncd18969\tchromodomain\ncd18972\tCD_POL_like\npfam04383\tKilA-N\npfam12943\tDUF3839\nsd00017\tZF_C2H2\npfam13894\tzf-C2H2_4\ncd00167\tSANT\npfam05485\tTHAP\npfam12017\tTnp_P_element\nsmart00692\tDM3\nsmart00980\tTHAP\npfam12596\tTnp_P_element_C\nPRK07764\tPRK07764\npfam09004\tDUF1891\npfam01021\tTYA\npfam09668\tAsp_protease\nTIGR03033\thypothetical_protein\npfam09588\tYqaJ\npfam13613\tHTH_Tnp_4\ncd15682\tPHD_ING1\ncd15683\tPHD_ING2\ncd15685\tPHD_ING5\ncd15584\tPHD_ING1_2\ncd15684\tPHD_ING4\npfam02037\tSAP\npfam16012\tDUF4780\ncd22761\tOTU_OTUD6\ncd22770\tOTU_OTUD3\ncd22797\tOTU_plant_OTU5-like\ncd22760\tOTU_plant_OTU4-like\ncd22746\tOTU_plant_OTU3_4-like\ncd22795\tOTU_ALG13\ncd22759\tOTU_plant_OTU3-like\ncd00096\tIg\ncd05717\tIgV_1_Necl_like\ncd05718\tIgV_1_PVR_like\ncd05719\tIgC1_2_PVR_like\ncd05725\tIgI_3_Robo\ncd05846\tIgV_1_MRC-OX-2_like\ncd05882\tIgV_1_Necl-1\ncd05886\tIgV_1_Nectin-1_like\ncd05887\tIgV_1_Nectin-3_like\ncd05888\tIgV_1_Nectin-4_like\ncd05889\tIgV_1_DNAM-1_like\ncd05890\tIgC1_2_Nectin-1_like\ncd07703\tIgC1_2_Nectin-2_Necl-5_like\ncd07704\tIgC1_2_Nectin-3-4_like\ncd16089\tIgV_CRIg\ncd20930\tIg3_Nectin-5_like\ncd20989\tIgV_1_Nectin-2_NecL-5_like_CD112_CD155\npfam07686\tV-set\npfam08205\tC2-set_2\npfam13927\tIg_3\nsmart00410\tIG_like\npfam16124\tRecQ_Zn_bind\nNF033563\ttranspos_IS30\ncd00590\tRRM_SF\ncd12224\tRRM_RBM22\ncd12229\tRRM_G3BP\ncd12233\tRRM_Srp1p_AtRSp31_like\ncd12236\tRRM_snRNP70\ncd12237\tRRM_snRNP35\ncd12242\tRRM_SLIRP\ncd12297\tRRM2_Prp24\ncd12312\tRRM_SRSF10_SRSF12\ncd12316\tRRM3_RBM19_RRM2_MRD1\ncd12320\tRRM6_RBM19_RRM5_MRD1\ncd12324\tRRM_RBM8\ncd12337\tRRM1_SRSF4_like\ncd12338\tRRM1_SRSF1_like\ncd12341\tRRM_hnRNPC_like\ncd12343\tRRM1_2_CoAA_like\ncd12347\tRRM_PPIE\ncd12363\tRRM_TRA2\ncd12373\tRRM_SRSF3_like\ncd12382\tRRM_RBMX_like\ncd12390\tRRM3_RAVER\ncd12391\tRRM1_SART3\ncd12392\tRRM2_SART3\ncd12398\tRRM_CSTF2_RNA15_like\ncd12399\tRRM_HP0827_like\ncd12400\tRRM_Nop6\ncd12402\tRRM_eIF4B\ncd12404\tRRM2_NCL\ncd12412\tRRM_DAZL_BOULE\ncd12413\tRRM1_RBM28_like\ncd12414\tRRM2_RBM28_like\ncd12448\tRRM2_gar2\ncd12449\tRRM_CIRBP_RBM3\ncd12450\tRRM1_NUCLs\ncd12494\tRRM3_hnRNPR\ncd12597\tRRM1_SRSF1\ncd12599\tRRM1_SF2_plant_like\ncd12608\tRRM1_CoAA\ncd12641\tRRM_TRA2B\ncd12642\tRRM_TRA2A\ncd12672\tRRM_DAZL\ncd12674\tRRM1_Nop4p\ncd12676\tRRM3_Nop4p\ncd21607\tRRM3_HRB1_GBP2\ncd21608\tRRM2_NsCP33_like\ncd21615\tRRM_SNP1_like\nCOG1877\tOtsB\npfam00076\tRRM_1\nsmart00360\tRRM\ncd15492\tPHD_BRPF_JADE_like\npfam02257\tRFX_DNA_binding\ncd15536\tPHD_PHRF1\ncd15573\tPHD_JADE\ncd15515\tPHD1_KDM5A_like\ncd15532\tPHD2_CHD_II\ncd15572\tPHD_BRPF\ncd15603\tPHD1_KDM5B\ncd15628\tPHD_BAZ1B\ncd15677\tPHD_BRPF2\ncd15680\tPHD_JADE2\npfam12012\tDUF3504\ncd06141\tWRN_exo\ncd06146\tmut-7_like_exo\ncd06129\tRNaseD_like\ncd09018\tDEDDy_polA_RNaseD_like_exo\npfam01612\tDNA_pol_A_exo1\nsmart00474\t35EXOc\ncd06142\tRNaseD_exo\npfam06975\tDUF1299\npfam18717\tCxC4\npfam18758\tKDZ\npfam18866\tCxC7\npfam12550\tGCR1_C\npfam16787\tNDC10_II\nNF033547\ttranspos_IS1595\npfam12762\tDDE_Tnp_IS1595\ncd22792\tOTU_RDRP-like\n"
},
{
"path": "scripts/TEstrainer/data/additional_domains.tsv",
"content": "class\tref\tabbrev\tclade\nPao\tNF033838\tPspC_subgroup_1\tAmphibia\nPao\tPRK14475\tPRK14475\tAmphibia\nPao\tTIGR02499\ttranslocation_protein\tAmphibia\nPao\tcd06503\tATP-synt_Fo_b\tAmphibia\nPao\tpfam16999\tV-ATPase_G_2\tAmphibia\nPao\tPRK02292\tPRK02292\tAmphibia\nPao\tcd06503\tATP-synt_Fo_b\tAmphibia\nPao\tpfam16999\tV-ATPase_G_2\tAmphibia\nPao\tsmart00935\tOmpH\tAmphibia\nPao\tpfam00430\tATP-synt_B\tAmphibia\nPao\tCOG0711\tAtpF\tAmphibia\nPao\tPRK08475\tPRK08475\tAmphibia\nPao\tpfam12072\tDUF3552\tAmphibia\nPao\tPRK05759\tPRK05759\tAmphibia\nPao\tPRK07353\tPRK07353\tAmphibia\nPao\tCOG2811\tNtpF\tAmphibia\nPao\tTIGR02926\tUncharacterized_protein_AF_1158\tAmphibia\nPao\tpfam15975\tFlot\tAmphibia\nGypsy\tpfam14893\tPNMA\tAmphibia\nTcMar-Tc1\tCOG2512\tCOG2512\tAmphibia\nTcMar-Tc1\tCOG2522\tCOG2522\tAmphibia\nLINE\tcd14695\tbZIP_HLF\tAmphibia\nLINE\tCOG1579\tCOG1579\tAmphibia\nLINE\tpfam08614\tATG16\tAmphibia\nPIF-Harbinger\tpfam02944\tBESS\tAmphibia\nGypsy\tcd07936\tSCAN\tAmphibia\nERV\tpfam07679\tI-set\tAmphibia\nERV\tcd07700\tIgV_CD8_beta\tAmphibia\nGypsy\tpfam13696\tzf-CCHC_2\tAmphibia\nGypsy\tcd18977\tCD_POL_like\tAmphibia\nHelitron\tcd17914\tDExxQc_SF1-N\tAmphibia\nHelitron\tcd02661\tPeptidase_C19E\tAmphibia\nHelitron\tpfam00443\tUCH\tAmphibia\nHelitron\tcd02668\tPeptidase_C19L\tAmphibia\nHelitron\tCOG5533\tUBP5\tAmphibia\nHelitron\tcd02667\tPeptidase_C19K\tAmphibia\nHelitron\tcd02257\tPeptidase_C19\tAmphibia\nHelitron\tcd02674\tPeptidase_C19R\tAmphibia\nHelitron\tcd02662\tPeptidase_C19F\tAmphibia\nHelitron\tcd02658\tPeptidase_C19B\tAmphibia\nHelitron\tTIGR01447\trecD\tAmphibia\nGypsy\tcd18964\tchromodomain\tAmphibia\nHelitron\tcd02673\tPeptidase_C19Q\tAmphibia\nERV\tpfam02813\tRetro_M\tAves\nERV\tpfam03708\tAvian_gp85\tAves\nR2\tpfam13921\tMyb_DNA-bind_6\tAves\nERV\tCOG1340\tCOG1340\tAves\nERV\tCOG4026\tCOG4026\tAves\nERV\tpfam00769\tERM\tAves\nERV\tCOG4942\tCOG4942\tAves\nDNA_virus\tpfam00242\tDNA_pol_viral_N\tAves\nDNA_virus\tpfam00336\tDNA_pol_viral_C\tAves\nDNA_virus\tpfam00906\tHepatitis_core\tAves\nDNA_virus\tpfam00695\tvMSA\tAves\nERV\tpfam10473\tCENP-F_leu_zip\tAves\nERV\tpfam16888\tDUF5082\tAves\nERV\tpfam10473\tCENP-F_leu_zip\tAves\nERV\tpfam02403\tSeryl_tRNA_N\tAves\nERV\tTIGR03698\tclan_AA_DTGF\tAves\nERV\tCOG5550\tCOG5550\tAves\nERV\tpfam00047\tig\tAves\nERV\tcd05740\tIgI_hCEACAM_2_4_6_like\tAves\nERV\tcd20935\tIgV_B7-H2\tAves\nERV\tTIGR02794\tProtein_TolA\tAves\nERV\tpfam05917\tDUF874\tAves\nERV\tpfam11559\tADIP\tAves\nERV\tpfam03938\tOmpH\tAves\nERV\tcd14717\tbZIP_Maf_small\tAves\nERV\tcd22887\tAtg16_CCD\tAves\nERV\tpfam14662\tKASH_CCD\tAves\nERV\tcd22887\tAtg16_CCD\tAves\nERV\tpfam00529\tHlyD\tAves\nERV\tpfam12795\tMscS_porin\tAves\nERV\tcd22887\tAtg16_CCD\tAves\nL1\tpfam06825\tHSBP1\tMammalia\nERV\tPRK13956\tdut\tMammalia\nERV\tpfam13804\tHERV-K_env_2\tMammalia\nERV\tpfam01141\tGag_p12\tMammalia\nERV-Foamy\tpfam03276\tGag_spuma\tMammalia\nERV-Foamy\tpfam03539\tSpuma_A9PTase\tMammalia\nERV-Foamy\tpfam03408\tFoamy_virus_ENV\tMammalia\nL1\tpfam01442\tApolipoprotein\tLepidosauria\nGypsy\tpfam04814\tHNF-1_N\tTestudines\nGypsy\tcd14693\tbZIP_CEBP\tTestudines\nERV\tcd02901\tMacro_Poa1p-like\tActinopterygii\nERV\tsmart00506\tA1pp\tActinopterygii\nERV\tcd02749\tMacro_SF\tActinopterygii\nERV\tPHA02595\ttk.4\tActinopterygii\nERV\tpfam01661\tMacro\tActinopterygii\nRex-Babar\tpfam16761\tClr2_transil\tActinopterygii\nGypsy\tpfam02023\tSCAN\tActinopterygii\nGypsy\tsmart00431\tSCAN\tActinopterygii\nGypsy\tpfam03078\tATHILA\tPlantae\nGypsy\tpfam04195\tTransposase_28\tPlantae\nCaulimovirus\tpfam03938\tOmpH\tPlantae\nCaulimovirus\tPRK03957\tPRK03957\tPlantae\nCopia\tpfam13696\tzf-CCHC_2\tPlantae\nGypsy\tpfam15294\tLeu_zip\tPlantae\nGypsy\tPRK07238\tPRK07238\tPlantae\nMULE-MuDR\tcd06410\tPB1_UP2\tPlantae\nMULE-MuDR\tpfam09322\tDUF1979\tPlantae\nMULE-MuDR\tpfam05928\tZea_mays_MuDR\tPlantae\n"
},
{
"path": "scripts/TEstrainer/data/exceptional_domains.tsv",
"content": "class\tref\tabbrev\tclade\nERV\tcd00096\tIg\tVertebrata\nERV\tcd05717\tIgV_1_Necl_like\tVertebrata\nERV\tcd05718\tIgV_1_PVR_like\tVertebrata\nERV\tcd05719\tIgC1_2_PVR_like\tVertebrata\nERV\tcd05725\tIgI_3_Robo\tVertebrata\nERV\tcd05759\tIgI_2_KIRREL3-like\tVertebrata\nERV\tcd05846\tIgV_1_MRC-OX-2_like\tVertebrata\nERV\tcd05882\tIgV_1_Necl-1\tVertebrata\nERV\tcd05886\tIgV_1_Nectin-1_like\tVertebrata\nERV\tcd05887\tIgV_1_Nectin-3_like\tVertebrata\nERV\tcd05888\tIgV_1_Nectin-4_like\tVertebrata\nERV\tcd05889\tIgV_1_DNAM-1_like\tVertebrata\nERV\tcd05890\tIgC1_2_Nectin-1_like\tVertebrata\nERV\tcd07703\tIgC1_2_Nectin-2_Necl-5_like\tVertebrata\nERV\tcd07704\tIgC1_2_Nectin-3-4_like\tVertebrata\nERV\tcd16089\tIgV_CRIg\tVertebrata\nERV\tcd20930\tIg3_Nectin-5_like\tVertebrata\nERV\tcd20946\tIgV_1_JAM1-like\tVertebrata\nERV\tcd20989\tIgV_1_Nectin-2_NecL-5_like_CD112_CD155\tVertebrata\nERV\tpfam07686\tV-set\tVertebrata\nERV\tpfam13927\tIg_3\tVertebrata\nERV\tsmart00410\tIG_like\tVertebrata\nPolinton\tcd04277\tZnMc_serralysin_like\tInsecta\nPolinton\tcd04278\tZnMc_MMP\tInsecta\nPolinton\tcd04279\tZnMc_MMP_like_1\tInsecta\nPolinton\tsmart00235\tZnMc\tInsecta\n"
},
{
"path": "scripts/TEstrainer/data/old_acceptable_domains.tsv",
"content": "ref\tabbrev\ncd09274\tRNase_HI_RT_Ty3\ncd01647\tRT_LTR\npfam17917\tRT_RNaseH\npfam17919\tRT_RNaseH_2\npfam00078\tRVT_1\npfam17921\tIntegrase_H2C2\ncd03715\tRT_ZFREV_like\ncd01645\tRT_Rtv\npfam00665\trve\ncd00303\tretropepsin_like\ncd03714\tRT_DIRS1\ncd09275\tRNase_HI_RT_DIRS1\ncd00304\tRT_like\npfam03732\tRetrotrans_gag\npfam13975\tgag-asp_proteas\nNF033577\ttranspos_IS481\npfam13650\tAsp_protease_2\ncd06222\tRNase_H_like\ncd05484\tretropepsin_like_LTR_2\npfam09337\tzf-H2C2\ncd05483\tretropepsin_like_bacteria\npfam08284\tRVP_2\ncd01644\tRT_pepA17\ncd00024\tCD_CSD\npfam00385\tChromo\ncd05481\tretropepsin_like_LTR_1\nsmart00298\tCHROMO\ncd18975\tCD_MarY1_POL_like\ncd18978\tCD_DDE_transposase_like\ncd18973\tCD_Tf2-1_POL_like\ncd18631\tCD_HP1_like\ncd18966\tchromodomain\ncd18968\tchromodomain\nsmart00343\tZnF_C2HC\ncd18979\tCD_POL_like\ncd18634\tCD_CDY\ncd18635\tCD_CMT3_like\ncd18970\tCD_POL_like\npfam13456\tRVT_3\ncd18644\tCD_polycomb\ncd06095\tRP_RTVL_H_like\ncd18976\tCD_POL_like\ncd18980\tCD_NC-like\ncd18650\tCD_HP1beta_Cbx1\ncd09279\tRNase_HI_like\ncd18630\tCD_Rhino\ncd18653\tCD_HP1a_insect\ncd05479\tRP_DDI\ncd18627\tCD_polycomb_like\ncd18651\tCD_HP1alpha_Cbx5\ncd18659\tCD2_tandem\ncd18647\tCD_Cbx2\ncd18633\tCD_MMP8\ncd18974\tCD_POL_like\ncd18960\tCD_HP1_like\ncd18638\tCD_EhHp1_like\nCOG0328\tRnhA\ncd18971\tCD_POL_like\ncd18977\tCD_POL_like\nPTZ00368\tPTZ00368\npfam00098\tzf-CCHC\ncd06094\tRP_Saci_like\ncd18652\tCD_HP1gamma_Cbx3\ncd18969\tchromodomain\ncd18632\tCD_Clr4_like\ncd09272\tRNase_HI_RT_Ty1\npfam07727\tRVT_2\npfam13976\tgag_pre-integrs\npfam14223\tRetrotran_gag_2\npfam14244\tRetrotran_gag_3\npfam13961\tDUF4219\npfam13683\trve_3\ncd01650\tRT_nLTR_like\ncd09076\tL1-EN\npfam03372\tExo_endo_phos\ncd08372\tEEP\ncd01651\tRT_G2_intron\ncd09073\tExoIII_AP-endo\npfam13966\tzf-RVT\ncd09087\tApe1-like_AP-endo\ncd09086\tExoIII-like_AP-endo\ncd09085\tMth212-like_AP-endo\ncd10281\tNape_like_AP-endo\npfam14529\tExo_endo_phos_2\nCOG0708\tXthA\ncd09077\tR1-I-EN\ncd09088\tApe2-like_AP-endo\ncd03487\tRT_Bac_retron_II\npfam02994\tTransposase_22\npfam14111\tDUF4283\ncd09080\tTDP2\npfam17490\tTnp_22_dsRBD\ncd09084\tEEP-2\npfam14392\tzf-CCHC_4\nTIGR00633\tDNA-apurinic_or_apyrimidinic_site_lyase\nPRK11756\tPRK11756\nPRK13911\tPRK13911\ncd09280\tRNase_HI_eukaryote_like\ncd09276\tRnase_HI_RT_non_LTR\npfam00075\tRNase_H\npfam05699\tDimer_Tnp_hAT\nsmart00614\tZnF_BED\npfam14291\tDUF4371\npfam14372\tDUF4413\npfam04937\tDUF659\npfam02892\tzf-BED\nsmart00597\tZnF_TTF\npfam18658\tzf-C2H2_12\nNF033545\ttranspos_IS630\npfam13358\tDDE_3\npfam03184\tDDE_1\npfam01498\tHTH_Tnp_Tc3_2\npfam13565\tHTH_32\nCOG3415\tCOG3415\npfam01359\tTransposase_1\npfam03221\tHTH_Tnp_Tc5\nsmart00674\tCENPB\npfam17906\tHTH_48\npfam13518\tHTH_28\npfam13384\tHTH_23\nCOG3335\tCOG3335\npfam04218\tCENP-B_N\npfam13936\tHTH_38\npfam05225\tHTH_psq\npfam16087\tDUF4817\npfam13412\tHTH_24\npfam18701\tDUF5641\npfam05380\tPeptidase_A17\npfam03564\tDUF1759\npfam05585\tDUF1758\nsmart00249\tPHD\ncd15489\tPHD_SF\ncd15522\tPHD_TAF3\npfam00628\tPHD\ncd15517\tPHD_TCF19_like\ncd15560\tPHD2_3_BPTF\ncd15552\tPHD_PHF3_like\ncd15610\tPHD3_KDM5A_like\ncd15553\tPHD_Cfp1\ncd15556\tPHD_MMD1_like\ncd16039\tPHD_SPP1\ncd15529\tPHD2_PHF10\ncd15554\tPHD_PHF2_like\ncd15550\tPHD_MLL5\ncd15518\tPHD_Ecm5p_Lid2p_like\ncd15505\tPHD_ING\ncd15640\tPHD_KDM7\ncd15551\tPHD_PYGO\ncd15525\tPHD_UHRF1_2\ncd15638\tPHD_PHF3\ncd15510\tPHD2_KMT2C_like\ncd15520\tPHD3_Lid2p_like\ncd15613\tPHD_AL_plant\ncd15587\tPHD_Yng1p_like\ncd15585\tPHD_ING3\ncd15627\tPHD_BAZ1A\ncd15687\tPHD3_KDM5B\ncd01646\tRT_Bac_retron_I\ncd00229\tSGNH_hydrolase\npfam13472\tLipase_GDSL_2\ncd01828\tsialate_O-acetylesterase_like2\nCOG3344\tCOG3344\ncd04501\tSGNH_hydrolase_like_4\ncd04502\tSGNH_hydrolase_like_7\ncd01832\tSGNH_hydrolase_like_1\ncd15636\tPHD_PYGO2\nCOG2755\tTesA\ncd01841\tNnaC_like\ncd15635\tPHD_PYGO1\ncd01833\tXynB_like\npfam10551\tMULE\nsmart00575\tZnF_PMZ\npfam04434\tSWIM\npfam03108\tDBD_Tnp_Mut\npfam03101\tFAR1\nPLN03097\tFHY3\npfam09331\tDUF1985\npfam02902\tPeptidase_C48\nNF033543\ttranspos_IS256\npfam10536\tPMD\npfam00872\tTransposase_mut\nCOG3328\tCOG3328\ncd00799\tINT_Cre_C\npfam11560\tLAP2alpha\ncd00397\tDNA_BRE_C\npfam00589\tPhage_integrase\npfam02899\tPhage_int_SAM_1\npfam13495\tPhage_int_SAM_4\npfam05869\tDam\ncd01194\tINT_C_like_4\npfam02992\tTransposase_21\npfam13960\tDUF4218\npfam13963\tTranspos_assoc\npfam13952\tDUF4216\npfam03004\tTransposase_24\npfam03017\tTransposase_23\ncd01648\tTERT\ncd10442\tGIY-YIG_PLEs\ncd10443\tGIY-YIG_HE_Tlr8p_PBC-V_like\ncd00719\tGIY-YIG_SF\ncd09278\tRNase_HI_prokaryote_like\ncd09277\tRNase_HI_bacteria_like\ncd09273\tRNase_HI_RT_Bel\nPRK00203\trnhA\npfam13359\tDDE_Tnp_4\npfam04827\tPlant_tran\nNF033581\ttranspos_IS5_4\npfam14303\tNAM-associated\npfam13837\tMyb_DNA-bind_4\ncd12203\tGT1\npfam01609\tDDE_Tnp_1\nsd00010\tSLR\npfam18697\tMLVIN_C\npfam00077\tRVP\ncd09950\tENVV1-like_HR1-HR2\ncd09948\tEbola_RSV-like_HR1-HR2\npfam02093\tGag_p30\ncd09951\tHERV-Rb-like_HR1-HR2\ncd09851\tHTLV-1-like_HR1-HR2\ncd09850\tEbola-like_HR1-HR2\ncd09947\tEbola_HIV-1-like_HR1-HR2\ncd13934\tRNase_H_Dikarya_like\npfam00429\tTLV_coat\npfam01140\tGag_MA\ncd05482\tHIV_retropepsin_like\ncd09949\tRSV-like_HR1-HR2\nPRK08719\tPRK08719\npfam16721\tzf-H3C2\nCOG2801\tTra5\ncd18037\tDEXSc_Pif1_like\npfam05970\tPIF1\npfam14214\tHelitron_like_N\ncd17933\tDEXSc_RecD-like\ncd18809\tSF1_C_RecD\npfam13604\tAAA_30\ncd18786\tSF1_C\ncd04480\tRPA1_DBD_A_like\npfam13245\tAAA_19\ncd17914\tDExxQc_SF1-N\nCOG0507\tRecD\npfam02721\tDUF223\ncd04476\tRPA1_DBD_C\ncd04481\tRPA1_DBD_B_like\npfam13401\tAAA_22\npfam13538\tUvrD_C_2\npfam04843\tHerpes_teg_N\nsmart00382\tAAA\nCOG3568\tElsH\npfam07530\tPRE_C2HC\nsmart00596\tPRE_C2HC\npfam00607\tGag_p24\npfam06817\tRVT_thumb\npfam02022\tIntegrase_Zn\npfam00552\tIN_DBD_C\npfam00692\tdUTPase\npfam14787\tzf-CCHC_5\ncd07557\ttrimeric_dUTPase\nPRK00601\tdut\npfam02337\tGag_p10\nCOG0756\tDut\nPHA03094\tPHA03094\nPLN02547\tPLN02547\nPHA02703\tPHA02703\nsmart00443\tG_patch\npfam01585\tG-patch\nTIGR00576\tDeoxyuridine_5'-triphosphate_nucleotidohydrolase\nPRK13956\tdut\npfam00517\tGP41\ncd09909\tHIV-1-like_HR1-HR2\npfam04160\tBorrelia_orfX\nPTZ00143\tPTZ00143\nPHA02517\tPHA02517\npfam13843\tDDE_Tnp_1_7\ncd22343\tPDDEXK_lambda_exonuclease-like\nTIGR00195\tExodeoxyribonuclease_III\npfam01107\tMP\npfam02160\tPeptidase_A3\ncd17920\tDEXHc_RecQ\nsmart00490\tHELICc\ncd18018\tDEXHc_RecQ4-like\npfam00271\tHelicase_C\ncd18794\tSF2_C_RecQ\ncd18017\tDEXHc_RecQ3\npfam00270\tDEAD\ncd18016\tDEXHc_RecQ2_BLM\ncd18015\tDEXHc_RecQ1\nsmart00487\tDEXDc\nCOG0514\tRecQ\ncd18014\tDEXHc_RecQ5\ncd00046\tSF2-N\ncd17921\tDEXHc_Ski2\nPRK11057\tPRK11057\ncd18785\tSF2_C\ncd18787\tSF2_C_DEAD\ncd17923\tDEXHc_Hrq1-like\ncd18797\tSF2_C_Hrq\ncd18793\tSF2_C_SNF\ncd18796\tSF2_C_LHR\ncd17927\tDEXHc_RIG-I\nCOG0513\tSrmB\npfam04851\tResIII\ncd18028\tDEXHc_archSki2\ncd18802\tSF2_C_dicer\ncd17926\tDEXHc_RE\ncd18023\tDEXHc_HFM1\ncd18795\tSF2_C_Ski2\ncd01820\tPAF_acetylesterase_like\npfam03175\tDNA_pol_B_2\nPHA02563\tPHA02563\npfam04665\tPox_A32\nNF033516\ttranspos_IS3\npfam08398\tParvo_coat_N\ncd18646\tCD_Cbx7\ncd18965\tchromodomain\ncd18972\tCD_POL_like\npfam02229\tPC4\ncd18962\tCD_MT_like\ncd18645\tCD_Cbx4\ncd18649\tCD_Cbx8\npfam00352\tTBP\ncd00022\tBIR\ncd04518\tTBP_archaea\npfam00653\tBIR\npfam13894\tzf-C2H2_4\ncd00167\tSANT\npfam05485\tTHAP\nsmart00980\tTHAP\npfam12017\tTnp_P_element\nsmart00692\tDM3\npfam12596\tTnp_P_element_C\nPRK13907\trnhA\nPRK07708\tPRK07708\npfam09004\tDUF1891\npfam09588\tYqaJ\npfam13613\tHTH_Tnp_4\nTIGR03033\thypothetical_protein\ncd15586\tPHD_ING4_5\ncd15682\tPHD_ING1\ncd15683\tPHD_ING2\ncd15685\tPHD_ING5\npfam16012\tDUF4780\nTIGR04416\thypothetical_protein\npfam02338\tOTU\npfam16124\tRecQ_Zn_bind\ncd00590\tRRM_SF\ncd12233\tRRM_Srp1p_AtRSp31_like\ncd12236\tRRM_snRNP70\ncd12237\tRRM_snRNP35\ncd12312\tRRM_SRSF10_SRSF12\ncd12320\tRRM6_RBM19_RRM5_MRD1\ncd12338\tRRM1_SRSF1_like\ncd12341\tRRM_hnRNPC_like\ncd12343\tRRM1_2_CoAA_like\ncd12363\tRRM_TRA2\ncd12373\tRRM_SRSF3_like\ncd12382\tRRM_RBMX_like\ncd12390\tRRM3_RAVER\ncd12392\tRRM2_SART3\ncd12399\tRRM_HP0827_like\ncd12404\tRRM2_NCL\ncd12412\tRRM_DAZL_BOULE\ncd12413\tRRM1_RBM28_like\ncd12414\tRRM2_RBM28_like\ncd12448\tRRM2_gar2\ncd12449\tRRM_CIRBP_RBM3\ncd12450\tRRM1_NUCLs\ncd12599\tRRM1_SF2_plant_like\ncd12608\tRRM1_CoAA\ncd12641\tRRM_TRA2B\ncd12642\tRRM_TRA2A\ncd12674\tRRM1_Nop4p\ncd21607\tRRM3_HRB1_GBP2\ncd21608\tRRM2_NsCP33_like\ncd21615\tRRM_SNP1_like\nCOG1877\tOtsB\npfam00076\tRRM_1\nsmart00360\tRRM\ncd15492\tPHD_BRPF_JADE_like\npfam02257\tRFX_DNA_binding\nPRK06548\tPRK06548\ncd06141\tWRN_exo\ncd06146\tmut-7_like_exo\ncd06129\tRNaseD_like\ncd09018\tDEDDy_polA_RNaseD_like_exo\npfam01612\tDNA_pol_A_exo1\nsmart00474\t35EXOc\npfam06975\tDUF1299\ncd15637\tPHD_dPYGO\npfam18717\tCxC4\npfam18758\tKDZ\nCOG3147\tDedD\npfam12550\tGCR1_C\npfam16787\tNDC10_II\nNF033547\ttranspos_IS1595\npfam12762\tDDE_Tnp_IS1595\ncd21575\tKLF18_N\nsmart00238\tBIR\nsd00033\tLRR_RI\npfam00400\tWD40\nsd00039\t7WD40\nsmart00320\tWD40\ncd00200\tWD40\nCOG2319\tCOG2319\nsd00031\tLRR_1\npfam00931\tNB-ARC\npfam13855\tLRR_8\n"
},
{
"path": "scripts/TEstrainer/data/unacceptable_domains.tsv",
"content": "ref\tabbrev\torganism\ncd20663\tCYP2D\tCeratotherium_simum_cottoni\ncd11026\tCYP2\tCeratotherium_simum_cottoni\npfam00067\tp450\tCeratotherium_simum_cottoni\ncd11027\tCYP17A1-like\tCeratotherium_simum_cottoni\npfam00001\t7tm_1\tCeratotherium_simum_cottoni\ncd15337\t7tmA_Opsin_Gq_invertebrates\tCeratotherium_simum_cottoni\ncd15001\t7tmA_GPRnna14-like\tCeratotherium_simum_cottoni\ncd15160\t7tmA_Proton-sensing_R\tCeratotherium_simum_cottoni\ncd15054\t7tmA_5-HT6\tCeratotherium_simum_cottoni\ncd15917\t7tmA_OR51_52-like\tCeratotherium_simum_cottoni\ncd15953\t7tmA_OR52P-like\tCeratotherium_simum_cottoni\ncd15221\t7tmA_OR52B-like\tCeratotherium_simum_cottoni\ncd15952\t7tmA_OR52E-like\tCeratotherium_simum_cottoni\ncd15951\t7tmA_OR52R_52L-like\tCeratotherium_simum_cottoni\npfam13853\t7tm_4\tCeratotherium_simum_cottoni\ncd15948\t7tmA_OR52K-like\tCeratotherium_simum_cottoni\ncd15954\t7tmA_OR52N-like\tCeratotherium_simum_cottoni\ncd15955\t7tmA_OR52A-like\tCeratotherium_simum_cottoni\ncd15950\t7tmA_OR52I-like\tCeratotherium_simum_cottoni\ncd14972\t7tmA_EDG-like\tCeratotherium_simum_cottoni\ncd14971\t7tmA_Galanin_R-like\tCeratotherium_simum_cottoni\ncd14978\t7tmA_FMRFamide_R-like\tCeratotherium_simum_cottoni\ncd15307\t7tmA_5-HT2_insect-like\tCeratotherium_simum_cottoni\ncd15411\t7tmA_OR8H-like\tCeratotherium_simum_cottoni\ncd15413\t7tmA_OR8K-like\tCeratotherium_simum_cottoni\ncd15429\t7tmA_OR2F-like\tCeratotherium_simum_cottoni\ncd15947\t7tmA_OR2B-like\tCeratotherium_simum_cottoni\ncd15416\t7tmA_OR5P-like\tCeratotherium_simum_cottoni\ncd15407\t7tmA_OR5B-like\tCeratotherium_simum_cottoni\ncd15915\t7tmA_OR12D-like\tCeratotherium_simum_cottoni\ncd15415\t7tmA_OR5J-like\tCeratotherium_simum_cottoni\ncd15405\t7tmA_OR8B-like\tCeratotherium_simum_cottoni\ncd15942\t7tmA_OR10G6-like\tCeratotherium_simum_cottoni\ncd15222\t7tmA_OR51-like\tCeratotherium_simum_cottoni\ncd15956\t7tmA_OR52W-like\tCeratotherium_simum_cottoni\ncd00637\t7tm_classA_rhodopsin-like\tCeratotherium_simum_cottoni\ncd15067\t7tmA_Dop1R2-like\tCeratotherium_simum_cottoni\ncd15208\t7tmA_OXR\tCeratotherium_simum_cottoni\ncd15366\t7tmA_GPR4\tCeratotherium_simum_cottoni\ncd15234\t7tmA_OR7-like\tCeratotherium_simum_cottoni\ncd13954\t7tmA_OR\tCeratotherium_simum_cottoni\ncd15225\t7tmA_OR10A-like\tCeratotherium_simum_cottoni\ncd15230\t7tmA_OR5-like\tCeratotherium_simum_cottoni\ncd15236\t7tmA_OR1E-like\tCeratotherium_simum_cottoni\ncd15911\t7tmA_OR11A-like\tCeratotherium_simum_cottoni\ncd15231\t7tmA_OR5V1-like\tCeratotherium_simum_cottoni\ncd15229\t7tmA_OR8S1-like\tCeratotherium_simum_cottoni\ncd15227\t7tmA_OR14-like\tCeratotherium_simum_cottoni\ncd15431\t7tmA_OR13H-like\tCeratotherium_simum_cottoni\ncd15912\t7tmA_OR6C-like\tCeratotherium_simum_cottoni\ncd15417\t7tmA_OR5A1-like\tCeratotherium_simum_cottoni\ncd15421\t7tmA_OR2T-like\tCeratotherium_simum_cottoni\ncd15232\t7tmA_OR13-like\tCeratotherium_simum_cottoni\ncd15420\t7tmA_OR2A-like\tCeratotherium_simum_cottoni\ncd15414\t7tmA_OR5G-like\tCeratotherium_simum_cottoni\ncd15419\t7tmA_OR9K2-like\tCeratotherium_simum_cottoni\ncd15409\t7tmA_OR5H-like\tCeratotherium_simum_cottoni\ncd15224\t7tmA_OR6B-like\tCeratotherium_simum_cottoni\ncd15233\t7tmA_OR3A-like\tCeratotherium_simum_cottoni\ncd15424\t7tmA_OR2_unk\tCeratotherium_simum_cottoni\ncd15913\t7tmA_OR11G-like\tCeratotherium_simum_cottoni\ncd15946\t7tmA_OR1330-like\tCeratotherium_simum_cottoni\ncd15228\t7tmA_OR10D-like\tCeratotherium_simum_cottoni\ncd15914\t7tmA_OR6N-like\tCeratotherium_simum_cottoni\ncd15916\t7tmA_OR10G-like\tCeratotherium_simum_cottoni\ncd15428\t7tmA_OR2D-like\tCeratotherium_simum_cottoni\ncd15226\t7tmA_OR4-like\tCeratotherium_simum_cottoni\ncd15941\t7tmA_OR10S1-like\tCeratotherium_simum_cottoni\ncd15940\t7tmA_OR4E-like\tCeratotherium_simum_cottoni\ncd15936\t7tmA_OR4D-like\tCeratotherium_simum_cottoni\ncd15223\t7tmA_OR56-like\tCeratotherium_simum_cottoni\ncd15938\t7tmA_OR4Q2-like\tCeratotherium_simum_cottoni\ncd15935\t7tmA_OR4Q3-like\tCeratotherium_simum_cottoni\ncd15937\t7tmA_OR4N-like\tCeratotherium_simum_cottoni\ncd14967\t7tmA_amine_R-like\tCeratotherium_simum_cottoni\ncd14993\t7tmA_CCKR-like\tCeratotherium_simum_cottoni\ncd15203\t7tmA_NPYR-like\tCeratotherium_simum_cottoni\ncd14969\t7tmA_Opsins_type2_animals\tCeratotherium_simum_cottoni\ncd15314\t7tmA_TAAR1\tCeratotherium_simum_cottoni\ncd15055\t7tmA_TAARs\tCeratotherium_simum_cottoni\ncd15062\t7tmA_alpha1_AR\tCeratotherium_simum_cottoni\ncd15202\t7tmA_TACR-like\tCeratotherium_simum_cottoni\ncd15079\t7tmA_photoreceptors_insect\tCeratotherium_simum_cottoni\ncd15325\t7tmA_alpha1A_AR\tCeratotherium_simum_cottoni\ncd15012\t7tmA_Trissin_R\tCeratotherium_simum_cottoni\ncd15061\t7tmA_tyramine_R-like\tCeratotherium_simum_cottoni\ncd15213\t7tmA_PSP24-like\tCeratotherium_simum_cottoni\ncd14982\t7tmA_purinoceptor-like\tCeratotherium_simum_cottoni\ncd15918\t7tmA_OR1_7-like\tCeratotherium_simum_cottoni\ncd15237\t7tmA_OR2-like\tCeratotherium_simum_cottoni\ncd15412\t7tmA_OR5M-like\tCeratotherium_simum_cottoni\ncd15434\t7tmA_OR2W-like\tCeratotherium_simum_cottoni\ncd15430\t7tmA_OR13-like\tCeratotherium_simum_cottoni\ncd15432\t7tmA_OR2B2-like\tCeratotherium_simum_cottoni\ncd15433\t7tmA_OR2Y-like\tCeratotherium_simum_cottoni\ncd15939\t7tmA_OR4A-like\tCeratotherium_simum_cottoni\ncd15083\t7tmA_Melanopsin-like\tCeratotherium_simum_cottoni\ncd15235\t7tmA_OR1A-like\tCeratotherium_simum_cottoni\ncd15418\t7tmA_OR9G-like\tCeratotherium_simum_cottoni\ncd15406\t7tmA_OR8D-like\tCeratotherium_simum_cottoni\ncd15410\t7tmA_OR5D-like\tCeratotherium_simum_cottoni\ncd15408\t7tmA_OR5AK3-like\tCeratotherium_simum_cottoni\ncd15945\t7tmA_OR5C1-like\tCeratotherium_simum_cottoni\ncd15944\t7tmA_OR5AR1-like\tCeratotherium_simum_cottoni\ncd15943\t7tmA_OR5AP2-like\tCeratotherium_simum_cottoni\ncd15949\t7tmA_OR52M-like\tCeratotherium_simum_cottoni\ncd15064\t7tmA_5-HT1_5_7\tCeratotherium_simum_cottoni\ncd15134\t7tmA_capaR\tCeratotherium_simum_cottoni\ncd15196\t7tmA_Vasopressin_Oxytocin\tCeratotherium_simum_cottoni\npfam00129\tMHC_I\tEquus_caballus\ncd16001\t7tmA_P2Y3-like\tChiloscyllium_punctatum\ncd15393\t7tmA_leucokinin-like\tChiloscyllium_punctatum\ncd15392\t7tmA_PR4-like\tChiloscyllium_punctatum\ncd15168\t7tmA_P2Y1-like\tChiloscyllium_punctatum\ncd14979\t7tmA_NTSR-like\tChiloscyllium_punctatum\ncd15928\t7tmA_GHSR-like\tChiloscyllium_punctatum\ncd15096\t7tmA_AstA_R_insect\tChiloscyllium_punctatum\ncd15206\t7tmA_CCK_R\tChiloscyllium_punctatum\ncd15919\t7tmA_GPR139\tChiloscyllium_punctatum\n"
},
{
"path": "scripts/TEstrainer/scripts/Dfam_extractor.py",
"content": "#!/usr/bin/env python\n\nimport os\nfrom re import sub\nfrom os.path import exists\nimport sys\nimport argparse\nfrom Bio import SeqIO\n\nparser = argparse.ArgumentParser()\nparser.add_argument('-l', '--library', type=str, required=True,\n help='Input library fasta')\nparser.add_argument('-d', '--directory', type=str, required=True,\n help='Path to working data directory') \nargs = parser.parse_args()\n\n# check file/folder exists\nif(exists(args.library) == False):\n sys.exit('Input fasta not found')\nif(exists(args.directory) == False):\n os.mkdir(args.out_dir)\n\n# name of out file\nout_library=args.directory+'/'+sub('.*/', '', args.library)\nnondfam_library=args.directory+'/non-dfam.'+sub('.*/', '', args.library)\n\n# write Dfam pseudo-curated sequences to out_library\n# sequence headers beginning with DR and having digits between \"DR\" and \"#\" treated as as Dfam\nwith open(out_library, 'w') as out_handle:\n with open(args.library, 'r') as in_handle:\n for record in SeqIO.parse(in_handle, \"fasta\"):\n if record.name.startswith(\"DR\"):\n if sub('DR', '', sub('#.*', '', record.name)).isdigit():\n SeqIO.write(record, out_handle, \"fasta\")\n\n# write non-Dfam sequences to out_library\n# sequence headers beginning with DR and having digits between \"DR\" and \"#\" treated as as Dfam\nwith open(nondfam_library, 'w') as out_handle:\n with open(args.library, 'r') as in_handle:\n for record in SeqIO.parse(in_handle, \"fasta\"):\n if record.name.startswith(\"DR\") is False:\n SeqIO.write(record, out_handle, \"fasta\")\n elif sub('DR', '', sub('#.*', '', record.name)).isdigit() is False:\n SeqIO.write(record, out_handle, \"fasta\")\n"
},
{
"path": "scripts/TEstrainer/scripts/TEtrim.py",
"content": "import os\nimport sys\nimport string\nimport statistics\nimport argparse\nimport re\nimport Bio\nfrom Bio import SeqIO, AlignIO\nfrom Bio.Align import AlignInfo, MultipleSeqAlignment\nfrom Bio.Seq import Seq\nfrom Bio.SeqRecord import SeqRecord\nimport pandas as pd\nparser = argparse.ArgumentParser()\nparser.add_argument('-i', '--in_seq', type=str, required=True,\n help='Input MSA to be trimmed')\nparser.add_argument('-n', '--iteration', type=str, required=True,\n help='iteration number')\nparser.add_argument('-d', '--directory', type=str, required=True,\n help='Output directory') \nparser.add_argument('-t', '--threads', type=str,\n help='Threads to use', default=1)\nparser.add_argument('-f', '--flank', type=int,\n help='Length of flanks used', default=1000)\nparser.add_argument('-D', '--debug', type=str,\n help='Print debug messages', default='FALSE')\nparser.add_argument('-w', '--window', type=int,\n help='Window size for blocks', default=5)\nparser.add_argument('-m', '--minimum_seq', type=int,\n help='Minimum sequences required for MSA', default=3)\n\nargs = parser.parse_args()\n\n# function for removing single base pair insertions\ndef single_trim(aln_in):\n # make empty alignment\n good=aln_in[:,0:0]\n for x in range(aln_in.get_alignment_length()):\n # extract columns with more than 1 base pair\n if len(aln_in) - aln_in[:, x].count(\"-\") > 1:\n good=good+aln_in[:,x:x+1]\n return(good)\n# function for making first consensus from alignment\ndef con_maker(aln_in):\n con_seq=str()\n for x in range(aln_in.get_alignment_length()):\n a = aln_in[:, x].count(\"a\") + aln_in[:, x].count(\"A\")\n t = aln_in[:, x].count(\"t\") + aln_in[:, x].count(\"T\")\n c = aln_in[:, x].count(\"c\") + aln_in[:, x].count(\"C\")\n g = aln_in[:, x].count(\"g\") + aln_in[:, x].count(\"G\")\n gap = aln_in[:, x].count(\"-\")\n if (len(aln_in) - gap) > 2:\n pos_count = {'A': a, 'T': t, 'C': c, 'G': g}\n if len([k for k, v in pos_count.items() if v == max(pos_count.values())])>1:\n con_seq+='n'\n else:\n con_seq+=max(pos_count, key=pos_count.get)\n con_seq=Seq(con_seq)\n return(con_seq)\n# set names\nseq_name=re.sub('.*/', '', args.in_seq)\nin_seq_path=args.directory+'/run_'+args.iteration+'/mafft/'+seq_name\nout_seq_path=args.directory+'/run_'+args.iteration+'/TEtrim/'+seq_name\n# read sequence\nif args.debug == 'TRUE':\n print('Reading og alignment')\nalign = AlignIO.read(in_seq_path, \"fasta\")\n# get name of first sequence (the consensus)\nfinal_id = align[0].id\nif args.debug == 'TRUE':\n print(final_id)\nif(float(Bio.__version__) >=1.79):\n og_con = SeqRecord(seq= align[0].seq.replace(\"-\", \"\"), id=align[0].id, name=align[0].id)\nelse:\n og_con = SeqRecord(seq= align[0].seq.ungap(\"-\"), id=align[0].id, name=align[0].id)\nSeqIO.write(og_con, (args.directory+'/run_'+args.iteration+'/TEtrim_con/og_'+seq_name),\"fasta\")\nalign = align[1:len(align)]\n\n# cancel if less than minimum_seq sequences, write to file for troubleshooting\nif args.debug == 'TRUE':\n print('Reading determining number of sequences '+str(len(align)))\nif(len(align)=1.79):\n record.seq = record.seq.replace(\"-\", \"\")\n else:\n record.seq = record.seq.ungap(\"-\")\n SeqIO.write(record, o, \"fasta-2line\")\n\n### run blast ###\nif args.debug == 'TRUE':\n print('Determining number of sequences')\nos.system('blastn -query '+args.directory+'/run_'+args.iteration+'/TEtrim_con/og_'+seq_name+' -subject '+args.directory+'/run_'+args.iteration+'/TEtrim_unaln/temp_'+seq_name+' -outfmt \"6 qseqid sseqid qcovs\" -task blastn | uniq > '+args.directory+'/run_'+args.iteration+'/TEtrim_blast/'+seq_name+'.tsv')\n# read initial blast, determine acceptable passed on coverage >50% mean of coverage\nif args.debug == 'TRUE':\n print('Reading initial blast')\ndf = pd.read_table((args.directory+'/run_'+args.iteration+'/TEtrim_blast/'+seq_name+'.tsv'), names=['qseqid', 'sseqid', 'qcovs'])\nif df.empty is True:\n SeqIO.write(og_con, (args.directory+'/run_'+args.iteration+'/TEtrim_complete/'+seq_name),\"fasta\")\n if args.debug == 'TRUE': \n sys.exit(('New consensus of '+seq_name+\" has no homology to original sequence\"))\n else:\n sys.exit()\nmean_covs=(statistics.mean(df.qcovs)/2)\nacceptable=list(df.query(\"(qcovs>@mean_covs) or (qcovs>50)\")['sseqid'])\n\n# Check at least minimum_seq\nif len(acceptable) < args.minimum_seq:\n SeqIO.write(og_con, (args.directory+'/run_'+args.iteration+'/TEtrim_complete/'+seq_name),\"fasta\")\n if args.debug == 'TRUE':\n sys.exit((seq_name+\" contains less than \"+args.minimum_seq+\" acceptable sequences\"))\n else:\n sys.exit()\n\n# create unaligned fasta of acceptables\nif args.debug == 'TRUE':\n print('Creating list of unaligned acceptable sequences')\nwith open((args.directory+'/run_'+args.iteration+'/TEtrim_unaln/unaln_'+seq_name), \"w\") as o:\n for record in SeqIO.parse((args.directory+'/run_'+args.iteration+'/TEtrim_unaln/temp_'+seq_name), \"fasta-2line\"):\n if record.id in acceptable:\n SeqIO.write(record, o, \"fasta-2line\")\n### run mafft ###\nif args.debug == 'TRUE':\n print('Running initial mafft')\nos.system('mafft --quiet --thread '+args.threads+' --localpair '+args.directory+'/run_'+args.iteration+'/TEtrim_unaln/unaln_'+seq_name+' > '+args.directory+'/run_'+args.iteration+'/TEtrim_mafft/mafft_'+seq_name)\n# read new alignment, make consensus\nif args.debug == 'TRUE':\n print('Reading new alignment and making consensus')\nalign_2=AlignIO.read((args.directory+'/run_'+args.iteration+'/TEtrim_mafft/mafft_'+seq_name), \"fasta\")\ncon_2=SeqRecord(con_maker(align_2),id=final_id,description=final_id)\nSeqIO.write(con_2, (args.directory+'/run_'+args.iteration+'/TEtrim_con/cleaned_'+seq_name),\"fasta\")\n# Run final blast\nif args.debug == 'TRUE':\n print('Final blast')\nos.system('blastn -query '+args.directory+'/run_'+args.iteration+'/TEtrim_con/og_'+seq_name+' -subject '+args.directory+'/run_'+args.iteration+'/TEtrim_con/cleaned_'+seq_name+' -outfmt \"6 length qlen slen qcovs\" -task dc-megablast -out '+args.directory+'/run_'+args.iteration+'/TEtrim_blast/check_'+seq_name+'.tsv')\nif os.path.getsize(args.directory+'/run_'+args.iteration+'/TEtrim_blast/check_'+seq_name+'.tsv') == 0:\n SeqIO.write(og_con, (args.directory+'/run_'+args.iteration+'/TEtrim_complete/'+seq_name),\"fasta\")\n if args.debug == 'TRUE':\n sys.exit(('New consensus of '+seq_name+\" has no homology to original sequence\"))\n else:\n sys.exit()\n# read in first line of blast\ndf2 = pd.read_table((args.directory+'/run_'+args.iteration+'/TEtrim_blast/check_'+seq_name+'.tsv'), names=['length', 'qlen', 'slen', 'qcovs']).head(1)\n# determine values from blast table\nlength=df2.values[0,0]\nqlen=df2.values[0,1]\nslen=df2.values[0,2]\nqcovs=df2.values[0,3]\nif qcovs <= 80:\n SeqIO.write(og_con, (args.directory+'/run_'+args.iteration+'/TEtrim_complete/'+seq_name),\"fasta\")\n if args.debug == 'TRUE':\n sys.exit(('New consensus of '+seq_name+\" covers < 80% than original sequence\"))\nelif (slen <= (0.9 * qlen)):\n SeqIO.write(og_con, (args.directory+'/run_'+args.iteration+'/TEtrim_complete/'+seq_name),\"fasta\")\n if args.debug == 'TRUE':\n sys.exit(('New consensus of '+seq_name+\" is shorter than original sequence\"))\nelif ((slen - qlen) <= (0.5 * args.flank)):\n SeqIO.write(con_2, (args.directory+'/run_'+args.iteration+'/TEtrim_complete/'+seq_name),\"fasta\")\n if args.debug == 'TRUE':\n sys.exit(('New consensus of '+seq_name+\" is good, no further work needed\"))\nelse:\n SeqIO.write(con_2, (args.directory+'/run_'+args.iteration+'/TEtrim_further/'+seq_name),\"fasta\")\n if args.debug == 'TRUE':\n sys.exit(('New consensus of '+seq_name+\" is good, needs further extension.\"))\n"
},
{
"path": "scripts/TEstrainer/scripts/domain_plotter.R",
"content": "library(tidyverse)\n\nin_seq <- \"seq/Chiloscyllium_punctatum-families.fa\"\ndata_dir <- \"TS_Chiloscyllium_punctatum-families.fa_5438/\"\nseq_name <- sub(\".*\\\\/\", \"\", in_seq)\n\nog_seq<-Biostrings::readDNAStringSet(in_seq)\nog_tibble <- tibble(seqnames = names(og_seq), width = width(og_seq)) %>%\n mutate(raw_names = sub(\"#.*\", \"\", seqnames))\n\nchimeric_seq<-Biostrings::readDNAStringSet(paste0(data_dir, \"/chimeras/chimeric_\", seq_name))\nunacceptable_seq <-Biostrings::readDNAStringSet(paste0(data_dir, \"/chimeras/questionable_\", seq_name))\nchimeric_seq <- c(chimeric_seq, unacceptable_seq)\nchimeric_tibble <- tibble(seqnames = names(chimeric_seq), width = width(chimeric_seq)) %>%\n mutate(raw_names = sub(\"#.*\", \"\", seqnames))\n\nrps_blast_out <- read_tsv(file = paste0(data_dir, \"chimeras/\", seq_name, \".rps.out\"),\n col_names = c(\"seqnames\", \"qstart\", \"qend\", \"qlen\", \"sseqid\", \"sstart\", \"send\", \"slen\",\n \"pident\", \"length\", \"mismatch\", \"gapopen\", \"evalue\", \"bitscore\", \"qcovs\", \"stitle\"),\n show_col_types = FALSE) %>%\n tidyr::separate(stitle, into = c(\"ref\", \"abbrev\", \"full\"), sep = \", \", extra = \"drop\") %>%\n dplyr::filter(length >= 0.5*slen)\n\nacceptable_domains <- rbind(read_tsv(\"data/acceptable_domains.tsv\"), read_tsv(\"data/additional_domains.tsv\", col_select = c(ref, abbrev)))\nunacceptable_domains <- rbind(read_tsv(\"data/unacceptable_domains.tsv\", col_select = c(ref, abbrev)))\n\nchimeric_tibble <- rps_blast_out[sub(\"#.*\", \"\", rps_blast_out$seqnames) %in% c(chimeric_tibble$raw_names),]\n\nunacceptable <- chimeric_tibble[!chimeric_tibble$ref %in% acceptable_domains$ref,]$seqnames\n\nchimeric_tibble <- rps_blast_out[sub(\"#.*\", \"\", rps_blast_out$seqnames) %in% sub(\"#.*\", \"\", unacceptable),]\n\nchimeric_tibble <- chimeric_tibble %>%\n mutate(missing = ifelse(ref %in% acceptable_domains$ref, FALSE, TRUE))\nchimeric_names <- base::unique(chimeric_tibble$seqnames)\nchimeric_names\n\ni=4\n\nto_plot <- chimeric_tibble[chimeric_tibble$seqnames == chimeric_names[i],] %>% arrange(qstart)\nto_plot$y = 1:nrow(to_plot)\nto_plot$text_x = (to_plot$qstart + to_plot$qend)/2\nggplot(to_plot) +\n geom_rect(aes(xmin = qstart, xmax = qend, ymin = y-1, ymax = y, fill = missing)) +\n geom_text(aes(x = text_x, y = y-0.5, label = abbrev)) +\n scale_x_continuous(limits = c(0,to_plot$qlen[1])) +\n ggtitle(to_plot$seqnames[1])\nto_plot %>%\n filter(!ref %in% unacceptable_domains$ref, !ref %in% acceptable_domains$ref) %>%\n dplyr::select(ref, abbrev) %>%\n base::unique() %>%\n mutate(organism = \"Chiloscyllium_punctatum\") %>%\n write_tsv(\"new_unacceptable_domains.tsv\")\n"
},
{
"path": "scripts/TEstrainer/scripts/final_sorter.R",
"content": "#!/usr/bin/Rscript\n\nlibrary(optparse)\n\noption_list <- list(\n make_option(c(\"-i\", \"--in_seq\"), default=NA, type = \"character\", help=\"Input sequence (required)\")\n)\nopt <- parse_args(OptionParser(option_list=option_list))\nif(is.na(opt$in_seq)){\n stop(\"Path to in sequence must be supplied\")\n}\n\n# make empty variable function\nsuppressPackageStartupMessages(library(tidyverse))\nsuppressPackageStartupMessages(library(plyranges))\nsuppressPackageStartupMessages(library(BSgenome))\n\n# read in sequence\nin_seq <- readDNAStringSet(opt$in_seq)\n\n# sort by package used, round number and family number\nsorted_ranges <-\n tibble(seqnames = names(in_seq), start = 1, end = width(in_seq)) %>%\n dplyr::mutate(pkg = substr(seqnames, 1, 3),\n rnd = as.integer(sub(\".*-\", \"\", sub(\"_family.*\", \"\", seqnames))),\n family = as.integer(sub(\"#.*\", \"\", sub(\".*_family-\", \"\", seqnames)))) %>%\n dplyr::arrange(pkg, rnd, family) %>%\n plyranges::as_granges()\n\n# Get sequence\nout_seq = getSeq(in_seq, sorted_ranges)\nnames(out_seq) <- seqnames(sorted_ranges)\n\n# Write to file\nwriteXStringSet(out_seq, opt$in_seq)"
},
{
"path": "scripts/TEstrainer/scripts/indexer.py",
"content": "#!/usr/bin/env python\n\nimport argparse\nfrom pyfaidx import Faidx\n\nparser = argparse.ArgumentParser()\nparser.add_argument('-g', '--genome', type=str, required=True,\n help='Path to genome sequence')\nargs = parser.parse_args()\n\nFaidx(args.genome)\n"
},
{
"path": "scripts/TEstrainer/scripts/initial_domain_finder.R",
"content": "#!/usr/bin/Rscript\n\nlibrary(ORFik)\nlibrary(tidyverse)\nlibrary(plyranges)\nlibrary(BSgenome)\n\n# read in sequences\nrepbase_dna <- Biostrings::readDNAStringSet(\"~/Databases/Repbase/RepBase5May2021.fasta\")\ndfam_dna <- Biostrings::readDNAStringSet(\"seq/Dfam/Dfam_3.6_curatedonly.fasta\")\n\n# get info about sequences\nrepbase_seq_info <- tibble(seqnames = names(repbase_dna), width = width(repbase_dna))\nrepbase_seq_info <- repbase_seq_info %>%\n tidyr::separate(col = seqnames, into = c(\"seqnames\", \"class\", \"species\"), sep = \"\\t\") %>%\n dplyr::select(-species) %>%\n filter(!class %in% c(\"Unknown\", \"Satellite\",\"Simple Repeat\"))\n\n# convert dfam classifications to Repbase\ndfam_seq_info <- tibble(seqnames = names(dfam_dna), width = width(dfam_dna)) %>%\n tidyr::separate(col = seqnames, into = c(\"seqnames\", \"class\"), sep = \"#\") %>%\n mutate(class = ifelse(grepl(\"ERV\", class), \"ERV\", class)) %>%\n dplyr::mutate(class = case_when(\n grepl(\"SINE\", class) ~ \"SINE\",\n grepl(\"/\", class) ~ sub(\".*/\", \"\", class),\n TRUE ~ class)\n ) %>%\n mutate(\n class = ifelse(class %in% c(\"TcMar\", \"TcMar-ISRm11\", \"TcMar-m44\", \"TcMar-Pogo\", \"TcMar-Fot1\", \"TcMar-Tc2\", \"TcMar-Mariner\", \"TcMar-Tc1\", \"TcMar-Tc4\", \"TcMar-Tigger\"), \"Mariner/Tc1\", class),\n class = ifelse(class %in% c(\"hAT-hAT5\", \"hAT-hAT19\", \"hAT-hATx\", \"hAT-hATm\", \"hAT-Tag1\", \"hAT-Ac\", \"hAT-Blackjack\", \"hAT-Tip100\", \"hAT-Charlie\"), \"hAT\", class),\n class = ifelse(class %in% c(\"Pao\"), \"BEL\", class),\n class = ifelse(class %in% c(\"CMC-Transib\"), \"Transib\", class),\n class = ifelse(class %in% c(\"PiggyBac\"), \"piggyBac\", class),\n class = ifelse(class %in% c(\"Naiad\", \"Chlamys\"), \"Penelope\", class),\n class = ifelse(class %in% c(\"CMC-Chapaev\", \"CMC-Chapaev-3\", \"CMC-EnSpm\"), \"EnSpm/CACTA\", class),\n class = ifelse(class %in% c(\"Academ-1\"), \"Academ\", class),\n class = ifelse(class %in% c(\"Ngaro\"), \"DIRS\", class),\n class = ifelse(class %in% c(\"PIF-ISL2EU\"), \"ISL2EU\", class),\n class = ifelse(class %in% c(\"R2-Hero\"), \"Hero\", class),\n class = ifelse(class %in% c(\"CR1-Zenon\"), \"CR1\", class),\n class = ifelse(class %in% c(\"endogenous retrovirus\", \"Endogenous Retrovirus\", \"ERV-1_PM-I\", \"ERV-2_PM-I\"), \"ERV\", class),\n class = ifelse(class %in% c(\"LTR retrotransposon\", \"LTR Retrotransposon\"), \"LTR\", class),\n class = ifelse(class %in% c(\"RTE-RTE\", \"RTE-BovB\"), \"RTE\", class)\n ) %>%\n filter(!seqnames %in% repbase_seq_info$seqnames)\n\ndfam_dna <- dfam_dna[sub(\"#.*\", \"\", names(dfam_dna)) %in% dfam_seq_info$seqnames,]\nrepbase_dna <- c(repbase_dna, dfam_dna)\nrepbase_seq_info_class_n <- as_tibble(as.data.frame(table(repbase_seq_info$class))) %>%\n dplyr::filter(Freq > 5) %>%\n dplyr::mutate(Var1 = as.character(Var1)) %>%\n dplyr::rename(class = Var1)\nrepbase_seq_info <- rbind(repbase_seq_info, dfam_seq_info) %>%\n filter(!class %in% c(\"Unknown\", \"Satellite\", \"subtelomeric\", \"Y-chromosome\", \"centromeric\", \"tRNA\", \"snRNA\", \"rRNA\", \"tRNA\", \"Multicopy gene\"),\n class %in% repbase_seq_info_class_n$class)\nrepbase_seq_info$name = 1:nrow(repbase_seq_info)\n\nrepbase_dna <- repbase_dna[sub(\"#.*\", \"\", sub(\"\\t.*\", \"\", names(repbase_dna))) %in% repbase_seq_info$seqnames,]\nnames(repbase_dna) <- sub(\"#.*\", \"\", sub(\"\\t.*\", \"\", names(repbase_dna)))\nwriteXStringSet(repbase_dna, \"predata/repbase_dfam_compiled.fasta\")\n\n#### RUN RPSTBLASTN AGAINST CDD ####\n\n\n# read in repbase rpstblastn\nrepbase_rps_out <- readr::read_tsv(file = \"predata/repbase_dfam_compiled.rps.out\",\n col_names = c(\"seqnames\", \"qstart\", \"qend\", \"qlen\", \"sseqid\", \"sstart\", \"send\", \"slen\",\n \"pident\", \"length\", \"mismatch\", \"gapopen\", \"evalue\", \"bitscore\", \"qcovs\", \"stitle\"),\n show_col_types = F) %>%\n dplyr::filter(length >= 0.5*slen, evalue <= 0.01)\n\nmissing <- repbase_rps_out %>% filter(!seqnames %in% repbase_seq_info$seqnames)\n\nrepbase_rps_out <- tidyr::separate(data = repbase_rps_out, col = stitle, into = c(\"ref\", \"abbrev\", \"full\"), sep = \", \")\nrepbase_rps_out <- dplyr::inner_join(repbase_rps_out, repbase_seq_info) %>%\n filter(class != \"Multicopy gene\")\ndomain_info <- repbase_rps_out %>% dplyr::select(ref, abbrev, full) %>% base::unique()\n\n# get info about repbase classes\nrepbase_class_info <- dplyr::as_tibble(BiocGenerics::as.data.frame(table(repbase_seq_info$class)))\nrepbase_class_info <- dplyr::mutate(.data = repbase_class_info, Var1 = as.character(Var1))\nrepbase_class_info <- dplyr::rename(.data = repbase_class_info, class = Var1)\nrepbase_class_info_min <- repbase_class_info[repbase_class_info$Freq < 10,]\nrepbase_class_info_max <- repbase_class_info[repbase_class_info$Freq >= 10,]\nrepbase_class_info_max <- dplyr::arrange(.data = repbase_class_info_max, -Freq)\n\n# loop over to get info\ncommon_domains <- tibble()\nfor(i in 1:nrow(repbase_class_info_max)){\n holder <- repbase_rps_out[repbase_rps_out$class == repbase_class_info_max$class[i],] %>%\n group_by(seqnames, abbrev) %>%\n arrange(-bitscore) %>%\n dplyr::slice(1) %>% # ensure one representative for each domain for each repeat\n ungroup()\n if(nrow(holder >= 1)){\n if(repbase_class_info_max$Freq[i] > 299){\n holder_tbl <- as_tibble(as.data.frame(table(holder$ref))) %>%\n mutate(Var1 = as.character(Var1),\n perc = Freq/repbase_class_info_max$Freq[i]) %>%\n dplyr::rename(ref = Var1) %>%\n filter(perc >= 0.01, Freq >= 3) %>%\n inner_join(domain_info) %>%\n mutate(class = repbase_class_info_max$class[i]) %>%\n dplyr::arrange(-Freq)\n } else {\n holder_tbl <- as_tibble(as.data.frame(table(holder$ref))) %>%\n mutate(Var1 = as.character(Var1),\n perc = Freq/repbase_class_info_max$Freq[i]) %>%\n dplyr::rename(ref = Var1) %>%\n filter(Freq >= 3) %>%\n inner_join(domain_info) %>%\n mutate(class = repbase_class_info_max$class[i]) %>%\n dplyr::arrange(-Freq)\n }\n common_domains <- rbind(common_domains, holder_tbl)\n }\n}\n\n# write to file\nbase::unique(common_domains %>% select(ref, abbrev)) %>%\n dplyr::filter(!startsWith(tolower(abbrev), \"ig\"),\n !startsWith(tolower(abbrev), \"znmc\"),\n abbrev != \"V-set\") %>%\n write_tsv(\"data/acceptable_domains.tsv\")\n\nbase::unique(common_domains %>% select(class, ref, abbrev)) %>%\n dplyr::filter(startsWith(tolower(abbrev), \"ig\") | startsWith(tolower(abbrev), \"znmc\") | abbrev == \"V-set\") %>%\n write_tsv(\"data/exceptional_domains.tsv\")\n"
},
{
"path": "scripts/TEstrainer/scripts/initial_mafft_setup.py",
"content": "#!/usr/bin/env python\n\nimport argparse\nimport sys\nimport os\nfrom os.path import exists\n\nparser = argparse.ArgumentParser()\nparser.add_argument('-d', '--directory', type=str, required=True,\n help='Directory housing TEstrainer run')\nparser.add_argument('-r', '--iteration', type=str, required=True,\n help='Interation number of TEstrainer curation')\nparser.add_argument('-s', '--seq_name', type=str, required=True,\n help='Sequence being prepared for alignment')\nparser.add_argument('-g', '--genome', type=str, required=True,\n help='Path to genome sequence')\nparser.add_argument('-f', '--flank', type=int, default=1000,\n help='Length of flank to be extended')\nparser.add_argument('-n', '--no_seq', type=int, default=20,\n help='Number of sequences to use for alignment')\nparser.add_argument('-D', '--debug', action='store_true',\n help='Set for full messaging')\nargs = parser.parse_args()\n\n# function to check files exist\ndef file_check(file_name, debug):\n if(exists(file_name) == False):\n if(debug == False):\n exit()\n else:\n sys.exit((file_name+\" not found\"))\n\n# check input files/folders exist\nfile_check((args.directory+\"/run_\"+args.iteration+\"/raw/\"+args.seq_name), args.debug) # current consensus\nfile_check((args.directory+\"/run_0/og/\"+args.seq_name), args.debug) # og consensus\nfile_check(args.genome, args.debug) # genome sequence\nfile_check((args.directory+'/run_'+args.iteration+'/self_search/'), args.debug) # self search folder\nfile_check((args.directory+'/run_'+args.iteration+'/to_align/'), args.debug) # final output folder\nfile_check((args.directory+'/run_'+args.iteration+'/TEtrim_complete/'), args.debug) # alternate final output folder\n\nimport string\nfrom os import system\nimport statistics\nimport re\nfrom Bio import SeqIO\nfrom Bio.Seq import Seq\nfrom Bio.SeqRecord import SeqRecord\nimport pandas as pd\nimport numpy as np\nimport pyranges as pr\n\n# function to output current consensus and exit if less than a certain number of sequences are found\ndef size_check(var_name, size):\n if(len(var_name) < size):\n with open((args.directory+'/run_'+args.iteration+'/TEtrim_complete/'+args.seq_name), \"w\") as o:\n SeqIO.write(start_seq, o, \"fasta-2line\")\n exit()\n\n# Function for writing sequence from df to file \ndef df_to_fasta(df, path, name, overwrite):\n df = df.astype('str')\n if('Chromosome' not in df.columns) or ('Start' not in df.columns) or ('End' not in df.columns) or ('Seq' not in df.columns):\n sys.exit(status=\"Necessary sequence columns not found in df\")\n # Name sequences\n if('Name' not in df.columns) or (name is True):\n if('Strand' in df.columns):\n df['Name']=(df.Chromosome+\":\"+df.Start+\"-\"+df.End+\"(\"+df.Strand+\")\")\n else:\n df['Name']=(df.Chromosome+\":\"+df.Start+\"-\"+df.End)\n # Write sequences\n if(overwrite is True):\n with open((path), \"w\") as o:\n for x in range(len(df)):\n SeqIO.write(SeqRecord(seq=Seq(df.Seq[x]),id=df.Name[x],name=df.Name[x], description=\"\"), o, \"fasta-2line\")\n else:\n with open((path), \"a\") as o:\n for x in range(len(df)):\n SeqIO.write(SeqRecord(seq=Seq(df.Seq[x]),id=df.Name[x],name=df.Name[x], description=\"\"), o, \"fasta-2line\")\n\n# Function for adjusting blast output for coordinates direction, convert to base 0\ndef blast_to_bed(df):\n rev = df['Start'] > df['End']\n fwd = df['Start'] < blast_df['End']\n df.loc[rev, ['Start', 'End']] = (df.loc[rev, ['End', 'Start']].values)\n df.loc[fwd, ['Start']] = df.loc[fwd, ['Start']] - 1\n df.loc[rev, ['End']] = df.loc[rev, ['End']] +1\n return(df)\n\n# read in starting seq\nstart_seq = SeqIO.read((args.directory+\"/run_\"+args.iteration+\"/raw/\"+args.seq_name), \"fasta\")\n\n# check if sequence is largely tandem repeat\n# make pd.df to use as holder for trf coordinates\nte_len=len(start_seq.seq)\ntrf_df = pd.DataFrame(columns=['Chromosome', 'Start', 'End'])\nwith open((args.directory+'/run_'+args.iteration+'/raw/'+args.seq_name+'.trf'), 'r') as trf:\n for line in trf:\n if line.startswith('@'):\n seqnames = re.sub('@', '', line.split()[0])\n elif float(line.split()[3]) > 5:\n trf_df = pd.concat([trf_df, pd.DataFrame({'Chromosome':[seqnames], 'Start':[int(line.split()[0])], 'End':[int(line.split()[1])]})])\n# Merge trf pr\ntrf_pr=pr.PyRanges(df=trf_df).merge()\n# Calculate total length of satellite sequences\n# TEs >90% consider satellites\nif trf_pr.length/te_len > 0.9:\n with open((args.directory+'/run_'+args.iteration+'/TEtrim_complete/'+args.seq_name), \"w\") as o:\n if args.debug is True:\n print('Hold your horses, '+start_seq.name+' is likely a tandem repeat')\n SeqIO.write(start_seq, o, \"fasta-2line\")\n exit()\n# TEs >50% consider TEs needing trimming\nelif (trf_pr.length/te_len > 0.5) & (len(start_seq.seq) > 500):\n if args.debug is True:\n print('Whoa boy, '+start_seq.name+' is inside tandem repeat')\n # make pr object of whole repeat\n te_pr=pr.from_dict({'Chromosome': [seqnames], 'Start': [0], 'End': [te_len]})\n # get inverse or trf, i.e. non tandem repeat section\n trf_inverse_pr=te_pr.subtract(trf_pr)\n # convert to df\n trf_inverse_df=trf_inverse_pr.as_df()\n # calculate width\n trf_inverse_df['Width']=trf_inverse_df.End-trf_inverse_df.Start+1\n # select longest non TR section and convert to ranges\n non_trf_longest_df=trf_inverse_df[trf_inverse_df.Width==max(trf_inverse_df.Width)].head(n=1)\n # handle potential empty DataFrame to avoid IndexError on iloc[0]\n if non_trf_longest_df.empty:\n if args.debug is True:\n print('No non-tandem repeat section found for '+start_seq.name+'; writing original sequence.')\n with open((args.directory+'/run_'+args.iteration+'/TEtrim_complete/'+args.seq_name), \"w\") as o:\n SeqIO.write(start_seq, o, \"fasta-2line\")\n exit()\n non_trf_longest_pr=pr.PyRanges(df=non_trf_longest_df)\n # get sequence of non TR section, write to file\n non_trf_seq=SeqRecord(seq=Seq(str(start_seq.seq)[int(non_trf_longest_df['Start'].iloc[0]):int(non_trf_longest_df['End'].iloc[0])]), id=start_seq.id, name=start_seq.name)\n with open((args.directory+'/run_'+args.iteration+'/TEtrim_complete/'+args.seq_name), \"w\") as o:\n SeqIO.write(non_trf_seq, o, \"fasta-2line\")\n exit()\n\n# perform initial blast\nsystem(\"blastn -task dc-megablast -query \"+args.directory+\"/run_\"+args.iteration+\"/raw/\"+args.seq_name+\" -db \"+args.genome+\" -evalue 1e-5 -outfmt \\\"6 qseqid sseqid pident length qstart qend qlen sstart send slen evalue bitscore qcovs\\\" -out \"+args.directory+\"/run_\"+args.iteration+\"/initial_blast/\"+args.seq_name+\".out -num_threads 1\")\n\n# check if any hits found, if not write to file and exit\nif os.path.getsize(args.directory+'/run_'+args.iteration+'/initial_blast/'+args.seq_name+'.out') == 0:\n with open((args.directory+'/run_'+args.iteration+'/TEtrim_complete/'+args.seq_name), \"w\") as o:\n SeqIO.write(start_seq, o, \"fasta-2line\")\n sys.exit()\n\n# read in blast table and filter\nblast_df = pd.read_table((args.directory+'/run_'+args.iteration+'/initial_blast/'+args.seq_name+'.out'), names=['qseqid', 'Chromosome', 'pident', 'length', 'qstart', 'qend', 'qlen', 'Start', 'End', 'slen', 'evalue', 'bitscore', 'qcovs'])\nblast_df = blast_df.query('pident >= 70 & qcovs >= 50').copy()\nsize_check(blast_df, 3)\n\n# adjust for coordinates direction, convert to base 0\nblast_to_bed(blast_df)\n\n# select 50 best, add flanks and convert to ranges\nblast_df = blast_df.sort_values(by = 'bitscore', ascending=False)\nblast_df = blast_df.iloc[:args.no_seq]\nblast_df['Start'] = blast_df['Start'] - args.flank\nblast_df['End'] = blast_df['End'] + args.flank\nblast_df.loc[blast_df['Start'] < 1, 'Start'] = 0\nblast_df.loc[blast_df['End'] > blast_df['slen'], 'End'] = blast_df['slen']\nblast_df = blast_df.sort_values(by = ['Chromosome', 'Start'])\nblast_gr = pr.from_dict({\"Chromosome\": blast_df.Chromosome, \"Start\": blast_df.Start, \"End\": blast_df.End, \"Bitscore\": blast_df.bitscore, \"slen\" : blast_df.slen})\n\n# reduce/merge ranges\nbest_hits_df = blast_gr.cluster(strand=False).df.groupby(['Cluster']).agg({'Chromosome':'first', 'Start':'min', 'End':'max', 'Bitscore':'max'})[['Chromosome','Start','End','Bitscore']].reset_index()\nbest_hits_py = pr.from_dict({\"Chromosome\": best_hits_df.Chromosome, \"Start\": best_hits_df.Start, \"End\": best_hits_df.End, \"Bitscore\": best_hits_df.Bitscore})\n\n# get sequence\nbest_hits_seq = pr.get_fasta(best_hits_py, path=args.genome)\n\n# convert df to strings\nbest_hits_df = best_hits_df.astype('str')\nbest_hits_df['Name'] = (best_hits_df.Chromosome+\":\"+best_hits_df.Start+\"-\"+best_hits_df.End+\"#\"+best_hits_df.Bitscore)\nbest_hits_df['Seq'] = best_hits_seq\nbest_hits_outpath = (args.directory+'/run_'+args.iteration+'/self_search/'+args.seq_name+'_check_1')\n\n# write sequences to file\ndf_to_fasta(best_hits_df, best_hits_outpath, False, True)\n\n# blast against og sequence,read in and filter\nsystem(\"blastn -task dc-megablast -query \"+args.directory+'/run_0/og/'+args.seq_name+\" -subject \"+args.directory+\"/run_\"+args.iteration+\"/self_search/\"+args.seq_name+\"_check_1 -outfmt \\\"6 sseqid pident qcovs\\\" -out \"+args.directory+\"/run_\"+args.iteration+\"/self_search/\"+args.seq_name+\"_check_1.out\")\ncheck_df = pd.read_table((args.directory+\"/run_\"+args.iteration+\"/self_search/\"+args.seq_name+\"_check_1.out\"), names=['Chromosome', 'pident', 'qcovs'])\ncheck_df = check_df.query('pident >= 70 & qcovs >= 50')\nsize_check(check_df, 3)\n\n# select one instance of each sequence\ncheck_df = check_df.groupby(['Chromosome']).head(n=1).reset_index()\ncheck_df[['Chromosome', 'bitscore']] = check_df['Chromosome'].str.split('#', n=1, expand = True)\nbest_hits_df[['Name', 'bitscore']] = best_hits_df['Name'].str.split('#', n=1, expand = True)\nsize_check(check_df, 3)\n\n# sort by bitscore and select no_seq (default 20) highest bitscores\ncheck_df = check_df.astype({'bitscore': 'float'})\ncheck_df = check_df.sort_values(by = 'bitscore', ascending=False)\n\n# select accurate and write ready for self blast\ncorrect_df = best_hits_df.loc[best_hits_df.Name.isin(check_df.Chromosome)].reset_index()\ncorrect_df_path = (args.directory+'/run_'+args.iteration+'/self_search/'+args.seq_name+'_check_2')\ndf_to_fasta(correct_df, correct_df_path, False, True)\n\n# self blast\nsystem(\"blastn -task dc-megablast -query \"+correct_df_path+\" -subject \"+correct_df_path+\" -outfmt \\\"6 qseqid sseqid length pident qstart qend bitscore\\\" -out \"+correct_df_path+\".out\")\nself_blast_df = pd.read_table((args.directory+\"/run_\"+args.iteration+\"/self_search/\"+args.seq_name+\"_check_2.out\"), names=['Chromosome', 'sseqid', 'length', 'pident', 'Start', 'End', 'bitscore'])\nsize_check(self_blast_df, 3)\n\n# fix coordinates\nself_blast_df.Start = self_blast_df.Start - 1\n\n# filter self hits, small hits\nself_blast_df = self_blast_df[self_blast_df.Chromosome != self_blast_df.sseqid].copy()\nself_blast_df = self_blast_df[self_blast_df.length >= (0.5*len(start_seq))].copy()\nsize_check(self_blast_df, 3)\n\n# calculate and filter quantiles\nself_blast_df['q1'] = self_blast_df.groupby('Chromosome').Start.transform(lambda x: x.quantile(0.1))\nself_blast_df['q9'] = self_blast_df.groupby('Chromosome').End.transform(lambda x: x.quantile(0.9))\nself_blast_df = self_blast_df[(self_blast_df.Start.astype(float) >= self_blast_df.q1) & (self_blast_df.End.astype(float) <= self_blast_df.q9)].copy()\nsize_check(self_blast_df, 3)\n\n# create and reduce/merge ranges\nself_blast_gr = pr.from_dict({\"Chromosome\": self_blast_df.Chromosome, \"Start\": self_blast_df.Start, \"End\": self_blast_df.End})\nself_hits_trimmed_df = self_blast_gr.cluster(strand=False).df.groupby(['Cluster']).agg({'Chromosome':'first', 'Start':'min', 'End':'max'})[['Chromosome','Start','End']].reset_index()\nself_hits_trimmed_py = pr.from_dict({\"Chromosome\": self_hits_trimmed_df.Chromosome, \"Start\": self_hits_trimmed_df.Start, \"End\": self_hits_trimmed_df.End})\nsize_check(self_hits_trimmed_py, 3)\n \n# get trimmed sequence\nself_hits_trimmed_seq = pr.get_fasta(self_hits_trimmed_py, path=(args.directory+'/run_'+args.iteration+'/self_search/'+args.seq_name+'_check_2'))\nself_hits_trimmed_df['Seq'] = self_hits_trimmed_seq\nself_hits_trimmed_outpath = (args.directory+'/run_'+args.iteration+'/to_align/'+args.seq_name)\n\n# write current consensus fasta to file\nwith open(self_hits_trimmed_outpath, \"w\") as o:\n SeqIO.write(start_seq, o, \"fasta-2line\")\n\n# write trimmed fasta to file\ndf_to_fasta(self_hits_trimmed_df, self_hits_trimmed_outpath, True, False)\n"
},
{
"path": "scripts/TEstrainer/scripts/mreps_parser.sh",
"content": "#!/bin/bash\n# run mreps and parse to usable format\n\nusage() { echo \"Usage: [-i input sequence (fasta file)] [-h Print this help]\" 1>&2; exit 1; }\n\nwhile getopts i:h flag\ndo\n case \"${flag}\" in\n i) INFILE=${OPTARG};;\n h | *)\n usage\n exit\n esac\ndone\n\nif [ ! -f \"${INFILE}\" ]; then\necho \"Error: file not found\"\nexit\nfi\n\nmreps -minsize 20 -maxsize 50000 -minperiod 1 -maxperiod 5000 -exp 2 -fasta $INFILE | grep \"\\->\" | grep -v \"from\" | sed 's/ \\+/\\t/g' | sed 's/\\t\\+/\\t/g' | sed 's/\\t//10g' | cut -f2,4,6,7,8,9,10 | sed 's/ ://g' | sed 's/>//g' | sed 's/ ${INFILE}.mreps.temp\nawk '{OFS=\"\\t\"}{print FILENAME,$1,$2,$3,$4,$5,$6,$7}' ${INFILE}.mreps.temp | sed 's/.*\\///;s/.fasta.mreps.temp//' > ${INFILE}.mreps\nrm ${INFILE}.mreps.temp\n"
},
{
"path": "scripts/TEstrainer/scripts/simple_repeat_filter_trim.R",
"content": "library(optparse)\n\noption_list <- list(\n make_option(c(\"-i\", \"--in_seq\"), default=NA, type = \"character\", help=\"Input sequence (required)\"),\n make_option(c(\"-d\", \"--directory\"), type=\"character\", default=NULL, help=\"Path to data directory (required)\", metavar=\"character\"),\n make_option(c(\"-p\", \"--sat_perc\"), type=\"integer\", default=60, help=\"Minimum percentage of total sequence classed as SSR for repeat to be considered a satellite\")\n)\n\nopt <- parse_args(OptionParser(option_list=option_list))\n\n# check variables provided\nif(is.na(opt$in_seq)){\n stop(\"Path to in sequence must be supplied\")\n}\nif(is.na(opt$directory)){\n stop(\"Path to in sequence must be supplied\")\n}\n\nopt$out_seq <- sub(\".*/\", \"\", opt$in_seq)\n\n# make empty variable function\nsuppressPackageStartupMessages(library(tidyverse))\nsuppressPackageStartupMessages(library(plyranges))\nsuppressPackageStartupMessages(library(BSgenome))\n\n# read in library and make bed like table\nin_seq <- readDNAStringSet(opt$in_seq)\nin_seq_tbl <- tibble(seqnames = names(in_seq), og_width = width(in_seq)) %>%\n dplyr::mutate(draft_seqnames = sub(\"#.*\", \"\", seqnames))\n\n# If no simple repeats were found by any of the three tools exit early, creating necessary files\nif(\n nrow(\n read_tsv(paste0(opt$directory, \"/trf/\", opt$out_seq, \".sassr\"),\n skip = 1, col_names = c(\"seqnames\", \"ssr\", \"count\", \"start\"), show_col_types = F)) == 0 &\n file.size(paste0(opt$directory, \"/trf/\", opt$out_seq, \".trf\")) == 0 &\n file.size(paste0(opt$directory, \"/trf/\", opt$out_seq, \".mreps\")) == 0\n){\n writeXStringSet(in_seq, paste0(opt$directory, \"/trf/\", opt$out_seq, \".nonsatellite\"))\n message(\"No satellites found\")\n} else {\n\n # read in, rearrange and calculate percent tandem repeats for SA-SSR data\n # check sassr only header, if so create empty tbl, else analyse data\n if(nrow(\n read_tsv(paste0(opt$directory, \"/trf/\", opt$out_seq, \".sassr\"),\n skip = 1, col_names = c(\"seqnames\", \"ssr\", \"count\", \"start\"), show_col_types = F)) == 0){\n \n sassr_calc <- tibble(seqnames = character(), sassr_perc_tr = double())\n \n sassr_select <- tibble(seqnames = character(), start = double(), end = double(),\n period = double(), count = double(), og_width = integer(),\n ssr = character(), ssr_width = double(), package = \"sassr\")\n \n } else {\n \n sassr <- read_tsv(paste0(opt$directory, \"/trf/\", opt$out_seq, \".sassr\"), \n skip = 1, col_names = c(\"seqnames\", \"ssr\", \"count\", \"start\"), show_col_types = F, col_types = \"ccdd\") %>%\n mutate_if(is.logical, ~gsub(\"RUE|ALSE\", \"\", .)) %>%\n dplyr::mutate(ssr = ifelse(is.na(ssr), \"NA\", ssr),\n period = as.double(width(ssr))) %>%\n dplyr::mutate(ssr_width = count*period, end = start + ssr_width, start = start +1) %>%\n mutate(draft_seqnames = sub(\"#.*\", \"\", seqnames)) %>%\n inner_join(in_seq_tbl, by = \"seqnames\") %>%\n arrange(seqnames) %>%\n filter(count > 2)\n \n sassr_calc <- as_tibble(GenomicRanges::reduce(as_granges(sassr))) %>%\n dplyr::select(-strand) %>%\n dplyr::mutate(seqnames = as.character(seqnames)) %>%\n group_by(seqnames) %>%\n mutate(total_width = sum(width)) %>%\n ungroup() %>%\n inner_join(in_seq_tbl, by = \"seqnames\") %>%\n dplyr::mutate(sassr_perc_tr = 100*total_width/og_width) %>%\n dplyr::select(seqnames, sassr_perc_tr) %>%\n base::unique()\n \n sassr_select <- sassr %>% select(seqnames, start, end, period, count, og_width, ssr, ssr_width) %>%\n mutate(package = \"sassr\")\n \n }\n \n # check trf empty, if so create empty tbl, else analyse data\n # read in, rearrange and calculate percent tandem repeats for TRF data\n if(file.size(paste0(opt$directory, \"/trf/\", opt$out_seq, \".trf\")) == 0){\n trf_select <- tibble(seqnames = character(), start = double(), end = double(),\n period = double(), count = double(), og_width = integer(),\n ssr = character(), ssr_width = double(), package = \"trf\")\n \n trf_calc <- tibble(seqnames = character(), trf_perc_tr = double())\n \n } else {\n trf <- read_tsv(paste0(opt$directory, \"/trf/\", opt$out_seq, \".trf\"),\n col_names = c(\"draft_seqnames\", \"start\", \"end\", \"period\", \"count\", \"ssr\"), show_col_types = F, col_types = \"cddddc\") %>%\n mutate_if(is.logical, ~gsub(\"RUE|ALSE\", \"\", .)) %>%\n mutate(ssr = ifelse(is.na(ssr), \"NA\", ssr),\n draft_seqnames = sub(\"@\", \"\", sub(\"#.*\", \"\", draft_seqnames))) %>%\n dplyr::mutate(ssr_width = end - start + 1) %>%\n inner_join(in_seq_tbl, by = \"draft_seqnames\")\n \n trf_select <- trf %>%\n filter(count > 2) %>%\n dplyr::select(seqnames, start, end)\n \n trf_calc <- as_tibble(GenomicRanges::reduce(as_granges(trf_select))) %>%\n dplyr::select(-strand) %>%\n dplyr::mutate(seqnames = as.character(seqnames)) %>%\n group_by(seqnames) %>%\n mutate(total_width = sum(width)) %>%\n ungroup() %>%\n inner_join(in_seq_tbl, by = \"seqnames\") %>%\n dplyr::mutate(trf_perc_tr = 100*total_width/og_width) %>%\n dplyr::select(seqnames, trf_perc_tr) %>%\n base::unique()\n \n trf_select <- trf %>% select(seqnames, start, end, period, count, og_width, ssr, ssr_width) %>%\n mutate(package = \"trf\")\n \n }\n \n # read in, rearrange and calculate percent tandem repeats for MREPS data\n if(file.size(paste0(opt$directory, \"/trf/\", opt$out_seq, \".mreps\")) == 0){\n mreps_select <- tibble(seqnames = character(), start = double(), end = double(),\n period = double(), count = double(), og_width = integer(),\n ssr = character(), ssr_width = double(), package = \"mreps\")\n \n mreps_calc <- tibble(seqnames = character(), mreps_perc_tr = double())\n \n } else {\n mreps <- read_tsv(paste0(opt$directory, \"/trf/\", opt$out_seq, \".mreps\"),\n col_names = c(\"draft_seqnames\", \"start\", \"end\", \"ssr_width\", \"period\", \"count\", \"error\", \"sequence\"), show_col_types = F, col_types = \"cdddddcc\") %>%\n mutate_if(is.logical, ~gsub(\"RUE|ALSE\", \"\", .)) %>%\n mutate(ssr = substr(x = sequence, start = 0, stop = period)) %>%\n inner_join(in_seq_tbl, by = \"draft_seqnames\")\n \n mreps_select <- mreps %>%\n filter(count > 2) %>%\n dplyr::select(seqnames, start, end)\n \n mreps_calc <- as_tibble(GenomicRanges::reduce(as_granges(mreps_select))) %>%\n dplyr::select(-strand) %>%\n dplyr::mutate(seqnames = as.character(seqnames)) %>%\n group_by(seqnames) %>%\n mutate(total_width = sum(width)) %>%\n ungroup() %>%\n inner_join(in_seq_tbl, by = \"seqnames\") %>%\n dplyr::mutate(mreps_perc_tr = 100*total_width/og_width) %>%\n dplyr::select(seqnames, mreps_perc_tr) %>%\n base::unique()\n \n mreps_select <- mreps %>% select(seqnames, start, end, period, count, og_width, ssr, ssr_width) %>%\n mutate(package = \"mreps\")\n }\n \n # Compile data\n compiled_tr <- rbind(rbind(trf_select, mreps_select), sassr_select)\n stats_tr <- full_join(sassr_calc, full_join(mreps_calc, trf_calc, by = \"seqnames\"), by = \"seqnames\")\n \n # if no data write to og seq to file as nonsatellite and exit\n if(nrow(compiled_tr) == 0){\n writeXStringSet(in_seq, paste0(opt$directory, \"/trf/\", opt$out_seq, \".nonsatellite\"))\n stop()\n }\n \n # Identifying satellite/simple repeats\n over50tr <- stats_tr %>%\n filter(sassr_perc_tr >= opt$sat_perc | mreps_perc_tr >= opt$sat_perc | trf_perc_tr >= opt$sat_perc) %>%\n mutate(sassr_perc_tr = ifelse(is.na(sassr_perc_tr), 0, sassr_perc_tr),\n mreps_perc_tr = ifelse(is.na(mreps_perc_tr), 0, mreps_perc_tr),\n trf_perc_tr = ifelse(is.na(trf_perc_tr), 0, trf_perc_tr))\n \n satellites <- compiled_tr[compiled_tr$seqnames %in% over50tr$seqnames,] %>%\n dplyr::group_by(seqnames) %>%\n dplyr::arrange(-ssr_width, period) %>%\n dplyr::slice(1) %>%\n dplyr::ungroup() %>%\n dplyr::mutate(perc_tr = 100*ssr_width/og_width) %>%\n inner_join(over50tr, by = \"seqnames\") %>%\n dplyr::mutate(max_perc_tr = pmax(sassr_perc_tr, mreps_perc_tr, trf_perc_tr)) %>%\n dplyr::arrange(max_perc_tr)\n \n # for macrosatellites trim to get single copy\n macrosatellites <- satellites %>%\n filter(max_perc_tr > 90, count >= 2) %>%\n filter(period > 200) %>%\n mutate(start = start + period,\n end = start + period,\n end = ifelse(end > og_width, og_width, end)) %>%\n as_granges()\n macrosatellites_seq <- getSeq(in_seq, macrosatellites)\n names(macrosatellites_seq) <- sub(\"#.*\", \"#Satellite\", seqnames(macrosatellites))\n \n # leave micro to minisatellite repeats intact\n other_satellites <- satellites %>%\n dplyr::filter(!seqnames %in% seqnames(macrosatellites)) %>%\n dplyr::mutate(start = 1, end = og_width) %>%\n as_granges()\n other_satellites_seq <- getSeq(in_seq, other_satellites)\n names(other_satellites_seq) <- sub(\"#.*\", \"#Satellite\", seqnames(other_satellites))\n \n # Combine satellite repeats and write to file\n writeXStringSet(c(other_satellites_seq, macrosatellites_seq), paste0(opt$directory, \"/trf/\", opt$out_seq, \".satellites\"))\n \n # filtering for trimming\n trim_3 <- compiled_tr %>%\n mutate(tr_len = period*count) %>%\n filter(count >=3, !seqnames %in% over50tr$seqnames, period < 20) %>%\n filter(og_width-end <= period) %>%\n dplyr::group_by(seqnames) %>%\n arrange(-end, period) %>%\n dplyr::slice(1)\n trim_5 <- compiled_tr %>%\n mutate(tr_len = period*count) %>%\n filter(count >=3, !seqnames %in% over50tr$seqnames, period < 20) %>%\n filter(start <= period) %>%\n dplyr::group_by(seqnames) %>%\n arrange(-end, period) %>%\n dplyr::slice(1)\n \n # Determine thos to be trimmed both ends\n trim_both_5 <- trim_5[trim_5$seqnames %in% trim_3$seqnames,] %>%\n dplyr::mutate(start = ifelse(period < 4, end - 8, end - (2*period))) %>%\n dplyr::mutate(start = ifelse(start < 1, 1, start)) %>%\n dplyr::select(seqnames, start)\n trim_both_3 <- trim_3[trim_3$seqnames %in% trim_5$seqnames,] %>%\n dplyr::mutate(end = ifelse(period < 4, start + 8, start + (2*period))) %>%\n dplyr::mutate(end = ifelse(end > og_width, og_width, end)) %>%\n dplyr::select(seqnames, end)\n trim_both <- inner_join(trim_both_5, trim_both_3, by = \"seqnames\")\n \n # determine those to be trimmed one end\n trim_only_5 <- trim_5[!trim_5$seqnames %in% trim_3$seqnames,] %>%\n dplyr::mutate(start = ifelse(period < 4, end - 8, end - (2*period))) %>%\n dplyr::mutate(start = ifelse(start < 1, 1, start), end = og_width) %>%\n dplyr::select(seqnames, start, end)\n trim_only_3 <- trim_3[!trim_3$seqnames %in% trim_5$seqnames,] %>%\n dplyr::mutate(end = ifelse(period < 4, start + 8, start + (2*period))) %>%\n dplyr::mutate(end = ifelse(end > og_width, og_width, end), start = 1) %>%\n dplyr::select(seqnames, start, end)\n \n to_trim <- as_granges(rbind(trim_both, rbind(trim_only_5, trim_only_3)))\n trimmed_seq <- getSeq(in_seq, to_trim)\n names(trimmed_seq) <- seqnames(to_trim)\n \n # determine untouched sequences\n untouched_seq <-\n in_seq[!sub(\"#.*\", \"\", names(in_seq)) %in% sub(\"#.*\", \"\", c(names(trimmed_seq), names(macrosatellites_seq), names(other_satellites_seq))),]\n \n # combine untouched and trimmed sequences and write to file\n writeXStringSet(c(trimmed_seq, untouched_seq), paste0(opt$directory, \"/trf/\", opt$out_seq, \".nonsatellite\"))\n \n}\n"
},
{
"path": "scripts/TEstrainer/scripts/splitter.py",
"content": "#!/usr/bin/env python\n\nimport os\nimport re\nfrom os.path import exists\nimport sys\nimport argparse\nfrom Bio import SeqIO\n\nparser = argparse.ArgumentParser()\nparser.add_argument('-i', '--in_seq', type=str, required=True,\n help='Input multi-fasta to be split')\nparser.add_argument('-o', '--out_dir', type=str, required=True,\n help='Output directory') \nargs = parser.parse_args()\nif(exists(args.in_seq) == False):\n sys.exit('File not found')\nif(exists(args.out_dir) == False):\n os.mkdir(args.out_dir)\nfile_list=[]\n\n# split fasta file\nwith open(args.in_seq, 'r') as handle:\n for record in SeqIO.parse(handle, \"fasta\"):\n file_name = (args.out_dir+\"/\"+record.name.split(sep=\"#\")[0]+\".fasta\")\n file_list.append(record.name.split(sep=\"#\")[0]+\".fasta\")\n SeqIO.write(record, file_name, \"fasta-2line\")\n# write file list\nwith open((args.out_dir+\"/\"+re.sub('.*/', '', args.in_seq)+\"_split.txt\"), 'w') as fp:\n for item in file_list:\n # write each item on a new line\n fp.write(\"%s\\n\" % item)\n"
},
{
"path": "scripts/TEstrainer/scripts/strainer.R",
"content": "#!/usr/bin/Rscript\n\nlibrary(optparse)\n\noption_list <- list(\n make_option(c(\"-i\", \"--in_seq\"), default=NA, type = \"character\", help=\"Path to input sequence (required)\"),\n make_option(c(\"-p\", \"--plot_chimeras\"), type = \"logical\", help=\"Set to TRUE to make plots of chimeras\", default = TRUE),\n make_option(c(\"-d\", \"--directory\"), default=NA, type = \"character\", help=\"Path to directory (required)\",)\n)\n\nopt <- parse_args(OptionParser(option_list=option_list))\n\n# check variables provided\nif(is.na(opt$in_seq)){\n stop(\"Path to in sequence must be supplied\")\n}\nif(is.na(opt$directory)){\n stop(\"Path to directory must be supplied\")\n}\n\nopt$out_seq <- sub(\".*/\", \"\", opt$in_seq)\nopt$rps_table <- paste0(opt$directory, \"/chimeras/\", opt$out_seq, \".rps.out\")\n\n# make empty variable function\nsuppressPackageStartupMessages(library(tidyverse))\nsuppressPackageStartupMessages(library(plyranges))\nsuppressPackageStartupMessages(library(BSgenome))\nsuppressPackageStartupMessages(library(grid))\n\n# read in fasta\nrm_seq_in <- readDNAStringSet(paste0(opt$in_seq))\nnames(rm_seq_in) <- sub(\" .*\", \"\", names(rm_seq_in))\nrm_seq_info <- tibble(seqnames = names(rm_seq_in), width = width(rm_seq_in))\n\ncompiled_acceptable <- tibble()\ntruly_chimeric_ranges <- GRanges()\nquestionable <- tibble()\nno_domains_seq <- DNAStringSet()\n\n# read in acceptable domains and exceptional domains and manually added additional domains\nexceptional_domains <- read_tsv(\"data/exceptional_domains.tsv\", show_col_types = FALSE) %>%\n dplyr::select(ref, class) %>% dplyr::rename(excep_class = class)\nadditional_domains <- read_tsv(\"data/additional_domains.tsv\", show_col_types = FALSE) %>%\n dplyr::select(ref, class) %>% dplyr::rename(excep_class = class)\nexceptional_domains <- rbind(exceptional_domains, additional_domains)\n\nacceptable_domains <- read_tsv(\"data/acceptable_domains.tsv\", show_col_types = FALSE)\n\nif(file.size(opt$rps_table)==0){\n writeXStringSet(rm_seq_in, paste0(opt$directory, \"/chimeras/clean_\", opt$out_seq))\n quit()\n}\n\n# read rps blast out\nrps_blast_out <- read_tsv(file = opt$rps_table,\n col_names = c(\"seqnames\", \"qstart\", \"qend\", \"qlen\", \"sseqid\", \"sstart\", \"send\", \"slen\",\n \"pident\", \"length\", \"mismatch\", \"gapopen\", \"evalue\", \"bitscore\", \"qcovs\", \"stitle\"),\n show_col_types = FALSE) %>%\n tidyr::separate(stitle, into = c(\"ref\", \"abbrev\", \"full\"), sep = \", \", extra = \"drop\") %>%\n dplyr::filter(length >= 0.5*slen)\n\nrps_blast_out$n <- 1:nrow(rps_blast_out)\n\n# select acceptable and non acceptable\ncontains_acceptable <- rps_blast_out[rps_blast_out$ref %in% acceptable_domains$ref,]\ncontains_unacceptable <- rps_blast_out[!rps_blast_out$ref %in% acceptable_domains$ref,]\n\n# Filter repeats containing only suitable domains\ncompletely_acceptable <- rps_blast_out %>%\n filter(seqnames %in% contains_acceptable$seqnames,\n !seqnames %in% contains_unacceptable$seqnames) %>%\n arrange(seqnames, qstart)\n\n# Filter repeats containing no suitable domains\nquestionable <- rps_blast_out %>%\n filter(!seqnames %in% contains_acceptable$seqnames,\n seqnames %in% contains_unacceptable$seqnames) %>%\n arrange(seqnames, qstart)\n\n# Filter repeats containing both suitable and unsuitable domains\nchimeric <- rps_blast_out %>%\n filter(seqnames %in% contains_acceptable$seqnames,\n seqnames %in% contains_unacceptable$seqnames) %>%\n arrange(seqnames, qstart) %>%\n mutate(unacceptable = ifelse(ref %in% acceptable_domains$ref, \"false\", \"true\"))\n\n# check if domains fit exceptional circumstances (grepl used to allow for the various ERVs)\nexcep_check <- chimeric %>%\n inner_join(exceptional_domains)\n\nfor( i in 1:nrow(excep_check)){\n if(grepl(pattern = excep_check$excep_class[i], x = excep_check$seqnames[i])){\n excep_check$unacceptable[i] <- \"false\"\n } else {\n excep_check$unacceptable[i] <- \"true\"\n }\n}\n\n# select single for each excep check, ensuring acceptable retained if exists\nexcep_check <- excep_check %>%\n group_by(n) %>%\n dplyr::arrange(unacceptable) %>%\n dplyr::slice(1) %>%\n dplyr::select(-excep_class)\n\n# add excep_checked back into fold\nchimeric <- chimeric %>%\n dplyr::filter(!n %in% excep_check$n) %>%\n base::rbind(excep_check) %>%\n arrange(n)\n\n# Determine regions with acceptable domains overlapping \"unacceptable\"\nchimeric_ranges <- chimeric %>%\n dplyr::mutate(start = ifelse(qstart < qend, qstart, qend),\n end = ifelse(qstart > qend, qstart, qend),\n strand = ifelse(qstart < qend, \"+\", \"-\")) %>%\n dplyr::select(seqnames, start, end, strand, abbrev, unacceptable) %>%\n plyranges::as_granges()\n\nchimeric_ranges_unacceptable_false <- chimeric_ranges[chimeric_ranges$unacceptable == \"false\",]\nchimeric_ranges_unacceptable_true <- chimeric_ranges[chimeric_ranges$unacceptable == \"true\",]\n\n# find regions which don't overlap with acceptable domains\ntruly_chimeric_ranges <- filter_by_non_overlaps(chimeric_ranges_unacceptable_true, chimeric_ranges_unacceptable_false)\n\ntruly_chimeric <- chimeric[chimeric$seqnames %in% seqnames(truly_chimeric_ranges),]\nfalse_positive_chimeric <- chimeric[!chimeric$seqnames %in% seqnames(truly_chimeric_ranges),] %>%\n dplyr::select(-unacceptable)\n\ncompiled_acceptable <- rbind(completely_acceptable, false_positive_chimeric) %>%\n dplyr::mutate(strand = ifelse(qstart < qend, \"+\", \"-\"), start = 1, end = qlen) %>%\n dplyr::group_by(seqnames) %>%\n dplyr::arrange(-bitscore) %>%\n dplyr::slice(1) %>%\n dplyr::ungroup() %>%\n plyranges::as_granges()\n\n# identify sequences with no domains\nno_domains_seq <- rm_seq_in[!names(rm_seq_in) %in% c(chimeric$seqnames, completely_acceptable$seqnames, questionable$seqnames)]\n\n## add step to combine data\n# write to file (check if any filtered, if not write all in to output)\ncompletely_acceptable_seq <- getSeq(rm_seq_in, compiled_acceptable)\nnames(completely_acceptable_seq) <- seqnames(compiled_acceptable)\nwriteXStringSet(c(completely_acceptable_seq, no_domains_seq), paste0(opt$directory, \"/chimeras/clean_\", opt$out_seq))\n\n# in writing to file flip if appropriate\nif(nrow(truly_chimeric) > 0){\n \n truly_chimeric_ranges <- truly_chimeric %>%\n dplyr::mutate(strand = ifelse(qstart < qend, \"+\", \"-\"), start = 1, end = qlen) %>%\n dplyr::group_by(seqnames) %>%\n dplyr::arrange(-bitscore) %>%\n dplyr::slice(1) %>%\n dplyr::ungroup() %>%\n plyranges::as_granges()\n \n chimeric_seq <- getSeq(rm_seq_in, truly_chimeric_ranges)\n names(chimeric_seq) <- seqnames(truly_chimeric_ranges)\n writeXStringSet(chimeric_seq, paste0(opt$directory, \"/chimeras/chimeric_\", opt$out_seq))\n\n}\n\nif(nrow(questionable) > 0){\n \n questionable_ranges <- questionable %>%\n dplyr::mutate(strand = ifelse(qstart < qend, \"+\", \"-\"), start = 1, end = qlen) %>%\n dplyr::group_by(seqnames) %>%\n dplyr::arrange(-bitscore) %>%\n dplyr::slice(1) %>%\n dplyr::ungroup() %>%\n plyranges::as_granges()\n \n questionable_seq <- getSeq(rm_seq_in, questionable_ranges)\n names(questionable_seq) <- seqnames(questionable_ranges)\n writeXStringSet(questionable_seq, paste0(opt$directory, \"/chimeras/questionable_\", opt$out_seq))\n \n}\n# plotting section for manual consideration of truly_chimeric elements\nif(opt$plot == TRUE & nrow(truly_chimeric) > 0){\n \n if(!file.exists(paste0(opt$directory, \"/chimeras/plots/\"))){\n dir.create(paste0(opt$directory, \"/chimeras/plots/\"))\n }\n \n for(i in 1:length(base::unique(truly_chimeric$seqnames))){\n \n to_plot <- truly_chimeric[truly_chimeric$seqnames == base::unique(truly_chimeric$seqnames)[i],] %>% arrange(qstart) %>%\n mutate(missing = ifelse(ref %in% acceptable_domains$ref, FALSE, TRUE))\n to_plot$y = 1:nrow(to_plot)\n to_plot$text_x = (to_plot$qstart + to_plot$qend)/2\n ggplot(to_plot) +\n geom_rect(aes(xmin = qstart, xmax = qend, ymin = y-1, ymax = y, fill = missing)) +\n geom_segment(aes(x = qstart, xend = qend, y = y, yend = y),\n arrow = arrow(length = unit(10,\"points\"))) +\n geom_text(aes(x = text_x, y = y-0.5, label = abbrev)) +\n scale_x_continuous(limits = c(0,to_plot$qlen[1]), name = \"Coordinates (bp)\", expand = c(0,0)) +\n scale_y_continuous(name = NULL, breaks = NULL) +\n scale_fill_discrete(name=\"Atypical of TEs?\") +\n ggtitle(to_plot$seqnames[1]) +\n theme_bw()\n \n ggsave(filename = paste0(opt$directory, \"/chimeras/plots/\", gsub(\"/\", \"_\", base::unique(truly_chimeric$seqnames)[i]), \".svg\"), device = \"svg\")\n \n }\n \n}\n"
},
{
"path": "scripts/TEstrainer/scripts/trf_parser.py",
"content": "#!/usr/bin/env python\n\nfrom argparse import (ArgumentParser, FileType)\n\ndef parse_args():\n \"Parse the input arguments, use '-h' for help\"\n parser = ArgumentParser(description='Convert Tandem Repeat Finder (TRF) dat file to bed format with repeat units for microsatellite genotyping')\n parser.add_argument(\n '-i', '--trf', type=str, required=True,\n help='Input file produced by Tandem Repeat Finder (TRF)')\n parser.add_argument(\n '-o', '--out', type=str, required=True,\n help='Output file containing genomic locations and repeat units of microsatellites.')\n\n return parser.parse_args() \n\ndef main():\n # Parse command line arguments\n args = parse_args()\n trffile = args.trf\n outfile = args.out\n\n with open(outfile, 'w') as out:\n chrom = \"\"\n with open(trffile, 'r') as trf:\n for line in trf:\n splitline = line.split()\n if line.startswith(\"@\"):\n chrom = line.split()[0]\n else:\n start = splitline[0]\n end = splitline[1]\n period = splitline[2]\n count = splitline[3]\n ssr = splitline[13]\n out.write('\\t'.join([chrom,start,end,period,count,ssr]) + '\\n')\n\nif __name__ == '__main__':\n main()\n "
},
{
"path": "scripts/autoLand.R",
"content": "# load libraries\n\nlibrary(tidyverse)\nlibrary(data.table)\nlibrary(magrittr)\n\n# set options\n\noptions(scipen = 100, stringsAsFactors = FALSE)\n\n# get inputs\n\nargs <- commandArgs()\n# print(args)\ninDist <- args[6]\ngen <- args[7]\nspec <- args[8]\nsaveLand <- args[9]\n## bed file\n## genome size\n## landscape name\n\n# test\n#inDist = \"~/projects/monarch/datingInsertions/danausPlexippus.divsum.unaltered\"\n#gen = 248676414\n#spec = \"test\"\n\n# read table\n\ndist <- as.data.frame(t(fread(inDist, skip = \"Div\", header = TRUE)))\ndist$species <- spec\ndist$type <- rownames(dist)\n\n# fix shitty columns\n\ndist <- dist[! dist$type == \"X\",]\ncolnames(dist) <- c(dist[1,1:71], \"species\", \"type\")\ndist <- dist[! dist$type == \"Div\",]\nrownames(dist) <- 1:nrow(dist)\n\n# get sidple TE classification levels\n\ndist$classif <- dist$type\ndist$classif <- gsub(\"^DNA.*\", \"DNA\", dist$classif)\ndist$classif <- gsub(\"^LTR.*\", \"LTR\", dist$classif)\ndist$classif <- gsub(\".*Penelope|^PLE.*\", \"Penelope\", dist$classif)\ndist$classif <- gsub(\"^LINE.*\", \"LINE\", dist$classif)\ndist$classif <- gsub(\"^SINE.*\", \"SINE\", dist$classif)\ndist$classif <- gsub(\"^RC.*\", \"Rolling Circle\", dist$classif)\ndist$classif <- gsub(\"^Unknown.*|Retroposon.*|Unspecified.*\", \"Unclassified\", dist$classif)\ndist$classif <- gsub(\".*RNA.*|^Satellite.*|^Simple_repeat.*|^Low_complexity.*|^ARTEFACT.*|^Other.*|^Segmenta.*\", \"Other (Simple Repeat, Microsatellite, RNA)\", dist$classif)\n\n\n# long format\n\npiv <- dist[,c(1:72, 74)] %>% pivot_longer(!c(\"species\", \"classif\"), names_to = \"Divergence\", values_to = \"count\") %>% group_by(species, classif, Divergence) %>% summarise(Count = sum(count))\n\npiv$Divergence <- as.numeric(piv$Divergence)\n\ncolourOrder <- c(\"DNA\",\n \"Rolling Circle\",\n \"Penelope\",\n \"LINE\",\n \"SINE\",\n \"LTR\",\n \"Other (Simple Repeat, Microsatellite, RNA)\",\n \"Unclassified\")\n\npiv$classif %<>% as.factor() %>% ordered(levels = colourOrder)\n\n# add genome size and scale number of elements\n\ngenomeSizes <- as.data.frame(matrix(c(spec, gen),\n byrow = TRUE, ncol = 2))\n\ncolnames(genomeSizes) <- c(\"species\", \"genomeSize\")\n\npiv <- merge(piv, genomeSizes)\n\npiv$genomeSize <- as.numeric(piv$genomeSize)\n\npiv$proportion <- piv$Count / piv$genomeSize\n\nfor (i in 1:nrow(piv)) {\n if (piv$Count[i] < 0) {\n piv$Count[i] <- 0\n }\n}\n\nfor (i in 1:length(piv$proportion)){\n if (piv$proportion[i] < 0) {\n piv$proportion[i] <- 0\n }\n if (i %% 100 == 0) { #if x is divisible by 100\n cat(sprintf(\"Completed: %s\\r\", i)) #print Completed: %s\\r\n }\n}\n\n\n# summarise\n\npiv <- piv %>% distinct()\npiv2 <- piv %>% group_by(species, Divergence, classif) %>% summarise(proportion = sum(proportion))\n\n# set plot limits\n\nlimits <- piv2 %>% group_by(Divergence) %>% summarise(perc = sum(proportion * 100))\nylimit <- max(limits$perc) * 1.1\n\n# set colours\n\ncolours <- as.data.frame(matrix(c(\"DNA\", \"#E32017\",\n \"Rolling Circle\", \"#EE7C0E\",\n \"Penelope\", \"#7156A5\",\n \"LINE\", \"#0098D4\",\n \"SINE\", \"#9B0056\",\n \"LTR\", \"#00782A\",\n \"Other (Simple Repeat, Microsatellite, RNA)\", \"#F3A9BB\",\n \"Unclassified\", \"#A0A5A9\"), ncol = 2, byrow = TRUE))\n\ncolnames(colours) <- c(\"tclassif\", \"colour\")\ncol <- colours$colour\nnames(col) <- colours$tclassif\n\n# plot\n\nplot <- ggplot(piv2[! piv2$classif %like% \"Other\",], aes(x = Divergence, y = proportion*100, fill = classif)) +\n geom_bar(stat = \"identity\") +\n ylim(0, ylimit) +\n scale_x_reverse() +\n scale_fill_manual(values = col) +\n theme_classic() +\n theme(strip.background = element_blank(), strip.text.y = element_text(face = \"italic\", angle = 0),\n axis.text.y = element_text(size = 8)) +\n labs(y = \"Percentage of Genome\", x = \"Kimura Distance (CpG Adjusted)\", fill = \"\")\n\nggsave(saveLand,\n plot = plot,\n scale = 1,\n width = 297,\n height = 210,\n units = \"mm\",\n dpi = 300,\n limitsize = FALSE)\n"
},
{
"path": "scripts/autoPie.R",
"content": "# load libraries\n\nsuppressPackageStartupMessages(library(tidyverse))\nsuppressPackageStartupMessages(library(data.table))\nsuppressPackageStartupMessages(library(kableExtra))\n\n# set options\n\noptions(scipen = 100, stringsAsFactors = FALSE)\n\n# get inputs\n\n\nargs <- commandArgs()\n# print(args)\ninBed <- args[6]\ninGff <- args[7]\ngen <- args[8]\nsavePie <- args[9]\nsaveTab <- args[10]\n\n# read table\n\ninput <- read.table(inBed, header = TRUE, sep = \"\\t\")\ninput2 <- read.table(inGff, header = FALSE, sep = \"\\t\")\ngen <- as.numeric(gen)\n\n# set colnames\ncolnames(input2) <- c(\"scaf\", \"method\", \"classif\", \"start\", \"end\", \"score\", \"strand\", \"dot\", \"attributes\")\n\n# simple classification\ninput$tclassif <- input$Family\ninput$tclassif <- gsub(\"^DNA.*\", \"DNA\", input$tclassif)\ninput$tclassif <- gsub(\"^RC.*\", \"Rolling Circle\", input$tclassif)\ninput$tclassif <- gsub(\".*Penelope|^PLE.*\", \"Penelope\", input$tclassif)\ninput$tclassif <- gsub(\"^LINE.*\", \"LINE\", input$tclassif)\ninput$tclassif <- gsub(\"^SINE.*\", \"SINE\", input$tclassif)\ninput$tclassif <- gsub(\"^LTR.*\", \"LTR\", input$tclassif)\ninput$tclassif <- gsub(\"^Unknown.*|Retroposon.*|Unspecified.*|^unknown.*\", \"Unclassified\", input$tclassif)\ninput$tclassif <- gsub(\".*RNA.*|^Satellite.*|^Simple_repeat.*|^Low_complexity.*|^ARTEFACT.*|^repeat.*|^Other.*|^Microsatellite.*\", \"Other (Simple Repeat, Microsatellite, RNA)\", input$tclassif)\ninput <- input %>%\n mutate(tclassif = case_when(input$Family %like% \"-nested\" ~ paste0(tclassif, \"-nested\"),\n .default = tclassif))\ninput$tclassif <- gsub(\"Unmasked\", \"Non-Repeat\", input$tclassif)\n\ninput2$tclassif <- input2$classif\ninput2$tclassif <- gsub(\"^DNA.*\", \"DNA\", input2$tclassif)\ninput2$tclassif <- gsub(\"^RC.*\", \"Rolling Circle\", input2$tclassif)\ninput2$tclassif <- gsub(\".*Penelope|^PLE.*\", \"Penelope\", input2$tclassif)\ninput2$tclassif <- gsub(\"^LINE.*\", \"LINE\", input2$tclassif)\ninput2$tclassif <- gsub(\"^SINE.*\", \"SINE\", input2$tclassif)\ninput2$tclassif <- gsub(\"^LTR.*\", \"LTR\", input2$tclassif)\ninput2$tclassif <- gsub(\"^Unknown.*|Retroposon.*|Unspecified.*|^unknown.*\", \"Unclassified\", input2$tclassif)\ninput2$tclassif <- gsub(\".*RNA.*|^Satellite.*|^Simple_repeat.*|^Low_complexity.*|^ARTEFACT.*|^repeat.*|^Other.*|^Microsatellite.*\", \"Other (Simple Repeat, Microsatellite, RNA)\", input2$tclassif)\ninput2 <- input2 %>%\n mutate(tclassif = case_when(attributes %like% \"FULLY_NESTED\" ~ paste0(tclassif, \"-nested\"),\n .default = tclassif))\n\n\n# set colours\n\n# https://oobrien.com/2012/01/tube-colours/\n\n# DNA = #E32017\n# Rolling Circle = #EE7C0E\n# Penelope = #7156A5\n# LINE = #0098D4\n# SINE = #9B0056\n# LTR = #00782A\n# Other = #F3A9BB\n# Unclas = #A0A5A9\n# Non-Repeat = #000000\n\ncolours <- as.data.frame(matrix(c(\"DNA\", \"#E32017\",\n \"Rolling Circle\", \"#EE7C0E\",\n \"Penelope\", \"#7156A5\",\n \"LINE\", \"#0098D4\",\n \"SINE\", \"#9B0056\",\n \"LTR\", \"#00782A\",\n \"Other (Simple Repeat, Microsatellite, RNA)\", \"#F3A9BB\",\n \"Unclassified\", \"#A0A5A9\",\n \"Non-Repeat\", \"#000000\"), ncol = 2, byrow = TRUE))\n\ncolnames(colours) <- c(\"tclassif\", \"colour\")\ncol <- colours$colour\nnames(col) <- colours$tclassif\n\n# summary table\n\n# use summary table generated previously...\npieCou <- input %>% \n select(tclassif, Number.of.Elements)\ncolnames(pieCou) <- c(\"tclassif\", \"count\")\npieSum <- input\n\n# set factor order\npieSum$tclassif %<>% as.factor() %>% ordered(levels = c(\"DNA\",\n \"DNA-nested\",\n \"Rolling Circle\",\n \"Rolling Circle-nested\",\n \"Penelope\",\n \"Penelope-nested\",\n \"LINE\",\n \"LINE-nested\",\n \"SINE\",\n \"SINE-nested\",\n \"LTR\",\n \"LTR-nested\",\n \"Other (Simple Repeat, Microsatellite, RNA)\",\n \"Other (Simple Repeat, Microsatellite, RNA)-nested\",\n \"Unclassified\",\n \"Unclassified-nested\",\n \"Total Interspersed Repeat\",\n \"Non-Repeat\"))\n\n# get proportions\n\npieSum$proportion <- pieSum$Percentage.of.Sequence / 100\npieSum$percentage <- pieSum$Percentage.of.Sequence\npieSum$genome_size <- pieSum$GenomeSize\n\n# plot pie\n\nplot <- ggplot(pieSum %>%\n filter(! tclassif %like% \"-nested\",\n ! tclassif %in% c(\"Total Interspersed Repeat\")), aes(x = as.numeric(gen/2), y = proportion, fill = tclassif, width = as.numeric(gen))) +\n geom_bar(stat = \"identity\", position = \"fill\") +\n coord_polar(\"y\", start = 0) +\n scale_fill_manual(values = col) +\n theme_classic() +\n theme(axis.line = element_blank(), axis.text = element_blank(), axis.ticks = element_blank(), axis.title = element_blank(),\n strip.text = element_text(size = 16, face = \"italic\"), legend.text = element_text(size = 16), legend.title = element_blank())\n\nggsave(savePie,\n plot = plot,\n scale = 1,\n width = 297,\n height = 210,\n units = \"mm\",\n dpi = 300,\n limitsize = FALSE)\n\n\n\n# generate summary of TE family abundance\n\nfamilyAbundance <- input2 %>%\n mutate(family = gsub(\".*NAME=\", \"\", gsub(\";TSTART.*\", \"\", attributes))) %>%\n mutate(name = paste0(family, \"#\", classif)) %>%\n group_by(name) %>%\n summarise(coverage = sum((end - start) + 1)) %>% \n arrange(-coverage)\n\nfamilyTally <- input2 %>% \n mutate(family = gsub(\".*NAME=\", \"\", gsub(\";TSTART.*\", \"\", attributes))) %>%\n mutate(name = paste0(family, \"#\", classif)) %>%\n group_by(name) %>%\n tally(name = \"copy_number\")\n\nfamilyAbundance <- merge(familyAbundance, familyTally) %>%\n arrange(-coverage)\n\ncolnames(familyAbundance) <- c(\"TE Family\", \"Coverage (bp)\", \"Copy Number\")\n\nsaveFam <- gsub(\"highLevelCount\", \"familyLevelCount\", saveTab)\n\nclassTally <- input2 %>% \n group_by(tclassif) %>%\n tally(name = \"Copy.Number\") %>%\n rename(\"Family\" = \"tclassif\")\n\nclassFamilyCount <- input2 %>%\n mutate(family = gsub(\".*NAME=\", \"\", gsub(\";TSTART.*\", \"\", attributes))) %>%\n select(family, tclassif) %>%\n distinct() %>%\n group_by(tclassif) %>%\n tally(name = \"Family.Count\") %>%\n rename(\"Family\" = \"tclassif\")\n\npieSum <- pieSum %>%\n ungroup() %>%\n select(-c(Family)) %>%\n rename(\"Family\" = \"tclassif\") %>%\n left_join(classTally) %>%\n left_join(classFamilyCount)\n\n# finalise table\npieSum$Family %<>% as.factor() %>% ordered(levels = c(\"DNA\",\n \"DNA-nested\",\n \"Rolling Circle\",\n \"Rolling Circle-nested\",\n \"Penelope\",\n \"Penelope-nested\",\n \"LINE\",\n \"LINE-nested\",\n \"SINE\",\n \"SINE-nested\",\n \"LTR\",\n \"LTR-nested\",\n \"Other (Simple Repeat, Microsatellite, RNA)\",\n \"Other (Simple Repeat, Microsatellite, RNA)-nested\",\n \"Unclassified\",\n \"Unclassified-nested\",\n \"Total Interspersed Repeat\",\n \"Non-Repeat\"))\n\npieSum <- pieSum %>%\n arrange(Family) %>%\n select(-c(percentage, genome_size, proportion, Copy.Number)) %>%\n relocate(Family, Length.Occupied, Number.of.Elements, Percentage.of.Sequence, GenomeSize, Family.Count)\n\ncolnames(pieSum) <- c(\"TE Classification\", \"Coverage (bp)\", \"Copy Number\", \"% Genome Coverage\", \"Genome Size\", \"TE Family Count\")\n\nwrite.table(pieSum, saveTab, col.names = TRUE, row.names = FALSE, quote = FALSE, sep = \"\\t\")\nwrite.table(familyAbundance, saveFam, col.names = TRUE, row.names = FALSE, quote = FALSE, sep = \"\\t\")\n\n# add a user-readable version of the final table\nsaveKable <- gsub(\".txt\", \".kable\", saveTab)\nkable(pieSum, format = \"pipe\") %>%\n writeLines(saveKable)\n\nsaveKableFam <- gsub(\".txt\", \".kable\", saveFam)\nkable(familyAbundance, format = \"pipe\") %>%\n writeLines(saveKableFam)\n"
},
{
"path": "scripts/autoPie.sh",
"content": "#!/bin/bash\n\n\nusage()\n{\n\techo \"autoPie.sh -i mergedRepeat.bed -t finalRepeatmasker.tbl -p plotName.pdf -o summaryTable.txt -h show this message\"\n}\n\nmain()\n{\n\tgenome_size=$(sed -n '4p' $intbl | rev | cut -f1,1 -d ':' | rev | sed 's/ bp.*//g; s/ //g')\n\tRscript ${SCRIPT_DIR}/autoPie.R $inbed ${inbed%.summary}.gff $genome_size $outplot $sumtab\n}\n\n\n\n\n\n## MAIN ##\n\nwhile getopts i:t:p:o:h option\ndo\n case \"${option}\"\n in\n i) inbed=${OPTARG};;\n t) intbl=${OPTARG};;\n p) outplot=${OPTARG};;\n\t o) sumtab=${OPTARG};;\n h) usage; exit 1;;\n esac\ndone\n\n# step 1 - Get genome size and run Rscript\n\nSCRIPT_DIR=/data/toby/EarlGrey/scripts\nmain\n"
},
{
"path": "scripts/backSwap.py",
"content": "import pandas as pd\nimport sys\nimport csv\n\nargs = sys.argv\n\ninput = args[1]\ndictionary = args[2]\noutput = args[3]\n\norig = dict(csv.reader(open(dictionary), delimiter=\"\\t\"))\ndict = {value:key for key, value in orig.items()}\n\ntable = pd.read_csv(input, names = ['scaf', 'start', 'end', 'repeat', 'score', 'strand'], sep=r'\\s+', header = None)\ntable['scaf'] = table['scaf'].astype(str)\n# table['scaf'].replace(dict, inplace = True)\ntable['scaf'] = table['scaf'].replace(dict)\n\ntable.to_csv(output, index = False, header = False, sep = '\\t') \n"
},
{
"path": "scripts/backSwapGFF.py",
"content": "import pandas as pd\nimport sys\nimport csv\n\nargs = sys.argv\n\ninput = args[1]\ndictionary = args[2]\noutput = args[3]\n\norig = dict(csv.reader(open(dictionary), delimiter=\"\\t\"))\ndict = {value:key for key, value in orig.items()}\n\ntable = pd.read_csv(input, names = ['scaf', 'two', 'three', 'four', 'five', 'six', 'seven', 'eight', 'nine'], sep='\\t', header = None)\ntable['scaf'] = table['scaf'].astype(str)\n# table['scaf'].replace(dict, inplace = True)\ntable['scaf'] = table['scaf'].replace(dict)\n\ntable.to_csv(output, index = False, header = False, sep = '\\t') \n"
},
{
"path": "scripts/bin/LTR_FINDER.x86_64-1.0.7/tRNA.list.txt",
"content": ">Eukarya\nArabidopsis thaliana\tAthal-tRNAs.fa\nCaenorhabditis briggsae (cb25.agp8 July 2002)\tCbrig-tRNAs.fa\nCaenorhabditis elegans (May 2003)\tCeleg-tRNAs.fa\nCanis familiaris (canFam1 July 2004)\tCfam1-tRNAs.fa\nDrosophila melanogaster (Release 4)\tDm-tRNAs.fa\nTakifugu rubripes (v.3.0 Aug. 2002) \tFr1-tRNAs.fa\nGallus gallus (galGal2 Feb 2004)\tGgal2-tRNAs-all.fa\nHomo sapiens (Hg17 May 2004 assembly)\tHg17-tRNAs.fa\nMus musculus (mm6 March 2005)\tMm6-tRNAs.fa\nOryza Sativa (indica 2005)\tOs-tRNAs.fa\nPlasmodium falciparum (Oct 2002)\tPfalc-tRNAs.fa\nPan troglodytes (chimp) (Nov 2003 assembly)\tpanTro1-tRNAs-all.fa\nRattus norvegicus (rn3 June 2003 assembly)\tRn3-tRNAs.fa\nSaccharomyces cerevisiae\tSc-tRNAs.fa\nSchizosaccharomyces pombe (v. July 2004)\tSp-tRNAs.fa\n>Archaea\nAeropyrum pernix\tAero_pern-tRNAs.fa\nArchaeoglobus fulgidus\tArch_fulg-tRNAs.fa\nHaloarcula marismortui ATCC 43049\tHalo_mari_ATCC_43049-tRNAs.fa\nHalobacterium sp\tHalo_sp-tRNAs.fa\nMethanosarcina acetivorans\tMeth_acet-tRNAs.fa\nMethanococcus jannaschii\tMeth_jann-tRNAs.fa\nMethanopyrus kandleri\tMeth_kand-tRNAs.fa\nMethanococcus maripaludis S2\tMeth_mari_S2-tRNAs.fa\nMethanosarcina mazei\tMeth_maze-tRNAs.fa\nMethanobacterium thermoautotrophicum\tMeth_ther-tRNAs.fa\nNanoarchaeum equitans\tNano_equi-tRNAs.fa\nPicrophilus torridus DSM 9790\tPicr_torr_DSM_9790-tRNAs.fa\nPyrococcus abyssi\tPyro_abys-tRNAs.fa\nPyrobaculum aerophilum\tPyro_aero-tRNAs.fa\nPyrococcus furiosus\tPyro_furi-tRNAs.fa\nPyrococcus horikoshii\tPyro_hori-tRNAs.fa\nSulfolobus acidocaldarius DSM 639\tSulf_acid_DSM_639-tRNAs.fa\nSulfolobus solfataricus\tSulf_solf-tRNAs.fa\nSulfolobus tokodaii\tSulf_toko-tRNAs.fa\nThermoplasma acidophilum\tTher_acid-tRNAs.fa\nThermococcus kodakaraensis KOD1\tTher_koda_KOD1-tRNAs.fa\nThermoplasma volcanium\tTher_volc-tRNAs.fa\n>Bacteria\nAcinetobacter sp ADP1\tAcin_sp_ADP1-tRNAs.fa\nAgrobacterium tumefaciens C58 Cereon\tAgro_tume_C58_Cereon-tRNAs.fa\nAgrobacterium tumefaciens C58 UWash\tAgro_tume_C58_UWash-tRNAs.fa\nAnaplasma marginale St Maries\tAnap_marg_St_Maries-tRNAs.fa\nAquifex aeolicus\tAqui_aeol-tRNAs.fa\nAzoarcus sp EbN1\tAzoa_sp_EbN1-tRNAs.fa\nBacillus anthracis A2012\tBaci_anth_A2012-tRNAs.fa\nBacillus anthracis Ames\tBaci_anth_Ames-tRNAs.fa\nBacillus anthracis Ames 0581\tBaci_anth_Ames_0581-tRNAs.fa\nBacillus anthracis str Sterne\tBaci_anth_str_Sterne-tRNAs.fa\nBacillus cereus ATCC14579\tBaci_cere_ATCC14579-tRNAs.fa\nBacillus cereus ATCC 10987\tBaci_cere_ATCC_10987-tRNAs.fa\nBacillus cereus ZK\tBaci_cere_ZK-tRNAs.fa\nBacillus clausii KSM-K16\tBaci_clau_KSM-K16-tRNAs.fa\nBacillus halodurans\tBaci_halo-tRNAs.fa\nBacillus licheniformis ATCC 14580\tBaci_lich_ATCC_14580-tRNAs.fa\nBacillus licheniformis DSM 13\tBaci_lich_DSM_13-tRNAs.fa\nBacillus subtilis\tBaci_subt-tRNAs.fa\nBacillus thuringiensis konkukian\tBaci_thur_konkukian-tRNAs.fa\nBacteroides fragilis NCTC 9434\tBact_frag_NCTC_9434-tRNAs.fa\nBacteroides fragilis YCH46\tBact_frag_YCH46-tRNAs.fa\nBacteroides thetaiotaomicron VPI-5482\tBact_thet_VPI-5482-tRNAs.fa\nBartonella henselae Houston-1\tBart_hens_Houston-1-tRNAs.fa\nBartonella quintana Toulouse\tBart_quin_Toulouse-tRNAs.fa\nBdellovibrio bacteriovorus\tBdel_bact-tRNAs.fa\nBifidobacterium longum\tBifi_long-tRNAs.fa\nBlochmannia floridanus\tBloc_flor-tRNAs.fa\nBordetella bronchiseptica\tBord_bron-tRNAs.fa\nBordetella parapertussis\tBord_para-tRNAs.fa\nBordetella pertussis\tBord_pert-tRNAs.fa\nBorrelia burgdorferi\tBorr_burg-tRNAs.fa\nBorrelia garinii PBi\tBorr_gari_PBi-tRNAs.fa\nBradyrhizobium japonicum\tBrad_japo-tRNAs.fa\nBrucella abortus 9-941\tBruc_abor_9-941-tRNAs.fa\nBrucella melitensis\tBruc_meli-tRNAs.fa\nBrucella suis 1330\tBruc_suis_1330-tRNAs.fa\nBuchnera aphidicola\tBuch_aphi-tRNAs.fa\nBuchnera aphidicola Sg\tBuch_aphi_Sg-tRNAs.fa\nBuchnera sp\tBuch_sp-tRNAs.fa\nBurkholderia mallei ATCC 23344\tBurk_mall_ATCC_23344-tRNAs.fa\nBurkholderia pseudomallei K96243\tBurk_pseu_K96243-tRNAs.fa\nCampylobacter jejuni\tCamp_jeju-tRNAs.fa\nCampylobacter jejuni RM1221\tCamp_jeju_RM1221-tRNAs.fa\nCaulobacter crescentus\tCaul_cres-tRNAs.fa\nChlamydophila abortus S26 3\tChla_abor_S26_3-tRNAs.fa\nChlamydophila caviae\tChla_cavi-tRNAs.fa\nChlamydia muridarum\tChla_muri-tRNAs.fa\nChlamydophila pneumoniae AR39\tChla_pneu_AR39-tRNAs.fa\nChlamydophila pneumoniae CWL029\tChla_pneu_CWL029-tRNAs.fa\nChlamydophila pneumoniae J138\tChla_pneu_J138-tRNAs.fa\nChlamydophila pneumoniae TW 183\tChla_pneu_TW_183-tRNAs.fa\nChlamydia trachomatis\tChla_trac-tRNAs.fa\nChlorobium tepidum TLS\tChlo_tepi_TLS-tRNAs.fa\nChromobacterium violaceum\tChro_viol-tRNAs.fa\nClostridium acetobutylicum\tClos_acet-tRNAs.fa\nClostridium perfringens\tClos_perf-tRNAs.fa\nClostridium tetani E88\tClos_teta_E88-tRNAs.fa\nCorynebacterium diphtheriae\tCory_diph-tRNAs.fa\nCorynebacterium efficiens YS-314\tCory_effi_YS-314-tRNAs.fa\nCorynebacterium glutamicum\tCory_glut-tRNAs.fa\nCorynebacterium glutamicum ATCC 13032 Bielefeld\tCory_glut_ATCC_13032_Bielefeld-tRNAs.fa\nCorynebacterium glutamicum ATCC 13032 Kitasato\tCory_glut_ATCC_13032_Kitasato-tRNAs.fa\nCoxiella burnetii\tCoxi_burn-tRNAs.fa\nDehalococcoides ethenogenes 195\tDeha_ethe_195-tRNAs.fa\nDeinococcus radiodurans\tDein_radi-tRNAs.fa\nDesulfotalea psychrophila LSv54\tDesu_psyc_LSv54-tRNAs.fa\nDesulfovibrio vulgaris Hildenborough\tDesu_vulg_Hildenborough-tRNAs.fa\nEhrlichia ruminantium Gardel\tEhrl_rumi_Gardel-tRNAs.fa\nEhrlichia ruminantium Welgevonden\tEhrl_rumi_Welgevonden-tRNAs.fa\nEhrlichia ruminantium str. Welgevonden\tEhrl_rumi_str._Welgevonden-tRNAs.fa\nEnterococcus faecalis V583\tEnte_faec_V583-tRNAs.fa\nErwinia carotovora atroseptica SCRI1043\tErwi_caro_atroseptica_SCRI1043-tRNAs.fa\nEscherichia coli CFT073\tEsch_coli_CFT073-tRNAs.fa\nEscherichia coli K12\tEsch_coli_K12-tRNAs.fa\nEscherichia coli O157H7\tEsch_coli_O157H7-tRNAs.fa\nEscherichia coli O157H7 EDL933\tEsch_coli_O157H7_EDL933-tRNAs.fa\nFrancisella tularensis tularensis\tFran_tula_tularensis-tRNAs.fa\nFusobacterium nucleatum\tFuso_nucl-tRNAs.fa\nGeobacillus kaustophilus HTA426\tGeob_kaus_HTA426-tRNAs.fa\nGeobacter sulfurreducens\tGeob_sulf-tRNAs.fa\nGloeobacter violaceus\tGloe_viol-tRNAs.fa\nGluconobacter oxydans 621H\tGluc_oxyd_621H-tRNAs.fa\nHaemophilus ducreyi 35000HP\tHaem_ducr_35000HP-tRNAs.fa\nHaemophilus influenzae\tHaem_infl-tRNAs.fa\nHaemophilus influenzae 86 028NP\tHaem_infl_86_028NP-tRNAs.fa\nHelicobacter hepaticus\tHeli_hepa-tRNAs.fa\nHelicobacter pylori 26695\tHeli_pylo_26695-tRNAs.fa\nHelicobacter pylori J99\tHeli_pylo_J99-tRNAs.fa\nIdiomarina loihiensis L2TR\tIdio_loih_L2TR-tRNAs.fa\nLactobacillus acidophilus NCFM\tLact_acid_NCFM-tRNAs.fa\nLactobacillus johnsonii NCC 533\tLact_john_NCC_533-tRNAs.fa\nLactococcus lactis\tLact_lact-tRNAs.fa\nLactobacillus plantarum\tLact_plan-tRNAs.fa\nLegionella pneumophila Lens\tLegi_pneu_Lens-tRNAs.fa\nLegionella pneumophila Paris\tLegi_pneu_Paris-tRNAs.fa\nLegionella pneumophila Philadelphia 1\tLegi_pneu_Philadelphia_1-tRNAs.fa\nLeifsonia xyli xyli CTCB0\tLeif_xyli_xyli_CTCB0-tRNAs.fa\nLeptospira interrogans\tLept_inte-tRNAs.fa\nLeptospira interrogans serovar Copenhageni\tLept_inte_serovar_Copenhageni-tRNAs.fa\nLeptospira interrogans serovar Lai\tLept_inte_serovar_Lai-tRNAs.fa\nListeria innocua\tList_inno-tRNAs.fa\nListeria monocytogenes\tList_mono-tRNAs.fa\nListeria monocytogenes 4b F2365\tList_mono_4b_F2365-tRNAs.fa\nMannheimia succiniciproducens MBEL55E\tMann_succ_MBEL55E-tRNAs.fa\nMesoplasma florum L1\tMeso_flor_L1-tRNAs.fa\nMesorhizobium loti\tMeso_loti-tRNAs.fa\nMethylococcus capsulatus Bath\tMeth_caps_Bath-tRNAs.fa\nMycobacterium avium paratuberculosis\tMyco_aviu_paratuberculosis-tRNAs.fa\nMycobacterium bovis\tMyco_bovi-tRNAs.fa\nMycoplasma gallisepticum\tMyco_gall-tRNAs.fa\nMycoplasma genitalium\tMyco_geni-tRNAs.fa\nMycoplasma hyopneumoniae 232\tMyco_hyop_232-tRNAs.fa\nMycobacterium leprae\tMyco_lepr-tRNAs.fa\nMycoplasma mobile 163K\tMyco_mobi_163K-tRNAs.fa\nMycoplasma mycoides\tMyco_myco-tRNAs.fa\nMycoplasma penetrans\tMyco_pene-tRNAs.fa\nMycoplasma pneumoniae\tMyco_pneu-tRNAs.fa\nMycoplasma pulmonis\tMyco_pulm-tRNAs.fa\nMycobacterium tuberculosis CDC1551\tMyco_tube_CDC1551-tRNAs.fa\nMycobacterium tuberculosis H37Rv\tMyco_tube_H37Rv-tRNAs.fa\nNeisseria gonorrhoeae FA 1090\tNeis_gono_FA_1090-tRNAs.fa\nNeisseria meningitidis MC58\tNeis_meni_MC58-tRNAs.fa\nNeisseria meningitidis Z2491\tNeis_meni_Z2491-tRNAs.fa\nNitrosomonas europaea\tNitr_euro-tRNAs.fa\nNocardia farcinica IFM10152\tNoca_farc_IFM10152-tRNAs.fa\nNostoc sp\tNost_sp-tRNAs.fa\nOceanobacillus iheyensis\tOcea_ihey-tRNAs.fa\nOnion yellows phytoplasma\tOnio_yell_phytoplasma-tRNAs.fa\nParachlamydia sp UWE25\tPara_sp_UWE25-tRNAs.fa\nPasteurella multocida\tPast_mult-tRNAs.fa\nPhotorhabdus luminescens\tPhot_lumi-tRNAs.fa\nPhotobacterium profundum SS9\tPhot_prof_SS9-tRNAs.fa\nPirellula sp\tPire_sp-tRNAs.fa\nPorphyromonas gingivalis W83\tPorp_ging_W83-tRNAs.fa\nProchlorococcus marinus CCMP1375\tProc_mari_CCMP1375-tRNAs.fa\nProchlorococcus marinus MED4\tProc_mari_MED4-tRNAs.fa\nProchlorococcus marinus MIT9313\tProc_mari_MIT9313-tRNAs.fa\nPropionibacterium acnes KPA171202\tProp_acne_KPA171202-tRNAs.fa\nPseudomonas aeruginosa\tPseu_aeru-tRNAs.fa\nPseudomonas fluorescens Pf-5\tPseu_fluo_Pf-5-tRNAs.fa\nPseudomonas putida KT2440\tPseu_puti_KT2440-tRNAs.fa\nPseudomonas syringae\tPseu_syri-tRNAs.fa\nPseudomonas syringae pv B728a\tPseu_syri_pv_B728a-tRNAs.fa\nRalstonia solanacearum\tRals_sola-tRNAs.fa\nRhodopseudomonas palustris CGA009\tRhod_palu_CGA009-tRNAs.fa\nRickettsia conorii\tRick_cono-tRNAs.fa\nRickettsia felis URRWXCal2\tRick_feli_URRWXCal2-tRNAs.fa\nRickettsia prowazekii\tRick_prow-tRNAs.fa\nRickettsia typhi wilmington\tRick_typh_wilmington-tRNAs.fa\nSalmonella enterica Choleraesuis\tSalm_ente_Choleraesuis-tRNAs.fa\nSalmonella enterica Paratypi ATCC 9150\tSalm_ente_Paratypi_ATCC_9150-tRNAs.fa\nSalmonella typhi\tSalm_typh-tRNAs.fa\nSalmonella typhimurium LT2\tSalm_typh_LT2-tRNAs.fa\nSalmonella typhi Ty2\tSalm_typh_Ty2-tRNAs.fa\nShewanella oneidensis\tShew_onei-tRNAs.fa\nShigella flexneri 2a\tShig_flex_2a-tRNAs.fa\nShigella flexneri 2a 2457T\tShig_flex_2a_2457T-tRNAs.fa\nSilicibacter pomeroyi DSS-3\tSili_pome_DSS-3-tRNAs.fa\nSinorhizobium meliloti\tSino_meli-tRNAs.fa\nStaphylococcus aureus COL\tStap_aure_COL-tRNAs.fa\nStaphylococcus aureus MW2\tStap_aure_MW2-tRNAs.fa\nStaphylococcus aureus Mu50\tStap_aure_Mu50-tRNAs.fa\nStaphylococcus aureus N315\tStap_aure_N315-tRNAs.fa\nStaphylococcus aureus aureus MRSA252\tStap_aure_aureus_MRSA252-tRNAs.fa\nStaphylococcus aureus aureus MSSA476\tStap_aure_aureus_MSSA476-tRNAs.fa\nStaphylococcus epidermidis ATCC 12228\tStap_epid_ATCC_12228-tRNAs.fa\nStaphylococcus epidermidis RP62A\tStap_epid_RP62A-tRNAs.fa\nStaphylococcus haemolyticus\tStap_haem-tRNAs.fa\nStreptococcus agalactiae 2603\tStre_agal_2603-tRNAs.fa\nStreptococcus agalactiae NEM316\tStre_agal_NEM316-tRNAs.fa\nStreptomyces avermitilis\tStre_aver-tRNAs.fa\nStreptomyces coelicolor\tStre_coel-tRNAs.fa\nStreptococcus mutans\tStre_muta-tRNAs.fa\nStreptococcus pneumoniae R6\tStre_pneu_R6-tRNAs.fa\nStreptococcus pneumoniae TIGR4\tStre_pneu_TIGR4-tRNAs.fa\nStreptococcus pyogenes\tStre_pyog-tRNAs.fa\nStreptococcus pyogenes M1 GAS\tStre_pyog_M1_GAS-tRNAs.fa\nStreptococcus pyogenes M49 591\tStre_pyog_M49_591-tRNAs.fa\nStreptococcus pyogenes MGAS10394\tStre_pyog_MGAS10394-tRNAs.fa\nStreptococcus pyogenes MGAS315\tStre_pyog_MGAS315-tRNAs.fa\nStreptococcus pyogenes MGAS8232\tStre_pyog_MGAS8232-tRNAs.fa\nStreptococcus pyogenes SSI-1\tStre_pyog_SSI-1-tRNAs.fa\nStreptococcus thermophilus CNRZ1066\tStre_ther_CNRZ1066-tRNAs.fa\nStreptococcus thermophilus LMG 18311\tStre_ther_LMG_18311-tRNAs.fa\nSymbiobacterium thermophilum IAM14863\tSymb_ther_IAM14863-tRNAs.fa\nSynechocystis PCC6803\tSyne_PCC6-tRNAs.fa\nSynechococcus elongatus PCC 6301\tSyne_elon_PCC_6301-tRNAs.fa\nSynechococcus sp WH8102\tSyne_sp_WH8102-tRNAs.fa\nThermosynechococcus elongatus\tTher_elon-tRNAs.fa\nThermotoga maritima\tTher_mari-tRNAs.fa\nThermoanaerobacter tengcongensis\tTher_teng-tRNAs.fa\nThermus thermophilus HB27\tTher_ther_HB27-tRNAs.fa\nThermus thermophilus HB8\tTher_ther_HB8-tRNAs.fa\nTreponema denticola ATCC 35405\tTrep_dent_ATCC_35405-tRNAs.fa\nTreponema pallidum\tTrep_pall-tRNAs.fa\nTropheryma whipplei TW08 27\tTrop_whip_TW08_27-tRNAs.fa\nTropheryma whipplei Twist\tTrop_whip_Twist-tRNAs.fa\nUreaplasma urealyticum\tUrea_urea-tRNAs.fa\nVibrio cholerae\tVibr_chol-tRNAs.fa\nVibrio fischeri ES114\tVibr_fisc_ES114-tRNAs.fa\nVibrio parahaemolyticus\tVibr_para-tRNAs.fa\nVibrio vulnificus CMCP6\tVibr_vuln_CMCP6-tRNAs.fa\nVibrio vulnificus YJ016\tVibr_vuln_YJ016-tRNAs.fa\nWigglesworthia brevipalpis\tWigg_brev-tRNAs.fa\nWolbachia endosymbiont of Brugia malayi TRS\tWolb_endo_of_Brugia_malayi_TRS-tRNAs.fa\nWolbachia endosymbiont of Drosophila melanogaster\tWolb_endo_of_Drosophila_melanogaster-tRNAs.fa\nWolinella succinogenes\tWoli_succ-tRNAs.fa\nXanthomonas campestris\tXant_camp-tRNAs.fa\nXanthomonas campestris 8004\tXant_camp_8004-tRNAs.fa\nXanthomonas citri\tXant_citr-tRNAs.fa\nXanthomonas oryzae KACC10331\tXant_oryz_KACC10331-tRNAs.fa\nXylella fastidiosa\tXyle_fast-tRNAs.fa\nXylella fastidiosa Temecula1\tXyle_fast_Temecula1-tRNAs.fa\nYersinia pestis CO92\tYers_pest_CO92-tRNAs.fa\nYersinia pestis KIM\tYers_pest_KIM-tRNAs.fa\nYersinia pestis biovar Mediaevails\tYers_pest_biovar_Mediaevails-tRNAs.fa\nYersinia pseudotuberculosis IP32953\tYers_pseu_IP32953-tRNAs.fa\nZymomonas mobilis ZM4\tZymo_mobi_ZM4-tRNAs.fa\n"
},
{
"path": "scripts/bin/cut.pl",
"content": "#!usr/bin/perl -w\nuse strict;\n#usage: perl cut.pl seq [-s|-S|-l int]\n\nmy $length=5000000; #length of a sequence\nmy $separate=1; #1 for 1 sequence per file, 0 for all sequence in 1 files.\nmy $size=0; #no size control; if set to 10000; program will output sequence files every $length base (roundup to single sequence)\nmy $i=0;\nforeach (@ARGV){\n\t$separate=1 if $_=~/-s|Separate/;\n\t$length=$ARGV[$i+1] if $_=~/-l|Length/i;\n\t$size=1 if /-S|size/;\n\t$i++;\n\t}\nopen Seq, \"<$ARGV[0]\" or die $!;\nopen List, \">$ARGV[0].list\" if $separate==1;\n\n$/=\"\\n>\";\nmy $accu_len=0;\nmy $k=1;\n\topen File, \">$ARGV[0]_sub$k\" if $size==1;\nwhile (){\n\ts/>//g;\n\tchomp;\n\tnext if /^\\s+$/;\n\tmy ($id, $seq)=(split /\\n/, $_, 2);\n\t$seq=~s/\\s+//g;\n\tmy $j=1;\n\tif ($size==0){\n\tmy $len=$length;\n\tfor (my $i=0; ; $i+=$len){\n\t\t$len=length($seq)-$i if length($seq)<$i+$len;\n\t\tmy $seq_sub=substr($seq, $i, $len);\n\t\tlast if length($seq_sub)<1;\n\t\tif ($separate==0){\n\t\t\tprint \">${id}_sub$j\\n$seq_sub\\n\";\n\t\t\t}\n\t\telsif ($separate==1){\n\t\t\tprint List \"${id}_sub$j\\n\";\n\t\t\topen File, \">${id}_sub$j\" or die $!;\n\t\t\tprint File \">${id}_sub$j\\n$seq_sub\\n\";\n\t\t\tclose File;\n\t\t\t}\n\t\t$j++;\n\t\t}\n\t\t} else {\n\t\tprint File \">$id\\n$seq\\n\";\n\t\t$accu_len+=length $seq;\n\t\tif ($accu_len>$length){\nprint \"$accu_len\\n\";\n\t\t\t$k++;\n\t\t\t$accu_len=0;\n\t\t\tclose File;\n\t\t\topen File, \">$ARGV[0]_sub$k\" or die $!;\n#\t\t\tprint File \">$id\\n$seq\\n\";\n\t\t\t}\n\t\t}\n\n\t}\n#system \"rm $ARGV[0]_sub$k\" if $size==1;\nclose Seq;\n\n"
},
{
"path": "scripts/divergenceCalc/divergence_calc.py",
"content": "import os\nfrom os.path import exists, getsize\nimport sys\nimport argparse\nimport numpy as np\nimport pandas as pd\nimport multiprocessing\nimport pybedtools\nimport subprocess\nimport shlex\nimport shutil\nfrom Bio import AlignIO, SeqIO\nfrom math import log, sqrt\nfrom functools import partial\nfrom time import time\nfrom re import sub\n\nparser = argparse.ArgumentParser()\nparser.add_argument('-l', '--repeat_library', type=str, required=True,\n help='repeat_library')\nparser.add_argument('-i', '--in_gff', type=str, required=True,\n help='Path to gff')\nparser.add_argument('-g', '--genome', type=str, required=True,\n help='Path to genome')\nparser.add_argument('-o', '--out_gff', type=str, required=True,\n help='Output gff')\nparser.add_argument('-tmp', '--temp_dir', type=str, default='tmp/',\n help='Temporary directory')\nparser.add_argument('-t', '--cores', type=int, default=4,\n help='Number of cores')\nparser.add_argument('-k', '--timeout', type=int, default=30,\n help='Seconds after which matcher will be cancelled and repeat treated as unalignable')\n\nargs = parser.parse_args()\n\ndef file_check(repeat_library, in_gff, genome, out_gff, temp_dir):\n if(exists(repeat_library) == False or exists(in_gff) == False or exists(genome) == False):\n sys.exit('Files not found. Requires the repeat library, path to the genome, and path to gff containing coordinates and corresponding repeat files')\n if(exists(genome+\".fai\") == False):\n print(\"Indexing genome\")\n subprocess.run([\"samtools\",\"faidx\",genome], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)\n if(exists(temp_dir) == False):\n os.mkdir(temp_dir)\n if(exists(os.path.join(temp_dir, \"qseqs\")) == False):\n os.mkdir(os.path.join(temp_dir, \"qseqs\"))\n if(exists(os.path.join(temp_dir, \"split_library\")) == False):\n os.mkdir(os.path.join(temp_dir, \"split_library\"))\n \n\ndef splitter(in_seq, temp_dir):\n with open(in_seq, 'r') as handle:\n for record in SeqIO.parse(handle, \"fasta\"):\n repeat_name = record.name.split(sep=\"#\")[0]\n repeat_name = repeat_name.lower()\n file_name = os.path.join(temp_dir, \"split_library\", repeat_name + \".fasta\")\n SeqIO.write(record, file_name, \"fasta-2line\")\n\ndef parse_gff(in_gff):\n gff = pd.read_table(in_gff, header = None, names=['seqnames', 'tool', 'repeat_class', 'start', 'end', 'score', 'strand', 'phase', 'metadata'])\n simple_gff = gff[gff['repeat_class'].str.contains('Simple_repeat|Satellite|Low_complexity')].reset_index()\n gff = gff[~gff['repeat_class'].str.contains('Simple_repeat|Satellite|Low_complexity')].reset_index()\n other_gff = gff[~gff['tool'].str.contains('Earl_Grey|RepeatMasker')].reset_index()\n gff = gff[gff['tool'].str.contains('Earl_Grey|RepeatMasker')].reset_index()\n gff['metadata_tmp'] = gff['metadata'].str.replace(';TSTART.*', '', regex=True)\n gff[['repeat_id', 'repeat_family']] = gff['metadata_tmp'].str.split(';', n=2, expand=True)\n gff = gff.drop(columns = ['metadata_tmp', 'repeat_id'])\n gff['repeat_family'] = gff['repeat_family'].str.replace('NAME=', '', regex=True)\n gff['repeat_family'] = gff['repeat_family'].str.lower()\n return(gff, simple_gff, other_gff)\n\ndef file_name_generator():\n import random\n import string\n file_name = ''.join(random.sample(string.ascii_letters, 12))+'.tmp'\n return(file_name)\n\ndef Kimura80(qseq, sseq):\n \"\"\"\n Calculations adapted from https://github.com/kgori/python_tools_on_github/blob/master/pairwise_distances.py\n \"\"\"\n # define transitions, transversions, matches\n transitions = [ \"AG\", \"GA\", \"CT\", \"TC\"]\n transversions = [ \"AC\", \"CA\", \"AT\", \"TA\",\n \"GC\", \"CG\", \"GT\", \"TG\" ]\n matches = [ \"AA\", \"GG\", \"CC\", \"TT\"]\n # set counters to 0\n m,ts,tv=0,0,0\n # count transitions, transversions, matches\n for i, j in zip(qseq, sseq):\n if i+j in matches: m+=1\n if i+j in transitions: ts+=1\n if i+j in transversions: tv+=1\n # count number of bp which align (excludes gaps, Ns)\n aln_len = m + ts + tv\n \n if aln_len != 0:\n # calculate p and q \n p = ts/aln_len\n q = tv/aln_len\n \n # calculate Kimura distance\n Kimura_dist = -0.5 * log((1 - 2*p - q) * sqrt( 1 - 2*q ))\n else:\n Kimura_dist = \"NA\"\n \n return(Kimura_dist)\n\ndef outer_func(genome_path, temp_dir, timeoutSeconds, chunk_path):\n # Set pybedtools temp directory within this worker (required when using forkserver).\n pybedtools.set_tempdir(os.path.join(temp_dir, 'pybedtools'))\n # Load the chunk from its temp file and delete it immediately to reclaim disk space.\n # index_col=0 restores the original GFF row indices, which are used as unique\n # per-row filenames in qseqs/. Without this, every chunk restarts at 0 and\n # concurrent workers collide on the same paths.\n gff = pd.read_csv(chunk_path, sep=\"\\t\", index_col=0)\n os.remove(chunk_path)\n generated_name = file_name_generator()\n holder_file_name = os.path.join(temp_dir, generated_name)\n failed_file_name = os.path.join(temp_dir, \"failed_\" + generated_name)\n row_counter = 0\n with open(holder_file_name, 'w') as tmp_out:\n header = list(gff.columns.values)[1:] + [\"Kimura\"]\n header = \"\\t\".join(header)+\"\\n\"\n tmp_out.write(header)\n for row in gff.iterrows():\n # Set index\n idx = row[0]\n row_counter += 1\n # Periodically release pybedtools temp files to prevent gradual memory growth.\n if row_counter % 500 == 0:\n pybedtools.cleanup(remove_all=False)\n # Set scaffold, coordinates, strand, repeat family\n seqnames, start, end, strand, repeat_family = str(row[1]['seqnames']), str(row[1]['start'] - 1), str(row[1]['end']), str(row[1]['strand']), str(row[1]['repeat_family'])\n # Create BED string for BEDtools\n bed_str = \" \".join([seqnames, start, end, \".\", \".\", strand])\n # Set path for query sequence\n query_path = os.path.join(temp_dir, \"qseqs\", str(idx))\n # Create bedtools command and getfasta\n a=pybedtools.BedTool(bed_str, from_string=True)\n try:\n a = a.sequence(fi=genome_path, fo=query_path, s=True)\n except: # Occasionally a samtools error occurs, this overcomes this\n with open(failed_file_name, \"a\") as failed_file:\n failed_file.write(seqnames+\":\"+start+\"-\"+end+\"_\"+strand+\"_\"+repeat_family+\"\\n\")\n if exists(query_path) is True and getsize(query_path) > 0:\n # Set path to subject sequence\n subject_path = os.path.join(temp_dir, \"split_library\", repeat_family + \".fasta\")\n # Run matcher, with timeout exception\n test_command = shlex.split(\"matcher \"+query_path+\" \"+subject_path+\" -outfile \"+query_path+\".matcher -aformat fasta\")\n # Run test and kill if it takes more than 10 seconds\n alignment_p = subprocess.Popen(test_command, stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)\n try:\n alignment_p.wait(timeoutSeconds)\n except subprocess.TimeoutExpired:\n # if matcher fails to complete before timeout, kill and move on\n with open(failed_file_name, \"a\") as failed_file:\n failed_file.write(seqnames+\":\"+start+\"-\"+end+\"_\"+strand+\"_\"+repeat_family+\"\\n\")\n alignment_p.kill()\n\n if exists(query_path+\".matcher\") is False or getsize(query_path+\".matcher\") == 0:\n # If no alignment is possible, set distances to NA and alignment length to 0\n Kdist = \"NA\"\n if exists(query_path):\n os.remove(query_path)\n if exists(query_path+\".matcher\") is True:\n os.remove(query_path+\".matcher\")\n else:\n # Read in alignments\n aln = list(SeqIO.parse(query_path+\".matcher\", 'fasta'))\n ref_seq, gen_seq = str(aln[0].seq).upper(), str(aln[1].seq).upper()\n # Check ref and genome sequence are same length, set Kdist to NA if not\n if len(ref_seq) == len(gen_seq):\n # Calculate distances based on model\n Kdist = Kimura80(ref_seq, gen_seq)\n # Convert numbers to strings\n if Kdist != \"NA\":\n Kdist = str(round(Kdist, 4))\n else:\n Kdist = \"NA\"\n # Delete temporary files\n os.remove(query_path+\".matcher\")\n if exists(query_path):\n os.remove(query_path)\n # Make line for temporary file and write to file\n tmp_holder = row[1].to_list()[1:]\n tmp_holder = \"\\t\".join(str(x) for x in tmp_holder)+\"\\t\"+Kdist+\"\\n\"\n tmp_out.write(tmp_holder)\n\n return(holder_file_name)\n\ndef tmp_out_parser(file_list, simple_gff, other_gff):\n # Loop through results \n gff=pd.DataFrame()\n for file in file_list:\n # read in gff\n in_gff = pd.read_csv(file, sep = \"\\t\")\n # concatenate gff\n gff = pd.concat([gff, in_gff], ignore_index=True)\n # Convert numbers to strings for concatenation\n gff['Kimura'] = gff['Kimura'].astype(str)\n # Convert new data onto metadata\n gff['metadata'] = gff['metadata'] + \";KIMURA80=\" + gff['Kimura']\n # Remove unnecessary rows\n gff = gff.drop(columns = ['Kimura', 'repeat_family'])\n # Combine columns, sort and drop unneccessary columns\n gff = pd.concat([gff, simple_gff, other_gff], ignore_index=True)\n gff = gff.drop(columns = ['level_0', 'index'])\n gff = gff.sort_values(by=['seqnames', 'start'])\n gff = gff.reset_index()\n\n return(gff)\n\nif __name__ == \"__main__\":\n # Use forkserver so workers start from a clean process rather than inheriting\n # a fork-copy of the parent address space (which holds the full GFF DataFrame).\n # This is the primary mechanism for reducing peak RAM at high thread counts.\n if multiprocessing.get_start_method(allow_none=True) is None:\n try:\n multiprocessing.set_start_method('forkserver')\n except (RuntimeError, ValueError):\n # Fall back to the existing/default start method if forkserver is unavailable\n # or if a start method has already been set by the environment.\n pass\n\n start_time = time()\n\n # check files exist\n file_check(args.repeat_library, args.in_gff, args.genome, args.out_gff, args.temp_dir)\n \n # split library file\n print(\"Splitting repeat library\")\n splitter(args.repeat_library, args.temp_dir)\n\n # read in gff and take head\n print(\"Reading in gff\")\n in_gff, simple_gff, other_gff = parse_gff(args.in_gff)\n \n # create as many processes as instructed cores\n num_processes = args.cores\n\n # break into exactly num_processes chunks, distributing any remainder evenly\n chunks = [in_gff.iloc[idx] for idx in np.array_split(range(len(in_gff)), num_processes)]\n\n # Write chunks to temp TSV files so the parent DataFrame can be freed before workers\n # are created. Workers read from disk rather than receiving pickled DataFrames via IPC.\n print(\"Writing chunks to disk\")\n chunk_files = []\n for i, chunk in enumerate(chunks):\n chunk_path = os.path.join(args.temp_dir, f\"chunk_{i}.tsv\")\n chunk.to_csv(chunk_path, sep=\"\\t\", index=True)\n chunk_files.append(chunk_path)\n\n # Free the main GFF DataFrame and chunk list from the parent process before the pool\n # is created. With forkserver this is already avoided, but freeing here also reduces\n # parent RSS during the pool run, which matters on memory-constrained machines.\n del chunks\n del in_gff\n\n # set pybedtools temp path (also set per-worker inside outer_func for forkserver)\n try:\n os.mkdir(os.path.join(args.temp_dir, \"pybedtools\"))\n except FileExistsError:\n pass\n pybedtools.set_tempdir(os.path.join(args.temp_dir, 'pybedtools'))\n\n print(\"Starting calculations\") \n # Perform calculations in parallel. maxtasksperchild=1 restarts each worker after\n # processing one chunk, releasing any lingering pybedtools handles or cached objects.\n func = partial(outer_func, args.genome, args.temp_dir, args.timeout)\n pool = multiprocessing.Pool(processes=num_processes, maxtasksperchild=1)\n results = list(pool.imap_unordered(func, chunk_files))\n pool.close()\n pool.join()\n print(\"Finished calculations\") \n\n # Read in temp files, fix metadata, add simple repeats back, and sort\n calc_gff = tmp_out_parser(results, simple_gff, other_gff)\n \n # remove first column and write to file\n calc_gff.drop(columns = ['index']).to_csv(args.out_gff, sep = \"\\t\", header = False, index=False)\n\n # print run time for number of rows\n run_time = time() - start_time\n print(\"Total run time for \", len(calc_gff), \" rows was \", run_time, \" seconds\")\n\n # Delete folder of split library\n shutil.rmtree(os.path.join(args.temp_dir, \"split_library\"), ignore_errors=True)\n"
},
{
"path": "scripts/divergenceCalc/divergence_plot.R",
"content": "\nlibrary(optparse)\n\noption_list <- list(\n make_option(c(\"-s\", \"--species_name\"), default=NA, type = \"character\",\n help=\"Species name (required)\"),\n make_option(c(\"-g\", \"--in_gff\"), default=NA, type = \"character\",\n help=\"GFF with Kimura distances (required)\"),\n make_option(c(\"-o\", \"--out_directory\"), default=NA, type = \"character\",\n help=\"Directory to write plots to (required)\"),\n make_option(c(\"-f\", \"--axis_flip\"), action = \"store_true\",\n default=TRUE, type = \"logical\", help=\"Flip x-axis on plots\")\n)\n\nopt <- parse_args(OptionParser(option_list=option_list))\n\n# Check variables are set\nif(is.na(opt$species_name)){\n stop(\"Species name must be supplied\")\n}\nif(is.na(opt$in_gff)){\n stop(\"Path to input gff must be supplied\")\n}\nif(is.na(opt$out_directory)){\n stop(\"Path to output directory must be supplied\")\n}\n\n# Check output folder exists or create\nif(!dir.exists(opt$out_directory)){\n message(\"Creating output directory\")\n dir.create(opt$out_directory, recursive = TRUE)\n}\nsuppressPackageStartupMessages(library(tidyverse))\nsuppressPackageStartupMessages(library(plyranges))\nsuppressPackageStartupMessages(library(viridis))\nsuppressPackageStartupMessages(library(cowplot))\nsuppressPackageStartupMessages(library(ggtext))\n\n# set variables\nspecies_name=opt$species_name\nin_gff=opt$in_gff\nout_directory=opt$out_directory\n\n# Created plot title\nplot_title <- paste0(\"Repeat landscape of *\", gsub(\"_\", \" \", species_name), \"*\")\ntitle_plot <- ggplot() + labs(title = plot_title) + theme(plot.title = element_markdown(hjust = 0.5)) + theme(panel.background = element_blank())\n\n# Read in data\ndivergence_eg_gff <- read_gff(in_gff)\n\n# Breakdown classification of repeats\ndivergence_eg_tes_gff <- divergence_eg_gff %>%\n dplyr::mutate(subclass = sub(\"/.*\", \"\", type),\n superfamily = sub(\"-.*\", \"\", sub(\".*/\", \"\", type)))\n\n# Fix Penelopes\ndivergence_eg_tes_gff <- divergence_eg_tes_gff %>%\n dplyr::mutate(subclass = ifelse(superfamily == \"Penelope\", \"PLE\", subclass)) %>%\n dplyr::mutate(subclass = ifelse(subclass %in% c(\"DNA\", \"LINE\", \"LTR\", \"PLE\", \"RC\", \"SINE\", \"Unknown\"), subclass, \"Other\")) %>%\n dplyr::mutate(named_subclass = case_when(subclass == \"DNA\" ~ \"DNA Transposon\",\n subclass == \"LTR\" ~ \"LTR Retrotransposon\",\n subclass == \"PLE\" ~ \"Penelope\",\n subclass == \"RC\" ~ \"Rolling Circle\",\n .default = subclass))\n\n# Create summary of families\nsummary_table <- divergence_eg_tes_gff %>%\n as_tibble %>%\n group_by(NAME) %>%\n dplyr::select(width, KIMURA80, subclass, superfamily, NAME) %>%\n mutate(total_bp = sum(width),\n mean_width = round(mean(width), digits = 2),\n min_width = round(min(width), digits = 2),\n max_width = round(max(width), digits = 2),\n sd_width = round(sd(width), digits = 2),\n div = as.numeric(KIMURA80),\n mean_div = round(mean(div), digits = 2),\n min_div = round(min(div), digits = 2),\n max_div = round(max(div), digits = 2),\n sd_div = round(sd(div), digits = 2)\n ) %>%\n dplyr::select(-width, -div, -KIMURA80) %>%\n base::unique() %>%\n dplyr::arrange(subclass, superfamily, NAME) %>%\n dplyr::rename(family = NAME)\n\n# Save summary to file\nreadr::write_tsv(x = summary_table, file = paste0(out_directory, \"/\", species_name, \"_summary_table.tsv\"))\n\n# Sum lengths to create data for plots (remove subclasses not in standard set and repeats which Kimura was not calculated for)\ndivergence_eg_tes_rounded_for_plot <- divergence_eg_tes_gff %>%\n filter(!is.na(KIMURA80)) %>%\n mutate(KIMURA80 = as.numeric(KIMURA80)) %>%\n dplyr::filter(KIMURA80 <= 0.5) %>%\n as_tibble() %>%\n dplyr::mutate(KIMURA80 = round(x = KIMURA80, digits = 2)) %>%\n group_by(named_subclass, KIMURA80) %>%\n mutate(KIMURA_SUM = sum(width)) %>%\n ungroup() %>%\n dplyr::select(subclass, named_subclass, KIMURA80, KIMURA_SUM) %>%\n base::unique() %>%\n arrange(named_subclass, KIMURA80)\n\n# Set fill colours\nfill_colours <- tibble(subclass = c(\"DNA\", \"LINE\", \"LTR\", \"PLE\", \"RC\", \"SINE\", \"Other\", \"Unknown\"),\n named_subclass = c(\"DNA Transposon\", \"LINE\", \"LTR Retrotransposon\", \"Penelope\", \"Rolling Circle\", \"SINE\", \"Other\", \"Unknown\"),\n fill_colour = c(\"#E32017\", \"#0098D4\", \"#00782A\", \"#7156A5\", \"#EE7C0E\", \"#9B0056\", \"#F3A9BB\", \"#A0A5A9\")) %>%\n filter(subclass %in% divergence_eg_tes_rounded_for_plot$subclass) %>%\n arrange(named_subclass) %>%\n filter(subclass %in% divergence_eg_tes_rounded_for_plot$subclass)\n\n# Create and save main plots\nkimura_plot <- ggplot(divergence_eg_tes_rounded_for_plot,\n aes(x = KIMURA80, y = KIMURA_SUM, fill = named_subclass)) +\n geom_col(position = \"stack\", width = 0.01) +\n theme_bw() +\n labs(title = plot_title) + theme(plot.title = element_markdown(hjust = 0.5)) +\n scale_fill_manual(values = fill_colours$fill_colour, name = \"TE Subclass\")\nsubclass_kimura_plot <- kimura_plot + scale_y_continuous(expand = c(0.01,0), name = \"Base pairs\")\n# Flip axis if desired\nif(opt$axis_flip == TRUE){\n subclass_kimura_plot <- subclass_kimura_plot +\n scale_x_continuous(limits = c(0.51, -0.01),\n expand = c(0,0), name = \"Kimura 2-Parameter Distance\",\n trans = \"reverse\")\n} else {\n subclass_kimura_plot <- subclass_kimura_plot +\n scale_x_continuous(limits = c(-0.01, 0.51),\n expand = c(0,0), name = \"Kimura 2-Parameter Distance\")\n}\nggsave(plot = subclass_kimura_plot, filename = paste0(out_directory, \"/\", species_name, \"_classification_landscape.pdf\"), device = \"pdf\", width = 12.85, height = 8.5)\n\nsplit_subclass_kimura_plot <- kimura_plot + scale_y_continuous(name = \"Base pairs\", labels = function(x) format(x, scientific = TRUE)) + facet_grid(subclass~., scales = \"free\")\n# Flip axis if desired\nif(opt$axis_flip == TRUE){\n split_subclass_kimura_plot <- split_subclass_kimura_plot +\n scale_x_continuous(limits = c(0.51, -0.01),\n expand = c(0,0), name = \"Kimura 2-Parameter Distance\",\n trans = \"reverse\")\n} else {\n split_subclass_kimura_plot <- split_subclass_kimura_plot +\n scale_x_continuous(limits = c(-0.01, 0.51),\n expand = c(0,0), name = \"Kimura 2-Parameter Distance\")\n}\nggsave(plot = split_subclass_kimura_plot, filename = paste0(out_directory, \"/\", species_name, \"_split_class_landscape.pdf\"), device = \"pdf\", width = 12.85, height = 8.5)\n\n# Perform maths for more divided plot\ndivergence_eg_tes_rounded_for_superfamily_plot <- divergence_eg_tes_gff %>%\n as_tibble() %>%\n filter(!is.na(KIMURA80)) %>%\n mutate(KIMURA80 = as.numeric(KIMURA80)) %>%\n dplyr::filter(KIMURA80 <= 0.5) %>%\n dplyr::mutate(KIMURA80 = round(x = KIMURA80, digits = 2),\n type = sub(\"-.*\", \"\", type)) %>%\n group_by(superfamily, KIMURA80) %>%\n mutate(KIMURA_SUM = sum(width)) %>%\n ungroup() %>%\n dplyr::select(type, subclass, superfamily, KIMURA80, KIMURA_SUM) %>%\n base::unique() %>%\n arrange(subclass, superfamily, KIMURA80)\n\n# Split data as necessary\ndivergence_eg_tes_rounded_for_superfamily_plot <- split(divergence_eg_tes_rounded_for_superfamily_plot,\n f = divergence_eg_tes_rounded_for_superfamily_plot$subclass)\n\n# Create plots of superfamilies of DNA transposons, LINEs, LTR retrotransposons and SINEs\n\nif (!is.null(divergence_eg_tes_rounded_for_superfamily_plot[[\"DNA\"]]) && nrow(divergence_eg_tes_rounded_for_superfamily_plot[[\"DNA\"]]) > 0) {\n kimura_superfamily_plot_1 <- ggplot(divergence_eg_tes_rounded_for_superfamily_plot$DNA,\n aes(x = KIMURA80, y = KIMURA_SUM, fill = superfamily)) +\n geom_col(position = \"stack\", width = 0.01, colour = \"black\", linewidth = 0.2) +\n theme_bw() +\n theme(legend.title=element_blank()) +\n scale_y_continuous(name = \"Base pairs\", labels = function(x) format(x, scientific = TRUE)) +\n facet_grid(subclass~., scales = \"free\") +\n guides(fill=guide_legend(ncol=3))\n} else {\n kimura_superfamily_plot_1 <- NULL\n}\n\nif (!is.null(divergence_eg_tes_rounded_for_superfamily_plot[[\"LINE\"]]) && nrow(divergence_eg_tes_rounded_for_superfamily_plot[[\"LINE\"]]) > 0) {\n kimura_superfamily_plot_2 <- ggplot(divergence_eg_tes_rounded_for_superfamily_plot$LINE,\n aes(x = KIMURA80, y = KIMURA_SUM, fill = superfamily)) +\n geom_col(position = \"stack\", width = 0.01, colour = \"black\", linewidth = 0.2) +\n theme_bw() +\n theme(legend.title=element_blank()) +\n scale_y_continuous(name = \"Base pairs\", labels = function(x) format(x, scientific = TRUE)) +\n facet_grid(subclass~., scales = \"free\") +\n guides(fill=guide_legend(ncol=3)) +\n scale_fill_brewer(palette = \"Blues\", direction = -1)\n} else {\n kimura_superfamily_plot_2 <- NULL\n}\n\nif (!is.null(divergence_eg_tes_rounded_for_superfamily_plot[[\"LTR\"]]) && nrow(divergence_eg_tes_rounded_for_superfamily_plot[[\"LTR\"]]) > 0) {\n kimura_superfamily_plot_3 <- ggplot(divergence_eg_tes_rounded_for_superfamily_plot$LTR,\n aes(x = KIMURA80, y = KIMURA_SUM, fill = superfamily)) +\n geom_col(position = \"stack\", width = 0.01, colour = \"black\", linewidth = 0.2) +\n theme_bw() +\n theme(legend.title=element_blank()) +\n scale_y_continuous(name = \"Base pairs\", labels = function(x) format(x, scientific = TRUE)) +\n facet_grid(subclass~., scales = \"free\") +\n guides(fill=guide_legend(ncol=3)) +\n scale_fill_brewer(palette = \"Greens\", direction = -1)\n} else {\n kimura_superfamily_plot_3 <- NULL\n}\n\nif (!is.null(divergence_eg_tes_rounded_for_superfamily_plot[[\"SINE\"]]) && nrow(divergence_eg_tes_rounded_for_superfamily_plot[[\"SINE\"]]) > 0) {\n kimura_superfamily_plot_4 <- ggplot(divergence_eg_tes_rounded_for_superfamily_plot$SINE,\n aes(x = KIMURA80, y = KIMURA_SUM, fill = superfamily)) +\n geom_col(position = \"stack\", width = 0.01, colour = \"black\", linewidth = 0.2) +\n theme_bw() +\n theme(legend.title=element_blank()) +\n scale_y_continuous(name = \"Base pairs\", labels = function(x) format(x, scientific = TRUE)) +\n facet_grid(subclass~., scales = \"free\") +\n guides(fill=guide_legend(ncol=3)) +\n scale_fill_brewer(palette = \"YlOrRd\", direction = -1)\n} else {\n kimura_superfamily_plot_4 <- NULL\n}\n\nif (!is.null(divergence_eg_tes_rounded_for_superfamily_plot[[\"PLE\"]]) && nrow(divergence_eg_tes_rounded_for_superfamily_plot[[\"PLE\"]]) > 0) {\n kimura_superfamily_plot_5 <- ggplot(divergence_eg_tes_rounded_for_superfamily_plot$PLE,\n aes(x = KIMURA80, y = KIMURA_SUM, fill = superfamily)) +\n geom_col(position = \"stack\", width = 0.01, colour = \"black\", linewidth = 0.2) +\n theme_bw() +\n theme(legend.title=element_blank()) +\n scale_y_continuous(name = \"Base pairs\", labels = function(x) format(x, scientific = TRUE)) +\n facet_grid(subclass~., scales = \"free\") +\n guides(fill=guide_legend(ncol=3)) +\n scale_fill_brewer(palette = \"Purples\", direction = -1)\n} else {\n kimura_superfamily_plot_5 <- NULL\n}\n\nif (!is.null(divergence_eg_tes_rounded_for_superfamily_plot[[\"RC\"]]) && nrow(divergence_eg_tes_rounded_for_superfamily_plot[[\"RC\"]]) > 0) {\n kimura_superfamily_plot_6 <- ggplot(divergence_eg_tes_rounded_for_superfamily_plot$RC,\n aes(x = KIMURA80, y = KIMURA_SUM, fill = superfamily)) +\n geom_col(position = \"stack\", width = 0.01, colour = \"black\", linewidth = 0.2) +\n theme_bw() +\n theme(legend.title=element_blank()) +\n scale_y_continuous(name = \"Base pairs\", labels = function(x) format(x, scientific = TRUE)) +\n facet_grid(subclass~., scales = \"free\") +\n guides(fill=guide_legend(ncol=3)) +\n scale_fill_brewer(palette = \"Oranges\", direction = -1)\n} else {\n kimura_superfamily_plot_6 <- NULL\n}\n\n# Flip axis if desired — only applied to non-NULL plots\n# The last non-NULL plot gets the axis label; all others get an empty name\nall_superfamily_plots <- list(\n kimura_superfamily_plot_1, kimura_superfamily_plot_2, kimura_superfamily_plot_3,\n kimura_superfamily_plot_4, kimura_superfamily_plot_5, kimura_superfamily_plot_6\n)\nnon_null_idx <- which(!sapply(all_superfamily_plots, is.null))\nlast_idx <- if (length(non_null_idx) > 0) max(non_null_idx) else NA\n\nfor (i in seq_along(all_superfamily_plots)) {\n if (is.null(all_superfamily_plots[[i]])) next\n axis_label <- if (!is.na(last_idx) && i == last_idx) \"Kimura 2-Parameter Distance\" else \"\"\n if (opt$axis_flip == TRUE) {\n all_superfamily_plots[[i]] <- all_superfamily_plots[[i]] +\n scale_x_continuous(limits = c(0.51, -0.01), expand = c(0, 0),\n name = axis_label, trans = \"reverse\")\n } else {\n all_superfamily_plots[[i]] <- all_superfamily_plots[[i]] +\n scale_x_continuous(limits = c(-0.01, 0.51), expand = c(0, 0),\n name = axis_label)\n }\n}\nvalid_plots <- all_superfamily_plots[!sapply(all_superfamily_plots, is.null)]\n\n# Combine plots and title\nsuperfamily_kimura_plot <- plot_grid(plotlist = valid_plots, \n ncol = 1, align = \"v\")\nsuperfamily_kimura_plot_titled <- plot_grid(title_plot, superfamily_kimura_plot, ncol = 1, rel_heights = c(1, 30))\n\n# Save divided plot\nggsave(plot = superfamily_kimura_plot_titled, filename = paste0(out_directory, \"/\", species_name, \"_superfamily_div_plot.pdf\"), device = \"pdf\", width = 12.85, height = 8.5)\n"
},
{
"path": "scripts/extract_align.py",
"content": "import argparse\nimport shutil\nimport re\nimport os\nimport subprocess\nfrom Bio import SeqIO\nimport pandas as pd\nimport numpy as np\nfrom pybedtools import BedTool\nfrom pyfaidx import Fasta\nimport logging\nimport time\nimport datetime\npd.options.mode.chained_assignment = None # default='warn'\n\nLOGGER = logging.getLogger(__name__)\n\n## Set up input arguments\ndef get_args():\n\tparser = argparse.ArgumentParser(description=\"Will process a blast output generated using a file of putative TEs (usually generated by RepeatModeler. For each putative consensus in the input putative TE library, it will generate an aligned file with N buffered instances from the queried genome, the input consensus, and, if requested, a new revised and extended consensus for inspection.\", formatter_class=argparse.ArgumentDefaultsHelpFormatter)\n\tparser.add_argument('-g', '--genome_fasta', type=str, help='Name of the fasta formatted genome to be queried.', required=True)\n\tparser.add_argument('-b', '--blastfile', type=str, help='Blast output to be used. Must be formatted using \"outfmt 6\".', required = True)\n\tparser.add_argument('-l', '--library', type=str, help='Library of putative TE consensus sequences to be extracted and aligned. Must be in fasta format with no # or / in the headers.', required = True)\n\tparser.add_argument('-lb', '--leftbuffer', type=int, help='Left buffer size. The number of bp of flanking sequence for each hit to be extracted along with the hit. Optional, Default = 1000', default = 1000)\n\tparser.add_argument('-rb', '--rightbuffer', type=int, help='Right beffer size. The number of bp of flanking sequence for each hit to be extracted along with the hit. Optional, Default = 1000', default = 1000)\n\tparser.add_argument('-n', '--hitnumber', type=int, help='The number of hits to be exracted. Optional. Default = 50.', default = 50)\n\tparser.add_argument('-a', '--align', type=str, help='Align the output fasta file, y or n?. Default is y.', default = 'y')\n\tparser.add_argument('-t', '--trimal', type=str, help='Use trimal to remove low-aligning regions, y or n? Trimal can sometimes encounter an error that prevents it from working, this results in an empty file in downstream analyses. Default is y.', default = 'y')\n\tparser.add_argument('-e', '--emboss', type=str, help='Generate a trimal/emboss consensus, y or n. Optional.', default = 'y')\n\tparser.add_argument(\"-log\", \"--log_level\", default=\"INFO\")\n\n\targs = parser.parse_args()\n\tGENOMEFA = args.genome_fasta\n\tBLAST = args.blastfile\n\tLIB = args.library\n\tLBUFFER = args.leftbuffer\n\tRBUFFER = args.rightbuffer\n\tHITNUM = args.hitnumber\n\tALIGN = args.align\n\tTRIMAL = args.trimal\n\tEMBOSS = args.emboss\n\tLOG = args.log_level\n\n\treturn GENOMEFA, BLAST, LIB, LBUFFER, RBUFFER, HITNUM, ALIGN, TRIMAL, EMBOSS, LOG\n\n## Create TE outfiles function. Creates files for populating with blast hits.\ndef CREATE_TE_OUTFILES(LIBRARY):\n\tfor record in SeqIO.parse(LIBRARY, 'fasta'):\n\t\tNEWID = re.sub('#', '__', record.id)\n\t\tNEWID = re.sub('/', '___', NEWID)\n\t\trecord.id = 'CONSENSUS-' + NEWID\n\t\trecord.description = ''\n\t\tSeqIO.write(record, 'tmpTEfiles/' + NEWID + '.fa', 'fasta')\n\t\t\t\t\n## Organize blast hits function. Will read in blast file, sort based on e-value and bitscore, deterine top BUFFER hits for extraction, extract, and combine with TE file from previous function.\ndef EXTRACT_BLAST_HITS(GENOME, BLAST, LBUFFER, RBUFFER, HITNUM):\n##Read in blast data\n\tBLASTDF = pd.read_table(BLAST, sep='\\t', names=['QUERYNAME', 'SCAFFOLD', 'C', 'D', 'E', 'F', 'QUERYSTART', 'QUERYSTOP', 'SCAFSTART', 'SCAFSTOP', 'E-VALUE', 'BITSCORE'])\n##Convert to bed format\n\tBLASTBED = BLASTDF[['SCAFFOLD', 'SCAFSTART', 'SCAFSTOP', 'QUERYNAME', 'E-VALUE', 'BITSCORE']]\n\tBLASTBED.insert(5, 'STRAND', '+')\n\tBLASTBED.loc[BLASTBED.SCAFSTOP < BLASTBED.SCAFSTART, 'STRAND'] = '-'\n\tBLASTBED.SCAFSTART, BLASTBED.SCAFSTOP = np.where(BLASTBED.SCAFSTART > BLASTBED.SCAFSTOP, [BLASTBED.SCAFSTOP, BLASTBED.SCAFSTART], [BLASTBED.SCAFSTART, BLASTBED.SCAFSTOP])\n##Generate list of query names\n\tQUERYLIST = BLASTBED.QUERYNAME.unique()\n\tLOGGER.info('There are ' + str(len(QUERYLIST)) + ' consensus sequences to process')\n#\tCOUNTER = 1\n##Sort subsets of df based on query names, keep the top BUFFER hits, make bedfiles, extract, and combine\n\tfor QUERY in QUERYLIST:\n#\t\tLOGGER.info('Extracting for TE: ' + str(COUNTER))\n\t\tQUERYFRAME = BLASTBED[BLASTBED['QUERYNAME'] == QUERY]\n\t\tQUERYFRAME = QUERYFRAME.sort_values(by=['E-VALUE', 'BITSCORE'], ascending=[True, False])\n\t\tQUERYFRAME = QUERYFRAME.head(HITNUM)\n\t\tQUERYFRAMESAVE = 'tmpbedfiles/' + QUERY + '.bed'\n\t\tQUERYFRAME.to_csv('tmpbedfiles/' + QUERY + '.bed', sep='\\t', header=False, index=False)\n\t\tCURRENTBED = BedTool('tmpbedfiles/' + QUERY + '.bed')\n\t\tGENOMEPREFIX = os.path.splitext(GENOME)[0]\n\t\tSLOPBED = CURRENTBED.slop(g=GENOMEPREFIX + '.fai', l=LBUFFER, r=RBUFFER, output='tmpbedfiles/' + QUERY + '.slop')\n\t\tSLOPBED = BedTool('tmpbedfiles/' + QUERY + '.slop')\n\t\tFASTA = SLOPBED.sequence(fi=GENOME, s=True)\n\t\tFASTASAVE = SLOPBED.save_seqs('tmpextracts/' + QUERY + '.fa')\n\t\tos.remove('tmpbedfiles/' + QUERY + '.slop')\n\t\tos.remove('tmpbedfiles/' + QUERY + '.bed')\n\t\tsubprocess.run('cat {} {} >{}'.format('tmpextracts/' + QUERY + '.fa', 'tmpTEfiles/' + QUERY +'.fa', 'catTEfiles/' + QUERY +'.fa'), shell=True)\n#\t\tCOUNTER = COUNTER + 1\n\t\t\n##Alignment function\ndef MUSCLE(TOALIGN):\n\tTOALIGNPREFIX = os.path.splitext(TOALIGN)[0]\n\tSOFTWARE = '/usr/bin'\n\tsubprocess.check_call(SOFTWARE + '/muscle -in {} -out {}'.format('catTEfiles/' + TOALIGN, 'muscle/' + TOALIGNPREFIX + '.fa'), shell=True)\n\n##Consensus generation function\ndef CONSENSUSGEN(ALIGNED):\n\tFILEPREFIX = os.path.splitext(ALIGNED)[0] \n\tSOFTWARE = '/home/toby/Programs/'\n\tif TRIMAL == 'y':\n\t\tsubprocess.run(SOFTWARE + 'trimal/source/trimal -in {} -gt 0.6 -cons 60 -fasta -out {}'.format('muscle/' + ALIGNED, 'muscle/' + FILEPREFIX + '_trimal.fa'), shell=True)\n\t\tsubprocess.run(SOFTWARE + 'EMBOSS-6.6.0/emboss/cons -sequence muscle/' + FILEPREFIX + '_trimal.fa -outseq muscle/' + FILEPREFIX + '_cons.fa -name ' + FILEPREFIX + '_cons -plurality 3 -identity 3', shell=True)\n\t\tsubprocess.run('cat {} {} >{}'.format('muscle/' + FILEPREFIX + '_trimal.fa', 'muscle/' + FILEPREFIX + '_cons.fa', 'consensusfiles/' + FILEPREFIX + '_cons.fa'), shell=True)\n\tif TRIMAL == 'n':\n#\t\tsubprocess.run(SOFTWARE + 'trimal/source/trimal -in {} -gt 0.6 -cons 60 -fasta -out {}'.format('muscle/' + ALIGNED, 'muscle/' + FILEPREFIX + '_trimal.fa'), shell=True)\n\t\tsubprocess.run(SOFTWARE + 'EMBOSS-6.6.0/emboss/cons -sequence muscle/' + ALIGNED + ' -outseq muscle/' + FILEPREFIX + '_cons.fa -name ' + FILEPREFIX + '_cons -plurality 3 -identity 3', shell=True)\n\t\tsubprocess.run('cat {} {} >{}'.format('muscle/' + ALIGNED, 'muscle/' + FILEPREFIX + '_cons.fa', 'consensusfiles/' + FILEPREFIX + '_cons.fa'), shell=True)\n\ndef DIRS(DIR):\n\tif os.path.exists(DIR):\n\t\tshutil.rmtree(DIR)\n\tos.mkdir(DIR)\n\n####MAIN function\ndef main():\t\n##Get input arguments\n\tGENOMEFA, BLAST, LIB, LBUFFER, RBUFFER, HITNUM, ALIGN, TRIMAL, EMBOSS, LOG = get_args()\n\n# Setup logging and script timing\n\thandlers = [logging.FileHandler('extract_align.log'), logging.StreamHandler()]\n\tlogging.basicConfig(format='', handlers = handlers)\n\tlogging.getLogger().setLevel(getattr(logging, LOG.upper()))\n\n\tstart_time = time.time()\n\n\tLOGGER.info('#\\n# extract_align.py\\n#')\n\n\tLOGGER.info('Genome file: ' + GENOMEFA)\n\tLOGGER.info('Blast file: ' + BLAST)\n\tLOGGER.info('TE library: ' + LIB)\n\tLOGGER.info('Left buffer size: ' + str(LBUFFER))\n\tLOGGER.info('Right buffer size: ' + str(RBUFFER))\n\tLOGGER.info('Number of hits evaluated: ' + str(HITNUM))\n\tLOGGER.info('Muscle alignment = ' + ALIGN)\n\tLOGGER.info('Trimal processing = ' + TRIMAL)\n\tLOGGER.info('Emboss consensus = ' + EMBOSS)\n\tLOGGER.info('Log level: ' + LOG)\n\n## Index the genome \n\tLOGGER.info('Indexing the genome')\n\tGENOMEIDX = Fasta(GENOMEFA)\n\tGENOMEPREFIX = os.path.splitext(GENOMEFA)[0]\n\tFAIDX = pd.read_table(GENOMEFA + '.fai', sep='\\t', names=['one', 'two', 'three', 'four', 'five'])\n\tFAIDX = FAIDX[['one', 'two']]\n\tFAIDX.to_csv(GENOMEPREFIX + '.fai', sep='\\t', header=False, index=False)\n\t\t\n## Set up directories\t\n\tLOGGER.info('Creating tmp and permanent directories')\n\tDIRS('tmpTEfiles')\n\tDIRS('tmpbedfiles')\n\tif ALIGN == 'y':\n\t\tDIRS('muscle')\n\tif EMBOSS == 'y':\n\t\tDIRS('consensusfiles')\n\tDIRS('tmpextracts')\n\tDIRS('catTEfiles')\n\t\n##Determine optional arguments and print to screen.\n\tif ALIGN == 'n' and EMBOSS == 'y':\n\t\tLOGGER.info('Input is contradictory. Generating a new consensus with emboss requires muscle alignment.')\n\telif ALIGN == 'y' and EMBOSS == 'y':\n\t\tLOGGER.info('Output files will be aligned and a new consensus will be generated with emboss and trimal.')\n\telif ALIGN == 'y' and EMBOSS == 'n':\n\t\tLOGGER.info('Output files will be aligned but without a new emboss/trimal consensus.')\n\telif ALIGN == 'n' and EMBOSS == 'n':\n\t\tLOGGER.info('Extractions will be made but no alignment.')\n\telse:\n\t\tLOGGER.info('Invalid input for either align, or emboss, or both.')\n\n##Create TE out files to populate with blast hits\n\tCREATE_TE_OUTFILES(LIB)\n\t\n##Extract hits and combine them with the TE out files if flagged\n\tEXTRACT_BLAST_HITS(GENOMEFA, BLAST, LBUFFER, RBUFFER, HITNUM)\n\t\n##Align extracted hits if flagged\n\tif ALIGN == 'y':\n\t\tCOUNTER = 1\n\t\tfor FILE in os.listdir('tmpextracts'):\n\t\t\tLOGGER.info('Aligning TE: ' + str(COUNTER))\n\t\t\tMUSCLE(FILE)\n\t\t\tCOUNTER = COUNTER + 1\n\n##Generate new consensus with emboss if flagged\n\tif EMBOSS == 'y':\n\t\tfor FILE in os.listdir('muscle'):\n\t\t\tCONSENSUSGEN(FILE)\n\t\t\t\n##Remove empty tmp directories and unneeded files\n\tLOGGER.info('Removing tmp directories and extraneous files')\n\tshutil.rmtree('tmpbedfiles/')\n\tshutil.rmtree('tmpextracts/')\n\tshutil.rmtree('tmpTEfiles/')\n\t#FILES = [F for F in os.listdir('muscle/') if F.endswith('_cons.fa')]\n\t#for FILE in FILES:\n\t#\tos.remove('muscle/' + FILE)\n\t#FILES = [F for F in os.listdir('muscle/') if F.endswith('_trimal.fa')]\n\t#for FILE in FILES:\n\t#\tos.remove('muscle/' + FILE)\n\t\n\tend_time = time.time()\n\tLOGGER.info('Run time: ' + str(datetime.timedelta(seconds=end_time-start_time)))\n#\n# Wrap script functionality in main() to avoid automatic execution\n# when imported ( e.g. when help is called on file )\n#\nif __name__ ==\"__main__\":main()\n\t\t\n"
},
{
"path": "scripts/faswap.py",
"content": "#!/usr/bin/env python3\n\nimport sys\nimport csv\n\nargs = sys.argv\n\ndictionary = args[1]\nfastaIn = args[2]\n\na=dict(csv.reader(open(dictionary),delimiter=\"\\t\"))\nlines=open(fastaIn).read().split(\"\\n\")\nlines=map(lambda x: \">\"+a[x[1:]] if x and x[0]==\">\" else x,lines);print(\"\\n\".join(lines))\n"
},
{
"path": "scripts/filteringOverlappingRepeats.R",
"content": "# load libraries\nsuppressPackageStartupMessages(library(GenomicRanges))\nsuppressPackageStartupMessages(library(ape))\nsuppressPackageStartupMessages(library(tidyverse))\n\n# set options\n\noptions(scipen = 100, stringsAsFactors = FALSE)\n\n#get inputs\n\nargs <- commandArgs()\n# print(args)\ngff.in <- args[6]\ngff.out <- args[7]\n\n# read gff\n\ninput <- read.gff(gff.in)\n\n# cut overlapping regions in half\ninput %<>%\n arrange(seqid, start) %>%\n mutate(new.start = case_when(seqid == lag(seqid) & start < lag(end) & end > lag(end) ~ as.integer((start + ((lag(end) - start)/2)) + 1),\n seqid == lag(seqid) & start == lag(end) ~ as.integer(start + 1),\n .default = start),\n new.end = case_when(seqid == lead(seqid) & end > lead(start) & end < lead(end) ~ as.integer((end - (end - lead(start))/2)),\n seqid == lead(seqid) & end == lead(start) ~ as.integer(end),\n .default = end)) %>%\n mutate(start = new.start,\n end = new.end) %>%\n select(-c(new.start, new.end))\n\n# identify nested and unnested TEs\n\n# nest detect function\ndetect_nesting <- function(df) {\n df %>%\n arrange(seqid, start) %>%\n mutate(\n nested = case_when(\n seqid == lag(seqid) &\n start > lag(start) &\n end < lag(end) ~ \"FULLY_NESTED\",\n TRUE ~ \"NOT_NESTED\"\n )\n )\n}\n\n# iterate to detect all nestings\nall_nested <- list()\n\niteration <- 1\ncurrent <- input\n\nrepeat {\n print(paste0(\"Searching for Nested TEs. Iteration:\", iteration))\n annotated <- detect_nesting(current)\n \n newly_nested <- annotated %>%\n filter(nested == \"FULLY_NESTED\") %>%\n mutate(nesting_round = iteration)\n \n if (nrow(newly_nested) == 0) break\n \n all_nested[[iteration]] <- newly_nested\n \n current <- annotated %>%\n filter(nested == \"NOT_NESTED\") %>%\n select(-nested)\n \n iteration <- iteration + 1\n}\n\nnested_only <- bind_rows(all_nested)\nunnested_only <- current\n\n# recombine to make GFF\nif (nrow(nested_only) > 0) {\n nested_only <- nested_only %>%\n mutate(attributes = paste0(attributes, \";nested=\", nested, \"-round\", nesting_round)) %>%\n select(-c(nested, nesting_round))\n output.gff <- bind_rows(nested_only, unnested_only) %>%\n arrange(seqid, start)\n} else {\n # No nested TEs found — write unnested output directly\n output.gff <- unnested_only %>%\n arrange(seqid, start)\n}\n\n# Write Table\nwrite.table(output.gff, gff.out, sep = \"\\t\", quote = F, row.names = F, col.names = F)\n"
},
{
"path": "scripts/headSwap.sh",
"content": "#!/bin/bash\n\nwhile getopts i:f:o: option\ndo\n case \"${option}\" \n in\n i) input=${OPTARG};;\n o) output=${OPTARG};;\n esac\ndone\n\n\n# 1 - Create Dictionary\nSCRIPT_DIR=/data/toby/EarlGrey/scripts\n\ngrep \">\" $input | awk '{printf(\"ctg_%01d %s\\n\", NR, $0)}' | sed 's/ /\\t/g; s/>//g' | awk '{OFS=\"\\t\"}{print $2, $1}' > ${input}.dict\ntr -d $'\\r' < ${input}.dict > ${input}.dict.1 && mv ${input}.dict{.1,}\n\n# 2 - Replace Fasta Headers\n\n${SCRIPT_DIR}/faswap.py ${input}.dict $input > $output\n"
},
{
"path": "scripts/install_r_packages.R",
"content": "# Install R Packages\r\n\r\ninstall.packages(c(\"BiocManager\",\"optparse\", \"ape\", \"optparse\"), repos=\"https://www.stats.bris.ac.uk/R/\")\r\nBiocManager::install(c(\"plyranges\", \"BSgenome\"), update = FALSE)\r\n"
},
{
"path": "scripts/makeGff.R",
"content": "suppressPackageStartupMessages(library(tidyverse))\nsuppressPackageStartupMessages(library(data.table))\nsuppressPackageStartupMessages(library(magrittr))\n\nargs <- commandArgs()\n# print(args)\nbedIn <- args[6]\ngffIn <- args[7]\ngffOut <- args[8]\n\n# test\n#bedIn <- \"~/projects/monarch/sortingSINEs/repeatMasker/merge/danausPlexippus.filteredRepeats.bed\"\n#gffIn <- \"/home/toby/projects/monarch/sortingSINEs/repeatMasker/merge/remerge/danausPlexippus.rmerge.gff.filtered\"\n#gffOut <- \"/home/toby/projects/monarch/sortingSINEs/repeatMasker/merge/remerge/danausPlexippus.filteredRepeats.gff\"\n\n# step 1 - read in tables\n\ntab1 <- read.table(bedIn)\ntab2 <- read.table(gffIn, sep = \"\\t\")\n\ntab1$id <- paste(tab1$V1, tab1$V2, tab1$V3, tab1$V4, tab1$V5, sep = \"_\")\ntab2$id <- paste(tab2$V1, (tab2$V4 - 1), tab2$V5, tab2$V3, tab2$V6, sep = \"_\")\n\ntab2 <- tab2 %>% distinct()\n\nfil <- tab2[tab2$id %in% tab1$id,]\nfil <- fil[,c(1:9)]\nfil$V8 <- \".\" #set phase to be Geneious-Friendly\n\nfor (i in 1:length(fil$V1)) {\n if (fil$V5[i] < fil$V4[i]) {\n start <- fil$V5[i]\n end <- fil$V4[i]\n fil$V4[i] <- start\n fil$V5[i] <- end\n }\n}\n\nfil <- fil %>%\n mutate(V2 = case_when(V9 %like% \"^ID=HLE*\" ~ \"HELIANO\",\n .default = \"Earl_Grey\"),\n V9 = toupper(V9),\n ID = paste0(sub(\".*;ID=\", \"ID=\", sub(\";SHORTTE.*$\", \"\", V9)), \"_\", row_number()),\n Name = paste0(sub(\".*ID=\", \"Name=\", sub(\";SHORTTE.*$\", \"\", V9))),\n V9 = paste0(ID, \";\", Name, \";\", sub(\"ID=.*\", \"\", V9), \";\", sub(\".*SHORTTE\", \"SHORTTE\", V9))) %>%\n select(-c(ID, Name))\n\nwrite.table(fil, gffOut, quote = FALSE, col.names = FALSE, row.names = FALSE, sep = \"\\t\")\n"
},
{
"path": "scripts/mergeRepeats.R",
"content": "suppressPackageStartupMessages(library(tidyverse))\nsuppressPackageStartupMessages(library(plyr))\nsuppressPackageStartupMessages(library(dplyr))\nsuppressPackageStartupMessages(library(magrittr))\nsuppressPackageStartupMessages(library(data.table))\noptions(scipen = 100, stringsAsFactors = FALSE)\n\n#####\n\nargs <- commandArgs()\n# print(args)\nfile <- args[6]\noutput <- args[7]\ngenomeSize <- args[8]\noutput2 <- args[9]\nfiltBed <- args[10]\nfiltSum <- args[11]\nlowend <- args[12]\n\n# Step 1 - Read in rmerge table\n\nrepeats <- fread(file)\ncolnames(repeats) <- c(\"Scaffold\", \"ReMa\", \"Repeat\", \"Start\", \"End\", \"Score\", \"Strand\", \"Dot\", \"Desc\")\n\n# Step 2 - Calculate size of repeats\n\n# calculate size of all repeats\nrepeats$size <- abs((repeats$End - repeats$Start)) + 1\nrepeats$Start <- as.double(repeats$Start) - 1\nrepeats$End <- as.double(repeats$End)\n\nmergedRepeats <- repeats\nmergedRepeatsOut <- repeats[,c(1,4,5,3,6,7)]\nwrite.table(mergedRepeatsOut, file = output, quote = FALSE, row.names = FALSE, sep = \"\\t\", col.names = FALSE)\n\n# Generate RM style summary table for all unnested repeats, and nested ones too\n\nsine <- mergedRepeats %>% filter(str_detect(Repeat, \"SINE\")) %>% filter(! str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nline <- mergedRepeats %>% filter(str_detect(Repeat, \"LINE\")) %>% filter(! str_detect(Repeat, \"Penelope\")) %>% filter(! str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nltr <- mergedRepeats %>% filter(str_detect(Repeat, \"LTR\")) %>% filter(! str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\ndna <- mergedRepeats %>% filter(str_detect(Repeat, \"DNA\")) %>% filter(! str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nrc <- mergedRepeats %>% filter(str_detect(Repeat, \"RC\")) %>% filter(! str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nple <- mergedRepeats %>% filter(str_detect(Repeat, \"Penelope\")) %>% filter(! str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nunknown <- mergedRepeats %>% filter(str_detect(Repeat, \"Unknown\")) %>% filter(! str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nother <- mergedRepeats %>% filter(str_detect(Repeat, \"ARTEFACT|Low|Retrop|Satellite|Simple|RNA\")) %>% filter(! str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nother <- other[! other$Repeat == \"SINE\",]\n\nsine.nest <- mergedRepeats %>% filter(str_detect(Repeat, \"SINE\")) %>% filter(str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nline.nest <- mergedRepeats %>% filter(str_detect(Repeat, \"LINE\")) %>% filter(! str_detect(Repeat, \"Penelope\")) %>% filter(str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nltr.nest <- mergedRepeats %>% filter(str_detect(Repeat, \"LTR\")) %>% filter(str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\ndna.nest <- mergedRepeats %>% filter(str_detect(Repeat, \"DNA\")) %>% filter(str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nrc.nest <- mergedRepeats %>% filter(str_detect(Repeat, \"RC\")) %>% filter(str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nple.nest <- mergedRepeats %>% filter(str_detect(Repeat, \"Penelope\")) %>% filter(str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nunknown.nest <- mergedRepeats %>% filter(str_detect(Repeat, \"Unknown\")) %>% filter(str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nother.nest <- mergedRepeats %>% filter(str_detect(Repeat, \"ARTEFACT|Low|Retrop|Satellite|Simple|RNA\")) %>% filter(str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nother.nest <- other.nest[! other.nest$Repeat == \"SINE\",]\n\n# Total bp coverage of each class\nsineCov <- sum(as.numeric(as.character(sine$size)))\nlineCov <- sum(as.numeric(as.character(line$size)))\nltrCov <- sum(as.numeric(as.character(ltr$size)))\ndnaCov <- sum(as.numeric(as.character(dna$size)))\nrcCov <- sum(as.numeric(as.character(rc$size)))\npleCov <- sum(as.numeric(as.character(ple$size)))\nunknownCov <- sum(as.numeric(as.character(unknown$size)))\notherCov <- sum(as.numeric(as.character(other$size)))\ntotalCov <- (sineCov + lineCov + ltrCov + dnaCov + rcCov + pleCov + unknownCov + otherCov) \nunmaskCov <- (as.numeric(genomeSize) - as.numeric(totalCov))\n\nsineCov.nest <- sum(as.numeric(as.character(sine.nest$size)))\nlineCov.nest <- sum(as.numeric(as.character(line.nest$size)))\nltrCov.nest <- sum(as.numeric(as.character(ltr.nest$size)))\ndnaCov.nest <- sum(as.numeric(as.character(dna.nest$size)))\nrcCov.nest <- sum(as.numeric(as.character(rc.nest$size)))\npleCov.nest <- sum(as.numeric(as.character(ple.nest$size)))\nunknownCov.nest <- sum(as.numeric(as.character(unknown.nest$size)))\notherCov.nest <- sum(as.numeric(as.character(other.nest$size)))\n\n# Number of elements\nsineTal <- as.numeric(tally(sine))\nlineTal <- as.numeric(tally(line))\nltrTal <- as.numeric(tally(ltr))\ndnaTal <- as.numeric(tally(dna))\nrcTal <- as.numeric(tally(rc))\npleTal <- as.numeric(tally(ple))\nunknownTal <- as.numeric(tally(unknown))\notherTal <- as.numeric(tally(other))\ntotalTal <- (sineTal + lineTal + ltrTal + dnaTal + rcTal + pleTal + unknownTal + otherTal)\n\nsineTal.nest <- as.numeric(tally(sine.nest))\nlineTal.nest <- as.numeric(tally(line.nest))\nltrTal.nest <- as.numeric(tally(ltr.nest))\ndnaTal.nest <- as.numeric(tally(dna.nest))\nrcTal.nest <- as.numeric(tally(rc.nest))\npleTal.nest <- as.numeric(tally(ple.nest))\nunknownTal.nest <- as.numeric(tally(unknown.nest))\notherTal.nest <- as.numeric(tally(other.nest))\n\n# Calculate seq percentage\ngenomeSize <- as.numeric(genomeSize)\nsinePer <- (sineCov / genomeSize) * 100\nlinePer <- (lineCov / genomeSize) * 100\nltrPer <- (ltrCov / genomeSize) * 100\ndnaPer <- (dnaCov / genomeSize) * 100\nrcPer <- (rcCov / genomeSize) * 100\nplePer <- (pleCov / genomeSize) * 100\nunknownPer <- (unknownCov / genomeSize) * 100\notherPer <- (otherCov / genomeSize) * 100\ntotalPer <- (sinePer + linePer + ltrPer + dnaPer + rcPer + plePer + unknownPer + otherPer)\nunmaskPer <- (100 - totalPer)\n\nsinePer.nest <- (sineCov.nest / genomeSize) * 100\nlinePer.nest <- (lineCov.nest / genomeSize) * 100\nltrPer.nest <- (ltrCov.nest / genomeSize) * 100\ndnaPer.nest <- (dnaCov.nest / genomeSize) * 100\nrcPer.nest <- (rcCov.nest / genomeSize) * 100\nplePer.nest <- (pleCov.nest / genomeSize) * 100\nunknownPer.nest <- (unknownCov.nest / genomeSize) * 100\notherPer.nest <- (otherCov.nest / genomeSize) * 100\n\n# Make Table \neleTyp <- c(\"SINE\", \"LINE\", \"LTR\", \"DNA\", \"RC\", \"PLE\", \"Unknown\", \"Total Interspersed Repeat\", \"Other\", \"Unmasked\",\n \"SINE-nested\", \"LINE-nested\", \"LTR-nested\", \"DNA-nested\", \"RC-nested\", \"PLE-nested\", \"Unknown-nested\", \"Other-nested\")\nnumEle <- c(sineTal, lineTal, ltrTal, dnaTal, rcTal, pleTal, unknownTal, totalTal, otherTal, as.character(\"notApplicable\"),\n sineTal.nest, lineTal.nest, ltrTal.nest, dnaTal.nest, rcTal.nest, pleTal.nest, unknownTal.nest, otherTal.nest)\nlenOcc <- c(sineCov, lineCov, ltrCov, dnaCov, rcCov, pleCov, unknownCov, totalCov, otherCov, unmaskCov,\n sineCov.nest, lineCov.nest, ltrCov.nest, dnaCov.nest, rcCov.nest, pleCov.nest, unknownCov.nest, otherCov.nest)\nperSeq <- c(sinePer, linePer, ltrPer, dnaPer, rcPer, plePer, unknownPer, totalPer, otherPer, unmaskPer,\n sinePer.nest, linePer.nest, ltrPer.nest, dnaPer.nest, rcPer.nest, plePer.nest, unknownPer.nest, otherPer.nest)\nsize <- c(rep(genomeSize, 18))\n\nsummary <- data.frame(eleTyp, numEle, lenOcc, perSeq, size)\ncolnames(summary) <- c(\"Family\", \"Number of Elements\", \"Length Occupied\", \"Percentage of Sequence\", \"GenomeSize\")\n\n# Print table\nwrite.table(summary, output2, col.names = TRUE, row.names = FALSE, quote = FALSE, sep = \"\\t\")\n\nfilteredRepeatsOut <- mergedRepeatsOut\nfilteredRepeatsOut$length <- abs(filteredRepeatsOut$End - filteredRepeatsOut$Start)\nif (lowend == \"yes\") {\n filteredRepeatsOut <- filteredRepeatsOut[filteredRepeatsOut$length > 100,]\n}\n\n# if end coordinate is before start, switch\nfilteredRepeatsOut2 <- filteredRepeatsOut[,1:6] %>%\n mutate(nStart = ifelse(End < Start, End, Start),\n nEnd = ifelse(End < Start, Start, End)) %>%\n mutate(Start = nStart,\n End = nEnd) %>%\n select(! c(nStart, nEnd))\n\nwrite.table(filteredRepeatsOut2, file = filtBed, quote = FALSE, row.names = FALSE, sep = \"\\t\", col.names = FALSE)\n\n\n# Generate RM style summary table for large Repeats\n\nsine <- mergedRepeats %>% filter(str_detect(Repeat, \"SINE\")) %>% filter(! str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nline <- mergedRepeats %>% filter(str_detect(Repeat, \"LINE\")) %>% filter(! str_detect(Repeat, \"Penelope\")) %>% filter(! str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nltr <- mergedRepeats %>% filter(str_detect(Repeat, \"LTR\")) %>% filter(! str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\ndna <- mergedRepeats %>% filter(str_detect(Repeat, \"DNA\")) %>% filter(! str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nrc <- mergedRepeats %>% filter(str_detect(Repeat, \"RC\")) %>% filter(! str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nple <- mergedRepeats %>% filter(str_detect(Repeat, \"Penelope\")) %>% filter(! str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nunknown <- mergedRepeats %>% filter(str_detect(Repeat, \"Unknown\")) %>% filter(! str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nother <- mergedRepeats %>% filter(str_detect(Repeat, \"ARTEFACT|Low|Retrop|Satellite|Simple|RNA\")) %>% filter(! str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nother <- other[! other$Repeat == \"SINE\",]\n\nsine.nest <- mergedRepeats %>% filter(str_detect(Repeat, \"SINE\")) %>% filter(str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nline.nest <- mergedRepeats %>% filter(str_detect(Repeat, \"LINE\")) %>% filter(! str_detect(Repeat, \"Penelope\")) %>% filter(str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nltr.nest <- mergedRepeats %>% filter(str_detect(Repeat, \"LTR\")) %>% filter(str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\ndna.nest <- mergedRepeats %>% filter(str_detect(Repeat, \"DNA\")) %>% filter(str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nrc.nest <- mergedRepeats %>% filter(str_detect(Repeat, \"RC\")) %>% filter(str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nple.nest <- mergedRepeats %>% filter(str_detect(Repeat, \"Penelope\")) %>% filter(str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nunknown.nest <- mergedRepeats %>% filter(str_detect(Repeat, \"Unknown\")) %>% filter(str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nother.nest <- mergedRepeats %>% filter(str_detect(Repeat, \"ARTEFACT|Low|Retrop|Satellite|Simple|RNA\")) %>% filter(str_detect(Desc, \"FULLY_NESTED\")) %>% distinct()\nother.nest <- other.nest[! other.nest$Repeat == \"SINE\",]\n\n# Total bp coverage of each class\nsineCov <- sum(as.numeric(as.character(sine$size)))\nlineCov <- sum(as.numeric(as.character(line$size)))\nltrCov <- sum(as.numeric(as.character(ltr$size)))\ndnaCov <- sum(as.numeric(as.character(dna$size)))\nrcCov <- sum(as.numeric(as.character(rc$size)))\npleCov <- sum(as.numeric(as.character(ple$size)))\nunknownCov <- sum(as.numeric(as.character(unknown$size)))\notherCov <- sum(as.numeric(as.character(other$size)))\ntotalCov <- (sineCov + lineCov + ltrCov + dnaCov + rcCov + pleCov + unknownCov + otherCov) \nunmaskCov <- (as.numeric(genomeSize) - as.numeric(totalCov))\n\nsineCov.nest <- sum(as.numeric(as.character(sine.nest$size)))\nlineCov.nest <- sum(as.numeric(as.character(line.nest$size)))\nltrCov.nest <- sum(as.numeric(as.character(ltr.nest$size)))\ndnaCov.nest <- sum(as.numeric(as.character(dna.nest$size)))\nrcCov.nest <- sum(as.numeric(as.character(rc.nest$size)))\npleCov.nest <- sum(as.numeric(as.character(ple.nest$size)))\nunknownCov.nest <- sum(as.numeric(as.character(unknown.nest$size)))\notherCov.nest <- sum(as.numeric(as.character(other.nest$size)))\n\n# Number of elements\nsineTal <- as.numeric(tally(sine))\nlineTal <- as.numeric(tally(line))\nltrTal <- as.numeric(tally(ltr))\ndnaTal <- as.numeric(tally(dna))\nrcTal <- as.numeric(tally(rc))\npleTal <- as.numeric(tally(ple))\nunknownTal <- as.numeric(tally(unknown))\notherTal <- as.numeric(tally(other))\ntotalTal <- (sineTal + lineTal + ltrTal + dnaTal + rcTal + pleTal + unknownTal + otherTal)\n\nsineTal.nest <- as.numeric(tally(sine.nest))\nlineTal.nest <- as.numeric(tally(line.nest))\nltrTal.nest <- as.numeric(tally(ltr.nest))\ndnaTal.nest <- as.numeric(tally(dna.nest))\nrcTal.nest <- as.numeric(tally(rc.nest))\npleTal.nest <- as.numeric(tally(ple.nest))\nunknownTal.nest <- as.numeric(tally(unknown.nest))\notherTal.nest <- as.numeric(tally(other.nest))\n\n# Calculate seq percentage\ngenomeSize <- as.numeric(genomeSize)\nsinePer <- (sineCov / genomeSize) * 100\nlinePer <- (lineCov / genomeSize) * 100\nltrPer <- (ltrCov / genomeSize) * 100\ndnaPer <- (dnaCov / genomeSize) * 100\nrcPer <- (rcCov / genomeSize) * 100\nplePer <- (pleCov / genomeSize) * 100\nunknownPer <- (unknownCov / genomeSize) * 100\notherPer <- (otherCov / genomeSize) * 100\ntotalPer <- (sinePer + linePer + ltrPer + dnaPer + rcPer + plePer + unknownPer + otherPer)\nunmaskPer <- (100 - totalPer)\n\nsinePer.nest <- (sineCov.nest / genomeSize) * 100\nlinePer.nest <- (lineCov.nest / genomeSize) * 100\nltrPer.nest <- (ltrCov.nest / genomeSize) * 100\ndnaPer.nest <- (dnaCov.nest / genomeSize) * 100\nrcPer.nest <- (rcCov.nest / genomeSize) * 100\nplePer.nest <- (pleCov.nest / genomeSize) * 100\nunknownPer.nest <- (unknownCov.nest / genomeSize) * 100\notherPer.nest <- (otherCov.nest / genomeSize) * 100\n\n# Make Table \neleTyp <- c(\"SINE\", \"LINE\", \"LTR\", \"DNA\", \"RC\", \"PLE\", \"Unknown\", \"Total Interspersed Repeat\", \"Other\", \"Unmasked\",\n \"SINE-nested\", \"LINE-nested\", \"LTR-nested\", \"DNA-nested\", \"RC-nested\", \"PLE-nested\", \"Unknown-nested\", \"Other-nested\")\nnumEle <- c(sineTal, lineTal, ltrTal, dnaTal, rcTal, pleTal, unknownTal, totalTal, otherTal, as.character(\"notApplicable\"),\n sineTal.nest, lineTal.nest, ltrTal.nest, dnaTal.nest, rcTal.nest, pleTal.nest, unknownTal.nest, otherTal.nest)\nlenOcc <- c(sineCov, lineCov, ltrCov, dnaCov, rcCov, pleCov, unknownCov, totalCov, otherCov, unmaskCov,\n sineCov.nest, lineCov.nest, ltrCov.nest, dnaCov.nest, rcCov.nest, pleCov.nest, unknownCov.nest, otherCov.nest)\nperSeq <- c(sinePer, linePer, ltrPer, dnaPer, rcPer, plePer, unknownPer, totalPer, otherPer, unmaskPer,\n sinePer.nest, linePer.nest, ltrPer.nest, dnaPer.nest, rcPer.nest, plePer.nest, unknownPer.nest, otherPer.nest)\nsize <- c(rep(genomeSize, 18))\n\nsummary2 <- data.frame(eleTyp, numEle, lenOcc, perSeq, size)\ncolnames(summary2) <- c(\"Family\", \"Number of Elements\", \"Length Occupied\", \"Percentage of Sequence\", \"GenomeSize\")\n\n# Print table\nwrite.table(summary2, file = filtSum, col.names = TRUE, row.names = FALSE, quote = FALSE, sep = \"\\t\")\n"
},
{
"path": "scripts/rcMergeRepeats",
"content": "#!/bin/bash \n\nusage()\n{\n echo \"Usage: rcMergeRepeats -f genome.fasta -s species -d output_directory -u RepeatMasker.out -q RepeatMasker.tbl -c repeatcraft.cfg -t ThreadNum -b dictionary -m yes|no -e heliano_gff -h\"\n}\n\n\nwhile getopts f:s:d:u:q:t:b:m:e:h option\n do\n case \"${option}\"\n in\n f) genome=${OPTARG};;\n s) spe=${OPTARG};;\n d) dir=${OPTARG};;\n u) rmOut=${OPTARG};;\n q) table=${OPTARG};;\n t) threads=${OPTARG};;\n b) dict=${OPTARG};;\n m) cutoff=${OPTARG};;\n e) heli=${OPTARG};;\n h) usage; exit;; \n esac\n done\n\n##### Subprocess ltrfinder\n\nltrfinder()\n{\n mkdir ${dir}/${spe}LtrFinder/\n cd ${dir}/${spe}LtrFinder/\n ${SCRIPT_DIR}/LTR_FINDER_parallel -seq $genome -threads $threads \n cd ${dir}\n}\n\n##### Subprocess convert rmOut to GFF3 and then to GFF2\n\nconvert()\n{\n rmOutToGFF3.pl $rmOut > $dir/${spe}.out.gff3;\n cat $dir/${spe}.out.gff3 | sed '1,2d; s/Target=/Target \"Motif:/g' | awk '{OFS=\"\\t\"}{gsub(/$/,\"\\\"\", $10); print}' | sed \"s/Target\\t/Target /g; 1s/^/##gff-version 2\\n##date xxxx-xx-xx\\n/; s/\\(.*\\)\\t/\\1 /; s/\\(.*\\)\\t/\\1 /\" > ${dir}/${spe}.out.gff2\n sed -i '/##sequence-region.*/d' ${dir}/${spe}.out.gff2\n}\n\n\n##### Subprocess repeatcraft\n\nrepeatcraft()\n{\n mkdir ${dir}/${spe}RepeatCraft/\n sed \"s#ltr_finder_gff:.*#ltr_finder_gff: ${dir}/${spe}LtrFinder/$(basename ${genome}).finder.combine.gff3 #g\" ${SCRIPT_DIR}/repeatCraft/example/repeatcraft.cfg > ${dir}/${spe}RepeatCraft/repeatcraft.cfg\n python ${SCRIPT_DIR}/repeatCraft/repeatcraft.py -r ${dir}/${spe}.out.gff2 -u $rmOut -c ${dir}/${spe}RepeatCraft/repeatcraft.cfg -o ${dir}/${spe} -m strict\n}\n\n##### Subprocess mergeRepeats\n\nmergeRepeats()\n{\n sort -s -k1,1 ${dir}/${spe}.rmerge.gff > ${dir}/${spe}.rmerge.gff.sorted\n if [ -s \"$heli\" ]; then\n bedtools intersect -v -a ${dir}/${spe}.rmerge.gff.sorted -b $heli > ${dir}/${spe}.rmerge.gff.noHeli\n cat ${dir}/${spe}.rmerge.gff.noHeli $heli | sort -k4,4n > ${dir}/${spe}.rmerge.gff.sorted\n fi\n Rscript ${SCRIPT_DIR}/filteringOverlappingRepeats.R ${dir}/${spe}.rmerge.gff.sorted ${dir}/${spe}.rmerge.gff.filtered\n Rscript ${SCRIPT_DIR}/mergeRepeats.R ${dir}/${spe}.rmerge.gff.filtered ${dir}/${spe}.mergedRepeats.bed $genSize ${dir}/${spe}.mergedRepeats.revisedTable ${dir}/${spe}.filteredRepeats.bed ${dir}/${spe}.filteredRepeats.summary $cutoff\n} \n\n\n##### Subprocess scafDeedPoll\nscafDeedPoll()\n{\n cp ${dir}/${spe}.filteredRepeats.bed ${dir}/${spe}.filteredRepeats.bed.bak\n python ${SCRIPT_DIR}/backSwap.py ${dir}/${spe}.filteredRepeats.bed ${dict} ${dir}/${spe}.filteredRepeats.bed.2\n mv ${dir}/${spe}.filteredRepeats.bed{.2,}\n python ${SCRIPT_DIR}/backSwapGFF.py ${dir}/${spe}.rmerge.gff.filtered ${dict} ${dir}/${spe}.rmerge.gff.filtered.2\n mv ${dir}/${spe}.rmerge.gff.filtered{.2,}\n Rscript ${SCRIPT_DIR}/makeGff.R ${dir}/${spe}.filteredRepeats.bed ${dir}/${spe}.rmerge.gff.filtered ${dir}/${spe}.filteredRepeats.gff\n}\n\n##### Subprocess Running Cow\nrunningCow()\n{\n\techo \" \n ) (\n\t ( ) )\n\t ) ( (\n\t _______)_\n\t.-'---------| \n ( C|/\\/\\/\\/\\/|\n\t'-./\\/\\/\\/\\/|\n\t '_________'\n\t '-------'\n\t<<< $stage >>>\"\n}\n\n\n##### Subprocess Size\ngenomesize()\n{\n genSize=$(sed -n '4p' $table | rev | cut -f1,1 -d ':' | rev | sed 's/ bp.*//g; s/ //g')\n}\n\n##### Main\n\n# Step 1 - Run ltr_finder\n\nSCRIPT_DIR=/data/toby/EarlGrey/scripts\nstage=\"Running LTR Finder\"\nrunningCow\nltrfinder \n\n# Step 2 - Generate all respective GFF2 files\n \nstage=\"Generating GFF Files\"\nrunningCow\nconvert\n\n# Step 3 - Run RepeatCraft\n\nstage=\"Running RepeatCraft\"\nrunningCow\nrepeatcraft\n\n# Step 4 - Merge mergedRepeats\n\nstage=\"Resolving Overlapping Repeats\"\nrunningCow\ngenomesize\nmergeRepeats\nscafDeedPoll\n\n# Step 5 - Done\n\nstage=\"Done!\"\nrunningCow\n\n"
},
{
"path": "scripts/rcMergeRepeatsLoose",
"content": "#!/bin/bash \n\nusage()\n{\n echo \"Usage: rcMergeRepeats -f genome.fasta -s species -d output_directory -u RepeatMasker.out -q RepeatMasker.tbl -c repeatcraft.cfg -t ThreadNum -b dictionary -m yes|no -e heliano_gff -h\"\n}\n\n\nwhile getopts f:s:d:u:q:t:b:m:e:h option\n do\n case \"${option}\"\n in\n f) genome=${OPTARG};;\n s) spe=${OPTARG};;\n d) dir=${OPTARG};;\n u) rmOut=${OPTARG};;\n q) table=${OPTARG};;\n t) threads=${OPTARG};;\n b) dict=${OPTARG};;\n m) cutoff=${OPTARG};;\n e) heli=${OPTARG};;\n h) usage; exit;; \n esac\n done\n\n##### Subprocess ltrfinder\n\nltrfinder()\n{\n mkdir ${dir}/${spe}LtrFinder/\n cd ${dir}/${spe}LtrFinder/\n ${SCRIPT_DIR}/LTR_FINDER_parallel -seq $genome -threads $threads \n cd ${dir}\n}\n\n##### Subprocess convert rmOut to GFF3 and then to GFF2\n\nconvert()\n{\n rmOutToGFF3.pl $rmOut > $dir/${spe}.out.gff3;\n cat $dir/${spe}.out.gff3 | sed '1,2d; s/Target=/Target \"Motif:/g' | awk '{OFS=\"\\t\"}{gsub(/$/,\"\\\"\", $10); print}' | sed \"s/Target\\t/Target /g; 1s/^/##gff-version 2\\n##date xxxx-xx-xx\\n/; s/\\(.*\\)\\t/\\1 /; s/\\(.*\\)\\t/\\1 /\" > ${dir}/${spe}.out.gff2\n sed -i '/##sequence-region.*/d' ${dir}/${spe}.out.gff2\n}\n\n\n##### Subprocess repeatcraft\n\nrepeatcraft()\n{\n mkdir ${dir}/${spe}RepeatCraft/\n sed \"s#ltr_finder_gff:.*#ltr_finder_gff: ${dir}/${spe}LtrFinder/$(basename ${genome}).finder.combine.gff3 #g\" ${SCRIPT_DIR}/repeatCraft/example/repeatcraft.cfg > ${dir}/${spe}RepeatCraft/repeatcraft.cfg\n python ${SCRIPT_DIR}/repeatCraft/repeatcraft.py -r ${dir}/${spe}.out.gff2 -u $rmOut -c ${dir}/${spe}RepeatCraft/repeatcraft.cfg -o ${dir}/${spe} -m loose\n}\n\n##### Subprocess mergeRepeats\n\nmergeRepeats()\n{\n sort -s -k1,1 ${dir}/${spe}.rmerge.gff > ${dir}/${spe}.rmerge.gff.sorted\n if [ -s \"$heli\" ]; then\n bedtools intersect -v -a ${dir}/${spe}.rmerge.gff.sorted -b $heli > ${dir}/${spe}.rmerge.gff.noHeli\n # for debugging\n bedtools intersect -a ${dir}/${spe}.rmerge.gff.sorted -b $heli > ${dir}/${spe}.rmerge.gff.notFilteredHeli\n cat ${dir}/${spe}.rmerge.gff.noHeli $heli | sort -k4,4n > ${dir}/${spe}.rmerge.gff.sorted\n fi\n Rscript ${SCRIPT_DIR}/filteringOverlappingRepeats.R ${dir}/${spe}.rmerge.gff.sorted ${dir}/${spe}.rmerge.gff.filtered\n Rscript ${SCRIPT_DIR}/mergeRepeats.R ${dir}/${spe}.rmerge.gff.filtered ${dir}/${spe}.mergedRepeats.bed $genSize ${dir}/${spe}.mergedRepeats.revisedTable ${dir}/${spe}.filteredRepeats.bed ${dir}/${spe}.filteredRepeats.summary $cutoff\n} \n\n\n##### Subprocess scafDeedPoll\nscafDeedPoll()\n{\n cp ${dir}/${spe}.filteredRepeats.bed ${dir}/${spe}.filteredRepeats.bed.bak\n python ${SCRIPT_DIR}/backSwap.py ${dir}/${spe}.filteredRepeats.bed ${dict} ${dir}/${spe}.filteredRepeats.bed.2\n mv ${dir}/${spe}.filteredRepeats.bed{.2,}\n python ${SCRIPT_DIR}/backSwapGFF.py ${dir}/${spe}.rmerge.gff.filtered ${dict} ${dir}/${spe}.rmerge.gff.filtered.2\n mv ${dir}/${spe}.rmerge.gff.filtered{.2,}\n Rscript ${SCRIPT_DIR}/makeGff.R ${dir}/${spe}.filteredRepeats.bed ${dir}/${spe}.rmerge.gff.filtered ${dir}/${spe}.filteredRepeats.gff\n}\n\n##### Subprocess Running Cow\nrunningCow()\n{\n\techo \" \n ) (\n\t ( ) )\n\t ) ( (\n\t _______)_\n\t.-'---------| \n ( C|/\\/\\/\\/\\/|\n\t'-./\\/\\/\\/\\/|\n\t '_________'\n\t '-------'\n\t<<< $stage >>>\"\n}\n\n\n##### Subprocess Size\ngenomesize()\n{\n genSize=$(sed -n '4p' $table | rev | cut -f1,1 -d ':' | rev | sed 's/ bp.*//g; s/ //g')\n}\n\n##### Main\n\n# Step 1 - Run ltr_finder\n\nSCRIPT_DIR=/data/toby/EarlGrey/scripts\nstage=\"Running LTR Finder\"\nrunningCow\nltrfinder \n\n# Step 2 - Generate all respective GFF2 files\n \nstage=\"Generating GFF Files\"\nrunningCow\nconvert\n\n# Step 3 - Run RepeatCraft\n\nstage=\"Running RepeatCraft\"\nrunningCow\nrepeatcraft\n\n# Step 4 - Merge mergedRepeats\n\nstage=\"Resolving Overlapping Repeats\"\nrunningCow\ngenomesize\nmergeRepeats\nscafDeedPoll\n\n# Step 5 - Done\n\nstage=\"Done!\"\nrunningCow\n\n"
},
{
"path": "scripts/repeatCraft/Dockerfile",
"content": "FROM python:3.6-alpine3.6\n\nRUN apk add --no-cache bash\n\nRUN mkdir /repeatcraft\nCOPY ./repeatcraft.py /repeatcraft\nRUN mkdir /repeatcraft/helper\nCOPY ./helper/* /repeatcraft/helper/\n\nWORKDIR \"/repeatcraft\"\n\n\n\n"
},
{
"path": "scripts/repeatCraft/LICENSE",
"content": "MIT License\n\nCopyright (c) 2018 Wai Yee Wong\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"
},
{
"path": "scripts/repeatCraft/README.md",
"content": "[](https://travis-ci.org/niccw/repeatcraftp)\n# RepeatCraft pipeline\n\n#### Introduction\nRepeatCraft pipeline incorporates tools for repeat element classification based on both sequence similarity and structural features. The pipeline defragments closely spanced repeat loci in the genomes, reconstructing longer copies thus allowing for a better annotation and sequence comparisons. A Shiny web application is also supplemented as a user interface for the RepeatCraft pipeline, and it allows easy access and interactive study of repeat elements. The web server is available at [http://131.130.65.134:3838/repeatcraft](http://131.130.65.134:3838/repeatcraft). You can also run it on your machine by installing the repeatcraft R package [https://github.com/niccw/repeatCraft](https://github.com/niccw/repeatCraft).\n\n#### What does RepeatCraft do?\n##### Merge repeat 'fragments'\n\nAfter the repeat-based repeat identification and annotation (i.e. RepeatModeler and RepeatMasker (Smit, 2013-2015)), fragments of repeats beloning to the same repeat family can be found in certain genomic regions (for example: blcok b,c,d in track B). These repeats are closely spanced and non-overlap in the consensus sequence. Multiple counting of repeats may affect the result of downstream analysis (e.g. age estimation). Therefore, we develope RepeatCraft as a easy-to-use tool for mering repeat fragments, with flexible parameter setting. Group labels are first added to the attribute field of repeats to be merge, then RepeatCraft merges the repeat based on the TE family label and LTR label (see below). The strict mode only merges consective repeats while loose mode also works on non-executive repeats, the annotation information of other repeats (i.e. simple repeats) between the fragments are not modified.\n\n##### Annotation from LTR_FINDER (optional but highly recommended)\n\nRepeatCraft makes use of the annotation information from LTR_FINDER (Xu and Wang, 2017), a popular structural-based LTR (Long Tandem Repeat) identification tool. It identify the full-length LTR by searching structural features. Using the results from LTR_FINDER, RepeatCraft merges the LTR annotated in the RepeatMasker GFF (which belong to the same LTR subgroup) and further decreases the degree of fragmentization. \n\n##### Other\nShort repeats are also labelled in the early step. Repeats with length less than certain size (default = 100bp, except simple repeats,low complexity, satellite and snRNA; user can define the threshold for each repeat class in **mapfile.tsv**) would have a label of *shortTE=T* in the attribute column. The *shortTE* label are not used in merging repeat fragments, user can study the distribution of short read on the RepeatCraft web application by uploading the output.\n\n###### General workflow of RepeatCraft\n![](\"figures/RepeatCraft_workflow_resize.png\")
\n\n***\n\n### Run RepeatCraft\nSimply run RepeatCraft by calling **repeatcraft.py** (python >= 3.5), it takes the following arguments and options:\n```\nusage: repeatcraft.py [-h] [-r RMGFF] [-u RMOUT] [-c CONFIG] [-o OUTPUT]\n [-m MODE]\n\nRepeatCraft pipeline for improving repeat elements annotation by defragments\nclosely spanced repeat elements,based on sequence similarity and structural\nfeatures from different annotators\n\noptional arguments:\n -h, --help show this help message and exit\n -r RMGFF, --rmgff RMGFF\n RepeatMasker GFF\n -u RMOUT, --rmout RMOUT\n RepeatMasker OUT\n -c CONFIG, --config CONFIG\n Configuration file\n -o OUTPUT, --output OUTPUT\n Output file name\n -m MODE, --mode MODE Merge mode. strict or loose. Default = loose\n```\nExample run:\n`./repeatcraft.py -r example_input.gff -u example_input.out -c repeatcraft.cfg -o example -m strict`\n\n### Run in docker container\n**Build image from Dockerfile**\n\n`git clone https://github.com/niccw/repeatcraftp.git && cd repeatcraftp`\n\n`docker build -t repeatcraft:v1 .`\n\n**Run repeatcraft inside docker conatiner**\n\n`docker run --rm -v \"PATH_TO_INPUT_FILES\":/repeatcraft/run repeatcraft:v1 ./repeatcraft.py -r run/your_input.gff -u run/your_input.out -c run/your_repeatcraft.cfg -o example -m strict`\n\n\n#### Inputs\nThe example folder contains the example inputs **example_input.gff**, **example_input.out** (GFF and OUT files from RepeatMasker), **example_ltrfinder.gff** (LTR_FINDER output in GFF/GTF format), **repeatcraft.cfg** (configuration file) and **mapfile.tsv** (which is used in short TEs labelling) . Currently RepeatCraft only supports GFF generated by RepeatMasker v4.0.7 and v4.0.6. RepeatCraft also works on GFF made by RepeatMasker using custom library generated by RepeatModeler. However, the repeat family names should not contain either space of pipe (\"|\").\nThe table output from LTR_FINDER can be converted to GFF using the `cnv_ltrfinder2gff.pl` script from the [DAWGPAWS pipeline](https://github.com/jestill/dawgpaws). \n\n\n#### Outputs\nRepeatCraft generates 3 outputs:\n1) rclabel.gff : This is the reformatted GFF from Repeatmasker (type column is the repeat class but not \"similarity\"), with labels (TEgroup, LTRgroup) added to the attr. column. The TEs are not merged in this GFF.\n2) rmerge.gff : This is the GFF after merging based on the labels in rclabel.gff.\n3) summary.txt : This summary file lists the number of TEs (by class) before and after merging. It also shows how many TEs are merged by family (TEgroup) and/or based on information from LTR_FINDER (LTRgroup).\n\nThe example outputs (**example.rclabel.gff**, **example.rmerge.gff** and **example.summary.txt** can be found in the example folder)\n\nBy default, RepeatCraft is in \"loose\" merging mode. Repeats are merged based on the labels (TEgroup, LTRgroup) added from the previous steps, and the strand (+/-) will follow the orientation of the fragment with longest consensus overlapping in the group.\n\n\n#### Parameters in repeatcraft.cfg\nOther options can be set in the **repeatcraft.cgf**.\n```\n# Label short TEs\nshortTE_size: 100\nmapfile: None\n\n# LTR grouping (based on LTR_FINER result)\nltr_finder_gff: example_ltrfinder.gff\nmax_LTR_size: 10000\nLTR_flanking_size: 200\n\n# TEs grouping\ngap_size: 150\n```\n\nIf **mapfile.tsv** (see example) is missing, a unite size (default = 100 bp) will by applied to all repeat classes. Since certain repeat classes (i.e. SINE) have shorter length compare to others (i.e. LTR or LINE), it is recommend to label different classes using different length threshold.\n\nTo incoporate the LTR_FINDER annotation, the path (absolute path is preferable, if relative path is used it should be relative to where the **repeatcraft.py** is in) should be stated in **ltr_finder_gff** field. If no LTR_FINDER annotation is available, specify None in that field. The **max_LTR_size** helps filter out overlapping LTR annotation from LTR_FINDER, and **LTR_flanking_size** allows shift of LTR annotated by RepeatMasker. The following figure shows how these two parameters work.\n\n\n\nFinally, **gap_size** refer to the maximum distance between two constitute repeats to be merged, by default it is 150 bp.\n***\n#### References\nJC Estill and JL Bennetzen. 2009. \"The DAWGPAWS pipeline for the annotation of genes and transposable elements in plant genomes.\" Plant Methods. 5:8\n\nSmit, AFA, Hubley, R & Green, P. RepeatMasker Open-4.0. 2013-2015