Repository: TobyBaril/EarlGrey
Branch: main
Commit: d6d8654acc2b
Files: 89
Total size: 11.4 MB
Directory structure:
gitextract_2xvj2bh0/
├── .gitpod.yml
├── CITATION.cff
├── CODE_OF_CONDUCT.md
├── Docker/
│ ├── Dockerfile
│ ├── README.md
│ ├── configInstructions.txt
│ └── getFiles.sh
├── LICENSE
├── README.md
├── Singularity/
│ └── README.md
├── conda/
│ ├── build.sh
│ ├── developmentNotes.md
│ └── meta.yaml
├── earlGrey
├── earlGreyAnnotationOnly
├── earlGreyLibConstruct
├── legacyDocs.md
├── modules/
│ └── trf409.linux64
└── scripts/
├── LTR_FINDER_parallel
├── TEstrainer/
│ ├── README.md
│ ├── TEstrainer
│ ├── TEstrainer.Rproj
│ ├── TEstrainer_for_earlGrey.sh
│ ├── data/
│ │ ├── acceptable_domains.tsv
│ │ ├── additional_domains.tsv
│ │ ├── exceptional_domains.tsv
│ │ ├── old_acceptable_domains.tsv
│ │ └── unacceptable_domains.tsv
│ └── scripts/
│ ├── Dfam_extractor.py
│ ├── TEtrim.py
│ ├── domain_plotter.R
│ ├── final_sorter.R
│ ├── indexer.py
│ ├── initial_domain_finder.R
│ ├── initial_mafft_setup.py
│ ├── mreps_parser.sh
│ ├── simple_repeat_filter_trim.R
│ ├── splitter.py
│ ├── strainer.R
│ └── trf_parser.py
├── autoLand.R
├── autoPie.R
├── autoPie.sh
├── backSwap.py
├── backSwapGFF.py
├── bin/
│ ├── LTR_FINDER.x86_64-1.0.7/
│ │ ├── ltr_finder
│ │ └── tRNA.list.txt
│ └── cut.pl
├── divergenceCalc/
│ ├── divergence_calc.py
│ └── divergence_plot.R
├── extract_align.py
├── faswap.py
├── filteringOverlappingRepeats.R
├── headSwap.sh
├── install_r_packages.R
├── makeGff.R
├── mergeRepeats.R
├── rcMergeRepeats
├── rcMergeRepeatsLoose
├── repeatCraft/
│ ├── Dockerfile
│ ├── LICENSE
│ ├── README.md
│ ├── example/
│ │ ├── example.rclabel.gff
│ │ ├── example.rmerge.gff
│ │ ├── example.summary.txt
│ │ ├── example_input.gff
│ │ ├── example_input.out
│ │ ├── example_ltrfinder.gff
│ │ ├── mapfile.tsv
│ │ ├── repeatcraft.cfg
│ │ └── repeatcraft_strict.cfg
│ ├── helper/
│ │ ├── combineGFFoverlapm.py
│ │ ├── extraFuseTEm.py
│ │ ├── extraTrueMergeTEm.py
│ │ ├── filtershortm.py
│ │ ├── fuseltr.py
│ │ ├── fusetem.py
│ │ ├── rcStatm.py
│ │ ├── reformatm.py
│ │ ├── repeatcraftHelper.py
│ │ ├── truemergeltrm.py
│ │ ├── truemergeltrmErrorManagement.py
│ │ └── truemergetem.py
│ ├── repeatcraft.py
│ ├── repeatcraftErrorManagement.py
│ └── test/
│ ├── ci.sh
│ └── requirements.txt
├── repeatcraft.py
└── repeatcraftErrorManagement.py
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitpod.yml
================================================
image: gitpod/workspace-base
tasks:
- name: install mamba and earlgrey
init: |
curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-$(uname)-$(uname -m).sh"
bash Mambaforge-$(uname)-$(uname -m).sh -b -p /workspace/conda && rm Mambaforge-$(uname)-$(uname -m).sh
/workspace/conda/bin/mamba init bash
source ${HOME}/.bashrc
mamba create -n earlgrey -c conda-forge -c bioconda \
earlgrey -y
mamba clean --all -y
mamba activate earlGrey
cd /workspace
wget https://sk13.cog.sanger.ac.uk/NC_045808_EarlWorkshop.fasta
command: |
/workspace/conda/bin/mamba init bash
source ~/.bashrc
mamba activate earlgrey
clear
github:
prebuilds:
# enable for the master/default branch (defaults to true)
master: true
# enable for all branches in this repo (defaults to false)
branches: true
# enable for pull requests coming from this repo (defaults to true)
pullRequests: true
# enable for pull requests coming from forks (defaults to false)
pullRequestsFromForks: true
# add a "Review in Gitpod" button as a comment to pull requests (defaults to true)
addComment: true
# add a "Review in Gitpod" button to pull requests (defaults to false)
addBadge: false
# add a label once the prebuild is ready to pull requests (defaults to false)
addLabel: prebuilt-in-gitpod
vscode:
extensions:
- anwar.papyrus-pdf
workspaceLocation: "/workspace"
================================================
FILE: CITATION.cff
================================================
# This CITATION.cff file was generated with cffinit.
# Visit https://bit.ly/cffinit to generate yours today!
cff-version: 1.2.0
title: Earl Grey
message: >-
If you use this software, please cite it using the
metadata from this file.
type: software
version: "7.2.4"
authors:
- given-names: Tobias
family-names: Baril
email: toby.baril@icloud.com
affiliation: Université de Neuchâtel
orcid: 'https://orcid.org/0000-0002-5936-7531'
- given-names: James
family-names: Galbraith
affiliation: University of Exeter
orcid: 'https://orcid.org/0000-0002-1871-2108'
- given-names: Alex
family-names: Hayward
affiliation: University of Exeter
orcid: 'https://orcid.org/0000-0001-7413-718X'
================================================
FILE: CODE_OF_CONDUCT.md
================================================
# Contributor Covenant Code of Conduct
## Our Pledge
We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, religion, or sexual identity
and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.
## Our Standards
Examples of behavior that contributes to a positive environment for our
community include:
* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes,
and learning from the experience
* Focusing on what is best not just for us as individuals, but for the
overall community
Examples of unacceptable behavior include:
* The use of sexualized language or imagery, and sexual attention or
advances of any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email
address, without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.
Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.
## Scope
This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official e-mail address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement.
All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the
reporter of any incident.
## Enforcement Guidelines
Community leaders will follow these Community Impact Guidelines in determining
the consequences for any action they deem in violation of this Code of Conduct:
### 1. Correction
**Community Impact**: Use of inappropriate language or other behavior deemed
unprofessional or unwelcome in the community.
**Consequence**: A private, written warning from community leaders, providing
clarity around the nature of the violation and an explanation of why the
behavior was inappropriate. A public apology may be requested.
### 2. Warning
**Community Impact**: A violation through a single incident or series
of actions.
**Consequence**: A warning with consequences for continued behavior. No
interaction with the people involved, including unsolicited interaction with
those enforcing the Code of Conduct, for a specified period of time. This
includes avoiding interactions in community spaces as well as external channels
like social media. Violating these terms may lead to a temporary or
permanent ban.
### 3. Temporary Ban
**Community Impact**: A serious violation of community standards, including
sustained inappropriate behavior.
**Consequence**: A temporary ban from any sort of interaction or public
communication with the community for a specified period of time. No public or
private interaction with the people involved, including unsolicited interaction
with those enforcing the Code of Conduct, is allowed during this period.
Violating these terms may lead to a permanent ban.
### 4. Permanent Ban
**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior, harassment of an
individual, or aggression toward or disparagement of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within
the community.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.0, available at
https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
Community Impact Guidelines were inspired by [Mozilla's code of conduct
enforcement ladder](https://github.com/mozilla/diversity).
[homepage]: https://www.contributor-covenant.org
For answers to common questions about this code of conduct, see the FAQ at
https://www.contributor-covenant.org/faq. Translations are available at
https://www.contributor-covenant.org/translations.
================================================
FILE: Docker/Dockerfile
================================================
# Based on Dockerfile from Dfam TETools https://github.com/Dfam-consortium/TETools, with added database configuration, conda configuration, and Earl Grey installation
# and configuration
FROM debian:latest AS builder
RUN apt-get -y update && apt-get -y install \
curl gcc g++ make zlib1g-dev libgomp1 \
perl \
python3-h5py \
libfile-which-perl \
libtext-soundex-perl \
libjson-perl liburi-perl libwww-perl
# Install dependencies and some basic utilities
RUN apt-get -y update \
&& apt-get -y install \
aptitude \
libgomp1 \
perl \
python3-h5py \
libfile-which-perl \
libtext-soundex-perl \
libjson-perl liburi-perl libwww-perl \
libdevel-size-perl \
&& aptitude install -y ~pstandard ~prequired \
curl wget \
vim nano \
procps strace \
libpam-systemd-
COPY src/* /opt/src/
WORKDIR /opt/src
# Extract RMBlast
RUN cd /opt \
&& mkdir rmblast \
&& tar --strip-components=1 -x -f src/rmblast-2.13.0+-x64-linux.tar.gz -C rmblast \
&& rm src/rmblast-2.13.0+-x64-linux.tar.gz
# Compile HMMER
RUN tar -x -f hmmer-3.3.2.tar.gz \
&& cd hmmer-3.3.2 \
&& ./configure --prefix=/opt/hmmer && make && make install \
&& make clean
# Compile TRF
RUN tar -x -f trf-4.09.1.tar.gz \
&& cd TRF-4.09.1 \
&& mkdir build && cd build \
&& ../configure && make && cp ./src/trf /opt/trf \
&& cd .. && rm -r build
# Compile RepeatScout
RUN tar -x -f RepeatScout-1.0.6.tar.gz \
&& cd RepeatScout-1.0.6 \
&& sed -i 's#^INSTDIR =.*#INSTDIR = /opt/RepeatScout#' Makefile \
&& make && make install
# Compile and configure RECON
RUN tar -x -f RECON-1.08.tar.gz \
&& mv RECON-1.08 ../RECON \
&& cd ../RECON \
&& make -C src && make -C src install \
&& cp 00README bin/ \
&& sed -i 's#^\$path =.*#$path = "/opt/RECON/bin";#' scripts/recon.pl
# Compile cd-hit
RUN tar -x -f cd-hit-v4.8.1-2019-0228.tar.gz \
&& cd cd-hit-v4.8.1-2019-0228 \
&& make && mkdir /opt/cd-hit && PREFIX=/opt/cd-hit make install
# Compile genometools (for ltrharvest)
RUN tar -x -f gt-1.6.2.tar.gz \
&& cd genometools-1.6.2 \
&& make -i -j4 cairo=no && make -i cairo=no prefix=/opt/genometools install \
&& make cleanup
# Configure LTR_retriever
RUN cd /opt \
&& tar -x -f src/LTR_retriever-2.9.0.tar.gz \
&& mv LTR_retriever-2.9.0 LTR_retriever \
&& cd LTR_retriever \
&& sed -i \
-e 's#BLAST+=#BLAST+=/opt/rmblast/bin#' \
-e 's#RepeatMasker=#RepeatMasker=/opt/RepeatMasker#' \
-e 's#HMMER=#HMMER=/opt/hmmer/bin#' \
-e 's#CDHIT=#CDHIT=/opt/cd-hit#' \
paths
# Compile MAFFT
RUN tar -x -f mafft-7.505-without-extensions-src.tgz \
&& cd mafft-7.505-without-extensions/core \
&& sed -i 's#^PREFIX =.*#PREFIX = /opt/mafft#' Makefile \
&& make clean && make && make install \
&& make clean
# Compile NINJA
RUN cd /opt \
&& mkdir NINJA \
&& tar --strip-components=1 -x -f src/NINJA-cluster.tar.gz -C NINJA \
&& cd NINJA/NINJA \
&& make clean && make all
# Move UCSC tools
RUN mkdir /opt/ucsc_tools \
&& mv faToTwoBit twoBitInfo twoBitToFa /opt/ucsc_tools \
&& chmod +x /opt/ucsc_tools/*
#COPY LICENSE.ucsc /opt/ucsc_tools/LICENSE
# Compile and configure coseg
RUN cd /opt \
&& tar -x -f src/coseg-0.2.2.tar.gz \
&& cd coseg \
&& sed -i 's@#!.*perl@#!/usr/bin/perl@' preprocessAlignments.pl runcoseg.pl refineConsSeqs.pl \
&& sed -i 's#use lib "/usr/local/RepeatMasker";#use lib "/opt/RepeatMasker";#' preprocessAlignments.pl \
&& make -i
# Configure RepeatMasker
RUN cd /opt \
&& tar -x -f src/RepeatMasker-4.1.4.tar.gz \
&& chmod a+w RepeatMasker/Libraries \
&& cd RepeatMasker/Libraries \
&& cp /opt/src/Dfam.h5.gz . \
&& gunzip -f Dfam.h5.gz \
&& cd /opt/RepeatMasker \
#&& cp /opt/src/RepBaseRepeatMaskerEdition-20181026.tar.gz . \
#&& tar -zxf RepBaseRepeatMaskerEdition-20181026.tar.gz \
#&& wget https://github.com/rmhubley/RepeatMasker/blob/development/Libraries/RMRBMeta.embl \
#&& cd /opt/RepeatMasker \
&& mv famdb.py famdb.py.bak \
&& cp /opt/src/famdb.py . \
&& chmod 755 famdb.py \
&& perl configure \
-hmmer_dir=/opt/hmmer/bin \
-rmblast_dir=/opt/rmblast/bin \
-libdir=/opt/RepeatMasker/Libraries \
-trf_prgm=/opt/trf \
-default_search_engine=rmblast \
&& echo "RepeatMasker configured..." && cd .. && rm src/RepeatMasker-4.1.4.tar.gz
# Configure RepeatModeler
RUN cd /opt \
&& tar -x -f src/RepeatModeler-2.0.4.tar.gz \
&& mv RepeatModeler-2.0.4 RepeatModeler \
&& cd RepeatModeler \
&& perl configure \
-cdhit_dir=/opt/cd-hit -genometools_dir=/opt/genometools/bin \
-ltr_retriever_dir=/opt/LTR_retriever -mafft_dir=/opt/mafft/bin \
-ninja_dir=/opt/NINJA/NINJA -recon_dir=/opt/RECON/bin \
-repeatmasker_dir=/opt/RepeatMasker \
-rmblast_dir=/opt/rmblast/bin -rscout_dir=/opt/RepeatScout \
-trf_dir=/opt/ \
-ucsctools_dir=/opt/ucsc_tools
FROM debian:latest
# Install dependencies and some basic utilities
RUN apt update && apt-get -y install git
RUN apt-get -y update \
&& apt-get -y install \
aptitude \
libgomp1 \
perl \
python3-h5py \
libfile-which-perl \
libtext-soundex-perl \
libjson-perl liburi-perl libwww-perl \
libdevel-size-perl \
&& aptitude install -y ~pstandard ~prequired \
curl wget \
vim nano \
procps strace \
libpam-systemd-
RUN cd /opt/ \
&& curl https://repo.anaconda.com/archive/Anaconda3-2022.10-Linux-x86_64.sh --output anaconda.sh \
&& bash ./anaconda.sh -b -p /anaconda3 \
&& eval "$(/anaconda3/bin/conda shell.bash hook)"
ENV PATH=$PATH:/opt/RepeatMasker:/opt/RepeatMasker/util:/opt/RepeatModeler:/opt/RepeatModeler/util:/opt/coseg:/opt/ucsc_tools:/opt:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/cd-hit/
COPY --from=builder /opt /opt
RUN echo "PS1='(earlgrey \$(pwd))\\\$ '" >> /etc/bash.bashrc
ENV LANG=C
ENV PYTHONIOENCODING=utf8
RUN apt update && apt install build-essential -y --no-install-recommends
RUN apt update && apt install libcurl4-openssl-dev -y && apt-get install libbz2-dev liblzma-dev libtiff-dev libfreetype6-dev pkgconf libfontconfig1-dev libharfbuzz-dev libfribidi-dev bc -y
ARG USER_ID
ARG GROUP_ID
RUN addgroup --gid $GROUP_ID user
RUN adduser --disabled-password --gecos '' --uid $USER_ID --gid $GROUP_ID --home /home/user user
RUN chmod a+rwx /opt/*
RUN chmod a+rwx /home/user
RUN chown -R user /home/user
RUN cd /home/user \
&& git clone https://github.com/ridgelab/SA-SSR \
&& cd SA-SSR/ \
&& sed -i "s|PREFIX=/usr/local/bin|PREFIX=/home/user/SA-SSR/|g" Makefile && make && make install
RUN chmod a+rwx /home/user/SA-SSR/bin/* /home/user/SA-SSR/*
RUN chown -R user /home/user/SA-SSR/
USER user
WORKDIR /home/user
RUN git clone https://github.com/TobyBaril/EarlGrey \
&& cd EarlGrey \
&& chmod +x ./configureForDocker \
&& eval "$(/anaconda3/bin/conda shell.bash hook)" \
&& ./configureForDocker
ENV PATH=$PATH:/home/user/EarlGrey/:/home/user/SA-SSR/bin/
#RUN echo "#!/bin/bash" > /work/start.sh \
# && echo 'eval "$(/anaconda3/bin/conda shell.bash hook)"' >> /work/start.sh \
# && echo "conda activate earlGrey" >> /work/start.sh \
# && chmod +x /work/start.sh
================================================
FILE: Docker/README.md
================================================
# Docker Container
A Docker container has been generated with none of Dfam 3.9, but with script generation to source required partitions
I try to keep an up-to-date container in docker hub, but this might not always be the case depending on if I have had time to build and upload a new image. Currently, the recommended image ready for use is `-nodfam` version. Upon running the container interactively and running the command `earlGrey`, instructions will print to `stdout` and a script that you can use will be placed in `/usr/local/share/RepeatMasker/Libraries/famdb/` when the container is running.
```
# Interactive mode
# Version 7.2.4 with no preconfigured partitions (RECOMMENDED!) - bind a directory, in my case the current directory using pwd
docker run -it -v 'pwd':/data/ tobybaril/earlgrey:latest-nodfam
# change to library directory
cd /data/
# run earlGrey to make the configuration script
earlGrey
# then alter script with required partitions and run the configuration script
# change 0-16 to whichever you require, but at least 0. This relates to the partitions of Dfam 3.9 (https://www.dfam.org/releases/Dfam_3.9/families/FamDB/)
##### e.g. for 0-5:
sed -i '/^curl/ s/0-16/0-5/g' configure_dfam39.sh
##### e.g for 1,3,5:
sed -i '/^curl/ s/0-16/1,3,5/g' configure_dfam39.sh
# run the configuration script
bash configure_dfam39.sh
# return to your data directory
cd /data/
```
================================================
FILE: Docker/configInstructions.txt
================================================
# Get Earl Grey to run in Docker
# Download dependencies
./getFiles.sh
# NOTE: please make sure that Dfam.h5.gz and RepBase libraries tar.gz are inside the ./src directory after running ./getsrc.sh .
# If you need to download these,
# Dfam.h5.gz can be downloaded by running:
wget https://www.dfam.org/releases/Dfam_3.7/families/Dfam.h5.gz # THIS IS A BIG FILE!
# Repbase is now behind a paywall, if you do not have access, please comment out the following lines in the Dockerfile (lines 105-107)
# && cd /opt/RepeatMasker \
# && cp /opt/src/RepBaseRepeatMaskerEdition-20181026.tar.gz . \
# && tar -zxf RepBaseRepeatMaskerEdition-20181026.tar.gz \
# Build a docker container (run from inside the directory where the Dockerfile for EarlGrey is stored)
docker build . -t earlgrey
# start the docker container
docker run -it --rm --init --mount type=bind,source="$(pwd)",target=/work --user "$(id -u):$(id -g)" --workdir "/work" --env "HOME=/work" earlgrey "$@"
# IMPORTANT - once the docker container has started, run these commands to activate the conda environment for EarlGrey
# we strongly recommend upgrading the RepeatMasker contained within Docker with the Dfam libraries and RepBase, information on this is found: https://github.com/Dfam-consortium/TETools
# note there are only ~6000 seqs in the basic RepeatMasker library in this container, real Dfam is much larger!
eval "$(/anaconda3/bin/conda shell.bash hook)"
conda activate earlGrey
================================================
FILE: Docker/getFiles.sh
================================================
#!/bin/sh
set -eu
# download src name
download() {
src="$1"
shift
if [ $# -ge 1 ]; then
name="$1"
else
name="${src##*/}"
fi
dest="src/$name"
if [ -n "${ALWAYS-}" ] || ! [ -f "$dest" ]; then
echo "Downloading $src to $dest"
curl -sSL "$src" > "$dest"
fi
}
mkdir -p src
download https://www.repeatmasker.org/rmblast/rmblast-2.13.0+-x64-linux.tar.gz
download http://eddylab.org/software/hmmer/hmmer-3.3.2.tar.gz
download https://github.com/Benson-Genomics-Lab/TRF/archive/v4.09.1.tar.gz trf-4.09.1.tar.gz
download https://www.repeatmasker.org/RepeatScout-1.0.6.tar.gz
download https://www.repeatmasker.org/RepeatModeler/RECON-1.08.tar.gz
download https://github.com/weizhongli/cdhit/releases/download/V4.8.1/cd-hit-v4.8.1-2019-0228.tar.gz
download https://github.com/genometools/genometools/archive/v1.6.2.tar.gz gt-1.6.2.tar.gz
download https://github.com/oushujun/LTR_retriever/archive/v2.9.0.tar.gz LTR_retriever-2.9.0.tar.gz
download https://mafft.cbrc.jp/alignment/software/mafft-7.505-without-extensions-src.tgz
download https://github.com/TravisWheelerLab/NINJA/archive/0.97-cluster_only.tar.gz NINJA-cluster.tar.gz
download https://www.repeatmasker.org/coseg-0.2.2.tar.gz
download https://www.repeatmasker.org/RepeatMasker/RepeatMasker-4.1.4.tar.gz
download https://github.com/Dfam-consortium/RepeatModeler/archive/2.0.4.tar.gz RepeatModeler-2.0.4.tar.gz
download https://github.com/Dfam-consortium/FamDB/raw/master/famdb.py famdb.py
download https://www.dfam.org/releases/Dfam_3.7/families/Dfam_curatedonly.h5.gz Dfam.h5.gz
# TODO: /exe/ only includes binaries of the "latest" version at the time of download.
# The version listed in README.md is obtained by running 'strings src/faToTwoBit | grep kent'
# On whatever was downloaded.
# Consider building these tools from source instead.
for tool in faToTwoBit twoBitInfo twoBitToFa; do
download https://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64/"$tool"
done
================================================
FILE: LICENSE
================================================
Open Software License v. 2.1
============================
This Open Software License (the "License") applies to any original work
of authorship (the "Original Work") whose owner (the "Licensor") has
placed the following notice immediately following the copyright notice
for the Original Work: Licensed under the Open Software License version 2.1
1) Grant of Copyright License. Licensor hereby grants You a world-wide,
royalty-free, non-exclusive, perpetual, sublicenseable license to do
the following:
to reproduce the Original Work in copies;
to prepare derivative works ("Derivative Works") based upon the Original Work;
to distribute copies of the Original Work and Derivative Works to the
public, with the proviso that copies of Original Work or Derivative Works
that You distribute shall be licensed under the Open Software License;
to perform the Original Work publicly; and
to display the Original Work publicly.
2) Grant of Patent License. Licensor hereby grants You a world-wide,
royalty-free, non-exclusive, perpetual, sublicenseable license, under
patent claims owned or controlled by the Licensor that are embodied in
the Original Work as furnished by the Licensor, to make, use, sell and
offer for sale the Original Work and Derivative Works.
3) Grant of Source Code License. The term "Source Code" means the
preferred form of the Original Work for making modifications to it
and all available documentation describing how to modify the Original
Work. Licensor hereby agrees to provide a machine-readable copy of the
Source Code of the Original Work along with each copy of the Original
Work that Licensor distributes. Licensor reserves the right to satisfy
this obligation by placing a machine-readable copy of the Source Code in
an information repository reasonably calculated to permit inexpensive and
convenient access by You for as long as Licensor continues to distribute
the Original Work, and by publishing the address of that information
repository in a notice immediately following the copyright notice that
applies to the Original Work.
4) Exclusions From License Grant. Neither the names of Licensor, nor
the names of any contributors to the Original Work, nor any of their
trademarks or service marks, may be used to endorse or promote products
derived from this Original Work without express prior written permission
of the Licensor. Nothing in this License shall be deemed to grant any
rights to trademarks, copyrights, patents, trade secrets or any other
intellectual property of Licensor except as expressly stated herein. No
patent license is granted to make, use, sell or offer to sell embodiments
of any patent claims other than the licensed claims defined in Section
2. No right is granted to the trademarks of Licensor even if such marks
are included in the Original Work. Nothing in this License shall be
interpreted to prohibit Licensor from licensing under different terms
from this License any Original Work that Licensor otherwise would have
a right to license.
5) External Deployment. The term "External Deployment" means the use or
distribution of the Original Work or Derivative Works in any way such that
the Original Work or Derivative Works may be used by anyone other than
You, whether the Original Work or Derivative Works are distributed to
those persons or made available as an application intended for use over
a computer network. As an express condition for the grants of license
hereunder, You agree that any External Deployment by You of a Derivative
Work shall be deemed a distribution and shall be licensed to all under
the terms of this License, as prescribed in section 1(c) herein.
6) Attribution Rights. You must retain, in the Source Code of any
Derivative Works that You create, all copyright, patent or trademark
notices from the Source Code of the Original Work, as well as any
notices of licensing and any descriptive text identified therein as an
"Attribution Notice." You must cause the Source Code for any Derivative
Works that You create to carry a prominent Attribution Notice reasonably
calculated to inform recipients that You have modified the Original Work.
7) Warranty of Provenance and Disclaimer of Warranty. Licensor
warrants that the copyright in and to the Original Work and the patent
rights granted herein by Licensor are owned by the Licensor or are
sublicensed to You under the terms of this License with the permission
of the contributor(s) of those copyrights and patent rights. Except as
expressly stated in the immediately proceeding sentence, the Original
Work is provided under this License on an "AS IS" BASIS and WITHOUT
WARRANTY, either express or implied, including, without limitation,
the warranties of NON-INFRINGEMENT, MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY OF THE ORIGINAL
WORK IS WITH YOU. This DISCLAIMER OF WARRANTY constitutes an essential
part of this License. No license to Original Work is granted hereunder
except under this disclaimer.
8) Limitation of Liability. Under no circumstances and under no legal
theory, whether in tort (including negligence), contract, or otherwise,
shall the Licensor be liable to any person for any direct, indirect,
special, incidental, or consequential damages of any character arising
as a result of this License or the use of the Original Work including,
without limitation, damages for loss of goodwill, work stoppage, computer
failure or malfunction, or any and all other commercial damages or
losses. This limitation of liability shall not apply to liability for
death or personal injury resulting from Licensor's negligence to the
extent applicable law prohibits such limitation. Some jurisdictions do
not allow the exclusion or limitation of incidental or consequential
damages, so this exclusion and limitation may not apply to You.
9) Acceptance and Termination. If You distribute copies of the Original
Work or a Derivative Work, You must make a reasonable effort under the
circumstances to obtain the express assent of recipients to the terms of
this License. Nothing else but this License (or another written agreement
between Licensor and You) grants You permission to create Derivative Works
based upon the Original Work or to exercise any of the rights granted
in Section 1 herein, and any attempt to do so except under the terms of
this License (or another written agreement between Licensor and You)
is expressly prohibited by U.S. copyright law, the equivalent laws of
other countries, and by international treaty. Therefore, by exercising
any of the rights granted to You in Section 1 herein, You indicate Your
acceptance of this License and all of its terms and conditions. This
License shall terminate immediately and you may no longer exercise any
of the rights granted to You by this License upon Your failure to honor
the proviso in Section 1(c) herein.
10) Termination for Patent Action. This License shall terminate
automatically and You may no longer exercise any of the rights granted
to You by this License as of the date You commence an action, including
a cross-claim or counterclaim, against Licensor or any licensee alleging
that the Original Work infringes a patent. This termination provision
shall not apply for an action alleging patent infringement by combinations
of the Original Work with other software or hardware.
11) Jurisdiction, Venue and Governing Law. Any action or suit relating to this License may be brought only in the courts of a jurisdiction wherein the Licensor resides or in which Licensor conducts its primary business, and under the laws of that jurisdiction excluding its conflict-of-law provisions. The application of the United Nations Convention on Contracts for the International Sale of Goods is expressly excluded. Any use of the Original Work outside the scope of this License or after its termination shall be subject to the requirements and penalties of the U.S. Copyright Act, 17 U.S.C. § 101 et seq., the equivalent laws of other countries, and international treaty. This section shall survive the termination of this License.
12) Attorneys Fees. In any action to enforce the terms of this License or
seeking damages relating thereto, the prevailing party shall be entitled
to recover its costs and expenses, including, without limitation,
reasonable attorneys' fees and costs incurred in connection with such
action, including any appeal of such action. This section shall survive
the termination of this License.
13) Miscellaneous. This License represents the complete agreement
concerning the subject matter hereof. If any provision of this License
is held to be unenforceable, such provision shall be reformed only to
the extent necessary to make it enforceable.
14) Definition of "You" in This License. "You" throughout this License,
whether in upper or lower case, means an individual or a legal entity
exercising rights under, and complying with all of the terms of, this
License. For legal entities, "You" includes any entity that controls,
is controlled by, or is under common control with you. For purposes of
this definition, "control" means (i) the power, direct or indirect, to
cause the direction or management of such entity, whether by contract
or otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
15) Right to Use. You may use the Original Work in all ways not otherwise
restricted or conditioned by this License or by law, and Licensor promises
not to interfere with or be responsible for such uses by You.
This license is Copyright (C) 2003-2004 Lawrence E. Rosen. All rights
reserved. Permission is hereby granted to copy and distribute this
license without modification. This license may not be modified without
the express written permission of its copyright owner.
================================================
FILE: README.md
================================================
[](https://zenodo.org/badge/latestdoi/412126708) [](https://anaconda.org/bioconda/earlgrey) [](
https://anaconda.org/bioconda/earlgrey) [](https://anaconda.org/bioconda/earlgrey) [](https://anaconda.org/bioconda/earlgrey)
# Earl Grey
Earl Grey is a full-automated transposable element (TE) annotation pipeline, leveraging the most widely-used tools and combining these with a consensus elongation process to better define _de novo_ consensus sequences when annotating new genome assemblies.
# Contents
[Changes in Latest Release](#changes-in-latest-release)
[Example Run](#example)
[References and Acknowledgements](#references-and-acknowledgements)
[Usage Without Installation](#usage-without-installation)
[Recommended Installation](#recommended-installation-with-conda-or-mamba)
[Docker Container](#docker-container)
# Important Considerations
Earl Grey version 6 uses Dfam 3.9. After installation, you MUST configure Dfam partitions as needed. Earl Grey will generate the script to do this and provide guidance when you run it for the first time. You need to specify which partitions of Dfam and/or RepBase to configure Earl Grey with. Choose partitions carefully as the combination will highly influence your results, especially if you want to pre-mask your input genome. Please make use of issues and discussions tabs if you have questions about this, we are always happy to help!
# Notes / Updates
We often get questions related to runtime. TE curation and annotation remains resource and time intensive. Fast is not necessarily better, and runtime is highly dependent on genome size, complexity, and repeat content. Runs will likely take longer than you might expect, and be very RAM-hungry. As some generic benchmarks, a 40Mb genome can take anywhere from a few hours to a day, 400Mb up to around 4-5 days, a 3Gb genome ~a week, and a 25Gb genome several weeks! Things will be running even if it doesn't look like they are. Each step checkpoints, so if you have server limits, you can resubmit the same script with the same parameters, and Earl Grey will skip completed steps. `TEstrainer` and the final `divergence calculator` use a lot of memory. Check carefully for OOM errors in the logs! As a rule of thumb, you need _at least_ 3GB of RAM _per thread_, with more being better. Therefore, 16 threads requires at least 48GB of RAM depending on repeat complexity of the input genome.
We have been made aware of some instability in repeat annotation percentages when high numbers of CPUs are employed in certain server environments. Please be sure to check logs carefully for instances of interruption. Known cases so far will show the following message:
```
OpenBLAS blas_thread_init: pthread_create failed for thread X of X: Resource temporarily unavailable
OpenBLAS blas_thread_init: ensure that your address space and process count limits are big enough (ulimit -a)
OpenBLAS blas_thread_init: or set a smaller OPENBLAS_NUM_THREADS to fit into what you have available
```
If you see this message, re-run analysis with less threads. Alternatively, you can modify your instance of the TEstrainer script `initial_mafft_setup.py` to add the following after `import os`:
```
os.environ['OPENBLAS_NUM_THREADS'] = '1'
```
# Changes in Latest Release
Earl Grey v7.2.4 improves pipeline robustness and RepeatModeler handling for small or fragmented genomes:
- **Proactive RepeatModeler sampling cap** (`earlGrey`, `earlGreyLibConstruct`): the `deNovo1` function previously used a nested retry loop that ran RepeatModeler up to three times, falling back to progressively smaller `-genomeSampleSizeMax` values after each failure. It is now replaced by a single proactive run. The samplable genome size (sum of contigs ≥ 40 kb, the threshold below which RepeatModeler discards contigs) is computed upfront using `awk`, and the appropriate `-genomeSampleSizeMax` cap is derived from the cumulative RECON round thresholds (none for ≥ 363 Mb; 81 Mb for ≥ 120 Mb; 27 Mb for ≥ 39 Mb; 9 Mb for ≥ 12 Mb; 3 Mb for < 12 Mb). RepeatModeler runs exactly once with the correct flag, eliminating wasted compute on small or fragmented genomes.
- **`set -eo pipefail`** added to `earlGrey`, `earlGreyLibConstruct`, and `earlGreyAnnotationOnly`: any command that fails mid-pipeline now immediately aborts execution, preventing downstream stages from running with missing or corrupt intermediate files.
- **`expr` replaced with `$(( ))`** throughout all three scripts: `expr N / 4` returns exit code 1 when the result is 0 (i.e. fewer than 4 threads requested), which would abort the pipeline under `set -e` before any tool was invoked. All thread-division expressions now use bash arithmetic `$(( ProcNum / 4 ))`.
- **`ls | head -n 1` pipelines guarded with `|| true`**: under `pipefail`, `ls` receives SIGPIPE (exit 141) when `head` exits after reading one line. All affected pipelines now append `|| true` to suppress this expected signal.
### Previous Changes
Earl Grey v7.2.3 fixes five bugs reported by users:
- **Off-by-one chunking in divergence_calc.py** ([#290](https://github.com/TobyBaril/EarlGrey/issues/290)): The previous floor-division chunking could produce one extra chunk when the row count was not evenly divisible by the thread count, causing the final chunk to run serially after all parallel workers had finished (effectively doubling wall-clock time in the worst case). Chunking now uses `np.array_split`, which always produces exactly `num_processes` chunks and distributes any remainder one row at a time.
- **OSError: AF_UNIX path too long in divergence_calc.py** ([#294](https://github.com/TobyBaril/EarlGrey/issues/294)): `pybedtools.set_tempdir()` sets Python's `tempfile.tempdir` globally. The `forkserver` start method creates its Unix domain socket under `tempfile.tempdir`, so deeply-nested output paths could exceed the 108-character AF_UNIX socket limit. The divergence calculator is now passed a short per-species path in `/tmp` via `-tmp`, keeping the socket path well within the limit regardless of output directory depth.
- **Crash when no nested TEs are found** ([#292](https://github.com/TobyBaril/EarlGrey/issues/292)): `filteringOverlappingRepeats.R` crashed with `object 'nested' not found` on genomes with low TE content where no fully-nested elements were detected. When the nesting-detection loop exits without finding any nesting, `bind_rows()` produces a zero-row data frame that lacks the `nested` and `nesting_round` columns, causing the downstream `mutate()` to fail. The fix guards the mutate block with an `nrow()` check — if no nested TEs are found the unnested output is written directly.
- **Temp file path errors in divergence_calc.py under Singularity and other environments** ([#289](https://github.com/TobyBaril/EarlGrey/issues/289)): All path constructions in `divergence_calc.py` used string concatenation (e.g. `temp_dir + "/qseqs"`), which produces double slashes when `temp_dir` already has a trailing slash. This is particularly common in Singularity where bound paths are often passed with trailing slashes. All 13 affected path constructions have been replaced with `os.path.join()`, which correctly handles trailing slashes and redundant separators in all environments.
- **TEstrainer parallel progress bar cluttering batch job logs** ([#293](https://github.com/TobyBaril/EarlGrey/issues/293)): A new `-q yes` flag on `earlGrey` (and `-q` on `TEstrainer_for_earlGrey.sh` directly) suppresses the GNU `parallel --bar` progress bar. This is particularly useful for `sbatch`/HPC log files where the animated bar produces thousands of redundant lines.
### Previous Changes
Earl Grey v7.2.1 patches a bug where the curated library directory was not created when an existing library was supplied via `-l` (without `-r`). In this case, `earlGreyAnnotationOnly` could fail when attempting to change into the directory during the final masking step. The directory is now created with `mkdir -p` before use, matching the fix already applied to the main `earlGrey` script.
### Older Changes
Earl Grey v7.2.0 significantly reduces peak RAM usage in two RAM-intensive components: TEstrainer and divergence_calc.py. These changes prevent OOM kills when running with large thread counts on memory-constrained compute nodes, with no change to output.
**TEstrainer / TEstrainer_for_earlGrey.sh:**
- The default `MEM_FREE` threshold raised from `200M` to `1G`. The previous value was lower than the startup cost of a single Python interpreter with heavy scientific libraries, making the guard ineffective.
- All GNU `parallel` calls in the BEAT curation loop (trf, initial_mafft_setup, mafft, TEtrim) now carry `--memfree ${MEM_FREE}`, throttling job dispatch when free RAM drops below the threshold.
- A RAM-cap guard is applied at startup: the requested thread count is capped based on available RAM (`free -m`) at an estimate of 800 MB per concurrent job, with a warning printed if a cap is applied.
**divergence_calc.py:**
- Switched from the default `fork` multiprocessing start method to `forkserver`. On Linux, `fork` duplicates the full parent address space (including the GFF DataFrame) into every worker; `forkserver` workers start clean and receive only a file path, eliminating N-fold GFF copies in RAM.
- GFF chunks are now serialised to temp TSV files on disk before the pool is created. The parent DataFrame is freed before any workers are launched, reducing parent RSS during the pool run.
- `pool.imap_unordered` replaces `pool.map`, allowing workers to be retired as they finish rather than all buffering results simultaneously.
- `maxtasksperchild=1` forces worker process restart after each chunk, releasing accumulated pybedtools handles and BioPython caches between chunks.
- Periodic `pybedtools.cleanup()` every 500 rows prevents temp-file accumulation in long-running workers.
- A pre-existing bug was fixed: `os.remove(query_path)` was called unconditionally in both cleanup branches even when the file was never created (e.g. when `pybedtools.sequence()` raised a samtools exception). Both removal calls are now guarded with `if exists(query_path)`.
Benchmarked results confirm peak RSS for `divergence_calc.py` is flat at ~76 MB across 1–16 threads (previously scaling linearly), and TEstrainer peak RSS is ~877 MB at both 4 and 32 threads (previously unbounded with thread count).
Earl Grey v7.1.1 patches a small bug in TEstrainer where consensus sequences comprised of tandem repeats were triggering a warning due to the change to `pandas >2.0`. The output results were not affected. The codebase has now been updated to handle these cases without throwing warnings, with the same expected behaviour and handling of tandem repeats as before.
Earl Grey v7.1.0 removes the dependency on Python 3.9, which is no longer supported. Earl Grey can now be built and run with Python 3.10 and above. This change was made to ensure that Earl Grey remains compatible with newer versions of Python and to remove the reliance on an unsupported version.
Earl Grey v7.0.3 fixes an issue with final calculation tables which did not count _other_ repeats towards total repeat content.
Earl Grey v7.0.2 adds RepeatLandscapes for _Penelope_-like elements and SINEs. Importantly, the `-norna` option in RepeatMasker is no longer invoked as default behaviour, which will improver the detection and masking of small tRNA-derived SINEs.
Earl Grey v7.0.1 patches the summary table generation, where LINEs and Penelopes were being counted in both categories for nested repeats only.
☕ Earl Grey v7.0.0 is here!
Some long-awaited changes in this release—thank you for your patience while I found the time to properly test and implement them.
🐞 RepeatCraft fixes
First, I’ve fixed an issue in RepeatCraft where distal TEs could be grouped erroneously. This occurred in rare edge cases where the internal counter failed to iterate when new distal copies of an existing TE were detected on the same contig. This should now be fully resolved.
🧬 Fully nested TE handling (major update)
I’ve also completely revamped the Earl Grey post-processing and summary steps to properly deal with Fully Nested TEs.
Partially overlapping TEs are handled as before, following the approach described in the original Earl Grey paper.
Fully nested TEs are now identified using a new iterative process:
The GFF is scanned for nested TEs
These are labelled and stored in a separate file
The nested TE is removed and the GFF is re-scanned to detect deeper (multi-level) nesting
This continues until no new nested TEs are found
📊 Changes to coverage and summaries (important)
Nested TEs are no longer included in the TE coverage calculation used to generate the pie chart in the `summaryFiles` directory.
Instead:
Summary tables now include new categories showing how many base pairs are comprised of nested TEs
These base pairs are not counted toward Total Interspersed Repeats, as doing so would result in double-counting genomic space.
⚠️ This represents a substantial change from previous versions, so please be aware of this difference when upgrading to v7.
The output table summary now has the following format:
```
|TE Classification | Coverage (bp)|Copy Number | % Genome Coverage| Genome Size| TE Family Count|
|:-------------------------------------------------|-------------:|:-------------|-----------------:|-----------:|---------------:|
|DNA | 80886|326 | 0.7607795| 10631990| 326|
|DNA-nested | 1449|29 | 0.0136287| 10631990| 29|
|Rolling Circle | 8022|31 | 0.0754515| 10631990| 31|
|Rolling Circle-nested | 0|0 | 0.0000000| 10631990| NA|
|Penelope | 112822|812 | 1.0611560| 10631990| 812|
|Penelope-nested | 1268|13 | 0.0119263| 10631990| 13|
|LINE | 132133|426 | 1.2427871| 10631990| 426|
|LINE-nested | 1268|13 | 0.0119263| 10631990| 4|
|SINE | 0|0 | 0.0000000| 10631990| NA|
|SINE-nested | 0|0 | 0.0000000| 10631990| NA|
|LTR | 0|0 | 0.0000000| 10631990| NA|
|LTR-nested | 0|0 | 0.0000000| 10631990| NA|
|Other (Simple Repeat, Microsatellite, RNA) | 189999|4241 | 1.7870502| 10631990| 4241|
|Other (Simple Repeat, Microsatellite, RNA)-nested | 1225|34 | 0.0115218| 10631990| 34|
|Unclassified | 274026|1291 | 2.5773726| 10631990| 1291|
|Unclassified-nested | 0|0 | 0.0000000| 10631990| NA|
|Total Interspersed Repeat | 607889|2886 | 5.7175468| 10631990| NA|
|Non-Repeat | 10024101|notApplicable | 94.2824532| 10631990| NA|
```
In the final GFF output, nested TEs are clearly labelled with a `nested=FULLY_NESTED` attribute in column 9, enabling quick identification and downstream filtering.
```
NC_045808.1 Earl_Grey Simple_repeat 51840 51871 14 + . ID=(CGCA)N_33;Name=(CGCA)N;tstart=1;tend=31;shortte=F;nested=FULLY_NESTED-ROUND1
```
As always, thank you to the TE community for your enthusiasm in using Earl Grey, and for your invaluable feedback and bug reports. I’ll continue to incorporate improvements and fixes as quickly—and carefully—as possible.
Happy New Year! 🎉
Earl Grey v6.3.6 patches a RepeatCraft bug that arises extremely rarely in specific genomes, linked to dictionary initialisation.
Earl Grey v6.3.5 patches the annotation only pipeline to use the correct divergence calculator when a custom library is used without a RepBase term.
Earl Grey v6.3.4 adds small patches to correct TE family count table, which was previously using ID rather than Name. Also, a new awk one-liner has been added to clean the final GFF to make all attributes compliant with genometools, geneious etc.
Earl Grey v6.3.3 adds small updates to improve user-friendliness when installing and running for the first time.
Earl Grey v6.3.0 makes some important changes. Firstly, RepeatMasker now runs without checking for IS element contamination `-no_is`. In final GFF files, each insertion is now given a unique ID with `ID=`. The TE family is designated with `Name=`. This should enable parsing with Geneious or other visualisers. Other script changes are related to this change in field designation.
Earl Grey v6.2.0 now includes better checkpoints for TEstrainer. In the event of an interrupted run, resubmit your `earlGrey` command with _exactly_ the same parameters and Earl Grey will skip successfully completed steps. With this new update, TEstrainer intermediate files will no longer be deleted and a new run started from scratch. Now, TEstrainer can be recovered mid-run to save time and resources.
Earl Grey v6.1.1 patches a bug where threads set to <4 caused TEstrainer to crash (only present in 6.1.0 and not earlier versions)
Earl Grey v6.1.0 reintroduces the `--curated` flag when known elements are used to pre-mask your input genome. As usual, unless a good quality TE library already exists for your species of interest, or a _very_ closely related one, we do not recommend pre-masking your input genome with known repeats. This reduces the amount of information available for _de novo_ annotation, and can lead to overestimations of TE divergence and lower-quality consensus sequences. If in doubt, leave this option out!
Earl Grey v6.0.3 reduces CPU usage for TEstrainer to reduce memory pressure.
Earl Grey v6.0.2 patches an issue where the use of existing libraries did not work with the new `famdb` formats.
Earl Grey v6.0.1 contains small bug fixes to verify installed RepeatMasker Libraries correctly. There is now a Docker container for Earl Grey v6.0.1 that contains all partitions of Dfam (so it is BIG!).
*Earl Grey v6.0.0 is here!*
There are some relatively large changes in this release, resulting in the jump to v6.0.0.
Importantly, Earl Grey has been updated to use *Dfam version 3.9*, with RepeatMasker 4.1.8 and famdb 2.0.1. This means that there is some extra configuration required to get the pipeline running. Upon first installation and running of Earl Grey, the pipeline will check whether RepeatMasker has been configured with the correct Dfam database partitions. If not, it will warn you, generate a script that you can modify and run to configure RepeatMasker, and provide instructions to `stdout` if you want to do this yourself.
Please take care to configure Earl Grey v6 with ALL required partitions. More information on the partitioning can be found at [Dfam.org](https://dfam.org/releases/current/families/FamDB/README.txt).
Earl Grey v5.1.1 will continue to work for those who are happy with Dfam v3.7, but we recommend upgrading to v6.0.0 to keep up to date with the latest improvements to the database.
Earl Grey v5.1.1 contains very small patches to improve compatibility with publicly available genome sequencing data. In rare instances, strange characters in fasta headers were causing issues preventing the pipeline from running. These have been resolved in the preparation step.
In addition, new _pretty_ tables are now generated in the `summaryFiles` directory and at the end of a successful run. These contain the same information as the `txt` tables, but in the familiar pipe format for readability. These can be added to markdown files if required. One is produced for the high level count as well as for the family level count. Below is an example of the table that is printed at the end of Earl Grey runs as of `v5.1.1`:
```
|TE Classification | Coverage (bp)| Copy Number| % Genome Coverage| Genome Size| TE Family Count|
|:------------------------------------------|-------------:|-----------:|-----------------:|-----------:|---------------:|
|DNA | 69284| 190| 0.6516560| 10631990| 9|
|Rolling Circle | 71788| 316| 0.6752076| 10631990| 12|
|Penelope | 139334| 869| 1.3105167| 10631990| 6|
|LINE | 141841| 491| 1.3340964| 10631990| 22|
|SINE | 44137| 191| 0.4151339| 10631990| 1|
|Other (Simple Repeat, Microsatellite, RNA) | 191645| 4235| 1.8025318| 10631990| 940|
|Unclassified | 155836| 905| 1.4657275| 10631990| 35|
|Non-Repeat | 9818125| NA| 92.3451301| 10631990| NA|
```
Earl Grey v5.1.0 contains small changes that drastically improve memory usage in the divergence calculator. We have replaced the use of EMBOSS `water` with EMBOSS `matcher`, which reduces memory consumption on large alignments whilst remaining rigorous. For more information, please see the notes section in the [EMBOSS Manual](https://www.bioinformatics.nl/cgi-bin/emboss/help/matcher). This should prevent jobs running out of memory, particularly when using queuing systems and shared resources.
Big changes in the latest release!
*Earl Grey v5.0.0 is here!*
This release incorporates the incremental improvements made throughout the life of version 4.
It is now possible to run some subroutines in Earl Grey (run either of these new commands with `-h` to see a list of options):
- `earlGreyLibConstruct` can be used to run Earl Grey for _de novo_ TE detection, consensus generation, and improvement through the BEAT process. The output will be the strained TE consensus sequences, which can then be used for subsequent annotation. This is useful when you want to make a combined library from the libraries of several different genomes, where it is no longer required to waste time running annotations. Once the libraries are generated and you have curated them, you can then run the next step in isolation (next point!).
- `earlGreyAnnotationOnly` can be used to run the final annotation and defragmentation steps in Earl Grey. This is useful if you have already run the BEAT process and have a library of _de novo_ TE consensus sequences that you would like to use to annotate a given genome. This script is also compatible with the `-r` flag to take known repeats from the databases used to configure RepeatMasker in addition to the custom repeat library.
- *EXPERIMENTAL FEATURE:* I have also added an option to run [HELIANO](https://github.com/Zhenlisme/heliano) for improved detection of Helitrons, which are notoriously difficult to detect and classify using homology methods. This can be implemented by adding `-e yes` to the command line options after upgrading to v5.0.0. Currently, HELIANO annotations replace those which they overlap following the RepeatMasker run, which is performed during defragmentation (in a similar way to full-length LTRs being dealt with in `RepeatCraft`). Feedback is welcomed on this implementation, and I am continuing to test and improve the implementation of HELIANO within Earl Grey.
- The settings used for HELIANO are: `--nearest -dn 6000 -flank_sim 0.5 -w 10000`. These can be modified in the earlGrey script of your specific installation.
Thank you for your continued support and enthusiasm for Earl Grey!
# Example
Given an input genome, Earl Grey will run through numerous steps to identify, curate, and annotate transposable elements (TEs). We recommend running earlGrey within a tmux or screen session, so that you can log off and leave Earl Grey running.
There are several required and optional parameters for your Earl Grey run:
```
Required Parameters:
-g == genome.fasta
-s == species name
-o == output directory
Optional Parameters:
-t == Number of Threads (DO NOT specify more than are available)
-r == RepeatMasker search term (e.g arthropoda/eukarya)
-l == Starting consensus library for an initial mask (in fasta format)
-i == Number of Iterations to BLAST, Extract, Extend (Default: 10)
-f == Number flanking basepairs to extract (Default: 1000)
-c == Cluster TE library to reduce redundancy? (yes/no, Default: no)
-m == Remove putative spurious TE annotations <100bp? (yes/no, Default: no)
-d == Create soft-masked genome at the end? (yes/no, Default: no)
-n == Max number of sequences used to generate consensus sequences (Default: 20)
-a == minimum number of sequences required to build a consensus sequence (Default: 3)
-e == Optional: Run HELIANO for detection of Helitrons (yes/no, Default: no)
-h == Show help
```
```
# remember to activate the conda environment before running (NOTE: depending on your install, the environment name might vary)
conda activate earlgrey
# run earl grey with minimum command options
earlGrey -g [genome.fasta] -s [speciesName] -o [outputDirectory]
# e.g
earlGrey -g myzusPersicae.fasta -s myzusPersicae -o ./earlGreyOutputs
```
Following this, Earl Grey will run through several processes depending on the options selected. The general pipeline is illustrated below:
For a more in-depth description of Earl Grey's steps, please refer to the implementation section in the [manuscript](https://academic.oup.com/mbe/article/41/4/msae068/7635926).
The runtime of Earl Grey will depend on the repeat content of your input genome. Once finished, you will notice that a number of directories have been created by Earl Grey. The most important results are found within the "summaryFiles" folder, however intermediate results are kept in case you wish to use alignments for further manual curation or investigation, for example. NOTE: RepeatModeler2 remains the rate-limiting step, with runtimes leading into several days, or even weeks, with large repeat-rich genomes. This is normal.
Directories created by earl grey:
```
[speciesName]EarlGrey/
|
|---[speciesName]_RepeatMasker/ [OPTIONAL: only created with -r or -l flags]
| + Results of the initial RepeatMasker run used to pre-mask previously characterised TEs
|---[speciesName]_Database/
| + BLAST database built from the (optionally pre-masked) genome, used by RepeatModeler
| + RepeatModeler2 consensus output ([speciesName]-families.fa, [speciesName]-families.stk, [speciesName]-rmod.log)
|---[speciesName]_RepeatModeler/
| + RepeatModeler2 working directory (RM_* subdirectory with per-round outputs and intermediate consensi)
|---[speciesName]_strainer/
| + TEstrainer working directory (TS_* subdirectory) containing BEAT curation intermediate files
| + Strained consensus library ([speciesName]-families.fa.strained)
|---[speciesName]_Curated_Library/ [OPTIONAL: only created with -r or -l flags]
| + Known repeat library extracted from the configured database ([speciesName].RepeatMasker.lib)
| + Combined _de novo_ and known repeat library ([speciesName]_combined_library.fasta)
|---[speciesName]_heliano/ [OPTIONAL: only created with -e yes flag]
| + HELIANO Helitron detection results (HEL_* subdirectory)
|---[speciesName]_RepeatMasker_Against_Custom_Library/
| + Results of RepeatMasker run against the final curated repeat library (.out, .tbl, .cat.gz, .masked)
|---[speciesName]_RepeatLandscape/
| + Divergence-annotated GFF ([speciesName].filteredRepeats.withDivergence.gff)
| + Repeat landscape PDFs (classification landscape, split-class landscape, superfamily divergence plot)
| + Divergence summary table ([speciesName]_summary_table.tsv)
|---[speciesName]_mergedRepeats/
| + looseMerge/ subdirectory containing:
| - LTR_FINDER full-length LTR results (ltrfinder GFF and working subdirectory)
| - RepeatCraft TE defragmentation output (RepeatCraft working subdirectory, merged GFFs and BEDs)
| - Final filtered and merged repeat annotations (filteredRepeats.bed, filteredRepeats.gff, filteredRepeats.summary)
|---[speciesName]_summaryFiles/
+ Final TE annotations in GFF3 format ([speciesName].filteredRepeats.gff)
(fully nested TEs labelled with 'nested=FULLY_NESTED' attribute in column 9)
+ Final TE annotations in BED format ([speciesName].filteredRepeats.bed)
+ High-level TE quantification table (tab-delimited .txt and markdown-formatted .kable)
+ Family-level TE quantification table (tab-delimited .txt and markdown-formatted .kable)
+ Divergence summary table ([speciesName]_divergence_summary_table.tsv)
+ Pie chart of genome repeat content ([speciesName].summaryPie.pdf)
+ Repeat landscape PDFs (classification landscape, split-class landscape, superfamily divergence plot)
+ _de novo_ strained TE library ([speciesName]-families.fa.strained)
+ Combined repeat library ([speciesName]_combined_library.fasta) [OPTIONAL: only with -r or -l flags]
+ Soft-masked genome FASTA ([speciesName].softmasked.fasta) [OPTIONAL: only with --softMask yes]
```
As of `v4.4.5`, there is an option to generate _de novo_ TE libraries without running subsequent annotation. To run this option, use `earlGreyLibConstruct` with the same command-line options as `earlGrey`. This will run everything up to the end of TEstrainer, and leave you a `families-fa.strained` library file in the `summaryFiles` directory, which you can then use for manual curation, or for pangenome studies.
### Example Outputs (NOTE: example data has been used here):
- Pie chart summarising TE content in input genome
- RepeatLandscapes summarising relative TE activity using Kimura 2-Parameter Divergence (recent activity towards the RHS)
- TE annotations - These are in standard genomic information formats to be compatible with downstream analyses. NOTE: TE divergence calculated using Kimura 2-Parameter distance is now supplied for each insertion in column 9 of the GFF3 file:
```
# BED format
NC45808.1 4964941 4965925 LINE/Penelope 5073 +
NC45808.1 7291353 7291525 LINE/L2 1279 +
NC_045808.1 8922477 8923791 DNA/TcMar-Tc1 11957 +
# GFF3 format
scaffold_1 Earl_Grey DNA/Mariner 71618 71814 892 + . TSTART=13;TEND=233;ID=RND-1_FAMILY-1;SHORTTE=F;KIMURA80=0.2141
scaffold_1 Earl_Grey Unknown 81757 81927 785 - . TSTART=16;TEND=194;ID=RND-1_FAMILY-0;SHORTTE=F;KIMURA80=0.2037
```
# References and Acknowledgements
This pipeline has been designed to be used and shared openly by the community.
### When using Earl Grey, please cite:
Baril, T., Galbraith, J.G., and Hayward, A., Earl Grey: A Fully Automated User-Friendly Transposable Element Annotation and Analysis Pipeline, Molecular Biology and Evolution, Volume 41, Issue 4, April 2024, msae068 [doi:10.1093/molbev/msae068](https://doi.org/10.1093/molbev/msae068)
Baril, Tobias., Galbraith, James., and Hayward, Alexander. (2023) Earl Grey. Zenodo [doi:10.5281/zenodo.5654615](https://doi.org/10.5281/zenodo.5654615)
### This pipeline makes use of scripts from:
[RepeatCraft](https://github.com/niccw/repeatcraftp) - Wong WY, Simakov O. RepeatCraft: a meta-pipeline for repetitive element de-fragmentation and annotation. Bioinformatics 2018;35:1051–2. https://doi.org/10.1093/bioinformatics/bty745.
### The following open source software are utilised by this pipeline:
Smit AFA, Hubley RR, Green PR. RepeatMasker Open-4.0. Http://RepeatmaskerOrg 2013.
Flynn, J.M., Hubley, R., Goubert, C., Rosen, J., Clark, A.G., Feschotte, C. and Smit, A.F. RepeatModeler2 for automated genomic discovery of transposable element families. Proceedings of the National Academy of Sciences 2020;17;9451-9457. https://doi.org/10.1073/pnas.1921046117
Bao Z, Eddy SR. Automated De Novo Identification of Repeat Sequence Families in Sequenced Genomes. Genome Res 2002;12:1269–76. https://doi.org/10.1101/gr.88502.
Price AL, Jones NC, Pevzner PA. De novo identification of repeat families in large genomes. Bioinformatics 2005;21:i351–8.
Camacho C, Coulouris G, Avagyan V, Ma N, Papadopoulos J, Bealer K, et al. BLAST+: Architecture and applications. BMC Bioinformatics 2009;10: 1–9. https://doi.org/10.1186/1471-2105-10-421.
Katoh K, Standley DM. MAFFT multiple sequence alignment software version 7: Improvements in performance and usability. Mol Biol Evol 2013;30:772–80. https://doi.org/10.1093/molbev/mst010.
Capella-Gutiérrez S, Silla-Martínez JM, Gabaldón T. trimAl: A tool for automated alignment trimming in large-scale phylogenetic analyses. Bioinformatics 2009;25:1972–3. https://doi.org/10.1093/bioinformatics/btp348.
Rice P, Longden L, Bleasby A. EMBOSS: The European Molecular Biology Open Software Suite. Trends Genet 2000;16:276–7. https://doi.org/10.1016/S0168-9525(00)02024-2.
Xu Z, Wang H. LTR_FINDER: an efficient tool for the prediction of full-length LTR retrotransposons. Nucleic Acids Res 2007;35:W265–8. https://doi.org/10.1093/nar/gkm286.
Ou S, Jiang N. LTR_FINDER_parallel: parallelization of LTR_FINDER enabling rapid identification of long terminal repeat retrotransposons. BioRxiv 2019:2–6.
# Usage without installation
If you would like to try Earl Grey, or prefer to use it in a browser, you can do this through [gitpod](https://gitpod.io). You can get 50 hours free per month and use Earl Grey within a preconfigured environment. Simply select this repository by pasting the repository URL, and the system will be automatically configured for you. You can then upload your genome of interest, and run Earl Grey as you would on the command line.
# Recommended Installation with Conda or Mamba
Earl Grey version 6 uses Dfam 3.9. After installation, you MUST configure Dfam partitions as needed. Earl Grey will generate the script to do this and provide guidance when you run it for the first time. You need to specify which partitions of Dfam and/or RepBase to configure Earl Grey with. Choose partitions carefully as the combination will highly influence your results, especially if you want to pre-mask your input genome.
Earl Grey version 7.2.4 (latest stable release) with all required and configured dependencies is found in the `biooconda` conda channel. To install, simply run the following depending on your installation:
```
# With conda
conda create -n earlgrey -c conda-forge -c bioconda earlgrey=7.2.4
# With mamba
mamba create -n earlgrey -c conda-forge -c bioconda earlgrey=7.2.4
# Then run
earlGrey
# a script will be output to stdout and generated in the current directory to aid in setup
```
# Recommended Installation on ARM-based Mac Systems (M chips) Using Docker
It is possible to install x86-based Docker environments on M-chip Mac systems using Rosetta. The below form a guide that should work, but please reach out if you have any trouble!
Please first follow the Docker on Mac installation [instructions here](https://docs.docker.com/desktop/setup/install/mac-install/). Ensure you have installed Rosetta2, as this is required to get Earl Grey to behave as expected.
Next, we will create aliases to switch between arm and rosetta:
We want a simple way to activate the x86 architecture. We can do this by adding the following to `~/.zshrc`:
```
alias arm="env /usr/bin/arch -arm64 /bin/zsh --login"
alias intel="env /usr/bin/arch -x86_64 /bin/zsh --login"
```
Then close the terminal and open a new one.
To activate the intel/rosetta environment, run the following in a new terminal:
```
intel
```
Next, get the Docker installation and run in an interactive terminal following the instructions in the next section below. You will only need to pull the container once, then can use it for all your Earl Grey needs.
After this, you are ready to go! Just remember to activate the _intel_ terminal before starting the interactive container and running Earl Grey.
# Docker Container
A Docker container has been generated with none of Dfam 3.9, but with script generation to source required partitions
I try to keep an up-to-date container in docker hub, but this might not always be the case depending on if I have had time to build and upload a new image. Currently, the recommended image ready for use is `-nodfam` version. Upon running the container interactively and running the command `earlGrey`, instructions will print to `stdout` and a script that you can use will be placed in your current working directory. After an initial setup and configuration in an interative version of the container, you can commit the changes (i.e. the Dfam configuration) using `docker commit [container_ID] yourdockerusername/earlgrey:version7.2.4-configured`. Then, you can run this container interactively, or non-interatively, to annotate focal genomes.
```
# Interactive mode
# Version 7.2.4 with no preconfigured partitions (RECOMMENDED!) - bind a directory, in my case the current directory using pwd
docker run -it -v 'pwd':/data/ tobybaril/earlgrey:latest-nodfam
# change to library directory
cd /data/
# run earlGrey to make the configuration script
earlGrey
# then alter script with required partitions and run the configuration script
# change 0-16 to whichever you require, but at least 0. This relates to the partitions of Dfam 3.9 (https://www.dfam.org/releases/Dfam_3.9/families/FamDB/)
##### e.g. for 0-5:
sed -i '/^curl/ s/0-16/0-5/g' configure_dfam39.sh
##### e.g for 1,3,5:
sed -i '/^curl/ s/0-16/1,3,5/g' configure_dfam39.sh
# run the configuration script
bash configure_dfam39.sh
# return to your data directory
cd /data/
# to quit the container and leave it running so you can commit the configured changes
# ctrl + p; ctrl + q
# get container ID to commit (not necessarily required if you leave the container running. If you kill the container and start again, you would need to reconfigure the DFAM libraries, so this is recommended. However, the container will be BIG depending on which parts of DFAM you pull.
docker ps -a
# commit the modified container so you can use at will (replace yourdockerusername with your docker username)
docker commit [container_ID] yourdockerusername/earlgrey:version7.2.4-configured
# you can then run non-interatively if required:
docker run -v 'pwd':/data/ yourdockerusername/earlgrey:version7.2.4-configured earlGrey -g /data/GENOME.fasta -s nonInteractiveTest -o /data/ -t 8
# alternatively you can still run interactive sessions
docker run -it -v 'pwd':/data/ yourdockerusername/earlgrey:version7.2.4-configured
```
================================================
FILE: Singularity/README.md
================================================
# Singularity image for Earl Grey
A Singularity image can be built from the Docker image hosted on Docker Hub. The image contains no pre-configured Dfam partitions, but will generate a configuration script on first run of `earlGrey`. You will need to configure the Dfam libraries once, then you can use the configured sandbox for all subsequent analyses.
Wherever you start this is where your `/data` directory will point to inside the container. Make sure your genome assembly is found here, or change `$(pwd)` in the below commands to the directory where you want to perform your analysis.
**Write access is required inside the container to download and configure the Dfam partitions.** The recommended approach is to build a writable sandbox, configure Dfam, and then optionally convert the sandbox to a portable SIF file.
## First-time setup: building and configuring the sandbox
```bash
# Build a writable sandbox from the Docker image
singularity build --sandbox earlgrey_sandbox/ docker://tobybaril/earlgrey:latest-nodfam
# Run the sandbox interactively with write access, binding your working directory to /data
singularity shell -C --bind $(pwd):/data --writable earlgrey_sandbox/
# Inside the container: change to the data directory and run earlGrey to generate the configuration script
cd /data/
earlGrey
# Modify the configuration script to select the required Dfam 3.9 partitions
# (https://www.dfam.org/releases/Dfam_3.9/families/FamDB/)
# At minimum, partition 0 is required. Change 0-16 to whichever partitions you need.
##### e.g. for partitions 0-5:
sed -i '/^curl/ s/0-16/0-5/g' configure_dfam39.sh
##### e.g. for specific partitions 1, 3, and 5:
sed -i '/^curl/ s/0-16/1,3,5/g' configure_dfam39.sh
# Run the configuration script to download and install the selected Dfam partitions
bash configure_dfam39.sh
# Return to your data directory
cd /data/
# Exit the sandbox
exit
```
## Running Earl Grey after configuration
```bash
# Run the configured sandbox interactively
singularity shell -C --bind $(pwd):/data --writable earlgrey_sandbox/
# Or run non-interactively
singularity exec -C --bind $(pwd):/data --writable earlgrey_sandbox/ earlGrey -g /data/GENOME.fasta -s mySpecies -o /data/ -t 8
```
## Optional: convert the configured sandbox to a portable SIF
Once Dfam is configured, you can convert the sandbox to a read-only SIF file for easier distribution and deployment. Note that the resulting SIF will be large depending on which Dfam partitions were downloaded.
```bash
singularity build earlgrey_configured.sif earlgrey_sandbox/
# Run non-interactively with the configured SIF
singularity exec -C --bind $(pwd):/data earlgrey_configured.sif earlGrey -g /data/GENOME.fasta -s mySpecies -o /data/ -t 8
```
================================================
FILE: conda/build.sh
================================================
#!/bin/bash
#Based on https://github.com/TobyBaril/EarlGrey/blob/main/configure
set -x
# Define paths
PACKAGE_HOME=${PREFIX}/share/${PKG_NAME}-${PKG_VERSION}-${PKG_BUILDNUM}
SCRIPT_DIR="${PACKAGE_HOME}/scripts/"
# Create directories
mkdir -p ${PREFIX}/bin
mkdir -p ${PACKAGE_HOME}
# Put package in share directory
cp -rf * ${PACKAGE_HOME}/
# Install SA-SSR (has to be done here because SA-SSR is an ancient repository without releases)
git clone --depth 1 https://github.com/TobyBaril/SA-SSR
cd SA-SSR
make -j"${CPU_COUNT}"
install -v -m 0755 bin/sa-ssr "${PREFIX}/bin"
cd ../ && rm -rf SA-SSR/
# Fixes to earlGrey executable
sed -i.bak "/CONDA_DEFAULT_ENV/,+4d" ${PACKAGE_HOME}/earlGrey #remove check that conda environment has a specific name
sed -i.bak "/CONDA_DEFAULT_ENV/,+4d" ${PACKAGE_HOME}/earlGreyLibConstruct #remove check that conda environment has a specific name
sed -i.bak "/CONDA_DEFAULT_ENV/,+4d" ${PACKAGE_HOME}/earlGreyAnnotationOnly #remove check that conda environment has a specific name
# Fixes sed command for executables so that it works on both linux and macos
sed -i.bak "s|sed -i |sed -i.bak |g" ${PACKAGE_HOME}/earlGrey ${PACKAGE_HOME}/earlGreyLibConstruct ${PACKAGE_HOME}/earlGreyAnnotationOnly ${SCRIPT_DIR}/rcMergeRepeat* ${SCRIPT_DIR}/TEstrainer/TEstrainer_for_earlGrey.sh
# Remove -pa from RepeatClassifier
sed -i.bak 's/RepeatClassifier -pa ${THREADS} /RepeatClassifier /' ${SCRIPT_DIR}/TEstrainer/TEstrainer
# Remove -t parameter from sa-ssr (since multithreading doesn't work on OSX)
sed -i.bak 's/-t ${THREADS} / /' ${SCRIPT_DIR}/TEstrainer/TEstrainer_for_earlGrey.sh
sed -i.bak 's/-t ${THREADS} / /' ${SCRIPT_DIR}/TEstrainer/TEstrainer
# Add SCRIPT_DIR to correct path
sed -i.bak "s|SCRIPT_DIR=.*|SCRIPT_DIR=${SCRIPT_DIR}|g" ${PACKAGE_HOME}/earlGrey
sed -i.bak "s|SCRIPT_DIR=.*|SCRIPT_DIR=${SCRIPT_DIR}|g" ${PACKAGE_HOME}/earlGreyLibConstruct
sed -i.bak "s|SCRIPT_DIR=.*|SCRIPT_DIR=${SCRIPT_DIR}|g" ${PACKAGE_HOME}/earlGreyAnnotationOnly
sed -i.bak "s|SCRIPT_DIR=.*|SCRIPT_DIR=${SCRIPT_DIR}|g" ${SCRIPT_DIR}/rcMergeRepeat*
sed -i.bak "s|SCRIPT_DIR=.*|SCRIPT_DIR=${SCRIPT_DIR}|g" ${SCRIPT_DIR}/headSwap.sh
sed -i.bak "s|SCRIPT_DIR=.*|SCRIPT_DIR=${SCRIPT_DIR}|g" ${SCRIPT_DIR}/autoPie.sh
sed -i.bak "s|STRAIN_SCRIPTS=.*|STRAIN_SCRIPTS=${SCRIPT_DIR}/TEstrainer/scripts/|g" ${SCRIPT_DIR}/TEstrainer/TEstrainer_for_earlGrey.sh
# Set permissions to files
chmod +x ${PACKAGE_HOME}/earlGrey
chmod +x ${PACKAGE_HOME}/earlGreyLibConstruct
chmod +x ${PACKAGE_HOME}/earlGreyAnnotationOnly
chmod +x ${SCRIPT_DIR}/TEstrainer/TEstrainer_for_earlGrey.sh
chmod +x ${SCRIPT_DIR}/* > /dev/null 2>&1
chmod +x ${SCRIPT_DIR}/bin/LTR_FINDER.x86_64-1.0.7/ltr_finder
chmod a+w ${SCRIPT_DIR}/repeatCraft/example
# Extract tRNAdb
tar -zxf ${SCRIPT_DIR}/bin/LTR_FINDER.x86_64-1.0.7/tRNAdb.tar.gz --directory ${SCRIPT_DIR}/bin/LTR_FINDER.x86_64-1.0.7 && rm -r ${SCRIPT_DIR}/bin/LTR_FINDER.x86_64-1.0.7/tRNAdb.tar.gz
# test for conda
df -h
# Set PERL5LIB upon activate/deactivate
for CHANGE in "activate" "deactivate";
do
mkdir -p "${PREFIX}/etc/conda/${CHANGE}.d"
done
echo "#!/bin/sh" > "${PREFIX}/etc/conda/activate.d/${PKG_NAME}_activate.sh"
echo "export PERL5LIB=${PREFIX}/share/RepeatMasker/:${PREFIX}/share/RepeatModeler/" >> "${PREFIX}/etc/conda/activate.d/${PKG_NAME}_activate.sh"
echo "#!/bin/sh" > "${PREFIX}/etc/conda/deactivate.d/${PKG_NAME}_deactivate.sh"
echo "unset PERL5LIB" >> "${PREFIX}/etc/conda/deactivate.d/${PKG_NAME}_deactivate.sh"
# Put earlGrey executable in bin
cd ${PREFIX}/bin
ln -sf ${PACKAGE_HOME}/earlGrey .
ln -sf ${PACKAGE_HOME}/earlGreyLibConstruct .
ln -sf ${PACKAGE_HOME}/earlGreyAnnotationOnly .
================================================
FILE: conda/developmentNotes.md
================================================
# Updating Earl Grey to remove Python 3.9 dependency
Bjorn wants me to update Earl Grey to remove the Python 3.9 dependency. This is because Python 3.9 is no longer supported and we want to ensure that Earl Grey can run on newer versions of Python.
I will build Earl Grey with the meta dependency removed and test it to ensure that it still works correctly. I will also update the documentation to reflect the change in dependencies.
Build the conda package:
```bash
conda build .
```
Make a new conda environment and install the package:
```bash
conda create -n earlgrey_python_test -c conda-forge -c bioconda earlgrey --use-local
conda activate earlgrey_python_test
```
This still builds with 3.9. I will force it to build with 3.10 by adding `python >=3.10` to the `run` dependencies in `meta.yaml`. After making this change, I will rebuild the package and test it again. I also removed the dependency on `ncls` version 0.0.64 and replaced it with `ncls` without a specific version to allow for newer versions of `ncls` to be used.
```bash
conda build .
conda create -n earlgrey_python_test -c conda-forge -c bioconda earlgrey --use-local
conda activate earlgrey_python_test
```
Link the dfam libraries and configure RepeatMasker:
```bash
ln -sf ln -sf /data/toby/tools/earlgrey_databases/Libraries/famdb/* \
/data/toby/miniforge3/envs/earlgrey_python_test/share/RepeatMasker/Libraries/famdb/
cd /data/toby/miniforge3/envs/earlgrey_python_test/share/RepeatMasker
perl configure
```
Run a test command to ensure that Earl Grey is working correctly:
```bash
cd /data/toby/testDIR/
earlGrey -g /data/toby/tools/test.fasta -s test_python312 -o . -t 16
```
This passed with a small fix to the `backSwap.py` script where I changed the regular expression for splitting the input file from `'\s+'` to `r'\s+'` to ensure that it correctly handles whitespace. After making this change, I will test the script again to ensure that it works correctly with the new dependencies.
```bash
cd /data/toby/testDIR/
earlGrey -g /data/toby/testDIR/saturationTests/1_4genomesNoRepMasker/1A5.fa -s test_python312_2 -o . -t 32
```
---
# Reducing peak RAM usage in TEstrainer and divergence_calc.py
## Background
Two components were identified as disproportionately RAM-intensive when run with large thread counts, leading to OOM (out-of-memory) kills on compute nodes with limited RAM:
- **TEstrainer** / **TEstrainer_for_earlGrey.sh** — the GNU parallel jobs dispatched during the BEET curation loop (trf, initial_mafft_setup.py, mafft, TEtrim.py) had no memory back-pressure. With 32–64 threads all jobs fired simultaneously, each spawning a Python interpreter that imports `pandas`, `numpy`, `pyranges` and `BioPython` (~300–600 MB each at startup), along with concurrent `--localpair` mafft alignments.
- **divergence_calc.py** — used `multiprocessing.Pool` with the default `fork` start method. On Linux this fork-copies the entire parent address space (including the full GFF DataFrame, which can be multiple GB for large genomes) into every worker process. At 32 cores this means 32 nearly-identical copies of the GFF in RAM simultaneously, even though each worker only needs its small chunk.
## Changes made
### `scripts/TEstrainer/TEstrainer` and `scripts/TEstrainer/TEstrainer_for_earlGrey.sh`
**1. Default `MEM_FREE` raised from `200M` to `1G`**
The previous default of 200 MB was smaller than the memory footprint of a single Python interpreter after importing the heavy scientific libraries used by `initial_mafft_setup.py` and `TEtrim.py`. This meant `--memfree 200M` gave essentially no throttling protection. 1 GB is a conservative lower bound that reflects real startup costs.
**2. RAM-cap guard added after argument parsing**
A block querying `/proc/meminfo` via `free -m` is inserted immediately after `MAFFT_THREADS` is calculated. If the requested thread count exceeds what is safely supportable (at an estimate of 800 MB per concurrent job), `THREADS` (and `MAFFT_THREADS`) are silently capped with a warning message to stderr. The 800 MB estimate is conservative; real usage per job is closer to 300–500 MB for typical inputs but spikes during mafft's `--localpair` DP phase.
```bash
AVAIL_MEM_MB=$(free -m | awk '/^Mem:/{print $7}')
MEM_PER_JOB_MB=800
MAX_SAFE_THREADS=$(( AVAIL_MEM_MB / MEM_PER_JOB_MB ))
if [[ $MAX_SAFE_THREADS -gt 0 && $THREADS -gt $MAX_SAFE_THREADS ]]; then
echo "Warning: capping threads from ${THREADS} to ${MAX_SAFE_THREADS} based on available RAM (${AVAIL_MEM_MB} MB free)"
THREADS=$MAX_SAFE_THREADS
if [[ $THREADS -gt 4 ]]; then MAFFT_THREADS=$(($(($THREADS / 4)))); else MAFFT_THREADS=1; fi
fi
```
**3. `--memfree ${MEM_FREE}` added to all GNU parallel calls that were missing it**
In the standalone `TEstrainer`, the four `parallel` calls inside the BEET curation loop (trf, initial_mafft_setup, mafft, TEtrim) were missing `--memfree`. In `TEstrainer_for_earlGrey.sh`, the mafft and TEtrim calls were missing it (the others already had it). GNU parallel's `--memfree` halts job dispatch when free system RAM drops below the threshold, queuing further jobs until a running job finishes and memory is reclaimed. This is the most direct mechanism for preventing simultaneous over-subscription.
### `scripts/divergenceCalc/divergence_calc.py`
**1. `forkserver` multiprocessing start method**
```python
multiprocessing.set_start_method('forkserver')
```
On Linux the default is `fork`, which duplicates the entire parent address space into every worker via `clone()`. With a multi-GB GFF DataFrame in memory this creates N near-identical copies. `forkserver` instead spawns a minimal server process at startup; each worker is forked from that clean server and receives only the data it actually needs (the chunk file path string) via IPC. This is the primary RAM reduction mechanism.
**2. GFF DataFrame serialised to per-chunk TSV files before pool creation**
Previously the chunk DataFrames were passed to `pool.map()` directly — they were pickled in the parent and unpickled in each worker. With `forkserver` this already avoids the fork-copy problem, but the parent still holds the full `in_gff` DataFrame in memory throughout the entire pool run. The new approach serialises each chunk to a temp TSV on disk, then deletes `in_gff` and `chunks` from the parent before the pool is even created:
```python
chunk_files = []
for i, chunk in enumerate(chunks):
chunk_path = os.path.join(args.temp_dir, f"chunk_{i}.tsv")
chunk.to_csv(chunk_path, sep="\t", index=True)
chunk_files.append(chunk_path)
del chunks
del in_gff
```
Workers then receive a file path string (not a DataFrame), read the chunk at the start of `outer_func`, and immediately delete the file. Parent peak RAM during the pool run is reduced to `simple_gff + other_gff + Python overhead` rather than `full_gff + simple_gff + other_gff`.
**3. `outer_func` signature updated and `pybedtools.set_tempdir` moved inside worker**
The first argument to `outer_func` is now the chunk file path. `pybedtools.set_tempdir()` is now called at the top of `outer_func` rather than only in the parent. This is required for `forkserver` workers, which do not inherit the parent's in-memory pybedtools state.
**4. Periodic `pybedtools.cleanup()` inside `outer_func`**
```python
if row_counter % 500 == 0:
pybedtools.cleanup(remove_all=False)
```
Each row creates a pybedtools temp file. In a long-running worker processing thousands of rows, these accumulate in the temp directory and inflate virtual memory. Calling `cleanup(remove_all=False)` every 500 rows removes files that are no longer referenced by any active Python object, releasing both disk and virtual-memory pressure without affecting open handles.
**5. `imap_unordered` instead of `pool.map`**
```python
results = list(pool.imap_unordered(func, chunk_files))
```
`pool.map` buffers all results in memory until every worker finishes. `imap_unordered` yields results as each worker completes. Since `outer_func` returns only a file path string this has negligible direct memory impact, but it allows the Python runtime to close completed worker processes and release their memory sooner when combined with `maxtasksperchild`.
**6. `maxtasksperchild=1`**
```python
pool = multiprocessing.Pool(processes=num_processes, maxtasksperchild=1)
```
Forces each worker process to exit and be replaced after processing one chunk. This ensures that any memory accumulated during a chunk run (pybedtools internal state, BioPython caches, subprocess residuals) is released to the OS at chunk boundaries rather than accumulating across the lifetime of the pool.
---
## Tests
The following tests should be run to verify correctness and confirm the memory ceiling has been reduced. Run them from the root of the EarlGrey repo unless stated otherwise.
### Dependencies for memory monitoring
```bash
# Install memory_profiler if not already present
pip install memory_profiler psutil
# /usr/bin/time -v is available on Linux via the 'time' package
which /usr/bin/time || sudo apt-get install -y time
```
---
### Test 1 — Static audit: verify `--memfree` presence in all parallel calls
Confirms no `parallel` call in either TEstrainer script is dispatching jobs without a memory guard.
```bash
echo ""
echo "=== TEstrainer_for_earlGrey.sh ==="
grep -n 'parallel' scripts/TEstrainer/TEstrainer_for_earlGrey.sh | grep -v '#'
```
**Expected:** Every line containing `parallel --bar` should also contain `--memfree`.
---
### Test 2 — Static audit: verify MEM_FREE defaults
```bash
grep 'MEM_FREE=' scripts/TEstrainer/TEstrainer_for_earlGrey.sh
```
**Expected:** Both lines should read `MEM_FREE="1G"`.
---
### Test 3 — Static audit: verify forkserver and chunk-file pattern in divergence_calc.py
```bash
grep -n 'set_start_method\|chunk_path\|chunk_files\|imap_unordered\|maxtasksperchild\|pybedtools.cleanup' \
scripts/divergenceCalc/divergence_calc.py
```
**Expected output should contain all of:**
- `set_start_method('forkserver')`
- `chunk_path` (argument to `outer_func`)
- `chunk_files` (list built in `__main__`)
- `imap_unordered`
- `maxtasksperchild=1`
- `pybedtools.cleanup`
---
### Test 4 — RAM-cap guard: unit test on the bash logic
```bash
# Simulate a machine with 4 GB available and a 64-thread request.
# The guard should cap THREADS to (4096 / 800) = 5.
bash -c '
THREADS=64
AVAIL_MEM_MB=4096
MEM_PER_JOB_MB=800
MAX_SAFE_THREADS=$(( AVAIL_MEM_MB / MEM_PER_JOB_MB ))
if [[ $MAX_SAFE_THREADS -gt 0 && $THREADS -gt $MAX_SAFE_THREADS ]]; then
echo "CAPPED: THREADS set to ${MAX_SAFE_THREADS}"
THREADS=$MAX_SAFE_THREADS
else
echo "NOT CAPPED: THREADS remains ${THREADS}"
fi
echo "Final THREADS: ${THREADS}"
'
```
**Expected:**
```
CAPPED: THREADS set to 5
Final THREADS: 5
```
---
### Test 5 — Correctness: divergence_calc.py output matches reference
Generate a reference output using the old code path on a small test GFF, then confirm the new code produces identical output. A pre-existing completed EarlGrey run in `testDIR` can supply the required files.
```bash
# Adjust paths to a completed EarlGrey run with a .gff and matching library
TESTDIR=/data/toby/testDIR/condaPull
GENOME=/data/toby/testDIR/test.fasta
GFF=/data/toby/testDIR/condaPull/genome1_EarlGrey/genome1_mergedRepeats/looseMerge/genome1.filteredRepeats.gff
LIB=/data/toby/testDIR/condaPull/genome1_EarlGrey/genome1_strainer/genome1-families.fa.strained
# Run with the updated script
python scripts/divergenceCalc/divergence_calc.py \
-l "${LIB}" \
-i "${GFF}" \
-g "${GENOME}" \
-o /tmp/divergence_new.gff \
-tmp /tmp/divtest_new/ \
-t 4
echo "Exit code: $?"
echo "Output lines: $(wc -l < /tmp/divergence_new.gff)"
```
**Expected:** Zero exit code, output line count matches the input GFF line count, all lines end with `;KIMURA80=`.
```bash
# Verify all non-simple-repeat lines carry a KIMURA80 tag
grep -v 'Simple_repeat\|Satellite\|Low_complexity' /tmp/divergence_new.gff | \
grep -v 'KIMURA80' | wc -l
```
**Expected:** 0 for a pure RepeatMasker/Earl Grey GFF. Annotations from other tools (e.g. HELIANO, which uses a different `tool` column value) are passed through unmodified by `parse_gff()` and will not carry a `KIMURA80` tag — one such line per non-RM/EG annotation is acceptable and expected.
---
### Test 6 — Memory ceiling: peak RSS comparison for divergence_calc.py
Measure peak RSS with different thread counts to confirm sub-linear scaling (the old `fork` behaviour would show near-linear scaling of the peak).
```bash
TESTDIR=/data/toby/testDIR/condaPull
GENOME=/data/toby/testDIR/test.fasta
GFF=/data/toby/testDIR/condaPull/genome1_EarlGrey/genome1_mergedRepeats/looseMerge/genome1.filteredRepeats.gff
LIB=/data/toby/testDIR/condaPull/genome1_EarlGrey/genome1_strainer/genome1-families.fa.strained
for T in 1 4 8 16; do
rm -rf /tmp/divmem_t${T}
echo -n "Threads=${T}: "
/usr/bin/time -v python scripts/divergenceCalc/divergence_calc.py \
-l "${LIB}" -i "${GFF}" -g "${GENOME}" \
-o /tmp/divmem_t${T}.gff -tmp /tmp/divmem_t${T}/ -t ${T} \
2>&1 | grep 'Maximum resident'
done
```
**Expected:** Peak RSS should not scale linearly with thread count. With the old `fork` code, doubling the thread count roughly doubled peak RSS. With the new code, peak RSS should plateau once per-worker overheads are comparable to the chunk size — a significant reduction at 8+ threads on large GFFs.
As a rough guide, if the single-thread peak RSS is R MB, you should aim to see the 16-thread run use no more than ~2–3× R (library imports + 16 small chunks), compared to potentially 16× R with the old fork-based approach.
**Observed results (test genome, condaPull/genome1):**
| Threads | Peak RSS (kB) |
|---------|--------------|
| 1 | 75,992 |
| 4 | 75,940 |
| 8 | 75,920 |
| 16 | 76,092 |
Peak RSS is effectively flat across all thread counts (~76 MB). This confirms that `forkserver` workers start from a clean process and do not inherit the parent GFF DataFrame — the old `fork` implementation would have shown peak RSS scaling approximately linearly with thread count (potentially ~16× higher at 16 threads for a large GFF).
---
### Test 7 — Memory ceiling: peak RSS comparison for TEstrainer
```bash
TESTLIB=/data/toby/testDIR/condaPull/genome1_EarlGrey/genome1_strainer/genome1-families.fa.strained # use a small library for speed
GENOME=/data/toby/testDIR/test.fasta
# Baseline: 4 threads
/usr/bin/time -v bash scripts/TEstrainer/TEstrainer_for_earlGrey.sh \
-l ${TESTLIB} -g ${GENOME} -t 4 -r 1 -d /tmp/ts_mem_t4 \
2>&1 | grep 'Maximum resident'
# Higher thread count: confirm MEM_FREE throttles correctly
/usr/bin/time -v bash scripts/TEstrainer/TEstrainer_for_earlGrey.sh \
-l ${TESTLIB} -g ${GENOME} -t 32 -r 1 -d /tmp/ts_mem_t32 \
2>&1 | grep 'Maximum resident'
```
**Expected:** The 32-thread run should emit a `Warning: capping threads` message if available RAM is insufficient, and peak RSS should not grow proportionally with the requested thread count.
**Observed results (test genome, condaPull/genome1 library):**
| Threads | Peak RSS (kB) |
|---------|--------------|
| 4 | 897,860 |
| 32 | 891,420 |
Peak RSS is effectively identical between 4 and 32 threads (~877 MB). Without `--memfree` throttling, the 32-thread run would dispatch all jobs simultaneously and peak RSS would scale with the number of concurrent Python interpreter startups. The flat result confirms that GNU parallel is correctly queuing jobs when free RAM drops below the `MEM_FREE` threshold, keeping concurrent memory usage stable regardless of the requested thread count.
---
### Test 8 — `--memfree` throttling behaviour
Verify that GNU parallel actually pauses job dispatch when free RAM is low, rather than launching all jobs at once.
```bash
# Install stress-ng to consume a controllable amount of RAM
which stress-ng || sudo apt-get install -y stress-ng
# Consume ~80% of available RAM, then run TEstrainer with many threads.
# parallel should pause on subsequent jobs when memfree drops below MEM_FREE.
AVAIL=$(free -m | awk '/^Mem:/{print $7}')
CONSUME_MB=$(( AVAIL * 80 / 100 ))
stress-ng --vm 1 --vm-bytes ${CONSUME_MB}M --vm-keep &
STRESS_PID=$!
sleep 3
# Now attempt to run parallel with --memfree 1G and many small jobs.
# Observe that parallel waits rather than dispatching all at once.
seq 1 20 | parallel --jobs 8 --memfree 1G 'sleep 0.5 && echo job {} done'
kill $STRESS_PID
```
**Expected:** The 20 sleep jobs complete without error. If free RAM is below 1 GB after stress-ng starts, parallel will hold back some jobs rather than launching all 20 simultaneously, and the output will appear staggered as memory is reclaimed.
---
### Test 9 — Worker isolation: confirm forkserver workers do not inherit parent GFF
This Python script directly tests that a worker process does not see the parent's `in_gff` variable, confirming the forkserver isolation is working.
```python
# save as /tmp/test_forkserver_isolation.py and run with: python /tmp/test_forkserver_isolation.py
import multiprocessing
import os
import pandas as pd
def worker_mem_check(_):
"""Return RSS of this worker process in MB."""
import psutil
proc = psutil.Process(os.getpid())
return proc.memory_info().rss / 1024 / 1024
if __name__ == "__main__":
# Allocate a large DataFrame in the parent to simulate a big GFF.
big_df = pd.DataFrame({'a': range(5_000_000), 'b': range(5_000_000)})
parent_rss_mb = __import__('psutil').Process(os.getpid()).memory_info().rss / 1024 / 1024
print(f"Parent RSS after allocating large DataFrame: {parent_rss_mb:.0f} MB")
multiprocessing.set_start_method('forkserver')
with multiprocessing.Pool(processes=4) as pool:
worker_rss_values = pool.map(worker_mem_check, range(4))
for i, rss in enumerate(worker_rss_values):
print(f"Worker {i} RSS: {rss:.0f} MB")
print(f"\nParent RSS: {parent_rss_mb:.0f} MB")
print(f"Mean worker RSS: {sum(worker_rss_values)/len(worker_rss_values):.0f} MB")
print(f"Ratio (parent/worker): {parent_rss_mb / (sum(worker_rss_values)/len(worker_rss_values)):.1f}x")
```
**Expected:** Worker RSS values should be substantially smaller than the parent RSS (typically 5–20× smaller), confirming that workers do not inherit the large `big_df` from the parent. With the old `fork` start method, worker RSS would be approximately equal to parent RSS.
To contrast, change `set_start_method('forkserver')` to `set_start_method('fork')` and re-run — worker RSS values should roughly match the parent.
**Observed results:**
```
Parent RSS after allocating large DataFrame: 135 MB
Worker 0 RSS: 54 MB
Worker 1 RSS: 54 MB
Worker 2 RSS: 54 MB
Worker 3 RSS: 54 MB
Parent RSS: 135 MB
Mean worker RSS: 54 MB
Ratio (parent/worker): 2.5x
```
Workers consume 54 MB (Python + library imports only) versus 135 MB in the parent (which includes the 5M-row DataFrame). The workers carry no copy of the parent's data, confirming `forkserver` isolation is working correctly. With the old `fork` start method all four workers would each have shown ~135 MB.
---
### Test 10 — End-to-end regression: full EarlGrey run
A complete run to confirm that all components still produce correct outputs after the changes.
```bash
cd /data/toby/testDIR/
earlGrey -g test.fasta -s mem_regression_test -o . -t 16
```
**Expected:** Run completes without error. Output directory `mem_regression_test_EarlGrey/` is populated with the expected files including `*-families.fa`, `*.gff`, and `*.divergence.pdf`. Compare output against a previous known-good run to confirm no regressions in repeat library composition or divergence values.
**Observed:** Run completed without error (exit code 0). All memory-reduction changes are confirmed compatible with a full end-to-end EarlGrey pipeline run.
This also passed successfully, confirming that Earl Grey is now compatible with Python 3.10 and does not have a dependency on Python 3.9. I will now update the documentation to reflect these changes and ensure that users are aware of the new dependencies when installing Earl Grey.
---
# Version 7.2.3 — bug fixes, robustness improvements, and `-q` quiet-bar flag
## Background
Several bugs were found and fixed after the v7.2.0 memory-refinement release. The changes also add a new `-q` flag to `earlGrey` to suppress the GNU parallel progress bar, which is useful when running jobs through batch schedulers (SLURM `sbatch`, PBS, etc.) where the ANSI escape codes written by `--bar` corrupt log files.
## Changes made
### `conda/meta.yaml`
- Bumped version to `7.2.3`.
- Added `numpy` to the `run` dependencies. `divergence_calc.py` now imports `numpy` directly (for `np.array_split`), so it must be declared explicitly.
- Changed `source` from a remote tarball + SHA256 to `path: ..` to allow local conda builds during development without needing a published release tarball.
### `earlGrey` (main pipeline script)
**1. New `-q` flag to suppress the TEstrainer parallel progress bar**
```bash
earlGrey -g genome.fa -s myspecies -o outdir -t 16 -q yes
```
When `-q yes` is passed, `earlGrey` appends `-q` to the `TEstrainer_for_earlGrey.sh` invocation. This propagates to every GNU `parallel --bar` call inside TEstrainer, replacing `--bar` with nothing so the progress bar is omitted. Useful for `sbatch` jobs where `--bar` writes ANSI codes into log files.
**2. Fixed `AF_UNIX` socket path-too-long crash in `divergence_calc.py`**
`pybedtools.set_tempdir()` sets `tempfile.tempdir` globally, which is also used by `forkserver` as the base directory for its `AF_UNIX` socket. If the EarlGrey output path is deeply nested, the socket path can exceed the 108-character kernel limit, causing an `OSError: AF_UNIX path too long`. The fix creates a short per-run temp directory under `/tmp`:
```bash
divcalc_tmp="/tmp/egdiv_${species}"
mkdir -p "${divcalc_tmp}"
python .../divergence_calc.py ... -tmp "${divcalc_tmp}"
# ...
rm -rf "${divcalc_tmp}"
```
### `scripts/TEstrainer/TEstrainer_for_earlGrey.sh`
Added a `-q` flag that sets `BAR_FLAG=""` (default `BAR_FLAG="--bar"`). The variable is substituted into every `parallel` call so a single flag controls all progress bars in the script.
### `scripts/divergenceCalc/divergence_calc.py`
**1. Added `import numpy as np`** — used for `np.array_split` (see below).
**2. Replaced manual chunk splitting with `np.array_split`**
The old code computed `chunk_size = int(rows / num_processes)` and stepped through the DataFrame with a fixed stride. For inputs where `rows` is not evenly divisible by `num_processes` this could silently drop up to `num_processes - 1` rows. `np.array_split` distributes any remainder evenly:
```python
chunks = [in_gff.iloc[idx] for idx in np.array_split(range(len(in_gff)), num_processes)]
```
**3. Replaced all path string concatenation with `os.path.join`**
All occurrences of `temp_dir + "/subdir/" + name` have been replaced with `os.path.join(temp_dir, "subdir", name)`. This fixes `OSError`s that occurred when `args.temp_dir` was passed with a trailing slash (e.g. `/tmp/egdiv_myspecies/`), which caused double-slash paths like `//pybedtools` that could fail on some systems.
### `scripts/filteringOverlappingRepeats.R`
Fixed a crash that occurred when no nested TEs were found in the genome. The previous code unconditionally called `mutate()` and `bind_rows()` on `nested_only`, which threw an error when `nested_only` was an empty data frame. The fix wraps the nested-TE processing in an `if (nrow(nested_only) > 0)` guard and falls through to writing `unnested_only` directly when there are no nested elements:
```r
if (nrow(nested_only) > 0) {
nested_only <- nested_only %>%
mutate(attributes = paste0(attributes, ";nested=", nested, "-round", nesting_round)) %>%
select(-c(nested, nesting_round))
output.gff <- bind_rows(nested_only, unnested_only) %>% arrange(seqid, start)
} else {
output.gff <- unnested_only %>% arrange(seqid, start)
}
```
### `scripts/divergenceCalc/divergence_plot.R`
Fixed crashes in the per-superfamily Kimura divergence plots when a TE class (DNA, LINE, LTR, SINE, PLE, RC) is absent from the genome. The previous code built each plot unconditionally and then caught errors with `try(ggplot_build(...))`. The new approach checks `!is.null(...) && nrow(...) > 0` before constructing the ggplot object:
```r
if (!is.null(divergence_eg_tes_rounded_for_superfamily_plot[["DNA"]]) &&
nrow(divergence_eg_tes_rounded_for_superfamily_plot[["DNA"]]) > 0) {
kimura_superfamily_plot_1 <- ggplot(...) + ...
} else {
kimura_superfamily_plot_1 <- NULL
}
```
The axis-flipping logic was also updated to skip `NULL` plots rather than attempting to add `scale_x_continuous()` to them (which would previously crash).
## Build and test environment
Build the conda package from the local source tree:
```bash
cd /data/toby/EarlGrey/conda
conda build .
```
Create a fresh environment and install:
```bash
conda create -n earlgrey_723_test -c conda-forge -c bioconda earlgrey --use-local
conda activate earlgrey_723_test
```
Link the Dfam libraries and configure RepeatMasker:
```bash
ln -sf /data/toby/tools/earlgrey_databases/Libraries/famdb/* \
/data/toby/miniforge3/envs/earlgrey_723_test/share/RepeatMasker/Libraries/famdb/
cd /data/toby/miniforge3/envs/earlgrey_723_test/share/RepeatMasker
perl configure
```
### Static checks
```bash
cd /data/toby/EarlGrey
# 1. Confirm numpy is in meta.yaml
grep 'numpy' conda/meta.yaml
# 2. Confirm -q flag is wired up in earlGrey
grep -n 'quietBar\|BAR_FLAG\|\-q' earlGrey | head -20
# 3. Confirm BAR_FLAG is used in TEstrainer_for_earlGrey.sh
grep -n 'BAR_FLAG\|bar' scripts/TEstrainer/TEstrainer_for_earlGrey.sh | grep -v '#'
# 4. Confirm os.path.join used throughout divergence_calc.py (no string concat with temp_dir)
grep 'temp_dir *+' scripts/divergenceCalc/divergence_calc.py # should return nothing
# 5. Confirm np.array_split is used for chunking
grep 'array_split' scripts/divergenceCalc/divergence_calc.py
# 6. Confirm nrow guard in filteringOverlappingRepeats.R
grep -n 'nrow(nested_only)' scripts/filteringOverlappingRepeats.R
```
### Functional tests
```bash
cd /data/toby/testDIR/
# Basic run — confirm pipeline completes with -q yes
earlGrey -g test.fasta -s test_723_quiet -o . -t 16 -q yes
# Confirm no ANSI bar codes in the log (stdout should have no ESC sequences)
earlGrey -g test.fasta -s test_723_quiet2 -o . -t 16 -q yes 2>&1 | cat | grep -c $'\033' || echo "No ANSI codes — good"
# Confirm pipeline still works without -q (bar shown as before)
earlGrey -g test.fasta -s test_723_noq -o . -t 16
```
---
# Version 7.2.4 — pipeline error handling, samplable genome size for RepeatModeler, and bash safety
## Background
Two complementary robustness improvements were identified and implemented across the three main Earl Grey pipeline scripts (`earlGrey`, `earlGreyLibConstruct`, `earlGreyAnnotationOnly`):
1. The `deNovo1()` function in `earlGrey` and `earlGreyLibConstruct` used a deeply nested retry loop to work around RepeatModeler failures on small or fragmented genomes. This approach was reactive: RepeatModeler would run, fail, be retried with a smaller `-genomeSampleSizeMax`, fail again, and be retried once more — wasting potentially hours of compute and producing confusing log output. The ParTEA (pangenome) version of Earl Grey had already solved this more elegantly by pre-computing the samplable genome size and choosing the correct flag upfront.
2. All three scripts lacked `set -eo pipefail`, meaning that any command that exited non-zero in the middle of a function body (outside an explicit `if` check) would silently be ignored and execution would continue into the next stage with corrupt or missing intermediate files. Additionally, two common shell idioms in the scripts would produce false failures under strict error handling: `expr N / 4` (returns exit code 1 when the result is 0) and `ls ... | head -n 1` pipelines (SIGPIPE exit 141 from `ls` when `head` exits early under `pipefail`).
## Changes made
### `earlGrey` and `earlGreyLibConstruct` — `deNovo1()` redesign
The previous implementation:
```bash
deNovo1() {
RepeatModeler -threads ${ProcNum} -database ...
if [ ! -e ...-families.fa ]; then
echo "ERROR: Retrying with limit set as Round 5"
RepeatModeler ... -genomeSampleSizeMax 81000000
if [ ! -e ...-families.fa ]; then
echo "ERROR: Retrying with limit set as Round 4"
RepeatModeler ... -genomeSampleSizeMax 27000000
if [ ! -e ...-families.fa ]; then
echo "ERROR: RepeatModeler Failed" && exit 2
fi
fi
fi
}
```
This ran RepeatModeler up to three times, each time with a smaller sampling cap, without any principled basis for which cap to choose. It also only covered two fallback tiers (rounds 5 and 4), leaving genomes smaller than 12 Mb (rounds 1–2) unhandled.
The new implementation mirrors the approach used in ParTEA. Before invoking RepeatModeler, the samplable genome size is computed as the sum of all contigs ≥ 40 kb (the threshold below which RepeatModeler discards contigs during sampling). The appropriate `-genomeSampleSizeMax` cap is then derived from the cumulative RECON round thresholds and set in `SAMPLE_FLAG`. RepeatModeler is invoked exactly once with the correct flag, and a single post-run existence check is retained as a safety net.
**Cumulative RECON round thresholds:**
| Rounds supported | Cumulative threshold | Cap applied |
|-----------------|---------------------|-------------|
| All 6 rounds | ≥ 363 Mb (3+9+27+81+243) | none (default) |
| Rounds 1–5 | ≥ 120 Mb (3+9+27+81) | `-genomeSampleSizeMax 81000000` |
| Rounds 1–4 | ≥ 39 Mb (3+9+27) | `-genomeSampleSizeMax 27000000` |
| Rounds 1–3 | ≥ 12 Mb (3+9) | `-genomeSampleSizeMax 9000000` |
| Rounds 1–2 | < 12 Mb | `-genomeSampleSizeMax 3000000` |
The samplable genome size is computed using `awk` directly on `$genome` (the `.prep` file, which is already available at the `deNovo1` call site):
```bash
GENOME_SIZE=$(awk '/^>/{if(len>=40000)sum+=len; len=0; next}{len+=length($0)} END{if(len>=40000)sum+=len; print sum+0}' "${genome}")
```
### `earlGrey`, `earlGreyLibConstruct`, `earlGreyAnnotationOnly` — `set -eo pipefail`
`set -eo pipefail` was added immediately after `#!/bin/bash` in all three scripts.
- `-e`: exits the script immediately if any simple command returns a non-zero exit code and is not part of a condition test.
- `-o pipefail`: extends `-e` to pipelines — a pipeline fails if any stage within it returns non-zero, not just the last stage.
Together these ensure that a failure in any tool invocation within a function body (RepeatMasker, BuildDatabase, TEstrainer, rcMergeRepeats, R scripts, etc.) will abort the pipeline at the point of failure rather than silently continuing to the next stage.
### `earlGrey`, `earlGreyLibConstruct`, `earlGreyAnnotationOnly` — `expr` replaced with `$(( ))`
All occurrences of `rmthreads=$(expr $ProcNum / 4)` and `strainthreads=$(expr $ProcNum / 4)` were replaced with the equivalent `$(( ProcNum / 4 ))`. The `expr` utility returns exit code 1 when the arithmetic result is 0, which occurs when `ProcNum < 4`. Under `set -e` this would abort the script before any tool was invoked if the user specified fewer than 4 threads.
### `earlGrey`, `earlGreyLibConstruct`, `earlGreyAnnotationOnly` — `ls | head -n 1` pipeline fix
All `ls -td ... | head -n 1` pipeline expressions used to retrieve the most recently created directory (TEstrainer output directories, HELIANO output directories) were appended with `|| true`:
```bash
latestFile="$(realpath $(ls -td -- ${OUTDIR}/${species}_strainer/*/ | head -n 1) || true)/..."
```
Under `pipefail`, when `head` exits after reading one line, `ls` receives SIGPIPE (exit code 141), causing the pipeline to return non-zero. This is expected behaviour — `ls` has more output but `head` is done — and `|| true` suppresses this signal without masking genuine `ls` errors (which would produce a different non-zero exit and an error message to stderr).
## Static verification
```bash
cd /data/toby/EarlGrey
# 1. Confirm set -eo pipefail is present in all three scripts
grep -n 'set -eo pipefail' earlGrey earlGreyLibConstruct earlGreyAnnotationOnly
# 2. Confirm no remaining expr calls
grep -n 'expr' earlGrey earlGreyLibConstruct earlGreyAnnotationOnly
# 3. Confirm ls|head pipelines are guarded
grep -n 'head -n 1' earlGrey earlGreyLibConstruct earlGreyAnnotationOnly
# 4. Confirm GENOME_SIZE and SAMPLE_FLAG logic is present in deNovo1
grep -n 'GENOME_SIZE\|SAMPLE_FLAG' earlGrey earlGreyLibConstruct
```
**Expected for check 1:** Three lines, one per script, each showing `set -eo pipefail` at line 2.
**Expected for check 2:** No output (all `expr` calls replaced).
**Expected for check 3:** All matching lines should contain `|| true`.
**Expected for check 4:** Both scripts should show `GENOME_SIZE` and `SAMPLE_FLAG` variable assignments and the five-tier `if/elif/else` block inside `deNovo1`.
================================================
FILE: conda/meta.yaml
================================================
{% set name = "EarlGrey" %}
{% set version = "7.2.4" %}
package:
name: {{ name|lower }}
version: {{ version }}
source:
path: ..
build:
number: 0
skip: True # [osx]
run_exports:
- {{ pin_subpackage('earlgrey', max_pin="x") }}
requirements:
build:
- make
- {{ compiler('cxx') }}
run:
- python >=3.10
- hmmer
- trf
- cd-hit
- genometools-genometools
- numpy
- pandas
- ncls
- pyfaidx
- pyranges
- parallel
- repeatmasker >=4.2.3
- ltr_retriever
- mafft
- mreps
- ninja-nj
- repeatscout
- recon
- repeatmodeler >=2.0.4
- bioconductor-genomeinfodb
- bioconductor-genomeinfodbdata
- bioconductor-bsgenome
- bioconductor-plyranges
- r-ape
- r-optparse
- r-tidyverse
- r-plyr
- r-viridis
- r-cowplot
- r-ggtext
- r-data.table
- r-magrittr
- bedtools
- emboss
- pybedtools
- samtools
- heliano
- r-kableextra
test:
commands:
- df -h
- earlGrey -h
about:
home: "https://github.com/TobyBaril/EarlGrey"
dev_url: "https://github.com/TobyBaril/EarlGrey"
doc_url: "https://github.com/TobyBaril/EarlGrey/blob/v{{ version }}/README.md"
license: "OSL-2.1"
license_file: LICENSE
summary: "Earl Grey: A fully automated TE curation and annotation pipeline."
description: |
Earl Grey is a full-automated transposable element (TE) annotation pipeline,
leveraging the most widely-used tools and combining these with a consensus
elongation process (BEAT) to better define de novo consensus sequences when
annotating new genome assemblies.
extra:
identifiers:
- doi:10.1093/molbev/msae068
================================================
FILE: earlGrey
================================================
#!/bin/bash
set -eo pipefail
usage()
{
echo " #############################
earlGrey version 7.2.4
Required Parameters:
-g == genome.fasta
-s == species name
-o == output directory
Optional Parameters:
-t == Number of Threads (DO NOT specify more than are available)
-r == RepeatMasker search term (e.g arthropoda/eukarya)
-l == Starting consensus library for an initial mask (in fasta format)
-i == Number of Iterations to BLAST, Extract, Extend (Default: 10)
-f == Number flanking basepairs to extract (Default: 1000)
-c == Cluster TE library to reduce redundancy? (yes/no, Default: no)
-m == Remove putative spurious TE annotations <100bp? (yes/no, Default: no)
-d == Create soft-masked genome at the end? (yes/no, Default: no)
-n == Max number of sequences used to generate consensus sequences (Default: 20)
-a == minimum number of sequences required to build a consensus sequence (Default: 3)
-e == Run HELIANO as an optional step to detect Helitrons (yes/no, Default: no)
-q == Suppress TEstrainer parallel progress bar (yes/no, Default: no, useful for batch/sbatch jobs)
-h == Show help
Example Usage:
earlGrey -g bombyxMori.fasta -s bombyxMori -o /home/toby/bombyxMori/repeatAnnotation/ -t 16
Queries can be sent to:
tobias.baril[at]unine.ch
Please make use of the GitHub Issues and Discussion Tabs at: https://github.com/TobyBaril/EarlGrey
#############################"
}
# Subprocess Make Directories #
makeDirectory()
{
directory=$(realpath ${directory})
mkdir -p ${directory}/${species}_EarlGrey/ && OUTDIR=${directory}/${species}_EarlGrey
if [ ! -z "$RepSpec" ] || [ ! -z "$startCust" ] ; then
mkdir -p $OUTDIR/${species}_RepeatMasker/
fi
mkdir -p $OUTDIR/${species}_Database/
mkdir -p $OUTDIR/${species}_RepeatModeler/
mkdir -p $OUTDIR/${species}_strainer/
mkdir -p $OUTDIR/${species}_RepeatMasker_Against_Custom_Library/
mkdir -p $OUTDIR/${species}_RepeatLandscape/
mkdir -p $OUTDIR/${species}_mergedRepeats/
mkdir -p ${OUTDIR}/${species}_summaryFiles/
if [ ! -z "$heli" ]; then
mkdir -p ${OUTDIR}/${species}_heliano/
fi
}
# Subprocess PrepGenome #
prepGenome()
{
if [ ! -L ${genome} ]; then
genome=$(realpath ${genome})
fi
if [ -L $genome ]; then
genome=$(realpath -s ${genome})
fi
if [ ! -f ${genome}.prep ] || [ ! -f ${genome}.dict ]; then
cp ${genome} ${genome}.bak && gzip -f ${genome}.bak
sed '/>/ s/[[:space:]].*//g; /^$/d' ${genome} > ${genome}.tmp
${SCRIPT_DIR}/headSwap.sh -i ${genome}.tmp -o ${genome}.prep && rm ${genome}.tmp
mv ${genome}.tmp.dict ${genome}.dict
sed -i '/^>/! s/[DVHBPE]/N/g' ${genome}.prep
dict=${genome}.dict
genOrig=${genome}
genome=${genome}.prep
else
dict=${genome}.dict
genOrig=${genome}
genome=${genome}.prep
fi
}
# Subprocess getRepeatMaskerFasta
# Generate a copy of the RepeatMasker library subset used for the initial TE mask in FASTA format (only used if a RepeatMasker run is specified)
getRepeatMaskerFasta()
{
if [[ $RepSpec = *" "* ]]; then
echo "ERROR: You have entered a species name that contains a space, please use the NCBI TaxID rather than name. E.G In place of \"Homo sapiens\" use \"9606\""
exit 2
fi
if [[ $(which RepeatMasker) == *"bin"* ]]; then
libpath="$(which RepeatMasker | sed 's|bin/RepeatMasker|share/RepeatMasker/Libraries/famdb/|')"
else
libpath="$(which RepeatMasker | sed 's|/[^/]*$||g')/Libraries/famdb/"
fi
if [[ $(which RepeatMasker) == *"bin"* ]]; then
PATH=$PATH:"$(which RepeatMasker | sed 's|bin/RepeatMasker|share/RepeatMasker/|')"
fi
mkdir -p $OUTDIR/${species}_Curated_Library/
famdb.py -i $libpath families -f fasta_name --include-class-in-name -a -d --curated $RepSpec > ${OUTDIR}/${species}_Curated_Library/${RepSpec}.RepeatMasker.lib
RepSub=${OUTDIR}/${species}_Curated_Library/${RepSpec}.RepeatMasker.lib
}
# Subprocess firstMask
# Run the initial RepeatMasker run with the specified species subset (only used if a RepeatMasker run is specified)
firstMask()
{
cd ${OUTDIR}/${species}_RepeatMasker
rmthreads=$(( ProcNum / 4 ))
RepeatMasker -species $RepSpec -no_is -lcambig -s -a -pa $rmthreads -dir $OUTDIR/${species}_RepeatMasker $genome
if [ ! -f ${OUTDIR}/${species}_RepeatMasker/*.masked ]; then
echo "ERROR: RepeatMasker failed, please check logs. This is likely because of an invalid species search term, if issue persists please use NCBI Taxids (E.G Drosophila is replaced with 7125)"; exit 2
fi
}
# Subprocess firstMaskCustomLib
# Run the initial RepeatMasker run with the specific fasta consensus library
firstMaskCustomLib()
{
cd ${OUTDIR}/${species}_RepeatMasker
rmthreads=$(( ProcNum / 4 ))
RepeatMasker -lib ${startCust} -no_is -lcambig -s -a -pa $rmthreads -dir $OUTDIR/${species}_RepeatMasker $genome
RepSub=${startCust}
if [ ! -f ${OUTDIR}/${species}_RepeatMasker/*.masked ]; then
echo "ERROR: RepeatMasker failed, please check logs. This is likely because of an invalid custom consensus file, check the fasta file looks as expected"; exit 2
fi
}
# Subprocess buildDB
# if masked genome exists, build database from this. If not, build database from original input genome
buildDB()
{
if [ -f ${OUTDIR}/${species}_RepeatMasker/*.masked ]; then
cd ${OUTDIR}/${species}_Database
BuildDatabase -name ${species} ${OUTDIR}/${species}_RepeatMasker/*.masked
else
cd ${OUTDIR}/${species}_Database
BuildDatabase -name ${species} $genome
fi
}
# Subprocess deNovo1
# Run the initial de novo annotation run with RepeatModeler2.
# The genomeSampleSizeMax flag is set based on the samplable genome size (contigs >= 40 kb)
# to avoid RepeatModeler failures on small or fragmented genomes.
deNovo1()
{
cd ${OUTDIR}/${species}_RepeatModeler
# Compute samplable genome size: sum of contigs >= 40 kb only.
# RepeatModeler discards contigs shorter than 40 kb during sampling, so using
# the total genome size would overestimate the sequence available per round.
GENOME_SIZE=$(awk '/^>/{if(len>=40000)sum+=len; len=0; next}{len+=length($0)} END{if(len>=40000)sum+=len; print sum+0}' "${genome}")
echo "Samplable genome size (contigs >= 40 kb): ${GENOME_SIZE} bp"
# Set -genomeSampleSizeMax to the highest RECON round threshold the genome
# can support. Thresholds are cumulative across rounds (r2=3M, r3=9M, r4=27M,
# r5=81M, r6=243M), so the samplable size must cover all rounds up to the cap:
# >= 363M (3+9+27+81+243): no cap -> all 6 rounds
# >= 120M (3+9+27+81): cap 81M -> rounds 1-5
# >= 39M (3+9+27): cap 27M -> rounds 1-4
# >= 12M (3+9): cap 9M -> rounds 1-3
# < 12M: cap 3M -> rounds 1-2
SAMPLE_FLAG=""
if [ "${GENOME_SIZE}" -ge 363000000 ] 2>/dev/null; then
: # genome large enough for all rounds; use RepeatModeler default
elif [ "${GENOME_SIZE}" -ge 120000000 ] 2>/dev/null; then
echo "Capping at round 5 (-genomeSampleSizeMax 81000000)"
SAMPLE_FLAG="-genomeSampleSizeMax 81000000"
elif [ "${GENOME_SIZE}" -ge 39000000 ] 2>/dev/null; then
echo "Capping at round 4 (-genomeSampleSizeMax 27000000)"
SAMPLE_FLAG="-genomeSampleSizeMax 27000000"
elif [ "${GENOME_SIZE}" -ge 12000000 ] 2>/dev/null; then
echo "Capping at round 3 (-genomeSampleSizeMax 9000000)"
SAMPLE_FLAG="-genomeSampleSizeMax 9000000"
else
echo "Capping at round 2 (-genomeSampleSizeMax 3000000)"
SAMPLE_FLAG="-genomeSampleSizeMax 3000000"
fi
RepeatModeler -threads ${ProcNum} -database ${OUTDIR}/${species}_Database/${species} ${SAMPLE_FLAG}
if [ ! -e ${OUTDIR}/${species}_Database/${species}-families.fa ]; then
echo "ERROR: RepeatModeler Failed"
exit 2
fi
}
# Subprocess strainer
# contains the BLAST, Extract, Extend, Trim pipeline from James Galbraith
strainer()
{
cd ${OUTDIR}/${species}_strainer/
if [ "$ProcNum" -ge 4 ]; then
strainthreads=$(( ProcNum / 4 ))
else
strainthreads=$ProcNum
fi
${SCRIPT_DIR}/TEstrainer/TEstrainer_for_earlGrey.sh -g $genome -l ${OUTDIR}/${species}_Database/${species}-families.fa -t ${strainthreads} -f $Flank -r $num -n $no_seq -m $min_seq$([ "$quietBar" == "yes" ] && echo " -q")
latestFile="$(realpath $(ls -td -- ${OUTDIR}/${species}_strainer/*/ | head -n 1) || true)/${species}-families.fa.strained"
cp $latestFile ${OUTDIR}/${species}_strainer/
}
# Subprocess strainerResume
# resumes interrupted TEstrainer runs
strainerResume() {
cd ${OUTDIR}/${species}_strainer/
strainDataDir=$(find . -maxdepth 1 -type d -name "TS_${species}-families.fa_*")
if [ ! $(echo "$strainDataDir" | wc -l) -eq 1 ]; then
echo "ERROR: There are more TEstrainer directories than expected present"
exit 2
fi
if [ "$ProcNum" -ge 4 ]; then
strainthreads=$(( ProcNum / 4 ))
else
strainthreads=$ProcNum
fi
${SCRIPT_DIR}/TEstrainer/TEstrainer_for_earlGrey.sh -g $genome -l ${OUTDIR}/${species}_Database/${species}-families.fa -t ${strainthreads} -f $Flank -r $num -n $no_seq -m $min_seq$([ "$quietBar" == "yes" ] && echo " -q") -d ${strainDataDir} -R
latestFile="$(realpath $(ls -td -- ${OUTDIR}/${species}_strainer/*/ | head -n 1) || true)/${species}-families.fa.strained"
cp $latestFile ${OUTDIR}/${species}_strainer/
}
# Subprocess clust
# optional wicker clustering of TE library consensus sequences to reduce redundancy
clust()
{
cd ${OUTDIR}/${species}_strainer/
cd-hit-est -d 0 -aS 0.8 -c 0.8 -G 0 -g 1 -b 500 -r 1 -i $latestFile -o ${latestFile}.clstrd.fa
latestFile=$latestFile.clstrd.fa
}
# Subprocess novoMask
# Run RepeatMasker with the final processed library
novoMask()
{
if [ -z "$RepSpec" ] && [ -z "$startCust" ]; then
cd ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/
rmthreads=$(( ProcNum / 4 ))
RepeatMasker -lib $latestFile -no_is -lcambig -s -a -pa $rmthreads -dir ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/ $genome
else
mkdir -p $OUTDIR/${species}_Curated_Library/
cd ${OUTDIR}/${species}_Curated_Library/
cat $latestFile $RepSub > ${species}_combined_library.fasta
latestFile=$(realpath ${species}_combined_library.fasta)
cd ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/
rmthreads=$(( ProcNum / 4 ))
RepeatMasker -lib $latestFile -no_is -lcambig -s -a -pa $rmthreads -dir ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/ $genome
fi
}
# Subprocess heliano
# Run HELIANO as an optional step, then replace overlapping repeats in the merged output with HELIANO outputs
### TODO:
heliano_optional()
{
cd ${OUTDIR}/${species}_heliano/
heliano -g $genome --nearest -dn 6000 -flank_sim 0.5 -o ${OUTDIR}/${species}_heliano/HEL_${timestamp} -w 10000 -n $ProcNum
awk '{OFS="\t"}{print $1, "HELIANO", "RC/Helitron", $2+1, $3, $5, $6, ".", "ID="$9"_"$11";shortTE=F"}' ${OUTDIR}/${species}_heliano/HEL_${timestamp}/RC.representative.bed > ${OUTDIR}/${species}_heliano/HEL_${timestamp}/RC.representative.gff
helitron_gff=${OUTDIR}/${species}_heliano/HEL_${timestamp}/RC.representative.gff
}
# Subprocess rcMergeRepeats
# Defragment repeat sequences to adjust for insertion times
mergeRep()
{
mkdir ${OUTDIR}/${species}_mergedRepeats/looseMerge
if [ -s "$helitron_gff" ]; then
echo "Running loose merge with HELIANO output"
${SCRIPT_DIR}/rcMergeRepeatsLoose -f $genome -s $species -d ${OUTDIR}/${species}_mergedRepeats/looseMerge -u ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).out -q ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).tbl -t $ProcNum -b ${dict} -m $margin -e $helitron_gff
else
${SCRIPT_DIR}/rcMergeRepeatsLoose -f $genome -s $species -d ${OUTDIR}/${species}_mergedRepeats/looseMerge -u ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).out -q ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).tbl -t $ProcNum -b ${dict} -m $margin
fi
if [ -f "${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed" ]; then
awk '{OFS="\t"}{print $1, $2, $3, $4, $5, $6, $7, $8, toupper($9)}' ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff > ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff.1 && mv ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff{.1,}
fi
if [ ! -f "${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed" ]; then
echo "ERROR: loose merge defragmentation failed, trying strict merge..."
cd ${OUTDIR}/${species}_mergedRepeats/
if [ -s "$helitron_gff" ]; then
${SCRIPT_DIR}/rcMergeRepeats -f $genome -s $species -d ${OUTDIR}/${species}_mergedRepeats/ -u ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).out -q ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).tbl -t $ProcNum -b ${dict} -m $margin -e $helitron_gff
else
${SCRIPT_DIR}/rcMergeRepeats -f $genome -s $species -d ${OUTDIR}/${species}_mergedRepeats/ -u ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).out -q ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).tbl -t $ProcNum -b ${dict} -m $margin
fi
if [ ! -f "${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.bed" ]; then
echo "ERROR: strict merge also failed, check ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).out looks as expected"
exit 2
fi
fi
}
# Subprocess pieChart
# Generate a pie chart summary of TE content
charts()
{
cd ${OUTDIR}/${species}_summaryFiles/
if [ -f "${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed" ]; then
${SCRIPT_DIR}/autoPie.sh -i ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.summary -t ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).tbl -p ${OUTDIR}/${species}_summaryFiles/${species}.summaryPie.pdf -o ${OUTDIR}/${species}_summaryFiles/${species}.highLevelCount.txt
else
${SCRIPT_DIR}/autoPie.sh -i ${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.summary -t ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).tbl -p ${OUTDIR}/${species}_summaryFiles/${species}.summaryPie.pdf -o ${OUTDIR}/${species}_summaryFiles/${species}.highLevelCount.txt
fi
}
# Subprocess calcDivRL
# Calculate divergence estimates
calcDivRL()
{
cd ${OUTDIR}/${species}_RepeatLandscape
# divergence_calc.py calls pybedtools.set_tempdir() which sets tempfile.tempdir
# globally. forkserver creates its AF_UNIX socket under tempfile.tempdir, so a
# long output path can exceed the 108-char socket limit (OSError: AF_UNIX path
# too long). Pass a short per-species path in /tmp to avoid this.
divcalc_tmp="/tmp/egdiv_${species}"
mkdir -p "${divcalc_tmp}"
if [ -z "$RepSpec" ] && [ -z "$startCust" ]; then
python ${SCRIPT_DIR}/divergenceCalc/divergence_calc.py -l $latestFile -g $genOrig -i ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff -o ${OUTDIR}/${species}_RepeatLandscape/${species}.filteredRepeats.withDivergence.gff -t $ProcNum -tmp "${divcalc_tmp}"
Rscript ${SCRIPT_DIR}/divergenceCalc/divergence_plot.R -s $species -g ${OUTDIR}/${species}_RepeatLandscape/${species}.filteredRepeats.withDivergence.gff -o ${OUTDIR}/${species}_RepeatLandscape/ && \
awk -F'\t' 'BEGIN{OFS="\t"} {gsub("NAME=","Name=",$9); gsub("TSTART=","tstart=",$9); gsub("TEND=","tend=",$9); gsub("SHORTTE=","shortte=",$9); gsub("TEGROUP=","tegroup=",$9); gsub("KIMURA80=","kimura80=",$9); gsub("NESTED=", "nested=", $9); print}' ${OUTDIR}/${species}_RepeatLandscape/${species}.filteredRepeats.withDivergence.gff > ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff
rm -rf "${divcalc_tmp}"
cp ${OUTDIR}/${species}_RepeatLandscape/*.pdf ${OUTDIR}/${species}_summaryFiles/
cp ${OUTDIR}/${species}_RepeatLandscape/*_summary_table.tsv ${OUTDIR}/${species}_summaryFiles/${species}_divergence_summary_table.tsv
else
python ${SCRIPT_DIR}/divergenceCalc/divergence_calc.py -l ${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta -g $genOrig -i ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff -o ${OUTDIR}/${species}_RepeatLandscape/${species}.filteredRepeats.withDivergence.gff -t $ProcNum -tmp "${divcalc_tmp}"
Rscript ${SCRIPT_DIR}/divergenceCalc/divergence_plot.R -s $species -g ${OUTDIR}/${species}_RepeatLandscape/${species}.filteredRepeats.withDivergence.gff -o ${OUTDIR}/${species}_RepeatLandscape/ && \
awk -F'\t' 'BEGIN{OFS="\t"} {gsub("NAME=","Name=",$9); gsub("TSTART=","tstart=",$9); gsub("TEND=","tend=",$9); gsub("SHORTTE=","shortte=",$9); gsub("TEGROUP=","tegroup=",$9); gsub("KIMURA80=","kimura80=",$9); gsub("NESTED=", "nested=", $9); print}' ${OUTDIR}/${species}_RepeatLandscape/${species}.filteredRepeats.withDivergence.gff > ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff
rm -rf "${divcalc_tmp}"
cp ${OUTDIR}/${species}_RepeatLandscape/*.pdf ${OUTDIR}/${species}_summaryFiles/
cp ${OUTDIR}/${species}_RepeatLandscape/*_summary_table.tsv ${OUTDIR}/${species}_summaryFiles/${species}_divergence_summary_table.tsv
fi
}
# Subprocess sweepUp
# Puts required files into a summary folder
sweepUp()
{
cd ${OUTDIR}/${species}_summaryFiles/
if [ -f "${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed" ] && [ ! -f "${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta" ]; then
cp ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff $latestFile ${OUTDIR}/${species}_summaryFiles/
elif [ -f "${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed" ] && [ -f "${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta" ]; then
cp ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff ${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta $latestFile ${OUTDIR}/${species}_summaryFiles/
elif [ ! -f "${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed" ] && [ ! -f "${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta" ]; then
cp ${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.bed ${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.gff $latestFile ${OUTDIR}/${species}_summaryFiles/
elif [ ! -f "${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed" ] && [ -f "${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta" ]; then
cp ${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.bed ${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.gff ${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta $latestFile ${OUTDIR}/${species}_summaryFiles/
else
echo "Error: Cannot find required summary files, please check one of the following is present: ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed and/or ${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.bed"
fi
}
# Subprocess runningTea
runningTea()
{
echo "
) (
( ) )
) ( (
_______)_
.-'---------|
( C|/\/\/\/\/|
'-./\/\/\/\/|
'_________'
'-------'
<<< $stage >>>"
}
# Subprocess GetTime
convertsecs()
{
h=$(bc <<< "${1}/3600")
m=$(bc <<< "(${1}%3600)/60")
s=$(bc <<< "${1}%60")
printf "%02d:%02d:%05.2f\n" $h $m $s
}
# Subprocess Checks
Checks()
{
if [ -z "$genome" ] || [ -z "$species" ] || [ -z "$directory" ] ; then
usage; exit 1
fi
if [ -z "$ProcNum" ] ; then
ProcNum=1; echo "$ProcNum Cores Will Be Used"
else
echo "$ProcNum Cores Will Be Used"
fi
if [ -z "$RepSpec" ] && [ -z "$startCust" ] ; then
echo "RepeatMasker species not specified, running Earl Grey without an initial mask with known repeats"
else
echo "Running Earl Grey with an intial mask with known repeats"
fi
if [ -z "$num" ] ; then
num=10; echo "De Novo Sequences Will Be Extended Through a Maximum of $num Iterations"
else
echo "De Novo Sequences Will Be Extended Through a Maximum of $num Iterations"
fi
if [ -z "$no_seq" ] ; then
no_seq=20; echo "$no_seq sequences will be used in BEAT consensus generation"
else
echo "$no_seq sequences will be used in BEAT consensus generation"
fi
if [ -z "$cluster" ] || [ "$cluster" == "no" ] ; then
cluster=no; echo "TE library consensus sequences will not be clustered"
elif [ "$cluster" == "yes" ]; then
cluster=yes; echo "TE library consensus sequences will be clustered, be aware of the impact this can have on subfamilies and chimeric TEs!"
else
cluster=no; echo "Clustering not specified using (yes/no). Using default parameter (no)."
fi
if [ -z "$softMask" ] || [ "$softMask" == "no" ] ; then
softMask=no; echo "Softmasked genome will not be generated"
elif [ "$softMask" == "yes" ]; then
softMask=yes; echo "Softmasked genome will be generated"
else
softMask=no; echo "Softmask not specified using (yes/no). Using default parameter (no)."
fi
if [ "$margin" == "yes" ] ; then
margin=yes; echo "Short TE sequences <100bp will be removed from annotation"
elif [ -z "$margin" ] || [ "$margin" == "no" ]; then
margin=no; echo "Short TE sequences <100bp will not be removed from annotation"
else
margin=no; echo "Margin not specified using (yes/no). Using default parameter (no)."
fi
if [ -z "$Flank" ]; then
Flank=1000; echo "Blast, Extend, Align, Trim Process Will Add 1000bp to Each End in Each Iteration"
else
echo "Blast, Extract, Extend, Trim Process Will Add $Flank to Each End in Each Iteration"
fi
if [ -z "$min_seq" ]; then
min_seq=3; echo "Blast, Extend, Align, Trim Process Will Require 3 Sequences to Generate a New Consensus Sequence"
else
echo "Blast, Extend, Align, Trim Process Will Require $min_seq Sequences to Generate a New Consensus Sequence"
fi
if [ ! -d "$SCRIPT_DIR" ]; then
echo "ERROR: Script directory variable not set, please run the configure script in the Earl Grey directory before attempting to run Earl Grey"; exit 1
fi
if [ ! -d "$SCRIPT_DIR/TEstrainer/" ]; then
echo "ERROR: teStrainer module not found, please check all modules are present and run the configure script in the Earl Grey directory before attempting to run Earl Grey"; exit 1
fi
if [ "$heli" == "yes" ] ; then
heli=yes; echo "HELITRON detection will be run using HELIANO"
elif [ -z "$heli" ] || [ "$heli" == "no" ]; then
heli=no; echo "HELITRON detection will not be run"
else
heli=no; echo "HELITRON detection not specified using (yes/no). Using default parameter (no)."
fi
if [ "$quietBar" == "yes" ] ; then
quietBar=yes; echo "TEstrainer parallel progress bar will be suppressed"
elif [ -z "$quietBar" ] || [ "$quietBar" == "no" ]; then
quietBar=no; echo "TEstrainer parallel progress bar will be shown"
else
quietBar=no; echo "quietBar not specified using (yes/no). Using default parameter (no)."
fi
echo "Please cite the following paper when using this software:"
echo "Baril, T., Galbraith, J. and Hayward, A., 2024. Earl Grey: a fully automated user-friendly transposable element annotation and analysis pipeline. Molecular Biology and Evolution, 41(4), p.msae068."
}
# Subprocess Dfam Checks
dfamCheck()
{
# Dfam 3.9 checks
library_path=$(which RepeatMasker | sed 's|/bin/RepeatMasker|/share/RepeatMasker/Libraries/famdb|g')
expected_file="${library_path}/dfam39_full.0.h5"
file_count=$(find "$library_path" -maxdepth 1 -type f | wc -l)
if [ $file_count -eq 2 ] && [ -f $expected_file ] && [ -f ${library_path}/rmlib.config ] ; then
echo "WARNING: Earl Grey v6.1.0 has updated to Dfam v3.9."
echo "Before using Earl Grey, you MUST download the required partitions from Dfam (https://dfam.org/releases/current/families/FamDB/)"
echo "These must be located at ${library_path}/. Then, you must reconfigure RepeatMasker before using Earl Grey."
echo "I have written the required commands for you to faciliate the process."
echo "Please follow these carefully to ensure successful configuration of RepeatMasker with Dfam 3.9."
echo "As a note, these DB paritions are BIG, so only download the ones you need for your studies (or, if you do need all of them, make sure you have the space for them)"
echo "########################################################"
echo "A script for modification and automation of these steps has been generated in $(pwd)/configure_dfam39.sh"
echo "Please open the script in your preferred text editor (e.g. nano or vim) and change the partition numbers in [] to those that you would like to download"
echo "then, make the script executable (chmod +x $(pwd)/configure_dfam39.sh) and run it: ./configure_dfam39.sh"
echo "Alternatively, you can run the steps below one-by-one to achieve the same outcome:"
echo "########################################################"
echo ""
if [ -f "$(pwd)/configure_dfam39.sh" ]; then
rm $(pwd)/configure_dfam39.sh
fi
echo "# first, change directory to the famdb library location" | tee -a $(pwd)/configure_dfam39.sh
echo "cd ${library_path}/" | tee -a $(pwd)/configure_dfam39.sh
echo "" | tee -a $(pwd)/configure_dfam39.sh
echo "# download the partitions you require from Dfam 3.9. In the below, change the numbers or range inside the square brackets to choose your subsets." | tee -a $(pwd)/configure_dfam39.sh
echo "# e.g. to download partitions 0 to 10: [0-10]; or to download partitions 3,5, and 7: [3,5,7]; [0-16] is ALL PARTITIONS" | tee -a $(pwd)/configure_dfam39.sh
echo "curl -o 'dfam39_full.#1.h5.gz' 'https://dfam.org/releases/current/families/FamDB/dfam39_full.[0-16].h5.gz'" | tee -a $(pwd)/configure_dfam39.sh
echo "" | tee -a $(pwd)/configure_dfam39.sh
echo "# decompress Dfam 3.9 paritions" | tee -a $(pwd)/configure_dfam39.sh
echo "gunzip *.gz" | tee -a $(pwd)/configure_dfam39.sh
echo "" | tee -a $(pwd)/configure_dfam39.sh
echo "# move up to RepeatMasker main directory" | tee -a $(pwd)/configure_dfam39.sh
echo "cd $(which RepeatMasker | sed 's|/bin/RepeatMasker|/share/RepeatMasker/|g')" | tee -a $(pwd)/configure_dfam39.sh
echo "" | tee -a $(pwd)/configure_dfam39.sh
#echo "# Replace famdb with version 2.0.1 (required for the Dfam 3.9)" | tee -a $(pwd)/configure_dfam39.sh
#echo "wget https://github.com/Dfam-consortium/FamDB/archive/refs/tags/2.0.1.tar.gz && tar --wildcards -zxvf 2.0.1.tar.gz FamDB-2.0.1/famdb.py FamDB-2.0.1/famdb_*.py" | tee -a $(pwd)/configure_dfam39.sh
#echo "mv FamDB-2.0.1/*.py . && rm -rf 2.0.1.tar.gz FamDB-2.0.1/" | tee -a $(pwd)/configure_dfam39.sh
#echo "" | tee -a $(pwd)/configure_dfam39.sh
echo "# save the min_init partition as a backup, just in case!" | tee -a $(pwd)/configure_dfam39.sh
echo "mv ${library_path}/min_init.0.h5 ${library_path}/min_init.0.h5.bak" | tee -a $(pwd)/configure_dfam39.sh
echo "" | tee -a $(pwd)/configure_dfam39.sh
echo "# Rerun RepeatMasker configuration" | tee -a $(pwd)/configure_dfam39.sh
echo "perl ./configure -libdir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/share/RepeatMasker/Libraries/|g') \
-trf_prgm $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin/trf|g') \
-rmblast_dir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin|g') \
-hmmer_dir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin|g') \
-abblast_dir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin|g') \
-crossmatch_dir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin|g') \
-default_search_engine rmblast" | tee -a $(pwd)/configure_dfam39.sh
echo "" | tee -a $(pwd)/configure_dfam39.sh
sed -i '/^perl/ s/\s\s*/ /g' $(pwd)/configure_dfam39.sh
echo "########################################################"
echo ""
echo "Following these steps should enable you to use Dfam 3.9 with RepeatMasker and Earl Grey"
echo "You will not see this message again if configuration is successful"
echo "If you only want partition 0, you do not have to do anything except rerun your Earl Grey command"
echo "If you are still having trouble, please open an issue on the Earl Grey github and we will try to help!"
touch ${library_path}/.earlgrey.config.complete
exit 2
fi
}
# Main #
while getopts g:s:o:t:f:i:r:c:l:m:d:n:a:e:q:h option
do
case "${option}"
in
g) genome=${OPTARG};;
s) species=${OPTARG};;
o) directory=${OPTARG};;
t) ProcNum=${OPTARG};;
f) Flank=${OPTARG};;
i) num=${OPTARG};;
r) RepSpec=${OPTARG};;
l) startCust=${OPTARG};;
c) cluster=${OPTARG};;
m) margin=${OPTARG};;
d) softMask=${OPTARG};;
n) no_seq=${OPTARG};;
a) min_seq=${OPTARG};;
e) heli=${OPTARG};;
q) quietBar=${OPTARG};;
h) usage; exit 0;;
esac
done
SECONDS=0
# Step 1 - set up the directories and make sure all modules are present
stage="Checking Parameters" && runningTea
SCRIPT_DIR=/data/toby/EarlGrey/scripts/
dfamCheck
Checks
stage="Making Directories" && runningTea
makeDirectory
# Start Logs - only bother starting if everything is present and correct
exec > >(tee "${OUTDIR}/${species}EarlGrey.log") 2>&1
stage="Cleaning Genome" && runningTea
prepGenome
sleep 1
# Step 2 - initial annotation, either de novo, or if required an initial RepeatMasker followed by a de novo run
if [ -z "$RepSpec" ]; then
sleep 1
else
if [ ! -f ${OUTDIR}/${species}_RepeatMasker/*.masked ]; then
stage="Getting RepeatMasker Sequences for $RepSpec and Saving as Fasta" && runningTea
getRepeatMaskerFasta
stage="Running Initial Mask with Known Repeats" && runningTea
firstMask
else
stage="Genome has already been masked, skipping..." && runningTea
getRepeatMaskerFasta
fi
fi
if [ -z "$startCust" ]; then
sleep 1
else
if [ ! -f ${OUTDIR}/${species}_RepeatMasker/*.masked ]; then
stage="Running Initial Mask with Known Repeats in Custom Library" && runningTea
firstMaskCustomLib
else
stage="Genome has already been masked, skipping..." && runningTea
fi
fi
# Step 3 - make a database from the genome, and then run a de novo TE annotation using RepeatModeler2
if [ ! -f ${OUTDIR}/${species}_Database/${species}-families.fa ]; then
stage="Detecting Novel Repeats" && runningTea
buildDB
deNovo1
sleep 1
else
stage="De novo repeats have already been detected, skipping..." && runningTea
sleep 1
fi
# Step 4 - run TEstrainer, which runs a BLAST, extract, extent, trim, on the the de novo repeat library to refine consensus sequences
if [ ! -s ${OUTDIR}/${species}_strainer/${species}-families.fa.strained ]; then
stage="Straining TEs and Refining de novo Consensus Sequences" && runningTea
if [ -f ${OUTDIR}/${species}_strainer/${species}-families.fa.strained ]; then
rm -r ${OUTDIR}/${species}_strainer/${species}-families.fa.strained ${OUTDIR}/${species}_strainer/TS*
strainer
elif [ -d ${OUTDIR}/${species}_strainer/TS_${species}-families.fa_* ]; then
stage="Resuming TEstrainer" && runningTea
strainerResume
else
strainer
fi
if [ ! -s ${OUTDIR}/${species}_strainer/${species}-families.fa.strained ]; then
echo "WARNING: TEstrainer failed to produce a strain file, please check the log file for more information. If you have run an intial mask with known repeats, this could be due RepeatModeler2 failing to identify any new repeats. Please check if this is expected."
fi
sleep 1
else
stage="TEs already strained, skipping..." && runningTea
latestFile=${OUTDIR}/${species}_strainer/${species}-families.fa.strained
sleep 1
fi
# Step 4.5 - cluster TE library IF REQUIRED
if [ "$cluster" == "yes" ]; then
stage="Clustering TE sequences to Wicker TE Family Level (80-80-80 rule)" && runningTea
clust
sleep 1
fi
# Stage 5 - run the final RepeatMasker annotation using the refined TE consensus library
if [ ! -f ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/*.tbl ]; then
stage="Identifying Repeats Using Species-Specific Library" && runningTea
novoMask
if [ ! -f ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/*.tbl ]; then
echo "ERROR: RepeatMasker failed, please check logs" && exit 2
fi
sleep 1
else
stage="Final masking already complete, skipping..." && runningTea
sleep 1
fi
# Stage 5.5 - Run HELIANO as an optional step
if [ "$heli" == "yes" ]; then
if [ ! -s ${OUTDIR}/${species}_heliano/*/RC.representative.gff ]; then
stage="Running HELIANO to Detect Helitrons" && runningTea
timestamp=$(date +"%Y%m%d_%H%M")
heliano_optional
else
stage="HELITRON detection already complete, skipping..." && runningTea
helitron_gff="$(realpath $(ls -td -- ${OUTDIR}/${species}_heliano/*/ | head -n 1) || true)/RC.representative.gff"
echo "HELITRON GFF: $helitron_gff"
fi
sleep 1
fi
# Stage 6
if [ ! -f ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed ] ; then
stage="Defragmenting Repeats" && runningTea
mergeRep
sleep 1
else
stage="Repeats already defragmented, skipping..." && runningTea
sleep 1
fi
# Stage 7
stage="Generating Summary Plots" && runningTea
charts
calcDivRL
sleep 1
# Stage 8
stage="Tidying Directories and Organising Important Files" && runningTea
sweepUp
sleep 1
# Stage 8.5 Optional Softmask
if [ "$softMask" == "yes" ]; then
stage="Generating Softmasked Genome" && runningTea
gunzip ${genome%.prep}.bak.gz && bedtools maskfasta -fi ${genome%.prep}.bak -bed ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed -fo ${OUTDIR}/${species}_summaryFiles/${species}.softmasked.fasta -soft && gzip ${genome%.prep}.bak
sleep 1
fi
# Stage 9
time=$(convertsecs $SECONDS)
stage="Done in $time" && runningTea
sleep 5
# Stage 10
stage="TE library, Summary Figures, and TE Quantifications in Standard Formats Can Be Found in ${OUTDIR}/${species}_summaryFiles/" && runningTea
cat ${OUTDIR}/${species}_summaryFiles/${species}.highLevelCount.kable
sleep 5
================================================
FILE: earlGreyAnnotationOnly
================================================
#!/bin/bash
set -eo pipefail
usage()
{
echo " #############################
earlGrey version 7.2.4 (AnnotationOnly)
Required Parameters:
-g == genome.fasta
-s == species name
-o == output directory
-l == Starting consensus library for annotation (in fasta format)
Optional Parameters:
-t == Number of Threads (DO NOT specify more than are available)
-r == RepeatMasker species for addition to custom library (Default: None)
-m == Remove putative spurious TE annotations <100bp? (yes/no, Default: no)
-d == Create soft-masked genome at the end? (yes/no, Default: no)
-e == Run HELIANO as an optional step to detect Helitrons (yes/no, Default: no)
-h == Show help
Example Usage:
earlGreyAnnotationOnly -g bombyxMori.fasta -s bombyxMori -o /home/toby/bombyxMori/repeatAnnotation/ -l bombyx-families.fa.strained -t 16
Queries can be sent to:
tobias.baril[at]unine.ch
Please make use of the GitHub Issues and Discussion Tabs at: https://github.com/TobyBaril/EarlGrey
#############################"
}
# Subprocess Make Directories #
makeDirectory()
{
directory=$(realpath ${directory})
mkdir -p ${directory}/${species}_EarlGrey/ && OUTDIR=${directory}/${species}_EarlGrey
mkdir -p ${OUTDIR}/${species}_Curated_Library/
mkdir -p $OUTDIR/${species}_RepeatMasker_Against_Custom_Library/
mkdir -p $OUTDIR/${species}_RepeatLandscape/
mkdir -p $OUTDIR/${species}_mergedRepeats/
mkdir -p ${OUTDIR}/${species}_summaryFiles/
if [ ! -z "$heli" ]; then
mkdir -p ${OUTDIR}/${species}_heliano/
fi
}
# Subprocess PrepGenome #
prepGenome()
{
if [ ! -L ${genome} ]; then
genome=$(realpath ${genome})
fi
if [ -L $genome ]; then
genome=$(realpath -s ${genome})
fi
if [ ! -f ${genome}.prep ] || [ ! -f ${genome}.dict ]; then
cp ${genome} ${genome}.bak && gzip -f ${genome}.bak
sed '/>/ s/[[:space:]].*//g; /^$/d' ${genome} > ${genome}.tmp
${SCRIPT_DIR}/headSwap.sh -i ${genome}.tmp -o ${genome}.prep && rm ${genome}.tmp
mv ${genome}.tmp.dict ${genome}.dict
sed -i '/^>/! s/[DVHBPE]/N/g' ${genome}.prep
dict=${genome}.dict
genOrig=${genome}
genome=${genome}.prep
else
dict=${genome}.dict
genOrig=${genome}
genome=${genome}.prep
fi
}
# Subprocess novoMask
# Run RepeatMasker with the final processed library
novoMask()
{
if [ -z "$RepSpec" ]; then
cd ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/
rmthreads=$(( ProcNum / 4 ))
RepeatMasker -lib $latestFile -no_is -lcambig -s -a -pa $rmthreads -dir ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/ $genome
else
mkdir -p $OUTDIR/${species}_Curated_Library/
cd ${OUTDIR}/${species}_Curated_Library/
cat $latestFile $RepSub > ${species}_combined_library.fasta
latestFile=$(realpath ${species}_combined_library.fasta)
cd ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/
rmthreads=$(( ProcNum / 4 ))
RepeatMasker -lib $latestFile -no_is -lcambig -s -a -pa $rmthreads -dir ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/ $genome
fi
}
# Subprocess heliano
# Run HELIANO as an optional step, then replace overlapping repeats in the merged output with HELIANO outputs
### TODO:
heliano_optional()
{
cd ${OUTDIR}/${species}_heliano/
heliano -g $genome --nearest -dn 6000 -flank_sim 0.5 -o ${OUTDIR}/${species}_heliano/HEL_${timestamp} -w 10000 -n $ProcNum
awk '{OFS="\t"}{print $1, "HELIANO", "RC/Helitron", $2+1, $3, $5, $6, ".", "ID="$9"_"$11";shortTE=F"}' ${OUTDIR}/${species}_heliano/HEL_${timestamp}/RC.representative.bed > ${OUTDIR}/${species}_heliano/HEL_${timestamp}/RC.representative.gff
helitron_gff=${OUTDIR}/${species}_heliano/HEL_${timestamp}/RC.representative.gff
}
# Subprocess rcMergeRepeats
# Defragment repeat sequences to adjust for insertion times
mergeRep()
{
mkdir ${OUTDIR}/${species}_mergedRepeats/looseMerge
if [ -s "$helitron_gff" ]; then
echo "Running loose merge with HELIANO output"
${SCRIPT_DIR}/rcMergeRepeatsLoose -f $genome -s $species -d ${OUTDIR}/${species}_mergedRepeats/looseMerge -u ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).out -q ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).tbl -t $ProcNum -b ${dict} -m $margin -e $helitron_gff
else
${SCRIPT_DIR}/rcMergeRepeatsLoose -f $genome -s $species -d ${OUTDIR}/${species}_mergedRepeats/looseMerge -u ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).out -q ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).tbl -t $ProcNum -b ${dict} -m $margin
fi
if [ -f "${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed" ]; then
awk '{OFS="\t"}{print $1, $2, $3, $4, $5, $6, $7, $8, toupper($9)}' ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff > ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff.1 && mv ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff{.1,}
fi
if [ ! -f "${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed" ]; then
echo "ERROR: loose merge defragmentation failed, trying strict merge..."
cd ${OUTDIR}/${species}_mergedRepeats/
if [ -s "$helitron_gff" ]; then
${SCRIPT_DIR}/rcMergeRepeats -f $genome -s $species -d ${OUTDIR}/${species}_mergedRepeats/ -u ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).out -q ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).tbl -t $ProcNum -b ${dict} -m $margin -e $helitron_gff
else
${SCRIPT_DIR}/rcMergeRepeats -f $genome -s $species -d ${OUTDIR}/${species}_mergedRepeats/ -u ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).out -q ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).tbl -t $ProcNum -b ${dict} -m $margin
fi
if [ ! -f "${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.bed" ]; then
echo "ERROR: strict merge also failed, check ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).out looks as expected"
exit 2
fi
fi
}
# Subprocess pieChart
# Generate a pie chart summary of TE content
charts()
{
cd ${OUTDIR}/${species}_summaryFiles/
if [ -f "${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed" ]; then
${SCRIPT_DIR}/autoPie.sh -i ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.summary -t ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).tbl -p ${OUTDIR}/${species}_summaryFiles/${species}.summaryPie.pdf -o ${OUTDIR}/${species}_summaryFiles/${species}.highLevelCount.txt
else
${SCRIPT_DIR}/autoPie.sh -i ${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.summary -t ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/$(basename $genome).tbl -p ${OUTDIR}/${species}_summaryFiles/${species}.summaryPie.pdf -o ${OUTDIR}/${species}_summaryFiles/${species}.highLevelCount.txt
fi
}
# Subprocess calcDivRL
# Calculate divergence estimates
calcDivRL()
{
cd ${OUTDIR}/${species}_RepeatLandscape
if [ -z "$RepSpec" ]; then
python ${SCRIPT_DIR}/divergenceCalc/divergence_calc.py -l $latestFile -g $genOrig -i ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff -o ${OUTDIR}/${species}_RepeatLandscape/${species}.filteredRepeats.withDivergence.gff -t $ProcNum
Rscript ${SCRIPT_DIR}/divergenceCalc/divergence_plot.R -s $species -g ${OUTDIR}/${species}_RepeatLandscape/${species}.filteredRepeats.withDivergence.gff -o ${OUTDIR}/${species}_RepeatLandscape/ && \
awk -F'\t' 'BEGIN{OFS="\t"} {gsub("NAME=","Name=",$9); gsub("TSTART=","tstart=",$9); gsub("TEND=","tend=",$9); gsub("SHORTTE=","shortte=",$9); gsub("TEGROUP=","tegroup=",$9); gsub("KIMURA80=","kimura80=",$9); gsub("NESTED=", "nested=", $9); print}' ${OUTDIR}/${species}_RepeatLandscape/${species}.filteredRepeats.withDivergence.gff > ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff
rm -rf ${OUTDIR}/${species}_RepeatLandscape/tmp/
cp ${OUTDIR}/${species}_RepeatLandscape/*.pdf ${OUTDIR}/${species}_summaryFiles/
cp ${OUTDIR}/${species}_RepeatLandscape/*_summary_table.tsv ${OUTDIR}/${species}_summaryFiles/${species}_divergence_summary_table.tsv
else
python ${SCRIPT_DIR}/divergenceCalc/divergence_calc.py -l ${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta -g $genOrig -i ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff -o ${OUTDIR}/${species}_RepeatLandscape/${species}.filteredRepeats.withDivergence.gff -t $ProcNum
Rscript ${SCRIPT_DIR}/divergenceCalc/divergence_plot.R -s $species -g ${OUTDIR}/${species}_RepeatLandscape/${species}.filteredRepeats.withDivergence.gff -o ${OUTDIR}/${species}_RepeatLandscape/ && \
awk -F'\t' 'BEGIN{OFS="\t"} {gsub("NAME=","Name=",$9); gsub("TSTART=","tstart=",$9); gsub("TEND=","tend=",$9); gsub("SHORTTE=","shortte=",$9); gsub("TEGROUP=","tegroup=",$9); gsub("KIMURA80=","kimura80=",$9); gsub("NESTED=", "nested=", $9); print}' ${OUTDIR}/${species}_RepeatLandscape/${species}.filteredRepeats.withDivergence.gff > ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff
rm -rf ${OUTDIR}/${species}_RepeatLandscape/tmp/
cp ${OUTDIR}/${species}_RepeatLandscape/*.pdf ${OUTDIR}/${species}_summaryFiles/
cp ${OUTDIR}/${species}_RepeatLandscape/*_summary_table.tsv ${OUTDIR}/${species}_summaryFiles/${species}_divergence_summary_table.tsv
fi
}
# Subprocess sweepUp
# Puts required files into a summary folder
sweepUp()
{
cd ${OUTDIR}/${species}_summaryFiles/
if [ -f "${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed" ] && [ ! -f "${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta" ]; then
cp ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff $latestFile ${OUTDIR}/${species}_summaryFiles/
elif [ -f "${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed" ] && [ -f "${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta" ]; then
cp ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.gff ${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta $latestFile ${OUTDIR}/${species}_summaryFiles/
elif [ ! -f "${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed" ] && [ ! -f "${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta" ]; then
cp ${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.bed ${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.gff $latestFile ${OUTDIR}/${species}_summaryFiles/
elif [ ! -f "${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed" ] && [ -f "${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta" ]; then
cp ${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.bed ${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.gff ${OUTDIR}/${species}_Curated_Library/${species}_combined_library.fasta $latestFile ${OUTDIR}/${species}_summaryFiles/
else
echo "Error: Cannot find required summary files, please check one of the following is present: ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed and/or ${OUTDIR}/${species}_mergedRepeats/${species}.filteredRepeats.bed"
fi
}
# Subprocess runningTea
runningTea()
{
echo "
) (
( ) )
) ( (
_______)_
.-'---------|
( C|/\/\/\/\/|
'-./\/\/\/\/|
'_________'
'-------'
<<< $stage >>>"
}
# Subprocess GetTime
convertsecs()
{
h=$(bc <<< "${1}/3600")
m=$(bc <<< "(${1}%3600)/60")
s=$(bc <<< "${1}%60")
printf "%02d:%02d:%05.2f\n" $h $m $s
}
# Subprocess Checks
Checks()
{
if [ -z "$genome" ] || [ -z "$species" ] || [ -z "$directory" ] || [ -z "$startCust" ] ; then
usage; exit 1
fi
if [ -z "$ProcNum" ] ; then
ProcNum=1; echo "$ProcNum Cores Will Be Used"
else
echo "$ProcNum Cores Will Be Used"
fi
if [ -z "$RepSpec" ]; then
echo "RepeatMasker species not specified, running Earl Grey without known repeats"
else
echo "Running Earl Grey with sequences from configured databases in addition to custom library"
fi
if [ -z "$softMask" ] || [ "$softMask" == "no" ] ; then
softMask=no; echo "Softmasked genome will not be generated"
elif [ "$softMask" == "yes" ]; then
softMask=yes; echo "Softmasked genome will be generated"
else
softMask=no; echo "Softmask not specified using (yes/no). Using default parameter (no)."
fi
if [ "$margin" == "yes" ] ; then
margin=yes; echo "Short TE sequences <100bp will be removed from annotation"
elif [ -z "$margin" ] || [ "$margin" == "no" ]; then
margin=no; echo "Short TE sequences <100bp will not be removed from annotation"
else
margin=no; echo "Margin not specified using (yes/no). Using default parameter (no)."
fi
if [ ! -d "$SCRIPT_DIR" ]; then
echo "ERROR: Script directory variable not set, please run the configure script in the Earl Grey directory before attempting to run Earl Grey"; exit 1
fi
if [ ! -d "$SCRIPT_DIR/TEstrainer/" ]; then
echo "ERROR: teStrainer module not found, please check all modules are present and run the configure script in the Earl Grey directory before attempting to run Earl Grey"; exit 1
fi
if [ "$heli" == "yes" ] ; then
heli=yes; echo "HELITRON detection will be run using HELIANO"
elif [ -z "$heli" ] || [ "$heli" == "no" ]; then
heli=no; echo "HELITRON detection will not be run"
else
heli=no; echo "HELITRON detection not specified using (yes/no). Using default parameter (no)."
fi
echo "Please cite the following paper when using this software:"
echo "Baril, T., Galbraith, J. and Hayward, A., 2024. Earl Grey: a fully automated user-friendly transposable element annotation and analysis pipeline. Molecular Biology and Evolution, 41(4), p.msae068."
# Dfam 3.9 checks
library_path=$(which RepeatMasker | sed 's|/bin/RepeatMasker|/share/RepeatMasker/Libraries/famdb|g')
expected_file="${library_path}/dfam39_full.0.h5"
file_count=$(find "$library_path" -maxdepth 1 -type f | wc -l)
if [ $file_count -eq 2 ] && [ -f $expected_file ] && [ -f ${library_path}/rmlib.config ] ; then
echo "WARNING: Earl Grey v6.1.0 has updated to Dfam v3.9."
echo "Before using Earl Grey, you MUST download the required partitions from Dfam (https://dfam.org/releases/current/families/FamDB/)"
echo "These must be located at ${library_path}/. Then, you must reconfigure RepeatMasker before using Earl Grey."
echo "I have written the required commands for you to faciliate the process."
echo "Please follow these carefully to ensure successful configuration of RepeatMasker with Dfam 3.9."
echo "As a note, these DB paritions are BIG, so only download the ones you need for your studies (or, if you do need all of them, make sure you have the space for them)"
echo "########################################################"
echo "A script for modification and automation of these steps has been generated in $(pwd)/configure_dfam39.sh"
echo "Please open the script in your preferred text editor (e.g. nano or vim) and change the partition numbers in [] to those that you would like to download"
echo "then, make the script executable (chmod +x $(pwd)/configure_dfam39.sh) and run it: ./configure_dfam39.sh"
echo "Alternatively, you can run the steps below one-by-one to achieve the same outcome:"
echo "########################################################"
echo ""
if [ -f "$(pwd)/configure_dfam39.sh" ]; then
rm $(pwd)/configure_dfam39.sh
fi
echo "# first, change directory to the famdb library location" | tee -a $(pwd)/configure_dfam39.sh
echo "cd ${library_path}/" | tee -a $(pwd)/configure_dfam39.sh
echo "" | tee -a $(pwd)/configure_dfam39.sh
echo "# download the partitions you require from Dfam 3.9. In the below, change the numbers or range inside the square brackets to choose your subsets." | tee -a $(pwd)/configure_dfam39.sh
echo "# e.g. to download partitions 0 to 10: [0-10]; or to download partitions 3,5, and 7: [3,5,7]; [0-16] is ALL PARTITIONS" | tee -a $(pwd)/configure_dfam39.sh
echo "curl -o 'dfam39_full.#1.h5.gz' 'https://dfam.org/releases/current/families/FamDB/dfam39_full.[0-16].h5.gz'" | tee -a $(pwd)/configure_dfam39.sh
echo "" | tee -a $(pwd)/configure_dfam39.sh
echo "# decompress Dfam 3.9 paritions" | tee -a $(pwd)/configure_dfam39.sh
echo "gunzip *.gz" | tee -a $(pwd)/configure_dfam39.sh
echo "" | tee -a $(pwd)/configure_dfam39.sh
echo "# move up to RepeatMasker main directory" | tee -a $(pwd)/configure_dfam39.sh
echo "cd $(which RepeatMasker | sed 's|/bin/RepeatMasker|/share/RepeatMasker/|g')" | tee -a $(pwd)/configure_dfam39.sh
echo "" | tee -a $(pwd)/configure_dfam39.sh
#echo "# Replace famdb with version 2.0.1 (required for the Dfam 3.9)" | tee -a $(pwd)/configure_dfam39.sh
#echo "wget https://github.com/Dfam-consortium/FamDB/archive/refs/tags/2.0.1.tar.gz && tar --wildcards -zxvf 2.0.1.tar.gz FamDB-2.0.1/famdb.py FamDB-2.0.1/famdb_*.py" | tee -a $(pwd)/configure_dfam39.sh
#echo "mv FamDB-2.0.1/*.py . && rm -rf 2.0.1.tar.gz FamDB-2.0.1/" | tee -a $(pwd)/configure_dfam39.sh
#echo "" | tee -a $(pwd)/configure_dfam39.sh
echo "# save the min_init partition as a backup, just in case!" | tee -a $(pwd)/configure_dfam39.sh
echo "mv ${library_path}/min_init.0.h5 ${library_path}/min_init.0.h5.bak" | tee -a $(pwd)/configure_dfam39.sh
echo "" | tee -a $(pwd)/configure_dfam39.sh
echo "# Rerun RepeatMasker configuration" | tee -a $(pwd)/configure_dfam39.sh
echo "perl ./configure -libdir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/share/RepeatMasker/Libraries/|g') \
-trf_prgm $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin/trf|g') \
-rmblast_dir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin|g') \
-hmmer_dir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin|g') \
-abblast_dir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin|g') \
-crossmatch_dir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin|g') \
-default_search_engine rmblast" | tee -a $(pwd)/configure_dfam39.sh
echo "" | tee -a $(pwd)/configure_dfam39.sh
sed -i '/^perl/ s/\s\s*/ /g' $(pwd)/configure_dfam39.sh
echo "########################################################"
echo ""
echo "Following these steps should enable you to use Dfam 3.9 with RepeatMasker and Earl Grey"
echo "You will not see this message again if configuration is successful"
echo "If you only want partition 0, you do not have to do anything except rerun your Earl Grey command"
echo "If you are still having trouble, please open an issue on the Earl Grey github and we will try to help!"
touch ${library_path}/.earlgrey.config.complete
exit 2
fi
}
# Main #
while getopts g:s:o:t:r:l:m:d:e:h option
do
case "${option}"
in
g) genome=${OPTARG};;
s) species=${OPTARG};;
o) directory=${OPTARG};;
t) ProcNum=${OPTARG};;
f) Flank=${OPTARG};;
i) num=${OPTARG};;
r) RepSpec=${OPTARG};;
l) startCust=${OPTARG};;
c) cluster=${OPTARG};;
m) margin=${OPTARG};;
d) softMask=${OPTARG};;
n) no_seq=${OPTARG};;
a) min_seq=${OPTARG};;
e) heli=${OPTARG};;
h) usage; exit 0;;
esac
done
SECONDS=0
# Step 1 - set up the directories and make sure all modules are present
stage="Checking Parameters" && runningTea
SCRIPT_DIR=/data/toby/EarlGrey/scripts/
Checks
stage="Making Directories" && runningTea
makeDirectory
# Start Logs - only bother starting if everything is present and correct
exec > >(tee "${OUTDIR}/${species}EarlGrey.log") 2>&1
stage="Cleaning Genome" && runningTea
prepGenome
sleep 1
# Stage 2 - Assign libraries to variables
latestFile=$(realpath $startCust)
if [ ! -z "$RepSpec" ]; then
if [[ $(which RepeatMasker) == *"bin"* ]]; then
libpath="$(which RepeatMasker | sed 's|bin/RepeatMasker|share/RepeatMasker/Libraries/famdb/|')"
else
libpath="$(which RepeatMasker | sed 's|/[^/]*$||g')/Libraries/famdb/"
fi
if [[ $(which RepeatMasker) == *"bin"* ]]; then
PATH=$PATH:"$(which RepeatMasker | sed 's|bin/RepeatMasker|share/RepeatMasker/|g')"
fi
famdb.py -i $libpath families -f fasta_name --include-class-in-name -a -d --curated $RepSpec > ${OUTDIR}/${species}_Curated_Library/${RepSpec}.RepeatMasker.lib
RepSub=${OUTDIR}/${species}_Curated_Library/${RepSpec}.RepeatMasker.lib
fi
# Stage 5 - run the final RepeatMasker annotation using the refined TE consensus library
if [ ! -f ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/*.tbl ]; then
stage="Identifying Repeats Using Species-Specific Library" && runningTea
novoMask
if [ ! -f ${OUTDIR}/${species}_RepeatMasker_Against_Custom_Library/*.tbl ]; then
echo "ERROR: RepeatMasker failed, please check logs" && exit 2
fi
sleep 1
else
stage="Final masking already complete, skipping..." && runningTea
sleep 1
fi
# Stage 5.5 - Run HELIANO as an optional step
if [ "$heli" == "yes" ]; then
if [ ! -s ${OUTDIR}/${species}_heliano/*/RC.representative.gff ]; then
stage="Running HELIANO to Detect Helitrons" && runningTea
timestamp=$(date +"%Y%m%d_%H%M")
heliano_optional
else
stage="HELITRON detection already complete, skipping..." && runningTea
helitron_gff="$(realpath $(ls -td -- ${OUTDIR}/${species}_heliano/*/ | head -n 1) || true)/RC.representative.gff"
echo "HELITRON GFF: $helitron_gff"
fi
sleep 1
fi
# Stage 6
if [ ! -f ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed ] ; then
stage="Defragmenting Repeats" && runningTea
mergeRep
sleep 1
else
stage="Repeats already defragmented, skipping..." && runningTea
sleep 1
fi
# Stage 7
stage="Generating Summary Plots" && runningTea
charts
calcDivRL
sleep 1
# Stage 8
stage="Tidying Directories and Organising Important Files" && runningTea
sweepUp
sleep 1
# Stage 8.5 Optional Softmask
if [ "$softMask" == "yes" ]; then
stage="Generating Softmasked Genome" && runningTea
gunzip ${genome%.prep}.bak.gz && bedtools maskfasta -fi ${genome%.prep}.bak -bed ${OUTDIR}/${species}_mergedRepeats/looseMerge/${species}.filteredRepeats.bed -fo ${OUTDIR}/${species}_summaryFiles/${species}.softmasked.fasta -soft && gzip ${genome%.prep}.bak
sleep 1
fi
# Stage 9
time=$(convertsecs $SECONDS)
stage="Done in $time" && runningTea
sleep 5
# Stage 10
stage="TE library, Summary Figures, and TE Quantifications in Standard Formats Can Be Found in ${OUTDIR}/${species}_summaryFiles/" && runningTea
cat ${OUTDIR}/${species}_summaryFiles/${species}.highLevelCount.kable
sleep 5
================================================
FILE: earlGreyLibConstruct
================================================
#!/bin/bash
set -eo pipefail
usage()
{
echo " #############################
earlGrey version 7.2.4 (Library Construction Only)
Required Parameters:
-g == genome.fasta
-s == species name
-o == output directory
Optional Parameters:
-t == Number of Threads (DO NOT specify more than are available)
-r == RepeatMasker search term (e.g arthropoda/eukarya)
-l == Starting consensus library for an initial mask (in fasta format)
-i == Number of Iterations to BLAST, Extract, Extend (Default: 10)
-f == Number flanking basepairs to extract (Default: 1000)
-n == Max number of sequences used to generate consensus sequences (Default: 20)
-a == minimum number of sequences required to build a consensus sequence (Default: 3)
-h == Show help
Example Usage:
earlGrey -g bombyxMori.fasta -s bombyxMori -o /home/toby/bombyxMori/repeatAnnotation/ -t 16
Queries can be sent to:
tobias.baril[at]unine.ch
Please make use of the GitHub Issues and Discussion Tabs at: https://github.com/TobyBaril/EarlGrey
#############################"
}
# Subprocess Make Directories #
makeDirectory()
{
directory=$(realpath ${directory})
mkdir -p ${directory}/${species}_EarlGrey/ && OUTDIR=${directory}/${species}_EarlGrey
if [ ! -z "$RepSpec" ] || [ ! -z "$startCust" ] ; then
mkdir -p $OUTDIR/${species}_RepeatMasker/
fi
mkdir -p $OUTDIR/${species}_Database/
mkdir -p $OUTDIR/${species}_RepeatModeler/
mkdir -p $OUTDIR/${species}_strainer/
mkdir -p ${OUTDIR}/${species}_summaryFiles/
}
# Subprocess PrepGenome #
prepGenome()
{
if [ ! -L ${genome} ]; then
genome=$(realpath ${genome})
fi
if [ -L $genome ]; then
genome=$(realpath -s ${genome})
fi
if [ ! -f ${genome}.prep ] || [ ! -f ${genome}.dict ]; then
cp ${genome} ${genome}.bak && gzip -f ${genome}.bak
sed '/>/ s/[[:space:]].*//g; /^$/d' ${genome} > ${genome}.tmp
${SCRIPT_DIR}/headSwap.sh -i ${genome}.tmp -o ${genome}.prep && rm ${genome}.tmp
mv ${genome}.tmp.dict ${genome}.dict
sed -i.bak '/^>/! s/[DVHBPE]/N/g' ${genome}.prep
dict=${genome}.dict
genOrig=${genome}
genome=${genome}.prep
else
dict=${genome}.dict
genOrig=${genome}
genome=${genome}.prep
fi
}
# Subprocess getRepeatMaskerFasta
# Generate a copy of the RepeatMasker library subset used for the initial TE mask in FASTA format (only used if a RepeatMasker run is specified)
getRepeatMaskerFasta()
{
if [[ $RepSpec = *" "* ]]; then
echo "ERROR: You have entered a species name that contains a space, please use the NCBI TaxID rather than name. E.G In place of \"Homo sapiens\" use \"9606\""
exit 2
fi
if [[ $(which RepeatMasker) == *"bin"* ]]; then
libpath="$(which RepeatMasker | sed 's|bin/RepeatMasker|share/RepeatMasker/Libraries/famdb/|')"
else
libpath="$(which RepeatMasker | sed 's|/[^/]*$||g')/Libraries/famdb/"
fi
if [[ $(which RepeatMasker) == *"bin"* ]]; then
PATH=$PATH:"$(which RepeatMasker | sed 's|bin/RepeatMasker|share/RepeatMasker/|')"
fi
mkdir -p $OUTDIR/${species}_Curated_Library/
famdb.py -i $libpath families -f fasta_name --include-class-in-name -a -d --curated $RepSpec > ${OUTDIR}/${species}_Curated_Library/${RepSpec}.RepeatMasker.lib
RepSub=${OUTDIR}/${species}_Curated_Library/${RepSpec}.RepeatMasker.lib
}
# Subprocess firstMask
# Run the initial RepeatMasker run with the specified species subset (only used if a RepeatMasker run is specified)
firstMask()
{
cd ${OUTDIR}/${species}_RepeatMasker
rmthreads=$(( ProcNum / 4 ))
RepeatMasker -species $RepSpec -no_is -lcambig -s -a -pa $rmthreads -dir $OUTDIR/${species}_RepeatMasker $genome
if [ ! -f ${OUTDIR}/${species}_RepeatMasker/*.masked ]; then
echo "ERROR: RepeatMasker failed, please check logs. This is likely because of an invalid species search term, if issue persists please use NCBI Taxids (E.G Drosophila is replaced with 7125)"; exit 2
fi
}
# Subprocess firstMaskCustomLib
# Run the initial RepeatMasker run with the specific fasta consensus library
firstMaskCustomLib()
{
cd ${OUTDIR}/${species}_RepeatMasker
rmthreads=$(( ProcNum / 4 ))
RepeatMasker -lib ${startCust} -no_is -lcambig -s -a -pa $rmthreads -dir $OUTDIR/${species}_RepeatMasker $genome
RepSub=${startCust}
if [ ! -f ${OUTDIR}/${species}_RepeatMasker/*.masked ]; then
echo "ERROR: RepeatMasker failed, please check logs. This is likely because of an invalid custom consensus file, check the fasta file looks as expected"; exit 2
fi
}
# Subprocess buildDB
# if masked genome exists, build database from this. If not, build database from original input genome
buildDB()
{
if [ -f ${OUTDIR}/${species}_RepeatMasker/*.masked ]; then
cd ${OUTDIR}/${species}_Database
BuildDatabase -name ${species} ${OUTDIR}/${species}_RepeatMasker/*.masked
else
cd ${OUTDIR}/${species}_Database
BuildDatabase -name ${species} $genome
fi
}
# Subprocess deNovo1
# Run the initial de novo annotation run with RepeatModeler2.
# The genomeSampleSizeMax flag is set based on the samplable genome size (contigs >= 40 kb)
# to avoid RepeatModeler failures on small or fragmented genomes.
deNovo1()
{
cd ${OUTDIR}/${species}_RepeatModeler
# Compute samplable genome size: sum of contigs >= 40 kb only.
# RepeatModeler discards contigs shorter than 40 kb during sampling, so using
# the total genome size would overestimate the sequence available per round.
GENOME_SIZE=$(awk '/^>/{if(len>=40000)sum+=len; len=0; next}{len+=length($0)} END{if(len>=40000)sum+=len; print sum+0}' "${genome}")
echo "Samplable genome size (contigs >= 40 kb): ${GENOME_SIZE} bp"
# Set -genomeSampleSizeMax to the highest RECON round threshold the genome
# can support. Thresholds are cumulative across rounds (r2=3M, r3=9M, r4=27M,
# r5=81M, r6=243M), so the samplable size must cover all rounds up to the cap:
# >= 363M (3+9+27+81+243): no cap -> all 6 rounds
# >= 120M (3+9+27+81): cap 81M -> rounds 1-5
# >= 39M (3+9+27): cap 27M -> rounds 1-4
# >= 12M (3+9): cap 9M -> rounds 1-3
# < 12M: cap 3M -> rounds 1-2
SAMPLE_FLAG=""
if [ "${GENOME_SIZE}" -ge 363000000 ] 2>/dev/null; then
: # genome large enough for all rounds; use RepeatModeler default
elif [ "${GENOME_SIZE}" -ge 120000000 ] 2>/dev/null; then
echo "Capping at round 5 (-genomeSampleSizeMax 81000000)"
SAMPLE_FLAG="-genomeSampleSizeMax 81000000"
elif [ "${GENOME_SIZE}" -ge 39000000 ] 2>/dev/null; then
echo "Capping at round 4 (-genomeSampleSizeMax 27000000)"
SAMPLE_FLAG="-genomeSampleSizeMax 27000000"
elif [ "${GENOME_SIZE}" -ge 12000000 ] 2>/dev/null; then
echo "Capping at round 3 (-genomeSampleSizeMax 9000000)"
SAMPLE_FLAG="-genomeSampleSizeMax 9000000"
else
echo "Capping at round 2 (-genomeSampleSizeMax 3000000)"
SAMPLE_FLAG="-genomeSampleSizeMax 3000000"
fi
RepeatModeler -threads ${ProcNum} -database ${OUTDIR}/${species}_Database/${species} ${SAMPLE_FLAG}
if [ ! -e ${OUTDIR}/${species}_Database/${species}-families.fa ]; then
echo "ERROR: RepeatModeler Failed"
exit 2
fi
}
# Subprocess strainer # CHECK FILE STRUCTURE FOR EARL GREY RUN
# contains the BLAST, Extract, Extend, Trim pipeline from James Galbraith
strainer()
{
cd ${OUTDIR}/${species}_strainer/
if [ "$ProcNum" -ge 4 ]; then
strainthreads=$(( ProcNum / 4 ))
else
strainthreads=$ProcNum
fi
${SCRIPT_DIR}/TEstrainer/TEstrainer_for_earlGrey.sh -g $genome -l ${OUTDIR}/${species}_Database/${species}-families.fa -t ${strainthreads} -f $Flank -r $num -n $no_seq -m $min_seq
latestFile="$(realpath $(ls -td -- ${OUTDIR}/${species}_strainer/*/ | head -n 1) || true)/${species}-families.fa.strained"
cp $latestFile ${OUTDIR}/${species}_strainer/
}
# Subprocess strainerResume
# resumes interrupted TEstrainer runs
strainerResume() {
cd ${OUTDIR}/${species}_strainer/
strainDataDir=$(find . -maxdepth 1 -type d -name "TS_${species}-families.fa_*")
if [ ! $(echo "$strainDataDir" | wc -l) -eq 1 ]; then
echo "ERROR: There are more TEstrainer directories than expected present"
exit 2
fi
if [ "$ProcNum" -ge 4 ]; then
strainthreads=$(( ProcNum / 4 ))
else
strainthreads=$ProcNum
fi
${SCRIPT_DIR}/TEstrainer/TEstrainer_for_earlGrey.sh -g $genome -l ${OUTDIR}/${species}_Database/${species}-families.fa -t ${strainthreads} -f $Flank -r $num -n $no_seq -m $min_seq -d ${strainDataDir} -R
latestFile="$(realpath $(ls -td -- ${OUTDIR}/${species}_strainer/*/ | head -n 1) || true)/${species}-families.fa.strained"
cp $latestFile ${OUTDIR}/${species}_strainer/
}
# Subprocess sweepUp
# Puts required files into a summary folder
sweepUp()
{
cd ${OUTDIR}/${species}_summaryFiles/
if [ -s ${latestFile} ]; then
cp $latestFile ${OUTDIR}/${species}_summaryFiles/
else
echo "ERROR: TEstrainer failed to generate a TE library, please check logs" && exit 1
fi
}
# Subprocess runningTea
runningTea()
{
echo "
) (
( ) )
) ( (
_______)_
.-'---------|
( C|/\/\/\/\/|
'-./\/\/\/\/|
'_________'
'-------'
<<< $stage >>>"
}
# Subprocess GetTime
convertsecs()
{
h=$(bc <<< "${1}/3600")
m=$(bc <<< "(${1}%3600)/60")
s=$(bc <<< "${1}%60")
printf "%02d:%02d:%05.2f\n" $h $m $s
}
# Subprocess Checks
Checks()
{
if [ -z "$genome" ] || [ -z "$species" ] || [ -z "$directory" ] ; then
usage; exit 1
fi
if [ -z "$ProcNum" ] ; then
ProcNum=1; echo "$ProcNum Cores Will Be Used"
else
echo "$ProcNum Cores Will Be Used"
fi
if [ -z "$RepSpec" ] && [ -z "$startCust" ] ; then
echo "RepeatMasker species not specified, running Earl Grey without an initial mask with known repeats"
else
echo "Running Earl Grey with an intial mask with known repeats"
fi
if [ -z "$num" ] ; then
num=10; echo "De Novo Sequences Will Be Extended Through a Maximum of $num Iterations"
else
echo "De Novo Sequences Will Be Extended Through a Maximum of $num Iterations"
fi
if [ -z "$no_seq" ] ; then
no_seq=20; echo "$no_seq sequences will be used in BEAT consensus generation"
else
echo "$no_seq sequences will be used in BEAT consensus generation"
fi
if [ -z "$Flank" ]; then
Flank=1000; echo "Blast, Extend, Align, Trim Process Will Add 1000bp to Each End in Each Iteration"
else
echo "Blast, Extract, Extend, Trim Process Will Add $Flank to Each End in Each Iteration"
fi
if [ -z "$min_seq" ]; then
min_seq=3; echo "Blast, Extend, Align, Trim Process Will Require 3 Sequences to Generate a New Consensus Sequence"
else
echo "Blast, Extend, Align, Trim Process Will Require $min_seq Sequences to Generate a New Consensus Sequence"
fi
if [ ! -d "$SCRIPT_DIR" ]; then
echo "ERROR: Script directory variable not set, please run the configure script in the Earl Grey directory before attempting to run Earl Grey"; exit 1
fi
if [ ! -d "$SCRIPT_DIR/TEstrainer/" ]; then
echo "ERROR: teStrainer module not found, please check all modules are present and run the configure script in the Earl Grey directory before attempting to run Earl Grey"; exit 1
fi
echo "Please cite the following paper when using this software:"
echo "Baril, T., Galbraith, J. and Hayward, A., 2024. Earl Grey: a fully automated user-friendly transposable element annotation and analysis pipeline. Molecular Biology and Evolution, 41(4), p.msae068."
}
# Subprocess Dfam Checks
dfamCheck()
{
# Dfam 3.9 checks
library_path=$(which RepeatMasker | sed 's|/bin/RepeatMasker|/share/RepeatMasker/Libraries/famdb|g')
expected_file="${library_path}/dfam39_full.0.h5"
file_count=$(find "$library_path" -maxdepth 1 -type f | wc -l)
if [ $file_count -eq 2 ] && [ -f $expected_file ] && [ -f ${library_path}/rmlib.config ] ; then
echo "WARNING: Earl Grey v6.1.0 has updated to Dfam v3.9."
echo "Before using Earl Grey, you MUST download the required partitions from Dfam (https://dfam.org/releases/current/families/FamDB/)"
echo "These must be located at ${library_path}/. Then, you must reconfigure RepeatMasker before using Earl Grey."
echo "I have written the required commands for you to faciliate the process."
echo "Please follow these carefully to ensure successful configuration of RepeatMasker with Dfam 3.9."
echo "As a note, these DB paritions are BIG, so only download the ones you need for your studies (or, if you do need all of them, make sure you have the space for them)"
echo "########################################################"
echo "A script for modification and automation of these steps has been generated in $(pwd)/configure_dfam39.sh"
echo "Please open the script in your preferred text editor (e.g. nano or vim) and change the partition numbers in [] to those that you would like to download"
echo "then, make the script executable (chmod +x $(pwd)/configure_dfam39.sh) and run it: ./configure_dfam39.sh"
echo "Alternatively, you can run the steps below one-by-one to achieve the same outcome:"
echo "########################################################"
echo ""
if [ -f "$(pwd)/configure_dfam39.sh" ]; then
rm $(pwd)/configure_dfam39.sh
fi
echo "# first, change directory to the famdb library location" | tee -a $(pwd)/configure_dfam39.sh
echo "cd ${library_path}/" | tee -a $(pwd)/configure_dfam39.sh
echo "" | tee -a $(pwd)/configure_dfam39.sh
echo "# download the partitions you require from Dfam 3.9. In the below, change the numbers or range inside the square brackets to choose your subsets." | tee -a $(pwd)/configure_dfam39.sh
echo "# e.g. to download partitions 0 to 10: [0-10]; or to download partitions 3,5, and 7: [3,5,7]; [0-16] is ALL PARTITIONS" | tee -a $(pwd)/configure_dfam39.sh
echo "curl -o 'dfam39_full.#1.h5.gz' 'https://dfam.org/releases/current/families/FamDB/dfam39_full.[0-16].h5.gz'" | tee -a $(pwd)/configure_dfam39.sh
echo "" | tee -a $(pwd)/configure_dfam39.sh
echo "# decompress Dfam 3.9 paritions" | tee -a $(pwd)/configure_dfam39.sh
echo "gunzip *.gz" | tee -a $(pwd)/configure_dfam39.sh
echo "" | tee -a $(pwd)/configure_dfam39.sh
echo "# move up to RepeatMasker main directory" | tee -a $(pwd)/configure_dfam39.sh
echo "cd $(which RepeatMasker | sed 's|/bin/RepeatMasker|/share/RepeatMasker/|g')" | tee -a $(pwd)/configure_dfam39.sh
echo "" | tee -a $(pwd)/configure_dfam39.sh
#echo "# Replace famdb with version 2.0.1 (required for the Dfam 3.9)" | tee -a $(pwd)/configure_dfam39.sh
#echo "wget https://github.com/Dfam-consortium/FamDB/archive/refs/tags/2.0.1.tar.gz && tar --wildcards -zxvf 2.0.1.tar.gz FamDB-2.0.1/famdb.py FamDB-2.0.1/famdb_*.py" | tee -a $(pwd)/configure_dfam39.sh
#echo "mv FamDB-2.0.1/*.py . && rm -rf 2.0.1.tar.gz FamDB-2.0.1/" | tee -a $(pwd)/configure_dfam39.sh
#echo "" | tee -a $(pwd)/configure_dfam39.sh
echo "# save the min_init partition as a backup, just in case!" | tee -a $(pwd)/configure_dfam39.sh
echo "mv ${library_path}/min_init.0.h5 ${library_path}/min_init.0.h5.bak" | tee -a $(pwd)/configure_dfam39.sh
echo "" | tee -a $(pwd)/configure_dfam39.sh
echo "# Rerun RepeatMasker configuration" | tee -a $(pwd)/configure_dfam39.sh
echo "perl ./configure -libdir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/share/RepeatMasker/Libraries/|g') \
-trf_prgm $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin/trf|g') \
-rmblast_dir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin|g') \
-hmmer_dir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin|g') \
-abblast_dir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin|g') \
-crossmatch_dir $(which RepeatMasker | sed 's|/bin/RepeatMasker|/bin|g') \
-default_search_engine rmblast" | tee -a $(pwd)/configure_dfam39.sh
echo "" | tee -a $(pwd)/configure_dfam39.sh
sed -i '/^perl/ s/\s\s*/ /g' $(pwd)/configure_dfam39.sh
echo "########################################################"
echo ""
echo "Following these steps should enable you to use Dfam 3.9 with RepeatMasker and Earl Grey"
echo "You will not see this message again if configuration is successful"
echo "If you only want partition 0, you do not have to do anything except rerun your Earl Grey command"
echo "If you are still having trouble, please open an issue on the Earl Grey github and we will try to help!"
touch ${library_path}/.earlgrey.config.complete
exit 2
fi
}
# Main #
while getopts g:s:o:t:f:i:r:l:n:a:h option
do
case "${option}"
in
g) genome=${OPTARG};;
s) species=${OPTARG};;
o) directory=${OPTARG};;
t) ProcNum=${OPTARG};;
f) Flank=${OPTARG};;
i) num=${OPTARG};;
r) RepSpec=${OPTARG};;
l) startCust=${OPTARG};;
n) no_seq=${OPTARG};;
a) min_seq=${OPTARG};;
h) usage; exit 0;;
esac
done
SECONDS=0
# Step 1 - set up the directories and make sure all modules are present
stage="Checking Parameters" && runningTea
SCRIPT_DIR=/data/toby/mambaforge/envs/earlgrey_440/share/earlgrey-4.4.0-0/scripts/
dfamCheck
Checks
stage="Making Directories" && runningTea
makeDirectory
# Start Logs - only bother starting if everything is present and correct
exec > >(tee "${OUTDIR}/${species}EarlGrey.log") 2>&1
stage="Cleaning Genome" && runningTea
prepGenome
sleep 1
# Step 2 - initial annotation, either de novo, or if required an initial RepeatMasker followed by a de novo run
if [ -z "$RepSpec" ]; then
sleep 1
else
if [ ! -f ${OUTDIR}/${species}_RepeatMasker/*.masked ]; then
stage="Getting RepeatMasker Sequences for $RepSpec and Saving as Fasta" && runningTea
getRepeatMaskerFasta
stage="Running Initial Mask with Known Repeats" && runningTea
firstMask
else
stage="Genome has already been masked, skipping..." && runningTea
getRepeatMaskerFasta
fi
fi
if [ -z "$startCust" ]; then
sleep 1
else
if [ ! -f ${OUTDIR}/${species}_RepeatMasker/*.masked ]; then
stage="Running Initial Mask with Known Repeats in Custom Library" && runningTea
firstMaskCustomLib
else
stage="Genome has already been masked, skipping..." && runningTea
fi
fi
# Step 3 - make a database from the genome, and then run a de novo TE annotation using RepeatModeler2
if [ ! -f ${OUTDIR}/${species}_Database/${species}-families.fa ]; then
stage="Detecting Novel Repeats" && runningTea
buildDB
deNovo1
sleep 1
else
stage="De novo repeats have already been detected, skipping..." && runningTea
sleep 1
fi
# Step 4 - run TEstrainer, which runs a BLAST, extract, extent, trim, on the the de novo repeat library to refine consensus sequences
if [ ! -s ${OUTDIR}/${species}_strainer/${species}-families.fa.strained ]; then
stage="Straining TEs and Refining de novo Consensus Sequences" && runningTea
if [ -f ${OUTDIR}/${species}_strainer/${species}-families.fa.strained ]; then
rm -r ${OUTDIR}/${species}_strainer/${species}-families.fa.strained ${OUTDIR}/${species}_strainer/TS*
strainer
elif [ -d ${OUTDIR}/${species}_strainer/TS_${species}-families.fa_* ]; then
stage="Resuming TEstrainer" && runningTea
strainerResume
else
strainer
fi
if [ ! -s ${OUTDIR}/${species}_strainer/${species}-families.fa.strained ]; then
echo "WARNING: TEstrainer failed to produce a strain file, please check the log file for more information. If you have run an intial mask with known repeats, this could be due RepeatModeler2 failing to identify any new repeats. Please check if this is expected."
fi
sleep 1
else
stage="TEs already strained, skipping..." && runningTea
latestFile=${OUTDIR}/${species}_strainer/${species}-families.fa.strained
cp ${latestFile} ${OUTDIR}/${species}_summaryFiles/
sleep 1
fi
# Stage 5 - Tidy up!
stage="Tidying Directories and Organising Important Files" && runningTea
sweepUp
sleep 1
# Stage 6
time=$(convertsecs $SECONDS)
stage="Done in $time" && runningTea
sleep 5
# Stage 10
stage="TE library in Standard Format Can Be Found in ${OUTDIR}/${species}_summaryFiles/" && runningTea
sleep 5
================================================
FILE: legacyDocs.md
================================================
# This file contains installation instructions for legacy versions of Earl Grey
## We recommend installing via conda or docker for the latest stable releases of Earl Grey, the below are retained as a reference
#==============================================================================================================================================================================#
## Earl Grey Installation and Configuration (If you already have RepeatMasker and RepeatModeler)
If you do not currently have RepeatMasker and RepeatModeler installed, the instructions are provided further down this page. If you do have them installed, **ensure the executables are in your PATH environment, including the RepeatMasker/util/ directory!**
All of the scripts and associated modules are contained within this github repository. Earl Grey runs inside an anaconda environment to ensure all required packages are present and are the correct version. Therefore to run Earl Grey, you will require anaconda to be installed on your system.
**If anaconda is NOT installed on your system, please install it following these instructions:**
```
# Change to /tmp directory as we won't need the script after running it
cd /tmp
# Download the anaconda installation script
curl https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh --output anaconda.sh
# Run the script to install anaconda
bash anaconda.sh
# answer yes when asked, and install anaconda3 in the specified location (recommended) unless you want it to be installed elsewhere.
# When asked "Do you wish the installer to initialize Anaconda3 by running conda init?", answer "yes" for ease of use.
# Activate conda by refreshing terminal session
source ~/.bashrc
# If successful, you should now see (base) on the left of your username on the command line
```
**Now that Anaconda is installed, we can install Earl Grey**
Clone the Earl Grey github repo
```
# Clone into a home directory, or somewhere you want to install Earl Grey
git clone https://github.com/TobyBaril/EarlGrey
```
Enter the Earl Grey directory and configure the program
```
cd ./EarlGrey
chmod +x ./configure
./configure
```
Once this is complete, remember to activate the earlGrey conda environment before attempting to run the Earl Grey pipeline
```
conda activate earlGrey
earlGrey -g genome.fasta -s speciesName -o outputDirectory -t threads
```
For suggestions or questions, please use the discussion and issues functions in github.
Thank you for trying Earl Grey!
#==============================================================================================================================================================================#
## Earl Grey Installation and Configuration (If you DO NOT have RepeatMasker and RepeatModeler) - WITH SUDO PRIVILEGES
These instructions will guide you through configuring all required programs and scripts to run Earl Grey.
All of the scripts and associated modules are contained within this github repository. Earl Grey runs inside an anaconda environment to ensure all required packages are present and are the correct version. Therefore to run Earl Grey, you will require anaconda to be installed on your system.
**If anaconda is NOT installed on your system, please install it following these instructions:**
```
# Change to /tmp directory as we won't need the script after running it
cd /tmp
# Download the anaconda installation script
curl https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh --output anaconda.sh
# Run the script to install anaconda
bash anaconda.sh
# answer yes when asked, and install anaconda3 in the specified location (recommended) unless you want it to be installed elsewhere.
# When asked "Do you wish the installer to initialize Anaconda3 by running conda init?", answer "yes" for ease of use.
# Activate conda by refreshing terminal session
source ~/.bashrc
# If successful, you should now see (base) on the left of your username on the command line
```
**Now that Anaconda is installed, we can install Earl Grey**
### To install RepeatMasker
RepeatMasker can be downloaded from: http://www.repeatmasker.org/RepeatMasker/. Installation instructions can be found on the website. Alternatively, please use the code below:
You will need to download and install a couple of programs for RepeatMasker to work.
Download rmblast
```wget https://www.repeatmasker.org/rmblast/rmblast-2.13.0+-x64-linux.tar.gz```
Extract the rmblast package
```tar -zxvf rmblast-2.13.0+-x64-linux.tar.gz```
Make a note of the full path to rmblast-2.13.0/bin/ as you will need this in the RepeatMasker configuration
If you are not certain of the full path, please run the following command
```realpath ./rmblast-2.13.0/bin/```
Download RepeatMasker (this will download it to the current directory). NOTE some extra steps are required for Dfam 3.8, please refer to repeatmasker.org/RepeatMasker.
```wget https://www.repeatmasker.org/RepeatMasker/RepeatMasker-4.1.6.tar.gz```
Copy the RepeatMasker package to /usr/local/, or somewhere that all users will be able to access the installation.
Copying to /usr/local/ might require sudo privileges
```sudo cp RepeatMasker-4.1.6.tar.gz /usr/local/```
Change directory to /usr/local, and extract the RepeatMasker package. This might require sudo privileges.
```cd /usr/local/```
```sudo tar -zxvf RepeatMasker-4.1.6.tar.gz```
Install the required RepeatMasker libraries - Earl Grey has been tested with Dfam 3.4 onwards and RepBase.
Unfortunately, RepBase is now behind a paywall, but to ensure Earl Grey remains open it does not rely on RepBase, although inclusion of RepBase can improve classification of repeats by RepeatModeler. If you have access to this database, please include it in your configuration of RepeatMasker.
We recommend that you download Dfam 3.8 before using Earl Grey, and this is required for RepeatMasker 4.1.6. The Dfam library is large - this could take a while!
We recommend downloading Dfam into your home directory (~/) or a subdirectory of home
Change directory to home
```cd ~/```
Download lastest Dfam release - This may take a while
*** NOTE: There are many partitions for Dfam, each containing curated and uncurated TE consensi split by taxonomy. As a minimum, you need to configure RepeatMasker with the root partition, but you can also include others. Note: There may be erroneous annotations in the uncurated Dfam database, use at your own risk***
For whole Dfam library run all of the below, or run each part as required (minimum the root (0) partition):
```
wget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.0.h5.gz
wget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.1.h5.gz
wget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.2.h5.gz
wget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.3.h5.gz
wget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.4.h5.gz
wget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.5.h5.gz
wget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.6.h5.gz
wget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.7.h5.gz
wget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.8.h5.gz
```
Unzip the Dfam release - This may take a while with no indication that anything is happening, please be patient!
For whole Dfam library (including uncurated elements):
```gunzip *.h5.gz```
Move the Dfam library to the RepeatMasker library folder
NOTE, a warning might come up that this will overwrite the existing file, allow this by pressing "y" then Enter
```mv *.h5 /usr/local/RepeatMasker/Libraries/famdb/```
DO NOT configure RepeatMasker just yet...
Add RepeatMasker and util directory to your path environment
```echo 'export PATH=$PATH:/usr/local/RepeatMasker:/usr/local/RepeatMasker/util/' >> ~/.bashrc```
### To install RepeatModeler
RepeatModeler can be downloaded from: http://www.repeatmasker.org/RepeatModeler/. Installation instructions can be found on the website. Alternatively, please use the code below:
You will need to download and install a couple of programs for RepeatModeler to work.
Download RECON and RepeatScout
```
wget http://www.repeatmasker.org/RepeatModeler/RECON-1.08.tar.gz
wget http://www.repeatmasker.org/RepeatScout-1.0.6.tar.gz
```
Extract the RECON and RepeatScout packages
```
tar -zxvf RECON-1.08.tar.gz
tar -zxvf RepeatScout-1.0.6.tar.gz
```
Enter the RECON directory and make from source
```
cd ./RECON-1.08/src/
make
make install
```
Enter the RepeatScout directory and make from source
```
cd ../../RepeatScout-1.0.6/
make
```
Make a note of the full path to RECON-1.08/bin/ and RepeatScout-1.0.6/ as you will need these in the RepeatModeler configuration
If you are not certain of the full paths, please run the following commands
```
realpath ./RECON-1.08/bin/
realpath ./RepeatScout-1.0.6/
```
Install UCSC TwoBit Tools. Make a note of the full path to the directory where these have been installed as you will need these in the RepeatModeler configuration
```
mkdir ~/ucscTwoBitTools/
cd ~/ucscTwoBitTools/
wget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/faToTwoBit
wget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/twoBitToFa
wget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/twoBitInfo
wget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/twoBitMask
wget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/twoBitDup
# make scripts executable
chmod +x ~/ucscTwoBitTools/*
```
For Earl Grey, the LTR structural pipeline of RepeatModeler is not required, as this is run as part of Earl Grey's defragmentation process.
Download RepeatModeler
```wget http://www.repeatmasker.org/RepeatModeler/RepeatModeler-2.0.4.tar.gz```
Copy the RepeatModeler package to /usr/local/, or somewhere that all users will be able to access the installation.
Copying to /usr/local/ might require sudo privileges
```sudo cp RepeatModeler-2.0.4.tar.gz /usr/local/```
Change directory to /usr/local, and extract the RepeatModeler package. This might require sudo privileges.
```cd /usr/local/```
```sudo tar -zxvf RepeatModeler-2.0.4.tar.gz```
DO NOT configure RepeatModeler just yet...
Add RepeatModeler directory to your path environment
```echo 'export PATH=$PATH:/usr/local/RepeatModeler-2.0.4/' >> ~/.bashrc```
### To Install Earl Grey and Configure all Programs
Clone the Earl Grey github repo
```
# Clone into a home directory, or somewhere you want to install Earl Grey
git clone https://github.com/TobyBaril/EarlGrey
```
Enter the Earl Grey directory and configure the program
```
cd ./EarlGrey
chmod +x ./configure
./configure
```
Activate the conda environment
```
conda activate earlGrey
```
For configuring RepeatMasker with sudo, h5py is required in the root path. The easiest way to do this is to run the following two lines:
```
sudo apt install python3-pip
sudo pip install h5py
```
The linux64 version of TRF is included in the modules directory of Earl Grey. Make a note of the full path to this file as this will be needed in the RepeatMasker configuration.
```realpath ./modules/trf409*```
NOTE: For the latest version of RepeatModeler, this must be called trf. We can make a symlinked version: ```ln -s ./modules/trf409* ./modules/trf```
Now to finally configure RepeatMasker! - Run these lines and then follow the on-screen prompts from RepeatMasker. Again, this might require sudo privileges.
```
cd /usr/local/RepeatMasker
# check which perl interpreter you should use
which perl
# replace [perl] in the below command with the path printed from the command above and use this as your perl interpreter (this should be the anaconda perl interpreter)
sudo [perl] ./configure
```
Time to configure RepeatModeler! You need to enter the paths to lots of the directories of programs we have installed, please note them down before running the configuration script!!
```
cd /usr/local/RepeatModeler-2.0.4/
# check which perl interpreter you should use - COPY THIS
which perl
# replace perl in the below command with the path printed from the command above and use this as your perl interpreter
sudo perl ./configure
```
Once this is complete, remember to activate the earlGrey conda environment before attempting to run the Earl Grey pipeline
```
conda activate earlGrey
earlGrey -g genome.fasta -s speciesName -o outputDirectory -t threads
```
#==============================================================================================================================================================================#
## Earl Grey Installation and Configuration (If you DO NOT have RepeatMasker and RepeatModeler) - WITHOUT SUDO PRIVILEGES
These instructions will guide you through configuring all required programs and scripts to run Earl Grey.
All of the scripts and associated modules are contained within this github repository. Earl Grey runs inside an anaconda environment to ensure all required packages are present and are the correct version. Therefore to run Earl Grey, you will require anaconda to be installed on your system.
**If anaconda is NOT installed on your system, please install it following these instructions:**
```
# Change to /tmp directory as we won't need the script after running it
cd /tmp
# Download the anaconda installation script
curl https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh --output anaconda.sh
# Run the script to install anaconda
bash anaconda.sh
# answer yes when asked, and install anaconda3 in the specified location (recommended) unless you want it to be installed elsewhere.
# When asked "Do you wish the installer to initialize Anaconda3 by running conda init?", answer "yes" for ease of use.
# Activate conda by refreshing terminal session
source ~/.bashrc
# If successful, you should now see (base) on the left of your username on the command line
```
**Now that Anaconda is installed, we can install Earl Grey**
### To install RepeatMasker
RepeatMasker can be downloaded from: http://www.repeatmasker.org/RepeatMasker/. Installation instructions can be found on the website. Alternatively, please use the code below:
You will need to download and install a couple of programs for RepeatMasker to work.
Download rmblast
```wget https://www.repeatmasker.org/rmblast/rmblast-2.13.0+-x64-linux.tar.gz```
Extract the rmblast package
```tar -zxvf rmblast-2.13.0+-x64-linux.tar.gz```
Make a note of the full path to rmblast-2.13.0/bin/ as you will need this in the RepeatMasker configuration
If you are not certain of the full path, please run the following command
```realpath ./rmblast-2.13.0/bin/```
Download RepeatMasker (this will download it to the current directory). NOTE some extra steps are required if you wish to use Dfam 3.7, please refer to repeatmasker.org/RepeatMasker for these extra steps if required.
```wget https://www.repeatmasker.org/RepeatMasker/RepeatMasker-4.1.6.tar.gz```
Extract the RepeatMasker package.
```
tar -zxvf RepeatMasker-4.1.6.tar.gz
```
Install the required RepeatMasker libraries - Earl Grey has been tested with Dfam 3.3 and RepBase.
Unfortunately, RepBase is now behind a paywall, but to ensure Earl Grey remains open it does not rely on RepBase, although inclusion of RepBase can improve classification of repeats by RepeatModeler. If you have access to this database, please include it in your configuration of RepeatMasker.
We recommend that you download Dfam 3.8 before using Earl Grey, and this is required for RepeatMasker 4.1.6. The Dfam library is large - this could take a while!
Change directory to RepeatMasker Libraries
```cd ./RepeatMasker/Libraries/famdb/```
Download lastest Dfam release - This may take a while
*** NOTE: There are many partitions for Dfam, each containing curated and uncurated TE consensi split by taxonomy. As a minimum, you need to configure RepeatMasker with the root partition, but you can also include others. Note: There may be erroneous annotations in the uncurated Dfam database, use at your own risk***
For whole Dfam library run all of the below, or run each part as required (minimum the root (0) partition):
```
wget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.0.h5.gz
wget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.1.h5.gz
wget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.2.h5.gz
wget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.3.h5.gz
wget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.4.h5.gz
wget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.5.h5.gz
wget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.6.h5.gz
wget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.7.h5.gz
wget https://dfam.org/releases/Dfam_3.8/families/FamDB/dfam38_full.8.h5.gz
```
Unzip the Dfam release - This may take a while with no indication that anything is happening, please be patient!
For whole Dfam library (including uncurated elements):
```gunzip *.h5.gz```
DO NOT configure RepeatMasker just yet...
Add RepeatMasker and util directory to your path environment (Replace /path/to/ with the full path to your installation directory)
```echo 'export PATH=$PATH:/path/to/RepeatMasker/:/path/to/RepeatMasker/util/' >> ~/.bashrc```
### To install RepeatModeler
RepeatModeler can be downloaded from: http://www.repeatmasker.org/RepeatModeler/. Installation instructions can be found on the website. Alternatively, please use the code below:
You will need to download and install a couple of programs for RepeatModeler to work.
Download RECON and RepeatScout (Make sure you are no longer inside your RepeatMasker directory!)
```
wget http://www.repeatmasker.org/RepeatModeler/RECON-1.08.tar.gz
wget http://www.repeatmasker.org/RepeatScout-1.0.6.tar.gz
```
Extract the RECON and RepeatScout packages
```
tar -zxvf RECON-1.08.tar.gz
tar -zxvf RepeatScout-1.0.6.tar.gz
```
Enter the RECON directory and make from source
```
cd ./RECON-1.08/src/
make
make install
```
Enter the RepeatScout directory and make from source
```
cd ../../RepeatScout-1.0.6/
make
```
Make a note of the full path to RECON-1.08/bin/ and RepeatScout-1.0.6/ as you will need these in the RepeatModeler configuration
If you are not certain of the full paths, please run the following commands
```
realpath ./RECON-1.08/bin/
realpath ./RepeatScout-1.0.6/
```
Install UCSC TwoBit Tools. Make a note of the full path to the directory where these have been installed as you will need these in the RepeatModeler configuration
```
mkdir ./ucscTwoBitTools/
cd ./ucscTwoBitTools/
wget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/faToTwoBit
wget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/twoBitToFa
wget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/twoBitInfo
wget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/twoBitMask
wget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/twoBitDup
# make scripts executable
chmod +x ./ucscTwoBitTools/*
```
For Earl Grey, the LTR structural pipeline of RepeatModeler is not required, as this is run as part of Earl Grey's defragmentation process.
Download RepeatModeler
```wget http://www.repeatmasker.org/RepeatModeler/RepeatModeler-2.0.4.tar.gz```
Unpack RepeatModeler
```sudo tar -zxvf RepeatModeler-2.0.4.tar.gz```
DO NOT configure RepeatModeler just yet...
Add RepeatModeler directory to your path environment (Replace /path/to/ with the full path to your installation directory)
```echo 'export PATH=$PATH:/path/to/RepeatModeler-2.0.4/' >> ~/.bashrc```
### To Install Earl Grey and Configure all Programs
Clone the Earl Grey github repo
```
# Clone into a home directory, or somewhere you want to install Earl Grey
git clone https://github.com/TobyBaril/EarlGrey
```
Enter the Earl Grey directory and configure the program
```
cd ./EarlGrey
chmod +x ./configure
./configure
```
Activate the conda environment
```
conda activate earlGrey
```
The linux64 version of TRF is included in the modules directory of Earl Grey. Make a note of the full path to this file as this will be needed in the RepeatMasker configuration.
```realpath ./modules/trf409*```
NOTE: For the latest version of RepeatModeler, this must be called trf. We can make a symlinked version: ```ln -s ./modules/trf409* ./modules/trf```
Now to finally configure RepeatMasker! - Run these lines and then follow the on-screen prompts from RepeatMasker. Again, this might require sudo privileges.
```
cd /path/to/RepeatMasker
perl ./configure
```
Time to configure RepeatModeler! You need to enter the paths to lots of the directories of programs we have installed, please note them down before running the configuration script!!
```
cd /path/to/RepeatModeler-2.0.4/
perl ./configure
```
Once this is complete, remember to activate the earlGrey conda environment before attempting to run the Earl Grey pipeline
```
conda activate earlGrey
earlGrey -g genome.fasta -s speciesName -o outputDirectory -t threads
```
# Alternative installation methods
Before using Earl Grey, please ensure RepeatMasker (>=version 4.1.4) and RepeatModeler (>=version 2.0.4) are installed and configured. If these are not, please follow the instructions below to install these before continuing with Earl Grey Installation.
NOTE: These instructions are provided to install RepeatMasker, RepeatModeler and related programs with sudo privileges. If you are working on a shared cluster, please request installation of RepeatMasker and RepeatModeler by your sysadmin before working with Earl Grey. Earl Grey will function with RepeatMasker and RepeatModeler installed in the local path environment, or when the modules are loaded on a HPC cluster.
#==============================================================================================================================================================================#
## Earl Grey is available in a [Docker container](./Docker) preconfigured with Dfam version 3.9 or empty, but with the option to install required partitions
To use this container, please make sure Docker is installed and configured on your system. All instructions relating to the docker installation are found within the [Docker directory](./Docker) in this repository. Please consult the README in the Docker directory for installation instructions.
#==============================================================================================================================================================================#
## Earl Grey is available for [Singularity](./Singularity) preconfigured with Dfam version 3.9 or empty, but with the option to install required partitions
To use this container, please make sure Singularity is installed and configured on your system. All instructions relating to the singularity installation are found within the [Singularity directory](./Singularity) in this repository. Please consult the README in the Singularity directory for installation instructions.
================================================
FILE: scripts/LTR_FINDER_parallel
================================================
#!/usr/bin/perl -w
use strict;
use threads;
use Thread::Queue;
use File::Basename;
use FindBin;
# customized parameters
my $seq_path = ""; #specify the sequence file
my $finder_para = "-w 2 -C -D 15000 -d 1000 -L 7000 -l 100 -p 20 -M 0.85"; #specify ltr_finder parameters. The "-w 2" parameter is required.
my $harvest_format = 0; #0, print out format is in LTR_finder table output format (default); 1, LTRharvest screen output format
my $size_of_each_piece = 5000000; #5 Mb per piece
my $timeout = 1500; #set maximum time for a thread to run. After $timeout seconds, the child thread is killed.
my $try1 = 1; #1, further split to 50 Kb regions if thread killed by timeout. 0, no further split.
my $next = 0; #0, run LTR_FINDER. 1, only summarize results if the .finder folder is retained.
my $verbose = 0; #0, remove the .finder folder after finished; 1, retain the folder for later use.
my $threads = 4; #specify thread number to run ltr_finder
my $annotator = "LTR_FINDER_parallel"; #the annotator name used in gff output
# dependencies
#my $script_path = dirname(__FILE__);
my $script_path = $FindBin::Bin;
my $cut = "$script_path/bin/cut.pl"; #the program to cut sequence
my $ltr_finder = "$script_path/bin/LTR_FINDER.x86_64-1.0.7/ltr_finder"; #the path to ltr_finder
my $usage = "
~ ~ ~ Run LTR_FINDER in parallel ~ ~ ~
Author: Shujun Ou (shujun.ou.1\@gmail.com)
Date: 09/19/2018
Update: 09/27/2019
Usage: perl LTR_FINDER_parallel -seq [file] -size [int] -threads [int]
Options: -seq [file] Specify the sequence file.
-size [int] Specify the size you want to split the genome sequence.
Please make it large enough to avoid spliting too many LTR elements. Default 5000000 (bp)
-time [int] Specify the maximum time to run a subregion (a thread).
This helps to skip simple repeat regions that take a substantial of time to run. Default: 1500 (seconds).
Suggestion: 300 for -size 1000000. Increase -time when -size increased.
-try1 [0|1] If a region requires more time than the specified -time (timeout), decide:
0, discard the entire region.
1, further split to 50 Kb regions to salvage LTR candidates (default);
-harvest_out Output LTRharvest format if specified. Default: output LTR_FINDER table format.
-next Only summarize the results for previous jobs without rerunning LTR_FINDER (for -v).
-verbose|-v Retain LTR_FINDER outputs for each sequence piece.
-finder [file] The path to the program LTR_FINDER (default v1.0.7, included in this package).
-threads|-t [int] Indicate how many CPU/threads you want to run LTR_FINDER.
-help|-h Display this help information.
\n";
# read user parameters
my $i=0;
foreach (@ARGV){
$seq_path = $ARGV[$i+1] if /^-seq$/i;
$size_of_each_piece = $ARGV[$i+1] if /^-size$/i;
$timeout = $ARGV[$i+1] if /^-time$/i;
$try1 = $ARGV[$i+1] if /^-try1$/i;
$harvest_format = 1 if /^-harvest_out$/i;
$next = 1 if /^-next$/i;
$verbose = 1 if /^-verbose$|^-v$/i;
$cut = $ARGV[$i+1] if /^-cut$/i;
$ltr_finder = $ARGV[$i+1] if /^-finder$/i;
$threads = $ARGV[$i+1] if /^-threads$|^-t$/i;
die $usage if /^-help$|^-h$/i;
$i++;
}
# make a softlink to the genome file
die $usage unless -s $seq_path;
my $seq_file = basename($seq_path);
`ln -s $seq_path $seq_file` unless -s $seq_file;
# read genome in memory
open SEQ, "<$seq_file" or die $usage;
open GFF, ">$seq_file.finder.combine.gff3" or die $!;
print GFF "##gff-version 3\n";
my %seq; #a hash to store seq name and length info
my %order; #store seq order in genome
my $chr_info; #store seq id in genome
$i=0;
$/ = "\n>";
while (){
s/>//g;
my ($id, $seq) = (split /\n/, $_, 2);
$seq =~ s/\s+//g;
my $len = length $seq;
print GFF "##sequence-region $id 1 $len\n";
$chr_info.="#$id\n";
$seq{$id} = $len;
$order{$id} = $i;
$i++;
}
print GFF "$chr_info";
close SEQ;
$/="\n";
goto Next if $next == 1; #run the next step if all LTR_finder processes are done
`mkdir $seq_file.finder` unless -d "$seq_file.finder";
chdir "$seq_file.finder";
`perl $cut ../$seq_file -s -l $size_of_each_piece`;
##multi-threading using queue, create a worker module for parallel computation
my $process_q=Thread::Queue->new();
sub worker {
while (my $seq = $process_q -> dequeue()){
chomp $seq;
$seq =~ s/\s+//;
print localtime() ." CPU".threads -> self() -> tid().": running on $seq\n";
`timeout $timeout $ltr_finder $finder_para $seq > $seq.finder.scn`; #set the max running time for a thread as $timeout seconds
if ($? ne 0 and $try1 ne 0){
print localtime() ." CPU".threads -> self() -> tid().": $seq timeout, process it with the salvage mode\n";
my $in=`perl $script_path/LTR_FINDER_parallel -seq $seq -size 50000 -time 10 -try1 0 -threads 1 -cut $cut -finder $ltr_finder`;
`mv $seq.finder.combine.scn $seq.finder.scn`;
`rm -rf ${seq}.finder`; #remove?
}
}
}
#insert seq names into the worker queue
open List, "<../$seq_file.list" or die $!;
$process_q -> enqueue ();
$process_q -> end(); #stop adding items to the queue
close List;
#work and finish
for (1..$threads){
threads -> create ( \&worker );
}
foreach my $thr (threads -> list()){
$thr -> join();
}
chdir "../";
Next:
#combine split ltr_finder results
open Out, ">$seq_file.finder.combine.scn" or die $!;
#print out headers
if ($harvest_format == 0){
print Out "index SeqID Location LTR len Inserted element len TSR PBS PPT RT IN (core) IN (c-term) RH Strand Score Sharpness Similarity\n";
} else {
print Out "#LTR_FINDER_parallel -seq $seq_file -size $size_of_each_piece -time $timeout -try1 $try1 -harvest_out -threads $threads -cut $cut -finder $ltr_finder
# LTR_FINDER args=$finder_para
# predictions are reported in the following way
# s(ret) e(ret) l(ret) s(lLTR) e(lLTR) l(lLTR) s(rLTR) e(rLTR) l(rLTR) sim(LTRs) seq-nr chr
# where:
# s = starting position
# e = ending position
# l = length
# ret = LTR-retrotransposon
# lLTR = left LTR
# rLTR = right LTR
# sim = similarity
# seq-nr = sequence order\n";
}
open List, "<$seq_file.list" or die $!;
my $count = 1; #count repeats in gff
foreach my $seq (){
$seq =~ s/\s+//;
my ($base, $order) = ($1, $2) if $seq =~ /(.*)_sub([0-9]+)$/;
my $coord_adj = ($order - 1) * $size_of_each_piece;
next unless -e "$seq_file.finder/$seq.finder.scn";
open Scn, "<$seq_file.finder/$seq.finder.scn" or die $!;
while (){
next unless /^\[/;
s/^\[\s+[0-9]+\]/\[NA\]/;
my ($index, $id, $loc, $ltr_len, $ele_len, $TSR, $PBS, $PPT, $RT, $IN_core, $IN_cterm, $RH, $strand, $score, $sharpness, $sim) = (split);
#convert coordinates back to the genome scale
my @coord = ($loc, $PBS, $PPT, $RT, $IN_core, $IN_cterm, $RH);
my $i = -1;
foreach (@coord) {
$i++;
next if /^N-N$/;
my ($start, $end) = ($1+$coord_adj, $2+$coord_adj) if /([0-9]+)\-([0-9]+)/;
$coord[$i] = "$start-$end";
}
my ($start, $end) = ($1, $2) if $coord[0] =~ /([0-9]+)\-([0-9]+)/;
my ($lltr, $rltr) = ($1, $2) if $ltr_len=~/([0-9]+),([0-9]+)/;
my ($lltr_e, $rltr_s) = ($start+$lltr-1, $end-$rltr+1);
#output LTRharvest or LTR_FINDER (-w 2) format
if ($harvest_format == 0){
print Out "[NA]\t$base\t$coord[0]\t$ltr_len\t$ele_len\t$TSR\t$coord[1]\t$coord[2]\t$coord[3]\t$coord[4]\t$coord[5]\t$coord[6]\t$strand\t$score\t$sharpness\t$sim\n";
} else {
$sim*=100;
print Out "$start $end $ele_len $start $lltr_e $lltr $rltr_s $end $rltr $sim $order{$base} $base\n";
}
#print GFF format
my $chr = $base;
print GFF "$chr\t$annotator\trepeat_region\t$start\t$end\t.\t$strand\t.\tID=repeat_region$count\n";
#print GFF "$chr\t$annotator\ttarget_site_duplication\t$lTSD\t.\t$strand\t.\tParent=repeat_region$count\n" unless $TSD eq "NA";
print GFF "$chr\t$annotator\tLTR_retrotransposon\t$start\t$end\t.\t$strand\t.\tID=LTR_retrotransposon$count;Parent=repeat_region$count;tsd=$TSR;ltr_identity=$sim;seq_number=$order{$chr}\n";
print GFF "$chr\t$annotator\tlong_terminal_repeat\t$start\t$lltr_e\t.\t$strand\t.\tParent=LTR_retrotransposon$count\n";
print GFF "$chr\t$annotator\tlong_terminal_repeat\t$rltr_s\t$end\t.\t$strand\t.\tParent=LTR_retrotransposon$count\n";
#print GFF "$chr\t$annotator\ttarget_site_duplication\t$rTSD\t.\t$strand\t.\tParent=repeat_region$count\n" unless $TSD eq "NA";
print GFF "###\n";
}
$count++;
}
close List;
close Scn;
close GFF;
`rm -rf ./$seq_file.finder` if $verbose eq 0;
print localtime() ." Job finished! Check out $seq_file.finder.combine.scn\n";
================================================
FILE: scripts/TEstrainer/README.md
================================================
# TEstrainer
A pipeline to dramatically improve repeat libraries before genome annotation through curation of each repeat, better identification of satellite repeats, removal of multi-copy genes and reclassification of cleaned libraries.
TEstrainer can be used on both de novo libraries (generated by tools such as RepeatModeler, REPET, EDTA, etc) or on libraries from databases (e.g. Repbase, Dfam).
The iterative curation of repeats is for use on de novo libraries, and will lengthen repeat consensus sequences as close as possible to the full length. This lengthening is accomplished by repeating the following procedure (developed from ):
Example command
```
./TEstrainer -l seq/Camellia sinensis-families.fa -g seq/ASM1731120v1.fasta -t 64 -r 12 -C
```
1) Using BLAST to identify copies of the repeats in the source genome
2) Extending their coordinates, extracting the extended coordinates
3) Local aligning the each extracted sequence to each other to identify regions of the flanks which do not align
4) Trimming off these flanks
5) Creating a multiple sequence alignment (MSA) of the trimmed sequences
6) Creating a consensus sequence from the MSA, ensuring coverage of at least three nucleotides along the length of the sequences
If a repeat's new consensus is more than 25% longer than its previously consensus sequence and aligns to the starting consensus sequence it this cycle is repeated.
## Required packages and databases
BLAST+ - Altschul, S.F. et al. (1997). https://doi.org/10.1089/10665270050081478.
MAFFT - Katoh, K and Standley, D.M. (2013). https://doi.org/10.1093/molbev/mst010
mreps - Kolpakov, R et al. (2003). https://doi.org/10.1093/nar/gkg617
sa-ssr - Pickett, B.D. (2016). https://doi.org/10.1093/bioinformatics/btw298
GNU Parallel - Tange, O. (2011) The USENIX Magazine, February 2011:42-47.
cd-hit (Optional, used for clustering) - Li, W (2006) https://doi.org/10.1093/bioinformatics/btl158
RepeatModeler (Optional, used for reclassifying) - Flynn, J.M. (2020). https://doi.org/10.1073/pnas.1921046117
### Python3 packages
pyranges - Stovner, E.B and Sætrom, P. (2020). https://doi.org/10.1093/bioinformatics/btz615
pyfaidx - Shirley MD, et al. (2015). https://doi.org/10.7287/peerj.preprints.970v1
biopython - Cock, P.A. et al. (2009). https://doi.org/10.1093/bioinformatics/btp163
pandas - McKinney, W. (2010). https://doi.org/10.25080/Majora-92bf1922-00a
numpy - Harris, C.R. et al. (2020). https://doi.org/10.1038/s41586-020-2649-2.
### R packages
optparse
tidyverse - Wickham, H. et al. (2019). https://doi.org/doi:10.21105/joss.01686.
BSgenome - Pagès H (2022). https://bioconductor.org/packages/BSgenome.
plyranges - Lee, S. et al. (2019). https://doi.org/10.1186/s13059-018-1597-8.
### Database
NCBI CDD - Lu, S. et al (2020). https://doi.org/10.1093/nar/gkz991
================================================
FILE: scripts/TEstrainer/TEstrainer
================================================
#!/bin/bash
usage() { echo "Usage: [-l Repeat library] [-g Genome ] [-t Threads (default 4) ] [-f Flank (default 1000) ] [-r Iterations of BEET to run (default 0) ] [-d Out directory, if not specified wil be created ] [-c Set if clustering ] [-C Set if reclassifying ] [-s Set if sorting (only for RepeatModeler output) ] [-D Set if printing all messages ] [-h Print this help] [-M Ammount of memory TEstrainer needs to keep free" 1>&2; exit 1; }
# default values
FLANK=1000
THREADS=4
RUNS=0
CLASSIFY=FALSE
SORT=FALSE
DEBUG=FALSE
# DFAM=FALSE
# for potential folder name
TIME=$(date +"%s")
TIME=${TIME: -4}
MEM_FREE="1G"
# parsing
while getopts l:g:t:f:r:d:M:CsDh flag; do
case "${flag}" in
l) RM_LIBRARY_PATH=${OPTARG};;
g) GENOME=${OPTARG};;
t) THREADS=${OPTARG};;
f) FLANK=${OPTARG};;
r) RUNS=${OPTARG};;
d) DATA_DIR=${OPTARG};;
M) MEM_FREE=${OPTARG};;
C) CLASSIFY=TRUE;;
s) SORT=TRUE ;;
D) DEBUG=TRUE ;;
h | *)
print_usage
exit_script
esac
done
echo "Running using"$GENOME" and "$RM_LIBRARY_PATH
# determine further variables and check files exist
if [ ! -f ${RM_LIBRARY_PATH} ]; then echo "Library not found"; usage; fi
if [ -z ${RM_LIBRARY_PATH} ]; then echo "Library must be supplied"; usage; else RM_LIBRARY=$(echo $RM_LIBRARY_PATH | sed 's/.*\///'); fi
if [[ $RUNS -gt 0 ]]; then
if [ -z ${GENOME} ]; then echo "If refining genome must be supplied"; usage; fi
if [ ! -f ${GENOME} ]; then echo "Refining genome not found"; usage; fi
fi
if [ -z "$DATA_DIR" ]; then DATA_DIR=$(echo "TS_"${RM_LIBRARY}"_"${TIME}); fi
if [ "$DEBUG" == TRUE ]; then echo "Files all exist" ;fi
# create data dir if missing
if [ ! -d ${DATA_DIR} ]
then
mkdir ${DATA_DIR}
fi
if [[ $THREADS -gt 4 ]]; then MAFFT_THREADS=$(($(($THREADS / 4)))); else MAFFT_THREADS=1; fi
# Cap threads based on available RAM to prevent OOM errors
AVAIL_MEM_MB=$(free -m | awk '/^Mem:/{print $7}')
MEM_PER_JOB_MB=800
MAX_SAFE_THREADS=$(( AVAIL_MEM_MB / MEM_PER_JOB_MB ))
if [[ $MAX_SAFE_THREADS -gt 0 && $THREADS -gt $MAX_SAFE_THREADS ]]; then
echo "Warning: capping threads from ${THREADS} to ${MAX_SAFE_THREADS} based on available RAM (${AVAIL_MEM_MB} MB free)"
THREADS=$MAX_SAFE_THREADS
if [[ $THREADS -gt 4 ]]; then MAFFT_THREADS=$(($(($THREADS / 4)))); else MAFFT_THREADS=1; fi
fi
# make directories
mkdir -p ${DATA_DIR}/run_0/
# # dfam extract
# if [ "$DFAM" == TRUE ]; then python scripts/Dfam_extractor.py -l ${RM_LIBRARY_PATH} -d ${DATA_DIR} ; else cp ${RM_LIBRARY_PATH} ${DATA_DIR}/${RM_LIBRARY};fi
# initial copy
if [ "$DEBUG" == TRUE ]; then echo "Copying library to data dir" ;fi
cp ${RM_LIBRARY_PATH} ${DATA_DIR}/run_0/further_${RM_LIBRARY}
# setup for BEET runs
if [[ $RUNS -gt 0 ]]; then
# index genome and make blast db
if [ "$DEBUG" == TRUE ]; then echo "Indexing genome" ;fi
python scripts/indexer.py -g ${GENOME}
if [ ! -f "${GENOME}".nsq ]; then
makeblastdb -in ${GENOME} -dbtype nucl -out ${GENOME} # makeblastb if needed
fi
# create og reference
python scripts/splitter.py -i ${DATA_DIR}/run_0/further_${RM_LIBRARY} -o ${DATA_DIR}/run_0/og
RUN_NO=1
# BEET runs
while [ $RUN_NO -le $RUNS ]
do
if [ "$DEBUG" == TRUE ]; then echo "Making "${RUN_NO}" directories"; fi
# make directories
mkdir -p ${DATA_DIR}/run_${RUN_NO}/raw \
${DATA_DIR}/run_${RUN_NO}/TEtrim_complete \
${DATA_DIR}/run_${RUN_NO}/initial_blast \
${DATA_DIR}/run_${RUN_NO}/self_search \
${DATA_DIR}/run_${RUN_NO}/to_align \
${DATA_DIR}/run_${RUN_NO}/mafft \
${DATA_DIR}/run_${RUN_NO}/TEtrim_con \
${DATA_DIR}/run_${RUN_NO}/TEtrim_unaln \
${DATA_DIR}/run_${RUN_NO}/TEtrim_blast \
${DATA_DIR}/run_${RUN_NO}/TEtrim_mafft \
${DATA_DIR}/run_${RUN_NO}/TEtrim_further \
${DATA_DIR}/run_${RUN_NO}/TEtrim_bp
# split
cp ${DATA_DIR}/run_$(expr $RUN_NO - 1)/further_${RM_LIBRARY} ${DATA_DIR}/run_${RUN_NO}/${RM_LIBRARY}
echo "Splitting run "${RUN_NO}
python scripts/splitter.py -i ${DATA_DIR}/run_${RUN_NO}/${RM_LIBRARY} -o ${DATA_DIR}/run_${RUN_NO}/raw
# run trf to determine if sequence is tandem repeat
echo "Initial trf check for "${RUN_NO}
parallel --bar --jobs ${THREADS} --memfree ${MEM_FREE} -a ${DATA_DIR}/run_${RUN_NO}/raw/${RM_LIBRARY}_split.txt trf ${DATA_DIR}/run_${RUN_NO}/raw/{} 2 7 7 80 10 50 500 -d -h -ngs ">" ${DATA_DIR}/run_${RUN_NO}/raw/{}.trf
echo "Initial blast and preparation for MSA "${RUN_NO}
# initial blast and extention
parallel --bar --jobs ${THREADS} --memfree ${MEM_FREE} -a ${DATA_DIR}/run_${RUN_NO}/raw/${RM_LIBRARY}_split.txt python3 scripts/initial_mafft_setup.py -d ${DATA_DIR} -r ${RUN_NO} -s {} -g ${GENOME} -f ${FLANK} -D
## first mafft alignment
find ${DATA_DIR}/run_${RUN_NO}/to_align -type f | sed 's/.*\///' > ${DATA_DIR}/run_${RUN_NO}/to_align.txt
echo "Primary alignment run "${RUN_NO}
parallel --bar --jobs $MAFFT_THREADS --memfree ${MEM_FREE} -a ${DATA_DIR}/run_${RUN_NO}/to_align.txt mafft --thread 4 --quiet --localpair --adjustdirectionaccurately ${DATA_DIR}/run_${RUN_NO}/to_align/{} ">" ${DATA_DIR}/run_${RUN_NO}/mafft/{}
# trim
echo "Trimming run "${RUN_NO}
parallel --bar --jobs $MAFFT_THREADS --memfree ${MEM_FREE} -a ${DATA_DIR}/run_${RUN_NO}/to_align.txt python scripts/TEtrim.py -i ${DATA_DIR}/run_${RUN_NO}/mafft/{} -t 4 -f ${FLANK} -n ${RUN_NO} -d ${DATA_DIR}
# compile completed curations
if [ -n "$(ls -A ${DATA_DIR}/run_${RUN_NO}/TEtrim_complete/ 2>/dev/null)" ]; then
cat ${DATA_DIR}/run_${RUN_NO}/TEtrim_complete/*fasta > ${DATA_DIR}/run_${RUN_NO}/complete_${RM_LIBRARY}
fi
# Either compile consensuses for further curation
if [ -n "$(ls -A ${DATA_DIR}/run_${RUN_NO}/TEtrim_further/ 2>/dev/null)" ]; then
cat ${DATA_DIR}/run_${RUN_NO}/TEtrim_further/*fasta > ${DATA_DIR}/run_${RUN_NO}/further_${RM_LIBRARY}
echo "Ready for " $(( RUN_NO++ ))
else
# or exit loop if no more curation needed
echo "Finished extension"
break
fi
done
# Compile all completed
cat ${DATA_DIR}/run_*/complete_${RM_LIBRARY} > ${DATA_DIR}/${RM_LIBRARY}
# if more could have been extended append to compilation
if [ -s ${DATA_DIR}/run_${RUNS}/further_${RM_LIBRARY} ]; then
cat ${DATA_DIR}/run_${RUNS}/further_${RM_LIBRARY} >> ${DATA_DIR}/${RM_LIBRARY}
fi
# add any families which went missing along the way due to alignment issues
find ${DATA_DIR}/run_*/mafft/ -empty | sed 's/mafft/raw/' > ${DATA_DIR}/missing_consensi.txt
if [[ -s ${DATA_DIR}/missing_consensi.txt ]]; then
while read file_name; do
cat $file_name >> ${DATA_DIR}/${RM_LIBRARY}
done < ${DATA_DIR}/missing_consensi.txt
fi
else
echo "No curation"
cp ${RM_LIBRARY_PATH} ${DATA_DIR}/${RM_LIBRARY}
fi
# Identify simple repeats and satellites, trim ends of LINEs/SINEs
echo "Splitting for simple/satellite packages"
mkdir -p ${DATA_DIR}/trf/split
python scripts/splitter.py -i ${DATA_DIR}/${RM_LIBRARY} -o ${DATA_DIR}/trf/split
cp ${DATA_DIR}/${RM_LIBRARY} ${DATA_DIR}/trf/
# Run and parse TRF
echo "Running TRF"
parallel --bar --jobs ${THREADS} --memfree ${MEM_FREE} -a ${DATA_DIR}/trf/split/${RM_LIBRARY}_split.txt trf ${DATA_DIR}/trf/split/{} 2 7 7 80 10 50 500 -d -h -ngs ">" ${DATA_DIR}/trf/split/{}.trf
parallel --bar --jobs ${THREADS} --memfree ${MEM_FREE} -a ${DATA_DIR}/trf/split/${RM_LIBRARY}_split.txt python3 scripts/trf_parser.py --trf ${DATA_DIR}/trf/split/{}.trf --out ${DATA_DIR}/trf/split/{}.trf.tsv
find ./${DATA_DIR}/trf/split/ -type f -name "*trf.tsv" -exec cat {} + | cat > ${DATA_DIR}/trf/${RM_LIBRARY}.trf
# Run SA-SSR
echo "Running SA-SSR"
sa-ssr -e -l 20 -L 50000 -m 1 -M 5000 -t ${THREADS} ${DATA_DIR}/${RM_LIBRARY} ${DATA_DIR}/trf/${RM_LIBRARY}.sassr
# Run and compile mreps
echo "Running mreps"
parallel --bar --jobs ${THREADS} --memfree ${MEM_FREE} -a ${DATA_DIR}/trf/split/${RM_LIBRARY}_split.txt bash scripts/mreps_parser.sh -i ${DATA_DIR}/trf/split/{} "2>"/dev/null
find ./${DATA_DIR}/trf/split/ -type f -name "*mreps" -exec cat {} + | cat > ${DATA_DIR}/trf/${RM_LIBRARY}.mreps
# Interpret mreps, TRF and SA-SSR
echo "Trimming and sorting based on mreps, TRF, SA-SSR"
Rscript scripts/simple_repeat_filter_trim.R -i ${DATA_DIR}/${RM_LIBRARY} -d ${DATA_DIR}
# Identify and trim chimeric repeats from non-satellite repeats, remove proteins
mkdir -p ${DATA_DIR}/chimeras/split/
cp ${DATA_DIR}/trf/${RM_LIBRARY}.nonsatellite ${DATA_DIR}/chimeras/${RM_LIBRARY}
echo "Split ready for rpstblastn"
python scripts/splitter.py -i ${DATA_DIR}/chimeras/${RM_LIBRARY} -o ${DATA_DIR}/chimeras/split/
echo "Run rpstblastn"
parallel --bar --jobs $THREADS --memfree ${MEM_FREE} -a ${DATA_DIR}/chimeras/split/${RM_LIBRARY}_split.txt rpstblastn -query ${DATA_DIR}/chimeras/split/{} -db ~/Databases/cdd/Cdd -out ${DATA_DIR}/chimeras/split/{}.out -outfmt \"6 qseqid qstart qend qlen sseqid sstart send slen pident length mismatch gapopen evalue bitscore qcovs stitle\" -evalue 0.01 -num_threads 1
find ./${DATA_DIR}/chimeras/split/ -type f -name "*.out" -exec cat {} + | cat > ${DATA_DIR}/chimeras/${RM_LIBRARY}.rps.out
echo "Running strainer.R"
Rscript scripts/strainer.R --in_seq ${DATA_DIR}/chimeras/${RM_LIBRARY} --directory ${DATA_DIR}
cat ${DATA_DIR}/chimeras/clean_${RM_LIBRARY} ${DATA_DIR}/chimeras/chimeric_${RM_LIBRARY} > ${DATA_DIR}/chimeras/rinsed_${RM_LIBRARY}.nonsatellite
# Delete temp files
if [ "$DEBUG" == FALSE ]; then
echo "Removing temporary files"
rm -r ${DATA_DIR}/*/split/
find ${DATA_DIR}/ -mindepth 1 -name "run_*" -exec rm -r {} +
fi
if [[ $RUNS -gt 0 ]]; then
rm ${GENOME}.n*
fi
# Classify ?
if [ "$CLASSIFY" == TRUE ]; then
# Classify improved consensi using RepeatModeler's RepeatClassifier
echo "Reclassifying repeats"
mkdir -p ${DATA_DIR}/classify/
cp ${DATA_DIR}/chimeras/rinsed_${RM_LIBRARY}.nonsatellite ${DATA_DIR}/classify/
cd ${DATA_DIR}/classify/
RepeatClassifier -pa ${THREADS} -consensi rinsed_${RM_LIBRARY}.nonsatellite
cd -
cp ${DATA_DIR}/classify/rinsed_${RM_LIBRARY}.nonsatellite.classified ${DATA_DIR}/${RM_LIBRARY}
echo "Compiling library"
cat ${DATA_DIR}/trf/${RM_LIBRARY}.satellites >> ${DATA_DIR}/${RM_LIBRARY}
else
echo "Compiling library"
cat ${DATA_DIR}/chimeras/rinsed_${RM_LIBRARY}.nonsatellite ${DATA_DIR}/trf/${RM_LIBRARY}.satellites >> ${DATA_DIR}/${RM_LIBRARY}
fi
# Sort? (for RepeatModeler output)
if [ "$SORT" == TRUE ]; then
echo "Sorting library"
Rscript scripts/final_sorter.R -i ${DATA_DIR}/${RM_LIBRARY}
fi
================================================
FILE: scripts/TEstrainer/TEstrainer.Rproj
================================================
Version: 1.0
RestoreWorkspace: Default
SaveWorkspace: Default
AlwaysSaveHistory: Default
EnableCodeIndexing: Yes
UseSpacesForTab: Yes
NumSpacesForTab: 2
Encoding: UTF-8
RnwWeave: Sweave
LaTeX: pdfLaTeX
================================================
FILE: scripts/TEstrainer/TEstrainer_for_earlGrey.sh
================================================
#!/bin/bash
usage() { echo "Usage: [-l Repeat library] [-g Genome ] [-t Threads (default 4) ] [-f Flank (default 1000) ] [-m Minimum number of sequences required in multiple sequence alignments during BEAT] [-r Number of iterations of BEAT to run (deafult 10)] [-d Out directory, if not specified wil be created ] [-h Print this help] [-M Amount of memory TEstrainer needs to keep free] [-R Resume interrupted run in directory specified by -d] [-q Suppress parallel progress bar (useful for batch/sbatch jobs)]" 1>&2; exit 1; }
# default values
STRAIN_SCRIPTS=/data/toby/EarlGrey/scripts/TEstrainer/scripts/
FLANK=1000
THREADS=4
RUNS=10
NO_SEQ=20
# for potential folder name
TIME=$(date +"%s")
TIME=${TIME: -4}
MEM_FREE="1G"
MIN_SEQ=3
BAR_FLAG="--bar"
# parsing
while getopts l:g:t:f:m:r:d:h:n:M:Rq flag; do
case "${flag}" in
l) RM_LIBRARY_PATH=${OPTARG};;
g) GENOME=${OPTARG};;
t) THREADS=${OPTARG};;
f) FLANK=${OPTARG};;
m) MIN_SEQ=${OPTARG};;
r) RUNS=${OPTARG};;
d) DATA_DIR=${OPTARG};;
n) NO_SEQ=${OPTARG};;
M) MEM_FREE=${OPTARG};;
R) RESUME=true;;
q) BAR_FLAG="";;
h | *)
print_usage
exit_script
esac
done
# determine further variables and check files exist
if [ ! -f ${RM_LIBRARY_PATH} ]; then echo "Library not found"; usage; fi
if [ -z ${RM_LIBRARY_PATH} ]; then echo "Library must be supplied"; usage; else RM_LIBRARY=$(echo $RM_LIBRARY_PATH | sed 's/.*\///'); fi
if [[ $RUNS -gt 0 ]]; then
if [ -z ${GENOME} ]; then echo "If refining genome must be supplied"; usage; fi
if [ ! -f ${GENOME} ]; then echo "Refining genome not found"; usage; fi
fi
if [[ $THREADS -gt 4 ]]; then MAFFT_THREADS=$(($(($THREADS / 4)))); else MAFFT_THREADS=1; fi
# Cap threads based on available RAM to prevent OOM errors
AVAIL_MEM_MB=$(free -m | awk '/^Mem:/{print $7}')
MEM_PER_JOB_MB=800
MAX_SAFE_THREADS=$(( AVAIL_MEM_MB / MEM_PER_JOB_MB ))
if [[ $MAX_SAFE_THREADS -gt 0 && $THREADS -gt $MAX_SAFE_THREADS ]]; then
echo "Warning: capping threads from ${THREADS} to ${MAX_SAFE_THREADS} based on available RAM (${AVAIL_MEM_MB} MB free)"
THREADS=$MAX_SAFE_THREADS
if [[ $THREADS -gt 4 ]]; then MAFFT_THREADS=$(($(($THREADS / 4)))); else MAFFT_THREADS=1; fi
fi
if [ -n "$RESUME" ]
then
if [ -z "$DATA_DIR" ]; then echo "When resuming, out directory (-d) must be specified"; usage; fi
RUN_NO=$(ls ${DATA_DIR} | grep -P '^run_\d+$' | sed 's/run_//' | sort -n | tail -n 1)
if [ -z "$RUN_NO" ] || [ $RUN_NO -eq 0 ]
then
echo "Attempting to resume a run that was interrupted before iterations started"
unset RESUME
if [ -d ${DATA_DIR}/run_0/ ]; then rm -r ${DATA_DIR}/run_0/ ; fi
if [ -d ${DATA_DIR}/tmp/ ]; then rm -r ${DATA_DIR}/tmp/ ; fi
elif [ $RUN_NO -gt $RUNS ]
then
echo "Trying to resume, but more runs than expected have already been completed"
usage
elif [ $RUN_NO -eq $RUNS ] && [ -f ${DATA_DIR}/${RM_LIBRARY} ]
then
RUN_NO=$RUN_NO+1
else
rm -r ${DATA_DIR}/run_${RUN_NO}
fi
fi
if [ -z "$RESUME" ]
then
if [ -z "$DATA_DIR" ]; then DATA_DIR=$(echo "TS_"${RM_LIBRARY}"_"${TIME}); fi
# create data dir if missing
if [ ! -d ${DATA_DIR} ]
then
mkdir ${DATA_DIR}
fi
# make directories
mkdir -p ${DATA_DIR}/run_0/
mkdir -p ${DATA_DIR}/tmp/
# initial copy
cp ${RM_LIBRARY_PATH} ${DATA_DIR}/${RM_LIBRARY}
# cp starting seq to starting directory
mkdir -p ${DATA_DIR}/run_0/og
cp ${RM_LIBRARY_PATH} ${DATA_DIR}/run_0/further_${RM_LIBRARY}
# create reference of original sequences
python ${STRAIN_SCRIPTS}/splitter.py -i ${DATA_DIR}/run_0/further_${RM_LIBRARY} -o ${DATA_DIR}/run_0/og
# runs
python ${STRAIN_SCRIPTS}/indexer.py -g ${GENOME}
if [ ! -f "${GENOME}".nsq ]; then
makeblastdb -in ${GENOME} -dbtype nucl -out ${GENOME} # makeblastb if needed
fi
# curation
RUN_NO=1
fi
while [ $RUN_NO -le $RUNS ]
do
# make directories
mkdir -p ${DATA_DIR}/run_${RUN_NO}/raw \
${DATA_DIR}/run_${RUN_NO}/TEtrim_complete \
${DATA_DIR}/run_${RUN_NO}/initial_blast \
${DATA_DIR}/run_${RUN_NO}/self_search \
${DATA_DIR}/run_${RUN_NO}/to_align \
${DATA_DIR}/run_${RUN_NO}/mafft \
${DATA_DIR}/run_${RUN_NO}/TEtrim_con \
${DATA_DIR}/run_${RUN_NO}/TEtrim_unaln \
${DATA_DIR}/run_${RUN_NO}/TEtrim_blast \
${DATA_DIR}/run_${RUN_NO}/TEtrim_mafft \
${DATA_DIR}/run_${RUN_NO}/TEtrim_further \
${DATA_DIR}/run_${RUN_NO}/TEtrim_bp
# split
cp ${DATA_DIR}/run_$(expr $RUN_NO - 1)/further_${RM_LIBRARY} ${DATA_DIR}/run_${RUN_NO}/${RM_LIBRARY}
echo "Splitting run "${RUN_NO}
python ${STRAIN_SCRIPTS}/splitter.py -i ${DATA_DIR}/run_${RUN_NO}/${RM_LIBRARY} -o ${DATA_DIR}/run_${RUN_NO}/raw
# run trf to determine if sequence is tandem repeat
echo "Initial trf check for "${RUN_NO}
parallel ${BAR_FLAG} --tmpdir ${DATA_DIR}/tmp/ --jobs ${THREADS} --memfree ${MEM_FREE} -a ${DATA_DIR}/run_${RUN_NO}/raw/${RM_LIBRARY}_split.txt trf ${DATA_DIR}/run_${RUN_NO}/raw/{} 2 7 7 80 10 50 500 -d -h -ngs ">" ${DATA_DIR}/run_${RUN_NO}/raw/{}.trf
echo "Initial blast and preparation for MSA "${RUN_NO}
# initial blast and extention
parallel ${BAR_FLAG} --tmpdir ${DATA_DIR}/tmp/ --jobs ${THREADS} --memfree ${MEM_FREE} -a ${DATA_DIR}/run_${RUN_NO}/raw/${RM_LIBRARY}_split.txt python3 ${STRAIN_SCRIPTS}/initial_mafft_setup.py -d ${DATA_DIR} -r ${RUN_NO} -s {} -g ${GENOME} -f ${FLANK} -D -n ${NO_SEQ}
## first mafft alignment
find ${DATA_DIR}/run_${RUN_NO}/to_align -type f | sed 's/.*\///' > ${DATA_DIR}/run_${RUN_NO}/to_align.txt
echo "Primary alignment run "${RUN_NO}
parallel ${BAR_FLAG} --tmpdir ${DATA_DIR}/tmp/ --jobs $MAFFT_THREADS --memfree ${MEM_FREE} -a ${DATA_DIR}/run_${RUN_NO}/to_align.txt mafft --thread 4 --quiet --localpair --adjustdirectionaccurately ${DATA_DIR}/run_${RUN_NO}/to_align/{} ">" ${DATA_DIR}/run_${RUN_NO}/mafft/{}
# trim
echo "Trimming run "${RUN_NO}
parallel ${BAR_FLAG} --tmpdir ${DATA_DIR}/tmp/ --jobs $MAFFT_THREADS --memfree ${MEM_FREE} -a ${DATA_DIR}/run_${RUN_NO}/to_align.txt python ${STRAIN_SCRIPTS}/TEtrim.py -i ${DATA_DIR}/run_${RUN_NO}/mafft/{} -t 4 -f ${FLANK} -n ${RUN_NO} -d ${DATA_DIR} -m ${MIN_SEQ}
# compile completed curations
if [ -n "$(ls -A ${DATA_DIR}/run_${RUN_NO}/TEtrim_complete/ 2>/dev/null)" ]; then
cat ${DATA_DIR}/run_${RUN_NO}/TEtrim_complete/*fasta > ${DATA_DIR}/run_${RUN_NO}/complete_${RM_LIBRARY}
fi
# Either compile consensuses for further curation
if [ -n "$(ls -A ${DATA_DIR}/run_${RUN_NO}/TEtrim_further/ 2>/dev/null)" ]; then
cat ${DATA_DIR}/run_${RUN_NO}/TEtrim_further/*fasta > ${DATA_DIR}/run_${RUN_NO}/further_${RM_LIBRARY}
echo "Ready for " $(( RUN_NO++ ))
else
# or exit loop if no more curation needed
echo "Finished extension"
break
fi
done
# Compile all completed
cat ${DATA_DIR}/run_*/complete_${RM_LIBRARY} > ${DATA_DIR}/${RM_LIBRARY}
# if more could have been extended append to compilation
if [ -s ${DATA_DIR}/run_${RUNS}/further_${RM_LIBRARY} ]; then
cat ${DATA_DIR}/run_${RUNS}/further_${RM_LIBRARY} >> ${DATA_DIR}/${RM_LIBRARY}
fi
# add any families which went missing along the way due to alignment issues
find ${DATA_DIR}/run_*/mafft/ -empty -type f | sed 's/mafft/raw/' > ${DATA_DIR}/missing_consensi.txt
if [[ -s ${DATA_DIR}/missing_consensi.txt ]]; then
while read file_name; do
cat $file_name >> ${DATA_DIR}/${RM_LIBRARY}
done < ${DATA_DIR}/missing_consensi.txt
fi
sed -i 's/ .*//' ${DATA_DIR}/${RM_LIBRARY}
# Identify simple repeats and satellites, trim ends of LINEs/SINEs
echo "Splitting for simple/satellite packages"
mkdir -p ${DATA_DIR}/trf/split
python ${STRAIN_SCRIPTS}/splitter.py -i ${DATA_DIR}/${RM_LIBRARY} -o ${DATA_DIR}/trf/split
cp ${DATA_DIR}/${RM_LIBRARY} ${DATA_DIR}/trf/
# Run and parse TRF
echo "Running TRF"
parallel ${BAR_FLAG} --tmpdir ${DATA_DIR}/tmp/ --jobs ${THREADS} --memfree ${MEM_FREE} -a ${DATA_DIR}/trf/split/${RM_LIBRARY}_split.txt trf ${DATA_DIR}/trf/split/{} 2 7 7 80 10 50 500 -d -h -ngs ">" ${DATA_DIR}/trf/split/{}.trf
parallel ${BAR_FLAG} --tmpdir ${DATA_DIR}/tmp/ --jobs ${THREADS} --memfree ${MEM_FREE} -a ${DATA_DIR}/trf/split/${RM_LIBRARY}_split.txt python3 ${STRAIN_SCRIPTS}/trf_parser.py --trf ${DATA_DIR}/trf/split/{}.trf --out ${DATA_DIR}/trf/split/{}.trf.tsv
find ${DATA_DIR}/trf/split/ -type f -name "*trf.tsv" -exec cat {} + | cat > ${DATA_DIR}/trf/${RM_LIBRARY}.trf
# Run SA-SSR
echo "Running SA-SSR"
sa-ssr -e -l 20 -L 50000 -m 1 -M 5000 -t ${THREADS} ${DATA_DIR}/${RM_LIBRARY} ${DATA_DIR}/trf/${RM_LIBRARY}.sassr
# Run and compile mreps
echo "Running mreps"
parallel ${BAR_FLAG} --tmpdir ${DATA_DIR}/tmp/ --jobs ${THREADS} --memfree ${MEM_FREE} -a ${DATA_DIR}/trf/split/${RM_LIBRARY}_split.txt bash ${STRAIN_SCRIPTS}/mreps_parser.sh -i ${DATA_DIR}/trf/split/{} "2>"/dev/null
find ${DATA_DIR}/trf/split/ -type f -name "*mreps" -exec cat {} + | cat > ${DATA_DIR}/trf/${RM_LIBRARY}.mreps
# Interpret mreps, TRF and SA-SSR
echo "Trimming and sorting based on mreps, TRF, SA-SSR"
if [ ! -f ${DATA_DIR}/trf/${RM_LIBRARY}.sassr ]; then
touch ${DATA_DIR}/trf/${RM_LIBRARY}.sassr
fi
Rscript ${STRAIN_SCRIPTS}/simple_repeat_filter_trim.R -i ${DATA_DIR}/${RM_LIBRARY} -d ${DATA_DIR} -p 50
# Delete temp files
echo "Removing temporary files"
rm -r ${DATA_DIR}/*/split/
rm -r ${DATA_DIR}/tmp/
find ${DATA_DIR}/ -mindepth 1 -name "run_*" -exec rm -r {} +
if [[ $RUNS -gt 0 ]]; then rm ${GENOME}.n*; fi
# Classify improved consensi using RepeatModeler's RepeatClassifier
echo "Reclassifying repeats"
mkdir -p ${DATA_DIR}/classify/
cp ${DATA_DIR}/trf/${RM_LIBRARY}.nonsatellite ${DATA_DIR}/classify/
cd ${DATA_DIR}/classify/
if [ -s ${RM_LIBRARY}.nonsatellite ]; then
RepeatClassifier -threads ${THREADS} -consensi ${RM_LIBRARY}.nonsatellite
fi
# legacy command for older installations - DEPRECATED
# RepeatClassifier -pa ${THREADS} -consensi ${RM_LIBRARY}.nonsatellite
cd -
# Compile classified files
if [ -f ${DATA_DIR}/classify/${RM_LIBRARY}.nonsatellite.classified ]; then
cp ${DATA_DIR}/classify/${RM_LIBRARY}.nonsatellite.classified ${DATA_DIR}/${RM_LIBRARY}.strained
else
touch ${DATA_DIR}/${RM_LIBRARY}.strained
fi
echo "Compiling library"
if [ -f ${DATA_DIR}/trf/${RM_LIBRARY}.satellites ]; then
cat ${DATA_DIR}/trf/${RM_LIBRARY}.satellites >> ${DATA_DIR}/${RM_LIBRARY}.strained
fi
================================================
FILE: scripts/TEstrainer/data/acceptable_domains.tsv
================================================
ref abbrev
pfam17917 RT_RNaseH
cd09274 RNase_HI_RT_Ty3
cd01647 RT_LTR
pfam17919 RT_RNaseH_2
pfam00078 RVT_1
pfam17921 Integrase_H2C2
cd03715 RT_ZFREV_like
pfam00665 rve
cd01645 RT_Rtv
cd00303 retropepsin_like
cd03714 RT_DIRS1
cd09275 RNase_HI_RT_DIRS1
cd00304 RT_like
pfam13975 gag-asp_proteas
pfam03732 Retrotrans_gag
pfam13650 Asp_protease_2
NF033577 transpos_IS481
cd06222 RNase_H_like
cd05484 retropepsin_like_LTR_2
cd05483 retropepsin_like_bacteria
pfam08284 RVP_2
cd01644 RT_pepA17
pfam09337 zf-H2C2
cd00024 CD_CSD
pfam00385 Chromo
cd05481 retropepsin_like_LTR_1
smart00298 CHROMO
pfam19259 Ty3_capsid
cd18975 CD_MarY1_POL_like
cd18978 CD_DDE_transposase_like
cd18973 CD_Tf2-1_POL_like
cd18631 CD_HP1_like
pfam00098 zf-CCHC
smart00343 ZnF_C2HC
cd18634 CD_CDY
cd18635 CD_CMT3_like
pfam13456 RVT_3
cd18644 CD_polycomb
cd06095 RP_RTVL_H_like
cd18980 CD_NC-like
cd18650 CD_HP1beta_Cbx1
cd18979 CD_POL_like
cd09279 RNase_HI_like
cd18630 CD_Rhino
cd18653 CD_HP1a_insect
cd05479 RP_DDI
cd18627 CD_polycomb_like
cd18651 CD_HP1alpha_Cbx5
cd18659 CD2_tandem
cd18647 CD_Cbx2
cd18633 CD_MMP8
cd18638 CD_EhHp1_like
COG0328 RnhA
cd06094 RP_Saci_like
cd18652 CD_HP1gamma_Cbx3
cd18632 CD_Clr4_like
cd18968 chromodomain
cd18639 CD_SUV39H1_like
cd18645 CD_Cbx4
pfam16297 DUF4939
cd18637 CD_Swi6_like
cd18646 CD_Cbx7
cd18649 CD_Cbx8
pfam13683 rve_3
cd18966 chromodomain
PRK07708 PRK07708
PTZ00368 PTZ00368
cd18648 CD_Cbx6
cd18962 CD_MT_like
cd18970 CD_POL_like
cd18640 CD_Chro-like
PRK13907 rnhA
cd09272 RNase_HI_RT_Ty1
pfam07727 RVT_2
pfam13976 gag_pre-integrs
pfam14223 Retrotran_gag_2
pfam14244 Retrotran_gag_3
pfam13961 DUF4219
COG2801 Tra5
cd01650 RT_nLTR_like
cd09076 L1-EN
cd08372 EEP
pfam03372 Exo_endo_phos
cd01651 RT_G2_intron
cd09073 ExoIII_AP-endo
pfam13966 zf-RVT
cd09087 Ape1-like_AP-endo
cd09086 ExoIII-like_AP-endo
TIGR00633 DNA-apurinic_or_apyrimidinic_site_lyase
cd09085 Mth212-like_AP-endo
cd10281 Nape_like_AP-endo
pfam14529 Exo_endo_phos_2
COG0708 XthA
cd09077 R1-I-EN
cd09088 Ape2-like_AP-endo
cd03487 RT_Bac_retron_II
pfam02994 Transposase_22
pfam14111 DUF4283
cd09080 TDP2
TIGR00195 Exodeoxyribonuclease_III
pfam17490 Tnp_22_dsRBD
cd09084 EEP-2
PRK13911 PRK13911
PRK11756 PRK11756
pfam14392 zf-CCHC_4
cd09280 RNase_HI_eukaryote_like
cd09276 Rnase_HI_RT_non_LTR
pfam00075 RNase_H
COG3568 ElsH
cd09278 RNase_HI_prokaryote_like
pfam05699 Dimer_Tnp_hAT
smart00614 ZnF_BED
pfam14291 DUF4371
pfam14372 DUF4413
pfam04937 DUF659
pfam02892 zf-BED
smart00597 ZnF_TTF
pfam18658 zf-C2H2_12
NF033545 transpos_IS630
pfam13358 DDE_3
pfam03184 DDE_1
pfam13565 HTH_32
pfam01498 HTH_Tnp_Tc3_2
COG3415 COG3415
pfam03221 HTH_Tnp_Tc5
smart00674 CENPB
pfam01359 Transposase_1
pfam17906 HTH_48
pfam13384 HTH_23
pfam13518 HTH_28
COG3335 COG3335
pfam04218 CENP-B_N
pfam13936 HTH_38
pfam05225 HTH_psq
pfam16087 DUF4817
pfam13412 HTH_24
smart00351 PAX
pfam18701 DUF5641
pfam05380 Peptidase_A17
pfam03564 DUF1759
pfam05585 DUF1758
cd15489 PHD_SF
cd15522 PHD_TAF3
cd15517 PHD_TCF19_like
smart00249 PHD
cd15560 PHD2_3_BPTF
cd15552 PHD_PHF3_like
cd15610 PHD3_KDM5A_like
cd15553 PHD_Cfp1
cd15556 PHD_MMD1_like
cd16039 PHD_SPP1
cd15529 PHD2_PHF10
cd15554 PHD_PHF2_like
cd15550 PHD_MLL5
cd15518 PHD_Ecm5p_Lid2p_like
cd15505 PHD_ING
cd15640 PHD_KDM7
cd15551 PHD_PYGO
cd15525 PHD_UHRF1_2
cd15638 PHD_PHF3
cd15510 PHD2_KMT2C_like
cd15520 PHD3_Lid2p_like
cd15613 PHD_AL_plant
cd15587 PHD_Yng1p_like
cd15585 PHD_ING3
cd15627 PHD_BAZ1A
cd15687 PHD3_KDM5B
cd15570 PHD_Bye1p_SIZ1_like
cd15563 PHD3_PHF14
cd15609 PHD_TCF19
cd15639 PHD_DIDO1_like
cd15543 PHD_RSF1
cd15544 PHD_BAZ1A_like
cd15594 PHD2_KMT2C
cd15508 PHD3_KMT2A_like
cd15495 PHD_ATX3_4_5_like
cd15506 PHD1_KMT2A_like
cd15595 PHD2_KMT2D
pfam00628 PHD
cd15545 PHD_BAZ2A_like
cd15519 PHD1_Lid2p_like
cd15586 PHD_ING4_5
cd15681 PHD_JADE3
cd15535 PHD1_Rco1
cd15632 PHD_PHF13
cd15679 PHD_JADE1
cd01646 RT_Bac_retron_I
TIGR04416 hypothetical_protein
cd00229 SGNH_hydrolase
pfam13472 Lipase_GDSL_2
cd01828 sialate_O-acetylesterase_like2
COG3344 COG3344
cd04501 SGNH_hydrolase_like_4
cd01832 SGNH_hydrolase_like_1
cd04502 SGNH_hydrolase_like_7
cd15636 PHD_PYGO2
COG2755 TesA
cd01841 NnaC_like
cd15635 PHD_PYGO1
cd01833 XynB_like
cd01648 TERT
cd01836 FeeA_FeeB_like
cd01825 SGNH_hydrolase_peri1
cd15637 PHD_dPYGO
pfam10551 MULE
smart00575 ZnF_PMZ
pfam04434 SWIM
pfam03108 DBD_Tnp_Mut
pfam03101 FAR1
PLN03097 FHY3
pfam09331 DUF1985
pfam02902 Peptidase_C48
pfam10536 PMD
pfam00872 Transposase_mut
COG3328 COG3328
cd22744 OTU
cd00799 INT_Cre_C
pfam11560 LAP2alpha
cd00397 DNA_BRE_C
pfam00589 Phage_integrase
pfam05869 Dam
pfam13495 Phage_int_SAM_4
pfam02899 Phage_int_SAM_1
cd01194 INT_C_like_4
cd01188 INT_RitA_C_like
cd00796 INT_Rci_Hp1_C
pfam02992 Transposase_21
pfam13963 Transpos_assoc
pfam13960 DUF4218
pfam13952 DUF4216
pfam03004 Transposase_24
pfam03017 Transposase_23
pfam20049 DUF6451
cd10442 GIY-YIG_PLEs
cd10443 GIY-YIG_HE_Tlr8p_PBC-V_like
cd00719 GIY-YIG_SF
cd10437 GIY-YIG_HE_I-TevI_like
cd09277 RNase_HI_bacteria_like
cd09273 RNase_HI_RT_Bel
PRK00203 rnhA
cd09081 CdtB
pfam13359 DDE_Tnp_4
pfam04827 Plant_tran
pfam14303 NAM-associated
NF033581 transpos_IS5_4
pfam13837 Myb_DNA-bind_4
cd12203 GT1
pfam01609 DDE_Tnp_1
cd10522 SET_LegAS4-like
cd19174 SET_ASH1L
cd19183 SET_SpSET3-like
cd20071 SET_SMYD
cd10529 SET_SETD5-like
cd10531 SET_SETD2-like
cd10533 SET_EHMT2
cd10535 SET_EHMT1
cd10538 SET_SETDB-like
cd19172 SET_SETD2
cd19175 SET_ASHR3-like
cd19211 SET_NSD2
cd20072 SET_SET1
pfam10545 MADF_DNA_bdg
pfam18697 MLVIN_C
pfam00077 RVP
cd09950 ENVV1-like_HR1-HR2
cd09948 Ebola_RSV-like_HR1-HR2
pfam02093 Gag_p30
cd09951 HERV-Rb-like_HR1-HR2
cd09851 HTLV-1-like_HR1-HR2
cd09850 Ebola-like_HR1-HR2
cd09947 Ebola_HIV-1-like_HR1-HR2
cd13934 RNase_H_Dikarya_like
pfam00429 TLV_coat
pfam01140 Gag_MA
cd05482 HIV_retropepsin_like
cd09949 RSV-like_HR1-HR2
PRK08719 PRK08719
pfam16721 zf-H3C2
PRK06548 PRK06548
cd18037 DEXSc_Pif1_like
pfam05970 PIF1
pfam14214 Helitron_like_N
cd17933 DEXSc_RecD-like
cd18809 SF1_C_RecD
pfam13604 AAA_30
cd18786 SF1_C
pfam13245 AAA_19
cd04480 RPA1_DBD_A_like
COG0507 RecD
cd04476 RPA1_DBD_C
cd04481 RPA1_DBD_B_like
pfam02721 DUF223
pfam13401 AAA_22
pfam13538 UvrD_C_2
pfam04843 Herpes_teg_N
cd22758 OTU_232R-like
smart00382 AAA
TIGR01448 recD_rel
cd22748 OTU_OTUD6-like
cd22755 OTU_CeDUB-like
cd22756 OTU_OTUD3-like
cd22747 OTU_OTUD1
cd22752 OTU_OTUD5-like
cd22753 OTU_ALG13-like
cd22771 OTU_plant_OTU7-like
cd22751 OTU_plant_OTU9-like
cd22757 OTU_P87_VP80-like
cd22762 OTU_fungi_OTU2-like
cd22794 OTU_OTUD4
cd22796 OTU_plant_OTU6-like
cd09083 EEP-1
pfam07530 PRE_C2HC
pfam00607 Gag_p24
pfam06817 RVT_thumb
pfam19317 Gag_p24_C
pfam02022 Integrase_Zn
pfam00552 IN_DBD_C
pfam00692 dUTPase
pfam14787 zf-CCHC_5
cd07557 trimeric_dUTPase
PRK00601 dut
pfam02337 Gag_p10
TIGR00576 Deoxyuridine_5'-triphosphate_nucleotidohydrolase
COG0756 Dut
PLN02547 PLN02547
PHA02703 PHA02703
PHA03094 PHA03094
smart00443 G_patch
pfam01585 G-patch
pfam00517 GP41
cd09909 HIV-1-like_HR1-HR2
PTZ00143 PTZ00143
pfam04160 Borrelia_orfX
PHA02517 PHA02517
pfam15695 HERV-K_REC
pfam12656 G-patch_2
pfam13804 HERV-K_env_2
NF033594 transpos_ISNCY_2
PHA03131 PHA03131
pfam02813 Retro_M
COG5082 AIR1
pfam13843 DDE_Tnp_1_7
cd22343 PDDEXK_lambda_exonuclease-like
cd00180 PKc
cd05122 PKc_STE
cd06917 STKc_NAK1_like
cd07832 STKc_CCRK
cd07847 STKc_CDKL1_4
cd13982 STKc_IRE1
cd14014 STKc_PknB_like
cd14133 PKc_DYRK_like
cd14229 STKc_HIPK3
pfam13842 Tnp_zf-ribbon_2
pfam01107 MP
pfam02160 Peptidase_A3
cd17920 DEXHc_RecQ
smart00490 HELICc
cd18018 DEXHc_RecQ4-like
pfam00271 Helicase_C
cd18794 SF2_C_RecQ
cd18017 DEXHc_RecQ3
pfam00270 DEAD
cd18016 DEXHc_RecQ2_BLM
smart00487 DEXDc
cd18015 DEXHc_RecQ1
cd18014 DEXHc_RecQ5
COG0514 RecQ
cd00046 SF2-N
cd17921 DEXHc_Ski2
PRK11057 PRK11057
TIGR00614 Werner_syndrome_ATP-dependent_helicase
cd18785 SF2_C
cd18787 SF2_C_DEAD
cd17923 DEXHc_Hrq1-like
cd18797 SF2_C_Hrq
cd18793 SF2_C_SNF
cd18796 SF2_C_LHR
cd17927 DEXHc_RIG-I
COG0513 SrmB
pfam04851 ResIII
cd18028 DEXHc_archSki2
cd18802 SF2_C_dicer
cd17926 DEXHc_RE
cd18023 DEXHc_HFM1
cd18795 SF2_C_Ski2
cd09639 Cas3_I
cd17922 DEXHc_LHR-like
cd18811 SF2_C_RecG
cd01820 PAF_acetylesterase_like
pfam03175 DNA_pol_B_2
pfam08398 Parvo_coat_N
PHA02563 PHA02563
pfam04665 Pox_A32
NF033516 transpos_IS3
cd18974 CD_POL_like
pfam02229 PC4
cd18965 chromodomain
pfam19263 DUF5906
cd00652 TBP_TLF
cd04516 TBP_eukaryotes
cd04517 TLF
cd04518 TBP_archaea
cd18976 CD_POL_like
COG2101 SPT15
pfam00352 TBP
pfam02945 Endonuclease_7
PLN00062 PLN00062
PRK00394 PRK00394
pfam05272 VirE
pfam18930 DUF5679
cd00009 AAA
cd00022 BIR
cd00094 HX
cd18969 chromodomain
cd18972 CD_POL_like
pfam04383 KilA-N
pfam12943 DUF3839
sd00017 ZF_C2H2
pfam13894 zf-C2H2_4
cd00167 SANT
pfam05485 THAP
pfam12017 Tnp_P_element
smart00692 DM3
smart00980 THAP
pfam12596 Tnp_P_element_C
PRK07764 PRK07764
pfam09004 DUF1891
pfam01021 TYA
pfam09668 Asp_protease
TIGR03033 hypothetical_protein
pfam09588 YqaJ
pfam13613 HTH_Tnp_4
cd15682 PHD_ING1
cd15683 PHD_ING2
cd15685 PHD_ING5
cd15584 PHD_ING1_2
cd15684 PHD_ING4
pfam02037 SAP
pfam16012 DUF4780
cd22761 OTU_OTUD6
cd22770 OTU_OTUD3
cd22797 OTU_plant_OTU5-like
cd22760 OTU_plant_OTU4-like
cd22746 OTU_plant_OTU3_4-like
cd22795 OTU_ALG13
cd22759 OTU_plant_OTU3-like
cd00096 Ig
cd05717 IgV_1_Necl_like
cd05718 IgV_1_PVR_like
cd05719 IgC1_2_PVR_like
cd05725 IgI_3_Robo
cd05846 IgV_1_MRC-OX-2_like
cd05882 IgV_1_Necl-1
cd05886 IgV_1_Nectin-1_like
cd05887 IgV_1_Nectin-3_like
cd05888 IgV_1_Nectin-4_like
cd05889 IgV_1_DNAM-1_like
cd05890 IgC1_2_Nectin-1_like
cd07703 IgC1_2_Nectin-2_Necl-5_like
cd07704 IgC1_2_Nectin-3-4_like
cd16089 IgV_CRIg
cd20930 Ig3_Nectin-5_like
cd20989 IgV_1_Nectin-2_NecL-5_like_CD112_CD155
pfam07686 V-set
pfam08205 C2-set_2
pfam13927 Ig_3
smart00410 IG_like
pfam16124 RecQ_Zn_bind
NF033563 transpos_IS30
cd00590 RRM_SF
cd12224 RRM_RBM22
cd12229 RRM_G3BP
cd12233 RRM_Srp1p_AtRSp31_like
cd12236 RRM_snRNP70
cd12237 RRM_snRNP35
cd12242 RRM_SLIRP
cd12297 RRM2_Prp24
cd12312 RRM_SRSF10_SRSF12
cd12316 RRM3_RBM19_RRM2_MRD1
cd12320 RRM6_RBM19_RRM5_MRD1
cd12324 RRM_RBM8
cd12337 RRM1_SRSF4_like
cd12338 RRM1_SRSF1_like
cd12341 RRM_hnRNPC_like
cd12343 RRM1_2_CoAA_like
cd12347 RRM_PPIE
cd12363 RRM_TRA2
cd12373 RRM_SRSF3_like
cd12382 RRM_RBMX_like
cd12390 RRM3_RAVER
cd12391 RRM1_SART3
cd12392 RRM2_SART3
cd12398 RRM_CSTF2_RNA15_like
cd12399 RRM_HP0827_like
cd12400 RRM_Nop6
cd12402 RRM_eIF4B
cd12404 RRM2_NCL
cd12412 RRM_DAZL_BOULE
cd12413 RRM1_RBM28_like
cd12414 RRM2_RBM28_like
cd12448 RRM2_gar2
cd12449 RRM_CIRBP_RBM3
cd12450 RRM1_NUCLs
cd12494 RRM3_hnRNPR
cd12597 RRM1_SRSF1
cd12599 RRM1_SF2_plant_like
cd12608 RRM1_CoAA
cd12641 RRM_TRA2B
cd12642 RRM_TRA2A
cd12672 RRM_DAZL
cd12674 RRM1_Nop4p
cd12676 RRM3_Nop4p
cd21607 RRM3_HRB1_GBP2
cd21608 RRM2_NsCP33_like
cd21615 RRM_SNP1_like
COG1877 OtsB
pfam00076 RRM_1
smart00360 RRM
cd15492 PHD_BRPF_JADE_like
pfam02257 RFX_DNA_binding
cd15536 PHD_PHRF1
cd15573 PHD_JADE
cd15515 PHD1_KDM5A_like
cd15532 PHD2_CHD_II
cd15572 PHD_BRPF
cd15603 PHD1_KDM5B
cd15628 PHD_BAZ1B
cd15677 PHD_BRPF2
cd15680 PHD_JADE2
pfam12012 DUF3504
cd06141 WRN_exo
cd06146 mut-7_like_exo
cd06129 RNaseD_like
cd09018 DEDDy_polA_RNaseD_like_exo
pfam01612 DNA_pol_A_exo1
smart00474 35EXOc
cd06142 RNaseD_exo
pfam06975 DUF1299
pfam18717 CxC4
pfam18758 KDZ
pfam18866 CxC7
pfam12550 GCR1_C
pfam16787 NDC10_II
NF033547 transpos_IS1595
pfam12762 DDE_Tnp_IS1595
cd22792 OTU_RDRP-like
================================================
FILE: scripts/TEstrainer/data/additional_domains.tsv
================================================
class ref abbrev clade
Pao NF033838 PspC_subgroup_1 Amphibia
Pao PRK14475 PRK14475 Amphibia
Pao TIGR02499 translocation_protein Amphibia
Pao cd06503 ATP-synt_Fo_b Amphibia
Pao pfam16999 V-ATPase_G_2 Amphibia
Pao PRK02292 PRK02292 Amphibia
Pao cd06503 ATP-synt_Fo_b Amphibia
Pao pfam16999 V-ATPase_G_2 Amphibia
Pao smart00935 OmpH Amphibia
Pao pfam00430 ATP-synt_B Amphibia
Pao COG0711 AtpF Amphibia
Pao PRK08475 PRK08475 Amphibia
Pao pfam12072 DUF3552 Amphibia
Pao PRK05759 PRK05759 Amphibia
Pao PRK07353 PRK07353 Amphibia
Pao COG2811 NtpF Amphibia
Pao TIGR02926 Uncharacterized_protein_AF_1158 Amphibia
Pao pfam15975 Flot Amphibia
Gypsy pfam14893 PNMA Amphibia
TcMar-Tc1 COG2512 COG2512 Amphibia
TcMar-Tc1 COG2522 COG2522 Amphibia
LINE cd14695 bZIP_HLF Amphibia
LINE COG1579 COG1579 Amphibia
LINE pfam08614 ATG16 Amphibia
PIF-Harbinger pfam02944 BESS Amphibia
Gypsy cd07936 SCAN Amphibia
ERV pfam07679 I-set Amphibia
ERV cd07700 IgV_CD8_beta Amphibia
Gypsy pfam13696 zf-CCHC_2 Amphibia
Gypsy cd18977 CD_POL_like Amphibia
Helitron cd17914 DExxQc_SF1-N Amphibia
Helitron cd02661 Peptidase_C19E Amphibia
Helitron pfam00443 UCH Amphibia
Helitron cd02668 Peptidase_C19L Amphibia
Helitron COG5533 UBP5 Amphibia
Helitron cd02667 Peptidase_C19K Amphibia
Helitron cd02257 Peptidase_C19 Amphibia
Helitron cd02674 Peptidase_C19R Amphibia
Helitron cd02662 Peptidase_C19F Amphibia
Helitron cd02658 Peptidase_C19B Amphibia
Helitron TIGR01447 recD Amphibia
Gypsy cd18964 chromodomain Amphibia
Helitron cd02673 Peptidase_C19Q Amphibia
ERV pfam02813 Retro_M Aves
ERV pfam03708 Avian_gp85 Aves
R2 pfam13921 Myb_DNA-bind_6 Aves
ERV COG1340 COG1340 Aves
ERV COG4026 COG4026 Aves
ERV pfam00769 ERM Aves
ERV COG4942 COG4942 Aves
DNA_virus pfam00242 DNA_pol_viral_N Aves
DNA_virus pfam00336 DNA_pol_viral_C Aves
DNA_virus pfam00906 Hepatitis_core Aves
DNA_virus pfam00695 vMSA Aves
ERV pfam10473 CENP-F_leu_zip Aves
ERV pfam16888 DUF5082 Aves
ERV pfam10473 CENP-F_leu_zip Aves
ERV pfam02403 Seryl_tRNA_N Aves
ERV TIGR03698 clan_AA_DTGF Aves
ERV COG5550 COG5550 Aves
ERV pfam00047 ig Aves
ERV cd05740 IgI_hCEACAM_2_4_6_like Aves
ERV cd20935 IgV_B7-H2 Aves
ERV TIGR02794 Protein_TolA Aves
ERV pfam05917 DUF874 Aves
ERV pfam11559 ADIP Aves
ERV pfam03938 OmpH Aves
ERV cd14717 bZIP_Maf_small Aves
ERV cd22887 Atg16_CCD Aves
ERV pfam14662 KASH_CCD Aves
ERV cd22887 Atg16_CCD Aves
ERV pfam00529 HlyD Aves
ERV pfam12795 MscS_porin Aves
ERV cd22887 Atg16_CCD Aves
L1 pfam06825 HSBP1 Mammalia
ERV PRK13956 dut Mammalia
ERV pfam13804 HERV-K_env_2 Mammalia
ERV pfam01141 Gag_p12 Mammalia
ERV-Foamy pfam03276 Gag_spuma Mammalia
ERV-Foamy pfam03539 Spuma_A9PTase Mammalia
ERV-Foamy pfam03408 Foamy_virus_ENV Mammalia
L1 pfam01442 Apolipoprotein Lepidosauria
Gypsy pfam04814 HNF-1_N Testudines
Gypsy cd14693 bZIP_CEBP Testudines
ERV cd02901 Macro_Poa1p-like Actinopterygii
ERV smart00506 A1pp Actinopterygii
ERV cd02749 Macro_SF Actinopterygii
ERV PHA02595 tk.4 Actinopterygii
ERV pfam01661 Macro Actinopterygii
Rex-Babar pfam16761 Clr2_transil Actinopterygii
Gypsy pfam02023 SCAN Actinopterygii
Gypsy smart00431 SCAN Actinopterygii
Gypsy pfam03078 ATHILA Plantae
Gypsy pfam04195 Transposase_28 Plantae
Caulimovirus pfam03938 OmpH Plantae
Caulimovirus PRK03957 PRK03957 Plantae
Copia pfam13696 zf-CCHC_2 Plantae
Gypsy pfam15294 Leu_zip Plantae
Gypsy PRK07238 PRK07238 Plantae
MULE-MuDR cd06410 PB1_UP2 Plantae
MULE-MuDR pfam09322 DUF1979 Plantae
MULE-MuDR pfam05928 Zea_mays_MuDR Plantae
================================================
FILE: scripts/TEstrainer/data/exceptional_domains.tsv
================================================
class ref abbrev clade
ERV cd00096 Ig Vertebrata
ERV cd05717 IgV_1_Necl_like Vertebrata
ERV cd05718 IgV_1_PVR_like Vertebrata
ERV cd05719 IgC1_2_PVR_like Vertebrata
ERV cd05725 IgI_3_Robo Vertebrata
ERV cd05759 IgI_2_KIRREL3-like Vertebrata
ERV cd05846 IgV_1_MRC-OX-2_like Vertebrata
ERV cd05882 IgV_1_Necl-1 Vertebrata
ERV cd05886 IgV_1_Nectin-1_like Vertebrata
ERV cd05887 IgV_1_Nectin-3_like Vertebrata
ERV cd05888 IgV_1_Nectin-4_like Vertebrata
ERV cd05889 IgV_1_DNAM-1_like Vertebrata
ERV cd05890 IgC1_2_Nectin-1_like Vertebrata
ERV cd07703 IgC1_2_Nectin-2_Necl-5_like Vertebrata
ERV cd07704 IgC1_2_Nectin-3-4_like Vertebrata
ERV cd16089 IgV_CRIg Vertebrata
ERV cd20930 Ig3_Nectin-5_like Vertebrata
ERV cd20946 IgV_1_JAM1-like Vertebrata
ERV cd20989 IgV_1_Nectin-2_NecL-5_like_CD112_CD155 Vertebrata
ERV pfam07686 V-set Vertebrata
ERV pfam13927 Ig_3 Vertebrata
ERV smart00410 IG_like Vertebrata
Polinton cd04277 ZnMc_serralysin_like Insecta
Polinton cd04278 ZnMc_MMP Insecta
Polinton cd04279 ZnMc_MMP_like_1 Insecta
Polinton smart00235 ZnMc Insecta
================================================
FILE: scripts/TEstrainer/data/old_acceptable_domains.tsv
================================================
ref abbrev
cd09274 RNase_HI_RT_Ty3
cd01647 RT_LTR
pfam17917 RT_RNaseH
pfam17919 RT_RNaseH_2
pfam00078 RVT_1
pfam17921 Integrase_H2C2
cd03715 RT_ZFREV_like
cd01645 RT_Rtv
pfam00665 rve
cd00303 retropepsin_like
cd03714 RT_DIRS1
cd09275 RNase_HI_RT_DIRS1
cd00304 RT_like
pfam03732 Retrotrans_gag
pfam13975 gag-asp_proteas
NF033577 transpos_IS481
pfam13650 Asp_protease_2
cd06222 RNase_H_like
cd05484 retropepsin_like_LTR_2
pfam09337 zf-H2C2
cd05483 retropepsin_like_bacteria
pfam08284 RVP_2
cd01644 RT_pepA17
cd00024 CD_CSD
pfam00385 Chromo
cd05481 retropepsin_like_LTR_1
smart00298 CHROMO
cd18975 CD_MarY1_POL_like
cd18978 CD_DDE_transposase_like
cd18973 CD_Tf2-1_POL_like
cd18631 CD_HP1_like
cd18966 chromodomain
cd18968 chromodomain
smart00343 ZnF_C2HC
cd18979 CD_POL_like
cd18634 CD_CDY
cd18635 CD_CMT3_like
cd18970 CD_POL_like
pfam13456 RVT_3
cd18644 CD_polycomb
cd06095 RP_RTVL_H_like
cd18976 CD_POL_like
cd18980 CD_NC-like
cd18650 CD_HP1beta_Cbx1
cd09279 RNase_HI_like
cd18630 CD_Rhino
cd18653 CD_HP1a_insect
cd05479 RP_DDI
cd18627 CD_polycomb_like
cd18651 CD_HP1alpha_Cbx5
cd18659 CD2_tandem
cd18647 CD_Cbx2
cd18633 CD_MMP8
cd18974 CD_POL_like
cd18960 CD_HP1_like
cd18638 CD_EhHp1_like
COG0328 RnhA
cd18971 CD_POL_like
cd18977 CD_POL_like
PTZ00368 PTZ00368
pfam00098 zf-CCHC
cd06094 RP_Saci_like
cd18652 CD_HP1gamma_Cbx3
cd18969 chromodomain
cd18632 CD_Clr4_like
cd09272 RNase_HI_RT_Ty1
pfam07727 RVT_2
pfam13976 gag_pre-integrs
pfam14223 Retrotran_gag_2
pfam14244 Retrotran_gag_3
pfam13961 DUF4219
pfam13683 rve_3
cd01650 RT_nLTR_like
cd09076 L1-EN
pfam03372 Exo_endo_phos
cd08372 EEP
cd01651 RT_G2_intron
cd09073 ExoIII_AP-endo
pfam13966 zf-RVT
cd09087 Ape1-like_AP-endo
cd09086 ExoIII-like_AP-endo
cd09085 Mth212-like_AP-endo
cd10281 Nape_like_AP-endo
pfam14529 Exo_endo_phos_2
COG0708 XthA
cd09077 R1-I-EN
cd09088 Ape2-like_AP-endo
cd03487 RT_Bac_retron_II
pfam02994 Transposase_22
pfam14111 DUF4283
cd09080 TDP2
pfam17490 Tnp_22_dsRBD
cd09084 EEP-2
pfam14392 zf-CCHC_4
TIGR00633 DNA-apurinic_or_apyrimidinic_site_lyase
PRK11756 PRK11756
PRK13911 PRK13911
cd09280 RNase_HI_eukaryote_like
cd09276 Rnase_HI_RT_non_LTR
pfam00075 RNase_H
pfam05699 Dimer_Tnp_hAT
smart00614 ZnF_BED
pfam14291 DUF4371
pfam14372 DUF4413
pfam04937 DUF659
pfam02892 zf-BED
smart00597 ZnF_TTF
pfam18658 zf-C2H2_12
NF033545 transpos_IS630
pfam13358 DDE_3
pfam03184 DDE_1
pfam01498 HTH_Tnp_Tc3_2
pfam13565 HTH_32
COG3415 COG3415
pfam01359 Transposase_1
pfam03221 HTH_Tnp_Tc5
smart00674 CENPB
pfam17906 HTH_48
pfam13518 HTH_28
pfam13384 HTH_23
COG3335 COG3335
pfam04218 CENP-B_N
pfam13936 HTH_38
pfam05225 HTH_psq
pfam16087 DUF4817
pfam13412 HTH_24
pfam18701 DUF5641
pfam05380 Peptidase_A17
pfam03564 DUF1759
pfam05585 DUF1758
smart00249 PHD
cd15489 PHD_SF
cd15522 PHD_TAF3
pfam00628 PHD
cd15517 PHD_TCF19_like
cd15560 PHD2_3_BPTF
cd15552 PHD_PHF3_like
cd15610 PHD3_KDM5A_like
cd15553 PHD_Cfp1
cd15556 PHD_MMD1_like
cd16039 PHD_SPP1
cd15529 PHD2_PHF10
cd15554 PHD_PHF2_like
cd15550 PHD_MLL5
cd15518 PHD_Ecm5p_Lid2p_like
cd15505 PHD_ING
cd15640 PHD_KDM7
cd15551 PHD_PYGO
cd15525 PHD_UHRF1_2
cd15638 PHD_PHF3
cd15510 PHD2_KMT2C_like
cd15520 PHD3_Lid2p_like
cd15613 PHD_AL_plant
cd15587 PHD_Yng1p_like
cd15585 PHD_ING3
cd15627 PHD_BAZ1A
cd15687 PHD3_KDM5B
cd01646 RT_Bac_retron_I
cd00229 SGNH_hydrolase
pfam13472 Lipase_GDSL_2
cd01828 sialate_O-acetylesterase_like2
COG3344 COG3344
cd04501 SGNH_hydrolase_like_4
cd04502 SGNH_hydrolase_like_7
cd01832 SGNH_hydrolase_like_1
cd15636 PHD_PYGO2
COG2755 TesA
cd01841 NnaC_like
cd15635 PHD_PYGO1
cd01833 XynB_like
pfam10551 MULE
smart00575 ZnF_PMZ
pfam04434 SWIM
pfam03108 DBD_Tnp_Mut
pfam03101 FAR1
PLN03097 FHY3
pfam09331 DUF1985
pfam02902 Peptidase_C48
NF033543 transpos_IS256
pfam10536 PMD
pfam00872 Transposase_mut
COG3328 COG3328
cd00799 INT_Cre_C
pfam11560 LAP2alpha
cd00397 DNA_BRE_C
pfam00589 Phage_integrase
pfam02899 Phage_int_SAM_1
pfam13495 Phage_int_SAM_4
pfam05869 Dam
cd01194 INT_C_like_4
pfam02992 Transposase_21
pfam13960 DUF4218
pfam13963 Transpos_assoc
pfam13952 DUF4216
pfam03004 Transposase_24
pfam03017 Transposase_23
cd01648 TERT
cd10442 GIY-YIG_PLEs
cd10443 GIY-YIG_HE_Tlr8p_PBC-V_like
cd00719 GIY-YIG_SF
cd09278 RNase_HI_prokaryote_like
cd09277 RNase_HI_bacteria_like
cd09273 RNase_HI_RT_Bel
PRK00203 rnhA
pfam13359 DDE_Tnp_4
pfam04827 Plant_tran
NF033581 transpos_IS5_4
pfam14303 NAM-associated
pfam13837 Myb_DNA-bind_4
cd12203 GT1
pfam01609 DDE_Tnp_1
sd00010 SLR
pfam18697 MLVIN_C
pfam00077 RVP
cd09950 ENVV1-like_HR1-HR2
cd09948 Ebola_RSV-like_HR1-HR2
pfam02093 Gag_p30
cd09951 HERV-Rb-like_HR1-HR2
cd09851 HTLV-1-like_HR1-HR2
cd09850 Ebola-like_HR1-HR2
cd09947 Ebola_HIV-1-like_HR1-HR2
cd13934 RNase_H_Dikarya_like
pfam00429 TLV_coat
pfam01140 Gag_MA
cd05482 HIV_retropepsin_like
cd09949 RSV-like_HR1-HR2
PRK08719 PRK08719
pfam16721 zf-H3C2
COG2801 Tra5
cd18037 DEXSc_Pif1_like
pfam05970 PIF1
pfam14214 Helitron_like_N
cd17933 DEXSc_RecD-like
cd18809 SF1_C_RecD
pfam13604 AAA_30
cd18786 SF1_C
cd04480 RPA1_DBD_A_like
pfam13245 AAA_19
cd17914 DExxQc_SF1-N
COG0507 RecD
pfam02721 DUF223
cd04476 RPA1_DBD_C
cd04481 RPA1_DBD_B_like
pfam13401 AAA_22
pfam13538 UvrD_C_2
pfam04843 Herpes_teg_N
smart00382 AAA
COG3568 ElsH
pfam07530 PRE_C2HC
smart00596 PRE_C2HC
pfam00607 Gag_p24
pfam06817 RVT_thumb
pfam02022 Integrase_Zn
pfam00552 IN_DBD_C
pfam00692 dUTPase
pfam14787 zf-CCHC_5
cd07557 trimeric_dUTPase
PRK00601 dut
pfam02337 Gag_p10
COG0756 Dut
PHA03094 PHA03094
PLN02547 PLN02547
PHA02703 PHA02703
smart00443 G_patch
pfam01585 G-patch
TIGR00576 Deoxyuridine_5'-triphosphate_nucleotidohydrolase
PRK13956 dut
pfam00517 GP41
cd09909 HIV-1-like_HR1-HR2
pfam04160 Borrelia_orfX
PTZ00143 PTZ00143
PHA02517 PHA02517
pfam13843 DDE_Tnp_1_7
cd22343 PDDEXK_lambda_exonuclease-like
TIGR00195 Exodeoxyribonuclease_III
pfam01107 MP
pfam02160 Peptidase_A3
cd17920 DEXHc_RecQ
smart00490 HELICc
cd18018 DEXHc_RecQ4-like
pfam00271 Helicase_C
cd18794 SF2_C_RecQ
cd18017 DEXHc_RecQ3
pfam00270 DEAD
cd18016 DEXHc_RecQ2_BLM
cd18015 DEXHc_RecQ1
smart00487 DEXDc
COG0514 RecQ
cd18014 DEXHc_RecQ5
cd00046 SF2-N
cd17921 DEXHc_Ski2
PRK11057 PRK11057
cd18785 SF2_C
cd18787 SF2_C_DEAD
cd17923 DEXHc_Hrq1-like
cd18797 SF2_C_Hrq
cd18793 SF2_C_SNF
cd18796 SF2_C_LHR
cd17927 DEXHc_RIG-I
COG0513 SrmB
pfam04851 ResIII
cd18028 DEXHc_archSki2
cd18802 SF2_C_dicer
cd17926 DEXHc_RE
cd18023 DEXHc_HFM1
cd18795 SF2_C_Ski2
cd01820 PAF_acetylesterase_like
pfam03175 DNA_pol_B_2
PHA02563 PHA02563
pfam04665 Pox_A32
NF033516 transpos_IS3
pfam08398 Parvo_coat_N
cd18646 CD_Cbx7
cd18965 chromodomain
cd18972 CD_POL_like
pfam02229 PC4
cd18962 CD_MT_like
cd18645 CD_Cbx4
cd18649 CD_Cbx8
pfam00352 TBP
cd00022 BIR
cd04518 TBP_archaea
pfam00653 BIR
pfam13894 zf-C2H2_4
cd00167 SANT
pfam05485 THAP
smart00980 THAP
pfam12017 Tnp_P_element
smart00692 DM3
pfam12596 Tnp_P_element_C
PRK13907 rnhA
PRK07708 PRK07708
pfam09004 DUF1891
pfam09588 YqaJ
pfam13613 HTH_Tnp_4
TIGR03033 hypothetical_protein
cd15586 PHD_ING4_5
cd15682 PHD_ING1
cd15683 PHD_ING2
cd15685 PHD_ING5
pfam16012 DUF4780
TIGR04416 hypothetical_protein
pfam02338 OTU
pfam16124 RecQ_Zn_bind
cd00590 RRM_SF
cd12233 RRM_Srp1p_AtRSp31_like
cd12236 RRM_snRNP70
cd12237 RRM_snRNP35
cd12312 RRM_SRSF10_SRSF12
cd12320 RRM6_RBM19_RRM5_MRD1
cd12338 RRM1_SRSF1_like
cd12341 RRM_hnRNPC_like
cd12343 RRM1_2_CoAA_like
cd12363 RRM_TRA2
cd12373 RRM_SRSF3_like
cd12382 RRM_RBMX_like
cd12390 RRM3_RAVER
cd12392 RRM2_SART3
cd12399 RRM_HP0827_like
cd12404 RRM2_NCL
cd12412 RRM_DAZL_BOULE
cd12413 RRM1_RBM28_like
cd12414 RRM2_RBM28_like
cd12448 RRM2_gar2
cd12449 RRM_CIRBP_RBM3
cd12450 RRM1_NUCLs
cd12599 RRM1_SF2_plant_like
cd12608 RRM1_CoAA
cd12641 RRM_TRA2B
cd12642 RRM_TRA2A
cd12674 RRM1_Nop4p
cd21607 RRM3_HRB1_GBP2
cd21608 RRM2_NsCP33_like
cd21615 RRM_SNP1_like
COG1877 OtsB
pfam00076 RRM_1
smart00360 RRM
cd15492 PHD_BRPF_JADE_like
pfam02257 RFX_DNA_binding
PRK06548 PRK06548
cd06141 WRN_exo
cd06146 mut-7_like_exo
cd06129 RNaseD_like
cd09018 DEDDy_polA_RNaseD_like_exo
pfam01612 DNA_pol_A_exo1
smart00474 35EXOc
pfam06975 DUF1299
cd15637 PHD_dPYGO
pfam18717 CxC4
pfam18758 KDZ
COG3147 DedD
pfam12550 GCR1_C
pfam16787 NDC10_II
NF033547 transpos_IS1595
pfam12762 DDE_Tnp_IS1595
cd21575 KLF18_N
smart00238 BIR
sd00033 LRR_RI
pfam00400 WD40
sd00039 7WD40
smart00320 WD40
cd00200 WD40
COG2319 COG2319
sd00031 LRR_1
pfam00931 NB-ARC
pfam13855 LRR_8
================================================
FILE: scripts/TEstrainer/data/unacceptable_domains.tsv
================================================
ref abbrev organism
cd20663 CYP2D Ceratotherium_simum_cottoni
cd11026 CYP2 Ceratotherium_simum_cottoni
pfam00067 p450 Ceratotherium_simum_cottoni
cd11027 CYP17A1-like Ceratotherium_simum_cottoni
pfam00001 7tm_1 Ceratotherium_simum_cottoni
cd15337 7tmA_Opsin_Gq_invertebrates Ceratotherium_simum_cottoni
cd15001 7tmA_GPRnna14-like Ceratotherium_simum_cottoni
cd15160 7tmA_Proton-sensing_R Ceratotherium_simum_cottoni
cd15054 7tmA_5-HT6 Ceratotherium_simum_cottoni
cd15917 7tmA_OR51_52-like Ceratotherium_simum_cottoni
cd15953 7tmA_OR52P-like Ceratotherium_simum_cottoni
cd15221 7tmA_OR52B-like Ceratotherium_simum_cottoni
cd15952 7tmA_OR52E-like Ceratotherium_simum_cottoni
cd15951 7tmA_OR52R_52L-like Ceratotherium_simum_cottoni
pfam13853 7tm_4 Ceratotherium_simum_cottoni
cd15948 7tmA_OR52K-like Ceratotherium_simum_cottoni
cd15954 7tmA_OR52N-like Ceratotherium_simum_cottoni
cd15955 7tmA_OR52A-like Ceratotherium_simum_cottoni
cd15950 7tmA_OR52I-like Ceratotherium_simum_cottoni
cd14972 7tmA_EDG-like Ceratotherium_simum_cottoni
cd14971 7tmA_Galanin_R-like Ceratotherium_simum_cottoni
cd14978 7tmA_FMRFamide_R-like Ceratotherium_simum_cottoni
cd15307 7tmA_5-HT2_insect-like Ceratotherium_simum_cottoni
cd15411 7tmA_OR8H-like Ceratotherium_simum_cottoni
cd15413 7tmA_OR8K-like Ceratotherium_simum_cottoni
cd15429 7tmA_OR2F-like Ceratotherium_simum_cottoni
cd15947 7tmA_OR2B-like Ceratotherium_simum_cottoni
cd15416 7tmA_OR5P-like Ceratotherium_simum_cottoni
cd15407 7tmA_OR5B-like Ceratotherium_simum_cottoni
cd15915 7tmA_OR12D-like Ceratotherium_simum_cottoni
cd15415 7tmA_OR5J-like Ceratotherium_simum_cottoni
cd15405 7tmA_OR8B-like Ceratotherium_simum_cottoni
cd15942 7tmA_OR10G6-like Ceratotherium_simum_cottoni
cd15222 7tmA_OR51-like Ceratotherium_simum_cottoni
cd15956 7tmA_OR52W-like Ceratotherium_simum_cottoni
cd00637 7tm_classA_rhodopsin-like Ceratotherium_simum_cottoni
cd15067 7tmA_Dop1R2-like Ceratotherium_simum_cottoni
cd15208 7tmA_OXR Ceratotherium_simum_cottoni
cd15366 7tmA_GPR4 Ceratotherium_simum_cottoni
cd15234 7tmA_OR7-like Ceratotherium_simum_cottoni
cd13954 7tmA_OR Ceratotherium_simum_cottoni
cd15225 7tmA_OR10A-like Ceratotherium_simum_cottoni
cd15230 7tmA_OR5-like Ceratotherium_simum_cottoni
cd15236 7tmA_OR1E-like Ceratotherium_simum_cottoni
cd15911 7tmA_OR11A-like Ceratotherium_simum_cottoni
cd15231 7tmA_OR5V1-like Ceratotherium_simum_cottoni
cd15229 7tmA_OR8S1-like Ceratotherium_simum_cottoni
cd15227 7tmA_OR14-like Ceratotherium_simum_cottoni
cd15431 7tmA_OR13H-like Ceratotherium_simum_cottoni
cd15912 7tmA_OR6C-like Ceratotherium_simum_cottoni
cd15417 7tmA_OR5A1-like Ceratotherium_simum_cottoni
cd15421 7tmA_OR2T-like Ceratotherium_simum_cottoni
cd15232 7tmA_OR13-like Ceratotherium_simum_cottoni
cd15420 7tmA_OR2A-like Ceratotherium_simum_cottoni
cd15414 7tmA_OR5G-like Ceratotherium_simum_cottoni
cd15419 7tmA_OR9K2-like Ceratotherium_simum_cottoni
cd15409 7tmA_OR5H-like Ceratotherium_simum_cottoni
cd15224 7tmA_OR6B-like Ceratotherium_simum_cottoni
cd15233 7tmA_OR3A-like Ceratotherium_simum_cottoni
cd15424 7tmA_OR2_unk Ceratotherium_simum_cottoni
cd15913 7tmA_OR11G-like Ceratotherium_simum_cottoni
cd15946 7tmA_OR1330-like Ceratotherium_simum_cottoni
cd15228 7tmA_OR10D-like Ceratotherium_simum_cottoni
cd15914 7tmA_OR6N-like Ceratotherium_simum_cottoni
cd15916 7tmA_OR10G-like Ceratotherium_simum_cottoni
cd15428 7tmA_OR2D-like Ceratotherium_simum_cottoni
cd15226 7tmA_OR4-like Ceratotherium_simum_cottoni
cd15941 7tmA_OR10S1-like Ceratotherium_simum_cottoni
cd15940 7tmA_OR4E-like Ceratotherium_simum_cottoni
cd15936 7tmA_OR4D-like Ceratotherium_simum_cottoni
cd15223 7tmA_OR56-like Ceratotherium_simum_cottoni
cd15938 7tmA_OR4Q2-like Ceratotherium_simum_cottoni
cd15935 7tmA_OR4Q3-like Ceratotherium_simum_cottoni
cd15937 7tmA_OR4N-like Ceratotherium_simum_cottoni
cd14967 7tmA_amine_R-like Ceratotherium_simum_cottoni
cd14993 7tmA_CCKR-like Ceratotherium_simum_cottoni
cd15203 7tmA_NPYR-like Ceratotherium_simum_cottoni
cd14969 7tmA_Opsins_type2_animals Ceratotherium_simum_cottoni
cd15314 7tmA_TAAR1 Ceratotherium_simum_cottoni
cd15055 7tmA_TAARs Ceratotherium_simum_cottoni
cd15062 7tmA_alpha1_AR Ceratotherium_simum_cottoni
cd15202 7tmA_TACR-like Ceratotherium_simum_cottoni
cd15079 7tmA_photoreceptors_insect Ceratotherium_simum_cottoni
cd15325 7tmA_alpha1A_AR Ceratotherium_simum_cottoni
cd15012 7tmA_Trissin_R Ceratotherium_simum_cottoni
cd15061 7tmA_tyramine_R-like Ceratotherium_simum_cottoni
cd15213 7tmA_PSP24-like Ceratotherium_simum_cottoni
cd14982 7tmA_purinoceptor-like Ceratotherium_simum_cottoni
cd15918 7tmA_OR1_7-like Ceratotherium_simum_cottoni
cd15237 7tmA_OR2-like Ceratotherium_simum_cottoni
cd15412 7tmA_OR5M-like Ceratotherium_simum_cottoni
cd15434 7tmA_OR2W-like Ceratotherium_simum_cottoni
cd15430 7tmA_OR13-like Ceratotherium_simum_cottoni
cd15432 7tmA_OR2B2-like Ceratotherium_simum_cottoni
cd15433 7tmA_OR2Y-like Ceratotherium_simum_cottoni
cd15939 7tmA_OR4A-like Ceratotherium_simum_cottoni
cd15083 7tmA_Melanopsin-like Ceratotherium_simum_cottoni
cd15235 7tmA_OR1A-like Ceratotherium_simum_cottoni
cd15418 7tmA_OR9G-like Ceratotherium_simum_cottoni
cd15406 7tmA_OR8D-like Ceratotherium_simum_cottoni
cd15410 7tmA_OR5D-like Ceratotherium_simum_cottoni
cd15408 7tmA_OR5AK3-like Ceratotherium_simum_cottoni
cd15945 7tmA_OR5C1-like Ceratotherium_simum_cottoni
cd15944 7tmA_OR5AR1-like Ceratotherium_simum_cottoni
cd15943 7tmA_OR5AP2-like Ceratotherium_simum_cottoni
cd15949 7tmA_OR52M-like Ceratotherium_simum_cottoni
cd15064 7tmA_5-HT1_5_7 Ceratotherium_simum_cottoni
cd15134 7tmA_capaR Ceratotherium_simum_cottoni
cd15196 7tmA_Vasopressin_Oxytocin Ceratotherium_simum_cottoni
pfam00129 MHC_I Equus_caballus
cd16001 7tmA_P2Y3-like Chiloscyllium_punctatum
cd15393 7tmA_leucokinin-like Chiloscyllium_punctatum
cd15392 7tmA_PR4-like Chiloscyllium_punctatum
cd15168 7tmA_P2Y1-like Chiloscyllium_punctatum
cd14979 7tmA_NTSR-like Chiloscyllium_punctatum
cd15928 7tmA_GHSR-like Chiloscyllium_punctatum
cd15096 7tmA_AstA_R_insect Chiloscyllium_punctatum
cd15206 7tmA_CCK_R Chiloscyllium_punctatum
cd15919 7tmA_GPR139 Chiloscyllium_punctatum
================================================
FILE: scripts/TEstrainer/scripts/Dfam_extractor.py
================================================
#!/usr/bin/env python
import os
from re import sub
from os.path import exists
import sys
import argparse
from Bio import SeqIO
parser = argparse.ArgumentParser()
parser.add_argument('-l', '--library', type=str, required=True,
help='Input library fasta')
parser.add_argument('-d', '--directory', type=str, required=True,
help='Path to working data directory')
args = parser.parse_args()
# check file/folder exists
if(exists(args.library) == False):
sys.exit('Input fasta not found')
if(exists(args.directory) == False):
os.mkdir(args.out_dir)
# name of out file
out_library=args.directory+'/'+sub('.*/', '', args.library)
nondfam_library=args.directory+'/non-dfam.'+sub('.*/', '', args.library)
# write Dfam pseudo-curated sequences to out_library
# sequence headers beginning with DR and having digits between "DR" and "#" treated as as Dfam
with open(out_library, 'w') as out_handle:
with open(args.library, 'r') as in_handle:
for record in SeqIO.parse(in_handle, "fasta"):
if record.name.startswith("DR"):
if sub('DR', '', sub('#.*', '', record.name)).isdigit():
SeqIO.write(record, out_handle, "fasta")
# write non-Dfam sequences to out_library
# sequence headers beginning with DR and having digits between "DR" and "#" treated as as Dfam
with open(nondfam_library, 'w') as out_handle:
with open(args.library, 'r') as in_handle:
for record in SeqIO.parse(in_handle, "fasta"):
if record.name.startswith("DR") is False:
SeqIO.write(record, out_handle, "fasta")
elif sub('DR', '', sub('#.*', '', record.name)).isdigit() is False:
SeqIO.write(record, out_handle, "fasta")
================================================
FILE: scripts/TEstrainer/scripts/TEtrim.py
================================================
import os
import sys
import string
import statistics
import argparse
import re
import Bio
from Bio import SeqIO, AlignIO
from Bio.Align import AlignInfo, MultipleSeqAlignment
from Bio.Seq import Seq
from Bio.SeqRecord import SeqRecord
import pandas as pd
parser = argparse.ArgumentParser()
parser.add_argument('-i', '--in_seq', type=str, required=True,
help='Input MSA to be trimmed')
parser.add_argument('-n', '--iteration', type=str, required=True,
help='iteration number')
parser.add_argument('-d', '--directory', type=str, required=True,
help='Output directory')
parser.add_argument('-t', '--threads', type=str,
help='Threads to use', default=1)
parser.add_argument('-f', '--flank', type=int,
help='Length of flanks used', default=1000)
parser.add_argument('-D', '--debug', type=str,
help='Print debug messages', default='FALSE')
parser.add_argument('-w', '--window', type=int,
help='Window size for blocks', default=5)
parser.add_argument('-m', '--minimum_seq', type=int,
help='Minimum sequences required for MSA', default=3)
args = parser.parse_args()
# function for removing single base pair insertions
def single_trim(aln_in):
# make empty alignment
good=aln_in[:,0:0]
for x in range(aln_in.get_alignment_length()):
# extract columns with more than 1 base pair
if len(aln_in) - aln_in[:, x].count("-") > 1:
good=good+aln_in[:,x:x+1]
return(good)
# function for making first consensus from alignment
def con_maker(aln_in):
con_seq=str()
for x in range(aln_in.get_alignment_length()):
a = aln_in[:, x].count("a") + aln_in[:, x].count("A")
t = aln_in[:, x].count("t") + aln_in[:, x].count("T")
c = aln_in[:, x].count("c") + aln_in[:, x].count("C")
g = aln_in[:, x].count("g") + aln_in[:, x].count("G")
gap = aln_in[:, x].count("-")
if (len(aln_in) - gap) > 2:
pos_count = {'A': a, 'T': t, 'C': c, 'G': g}
if len([k for k, v in pos_count.items() if v == max(pos_count.values())])>1:
con_seq+='n'
else:
con_seq+=max(pos_count, key=pos_count.get)
con_seq=Seq(con_seq)
return(con_seq)
# set names
seq_name=re.sub('.*/', '', args.in_seq)
in_seq_path=args.directory+'/run_'+args.iteration+'/mafft/'+seq_name
out_seq_path=args.directory+'/run_'+args.iteration+'/TEtrim/'+seq_name
# read sequence
if args.debug == 'TRUE':
print('Reading og alignment')
align = AlignIO.read(in_seq_path, "fasta")
# get name of first sequence (the consensus)
final_id = align[0].id
if args.debug == 'TRUE':
print(final_id)
if(float(Bio.__version__) >=1.79):
og_con = SeqRecord(seq= align[0].seq.replace("-", ""), id=align[0].id, name=align[0].id)
else:
og_con = SeqRecord(seq= align[0].seq.ungap("-"), id=align[0].id, name=align[0].id)
SeqIO.write(og_con, (args.directory+'/run_'+args.iteration+'/TEtrim_con/og_'+seq_name),"fasta")
align = align[1:len(align)]
# cancel if less than minimum_seq sequences, write to file for troubleshooting
if args.debug == 'TRUE':
print('Reading determining number of sequences '+str(len(align)))
if(len(align)=1.79):
record.seq = record.seq.replace("-", "")
else:
record.seq = record.seq.ungap("-")
SeqIO.write(record, o, "fasta-2line")
### run blast ###
if args.debug == 'TRUE':
print('Determining number of sequences')
os.system('blastn -query '+args.directory+'/run_'+args.iteration+'/TEtrim_con/og_'+seq_name+' -subject '+args.directory+'/run_'+args.iteration+'/TEtrim_unaln/temp_'+seq_name+' -outfmt "6 qseqid sseqid qcovs" -task blastn | uniq > '+args.directory+'/run_'+args.iteration+'/TEtrim_blast/'+seq_name+'.tsv')
# read initial blast, determine acceptable passed on coverage >50% mean of coverage
if args.debug == 'TRUE':
print('Reading initial blast')
df = pd.read_table((args.directory+'/run_'+args.iteration+'/TEtrim_blast/'+seq_name+'.tsv'), names=['qseqid', 'sseqid', 'qcovs'])
if df.empty is True:
SeqIO.write(og_con, (args.directory+'/run_'+args.iteration+'/TEtrim_complete/'+seq_name),"fasta")
if args.debug == 'TRUE':
sys.exit(('New consensus of '+seq_name+" has no homology to original sequence"))
else:
sys.exit()
mean_covs=(statistics.mean(df.qcovs)/2)
acceptable=list(df.query("(qcovs>@mean_covs) or (qcovs>50)")['sseqid'])
# Check at least minimum_seq
if len(acceptable) < args.minimum_seq:
SeqIO.write(og_con, (args.directory+'/run_'+args.iteration+'/TEtrim_complete/'+seq_name),"fasta")
if args.debug == 'TRUE':
sys.exit((seq_name+" contains less than "+args.minimum_seq+" acceptable sequences"))
else:
sys.exit()
# create unaligned fasta of acceptables
if args.debug == 'TRUE':
print('Creating list of unaligned acceptable sequences')
with open((args.directory+'/run_'+args.iteration+'/TEtrim_unaln/unaln_'+seq_name), "w") as o:
for record in SeqIO.parse((args.directory+'/run_'+args.iteration+'/TEtrim_unaln/temp_'+seq_name), "fasta-2line"):
if record.id in acceptable:
SeqIO.write(record, o, "fasta-2line")
### run mafft ###
if args.debug == 'TRUE':
print('Running initial mafft')
os.system('mafft --quiet --thread '+args.threads+' --localpair '+args.directory+'/run_'+args.iteration+'/TEtrim_unaln/unaln_'+seq_name+' > '+args.directory+'/run_'+args.iteration+'/TEtrim_mafft/mafft_'+seq_name)
# read new alignment, make consensus
if args.debug == 'TRUE':
print('Reading new alignment and making consensus')
align_2=AlignIO.read((args.directory+'/run_'+args.iteration+'/TEtrim_mafft/mafft_'+seq_name), "fasta")
con_2=SeqRecord(con_maker(align_2),id=final_id,description=final_id)
SeqIO.write(con_2, (args.directory+'/run_'+args.iteration+'/TEtrim_con/cleaned_'+seq_name),"fasta")
# Run final blast
if args.debug == 'TRUE':
print('Final blast')
os.system('blastn -query '+args.directory+'/run_'+args.iteration+'/TEtrim_con/og_'+seq_name+' -subject '+args.directory+'/run_'+args.iteration+'/TEtrim_con/cleaned_'+seq_name+' -outfmt "6 length qlen slen qcovs" -task dc-megablast -out '+args.directory+'/run_'+args.iteration+'/TEtrim_blast/check_'+seq_name+'.tsv')
if os.path.getsize(args.directory+'/run_'+args.iteration+'/TEtrim_blast/check_'+seq_name+'.tsv') == 0:
SeqIO.write(og_con, (args.directory+'/run_'+args.iteration+'/TEtrim_complete/'+seq_name),"fasta")
if args.debug == 'TRUE':
sys.exit(('New consensus of '+seq_name+" has no homology to original sequence"))
else:
sys.exit()
# read in first line of blast
df2 = pd.read_table((args.directory+'/run_'+args.iteration+'/TEtrim_blast/check_'+seq_name+'.tsv'), names=['length', 'qlen', 'slen', 'qcovs']).head(1)
# determine values from blast table
length=df2.values[0,0]
qlen=df2.values[0,1]
slen=df2.values[0,2]
qcovs=df2.values[0,3]
if qcovs <= 80:
SeqIO.write(og_con, (args.directory+'/run_'+args.iteration+'/TEtrim_complete/'+seq_name),"fasta")
if args.debug == 'TRUE':
sys.exit(('New consensus of '+seq_name+" covers < 80% than original sequence"))
elif (slen <= (0.9 * qlen)):
SeqIO.write(og_con, (args.directory+'/run_'+args.iteration+'/TEtrim_complete/'+seq_name),"fasta")
if args.debug == 'TRUE':
sys.exit(('New consensus of '+seq_name+" is shorter than original sequence"))
elif ((slen - qlen) <= (0.5 * args.flank)):
SeqIO.write(con_2, (args.directory+'/run_'+args.iteration+'/TEtrim_complete/'+seq_name),"fasta")
if args.debug == 'TRUE':
sys.exit(('New consensus of '+seq_name+" is good, no further work needed"))
else:
SeqIO.write(con_2, (args.directory+'/run_'+args.iteration+'/TEtrim_further/'+seq_name),"fasta")
if args.debug == 'TRUE':
sys.exit(('New consensus of '+seq_name+" is good, needs further extension."))
================================================
FILE: scripts/TEstrainer/scripts/domain_plotter.R
================================================
library(tidyverse)
in_seq <- "seq/Chiloscyllium_punctatum-families.fa"
data_dir <- "TS_Chiloscyllium_punctatum-families.fa_5438/"
seq_name <- sub(".*\\/", "", in_seq)
og_seq<-Biostrings::readDNAStringSet(in_seq)
og_tibble <- tibble(seqnames = names(og_seq), width = width(og_seq)) %>%
mutate(raw_names = sub("#.*", "", seqnames))
chimeric_seq<-Biostrings::readDNAStringSet(paste0(data_dir, "/chimeras/chimeric_", seq_name))
unacceptable_seq <-Biostrings::readDNAStringSet(paste0(data_dir, "/chimeras/questionable_", seq_name))
chimeric_seq <- c(chimeric_seq, unacceptable_seq)
chimeric_tibble <- tibble(seqnames = names(chimeric_seq), width = width(chimeric_seq)) %>%
mutate(raw_names = sub("#.*", "", seqnames))
rps_blast_out <- read_tsv(file = paste0(data_dir, "chimeras/", seq_name, ".rps.out"),
col_names = c("seqnames", "qstart", "qend", "qlen", "sseqid", "sstart", "send", "slen",
"pident", "length", "mismatch", "gapopen", "evalue", "bitscore", "qcovs", "stitle"),
show_col_types = FALSE) %>%
tidyr::separate(stitle, into = c("ref", "abbrev", "full"), sep = ", ", extra = "drop") %>%
dplyr::filter(length >= 0.5*slen)
acceptable_domains <- rbind(read_tsv("data/acceptable_domains.tsv"), read_tsv("data/additional_domains.tsv", col_select = c(ref, abbrev)))
unacceptable_domains <- rbind(read_tsv("data/unacceptable_domains.tsv", col_select = c(ref, abbrev)))
chimeric_tibble <- rps_blast_out[sub("#.*", "", rps_blast_out$seqnames) %in% c(chimeric_tibble$raw_names),]
unacceptable <- chimeric_tibble[!chimeric_tibble$ref %in% acceptable_domains$ref,]$seqnames
chimeric_tibble <- rps_blast_out[sub("#.*", "", rps_blast_out$seqnames) %in% sub("#.*", "", unacceptable),]
chimeric_tibble <- chimeric_tibble %>%
mutate(missing = ifelse(ref %in% acceptable_domains$ref, FALSE, TRUE))
chimeric_names <- base::unique(chimeric_tibble$seqnames)
chimeric_names
i=4
to_plot <- chimeric_tibble[chimeric_tibble$seqnames == chimeric_names[i],] %>% arrange(qstart)
to_plot$y = 1:nrow(to_plot)
to_plot$text_x = (to_plot$qstart + to_plot$qend)/2
ggplot(to_plot) +
geom_rect(aes(xmin = qstart, xmax = qend, ymin = y-1, ymax = y, fill = missing)) +
geom_text(aes(x = text_x, y = y-0.5, label = abbrev)) +
scale_x_continuous(limits = c(0,to_plot$qlen[1])) +
ggtitle(to_plot$seqnames[1])
to_plot %>%
filter(!ref %in% unacceptable_domains$ref, !ref %in% acceptable_domains$ref) %>%
dplyr::select(ref, abbrev) %>%
base::unique() %>%
mutate(organism = "Chiloscyllium_punctatum") %>%
write_tsv("new_unacceptable_domains.tsv")
================================================
FILE: scripts/TEstrainer/scripts/final_sorter.R
================================================
#!/usr/bin/Rscript
library(optparse)
option_list <- list(
make_option(c("-i", "--in_seq"), default=NA, type = "character", help="Input sequence (required)")
)
opt <- parse_args(OptionParser(option_list=option_list))
if(is.na(opt$in_seq)){
stop("Path to in sequence must be supplied")
}
# make empty variable function
suppressPackageStartupMessages(library(tidyverse))
suppressPackageStartupMessages(library(plyranges))
suppressPackageStartupMessages(library(BSgenome))
# read in sequence
in_seq <- readDNAStringSet(opt$in_seq)
# sort by package used, round number and family number
sorted_ranges <-
tibble(seqnames = names(in_seq), start = 1, end = width(in_seq)) %>%
dplyr::mutate(pkg = substr(seqnames, 1, 3),
rnd = as.integer(sub(".*-", "", sub("_family.*", "", seqnames))),
family = as.integer(sub("#.*", "", sub(".*_family-", "", seqnames)))) %>%
dplyr::arrange(pkg, rnd, family) %>%
plyranges::as_granges()
# Get sequence
out_seq = getSeq(in_seq, sorted_ranges)
names(out_seq) <- seqnames(sorted_ranges)
# Write to file
writeXStringSet(out_seq, opt$in_seq)
================================================
FILE: scripts/TEstrainer/scripts/indexer.py
================================================
#!/usr/bin/env python
import argparse
from pyfaidx import Faidx
parser = argparse.ArgumentParser()
parser.add_argument('-g', '--genome', type=str, required=True,
help='Path to genome sequence')
args = parser.parse_args()
Faidx(args.genome)
================================================
FILE: scripts/TEstrainer/scripts/initial_domain_finder.R
================================================
#!/usr/bin/Rscript
library(ORFik)
library(tidyverse)
library(plyranges)
library(BSgenome)
# read in sequences
repbase_dna <- Biostrings::readDNAStringSet("~/Databases/Repbase/RepBase5May2021.fasta")
dfam_dna <- Biostrings::readDNAStringSet("seq/Dfam/Dfam_3.6_curatedonly.fasta")
# get info about sequences
repbase_seq_info <- tibble(seqnames = names(repbase_dna), width = width(repbase_dna))
repbase_seq_info <- repbase_seq_info %>%
tidyr::separate(col = seqnames, into = c("seqnames", "class", "species"), sep = "\t") %>%
dplyr::select(-species) %>%
filter(!class %in% c("Unknown", "Satellite","Simple Repeat"))
# convert dfam classifications to Repbase
dfam_seq_info <- tibble(seqnames = names(dfam_dna), width = width(dfam_dna)) %>%
tidyr::separate(col = seqnames, into = c("seqnames", "class"), sep = "#") %>%
mutate(class = ifelse(grepl("ERV", class), "ERV", class)) %>%
dplyr::mutate(class = case_when(
grepl("SINE", class) ~ "SINE",
grepl("/", class) ~ sub(".*/", "", class),
TRUE ~ class)
) %>%
mutate(
class = ifelse(class %in% c("TcMar", "TcMar-ISRm11", "TcMar-m44", "TcMar-Pogo", "TcMar-Fot1", "TcMar-Tc2", "TcMar-Mariner", "TcMar-Tc1", "TcMar-Tc4", "TcMar-Tigger"), "Mariner/Tc1", class),
class = ifelse(class %in% c("hAT-hAT5", "hAT-hAT19", "hAT-hATx", "hAT-hATm", "hAT-Tag1", "hAT-Ac", "hAT-Blackjack", "hAT-Tip100", "hAT-Charlie"), "hAT", class),
class = ifelse(class %in% c("Pao"), "BEL", class),
class = ifelse(class %in% c("CMC-Transib"), "Transib", class),
class = ifelse(class %in% c("PiggyBac"), "piggyBac", class),
class = ifelse(class %in% c("Naiad", "Chlamys"), "Penelope", class),
class = ifelse(class %in% c("CMC-Chapaev", "CMC-Chapaev-3", "CMC-EnSpm"), "EnSpm/CACTA", class),
class = ifelse(class %in% c("Academ-1"), "Academ", class),
class = ifelse(class %in% c("Ngaro"), "DIRS", class),
class = ifelse(class %in% c("PIF-ISL2EU"), "ISL2EU", class),
class = ifelse(class %in% c("R2-Hero"), "Hero", class),
class = ifelse(class %in% c("CR1-Zenon"), "CR1", class),
class = ifelse(class %in% c("endogenous retrovirus", "Endogenous Retrovirus", "ERV-1_PM-I", "ERV-2_PM-I"), "ERV", class),
class = ifelse(class %in% c("LTR retrotransposon", "LTR Retrotransposon"), "LTR", class),
class = ifelse(class %in% c("RTE-RTE", "RTE-BovB"), "RTE", class)
) %>%
filter(!seqnames %in% repbase_seq_info$seqnames)
dfam_dna <- dfam_dna[sub("#.*", "", names(dfam_dna)) %in% dfam_seq_info$seqnames,]
repbase_dna <- c(repbase_dna, dfam_dna)
repbase_seq_info_class_n <- as_tibble(as.data.frame(table(repbase_seq_info$class))) %>%
dplyr::filter(Freq > 5) %>%
dplyr::mutate(Var1 = as.character(Var1)) %>%
dplyr::rename(class = Var1)
repbase_seq_info <- rbind(repbase_seq_info, dfam_seq_info) %>%
filter(!class %in% c("Unknown", "Satellite", "subtelomeric", "Y-chromosome", "centromeric", "tRNA", "snRNA", "rRNA", "tRNA", "Multicopy gene"),
class %in% repbase_seq_info_class_n$class)
repbase_seq_info$name = 1:nrow(repbase_seq_info)
repbase_dna <- repbase_dna[sub("#.*", "", sub("\t.*", "", names(repbase_dna))) %in% repbase_seq_info$seqnames,]
names(repbase_dna) <- sub("#.*", "", sub("\t.*", "", names(repbase_dna)))
writeXStringSet(repbase_dna, "predata/repbase_dfam_compiled.fasta")
#### RUN RPSTBLASTN AGAINST CDD ####
# read in repbase rpstblastn
repbase_rps_out <- readr::read_tsv(file = "predata/repbase_dfam_compiled.rps.out",
col_names = c("seqnames", "qstart", "qend", "qlen", "sseqid", "sstart", "send", "slen",
"pident", "length", "mismatch", "gapopen", "evalue", "bitscore", "qcovs", "stitle"),
show_col_types = F) %>%
dplyr::filter(length >= 0.5*slen, evalue <= 0.01)
missing <- repbase_rps_out %>% filter(!seqnames %in% repbase_seq_info$seqnames)
repbase_rps_out <- tidyr::separate(data = repbase_rps_out, col = stitle, into = c("ref", "abbrev", "full"), sep = ", ")
repbase_rps_out <- dplyr::inner_join(repbase_rps_out, repbase_seq_info) %>%
filter(class != "Multicopy gene")
domain_info <- repbase_rps_out %>% dplyr::select(ref, abbrev, full) %>% base::unique()
# get info about repbase classes
repbase_class_info <- dplyr::as_tibble(BiocGenerics::as.data.frame(table(repbase_seq_info$class)))
repbase_class_info <- dplyr::mutate(.data = repbase_class_info, Var1 = as.character(Var1))
repbase_class_info <- dplyr::rename(.data = repbase_class_info, class = Var1)
repbase_class_info_min <- repbase_class_info[repbase_class_info$Freq < 10,]
repbase_class_info_max <- repbase_class_info[repbase_class_info$Freq >= 10,]
repbase_class_info_max <- dplyr::arrange(.data = repbase_class_info_max, -Freq)
# loop over to get info
common_domains <- tibble()
for(i in 1:nrow(repbase_class_info_max)){
holder <- repbase_rps_out[repbase_rps_out$class == repbase_class_info_max$class[i],] %>%
group_by(seqnames, abbrev) %>%
arrange(-bitscore) %>%
dplyr::slice(1) %>% # ensure one representative for each domain for each repeat
ungroup()
if(nrow(holder >= 1)){
if(repbase_class_info_max$Freq[i] > 299){
holder_tbl <- as_tibble(as.data.frame(table(holder$ref))) %>%
mutate(Var1 = as.character(Var1),
perc = Freq/repbase_class_info_max$Freq[i]) %>%
dplyr::rename(ref = Var1) %>%
filter(perc >= 0.01, Freq >= 3) %>%
inner_join(domain_info) %>%
mutate(class = repbase_class_info_max$class[i]) %>%
dplyr::arrange(-Freq)
} else {
holder_tbl <- as_tibble(as.data.frame(table(holder$ref))) %>%
mutate(Var1 = as.character(Var1),
perc = Freq/repbase_class_info_max$Freq[i]) %>%
dplyr::rename(ref = Var1) %>%
filter(Freq >= 3) %>%
inner_join(domain_info) %>%
mutate(class = repbase_class_info_max$class[i]) %>%
dplyr::arrange(-Freq)
}
common_domains <- rbind(common_domains, holder_tbl)
}
}
# write to file
base::unique(common_domains %>% select(ref, abbrev)) %>%
dplyr::filter(!startsWith(tolower(abbrev), "ig"),
!startsWith(tolower(abbrev), "znmc"),
abbrev != "V-set") %>%
write_tsv("data/acceptable_domains.tsv")
base::unique(common_domains %>% select(class, ref, abbrev)) %>%
dplyr::filter(startsWith(tolower(abbrev), "ig") | startsWith(tolower(abbrev), "znmc") | abbrev == "V-set") %>%
write_tsv("data/exceptional_domains.tsv")
================================================
FILE: scripts/TEstrainer/scripts/initial_mafft_setup.py
================================================
#!/usr/bin/env python
import argparse
import sys
import os
from os.path import exists
parser = argparse.ArgumentParser()
parser.add_argument('-d', '--directory', type=str, required=True,
help='Directory housing TEstrainer run')
parser.add_argument('-r', '--iteration', type=str, required=True,
help='Interation number of TEstrainer curation')
parser.add_argument('-s', '--seq_name', type=str, required=True,
help='Sequence being prepared for alignment')
parser.add_argument('-g', '--genome', type=str, required=True,
help='Path to genome sequence')
parser.add_argument('-f', '--flank', type=int, default=1000,
help='Length of flank to be extended')
parser.add_argument('-n', '--no_seq', type=int, default=20,
help='Number of sequences to use for alignment')
parser.add_argument('-D', '--debug', action='store_true',
help='Set for full messaging')
args = parser.parse_args()
# function to check files exist
def file_check(file_name, debug):
if(exists(file_name) == False):
if(debug == False):
exit()
else:
sys.exit((file_name+" not found"))
# check input files/folders exist
file_check((args.directory+"/run_"+args.iteration+"/raw/"+args.seq_name), args.debug) # current consensus
file_check((args.directory+"/run_0/og/"+args.seq_name), args.debug) # og consensus
file_check(args.genome, args.debug) # genome sequence
file_check((args.directory+'/run_'+args.iteration+'/self_search/'), args.debug) # self search folder
file_check((args.directory+'/run_'+args.iteration+'/to_align/'), args.debug) # final output folder
file_check((args.directory+'/run_'+args.iteration+'/TEtrim_complete/'), args.debug) # alternate final output folder
import string
from os import system
import statistics
import re
from Bio import SeqIO
from Bio.Seq import Seq
from Bio.SeqRecord import SeqRecord
import pandas as pd
import numpy as np
import pyranges as pr
# function to output current consensus and exit if less than a certain number of sequences are found
def size_check(var_name, size):
if(len(var_name) < size):
with open((args.directory+'/run_'+args.iteration+'/TEtrim_complete/'+args.seq_name), "w") as o:
SeqIO.write(start_seq, o, "fasta-2line")
exit()
# Function for writing sequence from df to file
def df_to_fasta(df, path, name, overwrite):
df = df.astype('str')
if('Chromosome' not in df.columns) or ('Start' not in df.columns) or ('End' not in df.columns) or ('Seq' not in df.columns):
sys.exit(status="Necessary sequence columns not found in df")
# Name sequences
if('Name' not in df.columns) or (name is True):
if('Strand' in df.columns):
df['Name']=(df.Chromosome+":"+df.Start+"-"+df.End+"("+df.Strand+")")
else:
df['Name']=(df.Chromosome+":"+df.Start+"-"+df.End)
# Write sequences
if(overwrite is True):
with open((path), "w") as o:
for x in range(len(df)):
SeqIO.write(SeqRecord(seq=Seq(df.Seq[x]),id=df.Name[x],name=df.Name[x], description=""), o, "fasta-2line")
else:
with open((path), "a") as o:
for x in range(len(df)):
SeqIO.write(SeqRecord(seq=Seq(df.Seq[x]),id=df.Name[x],name=df.Name[x], description=""), o, "fasta-2line")
# Function for adjusting blast output for coordinates direction, convert to base 0
def blast_to_bed(df):
rev = df['Start'] > df['End']
fwd = df['Start'] < blast_df['End']
df.loc[rev, ['Start', 'End']] = (df.loc[rev, ['End', 'Start']].values)
df.loc[fwd, ['Start']] = df.loc[fwd, ['Start']] - 1
df.loc[rev, ['End']] = df.loc[rev, ['End']] +1
return(df)
# read in starting seq
start_seq = SeqIO.read((args.directory+"/run_"+args.iteration+"/raw/"+args.seq_name), "fasta")
# check if sequence is largely tandem repeat
# make pd.df to use as holder for trf coordinates
te_len=len(start_seq.seq)
trf_df = pd.DataFrame(columns=['Chromosome', 'Start', 'End'])
with open((args.directory+'/run_'+args.iteration+'/raw/'+args.seq_name+'.trf'), 'r') as trf:
for line in trf:
if line.startswith('@'):
seqnames = re.sub('@', '', line.split()[0])
elif float(line.split()[3]) > 5:
trf_df = pd.concat([trf_df, pd.DataFrame({'Chromosome':[seqnames], 'Start':[int(line.split()[0])], 'End':[int(line.split()[1])]})])
# Merge trf pr
trf_pr=pr.PyRanges(df=trf_df).merge()
# Calculate total length of satellite sequences
# TEs >90% consider satellites
if trf_pr.length/te_len > 0.9:
with open((args.directory+'/run_'+args.iteration+'/TEtrim_complete/'+args.seq_name), "w") as o:
if args.debug is True:
print('Hold your horses, '+start_seq.name+' is likely a tandem repeat')
SeqIO.write(start_seq, o, "fasta-2line")
exit()
# TEs >50% consider TEs needing trimming
elif (trf_pr.length/te_len > 0.5) & (len(start_seq.seq) > 500):
if args.debug is True:
print('Whoa boy, '+start_seq.name+' is inside tandem repeat')
# make pr object of whole repeat
te_pr=pr.from_dict({'Chromosome': [seqnames], 'Start': [0], 'End': [te_len]})
# get inverse or trf, i.e. non tandem repeat section
trf_inverse_pr=te_pr.subtract(trf_pr)
# convert to df
trf_inverse_df=trf_inverse_pr.as_df()
# calculate width
trf_inverse_df['Width']=trf_inverse_df.End-trf_inverse_df.Start+1
# select longest non TR section and convert to ranges
non_trf_longest_df=trf_inverse_df[trf_inverse_df.Width==max(trf_inverse_df.Width)].head(n=1)
# handle potential empty DataFrame to avoid IndexError on iloc[0]
if non_trf_longest_df.empty:
if args.debug is True:
print('No non-tandem repeat section found for '+start_seq.name+'; writing original sequence.')
with open((args.directory+'/run_'+args.iteration+'/TEtrim_complete/'+args.seq_name), "w") as o:
SeqIO.write(start_seq, o, "fasta-2line")
exit()
non_trf_longest_pr=pr.PyRanges(df=non_trf_longest_df)
# get sequence of non TR section, write to file
non_trf_seq=SeqRecord(seq=Seq(str(start_seq.seq)[int(non_trf_longest_df['Start'].iloc[0]):int(non_trf_longest_df['End'].iloc[0])]), id=start_seq.id, name=start_seq.name)
with open((args.directory+'/run_'+args.iteration+'/TEtrim_complete/'+args.seq_name), "w") as o:
SeqIO.write(non_trf_seq, o, "fasta-2line")
exit()
# perform initial blast
system("blastn -task dc-megablast -query "+args.directory+"/run_"+args.iteration+"/raw/"+args.seq_name+" -db "+args.genome+" -evalue 1e-5 -outfmt \"6 qseqid sseqid pident length qstart qend qlen sstart send slen evalue bitscore qcovs\" -out "+args.directory+"/run_"+args.iteration+"/initial_blast/"+args.seq_name+".out -num_threads 1")
# check if any hits found, if not write to file and exit
if os.path.getsize(args.directory+'/run_'+args.iteration+'/initial_blast/'+args.seq_name+'.out') == 0:
with open((args.directory+'/run_'+args.iteration+'/TEtrim_complete/'+args.seq_name), "w") as o:
SeqIO.write(start_seq, o, "fasta-2line")
sys.exit()
# read in blast table and filter
blast_df = pd.read_table((args.directory+'/run_'+args.iteration+'/initial_blast/'+args.seq_name+'.out'), names=['qseqid', 'Chromosome', 'pident', 'length', 'qstart', 'qend', 'qlen', 'Start', 'End', 'slen', 'evalue', 'bitscore', 'qcovs'])
blast_df = blast_df.query('pident >= 70 & qcovs >= 50').copy()
size_check(blast_df, 3)
# adjust for coordinates direction, convert to base 0
blast_to_bed(blast_df)
# select 50 best, add flanks and convert to ranges
blast_df = blast_df.sort_values(by = 'bitscore', ascending=False)
blast_df = blast_df.iloc[:args.no_seq]
blast_df['Start'] = blast_df['Start'] - args.flank
blast_df['End'] = blast_df['End'] + args.flank
blast_df.loc[blast_df['Start'] < 1, 'Start'] = 0
blast_df.loc[blast_df['End'] > blast_df['slen'], 'End'] = blast_df['slen']
blast_df = blast_df.sort_values(by = ['Chromosome', 'Start'])
blast_gr = pr.from_dict({"Chromosome": blast_df.Chromosome, "Start": blast_df.Start, "End": blast_df.End, "Bitscore": blast_df.bitscore, "slen" : blast_df.slen})
# reduce/merge ranges
best_hits_df = blast_gr.cluster(strand=False).df.groupby(['Cluster']).agg({'Chromosome':'first', 'Start':'min', 'End':'max', 'Bitscore':'max'})[['Chromosome','Start','End','Bitscore']].reset_index()
best_hits_py = pr.from_dict({"Chromosome": best_hits_df.Chromosome, "Start": best_hits_df.Start, "End": best_hits_df.End, "Bitscore": best_hits_df.Bitscore})
# get sequence
best_hits_seq = pr.get_fasta(best_hits_py, path=args.genome)
# convert df to strings
best_hits_df = best_hits_df.astype('str')
best_hits_df['Name'] = (best_hits_df.Chromosome+":"+best_hits_df.Start+"-"+best_hits_df.End+"#"+best_hits_df.Bitscore)
best_hits_df['Seq'] = best_hits_seq
best_hits_outpath = (args.directory+'/run_'+args.iteration+'/self_search/'+args.seq_name+'_check_1')
# write sequences to file
df_to_fasta(best_hits_df, best_hits_outpath, False, True)
# blast against og sequence,read in and filter
system("blastn -task dc-megablast -query "+args.directory+'/run_0/og/'+args.seq_name+" -subject "+args.directory+"/run_"+args.iteration+"/self_search/"+args.seq_name+"_check_1 -outfmt \"6 sseqid pident qcovs\" -out "+args.directory+"/run_"+args.iteration+"/self_search/"+args.seq_name+"_check_1.out")
check_df = pd.read_table((args.directory+"/run_"+args.iteration+"/self_search/"+args.seq_name+"_check_1.out"), names=['Chromosome', 'pident', 'qcovs'])
check_df = check_df.query('pident >= 70 & qcovs >= 50')
size_check(check_df, 3)
# select one instance of each sequence
check_df = check_df.groupby(['Chromosome']).head(n=1).reset_index()
check_df[['Chromosome', 'bitscore']] = check_df['Chromosome'].str.split('#', n=1, expand = True)
best_hits_df[['Name', 'bitscore']] = best_hits_df['Name'].str.split('#', n=1, expand = True)
size_check(check_df, 3)
# sort by bitscore and select no_seq (default 20) highest bitscores
check_df = check_df.astype({'bitscore': 'float'})
check_df = check_df.sort_values(by = 'bitscore', ascending=False)
# select accurate and write ready for self blast
correct_df = best_hits_df.loc[best_hits_df.Name.isin(check_df.Chromosome)].reset_index()
correct_df_path = (args.directory+'/run_'+args.iteration+'/self_search/'+args.seq_name+'_check_2')
df_to_fasta(correct_df, correct_df_path, False, True)
# self blast
system("blastn -task dc-megablast -query "+correct_df_path+" -subject "+correct_df_path+" -outfmt \"6 qseqid sseqid length pident qstart qend bitscore\" -out "+correct_df_path+".out")
self_blast_df = pd.read_table((args.directory+"/run_"+args.iteration+"/self_search/"+args.seq_name+"_check_2.out"), names=['Chromosome', 'sseqid', 'length', 'pident', 'Start', 'End', 'bitscore'])
size_check(self_blast_df, 3)
# fix coordinates
self_blast_df.Start = self_blast_df.Start - 1
# filter self hits, small hits
self_blast_df = self_blast_df[self_blast_df.Chromosome != self_blast_df.sseqid].copy()
self_blast_df = self_blast_df[self_blast_df.length >= (0.5*len(start_seq))].copy()
size_check(self_blast_df, 3)
# calculate and filter quantiles
self_blast_df['q1'] = self_blast_df.groupby('Chromosome').Start.transform(lambda x: x.quantile(0.1))
self_blast_df['q9'] = self_blast_df.groupby('Chromosome').End.transform(lambda x: x.quantile(0.9))
self_blast_df = self_blast_df[(self_blast_df.Start.astype(float) >= self_blast_df.q1) & (self_blast_df.End.astype(float) <= self_blast_df.q9)].copy()
size_check(self_blast_df, 3)
# create and reduce/merge ranges
self_blast_gr = pr.from_dict({"Chromosome": self_blast_df.Chromosome, "Start": self_blast_df.Start, "End": self_blast_df.End})
self_hits_trimmed_df = self_blast_gr.cluster(strand=False).df.groupby(['Cluster']).agg({'Chromosome':'first', 'Start':'min', 'End':'max'})[['Chromosome','Start','End']].reset_index()
self_hits_trimmed_py = pr.from_dict({"Chromosome": self_hits_trimmed_df.Chromosome, "Start": self_hits_trimmed_df.Start, "End": self_hits_trimmed_df.End})
size_check(self_hits_trimmed_py, 3)
# get trimmed sequence
self_hits_trimmed_seq = pr.get_fasta(self_hits_trimmed_py, path=(args.directory+'/run_'+args.iteration+'/self_search/'+args.seq_name+'_check_2'))
self_hits_trimmed_df['Seq'] = self_hits_trimmed_seq
self_hits_trimmed_outpath = (args.directory+'/run_'+args.iteration+'/to_align/'+args.seq_name)
# write current consensus fasta to file
with open(self_hits_trimmed_outpath, "w") as o:
SeqIO.write(start_seq, o, "fasta-2line")
# write trimmed fasta to file
df_to_fasta(self_hits_trimmed_df, self_hits_trimmed_outpath, True, False)
================================================
FILE: scripts/TEstrainer/scripts/mreps_parser.sh
================================================
#!/bin/bash
# run mreps and parse to usable format
usage() { echo "Usage: [-i input sequence (fasta file)] [-h Print this help]" 1>&2; exit 1; }
while getopts i:h flag
do
case "${flag}" in
i) INFILE=${OPTARG};;
h | *)
usage
exit
esac
done
if [ ! -f "${INFILE}" ]; then
echo "Error: file not found"
exit
fi
mreps -minsize 20 -maxsize 50000 -minperiod 1 -maxperiod 5000 -exp 2 -fasta $INFILE | grep "\->" | grep -v "from" | sed 's/ \+/\t/g' | sed 's/\t\+/\t/g' | sed 's/\t//10g' | cut -f2,4,6,7,8,9,10 | sed 's/ ://g' | sed 's/>//g' | sed 's/ ${INFILE}.mreps.temp
awk '{OFS="\t"}{print FILENAME,$1,$2,$3,$4,$5,$6,$7}' ${INFILE}.mreps.temp | sed 's/.*\///;s/.fasta.mreps.temp//' > ${INFILE}.mreps
rm ${INFILE}.mreps.temp
================================================
FILE: scripts/TEstrainer/scripts/simple_repeat_filter_trim.R
================================================
library(optparse)
option_list <- list(
make_option(c("-i", "--in_seq"), default=NA, type = "character", help="Input sequence (required)"),
make_option(c("-d", "--directory"), type="character", default=NULL, help="Path to data directory (required)", metavar="character"),
make_option(c("-p", "--sat_perc"), type="integer", default=60, help="Minimum percentage of total sequence classed as SSR for repeat to be considered a satellite")
)
opt <- parse_args(OptionParser(option_list=option_list))
# check variables provided
if(is.na(opt$in_seq)){
stop("Path to in sequence must be supplied")
}
if(is.na(opt$directory)){
stop("Path to in sequence must be supplied")
}
opt$out_seq <- sub(".*/", "", opt$in_seq)
# make empty variable function
suppressPackageStartupMessages(library(tidyverse))
suppressPackageStartupMessages(library(plyranges))
suppressPackageStartupMessages(library(BSgenome))
# read in library and make bed like table
in_seq <- readDNAStringSet(opt$in_seq)
in_seq_tbl <- tibble(seqnames = names(in_seq), og_width = width(in_seq)) %>%
dplyr::mutate(draft_seqnames = sub("#.*", "", seqnames))
# If no simple repeats were found by any of the three tools exit early, creating necessary files
if(
nrow(
read_tsv(paste0(opt$directory, "/trf/", opt$out_seq, ".sassr"),
skip = 1, col_names = c("seqnames", "ssr", "count", "start"), show_col_types = F)) == 0 &
file.size(paste0(opt$directory, "/trf/", opt$out_seq, ".trf")) == 0 &
file.size(paste0(opt$directory, "/trf/", opt$out_seq, ".mreps")) == 0
){
writeXStringSet(in_seq, paste0(opt$directory, "/trf/", opt$out_seq, ".nonsatellite"))
message("No satellites found")
} else {
# read in, rearrange and calculate percent tandem repeats for SA-SSR data
# check sassr only header, if so create empty tbl, else analyse data
if(nrow(
read_tsv(paste0(opt$directory, "/trf/", opt$out_seq, ".sassr"),
skip = 1, col_names = c("seqnames", "ssr", "count", "start"), show_col_types = F)) == 0){
sassr_calc <- tibble(seqnames = character(), sassr_perc_tr = double())
sassr_select <- tibble(seqnames = character(), start = double(), end = double(),
period = double(), count = double(), og_width = integer(),
ssr = character(), ssr_width = double(), package = "sassr")
} else {
sassr <- read_tsv(paste0(opt$directory, "/trf/", opt$out_seq, ".sassr"),
skip = 1, col_names = c("seqnames", "ssr", "count", "start"), show_col_types = F, col_types = "ccdd") %>%
mutate_if(is.logical, ~gsub("RUE|ALSE", "", .)) %>%
dplyr::mutate(ssr = ifelse(is.na(ssr), "NA", ssr),
period = as.double(width(ssr))) %>%
dplyr::mutate(ssr_width = count*period, end = start + ssr_width, start = start +1) %>%
mutate(draft_seqnames = sub("#.*", "", seqnames)) %>%
inner_join(in_seq_tbl, by = "seqnames") %>%
arrange(seqnames) %>%
filter(count > 2)
sassr_calc <- as_tibble(GenomicRanges::reduce(as_granges(sassr))) %>%
dplyr::select(-strand) %>%
dplyr::mutate(seqnames = as.character(seqnames)) %>%
group_by(seqnames) %>%
mutate(total_width = sum(width)) %>%
ungroup() %>%
inner_join(in_seq_tbl, by = "seqnames") %>%
dplyr::mutate(sassr_perc_tr = 100*total_width/og_width) %>%
dplyr::select(seqnames, sassr_perc_tr) %>%
base::unique()
sassr_select <- sassr %>% select(seqnames, start, end, period, count, og_width, ssr, ssr_width) %>%
mutate(package = "sassr")
}
# check trf empty, if so create empty tbl, else analyse data
# read in, rearrange and calculate percent tandem repeats for TRF data
if(file.size(paste0(opt$directory, "/trf/", opt$out_seq, ".trf")) == 0){
trf_select <- tibble(seqnames = character(), start = double(), end = double(),
period = double(), count = double(), og_width = integer(),
ssr = character(), ssr_width = double(), package = "trf")
trf_calc <- tibble(seqnames = character(), trf_perc_tr = double())
} else {
trf <- read_tsv(paste0(opt$directory, "/trf/", opt$out_seq, ".trf"),
col_names = c("draft_seqnames", "start", "end", "period", "count", "ssr"), show_col_types = F, col_types = "cddddc") %>%
mutate_if(is.logical, ~gsub("RUE|ALSE", "", .)) %>%
mutate(ssr = ifelse(is.na(ssr), "NA", ssr),
draft_seqnames = sub("@", "", sub("#.*", "", draft_seqnames))) %>%
dplyr::mutate(ssr_width = end - start + 1) %>%
inner_join(in_seq_tbl, by = "draft_seqnames")
trf_select <- trf %>%
filter(count > 2) %>%
dplyr::select(seqnames, start, end)
trf_calc <- as_tibble(GenomicRanges::reduce(as_granges(trf_select))) %>%
dplyr::select(-strand) %>%
dplyr::mutate(seqnames = as.character(seqnames)) %>%
group_by(seqnames) %>%
mutate(total_width = sum(width)) %>%
ungroup() %>%
inner_join(in_seq_tbl, by = "seqnames") %>%
dplyr::mutate(trf_perc_tr = 100*total_width/og_width) %>%
dplyr::select(seqnames, trf_perc_tr) %>%
base::unique()
trf_select <- trf %>% select(seqnames, start, end, period, count, og_width, ssr, ssr_width) %>%
mutate(package = "trf")
}
# read in, rearrange and calculate percent tandem repeats for MREPS data
if(file.size(paste0(opt$directory, "/trf/", opt$out_seq, ".mreps")) == 0){
mreps_select <- tibble(seqnames = character(), start = double(), end = double(),
period = double(), count = double(), og_width = integer(),
ssr = character(), ssr_width = double(), package = "mreps")
mreps_calc <- tibble(seqnames = character(), mreps_perc_tr = double())
} else {
mreps <- read_tsv(paste0(opt$directory, "/trf/", opt$out_seq, ".mreps"),
col_names = c("draft_seqnames", "start", "end", "ssr_width", "period", "count", "error", "sequence"), show_col_types = F, col_types = "cdddddcc") %>%
mutate_if(is.logical, ~gsub("RUE|ALSE", "", .)) %>%
mutate(ssr = substr(x = sequence, start = 0, stop = period)) %>%
inner_join(in_seq_tbl, by = "draft_seqnames")
mreps_select <- mreps %>%
filter(count > 2) %>%
dplyr::select(seqnames, start, end)
mreps_calc <- as_tibble(GenomicRanges::reduce(as_granges(mreps_select))) %>%
dplyr::select(-strand) %>%
dplyr::mutate(seqnames = as.character(seqnames)) %>%
group_by(seqnames) %>%
mutate(total_width = sum(width)) %>%
ungroup() %>%
inner_join(in_seq_tbl, by = "seqnames") %>%
dplyr::mutate(mreps_perc_tr = 100*total_width/og_width) %>%
dplyr::select(seqnames, mreps_perc_tr) %>%
base::unique()
mreps_select <- mreps %>% select(seqnames, start, end, period, count, og_width, ssr, ssr_width) %>%
mutate(package = "mreps")
}
# Compile data
compiled_tr <- rbind(rbind(trf_select, mreps_select), sassr_select)
stats_tr <- full_join(sassr_calc, full_join(mreps_calc, trf_calc, by = "seqnames"), by = "seqnames")
# if no data write to og seq to file as nonsatellite and exit
if(nrow(compiled_tr) == 0){
writeXStringSet(in_seq, paste0(opt$directory, "/trf/", opt$out_seq, ".nonsatellite"))
stop()
}
# Identifying satellite/simple repeats
over50tr <- stats_tr %>%
filter(sassr_perc_tr >= opt$sat_perc | mreps_perc_tr >= opt$sat_perc | trf_perc_tr >= opt$sat_perc) %>%
mutate(sassr_perc_tr = ifelse(is.na(sassr_perc_tr), 0, sassr_perc_tr),
mreps_perc_tr = ifelse(is.na(mreps_perc_tr), 0, mreps_perc_tr),
trf_perc_tr = ifelse(is.na(trf_perc_tr), 0, trf_perc_tr))
satellites <- compiled_tr[compiled_tr$seqnames %in% over50tr$seqnames,] %>%
dplyr::group_by(seqnames) %>%
dplyr::arrange(-ssr_width, period) %>%
dplyr::slice(1) %>%
dplyr::ungroup() %>%
dplyr::mutate(perc_tr = 100*ssr_width/og_width) %>%
inner_join(over50tr, by = "seqnames") %>%
dplyr::mutate(max_perc_tr = pmax(sassr_perc_tr, mreps_perc_tr, trf_perc_tr)) %>%
dplyr::arrange(max_perc_tr)
# for macrosatellites trim to get single copy
macrosatellites <- satellites %>%
filter(max_perc_tr > 90, count >= 2) %>%
filter(period > 200) %>%
mutate(start = start + period,
end = start + period,
end = ifelse(end > og_width, og_width, end)) %>%
as_granges()
macrosatellites_seq <- getSeq(in_seq, macrosatellites)
names(macrosatellites_seq) <- sub("#.*", "#Satellite", seqnames(macrosatellites))
# leave micro to minisatellite repeats intact
other_satellites <- satellites %>%
dplyr::filter(!seqnames %in% seqnames(macrosatellites)) %>%
dplyr::mutate(start = 1, end = og_width) %>%
as_granges()
other_satellites_seq <- getSeq(in_seq, other_satellites)
names(other_satellites_seq) <- sub("#.*", "#Satellite", seqnames(other_satellites))
# Combine satellite repeats and write to file
writeXStringSet(c(other_satellites_seq, macrosatellites_seq), paste0(opt$directory, "/trf/", opt$out_seq, ".satellites"))
# filtering for trimming
trim_3 <- compiled_tr %>%
mutate(tr_len = period*count) %>%
filter(count >=3, !seqnames %in% over50tr$seqnames, period < 20) %>%
filter(og_width-end <= period) %>%
dplyr::group_by(seqnames) %>%
arrange(-end, period) %>%
dplyr::slice(1)
trim_5 <- compiled_tr %>%
mutate(tr_len = period*count) %>%
filter(count >=3, !seqnames %in% over50tr$seqnames, period < 20) %>%
filter(start <= period) %>%
dplyr::group_by(seqnames) %>%
arrange(-end, period) %>%
dplyr::slice(1)
# Determine thos to be trimmed both ends
trim_both_5 <- trim_5[trim_5$seqnames %in% trim_3$seqnames,] %>%
dplyr::mutate(start = ifelse(period < 4, end - 8, end - (2*period))) %>%
dplyr::mutate(start = ifelse(start < 1, 1, start)) %>%
dplyr::select(seqnames, start)
trim_both_3 <- trim_3[trim_3$seqnames %in% trim_5$seqnames,] %>%
dplyr::mutate(end = ifelse(period < 4, start + 8, start + (2*period))) %>%
dplyr::mutate(end = ifelse(end > og_width, og_width, end)) %>%
dplyr::select(seqnames, end)
trim_both <- inner_join(trim_both_5, trim_both_3, by = "seqnames")
# determine those to be trimmed one end
trim_only_5 <- trim_5[!trim_5$seqnames %in% trim_3$seqnames,] %>%
dplyr::mutate(start = ifelse(period < 4, end - 8, end - (2*period))) %>%
dplyr::mutate(start = ifelse(start < 1, 1, start), end = og_width) %>%
dplyr::select(seqnames, start, end)
trim_only_3 <- trim_3[!trim_3$seqnames %in% trim_5$seqnames,] %>%
dplyr::mutate(end = ifelse(period < 4, start + 8, start + (2*period))) %>%
dplyr::mutate(end = ifelse(end > og_width, og_width, end), start = 1) %>%
dplyr::select(seqnames, start, end)
to_trim <- as_granges(rbind(trim_both, rbind(trim_only_5, trim_only_3)))
trimmed_seq <- getSeq(in_seq, to_trim)
names(trimmed_seq) <- seqnames(to_trim)
# determine untouched sequences
untouched_seq <-
in_seq[!sub("#.*", "", names(in_seq)) %in% sub("#.*", "", c(names(trimmed_seq), names(macrosatellites_seq), names(other_satellites_seq))),]
# combine untouched and trimmed sequences and write to file
writeXStringSet(c(trimmed_seq, untouched_seq), paste0(opt$directory, "/trf/", opt$out_seq, ".nonsatellite"))
}
================================================
FILE: scripts/TEstrainer/scripts/splitter.py
================================================
#!/usr/bin/env python
import os
import re
from os.path import exists
import sys
import argparse
from Bio import SeqIO
parser = argparse.ArgumentParser()
parser.add_argument('-i', '--in_seq', type=str, required=True,
help='Input multi-fasta to be split')
parser.add_argument('-o', '--out_dir', type=str, required=True,
help='Output directory')
args = parser.parse_args()
if(exists(args.in_seq) == False):
sys.exit('File not found')
if(exists(args.out_dir) == False):
os.mkdir(args.out_dir)
file_list=[]
# split fasta file
with open(args.in_seq, 'r') as handle:
for record in SeqIO.parse(handle, "fasta"):
file_name = (args.out_dir+"/"+record.name.split(sep="#")[0]+".fasta")
file_list.append(record.name.split(sep="#")[0]+".fasta")
SeqIO.write(record, file_name, "fasta-2line")
# write file list
with open((args.out_dir+"/"+re.sub('.*/', '', args.in_seq)+"_split.txt"), 'w') as fp:
for item in file_list:
# write each item on a new line
fp.write("%s\n" % item)
================================================
FILE: scripts/TEstrainer/scripts/strainer.R
================================================
#!/usr/bin/Rscript
library(optparse)
option_list <- list(
make_option(c("-i", "--in_seq"), default=NA, type = "character", help="Path to input sequence (required)"),
make_option(c("-p", "--plot_chimeras"), type = "logical", help="Set to TRUE to make plots of chimeras", default = TRUE),
make_option(c("-d", "--directory"), default=NA, type = "character", help="Path to directory (required)",)
)
opt <- parse_args(OptionParser(option_list=option_list))
# check variables provided
if(is.na(opt$in_seq)){
stop("Path to in sequence must be supplied")
}
if(is.na(opt$directory)){
stop("Path to directory must be supplied")
}
opt$out_seq <- sub(".*/", "", opt$in_seq)
opt$rps_table <- paste0(opt$directory, "/chimeras/", opt$out_seq, ".rps.out")
# make empty variable function
suppressPackageStartupMessages(library(tidyverse))
suppressPackageStartupMessages(library(plyranges))
suppressPackageStartupMessages(library(BSgenome))
suppressPackageStartupMessages(library(grid))
# read in fasta
rm_seq_in <- readDNAStringSet(paste0(opt$in_seq))
names(rm_seq_in) <- sub(" .*", "", names(rm_seq_in))
rm_seq_info <- tibble(seqnames = names(rm_seq_in), width = width(rm_seq_in))
compiled_acceptable <- tibble()
truly_chimeric_ranges <- GRanges()
questionable <- tibble()
no_domains_seq <- DNAStringSet()
# read in acceptable domains and exceptional domains and manually added additional domains
exceptional_domains <- read_tsv("data/exceptional_domains.tsv", show_col_types = FALSE) %>%
dplyr::select(ref, class) %>% dplyr::rename(excep_class = class)
additional_domains <- read_tsv("data/additional_domains.tsv", show_col_types = FALSE) %>%
dplyr::select(ref, class) %>% dplyr::rename(excep_class = class)
exceptional_domains <- rbind(exceptional_domains, additional_domains)
acceptable_domains <- read_tsv("data/acceptable_domains.tsv", show_col_types = FALSE)
if(file.size(opt$rps_table)==0){
writeXStringSet(rm_seq_in, paste0(opt$directory, "/chimeras/clean_", opt$out_seq))
quit()
}
# read rps blast out
rps_blast_out <- read_tsv(file = opt$rps_table,
col_names = c("seqnames", "qstart", "qend", "qlen", "sseqid", "sstart", "send", "slen",
"pident", "length", "mismatch", "gapopen", "evalue", "bitscore", "qcovs", "stitle"),
show_col_types = FALSE) %>%
tidyr::separate(stitle, into = c("ref", "abbrev", "full"), sep = ", ", extra = "drop") %>%
dplyr::filter(length >= 0.5*slen)
rps_blast_out$n <- 1:nrow(rps_blast_out)
# select acceptable and non acceptable
contains_acceptable <- rps_blast_out[rps_blast_out$ref %in% acceptable_domains$ref,]
contains_unacceptable <- rps_blast_out[!rps_blast_out$ref %in% acceptable_domains$ref,]
# Filter repeats containing only suitable domains
completely_acceptable <- rps_blast_out %>%
filter(seqnames %in% contains_acceptable$seqnames,
!seqnames %in% contains_unacceptable$seqnames) %>%
arrange(seqnames, qstart)
# Filter repeats containing no suitable domains
questionable <- rps_blast_out %>%
filter(!seqnames %in% contains_acceptable$seqnames,
seqnames %in% contains_unacceptable$seqnames) %>%
arrange(seqnames, qstart)
# Filter repeats containing both suitable and unsuitable domains
chimeric <- rps_blast_out %>%
filter(seqnames %in% contains_acceptable$seqnames,
seqnames %in% contains_unacceptable$seqnames) %>%
arrange(seqnames, qstart) %>%
mutate(unacceptable = ifelse(ref %in% acceptable_domains$ref, "false", "true"))
# check if domains fit exceptional circumstances (grepl used to allow for the various ERVs)
excep_check <- chimeric %>%
inner_join(exceptional_domains)
for( i in 1:nrow(excep_check)){
if(grepl(pattern = excep_check$excep_class[i], x = excep_check$seqnames[i])){
excep_check$unacceptable[i] <- "false"
} else {
excep_check$unacceptable[i] <- "true"
}
}
# select single for each excep check, ensuring acceptable retained if exists
excep_check <- excep_check %>%
group_by(n) %>%
dplyr::arrange(unacceptable) %>%
dplyr::slice(1) %>%
dplyr::select(-excep_class)
# add excep_checked back into fold
chimeric <- chimeric %>%
dplyr::filter(!n %in% excep_check$n) %>%
base::rbind(excep_check) %>%
arrange(n)
# Determine regions with acceptable domains overlapping "unacceptable"
chimeric_ranges <- chimeric %>%
dplyr::mutate(start = ifelse(qstart < qend, qstart, qend),
end = ifelse(qstart > qend, qstart, qend),
strand = ifelse(qstart < qend, "+", "-")) %>%
dplyr::select(seqnames, start, end, strand, abbrev, unacceptable) %>%
plyranges::as_granges()
chimeric_ranges_unacceptable_false <- chimeric_ranges[chimeric_ranges$unacceptable == "false",]
chimeric_ranges_unacceptable_true <- chimeric_ranges[chimeric_ranges$unacceptable == "true",]
# find regions which don't overlap with acceptable domains
truly_chimeric_ranges <- filter_by_non_overlaps(chimeric_ranges_unacceptable_true, chimeric_ranges_unacceptable_false)
truly_chimeric <- chimeric[chimeric$seqnames %in% seqnames(truly_chimeric_ranges),]
false_positive_chimeric <- chimeric[!chimeric$seqnames %in% seqnames(truly_chimeric_ranges),] %>%
dplyr::select(-unacceptable)
compiled_acceptable <- rbind(completely_acceptable, false_positive_chimeric) %>%
dplyr::mutate(strand = ifelse(qstart < qend, "+", "-"), start = 1, end = qlen) %>%
dplyr::group_by(seqnames) %>%
dplyr::arrange(-bitscore) %>%
dplyr::slice(1) %>%
dplyr::ungroup() %>%
plyranges::as_granges()
# identify sequences with no domains
no_domains_seq <- rm_seq_in[!names(rm_seq_in) %in% c(chimeric$seqnames, completely_acceptable$seqnames, questionable$seqnames)]
## add step to combine data
# write to file (check if any filtered, if not write all in to output)
completely_acceptable_seq <- getSeq(rm_seq_in, compiled_acceptable)
names(completely_acceptable_seq) <- seqnames(compiled_acceptable)
writeXStringSet(c(completely_acceptable_seq, no_domains_seq), paste0(opt$directory, "/chimeras/clean_", opt$out_seq))
# in writing to file flip if appropriate
if(nrow(truly_chimeric) > 0){
truly_chimeric_ranges <- truly_chimeric %>%
dplyr::mutate(strand = ifelse(qstart < qend, "+", "-"), start = 1, end = qlen) %>%
dplyr::group_by(seqnames) %>%
dplyr::arrange(-bitscore) %>%
dplyr::slice(1) %>%
dplyr::ungroup() %>%
plyranges::as_granges()
chimeric_seq <- getSeq(rm_seq_in, truly_chimeric_ranges)
names(chimeric_seq) <- seqnames(truly_chimeric_ranges)
writeXStringSet(chimeric_seq, paste0(opt$directory, "/chimeras/chimeric_", opt$out_seq))
}
if(nrow(questionable) > 0){
questionable_ranges <- questionable %>%
dplyr::mutate(strand = ifelse(qstart < qend, "+", "-"), start = 1, end = qlen) %>%
dplyr::group_by(seqnames) %>%
dplyr::arrange(-bitscore) %>%
dplyr::slice(1) %>%
dplyr::ungroup() %>%
plyranges::as_granges()
questionable_seq <- getSeq(rm_seq_in, questionable_ranges)
names(questionable_seq) <- seqnames(questionable_ranges)
writeXStringSet(questionable_seq, paste0(opt$directory, "/chimeras/questionable_", opt$out_seq))
}
# plotting section for manual consideration of truly_chimeric elements
if(opt$plot == TRUE & nrow(truly_chimeric) > 0){
if(!file.exists(paste0(opt$directory, "/chimeras/plots/"))){
dir.create(paste0(opt$directory, "/chimeras/plots/"))
}
for(i in 1:length(base::unique(truly_chimeric$seqnames))){
to_plot <- truly_chimeric[truly_chimeric$seqnames == base::unique(truly_chimeric$seqnames)[i],] %>% arrange(qstart) %>%
mutate(missing = ifelse(ref %in% acceptable_domains$ref, FALSE, TRUE))
to_plot$y = 1:nrow(to_plot)
to_plot$text_x = (to_plot$qstart + to_plot$qend)/2
ggplot(to_plot) +
geom_rect(aes(xmin = qstart, xmax = qend, ymin = y-1, ymax = y, fill = missing)) +
geom_segment(aes(x = qstart, xend = qend, y = y, yend = y),
arrow = arrow(length = unit(10,"points"))) +
geom_text(aes(x = text_x, y = y-0.5, label = abbrev)) +
scale_x_continuous(limits = c(0,to_plot$qlen[1]), name = "Coordinates (bp)", expand = c(0,0)) +
scale_y_continuous(name = NULL, breaks = NULL) +
scale_fill_discrete(name="Atypical of TEs?") +
ggtitle(to_plot$seqnames[1]) +
theme_bw()
ggsave(filename = paste0(opt$directory, "/chimeras/plots/", gsub("/", "_", base::unique(truly_chimeric$seqnames)[i]), ".svg"), device = "svg")
}
}
================================================
FILE: scripts/TEstrainer/scripts/trf_parser.py
================================================
#!/usr/bin/env python
from argparse import (ArgumentParser, FileType)
def parse_args():
"Parse the input arguments, use '-h' for help"
parser = ArgumentParser(description='Convert Tandem Repeat Finder (TRF) dat file to bed format with repeat units for microsatellite genotyping')
parser.add_argument(
'-i', '--trf', type=str, required=True,
help='Input file produced by Tandem Repeat Finder (TRF)')
parser.add_argument(
'-o', '--out', type=str, required=True,
help='Output file containing genomic locations and repeat units of microsatellites.')
return parser.parse_args()
def main():
# Parse command line arguments
args = parse_args()
trffile = args.trf
outfile = args.out
with open(outfile, 'w') as out:
chrom = ""
with open(trffile, 'r') as trf:
for line in trf:
splitline = line.split()
if line.startswith("@"):
chrom = line.split()[0]
else:
start = splitline[0]
end = splitline[1]
period = splitline[2]
count = splitline[3]
ssr = splitline[13]
out.write('\t'.join([chrom,start,end,period,count,ssr]) + '\n')
if __name__ == '__main__':
main()
================================================
FILE: scripts/autoLand.R
================================================
# load libraries
library(tidyverse)
library(data.table)
library(magrittr)
# set options
options(scipen = 100, stringsAsFactors = FALSE)
# get inputs
args <- commandArgs()
# print(args)
inDist <- args[6]
gen <- args[7]
spec <- args[8]
saveLand <- args[9]
## bed file
## genome size
## landscape name
# test
#inDist = "~/projects/monarch/datingInsertions/danausPlexippus.divsum.unaltered"
#gen = 248676414
#spec = "test"
# read table
dist <- as.data.frame(t(fread(inDist, skip = "Div", header = TRUE)))
dist$species <- spec
dist$type <- rownames(dist)
# fix shitty columns
dist <- dist[! dist$type == "X",]
colnames(dist) <- c(dist[1,1:71], "species", "type")
dist <- dist[! dist$type == "Div",]
rownames(dist) <- 1:nrow(dist)
# get sidple TE classification levels
dist$classif <- dist$type
dist$classif <- gsub("^DNA.*", "DNA", dist$classif)
dist$classif <- gsub("^LTR.*", "LTR", dist$classif)
dist$classif <- gsub(".*Penelope|^PLE.*", "Penelope", dist$classif)
dist$classif <- gsub("^LINE.*", "LINE", dist$classif)
dist$classif <- gsub("^SINE.*", "SINE", dist$classif)
dist$classif <- gsub("^RC.*", "Rolling Circle", dist$classif)
dist$classif <- gsub("^Unknown.*|Retroposon.*|Unspecified.*", "Unclassified", dist$classif)
dist$classif <- gsub(".*RNA.*|^Satellite.*|^Simple_repeat.*|^Low_complexity.*|^ARTEFACT.*|^Other.*|^Segmenta.*", "Other (Simple Repeat, Microsatellite, RNA)", dist$classif)
# long format
piv <- dist[,c(1:72, 74)] %>% pivot_longer(!c("species", "classif"), names_to = "Divergence", values_to = "count") %>% group_by(species, classif, Divergence) %>% summarise(Count = sum(count))
piv$Divergence <- as.numeric(piv$Divergence)
colourOrder <- c("DNA",
"Rolling Circle",
"Penelope",
"LINE",
"SINE",
"LTR",
"Other (Simple Repeat, Microsatellite, RNA)",
"Unclassified")
piv$classif %<>% as.factor() %>% ordered(levels = colourOrder)
# add genome size and scale number of elements
genomeSizes <- as.data.frame(matrix(c(spec, gen),
byrow = TRUE, ncol = 2))
colnames(genomeSizes) <- c("species", "genomeSize")
piv <- merge(piv, genomeSizes)
piv$genomeSize <- as.numeric(piv$genomeSize)
piv$proportion <- piv$Count / piv$genomeSize
for (i in 1:nrow(piv)) {
if (piv$Count[i] < 0) {
piv$Count[i] <- 0
}
}
for (i in 1:length(piv$proportion)){
if (piv$proportion[i] < 0) {
piv$proportion[i] <- 0
}
if (i %% 100 == 0) { #if x is divisible by 100
cat(sprintf("Completed: %s\r", i)) #print Completed: %s\r
}
}
# summarise
piv <- piv %>% distinct()
piv2 <- piv %>% group_by(species, Divergence, classif) %>% summarise(proportion = sum(proportion))
# set plot limits
limits <- piv2 %>% group_by(Divergence) %>% summarise(perc = sum(proportion * 100))
ylimit <- max(limits$perc) * 1.1
# set colours
colours <- as.data.frame(matrix(c("DNA", "#E32017",
"Rolling Circle", "#EE7C0E",
"Penelope", "#7156A5",
"LINE", "#0098D4",
"SINE", "#9B0056",
"LTR", "#00782A",
"Other (Simple Repeat, Microsatellite, RNA)", "#F3A9BB",
"Unclassified", "#A0A5A9"), ncol = 2, byrow = TRUE))
colnames(colours) <- c("tclassif", "colour")
col <- colours$colour
names(col) <- colours$tclassif
# plot
plot <- ggplot(piv2[! piv2$classif %like% "Other",], aes(x = Divergence, y = proportion*100, fill = classif)) +
geom_bar(stat = "identity") +
ylim(0, ylimit) +
scale_x_reverse() +
scale_fill_manual(values = col) +
theme_classic() +
theme(strip.background = element_blank(), strip.text.y = element_text(face = "italic", angle = 0),
axis.text.y = element_text(size = 8)) +
labs(y = "Percentage of Genome", x = "Kimura Distance (CpG Adjusted)", fill = "")
ggsave(saveLand,
plot = plot,
scale = 1,
width = 297,
height = 210,
units = "mm",
dpi = 300,
limitsize = FALSE)
================================================
FILE: scripts/autoPie.R
================================================
# load libraries
suppressPackageStartupMessages(library(tidyverse))
suppressPackageStartupMessages(library(data.table))
suppressPackageStartupMessages(library(kableExtra))
# set options
options(scipen = 100, stringsAsFactors = FALSE)
# get inputs
args <- commandArgs()
# print(args)
inBed <- args[6]
inGff <- args[7]
gen <- args[8]
savePie <- args[9]
saveTab <- args[10]
# read table
input <- read.table(inBed, header = TRUE, sep = "\t")
input2 <- read.table(inGff, header = FALSE, sep = "\t")
gen <- as.numeric(gen)
# set colnames
colnames(input2) <- c("scaf", "method", "classif", "start", "end", "score", "strand", "dot", "attributes")
# simple classification
input$tclassif <- input$Family
input$tclassif <- gsub("^DNA.*", "DNA", input$tclassif)
input$tclassif <- gsub("^RC.*", "Rolling Circle", input$tclassif)
input$tclassif <- gsub(".*Penelope|^PLE.*", "Penelope", input$tclassif)
input$tclassif <- gsub("^LINE.*", "LINE", input$tclassif)
input$tclassif <- gsub("^SINE.*", "SINE", input$tclassif)
input$tclassif <- gsub("^LTR.*", "LTR", input$tclassif)
input$tclassif <- gsub("^Unknown.*|Retroposon.*|Unspecified.*|^unknown.*", "Unclassified", input$tclassif)
input$tclassif <- gsub(".*RNA.*|^Satellite.*|^Simple_repeat.*|^Low_complexity.*|^ARTEFACT.*|^repeat.*|^Other.*|^Microsatellite.*", "Other (Simple Repeat, Microsatellite, RNA)", input$tclassif)
input <- input %>%
mutate(tclassif = case_when(input$Family %like% "-nested" ~ paste0(tclassif, "-nested"),
.default = tclassif))
input$tclassif <- gsub("Unmasked", "Non-Repeat", input$tclassif)
input2$tclassif <- input2$classif
input2$tclassif <- gsub("^DNA.*", "DNA", input2$tclassif)
input2$tclassif <- gsub("^RC.*", "Rolling Circle", input2$tclassif)
input2$tclassif <- gsub(".*Penelope|^PLE.*", "Penelope", input2$tclassif)
input2$tclassif <- gsub("^LINE.*", "LINE", input2$tclassif)
input2$tclassif <- gsub("^SINE.*", "SINE", input2$tclassif)
input2$tclassif <- gsub("^LTR.*", "LTR", input2$tclassif)
input2$tclassif <- gsub("^Unknown.*|Retroposon.*|Unspecified.*|^unknown.*", "Unclassified", input2$tclassif)
input2$tclassif <- gsub(".*RNA.*|^Satellite.*|^Simple_repeat.*|^Low_complexity.*|^ARTEFACT.*|^repeat.*|^Other.*|^Microsatellite.*", "Other (Simple Repeat, Microsatellite, RNA)", input2$tclassif)
input2 <- input2 %>%
mutate(tclassif = case_when(attributes %like% "FULLY_NESTED" ~ paste0(tclassif, "-nested"),
.default = tclassif))
# set colours
# https://oobrien.com/2012/01/tube-colours/
# DNA = #E32017
# Rolling Circle = #EE7C0E
# Penelope = #7156A5
# LINE = #0098D4
# SINE = #9B0056
# LTR = #00782A
# Other = #F3A9BB
# Unclas = #A0A5A9
# Non-Repeat = #000000
colours <- as.data.frame(matrix(c("DNA", "#E32017",
"Rolling Circle", "#EE7C0E",
"Penelope", "#7156A5",
"LINE", "#0098D4",
"SINE", "#9B0056",
"LTR", "#00782A",
"Other (Simple Repeat, Microsatellite, RNA)", "#F3A9BB",
"Unclassified", "#A0A5A9",
"Non-Repeat", "#000000"), ncol = 2, byrow = TRUE))
colnames(colours) <- c("tclassif", "colour")
col <- colours$colour
names(col) <- colours$tclassif
# summary table
# use summary table generated previously...
pieCou <- input %>%
select(tclassif, Number.of.Elements)
colnames(pieCou) <- c("tclassif", "count")
pieSum <- input
# set factor order
pieSum$tclassif %<>% as.factor() %>% ordered(levels = c("DNA",
"DNA-nested",
"Rolling Circle",
"Rolling Circle-nested",
"Penelope",
"Penelope-nested",
"LINE",
"LINE-nested",
"SINE",
"SINE-nested",
"LTR",
"LTR-nested",
"Other (Simple Repeat, Microsatellite, RNA)",
"Other (Simple Repeat, Microsatellite, RNA)-nested",
"Unclassified",
"Unclassified-nested",
"Total Interspersed Repeat",
"Non-Repeat"))
# get proportions
pieSum$proportion <- pieSum$Percentage.of.Sequence / 100
pieSum$percentage <- pieSum$Percentage.of.Sequence
pieSum$genome_size <- pieSum$GenomeSize
# plot pie
plot <- ggplot(pieSum %>%
filter(! tclassif %like% "-nested",
! tclassif %in% c("Total Interspersed Repeat")), aes(x = as.numeric(gen/2), y = proportion, fill = tclassif, width = as.numeric(gen))) +
geom_bar(stat = "identity", position = "fill") +
coord_polar("y", start = 0) +
scale_fill_manual(values = col) +
theme_classic() +
theme(axis.line = element_blank(), axis.text = element_blank(), axis.ticks = element_blank(), axis.title = element_blank(),
strip.text = element_text(size = 16, face = "italic"), legend.text = element_text(size = 16), legend.title = element_blank())
ggsave(savePie,
plot = plot,
scale = 1,
width = 297,
height = 210,
units = "mm",
dpi = 300,
limitsize = FALSE)
# generate summary of TE family abundance
familyAbundance <- input2 %>%
mutate(family = gsub(".*NAME=", "", gsub(";TSTART.*", "", attributes))) %>%
mutate(name = paste0(family, "#", classif)) %>%
group_by(name) %>%
summarise(coverage = sum((end - start) + 1)) %>%
arrange(-coverage)
familyTally <- input2 %>%
mutate(family = gsub(".*NAME=", "", gsub(";TSTART.*", "", attributes))) %>%
mutate(name = paste0(family, "#", classif)) %>%
group_by(name) %>%
tally(name = "copy_number")
familyAbundance <- merge(familyAbundance, familyTally) %>%
arrange(-coverage)
colnames(familyAbundance) <- c("TE Family", "Coverage (bp)", "Copy Number")
saveFam <- gsub("highLevelCount", "familyLevelCount", saveTab)
classTally <- input2 %>%
group_by(tclassif) %>%
tally(name = "Copy.Number") %>%
rename("Family" = "tclassif")
classFamilyCount <- input2 %>%
mutate(family = gsub(".*NAME=", "", gsub(";TSTART.*", "", attributes))) %>%
select(family, tclassif) %>%
distinct() %>%
group_by(tclassif) %>%
tally(name = "Family.Count") %>%
rename("Family" = "tclassif")
pieSum <- pieSum %>%
ungroup() %>%
select(-c(Family)) %>%
rename("Family" = "tclassif") %>%
left_join(classTally) %>%
left_join(classFamilyCount)
# finalise table
pieSum$Family %<>% as.factor() %>% ordered(levels = c("DNA",
"DNA-nested",
"Rolling Circle",
"Rolling Circle-nested",
"Penelope",
"Penelope-nested",
"LINE",
"LINE-nested",
"SINE",
"SINE-nested",
"LTR",
"LTR-nested",
"Other (Simple Repeat, Microsatellite, RNA)",
"Other (Simple Repeat, Microsatellite, RNA)-nested",
"Unclassified",
"Unclassified-nested",
"Total Interspersed Repeat",
"Non-Repeat"))
pieSum <- pieSum %>%
arrange(Family) %>%
select(-c(percentage, genome_size, proportion, Copy.Number)) %>%
relocate(Family, Length.Occupied, Number.of.Elements, Percentage.of.Sequence, GenomeSize, Family.Count)
colnames(pieSum) <- c("TE Classification", "Coverage (bp)", "Copy Number", "% Genome Coverage", "Genome Size", "TE Family Count")
write.table(pieSum, saveTab, col.names = TRUE, row.names = FALSE, quote = FALSE, sep = "\t")
write.table(familyAbundance, saveFam, col.names = TRUE, row.names = FALSE, quote = FALSE, sep = "\t")
# add a user-readable version of the final table
saveKable <- gsub(".txt", ".kable", saveTab)
kable(pieSum, format = "pipe") %>%
writeLines(saveKable)
saveKableFam <- gsub(".txt", ".kable", saveFam)
kable(familyAbundance, format = "pipe") %>%
writeLines(saveKableFam)
================================================
FILE: scripts/autoPie.sh
================================================
#!/bin/bash
usage()
{
echo "autoPie.sh -i mergedRepeat.bed -t finalRepeatmasker.tbl -p plotName.pdf -o summaryTable.txt -h show this message"
}
main()
{
genome_size=$(sed -n '4p' $intbl | rev | cut -f1,1 -d ':' | rev | sed 's/ bp.*//g; s/ //g')
Rscript ${SCRIPT_DIR}/autoPie.R $inbed ${inbed%.summary}.gff $genome_size $outplot $sumtab
}
## MAIN ##
while getopts i:t:p:o:h option
do
case "${option}"
in
i) inbed=${OPTARG};;
t) intbl=${OPTARG};;
p) outplot=${OPTARG};;
o) sumtab=${OPTARG};;
h) usage; exit 1;;
esac
done
# step 1 - Get genome size and run Rscript
SCRIPT_DIR=/data/toby/EarlGrey/scripts
main
================================================
FILE: scripts/backSwap.py
================================================
import pandas as pd
import sys
import csv
args = sys.argv
input = args[1]
dictionary = args[2]
output = args[3]
orig = dict(csv.reader(open(dictionary), delimiter="\t"))
dict = {value:key for key, value in orig.items()}
table = pd.read_csv(input, names = ['scaf', 'start', 'end', 'repeat', 'score', 'strand'], sep=r'\s+', header = None)
table['scaf'] = table['scaf'].astype(str)
# table['scaf'].replace(dict, inplace = True)
table['scaf'] = table['scaf'].replace(dict)
table.to_csv(output, index = False, header = False, sep = '\t')
================================================
FILE: scripts/backSwapGFF.py
================================================
import pandas as pd
import sys
import csv
args = sys.argv
input = args[1]
dictionary = args[2]
output = args[3]
orig = dict(csv.reader(open(dictionary), delimiter="\t"))
dict = {value:key for key, value in orig.items()}
table = pd.read_csv(input, names = ['scaf', 'two', 'three', 'four', 'five', 'six', 'seven', 'eight', 'nine'], sep='\t', header = None)
table['scaf'] = table['scaf'].astype(str)
# table['scaf'].replace(dict, inplace = True)
table['scaf'] = table['scaf'].replace(dict)
table.to_csv(output, index = False, header = False, sep = '\t')
================================================
FILE: scripts/bin/LTR_FINDER.x86_64-1.0.7/tRNA.list.txt
================================================
>Eukarya
Arabidopsis thaliana Athal-tRNAs.fa
Caenorhabditis briggsae (cb25.agp8 July 2002) Cbrig-tRNAs.fa
Caenorhabditis elegans (May 2003) Celeg-tRNAs.fa
Canis familiaris (canFam1 July 2004) Cfam1-tRNAs.fa
Drosophila melanogaster (Release 4) Dm-tRNAs.fa
Takifugu rubripes (v.3.0 Aug. 2002) Fr1-tRNAs.fa
Gallus gallus (galGal2 Feb 2004) Ggal2-tRNAs-all.fa
Homo sapiens (Hg17 May 2004 assembly) Hg17-tRNAs.fa
Mus musculus (mm6 March 2005) Mm6-tRNAs.fa
Oryza Sativa (indica 2005) Os-tRNAs.fa
Plasmodium falciparum (Oct 2002) Pfalc-tRNAs.fa
Pan troglodytes (chimp) (Nov 2003 assembly) panTro1-tRNAs-all.fa
Rattus norvegicus (rn3 June 2003 assembly) Rn3-tRNAs.fa
Saccharomyces cerevisiae Sc-tRNAs.fa
Schizosaccharomyces pombe (v. July 2004) Sp-tRNAs.fa
>Archaea
Aeropyrum pernix Aero_pern-tRNAs.fa
Archaeoglobus fulgidus Arch_fulg-tRNAs.fa
Haloarcula marismortui ATCC 43049 Halo_mari_ATCC_43049-tRNAs.fa
Halobacterium sp Halo_sp-tRNAs.fa
Methanosarcina acetivorans Meth_acet-tRNAs.fa
Methanococcus jannaschii Meth_jann-tRNAs.fa
Methanopyrus kandleri Meth_kand-tRNAs.fa
Methanococcus maripaludis S2 Meth_mari_S2-tRNAs.fa
Methanosarcina mazei Meth_maze-tRNAs.fa
Methanobacterium thermoautotrophicum Meth_ther-tRNAs.fa
Nanoarchaeum equitans Nano_equi-tRNAs.fa
Picrophilus torridus DSM 9790 Picr_torr_DSM_9790-tRNAs.fa
Pyrococcus abyssi Pyro_abys-tRNAs.fa
Pyrobaculum aerophilum Pyro_aero-tRNAs.fa
Pyrococcus furiosus Pyro_furi-tRNAs.fa
Pyrococcus horikoshii Pyro_hori-tRNAs.fa
Sulfolobus acidocaldarius DSM 639 Sulf_acid_DSM_639-tRNAs.fa
Sulfolobus solfataricus Sulf_solf-tRNAs.fa
Sulfolobus tokodaii Sulf_toko-tRNAs.fa
Thermoplasma acidophilum Ther_acid-tRNAs.fa
Thermococcus kodakaraensis KOD1 Ther_koda_KOD1-tRNAs.fa
Thermoplasma volcanium Ther_volc-tRNAs.fa
>Bacteria
Acinetobacter sp ADP1 Acin_sp_ADP1-tRNAs.fa
Agrobacterium tumefaciens C58 Cereon Agro_tume_C58_Cereon-tRNAs.fa
Agrobacterium tumefaciens C58 UWash Agro_tume_C58_UWash-tRNAs.fa
Anaplasma marginale St Maries Anap_marg_St_Maries-tRNAs.fa
Aquifex aeolicus Aqui_aeol-tRNAs.fa
Azoarcus sp EbN1 Azoa_sp_EbN1-tRNAs.fa
Bacillus anthracis A2012 Baci_anth_A2012-tRNAs.fa
Bacillus anthracis Ames Baci_anth_Ames-tRNAs.fa
Bacillus anthracis Ames 0581 Baci_anth_Ames_0581-tRNAs.fa
Bacillus anthracis str Sterne Baci_anth_str_Sterne-tRNAs.fa
Bacillus cereus ATCC14579 Baci_cere_ATCC14579-tRNAs.fa
Bacillus cereus ATCC 10987 Baci_cere_ATCC_10987-tRNAs.fa
Bacillus cereus ZK Baci_cere_ZK-tRNAs.fa
Bacillus clausii KSM-K16 Baci_clau_KSM-K16-tRNAs.fa
Bacillus halodurans Baci_halo-tRNAs.fa
Bacillus licheniformis ATCC 14580 Baci_lich_ATCC_14580-tRNAs.fa
Bacillus licheniformis DSM 13 Baci_lich_DSM_13-tRNAs.fa
Bacillus subtilis Baci_subt-tRNAs.fa
Bacillus thuringiensis konkukian Baci_thur_konkukian-tRNAs.fa
Bacteroides fragilis NCTC 9434 Bact_frag_NCTC_9434-tRNAs.fa
Bacteroides fragilis YCH46 Bact_frag_YCH46-tRNAs.fa
Bacteroides thetaiotaomicron VPI-5482 Bact_thet_VPI-5482-tRNAs.fa
Bartonella henselae Houston-1 Bart_hens_Houston-1-tRNAs.fa
Bartonella quintana Toulouse Bart_quin_Toulouse-tRNAs.fa
Bdellovibrio bacteriovorus Bdel_bact-tRNAs.fa
Bifidobacterium longum Bifi_long-tRNAs.fa
Blochmannia floridanus Bloc_flor-tRNAs.fa
Bordetella bronchiseptica Bord_bron-tRNAs.fa
Bordetella parapertussis Bord_para-tRNAs.fa
Bordetella pertussis Bord_pert-tRNAs.fa
Borrelia burgdorferi Borr_burg-tRNAs.fa
Borrelia garinii PBi Borr_gari_PBi-tRNAs.fa
Bradyrhizobium japonicum Brad_japo-tRNAs.fa
Brucella abortus 9-941 Bruc_abor_9-941-tRNAs.fa
Brucella melitensis Bruc_meli-tRNAs.fa
Brucella suis 1330 Bruc_suis_1330-tRNAs.fa
Buchnera aphidicola Buch_aphi-tRNAs.fa
Buchnera aphidicola Sg Buch_aphi_Sg-tRNAs.fa
Buchnera sp Buch_sp-tRNAs.fa
Burkholderia mallei ATCC 23344 Burk_mall_ATCC_23344-tRNAs.fa
Burkholderia pseudomallei K96243 Burk_pseu_K96243-tRNAs.fa
Campylobacter jejuni Camp_jeju-tRNAs.fa
Campylobacter jejuni RM1221 Camp_jeju_RM1221-tRNAs.fa
Caulobacter crescentus Caul_cres-tRNAs.fa
Chlamydophila abortus S26 3 Chla_abor_S26_3-tRNAs.fa
Chlamydophila caviae Chla_cavi-tRNAs.fa
Chlamydia muridarum Chla_muri-tRNAs.fa
Chlamydophila pneumoniae AR39 Chla_pneu_AR39-tRNAs.fa
Chlamydophila pneumoniae CWL029 Chla_pneu_CWL029-tRNAs.fa
Chlamydophila pneumoniae J138 Chla_pneu_J138-tRNAs.fa
Chlamydophila pneumoniae TW 183 Chla_pneu_TW_183-tRNAs.fa
Chlamydia trachomatis Chla_trac-tRNAs.fa
Chlorobium tepidum TLS Chlo_tepi_TLS-tRNAs.fa
Chromobacterium violaceum Chro_viol-tRNAs.fa
Clostridium acetobutylicum Clos_acet-tRNAs.fa
Clostridium perfringens Clos_perf-tRNAs.fa
Clostridium tetani E88 Clos_teta_E88-tRNAs.fa
Corynebacterium diphtheriae Cory_diph-tRNAs.fa
Corynebacterium efficiens YS-314 Cory_effi_YS-314-tRNAs.fa
Corynebacterium glutamicum Cory_glut-tRNAs.fa
Corynebacterium glutamicum ATCC 13032 Bielefeld Cory_glut_ATCC_13032_Bielefeld-tRNAs.fa
Corynebacterium glutamicum ATCC 13032 Kitasato Cory_glut_ATCC_13032_Kitasato-tRNAs.fa
Coxiella burnetii Coxi_burn-tRNAs.fa
Dehalococcoides ethenogenes 195 Deha_ethe_195-tRNAs.fa
Deinococcus radiodurans Dein_radi-tRNAs.fa
Desulfotalea psychrophila LSv54 Desu_psyc_LSv54-tRNAs.fa
Desulfovibrio vulgaris Hildenborough Desu_vulg_Hildenborough-tRNAs.fa
Ehrlichia ruminantium Gardel Ehrl_rumi_Gardel-tRNAs.fa
Ehrlichia ruminantium Welgevonden Ehrl_rumi_Welgevonden-tRNAs.fa
Ehrlichia ruminantium str. Welgevonden Ehrl_rumi_str._Welgevonden-tRNAs.fa
Enterococcus faecalis V583 Ente_faec_V583-tRNAs.fa
Erwinia carotovora atroseptica SCRI1043 Erwi_caro_atroseptica_SCRI1043-tRNAs.fa
Escherichia coli CFT073 Esch_coli_CFT073-tRNAs.fa
Escherichia coli K12 Esch_coli_K12-tRNAs.fa
Escherichia coli O157H7 Esch_coli_O157H7-tRNAs.fa
Escherichia coli O157H7 EDL933 Esch_coli_O157H7_EDL933-tRNAs.fa
Francisella tularensis tularensis Fran_tula_tularensis-tRNAs.fa
Fusobacterium nucleatum Fuso_nucl-tRNAs.fa
Geobacillus kaustophilus HTA426 Geob_kaus_HTA426-tRNAs.fa
Geobacter sulfurreducens Geob_sulf-tRNAs.fa
Gloeobacter violaceus Gloe_viol-tRNAs.fa
Gluconobacter oxydans 621H Gluc_oxyd_621H-tRNAs.fa
Haemophilus ducreyi 35000HP Haem_ducr_35000HP-tRNAs.fa
Haemophilus influenzae Haem_infl-tRNAs.fa
Haemophilus influenzae 86 028NP Haem_infl_86_028NP-tRNAs.fa
Helicobacter hepaticus Heli_hepa-tRNAs.fa
Helicobacter pylori 26695 Heli_pylo_26695-tRNAs.fa
Helicobacter pylori J99 Heli_pylo_J99-tRNAs.fa
Idiomarina loihiensis L2TR Idio_loih_L2TR-tRNAs.fa
Lactobacillus acidophilus NCFM Lact_acid_NCFM-tRNAs.fa
Lactobacillus johnsonii NCC 533 Lact_john_NCC_533-tRNAs.fa
Lactococcus lactis Lact_lact-tRNAs.fa
Lactobacillus plantarum Lact_plan-tRNAs.fa
Legionella pneumophila Lens Legi_pneu_Lens-tRNAs.fa
Legionella pneumophila Paris Legi_pneu_Paris-tRNAs.fa
Legionella pneumophila Philadelphia 1 Legi_pneu_Philadelphia_1-tRNAs.fa
Leifsonia xyli xyli CTCB0 Leif_xyli_xyli_CTCB0-tRNAs.fa
Leptospira interrogans Lept_inte-tRNAs.fa
Leptospira interrogans serovar Copenhageni Lept_inte_serovar_Copenhageni-tRNAs.fa
Leptospira interrogans serovar Lai Lept_inte_serovar_Lai-tRNAs.fa
Listeria innocua List_inno-tRNAs.fa
Listeria monocytogenes List_mono-tRNAs.fa
Listeria monocytogenes 4b F2365 List_mono_4b_F2365-tRNAs.fa
Mannheimia succiniciproducens MBEL55E Mann_succ_MBEL55E-tRNAs.fa
Mesoplasma florum L1 Meso_flor_L1-tRNAs.fa
Mesorhizobium loti Meso_loti-tRNAs.fa
Methylococcus capsulatus Bath Meth_caps_Bath-tRNAs.fa
Mycobacterium avium paratuberculosis Myco_aviu_paratuberculosis-tRNAs.fa
Mycobacterium bovis Myco_bovi-tRNAs.fa
Mycoplasma gallisepticum Myco_gall-tRNAs.fa
Mycoplasma genitalium Myco_geni-tRNAs.fa
Mycoplasma hyopneumoniae 232 Myco_hyop_232-tRNAs.fa
Mycobacterium leprae Myco_lepr-tRNAs.fa
Mycoplasma mobile 163K Myco_mobi_163K-tRNAs.fa
Mycoplasma mycoides Myco_myco-tRNAs.fa
Mycoplasma penetrans Myco_pene-tRNAs.fa
Mycoplasma pneumoniae Myco_pneu-tRNAs.fa
Mycoplasma pulmonis Myco_pulm-tRNAs.fa
Mycobacterium tuberculosis CDC1551 Myco_tube_CDC1551-tRNAs.fa
Mycobacterium tuberculosis H37Rv Myco_tube_H37Rv-tRNAs.fa
Neisseria gonorrhoeae FA 1090 Neis_gono_FA_1090-tRNAs.fa
Neisseria meningitidis MC58 Neis_meni_MC58-tRNAs.fa
Neisseria meningitidis Z2491 Neis_meni_Z2491-tRNAs.fa
Nitrosomonas europaea Nitr_euro-tRNAs.fa
Nocardia farcinica IFM10152 Noca_farc_IFM10152-tRNAs.fa
Nostoc sp Nost_sp-tRNAs.fa
Oceanobacillus iheyensis Ocea_ihey-tRNAs.fa
Onion yellows phytoplasma Onio_yell_phytoplasma-tRNAs.fa
Parachlamydia sp UWE25 Para_sp_UWE25-tRNAs.fa
Pasteurella multocida Past_mult-tRNAs.fa
Photorhabdus luminescens Phot_lumi-tRNAs.fa
Photobacterium profundum SS9 Phot_prof_SS9-tRNAs.fa
Pirellula sp Pire_sp-tRNAs.fa
Porphyromonas gingivalis W83 Porp_ging_W83-tRNAs.fa
Prochlorococcus marinus CCMP1375 Proc_mari_CCMP1375-tRNAs.fa
Prochlorococcus marinus MED4 Proc_mari_MED4-tRNAs.fa
Prochlorococcus marinus MIT9313 Proc_mari_MIT9313-tRNAs.fa
Propionibacterium acnes KPA171202 Prop_acne_KPA171202-tRNAs.fa
Pseudomonas aeruginosa Pseu_aeru-tRNAs.fa
Pseudomonas fluorescens Pf-5 Pseu_fluo_Pf-5-tRNAs.fa
Pseudomonas putida KT2440 Pseu_puti_KT2440-tRNAs.fa
Pseudomonas syringae Pseu_syri-tRNAs.fa
Pseudomonas syringae pv B728a Pseu_syri_pv_B728a-tRNAs.fa
Ralstonia solanacearum Rals_sola-tRNAs.fa
Rhodopseudomonas palustris CGA009 Rhod_palu_CGA009-tRNAs.fa
Rickettsia conorii Rick_cono-tRNAs.fa
Rickettsia felis URRWXCal2 Rick_feli_URRWXCal2-tRNAs.fa
Rickettsia prowazekii Rick_prow-tRNAs.fa
Rickettsia typhi wilmington Rick_typh_wilmington-tRNAs.fa
Salmonella enterica Choleraesuis Salm_ente_Choleraesuis-tRNAs.fa
Salmonella enterica Paratypi ATCC 9150 Salm_ente_Paratypi_ATCC_9150-tRNAs.fa
Salmonella typhi Salm_typh-tRNAs.fa
Salmonella typhimurium LT2 Salm_typh_LT2-tRNAs.fa
Salmonella typhi Ty2 Salm_typh_Ty2-tRNAs.fa
Shewanella oneidensis Shew_onei-tRNAs.fa
Shigella flexneri 2a Shig_flex_2a-tRNAs.fa
Shigella flexneri 2a 2457T Shig_flex_2a_2457T-tRNAs.fa
Silicibacter pomeroyi DSS-3 Sili_pome_DSS-3-tRNAs.fa
Sinorhizobium meliloti Sino_meli-tRNAs.fa
Staphylococcus aureus COL Stap_aure_COL-tRNAs.fa
Staphylococcus aureus MW2 Stap_aure_MW2-tRNAs.fa
Staphylococcus aureus Mu50 Stap_aure_Mu50-tRNAs.fa
Staphylococcus aureus N315 Stap_aure_N315-tRNAs.fa
Staphylococcus aureus aureus MRSA252 Stap_aure_aureus_MRSA252-tRNAs.fa
Staphylococcus aureus aureus MSSA476 Stap_aure_aureus_MSSA476-tRNAs.fa
Staphylococcus epidermidis ATCC 12228 Stap_epid_ATCC_12228-tRNAs.fa
Staphylococcus epidermidis RP62A Stap_epid_RP62A-tRNAs.fa
Staphylococcus haemolyticus Stap_haem-tRNAs.fa
Streptococcus agalactiae 2603 Stre_agal_2603-tRNAs.fa
Streptococcus agalactiae NEM316 Stre_agal_NEM316-tRNAs.fa
Streptomyces avermitilis Stre_aver-tRNAs.fa
Streptomyces coelicolor Stre_coel-tRNAs.fa
Streptococcus mutans Stre_muta-tRNAs.fa
Streptococcus pneumoniae R6 Stre_pneu_R6-tRNAs.fa
Streptococcus pneumoniae TIGR4 Stre_pneu_TIGR4-tRNAs.fa
Streptococcus pyogenes Stre_pyog-tRNAs.fa
Streptococcus pyogenes M1 GAS Stre_pyog_M1_GAS-tRNAs.fa
Streptococcus pyogenes M49 591 Stre_pyog_M49_591-tRNAs.fa
Streptococcus pyogenes MGAS10394 Stre_pyog_MGAS10394-tRNAs.fa
Streptococcus pyogenes MGAS315 Stre_pyog_MGAS315-tRNAs.fa
Streptococcus pyogenes MGAS8232 Stre_pyog_MGAS8232-tRNAs.fa
Streptococcus pyogenes SSI-1 Stre_pyog_SSI-1-tRNAs.fa
Streptococcus thermophilus CNRZ1066 Stre_ther_CNRZ1066-tRNAs.fa
Streptococcus thermophilus LMG 18311 Stre_ther_LMG_18311-tRNAs.fa
Symbiobacterium thermophilum IAM14863 Symb_ther_IAM14863-tRNAs.fa
Synechocystis PCC6803 Syne_PCC6-tRNAs.fa
Synechococcus elongatus PCC 6301 Syne_elon_PCC_6301-tRNAs.fa
Synechococcus sp WH8102 Syne_sp_WH8102-tRNAs.fa
Thermosynechococcus elongatus Ther_elon-tRNAs.fa
Thermotoga maritima Ther_mari-tRNAs.fa
Thermoanaerobacter tengcongensis Ther_teng-tRNAs.fa
Thermus thermophilus HB27 Ther_ther_HB27-tRNAs.fa
Thermus thermophilus HB8 Ther_ther_HB8-tRNAs.fa
Treponema denticola ATCC 35405 Trep_dent_ATCC_35405-tRNAs.fa
Treponema pallidum Trep_pall-tRNAs.fa
Tropheryma whipplei TW08 27 Trop_whip_TW08_27-tRNAs.fa
Tropheryma whipplei Twist Trop_whip_Twist-tRNAs.fa
Ureaplasma urealyticum Urea_urea-tRNAs.fa
Vibrio cholerae Vibr_chol-tRNAs.fa
Vibrio fischeri ES114 Vibr_fisc_ES114-tRNAs.fa
Vibrio parahaemolyticus Vibr_para-tRNAs.fa
Vibrio vulnificus CMCP6 Vibr_vuln_CMCP6-tRNAs.fa
Vibrio vulnificus YJ016 Vibr_vuln_YJ016-tRNAs.fa
Wigglesworthia brevipalpis Wigg_brev-tRNAs.fa
Wolbachia endosymbiont of Brugia malayi TRS Wolb_endo_of_Brugia_malayi_TRS-tRNAs.fa
Wolbachia endosymbiont of Drosophila melanogaster Wolb_endo_of_Drosophila_melanogaster-tRNAs.fa
Wolinella succinogenes Woli_succ-tRNAs.fa
Xanthomonas campestris Xant_camp-tRNAs.fa
Xanthomonas campestris 8004 Xant_camp_8004-tRNAs.fa
Xanthomonas citri Xant_citr-tRNAs.fa
Xanthomonas oryzae KACC10331 Xant_oryz_KACC10331-tRNAs.fa
Xylella fastidiosa Xyle_fast-tRNAs.fa
Xylella fastidiosa Temecula1 Xyle_fast_Temecula1-tRNAs.fa
Yersinia pestis CO92 Yers_pest_CO92-tRNAs.fa
Yersinia pestis KIM Yers_pest_KIM-tRNAs.fa
Yersinia pestis biovar Mediaevails Yers_pest_biovar_Mediaevails-tRNAs.fa
Yersinia pseudotuberculosis IP32953 Yers_pseu_IP32953-tRNAs.fa
Zymomonas mobilis ZM4 Zymo_mobi_ZM4-tRNAs.fa
================================================
FILE: scripts/bin/cut.pl
================================================
#!usr/bin/perl -w
use strict;
#usage: perl cut.pl seq [-s|-S|-l int]
my $length=5000000; #length of a sequence
my $separate=1; #1 for 1 sequence per file, 0 for all sequence in 1 files.
my $size=0; #no size control; if set to 10000; program will output sequence files every $length base (roundup to single sequence)
my $i=0;
foreach (@ARGV){
$separate=1 if $_=~/-s|Separate/;
$length=$ARGV[$i+1] if $_=~/-l|Length/i;
$size=1 if /-S|size/;
$i++;
}
open Seq, "<$ARGV[0]" or die $!;
open List, ">$ARGV[0].list" if $separate==1;
$/="\n>";
my $accu_len=0;
my $k=1;
open File, ">$ARGV[0]_sub$k" if $size==1;
while (){
s/>//g;
chomp;
next if /^\s+$/;
my ($id, $seq)=(split /\n/, $_, 2);
$seq=~s/\s+//g;
my $j=1;
if ($size==0){
my $len=$length;
for (my $i=0; ; $i+=$len){
$len=length($seq)-$i if length($seq)<$i+$len;
my $seq_sub=substr($seq, $i, $len);
last if length($seq_sub)<1;
if ($separate==0){
print ">${id}_sub$j\n$seq_sub\n";
}
elsif ($separate==1){
print List "${id}_sub$j\n";
open File, ">${id}_sub$j" or die $!;
print File ">${id}_sub$j\n$seq_sub\n";
close File;
}
$j++;
}
} else {
print File ">$id\n$seq\n";
$accu_len+=length $seq;
if ($accu_len>$length){
print "$accu_len\n";
$k++;
$accu_len=0;
close File;
open File, ">$ARGV[0]_sub$k" or die $!;
# print File ">$id\n$seq\n";
}
}
}
#system "rm $ARGV[0]_sub$k" if $size==1;
close Seq;
================================================
FILE: scripts/divergenceCalc/divergence_calc.py
================================================
import os
from os.path import exists, getsize
import sys
import argparse
import numpy as np
import pandas as pd
import multiprocessing
import pybedtools
import subprocess
import shlex
import shutil
from Bio import AlignIO, SeqIO
from math import log, sqrt
from functools import partial
from time import time
from re import sub
parser = argparse.ArgumentParser()
parser.add_argument('-l', '--repeat_library', type=str, required=True,
help='repeat_library')
parser.add_argument('-i', '--in_gff', type=str, required=True,
help='Path to gff')
parser.add_argument('-g', '--genome', type=str, required=True,
help='Path to genome')
parser.add_argument('-o', '--out_gff', type=str, required=True,
help='Output gff')
parser.add_argument('-tmp', '--temp_dir', type=str, default='tmp/',
help='Temporary directory')
parser.add_argument('-t', '--cores', type=int, default=4,
help='Number of cores')
parser.add_argument('-k', '--timeout', type=int, default=30,
help='Seconds after which matcher will be cancelled and repeat treated as unalignable')
args = parser.parse_args()
def file_check(repeat_library, in_gff, genome, out_gff, temp_dir):
if(exists(repeat_library) == False or exists(in_gff) == False or exists(genome) == False):
sys.exit('Files not found. Requires the repeat library, path to the genome, and path to gff containing coordinates and corresponding repeat files')
if(exists(genome+".fai") == False):
print("Indexing genome")
subprocess.run(["samtools","faidx",genome], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)
if(exists(temp_dir) == False):
os.mkdir(temp_dir)
if(exists(os.path.join(temp_dir, "qseqs")) == False):
os.mkdir(os.path.join(temp_dir, "qseqs"))
if(exists(os.path.join(temp_dir, "split_library")) == False):
os.mkdir(os.path.join(temp_dir, "split_library"))
def splitter(in_seq, temp_dir):
with open(in_seq, 'r') as handle:
for record in SeqIO.parse(handle, "fasta"):
repeat_name = record.name.split(sep="#")[0]
repeat_name = repeat_name.lower()
file_name = os.path.join(temp_dir, "split_library", repeat_name + ".fasta")
SeqIO.write(record, file_name, "fasta-2line")
def parse_gff(in_gff):
gff = pd.read_table(in_gff, header = None, names=['seqnames', 'tool', 'repeat_class', 'start', 'end', 'score', 'strand', 'phase', 'metadata'])
simple_gff = gff[gff['repeat_class'].str.contains('Simple_repeat|Satellite|Low_complexity')].reset_index()
gff = gff[~gff['repeat_class'].str.contains('Simple_repeat|Satellite|Low_complexity')].reset_index()
other_gff = gff[~gff['tool'].str.contains('Earl_Grey|RepeatMasker')].reset_index()
gff = gff[gff['tool'].str.contains('Earl_Grey|RepeatMasker')].reset_index()
gff['metadata_tmp'] = gff['metadata'].str.replace(';TSTART.*', '', regex=True)
gff[['repeat_id', 'repeat_family']] = gff['metadata_tmp'].str.split(';', n=2, expand=True)
gff = gff.drop(columns = ['metadata_tmp', 'repeat_id'])
gff['repeat_family'] = gff['repeat_family'].str.replace('NAME=', '', regex=True)
gff['repeat_family'] = gff['repeat_family'].str.lower()
return(gff, simple_gff, other_gff)
def file_name_generator():
import random
import string
file_name = ''.join(random.sample(string.ascii_letters, 12))+'.tmp'
return(file_name)
def Kimura80(qseq, sseq):
"""
Calculations adapted from https://github.com/kgori/python_tools_on_github/blob/master/pairwise_distances.py
"""
# define transitions, transversions, matches
transitions = [ "AG", "GA", "CT", "TC"]
transversions = [ "AC", "CA", "AT", "TA",
"GC", "CG", "GT", "TG" ]
matches = [ "AA", "GG", "CC", "TT"]
# set counters to 0
m,ts,tv=0,0,0
# count transitions, transversions, matches
for i, j in zip(qseq, sseq):
if i+j in matches: m+=1
if i+j in transitions: ts+=1
if i+j in transversions: tv+=1
# count number of bp which align (excludes gaps, Ns)
aln_len = m + ts + tv
if aln_len != 0:
# calculate p and q
p = ts/aln_len
q = tv/aln_len
# calculate Kimura distance
Kimura_dist = -0.5 * log((1 - 2*p - q) * sqrt( 1 - 2*q ))
else:
Kimura_dist = "NA"
return(Kimura_dist)
def outer_func(genome_path, temp_dir, timeoutSeconds, chunk_path):
# Set pybedtools temp directory within this worker (required when using forkserver).
pybedtools.set_tempdir(os.path.join(temp_dir, 'pybedtools'))
# Load the chunk from its temp file and delete it immediately to reclaim disk space.
# index_col=0 restores the original GFF row indices, which are used as unique
# per-row filenames in qseqs/. Without this, every chunk restarts at 0 and
# concurrent workers collide on the same paths.
gff = pd.read_csv(chunk_path, sep="\t", index_col=0)
os.remove(chunk_path)
generated_name = file_name_generator()
holder_file_name = os.path.join(temp_dir, generated_name)
failed_file_name = os.path.join(temp_dir, "failed_" + generated_name)
row_counter = 0
with open(holder_file_name, 'w') as tmp_out:
header = list(gff.columns.values)[1:] + ["Kimura"]
header = "\t".join(header)+"\n"
tmp_out.write(header)
for row in gff.iterrows():
# Set index
idx = row[0]
row_counter += 1
# Periodically release pybedtools temp files to prevent gradual memory growth.
if row_counter % 500 == 0:
pybedtools.cleanup(remove_all=False)
# Set scaffold, coordinates, strand, repeat family
seqnames, start, end, strand, repeat_family = str(row[1]['seqnames']), str(row[1]['start'] - 1), str(row[1]['end']), str(row[1]['strand']), str(row[1]['repeat_family'])
# Create BED string for BEDtools
bed_str = " ".join([seqnames, start, end, ".", ".", strand])
# Set path for query sequence
query_path = os.path.join(temp_dir, "qseqs", str(idx))
# Create bedtools command and getfasta
a=pybedtools.BedTool(bed_str, from_string=True)
try:
a = a.sequence(fi=genome_path, fo=query_path, s=True)
except: # Occasionally a samtools error occurs, this overcomes this
with open(failed_file_name, "a") as failed_file:
failed_file.write(seqnames+":"+start+"-"+end+"_"+strand+"_"+repeat_family+"\n")
if exists(query_path) is True and getsize(query_path) > 0:
# Set path to subject sequence
subject_path = os.path.join(temp_dir, "split_library", repeat_family + ".fasta")
# Run matcher, with timeout exception
test_command = shlex.split("matcher "+query_path+" "+subject_path+" -outfile "+query_path+".matcher -aformat fasta")
# Run test and kill if it takes more than 10 seconds
alignment_p = subprocess.Popen(test_command, stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)
try:
alignment_p.wait(timeoutSeconds)
except subprocess.TimeoutExpired:
# if matcher fails to complete before timeout, kill and move on
with open(failed_file_name, "a") as failed_file:
failed_file.write(seqnames+":"+start+"-"+end+"_"+strand+"_"+repeat_family+"\n")
alignment_p.kill()
if exists(query_path+".matcher") is False or getsize(query_path+".matcher") == 0:
# If no alignment is possible, set distances to NA and alignment length to 0
Kdist = "NA"
if exists(query_path):
os.remove(query_path)
if exists(query_path+".matcher") is True:
os.remove(query_path+".matcher")
else:
# Read in alignments
aln = list(SeqIO.parse(query_path+".matcher", 'fasta'))
ref_seq, gen_seq = str(aln[0].seq).upper(), str(aln[1].seq).upper()
# Check ref and genome sequence are same length, set Kdist to NA if not
if len(ref_seq) == len(gen_seq):
# Calculate distances based on model
Kdist = Kimura80(ref_seq, gen_seq)
# Convert numbers to strings
if Kdist != "NA":
Kdist = str(round(Kdist, 4))
else:
Kdist = "NA"
# Delete temporary files
os.remove(query_path+".matcher")
if exists(query_path):
os.remove(query_path)
# Make line for temporary file and write to file
tmp_holder = row[1].to_list()[1:]
tmp_holder = "\t".join(str(x) for x in tmp_holder)+"\t"+Kdist+"\n"
tmp_out.write(tmp_holder)
return(holder_file_name)
def tmp_out_parser(file_list, simple_gff, other_gff):
# Loop through results
gff=pd.DataFrame()
for file in file_list:
# read in gff
in_gff = pd.read_csv(file, sep = "\t")
# concatenate gff
gff = pd.concat([gff, in_gff], ignore_index=True)
# Convert numbers to strings for concatenation
gff['Kimura'] = gff['Kimura'].astype(str)
# Convert new data onto metadata
gff['metadata'] = gff['metadata'] + ";KIMURA80=" + gff['Kimura']
# Remove unnecessary rows
gff = gff.drop(columns = ['Kimura', 'repeat_family'])
# Combine columns, sort and drop unneccessary columns
gff = pd.concat([gff, simple_gff, other_gff], ignore_index=True)
gff = gff.drop(columns = ['level_0', 'index'])
gff = gff.sort_values(by=['seqnames', 'start'])
gff = gff.reset_index()
return(gff)
if __name__ == "__main__":
# Use forkserver so workers start from a clean process rather than inheriting
# a fork-copy of the parent address space (which holds the full GFF DataFrame).
# This is the primary mechanism for reducing peak RAM at high thread counts.
if multiprocessing.get_start_method(allow_none=True) is None:
try:
multiprocessing.set_start_method('forkserver')
except (RuntimeError, ValueError):
# Fall back to the existing/default start method if forkserver is unavailable
# or if a start method has already been set by the environment.
pass
start_time = time()
# check files exist
file_check(args.repeat_library, args.in_gff, args.genome, args.out_gff, args.temp_dir)
# split library file
print("Splitting repeat library")
splitter(args.repeat_library, args.temp_dir)
# read in gff and take head
print("Reading in gff")
in_gff, simple_gff, other_gff = parse_gff(args.in_gff)
# create as many processes as instructed cores
num_processes = args.cores
# break into exactly num_processes chunks, distributing any remainder evenly
chunks = [in_gff.iloc[idx] for idx in np.array_split(range(len(in_gff)), num_processes)]
# Write chunks to temp TSV files so the parent DataFrame can be freed before workers
# are created. Workers read from disk rather than receiving pickled DataFrames via IPC.
print("Writing chunks to disk")
chunk_files = []
for i, chunk in enumerate(chunks):
chunk_path = os.path.join(args.temp_dir, f"chunk_{i}.tsv")
chunk.to_csv(chunk_path, sep="\t", index=True)
chunk_files.append(chunk_path)
# Free the main GFF DataFrame and chunk list from the parent process before the pool
# is created. With forkserver this is already avoided, but freeing here also reduces
# parent RSS during the pool run, which matters on memory-constrained machines.
del chunks
del in_gff
# set pybedtools temp path (also set per-worker inside outer_func for forkserver)
try:
os.mkdir(os.path.join(args.temp_dir, "pybedtools"))
except FileExistsError:
pass
pybedtools.set_tempdir(os.path.join(args.temp_dir, 'pybedtools'))
print("Starting calculations")
# Perform calculations in parallel. maxtasksperchild=1 restarts each worker after
# processing one chunk, releasing any lingering pybedtools handles or cached objects.
func = partial(outer_func, args.genome, args.temp_dir, args.timeout)
pool = multiprocessing.Pool(processes=num_processes, maxtasksperchild=1)
results = list(pool.imap_unordered(func, chunk_files))
pool.close()
pool.join()
print("Finished calculations")
# Read in temp files, fix metadata, add simple repeats back, and sort
calc_gff = tmp_out_parser(results, simple_gff, other_gff)
# remove first column and write to file
calc_gff.drop(columns = ['index']).to_csv(args.out_gff, sep = "\t", header = False, index=False)
# print run time for number of rows
run_time = time() - start_time
print("Total run time for ", len(calc_gff), " rows was ", run_time, " seconds")
# Delete folder of split library
shutil.rmtree(os.path.join(args.temp_dir, "split_library"), ignore_errors=True)
================================================
FILE: scripts/divergenceCalc/divergence_plot.R
================================================
library(optparse)
option_list <- list(
make_option(c("-s", "--species_name"), default=NA, type = "character",
help="Species name (required)"),
make_option(c("-g", "--in_gff"), default=NA, type = "character",
help="GFF with Kimura distances (required)"),
make_option(c("-o", "--out_directory"), default=NA, type = "character",
help="Directory to write plots to (required)"),
make_option(c("-f", "--axis_flip"), action = "store_true",
default=TRUE, type = "logical", help="Flip x-axis on plots")
)
opt <- parse_args(OptionParser(option_list=option_list))
# Check variables are set
if(is.na(opt$species_name)){
stop("Species name must be supplied")
}
if(is.na(opt$in_gff)){
stop("Path to input gff must be supplied")
}
if(is.na(opt$out_directory)){
stop("Path to output directory must be supplied")
}
# Check output folder exists or create
if(!dir.exists(opt$out_directory)){
message("Creating output directory")
dir.create(opt$out_directory, recursive = TRUE)
}
suppressPackageStartupMessages(library(tidyverse))
suppressPackageStartupMessages(library(plyranges))
suppressPackageStartupMessages(library(viridis))
suppressPackageStartupMessages(library(cowplot))
suppressPackageStartupMessages(library(ggtext))
# set variables
species_name=opt$species_name
in_gff=opt$in_gff
out_directory=opt$out_directory
# Created plot title
plot_title <- paste0("Repeat landscape of *", gsub("_", " ", species_name), "*")
title_plot <- ggplot() + labs(title = plot_title) + theme(plot.title = element_markdown(hjust = 0.5)) + theme(panel.background = element_blank())
# Read in data
divergence_eg_gff <- read_gff(in_gff)
# Breakdown classification of repeats
divergence_eg_tes_gff <- divergence_eg_gff %>%
dplyr::mutate(subclass = sub("/.*", "", type),
superfamily = sub("-.*", "", sub(".*/", "", type)))
# Fix Penelopes
divergence_eg_tes_gff <- divergence_eg_tes_gff %>%
dplyr::mutate(subclass = ifelse(superfamily == "Penelope", "PLE", subclass)) %>%
dplyr::mutate(subclass = ifelse(subclass %in% c("DNA", "LINE", "LTR", "PLE", "RC", "SINE", "Unknown"), subclass, "Other")) %>%
dplyr::mutate(named_subclass = case_when(subclass == "DNA" ~ "DNA Transposon",
subclass == "LTR" ~ "LTR Retrotransposon",
subclass == "PLE" ~ "Penelope",
subclass == "RC" ~ "Rolling Circle",
.default = subclass))
# Create summary of families
summary_table <- divergence_eg_tes_gff %>%
as_tibble %>%
group_by(NAME) %>%
dplyr::select(width, KIMURA80, subclass, superfamily, NAME) %>%
mutate(total_bp = sum(width),
mean_width = round(mean(width), digits = 2),
min_width = round(min(width), digits = 2),
max_width = round(max(width), digits = 2),
sd_width = round(sd(width), digits = 2),
div = as.numeric(KIMURA80),
mean_div = round(mean(div), digits = 2),
min_div = round(min(div), digits = 2),
max_div = round(max(div), digits = 2),
sd_div = round(sd(div), digits = 2)
) %>%
dplyr::select(-width, -div, -KIMURA80) %>%
base::unique() %>%
dplyr::arrange(subclass, superfamily, NAME) %>%
dplyr::rename(family = NAME)
# Save summary to file
readr::write_tsv(x = summary_table, file = paste0(out_directory, "/", species_name, "_summary_table.tsv"))
# Sum lengths to create data for plots (remove subclasses not in standard set and repeats which Kimura was not calculated for)
divergence_eg_tes_rounded_for_plot <- divergence_eg_tes_gff %>%
filter(!is.na(KIMURA80)) %>%
mutate(KIMURA80 = as.numeric(KIMURA80)) %>%
dplyr::filter(KIMURA80 <= 0.5) %>%
as_tibble() %>%
dplyr::mutate(KIMURA80 = round(x = KIMURA80, digits = 2)) %>%
group_by(named_subclass, KIMURA80) %>%
mutate(KIMURA_SUM = sum(width)) %>%
ungroup() %>%
dplyr::select(subclass, named_subclass, KIMURA80, KIMURA_SUM) %>%
base::unique() %>%
arrange(named_subclass, KIMURA80)
# Set fill colours
fill_colours <- tibble(subclass = c("DNA", "LINE", "LTR", "PLE", "RC", "SINE", "Other", "Unknown"),
named_subclass = c("DNA Transposon", "LINE", "LTR Retrotransposon", "Penelope", "Rolling Circle", "SINE", "Other", "Unknown"),
fill_colour = c("#E32017", "#0098D4", "#00782A", "#7156A5", "#EE7C0E", "#9B0056", "#F3A9BB", "#A0A5A9")) %>%
filter(subclass %in% divergence_eg_tes_rounded_for_plot$subclass) %>%
arrange(named_subclass) %>%
filter(subclass %in% divergence_eg_tes_rounded_for_plot$subclass)
# Create and save main plots
kimura_plot <- ggplot(divergence_eg_tes_rounded_for_plot,
aes(x = KIMURA80, y = KIMURA_SUM, fill = named_subclass)) +
geom_col(position = "stack", width = 0.01) +
theme_bw() +
labs(title = plot_title) + theme(plot.title = element_markdown(hjust = 0.5)) +
scale_fill_manual(values = fill_colours$fill_colour, name = "TE Subclass")
subclass_kimura_plot <- kimura_plot + scale_y_continuous(expand = c(0.01,0), name = "Base pairs")
# Flip axis if desired
if(opt$axis_flip == TRUE){
subclass_kimura_plot <- subclass_kimura_plot +
scale_x_continuous(limits = c(0.51, -0.01),
expand = c(0,0), name = "Kimura 2-Parameter Distance",
trans = "reverse")
} else {
subclass_kimura_plot <- subclass_kimura_plot +
scale_x_continuous(limits = c(-0.01, 0.51),
expand = c(0,0), name = "Kimura 2-Parameter Distance")
}
ggsave(plot = subclass_kimura_plot, filename = paste0(out_directory, "/", species_name, "_classification_landscape.pdf"), device = "pdf", width = 12.85, height = 8.5)
split_subclass_kimura_plot <- kimura_plot + scale_y_continuous(name = "Base pairs", labels = function(x) format(x, scientific = TRUE)) + facet_grid(subclass~., scales = "free")
# Flip axis if desired
if(opt$axis_flip == TRUE){
split_subclass_kimura_plot <- split_subclass_kimura_plot +
scale_x_continuous(limits = c(0.51, -0.01),
expand = c(0,0), name = "Kimura 2-Parameter Distance",
trans = "reverse")
} else {
split_subclass_kimura_plot <- split_subclass_kimura_plot +
scale_x_continuous(limits = c(-0.01, 0.51),
expand = c(0,0), name = "Kimura 2-Parameter Distance")
}
ggsave(plot = split_subclass_kimura_plot, filename = paste0(out_directory, "/", species_name, "_split_class_landscape.pdf"), device = "pdf", width = 12.85, height = 8.5)
# Perform maths for more divided plot
divergence_eg_tes_rounded_for_superfamily_plot <- divergence_eg_tes_gff %>%
as_tibble() %>%
filter(!is.na(KIMURA80)) %>%
mutate(KIMURA80 = as.numeric(KIMURA80)) %>%
dplyr::filter(KIMURA80 <= 0.5) %>%
dplyr::mutate(KIMURA80 = round(x = KIMURA80, digits = 2),
type = sub("-.*", "", type)) %>%
group_by(superfamily, KIMURA80) %>%
mutate(KIMURA_SUM = sum(width)) %>%
ungroup() %>%
dplyr::select(type, subclass, superfamily, KIMURA80, KIMURA_SUM) %>%
base::unique() %>%
arrange(subclass, superfamily, KIMURA80)
# Split data as necessary
divergence_eg_tes_rounded_for_superfamily_plot <- split(divergence_eg_tes_rounded_for_superfamily_plot,
f = divergence_eg_tes_rounded_for_superfamily_plot$subclass)
# Create plots of superfamilies of DNA transposons, LINEs, LTR retrotransposons and SINEs
if (!is.null(divergence_eg_tes_rounded_for_superfamily_plot[["DNA"]]) && nrow(divergence_eg_tes_rounded_for_superfamily_plot[["DNA"]]) > 0) {
kimura_superfamily_plot_1 <- ggplot(divergence_eg_tes_rounded_for_superfamily_plot$DNA,
aes(x = KIMURA80, y = KIMURA_SUM, fill = superfamily)) +
geom_col(position = "stack", width = 0.01, colour = "black", linewidth = 0.2) +
theme_bw() +
theme(legend.title=element_blank()) +
scale_y_continuous(name = "Base pairs", labels = function(x) format(x, scientific = TRUE)) +
facet_grid(subclass~., scales = "free") +
guides(fill=guide_legend(ncol=3))
} else {
kimura_superfamily_plot_1 <- NULL
}
if (!is.null(divergence_eg_tes_rounded_for_superfamily_plot[["LINE"]]) && nrow(divergence_eg_tes_rounded_for_superfamily_plot[["LINE"]]) > 0) {
kimura_superfamily_plot_2 <- ggplot(divergence_eg_tes_rounded_for_superfamily_plot$LINE,
aes(x = KIMURA80, y = KIMURA_SUM, fill = superfamily)) +
geom_col(position = "stack", width = 0.01, colour = "black", linewidth = 0.2) +
theme_bw() +
theme(legend.title=element_blank()) +
scale_y_continuous(name = "Base pairs", labels = function(x) format(x, scientific = TRUE)) +
facet_grid(subclass~., scales = "free") +
guides(fill=guide_legend(ncol=3)) +
scale_fill_brewer(palette = "Blues", direction = -1)
} else {
kimura_superfamily_plot_2 <- NULL
}
if (!is.null(divergence_eg_tes_rounded_for_superfamily_plot[["LTR"]]) && nrow(divergence_eg_tes_rounded_for_superfamily_plot[["LTR"]]) > 0) {
kimura_superfamily_plot_3 <- ggplot(divergence_eg_tes_rounded_for_superfamily_plot$LTR,
aes(x = KIMURA80, y = KIMURA_SUM, fill = superfamily)) +
geom_col(position = "stack", width = 0.01, colour = "black", linewidth = 0.2) +
theme_bw() +
theme(legend.title=element_blank()) +
scale_y_continuous(name = "Base pairs", labels = function(x) format(x, scientific = TRUE)) +
facet_grid(subclass~., scales = "free") +
guides(fill=guide_legend(ncol=3)) +
scale_fill_brewer(palette = "Greens", direction = -1)
} else {
kimura_superfamily_plot_3 <- NULL
}
if (!is.null(divergence_eg_tes_rounded_for_superfamily_plot[["SINE"]]) && nrow(divergence_eg_tes_rounded_for_superfamily_plot[["SINE"]]) > 0) {
kimura_superfamily_plot_4 <- ggplot(divergence_eg_tes_rounded_for_superfamily_plot$SINE,
aes(x = KIMURA80, y = KIMURA_SUM, fill = superfamily)) +
geom_col(position = "stack", width = 0.01, colour = "black", linewidth = 0.2) +
theme_bw() +
theme(legend.title=element_blank()) +
scale_y_continuous(name = "Base pairs", labels = function(x) format(x, scientific = TRUE)) +
facet_grid(subclass~., scales = "free") +
guides(fill=guide_legend(ncol=3)) +
scale_fill_brewer(palette = "YlOrRd", direction = -1)
} else {
kimura_superfamily_plot_4 <- NULL
}
if (!is.null(divergence_eg_tes_rounded_for_superfamily_plot[["PLE"]]) && nrow(divergence_eg_tes_rounded_for_superfamily_plot[["PLE"]]) > 0) {
kimura_superfamily_plot_5 <- ggplot(divergence_eg_tes_rounded_for_superfamily_plot$PLE,
aes(x = KIMURA80, y = KIMURA_SUM, fill = superfamily)) +
geom_col(position = "stack", width = 0.01, colour = "black", linewidth = 0.2) +
theme_bw() +
theme(legend.title=element_blank()) +
scale_y_continuous(name = "Base pairs", labels = function(x) format(x, scientific = TRUE)) +
facet_grid(subclass~., scales = "free") +
guides(fill=guide_legend(ncol=3)) +
scale_fill_brewer(palette = "Purples", direction = -1)
} else {
kimura_superfamily_plot_5 <- NULL
}
if (!is.null(divergence_eg_tes_rounded_for_superfamily_plot[["RC"]]) && nrow(divergence_eg_tes_rounded_for_superfamily_plot[["RC"]]) > 0) {
kimura_superfamily_plot_6 <- ggplot(divergence_eg_tes_rounded_for_superfamily_plot$RC,
aes(x = KIMURA80, y = KIMURA_SUM, fill = superfamily)) +
geom_col(position = "stack", width = 0.01, colour = "black", linewidth = 0.2) +
theme_bw() +
theme(legend.title=element_blank()) +
scale_y_continuous(name = "Base pairs", labels = function(x) format(x, scientific = TRUE)) +
facet_grid(subclass~., scales = "free") +
guides(fill=guide_legend(ncol=3)) +
scale_fill_brewer(palette = "Oranges", direction = -1)
} else {
kimura_superfamily_plot_6 <- NULL
}
# Flip axis if desired — only applied to non-NULL plots
# The last non-NULL plot gets the axis label; all others get an empty name
all_superfamily_plots <- list(
kimura_superfamily_plot_1, kimura_superfamily_plot_2, kimura_superfamily_plot_3,
kimura_superfamily_plot_4, kimura_superfamily_plot_5, kimura_superfamily_plot_6
)
non_null_idx <- which(!sapply(all_superfamily_plots, is.null))
last_idx <- if (length(non_null_idx) > 0) max(non_null_idx) else NA
for (i in seq_along(all_superfamily_plots)) {
if (is.null(all_superfamily_plots[[i]])) next
axis_label <- if (!is.na(last_idx) && i == last_idx) "Kimura 2-Parameter Distance" else ""
if (opt$axis_flip == TRUE) {
all_superfamily_plots[[i]] <- all_superfamily_plots[[i]] +
scale_x_continuous(limits = c(0.51, -0.01), expand = c(0, 0),
name = axis_label, trans = "reverse")
} else {
all_superfamily_plots[[i]] <- all_superfamily_plots[[i]] +
scale_x_continuous(limits = c(-0.01, 0.51), expand = c(0, 0),
name = axis_label)
}
}
valid_plots <- all_superfamily_plots[!sapply(all_superfamily_plots, is.null)]
# Combine plots and title
superfamily_kimura_plot <- plot_grid(plotlist = valid_plots,
ncol = 1, align = "v")
superfamily_kimura_plot_titled <- plot_grid(title_plot, superfamily_kimura_plot, ncol = 1, rel_heights = c(1, 30))
# Save divided plot
ggsave(plot = superfamily_kimura_plot_titled, filename = paste0(out_directory, "/", species_name, "_superfamily_div_plot.pdf"), device = "pdf", width = 12.85, height = 8.5)
================================================
FILE: scripts/extract_align.py
================================================
import argparse
import shutil
import re
import os
import subprocess
from Bio import SeqIO
import pandas as pd
import numpy as np
from pybedtools import BedTool
from pyfaidx import Fasta
import logging
import time
import datetime
pd.options.mode.chained_assignment = None # default='warn'
LOGGER = logging.getLogger(__name__)
## Set up input arguments
def get_args():
parser = argparse.ArgumentParser(description="Will process a blast output generated using a file of putative TEs (usually generated by RepeatModeler. For each putative consensus in the input putative TE library, it will generate an aligned file with N buffered instances from the queried genome, the input consensus, and, if requested, a new revised and extended consensus for inspection.", formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser.add_argument('-g', '--genome_fasta', type=str, help='Name of the fasta formatted genome to be queried.', required=True)
parser.add_argument('-b', '--blastfile', type=str, help='Blast output to be used. Must be formatted using "outfmt 6".', required = True)
parser.add_argument('-l', '--library', type=str, help='Library of putative TE consensus sequences to be extracted and aligned. Must be in fasta format with no # or / in the headers.', required = True)
parser.add_argument('-lb', '--leftbuffer', type=int, help='Left buffer size. The number of bp of flanking sequence for each hit to be extracted along with the hit. Optional, Default = 1000', default = 1000)
parser.add_argument('-rb', '--rightbuffer', type=int, help='Right beffer size. The number of bp of flanking sequence for each hit to be extracted along with the hit. Optional, Default = 1000', default = 1000)
parser.add_argument('-n', '--hitnumber', type=int, help='The number of hits to be exracted. Optional. Default = 50.', default = 50)
parser.add_argument('-a', '--align', type=str, help='Align the output fasta file, y or n?. Default is y.', default = 'y')
parser.add_argument('-t', '--trimal', type=str, help='Use trimal to remove low-aligning regions, y or n? Trimal can sometimes encounter an error that prevents it from working, this results in an empty file in downstream analyses. Default is y.', default = 'y')
parser.add_argument('-e', '--emboss', type=str, help='Generate a trimal/emboss consensus, y or n. Optional.', default = 'y')
parser.add_argument("-log", "--log_level", default="INFO")
args = parser.parse_args()
GENOMEFA = args.genome_fasta
BLAST = args.blastfile
LIB = args.library
LBUFFER = args.leftbuffer
RBUFFER = args.rightbuffer
HITNUM = args.hitnumber
ALIGN = args.align
TRIMAL = args.trimal
EMBOSS = args.emboss
LOG = args.log_level
return GENOMEFA, BLAST, LIB, LBUFFER, RBUFFER, HITNUM, ALIGN, TRIMAL, EMBOSS, LOG
## Create TE outfiles function. Creates files for populating with blast hits.
def CREATE_TE_OUTFILES(LIBRARY):
for record in SeqIO.parse(LIBRARY, 'fasta'):
NEWID = re.sub('#', '__', record.id)
NEWID = re.sub('/', '___', NEWID)
record.id = 'CONSENSUS-' + NEWID
record.description = ''
SeqIO.write(record, 'tmpTEfiles/' + NEWID + '.fa', 'fasta')
## Organize blast hits function. Will read in blast file, sort based on e-value and bitscore, deterine top BUFFER hits for extraction, extract, and combine with TE file from previous function.
def EXTRACT_BLAST_HITS(GENOME, BLAST, LBUFFER, RBUFFER, HITNUM):
##Read in blast data
BLASTDF = pd.read_table(BLAST, sep='\t', names=['QUERYNAME', 'SCAFFOLD', 'C', 'D', 'E', 'F', 'QUERYSTART', 'QUERYSTOP', 'SCAFSTART', 'SCAFSTOP', 'E-VALUE', 'BITSCORE'])
##Convert to bed format
BLASTBED = BLASTDF[['SCAFFOLD', 'SCAFSTART', 'SCAFSTOP', 'QUERYNAME', 'E-VALUE', 'BITSCORE']]
BLASTBED.insert(5, 'STRAND', '+')
BLASTBED.loc[BLASTBED.SCAFSTOP < BLASTBED.SCAFSTART, 'STRAND'] = '-'
BLASTBED.SCAFSTART, BLASTBED.SCAFSTOP = np.where(BLASTBED.SCAFSTART > BLASTBED.SCAFSTOP, [BLASTBED.SCAFSTOP, BLASTBED.SCAFSTART], [BLASTBED.SCAFSTART, BLASTBED.SCAFSTOP])
##Generate list of query names
QUERYLIST = BLASTBED.QUERYNAME.unique()
LOGGER.info('There are ' + str(len(QUERYLIST)) + ' consensus sequences to process')
# COUNTER = 1
##Sort subsets of df based on query names, keep the top BUFFER hits, make bedfiles, extract, and combine
for QUERY in QUERYLIST:
# LOGGER.info('Extracting for TE: ' + str(COUNTER))
QUERYFRAME = BLASTBED[BLASTBED['QUERYNAME'] == QUERY]
QUERYFRAME = QUERYFRAME.sort_values(by=['E-VALUE', 'BITSCORE'], ascending=[True, False])
QUERYFRAME = QUERYFRAME.head(HITNUM)
QUERYFRAMESAVE = 'tmpbedfiles/' + QUERY + '.bed'
QUERYFRAME.to_csv('tmpbedfiles/' + QUERY + '.bed', sep='\t', header=False, index=False)
CURRENTBED = BedTool('tmpbedfiles/' + QUERY + '.bed')
GENOMEPREFIX = os.path.splitext(GENOME)[0]
SLOPBED = CURRENTBED.slop(g=GENOMEPREFIX + '.fai', l=LBUFFER, r=RBUFFER, output='tmpbedfiles/' + QUERY + '.slop')
SLOPBED = BedTool('tmpbedfiles/' + QUERY + '.slop')
FASTA = SLOPBED.sequence(fi=GENOME, s=True)
FASTASAVE = SLOPBED.save_seqs('tmpextracts/' + QUERY + '.fa')
os.remove('tmpbedfiles/' + QUERY + '.slop')
os.remove('tmpbedfiles/' + QUERY + '.bed')
subprocess.run('cat {} {} >{}'.format('tmpextracts/' + QUERY + '.fa', 'tmpTEfiles/' + QUERY +'.fa', 'catTEfiles/' + QUERY +'.fa'), shell=True)
# COUNTER = COUNTER + 1
##Alignment function
def MUSCLE(TOALIGN):
TOALIGNPREFIX = os.path.splitext(TOALIGN)[0]
SOFTWARE = '/usr/bin'
subprocess.check_call(SOFTWARE + '/muscle -in {} -out {}'.format('catTEfiles/' + TOALIGN, 'muscle/' + TOALIGNPREFIX + '.fa'), shell=True)
##Consensus generation function
def CONSENSUSGEN(ALIGNED):
FILEPREFIX = os.path.splitext(ALIGNED)[0]
SOFTWARE = '/home/toby/Programs/'
if TRIMAL == 'y':
subprocess.run(SOFTWARE + 'trimal/source/trimal -in {} -gt 0.6 -cons 60 -fasta -out {}'.format('muscle/' + ALIGNED, 'muscle/' + FILEPREFIX + '_trimal.fa'), shell=True)
subprocess.run(SOFTWARE + 'EMBOSS-6.6.0/emboss/cons -sequence muscle/' + FILEPREFIX + '_trimal.fa -outseq muscle/' + FILEPREFIX + '_cons.fa -name ' + FILEPREFIX + '_cons -plurality 3 -identity 3', shell=True)
subprocess.run('cat {} {} >{}'.format('muscle/' + FILEPREFIX + '_trimal.fa', 'muscle/' + FILEPREFIX + '_cons.fa', 'consensusfiles/' + FILEPREFIX + '_cons.fa'), shell=True)
if TRIMAL == 'n':
# subprocess.run(SOFTWARE + 'trimal/source/trimal -in {} -gt 0.6 -cons 60 -fasta -out {}'.format('muscle/' + ALIGNED, 'muscle/' + FILEPREFIX + '_trimal.fa'), shell=True)
subprocess.run(SOFTWARE + 'EMBOSS-6.6.0/emboss/cons -sequence muscle/' + ALIGNED + ' -outseq muscle/' + FILEPREFIX + '_cons.fa -name ' + FILEPREFIX + '_cons -plurality 3 -identity 3', shell=True)
subprocess.run('cat {} {} >{}'.format('muscle/' + ALIGNED, 'muscle/' + FILEPREFIX + '_cons.fa', 'consensusfiles/' + FILEPREFIX + '_cons.fa'), shell=True)
def DIRS(DIR):
if os.path.exists(DIR):
shutil.rmtree(DIR)
os.mkdir(DIR)
####MAIN function
def main():
##Get input arguments
GENOMEFA, BLAST, LIB, LBUFFER, RBUFFER, HITNUM, ALIGN, TRIMAL, EMBOSS, LOG = get_args()
# Setup logging and script timing
handlers = [logging.FileHandler('extract_align.log'), logging.StreamHandler()]
logging.basicConfig(format='', handlers = handlers)
logging.getLogger().setLevel(getattr(logging, LOG.upper()))
start_time = time.time()
LOGGER.info('#\n# extract_align.py\n#')
LOGGER.info('Genome file: ' + GENOMEFA)
LOGGER.info('Blast file: ' + BLAST)
LOGGER.info('TE library: ' + LIB)
LOGGER.info('Left buffer size: ' + str(LBUFFER))
LOGGER.info('Right buffer size: ' + str(RBUFFER))
LOGGER.info('Number of hits evaluated: ' + str(HITNUM))
LOGGER.info('Muscle alignment = ' + ALIGN)
LOGGER.info('Trimal processing = ' + TRIMAL)
LOGGER.info('Emboss consensus = ' + EMBOSS)
LOGGER.info('Log level: ' + LOG)
## Index the genome
LOGGER.info('Indexing the genome')
GENOMEIDX = Fasta(GENOMEFA)
GENOMEPREFIX = os.path.splitext(GENOMEFA)[0]
FAIDX = pd.read_table(GENOMEFA + '.fai', sep='\t', names=['one', 'two', 'three', 'four', 'five'])
FAIDX = FAIDX[['one', 'two']]
FAIDX.to_csv(GENOMEPREFIX + '.fai', sep='\t', header=False, index=False)
## Set up directories
LOGGER.info('Creating tmp and permanent directories')
DIRS('tmpTEfiles')
DIRS('tmpbedfiles')
if ALIGN == 'y':
DIRS('muscle')
if EMBOSS == 'y':
DIRS('consensusfiles')
DIRS('tmpextracts')
DIRS('catTEfiles')
##Determine optional arguments and print to screen.
if ALIGN == 'n' and EMBOSS == 'y':
LOGGER.info('Input is contradictory. Generating a new consensus with emboss requires muscle alignment.')
elif ALIGN == 'y' and EMBOSS == 'y':
LOGGER.info('Output files will be aligned and a new consensus will be generated with emboss and trimal.')
elif ALIGN == 'y' and EMBOSS == 'n':
LOGGER.info('Output files will be aligned but without a new emboss/trimal consensus.')
elif ALIGN == 'n' and EMBOSS == 'n':
LOGGER.info('Extractions will be made but no alignment.')
else:
LOGGER.info('Invalid input for either align, or emboss, or both.')
##Create TE out files to populate with blast hits
CREATE_TE_OUTFILES(LIB)
##Extract hits and combine them with the TE out files if flagged
EXTRACT_BLAST_HITS(GENOMEFA, BLAST, LBUFFER, RBUFFER, HITNUM)
##Align extracted hits if flagged
if ALIGN == 'y':
COUNTER = 1
for FILE in os.listdir('tmpextracts'):
LOGGER.info('Aligning TE: ' + str(COUNTER))
MUSCLE(FILE)
COUNTER = COUNTER + 1
##Generate new consensus with emboss if flagged
if EMBOSS == 'y':
for FILE in os.listdir('muscle'):
CONSENSUSGEN(FILE)
##Remove empty tmp directories and unneeded files
LOGGER.info('Removing tmp directories and extraneous files')
shutil.rmtree('tmpbedfiles/')
shutil.rmtree('tmpextracts/')
shutil.rmtree('tmpTEfiles/')
#FILES = [F for F in os.listdir('muscle/') if F.endswith('_cons.fa')]
#for FILE in FILES:
# os.remove('muscle/' + FILE)
#FILES = [F for F in os.listdir('muscle/') if F.endswith('_trimal.fa')]
#for FILE in FILES:
# os.remove('muscle/' + FILE)
end_time = time.time()
LOGGER.info('Run time: ' + str(datetime.timedelta(seconds=end_time-start_time)))
#
# Wrap script functionality in main() to avoid automatic execution
# when imported ( e.g. when help is called on file )
#
if __name__ =="__main__":main()
================================================
FILE: scripts/faswap.py
================================================
#!/usr/bin/env python3
import sys
import csv
args = sys.argv
dictionary = args[1]
fastaIn = args[2]
a=dict(csv.reader(open(dictionary),delimiter="\t"))
lines=open(fastaIn).read().split("\n")
lines=map(lambda x: ">"+a[x[1:]] if x and x[0]==">" else x,lines);print("\n".join(lines))
================================================
FILE: scripts/filteringOverlappingRepeats.R
================================================
# load libraries
suppressPackageStartupMessages(library(GenomicRanges))
suppressPackageStartupMessages(library(ape))
suppressPackageStartupMessages(library(tidyverse))
# set options
options(scipen = 100, stringsAsFactors = FALSE)
#get inputs
args <- commandArgs()
# print(args)
gff.in <- args[6]
gff.out <- args[7]
# read gff
input <- read.gff(gff.in)
# cut overlapping regions in half
input %<>%
arrange(seqid, start) %>%
mutate(new.start = case_when(seqid == lag(seqid) & start < lag(end) & end > lag(end) ~ as.integer((start + ((lag(end) - start)/2)) + 1),
seqid == lag(seqid) & start == lag(end) ~ as.integer(start + 1),
.default = start),
new.end = case_when(seqid == lead(seqid) & end > lead(start) & end < lead(end) ~ as.integer((end - (end - lead(start))/2)),
seqid == lead(seqid) & end == lead(start) ~ as.integer(end),
.default = end)) %>%
mutate(start = new.start,
end = new.end) %>%
select(-c(new.start, new.end))
# identify nested and unnested TEs
# nest detect function
detect_nesting <- function(df) {
df %>%
arrange(seqid, start) %>%
mutate(
nested = case_when(
seqid == lag(seqid) &
start > lag(start) &
end < lag(end) ~ "FULLY_NESTED",
TRUE ~ "NOT_NESTED"
)
)
}
# iterate to detect all nestings
all_nested <- list()
iteration <- 1
current <- input
repeat {
print(paste0("Searching for Nested TEs. Iteration:", iteration))
annotated <- detect_nesting(current)
newly_nested <- annotated %>%
filter(nested == "FULLY_NESTED") %>%
mutate(nesting_round = iteration)
if (nrow(newly_nested) == 0) break
all_nested[[iteration]] <- newly_nested
current <- annotated %>%
filter(nested == "NOT_NESTED") %>%
select(-nested)
iteration <- iteration + 1
}
nested_only <- bind_rows(all_nested)
unnested_only <- current
# recombine to make GFF
if (nrow(nested_only) > 0) {
nested_only <- nested_only %>%
mutate(attributes = paste0(attributes, ";nested=", nested, "-round", nesting_round)) %>%
select(-c(nested, nesting_round))
output.gff <- bind_rows(nested_only, unnested_only) %>%
arrange(seqid, start)
} else {
# No nested TEs found — write unnested output directly
output.gff <- unnested_only %>%
arrange(seqid, start)
}
# Write Table
write.table(output.gff, gff.out, sep = "\t", quote = F, row.names = F, col.names = F)
================================================
FILE: scripts/headSwap.sh
================================================
#!/bin/bash
while getopts i:f:o: option
do
case "${option}"
in
i) input=${OPTARG};;
o) output=${OPTARG};;
esac
done
# 1 - Create Dictionary
SCRIPT_DIR=/data/toby/EarlGrey/scripts
grep ">" $input | awk '{printf("ctg_%01d %s\n", NR, $0)}' | sed 's/ /\t/g; s/>//g' | awk '{OFS="\t"}{print $2, $1}' > ${input}.dict
tr -d $'\r' < ${input}.dict > ${input}.dict.1 && mv ${input}.dict{.1,}
# 2 - Replace Fasta Headers
${SCRIPT_DIR}/faswap.py ${input}.dict $input > $output
================================================
FILE: scripts/install_r_packages.R
================================================
# Install R Packages
install.packages(c("BiocManager","optparse", "ape", "optparse"), repos="https://www.stats.bris.ac.uk/R/")
BiocManager::install(c("plyranges", "BSgenome"), update = FALSE)
================================================
FILE: scripts/makeGff.R
================================================
suppressPackageStartupMessages(library(tidyverse))
suppressPackageStartupMessages(library(data.table))
suppressPackageStartupMessages(library(magrittr))
args <- commandArgs()
# print(args)
bedIn <- args[6]
gffIn <- args[7]
gffOut <- args[8]
# test
#bedIn <- "~/projects/monarch/sortingSINEs/repeatMasker/merge/danausPlexippus.filteredRepeats.bed"
#gffIn <- "/home/toby/projects/monarch/sortingSINEs/repeatMasker/merge/remerge/danausPlexippus.rmerge.gff.filtered"
#gffOut <- "/home/toby/projects/monarch/sortingSINEs/repeatMasker/merge/remerge/danausPlexippus.filteredRepeats.gff"
# step 1 - read in tables
tab1 <- read.table(bedIn)
tab2 <- read.table(gffIn, sep = "\t")
tab1$id <- paste(tab1$V1, tab1$V2, tab1$V3, tab1$V4, tab1$V5, sep = "_")
tab2$id <- paste(tab2$V1, (tab2$V4 - 1), tab2$V5, tab2$V3, tab2$V6, sep = "_")
tab2 <- tab2 %>% distinct()
fil <- tab2[tab2$id %in% tab1$id,]
fil <- fil[,c(1:9)]
fil$V8 <- "." #set phase to be Geneious-Friendly
for (i in 1:length(fil$V1)) {
if (fil$V5[i] < fil$V4[i]) {
start <- fil$V5[i]
end <- fil$V4[i]
fil$V4[i] <- start
fil$V5[i] <- end
}
}
fil <- fil %>%
mutate(V2 = case_when(V9 %like% "^ID=HLE*" ~ "HELIANO",
.default = "Earl_Grey"),
V9 = toupper(V9),
ID = paste0(sub(".*;ID=", "ID=", sub(";SHORTTE.*$", "", V9)), "_", row_number()),
Name = paste0(sub(".*ID=", "Name=", sub(";SHORTTE.*$", "", V9))),
V9 = paste0(ID, ";", Name, ";", sub("ID=.*", "", V9), ";", sub(".*SHORTTE", "SHORTTE", V9))) %>%
select(-c(ID, Name))
write.table(fil, gffOut, quote = FALSE, col.names = FALSE, row.names = FALSE, sep = "\t")
================================================
FILE: scripts/mergeRepeats.R
================================================
suppressPackageStartupMessages(library(tidyverse))
suppressPackageStartupMessages(library(plyr))
suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(magrittr))
suppressPackageStartupMessages(library(data.table))
options(scipen = 100, stringsAsFactors = FALSE)
#####
args <- commandArgs()
# print(args)
file <- args[6]
output <- args[7]
genomeSize <- args[8]
output2 <- args[9]
filtBed <- args[10]
filtSum <- args[11]
lowend <- args[12]
# Step 1 - Read in rmerge table
repeats <- fread(file)
colnames(repeats) <- c("Scaffold", "ReMa", "Repeat", "Start", "End", "Score", "Strand", "Dot", "Desc")
# Step 2 - Calculate size of repeats
# calculate size of all repeats
repeats$size <- abs((repeats$End - repeats$Start)) + 1
repeats$Start <- as.double(repeats$Start) - 1
repeats$End <- as.double(repeats$End)
mergedRepeats <- repeats
mergedRepeatsOut <- repeats[,c(1,4,5,3,6,7)]
write.table(mergedRepeatsOut, file = output, quote = FALSE, row.names = FALSE, sep = "\t", col.names = FALSE)
# Generate RM style summary table for all unnested repeats, and nested ones too
sine <- mergedRepeats %>% filter(str_detect(Repeat, "SINE")) %>% filter(! str_detect(Desc, "FULLY_NESTED")) %>% distinct()
line <- mergedRepeats %>% filter(str_detect(Repeat, "LINE")) %>% filter(! str_detect(Repeat, "Penelope")) %>% filter(! str_detect(Desc, "FULLY_NESTED")) %>% distinct()
ltr <- mergedRepeats %>% filter(str_detect(Repeat, "LTR")) %>% filter(! str_detect(Desc, "FULLY_NESTED")) %>% distinct()
dna <- mergedRepeats %>% filter(str_detect(Repeat, "DNA")) %>% filter(! str_detect(Desc, "FULLY_NESTED")) %>% distinct()
rc <- mergedRepeats %>% filter(str_detect(Repeat, "RC")) %>% filter(! str_detect(Desc, "FULLY_NESTED")) %>% distinct()
ple <- mergedRepeats %>% filter(str_detect(Repeat, "Penelope")) %>% filter(! str_detect(Desc, "FULLY_NESTED")) %>% distinct()
unknown <- mergedRepeats %>% filter(str_detect(Repeat, "Unknown")) %>% filter(! str_detect(Desc, "FULLY_NESTED")) %>% distinct()
other <- mergedRepeats %>% filter(str_detect(Repeat, "ARTEFACT|Low|Retrop|Satellite|Simple|RNA")) %>% filter(! str_detect(Desc, "FULLY_NESTED")) %>% distinct()
other <- other[! other$Repeat == "SINE",]
sine.nest <- mergedRepeats %>% filter(str_detect(Repeat, "SINE")) %>% filter(str_detect(Desc, "FULLY_NESTED")) %>% distinct()
line.nest <- mergedRepeats %>% filter(str_detect(Repeat, "LINE")) %>% filter(! str_detect(Repeat, "Penelope")) %>% filter(str_detect(Desc, "FULLY_NESTED")) %>% distinct()
ltr.nest <- mergedRepeats %>% filter(str_detect(Repeat, "LTR")) %>% filter(str_detect(Desc, "FULLY_NESTED")) %>% distinct()
dna.nest <- mergedRepeats %>% filter(str_detect(Repeat, "DNA")) %>% filter(str_detect(Desc, "FULLY_NESTED")) %>% distinct()
rc.nest <- mergedRepeats %>% filter(str_detect(Repeat, "RC")) %>% filter(str_detect(Desc, "FULLY_NESTED")) %>% distinct()
ple.nest <- mergedRepeats %>% filter(str_detect(Repeat, "Penelope")) %>% filter(str_detect(Desc, "FULLY_NESTED")) %>% distinct()
unknown.nest <- mergedRepeats %>% filter(str_detect(Repeat, "Unknown")) %>% filter(str_detect(Desc, "FULLY_NESTED")) %>% distinct()
other.nest <- mergedRepeats %>% filter(str_detect(Repeat, "ARTEFACT|Low|Retrop|Satellite|Simple|RNA")) %>% filter(str_detect(Desc, "FULLY_NESTED")) %>% distinct()
other.nest <- other.nest[! other.nest$Repeat == "SINE",]
# Total bp coverage of each class
sineCov <- sum(as.numeric(as.character(sine$size)))
lineCov <- sum(as.numeric(as.character(line$size)))
ltrCov <- sum(as.numeric(as.character(ltr$size)))
dnaCov <- sum(as.numeric(as.character(dna$size)))
rcCov <- sum(as.numeric(as.character(rc$size)))
pleCov <- sum(as.numeric(as.character(ple$size)))
unknownCov <- sum(as.numeric(as.character(unknown$size)))
otherCov <- sum(as.numeric(as.character(other$size)))
totalCov <- (sineCov + lineCov + ltrCov + dnaCov + rcCov + pleCov + unknownCov + otherCov)
unmaskCov <- (as.numeric(genomeSize) - as.numeric(totalCov))
sineCov.nest <- sum(as.numeric(as.character(sine.nest$size)))
lineCov.nest <- sum(as.numeric(as.character(line.nest$size)))
ltrCov.nest <- sum(as.numeric(as.character(ltr.nest$size)))
dnaCov.nest <- sum(as.numeric(as.character(dna.nest$size)))
rcCov.nest <- sum(as.numeric(as.character(rc.nest$size)))
pleCov.nest <- sum(as.numeric(as.character(ple.nest$size)))
unknownCov.nest <- sum(as.numeric(as.character(unknown.nest$size)))
otherCov.nest <- sum(as.numeric(as.character(other.nest$size)))
# Number of elements
sineTal <- as.numeric(tally(sine))
lineTal <- as.numeric(tally(line))
ltrTal <- as.numeric(tally(ltr))
dnaTal <- as.numeric(tally(dna))
rcTal <- as.numeric(tally(rc))
pleTal <- as.numeric(tally(ple))
unknownTal <- as.numeric(tally(unknown))
otherTal <- as.numeric(tally(other))
totalTal <- (sineTal + lineTal + ltrTal + dnaTal + rcTal + pleTal + unknownTal + otherTal)
sineTal.nest <- as.numeric(tally(sine.nest))
lineTal.nest <- as.numeric(tally(line.nest))
ltrTal.nest <- as.numeric(tally(ltr.nest))
dnaTal.nest <- as.numeric(tally(dna.nest))
rcTal.nest <- as.numeric(tally(rc.nest))
pleTal.nest <- as.numeric(tally(ple.nest))
unknownTal.nest <- as.numeric(tally(unknown.nest))
otherTal.nest <- as.numeric(tally(other.nest))
# Calculate seq percentage
genomeSize <- as.numeric(genomeSize)
sinePer <- (sineCov / genomeSize) * 100
linePer <- (lineCov / genomeSize) * 100
ltrPer <- (ltrCov / genomeSize) * 100
dnaPer <- (dnaCov / genomeSize) * 100
rcPer <- (rcCov / genomeSize) * 100
plePer <- (pleCov / genomeSize) * 100
unknownPer <- (unknownCov / genomeSize) * 100
otherPer <- (otherCov / genomeSize) * 100
totalPer <- (sinePer + linePer + ltrPer + dnaPer + rcPer + plePer + unknownPer + otherPer)
unmaskPer <- (100 - totalPer)
sinePer.nest <- (sineCov.nest / genomeSize) * 100
linePer.nest <- (lineCov.nest / genomeSize) * 100
ltrPer.nest <- (ltrCov.nest / genomeSize) * 100
dnaPer.nest <- (dnaCov.nest / genomeSize) * 100
rcPer.nest <- (rcCov.nest / genomeSize) * 100
plePer.nest <- (pleCov.nest / genomeSize) * 100
unknownPer.nest <- (unknownCov.nest / genomeSize) * 100
otherPer.nest <- (otherCov.nest / genomeSize) * 100
# Make Table
eleTyp <- c("SINE", "LINE", "LTR", "DNA", "RC", "PLE", "Unknown", "Total Interspersed Repeat", "Other", "Unmasked",
"SINE-nested", "LINE-nested", "LTR-nested", "DNA-nested", "RC-nested", "PLE-nested", "Unknown-nested", "Other-nested")
numEle <- c(sineTal, lineTal, ltrTal, dnaTal, rcTal, pleTal, unknownTal, totalTal, otherTal, as.character("notApplicable"),
sineTal.nest, lineTal.nest, ltrTal.nest, dnaTal.nest, rcTal.nest, pleTal.nest, unknownTal.nest, otherTal.nest)
lenOcc <- c(sineCov, lineCov, ltrCov, dnaCov, rcCov, pleCov, unknownCov, totalCov, otherCov, unmaskCov,
sineCov.nest, lineCov.nest, ltrCov.nest, dnaCov.nest, rcCov.nest, pleCov.nest, unknownCov.nest, otherCov.nest)
perSeq <- c(sinePer, linePer, ltrPer, dnaPer, rcPer, plePer, unknownPer, totalPer, otherPer, unmaskPer,
sinePer.nest, linePer.nest, ltrPer.nest, dnaPer.nest, rcPer.nest, plePer.nest, unknownPer.nest, otherPer.nest)
size <- c(rep(genomeSize, 18))
summary <- data.frame(eleTyp, numEle, lenOcc, perSeq, size)
colnames(summary) <- c("Family", "Number of Elements", "Length Occupied", "Percentage of Sequence", "GenomeSize")
# Print table
write.table(summary, output2, col.names = TRUE, row.names = FALSE, quote = FALSE, sep = "\t")
filteredRepeatsOut <- mergedRepeatsOut
filteredRepeatsOut$length <- abs(filteredRepeatsOut$End - filteredRepeatsOut$Start)
if (lowend == "yes") {
filteredRepeatsOut <- filteredRepeatsOut[filteredRepeatsOut$length > 100,]
}
# if end coordinate is before start, switch
filteredRepeatsOut2 <- filteredRepeatsOut[,1:6] %>%
mutate(nStart = ifelse(End < Start, End, Start),
nEnd = ifelse(End < Start, Start, End)) %>%
mutate(Start = nStart,
End = nEnd) %>%
select(! c(nStart, nEnd))
write.table(filteredRepeatsOut2, file = filtBed, quote = FALSE, row.names = FALSE, sep = "\t", col.names = FALSE)
# Generate RM style summary table for large Repeats
sine <- mergedRepeats %>% filter(str_detect(Repeat, "SINE")) %>% filter(! str_detect(Desc, "FULLY_NESTED")) %>% distinct()
line <- mergedRepeats %>% filter(str_detect(Repeat, "LINE")) %>% filter(! str_detect(Repeat, "Penelope")) %>% filter(! str_detect(Desc, "FULLY_NESTED")) %>% distinct()
ltr <- mergedRepeats %>% filter(str_detect(Repeat, "LTR")) %>% filter(! str_detect(Desc, "FULLY_NESTED")) %>% distinct()
dna <- mergedRepeats %>% filter(str_detect(Repeat, "DNA")) %>% filter(! str_detect(Desc, "FULLY_NESTED")) %>% distinct()
rc <- mergedRepeats %>% filter(str_detect(Repeat, "RC")) %>% filter(! str_detect(Desc, "FULLY_NESTED")) %>% distinct()
ple <- mergedRepeats %>% filter(str_detect(Repeat, "Penelope")) %>% filter(! str_detect(Desc, "FULLY_NESTED")) %>% distinct()
unknown <- mergedRepeats %>% filter(str_detect(Repeat, "Unknown")) %>% filter(! str_detect(Desc, "FULLY_NESTED")) %>% distinct()
other <- mergedRepeats %>% filter(str_detect(Repeat, "ARTEFACT|Low|Retrop|Satellite|Simple|RNA")) %>% filter(! str_detect(Desc, "FULLY_NESTED")) %>% distinct()
other <- other[! other$Repeat == "SINE",]
sine.nest <- mergedRepeats %>% filter(str_detect(Repeat, "SINE")) %>% filter(str_detect(Desc, "FULLY_NESTED")) %>% distinct()
line.nest <- mergedRepeats %>% filter(str_detect(Repeat, "LINE")) %>% filter(! str_detect(Repeat, "Penelope")) %>% filter(str_detect(Desc, "FULLY_NESTED")) %>% distinct()
ltr.nest <- mergedRepeats %>% filter(str_detect(Repeat, "LTR")) %>% filter(str_detect(Desc, "FULLY_NESTED")) %>% distinct()
dna.nest <- mergedRepeats %>% filter(str_detect(Repeat, "DNA")) %>% filter(str_detect(Desc, "FULLY_NESTED")) %>% distinct()
rc.nest <- mergedRepeats %>% filter(str_detect(Repeat, "RC")) %>% filter(str_detect(Desc, "FULLY_NESTED")) %>% distinct()
ple.nest <- mergedRepeats %>% filter(str_detect(Repeat, "Penelope")) %>% filter(str_detect(Desc, "FULLY_NESTED")) %>% distinct()
unknown.nest <- mergedRepeats %>% filter(str_detect(Repeat, "Unknown")) %>% filter(str_detect(Desc, "FULLY_NESTED")) %>% distinct()
other.nest <- mergedRepeats %>% filter(str_detect(Repeat, "ARTEFACT|Low|Retrop|Satellite|Simple|RNA")) %>% filter(str_detect(Desc, "FULLY_NESTED")) %>% distinct()
other.nest <- other.nest[! other.nest$Repeat == "SINE",]
# Total bp coverage of each class
sineCov <- sum(as.numeric(as.character(sine$size)))
lineCov <- sum(as.numeric(as.character(line$size)))
ltrCov <- sum(as.numeric(as.character(ltr$size)))
dnaCov <- sum(as.numeric(as.character(dna$size)))
rcCov <- sum(as.numeric(as.character(rc$size)))
pleCov <- sum(as.numeric(as.character(ple$size)))
unknownCov <- sum(as.numeric(as.character(unknown$size)))
otherCov <- sum(as.numeric(as.character(other$size)))
totalCov <- (sineCov + lineCov + ltrCov + dnaCov + rcCov + pleCov + unknownCov + otherCov)
unmaskCov <- (as.numeric(genomeSize) - as.numeric(totalCov))
sineCov.nest <- sum(as.numeric(as.character(sine.nest$size)))
lineCov.nest <- sum(as.numeric(as.character(line.nest$size)))
ltrCov.nest <- sum(as.numeric(as.character(ltr.nest$size)))
dnaCov.nest <- sum(as.numeric(as.character(dna.nest$size)))
rcCov.nest <- sum(as.numeric(as.character(rc.nest$size)))
pleCov.nest <- sum(as.numeric(as.character(ple.nest$size)))
unknownCov.nest <- sum(as.numeric(as.character(unknown.nest$size)))
otherCov.nest <- sum(as.numeric(as.character(other.nest$size)))
# Number of elements
sineTal <- as.numeric(tally(sine))
lineTal <- as.numeric(tally(line))
ltrTal <- as.numeric(tally(ltr))
dnaTal <- as.numeric(tally(dna))
rcTal <- as.numeric(tally(rc))
pleTal <- as.numeric(tally(ple))
unknownTal <- as.numeric(tally(unknown))
otherTal <- as.numeric(tally(other))
totalTal <- (sineTal + lineTal + ltrTal + dnaTal + rcTal + pleTal + unknownTal + otherTal)
sineTal.nest <- as.numeric(tally(sine.nest))
lineTal.nest <- as.numeric(tally(line.nest))
ltrTal.nest <- as.numeric(tally(ltr.nest))
dnaTal.nest <- as.numeric(tally(dna.nest))
rcTal.nest <- as.numeric(tally(rc.nest))
pleTal.nest <- as.numeric(tally(ple.nest))
unknownTal.nest <- as.numeric(tally(unknown.nest))
otherTal.nest <- as.numeric(tally(other.nest))
# Calculate seq percentage
genomeSize <- as.numeric(genomeSize)
sinePer <- (sineCov / genomeSize) * 100
linePer <- (lineCov / genomeSize) * 100
ltrPer <- (ltrCov / genomeSize) * 100
dnaPer <- (dnaCov / genomeSize) * 100
rcPer <- (rcCov / genomeSize) * 100
plePer <- (pleCov / genomeSize) * 100
unknownPer <- (unknownCov / genomeSize) * 100
otherPer <- (otherCov / genomeSize) * 100
totalPer <- (sinePer + linePer + ltrPer + dnaPer + rcPer + plePer + unknownPer + otherPer)
unmaskPer <- (100 - totalPer)
sinePer.nest <- (sineCov.nest / genomeSize) * 100
linePer.nest <- (lineCov.nest / genomeSize) * 100
ltrPer.nest <- (ltrCov.nest / genomeSize) * 100
dnaPer.nest <- (dnaCov.nest / genomeSize) * 100
rcPer.nest <- (rcCov.nest / genomeSize) * 100
plePer.nest <- (pleCov.nest / genomeSize) * 100
unknownPer.nest <- (unknownCov.nest / genomeSize) * 100
otherPer.nest <- (otherCov.nest / genomeSize) * 100
# Make Table
eleTyp <- c("SINE", "LINE", "LTR", "DNA", "RC", "PLE", "Unknown", "Total Interspersed Repeat", "Other", "Unmasked",
"SINE-nested", "LINE-nested", "LTR-nested", "DNA-nested", "RC-nested", "PLE-nested", "Unknown-nested", "Other-nested")
numEle <- c(sineTal, lineTal, ltrTal, dnaTal, rcTal, pleTal, unknownTal, totalTal, otherTal, as.character("notApplicable"),
sineTal.nest, lineTal.nest, ltrTal.nest, dnaTal.nest, rcTal.nest, pleTal.nest, unknownTal.nest, otherTal.nest)
lenOcc <- c(sineCov, lineCov, ltrCov, dnaCov, rcCov, pleCov, unknownCov, totalCov, otherCov, unmaskCov,
sineCov.nest, lineCov.nest, ltrCov.nest, dnaCov.nest, rcCov.nest, pleCov.nest, unknownCov.nest, otherCov.nest)
perSeq <- c(sinePer, linePer, ltrPer, dnaPer, rcPer, plePer, unknownPer, totalPer, otherPer, unmaskPer,
sinePer.nest, linePer.nest, ltrPer.nest, dnaPer.nest, rcPer.nest, plePer.nest, unknownPer.nest, otherPer.nest)
size <- c(rep(genomeSize, 18))
summary2 <- data.frame(eleTyp, numEle, lenOcc, perSeq, size)
colnames(summary2) <- c("Family", "Number of Elements", "Length Occupied", "Percentage of Sequence", "GenomeSize")
# Print table
write.table(summary2, file = filtSum, col.names = TRUE, row.names = FALSE, quote = FALSE, sep = "\t")
================================================
FILE: scripts/rcMergeRepeats
================================================
#!/bin/bash
usage()
{
echo "Usage: rcMergeRepeats -f genome.fasta -s species -d output_directory -u RepeatMasker.out -q RepeatMasker.tbl -c repeatcraft.cfg -t ThreadNum -b dictionary -m yes|no -e heliano_gff -h"
}
while getopts f:s:d:u:q:t:b:m:e:h option
do
case "${option}"
in
f) genome=${OPTARG};;
s) spe=${OPTARG};;
d) dir=${OPTARG};;
u) rmOut=${OPTARG};;
q) table=${OPTARG};;
t) threads=${OPTARG};;
b) dict=${OPTARG};;
m) cutoff=${OPTARG};;
e) heli=${OPTARG};;
h) usage; exit;;
esac
done
##### Subprocess ltrfinder
ltrfinder()
{
mkdir ${dir}/${spe}LtrFinder/
cd ${dir}/${spe}LtrFinder/
${SCRIPT_DIR}/LTR_FINDER_parallel -seq $genome -threads $threads
cd ${dir}
}
##### Subprocess convert rmOut to GFF3 and then to GFF2
convert()
{
rmOutToGFF3.pl $rmOut > $dir/${spe}.out.gff3;
cat $dir/${spe}.out.gff3 | sed '1,2d; s/Target=/Target "Motif:/g' | awk '{OFS="\t"}{gsub(/$/,"\"", $10); print}' | sed "s/Target\t/Target /g; 1s/^/##gff-version 2\n##date xxxx-xx-xx\n/; s/\(.*\)\t/\1 /; s/\(.*\)\t/\1 /" > ${dir}/${spe}.out.gff2
sed -i '/##sequence-region.*/d' ${dir}/${spe}.out.gff2
}
##### Subprocess repeatcraft
repeatcraft()
{
mkdir ${dir}/${spe}RepeatCraft/
sed "s#ltr_finder_gff:.*#ltr_finder_gff: ${dir}/${spe}LtrFinder/$(basename ${genome}).finder.combine.gff3 #g" ${SCRIPT_DIR}/repeatCraft/example/repeatcraft.cfg > ${dir}/${spe}RepeatCraft/repeatcraft.cfg
python ${SCRIPT_DIR}/repeatCraft/repeatcraft.py -r ${dir}/${spe}.out.gff2 -u $rmOut -c ${dir}/${spe}RepeatCraft/repeatcraft.cfg -o ${dir}/${spe} -m strict
}
##### Subprocess mergeRepeats
mergeRepeats()
{
sort -s -k1,1 ${dir}/${spe}.rmerge.gff > ${dir}/${spe}.rmerge.gff.sorted
if [ -s "$heli" ]; then
bedtools intersect -v -a ${dir}/${spe}.rmerge.gff.sorted -b $heli > ${dir}/${spe}.rmerge.gff.noHeli
cat ${dir}/${spe}.rmerge.gff.noHeli $heli | sort -k4,4n > ${dir}/${spe}.rmerge.gff.sorted
fi
Rscript ${SCRIPT_DIR}/filteringOverlappingRepeats.R ${dir}/${spe}.rmerge.gff.sorted ${dir}/${spe}.rmerge.gff.filtered
Rscript ${SCRIPT_DIR}/mergeRepeats.R ${dir}/${spe}.rmerge.gff.filtered ${dir}/${spe}.mergedRepeats.bed $genSize ${dir}/${spe}.mergedRepeats.revisedTable ${dir}/${spe}.filteredRepeats.bed ${dir}/${spe}.filteredRepeats.summary $cutoff
}
##### Subprocess scafDeedPoll
scafDeedPoll()
{
cp ${dir}/${spe}.filteredRepeats.bed ${dir}/${spe}.filteredRepeats.bed.bak
python ${SCRIPT_DIR}/backSwap.py ${dir}/${spe}.filteredRepeats.bed ${dict} ${dir}/${spe}.filteredRepeats.bed.2
mv ${dir}/${spe}.filteredRepeats.bed{.2,}
python ${SCRIPT_DIR}/backSwapGFF.py ${dir}/${spe}.rmerge.gff.filtered ${dict} ${dir}/${spe}.rmerge.gff.filtered.2
mv ${dir}/${spe}.rmerge.gff.filtered{.2,}
Rscript ${SCRIPT_DIR}/makeGff.R ${dir}/${spe}.filteredRepeats.bed ${dir}/${spe}.rmerge.gff.filtered ${dir}/${spe}.filteredRepeats.gff
}
##### Subprocess Running Cow
runningCow()
{
echo "
) (
( ) )
) ( (
_______)_
.-'---------|
( C|/\/\/\/\/|
'-./\/\/\/\/|
'_________'
'-------'
<<< $stage >>>"
}
##### Subprocess Size
genomesize()
{
genSize=$(sed -n '4p' $table | rev | cut -f1,1 -d ':' | rev | sed 's/ bp.*//g; s/ //g')
}
##### Main
# Step 1 - Run ltr_finder
SCRIPT_DIR=/data/toby/EarlGrey/scripts
stage="Running LTR Finder"
runningCow
ltrfinder
# Step 2 - Generate all respective GFF2 files
stage="Generating GFF Files"
runningCow
convert
# Step 3 - Run RepeatCraft
stage="Running RepeatCraft"
runningCow
repeatcraft
# Step 4 - Merge mergedRepeats
stage="Resolving Overlapping Repeats"
runningCow
genomesize
mergeRepeats
scafDeedPoll
# Step 5 - Done
stage="Done!"
runningCow
================================================
FILE: scripts/rcMergeRepeatsLoose
================================================
#!/bin/bash
usage()
{
echo "Usage: rcMergeRepeats -f genome.fasta -s species -d output_directory -u RepeatMasker.out -q RepeatMasker.tbl -c repeatcraft.cfg -t ThreadNum -b dictionary -m yes|no -e heliano_gff -h"
}
while getopts f:s:d:u:q:t:b:m:e:h option
do
case "${option}"
in
f) genome=${OPTARG};;
s) spe=${OPTARG};;
d) dir=${OPTARG};;
u) rmOut=${OPTARG};;
q) table=${OPTARG};;
t) threads=${OPTARG};;
b) dict=${OPTARG};;
m) cutoff=${OPTARG};;
e) heli=${OPTARG};;
h) usage; exit;;
esac
done
##### Subprocess ltrfinder
ltrfinder()
{
mkdir ${dir}/${spe}LtrFinder/
cd ${dir}/${spe}LtrFinder/
${SCRIPT_DIR}/LTR_FINDER_parallel -seq $genome -threads $threads
cd ${dir}
}
##### Subprocess convert rmOut to GFF3 and then to GFF2
convert()
{
rmOutToGFF3.pl $rmOut > $dir/${spe}.out.gff3;
cat $dir/${spe}.out.gff3 | sed '1,2d; s/Target=/Target "Motif:/g' | awk '{OFS="\t"}{gsub(/$/,"\"", $10); print}' | sed "s/Target\t/Target /g; 1s/^/##gff-version 2\n##date xxxx-xx-xx\n/; s/\(.*\)\t/\1 /; s/\(.*\)\t/\1 /" > ${dir}/${spe}.out.gff2
sed -i '/##sequence-region.*/d' ${dir}/${spe}.out.gff2
}
##### Subprocess repeatcraft
repeatcraft()
{
mkdir ${dir}/${spe}RepeatCraft/
sed "s#ltr_finder_gff:.*#ltr_finder_gff: ${dir}/${spe}LtrFinder/$(basename ${genome}).finder.combine.gff3 #g" ${SCRIPT_DIR}/repeatCraft/example/repeatcraft.cfg > ${dir}/${spe}RepeatCraft/repeatcraft.cfg
python ${SCRIPT_DIR}/repeatCraft/repeatcraft.py -r ${dir}/${spe}.out.gff2 -u $rmOut -c ${dir}/${spe}RepeatCraft/repeatcraft.cfg -o ${dir}/${spe} -m loose
}
##### Subprocess mergeRepeats
mergeRepeats()
{
sort -s -k1,1 ${dir}/${spe}.rmerge.gff > ${dir}/${spe}.rmerge.gff.sorted
if [ -s "$heli" ]; then
bedtools intersect -v -a ${dir}/${spe}.rmerge.gff.sorted -b $heli > ${dir}/${spe}.rmerge.gff.noHeli
# for debugging
bedtools intersect -a ${dir}/${spe}.rmerge.gff.sorted -b $heli > ${dir}/${spe}.rmerge.gff.notFilteredHeli
cat ${dir}/${spe}.rmerge.gff.noHeli $heli | sort -k4,4n > ${dir}/${spe}.rmerge.gff.sorted
fi
Rscript ${SCRIPT_DIR}/filteringOverlappingRepeats.R ${dir}/${spe}.rmerge.gff.sorted ${dir}/${spe}.rmerge.gff.filtered
Rscript ${SCRIPT_DIR}/mergeRepeats.R ${dir}/${spe}.rmerge.gff.filtered ${dir}/${spe}.mergedRepeats.bed $genSize ${dir}/${spe}.mergedRepeats.revisedTable ${dir}/${spe}.filteredRepeats.bed ${dir}/${spe}.filteredRepeats.summary $cutoff
}
##### Subprocess scafDeedPoll
scafDeedPoll()
{
cp ${dir}/${spe}.filteredRepeats.bed ${dir}/${spe}.filteredRepeats.bed.bak
python ${SCRIPT_DIR}/backSwap.py ${dir}/${spe}.filteredRepeats.bed ${dict} ${dir}/${spe}.filteredRepeats.bed.2
mv ${dir}/${spe}.filteredRepeats.bed{.2,}
python ${SCRIPT_DIR}/backSwapGFF.py ${dir}/${spe}.rmerge.gff.filtered ${dict} ${dir}/${spe}.rmerge.gff.filtered.2
mv ${dir}/${spe}.rmerge.gff.filtered{.2,}
Rscript ${SCRIPT_DIR}/makeGff.R ${dir}/${spe}.filteredRepeats.bed ${dir}/${spe}.rmerge.gff.filtered ${dir}/${spe}.filteredRepeats.gff
}
##### Subprocess Running Cow
runningCow()
{
echo "
) (
( ) )
) ( (
_______)_
.-'---------|
( C|/\/\/\/\/|
'-./\/\/\/\/|
'_________'
'-------'
<<< $stage >>>"
}
##### Subprocess Size
genomesize()
{
genSize=$(sed -n '4p' $table | rev | cut -f1,1 -d ':' | rev | sed 's/ bp.*//g; s/ //g')
}
##### Main
# Step 1 - Run ltr_finder
SCRIPT_DIR=/data/toby/EarlGrey/scripts
stage="Running LTR Finder"
runningCow
ltrfinder
# Step 2 - Generate all respective GFF2 files
stage="Generating GFF Files"
runningCow
convert
# Step 3 - Run RepeatCraft
stage="Running RepeatCraft"
runningCow
repeatcraft
# Step 4 - Merge mergedRepeats
stage="Resolving Overlapping Repeats"
runningCow
genomesize
mergeRepeats
scafDeedPoll
# Step 5 - Done
stage="Done!"
runningCow
================================================
FILE: scripts/repeatCraft/Dockerfile
================================================
FROM python:3.6-alpine3.6
RUN apk add --no-cache bash
RUN mkdir /repeatcraft
COPY ./repeatcraft.py /repeatcraft
RUN mkdir /repeatcraft/helper
COPY ./helper/* /repeatcraft/helper/
WORKDIR "/repeatcraft"
================================================
FILE: scripts/repeatCraft/LICENSE
================================================
MIT License
Copyright (c) 2018 Wai Yee Wong
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: scripts/repeatCraft/README.md
================================================
[](https://travis-ci.org/niccw/repeatcraftp)
# RepeatCraft pipeline
#### Introduction
RepeatCraft pipeline incorporates tools for repeat element classification based on both sequence similarity and structural features. The pipeline defragments closely spanced repeat loci in the genomes, reconstructing longer copies thus allowing for a better annotation and sequence comparisons. A Shiny web application is also supplemented as a user interface for the RepeatCraft pipeline, and it allows easy access and interactive study of repeat elements. The web server is available at [http://131.130.65.134:3838/repeatcraft](http://131.130.65.134:3838/repeatcraft). You can also run it on your machine by installing the repeatcraft R package [https://github.com/niccw/repeatCraft](https://github.com/niccw/repeatCraft).
#### What does RepeatCraft do?
##### Merge repeat 'fragments'
After the repeat-based repeat identification and annotation (i.e. RepeatModeler and RepeatMasker (Smit, 2013-2015)), fragments of repeats beloning to the same repeat family can be found in certain genomic regions (for example: blcok b,c,d in track B). These repeats are closely spanced and non-overlap in the consensus sequence. Multiple counting of repeats may affect the result of downstream analysis (e.g. age estimation). Therefore, we develope RepeatCraft as a easy-to-use tool for mering repeat fragments, with flexible parameter setting. Group labels are first added to the attribute field of repeats to be merge, then RepeatCraft merges the repeat based on the TE family label and LTR label (see below). The strict mode only merges consective repeats while loose mode also works on non-executive repeats, the annotation information of other repeats (i.e. simple repeats) between the fragments are not modified.
##### Annotation from LTR_FINDER (optional but highly recommended)
RepeatCraft makes use of the annotation information from LTR_FINDER (Xu and Wang, 2017), a popular structural-based LTR (Long Tandem Repeat) identification tool. It identify the full-length LTR by searching structural features. Using the results from LTR_FINDER, RepeatCraft merges the LTR annotated in the RepeatMasker GFF (which belong to the same LTR subgroup) and further decreases the degree of fragmentization.
##### Other
Short repeats are also labelled in the early step. Repeats with length less than certain size (default = 100bp, except simple repeats,low complexity, satellite and snRNA; user can define the threshold for each repeat class in **mapfile.tsv**) would have a label of *shortTE=T* in the attribute column. The *shortTE* label are not used in merging repeat fragments, user can study the distribution of short read on the RepeatCraft web application by uploading the output.
###### General workflow of RepeatCraft
***
### Run RepeatCraft
Simply run RepeatCraft by calling **repeatcraft.py** (python >= 3.5), it takes the following arguments and options:
```
usage: repeatcraft.py [-h] [-r RMGFF] [-u RMOUT] [-c CONFIG] [-o OUTPUT]
[-m MODE]
RepeatCraft pipeline for improving repeat elements annotation by defragments
closely spanced repeat elements,based on sequence similarity and structural
features from different annotators
optional arguments:
-h, --help show this help message and exit
-r RMGFF, --rmgff RMGFF
RepeatMasker GFF
-u RMOUT, --rmout RMOUT
RepeatMasker OUT
-c CONFIG, --config CONFIG
Configuration file
-o OUTPUT, --output OUTPUT
Output file name
-m MODE, --mode MODE Merge mode. strict or loose. Default = loose
```
Example run:
`./repeatcraft.py -r example_input.gff -u example_input.out -c repeatcraft.cfg -o example -m strict`
### Run in docker container
**Build image from Dockerfile**
`git clone https://github.com/niccw/repeatcraftp.git && cd repeatcraftp`
`docker build -t repeatcraft:v1 .`
**Run repeatcraft inside docker conatiner**
`docker run --rm -v "PATH_TO_INPUT_FILES":/repeatcraft/run repeatcraft:v1 ./repeatcraft.py -r run/your_input.gff -u run/your_input.out -c run/your_repeatcraft.cfg -o example -m strict`
#### Inputs
The example folder contains the example inputs **example_input.gff**, **example_input.out** (GFF and OUT files from RepeatMasker), **example_ltrfinder.gff** (LTR_FINDER output in GFF/GTF format), **repeatcraft.cfg** (configuration file) and **mapfile.tsv** (which is used in short TEs labelling) . Currently RepeatCraft only supports GFF generated by RepeatMasker v4.0.7 and v4.0.6. RepeatCraft also works on GFF made by RepeatMasker using custom library generated by RepeatModeler. However, the repeat family names should not contain either space of pipe ("|").
The table output from LTR_FINDER can be converted to GFF using the `cnv_ltrfinder2gff.pl` script from the [DAWGPAWS pipeline](https://github.com/jestill/dawgpaws).
#### Outputs
RepeatCraft generates 3 outputs:
1) rclabel.gff : This is the reformatted GFF from Repeatmasker (type column is the repeat class but not "similarity"), with labels (TEgroup, LTRgroup) added to the attr. column. The TEs are not merged in this GFF.
2) rmerge.gff : This is the GFF after merging based on the labels in rclabel.gff.
3) summary.txt : This summary file lists the number of TEs (by class) before and after merging. It also shows how many TEs are merged by family (TEgroup) and/or based on information from LTR_FINDER (LTRgroup).
The example outputs (**example.rclabel.gff**, **example.rmerge.gff** and **example.summary.txt** can be found in the example folder)
By default, RepeatCraft is in "loose" merging mode. Repeats are merged based on the labels (TEgroup, LTRgroup) added from the previous steps, and the strand (+/-) will follow the orientation of the fragment with longest consensus overlapping in the group.
#### Parameters in repeatcraft.cfg
Other options can be set in the **repeatcraft.cgf**.
```
# Label short TEs
shortTE_size: 100
mapfile: None
# LTR grouping (based on LTR_FINER result)
ltr_finder_gff: example_ltrfinder.gff
max_LTR_size: 10000
LTR_flanking_size: 200
# TEs grouping
gap_size: 150
```
If **mapfile.tsv** (see example) is missing, a unite size (default = 100 bp) will by applied to all repeat classes. Since certain repeat classes (i.e. SINE) have shorter length compare to others (i.e. LTR or LINE), it is recommend to label different classes using different length threshold.
To incoporate the LTR_FINDER annotation, the path (absolute path is preferable, if relative path is used it should be relative to where the **repeatcraft.py** is in) should be stated in **ltr_finder_gff** field. If no LTR_FINDER annotation is available, specify None in that field. The **max_LTR_size** helps filter out overlapping LTR annotation from LTR_FINDER, and **LTR_flanking_size** allows shift of LTR annotated by RepeatMasker. The following figure shows how these two parameters work.
Finally, **gap_size** refer to the maximum distance between two constitute repeats to be merged, by default it is 150 bp.
***
#### References
JC Estill and JL Bennetzen. 2009. "The DAWGPAWS pipeline for the annotation of genes and transposable elements in plant genomes." Plant Methods. 5:8
Smit, AFA, Hubley, R & Green, P. RepeatMasker Open-4.0. 2013-2015