[
{
"path": ".github/ISSUE_TEMPLATE/bug_report.md",
"content": "---\nname: Bug report\nabout: Create a report to help us improve\ntitle: \"[BUG] XXX\"\nlabels: bug\nassignees: ''\n\n---\n\n**Describe the bug**\nA clear and concise description of what the bug is.\n\n**To Reproduce**\nSteps to reproduce the behavior:\n1. Describe the data you are using and provide a sample of your data if possible. For example, the paired-end reads are generated by 10x scATAC-seq. The read length is 50bp and the barcode length is 16bp.\n2. Get the Chromap version by running ```chromap -v``` and post it here.\n3. Provide the full command line you used to run Chromap.\n4. Provide the log output by Chromap and highlight the error message.\n\n**Expected behavior**\nA clear and concise description of what you expected to happen.\n\n**Screenshots**\nIf applicable, add screenshots to help explain your problem.\n\n**Environment (please complete the following information):**\n - OS: [e.g. Ubuntu 22.10]\n - Way you install Chromap [e.g. use Bioconda, download binary, build from source]\n - If you compiled Chromap from source yourself, please provide the compiler version [e.g. GCC 7.4.0]\n\n**Additional context**\nAdd any other context about the problem here.\n"
},
{
"path": ".github/workflows/ci.yml",
"content": "name: CI\n\non:\n push:\n branches: [ master ]\n pull_request:\n branches: [ master ]\n\nenv:\n DEVELOPER_DIR: /Applications/Xcode.app/Contents/Developer\n\njobs:\n ubuntu:\n runs-on: ubuntu-latest\n strategy:\n matrix:\n compiler: [g++, clang++]\n steps:\n - uses: actions/checkout@v2\n - name: install-deps\n run:\n sudo apt-get update; sudo apt-get install -y clang libomp5 libomp-dev\n - name: build-chromap\n run:\n make CXX=${{ matrix.compiler }}\n - name: test-chromap\n run:\n ./chromap -h\n\n macos:\n runs-on: macos-latest\n strategy:\n matrix:\n compiler: [clang++]\n steps:\n - uses: actions/checkout@v2\n - name: cache-openmp\n id: cache-openmp\n uses: actions/cache@v3\n with:\n path: openmp-install\n key: openmp-macos-install\n - name: build-openmp\n if: steps.cache-openmp.outputs.cache-hit != 'true'\n run: |\n wget https://github.com/llvm/llvm-project/releases/download/llvmorg-14.0.0/openmp-14.0.0.src.tar.xz\n tar -xf openmp-14.0.0.src.tar.xz\n cd openmp-14.0.0.src\n sed -i'' -e '/.size __kmp_unnamed_critical_addr/d' runtime/src/z_Linux_asm.S\n sed -i'' -e 's/__kmp_unnamed_critical_addr/___kmp_unnamed_critical_addr/g' runtime/src/z_Linux_asm.S\n mkdir -p build && cd build\n cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=install -DCMAKE_OSX_ARCHITECTURES=\"x86_64;arm64\" \\\n -DLIBOMP_ENABLE_SHARED=OFF -DLIBOMP_OMPT_SUPPORT=OFF -DLIBOMP_USE_HWLOC=OFF ..\n cmake --build . -j 3\n cmake --build . --target install\n mkdir $GITHUB_WORKSPACE/openmp-install\n cp -r install/* $GITHUB_WORKSPACE/openmp-install\n - name: install-openmp\n run: |\n sudo cp $GITHUB_WORKSPACE/openmp-install/include/* $DEVELOPER_DIR/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include\n sudo cp $GITHUB_WORKSPACE/openmp-install/lib/libomp.a $DEVELOPER_DIR/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib\n - name: build-chromap\n run:\n make CXX=${{ matrix.compiler }} CXXFLAGS=\"-arch x86_64 -isysroot $DEVELOPER_DIR/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -std=c++11 -Wall -O3 -Xclang -fopenmp -msse4.1\" LDFLAGS=\"-L$DEVELOPER_DIR/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib -rpath $DEVELOPER_DIR/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib -lm -lz -lomp\"\n - name: test-chromap\n run:\n ./chromap -h\n"
},
{
"path": "LICENSE",
"content": "MIT License\n\nCopyright (c) 2019 Haowen Zhang, Li Song, X. Shirley Liu, Heng Li\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"
},
{
"path": "Makefile",
"content": "CXX=g++\nCXXFLAGS=-std=c++11 -Wall -O3 -fopenmp -msse4.1\nLDFLAGS=-lm -lz\n\ncpp_source=sequence_batch.cc index.cc minimizer_generator.cc candidate_processor.cc alignment.cc feature_barcode_matrix.cc ksw.cc draft_mapping_generator.cc mapping_generator.cc mapping_writer.cc chromap.cc chromap_driver.cc\nsrc_dir=src\nobjs_dir=objs\nobjs+=$(patsubst %.cc,$(objs_dir)/%.o,$(cpp_source))\n\nexec=chromap\n\nifneq ($(asan),)\n\tCXXFLAGS+=-fsanitize=address -g\n\tLDFLAGS+=-fsanitize=address -ldl -g\nendif\n\nall: dir $(exec) \n\t\ndir:\n\tmkdir -p $(objs_dir)\n\n$(exec): $(objs)\n\t$(CXX) $(CXXFLAGS) $(objs) -o $(exec) $(LDFLAGS)\n\t\n$(objs_dir)/%.o: $(src_dir)/%.cc\n\t$(CXX) $(CXXFLAGS) -c $< -o $@\n\n.PHONY: clean\nclean:\n\t-rm -rf $(exec) $(objs_dir)\n"
},
{
"path": "README.md",
"content": "[](https://github.com/haowenz/chromap/actions/workflows/ci.yml) [](https://github.com/haowenz/chromap/blob/master/LICENSE) [](https://anaconda.org/bioconda/chromap) [](https://anaconda.org/bioconda/chromap) [](https://anaconda.org/bioconda/chromap)\n\n## Getting Started\n```sh\ngit clone https://github.com/haowenz/chromap.git\ncd chromap && make\n# create an index first and then map\n./chromap -i -r test/ref.fa -o ref.index\n./chromap -x ref.index -r test/ref.fa -1 test/read1.fq -2 test/read2.fq -o test.bed\n# use presets (no test data)\n./chromap --preset atac -x index -r ref.fa -1 read1.fq -2 read2.fq -o aln.bed # ATAC-seq reads\n./chromap --preset atac -x index -r ref.fa -1 read1.fq -2 read2.fq -o aln.bed \\\n -b barcode.fq.gz --barcode-whitelist whitelist.txt # scATAC-seq reads\n./chromap --preset chip -x index -r ref.fa -1 read1.fq -2 read2.fq -o aln.bed # ChIP-seq reads\n./chromap --preset hic -x index -r ref.fa -1 read1.fq -2 read2.fq -o aln.pairs # Hi-C reads and pairs output\n./chromap --preset hic -x index -r ref.fa -1 read1.fq -2 read2.fq --SAM -o aln.sam # Hi-C reads and SAM output\n```\n## Table of Contents\n\n- [Getting Started](#started)\n- [User Guide](#uguide)\n - [Installation](#install)\n - [General usage](#general)\n - [Use cases](#cases)\n - [Map ChIP-seq short reads](#map-chip)\n - [Map ATAC-seq/scATAC-seq short reads](#map-atac)\n - [Map Hi-C short reads](#map-hic)\n - [Summarizing mapping statistics/quality control](#atacseq-qc)\n - [Summary File](#summaryfile)\n - [Estimating FRiP](#estfrip)\n - [Features to assist in doublet detection](#doublet)\n - [Getting help](#help)\n - [Citing Chromap](#cite)\n\n## User Guide\n\nChromap is an ultrafast method for aligning and preprocessing high throughput chromatin profiles. Typical use cases include: (1) trimming sequencing adapters, mapping bulk ATAC-seq or ChIP-seq genomic reads to the human genome and removing duplicates; (2) trimming sequencing adapters, mapping single cell ATAC-seq genomic reads to the human genome, correcting barcodes, removing duplicates and performing Tn5 shift; (3) split alignment of Hi-C reads against a reference genome. In all these three cases, Chromap is 10-20 times faster while being accurate.\n\n### Installation\n\nTo compile from the source, you need to have the GCC compiler with version>=7.3.0, GNU make and zlib development files installed. Then type `make` in the source code directory to compile. \n\nChromap is also available on [bioconda][bioconda]. Thus you can easily install Chromap with Conda\n```sh\nconda install -c conda-forge -c bioconda chromap\n```\n\n### General usage\nBefore mapping, an index of the reference needs to be created and saved on the disk:\n```sh\nchromap -i -r ref.fa -o index\n```\nThe users can input the min fragment length expected in their sequencing experiments, e.g. read length, by **--min-frag-length**. Then Chromap will choose proper k-mer length and window size to build the index. For human genome, it only takes a few minutes to build the index. Without any preset parameters, Chromap takes a reference database and a query sequence file as input and produce approximate mapping, without base-level alignment in the [BED format][bed]:\n\n```sh\nchromap -x index -r ref.fa -1 query.fq -o approx-mapping.bed\n```\nYou may ask Chromap to output alignments in the [SAM format][sam]:\n\n```sh\nchromap -x index -r ref.fa -1 query.fq --SAM -o alignment.sam\n```\nBut note that the the processing of SAM files is not fully optimized and can be slow. Thus generating the output in SAM format is not preferred and should be avoided when possible. Chromap can take multiple input read files:\n\n```sh\nchromap -x index -r ref.fa -1 query1.fq,query2.fq,query3.fq --SAM -o alignment.sam\n```\nChromap also supports wildcards in the read file names and will find all matched read files. To use this function, the read file names ***must*** be put in quotation marks:\n\n```sh\nchromap -x index -r ref.fa -1 \"query*.fq\" --SAM -o alignment.sam\n```\nChromap works with gzip'd FASTA and FASTQ formats as input. You don't need to convert between FASTA and FASTQ or decompress gzip'd files first. \n\n***Importantly***, it should be noted that once you build the index, indexing parameters such as **-k**, **-w** and **--min-frag-length** can't be changed during mapping. If you are running Chromap for different data types, you will probably need to keep multiple indexes generated with different parameters.\nThis makes Chromap different from BWA which always uses the same index regardless of query data types. Chromap can build the human genome index file in a few minutes.\n\nDetailed explanations for the options can be found at the [manpage][manpage].\n\n### Use cases\n\nTo support different data types (e.g. ChIP-seq, Hi-C, ATAC-seq), Chromap needs to be tuned for optimal performance and accuracy. It is usually recommended to choose a preset with option **--preset**, which sets multiple parameters at the same time.\n\n#### Map ChIP-seq short reads\n\n```sh\nchromap --preset chip -x index -r ref.fa -1 read1.fq.gz -2 read2.fq.gz -o aln.bed # ChIP-seq reads\n```\nThis set of parameters is tuned for mapping ChIP-seq reads. Chromap will map the paired-end reads with max insert size up to 2000 (**-l 2000**) and then remove duplicates (**--remove-pcr-duplicates**) using the low memory mode (**--low-mem**). The output is in BED format (**--BED**). In the output BED file, each row is a mapping of a fragment (i.e., a read pair) and the columns are\n\n chrom chrom_start chrom_end N mapq strand\nThe strand here is the strand of the first read in a read pair (specified by **-1**). If the mapping start and end locations of each read in a read pair are desired, **--TagAlign** should be used to overide **--BED** in the preset parameters as following\n```sh\nchromap --preset chip -x index -r ref.fa -1 read1.fq.gz -2 read2.fq.gz --TagAlign -o aln.tagAlign # ChIP-seq reads\n```\nFor each read pair, there will be two rows in the output file, one for each read in the pair respectively. The meaning of the columns remains the same.\n\n#### Map ATAC-seq/scATAC-seq short reads\n\n```sh\nchromap --preset atac -x index -r ref.fa -1 read1.fq.gz -2 read2.fq.gz -o aln.bed # ATAC-seq reads\nchromap --preset atac -x index -r ref.fa -1 read1.fq.gz -2 read2.fq.gz -o aln.bed\\\n -b barcode.fq.gz --barcode-whitelist whitelist.txt # scATAC-seq reads\n```\nThis set of parameters is tuned for mapping ATAC-seq/scATAC-seq reads. Chromap will trim the adapters on 3' end (**--trim-adapters**), map the paired-end reads with max insert size up to 2000 (**-l 2000**) and then remove duplicates at cell level (**--remove-pcr-duplicates-at-cell-level**). Tn5 shift will also be applied to the fragments (**--Tn5-shift**). The forward mapping start positions are increased by 4bp and the reverse mapping end positions are decreased by 5bp. The processing is run in the low memory mode (**--low-mem**).\n\nIf no barcode whitelist file is given, Chromap will skip barcode correction. When barcodes and a whitelist are given as input, by default Chromap will estimate barcode abundance and use this information to perform barcode correction with up to 1 Hamming distance from a whitelist barcode. By setting **--bc-error-threshold** to 2, Chromap is able to correct barcodes with up to 2 Hamming distance from a whitelist barcode. User can also increase the probability threshold to make a correction by setting **--bc-probability-threshold** (set to 0.9 by default) to a large value (e.g., 0.975) to only make reliable corrections. For scATAC-seq data with multiple read and barcode files, you can use \",\" to concatenate multiple input files as the example [above](#general). \n\nChromap also supports user-defined barcode format, including mixed barcode and genomic data case. User can specify the sequence structure through option **--read-format**. The value is a comma-separated string, each field in the string is also a semi-comma-splitted string\n\n [r1|r2|bc]:start:end:strand\nThe start and end are inclusive and -1 means the end of the read. User may use multiple fields to specify non-consecutive segments, e.g. bc:0:15,bc:32:-1. The strand is presented by '+' and '-' symbol, if '-' the barcode will be reverse-complemented after extraction. The strand symbol can be omitted if it is '+' and is ignored on r1 and r2. For example, when the barcode is in the first 16bp of read1, one can use the option `-1 read1.fq.gz -2 read2.fq.gz --barcode read1.fq.gz --read-format bc:0:15,r1:16:-1`.\n\nThe output file formats for bulk and single-cell data are different except for the first three columns. For bulk data, the columns are\n\n chrom chrom_start chrom_end N mapq strand duplicate_count\nFor single-cell data, the columns are \n \n chrom chrom_start chrom_end barcode duplicate_count\nthe same as the definition of the fragment file in [CellRanger][cellranger]. Note that chrom_end is open-end. This output fragment file can be used as input of downstream analysis tools such as [MAESTRO][MAESTRO], [ArchR][ArchR], [signac][signac] and etc.\n\nBesides, Chromap can translate input cell barcodes to another set of barcodes. Users can specify the translation file through the option **--barcode-translate**. The translation file is a two-column tsv/csv file with the translated barcode on the first column and the original barcode on the second column. This is useful for 10x Multiome data, where scATAC-seq and scRNA-seq data use different sets of barcodes. This option also supports combinatorial barcoding, such as SHARE-seq. Chromap can translate each barcode segment provided in the second column to the ID in the first column and add \"-\" to concatenate the IDs in the output.\n\n#### Map Hi-C short reads\n\n```sh\nchromap --preset hic -x index -r ref.fa -1 read1.fa -2 read2.fa -o aln.pairs # Hi-C reads and pairs output\n```\nChromap will perform split alignment (**--split-alignment**) on Hi-C reads and output mappings in [pairs][pairs] format (**--pairs**), which is used in [4DN Hi-C data processing pipeline][4DN]. Some Hi-C data analysis pipelines may require the reads are sorted in specific chromosome order other than the one in the index. Therefore, Chromap provides the option **--chr-order** to specify the alignment order, and **--pairs-natural-chr-order** for flipping the pair in the pairs format. \n\n### Summarizing mapping statistics/quality control\n\nChromap allows you to summarize the dataset's mapping statistics as well as quality metrics at either a *bulk* or *single cell* level. To enable this feature, users can specify a file path using this option, **--summary [FILE]**, where a csv file will be saved.\n\nThis summary file will output a series of metrics for each barcode (or the overall dataset if it is bulk). Here are the different columns contained within the summary file:\n\n```sh\nbarcode,total,duplicate,unmapped,lowmapq,cachehit,fric,estfrip,numcacheslots\n```\n\n- `barcode` - Barcode label for cell\n- `total` - Total number of fragments\n- `duplicate` - Number of duplicate fragments\n- `unmapped` - Number of unmapped fragments \n- `lowmapq` - Number of fragments with a low MAPQ\n- `cachehit` - Number of fragments that were found in the chromap cache during alignment\n- `fric` - Fraction of fragments in the chromap cache\n- `estfrip` - Estimated FRiP value based on a linear model ([See below for more details](#estfrip))\n- `numcacheslots` - Number of unique associated cache slots for this barcode (Relevant feature for doublet detection, [see below for more](#doublet))\n\nThe summary contains metrics relevant to the mappability of fragments from each barcode. \nHowever, it also contains metrics (`estfrip` and `numcacheslots`) relevant to quality control for chromatin profiling assays like scATAC-seq. These cache-related metrics require overall deep sequencing depth, so it is more useful for single-cell data. \nThe next two sections briefly describe these two metrics and how they can be useful for users.\n\n#### Estimating FRiP\n\nThe `estfrip` column in Chromap's summary file represents an estimate of the FRiP score (Fraction of Reads in Peak Regions) computed by Chromap.\nChromap uses a simple multi-variate linear model to estimate the FRiP for each barcode and the features used in this model are `fric`, `duplicate`, `unmapped` and `lowmapq`.\n\nTypically, the FRiP score is used to assess the quality of chromatin profiles, where typically the higher the FRiP score the better. \n\nFor users, this `estfrip` can be used to quickly gauge the quality of the data by plotting all the values in a histogram and looking to see if you a multi-modal distribution.\nIn addition, when combining Chromap with downstream analysis tools such as [SnapATAC2](https://github.com/kaizhang/SnapATAC2) that perform clustering, the `estfrip` can be used to quickly identify any specific clusters that are lower quality than the rest.\n\n**An important note to users**, the `estfrip` values for every barcode should not be taken by themselves and used as the true FRiP score.\nThese estimates are mainly intended to be used for quality control at a dataset level where we compare different `estfrip` values to each other.\n\n#### Features to assist in doublet detection\n\nThe `numcacheslots` column in Chromap's summary file estimates the number of unique cache slots queried for each barcode during the alignment. This feature can be useful in assisting users for doublet detection/filtering.\n\nTypically for doublet detection in single-cell datasets, a simple and naive metric used to identify potential doublets is the number of fragments in cells (i.e. more reads, more likely a doublet). \n\nChromap uses the simple intuition that barcodes with higher number of peaks than usual, could be doublets. The number of unique cache slots that are queried can be seen as a proxy for the number of peaks. In our experiments, using `numcacheslots` yields a larger AUC compared using `total` for binary classification of doublets. Therefore, users can potentially use this metric as an additional check/feature along with other doublet-detection specific methods.\n\n\n### Getting help\n\nDetailed description of Chromap command line options and optional tags can be displayed by running Chromap with **-h** or be found at the [manpage][manpage]. If you encounter bugs or have further questions or requests, you can raise an issue at the [issue page][issue].\n\n### Citing Chromap\n\nIf you use Chromap, please cite:\n\n> Zhang, H., Song, L., Wang, X., Cheng, H., Wang, C., Meyer, C. A., ..., Liu, X. S., Li, H. (2021). Fast alignment and preprocessing of chromatin profiles with Chromap. Nature communications, 12(1), 1-6.\n> https://doi.org/10.1038/s41467-021-26865-w\n\nThe summary file for QC is described in the manuscript:\n> Ahmed, O., Zhang, H., Langmead, B., Song, L. (2025). Quality control of single-cell ATAC-seq data without peak calling using Chromap. Biorxiv.\n> https://doi.org/10.1101/2025.07.15.664951\n\n[bed]: https://genome.ucsc.edu/FAQ/FAQformat.html#format1\n[paf]: https://github.com/lh3/miniasm/blob/master/PAF.md\n[sam]: https://samtools.github.io/hts-specs/SAMv1.pdf\n[pairs]: https://github.com/4dn-dcic/pairix/blob/master/pairs_format_specification.md\n[4DN]: https://data.4dnucleome.org/resources/data-analysis/hi_c-processing-pipeline\n[minimap]: https://github.com/lh3/minimap\n[release]: https://github.com/haowenz/chromap/releases\n[issue]: https://github.com/haowenz/chromap/issues\n[cellranger]: https://support.10xgenomics.com/single-cell-atac/software/pipelines/latest/output/fragments\n[manpage]: https://haowenz.github.io/chromap/chromap.html\n[bioconda]: https://anaconda.org/bioconda/chromap\n[ArchR]: https://www.archrproject.com/index.html\n[MAESTRO]: https://github.com/liulab-dfci/MAESTRO\n[signac]: https://satijalab.org/signac/articles/pbmc_vignette.html\n"
},
{
"path": "chromap.1",
"content": ".TH chromap 1 \"25 Jan 2024\" \"chromap-0.2.6 (r490)\" \"Bioinformatics tools\"\n.SH NAME\n.PP\nchromap - fast alignment and preprocessing of chromatin profiles\n.SH SYNOPSIS\n* Indexing the reference genome:\n.RS 4\nchromap\n.B -i\n.RB [ -k\n.IR kmer ]\n.RB [ -w\n.IR miniWinSize ]\n.B -r\n.I ref.fa\n.B -o\n.I ref.index\n.RE\n\n* Mapping (sc)ATAC-seq reads:\n.RS 4\nchromap\n.B --preset\n.I atac\n.B -r\n.I ref.fa\n.B -x\n.I ref.index\n.B -1 \n.I read1.fq\n.B -2\n.I read2.fq\n.B -o \n.I aln.bed\n.RB [ -b \n.IR barcode.fq.gz ] \n.RB [ --barcode-whitelist \n.IR whitelist.txt ]\n.RE\n\n* Mapping ChIP-seq reads:\n.RS 4\nchromap\n.B --preset\n.I chip\n.B -r\n.I ref.fa\n.B -x\n.I ref.index\n.B -1\n.I read1.fq\n.B -2\n.I read2.fq\n.B -o\n.I aln.bed\n.RE\n\n* Mapping Hi-C reads:\n.RS 4\nchromap \n.B --preset\n.I hic\n.B -r\n.I ref.fa\n.B -x\n.I ref.index\n.B -1\n.I read1.fq\n.B -2\n.I read2.fq\n.B -o\n.I aln.pairs\n.br\nchromap \n.B --preset\n.I hic\n.B -r\n.I ref.fa\n.B -x\n.I ref.index\n.B -1\n.I read1.fq\n.B -2\n.I read2.fq\n.B --SAM\n.B -o\n.I aln.sam\n.RE\n\n.SH DESCRIPTION\n.PP\nChromap is an ultrafast method for aligning and preprocessing high throughput\nchromatin profiles. Typical use cases include: (1) trimming sequencing adapters,\nmapping bulk ATAC-seq or ChIP-seq genomic reads to the human genome and removing\nduplicates; (2) trimming sequencing adapters, mapping single cell ATAC-seq\ngenomic reads to the human genome, correcting barcodes, removing duplicates and\nperforming Tn5 shift; (3) split alignment of Hi-C reads against a reference\ngenome. In all these three cases, Chromap is 10-20 times faster while being\naccurate.\n.SH OPTIONS\n.SS Indexing options\n.TP 10\n.BI -k \\ INT\nMinimizer k-mer length [17].\n.TP\n.BI -w \\ INT\nMinimizer window size [7]. A minimizer is the smallest k-mer\nin a window of w consecutive k-mers.\n.TP\n.B --min-frag-length\nMin fragment length for choosing k and w automatically [30]. Users can increase\nthis value when the min length of the fragments of interest is long, which can\nincrease the mapping speed. Note that the default value 30 is the min fragment\nlength that chromap can map. \n\n.SS Mapping options\n.TP 10\n.BI --split-alignment\nAllow split alignments. This option should be set only when mapping Hi-C reads.\n.TP\n.BI -e \\ INT\nMax edit distance allowed to map a read [8].\n.TP\n.BI -s \\ INT\nMin number of minimizers required to map a read [2].\n.TP\n.BI -f \\ INT1 [, INT2 ]\nIgnore minimizers occuring more than\n.I INT1\n[500] times.\n.I INT2\n[1000] is the threshold for a second round of seeding.\n.TP\n.BI -l \\ INT\nMax insert size, only for paired-end read mapping [1000].\n.TP\n.BI -q \\ INT\nMin MAPQ in range [0, 60] for mappings to be output [30].\n.TP\n.BI --min-read-length \\ INT\nSkip mapping the reads of length less than \n.I INT \n[30]. Note that this is different from the index option\n.BR --min-frag-length\n, which set\n.BR -k\nand\n.BR -w\nfor indexing the genome.\n.TP\n.BI --trim-adapters\nTry to trim adapters on 3'. This only works for paired-end reads. When the\nfragment length indicated by the read pair is less than the length of the reads,\nthe two mates are overlapped with each other. Then the regions outside the\noverlap are regarded as adapters and trimmed.\n.TP\n.BI --remove-pcr-duplicates\nRemove PCR duplicates.\n.TP\n.BI --remove-pcr-duplicates-at-bulk-level\nRemove PCR duplicates at bulk level for single cell data.\n.TP\n.BI --remove-pcr-duplicates-at-cell-level\nRemove PCR duplicates at cell level for single cell data.\n.TP\n.BI --Tn5-shift\nPerform Tn5 shift. When this option is turned on, the forward mapping start\npositions are increased by 4bp and the reverse mapping end positions are\ndecreased by 5bp. Note that this works only when\n.BR --SAM\nis NOT set.\n.TP\n.BI --low-mem\nUse low memory mode. When this option is set, multiple temporary intermediate\nmapping files might be generated on disk and they are merged at the end of\nprocessing to reduce memory usage. When this is NOT set, all the mapping results\nare kept in the memory before they are saved on disk, which works more\nefficiently for datasets that are not too large.\n.TP\n.BI --bc-error-threshold \\ INT\nMax Hamming distance allowed to correct a barcode [1]. Note that the max \nsupported threshold is 2.\n.TP\n.BI --bc-probability-threshold \\ FLT\nMin probability to correct a barcode [0.9]. When there are multiple whitelisted\nbarcodes with the same Hamming distance to the barcode to correct, chromap will\nprocess the base quality of the mismatched bases, and compute a probability that\nthe correction is right.\n.TP\n.BI -t \\ INT\nThe number of threads for mapping [1].\n\n.SS Input options\n.TP 10\n.BI -r \\ FILE\nReference file.\n.TP\n.BI -x \\ FILE\nIndex file.\n.TP\n.BI -1 \\ FILE\nSingle-end read files or paired-end read files 1. Chromap supports mulitple\ninput files concatenate by \",\". For example, setting this option to \n\"Library1_R1.fastq.gz,Library2_R1.fastq.gz,Library3_R1.fastq.gz\" will make \nall three files as input and map them in this order. Similarly,\n.BR -2\nand\n.BR -b\nalso support multiple input files. And the ordering of the input files for all\nthe three options should match.\n.TP\n.BI -2 \\ FILE\nPaired-end read files 2.\n.TP\n.BI -b \\ FILE\nCell barcode files.\n.TP\n.BI --barcode-whitelist \\ FILE\nCell barcode whitelist file. This is supposed to be a txt file where each line\nis a whitelisted barcode.\n.TP\n.BI --read-format \\ STR\nFormat for read files and barcode files [\"r1:0:-1,bc:0:-1\"] as 10x Genomics \nsingle-end format.\n\n.SS Output options\n.TP 10\n.BR -o \\ FILE\nOutput file.\n.TP\n.BR --output-mappings-not-in-whitelist\nOutput mappings with barcode not in the whitelist.\n.TP\n.BR --chr-order \\ FILE \nCustom chromosome order file. If not specified, the order of reference sequences will be used.\n.TP\n.BR --BED\nOutput mappings in BED/BEDPE format. Note that only one of the formats should be\nset.\n.TP\n.BR --TagAlign\nOutput mappings in TagAlign/PairedTagAlign format.\n.TP\n.BR --SAM\nOutput mappings in SAM format.\n.TP\n.BR --pairs\nOutput mappings in pairs format (defined by 4DN for HiC data).\n.TP\n.BR --pairs-natural-chr-order \\ FILE\nCustom chromosome order file for pairs flipping. If not specified, the custom chromosome order will be used.\n.TP\n.BR --barcode-translate \\ FILE\nConvert input barcodes to another set of barcodes in the output.\n.TP\n.BR --summary \\ FILE\nSummarize the mapping statistics at bulk or barcode level.\n.TP\n.B -v\nPrint version number to stdout.\n\n.SS Preset options\n.TP 10\n.BI --preset \\ STR\nPreset []. This option applies multiple options at the same time. It should be\napplied before other options because options applied later will overwrite the\nvalues set by\n.BR --preset .\nAvailable\n.I STR\nare:\n.RS\n.TP 10\n.B chip \nMapping ChIP-seq reads\n.RB ( -l\n.I 2000\n.B --remove-pcr-duplicates --low-mem\n.BR --BED ).\n.TP\n.B atac\nMapping ATAC-seq/scATAC-seq reads\n.RB ( -l \n.I 2000\n.B --remove-pcr-duplicates --low-mem --trim-adapters --Tn5-shift\n.B --remove-pcr-duplicates-at-cell-level\n.BR --BED ).\n.TP\n.B hic\nMapping Hi-C reads\n.RB ( -e \n.I 4\n.B -q\n.I 1 \n.B --low-mem --split-alignment\n.BR --pairs ).\n"
},
{
"path": "docs/_config.yml",
"content": "theme: jekyll-theme-modernist"
},
{
"path": "docs/chromap.html",
"content": "\n\n\n\n\n\n\n\n\nchromap\n\n\n\n\nchromap
\n\nNAME
\nSYNOPSIS
\nDESCRIPTION
\nOPTIONS
\n\n
\n\n\nNAME\n\n
\n\n\nchromap - fast\nalignment and preprocessing of chromatin profiles
\n\nSYNOPSIS\n\n
\n\n\n* Indexing the\nreference genome:
\n\nchromap -i [-k\nkmer] [-w miniWinSize] -r\nref.fa -o ref.index
\n\n* Mapping\n(sc)ATAC-seq reads:
\n\nchromap --preset\natac -r ref.fa -x\nref.index -1 read1.fq -2\nread2.fq -o aln.bed [-b\nbarcode.fq.gz] [--barcode-whitelist\nwhitelist.txt]
\n\n* Mapping\nChIP-seq reads:
\n\nchromap --preset\nchip -r ref.fa -x\nref.index -1 read1.fq -2\nread2.fq -o aln.bed
\n\n* Mapping Hi-C\nreads:
\n\nchromap --preset\nhic -r ref.fa -x\nref.index -1 read1.fq -2\nread2.fq -o aln.pairs
\nchromap --preset hic -r ref.fa\n-x ref.index -1 read1.fq\n-2 read2.fq --SAM -o aln.sam
\n\nDESCRIPTION\n\n
\n\n\nChromap is an\nultrafast method for aligning and preprocessing high\nthroughput chromatin profiles. Typical use cases include:\n(1) trimming sequencing adapters, mapping bulk ATAC-seq or\nChIP-seq genomic reads to the human genome and removing\nduplicates; (2) trimming sequencing adapters, mapping single\ncell ATAC-seq genomic reads to the human genome, correcting\nbarcodes, removing duplicates and performing Tn5 shift; (3)\nsplit alignment of Hi-C reads against a reference genome. In\nall these three cases, Chromap is 10-20 times faster while\nbeing accurate.
\n\nOPTIONS\n\n
\n\n\nIndexing\noptions
\n\n\n\n | \n\n\n\n -k INT | \n | \n\n\n\n Minimizer k-mer length [17]. |
\n\n | \n\n\n\n -w INT | \n | \n\n\n\n Minimizer window size [7]. A minimizer is the smallest\nk-mer in a window of w consecutive k-mers. |
\n
\n\n--min-frag-length
\n\nMin fragment length for\nchoosing k and w automatically [30]. Users can increase this\nvalue when the min length of the fragments of interest is\nlong, which can increase the mapping speed. Note that the\ndefault value 30 is the min fragment length that chromap can\nmap.
\n\nMapping\noptions
\n--split-alignment
\n\nAllow split alignments. This\noption should be set only when mapping Hi-C reads.
\n\n\n\n | \n\n\n\n -e INT | \n | \n\n\n\n Max edit distance allowed to map a read [8]. |
\n\n | \n\n\n\n -s INT | \n | \n\n\n\n Min number of minimizers required to map a read [2]. |
\n
\n\n\n-f INT1[,INT2]
\n\nIgnore minimizers occuring more\nthan INT1 [500] times. INT2 [1000] is the\nthreshold for a second round of seeding.
\n\n\n\n | \n\n\n\n -l INT | \n | \n\n\n\n Max insert size, only for paired-end read mapping\n[1000]. |
\n\n | \n\n\n\n -q INT | \n | \n\n\n\n Min MAPQ in range [0, 60] for mappings to be output\n[30]. |
\n
\n\n\n--min-read-length INT
\n\nSkip mapping the reads of\nlength less than INT [30]. Note that this is\ndifferent from the index option --min-frag-length ,\nwhich set -k and -w for indexing the\ngenome.
\n\n--trim-adapters
\n\nTry to trim adapters on\n3’. This only works for paired-end reads. When the\nfragment length indicated by the read pair is less than the\nlength of the reads, the two mates are overlapped with each\nother. Then the regions outside the overlap are regarded as\nadapters and trimmed.
\n\n\n--remove-pcr-duplicates
\n\nRemove PCR duplicates.
\n\n\n--remove-pcr-duplicates-at-bulk-level
\n\nRemove PCR duplicates at bulk\nlevel for single cell data.
\n\n\n--remove-pcr-duplicates-at-cell-level
\n\nRemove PCR duplicates at cell\nlevel for single cell data.
\n\n--Tn5-shift
\n\nPerform Tn5 shift. When this\noption is turned on, the forward mapping start positions are\nincreased by 4bp and the reverse mapping end positions are\ndecreased by 5bp. Note that this works only when\n--SAM is NOT set.
\n\n\n\n | \n\n\n\n --low-mem | \n | \n\n\n\n Use low memory mode. When this option is set, multiple\ntemporary intermediate mapping files might be generated on\ndisk and they are merged at the end of processing to reduce\nmemory usage. When this is NOT set, all the mapping results\nare kept in the memory before they are saved on disk, which\nworks more efficiently for datasets that are not too\nlarge. |
\n
\n\n\n--bc-error-threshold INT
\n\nMax Hamming distance allowed to\ncorrect a barcode [1]. Note that the max supported threshold\nis 2.
\n\n\n--bc-probability-threshold FLT
\n\nMin probability to correct a\nbarcode [0.9]. When there are multiple whitelisted barcodes\nwith the same Hamming distance to the barcode to correct,\nchromap will process the base quality of the mismatched\nbases, and compute a probability that the correction is\nright.
\n\n\n\n | \n\n\n\n -t INT | \n | \n\n\n\n The number of threads for mapping [1]. | \n\n |
\n
\n\nInput\noptions
\n\n\n\n | \n\n\n\n -r FILE | \n | \n\n\n\n Reference file. |
\n\n | \n\n\n\n -x FILE | \n | \n\n\n\n Index file. |
\n\n | \n\n\n\n -1 FILE | \n | \n\n\n\n Single-end read files or paired-end read files 1.\nChromap supports mulitple input files concatenate by\n",". For example, setting this option to\n"read11.fq,read12.fq,read13.fq" will make all\nthree files as input and map them in this order. Similarly,\n-2 and -b also support multiple input files.\nAnd the ordering of the input files for all the three\noptions should match. |
\n\n | \n\n\n\n -2 FILE | \n | \n\n\n\n Paired-end read files 2. |
\n\n | \n\n\n\n -b FILE | \n | \n\n\n\n Cell barcode files. |
\n
\n\n\n--barcode-whitelist FILE
\n\nCell barcode whitelist file.\nThis is supposed to be a txt file where each line is a\nwhitelisted barcode.
\n\n\n--read-format STR
\n\nFormat for read files and\nbarcode files ["r1:0:-1,bc:0:-1"] as 10x Genomics\nsingle-end format.
\n\nOutput\noptions
\n\n\n\n | \n\n\n\n -o FILE | \n | \n\n\n\n Output file. | \n\n |
\n
\n\n\n--output-mappings-not-in-whitelist
\n\nOutput mappings with barcode\nnot in the whitelist.
\n\n\n--chr-order FILE
\n\nCustomized chromsome order.
\n\n\n\n | \n\n\n\n --BED | \n | \n\n\n\n Output mappings in BED/BEDPE format. Note that only one\nof the formats should be set. |
\n
\n\n--TagAlign
\n\nOutput mappings in\nTagAlign/PairedTagAlign format.
\n\n\n\n | \n\n\n\n --SAM | \n | \n\n\n\n Output mappings in SAM format. |
\n\n | \n\n\n\n --pairs | \n | \n\n\n\n Output mappings in pairs format (defined by 4DN for HiC\ndata). |
\n
\n\n\n--pairs-natural-chr-order FILE
\n\nNatural chromosome order for\npairs flipping.
\n\n\n\n | \n\n\n\n -v | \n | \n\n\n\n Print version number to stdout. | \n\n |
\n
\n\nPreset\noptions
\n--preset STR
\n\nPreset []. This option applies\nmultiple options at the same time. It should be applied\nbefore other options because options applied later will\noverwrite the values set by --preset. Available\nSTR are:
\n\n\n\n | \n\n\n\n chip | \n | \n\n\n\n Mapping ChIP-seq reads (-l 2000\n--remove-pcr-duplicates --low-mem --BED). |
\n\n | \n\n\n\n atac | \n | \n\n\n\n Mapping ATAC-seq/scATAC-seq reads (-l 2000\n--remove-pcr-duplicates --low-mem --trim-adapters\n--Tn5-shift --remove-pcr-duplicates-at-cell-level\n--BED). |
\n\n | \n\n\n\n hic | \n | \n\n\n\n Mapping Hi-C reads (-e 4 -q\n1 --low-mem --split-alignment --pairs). |
\n
\n
\n\n\n"
},
{
"path": "docs/index.md",
"content": "## Getting help\n\n* [README][doc]: general documentation\n* [Manpage](chromap.html): explanation of command-line options\n* [Preprint][biorxiv]: free of charge preprint that describes the method\n* [GitHub Issues page][issue]: report bugs, request features and ask questions\n\n## Acquiring Chromap\n\n* `git clone https://github.com/haowenz/chromap.git`\n* [GitHub Release page][release]: versioned packages\n* Also [available from BioConda][bioconda]\n\n[doc]: https://github.com/haowenz/chromap/blob/master/README.md\n[biorxiv]: https://www.biorxiv.org/content/10.1101/2021.06.18.448995v1\n[bioconda]: https://anaconda.org/bioconda/chromap\n[release]: https://github.com/haowenz/chromap/releases\n[issue]: https://github.com/haowenz/chromap/issues\n"
},
{
"path": "src/alignment.cc",
"content": "#include \"alignment.h\"\n\n#include \n\nnamespace chromap {\n\nint GetLongestMatchLength(const char *pattern, const char *text,\n const int read_length) {\n int max_match = 0;\n int tmp = 0;\n for (int i = 0; i < read_length; ++i) {\n if (CharToUint8(pattern[i]) == CharToUint8(text[i])) {\n ++tmp;\n } else if (tmp > max_match) {\n max_match = tmp;\n }\n }\n if (tmp > max_match) {\n max_match = tmp;\n }\n return max_match;\n}\n\nint AdjustGapBeginning(const Strand mapping_strand, const char *ref,\n const char *read, int *gap_beginning, int read_end,\n int ref_start_position, int ref_end_position,\n int *n_cigar, uint32_t **cigar) {\n int i, j;\n if (mapping_strand == kPositive) {\n if (*gap_beginning <= 0) {\n return ref_start_position;\n }\n\n // printf(\"%d\\n\", *gap_beginning);\n\n for (i = *gap_beginning - 1, j = ref_start_position - 1; i >= 0 && j >= 0;\n --i, --j) {\n // printf(\"%c %c\\n\", read[i], ref[j]);\n if (read[i] != ref[j] && read[i] != ref[j] - 'a' + 'A') {\n break;\n }\n }\n\n *gap_beginning = i + 1;\n // TODO: add soft clip in cigar\n if (n_cigar && *n_cigar > 0) {\n if (((*cigar)[0] & 0xf) == BAM_CMATCH) {\n (*cigar)[0] += (ref_start_position - 1 - j) << 4;\n }\n }\n\n return j + 1;\n }\n\n if (*gap_beginning <= 0) {\n return ref_end_position;\n }\n\n // printf(\"%d\\n\", *gap_beginning);\n /*char *tmp = new char[255] ;\n strncpy(tmp, ref + ref_start_position, ref_end_position - ref_start_position\n + 1 + 10) ; printf(\"%s %d. %d %d\\n\", tmp, strlen(tmp), ref_end_position -\n ref_start_position + 1 + 10, strlen(ref)) ; delete[] tmp;*/\n\n for (i = read_end + 1, j = ref_end_position + 1; read[i] && ref[j];\n ++i, ++j) {\n // printf(\"%c %c %c %c %c %c\\n\", read[i], ref[j - 1], ref[j], ref[j + 1],\n // ref[j + 2], ref[j + 3]);\n if (read[i] != ref[j] && read[i] != ref[j] - 'a' + 'A') {\n break;\n }\n }\n\n *gap_beginning = *gap_beginning + i - (read_end + 1);\n\n if (n_cigar && *n_cigar > 0) {\n if (((*cigar)[*n_cigar - 1] & 0xf) == BAM_CMATCH) {\n (*cigar)[*n_cigar - 1] += (j - (ref_end_position + 1)) << 4;\n }\n }\n\n return j - 1;\n}\n\nvoid GenerateNMAndMDTag(const char *pattern, const char *text,\n int mapping_start_position,\n MappingInMemory &mapping_in_memory) {\n const char *read = text;\n const char *reference = pattern + mapping_start_position;\n\n const uint32_t *cigar = mapping_in_memory.cigar;\n const int n_cigar = mapping_in_memory.n_cigar;\n mapping_in_memory.NM = 0;\n mapping_in_memory.MD_tag.clear();\n\n int num_matches = 0;\n int read_position = 0;\n int reference_position = 0;\n\n for (int ci = 0; ci < n_cigar; ++ci) {\n uint32_t current_cigar_uint = cigar[ci];\n uint8_t cigar_operation = bam_cigar_op(current_cigar_uint);\n int num_cigar_operations = bam_cigar_oplen(current_cigar_uint);\n if (cigar_operation == BAM_CMATCH) {\n for (int opi = 0; opi < num_cigar_operations; ++opi) {\n if (reference[reference_position] == read[read_position] ||\n reference[reference_position] - 'a' + 'A' == read[read_position]) {\n // a match\n ++num_matches;\n } else {\n // a mismatch\n ++mapping_in_memory.NM;\n \n mapping_in_memory.MD_tag.append(std::to_string(num_matches));\n num_matches = 0;\n mapping_in_memory.MD_tag.push_back(reference[reference_position]);\n }\n ++reference_position;\n ++read_position;\n }\n } else if (cigar_operation == BAM_CINS) {\n mapping_in_memory.NM += num_cigar_operations;\n read_position += num_cigar_operations;\n } else if (cigar_operation == BAM_CDEL) {\n mapping_in_memory.NM += num_cigar_operations;\n \n mapping_in_memory.MD_tag.append(std::to_string(num_matches));\n num_matches = 0;\n mapping_in_memory.MD_tag.push_back('^');\n for (int opi = 0; opi < num_cigar_operations; ++opi) {\n mapping_in_memory.MD_tag.push_back(reference[reference_position]);\n ++reference_position;\n }\n } else {\n std::cerr << \"Unexpected cigar op: \" << (int)cigar_operation << \"\\n\";\n }\n }\n mapping_in_memory.MD_tag.append(std::to_string(num_matches));\n}\n\nint BandedAlignPatternToText(int error_threshold, const char *pattern,\n const char *text, const int read_length,\n int *mapping_end_position) {\n uint32_t Peq[5] = {0, 0, 0, 0, 0};\n for (int i = 0; i < 2 * error_threshold; i++) {\n uint8_t base = CharToUint8(pattern[i]);\n Peq[base] = Peq[base] | (1 << i);\n }\n uint32_t highest_bit_in_band_mask = 1 << (2 * error_threshold);\n uint32_t lowest_bit_in_band_mask = 1;\n uint32_t VP = 0;\n uint32_t VN = 0;\n uint32_t X = 0;\n uint32_t D0 = 0;\n uint32_t HN = 0;\n uint32_t HP = 0;\n int num_errors_at_band_start_position = 0;\n for (int i = 0; i < read_length; i++) {\n uint8_t pattern_base = CharToUint8(pattern[i + 2 * error_threshold]);\n Peq[pattern_base] = Peq[pattern_base] | highest_bit_in_band_mask;\n X = Peq[CharToUint8(text[i])] | VN;\n D0 = ((VP + (X & VP)) ^ VP) | X;\n HN = VP & D0;\n HP = VN | ~(VP | D0);\n X = D0 >> 1;\n VN = X & HP;\n VP = HN | ~(X | HP);\n num_errors_at_band_start_position += 1 - (D0 & lowest_bit_in_band_mask);\n if (num_errors_at_band_start_position > 3 * error_threshold) {\n return error_threshold + 1;\n }\n for (int ai = 0; ai < 5; ai++) {\n Peq[ai] >>= 1;\n }\n }\n int band_start_position = read_length - 1;\n int min_num_errors = num_errors_at_band_start_position;\n *mapping_end_position = band_start_position;\n for (int i = 0; i < 2 * error_threshold; i++) {\n num_errors_at_band_start_position =\n num_errors_at_band_start_position + ((VP >> i) & (uint32_t)1);\n num_errors_at_band_start_position =\n num_errors_at_band_start_position - ((VN >> i) & (uint32_t)1);\n if (num_errors_at_band_start_position < min_num_errors ||\n (num_errors_at_band_start_position == min_num_errors &&\n i + 1 == error_threshold)) {\n min_num_errors = num_errors_at_band_start_position;\n *mapping_end_position = band_start_position + 1 + i;\n }\n }\n return min_num_errors;\n}\n\n// Return negative number if the termination are deemed at the beginning of the\n// read mappping_end_position is relative to pattern (reference)\n// read_mapping_length is for text (read)\nint BandedAlignPatternToTextWithDropOff(int error_threshold,\n const char *pattern, const char *text,\n const int read_length,\n int *mapping_end_position,\n int *read_mapping_length) {\n uint32_t Peq[5] = {0, 0, 0, 0, 0};\n for (int i = 0; i < 2 * error_threshold; i++) {\n uint8_t base = CharToUint8(pattern[i]);\n Peq[base] = Peq[base] | (1 << i);\n }\n uint32_t highest_bit_in_band_mask = 1 << (2 * error_threshold);\n uint32_t lowest_bit_in_band_mask = 1;\n uint32_t VP = 0;\n uint32_t VN = 0;\n uint32_t X = 0;\n uint32_t D0 = 0;\n uint32_t HN = 0;\n uint32_t HP = 0;\n uint32_t prev_VP = 0;\n uint32_t prev_VN = 0;\n int num_errors_at_band_start_position = 0;\n int i = 0;\n int fail_beginning = 0; // the alignment failed at the beginning part\n int prev_num_errors_at_band_start_position = 0;\n for (; i < read_length; i++) {\n uint8_t pattern_base = CharToUint8(pattern[i + 2 * error_threshold]);\n Peq[pattern_base] = Peq[pattern_base] | highest_bit_in_band_mask;\n X = Peq[CharToUint8(text[i])] | VN;\n D0 = ((VP + (X & VP)) ^ VP) | X;\n HN = VP & D0;\n HP = VN | ~(VP | D0);\n X = D0 >> 1;\n prev_VN = VN;\n prev_VP = VP;\n VN = X & HP;\n VP = HN | ~(X | HP);\n prev_num_errors_at_band_start_position = num_errors_at_band_start_position;\n num_errors_at_band_start_position += 1 - (D0 & lowest_bit_in_band_mask);\n if (num_errors_at_band_start_position > 2 * error_threshold) {\n // return error_threshold + 1;\n // the min error in this band could be still less than the\n // error_threshold, and could but this should be fine since it does not\n // affect the 5' end of the read.\n if (i < 4 * error_threshold && i < read_length / 2) {\n fail_beginning = 1;\n }\n break;\n }\n for (int ai = 0; ai < 5; ai++) {\n Peq[ai] >>= 1;\n }\n }\n\n /*char tmp[255] ;\n strncpy(tmp, pattern, read_length + 2 * error_threshold);\n printf(\"%s\\n%s\\n\", tmp, text);\n printf(\"%d\\n\", i) ;\n fflush(stdout);*/\n if (i < read_length) {\n num_errors_at_band_start_position = prev_num_errors_at_band_start_position;\n VN = prev_VN;\n VP = prev_VP;\n }\n int band_start_position = i - 1;\n int min_num_errors = num_errors_at_band_start_position;\n *read_mapping_length = i;\n *mapping_end_position = band_start_position;\n\n for (i = 0; i < 2 * error_threshold; i++) {\n num_errors_at_band_start_position =\n num_errors_at_band_start_position + ((VP >> i) & (uint32_t)1);\n num_errors_at_band_start_position =\n num_errors_at_band_start_position - ((VN >> i) & (uint32_t)1);\n if (num_errors_at_band_start_position < min_num_errors ||\n (num_errors_at_band_start_position == min_num_errors &&\n i + 1 == error_threshold)) {\n min_num_errors = num_errors_at_band_start_position;\n *mapping_end_position = band_start_position + 1 + i;\n }\n }\n if (fail_beginning ||\n (read_length > 60 &&\n *mapping_end_position + 1 - error_threshold - min_num_errors < 30)) {\n *mapping_end_position = -*mapping_end_position;\n }\n return min_num_errors;\n}\n\nint BandedAlignPatternToTextWithDropOffFrom3End(int error_threshold,\n const char *pattern,\n const char *text,\n const int read_length,\n int *mapping_end_position,\n int *read_mapping_length) {\n uint32_t Peq[5] = {0, 0, 0, 0, 0};\n for (int i = 0; i < 2 * error_threshold; i++) {\n uint8_t base =\n CharToUint8(pattern[read_length + 2 * error_threshold - 1 - i]);\n Peq[base] = Peq[base] | (1 << i);\n }\n uint32_t highest_bit_in_band_mask = 1 << (2 * error_threshold);\n uint32_t lowest_bit_in_band_mask = 1;\n uint32_t VP = 0;\n uint32_t VN = 0;\n uint32_t X = 0;\n uint32_t D0 = 0;\n uint32_t HN = 0;\n uint32_t HP = 0;\n uint32_t prev_VP = 0;\n uint32_t prev_VN = 0;\n int num_errors_at_band_start_position = 0;\n int i = 0;\n int fail_beginning = 0; // the alignment failed at the beginning part\n int prev_num_errors_at_band_start_position = 0;\n for (; i < read_length; i++) {\n // printf(\"%c %c %d\\n\", pattern[read_length - 1 - i], pattern[read_length -\n // 1 - i + error_threshold], text[read_length - 1 - i]);\n uint8_t pattern_base = CharToUint8(pattern[read_length - 1 - i]);\n Peq[pattern_base] = Peq[pattern_base] | highest_bit_in_band_mask;\n X = Peq[CharToUint8(text[read_length - 1 - i])] | VN;\n D0 = ((VP + (X & VP)) ^ VP) | X;\n HN = VP & D0;\n HP = VN | ~(VP | D0);\n X = D0 >> 1;\n prev_VN = VN;\n prev_VP = VP;\n VN = X & HP;\n VP = HN | ~(X | HP);\n prev_num_errors_at_band_start_position = num_errors_at_band_start_position;\n num_errors_at_band_start_position += 1 - (D0 & lowest_bit_in_band_mask);\n /*printf(\"->%d %d %c %c\", i, num_errors_at_band_start_position,\n pattern[read_length - 1 - i], text[read_length - 1 - i]) ; int tmp =\n num_errors_at_band_start_position; for (int j = 0; j < 2 * error_threshold;\n j++) { tmp = tmp + ((VP >> j) & (uint32_t) 1); tmp = tmp - ((VN >> j) &\n (uint32_t) 1); printf(\" %d\", tmp);\n }\n printf(\"\\n\");*/\n if (num_errors_at_band_start_position > 2 * error_threshold) {\n // return error_threshold + 1;\n if (i < 4 * error_threshold && i < read_length / 2) {\n fail_beginning = 1;\n }\n break;\n }\n for (int ai = 0; ai < 5; ai++) {\n Peq[ai] >>= 1;\n }\n }\n // printf(\"li %d: %d %d %d\\n\", fail_beginning, i, error_threshold,\n // read_length);\n if (i < read_length) {\n num_errors_at_band_start_position = prev_num_errors_at_band_start_position;\n VN = prev_VN;\n VP = prev_VP;\n }\n int band_start_position = i - 1;\n int min_num_errors = num_errors_at_band_start_position;\n *read_mapping_length = i;\n *mapping_end_position = band_start_position;\n // printf(\"-1: %d\\n\", num_errors_at_band_start_position);\n for (i = 0; i < 2 * error_threshold; i++) {\n num_errors_at_band_start_position =\n num_errors_at_band_start_position + ((VP >> i) & (uint32_t)1);\n num_errors_at_band_start_position =\n num_errors_at_band_start_position - ((VN >> i) & (uint32_t)1);\n // printf(\"%d: %d\\n\", i, num_errors_at_band_start_position);\n if (num_errors_at_band_start_position < min_num_errors ||\n (num_errors_at_band_start_position == min_num_errors &&\n i + 1 == error_threshold)) {\n min_num_errors = num_errors_at_band_start_position;\n *mapping_end_position = band_start_position + (1 + i);\n }\n }\n if (fail_beginning ||\n (read_length > 60 &&\n *mapping_end_position + 1 - error_threshold - min_num_errors < 30)) {\n *mapping_end_position = -*mapping_end_position;\n }\n return min_num_errors;\n}\n\nvoid BandedAlign4PatternsToText(int error_threshold, const char **patterns,\n const char *text, int read_length,\n int32_t *mapping_edit_distances,\n int32_t *mapping_end_positions) {\n int ALPHABET_SIZE = 5;\n const char *reference_sequence0 = patterns[0];\n const char *reference_sequence1 = patterns[1];\n const char *reference_sequence2 = patterns[2];\n const char *reference_sequence3 = patterns[3];\n uint32_t highest_bit_in_band_mask = 1 << (2 * error_threshold);\n __m128i highest_bit_in_band_mask_vpu0 =\n _mm_set_epi32(0, 0, 0, highest_bit_in_band_mask);\n __m128i highest_bit_in_band_mask_vpu1 =\n _mm_set_epi32(0, 0, highest_bit_in_band_mask, 0);\n __m128i highest_bit_in_band_mask_vpu2 =\n _mm_set_epi32(0, highest_bit_in_band_mask, 0, 0);\n __m128i highest_bit_in_band_mask_vpu3 =\n _mm_set_epi32(highest_bit_in_band_mask, 0, 0, 0);\n // Init Peq\n __m128i Peq[ALPHABET_SIZE];\n for (int ai = 0; ai < ALPHABET_SIZE; ai++) {\n Peq[ai] = _mm_setzero_si128();\n }\n for (int i = 0; i < 2 * error_threshold; i++) {\n uint8_t base0 = CharToUint8(reference_sequence0[i]);\n uint8_t base1 = CharToUint8(reference_sequence1[i]);\n uint8_t base2 = CharToUint8(reference_sequence2[i]);\n uint8_t base3 = CharToUint8(reference_sequence3[i]);\n Peq[base0] = _mm_or_si128(highest_bit_in_band_mask_vpu0, Peq[base0]);\n Peq[base1] = _mm_or_si128(highest_bit_in_band_mask_vpu1, Peq[base1]);\n Peq[base2] = _mm_or_si128(highest_bit_in_band_mask_vpu2, Peq[base2]);\n Peq[base3] = _mm_or_si128(highest_bit_in_band_mask_vpu3, Peq[base3]);\n for (int ai = 0; ai < ALPHABET_SIZE; ai++) {\n Peq[ai] = _mm_srli_epi32(Peq[ai], 1);\n }\n }\n\n uint32_t lowest_bit_in_band_mask = 1;\n __m128i lowest_bit_in_band_mask_vpu = _mm_set1_epi32(lowest_bit_in_band_mask);\n __m128i VP = _mm_setzero_si128();\n __m128i VN = _mm_setzero_si128();\n __m128i X = _mm_setzero_si128();\n __m128i D0 = _mm_setzero_si128();\n __m128i HN = _mm_setzero_si128();\n __m128i HP = _mm_setzero_si128();\n __m128i max_mask_vpu = _mm_set1_epi32(0xffffffff);\n __m128i num_errors_at_band_start_position_vpu = _mm_setzero_si128();\n __m128i early_stop_threshold_vpu = _mm_set1_epi32(error_threshold * 3);\n for (int i = 0; i < read_length; i++) {\n uint8_t base0 = CharToUint8(reference_sequence0[i + 2 * error_threshold]);\n uint8_t base1 = CharToUint8(reference_sequence1[i + 2 * error_threshold]);\n uint8_t base2 = CharToUint8(reference_sequence2[i + 2 * error_threshold]);\n uint8_t base3 = CharToUint8(reference_sequence3[i + 2 * error_threshold]);\n Peq[base0] = _mm_or_si128(highest_bit_in_band_mask_vpu0, Peq[base0]);\n Peq[base1] = _mm_or_si128(highest_bit_in_band_mask_vpu1, Peq[base1]);\n Peq[base2] = _mm_or_si128(highest_bit_in_band_mask_vpu2, Peq[base2]);\n Peq[base3] = _mm_or_si128(highest_bit_in_band_mask_vpu3, Peq[base3]);\n X = _mm_or_si128(Peq[CharToUint8(text[i])], VN);\n D0 = _mm_and_si128(X, VP);\n D0 = _mm_add_epi32(D0, VP);\n D0 = _mm_xor_si128(D0, VP);\n D0 = _mm_or_si128(D0, X);\n HN = _mm_and_si128(VP, D0);\n HP = _mm_or_si128(VP, D0);\n HP = _mm_xor_si128(HP, max_mask_vpu);\n HP = _mm_or_si128(HP, VN);\n X = _mm_srli_epi32(D0, 1);\n VN = _mm_and_si128(X, HP);\n VP = _mm_or_si128(X, HP);\n VP = _mm_xor_si128(VP, max_mask_vpu);\n VP = _mm_or_si128(VP, HN);\n __m128i E = _mm_and_si128(D0, lowest_bit_in_band_mask_vpu);\n E = _mm_xor_si128(E, lowest_bit_in_band_mask_vpu);\n num_errors_at_band_start_position_vpu =\n _mm_add_epi32(num_errors_at_band_start_position_vpu, E);\n __m128i early_stop = _mm_cmpgt_epi32(num_errors_at_band_start_position_vpu,\n early_stop_threshold_vpu);\n int tmp = _mm_movemask_epi8(early_stop);\n if (tmp == 0xffff) {\n _mm_store_si128((__m128i *)mapping_edit_distances,\n num_errors_at_band_start_position_vpu);\n return;\n }\n for (int ai = 0; ai < ALPHABET_SIZE; ai++) {\n Peq[ai] = _mm_srli_epi32(Peq[ai], 1);\n }\n }\n int band_start_position = read_length - 1;\n __m128i min_num_errors_vpu = num_errors_at_band_start_position_vpu;\n for (int i = 0; i < 2 * error_threshold; i++) {\n __m128i lowest_bit_in_VP_vpu =\n _mm_and_si128(VP, lowest_bit_in_band_mask_vpu);\n __m128i lowest_bit_in_VN_vpu =\n _mm_and_si128(VN, lowest_bit_in_band_mask_vpu);\n num_errors_at_band_start_position_vpu = _mm_add_epi32(\n num_errors_at_band_start_position_vpu, lowest_bit_in_VP_vpu);\n num_errors_at_band_start_position_vpu = _mm_sub_epi32(\n num_errors_at_band_start_position_vpu, lowest_bit_in_VN_vpu);\n __m128i mapping_end_positions_update_mask_vpu = _mm_cmplt_epi32(\n num_errors_at_band_start_position_vpu, min_num_errors_vpu);\n __m128i mapping_end_positions_update_mask_vpu1 = _mm_cmpeq_epi32(\n num_errors_at_band_start_position_vpu, min_num_errors_vpu);\n int mapping_end_positions_update_mask =\n _mm_movemask_epi8(mapping_end_positions_update_mask_vpu);\n int mapping_end_positions_update_mask1 =\n _mm_movemask_epi8(mapping_end_positions_update_mask_vpu1);\n for (int li = 0; li < 4; ++li) {\n if ((mapping_end_positions_update_mask & 1) == 1 ||\n ((mapping_end_positions_update_mask1 & 1) == 1 &&\n i + 1 == error_threshold)) {\n mapping_end_positions[li] = band_start_position + 1 + i;\n }\n mapping_end_positions_update_mask =\n mapping_end_positions_update_mask >> 4;\n mapping_end_positions_update_mask1 =\n mapping_end_positions_update_mask1 >> 4;\n }\n min_num_errors_vpu = _mm_min_epi32(min_num_errors_vpu,\n num_errors_at_band_start_position_vpu);\n VP = _mm_srli_epi32(VP, 1);\n VN = _mm_srli_epi32(VN, 1);\n }\n _mm_store_si128((__m128i *)mapping_edit_distances, min_num_errors_vpu);\n}\n\nvoid BandedAlign8PatternsToText(int error_threshold, const char **patterns,\n const char *text, int read_length,\n int16_t *mapping_edit_distances,\n int16_t *mapping_end_positions) {\n int ALPHABET_SIZE = 5;\n const char *reference_sequence0 = patterns[0];\n const char *reference_sequence1 = patterns[1];\n const char *reference_sequence2 = patterns[2];\n const char *reference_sequence3 = patterns[3];\n const char *reference_sequence4 = patterns[4];\n const char *reference_sequence5 = patterns[5];\n const char *reference_sequence6 = patterns[6];\n const char *reference_sequence7 = patterns[7];\n uint16_t highest_bit_in_band_mask = 1 << (2 * error_threshold);\n __m128i highest_bit_in_band_mask_vpu0 =\n _mm_set_epi16(0, 0, 0, 0, 0, 0, 0, highest_bit_in_band_mask);\n __m128i highest_bit_in_band_mask_vpu1 =\n _mm_set_epi16(0, 0, 0, 0, 0, 0, highest_bit_in_band_mask, 0);\n __m128i highest_bit_in_band_mask_vpu2 =\n _mm_set_epi16(0, 0, 0, 0, 0, highest_bit_in_band_mask, 0, 0);\n __m128i highest_bit_in_band_mask_vpu3 =\n _mm_set_epi16(0, 0, 0, 0, highest_bit_in_band_mask, 0, 0, 0);\n __m128i highest_bit_in_band_mask_vpu4 =\n _mm_set_epi16(0, 0, 0, highest_bit_in_band_mask, 0, 0, 0, 0);\n __m128i highest_bit_in_band_mask_vpu5 =\n _mm_set_epi16(0, 0, highest_bit_in_band_mask, 0, 0, 0, 0, 0);\n __m128i highest_bit_in_band_mask_vpu6 =\n _mm_set_epi16(0, highest_bit_in_band_mask, 0, 0, 0, 0, 0, 0);\n __m128i highest_bit_in_band_mask_vpu7 =\n _mm_set_epi16(highest_bit_in_band_mask, 0, 0, 0, 0, 0, 0, 0);\n // Init Peq\n __m128i Peq[ALPHABET_SIZE];\n for (int ai = 0; ai < ALPHABET_SIZE; ai++) {\n Peq[ai] = _mm_setzero_si128();\n }\n for (int i = 0; i < 2 * error_threshold; i++) {\n uint8_t base0 = CharToUint8(reference_sequence0[i]);\n uint8_t base1 = CharToUint8(reference_sequence1[i]);\n uint8_t base2 = CharToUint8(reference_sequence2[i]);\n uint8_t base3 = CharToUint8(reference_sequence3[i]);\n uint8_t base4 = CharToUint8(reference_sequence4[i]);\n uint8_t base5 = CharToUint8(reference_sequence5[i]);\n uint8_t base6 = CharToUint8(reference_sequence6[i]);\n uint8_t base7 = CharToUint8(reference_sequence7[i]);\n Peq[base0] = _mm_or_si128(highest_bit_in_band_mask_vpu0, Peq[base0]);\n Peq[base1] = _mm_or_si128(highest_bit_in_band_mask_vpu1, Peq[base1]);\n Peq[base2] = _mm_or_si128(highest_bit_in_band_mask_vpu2, Peq[base2]);\n Peq[base3] = _mm_or_si128(highest_bit_in_band_mask_vpu3, Peq[base3]);\n Peq[base4] = _mm_or_si128(highest_bit_in_band_mask_vpu4, Peq[base4]);\n Peq[base5] = _mm_or_si128(highest_bit_in_band_mask_vpu5, Peq[base5]);\n Peq[base6] = _mm_or_si128(highest_bit_in_band_mask_vpu6, Peq[base6]);\n Peq[base7] = _mm_or_si128(highest_bit_in_band_mask_vpu7, Peq[base7]);\n for (int ai = 0; ai < ALPHABET_SIZE; ai++) {\n Peq[ai] = _mm_srli_epi16(Peq[ai], 1);\n }\n }\n\n uint16_t lowest_bit_in_band_mask = 1;\n __m128i lowest_bit_in_band_mask_vpu = _mm_set1_epi16(lowest_bit_in_band_mask);\n __m128i VP = _mm_setzero_si128();\n __m128i VN = _mm_setzero_si128();\n __m128i X = _mm_setzero_si128();\n __m128i D0 = _mm_setzero_si128();\n __m128i HN = _mm_setzero_si128();\n __m128i HP = _mm_setzero_si128();\n __m128i max_mask_vpu = _mm_set1_epi16(0xffff);\n __m128i num_errors_at_band_start_position_vpu = _mm_setzero_si128();\n __m128i early_stop_threshold_vpu = _mm_set1_epi16(error_threshold * 3);\n for (int i = 0; i < read_length; i++) {\n uint8_t base0 = CharToUint8(reference_sequence0[i + 2 * error_threshold]);\n uint8_t base1 = CharToUint8(reference_sequence1[i + 2 * error_threshold]);\n uint8_t base2 = CharToUint8(reference_sequence2[i + 2 * error_threshold]);\n uint8_t base3 = CharToUint8(reference_sequence3[i + 2 * error_threshold]);\n uint8_t base4 = CharToUint8(reference_sequence4[i + 2 * error_threshold]);\n uint8_t base5 = CharToUint8(reference_sequence5[i + 2 * error_threshold]);\n uint8_t base6 = CharToUint8(reference_sequence6[i + 2 * error_threshold]);\n uint8_t base7 = CharToUint8(reference_sequence7[i + 2 * error_threshold]);\n Peq[base0] = _mm_or_si128(highest_bit_in_band_mask_vpu0, Peq[base0]);\n Peq[base1] = _mm_or_si128(highest_bit_in_band_mask_vpu1, Peq[base1]);\n Peq[base2] = _mm_or_si128(highest_bit_in_band_mask_vpu2, Peq[base2]);\n Peq[base3] = _mm_or_si128(highest_bit_in_band_mask_vpu3, Peq[base3]);\n Peq[base4] = _mm_or_si128(highest_bit_in_band_mask_vpu4, Peq[base4]);\n Peq[base5] = _mm_or_si128(highest_bit_in_band_mask_vpu5, Peq[base5]);\n Peq[base6] = _mm_or_si128(highest_bit_in_band_mask_vpu6, Peq[base6]);\n Peq[base7] = _mm_or_si128(highest_bit_in_band_mask_vpu7, Peq[base7]);\n X = _mm_or_si128(Peq[CharToUint8(text[i])], VN);\n D0 = _mm_and_si128(X, VP);\n D0 = _mm_add_epi16(D0, VP);\n D0 = _mm_xor_si128(D0, VP);\n D0 = _mm_or_si128(D0, X);\n HN = _mm_and_si128(VP, D0);\n HP = _mm_or_si128(VP, D0);\n HP = _mm_xor_si128(HP, max_mask_vpu);\n HP = _mm_or_si128(HP, VN);\n X = _mm_srli_epi16(D0, 1);\n VN = _mm_and_si128(X, HP);\n VP = _mm_or_si128(X, HP);\n VP = _mm_xor_si128(VP, max_mask_vpu);\n VP = _mm_or_si128(VP, HN);\n __m128i E = _mm_and_si128(D0, lowest_bit_in_band_mask_vpu);\n E = _mm_xor_si128(E, lowest_bit_in_band_mask_vpu);\n num_errors_at_band_start_position_vpu =\n _mm_add_epi16(num_errors_at_band_start_position_vpu, E);\n __m128i early_stop = _mm_cmpgt_epi16(num_errors_at_band_start_position_vpu,\n early_stop_threshold_vpu);\n int tmp = _mm_movemask_epi8(early_stop);\n if (tmp == 0xffff) {\n _mm_store_si128((__m128i *)mapping_edit_distances,\n num_errors_at_band_start_position_vpu);\n return;\n }\n for (int ai = 0; ai < ALPHABET_SIZE; ai++) {\n Peq[ai] = _mm_srli_epi16(Peq[ai], 1);\n }\n }\n int band_start_position = read_length - 1;\n __m128i min_num_errors_vpu = num_errors_at_band_start_position_vpu;\n for (int i = 0; i < 2 * error_threshold; i++) {\n __m128i lowest_bit_in_VP_vpu =\n _mm_and_si128(VP, lowest_bit_in_band_mask_vpu);\n __m128i lowest_bit_in_VN_vpu =\n _mm_and_si128(VN, lowest_bit_in_band_mask_vpu);\n num_errors_at_band_start_position_vpu = _mm_add_epi16(\n num_errors_at_band_start_position_vpu, lowest_bit_in_VP_vpu);\n num_errors_at_band_start_position_vpu = _mm_sub_epi16(\n num_errors_at_band_start_position_vpu, lowest_bit_in_VN_vpu);\n __m128i mapping_end_positions_update_mask_vpu = _mm_cmplt_epi16(\n num_errors_at_band_start_position_vpu, min_num_errors_vpu);\n __m128i mapping_end_positions_update_mask_vpu1 = _mm_cmpeq_epi16(\n num_errors_at_band_start_position_vpu, min_num_errors_vpu);\n int mapping_end_positions_update_mask =\n _mm_movemask_epi8(mapping_end_positions_update_mask_vpu);\n int mapping_end_positions_update_mask1 =\n _mm_movemask_epi8(mapping_end_positions_update_mask_vpu1);\n for (int li = 0; li < 8; ++li) {\n if ((mapping_end_positions_update_mask & 1) == 1 ||\n ((mapping_end_positions_update_mask1 & 1) == 1 &&\n i + 1 == error_threshold)) {\n mapping_end_positions[li] = band_start_position + 1 + i;\n }\n mapping_end_positions_update_mask =\n mapping_end_positions_update_mask >> 2;\n mapping_end_positions_update_mask1 =\n mapping_end_positions_update_mask1 >> 2;\n }\n min_num_errors_vpu = _mm_min_epi16(min_num_errors_vpu,\n num_errors_at_band_start_position_vpu);\n VP = _mm_srli_epi16(VP, 1);\n VN = _mm_srli_epi16(VN, 1);\n }\n _mm_store_si128((__m128i *)mapping_edit_distances, min_num_errors_vpu);\n}\n\nvoid BandedTraceback(int error_threshold, int min_num_errors,\n const char *pattern, const char *text,\n const int read_length, int *mapping_start_position) {\n // fisrt calculate the hamming distance and see whether it's equal to # errors\n if (min_num_errors == 0) {\n *mapping_start_position = error_threshold;\n return;\n }\n int error_count = 0;\n for (int i = 0; i < read_length; ++i) {\n if (pattern[i + error_threshold] != text[i]) {\n ++error_count;\n }\n }\n if (error_count == min_num_errors) {\n *mapping_start_position = error_threshold;\n return;\n }\n // if not then there are gaps so that we have to traceback with edit distance.\n uint32_t Peq[5] = {0, 0, 0, 0, 0};\n for (int i = 0; i < 2 * error_threshold; i++) {\n uint8_t base =\n CharToUint8(pattern[read_length - 1 + 2 * error_threshold - i]);\n Peq[base] = Peq[base] | (1 << i);\n }\n uint32_t highest_bit_in_band_mask = 1 << (2 * error_threshold);\n uint32_t lowest_bit_in_band_mask = 1;\n uint32_t VP = 0;\n uint32_t VN = 0;\n uint32_t X = 0;\n uint32_t D0 = 0;\n uint32_t HN = 0;\n uint32_t HP = 0;\n int num_errors_at_band_start_position = 0;\n for (int i = 0; i < read_length; i++) {\n uint8_t pattern_base = CharToUint8(pattern[read_length - 1 - i]);\n Peq[pattern_base] = Peq[pattern_base] | highest_bit_in_band_mask;\n X = Peq[CharToUint8(text[read_length - 1 - i])] | VN;\n D0 = ((VP + (X & VP)) ^ VP) | X;\n HN = VP & D0;\n HP = VN | ~(VP | D0);\n X = D0 >> 1;\n VN = X & HP;\n VP = HN | ~(X | HP);\n num_errors_at_band_start_position += 1 - (D0 & lowest_bit_in_band_mask);\n for (int ai = 0; ai < 5; ai++) {\n Peq[ai] >>= 1;\n }\n }\n *mapping_start_position = 2 * error_threshold;\n for (int i = 0; i < 2 * error_threshold; i++) {\n num_errors_at_band_start_position =\n num_errors_at_band_start_position + ((VP >> i) & (uint32_t)1);\n num_errors_at_band_start_position =\n num_errors_at_band_start_position - ((VN >> i) & (uint32_t)1);\n if (num_errors_at_band_start_position == min_num_errors) {\n *mapping_start_position = 2 * error_threshold - (1 + i);\n if (i + 1 == error_threshold) {\n return;\n }\n }\n }\n}\n\nvoid BandedTracebackToEnd(int error_threshold, int min_num_errors,\n const char *pattern, const char *text,\n const int read_length, int *mapping_end_position) {\n // fisrt calculate the hamming distance and see whether it's equal to # errors\n if (min_num_errors == 0) {\n *mapping_end_position = read_length + error_threshold;\n return;\n }\n int error_count = 0;\n for (int i = 0; i < read_length; ++i) {\n if (pattern[i + error_threshold] != text[i]) {\n ++error_count;\n }\n }\n if (error_count == min_num_errors) {\n *mapping_end_position = read_length + error_threshold;\n return;\n }\n // if not then there are gaps so that we have to traceback with edit distance.\n uint32_t Peq[5] = {0, 0, 0, 0, 0};\n for (int i = 0; i < 2 * error_threshold; i++) {\n uint8_t base = CharToUint8(pattern[i]);\n Peq[base] = Peq[base] | (1 << i);\n }\n uint32_t highest_bit_in_band_mask = 1 << (2 * error_threshold);\n uint32_t lowest_bit_in_band_mask = 1;\n uint32_t VP = 0;\n uint32_t VN = 0;\n uint32_t X = 0;\n uint32_t D0 = 0;\n uint32_t HN = 0;\n uint32_t HP = 0;\n int num_errors_at_band_start_position = 0;\n for (int i = 0; i < read_length; i++) {\n // printf(\"=>%d %d %c %c\\n\", i, num_errors_at_band_start_position, pattern[i\n // + 2 * error_threshold], text[i]) ;\n uint8_t pattern_base = CharToUint8(pattern[i + 2 * error_threshold]);\n Peq[pattern_base] = Peq[pattern_base] | highest_bit_in_band_mask;\n X = Peq[CharToUint8(text[i])] | VN;\n D0 = ((VP + (X & VP)) ^ VP) | X;\n HN = VP & D0;\n HP = VN | ~(VP | D0);\n X = D0 >> 1;\n VN = X & HP;\n VP = HN | ~(X | HP);\n num_errors_at_band_start_position += 1 - (D0 & lowest_bit_in_band_mask);\n for (int ai = 0; ai < 5; ai++) {\n Peq[ai] >>= 1;\n }\n }\n int band_start_position = read_length;\n *mapping_end_position = band_start_position + 1;\n for (int i = 0; i < 2 * error_threshold; i++) {\n num_errors_at_band_start_position =\n num_errors_at_band_start_position + ((VP >> i) & (uint32_t)1);\n num_errors_at_band_start_position =\n num_errors_at_band_start_position - ((VN >> i) & (uint32_t)1);\n if (num_errors_at_band_start_position == min_num_errors) {\n *mapping_end_position = band_start_position + (i + 1);\n if (i + 1 == error_threshold) {\n return;\n }\n }\n }\n}\n\n} // namespace chromap\n"
},
{
"path": "src/alignment.h",
"content": "#ifndef ALIGNMENT_H_\n#define ALIGNMENT_H_\n\n#include \"mapping_in_memory.h\"\n#include \"sam_mapping.h\"\n#include \"sequence_batch.h\"\n#include \"utils.h\"\n\nnamespace chromap {\n\nint GetLongestMatchLength(const char *pattern, const char *text,\n const int read_length);\n\n// Return newly adjusted reference start/end position for kPositive/kNegative\n// mappings.\nint AdjustGapBeginning(const Strand mapping_strand, const char *ref,\n const char *read, int *gap_beginning, int read_end,\n int ref_start_position, int ref_end_position,\n int *n_cigar, uint32_t **cigar);\n\n// Reference (pattern) mapping start postion and cigar must be computed before\n// calling this function. Read (text) must be already at the start position.\nvoid GenerateNMAndMDTag(const char *pattern, const char *text,\n int mapping_start_position,\n MappingInMemory &mapping_in_memory);\n\nint BandedAlignPatternToText(int error_threshold, const char *pattern,\n const char *text, const int read_length,\n int *mapping_end_position);\n\n// Return negative number if the termination are deemed at the beginning of the\n// read mappping_end_position is relative to pattern (reference)\n// read_mapping_length is for text (read)\nint BandedAlignPatternToTextWithDropOff(int error_threshold,\n const char *pattern, const char *text,\n const int read_length,\n int *mapping_end_position,\n int *read_mapping_length);\n\nint BandedAlignPatternToTextWithDropOffFrom3End(\n int error_threshold, const char *pattern, const char *text,\n const int read_length, int *mapping_end_position, int *read_mapping_length);\n\nvoid BandedAlign4PatternsToText(int error_threshold, const char **patterns,\n const char *text, int read_length,\n int32_t *mapping_edit_distances,\n int32_t *mapping_end_positions);\n\nvoid BandedAlign8PatternsToText(int error_threshold, const char **patterns,\n const char *text, int read_length,\n int16_t *mapping_edit_distances,\n int16_t *mapping_end_positions);\n\nvoid BandedTraceback(int error_threshold, int min_num_errors,\n const char *pattern, const char *text,\n const int read_length, int *mapping_start_position);\n\nvoid BandedTracebackToEnd(int error_threshold, int min_num_errors,\n const char *pattern, const char *text,\n const int read_length, int *mapping_end_position);\n\n} // namespace chromap\n\n#endif // ALIGNMENT_H_\n"
},
{
"path": "src/barcode_translator.h",
"content": "#ifndef BARCODETRANSLATOR_H_\n#define BARCODETRANSLATOR_H_\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n\n#include \"khash.h\"\n#include \"utils.h\"\n\nnamespace chromap {\n\nKHASH_INIT(k64_str, uint64_t, char *, 1, kh_int64_hash_func,\n kh_int64_hash_equal);\n\n// The class for handling barcode convertion.\nclass BarcodeTranslator {\n public:\n BarcodeTranslator() {\n barcode_translate_table_ = NULL;\n from_bc_length_ = -1;\n }\n\n ~BarcodeTranslator() {\n if (barcode_translate_table_ != NULL) {\n khiter_t k;\n for (k = kh_begin(barcode_translate_table_);\n k != kh_end(barcode_translate_table_); ++k) {\n if (kh_exist(barcode_translate_table_, k))\n free(kh_value(barcode_translate_table_, k));\n }\n kh_destroy(k64_str, barcode_translate_table_);\n }\n }\n\n void SetTranslateTable(const std::string &file) {\n barcode_translate_table_ = kh_init(k64_str);\n \n if (1) {\n gzFile barcode_translate_file = gzopen(file.c_str(), \"r\");\n const uint32_t line_buffer_size = 512;\n char file_line[line_buffer_size];\n while (gzgets(barcode_translate_file, file_line, line_buffer_size) != NULL) {\n int line_len = strlen(file_line);\n if (file_line[line_len - 1] == '\\n') {\n file_line[line_len - 1] = '\\0';\n }\n std::string tmp_string(file_line);\n ProcessTranslateFileLine(tmp_string);\n }\n } else {\n // Old implementation, which does not support gzipped input.\n std::ifstream file_stream(file);\n std::string file_line;\n while (getline(file_stream, file_line)) {\n ProcessTranslateFileLine(file_line);\n }\n }\n\n mask_ = (1ull << (2 * from_bc_length_)) - 1;\n /*for (int i = 0; i < from_bc_length_; ++i)\n {\n mask_ |= (3ull << (2*i));\n }*/\n }\n\n std::string Translate(uint64_t bc, uint32_t bc_length) {\n if (barcode_translate_table_ == NULL) {\n return Seed2Sequence(bc, bc_length);\n }\n\n std::string ret;\n uint64_t i;\n for (i = 0; i < bc_length / from_bc_length_; ++i) {\n uint64_t seed = (bc << (2 * i * from_bc_length_)) >>\n (2 * (bc_length / from_bc_length_ - 1) * from_bc_length_);\n seed &= mask_;\n khiter_t barcode_translate_table_iter =\n kh_get(k64_str, barcode_translate_table_, seed);\n if (barcode_translate_table_iter == kh_end(barcode_translate_table_)) {\n std::cerr << \"Barcode does not exist in the translation table.\"\n << std::endl;\n exit(-1);\n }\n std::string bc_to(\n kh_value(barcode_translate_table_, barcode_translate_table_iter));\n if (i == 0) {\n ret = bc_to;\n } else {\n ret += \"-\" + bc_to;\n }\n }\n return ret;\n }\n\n private:\n khash_t(k64_str) * barcode_translate_table_;\n int from_bc_length_;\n uint64_t mask_;\n\n std::string Seed2Sequence(uint64_t seed, uint32_t seed_length) const {\n std::string sequence;\n sequence.reserve(seed_length);\n uint64_t mask_ = 3;\n for (uint32_t i = 0; i < seed_length; ++i) {\n sequence.push_back(\n Uint8ToChar((seed >> ((seed_length - 1 - i) * 2)) & mask_));\n }\n return sequence;\n }\n\n void ProcessTranslateFileLine(std::string &line) {\n int i;\n int len = line.length();\n std::string to;\n for (i = 0; i < len; ++i) {\n if (line[i] == ',' || line[i] == '\\t') break;\n }\n\n to = line.substr(0, i);\n // from = line.substr(i + 1, len - i - 1);\n from_bc_length_ = len - i - 1;\n uint64_t from_seed =\n GenerateSeedFromSequence(line.c_str(), len, i + 1, from_bc_length_);\n\n int khash_return_code;\n khiter_t barcode_translate_table_iter = kh_put(\n k64_str, barcode_translate_table_, from_seed, &khash_return_code);\n kh_value(barcode_translate_table_, barcode_translate_table_iter) =\n strdup(to.c_str());\n }\n};\n\n} // namespace chromap\n#endif\n"
},
{
"path": "src/bed_mapping.h",
"content": "#ifndef BEDMAPPING_H_\n#define BEDMAPPING_H_\n\n#include \n\n#include \"mapping.h\"\n\nnamespace chromap {\n\nclass MappingWithBarcode : public Mapping {\n public:\n uint32_t read_id_;\n uint64_t cell_barcode_;\n uint32_t fragment_start_position_;\n uint16_t fragment_length_;\n uint8_t mapq_ : 6, direction_ : 1, is_unique_ : 1;\n uint8_t num_dups_;\n // uint8_t mapq;\n MappingWithBarcode() : num_dups_(0) {}\n MappingWithBarcode(uint32_t read_id, uint64_t cell_barcode,\n uint32_t fragment_start_position, uint16_t fragment_length,\n uint8_t mapq, uint8_t direction, uint8_t is_unique,\n uint8_t num_dups)\n : read_id_(read_id),\n cell_barcode_(cell_barcode),\n fragment_start_position_(fragment_start_position),\n fragment_length_(fragment_length),\n mapq_(mapq),\n direction_(direction),\n is_unique_(is_unique),\n num_dups_(num_dups) {}\n bool operator<(const MappingWithBarcode &m) const {\n return std::tie(fragment_start_position_, fragment_length_, cell_barcode_,\n mapq_, direction_, is_unique_, read_id_) <\n std::tie(m.fragment_start_position_, m.fragment_length_,\n m.cell_barcode_, m.mapq_, m.direction_, m.is_unique_,\n m.read_id_);\n }\n bool operator==(const MappingWithBarcode &m) const {\n return std::tie(cell_barcode_, fragment_start_position_) ==\n std::tie(m.cell_barcode_, m.fragment_start_position_);\n }\n bool IsSamePosition(const MappingWithBarcode &m) const {\n return std::tie(fragment_start_position_) ==\n std::tie(m.fragment_start_position_);\n }\n uint64_t GetBarcode() const { return cell_barcode_; }\n void Tn5Shift() {\n if (direction_ == 1) {\n fragment_start_position_ += 4;\n } else {\n fragment_length_ -= 5;\n }\n }\n bool IsPositiveStrand() const { return direction_ > 0 ? true : false; }\n uint32_t GetStartPosition() const { // inclusive\n return fragment_start_position_;\n }\n uint32_t GetEndPosition() const { // exclusive\n return fragment_start_position_ + fragment_length_;\n }\n};\n\nclass MappingWithoutBarcode : public Mapping {\n public:\n uint32_t read_id_;\n uint32_t fragment_start_position_;\n uint16_t fragment_length_;\n // uint8_t mapq;\n uint8_t mapq_ : 6, direction_ : 1, is_unique_ : 1;\n uint16_t num_dups_; // Need higher limit in bulk setting\n\n MappingWithoutBarcode() : num_dups_(0) {}\n MappingWithoutBarcode(uint32_t read_id, uint32_t fragment_start_position,\n uint16_t fragment_length, uint16_t mapq,\n uint8_t direction, uint8_t is_unique, uint8_t num_dups)\n : read_id_(read_id),\n fragment_start_position_(fragment_start_position),\n fragment_length_(fragment_length),\n mapq_(mapq),\n direction_(direction),\n is_unique_(is_unique),\n num_dups_(num_dups) {}\n\n bool operator<(const MappingWithoutBarcode &m) const {\n return std::tie(fragment_start_position_, fragment_length_, mapq_,\n direction_, is_unique_, read_id_) <\n std::tie(m.fragment_start_position_, m.fragment_length_, m.mapq_,\n m.direction_, m.is_unique_, m.read_id_);\n }\n bool operator==(const MappingWithoutBarcode &m) const {\n return std::tie(fragment_start_position_) ==\n std::tie(m.fragment_start_position_);\n }\n bool IsSamePosition(const MappingWithoutBarcode &m) const {\n return std::tie(fragment_start_position_) ==\n std::tie(m.fragment_start_position_);\n }\n uint64_t GetBarcode() const { return 0; }\n void Tn5Shift() {\n if (direction_ == 1) {\n fragment_start_position_ += 4;\n } else {\n fragment_length_ -= 5;\n }\n }\n bool IsPositiveStrand() const { return direction_ > 0 ? true : false; }\n uint32_t GetStartPosition() const { // inclusive\n return fragment_start_position_;\n }\n uint32_t GetEndPosition() const { // exclusive\n return fragment_start_position_ + fragment_length_;\n }\n};\n\nclass PairedEndMappingWithBarcode : public Mapping {\n public:\n uint32_t read_id_;\n uint64_t cell_barcode_;\n uint32_t fragment_start_position_;\n uint16_t fragment_length_;\n uint8_t mapq_ : 6, direction_ : 1, is_unique_ : 1;\n uint8_t num_dups_;\n // uint8_t mapq;\n uint16_t positive_alignment_length_;\n uint16_t negative_alignment_length_;\n PairedEndMappingWithBarcode() : num_dups_(0) {}\n PairedEndMappingWithBarcode(uint32_t read_id, uint64_t cell_barcode,\n uint32_t fragment_start_position,\n uint16_t fragment_length, uint8_t mapq,\n uint8_t direction, uint8_t is_unique,\n uint8_t num_dups,\n uint16_t positive_alignment_length,\n uint16_t negative_alignment_length)\n : read_id_(read_id),\n cell_barcode_(cell_barcode),\n fragment_start_position_(fragment_start_position),\n fragment_length_(fragment_length),\n mapq_(mapq),\n direction_(direction),\n is_unique_(is_unique),\n num_dups_(num_dups),\n positive_alignment_length_(positive_alignment_length),\n negative_alignment_length_(negative_alignment_length) {}\n bool operator<(const PairedEndMappingWithBarcode &m) const {\n return std::tie(fragment_start_position_, fragment_length_, cell_barcode_,\n mapq_, direction_, is_unique_, read_id_,\n positive_alignment_length_, negative_alignment_length_) <\n std::tie(m.fragment_start_position_, m.fragment_length_,\n m.cell_barcode_, m.mapq_, m.direction_, m.is_unique_,\n m.read_id_, m.positive_alignment_length_,\n m.negative_alignment_length_);\n }\n bool operator==(const PairedEndMappingWithBarcode &m) const {\n return std::tie(cell_barcode_, fragment_start_position_,\n fragment_length_) == std::tie(m.cell_barcode_,\n m.fragment_start_position_,\n m.fragment_length_);\n }\n bool IsSamePosition(const PairedEndMappingWithBarcode &m) const {\n return std::tie(fragment_start_position_, fragment_length_) ==\n std::tie(m.fragment_start_position_, m.fragment_length_);\n }\n uint64_t GetBarcode() const { return cell_barcode_; }\n void Tn5Shift() {\n fragment_start_position_ += 4;\n positive_alignment_length_ -= 4;\n fragment_length_ -= 9;\n negative_alignment_length_ -= 5;\n }\n bool IsPositiveStrand() const { return direction_ > 0 ? true : false; }\n uint32_t GetStartPosition() const { // inclusive\n return fragment_start_position_;\n }\n uint32_t GetEndPosition() const { // exclusive\n return fragment_start_position_ + fragment_length_;\n }\n};\n\nclass PairedEndMappingWithoutBarcode : public Mapping {\n public:\n uint32_t read_id_;\n uint32_t fragment_start_position_;\n uint16_t fragment_length_;\n uint8_t mapq_ : 6, direction_ : 1, is_unique_ : 1;\n uint8_t num_dups_;\n // uint8_t mapq;\n uint16_t positive_alignment_length_;\n uint16_t negative_alignment_length_;\n PairedEndMappingWithoutBarcode() : num_dups_(0) {}\n PairedEndMappingWithoutBarcode(uint32_t read_id,\n uint32_t fragment_start_position,\n uint16_t fragment_length, uint8_t mapq,\n uint8_t direction, uint8_t is_unique,\n uint16_t num_dups,\n uint16_t positive_alignment_length,\n uint16_t negative_alignment_length)\n : read_id_(read_id),\n fragment_start_position_(fragment_start_position),\n fragment_length_(fragment_length),\n mapq_(mapq),\n direction_(direction),\n is_unique_(is_unique),\n num_dups_(num_dups),\n positive_alignment_length_(positive_alignment_length),\n negative_alignment_length_(negative_alignment_length) {}\n\n bool operator<(const PairedEndMappingWithoutBarcode &m) const {\n return std::tie(fragment_start_position_, fragment_length_, mapq_,\n direction_, is_unique_, read_id_,\n positive_alignment_length_, negative_alignment_length_) <\n std::tie(m.fragment_start_position_, m.fragment_length_, m.mapq_,\n m.direction_, m.is_unique_, m.read_id_,\n m.positive_alignment_length_, m.negative_alignment_length_);\n }\n bool operator==(const PairedEndMappingWithoutBarcode &m) const {\n return std::tie(fragment_start_position_, fragment_length_) ==\n std::tie(m.fragment_start_position_, m.fragment_length_);\n }\n bool IsSamePosition(const PairedEndMappingWithoutBarcode &m) const {\n return std::tie(fragment_start_position_, fragment_length_) ==\n std::tie(m.fragment_start_position_, m.fragment_length_);\n }\n uint64_t GetBarcode() const { return 0; }\n void Tn5Shift() {\n fragment_start_position_ += 4;\n positive_alignment_length_ -= 4;\n fragment_length_ -= 9;\n negative_alignment_length_ -= 5;\n }\n bool IsPositiveStrand() const { return direction_ > 0 ? true : false; }\n uint32_t GetStartPosition() const { // inclusive\n return fragment_start_position_;\n }\n uint32_t GetEndPosition() const { // exclusive\n return fragment_start_position_ + fragment_length_;\n }\n};\n\n} // namespace chromap\n\n#endif // BEDMAPPING_H_\n"
},
{
"path": "src/candidate.h",
"content": "#ifndef CANDIDATE_H_\n#define CANDIDATE_H_\n\n#include \n\nnamespace chromap {\n\nstruct Candidate {\n // The high 32 bits save the reference sequence index in the reference\n // sequence batch. The low 32 bits save the reference position on that\n // sequence.\n uint64_t position = 0;\n\n // The number of minimizers supports the position.\n uint8_t count = 0;\n\n inline uint32_t GetReferenceSequenceIndex() const { return (position >> 32); }\n\n inline uint32_t GetReferenceSequencePosition() const { return position; }\n\n inline uint8_t GetCount() { return count; }\n\n inline bool operator<(const Candidate &c) const {\n if (count > c.count) {\n return true;\n }\n\n if (count < c.count) {\n return false;\n }\n\n return position < c.position;\n }\n};\n\n} // namespace chromap\n\n#endif // CANDIDATE_H_\n"
},
{
"path": "src/candidate_position_generating_config.h",
"content": "#ifndef CANDIDATE_POSITION_GENERATING_CONFIG_H_\n#define CANDIDATE_POSITION_GENERATING_CONFIG_H_\n\n#include \n\nnamespace chromap {\n\n// This class holds the parameters to generate candidate position. Using the\n// parameters, it can check whether a seed is frequent or repetitive.\nclass CandidatePositionGeneratingConfig {\n public:\n CandidatePositionGeneratingConfig() = delete;\n\n CandidatePositionGeneratingConfig(uint32_t max_seed_frequency,\n uint32_t repetitive_seed_frequency,\n bool use_heap_merge)\n : max_seed_frequency_(max_seed_frequency),\n repetitive_seed_frequency_(repetitive_seed_frequency),\n use_heap_merge_(use_heap_merge) {}\n\n ~CandidatePositionGeneratingConfig() = default;\n\n inline bool IsFrequentSeed(uint32_t seed_frequency) const {\n return seed_frequency >= max_seed_frequency_;\n }\n\n inline bool IsRepetitiveSeed(uint32_t seed_frequency) const {\n return seed_frequency >= repetitive_seed_frequency_;\n }\n\n inline bool UseHeapMerge() const { return use_heap_merge_; }\n\n inline uint32_t GetMaxSeedFrequency() const { return max_seed_frequency_; }\n\n private:\n // Only seeds with frequency less than this threshold will be used.\n const uint32_t max_seed_frequency_;\n\n // Seeds with frequency greater than or equal to this threshold will be\n // considered as repetitive seeds.\n const uint32_t repetitive_seed_frequency_;\n\n // When the number of candidate positions is really large, use heap merge to\n // merge sorted candidate lists.\n const bool use_heap_merge_;\n};\n\n} // namespace chromap\n\n#endif // CANDIDATE_POSITION_GENERATING_CONFIG_H_\n"
},
{
"path": "src/candidate_processor.cc",
"content": "#include \"candidate_processor.h\"\n\n#include \n#include \n#include \n#include \n#include \n#include \n\nnamespace chromap {\n\nvoid CandidateProcessor::GenerateCandidates(\n int error_threshold, const Index &index,\n MappingMetadata &mapping_metadata) const {\n const std::vector &minimizers = mapping_metadata.minimizers_;\n std::vector &positive_hits = mapping_metadata.positive_hits_;\n std::vector &negative_hits = mapping_metadata.negative_hits_;\n std::vector &positive_candidates =\n mapping_metadata.positive_candidates_;\n std::vector &negative_candidates =\n mapping_metadata.negative_candidates_;\n uint32_t &repetitive_seed_length = mapping_metadata.repetitive_seed_length_;\n\n const CandidatePositionGeneratingConfig first_round_generating_config(\n /*max_seed_frequency=*/max_seed_frequencies_[0],\n /*repetitive_seed_frequency=*/max_seed_frequencies_[0],\n /*use_heap_merge=*/false);\n\n repetitive_seed_length = 0;\n int repetitive_seed_count = index.GenerateCandidatePositions(\n first_round_generating_config, mapping_metadata);\n\n bool use_high_frequency_minimizers = false;\n if (positive_hits.size() + negative_hits.size() == 0) {\n positive_hits.clear();\n negative_hits.clear();\n repetitive_seed_length = 0;\n\n const CandidatePositionGeneratingConfig second_round_generating_config(\n /*max_seed_frequency=*/max_seed_frequencies_[1],\n /*repetitive_seed_frequency=*/max_seed_frequencies_[0],\n /*use_heap_merge=*/true);\n\n repetitive_seed_count = index.GenerateCandidatePositions(\n second_round_generating_config, mapping_metadata);\n use_high_frequency_minimizers = true;\n if (positive_hits.size() == 0 || negative_hits.size() == 0) {\n use_high_frequency_minimizers = false;\n }\n }\n\n int num_required_seeds = minimizers.size() - repetitive_seed_count;\n num_required_seeds = num_required_seeds > 1 ? num_required_seeds : 1;\n num_required_seeds = num_required_seeds > min_num_seeds_required_for_mapping_\n ? min_num_seeds_required_for_mapping_\n : num_required_seeds;\n if (use_high_frequency_minimizers) {\n num_required_seeds = min_num_seeds_required_for_mapping_;\n }\n\n // std::cerr << \"Normal positive gen on one dir\\n\";\n GenerateCandidatesOnOneStrand(error_threshold, num_required_seeds,\n minimizers.size(), positive_hits,\n positive_candidates);\n // std::cerr << \"Normal negative gen on one dir\\n\";\n GenerateCandidatesOnOneStrand(error_threshold, num_required_seeds,\n minimizers.size(), negative_hits,\n negative_candidates);\n // fprintf(stderr, \"p+n: %d\\n\", positive_candidates->size() +\n // negative_candidates->size()) ;\n}\n\n// Return 0 if it supplements normally. Return 1 if the supplement could be too\n// aggressive, and MAPQ needs setting to 0.\nint CandidateProcessor::SupplementCandidates(\n int error_threshold, uint32_t search_range, const Index &index,\n PairedEndMappingMetadata &paired_end_mapping_metadata) const {\n std::vector augment_positive_candidates1;\n std::vector augment_positive_candidates2;\n std::vector augment_negative_candidates1;\n std::vector augment_negative_candidates2;\n\n int ret = 0;\n\n for (int mate = 0; mate <= 1; ++mate) {\n std::vector *minimizers;\n std::vector *positive_hits;\n std::vector *negative_hits;\n std::vector *positive_candidates;\n std::vector *negative_candidates;\n std::vector *mate_positive_candidates;\n std::vector *mate_negative_candidates;\n std::vector *augment_positive_candidates;\n std::vector *augment_negative_candidates;\n uint32_t *repetitive_seed_length;\n\n if (mate == 0) {\n minimizers = &paired_end_mapping_metadata.mapping_metadata1_.minimizers_;\n positive_hits =\n &paired_end_mapping_metadata.mapping_metadata1_.positive_hits_;\n negative_hits =\n &paired_end_mapping_metadata.mapping_metadata1_.negative_hits_;\n positive_candidates =\n &paired_end_mapping_metadata.mapping_metadata1_.positive_candidates_;\n negative_candidates =\n &paired_end_mapping_metadata.mapping_metadata1_.negative_candidates_;\n mate_positive_candidates =\n &paired_end_mapping_metadata.mapping_metadata2_.positive_candidates_;\n mate_negative_candidates =\n &paired_end_mapping_metadata.mapping_metadata2_.negative_candidates_;\n augment_positive_candidates = &augment_positive_candidates1;\n augment_negative_candidates = &augment_negative_candidates1;\n repetitive_seed_length = &paired_end_mapping_metadata.mapping_metadata1_\n .repetitive_seed_length_;\n } else {\n minimizers = &paired_end_mapping_metadata.mapping_metadata2_.minimizers_;\n positive_hits =\n &paired_end_mapping_metadata.mapping_metadata2_.positive_hits_;\n negative_hits =\n &paired_end_mapping_metadata.mapping_metadata2_.negative_hits_;\n positive_candidates =\n &paired_end_mapping_metadata.mapping_metadata2_.positive_candidates_;\n negative_candidates =\n &paired_end_mapping_metadata.mapping_metadata2_.negative_candidates_;\n mate_positive_candidates =\n &paired_end_mapping_metadata.mapping_metadata1_.positive_candidates_;\n mate_negative_candidates =\n &paired_end_mapping_metadata.mapping_metadata1_.negative_candidates_;\n augment_positive_candidates = &augment_positive_candidates2;\n augment_negative_candidates = &augment_negative_candidates2;\n repetitive_seed_length = &paired_end_mapping_metadata.mapping_metadata2_\n .repetitive_seed_length_;\n }\n\n uint32_t mm_count = minimizers->size();\n bool augment_flag = true;\n uint32_t candidate_num = positive_candidates->size();\n\n for (uint32_t i = 0; i < candidate_num; ++i) {\n if ((*positive_candidates)[i].count >= mm_count / 2) {\n augment_flag = false;\n break;\n }\n }\n\n candidate_num = negative_candidates->size();\n if (augment_flag) {\n for (uint32_t i = 0; i < candidate_num; ++i) {\n if ((*negative_candidates)[i].count >= mm_count / 2) {\n augment_flag = false;\n break;\n }\n }\n }\n\n if (augment_flag) {\n positive_hits->clear();\n negative_hits->clear();\n positive_hits->reserve(max_seed_frequencies_[0]);\n negative_hits->reserve(max_seed_frequencies_[0]);\n int positive_rescue_result = 0;\n int negative_rescue_result = 0;\n if (mate_positive_candidates->size() > 0) {\n positive_rescue_result =\n GenerateCandidatesFromRepetitiveReadWithMateInfoOnOneStrand(\n kNegative, search_range, error_threshold, index, *minimizers,\n *mate_positive_candidates, *repetitive_seed_length,\n *negative_hits, *augment_negative_candidates);\n }\n\n if (mate_negative_candidates->size() > 0) {\n negative_rescue_result =\n GenerateCandidatesFromRepetitiveReadWithMateInfoOnOneStrand(\n kPositive, search_range, error_threshold, index, *minimizers,\n *mate_negative_candidates, *repetitive_seed_length,\n *positive_hits, *augment_positive_candidates);\n }\n\n // If one of the strand did not supplement due to too many best candidate,\n // and the filtered strand have better best candidates,\n // and there is no candidate directly from minimizers,\n // then we remove the supplement\n if (((positive_rescue_result < 0 && negative_rescue_result > 0 &&\n -positive_rescue_result >= negative_rescue_result) ||\n (positive_rescue_result > 0 && negative_rescue_result < 0 &&\n positive_rescue_result <= -negative_rescue_result)) &&\n positive_candidates->size() + negative_candidates->size() == 0) {\n // augment_positive_candidates->clear();\n // augment_negative_candidates->clear();\n ret = 1;\n }\n }\n }\n\n if (augment_positive_candidates1.size() > 0) {\n MergeCandidates(\n error_threshold,\n paired_end_mapping_metadata.mapping_metadata1_.positive_candidates_,\n augment_positive_candidates1,\n paired_end_mapping_metadata.mapping_metadata1_\n .positive_candidates_buffer_);\n }\n\n if (augment_negative_candidates1.size() > 0) {\n MergeCandidates(\n error_threshold,\n paired_end_mapping_metadata.mapping_metadata1_.negative_candidates_,\n augment_negative_candidates1,\n paired_end_mapping_metadata.mapping_metadata1_\n .negative_candidates_buffer_);\n }\n\n if (augment_positive_candidates2.size() > 0) {\n MergeCandidates(\n error_threshold,\n paired_end_mapping_metadata.mapping_metadata2_.positive_candidates_,\n augment_positive_candidates2,\n paired_end_mapping_metadata.mapping_metadata2_\n .positive_candidates_buffer_);\n }\n\n if (augment_negative_candidates2.size() > 0) {\n MergeCandidates(\n error_threshold,\n paired_end_mapping_metadata.mapping_metadata2_.negative_candidates_,\n augment_negative_candidates2,\n paired_end_mapping_metadata.mapping_metadata2_\n .negative_candidates_buffer_);\n }\n return ret;\n}\n\nvoid CandidateProcessor::ReduceCandidatesForPairedEndRead(\n uint32_t mapping_positions_distance,\n PairedEndMappingMetadata &paired_end_mapping_metadata) const {\n const std::vector &positive_candidates1 =\n paired_end_mapping_metadata.mapping_metadata1_\n .positive_candidates_buffer_;\n const std::vector &negative_candidates1 =\n paired_end_mapping_metadata.mapping_metadata1_\n .negative_candidates_buffer_;\n const std::vector &positive_candidates2 =\n paired_end_mapping_metadata.mapping_metadata2_\n .positive_candidates_buffer_;\n const std::vector &negative_candidates2 =\n paired_end_mapping_metadata.mapping_metadata2_\n .negative_candidates_buffer_;\n std::vector &filtered_positive_candidates1 =\n paired_end_mapping_metadata.mapping_metadata1_.positive_candidates_;\n std::vector &filtered_negative_candidates1 =\n paired_end_mapping_metadata.mapping_metadata1_.negative_candidates_;\n std::vector &filtered_positive_candidates2 =\n paired_end_mapping_metadata.mapping_metadata2_.positive_candidates_;\n std::vector &filtered_negative_candidates2 =\n paired_end_mapping_metadata.mapping_metadata2_.negative_candidates_;\n\n ReduceCandidatesForPairedEndReadOnOneDirection(\n mapping_positions_distance, positive_candidates1, negative_candidates2,\n filtered_positive_candidates1, filtered_negative_candidates2);\n ReduceCandidatesForPairedEndReadOnOneDirection(\n mapping_positions_distance, negative_candidates1, positive_candidates2,\n filtered_negative_candidates1, filtered_positive_candidates2);\n}\n\nint CandidateProcessor::\n GenerateCandidatesFromRepetitiveReadWithMateInfoOnOneStrand(\n const Strand strand, uint32_t search_range, int error_threshold,\n const Index &index, const std::vector &minimizers,\n const std::vector &mate_candidates,\n uint32_t &repetitive_seed_length, std::vector &hits,\n std::vector &candidates) const {\n int max_seed_count =\n index.GenerateCandidatePositionsFromRepetitiveReadWithMateInfoOnOneStrand(\n strand, search_range, min_num_seeds_required_for_mapping_,\n max_seed_frequencies_[0], error_threshold, minimizers,\n mate_candidates, repetitive_seed_length, hits);\n\n GenerateCandidatesOnOneStrand(error_threshold, /*num_seeds_required=*/1,\n minimizers.size(), hits, candidates);\n return max_seed_count;\n}\n\nvoid CandidateProcessor::GenerateCandidatesOnOneStrand(\n int error_threshold, int num_seeds_required, uint32_t num_minimizers,\n std::vector &hits, std::vector &candidates) const {\n hits.emplace_back(UINT64_MAX);\n if (hits.size() > 0) {\n int minimizer_count = 1;\n // The number of seeds with the exact same reference position.\n int equal_count = 1;\n int best_equal_count = 1;\n uint64_t previous_hit = hits[0];\n uint32_t previous_reference_id = previous_hit >> 32;\n uint32_t previous_reference_position = previous_hit;\n uint64_t best_local_hit = hits[0];\n for (uint32_t pi = 1; pi < hits.size(); ++pi) {\n uint32_t current_reference_id = hits[pi] >> 32;\n uint32_t current_reference_position = hits[pi];\n#ifdef LI_DEBUG\n printf(\"%s: %d %d\\n\", __func__, current_reference_id,\n current_reference_position);\n#endif\n if (current_reference_id != previous_reference_id ||\n current_reference_position >\n previous_reference_position + error_threshold ||\n ((uint32_t)minimizer_count >= num_minimizers &&\n current_reference_position >\n (uint32_t)best_local_hit + error_threshold)) {\n if (minimizer_count >= num_seeds_required) {\n Candidate candidate;\n candidate.position = best_local_hit;\n candidate.count = best_equal_count;\n candidates.push_back(candidate);\n }\n\n minimizer_count = 1;\n equal_count = 1;\n best_equal_count = 1;\n best_local_hit = hits[pi];\n } else {\n if (hits[pi] == best_local_hit) {\n ++equal_count;\n ++best_equal_count;\n } else if (hits[pi] == previous_hit) {\n ++equal_count;\n if (equal_count > best_equal_count) {\n best_local_hit = previous_hit;\n best_equal_count = equal_count;\n }\n } else {\n equal_count = 1;\n }\n\n ++minimizer_count;\n }\n\n previous_hit = hits[pi];\n previous_reference_id = current_reference_id;\n previous_reference_position = current_reference_position;\n }\n }\n}\n\n// Merge c1 and c2 into buffer and then swap the results into c1.\nvoid CandidateProcessor::MergeCandidates(int error_threshold,\n std::vector &c1,\n std::vector &c2,\n std::vector &buffer) const {\n if (c1.size() == 0) {\n c1.swap(c2);\n return;\n }\n\n uint32_t i, j;\n uint32_t size1, size2;\n size1 = c1.size();\n size2 = c2.size();\n buffer.clear();\n\n#ifdef LI_DEBUG\n for (i = 0; i < size1; ++i)\n printf(\"c1: %d %d %d\\n\", (int)(c1[i].position >> 32), (int)c1[i].position,\n c1[i].count);\n for (i = 0; i < size2; ++i)\n printf(\"c2: %d %d %d\\n\", (int)(c2[i].position >> 32), (int)c2[i].position,\n c2[i].count);\n#endif\n\n i = 0;\n j = 0;\n while (i < size1 && j < size2) {\n if (c1[i].position == c2[j].position) {\n if (buffer.empty() ||\n c1[i].position > buffer.back().position + error_threshold) {\n if (c1[i].count > c2[j].count) {\n buffer.push_back(c1[i]);\n } else {\n buffer.push_back(c2[j]);\n }\n }\n ++i, ++j;\n } else if (c1[i].position < c2[j].position) {\n if (buffer.empty() ||\n c1[i].position > buffer.back().position + error_threshold) {\n buffer.push_back(c1[i]);\n }\n ++i;\n } else {\n if (buffer.empty() ||\n c2[j].position > buffer.back().position + error_threshold) {\n buffer.push_back(c2[j]);\n }\n ++j;\n }\n }\n\n while (i < size1) {\n if (buffer.empty() ||\n c1[i].position > buffer.back().position + error_threshold) {\n buffer.push_back(c1[i]);\n }\n ++i;\n }\n\n while (j < size2) {\n if (buffer.empty() ||\n c2[j].position > buffer.back().position + error_threshold) {\n buffer.push_back(c2[j]);\n }\n ++j;\n }\n\n c1.swap(buffer);\n}\n\nvoid CandidateProcessor::ReduceCandidatesForPairedEndReadOnOneDirection(\n uint32_t mapping_positions_distance,\n const std::vector &candidates1,\n const std::vector &candidates2,\n std::vector &filtered_candidates1,\n std::vector &filtered_candidates2) const {\n uint32_t i1 = 0;\n uint32_t i2 = 0;\n int num_unpaired_candidate1 = 0;\n int num_unpaired_candidate2 = 0;\n int num_unpaired_candidate_threshold = 5;\n int max_candidate_count1 = 6;\n int max_candidate_count2 = 6;\n uint32_t previous_end_i2 = i2;\n#ifdef LI_DEBUG\n for (uint32_t i = 0; i < candidates1.size(); ++i)\n printf(\"%s 0: %d %d:%d\\n\", __func__, i,\n (int)(candidates1[i].position >> 32), (int)candidates1[i].position);\n for (uint32_t i = 0; i < candidates2.size(); ++i)\n printf(\"%s 1: %d %d:%d\\n\", __func__, i,\n (int)(candidates2[i].position >> 32), (int)candidates2[i].position);\n#endif\n while (i1 < candidates1.size() && i2 < candidates2.size()) {\n if (candidates1[i1].position >\n candidates2[i2].position + mapping_positions_distance) {\n if (i2 >= previous_end_i2 &&\n num_unpaired_candidate2 < num_unpaired_candidate_threshold &&\n (candidates1[i1].position >> 32) ==\n (candidates2[i2].position >> 32) &&\n candidates2[i2].count >= max_candidate_count2) {\n filtered_candidates2.emplace_back(candidates2[i2]);\n ++num_unpaired_candidate2;\n }\n ++i2;\n } else if (candidates2[i2].position >\n candidates1[i1].position + mapping_positions_distance) {\n if (num_unpaired_candidate1 < num_unpaired_candidate_threshold &&\n (candidates1[i1].position >> 32) ==\n (candidates2[i2].position >> 32) &&\n candidates1[i1].count >= max_candidate_count1) {\n filtered_candidates1.emplace_back(candidates1[i1]);\n ++num_unpaired_candidate1;\n }\n ++i1;\n } else {\n // ok, find a pair, we store current ni2 somewhere and keep looking until\n // we go out of the range, then we go back and then move to next pi1 and\n // keep doing the similar thing.\n filtered_candidates1.emplace_back(candidates1[i1]);\n if (candidates1[i1].count > max_candidate_count1) {\n max_candidate_count1 = candidates1[i1].count;\n }\n uint32_t current_i2 = i2;\n while (current_i2 < candidates2.size() &&\n candidates2[current_i2].position <=\n candidates1[i1].position + mapping_positions_distance) {\n if (current_i2 >= previous_end_i2) {\n filtered_candidates2.emplace_back(candidates2[current_i2]);\n if (candidates2[current_i2].count > max_candidate_count2) {\n max_candidate_count2 = candidates2[current_i2].count;\n }\n }\n ++current_i2;\n }\n previous_end_i2 = current_i2;\n ++i1;\n }\n }\n}\n\n} // namespace chromap\n"
},
{
"path": "src/candidate_processor.h",
"content": "#ifndef CANDIDATE_PROCESSOR_H_\n#define CANDIDATE_PROCESSOR_H_\n\n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \"candidate.h\"\n#include \"index.h\"\n#include \"mapping_metadata.h\"\n#include \"paired_end_mapping_metadata.h\"\n#include \"sequence_batch.h\"\n#include \"utils.h\"\n\nnamespace chromap {\n\nclass CandidateProcessor {\n public:\n CandidateProcessor() = delete;\n\n CandidateProcessor(int min_num_seeds_required_for_mapping,\n const std::vector max_seed_frequencies)\n : min_num_seeds_required_for_mapping_(min_num_seeds_required_for_mapping),\n max_seed_frequencies_(max_seed_frequencies) {}\n\n ~CandidateProcessor() = default;\n\n void GenerateCandidates(int error_threshold, const Index &index,\n MappingMetadata &mapping_metadata) const;\n\n int SupplementCandidates(\n int error_threshold, uint32_t search_range, const Index &index,\n PairedEndMappingMetadata &paired_end_mapping_metadata) const;\n\n void ReduceCandidatesForPairedEndRead(\n uint32_t mapping_positions_distance,\n PairedEndMappingMetadata &paired_end_mapping_metadata) const;\n\n private:\n void GenerateCandidatesOnOneStrand(int error_threshold,\n int num_seeds_required,\n uint32_t num_minimizers,\n std::vector &hits,\n std::vector &candidates) const;\n\n int GenerateCandidatesFromRepetitiveReadWithMateInfoOnOneStrand(\n const Strand strand, uint32_t search_range, int error_threshold,\n const Index &index, const std::vector &minimizers,\n const std::vector &mate_candidates,\n uint32_t &repetitive_seed_length, std::vector &hits,\n std::vector &candidates) const;\n\n void MergeCandidates(int error_threshold, std::vector &c1,\n std::vector &c2,\n std::vector &buffer) const;\n\n void ReduceCandidatesForPairedEndReadOnOneDirection(\n uint32_t mapping_positions_distance,\n const std::vector &candidates1,\n const std::vector &candidates2,\n std::vector &filtered_candidates1,\n std::vector &filtered_candidates2) const;\n\n const int min_num_seeds_required_for_mapping_;\n // Vector of size 2. The first element is the frequency threshold, and the\n // second element is the frequency threshold to run rescue. The second element\n // should always larger than the first one.\n // TODO(Haowen): add an error check.\n const std::vector max_seed_frequencies_;\n};\n\n} // namespace chromap\n\n#endif // CANDIDATE_PROCESSOR_H_\n"
},
{
"path": "src/chromap.cc",
"content": "#include \"chromap.h\"\n\n#include \n#include \n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\nnamespace chromap {\n\nvoid Chromap::ConstructIndex() {\n // TODO(Haowen): Need a faster algorithm\n // Load all sequences in the reference into one batch\n SequenceBatch reference;\n reference.InitializeLoading(index_parameters_.reference_file_path);\n reference.LoadAllSequences();\n const uint32_t num_sequences = reference.GetNumSequences();\n Index index(index_parameters_);\n index.Construct(num_sequences, reference);\n index.Statistics(num_sequences, reference);\n index.Save();\n reference.FinalizeLoading();\n}\n\nuint32_t Chromap::LoadSingleEndReadsWithBarcodes(SequenceBatch &read_batch,\n SequenceBatch &barcode_batch,\n bool parallel_parsing) {\n //double real_start_time = GetRealTime();\n uint32_t num_loaded_reads = 0;\n\n if (!parallel_parsing || mapping_parameters_.is_bulk_data) {\n while (num_loaded_reads < read_batch_size_) {\n bool no_more_read = read_batch.LoadOneSequenceAndSaveAt(num_loaded_reads);\n bool no_more_barcode = no_more_read;\n if (!mapping_parameters_.is_bulk_data) {\n no_more_barcode =\n barcode_batch.LoadOneSequenceAndSaveAt(num_loaded_reads);\n }\n\n if (no_more_read && no_more_barcode) {\n break;\n } else if (no_more_read || no_more_barcode){\n ExitWithMessage(\"Numbers of reads and barcodes don't match!\");\n }\n ++num_loaded_reads;\n }\n } else {\n uint32_t num_loaded_barcode = 0 ;\n \n#pragma omp task shared(num_loaded_reads, read_batch)\n {\n uint32_t i = 0 ;\n for (i = 0 ; i < read_batch_size_; ++i) {\n if (read_batch.LoadOneSequenceAndSaveAt(i) == true) { // true: no more read\n break ;\n }\n }\n num_loaded_reads = i ;\n }\n\n#pragma omp task shared(num_loaded_barcode, barcode_batch)\n { // bulk data will go to the other big branch\n uint32_t i = 0 ;\n for (i = 0 ; i < read_batch_size_; ++i) {\n if (barcode_batch.LoadOneSequenceAndSaveAt(i) == true) { // true: no more read\n break ;\n }\n }\n num_loaded_barcode = i ;\n }\n\n#pragma omp taskwait\n\n if (num_loaded_reads != num_loaded_barcode) {\n ExitWithMessage(\"Numbers of reads and barcodes don't match!\");\n }\n }\n /*if (num_loaded_reads > 0) {\n std::cerr << \"Loaded \" << num_loaded_reads << \" reads in \"\n << GetRealTime() - real_start_time << \"s.\\n\";\n } else {\n std::cerr << \"No more reads.\\n\";\n }*/\n return num_loaded_reads;\n}\n\nuint32_t Chromap::LoadPairedEndReadsWithBarcodes(SequenceBatch &read_batch1,\n SequenceBatch &read_batch2,\n SequenceBatch &barcode_batch,\n bool parallel_parsing) {\n // double real_start_time = Chromap<>::GetRealTime();\n uint32_t num_loaded_pairs = 0;\n \n if (!parallel_parsing) {\n while (num_loaded_pairs < read_batch_size_) {\n bool no_more_read1 = read_batch1.LoadOneSequenceAndSaveAt(num_loaded_pairs);\n bool no_more_read2 = read_batch2.LoadOneSequenceAndSaveAt(num_loaded_pairs);\n bool no_more_barcode = no_more_read2;\n if (!mapping_parameters_.is_bulk_data) {\n no_more_barcode =\n barcode_batch.LoadOneSequenceAndSaveAt(num_loaded_pairs);\n }\n \n if (no_more_read1 && no_more_read2 && no_more_barcode) {\n break;\n } else if (no_more_read1 || no_more_read2 || no_more_barcode){\n ExitWithMessage(\"Numbers of reads and barcodes don't match!\");\n }\n ++num_loaded_pairs;\n }\n } else {\n uint32_t num_loaded_read1 = 0;\n uint32_t num_loaded_read2 = 0;\n uint32_t num_loaded_barcode = 0;\n \n#pragma omp task shared(num_loaded_read1, read_batch1)\n {\n uint32_t i = 0 ;\n for (i = 0 ; i < read_batch_size_; ++i) {\n if (read_batch1.LoadOneSequenceAndSaveAt(i) == true) { // true: no more read\n break ;\n }\n }\n num_loaded_read1 = i ;\n }\n\n#pragma omp task shared(num_loaded_read2, read_batch2)\n {\n uint32_t i = 0 ;\n for (i = 0 ; i < read_batch_size_; ++i) {\n if (read_batch2.LoadOneSequenceAndSaveAt(i) == true) { // true: no more read\n break ;\n }\n }\n num_loaded_read2 = i ;\n }\n\n#pragma omp task shared(num_loaded_barcode, barcode_batch)\n {\n if (!mapping_parameters_.is_bulk_data) {\n uint32_t i = 0 ;\n for (i = 0 ; i < read_batch_size_; ++i) {\n if (barcode_batch.LoadOneSequenceAndSaveAt(i) == true) { // true: no more read\n break ;\n }\n }\n num_loaded_barcode = i ;\n }\n }\n\n#pragma omp taskwait\n if (mapping_parameters_.is_bulk_data) {\n num_loaded_barcode = num_loaded_read2;\n }\n if (num_loaded_read1 != num_loaded_read2 || num_loaded_read2 != num_loaded_barcode) {\n ExitWithMessage(\"Numbers of reads and barcodes don't match!\");\n }\n\n num_loaded_pairs = num_loaded_read1 ;\n }\n // if (num_loaded_pairs > 0) {\n // std::cerr << \"Loaded \" << num_loaded_pairs << \" pairs in \"<<\n // Chromap<>::GetRealTime() - real_start_time << \"s. \";\n //} else {\n // std::cerr << \"No more reads.\\n\";\n //}\n return num_loaded_pairs;\n}\n\nvoid Chromap::TrimAdapterForPairedEndRead(uint32_t pair_index,\n SequenceBatch &read_batch1,\n SequenceBatch &read_batch2) {\n const uint32_t raw_read1_length = read_batch1.GetSequenceLengthAt(pair_index);\n const uint32_t raw_read2_length = read_batch2.GetSequenceLengthAt(pair_index);\n const char *raw_read1 = read_batch1.GetSequenceAt(pair_index);\n const char *raw_read2 = read_batch2.GetSequenceAt(pair_index);\n const std::string &raw_negative_read1 =\n read_batch1.GetNegativeSequenceAt(pair_index);\n const std::string &raw_negative_read2 =\n read_batch2.GetNegativeSequenceAt(pair_index);\n\n // In the actual adaptor trimming, we assuem length(read1)<=length(read2). So\n // we can have the case that read1 is a subset of read2.\n const char *read1 =\n raw_read1_length <= raw_read2_length ? raw_read1 : raw_read2;\n const std::string &negative_read2 = raw_read1_length <= raw_read2_length\n ? raw_negative_read2\n : raw_negative_read1;\n const uint32_t read1_length = raw_read1_length <= raw_read2_length\n ? raw_read1_length\n : raw_read2_length;\n const uint32_t read2_length = raw_read1_length <= raw_read2_length\n ? raw_read2_length\n : raw_read1_length;\n\n const int min_overlap_length = mapping_parameters_.min_read_length;\n const int seed_length = min_overlap_length / 2;\n const int error_threshold_for_merging = 1;\n bool is_merged = false;\n\n for (int si = 0; si < error_threshold_for_merging + 1; ++si) {\n size_t seed_start_position =\n negative_read2.find(read1 + si * seed_length, 0, seed_length);\n\n while (seed_start_position != std::string::npos) {\n const bool before_seed_is_enough_long =\n seed_start_position >= (size_t)(si * seed_length);\n const bool overlap_is_enough_long =\n (int)(read2_length - seed_start_position + seed_length * si) >=\n min_overlap_length;\n\n if (!before_seed_is_enough_long || !overlap_is_enough_long) {\n seed_start_position = negative_read2.find(\n read1 + si * seed_length, seed_start_position + 1, seed_length);\n continue;\n }\n\n bool can_merge = true;\n int num_errors = 0;\n\n // The bases before the seed.\n for (int i = 0; i < seed_length * si; ++i) {\n if (negative_read2[seed_start_position - si * seed_length + i] !=\n read1[i]) {\n ++num_errors;\n }\n if (num_errors > error_threshold_for_merging) {\n can_merge = false;\n break;\n }\n }\n\n // The bases after the seed.\n for (uint32_t i = seed_length; i + seed_start_position < read2_length &&\n si * seed_length + i < read1_length;\n ++i) {\n if (negative_read2[seed_start_position + i] !=\n read1[si * seed_length + i]) {\n ++num_errors;\n }\n if (num_errors > error_threshold_for_merging) {\n can_merge = false;\n break;\n }\n }\n\n if (can_merge) {\n // Trim adapters and TODO: fix sequencing errors\n int overlap_length =\n read2_length - seed_start_position + si * seed_length;\n int read2_offset = 0;\n // The case that read1 is strictly contained in read2. overlap_length is\n // inferred from the longer read2, which could be longer than read1. In\n // that case, we don't trim read1 (make overlap length equal to read1\n // length) and trim read2 as the original plan.\n if (overlap_length > (int)read1_length) {\n read2_offset = overlap_length - read1_length;\n overlap_length = read1_length;\n }\n\n if (raw_read1_length <= raw_read2_length) {\n read_batch1.TrimSequenceAt(pair_index, overlap_length);\n read_batch2.TrimSequenceAt(pair_index, overlap_length + read2_offset);\n } else {\n read_batch1.TrimSequenceAt(pair_index, overlap_length + read2_offset);\n read_batch2.TrimSequenceAt(pair_index, overlap_length);\n }\n\n is_merged = true;\n // std::cerr << \"Trimed! overlap length: \" << overlap_length << \", \" <<\n // read1.GetLength() << \" \" << read2.GetLength() << \"\\n\";\n break;\n }\n\n seed_start_position = negative_read2.find(\n read1 + si * seed_length, seed_start_position + 1, seed_length);\n }\n\n if (is_merged) {\n break;\n }\n }\n}\n\nbool Chromap::PairedEndReadWithBarcodeIsDuplicate(\n uint32_t pair_index, const SequenceBatch &barcode_batch,\n const SequenceBatch &read_batch1, const SequenceBatch &read_batch2) {\n int dedupe_seed_length = 16;\n uint32_t barcode_length = barcode_batch.GetSequenceLengthAt(pair_index);\n uint64_t barcode_key =\n barcode_batch.GenerateSeedFromSequenceAt(pair_index, 0, barcode_length);\n uint64_t read1_seed1 =\n read_batch1.GenerateSeedFromSequenceAt(pair_index, 0, dedupe_seed_length);\n uint64_t read2_seed1 =\n read_batch2.GenerateSeedFromSequenceAt(pair_index, 0, dedupe_seed_length);\n uint64_t read_seed_key =\n (read1_seed1 << (dedupe_seed_length * 2)) | read2_seed1;\n uint64_t read1_seed2 = read_batch1.GenerateSeedFromSequenceAt(\n pair_index, dedupe_seed_length, dedupe_seed_length * 2);\n uint64_t read2_seed2 = read_batch2.GenerateSeedFromSequenceAt(\n pair_index, dedupe_seed_length, dedupe_seed_length * 2);\n khiter_t barcode_table_iterator =\n kh_get(k64_seq, barcode_lookup_table_, barcode_key);\n if (barcode_table_iterator != kh_end(barcode_lookup_table_)) {\n uint32_t read_lookup_table_index =\n kh_value(barcode_lookup_table_, barcode_table_iterator);\n // std::cerr << \"Have barcode, try to check read. \" <<\n // read_lookup_table_index << \"\\n\";\n khash_t(k128) *read_lookup_table =\n read_lookup_tables_[read_lookup_table_index];\n khiter_t read_lookup_table_iterator =\n kh_get(k128, read_lookup_table, read_seed_key);\n if (read_lookup_table_iterator != kh_end(read_lookup_table)) {\n // std::cerr << \"Have barcode, have read, try whether match.\\n\";\n uint128_t read_seeds =\n kh_value(read_lookup_table, read_lookup_table_iterator);\n if (read_seeds.first == read1_seed2 && read_seeds.second == read2_seed2) {\n // std::cerr << \"Have barcode, have read, and match.\\n\";\n return true;\n } else {\n // std::cerr << \"Have barcode, have read, but don't match.\\n\";\n return false;\n }\n } else {\n // std::cerr << \"Have barcode, no read.\\n\";\n uint128_t read_seeds = {.first = read1_seed2, .second = read2_seed2};\n int khash_return_code;\n khiter_t read_lookup_table_insert_iterator =\n kh_put(k128, read_lookup_table, read_seed_key, &khash_return_code);\n assert(khash_return_code != -1 && khash_return_code != 0);\n kh_value(read_lookup_table, read_lookup_table_insert_iterator) =\n read_seeds;\n // std::cerr << \"Have barcode, no read.\\n\";\n return false;\n }\n } else {\n // insert the barcode and append a new read hash table to tables and then\n // insert the reads\n // std::cerr << \"No barcode, no read.\\n\";\n int khash_return_code;\n khiter_t barcode_table_insert_iterator =\n kh_put(k64_seq, barcode_lookup_table_, barcode_key, &khash_return_code);\n assert(khash_return_code != -1 && khash_return_code != 0);\n kh_value(barcode_lookup_table_, barcode_table_insert_iterator) =\n read_lookup_tables_.size();\n khash_t(k128) *read_lookup_table = kh_init(k128);\n khiter_t read_lookup_table_iterator =\n kh_put(k128, read_lookup_table, read_seed_key, &khash_return_code);\n assert(khash_return_code != -1 && khash_return_code != 0);\n uint128_t read_seeds = {.first = read1_seed2, .second = read2_seed2};\n kh_value(read_lookup_table, read_lookup_table_iterator) = read_seeds;\n read_lookup_tables_.push_back(read_lookup_table);\n // std::cerr << \"No barcode, no read.\\n\";\n return false;\n }\n}\n\nuint32_t Chromap::SampleInputBarcodesAndExamineLength() {\n if (mapping_parameters_.is_bulk_data) {\n return 0;\n }\n\n uint32_t sample_batch_size = 1000;\n SequenceBatch barcode_batch(sample_batch_size, barcode_effective_range_);\n\n barcode_batch.InitializeLoading(mapping_parameters_.barcode_file_paths[0]);\n\n uint32_t num_loaded_barcodes = barcode_batch.LoadBatch();\n\n uint32_t cell_barcode_length = barcode_batch.GetSequenceLengthAt(0);\n for (uint32_t i = 1; i < num_loaded_barcodes; ++i) {\n if (barcode_batch.GetSequenceLengthAt(i) != cell_barcode_length) {\n ExitWithMessage(\"ERROR: barcode lengths are not equal in the sample!\");\n }\n }\n\n barcode_batch.FinalizeLoading();\n\n return cell_barcode_length;\n}\n\nvoid Chromap::LoadBarcodeWhitelist() {\n double real_start_time = GetRealTime();\n int num_barcodes = 0;\n\n if (1) {\n gzFile barcode_whitelist_file = \n gzopen(mapping_parameters_.barcode_whitelist_file_path.c_str(), \"r\"); \n const uint32_t barcode_buffer_size = 256;\n char barcode[barcode_buffer_size];\n while (gzgets(barcode_whitelist_file, barcode, barcode_buffer_size) != NULL) {\n size_t barcode_length = strlen(barcode);\n if (barcode[barcode_length - 1] == '\\n') {\n barcode[barcode_length - 1] = '\\0';\n --barcode_length;\n }\n if (barcode_length > 32) {\n ExitWithMessage(\"ERROR: barcode length is greater than 32!\");\n }\n\n if (barcode_length != barcode_length_) {\n if (num_barcodes == 0) {\n ExitWithMessage(\n \"ERROR: whitelist and input barcode lengths are not equal!\");\n } else {\n ExitWithMessage(\n \"ERROR: barcode lengths are not equal in the whitelist!\");\n }\n }\n \n uint64_t barcode_key = GenerateSeedFromSequence(\n barcode, barcode_length, 0, barcode_length);\n \n int khash_return_code;\n khiter_t barcode_whitelist_lookup_table_iterator =\n kh_put(k64_seq, barcode_whitelist_lookup_table_, barcode_key,\n &khash_return_code);\n kh_value(barcode_whitelist_lookup_table_,\n barcode_whitelist_lookup_table_iterator) = 0;\n assert(khash_return_code != -1 && khash_return_code != 0);\n ++num_barcodes;\n }\n if (!gzeof(barcode_whitelist_file)) {\n ExitWithMessage(\"ERROR: barcode whitelist file does not exist or is truncated!\");\n }\n gzclose(barcode_whitelist_file);\n } else { \n std::ifstream barcode_whitelist_file_stream(\n mapping_parameters_.barcode_whitelist_file_path);\n std::string barcode_whitelist_file_line;\n // bool first_line = true;\n while (getline(barcode_whitelist_file_stream, barcode_whitelist_file_line)) {\n std::stringstream barcode_whitelist_file_line_string_stream(\n barcode_whitelist_file_line);\n //// skip the header\n // if (barcode_whitelist_file_line[0] == '#' ||\n // barcode_whitelist_file_line.find(\"kmer\") == 0) {\n // continue;\n //}\n std::string barcode;\n barcode_whitelist_file_line_string_stream >> barcode;\n size_t barcode_length = barcode.length();\n if (barcode_length > 32) {\n ExitWithMessage(\"ERROR: barcode length is greater than 32!\");\n }\n\n if (barcode_length != barcode_length_) {\n if (num_barcodes == 0) {\n ExitWithMessage(\n \"ERROR: whitelist and input barcode lengths are not equal!\");\n } else {\n ExitWithMessage(\n \"ERROR: barcode lengths are not equal in the whitelist!\");\n }\n }\n\n // if (first_line) {\n // //size_t barcode_length = kmer.length();\n // // Allocate memory to save pore model parameters\n // //size_t num_pore_models = 1 << (kmer_size_ * 2);\n // //pore_models_.assign(num_pore_models, PoreModelParameters());\n // //first_line = false;\n //}\n // assert(kmer.length() == (size_t)kmer_size_);\n uint64_t barcode_key = GenerateSeedFromSequence(\n barcode.data(), barcode_length, 0, barcode_length);\n // PoreModelParameters &pore_model_parameters =\n // pore_models_[kmer_hash_value]; barcode_whitelist_file_line_string_stream\n // >> pore_model_parameters.level_mean >> pore_model_parameters.level_stdv\n // >> pore_model_parameters.sd_mean >> pore_model_parameters.sd_stdv;\n int khash_return_code;\n khiter_t barcode_whitelist_lookup_table_iterator =\n kh_put(k64_seq, barcode_whitelist_lookup_table_, barcode_key,\n &khash_return_code);\n kh_value(barcode_whitelist_lookup_table_,\n barcode_whitelist_lookup_table_iterator) = 0;\n assert(khash_return_code != -1 && khash_return_code != 0);\n ++num_barcodes;\n }\n barcode_whitelist_file_stream.close();\n }\n std::cerr << \"Loaded \" << num_barcodes << \" barcodes in \"\n << GetRealTime() - real_start_time << \"s.\\n\";\n}\n\nvoid Chromap::ComputeBarcodeAbundance(uint64_t max_num_sample_barcodes) {\n double real_start_time = GetRealTime();\n SequenceBatch barcode_batch(read_batch_size_, barcode_effective_range_);\n for (size_t read_file_index = 0;\n read_file_index < mapping_parameters_.read_file1_paths.size();\n ++read_file_index) {\n barcode_batch.InitializeLoading(\n mapping_parameters_.barcode_file_paths[read_file_index]);\n uint32_t num_loaded_barcodes = barcode_batch.LoadBatch();\n while (num_loaded_barcodes > 0) {\n for (uint32_t barcode_index = 0; barcode_index < num_loaded_barcodes;\n ++barcode_index) {\n std::vector N_pos; // position of Ns\n barcode_batch.GetSequenceNsAt(barcode_index, /*little_endian=*/true,\n N_pos);\n if (N_pos.size() > 0) continue;\n\n uint32_t barcode_length =\n barcode_batch.GetSequenceLengthAt(barcode_index);\n uint64_t barcode_key = barcode_batch.GenerateSeedFromSequenceAt(\n barcode_index, 0, barcode_length);\n khiter_t barcode_whitelist_lookup_table_iterator =\n kh_get(k64_seq, barcode_whitelist_lookup_table_, barcode_key);\n if (barcode_whitelist_lookup_table_iterator !=\n kh_end(barcode_whitelist_lookup_table_)) {\n // Correct barcode\n kh_value(barcode_whitelist_lookup_table_,\n barcode_whitelist_lookup_table_iterator) += 1;\n ++num_sample_barcodes_;\n }\n }\n if (!mapping_parameters_.skip_barcode_check &&\n num_sample_barcodes_ * 20 < num_loaded_barcodes) {\n // Since num_loaded_pairs is a constant, this if is actuaclly only\n // effective in the first iteration\n ExitWithMessage(\n \"Less than 5\\% barcodes can be found or corrected based on the \"\n \"barcode whitelist.\\nPlease check whether the barcode whitelist \"\n \"matches the data, e.g. length, reverse-complement. If this is a \"\n \"false warning, please run Chromap with the option \"\n \"--skip-barcode-check.\");\n }\n\n if (num_sample_barcodes_ >= max_num_sample_barcodes) {\n break;\n }\n num_loaded_barcodes = barcode_batch.LoadBatch();\n }\n barcode_batch.FinalizeLoading();\n if (num_sample_barcodes_ >= max_num_sample_barcodes) {\n break;\n }\n }\n\n std::cerr << \"Compute barcode abundance using \" << num_sample_barcodes_\n << \" in \" << GetRealTime() - real_start_time << \"s.\\n\";\n}\n\nvoid Chromap::UpdateBarcodeAbundance(uint32_t num_loaded_barcodes,\n const SequenceBatch &barcode_batch) {\n double real_start_time = GetRealTime();\n for (uint32_t barcode_index = 0; barcode_index < num_loaded_barcodes;\n ++barcode_index) {\n uint32_t barcode_length = barcode_batch.GetSequenceLengthAt(barcode_index);\n uint64_t barcode_key = barcode_batch.GenerateSeedFromSequenceAt(\n barcode_index, 0, barcode_length);\n khiter_t barcode_whitelist_lookup_table_iterator =\n kh_get(k64_seq, barcode_whitelist_lookup_table_, barcode_key);\n if (barcode_whitelist_lookup_table_iterator !=\n kh_end(barcode_whitelist_lookup_table_)) {\n // Correct barcode\n kh_value(barcode_whitelist_lookup_table_,\n barcode_whitelist_lookup_table_iterator) += 1;\n ++num_sample_barcodes_;\n }\n }\n std::cerr << \"Update barcode abundance using \" << num_sample_barcodes_\n << \" in \" << GetRealTime() - real_start_time << \"s.\\n\";\n}\n\nbool Chromap::CorrectBarcodeAt(uint32_t barcode_index,\n SequenceBatch &barcode_batch,\n uint64_t &num_barcode_in_whitelist,\n uint64_t &num_corrected_barcode) {\n const uint32_t barcode_length =\n barcode_batch.GetSequenceLengthAt(barcode_index);\n const uint64_t barcode_key = barcode_batch.GenerateSeedFromSequenceAt(\n barcode_index, 0, barcode_length);\n khiter_t barcode_whitelist_lookup_table_iterator =\n kh_get(k64_seq, barcode_whitelist_lookup_table_, barcode_key);\n std::vector N_pos; // position of Ns\n\n barcode_batch.GetSequenceNsAt(barcode_index, /*little_endian=*/true, N_pos);\n if (N_pos.size() >\n (uint32_t)mapping_parameters_.barcode_correction_error_threshold)\n return false;\n\n if (N_pos.size() == 0 && barcode_whitelist_lookup_table_iterator !=\n kh_end(barcode_whitelist_lookup_table_)) {\n // Correct barcode\n ++num_barcode_in_whitelist;\n return true;\n } else if (mapping_parameters_.barcode_correction_error_threshold > 0) {\n // Need to correct this barcode\n // const char *barcode = barcode_batch->GetSequenceAt(barcode_index);\n // std::cerr << barcode_index << \" barcode \" << barcode << \" needs\n // correction\\n\";\n const char *barcode_qual = barcode_batch.GetSequenceQualAt(barcode_index);\n std::vector corrected_barcodes_with_quals;\n uint64_t mask = (uint64_t)3;\n uint32_t i_start = 0;\n uint32_t i_end = barcode_length;\n uint32_t ti_limit = 3;\n if (N_pos.size() > 0) {\n i_start = N_pos[0];\n i_end = N_pos[0] + 1;\n ti_limit = 4;\n }\n for (uint32_t i = i_start; i < i_end; ++i) {\n uint64_t barcode_key_to_change = mask << (2 * i);\n barcode_key_to_change = ~barcode_key_to_change;\n barcode_key_to_change &= barcode_key;\n uint64_t base_to_change1 = (barcode_key >> (2 * i)) & mask;\n for (uint32_t ti = 0; ti < ti_limit; ++ti) {\n // change the base\n base_to_change1 += 1;\n base_to_change1 &= mask;\n // generate the corrected key\n uint64_t corrected_barcode_key =\n barcode_key_to_change | (base_to_change1 << (2 * i));\n barcode_whitelist_lookup_table_iterator = kh_get(\n k64_seq, barcode_whitelist_lookup_table_, corrected_barcode_key);\n if (barcode_whitelist_lookup_table_iterator !=\n kh_end(barcode_whitelist_lookup_table_)) {\n // find one possible corrected barcode\n double barcode_abundance =\n kh_value(barcode_whitelist_lookup_table_,\n barcode_whitelist_lookup_table_iterator) /\n (double)num_sample_barcodes_;\n int qual_offset = 33;\n int adjusted_qual =\n barcode_qual[barcode_length - 1 - i] - qual_offset;\n adjusted_qual = adjusted_qual > 40 ? 40 : adjusted_qual;\n adjusted_qual = adjusted_qual < 3 ? 3 : adjusted_qual;\n double score =\n pow(10.0, ((-adjusted_qual) / 10.0)) * barcode_abundance;\n corrected_barcodes_with_quals.emplace_back(\n BarcodeWithQual{barcode_length - 1 - i,\n Uint8ToChar(base_to_change1), 0, 0, score});\n // std::cerr << \"1score: \" << score << \" pos1: \" << barcode_length - 1\n // - i << \" b1: \" << base_to_change1 << \" pos2: \" << 0 << \" b2: \" <<\n // (char)0 << \"\\n\";\n }\n if (mapping_parameters_.barcode_correction_error_threshold == 2) {\n uint32_t j_start = i + 1;\n uint32_t j_end = barcode_length;\n uint32_t ti2_limit = 3;\n if (N_pos.size() == 2) {\n j_start = N_pos[1];\n j_end = N_pos[1] + 1;\n ti2_limit = 4;\n }\n for (uint32_t j = j_start; j < j_end; ++j) {\n uint64_t barcode_key_to_change2 = mask << (2 * i);\n barcode_key_to_change2 = mask << (2 * j);\n barcode_key_to_change2 = ~barcode_key_to_change2;\n barcode_key_to_change2 &= corrected_barcode_key;\n uint64_t base_to_change2 =\n (corrected_barcode_key >> (2 * j)) & mask;\n for (uint32_t ti2 = 0; ti2 < ti2_limit; ++ti2) {\n // change the base\n base_to_change2 += 1;\n base_to_change2 &= mask;\n // generate the corrected key\n uint64_t corrected_barcode_key2 =\n barcode_key_to_change2 | (base_to_change2 << (2 * j));\n barcode_whitelist_lookup_table_iterator =\n kh_get(k64_seq, barcode_whitelist_lookup_table_,\n corrected_barcode_key2);\n if (barcode_whitelist_lookup_table_iterator !=\n kh_end(barcode_whitelist_lookup_table_)) {\n // find one possible corrected barcode\n double barcode_abundance =\n kh_value(barcode_whitelist_lookup_table_,\n barcode_whitelist_lookup_table_iterator) /\n (double)num_sample_barcodes_;\n int qual_offset = 33;\n int adjusted_qual =\n barcode_qual[barcode_length - 1 - j] - qual_offset;\n adjusted_qual = adjusted_qual > 40 ? 40 : adjusted_qual;\n adjusted_qual = adjusted_qual < 3 ? 3 : adjusted_qual;\n int adjusted_qual1 =\n barcode_qual[barcode_length - 1 - i] - qual_offset;\n adjusted_qual1 = adjusted_qual1 > 40 ? 40 : adjusted_qual1;\n adjusted_qual1 = adjusted_qual1 < 3 ? 3 : adjusted_qual1;\n adjusted_qual += adjusted_qual1;\n double score =\n pow(10.0, ((-adjusted_qual) / 10.0)) * barcode_abundance;\n corrected_barcodes_with_quals.emplace_back(BarcodeWithQual{\n barcode_length - 1 - i, Uint8ToChar(base_to_change1),\n barcode_length - 1 - j, Uint8ToChar(base_to_change2),\n score});\n // std::cerr << \"2score: \" << score << \" pos1: \" <<\n // barcode_length - 1 - i << \" b1: \" << base_to_change1 << \"\n // pos2: \" << barcode_length - 1 -j << \" b2: \" <<\n // base_to_change2\n // << \"\\n\";\n }\n }\n }\n }\n }\n }\n size_t num_possible_corrected_barcodes =\n corrected_barcodes_with_quals.size();\n if (num_possible_corrected_barcodes == 0) {\n // Barcode cannot be corrected, leave it for downstream\n return false;\n } else if (num_possible_corrected_barcodes == 1) {\n // Just correct it\n // std::cerr << \"Corrected the barcode from \" << barcode << \" to \";\n barcode_batch.CorrectBaseAt(\n barcode_index, corrected_barcodes_with_quals[0].corrected_base_index1,\n corrected_barcodes_with_quals[0].correct_base1);\n if (corrected_barcodes_with_quals[0].correct_base2 != 0) {\n barcode_batch.CorrectBaseAt(\n barcode_index,\n corrected_barcodes_with_quals[0].corrected_base_index2,\n corrected_barcodes_with_quals[0].correct_base2);\n }\n // std::cerr << barcode << \"\\n\";\n // std::cerr << \"score: \" << corrected_barcodes_with_quals[0].score <<\n // \"\\n\"; std::cerr << \"score: \" << corrected_barcodes_with_quals[0].score\n // << \" pos1: \" << corrected_barcodes_with_quals[0].corrected_base_index1\n // << \" b1: \" << corrected_barcodes_with_quals[0].correct_base1 << \" pos2:\n // \" << corrected_barcodes_with_quals[0].corrected_base_index2 << \" b2: \"\n // << corrected_barcodes_with_quals[0].correct_base2 << \"\\n\";\n ++num_corrected_barcode;\n return true;\n } else {\n // Select the best correction\n std::sort(corrected_barcodes_with_quals.begin(),\n corrected_barcodes_with_quals.end(),\n std::greater());\n // int num_ties = 0;\n double sum_score = 0;\n for (size_t ci = 0; ci < num_possible_corrected_barcodes; ++ci) {\n sum_score += corrected_barcodes_with_quals[ci].score;\n // std::cerr << ci << \" score: \" <<\n // corrected_barcodes_with_quals[ci].score << \" pos1: \" <<\n // corrected_barcodes_with_quals[ci].corrected_base_index1 << \" b1: \" <<\n // corrected_barcodes_with_quals[ci].correct_base1 << \" pos2: \" <<\n // corrected_barcodes_with_quals[ci].corrected_base_index2 << \" b2: \" <<\n // corrected_barcodes_with_quals[ci].correct_base2 << \"\\n\"; if\n // (corrected_barcodes_with_quals[ci].qual ==\n // corrected_barcodes_with_quals[0].qual) {\n // ++num_ties;\n //}\n }\n int best_corrected_barcode_index = 0;\n // if (num_ties > 0) {\n // std::mt19937 tmp_generator(11);\n // std::uniform_int_distribution distribution(0, num_ties); //\n // important: inclusive range best_corrected_barcode_index =\n // distribution(tmp_generator);\n //}\n // std::cerr << \"Corrected the barcode from \" << barcode << \" to \";\n double confidence_threshold =\n mapping_parameters_.barcode_correction_probability_threshold;\n if (corrected_barcodes_with_quals[best_corrected_barcode_index].score /\n sum_score >\n confidence_threshold) {\n barcode_batch.CorrectBaseAt(\n barcode_index,\n corrected_barcodes_with_quals[best_corrected_barcode_index]\n .corrected_base_index1,\n corrected_barcodes_with_quals[best_corrected_barcode_index]\n .correct_base1);\n if (corrected_barcodes_with_quals[best_corrected_barcode_index]\n .correct_base2 != 0) {\n barcode_batch.CorrectBaseAt(\n barcode_index,\n corrected_barcodes_with_quals[best_corrected_barcode_index]\n .corrected_base_index2,\n corrected_barcodes_with_quals[best_corrected_barcode_index]\n .correct_base2);\n }\n // std::cerr << barcode << \"\\n\";\n // std::cerr << \"score: \" <<\n // corrected_barcodes_with_quals[best_corrected_barcode_index].score <<\n // \"\\n\"; std::cerr << \"best score: \" <<\n // corrected_barcodes_with_quals[best_corrected_barcode_index].score <<\n // \" sum score: \" << sum_score << \"\\n\";\n ++num_corrected_barcode;\n return true;\n } else {\n // std::cerr << \"Didnt pass filter: \" <<\n // corrected_barcodes_with_quals[best_corrected_barcode_index].score /\n // sum_score << \"\\n\"; std::cerr << \"best score: \" <<\n // corrected_barcodes_with_quals[best_corrected_barcode_index].score <<\n // \" sum score: \" << sum_score << \"\\n\";\n return false;\n }\n }\n } else {\n return false;\n }\n}\n\nvoid Chromap::OutputBarcodeStatistics() {\n std::cerr << \"Number of barcodes in whitelist: \" << num_barcode_in_whitelist_\n << \".\\n\";\n std::cerr << \"Number of corrected barcodes: \" << num_corrected_barcode_\n << \".\\n\";\n}\n\nvoid Chromap::OutputMappingStatistics() {\n std::cerr << \"Number of reads: \" << num_reads_ << \".\\n\";\n // std::cerr << \"Number of duplicated reads: \" << num_duplicated_reads_ <<\n // \".\\n\";\n std::cerr << \"Number of mapped reads: \" << num_mapped_reads_ << \".\\n\";\n std::cerr << \"Number of uniquely mapped reads: \" << num_uniquely_mapped_reads_\n << \".\\n\";\n std::cerr << \"Number of reads have multi-mappings: \"\n << num_mapped_reads_ - num_uniquely_mapped_reads_ << \".\\n\";\n std::cerr << \"Number of candidates: \" << num_candidates_ << \".\\n\";\n std::cerr << \"Number of mappings: \" << num_mappings_ << \".\\n\";\n std::cerr << \"Number of uni-mappings: \" << num_uniquely_mapped_reads_\n << \".\\n\";\n std::cerr << \"Number of multi-mappings: \"\n << num_mappings_ - num_uniquely_mapped_reads_ << \".\\n\";\n}\n\nvoid Chromap::ParseReadFormat(const std::string &read_format) {\n if (read_format.empty()) {\n return;\n }\n\n read1_effective_range_.InitializeParsing();\n read2_effective_range_.InitializeParsing();\n barcode_effective_range_.InitializeParsing();\n\n uint32_t i, j;\n for (i = 0; i < read_format.size();) {\n for (j = i + 1; j < read_format.size() && read_format[j] != ','; ++j)\n ;\n bool parse_success = true;\n if (read_format[i] == 'r' && read_format[i + 1] == '1') {\n parse_success =\n read1_effective_range_.ParseFormatStringAndAppendEffectiveRange(\n read_format.c_str() + i, j - i);\n } else if (read_format[i] == 'r' && read_format[i + 1] == '2') {\n parse_success =\n read2_effective_range_.ParseFormatStringAndAppendEffectiveRange(\n read_format.c_str() + i, j - i);\n } else if (read_format[i] == 'b' && read_format[i + 1] == 'c') {\n parse_success =\n barcode_effective_range_.ParseFormatStringAndAppendEffectiveRange(\n read_format.c_str() + i, j - i);\n } else {\n parse_success = false;\n }\n\n if (!parse_success) {\n ExitWithMessage(\"Unknown read format: \" + read_format + \"\\n\");\n }\n\n i = j + 1;\n }\n\n read1_effective_range_.FinalizeParsing();\n read2_effective_range_.FinalizeParsing();\n barcode_effective_range_.FinalizeParsing();\n}\n\nvoid Chromap::GenerateCustomRidRanks(\n const std::string &custom_rid_order_file_path,\n uint32_t num_reference_sequences, const SequenceBatch &reference,\n std::vector &rid_ranks) {\n for (uint32_t i = 0; i < num_reference_sequences; ++i) {\n rid_ranks.emplace_back(i);\n }\n\n if (custom_rid_order_file_path.empty()) {\n return;\n }\n\n std::unordered_map ref_name_to_rank;\n std::ifstream custom_rid_order_file_stream(custom_rid_order_file_path);\n std::string ref_name;\n uint32_t ref_rank = 0;\n while (getline(custom_rid_order_file_stream, ref_name)) {\n ref_name_to_rank[ref_name] = ref_rank;\n ref_rank += 1;\n }\n custom_rid_order_file_stream.close();\n\n // First, rank the chromosomes in the custom order provided by users.\n for (uint32_t i = 0; i < num_reference_sequences; ++i) {\n std::string ref_name(reference.GetSequenceNameAt(i));\n if (ref_name_to_rank.find(ref_name) != ref_name_to_rank.end()) {\n rid_ranks[i] = ref_name_to_rank[ref_name];\n } else {\n rid_ranks[i] = -1;\n }\n }\n\n // There might be some rids without any custom order. We just order them based\n // on their original order in the reference file.\n uint32_t k = ref_name_to_rank.size();\n // Rank the remaining chromosomes.\n for (uint32_t i = 0; i < num_reference_sequences; ++i) {\n if (rid_ranks[i] == -1) {\n rid_ranks[i] = k;\n ++k;\n }\n }\n\n if (k > num_reference_sequences) {\n ExitWithMessage(\n \"ERROR: unknown chromsome names found in chromosome order file.\");\n }\n}\n\nvoid Chromap::RerankCandidatesRid(std::vector &candidates) {\n for (size_t i = 0; i < candidates.size(); ++i) {\n uint64_t rid = (uint32_t)(candidates[i].position >> 32);\n rid = custom_rid_rank_[rid];\n candidates[i].position =\n (candidates[i].position & (uint64_t)0xffffffff) | (rid << 32);\n }\n}\n\n} // namespace chromap\n"
},
{
"path": "src/chromap.h",
"content": "#ifndef CHROMAP_H_\n#define CHROMAP_H_\n\n#include \n\n#include \n#include \n#include \n#include \n#include \n\n#include // Used these two for k-minhash\n#include \n\n#include // Used for frip est params splitting\n\n#include \"candidate_processor.h\"\n#include \"cxxopts.hpp\"\n#include \"draft_mapping_generator.h\"\n#include \"feature_barcode_matrix.h\"\n#include \"index.h\"\n#include \"index_parameters.h\"\n#include \"khash.h\"\n#include \"mapping_generator.h\"\n#include \"mapping_metadata.h\"\n#include \"mapping_parameters.h\"\n#include \"mapping_processor.h\"\n#include \"mapping_writer.h\"\n#include \"minimizer_generator.h\"\n#include \"mmcache.hpp\"\n#include \"paired_end_mapping_metadata.h\"\n#include \"sequence_batch.h\"\n#include \"sequence_effective_range.h\"\n#include \"temp_mapping.h\"\n#include \"utils.h\"\n\n#define CHROMAP_VERSION \"0.3.3-r521\"\n\nnamespace chromap {\n\nclass K_MinHash {\npublic:\n /*\n * MinHash Class - used to estimate the number of unique cache slots \n * hit by each barcode\n *\n * @param k - size of MinHash sketch\n * @param range - range of possible cache ids\n */\n K_MinHash(size_t k, size_t range) : k_(k), range_(range) {}\n\n inline void add(size_t num) {\n /* If num is not present in queue, we will add it */\n if (unique_slots_.find(num) == unique_slots_.end()) {\n unique_slots_.insert(num);\n pq_.push(num);\n // only keep smallest k numbers\n if (pq_.size() > k_) {\n unique_slots_.erase(pq_.top());\n pq_.pop();\n }\n }\n }\n\n inline size_t compute_cardinality() {\n /* Use k-MinHash estimator to return estimated cardinality */\n if (pq_.size() < k_) {return 0;}\n size_t cardinality = (k_ * range_)/pq_.top() - 1;\n return cardinality;\n }\n\nprivate:\n size_t k_;\n size_t range_;\n\n /* Uses an unordered set to have O(1) find queries*/\n std::priority_queue pq_; // max-heap\n std::unordered_set unique_slots_; // keep track of unique values\n};\n\nclass Chromap {\n public:\n Chromap() = delete;\n\n // For index construction\n Chromap(const IndexParameters &index_parameters)\n : index_parameters_(index_parameters) {\n barcode_lookup_table_ = NULL;\n barcode_whitelist_lookup_table_ = NULL;\n }\n\n // For mapping\n Chromap(const MappingParameters &mapping_parameters)\n : mapping_parameters_(mapping_parameters) {\n barcode_lookup_table_ = kh_init(k64_seq);\n barcode_whitelist_lookup_table_ = kh_init(k64_seq);\n\n ParseReadFormat(mapping_parameters.read_format);\n }\n\n ~Chromap() {\n if (barcode_whitelist_lookup_table_ != NULL) {\n kh_destroy(k64_seq, barcode_whitelist_lookup_table_);\n }\n\n if (barcode_lookup_table_ != NULL) {\n kh_destroy(k64_seq, barcode_lookup_table_);\n }\n if (read_lookup_tables_.size() > 0) {\n for (uint32_t i = 0; i < read_lookup_tables_.size(); ++i) {\n kh_destroy(k128, read_lookup_tables_[i]);\n }\n }\n }\n\n void ConstructIndex();\n\n template \n void MapSingleEndReads();\n\n template \n void MapPairedEndReads();\n\n private:\n uint32_t LoadSingleEndReadsWithBarcodes(SequenceBatch &read_batch,\n SequenceBatch &barcode_batch,\n bool parallel_parsing);\n\n uint32_t LoadPairedEndReadsWithBarcodes(SequenceBatch &read_batch1,\n SequenceBatch &read_batch2,\n SequenceBatch &barcode_batch,\n bool parallel_parsing);\n\n void TrimAdapterForPairedEndRead(uint32_t pair_index,\n SequenceBatch &read_batch1,\n SequenceBatch &read_batch2);\n\n bool PairedEndReadWithBarcodeIsDuplicate(uint32_t pair_index,\n const SequenceBatch &barcode_batch,\n const SequenceBatch &read_batch1,\n const SequenceBatch &read_batch2);\n\n uint32_t SampleInputBarcodesAndExamineLength();\n\n void LoadBarcodeWhitelist();\n\n void ComputeBarcodeAbundance(uint64_t max_num_sample_barcodes);\n\n void UpdateBarcodeAbundance(uint32_t num_loaded_barcodes,\n const SequenceBatch &barcode_batch);\n\n bool CorrectBarcodeAt(uint32_t barcode_index, SequenceBatch &barcode_batch,\n uint64_t &num_barcode_in_whitelist,\n uint64_t &num_corrected_barcode);\n\n void OutputBarcodeStatistics();\n\n void OutputMappingStatistics();\n\n void ParseReadFormat(const std::string &read_format);\n\n // User custom rid order file contains a column of reference sequence names\n // and there is one name on each row. The reference sequence name on the ith\n // row means the rank of this sequence is i. This function loads the custom\n // rid order file and generates a mapping from the original rids to their\n // custom ranks, e.g., rid_ranks[i] is the custom rank of the ith rid in the\n // reference.\n void GenerateCustomRidRanks(const std::string &custom_rid_order_file_path,\n uint32_t num_reference_sequences,\n const SequenceBatch &reference,\n std::vector &rid_ranks);\n\n // TODO: generate reranked candidates directly.\n void RerankCandidatesRid(std::vector &candidates);\n\n // Parameters\n const IndexParameters index_parameters_;\n const MappingParameters mapping_parameters_;\n\n // Default batch size, # reads for single-end reads, # read pairs for\n // paired-end reads.\n const uint32_t read_batch_size_ = 500000;\n\n // 0-start, 1-end (includsive), 2-strand(-1:minus, 1:plus)\n SequenceEffectiveRange barcode_effective_range_;\n SequenceEffectiveRange read1_effective_range_;\n SequenceEffectiveRange read2_effective_range_;\n\n std::vector custom_rid_rank_;\n std::vector pairs_custom_rid_rank_;\n\n khash_t(k64_seq) * barcode_whitelist_lookup_table_;\n\n // For identical read dedupe\n khash_t(k64_seq) * barcode_lookup_table_;\n std::vector read_lookup_tables_;\n\n // For mapping.\n const int min_unique_mapping_mapq_ = 4;\n\n // For mapping stats.\n uint64_t num_candidates_ = 0;\n uint64_t num_mappings_ = 0;\n uint64_t num_mapped_reads_ = 0;\n uint64_t num_uniquely_mapped_reads_ = 0;\n uint64_t num_reads_ = 0;\n // # identical reads.\n // uint64_t num_duplicated_reads_ = 0;\n\n // For barcode stats.\n const uint64_t initial_num_sample_barcodes_ = 20000000;\n uint64_t num_sample_barcodes_ = 0;\n uint64_t num_barcode_in_whitelist_ = 0;\n uint64_t num_corrected_barcode_ = 0;\n uint32_t barcode_length_ = 0;\n};\n\ntemplate \nvoid Chromap::MapSingleEndReads() {\n double real_start_time = GetRealTime();\n\n SequenceBatch reference;\n reference.InitializeLoading(mapping_parameters_.reference_file_path);\n reference.LoadAllSequences();\n uint32_t num_reference_sequences = reference.GetNumSequences();\n if (mapping_parameters_.custom_rid_order_file_path.length() > 0) {\n GenerateCustomRidRanks(mapping_parameters_.custom_rid_order_file_path,\n num_reference_sequences, reference,\n custom_rid_rank_);\n reference.ReorderSequences(custom_rid_rank_);\n }\n\n Index index(mapping_parameters_.index_file_path);\n index.Load();\n const int kmer_size = index.GetKmerSize();\n const int window_size = index.GetWindowSize();\n // index.Statistics(num_sequences, reference);\n\n SequenceBatch read_batch(read_batch_size_, read1_effective_range_);\n SequenceBatch read_batch_for_loading(read_batch_size_,\n read1_effective_range_);\n SequenceBatch barcode_batch(read_batch_size_, barcode_effective_range_);\n SequenceBatch barcode_batch_for_loading(read_batch_size_,\n barcode_effective_range_);\n\n std::vector> mappings_on_diff_ref_seqs;\n mappings_on_diff_ref_seqs.reserve(num_reference_sequences);\n for (uint32_t i = 0; i < num_reference_sequences; ++i) {\n mappings_on_diff_ref_seqs.emplace_back(std::vector());\n }\n\n std::vector> temp_mapping_file_handles;\n\n // Preprocess barcodes for single cell data\n if (!mapping_parameters_.is_bulk_data) {\n barcode_length_ = SampleInputBarcodesAndExamineLength();\n if (!mapping_parameters_.barcode_whitelist_file_path.empty()) {\n LoadBarcodeWhitelist();\n ComputeBarcodeAbundance(initial_num_sample_barcodes_);\n }\n }\n\n MinimizerGenerator minimizer_generator(kmer_size, window_size);\n\n CandidateProcessor candidate_processor(\n mapping_parameters_.min_num_seeds_required_for_mapping,\n mapping_parameters_.max_seed_frequencies);\n\n MappingProcessor mapping_processor(mapping_parameters_,\n min_unique_mapping_mapq_);\n\n DraftMappingGenerator draft_mapping_generator(mapping_parameters_);\n\n MappingGenerator mapping_generator(mapping_parameters_,\n pairs_custom_rid_rank_);\n\n MappingWriter mapping_writer(\n mapping_parameters_, barcode_length_, pairs_custom_rid_rank_);\n\n mapping_writer.OutputHeader(num_reference_sequences, reference);\n\n uint32_t num_mappings_in_mem = 0;\n uint64_t max_num_mappings_in_mem =\n 1 * ((uint64_t)1 << 30) / sizeof(MappingRecord);\n if (mapping_parameters_.mapping_output_format == MAPPINGFORMAT_SAM ||\n mapping_parameters_.mapping_output_format == MAPPINGFORMAT_PAF ||\n mapping_parameters_.mapping_output_format == MAPPINGFORMAT_PAIRS) {\n max_num_mappings_in_mem = 1 * ((uint64_t)1 << 29) / sizeof(MappingRecord);\n }\n\n mm_cache mm_to_candidates_cache(2000003);\n mm_to_candidates_cache.SetKmerLength(kmer_size);\n struct _mm_history *mm_history = new struct _mm_history[read_batch_size_];\n // Use bit encoding to represent mapping results\n // bit 0: is barcode in whitelist\n uint8_t *read_map_summary = NULL ; \n if (!mapping_parameters_.summary_metadata_file_path.empty()) {\n read_map_summary = new uint8_t[read_batch_size_];\n memset(read_map_summary, 1, sizeof(*read_map_summary)*read_batch_size_);\n }\n\n static uint64_t thread_num_candidates = 0;\n static uint64_t thread_num_mappings = 0;\n static uint64_t thread_num_mapped_reads = 0;\n static uint64_t thread_num_uniquely_mapped_reads = 0;\n static uint64_t thread_num_barcode_in_whitelist = 0;\n static uint64_t thread_num_corrected_barcode = 0;\n#pragma omp threadprivate( \\\n thread_num_candidates, thread_num_mappings, thread_num_mapped_reads, \\\n thread_num_uniquely_mapped_reads, thread_num_barcode_in_whitelist, \\\n thread_num_corrected_barcode)\n double real_start_mapping_time = GetRealTime();\n for (size_t read_file_index = 0;\n read_file_index < mapping_parameters_.read_file1_paths.size();\n ++read_file_index) {\n read_batch_for_loading.InitializeLoading(\n mapping_parameters_.read_file1_paths[read_file_index]);\n\n if (!mapping_parameters_.is_bulk_data) {\n barcode_batch_for_loading.InitializeLoading(\n mapping_parameters_.barcode_file_paths[read_file_index]);\n }\n\n uint32_t num_loaded_reads_for_loading = 0;\n uint32_t num_loaded_reads = LoadSingleEndReadsWithBarcodes(\n read_batch_for_loading, barcode_batch_for_loading,\n mapping_parameters_.num_threads >= 3 ? true : false);\n read_batch_for_loading.SwapSequenceBatch(read_batch);\n\n if (!mapping_parameters_.is_bulk_data) {\n barcode_batch_for_loading.SwapSequenceBatch(barcode_batch);\n }\n\n std::vector>>\n mappings_on_diff_ref_seqs_for_diff_threads;\n std::vector>>\n mappings_on_diff_ref_seqs_for_diff_threads_for_saving;\n mappings_on_diff_ref_seqs_for_diff_threads.reserve(\n mapping_parameters_.num_threads);\n mappings_on_diff_ref_seqs_for_diff_threads_for_saving.reserve(\n mapping_parameters_.num_threads);\n for (int ti = 0; ti < mapping_parameters_.num_threads; ++ti) {\n mappings_on_diff_ref_seqs_for_diff_threads.emplace_back(\n std::vector>(num_reference_sequences));\n mappings_on_diff_ref_seqs_for_diff_threads_for_saving.emplace_back(\n std::vector>(num_reference_sequences));\n for (uint32_t i = 0; i < num_reference_sequences; ++i) {\n mappings_on_diff_ref_seqs_for_diff_threads[ti][i].reserve(\n (num_loaded_reads + num_loaded_reads / 1000 *\n mapping_parameters_.max_num_best_mappings) /\n mapping_parameters_.num_threads / num_reference_sequences);\n mappings_on_diff_ref_seqs_for_diff_threads_for_saving[ti][i].reserve(\n (num_loaded_reads + num_loaded_reads / 1000 *\n mapping_parameters_.max_num_best_mappings) /\n mapping_parameters_.num_threads / num_reference_sequences);\n }\n }\n#pragma omp parallel shared(num_reads_, mm_history, read_map_summary, reference, index, read_batch, barcode_batch, read_batch_for_loading, barcode_batch_for_loading, std::cerr, num_loaded_reads_for_loading, num_loaded_reads, num_reference_sequences, mappings_on_diff_ref_seqs_for_diff_threads, mappings_on_diff_ref_seqs_for_diff_threads_for_saving, mappings_on_diff_ref_seqs, temp_mapping_file_handles, mm_to_candidates_cache, mapping_writer, minimizer_generator, candidate_processor, mapping_processor, draft_mapping_generator, mapping_generator, num_mappings_in_mem, max_num_mappings_in_mem) num_threads(mapping_parameters_.num_threads) reduction(+:num_candidates_, num_mappings_, num_mapped_reads_, num_uniquely_mapped_reads_, num_barcode_in_whitelist_, num_corrected_barcode_)\n {\n thread_num_candidates = 0;\n thread_num_mappings = 0;\n thread_num_mapped_reads = 0;\n thread_num_uniquely_mapped_reads = 0;\n thread_num_barcode_in_whitelist = 0;\n thread_num_corrected_barcode = 0;\n MappingMetadata mapping_metadata;\n#pragma omp single\n {\n while (num_loaded_reads > 0) {\n double real_batch_start_time = GetRealTime();\n num_reads_ += num_loaded_reads;\n#pragma omp task\n {\n num_loaded_reads_for_loading = LoadSingleEndReadsWithBarcodes(\n read_batch_for_loading, barcode_batch_for_loading,\n mapping_parameters_.num_threads >= 12 ? true : false);\n } // end of openmp loading task\n uint32_t history_update_threshold =\n mm_to_candidates_cache.GetUpdateThreshold(num_loaded_reads,\n num_reads_, \n false,\n 0.01);\n // int grain_size = 10000;\n//#pragma omp taskloop grainsize(grain_size) //num_tasks(num_threads_* 50)\n#pragma omp taskloop num_tasks( \\\n mapping_parameters_.num_threads *mapping_parameters_.num_threads)\n for (uint32_t read_index = 0; read_index < num_loaded_reads;\n ++read_index) {\n bool current_barcode_is_whitelisted = true;\n if (!mapping_parameters_.barcode_whitelist_file_path.empty()) {\n current_barcode_is_whitelisted = CorrectBarcodeAt(\n read_index, barcode_batch, thread_num_barcode_in_whitelist,\n thread_num_corrected_barcode);\n }\n\n if (!(current_barcode_is_whitelisted ||\n mapping_parameters_.output_mappings_not_in_whitelist)) {\n if (read_map_summary != NULL)\n read_map_summary[read_index] = 0;\n continue;\n }\n \n if (read_batch.GetSequenceLengthAt(read_index) <\n (uint32_t)mapping_parameters_.min_read_length) {\n continue; // reads are too short, just drop.\n }\n\n read_batch.PrepareNegativeSequenceAt(read_index);\n\n mapping_metadata.PrepareForMappingNextRead(\n mapping_parameters_.max_seed_frequencies[0]);\n\n minimizer_generator.GenerateMinimizers(\n read_batch, read_index, mapping_metadata.minimizers_);\n\n if (mapping_metadata.minimizers_.size() > 0) {\n if (mapping_parameters_.custom_rid_order_file_path.length() > 0) {\n RerankCandidatesRid(mapping_metadata.positive_candidates_);\n RerankCandidatesRid(mapping_metadata.negative_candidates_);\n }\n\n if (mm_to_candidates_cache.Query(\n mapping_metadata,\n read_batch.GetSequenceLengthAt(read_index)) == -1) {\n candidate_processor.GenerateCandidates(\n mapping_parameters_.error_threshold, index,\n mapping_metadata);\n }\n\n if (read_index < history_update_threshold) {\n mm_history[read_index].timestamp = num_reads_;\n mm_history[read_index].minimizers =\n mapping_metadata.minimizers_;\n mm_history[read_index].positive_candidates =\n mapping_metadata.positive_candidates_;\n mm_history[read_index].negative_candidates =\n mapping_metadata.negative_candidates_;\n mm_history[read_index].repetitive_seed_length =\n mapping_metadata.repetitive_seed_length_;\n }\n\n size_t current_num_candidates =\n mapping_metadata.GetNumCandidates();\n if (current_num_candidates > 0) {\n thread_num_candidates += current_num_candidates;\n draft_mapping_generator.GenerateDraftMappings(\n read_batch, read_index, reference, mapping_metadata);\n\n const size_t current_num_draft_mappings =\n mapping_metadata.GetNumDraftMappings();\n if (current_num_draft_mappings > 0) {\n std::vector>\n &mappings_on_diff_ref_seqs =\n mappings_on_diff_ref_seqs_for_diff_threads\n [omp_get_thread_num()];\n\n mapping_generator.GenerateBestMappingsForSingleEndRead(\n read_batch, read_index, reference, barcode_batch,\n mapping_metadata, mappings_on_diff_ref_seqs);\n\n thread_num_mappings +=\n std::min(mapping_metadata.GetNumBestMappings(),\n mapping_parameters_.max_num_best_mappings);\n ++thread_num_mapped_reads;\n\n if (mapping_metadata.GetNumBestMappings() == 1) {\n ++thread_num_uniquely_mapped_reads;\n }\n }\n }\n }\n }\n#pragma omp taskwait\n for (uint32_t read_index = 0; read_index < history_update_threshold;\n ++read_index) {\n if (mm_history[read_index].timestamp != num_reads_) continue;\n mm_to_candidates_cache.Update(\n mm_history[read_index].minimizers,\n mm_history[read_index].positive_candidates,\n mm_history[read_index].negative_candidates,\n mm_history[read_index].repetitive_seed_length);\n if (mm_history[read_index].positive_candidates.size() <\n mm_history[read_index].positive_candidates.capacity() / 2) {\n std::vector().swap(\n mm_history[read_index].positive_candidates);\n }\n if (mm_history[read_index].negative_candidates.size() <\n mm_history[read_index].negative_candidates.capacity() / 2) {\n std::vector().swap(\n mm_history[read_index].negative_candidates);\n }\n }\n // std::cerr<<\"cache memusage: \" <<\n // mm_to_candidates_cache.GetMemoryBytes() <<\"\\n\" ;\n if (!mapping_parameters_.summary_metadata_file_path.empty()) {\n if (mapping_parameters_.is_bulk_data) \n mapping_writer.UpdateSummaryMetadata(0, SUMMARY_METADATA_TOTAL, \n num_loaded_reads) ;\n else {\n uint32_t nonwhitelist_count = 0;\n for (uint32_t read_index = 0; read_index < num_loaded_reads; ++read_index)\n if (read_map_summary[read_index] & 1) {\n mapping_writer.UpdateSummaryMetadata(\n barcode_batch.GenerateSeedFromSequenceAt(read_index, 0, barcode_length_), \n SUMMARY_METADATA_TOTAL, 1);\n } else {\n ++nonwhitelist_count;\n }\n \n mapping_writer.UpdateSpeicalCategorySummaryMetadata(/*nonwhitelist*/0, \n SUMMARY_METADATA_TOTAL, nonwhitelist_count);\n }\n\n // By default, set the lowest bit to 1 (whether the barcode is in the whitelist)\n memset(read_map_summary, 1, sizeof(*read_map_summary)*read_batch_size_);\n }\n num_loaded_reads = num_loaded_reads_for_loading;\n read_batch_for_loading.SwapSequenceBatch(read_batch);\n barcode_batch_for_loading.SwapSequenceBatch(barcode_batch);\n mappings_on_diff_ref_seqs_for_diff_threads.swap(\n mappings_on_diff_ref_seqs_for_diff_threads_for_saving);\n#pragma omp task\n {\n num_mappings_in_mem +=\n mapping_processor.MoveMappingsInBuffersToMappingContainer(\n num_reference_sequences,\n mappings_on_diff_ref_seqs_for_diff_threads_for_saving,\n mappings_on_diff_ref_seqs);\n if (mapping_parameters_.low_memory_mode &&\n num_mappings_in_mem > max_num_mappings_in_mem) {\n mapping_processor.ParallelSortOutputMappings(num_reference_sequences,\n mappings_on_diff_ref_seqs, 0);\n\n mapping_writer.OutputTempMappings(num_reference_sequences,\n mappings_on_diff_ref_seqs,\n temp_mapping_file_handles);\n\n if (temp_mapping_file_handles.size() > 850\n && temp_mapping_file_handles.size() % 10 == 1) { // every 10 temp files, double the temp file size\n max_num_mappings_in_mem <<= 1;\n std::cerr << \"Used \" << temp_mapping_file_handles.size() << \"temp files. Double the temp file volume to \" << max_num_mappings_in_mem << \"\\n\" ;\n }\n num_mappings_in_mem = 0;\n }\n }\n std::cerr << \"Mapped \" << num_loaded_reads << \" reads in \"\n << GetRealTime() - real_batch_start_time << \"s.\\n\";\n }\n } // end of openmp single\n {\n num_barcode_in_whitelist_ += thread_num_barcode_in_whitelist;\n num_corrected_barcode_ += thread_num_corrected_barcode;\n num_candidates_ += thread_num_candidates;\n num_mappings_ += thread_num_mappings;\n num_mapped_reads_ += thread_num_mapped_reads;\n num_uniquely_mapped_reads_ += thread_num_uniquely_mapped_reads;\n } // end of updating shared mapping stats\n } // end of openmp parallel region\n read_batch_for_loading.FinalizeLoading();\n if (!mapping_parameters_.is_bulk_data) {\n barcode_batch_for_loading.FinalizeLoading();\n }\n }\n\n std::cerr << \"Mapped all reads in \" << GetRealTime() - real_start_mapping_time\n << \"s.\\n\";\n\n delete[] mm_history;\n if (read_map_summary != NULL)\n delete[] read_map_summary;\n\n OutputMappingStatistics();\n if (!mapping_parameters_.is_bulk_data) {\n OutputBarcodeStatistics();\n }\n\n index.Destroy();\n\n if (mapping_parameters_.low_memory_mode) {\n // First, process the remaining mappings in the memory and save them on\n // disk.\n if (num_mappings_in_mem > 0) {\n mapping_processor.SortOutputMappings(num_reference_sequences,\n mappings_on_diff_ref_seqs);\n\n mapping_writer.OutputTempMappings(num_reference_sequences,\n mappings_on_diff_ref_seqs,\n temp_mapping_file_handles);\n num_mappings_in_mem = 0;\n }\n\n mapping_writer.ProcessAndOutputMappingsInLowMemory(\n num_mappings_in_mem, num_reference_sequences, reference,\n barcode_whitelist_lookup_table_, temp_mapping_file_handles);\n } else {\n if (mapping_parameters_.Tn5_shift) {\n mapping_processor.ApplyTn5ShiftOnMappings(num_reference_sequences,\n mappings_on_diff_ref_seqs);\n }\n\n if (mapping_parameters_.remove_pcr_duplicates) {\n mapping_processor.RemovePCRDuplicate(num_reference_sequences,\n mappings_on_diff_ref_seqs,\n mapping_parameters_.num_threads);\n std::cerr << \"After removing PCR duplications, \";\n mapping_processor.OutputMappingStatistics(num_reference_sequences,\n mappings_on_diff_ref_seqs);\n } else {\n mapping_processor.ParallelSortOutputMappings(num_reference_sequences,\n mappings_on_diff_ref_seqs,\n mapping_parameters_.num_threads);\n }\n\n if (mapping_parameters_.allocate_multi_mappings) {\n const uint64_t num_multi_mappings =\n num_mapped_reads_ - num_uniquely_mapped_reads_;\n mapping_processor.AllocateMultiMappings(\n num_reference_sequences, num_multi_mappings,\n mapping_parameters_.multi_mapping_allocation_distance,\n mappings_on_diff_ref_seqs);\n std::cerr << \"After allocating multi-mappings, \";\n mapping_processor.OutputMappingStatistics(num_reference_sequences,\n mappings_on_diff_ref_seqs);\n mapping_processor.SortOutputMappings(num_reference_sequences,\n mappings_on_diff_ref_seqs);\n }\n mapping_writer.OutputMappings(num_reference_sequences, reference,\n mappings_on_diff_ref_seqs);\n }\n mapping_writer.OutputSummaryMetadata();\n\n reference.FinalizeLoading();\n std::cerr << \"Total time: \" << GetRealTime() - real_start_time << \"s.\\n\";\n}\n\ntemplate \nvoid Chromap::MapPairedEndReads() {\n double real_start_time = GetRealTime();\n\n // Load reference\n SequenceBatch reference;\n reference.InitializeLoading(mapping_parameters_.reference_file_path);\n reference.LoadAllSequences();\n uint32_t num_reference_sequences = reference.GetNumSequences();\n \n // Debugging Info (printing out reference information)\n if (mapping_parameters_.debug_cache) {\n for (size_t i = 0; i < num_reference_sequences; i++){\n std::cout << \"[DEBUG][INDEX] seq_i = \" << i \n << \" , seq_i_name = \" << reference.GetSequenceNameAt(i) << std::endl;\n }\n }\n \n if (mapping_parameters_.custom_rid_order_file_path.length() > 0) {\n GenerateCustomRidRanks(mapping_parameters_.custom_rid_order_file_path,\n num_reference_sequences, reference,\n custom_rid_rank_);\n reference.ReorderSequences(custom_rid_rank_);\n }\n if (mapping_parameters_.mapping_output_format == MAPPINGFORMAT_PAIRS) {\n GenerateCustomRidRanks(\n mapping_parameters_.pairs_flipping_custom_rid_order_file_path,\n num_reference_sequences, reference, pairs_custom_rid_rank_);\n }\n\n // Load index\n Index index(mapping_parameters_.index_file_path);\n index.Load();\n const int kmer_size = index.GetKmerSize();\n const int window_size = index.GetWindowSize();\n // index.Statistics(num_sequences, reference);\n\n // Initialize read batches\n SequenceBatch read_batch1(read_batch_size_, read1_effective_range_);\n SequenceBatch read_batch2(read_batch_size_, read2_effective_range_);\n SequenceBatch barcode_batch(read_batch_size_, barcode_effective_range_);\n SequenceBatch read_batch1_for_loading(read_batch_size_,\n read1_effective_range_);\n SequenceBatch read_batch2_for_loading(read_batch_size_,\n read2_effective_range_);\n SequenceBatch barcode_batch_for_loading(read_batch_size_,\n barcode_effective_range_);\n\n // Check cache-related parameters\n std::cerr << \"Cache Size: \" << mapping_parameters_.cache_size << std::endl;\n std::cerr << \"Cache Update Param: \" << mapping_parameters_.cache_update_param << std::endl;\n \n std::vector seeds_for_batch(500000, 0);\n\n // Variables used for counting number of associated cache slots\n bool output_num_cache_slots_info = mapping_parameters_.output_num_uniq_cache_slots;\n if (mapping_parameters_.summary_metadata_file_path.empty()) {\n output_num_cache_slots_info = false;\n }\n const size_t k_for_minhash = mapping_parameters_.k_for_minhash;\n\n std::cerr << \"Output number of associated cache slots: \" << output_num_cache_slots_info << std::endl;\n std::cerr << \"K for MinHash: \" << k_for_minhash << std::endl;\n\n int num_locks_for_map = 1000;\n omp_lock_t map_locks[num_locks_for_map];\n for (int i = 0; i < num_locks_for_map; ++i) {omp_init_lock(&map_locks[i]);}\n \n std::vector> barcode_peak_map(num_locks_for_map);\n\n // Parse out the parameters for chromap score (const, fric, dup, unmapped, lowmapq)\n std::vector frip_est_params; \n std::stringstream ss(mapping_parameters_.frip_est_params);\n std::string token;\n\n while(std::getline(ss, token, ';')) {\n try {\n auto curr_param = std::stod(token);\n frip_est_params.push_back(curr_param);\n } catch(...) {\n chromap::ExitWithMessage(\n \"\\nException occurred while processing chromap score parameters\\n\"\n );\n }\n }\n if (frip_est_params.size() != 5) {\n chromap::ExitWithMessage(\n \"\\nInvalid number of parameters, expecting 5 parameters but found \" \n + std::to_string(frip_est_params.size()) \n + \" parameters\\n\"\n );\n }\n\n // Initialize vector to keep track of cache hits for each thread\n std::vector cache_hits_per_thread(mapping_parameters_.num_threads, 0);\n\n // Initialize cache\n mm_cache mm_to_candidates_cache(mapping_parameters_.cache_size);\n mm_to_candidates_cache.SetKmerLength(kmer_size);\n\n struct _mm_history *mm_history1 = new struct _mm_history[read_batch_size_];\n struct _mm_history *mm_history2 = new struct _mm_history[read_batch_size_];\n \n // The explanation for read_map_summary is in the single-end mapping function\n uint8_t *read_map_summary = NULL ;\n if (!mapping_parameters_.summary_metadata_file_path.empty()) {\n read_map_summary = new uint8_t[read_batch_size_];\n memset(read_map_summary, 1, sizeof(*read_map_summary)*read_batch_size_);\n }\n std::vector> mappings_on_diff_ref_seqs;\n \n // Initialize mapping container\n mappings_on_diff_ref_seqs.reserve(num_reference_sequences);\n for (uint32_t i = 0; i < num_reference_sequences; ++i) {\n mappings_on_diff_ref_seqs.emplace_back(std::vector());\n }\n std::vector> temp_mapping_file_handles;\n\n // Preprocess barcodes for single cell data\n if (!mapping_parameters_.is_bulk_data) {\n barcode_length_ = SampleInputBarcodesAndExamineLength();\n if (!mapping_parameters_.barcode_whitelist_file_path.empty()) {\n LoadBarcodeWhitelist();\n ComputeBarcodeAbundance(initial_num_sample_barcodes_);\n }\n }\n\n MinimizerGenerator minimizer_generator(kmer_size, window_size);\n\n CandidateProcessor candidate_processor(\n mapping_parameters_.min_num_seeds_required_for_mapping,\n mapping_parameters_.max_seed_frequencies);\n\n MappingProcessor mapping_processor(mapping_parameters_,\n min_unique_mapping_mapq_);\n\n DraftMappingGenerator draft_mapping_generator(mapping_parameters_);\n\n MappingGenerator mapping_generator(mapping_parameters_,\n pairs_custom_rid_rank_);\n\n MappingWriter mapping_writer(\n mapping_parameters_, barcode_length_, pairs_custom_rid_rank_);\n mapping_writer.OutputHeader(num_reference_sequences, reference);\n\n uint32_t num_mappings_in_mem = 0;\n uint64_t max_num_mappings_in_mem =\n 1 * ((uint64_t)1 << 30) / sizeof(MappingRecord);\n if (mapping_parameters_.mapping_output_format == MAPPINGFORMAT_SAM ||\n mapping_parameters_.mapping_output_format == MAPPINGFORMAT_PAF ||\n mapping_parameters_.mapping_output_format == MAPPINGFORMAT_PAIRS) {\n max_num_mappings_in_mem = 1 * ((uint64_t)1 << 29) / sizeof(MappingRecord);\n }\n\n static uint64_t thread_num_candidates = 0;\n static uint64_t thread_num_mappings = 0;\n static uint64_t thread_num_mapped_reads = 0;\n static uint64_t thread_num_uniquely_mapped_reads = 0;\n static uint64_t thread_num_barcode_in_whitelist = 0;\n static uint64_t thread_num_corrected_barcode = 0;\n#pragma omp threadprivate( \\\n thread_num_candidates, thread_num_mappings, thread_num_mapped_reads, \\\n thread_num_uniquely_mapped_reads, thread_num_barcode_in_whitelist, \\\n thread_num_corrected_barcode)\n double real_start_mapping_time = GetRealTime();\n for (size_t read_file_index = 0;\n read_file_index < mapping_parameters_.read_file1_paths.size();\n ++read_file_index) {\n // Set read batches to the current read files.\n read_batch1_for_loading.InitializeLoading(\n mapping_parameters_.read_file1_paths[read_file_index]);\n read_batch2_for_loading.InitializeLoading(\n mapping_parameters_.read_file2_paths[read_file_index]);\n if (!mapping_parameters_.is_bulk_data) {\n barcode_batch_for_loading.InitializeLoading(\n mapping_parameters_.barcode_file_paths[read_file_index]);\n }\n\n // Load the first batches.\n uint32_t num_loaded_pairs_for_loading = 0;\n uint32_t num_loaded_pairs = LoadPairedEndReadsWithBarcodes(\n read_batch1_for_loading, read_batch2_for_loading,\n barcode_batch_for_loading, mapping_parameters_.num_threads >= 3 ? true : false);\n read_batch1_for_loading.SwapSequenceBatch(read_batch1);\n read_batch2_for_loading.SwapSequenceBatch(read_batch2);\n if (!mapping_parameters_.is_bulk_data) {\n barcode_batch_for_loading.SwapSequenceBatch(barcode_batch);\n }\n\n // Setup thread private vectors to save mapping results.\n std::vector>>\n mappings_on_diff_ref_seqs_for_diff_threads;\n std::vector>>\n mappings_on_diff_ref_seqs_for_diff_threads_for_saving;\n mappings_on_diff_ref_seqs_for_diff_threads.reserve(\n mapping_parameters_.num_threads);\n mappings_on_diff_ref_seqs_for_diff_threads_for_saving.reserve(\n mapping_parameters_.num_threads);\n for (int ti = 0; ti < mapping_parameters_.num_threads; ++ti) {\n mappings_on_diff_ref_seqs_for_diff_threads.emplace_back(\n std::vector>(num_reference_sequences));\n mappings_on_diff_ref_seqs_for_diff_threads_for_saving.emplace_back(\n std::vector>(num_reference_sequences));\n for (uint32_t i = 0; i < num_reference_sequences; ++i) {\n mappings_on_diff_ref_seqs_for_diff_threads[ti][i].reserve(\n (num_loaded_pairs + num_loaded_pairs / 1000 *\n mapping_parameters_.max_num_best_mappings) /\n mapping_parameters_.num_threads / num_reference_sequences);\n mappings_on_diff_ref_seqs_for_diff_threads_for_saving[ti][i].reserve(\n (num_loaded_pairs + num_loaded_pairs / 1000 *\n mapping_parameters_.max_num_best_mappings) /\n mapping_parameters_.num_threads / num_reference_sequences);\n }\n }\n\n#pragma omp parallel shared(num_reads_, num_reference_sequences, reference, index, read_batch1, read_batch2, barcode_batch, read_batch1_for_loading, read_batch2_for_loading, barcode_batch_for_loading, minimizer_generator, candidate_processor, mapping_processor, draft_mapping_generator, mapping_generator, mapping_writer, std::cerr, num_loaded_pairs_for_loading, num_loaded_pairs, mappings_on_diff_ref_seqs_for_diff_threads, mappings_on_diff_ref_seqs_for_diff_threads_for_saving, mappings_on_diff_ref_seqs, num_mappings_in_mem, max_num_mappings_in_mem, temp_mapping_file_handles, mm_to_candidates_cache, mm_history1, mm_history2, read_map_summary) num_threads(mapping_parameters_.num_threads) reduction(+:num_candidates_, num_mappings_, num_mapped_reads_, num_uniquely_mapped_reads_, num_barcode_in_whitelist_, num_corrected_barcode_)\n {\n thread_num_candidates = 0;\n thread_num_mappings = 0;\n thread_num_mapped_reads = 0;\n thread_num_uniquely_mapped_reads = 0;\n thread_num_barcode_in_whitelist = 0;\n thread_num_corrected_barcode = 0;\n PairedEndMappingMetadata paired_end_mapping_metadata;\n\n std::vector best_mapping_indices(\n mapping_parameters_.max_num_best_mappings);\n std::mt19937 generator(11);\n#pragma omp single\n {\n double real_batch_start_time = GetRealTime();\n while (num_loaded_pairs > 0) {\n num_reads_ += num_loaded_pairs;\n num_reads_ += num_loaded_pairs;\n\n#pragma omp task\n {\n num_loaded_pairs_for_loading = LoadPairedEndReadsWithBarcodes(\n read_batch1_for_loading, read_batch2_for_loading,\n barcode_batch_for_loading, \n mapping_parameters_.num_threads >= 12 ? true : false);\n } // end of openmp loading task\n\n int grain_size = 5000;\n uint32_t history_update_threshold =\n mm_to_candidates_cache.GetUpdateThreshold(num_loaded_pairs,\n num_reads_, \n true,\n mapping_parameters_.cache_update_param\n );\n std::fill(cache_hits_per_thread.begin(), cache_hits_per_thread.end(), 0);\n\n if (mapping_parameters_.debug_cache) {\n std::cout << \"[DEBUG][UPDATE] update_threshold = \" << history_update_threshold << std::endl;\n }\n\n#pragma omp taskloop grainsize(grain_size)\n for (uint32_t pair_index = 0; pair_index < num_loaded_pairs;\n ++pair_index) {\n int thread_id = omp_get_thread_num();\n \n bool current_barcode_is_whitelisted = true;\n if (!mapping_parameters_.barcode_whitelist_file_path.empty()) {\n current_barcode_is_whitelisted = CorrectBarcodeAt(\n pair_index, barcode_batch, thread_num_barcode_in_whitelist,\n thread_num_corrected_barcode);\n }\n\n // calculate seed value for each barcode to use later (below and summary update)\n size_t curr_seed_val = barcode_batch.GenerateSeedFromSequenceAt(pair_index, 0, barcode_length_);\n seeds_for_batch[pair_index] = curr_seed_val;\n\n if (current_barcode_is_whitelisted ||\n mapping_parameters_.output_mappings_not_in_whitelist) {\n \n if (read_batch1.GetSequenceLengthAt(pair_index) <\n (uint32_t)mapping_parameters_.min_read_length ||\n read_batch2.GetSequenceLengthAt(pair_index) <\n (uint32_t)mapping_parameters_.min_read_length) {\n continue; // reads are too short, just drop.\n }\n\n read_batch1.PrepareNegativeSequenceAt(pair_index);\n read_batch2.PrepareNegativeSequenceAt(pair_index);\n\n if (mapping_parameters_.trim_adapters) {\n TrimAdapterForPairedEndRead(pair_index, read_batch1,\n read_batch2);\n }\n\n paired_end_mapping_metadata.PreparedForMappingNextReadPair(\n mapping_parameters_.max_seed_frequencies[0]);\n\n minimizer_generator.GenerateMinimizers(\n read_batch1, pair_index,\n paired_end_mapping_metadata.mapping_metadata1_.minimizers_);\n minimizer_generator.GenerateMinimizers(\n read_batch2, pair_index,\n paired_end_mapping_metadata.mapping_metadata2_.minimizers_);\n\n if (paired_end_mapping_metadata.BothEndsHaveMinimizers()) {\n // declare temp local variable for cache result\n int cache_query_result1 = 0;\n int cache_query_result2 = 0;\n int cache_miss = 0;\n\n cache_query_result1 = mm_to_candidates_cache.Query(paired_end_mapping_metadata.mapping_metadata1_,\n read_batch1.GetSequenceLengthAt(pair_index));\n if (cache_query_result1 == -1) \n {\n candidate_processor.GenerateCandidates(\n mapping_parameters_.error_threshold, \n index,\n paired_end_mapping_metadata.mapping_metadata1_\n );\n ++cache_miss;\n }\n size_t current_num_candidates1 = paired_end_mapping_metadata.mapping_metadata1_.GetNumCandidates();\n\n\n cache_query_result2 = mm_to_candidates_cache.Query(paired_end_mapping_metadata.mapping_metadata2_,\n read_batch2.GetSequenceLengthAt(pair_index));\n if (cache_query_result2 == -1) \n {\n candidate_processor.GenerateCandidates(\n mapping_parameters_.error_threshold, \n index,\n paired_end_mapping_metadata.mapping_metadata2_\n );\n ++cache_miss;\n }\n size_t current_num_candidates2 = paired_end_mapping_metadata.mapping_metadata2_.GetNumCandidates();\n\n // increment variable for cache_hits\n bool curr_read_hit_cache = false;\n if (cache_query_result1 >= 0 || cache_query_result2 >= 0) {\n cache_hits_per_thread[thread_id]++;\n curr_read_hit_cache = true;\n }\n\n // update the peak counting data-structure\n if (output_num_cache_slots_info && curr_read_hit_cache) {\n // calculate which map this barcode is in\n size_t map_id = curr_seed_val % num_locks_for_map;\n \n // grab lock for this map, and add to the K-MinHash for this particular barcode\n omp_set_lock(&map_locks[map_id]);\n auto it = barcode_peak_map[map_id].emplace(curr_seed_val, K_MinHash(k_for_minhash, mapping_parameters_.cache_size)).first;\n if (cache_query_result1 >= 0) {it->second.add(cache_query_result1);}\n if (cache_query_result2 >= 0) {it->second.add(cache_query_result2);}\n omp_unset_lock(&map_locks[map_id]);\n }\n\n if (pair_index < history_update_threshold) {\n mm_history1[pair_index].timestamp =\n mm_history2[pair_index].timestamp = num_reads_;\n mm_history1[pair_index].minimizers =\n paired_end_mapping_metadata.mapping_metadata1_\n .minimizers_;\n mm_history1[pair_index].positive_candidates =\n paired_end_mapping_metadata.mapping_metadata1_\n .positive_candidates_;\n mm_history1[pair_index].negative_candidates =\n paired_end_mapping_metadata.mapping_metadata1_\n .negative_candidates_;\n mm_history1[pair_index].repetitive_seed_length =\n paired_end_mapping_metadata.mapping_metadata1_\n .repetitive_seed_length_;\n mm_history2[pair_index].minimizers =\n paired_end_mapping_metadata.mapping_metadata2_\n .minimizers_;\n mm_history2[pair_index].positive_candidates =\n paired_end_mapping_metadata.mapping_metadata2_\n .positive_candidates_;\n mm_history2[pair_index].negative_candidates =\n paired_end_mapping_metadata.mapping_metadata2_\n .negative_candidates_;\n mm_history2[pair_index].repetitive_seed_length =\n paired_end_mapping_metadata.mapping_metadata2_\n .repetitive_seed_length_;\n }\n\n // Test whether we need to augment the candidate list with mate\n // information.\n int supplementCandidateResult = 0;\n if (!mapping_parameters_.split_alignment) {\n supplementCandidateResult =\n candidate_processor.SupplementCandidates(\n mapping_parameters_.error_threshold,\n /*search_range=*/2 *\n mapping_parameters_.max_insert_size,\n index, paired_end_mapping_metadata);\n current_num_candidates1 =\n paired_end_mapping_metadata.mapping_metadata1_\n .GetNumCandidates();\n current_num_candidates2 =\n paired_end_mapping_metadata.mapping_metadata2_\n .GetNumCandidates();\n }\n\n if (current_num_candidates1 > 0 &&\n current_num_candidates2 > 0 &&\n !mapping_parameters_.split_alignment) {\n paired_end_mapping_metadata.MoveCandidiatesToBuffer();\n\n // Paired-end filter\n candidate_processor.ReduceCandidatesForPairedEndRead(\n mapping_parameters_.max_insert_size,\n paired_end_mapping_metadata);\n\n current_num_candidates1 =\n paired_end_mapping_metadata.mapping_metadata1_\n .GetNumCandidates();\n current_num_candidates2 =\n paired_end_mapping_metadata.mapping_metadata2_\n .GetNumCandidates();\n }\n\n // Verify candidates\n if (current_num_candidates1 > 0 &&\n current_num_candidates2 > 0) {\n thread_num_candidates +=\n current_num_candidates1 + current_num_candidates2;\n\n if (mapping_parameters_.custom_rid_order_file_path.length() >\n 0) {\n RerankCandidatesRid(\n paired_end_mapping_metadata.mapping_metadata1_\n .positive_candidates_);\n RerankCandidatesRid(\n paired_end_mapping_metadata.mapping_metadata1_\n .negative_candidates_);\n RerankCandidatesRid(\n paired_end_mapping_metadata.mapping_metadata2_\n .positive_candidates_);\n RerankCandidatesRid(\n paired_end_mapping_metadata.mapping_metadata2_\n .negative_candidates_);\n }\n\n draft_mapping_generator.GenerateDraftMappings(\n read_batch1, pair_index, reference,\n paired_end_mapping_metadata.mapping_metadata1_);\n\n const size_t current_num_draft_mappings1 =\n paired_end_mapping_metadata.mapping_metadata1_\n .GetNumDraftMappings();\n\n draft_mapping_generator.GenerateDraftMappings(\n read_batch2, pair_index, reference,\n paired_end_mapping_metadata.mapping_metadata2_);\n\n const size_t current_num_draft_mappings2 =\n paired_end_mapping_metadata.mapping_metadata2_\n .GetNumDraftMappings();\n\n if (current_num_draft_mappings1 > 0 &&\n current_num_draft_mappings2 > 0) {\n std::vector>\n &mappings_on_diff_ref_seqs =\n mappings_on_diff_ref_seqs_for_diff_threads\n [omp_get_thread_num()];\n\n if (!mapping_parameters_.split_alignment) {\n // GenerateBestMappingsForPairedEndRead assumes the\n // mappings are sorted by coordinate for non split\n // alignments. In split alignment, we don't want to sort\n // and this keeps mapping and split_sites vectors\n // consistent.\n paired_end_mapping_metadata.SortMappingsByPositions();\n }\n\n int force_mapq = -1;\n if (supplementCandidateResult != 0) {\n force_mapq = 0;\n }\n\n mapping_generator.GenerateBestMappingsForPairedEndRead(\n pair_index, read_batch1, read_batch2, barcode_batch,\n reference, best_mapping_indices, generator, force_mapq,\n paired_end_mapping_metadata, mappings_on_diff_ref_seqs);\n\n if (paired_end_mapping_metadata.GetNumBestMappings() == 1) {\n ++thread_num_uniquely_mapped_reads;\n ++thread_num_uniquely_mapped_reads;\n }\n\n thread_num_mappings += std::min(\n paired_end_mapping_metadata.GetNumBestMappings(),\n mapping_parameters_.max_num_best_mappings);\n thread_num_mappings += std::min(\n paired_end_mapping_metadata.GetNumBestMappings(),\n mapping_parameters_.max_num_best_mappings);\n if (paired_end_mapping_metadata.GetNumBestMappings() > 0) {\n ++thread_num_mapped_reads;\n ++thread_num_mapped_reads;\n\n if (read_map_summary != NULL)\n read_map_summary[pair_index] |= (cache_miss < 2 ? 2 : 0) ;\n }\n }\n } // verify candidate\n }\n } else {\n if (read_map_summary != NULL)\n read_map_summary[pair_index] = 0 ;\n }\n } // end of for pair_index\n\n // if (num_reads_ / 2 > initial_num_sample_barcodes_) {\n // if (!is_bulk_data_) {\n // if (!barcode_whitelist_file_path_.empty()) {\n // UpdateBarcodeAbundance(num_loaded_pairs, barcode_batch);\n // }\n // }\n //}\n#pragma omp taskloop grainsize( std::max(history_update_threshold / mapping_parameters_.num_threads, (unsigned int)grain_size) )\n // Update cache\n for (uint32_t pair_index = 0; pair_index < history_update_threshold;\n ++pair_index) {\n if (mm_history1[pair_index].timestamp != num_reads_) continue;\n\n mm_to_candidates_cache.Update(\n mm_history1[pair_index].minimizers,\n mm_history1[pair_index].positive_candidates,\n mm_history1[pair_index].negative_candidates,\n mm_history1[pair_index].repetitive_seed_length,\n mapping_parameters_.debug_cache);\n mm_to_candidates_cache.Update(\n mm_history2[pair_index].minimizers,\n mm_history2[pair_index].positive_candidates,\n mm_history2[pair_index].negative_candidates,\n mm_history2[pair_index].repetitive_seed_length,\n mapping_parameters_.debug_cache);\n\n if (mm_history1[pair_index].positive_candidates.size() > 50) {\n std::vector().swap(\n mm_history1[pair_index].positive_candidates);\n }\n if (mm_history1[pair_index].negative_candidates.size() > 50) {\n std::vector().swap(\n mm_history1[pair_index].negative_candidates);\n }\n if (mm_history2[pair_index].positive_candidates.size() > 50) {\n std::vector().swap(\n mm_history2[pair_index].positive_candidates);\n }\n if (mm_history2[pair_index].negative_candidates.size() > 50) {\n std::vector().swap(\n mm_history2[pair_index].negative_candidates);\n }\n }\n\n#pragma omp taskwait\n if (!mapping_parameters_.summary_metadata_file_path.empty()) {\n // Update total read count and number of cache hits\n if (mapping_parameters_.is_bulk_data) {\n // Sum up cache hits for each thread\n int cache_hits_for_batch = 0;\n for (int hits: cache_hits_per_thread) {\n cache_hits_for_batch += hits;\n }\n mapping_writer.UpdateSummaryMetadata(0, \n SUMMARY_METADATA_TOTAL, \n num_loaded_pairs);\n mapping_writer.UpdateSummaryMetadata(0,\n SUMMARY_METADATA_CACHEHIT, \n cache_hits_for_batch);\n }\n else {\n uint32_t nonwhitelist_count = 0;\n for (uint32_t pair_index = 0; pair_index < num_loaded_pairs; ++pair_index) {\n uint64_t pair_seed = seeds_for_batch[pair_index];\n if (read_map_summary[pair_index] & 1) {\n mapping_writer.UpdateSummaryMetadata(\n pair_seed, \n SUMMARY_METADATA_TOTAL, \n 1);\n } else {\n ++nonwhitelist_count ;\n }\n\n if (read_map_summary[pair_index] & 2) {\n mapping_writer.UpdateSummaryMetadata( \n pair_seed,\n SUMMARY_METADATA_CACHEHIT, \n 1);\n }\n }\n mapping_writer.UpdateSpeicalCategorySummaryMetadata(/*nonwhitelist*/0, \n SUMMARY_METADATA_TOTAL, nonwhitelist_count);\n } \n\n memset(read_map_summary, 1, sizeof(*read_map_summary)*read_batch_size_);\n }\n\n std::cerr << \"Mapped \" << num_loaded_pairs << \" read pairs in \"\n << GetRealTime() - real_batch_start_time << \"s.\\n\";\n real_batch_start_time = GetRealTime();\n\n // Swap to next batch\n num_loaded_pairs = num_loaded_pairs_for_loading;\n read_batch1_for_loading.SwapSequenceBatch(read_batch1);\n read_batch2_for_loading.SwapSequenceBatch(read_batch2);\n barcode_batch_for_loading.SwapSequenceBatch(barcode_batch);\n mappings_on_diff_ref_seqs_for_diff_threads.swap(\n mappings_on_diff_ref_seqs_for_diff_threads_for_saving);\n\n // Reset for next batch\n std::fill(seeds_for_batch.begin(), seeds_for_batch.end(), 0);\n\n#pragma omp task\n {\n // Handle output\n num_mappings_in_mem +=\n mapping_processor.MoveMappingsInBuffersToMappingContainer(\n num_reference_sequences,\n mappings_on_diff_ref_seqs_for_diff_threads_for_saving,\n mappings_on_diff_ref_seqs);\n if (mapping_parameters_.low_memory_mode &&\n num_mappings_in_mem > max_num_mappings_in_mem) {\n mapping_processor.ParallelSortOutputMappings(num_reference_sequences,\n mappings_on_diff_ref_seqs, 0);\n\n mapping_writer.OutputTempMappings(num_reference_sequences,\n mappings_on_diff_ref_seqs,\n temp_mapping_file_handles);\n if (temp_mapping_file_handles.size() > 850\n && temp_mapping_file_handles.size() % 10 == 1) { // every 10 temp files, double the temp file size\n max_num_mappings_in_mem <<= 1;\n std::cerr << \"Used \" << temp_mapping_file_handles.size() << \"temp files. Double the temp file volume to \" << max_num_mappings_in_mem << \"\\n\" ;\n }\n num_mappings_in_mem = 0;\n }\n } // end of omp task to handle output\n } // end of while num_loaded_pairs\n } // end of openmp single\n\n num_barcode_in_whitelist_ += thread_num_barcode_in_whitelist;\n num_corrected_barcode_ += thread_num_corrected_barcode;\n num_candidates_ += thread_num_candidates;\n num_mappings_ += thread_num_mappings;\n num_mapped_reads_ += thread_num_mapped_reads;\n num_uniquely_mapped_reads_ += thread_num_uniquely_mapped_reads;\n } // end of openmp parallel region\n\n read_batch1_for_loading.FinalizeLoading();\n read_batch2_for_loading.FinalizeLoading();\n\n if (!mapping_parameters_.is_bulk_data) {\n barcode_batch_for_loading.FinalizeLoading();\n }\n } // end of for read_file_index\n\n std::cerr << \"Mapped all reads in \" << GetRealTime() - real_start_mapping_time\n << \"s.\\n\";\n\n delete[] mm_history1;\n delete[] mm_history2;\n if (read_map_summary != NULL)\n delete[] read_map_summary;\n\n OutputMappingStatistics();\n if (!mapping_parameters_.is_bulk_data) {\n OutputBarcodeStatistics();\n }\n\n index.Destroy();\n\n if (mapping_parameters_.low_memory_mode) {\n // First, process the remaining mappings in the memory and save them on\n // disk.\n if (num_mappings_in_mem > 0) {\n mapping_processor.SortOutputMappings(num_reference_sequences,\n mappings_on_diff_ref_seqs);\n\n mapping_writer.OutputTempMappings(num_reference_sequences,\n mappings_on_diff_ref_seqs,\n temp_mapping_file_handles);\n num_mappings_in_mem = 0;\n }\n\n mapping_writer.ProcessAndOutputMappingsInLowMemory(\n num_mappings_in_mem, num_reference_sequences, reference,\n barcode_whitelist_lookup_table_, temp_mapping_file_handles);\n } \n else {\n if (mapping_parameters_.Tn5_shift) {\n mapping_processor.ApplyTn5ShiftOnMappings(num_reference_sequences,\n mappings_on_diff_ref_seqs);\n }\n\n if (mapping_parameters_.remove_pcr_duplicates) {\n mapping_processor.RemovePCRDuplicate(num_reference_sequences,\n mappings_on_diff_ref_seqs,\n mapping_parameters_.num_threads);\n std::cerr << \"After removing PCR duplications, \";\n mapping_processor.OutputMappingStatistics(num_reference_sequences,\n mappings_on_diff_ref_seqs);\n } else {\n mapping_processor.ParallelSortOutputMappings(num_reference_sequences,\n mappings_on_diff_ref_seqs,\n mapping_parameters_.num_threads);\n }\n\n if (mapping_parameters_.allocate_multi_mappings) {\n const uint64_t num_multi_mappings =\n num_mapped_reads_ - num_uniquely_mapped_reads_;\n mapping_processor.AllocateMultiMappings(\n num_reference_sequences, num_multi_mappings,\n mapping_parameters_.multi_mapping_allocation_distance,\n mappings_on_diff_ref_seqs);\n std::cerr << \"After allocating multi-mappings, \";\n mapping_processor.OutputMappingStatistics(num_reference_sequences,\n mappings_on_diff_ref_seqs);\n mapping_processor.SortOutputMappings(num_reference_sequences,\n mappings_on_diff_ref_seqs);\n }\n mapping_writer.OutputMappings(num_reference_sequences, reference,\n mappings_on_diff_ref_seqs);\n // Temporarily disable feature matrix output. Do not delete the following\n // commented code.\n // if (!is_bulk_data_ && !matrix_output_prefix_.empty()) {\n // if constexpr (std::is_same::value) {\n // FeatureBarcodeMatrix feature_barcode_matrix(\n // cell_by_bin_, bin_size_, multi_mapping_allocation_distance_,\n // depth_cutoff_to_call_peak_);\n // std::vector> &mappings =\n // allocate_multi_mappings_\n // ? allocated_mappings_on_diff_ref_seqs\n // : (remove_pcr_duplicates_ ? deduped_mappings_on_diff_ref_seqs\n // : mappings_on_diff_ref_seqs);\n\n // feature_barcode_matrix.OutputFeatureMatrix(num_reference_sequences,\n // reference, mappings,\n // matrix_output_prefix_);\n // }\n //}\n }\n \n if (mapping_parameters_.mapping_output_format == MAPPINGFORMAT_SAM)\n mapping_writer.AdjustSummaryPairedEndOverCount() ;\n\n // Destory the locks used for map\n for (int i = 0; i < num_locks_for_map; ++i) {\n omp_destroy_lock(&map_locks[i]);\n }\n\n // Add cardinality information to summary metadata\n if (output_num_cache_slots_info) {\n for (auto curr_map: barcode_peak_map) {\n for (auto &pair: curr_map) {\n size_t curr_seed = pair.first;\n size_t est_num_slots = pair.second.compute_cardinality();\n\n mapping_writer.UpdateSummaryMetadata( \n curr_seed,\n SUMMARY_METADATA_CARDINALITY, \n est_num_slots);\n }\n }\n }\n\n mapping_writer.OutputSummaryMetadata(frip_est_params, output_num_cache_slots_info);\n reference.FinalizeLoading();\n if (mapping_parameters_.debug_cache) {mm_to_candidates_cache.PrintStats();}\n \n std::cerr << \"Total time: \" << GetRealTime() - real_start_time << \"s.\\n\";\n}\n\n} // namespace chromap\n\n#endif // CHROMAP_H_\n"
},
{
"path": "src/chromap_driver.cc",
"content": "#include \"chromap_driver.h\"\n\n#include \n\n#include \n#include \n#include \n#include \n\n#include \"chromap.h\"\n#include \"cxxopts.hpp\"\n\nnamespace chromap {\nnamespace {\n\nvoid AddIndexingOptions(cxxopts::Options &options) {\n options.add_options(\"Indexing\")(\"i,build-index\", \"Build index\")(\n \"min-frag-length\",\n \"Min fragment length for choosing k and w automatically [30]\",\n cxxopts::value(),\n \"INT\")(\"k,kmer\", \"Kmer length [17]\", cxxopts::value(), \"INT\")(\n \"w,window\", \"Window size [7]\", cxxopts::value(), \"INT\");\n}\n\nvoid AddMappingOptions(cxxopts::Options &options) {\n options.set_width(120).add_options(\"Mapping\")(\n \"preset\",\n \"Preset parameters for mapping reads (always applied before other \"\n \"options) []\\natac: mapping ATAC-seq/scATAC-seq reads\\nchip: mapping \"\n \"ChIP-seq reads\\nhic: mapping Hi-C reads\",\n cxxopts::value(),\n \"STR\")(\"split-alignment\", \"Allow split alignments\")(\n \"e,error-threshold\", \"Max # errors allowed to map a read [8]\",\n cxxopts::value(), \"INT\")\n //(\"A,match-score\", \"Match score [1]\", cxxopts::value(), \"INT\")\n //(\"B,mismatch-penalty\", \"Mismatch penalty [4]\", cxxopts::value(),\n //\"INT\")\n //(\"O,gap-open-penalties\", \"Gap open penalty [6,6]\",\n // cxxopts::value>(), \"INT[,INT]\")\n //(\"E,gap-extension-penalties\", \"Gap extension penalty [1,1]\",\n // cxxopts::value>(), \"INT[,INT]\")\n (\"s,min-num-seeds\", \"Min # seeds to try to map a read [2]\",\n cxxopts::value(),\n \"INT\")(\"f,max-seed-frequencies\",\n \"Max seed frequencies for a seed to be selected [500,1000]\",\n cxxopts::value>(), \"INT[,INT]\")\n //(\"n,max-num-best-mappings\", \"Only report n best mappings [1]\",\n // cxxopts::value(), \"INT\")\n (\"l,max-insert-size\",\n \"Max insert size, only for paired-end read mapping [1000]\",\n cxxopts::value(),\n \"INT\")(\"q,MAPQ-threshold\",\n \"Min MAPQ in range [0, 60] for mappings to be output [30]\",\n cxxopts::value(),\n \"INT\")(\"min-read-length\", \"Min read length [30]\",\n cxxopts::value(), \"INT\")\n //(\"multi-mapping-allocation-distance\", \"Uni-mappings within this distance\n // from any end of multi-mappings are used for allocation [0]\",\n // cxxopts::value(), \"INT\")\n //(\"multi-mapping-allocation-seed\", \"Seed for random number generator in\n // multi-mapping allocation [11]\", cxxopts::value(), \"INT\")\n //(\"drop-repetitive-reads\", \"Drop reads with too many best mappings\n //[500000]\", cxxopts::value(), \"INT\")\n (\"trim-adapters\", \"Try to trim adapters on 3'\")(\"remove-pcr-duplicates\",\n \"Remove PCR duplicates\")(\n \"remove-pcr-duplicates-at-bulk-level\",\n \"Remove PCR duplicates at bulk level for single cell data\")(\n \"remove-pcr-duplicates-at-cell-level\",\n \"Remove PCR duplicates at cell level for single cell data\")\n //(\"allocate-multi-mappings\", \"Allocate multi-mappings\")\n (\"Tn5-shift\", \"Perform Tn5 shift\")(\"low-mem\", \"Use low memory mode\")(\n \"bc-error-threshold\",\n \"Max Hamming distance allowed to correct a barcode [1]\",\n cxxopts::value(),\n \"INT\")(\"bc-probability-threshold\",\n \"Min probability to correct a barcode [0.9]\",\n cxxopts::value(),\n \"FLT\")(\"t,num-threads\", \"# threads for mapping [1]\",\n cxxopts::value(), \"INT\")\n (\"frip-est-params\", \"coefficients used for frip est calculation, separated by semi-colons\",\n cxxopts::value(), \"STR\")\n (\"turn-off-num-uniq-cache-slots\", \"turn off the output of number of cache slots in summary file\");\n}\n\nvoid AddInputOptions(cxxopts::Options &options) {\n options.add_options(\"Input\")(\"r,ref\", \"Reference file\",\n cxxopts::value(), \"FILE\")(\n \"x,index\", \"Index file\", cxxopts::value(), \"FILE\")(\n \"1,read1\", \"Single-end read files or paired-end read files 1\",\n cxxopts::value>(),\n \"FILE\")(\"2,read2\", \"Paired-end read files 2\",\n cxxopts::value>(),\n \"FILE\")(\"b,barcode\", \"Cell barcode files\",\n cxxopts::value>(), \"FILE\")(\n \"barcode-whitelist\", \"Cell barcode whitelist file\",\n cxxopts::value(),\n \"FILE\")(\"read-format\",\n \"Format for read files and barcode files [\\\"r1:0:-1,bc:0:-1\\\" \"\n \"as 10x Genomics single-end format]\",\n cxxopts::value(), \"STR\");\n}\n\nvoid AddOutputOptions(cxxopts::Options &options) {\n options.add_options(\"Output\")(\"o,output\", \"Output file\",\n cxxopts::value(), \"FILE\")\n //(\"p,matrix-output-prefix\", \"Prefix of matrix output files\",\n // cxxopts::value(), \"FILE\")\n (\"output-mappings-not-in-whitelist\",\n \"Output mappings with barcode not in the whitelist\")(\n \"chr-order\",\n \"Custom chromosome order file. If not specified, the order of \"\n \"reference sequences will be used\",\n cxxopts::value(),\n \"FILE\")(\"BED\", \"Output mappings in BED/BEDPE format\")(\n \"TagAlign\", \"Output mappings in TagAlign/PairedTagAlign format\")(\n \"SAM\", \"Output mappings in SAM format\")(\n \"pairs\",\n \"Output mappings in pairs format (defined by 4DN for HiC data)\")(\n \"pairs-natural-chr-order\",\n \"Custom chromosome order file for pairs flipping. If not specified, \"\n \"the custom chromosome order will be used\",\n cxxopts::value(),\n \"FILE\")(\"barcode-translate\",\n \"Convert barcode to the specified sequences during output\",\n cxxopts::value(), \"FILE\")(\n \"summary\",\n \"Summarize the mapping statistics at bulk or barcode level\",\n cxxopts::value(), \"FILE\");\n //(\"PAF\", \"Output mappings in PAF format (only for test)\");\n}\n\nvoid AddDevelopmentOptions(cxxopts::Options &options) {\n options.add_options(\"Development options\")(\"A,match-score\", \"Match score [1]\",\n cxxopts::value(), \"INT\")(\n \"B,mismatch-penalty\", \"Mismatch penalty [4]\", cxxopts::value(),\n \"INT\")(\"O,gap-open-penalties\", \"Gap open penalty [6,6]\",\n cxxopts::value>(), \"INT[,INT]\")(\n \"E,gap-extension-penalties\", \"Gap extension penalty [1,1]\",\n cxxopts::value>(),\n \"INT[,INT]\")(\"n,max-num-best-mappings\", \"Only report n best mappings [1]\",\n cxxopts::value(),\n \"INT\")(\"multi-mapping-allocation-distance\",\n \"Uni-mappings within this distance from any end of \"\n \"multi-mappings are used for allocation [0]\",\n cxxopts::value(), \"INT\")(\n \"multi-mapping-allocation-seed\",\n \"Seed for random number generator in multi-mapping allocation [11]\",\n cxxopts::value(), \"INT\")(\n \"drop-repetitive-reads\",\n \"Drop reads with too many best mappings [500000]\", cxxopts::value(),\n \"INT\")(\"allocate-multi-mappings\", \"Allocate multi-mappings\")(\n \"PAF\", \"Output mappings in PAF format (only for test)\")(\n \"skip-barcode-check\",\n \"Do not check whether too few barcodes are in the whitelist\")\n (\"cache-size\", \"number of cache entries [4000003]\", cxxopts::value(), \"INT\")\n (\"cache-update-param\", \"value used to control number of reads sampled [0.01]\", cxxopts::value(), \"FLT\")\n (\"debug-cache\", \"verbose output for debugging cache used in chromap\")\n (\"k-for-minhash\", \"number of values stored in each MinHash sketch [250]\", cxxopts::value(), \"INT\");\n}\n\nvoid AddPeakOptions(cxxopts::Options &options) {\n options.add_options(\"Peak\")(\"cell-by-bin\", \"Generate cell-by-bin matrix\")(\n \"bin-size\", \"Bin size to generate cell-by-bin matrix [5000]\",\n cxxopts::value(),\n \"INT\")(\"depth-cutoff\", \"Depth cutoff for peak calling [3]\",\n cxxopts::value(),\n \"INT\")(\"peak-min-length\", \"Min length of peaks to report [30]\",\n cxxopts::value(), \"INT\")(\n \"peak-merge-max-length\", \"Peaks within this length will be merged [30]\",\n cxxopts::value(), \"INT\");\n}\n\n// Return all file paths that match the input pattern.\nstd::vector GetMatchedFilePaths(const std::string &pattern) {\n glob_t glob_result;\n memset(&glob_result, 0, sizeof(glob_result));\n\n const int return_value =\n glob(pattern.c_str(), GLOB_TILDE, NULL, &glob_result);\n\n if (return_value != 0) {\n globfree(&glob_result);\n chromap::ExitWithMessage(\"glob() failed with return value \" +\n std::to_string(return_value) + \"\\n\");\n }\n\n std::vector matched_file_paths;\n matched_file_paths.reserve(glob_result.gl_pathc);\n for (size_t i = 0; i < glob_result.gl_pathc; ++i) {\n matched_file_paths.push_back(std::string(glob_result.gl_pathv[i]));\n std::cerr << matched_file_paths.back() << \"\\n\";\n }\n globfree(&glob_result);\n\n return matched_file_paths;\n}\n\n// Return all file paths that match the input patterns.\nstd::vector GetMatchedFilePaths(\n const std::vector &patterns) {\n std::vector all_matched_file_paths;\n for (const auto &pattern : patterns) {\n std::vector matched_file_paths = GetMatchedFilePaths(pattern);\n all_matched_file_paths.reserve(all_matched_file_paths.size() +\n matched_file_paths.size());\n all_matched_file_paths.insert(\n std::end(all_matched_file_paths),\n std::make_move_iterator(std::begin(matched_file_paths)),\n std::make_move_iterator(std::end(matched_file_paths)));\n }\n return all_matched_file_paths;\n}\n\n} // namespace\n\nvoid ChromapDriver::ParseArgsAndRun(int argc, char *argv[]) {\n cxxopts::Options options(\n \"chromap\", \"Fast alignment and preprocessing of chromatin profiles\");\n\n options.add_options()(\"v,version\", \"Print version\")(\"h,help\", \"Print help\");\n\n AddIndexingOptions(options);\n AddMappingOptions(options);\n\n // We don't support peak options for now.\n // AddPeakOptions(options);\n\n AddInputOptions(options);\n AddOutputOptions(options);\n\n AddDevelopmentOptions(options);\n\n auto result = options.parse(argc, argv);\n if (result.count(\"h\")) {\n std::cerr << options.help(\n {\"\", \"Indexing\", \"Mapping\", \"Peak\", \"Input\", \"Output\"});\n return;\n }\n if (result.count(\"v\")) {\n std::cerr << CHROMAP_VERSION << \"\\n\";\n return;\n }\n // Parameters and their default\n IndexParameters index_parameters;\n MappingParameters mapping_parameters;\n\n if (result.count(\"preset\")) {\n std::string read_type = result[\"preset\"].as();\n if (read_type == \"atac\") {\n std::cerr << \"Preset parameters for ATAC-seq/scATAC-seq are used.\\n\";\n mapping_parameters.max_insert_size = 2000;\n mapping_parameters.trim_adapters = true;\n mapping_parameters.remove_pcr_duplicates = true;\n mapping_parameters.remove_pcr_duplicates_at_bulk_level = false;\n mapping_parameters.Tn5_shift = true;\n mapping_parameters.mapping_output_format = MAPPINGFORMAT_BED;\n mapping_parameters.low_memory_mode = true;\n } else if (read_type == \"chip\") {\n std::cerr << \"Preset parameters for ChIP-seq are used.\\n\";\n mapping_parameters.max_insert_size = 2000;\n mapping_parameters.remove_pcr_duplicates = true;\n mapping_parameters.low_memory_mode = true;\n mapping_parameters.mapping_output_format = MAPPINGFORMAT_BED;\n } else if (read_type == \"hic\") {\n std::cerr << \"Preset parameters for Hi-C are used.\\n\";\n mapping_parameters.error_threshold = 4;\n mapping_parameters.mapq_threshold = 1;\n mapping_parameters.split_alignment = true;\n mapping_parameters.low_memory_mode = true;\n mapping_parameters.mapping_output_format = MAPPINGFORMAT_PAIRS;\n } else {\n chromap::ExitWithMessage(\"Unrecognized preset parameters \" + read_type +\n \"\\n\");\n }\n }\n // Optional parameters\n if (result.count(\"min-frag-length\")) {\n int min_fragment_length = result[\"min-frag-length\"].as();\n if (min_fragment_length <= 60) {\n index_parameters.kmer_size = 17;\n index_parameters.window_size = 7;\n } else if (min_fragment_length <= 80) {\n index_parameters.kmer_size = 19;\n index_parameters.window_size = 10;\n } else {\n index_parameters.kmer_size = 23;\n index_parameters.window_size = 11;\n }\n }\n if (result.count(\"k\")) {\n index_parameters.kmer_size = result[\"kmer\"].as();\n }\n if (result.count(\"w\")) {\n index_parameters.window_size = result[\"window\"].as();\n }\n if (result.count(\"e\")) {\n mapping_parameters.error_threshold = result[\"error-threshold\"].as();\n }\n if (result.count(\"A\")) {\n mapping_parameters.match_score = result[\"match-score\"].as();\n }\n if (result.count(\"B\")) {\n mapping_parameters.mismatch_penalty = result[\"mismatch-penalty\"].as();\n }\n if (result.count(\"O\")) {\n mapping_parameters.gap_open_penalties =\n result[\"gap-open-penalties\"].as>();\n }\n if (result.count(\"E\")) {\n mapping_parameters.gap_extension_penalties =\n result[\"gap-extension-penalties\"].as>();\n }\n if (result.count(\"s\")) {\n mapping_parameters.min_num_seeds_required_for_mapping =\n result[\"min-num-seeds\"].as();\n }\n if (result.count(\"f\")) {\n mapping_parameters.max_seed_frequencies =\n result[\"max-seed-frequencies\"].as>();\n }\n if (result.count(\"n\")) {\n mapping_parameters.max_num_best_mappings =\n result[\"max-num-best-mappings\"].as();\n }\n if (result.count(\"l\")) {\n mapping_parameters.max_insert_size = result[\"max-insert-size\"].as();\n }\n if (result.count(\"q\")) {\n mapping_parameters.mapq_threshold = result[\"MAPQ-threshold\"].as();\n }\n if (result.count(\"t\")) {\n mapping_parameters.num_threads = result[\"num-threads\"].as();\n }\n\n\n // check cache-related parameters\n if (result.count(\"cache-update-param\")) {\n mapping_parameters.cache_update_param = result[\"cache-update-param\"].as();\n if (mapping_parameters.cache_update_param < 0.0 || mapping_parameters.cache_update_param > 1.0){\n chromap::ExitWithMessage(\"cache update param is not approriate, must be in this range (0, 1]\");\n }\n } \n if (result.count(\"cache-size\")) {\n mapping_parameters.cache_size = result[\"cache-size\"].as();\n if (mapping_parameters.cache_size < 2000000 || mapping_parameters.cache_size > 15000000) {\n chromap::ExitWithMessage(\"cache size is not in appropriate range\\n\");\n }\n }\n if (result.count(\"debug-cache\")) {\n mapping_parameters.debug_cache = true;\n }\n if (result.count(\"frip-est-params\")) {\n mapping_parameters.frip_est_params = result[\"frip-est-params\"].as();\n }\n if (result.count(\"turn-off-num-uniq-cache-slots\")) {\n mapping_parameters.output_num_uniq_cache_slots = false;\n } \n if (result.count(\"k-for-minhash\")) {\n mapping_parameters.k_for_minhash = result[\"k-for-minhash\"].as();\n if (mapping_parameters.k_for_minhash < 1 || mapping_parameters.k_for_minhash >= 2000) {\n chromap::ExitWithMessage(\"Invalid paramter for size of MinHash sketch (--k-for-minhash)\");\n }\n }\n\n\n if (result.count(\"min-read-length\")) {\n mapping_parameters.min_read_length = result[\"min-read-length\"].as();\n }\n if (result.count(\"bc-error-threshold\")) {\n mapping_parameters.barcode_correction_error_threshold =\n result[\"bc-error-threshold\"].as();\n }\n if (result.count(\"bc-probability-threshold\")) {\n mapping_parameters.barcode_correction_probability_threshold =\n result[\"bc-probability-threshold\"].as();\n }\n if (result.count(\"multi-mapping-allocation-distance\")) {\n mapping_parameters.multi_mapping_allocation_distance =\n result[\"multi-mapping-allocation-distance\"].as();\n }\n if (result.count(\"multi-mapping-allocation-seed\")) {\n mapping_parameters.multi_mapping_allocation_seed =\n result[\"multi-mapping-allocation-seed\"].as();\n }\n if (result.count(\"drop-repetitive-reads\")) {\n mapping_parameters.drop_repetitive_reads =\n result[\"drop-repetitive-reads\"].as();\n }\n if (result.count(\"trim-adapters\")) {\n mapping_parameters.trim_adapters = true;\n }\n if (result.count(\"remove-pcr-duplicates\")) {\n mapping_parameters.remove_pcr_duplicates = true;\n }\n if (result.count(\"remove-pcr-duplicates-at-bulk-level\")) {\n mapping_parameters.remove_pcr_duplicates_at_bulk_level = true;\n }\n if (result.count(\"remove-pcr-duplicates-at-cell-level\")) {\n mapping_parameters.remove_pcr_duplicates_at_bulk_level = false;\n }\n if (result.count(\"allocate-multi-mappings\")) {\n mapping_parameters.allocate_multi_mappings = true;\n mapping_parameters.only_output_unique_mappings = false;\n }\n if (result.count(\"Tn5-shift\")) {\n mapping_parameters.Tn5_shift = true;\n }\n if (result.count(\"split-alignment\")) {\n mapping_parameters.split_alignment = true;\n }\n if (result.count(\"output-mappings-not-in-whitelist\")) {\n mapping_parameters.output_mappings_not_in_whitelist = true;\n }\n if (result.count(\"BED\")) {\n mapping_parameters.mapping_output_format = MAPPINGFORMAT_BED;\n }\n if (result.count(\"TagAlign\")) {\n mapping_parameters.mapping_output_format = MAPPINGFORMAT_TAGALIGN;\n }\n if (result.count(\"PAF\")) {\n mapping_parameters.mapping_output_format = MAPPINGFORMAT_PAF;\n }\n if (result.count(\"pairs\")) {\n mapping_parameters.mapping_output_format = MAPPINGFORMAT_PAIRS;\n }\n if (result.count(\"SAM\")) {\n mapping_parameters.mapping_output_format = MAPPINGFORMAT_SAM;\n }\n if (result.count(\"low-mem\")) {\n mapping_parameters.low_memory_mode = true;\n }\n if (result.count(\"cell-by-bin\")) {\n mapping_parameters.cell_by_bin = true;\n }\n if (result.count(\"bin-size\")) {\n mapping_parameters.bin_size = result[\"bin-size\"].as();\n }\n if (result.count(\"depth-cutoff\")) {\n mapping_parameters.depth_cutoff_to_call_peak =\n result[\"depth-cutoff\"].as();\n }\n if (result.count(\"peak-min-length\")) {\n mapping_parameters.peak_min_length = result[\"peak-min-length\"].as();\n }\n if (result.count(\"peak-merge-max-length\")) {\n mapping_parameters.peak_merge_max_length =\n result[\"peak-merge-max-length\"].as();\n }\n\n std::cerr << std::setprecision(2) << std::fixed;\n if (result.count(\"i\")) {\n if (result.count(\"r\")) {\n index_parameters.reference_file_path = result[\"ref\"].as();\n } else {\n chromap::ExitWithMessage(\"No reference specified!\");\n }\n if (result.count(\"o\")) {\n index_parameters.index_output_file_path =\n result[\"output\"].as();\n } else {\n chromap::ExitWithMessage(\"No output file specified!\");\n }\n std::cerr << \"Build index for the reference.\\n\";\n std::cerr << \"Kmer length: \" << index_parameters.kmer_size\n << \", window size: \" << index_parameters.window_size << \"\\n\";\n std::cerr << \"Reference file: \" << index_parameters.reference_file_path\n << \"\\n\";\n std::cerr << \"Output file: \" << index_parameters.index_output_file_path\n << \"\\n\";\n chromap::Chromap chromap_for_indexing(index_parameters);\n chromap_for_indexing.ConstructIndex();\n } else if (result.count(\"1\")) {\n std::cerr << \"Start to map reads.\\n\";\n if (result.count(\"r\")) {\n mapping_parameters.reference_file_path = result[\"ref\"].as();\n } else {\n chromap::ExitWithMessage(\"No reference specified!\");\n }\n if (result.count(\"o\")) {\n mapping_parameters.mapping_output_file_path =\n result[\"output\"].as();\n } else {\n chromap::ExitWithMessage(\"No output file specified!\");\n }\n if (result.count(\"x\")) {\n mapping_parameters.index_file_path = result[\"index\"].as();\n } else {\n chromap::ExitWithMessage(\"No index file specified!\");\n }\n if (result.count(\"1\")) {\n mapping_parameters.read_file1_paths =\n GetMatchedFilePaths(result[\"read1\"].as>());\n } else {\n chromap::ExitWithMessage(\"No read file specified!\");\n }\n if (result.count(\"2\")) {\n mapping_parameters.read_file2_paths =\n GetMatchedFilePaths(result[\"read2\"].as>());\n }\n\n if (result.count(\"b\")) {\n mapping_parameters.is_bulk_data = false;\n mapping_parameters.barcode_file_paths =\n GetMatchedFilePaths(result[\"barcode\"].as>());\n if (result.count(\"barcode-whitelist\") == 0) {\n std::cerr << \"WARNING: there are input barcode files but a barcode \"\n \"whitelist file is missing!\\n\";\n }\n }\n\n if (result.count(\"barcode-whitelist\")) {\n if (mapping_parameters.is_bulk_data) {\n chromap::ExitWithMessage(\n \"No barcode file specified but the barcode whitelist file is \"\n \"given!\");\n }\n mapping_parameters.barcode_whitelist_file_path =\n result[\"barcode-whitelist\"].as();\n }\n\n if (result.count(\"p\")) {\n mapping_parameters.matrix_output_prefix =\n result[\"matrix-output-prefix\"].as();\n if (mapping_parameters.is_bulk_data) {\n chromap::ExitWithMessage(\n \"No barcode file specified but asked to output matrix files!\");\n }\n }\n if (result.count(\"read-format\")) {\n mapping_parameters.read_format = result[\"read-format\"].as();\n }\n\n if (result.count(\"chr-order\")) {\n mapping_parameters.custom_rid_order_file_path =\n result[\"chr-order\"].as