Repository: openai/tiktoken
Branch: main
Commit: 6ec814981227
Files: 29
Total size: 112.3 KB
Directory structure:
gitextract_vtvi0nnb/
├── .github/
│ └── workflows/
│ └── build_wheels.yml
├── .gitignore
├── CHANGELOG.md
├── Cargo.toml
├── LICENSE
├── MANIFEST.in
├── README.md
├── pyproject.toml
├── scripts/
│ ├── benchmark.py
│ ├── redact.py
│ └── wheel_download.py
├── setup.py
├── src/
│ ├── lib.rs
│ └── py.rs
├── tests/
│ ├── __init__.py
│ ├── test_encoding.py
│ ├── test_helpers.py
│ ├── test_misc.py
│ ├── test_offsets.py
│ ├── test_pickle.py
│ └── test_simple_public.py
├── tiktoken/
│ ├── __init__.py
│ ├── _educational.py
│ ├── core.py
│ ├── load.py
│ ├── model.py
│ ├── py.typed
│ └── registry.py
└── tiktoken_ext/
└── openai_public.py
================================================
FILE CONTENTS
================================================
================================================
FILE: .github/workflows/build_wheels.yml
================================================
name: Build wheels
on: [push, pull_request, workflow_dispatch]
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
jobs:
build_wheels:
name: py${{ matrix.python-version }} on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
# cibuildwheel builds linux wheels inside a manylinux container
# it also takes care of procuring the correct python version for us
os: [ubuntu-latest, windows-latest, macos-latest]
python-version: [39, 310, 311, 312, 313, 313t, 314, 314t]
steps:
- uses: actions/checkout@v6
- uses: pypa/cibuildwheel@v3.1.4
env:
CIBW_BUILD: "cp${{ matrix.python-version}}-*"
CIBW_ENABLE: cpython-freethreading
- uses: actions/upload-artifact@v6
with:
name: cibw-wheels-${{ matrix.os }}-${{ strategy.job-index }}
path: ./wheelhouse/*.whl
build_wheels_aarch64:
name: py${{ matrix.python-version }} on ${{ matrix.os }} (aarch64)
runs-on: ${{ matrix.os }}
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
os: [ubuntu-24.04-arm]
python-version: [39, 310, 311, 312, 313, 313t, 314, 314t]
steps:
- uses: actions/checkout@v6
- name: Build wheels
uses: pypa/cibuildwheel@v3.1.4
env:
CIBW_BUILD: "cp${{ matrix.python-version}}-*"
CIBW_ARCHS: aarch64
CIBW_BUILD_VERBOSITY: 3
# https://github.com/rust-lang/cargo/issues/10583
CIBW_ENVIRONMENT_LINUX: PATH="$PATH:$HOME/.cargo/bin" CARGO_NET_GIT_FETCH_WITH_CLI=true
CIBW_ENABLE: cpython-freethreading
- uses: actions/upload-artifact@v6
with:
name: cibw-wheels-aarch64-${{ matrix.os }}-${{ strategy.job-index }}
path: ./wheelhouse/*.whl
build_sdist:
name: sdist
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- uses: actions/checkout@v6
- uses: actions/setup-python@v6
name: Install Python
with:
python-version: "3.9"
- name: Run check-manifest
run: |
pip install check-manifest
check-manifest -v
- name: Build sdist
run: |
pip install --upgrade build
python -m build --sdist
- uses: actions/upload-artifact@v6
with:
name: cibw-wheels-${{ matrix.os }}-${{ strategy.job-index }}
path: ./dist/*.tar.gz
join_artifacts:
name: Join artifacts
runs-on: ubuntu-latest
needs: [build_wheels, build_wheels_aarch64, build_sdist]
steps:
- name: Merge artifacts
uses: actions/upload-artifact/merge@v4
with:
name: cibw-wheels
pattern: cibw-wheels-*
delete-merged: true
================================================
FILE: .gitignore
================================================
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# Environments
.env
.venv
# Tools
.mypy_cache
.coverage
.hypothesis
htmlcov
# General
.DS_Store
Cargo.lock
target/
================================================
FILE: CHANGELOG.md
================================================
# Changelog
This is the changelog for the open source version of tiktoken.
## [v0.12.0]
- Build wheels for Python 3.14
- Build musllinux aarch64 wheels
- Support for free-threaded Python
- Update version of `pyo3` and `rustc-hash`
- Avoid use of `blobfile` for reading local files
- Recognise `gpt-5` model identifier
- Minor performance improvement for file reading
## [v0.11.0]
- Support for `GPT-5`
- Update version of `pyo3`
- Use new Rust edition
- Fix special token handling in `encode_to_numpy`
- Better error handling
- Improvements to private APIs
## [v0.10.0]
- Support for newer models
- Improvements to private APIs
## [v0.9.0]
- Support for `o1` and `o3` models
- Better error messages when loading invalid vocabulary files
- Support for encoding to numpy arrays
- Delayed imports when not strictly necessary
## [v0.8.0]
- Support for `o1-` and `chatgpt-4o-` models
- Build wheels for Python 3.13
- Add possessive quantifiers to limit backtracking in regular expressions, thanks to @l0rinc!
- Provide a better error message and type for invalid token decode
- Permit tuples in type hints
- Better error message for passing invalid input to `get_encoding`
- Better error messages during plugin loading
- Add a `__version__` attribute
- Update versions of `pyo3`, `regex`, `fancy-regex`
- Drop support for Python 3.8
## [v0.7.0]
- Support for `gpt-4o`
- Performance improvements
## [v0.6.0]
- Optimise regular expressions for a 20% performance improvement, thanks to @paplorinc!
- Add `text-embedding-3-*` models to `encoding_for_model`
- Check content hash for downloaded files
- Allow pickling `Encoding` objects. Registered `Encoding` will be pickled by reference
- Workaround PyO3 bug for frozenset conversion
Thank you to @paplorinc, @mdwelsh, @Praneet460!
## [v0.5.2]
- Build wheels for Python 3.12
- Update version of PyO3 to allow multiple imports
- Avoid permission errors when using default cache logic
## [v0.5.1]
- Add `encoding_name_for_model`, undo some renames to variables that are implementation details
## [v0.5.0]
- Add `tiktoken._educational` submodule to better document how byte pair encoding works
- Ensure `encoding_for_model` knows about several new models
- Add `decode_with_offets`
- Better error for failures with the plugin mechanism
- Make more tests public
- Update versions of dependencies
## [v0.4.0]
- Add `decode_batch` and `decode_bytes_batch`
- Improve error messages and handling
## [v0.3.3]
- `tiktoken` will now make a best effort attempt to replace surrogate pairs with the corresponding
Unicode character and will replace lone surrogates with the Unicode replacement character.
## [v0.3.2]
- Add encoding for GPT-4
## [v0.3.1]
- Build aarch64 wheels
- Make `blobfile` an optional dependency
Thank you to @messense for the environment variable that makes cargo not OOM under emulation!
## [v0.3.0]
- Improve performance by 5-20%; thank you to @nistath!
- Add `gpt-3.5-turbo` models to `encoding_for_model`
- Add prefix matching to `encoding_for_model` to better support future model versions
- Fix a bug in the README instructions on extending tiktoken
- Update the set of available encodings
- Add packaging metadata
## [v0.2.0]
- Add `tiktoken.encoding_for_model` to get the encoding for a specific model
- Improve portability of caching logic
Thank you to @fritzo, @arvid220u, @khanhvu207, @henriktorget for various small corrections
## [v0.1.2]
- Avoid use of `blobfile` for public files
- Add support for Python 3.8
- Add py.typed
- Improve the public tests
## [v0.1.1]
- Initial release
================================================
FILE: Cargo.toml
================================================
[package]
name = "tiktoken"
version = "0.12.0"
edition = "2024"
[lib]
name = "tiktoken"
crate-type = ["cdylib", "rlib"]
[features]
default = []
python = [
"pyo3",
]
[dependencies]
pyo3 = { version = "0.27.2", default-features = false, features = [
"extension-module",
"macros",
], optional = true }
# tiktoken dependencies
fancy-regex = "0.17.0"
regex = "1.10.3"
rustc-hash = "2"
bstr = "1.5.0"
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2022 OpenAI, Shantanu Jain
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: MANIFEST.in
================================================
include *.svg
include *.toml
include *.md
include Makefile
global-include py.typed
recursive-include scripts *.py
recursive-include tests *.py
recursive-include src *.rs
================================================
FILE: README.md
================================================
# ⏳ tiktoken
tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with
OpenAI's models.
```python
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
assert enc.decode(enc.encode("hello world")) == "hello world"
# To get the tokeniser corresponding to a specific model in the OpenAI API:
enc = tiktoken.encoding_for_model("gpt-4o")
```
The open source version of `tiktoken` can be installed from [PyPI](https://pypi.org/project/tiktoken):
```
pip install tiktoken
```
The tokeniser API is documented in `tiktoken/core.py`.
Example code using `tiktoken` can be found in the
[OpenAI Cookbook](https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb).
## Performance
`tiktoken` is between 3-6x faster than a comparable open source tokeniser:
Performance measured on 1GB of text using the GPT-2 tokeniser, using `GPT2TokenizerFast` from
`tokenizers==0.13.2`, `transformers==4.24.0` and `tiktoken==0.2.0`.
## Getting help
Please post questions in the [issue tracker](https://github.com/openai/tiktoken/issues).
If you work at OpenAI, make sure to check the internal documentation or feel free to contact
@shantanu.
## What is BPE anyway?
Language models don't see text like you and I, instead they see a sequence of numbers (known as tokens).
Byte pair encoding (BPE) is a way of converting text into tokens. It has a couple desirable
properties:
1) It's reversible and lossless, so you can convert tokens back into the original text
2) It works on arbitrary text, even text that is not in the tokeniser's training data
3) It compresses the text: the token sequence is shorter than the bytes corresponding to the
original text. On average, in practice, each token corresponds to about 4 bytes.
4) It attempts to let the model see common subwords. For instance, "ing" is a common subword in
English, so BPE encodings will often split "encoding" into tokens like "encod" and "ing"
(instead of e.g. "enc" and "oding"). Because the model will then see the "ing" token again and
again in different contexts, it helps models generalise and better understand grammar.
`tiktoken` contains an educational submodule that is friendlier if you want to learn more about
the details of BPE, including code that helps visualise the BPE procedure:
```python
from tiktoken._educational import *
# Train a BPE tokeniser on a small amount of text
enc = train_simple_encoding()
# Visualise how the GPT-4 encoder encodes text
enc = SimpleBytePairEncoding.from_tiktoken("cl100k_base")
enc.encode("hello world aaaaaaaaaaaa")
```
## Extending tiktoken
You may wish to extend `tiktoken` to support new encodings. There are two ways to do this.
**Create your `Encoding` object exactly the way you want and simply pass it around.**
```python
cl100k_base = tiktoken.get_encoding("cl100k_base")
# In production, load the arguments directly instead of accessing private attributes
# See openai_public.py for examples of arguments for specific encodings
enc = tiktoken.Encoding(
# If you're changing the set of special tokens, make sure to use a different name
# It should be clear from the name what behaviour to expect.
name="cl100k_im",
pat_str=cl100k_base._pat_str,
mergeable_ranks=cl100k_base._mergeable_ranks,
special_tokens={
**cl100k_base._special_tokens,
"<|im_start|>": 100264,
"<|im_end|>": 100265,
}
)
```
**Use the `tiktoken_ext` plugin mechanism to register your `Encoding` objects with `tiktoken`.**
This is only useful if you need `tiktoken.get_encoding` to find your encoding, otherwise prefer
option 1.
To do this, you'll need to create a namespace package under `tiktoken_ext`.
Layout your project like this, making sure to omit the `tiktoken_ext/__init__.py` file:
```
my_tiktoken_extension
├── tiktoken_ext
│ └── my_encodings.py
└── setup.py
```
`my_encodings.py` should be a module that contains a variable named `ENCODING_CONSTRUCTORS`.
This is a dictionary from an encoding name to a function that takes no arguments and returns
arguments that can be passed to `tiktoken.Encoding` to construct that encoding. For an example, see
`tiktoken_ext/openai_public.py`. For precise details, see `tiktoken/registry.py`.
Your `setup.py` should look something like this:
```python
from setuptools import setup, find_namespace_packages
setup(
name="my_tiktoken_extension",
packages=find_namespace_packages(include=['tiktoken_ext*']),
install_requires=["tiktoken"],
...
)
```
Then simply `pip install ./my_tiktoken_extension` and you should be able to use your
custom encodings! Make sure **not** to use an editable install.
================================================
FILE: pyproject.toml
================================================
[project]
name = "tiktoken"
version = "0.12.0"
description = "tiktoken is a fast BPE tokeniser for use with OpenAI's models"
readme = "README.md"
license = { file = "LICENSE" }
authors = [{ name = "Shantanu Jain" }, { email = "shantanu@openai.com" }]
dependencies = ["regex", "requests"]
optional-dependencies = { blobfile = ["blobfile>=3"] }
requires-python = ">=3.9"
[project.urls]
homepage = "https://github.com/openai/tiktoken"
repository = "https://github.com/openai/tiktoken"
changelog = "https://github.com/openai/tiktoken/blob/main/CHANGELOG.md"
[build-system]
build-backend = "setuptools.build_meta"
requires = ["setuptools>=62.4", "wheel", "setuptools-rust>=1.5.2"]
[tool.cibuildwheel]
build-frontend = "build"
build-verbosity = 1
linux.before-all = "curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y"
linux.environment = { PATH = "$PATH:$HOME/.cargo/bin" }
macos.before-all = "rustup target add aarch64-apple-darwin x86_64-apple-darwin"
macos.environment = { MACOSX_DEPLOYMENT_TARGET = "10.12" }
skip = [
"*-manylinux_i686",
"*-musllinux_i686",
"*-win32",
]
macos.archs = ["x86_64", "arm64"]
# When cross-compiling on Intel, it is not possible to test arm64 wheels.
# Warnings will be silenced with following CIBW_TEST_SKIP
test-skip = "*-macosx_arm64"
before-test = "pip install pytest hypothesis"
test-command = "pytest {project}/tests --import-mode=append"
================================================
FILE: scripts/benchmark.py
================================================
import base64
import functools
import gzip
import json
import os
import random
import time
from typing import Any, cast
import blobfile
import tiktoken
def benchmark_batch(documents: list[str]) -> None:
num_threads = int(os.environ["RAYON_NUM_THREADS"])
num_bytes = sum(map(len, map(str.encode, documents)))
print(f"num_threads: {num_threads}, num_bytes: {num_bytes}")
enc = tiktoken.get_encoding("gpt2")
enc.encode("warmup")
start = time.perf_counter_ns()
enc.encode_ordinary_batch(documents, num_threads=num_threads)
end = time.perf_counter_ns()
print(f"tiktoken \t{num_bytes / (end - start) * 1e9} bytes / s")
import transformers
hf_enc = cast(Any, transformers).GPT2TokenizerFast.from_pretrained("gpt2")
hf_enc.model_max_length = 1e30 # silence!
hf_enc.encode("warmup")
start = time.perf_counter_ns()
hf_enc(documents)
end = time.perf_counter_ns()
print(f"huggingface \t{num_bytes / (end - start) * 1e9} bytes / s")
================================================
FILE: scripts/redact.py
================================================
import argparse
import re
import subprocess
from pathlib import Path
def redact_file(path: Path, dry_run: bool) -> None:
if not path.exists() or path.is_dir():
return
text = path.read_text()
if not text:
return
first_line = text.splitlines()[0]
if "redact" in first_line:
if not dry_run:
path.unlink()
print(f"Deleted {path}")
return
pattern = "|".join(
r" *" + re.escape(x)
for x in [
"# ===== redact-beg =====\n",
"# ===== redact-end =====\n",
"<!--- redact-beg -->\n",
"<!--- redact-end -->\n",
]
)
if re.search(pattern, text):
redacted_text = "".join(re.split(pattern, text)[::2])
if not dry_run:
path.write_text(redacted_text)
print(f"Redacted {path}")
return
print(f"Skipped {path}")
def redact(dry_run: bool) -> None:
tiktoken_root = Path(__file__).parent.parent
assert tiktoken_root.name == "tiktoken"
assert (tiktoken_root / "pyproject.toml").exists()
try:
output = subprocess.check_output(["git", "ls-files"], cwd=tiktoken_root, text=True)
paths = [Path(p) for p in output.splitlines()]
except subprocess.CalledProcessError:
paths = list(tiktoken_root.glob("**/*"))
for path in paths:
redact_file(path, dry_run=dry_run)
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--dry-run", type=lambda x: not x or x[0].lower() != "f", default=True)
args = parser.parse_args()
redact(args.dry_run)
if args.dry_run:
print("Dry run, use --dry-run=false to actually redact files")
if __name__ == "__main__":
main()
================================================
FILE: scripts/wheel_download.py
================================================
import argparse
import zipfile
from pathlib import Path
import requests
def download_artifacts(token, owner, repo, run_id, output_dir):
headers = {"Authorization": f"token {token}", "Accept": "application/vnd.github.v3+json"}
# Get list of artifacts
artifacts_url = f"https://api.github.com/repos/{owner}/{repo}/actions/runs/{run_id}/artifacts"
response = requests.get(artifacts_url, headers=headers)
response.raise_for_status()
artifacts = response.json()["artifacts"]
if not artifacts:
print(f"No artifacts found for run ID: {run_id}")
return
output_dir = Path(output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
print(f"Found {len(artifacts)} artifacts")
for artifact in artifacts:
name = artifact["name"]
download_url = artifact["archive_download_url"]
print(f"Downloading {name}...")
response = requests.get(download_url, headers=headers, stream=True)
response.raise_for_status()
temp_zip = output_dir / f"{name}.zip"
with open(temp_zip, "wb") as f:
for chunk in response.iter_content(chunk_size=8192):
f.write(chunk)
with zipfile.ZipFile(temp_zip, "r") as zip_ref:
zip_ref.extractall(output_dir)
temp_zip.unlink()
print(f"Downloaded and extracted {name}")
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Download artifacts from a GitHub Actions run")
parser.add_argument("--token", required=True, help="GitHub Personal Access Token")
parser.add_argument("--owner", required=True, help="Repository owner")
parser.add_argument("--repo", required=True, help="Repository name")
parser.add_argument("--run-id", required=True, help="Workflow run ID")
parser.add_argument(
"--output-dir", default="artifacts", help="Output directory for downloaded artifacts"
)
args = parser.parse_args()
download_artifacts(args.token, args.owner, args.repo, args.run_id, args.output_dir)
================================================
FILE: setup.py
================================================
from setuptools import setup
from setuptools_rust import Binding, RustExtension
setup(
name="tiktoken",
rust_extensions=[
RustExtension(
"tiktoken._tiktoken",
binding=Binding.PyO3,
# Between our use of editable installs and wanting to use Rust for performance sensitive
# code, it makes sense to just always use --release
debug=False,
features=["python"],
)
],
package_data={"tiktoken": ["py.typed"]},
packages=["tiktoken", "tiktoken_ext"],
zip_safe=False,
)
================================================
FILE: src/lib.rs
================================================
use std::collections::HashSet;
use std::num::NonZeroU64;
use std::thread;
use fancy_regex::Regex;
#[cfg(feature = "python")]
use pyo3::prelude::*;
use rustc_hash::FxHashMap as HashMap;
#[cfg(feature = "python")]
mod py;
pub type Rank = u32;
use std::collections::BinaryHeap;
#[derive(Eq, PartialEq, Clone, Copy)]
struct Merge {
start: usize,
rank: Rank,
}
impl Ord for Merge {
#[inline]
fn cmp(&self, other: &Self) -> std::cmp::Ordering {
other
.rank
.cmp(&self.rank)
.then_with(|| other.start.cmp(&self.start))
}
}
impl PartialOrd for Merge {
fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {
Some(self.cmp(other))
}
}
struct State {
prev: usize,
end: usize,
next_end: usize,
next_rank: Rank,
cur_rank: Rank,
}
fn _byte_pair_merge_large(ranks: &HashMap<Vec<u8>, Rank>, piece: &[u8]) -> Vec<Rank> {
let mut state = Vec::with_capacity(piece.len());
state.push(State {
prev: usize::MAX,
end: 1,
next_end: 2,
next_rank: Rank::MAX,
cur_rank: Rank::MAX,
});
let mut heap = BinaryHeap::with_capacity(piece.len());
for i in 0..piece.len() - 1 {
if let Some(&rank) = ranks.get(&piece[i..i + 2]) {
heap.push(Merge { start: i, rank });
state[i].next_rank = rank;
}
// note this is happening offset by 1
state.push(State {
prev: i,
end: i + 2,
next_end: i + 3,
next_rank: Rank::MAX,
cur_rank: Rank::MAX,
});
}
// Repeatedly find the valid merge with smallest rank. We merge the (left) token that
// starts at `start` and ends at `state[start].end` with the (right) token that starts at
// `state[start].end` and ends at `state[start].next_end`. We invalidate the old merges
// (the ones that started at `state[start].end` and ended at `state[start]`) and add the two
// new potential merges to the heap.
let potential_merge = {
#[inline(always)]
|state: &mut Vec<State>,
heap: &mut BinaryHeap<Merge>,
start: usize,
next_end_item: usize| {
state[start].next_end = next_end_item;
state[start].next_rank = Rank::MAX; // Always invalidate the old merge
if next_end_item <= piece.len()
&& let Some(&rank) = ranks.get(&piece[start..next_end_item])
{
// We have a valid potential merge!
heap.push(Merge { start, rank });
state[start].next_rank = rank;
}
}
};
while let Some(left) = heap.pop() {
if left.rank == Rank::MAX {
break;
}
if left.rank != state[left.start].next_rank {
continue; // This merge was invalidated, ignore it
}
let left_start = left.start;
let right_start = state[left_start].end;
let right_end = state[left_start].next_end;
debug_assert!(right_end == state[right_start].end);
let right_next_end = state[right_start].next_end;
// Merge left and right into a single token
state[left_start].cur_rank = state[left_start].next_rank;
state[left_start].end = right_end;
potential_merge(&mut state, &mut heap, left_start, right_next_end);
if right_end < state.len() {
state[right_end].prev = left_start;
}
// Update the merge that ends at left_start
if left_start > 0 {
let prev_start = state[left_start].prev;
potential_merge(&mut state, &mut heap, prev_start, right_end);
}
// Invalidate the merge starting at right_start, so we ignore it when it comes off the heap
state[right_start].next_rank = Rank::MAX;
}
let mut result = Vec::new();
let mut i = 0;
while i < state.len() {
if state[i].cur_rank != Rank::MAX {
result.push(state[i].cur_rank);
} else {
result.push(ranks[&piece[i..state[i].end]]);
}
i = state[i].end;
}
result
}
fn _byte_pair_merge(ranks: &HashMap<Vec<u8>, Rank>, piece: &[u8]) -> Vec<(usize, Rank)> {
// This is a vector of (start, rank).
// The rank is of the pair starting at position start.
let mut parts = Vec::with_capacity(piece.len() + 1);
// Note that we hash bytes when indexing into `ranks`, not token pairs. As long as we train BPE
// the way we currently do, this is equivalent. An easy way to break this would be to decouple
// merge priority from token index or to prevent specific token merges.
let mut min_rank: (Rank, usize) = (Rank::MAX, usize::MAX);
for i in 0..piece.len() - 1 {
let rank = *ranks.get(&piece[i..i + 2]).unwrap_or(&Rank::MAX);
if rank < min_rank.0 {
min_rank = (rank, i);
}
parts.push((i, rank));
}
parts.push((piece.len() - 1, Rank::MAX));
parts.push((piece.len(), Rank::MAX));
let get_rank = {
#[inline(always)]
|parts: &Vec<(usize, Rank)>, i: usize| {
if (i + 3) < parts.len() {
// Similar to `piece[i..i + 2]` above. The +3 is because we haven't yet deleted
// parts[i + 1], see comment in the main loop.
*ranks
.get(&piece[parts[i].0..parts[i + 3].0])
.unwrap_or(&Rank::MAX)
} else {
Rank::MAX
}
}
};
// If you have n parts and m merges, this does O(mn) work.
// We could do something with a heap and do O(m log n) work.
// n is often very small so considerations like cache-locality outweigh the algorithmic
// complexity downsides of the `parts` vector.
while min_rank.0 != Rank::MAX {
let i = min_rank.1;
// Update parts[i] and parts[i - 1] before removing parts[i + 1], since
// `parts.remove(i + 1)` will thrash the cache.
if i > 0 {
parts[i - 1].1 = get_rank(&parts, i - 1);
}
parts[i].1 = get_rank(&parts, i);
parts.remove(i + 1);
min_rank = (Rank::MAX, usize::MAX);
for (i, &(_, rank)) in parts[..parts.len() - 1].iter().enumerate() {
if rank < min_rank.0 {
min_rank = (rank, i);
}
}
}
parts
}
pub fn byte_pair_encode(piece: &[u8], ranks: &HashMap<Vec<u8>, Rank>) -> Vec<Rank> {
let piece_len = piece.len();
if piece_len == 1 {
return vec![ranks[piece]];
}
if piece_len < 100 {
return _byte_pair_merge(ranks, piece)
.windows(2)
.map(|part| ranks[&piece[part[0].0..part[1].0]])
.collect();
}
_byte_pair_merge_large(ranks, piece)
}
pub fn byte_pair_split<'a>(piece: &'a [u8], ranks: &HashMap<Vec<u8>, Rank>) -> Vec<&'a [u8]> {
assert!(piece.len() > 1);
_byte_pair_merge(ranks, piece)
.windows(2)
.map(|part| &piece[part[0].0..part[1].0])
.collect()
}
// Various performance notes:
//
// Regex
// =====
// Most of the time is spent in regex. The easiest way to speed this up is by using less fancy
// regex features. For instance, using a regex parse-able by `regex` crate is 3x faster than
// the usual regex we use.
//
// However, given that we're using a regex parse-able by `regex`, there isn't much difference
// between using the `regex` crate and using the `fancy_regex` crate.
//
// There is an important interaction between threading, `regex` and `fancy_regex`.
// When using `fancy_regex`, we hit `regex.find_at`. It turns out that this causes contention on
// some mutable scratch space inside of `regex`. This absolutely kills performance. When using plain
// old `regex`, we don't hit this, because `find_iter` has a different code path.
// Related: https://github.com/rust-lang/regex/blob/master/PERFORMANCE.md
// Anyway, the way we get around this is with having a (mostly) thread local clone of the regex for
// each thread.
//
// Threading
// =========
// I tried using `rayon`. It wasn't really faster than using Python threads and releasing the GIL.
// So goodbye `rayon`! Let thread count etc be in control of our Python users.
//
// Caching
// =======
// The reference tokeniser has an lru cache over the equivalent of `byte_pair_encode`.
// Originally, we had one too! Without it, we were only vaguely faster than Python.
// I used an RWLock to protect the cache. This didn't seem to hurt single threaded performance
// noticeably, but it did affect multi-threaded performance. Weirdly, it seemed to affect
// multi-threaded performance even when I only had readers (maybed I messed something up?).
// Anyway, I realised that we could get rid of the cache, if we treat the set of tokens as a cache!
// These are exactly the set or merges that are likely to be hot. And now we don't have to think
// about interior mutability, memory use, or cloning.
//
// Hashing
// =======
// We use FxHashMap instead of the standard HashMap. This is maybe like a 5-10% win?
// The current implementation ends up doing a lot of hashing of bytes. In theory, this could be made
// to be hashing of two-tuples of ints, which looks like it may also be a couple percent faster.
struct FakeThreadId(NonZeroU64);
fn hash_current_thread() -> usize {
// It's easier to use unsafe than to use nightly. Rust has this nice u64 thread id counter
// that works great for our use case of avoiding collisions in our array. Unfortunately,
// it's private. However, there are only so many ways you can layout a u64, so just transmute
// https://github.com/rust-lang/rust/issues/67939
const _: [u8; 8] = [0; std::mem::size_of::<std::thread::ThreadId>()];
const _: [u8; 8] = [0; std::mem::size_of::<FakeThreadId>()];
let x = unsafe {
std::mem::transmute::<std::thread::ThreadId, FakeThreadId>(thread::current().id()).0
};
u64::from(x) as usize
}
#[derive(Debug, Clone)]
pub struct DecodeKeyError {
pub token: Rank,
}
impl std::fmt::Display for DecodeKeyError {
fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
write!(f, "Invalid token for decoding: {}", self.token)
}
}
impl std::error::Error for DecodeKeyError {}
#[derive(Debug, Clone)]
pub struct DecodeError {
pub message: String,
}
impl std::fmt::Display for DecodeError {
fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
write!(f, "Could not decode tokens: {}", self.message)
}
}
impl std::error::Error for DecodeError {}
#[derive(Debug, Clone)]
pub struct EncodeError {
pub message: String,
}
impl std::fmt::Display for EncodeError {
fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
write!(f, "Could not encode string: {}", self.message)
}
}
impl std::error::Error for EncodeError {}
const MAX_NUM_THREADS: usize = 128;
#[cfg_attr(feature = "python", pyclass(frozen))]
#[derive(Clone)]
pub struct CoreBPE {
encoder: HashMap<Vec<u8>, Rank>,
special_tokens_encoder: HashMap<String, Rank>,
decoder: HashMap<Rank, Vec<u8>>,
special_tokens_decoder: HashMap<Rank, Vec<u8>>,
regex_tls: Vec<Regex>,
special_regex_tls: Vec<Regex>,
sorted_token_bytes: Vec<Vec<u8>>,
}
impl CoreBPE {
fn _get_tl_regex(&self) -> &Regex {
// See performance notes above for what this is about
// It's also a little janky, please make a better version of it!
// However, it's nice that this doesn't leak memory to short-lived threads
&self.regex_tls[hash_current_thread() % MAX_NUM_THREADS]
}
fn _get_tl_special_regex(&self) -> &Regex {
&self.special_regex_tls[hash_current_thread() % MAX_NUM_THREADS]
}
/// Decodes tokens into a list of bytes.
///
/// The bytes are not gauranteed to be a valid utf-8 string.
fn decode_bytes(&self, tokens: &[Rank]) -> Result<Vec<u8>, DecodeKeyError> {
let mut ret = Vec::with_capacity(tokens.len() * 2);
for &token in tokens {
let token_bytes = match self.decoder.get(&token) {
Some(bytes) => bytes,
None => self
.special_tokens_decoder
.get(&token)
.ok_or(DecodeKeyError { token })?,
};
ret.extend(token_bytes);
}
Ok(ret)
}
pub fn encode_ordinary(&self, text: &str) -> Vec<Rank> {
// This is the core of the encoding logic; the other functions in here
// just make things complicated :-)
let regex = self._get_tl_regex();
let mut ret = vec![];
for mat in regex.find_iter(text) {
let piece = mat.unwrap().as_str().as_bytes();
match self.encoder.get(piece) {
Some(token) => ret.push(*token),
None => ret.extend(&byte_pair_encode(piece, &self.encoder)),
}
}
ret
}
pub fn encode(
&self,
text: &str,
allowed_special: &HashSet<&str>,
) -> Result<(Vec<Rank>, usize), EncodeError> {
let special_regex = self._get_tl_special_regex();
let regex = self._get_tl_regex();
let mut ret = vec![];
let mut start = 0;
let mut last_piece_token_len = 0;
loop {
let mut next_special;
let mut start_find = start;
loop {
// Find the next allowed special token, if any
next_special = special_regex.find_from_pos(text, start_find).unwrap();
match next_special {
Some(m) => {
if allowed_special.contains(&text[m.start()..m.end()]) {
break;
}
start_find = m.start() + 1;
}
None => break,
}
}
let end = next_special.map_or(text.len(), |m| m.start());
// Okay, here we go, compare this logic to encode_ordinary
for mat_res in regex.find_iter(&text[start..end]) {
let mat = match mat_res {
Ok(m) => m,
Err(e) => {
return Err(EncodeError {
message: format!("Regex error while tokenizing: {e}"),
});
}
};
let piece = mat.as_str().as_bytes();
if let Some(token) = self.encoder.get(piece) {
last_piece_token_len = 1;
ret.push(*token);
continue;
}
let tokens = byte_pair_encode(piece, &self.encoder);
last_piece_token_len = tokens.len();
ret.extend(&tokens);
}
match next_special {
// And here we push the special token
Some(m) => {
let piece = m.as_str();
let token = self.special_tokens_encoder[piece];
ret.push(token);
start = m.end();
last_piece_token_len = 0;
}
None => break,
}
}
// last_piece_token_len is how many tokens came from the last regex split. This is used
// for determining unstable tokens, since you can't merge across (stable) regex splits
Ok((ret, last_piece_token_len))
}
fn _increase_last_piece_token_len(
&self,
tokens: Vec<Rank>,
mut last_piece_token_len: usize,
) -> (Vec<Rank>, usize) {
// Unfortunately, the locations where our regex splits can be unstable.
// For the purposes of determining unstable tokens, unstable regex splitting
// is only a problem if a split that was present disappears, since this can
// lead to merging of tokens otherwise thought to be stable.
// cl100k_base makes our life hard by including the \s*[\r\n]+
// pattern. This can e.g. cause "\n" + " " to become "\n \n".
// Here is a quick and dirty fix:
{
let token_is_all_space = |token| {
self.decoder
.get(token)
.map(|token_bytes| {
token_bytes
.iter()
.rev()
.all(|&b| [b' ', b'\n', b'\t'].contains(&b))
})
.unwrap_or(false)
};
if last_piece_token_len > 0
&& token_is_all_space(&tokens[tokens.len() - last_piece_token_len])
{
while (last_piece_token_len < tokens.len())
&& token_is_all_space(&tokens[tokens.len() - last_piece_token_len - 1])
{
last_piece_token_len += 1;
}
}
}
debug_assert!(last_piece_token_len <= tokens.len());
(tokens, last_piece_token_len)
}
pub fn _encode_unstable_native(
&self,
text: &str,
allowed_special: &HashSet<&str>,
) -> (Vec<Rank>, HashSet<Vec<Rank>>) {
let (tokens, last_piece_token_len) = self.encode(text, allowed_special).unwrap();
if last_piece_token_len == 0 {
// If last_piece_token_len is zero, the last token was a special token and we have
// no unstable bytes
return (tokens, HashSet::new());
}
let (mut tokens, last_piece_token_len) =
self._increase_last_piece_token_len(tokens, last_piece_token_len);
let unstable_bytes = self
.decode_bytes(&tokens[tokens.len() - last_piece_token_len..])
.unwrap();
tokens.truncate(tokens.len() - last_piece_token_len);
// TODO: we should try harder to find additional stable tokens
// This would reduce the amount of retokenising when determining completions
// Refer to the logic in an older version of this file
let mut completions = HashSet::new();
if unstable_bytes.is_empty() {
return (tokens, completions);
}
// This is the easy bit. Just find all single tokens that start with unstable_bytes
// (including tokens that exactly match unstable_bytes)
// Separating this from the loop below helps with performance in a common case.
let mut point = self
.sorted_token_bytes
.partition_point(|x| x.as_slice() < unstable_bytes.as_slice());
while point < self.sorted_token_bytes.len()
&& self.sorted_token_bytes[point].starts_with(&unstable_bytes)
{
completions.insert(vec![
self.encoder[self.sorted_token_bytes[point].as_slice()],
]);
point += 1;
}
// Now apply even more brute force. At every (other) possible position for the straddling
// token, concatenate additional bytes from that token (if any) to unstable_bytes,
// and retokenise the whole thing and see what we get.
for i in 1..unstable_bytes.len() {
let prefix = &unstable_bytes[..i];
let suffix = &unstable_bytes[i..];
let mut point = self
.sorted_token_bytes
.partition_point(|x| x.as_slice() < suffix);
// TODO: Perf optimisation if suffix starts with " "?
while point < self.sorted_token_bytes.len()
&& self.sorted_token_bytes[point].starts_with(suffix)
{
let possibility = [prefix, self.sorted_token_bytes[point].as_slice()].concat();
let encoded = match std::str::from_utf8(&possibility) {
// Morally, this is byte_pair_encode(&possibility, &self.encoder)
// But we might have introduced a regex split which would prevent merges.
// (particularly possible in the presence of unstable regex splits)
// So convert to UTF-8 and do regex splitting.
// E.g. with cl100k_base " !" gets split to " " + " !",
// but byte_pair_encode(" !") != byte_pair_encode(" ")
Ok(s) => self.encode_ordinary(s),
// Technically, whether or not this arm is correct depends on whether there
// would be a regex split before the UTF-8 truncation point.
// Probably niche enough that no one will ever notice (after all, people didn't
// notice all the big holes in the previous unstable token implementation)
Err(_) => byte_pair_encode(&possibility, &self.encoder),
// Something like the following is intriguing but incorrect:
// Err(e) => self.encode_ordinary(unsafe {
// std::str::from_utf8_unchecked(&possibility[..e.valid_up_to()])
// }),
};
let mut seq = Vec::new();
let mut seq_len = 0;
for token in encoded {
seq.push(token);
seq_len += self.decoder[&token].len();
if seq_len >= unstable_bytes.len() {
break;
}
}
completions.insert(seq);
point += 1;
}
}
// This is also not straightforward. While we generally assume that regex splits are stable,
// unfortunately, they are not. That is, if adding bytes were to make a split appear in
// unstable_bytes, this could make tokens possible which our logic would otherwise think
// would be merged.
// For example, with gpt2, the use of \s+(?!\S) means that "\n\n" could
// develop a split, e.g. "\n\n0" splits into "\n"+"\n"+"0", making "\n" a possible token.
// Here is a quick and dirty fix:
// This isn't right if we ever remove \s+(?!\S)
if unstable_bytes.len() > 1 {
let last_decoded = bstr::decode_last_utf8(unstable_bytes.as_slice());
if unstable_bytes.len() - last_decoded.1 > 0
&& last_decoded.0.is_some_and(|c| c.is_whitespace())
{
let mut reencoded = byte_pair_encode(
&unstable_bytes[..unstable_bytes.len() - last_decoded.1],
&self.encoder,
);
reencoded.extend(byte_pair_encode(
&unstable_bytes[unstable_bytes.len() - last_decoded.1..],
&self.encoder,
));
completions.insert(reencoded);
}
}
(tokens, completions)
}
pub fn new<E, SE, NSE>(
encoder: E,
special_tokens_encoder: SE,
pattern: &str,
) -> Result<Self, Box<dyn std::error::Error + Send + Sync>>
where
E: IntoIterator<Item = (Vec<u8>, Rank)>,
SE: IntoIterator<Item = (String, Rank)>,
NSE: IntoIterator<Item = (String, (Rank, Rank))>,
{
Self::new_internal(
HashMap::from_iter(encoder),
HashMap::from_iter(special_tokens_encoder),
pattern,
)
}
fn new_internal(
encoder: HashMap<Vec<u8>, Rank>,
special_tokens_encoder: HashMap<String, Rank>,
pattern: &str,
) -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
let regex = Regex::new(pattern)?;
let special_regex = {
let parts = special_tokens_encoder
.keys()
.map(|s| fancy_regex::escape(s))
.collect::<Vec<_>>();
Regex::new(&parts.join("|"))?
};
let decoder: HashMap<Rank, Vec<u8>> =
encoder.iter().map(|(k, v)| (*v, k.clone())).collect();
assert!(
encoder.len() == decoder.len(),
"Encoder and decoder must be of equal length. Encoder length: {}, decoder length: {}.\nMaybe you had duplicate token indices in your encoder?",
encoder.len(),
decoder.len()
);
let special_tokens_decoder: HashMap<Rank, Vec<u8>> = special_tokens_encoder
.iter()
.map(|(k, v)| (*v, k.as_bytes().to_vec()))
.collect();
// Clone because I don't know how to tell Rust I'm not going to change the map
let mut sorted_token_bytes: Vec<Vec<u8>> = encoder.keys().cloned().collect();
sorted_token_bytes.sort();
Ok(Self {
encoder,
special_tokens_encoder,
decoder,
special_tokens_decoder,
regex_tls: (0..MAX_NUM_THREADS).map(|_| regex.clone()).collect(),
special_regex_tls: (0..MAX_NUM_THREADS)
.map(|_| special_regex.clone())
.collect(),
sorted_token_bytes,
})
}
pub fn special_tokens(&self) -> HashSet<&str> {
self.special_tokens_encoder
.keys()
.map(|s| s.as_str())
.collect()
}
pub fn encode_with_special_tokens(&self, text: &str) -> Vec<Rank> {
let allowed_special = self.special_tokens();
self.encode(text, &allowed_special).unwrap().0
}
}
#[cfg(test)]
mod tests {
use fancy_regex::Regex;
use rustc_hash::FxHashMap as HashMap;
use crate::{Rank, byte_pair_split};
fn setup_ranks() -> HashMap<Vec<u8>, Rank> {
HashMap::from_iter([(b"ab".to_vec(), 0), (b"cd".to_vec(), 1)])
}
#[test]
fn test_simple_characters() {
let ranks = setup_ranks();
let res = byte_pair_split(b"abcd", &ranks);
assert_eq!(res, vec![b"ab", b"cd"]);
}
#[test]
fn test_repeated_characters() {
let ranks = setup_ranks();
let res = byte_pair_split(b"abab", &ranks);
assert_eq!(res, vec![b"ab", b"ab"]);
}
}
================================================
FILE: src/py.rs
================================================
use std::collections::HashSet;
use pyo3::{
IntoPyObjectExt, PyResult, exceptions,
prelude::*,
pybacked::PyBackedStr,
types::{PyBytes, PyList},
};
use rustc_hash::FxHashMap as HashMap;
use crate::{CoreBPE, Rank, byte_pair_encode};
#[pymethods]
impl CoreBPE {
#[new]
fn py_new(
encoder: HashMap<Vec<u8>, Rank>,
special_tokens_encoder: HashMap<String, Rank>,
pattern: &str,
) -> PyResult<Self> {
Self::new_internal(encoder, special_tokens_encoder, pattern)
.map_err(|e| PyErr::new::<exceptions::PyValueError, _>(e.to_string()))
}
// ====================
// Encoding
// ====================
#[pyo3(name = "encode_ordinary")]
fn py_encode_ordinary(&self, py: Python, text: &str) -> Vec<Rank> {
py.detach(|| self.encode_ordinary(text))
}
#[pyo3(name = "encode")]
fn py_encode(
&self,
py: Python,
text: &str,
allowed_special: HashSet<PyBackedStr>,
) -> PyResult<Vec<Rank>> {
py.detach(|| {
let allowed_special: HashSet<&str> =
allowed_special.iter().map(|s| s.as_ref()).collect();
match self.encode(text, &allowed_special) {
Ok((tokens, _)) => Ok(tokens),
Err(e) => Err(PyErr::new::<exceptions::PyValueError, _>(e.message)),
}
})
}
fn encode_to_tiktoken_buffer(
&self,
py: Python,
text: &str,
allowed_special: HashSet<PyBackedStr>,
) -> PyResult<Py<PyAny>> {
let tokens_res = py.detach(|| {
let allowed_special: HashSet<&str> =
allowed_special.iter().map(|s| s.as_ref()).collect();
self.encode(text, &allowed_special)
});
let tokens = match tokens_res {
Ok((tokens, _)) => tokens,
Err(e) => return Err(PyErr::new::<exceptions::PyValueError, _>(e.message)),
};
let buffer = TiktokenBuffer { tokens };
buffer.into_py_any(py)
}
fn _encode_bytes(&self, py: Python, bytes: &[u8]) -> Vec<Rank> {
py.detach(|| {
match std::str::from_utf8(bytes) {
// Straightforward case
Ok(text) => self.encode_ordinary(text),
// Oops, don't actually have UTF-8. But we need to do the regex splitting in
// Unicode space, so we make our best guess at where we would have splits
Err(e) => {
let text = unsafe { std::str::from_utf8_unchecked(&bytes[..e.valid_up_to()]) };
let (tokens, last_piece_token_len) =
self.encode(text, &HashSet::new()).unwrap();
let (mut tokens, last_piece_token_len) =
self._increase_last_piece_token_len(tokens, last_piece_token_len);
let mut unstable_bytes;
if !tokens.is_empty() && last_piece_token_len > 0 {
// Lop off the tokens from the last piece and run BPE on the remaining bytes
// This likely matches what models see better, e.g. if you assume we're
// dealing with truncated UTF-8 bytes.
// Niche, but note this may not be correct if we'd have had a regex
// split between the valid UTF-8 and the invalid bytes.
unstable_bytes = self
.decode_bytes(&tokens[tokens.len() - last_piece_token_len..])
.unwrap();
unstable_bytes.extend_from_slice(&bytes[e.valid_up_to()..]);
tokens.truncate(tokens.len() - last_piece_token_len);
} else {
unstable_bytes = bytes[e.valid_up_to()..].to_vec();
}
if !unstable_bytes.is_empty() {
match self.encoder.get(&unstable_bytes) {
Some(token) => tokens.push(*token),
None => {
tokens.extend(&byte_pair_encode(&unstable_bytes, &self.encoder))
}
}
}
tokens
}
}
})
}
#[pyo3(name = "encode_with_unstable")]
fn py_encode_with_unstable(
&self,
py: Python,
text: &str,
allowed_special: HashSet<PyBackedStr>,
) -> PyResult<(Vec<Rank>, Py<PyList>)> {
let (tokens, completions): (Vec<Rank>, HashSet<Vec<Rank>>) = py.detach(|| {
let allowed_special: HashSet<&str> =
allowed_special.iter().map(|s| s.as_ref()).collect();
self._encode_unstable_native(text, &allowed_special)
});
let py_completions = PyList::new(py, completions.into_iter())?;
Ok((tokens, py_completions.into()))
}
fn encode_single_token(&self, piece: &[u8]) -> PyResult<Rank> {
if let Some(token) = self.encoder.get(piece).copied() {
return Ok(token);
}
if let Ok(piece_str) = std::str::from_utf8(piece) {
if let Some(token) = self.special_tokens_encoder.get(piece_str).copied() {
return Ok(token);
}
}
Err(PyErr::new::<exceptions::PyKeyError, _>(piece.to_owned()))
}
fn encode_single_piece(&self, piece: &[u8]) -> Vec<Rank> {
if let Some(token) = self.encoder.get(piece) {
return vec![*token];
}
byte_pair_encode(piece, &self.encoder)
}
// ====================
// Decoding
// ====================
#[pyo3(name = "decode_bytes")]
fn py_decode_bytes(&self, py: Python, tokens: Vec<Rank>) -> Result<Py<PyBytes>, PyErr> {
match py.detach(|| self.decode_bytes(&tokens)) {
Ok(bytes) => Ok(PyBytes::new(py, &bytes).into()),
Err(e) => Err(pyo3::exceptions::PyKeyError::new_err(format!("{}", e))),
}
}
fn decode_single_token_bytes(&self, py: Python, token: Rank) -> PyResult<Py<PyBytes>> {
if let Some(bytes) = self.decoder.get(&token) {
return Ok(PyBytes::new(py, bytes).into());
}
if let Some(bytes) = self.special_tokens_decoder.get(&token) {
return Ok(PyBytes::new(py, bytes).into());
}
Err(PyErr::new::<exceptions::PyKeyError, _>(token.to_string()))
}
// ====================
// Miscellaneous
// ====================
fn token_byte_values(&self, py: Python) -> Vec<Py<PyBytes>> {
self.sorted_token_bytes
.iter()
.map(|x| PyBytes::new(py, x).into())
.collect()
}
}
#[pyclass(frozen)]
struct TiktokenBuffer {
tokens: Vec<Rank>,
}
#[pymethods]
impl TiktokenBuffer {
// Based on https://github.com/PyO3/pyo3/blob/v0.22.2/tests/test_buffer_protocol.rs#L25
unsafe fn __getbuffer__(
slf: Bound<'_, Self>,
view: *mut pyo3::ffi::Py_buffer,
flags: std::os::raw::c_int,
) -> PyResult<()> {
if view.is_null() {
return Err(pyo3::exceptions::PyBufferError::new_err("View is null"));
}
if (flags & pyo3::ffi::PyBUF_WRITABLE) == pyo3::ffi::PyBUF_WRITABLE {
return Err(pyo3::exceptions::PyBufferError::new_err(
"Object is not writable",
));
}
unsafe {
let view_ref = &mut *view;
view_ref.obj = slf.clone().into_any().into_ptr();
let data = &slf.borrow().tokens;
view_ref.buf = data.as_ptr() as *mut std::os::raw::c_void;
view_ref.len = (data.len() * std::mem::size_of::<Rank>()) as isize;
view_ref.readonly = 1;
view_ref.itemsize = std::mem::size_of::<Rank>() as isize;
view_ref.format = if (flags & pyo3::ffi::PyBUF_FORMAT) == pyo3::ffi::PyBUF_FORMAT {
let msg = std::ffi::CString::new("I").unwrap();
msg.into_raw()
} else {
std::ptr::null_mut()
};
view_ref.ndim = 1;
view_ref.shape = if (flags & pyo3::ffi::PyBUF_ND) == pyo3::ffi::PyBUF_ND {
&mut view_ref.len
} else {
std::ptr::null_mut()
};
view_ref.strides = if (flags & pyo3::ffi::PyBUF_STRIDES) == pyo3::ffi::PyBUF_STRIDES {
&mut view_ref.itemsize
} else {
std::ptr::null_mut()
};
view_ref.suboffsets = std::ptr::null_mut();
view_ref.internal = std::ptr::null_mut();
}
Ok(())
}
unsafe fn __releasebuffer__(&self, view: *mut pyo3::ffi::Py_buffer) {
// Note that Py_buffer doesn't have a Drop impl
unsafe {
let view_ref = &mut *view;
if !view_ref.format.is_null() {
std::mem::drop(std::ffi::CString::from_raw(view_ref.format));
}
}
}
}
#[pymodule(gil_used = false)]
fn _tiktoken(_py: Python, m: &Bound<PyModule>) -> PyResult<()> {
m.add_class::<CoreBPE>()?;
Ok(())
}
================================================
FILE: tests/__init__.py
================================================
================================================
FILE: tests/test_encoding.py
================================================
# Note that there are more actual tests, they're just not currently public :-)
from typing import Callable
import hypothesis
import hypothesis.strategies as st
import pytest
import tiktoken
from .test_helpers import ENCODING_FACTORIES, MAX_EXAMPLES
def test_simple():
enc = tiktoken.get_encoding("gpt2")
assert enc.encode("hello world") == [31373, 995]
assert enc.decode([31373, 995]) == "hello world"
assert enc.encode("hello <|endoftext|>", allowed_special="all") == [31373, 220, 50256]
enc = tiktoken.get_encoding("cl100k_base")
assert enc.encode("hello world") == [15339, 1917]
assert enc.decode([15339, 1917]) == "hello world"
assert enc.encode("hello <|endoftext|>", allowed_special="all") == [15339, 220, 100257]
for enc_name in tiktoken.list_encoding_names():
enc = tiktoken.get_encoding(enc_name)
for token in range(min(10_000, enc.max_token_value - 1)):
assert enc.encode_single_token(enc.decode_single_token_bytes(token)) == token
def test_simple_repeated():
enc = tiktoken.get_encoding("gpt2")
assert enc.encode("0") == [15]
assert enc.encode("00") == [405]
assert enc.encode("000") == [830]
assert enc.encode("0000") == [2388]
assert enc.encode("00000") == [20483]
assert enc.encode("000000") == [10535]
assert enc.encode("0000000") == [24598]
assert enc.encode("00000000") == [8269]
assert enc.encode("000000000") == [10535, 830]
assert enc.encode("0000000000") == [8269, 405]
assert enc.encode("00000000000") == [8269, 830]
assert enc.encode("000000000000") == [8269, 2388]
assert enc.encode("0000000000000") == [8269, 20483]
assert enc.encode("00000000000000") == [8269, 10535]
assert enc.encode("000000000000000") == [8269, 24598]
assert enc.encode("0000000000000000") == [25645]
assert enc.encode("00000000000000000") == [8269, 10535, 830]
def test_large_repeated():
enc = tiktoken.get_encoding("o200k_base")
# Large inputs should be handled without raising.
tokens = enc.encode("x" * 1_000_000)
assert tokens
def test_simple_regex():
enc = tiktoken.get_encoding("cl100k_base")
assert enc.encode("rer") == [38149]
assert enc.encode("'rer") == [2351, 81]
assert enc.encode("today\n ") == [31213, 198, 220]
assert enc.encode("today\n \n") == [31213, 27907]
assert enc.encode("today\n \n") == [31213, 14211]
def test_basic_encode():
enc = tiktoken.get_encoding("r50k_base")
assert enc.encode("hello world") == [31373, 995]
enc = tiktoken.get_encoding("p50k_base")
assert enc.encode("hello world") == [31373, 995]
enc = tiktoken.get_encoding("cl100k_base")
assert enc.encode("hello world") == [15339, 1917]
assert enc.encode(" \x850") == [220, 126, 227, 15]
def test_encode_empty():
enc = tiktoken.get_encoding("r50k_base")
assert enc.encode("") == []
def test_encode_bytes():
enc = tiktoken.get_encoding("cl100k_base")
assert enc._encode_bytes(b" \xec\x8b\xa4\xed") == [62085]
for i in range(10):
bytestring = b"\x80" * i
assert enc.decode_bytes(enc._encode_bytes(bytestring)) == bytestring
@pytest.mark.parametrize("make_enc", ENCODING_FACTORIES)
@hypothesis.given(bytestring=st.binary())
@hypothesis.settings(deadline=None, max_examples=MAX_EXAMPLES)
def test_hyp_encode_bytes(make_enc: Callable[[], tiktoken.Encoding], bytestring: bytes):
enc = make_enc()
assert enc.decode_bytes(enc._encode_bytes(bytestring)) == bytestring
def test_encode_surrogate_pairs():
enc = tiktoken.get_encoding("cl100k_base")
assert enc.encode("👍") == [9468, 239, 235]
# surrogate pair gets converted to codepoint
assert enc.encode("\ud83d\udc4d") == [9468, 239, 235]
# lone surrogate just gets replaced
assert enc.encode("\ud83d") == enc.encode("�")
@pytest.mark.parametrize("make_enc", ENCODING_FACTORIES)
def test_catastrophically_repetitive(make_enc: Callable[[], tiktoken.Encoding]):
enc = make_enc()
for c in ["^", "0", "a", "'s", " ", "\n"]:
big_value = c * 10_000
assert big_value == enc.decode(enc.encode(big_value))
big_value = " " + big_value
assert big_value == enc.decode(enc.encode(big_value))
big_value = big_value + "\n"
assert big_value == enc.decode(enc.encode(big_value))
# ====================
# Roundtrip
# ====================
@pytest.mark.parametrize("make_enc", ENCODING_FACTORIES)
def test_basic_roundtrip(make_enc):
enc = make_enc()
for value in (
"hello",
"hello ",
"hello ",
" hello",
" hello ",
" hello ",
"hello world",
"请考试我的软件！12345",
):
assert value == enc.decode(enc.encode(value))
assert value == enc.decode(enc.encode_ordinary(value))
@pytest.mark.parametrize("make_enc", ENCODING_FACTORIES)
@hypothesis.given(text=st.text())
@hypothesis.settings(deadline=None, max_examples=MAX_EXAMPLES)
def test_hyp_roundtrip(make_enc: Callable[[], tiktoken.Encoding], text):
enc = make_enc()
assert text == enc.decode(enc.encode(text))
@pytest.mark.parametrize("make_enc", ENCODING_FACTORIES)
def test_single_token_roundtrip(make_enc: Callable[[], tiktoken.Encoding]):
enc = make_enc()
for token in range(enc.n_vocab):
try:
token_bytes = enc.decode_single_token_bytes(token)
except KeyError:
continue
assert enc.encode_single_token(token_bytes) == token
# ====================
# Special tokens
# ====================
def test_special_token():
enc = tiktoken.get_encoding("cl100k_base")
eot = enc.encode_single_token("<|endoftext|>")
assert eot == enc.eot_token
fip = enc.encode_single_token("<|fim_prefix|>")
fim = enc.encode_single_token("<|fim_middle|>")
text = "<|endoftext|> hello <|fim_prefix|>"
assert eot not in enc.encode(text, disallowed_special=())
with pytest.raises(ValueError):
enc.encode(text)
with pytest.raises(ValueError):
enc.encode(text, disallowed_special="all")
with pytest.raises(ValueError):
enc.encode(text, disallowed_special={"<|endoftext|>"})
with pytest.raises(ValueError):
enc.encode(text, disallowed_special={"<|fim_prefix|>"})
text = "<|endoftext|> hello <|fim_prefix|> there <|fim_middle|>"
tokens = enc.encode(text, disallowed_special=())
assert eot not in tokens
assert fip not in tokens
assert fim not in tokens
tokens = enc.encode(text, allowed_special="all", disallowed_special=())
assert eot in tokens
assert fip in tokens
assert fim in tokens
tokens = enc.encode(text, allowed_special="all", disallowed_special="all")
assert eot in tokens
assert fip in tokens
assert fim in tokens
tokens = enc.encode(text, allowed_special={"<|fim_prefix|>"}, disallowed_special=())
assert eot not in tokens
assert fip in tokens
assert fim not in tokens
tokens = enc.encode(text, allowed_special={"<|endoftext|>"}, disallowed_special=())
assert eot in tokens
assert fip not in tokens
assert fim not in tokens
tokens = enc.encode(text, allowed_special={"<|fim_middle|>"}, disallowed_special=())
assert eot not in tokens
assert fip not in tokens
assert fim in tokens
@pytest.mark.parametrize("make_enc", ENCODING_FACTORIES)
@hypothesis.given(text=st.text())
@hypothesis.settings(deadline=None, max_examples=MAX_EXAMPLES)
def test_hyp_special_ordinary(make_enc, text: str):
enc = make_enc()
assert enc.encode_ordinary(text) == enc.encode(text, disallowed_special=())
# ====================
# Batch encoding
# ====================
@pytest.mark.parametrize("make_enc", ENCODING_FACTORIES)
def test_batch_encode(make_enc: Callable[[], tiktoken.Encoding]):
enc = make_enc()
text1 = "hello world"
text2 = "goodbye world"
assert enc.encode_batch([text1]) == [enc.encode(text1)]
assert enc.encode_batch([text1, text2]) == [enc.encode(text1), enc.encode(text2)]
assert enc.encode_ordinary_batch([text1]) == [enc.encode_ordinary(text1)]
assert enc.encode_ordinary_batch([text1, text2]) == [
enc.encode_ordinary(text1),
enc.encode_ordinary(text2),
]
@pytest.mark.parametrize("make_enc", ENCODING_FACTORIES)
@hypothesis.given(batch=st.lists(st.text()))
@hypothesis.settings(deadline=None, max_examples=MAX_EXAMPLES)
def test_hyp_batch_roundtrip(make_enc: Callable[[], tiktoken.Encoding], batch):
enc = make_enc()
encoded = enc.encode_batch(batch, allowed_special="all")
assert encoded == [enc.encode(t, allowed_special="all") for t in batch]
decoded = enc.decode_batch(encoded)
assert decoded == batch
================================================
FILE: tests/test_helpers.py
================================================
import bisect
import functools
import os
import pytest
import tiktoken
MAX_EXAMPLES: int = int(os.environ.get("TIKTOKEN_MAX_EXAMPLES", "100"))
ENCODINGS = ["r50k_base", "cl100k_base"]
SOME_ENCODINGS = ["cl100k_base"]
ENCODING_FACTORIES = [
pytest.param(functools.partial(tiktoken.get_encoding, name), id=name) for name in ENCODINGS
]
SOME_ENCODING_FACTORIES = [
pytest.param(functools.partial(tiktoken.get_encoding, name), id=name) for name in SOME_ENCODINGS
]
================================================
FILE: tests/test_misc.py
================================================
import subprocess
import sys
import tiktoken
def test_encoding_for_model():
enc = tiktoken.encoding_for_model("gpt2")
assert enc.name == "gpt2"
enc = tiktoken.encoding_for_model("text-davinci-003")
assert enc.name == "p50k_base"
enc = tiktoken.encoding_for_model("text-davinci-edit-001")
assert enc.name == "p50k_edit"
enc = tiktoken.encoding_for_model("gpt-3.5-turbo-0301")
assert enc.name == "cl100k_base"
enc = tiktoken.encoding_for_model("gpt-4")
assert enc.name == "cl100k_base"
enc = tiktoken.encoding_for_model("gpt-4o")
assert enc.name == "o200k_base"
enc = tiktoken.encoding_for_model("gpt-oss-120b")
assert enc.name == "o200k_harmony"
def test_optional_blobfile_dependency():
prog = """
import tiktoken
import sys
assert "blobfile" not in sys.modules
"""
subprocess.check_call([sys.executable, "-c", prog])
================================================
FILE: tests/test_offsets.py
================================================
from typing import Callable
import hypothesis
import pytest
from hypothesis import strategies as st
import tiktoken
from .test_helpers import MAX_EXAMPLES, SOME_ENCODING_FACTORIES
def _common_prefix_len(a, b):
i = 0
while i < len(a) and i < len(b) and a[i] == b[i]:
i += 1
return i
def _token_offsets_reference(enc, tokens):
text = enc.decode(tokens, errors="strict")
res = []
for i in range(len(tokens)):
prefix = enc.decode(tokens[:i], errors="ignore")
res.append(_common_prefix_len(text, prefix))
return res
@pytest.mark.parametrize("make_enc", SOME_ENCODING_FACTORIES)
@hypothesis.given(data=st.data())
@hypothesis.settings(deadline=None, max_examples=MAX_EXAMPLES)
def test_hyp_offsets(make_enc: Callable[[], tiktoken.Encoding], data):
enc = make_enc()
tokens_st = st.lists(
st.integers(0, enc.n_vocab - 1).filter(
lambda x: x in enc._special_tokens.values() or x in enc._mergeable_ranks.values()
),
min_size=1,
max_size=20,
)
tokens = data.draw(tokens_st)
# This is a dumb hack to make sure that our tokens are a valid UTF-8 string
# We could potentially drop this, see the TODO in decode_with_offsets
tokens = enc.encode(enc.decode(tokens, errors="ignore"), allowed_special="all")
assert enc.decode_with_offsets(tokens)[1] == _token_offsets_reference(enc, tokens)
def test_basic_offsets():
enc = tiktoken.get_encoding("cl100k_base")
prompt = "hello world"
p, o = enc.decode_with_offsets(enc.encode(prompt))
assert p == prompt
assert o == [0, 5]
prompt = "hello world<|endoftext|> green cow"
p, o = enc.decode_with_offsets(enc.encode(prompt, allowed_special="all"))
assert p == prompt
assert o == [0, 5, 11, 24, 30]
prompt = "我非常渴望与人工智能一起工作"
p, o = enc.decode_with_offsets(enc.encode(prompt))
assert p == prompt
assert o == [0, 1, 2, 3, 3, 4, 4, 5, 6, 7, 8, 8, 9, 10, 11, 12, 13]
# contains the interesting tokens b'\xe0\xae\xbf\xe0\xae' and b'\xe0\xaf\x8d\xe0\xae'
# in which \xe0 is the start of a 3-byte UTF-8 character
prompt = "நடிகர் சூர்யா"
p, o = enc.decode_with_offsets(enc.encode(prompt))
assert p == prompt
assert o == [0, 0, 1, 1, 2, 3, 4, 4, 5, 6, 7, 8, 8, 9, 9, 10, 11, 12, 12]
# contains the interesting token b'\xa0\xe9\x99\xa4'
# in which \xe9 is the start of a 3-byte UTF-8 character and \xa0 is a continuation byte
prompt = " Ġ除"
p, o = enc.decode_with_offsets(enc.encode(prompt))
assert p == prompt
assert o == [0, 1]
================================================
FILE: tests/test_pickle.py
================================================
import tiktoken
def test_pickle():
import pickle
enc_old = tiktoken.get_encoding("r50k_base")
enc_new = pickle.loads(pickle.dumps(enc_old))
assert enc_old.encode("hello world") == enc_new.encode("hello world")
enc_old = tiktoken.Encoding(
name="custom_enc",
pat_str=enc_old._pat_str,
mergeable_ranks=enc_old._mergeable_ranks,
special_tokens={"<|pickle|>": 100_000},
)
enc_new = pickle.loads(pickle.dumps(enc_old))
assert enc_old.encode("hello world") == enc_new.encode("hello world")
assert (
enc_old.encode("<|pickle|>", allowed_special="all")
== enc_new.encode("<|pickle|>", allowed_special="all")
== [100_000]
)
================================================
FILE: tests/test_simple_public.py
================================================
import subprocess
import sys
import tiktoken
def test_simple():
# Note that there are more actual tests, they're just not currently public :-)
enc = tiktoken.get_encoding("gpt2")
assert enc.encode("hello world") == [31373, 995]
assert enc.decode([31373, 995]) == "hello world"
assert enc.encode("hello <|endoftext|>", allowed_special="all") == [31373, 220, 50256]
enc = tiktoken.get_encoding("cl100k_base")
assert enc.encode("hello world") == [15339, 1917]
assert enc.decode([15339, 1917]) == "hello world"
assert enc.encode("hello <|endoftext|>", allowed_special="all") == [15339, 220, 100257]
for enc_name in tiktoken.list_encoding_names():
enc = tiktoken.get_encoding(enc_name)
for token in range(10_000):
assert enc.encode_single_token(enc.decode_single_token_bytes(token)) == token
def test_encoding_for_model():
enc = tiktoken.encoding_for_model("gpt2")
assert enc.name == "gpt2"
enc = tiktoken.encoding_for_model("text-davinci-003")
assert enc.name == "p50k_base"
enc = tiktoken.encoding_for_model("text-davinci-edit-001")
assert enc.name == "p50k_edit"
enc = tiktoken.encoding_for_model("gpt-3.5-turbo-0301")
assert enc.name == "cl100k_base"
def test_optional_blobfile_dependency():
prog = """
import tiktoken
import sys
assert "blobfile" not in sys.modules
"""
subprocess.check_call([sys.executable, "-c", prog])
================================================
FILE: tiktoken/__init__.py
================================================
# This is the public API of tiktoken
from .core import Encoding as Encoding
from .model import encoding_for_model as encoding_for_model
from .model import encoding_name_for_model as encoding_name_for_model
from .registry import get_encoding as get_encoding
from .registry import list_encoding_names as list_encoding_names
__version__ = "0.12.0"
================================================
FILE: tiktoken/_educational.py
================================================
"""This is an educational implementation of the byte pair encoding algorithm."""
from __future__ import annotations
import collections
import regex
import tiktoken
class SimpleBytePairEncoding:
def __init__(self, *, pat_str: str, mergeable_ranks: dict[bytes, int]) -> None:
"""Creates an Encoding object."""
# A regex pattern string that is used to split the input text
self.pat_str = pat_str
# A dictionary mapping token bytes to their ranks. The ranks correspond to merge priority
self.mergeable_ranks = mergeable_ranks
self._decoder = {token: token_bytes for token_bytes, token in mergeable_ranks.items()}
self._pat = regex.compile(pat_str)
def encode(self, text: str, visualise: str | None = "colour") -> list[int]:
"""Encodes a string into tokens.
>>> enc.encode("hello world")
[388, 372]
"""
# Use the regex to split the text into (approximately) words
words = self._pat.findall(text)
tokens = []
for word in words:
# Turn each word into tokens, using the byte pair encoding algorithm
word_bytes = word.encode("utf-8")
word_tokens = bpe_encode(self.mergeable_ranks, word_bytes, visualise=visualise)
tokens.extend(word_tokens)
return tokens
def decode_bytes(self, tokens: list[int]) -> bytes:
"""Decodes a list of tokens into bytes.
>>> enc.decode_bytes([388, 372])
b'hello world'
"""
return b"".join(self._decoder[token] for token in tokens)
def decode(self, tokens: list[int]) -> str:
"""Decodes a list of tokens into a string.
Decoded bytes are not guaranteed to be valid UTF-8. In that case, we replace
the invalid bytes with the replacement character "�".
>>> enc.decode([388, 372])
'hello world'
"""
return self.decode_bytes(tokens).decode("utf-8", errors="replace")
def decode_tokens_bytes(self, tokens: list[int]) -> list[bytes]:
"""Decodes a list of tokens into a list of bytes.
Useful for visualising how a string is tokenised.
>>> enc.decode_tokens_bytes([388, 372])
[b'hello', b' world']
"""
return [self._decoder[token] for token in tokens]
@staticmethod
def train(training_data: str, vocab_size: int, pat_str: str):
"""Train a BPE tokeniser on some data!"""
mergeable_ranks = bpe_train(data=training_data, vocab_size=vocab_size, pat_str=pat_str)
return SimpleBytePairEncoding(pat_str=pat_str, mergeable_ranks=mergeable_ranks)
@staticmethod
def from_tiktoken(encoding):
if isinstance(encoding, str):
encoding = tiktoken.get_encoding(encoding)
return SimpleBytePairEncoding(
pat_str=encoding._pat_str, mergeable_ranks=encoding._mergeable_ranks
)
def bpe_encode(
mergeable_ranks: dict[bytes, int], input: bytes, visualise: str | None = "colour"
) -> list[int]:
parts = [bytes([b]) for b in input]
while True:
# See the intermediate merges play out!
if visualise:
if visualise in ["colour", "color"]:
visualise_tokens(parts)
elif visualise == "simple":
print(parts)
# Iterate over all pairs and find the pair we want to merge the most
min_idx = None
min_rank = None
for i, pair in enumerate(zip(parts[:-1], parts[1:])):
rank = mergeable_ranks.get(pair[0] + pair[1])
if rank is not None and (min_rank is None or rank < min_rank):
min_idx = i
min_rank = rank
# If there were no pairs we could merge, we're done!
if min_rank is None:
break
assert min_idx is not None
# Otherwise, merge that pair and leave the rest unchanged. Then repeat.
parts = parts[:min_idx] + [parts[min_idx] + parts[min_idx + 1]] + parts[min_idx + 2 :]
if visualise:
print()
tokens = [mergeable_ranks[part] for part in parts]
return tokens
def bpe_train(
data: str, vocab_size: int, pat_str: str, visualise: str | None = "colour"
) -> dict[bytes, int]:
# First, add tokens for each individual byte value
if vocab_size < 2**8:
raise ValueError("vocab_size must be at least 256, so we can encode all bytes")
ranks = {}
for i in range(2**8):
ranks[bytes([i])] = i
# Splinter up our data into lists of bytes
# data = "Hello world"
# words = [
# [b'H', b'e', b'l', b'l', b'o'],
# [b' ', b'w', b'o', b'r', b'l', b'd']
# ]
words: list[list[bytes]] = [
[bytes([b]) for b in word.encode("utf-8")] for word in regex.findall(pat_str, data)
]
# Now, use our data to figure out which merges we should make
while len(ranks) < vocab_size:
# Find the most common pair. This will become our next token
stats = collections.Counter()
for piece in words:
for pair in zip(piece[:-1], piece[1:]):
stats[pair] += 1
most_common_pair = max(stats, key=lambda x: stats[x])
token_bytes = most_common_pair[0] + most_common_pair[1]
token = len(ranks)
# Add the new token!
ranks[token_bytes] = token
# Now merge that most common pair in all the words. That is, update our training data
# to reflect our decision to make that pair into a new token.
new_words = []
for word in words:
new_word = []
i = 0
while i < len(word) - 1:
if (word[i], word[i + 1]) == most_common_pair:
# We found our pair! Merge it
new_word.append(token_bytes)
i += 2
else:
new_word.append(word[i])
i += 1
if i == len(word) - 1:
new_word.append(word[i])
new_words.append(new_word)
words = new_words
# See the intermediate merges play out!
if visualise:
print(f"The current most common pair is {most_common_pair[0]} + {most_common_pair[1]}")
print(f"So we made {token_bytes} our {len(ranks)}th token")
if visualise in ["colour", "color"]:
print("Now the first fifty words in our training data look like:")
visualise_tokens([token for word in words[:50] for token in word])
elif visualise == "simple":
print("Now the first twenty words in our training data look like:")
for word in words[:20]:
print(word)
print("\n")
return ranks
def visualise_tokens(token_values: list[bytes]) -> None:
background = [f"\u001b[48;5;{i}m" for i in [167, 179, 185, 77, 80, 68, 134]]
# If token boundaries do not occur at unicode character boundaries, it's unclear how best to
# visualise the token. Here, we'll just use the unicode replacement character to represent some
# fraction of a character.
unicode_token_values = [x.decode("utf-8", errors="replace") for x in token_values]
running_length = 0
last_color = None
for token in unicode_token_values:
color = background[running_length % len(background)]
if color == last_color:
color = background[(running_length + 1) % len(background)]
assert color != last_color
last_color = color
running_length += len(token)
print(color + token, end="")
print("\u001b[0m")
def train_simple_encoding():
gpt2_pattern = (
r"""'s|'t|'re|'ve|'m|'ll|'d| ?[\p{L}]+| ?[\p{N}]+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+"""
)
with open(__file__) as f:
data = f.read()
enc = SimpleBytePairEncoding.train(data, vocab_size=600, pat_str=gpt2_pattern)
print("This is the sequence of merges performed in order to encode 'hello world':")
tokens = enc.encode("hello world")
assert enc.decode(tokens) == "hello world"
assert enc.decode_bytes(tokens) == b"hello world"
assert enc.decode_tokens_bytes(tokens) == [b"hello", b" world"]
return enc
================================================
FILE: tiktoken/core.py
================================================
from __future__ import annotations
import functools
from concurrent.futures import ThreadPoolExecutor
from typing import TYPE_CHECKING, AbstractSet, Collection, Literal, NoReturn, Sequence
from tiktoken import _tiktoken
if TYPE_CHECKING:
import re
import numpy as np
import numpy.typing as npt
class Encoding:
def __init__(
self,
name: str,
*,
pat_str: str,
mergeable_ranks: dict[bytes, int],
special_tokens: dict[str, int],
explicit_n_vocab: int | None = None,
):
"""Creates an Encoding object.
See openai_public.py for examples of how to construct an Encoding object.
Args:
name: The name of the encoding. It should be clear from the name of the encoding
what behaviour to expect, in particular, encodings with different special tokens
should have different names.
pat_str: A regex pattern string that is used to split the input text.
mergeable_ranks: A dictionary mapping mergeable token bytes to their ranks. The ranks
must correspond to merge priority.
special_tokens: A dictionary mapping special token strings to their token values.
explicit_n_vocab: The number of tokens in the vocabulary. If provided, it is checked
that the number of mergeable tokens and special tokens is equal to this number.
"""
self.name = name
self._pat_str = pat_str
self._mergeable_ranks = mergeable_ranks
self._special_tokens = special_tokens
self.max_token_value = max(
max(mergeable_ranks.values()), max(special_tokens.values(), default=0)
)
if explicit_n_vocab:
assert len(mergeable_ranks) + len(special_tokens) == explicit_n_vocab
assert self.max_token_value == explicit_n_vocab - 1
self._core_bpe = _tiktoken.CoreBPE(mergeable_ranks, special_tokens, pat_str)
def __repr__(self) -> str:
return f"<Encoding {self.name!r}>"
# ====================
# Encoding
# ====================
def encode_ordinary(self, text: str) -> list[int]:
"""Encodes a string into tokens, ignoring special tokens.
This is equivalent to `encode(text, disallowed_special=())` (but slightly faster).
```
>>> enc.encode_ordinary("hello world")
[31373, 995]
"""
try:
return self._core_bpe.encode_ordinary(text)
except UnicodeEncodeError:
# See comment in encode
text = text.encode("utf-16", "surrogatepass").decode("utf-16", "replace")
return self._core_bpe.encode_ordinary(text)
def encode(
self,
text: str,
*,
allowed_special: Literal["all"] | AbstractSet[str] = set(), # noqa: B006
disallowed_special: Literal["all"] | Collection[str] = "all",
) -> list[int]:
"""Encodes a string into tokens.
Special tokens are artificial tokens used to unlock capabilities from a model,
such as fill-in-the-middle. So we want to be careful about accidentally encoding special
tokens, since they can be used to trick a model into doing something we don't want it to do.
Hence, by default, encode will raise an error if it encounters text that corresponds
to a special token. This can be controlled on a per-token level using the `allowed_special`
and `disallowed_special` parameters. In particular:
- Setting `disallowed_special` to () will prevent this function from raising errors and
cause all text corresponding to special tokens to be encoded as natural text.
- Setting `allowed_special` to "all" will cause this function to treat all text
corresponding to special tokens to be encoded as special tokens.
```
>>> enc.encode("hello world")
[31373, 995]
>>> enc.encode("<|endoftext|>", allowed_special={"<|endoftext|>"})
[50256]
>>> enc.encode("<|endoftext|>", allowed_special="all")
[50256]
>>> enc.encode("<|endoftext|>")
# Raises ValueError
>>> enc.encode("<|endoftext|>", disallowed_special=())
[27, 91, 437, 1659, 5239, 91, 29]
```
"""
if allowed_special == "all":
allowed_special = self.special_tokens_set
if disallowed_special == "all":
disallowed_special = self.special_tokens_set - allowed_special
if disallowed_special:
if not isinstance(disallowed_special, frozenset):
disallowed_special = frozenset(disallowed_special)
if match := _special_token_regex(disallowed_special).search(text):
raise_disallowed_special_token(match.group())
try:
return self._core_bpe.encode(text, allowed_special)
except UnicodeEncodeError:
# BPE operates on bytes, but the regex operates on unicode. If we pass a str that is
# invalid UTF-8 to Rust, it will rightfully complain. Here we do a quick and dirty
# fixup for any surrogate pairs that may have sneaked their way into the text.
# Technically, this introduces a place where encode + decode doesn't roundtrip a Python
# string, but given that this is input we want to support, maybe that's okay.
# Also we use errors="replace" to handle weird things like lone surrogates.
text = text.encode("utf-16", "surrogatepass").decode("utf-16", "replace")
return self._core_bpe.encode(text, allowed_special)
def encode_to_numpy(
self,
text: str,
*,
allowed_special: Literal["all"] | AbstractSet[str] = set(), # noqa: B006
disallowed_special: Literal["all"] | Collection[str] = "all",
) -> npt.NDArray[np.uint32]:
"""Encodes a string into tokens, returning a numpy array.
Avoids the overhead of copying the token buffer into a Python list.
"""
if allowed_special == "all":
allowed_special = self.special_tokens_set
if disallowed_special == "all":
disallowed_special = self.special_tokens_set - allowed_special
if disallowed_special:
if not isinstance(disallowed_special, frozenset):
disallowed_special = frozenset(disallowed_special)
if match := _special_token_regex(disallowed_special).search(text):
raise_disallowed_special_token(match.group())
import numpy as np
buffer = self._core_bpe.encode_to_tiktoken_buffer(text, allowed_special)
return np.frombuffer(buffer, dtype=np.uint32)
def encode_ordinary_batch(self, text: list[str], *, num_threads: int = 8) -> list[list[int]]:
"""Encodes a list of strings into tokens, in parallel, ignoring special tokens.
This is equivalent to `encode_batch(text, disallowed_special=())` (but slightly faster).
```
>>> enc.encode_ordinary_batch(["hello world", "goodbye world"])
[[31373, 995], [11274, 16390, 995]]
```
"""
encoder = functools.partial(self.encode_ordinary)
with ThreadPoolExecutor(num_threads) as e:
return list(e.map(encoder, text))
def encode_batch(
self,
text: list[str],
*,
num_threads: int = 8,
allowed_special: Literal["all"] | AbstractSet[str] = set(), # noqa: B006
disallowed_special: Literal["all"] | Collection[str] = "all",
) -> list[list[int]]:
"""Encodes a list of strings into tokens, in parallel.
See `encode` for more details on `allowed_special` and `disallowed_special`.
```
>>> enc.encode_batch(["hello world", "goodbye world"])
[[31373, 995], [11274, 16390, 995]]
```
"""
if allowed_special == "all":
allowed_special = self.special_tokens_set
if disallowed_special == "all":
disallowed_special = self.special_tokens_set - allowed_special
if not isinstance(disallowed_special, frozenset):
disallowed_special = frozenset(disallowed_special)
encoder = functools.partial(
self.encode, allowed_special=allowed_special, disallowed_special=disallowed_special
)
with ThreadPoolExecutor(num_threads) as e:
return list(e.map(encoder, text))
def encode_with_unstable(
self,
text: str,
*,
allowed_special: Literal["all"] | AbstractSet[str] = set(), # noqa: B006
disallowed_special: Literal["all"] | Collection[str] = "all",
) -> tuple[list[int], list[list[int]]]:
"""Encodes a string into stable tokens and possible completion sequences.
Note that the stable tokens will only represent a substring of `text`.
See `encode` for more details on `allowed_special` and `disallowed_special`.
This API should itself be considered unstable.
```
>>> enc.encode_with_unstable("hello fanta")
([31373], [(277, 4910), (5113, 265), ..., (8842,)])
>>> text = "..."
>>> stable_tokens, completions = enc.encode_with_unstable(text)
>>> assert text.encode().startswith(enc.decode_bytes(stable_tokens))
>>> assert all(enc.decode_bytes(stable_tokens + seq).startswith(text.encode()) for seq in completions)
```
"""
if allowed_special == "all":
allowed_special = self.special_tokens_set
if disallowed_special == "all":
disallowed_special = self.special_tokens_set - allowed_special
if disallowed_special:
if not isinstance(disallowed_special, frozenset):
disallowed_special = frozenset(disallowed_special)
if match := _special_token_regex(disallowed_special).search(text):
raise_disallowed_special_token(match.group())
return self._core_bpe.encode_with_unstable(text, allowed_special)
def encode_single_token(self, text_or_bytes: str | bytes) -> int:
"""Encodes text corresponding to a single token to its token value.
NOTE: this will encode all special tokens.
Raises `KeyError` if the token is not in the vocabulary.
```
>>> enc.encode_single_token("hello")
31373
```
"""
if isinstance(text_or_bytes, str):
text_or_bytes = text_or_bytes.encode("utf-8")
return self._core_bpe.encode_single_token(text_or_bytes)
# ====================
# Decoding
# ====================
def decode_bytes(self, tokens: Sequence[int]) -> bytes:
"""Decodes a list of tokens into bytes.
```
>>> enc.decode_bytes([31373, 995])
b'hello world'
```
"""
return self._core_bpe.decode_bytes(tokens)
def decode(self, tokens: Sequence[int], errors: str = "replace") -> str:
"""Decodes a list of tokens into a string.
WARNING: the default behaviour of this function is lossy, since decoded bytes are not
guaranteed to be valid UTF-8. You can control this behaviour using the `errors` parameter,
for instance, setting `errors=strict`.
```
>>> enc.decode([31373, 995])
'hello world'
```
"""
return self._core_bpe.decode_bytes(tokens).decode("utf-8", errors=errors)
def decode_single_token_bytes(self, token: int) -> bytes:
"""Decodes a token into bytes.
NOTE: this will decode all special tokens.
Raises `KeyError` if the token is not in the vocabulary.
```
>>> enc.decode_single_token_bytes(31373)
b'hello'
```
"""
return self._core_bpe.decode_single_token_bytes(token)
def decode_tokens_bytes(self, tokens: Sequence[int]) -> list[bytes]:
"""Decodes a list of tokens into a list of bytes.
Useful for visualising tokenisation.
>>> enc.decode_tokens_bytes([31373, 995])
[b'hello', b' world']
"""
return [self.decode_single_token_bytes(token) for token in tokens]
def decode_with_offsets(self, tokens: Sequence[int]) -> tuple[str, list[int]]:
"""Decodes a list of tokens into a string and a list of offsets.
Each offset is the index into text corresponding to the start of each token.
If UTF-8 character boundaries do not line up with token boundaries, the offset is the index
of the first character that contains bytes from the token.
This will currently raise if given tokens that decode to invalid UTF-8; this behaviour may
change in the future to be more permissive.
>>> enc.decode_with_offsets([31373, 995])
('hello world', [0, 5])
"""
token_bytes = self.decode_tokens_bytes(tokens)
text_len = 0
offsets = []
for token in token_bytes:
offsets.append(max(0, text_len - (0x80 <= token[0] < 0xC0)))
text_len += sum(1 for c in token if not 0x80 <= c < 0xC0)
# TODO: assess correctness for errors="ignore" and errors="replace"
text = b"".join(token_bytes).decode("utf-8", errors="strict")
return text, offsets
def decode_batch(
self, batch: Sequence[Sequence[int]], *, errors: str = "replace", num_threads: int = 8
) -> list[str]:
"""Decodes a batch (list of lists of tokens) into a list of strings."""
decoder = functools.partial(self.decode, errors=errors)
with ThreadPoolExecutor(num_threads) as e:
return list(e.map(decoder, batch))
def decode_bytes_batch(
self, batch: Sequence[Sequence[int]], *, num_threads: int = 8
) -> list[bytes]:
"""Decodes a batch (list of lists of tokens) into a list of bytes."""
with ThreadPoolExecutor(num_threads) as e:
return list(e.map(self.decode_bytes, batch))
# ====================
# Miscellaneous
# ====================
def token_byte_values(self) -> list[bytes]:
"""Returns the list of all token byte values."""
return self._core_bpe.token_byte_values()
@property
def eot_token(self) -> int:
return self._special_tokens["<|endoftext|>"]
@functools.cached_property
def special_tokens_set(self) -> set[str]:
return set(self._special_tokens.keys())
def is_special_token(self, token: int) -> bool:
assert isinstance(token, int)
return token in self._special_token_values
@property
def n_vocab(self) -> int:
"""For backwards compatibility. Prefer to use `enc.max_token_value + 1`."""
return self.max_token_value + 1
# ====================
# Private
# ====================
def _encode_single_piece(self, text_or_bytes: str | bytes) -> list[int]:
"""Encodes text corresponding to bytes without a regex split.
NOTE: this will not encode any special tokens.
```
>>> enc.encode_single_piece("helloqqqq")
[31373, 38227, 38227]
```
"""
if isinstance(text_or_bytes, str):
text_or_bytes = text_or_bytes.encode("utf-8")
return self._core_bpe.encode_single_piece(text_or_bytes)
def _encode_only_native_bpe(self, text: str) -> list[int]:
"""Encodes a string into tokens, but do regex splitting in Python."""
# We need specifically `regex` in order to compile pat_str due to e.g. \p
import regex
_unused_pat = regex.compile(self._pat_str)
ret = []
for piece in regex.findall(_unused_pat, text):
ret.extend(self._core_bpe.encode_single_piece(piece.encode("utf-8")))
return ret
def _encode_bytes(self, text: bytes) -> list[int]:
return self._core_bpe._encode_bytes(text)
def __getstate__(self) -> object:
import tiktoken.registry
# As an optimisation, pickle registered encodings by reference
if self is tiktoken.registry.ENCODINGS.get(self.name):
return self.name
return {
"name": self.name,
"pat_str": self._pat_str,
"mergeable_ranks": self._mergeable_ranks,
"special_tokens": self._special_tokens,
}
def __setstate__(self, value: object) -> None:
import tiktoken.registry
if isinstance(value, str):
self.__dict__ = tiktoken.registry.get_encoding(value).__dict__
return
self.__init__(**value)
@functools.lru_cache(maxsize=128)
def _special_token_regex(tokens: frozenset[str]) -> re.Pattern[str]:
try:
import regex as re
except ImportError:
import re
inner = "|".join(re.escape(token) for token in tokens)
return re.compile(f"({inner})")
def raise_disallowed_special_token(token: str) -> NoReturn:
raise ValueError(
f"Encountered text corresponding to disallowed special token {token!r}.\n"
"If you want this text to be encoded as a special token, "
f"pass it to `allowed_special`, e.g. `allowed_special={{{token!r}, ...}}`.\n"
f"If you want this text to be encoded as normal text, disable the check for this token "
f"by passing `disallowed_special=(enc.special_tokens_set - {{{token!r}}})`.\n"
"To disable this check for all special tokens, pass `disallowed_special=()`.\n"
)
================================================
FILE: tiktoken/load.py
================================================
from __future__ import annotations
import base64
import hashlib
import os
def read_file(blobpath: str) -> bytes:
if "://" not in blobpath:
with open(blobpath, "rb", buffering=0) as f:
return f.read()
if blobpath.startswith(("http://", "https://")):
# avoiding blobfile for public files helps avoid auth issues, like MFA prompts.
import requests
resp = requests.get(blobpath)
resp.raise_for_status()
return resp.content
try:
import blobfile
except ImportError as e:
raise ImportError(
"blobfile is not installed. Please install it by running `pip install blobfile`."
) from e
return blobfile.read_bytes(blobpath)
def check_hash(data: bytes, expected_hash: str) -> bool:
actual_hash = hashlib.sha256(data).hexdigest()
return actual_hash == expected_hash
def read_file_cached(blobpath: str, expected_hash: str | None = None) -> bytes:
user_specified_cache = True
if "TIKTOKEN_CACHE_DIR" in os.environ:
cache_dir = os.environ["TIKTOKEN_CACHE_DIR"]
elif "DATA_GYM_CACHE_DIR" in os.environ:
cache_dir = os.environ["DATA_GYM_CACHE_DIR"]
else:
import tempfile
cache_dir = os.path.join(tempfile.gettempdir(), "data-gym-cache")
user_specified_cache = False
if cache_dir == "":
# disable caching
return read_file(blobpath)
cache_key = hashlib.sha1(blobpath.encode()).hexdigest()
cache_path = os.path.join(cache_dir, cache_key)
if os.path.exists(cache_path):
with open(cache_path, "rb", buffering=0) as f:
data = f.read()
if expected_hash is None or check_hash(data, expected_hash):
return data
# the cached file does not match the hash, remove it and re-fetch
try:
os.remove(cache_path)
except OSError:
pass
contents = read_file(blobpath)
if expected_hash and not check_hash(contents, expected_hash):
raise ValueError(
f"Hash mismatch for data downloaded from {blobpath} (expected {expected_hash}). "
f"This may indicate a corrupted download. Please try again."
)
import uuid
try:
os.makedirs(cache_dir, exist_ok=True)
tmp_filename = cache_path + "." + str(uuid.uuid4()) + ".tmp"
with open(tmp_filename, "wb") as f:
f.write(contents)
os.rename(tmp_filename, cache_path)
except OSError:
# don't raise if we can't write to the default cache, e.g. issue #75
if user_specified_cache:
raise
return contents
def data_gym_to_mergeable_bpe_ranks(
vocab_bpe_file: str,
encoder_json_file: str,
vocab_bpe_hash: str | None = None,
encoder_json_hash: str | None = None,
clobber_one_byte_tokens: bool = False,
) -> dict[bytes, int]:
# NB: do not add caching to this function
rank_to_intbyte = [b for b in range(2**8) if chr(b).isprintable() and chr(b) != " "]
data_gym_byte_to_byte = {chr(b): b for b in rank_to_intbyte}
n = 0
for b in range(2**8):
if b not in rank_to_intbyte:
rank_to_intbyte.append(b)
data_gym_byte_to_byte[chr(2**8 + n)] = b
n += 1
assert len(rank_to_intbyte) == 2**8
# vocab_bpe contains the merges along with associated ranks
vocab_bpe_contents = read_file_cached(vocab_bpe_file, vocab_bpe_hash).decode()
bpe_merges = [tuple(merge_str.split()) for merge_str in vocab_bpe_contents.split("\n")[1:-1]]
def decode_data_gym(value: str) -> bytes:
return bytes(data_gym_byte_to_byte[b] for b in value)
# add the single byte tokens
# if clobber_one_byte_tokens is True, we'll replace these with ones from the encoder json
bpe_ranks = {bytes([b]): i for i, b in enumerate(rank_to_intbyte)}
del rank_to_intbyte
# add the merged tokens
n = len(bpe_ranks)
for first, second in bpe_merges:
bpe_ranks[decode_data_gym(first) + decode_data_gym(second)] = n
n += 1
import json
# check that the encoder file matches the merges file
# this sanity check is important since tiktoken assumes that ranks are ordered the same
# as merge priority
encoder_json = json.loads(read_file_cached(encoder_json_file, encoder_json_hash))
encoder_json_loaded = {decode_data_gym(k): v for k, v in encoder_json.items()}
# drop these two special tokens if present, since they're not mergeable bpe tokens
encoder_json_loaded.pop(b"<|endoftext|>", None)
encoder_json_loaded.pop(b"<|startoftext|>", None)
if clobber_one_byte_tokens:
for k in encoder_json_loaded:
if len(k) == 1:
bpe_ranks[k] = encoder_json_loaded[k]
assert bpe_ranks == encoder_json_loaded
return bpe_ranks
def dump_tiktoken_bpe(bpe_ranks: dict[bytes, int], tiktoken_bpe_file: str) -> None:
try:
import blobfile
except ImportError as e:
raise ImportError(
"blobfile is not installed. Please install it by running `pip install blobfile`."
) from e
with blobfile.BlobFile(tiktoken_bpe_file, "wb") as f:
for token, rank in sorted(bpe_ranks.items(), key=lambda x: x[1]):
f.write(base64.b64encode(token) + b" " + str(rank).encode() + b"\n")
def load_tiktoken_bpe(tiktoken_bpe_file: str, expected_hash: str | None = None) -> dict[bytes, int]:
# NB: do not add caching to this function
contents = read_file_cached(tiktoken_bpe_file, expected_hash)
ret = {}
for line in contents.splitlines():
if not line:
continue
try:
token, rank = line.split()
ret[base64.b64decode(token)] = int(rank)
except Exception as e:
raise ValueError(f"Error parsing line {line!r} in {tiktoken_bpe_file}") from e
return ret
================================================
FILE: tiktoken/model.py
================================================
from __future__ import annotations
from .core import Encoding
from .registry import get_encoding
# TODO: these will likely be replaced by an API endpoint
MODEL_PREFIX_TO_ENCODING: dict[str, str] = {
"o1-": "o200k_base",
"o3-": "o200k_base",
"o4-mini-": "o200k_base",
# chat
"gpt-5-": "o200k_base",
"gpt-4.5-": "o200k_base",
"gpt-4.1-": "o200k_base",
"chatgpt-4o-": "o200k_base",
"gpt-4o-": "o200k_base", # e.g., gpt-4o-2024-05-13
"gpt-4-": "cl100k_base", # e.g., gpt-4-0314, etc., plus gpt-4-32k
"gpt-3.5-turbo-": "cl100k_base", # e.g, gpt-3.5-turbo-0301, -0401, etc.
"gpt-35-turbo-": "cl100k_base", # Azure deployment name
"gpt-oss-": "o200k_harmony",
# fine-tuned
"ft:gpt-4o": "o200k_base",
"ft:gpt-4": "cl100k_base",
"ft:gpt-3.5-turbo": "cl100k_base",
"ft:davinci-002": "cl100k_base",
"ft:babbage-002": "cl100k_base",
}
MODEL_TO_ENCODING: dict[str, str] = {
# reasoning
"o1": "o200k_base",
"o3": "o200k_base",
"o4-mini": "o200k_base",
# chat
"gpt-5": "o200k_base",
"gpt-4.1": "o200k_base",
"gpt-4o": "o200k_base",
"gpt-4": "cl100k_base",
"gpt-3.5-turbo": "cl100k_base",
"gpt-3.5": "cl100k_base", # Common shorthand
"gpt-35-turbo": "cl100k_base", # Azure deployment name
# base
"davinci-002": "cl100k_base",
"babbage-002": "cl100k_base",
# embeddings
"text-embedding-ada-002": "cl100k_base",
"text-embedding-3-small": "cl100k_base",
"text-embedding-3-large": "cl100k_base",
# DEPRECATED MODELS
# text (DEPRECATED)
"text-davinci-003": "p50k_base",
"text-davinci-002": "p50k_base",
"text-davinci-001": "r50k_base",
"text-curie-001": "r50k_base",
"text-babbage-001": "r50k_base",
"text-ada-001": "r50k_base",
"davinci": "r50k_base",
"curie": "r50k_base",
"babbage": "r50k_base",
"ada": "r50k_base",
# code (DEPRECATED)
"code-davinci-002": "p50k_base",
"code-davinci-001": "p50k_base",
"code-cushman-002": "p50k_base",
"code-cushman-001": "p50k_base",
"davinci-codex": "p50k_base",
"cushman-codex": "p50k_base",
# edit (DEPRECATED)
"text-davinci-edit-001": "p50k_edit",
"code-davinci-edit-001": "p50k_edit",
# old embeddings (DEPRECATED)
"text-similarity-davinci-001": "r50k_base",
"text-similarity-curie-001": "r50k_base",
"text-similarity-babbage-001": "r50k_base",
"text-similarity-ada-001": "r50k_base",
"text-search-davinci-doc-001": "r50k_base",
"text-search-curie-doc-001": "r50k_base",
"text-search-babbage-doc-001": "r50k_base",
"text-search-ada-doc-001": "r50k_base",
"code-search-babbage-code-001": "r50k_base",
"code-search-ada-code-001": "r50k_base",
# open source
"gpt2": "gpt2",
"gpt-2": "gpt2", # Maintains consistency with gpt-4
}
def encoding_name_for_model(model_name: str) -> str:
"""Returns the name of the encoding used by a model.
Raises a KeyError if the model name is not recognised.
"""
encoding_name = None
if model_name in MODEL_TO_ENCODING:
encoding_name = MODEL_TO_ENCODING[model_name]
else:
# Check if the model matches a known prefix
# Prefix matching avoids needing library updates for every model version release
# Note that this can match on non-existent models (e.g., gpt-3.5-turbo-FAKE)
for model_prefix, model_encoding_name in MODEL_PREFIX_TO_ENCODING.items():
if model_name.startswith(model_prefix):
return model_encoding_name
if encoding_name is None:
raise KeyError(
f"Could not automatically map {model_name} to a tokeniser. "
"Please use `tiktoken.get_encoding` to explicitly get the tokeniser you expect."
) from None
return encoding_name
def encoding_for_model(model_name: str) -> Encoding:
"""Returns the encoding used by a model.
Raises a KeyError if the model name is not recognised.
"""
return get_encoding(encoding_name_for_model(model_name))
================================================
FILE: tiktoken/py.typed
================================================
================================================
FILE: tiktoken/registry.py
================================================
from __future__ import annotations
import functools
import importlib
import pkgutil
import threading
from typing import Any, Callable, Sequence
import tiktoken_ext
import tiktoken
from tiktoken.core import Encoding
_lock = threading.RLock()
ENCODINGS: dict[str, Encoding] = {}
ENCODING_CONSTRUCTORS: dict[str, Callable[[], dict[str, Any]]] | None = None
@functools.lru_cache
def _available_plugin_modules() -> Sequence[str]:
# tiktoken_ext is a namespace package
# submodules inside tiktoken_ext will be inspected for ENCODING_CONSTRUCTORS attributes
# - we use namespace package pattern so `pkgutil.iter_modules` is fast
# - it's a separate top-level package because namespace subpackages of non-namespace
# packages don't quite do what you want with editable installs
mods = []
plugin_mods = pkgutil.iter_modules(tiktoken_ext.__path__, tiktoken_ext.__name__ + ".")
for _, mod_name, _ in plugin_mods:
mods.append(mod_name)
return mods
def _find_constructors() -> None:
global ENCODING_CONSTRUCTORS
with _lock:
if ENCODING_CONSTRUCTORS is not None:
return
ENCODING_CONSTRUCTORS = {}
try:
for mod_name in _available_plugin_modules():
mod = importlib.import_module(mod_name)
try:
constructors = mod.ENCODING_CONSTRUCTORS
except AttributeError as e:
raise ValueError(
f"tiktoken plugin {mod_name} does not define ENCODING_CONSTRUCTORS"
) from e
for enc_name, constructor in constructors.items():
if enc_name in ENCODING_CONSTRUCTORS:
raise ValueError(
f"Duplicate encoding name {enc_name} in tiktoken plugin {mod_name}"
)
ENCODING_CONSTRUCTORS[enc_name] = constructor
except Exception:
# Ensure we idempotently raise errors
ENCODING_CONSTRUCTORS = None
raise
def get_encoding(encoding_name: str) -> Encoding:
if not isinstance(encoding_name, str):
raise ValueError(f"Expected a string in get_encoding, got {type(encoding_name)}")
if encoding_name in ENCODINGS:
return ENCODINGS[encoding_name]
with _lock:
if encoding_name in ENCODINGS:
return ENCODINGS[encoding_name]
if ENCODING_CONSTRUCTORS is None:
_find_constructors()
assert ENCODING_CONSTRUCTORS is not None
if encoding_name not in ENCODING_CONSTRUCTORS:
raise ValueError(
f"Unknown encoding {encoding_name}.\n"
f"Plugins found: {_available_plugin_modules()}\n"
f"tiktoken version: {tiktoken.__version__} (are you on latest?)"
)
constructor = ENCODING_CONSTRUCTORS[encoding_name]
enc = Encoding(**constructor())
ENCODINGS[encoding_name] = enc
return enc
def list_encoding_names() -> list[str]:
with _lock:
if ENCODING_CONSTRUCTORS is None:
_find_constructors()
assert ENCODING_CONSTRUCTORS is not None
return list(ENCODING_CONSTRUCTORS)
================================================
FILE: tiktoken_ext/openai_public.py
================================================
from tiktoken.load import data_gym_to_mergeable_bpe_ranks, load_tiktoken_bpe
ENDOFTEXT = "<|endoftext|>"
FIM_PREFIX = "<|fim_prefix|>"
FIM_MIDDLE = "<|fim_middle|>"
FIM_SUFFIX = "<|fim_suffix|>"
ENDOFPROMPT = "<|endofprompt|>"
# The pattern in the original GPT-2 release is:
# r"""'s|'t|'re|'ve|'m|'ll|'d| ?[\p{L}]+| ?[\p{N}]+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+"""
# This is equivalent, but executes faster:
r50k_pat_str = (
r"""'(?:[sdmt]|ll|ve|re)| ?\p{L}++| ?\p{N}++| ?[^\s\p{L}\p{N}]++|\s++$|\s+(?!\S)|\s"""
)
def gpt2():
mergeable_ranks = data_gym_to_mergeable_bpe_ranks(
vocab_bpe_file="https://openaipublic.blob.core.windows.net/gpt-2/encodings/main/vocab.bpe",
encoder_json_file="https://openaipublic.blob.core.windows.net/gpt-2/encodings/main/encoder.json",
vocab_bpe_hash="1ce1664773c50f3e0cc8842619a93edc4624525b728b188a9e0be33b7726adc5",
encoder_json_hash="196139668be63f3b5d6574427317ae82f612a97c5d1cdaf36ed2256dbf636783",
)
return {
"name": "gpt2",
"explicit_n_vocab": 50257,
"pat_str": r50k_pat_str,
"mergeable_ranks": mergeable_ranks,
"special_tokens": {ENDOFTEXT: 50256},
}
def r50k_base():
mergeable_ranks = load_tiktoken_bpe(
"https://openaipublic.blob.core.windows.net/encodings/r50k_base.tiktoken",
expected_hash="306cd27f03c1a714eca7108e03d66b7dc042abe8c258b44c199a7ed9838dd930",
)
return {
"name": "r50k_base",
"explicit_n_vocab": 50257,
"pat_str": r50k_pat_str,
"mergeable_ranks": mergeable_ranks,
"special_tokens": {ENDOFTEXT: 50256},
}
def p50k_base():
mergeable_ranks = load_tiktoken_bpe(
"https://openaipublic.blob.core.windows.net/encodings/p50k_base.tiktoken",
expected_hash="94b5ca7dff4d00767bc256fdd1b27e5b17361d7b8a5f968547f9f23eb70d2069",
)
return {
"name": "p50k_base",
"explicit_n_vocab": 50281,
"pat_str": r50k_pat_str,
"mergeable_ranks": mergeable_ranks,
"special_tokens": {ENDOFTEXT: 50256},
}
def p50k_edit():
mergeable_ranks = load_tiktoken_bpe(
"https://openaipublic.blob.core.windows.net/encodings/p50k_base.tiktoken",
expected_hash="94b5ca7dff4d00767bc256fdd1b27e5b17361d7b8a5f968547f9f23eb70d2069",
)
special_tokens = {ENDOFTEXT: 50256, FIM_PREFIX: 50281, FIM_MIDDLE: 50282, FIM_SUFFIX: 50283}
return {
"name": "p50k_edit",
"pat_str": r50k_pat_str,
"mergeable_ranks": mergeable_ranks,
"special_tokens": special_tokens,
}
def cl100k_base():
mergeable_ranks = load_tiktoken_bpe(
"https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken",
expected_hash="223921b76ee99bde995b7ff738513eef100fb51d18c93597a113bcffe865b2a7",
)
special_tokens = {
ENDOFTEXT: 100257,
FIM_PREFIX: 100258,
FIM_MIDDLE: 100259,
FIM_SUFFIX: 100260,
ENDOFPROMPT: 100276,
}
return {
"name": "cl100k_base",
"pat_str": r"""'(?i:[sdmt]|ll|ve|re)|[^\r\n\p{L}\p{N}]?+\p{L}++|\p{N}{1,3}+| ?[^\s\p{L}\p{N}]++[\r\n]*+|\s++$|\s*[\r\n]|\s+(?!\S)|\s""",
"mergeable_ranks": mergeable_ranks,
"special_tokens": special_tokens,
}
def o200k_base():
mergeable_ranks = load_tiktoken_bpe(
"https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken",
expected_hash="446a9538cb6c348e3516120d7c08b09f57c36495e2acfffe59a5bf8b0cfb1a2d",
)
special_tokens = {ENDOFTEXT: 199999, ENDOFPROMPT: 200018}
# This regex could be made more efficient. If I was the one working on this encoding, I would
# have done a few other things differently too, e.g. I think you can allocate tokens more
# efficiently across languages.
pat_str = "|".join(
[
r"""[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+(?i:'s|'t|'re|'ve|'m|'ll|'d)?""",
r"""[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+[\p{Ll}\p{Lm}\p{Lo}\p{M}]*(?i:'s|'t|'re|'ve|'m|'ll|'d)?""",
r"""\p{N}{1,3}""",
r""" ?[^\s\p{L}\p{N}]+[\r\n/]*""",
r"""\s*[\r\n]+""",
r"""\s+(?!\S)""",
r"""\s+""",
]
)
return {
"name": "o200k_base",
"pat_str": pat_str,
"mergeable_ranks": mergeable_ranks,
"special_tokens": special_tokens,
}
def o200k_harmony():
base_enc = o200k_base()
name = "o200k_harmony"
pat_str = base_enc["pat_str"]
mergeable_ranks = base_enc["mergeable_ranks"]
special_tokens = {
**base_enc["special_tokens"],
"<|startoftext|>": 199998,
"<|endoftext|>": 199999,
"<|reserved_200000|>": 200000,
"<|reserved_200001|>": 200001,
"<|return|>": 200002,
"<|constrain|>": 200003,
"<|reserved_200004|>": 200004,
"<|channel|>": 200005,
"<|start|>": 200006,
"<|end|>": 200007,
"<|message|>": 200008,
"<|reserved_200009|>": 200009,
"<|reserved_200010|>": 200010,
"<|reserved_200011|>": 200011,
"<|call|>": 200012,
} | {f"<|reserved_{i}|>": i for i in range(200013, 201088)}
return {
"name": name,
"pat_str": pat_str,
"mergeable_ranks": mergeable_ranks,
"special_tokens": special_tokens,
}
ENCODING_CONSTRUCTORS = {
"gpt2": gpt2,
"r50k_base": r50k_base,
"p50k_base": p50k_base,
"p50k_edit": p50k_edit,
"cl100k_base": cl100k_base,
"o200k_base": o200k_base,
"o200k_harmony": o200k_harmony,
}
gitextract_vtvi0nnb/
├── .github/
│ └── workflows/
│ └── build_wheels.yml
├── .gitignore
├── CHANGELOG.md
├── Cargo.toml
├── LICENSE
├── MANIFEST.in
├── README.md
├── pyproject.toml
├── scripts/
│ ├── benchmark.py
│ ├── redact.py
│ └── wheel_download.py
├── setup.py
├── src/
│ ├── lib.rs
│ └── py.rs
├── tests/
│ ├── __init__.py
│ ├── test_encoding.py
│ ├── test_helpers.py
│ ├── test_misc.py
│ ├── test_offsets.py
│ ├── test_pickle.py
│ └── test_simple_public.py
├── tiktoken/
│ ├── __init__.py
│ ├── _educational.py
│ ├── core.py
│ ├── load.py
│ ├── model.py
│ ├── py.typed
│ └── registry.py
└── tiktoken_ext/
└── openai_public.py
SYMBOL INDEX (140 symbols across 16 files)
FILE: scripts/benchmark.py
function benchmark_batch (line 15) | def benchmark_batch(documents: list[str]) -> None:
FILE: scripts/redact.py
function redact_file (line 7) | def redact_file(path: Path, dry_run: bool) -> None:
function redact (line 42) | def redact(dry_run: bool) -> None:
function main (line 57) | def main() -> None:
FILE: scripts/wheel_download.py
function download_artifacts (line 8) | def download_artifacts(token, owner, repo, run_id, output_dir):
FILE: src/lib.rs
type Rank (line 13) | pub type Rank = u32;
type Merge (line 18) | struct Merge {
method cmp (line 25) | fn cmp(&self, other: &Self) -> std::cmp::Ordering {
method partial_cmp (line 34) | fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {
type State (line 39) | struct State {
function _byte_pair_merge_large (line 47) | fn _byte_pair_merge_large(ranks: &HashMap<Vec<u8>, Rank>, piece: &[u8]) ...
function _byte_pair_merge (line 140) | fn _byte_pair_merge(ranks: &HashMap<Vec<u8>, Rank>, piece: &[u8]) -> Vec...
function byte_pair_encode (line 198) | pub fn byte_pair_encode(piece: &[u8], ranks: &HashMap<Vec<u8>, Rank>) ->...
function byte_pair_split (line 213) | pub fn byte_pair_split<'a>(piece: &'a [u8], ranks: &HashMap<Vec<u8>, Ran...
type FakeThreadId (line 262) | struct FakeThreadId(NonZeroU64);
function hash_current_thread (line 264) | fn hash_current_thread() -> usize {
type DecodeKeyError (line 278) | pub struct DecodeKeyError {
method fmt (line 283) | fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
type DecodeError (line 291) | pub struct DecodeError {
method fmt (line 296) | fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
type EncodeError (line 304) | pub struct EncodeError {
method fmt (line 309) | fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
constant MAX_NUM_THREADS (line 316) | const MAX_NUM_THREADS: usize = 128;
type CoreBPE (line 320) | pub struct CoreBPE {
method _get_tl_regex (line 331) | fn _get_tl_regex(&self) -> &Regex {
method _get_tl_special_regex (line 338) | fn _get_tl_special_regex(&self) -> &Regex {
method decode_bytes (line 345) | fn decode_bytes(&self, tokens: &[Rank]) -> Result<Vec<u8>, DecodeKeyEr...
method encode_ordinary (line 360) | pub fn encode_ordinary(&self, text: &str) -> Vec<Rank> {
method encode (line 375) | pub fn encode(
method _increase_last_piece_token_len (line 444) | fn _increase_last_piece_token_len(
method _encode_unstable_native (line 483) | pub fn _encode_unstable_native(
method new (line 601) | pub fn new<E, SE, NSE>(
method new_internal (line 618) | fn new_internal(
method special_tokens (line 665) | pub fn special_tokens(&self) -> HashSet<&str> {
method encode_with_special_tokens (line 672) | pub fn encode_with_special_tokens(&self, text: &str) -> Vec<Rank> {
function setup_ranks (line 685) | fn setup_ranks() -> HashMap<Vec<u8>, Rank> {
function test_simple_characters (line 690) | fn test_simple_characters() {
function test_repeated_characters (line 697) | fn test_repeated_characters() {
FILE: src/py.rs
method py_new (line 16) | fn py_new(
method py_encode_ordinary (line 30) | fn py_encode_ordinary(&self, py: Python, text: &str) -> Vec<Rank> {
method py_encode (line 35) | fn py_encode(
method encode_to_tiktoken_buffer (line 51) | fn encode_to_tiktoken_buffer(
method _encode_bytes (line 72) | fn _encode_bytes(&self, py: Python, bytes: &[u8]) -> Vec<Rank> {
method py_encode_with_unstable (line 118) | fn py_encode_with_unstable(
method encode_single_token (line 133) | fn encode_single_token(&self, piece: &[u8]) -> PyResult<Rank> {
method encode_single_piece (line 145) | fn encode_single_piece(&self, piece: &[u8]) -> Vec<Rank> {
method py_decode_bytes (line 157) | fn py_decode_bytes(&self, py: Python, tokens: Vec<Rank>) -> Result<Py<Py...
method decode_single_token_bytes (line 164) | fn decode_single_token_bytes(&self, py: Python, token: Rank) -> PyResult...
method token_byte_values (line 178) | fn token_byte_values(&self, py: Python) -> Vec<Py<PyBytes>> {
type TiktokenBuffer (line 187) | struct TiktokenBuffer {
method __getbuffer__ (line 194) | unsafe fn __getbuffer__(
method __releasebuffer__ (line 240) | unsafe fn __releasebuffer__(&self, view: *mut pyo3::ffi::Py_buffer) {
function _tiktoken (line 252) | fn _tiktoken(_py: Python, m: &Bound<PyModule>) -> PyResult<()> {
FILE: tests/test_encoding.py
function test_simple (line 14) | def test_simple():
function test_simple_repeated (line 31) | def test_simple_repeated():
function test_large_repeated (line 52) | def test_large_repeated():
function test_simple_regex (line 60) | def test_simple_regex():
function test_basic_encode (line 69) | def test_basic_encode():
function test_encode_empty (line 81) | def test_encode_empty():
function test_encode_bytes (line 86) | def test_encode_bytes():
function test_hyp_encode_bytes (line 97) | def test_hyp_encode_bytes(make_enc: Callable[[], tiktoken.Encoding], byt...
function test_encode_surrogate_pairs (line 102) | def test_encode_surrogate_pairs():
function test_catastrophically_repetitive (line 114) | def test_catastrophically_repetitive(make_enc: Callable[[], tiktoken.Enc...
function test_basic_roundtrip (line 133) | def test_basic_roundtrip(make_enc):
function test_hyp_roundtrip (line 152) | def test_hyp_roundtrip(make_enc: Callable[[], tiktoken.Encoding], text):
function test_single_token_roundtrip (line 159) | def test_single_token_roundtrip(make_enc: Callable[[], tiktoken.Encoding]):
function test_special_token (line 175) | def test_special_token():
function test_hyp_special_ordinary (line 229) | def test_hyp_special_ordinary(make_enc, text: str):
function test_batch_encode (line 240) | def test_batch_encode(make_enc: Callable[[], tiktoken.Encoding]):
function test_hyp_batch_roundtrip (line 258) | def test_hyp_batch_roundtrip(make_enc: Callable[[], tiktoken.Encoding], ...
FILE: tests/test_misc.py
function test_encoding_for_model (line 7) | def test_encoding_for_model():
function test_optional_blobfile_dependency (line 24) | def test_optional_blobfile_dependency():
FILE: tests/test_offsets.py
function _common_prefix_len (line 12) | def _common_prefix_len(a, b):
function _token_offsets_reference (line 19) | def _token_offsets_reference(enc, tokens):
function test_hyp_offsets (line 31) | def test_hyp_offsets(make_enc: Callable[[], tiktoken.Encoding], data):
function test_basic_offsets (line 49) | def test_basic_offsets():
FILE: tests/test_pickle.py
function test_pickle (line 4) | def test_pickle():
FILE: tests/test_simple_public.py
function test_simple (line 7) | def test_simple():
function test_encoding_for_model (line 25) | def test_encoding_for_model():
function test_optional_blobfile_dependency (line 36) | def test_optional_blobfile_dependency():
FILE: tiktoken/_educational.py
class SimpleBytePairEncoding (line 12) | class SimpleBytePairEncoding:
method __init__ (line 13) | def __init__(self, *, pat_str: str, mergeable_ranks: dict[bytes, int])...
method encode (line 23) | def encode(self, text: str, visualise: str | None = "colour") -> list[...
method decode_bytes (line 39) | def decode_bytes(self, tokens: list[int]) -> bytes:
method decode (line 47) | def decode(self, tokens: list[int]) -> str:
method decode_tokens_bytes (line 58) | def decode_tokens_bytes(self, tokens: list[int]) -> list[bytes]:
method train (line 69) | def train(training_data: str, vocab_size: int, pat_str: str):
method from_tiktoken (line 75) | def from_tiktoken(encoding):
function bpe_encode (line 83) | def bpe_encode(
function bpe_train (line 119) | def bpe_train(
function visualise_tokens (line 188) | def visualise_tokens(token_values: list[bytes]) -> None:
function train_simple_encoding (line 208) | def train_simple_encoding():
FILE: tiktoken/core.py
class Encoding (line 16) | class Encoding:
method __init__ (line 17) | def __init__(
method __repr__ (line 56) | def __repr__(self) -> str:
method encode_ordinary (line 63) | def encode_ordinary(self, text: str) -> list[int]:
method encode (line 79) | def encode(
method encode_to_numpy (line 135) | def encode_to_numpy(
method encode_ordinary_batch (line 161) | def encode_ordinary_batch(self, text: list[str], *, num_threads: int =...
method encode_batch (line 175) | def encode_batch(
method encode_with_unstable (line 205) | def encode_with_unstable(
method encode_single_token (line 242) | def encode_single_token(self, text_or_bytes: str | bytes) -> int:
method decode_bytes (line 262) | def decode_bytes(self, tokens: Sequence[int]) -> bytes:
method decode (line 272) | def decode(self, tokens: Sequence[int], errors: str = "replace") -> str:
method decode_single_token_bytes (line 286) | def decode_single_token_bytes(self, token: int) -> bytes:
method decode_tokens_bytes (line 300) | def decode_tokens_bytes(self, tokens: Sequence[int]) -> list[bytes]:
method decode_with_offsets (line 309) | def decode_with_offsets(self, tokens: Sequence[int]) -> tuple[str, lis...
method decode_batch (line 334) | def decode_batch(
method decode_bytes_batch (line 342) | def decode_bytes_batch(
method token_byte_values (line 353) | def token_byte_values(self) -> list[bytes]:
method eot_token (line 358) | def eot_token(self) -> int:
method special_tokens_set (line 362) | def special_tokens_set(self) -> set[str]:
method is_special_token (line 365) | def is_special_token(self, token: int) -> bool:
method n_vocab (line 370) | def n_vocab(self) -> int:
method _encode_single_piece (line 378) | def _encode_single_piece(self, text_or_bytes: str | bytes) -> list[int]:
method _encode_only_native_bpe (line 392) | def _encode_only_native_bpe(self, text: str) -> list[int]:
method _encode_bytes (line 403) | def _encode_bytes(self, text: bytes) -> list[int]:
method __getstate__ (line 406) | def __getstate__(self) -> object:
method __setstate__ (line 419) | def __setstate__(self, value: object) -> None:
function _special_token_regex (line 429) | def _special_token_regex(tokens: frozenset[str]) -> re.Pattern[str]:
function raise_disallowed_special_token (line 438) | def raise_disallowed_special_token(token: str) -> NoReturn:
FILE: tiktoken/load.py
function read_file (line 8) | def read_file(blobpath: str) -> bytes:
function check_hash (line 30) | def check_hash(data: bytes, expected_hash: str) -> bool:
function read_file_cached (line 35) | def read_file_cached(blobpath: str, expected_hash: str | None = None) ->...
function data_gym_to_mergeable_bpe_ranks (line 89) | def data_gym_to_mergeable_bpe_ranks(
function dump_tiktoken_bpe (line 147) | def dump_tiktoken_bpe(bpe_ranks: dict[bytes, int], tiktoken_bpe_file: st...
function load_tiktoken_bpe (line 159) | def load_tiktoken_bpe(tiktoken_bpe_file: str, expected_hash: str | None ...
FILE: tiktoken/model.py
function encoding_name_for_model (line 88) | def encoding_name_for_model(model_name: str) -> str:
function encoding_for_model (line 113) | def encoding_for_model(model_name: str) -> Encoding:
FILE: tiktoken/registry.py
function _available_plugin_modules (line 20) | def _available_plugin_modules() -> Sequence[str]:
function _find_constructors (line 33) | def _find_constructors() -> None:
function get_encoding (line 63) | def get_encoding(encoding_name: str) -> Encoding:
function list_encoding_names (line 91) | def list_encoding_names() -> list[str]:
FILE: tiktoken_ext/openai_public.py
function gpt2 (line 17) | def gpt2():
function r50k_base (line 33) | def r50k_base():
function p50k_base (line 47) | def p50k_base():
function p50k_edit (line 61) | def p50k_edit():
function cl100k_base (line 75) | def cl100k_base():
function o200k_base (line 95) | def o200k_base():
function o200k_harmony (line 123) | def o200k_harmony():
Condensed preview — 29 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (121K chars).
[
{
"path": ".github/workflows/build_wheels.yml",
"chars": 2861,
"preview": "name: Build wheels\n\non: [push, pull_request, workflow_dispatch]\n\nconcurrency:\n group: ${{ github.workflow }}-${{ github"
},
{
"path": ".gitignore",
"chars": 403,
"preview": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packagi"
},
{
"path": "CHANGELOG.md",
"chars": 3583,
"preview": "# Changelog\n\nThis is the changelog for the open source version of tiktoken.\n\n## [v0.12.0]\n- Build wheels for Python 3.14"
},
{
"path": "Cargo.toml",
"chars": 411,
"preview": "[package]\nname = \"tiktoken\"\nversion = \"0.12.0\"\nedition = \"2024\"\n\n[lib]\nname = \"tiktoken\"\ncrate-type = [\"cdylib\", \"rlib\"]"
},
{
"path": "LICENSE",
"chars": 1078,
"preview": "MIT License\n\nCopyright (c) 2022 OpenAI, Shantanu Jain\n\nPermission is hereby granted, free of charge, to any person obtai"
},
{
"path": "MANIFEST.in",
"chars": 170,
"preview": "include *.svg\ninclude *.toml\ninclude *.md\ninclude Makefile\nglobal-include py.typed\nrecursive-include scripts *.py\nrecurs"
},
{
"path": "README.md",
"chars": 4783,
"preview": "# ⏳ tiktoken\n\ntiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with\nOpenAI's"
},
{
"path": "pyproject.toml",
"chars": 1405,
"preview": "[project]\nname = \"tiktoken\"\nversion = \"0.12.0\"\ndescription = \"tiktoken is a fast BPE tokeniser for use with OpenAI's mod"
},
{
"path": "scripts/benchmark.py",
"chars": 1000,
"preview": "import base64\nimport functools\nimport gzip\nimport json\nimport os\nimport random\nimport time\nfrom typing import Any, cast\n"
},
{
"path": "scripts/redact.py",
"chars": 1746,
"preview": "import argparse\nimport re\nimport subprocess\nfrom pathlib import Path\n\n\ndef redact_file(path: Path, dry_run: bool) -> Non"
},
{
"path": "scripts/wheel_download.py",
"chars": 2040,
"preview": "import argparse\nimport zipfile\nfrom pathlib import Path\n\nimport requests\n\n\ndef download_artifacts(token, owner, repo, ru"
},
{
"path": "setup.py",
"chars": 572,
"preview": "from setuptools import setup\nfrom setuptools_rust import Binding, RustExtension\n\nsetup(\n name=\"tiktoken\",\n rust_ex"
},
{
"path": "src/lib.rs",
"chars": 26068,
"preview": "use std::collections::HashSet;\nuse std::num::NonZeroU64;\nuse std::thread;\n\nuse fancy_regex::Regex;\n#[cfg(feature = \"pyth"
},
{
"path": "src/py.rs",
"chars": 9203,
"preview": "use std::collections::HashSet;\n\nuse pyo3::{\n IntoPyObjectExt, PyResult, exceptions,\n prelude::*,\n pybacked::PyB"
},
{
"path": "tests/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "tests/test_encoding.py",
"chars": 8733,
"preview": "# Note that there are more actual tests, they're just not currently public :-)\n\nfrom typing import Callable\n\nimport hypo"
},
{
"path": "tests/test_helpers.py",
"chars": 477,
"preview": "import bisect\nimport functools\nimport os\n\nimport pytest\n\nimport tiktoken\n\nMAX_EXAMPLES: int = int(os.environ.get(\"TIKTOK"
},
{
"path": "tests/test_misc.py",
"chars": 886,
"preview": "import subprocess\nimport sys\n\nimport tiktoken\n\n\ndef test_encoding_for_model():\n enc = tiktoken.encoding_for_model(\"gp"
},
{
"path": "tests/test_offsets.py",
"chars": 2590,
"preview": "from typing import Callable\n\nimport hypothesis\nimport pytest\nfrom hypothesis import strategies as st\n\nimport tiktoken\n\nf"
},
{
"path": "tests/test_pickle.py",
"chars": 715,
"preview": "import tiktoken\n\n\ndef test_pickle():\n import pickle\n\n enc_old = tiktoken.get_encoding(\"r50k_base\")\n enc_new = p"
},
{
"path": "tests/test_simple_public.py",
"chars": 1439,
"preview": "import subprocess\nimport sys\n\nimport tiktoken\n\n\ndef test_simple():\n # Note that there are more actual tests, they're "
},
{
"path": "tiktoken/__init__.py",
"chars": 346,
"preview": "# This is the public API of tiktoken\nfrom .core import Encoding as Encoding\nfrom .model import encoding_for_model as enc"
},
{
"path": "tiktoken/_educational.py",
"chars": 8227,
"preview": "\"\"\"This is an educational implementation of the byte pair encoding algorithm.\"\"\"\n\nfrom __future__ import annotations\n\nim"
},
{
"path": "tiktoken/core.py",
"chars": 17458,
"preview": "from __future__ import annotations\n\nimport functools\nfrom concurrent.futures import ThreadPoolExecutor\nfrom typing impor"
},
{
"path": "tiktoken/load.py",
"chars": 5887,
"preview": "from __future__ import annotations\n\nimport base64\nimport hashlib\nimport os\n\n\ndef read_file(blobpath: str) -> bytes:\n "
},
{
"path": "tiktoken/model.py",
"chars": 4061,
"preview": "from __future__ import annotations\n\nfrom .core import Encoding\nfrom .registry import get_encoding\n\n# TODO: these will li"
},
{
"path": "tiktoken/py.typed",
"chars": 0,
"preview": ""
},
{
"path": "tiktoken/registry.py",
"chars": 3256,
"preview": "from __future__ import annotations\n\nimport functools\nimport importlib\nimport pkgutil\nimport threading\nfrom typing import"
},
{
"path": "tiktoken_ext/openai_public.py",
"chars": 5613,
"preview": "from tiktoken.load import data_gym_to_mergeable_bpe_ranks, load_tiktoken_bpe\n\nENDOFTEXT = \"<|endoftext|>\"\nFIM_PREFIX = \""
}
]
About this extraction
This page contains the full source code of the openai/tiktoken GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 29 files (112.3 KB), approximately 30.0k tokens, and a symbol index with 140 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.