Copy disabled (too large)
Download .txt
Showing preview only (12,966K chars total). Download the full file to get everything.
Repository: MaartenGr/BERTopic
Branch: master
Commit: b2ce08422250
Files: 184
Total size: 12.3 MB
Directory structure:
gitextract_s6becou6/
├── .git-blame-ignore-revs
├── .gitattributes
├── .github/
│ ├── ISSUE_TEMPLATE/
│ │ ├── bug_report.yml
│ │ ├── config.yml
│ │ └── feature_request.yml
│ ├── PULL_REQUEST_TEMPLATE.md
│ └── workflows/
│ └── testing.yml
├── .gitignore
├── .pre-commit-config.yaml
├── CONTRIBUTING.md
├── LICENSE
├── Makefile
├── README.md
├── bertopic/
│ ├── __init__.py
│ ├── _bertopic.py
│ ├── _save_utils.py
│ ├── _utils.py
│ ├── backend/
│ │ ├── __init__.py
│ │ ├── _base.py
│ │ ├── _cohere.py
│ │ ├── _fastembed.py
│ │ ├── _flair.py
│ │ ├── _gensim.py
│ │ ├── _hftransformers.py
│ │ ├── _langchain.py
│ │ ├── _model2vec.py
│ │ ├── _multimodal.py
│ │ ├── _openai.py
│ │ ├── _sentencetransformers.py
│ │ ├── _sklearn.py
│ │ ├── _spacy.py
│ │ ├── _use.py
│ │ ├── _utils.py
│ │ └── _word_doc.py
│ ├── cluster/
│ │ ├── __init__.py
│ │ ├── _base.py
│ │ └── _utils.py
│ ├── dimensionality/
│ │ ├── __init__.py
│ │ └── _base.py
│ ├── plotting/
│ │ ├── __init__.py
│ │ ├── _approximate_distribution.py
│ │ ├── _barchart.py
│ │ ├── _datamap.py
│ │ ├── _distribution.py
│ │ ├── _documents.py
│ │ ├── _heatmap.py
│ │ ├── _hierarchical_documents.py
│ │ ├── _hierarchy.py
│ │ ├── _term_rank.py
│ │ ├── _topics.py
│ │ ├── _topics_over_time.py
│ │ └── _topics_per_class.py
│ ├── representation/
│ │ ├── __init__.py
│ │ ├── _base.py
│ │ ├── _cohere.py
│ │ ├── _keybert.py
│ │ ├── _langchain.py
│ │ ├── _litellm.py
│ │ ├── _llamacpp.py
│ │ ├── _mmr.py
│ │ ├── _openai.py
│ │ ├── _pos.py
│ │ ├── _textgeneration.py
│ │ ├── _utils.py
│ │ ├── _visual.py
│ │ └── _zeroshot.py
│ └── vectorizers/
│ ├── __init__.py
│ ├── _ctfidf.py
│ └── _online_cv.py
├── docs/
│ ├── algorithm/
│ │ └── algorithm.md
│ ├── api/
│ │ ├── backends.md
│ │ ├── bertopic.md
│ │ ├── cluster copy.md
│ │ ├── cluster.md
│ │ ├── ctfidf.md
│ │ ├── dimensionality.md
│ │ ├── plotting/
│ │ │ ├── barchart.md
│ │ │ ├── distribution.md
│ │ │ ├── document_datamap.md
│ │ │ ├── documents.md
│ │ │ ├── dtm.md
│ │ │ ├── heatmap.md
│ │ │ ├── hierarchical_documents.md
│ │ │ ├── hierarchy.md
│ │ │ ├── term.md
│ │ │ ├── topics.md
│ │ │ └── topics_per_class.md
│ │ ├── plotting.md
│ │ ├── representations.md
│ │ └── vectorizers.md
│ ├── changelog.md
│ ├── faq.md
│ ├── getting_started/
│ │ ├── best_practices/
│ │ │ └── best_practices.md
│ │ ├── clustering/
│ │ │ └── clustering.md
│ │ ├── ctfidf/
│ │ │ └── ctfidf.md
│ │ ├── dim_reduction/
│ │ │ └── dim_reduction.md
│ │ ├── distribution/
│ │ │ ├── distribution.md
│ │ │ └── distribution_viz.html
│ │ ├── embeddings/
│ │ │ └── embeddings.md
│ │ ├── guided/
│ │ │ └── guided.md
│ │ ├── hierarchicaltopics/
│ │ │ ├── hierarchical_topics.html
│ │ │ └── hierarchicaltopics.md
│ │ ├── manual/
│ │ │ └── manual.md
│ │ ├── merge/
│ │ │ └── merge.md
│ │ ├── multiaspect/
│ │ │ └── multiaspect.md
│ │ ├── multimodal/
│ │ │ └── multimodal.md
│ │ ├── online/
│ │ │ └── online.md
│ │ ├── outlier_reduction/
│ │ │ ├── fig_base.html
│ │ │ ├── fig_reduced.html
│ │ │ └── outlier_reduction.md
│ │ ├── parameter tuning/
│ │ │ └── parametertuning.md
│ │ ├── quickstart/
│ │ │ ├── quickstart.md
│ │ │ └── viz.html
│ │ ├── representation/
│ │ │ ├── llm.md
│ │ │ └── representation.md
│ │ ├── search/
│ │ │ └── search.md
│ │ ├── seed_words/
│ │ │ └── seed_words.md
│ │ ├── semisupervised/
│ │ │ └── semisupervised.md
│ │ ├── serialization/
│ │ │ └── serialization.md
│ │ ├── supervised/
│ │ │ └── supervised.md
│ │ ├── tips_and_tricks/
│ │ │ └── tips_and_tricks.md
│ │ ├── topicreduction/
│ │ │ └── topicreduction.md
│ │ ├── topicrepresentation/
│ │ │ └── topicrepresentation.md
│ │ ├── topicsovertime/
│ │ │ ├── topicsovertime.md
│ │ │ └── trump.html
│ │ ├── topicsperclass/
│ │ │ ├── topics_per_class.html
│ │ │ └── topicsperclass.md
│ │ ├── vectorizers/
│ │ │ └── vectorizers.md
│ │ ├── visualization/
│ │ │ ├── bar_chart.html
│ │ │ ├── datamapplot.html
│ │ │ ├── documents.html
│ │ │ ├── heatmap.html
│ │ │ ├── hierarchical_documents.html
│ │ │ ├── hierarchical_topics.html
│ │ │ ├── hierarchy.html
│ │ │ ├── probabilities.html
│ │ │ ├── term_rank.html
│ │ │ ├── term_rank_log.html
│ │ │ ├── topics_per_class.html
│ │ │ ├── trump.html
│ │ │ ├── visualization.md
│ │ │ ├── visualize_documents.md
│ │ │ ├── visualize_hierarchy.md
│ │ │ ├── visualize_terms.md
│ │ │ ├── visualize_topics.md
│ │ │ └── viz.html
│ │ └── zeroshot/
│ │ └── zeroshot.md
│ ├── img/
│ │ └── probabilities.html
│ ├── index.md
│ ├── stylesheets/
│ │ └── extra.css
│ └── usecases.md
├── mkdocs.yml
├── pyproject.toml
└── tests/
├── __init__.py
├── conftest.py
├── test_bertopic.py
├── test_other.py
├── test_plotting/
│ ├── __init__.py
│ ├── test_approximate.py
│ ├── test_bar.py
│ ├── test_documents.py
│ ├── test_dynamic.py
│ ├── test_heatmap.py
│ ├── test_term_rank.py
│ └── test_topics.py
├── test_reduction/
│ ├── __init__.py
│ ├── test_delete.py
│ └── test_merge.py
├── test_representation/
│ ├── __init__.py
│ ├── test_get.py
│ ├── test_labels.py
│ └── test_representations.py
├── test_sub_models/
│ ├── __init__.py
│ ├── test_cluster.py
│ ├── test_dim_reduction.py
│ └── test_embeddings.py
├── test_utils.py
├── test_variations/
│ ├── __init__.py
│ ├── test_class.py
│ ├── test_dynamic.py
│ └── test_hierarchy.py
└── test_vectorizers/
├── __init__.py
├── test_ctfidf.py
└── test_online_cv.py
================================================
FILE CONTENTS
================================================
================================================
FILE: .git-blame-ignore-revs
================================================
# Initial Ruff Linting
70d7725e5c89bccfe7d4e5a3ccd87e05c642d74b
# Change line-length and ruff format
39bbfdb8298b5faa32e4bc052080d240f6140bea
# pre-commit hooks and ruff
6ed123ecc4aec9da26bd48748df670cd5b42b3cd
================================================
FILE: .gitattributes
================================================
*.ipynb linguist-documentation
================================================
FILE: .github/ISSUE_TEMPLATE/bug_report.yml
================================================
name: "\U0001F41B Bug Report"
description: Report your bug here.
labels: ["bug"]
body:
- type: markdown
attributes:
value: |
Thanks for taking the time to fill out this bug report! Any information you can provide about your system and the issue you encountered will help to resolve it faster.
- type: checkboxes
attributes:
label: Have you searched existing issues? 🔎
description: Please search to see if an [issue](https://github.com/MaartenGr/BERTopic/issues) already exists for the issue you encountered.
options:
- label: I have searched and found no existing issues
required: true
- type: textarea
id: describe_the_bug
attributes:
label: Desribe the bug
description: Please provide a concise description of the bug. If there is an error, make sure to provide the **full** error log.
placeholder: Describe the bug
validations:
required: true
- type: textarea
id: reproduction
attributes:
label: Reproduction
description: Please provide a minimal example, with code, that can be run to reproduce the issue.
value: |
```python
from bertopic import BERTopic
```
- type: input
id: bertopic_version
attributes:
label: BERTopic Version
description: What version of BERTopic are you using?
validations:
required: true
================================================
FILE: .github/ISSUE_TEMPLATE/config.yml
================================================
blank_issues_enabled: true
contact_links:
- name: 💡 General questions
url: https://github.com/MaartenGr/BERTopic/discussions
about: Ask a question there!
- name: Want to contribute?
url: https://github.com/MaartenGr/BERTopic/blob/master/CONTRIBUTING.md
about: Head to the contributing guidelines
================================================
FILE: .github/ISSUE_TEMPLATE/feature_request.yml
================================================
name: "\U0001F680 Feature request"
description: Submit a proposal/request for a new BERTopic feature
labels: ["Feature request"]
body:
- type: textarea
id: feature-request
validations:
required: true
attributes:
label: Feature request
description: |
A clear and concise description of the feature proposal.
- type: textarea
id: motivation
validations:
required: true
attributes:
label: Motivation
description: |
Please outline the motivation for the proposal. If this is related to another GitHub issue, please link here too.
- type: textarea
id: contribution
validations:
required: true
attributes:
label: Your contribution
description: |
Any help on the implementation of this feature would be greatly appreciated. If you are interested in working on this, make sure to read the [CONTRIBUTING.MD guide](https://github.com/MaartenGr/BERTopic/blob/master/CONTRIBUTING.md)
================================================
FILE: .github/PULL_REQUEST_TEMPLATE.md
================================================
# What does this PR do?
<!--
Thank you for considering creating a PR! Before you do, make sure to read through [contributor guideline](https://github.com/MaartenGr/BERTopic/blob/master/CONTRIBUTING.md)
-->
<!-- Remove if not applicable -->
Fixes # (issue)
## Before submitting
- [ ] This PR fixes a typo or improves the docs (if yes, ignore all other checks!).
- [ ] Did you read the [contributor guideline](https://github.com/MaartenGr/BERTopic/blob/master/CONTRIBUTING.md)?
- [ ] Was this discussed/approved via a Github issue? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes (if applicable)?
- [ ] Did you write any new necessary tests?
================================================
FILE: .github/workflows/testing.yml
================================================
name: Code Checks
on:
push:
branches:
- master
- dev
pull_request:
branches:
- master
- dev
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/setup-python@v6
with:
python-version: "3.13"
# Ref: https://github.com/tox-dev/action-pre-commit-uv
- uses: tox-dev/action-pre-commit-uv@246b66536e366bb885f52d61983bf32f7c95e8b1 # v1.0.3
build:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.10", "3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@v5
- name: Install latest version of uv and set up Python ${{ matrix.python-version }}
uses: astral-sh/setup-uv@v7
with:
python-version: ${{ matrix.python-version }}
activate-environment: "true"
- name: Install dependencies (including test group)
run: uv sync --extra test
- name: Display the project's dependency tree (including information on outdated packages)
run: uv tree --outdated
- name: Run Checking Mechanisms
run: make check
================================================
FILE: .gitignore
================================================
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
pip-wheel-metadata/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
model_dir
model_dir/
test
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/
.pytest_cache/
# Sphinx documentation
docs/_build/
# Jupyter Notebook
.ipynb_checkpoints
notebooks/
# IPython
profile_default/
ipython_config.py
# pyenv
.python-version
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
*.lock
# Artifacts
.idea
.idea/
.vscode
.DS_Store
# mkdocs
site/
================================================
FILE: .pre-commit-config.yaml
================================================
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v6.0.0
hooks:
- id: trailing-whitespace
exclude: |
(?x)^(
README.md|
docs/
)$
- id: end-of-file-fixer
exclude_types: [html, svg]
- id: check-yaml
- id: check-added-large-files
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.14.0
hooks:
- id: ruff-check
args: [--fix, --show-fixes, --exit-non-zero-on-fix]
- id: ruff-format
================================================
FILE: CONTRIBUTING.md
================================================
# Contributing to BERTopic
Hi! Thank you for considering contributing to BERTopic. With the modular nature of BERTopic, many new add-ons, backends, representation models, sub-models, and LLMs, can quickly be added to keep up with the incredibly fast-pacing field.
Whether contributions are new features, better documentation, bug fixes, or improvement on the repository itself, anything is appreciated!
## 📚 Guidelines
### 🤖 Contributing Code
To contribute to this project, we follow an `issue -> pull request` approach for main features and bug fixes. This means that any new feature, bug fix, or anything else that touches on code directly needs to start from an issue first. That way, the main discussion about what needs to be added/fixed can be done in the issue before creating a pull request. This makes sure that we are on the same page before you start coding your pull request. If you start working on an issue, please assign it to yourself but do so after there is an agreement with the maintainer, [@MaartenGr](https://github.com/MaartenGr).
When there is agreement on the assigned approach, a pull request can be created in which the fix/feature can be added. This follows a ["fork and pull request"](https://docs.github.com/en/get-started/quickstart/contributing-to-projects) workflow.
Please do not try to push directly to this repo unless you are a maintainer.
There are exceptions to the `issue -> pull request` approach that are typically small changes that do not need agreements, such as:
* Documentation
* Spelling/grammar issues
* Docstrings
* etc.
There is a large focus on documentation in this repository, so please make sure to add extensive descriptions of features when creating the pull request.
Note that the main focus of pull requests and code should be:
* Easy readability
* Clear communication
* Sufficient documentation
## 🚀 Quick Start
To start contributing, make sure to first start from a fresh environment. Using an environment manager, such as `conda` or `pyenv` helps in making sure that your code is reproducible and tracks the versions you have in your environment.
If you are using conda, you can approach it as follows:
1. Create and activate a new conda environment (e.g., `conda create -n bertopic python=3.10`)
2. Install requirements (e.g., `pip install .[dev]`)
* This makes sure to also install documentation and testing packages
3. (Optional) Run `make docs` to build your documentation
4. (Optional) Run `make test` to run the unit tests and `make coverage` to check the coverage of unit tests
❗Note: Unit testing the package can take quite some time since it needs to run several variants of the BERTopic pipeline.
## 🧹 Linting and Formatting
We use [Ruff](https://docs.astral.sh/ruff/) to ensure code is uniformly formatted and to avoid common mistakes and bad practices.
* To automatically re-format code, run `make format`
* To check for linting issues, run `make lint` - some issues may be automatically fixed, some will not be
When a pull request is made, the CI will automatically check for linting and formatting issues. However, it will not automatically apply any fixes, so it is easiest to run locally.
If you believe an error is incorrectly flagged, use a [`# noqa:` comment to suppress](https://docs.astral.sh/ruff/linter/#error-suppression), but this is discouraged unless strictly necessary.
## 🤓 Collaborative Efforts
When you run into any issue with the above or need help to start with a pull request, feel free to reach out in the issues! As with all repositories, this one has its particularities as a result of the maintainer's view. Each repository is quite different and so will their processes.
## 🏆 Recognition
If your contribution has made its way into a new release of BERTopic, you will be given credit in the changelog of the new release! Regardless of the size of the contribution, any help is greatly appreciated.
## 🎈 Release
BERTopic tries to mostly follow [semantic versioning](https://semver.org/) for its new releases. Even though BERTopic has been around for a few years now, it is still pre-1.0 software. With the rapid chances in the field and as a way to keep up, this versioning is on purpose. Backwards-compatibility is taken into account but integrating new features and thereby keeping up with the field takes priority. Especially since BERTopic focuses on modularity, flexibility is necessary.
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2024, Maarten P. Grootendorst
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: Makefile
================================================
test:
pytest
test-no-plotly:
uv sync --extra test
uv pip uninstall plotly
pytest tests/test_other.py -k plotly --pdb
uv sync --extra test
pytest tests/test_other.py -k plotly
coverage:
pytest --cov
format:
ruff format
lint:
ruff check --fix
install:
python -m pip install -e .
install-test:
python -m pip install -e ".[dev]"
docs:
mkdocs serve
pypi:
python -m build
twine upload dist/*
clean:
rm -rf **/.ipynb_checkpoints **/.pytest_cache **/__pycache__ **/**/__pycache__ .ipynb_checkpoints .pytest_cache
check: test clean
================================================
FILE: README.md
================================================
[](https://pepy.tech/projects/bertopic)
[](https://pypi.org/project/bertopic/)
[](https://github.com/MaartenGr/BERTopic/actions)
[](https://maartengr.github.io/BERTopic/)
[](https://pypi.org/project/bertopic/)
[](https://github.com/MaartenGr/VLAC/blob/master/LICENSE)
[](https://arxiv.org/abs/2203.05794)
# BERTopic
<img src="images/logo.png" width="35%" align="right" />
BERTopic is a topic modeling technique that leverages 🤗 transformers and c-TF-IDF to create dense clusters
allowing for easily interpretable topics whilst keeping important words in the topic descriptions.
BERTopic supports all kinds of topic modeling techniques:
<table>
<tr>
<td><a href="https://maartengr.github.io/BERTopic/getting_started/guided/guided.html">Guided</a></td>
<td><a href="https://maartengr.github.io/BERTopic/getting_started/supervised/supervised.html">Supervised</a></td>
<td><a href="https://maartengr.github.io/BERTopic/getting_started/semisupervised/semisupervised.html">Semi-supervised</a></td>
</tr>
<tr>
<td><a href="https://maartengr.github.io/BERTopic/getting_started/manual/manual.html">Manual</a></td>
<td><a href="https://maartengr.github.io/BERTopic/getting_started/distribution/distribution.html">Multi-topic distributions</a></td>
<td><a href="https://maartengr.github.io/BERTopic/getting_started/hierarchicaltopics/hierarchicaltopics.html">Hierarchical</a></td>
</tr>
<tr>
<td><a href="https://maartengr.github.io/BERTopic/getting_started/topicsperclass/topicsperclass.html">Class-based</a></td>
<td><a href="https://maartengr.github.io/BERTopic/getting_started/topicsovertime/topicsovertime.html">Dynamic</a></td>
<td><a href="https://maartengr.github.io/BERTopic/getting_started/online/online.html">Online/Incremental</a></td>
</tr>
<tr>
<td><a href="https://maartengr.github.io/BERTopic/getting_started/multimodal/multimodal.html">Multimodal</a></td>
<td><a href="https://maartengr.github.io/BERTopic/getting_started/multiaspect/multiaspect.html">Multi-aspect</a></td>
<td><a href="https://maartengr.github.io/BERTopic/getting_started/representation/llm.html">Text Generation/LLM</a></td>
</tr>
<tr>
<td><a href="https://maartengr.github.io/BERTopic/getting_started/zeroshot/zeroshot.html">Zero-shot <b>(new!)</b></a></td>
<td><a href="https://maartengr.github.io/BERTopic/getting_started/merge/merge.html">Merge Models <b>(new!)</b></a></td>
<td><a href="https://maartengr.github.io/BERTopic/getting_started/seed_words/seed_words.html">Seed Words <b>(new!)</b></a></td>
</tr>
</table>
Corresponding medium posts can be found [here](https://medium.com/data-science/topic-modeling-with-bert-779f7db187e6?sk=0b5a470c006d1842ad4c8a3057063a99
), [here](https://medium.com/data-science/using-whisper-and-bertopic-to-model-kurzgesagts-videos-7d8a63139bdf?sk=b1e0fd46f70cb15e8422b4794a81161d
) and [here](https://medium.com/data-science/interactive-topic-modeling-with-bertopic-1ea55e7d73d8?sk=03c2168e9e74b6bda2a1f3ed953427e4
). For a more detailed overview, you can read the [paper](https://arxiv.org/abs/2203.05794) or see a [brief overview](https://maartengr.github.io/BERTopic/algorithm/algorithm.html).
## Installation
Installation, with sentence-transformers, can be done using [uv](https://docs.astral.sh/uv/):
```bash
uv add bertopic
```
or with [pip](https://github.com/pypa/pip):
```bash
pip install bertopic
```
If you want to install BERTopic with other embedding models, you can choose one of the following:
```bash
# Choose an embedding backend
pip install bertopic[flair,gensim,spacy,use]
# Topic modeling with images
pip install bertopic[vision]
```
For a *light-weight installation* without transformers, UMAP and/or HDBSCAN (for training with Model2Vec or inference), see [this tutorial](https://maartengr.github.io/BERTopic/getting_started/tips_and_tricks/tips_and_tricks.html#lightweight-installation).
## Getting Started
For an in-depth overview of the features of BERTopic
you can check the [**full documentation**](https://maartengr.github.io/BERTopic/) or you can follow along
with one of the examples below:
| Name | Link |
|---|---|
| Start Here - **Best Practices in BERTopic** | [](https://colab.research.google.com/drive/1BoQ_vakEVtojsd2x_U6-_x52OOuqruj2?usp=sharing) |
| **🆕 New!** - Topic Modeling on Large Data (GPU Acceleration) | [](https://colab.research.google.com/drive/1W7aEdDPxC29jP99GGZphUlqjMFFVKtBC?usp=sharing) |
| **🆕 New!** - Topic Modeling with Llama 2 🦙 | [](https://colab.research.google.com/drive/1QCERSMUjqGetGGujdrvv_6_EeoIcd_9M?usp=sharing) |
| **🆕 New!** - Topic Modeling with Quantized LLMs | [](https://colab.research.google.com/drive/1DdSHvVPJA3rmNfBWjCo2P1E9686xfxFx?usp=sharing) |
| Topic Modeling with BERTopic | [](https://colab.research.google.com/drive/1FieRA9fLdkQEGDIMYl0I3MCjSUKVF8C-?usp=sharing) |
| (Custom) Embedding Models in BERTopic | [](https://colab.research.google.com/drive/18arPPe50szvcCp_Y6xS56H2tY0m-RLqv?usp=sharing) |
| Advanced Customization in BERTopic | [](https://colab.research.google.com/drive/1ClTYut039t-LDtlcd-oQAdXWgcsSGTw9?usp=sharing) |
| (semi-)Supervised Topic Modeling with BERTopic | [](https://colab.research.google.com/drive/1bxizKzv5vfxJEB29sntU__ZC7PBSIPaQ?usp=sharing) |
| Dynamic Topic Modeling with Trump's Tweets | [](https://colab.research.google.com/drive/1un8ooI-7ZNlRoK0maVkYhmNRl0XGK88f?usp=sharing) |
| Topic Modeling arXiv Abstracts | [](https://www.kaggle.com/maartengr/topic-modeling-arxiv-abstract-with-bertopic) |
## Quick Start
We start by extracting topics from the well-known 20 newsgroups dataset containing English documents:
```python
from bertopic import BERTopic
from sklearn.datasets import fetch_20newsgroups
docs = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes'))['data']
topic_model = BERTopic()
topics, probs = topic_model.fit_transform(docs)
```
After generating topics and their probabilities, we can access all of the topics together with their topic representations:
```python
>>> topic_model.get_topic_info()
Topic Count Name
-1 4630 -1_can_your_will_any
0 693 49_windows_drive_dos_file
1 466 32_jesus_bible_christian_faith
2 441 2_space_launch_orbit_lunar
3 381 22_key_encryption_keys_encrypted
...
```
The `-1` topic refers to all outlier documents and are typically ignored. Each word in a topic describes the underlying theme of that topic and can be used
for interpreting that topic. Next, let's take a look at the most frequent topic that was generated:
```python
>>> topic_model.get_topic(0)
[('windows', 0.006152228076250982),
('drive', 0.004982897610645755),
('dos', 0.004845038866360651),
('file', 0.004140142872194834),
('disk', 0.004131678774810884),
('mac', 0.003624848635985097),
('memory', 0.0034840976976789903),
('software', 0.0034415334250699077),
('email', 0.0034239554442333257),
('pc', 0.003047105930670237)]
```
Using `.get_document_info`, we can also extract information on a document level, such as their corresponding topics, probabilities, whether they are representative documents for a topic, etc.:
```python
>>> topic_model.get_document_info(docs)
Document Topic Name Top_n_words Probability ...
I am sure some bashers of Pens... 0 0_game_team_games_season game - team - games... 0.200010 ...
My brother is in the market for... -1 -1_can_your_will_any can - your - will... 0.420668 ...
Finally you said what you dream... -1 -1_can_your_will_any can - your - will... 0.807259 ...
Think! It's the SCSI card doing... 49 49_windows_drive_dos_file windows - drive - docs... 0.071746 ...
1) I have an old Jasmine drive... 49 49_windows_drive_dos_file windows - drive - docs... 0.038983 ...
```
**`🔥 Tip`**: Use `BERTopic(language="multilingual")` to select a model that supports 50+ languages.
## Fine-tune Topic Representations
In BERTopic, there are a number of different [topic representations](https://maartengr.github.io/BERTopic/getting_started/representation/representation.html) that we can choose from. They are all quite different from one another and give interesting perspectives and variations of topic representations. A great start is `KeyBERTInspired`, which for many users increases the coherence and reduces stopwords from the resulting topic representations:
```python
from bertopic.representation import KeyBERTInspired
# Fine-tune your topic representations
representation_model = KeyBERTInspired()
topic_model = BERTopic(representation_model=representation_model)
```
However, you might want to use something more powerful to describe your clusters. You can even use ChatGPT or other models from OpenAI to generate labels, summaries, phrases, keywords, and more:
```python
import openai
from bertopic.representation import OpenAI
# Fine-tune topic representations with GPT
client = openai.OpenAI(api_key="sk-...")
representation_model = OpenAI(client, model="gpt-4o-mini", chat=True)
topic_model = BERTopic(representation_model=representation_model)
```
**`🔥 Tip`**: Instead of iterating over all of these different topic representations, you can model them simultaneously with [multi-aspect topic representations](https://maartengr.github.io/BERTopic/getting_started/multiaspect/multiaspect.html) in BERTopic.
## Visualizations
After having trained our BERTopic model, we can iteratively go through hundreds of topics to get a good
understanding of the topics that were extracted. However, that takes quite some time and lacks a global representation. Instead, we can use one of the [many visualization options](https://maartengr.github.io/BERTopic/getting_started/visualization/visualization.html) in BERTopic.
For example, we can visualize the topics that were generated in a way very similar to
[LDAvis](https://github.com/cpsievert/LDAvis):
```python
topic_model.visualize_topics()
```
<img src="images/topic_visualization.gif" width="80%" align="center" />
## Modularity
By default, the [main steps](https://maartengr.github.io/BERTopic/algorithm/algorithm.html) for topic modeling with BERTopic are sentence-transformers, UMAP, HDBSCAN, and c-TF-IDF run in sequence. However, it assumes some independence between these steps which makes BERTopic quite modular. In other words, BERTopic not only allows you to build your own topic model but to explore several topic modeling techniques on top of your customized topic model:
https://user-images.githubusercontent.com/25746895/218420473-4b2bb539-9dbe-407a-9674-a8317c7fb3bf.mp4
You can swap out any of these models or even remove them entirely. The following steps are completely modular:
1. [Embedding](https://maartengr.github.io/BERTopic/getting_started/embeddings/embeddings.html) documents
2. [Reducing dimensionality](https://maartengr.github.io/BERTopic/getting_started/dim_reduction/dim_reduction.html) of embeddings
3. [Clustering](https://maartengr.github.io/BERTopic/getting_started/clustering/clustering.html) reduced embeddings into topics
4. [Tokenization](https://maartengr.github.io/BERTopic/getting_started/vectorizers/vectorizers.html) of topics
5. [Weight](https://maartengr.github.io/BERTopic/getting_started/ctfidf/ctfidf.html) tokens
6. [Represent topics](https://maartengr.github.io/BERTopic/getting_started/representation/representation.html) with one or [multiple](https://maartengr.github.io/BERTopic/getting_started/multiaspect/multiaspect.html) representations
## Functionality
BERTopic has many functions that quickly can become overwhelming. To alleviate this issue, you will find an overview
of all methods and a short description of its purpose.
### Common
Below, you will find an overview of common functions in BERTopic.
| Method | Code |
|-----------------------|---|
| Fit the model | `.fit(docs)` |
| Fit the model and predict documents | `.fit_transform(docs)` |
| Predict new documents | `.transform([new_doc])` |
| Access single topic | `.get_topic(topic=12)` |
| Access all topics | `.get_topics()` |
| Get topic freq | `.get_topic_freq()` |
| Get all topic information| `.get_topic_info()` |
| Get all document information| `.get_document_info(docs)` |
| Get representative docs per topic | `.get_representative_docs()` |
| Update topic representation | `.update_topics(docs, n_gram_range=(1, 3))` |
| Generate topic labels | `.generate_topic_labels()` |
| Set topic labels | `.set_topic_labels(my_custom_labels)` |
| Merge topics | `.merge_topics(docs, topics_to_merge)` |
| Reduce nr of topics | `.reduce_topics(docs, nr_topics=30)` |
| Reduce outliers | `.reduce_outliers(docs, topics)` |
| Find topics | `.find_topics("vehicle")` |
| Save model | `.save("my_model", serialization="safetensors")` |
| Load model | `BERTopic.load("my_model")` |
| Get parameters | `.get_params()` |
### Attributes
After having trained your BERTopic model, several attributes are saved within your model. These attributes, in part,
refer to how model information is stored on an estimator during fitting. The attributes that you see below all end in `_` and are
public attributes that can be used to access model information.
| Attribute | Description |
|------------------------|---------------------------------------------------------------------------------------------|
| `.topics_` | The topics that are generated for each document after training or updating the topic model. |
| `.probabilities_` | The probabilities that are generated for each document if HDBSCAN is used. |
| `.topic_sizes_` | The size of each topic |
| `.topic_mapper_` | A class for tracking topics and their mappings anytime they are merged/reduced. |
| `.topic_representations_` | The top *n* terms per topic and their respective c-TF-IDF values. |
| `.c_tf_idf_` | The topic-term matrix as calculated through c-TF-IDF. |
| `.topic_aspects_` | The different aspects, or representations, of each topic. |
| `.topic_labels_` | The default labels for each topic. |
| `.custom_labels_` | Custom labels for each topic as generated through `.set_topic_labels`. |
| `.topic_embeddings_` | The embeddings for each topic if `embedding_model` was used. |
| `.representative_docs_` | The representative documents for each topic if HDBSCAN is used. |
### Variations
There are many different use cases in which topic modeling can be used. As such, several variations of BERTopic have been developed such that one package can be used across many use cases.
| Method | Code |
|-----------------------|---|
| [Topic Distribution Approximation](https://maartengr.github.io/BERTopic/getting_started/distribution/distribution.html) | `.approximate_distribution(docs)` |
| [Online Topic Modeling](https://maartengr.github.io/BERTopic/getting_started/online/online.html) | `.partial_fit(doc)` |
| [Semi-supervised Topic Modeling](https://maartengr.github.io/BERTopic/getting_started/semisupervised/semisupervised.html) | `.fit(docs, y=y)` |
| [Supervised Topic Modeling](https://maartengr.github.io/BERTopic/getting_started/supervised/supervised.html) | `.fit(docs, y=y)` |
| [Manual Topic Modeling](https://maartengr.github.io/BERTopic/getting_started/manual/manual.html) | `.fit(docs, y=y)` |
| [Multimodal Topic Modeling](https://maartengr.github.io/BERTopic/getting_started/multimodal/multimodal.html) | ``.fit(docs, images=images)`` |
| [Topic Modeling per Class](https://maartengr.github.io/BERTopic/getting_started/topicsperclass/topicsperclass.html) | `.topics_per_class(docs, classes)` |
| [Dynamic Topic Modeling](https://maartengr.github.io/BERTopic/getting_started/topicsovertime/topicsovertime.html) | `.topics_over_time(docs, timestamps)` |
| [Hierarchical Topic Modeling](https://maartengr.github.io/BERTopic/getting_started/hierarchicaltopics/hierarchicaltopics.html) | `.hierarchical_topics(docs)` |
| [Guided Topic Modeling](https://maartengr.github.io/BERTopic/getting_started/guided/guided.html) | `BERTopic(seed_topic_list=seed_topic_list)` |
| [Zero-shot Topic Modeling](https://maartengr.github.io/BERTopic/getting_started/zeroshot/zeroshot.html) | `BERTopic(zeroshot_topic_list=zeroshot_topic_list)` |
| [Merge Multiple Models](https://maartengr.github.io/BERTopic/getting_started/merge/merge.html) | `BERTopic.merge_models([topic_model_1, topic_model_2])` |
### Visualizations
Evaluating topic models can be rather difficult due to the somewhat subjective nature of evaluation.
Visualizing different aspects of the topic model helps in understanding the model and makes it easier
to tweak the model to your liking.
| Method | Code |
|-----------------------|---|
| Visualize Topics | `.visualize_topics()` |
| Visualize Documents | `.visualize_documents()` |
| Visualize Document Hierarchy | `.visualize_hierarchical_documents()` |
| Visualize Topic Hierarchy | `.visualize_hierarchy()` |
| Visualize Topic Tree | `.get_topic_tree(hierarchical_topics)` |
| Visualize Topic Terms | `.visualize_barchart()` |
| Visualize Topic Similarity | `.visualize_heatmap()` |
| Visualize Term Score Decline | `.visualize_term_rank()` |
| Visualize Topic Probability Distribution | `.visualize_distribution(probs[0])` |
| Visualize Topics over Time | `.visualize_topics_over_time(topics_over_time)` |
| Visualize Topics per Class | `.visualize_topics_per_class(topics_per_class)` |
## Citation
To cite the [BERTopic paper](https://arxiv.org/abs/2203.05794), please use the following bibtex reference:
```bibtext
@article{grootendorst2022bertopic,
title={BERTopic: Neural topic modeling with a class-based TF-IDF procedure},
author={Grootendorst, Maarten},
journal={arXiv preprint arXiv:2203.05794},
year={2022}
}
```
================================================
FILE: bertopic/__init__.py
================================================
from importlib.metadata import version
from bertopic._bertopic import BERTopic
__version__ = version("bertopic")
__all__ = [
"BERTopic",
]
================================================
FILE: bertopic/_bertopic.py
================================================
# ruff: noqa: E402
import yaml
import warnings
warnings.filterwarnings("ignore", category=FutureWarning)
warnings.filterwarnings("ignore", category=UserWarning)
try:
yaml._warnings_enabled["YAMLLoadWarning"] = False
except (KeyError, AttributeError, TypeError):
pass
import re
import math
import joblib
import inspect
import collections
import numpy as np
import pandas as pd
import scipy.sparse as sp
from copy import deepcopy
from tqdm import tqdm
from pathlib import Path
from packaging import version
from tempfile import TemporaryDirectory
from collections import defaultdict, Counter
from scipy.sparse import csr_matrix
from scipy.cluster import hierarchy as sch
from importlib.util import find_spec
from typing import List, Tuple, Union, Mapping, Any, Callable, Iterable, TYPE_CHECKING, Literal
# Plotting
if find_spec("plotly") is None:
from bertopic._utils import MockPlotlyModule
plotting = MockPlotlyModule()
else:
from bertopic import plotting
if TYPE_CHECKING:
import plotly.graph_objs as go
import matplotlib.figure as fig
# Models
try:
from hdbscan import HDBSCAN
HAS_HDBSCAN = True
except (ImportError, ModuleNotFoundError):
HAS_HDBSCAN = False
from sklearn.cluster import HDBSCAN as SK_HDBSCAN
from sklearn.preprocessing import normalize
from sklearn import __version__ as sklearn_version
from sklearn.cluster import AgglomerativeClustering
from sklearn.decomposition import PCA
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
# BERTopic
from bertopic.cluster import BaseCluster
from bertopic.backend import BaseEmbedder
from bertopic.representation._mmr import mmr
from bertopic.backend._utils import select_backend
from bertopic.vectorizers import ClassTfidfTransformer
from bertopic.representation import BaseRepresentation, KeyBERTInspired
from bertopic.dimensionality import BaseDimensionalityReduction
from bertopic.cluster._utils import hdbscan_delegator, is_supported_hdbscan
from bertopic._utils import (
MyLogger,
check_documents_type,
check_embeddings_shape,
check_is_fitted,
validate_distance_matrix,
select_topic_representation,
get_unique_distances,
)
import bertopic._save_utils as save_utils
logger = MyLogger()
logger.configure("WARNING")
class BERTopic:
"""BERTopic is a topic modeling technique that leverages BERT embeddings and
c-TF-IDF to create dense clusters allowing for easily interpretable topics
whilst keeping important words in the topic descriptions.
The default embedding model is `all-MiniLM-L6-v2` when selecting `language="english"`
and `paraphrase-multilingual-MiniLM-L12-v2` when selecting `language="multilingual"`.
Attributes:
topics_ (List[int]) : The topics that are generated for each document after training or updating
the topic model. The most recent topics are tracked.
probabilities_ (List[float]): The probability of the assigned topic per document. These are
only calculated if a HDBSCAN model is used for the clustering step.
When `calculate_probabilities=True`, then it is the probabilities
of all topics per document.
topic_sizes_ (Mapping[int, int]) : The size of each topic.
topic_mapper_ (TopicMapper) : A class for tracking topics and their mappings anytime they are
merged, reduced, added, or removed.
topic_representations_ (Mapping[int, Tuple[int, float]]) : The top n terms per topic and their respective
c-TF-IDF values.
c_tf_idf_ (csr_matrix) : The topic-term matrix as calculated through c-TF-IDF. To access its respective
words, run `.vectorizer_model.get_feature_names()` or
`.vectorizer_model.get_feature_names_out()`
topic_labels_ (Mapping[int, str]) : The default labels for each topic.
custom_labels_ (List[str]) : Custom labels for each topic.
topic_embeddings_ (np.ndarray) : The embeddings for each topic. They are calculated by taking the
centroid embedding of each cluster.
representative_docs_ (Mapping[int, str]) : The representative documents for each topic.
Examples:
```python
from bertopic import BERTopic
from sklearn.datasets import fetch_20newsgroups
docs = fetch_20newsgroups(subset='all')['data']
topic_model = BERTopic()
topics, probabilities = topic_model.fit_transform(docs)
```
If you want to use your own embedding model, use it as follows:
```python
from bertopic import BERTopic
from sklearn.datasets import fetch_20newsgroups
from sentence_transformers import SentenceTransformer
docs = fetch_20newsgroups(subset='all')['data']
sentence_model = SentenceTransformer("all-MiniLM-L6-v2")
topic_model = BERTopic(embedding_model=sentence_model)
```
Due to the stochastic nature of UMAP, the results from BERTopic might differ
and the quality can degrade. Using your own embeddings allows you to
try out BERTopic several times until you find the topics that suit
you best.
"""
def __init__(
self,
language: str = "english",
top_n_words: int = 10,
n_gram_range: Tuple[int, int] = (1, 1),
min_topic_size: int = 10,
nr_topics: Union[int, str] | None = None,
low_memory: bool = False,
calculate_probabilities: bool = False,
seed_topic_list: List[List[str]] | None = None,
zeroshot_topic_list: List[str] | None = None,
zeroshot_min_similarity: float = 0.7,
embedding_model=None,
umap_model=None,
hdbscan_model=None,
vectorizer_model: CountVectorizer = None,
ctfidf_model: TfidfTransformer = None,
representation_model: BaseRepresentation = None,
verbose: bool = False,
):
"""BERTopic initialization.
Arguments:
language: The main language used in your documents. The default sentence-transformers
model for "english" is `all-MiniLM-L6-v2`. For a full overview of
supported languages see bertopic.backend.languages. Select
"multilingual" to load in the `paraphrase-multilingual-MiniLM-L12-v2`
sentence-transformers model that supports 50+ languages.
NOTE: This is not used if `embedding_model` is used.
top_n_words: The number of words per topic to extract. Setting this
too high can negatively impact topic embeddings as topics
are typically best represented by at most 10 words.
n_gram_range: The n-gram range for the CountVectorizer.
Advised to keep high values between 1 and 3.
More would likely lead to memory issues.
NOTE: This param will not be used if you pass in your own
CountVectorizer.
min_topic_size: The minimum size of the topic. Increasing this value will lead
to a lower number of clusters/topics and vice versa.
It is the same parameter as `min_cluster_size` in HDBSCAN.
NOTE: This param will not be used if you are using `hdbscan_model`.
nr_topics: Specifying the number of topics will reduce the initial
number of topics to the value specified. This reduction can take
a while as each reduction in topics (-1) activates a c-TF-IDF
calculation. If this is set to None, no reduction is applied. Use
"auto" to automatically reduce topics using HDBSCAN.
NOTE: Controlling the number of topics is best done by adjusting
`min_topic_size` first before adjusting this parameter.
low_memory: Sets UMAP low memory to True to make sure less memory is used.
NOTE: This is only used in UMAP. For example, if you use PCA instead of UMAP
this parameter will not be used.
calculate_probabilities: Calculate the probabilities of all topics
per document instead of the probability of the assigned
topic per document. This could slow down the extraction
of topics if you have many documents (> 100_000).
NOTE: If false you cannot use the corresponding
visualization method `visualize_probabilities`.
NOTE: This is an approximation of topic probabilities
as used in HDBSCAN and not an exact representation.
seed_topic_list: A list of seed words per topic to converge around
zeroshot_topic_list: A list of topic names to use for zero-shot classification
zeroshot_min_similarity: The minimum similarity between a zero-shot topic and
a document for assignment. The higher this value, the more
confident the model needs to be to assign a zero-shot topic to a document.
verbose: Changes the verbosity of the model, Set to True if you want
to track the stages of the model.
embedding_model: Use a custom embedding model.
The following backends are currently supported
* SentenceTransformers
* Flair
* Spacy
* Gensim
* USE (TF-Hub)
You can also pass in a string that points to one of the following
sentence-transformers models:
* https://www.sbert.net/docs/pretrained_models.html
umap_model: Pass in a UMAP model to be used instead of the default.
NOTE: You can also pass in any dimensionality reduction algorithm as long
as it has `.fit` and `.transform` functions.
hdbscan_model: Pass in a hdbscan.HDBSCAN model to be used instead of the default
NOTE: You can also pass in any clustering algorithm as long as it has
`.fit` and `.predict` functions along with the `.labels_` variable.
vectorizer_model: Pass in a custom `CountVectorizer` instead of the default model.
ctfidf_model: Pass in a custom ClassTfidfTransformer instead of the default model.
representation_model: Pass in a model that fine-tunes the topic representations
calculated through c-TF-IDF. Models from `bertopic.representation`
are supported.
"""
# Topic-based parameters
if top_n_words > 100:
logger.warning(
"Note that extracting more than 100 words from a sparse can slow down computation quite a bit."
)
self.top_n_words = top_n_words
self.min_topic_size = min_topic_size
self.nr_topics = nr_topics
self.low_memory = low_memory
self.calculate_probabilities = calculate_probabilities
self.verbose = verbose
self.seed_topic_list = seed_topic_list
self.zeroshot_topic_list = zeroshot_topic_list
self.zeroshot_min_similarity = zeroshot_min_similarity
# Embedding model
self.language = language if not embedding_model else None
self.embedding_model = embedding_model
# Vectorizer
self.n_gram_range = n_gram_range
self.vectorizer_model = vectorizer_model or CountVectorizer(ngram_range=self.n_gram_range)
self.ctfidf_model = ctfidf_model or ClassTfidfTransformer()
# Representation model
self.representation_model = representation_model
# UMAP or another algorithm that has .fit and .transform functions
if umap_model is not None:
self.umap_model = umap_model
else:
try:
from umap import UMAP
self.umap_model = UMAP(
n_neighbors=15,
n_components=5,
min_dist=0.0,
metric="cosine",
low_memory=self.low_memory,
)
except (ImportError, ModuleNotFoundError):
self.umap_model = PCA(n_components=5)
# HDBSCAN or another clustering algorithm that has .fit and .predict functions and
# the .labels_ variable to extract the labels
if hdbscan_model is not None:
self.hdbscan_model = hdbscan_model
elif HAS_HDBSCAN:
self.hdbscan_model = HDBSCAN(
min_cluster_size=self.min_topic_size,
metric="euclidean",
cluster_selection_method="eom",
prediction_data=True,
)
else:
self.hdbscan_model = SK_HDBSCAN(
min_cluster_size=self.min_topic_size, metric="euclidean", cluster_selection_method="eom", n_jobs=-1
)
# Public attributes
self.topics_ = None
self.probabilities_ = None
self.topic_sizes_ = None
self.topic_mapper_ = None
self.topic_representations_ = None
self.topic_embeddings_ = None
self._topic_id_to_zeroshot_topic_idx = {}
self.custom_labels_ = None
self.c_tf_idf_ = None
self.representative_images_ = None
self.representative_docs_ = {}
self.topic_aspects_ = {}
# Private attributes for internal tracking purposes
self._merged_topics = None
if verbose:
logger.set_level("DEBUG")
else:
logger.set_level("WARNING")
@property
def _outliers(self):
"""Some algorithms have outlier labels (-1) that can be tricky to work
with if you are slicing data based on that labels. Therefore, we
track if there are outlier labels and act accordingly when slicing.
Returns:
An integer indicating whether outliers are present in the topic model
"""
return 1 if -1 in self.topic_sizes_ else 0
@property
def topic_labels_(self):
"""Map topic IDs to their labels.
A label is the topic ID, along with the first four words of the topic representation, joined using '_'.
Zeroshot topic labels come from self.zeroshot_topic_list rather than the calculated representation.
Returns:
topic_labels: a dict mapping a topic ID (int) to its label (str)
"""
topic_labels = {
key: f"{key}_" + "_".join([word[0] for word in values[:4]])
for key, values in self.topic_representations_.items()
}
if self._is_zeroshot():
# Need to correct labels from zero-shot topics
topic_id_to_zeroshot_label = {
topic_id: self.zeroshot_topic_list[zeroshot_topic_idx]
for topic_id, zeroshot_topic_idx in self._topic_id_to_zeroshot_topic_idx.items()
}
topic_labels.update(topic_id_to_zeroshot_label)
return topic_labels
def fit(
self,
documents: List[str],
embeddings: np.ndarray = None,
images: List[str] | None = None,
y: Union[List[int], np.ndarray] = None,
):
"""Fit the models on a collection of documents and generate topics.
Arguments:
documents: A list of documents to fit on
embeddings: Pre-trained document embeddings. These can be used
instead of the sentence-transformer model
images: A list of paths to the images to fit on or the images themselves
y: The target class for (semi)-supervised modeling. Use -1 if no class for a
specific instance is specified.
Examples:
```python
from bertopic import BERTopic
from sklearn.datasets import fetch_20newsgroups
docs = fetch_20newsgroups(subset='all')['data']
topic_model = BERTopic().fit(docs)
```
If you want to use your own embeddings, use it as follows:
```python
from bertopic import BERTopic
from sklearn.datasets import fetch_20newsgroups
from sentence_transformers import SentenceTransformer
# Create embeddings
docs = fetch_20newsgroups(subset='all')['data']
sentence_model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = sentence_model.encode(docs, show_progress_bar=True)
# Create topic model
topic_model = BERTopic().fit(docs, embeddings)
```
"""
self.fit_transform(documents=documents, embeddings=embeddings, y=y, images=images)
return self
def fit_transform(
self,
documents: List[str],
embeddings: np.ndarray = None,
images: List[str] | None = None,
y: Union[List[int], np.ndarray] = None,
) -> Tuple[List[int], Union[np.ndarray, None]]:
"""Fit the models on a collection of documents, generate topics,
and return the probabilities and topic per document.
Arguments:
documents: A list of documents to fit on
embeddings: Pre-trained document embeddings. These can be used
instead of the sentence-transformer model
images: A list of paths to the images to fit on or the images themselves
y: The target class for (semi)-supervised modeling. Use -1 if no class for a
specific instance is specified.
Returns:
predictions: Topic predictions for each documents
probabilities: The probability of the assigned topic per document.
If `calculate_probabilities` in BERTopic is set to True, then
it calculates the probabilities of all topics across all documents
instead of only the assigned topic. This, however, slows down
computation and may increase memory usage.
Examples:
```python
from bertopic import BERTopic
from sklearn.datasets import fetch_20newsgroups
docs = fetch_20newsgroups(subset='all')['data']
topic_model = BERTopic()
topics, probs = topic_model.fit_transform(docs)
```
If you want to use your own embeddings, use it as follows:
```python
from bertopic import BERTopic
from sklearn.datasets import fetch_20newsgroups
from sentence_transformers import SentenceTransformer
# Create embeddings
docs = fetch_20newsgroups(subset='all')['data']
sentence_model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = sentence_model.encode(docs, show_progress_bar=True)
# Create topic model
topic_model = BERTopic()
topics, probs = topic_model.fit_transform(docs, embeddings)
```
"""
if documents is not None:
check_documents_type(documents)
check_embeddings_shape(embeddings, documents)
doc_ids = range(len(documents)) if documents is not None else range(len(images))
documents = pd.DataFrame({"Document": documents, "ID": doc_ids, "Topic": None, "Image": images})
# Extract embeddings
if embeddings is None:
logger.info("Embedding - Transforming documents to embeddings.")
self.embedding_model = select_backend(self.embedding_model, language=self.language, verbose=self.verbose)
embeddings = self._extract_embeddings(
documents.Document.to_numpy().tolist(),
images=images,
method="document",
verbose=self.verbose,
)
logger.info("Embedding - Completed \u2713")
else:
if self.embedding_model is not None:
self.embedding_model = select_backend(
self.embedding_model, language=self.language, verbose=self.verbose
)
# Guided Topic Modeling
if self.seed_topic_list is not None and self.embedding_model is not None:
y, embeddings = self._guided_topic_modeling(embeddings)
# Reduce dimensionality and fit UMAP model
umap_embeddings = self._reduce_dimensionality(embeddings, y)
# Zero-shot Topic Modeling
if self._is_zeroshot():
documents, embeddings, assigned_documents, assigned_embeddings = self._zeroshot_topic_modeling(
documents, embeddings
)
# Filter UMAP embeddings to only non-assigned embeddings to be used for clustering
if len(documents) > 0:
umap_embeddings = self.umap_model.transform(embeddings)
if len(documents) > 0:
# Cluster reduced embeddings
documents, probabilities = self._cluster_embeddings(umap_embeddings, documents, y=y)
if self._is_zeroshot() and len(assigned_documents) > 0:
documents, embeddings = self._combine_zeroshot_topics(
documents, embeddings, assigned_documents, assigned_embeddings
)
else:
# All documents matches zero-shot topics
documents = assigned_documents
embeddings = assigned_embeddings
# Sort and Map Topic IDs by their frequency
if not self.nr_topics:
documents = self._sort_mappings_by_frequency(documents)
# Create documents from images if we have images only
if documents.Document.to_numpy()[0] is None:
custom_documents = self._images_to_text(documents, embeddings)
# Extract topics by calculating c-TF-IDF, reduce topics if needed, and get representations.
self._extract_topics(custom_documents, embeddings=embeddings, fine_tune_representation=not self.nr_topics)
if self.nr_topics:
custom_documents = self._reduce_topics(custom_documents)
self._create_topic_vectors(documents=documents, embeddings=embeddings)
# Save the top 3 most representative documents per topic
self._save_representative_docs(custom_documents)
else:
# Extract topics by calculating c-TF-IDF, reduce topics if needed, and get representations.
self._extract_topics(
documents, embeddings=embeddings, verbose=self.verbose, fine_tune_representation=not self.nr_topics
)
if self.nr_topics:
documents = self._reduce_topics(documents)
# Save the top 3 most representative documents per topic
self._save_representative_docs(documents)
# In the case of zero-shot topics, probability will come from cosine similarity,
# and the HDBSCAN model will be removed
if self._is_zeroshot() and len(assigned_documents) > 0:
self.hdbscan_model = BaseCluster()
sim_matrix = cosine_similarity(embeddings, np.array(self.topic_embeddings_))
if self.calculate_probabilities:
self.probabilities_ = sim_matrix
else:
self.probabilities_ = np.max(sim_matrix, axis=1)
else:
self.probabilities_ = self._map_probabilities(probabilities, original_topics=True)
predictions = documents.Topic.to_list()
return predictions, self.probabilities_
def transform(
self,
documents: Union[str, List[str]],
embeddings: np.ndarray = None,
images: List[str] | None = None,
) -> Tuple[List[int], np.ndarray]:
"""After having fit a model, use transform to predict new instances.
Arguments:
documents: A single document or a list of documents to predict on
embeddings: Pre-trained document embeddings. These can be used
instead of the sentence-transformer model.
images: A list of paths to the images to predict on or the images themselves
Returns:
predictions: Topic predictions for each documents
probabilities: The topic probability distribution which is returned by default.
If `calculate_probabilities` in BERTopic is set to False, then the
probabilities are not calculated to speed up computation and
decrease memory usage.
Examples:
```python
from bertopic import BERTopic
from sklearn.datasets import fetch_20newsgroups
docs = fetch_20newsgroups(subset='all')['data']
topic_model = BERTopic().fit(docs)
topics, probs = topic_model.transform(docs)
```
If you want to use your own embeddings:
```python
from bertopic import BERTopic
from sklearn.datasets import fetch_20newsgroups
from sentence_transformers import SentenceTransformer
# Create embeddings
docs = fetch_20newsgroups(subset='all')['data']
sentence_model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = sentence_model.encode(docs, show_progress_bar=True)
# Create topic model
topic_model = BERTopic().fit(docs, embeddings)
topics, probs = topic_model.transform(docs, embeddings)
```
"""
check_is_fitted(self)
check_embeddings_shape(embeddings, documents)
if isinstance(documents, str) or documents is None:
documents = [documents]
if embeddings is None:
embeddings = self._extract_embeddings(documents, images=images, method="document", verbose=self.verbose)
# Check if an embedding model was found
if embeddings is None:
raise ValueError(
"No embedding model was found to embed the documents."
"Make sure when loading in the model using BERTopic.load()"
"to also specify the embedding model."
)
# Transform without hdbscan_model and umap_model using only cosine similarity
elif type(self.hdbscan_model) is BaseCluster:
logger.info("Predicting topic assignments through cosine similarity of topic and document embeddings.")
sim_matrix = cosine_similarity(embeddings, np.array(self.topic_embeddings_))
predictions = np.argmax(sim_matrix, axis=1) - self._outliers
if self.calculate_probabilities:
probabilities = sim_matrix
else:
probabilities = np.max(sim_matrix, axis=1)
# Transform with full pipeline
else:
logger.info("Dimensionality - Reducing dimensionality of input embeddings.")
umap_embeddings = self.umap_model.transform(embeddings)
logger.info("Dimensionality - Completed \u2713")
# Extract predictions and probabilities if it is a HDBSCAN-like model
logger.info("Clustering - Approximating new points with `hdbscan_model`")
if is_supported_hdbscan(self.hdbscan_model):
predictions, probabilities = hdbscan_delegator(
self.hdbscan_model, "approximate_predict", umap_embeddings
)
# Calculate probabilities
if self.calculate_probabilities:
logger.info("Probabilities - Start calculation of probabilities with HDBSCAN")
probabilities = hdbscan_delegator(self.hdbscan_model, "membership_vector", umap_embeddings)
logger.info("Probabilities - Completed \u2713")
else:
predictions = self.hdbscan_model.predict(umap_embeddings)
probabilities = None
logger.info("Cluster - Completed \u2713")
# Map probabilities and predictions
probabilities = self._map_probabilities(probabilities, original_topics=True)
predictions = self._map_predictions(predictions)
return predictions, probabilities
def partial_fit(
self,
documents: List[str],
embeddings: np.ndarray = None,
y: Union[List[int], np.ndarray] = None,
):
"""Fit BERTopic on a subset of the data and perform online learning
with batch-like data.
Online topic modeling in BERTopic is performed by using dimensionality
reduction and cluster algorithms that support a `partial_fit` method
in order to incrementally train the topic model.
Likewise, the `bertopic.vectorizers.OnlineCountVectorizer` is used
to dynamically update its vocabulary when presented with new data.
It has several parameters for modeling decay and updating the
representations.
In other words, although the main algorithm stays the same, the training
procedure now works as follows:
For each subset of the data:
1. Generate embeddings with a pre-trained language model
2. Incrementally update the dimensionality reduction algorithm with `partial_fit`
3. Incrementally update the cluster algorithm with `partial_fit`
4. Incrementally update the OnlineCountVectorizer and apply some form of decay
Note that it is advised to use `partial_fit` with batches and
not single documents for the best performance.
Arguments:
documents: A list of documents to fit on
embeddings: Pre-trained document embeddings. These can be used
instead of the sentence-transformer model
y: The target class for (semi)-supervised modeling. Use -1 if no class for a
specific instance is specified.
Examples:
```python
from sklearn.datasets import fetch_20newsgroups
from sklearn.cluster import MiniBatchKMeans
from sklearn.decomposition import IncrementalPCA
from bertopic.vectorizers import OnlineCountVectorizer
from bertopic import BERTopic
# Prepare documents
docs = fetch_20newsgroups(subset=subset, remove=('headers', 'footers', 'quotes'))["data"]
# Prepare sub-models that support online learning
umap_model = IncrementalPCA(n_components=5)
cluster_model = MiniBatchKMeans(n_clusters=50, random_state=0)
vectorizer_model = OnlineCountVectorizer(stop_words="english", decay=.01)
topic_model = BERTopic(umap_model=umap_model,
hdbscan_model=cluster_model,
vectorizer_model=vectorizer_model)
# Incrementally fit the topic model by training on 1000 documents at a time
for index in range(0, len(docs), 1000):
topic_model.partial_fit(docs[index: index+1000])
```
"""
# Checks
check_embeddings_shape(embeddings, documents)
if not hasattr(self.hdbscan_model, "partial_fit"):
raise ValueError("In order to use `.partial_fit`, the cluster model should have a `.partial_fit` function.")
# Prepare documents
if isinstance(documents, str):
documents = [documents]
documents = pd.DataFrame({"Document": documents, "ID": range(len(documents)), "Topic": None})
# Extract embeddings
if embeddings is None:
if self.topic_representations_ is None:
self.embedding_model = select_backend(
self.embedding_model, language=self.language, verbose=self.verbose
)
embeddings = self._extract_embeddings(
documents.Document.to_numpy().tolist(),
method="document",
verbose=self.verbose,
)
else:
if self.embedding_model is not None and self.topic_representations_ is None:
self.embedding_model = select_backend(
self.embedding_model, language=self.language, verbose=self.verbose
)
# Reduce dimensionality
if self.seed_topic_list is not None and self.embedding_model is not None:
y, embeddings = self._guided_topic_modeling(embeddings)
umap_embeddings = self._reduce_dimensionality(embeddings, y, partial_fit=True)
# Cluster reduced embeddings
documents, self.probabilities_ = self._cluster_embeddings(umap_embeddings, documents, partial_fit=True)
topics = documents.Topic.to_list()
# Map and find new topics
if not self.topic_mapper_:
self.topic_mapper_ = TopicMapper(topics)
mappings = self.topic_mapper_.get_mappings()
new_topics = set(topics).difference(set(mappings.keys()))
new_topic_ids = {topic: max(mappings.values()) + index + 1 for index, topic in enumerate(new_topics)}
self.topic_mapper_.add_new_topics(new_topic_ids)
updated_mappings = self.topic_mapper_.get_mappings()
updated_topics = [updated_mappings[topic] for topic in topics]
documents["Topic"] = updated_topics
# Add missing topics (topics that were originally created but are now missing)
if self.topic_representations_:
missing_topics = set(self.topic_representations_.keys()).difference(set(updated_topics))
for missing_topic in missing_topics:
documents.loc[len(documents), :] = [" ", len(documents), missing_topic]
else:
missing_topics = {}
# Prepare documents
documents_per_topic = documents.sort_values("Topic").groupby(["Topic"], as_index=False)
updated_topics = documents_per_topic.first().Topic.astype(int)
documents_per_topic = documents_per_topic.agg({"Document": " ".join})
# Update topic representations
self.c_tf_idf_, updated_words = self._c_tf_idf(documents_per_topic, partial_fit=True)
self.topic_representations_ = self._extract_words_per_topic(
updated_words, documents, self.c_tf_idf_, calculate_aspects=False
)
self._create_topic_vectors()
# Update topic sizes
if len(missing_topics) > 0:
documents = documents.iloc[: -len(missing_topics)]
if self.topic_sizes_ is None:
self._update_topic_size(documents)
else:
sizes = documents.groupby(["Topic"], as_index=False).count()
for _, row in sizes.iterrows():
topic = int(row.Topic)
if self.topic_sizes_.get(topic) is not None and topic not in missing_topics:
self.topic_sizes_[topic] += int(row.Document)
elif self.topic_sizes_.get(topic) is None:
self.topic_sizes_[topic] = int(row.Document)
self.topics_ = documents.Topic.astype(int).tolist()
return self
def topics_over_time(
self,
docs: List[str],
timestamps: Union[List[str], List[int]],
topics: List[int] | None = None,
nr_bins: int | None = None,
datetime_format: str | None = None,
evolution_tuning: bool = True,
global_tuning: bool = True,
) -> pd.DataFrame:
"""Create topics over time.
To create the topics over time, BERTopic needs to be already fitted once.
From the fitted models, the c-TF-IDF representations are calculate at
each timestamp t. Then, the c-TF-IDF representations at timestamp t are
averaged with the global c-TF-IDF representations in order to fine-tune the
local representations.
Note:
Make sure to use a limited number of unique timestamps (<100) as the
c-TF-IDF representation will be calculated at each single unique timestamp.
Having a large number of unique timestamps can take some time to be calculated.
Moreover, there aren't many use-cases where you would like to see the difference
in topic representations over more than 100 different timestamps.
Arguments:
docs: The documents you used when calling either `fit` or `fit_transform`
timestamps: The timestamp of each document. This can be either a list of strings or ints.
If it is a list of strings, then the datetime format will be automatically
inferred. If it is a list of ints, then the documents will be ordered in
ascending order.
topics: A list of topics where each topic is related to a document in `docs` and
a timestamp in `timestamps`. You can use this to apply topics_over_time on
a subset of the data. Make sure that `docs`, `timestamps`, and `topics`
all correspond to one another and have the same size.
nr_bins: The number of bins you want to create for the timestamps. The left interval will
be chosen as the timestamp. An additional column will be created with the
entire interval.
datetime_format: The datetime format of the timestamps if they are strings, eg "%d/%m/%Y".
Set this to None if you want to have it automatically detect the format.
See strftime documentation for more information on choices:
https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior.
evolution_tuning: Fine-tune each topic representation at timestamp *t* by averaging its
c-TF-IDF matrix with the c-TF-IDF matrix at timestamp *t-1*. This creates
evolutionary topic representations.
global_tuning: Fine-tune each topic representation at timestamp *t* by averaging its c-TF-IDF matrix
with the global c-TF-IDF matrix. Turn this off if you want to prevent words in
topic representations that could not be found in the documents at timestamp *t*.
Returns:
topics_over_time: A dataframe that contains the topic, words, and frequency of topic
at timestamp *t*.
Examples:
The timestamps variable represents the timestamp of each document. If you have over
100 unique timestamps, it is advised to bin the timestamps as shown below:
```python
from bertopic import BERTopic
topic_model = BERTopic()
topics, probs = topic_model.fit_transform(docs)
topics_over_time = topic_model.topics_over_time(docs, timestamps, nr_bins=20)
```
"""
check_is_fitted(self)
check_documents_type(docs)
selected_topics = topics if topics else self.topics_
documents = pd.DataFrame({"Document": docs, "Topic": selected_topics, "Timestamps": timestamps})
global_c_tf_idf = normalize(self.c_tf_idf_, axis=1, norm="l1", copy=False)
all_topics = sorted(list(documents.Topic.unique()))
all_topics_indices = {topic: index for index, topic in enumerate(all_topics)}
if isinstance(timestamps[0], str):
infer_datetime_format = True if not datetime_format else False
documents["Timestamps"] = pd.to_datetime(
documents["Timestamps"],
infer_datetime_format=infer_datetime_format,
format=datetime_format,
)
if nr_bins:
documents["Bins"] = pd.cut(documents.Timestamps, bins=nr_bins)
documents["Timestamps"] = documents.apply(lambda row: row.Bins.left, 1)
# Sort documents in chronological order
documents = documents.sort_values("Timestamps")
timestamps = documents.Timestamps.unique()
if len(timestamps) > 100:
logger.warning(
f"There are more than 100 unique timestamps (i.e., {len(timestamps)}) "
"which significantly slows down the application. Consider setting `nr_bins` "
"to a value lower than 100 to speed up calculation. "
)
# For each unique timestamp, create topic representations
topics_over_time = []
for index, timestamp in tqdm(enumerate(timestamps), disable=not self.verbose):
# Calculate c-TF-IDF representation for a specific timestamp
selection = documents.loc[documents.Timestamps == timestamp, :]
documents_per_topic = selection.groupby(["Topic"], as_index=False).agg(
{"Document": " ".join, "Timestamps": "count"}
)
c_tf_idf, words = self._c_tf_idf(documents_per_topic, fit=False)
if global_tuning or evolution_tuning:
c_tf_idf = normalize(c_tf_idf, axis=1, norm="l1", copy=False)
# Fine-tune the c-TF-IDF matrix at timestamp t by averaging it with the c-TF-IDF
# matrix at timestamp t-1
if evolution_tuning and index != 0:
current_topics = sorted(list(documents_per_topic.Topic.values))
overlapping_topics = sorted(
list(set(previous_topics).intersection(set(current_topics))) # noqa: F821
)
current_overlap_idx = [current_topics.index(topic) for topic in overlapping_topics]
previous_overlap_idx = [
previous_topics.index(topic) # noqa: F821
for topic in overlapping_topics
]
c_tf_idf.tolil()[current_overlap_idx] = (
(
c_tf_idf[current_overlap_idx] + previous_c_tf_idf[previous_overlap_idx] # noqa: F821
)
/ 2.0
).tolil()
# Fine-tune the timestamp c-TF-IDF representation based on the global c-TF-IDF representation
# by simply taking the average of the two
if global_tuning:
selected_topics = [all_topics_indices[topic] for topic in documents_per_topic.Topic.to_numpy()]
c_tf_idf = (global_c_tf_idf[selected_topics] + c_tf_idf) / 2.0
# Extract the words per topic
words_per_topic = self._extract_words_per_topic(words, selection, c_tf_idf, calculate_aspects=False)
topic_frequency = pd.Series(
documents_per_topic.Timestamps.values, index=documents_per_topic.Topic
).to_dict()
# Fill dataframe with results
topics_at_timestamp = [
(
topic,
", ".join([words[0] for words in values][:5]),
topic_frequency[topic],
timestamp,
)
for topic, values in words_per_topic.items()
]
topics_over_time.extend(topics_at_timestamp)
if evolution_tuning:
previous_topics = sorted(list(documents_per_topic.Topic.values)) # noqa: F841
previous_c_tf_idf = c_tf_idf.copy() # noqa: F841
return pd.DataFrame(topics_over_time, columns=["Topic", "Words", "Frequency", "Timestamp"])
def topics_per_class(
self,
docs: List[str],
classes: Union[List[int], List[str]],
global_tuning: bool = True,
) -> pd.DataFrame:
"""Create topics per class.
To create the topics per class, BERTopic needs to be already fitted once.
From the fitted models, the c-TF-IDF representations are calculated at
each class c. Then, the c-TF-IDF representations at class c are
averaged with the global c-TF-IDF representations in order to fine-tune the
local representations. This can be turned off if the pure representation is
needed.
Note:
Make sure to use a limited number of unique classes (<100) as the
c-TF-IDF representation will be calculated at each single unique class.
Having a large number of unique classes can take some time to be calculated.
Arguments:
docs: The documents you used when calling either `fit` or `fit_transform`
classes: The class of each document. This can be either a list of strings or ints.
global_tuning: Fine-tune each topic representation for class c by averaging its c-TF-IDF matrix
with the global c-TF-IDF matrix. Turn this off if you want to prevent words in
topic representations that could not be found in the documents for class c.
Returns:
topics_per_class: A dataframe that contains the topic, words, and frequency of topics
for each class.
Examples:
```python
from bertopic import BERTopic
topic_model = BERTopic()
topics, probs = topic_model.fit_transform(docs)
topics_per_class = topic_model.topics_per_class(docs, classes)
```
"""
check_documents_type(docs)
documents = pd.DataFrame({"Document": docs, "Topic": self.topics_, "Class": classes})
global_c_tf_idf = normalize(self.c_tf_idf_, axis=1, norm="l1", copy=False)
# For each unique timestamp, create topic representations
topics_per_class = []
for _, class_ in tqdm(enumerate(set(classes)), disable=not self.verbose):
# Calculate c-TF-IDF representation for a specific timestamp
selection = documents.loc[documents.Class == class_, :]
documents_per_topic = selection.groupby(["Topic"], as_index=False).agg(
{"Document": " ".join, "Class": "count"}
)
c_tf_idf, words = self._c_tf_idf(documents_per_topic, fit=False)
# Fine-tune the timestamp c-TF-IDF representation based on the global c-TF-IDF representation
# by simply taking the average of the two
if global_tuning:
c_tf_idf = normalize(c_tf_idf, axis=1, norm="l1", copy=False)
c_tf_idf = (global_c_tf_idf[documents_per_topic.Topic.to_numpy() + self._outliers] + c_tf_idf) / 2.0
# Extract the words per topic
words_per_topic = self._extract_words_per_topic(words, selection, c_tf_idf, calculate_aspects=False)
topic_frequency = pd.Series(documents_per_topic.Class.to_numpy(), index=documents_per_topic.Topic).to_dict()
# Fill dataframe with results
topics_at_class = [
(
topic,
", ".join([words[0] for words in values][:5]),
topic_frequency[topic],
class_,
)
for topic, values in words_per_topic.items()
]
topics_per_class.extend(topics_at_class)
topics_per_class = pd.DataFrame(topics_per_class, columns=["Topic", "Words", "Frequency", "Class"])
return topics_per_class
def hierarchical_topics(
self,
docs: List[str],
use_ctfidf: bool = True,
linkage_function: Callable[[csr_matrix], np.ndarray] | None = None,
distance_function: Callable[[csr_matrix], csr_matrix] | None = None,
) -> pd.DataFrame:
"""Create a hierarchy of topics.
To create this hierarchy, BERTopic needs to be already fitted once.
Then, a hierarchy is calculated on the distance matrix of the c-TF-IDF or topic embeddings
representation using `scipy.cluster.hierarchy.linkage`.
Based on that hierarchy, we calculate the topic representation at each
merged step. This is a local representation, as we only assume that the
chosen step is merged and not all others which typically improves the
topic representation.
Arguments:
docs: The documents you used when calling either `fit` or `fit_transform`
use_ctfidf: Whether to calculate distances between topics based on c-TF-IDF embeddings. If False, the
embeddings from the embedding model are used.
linkage_function: The linkage function to use. Default is:
`lambda x: sch.linkage(x, 'ward', optimal_ordering=True)`
distance_function: The distance function to use on the c-TF-IDF matrix. Default is:
`lambda x: 1 - cosine_similarity(x)`.
You can pass any function that returns either a square matrix of
shape (n_samples, n_samples) with zeros on the diagonal and
non-negative values or condensed distance matrix of shape
(n_samples * (n_samples - 1) / 2,) containing the upper
triangular of the distance matrix.
Returns:
hierarchical_topics: A dataframe that contains a hierarchy of topics
represented by their parents and their children
Examples:
```python
from bertopic import BERTopic
topic_model = BERTopic()
topics, probs = topic_model.fit_transform(docs)
hierarchical_topics = topic_model.hierarchical_topics(docs)
```
A custom linkage function can be used as follows:
```python
from scipy.cluster import hierarchy as sch
from bertopic import BERTopic
topic_model = BERTopic()
topics, probs = topic_model.fit_transform(docs)
# Hierarchical topics
linkage_function = lambda x: sch.linkage(x, 'ward', optimal_ordering=True)
hierarchical_topics = topic_model.hierarchical_topics(docs, linkage_function=linkage_function)
```
"""
check_documents_type(docs)
if distance_function is None:
distance_function = lambda x: 1 - cosine_similarity(x)
if linkage_function is None:
linkage_function = lambda x: sch.linkage(x, "ward", optimal_ordering=True)
# Calculate distance
embeddings = select_topic_representation(self.c_tf_idf_, self.topic_embeddings_, use_ctfidf)[0][
self._outliers :
]
X = distance_function(embeddings)
X = validate_distance_matrix(X, embeddings.shape[0])
# Use the 1-D condensed distance matrix as an input instead of the raw distance matrix
Z = linkage_function(X)
# Ensuring that the distances between clusters are unique otherwise the flatting of the hierarchy with
# `sch.fcluster(...)` would produce incorrect values for "Topics" for these clusters
if len(Z[:, 2]) != len(np.unique(Z[:, 2])):
Z[:, 2] = get_unique_distances(Z[:, 2])
# Calculate basic bag-of-words to be iteratively merged later
documents = pd.DataFrame({"Document": docs, "ID": range(len(docs)), "Topic": self.topics_})
documents_per_topic = documents.groupby(["Topic"], as_index=False).agg({"Document": " ".join})
documents_per_topic = documents_per_topic.loc[documents_per_topic.Topic != -1, :]
clean_documents = self._preprocess_text(documents_per_topic.Document.values)
# Scikit-Learn Deprecation: get_feature_names is deprecated in 1.0
# and will be removed in 1.2. Please use get_feature_names_out instead.
if version.parse(sklearn_version) >= version.parse("1.0.0"):
words = self.vectorizer_model.get_feature_names_out()
else:
words = self.vectorizer_model.get_feature_names()
bow = self.vectorizer_model.transform(clean_documents)
# Extract clusters
hier_topics = pd.DataFrame(
columns=[
"Parent_ID",
"Parent_Name",
"Topics",
"Child_Left_ID",
"Child_Left_Name",
"Child_Right_ID",
"Child_Right_Name",
]
)
for index in tqdm(range(len(Z))):
# Find clustered documents
clusters = sch.fcluster(Z, t=Z[index][2], criterion="distance") - self._outliers
nr_clusters = len(clusters)
# Extract first topic we find to get the set of topics in a merged topic
topic = None
val = Z[index][0]
while topic is None:
if val - len(clusters) < 0:
topic = int(val)
else:
val = Z[int(val - len(clusters))][0]
clustered_topics = [i for i, x in enumerate(clusters) if x == clusters[topic]]
# Group bow per cluster, calculate c-TF-IDF and extract words
grouped = csr_matrix(bow[clustered_topics].sum(axis=0))
c_tf_idf = self.ctfidf_model.transform(grouped)
selection = documents.loc[documents.Topic.isin(clustered_topics), :]
selection.Topic = 0
words_per_topic = self._extract_words_per_topic(words, selection, c_tf_idf, calculate_aspects=False)
# Extract parent's name and ID
parent_id = index + len(clusters)
parent_name = "_".join([x[0] for x in words_per_topic[0]][:5])
# Extract child's name and ID
Z_id = Z[index][0]
child_left_id = Z_id if Z_id - nr_clusters < 0 else Z_id - nr_clusters
if Z_id - nr_clusters < 0:
child_left_name = "_".join([x[0] for x in self.get_topic(Z_id)][:5])
else:
child_left_name = hier_topics.iloc[int(child_left_id)].Parent_Name
# Extract child's name and ID
Z_id = Z[index][1]
child_right_id = Z_id if Z_id - nr_clusters < 0 else Z_id - nr_clusters
if Z_id - nr_clusters < 0:
child_right_name = "_".join([x[0] for x in self.get_topic(Z_id)][:5])
else:
child_right_name = hier_topics.iloc[int(child_right_id)].Parent_Name
# Save results
hier_topics.loc[len(hier_topics), :] = [
parent_id,
parent_name,
clustered_topics,
int(Z[index][0]),
child_left_name,
int(Z[index][1]),
child_right_name,
]
hier_topics["Distance"] = Z[:, 2]
hier_topics = hier_topics.sort_values("Parent_ID", ascending=False)
hier_topics[["Parent_ID", "Child_Left_ID", "Child_Right_ID"]] = hier_topics[
["Parent_ID", "Child_Left_ID", "Child_Right_ID"]
].astype(str)
return hier_topics
def approximate_distribution(
self,
documents: Union[str, List[str]],
window: int = 4,
stride: int = 1,
min_similarity: float = 0.1,
batch_size: int = 1000,
padding: bool = False,
use_embedding_model: bool = False,
calculate_tokens: bool = False,
separator: str = " ",
) -> Tuple[np.ndarray, Union[List[np.ndarray], None]]:
"""A post-hoc approximation of topic distributions across documents.
In order to perform this approximation, each document is split into tokens
according to the provided tokenizer in the `CountVectorizer`. Then, a
sliding window is applied on each document creating subsets of the document.
For example, with a window size of 3 and stride of 1, the sentence:
`Solving the right problem is difficult.`
can be split up into `solving the right`, `the right problem`, `right problem is`,
and `problem is difficult`. These are called tokensets. For each of these
tokensets, we calculate their c-TF-IDF representation and find out
how similar they are to the previously generated topics. Then, the
similarities to the topics for each tokenset are summed up in order to
create a topic distribution for the entire document.
We can also dive into this a bit deeper by then splitting these tokensets
up into individual tokens and calculate how much a word, in a specific sentence,
contributes to the topics found in that document. This can be enabled by
setting `calculate_tokens=True` which can be used for visualization purposes
in `topic_model.visualize_approximate_distribution`.
The main output, `topic_distributions`, can also be used directly in
`.visualize_distribution(topic_distributions[index])` by simply selecting
a single distribution.
Arguments:
documents: A single document or a list of documents for which we
approximate their topic distributions
window: Size of the moving window which indicates the number of
tokens being considered.
stride: How far the window should move at each step.
min_similarity: The minimum similarity of a document's tokenset
with respect to the topics.
batch_size: The number of documents to process at a time. If None,
then all documents are processed at once.
NOTE: With a large number of documents, it is not
advised to process all documents at once.
padding: Whether to pad the beginning and ending of a document with
empty tokens.
use_embedding_model: Whether to use the topic model's embedding
model to calculate the similarity between
tokensets and topics instead of using c-TF-IDF.
calculate_tokens: Calculate the similarity of tokens with all topics.
NOTE: This is computation-wise more expensive and
can require more memory. Using this over batches of
documents might be preferred.
separator: The separator used to merge tokens into tokensets.
Returns:
topic_distributions: A `n` x `m` matrix containing the topic distributions
for all input documents with `n` being the documents
and `m` the topics.
topic_token_distributions: A list of `t` x `m` arrays with `t` being the
number of tokens for the respective document
and `m` the topics.
Examples:
After fitting the model, the topic distributions can be calculated regardless
of the clustering model and regardless of whether the documents were previously
seen or not:
```python
topic_distr, _ = topic_model.approximate_distribution(docs)
```
As a result, the topic distributions are calculated in `topic_distr` for the
entire document based on a token set with a specific window size and stride.
If you want to calculate the topic distributions on a token-level:
```python
topic_distr, topic_token_distr = topic_model.approximate_distribution(docs, calculate_tokens=True)
```
The `topic_token_distr` then contains, for each token, the best fitting topics.
As with `topic_distr`, it can contain multiple topics for a single token.
"""
if isinstance(documents, str):
documents = [documents]
if batch_size is None:
batch_size = len(documents)
batches = 1
else:
batches = math.ceil(len(documents) / batch_size)
topic_distributions = []
topic_token_distributions = []
for i in tqdm(range(batches), disable=not self.verbose):
doc_set = documents[i * batch_size : (i + 1) * batch_size]
# Extract tokens
analyzer = self.vectorizer_model.build_tokenizer()
tokens = [analyzer(document) for document in doc_set]
# Extract token sets
all_sentences = []
all_indices = [0]
all_token_sets_ids = []
for tokenset in tokens:
if len(tokenset) < window:
token_sets = [tokenset]
token_sets_ids = [list(range(len(tokenset)))]
else:
# Extract tokensets using window and stride parameters
stride_indices = list(range(len(tokenset)))[::stride]
token_sets = []
token_sets_ids = []
for stride_index in stride_indices:
selected_tokens = tokenset[stride_index : stride_index + window]
if padding or len(selected_tokens) == window:
token_sets.append(selected_tokens)
token_sets_ids.append(
list(
range(
stride_index,
stride_index + len(selected_tokens),
)
)
)
# Add empty tokens at the beginning and end of a document
if padding:
padded = []
padded_ids = []
t = math.ceil(window / stride) - 1
for i in range(math.ceil(window / stride) - 1):
padded.append(tokenset[: window - ((t - i) * stride)])
padded_ids.append(list(range(0, window - ((t - i) * stride))))
token_sets = padded + token_sets
token_sets_ids = padded_ids + token_sets_ids
# Join the tokens
sentences = [separator.join(token) for token in token_sets]
all_sentences.extend(sentences)
all_token_sets_ids.extend(token_sets_ids)
all_indices.append(all_indices[-1] + len(sentences))
# Calculate similarity between embeddings of token sets and the topics
if use_embedding_model:
embeddings = self._extract_embeddings(all_sentences, method="document", verbose=True)
similarity = cosine_similarity(embeddings, self.topic_embeddings_[self._outliers :])
# Calculate similarity between c-TF-IDF of token sets and the topics
else:
bow_doc = self.vectorizer_model.transform(all_sentences)
c_tf_idf_doc = self.ctfidf_model.transform(bow_doc)
similarity = cosine_similarity(c_tf_idf_doc, self.c_tf_idf_[self._outliers :])
# Only keep similarities that exceed the minimum
similarity[similarity < min_similarity] = 0
# Aggregate results on an individual token level
if calculate_tokens:
topic_distribution = []
topic_token_distribution = []
for index, token in enumerate(tokens):
start = all_indices[index]
end = all_indices[index + 1]
if start == end:
end = end + 1
# Assign topics to individual tokens
token_id = [i for i in range(len(token))]
token_val = {index: [] for index in token_id}
for sim, token_set in zip(similarity[start:end], all_token_sets_ids[start:end]):
for token in token_set:
if token in token_val:
token_val[token].append(sim)
matrix = []
for _, value in token_val.items():
matrix.append(np.add.reduce(value))
# Take empty documents into account
matrix = np.array(matrix)
if len(matrix.shape) == 1:
matrix = np.zeros((1, len(self.topic_labels_) - self._outliers))
topic_token_distribution.append(np.array(matrix))
topic_distribution.append(np.add.reduce(matrix))
topic_distribution = normalize(topic_distribution, norm="l1", axis=1)
# Aggregate on a tokenset level indicated by the window and stride
else:
topic_distribution = []
for index in range(len(all_indices) - 1):
start = all_indices[index]
end = all_indices[index + 1]
if start == end:
end = end + 1
group = similarity[start:end].sum(axis=0)
topic_distribution.append(group)
topic_distribution = normalize(np.array(topic_distribution), norm="l1", axis=1)
topic_token_distribution = None
# Combine results
topic_distributions.append(topic_distribution)
if topic_token_distribution is None:
topic_token_distributions = None
else:
topic_token_distributions.extend(topic_token_distribution)
topic_distributions = np.vstack(topic_distributions)
return topic_distributions, topic_token_distributions
def find_topics(
self, search_term: str | None = None, image: str | None = None, top_n: int = 5
) -> Tuple[List[int], List[float]]:
"""Find topics most similar to a search_term.
Creates an embedding for a search query and compares that with
the topic embeddings. The most similar topics are returned
along with their similarity values.
The query is specified using search_term for text queries or image for image queries.
The search_term can be of any size but since it is compared
with the topic representation it is advised to keep it
below 5 words.
Arguments:
search_term: the term you want to use to search for topics.
image: path to the image you want to use to search for topics.
top_n: the number of topics to return
Returns:
similar_topics: the most similar topics from high to low
similarity: the similarity scores from high to low
Examples:
You can use the underlying embedding model to find topics that
best represent the search term:
```python
topics, similarity = topic_model.find_topics("sports", top_n=5)
```
Note that the search query is typically more accurate if the
search_term consists of a phrase or multiple words.
"""
if self.embedding_model is None:
raise Exception("This method can only be used if you did not use custom embeddings.")
topic_list = list(self.topic_representations_.keys())
topic_list.sort()
# Extract search_term embeddings and compare with topic embeddings
if search_term is not None:
search_embedding = self._extract_embeddings([search_term], method="word", verbose=False).flatten()
elif image is not None:
search_embedding = self._extract_embeddings(
[None], images=[image], method="document", verbose=False
).flatten()
sims = cosine_similarity(search_embedding.reshape(1, -1), self.topic_embeddings_).flatten()
# Extract topics most similar to search_term
ids = np.argsort(sims)[-top_n:]
similarity = [sims[i] for i in ids][::-1]
similar_topics = [topic_list[index] for index in ids][::-1]
return similar_topics, similarity
def update_topics(
self,
docs: List[str],
images: List[str] | None = None,
topics: List[int] | None = None,
top_n_words: int = 10,
n_gram_range: Tuple[int, int] | None = None,
vectorizer_model: CountVectorizer = None,
ctfidf_model: ClassTfidfTransformer = None,
representation_model: BaseRepresentation = None,
):
"""Updates the topic representation by recalculating c-TF-IDF with the new
parameters as defined in this function.
When you have trained a model and viewed the topics and the words that represent them,
you might not be satisfied with the representation. Perhaps you forgot to remove
stop_words or you want to try out a different n_gram_range. This function allows you
to update the topic representation after they have been formed.
Arguments:
docs: The documents you used when calling either `fit` or `fit_transform`
images: The images you used when calling either `fit` or `fit_transform`
topics: A list of topics where each topic is related to a document in `docs`.
Use this variable to change or map the topics.
NOTE: Using a custom list of topic assignments may lead to errors if
topic reduction techniques are used afterwards. Make sure that
manually assigning topics is the last step in the pipeline
top_n_words: The number of words per topic to extract. Setting this
too high can negatively impact topic embeddings as topics
are typically best represented by at most 10 words.
n_gram_range: The n-gram range for the CountVectorizer.
vectorizer_model: Pass in your own CountVectorizer from scikit-learn
ctfidf_model: Pass in your own c-TF-IDF model to update the representations
representation_model: Pass in a model that fine-tunes the topic representations
calculated through c-TF-IDF. Models from `bertopic.representation`
are supported.
Examples:
In order to update the topic representation, you will need to first fit the topic
model and extract topics from them. Based on these, you can update the representation:
```python
topic_model.update_topics(docs, n_gram_range=(2, 3))
```
You can also use a custom vectorizer to update the representation:
```python
from sklearn.feature_extraction.text import CountVectorizer
vectorizer_model = CountVectorizer(ngram_range=(1, 2), stop_words="english")
topic_model.update_topics(docs, vectorizer_model=vectorizer_model)
```
You can also use this function to change or map the topics to something else.
You can update them as follows:
```python
topic_model.update_topics(docs, my_updated_topics)
```
"""
check_documents_type(docs)
check_is_fitted(self)
if not n_gram_range:
n_gram_range = self.n_gram_range
if top_n_words > 100:
logger.warning(
"Note that extracting more than 100 words from a sparse can slow down computation quite a bit."
)
self.top_n_words = top_n_words
self.vectorizer_model = vectorizer_model or CountVectorizer(ngram_range=n_gram_range)
self.ctfidf_model = ctfidf_model or ClassTfidfTransformer()
self.representation_model = representation_model
if topics is None:
topics = self.topics_
else:
logger.warning(
"Using a custom list of topic assignments may lead to errors if "
"topic reduction techniques are used afterwards. Make sure that "
"manually assigning topics is the last step in the pipeline."
"Note that topic embeddings will also be created through weighted"
"c-TF-IDF embeddings instead of centroid embeddings."
)
documents = pd.DataFrame({"Document": docs, "Topic": topics, "ID": range(len(docs)), "Image": images})
documents_per_topic = documents.groupby(["Topic"], as_index=False).agg({"Document": " ".join})
# Update topic sizes and assignments
self._update_topic_size(documents)
# Extract words and update topic labels
self.c_tf_idf_, words = self._c_tf_idf(documents_per_topic)
self.topic_representations_ = self._extract_words_per_topic(words, documents)
# Update topic vectors
if set(topics) != set(self.topics_):
# Remove outlier topic embedding if all that has changed is the outlier class
same_position = all(
[
True if old_topic == new_topic else False
for old_topic, new_topic in zip(self.topics_, topics)
if old_topic != -1
]
)
if same_position and -1 not in topics and -1 in self.topics_:
self.topic_embeddings_ = self.topic_embeddings_[1:]
else:
self._create_topic_vectors()
def get_topics(self, full: bool = False) -> Mapping[str, Tuple[str, float]]:
"""Return topics with top n words and their c-TF-IDF score.
Arguments:
full: If True, returns all different forms of topic representations
for each topic, including aspects
Returns:
self.topic_representations_: The top n words per topic and the corresponding c-TF-IDF score
Examples:
```python
all_topics = topic_model.get_topics()
```
"""
check_is_fitted(self)
if full:
topic_representations = {"Main": self.topic_representations_}
topic_representations.update(self.topic_aspects_)
return topic_representations
else:
return self.topic_representations_
def get_topic(self, topic: int, full: bool = False) -> Union[Mapping[str, Tuple[str, float]], bool]:
"""Return top n words for a specific topic and their c-TF-IDF scores.
Arguments:
topic: A specific topic for which you want its representation
full: If True, returns all different forms of topic representations
for a topic, including aspects
Returns:
The top n words for a specific word and its respective c-TF-IDF scores
Examples:
```python
topic = topic_model.get_topic(12)
```
"""
check_is_fitted(self)
if topic in self.topic_representations_:
if full:
representations = {"Main": self.topic_representations_[topic]}
aspects = {aspect: representations[topic] for aspect, representations in self.topic_aspects_.items()}
representations.update(aspects)
return representations
else:
return self.topic_representations_[topic]
else:
return False
def get_topic_info(self, topic: int | None = None) -> pd.DataFrame:
"""Get information about each topic including its ID, frequency, and name.
Arguments:
topic: A specific topic for which you want the frequency
Returns:
info: The information relating to either a single topic or all topics
Examples:
```python
info_df = topic_model.get_topic_info()
```
"""
check_is_fitted(self)
info = pd.DataFrame(self.topic_sizes_.items(), columns=["Topic", "Count"]).sort_values("Topic")
info["Name"] = info.Topic.map(self.topic_labels_)
# Custom label
if self.custom_labels_ is not None:
if len(self.custom_labels_) == len(info):
labels = {topic - self._outliers: label for topic, label in enumerate(self.custom_labels_)}
info["CustomName"] = info["Topic"].map(labels)
# Main Keywords
values = {topic: list(next(zip(*values))) for topic, values in self.topic_representations_.items()}
info["Representation"] = info["Topic"].map(values)
# Extract all topic aspects
if self.topic_aspects_:
for aspect, values in self.topic_aspects_.items():
if isinstance(list(values.values())[-1], list):
if isinstance(list(values.values())[-1][0], tuple) or isinstance(
list(values.values())[-1][0], list
):
values = {topic: list(next(zip(*value))) for topic, value in values.items()}
elif isinstance(list(values.values())[-1][0], str):
values = {topic: " ".join(value).strip() for topic, value in values.items()}
info[aspect] = info["Topic"].map(values)
# Representative Docs / Images
if self.representative_docs_ is not None:
info["Representative_Docs"] = info["Topic"].map(self.representative_docs_)
if self.representative_images_ is not None:
info["Representative_Images"] = info["Topic"].map(self.representative_images_)
# Select specific topic to return
if topic is not None:
info = info.loc[info.Topic == topic, :]
return info.reset_index(drop=True)
def get_topic_freq(self, topic: int | None = None) -> Union[pd.DataFrame, int]:
"""Return the size of topics (descending order).
Arguments:
topic: A specific topic for which you want the frequency
Returns:
Either the frequency of a single topic or dataframe with
the frequencies of all topics
Examples:
To extract the frequency of all topics:
```python
frequency = topic_model.get_topic_freq()
```
To get the frequency of a single topic:
```python
frequency = topic_model.get_topic_freq(12)
```
"""
check_is_fitted(self)
if isinstance(topic, int):
return self.topic_sizes_[topic]
else:
return pd.DataFrame(self.topic_sizes_.items(), columns=["Topic", "Count"]).sort_values(
"Count", ascending=False
)
def get_document_info(
self,
docs: List[str],
df: pd.DataFrame = None,
metadata: Mapping[str, Any] | None = None,
) -> pd.DataFrame:
"""Get information about the documents on which the topic was trained
including the documents themselves, their respective topics, the name
of each topic, the top n words of each topic, whether it is a
representative document, and probability of the clustering if the cluster
model supports it.
There are also options to include other meta data, such as the topic
distributions or the x and y coordinates of the reduced embeddings.
Arguments:
docs: The documents on which the topic model was trained.
df: A dataframe containing the metadata and the documents on which
the topic model was originally trained on.
metadata: A dictionary with meta data for each document in the form
of column name (key) and the respective values (value).
Returns:
document_info: A dataframe with several statistics regarding
the documents on which the topic model was trained.
Usage:
To get the document info, you will only need to pass the documents on which
the topic model was trained:
```python
document_info = topic_model.get_document_info(docs)
```
There are additionally options to include meta data, such as the topic
distributions. Moreover, we can pass the original dataframe that contains
the documents and extend it with the information retrieved from BERTopic:
```python
from sklearn.datasets import fetch_20newsgroups
# The original data in a dataframe format to include the target variable
data = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes'))
df = pd.DataFrame({"Document": data['data'], "Class": data['target']})
# Add information about the percentage of the document that relates to the topic
topic_distr, _ = topic_model.approximate_distribution(docs, batch_size=1000)
distributions = [distr[topic] if topic != -1 else 0 for topic, distr in zip(topics, topic_distr)]
# Create our documents dataframe using the original dataframe and meta data about
# the topic distributions
document_info = topic_model.get_document_info(docs, df=df,
metadata={"Topic_distribution": distributions})
"""
check_documents_type(docs)
if df is not None:
document_info = df.copy()
document_info["Document"] = docs
document_info["Topic"] = self.topics_
else:
document_info = pd.DataFrame({"Document": docs, "Topic": self.topics_})
# Add topic info through `.get_topic_info()`
topic_info = self.get_topic_info().drop("Count", axis=1)
document_info = document_info.merge(topic_info, on="Topic", how="left")
# Add top n words
top_n_words = {topic: " - ".join(next(zip(*self.get_topic(topic)))) for topic in set(self.topics_)}
document_info["Top_n_words"] = document_info.Topic.map(top_n_words)
# Add flat probabilities
if self.probabilities_ is not None:
if len(self.probabilities_.shape) == 1:
document_info["Probability"] = self.probabilities_
else:
document_info["Probability"] = [
max(probs) if topic != -1 else 1 - sum(probs)
for topic, probs in zip(self.topics_, self.probabilities_)
]
# Add representative document labels
repr_docs = [repr_doc for repr_docs in self.representative_docs_.values() for repr_doc in repr_docs]
document_info["Representative_document"] = False
document_info.loc[document_info.Document.isin(repr_docs), "Representative_document"] = True
# Add custom meta data provided by the user
if metadata is not None:
for column, values in metadata.items():
document_info[column] = values
return document_info
def get_representative_docs(self, topic: int | None = None) -> List[str]:
"""Extract the best representing documents per topic.
Note:
This does not extract all documents per topic as all documents
are not saved within BERTopic. To get all documents, please
run the following:
```python
# When you used `.fit_transform`:
df = pd.DataFrame({"Document": docs, "Topic": topic})
# When you used `.fit`:
df = pd.DataFrame({"Document": docs, "Topic": topic_model.topics_})
```
Arguments:
topic: A specific topic for which you want
the representative documents
Returns:
Representative documents of the chosen topic
Examples:
To extract the representative docs of all topics:
```python
representative_docs = topic_model.get_representative_docs()
```
To get the representative docs of a single topic:
```python
representative_docs = topic_model.get_representative_docs(12)
```
"""
check_is_fitted(self)
if isinstance(topic, int):
if self.representative_docs_.get(topic):
return self.representative_docs_[topic]
else:
return None
else:
return self.representative_docs_
@staticmethod
def get_topic_tree(
hier_topics: pd.DataFrame,
max_distance: float | None = None,
tight_layout: bool = False,
) -> str:
"""Extract the topic tree such that it can be printed.
Arguments:
hier_topics: A dataframe containing the structure of the topic tree.
This is the output of `topic_model.hierarchical_topics()`
max_distance: The maximum distance between two topics. This value is
based on the Distance column in `hier_topics`.
tight_layout: Whether to use a tight layout (narrow width) for
easier readability if you have hundreds of topics.
Returns:
A tree that has the following structure when printed:
.
.
└─health_medical_disease_patients_hiv
├─patients_medical_disease_candida_health
│ ├─■──candida_yeast_infection_gonorrhea_infections ── Topic: 48
│ └─patients_disease_cancer_medical_doctor
│ ├─■──hiv_medical_cancer_patients_doctor ── Topic: 34
│ └─■──pain_drug_patients_disease_diet ── Topic: 26
└─■──health_newsgroup_tobacco_vote_votes ── Topic: 9
The blocks (■) indicate that the topic is one you can directly access
from `topic_model.get_topic`. In other words, they are the original un-grouped topics.
Examples:
```python
# Train model
from bertopic import BERTopic
topic_model = BERTopic()
topics, probs = topic_model.fit_transform(docs)
hierarchical_topics = topic_model.hierarchical_topics(docs)
# Print topic tree
tree = topic_model.get_topic_tree(hierarchical_topics)
print(tree)
```
"""
width = 1 if tight_layout else 4
if max_distance is None:
max_distance = hier_topics.Distance.max() + 1
max_original_topic = hier_topics.Parent_ID.astype(int).min() - 1
# Extract mapping from ID to name
topic_to_name = dict(zip(hier_topics.Child_Left_ID, hier_topics.Child_Left_Name))
topic_to_name.update(dict(zip(hier_topics.Child_Right_ID, hier_topics.Child_Right_Name)))
topic_to_name = {topic: name[:100] for topic, name in topic_to_name.items()}
# Create tree
tree = {
str(row[1].Parent_ID): [
str(row[1].Child_Left_ID),
str(row[1].Child_Right_ID),
]
for row in hier_topics.iterrows()
}
def get_tree(start, tree):
"""Based on: https://stackoverflow.com/a/51920869/10532563."""
def _tree(to_print, start, parent, tree, grandpa=None, indent=""):
# Get distance between merged topics
distance = hier_topics.loc[
(hier_topics.Child_Left_ID == parent) | (hier_topics.Child_Right_ID == parent),
"Distance",
]
distance = distance.to_numpy()[0] if len(distance) > 0 else 10
if parent != start:
if grandpa is None:
to_print += topic_to_name[parent]
else:
if int(parent) <= max_original_topic:
# Do not append topic ID if they are not merged
if distance < max_distance:
to_print += "■──" + topic_to_name[parent] + f" ── Topic: {parent}" + "\n"
else:
to_print += "O \n"
else:
to_print += topic_to_name[parent] + "\n"
if parent not in tree:
return to_print
for child in tree[parent][:-1]:
to_print += indent + "├" + "─"
to_print = _tree(to_print, start, child, tree, parent, indent + "│" + " " * width)
child = tree[parent][-1]
to_print += indent + "└" + "─"
to_print = _tree(to_print, start, child, tree, parent, indent + " " * (width + 1))
return to_print
to_print = "." + "\n"
to_print = _tree(to_print, start, start, tree)
return to_print
start = str(hier_topics.Parent_ID.astype(int).max())
return get_tree(start, tree)
def set_topic_labels(self, topic_labels: Union[List[str], Mapping[int, str]]) -> None:
"""Set custom topic labels in your fitted BERTopic model.
Arguments:
topic_labels: If a list of topic labels, it should contain the same number
of labels as there are topics. This must be ordered
from the topic with the lowest ID to the highest ID,
including topic -1 if it exists.
If a dictionary of `topic ID`: `topic_label`, it can have
any number of topics as it will only map the topics found
in the dictionary.
Examples:
First, we define our topic labels with `.generate_topic_labels` in which
we can customize our topic labels:
```python
topic_labels = topic_model.generate_topic_labels(nr_words=2,
topic_prefix=True,
word_length=10,
separator=", ")
```
Then, we pass these `topic_labels` to our topic model which
can be accessed at any time with `.custom_labels_`:
```python
topic_model.set_topic_labels(topic_labels)
topic_model.custom_labels_
```
You might want to change only a few topic labels instead of all of them.
To do so, you can pass a dictionary where the keys are the topic IDs and
its keys the topic labels:
```python
topic_model.set_topic_labels({0: "Space", 1: "Sports", 2: "Medicine"})
topic_model.custom_labels_
```
"""
unique_topics = sorted(set(self.topics_))
if isinstance(topic_labels, dict):
if self.custom_labels_ is not None:
original_labels = {topic: label for topic, label in zip(unique_topics, self.custom_labels_)}
else:
info = self.get_topic_info()
original_labels = dict(zip(info.Topic, info.Name))
custom_labels = [
topic_labels.get(topic) if topic_labels.get(topic) else original_labels[topic]
for topic in unique_topics
]
elif isinstance(topic_labels, list):
if len(topic_labels) == len(unique_topics):
custom_labels = topic_labels
else:
raise ValueError(
"Make sure that `topic_labels` contains the same number of labels as there are topics."
)
self.custom_labels_ = custom_labels
def generate_topic_labels(
self,
nr_words: int = 3,
topic_prefix: bool = True,
word_length: int | None = None,
separator: str = "_",
aspect: str | None = None,
) -> List[str]:
"""Get labels for each topic in a user-defined format.
Arguments:
nr_words: Top `n` words per topic to use
topic_prefix: Whether to use the topic ID as a prefix.
If set to True, the topic ID will be separated
using the `separator`
word_length: The maximum length of each word in the topic label.
Some words might be relatively long and setting this
value helps to make sure that all labels have relatively
similar lengths.
separator: The string with which the words and topic prefix will be
separated. Underscores are the default but a nice alternative
is `", "`.
aspect: The aspect from which to generate topic labels
Returns:
topic_labels: A list of topic labels sorted from the lowest topic ID to the highest.
If the topic model was trained using HDBSCAN, the lowest topic ID is -1,
otherwise it is 0.
Examples:
To create our custom topic labels, usage is rather straightforward:
```python
topic_labels = topic_model.generate_topic_labels(nr_words=2, separator=", ")
```
"""
unique_topics = sorted(set(self.topics_))
topic_labels = []
for topic in unique_topics:
if aspect:
words, _ = zip(*self.topic_aspects_[aspect][topic])
else:
words, _ = zip(*self.get_topic(topic))
if word_length:
words = [word[:word_length] for word in words][:nr_words]
else:
words = list(words)[:nr_words]
if topic_prefix:
topic_label = f"{topic}{separator}" + separator.join(words)
else:
topic_label = separator.join(words)
topic_labels.append(topic_label)
return topic_labels
def merge_topics(
self,
docs: List[str],
topics_to_merge: List[Union[Iterable[int], int]],
images: List[str] | None = None,
) -> None:
"""Arguments:
docs: The documents you used when calling either `fit` or `fit_transform`
topics_to_merge: Either a list of topics or a list of list of topics
to merge. For example:
[1, 2, 3] will merge topics 1, 2 and 3
[[1, 2], [3, 4]] will merge topics 1 and 2, and
separately merge topics 3 and 4.
images: A list of paths to the images used when calling either
`fit` or `fit_transform`.
Examples:
If you want to merge topics 1, 2, and 3:
```python
topics_to_merge = [1, 2, 3]
topic_model.merge_topics(docs, topics_to_merge)
```
or if you want to merge topics 1 and 2, and separately
merge topics 3 and 4:
```python
topics_to_merge = [[1, 2],
[3, 4]]
topic_model.merge_topics(docs, topics_to_merge)
```
"""
check_is_fitted(self)
check_documents_type(docs)
documents = pd.DataFrame(
{
"Document": docs,
"Topic": self.topics_,
"Image": images,
"ID": range(len(docs)),
}
)
mapping = {topic: topic for topic in set(self.topics_)}
if isinstance(topics_to_merge[0], int):
for topic in sorted(topics_to_merge):
mapping[topic] = topics_to_merge[0]
elif isinstance(topics_to_merge[0], Iterable):
for topic_group in sorted(topics_to_merge):
for topic in topic_group:
mapping[topic] = topic_group[0]
else:
raise ValueError("Make sure that `topics_to_merge` is eithera list of topics or a list of list of topics.")
# Track mappings and sizes of topics for merging topic embeddings
mappings = defaultdict(list)
for key, val in sorted(mapping.items()):
mappings[val].append(key)
mappings = {
topic_to: {
"topics_from": topics_from,
"topic_sizes": [self.topic_sizes_[topic] for topic in topics_from],
}
for topic_to, topics_from in mappings.items()
}
# Update topics
documents.Topic = documents.Topic.map(mapping)
self.topic_mapper_.add_mappings(mapping, topic_model=self)
documents = self._sort_mappings_by_frequency(documents)
self._extract_topics(documents, mappings=mappings)
self._update_topic_size(documents)
self._save_representative_docs(documents)
self.probabilities_ = self._map_probabilities(self.probabilities_)
def delete_topics(
self,
topics_to_delete: List[int],
) -> None:
"""Delete topics from the topic model.
The deleted topics will be mapped to -1 (outlier topic). Core topic attributes
like topic embeddings and c-TF-IDF will be automatically updated.
Arguments:
topics_to_delete: List of topics to delete
"""
check_is_fitted(self)
topics_df = pd.DataFrame({"Topic": self.topics_})
# Check if -1 exists in the current topics
had_outliers = -1 in set(self.topics_)
# If adding -1 for the first time, initialize its attributes
if not had_outliers and any(topic in topics_to_delete for topic in self.topics_):
# Initialize c-TF-IDF for -1 topic (zeros)
outlier_row = np.zeros((1, self.c_tf_idf_.shape[1]))
outlier_row = sp.csr_matrix(outlier_row)
self.c_tf_idf_ = sp.vstack([outlier_row, self.c_tf_idf_])
# Initialize topic embeddings for -1 topic (zeros)
outlier_embedding = np.zeros((1, self.topic_embeddings_.shape[1]))
self.topic_embeddings_ = np.vstack([outlier_embedding, self.topic_embeddings_])
# Initialize topic representations for -1 topic: ("", 1e-05)
self.topic_representations_[-1] = [("", 1e-05)]
# Initialize representative docs for -1 topic (empty list)
self.representative_docs_[-1] = []
# Initialize representative images for -1 topic if images are being used
if self.representative_images_ is not None:
outlier_image = np.zeros((1, self.representative_images_.shape[1]))
self.representative_images_ = np.vstack([outlier_image, self.representative_images_])
# Initialize custom labels for -1 topic if they exist
if hasattr(self, "custom_labels_") and self.custom_labels_ is not None:
self.custom_labels_[-1] = ""
# Initialize ctfidf model diagonal for -1 topic (ones) if it exists
if hasattr(self, "ctfidf_model") and self.ctfidf_model is not None:
n_features = self.ctfidf_model._idf_diag.shape[1]
outlier_diag = sp.csr_matrix(([1.0], ([0], [0])), shape=(1, n_features))
self.ctfidf_model._idf_diag = sp.vstack([outlier_diag, self.ctfidf_model._idf_diag])
# Initialize topic aspects for -1 topic (empty dict for each aspect) if they exist
if hasattr(self, "topic_aspects_") and self.topic_aspects_ is not None:
for aspect in self.topic_aspects_:
self.topic_aspects_[aspect][-1] = {}
# First map deleted topics to -1
mapping = {topic: -1 if topic in topics_to_delete else topic for topic in set(self.topics_)}
mapping[-1] = -1
# Track mappings and sizes of topics for merging topic embeddings
mappings = defaultdict(list)
for key, val in sorted(mapping.items()):
mappings[val].append(key)
mappings = {
topic_to: {
"topics_from": topics_from,
"topic_sizes": [self.topic_sizes_[topic] for topic in topics_from],
}
for topic_to, topics_from in mappings.items()
}
# remove deleted topics and update attributes
topics_df.Topic = topics_df.Topic.map(mapping)
self.topic_mapper_.add_mappings(mapping, topic_model=deepcopy(self))
topics_df = self._sort_mappings_by_frequency(topics_df)
self._update_topic_size(topics_df)
self.probabilities_ = self._map_probabilities(self.probabilities_)
final_mapping = self.topic_mapper_.get_mappings(original_topics=False)
# Update dictionary-based attributes to remove deleted topics
# Handle topic_aspects_ if it exists
if hasattr(self, "topic_aspects_") and self.topic_aspects_ is not None:
new_aspects = {
aspect: {
(final_mapping[old_topic] if old_topic != -1 else -1): content
for old_topic, content in topics.items()
if old_topic not in topics_to_delete
}
for aspect, topics in self.topic_aspects_.items()
}
self.topic_aspects_ = new_aspects
# Update custom labels if they exist
if hasattr(self, "custom_labels_") and self.custom_labels_ is not None:
new_labels = {
(final_mapping[old_topic] if old_topic != -1 else -1): label
for old_topic, label in self.custom_labels_.items()
if old_topic not in topics_to_delete
}
self.custom_labels_ = new_labels
# Update topic representations
new_representations = {
(final_mapping[old_topic] if old_topic != -1 else -1): content
for old_topic, content in self.topic_representations_.items()
if old_topic not in topics_to_delete
}
self.topic_representations_ = new_representations
# Update representative docs if they exist
new_representative_docs = {
(final_mapping[old_topic] if old_topic != -1 else -1): docs
for old_topic, docs in self.representative_docs_.items()
if old_topic not in topics_to_delete
}
self.representative_docs_ = new_representative_docs
# Update representative images if they exist
if self.representative_images_ is not None:
# Create a mask for non-deleted topics
mask = np.array([topic not in topics_to_delete for topic in range(len(self.representative_images_))])
self.representative_images_ = self.representative_images_[mask] if mask.any() else None
# Update array-based attributes using masks to remove deleted topics
for attr in ["topic_embeddings_", "c_tf_idf_"]:
matrix = getattr(self, attr)
mask = np.array([topic not in topics_to_delete for topic in range(matrix.shape[0])])
setattr(self, attr, matrix[mask])
# Update ctfidf model to remove deleted topics if it exists
if hasattr(self, "ctfidf_model") and self.ctfidf_model is not None:
mask = np.array([topic not in topics_to_delete for topic in range(self.ctfidf_model._idf_diag.shape[0])])
self.ctfidf_model._idf_diag = self.ctfidf_model._idf_diag[mask]
def reduce_topics(
self,
docs: List[str],
nr_topics: Union[int, str] = 20,
images: List[str] | None = None,
use_ctfidf: bool = False,
) -> None:
"""Reduce the number of topics to a fixed number of topics
or automatically.
If nr_topics is an integer, then the number of topics is reduced
to nr_topics using `AgglomerativeClustering` on the cosine distance matrix
of the topic c-TF-IDF or semantic embeddings.
If nr_topics is `"auto"`, then HDBSCAN is used to automatically
reduce the number of topics by running it on the topic embeddings.
The topics, their sizes, and representations are updated.
Arguments:
docs: The docs you used when calling either `fit` or `fit_transform`
nr_topics: The number of topics you want reduced to
images: A list of paths to the images used when calling either
`fit` or `fit_transform`
use_ctfidf: Whether to calculate distances between topics based on c-TF-IDF embeddings. If False, the
embeddings from the embedding model are used.
Updates:
topics_ : Assigns topics to their merged representations.
probabilities_ : Assigns probabilities to their merged representations.
Examples:
You can further reduce the topics by passing the documents with their
topics and probabilities (if they were calculated):
```python
topic_model.reduce_topics(docs, nr_topics=30)
```
You can then access the updated topics and probabilities with:
```python
topics = topic_model.topics_
probabilities = topic_model.probabilities_
```
"""
check_is_fitted(self)
check_documents_type(docs)
self.nr_topics = nr_topics
documents = pd.DataFrame(
{
"Document": docs,
"Topic": self.topics_,
"Image": images,
"ID": range(len(docs)),
}
)
# Reduce number of topics
documents = self._reduce_topics(documents, use_ctfidf)
self._merged_topics = None
self._save_representative_docs(documents)
self.probabilities_ = self._map_probabilities(self.probabilities_)
return self
def reduce_outliers(
self,
documents: List[str],
topics: List[int],
images: List[str] | None = None,
strategy: str = "distributions",
probabilities: np.ndarray = None,
threshold: float = 0,
embeddings: np.ndarray = None,
distributions_params: Mapping[str, Any] = {},
) -> List[int]:
"""Reduce outliers by merging them with their nearest topic according
to one of several strategies.
When using HDBSCAN, DBSCAN, or OPTICS, a number of outlier documents might be created
that do not fall within any of the created topics. These are labeled as -1.
This function allows the user to match outlier documents with their nearest topic
using one of the following strategies using the `strategy` parameter:
* "probabilities"
This uses the soft-clustering as performed by HDBSCAN to find the
best matching topic for each outlier document. To use this, make
sure to calculate the `probabilities` beforehand by instantiating
BERTopic with `calculate_probabilities=True`.
* "distributions"
Use the topic distributions, as calculated with `.approximate_distribution`
to find the most frequent topic in each outlier document. You can use the
`distributions_params` variable to tweak the parameters of
`.approximate_distribution`.
* "c-tf-idf"
Calculate the c-TF-IDF representation for each outlier document and
find the best matching c-TF-IDF topic representation using
cosine similarity.
* "embeddings"
Using the embeddings of each outlier documents, find the best
matching topic embedding using cosine similarity.
Arguments:
documents: A list of documents for which we reduce or remove the outliers.
topics: The topics that correspond to the documents
images: A list of paths to the images used when calling either
`fit` or `fit_transform`
strategy: The strategy used for reducing outliers.
Options:
* "probabilities"
This uses the soft-clustering as performed by HDBSCAN
to find the best matching topic for each outlier document.
* "distributions"
Use the topic distributions, as calculated with `.approximate_distribution`
to find the most frequent topic in each outlier document.
* "c-tf-idf"
Calculate the c-TF-IDF representation for outlier documents and
find the best matching c-TF-IDF topic representation.
* "embeddings"
Calculate the embeddings for outlier documents and
find the best matching topic embedding.
probabilities: Probabilities generated by HDBSCAN for each document when using the strategy `"probabilities"`.
threshold: The threshold for assigning topics to outlier documents. This value
represents the minimum probability when `strategy="probabilities"`.
For all other strategies, it represents the minimum similarity.
embeddings: The pre-computed embeddings to be used when `strategy="embeddings"`.
If this is None, then it will compute the embeddings for the outlier documents.
distributions_params: The parameters used in `.approximate_distribution` when using
the strategy `"distributions"`.
Returns:
new_topics: The updated topics
Usage:
The default settings uses the `"distributions"` strategy:
```python
new_topics = topic_model.reduce_outliers(docs, topics)
```
When you use the `"probabilities"` strategy, make sure to also pass the probabilities
as generated through HDBSCAN:
```python
from bertopic import BERTopic
topic_model = BERTopic(calculate_probabilities=True)
topics, probs = topic_model.fit_transform(docs)
new_topics = topic_model.reduce_outliers(docs, topics, probabilities=probs, strategy="probabilities")
```
"""
if not self._outliers:
raise ValueError("No outliers to reduce.")
if images is not None:
strategy = "embeddings"
# Check correct use of parameters
if strategy.lower() == "probabilities" and probabilities is None:
raise ValueError("Make sure to pass in `probabilities` in order to use the probabilities strategy")
# Reduce outliers by extracting most likely topics through the topic-term probability matrix
if strategy.lower() == "probabilities":
new_topics = [
np.argmax(prob) if np.max(prob) >= threshold and topic == -1 else topic
for topic, prob in zip(topics, probabilities)
]
# Reduce outliers by extracting most frequent topics through calculating of Topic Distributions
elif strategy.lower() == "distributions":
outlier_ids = [index for index, topic in enumerate(topics) if topic == -1]
outlier_docs = [documents[index] for index in outlier_ids]
topic_distr, _ = self.approximate_distribution(
outlier_docs, min_similarity=threshold, **distributions_params
)
outlier_topics = iter([np.argmax(prob) if sum(prob) > 0 else -1 for prob in topic_distr])
new_topics = [topic if topic != -1 else next(outlier_topics) for topic in topics]
# Reduce outliers by finding the most similar c-TF-IDF representations
elif strategy.lower() == "c-tf-idf":
outlier_ids = [index for index, topic in enumerate(topics) if topic == -1]
outlier_docs = [documents[index] for index in outlier_ids]
# Calculate c-TF-IDF of outlier documents with all topics
bow_doc = self.vectorizer_model.transform(outlier_docs)
c_tf_idf_doc = self.ctfidf_model.transform(bow_doc)
similarity = cosine_similarity(c_tf_idf_doc, self.c_tf_idf_[self._outliers :])
# Update topics
similarity[similarity < threshold] = 0
outlier_topics = iter([np.argmax(sim) if sum(sim) > 0 else -1 for sim in similarity])
new_topics = [topic if topic != -1 else next(outlier_topics) for topic in topics]
# Reduce outliers by finding the most similar topic embeddings
elif strategy.lower() == "embeddings":
if self.embedding_model is None and embeddings is None:
raise ValueError(
"To use this strategy, you will need to pass a model to `embedding_model`"
"when instantiating BERTopic."
)
outlier_ids = [index for index, topic in enumerate(topics) if topic == -1]
if images is not None:
outlier_docs = [images[index] for index in outlier_ids]
else:
outlier_docs = [documents[index] for index in outlier_ids]
# Extract or calculate embeddings for outlier documents
if embeddings is not None:
outlier_embeddings = np.array([embeddings[index] for index in outlier_ids])
elif images is not None:
outlier_images = [images[index] for index in outlier_ids]
outlier_embeddings = self.embedding_model.embed_images(outlier_images, verbose=self.verbose)
else:
outlier_embeddings = self.embedding_model.embed_documents(outlier_docs)
similarity = cosine_similarity(outlier_embeddings, self.topic_embeddings_[self._outliers :])
# Update topics
similarity[similarity < threshold] = 0
outlier_topics = iter([np.argmax(sim) if sum(sim) > 0 else -1 for sim in similarity])
new_topics = [topic if topic != -1 else next(outlier_topics) for topic in topics]
return new_topics
def visualize_topics(
self,
topics: List[int] | None = None,
top_n_topics: int | None = None,
use_ctfidf: bool = False,
custom_labels: bool = False,
title: str = "<b>Intertopic Distance Map</b>",
width: int = 650,
height: int = 650,
) -> "go.Figure":
"""Visualize topics, their sizes, and their corresponding words.
This visualization is highly inspired by LDAvis, a great visualization
technique typically reserved for LDA.
Arguments:
topics: A selection of topics to visualize
Not to be confused with the topics that you get from `.fit_transform`.
For example, if you want to visualize only topics 1 through 5:
`topics = [1, 2, 3, 4, 5]`.
top_n_topics: Only select the top n most frequent topics
use_ctfidf: Whether to use c-TF-IDF representations instead of the embeddings from the embedding model.
custom_labels: Whether to use custom topic labels that were defined using
`topic_model.set_topic_labels`.
title: Title of the plot.
width: The width of the figure.
height: The height of the figure.
Examples:
To visualize the topics simply run:
```python
topic_model.visualize_topics()
```
Or if you want to save the resulting figure:
```python
fig = topic_model.visualize_topics()
fig.write_html("path/to/file.html")
```
"""
check_is_fitted(self)
return plotting.visualize_topics(
self,
topics=topics,
top_n_topics=top_n_topics,
use_ctfidf=use_ctfidf,
custom_labels=custom_labels,
title=title,
width=width,
height=height,
)
def visualize_documents(
self,
docs: List[str],
topics: List[int] | None = None,
embeddings: np.ndarray = None,
reduced_embeddings: np.ndarray = None,
sample: float | None = None,
hide_annotations: bool = False,
hide_document_hover: bool = False,
custom_labels: bool = False,
title: str = "<b>Documents and Topics</b>",
width: int = 1200,
height: int = 750,
) -> "go.Figure":
"""Visualize documents and their topics in 2D.
Arguments:
topic_model: A fitted BERTopic instance.
docs: The documents you used when calling either `fit` or `fit_transform`
topics: A selection of topics to visualize.
Not to be confused with the topics that you get from `.fit_transform`.
For example, if you want to visualize only topics 1 through 5:
`topics = [1, 2, 3, 4, 5]`.
embeddings: The embeddings of all documents in `docs`.
reduced_embeddings: The 2D reduced embeddings of all documents in `docs`.
sample: The percentage of documents in each topic that you would like to keep.
Value can be between 0 and 1. Setting this value to, for example,
0.1 (10% of documents in each topic) makes it easier to visualize
millions of documents as a subset is chosen.
hide_annotations: Hide the names of the traces on top of each cluster.
hide_document_hover: Hide the content of the documents when hovering over
specific points. Helps to speed up generation of visualization.
custom_labels: Whether to use custom topic labels that were defined using
`topic_model.set_topic_labels`.
title: Title of the plot.
width: The width of the figure.
height: The height of the figure.
Examples:
To visualize the topics simply run:
```python
topic_model.visualize_documents(docs)
```
Do note that this re-calculates the embeddings and reduces them to 2D.
The advised and preferred pipeline for using this function is as follows:
```python
from sklearn.datasets import fetch_20newsgroups
from sentence_transformers import SentenceTransformer
from bertopic import BERTopic
from umap import UMAP
# Prepare embeddings
docs = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes'))['data']
sentence_model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = sentence_model.encode(docs, show_progress_bar=False)
# Train BERTopic
topic_model = BERTopic().fit(docs, embeddings)
# Reduce dimensionality of embeddings, this step is optional
# reduced_embeddings = UMAP(n_neighbors=10, n_components=2, min_dist=0.0, metric='cosine').fit_transform(embeddings)
# Run the visualization with the original embeddings
topic_model.visualize_documents(docs, embeddings=embeddings)
# Or, if you have reduced the original embeddings already:
topic_model.visualize_documents(docs, reduced_embeddings=reduced_embeddings)
```
Or if you want to save the resulting figure:
```python
fig = topic_model.visualize_documents(docs, reduced_embeddings=reduced_embeddings)
fig.write_html("path/to/file.html")
```
<iframe src="../getting_started/visualization/documents.html"
style="width:1000px; height: 800px; border: 0px;""></iframe>
"""
check_is_fitted(self)
check_documents_type(docs)
return plotting.visualize_documents(
self,
docs=docs,
topics=topics,
embeddings=embeddings,
reduced_embeddings=reduced_embeddings,
sample=sample,
hide_annotations=hide_annotations,
hide_document_hover=hide_document_hover,
custom_labels=custom_labels,
title=title,
width=width,
height=height,
)
def visualize_document_datamap(
self,
docs: List[str] | None = None,
topics: List[int] | None = None,
embeddings: np.ndarray = None,
reduced_embeddings: np.ndarray = None,
custom_labels: Union[bool, str] = False,
title: str = "Documents and Topics",
sub_title: Union[str, None] = None,
width: int = 1200,
height: int = 750,
interactive: bool = False,
enable_search: bool = False,
topic_prefix: bool = False,
datamap_kwds: dict = {},
int_datamap_kwds: dict = {},
) -> "fig.Figure":
"""Visualize documents and their topics in 2D as a static plot for publication using
DataMapPlot. This works best if there are between 5 and 60 topics. It is therefore best
to use a sufficiently large `min_topic_size` or set `nr_topics` when building the model.
Arguments:
topic_model: A fitted BERTopic instance.
docs: The documents you used when calling either `fit` or `fit_transform`.
topics: A selection of topics to visualize.
Not to be confused with the topics that you get from `.fit_transform`.
For example, if you want to visualize only topics 1 through 5:
`topics = [1, 2, 3, 4, 5]`. Documents not in these topics will be shown
as noise points.
embeddings: The embeddings of all documents in `docs`.
reduced_embeddings: The 2D reduced embeddings of all documents in `docs`.
custom_labels: If bool, whether to use custom topic labels that were defined using
`topic_model.set_topic_labels`.
If `str`, it uses labels from other aspects, e.g., "Aspect1".
title: Title of the plot.
sub_title: Sub-title of the plot.
width: The width of the figure.
height: The height of the figure.
interactive: Whether to create an interactive plot using DataMapPlot's `create_interactive_plot`.
enable_search: Whether to enable search in the interactive plot. Only works if `interactive=True`.
topic_prefix: Prefix to add to the topic number when displaying the topic name.
datamap_kwds: Keyword args be passed on to DataMapPlot's `create_plot` function
if you are not using the interactive version.
See the DataMapPlot documentation for more details.
int_datamap_kwds: Keyword args be passed on to DataMapPlot's `create_interactive_plot` function
if you are using the interactive version.
See the DataMapPlot documentation for more details.
Returns:
figure: A Matplotlib Figure object.
Examples:
To visualize the topics simply run:
```python
topic_model.visualize_document_datamap(docs)
```
Do note that this re-calculates the embeddings and reduces them to 2D.
The advised and preferred pipeline for using this function is as follows:
```python
from sklearn.datasets import fetch_20newsgroups
from sentence_transformers import SentenceTransformer
from bertopic import BERTopic
from umap import UMAP
# Prepare embeddings
docs = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes'))['data']
sentence_model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = sentence_model.encode(docs, show_progress_bar=False)
# Train BERTopic
topic_model = BERTopic(min_topic_size=36).fit(docs, embeddings)
# Reduce dimensionality of embeddings, this step is optional
# reduced_embeddings = UMAP(n_neighbors=10, n_components=2, min_dist=0.0, metric='cosine').fit_transform(embeddings)
# Run the visualization with the original embeddings
topic_model.visualize_document_datamap(docs, embeddings=embeddings)
# Or, if you have reduced the original embeddings already:
topic_model.visualize_document_datamap(docs, reduced_embeddings=reduced_embeddings)
```
Or if you want to save the resulting figure:
```python
fig = topic_model.visualize_document_datamap(docs, reduced_embeddings=reduced_embeddings)
fig.savefig("path/to/file.png", bbox_inches="tight")
```
"""
check_is_fitted(self)
return plotting.visualize_document_datamap(
self,
docs,
topics,
embeddings,
reduced_embeddings,
custom_labels,
title,
sub_title,
width,
height,
interactive,
enable_search,
topic_prefix,
datamap_kwds,
int_datamap_kwds,
)
def visualize_hierarchical_documents(
self,
docs: List[str],
hierarchical_topics: pd.DataFrame,
topics: List[int] | None = None,
embeddings: np.ndarray = None,
reduced_embeddings: np.ndarray = None,
sample: Union[float, int] | None = None,
hide_annotations: bool = False,
hide_document_hover: bool = True,
nr_levels: int = 10,
level_scale: str = "linear",
custom_labels: bool = False,
title: str = "<b>Hierarchical Documents and Topics</b>",
width: int = 1200,
height: int = 750,
) -> "go.Figure":
"""Visualize documents and their topics in 2D at different levels of hierarchy.
Arguments:
docs: The documents you used when calling either `fit` or `fit_transform`
hierarchical_topics: A dataframe that contains a hierarchy of topics
represented by their parents and their children
topics: A selection of topics to visualize.
Not to be confused with the topics that you get from `.fit_transform`.
For example, if you want to visualize only topics 1 through 5:
`topics = [1, 2, 3, 4, 5]`.
embeddings: The embeddings of all documents in `docs`.
reduced_embeddings: The 2D reduced embeddings of all documents in `docs`.
sample: The percentage of documents in each topic that you would like to keep.
Value can be between 0 and 1. Setting this value to, for example,
0.1 (10% of documents in each topic) makes it easier to visualize
millions of documents as a subset is chosen.
hide_annotations: Hide the names of the traces on top of each cluster.
hide_document_hover: Hide the content of the documents when hovering over
specific points. Helps to speed up generation of visualizations.
nr_levels: The number of levels to be visualized in the hierarchy. First, the distances
in `hierarchical_topics.Distance` are split in `nr_levels` lists of distances with
equal length. Then, for each list of distances, the merged topics, that have
a distance less or equal to the maximum distance of the selected list of distances, are selected.
NOTE: To get all possible merged steps, make sure that `nr_levels` is equal to
the length of `hierarchical_topics`.
level_scale: Whether to apply a linear or logarithmic ('log') scale levels of the distance
vector. Linear scaling will perform an equal number of merges at each level
while logarithmic scaling will perform more mergers in earlier levels to
provide more resolution at higher levels (this can be used for when the number
of topics is large).
custom_labels: Whether to use custom topic labels that were defined using
`topic_model.set_topic_labels`.
NOTE: Custom labels are only generated for the original
un-merged topics.
title: Title of the plot.
width: The width of the figure.
height: The height of the figure.
Examples:
To visualize the topics simply run:
```python
topic_model.visualize_hierarchical_documents(docs, hierarchical_topics)
```
Do note that this re-calculates the embeddings and reduces them to 2D.
The advised and preferred pipeline for using this function is as follows:
```python
from sklearn.datasets import fetch_20newsgroups
from sentence_transformers import SentenceTransformer
from bertopic import BERTopic
from umap import UMAP
# Prepare embeddings
docs = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes'))['data']
sentence_model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = sentence_model.encode(docs, show_progress_bar=False)
# Train BERTopic and extract hierarchical topics
topic_model = BERTopic().fit(docs, embeddings)
hierarchical_topics = topic_model.hierarchical_topics(docs)
# Reduce dimensionality of embeddings, this step is optional
# reduced_embeddings = UMAP(n_neighbors=10, n_components=2, min_dist=0.0, metric='cosine').fit_transform(embeddings)
# Run the visualization with the original embeddings
topic_model.visualize_hierarchical_documents(docs, hierarchical_topics, embeddings=embeddings)
# Or, if you have reduced the original embeddings already:
topic_model.visualize_hierarchical_documents(docs, hierarchical_topics, reduced_embeddings=reduced_embeddings)
```
Or if you want to save the resulting figure:
```python
fig = topic_model.visualize_hierarchical_documents(docs, hierarchical_topics, reduced_embeddings=reduced_embeddings)
fig.write_html("path/to/file.html")
```
<iframe src="../getting_started/visualization/hierarchical_documents.html"
style="width:1000px; height: 770px; border: 0px;""></iframe>
"""
check_is_fitted(self)
check_documents_type(docs)
return plotting.visualize_hierarchical_documents(
self,
docs=docs,
hierarchical_topics=hierarchical_topics,
topics=topics,
embeddings=embeddings,
reduced_embeddings=reduced_embeddings,
sample=sample,
hide_annotations=hide_annotations,
hide_document_hover=hide_document_hover,
nr_levels=nr_levels,
level_scale=level_scale,
custom_labels=custom_labels,
title=title,
width=width,
height=height,
)
def visualize_term_rank(
self,
topics: List[int] | None = None,
log_scale: bool = False,
custom_labels: bool = False,
title: str = "<b>Term score decline per Topic</b>",
width: int = 800,
height: int = 500,
) -> "go.Figure":
"""Visualize the ranks of all terms across all topics.
Each topic is represented by a set of words. These words, however,
do not all equally represent the topic. This visualization shows
how many words are needed to represent a topic and at which point
the beneficial effect of adding words starts to decline.
Arguments:
topics: A selection of topics to visualize. These will be colored
red where all others will be colored black.
log_scale: Whether to represent the ranking on a log scale
custom_labels: Whether to use custom topic labels that were defined using
`topic_model.set_topic_labels`.
title: Title of the plot.
width: The width of the figure.
height: The height of the figure.
Returns:
fig: A plotly figure
Examples:
To visualize the ranks of all words across
all topics simply run:
```python
topic_model.visualize_term_rank()
```
Or if you want to save the resulting figure:
```python
fig = topic_model.visualize_term_rank()
fig.write_html("path/to/file.html")
```
Reference:
This visualization was heavily inspired by the
"Term Probability Decline" visualization found in an
analysis by the amazing [tmtoolkit](https://tmtoolkit.readthedocs.io/).
Reference to that specific analysis can be found
[here](https://wzbsocialsciencecenter.github.io/tm_corona/tm_analysis.html).
"""
check_is_fitted(self)
return plotting.visualize_term_rank(
self,
topics=topics,
log_scale=log_scale,
custom_labels=custom_labels,
title=title,
width=width,
height=height,
)
def visualize_topics_over_time(
self,
topics_over_time: pd.DataFrame,
top_n_topics: int | None = None,
topics: List[int] | None = None,
normalize_frequency: bool = False,
custom_labels: bool = False,
title: str = "<b>Topics over Time</b>",
width: int = 1250,
height: int = 450,
) -> "go.Figure":
"""Visualize topics over time.
Arguments:
topics_over_time: The topics you would like to be visualized with the
corresponding topic representation
top_n_topics: To visualize the most frequent topics instead of all
topics: Select which topics you would like to be visualized
normalize_frequency: Whether to normalize each topic's frequency individually
custom_labels: Whether to use custom topic labels that were defined using
`topic_model.set_topic_labels`.
title: Title of the plot.
width: The width of the figure.
height: The height of the figure.
Returns:
A plotly.graph_objects.Figure including all traces
Examples:
To visualize the topics over time, simply run:
```python
topics_over_time = topic_model.topics_over_time(docs, timestamps)
topic_model.visualize_topics_over_time(topics_over_time)
```
Or if you want to save the resulting figure:
```python
fig = topic_model.visualize_topics_over_time(topics_over_time)
fig.write_html("path/to/file.html")
```
"""
check_is_fitted(self)
return plotting.visualize_topics_over_time(
self,
topics_over_time=topics_over_time,
top_n_topics=top_n_topics,
topics=topics,
normalize_frequency=normalize_frequency,
custom_labels=custom_labels,
title=title,
width=width,
height=height,
)
def visualize_topics_per_class(
self,
topics_per_class: pd.DataFrame,
top_n_topics: int = 10,
topics: List[int] | None = None,
normalize_frequency: bool = False,
custom_labels: bool = False,
title: str = "<b>Topics per Class</b>",
width: int = 1250,
height: int = 900,
) -> "go.Figure":
"""Visualize topics per class.
Arguments:
topics_per_class: The topics you would like to be visualized with the
corresponding topic representation
top_n_topics: To visualize the most frequent topics instead of all
topics: Select which topics you would like to be visualized
normalize_frequency: Whether to normalize each topic's frequency individually
custom_labels: Whether to use custom topic labels that were defined using
`topic_model.set_topic_labels`.
title: Title of the plot.
width: The width of the figure.
height: The height of the figure.
Returns:
A plotly.graph_objects.Figure including all traces
Examples:
To visualize the topics per class, simply run:
```python
topics_per_class = topic_model.topics_per_class(docs, classes)
topic_model.visualize_topics_per_class(topics_per_class)
```
Or if you want to save the resulting figure:
```python
fig = topic_model.visualize_topics_per_class(topics_per_class)
fig.write_html("path/to/file.html")
```
"""
check_is_fitted(self)
return plotting.visualize_topics_per_class(
self,
topics_per_class=topics_per_class,
top_n_topics=top_n_topics,
topics=topics,
normalize_frequency=normalize_frequency,
custom_labels=custom_labels,
title=title,
width=width,
height=height,
)
def visualize_distribution(
self,
probabilities: np.ndarray,
min_probability: float = 0.015,
custom_labels: bool = False,
title: str = "<b>Topic Probability Distribution</b>",
width: int = 800,
height: int = 600,
) -> "go.Figure":
"""Visualize the distribution of topic probabilities.
Arguments:
probabilities: An array of probability scores
min_probability: The minimum probability score to visualize.
All others are ignored.
custom_labels: Whether to use custom topic labels that were defined using
`topic_model.set_topic_labels`.
title: Title of the plot.
width: The width of the figure.
height: The height of the figure.
Examples:
Make sure to fit the model before and only input the
probabilities of a single document:
```python
topic_model.visualize_distribution(topic_model.probabilities_[0])
```
Or if you want to save the resulting figure:
```python
fig = topic_model.visualize_distribution(topic_model.probabilities_[0])
fig.write_html("path/to/file.html")
```
"""
check_is_fitted(self)
return plotting.visualize_distribution(
self,
probabilities=probabilities,
min_probability=min_probability,
custom_labels=custom_labels,
title=title,
width=width,
height=height,
)
def visualize_approximate_distribution(
self,
document: str,
topic_token_distribution: np.ndarray,
normalize: bool = False,
):
"""Visualize the topic distribution calculated by `.approximate_topic_distribution`
on a token level. Thereby indicating the extent to which a certain word or phrase belongs
to a specific topic. The assumption here is that a single word can belong to multiple
similar topics and as such can give information about the broader set of topics within
a single document.
Arguments:
topic_model: A fitted BERTopic instance.
document: The document for which you want to visualize
the approximated topic distribution.
topic_token_distribution: The topic-token distribution of the document as
extracted by `.approximate_topic_distribution`
normalize: Whether to normalize, between 0 and 1 (summing up to 1), the
topic distribution values.
Returns:
df: A stylized dataframe indicating the best fitting topics
for each token.
Examples:
```python
# Calculate the topic distributions on a token level
# Note that we need to have `calculate_token_level=True`
topic_distr, topic_token_distr = topic_model.approximate_distribution(
docs, calculate_token_level=True
)
# Visualize the approximated topic distributions
df = topic_model.visualize_approximate_distribution(docs[0], topic_token_distr[0])
df
```
To revert this stylized dataframe back to a regular dataframe,
you can run the following:
```python
df.data.columns = [column.strip() for column in df.data.columns]
df = df.data
```
"""
check_is_fitted(self)
return plotting.visualize_approximate_distribution(
self,
document=document,
topic_token_distribution=topic_token_distribution,
normalize=normalize,
)
def visualize_hierarchy(
self,
orientation: str = "left",
topics: List[int] | None = None,
top_n_topics: int | None = None,
use_ctfidf: bool = True,
custom_labels: bool = False,
title: str = "<b>Hierarchical Clustering</b>",
width: int = 1000,
height: int = 600,
hierarchical_topics: pd.DataFrame = None,
linkage_function: Callable[[csr_matrix], np.ndarray] | None = None,
distance_function: Callable[[csr_matrix], csr_matrix] | None = None,
color_threshold: int = 1,
) -> "go.Figure":
"""Visualize a hierarchical structure of the topics.
A ward linkage function is used to perform the
hierarchical clustering based on the cosine distance
matrix between c-TF-IDF or semantic embeddings of the topics.
Arguments:
topic_model: A fitted BERTopic instance.
orientation: The orientation of the figure.
Either 'left' or 'bottom'
topics: A selection of topics to visualize
top_n_topics: Only select the top n most frequent topics
use_ctfidf: Whether to calculate distances between topics based on c-TF-IDF embeddings. If False, the
embeddings from the embedding model are used.
custom_labels: Whether to use custom topic labels that were defined using
`topic_model.set_topic_labels`.
NOTE: Custom labels are only generated for the original
un-merged topics.
title: Title of the plot.
width: The width of the figure. Only works if orientation is set to 'left'
height: The height of the figure. Only works if orientation is set to 'bottom'
hierarchical_topics: A dataframe that contains a hierarchy of topics
represented by their parents and their children.
NOTE: The hierarchical topic names are only visualized
if both `topics` and `top_n_topics` are not set.
linkage_function: The linkage function to use. Default is:
`lambda x: sch.linkage(x, 'ward', optimal_ordering=True)`
NOTE: Make sure to use the same `linkage_function` as used
in `topic_model.hierarchical_topics`.
distance_function: The distance function to use on the c-TF-IDF matrix. Default is:
`lambda x: 1 - cosine_similarity(x)`
NOTE: Make sure to use the same `distance_function` as used
in `topic_model.hierarchical_topics`.
color_threshold: Value at which the separation of clusters will be made which
will result in different colors for different clusters.
A higher value will typically lead to less colored clusters.
Returns:
fig: A plotly figure
Examples:
To visualize the hierarchical structure of
topics simply run:
```python
topic_model.visualize_hierarchy()
```
If you also want the labels of hierarchical topics visualized,
run the following:
```python
# Extract hierarchical topics and their representations
hierarchical_topics = topic_model.hierarchical_topics(docs)
# Visualize these representations
topic_model.visualize_hierarchy(hierarchical_topics=hierarchical_topics)
```
If you want to save the resulting figure:
```python
fig = topic_model.visualize_hierarchy()
fig.write_html("path/to/file.html")
```
<iframe src="../getting_started/visualization/hierarchy.html"
style="width:1000px; height: 680px; border: 0px;""></iframe>
"""
check_is_fitted(self)
return plotting.visualize_hierarchy(
self,
orientation=orientation,
topics=topics,
top_n_topics=top_n_topics,
use_ctfidf=use_ctfidf,
custom_labels=custom_labels,
title=title,
width=width,
height=height,
hierarchical_topics=hierarchical_topics,
linkage_function=linkage_function,
distance_function=distance_function,
color_threshold=color_threshold,
)
def visualize_heatmap(
self,
topics: List[int] | None = None,
top_n_topics: int | None = None,
n_clusters: int | None = None,
use_ctfidf: bool = False,
custom_labels: bool = False,
title: str = "<b>Similarity Matrix</b>",
width: int = 800,
height: int = 800,
) -> "go.Figure":
"""Visualize a heatmap of the topic's similarity matrix.
Based on the cosine similarity matrix between c-TF-IDFs or semantic embeddings of the topics,
a heatmap is created showing the similarity between topics.
Arguments:
topics: A selection of topics to visualize.
top_n_topics: Only select the top n most frequent topics.
n_clusters: Create n clusters and order the similarity
matrix by those clusters.
use_ctfidf: Whether to calculate distances between topics based on c-TF-IDF embeddings. If False, the
embeddings from the embedding model are used.
custom_labels: Whether to use custom topic labels that were defined using
`topic_model.set_topic_labels`.
title: Title of the plot.
width: The width of the figure.
height: The height of the figure.
Returns:
fig: A plotly figure
Examples:
To visualize the similarity matrix of
topics simply run:
```python
topic_model.visualize_heatmap()
```
Or if you want to save the resulting figure:
```python
fig = topic_model.visualize_heatmap()
fig.write_html("path/to/file.html")
```
"""
check_is_fitted(self)
return plotting.visualize_heatmap(
self,
topics=topics,
top_n_topics=top_n_topics,
n_clusters=n_clusters,
use_ctfidf=use_ctfidf,
custom_labels=custom_labels,
title=title,
width=width,
height=height,
)
def visualize_barchart(
self,
topics: List[int] | None = None,
top_n_topics: int = 8,
n_words: int = 5,
custom_labels: bool = False,
title: str = "Topic Word Scores",
width: int = 250,
height: int = 250,
autoscale: bool = False,
) -> "go.Figure":
"""Visualize a barchart of selected topics.
Arguments:
topics: A selection of topics to visualize.
top_n_topics: Only select the top n most frequent topics.
n_words: Number of words to show in a topic
custom_labels: Whether to use custom topic labels that were defined using
`topic_model.set_topic_labels`.
title: Title of the plot.
width: The width of each figure.
height: The height of each figure.
autoscale: Whether to automatically calculate the height of the figures to fit the whole bar text
Returns:
fig: A plotly figure
Examples:
To visualize the barchart of selected topics
simply run:
```python
topic_model.visualize_barchart()
```
Or if you want to save the resulting figure:
```python
fig = topic_model.visualize_barchart()
fig.write_html("path/to/file.html")
```
"""
check_is_fitted(self)
return plotting.visualize_barchart(
self,
topics=topics,
top_n_topics=top_n_topics,
n_words=n_words,
custom_labels=custom_labels,
title=title,
width=width,
height=height,
autoscale=autoscale,
)
def save(
self,
path,
serialization: Literal["safetensors", "pickle", "pytorch"] = "pickle",
save_embedding_model: Union[bool, str] = True,
save_ctfidf: bool = False,
):
"""Saves the model to the specified path or folder.
When saving the model, make sure to also keep track of the versions
of dependencies and Python used. Loading and saving the model should
be done using the same dependencies and Python. Moreover, models
saved in one version of BERTopic should not be loaded in other versions.
Arguments:
path: If `serialization` is 'safetensors' or `pytorch`, this is a directory.
If `serialization` is `pickle`, then this is a file.
serialization: If `pickle`, the entire model will be pickled. If `safetensors`
or `pytorch` the model will be saved without the embedding,
dimensionality reduction, and clustering algorithms.
This is a very efficient format and typically advised.
save_embedding_model: If serialization is `pickle`, then you can choose to skip
saving the embedding model. If serialization is `safetensors`
or `pytorch`, this variable can be used as a string pointing
towards a huggingface model.
save_ctfidf: Whether to save c-TF-IDF information if serialization is `safetensors`
or `pytorch`
Examples:
To save the model in an efficient and safe format (safetensors) with c-TF-IDF information:
```python
topic_model.save("model_dir", serialization="safetensors", save_ctfidf=True)
```
If you wish to also add a pointer to the embedding model, which will be downloaded from
HuggingFace upon loading:
```python
embedding_model = "sentence-transformers/all-MiniLM-L6-v2"
topic_model.save("model_dir", serialization="safetensors", save_embedding_model=embedding_model)
```
or if you want save the full model with pickle:
```python
topic_model.save("my_model")
```
NOTE: Pickle can run arbitrary code and is generally considered to be less safe than
safetensors.
"""
if serialization == "pickle":
logger.warning(
"When you use `pickle` to save/load a BERTopic model,"
"please make sure that the environments in which you save"
"and load the model are **exactly** the same. The version of BERTopic,"
"its dependencies, and python need to remain the same."
)
with open(path, "wb") as file:
# This prevents the vectorizer from being too large in size if `min_df` was
# set to a value higher than 1
self.vectorizer_model.stop_words_ = None
if not save_embedding_model:
embedding_model = self.embedding_model
self.embedding_model = None
joblib.dump(self, file)
self.embedding_model = embedding_model
else:
joblib.dump(self, file)
elif serialization == "safetensors" or serialization == "pytorch":
# Directory
save_directory = Path(path)
save_directory.mkdir(exist_ok=True, parents=True)
# Check embedding model
if (
save_embedding_model
and hasattr(self.embedding_model, "_hf_model")
and not isinstance(save_embedding_model, str)
):
save_embedding_model = self.embedding_model._hf_model
elif not save_embedding_model:
logger.warning(
"You are saving a BERTopic model without explicitly defining an embedding model."
"If you are using a sentence-transformers model or a HuggingFace model supported"
"by sentence-transformers, please save the model by using a pointer towards that model."
"For example, `save_embedding_model='sentence-transformers/all-mpnet-base-v2'`"
)
# Minimal
save_utils.save_hf(model=self, save_directory=save_directory, serialization=serialization)
save_utils.save_topics(model=self, path=save_directory / "topics.json")
save_utils.save_images(model=self, path=save_directory / "images")
save_utils.save_config(
model=self,
path=save_directory / "config.json",
embedding_model=save_embedding_model,
)
# Additional
if save_ctfidf:
if self.c_tf_idf_ is None:
logger.warning(
"The c-TF-IDF matrix could not be saved as it was not found. "
"This typically occurs when merging BERTopic models with `BERTopic.merge_models`."
)
else:
save_utils.save_ctfidf(
model=self,
save_directory=save_directory,
serialization=serialization,
)
save_utils.save_ctfidf_config(model=self, path=save_directory / "ctfidf_config.json")
@classmethod
def load(cls, path: str, embedding_model=None):
"""Loads the model from the specified path or directory.
Arguments:
path: Either load a BERTopic model from a file (`.pickle`) or a folder containing
`.safetensors` or `.bin` files.
embedding_model: Additionally load in an embedding model if it was not saved
in the BERTopic model file or directory.
Examples:
```python
BERTopic.load("model_dir")
```
or if you did not save the embedding model:
```python
BERTopic.load("model_dir", embedding_model="all-MiniLM-L6-v2")
```
"""
file_or_dir = Path(path)
# Load from Pickle
if file_or_dir.is_file():
with open(file_or_dir, "rb") as file:
if embedding_model:
topic_model = joblib.load(file)
topic_model.embedding_model = select_backend(embedding_model, verbose=topic_model.verbose)
else:
topic_model = joblib.load(file)
return topic_model
# Load from directory or HF
if file_or_dir.is_dir():
topics, params, tensors, ctfidf_tensors, ctfidf_config, images = save_utils.load_local_files(file_or_dir)
elif "/" in str(path):
topics, params, tensors, ctfidf_tensors, ctfidf_config, images = save_utils.load_files_from_hf(path)
else:
raise ValueError("Make sure to either pass a valid directory or HF model.")
topic_model = _create_model_from_files(
topics,
params,
tensors,
ctfidf_tensors,
ctfidf_config,
images,
warn_no_backend=(embedding_model is None),
)
# Replace embedding model if one is specifically chosen
if embedding_model is not None:
topic_model.embedding_model = select_backend(embedding_model, verbose=topic_model.verbose)
return topic_model
@classmethod
def merge_models(cls, models, min_similarity: float = 0.7, embedding_model=None):
"""Merge multiple pre-trained BERTopic models into a single model.
The models are merged as if they were all saved using pytorch or
safetensors, so a minimal version without c-TF-IDF.
To do this, we choose the first model in the list of
models as a baseline. Then, we check each model whether
they contain topics that are not in the baseline.
This check is based on the cosine similarity between
topics embeddings. If topic embeddings between two models
are similar, then the topic of the second model is re-assigned
to the first. If they are dissimilar, the topic of the second
model is assigned to the first.
In essence, we simply check whether sufficiently "new"
topics emerge and add them.
Arguments:
models: A list of fitted BERTopic models
min_similarity: The minimum similarity for when topics are merged.
embedding_model: Additionally load in an embedding model if necessary.
Returns:
A new BERTopic model that was created as if you were
loading a model from the HuggingFace Hub without c-TF-IDF
Examples:
```python
from bertopic import BERTopic
from sklearn.datasets import fetch_20newsgroups
docs = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes'))['data']
# Create three separate models
topic_model_1 = BERTopic(min_topic_size=5).fit(docs[:4000])
topic_model_2 = BERTopic(min_topic_size=5).fit(docs[4000:8000])
topic_model_3 = BERTopic(min_topic_size=5).fit(docs[8000:])
# Combine all models into one
merged_model = BERTopic.merge_models([topic_model_1, topic_model_2, topic_model_3])
```
"""
def choose_backend():
"""Choose the backend to use for saving the model."""
try:
import torch # noqa: F401
return "pytorch"
except (ModuleNotFoundError, ImportError):
try:
import safetensors # noqa: F401
return "safetensors"
except (ModuleNotFoundError, ImportError):
raise ImportError(
"Neither pytorch nor safetensors is installed. "
"Please install at least one of these packages:\n"
" pip install torch\n"
" pip install safetensors"
)
# Temporarily save model and push to HF
with TemporaryDirectory() as tmpdir:
# Save model weights and config.
all_topics, all_params, all_tensors = [], [], []
for index, model in enumerate(models):
model.save(tmpdir, serialization=choose_backend())
topics, params, tensors, _, _, _ = save_utils.load_local_files(Path(tmpdir))
all_topics.append(topics)
all_params.append(params)
all_tensors.append(np.array(tensors["topic_embeddings"]))
# Create a base set of parameters
if index == 0:
merged_topics = topics
merged_params = params
merged_tensors = np.array(tensors["topic_embeddings"])
merged_topics["custom_labels"] = None
for tensors, selected_topics in zip(all_tensors[1:], all_topics[1:]):
# Calculate similarity matrix
sim_matrix = cosine_similarity(tensors, merged_tensors)
sims = np.max(sim_matrix, axis=1)
# Extract new topics
new_topics = sorted(
[index - selected_topics["_outliers"] for index, sim in enumerate(sims) if sim < min_similarity]
)
max_topic = max(set(merged_topics["topics"]))
# Merge Topic Representations
new_topics_dict = {}
for new_topic in new_topics:
if new_topic != -1:
max_topic += 1
new_topics_dict[new_topic] = max_topic
merged_topics["topic_representations"][str(max_topic)] = selected_topics["topic_representations"][
str(new_topic)
]
merged_topics["topic_labels"][str(max_topic)] = selected_topics["topic_labels"][str(new_topic)]
# Add new aspects
if selected_topics["topic_aspects"]:
aspects_1 = set(merged_topics["topic_aspects"].keys())
aspects_2 = set(selected_topics["topic_aspects"].keys())
aspects_diff = aspects_2.difference(aspects_1)
if aspects_diff:
for aspect in aspects_diff:
merged_topics["topic_aspects"][aspect] = {}
# If the original model does not have topic aspects but the to be added model does
if not merged_topics.get("topic_aspects"):
merged_topics["topic_aspects"] = selected_topics["topic_aspects"]
# If they both contain topic aspects, add to the existing set of aspects
else:
for aspect, values in selected_topics["topic_aspects"].items():
merged_topics["topic_aspects"][aspect][str(max_topic)] = values[str(new_topic)]
# Add new embeddings
new_tensors = tensors[new_topic + selected_topics["_outliers"]]
merged_tensors = np.vstack([merged_tensors, new_tensors])
# Topic Mapper
merged_topics["topic_mapper"] = TopicMapper(list(range(-1, max_topic + 1, 1))).mappings_
# Find similar topics and re-assign those from the new models
sims_idx = np.argmax(sim_matrix, axis=1)
sims = np.max(sim_matrix, axis=1)
to_merge = {
a - selected_topics["_outliers"]: b - merged_topics["_outliers"]
for a, (b, val) in enumerate(zip(sims_idx, sims))
if val >= min_similarity
}
to_merge.update(new_topics_dict)
to_merge[-1] = -1
topics = [to_merge[topic] for topic in selected_topics["topics"]]
merged_topics["topics"].extend(topics)
merged_topics["topic_sizes"] = dict(Counter(merged_topics["topics"]))
# Create a new model from the merged parameters
merged_tensors = {"topic_embeddings": merged_tensors}
merged_model = _create_model_from_files(
merged_topics,
merged_params,
merged_tensors,
None,
None,
None,
warn_no_backend=False,
)
merged_model.embedding_model = models[0].embedding_model
# Replace embedding model if one is specifically chosen
verbose = any([model.verbose for model in models])
if embedding_model is not None and type(merged_model.embedding_model) is BaseEmbedder:
merged_model.embedding_model = select_backend(embedding_model, verbose=verbose)
return merged_model
def push_to_hf_hub(
self,
repo_id: str,
commit_message: str = "Add BERTopic model",
token: str | None = None,
revision: str | None = None,
private: bool = False,
create_pr: bool = False,
model_card: bool = True,
serialization: str = "safetensors",
save_embedding_model: Union[str, bool] = True,
save_ctfidf: bool = False,
):
"""Push your BERTopic model to a HuggingFace Hub.
Whenever you want to upload files to the Hub, you need to log in to your HuggingFace account:
* Log in to your HuggingFace account with the following command:
```bash
huggingface-cli login
# or using an environment variable
huggingface-cli login --token $HUGGINGFACE_TOKEN
```
* Alternatively, you can programmatically login using login() in a notebook or a script:
```python
from huggingface_hub import login
login()
```
* Or you can give a token with the `token` variable
Arguments:
repo_id: The name of your HuggingFace repository
commit_message: A commit
gitextract_s6becou6/
├── .git-blame-ignore-revs
├── .gitattributes
├── .github/
│ ├── ISSUE_TEMPLATE/
│ │ ├── bug_report.yml
│ │ ├── config.yml
│ │ └── feature_request.yml
│ ├── PULL_REQUEST_TEMPLATE.md
│ └── workflows/
│ └── testing.yml
├── .gitignore
├── .pre-commit-config.yaml
├── CONTRIBUTING.md
├── LICENSE
├── Makefile
├── README.md
├── bertopic/
│ ├── __init__.py
│ ├── _bertopic.py
│ ├── _save_utils.py
│ ├── _utils.py
│ ├── backend/
│ │ ├── __init__.py
│ │ ├── _base.py
│ │ ├── _cohere.py
│ │ ├── _fastembed.py
│ │ ├── _flair.py
│ │ ├── _gensim.py
│ │ ├── _hftransformers.py
│ │ ├── _langchain.py
│ │ ├── _model2vec.py
│ │ ├── _multimodal.py
│ │ ├── _openai.py
│ │ ├── _sentencetransformers.py
│ │ ├── _sklearn.py
│ │ ├── _spacy.py
│ │ ├── _use.py
│ │ ├── _utils.py
│ │ └── _word_doc.py
│ ├── cluster/
│ │ ├── __init__.py
│ │ ├── _base.py
│ │ └── _utils.py
│ ├── dimensionality/
│ │ ├── __init__.py
│ │ └── _base.py
│ ├── plotting/
│ │ ├── __init__.py
│ │ ├── _approximate_distribution.py
│ │ ├── _barchart.py
│ │ ├── _datamap.py
│ │ ├── _distribution.py
│ │ ├── _documents.py
│ │ ├── _heatmap.py
│ │ ├── _hierarchical_documents.py
│ │ ├── _hierarchy.py
│ │ ├── _term_rank.py
│ │ ├── _topics.py
│ │ ├── _topics_over_time.py
│ │ └── _topics_per_class.py
│ ├── representation/
│ │ ├── __init__.py
│ │ ├── _base.py
│ │ ├── _cohere.py
│ │ ├── _keybert.py
│ │ ├── _langchain.py
│ │ ├── _litellm.py
│ │ ├── _llamacpp.py
│ │ ├── _mmr.py
│ │ ├── _openai.py
│ │ ├── _pos.py
│ │ ├── _textgeneration.py
│ │ ├── _utils.py
│ │ ├── _visual.py
│ │ └── _zeroshot.py
│ └── vectorizers/
│ ├── __init__.py
│ ├── _ctfidf.py
│ └── _online_cv.py
├── docs/
│ ├── algorithm/
│ │ └── algorithm.md
│ ├── api/
│ │ ├── backends.md
│ │ ├── bertopic.md
│ │ ├── cluster copy.md
│ │ ├── cluster.md
│ │ ├── ctfidf.md
│ │ ├── dimensionality.md
│ │ ├── plotting/
│ │ │ ├── barchart.md
│ │ │ ├── distribution.md
│ │ │ ├── document_datamap.md
│ │ │ ├── documents.md
│ │ │ ├── dtm.md
│ │ │ ├── heatmap.md
│ │ │ ├── hierarchical_documents.md
│ │ │ ├── hierarchy.md
│ │ │ ├── term.md
│ │ │ ├── topics.md
│ │ │ └── topics_per_class.md
│ │ ├── plotting.md
│ │ ├── representations.md
│ │ └── vectorizers.md
│ ├── changelog.md
│ ├── faq.md
│ ├── getting_started/
│ │ ├── best_practices/
│ │ │ └── best_practices.md
│ │ ├── clustering/
│ │ │ └── clustering.md
│ │ ├── ctfidf/
│ │ │ └── ctfidf.md
│ │ ├── dim_reduction/
│ │ │ └── dim_reduction.md
│ │ ├── distribution/
│ │ │ ├── distribution.md
│ │ │ └── distribution_viz.html
│ │ ├── embeddings/
│ │ │ └── embeddings.md
│ │ ├── guided/
│ │ │ └── guided.md
│ │ ├── hierarchicaltopics/
│ │ │ ├── hierarchical_topics.html
│ │ │ └── hierarchicaltopics.md
│ │ ├── manual/
│ │ │ └── manual.md
│ │ ├── merge/
│ │ │ └── merge.md
│ │ ├── multiaspect/
│ │ │ └── multiaspect.md
│ │ ├── multimodal/
│ │ │ └── multimodal.md
│ │ ├── online/
│ │ │ └── online.md
│ │ ├── outlier_reduction/
│ │ │ ├── fig_base.html
│ │ │ ├── fig_reduced.html
│ │ │ └── outlier_reduction.md
│ │ ├── parameter tuning/
│ │ │ └── parametertuning.md
│ │ ├── quickstart/
│ │ │ ├── quickstart.md
│ │ │ └── viz.html
│ │ ├── representation/
│ │ │ ├── llm.md
│ │ │ └── representation.md
│ │ ├── search/
│ │ │ └── search.md
│ │ ├── seed_words/
│ │ │ └── seed_words.md
│ │ ├── semisupervised/
│ │ │ └── semisupervised.md
│ │ ├── serialization/
│ │ │ └── serialization.md
│ │ ├── supervised/
│ │ │ └── supervised.md
│ │ ├── tips_and_tricks/
│ │ │ └── tips_and_tricks.md
│ │ ├── topicreduction/
│ │ │ └── topicreduction.md
│ │ ├── topicrepresentation/
│ │ │ └── topicrepresentation.md
│ │ ├── topicsovertime/
│ │ │ ├── topicsovertime.md
│ │ │ └── trump.html
│ │ ├── topicsperclass/
│ │ │ ├── topics_per_class.html
│ │ │ └── topicsperclass.md
│ │ ├── vectorizers/
│ │ │ └── vectorizers.md
│ │ ├── visualization/
│ │ │ ├── bar_chart.html
│ │ │ ├── datamapplot.html
│ │ │ ├── documents.html
│ │ │ ├── heatmap.html
│ │ │ ├── hierarchical_documents.html
│ │ │ ├── hierarchical_topics.html
│ │ │ ├── hierarchy.html
│ │ │ ├── probabilities.html
│ │ │ ├── term_rank.html
│ │ │ ├── term_rank_log.html
│ │ │ ├── topics_per_class.html
│ │ │ ├── trump.html
│ │ │ ├── visualization.md
│ │ │ ├── visualize_documents.md
│ │ │ ├── visualize_hierarchy.md
│ │ │ ├── visualize_terms.md
│ │ │ ├── visualize_topics.md
│ │ │ └── viz.html
│ │ └── zeroshot/
│ │ └── zeroshot.md
│ ├── img/
│ │ └── probabilities.html
│ ├── index.md
│ ├── stylesheets/
│ │ └── extra.css
│ └── usecases.md
├── mkdocs.yml
├── pyproject.toml
└── tests/
├── __init__.py
├── conftest.py
├── test_bertopic.py
├── test_other.py
├── test_plotting/
│ ├── __init__.py
│ ├── test_approximate.py
│ ├── test_bar.py
│ ├── test_documents.py
│ ├── test_dynamic.py
│ ├── test_heatmap.py
│ ├── test_term_rank.py
│ └── test_topics.py
├── test_reduction/
│ ├── __init__.py
│ ├── test_delete.py
│ └── test_merge.py
├── test_representation/
│ ├── __init__.py
│ ├── test_get.py
│ ├── test_labels.py
│ └── test_representations.py
├── test_sub_models/
│ ├── __init__.py
│ ├── test_cluster.py
│ ├── test_dim_reduction.py
│ └── test_embeddings.py
├── test_utils.py
├── test_variations/
│ ├── __init__.py
│ ├── test_class.py
│ ├── test_dynamic.py
│ └── test_hierarchy.py
└── test_vectorizers/
├── __init__.py
├── test_ctfidf.py
└── test_online_cv.py
SYMBOL INDEX (329 symbols across 73 files)
FILE: bertopic/_bertopic.py
class BERTopic (line 88) | class BERTopic:
method __init__ (line 145) | def __init__(
method _outliers (line 318) | def _outliers(self):
method topic_labels_ (line 329) | def topic_labels_(self):
method fit (line 350) | def fit(
method fit_transform (line 395) | def fit_transform(
method transform (line 545) | def transform(
method partial_fit (line 649) | def partial_fit(
method topics_over_time (line 797) | def topics_over_time(
method topics_per_class (line 956) | def topics_per_class(
method hierarchical_topics (line 1035) | def hierarchical_topics(
method approximate_distribution (line 1204) | def approximate_distribution(
method find_topics (line 1431) | def find_topics(
method update_topics (line 1488) | def update_topics(
method get_topics (line 1598) | def get_topics(self, full: bool = False) -> Mapping[str, Tuple[str, fl...
method get_topic (line 1622) | def get_topic(self, topic: int, full: bool = False) -> Union[Mapping[s...
method get_topic_info (line 1650) | def get_topic_info(self, topic: int | None = None) -> pd.DataFrame:
method get_topic_freq (line 1703) | def get_topic_freq(self, topic: int | None = None) -> Union[pd.DataFra...
method get_document_info (line 1734) | def get_document_info(
method get_representative_docs (line 1826) | def get_representative_docs(self, topic: int | None = None) -> List[str]:
method get_topic_tree (line 1872) | def get_topic_tree(
method set_topic_labels (line 1979) | def set_topic_labels(self, topic_labels: Union[List[str], Mapping[int,...
method generate_topic_labels (line 2042) | def generate_topic_labels(
method merge_topics (line 2101) | def merge_topics(
method delete_topics (line 2177) | def delete_topics(
method reduce_topics (line 2313) | def reduce_topics(
method reduce_outliers (line 2380) | def reduce_outliers(
method visualize_topics (line 2541) | def visualize_topics(
method visualize_documents (line 2595) | def visualize_documents(
method visualize_document_datamap (line 2694) | def visualize_document_datamap(
method visualize_hierarchical_documents (line 2805) | def visualize_hierarchical_documents(
method visualize_term_rank (line 2925) | def visualize_term_rank(
method visualize_topics_over_time (line 2988) | def visualize_topics_over_time(
method visualize_topics_per_class (line 3044) | def visualize_topics_per_class(
method visualize_distribution (line 3100) | def visualize_distribution(
method visualize_approximate_distribution (line 3147) | def visualize_approximate_distribution(
method visualize_hierarchy (line 3201) | def visualize_hierarchy(
method visualize_heatmap (line 3301) | def visualize_heatmap(
method visualize_barchart (line 3361) | def visualize_barchart(
method save (line 3416) | def save(
method load (line 3534) | def load(cls, path: str, embedding_model=None):
method merge_models (line 3590) | def merge_models(cls, models, min_similarity: float = 0.7, embedding_m...
method push_to_hf_hub (line 3751) | def push_to_hf_hub(
method get_params (line 3821) | def get_params(self, deep: bool = False) -> Mapping[str, Any]:
method _extract_embeddings (line 3844) | def _extract_embeddings(
method _images_to_text (line 3884) | def _images_to_text(self, documents: pd.DataFrame, embeddings: np.ndar...
method _map_predictions (line 3901) | def _map_predictions(self, predictions: List[int]) -> List[int]:
method _reduce_dimensionality (line 3907) | def _reduce_dimensionality(
method _cluster_embeddings (line 3958) | def _cluster_embeddings(
method _zeroshot_topic_modeling (line 4010) | def _zeroshot_topic_modeling(
method _is_zeroshot (line 4068) | def _is_zeroshot(self):
method _combine_zeroshot_topics (line 4078) | def _combine_zeroshot_topics(
method _guided_topic_modeling (line 4137) | def _guided_topic_modeling(self, embeddings: np.ndarray) -> Tuple[List...
method _extract_topics (line 4177) | def _extract_topics(
method _save_representative_docs (line 4217) | def _save_representative_docs(self, documents: pd.DataFrame):
method _extract_representative_docs (line 4235) | def _extract_representative_docs(
method _create_topic_vectors (line 4315) | def _create_topic_vectors(
method _c_tf_idf (line 4398) | def _c_tf_idf(
method _update_topic_size (line 4455) | def _update_topic_size(self, documents: pd.DataFrame):
method _extract_words_per_topic (line 4464) | def _extract_words_per_topic(
method _reduce_topics (line 4565) | def _reduce_topics(self, documents: pd.DataFrame, use_ctfidf: bool = F...
method _reduce_to_n_topics (line 4598) | def _reduce_to_n_topics(self, documents: pd.DataFrame, use_ctfidf: boo...
method _auto_reduce_topics (line 4655) | def _auto_reduce_topics(self, documents: pd.DataFrame, use_ctfidf: boo...
method _sort_mappings_by_frequency (line 4726) | def _sort_mappings_by_frequency(self, documents: pd.DataFrame) -> pd.D...
method _map_probabilities (line 4767) | def _map_probabilities(
method _preprocess_text (line 4804) | def _preprocess_text(self, documents: np.ndarray) -> List[str]:
method _top_n_idx_sparse (line 4819) | def _top_n_idx_sparse(matrix: csr_matrix, n: int) -> np.ndarray:
method _top_n_values_sparse (line 4841) | def _top_n_values_sparse(matrix: csr_matrix, indices: np.ndarray) -> n...
method _get_param_names (line 4858) | def _get_param_names(cls):
method __str__ (line 4870) | def __str__(self):
class TopicMapper (line 4887) | class TopicMapper:
method __init__ (line 4913) | def __init__(self, topics: List[int]):
method get_mappings (line 4923) | def get_mappings(self, original_topics: bool = True) -> Mapping[int, i...
method add_mappings (line 4950) | def add_mappings(self, mappings: Mapping[int, int], topic_model: BERTo...
method add_new_topics (line 5006) | def add_new_topics(self, mappings: Mapping[int, int]):
function _create_model_from_files (line 5018) | def _create_model_from_files(
FILE: bertopic/_save_utils.py
function push_to_hf_hub (line 106) | def push_to_hf_hub(
function load_local_files (line 174) | def load_local_files(path):
function load_files_from_hf (line 224) | def load_files_from_hf(path):
function generate_readme (line 271) | def generate_readme(model, repo_id: str):
function save_hf (line 318) | def save_hf(model, save_directory, serialization: str):
function save_ctfidf (line 331) | def save_ctfidf(model, save_directory: str, serialization: str):
function save_ctfidf_config (line 360) | def save_ctfidf_config(model, path):
function save_config (line 385) | def save_config(model, path: str, embedding_model):
function check_has_visual_aspect (line 401) | def check_has_visual_aspect(model):
function save_images (line 409) | def save_images(model, path: str):
function save_topics (line 424) | def save_topics(model, path: str):
function load_cfg_from_json (line 453) | def load_cfg_from_json(json_file: Union[str, os.PathLike]):
class NumpyEncoder (line 460) | class NumpyEncoder(json.JSONEncoder):
method default (line 461) | def default(self, obj):
function get_package_versions (line 469) | def get_package_versions():
function load_safetensors (line 521) | def load_safetensors(path):
function save_safetensors (line 531) | def save_safetensors(path, tensors):
FILE: bertopic/_utils.py
class MyLogger (line 10) | class MyLogger:
method __init__ (line 11) | def __init__(self):
method configure (line 14) | def configure(self, level):
method info (line 19) | def info(self, message):
method warning (line 22) | def warning(self, message):
method set_level (line 25) | def set_level(self, level):
method _add_handler (line 30) | def _add_handler(self):
function check_documents_type (line 40) | def check_documents_type(documents):
function check_embeddings_shape (line 51) | def check_embeddings_shape(embeddings, docs):
function check_is_fitted (line 65) | def check_is_fitted(topic_model):
class NotInstalled (line 83) | class NotInstalled:
method __init__ (line 88) | def __init__(self, tool, dep, custom_msg=None):
method __getattr__ (line 99) | def __getattr__(self, *args, **kwargs):
method __call__ (line 102) | def __call__(self, *args, **kwargs):
function validate_distance_matrix (line 106) | def validate_distance_matrix(X, n_samples):
function get_unique_distances (line 155) | def get_unique_distances(dists: np.array, noise_max=1e-7) -> np.array:
function select_topic_representation (line 179) | def select_topic_representation(
class MockPlotlyModule (line 231) | class MockPlotlyModule:
method __getattr__ (line 234) | def __getattr__(self, name: str) -> Any:
FILE: bertopic/backend/_base.py
class BaseEmbedder (line 5) | class BaseEmbedder:
method __init__ (line 17) | def __init__(self, embedding_model=None, word_embedding_model=None):
method embed (line 21) | def embed(self, documents: List[str], verbose: bool = False) -> np.nda...
method embed_words (line 35) | def embed_words(self, words: List[str], verbose: bool = False) -> np.n...
method embed_documents (line 50) | def embed_documents(self, document: List[str], verbose: bool = False) ...
FILE: bertopic/backend/_cohere.py
class CohereBackend (line 8) | class CohereBackend(BaseEmbedder):
method __init__ (line 43) | def __init__(
method embed (line 63) | def embed(self, documents: List[str], verbose: bool = False) -> np.nda...
method _chunks (line 92) | def _chunks(self, documents):
FILE: bertopic/backend/_fastembed.py
class FastEmbedBackend (line 8) | class FastEmbedBackend(BaseEmbedder):
method __init__ (line 27) | def __init__(self, embedding_model: str = "BAAI/bge-small-en-v1.5"):
method embed (line 41) | def embed(self, documents: List[str], verbose: bool = False) -> np.nda...
FILE: bertopic/backend/_flair.py
class FlairBackend (line 10) | class FlairBackend(BaseEmbedder):
method __init__ (line 33) | def __init__(self, embedding_model: Union[TokenEmbeddings, DocumentEmb...
method embed (line 55) | def embed(self, documents: List[str], verbose: bool = False) -> np.nda...
FILE: bertopic/backend/_gensim.py
class GensimBackend (line 8) | class GensimBackend(BaseEmbedder):
method __init__ (line 27) | def __init__(self, embedding_model: Word2VecKeyedVectors):
method embed (line 39) | def embed(self, documents: List[str], verbose: bool = False) -> np.nda...
FILE: bertopic/backend/_hftransformers.py
class HFTransformerBackend (line 12) | class HFTransformerBackend(BaseEmbedder):
method __init__ (line 34) | def __init__(self, embedding_model: Pipeline):
method embed (line 45) | def embed(self, documents: List[str], verbose: bool = False) -> np.nda...
method _embed (line 69) | def _embed(self, document: str, features: np.ndarray) -> np.ndarray:
class MyDataset (line 94) | class MyDataset(Dataset):
method __init__ (line 97) | def __init__(self, docs):
method __len__ (line 100) | def __len__(self):
method __getitem__ (line 103) | def __getitem__(self, idx):
FILE: bertopic/backend/_langchain.py
class LangChainBackend (line 8) | class LangChainBackend(BaseEmbedder):
method __init__ (line 25) | def __init__(self, embedding_model: Embeddings):
method embed (line 28) | def embed(self, documents: List[str], verbose: bool = False) -> np.nda...
FILE: bertopic/backend/_model2vec.py
class Model2VecBackend (line 9) | class Model2VecBackend(BaseEmbedder):
method __init__ (line 54) | def __init__(
method embed (line 90) | def embed(self, documents: List[str], verbose: bool = False) -> np.nda...
method _check_model2vec_installation (line 125) | def _check_model2vec_installation(self):
FILE: bertopic/backend/_multimodal.py
class MultiModalBackend (line 10) | class MultiModalBackend(BaseEmbedder):
method __init__ (line 45) | def __init__(
method embed (line 87) | def embed(self, documents: List[str], images: List[str] | None = None,...
method embed_documents (line 125) | def embed_documents(self, documents: List[str], verbose: bool = False)...
method embed_words (line 141) | def embed_words(self, words: List[str], verbose: bool = False) -> np.n...
method embed_images (line 156) | def embed_images(self, images, verbose):
method _truncate_document (line 188) | def _truncate_document(self, document):
FILE: bertopic/backend/_openai.py
class OpenAIBackend (line 9) | class OpenAIBackend(BaseEmbedder):
method __init__ (line 34) | def __init__(
method embed (line 54) | def embed(self, documents: List[str], verbose: bool = False) -> np.nda...
method _chunks (line 86) | def _chunks(self, documents):
FILE: bertopic/backend/_sentencetransformers.py
class SentenceTransformerBackend (line 9) | class SentenceTransformerBackend(BaseEmbedder):
method __init__ (line 53) | def __init__(self, embedding_model: Union[str, SentenceTransformer], m...
method embed (line 72) | def embed(self, documents: List[str], verbose: bool = False) -> np.nda...
FILE: bertopic/backend/_sklearn.py
class SklearnEmbedder (line 5) | class SklearnEmbedder(BaseEmbedder):
method __init__ (line 46) | def __init__(self, pipe):
method embed (line 50) | def embed(self, documents, verbose=False):
FILE: bertopic/backend/_spacy.py
class SpacyBackend (line 7) | class SpacyBackend(BaseEmbedder):
method __init__ (line 53) | def __init__(self, embedding_model):
method embed (line 64) | def embed(self, documents: List[str], verbose: bool = False) -> np.nda...
FILE: bertopic/backend/_use.py
class USEBackend (line 8) | class USEBackend(BaseEmbedder):
method __init__ (line 27) | def __init__(self, embedding_model):
method embed (line 40) | def embed(self, documents: List[str], verbose: bool = False) -> np.nda...
FILE: bertopic/backend/_utils.py
function select_backend (line 71) | def select_backend(embedding_model, language: str | None = None, verbose...
FILE: bertopic/backend/_word_doc.py
class WordDocEmbedder (line 7) | class WordDocEmbedder(BaseEmbedder):
method __init__ (line 10) | def __init__(self, embedding_model, word_embedding_model):
method embed_words (line 16) | def embed_words(self, words: List[str], verbose: bool = False) -> np.n...
method embed_documents (line 31) | def embed_documents(self, document: List[str], verbose: bool = False) ...
FILE: bertopic/cluster/_base.py
class BaseCluster (line 4) | class BaseCluster:
method fit (line 33) | def fit(self, X, y=None):
method transform (line 40) | def transform(self, X: np.ndarray) -> np.ndarray:
FILE: bertopic/cluster/_utils.py
function hdbscan_delegator (line 4) | def hdbscan_delegator(model, func: str, embeddings: np.ndarray = None):
function is_supported_hdbscan (line 67) | def is_supported_hdbscan(model):
FILE: bertopic/dimensionality/_base.py
class BaseDimensionalityReduction (line 4) | class BaseDimensionalityReduction:
method fit (line 22) | def fit(self, X: np.ndarray = None, y: np.ndarray = None):
method transform (line 25) | def transform(self, X: np.ndarray) -> np.ndarray:
FILE: bertopic/plotting/_approximate_distribution.py
function visualize_approximate_distribution (line 12) | def visualize_approximate_distribution(
FILE: bertopic/plotting/_barchart.py
function visualize_barchart (line 9) | def visualize_barchart(
FILE: bertopic/plotting/_datamap.py
class Figure (line 13) | class Figure(object):
function visualize_document_datamap (line 17) | def visualize_document_datamap(
FILE: bertopic/plotting/_distribution.py
function visualize_distribution (line 6) | def visualize_distribution(
FILE: bertopic/plotting/_documents.py
function visualize_documents (line 8) | def visualize_documents(
FILE: bertopic/plotting/_heatmap.py
function visualize_heatmap (line 11) | def visualize_heatmap(
FILE: bertopic/plotting/_hierarchical_documents.py
function visualize_hierarchical_documents (line 9) | def visualize_hierarchical_documents(
FILE: bertopic/plotting/_hierarchy.py
function visualize_hierarchy (line 16) | def visualize_hierarchy(
function _get_annotations (line 230) | def _get_annotations(
FILE: bertopic/plotting/_term_rank.py
function visualize_term_rank (line 6) | def visualize_term_rank(
FILE: bertopic/plotting/_topics.py
function visualize_topics (line 18) | def visualize_topics(
function _plotly_topic_visualization (line 123) | def _plotly_topic_visualization(df: pd.DataFrame, topic_list: List[str],...
FILE: bertopic/plotting/_topics_over_time.py
function visualize_topics_over_time (line 7) | def visualize_topics_over_time(
FILE: bertopic/plotting/_topics_per_class.py
function visualize_topics_per_class (line 7) | def visualize_topics_per_class(
FILE: bertopic/representation/_base.py
class BaseRepresentation (line 7) | class BaseRepresentation(BaseEstimator):
method extract_topics (line 10) | def extract_topics(
FILE: bertopic/representation/_cohere.py
class Cohere (line 41) | class Cohere(BaseRepresentation):
method __init__ (line 111) | def __init__(
method extract_topics (line 138) | def extract_topics(
method _create_prompt (line 184) | def _create_prompt(self, docs, topic, topics):
method _replace_documents (line 204) | def _replace_documents(prompt, docs):
FILE: bertopic/representation/_keybert.py
class KeyBERTInspired (line 12) | class KeyBERTInspired(BaseRepresentation):
method __init__ (line 13) | def __init__(
method extract_topics (line 68) | def extract_topics(
method _extract_candidate_words (line 112) | def _extract_candidate_words(
method _extract_embeddings (line 156) | def _extract_embeddings(
method _extract_top_words (line 194) | def _extract_top_words(
FILE: bertopic/representation/_langchain.py
class LangChain (line 12) | class LangChain(BaseRepresentation):
method __init__ (line 133) | def __init__(
method extract_topics (line 153) | def extract_topics(
FILE: bertopic/representation/_litellm.py
class LiteLLM (line 20) | class LiteLLM(BaseRepresentation):
method __init__ (line 79) | def __init__(
method extract_topics (line 103) | def extract_topics(
method _create_prompt (line 146) | def _create_prompt(self, docs, topic, topics):
method _replace_documents (line 166) | def _replace_documents(prompt, docs):
function chat_completions_with_backoff (line 174) | def chat_completions_with_backoff(**kwargs):
FILE: bertopic/representation/_llamacpp.py
class LlamaCPP (line 41) | class LlamaCPP(BaseRepresentation):
method __init__ (line 115) | def __init__(
method extract_topics (line 149) | def extract_topics(
method _create_prompt (line 190) | def _create_prompt(self, docs, topic, topics):
method _replace_documents (line 210) | def _replace_documents(prompt, docs):
FILE: bertopic/representation/_mmr.py
class MaximalMarginalRelevance (line 10) | class MaximalMarginalRelevance(BaseRepresentation):
method __init__ (line 39) | def __init__(self, diversity: float = 0.1, top_n_words: int = 10):
method extract_topics (line 43) | def extract_topics(
function mmr (line 86) | def mmr(
FILE: bertopic/representation/_openai.py
class OpenAI (line 50) | class OpenAI(BaseRepresentation):
method __init__ (line 133) | def __init__(
method extract_topics (line 182) | def extract_topics(
method _create_prompt (line 242) | def _create_prompt(self, docs, topic, topics):
method _replace_documents (line 262) | def _replace_documents(prompt, docs):
function chat_completions_with_backoff (line 270) | def chat_completions_with_backoff(client, **kwargs):
FILE: bertopic/representation/_pos.py
class PartOfSpeech (line 15) | class PartOfSpeech(BaseRepresentation):
method __init__ (line 66) | def __init__(
method extract_topics (line 94) | def extract_topics(
FILE: bertopic/representation/_textgeneration.py
class TextGeneration (line 17) | class TextGeneration(BaseRepresentation):
method __init__ (line 85) | def __init__(
method extract_topics (line 119) | def extract_topics(
method _create_prompt (line 169) | def _create_prompt(self, docs, topic, topics):
FILE: bertopic/representation/_utils.py
function truncate_document (line 6) | def truncate_document(topic_model, doc_length: Union[int, None], tokeniz...
function validate_truncate_document_parameters (line 62) | def validate_truncate_document_parameters(tokenizer, doc_length) -> Unio...
function retry_with_exponential_backoff (line 74) | def retry_with_exponential_backoff(
FILE: bertopic/representation/_visual.py
class VisualRepresentation (line 14) | class VisualRepresentation(BaseRepresentation):
method __init__ (line 48) | def __init__(
method extract_topics (line 76) | def extract_topics(
method _convert_image_to_text (line 131) | def _convert_image_to_text(self, images: List[str], verbose: bool = Fa...
method image_to_text (line 156) | def image_to_text(self, documents: pd.DataFrame, embeddings: np.ndarra...
method _chunks (line 204) | def _chunks(self, images):
function get_concat_h_multi_resize (line 209) | def get_concat_h_multi_resize(im_list):
function get_concat_v_multi_resize (line 227) | def get_concat_v_multi_resize(im_list):
function get_concat_tile_resize (line 241) | def get_concat_tile_resize(im_list_2d, image_height=600, image_squares=F...
FILE: bertopic/representation/_zeroshot.py
class ZeroShotClassification (line 9) | class ZeroShotClassification(BaseRepresentation):
method __init__ (line 38) | def __init__(
method extract_topics (line 59) | def extract_topics(
FILE: bertopic/vectorizers/_ctfidf.py
class ClassTfidfTransformer (line 9) | class ClassTfidfTransformer(TfidfTransformer):
method __init__ (line 41) | def __init__(
method fit (line 54) | def fit(self, X: sp.csr_matrix, multiplier: np.ndarray = None):
method transform (line 98) | def transform(self, X: sp.csr_matrix):
FILE: bertopic/vectorizers/_online_cv.py
class OnlineCountVectorizer (line 11) | class OnlineCountVectorizer(CountVectorizer):
method __init__ (line 71) | def __init__(self, decay: float | None = None, delete_min_df: float | ...
method partial_fit (line 76) | def partial_fit(self, raw_documents: List[str]) -> None:
method update_bow (line 102) | def update_bow(self, raw_documents: List[str]) -> csr_matrix:
method _clean_bow (line 144) | def _clean_bow(self) -> None:
FILE: tests/conftest.py
function embedding_model (line 17) | def embedding_model():
function document_embeddings (line 23) | def document_embeddings(documents, embedding_model):
function reduced_embeddings (line 29) | def reduced_embeddings(document_embeddings):
function documents (line 37) | def documents():
function targets (line 43) | def targets():
function base_topic_model (line 50) | def base_topic_model(documents, document_embeddings, embedding_model):
function zeroshot_topic_model (line 59) | def zeroshot_topic_model(documents, document_embeddings, embedding_model):
function custom_topic_model (line 74) | def custom_topic_model(documents, document_embeddings, embedding_model):
function representation_topic_model (line 92) | def representation_topic_model(documents, document_embeddings, embedding...
function reduced_topic_model (line 115) | def reduced_topic_model(custom_topic_model, documents):
function merged_topic_model (line 122) | def merged_topic_model(custom_topic_model, documents):
function kmeans_pca_topic_model (line 136) | def kmeans_pca_topic_model(documents, document_embeddings):
function supervised_topic_model (line 148) | def supervised_topic_model(documents, document_embeddings, embedding_mod...
function online_topic_model (line 161) | def online_topic_model(documents, document_embeddings, embedding_model):
function cuml_base_topic_model (line 181) | def cuml_base_topic_model(documents, document_embeddings, embedding_model):
FILE: tests/test_bertopic.py
function cuml_available (line 7) | def cuml_available():
function test_full_model (line 32) | def test_full_model(model, documents, request):
FILE: tests/test_other.py
function test_load_save_model (line 10) | def test_load_save_model():
function test_get_params (line 20) | def test_get_params():
function test_no_plotly (line 31) | def test_no_plotly():
FILE: tests/test_plotting/test_approximate.py
function test_approximate_distribution (line 17) | def test_approximate_distribution(batch_size, padding, model, documents,...
FILE: tests/test_plotting/test_bar.py
function test_barchart (line 16) | def test_barchart(model, request):
function test_barchart_outlier (line 42) | def test_barchart_outlier(model, request):
FILE: tests/test_plotting/test_documents.py
function test_documents (line 15) | def test_documents(model, reduced_embeddings, documents, request):
FILE: tests/test_plotting/test_dynamic.py
function test_dynamic (line 16) | def test_dynamic(model, documents, request):
FILE: tests/test_plotting/test_heatmap.py
function test_heatmap (line 15) | def test_heatmap(model, request):
FILE: tests/test_plotting/test_term_rank.py
function test_term_rank (line 6) | def test_term_rank(model, request):
FILE: tests/test_plotting/test_topics.py
function test_topics (line 16) | def test_topics(model, request):
function test_topics_outlier (line 40) | def test_topics_outlier(model, request):
FILE: tests/test_reduction/test_delete.py
function test_delete (line 16) | def test_delete(model, request):
FILE: tests/test_reduction/test_merge.py
function test_merge (line 16) | def test_merge(model, documents, request):
FILE: tests/test_representation/test_get.py
function test_get_topic (line 18) | def test_get_topic(model, request):
function test_get_topics (line 41) | def test_get_topics(model, request):
function test_get_topic_freq (line 60) | def test_get_topic_freq(model, request):
function test_get_representative_docs (line 85) | def test_get_representative_docs(model, request):
function test_get_topic_info (line 114) | def test_get_topic_info(model, request):
FILE: tests/test_representation/test_labels.py
function test_generate_topic_labels (line 16) | def test_generate_topic_labels(model, request):
function test_set_labels (line 40) | def test_set_labels(model, request):
FILE: tests/test_representation/test_representations.py
function test_update_topics (line 18) | def test_update_topics(model, documents, request):
function test_extract_topics (line 51) | def test_extract_topics(model, documents, request):
function test_extract_topics_custom_cv (line 84) | def test_extract_topics_custom_cv(model, documents, request):
function test_topic_reduction (line 121) | def test_topic_reduction(model, reduced_topics, documents, request):
function test_topic_reduction_edge_cases (line 149) | def test_topic_reduction_edge_cases(model, documents, request):
function test_find_topics (line 179) | def test_find_topics(model, request):
FILE: tests/test_sub_models/test_cluster.py
function test_hdbscan_cluster_embeddings (line 23) | def test_hdbscan_cluster_embeddings(cluster_model, samples, features, ce...
function test_custom_hdbscan_cluster_embeddings (line 58) | def test_custom_hdbscan_cluster_embeddings(cluster_model, samples, featu...
FILE: tests/test_sub_models/test_dim_reduction.py
function test_reduce_dimensionality (line 19) | def test_reduce_dimensionality(dim_model, embeddings, shape, n_components):
function test_custom_reduce_dimensionality (line 36) | def test_custom_reduce_dimensionality(model, request):
FILE: tests/test_sub_models/test_embeddings.py
function test_extract_embeddings (line 19) | def test_extract_embeddings(model, request):
function test_extract_embeddings_compare (line 50) | def test_extract_embeddings_compare(model, embedding_model, request):
function test_extract_incorrect_embeddings (line 62) | def test_extract_incorrect_embeddings():
FILE: tests/test_utils.py
function test_logger (line 15) | def test_logger():
function test_check_documents_type (line 31) | def test_check_documents_type(docs):
function test_check_embeddings_shape (line 36) | def test_check_embeddings_shape():
function test_make_unique_distances (line 42) | def test_make_unique_distances():
function test_select_topic_representation (line 57) | def test_select_topic_representation():
FILE: tests/test_variations/test_class.py
function test_class (line 19) | def test_class(model, documents, request):
FILE: tests/test_variations/test_dynamic.py
function test_dynamic (line 15) | def test_dynamic(model, documents, request):
FILE: tests/test_variations/test_hierarchy.py
function test_hierarchy (line 16) | def test_hierarchy(model, documents, request):
function test_linkage (line 36) | def test_linkage(model, documents, request):
function test_tree (line 59) | def test_tree(model, documents, request):
FILE: tests/test_vectorizers/test_ctfidf.py
function test_ctfidf (line 23) | def test_ctfidf(model, documents, request):
function test_ctfidf_custom_cv (line 68) | def test_ctfidf_custom_cv(model, documents, request):
FILE: tests/test_vectorizers/test_online_cv.py
function test_online_cv (line 16) | def test_online_cv(model, documents, request):
function test_clean_bow (line 30) | def test_clean_bow(model, request):
Copy disabled (too large)
Download .json
Condensed preview — 184 files, each showing path, character count, and a content snippet. Download the .json file for the full structured content (13,367K chars).
[
{
"path": ".git-blame-ignore-revs",
"chars": 211,
"preview": "# Initial Ruff Linting\n70d7725e5c89bccfe7d4e5a3ccd87e05c642d74b\n# Change line-length and ruff format\n39bbfdb8298b5faa32e"
},
{
"path": ".gitattributes",
"chars": 31,
"preview": "*.ipynb linguist-documentation\n"
},
{
"path": ".github/ISSUE_TEMPLATE/bug_report.yml",
"chars": 1406,
"preview": "name: \"\\U0001F41B Bug Report\"\ndescription: Report your bug here.\nlabels: [\"bug\"]\nbody:\n\n - type: markdown\n attribute"
},
{
"path": ".github/ISSUE_TEMPLATE/config.yml",
"chars": 316,
"preview": "blank_issues_enabled: true\ncontact_links:\n - name: 💡 General questions\n url: https://github.com/MaartenGr/BERTopic/d"
},
{
"path": ".github/ISSUE_TEMPLATE/feature_request.yml",
"chars": 994,
"preview": "name: \"\\U0001F680 Feature request\"\ndescription: Submit a proposal/request for a new BERTopic feature\nlabels: [\"Feature r"
},
{
"path": ".github/PULL_REQUEST_TEMPLATE.md",
"chars": 711,
"preview": "# What does this PR do?\n\n<!--\nThank you for considering creating a PR! Before you do, make sure to read through [contrib"
},
{
"path": ".github/workflows/testing.yml",
"chars": 1096,
"preview": "name: Code Checks\n\non:\n push:\n branches:\n - master\n - dev\n pull_request:\n branches:\n - master\n - dev"
},
{
"path": ".gitignore",
"chars": 1006,
"preview": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packagi"
},
{
"path": ".pre-commit-config.yaml",
"chars": 539,
"preview": "repos:\n- repo: https://github.com/pre-commit/pre-commit-hooks\n rev: v6.0.0\n hooks:\n - id: trailing-whitespa"
},
{
"path": "CONTRIBUTING.md",
"chars": 4420,
"preview": "# Contributing to BERTopic\n\nHi! Thank you for considering contributing to BERTopic. With the modular nature of BERTopic,"
},
{
"path": "LICENSE",
"chars": 1081,
"preview": "MIT License\n\nCopyright (c) 2024, Maarten P. Grootendorst\n\nPermission is hereby granted, free of charge, to any person ob"
},
{
"path": "Makefile",
"chars": 548,
"preview": "test:\n\tpytest\n\ntest-no-plotly:\n\tuv sync --extra test\n\tuv pip uninstall plotly\n\tpytest tests/test_other.py -k plotly --pd"
},
{
"path": "README.md",
"chars": 19496,
"preview": "[](https://pepy.tech/projects/bertopic)\n[![PyPI - Python](http"
},
{
"path": "bertopic/__init__.py",
"chars": 146,
"preview": "from importlib.metadata import version\n\nfrom bertopic._bertopic import BERTopic\n\n__version__ = version(\"bertopic\")\n\n__al"
},
{
"path": "bertopic/_bertopic.py",
"chars": 226883,
"preview": "# ruff: noqa: E402\nimport yaml\nimport warnings\n\nwarnings.filterwarnings(\"ignore\", category=FutureWarning)\nwarnings.filte"
},
{
"path": "bertopic/_save_utils.py",
"chars": 17569,
"preview": "import os\nimport json\nimport numpy as np\n\nfrom pathlib import Path\nfrom tempfile import TemporaryDirectory\n\n\n# HuggingFa"
},
{
"path": "bertopic/_utils.py",
"chars": 8813,
"preview": "import numpy as np\nimport pandas as pd\nimport logging\nfrom collections.abc import Iterable\nfrom scipy.sparse import csr_"
},
{
"path": "bertopic/backend/__init__.py",
"chars": 1745,
"preview": "from ._base import BaseEmbedder\nfrom ._word_doc import WordDocEmbedder\nfrom ._utils import languages\nfrom bertopic._util"
},
{
"path": "bertopic/backend/_base.py",
"chars": 2239,
"preview": "import numpy as np\nfrom typing import List\n\n\nclass BaseEmbedder:\n \"\"\"The Base Embedder used for creating embedding mo"
},
{
"path": "bertopic/backend/_cohere.py",
"chars": 3152,
"preview": "import time\nimport numpy as np\nfrom tqdm import tqdm\nfrom typing import Any, List, Mapping\nfrom bertopic.backend import "
},
{
"path": "bertopic/backend/_fastembed.py",
"chars": 1883,
"preview": "import numpy as np\nfrom typing import List\nfrom fastembed import TextEmbedding\n\nfrom bertopic.backend import BaseEmbedde"
},
{
"path": "bertopic/backend/_flair.py",
"chars": 2931,
"preview": "import numpy as np\nfrom tqdm import tqdm\nfrom typing import Union, List\nfrom flair.data import Sentence\nfrom flair.embed"
},
{
"path": "bertopic/backend/_gensim.py",
"chars": 2313,
"preview": "import numpy as np\nfrom tqdm import tqdm\nfrom typing import List\nfrom bertopic.backend import BaseEmbedder\nfrom gensim.m"
},
{
"path": "bertopic/backend/_hftransformers.py",
"chars": 3517,
"preview": "import numpy as np\n\nfrom tqdm import tqdm\nfrom typing import List\nfrom torch.utils.data import Dataset\nfrom sklearn.prep"
},
{
"path": "bertopic/backend/_langchain.py",
"chars": 1485,
"preview": "from typing import List\n\nimport numpy as np\nfrom bertopic.backend import BaseEmbedder\nfrom langchain_core.embeddings imp"
},
{
"path": "bertopic/backend/_model2vec.py",
"chars": 5200,
"preview": "import numpy as np\nfrom typing import List, Union\nfrom model2vec import StaticModel\nfrom sklearn.feature_extraction.text"
},
{
"path": "bertopic/backend/_multimodal.py",
"chars": 7760,
"preview": "import numpy as np\nfrom PIL import Image\nfrom tqdm import tqdm\nfrom typing import List, Union\nfrom sentence_transformers"
},
{
"path": "bertopic/backend/_openai.py",
"chars": 3319,
"preview": "import time\nimport openai\nimport numpy as np\nfrom tqdm import tqdm\nfrom typing import List, Mapping, Any\nfrom bertopic.b"
},
{
"path": "bertopic/backend/_sentencetransformers.py",
"chars": 3265,
"preview": "import numpy as np\nfrom typing import List, Union\nfrom sentence_transformers import SentenceTransformer\nfrom sentence_tr"
},
{
"path": "bertopic/backend/_sklearn.py",
"chars": 2479,
"preview": "from bertopic.backend import BaseEmbedder\nfrom sklearn.utils.validation import check_is_fitted, NotFittedError\n\n\nclass S"
},
{
"path": "bertopic/backend/_spacy.py",
"chars": 3053,
"preview": "import numpy as np\nfrom tqdm import tqdm\nfrom typing import List\nfrom bertopic.backend import BaseEmbedder\n\n\nclass Spacy"
},
{
"path": "bertopic/backend/_use.py",
"chars": 1697,
"preview": "import numpy as np\nfrom tqdm import tqdm\nfrom typing import List\n\nfrom bertopic.backend import BaseEmbedder\n\n\nclass USEB"
},
{
"path": "bertopic/backend/_utils.py",
"chars": 5440,
"preview": "from ._base import BaseEmbedder\n\n# Imports for light-weight variant of BERTopic\nfrom bertopic.backend._sklearn import Sk"
},
{
"path": "bertopic/backend/_word_doc.py",
"chars": 1497,
"preview": "import numpy as np\nfrom typing import List\nfrom bertopic.backend._base import BaseEmbedder\nfrom bertopic.backend._utils "
},
{
"path": "bertopic/cluster/__init__.py",
"chars": 65,
"preview": "from ._base import BaseCluster\n\n__all__ = [\n \"BaseCluster\",\n]\n"
},
{
"path": "bertopic/cluster/_base.py",
"chars": 1032,
"preview": "import numpy as np\n\n\nclass BaseCluster:\n \"\"\"The Base Cluster class.\n\n Using this class directly in BERTopic will m"
},
{
"path": "bertopic/cluster/_utils.py",
"chars": 2771,
"preview": "import numpy as np\n\n\ndef hdbscan_delegator(model, func: str, embeddings: np.ndarray = None):\n \"\"\"Function used to sel"
},
{
"path": "bertopic/dimensionality/__init__.py",
"chars": 97,
"preview": "from ._base import BaseDimensionalityReduction\n\n__all__ = [\n \"BaseDimensionalityReduction\",\n]\n"
},
{
"path": "bertopic/dimensionality/_base.py",
"chars": 667,
"preview": "import numpy as np\n\n\nclass BaseDimensionalityReduction:\n \"\"\"The Base Dimensionality Reduction class.\n\n You can use"
},
{
"path": "bertopic/plotting/__init__.py",
"chars": 997,
"preview": "from ._topics import visualize_topics\nfrom ._heatmap import visualize_heatmap\nfrom ._barchart import visualize_barchart\n"
},
{
"path": "bertopic/plotting/_approximate_distribution.py",
"chars": 3498,
"preview": "import numpy as np\nimport pandas as pd\n\ntry:\n from pandas.io.formats.style import Styler # noqa: F401\n\n HAS_JINJA"
},
{
"path": "bertopic/plotting/_barchart.py",
"chars": 4388,
"preview": "import itertools\nimport numpy as np\nfrom typing import List, Union\n\nimport plotly.graph_objects as go\nfrom plotly.subplo"
},
{
"path": "bertopic/plotting/_datamap.py",
"chars": 7577,
"preview": "import numpy as np\nimport pandas as pd\nfrom typing import List, Union\nfrom warnings import warn\n\ntry:\n import datamap"
},
{
"path": "bertopic/plotting/_distribution.py",
"chars": 3883,
"preview": "import numpy as np\nfrom typing import Union\nimport plotly.graph_objects as go\n\n\ndef visualize_distribution(\n topic_mo"
},
{
"path": "bertopic/plotting/_documents.py",
"chars": 9570,
"preview": "import numpy as np\nimport pandas as pd\nimport plotly.graph_objects as go\n\nfrom typing import List, Union\n\n\ndef visualize"
},
{
"path": "bertopic/plotting/_heatmap.py",
"chars": 5137,
"preview": "import numpy as np\nfrom typing import List, Union\nfrom scipy.cluster.hierarchy import fcluster, linkage\nfrom sklearn.met"
},
{
"path": "bertopic/plotting/_hierarchical_documents.py",
"chars": 15586,
"preview": "import numpy as np\nimport pandas as pd\nimport plotly.graph_objects as go\nimport math\n\nfrom typing import List, Union\n\n\nd"
},
{
"path": "bertopic/plotting/_hierarchy.py",
"chars": 14252,
"preview": "import numpy as np\nimport pandas as pd\nfrom typing import Callable, List, Union\nfrom scipy.sparse import csr_matrix\nfrom"
},
{
"path": "bertopic/plotting/_term_rank.py",
"chars": 4604,
"preview": "import numpy as np\nfrom typing import List, Union\nimport plotly.graph_objects as go\n\n\ndef visualize_term_rank(\n topic"
},
{
"path": "bertopic/plotting/_topics.py",
"chars": 7193,
"preview": "import numpy as np\nimport pandas as pd\n\ntry:\n from umap import UMAP\n\n HAS_UMAP = True\nexcept (ImportError, ModuleN"
},
{
"path": "bertopic/plotting/_topics_over_time.py",
"chars": 4962,
"preview": "import pandas as pd\nfrom typing import List, Union\nimport plotly.graph_objects as go\nfrom sklearn.preprocessing import n"
},
{
"path": "bertopic/plotting/_topics_per_class.py",
"chars": 5078,
"preview": "import pandas as pd\nfrom typing import List, Union\nimport plotly.graph_objects as go\nfrom sklearn.preprocessing import n"
},
{
"path": "bertopic/representation/__init__.py",
"chars": 2424,
"preview": "from bertopic._utils import NotInstalled\nfrom bertopic.representation._cohere import Cohere\nfrom bertopic.representation"
},
{
"path": "bertopic/representation/_base.py",
"chars": 1737,
"preview": "import pandas as pd\nfrom scipy.sparse import csr_matrix\nfrom sklearn.base import BaseEstimator\nfrom typing import Mappin"
},
{
"path": "bertopic/representation/_cohere.py",
"chars": 8570,
"preview": "import time\nimport pandas as pd\nfrom tqdm import tqdm\nfrom scipy.sparse import csr_matrix\nfrom typing import Mapping, Li"
},
{
"path": "bertopic/representation/_keybert.py",
"chars": 9551,
"preview": "import numpy as np\nimport pandas as pd\n\nfrom packaging import version\nfrom scipy.sparse import csr_matrix\nfrom typing im"
},
{
"path": "bertopic/representation/_langchain.py",
"chars": 8829,
"preview": "import pandas as pd\nfrom langchain.docstore.document import Document\nfrom scipy.sparse import csr_matrix\nfrom typing imp"
},
{
"path": "bertopic/representation/_litellm.py",
"chars": 6714,
"preview": "import time\nfrom litellm import completion\nimport pandas as pd\nfrom tqdm import tqdm\nfrom scipy.sparse import csr_matrix"
},
{
"path": "bertopic/representation/_llamacpp.py",
"chars": 9446,
"preview": "import pandas as pd\nfrom tqdm import tqdm\nfrom scipy.sparse import csr_matrix\nfrom llama_cpp import Llama\nfrom typing im"
},
{
"path": "bertopic/representation/_mmr.py",
"chars": 4701,
"preview": "import warnings\nimport numpy as np\nimport pandas as pd\nfrom typing import List, Mapping, Tuple\nfrom scipy.sparse import "
},
{
"path": "bertopic/representation/_openai.py",
"chars": 11060,
"preview": "import time\nimport openai\nimport pandas as pd\nfrom tqdm import tqdm\nfrom scipy.sparse import csr_matrix\nfrom typing impo"
},
{
"path": "bertopic/representation/_pos.py",
"chars": 5958,
"preview": "import numpy as np\nimport pandas as pd\n\nimport spacy\nfrom spacy.matcher import Matcher\nfrom spacy.language import Langua"
},
{
"path": "bertopic/representation/_textgeneration.py",
"chars": 7991,
"preview": "import pandas as pd\nfrom tqdm import tqdm\nfrom scipy.sparse import csr_matrix\nfrom transformers import pipeline, set_see"
},
{
"path": "bertopic/representation/_utils.py",
"chars": 4652,
"preview": "import random\nimport time\nfrom typing import Union\n\n\ndef truncate_document(topic_model, doc_length: Union[int, None], to"
},
{
"path": "bertopic/representation/_visual.py",
"chars": 10697,
"preview": "import numpy as np\nimport pandas as pd\n\nfrom PIL import Image\nfrom tqdm import tqdm\nfrom scipy.sparse import csr_matrix\n"
},
{
"path": "bertopic/representation/_zeroshot.py",
"chars": 4099,
"preview": "import pandas as pd\nfrom transformers import pipeline\nfrom transformers.pipelines.base import Pipeline\nfrom scipy.sparse"
},
{
"path": "bertopic/vectorizers/__init__.py",
"chars": 151,
"preview": "from ._ctfidf import ClassTfidfTransformer\nfrom ._online_cv import OnlineCountVectorizer\n\n__all__ = [\"ClassTfidfTransfor"
},
{
"path": "bertopic/vectorizers/_ctfidf.py",
"chars": 4262,
"preview": "from typing import List\nfrom sklearn.feature_extraction.text import TfidfTransformer\nfrom sklearn.preprocessing import n"
},
{
"path": "bertopic/vectorizers/_online_cv.py",
"chars": 6022,
"preview": "import numpy as np\nfrom itertools import chain\nfrom typing import List\n\nfrom scipy import sparse\nfrom scipy.sparse impor"
},
{
"path": "docs/algorithm/algorithm.md",
"chars": 12958,
"preview": "---\nhide:\n - navigation\n---\n\n# The Algorithm\n\nBelow, you will find different types of overviews of each step in BERTopi"
},
{
"path": "docs/api/backends.md",
"chars": 35,
"preview": "# `Backends`\n\n::: bertopic.backend\n"
},
{
"path": "docs/api/bertopic.md",
"chars": 46,
"preview": "# `BERTopic`\n\n::: bertopic._bertopic.BERTopic\n"
},
{
"path": "docs/api/cluster copy.md",
"chars": 56,
"preview": "# `BaseCluster`\n\n::: bertopic.cluster._base.BaseCluster\n"
},
{
"path": "docs/api/cluster.md",
"chars": 56,
"preview": "# `BaseCluster`\n\n::: bertopic.cluster._base.BaseCluster\n"
},
{
"path": "docs/api/ctfidf.md",
"chars": 61,
"preview": "# `c-TF-IDF`\n\n::: bertopic.vectorizers.ClassTfidfTransformer\n"
},
{
"path": "docs/api/dimensionality.md",
"chars": 95,
"preview": "# `BaseDimensionalityReduction`\n\n::: bertopic.dimensionality._base.BaseDimensionalityReduction\n"
},
{
"path": "docs/api/plotting/barchart.md",
"chars": 65,
"preview": "# `Barchart`\n\n::: bertopic.plotting._barchart.visualize_barchart\n"
},
{
"path": "docs/api/plotting/distribution.md",
"chars": 77,
"preview": "# `Distribution`\n\n::: bertopic.plotting._distribution.visualize_distribution\n"
},
{
"path": "docs/api/plotting/document_datamap.md",
"chars": 90,
"preview": "# `Documents with DataMapPlot`\n\n::: bertopic.plotting._datamap.visualize_document_datamap\n"
},
{
"path": "docs/api/plotting/documents.md",
"chars": 68,
"preview": "# `Documents`\n\n::: bertopic.plotting._documents.visualize_documents\n"
},
{
"path": "docs/api/plotting/dtm.md",
"chars": 76,
"preview": "# `DTM`\n\n::: bertopic.plotting._topics_over_time.visualize_topics_over_time\n"
},
{
"path": "docs/api/plotting/heatmap.md",
"chars": 62,
"preview": "# `Heatmap`\n\n::: bertopic.plotting._heatmap.visualize_heatmap\n"
},
{
"path": "docs/api/plotting/hierarchical_documents.md",
"chars": 107,
"preview": "# `Hierarchical Documents`\n\n::: bertopic.plotting._hierarchical_documents.visualize_hierarchical_documents\n"
},
{
"path": "docs/api/plotting/hierarchy.md",
"chars": 68,
"preview": "# `Hierarchy`\n\n::: bertopic.plotting._hierarchy.visualize_hierarchy\n"
},
{
"path": "docs/api/plotting/term.md",
"chars": 77,
"preview": "# `Term Score Decline`\n\n::: bertopic.plotting._term_rank.visualize_term_rank\n"
},
{
"path": "docs/api/plotting/topics.md",
"chars": 59,
"preview": "# `Topics`\n\n::: bertopic.plotting._topics.visualize_topics\n"
},
{
"path": "docs/api/plotting/topics_per_class.md",
"chars": 89,
"preview": "# `Topics per Class`\n\n::: bertopic.plotting._topics_per_class.visualize_topics_per_class\n"
},
{
"path": "docs/api/plotting.md",
"chars": 36,
"preview": "# `Plotting`\n\n::: bertopic.plotting\n"
},
{
"path": "docs/api/representations.md",
"chars": 49,
"preview": "# `Representations`\n\n::: bertopic.representation\n"
},
{
"path": "docs/api/vectorizers.md",
"chars": 75,
"preview": "# `Vectorizers`\n\n::: bertopic.vectorizers._online_cv.OnlineCountVectorizer\n"
},
{
"path": "docs/changelog.md",
"chars": 96536,
"preview": "---\nhide:\n - navigation\n---\n\n# Changelog\n\n\n## **Version 0.17.4**\n*Release date: 3 December, 2025*\n\n<h3><b>Highlights:</"
},
{
"path": "docs/faq.md",
"chars": 19349,
"preview": "---\nhide:\n - navigation\n---\n\n# Frequently Asked Questions\n\n## **Why are the results not consistent between runs?**\nDue "
},
{
"path": "docs/getting_started/best_practices/best_practices.md",
"chars": 20788,
"preview": "[](https://colab.research.google.com/drive/1Bo"
},
{
"path": "docs/getting_started/clustering/clustering.md",
"chars": 5517,
"preview": "After reducing the dimensionality of our input embeddings, we need to cluster them into groups of similar embeddings to "
},
{
"path": "docs/getting_started/ctfidf/ctfidf.md",
"chars": 4068,
"preview": "# c-TF-IDF\n\nIn BERTopic, in order to get an accurate representation of the topics from our bag-of-words matrix, TF-IDF w"
},
{
"path": "docs/getting_started/dim_reduction/dim_reduction.md",
"chars": 5307,
"preview": "An important aspect of BERTopic is the dimensionality reduction of the input embeddings. As embeddings are often high in"
},
{
"path": "docs/getting_started/distribution/distribution.md",
"chars": 6309,
"preview": "BERTopic approaches topic modeling as a cluster task and attempts to cluster semantically similar documents to extract c"
},
{
"path": "docs/getting_started/distribution/distribution_viz.html",
"chars": 10026,
"preview": "<html>\n<head><meta charset=\"utf-8\" /></head>\n<body>\n <div> <script type=\"text/javascript\">wind"
},
{
"path": "docs/getting_started/embeddings/embeddings.md",
"chars": 15623,
"preview": "# Embedding Models\nBERTopic starts with transforming our input documents into numerical representations. Although there "
},
{
"path": "docs/getting_started/guided/guided.md",
"chars": 3542,
"preview": "Guided Topic Modeling or Seeded Topic Modeling is a collection of techniques that guides the topic modeling approach by "
},
{
"path": "docs/getting_started/hierarchicaltopics/hierarchical_topics.html",
"chars": 57798,
"preview": "<html>\n<head><meta charset=\"utf-8\" /></head>\n<body>\n <div> <script type=\"text/javascript\">wind"
},
{
"path": "docs/getting_started/hierarchicaltopics/hierarchicaltopics.md",
"chars": 25365,
"preview": "When tweaking your topic model, the number of topics that are generated has a large effect on the quality of the topic r"
},
{
"path": "docs/getting_started/manual/manual.md",
"chars": 4473,
"preview": "Although topic modeling is typically done by discovering topics in an unsupervised manner, there might be times when you"
},
{
"path": "docs/getting_started/merge/merge.md",
"chars": 6412,
"preview": "# Merge Multiple Fitted Models\n\nAfter you have trained a new BERTopic model on your data, new data might still be coming"
},
{
"path": "docs/getting_started/multiaspect/multiaspect.md",
"chars": 2558,
"preview": "During the development of BERTopic, many different types of representations can be created, from keywords and phrases to"
},
{
"path": "docs/getting_started/multimodal/multimodal.md",
"chars": 7062,
"preview": "Documents or text are often accompanied by imagery or the other way around. For example, social media images with captio"
},
{
"path": "docs/getting_started/online/online.md",
"chars": 8456,
"preview": "Online topic modeling (sometimes called \"incremental topic modeling\") is the ability to learn incrementally from a mini-"
},
{
"path": "docs/getting_started/outlier_reduction/fig_base.html",
"chars": 409098,
"preview": "<html>\n<head><meta charset=\"utf-8\" /></head>\n<body>\n <div> <script type=\"text/javascript\">wind"
},
{
"path": "docs/getting_started/outlier_reduction/fig_reduced.html",
"chars": 421468,
"preview": "<html>\n<head><meta charset=\"utf-8\" /></head>\n<body>\n <div> <script type=\"text/javascript\">wind"
},
{
"path": "docs/getting_started/outlier_reduction/outlier_reduction.md",
"chars": 8710,
"preview": "When using HDBSCAN, DBSCAN, or OPTICS, a number of outlier documents might be created\nthat do not fall within any of the"
},
{
"path": "docs/getting_started/parameter tuning/parametertuning.md",
"chars": 7679,
"preview": "# Hyperparameter Tuning\n\nAlthough BERTopic works quite well out of the box, there are a number of hyperparameters to tun"
},
{
"path": "docs/getting_started/quickstart/quickstart.md",
"chars": 7470,
"preview": "## **Installation**\n\nInstallation, with sentence-transformers, can be done using [pypi](https://pypi.org/project/bertopi"
},
{
"path": "docs/getting_started/quickstart/viz.html",
"chars": 3673141,
"preview": "<html>\n<head><meta charset=\"utf-8\" /></head>\n<body>\n <div> <script type=\"text/javascript\">wind"
},
{
"path": "docs/getting_started/representation/llm.md",
"chars": 24620,
"preview": "As we have seen in the [previous section](https://maartengr.github.io/BERTopic/getting_started/representation/representa"
},
{
"path": "docs/getting_started/representation/representation.md",
"chars": 10564,
"preview": "One of the core components of BERTopic is its Bag-of-Words representation and weighting with c-TF-IDF. This method is fa"
},
{
"path": "docs/getting_started/search/search.md",
"chars": 1714,
"preview": "After having created a BERTopic model, you might end up with over a hundred topics. Searching through those\ncan be quite"
},
{
"path": "docs/getting_started/seed_words/seed_words.md",
"chars": 4408,
"preview": "When performing Topic Modeling, you are often faced with data that you are familiar with to a certain extend or that spe"
},
{
"path": "docs/getting_started/semisupervised/semisupervised.md",
"chars": 3659,
"preview": "In BERTopic, you have several options to nudge the creation of topics toward certain pre-specified topics. Here, we will"
},
{
"path": "docs/getting_started/serialization/serialization.md",
"chars": 5217,
"preview": "Saving, loading, and sharing a BERTopic model can be done in several ways. It is generally advised to go with `.safetens"
},
{
"path": "docs/getting_started/supervised/supervised.md",
"chars": 5275,
"preview": "Although topic modeling is typically done by discovering topics in an unsupervised manner, there might be times when you"
},
{
"path": "docs/getting_started/tips_and_tricks/tips_and_tricks.md",
"chars": 21103,
"preview": "# Tips & Tricks\n\n\n## **Document length**\nAs a default, we are using sentence-transformers to embed our documents. Howeve"
},
{
"path": "docs/getting_started/topicreduction/topicreduction.md",
"chars": 3809,
"preview": "BERTopic uses HDBSCAN for clustering the data and it cannot specify the number of clusters you would want. To a certain "
},
{
"path": "docs/getting_started/topicrepresentation/topicrepresentation.md",
"chars": 6554,
"preview": "The topics that are extracted from BERTopic are represented by words. These words are extracted from the documents\noccup"
},
{
"path": "docs/getting_started/topicsovertime/topicsovertime.md",
"chars": 6221,
"preview": "Dynamic topic modeling (DTM) is a collection of techniques aimed at analyzing the evolution of topics\nover time. These m"
},
{
"path": "docs/getting_started/topicsovertime/trump.html",
"chars": 23081,
"preview": "<html>\n<head><meta charset=\"utf-8\" /></head>\n<body>\n <div> <script type=\"text/javascript\">wind"
},
{
"path": "docs/getting_started/topicsperclass/topics_per_class.html",
"chars": 28274,
"preview": "<html>\n<head><meta charset=\"utf-8\" /></head>\n<body>\n <div> <script type=\"text/javascript\">wind"
},
{
"path": "docs/getting_started/topicsperclass/topicsperclass.md",
"chars": 2617,
"preview": "In some cases, you might be interested in how certain topics are represented over certain categories. Perhaps\nthere are "
},
{
"path": "docs/getting_started/vectorizers/vectorizers.md",
"chars": 14083,
"preview": "In topic modeling, the quality of the topic representations is key for interpreting the topics, communicating results, a"
},
{
"path": "docs/getting_started/visualization/bar_chart.html",
"chars": 12272,
"preview": "<html>\n<head><meta charset=\"utf-8\" /></head>\n<body>\n <div> <script type=\"text/javascript\">wind"
},
{
"path": "docs/getting_started/visualization/datamapplot.html",
"chars": 1966727,
"preview": "<!doctype html>\n<html>\n <head>\n <meta http-equiv=\"content-type\" content=\"text/html; charset=UTF-8\" />\n <title>Int"
},
{
"path": "docs/getting_started/visualization/documents.html",
"chars": 452548,
"preview": "<html>\n<head><meta charset=\"utf-8\" /></head>\n<body>\n <div> <script type=\"text/javascript\">wind"
},
{
"path": "docs/getting_started/visualization/heatmap.html",
"chars": 62445,
"preview": "<html>\n<head><meta charset=\"utf-8\" /></head>\n<body>\n <div> <script type=\"text/javascript\">wind"
},
{
"path": "docs/getting_started/visualization/hierarchical_documents.html",
"chars": 739908,
"preview": "<html>\n<head><meta charset=\"utf-8\" /></head>\n<body>\n <div> <script type=\"text/javascript\">wind"
},
{
"path": "docs/getting_started/visualization/hierarchical_topics.html",
"chars": 57798,
"preview": "<html>\n<head><meta charset=\"utf-8\" /></head>\n<body>\n <div> <script type=\"text/javascript\">wind"
},
{
"path": "docs/getting_started/visualization/hierarchy.html",
"chars": 22393,
"preview": "<html>\n<head><meta charset=\"utf-8\" /></head>\n<body>\n <div> <script type=\"text/javascript\">wind"
},
{
"path": "docs/getting_started/visualization/probabilities.html",
"chars": 10499,
"preview": "<html>\n<head><meta charset=\"utf-8\" /></head>\n<body>\n <div> <script type=\"text/javascript\">wind"
},
{
"path": "docs/getting_started/visualization/term_rank.html",
"chars": 81962,
"preview": "<html>\n<head><meta charset=\"utf-8\" /></head>\n<body>\n <div> <script type=\"text/javascript\">wind"
},
{
"path": "docs/getting_started/visualization/term_rank_log.html",
"chars": 80104,
"preview": "<html>\n<head><meta charset=\"utf-8\" /></head>\n<body>\n <div> <script type=\"text/javascript\">wind"
},
{
"path": "docs/getting_started/visualization/topics_per_class.html",
"chars": 28877,
"preview": "<html>\n<head><meta charset=\"utf-8\" /></head>\n<body>\n <div> <script type=\"text/javascript\">wind"
},
{
"path": "docs/getting_started/visualization/trump.html",
"chars": 23081,
"preview": "<html>\n<head><meta charset=\"utf-8\" /></head>\n<body>\n <div> <script type=\"text/javascript\">wind"
},
{
"path": "docs/getting_started/visualization/visualization.md",
"chars": 36598,
"preview": "Visualizing BERTopic and its derivatives is important in understanding the model, how it works, and more importantly, wh"
},
{
"path": "docs/getting_started/visualization/visualize_documents.md",
"chars": 6091,
"preview": "## **Visualize documents with Plotly**\n\nUsing the `.visualize_topics`, we can visualize the topics and get insight into "
},
{
"path": "docs/getting_started/visualization/visualize_hierarchy.md",
"chars": 24410,
"preview": "The topics that you create can be hierarchically reduced. In order to understand the potential hierarchical\nstructure of"
},
{
"path": "docs/getting_started/visualization/visualize_terms.md",
"chars": 2273,
"preview": "We can visualize the selected terms for a few topics by creating bar charts out of the c-TF-IDF scores\nfor each topic re"
},
{
"path": "docs/getting_started/visualization/visualize_topics.md",
"chars": 5056,
"preview": "Visualizing BERTopic and its derivatives is important in understanding the model, how it works, and more importantly, wh"
},
{
"path": "docs/getting_started/visualization/viz.html",
"chars": 3673141,
"preview": "<html>\n<head><meta charset=\"utf-8\" /></head>\n<body>\n <div> <script type=\"text/javascript\">wind"
},
{
"path": "docs/getting_started/zeroshot/zeroshot.md",
"chars": 4470,
"preview": "Zero-shot Topic Modeling is a technique that allows you to find topics in large amounts of documents that were predefine"
},
{
"path": "docs/img/probabilities.html",
"chars": 10327,
"preview": "<html>\n<head><meta charset=\"utf-8\" /></head>\n<body>\n <div> <div id=\"325bf5d5-b201-4a49-9ec"
},
{
"path": "docs/index.md",
"chars": 15668,
"preview": "---\nhide:\n - navigation\n---\n\n# BERTopic\n\n<img src=\"logo.png\" width=\"30%\" height=\"30%\" align=\"right\" />\n\nBERTopic is a t"
},
{
"path": "docs/stylesheets/extra.css",
"chars": 416,
"preview": ":root {\n --md-primary-fg-color: #234E70;\n\n }\n\n:root>* {\n--md-typeset-a-color: #016198;\n--md-text-link-color: "
},
{
"path": "docs/usecases.md",
"chars": 14592,
"preview": "---\nhide:\n - navigation\n---\n\nOver the last few years, BERTopic has been used on a wide variety of use cases and domains"
},
{
"path": "mkdocs.yml",
"chars": 5193,
"preview": "site_name: BERTopic\nrepo_url: https://github.com/MaartenGr/BERTopic\nsite_url: https://maartengr.github.io/BERTopic/\nsite"
},
{
"path": "pyproject.toml",
"chars": 2965,
"preview": "[build-system]\nrequires = [\"setuptools\"]\nbuild-backend = \"setuptools.build_meta\"\n\n[project]\nname = \"bertopic\"\nversion = "
},
{
"path": "tests/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "tests/conftest.py",
"chars": 6351,
"preview": "import copy\nimport pytest\nfrom umap import UMAP\nfrom hdbscan import HDBSCAN\nfrom bertopic import BERTopic\nfrom sklearn.d"
},
{
"path": "tests/test_bertopic.py",
"chars": 5419,
"preview": "import copy\nimport pytest\nfrom bertopic import BERTopic\nimport importlib.util\n\n\ndef cuml_available():\n try:\n r"
},
{
"path": "tests/test_other.py",
"chars": 1368,
"preview": "from bertopic import BERTopic\nfrom bertopic.dimensionality import BaseDimensionalityReduction\n\ntry:\n from plotly.grap"
},
{
"path": "tests/test_plotting/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "tests/test_plotting/test_approximate.py",
"chars": 1324,
"preview": "import copy\nimport pytest\n\n\n@pytest.mark.parametrize(\"batch_size\", [50, None])\n@pytest.mark.parametrize(\"padding\", [True"
},
{
"path": "tests/test_plotting/test_bar.py",
"chars": 1705,
"preview": "import copy\nimport pytest\n\n\n@pytest.mark.parametrize(\n \"model\",\n [\n (\"kmeans_pca_topic_model\"),\n (\"b"
},
{
"path": "tests/test_plotting/test_documents.py",
"chars": 697,
"preview": "import copy\nimport pytest\n\n\n@pytest.mark.parametrize(\n \"model\",\n [\n (\"kmeans_pca_topic_model\"),\n (\"b"
},
{
"path": "tests/test_plotting/test_dynamic.py",
"chars": 681,
"preview": "import copy\nimport pytest\n\n\n@pytest.mark.parametrize(\n \"model\",\n [\n (\"kmeans_pca_topic_model\"),\n (\"b"
},
{
"path": "tests/test_plotting/test_heatmap.py",
"chars": 595,
"preview": "import copy\nimport pytest\n\n\n@pytest.mark.parametrize(\n \"model\",\n [\n (\"kmeans_pca_topic_model\"),\n (\"b"
},
{
"path": "tests/test_plotting/test_term_rank.py",
"chars": 276,
"preview": "import copy\nimport pytest\n\n\n@pytest.mark.parametrize(\"model\", [(\"kmeans_pca_topic_model\"), (\"base_topic_model\"), (\"custo"
},
{
"path": "tests/test_plotting/test_topics.py",
"chars": 1562,
"preview": "import copy\nimport pytest\n\n\n@pytest.mark.parametrize(\n \"model\",\n [\n (\"kmeans_pca_topic_model\"),\n (\"b"
},
{
"path": "tests/test_reduction/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "tests/test_reduction/test_delete.py",
"chars": 2323,
"preview": "import copy\nimport pytest\n\n\n@pytest.mark.parametrize(\n \"model\",\n [\n (\"kmeans_pca_topic_model\"),\n (\"b"
},
{
"path": "tests/test_reduction/test_merge.py",
"chars": 1538,
"preview": "import copy\nimport pytest\n\n\n@pytest.mark.parametrize(\n \"model\",\n [\n (\"kmeans_pca_topic_model\"),\n (\"b"
},
{
"path": "tests/test_representation/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "tests/test_representation/test_get.py",
"chars": 3712,
"preview": "import copy\nimport pytest\nimport numpy as np\nimport pandas as pd\n\n\n@pytest.mark.parametrize(\n \"model\",\n [\n "
},
{
"path": "tests/test_representation/test_labels.py",
"chars": 2617,
"preview": "import copy\nimport pytest\n\n\n@pytest.mark.parametrize(\n \"model\",\n [\n (\"kmeans_pca_topic_model\"),\n (\"b"
},
{
"path": "tests/test_representation/test_representations.py",
"chars": 5874,
"preview": "import copy\nimport pytest\nimport numpy as np\nimport pandas as pd\nfrom sklearn.feature_extraction.text import CountVector"
},
{
"path": "tests/test_sub_models/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "tests/test_sub_models/test_cluster.py",
"chars": 2573,
"preview": "import pytest\nimport pandas as pd\n\nfrom sklearn.datasets import make_blobs\nfrom sklearn.cluster import KMeans\nfrom hdbsc"
},
{
"path": "tests/test_sub_models/test_dim_reduction.py",
"chars": 1201,
"preview": "import copy\nimport pytest\nimport numpy as np\nfrom umap import UMAP\nfrom sklearn.decomposition import PCA\n\nfrom bertopic "
},
{
"path": "tests/test_sub_models/test_embeddings.py",
"chars": 2052,
"preview": "import copy\nimport pytest\nimport numpy as np\nfrom bertopic import BERTopic\nfrom sklearn.metrics.pairwise import cosine_s"
},
{
"path": "tests/test_utils.py",
"chars": 3318,
"preview": "import pytest\nimport logging\nimport numpy as np\nfrom typing import List\nfrom bertopic._utils import (\n check_document"
},
{
"path": "tests/test_variations/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "tests/test_variations/test_class.py",
"chars": 1219,
"preview": "import copy\nimport pytest\nfrom sklearn.datasets import fetch_20newsgroups\n\ndata = fetch_20newsgroups(subset=\"all\", remov"
},
{
"path": "tests/test_variations/test_dynamic.py",
"chars": 703,
"preview": "import copy\nimport pytest\n\n\n@pytest.mark.parametrize(\n \"model\",\n [\n (\"kmeans_pca_topic_model\"),\n (\"c"
},
{
"path": "tests/test_variations/test_hierarchy.py",
"chars": 2408,
"preview": "import copy\nimport pytest\nfrom scipy.cluster import hierarchy as sch\n\n\n@pytest.mark.parametrize(\n \"model\",\n [\n "
},
{
"path": "tests/test_vectorizers/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "tests/test_vectorizers/test_ctfidf.py",
"chars": 3468,
"preview": "import copy\nimport pytest\nimport numpy as np\nimport pandas as pd\nfrom packaging import version\nfrom scipy.sparse import "
},
{
"path": "tests/test_vectorizers/test_online_cv.py",
"chars": 1344,
"preview": "import copy\nimport pytest\nfrom bertopic.vectorizers import OnlineCountVectorizer\n\n\n@pytest.mark.parametrize(\n \"model\""
}
]
About this extraction
This page contains the full source code of the MaartenGr/BERTopic GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 184 files (12.3 MB), approximately 3.2M tokens, and a symbol index with 329 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.