...
blocks, flattening their content."""
content = re.sub(r"', "", content)
content = re.sub(r"
", "", content)
return content
def process_overview():
url = "https://raw.githubusercontent.com/pingcap/docs/refs/heads/master/vector-search/vector-search-overview.md"
response = requests.get(url)
content = response.text
content = remove_custom_content_blocks(content)
content = collapse_extra_blank_lines(content)
content = convert_note_blocks(content)
content = remove_see_also_section(content)
content = replace_image_paths(content)
content = replace_relative_doc_links(content)
content = remove_overview_from_title(content)
save_to_file(content, "./src/ai/concepts/vector-search.md")
def process_llamaindex():
url = "https://raw.githubusercontent.com/pingcap/docs/master/vector-search/vector-search-integrate-with-llamaindex.md"
response = requests.get(url)
content = response.text
content = remove_front_matter(content)
content = remove_custom_content_blocks(content)
content = remove_simpletab_blocks(content)
content = collapse_extra_blank_lines(content)
content = convert_warning_blocks(content)
content = convert_note_blocks(content)
content = convert_tip_blocks(content)
content = remove_see_also_section(content)
content = replace_image_paths(content)
content = replace_relative_doc_links(content)
save_to_file(content, "./src/ai/integrations/llamaindex.md")
def process_langchain():
url = "https://raw.githubusercontent.com/pingcap/docs/master/vector-search/vector-search-integrate-with-langchain.md"
response = requests.get(url)
content = response.text
content = remove_front_matter(content)
content = remove_custom_content_blocks(content)
content = remove_simpletab_blocks(content)
content = collapse_extra_blank_lines(content)
content = convert_warning_blocks(content)
content = convert_note_blocks(content)
content = convert_tip_blocks(content)
content = remove_see_also_section(content)
content = replace_image_paths(content)
content = replace_relative_doc_links(content)
save_to_file(content, "./src/ai/integrations/langchain.md")
def save_to_file(content, filename):
"""Save the processed content to a file."""
with open(filename, "w") as f:
f.write(content)
def main():
if len(sys.argv) > 1:
arg = sys.argv[1].lower()
if arg in ("--llamaindex", "llamaindex"):
process_llamaindex()
print("llamaindex doc synced.")
return
elif arg in ("--langchain", "langchain"):
process_langchain()
print("langchain doc synced.")
return
elif arg in ("--overview", "overview"):
process_overview()
print("overview doc synced.")
return
else:
print(f"Unknown argument: {arg}")
print("Usage: python sync_from_tidb_docs.py [--llamaindex|--langchain|--overview]")
return
process_overview()
print("overview doc synced (default).")
if __name__ == "__main__":
main()
================================================
FILE: src/ai/concepts/vector-search.md
================================================
---
title: Vector Search Overview
summary: Learn about Vector Search in TiDB. This feature provides an advanced search solution for performing semantic similarity searches across various data types, including documents, images, audio, and video.
---
# Vector Search
Vector search offers a powerful solution for semantic similarity searches across diverse data types, such as documents, images, audio, and video. It allows developers to leverage their MySQL expertise to build scalable applications enriched with generative AI capabilities, simplifying the integration of advanced search functionality.
!!! note
The vector search feature is only available for TiDB Self-Managed clusters and [TiDB Cloud Starter](https://docs.pingcap.com/tidbcloud/select-cluster-tier#tidb-cloud-starter) clusters.
## Concepts
Vector search is a search method that prioritizes the meaning of your data to deliver relevant results.
Unlike traditional full-text search, which relies on exact keyword matching and word frequency, vector search converts various data types (such as text, images, or audio) into high-dimensional vectors and queries based on the similarity between these vectors. This search method captures the semantic meaning and contextual information of the data, leading to a more precise understanding of user intent.
Even when the search terms do not exactly match the content in the database, vector search can still provide results that align with the user's intent by analyzing the semantics of the data.
For example, a full-text search for "a swimming animal" only returns results containing these exact keywords. In contrast, vector search can return results for other swimming animals, such as fish or ducks, even if these results do not contain the exact keywords.
### Vector embedding
A vector embedding, also known as an embedding, is a sequence of numbers that represents real-world objects in a high-dimensional space. It captures the meaning and context of unstructured data, such as documents, images, audio, and videos.
Vector embeddings are essential in machine learning and serve as the foundation for semantic similarity searches.
TiDB introduces [Vector data types](https://docs.pingcap.com/tidbcloud/vector-search-data-types) and [Vector search index](https://docs.pingcap.com/tidbcloud/vector-search-index) designed to optimize the storage and retrieval of vector embeddings, enhancing their use in AI applications. You can store vector embeddings in TiDB and perform vector search queries to find the most relevant data using these data types.
### Embedding model
Embedding models are algorithms that transform data into [vector embeddings](#vector-embedding).
Choosing an appropriate embedding model is crucial for ensuring the accuracy and relevance of semantic search results. For unstructured text data, you can find top-performing text embedding models on the [Massive Text Embedding Benchmark (MTEB) Leaderboard](https://huggingface.co/spaces/mteb/leaderboard).
To learn how to generate vector embeddings for your specific data types, refer to integration tutorials or examples of embedding models.
## How vector search works
After converting raw data into vector embeddings and storing them in TiDB, your application can execute vector search queries to find the data most semantically or contextually relevant to a user's query.
TiDB vector search identifies the top-k nearest neighbor (KNN) vectors by using a [distance function](https://docs.pingcap.com/tidbcloud/vector-search-functions-and-operators) to calculate the distance between the given vector and vectors stored in the database. The vectors closest to the given vector in the query represent the most similar data in meaning.
As a relational database with integrated vector search capabilities, TiDB enables you to store data and their corresponding vector representations (that is, vector embeddings) together in one database. You can choose any of the following ways for storage:
- Store data and their corresponding vector representations in different columns of the same table.
- Store data and their corresponding vector representation in different tables. In this way, you need to use `JOIN` queries to combine the tables when retrieving data.
## Use cases
### Retrieval-Augmented Generation (RAG)
Retrieval-Augmented Generation (RAG) is an architecture designed to optimize the output of Large Language Models (LLMs). By using vector search, RAG applications can store vector embeddings in the database and retrieve relevant documents as additional context when the LLM generates responses, thereby improving the quality and relevance of the answers.
### Semantic search
Semantic search is a search technology that returns results based on the meaning of a query, rather than simply matching keywords. It interprets the meaning across different languages and various types of data (such as text, images, and audio) using embeddings. Vector search algorithms then use these embeddings to find the most relevant data that satisfies the user's query.
### Recommendation engine
A recommendation engine is a system that proactively suggests content, products, or services that are relevant and personalized to users. It accomplishes this by creating embeddings that represent user behavior and preferences. These embeddings help the system identify similar items that other users have interacted with or shown interest in. This increases the likelihood that the recommendations will be both relevant and appealing to the user.
================================================
FILE: src/ai/examples/auto-embedding-with-pytidb.md
================================================
---
title: Auto Embedding
description: "Automatically generate embeddings for your text data using built-in embedding models."
source_repo: "https://github.com/pingcap/pytidb/tree/main/examples/auto_embedding"
---
# Auto Embedding Demo
This example showcases how to use the auto embedding feature with PyTiDB Client.
* Connect to TiDB with PyTiDB Client
* Define a table with a VectorField configured for automatic embedding
* Insert plain text data, embeddings are populated automatically in the background
* Run vector searches with natural language queries, embedding happens transparently
## Prerequisites
- **Python 3.10+**
- **A TiDB Cloud Starter cluster**: Create a free cluster here: [tidbcloud.com ↗️](https://tidbcloud.com/?utm_source=github&utm_medium=referral&utm_campaign=pytidb_readme)
## How to run
**Step 1**: Clone the repository
```bash
git clone https://github.com/pingcap/pytidb.git
cd pytidb/examples/auto_embedding/
```
**Step 2**: Install the required packages
```bash
python -m venv .venv
source .venv/bin/activate
pip install -r reqs.txt
```
**Step 3**: Set up environment to connect to database
Go to [TiDB Cloud console](https://tidbcloud.com/clusters) to get the connection parameters and set up the environment variable like this:
```bash
cat > .env <E-commerce product search with full-text search
## Prerequisites - **Python 3.10+** - **A TiDB Cloud Starter cluster**: Create a free cluster here: [tidbcloud.com ↗️](https://tidbcloud.com/?utm_source=github&utm_medium=referral&utm_campaign=pytidb_readme) ## How to run **Step 1**: Clone the repository to local ```bash git clone https://github.com/pingcap/pytidb.git cd pytidb/examples/fulltext_search/; ``` **Step 2**: Install the required packages and setup environment ```bash python -m venv .venv source .venv/bin/activate pip install -r reqs.txt ``` **Step 3**: Set up environment to connect to database Go to the [TiDB Cloud console](https://tidbcloud.com/), create a new cluster if you don't have one, and then get the connection parameters on the connection dialog. ```bash cat > .env <TiDB Hybrid Search Demo
## Prerequisites * Python 3.10+ * TiDB database instance (👉 [Create a free TiDB Serverless Cluster](https://tidbcloud.com/free-trial)) * OpenAI API key (Go to [OpenAI](https://platform.openai.com/api-keys) to get the API key) > **Note** > > Currently, full-text search is only available for the following product option and region: > > - TiDB Cloud Starter: Frankfurt (eu-central-1), Singapore (ap-southeast-1) ## How to run **Step 1**: Clone the repository ```bash git clone https://github.com/pingcap/pytidb.git cd pytidb/examples/hybrid_search; ``` **Step 2**: Install the required packages and setup environment ```bash python -m venv .venv source .venv/bin/activate pip install -r reqs.txt ``` **Step 3**: Set up environment to connect to storage If you are using TiDB Cloud, you can find the connection parameters in the [TiDB Cloud console](https://tidbcloud.com/). ```bash cat > .env <
Pet image search via multimodal embeddings
## Prerequisites - **Python 3.10+** - **A TiDB Cloud Starter cluster**: Create a free cluster here: [tidbcloud.com ↗️](https://tidbcloud.com/?utm_source=github&utm_medium=referral&utm_campaign=pytidb_readme) - **Jina AI API Key**: Get your free API key at [jina.ai Embeddings ↗️](https://jina.ai/embeddings/) ## How to run **Step 1**: Clone the repository to local ```bash git clone https://github.com/pingcap/pytidb.git cd pytidb/examples/image_search/ ``` **Step 2**: Install the required packages ```bash python -m venv .venv source .venv/bin/activate # Windows: .venv\Scripts\activate pip install -r reqs.txt ``` **Step 3**: Set up environment variables Go to [TiDB Cloud console](https://tidbcloud.com/clusters) and get the connection parameters, then set up the environment variable like this: ```bash cat > .env <Demo Gallery
Explore hands-on demos showcasing how TiDB empowers AI applications.
Get started quickly with TiDB Cloud Starter to build your own AI-powered solutions.
⭐ Featured
🚀 Getting Started
🔍 Search & Retrieval
Vector Search
Implement semantic search using vector embeddings to find similar content.
Fulltext Search
Perform traditional text search using MySQL fulltext search capabilities.
Hybrid Search
Combine vector search and fulltext search for more comprehensive results.
Image Search
Build an image search application using multimodal embeddings for both text-to-image and image-to-image search.
🤖 AI Applications
Ready to build your AI application?
Start your AI journey with TiDB Cloud Starter. Follow our quickstart guide to build your first AI-powered application in minutes, or explore specific examples for your use case.
AI Agent with memory powered by TiDB
## Prerequisites - **Python 3.10+** - **A TiDB Cloud Starter cluster**: Create a free cluster here: [tidbcloud.com ↗️](https://tidbcloud.com/?utm_source=github&utm_medium=referral&utm_campaign=pytidb_readme) - **OpenAI API Key**: Get your API key at [OpenAI Platform ↗️](https://platform.openai.com/api-keys) ## How to run **Step 1**: Clone the repository to local ```bash git clone https://github.com/pingcap/pytidb.git cd pytidb/examples/memory/ ``` **Step 2**: Install the required packages ```bash python -m venv .venv source .venv/bin/activate # Windows: .venv\Scripts\activate pip install -r reqs.txt ``` **Step 3**: Set up environment variables Go to [TiDB Cloud console](https://tidbcloud.com/clusters) and get the connection parameters, then set up the environment variable like this: ```bash cat > .env <RAG application built with PyTiDB
## Prerequisites - **Python 3.10+** - **A TiDB Cloud Starter cluster**: Create a free cluster here: [tidbcloud.com ↗️](https://tidbcloud.com/?utm_source=github&utm_medium=referral&utm_campaign=pytidb_readme) - **Ollama**: You can install it from [Ollama ↗️](https://ollama.com/download) ## How to run **Step 1**: Prepare the inference API Pull the embedding and LLM model via ollama CLI: ```bash ollama pull mxbai-embed-large ollama pull gemma3:4b ollama run gemma3:4b ``` Test the `/embed` and `/generate` endpoints to make sure they are running: ```bash curl http://localhost:11434/api/embed -d '{ "model": "mxbai-embed-large", "input": "Llamas are members of the camelid family" }' ``` ```bash curl http://localhost:11434/api/generate -d '{ "model": "gemma3:4b", "prompt": "Hello, Who are you?" }' ``` **Step 2**: Clone the repository to local ```bash git clone https://github.com/pingcap/pytidb.git cd pytidb/examples/rag/; ``` **Step 3**: Install the required packages and setup environment ```bash python -m venv .venv source .venv/bin/activate pip install -r reqs.txt ``` **Step 4**: Set up environment to connect to database Go to [TiDB Cloud console](https://tidbcloud.com/clusters) and get the connection parameters, then set up the environment variable like this: ```bash cat > .env <Semantic search with vector embeddings
## Prerequisites - **Python 3.10+** - **A TiDB Cloud Starter cluster**: Create a free cluster here: [tidbcloud.com ↗️](https://tidbcloud.com/?utm_source=github&utm_medium=referral&utm_campaign=pytidb_readme) - **Ollama**: You can install it from [Ollama ↗️](https://ollama.com/download) ## How to run **Step 1**: Start the embedding service with Ollama Pull the embedding model: ```bash ollama pull mxbai-embed-large ``` Test the embedding service to make sure it is running: ```bash curl http://localhost:11434/api/embed -d '{ "model": "mxbai-embed-large", "input": "Llamas are members of the camelid family" }' ``` **Step 2**: Clone the repository to local ```bash git clone https://github.com/pingcap/pytidb.git cd pytidb/examples/vector_search/ ``` **Step 3**: Install the required packages and set up the environment ```bash python -m venv .venv source .venv/bin/activate pip install -r reqs.txt ``` **Step 4**: Set up environment to connect to TiDB Go to [TiDB Cloud console](https://tidbcloud.com/clusters) and get the connection parameters, then set up the environment variable like this: ```bash cat > .env <
