The complete pipeline

Does assigning token IDs to terms after tokenization count as part of embedding?

Korewaxnne

It does. But academically, assigning token IDs is normally called mapping or indexing, rather than embedding lookup. First, the full pipeline:

1
Raw text
2
  ↓ tokenize
3
["毕业", "后", "我", "搬回", "了", "老家"]
4
  ↓ mapping (look up vocabulary and map terms to integer IDs)
5
[1456, 78, 23, 8901, 45, 6723]
6
  ↓ embedding lookup (look up vector table and turn every ID into a vector)
7
[[0.23, -0.45, ...], [0.12, 0.78, ...], ...]   ← one vector per token
8
  ↓ pooling / [CLS] (compress a row of vectors into one sentence)
9
[0.56, -0.12, ...]                              ← one vector for the sentence, the API result

Your question falls on mapping, assigning token IDs after segmentation. Does it count as embedding?

• In the broad sense, yes: it is part of the full process of making text usable by the model, and is often loosely called embedding in ordinary discussion.
• Strictly, no: mapping is only a lookup from term to ID and has no vector operation. In academic context, embedding specifically means step three, embedding lookup, mapping discrete token IDs into a continuous vector space.

So both descriptions work; they merely differ in precision. Saying that you “mixed up tokenization and embedding” is accurate enough here. There is no need to worry about where mapping belongs.

A small tip: token counts can indeed be read after tokenization, through len(tokens). This is why models calling something a token normally mean an integer ID, not a vector.

Narrow definitions

Then let us narrow both terms: embedding means token_id_list -> vector; tokenization means segmentation; mapping is mapping.

Intuition for subwords

I remember that a modern BERT vocabulary is usually around thirty thousand words.

If a word has never been seen, and cannot even be assembled from roots, what happens when mapping it?

Korewaxnne

Two questions can be answered together: your direction is right, but “roots” is not quite the correct term.

BERT does not assemble roots; it uses subwords.

For example, Transformer may be missing from a vocabulary, but a BERT WordPiece tokenizer may split it as:

1
Transformer → Trans + ##former

## means this part attaches to the preceding text rather than standing alone. The vocabulary has Trans, a common prefix, and ##former, a common suffix, which can assemble Transformer.

This split is not based on linguistic morphemes, but statistics. During training, the algorithm records high-frequency substrings and includes them in the vocabulary. Sometimes they resemble roots, such as ##former; sometimes they are merely frequent fragments, such as ##tion.

---

What if none of a word’s subwords can be found in the vocabulary?

With BERT WordPiece, this almost never happens because of its fallback:

1
Input: "一个完全没见过的生僻词"
2
   ↓ try longest vocabulary substring
3
"一个完全没见过的生僻词" → if "生僻词" is absent
4
   ↓ recursively split to individual characters
5
"生", "僻", "词" → if even a character is absent
6
   ↓ final fallback
7
[UNK] (Unknown token)

In more modern tokenizers, such as GPT-family BPE and SentencePiece, even [UNK] does not appear. Training includes every individual character or byte, so every input can be assembled; there is no truly unseen input.

So the answer is: in modern tokenizers, “cannot even assemble its roots” is essentially impossible because individual characters exist in the vocabulary. In the worst case, it degrades to one [UNK] token.

The practical difference between a BERT vocabulary of roughly 30k and byte-level BPE is:

	BERT WordPiece	Byte-level BPE
Vocabulary size	~30k	~100k, including every byte
OOV?	Possible; falls back to [UNK]	Almost none; bytes can always assemble input
Split basis	Statistical subwords plus language preference	Pure statistics at byte level

This is why GPT-family models later abandoned WordPiece for BPE: no [UNK] handling, one problem removed.

It is truly a good idea. I remember learning C++:

1
char a = 'a';    // 1 byte
2
string b = "我";  // 3 bytes (UTF-8)
3
string c = "你好"; // 6 bytes (UTF-8)

The exact byte count depends on encoding: under UTF-8, one Chinese character is three bytes; under UTF-32, four.

Anything encodable can be assembled from bytes, and multi-character words are just collections of bytes. It is beautiful: it bypasses linguistic preference and depends completely on statistics.

I want to understand further how BPE and WordPiece differ and resemble each other when learning a vocabulary.

BPE vs. WordPiece

Korewaxnne

Their central difference is the merge criterion, which changes how vocabulary is formed:

	BPE	WordPiece
Merge basis	Frequency: merge the pair that occurs most often	Likelihood: merge the pair that increases model probability most
Training	Start from single characters and repeatedly merge the most frequent adjacent pair until vocabulary size is reached	Start from single characters and merge the pair that most improves training-data likelihood each time
Vocabulary	Purely statistical: frequent fragments mix together, such as ing, tion, ab, and ##s	More inclined toward linguistic boundaries, because improving likelihood often merges meaningful units
OOV handling	Almost none: every byte exists	Possible: fully unseen combinations fall back to [UNK]
Typical models	GPT family, LLaMA	BERT, DistilBERT

An intuitive comparison:

For input "unbelievable":

• BPE, frequency-driven → ["un", "believ", "able"] un and able are merged because they occur frequently; believ is also frequent, though possibly less stable than un.
• WordPiece, likelihood-driven → ["un", "##believ", "##able"] The result looks similar, but ##believ carries ## because the model finds an attached form more probable than an independent word.

The bigger difference is the vocabulary base:

BPE begins at the byte level, so its initial vocabulary contains all 256 byte values. WordPiece begins at the character level, with Unicode characters plus common multi-character combinations. Therefore:

• BPE naturally includes every byte → any input can be assembled → no [UNK]
• WordPiece does not include every Unicode character → rare characters may become [UNK]

Later BERT versions addressed this with full-character coverage, but that is another story.

A blunt summary: BPE is an illiterate statistician that cares only about “which two things occur together often.” WordPiece is half-literate: it knows what it does not know and leaves room for [UNK], but its splits have more linguistic intuition.

What each suits

What scenarios suit sparse vectors?

For example:

1. A user talks with an LLM, while another LLM continuously summarizes facts and preferences from the conversation as memory.
2. A user wants to quickly capture every article related to cats, using the keyword 猫.

Which suits dense or sparse vectors?

Earlier, we mentioned that embedding a single term is possible, but normally it has too little semantic information for the vector to say anything useful and recall becomes difficult. Dense vectors are better at complex semantic expression; they handle complex query needs better, but can struggle with queries lacking semantics or context.

BM25 and TF-IDF, by contrast, are designed for term retrieval. In scenario 2, the answer is almost instant.

Korewaxnne

Most of your intuition is right. Let me calibrate a few points.

Does BM25 have its own vector space?

Yes. Every term is a dimension. The dimension corresponding to “猫” has a value, while others are zero. Its vector space is an extremely high-dimensional sparse space whose dimensions are vocabulary terms, perhaps 50,000 dimensions, while every vector has only a few nonzero values. That is why it counts as a sparse vector.

Which suits scenarios 1 and 2?

Scenario	Matching type	Better fit	Why
1. Conversation summaries → memories	Semantic matching: “the user likes cats” ≈ “they own a Ragdoll”	Dense vectors	Memories need to understand meaning rather than exact words. If a user says “I own a Ragdoll,” dense vectors can associate it with “cat”; sparse vectors cannot.
2. Search articles about “猫”	Keyword matching: title or text contains “猫”	Sparse vectors, BM25	The query is too short for a dense vector to say much; sparse vectors can hit the exact term immediately.

But there is an implicit question worth expanding: why do dense vectors struggle with short queries?

It is not because they are short, but because they lack context. One word is too ambiguous:

• “猫” might mean the animal, feline species, cat food, cat videos, Tom and Jerry…
• “一只黑猫蹲在窗台上看雨” tells the model the approximate scene.

The more room the model has to infer, the more accurate semantic matching becomes. A one-word query gives it no room, so its vector does not know which direction to approach.

How does production solve this? Through hybrid search.

Modern search engines almost never use one method only:

1
query → BM25 (exact match) + dense vector (semantic expansion) → fused ranking → return

BM25 handles “do what was said”; dense handles “reason by analogy.” Their scores are weighted together—BM25 gets higher weight for short queries, dense gets more for long queries. This is also the complement of related-article recommendation discussed in your TF-IDF note.

From this perspective, memU and mem0 are both hybrid scenarios.

They need dense embeddings and BM25 as a supplement. But how BM25 should be used has much to discuss. Is it simply the hybrid-search use shown above, or can it do something more interesting?

When breaking down memU, we can capture how it uses BM25: simple hybrid search, or its own extension—What Is memU? Let’s Break It Down.