NLP

My plain-language explanation of TF-IDF, from terms and tokenization through TF, IDF, query scoring, BM25, and using it to recommend related posts on this blog. Apparently, it is much easier to pay back the debt of skipping NLP lectures once there is a reason to learn it.

Starting from the question of what sparse and dense really mean, this works through the embedding pipeline, subwords and vocabularies, input length, and their practical uses.

A discussion inspired by MoeChat about RRF and Hybrid Search: how should time retrieval participate in hybrid retrieval, how should it adapt across scenarios, and where are the boundaries between wikimem and XnneHangLab?