Information Theorymath
The mathematical study of information, entropy, and compression — how much can be encoded, and how surprising is an event.
TF-IDF, entropy-based pruning, and embedding information content all draw on information theory; it underlies why some terms are more informative than others.
Where it appears
- SKCE foundation for embeddings
- rag
- vector-search
Descend
Historical evolution
Shannon (1948) defined entropy; the basis of coding, retrieval weighting, and compression.
Implementation details
Upstream KeywordExtractorPass uses smoothed IDF: log((N+1)/(df+1))+1.
Sources
- [2026-07-11-knowledge-compiler-compiling-human-knowledge-into-static-semantic-artifacts] 2026-07-11-knowledge-compiler-compiling-human-knowledge-into-static-semantic-artifacts
- [2026-07-11-knowledge-compiler-compiling-human-knowledge-into-static-semantic-artifacts] 2026-07-11-knowledge-compiler-compiling-human-knowledge-into-static-semantic-artifacts