Tokenization notebook (#173)

* Tokenization notebook * remove comment * exclude special tokens before decoding * improve descriptions * add overlap; remove text file * run nbtest through Makefile * reformulate recommendation * remove paragraph I wanted to omit * move to `document-chunking` folder
elastic · Jan 26, 2024 · 5369ab6 · 5369ab6
1 parent 5264994
commit 5369ab6
Show file tree

Hide file tree

Showing 2 changed files with 330 additions and 0 deletions.
diff --git a/notebooks/document-chunking/Makefile b/notebooks/document-chunking/Makefile
@@ -1,5 +1,6 @@
 NBTEST = ../../bin/nbtest
 NOTEBOOKS = \
+	tokenization.ipynb \
 	with-index-pipelines.ipynb \
 	with-langchain-splitters.ipynb