infra: add CI job for running tutorial notebooks #26944
Conversation
dosubot
bot
added
the
size:XXL
This reverts commit 67312fe.
dosubot
bot
added
size:M
| - cron: '0 13 * * *' | ||
|
|
||
| env: | ||
| PYTHON_VERSION: "3.11" |
There was a problem hiding this comment.
might be worth parameterizing this to make sure they run in 3.9 + 3.12 mostly
There was a problem hiding this comment.
Will do separately as it's easier to debug workflow dispatch after merging the new workflow to master.
There was a problem hiding this comment.
Added python version as a parameter in #26956
.github/workflows/run_notebooks.yml
Outdated
| env: | ||
| # these won't actually be used because of the VCR cassettes | ||
| # but need to set them to avoid triggering getpass() | ||
| ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} | ||
| MISTRAL_API_KEY: ${{ secrets.MISTRAL_API_KEY }} | ||
| OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} | ||
| TAVILY_API_KEY: ${{ secrets.TAVILY_API_KEY }} | ||
| run: | | ||
| ./docs/scripts/execute_notebooks.sh |
There was a problem hiding this comment.
if we know they won't be needed, could we skip populating the real secrets and instead just run export ANTHROPIC_API_KEY=fake-api-key commands?
There was a problem hiding this comment.
Deleted this comment as these actually do get used for notebooks where we skip cassettes.
|
|
||
| NOTEBOOKS_NO_EXECUTION = [ | ||
| "docs/docs/tutorials/graph.ipynb", # Requires local graph db running | ||
| "docs/docs/tutorials/local_rag.ipynb", # Local LLMs |
There was a problem hiding this comment.
ooc why can't this be run off a cassette?
There was a problem hiding this comment.
Local LLM notebook I think may not make http calls, there's just significant overhead running local models (e.g., ollama). LMK if I've misunderstood.
docs/scripts/execute_notebooks.sh
Outdated
| file="$1" | ||
| echo "Starting execution of $file" | ||
| start_time=$(date +%s) | ||
| if ! output=$(time poetry run jupyter execute "$file" 2>&1); then |
There was a problem hiding this comment.
might be worth switching to jupyter nbconvert --to notebook --execute $file for later customizability if we ever want to process the outputs / look at the file
doesn't need to block this PR - this looks like it only gets errors from the jupyter runtime and might be tricky to get warnings/incorrect outputs from cells?
| export -f execute_notebook | ||
|
|
||
| # Find all notebooks and filter out those in the skip list | ||
| notebooks=$(find docs/docs/tutorials -name "*.ipynb" | grep -v ".ipynb_checkpoints" | grep -vFf <(echo "$SKIP_NOTEBOOKS")) |
There was a problem hiding this comment.
if there's a way to parameterize this, might make this CI step more customizable in future if we want to use it to run e.g. individual notebooks
There was a problem hiding this comment.
Will look into this after merging so we can debug workflow dispatch.
dosubot
bot
added
the
lgtm
Addressing some lingering comments from #26944, adding parameters for - python version - working directory 
No description provided.