This is a React-based curve editor GUI for prompt interpolation animations made with Stable Diffusion!
Since that sentence is a bit of a mouthful, how about I just show you how it works?
Users populate a "prompt graph", where the X-axis represents the progression of an animation, and the Y-axis
represents the relative "strength" of a prompt. Each curve in the graph corresponds with one of the user's prompts.
After crafting a Prompt Graph, a series of Stable Diffusion generations can be launched. The resulting images will correspond with the interpolation you defined in your graph:
This repo is a clone of v0.9 of basujindal's stable-diffusion fork! I picked this fork as a basis for my Prompt Graph app, since it contains a ton of optimizations that allow Stable Diffusion to be run on my local GPU (NVIDIA 2070 Super). In the future, I could easily swap out the "engine" Prompt Graph is using - by the time of publishing this, there'll probably be more insane optimizations to make things even faster. 😉
In order to install this app, you'll need to install a couple of pre-requesites:
-
node.js - Since this is a React app, you'll need to download
node.jsin order to have access tonpm. -
conda - Conda is a package management system that you'll use to install the right Python packages.
-
ffmpeg - FFMPEG is a command-line tool that enables a lot of video-related functionality. It's what I use to create the .mp4 files of the animations
Once you've downloaded both of these, you should move into setting up Stable Diffusion. The first thing you ought to do is actually download the weights for the model. I've used Stable Diffusion v1.4 successfully, but there might be newer models by the time you're reading this.
After the model is downloaded, you ought to rename the .ckpt file to model.ckpt, and then place it into the stable-diffusion/models/ldm/stable-diffusion-v1 folder.
Once you've placed the model into the correct folder, you ought to set up the Stable Diffusion conda environment. To set up this environment, run these commands from the root of the repo:
cd stable-diffusion/
conda env create -f environment.yaml
conda activate ldm
pip install -r ./requirements.txt --exists-action=wOnce this is finished, you'll need to install all of the dependencies for the React app. To do that, run these commands from the repo root:
cd prompt-graph/
npm install After this point, you should be all set up! Check out the Usage section below to learn more about how to actually run the app.
In order to run the React app, you can run the following commands from the root of the repo:
cd prompt-graph/
npm start
This will start a server running the React app at http://localhost:3000.
From there, you can specify the Prompt Graph that you want to create. (More detailed documentation / a video tutorial on how to use the app is to come!)
Once you're finished defining your prompt graph, click on the Generate Configuration File button. This will download a .json file containing all of the prompts you'd specified.
Finally, you should be able to run
python run_generation_from_config.py
from the root of the repo. This will ask you for the path of the configuration file you want to generate; once you paste in the path and hit Enter, then the generation will begin!
If you're having any trouble using the app, please reach out to me - either through a DM on Twitter, or by starting an issue on this repo!
I'm actively working on this repo, so I'd expect that things ought to change pretty radically in the future. As follows is a list of some of the things I'm actively working on:
- Simplifying installation process
- CLIP-guided recursive latent interpolation (to smooth transitions between images)
- Allowing users to load and edit already-created Prompt Graph configuration files
- Adding additional polish to the user interface & fixing bugs
- Allowing initial images to be used within the Prompt Graph
- Development of a Dash version of the React application (which'll allow for in-browser use via Google Colab)
- Better curve-editing capabilities / controls (i.e., non-linear curves)