Documentation for FasterAI updates

CHANGELOG.md

@@ -11,5 +11,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
     - Find datasets and learning methods based on `Block`s: `finddataset`, `findlearningmethods`
     - `loaddataset` for quickly loading data containers from configured recipes
 - Data container recipes (`DatasetRecipe`, `loadrecipe`)
+- Documentation setions for FasterAI interfaces:
+    - [Discovery](https://fluxml.ai/FastAI.jl/dev/docs/discovery.md.html)
+    - [Blocks and encodings](https://fluxml.ai/FastAI.jl/dev/docs/background/blocksencodings.md.html)
 
-### Changed
+
+
+### Changed
+
+- Documentation sections to reference FasterAI interfaces:
+    - [README](https://fluxml.ai/FastAI.jl/dev/README.md.html)
+    - [Introduction](https://fluxml.ai/FastAI.jl/dev/docs/introduction.md.html)
+    - [Data containers](https://fluxml.ai/FastAI.jl/dev/docs/data_containers.md.html)
+    - Combined how-tos on training [into a single page](https://fluxml.ai/FastAI.jl/dev/notebooks/training.ipynb.html)

README.md

@@ -14,28 +14,15 @@ Pkg.add("FastAI")
 or try it out with this [Google Colab template](https://colab.research.google.com/gist/lorenzoh/2fdc91f9e42a15e633861c640c68e5e8).
 
 
-## Example
-
-Training an image classification model:
+As an example, here is how to train an image classification model:
 
 ```julia
 using FastAI
-path = datasetpath("imagenette2-160")
-data = Datasets.loadfolderdata(
-    path,
-    filterfn=isimagefile,
-    loadfn=(loadfile, parentname))
-classes = unique(eachobs(data[2]))
-method = BlockMethod(
-    (Image{2}(), Label(classes)),
-    (
-        ProjectiveTransforms((128, 128), augmentations=augs_projection()),
-        ImagePreprocessing(),
-        OneHot()
-    )
-)
-learner = methodlearner(method, data, Models.xresnet18(), ToGPU(), Metrics(accuracy))
+data, blocks = loaddataset("imagenette2-160", (Image, Label))
+method = ImageClassificationSingle(blocks)
+learner = methodlearner(method, data, Models.xresnet18(), ToGPU())
 fitonecycle!(learner, 10)
+plotpredictions(method, learner)
 ```
 
 Please read [the documentation](https://fluxml.github.io/FastAI.jl/dev) for more information and see the [setup instructions](docs/setup.md).

docs/background/blocksencodings.md

@@ -0,0 +1,154 @@
+# Blocks and encodings
+
+_Unstructured notes on blocks and encodings_
+
+## Blocks
+
+> A **block** describes the meaning of a piece of data in the context of a learning task.
+
+- For example, for supervised learning tasks, there is an input block and a target block and we want to learn to predict targets from inputs. Learning to predict a cat/dog label (`Label(["cat", "dog"])`) from 2D images (`Image{2}()`) is a supervised image classification task.
+- A block is not a piece of data itself. Instead it describes the meaning of a piece of data in a context. That a piece of data is a block can be checked using [`checkblock`]`(block, data)`. A piece of data for the `Label` block above needs to be one of the labels, so `checkblock(Label(["cat", "dog"]), "cat") == true`, but `checkblock(Label(["cat", "dog"]), "cat") == false`.
+- We can say that a data container is compatible with a learning method if every observation in it is a valid sample of the sample block of the learning method. The sample block for supervised tasks is `sampleblock = (inputblock, targetblock)` so `sample = getobs(data, i)` from a compatible data container implies that `checkblock(sampleblock, sample)`. This also means that any data stored in blocks must not depend on individual samples; we can store the names of possible classes inside the `Label` block because they are the same across the whole dataset.
+
+
+## Data pipelines
+
+We can use blocks to formalize the data processing pipeline.
+
+During **training** we want to create pairs of data `(x, y)` s.t. `output = model(x)` and `loss = lossfn(output, y)`. In terms of blocks that means `model` is a function `(x,) -> output` and the loss function maps `(outputblock, yblock) -> loss`. Usually, `(input, target) != (x, y)` and instead we have an encoding step that transforms a sample into representations suitable to train a model on, i.e. `encode :: sample -> (x, y)`.
+
+- For the above image classification example we have `sampleblock = (Image{2}(), Label(["cat", "dog"]))` but we cannot put raw images into a model and get out a class. Instead, the image is converted to an array that includes the color dimension and its values are normalized; and the class label is one-hot encoded. So `xblock = ImageTensor{2}()` and `yblock = OneHotTensor{0}`. Hence to do training, we need a sample encoding function `(Image{2}, Label) -> (ImageTensor{2}, OneHotTensor{0})`
+
+During **inference**, we have an input and want to use a trained model to predict a target, i.e. `input -> target`. The model is again a mapping `xblock -> outputblock`, so we can build the transformation with an encoding step that encodes the input and a decoding step that takes the model output back into a target. 
+
+This gives us
+> `(predict :: input -> target) = decodeoutput ∘ model ∘ encodeinput`
+> where 
+> - `(encodeinput :: input -> x)`
+> - `(model :: x -> y)`
+> - `(decodeoutput :: y -> target)`
+
+- In the classification example we have, written in blocks, `predict :: Image{2} -> Label` and hence `encodeinput :: Image{2} -> ImageTensor{2}` and `decodeoutput :: OneHotTensor{0} -> Label`
+
+Where do we draw the line between model and data processing? In general, the encoding and decoding steps are **non-learnable** transformations, while the model is a **learnable** transformation.
+
+## Encodings
+
+> **Encodings** are reversible transformations that model the non-learnable parts (encoding and decoding) of the data pipeline.
+
+- What an encoding does depends on what block is passed in. Most encodings only transform specific blocks. For example, the [`ImagePreprocessing`](#) encoding maps blocks `Image{N} -> ImageTensor{N}`, but leaves other blocks unchanged. Encodings are called with `encode` and `decode` which take in the block and the data. The actual encoding and decoding takes in an additional context argument which can be specialized on to implement different behavior for e.g. training and validation.
+    {cell=main}
+    ```julia
+    using FastAI, Colors
+    using FastAI: ImageTensor
+    enc = ImagePreprocessing()
+    data = rand(RGB, 100, 100)
+    @show summary(data)
+    encdata = encode(enc, Training(), Image{2}(), data)
+    @show summary(encdata)  # (h, w, ch)-image tensor
+    data_ = decode(enc, Training(), ImageTensor{2}(3), encdata)
+    ```
+- Using an encoding to encode and then decode must be block-preserving, i.e. if, for an encoding, `encode :: Block1 -> Block2` then `decode :: Block2 -> Block1`. To see the resulting block of applying an encoding to a block, we can use [`encodedblock`](#) and [`decodedblock`](#).
+    {cell=main}
+    ```julia
+    using FastAI: encodedblock, decodedblock
+    enc = ImagePreprocessing()
+    @show encodedblock(enc, Image{2}())
+    @show decodedblock(enc, ImageTensor{2}(3))
+    Image{2}() == decodedblock(enc, encodedblock(enc, Image{2}()))
+    ```
+    You can use [`testencoding`](#) to test these invariants to make sure an encoding is implemented properly for a specific block.
+    {cell=main}
+
+    ```julia
+    FastAI.testencoding(enc, Image{2}())
+    ```
+- The default implementations of `encodedblock` and `decodedblock` is to return `nothing` indicating that it doesn't transform the data. This is overwritten for blocks for which `encode` and `decode` are implemented to indicate that the data is transformed. Using `encodedblock(block, data, true)` will replace returned `nothing`s with the unchanged block.
+    {cell=main}
+    ```julia
+    encodedblock(enc, Label(1:10)) === nothing
+    ```
+    {cell=main}
+    ```julia
+    encodedblock(enc, Label(1:10), true) == Label(1:10)
+    ```
+- Encodings can be applied to tuples of blocks. The default behavior is to apply the encoding to each block separately.
+    {cell=main}
+    ```julia
+    encodedblock(enc, (Image{2}(), Image{2}()))
+    ```
+
+- Applying a tuple of encodings will encode the data by applying one encoding after the other. When decoding, the order is reversed.
+
+## Block learning methods
+
+[`BlockMethod`](#) creates a learning method from blocks and encodings. You define the sample block (recall for supervised tasks this is a tuple of input and target) and a sequence of encodings that are applied to all blocks.
+
+The below example defines the same learning method as [`ImageClassificationSingle`](#) does. The first two encodings only change `Image`, and the last changes only `Label`, so it's simple to understand.
+
+{cell=main}
+```julia
+method = BlockMethod(
+    (Image{2}(), Label(["cats", "dogs"])),
+    (
+        ProjectiveTransforms((128, 128)),
+        ImagePreprocessing(),
+        OneHot(),
+    )
+)
+```
+
+Now `encode` expects a sample and just runs the encodings over that, giving us an encoded input `x` and an encoded target `y`.
+
+{cell=main}
+```julia
+data = loadfolderdata(joinpath(datasetpath("dogscats"), "train"), filterfn=isimagefile, loadfn=(loadfile, parentname))
+sample = getobs(data, 1)
+x, y = encode(method, Training(), sample)
+summary(x), summary(y)
+```
+
+This is equivalent to:
+
+{cell=main}
+```julia
+x, y = encode(method.encodings, Training(), method.blocks, sample)
+summary(x), summary(y)
+```
+
+Image segmentation looks almost the same except we use a `Mask` block as target. We're also using `OneHot` here, because it also has an `encode` method for `Mask`s. For this method, `ProjectiveTransforms` will be applied to both the `Image` and the `Mask`, using the same random state for cropping and augmentation.
+
+{cell=main}
+```julia
+method = BlockMethod(
+    (Image{2}(), Mask{2}(1:10)),
+    (
+        ProjectiveTransforms((128, 128)),
+        ImagePreprocessing(),
+        OneHot(),
+    )
+)
+```
+
+The easiest way to understand how encodings are applied to each block is to use [`describemethod`](#) and [`describeencodings`](#) which print a table of how each encoding is applied successively to each block. Rows where a block is **bolded** indicate that the data was transformed by that encoding.
+
+{cell=main}
+```julia
+describemethod(method)
+```
+
+The above tables make it clear what happens during training ("encoding a sample") and inference (encoding an input and "decoding an output"). The more general form [`describeencodings`](#) takes in encodings and blocks directly and can be useful for building an understanding of how encodings apply to some blocks.
+
+{cell=main}
+```julia
+FastAI.describeencodings(method.encodings, (Image{2}(),))
+```
+
+{cell=main}
+```julia
+FastAI.describeencodings((OneHot(),), (Label(1:10), Mask{2}(1:10), Image{2}()))
+```
+
+Notes
+
+- Since most encodings just operate on a small number of blocks and keep the rest unchanged, applying them to all blocks is usually not a problem. When it is because you want some encoding to apply to a specific block only, you can use [`Named`](#) and [`Only`](#) to get around it.

docs/data_containers.md

@@ -13,47 +13,35 @@ ENV["DATADEPS_ALWAYS_ACCEPT"] = "true"
 {cell=main}
 ```julia
 using FastAI
-
-NAME = "imagenette2-160"
-dir = datasetpath(NAME)
-data = Datasets.loadfolderdata(
-    dir,
-    filterfn=isimagefile,
-    loadfn=(loadfile, parentname))
+data, _ = loaddataset("imagenette2-160", (Image, Label))
 ```
 
-A data container is any type that holds observations of data and allows us to load them with `getobs` and query the number of observations with `nobs`:
+A data container is any type that holds observations of data and allows us to load them with `getobs` and query the number of observations with `nobs`. In this case, each observation is a tuple of an image and the corresponding class; after all, we want to use it for image classification. 
 
 {cell=main}
 ```julia
-obs = getobs(data, 1)
+image, class = obs = getobs(data, 1)
+@show class
+image
 ```
 
 {cell=main}
 ```julia
 nobs(data)
 ```
 
-In this case, each observation is a tuple of an image and the corresponding class; after all, we want to use it for image classification. 
+[`loaddataset`](#) makes it easy to a load a data container that is compatible with some block types, but to get a better feel for what it does, let's look under the hood by creating the same data container using some mid-level APIs.
+
+## Creating data containers from files
+
+Before we recreate the data container, [`datasetpath`](#) downloads a dataset and returns the path to the extracted files.
 
 {cell=main}
 ```julia
-image, class = obs
-@show class
-image
+dir = datasetpath("imagenette2-160")
 ```
 
-As you saw above, the `Datasets` submodule provides functions for loading and creating data containers. We used [`Datasets.datasetpath`](#) to download a dataset if it wasn't yet and get the folder it was downloaded to. Then, [`loadfolderdata`](#) took the folder and loaded a data container suitable for image classification. FastAI.jl makes it easy to download the datasets from fastai's collection on AWS Open Datasets. For the full list, see `Datasets.DATASETS`.
-
-
-### Exercises
-
-1. Have a look at the other image classification datasets in [`Datasets.DATASETS_IMAGECLASSIFICATION`](#) and change the above code to load a different dataset.
-
-
-## Creating data containers from files
-
-Now let's create the same data container, but using more general functions FastAI.jl provides to get a look behind the scenes. We'll start with [`FileDataset`](#) which creates a data container (here a `Vector`) of files given a path. We'll use the path of the downloaded dataset:
+Now we'll start with [`FileDataset`](#) which creates a data container (here a `Vector`) of files given a path. We'll use the path of the downloaded dataset:
 
 {cell=main}
 ```julia
@@ -93,7 +81,8 @@ data = mapobs(loadimageclass, files);
 
 ### Exercises
 
-1. Using `mapobs` and `loadfile`, create a data container where every observation is only an image.
+1. Using [`mapobs`](#) and [`loadfile`](#), create a data container where every observation is only an image.
+2. Change the above code to run on a different dataset from the list in `Datasets.DATASETS_IMAGECLASSIFICATION`.
 
 
 ## Splitting a data container into subsets
@@ -133,3 +122,16 @@ trainfiles, validfiles = datagroups["train"], datagroups["val"]
 ```
 
 Using this official split, it will be easier to compare the performance of your results with those of others'. 
+
+
+## Dataset recipes
+
+We saw above how different image classification datasets can be loaded with the same logic as long as they are in a common format. To encapsulate the logic for loading common dataset formats, FastAI.jl has `DatasetRecipe`s. When we used [`finddatasets`](#) in the [discovery tutorial](discovery.md), it returned pairs of a dataset name and a `DatasetRecipe`. For example, `"imagenette2-160"` has an associated [`ImageFolders`](#) recipe and we can load it using [`loadrecipe`] and the path to the downloaded dataset:
+
+{cell=main}
+```julia
+name, recipe = finddatasets(blocks=(Image, Label), name="imagenette2-160")[1]
+data, blocks = loadrecipe(recipe, datasetpath(name))
+```
+
+These recipes also take care of loading the data block information for the dataset. Read the [discovery tutorial](discovery.md) to find out more about that.

docs/discovery.md

@@ -0,0 +1,81 @@
+# Discovery
+
+As you may have seen in [the introduction](./introduction.md), FastAI.jl makes it possible to train models in just 5 lines of code. However, if you have a task in mind, you need to know what datasets you can train on and if there are convenience learning method constructors. For example, the introduction loads the `"imagenette2-160"` dataset and uses [`ImageClassificationSingle`](#) to construct a learning method. Now what if, instead of classifying an image into one class, we want to classify every single pixel into a class (semantic segmentation)? Now we need a dataset with pixel-level annotations and a learning method that can process those segmentation masks.
+
+For finding both, we can make use of `Block`s. A `Block` represents a kind of data, for example images, labels or keypoints. For supervised learning tasks, we have an input block and a target block. If we wanted to classify whether 2D images contain a cat or a dog, we could use the blocks `(Image{2}(), Label(["cat", "dog"]))`, while for semantic segmentation, we'll have an input `Image` block and a target [`Mask`](#) block.
+
+## Finding a dataset
+
+To find a dataset with compatible samples, we can pass the types of these blocks to [`finddatasets`](#) which will return a list of dataset names and recipes to load them in a suitable way.
+
+{cell=main}
+```julia
+using FastAI
+finddatasets(blocks=(Image, Mask))
+```
+
+We can see that the `"camvid_tiny"` dataset can be loaded so that each sample is a pair of an image and a segmentation mask. Let's use [`loaddataset`](#) to load a [data container](data_containers.md) and concrete blocks.
+
+{cell=main}
+```julia
+data, blocks = loaddataset("camvid_tiny", (Image, Mask))
+```
+
+As with every data container, we can load a sample using `getobs` which gives us a tuple of an image and a segmentation mask.
+
+{cell=main}
+```julia
+image, mask = sample = getobs(data, 1)
+size.(sample), eltype.(sample)
+```
+
+`loaddataset` also returned `blocks` which are the concrete `Block` instances for the dataset. We passed in _types_ of blocks (`(Image, Mask)`) and get back _instances_ since the specifics of some blocks depend on the dataset. For example, the returned target block carries the labels for every class that a pixel can belong to.
+
+{cell=main}
+```julia
+inputblock, targetblock = blocks
+targetblock
+```
+
+With these `blocks`, we can also validate a sample of data using [`checkblock`](#) which is useful as a sanity check when using custom data containers.
+
+{cell=main}
+```julia
+checkblock((inputblock, targetblock), (image, mask))
+```
+
+### Summary
+
+In short, if you have a learning task in mind and want to load a dataset for that task, then
+
+1. define the types of input and target block, e.g. `blocktypes = (Image, Label)`,
+2. use [`finddatasets`](#)`(blocks=blocktypes)` to find compatbile datasets; and
+3. run [`loaddataset`](#)`(datasetname, blocktypes)` to load a data container and the concrete blocks
+
+### Exercises
+
+1. Find and load a dataset for multi-label image classification. (Hint: the block for multi-category outputs is called `LabelMulti`).
+2. List all datasets with `Image` as input block and any target block. (Hint: the supertype of all types is `Any`)
+
+
+## Finding a learning method
+
+Armed with a dataset, we can go to the next step: creating a learning method. Since we already have blocks defined, this amounts to defining the encodings that are applied to the data before it is used in training. Here, FastAI.jl already defines some convenient constructors for learning methods and you can find them with [`findlearningmethods`](#). Here we can pass in either block types as above or the block instances we got from `loaddataset`.
+
+{cell=main}
+```julia
+findlearningmethods(blocks)
+```
+
+Looks like we can use the [`ImageSegmentation`](#) function to create a learning method for our learning task. Every function returned can be called with `blocks` and, optionally, some keyword arguments for customization.
+
+{cell=main}
+```julia
+method = ImageSegmentation(blocks; size = (64, 64))
+```
+
+And that's the basic workflow for getting started with a supervised task.
+
+### Exercises
+
+1. Find all learning method functions with images as inputs.