From b474f835f92ee13d6ceb28a3018500e9c05fdda3 Mon Sep 17 00:00:00 2001 From: Leon Derczynski Date: Mon, 23 Sep 2024 17:41:54 +0200 Subject: [PATCH 1/2] add example of generator config using config_root --- docs/source/configurable.rst | 59 +++++++++++++++++++++++++----------- 1 file changed, 42 insertions(+), 17 deletions(-) diff --git a/docs/source/configurable.rst b/docs/source/configurable.rst index ef3f378a..ff6973d8 100644 --- a/docs/source/configurable.rst +++ b/docs/source/configurable.rst @@ -129,18 +129,8 @@ For an example of how to use the ``detectors``, ``generators``, ``buffs``, * ``show_100_pass_modules`` - Should entries scoring 100% still be detailed in the HTML report? -Using a custom JSON config -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Some plugins can take a JSON config specified on the command line. This config -has the same structure as a YAML config, starting with the plugin model/type. -The config can either be written to a file and the path passed, with -`--generator_option_file` or `--probe_option_file`, or directly as JSON on the -command prompt, with `--generator_options` or `--probe_options`. An example -is given in `RestGenerator Config with JSON `_ below. - -Examples: quick configs -^^^^^^^^^^^^^^^^^^^^^^^ +Bundled quick configs +^^^^^^^^^^^^^^^^^^^^^ Garak comes bundled with some quick configs that can be loaded directly using ``--config``. These don't need the ``.yml`` extension when being requested. They include: @@ -174,8 +164,21 @@ probes and run each prompt just once: If we save this as ``latent1.yaml`` somewhere, then we can use it with ``garak --config latent1.yaml``. -Plugins -------- + + +Using a custom JSON config +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Some plugins can take a JSON config specified on the command line. This config +has the same structure as a YAML config, starting with the plugin model/type. +The config can either be written to a file and the path passed, with +`--generator_option_file` or `--probe_option_file`, or directly as JSON on the +command prompt, with `--generator_options` or `--probe_options`. An example +is given in `RestGenerator Config with JSON `_ below. + + +Configuring Plugins +------------------- Garak's functions are through its plugins. Most parts of garak are plugins, like the ``probes`` and ``detectors`` that do the actual examination of the target, @@ -250,8 +253,8 @@ is an example that is equivalent to the configuration above: openai: temperature: 1.0 -RestGenerator -^^^^^^^^^^^^^ +Example: RestGenerator +^^^^^^^^^^^^^^^^^^^^^^ RestGenerator is a slightly complex generator, though mostly because it exposes so many config values, allowing flexible integrations. This example sets @@ -317,4 +320,26 @@ This defines a REST endpoint where: This should be written to a file, and the file's path passed on the command -line with `-G`. \ No newline at end of file +line with `-G`. + +Configuration in code +--------------------- + +The preferred way to instantiate a plugin is using ``garak._plugins.load_plugin()``. +This function takes two parameters: + +* ``name``, the plugin's package, module, and class - e.g. ``generator.test.Lipsum`` +* (optional) ``config_root``, either garak._config or a dictionary of a config, beginning at a top-level plugin type. + +``load_plugin()`` returns a configured instance of the requested plugin. + +OpenAIGenerator config with dictionary +"""""""""""""""""""""""""""""""""""""" + +.. code-block:: python + + >>> import garak._plugins + >>> c = {"model_type":"openai.OpenAIGenerator", "generators":{"openai":{"OpenAIGenerator":{"seed":30,"name":"gpt-4"}}}} + >>> garak._plugins.load_plugin("generators.openai.OpenAIGenerator", config_root=c) + 🦜 loading generator: OpenAI: gpt-4 + From 58ec9cdfbf9aec1a463efa91852452ca6bb7d995 Mon Sep 17 00:00:00 2001 From: Leon Derczynski Date: Mon, 23 Sep 2024 17:42:38 +0200 Subject: [PATCH 2/2] rm spurious top-level entry --- docs/source/configurable.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/configurable.rst b/docs/source/configurable.rst index ff6973d8..69294ef6 100644 --- a/docs/source/configurable.rst +++ b/docs/source/configurable.rst @@ -339,7 +339,7 @@ OpenAIGenerator config with dictionary .. code-block:: python >>> import garak._plugins - >>> c = {"model_type":"openai.OpenAIGenerator", "generators":{"openai":{"OpenAIGenerator":{"seed":30,"name":"gpt-4"}}}} + >>> c = {"generators":{"openai":{"OpenAIGenerator":{"seed":30,"name":"gpt-4"}}}} >>> garak._plugins.load_plugin("generators.openai.OpenAIGenerator", config_root=c) 🦜 loading generator: OpenAI: gpt-4