Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

configurable should validate dict format expectations #926

Open
leondz opened this issue Sep 23, 2024 · 1 comment
Open

configurable should validate dict format expectations #926

leondz opened this issue Sep 23, 2024 · 1 comment
Assignees
Labels
enhancement Architectural upgrades

Comments

@leondz
Copy link
Owner

leondz commented Sep 23, 2024

Summary

garak._plugins.load_plugin() and Configurable expect to be passed either _config or a dict with a required format. Currently _load_config() does not validate a dict object meets the format requirements and error messages are unclear.

Basic example

import garak._plugins
c = {"generators":{'openai': {'name': 'gpt-4'}}}
g = _plugins.load_module("generators.openai", config_root=c)

Here is an example invalid config using plugins as the top level key instead of generators:

>>> c = {"plugins":{"generators":{'openai': {'name': 'gpt-4'}}}}
>>> import garak
>>> import garak._plugins
>>> g = garak._plugins.load_plugin("generators.openai", config_root=c)
WARNING:root:Exception instantiating garak.generators.openai.OpenAIGenerator: 'dict' object has no attribute 'plugins'
Traceback (most recent call last):
  File "/home/lderczynski/dev/garak/garak/_plugins.py", line 390, in load_plugin
    plugin_instance = klass(config_root=config_root)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lderczynski/dev/garak/garak/generators/openai.py", line 315, in __init__
    super().__init__(self.name, config_root=config_root)
  File "/home/lderczynski/dev/garak/garak/generators/openai.py", line 158, in __init__
    print(config_root.plugins.generators)
          ^^^^^^^^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute 'plugins'
Traceback (most recent call last):
  File "/home/lderczynski/dev/garak/garak/_plugins.py", line 390, in load_plugin
    plugin_instance = klass(config_root=config_root)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lderczynski/dev/garak/garak/generators/openai.py", line 315, in __init__
    super().__init__(self.name, config_root=config_root)
  File "/home/lderczynski/dev/garak/garak/generators/openai.py", line 158, in __init__
    print(config_root.plugins.generators)
          ^^^^^^^^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute 'plugins'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/lderczynski/dev/garak/garak/_plugins.py", line 400, in load_plugin
    raise GarakException(e) from e
garak.exception.GarakException: 'dict' object has no attribute 'plugins'

Motivation

This supports simpler maintenance and loading of generator instantiation, by validating the input in one location.

We also have some difficulty loading generators from places other than garak.cli, depending on the generator; improving this makes probes and detectors more powerful in the range of what generators they can use. For example, LLMaaJ #419 works well with GPT-4, but writing this in a detector means some logic duplication.

@leondz leondz added the enhancement Architectural upgrades label Sep 23, 2024
@jmartin-tech
Copy link
Collaborator

Let refactor this to clarify format for a configurable object when parsing config_root as a dict.

@jmartin-tech jmartin-tech self-assigned this Sep 23, 2024
@jmartin-tech jmartin-tech changed the title load_plugin should support generators configurable should validate dict format expectations Sep 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Architectural upgrades
Projects
None yet
Development

No branches or pull requests

2 participants