compatibility issue with latest gym (0.26.1) and stable -baseline3 (2.0.0)[

[Ganesh-mali]

Issue Description
I am encountering an error while running the code provided in the official CityLearn documentation. I have not modified a single line of the code, and I'm using the exact code snippet provided.

Expected Behavior
I expected the code to run without errors, as it's directly taken from the official CityLearn documentation.

Actual Behavior
I am facing the following error:
ValueError: not enough values to unpack (expected 2, got 1)

Steps to Reproduce
Install CityLearn (version 2.0b4), Stable Baselines 3 (version 2.0.0), and Gym (version 0.26.1).
Run the code provided in the official CityLearn documentation:(https://www.citylearn.net/quickstart.html)

Code:
from stable_baselines3.sac import SAC
from citylearn.citylearn import CityLearnEnv
from citylearn.wrappers import NormalizedObservationWrapper, StableBaselines3Wrapper

dataset_name = 'baeda_3dem'
env = CityLearnEnv(dataset_name, central_agent=True, simulation_end_time_step=1000)
env = NormalizedObservationWrapper(env)
env = StableBaselines3Wrapper(env)
model = SAC('MlpPolicy', env)
model.learn(total_timesteps=env.time_steps*2)

evaluate

observations = env.reset()

while not env.done:
actions, _ = model.predict(observations, deterministic=True)
observations, _, _, _ = env.step(actions)

kpis = env.evaluate().pivot(index='cost_function', columns='name', values='value')
kpis = kpis.dropna(how='all')
display(kpis)

Environment
CityLearn version: 2.0b4
Operating System: Windows
Python version: 3.10

Possible Solution
I have tried various solutions and referred to the official documentation, but I am unable to find a compatible version combination that resolves this issue. I'm looking for guidance from the community.

Additional Notes
I'm following the instructions exactly as provided in the official documentation, so I'm puzzled as to why I'm encountering this issue. If anyone has experience with these libraries and can provide guidance or suggestions, I would greatly appreciate it.

Thank you for your time and assistance!

[kingsleynweye]

Hi @Ganesh-mali, there is a related issue to reference for this #63. You need to downgrade your stable-baselines3 installation, preferably to 1.7.0 or 1.8.0. CityLearn is still a gym environment whereas later stable-baselines3 releases support gymnasium environments.

[Ganesh-mali]

hey @kingsleynweye thank you for your reply, I tried downgrading to both 1.7.0 as well as 1.8.0. I get new error with that :

Building wheels for collected packages: gym
Building wheel for gym (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [477 lines of output]

Let me know , if you need more info.

[kingsleynweye]

@Ganesh-mali this error is a recently known gym bug for version 0.21. I have been able to bypass it in the past by downgrading pip and setuptools before installing CityLearn and stable-baselines3:

pip install pip==21.3.1
pip install setuptools==41.2.0
pip install CityLearn==2.0b4
pip install stable-baselines3==1.7.0

[kingsleynweye]

@Ganesh-mali hmmnn I have not seen this before. If I understand correctly, it is the stable-baselines3 installation throwing this new error? Also, looks like you are having issues with anaconda virtual environments. I am not very familiar with using anaconda as I typically use venv to manage virtual environments.

[Ganesh-mali]

hey @kingsleynweye , this time I created a virtual environment using Venv and repeated your steps, Again same error, the first three ran successfully and the installation of stable-baseline3 failed again.

Installing collected packages: zipp, gym, importlib-metadata, stable-baselines3
  Attempting uninstall: gym
    Found existing installation: gym 0.26.2
    Uninstalling gym-0.26.2:
      Successfully uninstalled gym-0.26.2
  DEPRECATION: gym is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for gym ... error
  error: subprocess-exited-with-error

  × Running setup.py install for gym did not run successfully.
  │ exit code: 1
  ╰─> [573 lines of output]
      running install
      running build
      running build_py
      creating build

  This above message followed some 200 line output then it showed :

      , line 2449, in resolve
          module = __import__(self.module_name, fromlist=['__name__'], level=0)
        File "C:\COURSES_LEARNINGS\Rl_learning\city_learn\city_test\lib\site-packages\setuptools\command\bdist_wininst.py", line 1, in <module>
          import distutils.command.bdist_wininst as orig
      ModuleNotFoundError: No module named 'distutils.command.bdist_wininst'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  Rolling back uninstall of gym
  Moving to c:\courses_learnings\rl_learning\city_learn\city_test\lib\site-packages\gym-0.26.2.dist-info\
   from C:\COURSES_LEARNINGS\Rl_learning\city_learn\city_test\Lib\site-packages\~ym-0.26.2.dist-info
  Moving to c:\courses_learnings\rl_learning\city_learn\city_test\lib\site-packages\gym\
   from C:\COURSES_LEARNINGS\Rl_learning\city_learn\city_test\Lib\site-packages\~ym
error: legacy-install-failure

If you need more info please let me know. Or If you have any other suggestions, I will follow your guidelines I just want to run the code that is on Citylearn documentation.

[Ganesh-mali]

Hi @kingsleynweye

I have an update, I have installed all for packages as you said, After running the code I am getting this error now :

C:\Users\anaconda3\envs\new_city_test_env\lib\site-packages\torch\nn\modules\rnn.py:89: UserWarning: Failed to initialize NumPy: module compiled against API version 0x10 but this version of numpy is 0xe (Triggered internally at  ..\torch\csrc\utils\tensor_numpy.cpp:68.)
  w_ih = Parameter(torch.empty((gate_size, layer_input_size), **factory_kwargs))
Traceback (most recent call last):
  File "C:\COURSES_LEARNINGS\Rl_learning\city_learn\p1-city_learn.py", line 12, in <module>
    model.learn(total_timesteps=env.time_steps*20)
  File "C:\Users\anaconda3\envs\new_city_test_env\lib\site-packages\stable_baselines3\sac\sac.py", line 299, in learn
    return super().learn(
  File "C:\Users\\anaconda3\envs\new_city_test_env\lib\site-packages\stable_baselines3\common\off_policy_algorithm.py", line 323, in learn
    total_timesteps, callback = self._setup_learn(
  File "C:\Users\anaconda3\envs\new_city_test_env\lib\site-packages\stable_baselines3\common\off_policy_algorithm.py", line 305, in _setup_learn
    return super()._setup_learn(
  File "C:\Users\anaconda3\envs\new_city_test_env\lib\site-packages\stable_baselines3\common\base_class.py", line 408, in _setup_learn
    self._last_obs = self.env.reset()  # pytype: disable=annotation-type-mismatch
  File "C:\Users\anaconda3\envs\new_city_test_env\lib\site-packages\stable_baselines3\common\vec_env\dummy_vec_env.py", line 74, in reset
    obs = self.envs[env_idx].reset()
  File "C:\Users\anaconda3\envs\new_city_test_env\lib\site-packages\stable_baselines3\common\monitor.py", line 83, in reset
    return self.env.reset(**kwargs)
  File "C:\Users\anaconda3\envs\new_city_test_env\lib\site-packages\gym\core.py", line 323, in reset
    return self.env.reset(**kwargs)
  File "C:\Users\anaconda3\envs\new_city_test_env\lib\site-packages\gym\core.py", line 379, in reset
    obs, info = self.env.reset(**kwargs)
  File "C:\Users\anaconda3\envs\new_city_test_env\lib\site-packages\gym\core.py", line 323, in reset
    return self.env.reset(**kwargs)
  File "C:\Users\anaconda3\envs\new_city_test_env\lib\site-packages\gym\core.py", line 323, in reset
    return self.env.reset(**kwargs)
  File "C:\Users\anaconda3\envs\new_city_test_env\lib\site-packages\gym\core.py", line 379, in reset
    obs, info = self.env.reset(**kwargs)
ValueError: not enough values to unpack (expected 2, got 1)

[Ganesh-mali]

Hey @kingsleynweye any update on this ?

[H-Park]

This is indeed a bug. The library needs to depend on gymnasium, not gym. The OpenAI's gym package hasn't been maintained in years, and a independent developer took it upon themself to revive it and add features the RL community wanted, and in the process releases it on pypi as gymnasium to avoid issues.

[kingsleynweye]

@Ganesh-mali this part of the error:

obs, info = self.env.reset(**kwargs)
ValueError: not enough values to unpack (expected 2, got 1)

makes me believe the issue is caused by your installation of stable-baselines3 expecting a gymnasium environment where the reset function returns a tuple of two values. In line with what @H-Park mentioned, CityLearn is a gym environment and gym environments' step function returns a single value, obs.

To avoid this problem, you need to use a stable-baselines3 version compatible with gym (I think any stable-baselines3<=1.8.0).

[kingsleynweye]

This is indeed a bug. The library needs to depend on gymnasium, not gym. The OpenAI's gym package hasn't been maintained in years, and a independent developer took it upon themself to revive it and add features the RL community wanted, and in the process releases it on pypi as gymnasium to avoid issues.

@H-Park yeah, ideally, we would upgrade CityLearn to gymnasium and it is something on the list but not quite a priority yet :( . If this is a fix you will want to see sooner and have to know-how, please feel free to contribute as CityLearn is open source and I think migrating to gymnasium is a very welcomed community contribution through a pull request.

[kingsleynweye]

@Ganesh-mali this part of the error:

obs, info = self.env.reset(**kwargs)
ValueError: not enough values to unpack (expected 2, got 1)

makes me believe the issue is caused by your installation of stable-baselines3 expecting a gymnasium environment where the reset function returns a tuple of two values. In line with what @H-Park mentioned, CityLearn is a gym environment and gym environments' step function returns a single value, obs.

To avoid this problem, you need to use a stable-baselines3 version compatible with gym (I think any stable-baselines3<=1.8.0).

Forgot to mention that another way I have been able to solve this problem is by installing shimmy that provides a gymnasium binder.

[Ganesh-mali]

@kingsleynweye the dataset given in the example code "baeda_3dem" doesn't work, I even checked using the method Datset.get_names(). It printed all available datasets except baeda_3dem. Anyways, it is working with other datasets, I am able to run the code with SB3 (1.7.0), and gym (0.21.0). Thanks for all your suggestions.

[kingsleynweye]

@Ganesh-mali, if baeda_3dem is not getting printed then I suspect you are using a citylearn version that is <=1.8.0. It was only added after 1.8.0 It prints out for me for later versions.

[ravikiranbadathala]

This is the new issue with running the code, i am not sure...if any of these examples can be run as it is
can you please point me where i can run at least one full example on this...
Thanks

130 if not deterministic:
--> 131 self.update(observations, actions, rewards, next_observations, done=self.env.done)
132 else:
133 pass

TypeError: update() got an unexpected keyword argument 'done'

[kingsleynweye]

This is the new issue with running the code, i am not sure...if any of these examples can be run as it is can you please point me where i can run at least one full example on this... Thanks

130 if not deterministic: --> 131 self.update(observations, actions, rewards, next_observations, done=self.env.done) 132 else: 133 pass

TypeError: update() got an unexpected keyword argument 'done'

What examples are you referring to? With the current code, I was able to run all sections of quickstart.ipynb but Stable Baselines3 Reinforcement Learning Algorithms section in Google Colab. I will push a fix for Stable Baselines3 example soon.

[ravikiranbadathala]

https://github.com/intelligent-environments-lab/CityLearn/blob/master/examples/citylearn_rlem23_tutorial.ipynb

----------------- CALCULATE NUMBER OF TRAINING EPISODES -----------------

i = 3
m = tql_env.observation_space[0].n
n = tql_env.action_space[0].n
t = tql_env.time_steps - 1
tql_episodes = mni/t
tql_episodes = int(tql_episodes)
print('Q-Table dimension:', (m, n))
print('Number of episodes to train:', tql_episodes)

------------------------------- SET LOADER ------------------------------

loader = get_loader(max=tql_episodes*t)
display(loader)

----------------------- SET MODEL HYPERPARAMETERS -----------------------

tql_kwargs = {
'epsilon': 1.0,
'minimum_epsilon': 0.01,
'epsilon_decay': 0.0001,
'learning_rate': 0.005,
'discount_factor': 0.99,
}

----------------------- INITIALIZE AND TRAIN MODEL ----------------------

tql_model = CustomTabularQLearning(
env=tql_env,
loader=loader,
random_seed=RANDOM_SEED,
**tql_kwargs
)
_ = tql_model.learn(episodes=tql_episodes)

the last line giving this error. can you please check this once

[kingsleynweye]

https://github.com/intelligent-environments-lab/CityLearn/blob/master/examples/citylearn_rlem23_tutorial.ipynb

----------------- CALCULATE NUMBER OF TRAINING EPISODES -----------------

i = 3 m = tql_env.observation_space[0].n n = tql_env.action_space[0].n t = tql_env.time_steps - 1 tql_episodes = m_n_i/t tql_episodes = int(tql_episodes) print('Q-Table dimension:', (m, n)) print('Number of episodes to train:', tql_episodes)

------------------------------- SET LOADER ------------------------------

loader = get_loader(max=tql_episodes*t) display(loader)

----------------------- SET MODEL HYPERPARAMETERS -----------------------

tql_kwargs = { 'epsilon': 1.0, 'minimum_epsilon': 0.01, 'epsilon_decay': 0.0001, 'learning_rate': 0.005, 'discount_factor': 0.99, }

----------------------- INITIALIZE AND TRAIN MODEL ----------------------

tql_model = CustomTabularQLearning( env=tql_env, loader=loader, random_seed=RANDOM_SEED, **tql_kwargs ) _ = tql_model.learn(episodes=tql_episodes)

the last line giving this error. can you please check this once

Oh I see. The as-provided notebook works fine for me in Colab (https://colab.research.google.com/github/intelligent-environments-lab/CityLearn/blob/master/examples/citylearn_rlem23_tutorial.ipynb). Pleae, can I see what the error message is?

[Hnecent]

@kingsleynweye

Forgot to mention that another way I have been able to solve this problem is by installing shimmy that provides a gymnasium binder.

Hey，Can you explain specifically how to use [shimmy] to solve this bug? Thanks~

[kingsleynweye]

@kingsleynweye

Forgot to mention that another way I have been able to solve this problem is by installing shimmy that provides a gymnasium binder.

Hey，Can you explain specifically how to use [shimmy] to solve this bug? Thanks~

You will want to install it with stable-baselines3. For example:

pip install stable-baselines3==2.1.0
pip install shimmy==0.2.1

Other version numbers for either package might work.