Proposal: update rllib and pettingzoo examples to current release versions

[elliottower]

I'm planning to do some testing with the newly released ray 2.3.0 and I noticed the pettingzoo example is 9 months old and still uses gym rather than gymnasium, both should probably be updated. Will create a PR if I can get them to work but figure I'd create an issue first. Appreciate the repo, very excited to use it.

[jagapiou]

That sounds very useful thank you! I don't know the Ray stuff etc that well, so @duenez is best to comment on any details.

[duenez]

Hello Elliot,

That would be fantastic! Please let me know if there's anything I can do to help.

[elliottower]

Hello Elliot,

That would be fantastic! Please let me know if there's anything I can do to help.

Appreciate it, will reach out if I run into any issues. I got a good bit of progress today, although a major blocker to updating the entire pettingzoo tutorial is that stable-baselines3 doesn't officially support gymnasium yet (pettingzoo has migrated as of October 2022). There's some open pull requests which may get merged in the future (stable-baselines3 stable-baselines3-contrib but the only way to get it working currently is manually installing the feature branch: pip install git+https://github.com/DLR-RM/stable-baselines3@feat/gymnasium-support. I can do this in the meantime for development but I'd imagine merging the tutorial should probably wait on an official release.

Edit: looks like there's an API incompatibility with pettingzoo and the sb3 gymnasium support feature branch (link) so the pettingzoo tutorial will have to wait until that is fixed.

I got most of the pettingzoo code adapted though, I chose to use the SB3's frame stacking functionality rather than supersuit because the latter required casting types to float, stacking frames, and then casting back to uint8 for SB3. I've also heard that supersuit will be closed down and its functionality migrated directly into gymnasium and pettingzoo, so I figure it's best not to rely on it too much if possible.

My changes so far: elliottower@64b38e9

Luckily ray/rllib 2.3.0 does have gymnasium support so that shouldn't be a blocker, planning to work on that tomorrow.

[elliottower]

One minor issue I had with this repo was running the pylint script, didn't find any documentation on it and for some reason running pylint --rcfile .pylintrc didn't work.

I'm currently running the pytest suite but it seems to be going very slowly and says it has 4787 items. Edit: took 34 minutes 30 seconds on a 2019 intel mbp. The .sh install scripts worked great though and the tests ran pretty quickly there (less than 5 mins iirc)

[duenez]

Yes, some of our tests are very slow (the full substrate tests). I believe we don't run all the tests in the install.sh script.

[elliottower]

Is there a reason the pettingzoo env is defined with a separate superclass class _ParallelEnv? I got some issues when trying to use wrapper functions from pettingzoo/gymnasium/SB3 because the base class when you do env.unwrapped is _ParallelEnv.

Moving the EzPickle init line into the regular class (which is of the correct type parallel pettingzoo env) led to an error saying it can’t load the lab2d object from pickle. I don’t have experience with lab2d or the EzPickle function, imagine the specific definition of this ParallelEnv with the two superclasses (one for the env one for EzPickle) is what allows it to load properly?

Edit: Nevermind, I got the EzPickle component to work by restructuring the code and using wrappers. Will post a PR once I'm finished.

[elliottower]

Any idea on how to deal with these warnings? Getting them when running pettingzoo API tests, thinking they may be corner cases that the pettingzoo wrapper doesn't check for @jagapiou @duenez
W @/Users/elliottower/Documents/GitHub/meltingpot//meltingpot/lua/modules/prefab_utils.lua:83] Character x not found in the charPrefabMap. Ignoring.

W @/Users/elliottower/Documents/GitHub/meltingpot//meltingpot/lua/modules/base_simulation.lua:225] GameObject 'scene' did not have a Transform component, but explicitly specifying one is strongly preferred. Using a default.

W @/Users/elliottower/Documents/GitHub/meltingpot//meltingpot/lua/modules/base_simulation.lua:219] GameObject 'scene' did not have a StateManager component, but explicitly specifying one is strongly preferred. Using a default.

Also PR is up: #117, running final tests and will then make a PR on shimmy with melting pot compatibility. I think using shimmy is best as it has comprehensive tests and is used for other API conversions, so I can update this PR to use shimmy once that gets added (as opposed to examples/pettingzoo/utils.py)

[duenez]

Those warnings are not critical. They happen when a substrate doesn't define a scene, it when their ASCII map contains a character that is not present in the prefab map. Which substrate is this for? I could just add those and fix the warnings. Shimmy seems like a reasonable approach to unify APIs. Thanks. Finally, regarding the EzPickle, this is needed, I believe, to serialise the environment. I'm particular, the EzPickle class has a __set_state__ and __get_state__ pair of functions ( https://github.com/openai/gym/blob/master/gym/utils/ezpickle.py) that I imagine allow for saving the environment. If you can modernise the code and not need that, it should be fine.

…

On Sat, 4 Mar 2023, 00:49 Elliot Tower, ***@***.***> wrote: Any idea on how to deal with these warnings? Getting them when running pettingzoo API tests, thinking they may be corner cases that the pettingzoo wrapper doesn't check for @jagapiou <https://github.com/jagapiou> @duenez <https://github.com/duenez> W @/Users/elliottower/Documents/GitHub/meltingpot//meltingpot/lua/modules/prefab_utils.lua:83] Character x not found in the charPrefabMap. Ignoring. W @/Users/elliottower/Documents/GitHub/meltingpot//meltingpot/lua/modules/base_simulation.lua:225] GameObject 'scene' did not have a Transform component, but explicitly specifying one is strongly preferred. Using a default. W @/Users/elliottower/Documents/GitHub/meltingpot//meltingpot/lua/modules/base_simulation.lua:219] GameObject 'scene' did not have a StateManager component, but explicitly specifying one is strongly preferred. Using a default. Also PR is up: #117 <#117>, running final tests and will then make a PR on shimmy <https://github.com/Farama-Foundation/Shimmy> with melting pot compatibility. I think using shimmy is best as it has comprehensive tests and is used for other API conversions, so I can update this PR to use shimmy once that gets added (as opposed to examples/pettingzoo/utils.py) — Reply to this email directly, view it on GitHub <#113 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAY7XEO4XO7EPPHRXV6CDJ3W2KGR5ANCNFSM6AAAAAAVLTHO6M> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[elliottower]

Will get the list of substrates later today when I rerun the tests.

@duenez Do you happen to know if it’s possible to do rendering using PIL or something other than matplotlib? Not sure exactly how the rendering works internally but on the shimmy PR they raised concerns about not wanting to have to import matplotlib (most farama code uses pygame afaik and/or PIL, I think it might be a speed concern?)

And fwiw the pickle code is still used in other farama games like Atari and their devs said it looks fine.

[duenez]

Yes, of course. The observation is produced as a raw numpy array of height X width X channels (3, RGB). Si you can render this to anything you want. One possibility is doing imshow in matplotlib, but equally possible is using PIL.

[elliottower]

@duenez thanks, I ended up searching through the code more and finding an example using pygame rendering, which is what is used by all other pettingzoo envs, and mirrored that functionality in the shimmy wrapper.

[elliottower]

@duenez is it possible to reset the environment using a specific seed rather than re-initializing a new environment with a specified seed? The new gym API requires that the reset() function take a seed argument, I've been trying to adapt the code but it doesn't seem to be possible without initializing the environment again, which I also have had trouble figuring out how to do.

The only python file I see using seeding is builder_test.py but all of the example scripts and documentation seem to indicate that the higher level substrate.build() or substrate.build_from_config() are preferred to manually calling builder.builder.

I think the correct code should be something like this, but it says that the config is missing they key simulation.

config = meltingpot.python.substrate.get_config(substrate_name)
env = builder.build(config, env_seed=seed)

Have been having a lot of difficulty understanding the inner workings of this code and if it's even possible to do what I want to do. Maybe it's best to change the way the reset() function works internally to allow a seed to be specified?