Skip to content

Commit 6df4bb9

Browse files
authored
docs(README): add dashboard visualization; improve description of VanillaReplayStrategy
1 parent 81074c4 commit 6df4bb9

File tree

1 file changed

+25
-12
lines changed

1 file changed

+25
-12
lines changed

README.md

Lines changed: 25 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -158,27 +158,37 @@ pointing the cursor and left or right clicking, as described in this
158158

159159
### Visualize
160160

161-
Visualize the latest recording you created by running the following command:
161+
Quickly visualize the latest recording you created by running the following command:
162162

163163
```
164164
python -m openadapt.visualize
165165
```
166166

167-
This will open a scrollable window that looks something like this:
167+
This will generate an HTML file and open a tab in your browser that looks something like this:
168168

169-
<img width="1512" alt="image" src="https://github.com/OpenAdaptAI/OpenAdapt/assets/774615/451dd467-20ae-4ce7-a3b4-f888635afe8c">
169+
![image](https://github.com/OpenAdaptAI/OpenAdapt/assets/774615/5d7253b7-ae12-477c-94a3-b388e4f37587)
170170

171-
<img width="1511" alt="image" src="https://github.com/OpenAdaptAI/OpenAdapt/assets/774615/13264cf6-46c0-4413-a29d-59bdd040a32e">
171+
For a more powerful dashboard, run:
172+
173+
```
174+
python -m openadapt.app.dashboard.run
175+
```
176+
177+
This will start a web server locally, and then open a tab in your browser that looks something like this:
172178

173-
For a browser-based visualization, run:
179+
![image](https://github.com/OpenAdaptAI/OpenAdapt/assets/774615/48d27459-4be8-4b96-beb0-1973953b8a09)
180+
181+
For a desktop app-based visualization, run:
174182

175183
```
176-
python -m openadapt.deprecated.visualize
184+
python -m openadapt.app.visualize
177185
```
178186

179-
This will open up a tab in your browser that looks something like this:
187+
This will open a scrollable window that looks something like this:
180188

181-
![image](https://github.com/OpenAdaptAI/OpenAdapt/assets/774615/5d7253b7-ae12-477c-94a3-b388e4f37587)
189+
<img width="1512" alt="image" src="https://github.com/OpenAdaptAI/OpenAdapt/assets/774615/451dd467-20ae-4ce7-a3b4-f888635afe8c">
190+
191+
<img width="1511" alt="image" src="https://github.com/OpenAdaptAI/OpenAdapt/assets/774615/13264cf6-46c0-4413-a29d-59bdd040a32e">
182192

183193
### Playback
184194

@@ -190,12 +200,15 @@ python -m openadapt.replay NaiveReplayStrategy
190200

191201
Other replay strategies include:
192202

193-
- [`StatefulReplayStrategy`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/stateful.py): Proof-of-concept which uses the OpenAI GPT-4 API with prompts constructed via OS-level window data.
194-
- [`VanillaReplayStrategy`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/vanilla.py): If AGI or GPT6 happens, this script should be able to suddenly do the work. --LunjunZhang
195-
- [`VisualReplayStrategy`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/visual.py): Uses [Fast Segment Anything Model (FastSAM)](https://github.com/CASIA-IVA-Lab/FastSAM) to segment active window. Accepts an "instructions" parameter that is used to modify the recording, e.g.:
203+
- [`StatefulReplayStrategy`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/stateful.py): Early proof-of-concept which uses the OpenAI GPT-4 API with prompts constructed via OS-level window data.
204+
- (*)[`VisualReplayStrategy`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/visual.py): Uses [Fast Segment Anything Model (FastSAM)](https://github.com/CASIA-IVA-Lab/FastSAM) to segment active window.
205+
- (*)[`VanillaReplayStrategy`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/vanilla.py): Assumes the model is capable of directly reasoning on states and actions accurately. With future frontier models, we hope that this script will suddenly work a lot better.
206+
207+
208+
The (*) prefix indicates strategies which accept an "instructions" parameter that is used to modify the recording, e.g.:
196209

197210
```
198-
python -m openadapt.replay VisualReplayStrategy --instructions "Multiply 9x5 instead of 6x8"
211+
python -m openadapt.replay VanillaReplayStrategy --instructions "calculate 9-8"
199212
```
200213

201214
See https://github.com/OpenAdaptAI/OpenAdapt/tree/main/openadapt/strategies for a complete list. More ReplayStrategies coming soon! (see [Contributing](#Contributing)).

0 commit comments

Comments
 (0)