Skip to content

Commit 7b639e7

Browse files
jmchiltonjmchilton
committed
Implement a workflow for AI-assisted Selenium test case generation.
Automated Test Cases - Bad AI generated test cases are noisy and generally just bad - this is not that. A semi-automated approach I looked into was recording a script and having an AI convert it into Selenium commands - it wasn't very promising at all though. The selectors chosen don't really seem great and obviously most test cases can be bootstrapped and setup with a huge mountain of existing test helpers we've already had - reducing all of that to just sequences of Selectors would result in a ton of duplication, unreadable code, and less robustness (our helpers have a lot of good retry logic, adaptive waiting, rich debug messages, etc...). AI Assistance in Building Test Cases - Good! The semi-automatic approach that I think is more promising is to have the AI agent setup a rich environment for manually testing the UI and then provide a mechanism for turning that exploration directly into a test case. This PR adds a Claude slash command "/setup-selenium-test-notebook <feature description OR GitHub PR>". It can take a description of the feature to test or a PR. It will setup a Jupyter notebook with cells filled out for setting up the Selenium enviornment and talking with Galaxy. It tells the user about the config file they need to setup if it isn't present and tells the user how to run Jupyter. All this part is based on my prior work in #11177. The agent will pull down the PR description and try to come up with an idea for how to test it. The manual testing instructions we already provide are great for this. It will also "research" the code base and find related tests and will provide potentially relevant code from existing tests as Markdown comments right in the notebook - so you have a good idea of what helpers and components are already implemented that might help with the task of testing the PR. The agent seems smart enough to reason about when a managed history annotation is needed and how to deal with user login, etc... Developing in Jupyter is nice because it can sustain a persistent connection to the browser automation application. You don't have to re-run the whole test - you can work a line or two at a time with cells and preserve progress and just re-run what is needed as components are annotated, etc... I think the screenshots are a cool part of the framework we have - and these will appear right inside the notebook. After the notebook test case is ready go, claude seems pretty good at converting it directly to a test case. This can be done with '/extract-selenium-test <notebook path or description>'
1 parent 425ddc9 commit 7b639e7

File tree

8 files changed

+970
-1
lines changed

8 files changed

+970
-1
lines changed

‎lib/galaxy/selenium/context.py‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,10 +64,28 @@ def __init__(self, from_dict: Optional[dict] = None) -> None:
6464
self.url = from_dict.get("local_galaxy_url", "http://localhost:8080")
6565
self.target_url_from_selenium = from_dict.get("selenium_galaxy_url", self.url)
6666
self.timeout_multiplier = from_dict.get("timeout_multiplier", 1)
67+
# Optional properties...
68+
self.login_email = from_dict.get("login_email", "")
69+
self.login_password = from_dict.get("login_password", "")
6770

6871
def _screenshot_path(self, label, extension=".png"):
6972
return label + extension
7073

74+
# mirror TestWithSelenium method for doing this but use the config.
75+
def login(self):
76+
if self.login_email:
77+
assert (
78+
self.login_password
79+
), "If login_email is set, a password must be set also with login_password"
80+
self.home()
81+
self.submit_login(
82+
email=self.login_email,
83+
password=self.login_password,
84+
assert_valid=True,
85+
)
86+
else:
87+
self.register()
88+
7189

7290
def init(config=None, clazz=GalaxySeleniumContextImpl) -> GalaxySeleniumContext:
7391
if os.path.exists("galaxy_selenium_context.yml"):

‎lib/galaxy_test/selenium/.claude/README.md‎

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
# Claude Code Configuration for Galaxy Selenium Tests
2+
3+
This directory contains Claude Code configuration and slash commands for working with Galaxy's Selenium test suite.
4+
5+
## CLAUDE.md
6+
7+
The `CLAUDE.md` file in the parent directory (`../CLAUDE.md`) provides comprehensive documentation about Galaxy Selenium testing conventions. It includes:
8+
9+
- Test structure and base classes
10+
- Common patterns (login, navigation, data upload)
11+
- Decorator usage (`@selenium_test`, `@managed_history`, `@requires_admin`)
12+
- Component system access
13+
- Helper methods and wait types
14+
- Accessibility testing guidelines
15+
- Complete test templates for different scenarios
16+
17+
**Purpose**: This file serves as context for Claude Code when writing or modifying Selenium tests, ensuring consistency with Galaxy's testing conventions.
18+
19+
## Slash Commands
20+
21+
### Direct Test Creation
22+
23+
#### `/new-selenium-test`
24+
25+
Bootstrap a new Selenium test file from scratch.
26+
27+
**Usage:**
28+
```
29+
/new-selenium-test <feature description>
30+
```
31+
32+
**Examples:**
33+
```
34+
/new-selenium-test workflow import with accessibility testing
35+
/new-selenium-test admin user management features
36+
/new-selenium-test dataset metadata editing with managed history
37+
```
38+
39+
**What it does:**
40+
- Creates a new test file with proper structure
41+
- Includes appropriate imports and decorators
42+
- Follows Galaxy naming conventions
43+
- Uses correct template based on requirements
44+
- Asks clarifying questions if needed
45+
46+
#### `/add-selenium-test`
47+
48+
Add new test methods to an existing Selenium test file.
49+
50+
**Usage:**
51+
```
52+
/add-selenium-test <description and target file>
53+
```
54+
55+
**Examples:**
56+
```
57+
/add-selenium-test Add a test for tag filtering to test_history_panel.py
58+
/add-selenium-test Add admin test for user deletion to test_admin_app.py
59+
/add-selenium-test Add dataset copy test with managed history to test_dataset_edit.py
60+
```
61+
62+
**What it does:**
63+
- Reads existing file to match style
64+
- Adds properly structured test methods
65+
- Updates imports if necessary
66+
- Maintains consistency with existing code
67+
68+
### Interactive Jupyter Workflow
69+
70+
For complex tests or when you need to explore the DOM interactively, use the Jupyter notebook workflow:
71+
72+
#### `/setup-selenium-test-notebook`
73+
74+
Create a Jupyter notebook for interactively prototyping a Selenium test.
75+
76+
**Usage:**
77+
```
78+
/setup-selenium-test-notebook <feature description OR GitHub PR>
79+
```
80+
81+
**Examples (text mode):**
82+
```
83+
/setup-selenium-test-notebook dataset metadata editing
84+
/setup-selenium-test-notebook workflow import from URL
85+
/setup-selenium-test-notebook admin user quota management
86+
```
87+
88+
**Examples (GitHub PR mode):**
89+
```
90+
/setup-selenium-test-notebook https://github.com/galaxyproject/galaxy/pull/12345
91+
/setup-selenium-test-notebook #12345
92+
/setup-selenium-test-notebook 12345
93+
```
94+
95+
**What it does:**
96+
- Fetches PR context (if PR mode): description, screenshots, author
97+
- Launches research subagent to find similar existing tests
98+
- Creates a prototype notebook in `jupyter/` directory
99+
- Includes PR context, example code, and starter code
100+
- Prints complete setup instructions
101+
102+
**Workflow:**
103+
1. Run `/setup-selenium-test-notebook <feature>` to create notebook
104+
2. Follow printed instructions to start Jupyter
105+
3. Prototype test interactively with screenshots
106+
4. Run `/extract-selenium-test` when done to convert to test file
107+
108+
#### `/extract-selenium-test`
109+
110+
Extract test code from a Jupyter notebook and convert to a proper test file.
111+
112+
**Usage:**
113+
```
114+
/extract-selenium-test <notebook path or description>
115+
```
116+
117+
**Examples:**
118+
```
119+
/extract-selenium-test from jupyter/test_prototype_dataset_metadata.ipynb
120+
/extract-selenium-test jupyter/test_prototype_workflow_import.ipynb as test_workflow_import
121+
/extract-selenium-test notebook test_prototype_admin_users.ipynb
122+
```
123+
124+
**What it does:**
125+
- Reads notebook cells (skipping initialization)
126+
- Transforms `gx_selenium_context` to `self`
127+
- Detects required decorators and attributes
128+
- Creates properly structured Pytest test file for the Selenium test suite
129+
- Groups related cells into test methods
130+
131+
## Getting Started
132+
133+
### Quick Start (Direct Creation)
134+
135+
For straightforward tests with known requirements:
136+
137+
1. **Read `../CLAUDE.md`** to understand Galaxy Selenium testing conventions
138+
2. **Use `/new-selenium-test <description>`** to create a new test file
139+
3. **Use `/add-selenium-test <description>`** to extend an existing test file
140+
4. **Run** with `pytest lib/galaxy_test/selenium/test_<your_file>.py`
141+
142+
### Interactive Development (Jupyter Workflow)
143+
144+
For complex tests or DOM exploration:
145+
146+
1. **Create notebook**: `/setup-selenium-test-notebook <description>`
147+
2. **Start Galaxy & Jupyter** following the printed instructions
148+
3. **Prototype interactively** in the notebook with live screenshots
149+
4. **Extract to test**: `/extract-selenium-test <notebook_path>`
150+
5. **Run** with `pytest lib/galaxy_test/selenium/test_<your_file>.py`
151+
152+
## Tips
153+
154+
- **Be specific** in command descriptions (mention admin access, managed history, etc.)
155+
- **Use Jupyter workflow** when you need to explore the DOM or iterate quickly
156+
- **Use direct creation** when you know exactly what needs to be tested
157+
- Commands will **ask clarifying questions** if your request is ambiguous
158+
- **Review generated code** to ensure it meets your specific needs
159+
- **Reference existing tests** in the parent directory for patterns and examples

‎lib/galaxy_test/selenium/.claude/commands/add-selenium-test.md‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
---
2+
description: Add a new test method to an existing Galaxy Selenium test file
3+
---
4+
5+
The user input can be provided directly or as a command argument - you **MUST** consider it before proceeding.
6+
7+
User input:
8+
9+
$ARGUMENTS
10+
11+
## Context
12+
13+
You are adding a new test method to an existing Selenium test file in the Galaxy test suite. Read the CLAUDE.md file in this directory for complete context on Galaxy Selenium testing conventions.
14+
15+
## Task
16+
17+
Add one or more new test methods to an existing test file based on the user's description. The user should specify:
18+
- Which test file to modify (e.g., `test_history_panel.py`)
19+
- What scenario(s) to test
20+
- Any special requirements (managed history, admin access, etc.)
21+
22+
## Requirements
23+
24+
1. **Read CLAUDE.md** in the current directory for full context
25+
26+
2. **Read the target test file** to understand:
27+
- Existing class structure and attributes
28+
- Import statements already present
29+
- Coding style and patterns used
30+
- Helper methods defined in the class
31+
32+
3. **Match the existing style**:
33+
- Use same patterns as other methods in the class
34+
- Follow same naming conventions
35+
- Maintain consistent indentation and spacing
36+
- Add similar docstrings
37+
38+
4. **Add necessary imports** if new decorators/modules are needed:
39+
- `managed_history` if using `@managed_history`
40+
- `requires_admin` if using `@requires_admin`
41+
- Any other decorators from `galaxy.selenium.navigates_galaxy`
42+
43+
5. **Use appropriate decorators**:
44+
- Always use `@selenium_test`
45+
- Add `@managed_history` if the test needs predictable HIDs
46+
- Add `@requires_admin` if testing admin features
47+
- Add `@edit_details` or other navigation decorators as needed
48+
49+
6. **Follow test structure**:
50+
- Clear, descriptive method name: `test_<specific_scenario>`
51+
- Docstring explaining what is being tested
52+
- Proper setup and navigation
53+
- Clear assertions
54+
- Screenshots if helpful (use `self.screenshot("description")`)
55+
56+
7. **Respect class attributes**:
57+
- If class has `ensure_registered = True`, user is already logged in
58+
- If class has `run_as_admin = True`, tests run as admin
59+
- Use these to simplify test setup
60+
61+
## Output
62+
63+
1. **Confirm the target**:
64+
- Which file will be modified
65+
- What test method(s) will be added
66+
67+
2. **Add the test method(s)** to the appropriate class
68+
69+
3. **Update imports** if needed
70+
71+
4. **Explain what was added**:
72+
- What the new test does
73+
- Why certain decorators were used
74+
- How it fits with existing tests
75+
- How to run just this new test
76+
77+
## Example Usage
78+
79+
```
80+
/add-test Add a test for tag filtering to test_history_panel.py
81+
/add-test Add admin test for user deletion to test_admin_app.py
82+
/add-test Add dataset copy test with managed history to test_dataset_edit.py
83+
```
84+
85+
## Notes
86+
87+
- Always read the existing file first to match its style
88+
- Don't duplicate existing test scenarios
89+
- Keep tests independent and focused
90+
- Consider edge cases and error conditions
91+
- Add accessibility tests where appropriate

‎lib/galaxy_test/selenium/.claude/commands/extract-selenium-test.md‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
---
2+
description: Extract test code from a Jupyter notebook and convert it to a proper Selenium test file
3+
---
4+
5+
The user input can be provided directly or as a command argument - you **MUST** consider it before proceeding.
6+
7+
User input:
8+
9+
$ARGUMENTS
10+
11+
## Context
12+
13+
After prototyping a Selenium test interactively in a Jupyter notebook, this command extracts the test code and converts it into a proper test file following Galaxy's Selenium testing conventions.
14+
15+
## Task
16+
17+
Extract test cells from a Jupyter notebook and create a properly structured Selenium test file.
18+
19+
## Requirements
20+
21+
1. **Read the target notebook** specified by the user (e.g., `jupyter/test_prototype_<name>.ipynb`)
22+
23+
2. **Read CLAUDE.md** for full context on Galaxy Selenium test conventions
24+
25+
3. **Identify test cells** - Extract all cells after the initialization cell (cell 2):
26+
- Skip the parameters cell (cell 0)
27+
- Skip the initialization cell (cell 1)
28+
- Extract cells 2+ which contain the test implementation
29+
30+
4. **Transform notebook code to test code**:
31+
- Replace `gx_selenium_context.` with `self.`
32+
- Remove screenshot calls or keep them if desired for debugging
33+
- Group related operations into logical test methods
34+
- Add proper decorators (`@selenium_test`, `@managed_history`, etc.)
35+
- Add docstrings to test methods
36+
37+
5. **Determine test structure** from the notebook code:
38+
- Does it need `ensure_registered = True`? (if it calls `register()` or `login()`)
39+
- Does it need `run_as_admin = True`? (if it calls `admin_login()`)
40+
- Does it need `@managed_history`? (if it works with datasets/HIDs)
41+
- Does it need `@requires_admin`? (if testing admin features)
42+
43+
6. **Create proper test file**:
44+
- Add appropriate imports
45+
- Create test class with correct name
46+
- Add class attributes as needed
47+
- Create test methods from notebook cells
48+
- Group related cells into single test methods
49+
- Split unrelated operations into separate test methods
50+
51+
7. **Handle special cases**:
52+
- If notebook uses `dataset_populator`, keep those patterns
53+
- If notebook uses `current_history_id()`, consider `@managed_history`
54+
- If notebook has multiple distinct scenarios, create multiple test methods
55+
- Convert inline assertions to proper pytest assertions
56+
57+
8. **File naming**: Ask user for feature name if not clear from notebook name
58+
59+
## Output
60+
61+
1. **Analyze the notebook** and report:
62+
- Number of test cells found
63+
- Key operations detected (login, upload, navigation, etc.)
64+
- Recommended test structure (basic/managed history/admin)
65+
66+
2. **Ask for confirmation** if structure is ambiguous:
67+
- Should this be one test method or multiple?
68+
- What should the test methods be named?
69+
- Should screenshots be kept or removed?
70+
71+
3. **Create the test file** at `test_<feature_name>.py` with:
72+
- Proper imports
73+
- Correctly structured test class
74+
- Appropriate decorators and attributes
75+
- Well-named test methods with docstrings
76+
- Transformed code (gx_selenium_context → self)
77+
78+
4. **Report completion**:
79+
```
80+
✓ Extracted test from notebook: jupyter/test_prototype_<name>.ipynb
81+
✓ Created test file: test_<feature_name>.py
82+
83+
Test structure:
84+
- Class: Test<FeatureName>
85+
- Attributes: [ensure_registered, run_as_admin, etc.]
86+
- Methods: test_method_1, test_method_2, ...
87+
88+
To run the test:
89+
pytest lib/galaxy_test/selenium/test_<feature_name>.py
90+
91+
Or run a specific test:
92+
pytest lib/galaxy_test/selenium/test_<feature_name>.py::Test<FeatureName>::test_method_1
93+
```
94+
95+
5. **Provide review guidance**:
96+
- Suggest reviewing for proper wait/sleep patterns
97+
- Recommend adding assertions if few were found
98+
- Suggest accessibility testing if not present
99+
- Note any hardcoded values that should be parameterized
100+
101+
## Example Usage
102+
103+
```
104+
/extract-test from jupyter/test_prototype_dataset_metadata.ipynb
105+
/extract-test jupyter/test_prototype_workflow_import.ipynb as test_workflow_import
106+
/extract-test notebook test_prototype_admin_users.ipynb
107+
```
108+
109+
## Notes
110+
111+
- Read CLAUDE.md for complete context on test conventions
112+
- The notebook's initialization cells (0-1) are never included in the test
113+
- Multiple notebook cells can be combined into a single test method if related
114+
- Distinct scenarios should become separate test methods
115+
- The command should intelligently detect patterns (admin, managed history, etc.)
116+
- Preserve the logic but adapt to test framework conventions
117+
- Remove debug-only code (print statements, excessive screenshots)
118+
- Keep test-relevant screenshots if they help document expected states

0 commit comments

Comments
 (0)