Skip to content

Commit 6da0688

Browse files
IonutMuthiIonutMuthi
committed
testing: add README
Signed-off-by: IonutMuthi <[email protected]>
1 parent 3e1924c commit 6da0688

File tree

1 file changed

+227
-0
lines changed

1 file changed

+227
-0
lines changed

‎tools/testing/README.md‎

Lines changed: 227 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,227 @@
1+
# Scopy Testing Tools
2+
3+
A comprehensive set of Python tools for managing Scopy's manual testing workflow. These tools automate test environment setup, result parsing, and Excel report generation for release testing processes.
4+
5+
## Quick Start
6+
7+
### 1. **Generate Test Environment**
8+
```bash
9+
# Create test environment for v3.0.0 with all tests
10+
python3 setup_test_environment.py v3.0.0
11+
12+
# Filter by component
13+
python3 setup_test_environment.py v3.0.0 --component adc dac
14+
15+
# Filter by priority level
16+
python3 setup_test_environment.py v3.0.0 --rbp P0 P1
17+
18+
# Release testing: P0 tests + new features since last version
19+
python3 setup_test_environment.py v3.0.0 --rbp P0 --new-since v2.5.0
20+
```
21+
22+
### 2. **Execute Tests Manually**
23+
Edit the generated RST files in `testing_results/testing_results_v3.0.0/` and fill in:
24+
- **Tested OS:** Your operating system
25+
- **Comments:** Test execution notes
26+
- **Result:** PASS or FAIL
27+
28+
### 3. **Generate Excel Report**
29+
```bash
30+
# Parse completed test results and create Excel report
31+
python3 parseTestResults.py v3.0.0
32+
```
33+
34+
## Detailed Usage
35+
36+
### `setup_test_environment.py` - Test Environment Setup
37+
38+
**Purpose:** Creates filtered copies of test documentation for manual testing execution.
39+
40+
**Basic Usage:**
41+
```bash
42+
python3 setup_test_environment.py <version> [OPTIONS]
43+
```
44+
45+
**Parameters:**
46+
- `version` - Version name for the test directory (e.g., v3.0.0)
47+
- `--rbp` - Filter by RBP priority levels (P0, P1, P2, P3)
48+
- `--component` - Filter by component names (adc, dac, m2k, core, etc.)
49+
- `--new-since` - Include tests added since specified git version
50+
51+
**Examples:**
52+
```bash
53+
# Copy all tests for v3.0.0
54+
python3 setup_test_environment.py v3.0.0
55+
56+
# Only P0 and P1 priority tests
57+
python3 setup_test_environment.py v3.0.0 --rbp P0 P1
58+
59+
# Only ADC and DAC component tests
60+
python3 setup_test_environment.py v3.0.0 --component adc dac
61+
62+
# Combined filters: M2K component with P0 priority
63+
python3 setup_test_environment.py v3.0.0 --component m2k --rbp P0
64+
65+
# Release testing: All P0 tests + new tests since v2.5.0
66+
python3 setup_test_environment.py v3.0.0 --rbp P0 --new-since v2.5.0
67+
68+
# Only new features testing
69+
python3 setup_test_environment.py v3.0.0 --new-since v2.5.0
70+
```
71+
72+
**Output:**
73+
- Creates `testing_results/testing_results_v3.0.0/` directory
74+
- Contains filtered RST files ready for manual testing
75+
- Preserves RST structure for documentation compilation
76+
77+
### `parseTestResults.py` - Excel Report Generation
78+
79+
**Purpose:** Parses completed test results and generates Excel reports with execution statistics.
80+
81+
**Basic Usage:**
82+
```bash
83+
python3 parseTestResults.py <version> [OPTIONS]
84+
```
85+
86+
**Parameters:**
87+
- `version` - Version to parse (must match testing_results directory)
88+
- `--output` - Custom output Excel file path (optional)
89+
90+
**Examples:**
91+
```bash
92+
# Parse v3.0.0 test results
93+
python3 parseTestResults.py v3.0.0
94+
95+
# Custom output location
96+
python3 parseTestResults.py v3.0.0 --output /path/to/custom_results.xlsx
97+
```
98+
99+
**Output:**
100+
- Creates/updates `testing_results/scopy_test_results.xlsx`
101+
- Each version gets its own worksheet
102+
- Columns: Test UID | Component | RBP | Result | Tested OS | Comments | File
103+
104+
### `generateTestsTable.py` - Source Documentation Tracking
105+
106+
**Purpose:** Generates Excel tracking tables directly from source documentation (before test execution).
107+
108+
**Basic Usage:**
109+
```bash
110+
python3 generateTestsTable.py <version> [OPTIONS]
111+
```
112+
113+
**Parameters:**
114+
- `version` - Version name for Excel worksheet
115+
- `--sort` - Sort tests by RBP priority instead of UID
116+
- `--filter` - Filter tests by RBP levels (e.g., P0,P1)
117+
118+
**Examples:**
119+
```bash
120+
# Generate tracking table for v3.0.0
121+
python3 generateTestsTable.py v3.0.0
122+
123+
# Only P0 and P1 tests, sorted by priority
124+
python3 generateTestsTable.py v3.0.0 --filter P0,P1 --sort
125+
```
126+
127+
**Output:**
128+
- Creates/updates `scopy_tests_tracking.xlsx` in scopy root
129+
- Columns: Test UID | Test RBP | Test Pass/Fail (empty for manual entry)
130+
131+
## Complete Testing Workflow
132+
133+
### Release Testing Example (v3.0.0)
134+
135+
**1. Prepare Test Environment**
136+
```bash
137+
# Create comprehensive release testing environment
138+
python3 setup_test_environment.py v3.0.0 --rbp P0 --new-since v2.5.0
139+
```
140+
141+
**2. Execute Tests**
142+
```bash
143+
cd testing_results/testing_results_v3.0.0/
144+
# Manually edit RST files with test results:
145+
```
146+
147+
**Example test result format:**
148+
```rst
149+
**Tested OS:** Windows 11
150+
151+
**Comments:** Test passed successfully. Minor UI lag observed but within acceptable range.
152+
153+
**Result:** PASS
154+
```
155+
156+
**3. Generate Report**
157+
```bash
158+
# Parse results and create Excel report
159+
python3 parseTestResults.py v3.0.0
160+
```
161+
162+
**4. Review Results**
163+
- Open `testing_results/scopy_test_results.xlsx`
164+
- Check "v3.0.0" worksheet for complete test execution summary
165+
166+
167+
## Requirements
168+
169+
### Dependencies
170+
```bash
171+
pip install openpyxl # For Excel file generation
172+
```
173+
174+
### Git Requirements
175+
- Git repository (required for `--new-since` functionality)
176+
- Proper git tags for version references
177+
178+
### File Structure Requirements
179+
```
180+
scopy/
181+
├── docs/tests/ # Source test documentation
182+
├── testing_results/ # Generated test environments
183+
└── tools/testing/ # This toolset
184+
```
185+
186+
## Excel Report Format
187+
188+
### Columns
189+
- **Test UID** - Unique test identifier
190+
- **Component** - Test component (adc, dac, m2k, core, etc.)
191+
- **RBP** - Risk-based priority (P0, P1, P2, P3)
192+
- **Result** - Test outcome (PASSED, FAILED, SKIPPED)
193+
- **Tested OS** - Operating system used for testing
194+
- **Comments** - Test execution notes
195+
- **File** - Source RST filename
196+
197+
### Multiple Versions
198+
- Each version gets its own worksheet in the Excel file
199+
- Historical test data preserved across versions
200+
- Easy comparison between releases
201+
202+
## Troubleshooting
203+
204+
### Common Issues
205+
206+
**"Git tag not found" error:**
207+
```bash
208+
# Check available git tags
209+
git tag -l
210+
211+
# Use exact tag name
212+
python3 setup_test_environment.py v3.0.0 --new-since v2.5.0
213+
```
214+
215+
**"Not in a git repository" error:**
216+
```bash
217+
# Ensure you're in the scopy git repository root
218+
cd /path/to/scopy
219+
python3 tools/testing/setup_test_environment.py v3.0.0 --new-since v2.5.0
220+
```
221+
222+
**No tests found:**
223+
- Verify `docs/tests/` directory exists
224+
- Check test files contain proper `**UID:**` and `**RBP:**` metadata
225+
226+
227+

0 commit comments

Comments
 (0)