Testing updates

tools/testing/README.md

@@ -0,0 +1,205 @@
+# Scopy Testing Tools
+
+A comprehensive set of Python tools for managing Scopy's manual testing workflow. These tools automate test environment setup, result parsing, and CSV template generation for release testing processes.
+
+## Quick Start
+
+### 1. **Generate Test Environment**
+```bash
+# Create test environment for v3.0.0 with all tests
+python3 setup_test_environment.py v3.0.0
+
+# Filter by component
+python3 setup_test_environment.py v3.0.0 --component adc dac
+
+# Filter by priority level
+python3 setup_test_environment.py v3.0.0 --rbp P0 P1
+
+# Release testing: P0 tests + new features since last version
+python3 setup_test_environment.py v3.0.0 --rbp P0 --new-since v2.5.0
+```
+### 2. **CSV Template Auto-Generated**
+The CSV template is automatically created during step 1. You can also generate it manually:
+```bash
+python3 parseTestResults.py v3.0.0
+```
+
+### 3. **Execute Tests Manually**
+Edit the generated csv file to add test results based on provided columns
+
+
+## Detailed Usage
+
+### `setup_test_environment.py` - Test Environment Setup
+
+**Purpose:** Creates filtered copies of test documentation for manual testing execution.
+
+**Basic Usage:**
+```bash
+python3 setup_test_environment.py <version> [OPTIONS]
+```
+
+**Parameters:**
+- `version` - Version name for the test directory (e.g., v3.0.0)
+- `--rbp` - Filter by RBP priority levels (P0, P1, P2, P3)
+- `--component` - Filter by component names (adc, dac, m2k, core, etc.)
+- `--new-since` - Include tests added since specified git version (overrides all other filters so that all new tests are added)
+
+**Available Components:**
+Components are determined by folder structure in `docs/tests/`:
+- **Plugin components:** Any subdirectory in `docs/tests/plugins/` (e.g., adc, dac, m2k, ad936x, ad9084, adrv9002, fmcomms5, swiot1l, datalogger, debugger, jesd, pqm, registermap)
+- **Core component:** Files in `docs/tests/general/core/` subdirectory
+- **General meta-component:** Using `--component general` includes ALL files from the `general/` directory, including both core files and other general test files
+
+**Filter Logic:**
+- When using `--component` and/or `--rbp` without `--new-since`: Tests must match ALL specified filters (AND logic)
+- When using `--new-since`: Filter logic becomes `(component AND rbp) OR new-since`
+  - This means all new tests are included regardless of component/rbp filters
+  - Plus any existing tests matching the component/rbp criteria
+
+**Examples:**
+```bash
+# Copy all tests for v3.0.0
+python3 setup_test_environment.py v3.0.0
+
+# Only P0 and P1 priority tests
+python3 setup_test_environment.py v3.0.0 --rbp P0 P1
+
+# Only ADC and DAC component tests
+python3 setup_test_environment.py v3.0.0 --component adc dac
+
+# Core component tests only
+python3 setup_test_environment.py v3.0.0 --component core
+
+# All general tests (includes core and all other general/ files)
+python3 setup_test_environment.py v3.0.0 --component general
+
+# Combined filters: M2K component with P0 priority
+python3 setup_test_environment.py v3.0.0 --component m2k --rbp P0
+
+# Release testing: All P0 tests + All new tests since v2.5.0
+python3 setup_test_environment.py v3.0.0 --rbp P0 --new-since v2.5.0
+
+# Only new features testing
+python3 setup_test_environment.py v3.0.0 --new-since v2.5.0
+```
+
+**Output:**
+- Creates `testing_results/testing_results_v3.0.0/` directory
+- Contains filtered RST files ready for manual testing
+- Automatically generates CSV template
+- If directory exists, prompts to overwrite (this will also delete and regenerate the CSV file)
+
+### `parseTestResults.py` - CSV Template Generation
+
+**Purpose:** Generates CSV templates from testing directories for manual test execution tracking.
+
+**Basic Usage:**
+```bash
+python3 parseTestResults.py <version> [OPTIONS]
+```
+
+**Parameters:**
+- `version` - Version to parse (must match testing_results directory)
+- `--output` - Custom output CSV file path (optional)
+- `--force` - Force overwrite without prompting (optional)
+
+**Examples:**
+```bash
+# Parse v3.0.0 test results
+python3 parseTestResults.py v3.0.0
+
+# Custom output location
+python3 parseTestResults.py v3.0.0 --output /path/to/custom_results.csv
+
+# Force overwrite without prompting
+python3 parseTestResults.py v3.0.0 --force
+```
+
+**Output:**
+- Creates `testing_results/testing_results_{version}.csv`
+- Template with empty result fields for manual completion
+- Columns: Test UID | Component | RBP | Result | Tested OS | Comments | Tester | File
+
+## Complete Testing Workflow
+
+### Release Testing Example (v3.0.0)
+
+**1. Prepare Test Environment**
+```bash
+# Create comprehensive release testing environment
+python3 setup_test_environment.py v3.0.0 --rbp P0 --new-since v2.5.0
+```
+
+**2. Execute Tests and Track Results**
+- Open the auto-generated CSV template: `testing_results/testing_results_v3.0.0.csv`
+- For each test, fill in:
+  - **Result:** PASS, FAIL, or SKIP
+  - **Tested OS:** Operating system used
+  - **Comments:** Any observations or notes
+  - **Tester:** Your name or identifier
+
+**3. Review and Analyze**
+- Save the completed CSV file
+- Use spreadsheet software for analysis and reporting
+- Historical data is preserved across versions
+
+
+
+##  Requirements
+
+### Git Requirements
+- Git repository (required for `--new-since` functionality)
+- Proper git tags for version references
+
+### File Structure Requirements
+```
+scopy/
+├── docs/tests/           # Source test documentation
+├── testing_results/      # Generated test environments
+└── tools/testing/        # This toolset
+```
+
+## CSV Template Format
+
+### Columns
+- **Test UID** - Unique test identifier
+- **Component** - Test component (adc, dac, m2k, core, etc.)
+- **RBP** - Risk-based priority (P0, P1, P2, P3)
+- **Result** - Test outcome (empty template field for PASSED/FAILED/SKIPPED)
+- **Tested OS** - Operating system used for testing (empty template field)
+- **Comments** - Test execution notes (empty template field)
+- **Tester** - Person executing the tests (empty template field)
+- **File** - Source RST filename
+
+### Multiple Versions
+- Each version gets its own CSV file: `testing_results_{version}.csv`
+- Historical test data preserved across versions
+- Easy import into spreadsheet tools for analysis
+
+## Troubleshooting
+
+### Common Issues
+
+**"Git tag not found" error:**
+```bash
+# Check available git tags
+git tag -l
+
+# Use exact tag name
+python3 setup_test_environment.py v3.0.0 --new-since v2.5.0
+```
+
+**"Not in a git repository" error:**
+```bash
+# Ensure you're in the scopy git repository root
+cd /path/to/scopy
+python3 tools/testing/setup_test_environment.py v3.0.0 --new-since v2.5.0
+```
+
+**No tests found:**
+- Verify `docs/tests/` directory exists
+- Check test files contain proper `**UID:**` and `**RBP:**` metadata
+
+
+

tools/testing/file_filters.py

@@ -0,0 +1,86 @@
+#!/usr/bin/env python3
+"""
+File Filters Module
+Handles file and content filtering logic for test environment setup.
+
+This module provides functionality for:
+- Component-based file filtering
+- RBP-based content filtering
+- New test detection and filtering
+- Helper functions for filtering decisions
+"""
+
+from rst_parser import parse_rst_structure, rebuild_rst_file
+
+
+def _should_include_file(test_file, component_filter, new_uids):
+    """Determine if file should be included based on filters."""
+    # Skip all index files - they're not needed for CSV workflow
+    if 'index.rst' in test_file['relative_path']:
+        return False
+
+    # Special handling: 'general' meta-component includes entire general/ directory
+    # This includes both 'core' subdirectory and 'general' files
+    if 'general' in component_filter:
+        if test_file['relative_path'].startswith('general/'):
+            return True
+
+    # Include files from specified components
+    if test_file['component'] in component_filter:
+        return True
+
+    # Include files containing new tests (bypass component filter)
+    if new_uids and any(test['uid'] in new_uids for test in test_file['tests']):
+        return True
+
+    return False
+
+
+def filter_by_component(test_files, component_filter, new_uids=None):
+    """Simple file-level filtering by component, with new-since support."""
+    if not component_filter:
+        return test_files
+
+    return [tf for tf in test_files
+            if _should_include_file(tf, component_filter, new_uids)]
+
+
+def _should_include_test(test, rbp_filter, new_uids):
+    """Determine if test should be included based on filters."""
+    # Include if it's a new test (always)
+    if new_uids and test['uid'] in new_uids:
+        return True
+
+    # Include if matches RBP filter
+    if rbp_filter and test['rbp'] in rbp_filter:
+        return True
+
+    # Include if no filters specified
+    if not rbp_filter and not new_uids:
+        return True
+
+    return False
+
+
+def _extract_all_uids(test_files):
+    """Extract all UIDs from test files efficiently."""
+    return {test['uid'] for file_info in test_files
+            for test in file_info['tests']
+            if test['uid'] != 'MISSING'}
+
+
+def filter_rst_content(file_path, rbp_filter, new_uids=None):
+    """Parse RST file and keep only tests matching RBP filter or new tests."""
+
+    with open(file_path, 'r', encoding='utf-8') as f:
+        content = f.read()
+
+    # Split into header + test sections
+    structure = parse_rst_structure(content)
+
+    # Filter tests by RBP or new-since
+    filtered_tests = [test for test in structure['tests']
+                     if _should_include_test(test, rbp_filter, new_uids)]
+
+    # Reconstruct file
+    return rebuild_rst_file(structure['header'], filtered_tests)

tools/testing/file_operations.py

@@ -0,0 +1,60 @@
+#!/usr/bin/env python3
+"""
+File Operations Module
+Handles file copying operations and index file management.
+
+This module provides functionality for:
+- Copying test files with or without modification
+- Managing index file toctree entries
+- Component tracking for copied files
+- RST content-level filtering and file operations
+"""
+
+import os
+import shutil
+from file_filters import filter_rst_content
+
+
+def copy_files_unchanged(test_files, dest_dir, component_filter=None):
+    """Copy files without content modification."""
+    copied_count = 0
+
+    for test_file in test_files:
+        src_path = test_file['file_path']
+        dest_path = os.path.join(dest_dir, test_file['relative_path'])
+
+        # Create destination directory
+        os.makedirs(os.path.dirname(dest_path), exist_ok=True)
+
+        # Copy file unchanged
+        shutil.copy2(src_path, dest_path)
+        copied_count += 1
+
+    return copied_count
+
+
+def filter_by_rbp_content(test_files, rbp_filter, dest_dir, component_filter=None, new_uids=None):
+    """Content-level filtering: modify files to keep only specified RBP tests or new tests."""
+    if not rbp_filter and not new_uids:
+        # No filters - copy files as-is
+        return copy_files_unchanged(test_files, dest_dir, component_filter)
+
+    copied_count = 0
+
+    # Filter active - parse and modify content
+    for test_file in test_files:
+        dest_path = os.path.join(dest_dir, test_file['relative_path'])
+
+        # Create destination directory
+        os.makedirs(os.path.dirname(dest_path), exist_ok=True)
+
+        if test_file['tests']:  # File contains tests
+            filtered_content = filter_rst_content(test_file['file_path'], rbp_filter, new_uids)
+            with open(dest_path, 'w', encoding='utf-8') as f:
+                f.write(filtered_content)
+        else:  # File has no tests (docs, etc.)
+            shutil.copy2(test_file['file_path'], dest_path)
+
+        copied_count += 1
+
+    return copied_count