diff --git a/BREAKINGS.md b/BREAKINGS.md index 3b2f80f9..1d87f875 100644 --- a/BREAKINGS.md +++ b/BREAKINGS.md @@ -7,6 +7,7 @@ This document outlines all breaking changes introduced in CTBase v0.18.0-beta co - [Exception System Overhaul](#exception-system-overhaul) - [Module Reorganization](#module-reorganization) - [Extension System Introduction](#extension-system-introduction) +- [TestRunner Enhancements](#testrunner-enhancements) - [API Changes](#api-changes) - [Dependency Updates](#dependency-updates) - [Migration Guide](#migration-guide) @@ -234,6 +235,109 @@ CTBase.Unicode.ctindice(1) --- +## TestRunner Enhancements + +### 🚨 New Extension Features + +The TestRunner extension has been significantly enhanced with advanced capabilities for test execution and progress tracking. + +#### Progress Bar System Changes + +The progress bar implementation has been completely rewritten with adaptive width: + +```julia +# v0.18.0-beta (basic progress) +[████████████░░░░░░░░░░] ✓ [08/19] suite/exceptions/test_display.jl (2.5s) + +# v0.18.0-beta.1 (adaptive width) +[████████░░░░░░░░░░] ✓ [08/19] suite/exceptions/test_display.jl (2.5s) +[████████████████████] ✓ [19/19] suite/exceptions/test_exceptions.jl (0.6s) +``` + +#### Enhanced Failure Detection + +The failure detection mechanism is now more robust and accurate: + +```julia +# v0.18.0-beta (unreliable) +# @test failures sometimes showed as success + +# v0.18.0-beta.1 (robust detection) +# Correctly detects both exceptions and @test assertion failures +``` + +#### Path Prefix Stripping + +Users can now use `test/suite` and `suite` interchangeably: + +```julia +# v0.18.0-beta (explicit paths required) +julia --project -e 'using Pkg; Pkg.test(; test_args=["suite/exceptions"])' + +# v0.18.0-beta.1 (automatic stripping) +julia --project -e 'using Pkg; Pkg.test(; test_args=["test/suite"])' # Same result +``` + +#### Callback System Overhaul + +The callback system has been enhanced with rich context information: + +```julia +# v0.18.0-beta (basic callbacks) +on_test_done = info -> println("Done: $(info.status)") + +# v0.18.0-beta.1 (rich context) +on_test_done = info -> begin + if info.status == :error || info.status == :test_failed + println("❌ FAILED: $(info.spec)") + else + println("✓ Success: $(info.spec)") + end +end +``` + +#### Directory Protection + +The system now prevents ambiguous directory structures: + +```julia +# v0.18.0-beta (allowed) +test/ +├── test/ # This would cause ambiguity with prefix stripping + +# v0.18.0-beta.1 (protected) +ERROR: A subdirectory "test" exists inside the test directory +``` + +### 📚 Documentation Improvements + +#### Docstring Standards Compliance + +All TestRunner functions now follow the project's documentation standards: + +```julia +""" +$(TYPEDSIGNATURES) +Run tests with configurable file/function name builders... + +# Arguments +- `args::AbstractVector{<:AbstractString}`: Command-line arguments... +``` + +#### Cross-Reference Resolution + +All internal references now use fully qualified names to prevent header conflicts: + +```julia +# v0.18.0-beta (conflicts) +See also: [`TestRunInfo`](@ref) # Header conflicts + +# v0.18.0-beta.1 (resolved) +See also: [`TestRunner.TestRunInfo`](@ref) +``` + +--- + ## Dependency Updates ### 🚨 Version Requirements diff --git a/CHANGELOGS.md b/CHANGELOGS.md index ef9dcbd7..486c7ea7 100644 --- a/CHANGELOGS.md +++ b/CHANGELOGS.md @@ -153,6 +153,199 @@ This major release represents a significant investment in code quality, develope --- +## [0.18.0-beta.1] - 2025-02-09 + +### 🚀 TestRunner Enhancements + +#### **Advanced Progress Bar System** +- **Adaptive bar width**: Width equals total for ≤20 tests, fixed at 20 for >20 tests +- **Visual consistency**: Uses `█` (filled) and `░` (empty) characters without gaps +- **Smart failure detection**: Correctly detects both exceptions and `@test` assertion failures +- **Zero-padded indices**: Aligned test numbers for better readability + +#### **Robust Failure Detection** +- **Before/after results scanning**: Compares testset results before and after eval +- **Recursive failure detection**: Scans nested testsets for `Test.Fail` and `Test.Error` +- **More reliable than `anynonpass`**: Works regardless of testset completion timing + +#### **Enhanced User Experience** +- **Path prefix stripping**: Users can write `test/suite` or `suite` interchangeably +- **Guard against conflicts**: Prevents `test/` subdirectory in test directory +- **Comprehensive callbacks**: `on_test_start` and `on_test_done` with `TestRunInfo` context +- **Configurable progress**: Built-in progress bar with option for custom callbacks + +#### **Professional Documentation** +- **Complete docstring overhaul**: All functions follow project documentation standards +- **Safe, runnable examples**: All examples use `julia-repl` with proper imports +- **Cross-reference resolution**: Fully qualified `@ref` links to prevent header conflicts +- **API reference reorganization**: Clean separation in `src/api/` without prefixes + +### 📈 Documentation Improvements + +#### **Better Organization** +- **User guides in `src/guide/`**: Clear separation from API reference +- **API reference in `src/api/`**: Auto-generated with clean filenames +- **Updated navigation**: Changed "Tutorials" to "User Guides" for clarity +- **Fixed cleanup**: Generated API files properly removed after build + +### 🔧 Technical Improvements + +#### **Code Quality** +- **Fixed `@ref` conflicts**: All TestRunner references now use fully qualified names +- **Enhanced error handling**: Better detection and reporting of test failures +- **Improved type safety**: Comprehensive type annotations throughout +- **Memory optimization**: Efficient progress bar rendering and failure detection + +--- + +## [0.18.0-beta] - 2025-02-04 + +### 🚀 Major Features + +#### **Modular Architecture Overhaul** +- **Complete reorganization** of the codebase into thematic modules: + +```text +src/ +├── Core/ # Fundamental types and utilities +├── Exceptions/ # Enhanced error handling system +├── Unicode/ # Unicode character utilities +├── Descriptions/ # Description management +└── Extensions/ # Extension system with tag-based dispatch +``` + +- **Improved maintainability** through clear separation of concerns +- **Better testability** with isolated module boundaries + +#### **Enhanced Exception System** + +- **New exception types** with rich context: + - `PreconditionError`: Replaces `UnauthorizedCall` for state-related errors + - `ParsingError`: For syntax and structure validation errors + - `AmbiguousDescription`: Enhanced with diagnostic capabilities + - `ExtensionError`: Improved with feature and context information +- **Rich error messages** with optional fields for suggestions, context, and diagnostics +- **User-friendly display** with emojis and structured formatting +- **Better debugging experience** with detailed error context + +#### **Professional Extension System** + +- **Tag-based dispatch** for extension points: + - `AbstractDocumenterReferenceTag` / `DocumenterReferenceTag` + - `AbstractCoveragePostprocessingTag` / `CoveragePostprocessingTag` + - `AbstractTestRunnerTag` / `TestRunnerTag` +- **Three core extensions**: + - `DocumenterReference`: API documentation generation + - `CoveragePostprocessing`: Test coverage analysis and reporting + - `TestRunner`: Advanced test execution with glob patterns +- **Clean separation** between extension points and implementations + +#### **Advanced Test Runner** + +- **Glob pattern support** for test selection +- **Configurable filename/function name builders** +- **Dry run mode** for test planning +- **Recursive test discovery** in subdirectories +- **Integration with Julia's test ecosystem** + +### 📈 Enhancements + +#### **Documentation Improvements** + +- **Complete documentation rewrite** with modular tutorials +- **API reference generation** with automatic categorization +- **Enhanced coverage reporting** with visual summaries +- **Professional documentation guides** for developers + +#### **Testing Infrastructure** + +- **Modular test organization** by functionality (not source structure) +- **Comprehensive test coverage** with quality metrics +- **Automated code quality checks** with Aqua.jl +- **Performance and type stability testing** + +#### **Developer Experience** + +- **Improved error messages** with actionable suggestions +- **Better Unicode support** for mathematical notation +- **Enhanced description management** with intelligent completion +- **Professional project structure** following Julia best practices + +### 🔧 Internal Changes + +#### **Code Quality** + +- **Strict adherence** to Julia style guidelines +- **Comprehensive type annotations** for better performance +- **Memory allocation optimization** for critical paths +- **Extensive documentation** for all public APIs + +#### **Testing Standards** + +- **Contract-first testing** methodology +- **Unit and integration test separation** +- **Mock and fake implementations** for isolated testing +- **Deterministic and reproducible tests** + +#### **Performance** + +- **Type stability improvements** throughout the codebase +- **Reduced memory allocations** in hot paths +- **Optimized Unicode operations** +- **Efficient description matching algorithms** + +### 📦 Dependencies + +#### **New Dependencies** + +- `Aqua = "0.8"`: Code quality and consistency checks +- `OrderedCollections = "1"`: Ordered data structures for testing + +#### **Updated Compatibility** + +- Julia `1.10+` (increased from `1.8+`) +- Updated all package compatibility bounds + +### 🏗️ Project Structure + +#### **New Directory Organization** + +```text +src/ +├── Core/ # Fundamental types and utilities +├── Exceptions/ # Enhanced error handling system +├── Unicode/ # Unicode character utilities +├── Descriptions/ # Description management +└── Extensions/ # Extension system + +test/ +├── suite/ # Modular test suites by functionality +├── extras/ # Example tests and diagnostics +└── src/ # Source-level integration tests + +docs/ +└── src/ # Modular documentation tutorials +``` + +#### **Extension System** + +```text +ext/ +├── CoveragePostprocessing.jl # Coverage analysis +├── DocumenterReference.jl # API documentation +└── TestRunner.jl # Advanced test execution +``` + +### 🎯 Breaking Changes + +See [BREAKINGS.md](BREAKINGS.md) for detailed migration guide. + +### 🙏 Acknowledgments + +This major release represents a significant investment in code quality, developer experience, and long-term maintainability. The modular architecture provides a solid foundation for future enhancements while maintaining backward compatibility where possible. + +--- + ## [0.17.4] - Previous Release ### 📝 Previous Changes diff --git a/Project.toml b/Project.toml index 12bc735b..530cc41d 100644 --- a/Project.toml +++ b/Project.toml @@ -1,6 +1,6 @@ name = "CTBase" uuid = "54762871-cc72-4466-b8e8-f6c8b58076cd" -version = "0.18.0-beta" +version = "0.18.0-beta.1" authors = ["Olivier Cots ", "Jean-Baptiste Caillau "] [deps] diff --git a/docs/Project.toml b/docs/Project.toml index 79b107af..9539b6b4 100644 --- a/docs/Project.toml +++ b/docs/Project.toml @@ -1,5 +1,4 @@ [deps] -CTBase = "54762871-cc72-4466-b8e8-f6c8b58076cd" Coverage = "a2441757-f6aa-5fb2-8edb-039e3f45d037" Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4" MarkdownAST = "d0879d2d-cac2-40c8-9cee-1863dc0c7391" diff --git a/docs/api_reference.jl b/docs/api_reference.jl index e2687dec..dbddb18f 100644 --- a/docs/api_reference.jl +++ b/docs/api_reference.jl @@ -16,17 +16,17 @@ function generate_api_reference(src_dir::String) pages = [ CTBase.automatic_reference_documentation(; - subdirectory=".", + subdirectory="api", primary_modules=[CTBase.Core => src(joinpath("Core", "Core.jl"))], exclude=EXCLUDE_SYMBOLS, public=true, private=true, title="Core", title_in_menu="Core", - filename="api_core", + filename="core", ), CTBase.automatic_reference_documentation(; - subdirectory=".", + subdirectory="api", primary_modules=[ CTBase.Descriptions => src( joinpath("Descriptions", "Descriptions.jl"), @@ -43,10 +43,10 @@ function generate_api_reference(src_dir::String) private=true, title="Descriptions", title_in_menu="Descriptions", - filename="api_descriptions", + filename="descriptions", ), CTBase.automatic_reference_documentation(; - subdirectory=".", + subdirectory="api", primary_modules=[ CTBase.Exceptions => src( joinpath("Exceptions", "Exceptions.jl"), @@ -59,20 +59,20 @@ function generate_api_reference(src_dir::String) private=true, title="Exceptions", title_in_menu="Exceptions", - filename="api_exceptions", + filename="exceptions", ), CTBase.automatic_reference_documentation(; - subdirectory=".", + subdirectory="api", primary_modules=[CTBase.Unicode => src(joinpath("Unicode", "Unicode.jl"))], exclude=EXCLUDE_SYMBOLS, public=true, private=false, # there is no private API title="Unicode", title_in_menu="Unicode", - filename="api_unicode", + filename="unicode", ), CTBase.automatic_reference_documentation(; - subdirectory=".", + subdirectory="api", primary_modules=[ CTBase.Extensions => src(joinpath("Extensions", "Extensions.jl")) ], @@ -81,7 +81,7 @@ function generate_api_reference(src_dir::String) private=true, title="Extensions", title_in_menu="Extensions", - filename="api_extensions", + filename="extensions", ), ] @@ -101,7 +101,7 @@ function generate_api_reference(src_dir::String) push!( pages, CTBase.automatic_reference_documentation(; - subdirectory=".", + subdirectory="api", primary_modules=[DocumenterReference => ext("DocumenterReference.jl")], external_modules_to_document=[CTBase], exclude=EXCLUDE_DOCREF, @@ -119,7 +119,7 @@ function generate_api_reference(src_dir::String) push!( pages, CTBase.automatic_reference_documentation(; - subdirectory=".", + subdirectory="api", primary_modules=[ CoveragePostprocessing => ext("CoveragePostprocessing.jl") ], @@ -139,7 +139,7 @@ function generate_api_reference(src_dir::String) push!( pages, CTBase.automatic_reference_documentation(; - subdirectory=".", + subdirectory="api", primary_modules=[TestRunner => ext("TestRunner.jl")], external_modules_to_document=[CTBase], exclude=EXCLUDE_SYMBOLS, @@ -164,39 +164,23 @@ function with_api_reference(f::Function, src_dir::String) try f(pages) finally - # Clean up generated files - # The pages are Pairs: "Title" => "filename.md" (relative to build? no, relative to src) - # automatic_reference_documentation returns "filename" which is relative to docs/src if subdirectory="." - - # We need to reconstruct the full path to delete them. - # Assuming they are in docs/src (which is where makedocs runs from?) - # Wait, makedocs options say subdirectory=".". - # Typically automatic_reference_documentation writes to joinpath(@__DIR__, "src", subdirectory, filename.md) ?? - # I need to check where automatic_reference_documentation writes. - # Assuming we are running from docs/ (where make.jl is). - - # Let's assume the files are in `docs/src`. docs_src = abspath(joinpath(@__DIR__, "src")) + _cleanup_pages(docs_src, pages) + end +end - function cleanup_pages(pages) - for p in pages - content = last(p) - if content isa AbstractString - # file path - filename = content - fname = endswith(filename, ".md") ? filename : filename * ".md" - full_path = joinpath(docs_src, fname) - if isfile(full_path) - rm(full_path) - println("Removed temporary API doc: $full_path") - end - elseif content isa Vector - # nested pages - cleanup_pages(content) - end +function _cleanup_pages(docs_src::String, pages) + for p in pages + val = last(p) + if val isa AbstractString + fname = endswith(val, ".md") ? val : val * ".md" + full_path = joinpath(docs_src, fname) + if isfile(full_path) + rm(full_path) + println("Removed temporary API doc: $full_path") end + elseif val isa AbstractVector + _cleanup_pages(docs_src, val) end - - cleanup_pages(pages) end end diff --git a/docs/make.jl b/docs/make.jl index 01d12b7f..d52d5728 100644 --- a/docs/make.jl +++ b/docs/make.jl @@ -66,12 +66,12 @@ with_api_reference(src_dir) do api_pages ), pages=[ "Introduction" => "index.md", - "Tutorials" => [ - "Descriptions" => "descriptions.md", - "Exceptions" => "exceptions.md", - "Test Runner" => "test-runner.md", - "Coverage" => "coverage.md", - "API Documentation" => "api-documentation.md", + "User Guides" => [ + "Descriptions" => joinpath("guide", "descriptions.md"), + "Exceptions" => joinpath("guide", "exceptions.md"), + "Test Runner" => joinpath("guide", "test-runner.md"), + "Coverage" => joinpath("guide", "coverage.md"), + "API Documentation" => joinpath("guide", "api-documentation.md"), ], "API Reference" => api_pages, ], diff --git a/docs/src/exceptions_new.md b/docs/src/exceptions_new.md deleted file mode 100644 index cb74f341..00000000 --- a/docs/src/exceptions_new.md +++ /dev/null @@ -1,22 +0,0 @@ -# Error Handling and CTBase Exceptions - -CTBase defines a small hierarchy of domain-specific exceptions to make error -handling explicit and consistent across the control-toolbox ecosystem. - -All custom exceptions inherit from `CTBase.CTException`: - -```julia -abstract type CTBase.CTException <: Exception end -``` - -## Exception Hierarchy - -``` -CTException (abstract) -├── IncorrectArgument # Input validation errors -├── PreconditionError # Order of operations, state validation -├── NotImplemented # Unimplemented interface methods -├── ParsingError # Parsing errors -├── AmbiguousDescription # Ambiguous or incorrect descriptions -└── ExtensionError # Missing optional dependencies -``` diff --git a/docs/src/api-documentation.md b/docs/src/guide/api-documentation.md similarity index 100% rename from docs/src/api-documentation.md rename to docs/src/guide/api-documentation.md diff --git a/docs/src/coverage.md b/docs/src/guide/coverage.md similarity index 100% rename from docs/src/coverage.md rename to docs/src/guide/coverage.md diff --git a/docs/src/descriptions.md b/docs/src/guide/descriptions.md similarity index 100% rename from docs/src/descriptions.md rename to docs/src/guide/descriptions.md diff --git a/docs/src/exceptions.md b/docs/src/guide/exceptions.md similarity index 100% rename from docs/src/exceptions.md rename to docs/src/guide/exceptions.md diff --git a/docs/src/guide/test-runner.md b/docs/src/guide/test-runner.md new file mode 100644 index 00000000..df4bcf01 --- /dev/null +++ b/docs/src/guide/test-runner.md @@ -0,0 +1,376 @@ +# Test Runner Guide + +This guide explains how to set up a modular testing infrastructure for Julia packages using the **TestRunner** extension of `CTBase.jl`. This setup enables granular test execution and is friendly both for human developers and AI agents. + +## Architecture Overview + +A robust testing architecture typically involves: + +1. **Test Runner**: A `runtests.jl` file that allows running specific test groups via command-line arguments. +2. **Test Suite Structure**: Modular test files, each containing a main entry point function. + +### Recommended Directory Structure + +We recommend placing your tests in a `suite` subdirectory to keep the top-level `test/` folder clean. + +```text +MyPackage.jl/ +├── src/ +│ └── ... +├── test/ +│ ├── runtests.jl # Main test runner +│ └── suite/ # Directory containing test files +│ ├── test_utils.jl +│ ├── test_core.jl +│ └── ... +└── ... +``` + +!!! warning "Restriction" + The `test/` directory **must not** contain a subdirectory named `test`. + This would conflict with the automatic `test/` prefix stripping (see [Path prefix stripping](@ref)). + +## Setting up `runtests.jl` + +The `runtests.jl` file is the entry point for your test suite. By using `CTBase.run_tests`, you enable a powerful mechanism to filter and execute specific tests using command-line arguments. This is crucial for fast iteration cycles. + +### Example `test/runtests.jl` + +```julia +# Load dependencies +using Test +using CTBase # Provides run_tests +using MyPackage # Your package + +# Define where your tests are located +const TEST_DIR = @__DIR__ + +# Run tests using the CTBase test runner +CTBase.run_tests(; + args=String.(ARGS), # Pass command line arguments + testset_name="MyPackage Tests", # Name of the main testset + available_tests=[ # List of available test groups/files + "suite/*" # Use glob pattern to include all tests in suite/ + ], + filename_builder = name -> "test_$(name).jl", + funcname_builder = name -> "test_$(name)", + test_dir=TEST_DIR, # Directory containing test files +) +``` + +### Keyword Arguments + +| Argument | Type | Default | Description | +| :------- | :--- | :------ | :---------- | +| `args` | `AbstractVector{<:AbstractString}` | `String[]` | Command-line arguments (typically `String.(ARGS)`) | +| `testset_name` | `String` | `"Tests"` | Name of the main `@testset` | +| `available_tests` | `Vector` | `Symbol[]` | Allowed tests (Symbols, Strings, or glob patterns). Empty = auto-discovery | +| `filename_builder` | `Function` | `identity` | `name → filename` mapping | +| `funcname_builder` | `Function` | `identity` | `name → function_name` mapping (return `nothing` to skip eval) | +| `eval_mode` | `Bool` | `true` | Whether to call the function after `include` | +| `verbose` | `Bool` | `true` | Verbose `@testset` output | +| `showtiming` | `Bool` | `true` | Show timing in `@testset` output | +| `test_dir` | `String` | `joinpath(pwd(), "test")` | Root directory for test files | +| `on_test_start` | `Function` or `nothing` | `nothing` | Callback before eval (see [Callbacks](@ref)) | +| `on_test_done` | `Function` or `nothing` | `nothing` | Callback after eval (see [Callbacks](@ref)) | +| `progress` | `Bool` | `true` | Show built-in progress bar. Ignored when `on_test_done` is provided | + +## Writing Test Files + +To support the modular execution model, each test file should define a function (typically matching the filename) that contains the tests. This avoids scope pollution and makes the tests easy to invoke programmatically. + +### Example `test/suite/test_utils.jl` + +```julia +# The function name matches the `funcname_builder` logic in runtests.jl +function test_utils() + @testset "Utilities" begin + @test MyPackage.add(1, 1) == 2 + @test MyPackage.sub(2, 1) == 1 + end +end +``` + +## Running Tests + +### Default (all available tests) + +```bash +julia --project -e 'using Pkg; Pkg.test("MyPackage")' +``` + +### Run a specific test group + +```bash +julia --project -e 'using Pkg; Pkg.test("MyPackage"; test_args=["utils"])' +julia --project -e 'using Pkg; Pkg.test("MyPackage"; test_args=["core", "utils"])' +``` + +### Path prefix stripping + +Selection arguments starting with `test/` are **automatically stripped**, so the following are equivalent: + +```bash +# These two commands run the same tests: +julia --project -e 'using Pkg; Pkg.test("MyPackage"; test_args=["suite"])' +julia --project -e 'using Pkg; Pkg.test("MyPackage"; test_args=["test/suite"])' +``` + +This is convenient when tab-completing paths from the project root. + +### CLI flags + +- `-a` or `--all` — run all available tests (same as no arguments). +- `-n` or `--dryrun` — print the list of tests that would run, without executing them. +- Coverage flags (`coverage=true`, `--coverage`, etc.) are silently filtered out. + +### Directory selection + +Bare directory names are automatically expanded. For example, `suite/exceptions` is treated as `suite/exceptions/*`, selecting all test files in that directory. + +## Progress Bar + +By default, `run_tests` displays a progress bar after each test completes: + +```text +[████████░░░░░░░░░░░░] ✓ [08/19] suite/exceptions/test_display.jl (2.5s) +[█████████░░░░░░░░░░░] ✓ [09/19] suite/exceptions/test_exceptions.jl (0.6s) +[██████████░░░░░░░░░░] ✗ [10/19] suite/exceptions/test_types.jl FAILED, (0.9s) +``` + +### Visual elements + +- **Bar**: `█` (filled) and `░` (empty), enclosed in `[…]` +- **Status symbol**: `✓` (green, success), `✗` (red, failure/error), `○` (yellow, skipped) +- **Index**: zero-padded `[01/19]` for alignment +- **Spec**: test identifier (relative path or symbol name) +- **Time**: wall-clock elapsed time for the eval phase +- **FAILED**: appended in bold red when the test failed + +### Adaptive bar width + +The bar width adapts to the number of tests: + +- **≤ 20 tests**: width equals the total number of tests (one block per test). +- **> 20 tests**: fixed width of 20 characters. Some tests will not visually advance the bar (the fill is computed as `round(Int, index / total * 20)`). + +### Failure detection + +The progress bar correctly detects **both** types of failures: + +- **Exceptions**: errors thrown during test execution (caught via `try/catch`). +- **`@test` assertion failures**: detected by scanning the enclosing `DefaultTestSet` results before and after eval. This is more reliable than checking `anynonpass`, which is only updated when a testset finishes. + +### Disabling the progress bar + +Set `progress=false` to disable the built-in progress display: + +```julia +CTBase.run_tests(; args=String.(ARGS), progress=false) +``` + +The progress bar is also automatically disabled when a custom `on_test_done` callback is provided. + +## Callbacks + +The `on_test_start` and `on_test_done` callbacks allow custom actions during the test lifecycle. Both receive a `TestRunInfo` struct. + +### `TestRunInfo` + +```julia +struct TestRunInfo + spec::Union{Symbol,String} # Test identifier + filename::String # Absolute path of the test file + func_symbol::Union{Symbol,Nothing} # Function to call (nothing if eval_mode=false) + index::Int # 1-based index in the selected list + total::Int # Total number of selected tests + status::Symbol # See below + error::Union{Exception,Nothing} # Captured exception (only when status == :error) + elapsed::Union{Float64,Nothing} # Wall-clock seconds (only in on_test_done) +end +``` + +### Status values + +| Status | When | Callback | +| :----- | :--- | :------- | +| `:pre_eval` | After `include`, before `eval` | `on_test_start` | +| `:post_eval` | After successful `eval` | `on_test_done` | +| `:test_failed` | After `eval` with `@test` failures (no exception) | `on_test_done` | +| `:error` | After `eval` raised an exception | `on_test_done` | +| `:skipped` | When `eval_mode=false` or `on_test_start` returned `false` | `on_test_done` | + +### `on_test_start` + +Called after the test file is included but before the function is evaluated. Must return a `Bool`: + +- `true` — proceed with eval. +- `false` — skip eval (triggers `on_test_done` with `:skipped`). + +### `on_test_done` + +Called after eval completes (or after skip/error). The built-in progress bar is implemented as a default `on_test_done` callback. + +### Example: custom callbacks + +```julia +CTBase.run_tests(; + args=String.(ARGS), + on_test_start = info -> begin + print(" [$(info.index)/$(info.total)] $(info.spec)...") + return true # proceed with eval + end, + on_test_done = info -> begin + if info.status == :post_eval + println(" ✓ ($(round(info.elapsed; digits=1))s)") + elseif info.status == :error || info.status == :test_failed + println(" ✗ FAILED") + elseif info.status == :skipped + println(" ○ skipped") + end + end, +) +``` + +## Advanced Usage + +### Filtering Tests with Glob Patterns + +You can use glob patterns to organize tests hierarchically: + +```julia +CTBase.run_tests(; + args=String.(ARGS), + testset_name="MyPackage Tests", + available_tests=[ + "suite/core/*", # All core tests + "suite/utils/*", # All utility tests + "suite/integration/*" # All integration tests + ], + # ... +) +``` + +Selection arguments are matched against multiple representations of each candidate: + +- The candidate name (e.g. `:utils`) +- The full relative path (e.g. `suite/test_utils.jl`) +- The path without `.jl` extension +- The basename (e.g. `test_utils.jl`) +- The basename without extension +- The basename without `test_` prefix (e.g. `utils`) + +### Custom Test Options + +Pass custom options to your test suite: + +```julia +# In runtests.jl +const VERBOSE = "--verbose" in ARGS +const SHOWTIMING = "--timing" in ARGS + +# In test files +function test_utils() + @testset "Utilities" verbose=VERBOSE showtiming=SHOWTIMING begin + # tests here + end +end +``` + +## Debugging Test Failures + +### Common Issues and Solutions + +#### Issue: Test function not found + +**Error**: `UndefVarError: test_utils not defined` + +**Solution**: Ensure your test file exports the test function to the outer scope: + +```julia +# At the end of test/suite/test_utils.jl +test_utils() = TestUtils.test_utils() # Export to outer scope +``` + +#### Issue: Module conflicts + +**Error**: `WARNING: replacing module TestUtils` + +**Solution**: Use unique module names for each test file: + +```julia +module TestUtilsModule # Unique name +using Test +using MyPackage + +function test_utils() + @testset "Utilities" begin + # tests + end +end + +end # module + +test_utils() = TestUtilsModule.test_utils() +``` + +#### Issue: Tests not discovered + +**Error**: No tests run when specifying a test name + +**Solution**: Check that your `filename_builder` and `funcname_builder` match your file structure: + +```julia +# If your files are named "utils_test.jl" +filename_builder = name -> "$(name)_test.jl" + +# If your functions are named "run_utils_tests" +funcname_builder = name -> "run_$(name)_tests" +``` + +#### Issue: `@test` failure shown as success + +The progress bar detects `@test` failures by scanning the enclosing testset results before and after eval. If you see a `✓` for a test that should have failed, make sure the test function is wrapped in a `@testset` block so that failures are recorded in the results. + +### Debugging with Verbose Output + +Run tests with verbose output to see detailed information: + +```bash +julia --project -e 'using Pkg; Pkg.test("MyPackage"; test_args=["--verbose", "utils"])' +``` + +## Best Practices + +1. **One test function per file**: Keep test files focused and easy to navigate +2. **Use descriptive names**: Name test files and functions clearly (e.g., `test_optimization.jl`, `test_optimization()`) +3. **Organize by feature**: Group related tests in subdirectories +4. **Fast tests first**: Place quick unit tests before slow integration tests +5. **Isolate test state**: Each test should be independent and not rely on execution order +6. **Use test fixtures**: Create helper functions for common test setup +7. **Document test requirements**: Note any special dependencies or setup needed +8. **No `test/` subdirectory in `test/`**: Avoid naming a subdirectory `test` inside your test directory + +## Integration with CI/CD + +### GitHub Actions Example + +```yaml +name: Tests +on: [push, pull_request] +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + - uses: julia-actions/setup-julia@v1 + - name: Run all tests + run: julia --project -e 'using Pkg; Pkg.test()' + - name: Run specific test group + run: julia --project -e 'using Pkg; Pkg.test(test_args=["core"])' +``` + +## See Also + +- [Exception Handling](exceptions.md): Understanding test failures and exceptions +- [Coverage Guide](coverage.md): Measuring test coverage diff --git a/docs/src/index.md b/docs/src/index.md index 8cd9681d..d9beeb71 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -12,15 +12,15 @@ It provides the core types, utilities, and infrastructure used by other packages The root package is [OptimalControl.jl](https://github.com/control-toolbox/OptimalControl.jl) which aims to provide tools to model and solve optimal control problems with ordinary differential equations by direct and indirect methods, both on CPU and GPU. -## Features and Tutorials +## Features and User Guides CTBase provides several key features to build robust control-toolbox packages: -- **[Descriptions: encoding algorithms](descriptions.md)**: A declarative way to encode algorithms or configurations using tuples of symbols. -- **[Error handling and Exceptions](exceptions.md)**: A domain-specific exception hierarchy for consistent error reporting. -- **[Test Runner](test-runner.md)**: A modular test runner for granular test execution. -- **[Coverage Post-processing](coverage.md)**: Tools to generate readable coverage reports. -- **[API Documentation Generation](api-documentation.md)**: Automated API reference generation from docstrings. +- **[Descriptions: encoding algorithms](guide/descriptions.md)**: A declarative way to encode algorithms or configurations using tuples of symbols. +- **[Error handling and Exceptions](guide/exceptions.md)**: A domain-specific exception hierarchy for consistent error reporting. +- **[Test Runner](guide/test-runner.md)**: A modular test runner for granular test execution. +- **[Coverage Post-processing](guide/coverage.md)**: Tools to generate readable coverage reports. +- **[API Documentation Generation](guide/api-documentation.md)**: Automated API reference generation from docstrings. ## Note on Private Methods diff --git a/docs/src/test-runner.md b/docs/src/test-runner.md deleted file mode 100644 index e233e1c0..00000000 --- a/docs/src/test-runner.md +++ /dev/null @@ -1,224 +0,0 @@ -# Test Runner Guide - -This guide explains how to set up a modular testing infrastructure for Julia packages using the **TestRunner** extension of `CTBase.jl`. This setup enables granular test execution and is friendly both for human developers and AI agents. - -## Architecture Overview - -A robust testing architecture typically involves: - -1. **Test Runner**: A `runtests.jl` file that allows running specific test groups via command-line arguments. -2. **Test Suite Structure**: Modular test files, each containing a main entry point function. - -### Recommended Directory Structure - -We recommend placing your tests in a `suite` subdirectory to keep the top-level `test/` folder clean. - -```text -MyPackage.jl/ -├── src/ -│ └── ... -├── test/ -│ ├── runtests.jl # Main test runner -│ └── suite/ # Directory containing test files -│ ├── test_utils.jl -│ ├── test_core.jl -│ └── ... -└── ... -``` - -## Setting up `runtests.jl` - -The `runtests.jl` file is the entry point for your test suite. By using `CTBase.run_tests`, you enable a powerful mechanism to filter and execute specific tests using command-line arguments. This is crucial for fast iteration cycles. - -### Example `test/runtests.jl` - -```julia -# Load dependencies -using Test -using CTBase # Provides run_tests -using MyPackage # Your package - -# Define where your tests are located -const TEST_DIR = @__DIR__ - -# Run tests using the CTBase test runner -CTBase.run_tests(; - args=String.(ARGS), # Pass command line arguments - testset_name="MyPackage Tests", # Name of the main testset - available_tests=[ # List of available test groups/files - "suite/*" # Use glob pattern to include all tests in suite/ - ], - # Function to map a test name in ARGS (like "utils") to a filename - filename_builder = name -> "test_$(name).jl", - # Function to map a test name in ARGS to the function to call inside that file - funcname_builder = name -> "test_$(name)", - test_dir=TEST_DIR, # Directory containing test files - verbose=true, # Show verbose output - showtiming=true, # Show timing information -) -``` - -## Writing Test Files - -To support the modular execution model, each test file should define a function (typically matching the filename) that contains the tests. This avoids scope pollution and makes the tests easy to invoke programmatically. - -### Example `test/suite/test_utils.jl` - -```julia -# The function name matches the `funcname_builder` logic in runtests.jl -function test_utils() - @testset "Utilities" begin - @test MyPackage.add(1, 1) == 2 - @test MyPackage.sub(2, 1) == 1 - end -end -``` - -## Running Tests - -### Default (all available tests) - -```bash -julia --project -e 'using Pkg; Pkg.test("MyPackage")' -``` - -### Run a specific test group - -```bash -julia --project -e 'using Pkg; Pkg.test("MyPackage"; test_args=["utils"])' -julia --project -e 'using Pkg; Pkg.test("MyPackage"; test_args=["core", "utils"])' -``` - -Note: - -- Passing `-a` or `--all` is equivalent to running without arguments. -- Passing `--dry-run` will print the list of tests that would be run, but not execute them. - -## Advanced Usage - -### Filtering Tests with Glob Patterns - -You can use glob patterns to organize tests hierarchically: - -```julia -CTBase.run_tests(; - args=String.(ARGS), - testset_name="MyPackage Tests", - available_tests=[ - "suite/core/*", # All core tests - "suite/utils/*", # All utility tests - "suite/integration/*" # All integration tests - ], - # ... -) -``` - -### Custom Test Options - -Pass custom options to your test suite: - -```julia -# In runtests.jl -const VERBOSE = "--verbose" in ARGS -const SHOWTIMING = "--timing" in ARGS - -# In test files -function test_utils() - @testset "Utilities" verbose=VERBOSE showtiming=SHOWTIMING begin - # tests here - end -end -``` - -## Debugging Test Failures - -### Common Issues and Solutions - -#### Issue: Test function not found - -**Error**: `UndefVarError: test_utils not defined` - -**Solution**: Ensure your test file exports the test function to the outer scope: - -```julia -# At the end of test/suite/test_utils.jl -test_utils() = TestUtils.test_utils() # Export to outer scope -``` - -#### Issue: Module conflicts - -**Error**: `WARNING: replacing module TestUtils` - -**Solution**: Use unique module names for each test file: - -```julia -module TestUtilsModule # Unique name -using Test -using MyPackage - -function test_utils() - @testset "Utilities" begin - # tests - end -end - -end # module - -test_utils() = TestUtilsModule.test_utils() -``` - -#### Issue: Tests not discovered - -**Error**: No tests run when specifying a test name - -**Solution**: Check that your `filename_builder` and `funcname_builder` match your file structure: - -```julia -# If your files are named "utils_test.jl" -filename_builder = name -> "$(name)_test.jl" - -# If your functions are named "run_utils_tests" -funcname_builder = name -> "run_$(name)_tests" -``` - -### Debugging with Verbose Output - -Run tests with verbose output to see detailed information: - -```bash -julia --project -e 'using Pkg; Pkg.test("MyPackage"; test_args=["--verbose", "utils"])' -``` - -## Best Practices - -1. **One test function per file**: Keep test files focused and easy to navigate -2. **Use descriptive names**: Name test files and functions clearly (e.g., `test_optimization.jl`, `test_optimization()`) -3. **Organize by feature**: Group related tests in subdirectories -4. **Fast tests first**: Place quick unit tests before slow integration tests -5. **Isolate test state**: Each test should be independent and not rely on execution order -6. **Use test fixtures**: Create helper functions for common test setup -7. **Document test requirements**: Note any special dependencies or setup needed - -## Integration with CI/CD - -### GitHub Actions Example - -```yaml -name: Tests -on: [push, pull_request] -jobs: - test: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v2 - - uses: julia-actions/setup-julia@v1 - - name: Run all tests - run: julia --project -e 'using Pkg; Pkg.test()' - - name: Run specific test group - run: julia --project -e 'using Pkg; Pkg.test(test_args=["core"])' -``` - -## See Also - -- [Exception Handling](exceptions.md): Understanding test failures and exceptions -- [Coverage Guide](coverage.md): Measuring test coverage diff --git a/ext/TestRunner.jl b/ext/TestRunner.jl index 1878be6e..52b4dfb1 100644 --- a/ext/TestRunner.jl +++ b/ext/TestRunner.jl @@ -10,37 +10,126 @@ running testsets). module TestRunner using CTBase: CTBase +using DocStringExtensions using Test: Test, @testset -const TestSpec = Union{Symbol,String} - """ - run_tests(::CTBase.Extensions.TestRunnerTag; kwargs...) +$(TYPEDEF) -Run tests with configurable file/function name builders and optional available tests filter. +Union type representing a test specification. + +A test spec can be either: +- `Symbol`: A logical test name (e.g., `:utils`, `:core`) +- `String`: A relative file path or glob pattern (e.g., `"suite/test_utils.jl"`, `"suite/core/*"`) -# Keyword Arguments -- `testset_name::String = "Tests"` — name of the main testset -- `available_tests::Vector{Symbol} = Symbol[]` — if non-empty, only these tests are allowed -- `filename_builder::Function = identity` — `Symbol → Symbol`, builds the filename from the test name -- `funcname_builder::Function = identity` — `Symbol → Symbol|Nothing`, builds the function name (or nothing to skip eval) -- `eval_mode::Bool = true` — whether to eval the function after include -- `verbose::Bool = true` — verbose testset output -- `showtiming::Bool = true` — show timing in testset output +This type is used throughout TestRunner to represent both user-provided selections +and internal test identifiers. # Notes +- Symbol specs are resolved via `filename_builder` and `funcname_builder` +- String specs are treated as relative paths from `test_dir` +- Glob patterns are supported for String specs + +See also: [`CTBase.run_tests`](@ref), [`TestRunner._select_tests`](@ref) +""" +const TestSpec = Union{Symbol,String} + +""" +$(TYPEDEF) + +Context information passed to test callbacks (`on_test_start`, `on_test_done`). + +Provides details about the current test being executed, including progress +information (`index`, `total`) and execution results (`status`, `error`, `elapsed`). + +# Fields +- `spec::TestSpec`: test identifier (Symbol or relative path String) +- `filename::String`: absolute path of the included test file +- `func_symbol::Union{Symbol,Nothing}`: function to call (`nothing` if `eval_mode=false`) +- `index::Int`: 1-based index of the current test in the selected list +- `total::Int`: total number of selected tests +- `status::Symbol`: one of `:pre_eval`, `:post_eval`, `:skipped`, `:error`, `:test_failed` +- `error::Union{Exception,Nothing}`: captured exception when `status == :error` +- `elapsed::Union{Float64,Nothing}`: wall-clock seconds for the eval phase (only in `on_test_done`) + +# Example +```julia-repl +julia> using CTBase.TestRunner + +julia> info = TestRunner.TestRunInfo( + :utils, + "/path/to/test_utils.jl", + :test_utils, + 3, 10, + :post_eval, + nothing, + 1.23 + ) +TestRunner.TestRunInfo(:utils, "/path/to/test_utils.jl", :test_utils, 3, 10, :post_eval, nothing, 1.23) + +julia> info.status +:post_eval + +julia> info.elapsed +1.23 +``` +""" +struct TestRunInfo + spec::TestSpec + filename::String + func_symbol::Union{Symbol,Nothing} + index::Int + total::Int + status::Symbol + error::Union{Exception,Nothing} + elapsed::Union{Float64,Nothing} +end -- Test selection is driven by `Main.ARGS` (coverage flags are ignored). -- Selection arguments are interpreted as glob patterns and matched against both - the test name and the corresponding filename. +""" +$(TYPEDSIGNATURES) -# Usage sketch (non-executed) +Run tests with configurable file/function name builders and optional available tests filter. -```julia -using CTBase +# Arguments +- `::CTBase.Extensions.TestRunnerTag`: Dispatch tag for the TestRunner extension +- `args::AbstractVector{<:AbstractString}`: Command-line arguments (typically `String.(ARGS)`) +- `testset_name::String`: Name of the main testset (default: `"Tests"`) +- `available_tests`: Allowed tests (Symbols, Strings, or glob patterns). Empty = auto-discovery +- `filename_builder::Function`: `name → filename` mapping (default: `identity`) +- `funcname_builder::Function`: `name → function_name` mapping (default: `identity`) +- `eval_mode::Bool`: Whether to call the function after include (default: `true`) +- `verbose::Bool`: Verbose `@testset` output (default: `true`) +- `showtiming::Bool`: Show timing in `@testset` output (default: `true`) +- `test_dir::String`: Root directory for test files (default: `joinpath(pwd(), "test")`) +- `on_test_start::Union{Function,Nothing}`: Callback before eval (default: `nothing`) +- `on_test_done::Union{Function,Nothing}`: Callback after eval (default: `nothing`) +- `progress::Bool`: Show built-in progress bar (default: `true`) + +# Returns +- `Nothing`: Tests are executed via side effects -# CTBase.run_tests(; testset_name="Tests") +# Notes +- Test selection is driven by `args` (coverage flags are automatically filtered out) +- Selection arguments are interpreted as glob patterns and matched against both test names and filenames +- Arguments starting with `test/` are automatically stripped for convenience +- When `on_test_done` is provided, the built-in progress bar is disabled unless `progress=true` + +# Example +```julia-repl +julia> using CTBase.TestRunner + +julia> # Run all tests with default settings +julia> CTBase.run_tests() + +julia> # Run specific tests with custom callbacks +julia> CTBase.run_tests(; + args=["utils", "core"], + on_test_start = info -> (println("Running: ", info.spec); true), + on_test_done = info -> println("Done: ", info.status) + ) ``` + +See also: [`TestRunner.TestRunInfo`](@ref), [`TestRunner._parse_test_args`](@ref), [`TestRunner._select_tests`](@ref) """ function CTBase.run_tests( ::CTBase.Extensions.TestRunnerTag; @@ -53,7 +142,21 @@ function CTBase.run_tests( verbose::Bool=true, showtiming::Bool=true, test_dir::String=joinpath(pwd(), "test"), + on_test_start::Union{Function,Nothing}=nothing, + on_test_done::Union{Function,Nothing}=nothing, + progress::Bool=true, ) + # Guard: a subdirectory named "test" inside test_dir would conflict with + # the automatic `test/` prefix stripping in _parse_test_args. + if isdir(joinpath(test_dir, "test")) + error( + "A subdirectory \"test\" exists inside the test directory " * + "\"$(test_dir)\". This is not supported because selection " * + "arguments starting with \"test/\" are automatically stripped " * + "(e.g. \"test/suite\" → \"suite\"). Please rename the subdirectory." + ) + end + # Parse command-line arguments (selections, run_all, dry_run) = _parse_test_args(String.(args)) @@ -70,8 +173,19 @@ function CTBase.run_tests( return nothing end + total = length(selected) + + # Wire up default progress callback when no custom on_test_done is provided + effective_on_test_done = if on_test_done !== nothing + on_test_done + elseif progress + _default_on_test_done + else + nothing + end + @testset verbose = verbose showtiming = showtiming "$testset_name" begin - for spec in selected + for (idx, spec) in enumerate(selected) @testset "$(spec)" begin _run_single_test( spec; @@ -80,6 +194,10 @@ function CTBase.run_tests( funcname_builder, eval_mode, test_dir, + index=idx, + total, + on_test_start, + on_test_done=effective_on_test_done, ) end end @@ -87,15 +205,34 @@ function CTBase.run_tests( end """ - _parse_test_args(args::Vector{String}) -> Tuple{Vector{Symbol}, Bool, Bool} +$(TYPEDSIGNATURES) Parse command-line test arguments, filtering out coverage-related flags. -Returns `(selections, run_all, dry_run)` where: +# Arguments +- `args::Vector{String}`: Raw command-line arguments + +# Returns +- `Tuple{Vector{String}, Bool, Bool}`: `(selections, run_all, dry_run)` where: + - `selections`: selection patterns provided by the user (as strings) + - `run_all`: whether `-a` / `--all` was present + - `dry_run`: whether `-n` / `--dryrun` was present + +# Notes +- Coverage flags (`coverage=true`, `--coverage`, etc.) are automatically filtered out +- Selection patterns starting with `test/` or `test\\` are automatically stripped + so that users can write `test/suite/foo` or `suite/foo` interchangeably + +# Example +```julia-repl +julia> using CTBase.TestRunner + +julia> TestRunner._parse_test_args(["utils", "-a", "--dryrun"]) +(["utils"], true, true) -- `selections`: selection patterns provided by the user (as symbols) -- `run_all`: whether `-a` / `--all` was present -- `dry_run`: whether `-n` / `--dryrun` was present +julia> TestRunner._parse_test_args(["test/suite", "coverage=true"]) +(["suite"], false, false) +``` """ function _parse_test_args(args::Vector{String}) selections = String[] @@ -110,18 +247,124 @@ function _parse_test_args(args::Vector{String}) elseif arg == "-n" || arg == "--dryrun" dry_run = true else - push!(selections, arg) + push!(selections, _strip_test_prefix(arg)) end end return (selections, run_all, dry_run) end """ - _glob_to_regex(pattern::AbstractString) -> Regex +$(TYPEDSIGNATURES) + +Strip a leading `test/` or `test\\` prefix from a selection pattern. + +This allows users to type `test/suite/foo` instead of `suite/foo` since +the test directory is already the root for pattern matching. + +# Arguments +- `s::AbstractString`: Selection pattern to process + +# Returns +- `String`: Pattern with `test/` or `test\\` prefix stripped (if present) + +# Example +```julia-repl +julia> using CTBase.TestRunner + +julia> TestRunner._strip_test_prefix("test/suite/foo") +"suite/foo" + +julia> TestRunner._strip_test_prefix("suite/foo") +"suite/foo" + +julia> TestRunner._strip_test_prefix("test\\windows\\path") +"windows\\path" +``` +""" +function _strip_test_prefix(s::AbstractString) + for prefix in ("test/", "test\\") + if startswith(s, prefix) + return String(s[length(prefix)+1:end]) + end + end + return String(s) +end + +""" +$(TYPEDSIGNATURES) + +Normalize user-provided selection patterns before glob matching. + +Applied transformations: +- Strip trailing `/` (e.g. `"suite/exceptions/"` → `"suite/exceptions"`) +- If a selection contains no glob wildcard (`*` or `?`) and matches a directory + prefix of at least one candidate, expand it to `"selection/*"` so that all + files under that directory are selected. + +The original selection is always kept so that exact-name matches still work. + +# Arguments +- `selections::Vector{String}`: User-provided selection patterns +- `candidates::Vector{<:TestSpec}`: Available test candidates + +# Returns +- `Vector{String}`: Normalized selection patterns + +# Example +```julia-repl +julia> using CTBase.TestRunner + +julia> TestRunner._normalize_selections( + ["suite/"], + ["suite/test_a.jl", "suite/test_b.jl"] + ) +2-element Vector{String}: + "suite" + "suite/*" +``` +""" +function _normalize_selections(selections::Vector{String}, candidates::Vector{<:TestSpec}) + candidate_strs = [c isa Symbol ? String(c) : String(c) for c in candidates] + normalized = String[] + for sel in selections + # Strip trailing slash(es) + s = rstrip(sel, '/') + push!(normalized, s) + # If no glob wildcard, check if it looks like a directory prefix + if !occursin('*', s) && !occursin('?', s) + prefix = s * "/" + is_dir_prefix = any(c -> startswith(c, prefix), candidate_strs) + if is_dir_prefix + push!(normalized, s * "/*") + end + end + end + return unique(normalized) +end + +""" +$(TYPEDSIGNATURES) Convert a glob pattern (using `*` and `?`) into a regular expression. The returned regex is anchored (matches the full string). + +# Arguments +- `pattern::AbstractString`: Glob pattern to convert + +# Returns +- `Regex`: Anchored regular expression equivalent to the glob pattern + +# Example +```julia-repl +julia> using CTBase.TestRunner + +julia> TestRunner._glob_to_regex("test_*.jl") +r"^test_.*\\.jl\$" + +julia> TestRunner._glob_to_regex("suite/test_?.jl") +r"^suite/test.\\.jl\$" +``` """ function _glob_to_regex(pattern::AbstractString) # Escape special regex characters except * and ? @@ -145,14 +388,89 @@ function _glob_to_regex(pattern::AbstractString) return Regex("^" * regex_str * "\u0024") end +""" + _ensure_jl(filename::AbstractString) -> String + +Ensure that a filename ends with `.jl` extension. + +If the filename already ends with `.jl`, returns it unchanged. +Otherwise, appends `.jl` to the filename. + +# Arguments +- `filename::AbstractString`: Base filename with or without `.jl` extension + +# Returns +- `String`: Filename guaranteed to end with `.jl` + +# Example +```julia +julia> TestRunner._ensure_jl("test_utils") +"test_utils.jl" + +julia> TestRunner._ensure_jl("test_utils.jl") +"test_utils.jl" +``` +""" function _ensure_jl(filename::AbstractString) endswith(filename, ".jl") ? filename : filename * ".jl" end +""" + _builder_to_string(x) -> String + +Convert a Symbol or String to String. + +This helper function ensures that builder function outputs are always +converted to strings for consistent handling. + +# Arguments +- `x`: Symbol or String to convert + +# Returns +- `String`: The string representation of `x` + +# Example +```julia +julia> TestRunner._builder_to_string(:utils) +"utils" + +julia> TestRunner._builder_to_string("utils") +"utils" +``` +""" function _builder_to_string(x) x isa Symbol ? String(x) : String(x) end +""" + _normalize_available_tests(available_tests) -> Vector{TestSpec} + +Normalize and validate the `available_tests` argument. + +Converts the input to a `Vector{TestSpec}` and validates that all entries +are either `Symbol` or `String`. Returns an empty vector if `available_tests` +is `nothing`. + +# Arguments +- `available_tests`: `nothing`, `Vector`, or `Tuple` containing `Symbol` or `String` entries + +# Returns +- `Vector{TestSpec}`: Normalized vector of test specifications + +# Throws +- `ArgumentError`: If `available_tests` is not a Vector/Tuple or contains invalid entries + +# Example +```julia +julia> TestRunner._normalize_available_tests([:utils, "suite/*"]) +2-element Vector{Union{Symbol, String}}: + :utils + "suite/*" + +julia> TestRunner._normalize_available_tests(nothing) +Union{Symbol, String}[] +``` +""" function _normalize_available_tests(available_tests) available_tests === nothing && return TestSpec[] @@ -171,6 +489,32 @@ function _normalize_available_tests(available_tests) return out end +""" + _collect_test_files_recursive(test_dir::AbstractString) -> Vector{String} + +Recursively collect all `.jl` files in `test_dir` (excluding `runtests.jl`). + +Returns relative paths from `test_dir`, sorted alphabetically. + +# Arguments +- `test_dir::AbstractString`: Root directory to search + +# Returns +- `Vector{String}`: Relative paths to all `.jl` files (excluding `runtests.jl`) + +# Example +```julia +# Assuming test_dir contains: +# - test/utils.jl +# - test/core/test_core.jl +# - test/runtests.jl + +julia> TestRunner._collect_test_files_recursive("test") +2-element Vector{String}: + "test/core/test_core.jl" + "test/utils.jl" +``` +""" function _collect_test_files_recursive(test_dir::AbstractString) files = String[] for (root, _, fs) in walkdir(test_dir) @@ -185,6 +529,32 @@ function _collect_test_files_recursive(test_dir::AbstractString) return files end +""" + _find_symbol_test_file_rel(name::Symbol, filename_builder::Function; test_dir::AbstractString) -> Union{String,Nothing} + +Find the relative path to a test file for a given symbol name. + +Uses the `filename_builder` to construct the expected filename, then searches +for files matching that basename. If multiple matches exist (e.g., files in +different subdirectories), prefers the shallowest path. + +# Arguments +- `name::Symbol`: Test name to resolve +- `filename_builder::Function`: Function that maps test names to filenames +- `test_dir::AbstractString`: Root directory containing test files + +# Returns +- `String`: Relative path to the matching test file +- `nothing`: If no matching file is found + +# Notes +- Searches recursively in `test_dir` +- Excludes `runtests.jl` from consideration +- Prefers shallower paths when multiple matches exist +- Returns the exact relative path if found + +See also: [`TestRunner._collect_test_files_recursive`](@ref), [`TestRunner._ensure_jl`](@ref) +""" function _find_symbol_test_file_rel( name::Symbol, filename_builder::Function; test_dir::AbstractString ) @@ -204,15 +574,28 @@ function _find_symbol_test_file_rel( end """ +$(TYPEDSIGNATURES) + Determine which tests to run based on selections, available_tests filter, and file globbing. + 1. Identify potential test files in `test_dir` (default: `test/`). 2. Filter by `available_tests` if provided. 3. Filter by `selections` (interpreted as globs) if present. -# Notes +# Arguments +- `selections::Vector{String}`: User-provided selection patterns +- `available_tests::AbstractVector{<:TestSpec}`: Allowed tests (empty = auto-discovery) +- `run_all::Bool`: Whether to run all available tests +- `filename_builder::Function`: Function to map test names to filenames +- `test_dir::String`: Root directory containing test files -If `available_tests` is empty, this function falls back to an auto-discovery -heuristic using the filename stem as the candidate test name. +# Returns +- `Vector{TestSpec}`: Selected test specifications + +# Notes +- If `available_tests` is empty, this function falls back to an auto-discovery + heuristic using the filename stem as the candidate test name +- Selection arguments are matched against multiple representations of each candidate """ function _select_tests( selections::Vector{String}, @@ -300,8 +683,10 @@ function _select_tests( return candidates end - # 3. Filter candidates by selections (Patterns) - # selections are now patterns! e.g. [:utils, :test_u*] + # 3. Normalize selections: expand bare directory paths to dir/* + selections = _normalize_selections(selections, candidates) + + # 4. Filter candidates by selections (Patterns) filtered = TestSpec[] for candidate in candidates @@ -361,17 +746,32 @@ function _select_tests( end """ - _run_single_test(name::Symbol; kwargs...) +$(TYPEDSIGNATURES) Run a single selected test. This helper: - - Resolves a test filename via `filename_builder` - Includes the file into `Main` +- Calls `on_test_start` (if provided) after include, before eval - Optionally evaluates a function (via `funcname_builder`) when `eval_mode=true` +- Calls `on_test_done` (if provided) after eval, skip, or error + +# Arguments +- `spec::TestSpec`: Test specification to run +- `available_tests::AbstractVector{<:TestSpec}`: Available tests for validation +- `filename_builder::Function`: Function to map test names to filenames +- `funcname_builder::Function`: Function to map test names to function names +- `eval_mode::Bool`: Whether to evaluate the function after include +- `test_dir::String`: Root directory containing test files +- `index::Int`: 1-based index in the selected list (default: `1`) +- `total::Int`: Total number of selected tests (default: `1`) +- `on_test_start::Union{Function,Nothing}`: Callback before eval (default: `nothing`) +- `on_test_done::Union{Function,Nothing}`: Callback after eval (default: `nothing`) -This function is not part of the public API. +# Notes +- This function is not part of the public API +- Use `run_tests` for running multiple tests with proper orchestration """ function _run_single_test( spec::TestSpec; @@ -380,6 +780,116 @@ function _run_single_test( funcname_builder::Function, eval_mode::Bool, test_dir::String, + index::Int=1, + total::Int=1, + on_test_start::Union{Function,Nothing}=nothing, + on_test_done::Union{Function,Nothing}=nothing, +) + # --- Resolve filename and func_symbol --- + filename, func_symbol = _resolve_test( + spec; available_tests, filename_builder, funcname_builder, eval_mode, test_dir + ) + + # --- Include the file --- + Base.include(Main, filename) + + # --- Check function exists after include --- + if eval_mode && func_symbol !== nothing && !isdefined(Main, func_symbol) + error(""" + Function "$(func_symbol)" not found after including "$(filename)". + Make sure the file defines a function with this name. + """) + end + + # --- on_test_start callback --- + if on_test_start !== nothing + info = TestRunInfo(spec, filename, func_symbol, index, total, :pre_eval, nothing, nothing) + should_continue = on_test_start(info) + if should_continue === false + if on_test_done !== nothing + done_info = TestRunInfo(spec, filename, func_symbol, index, total, :skipped, nothing, nothing) + on_test_done(done_info) + end + return nothing + end + end + + # --- Skip eval if not in eval mode --- + if !eval_mode || func_symbol === nothing + if on_test_done !== nothing + done_info = TestRunInfo(spec, filename, func_symbol, index, total, :skipped, nothing, nothing) + on_test_done(done_info) + end + return nothing + end + + # --- Snapshot testset results before eval --- + ts = Test.get_testset() + n_before = (ts isa Test.DefaultTestSet) ? length(ts.results) : nothing + + # --- Eval the function --- + t0 = time() + try + Main.eval(Expr(:call, func_symbol)) + elapsed = time() - t0 + if on_test_done !== nothing + # Detect @test failures by scanning only the new results added during eval + status = if n_before !== nothing + _has_failures_in_results(ts, n_before + 1) ? :test_failed : :post_eval + else + :post_eval + end + done_info = TestRunInfo(spec, filename, func_symbol, index, total, status, nothing, elapsed) + on_test_done(done_info) + end + catch ex + elapsed = time() - t0 + if on_test_done !== nothing + done_info = TestRunInfo(spec, filename, func_symbol, index, total, :error, ex, elapsed) + on_test_done(done_info) + end + rethrow() + end + + return nothing +end + +""" +$(TYPEDSIGNATURES) + +Resolve a test spec into an absolute filename and function symbol. + +Handles both `String` specs (relative paths) and `Symbol` specs (logical names). +Raises errors if the file is not found or if `eval_mode=true` but no function can be determined. + +# Arguments +- `spec::TestSpec`: Test specification to resolve +- `available_tests::AbstractVector{<:TestSpec}`: Available tests for validation +- `filename_builder::Function`: Function to map test names to filenames +- `funcname_builder::Function`: Function to map test names to function names +- `eval_mode::Bool`: Whether to resolve a function name +- `test_dir::String`: Root directory containing test files + +# Returns +- `Tuple{String, Union{Symbol,Nothing}}`: `(filename, func_symbol)` where: + - `filename`: Absolute path to the test file + - `func_symbol`: Function symbol to call (or `nothing` if `eval_mode=false`) + +# Throws +- `ErrorException`: If the test file is not found +- `ErrorException`: If `eval_mode=true` but no function can be determined + +# Notes +- This function is not part of the public API +- Use `run_tests` for running tests with proper error handling +""" +function _resolve_test( + spec::TestSpec; + available_tests::AbstractVector{<:TestSpec}, + filename_builder::Function, + funcname_builder::Function, + eval_mode::Bool, + test_dir::String, ) if spec isa String rel = _ensure_jl(spec) @@ -390,22 +900,14 @@ function _run_single_test( Current directory: $(pwd()) """) end - Base.include(Main, filename) - - if !eval_mode - return nothing - end - func_symbol = Symbol(replace(basename(rel), ".jl" => "")) - if !isdefined(Main, func_symbol) - error(""" - Function "$(func_symbol)" not found after including "$(filename)". - Make sure the file defines a function with this name. - """) + func_symbol = if eval_mode + Symbol(replace(basename(rel), ".jl" => "")) + else + nothing end - Main.eval(Expr(:call, func_symbol)) - return nothing + return (filename, func_symbol) end name = spec @@ -425,9 +927,6 @@ function _run_single_test( Current directory: $(pwd()) """) - # Include the file - Base.include(Main, filename) - # Determine function name func_symbol = funcname_builder(name) @@ -441,24 +940,245 @@ function _run_single_test( ) end - # Skip eval if not in eval mode or funcname_builder returned nothing if !eval_mode || func_symbol === nothing - return nothing + return (filename, nothing) end func_symbol = func_symbol isa Symbol ? func_symbol : Symbol(String(func_symbol)) - # Check function exists before eval - if !isdefined(Main, func_symbol) - error(""" - Function "$(func_symbol)" not found after including "$(filename)". - Make sure the file defines a function with this name. - """) + return (filename, func_symbol) +end + +# ============================================================================ +# Progress display +# ============================================================================ + +""" +$(TYPEDSIGNATURES) + +Recursively scan a `DefaultTestSet` results for `Test.Fail` or `Test.Error` entries, +starting at index `from`. + +This is used to detect `@test` failures that occurred during a specific eval by +comparing the results count before and after the eval. The `anynonpass` field is +unreliable because it is only updated when a testset *finishes* (in `Test.finish`). + +# Arguments +- `ts::Test.DefaultTestSet`: TestSet to scan +- `from::Int`: Starting index for scanning (default: `1`) + +# Returns +- `Bool`: `true` if any failures are found, `false` otherwise + +# Example +```julia-repl +julia> using CTBase.TestRunner, Test + +julia> ts = Test.DefaultTestSet("test", []) +julia> Test.@testset "example" begin + Test.@test 1 == 1 + Test.@test 2 == 0 # This will fail + end +Test.DefaultTestSet("example", Any[Test.Pass(1), Test.Fail("false")]) + +julia> TestRunner._has_failures_in_results(ts) +true +``` +""" +function _has_failures_in_results(ts::Test.DefaultTestSet, from::Int=1) + for i in from:length(ts.results) + r = ts.results[i] + if r isa Test.DefaultTestSet + _has_failures_in_results(r) && return true + elseif r isa Test.Fail || r isa Test.Error + return true + end + end + return false +end + +""" +$(TYPEDSIGNATURES) + +Compute the progress bar character width based on the number of tests. + +- `total ≤ 20`: width equals `total` (one block per test). +- `total > 20`: fixed width of 20 (some tests skip a block advance). + +# Arguments +- `total::Int`: Total number of tests + +# Returns +- `Int`: Character width for the progress bar (0 if `total ≤ 0`) + +# Example +```julia-repl +julia> using CTBase.TestRunner + +julia> TestRunner._bar_width(10) +10 + +julia> TestRunner._bar_width(25) +20 + +julia> TestRunner._bar_width(0) +0 +``` +""" +function _bar_width(total::Int) + total <= 0 && return 0 + return min(total, 20) +end + +""" +$(TYPEDSIGNATURES) + +Render a progress bar string like `[████████░░░░░░░░░░░]`. + +When `width` is `nothing` (default), the width is computed automatically +via `_bar_width(total)`. Returns an empty string when the bar is hidden. + +# Arguments +- `index::Int`: current progress (1-based) +- `total::Int`: total number of items +- `width::Union{Int,Nothing}`: character width of the bar (default: auto) + +# Returns +- `String`: Progress bar string, or empty string if hidden + +# Example +```julia-repl +julia> using CTBase.TestRunner + +julia> TestRunner._progress_bar(5, 10) +"[█████░░░░░]" + +julia> TestRunner._progress_bar(5, 10; width=20) +"[████████████████████░░░░░░]" + +julia> TestRunner._progress_bar(0, 10; width=5) +"[░░░░░]" +``` +""" +function _progress_bar(index::Int, total::Int; width::Union{Int,Nothing}=nothing) + w = width === nothing ? _bar_width(total) : width + w <= 0 && return "" + total <= 0 && return "[" * repeat("░", w) * "]" + filled = round(Int, index / total * w) + filled = clamp(filled, 0, w) + return "[" * repeat("█", filled) * repeat("░", w - filled) * "]" +end + +""" +$(TYPEDSIGNATURES) + +Write a styled progress line for a completed test to `io`. + +Uses ANSI colors: green for success, red for errors, yellow for skipped. + +# Arguments +- `io::IO`: Output stream to write to +- `info::TestRunInfo`: Test execution information + +# Notes +- Format: `[progress_bar] symbol [index/total] spec (time) status` +- Colors: green for success, red for errors, yellow for skipped +- Time is displayed with one decimal place when available + +# Example +```julia-repl +julia> using CTBase.TestRunner, IOBuffer + +julia> info = TestRunner.TestRunInfo( + :test_example, + "/path/to/test.jl", + :test_example, + 5, 10, + :post_eval, + nothing, + 1.23 + ); + +julia> buf = IOBuffer(); +julia> TestRunner._format_progress_line(buf, info); +julia> String(take!(buf)) +"[█████░░░░░░░░░░░] ✓ [05/10] test_example (1.2s)" +``` +""" +function _format_progress_line(io::IO, info::TestRunInfo) + # ANSI codes + reset = "\e[0m" + bold = "\e[1m" + dim = "\e[2m" + green = "\e[32m" + red = "\e[31m" + yellow = "\e[33m" + cyan = "\e[36m" + + bar = _progress_bar(info.index, info.total) + + if info.status == :error || info.status == :test_failed + color = red + symbol = "✗" + elseif info.status == :skipped + color = yellow + symbol = "○" + else + color = green + symbol = "✓" + end + + w = ndigits(info.total) + idx_str = "[$(lpad(info.index, w, '0'))/$(info.total)]" + time_str = if info.elapsed !== nothing + " $(dim)($(round(info.elapsed; digits=1))s)$(reset)" + else + "" end + status_str = (info.status == :error || info.status == :test_failed) ? " $(bold)$(red)FAILED$(reset)$(dim)," : "" - # Eval the function - Main.eval(Expr(:call, func_symbol)) + if !isempty(bar) + print(io, "$(color)$(bar)$(reset) ") + end + print(io, "$(bold)$(color)$(symbol)$(reset) ") + print(io, "$(cyan)$(idx_str)$(reset) ") + print(io, "$(bold)$(info.spec)$(reset)") + println(io, "$(status_str)$(time_str)") + return nothing +end +""" +$(TYPEDSIGNATURES) + +Default progress callback for `on_test_done`. Prints to `stdout`. + +# Arguments +- `info::TestRunInfo`: Test execution information to display + +# Notes +- This is the default callback used when `progress=true` and no custom `on_test_done` is provided +- Outputs a formatted progress line to `stdout` with colors and timing information + +# Example +```julia-repl +julia> using CTBase.TestRunner + +julia> info = TestRunner.TestRunInfo( + :test_example, + "/path/to/test.jl", + :test_example, + 5, 10, + :post_eval, + nothing, + 1.23 + ); + +julia> TestRunner._default_on_test_done(info) +[█████░░░░░░░░░░░] ✓ [05/10] test_example (1.2s) +``` +""" +function _default_on_test_done(info::TestRunInfo) + _format_progress_line(stdout, info) return nothing end diff --git a/test/extras/Project.toml b/test/extras/Project.toml new file mode 100644 index 00000000..4d211f1e --- /dev/null +++ b/test/extras/Project.toml @@ -0,0 +1,3 @@ +[deps] +CTBase = "54762871-cc72-4466-b8e8-f6c8b58076cd" +Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40" diff --git a/test/extras/progress/bar.jl b/test/extras/progress/bar.jl new file mode 100644 index 00000000..009ddfc9 --- /dev/null +++ b/test/extras/progress/bar.jl @@ -0,0 +1,131 @@ +try + using Revise +catch + println("🔧 Revise not found - continuing without hot reload") +end + +using Pkg +Pkg.activate(joinpath(@__DIR__, "..")) + +# Add CTBase in development mode +if !haskey(Pkg.project().dependencies, "CTBase") + Pkg.develop(path=joinpath(@__DIR__, "..", "..", "..")) +end + +using CTBase +using Test + +# Include TestRunner directly since extension loading isn't working +include(joinpath(@__DIR__, "..", "..", "..", "ext", "TestRunner.jl")) +using .TestRunner: _bar_width, _progress_bar, _format_progress_line, TestRunInfo + +# ============================================================================ +# Block character samples +# ============================================================================ + +println("=" ^ 60) +println("BLOCK CHARACTER SAMPLES") +println("=" ^ 60) + +blocks = [ + ("█", "Full block (U+2588)"), + ("▓", "Dark shade (U+2593)"), + ("▒", "Medium shade (U+2592)"), + ("░", "Light shade (U+2591)"), + ("▌", "Left half block (U+258C)"), + ("▐", "Right half block (U+2590)"), + ("▍", "Left 3/8 block (U+258D)"), + ("▎", "Left 1/4 block (U+258E)"), + ("▏", "Left 1/8 block (U+258F)"), +] + +for (ch, desc) in blocks + println(" $(desc)") + for w in [5, 10, 15, 20] + bar = "[" * repeat(ch, w) * "]" + println(" w=$(lpad(w, 2)): $bar") + end + println() +end + +# Filled + empty combos +println("--- Filled + empty combos (half-filled, w=20) ---") +combos = [ + ("█", "░"), + ("▓", "░"), + ("▌", "░"), + ("▍", " "), + ("▎", " "), + ("█", "·"), + ("█", "─"), +] + +for (filled, empty) in combos + bar = "[" * repeat(filled, 10) * repeat(empty, 10) * "]" + println(" $filled + $empty : $bar") +end + +println() + +# ============================================================================ +# Simulate progress bar display for different test counts +# ============================================================================ + +println("=" ^ 60) +println("PROGRESS BAR VISUAL SIMULATION") +println("=" ^ 60) + +function simulate_progress(total_tests::Int) + println("\n--- $total_tests tests ---") + + # Show a few representative progress points + points = if total_tests <= 20 + collect(1:total_tests) + elseif total_tests <= 50 + [1, 5, 10, 25, 40, total_tests] + elseif total_tests <= 100 + [1, 10, 25, 50, 75, total_tests] + elseif total_tests <= 200 + [1, 20, 50, 100, 150, total_tests] + else + [1, 50, 100, 200, 400, total_tests] + end + + for i in points + if i > total_tests + continue + end + + # Simulate different statuses + status = if i == 1 + :post_eval # success + elseif i == total_tests + :test_failed # failure + else + :post_eval # success + end + + elapsed = 0.1 + (i * 0.05) # fake elapsed time + spec = "suite/test_file_$(lpad(i, 3)).jl" + + info = TestRunInfo( + Symbol(spec), + "/tmp/$(spec)", + Symbol("test_$(i)"), + i, total_tests, status, nothing, elapsed + ) + + buf = IOBuffer() + _format_progress_line(buf, info) + line = String(take!(buf)) + println(line) + end +end + +# Simulate different test counts including edge cases +for total in [5, 10, 20, 21, 30, 50, 100, 200, 500] + simulate_progress(total) +end + +println("\n" * "=" ^ 60) +println("Done.") \ No newline at end of file diff --git a/test/extras/progress/run_tests.jl b/test/extras/progress/run_tests.jl new file mode 100644 index 00000000..b0aba778 --- /dev/null +++ b/test/extras/progress/run_tests.jl @@ -0,0 +1,207 @@ +try + using Revise +catch + println("🔧 Revise not found - continuing without hot reload") +end + +using Pkg +Pkg.activate(joinpath(@__DIR__, "..")) + +# Add CTBase in development mode +if !haskey(Pkg.project().dependencies, "CTBase") + Pkg.develop(path=joinpath(@__DIR__, "..", "..", "..")) +end + +using CTBase +using Test + +# ============================================================================ +# Exploration: how does Test.get_testset() / anynonpass behave? +# ============================================================================ + +# Helper functions +function fake_test_with_failure() + @testset "Fake test" begin + @test 1 == 1 + @test 1 == 0 # fail + @test 2 == 2 + end +end + +function fake_test_passing() + @testset "Fake passing" begin + @test 1 == 1 + @test 2 == 2 + end +end + +# Recursive failure detection +function _has_failures(ts::Test.DefaultTestSet) + ts.anynonpass && return true + for r in ts.results + if r isa Test.DefaultTestSet + _has_failures(r) && return true + elseif r isa Test.Fail || r isa Test.Error + return true + end + end + return false +end + +# ============================================================================ +# EXPLORATION 1: anynonpass + results inspection after eval +# ============================================================================ +println("=" ^ 60) +println("EXPLORATION 1: anynonpass + results after eval") +println("=" ^ 60) + +try + @testset "expl1" begin + @testset "spec_fail" begin + fake_test_with_failure() + ts = Test.get_testset() + println(" [FAIL] anynonpass: $(ts.anynonpass)") + println(" [FAIL] results count: $(length(ts.results))") + for (i, r) in enumerate(ts.results) + if r isa Test.DefaultTestSet + println(" result[$i]: desc=$(r.description), anynonpass=$(r.anynonpass), n_passed=$(r.n_passed), results=$(length(r.results))") + for (j, rr) in enumerate(r.results) + println(" result[$i][$j]: $(typeof(rr))") + end + else + println(" result[$i]: $(typeof(r))") + end + end + println(" [FAIL] _has_failures: $(_has_failures(ts))") + end + + @testset "spec_pass" begin + fake_test_passing() + ts = Test.get_testset() + println(" [PASS] anynonpass: $(ts.anynonpass)") + println(" [PASS] results count: $(length(ts.results))") + for (i, r) in enumerate(ts.results) + if r isa Test.DefaultTestSet + println(" result[$i]: desc=$(r.description), anynonpass=$(r.anynonpass), n_passed=$(r.n_passed), results=$(length(r.results))") + else + println(" result[$i]: $(typeof(r))") + end + end + println(" [PASS] _has_failures: $(_has_failures(ts))") + end + end +catch e + println(" (testset threw: $(typeof(e)))") +end + +# ============================================================================ +# EXPLORATION 2: count Fail/Error types directly in results +# ============================================================================ +println() +println("=" ^ 60) +println("EXPLORATION 2: count Fail/Error in results tree") +println("=" ^ 60) + +function _count_failures(ts::Test.DefaultTestSet) + count = 0 + for r in ts.results + if r isa Test.DefaultTestSet + count += _count_failures(r) + elseif r isa Test.Fail || r isa Test.Error + count += 1 + end + end + return count +end + +try + @testset "expl2" begin + @testset "fail_case" begin + fake_test_with_failure() + ts = Test.get_testset() + println(" [FAIL] _count_failures: $(_count_failures(ts))") + println(" [FAIL] _has_failures: $(_has_failures(ts))") + end + + @testset "pass_case" begin + fake_test_passing() + ts = Test.get_testset() + println(" [PASS] _count_failures: $(_count_failures(ts))") + println(" [PASS] _has_failures: $(_has_failures(ts))") + end + end +catch e + println(" (testset threw: $(typeof(e)))") +end + +# ============================================================================ +# EXPLORATION 3: before/after results count (most robust approach) +# ============================================================================ +println() +println("=" ^ 60) +println("EXPLORATION 3: before/after results count") +println("=" ^ 60) + +function _has_failures_in_range(ts::Test.DefaultTestSet, from::Int) + for i in from:length(ts.results) + r = ts.results[i] + if r isa Test.DefaultTestSet + _has_failures(r) && return true + elseif r isa Test.Fail || r isa Test.Error + return true + end + end + return false +end + +try + @testset "expl3" begin + @testset "fail_before_after" begin + ts = Test.get_testset() + n_before = length(ts.results) + println(" [FAIL] results before eval: $n_before") + + fake_test_with_failure() + + n_after = length(ts.results) + println(" [FAIL] results after eval: $n_after") + println(" [FAIL] new results: $(n_after - n_before)") + has_fail = _has_failures_in_range(ts, n_before + 1) + println(" [FAIL] _has_failures_in_range: $has_fail") + end + + @testset "pass_before_after" begin + ts = Test.get_testset() + n_before = length(ts.results) + println(" [PASS] results before eval: $n_before") + + fake_test_passing() + + n_after = length(ts.results) + println(" [PASS] results after eval: $n_after") + println(" [PASS] new results: $(n_after - n_before)") + has_fail = _has_failures_in_range(ts, n_before + 1) + println(" [PASS] _has_failures_in_range: $has_fail") + end + end +catch e + println(" (testset threw: $(typeof(e)))") +end + +# ============================================================================ +# EXPLORATION 4: without any enclosing @testset (bare _run_single_test) +# ============================================================================ +println() +println("=" ^ 60) +println("EXPLORATION 4: no enclosing @testset") +println("=" ^ 60) + +try + ts = Test.get_testset() + println(" get_testset() outside @testset: $(typeof(ts))") +catch e + println(" get_testset() outside @testset throws: $(typeof(e))") +end + +println() +println("Done.") diff --git a/test/extras/README.md b/test/extras/save/README.md similarity index 100% rename from test/extras/README.md rename to test/extras/save/README.md diff --git a/test/extras/run_all_examples.jl b/test/extras/save/run_all_examples.jl similarity index 100% rename from test/extras/run_all_examples.jl rename to test/extras/save/run_all_examples.jl diff --git a/test/extras/test_ambiguous_description_examples.jl b/test/extras/save/test_ambiguous_description_examples.jl similarity index 100% rename from test/extras/test_ambiguous_description_examples.jl rename to test/extras/save/test_ambiguous_description_examples.jl diff --git a/test/extras/test_extension_error_examples.jl b/test/extras/save/test_extension_error_examples.jl similarity index 100% rename from test/extras/test_extension_error_examples.jl rename to test/extras/save/test_extension_error_examples.jl diff --git a/test/extras/test_incorrect_argument_examples.jl b/test/extras/save/test_incorrect_argument_examples.jl similarity index 100% rename from test/extras/test_incorrect_argument_examples.jl rename to test/extras/save/test_incorrect_argument_examples.jl diff --git a/test/extras/test_not_implemented_examples.jl b/test/extras/save/test_not_implemented_examples.jl similarity index 100% rename from test/extras/test_not_implemented_examples.jl rename to test/extras/save/test_not_implemented_examples.jl diff --git a/test/extras/test_parsing_error_examples.jl b/test/extras/save/test_parsing_error_examples.jl similarity index 100% rename from test/extras/test_parsing_error_examples.jl rename to test/extras/save/test_parsing_error_examples.jl diff --git a/test/extras/test_precondition_error_examples.jl b/test/extras/save/test_precondition_error_examples.jl similarity index 100% rename from test/extras/test_precondition_error_examples.jl rename to test/extras/save/test_precondition_error_examples.jl diff --git a/test/suite/extensions/test_testrunner.jl b/test/suite/extensions/test_testrunner.jl index 6ca6b2ff..65050c36 100644 --- a/test/suite/extensions/test_testrunner.jl +++ b/test/suite/extensions/test_testrunner.jl @@ -1,3 +1,13 @@ +module TestTestRunner + +using Test +using CTBase + +const TestRunner = Base.get_extension(CTBase, :TestRunner) + +const VERBOSE = isdefined(Main, :TestOptions) ? Main.TestOptions.VERBOSE : true +const SHOWTIMING = isdefined(Main, :TestOptions) ? Main.TestOptions.SHOWTIMING : true + struct DummyTestRunnerTag <: CTBase.Extensions.AbstractTestRunnerTag end function test_testrunner() @@ -29,7 +39,7 @@ function test_testrunner() # Access the private function via the loaded extension module parse_args = TestRunner._parse_test_args - @testset verbose = VERBOSE showtiming = SHOWTIMING "empty args returns empty with default flags" begin + @testset verbose = VERBOSE showtiming = SHOWTIMING "empty args with defaults" begin (sel, all_flag, dry_flag) = parse_args(String[]) @test sel == String[] @test all_flag == false @@ -76,6 +86,37 @@ function test_testrunner() @test run_all == true @test dry_run == true end + + @testset verbose = VERBOSE showtiming = SHOWTIMING "test/ prefix is stripped" begin + (sel, _, _) = parse_args(["test/suite/foo"]) + @test sel == ["suite/foo"] + + (sel, _, _) = parse_args(["test/suite"]) + @test sel == ["suite"] + + # No prefix → unchanged + (sel, _, _) = parse_args(["suite/foo"]) + @test sel == ["suite/foo"] + + # Exact "test" (no slash) → unchanged + (sel, _, _) = parse_args(["test"]) + @test sel == ["test"] + + # Multiple args with mixed prefixes + (sel, _, _) = parse_args(["test/suite/a", "suite/b"]) + @test sel == ["suite/a", "suite/b"] + end + end + + @testset verbose = VERBOSE showtiming = SHOWTIMING "_strip_test_prefix" begin + strip_prefix = TestRunner._strip_test_prefix + + @test strip_prefix("test/suite/foo") == "suite/foo" + @test strip_prefix("test/a") == "a" + @test strip_prefix("suite/foo") == "suite/foo" + @test strip_prefix("test") == "test" + @test strip_prefix("testing/foo") == "testing/foo" + @test strip_prefix("test/") == "" end @testset verbose = VERBOSE showtiming = SHOWTIMING "run_tests (dry run)" begin @@ -212,7 +253,7 @@ function test_testrunner() @test sel == [:core] end - @testset verbose = VERBOSE showtiming = SHOWTIMING "With available_tests (symbol located in subdirectory)" begin + @testset verbose = VERBOSE showtiming = SHOWTIMING "symbol in subdirectory" begin mkpath(joinpath(temp_dir, "suite_src")) touch(joinpath(temp_dir, "suite_src", "test_utils.jl")) @@ -224,7 +265,7 @@ function test_testrunner() @test sel == [:utils] end - @testset verbose = VERBOSE showtiming = SHOWTIMING "available_tests may include directories" begin + @testset verbose = VERBOSE showtiming = SHOWTIMING "available_tests may be directories" begin mkpath(joinpath(temp_dir, "suite")) touch(joinpath(temp_dir, "suite", "test_a.jl")) touch(joinpath(temp_dir, "suite", "test_b.jl")) @@ -239,7 +280,7 @@ function test_testrunner() @test sel == ["suite/test_a.jl"] end - @testset verbose = VERBOSE showtiming = SHOWTIMING "available_tests glob string: selection may match basename without test_ prefix" begin + @testset verbose = VERBOSE showtiming = SHOWTIMING "glob: match basename without test_ prefix" begin mkpath(joinpath(temp_dir, "suite_src")) touch(joinpath(temp_dir, "suite_src", "test_utils.jl")) @@ -248,6 +289,220 @@ function test_testrunner() sel = select_tests(["utils"], available, false, identity; test_dir=temp_dir) @test sel == ["suite_src/test_utils.jl"] end + + @testset verbose = VERBOSE showtiming = SHOWTIMING "bare directory selection" begin + mkpath(joinpath(temp_dir, "suite_norm")) + touch(joinpath(temp_dir, "suite_norm", "test_x.jl")) + touch(joinpath(temp_dir, "suite_norm", "test_y.jl")) + + available = TestRunner.TestSpec["suite_norm/*"] + + # "suite_norm" (no wildcard) should behave like "suite_norm/*" + sel = select_tests( + ["suite_norm"], available, false, identity; test_dir=temp_dir + ) + @test sort(sel) == ["suite_norm/test_x.jl", "suite_norm/test_y.jl"] + + # Trailing slash should also work + sel = select_tests( + ["suite_norm/"], available, false, identity; test_dir=temp_dir + ) + @test sort(sel) == ["suite_norm/test_x.jl", "suite_norm/test_y.jl"] + + # Explicit wildcard still works + sel = select_tests( + ["suite_norm/*"], available, false, identity; test_dir=temp_dir + ) + @test sort(sel) == ["suite_norm/test_x.jl", "suite_norm/test_y.jl"] + end + + @testset verbose = VERBOSE showtiming = SHOWTIMING "bare nested directory selection" begin + mkpath(joinpath(temp_dir, "suite_deep", "sub")) + touch(joinpath(temp_dir, "suite_deep", "sub", "test_z.jl")) + + available = TestRunner.TestSpec["suite_deep/sub/*"] + + # "suite_deep/sub" should match "suite_deep/sub/*" + sel = select_tests( + ["suite_deep/sub"], available, false, identity; test_dir=temp_dir + ) + @test sel == ["suite_deep/sub/test_z.jl"] + end + end + end + + # ============================================================================ + # UNIT TESTS - _normalize_selections + # ============================================================================ + + @testset verbose = VERBOSE showtiming = SHOWTIMING "_normalize_selections" begin + normalize = TestRunner._normalize_selections + + candidates = TestRunner.TestSpec[ + "suite/exceptions/test_display.jl", + "suite/exceptions/test_types.jl", + "suite/core/test_core.jl", + ] + + @testset "bare directory expands to dir/*" begin + result = normalize(["suite/exceptions"], candidates) + @test "suite/exceptions" in result + @test "suite/exceptions/*" in result + end + + @testset "trailing slash is stripped and expanded" begin + result = normalize(["suite/exceptions/"], candidates) + @test "suite/exceptions" in result + @test "suite/exceptions/*" in result + end + + @testset "pattern with wildcard is kept as-is" begin + result = normalize(["suite/exceptions/*"], candidates) + @test result == ["suite/exceptions/*"] + end + + @testset "non-matching directory is not expanded" begin + result = normalize(["nonexistent"], candidates) + @test result == ["nonexistent"] + end + + @testset "parent directory expands" begin + result = normalize(["suite"], candidates) + @test "suite" in result + @test "suite/*" in result + end + + @testset "no duplicates" begin + result = normalize(["suite/exceptions", "suite/exceptions/"], candidates) + @test count(==("suite/exceptions"), result) == 1 + @test count(==("suite/exceptions/*"), result) == 1 + end + end + + # ============================================================================ + # UNIT TESTS - Progress display + # ============================================================================ + + @testset verbose = VERBOSE showtiming = SHOWTIMING "Progress display" begin + progress_bar = TestRunner._progress_bar + bar_width = TestRunner._bar_width + default_on_test_done = TestRunner._default_on_test_done + + @testset "_bar_width adaptive" begin + @test bar_width(1) == 1 + @test bar_width(10) == 10 + @test bar_width(20) == 20 + @test bar_width(21) == 20 + @test bar_width(30) == 20 + @test bar_width(50) == 20 + @test bar_width(100) == 20 + @test bar_width(500) == 20 + @test bar_width(1000) == 20 + @test bar_width(0) == 0 + end + + @testset "_progress_bar rendering" begin + # Full bar (explicit width) + bar = progress_bar(10, 10; width=10) + @test bar == "[██████████]" + + # Empty bar + bar = progress_bar(0, 10; width=10) + @test bar == "[░░░░░░░░░░]" + + # Half bar + bar = progress_bar(5, 10; width=10) + @test bar == "[█████░░░░░]" + + # Auto width: 2 tests → width=2 + bar = progress_bar(1, 2) + @test length(bar) == 4 # 2 chars + 2 brackets + + # Auto width: 50 tests → width=20 + bar = progress_bar(25, 50) + @test length(bar) == 22 # 20 chars + 2 brackets + + # Auto width: 500 tests → width=20 + bar = progress_bar(250, 500) + @test length(bar) == 22 # 20 chars + 2 brackets + + # Edge case: total=0 + bar = progress_bar(0, 0; width=10) + @test bar == "[░░░░░░░░░░]" + end + + @testset "_format_progress_line output" begin + fmt = TestRunner._format_progress_line + + info = TestRunner.TestRunInfo( + :my_test, "/tmp/test_my_test.jl", :test_my_test, + 3, 10, :post_eval, nothing, 1.234, + ) + buf = IOBuffer() + fmt(buf, info) + output = String(take!(buf)) + @test contains(output, "✓") + @test contains(output, "[03/10]") + @test contains(output, "my_test") + @test contains(output, "1.2s") + @test contains(output, "█") + + # Error status + info_err = TestRunner.TestRunInfo( + "suite/test_fail.jl", "/tmp/test_fail.jl", :test_fail, + 5, 10, :error, ErrorException("boom"), 0.5, + ) + buf_err = IOBuffer() + fmt(buf_err, info_err) + output_err = String(take!(buf_err)) + @test contains(output_err, "✗") + @test contains(output_err, "FAILED") + @test contains(output_err, "[05/10]") + + # test_failed status (from @test failures) + info_tf = TestRunner.TestRunInfo( + :my_test, "/tmp/test_my_test.jl", :test_my_test, + 7, 10, :test_failed, nothing, 2.0, + ) + buf_tf = IOBuffer() + fmt(buf_tf, info_tf) + output_tf = String(take!(buf_tf)) + @test contains(output_tf, "✗") + @test contains(output_tf, "FAILED") + + # Skipped status + info_skip = TestRunner.TestRunInfo( + :skipped_test, "/tmp/test_skip.jl", :test_skip, + 1, 5, :skipped, nothing, nothing, + ) + buf_skip = IOBuffer() + fmt(buf_skip, info_skip) + output_skip = String(take!(buf_skip)) + @test contains(output_skip, "○") + @test !contains(output_skip, "FAILED") + + # Fixed bar of 20 chars even for >400 tests (empty at start) + info_large = TestRunner.TestRunInfo( + :big_test, "/tmp/test_big.jl", :test_big, + 1, 500, :post_eval, nothing, 0.1, + ) + buf_large = IOBuffer() + fmt(buf_large, info_large) + output_large = String(take!(buf_large)) + @test contains(output_large, "✓") + @test contains(output_large, "░") + + # Test with some progress (25%) + info_large_progress = TestRunner.TestRunInfo( + :big_test, "/tmp/test_big.jl", :test_big, + 125, 500, :post_eval, nothing, 0.1, + ) + buf_large_progress = IOBuffer() + fmt(buf_large_progress, info_large_progress) + output_large_progress = String(take!(buf_large_progress)) + @test contains(output_large_progress, "✓") + @test contains(output_large_progress, "█") + @test contains(output_large_progress, "░") end end @@ -261,7 +516,7 @@ function test_testrunner() normalize_available = TestRunner._normalize_available_tests find_symbol_file = TestRunner._find_symbol_test_file_rel - @testset verbose = VERBOSE showtiming = SHOWTIMING "mixed case and special characters in test names" begin + @testset verbose = VERBOSE showtiming = SHOWTIMING "mixed case & special chars" begin # Test names are converted to Symbols, preserving case (sel, _, _) = parse_args(["MyTest", "test_123"]) @test sel == ["MyTest", "test_123"] @@ -282,7 +537,7 @@ function test_testrunner() @test_throws ArgumentError normalize_available(Any[1]) end - @testset verbose = VERBOSE showtiming = SHOWTIMING "symbol resolution prefers shallowest match when root missing" begin + @testset verbose = VERBOSE showtiming = SHOWTIMING "symbol resolution: shallowest match" begin mktempdir() do temp_dir mkpath(joinpath(temp_dir, "a")) mkpath(joinpath(temp_dir, "a", "b")) @@ -319,7 +574,7 @@ function test_testrunner() end end - @testset verbose = VERBOSE showtiming = SHOWTIMING "duplicate selections are preserved" begin + @testset verbose = VERBOSE showtiming = SHOWTIMING "duplicate selections preserved" begin # Duplicates are not filtered (caller's responsibility) (sel, _, _) = parse_args(["utils", "utils"]) @test sel == ["utils", "utils"] @@ -352,7 +607,7 @@ function test_testrunner() @test occursin("not found", err.msg) end - @testset verbose = VERBOSE showtiming = SHOWTIMING "string spec: error when file not found" begin + @testset verbose = VERBOSE showtiming = SHOWTIMING "string spec: file not found" begin err = try run_single( "does_not_exist_abc123.jl"; @@ -370,7 +625,7 @@ function test_testrunner() @test occursin("not found", err.msg) end - @testset verbose = VERBOSE showtiming = SHOWTIMING "string spec: eval_mode=false does not call function" begin + @testset verbose = VERBOSE showtiming = SHOWTIMING "string spec: eval_mode=false" begin mktempdir() do temp_dir stem = "testrunner_evalmode_false" file = joinpath(temp_dir, stem * ".jl") @@ -397,7 +652,7 @@ function test_testrunner() end end - @testset verbose = VERBOSE showtiming = SHOWTIMING "symbol spec: eval_mode=false does not call function" begin + @testset verbose = VERBOSE showtiming = SHOWTIMING "symbol spec: eval_mode=false" begin mktempdir() do temp_dir touch(joinpath(temp_dir, "test_sym_noeval.jl")) write( @@ -423,7 +678,7 @@ function test_testrunner() end end - @testset verbose = VERBOSE showtiming = SHOWTIMING "string spec: error when expected function missing" begin + @testset verbose = VERBOSE showtiming = SHOWTIMING "string spec: missing function" begin mktempdir() do temp_dir stem = "testrunner_missing_func" file = joinpath(temp_dir, stem * ".jl") @@ -449,7 +704,7 @@ function test_testrunner() end end - @testset verbose = VERBOSE showtiming = SHOWTIMING "symbol spec: funcname_builder returns nothing in eval_mode" begin + @testset verbose = VERBOSE showtiming = SHOWTIMING "symbol spec: funcname_builder returns nothing" begin mktempdir() do temp_dir touch(joinpath(temp_dir, "test_foo.jl")) @@ -472,7 +727,7 @@ function test_testrunner() end end - @testset verbose = VERBOSE showtiming = SHOWTIMING "symbol spec: error when built function missing" begin + @testset verbose = VERBOSE showtiming = SHOWTIMING "symbol spec: built function missing" begin mktempdir() do temp_dir write(joinpath(temp_dir, "test_bar.jl"), "x = 1\n") @@ -496,7 +751,7 @@ function test_testrunner() end end - @testset verbose = VERBOSE showtiming = SHOWTIMING "funcname_builder may return String or Symbol" begin + @testset verbose = VERBOSE showtiming = SHOWTIMING "funcname_builder: String or Symbol" begin mktempdir() do temp_dir # Create a test file with a function write( @@ -535,5 +790,352 @@ function test_testrunner() # They are tested indirectly through normal test execution. end + # ============================================================================ + # UNIT TESTS - TestRunInfo struct + # ============================================================================ + + @testset verbose = VERBOSE showtiming = SHOWTIMING "TestRunInfo" begin + @testset verbose = VERBOSE showtiming = SHOWTIMING "construction with all fields" begin + info = TestRunner.TestRunInfo(:my_test, "/tmp/test_my_test.jl", :test_my_test, 1, 5, :pre_eval, nothing, nothing) + @test info.spec === :my_test + @test info.filename == "/tmp/test_my_test.jl" + @test info.func_symbol === :test_my_test + @test info.index == 1 + @test info.total == 5 + @test info.status === :pre_eval + @test info.error === nothing + @test info.elapsed === nothing + end + + @testset verbose = VERBOSE showtiming = SHOWTIMING "construction with String spec" begin + info = TestRunner.TestRunInfo("suite/test_a.jl", "/tmp/suite/test_a.jl", :test_a, 3, 10, :post_eval, nothing, 1.5) + @test info.spec == "suite/test_a.jl" + @test info.elapsed == 1.5 + end + + @testset verbose = VERBOSE showtiming = SHOWTIMING "construction with error" begin + ex = ErrorException("boom") + info = TestRunner.TestRunInfo(:failing, "/tmp/test_failing.jl", :test_failing, 2, 3, :error, ex, 0.1) + @test info.status === :error + @test info.error === ex + @test info.elapsed == 0.1 + end + end + + # ============================================================================ + # UNIT TESTS - Callbacks via _run_single_test + # ============================================================================ + + @testset verbose = VERBOSE showtiming = SHOWTIMING "Callbacks" begin + run_single = TestRunner._run_single_test + + @testset verbose = VERBOSE showtiming = SHOWTIMING "on_test_start receives :pre_eval" begin + mktempdir() do temp_dir + write( + joinpath(temp_dir, "test_cb_start.jl"), + "function test_cb_start()\n @test true\nend\n", + ) + + captured = Ref{Any}(nothing) + run_single( + :cb_start; + available_tests=Symbol[], + filename_builder=n -> "test_" * String(n), + funcname_builder=n -> Symbol(:test_, n), + eval_mode=true, + test_dir=temp_dir, + index=2, + total=7, + on_test_start=info -> (captured[] = info; true), + on_test_done=nothing, + ) + + info = captured[] + @test info isa TestRunner.TestRunInfo + @test info.spec === :cb_start + @test info.func_symbol === :test_cb_start + @test info.index == 2 + @test info.total == 7 + @test info.status === :pre_eval + @test info.error === nothing + @test info.elapsed === nothing + end + end + + @testset verbose = VERBOSE showtiming = SHOWTIMING "on_test_done receives :post_eval" begin + mktempdir() do temp_dir + write( + joinpath(temp_dir, "test_cb_done.jl"), + "function test_cb_done()\n @test true\nend\n", + ) + + captured = Ref{Any}(nothing) + run_single( + :cb_done; + available_tests=Symbol[], + filename_builder=n -> "test_" * String(n), + funcname_builder=n -> Symbol(:test_, n), + eval_mode=true, + test_dir=temp_dir, + index=1, + total=3, + on_test_start=nothing, + on_test_done=info -> (captured[] = info), + ) + + info = captured[] + @test info isa TestRunner.TestRunInfo + @test info.spec === :cb_done + @test info.status === :post_eval + @test info.error === nothing + @test info.elapsed isa Float64 + @test info.elapsed >= 0.0 + end + end + + @testset verbose = VERBOSE showtiming = SHOWTIMING "on_test_start false skips eval" begin + mktempdir() do temp_dir + write( + joinpath(temp_dir, "test_cb_skip.jl"), + "const __ctbase_cb_skip_flag__ = Ref(false)\n" * + "function test_cb_skip()\n" * + " __ctbase_cb_skip_flag__[] = true\n" * + " return nothing\n" * + "end\n", + ) + + done_captured = Ref{Any}(nothing) + run_single( + :cb_skip; + available_tests=Symbol[], + filename_builder=n -> "test_" * String(n), + funcname_builder=n -> Symbol(:test_, n), + eval_mode=true, + test_dir=temp_dir, + index=1, + total=1, + on_test_start=_ -> false, + on_test_done=info -> (done_captured[] = info), + ) + + # Function was NOT called + flag_value = Base.invokelatest(getfield, Main, :__ctbase_cb_skip_flag__) + @test flag_value[] == false + + # on_test_done was called with :skipped + info = done_captured[] + @test info isa TestRunner.TestRunInfo + @test info.status === :skipped + @test info.elapsed === nothing + end + end + + @testset verbose = VERBOSE showtiming = SHOWTIMING "on_test_done receives :skipped" begin + mktempdir() do temp_dir + write( + joinpath(temp_dir, "test_cb_noeval.jl"), + "x = 1\n", + ) + + done_captured = Ref{Any}(nothing) + run_single( + :cb_noeval; + available_tests=Symbol[], + filename_builder=n -> "test_" * String(n), + funcname_builder=n -> Symbol(:test_, n), + eval_mode=false, + test_dir=temp_dir, + index=1, + total=1, + on_test_start=nothing, + on_test_done=info -> (done_captured[] = info), + ) + + info = done_captured[] + @test info isa TestRunner.TestRunInfo + @test info.status === :skipped + @test info.func_symbol === nothing + end + end + + @testset verbose = VERBOSE showtiming = SHOWTIMING "on_test_done receives :error on eval failure" begin + mktempdir() do temp_dir + write( + joinpath(temp_dir, "test_cb_err.jl"), + "function test_cb_err()\n error(\"intentional failure\")\nend\n", + ) + + done_captured = Ref{Any}(nothing) + err = try + run_single( + :cb_err; + available_tests=Symbol[], + filename_builder=n -> "test_" * String(n), + funcname_builder=n -> Symbol(:test_, n), + eval_mode=true, + test_dir=temp_dir, + index=1, + total=1, + on_test_start=_ -> true, + on_test_done=info -> (done_captured[] = info), + ) + nothing + catch e + e + end + + # Error is rethrown + @test err isa ErrorException + @test occursin("intentional failure", err.msg) + + # on_test_done was called with :error + info = done_captured[] + @test info isa TestRunner.TestRunInfo + @test info.status === :error + @test info.error isa ErrorException + @test info.elapsed isa Float64 + end + end + + @testset verbose = VERBOSE showtiming = SHOWTIMING "both callbacks work together" begin + mktempdir() do temp_dir + write( + joinpath(temp_dir, "test_cb_both.jl"), + "function test_cb_both()\n @test true\nend\n", + ) + + start_captured = Ref{Any}(nothing) + done_captured = Ref{Any}(nothing) + run_single( + :cb_both; + available_tests=Symbol[], + filename_builder=n -> "test_" * String(n), + funcname_builder=n -> Symbol(:test_, n), + eval_mode=true, + test_dir=temp_dir, + index=3, + total=5, + on_test_start=info -> (start_captured[] = info; true), + on_test_done=info -> (done_captured[] = info), + ) + + @test start_captured[].status === :pre_eval + @test start_captured[].index == 3 + @test start_captured[].total == 5 + @test done_captured[].status === :post_eval + @test done_captured[].index == 3 + @test done_captured[].total == 5 + end + end + + @testset verbose = VERBOSE showtiming = SHOWTIMING "callbacks with String spec" begin + mktempdir() do temp_dir + write( + joinpath(temp_dir, "test_cb_str.jl"), + "function test_cb_str()\n @test true\nend\n", + ) + + start_captured = Ref{Any}(nothing) + done_captured = Ref{Any}(nothing) + run_single( + "test_cb_str.jl"; + available_tests=Symbol[], + filename_builder=identity, + funcname_builder=identity, + eval_mode=true, + test_dir=temp_dir, + index=1, + total=2, + on_test_start=info -> (start_captured[] = info; true), + on_test_done=info -> (done_captured[] = info), + ) + + @test start_captured[].spec == "test_cb_str.jl" + @test start_captured[].status === :pre_eval + @test done_captured[].spec == "test_cb_str.jl" + @test done_captured[].status === :post_eval + end + end + + @testset verbose = VERBOSE showtiming = SHOWTIMING "no callbacks (nothing) preserves original behavior" begin + mktempdir() do temp_dir + write( + joinpath(temp_dir, "test_cb_none.jl"), + "function test_cb_none()\n @test true\nend\n", + ) + + # Should not error when callbacks are nothing + run_single( + :cb_none; + available_tests=Symbol[], + filename_builder=n -> "test_" * String(n), + funcname_builder=n -> Symbol(:test_, n), + eval_mode=true, + test_dir=temp_dir, + on_test_start=nothing, + on_test_done=nothing, + ) + @test true + end + end + end + + # ============================================================================ + # INTEGRATION TESTS - Callbacks via run_tests + # ============================================================================ + + @testset verbose = VERBOSE showtiming = SHOWTIMING "Callbacks via run_tests" begin + @testset verbose = VERBOSE showtiming = SHOWTIMING "on_test_start and on_test_done called for each test" begin + mktempdir() do temp_dir + write( + joinpath(temp_dir, "test_int_a.jl"), + "function test_int_a()\n @test true\nend\n", + ) + write( + joinpath(temp_dir, "test_int_b.jl"), + "function test_int_b()\n @test true\nend\n", + ) + + start_log = TestRunner.TestRunInfo[] + done_log = TestRunner.TestRunInfo[] + + CTBase.run_tests(; + args=String[], + testset_name="CallbackIntegration", + available_tests=(:int_a, :int_b), + filename_builder=n -> Symbol(:test_, n), + funcname_builder=n -> Symbol(:test_, n), + eval_mode=true, + verbose=false, + showtiming=false, + test_dir=temp_dir, + on_test_start=info -> (push!(start_log, info); true), + on_test_done=info -> push!(done_log, info), + ) + + @test length(start_log) == 2 + @test length(done_log) == 2 + + # Check index/total progression + @test start_log[1].index == 1 + @test start_log[1].total == 2 + @test start_log[2].index == 2 + @test start_log[2].total == 2 + + # Check statuses + @test all(info -> info.status === :pre_eval, start_log) + @test all(info -> info.status === :post_eval, done_log) + + # Check elapsed is populated + @test all(info -> info.elapsed isa Float64, done_log) + end + end + end + return nothing end + +end # module + +# Re-export for TestRunner discovery +test_testrunner() = TestTestRunner.test_testrunner() diff --git a/test_dr.log b/test_dr.log deleted file mode 100644 index 98133ab3..00000000 --- a/test_dr.log +++ /dev/null @@ -1,193 +0,0 @@ - Testing CTBase - Status `/private/var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_UkyeZO/Project.toml` - [4c88cf16] Aqua v0.8.14 - [54762871] CTBase v0.17.4 `~/Research/logiciels/dev/control-toolbox/CTBase` - [a2441757] Coverage v1.8.1 - [ffbed154] DocStringExtensions v0.9.5 - [e30172f5] Documenter v1.16.1 - [d0879d2d] MarkdownAST v0.1.2 - [bac558e1] OrderedCollections v1.8.1 - [d6f4376e] Markdown v1.11.0 - [8dfed614] Test v1.11.0 - Status `/private/var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_UkyeZO/Manifest.toml` - [a4c015fc] ANSIColoredPrinters v0.0.1 - [1520ce14] AbstractTrees v0.4.5 - [4c88cf16] Aqua v0.8.14 - [c7e460c6] ArgParse v1.2.0 - [d1d4a3ce] BitFlags v0.1.9 - [54762871] CTBase v0.17.4 `~/Research/logiciels/dev/control-toolbox/CTBase` - [944b1d66] CodecZlib v0.7.8 - [34da2185] Compat v4.18.1 - [f0e56b4a] ConcurrentUtilities v2.5.0 - [a2441757] Coverage v1.8.1 - [c36e975a] CoverageTools v1.4.1 - [ffbed154] DocStringExtensions v0.9.5 - [e30172f5] Documenter v1.16.1 - [460bff9d] ExceptionUnwrapping v0.1.11 - [d7ba0133] Git v1.5.0 - [cd3eb016] HTTP v1.10.19 - [b5f81e59] IOCapture v1.0.0 - [692b3bcd] JLLWrappers v1.7.1 - [682c06a0] JSON v1.4.0 - [70703baa] JuliaSyntax v1.0.2 - [0e77f7df] LazilyInitializedFields v1.3.0 - [e6f89c97] LoggingExtras v1.2.0 - [d0879d2d] MarkdownAST v0.1.2 - [739be429] MbedTLS v1.1.9 - [4d8831e6] OpenSSL v1.6.1 - [bac558e1] OrderedCollections v1.8.1 - [69de0a69] Parsers v2.8.3 - [aea7be01] PrecompileTools v1.3.3 - [21216c6a] Preferences v1.5.1 - [2792f1a3] RegistryInstances v0.1.0 - [6c6a2e73] Scratch v1.3.0 - [777ac1f9] SimpleBufferStream v1.2.0 - [ec057cc2] StructUtils v2.6.2 - [b718987f] TextWrap v1.0.2 - [3bb67fe8] TranscodingStreams v0.11.3 - [5c2747f8] URIs v1.6.1 - [2e619515] Expat_jll v2.7.3+0 - [020c3dae] Git_LFS_jll v3.7.0+0 - [f8c6e375] Git_jll v2.52.0+0 - [94ce4f54] Libiconv_jll v1.18.0+0 - [c8ffd9c3] MbedTLS_jll v2.28.1010+0 - [9bd350c2] OpenSSH_jll v10.2.1+0 - [0dad84c5] ArgTools v1.1.2 - [56f22d72] Artifacts v1.11.0 - [2a0f44e3] Base64 v1.11.0 - [ade2ca70] Dates v1.11.0 - [f43a241f] Downloads v1.6.0 - [7b1f6079] FileWatching v1.11.0 - [b77e0a4c] InteractiveUtils v1.11.0 - [ac6e5ff7] JuliaSyntaxHighlighting v1.12.0 - [b27032c2] LibCURL v0.6.4 - [76f85450] LibGit2 v1.11.0 - [8f399da3] Libdl v1.11.0 - [56ddb016] Logging v1.11.0 - [d6f4376e] Markdown v1.11.0 - [ca575930] NetworkOptions v1.3.0 - [44cfe95a] Pkg v1.12.0 - [de0858da] Printf v1.11.0 - [3fa0cd96] REPL v1.11.0 - [9a3f8284] Random v1.11.0 - [ea8e919c] SHA v0.7.0 - [9e88b42a] Serialization v1.11.0 - [6462fe0b] Sockets v1.11.0 - [f489334b] StyledStrings v1.11.0 - [fa267f1f] TOML v1.0.3 - [a4e569a6] Tar v1.10.0 - [8dfed614] Test v1.11.0 - [cf7118a7] UUIDs v1.11.0 - [4ec0a83e] Unicode v1.11.0 - [deac9b47] LibCURL_jll v8.11.1+1 - [e37daf67] LibGit2_jll v1.9.0+0 - [29816b5a] LibSSH2_jll v1.11.3+1 - [14a3606d] MozillaCACerts_jll v2025.5.20 - [458c3c95] OpenSSL_jll v3.5.1+0 - [efcefdf7] PCRE2_jll v10.44.0+1 - [83775a58] Zlib_jll v1.3.1+2 - [8e850ede] nghttp2_jll v1.64.0+1 - [3f19e933] p7zip_jll v17.5.0+2 - Testing Running tests... -┌ Warning: No documentation found for no_doc in Main.DocumenterReferenceTestMod. Skipping from API reference. -└ @ DocumenterReference ~/Research/logiciels/dev/control-toolbox/CTBase/ext/DocumenterReference.jl:682 -┌ Warning: Unable to determine HTML(edit_link = ...) from remote HEAD branch, defaulting to "master". -│ Calling `git remote` failed with an exception. Set JULIA_DEBUG=Documenter to see the error. -│ Unless this is due to a configuration error, the relevant variable should be set explicitly. -└ @ Documenter ~/.julia/packages/Documenter/xvqbW/src/utilities/utilities.jl:680 -[ Info: APIBuilder: creating API reference -┌ Warning: No documentation found for AbstractFoo in Main.DocumenterReferenceTestMod. Skipping from API reference. -└ @ DocumenterReference ~/Research/logiciels/dev/control-toolbox/CTBase/ext/DocumenterReference.jl:682 -┌ Warning: No documentation found for Foo in Main.DocumenterReferenceTestMod. Skipping from API reference. -└ @ DocumenterReference ~/Research/logiciels/dev/control-toolbox/CTBase/ext/DocumenterReference.jl:682 -┌ Warning: No documentation found for MYCONST in Main.DocumenterReferenceTestMod. Skipping from API reference. -└ @ DocumenterReference ~/Research/logiciels/dev/control-toolbox/CTBase/ext/DocumenterReference.jl:682 -┌ Warning: No documentation found for no_doc in Main.DocumenterReferenceTestMod. Skipping from API reference. -└ @ DocumenterReference ~/Research/logiciels/dev/control-toolbox/CTBase/ext/DocumenterReference.jl:682 -ERROR: LoadError: Some tests did not pass: 168 passed, 2 failed, 0 errored, 0 broken. -in expression starting at /Users/ocots/Research/logiciels/dev/control-toolbox/CTBase/test/runtests.jl:47 -automatic_reference_documentation configuration: Test Failed at /Users/ocots/Research/logiciels/dev/control-toolbox/CTBase/test/suite/extensions/test_documenter_reference.jl:363 - Expression: pages2 == ("All API" => ["Public" => "ref/public.md", "Private" => "ref/private.md"]) - Evaluated: "All API" => ["Public" => "ref/api_public.md", "Private" => "ref/api_private.md"] == "All API" => ["Public" => "ref/public.md", "Private" => "ref/private.md"] - -Stacktrace: - [1] macro expansion - @ ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Test/src/Test.jl:680 [inlined] - [2] macro expansion - @ ~/Research/logiciels/dev/control-toolbox/CTBase/test/suite/extensions/test_documenter_reference.jl:363 [inlined] - [3] macro expansion - @ ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Test/src/Test.jl:1776 [inlined] - [4] test_documenter_reference() - @ Main ~/Research/logiciels/dev/control-toolbox/CTBase/test/suite/extensions/test_documenter_reference.jl:304 -Documenter.Selectors.order for APIBuilder: Test Failed at /Users/ocots/Research/logiciels/dev/control-toolbox/CTBase/test/suite/extensions/test_documenter_reference.jl:377 - Expression: Documenter.Selectors.order(DR.APIBuilder) == 0.0 - Evaluated: 0.5 == 0.0 - -Stacktrace: - [1] macro expansion - @ ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Test/src/Test.jl:680 [inlined] - [2] macro expansion - @ ~/Research/logiciels/dev/control-toolbox/CTBase/test/suite/extensions/test_documenter_reference.jl:377 [inlined] - [3] macro expansion - @ ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Test/src/Test.jl:1776 [inlined] - [4] test_documenter_reference() - @ Main ~/Research/logiciels/dev/control-toolbox/CTBase/test/suite/extensions/test_documenter_reference.jl:377 -Test Summary: | Pass Fail Total Time -CTBase tests | 168 2 170 7.2s - suite/extensions/test_documenter_reference.jl | 168 2 170 7.2s - Invalid primary_modules input | 1 1 0.0s - reset_config! clears CONFIG | 1 1 0.0s - _default_basename and _build_page_path | 7 7 0.0s - _classify_symbol and _to_string | 12 12 0.0s - _get_source_file | 4 4 0.0s - _get_source_from_methods | 1 1 0.0s - _parse_primary_modules with Pair | 1 1 0.0s - _has_documentation: module documented elsewhere | 1 1 0.0s - Type formatting helpers | 8 8 0.0s - _method_signature_string handles UnionAll | 2 2 0.0s - _iterate_over_symbols filtering | 3 3 0.3s - _iterate_over_symbols with source_files and include_without_source | 3 3 0.1s - automatic_reference_documentation configuration | 20 1 21 1.1s - automatic_reference_documentation default tag | 7 7 0.0s - Documenter.Selectors.order for APIBuilder | 1 1 0.0s - _exported_symbols classification | 16 16 0.0s - automatic_reference_documentation multi-module | 4 4 0.0s - automatic_reference_documentation multi-module with filename | 6 6 0.0s - _get_source_file expanded cases | 6 6 0.0s - Edge cases | 23 23 0.0s - _format_type_for_docs and helpers | 11 11 0.0s - _method_signature_string and method collection | 10 10 0.1s - Page content builders | 9 9 0.0s - external_modules_to_document | 2 2 0.1s - APIBuilder runner integration | 3 3 0.1s - _format_type_for_docs edge cases | 4 4 0.0s - _format_datatype_for_docs with TypeVar | 4 4 0.0s - _format_type_param edge cases | 3 3 0.0s - _method_signature_string edge cases | 3 3 0.0s -RNG of the outermost testset: Random.Xoshiro(0x3eccb6b182fcf254, 0x961144a60eae4d90, 0x814c8bfb77f61087, 0x10762156009528c1, 0xfaba29e70aea4aee) -ERROR: Package CTBase errored during testing -Stacktrace: - [1] pkgerror(msg::String) - @ Pkg.Types ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Pkg/src/Types.jl:68 - [2] test(ctx::Pkg.Types.Context, pkgs::Vector{PackageSpec}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool) - @ Pkg.Operations ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Pkg/src/Operations.jl:2427 - [3] test - @ ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Pkg/src/Operations.jl:2280 [inlined] - [4] test(ctx::Pkg.Types.Context, pkgs::Vector{PackageSpec}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Vector{String}, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool, kwargs::@Kwargs{io::IOContext{IO}}) - @ Pkg.API ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Pkg/src/API.jl:484 - [5] test(pkgs::Vector{PackageSpec}; io::IOContext{IO}, kwargs::@Kwargs{test_args::Vector{String}}) - @ Pkg.API ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Pkg/src/API.jl:164 - [6] test(pkgs::Vector{String}; kwargs::@Kwargs{test_args::Vector{String}}) - @ Pkg.API ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Pkg/src/API.jl:152 - [7] test - @ ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Pkg/src/API.jl:152 [inlined] - [8] #test#81 - @ ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Pkg/src/API.jl:151 [inlined] - [9] top-level scope - @ none:1 - [10] eval(m::Module, e::Any) - @ Core ./boot.jl:489 - [11] exec_options(opts::Base.JLOptions) - @ Base ./client.jl:283 - [12] _start() - @ Base ./client.jl:550 diff --git a/test_dr_v2.log b/test_dr_v2.log deleted file mode 100644 index edd98ce5..00000000 --- a/test_dr_v2.log +++ /dev/null @@ -1,110 +0,0 @@ - Testing CTBase - Status `/private/var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_l4Dyyv/Project.toml` - [4c88cf16] Aqua v0.8.14 - [54762871] CTBase v0.17.4 `~/Research/logiciels/dev/control-toolbox/CTBase` - [a2441757] Coverage v1.8.1 - [ffbed154] DocStringExtensions v0.9.5 - [e30172f5] Documenter v1.16.1 - [d0879d2d] MarkdownAST v0.1.2 - [bac558e1] OrderedCollections v1.8.1 - [d6f4376e] Markdown v1.11.0 - [8dfed614] Test v1.11.0 - Status `/private/var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_l4Dyyv/Manifest.toml` - [a4c015fc] ANSIColoredPrinters v0.0.1 - [1520ce14] AbstractTrees v0.4.5 - [4c88cf16] Aqua v0.8.14 - [c7e460c6] ArgParse v1.2.0 - [d1d4a3ce] BitFlags v0.1.9 - [54762871] CTBase v0.17.4 `~/Research/logiciels/dev/control-toolbox/CTBase` - [944b1d66] CodecZlib v0.7.8 - [34da2185] Compat v4.18.1 - [f0e56b4a] ConcurrentUtilities v2.5.0 - [a2441757] Coverage v1.8.1 - [c36e975a] CoverageTools v1.4.1 - [ffbed154] DocStringExtensions v0.9.5 - [e30172f5] Documenter v1.16.1 - [460bff9d] ExceptionUnwrapping v0.1.11 - [d7ba0133] Git v1.5.0 - [cd3eb016] HTTP v1.10.19 - [b5f81e59] IOCapture v1.0.0 - [692b3bcd] JLLWrappers v1.7.1 - [682c06a0] JSON v1.4.0 - [70703baa] JuliaSyntax v1.0.2 - [0e77f7df] LazilyInitializedFields v1.3.0 - [e6f89c97] LoggingExtras v1.2.0 - [d0879d2d] MarkdownAST v0.1.2 - [739be429] MbedTLS v1.1.9 - [4d8831e6] OpenSSL v1.6.1 - [bac558e1] OrderedCollections v1.8.1 - [69de0a69] Parsers v2.8.3 - [aea7be01] PrecompileTools v1.3.3 - [21216c6a] Preferences v1.5.1 - [2792f1a3] RegistryInstances v0.1.0 - [6c6a2e73] Scratch v1.3.0 - [777ac1f9] SimpleBufferStream v1.2.0 - [ec057cc2] StructUtils v2.6.2 - [b718987f] TextWrap v1.0.2 - [3bb67fe8] TranscodingStreams v0.11.3 - [5c2747f8] URIs v1.6.1 - [2e619515] Expat_jll v2.7.3+0 - [020c3dae] Git_LFS_jll v3.7.0+0 - [f8c6e375] Git_jll v2.52.0+0 - [94ce4f54] Libiconv_jll v1.18.0+0 - [c8ffd9c3] MbedTLS_jll v2.28.1010+0 - [9bd350c2] OpenSSH_jll v10.2.1+0 - [0dad84c5] ArgTools v1.1.2 - [56f22d72] Artifacts v1.11.0 - [2a0f44e3] Base64 v1.11.0 - [ade2ca70] Dates v1.11.0 - [f43a241f] Downloads v1.6.0 - [7b1f6079] FileWatching v1.11.0 - [b77e0a4c] InteractiveUtils v1.11.0 - [ac6e5ff7] JuliaSyntaxHighlighting v1.12.0 - [b27032c2] LibCURL v0.6.4 - [76f85450] LibGit2 v1.11.0 - [8f399da3] Libdl v1.11.0 - [56ddb016] Logging v1.11.0 - [d6f4376e] Markdown v1.11.0 - [ca575930] NetworkOptions v1.3.0 - [44cfe95a] Pkg v1.12.0 - [de0858da] Printf v1.11.0 - [3fa0cd96] REPL v1.11.0 - [9a3f8284] Random v1.11.0 - [ea8e919c] SHA v0.7.0 - [9e88b42a] Serialization v1.11.0 - [6462fe0b] Sockets v1.11.0 - [f489334b] StyledStrings v1.11.0 - [fa267f1f] TOML v1.0.3 - [a4e569a6] Tar v1.10.0 - [8dfed614] Test v1.11.0 - [cf7118a7] UUIDs v1.11.0 - [4ec0a83e] Unicode v1.11.0 - [deac9b47] LibCURL_jll v8.11.1+1 - [e37daf67] LibGit2_jll v1.9.0+0 - [29816b5a] LibSSH2_jll v1.11.3+1 - [14a3606d] MozillaCACerts_jll v2025.5.20 - [458c3c95] OpenSSL_jll v3.5.1+0 - [efcefdf7] PCRE2_jll v10.44.0+1 - [83775a58] Zlib_jll v1.3.1+2 - [8e850ede] nghttp2_jll v1.64.0+1 - [3f19e933] p7zip_jll v17.5.0+2 - Testing Running tests... -┌ Warning: No documentation found for no_doc in Main.DocumenterReferenceTestMod. Skipping from API reference. -└ @ DocumenterReference ~/Research/logiciels/dev/control-toolbox/CTBase/ext/DocumenterReference.jl:682 -┌ Warning: Unable to determine HTML(edit_link = ...) from remote HEAD branch, defaulting to "master". -│ Calling `git remote` failed with an exception. Set JULIA_DEBUG=Documenter to see the error. -│ Unless this is due to a configuration error, the relevant variable should be set explicitly. -└ @ Documenter ~/.julia/packages/Documenter/xvqbW/src/utilities/utilities.jl:680 -[ Info: APIBuilder: creating API reference -┌ Warning: No documentation found for AbstractFoo in Main.DocumenterReferenceTestMod. Skipping from API reference. -└ @ DocumenterReference ~/Research/logiciels/dev/control-toolbox/CTBase/ext/DocumenterReference.jl:682 -┌ Warning: No documentation found for Foo in Main.DocumenterReferenceTestMod. Skipping from API reference. -└ @ DocumenterReference ~/Research/logiciels/dev/control-toolbox/CTBase/ext/DocumenterReference.jl:682 -┌ Warning: No documentation found for MYCONST in Main.DocumenterReferenceTestMod. Skipping from API reference. -└ @ DocumenterReference ~/Research/logiciels/dev/control-toolbox/CTBase/ext/DocumenterReference.jl:682 -┌ Warning: No documentation found for no_doc in Main.DocumenterReferenceTestMod. Skipping from API reference. -└ @ DocumenterReference ~/Research/logiciels/dev/control-toolbox/CTBase/ext/DocumenterReference.jl:682 -Test Summary: | Pass Total Time -CTBase tests | 175 175 6.1s - suite/extensions/test_documenter_reference.jl | 175 175 6.1s - Testing CTBase tests passed diff --git a/test_ext.log b/test_ext.log deleted file mode 100644 index ea7b2f0a..00000000 --- a/test_ext.log +++ /dev/null @@ -1,191 +0,0 @@ - Testing CTBase - Status `/private/var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_SEUFgh/Project.toml` - [4c88cf16] Aqua v0.8.14 - [54762871] CTBase v0.17.4 `~/Research/logiciels/dev/control-toolbox/CTBase` - [a2441757] Coverage v1.8.1 - [ffbed154] DocStringExtensions v0.9.5 - [e30172f5] Documenter v1.16.1 - [d0879d2d] MarkdownAST v0.1.2 - [bac558e1] OrderedCollections v1.8.1 - [d6f4376e] Markdown v1.11.0 - [8dfed614] Test v1.11.0 - Status `/private/var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_SEUFgh/Manifest.toml` - [a4c015fc] ANSIColoredPrinters v0.0.1 - [1520ce14] AbstractTrees v0.4.5 - [4c88cf16] Aqua v0.8.14 - [c7e460c6] ArgParse v1.2.0 - [d1d4a3ce] BitFlags v0.1.9 - [54762871] CTBase v0.17.4 `~/Research/logiciels/dev/control-toolbox/CTBase` - [944b1d66] CodecZlib v0.7.8 - [34da2185] Compat v4.18.1 - [f0e56b4a] ConcurrentUtilities v2.5.0 - [a2441757] Coverage v1.8.1 - [c36e975a] CoverageTools v1.4.1 - [ffbed154] DocStringExtensions v0.9.5 - [e30172f5] Documenter v1.16.1 - [460bff9d] ExceptionUnwrapping v0.1.11 - [d7ba0133] Git v1.5.0 - [cd3eb016] HTTP v1.10.19 - [b5f81e59] IOCapture v1.0.0 - [692b3bcd] JLLWrappers v1.7.1 - [682c06a0] JSON v1.4.0 - [70703baa] JuliaSyntax v1.0.2 - [0e77f7df] LazilyInitializedFields v1.3.0 - [e6f89c97] LoggingExtras v1.2.0 - [d0879d2d] MarkdownAST v0.1.2 - [739be429] MbedTLS v1.1.9 - [4d8831e6] OpenSSL v1.6.1 - [bac558e1] OrderedCollections v1.8.1 - [69de0a69] Parsers v2.8.3 - [aea7be01] PrecompileTools v1.3.3 - [21216c6a] Preferences v1.5.1 - [2792f1a3] RegistryInstances v0.1.0 - [6c6a2e73] Scratch v1.3.0 - [777ac1f9] SimpleBufferStream v1.2.0 - [ec057cc2] StructUtils v2.6.2 - [b718987f] TextWrap v1.0.2 - [3bb67fe8] TranscodingStreams v0.11.3 - [5c2747f8] URIs v1.6.1 - [2e619515] Expat_jll v2.7.3+0 - [020c3dae] Git_LFS_jll v3.7.0+0 - [f8c6e375] Git_jll v2.52.0+0 - [94ce4f54] Libiconv_jll v1.18.0+0 - [c8ffd9c3] MbedTLS_jll v2.28.1010+0 - [9bd350c2] OpenSSH_jll v10.2.1+0 - [0dad84c5] ArgTools v1.1.2 - [56f22d72] Artifacts v1.11.0 - [2a0f44e3] Base64 v1.11.0 - [ade2ca70] Dates v1.11.0 - [f43a241f] Downloads v1.6.0 - [7b1f6079] FileWatching v1.11.0 - [b77e0a4c] InteractiveUtils v1.11.0 - [ac6e5ff7] JuliaSyntaxHighlighting v1.12.0 - [b27032c2] LibCURL v0.6.4 - [76f85450] LibGit2 v1.11.0 - [8f399da3] Libdl v1.11.0 - [56ddb016] Logging v1.11.0 - [d6f4376e] Markdown v1.11.0 - [ca575930] NetworkOptions v1.3.0 - [44cfe95a] Pkg v1.12.0 - [de0858da] Printf v1.11.0 - [3fa0cd96] REPL v1.11.0 - [9a3f8284] Random v1.11.0 - [ea8e919c] SHA v0.7.0 - [9e88b42a] Serialization v1.11.0 - [6462fe0b] Sockets v1.11.0 - [f489334b] StyledStrings v1.11.0 - [fa267f1f] TOML v1.0.3 - [a4e569a6] Tar v1.10.0 - [8dfed614] Test v1.11.0 - [cf7118a7] UUIDs v1.11.0 - [4ec0a83e] Unicode v1.11.0 - [deac9b47] LibCURL_jll v8.11.1+1 - [e37daf67] LibGit2_jll v1.9.0+0 - [29816b5a] LibSSH2_jll v1.11.3+1 - [14a3606d] MozillaCACerts_jll v2025.5.20 - [458c3c95] OpenSSL_jll v3.5.1+0 - [efcefdf7] PCRE2_jll v10.44.0+1 - [83775a58] Zlib_jll v1.3.1+2 - [8e850ede] nghttp2_jll v1.64.0+1 - [3f19e933] p7zip_jll v17.5.0+2 - Testing Running tests... -WARNING: Method definition _find_symbol_test_file_rel(Any, Any) in module TestRunner at /Users/ocots/Research/logiciels/dev/control-toolbox/CTBase/test/suite/extensions/test_coverage_edge_cases.jl:98 overwritten at /Users/ocots/Research/logiciels/dev/control-toolbox/CTBase/test/suite/extensions/test_coverage_edge_cases.jl:125. -WARNING: Method definition kwcall(NamedTuple{names, T} where T<:Tuple where names, typeof(TestRunner._find_symbol_test_file_rel), Any, Any) in module TestRunner at /Users/ocots/Research/logiciels/dev/control-toolbox/CTBase/test/suite/extensions/test_coverage_edge_cases.jl:98 overwritten at /Users/ocots/Research/logiciels/dev/control-toolbox/CTBase/test/suite/extensions/test_coverage_edge_cases.jl:125. -WARNING: Method definition test_baz() in module Main at /var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_jotrtG/test_baz.jl:1 overwritten on the same line (check for duplicate calls to `include`). -[ Info: CoverageTools.process_folder: Searching /private/var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_imKs6z/src for .jl files... -[ Info: CoverageTools.process_file: Detecting coverage for /private/var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_imKs6z/src/foo.jl -[ Info: CoverageTools.process_cov: processing /private/var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_imKs6z/src/foo.jl.1234.cov -[ Info: CoverageTools.process_folder: Searching /private/var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_imKs6z/test for .jl files... -[ Info: CoverageTools.process_folder: Searching src for .jl files... -[ Info: CoverageTools.process_file: Detecting coverage for src/foo.jl -[ Info: CoverageTools.process_cov: processing src/foo.jl.1234.cov -[ Info: CoverageTools.process_folder: Searching /var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_KtpdnW/src for .jl files... -[ Info: CoverageTools.process_file: Detecting coverage for /var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_KtpdnW/src/bar.jl -[ Info: CoverageTools.process_cov: processing /var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_KtpdnW/src/bar.jl.1234.cov -┌ Warning: No documentation found for no_doc in Main.DocumenterReferenceTestMod. Skipping from API reference. -└ @ DocumenterReference ~/Research/logiciels/dev/control-toolbox/CTBase/ext/DocumenterReference.jl:682 -┌ Warning: Unable to determine HTML(edit_link = ...) from remote HEAD branch, defaulting to "master". -│ Calling `git remote` failed with an exception. Set JULIA_DEBUG=Documenter to see the error. -│ Unless this is due to a configuration error, the relevant variable should be set explicitly. -└ @ Documenter ~/.julia/packages/Documenter/xvqbW/src/utilities/utilities.jl:680 -[ Info: APIBuilder: creating API reference -┌ Warning: No documentation found for AbstractFoo in Main.DocumenterReferenceTestMod. Skipping from API reference. -└ @ DocumenterReference ~/Research/logiciels/dev/control-toolbox/CTBase/ext/DocumenterReference.jl:682 -┌ Warning: No documentation found for Foo in Main.DocumenterReferenceTestMod. Skipping from API reference. -└ @ DocumenterReference ~/Research/logiciels/dev/control-toolbox/CTBase/ext/DocumenterReference.jl:682 -┌ Warning: No documentation found for MYCONST in Main.DocumenterReferenceTestMod. Skipping from API reference. -└ @ DocumenterReference ~/Research/logiciels/dev/control-toolbox/CTBase/ext/DocumenterReference.jl:682 -┌ Warning: No documentation found for no_doc in Main.DocumenterReferenceTestMod. Skipping from API reference. -└ @ DocumenterReference ~/Research/logiciels/dev/control-toolbox/CTBase/ext/DocumenterReference.jl:682 -ERROR: LoadError: Some tests did not pass: 261 passed, 1 failed, 0 errored, 0 broken. -in expression starting at /Users/ocots/Research/logiciels/dev/control-toolbox/CTBase/test/runtests.jl:47 -DocumenterReference: Edge cases: Test Failed at /Users/ocots/Research/logiciels/dev/control-toolbox/CTBase/test/suite/extensions/test_coverage_edge_cases.jl:141 - Expression: Documenter.Selectors.order(DR.APIBuilder) == 0.0 - Evaluated: 0.5 == 0.0 - -Stacktrace: - [1] macro expansion - @ ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Test/src/Test.jl:680 [inlined] - [2] macro expansion - @ ~/Research/logiciels/dev/control-toolbox/CTBase/test/suite/extensions/test_coverage_edge_cases.jl:141 [inlined] - [3] macro expansion - @ ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Test/src/Test.jl:1776 [inlined] - [4] macro expansion - @ ~/Research/logiciels/dev/control-toolbox/CTBase/test/suite/extensions/test_coverage_edge_cases.jl:141 [inlined] - [5] macro expansion - @ ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Test/src/Test.jl:1776 [inlined] - [6] test_coverage_edge_cases() - @ Main.TestCoverageEdgeCases ~/Research/logiciels/dev/control-toolbox/CTBase/test/suite/extensions/test_coverage_edge_cases.jl:89 -✓ Coverage post-processing start -✓ Reset coverage/ directory -✓ Coverage post-processing start -✓ Reset coverage/ directory -Cleaned 1 existing .cov files from source directories -✓ Moved 3 .cov files to /private/var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_LSvG6w/coverage/cov -✓ Coverage post-processing completed successfully -✓ Coverage post-processing start -✓ Reset coverage/ directory -✓ Generating coverage report -✓ Wrote coverage report to /private/var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_imKs6z/coverage/lcov.info -✓ Moved 1 .cov files to /private/var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_imKs6z/coverage/cov -✓ Coverage post-processing completed successfully -✓ Generating coverage report -✓ Wrote coverage report to /var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_NpnMWE/coverage/lcov.info -✓ Generating coverage report -✓ Wrote coverage report to /var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_FrIzXf/coverage/lcov.info -Test Summary: | Pass Fail Total Time -CTBase tests | 261 1 262 11.6s - suite/extensions/test_coverage_edge_cases.jl | 7 1 8 1.8s - Coverage and Test Edge Cases | 7 1 8 1.1s - TestRunner: file exists then disappears | 4 4 0.0s - DocumenterReference: Edge cases | 3 1 4 1.1s - suite/extensions/test_coverage_post_process.jl | 19 19 1.8s - suite/extensions/test_documenter_reference.jl | 175 175 5.4s - suite/extensions/test_testrunner.jl | 60 60 2.6s -RNG of the outermost testset: Random.Xoshiro(0xc4c7a939a50c7a1d, 0x3b2986f825734f14, 0xde337c1fd7181b33, 0x47fc227b13ff1a46, 0xc52a9c1ed4d8a742) -ERROR: Package CTBase errored during testing -Stacktrace: - [1] pkgerror(msg::String) - @ Pkg.Types ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Pkg/src/Types.jl:68 - [2] test(ctx::Pkg.Types.Context, pkgs::Vector{PackageSpec}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool) - @ Pkg.Operations ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Pkg/src/Operations.jl:2427 - [3] test - @ ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Pkg/src/Operations.jl:2280 [inlined] - [4] test(ctx::Pkg.Types.Context, pkgs::Vector{PackageSpec}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Vector{String}, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool, kwargs::@Kwargs{io::IOContext{IO}}) - @ Pkg.API ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Pkg/src/API.jl:484 - [5] test(pkgs::Vector{PackageSpec}; io::IOContext{IO}, kwargs::@Kwargs{test_args::Vector{String}}) - @ Pkg.API ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Pkg/src/API.jl:164 - [6] test(pkgs::Vector{String}; kwargs::@Kwargs{test_args::Vector{String}}) - @ Pkg.API ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Pkg/src/API.jl:152 - [7] test - @ ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Pkg/src/API.jl:152 [inlined] - [8] #test#81 - @ ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/Pkg/src/API.jl:151 [inlined] - [9] top-level scope - @ none:1 - [10] eval(m::Module, e::Any) - @ Core ./boot.jl:489 - [11] exec_options(opts::Base.JLOptions) - @ Base ./client.jl:283 - [12] _start() - @ Base ./client.jl:550 diff --git a/test_ext_v2.log b/test_ext_v2.log deleted file mode 100644 index 4afaf427..00000000 --- a/test_ext_v2.log +++ /dev/null @@ -1,143 +0,0 @@ - Testing CTBase - Status `/private/var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_7l99TA/Project.toml` - [4c88cf16] Aqua v0.8.14 - [54762871] CTBase v0.17.4 `~/Research/logiciels/dev/control-toolbox/CTBase` - [a2441757] Coverage v1.8.1 - [ffbed154] DocStringExtensions v0.9.5 - [e30172f5] Documenter v1.16.1 - [d0879d2d] MarkdownAST v0.1.2 - [bac558e1] OrderedCollections v1.8.1 - [d6f4376e] Markdown v1.11.0 - [8dfed614] Test v1.11.0 - Status `/private/var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_7l99TA/Manifest.toml` - [a4c015fc] ANSIColoredPrinters v0.0.1 - [1520ce14] AbstractTrees v0.4.5 - [4c88cf16] Aqua v0.8.14 - [c7e460c6] ArgParse v1.2.0 - [d1d4a3ce] BitFlags v0.1.9 - [54762871] CTBase v0.17.4 `~/Research/logiciels/dev/control-toolbox/CTBase` - [944b1d66] CodecZlib v0.7.8 - [34da2185] Compat v4.18.1 - [f0e56b4a] ConcurrentUtilities v2.5.0 - [a2441757] Coverage v1.8.1 - [c36e975a] CoverageTools v1.4.1 - [ffbed154] DocStringExtensions v0.9.5 - [e30172f5] Documenter v1.16.1 - [460bff9d] ExceptionUnwrapping v0.1.11 - [d7ba0133] Git v1.5.0 - [cd3eb016] HTTP v1.10.19 - [b5f81e59] IOCapture v1.0.0 - [692b3bcd] JLLWrappers v1.7.1 - [682c06a0] JSON v1.4.0 - [70703baa] JuliaSyntax v1.0.2 - [0e77f7df] LazilyInitializedFields v1.3.0 - [e6f89c97] LoggingExtras v1.2.0 - [d0879d2d] MarkdownAST v0.1.2 - [739be429] MbedTLS v1.1.9 - [4d8831e6] OpenSSL v1.6.1 - [bac558e1] OrderedCollections v1.8.1 - [69de0a69] Parsers v2.8.3 - [aea7be01] PrecompileTools v1.3.3 - [21216c6a] Preferences v1.5.1 - [2792f1a3] RegistryInstances v0.1.0 - [6c6a2e73] Scratch v1.3.0 - [777ac1f9] SimpleBufferStream v1.2.0 - [ec057cc2] StructUtils v2.6.2 - [b718987f] TextWrap v1.0.2 - [3bb67fe8] TranscodingStreams v0.11.3 - [5c2747f8] URIs v1.6.1 - [2e619515] Expat_jll v2.7.3+0 - [020c3dae] Git_LFS_jll v3.7.0+0 - [f8c6e375] Git_jll v2.52.0+0 - [94ce4f54] Libiconv_jll v1.18.0+0 - [c8ffd9c3] MbedTLS_jll v2.28.1010+0 - [9bd350c2] OpenSSH_jll v10.2.1+0 - [0dad84c5] ArgTools v1.1.2 - [56f22d72] Artifacts v1.11.0 - [2a0f44e3] Base64 v1.11.0 - [ade2ca70] Dates v1.11.0 - [f43a241f] Downloads v1.6.0 - [7b1f6079] FileWatching v1.11.0 - [b77e0a4c] InteractiveUtils v1.11.0 - [ac6e5ff7] JuliaSyntaxHighlighting v1.12.0 - [b27032c2] LibCURL v0.6.4 - [76f85450] LibGit2 v1.11.0 - [8f399da3] Libdl v1.11.0 - [56ddb016] Logging v1.11.0 - [d6f4376e] Markdown v1.11.0 - [ca575930] NetworkOptions v1.3.0 - [44cfe95a] Pkg v1.12.0 - [de0858da] Printf v1.11.0 - [3fa0cd96] REPL v1.11.0 - [9a3f8284] Random v1.11.0 - [ea8e919c] SHA v0.7.0 - [9e88b42a] Serialization v1.11.0 - [6462fe0b] Sockets v1.11.0 - [f489334b] StyledStrings v1.11.0 - [fa267f1f] TOML v1.0.3 - [a4e569a6] Tar v1.10.0 - [8dfed614] Test v1.11.0 - [cf7118a7] UUIDs v1.11.0 - [4ec0a83e] Unicode v1.11.0 - [deac9b47] LibCURL_jll v8.11.1+1 - [e37daf67] LibGit2_jll v1.9.0+0 - [29816b5a] LibSSH2_jll v1.11.3+1 - [14a3606d] MozillaCACerts_jll v2025.5.20 - [458c3c95] OpenSSL_jll v3.5.1+0 - [efcefdf7] PCRE2_jll v10.44.0+1 - [83775a58] Zlib_jll v1.3.1+2 - [8e850ede] nghttp2_jll v1.64.0+1 - [3f19e933] p7zip_jll v17.5.0+2 - Testing Running tests... -WARNING: Method definition _find_symbol_test_file_rel(Any, Any) in module TestRunner at /Users/ocots/Research/logiciels/dev/control-toolbox/CTBase/test/suite/extensions/test_coverage_edge_cases.jl:98 overwritten at /Users/ocots/Research/logiciels/dev/control-toolbox/CTBase/test/suite/extensions/test_coverage_edge_cases.jl:125. -WARNING: Method definition kwcall(NamedTuple{names, T} where T<:Tuple where names, typeof(TestRunner._find_symbol_test_file_rel), Any, Any) in module TestRunner at /Users/ocots/Research/logiciels/dev/control-toolbox/CTBase/test/suite/extensions/test_coverage_edge_cases.jl:98 overwritten at /Users/ocots/Research/logiciels/dev/control-toolbox/CTBase/test/suite/extensions/test_coverage_edge_cases.jl:125. -WARNING: Method definition test_baz() in module Main at /var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_NWLpqC/test_baz.jl:1 overwritten on the same line (check for duplicate calls to `include`). -[ Info: CoverageTools.process_folder: Searching /private/var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_ACRESI/src for .jl files... -[ Info: CoverageTools.process_file: Detecting coverage for /private/var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_ACRESI/src/foo.jl -[ Info: CoverageTools.process_cov: processing /private/var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_ACRESI/src/foo.jl.1234.cov -[ Info: CoverageTools.process_folder: Searching /private/var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_ACRESI/test for .jl files... -[ Info: CoverageTools.process_folder: Searching src for .jl files... -[ Info: CoverageTools.process_file: Detecting coverage for src/foo.jl -[ Info: CoverageTools.process_cov: processing src/foo.jl.1234.cov -[ Info: CoverageTools.process_folder: Searching /var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_fTwSBR/src for .jl files... -[ Info: CoverageTools.process_file: Detecting coverage for /var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_fTwSBR/src/bar.jl -[ Info: CoverageTools.process_cov: processing /var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_fTwSBR/src/bar.jl.1234.cov -┌ Warning: No documentation found for no_doc in Main.DocumenterReferenceTestMod. Skipping from API reference. -└ @ DocumenterReference ~/Research/logiciels/dev/control-toolbox/CTBase/ext/DocumenterReference.jl:682 -┌ Warning: Unable to determine HTML(edit_link = ...) from remote HEAD branch, defaulting to "master". -│ Calling `git remote` failed with an exception. Set JULIA_DEBUG=Documenter to see the error. -│ Unless this is due to a configuration error, the relevant variable should be set explicitly. -└ @ Documenter ~/.julia/packages/Documenter/xvqbW/src/utilities/utilities.jl:680 -[ Info: APIBuilder: creating API reference -┌ Warning: No documentation found for AbstractFoo in Main.DocumenterReferenceTestMod. Skipping from API reference. -└ @ DocumenterReference ~/Research/logiciels/dev/control-toolbox/CTBase/ext/DocumenterReference.jl:682 -┌ Warning: No documentation found for Foo in Main.DocumenterReferenceTestMod. Skipping from API reference. -└ @ DocumenterReference ~/Research/logiciels/dev/control-toolbox/CTBase/ext/DocumenterReference.jl:682 -┌ Warning: No documentation found for MYCONST in Main.DocumenterReferenceTestMod. Skipping from API reference. -└ @ DocumenterReference ~/Research/logiciels/dev/control-toolbox/CTBase/ext/DocumenterReference.jl:682 -┌ Warning: No documentation found for no_doc in Main.DocumenterReferenceTestMod. Skipping from API reference. -└ @ DocumenterReference ~/Research/logiciels/dev/control-toolbox/CTBase/ext/DocumenterReference.jl:682 -✓ Coverage post-processing start -✓ Reset coverage/ directory -✓ Coverage post-processing start -✓ Reset coverage/ directory -Cleaned 1 existing .cov files from source directories -✓ Moved 3 .cov files to /private/var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_F6g2jS/coverage/cov -✓ Coverage post-processing completed successfully -✓ Coverage post-processing start -✓ Reset coverage/ directory -✓ Generating coverage report -✓ Wrote coverage report to /private/var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_ACRESI/coverage/lcov.info -✓ Moved 1 .cov files to /private/var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_ACRESI/coverage/cov -✓ Coverage post-processing completed successfully -✓ Generating coverage report -✓ Wrote coverage report to /var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_I0zj0p/coverage/lcov.info -✓ Generating coverage report -✓ Wrote coverage report to /var/folders/xh/6cj27y_n2_v3l0h35s5p5pkh0000gp/T/jl_06HBhL/coverage/lcov.info -Test Summary: | Pass Total Time -CTBase tests | 262 262 10.7s - suite/extensions/test_coverage_edge_cases.jl | 8 8 0.7s - suite/extensions/test_coverage_post_process.jl | 19 19 1.8s - suite/extensions/test_documenter_reference.jl | 175 175 5.5s - suite/extensions/test_testrunner.jl | 60 60 2.6s - Testing CTBase tests passed