Migrate local analysis replay to jaspSyntax

.github/copilot-instructions.md

@@ -2,7 +2,7 @@
 
 ## Project Overview
 
-jaspTools is an R package that enables JASP developers to preview, debug, and test JASP analyses locally without rebuilding the entire JASP application. It replicates the JASP runtime environment in R, including RCPP bridges, data handling, and state management.
+jaspTools is an R package that enables JASP developers to preview, debug, and test JASP analyses locally without rebuilding the entire JASP application. It orchestrates the local JASP runtime in R while delegating native QML, saved `.jasp`, dataset encoding, and result-decoding semantics to `jaspSyntax` and Desktop.
 
 ## Architecture
 
@@ -16,10 +16,10 @@ jaspTools uses specialized environments for variable scoping:
 
 ### Analysis Execution Flow
 
-1. **Setup**: `setupJaspTools()` fetches dependencies (jaspBase, jaspGraphs, datasets, HTML resources) and validates paths
-2. **Options**: `analysisOptions()` parses QML files, JASP files, or JSON to generate option lists
+1. **Setup**: `setupJaspTools()` fetches dependencies (jaspBase, jaspSyntax, jaspGraphs, datasets, HTML resources) and validates paths
+2. **Options**: `analysisOptions()` reads QML defaults and saved `.jasp` options through `jaspSyntax`; JSON requests are parsed directly
 3. **Runtime Init**: `initAnalysisRuntime()` in `R/run.R` sets up dataset, state, and global RCPP masks
-4. **Execution**: `runAnalysis()` calls `jaspBase::runJaspResults()` with the analysis function
+4. **Execution**: `runAnalysis()` resolves the module QML through `jaspSyntax`, then calls `jaspBase::runWrappedAnalysis()` / `runJaspResults()`
 5. **Output**: Results are converted to JSON and optionally displayed via `view()` using JASP's HTML/JS/CSS
 
 **Critical**: S3 methods from `common.R` are temporarily exported to `.GlobalEnv` during analysis execution (see `Developers-note.md` "Handling of S3 methods").
@@ -70,28 +70,24 @@ options <- analysisOptions("path/to/analysis.jasp")  # Returns list if multiple
 # For multi-analysis files, access by index: options[[1]], options[[2]], etc.
 ```
 
-### Encoding Options and Datasets
+### Native Option and Dataset Semantics
 
-For reproducible testing, use `encodeOptionsAndDataset()` to standardize variable names and types:
+Do not reimplement option encoding, type coercion, QML parsing, or saved `.jasp`
+dataset reconstruction in jaspTools. Those semantics belong to `jaspSyntax` and
+Desktop. jaspTools should pass saved options and extracted datasets into the
+bridge and let `jaspSyntax` return the runtime-ready dataset, column mapping,
+and decoded results.
 
 ```r
-# Encode options and dataset for reproducible testing
-options <- analysisOptions("path/to/file.jasp")
+savedOptions <- analysisOptions("path/to/file.jasp")
 dataset <- extractDatasetFromJASPFile("path/to/file.jasp")
 
-encoded <- encodeOptionsAndDataset(options, dataset)
-# encoded$options: Options with variables renamed to jaspColumn1, jaspColumn2, etc.
-# encoded$dataset: Dataset with matching column names and proper type coercion
-# encoded$encodingMap: Mapping from original names to encoded names
-
-# Run with encoded data (skip type detection)
-runAnalysis("AnalysisName", encoded$dataset, encoded$options, encodedDataset = TRUE)
+runAnalysis("AnalysisName", dataset, savedOptions)
 ```
 
-The encoding process:
-1. Scans options for variables with `.types` metadata (e.g., `variables` and `variables.types`)
-2. Creates unique variable-type pairs and maps them to `jaspColumn1`, `jaspColumn2`, etc.
-3. Applies type coercion: `"nominal"` → factor, `"ordinal"` → ordered, `"scale"` → numeric
+Use `analysisRuntimeOptions()` only for inspecting the backend-prepared option
+shape. Do not feed those options back into `runAnalysis()`, because that would
+prepare an already-prepared option payload a second time.
 
 ### Generating Tests from JASP Example Files
 
@@ -122,9 +118,9 @@ makeTestsFromExamples(sanitize = TRUE)
 ```
 
 **Source Folders**: JASP example files are stored in `tests/testthat/jaspfiles/{library,verified,other}/`. The `source` argument controls which folders to process:
-- `"library"` — JASP files from the analysis library
-- `"verified"` — Manually verified/curated test files
-- `"other"` — Other/imported JASP files (default target for `path` imports)
+- `"library"` - JASP files from the analysis library
+- `"verified"` - Manually verified/curated test files
+- `"other"` - Other/imported JASP files (default target for `path` imports)
 
 **Defaults**: When `overwrite = FALSE` (default), all three sources are processed. When `overwrite = TRUE`, only `"library"` and `"other"` are processed to protect verified tests.
 
@@ -176,11 +172,15 @@ Analysis functions may have `Internal` suffixes. `findCorrectFunction()` searche
 
 ### QML Parsing
 
-`readQML()` in `R/options-parser-qml.R` strips comments, whitespace, and newlines, then uses regex to extract QML form elements. Supports `IntegerField`, `CheckBox`, `DropDown`, `RadioButtonGroup`, etc. Static elements like `SetSeed` and `BayesFactorType` inject default options.
+jaspTools does not parse QML directly. Use `jaspSyntax::readDefaultAnalysisOptions()`,
+`jaspSyntax::readAnalysisOptionsFromJaspFile()`, and
+`jaspSyntax::resolveAnalysisQml()` for native option semantics.
 
 ### State Management
 
-**State is ignored** in jaspTools (noted in README limitations). State from JASP files is stored in `.internal` but not persisted between runs. Analyses should be stateless or handle missing state gracefully.
+jaspTools keeps a standalone state file for `jaspBase` replay and decodes
+returned plot state after analysis execution. Browser-only interactions are not
+replayed.
 
 ## Common Pitfalls
 
@@ -196,17 +196,17 @@ Analysis functions may have `Internal` suffixes. `findCorrectFunction()` searche
 - **jaspGraphs**: Plotting system for JASP-compatible graphics
 - **jaspResults**: Legacy state container (now merged into jaspBase)
 - **vdiffr**: Visual regression testing for plots
-- **testthat**: Unit testing framework (requires ≥3.2.2)
+- **testthat**: Unit testing framework (requires >=3.2.2)
 
 Check versions with `.checkUpdatesJaspCorePkgs()` on package load.
 
 ## File Organization
 
-- `R/run.R`: Analysis execution, RCPP mask setup, JSON conversion. Supports `encodedDataset` parameter for pre-encoded data.
+- `R/run.R`: Analysis execution, RCPP mask setup, native QML provenance, JSON conversion, state-file replay
 - `R/test.R`: Testing infrastructure, `testAnalysis()`, `testAll()`
 - `R/test-agent.R`: Agent-friendly test wrappers, `agentTestAll()`, `agentTestAnalysis()`
 - `R/options.R`: Option parsing from QML/JASP/JSON. Cross-platform path handling for `.jasp` files.
-- `R/dataset.R`: Dataset loading, type conversion, `extractDatasetFromJASPFile()`, `encodeOptionsAndDataset()`
+- `R/dataset.R`: Dataset loading orchestration and `jaspSyntax` bridge wrappers
 - `R/rbridge.R`: RCPP bridge replacements (`.readDatasetToEndNative`, `.requestTempFileNameNative`, etc.)
 - `R/pkg-setup.R`: Initial setup, dependency fetching
 - `R/utils.R`: Module path resolution, validation, helper functions

DESCRIPTION

@@ -12,17 +12,17 @@ License: GNU General Public License
 Encoding: UTF-8
 LazyData: true
 Imports:
-  archive (>= 1.1.6),
+  callr,
   cli,
   data.table,
-  DBI,
   httr,
+  jaspBase (>= 0.20.5),
+  jaspSyntax (>= 1.3.2),
   jsonlite,
   lifecycle,
   pkgload,
   remotes,
   rjson,
-  RSQLite,
   stringi,
   stringr,
   testthat (>= 3.2.2),

Developers-note.md

@@ -10,7 +10,7 @@ jasptools uses three different mechanisms to share variables between inner funct
 1. `.internal`: environment that holds internal variables; these are used to pass information between functions.
 The state entry is special, because it interfaces directly with the JASP code. Rather than storing and then loading the state, it is passed to jasptools, which is much faster.
 2. `.pkgOptions`: environment that holds variables to be changed by users; mainly paths to resources.
-3. global variables: the rbridge in JASP defines certain global variables that JASP assumes are always globally accessible and these need to be matched in jasptools. They include `perform`, `.ppi` and `.baseCitation` (the global functions on the other hand are defined in the jasptools file rbridge.R). Global variables are set every time `run` is called through `.setRCPPMasks`.
+3. rbridge namespace objects: JASP Desktop defines Rcpp-backed objects that `jaspBase:::.fromRCPP()` can request while an analysis runs. `jaspTools` now keeps only the local developer-execution subset in its namespace, such as `.ppi`, `.baseCitation`, plot/state file providers, and `.imageBackground`. Dataset loading and column encode/decode semantics belong to `jaspSyntax`/SyntaxInterface or Desktop.
 
 #### Handling of S3 methods
 Currently it is necessary to export the S3 methods used for generic functions in JASP (e.g., those defined in `common.R`).  
@@ -22,8 +22,5 @@ jasp-desktop follows a fixed structure, meaning that in every development enviro
 Once jasptools verifies that it is located within /Tools/ it converts the relative paths to the resources to absolute paths.  
 Now, if these resources are changed, jasptools will need to be adjusted. Firstly, `zzz.R` needs to be modified where it states `# set locations of all required resources (json, analyses, html, packages)` and secondly `.pkgOptions` in `pkg-settings.R` must reflect the same changes.
 
-#### Changing to TOML
-At the moment jasptools only supports JSON format for the description files.  
-Should we decide to change this in the future, then the functions that interface with the description file must be adapted.  
-These functions can be found in `options.R` (`.analysisOptionsFromFile`), `main.R` (`run`) and in `utils.R` (`.getJSON`).  
-It is not a problem that the code that is send to `view` still uses JSON as this is unlikely to change.
+#### jaspSyntax boundary
+jaspTools now delegates module description parsing, QML option parsing, saved `.jasp` option extraction, and `.jasp` dataset extraction to `jaspSyntax`. The developer-facing contract lives in `options.R`, `dataset.R`, `run.R`, and `test-generator.R`; lower-level Desktop/SyntaxInterface behavior should stay in `jaspSyntax` rather than being reimplemented here.

NAMESPACE

@@ -2,24 +2,11 @@
 
 S3method(expect_doppelganger_fallback,default)
 S3method(expect_doppelganger_fallback,ggplot)
-S3method(extractData,AssignedVariablesList)
-S3method(extractData,CIField)
-S3method(extractData,CheckBox)
-S3method(extractData,Chi2TestTableView)
-S3method(extractData,DoubleField)
-S3method(extractData,DropDown)
-S3method(extractData,IntegerField)
-S3method(extractData,PercentField)
-S3method(extractData,RadioButtonGroup)
-S3method(extractData,Slider)
-S3method(extractData,default)
-S3method(extractData,repeatedMeasuresFactorsList)
 S3method(print,jaspAgentTestResults)
-export(addTypedDataSet)
 export(agentTestAll)
 export(agentTestAnalysis)
 export(analysisOptions)
-export(encodeOptionsAndDataset)
+export(analysisRuntimeOptions)
 export(expect_equal_plots)
 export(expect_equal_tables)
 export(extractDatasetFromJASPFile)