File upstream issues from languages ng

dev-docs/issues/.next-id.txt

@@ -1 +1 @@
-185
+190

dev-docs/issues/185-rep-parses-each-source-independently.md

@@ -0,0 +1,75 @@
+# 185 - plcc-rep parses each SOURCE independently, unlike PLCC's `rep`
+
+**Type:** docs
+**Date:** 2026-08-11
+
+<!--
+Classify by user-facing impact, not by whether something was "broken".
+`fix` and `feat` bump the release version (see [tool.semantic_release]
+in pyproject.toml); reserve them for changes to the shipped package
+(src/). A bug in a test, script, or CI workflow (bin/, tests/,
+.github/) is still a bug, but it's not user-facing — classify it
+`test` or `chore` instead so it doesn't spin the version. `docs` is for
+documentation content, and never bumps the version either way.
+-->
+
+## Description
+
+`plcc-rep` parses each `SOURCE` argument as an independent token stream.
+Old PLCC's `rep` joined its file arguments into a single stream, so a
+program split across two files parsed as one program. Under plcc-ng the
+same invocation fails.
+
+This is not a defect — per-source parsing is the design issue
+[#008](done/008-parse-multi-program-streaming.md) settled on ("Each source
+is parsed independently, producing one or more trees per source"), and
+[`SourceRunner.run`](../../src/plcc/cmd/source_runner.py) implements it
+uniformly for `plcc-scan`, `plcc-parse`, and `plcc-rep`. But it is an
+undocumented breaking change: [docs/migration.md](../../docs/migration.md)'s
+command table maps `rep [-t] [-n] [file...]` → `plcc-rep [file...]` with an
+empty notes column, which reads as "same behavior, new name."
+
+Splitting one program across files is a real course-material pattern — an
+exercise that hands students a partial program in one file and the
+remainder in another — so this silently changes what an existing
+demonstration does, and does it with a parse error rather than a diagnosis.
+
+## Steps to Reproduce
+
+1. With a spec for a simple expression language, put `+(3` in `p1` and
+   `,4)` in `p2`:
+
+   ```
+   $ plcc-rep p1 p2
+   plcc-parser-table: -:1:3: error: expected 'RPAREN', got end of file
+   plcc-parser-table: -:1:1: error: unexpected 'COMMA', no production for 'Program'
+   ```
+
+2. Concatenating first works as expected:
+
+   ```
+   $ cat p1 p2 | plcc-rep
+   7
+   ```
+
+## Notes
+
+The minimum fix is documentation: add a **Breaking behavior changes** entry
+and fill in the `rep` row's notes column in
+[docs/migration.md](../../docs/migration.md), stating that each `SOURCE` is
+its own token stream and that `cat f1 f2 | plcc-rep` reproduces the old
+behavior. Issue
+[#159](done/159-migration-guide-missing-breaking-changes-callout.md) is the
+precedent for that shape.
+
+Worth deciding at the same time whether the divergence is wanted at all for
+`plcc-rep` specifically. `plcc-scan` and `plcc-parse` have a clear reason to
+keep sources separate — their output is per-source records carrying a
+`source` field. `plcc-rep`'s output is the *program's* result, where "these
+files are one program" is at least as plausible a reading as "these files
+are separate programs." If joining is the wanted behavior, it is a `feat` on
+top of this docs entry, not a replacement for it: the divergence has already
+shipped and still needs a migration-guide note.
+
+Found while migrating a set of course languages to plcc-ng, where the
+affected example is a two-file program built exactly this way.

dev-docs/issues/186-rep-deadlocks-on-partial-stdout-line.md

@@ -0,0 +1,84 @@
+# 186 - plcc-rep deadlocks on a partial stdout line
+
+**Type:** fix
+**Date:** 2026-08-11
+
+<!--
+Classify by user-facing impact, not by whether something was "broken".
+`fix` and `feat` bump the release version (see [tool.semantic_release]
+in pyproject.toml); reserve them for changes to the shipped package
+(src/). A bug in a test, script, or CI workflow (bin/, tests/,
+.github/) is still a bug, but it's not user-facing — classify it
+`test` or `chore` instead so it doesn't spin the version. `docs` is for
+documentation content, and never bumps the version either way.
+-->
+
+## Description
+
+A semantic action that writes a partial line (no trailing newline) to
+stdout deadlocks `plcc-rep` with no diagnostic: no stdout, no stderr, no
+exit.
+
+`plcc-rep` runs the generated program as a subprocess and treats its stdout
+as a private, line-oriented JSON channel.
+[`_read_response`](../../src/plcc/cmd/rep.py) reads one line at a time; a
+line that fails to parse as JSON (or parses to something without a `kind`)
+is printed verbatim and the loop continues, waiting for the next line:
+
+```python
+line = raw.decode('utf-8', errors='replace').rstrip('\n')
+try:
+    record = json.loads(line)
+except json.JSONDecodeError:
+    print(line)
+    continue
+```
+
+That works when the stray text ends in a newline — the result record
+arrives intact on the next `readline()`. It does not work for a partial
+line: the unterminated text merges with the following JSON result line into
+one unparseable line, which is printed (destroying the result record), and
+`readline()` then blocks forever waiting for a result that will never come.
+
+Measured in both the Python and JavaScript targets, via stdin and via a
+`SOURCE` file: `timeout` reports exit 124, with no stdout and no stderr —
+the worst available failure mode, since nothing tells the caller what
+happened. A student who puts a `print` (or the target-language equivalent)
+in a semantic action and forgets the trailing newline gets a hang with no
+message, indistinguishable from an infinite loop in their own program.
+
+Note that the newline-terminated case survives only through the
+unparseable-line fallback above, which is an accident of the implementation
+rather than a supported output channel. It happens to print the raw line
+before the real result, which fakes the interleaving a language's `display`
+would want.
+
+## Steps to Reproduce
+
+1. A spec whose semantic action for some expression writes a partial line,
+   e.g. `sys.stdout.write("7")` with no trailing `\n`.
+2. `echo '<partial-write expression>' | timeout 5 plcc-rep`
+3. Actual: exit 124, no output at all. Expected: either the partial write is
+   surfaced, or `plcc-rep` reports the malformed channel — anything other
+   than hanging silently.
+
+## Notes
+
+The narrow fix is for `_read_response` to stop treating an unbounded
+`readline()` as acceptable: read incrementally, or bound the wait, or at
+minimum detect that the subprocess has gone idle with unterminated data
+buffered and report it. Whatever the mechanism, the requirement is that a
+partial-line write produce a message rather than a hang.
+
+The wider fix is issue [#187](187-rep-lacks-output-and-clean-exit-records.md):
+give semantic actions a supported `output` record kind, so nothing they emit
+travels as raw stdout in the first place. That would close this off at the
+source, but the defensive handling here is still worth having on its own —
+user code can always write to stdout directly.
+
+Found while migrating a course language whose `display`, `display#`, `putc`,
+`puts`, and `newline` primitives are partial-line writers by design (a
+trailing newline is exactly what its separate `newline` expression is for).
+That port works around this by buffering all output and returning it as part
+of `_run()`'s result rather than writing it directly — a workaround with its
+own cost, described in [#187](187-rep-lacks-output-and-clean-exit-records.md).

dev-docs/issues/187-rep-lacks-output-and-clean-exit-records.md

@@ -0,0 +1,99 @@
+# 187 - plcc-rep lacks output and clean-exit record kinds
+
+**Type:** feat
+**Date:** 2026-08-11
+
+<!--
+Classify by user-facing impact, not by whether something was "broken".
+`fix` and `feat` bump the release version (see [tool.semantic_release]
+in pyproject.toml); reserve them for changes to the shipped package
+(src/). A bug in a test, script, or CI workflow (bin/, tests/,
+.github/) is still a bug, but it's not user-facing — classify it
+`test` or `chore` instead so it doesn't spin the version. `docs` is for
+documentation content, and never bumps the version either way.
+-->
+
+## Description
+
+Semantic actions have no supported way to emit user-visible output, and no
+way to end the session cleanly. Both are missing record kinds in
+`plcc-rep`'s protocol.
+
+[`_render_record`](../../src/plcc/cmd/rep.py) already dispatches on
+`record['kind']` — `result`, `error`, `specification_error`, and anything
+else is a hard error — so the shape of the fix is a new record kind handled
+there, plus a hook in each target runtime that lets `_run()` emit it. Two
+gaps this would close:
+
+1. **Output.** There is no record kind for "the program printed this."
+   Writing directly to stdout collides with `plcc-rep`'s use of stdout as
+   its own JSON channel, and hangs outright on a partial line (issue
+   [#186](186-rep-deadlocks-on-partial-stdout-line.md)). The only way out
+   today is to buffer everything a semantic action would have printed and
+   fold it into `_run()`'s returned value. An `output` record kind, emitted
+   as each write happens rather than buffered and flushed at the end, would
+   let output interleave with results the way old PLCC's `rep` did, and
+   would remove the need for that workaround entirely.
+
+2. **Clean exit.** There is no record kind for "the program is
+   intentionally done." A language with an `exit` expression calls the
+   target language's process-exit function, which — because `plcc-rep` runs
+   the generated program as a subprocess — closes the pipe mid-protocol.
+   Measured:
+
+   ```
+   $ printf 'display 1\nexit\ndisplay 2\n' | plcc-rep
+   1nil                                          # stdout
+   plcc-rep: interpreter exited unexpectedly     # stderr
+   $ echo $?
+   1
+   ```
+
+   Under old PLCC, `rep` *was* the process, so this was a clean quit with
+   status 0. Under plcc-ng a deliberate quit reads as a crash: wrong stderr
+   message, wrong exit status, for want of a `session-end` (or similar)
+   record the subprocess could emit before exiting.
+
+### Second symptom: the buffering workaround swallows output when evaluation raises
+
+Measured on the Python target, in a port using the buffered-output
+workaround from (1):
+
+```
+$ printf '{display 1 ; error "boom"}\n' | plcc-rep
+[98,111,111,109]
+```
+
+The `1` is gone. Old PLCC's `System.out.print` had already written it to
+stdout by the time the exception was thrown; a buffer standing in for
+stdout is discarded along with the statement. A definition whose right-hand
+side raises loses its output the same way.
+
+That is not a defect in the workaround — it is what the workaround costs.
+Buffering is only necessary because there is no supported channel, and the
+one alternative that would preserve the output (writing to stdout as it is
+produced) is exactly what deadlocks the tool. An `output` record kind
+removes the trade-off: each write emits its record the moment it runs, so an
+error later in the same statement cannot retroactively swallow it, and
+nothing has to be held until `_run` returns.
+
+It is worth recording as a distinct symptom because it changes observable
+language behavior, where the deadlock only changes failure mode.
+
+## Notes
+
+Both additions are protocol changes, so they touch every target runtime
+(Python, Java, JavaScript, Haskell) as well as `_render_record`, and they
+belong in [docs/language-guide/](../../docs/language-guide/) once the
+record kinds are documented. `_wait_for_ready`'s `ready` record is the
+precedent for a non-`result` kind flowing across the same channel.
+
+Cross-links issue [#186](186-rep-deadlocks-on-partial-stdout-line.md), the
+partial-line deadlock this same addition would also close off, since output
+would travel through a typed record instead of raw stdout.
+
+Found while migrating a course language whose `display` family and `exit`
+expression need exactly these two channels. That port does not block on
+this: the buffered-output workaround ships, and `exit`'s status-code
+divergence ships as-is with a comment in each spec explaining it, since
+`exit` is used by no example programs and no tests.

dev-docs/issues/188-follow-set-omits-nullable-tail.md

@@ -0,0 +1,116 @@
+# 188 - FOLLOW set omits the nullable tail, breaking empty alternatives
+
+**Type:** fix
+**Date:** 2026-08-11
+
+<!--
+Classify by user-facing impact, not by whether something was "broken".
+`fix` and `feat` bump the release version (see [tool.semantic_release]
+in pyproject.toml); reserve them for changes to the shipped package
+(src/). A bug in a test, script, or CI workflow (bin/, tests/,
+.github/) is still a bug, but it's not user-facing — classify it
+`test` or `chore` instead so it doesn't spin the version. `docs` is for
+documentation content, and never bumps the version either way.
+-->
+
+## Description
+
+The FOLLOW set of a nonterminal that is followed by a *nullable* symbol is
+under-approximated. The predict set of an **empty alternative** is exactly
+FOLLOW of its nonterminal, so the generated parse table silently loses
+entries and valid programs fail to parse — while `plcc-ll1` still reports
+`is_ll1: true` and no conflicts.
+
+The defect is in
+[`FollowSetBuilder._updateWithSingleOccuranceOfNonterminalInProduction`](../../src/plcc/spec/syntax/validations/ll1/build_follow_sets.py):
+
+```python
+else:
+    self._addFirstOfNextSymbol(rules[index + 1], nonterminal)
+    if self._canDeriveEmpty(rules[index + 1:]):
+        self._addFollowOfLHS(lhs, nonterminal)
+```
+
+It adds FIRST of the **single** next symbol, then jumps straight to the
+"whole remainder is nullable" case. The standard algorithm walks forward
+from `index + 1`, adding `FIRST(X_j) \ {ε}` and stopping at the first `X_j`
+that is not nullable. When `X_{i+1}` is nullable but the remainder is not,
+every symbol between them is skipped.
+
+A repeating (`**=`) nonterminal makes this easy to hit, because it is
+nullable by construction and is a natural thing to place in a sequence.
+
+Distinct from issue
+[#170](done/170-arbno-follow-set-missing-eof.md), which was about
+nullability being tested against only a nonterminal's first-registered
+production. That fix landed (`_canDeriveEmpty` now consults the FIRST sets);
+this is the separate forward-walk defect in the same function.
+
+## Steps to Reproduce
+
+1. A grammar in which a nonterminal with an empty alternative is followed by
+   a nullable symbol — here `<Ext>` (which has an empty alternative) is
+   followed by `<Statics>` (nullable, because `**=`):
+
+   ```
+   <ClassDecl>     ::= CLASS <Ext> <Statics> <Fields> <Methods> END
+   <Ext:Ext1>      ::= EXTENDS <Exp>
+   <Ext:Ext0>      ::=
+   <Statics>       **= STATIC <SYMBOL> EQUALS <Exp>
+   <Fields>        **= FIELD <SYMBOL>
+   <Methods>       **= METHOD <SYMBOL> EQUALS <Proc>
+   ```
+
+2. `plcc-ll1` reports `is_ll1: true`, no conflicts, and
+   `FOLLOW(Ext) = ['STATIC']`. The correct set is
+   `{STATIC, FIELD, METHOD, END}`.
+
+3. `printf 'class static x = 3 end\n' | plcc-parse` — parses. This is the
+   one class shape whose next token happens to be in the truncated FOLLOW
+   set.
+
+4. `printf 'class field x end\n' | plcc-parse` — fails:
+
+   ```
+   plcc-parser-table: -:1:7: error: unexpected 'FIELD', no production for 'Ext'
+   ```
+
+   Same for `class method m = proc() 1 end` and for the empty `class end`.
+
+## Notes
+
+Two things make this worse than an ordinary parse bug.
+
+**It is silent.** `plcc-ll1` reports the grammar LL(1)-clean, so nothing
+warns the spec author. The failure appears later, on one particular input
+shape.
+
+**The diagnosis is inverted.** The message names the token that *is* there
+(`unexpected 'FIELD'`) and the nonterminal that has no entry for it, which
+reads like a grammar-ambiguity problem. The actual cause is several rules
+away, in a symbol the author never looked at.
+
+Suggested fix, replacing the `else` branch above:
+
+```python
+else:
+    for j in range(index + 1, len(rules)):
+        self._addFirstOfNextSymbol(rules[j], nonterminal)
+        if not self._canDeriveEmpty([rules[j]]):
+            break
+    else:
+        self._addFollowOfLHS(lhs, nonterminal)
+```
+
+Whoever picks this up should add a unit test at the `build_follow_sets`
+level directly, not only an end-to-end regression: a minimal grammar
+`S -> A B C`, `A -> a | ε`, `B -> b | ε`, `C -> c`, asserting
+`FOLLOW(A) = {b, c}` rather than `{b}`. The grammar above is the
+end-to-end case.
+
+Found while migrating a course language whose class-declaration rule has
+exactly this shape. That port works around it by splitting the class body
+into a non-nullable nonterminal, so the symbol immediately following the
+empty-alternative nonterminal has a correctly computed FIRST set; the
+accepted language is unchanged, and the workaround can be reverted once
+this is fixed.