Skip to content

Commit 0c48f7d

Browse files
DrunkOnJavaDrunkOnJava
committed
re(RE-18): tagged-ancestor walker ships — H7 refuted empirically
Added SchemaTable::tagged_ancestor() + tagged_ancestor_map() APIs with 6 unit tests. Walks the parent chain from an untagged class until it finds an ancestor with a tag, with cycle guard. Empirical finding via probe_tagged_ancestors against 4 corpus files: - Revit_IFC5_Einhoven.rvt (2023): 405 classes, 80 tagged, 0 resolved via ancestor - 2024_Core_Interior.rvt: 395 classes, 79 tagged, 0 resolved via ancestor - racbasicsamplefamily-2024.rfa: 395/79/0 - racbasicsamplefamily-2026.rfa: 349/60/0 The class-name literals we expected (Wall, Floor, Door, etc.) are NOT in any corpus schema under those exact names. Only HostObjAttr and Element exist. The 325 untagged classes have empty parent, so there is nothing to walk. H7's premise — "tagless Wall on the wire is a HostObjAttr tag after walking parent chain" — is refuted: Wall is not a thing on the wire. Concrete subtypes (ArcWall 0x0191, VWall 0x0192, WallCGDriver 0x0197) are carried directly by their own tags. Decision: RE-11 reverts to scanning partition chunks for the 80 tagged classes directly. No ancestor expansion needed. The tagged_ancestor API stays shipped because (a) it is correct and tested, (b) a future parser improvement that extracts more parent links will silently benefit callers, and (c) the aggregate map doubles as schema-parser coverage diagnostics. Synthesis: reports/element-framing/RE-18-synthesis.md (F16/F17/F18, D6/D7/D8, Q8/Q9, full table of ancestor-chain coverage per file). 722 tests pass (+6 new), probe clean, cargo check clean.
1 parent f08a75e commit 0c48f7d

3 files changed

Lines changed: 547 additions & 0 deletions

File tree

‎examples/probe_tagged_ancestors.rs‎

Lines changed: 199 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,199 @@
1+
//! RE-18 — schema tagged-ancestor walker. Given a parsed schema,
2+
//! for every untagged class resolve the first tag-carrying ancestor
3+
//! by walking the `parent` chain.
4+
//!
5+
//! Why this matters (H7, confidence 0.7, from RE-09):
6+
//! - Schema has 405 classes but only 80 are directly tagged.
7+
//! - `Wall`, `Floor`, `Door`, `Window`, `Level`, `Grid`,
8+
//! `FamilyInstance`, `Room` are all tagless abstract parents.
9+
//! - Their instances on the wire presumably carry a concrete
10+
//! subtype tag (e.g. `ArcWall` 0x0191, `VWall` 0x0192, or a
11+
//! generic ancestor like `HostObjAttr` 0x006b).
12+
//! - RE-11's tag scan needs the full set of "tag worth scanning
13+
//! for if I care about Walls" — that's the ancestor map.
14+
//!
15+
//! Output:
16+
//! - Counts (total / directly-tagged / resolved-via-ancestor /
17+
//! unresolvable).
18+
//! - Per "interesting class" the ancestor + tag resolution.
19+
//! - Distribution of ancestor depths (how many parent hops to
20+
//! reach a tag).
21+
//! - Sample of unresolvable classes (no tag in chain) — these are
22+
//! likely mixins, abstract protocols, or classes whose parents
23+
//! live in a different serializable scope.
24+
25+
use rvt::{RevitFile, compression, formats, streams};
26+
use std::collections::BTreeMap;
27+
28+
fn main() {
29+
let args: Vec<String> = std::env::args().skip(1).collect();
30+
let project_dir = std::env::var("RVT_PROJECT_CORPUS_DIR")
31+
.unwrap_or_else(|_| "/private/tmp/rvt-corpus-probe/magnetar/Revit".into());
32+
let targets: Vec<String> = if args.is_empty() {
33+
vec![
34+
format!("{project_dir}/Revit_IFC5_Einhoven.rvt"),
35+
format!("{project_dir}/2024_Core_Interior.rvt"),
36+
"../../samples/racbasicsamplefamily-2024.rfa".to_string(),
37+
"../../samples/racbasicsamplefamily-2026.rfa".to_string(),
38+
]
39+
} else {
40+
args
41+
};
42+
43+
for path in targets {
44+
if !std::path::Path::new(&path).exists() {
45+
continue;
46+
}
47+
let Ok(mut rf) = RevitFile::open(&path) else {
48+
continue;
49+
};
50+
let Ok(raw) = rf.read_stream(streams::FORMATS_LATEST) else {
51+
continue;
52+
};
53+
let Ok(decomp) = compression::inflate_at(&raw, 0) else {
54+
continue;
55+
};
56+
let Ok(schema) = formats::parse_schema(&decomp) else {
57+
continue;
58+
};
59+
60+
let fname = std::path::Path::new(&path)
61+
.file_name()
62+
.unwrap()
63+
.to_string_lossy();
64+
println!("\n=== {fname} ===");
65+
66+
let total = schema.classes.len();
67+
let direct_tag = schema.classes.iter().filter(|c| c.tag.is_some()).count();
68+
let untagged = total - direct_tag;
69+
70+
// Resolve every untagged class through the ancestor walker.
71+
let mut resolved_via_ancestor = 0usize;
72+
let mut unresolvable: Vec<&str> = Vec::new();
73+
let mut depth_distribution: BTreeMap<usize, usize> = BTreeMap::new();
74+
let mut ancestor_popularity: BTreeMap<&str, usize> = BTreeMap::new();
75+
76+
for c in &schema.classes {
77+
if c.tag.is_some() {
78+
continue;
79+
}
80+
// Walk parent chain manually to get depth count. Using the
81+
// method gives the answer; we redo the walk here so we can
82+
// count hops.
83+
let mut depth = 0usize;
84+
let mut current = c.name.as_str();
85+
let mut found: Option<(&str, u16)> = None;
86+
loop {
87+
let Some(entry) = schema.classes.iter().find(|x| x.name == current) else {
88+
break;
89+
};
90+
if let Some(t) = entry.tag {
91+
found = Some((entry.name.as_str(), t));
92+
break;
93+
}
94+
match entry.parent.as_deref() {
95+
Some(p) => {
96+
current = p;
97+
depth += 1;
98+
}
99+
None => break,
100+
}
101+
// Guard against pathological chains.
102+
if depth > 40 {
103+
break;
104+
}
105+
}
106+
match found {
107+
Some((anc, _)) => {
108+
resolved_via_ancestor += 1;
109+
*depth_distribution.entry(depth).or_insert(0) += 1;
110+
*ancestor_popularity.entry(anc).or_insert(0) += 1;
111+
}
112+
None => unresolvable.push(&c.name),
113+
}
114+
}
115+
116+
println!(
117+
" Schema: {total} classes ({direct_tag} directly tagged, {untagged} untagged)"
118+
);
119+
println!(
120+
" Untagged: {resolved_via_ancestor} resolved via ancestor chain, \
121+
{} unresolvable",
122+
unresolvable.len()
123+
);
124+
println!(
125+
" Coverage: {} of {total} total classes now resolvable ({:.1}%)",
126+
direct_tag + resolved_via_ancestor,
127+
100.0 * (direct_tag + resolved_via_ancestor) as f64 / total as f64,
128+
);
129+
130+
println!("\n Ancestor hop-count distribution (untagged classes):");
131+
for (depth, count) in &depth_distribution {
132+
println!(" {depth:>2} hops: {count} classes");
133+
}
134+
135+
println!("\n Top-10 most-popular tagged ancestors:");
136+
let mut popular: Vec<(&&str, &usize)> = ancestor_popularity.iter().collect();
137+
popular.sort_by_key(|(_, count)| std::cmp::Reverse(**count));
138+
for (anc, count) in popular.iter().take(10) {
139+
let tag = schema
140+
.classes
141+
.iter()
142+
.find(|c| &c.name == *anc)
143+
.and_then(|c| c.tag)
144+
.unwrap_or(0);
145+
println!(" {count:>4} classes → {anc} (tag 0x{tag:04x})");
146+
}
147+
148+
// Interesting-class resolution: what tag does each key
149+
// architectural class resolve to?
150+
let interesting = [
151+
"Wall",
152+
"Floor",
153+
"Door",
154+
"Window",
155+
"Stair",
156+
"Column",
157+
"Beam",
158+
"Roof",
159+
"Ceiling",
160+
"Level",
161+
"Grid",
162+
"FamilyInstance",
163+
"Room",
164+
"HostObject",
165+
"HostObjAttr",
166+
"Element",
167+
];
168+
println!("\n Interesting-class resolution:");
169+
for name in interesting {
170+
match schema.tagged_ancestor(name) {
171+
Some((anc, tag)) if anc == name => {
172+
println!(" {name:<18} direct tag 0x{tag:04x}");
173+
}
174+
Some((anc, tag)) => {
175+
println!(" {name:<18} → {anc} (tag 0x{tag:04x})");
176+
}
177+
None => {
178+
let exists = schema.classes.iter().any(|c| c.name == name);
179+
if exists {
180+
println!(" {name:<18} UNRESOLVABLE (in schema, no tagged ancestor)");
181+
} else {
182+
println!(" {name:<18} not in schema");
183+
}
184+
}
185+
}
186+
}
187+
188+
// Sample unresolvable untagged classes
189+
if !unresolvable.is_empty() {
190+
println!(
191+
"\n Sample unresolvable untagged classes (first 8 of {}):",
192+
unresolvable.len()
193+
);
194+
for name in unresolvable.iter().take(8) {
195+
println!(" {name}");
196+
}
197+
}
198+
}
199+
}

‎reports/element-framing/RE-18-synthesis.md‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
# RE-18 synthesis — tagged-ancestor walker ships, but hypothesis H7 refuted by data
2+
3+
**Date:** 2026-04-21
4+
**Scope:** Build `SchemaTable::tagged_ancestor()` + `tagged_ancestor_map()` API, validate against real corpus, test the load-bearing hypothesis H7 from `RE-09-synthesis.md`.
5+
6+
## TL;DR
7+
8+
Shipped: the ancestor-walker API (6 unit tests passing, probe CLI running clean).
9+
**H7 refuted:** on actual corpus, the tagged-ancestor approach does not resolve the
10+
classes we care about.
11+
12+
## H7 refuted — the empirical picture
13+
14+
Hypothesis H7 from RE-09 (confidence 0.7): *"Tagless classes (Wall, Floor, Door)
15+
use their parent-in-schema-tree tag on the wire. Walk class `parent` chain in the
16+
schema to find the first ancestor with a tag."*
17+
18+
Running `probe_tagged_ancestors` against Einhoven 2023 + 2024 Core Interior +
19+
2024/2026 sample RFAs:
20+
21+
| File | Classes | Directly tagged | Untagged resolved via ancestor | Unresolvable |
22+
|---|---:|---:|---:|---:|
23+
| Revit_IFC5_Einhoven.rvt (2023) | 405 | 80 | **0** | 325 |
24+
| 2024_Core_Interior.rvt | 395 | 79 | **0** | 316 |
25+
| racbasicsamplefamily-2024.rfa | 395 | 79 | **0** | 316 |
26+
| racbasicsamplefamily-2026.rfa | 349 | 60 | **0** | 289 |
27+
28+
Zero untagged classes resolve to a tagged ancestor — across 4 corpus files.
29+
30+
### Two reasons why
31+
32+
1. **The class-name literals we were looking for do not exist in the schema.**
33+
`Wall`, `Floor`, `Door`, `Window`, `Level`, `Grid`, `Column`, `Beam`, `Roof`,
34+
`Ceiling`, `FamilyInstance`, `Room`, `HostObject` — every one of these returns
35+
"not in schema" from both real .rvt files. Only `HostObjAttr` (tag 0x006b)
36+
and `Element` (tagless, parent-less) are present under the names we used.
37+
2. **The parser isn't extracting `parent` links for most untagged classes.** Of
38+
the 325 untagged classes on Einhoven, every one has `parent.is_none()` —
39+
the probe logged 0 entries in the hop-count distribution, meaning there
40+
was nothing to walk.
41+
42+
Possible parent-link shortfall causes:
43+
- The schema parser in `formats.rs` only records `parent` when a tagged class
44+
is followed by a `[u16 pad=0][u16 parent_name_len][parent_name]` block AND
45+
the `[u16 flag][u32 fc][u32 fc_dup]` preamble validates. That guard fires
46+
for tagged classes; tagless ones fall through the `raw_tag & 0x8000`
47+
branch entirely and never reach parent detection.
48+
- Tagless classes may have their parent stored in a different record format
49+
that the current parser doesn't recognize. Or they may have no parent in
50+
the on-disk schema at all (mixins / embedded primitives like `ElementId`,
51+
`Identifier`, `AppInfo` genuinely have no parent — they're fundamental
52+
types).
53+
- The 325 "untagged" classes include `ACDPtrWrapper`, `A3PartySECImage`,
54+
`ImportVocabulary`, `ADTGridTextLocation` — these look like utility /
55+
mixin / interop types that are embedded in other records as value
56+
wrappers, not top-level serializable entities. They were never candidates
57+
for a Wall-to-HostObjAttr chain.
58+
59+
### Implication for H7 itself
60+
61+
The premise of H7 was that `Wall` lives in the schema under its generic name
62+
with a parent chain pointing to some tagged class. The data says the schema
63+
uses concrete subtype names (`ArcWall` 0x0191, `VWall` 0x0192, `WallCGDriver`
64+
0x0197, etc.) directly, not a `Wall` abstract parent with `ArcWall` as a
65+
subtype. On the wire, a partition chunk representing an arc wall carries
66+
tag 0x0191 — no ancestor walk needed; we just look up 0x0191 directly.
67+
68+
**New posture (H7'):** Classes are carried on the wire by their own concrete
69+
tag. There is no need to walk ancestry to find the tag for `Wall` because
70+
`Wall` is not a thing on the wire — `ArcWall`, `VWall`, etc. are.
71+
72+
This simplifies the tag-scan approach: the target is the 80 tagged classes
73+
themselves. RE-11's original scan (which already included every tagged class,
74+
not just interesting ones) is the correct methodology — the ancestor step
75+
adds no value.
76+
77+
## Verified facts (new)
78+
79+
- **F16** — On 4 corpus files (Einhoven 2023, 2024 Core Interior, 2024 RFA,
80+
2026 RFA) the schema parser extracts `parent` for essentially zero
81+
untagged classes. Either parent data isn't present in the stream for
82+
them, or our parser isn't picking it up. Either way, the ancestor-walk
83+
approach does not materially expand the searchable tag set.
84+
- **F17** — Class-name literals `Wall`, `Floor`, `Door`, `Window`, `Stair`,
85+
`Column`, `Beam`, `Roof`, `Ceiling`, `Level`, `Grid`, `FamilyInstance`,
86+
`Room` are **not in the schema** under those exact names in any corpus
87+
file. Only `HostObjAttr` (direct tag 0x006b) and `Element` (untagged,
88+
parent-less) exist under the names we expected. Source: probe output.
89+
- **F18** — Across all 4 corpus files, `HostObjAttr` is the only "generic
90+
host container" tag, and it sits alone at the top of concrete-element
91+
hierarchies without a visible subtype chain in the parsed schema.
92+
93+
## Code shipped
94+
95+
- `SchemaTable::tagged_ancestor(&self, class_name: &str) -> Option<(&str, u16)>`
96+
- `SchemaTable::tagged_ancestor_map(&self) -> BTreeMap<String, (String, u16)>`
97+
- 6 unit tests covering: direct-tag, walked chain, no-tag-in-chain, cycle
98+
guard, unknown class, aggregate map
99+
- `examples/probe_tagged_ancestors.rs` — 180-line CLI that produces the
100+
empirical table above
101+
- This synthesis doc
102+
103+
The API is *correct* — the unit tests prove the walker works on synthesized
104+
schema data. It just doesn't *help* against real corpus because the parent
105+
data isn't there to walk. Keeping the API because:
106+
107+
1. Callers asking "does class X have a tag either directly or via ancestor?"
108+
get the right answer on both synthesized + real data.
109+
2. If the schema parser improves to extract more parent links (likely
110+
follow-up work), this API's results will silently improve.
111+
3. The `tagged_ancestor_map()` doubles as a sanity check for the schema
112+
parser — it shows what chain coverage we have.
113+
114+
## Decisions
115+
116+
**D6** — Keep the API. Mark the hypothesis that motivated it (H7) as refuted
117+
on this corpus. Move RE-11 execution back to its original plan: scan partition
118+
chunks for the 80 concrete tagged classes directly, not for their abstract
119+
ancestors.
120+
121+
**D7** — Open follow-up: investigate why 325 untagged classes have no `parent`.
122+
Is the schema genuinely silent on their ancestry, or does the parser skip a
123+
branch that contains the parent data? Lower priority than finishing RE-11/15
124+
because H7's refutation means ancestors are not load-bearing for element
125+
location.
126+
127+
**D8** — Do not add a more complex "mixin / aggregation / component graph"
128+
walker until we have evidence that elements carry mixin-tags on the wire.
129+
The 80-tag set is the right place to start; we can extend later if scanning
130+
reveals tags outside that set showing up in the right positions.
131+
132+
## Open questions
133+
134+
**Q8** — Of the 80 tagged classes, which ones are concrete "element"
135+
instances (wall-like, floor-like, door-like) vs which are internal
136+
machinery (transaction headers, index records, version stamps)? The list
137+
from RE-09 (tag 0x0191 ArcWall, 0x0192 VWall, 0x0197 WallCGDriver, etc.)
138+
is a start but we need to enumerate all 80 and decide which subset RE-11
139+
should treat as "interesting."
140+
141+
**Q9** — For Revit 2024 where u0 in partition chunks was quasi-monotonic
142+
(1810, 64397, 81127, 81549, …), does that quasi-monotonic value correspond
143+
to one of the 80 tagged classes' tag values? RE-19 correlated u0 with
144+
ElemTable IDs (negative). A fresh correlation with the 80-tag set is
145+
worth a pass.
146+
147+
## Recommended next steps
148+
149+
1. **RE-11 proper** — scan partition chunks for all 80 tagged classes in
150+
their u16 LE value form. Use `tagged_ancestor_map()` output to
151+
additionally resolve any hits back to class names. If a chunk
152+
consistently begins with a specific tag from the 80, that's the
153+
element envelope marker.
154+
2. **RE-11.5** — run the same scan with tag positions offset by 0, 2, 4,
155+
8 bytes (in case there's a length prefix or a fixed preamble before
156+
the tag). The RE-09 chunk-header probe was negative on 16-byte
157+
hypothesis but a variable-length preamble is still possible.
158+
3. **Parent-link parser fix (defer)** — if and only if RE-11 returns
159+
negative, revisit whether the schema parser is dropping parent data.
160+
161+
Coverage of current schema data as documented by this probe:
162+
163+
```
164+
Revit_IFC5_Einhoven.rvt 80/405 classes resolvable (19.8%)
165+
2024_Core_Interior.rvt 79/395 classes resolvable (20.0%)
166+
racbasicsamplefamily-2024 79/395 classes resolvable (20.0%)
167+
racbasicsamplefamily-2026 60/349 classes resolvable (17.2%)
168+
```
169+
170+
That ~20% is the 80 tagged classes. No ancestor walk expands it. This
171+
is the ceiling until the parser catches more parent data — if it exists.

0 commit comments

Comments
 (0)