(feat) LLM Mode - verbose mode and token economics

[andersonpem]

Hey guys. This is my very first time opening a PR to Graphify. I'm excited!

I like this project quite a lot. Thank you for the excellent piece of software this project is.

This PR introduces two flags to the Graphify CLI (plus their respective environment variable counterparts)

• The --verbose flag for showing verbose outputs of what Graphify has been working on
• The --tokens flag for showing only token economics (size of the prompt, input, output tokens used)

I have quite big projects. And my AI APIs are kinda restricted in terms of dashboards, so, calculating if it was viable to use the LLM-backed labelling of Graphify was very mysterious to me. I never knew if I was overworking the models or not.

Verbose mode will dump the prompts, the thinking process, the data, everything about a LLM run to the stdout. That is good especially for prompt engineering: refining how graphify deals with data, what can be improved, etc.

Tokens mode will only print out the prompt size, input, output tokens and an estimate of cost (in the case of Claude), so the user can have a north of how much that run costed. The cost data is merely informative, an estimate considering current prices. There are a lot of factors that can change the pricing of LLM usage.

If you have any doubts, or if the flags aren't clear enough, let me know.

Have a nice day.
Andy.

[graphify-labs]

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 3 advisory finding(s) below merit a look before merge.

Formal verification. 1 change(s) tested, no difference found (not proven).

---

Graphify review — findings

This PR adds two opt-in observability modes to the LLM-backed community labeling and dedup paths. - --verbose mode (also GRAPHIFY_LLM_VERBOSE=1) traces each LLM call's prompt, thinking, response, and per-call token counts to stderr, followed by a run-level token total with a cost estimate. It also shapes calls to surface reasoning (extended thinking on claude, stream-json on claude-cli, reasoning_content on OpenAI/Bedrock). - --tokens mode (also GRAPHIFY_LLM_TOKENS=1) prints only per-call token counts plus a run total, without the prompt/thinking/response dumps and without verbose's call-shaping side effects. The changes touch the cluster-only and label CLI subcommands (flag parsing, help text, run-total printing), the llm.py backend layer (new set_llm_verbose/set_llm_tokens toggles, per-backend tracing hooks, cost estimation), CHANGELOG/README docs, a .gitignore entry, and associated tests.

Worth a look

• label_token_usage accumulated non-atomically across concurrent labeling threads — graphify/cli.py:2006 · Escalate · high
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
• Env-enabled token tracing misses the run total — graphify/cli.py:2006 · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
• Module-level LLM verbose/tokens flags mutated without synchronization while ThreadPoolExecutor runs concurrent calls — graphify/llm.py · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 1026 functions depend on the 550 functions this change touches.

Health — this change adds coupling hotspots:

• worse: _call_llm() — 16 callers, 22 callees
• worse: dispatch_command() — 2 callers, 119 callees

Verification — 1026 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

• verification_scope: 967 function(s) in the blast radius were not formally verified this run

Formal verification

Could not verify: Could not verify dispatch\_command.

The verifier did not have enough to check dispatch\_command, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly SystemExit — names the real obstacle, not a sampling gap)

Could not verify: Could not verify \_call\_llm.

The verifier did not have enough to check \_call\_llm, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly TypeError — names the real obstacle, not a sampling gap)

No difference found (not proven): No behavior difference found in \_run\_cli (not a proof).

The verifier ran both versions of \_run\_cli on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.

Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.

Note: An input the sampler did not try could still differ.

· 2 more finding(s) on lines outside this diff (see the check run).

[andersonpem]

Nice catch, bot! I'll work on that later on the day :)

[andersonpem]

From "sight reading", I'd say the one that I can work on is the "Env-enabled token tracing misses the run total". It's more of a cosmetic thing, but I can add it.
The first one, each batch dict is merged in the main thread only.
The third one, the setters run only during startup, before any thread pool exists.

[graphify-labs]

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 1 advisory finding(s) below merit a look before merge.

---

Graphify review — findings

This pull request adds LLM call tracing to the community labeling and dedup tiebreaker paths. It introduces two new modes: --verbose (via the flag on cluster-only/label or GRAPHIFY_LLM_VERBOSE=1), which prints each LLM call's prompt, thinking, response, and per-call token counts to stderr plus a run-level total with a cost estimate, and --tokens (or GRAPHIFY_LLM_TOKENS=1), which prints only token accounting without the full exchange dumps or verbose's call-shaping side effects. The surface area touches CLI flag parsing and dispatch in cli.py, help text in __main__.py, and the LLM backend module (llm.py) — adding verbose/tokens state setters, per-call token tracing, cost estimation, backend detection, and reasoning-content surfacing across the OpenAI-compatible, Bedrock, claude, and claude-cli backends. Documentation (README, CHANGELOG) and a new/expanded test_llm_backends test suite are also updated, along with a .gitignore entry.

Worth a look

• Module-level LLM verbose/tokens state is unguarded global shared across parallel calls — graphify/cli.py:1779 · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 1029 functions depend on the 553 functions this change touches.

Health — this change adds coupling hotspots:

• worse: _call_llm() — 16 callers, 22 callees
• worse: dispatch_command() — 2 callers, 121 callees

Verification — 1029 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

• verification_scope: 970 function(s) in the blast radius were not formally verified this run

· 2 more finding(s) on lines outside this diff (see the check run).

[andersonpem]

Have a look and if you have any doubts let me know