Feature/compaction truncation

[janspoerer]

No description provided.

[janspoerer]

Old PR: https://github.com/evalstate/fast-agent/pull/252/files

[evalstate]

Hi @janspoerer - this one is still in draft; i was about to do some tidying up in this area, but will hold off for the moment to make any potential merge less painful. what do we need to do to undraft this?

[janspoerer]

Hi @evalstate,
Even though the truncation works, I would like to make some further changes to the provider classes. I would like to implement these changes, and then the PR can be undrafted.

[janspoerer]

Hi @evalstate, I just realized you also wanted to make sure to make merging less painful. Missed that earlier, sorry!

Please don't worry about conflicts from your other changes. I would be happy with merging any conflicts.

[storlien]

Hey, @janspoerer

I got this error when trying to test your feature:

num_tokens += len(encoding.encode(message.first_text())) ^^^^^^^^^^^^^^^^^^ AttributeError: 'dict' object has no attribute 'first_text'

Full stack:

[mcp_agent.llm.context_truncation] Model gpt-4.1-mini-2025-04-14 not found. Using cl100k_base tokenizer.
  Finished       | Altinity          / Elapsed Time 00:01:32

Usage Summary (Cumulative)
Agent               Input    Output     Total  Turns  Tools  Context%  Model                    
Altinity_Exe...     2,989        63     3,052      1      0      0.3%  gpt-4.1-mini-2025-04-14  

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/carl.edward.storlien/Documents/Altinn/altinity/fastagent/altinity.py", line 83, in <module>
    asyncio.run(main())
    ~~~~~~~~~~~^^^^^^^^
  File "/Users/carl.edward.storlien/.local/share/uv/python/cpython-3.13.5-macos-aarch64-none/lib/python3.13/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ~~~~~~~~~~^^^^^^
  File "/Users/carl.edward.storlien/.local/share/uv/python/cpython-3.13.5-macos-aarch64-none/lib/python3.13/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/Users/carl.edward.storlien/.local/share/uv/python/cpython-3.13.5-macos-aarch64-none/lib/python3.13/asyncio/base_events.py", line 725, in run_until_complete
    return future.result()
           ~~~~~~~~~~~~~^^
  File "/Users/carl.edward.storlien/Documents/Altinn/jan-repo/fast-agent/src/mcp_agent/core/direct_decorators.py", line 125, in wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/carl.edward.storlien/Documents/Altinn/jan-repo/fast-agent/src/mcp_agent/core/direct_decorators.py", line 125, in wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/carl.edward.storlien/Documents/Altinn/jan-repo/fast-agent/src/mcp_agent/core/direct_decorators.py", line 125, in wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  [Previous line repeated 1 more time]
  File "/Users/carl.edward.storlien/Documents/Altinn/altinity/fastagent/altinity.py", line 80, in main
    await agent.interactive()
  File "/Users/carl.edward.storlien/Documents/Altinn/jan-repo/fast-agent/src/mcp_agent/core/agent_app.py", line 295, in interactive
    return await prompt.prompt_loop(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
    )
    ^
  File "/Users/carl.edward.storlien/Documents/Altinn/jan-repo/fast-agent/src/mcp_agent/core/interactive_prompt.py", line 210, in prompt_loop
    result = await send_func(user_input, agent)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/carl.edward.storlien/Documents/Altinn/jan-repo/fast-agent/src/mcp_agent/core/agent_app.py", line 279, in send_wrapper
    result = await self.send(message, agent_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/carl.edward.storlien/Documents/Altinn/jan-repo/fast-agent/src/mcp_agent/core/agent_app.py", line 95, in send
    return await self._agent(agent_name).send(message)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/carl.edward.storlien/Documents/Altinn/jan-repo/fast-agent/src/mcp_agent/agents/base_agent.py", line 226, in send
    response = await self.generate([prompt], None)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/carl.edward.storlien/Documents/Altinn/jan-repo/fast-agent/src/mcp_agent/agents/base_agent.py", line 597, in generate
    return await self._llm.generate(multipart_messages, request_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/carl.edward.storlien/Documents/Altinn/jan-repo/fast-agent/src/mcp_agent/llm/augmented_llm.py", line 233, in generate
    assistant_response: PromptMessageMultipart = await self._apply_prompt_provider_specific(
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        multipart_messages, request_params
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/carl.edward.storlien/Documents/Altinn/jan-repo/fast-agent/src/mcp_agent/llm/providers/augmented_llm_openai.py", line 548, in _apply_prompt_provider_specific
    ] = await self._openai_completion(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
    )
    ^
  File "/Users/carl.edward.storlien/Documents/Altinn/jan-repo/fast-agent/src/mcp_agent/llm/providers/augmented_llm_openai.py", line 346, in _openai_completion
    if self.context_truncation.needs_truncation(
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self.history,
        ^^^^^^^^^^^^^
    ...<2 lines>...
        system_prompt,
        ^^^^^^^^^^^^^^
    ):
    ^
  File "/Users/carl.edward.storlien/Documents/Altinn/jan-repo/fast-agent/src/mcp_agent/llm/context_truncation.py", line 87, in needs_truncation
    current_tokens = self._estimate_tokens(memory.get(), model, system_prompt)
  File "/Users/carl.edward.storlien/Documents/Altinn/jan-repo/fast-agent/src/mcp_agent/llm/context_truncation.py", line 70, in _estimate_tokens
    num_tokens += len(encoding.encode(message.first_text()))
                                      ^^^^^^^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute 'first_text'