Migrate Evolving Agents Toolkit to Latest BeeAI Framework

[matiasmolinas]

The Evolving Agents Toolkit (EAT) currently utilizes beeai-framework==0.1.4. This version is significantly outdated compared to the latest releases of the BeeAI framework (e.g., v0.1.23 or newer). This migration aims to update EAT to the latest stable version of beeai-framework to leverage new features, bug fixes, performance improvements, and ensure better long-term maintainability and compatibility.

This is a significant undertaking due to the potential for breaking API changes in BeeAI and EAT's deeply integrated custom architecture (MongoDB backend, SmartLibrary, SmartAgentBus, SmartMemory, etc.).

2. Motivation & Benefits

Migrating to the latest BeeAI framework will provide EAT with:

• Access to Latest LLM Integrations & Features: Broader LLM provider support (via LiteLLM), advanced LLM capabilities (e.g., new modalities, tool_choice).
• Improved Core Agent Capabilities: Enhancements to BeeAI's ReActAgent (used by EAT's SystemAgent, ArchitectZero, etc.) and access to new agent types (e.g., ToolCallingAgent).
• Bug Fixes, Security, and Performance Improvements: Benefit from numerous fixes and optimizations in BeeAI.
• Enhanced BeeAI Tooling: Access to new built-in tools (SandboxTool, CodeInterpreterTool) and improvements to existing ones.
• Modernized Framework Internals: Stricter type safety, improved event system (Emitter, Run.on()), and potentially better agent/tool templating mechanisms.
• Potential for EAT Code Simplification: Evaluate if newer BeeAI features (e.g., caching) can replace or simplify parts of EAT's custom code, particularly the LLMCache.
• Stay Aligned with Dependency Evolution: Reduce technical debt and make it easier to get community support.
• Enhanced Interoperability: Potential to leverage newer BeeAI communication protocols (e.g., ACP).

3. Proposed Plan & Tasks

This migration will be approached in phases:

Phase 1: Preparation & Investigation (Target BeeAI Version: TBD - e.g., 0.1.23+)

• Determine Target BeeAI Version: Identify the latest stable and suitable version of beeai-framework.
• Detailed Changelog Review: Analyze BeeAI changelogs from v0.1.4 to the target version to identify all breaking changes, new features, and deprecations.
• Setup Migration Branch: Create a dedicated Git branch for this migration (e.g., feat/beeai-latest-migration).
• Update requirements.txt: Update beeai-framework to the target version. Resolve any immediate dependency conflicts. Test basic installation and import.
• Initial Impact Assessment: Identify immediate breaking changes by attempting to run a minimal EAT setup (e.g., instantiating LLMService or a basic ReActAgent). Document major API signature changes, import path errors, etc.

Phase 2: Core Component Migration

• LLMService (evolving_agents.core.llm_service):
  • Adapt to new ChatModel, EmbeddingModel interfaces in BeeAI (likely via LiteLLMChatModel and potentially new embedding abstractions in BeeAI).
  • Update provider-specific model instantiations (OpenAI, Ollama).
  • Critical Decision: Evaluate BeeAI's internal caching (if any for ChatModel/Tool in the target version) against EAT's custom MongoDB LLMCache.
    • If BeeAI's cache is suitable and MongoDB-compatible (or pluggable): Consider adopting it and potentially deprecating parts of EAT's LLMCache.
    • If not: Ensure EAT's LLMCache continues to function correctly and, if necessary, disable/override BeeAI's internal caching to prevent conflicts.
  • Refactor generate(), embed(), embed_batch(), generate_applicability_text() methods.
• BeeAI Provider (evolving_agents.providers.beeai_provider):
  • Update agent creation logic (create_agent_instance) to align with new BeeAI AgentFactory or agent instantiation patterns.
  • Update agent execution logic (execute_agent_instance) to use new BeeAI agent run methods or invocation patterns.
  • Adapt tool handling/registration if BeeAI's tool management has changed.
• EAT Agents (ReActAgent-based):
  • For SystemAgent, ArchitectZero, MemoryManagerAgent, IntentReviewAgent:
    • Update instantiation based on changes to beeai_framework.agents.react.ReActAgent.
    • Review and update prompt engineering if ReActAgent's internal prompting has changed.
    • Verify tool definition, registration, and execution flow within these agents.
    • Adapt to any changes in how AgentMeta, Memory (e.g., TokenMemory), or RunContext are handled.
• EAT Tools (BeeAI-based):
  • Ensure all EAT tools inheriting from/using beeai_framework.tools.tool.Tool are compatible with the latest interface.
  • Update evolving_agents.tools.tool_factory.DynamicTool and ToolFactory if base BeeAI tool structures have changed.
• EAT Adapters (OpenAI specific):
  • Review OpenAIToolAdapter, OpenAIGuardrailsAdapter, OpenAITracingAdapter.
  • Determine if newer BeeAI versions offer better native OpenAI Agents SDK integration that could simplify or replace these adapters.

Phase 3: Integration & System-Level Adjustments

• Dependency Container: Ensure all new/updated services (LLMService, potentially new BeeAI components) are correctly registered and resolved by DependencyContainer.
• EAT Agent/Tool Factories: Refactor evolving_agents.agents.agent_factory.AgentFactory and evolving_agents.tools.tool_factory.ToolFactory based on changes in BeeAIProvider and tool/agent interfaces.
• EAT Templates: Update evolving_agents/templates/beeai_agent_template.txt and evolving_agents/templates/beeai_tool_template.txt to generate code compatible with the new BeeAI version.
• Configuration: Review and update evolving_agents.config.py and .env.example for any new or changed configuration variables required by the new BeeAI framework or EAT's adaptations.
• Import Paths: Systematically check and update all import statements from beeai_framework throughout the EAT codebase (ref: BeeAI v0.1.12 breaking changes).

Phase 4: Testing

• Unit Tests: Write/update unit tests for all significantly refactored components (LLMService, providers, core agents).
• Integration Tests:
  • Test SmartLibrary component creation and retrieval (especially for BeeAI-based components).
  • Test SmartAgentBus registration and capability requests involving BeeAI agents.
  • Test SystemAgent's core orchestration loop with migrated components.
  • Test SmartMemory interactions (ContextBuilderTool, ExperienceRecorderTool via MemoryManagerAgent).
  • Test IntentReview system for BeeAI-related plans.
• End-to-End Testing (Examples):
  • Systematically test all examples in the examples/ directory.
  • Pay special attention to examples/invoice_processing/architect_zero_comprehensive_demo.py as a full system test.
  • Verify Smart Memory examples like examples/smart_memory_enhancement_real_evolution_demo.py (and scripts/test_smart_memory.py).
• Performance Testing: Basic performance comparison before/after to identify any significant regressions.

Phase 5: Documentation & Finalization

• Update EAT Documentation:
  • README.md: Update BeeAI version, any relevant architectural diagrams or descriptions.
  • docs/ARCHITECTURE.md: Reflect any changes due to BeeAI updates.
  • docs/MONGO-SETUP.md, docs/TUTORIAL.md: Ensure they remain accurate.
• Update Code Comments: Add/update comments explaining new integrations or changes.
• Code Review: Thorough code review of all changes.
• Merge: Merge the migration branch into the main development branch.

4. Acceptance Criteria

• EAT successfully builds and runs using the target version of beeai-framework.
• All core EAT functionalities are verified and working as expected:
  • SystemAgent orchestration.
  • SmartLibrary operations (create, search, evolve BeeAI components).
  • SmartAgentBus operations (register, discover, request for BeeAI agents).
  • SmartMemory operations.
  • IntentReview system.
  • LLMCache (EAT's or BeeAI's, as decided).
• All Python examples in the examples/ directory execute successfully and produce expected outcomes.
• The existing EAT test suite (once fully migrated to MongoDB) passes.
• EAT documentation (README.md, ARCHITECTURE.md, etc.) is updated to reflect the changes.
• No major performance regressions are observed in key EAT operations.
• Code is well-documented and follows EAT's coding standards.

5. Potential Risks & Challenges

• Extensive Refactoring: Significant code changes will be required across multiple EAT modules.
• Unforeseen Breaking Changes: BeeAI changelogs might not capture all subtle breaking changes or behavioral differences.
• Caching Strategy: Deciding on and implementing the correct caching strategy (EAT's custom vs. BeeAI's internal) will be complex.
• Compatibility with EAT's Custom Systems: Ensuring EAT's MongoDB-centric architecture (SmartLibrary, SmartAgentBus, SmartMemory) integrates smoothly with the new BeeAI version.
• Testing Complexity: Thoroughly testing all aspects of the complex EAT system will be time-consuming.
• Resource Allocation: This migration will require significant, dedicated developer time.

6. Relevant Context & Links

• EAT requirements.txt: [link to EAT's requirements.txt on GitHub]
• BeeAI Framework Repository: https://github.com/i-am-bee/beeai-framework
• BeeAI Python Changelog: https://github.com/i-am-bee/beeai-framework/blob/main/python/CHANGELOG.md
• Key EAT Modules for Migration:
  • evolving_agents/core/llm_service.py
  • evolving_agents/providers/beeai_provider.py
  • evolving_agents/agents/ (SystemAgent, ArchitectZero, etc.)
  • evolving_agents/tools/tool_factory.py & evolving_agents/agents/agent_factory.py
  • evolving_agents/templates/

---

This issue will serve as the main tracker for the migration effort. Sub-tasks can be created for specific components or phases as needed.

[matiasmolinas]

Project Upgrade Plan: Migrating evolving-agents to the Latest beeai-framework

This plan outlines the strategy, phases, and specific actions required to upgrade the evolving-agents toolkit (EAT) from beeai-framework v0.1.4 to the latest version. The migration will focus on refactoring deprecated patterns, simplifying the EAT codebase by adopting native beeai features, and enhancing capabilities with new tools and agent types.

1. Executive Summary & Goals

The evolving-agents toolkit is a powerful meta-framework built on an early version of beeai-framework. Since v0.1.4, beeai-framework has introduced significant new features, including a unified EmbeddingModel, advanced ToolCallingAgent and RequirementAgent types, a native caching system, and a sophisticated AgentWorkflow engine.

The primary goals of this migration are to:

1. Modernize the Core: Replace EAT's custom LLM and embedding service abstractions with beeai-framework's native, provider-agnostic ChatModel and EmbeddingModel.
2. Simplify the Codebase: Reduce maintenance overhead by removing redundant custom logic (e.g., caching, workflow processing) that is now standard in the framework.
3. Enhance Agent Capabilities: Leverage new agent types like ToolCallingAgent for more reliable tool use and AgentWorkflow for more robust orchestration.
4. Improve Performance and Reduce Cost: Adopt the native framework caching mechanisms for LLM calls and tool outputs.
5. Future-Proof the Project: Align EAT with the current architecture and future direction of beeai-framework, making subsequent updates easier.

---

2. Pre-Migration Analysis: Key Differences

A review of both projects highlights several key areas for migration:

Feature	evolving-agents (on beeai v0.1.4)	Latest beeai-framework	Migration Action
Embeddings	Custom EmbeddingModel classes in llm_service.py.	Native EmbeddingModel with provider-specific adapters (from_name).	Refactor LLMService to use the native EmbeddingModel.
Agent Types	Primarily uses ReActAgent.	Adds ToolCallingAgent and experimental RequirementAgent.	Upgrade SystemAgent to ToolCallingAgent; pilot RequirementAgent.
Workflows	Custom YAML-based engine (GenerateWorkflowTool).	Native AgentWorkflow class for multi-agent orchestration.	Replace custom engine with AgentWorkflow.
Caching	Custom MongoDB-based LLMCache in llm_service.py.	Native, pluggable Cache system (SlidingCache, etc.) for models and tools.	Migrate from custom cache to native framework cache.
Observability	Custom ComponentExperienceTracker.	Standardized Emitter with Run.on() API and OpenInference integration.	Refactor tracker to subscribe to the native event stream via .on().
Tools	All custom-built Tool subclasses.	Adds powerful built-in tools like OpenAPITool and SandboxTool.	Audit and replace custom tools where applicable.
Dependencies	Pinned to beeai-framework==0.1.4.	Uses latest beeai-framework and its dependencies.	Update requirements.txt and resolve conflicts.

---

3. Detailed Migration Plan (Phased Approach)

This plan is broken into sequential phases to manage complexity.

Phase 0: Preparation & Environment Setup

1. Branching: Create a new Git branch for the migration (e.g., feat/beeai-upgrade).
2. Dependency Upgrade:
   • In requirements.txt, update the beeai-framework line to the latest version (e.g., beeai-framework>=0.1.29).
   • Remove openai-agents if its functionality can be fully replaced by beeai adapters.
   • Run pip install -r requirements.txt in a clean virtual environment. Resolve any immediate dependency conflicts.
3. Initial Test Run:
   • Execute the existing test suite. Expect numerous failures.
   • Use the initial failure report to create a high-level checklist of broken areas (imports, class signatures, etc.).

Phase 1: Foundational Refactoring (Core Services)

This phase focuses on EAT's LLMService (evolving_agents/core/llm_service.py), which is the primary interface to the underlying framework.

1. Refactor Embedding Logic:

   • Action: Remove the custom EmbeddingModel, OpenAIEmbeddingModel, and OllamaEmbeddingModel classes from llm_service.py.
   • Action: In LLMService.__init__, instantiate the embedding model using the native beeai factory method:

     # In evolving_agents/core/llm_service.py
     from beeai_framework.backend import EmbeddingModel
     
     # ...
     embedding_model_name = "openai:text-embedding-3-small" # Or construct from config
     self.embedding_model = EmbeddingModel.from_name(embedding_model_name)

   • Action: Update the LLMService.embed() and embed_batch() methods to call self.embedding_model.create(). Note that create() returns an EmbeddingModelOutput object, so you'll need to extract the embeddings from output.embeddings.

2. Refactor Chat Model Logic:

   • Action: In LLMService.__init__, simplify chat model instantiation:

     # In evolving_agents/core/llm_service.py
     from beeai_framework.backend import ChatModel
     
     # ...
     chat_model_name = "openai:gpt-4o-mini" # Or construct from config
     self.chat_model = ChatModel.from_name(chat_model_name)

   • Action: Update LLMService.generate() to call self.chat_model.create(). The create method returns a ChatModelOutput object; use output.get_text_content() to get the string response.

3. Migrate to Native Caching:

   • Action: Remove the custom LLMCache class and its associated logic from llm_service.py.
   • Action: Instantiate beeai-framework's cache and pass it to the models during their creation in LLMService.__init__. This automatically enables caching for all create calls.

     # In evolving_agents/core/llm_service.py
     from beeai_framework.cache import UnconstrainedCache
     # ...
     
     llm_cache = UnconstrainedCache() # Or SlidingCache, etc.
     self.chat_model = ChatModel.from_name(chat_model_name, cache=llm_cache)
     self.embedding_model = EmbeddingModel.from_name(embedding_model_name, cache=llm_cache)

   • Note: This replaces the MongoDB cache. A data migration script could be written to move old cache entries, but it's likely easier to start with a fresh cache.

Phase 2: Component Migration (Agents & Tools)

1. Update Custom Tool Signatures:

   • Action: Systematically review all custom tools in evolving_agents/tools/.
   • Action: The _run method signature in beeai_framework.tools.tool.Tool now includes a context: RunContext argument. Update all subclasses to match:

     # Before
     async def _run(self, input: MyInput) -> MyOutput:
         # ...
     
     # After
     async def _run(self, input: MyInput, context: RunContext) -> MyOutput:
         # ...

2. Update Custom Agent Definitions:

   • Action: Review agents defined in evolving_agents/agents/ (e.g., ArchitectZeroAgentInitializer, MemoryManagerAgent).
   • Action: The base agent class in beeai now uses generics. Update any custom agent classes to specify their output model. Since most EAT agents are ReActAgents, this change may be handled by the initializer, but any direct subclassing of BaseAgent must be updated.
   • Action: Verify the __init__ signature of ReActAgent and ToolCallingAgent and update the factory methods in AgentFactory and initializers (ArchitectZeroAgentInitializer) accordingly. The way meta and tools are passed might have changed.

3. Refactor beeai_provider.py:

   • This file is the primary bridge. Its create_agent and execute_agent methods need to be updated to align with the latest beeai-framework API for agent instantiation and execution.
   • Action: The agent.run() method now returns a Run object. Execution is triggered by await-ing it. Ensure the provider correctly invokes the agent and handles the response.

Phase 3: New Feature Adoption & Modernization

1. Upgrade SystemAgent to ToolCallingAgent:

   • The SystemAgent is the central orchestrator and is a perfect candidate for the more robust ToolCallingAgent.
   • Action: In evolving_agents/providers/beeai_provider.py (or AgentFactory), add logic to instantiate beeai_framework.agents.tool_calling.ToolCallingAgent.
   • Action: Refactor the SystemAgent's core logic/prompt to leverage tool calling. This will likely simplify its ReAct-style prompt and improve reliability.

2. Replace Custom Workflow Engine with AgentWorkflow:

   • This is a significant but highly beneficial change.
   • Action: Remove GenerateWorkflowTool and ProcessWorkflowTool.
   • Action: Refactor the architect_zero_comprehensive_demo.py example. Instead of generating YAML, ArchitectZero should design a sequence of steps. The main script will then construct a beeai_framework.workflows.agent.AgentWorkflow with these steps.
   • Example Refactoring:

     # In architect_zero_comprehensive_demo.py (or similar orchestration script)
     from beeai_framework.workflows.agent import AgentWorkflow, AgentWorkflowInput
     
     # ... (Create specialized agents for each step)
     workflow = AgentWorkflow(name="Invoice Processing")
     
     workflow.add_agent(name="InvoiceReader", role="Extracts text from invoices.", ...)
     workflow.add_agent(name="DataExtractor", role="Extracts structured data from text.", ...)
     workflow.add_agent(name="Validator", role="Validates the extracted data.", ...)
     
     # Run the entire orchestrated process
     result = await workflow.run(inputs=[
         AgentWorkflowInput(prompt="Read the invoice file.", agent="InvoiceReader"),
         AgentWorkflowInput(prompt="Extract invoice details.", agent="DataExtractor"),
         # ...
     ])

3. Adopt New Built-in Tools:

   • Action: Audit all custom tools in EAT.
   • Action: Identify if any can be replaced. For example, if EAT has a tool for running Python code, it can likely be replaced by the more secure beeai_framework.tools.code.SandboxTool.

4. Modernize Observability:

   • Action: The ComponentExperienceTracker should be refactored. Instead of being called directly, it should subscribe to the beeai event stream.
   • Action: In orchestration scripts, use the .on() method to attach the tracker's logging functions to agent and tool events.

     # Example of hooking the tracker
     experience_tracker = ComponentExperienceTracker(...)
     
     agent = SystemAgent(...)
     await agent.run(prompt="...") \
                .on("tool:start", experience_tracker.on_tool_start) \
                .on("tool:success", experience_tracker.on_tool_success) \
                .on("tool:error", experience_tracker.on_tool_error)

   This decouples the tracking logic from the execution logic and provides much richer, standardized data.

Phase 4: Validation & Documentation

1. Testing:
   • Incrementally fix the test suite after each phase of the migration.
   • Add new tests for features adopted, such as AgentWorkflow execution and ToolCallingAgent behavior.
   • Pay special attention to tests for SmartLibrary and LLMService to validate the new embedding and caching backends.
2. Documentation Update:
   • Action: Update docs/ARCHITECTURE.md to reflect the new architecture. Remove references to the custom LLMCache and workflow engine. Add sections on AgentWorkflow and the native EmbeddingModel.
   • Action: Update docs/TUTORIAL.md and any examples in examples/ to use the new APIs.
   • Action: Update README.md to highlight the new capabilities inherited from the latest beeai-framework.
   • Action: Ensure requirements.txt is finalized.

---

4. Risks and Mitigation

• Risk: Breaking changes in beeai-framework are more extensive than anticipated.
  • Mitigation: The phased approach allows for tackling one area at a time. Referencing the beeai-framework CHANGELOG.md will be critical for identifying breaking changes.
• Risk: Performance degradation due to new abstractions.
  • Mitigation: Benchmark key operations (e.g., semantic search in SmartLibrary, complex agent execution) before and after the migration.
• Risk: The scope of refactoring is very large.
  • Mitigation: Prioritize the phases. Phase 1 is mandatory. Phases 2 and 3 can be broken into smaller, incremental tasks (e.g., migrate one agent at a time, adopt one new feature at a time).

By following this phased plan, the evolving-agents project can be successfully and methodically upgraded, resulting in a more robust, maintainable, and powerful toolkit.