Getting exception with MultiModalWebsurfer : Unknown tool 'WebSurfer_click'. #5451
Description
What happened?
I have a multi-modal websurfer in my autogen 0.4 team declared like so:
websurfer = MultimodalWebSurfer(
"WebSurfer",
model_client=model_client_gpt_4o_mini,
)
While it appears to be working fine most of the time, sometimes, I see an exception logged such as the one below:
---------- MagenticOneOrchestrator ----------
Please attempt to search for ticker symbol 'PFFA' again to gather its relevant data including name, NAV, market price, and dividend history.
---------- WebSurfer ----------
Web surfing error:
Traceback (most recent call last):
File "c:\PortableApps\Miniconda\envs\autogen\Lib\site-packages\autogen_ext\agents\web_surfer\_multimodal_web_surfer.py", line 408, in on_messages_stream
content = await self._generate_reply(cancellation_token=cancellation_token)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\PortableApps\Miniconda\envs\autogen\Lib\site-packages\autogen_ext\agents\web_surfer\_multimodal_web_surfer.py", line 546, in _generate_reply
return await self._execute_tool(message, rects, tool_names, cancellation_token=cancellation_token)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\PortableApps\Miniconda\envs\autogen\Lib\site-packages\autogen_ext\agents\web_surfer\_multimodal_web_surfer.py", line 691, in _execute_tool
raise ValueError(f"Unknown tool '{name}'. Please choose from:\n\n{tool_names}")
ValueError: Unknown tool 'WebSurfer_click'. Please choose from:
visit_url
web_search
history_back
click
input_text
answer_question
summarize_page
sleep
hover
page_down
What did you expect to happen?
websurfer should not throw exceptions
How can we reproduce it (as minimally and precisely as possible)?
from autogen_agentchat.agents import AssistantAgent
from autogen_agentchat.base import TaskResult
from autogen_agentchat.conditions import ExternalTermination, TextMentionTermination
from autogen_agentchat.teams import RoundRobinGroupChat,MagenticOneGroupChat
from autogen_agentchat.ui import Console
from autogen_core import CancellationToken
from autogen_ext.models.openai import AzureOpenAIChatCompletionClient
from autogen_ext.agents.web_surfer import MultimodalWebSurfer
from autogen_ext.agents.file_surfer import FileSurfer
from autogen_ext.agents.magentic_one import MagenticOneCoderAgent
from autogen_ext.code_executors.local import LocalCommandLineCodeExecutor
from autogen_ext.tools.code_execution import PythonCodeExecutionTool
from autogen_core.code_executor import CodeBlock
from autogen_ext.code_executors.local import LocalCommandLineCodeExecutor
import asyncio
from pathlib import Path
model_client_gpt_4o_mini = AzureOpenAIChatCompletionClient(
azure_deployment="gpt-4o-mini",
azure_endpoint="yyy",
model="gpt-4o-mini",
api_version="2024-08-01-preview",
api_key="xxx",
)
# Create the primary agent.
primary_agent = AssistantAgent(
"primary",
model_client=model_client_gpt_4o_mini,
system_message=""" You are a helpful assistant.""", tools=[pythontool], reflect_on_tool_use=True
)
# Create the critic agent.
critic_agent = AssistantAgent(
"critic",
model_client=model_client_gpt_4o_mini,
system_message="Provide constructive feedback. Respond with 'APPROVE' to when your feedbacks are addressed.",
)
websurfer = MultimodalWebSurfer(
"WebSurfer",
model_client=model_client_gpt_4o_mini,
)
prompt = """From the given list of stock and etf ticker symbols, perform a detailed analysis and create a report in markdown format containing following columns:
1. Symbol
2. Name
3. % Discount or Premium to the NAV
4. [True/False] If there is a history of sustained dividend payments, over 10 years.
5. [True/False] If the dividend payments have been increasing over the last 10 years.
6. [True/False] If the dividend payment is sufficient to cover the dividend payments and is not leading to a decrease in the NAV.
7. % Yield on current market price.
8. % Yield on NAV.
9. Frequency of dividend payments [Monthly, Quarterly, Semi-Annually, Annually etc.]
List of Ticker Symbols: PBDC,TLTW,ARCC,ASGI,PEO,PFFA,STWD,BST,PTY,DSL,DSU,OBDC,RLTY,BXSL,FSCO,ARDC,IIF,SPYI,PFN,XDTE,WDI,QQQI,EIC,XFLT,PDI,GOF,IWMI,FEPI,FDUS,SLRC,RC,RIV,YYY,BRW,BIGZ,BMEZ,OXLC,BSTZ,EXG,ETW,RITM,ASG
"""
# Create a team with the primary and critic agents.
team = MagenticOneGroupChat([primary_agent,critic_agent,websurfer], model_client=model_client_gpt_4o_mini)
async def main():
await Console(team.run_stream(task=prompt))
asyncio.run(main())
AutoGen version
0.4
Which package was this bug in
Extensions
Model used
No response
Python version
No response
Operating system
No response
Any additional info you think would be helpful for fixing this bug
No response