Adk evaluation failing with: "AttributeError: 'float' object has no attribute 'item'"

[iperdomocabrera]

Adk evaluation failing with: "AttributeError: 'float' object has no attribute 'item'"

Steps to reproduce the behavior:

1. Install google adk latest version. I'm using google-adk version 1.1.1, but I've also tried 0.0.4 and 1.0.0 with the same error.
   Install pytest and its dependencies

2. I can reproduce this issue calling adk eval or runinng my test using pytest.

Running adk eval: 
adk eval --config_file_path tests/evaluation/test_config.json --print_detailed_results sub_agents/analyze tests/evaluation/analyze_agent.test.json

Running test using pytest:
pytest tests/evaluation/test_analyze_agent.py -v --capture=no

3. Error:

self = <google.adk.evaluation.response_evaluator.ResponseEvaluator object at 0x00000141C47D2030>
eval_result = EvalResult(summary_metrics={'row_count': 1, 'rouge_1/mean': nan, 'rouge_1/std': np.float64(nan)}, metrics_table=      ...e_1/score
0  {   "audit_transaction": {     "request_docume...  ...          None

[1 rows x 6 columns], metadata=None)

   def _get_score(self, eval_result) -> float:
>     return eval_result.summary_metrics[f"{self._metric_name}/mean"].item()
E     AttributeError: 'float' object has no attribute 'item'

..\.env\Lib\site-packages\google\adk\evaluation\response_evaluator.py:120: AttributeError

Expected behavior
Expect to run adk evaluation on my analyze_agent to evaluate trajectory and response matching without errors.

Desktop (please complete the following information):

• Windows Server 2022
• Python version 3.12.0
• ADK version 1.1.1

Additional context
I'm attempting to run ADK evaluation on my agent but I'm encountering the following error: "AttributeError: 'float' object has no attribute 'item'". Here are the details:

• I'm using google-adk version 1.1.1, but I've also tried 0.0.4 and 1.0.0 with the same error.

• The eval-set I'm using was generated via ADK Web.

• I've executed the tests using both pytest and the adk eval CLI, and both produce the same results.

• The error appears to originate from the response_evaluator, specifically this line

These are the files Im using to run the evaluation including the test output:

test_config.json.txt
TestResults.txt
test_analyze_agent.py.txt
analyze_agent.test.json.txt

Any guidance would be appreciated!

[ankursharmas]

Thank you for bringing this to our attention.

This particular failure happened because you are missing GCP Project configurations. The messages in the attached TestResults.txt have the following lines:

WARNING  vertexai.preview.evaluation._evaluation:_evaluation.py:937 Error encountered for metric rouge_1 at dataset index 0: Error: Unable to find your project.

If you are running this eval using the command line, you need to set your GCP project id in the .env file. Here is some info on how to set that up: https://google.github.io/adk-docs/get-started/quickstart/#set-up-the-model

If you are running this eval using pytest, then follow the set up here: https://github.com/google/adk-python/blob/main/tests/integration/.env.example

[iperdomocabrera]

Hi @ankursharmas , thank you for the quick response and for sharing the documentation on setting up my GCP project.

I followed the setup steps, but I’m still encountering an issue when using the following test_config.json:

{
  "criteria": {
    "tool_trajectory_avg_score": 0.9,
    "response_match_score": 0.9
  }
}

When I run the evaluation using adk eval, I see that it successfully makes the Vertex Gen AI API call, but it fails with the following error:

AttributeError: 'float' object has no attribute 'item'

adk eval --config_file_path tests/evaluation/test_config.json  sub_agents/analyze tests/evaluation/analyze_agent.test.json
Using evaluation criteria: {'tool_trajectory_avg_score': 0.9, 'response_match_score': 0.9}
Running Eval: evalseta62195:case1e7104
Code search tool called with query: Check why nexus configuration might not apply
Computing metrics with a total of 1 Vertex Gen AI Evaluation Service API requests.
100%|███████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  4.78it/s]
1 errors encountered during evaluation. Continue to compute summary metrics for the rest of the dataset.
Error encountered for metric rouge_1 at dataset index 0: Error: 500 Internal error encountered.
Evaluation Took:0.2025716999778524 seconds
Eval failed for `evalseta62195:case1e7104`
Traceback (most recent call last):
  File "C:\Users\Administrator\Documents\triage-ai-project\.env\Lib\site-packages\google\adk\cli\cli_eval.py", line 196, in run_evals
    evaluation_result = metric_evaluator.evaluate_invocations(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Administrator\Documents\triage-ai-project\.env\Lib\site-packages\google\adk\evaluation\response_evaluator.py", line 76, in evaluate_invocations
    score = self._get_score(eval_case_result)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Administrator\Documents\triage-ai-project\.env\Lib\site-packages\google\adk\evaluation\response_evaluator.py", line 120, in _get_score
    return eval_result.summary_metrics[f"{self._metric_name}/mean"].item()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'float' object has no attribute 'item'
Running Eval: evalseta62195:case940152
Code search tool called with query: Check why nexus configuration might not apply
Computing metrics with a total of 1 Vertex Gen AI Evaluation Service API requests.
100%|███████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  4.80it/s]
1 errors encountered during evaluation. Continue to compute summary metrics for the rest of the dataset.
Error encountered for metric rouge_1 at dataset index 0: Error: 500 Internal error encountered.
Evaluation Took:0.21564670000225306 seconds
Eval failed for `evalseta62195:case940152`
Traceback (most recent call last):
  File "C:\Users\Administrator\Documents\triage-ai-project\.env\Lib\site-packages\google\adk\cli\cli_eval.py", line 196, in run_evals
    evaluation_result = metric_evaluator.evaluate_invocations(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Administrator\Documents\triage-ai-project\.env\Lib\site-packages\google\adk\evaluation\response_evaluator.py", line 76, in evaluate_invocations
    score = self._get_score(eval_case_result)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Administrator\Documents\triage-ai-project\.env\Lib\site-packages\google\adk\evaluation\response_evaluator.py", line 120, in _get_score
    return eval_result.summary_metrics[f"{self._metric_name}/mean"].item()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'float' object has no attribute 'item'
*********************************************************************
Eval Run Summary

It appears the error is triggered here:

return eval_result.summary_metrics[f"{self._metric_name}/mean"].item()

If I remove response_match_score from the config and use only:

{
  "criteria": {
    "tool_trajectory_avg_score": 0.9
  }
}

…the evaluation completes without raising an exception, though the test cases still fail (as expected due to tool trajectory mismatch).

adk eval --config_file_path tests/evaluation/test_config.json  sub_agents/analyze tests/evaluation/analyze_agent.test.json                        
Using evaluation criteria: {'tool_trajectory_avg_score': 0.9}
Running Eval: evalseta62195:case1e7104
Code search tool called with query: Check why nexus configuration might not apply
Result: ❌ Failed

Running Eval: evalseta62195:case940152
Code search tool called with query: Check why nexus configuration might not apply
Result: ❌ Failed

*********************************************************************
Eval Run Summary
evalseta62195:
  Tests passed: 0
  Tests failed: 2

Here’s a summary:

• ✅ Vertex AI API call is being made
• ❌ response_match_score causes an internal ADK error
• ✅ Removing it allows the eval to run without crashing

Could you please confirm if this is a known issue, or if there’s something I need to change on my end?

Thanks again for your help,
Indira

[ankursharmas]

For the response_match_score, I don't seem to have a local repro. I do see the following error in the logs you shared:

Error encountered for metric rouge_1 at dataset index 0: Error: 500 Internal error encountered.

I will try to fix the code and have these kind of failures bubble up and fail the eval. But, this won't solve your problem.

Can you try setting a break point here

adk-python/src/google/adk/evaluation/response_evaluator.py

Line 247 in 8d36dbd

return eval_task.evaluate()

and see if there are some logs of interest that get published after the execution (of that line).

[iperdomocabrera]

@ankursharmas , I've added some debug logging as you suggested—please find the logs attached.

To assist further investigation, I modified response_evaluator.py to output additional details related to the perform_eval parameters and evaluation case results. Kindly review the changes specifically at lines 78 and 253.

Appreciate your help!

response_evaluator.py.txt
Logs_06032025.txt

[iperdomocabrera]

Hi @ankursharmas,

I’ve created a small project based on the documentation example, and I was able to reproduce the error. I’m sharing it here in case it helps with your investigation.

Please do let me know if you uncover any findings—I’m keen to incorporate ADK evaluation into my enterprise project.

Find the project attached

travel_concierge.zip

Thank you,
Indira

This is how I am running adk eval for this project.

PS C:\Users\Administrator\Documents\travel_concierge> adk eval --config_file_path multi_tool_agent/eval/test_config.json --print_detailed_results multi_tool_agent multi_tool_agent/eval/weather.evalset.json Using evaluation criteria: {'tool_trajectory_avg_score': 0.8, 'response_match_score': 0.65} Running Eval: evalsetfba483:case5c239a Warning: there are non-text parts in the response: ['function_call'], returning concatenated text result from text parts. Check the full candidates.content.parts accessor to get the full model response. Computing metrics with a total of 1 Vertex Gen AI Evaluation Service API requests. 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 5.71it/s] 1 errors encountered during evaluation. Continue to compute summary metrics for the rest of the dataset. Error encountered for metric rouge_1 at dataset index 0: Error: 500 Internal error encountered. Evaluation Took:0.18599139992147684 seconds Eval failed for evalsetfba483:case5c239a`
Traceback (most recent call last):
File "C:\Users\Administrator\Documents\travel_concierge.env\Lib\site-packages\google\adk\cli\cli_eval.py", line 196, in run_evals
evaluation_result = metric_evaluator.evaluate_invocations(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\Documents\travel_concierge.env\Lib\site-packages\google\adk\evaluation\response_evaluator.py", line 78, in evaluate_invocations
score = self._get_score(eval_case_result)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\Documents\travel_concierge.env\Lib\site-packages\google\adk\evaluation\response_evaluator.py", line 122, in _get_score
return eval_result.summary_metrics[f"{self._metric_name}/mean"].item()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'float' object has no attribute 'item'

---

Eval Run Summary
PS C:\Users\Administrator\Documents\travel_concierge> `

[jcfiorenzano]

@ankursharmas any update on this issue?