when using tools model is unable to recognize a parameter which is an object

[kristoph]

I am able to define tool and have the model call tools that have simple parameters like strings but when I specify an object the response for the whole parameter becomes 'unknown' This is the full request ( which is part of test harness ). I believe I am following the notebook by @koverholt @holtskinner though that does not actually include an object ( would be helpful ).

{
  "generationConfig": {
    "maxOutputTokens": 64,
    "temperature": 0
  },
  "systemInstruction": {
    "role": "user",
    "parts": [
      {
        "text": "\n            You are integrated into a test platform. It is very important that you answer \n            succinctly and always provide the text in the requested format without any additional \n            text. \n          "
      }
    ]
  },
  "contents": [
    {
      "role": "user",
      "parts": [
        {
          "text": "I am in Seattle, WA\n"
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "Got it! Let me know if you need any local insights or information related to Seattle!\n"
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "text": "What is the current weather?\n"
        }
      ]
    }
  ],
  "tools": [
    {
      "function_declarations": [
        {
          "name": "get_weather",
          "description": "The get_weather tool will return the current weather in a given locality.",
          "parameters": {
            "type": "OBJECT",
            "properties": {
              "location": {
                "type": "OBJECT",
                "description": "The locality for which the weather will be returned",
                "properties": {
                  "city": {
                    "type": "STRING",
                    "description": "The city or town for which the current weather should be returned."
                  },
                  "state": {
                    "type": "STRING",
                    "description": "The state or province for which the current weather should be returned. If this is not provided the largest or most prominent city with the given name, in the given country or in the worldi, will be assumed."
                  },
                  "country": {
                    "type": "STRING",
                    "description": "The country for which the given weather should be returned. If this is not provided the largest or most prominent city with the given name will be returned."
                  }
                },
                "required": [
                  "city"
                ]
              }
            },
            "required": [
              "location"
            ]
          }
        }
      ]
    }
  ]
}

[kristoph]

This is what I get as a response body:

{
  "candidates": [
    {
      "content": {
        "parts": [
          {
            "functionCall": {
              "name": "get_weather",
              "args": {
                "location": "unknown"
              }
            }
          }
        ],
        "role": "model"
      },
      "finishReason": "STOP",
      "avgLogprobs": -2.3844768293201922e-08
    }
  ],
  "usageMetadata": {
    "promptTokenCount": 297,
    "candidatesTokenCount": 5,
    "totalTokenCount": 302
  },
  "modelVersion": "gemini-1.5-pro-002"
}

[koverholt]

Thanks for letting us know. I'm also seeing a similar type of failure in the final example shown in the function calling data structures sample notebook. I opened a bug at https://issuetracker.google.com/issues/376170319, so you can track updates and information there.

[kristoph]

Thank you for taking a look. I just ran the same request agains 1.5 pro and I received the same error so if this code worked historically I think it's an API rather than model issue.

{
  "generationConfig": {
    "maxOutputTokens": 64,
    "temperature": 0
  },
  "systemInstruction": {
    "role": "user",
    "parts": [
      {
        "text": "\n            You are integrated into a test platform. It is very important that you answer \n            succinctly and always provide the text in the requested format without any additional \n            text. \n          "
      }
    ]
  },
  "contents": [
    {
      "role": "user",
      "parts": [
        {
          "text": "I am in Seattle, WA\n"
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "Got it! Let me know if you need any local insights or information related to Seattle!\n"
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "text": "What is the current weather?\n"
        }
      ]
    }
  ],
  "tools": [
    {
      "function_declarations": [
        {
          "name": "get_weather",
          "description": "The get_weather tool returns the current weather in given locality.",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "object",
                "description": "The locality for which the weather will be returned",
                "properties": {
                  "city": {
                    "type": "string",
                    "description": "The city or town for which the current weather should be returned."
                  },
                  "state": {
                    "type": "string",
                    "description": "The state or province for which the current weather should be returned. If this is not provided the largest or most prominent city with the given name, in the given country or in the worldi, will be assumed."
                  },
                  "country": {
                    "type": "string",
                    "description": "The country for which the given weather should be returned. If this is not provided the largest or most prominent city with the given name will be returned."
                  }
                },
                "required": [
                  "city"
                ]
              }
            },
            "required": [
              "location"
            ]
          }
        }
      ]
    }
  ]
}

Produces the response:

{
  "candidates": [
    {
      "content": {
        "parts": [
          {
            "functionCall": {
              "name": "get_weather",
              "args": {
                "location": "unknown"
              }
            }
          }
        ],
        "role": "model"
      },
      "finishReason": "STOP"
    }
  ],
  "usageMetadata": {
    "promptTokenCount": 268,
    "candidatesTokenCount": 5,
    "totalTokenCount": 273
  },
  "modelVersion": "gemini-1.5-pro"
}

[kristoph]

Sorry to spam here but I've done some additional tests and it seems like Gemini 1.5 Flash correctly handles the object while, Gemini Flash 1.5 002, Gemini 1.5 Pro and 1.5 Pro 002 do not provide a correct response and just return 'unknown'.

Reliably repeatable with the above request at T0

[koverholt]

Thanks for the additional info. I recommend posting additional info and context to the bug report https://issuetracker.google.com/issues/376170319 so that the engineering team can see the info there. In my testing, Gemini 1.0 Pro 002 worked as well, in addition to Gemini 1.5 Flash like you mentioned. And keep in mind that gemini-1.5-pro will alias to the latest model version, so you might also be seeing different transient behavior between calls.