Integration package refinements #266

rodrigobr-msft · 2025-12-02T00:03:16Z

No description provided.

Copilot

Pull request overview

This PR introduces several refinements to the integration testing package, including support for invoke responses, renaming of the Selector class to ModelSelector, new utility functions, and a comprehensive benchmarking CLI tool. However, the changes contain several critical bugs that must be addressed before merging.

Key Changes

Added support for invoke activity responses in data-driven tests with new InvokeResponse handling
Renamed Selector to ModelSelector for better clarity across the codebase
Introduced resolve_env utility for loading environment configuration
Added comprehensive benchmark CLI tool with thread and coroutine execution modes

Reviewed changes

Copilot reviewed 14 out of 35 changed files in this pull request and generated 15 comments.

Show a summary per file

File	Description
test_ddt.py	Updated mock assertions to include `prefix=""` parameter
test_data_driven_test.py	Added test cases for expect_replies delivery mode and invoke response assertions
test_agent_client.py	Added test for new send_invoke_activity method
test_selector.py	Renamed Selector to ModelSelector throughout test cases
test_model_assertion.py	Renamed Selector to ModelSelector in assertion tests
resolve_env.py	New utility for resolving .env files from paths or directories
utils/init.py	Exported new resolve_env utility
data_driven_test.py	Added invoke response support with refactored input/assertion handling
response_client.py	Improved streaming activity handling with new _handle_streamed_activity implementation
agent_client.py	Added send_invoke_activity method and refactored sleep handling
auto_client.py	Removed commented-out placeholder code
test_auth.py	Added placeholder test file
auth_sample.py	New authentication sample implementation
benchmark/*	Complete benchmarking tool infrastructure with executors, configuration, and documentation
init.py	Added cli module export

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-02T00:09:10Z

...icrosoft-agents-testing/microsoft_agents/testing/integration/data_driven/data_driven_test.py

+        model_list: list
+
+        if step.get("assertion_type") == "activity":
+            model_list = responses
+
+        if step.get("assertion_type") == "invoke_response":
+            model_list = invoke_responses


The variable model_list is declared but may not be initialized before use. If assertion_type is neither "activity" nor "invoke_response", or if it's not set at all, the variable will be uninitialized when passed to assertion.check() on line 135, causing a UnboundLocalError.

Consider:

Initializing model_list with a default value (e.g., model_list: list = [])

Using elif on line 132 instead of a second if

Adding an else clause to handle unexpected assertion types

Suggested change

model_list: list

if step.get("assertion_type") == "activity":

model_list = responses

if step.get("assertion_type") == "invoke_response":

model_list = invoke_responses

model_list: list = []

assertion_type = step.get("assertion_type")

if assertion_type == "activity":

model_list = responses

elif assertion_type == "invoke_response":

model_list = invoke_responses

else:

raise ValueError(f"Unsupported assertion_type: {assertion_type!r} in step: {step}")

Copilot · 2025-12-02T00:09:11Z

...icrosoft-agents-testing/microsoft_agents/testing/integration/data_driven/data_driven_test.py

+                            if "invokeResponse" in selector:
+                                selector["model"] = selector["invokeResponse"]
+                            if "invoke_response" in selector:


Similar to the assertion_type logic issue, lines 85-90 have multiple if statements that could execute sequentially, potentially overwriting the selector["model"] value when multiple keys exist.

Use elif statements to ensure only one selector model is set:

if "activity" in selector: selector["model"] = selector["activity"] elif "invokeResponse" in selector: selector["model"] = selector["invokeResponse"] elif "invoke_response" in selector: selector["model"] = selector["invoke_response"]

Suggested change

if "invokeResponse" in selector:

selector["model"] = selector["invokeResponse"]

if "invoke_response" in selector:

elif "invokeResponse" in selector:

selector["model"] = selector["invokeResponse"]

elif "invoke_response" in selector:

Copilot · 2025-12-02T00:09:11Z

...microsoft-agents-testing/microsoft_agents/testing/integration/core/client/response_client.py

+            if (
+                stream_info.stream_sequence <= 0
+                and activity.type == ActivityTypes.typing
+            ):
+                raise ValueError(
+                    "Streamed activity's stream sequence should be a positive number."
+                )


The logic condition on lines 110-113 is confusing. It checks if stream_sequence <= 0 AND activity.type == typing, but the error message suggests it should be checking if stream_sequence <= 0 AND activity.type != typing.

The condition appears backwards - it raises an error when the sequence is <= 0 AND it's a typing activity, but the error message says streaming activities should have positive sequence numbers. This suggests the condition should be checking for streaming activities (non-typing messages), not typing activities.

Consider clarifying the logic or the error message to match the intended behavior.

Copilot · 2025-12-02T00:09:11Z

...microsoft-agents-testing/microsoft_agents/testing/integration/core/client/response_client.py

    async def _handle_streamed_activity(
-        self, activity: Activity, *args, **kwargs
+        self, activity: Activity, stream_info: Entity, cid: str
    ) -> bool:


The function _handle_streamed_activity is defined with parameters activity, stream_info, and cid, but is called with only activity. This will cause a TypeError at runtime.

The function should either:

Be called with all required parameters extracted from the activity

Have its signature updated to match the call site (taking only activity and extracting stream_info and cid internally)

Copilot · 2025-12-02T00:09:12Z

dev/microsoft-agents-testing/microsoft_agents/testing/cli/samples/auth_sample.py

+
+
+def create_auth_route(auth_handler_id: str, agent: AgentApplication):
+    def dynamic_function(context: TurnContext, state: TurnState):


The function dynamic_function is defined as a regular function but uses await on line 12, making it an async function. This will cause a SyntaxError.

Change line 11 to:

async def dynamic_function(context: TurnContext, state: TurnState):

Suggested change

def dynamic_function(context: TurnContext, state: TurnState):

async def dynamic_function(context: TurnContext, state: TurnState):

Copilot · 2025-12-02T00:09:13Z

dev/microsoft-agents-testing/microsoft_agents/testing/integration/client/agent_client.py

+    ) -> InvokeResponse:
+
+        if not activity.type == ActivityTypes.invoke:
+            raise ValueError("Activity type must be 'invoke' for send_invoke method.")


The error message on line 181 refers to "send_invoke method" but the actual method name is "send_invoke_activity". This inconsistency could confuse users debugging errors.

Update to:

raise ValueError("Activity type must be 'invoke' for send_invoke_activity method.")

Suggested change

raise ValueError("Activity type must be 'invoke' for send_invoke method.")

raise ValueError("Activity type must be 'invoke' for send_invoke_activity method.")

Copilot · 2025-12-02T00:09:14Z

dev/microsoft-agents-testing/microsoft_agents/testing/integration/client/agent_client.py

+        except ValidationError:
+            raise ValueError(
+                "Error when sending invoke activity: InvokeResponse not returned or invalid format."
+            )


The exception handling on lines 185-191 catches ValidationError but doesn't preserve the original error information. This makes debugging difficult when the InvokeResponse validation fails.

Consider preserving the original exception:

except ValidationError as e: raise ValueError( f"Error when sending invoke activity: InvokeResponse not returned or invalid format. Details: {e}" ) from e

Suggested change

except ValidationError:

raise ValueError(

"Error when sending invoke activity: InvokeResponse not returned or invalid format."

)

except ValidationError as e:

raise ValueError(

f"Error when sending invoke activity: InvokeResponse not returned or invalid format. Details: {e}"

) from e

Copilot · 2025-12-02T00:09:14Z

dev/microsoft-agents-testing/microsoft_agents/testing/cli/samples/auth_sample.py

+    @classmethod
+    async def get_config(cls) -> dict:
+        """Retrieve the configuration for the sample."""
+        load_dotenv("./src/tests/.env")
+        return dict(os.environ)


Missing async keyword in function definition. The function contains await on line 12 but is not declared as async, which will cause a SyntaxError.

Change the function signature to:

@classmethod async def get_config(cls) -> dict:

Copilot · 2025-12-02T00:09:14Z

dev/microsoft-agents-testing/microsoft_agents/testing/cli/samples/auth_sample.py

+        for auth_handler in app.config.authentication_handlers:
+            app.message(
+                auth_handler.name.lower(),
+                create_auth_route(auth_handler.name),


Call to function create_auth_route with too few arguments; should be no fewer than 2.

Suggested change

create_auth_route(auth_handler.name),

create_auth_route(auth_handler.name, app),

Copilot · 2025-12-02T00:09:14Z

dev/microsoft-agents-testing/microsoft_agents/testing/cli/test_auth.py

@@ -0,0 +1,5 @@
+import cli


Import of 'cli' is not used.

Suggested change

import cli

Copilot

Pull request overview

Copilot reviewed 27 out of 50 changed files in this pull request and generated 24 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-02T20:01:59Z

dev/microsoft-agents-testing/microsoft_agents/testing/cli/test_auth/command.py

+    environment.setup()
+
+    sample = AuthSample()
+    sample.run(resolve_env)


The sample.run(resolve_env) call appears incorrect. The run method likely expects a configuration dictionary, not the resolve_env function itself. This should probably be sample.run(resolve_env(config_path)) or similar.

Copilot · 2025-12-02T20:02:00Z

dev/microsoft-agents-testing/microsoft_agents/testing/integration/client/agent_client.py

+    ) -> InvokeResponse:
+
+        if not activity.type == ActivityTypes.invoke:
+            raise ValueError("Activity type must be 'invoke' for send_invoke method.")


The error message refers to "send_invoke method" but the actual method name is "send_invoke_activity". The error message should be updated to match the method name for clarity.

Copilot · 2025-12-02T20:02:00Z

dev/microsoft-agents-testing/microsoft_agents/testing/cli/samples/auth_sample.py

+class QuickstartSample(Sample):
+    """A quickstart sample implementation."""


The class is named QuickstartSample but should be AuthSample based on the file name and context. Additionally, the docstring "A quickstart sample implementation." doesn't accurately describe this authentication-specific sample.

Copilot · 2025-12-02T20:02:00Z

dev/microsoft-agents-testing/microsoft_agents/testing/cli/test_auth.py

@@ -0,0 +1,5 @@
+import cli


The import statement import cli will fail because cli is not a valid module. This appears to be an incomplete or incorrect import that should either import from .cli or be removed entirely.

Copilot · 2025-12-02T20:02:01Z

...icrosoft-agents-testing/microsoft_agents/testing/integration/data_driven/data_driven_test.py

+        assertion = self._load_assertion(step)
+        responses.extend(await response_client.pop())
+
+        model_list: list


The type annotation model_list: list is too generic. It should be model_list: list[Activity] | list[InvokeResponse] or use a Union type to properly reflect what's being stored. This improves type safety and code clarity.

Copilot · 2025-12-02T20:02:05Z

...microsoft-agents-testing/microsoft_agents/testing/integration/core/client/response_client.py

+            await self._add(activity)
+            if activity.type != ActivityTypes.typing:
+                await asyncio.sleep(0.05)  # Simulate processing delay
+


Trailing whitespace on this line should be removed.

Copilot · 2025-12-02T20:02:05Z

dev/microsoft-agents-testing/microsoft_agents/testing/cli/samples/auth_sample.py

+        for auth_handler in app.config.authentication_handlers:
+            app.message(
+                auth_handler.name.lower(),
+                create_auth_route(auth_handler.name),


Call to function create_auth_route with too few arguments; should be no fewer than 2.

Copilot · 2025-12-02T20:02:06Z

dev/microsoft-agents-testing/microsoft_agents/testing/cli/test_auth/auth_sample.py

+        for auth_handler in app.config.authentication_handlers:
+            app.message(
+                auth_handler.name.lower(),
+                create_auth_route(auth_handler.name),


Call to function create_auth_route with too few arguments; should be no fewer than 2.

Copilot · 2025-12-02T20:02:06Z

dev/microsoft-agents-testing/microsoft_agents/testing/integration/client/response_client.py

-    Activity,
-    ActivityTypes,
-)
+from microsoft_agents.activity import Activity, ActivityTypes, Entity


Import of 'Entity' is not used.

Copilot · 2025-12-02T20:02:06Z

dev/microsoft-agents-testing/microsoft_agents/testing/cli/test_auth.py

@@ -0,0 +1,5 @@
+import cli


Import of 'cli' is not used.

Copilot

Pull request overview

Copilot reviewed 37 out of 60 changed files in this pull request and generated 17 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-05T17:09:58Z

...icrosoft-agents-testing/microsoft_agents/testing/integration/data_driven/data_driven_test.py

+        model_list: list
+
+        if step.get("assertion_type") == "activity":
+            model_list = responses
+
+        if step.get("assertion_type") == "invoke_response":
+            model_list = invoke_responses


The variable model_list may be uninitialized if assertion_type is neither "activity" nor "invoke_response". This will cause an UnboundLocalError when model_list is used on line 146. Consider adding an else clause or initializing model_list before the if statements, or using elif for the second condition.

Copilot · 2025-12-05T17:09:58Z

dev/microsoft-agents-testing/microsoft_agents/testing/cli/test_auth/command.py

+from .auth_sample import AuthSample
+
+from microsoft_agents.testing.utils import resolve_env
+from microsoft_agents.testing.core import AiohttpEnvironment


Invalid import path. The import from microsoft_agents.testing.core import AiohttpEnvironment appears to be incorrect. Based on the codebase structure, the correct import should be from microsoft_agents.testing.integration.core import AiohttpEnvironment.

Copilot · 2025-12-05T17:09:59Z

dev/integration/tests/test_copilot_client.py

+def mock_mcs_handler(activity: Activity) -> Callable[[Request], Awaitable[Response]]:
+    """Creates a mock handler for MCS endpoint returning the given activity."""
+
+    async def handler(request: Request) -> Response:
+        activity_data = activity.model_dump_json(exclude_unset=True)
+        return Response(body=activity_data)
+
+    return handler
+
+


Duplicate function definition. The function mock_mcs_handler is defined twice with different signatures (lines 19-26 and 28-45), which will cause the second definition to override the first. The second definition is the one actually used on line 60. Consider renaming one of them or removing the unused first definition.

Suggested change

def mock_mcs_handler(activity: Activity) -> Callable[[Request], Awaitable[Response]]:

"""Creates a mock handler for MCS endpoint returning the given activity."""

async def handler(request: Request) -> Response:

activity_data = activity.model_dump_json(exclude_unset=True)

return Response(body=activity_data)

return handler

Copilot · 2025-12-05T17:09:59Z

dev/microsoft-agents-testing/microsoft_agents/testing/integration/client/response_client.py

+            await self._add(activity)
+            if activity.type != ActivityTypes.typing:
+                await asyncio.sleep(0.05)  # Simulate processing delay
+


Trailing whitespace detected. There are trailing spaces on this line which should be removed to follow Python style guidelines (PEP 8).

Copilot · 2025-12-05T17:10:00Z

...icrosoft-agents-testing/microsoft_agents/testing/integration/aiohttp/aiohttp_async_runner.py

+                await self._server_task
+            except asyncio.CancelledError:
+                pass
+


Trailing whitespace detected. There are trailing spaces on this line which should be removed to follow Python style guidelines (PEP 8).

Copilot · 2025-12-05T17:10:03Z

dev/microsoft-agents-testing/microsoft_agents/testing/assertions/check_model.py

            return True, None

        elif isinstance(baseline, list):
+


Trailing whitespace detected. There are trailing spaces on this line which should be removed to follow Python style guidelines (PEP 8).

Copilot · 2025-12-05T17:10:04Z

...icrosoft-agents-testing/microsoft_agents/testing/integration/data_driven/data_driven_test.py

        """Compile the data driven test to ensure all steps are valid."""
-        for step in self._test:
+        for i, step in enumerate(self._test):
+


Trailing whitespace detected. There are trailing spaces on this line which should be removed to follow Python style guidelines (PEP 8).

Copilot · 2025-12-05T17:10:04Z

...icrosoft-agents-testing/microsoft_agents/testing/integration/data_driven/data_driven_test.py

+                    else:
+                        step["assertion"] = {}
+                        step["assertion_type"] = "activity"
+


Trailing whitespace detected. There are trailing spaces on this line which should be removed to follow Python style guidelines (PEP 8).

Copilot · 2025-12-05T17:10:04Z

dev/microsoft-agents-testing/tests/integration/data_driven/test_data_driven_test.py

        assert call_args.locale == "en-US"
        assert call_args.channel_id == "test-channel"

+    async def test_with_invoke_response_assertion(self):


Missing @pytest.mark.asyncio decorator. This async test method lacks the required pytest asyncio decorator. All async test methods in this file use @pytest.mark.asyncio, but this one is missing it on line 718, which will cause the test to fail when executed.

Copilot · 2025-12-05T17:10:04Z

...icrosoft-agents-testing/microsoft_agents/testing/integration/aiohttp/aiohttp_async_runner.py

+            self._server_task.cancel()
+            try:
+                await self._server_task
+            except asyncio.CancelledError:


'except' clause does nothing but pass and there is no explanatory comment.

Copilot

Pull request overview

Copilot reviewed 93 out of 207 changed files in this pull request and generated 9 comments.

Comments suppressed due to low confidence (1)

dev/microsoft-agents-testing/microsoft_agents/testing/integration/client/response_client.py:13

Import of 'Entity' is not used.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-10T18:38:26Z

dev/tests/samples/quickstart_sample.py

+    @classmethod
+    async def get_config(cls) -> dict:
+        """Retrieve the configuration for the sample."""
+        load_dotenv("./src/tests/.env")


The file path "./src/tests/.env" appears to be incorrect. The directory structure suggests the .env file should be in the root or tests directory, not "./src/tests/.env".

Copilot · 2025-12-10T18:38:26Z

dev/tests/samples/quickstart_sample.py

+
+        @app.error
+        async def on_error(context: TurnContext, error: Exception):
+            # This check writes out errors to console log .vs. app insights.


There's a spelling/grammar issue: the abbreviation ".vs." should be "vs." or "versus". Additionally, this comment has inconsistent style with the next line using proper NOTE format.

Copilot · 2025-12-10T18:38:26Z

dev/tests/integration/test_copilot_client.py

+# import pytest
+
+# from typing import Awaitable, Callable, Iterable
+
+# from aiohttp.web import Request, Response, Application, StreamResponse
+
+# from microsoft_agents.activity import Activity
+
+# from microsoft_agents.copilotstudio.client import (
+#     CopilotClient,
+#     ConnectionSettings,
+#     PowerPlatformEnvironment,
+# )
+
+# from microsoft_agents.testing.integration.core import AiohttpRunner
+
+
+# def mock_mcs_handler(activity: Activity) -> Callable[[Request], Awaitable[Response]]:
+#     """Creates a mock handler for MCS endpoint returning the given activity."""
+
+#     async def handler(request: Request) -> Response:
+#         activity_data = activity.model_dump_json(exclude_unset=True)
+#         return Response(body=activity_data)
+
+#     return handler
+
+
+# def mock_mcs_handler(
+#     activities: Iterable[Activity],
+# ) -> Callable[[Request], Awaitable[StreamResponse]]:
+#     """Creates a mock handler for MCS endpoint returning SSE-formatted activity."""
+
+#     async def handler(request: Request) -> StreamResponse:
+#         response = StreamResponse(status=200)
+#         response.headers["Content-Type"] = "text/event-stream"
+#         response.headers["x-ms-conversationid"] = "test-conv-id"
+#         # response.headers['Content-Length'] = str(len(activity_data))
+#         await response.prepare(request)
+
+#         # Proper SSE format
+#         for activity in activities:
+#             activity_data = activity.model_dump_json(exclude_unset=True)
+#             await response.write(b"event: activity\n")
+#             await response.write(f"data: {activity_data}\n\n".encode("utf-8"))
+
+#         await response.write_eof()
+#         return response
+
+#     return handler
+
+
+# def mock_mcs_endpoint(
+#     mocker, activities: Iterable[Activity], path: str, port: int
+# ) -> AiohttpRunner:
+#     """Mock MCS responses for testing."""
+
+#     PowerPlatformEnvironment.get_copilot_studio_connection_url = mocker.MagicMock(
+#         return_value=f"http://localhost:{port}{path}"
+#     )
+
+#     app = Application()
+#     app.router.add_post(path, mock_mcs_handler(activities))
+
+#     return AiohttpRunner(app, port=port)
+
+
+# @pytest.mark.asyncio
+# async def test_start_conversation_and_ask_question_large_message(mocker):
+
+#     activity = Activity(
+#         type="message", text="*" * 1_000_000, conversation={"id": "conv-id"}
+#     )
+
+#     runner = mock_mcs_endpoint(mocker, [activity], "/mcs-endpoint", port=8081)
+
+#     async with runner:
+#         settings = ConnectionSettings("environment-id", "agent-id")
+#         client = CopilotClient(settings=settings, token="test-token")
+
+#         with pytest.raises(Exception, match="Chunk too big"):
+#             async for conv_activity in client.start_conversation():
+#                 assert conv_activity.type == "message"
+
+#         # with pytest.raises(Exception, match="Chunk too big"):
+#         #     async for question_activity in client.ask_question("Hello!", "conv-id"):
+#         #         assert question_activity.type == "message"
+
+
+# def activity_generator(activity: Activity, n: int) -> Iterable[Activity]:
+#     for i in range(n):
+#         yield activity
+
+
+# @pytest.mark.asyncio
+# async def test_start_conversation_many(mocker):
+
+#     activity = Activity(type="message", conversation={"id": "conv-id"})
+#     activities = activity_generator(activity, 100_000)
+
+#     runner = mock_mcs_endpoint(mocker, activities, "/mcs-endpoint", port=8081)
+
+#     async with runner:
+#         settings = ConnectionSettings("environment-id", "agent-id")
+#         client = CopilotClient(settings=settings, token="test-token")
+
+#         for i in range(100):
+#             # try:
+#             async for conv_activity in client.start_conversation():
+#                 assert conv_activity.type == "message"
+#             # except Exception as e:
+#             #     assert str(e) == "Chunk too big"
+
+#         # with pytest.raises(Exception, match="Chunk too big"):
+#         #     async for question_activity in client.ask_question("Hello!", "conv-id"):
+#         #         assert question_activity.type == "message"


The entire test file is commented out. Commented-out code should either be removed or uncommented if it's intended to be used.

Copilot · 2025-12-10T18:38:27Z

dev/microsoft-agents-testing/microsoft_agents/testing/assertions/types/_readonly.py

+        raise AttributeError(f"Cannot set item '{key}' on {type(self).__name__}")
+
+    def __delitem__(self, key):
+        raise AttributeError(f"Cannot delete item '{key}' on {type(self).__name__}")


This method always raises AttributeError - should raise a LookupError (KeyError or IndexError) instead.

Suggested change

raise AttributeError(f"Cannot set item '{key}' on {type(self).__name__}")

def __delitem__(self, key):

raise AttributeError(f"Cannot delete item '{key}' on {type(self).__name__}")

raise KeyError(f"Cannot set item '{key}' on {type(self).__name__}")

def __delitem__(self, key):

raise KeyError(f"Cannot delete item '{key}' on {type(self).__name__}")

Copilot · 2025-12-10T18:38:27Z

dev/microsoft-agents-testing/microsoft_agents/testing/assertions/assertion_context.py

+
+from .types.safe_object import (
+    SafeObject,
+    resolve,


Import of 'resolve' is not used.

Suggested change

resolve,

Copilot · 2025-12-10T18:38:28Z

dev/microsoft-agents-testing/microsoft_agents/testing/assertions/assertions.py

+from __future__ import annotations

-from typing import Any
+import inspect


Import of 'inspect' is not used.

Suggested change

import inspect

Copilot · 2025-12-10T18:38:28Z

dev/microsoft-agents-testing/microsoft_agents/testing/assertions/assertions.py


-from typing import Any
+import inspect
+from pathlib import PurePath


Import of 'PurePath' is not used.

Suggested change

from pathlib import PurePath

Copilot · 2025-12-10T18:38:28Z

dev/microsoft-agents-testing/microsoft_agents/testing/assertions/assertions.py

+from typing import Any, Callable

-from microsoft_agents.activity import AgentsModel
+from .types import SafeObject, DynamicObject


Import of 'DynamicObject' is not used.

Suggested change

from .types import SafeObject, DynamicObject

from .types import SafeObject

Copilot · 2025-12-10T18:38:28Z

dev/microsoft-agents-testing/microsoft_agents/testing/assertions/assertions.py

+    @staticmethod
+    def check_verbose(actual: Any, baseline: Any) -> tuple[bool, str]:
+        actual = SafeObject(actual)
+        context = AssertionContext(actual, baseline)


Call to AssertionContext.init with too few arguments; should be no fewer than 4.

Suggested change

context = AssertionContext(actual, baseline)

context = AssertionContext(actual, baseline, None, None)

Copilot

Pull request overview

Copilot reviewed 94 out of 208 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (1)

dev/microsoft-agents-testing/microsoft_agents/testing/integration/client/response_client.py:13

Import of 'Entity' is not used.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-10T19:51:56Z

dev/microsoft-agents-testing/microsoft_agents/testing/assertions/types/_readonly.py

+        raise AttributeError(f"Cannot set item '{key}' on {type(self).__name__}")
+
+    def __delitem__(self, key):
+        """Prevent deleting items on the readonly object."""
+        raise AttributeError(f"Cannot delete item '{key}' on {type(self).__name__}")


This method always raises AttributeError - should raise a LookupError (KeyError or IndexError) instead.

Suggested change

raise AttributeError(f"Cannot set item '{key}' on {type(self).__name__}")

def __delitem__(self, key):

"""Prevent deleting items on the readonly object."""

raise AttributeError(f"Cannot delete item '{key}' on {type(self).__name__}")

raise KeyError(f"Cannot set item '{key}' on {type(self).__name__}")

def __delitem__(self, key):

"""Prevent deleting items on the readonly object."""

raise KeyError(f"Cannot delete item '{key}' on {type(self).__name__}")

Copilot · 2025-12-10T19:51:56Z

dev/microsoft-agents-testing/tests/assertions/test_assertion_context.py

@@ -0,0 +1,477 @@
+import pytest
+from unittest.mock import Mock


Import of 'Mock' is not used.

Copilot · 2025-12-10T19:51:56Z

dev/microsoft-agents-testing/tests/assertions/test_assertion_context.py

+from unittest.mock import Mock
+
+from microsoft_agents.testing.assertions.assertion_context import AssertionContext
+from microsoft_agents.testing.assertions.types import SafeObject, DynamicObject, Unset


Import of 'Unset' is not used.

Copilot · 2025-12-10T19:51:57Z

dev/microsoft-agents-testing/tests/assertions/test_assertion_context.py

+
+from microsoft_agents.testing.assertions.assertion_context import AssertionContext
+from microsoft_agents.testing.assertions.types import SafeObject, DynamicObject, Unset
+from microsoft_agents.testing.assertions.types.safe_object import resolve, parent


Import of 'parent' is not used.

Copilot

Pull request overview

Copilot reviewed 94 out of 208 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

dev/microsoft-agents-testing/pytest.ini:1

The warning filter value should be 'pytest.PytestUnraisableExceptionWarning' but this is not a standard pytest warning category. The correct category should be 'PytestUnraisableExceptionWarning' without the 'pytest.' prefix, or use the full module path.
dev/microsoft-agents-testing/microsoft_agents/testing/utils/misc.py:1
The function uses the deprecated pdb.set_trace() method. Python 3.7+ includes a built-in breakpoint() function that should be used instead. Consider removing this wrapper function entirely and using breakpoint() directly in code.
dev/microsoft-agents-testing/microsoft_agents/testing/cli/cli_config.py:1
The add_trailing_slash function is called in the setter but also in the initialization default value. Consider calling it only in the setter to avoid duplication and ensure consistency.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…into users/robrandao/integration-refinements

Copilot

Pull request overview

Copilot reviewed 95 out of 208 changed files in this pull request and generated 6 comments.

Comments suppressed due to low confidence (2)

dev/tests/E2E/basic_agent/directline/SendActivity_SendsHi5_Returns5HiActivities.yaml:17

Line 17 contains an invalid YAML syntax. The value should be - type: skip (with proper indentation and "type:" key), not just |- skip. This will cause YAML parsing errors when the test is executed.
dev/microsoft-agents-testing/microsoft_agents/testing/integration/client/response_client.py:13
Import of 'Entity' is not used.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-16T22:04:44Z

dev/tests/integration/test_copilot_client.py

+def mock_mcs_handler(activity: Activity) -> Callable[[Request], Awaitable[Response]]:
+    """Creates a mock handler for MCS endpoint returning the given activity."""
+
+    async def handler(request: Request) -> Response:
+        activity_data = activity.model_dump_json(exclude_unset=True)
+        return Response(body=activity_data)
+
+    return handler
+
+
+def mock_mcs_handler(


There are two functions with the same name mock_mcs_handler defined at lines 18 and 28. The second definition (lines 28-49) completely shadows the first one (lines 18-25), making the first function unreachable. This appears to be leftover code from refactoring. Remove the first function definition or rename one of them if both are needed.

Copilot · 2025-12-16T22:04:44Z

dev/microsoft-agents-testing/microsoft_agents/testing/assertions/assertions.py

+from typing import Any, Callable

-from microsoft_agents.activity import AgentsModel
+from .types import SafeObject, DynamicObject, resolve, parent


Import of 'DynamicObject' is not used.
Import of 'parent' is not used.

Suggested change

from .types import SafeObject, DynamicObject, resolve, parent

from .types import SafeObject, resolve

Copilot · 2025-12-16T22:04:45Z

dev/microsoft-agents-testing/tests/assertions/test_assertions.py

@@ -0,0 +1,570 @@
+import pytest
+from unittest.mock import Mock


Import of 'Mock' is not used.

Copilot · 2025-12-16T22:04:45Z

dev/microsoft-agents-testing/tests/assertions/test_assertions.py

+
+from microsoft_agents.testing.assertions.assertions import Assertions
+from microsoft_agents.testing.assertions.assertion_context import AssertionContext
+from microsoft_agents.testing.assertions.types import SafeObject, DynamicObject, Unset


Import of 'DynamicObject' is not used.
Import of 'Unset' is not used.

Copilot · 2025-12-16T22:04:45Z

dev/microsoft-agents-testing/tests/assertions/test_model_query.py

@@ -0,0 +1,570 @@
+import pytest
+from unittest.mock import Mock


Import of 'Mock' is not used.

Copilot · 2025-12-16T22:04:45Z

dev/microsoft-agents-testing/tests/assertions/test_model_query.py

+
+from microsoft_agents.testing.assertions.assertions import Assertions
+from microsoft_agents.testing.assertions.assertion_context import AssertionContext
+from microsoft_agents.testing.assertions.types import SafeObject, DynamicObject, Unset


Import of 'DynamicObject' is not used.
Import of 'Unset' is not used.

Copilot

Pull request overview

Copilot reviewed 96 out of 209 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (2)

dev/tests/E2E/basic_agent/directline/SendActivity_SendsHi5_Returns5HiActivities.yaml:17

Typo in YAML - "- skip" should be "- type: skip" to match the expected YAML structure format.
dev/microsoft-agents-testing/microsoft_agents/testing/integration/client/response_client.py:13
Import of 'Entity' is not used.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-16T22:28:47Z

dev/microsoft-agents-testing/microsoft_agents/testing/assertions/types/_readonly.py

+
+    def __delitem__(self, key):
+        """Prevent deleting items on the readonly object."""
+        raise AttributeError(f"Cannot delete item '{key}' on {type(self).__name__}")


This method always raises AttributeError - should raise a LookupError (KeyError or IndexError) instead.

Suggested change

raise AttributeError(f"Cannot delete item '{key}' on {type(self).__name__}")

raise KeyError(f"Cannot delete item '{key}' on {type(self).__name__}")

Copilot · 2025-12-16T22:28:47Z

dev/tests/integration/test_copilot_client.py

+
+from typing import Awaitable, Callable, Iterable
+
+from aiohttp.web import Request, Response, Application, StreamResponse


Import of 'Response' is not used.

Copilot

Pull request overview

Copilot reviewed 100 out of 214 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (1)

dev/microsoft-agents-testing/microsoft_agents/testing/integration/client/response_client.py:13

Import of 'Entity' is not used.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-17T15:02:44Z

dev/microsoft-agents-testing/microsoft_agents/testing/assertions/types/_readonly.py

+        """Prevent setting items on the readonly object."""
+        raise AttributeError(f"Cannot set item '{key}' on {type(self).__name__}")
+
+    def __delitem__(self, key):


This method always raises AttributeError - should raise a LookupError (KeyError or IndexError) instead.

Copilot · 2025-12-17T15:02:44Z

dev/microsoft-agents-testing/microsoft_agents/testing/integration/data_driven_test.py

@@ -0,0 +1,4 @@
+class DataDrivenTest
+
+    def __init__(self, test_seq: list[DDTComponent]):


Syntax Error (in Python 3).

Copilot · 2025-12-17T15:02:44Z

dev/microsoft-agents-testing/microsoft_agents/testing/integration/ddt/ddt_component.py

+
+class Receive(DDTComponent):
+
+    def __init__(self, )


Syntax Error (in Python 3).

Copilot · 2025-12-17T15:02:45Z

dev/microsoft-agents-testing/microsoft_agents/testing/assertions/assertion_context.py

+from .types import (
+    SafeObject,
+    resolve,
+    parent
+)


Import of 'resolve' is not used.

Copilot · 2025-12-17T15:02:45Z

dev/microsoft-agents-testing/microsoft_agents/testing/assertions/model_query.py

@@ -0,0 +1,73 @@
+from typing import Protocol, TypeVar, overload, Iterable, Callable


Import of 'Protocol' is not used.

rodrigobr-msft added 10 commits December 1, 2025 12:36

Prelim work to handle sending invokes

3cb2022

Reorganizing AgentClient interface

6e25a90

Removing AutoClient file

fd25c32

Adding basic stream activity handling for ResponseClient

464ecc6

CLI starter

d1ec76e

Adding input response handling for DDT

6c0a56e

Adding resolve_env utility function

6a57b4c

Fixing test cases

943fd71

Handling of InvokeResponse assertions in DataDrivenTest class

efc7da2

Expect replies tests for DDT

5c77c65

Copilot AI review requested due to automatic review settings December 2, 2025 00:03

Copilot started reviewing on behalf of rodrigobr-msft December 2, 2025 00:03 View session

Copilot finished reviewing on behalf of rodrigobr-msft December 2, 2025 00:07

Copilot AI reviewed Dec 2, 2025

View reviewed changes

rodrigobr-msft added 2 commits December 2, 2025 09:53

Fixing integration tests

ecb2c6e

Fixing integration tests with streaming

3048e19

Copilot AI review requested due to automatic review settings December 2, 2025 19:54

Copilot started reviewing on behalf of rodrigobr-msft December 2, 2025 19:54 View session

Copilot finished reviewing on behalf of rodrigobr-msft December 2, 2025 19:58

Copilot AI reviewed Dec 2, 2025

View reviewed changes

rodrigobr-msft added 2 commits December 2, 2025 14:59

More adjustments to test cases

930bf09

Adding CopilotClient test case using integration package features

603050d

Copilot AI review requested due to automatic review settings December 5, 2025 17:03

Copilot started reviewing on behalf of rodrigobr-msft December 5, 2025 17:04 View session

Copilot finished reviewing on behalf of rodrigobr-msft December 5, 2025 17:07

Reformatting

415264b

Copilot AI reviewed Dec 5, 2025

View reviewed changes

rodrigobr-msft added 2 commits December 5, 2025 14:29

CLI grouping

cadb843

Moving ResponseClient to use AiohttpRunner instead of async version

902d412

Copilot AI review requested due to automatic review settings December 5, 2025 22:54

Robust query function handling

995d44e

Copilot AI review requested due to automatic review settings December 10, 2025 18:33

Copilot started reviewing on behalf of rodrigobr-msft December 10, 2025 18:33 View session

Copilot AI reviewed Dec 10, 2025

View reviewed changes

rodrigobr-msft added 2 commits December 10, 2025 10:50

Adding documentation

f077099

Adding AssertionContext tests

209c56d

Copilot AI review requested due to automatic review settings December 10, 2025 19:46

Copilot started reviewing on behalf of rodrigobr-msft December 10, 2025 19:47 View session

Copilot AI reviewed Dec 10, 2025

View reviewed changes

rodrigobr-msft added 2 commits December 10, 2025 11:55

Assertions tests

31d5f2f

Adding basic fixtures

73205e0

Copilot AI review requested due to automatic review settings December 10, 2025 19:58

Copilot started reviewing on behalf of rodrigobr-msft December 10, 2025 19:59 View session

Copilot AI reviewed Dec 10, 2025

View reviewed changes

rodrigobr-msft added 2 commits December 10, 2025 14:06

Stable version

42846ed

Updating old tests

ba50824

Copilot AI review requested due to automatic review settings December 16, 2025 21:59

Merge branch 'main' of https://github.com/microsoft/Agents-for-python …

8dc344c

…into users/robrandao/integration-refinements

Copilot started reviewing on behalf of rodrigobr-msft December 16, 2025 21:59 View session

Copilot AI reviewed Dec 16, 2025

View reviewed changes

rodrigobr-msft added 2 commits December 16, 2025 14:04

Adding CopilotClient integration test

ba92b0b

Quickstart E2E test port

66b818c

Copilot AI review requested due to automatic review settings December 16, 2025 22:22

Copilot started reviewing on behalf of rodrigobr-msft December 16, 2025 22:23 View session

Copilot AI reviewed Dec 16, 2025

View reviewed changes

rodrigobr-msft added 2 commits December 16, 2025 14:39

Fixed wrong model query in test case

284ab17

Added FutureVars

a21a856

Copilot AI review requested due to automatic review settings December 17, 2025 15:00

Copilot started reviewing on behalf of rodrigobr-msft December 17, 2025 15:01 View session

Copilot AI reviewed Dec 17, 2025

View reviewed changes



		def create_auth_route(auth_handler_id: str, agent: AgentApplication):
		def dynamic_function(context: TurnContext, state: TurnState):

	def dynamic_function(context: TurnContext, state: TurnState):
	async def dynamic_function(context: TurnContext, state: TurnState):

	raise ValueError("Activity type must be 'invoke' for send_invoke method.")
	raise ValueError("Activity type must be 'invoke' for send_invoke_activity method.")

	create_auth_route(auth_handler.name),
	create_auth_route(auth_handler.name, app),

		class QuickstartSample(Sample):
		"""A quickstart sample implementation."""

	from .types import SafeObject, DynamicObject
	from .types import SafeObject

	context = AssertionContext(actual, baseline)
	context = AssertionContext(actual, baseline, None, None)

		@@ -0,0 +1,477 @@
		import pytest
		from unittest.mock import Mock

	from .types import SafeObject, DynamicObject, resolve, parent
	from .types import SafeObject, resolve

		@@ -0,0 +1,570 @@
		import pytest
		from unittest.mock import Mock

	raise AttributeError(f"Cannot delete item '{key}' on {type(self).__name__}")
	raise KeyError(f"Cannot delete item '{key}' on {type(self).__name__}")


		from typing import Awaitable, Callable, Iterable

		from aiohttp.web import Request, Response, Application, StreamResponse

		@@ -0,0 +1,4 @@
		class DataDrivenTest

		def __init__(self, test_seq: list[DDTComponent]):


		class Receive(DDTComponent):

		def __init__(self, ) No newline at end of file

		@@ -0,0 +1,73 @@
		from typing import Protocol, TypeVar, overload, Iterable, Callable

Integration package refinements #266

Are you sure you want to change the base?

Integration package refinements #266

Uh oh!

Conversation

rodrigobr-msft commented Dec 2, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Key Changes

Reviewed changes

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview