Add ability to deploy custom elastic-agents on different OS or runtimes

[marc-gr]

There are some integrations that might require elastic-agents with custom configurations for them or their container ie: winlogbeat requires a windows container, auditbeat requires special container capabilities, etc.

I initially created #786 that adds ability to deploy custom agents as test services, there is still missing code specific to deal with the windows scenario.

I open this thread to discuss other approaches that might avoid adding the complexity to the test runner if possible.

EDIT:

As mentioned in #787 (comment) , there is now support in elastic-package to:

• run tests in independent Elastic Agents (for now in Linux):
  • these independent Elastic Agents can be customized (with capabilities and scripts).
  • each system test writes into its own data streams
• run system tests in parallel
  • being able to set through an environment variable the maximum number of routines to run in parallel

It will be pending here to allow running Elastic Agents in other OS (e.g. Windows) or in other runtimes (VMs?).

[mtojek]

Regarding #786:

I recommend not modifying the test runner much/at all. We'd like to rewrite that code eventually as it has too many responsibilities and it's error-prone now.

I see a few approaches that we can apply here.

Extend "profiles" with local patches

When a user or CI executes elastic-package stack up command, it checks if there are any local profile patches and creates a custom profile (if it doesn't exist) for that particular integration.

Extend Compose stack definition with environment variables

Let's not add anything special except allowing for var customizations in the Compose stack. Whenever a user or CI executes elastic-package stack up command, it will also load local vars.

Problem: it doesn't solve the problem of elastic-package stack booting on the Windows machine. Should we move it to another, separate issue? Actually, you can pair it with image overrides.

Hack: retag elatic-agent image

It's a hack but may work temporarily. Before running the elastic-package stack up we need to build the agent's Docker image for Windows and replace/retag the current one. It might be hard due to different stack versions and configuration changes.

[mtojek]

pinging @jsoriano for his thoughts around this and #786.

[jsoriano]

Another option could be to move agent initialization to the system test runner, and remove it from the default stack definition. Having it in the runner would allow it to have full control of the started agents, allowing to start them with different options, and handling platform-specific needings, as could be the case of Windows.
It could also allow to start different tests with different agent configuration, for example to test different auditbeat configurations with different capabilities.

We already have a custom agent for the Kubernetes service deployer, we could follow a similar strategy on any other deployer.
There could be general options, such as options to select the version to use, or things like capabilities to add. And there could be platform specific options, that would also allow things like selecting daemonset vs deployment in Kubernetes (#465). On a second iteration these options could be easily overridden with flags or environment variables.

If we remove agent from the default stack definition, we could still have a stack subcommand to start agents for manual tests. Something like this could also cover #548.

I think this could be a more future-proof option, but it can require an important effort.

---

And another option for the use case of starting an agent but no service, can be to add a new "system" deployer, that just starts an agent with a given configuration, intended for system-level monitoring. This could help with packages for auditbeat or for the system module itself. This could be extended in the future to start completely different OSs using VMs.

This would be more in line of #786, but without needing to hack over the current test runner and compose deployer.

[mtojek]

Another option could be to move agent initialization to the system test runner, and remove it from the default stack definition. Having it in the runner would allow it to have full control of the started agents, allowing to start them with different options, and handling platform-specific needings, as could be the case of Windows.
It could also allow to start different tests with different agent configuration, for example to test different auditbeat configurations with different capabilities.

There are two constraints related to this approach:

1. Don't forget that this is the mode we also use for development purposes. You can simply start the stack and have everything ready. It is really convenient.
2. Agent enrollment with fleet server takes time. We considered this option at the early stage and decided to follow the "enroll once" approach at startup. It's also easier to debug if you have the agent instance present, not wiped out.

We already have a custom agent for the Kubernetes service deployer, we could follow a similar strategy on any other deployer.
There could be general options, such as options to select the version to use, or things like capabilities to add. And there could be platform specific options, that would also allow things like selecting daemonset vs deployment in Kubernetes (#465). On a second iteration these options could be easily overridden with flags or environment variables.

I like the approach of having the custom agent setup. You're right that we could apply similar logic as for kind, to spawn a new agent. This way we don't need to modify test runners at all. Most likely we may need two setups: custom image properties and windows.

If we remove agent from the default stack definition, we could still have a stack subcommand to start agents for manual tests. Something like this could also cover #548.

I had that in mind before, hence the issue, but always considered its complexity as +Inf. Maybe we can evaluate it as a good first issue and "rebuild the stack command"?

---

To sum up, my vote would go to custom agent setup.

[jsoriano]

We already have a custom agent for the Kubernetes service deployer, we could follow a similar strategy on any other deployer.
There could be general options, such as options to select the version to use, or things like capabilities to add. And there could be platform specific options, that would also allow things like selecting daemonset vs deployment in Kubernetes (#465). On a second iteration these options could be easily overridden with flags or environment variables.

I like the approach of having the custom agent setup. You're right that we could apply similar logic as for kind, to spawn a new agent. This way we don't need to modify test runners at all. Most likely we may need two setups: custom image properties and windows.

Could this be done without modifying runners?

[mtojek]

Yes, I think so. Same way as closed most of the changes for Kubernetes service deployer in this file. There might be one inconvenience, the agent will be deployed during the first run of the system test.

[jsoriano]

Ah ok, but it would be modifying service deployers. Would you prefer to add an agent to the compose deployer, or to add a new deployer for these use cases?

[mtojek]

Would you prefer to add an agent to the compose deployer

It looks like it depends on the final infrastructure setup. Not sure if that option will work for @marc-gr and Windows containers.

add a new deployer for these use cases?

This option seems to be pluggable and flexible in terms of specific configuration properties or OS-specific logic. It has also an extra benefit, it will prevent copying a custom agent code to multiple places.

I'm thinking now if we aren't close to introducing a feature of using an agent under development. This way you could use even standalone builds. Maybe we should implement a proxy instead :)

[jsoriano]

Discussed offline about this with Marc, he is going to explore the option of implementing something like #786, but as a new deployer, so the runner is not modified. This could cover the current auditbeat needings.

We also discussed that probably we need something like vms for system tests, this will be neccesary to support running tests with windows, or even with linux if not enough privileges can be granted with containers for some use cases.
We could run these tests on specialized CI workers, as in elastic/integrations#1713 for Windows, but it'd be nice to have something in elastic-package so developers working on Mac/Linux can also run these tests locally.