Not working with parallel mode

Depends on test not working with pabot with mutiple test case run at the same time. I saw the not test cae not found.

[mentalisttraceur]

Thanks for bringing this up!

Could you tell me more about your specific usecase? Would be great to see example code that preserves the relationships between tests which run in parallel, and a description of what kind of behavior you are hoping for.

The big challenge is that in the general case, simple test-level parallelism like what pabot does is inherently incompatible with dependencies between tests. A smaller challenge is state sharing.

For example, let's say we run pabot with two executors on a suite with tests Foo, Bar, and Qux. A Depends on test Foo within Qux has no power over pabot scheduling the tests (and this library can't reach out and effect pabot like that, because this library's keywords within each test don't run until after the test is already started on some executor). So if Foo takes a while and Bar finishes first, then Qux might get scheduled to run on the second process where only Bar ran, while the first process is still not finished with Foo.

So the scheduling problem is that pabot might schedule Qux before Foo even finished. The state-sharing problem is that the instance of DependencyLibrary in the second process might not even get told about the execution of test Foo in the first process.

Actually, Qux could start executing before Foo even started! For example if we raise the parallelism to three or more executors, or if Qux is right after Foo in the test file, and you get unlucky with how your OS schedules the pabot processes that time.

There is only so much that DependencyLibrary could even do about this:

1. DependencyLibrary could try importing pabot's library and use it to check if it is running under pabot. It would have to take care to not misbehave or impose too much overhead if pabot is not installed or installed but not being used.

2. DependencyLibrary could use pabot APIs to communicate across all pabot executors each time it gets any test case event or test suite event that it normally records. It has to be every event to all executors because DependencyLibrary can't know in advance what dependencies are in all the future tests in the entire run.

3. DependencyLibrary could use pabot APIs to block in each "depends on ..." keyword if the test or suite being depended on is recorded as passed, failed, or skipped. It would have to take care to detect and report deadlocks as much as possible, because otherwise a mistake in dependencies could cause the tests to block forever.

Downsides with this approach:

• Really tricky to get right. Requires a lot of brute force testing of possible permutations of events to achieve anywhere close to decent test coverage. Realistic bugs in this kind of parallelism synchronization tend to be rare to strike, non-obvious, hard to debug, and nearly impossible to reproduce.
• Requires a different solution to coexist in the code for each alternative parallelization solution - for example if pabot ever makes a breaking change to their API, or if there is an alternative to pabot that users want to use. It's a bad sign in software design when generalizing a solution looks like O(n) scaling of implementation effort and conditional branches in the code.
• Deadlocks, when they happen, would not be entirely deterministic.
• Dependencies between tests still reduce how much code can be parallelized, no way around it. If you have most tests depending on each other, then no matter how much parallelism you throw at the problem, parallel test executors might spend most of their time waiting for each other to finish tests they depend on.
• Parallelism adds ambiguity/randomness. When you have a single linear well-defined order of execution, Depends on test Foo always refers to the first Foo that you find when reading upwards in the file. But Robot Framework allows multiple tests with the same name. Test-level parallelism means that a lexically earlier Foo could execute after a lexically later Foo, and if the results of those two tests are different, that could non-obviously change the behavior of the dependant test.
• If you declare a dependency on something that doesn't exist, instead of erroring out immediately, you'd now have to wait until all tests ran before you could be sure that a nonexistent dependency was asked for.
• Bug reports from any issues would be really annoying. I can already imagine someone opening an issue which takes a long time and a lot of back-and-forth and frustration to debug, only to end in "oh, it's because [one of the above points]".
• If you only have a couple tests that depend on each other in a large suite, a one-off hard-coded solution for just those tests is simpler and easier to get right than a generally robust solution like the kind appropriate at the library level.
• Most systems and tests should be factored so that the tests are independent. So this seems like it would have a rather small impact of helping people with tests that really need it, and a much larger impact of enabling and empowering bad practice in situations where the desire for parallelization is a good incentive for taking the time to refactor tests to have less dependencies.

So I need to really think about it before making DependencyLibrary get into the business of automagically working when parallelized with something like pabot.

Another alternative, a better one in some ways and worse in others, is for tools like pabot to treat "depends on ..." keywords as a standard for declaring dependencies between tests, statically analyze Robot tests for those dependency declarations schedule tests accordingly. This would in some ways be a lot better from an engineering perspective, because it allows much cleaner, uncoupled factoring: the static analysis piece could be independent (I'd be happy to provide that), DependencyLibrary could stay independent of the parallelization, some dependency deadlocks could be detected sooner, and test scheduling across executors could be optimized based on dependency information. In fact this is the same data dependency and scheduling problem that I'm sure has plenty of general solutions in reusable libraries already, since CPUs and optimizing compilers and schedulers and so on do this kind of analysis. (Static analysis also makes it easier to solve the problem of non-deterministic-ordering between depended-on tests with the same name.)

Of course, having a solution inside DependencyLibrary lets the solution still work when faced with use-cases and edge-cases which do not lend themselves to static analysis (for example, dynamically specified or conditional dependencies between tests), and if implemented properly, it would even allow a user to supply the glue at runtime between a new implementation of parallelizing tests and DependencyLibrary.

So I've got a lot to think about. Knowing more about your use-case/situation might help narrow things down or lead to a great partial/first-step solution.

P.S. By the way, this is one real world example of why dependencies between tests are discouraged in test design - one consequence of coupling tests like that is that many tools have to do a lot more work to work right if there are dependencies between tests.

[mentalisttraceur]

I've actually seen production tests where the only thing blocking a switch from suite-level parallelism to test-level parallelism was just one or two test dependencies within a suite.

If DependencyLibrary automagically provided under-the-covers logic to make test-level parallelism Just Work, at worst degrading back towards serial test execution with idling executors, that would be a real win in real cases.

And in fact it would create greater incentive to explicitly declare dependencies between tests, and I really want people to do that.

[mentalisttraceur]

Another thought: with a very tiny addition to pabot, having all this logic in DependencyLibrary becomes even more powerful - if a test keyword can tell the parallel executor to "skip me for now and try running me again later (after the test/suite named {{whatever}} is done)", for example by raising an exception type meant for the purpose, then test execution reordering for maximum throughput to avoid blocking on dependencies emerges dynamically with relatively little overhead at runtime without any static analysis.

[mentalisttraceur]

EDIT: nevermind this comment (explanation in next comment).

Actually, did you try using pabot.SharedLibrary? Like this:

-Library  DependencyLibrary
+Library  pabot.SharedLibrary  DependencyLibrary

Documented here, source here.

That should take care of the state sharing part, so if your tests are not also running into sequencing problems, this may be a good temporary workaround.

If you're doing this and still seeing "not found" messages which only happen during parallel runs, then you're going to need a more complete solution.

[mentalisttraceur]

Actually wait, that probably won't work because from a quick glance at that source code I think pabot.SharedLibrary doesn't forward Robot events, and DependencyLibrary needs the events from Robot to know which tests have executed and what their status is.

[mentalisttraceur]

Tomorrow I intend to start working on a PR for pabot.SharedLibrary to support Robot Framework's event listener API - that would be step one of supporting dependencies across multiple pabot runners.

Once that's in place, dependencies will at least be able to always know about each other, though you will have to use the separate test ordering feature to make sure dependencies always run before their dependents.

From there, I'm hoping to think of a way to make it a totally smooth, Just Works experience, but that might take much longer.

[mentalisttraceur]

Haven't ended up actually getting the time to work on this, but in other news, PreRunModifier looks potentially useful here.

[montsamu]

What we're doing is using a pre-pass to create a suiteordering.txt file, e.g. for the basic example:

$ cat suiteorder.txt
{
--test Test.Test.Ci360-Test-Scenarios.A Test That Depends On Passing Test #DEPENDS Test.Test.Ci360-Test-Scenarios.Passing Test
--test Test.Test.Ci360-Test-Scenarios.Passing Test
}

Then passing -ordering suiteorder.txt to our pabot invocation. This does force these dependencies to execute on the same pabot executor, which is one small price to pay for this "just working"!

[joonaskuisma]

Try DependencySolver

Greetings!

@montsamu 's suggestion is indeed a valid approach. As @mentalisttraceur suggest PreRunModifier works, so I've built my own PreRunModifier package that automatically generates the ordering.txt file based on dependencies defined using DependencyLibrary.

You can read more about the package here: https://pypi.org/project/robotframework-dependencysolver/
Or check it out directly on GitHub: https://github.com/joonaskuisma/robotframework-dependencysolver

Additionally, starting from pabot 4.1.0, pabot now includes support for my DependencySolver. DependencySolver also works directly with Robot Framework, even if you don't need pabot, but still want to run dependent test chains.
So, if you need to run dependent test chains in any case, give DependencySolver a try!

For now, I've marked the project as Beta, so any feedback and development ideas are highly welcome! 🙂