Draft: Update core assignment algorithm in benchexec/resources.py #892
+3,012
−330
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CGall42
added
the
resource allocation
PhilippWendler
requested changes
Jan 26, 2023
Member
PhilippWendler
left a comment
PhilippWendler
left a comment
There was a problem hiding this comment.
This is a preliminary review with some hints, mostly on code style and documentation. Please fix these issues and provide documentation, such that a full review becomes possible and makes sense.
As part of this (as first step actually), please format the source code with the formatter black and make sure to use it in the future for each commit. Consistent code formatting is a big help for readability.
And please also have a look at all the other CI failures. After each commit these checks will run automatically, so always check whether CI is green for your most recent commit. The check check-format is fixed automatically if you use the code formatter, and reuse only complains about the missing copyright header. But flake8 and pytype provide hints about potential errors in your code. And of course the unit-tests checks just execute our tests.
This method adds a "root" hierarchy level, if the system topology doesnt have one. Necessary for iterating through the whole topology.
now chooses the right starting core for the next thread
The pytest-based assertion will replace the use of the assertInvalid method
As we use the equivalent pytest-based assertion, the old assertion is redundant
The _test_nCoresPerRun method is a remainder of prior refactoring - as it only contains a single call to another function, we can remove and inline its content safely
As with the other decorators, this centralization of the logic allows us to use pytest parameterization later on
This commit, again, still has the old calls, to have both in one test run
All calls to assertValid (from within the actual test classes) are now also performed via decorator
The existing logic was quite hard to read - to make future maintenance easier, the logic is broken up a bit to use explicit variable names, bundled in a single auxillary function
Identical functionality to the existing function can be achieved via set comparision, resulting in significantly simplified logic.
Using pythonic methods, especially the code to remove superfluous hyperthreading cores can be made more readable, while keeping identical functionality.
blocked_cores is completely unused and serves no purpose even after excessive search
While it does not do much for readability in the while loop itself, this increases readability of the if branch conditions inside it
This prior use of a temporary variable before assigning a shallow copy to the actual variable does not serve any purpose
To reduce the complexity of core_allocation_algorithm, it seems like a good idea to decouple the high level parts of the algorithm itself from the actual crawling of lists and dicts. Note that this commit only duplicates the code, to be able to use an assertion to make sure the new auxillery function returns equivalent results. The existing code block (and assertion) is removed in the next commit.
(and assertion)
Using enumerate is the more pythonic way, and this code should be more readable in general
From the old name it was quite unclear what this function actually does
Instead of having a any() construction with an auxillery variable, we can use a negated all() construct
…num_of_values To be able to remove check_asymmetric_num_of_values, we move the actual code check_symmetric_num_of_values, removing the final negation from the all() statement
As check_asymmetric_num_of_values is only a negation of another function, and as "not is_symmetric_hierachy" is quite clear, we can remove this function
As we only really need the fast_cores list, this way, we simply can remove the slow_cores list comprehension (if we do not debug print)
To keep behavior consistent, we still print the list of unused cores if debug printing is enabled, but as this information is otherwise unused, the whole calculation is only executed in this case
A simplified implementation of the algorithm selecting the next free core on the first symmetrical level. To ensure consistency with the existing code, the old code is still left, with an assertion checking both results are equal.
Siloing the algorithm selecting cores for each run into a helper function enables splitting the monolithic core_allocation_algorithm method apart later and removes an additional while loop.
core_allocation_algorithm performs destructive operations on hierarchy_levels - to remove the unexpected side-effect of it emptying the dict, we create and work on a deepcopy of it inside the function
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Referring to issue #748, the core assignment can now handle additional hierarchy layers (such as a shared L3 cache).
The addition of further layers can be implemented without knowing the exact topology of a machine - the hierarchy of the layers (CPUs, NUMA nodes, L3 caches, hyperthreading, etc) is determined by the algorithm.
Fixes #748
Fixes #850
Kernel documentation: