hgb_scaling: disable OMP_PROC_BIND on hybrid-core machines - #36
Merged
Conversation
OMP_PROC_BIND=true binds each OpenMP thread to a place chosen from across the whole taskset's topology without distinguishing core type. On a hybrid (P-core/E-core) laptop that can place a thread team smaller than the taskset onto slower E-cores instead of packing it onto the faster P-cores the taskset was sized for - observed as a ~2x fit-time regression between two OMP_NUM_THREADS values that both resolved to the same actual tree_n_threads (see PR#34788 benchmark analysis). has_hybrid_cores() detects this via the cpu_core/cpu_atom sysfs directories the kernel exposes for Intel Hybrid Technology. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
configs/hgb_scaling.pysetOMP_PROC_BIND=trueunconditionally. On hybrid-core (P-core/E-core) machines, that binds each OpenMP thread to a place chosen from across the whole taskset's topology without distinguishing core type - so a thread team smaller than the taskset can land partly on slower E-cores instead of packing entirely onto the faster P-cores the taskset was sized for.hgb/both_threads_optim) benchmark results:ames_housingon the laptop runner resolved to the sametree_n_threads=4at bothOMP_NUM_THREADS=4(taskset0-3, all P-cores) andOMP_NUM_THREADS=16(taskset0-15, mixed P/E) yet ran ~2x slower in the wider-taskset case.LD_PRELOADing the repo's ownsklearn-dev-libomplibomp.so: with a narrow taskset all 4 threads bind to P-cores; with a wide hybrid taskset covering the same team size, some threads bind to Atom/E-cores instead.configs/_scaling.pygainshas_hybrid_cores(), detecting Intel Hybrid Technology via thecpu_core/cpu_atomPMU sysfs directories the kernel exposes (absent → treated as non-hybrid).hgb_scaling.py's_with_thread_countnow setsOMP_PROC_BINDto"false"on hybrid machines instead of"true", explicitly either way (same "don't rely on ambient state" rationale already used forOMP_NUM_THREADSin that function).configs/hgb_scaling_force_active_wait.pyneeded no change - it already rebuildsbench.envfrom scratch and never setsOMP_PROC_BIND.Test plan
PIXI_FROZEN=true pixi run -e sklearn-pypi python -c 'from _scaling import has_hybrid_cores; print(has_hybrid_cores())'→Trueon this (hybrid) machine.load_cases_from_script("configs/hgb_scaling.py")→ every generated case now carriesOMP_PROC_BIND=falseon this machine.hgb_scaling.pyon the actual laptop runner and confirm theames_housing4-vs-16-thread anomaly is gone.🤖 Generated with Claude Code