Inconsistent map of hardware threads from v3 onwards

[fgava90]

There is an inconsistency in mapping hardware threads from prrte v3 onwards.

I have a 2 sockets, 64 cores per socket, 2 threads per core system.

When using openmpi v4 options:
mpirun -np 4 --verbose --bind-to cpu-list:ordered --cpu-list 0,1,128,129 --report-bindings hostname
i obtain:

MCW rank 0 bound to socket 0[core 0[hwt 0]]
MCW rank 1 bound to socket 0[core 1[hwt 0]]
MCW rank 2 bound to socket 0[core 0[hwt 1]]
MCW rank 3 bound to socket 0[core 1[hwt 1]]

which i deem correct as other tools (hwloc, htop, ...) also consider 0-127 to be logical hwt 0, and 128-255 to be hwt 1.
Also /proc/cpuinfo gives:

...
...
processor   : 128
...
core id     : 0
...
...
processor   : 129
...
core id     : 1
....

When attempting the same with openmpi v5 i get this:
mpirun -np 4 --verbose --bind-to hwt --map-by pe-list=0,1,128,129:ordered:hwtcpus --report-bindings --mca mca_base_verbose debug --mca rmaps_base_verbose 100 hostname

Rank 0 bound to package[0][hwt:0]
Rank 1 bound to package[0][hwt:128]
Rank 2 bound to package[1][hwt:64]
Rank 3 bound to package[1][hwt:192]

To me this looks like the mapping logic assumes:

0 = core 0, hwt 0
1 = core 0, hwt 1
128 = core 64, hwt 0
129 = core 64, hwt 1

instead of the correct map:

0 = core 0, hwt 0
1 = core 1, hwt 0
128 = core 0, hwt 1
129 = core 1, hwt 1

I've tested this with the current mpi main branch (last release is 5.0.8) and:

1. internal pmix + prrte v3.0.11
2. pmix master+ prrte v4.0.0

Below the info for the 2nd case as it's the current commit/release on everything

ompi_info
                Open MPI: 5.1.0a1
  Open MPI repo revision: cfb5505
   Open MPI release date: Unreleased developer copy
                 MPI API: 3.1.0
            Ident string: 5.1.0a1

pmix_info
                    PMIX: 7.0.0a1
      PMIX repo revision: 5ee562a
       PMIX release date: Unreleased developer copy
           PMIX Standard: 4.1
       PMIX Standard ABI: Stable (0.0), Provisional (0.0)

prte_info
                    PRTE: 4.0.0a1v4.0.0
      PRTE repo revision: v4.0.0
       PRTE release date: @PMIX_RELEASE_DATE@
                    PMIx: OpenPMIx 7.0.0a1, repo rev: 5ee562a (PMIx Standard: 4.1, Stable ABI: 0.0, Provisional ABI: 0.0)

[rhc54]

Please post your topology output from "lstopo" in xml format.

Looks to me like this is just a case of reporting physical vs logical IDs in the print output.

[fgava90]

topo.xml.txt

Sorry i needed to edit out some sensitive info from the xml output.
You should have what's necessary for this issue.

Looks to me like this is just a case of reporting physical vs logical IDs in the print output.

I don't think that's the case as htop shows they are bound as the output of the mpirun command says

[fgava90]

See below the htop screengrabs that prove the binding is indeed as reported.
With OpenMPI-5.0.8:

mpirun -np 4 --verbose --bind-to hwthread --map-by pe-list=0,1,128,129:ordered:hwtcpus --report-bindings --mca mca_base_verbose debug --mca rmaps_base_verbose 100 ./use_cpu.sh
...
Rank 0 bound to package[0][hwt:0]
Rank 1 bound to package[0][hwt:128]
Rank 2 bound to package[1][hwt:64]
Rank 3 bound to package[1][hwt:192]

With OpenMPI-4.1.6:

mpirun -np 4 --verbose --bind-to cpu-list:ordered --cpu-list 0,1,128,129 --report-bindings --mca mca_base_verbose debug --mca rmaps_base_verbose 100 ./use_cpu.sh
...
MCW rank 0 bound to socket 0[core 0[hwt 0]]
MCW rank 1 bound to socket 0[core 1[hwt 0]] 
MCW rank 2 bound to socket 0[core 0[hwt 1]]
MCW rank 3 bound to socket 0[core 1[hwt 1]] 

[rhc54]

Afraid your topo file is damaged - can't read it:

./ompi-user2.xml:579: parser error : Opening and ending tag mismatch: topology line 3 and object
  </object>
           ^

Instead of pasting it into the comment box (which has issues with the greater-than/less-than symbols), it is usually better to put it in a gist or add it as a file.

And please don't post screenshots - virtually impossible to make any sense of them. Let me just have a little time to look into this - I'm busy with other things, so it may not happen right away.

[fgava90]

I have updated my comment above with the topology as a file.
I had to update the xml as .txt as xml is not supported by github issues.

The screenshots were to prove (with htop outputs) that the processes are indeed bound wrong.
If you have a better way to show it i'll provide that instead.

[rhc54]

Yeah, there's just something wrong about that topology file - hwloc refuses to ingest it. I'll use a different one to see if I can reproduce the problem.

Screenshots are very hard to read - yours were frankly impossible. I don't like getting into debates over "proving" something is wrong - just report the perceived problem and let me try to reproduce it. I'll be able to tell if there is a real problem, or simply an issue with how we are printing the report.

[fgava90]

Sorry, i don't know why that doesn't work.
If you look at this section:

        <object type="Core" os_index="0" ... >
          <object type="PU" os_index="0" ... />
          <object type="PU" os_index="128" ... />
        </object>

it shows that core 0 is made by PU 0 and PU 128, same for the others of course.

Screenshots were of htop, which shows CPUs at 100% usage where there's load, in this case only the bound cpus.
In the first one you can see the load on cpus 1,65,129,193 as reported by the openmpi-v5 output (there's a 1 shift as count starts from 1 instead of 0) .
In the 2nd one you can see the load on cpus 1,2,129,130 (same 1 shift) as requested with openmpi-v4.

[rhc54]

I cannot speak to what is going on with your topology file. On the roughly 30 I have here, the mapping works exactly as it should. The only "error" I can find is that the binding report shows physical CPU IDs instead of logical, even though the input is required to be in logical IDs. I also note that we aren't getting a good error output when the CPU list runs outside of the cpus on the node - e.g., if someone includes pe-list=0,1,300 and there aren't 300 logical cpus.

I can work the report and deal with the bad input error when I have time (probably the next couple of weeks), but I don't see any errors in the mapping/binding. This is all with head of PRRTE master branch, so it is possible that an error exists on some earlier release.

It is also possible that the problem on your machine lies in our reading of the topology given that hwloc errors out when trying to read the xml file - even though I configured PRRTE to use HWLOC v2.2.0 since that is what your file claims you are using. Or it could be that the output is simply confusing you into thinking the mapping is incorrect. 🤷‍♂️ Don't know.

[fgava90]

As i said, looking at the load on the node, the reporting is correct and the binding is done on the logical cpu id reported.
Can you try with this new topology file i've uploaded?
It's from a similar node with the same issue:

topo2.xml.txt

[fgava90]

On the roughly 30 I have here, the mapping works exactly as it should.

How do you test this?

[rhc54]

New topology worked fine - thanks! Tests show it also works as expected. The problem remains in the printed output:

Rank 0 bound to package[0][hwt:0]
Rank 1 bound to package[0][hwt:128]
Rank 2 bound to package[1][hwt:64]
Rank 3 bound to package[1][hwt:192]

is incorrectly reporting that:

• rank 0 is bound to physical cpu 0, which happens to be logical 0
• rank 1 is bound to physical cpu 128, which happens to be logical 1
• rank 2 is bound to physical cpu 64, which happens to be logical 128
• rank 3 is bound to physical cpu 192, which happens to be logical 129

which is precisely what you asked to have happen. We just need to get the printout correct 😄

[rhc54]

On the roughly 30 I have here, the mapping works exactly as it should.

How do you test this?

You can simulate any topology given an xml file, and then output the resulting map/bind pattern. For this case:

mpirun -np 4 --prtemca hwloc_use_topo_file ./ompi-user4.xml --runtime-options donotlaunch --map-by pe-list=0,94,2,88:ordered:hwt --display map hostname

results in:

========================   JOB MAP   ========================
Data for JOB prterun-Ralphs-iMac-39288@1 offset 0 Total slots allocated 128
    Mapping policy: PE-LIST:NOOVERSUBSCRIBE:ORDERED  Ranking policy: SLOT Binding policy: HWTHREAD:IF-SUPPORTED
    Cpu set: 0,94,2,88  PPR: N/A  Cpus-per-rank: N/A  Cpu Type: HWT


Data for node: Ralphs-iMac	Num slots: 128	Max slots: 0	Num procs: 4
        Process jobid: prterun-Ralphs-iMac-39288@1 App: 0 Process rank: 0 Bound: package[0][hwt:0]
        Process jobid: prterun-Ralphs-iMac-39288@1 App: 0 Process rank: 1 Bound: package[0][hwt:47]
        Process jobid: prterun-Ralphs-iMac-39288@1 App: 0 Process rank: 2 Bound: package[0][hwt:1]
        Process jobid: prterun-Ralphs-iMac-39288@1 App: 0 Process rank: 3 Bound: package[0][hwt:44]

Warning: This map has been generated with the DONOTLAUNCH option;
	The compute node architecture has not been probed, and the displayed
	map reflects the HEADNODE ARCHITECTURE. On systems with a different
	architecture between headnode and compute nodes, the map can be
	displayed using `prte --display map /bin/true`, which will launch
	enough of the DVM to probe the compute node architecture.

[fgava90]

I do not agree with your reply.
Or at least this is inconsistent with the same cpu list given to openmpi 4.
Are we then to infer that from openmpi 5 the pe-list is logical cpus and nor physical cpus anymore?
Running a script that simulates cpu load and having it monitored on a node i can clearly see the binding being different from the two versions.

[rhc54]

The documentation made the change to logical rather clear, I think - though I agree that many people don't bother to read it. The argument over whether to use logical vs physical IDs has raged for over 20 years. No matter which way we go, someone always argues that it should have been the other.

Note that it is typical for things to change between major releases. OMPI folks try to highlight those changes in the release notes as well as the docs - though, again, many people don't read either of those.

I plan to take a look at adding a "p" in front of the number to indicate it should be taken as physical. Not a high priority, so it may take a while before it shows up. Meantime, the system uses logical cpu IDs, and I'll fix the output to match.

[fgava90]

You are dinitely right.
I'm sorry about this.
I have read the doc but that "detail" escaped me.
It's quite a tricky and important change though, i would expect a big note there saying it changed from previous version rather than casually just mentioning it is logical cpus. But it was indeed mentioned so i'll take the blame for not reading more carefully.
I guess the wrong output made it all more confusing.
Thanks for the help!