BeeGFS becomes inconsistent when registering a storage node/target fails

[frsc-ku]

I'm observing a problem with BeeGFS 8 community version when adding a storage node/target fails. The list of targets in the storage_pool_default becomes inconsistent with the actual status of the system. It looks like the targets in the pool get added even though there occurs an error during registration. I now have a BeeGFS with no storage targets, but the default pool shows 36 targets as members.

How can I clean this up/recover from such a registration failure?

Here some more info from logs and command output:

Pool list:

# beegfs pool list
ID   ALIAS                 TARGETS                                                                                                                                                                                                                  MIRRORS
s:1  storage_pool_default  s:101,s:102,s:103,s:104,s:105,s:106,s:107,s:108,s:109,s:110,s:111,s:112,s:113,s:114,s:115,s:116,s:117,s:118,s:119,s:120,s:121,s:122,s:123,s:124,s:125,s:126,s:127,s:128,s:129,s:130,s:131,s:132,s:133,s:134,s:135,s:136

There seems to be no command to remove targets from a pool. I guess that should happen automatically - the mgmt-daemon should keep everything consistent? The naive attempt fails:

# beegfs target delete s:136
Error: rpc error: code = Unknown desc = Delete target: target storage:136 does not exist

Target list:

# beegfs target list
ID     TYPE  ALIAS            NODE   STORAGE_POOL
m:101  meta  target_meta_101  m:101  (n/a)
m:102  meta  target_meta_102  m:102  (n/a)
m:103  meta  target_meta_103  m:103  (n/a)
m:104  meta  target_meta_104  m:104  (n/a)

Node list:

# beegfs node list
ID     TYPE        ALIAS
m:101  meta        node_meta_101
m:102  meta        node_meta_102
m:103  meta        node_meta_103
m:104  meta        node_meta_104
mg:1   management  management

Daemon status:

beegfs-meta@meta_g5_a1.service     loaded  active  running  BeeGFS  Metadata    Server  (multimode)
beegfs-meta@meta_g5_a2.service     loaded  active  running  BeeGFS  Metadata    Server  (multimode)
beegfs-meta@meta_g5_a3.service     loaded  active  running  BeeGFS  Metadata    Server  (multimode)
beegfs-meta@meta_g5_a4.service     loaded  active  running  BeeGFS  Metadata    Server  (multimode)
beegfs-mgmtd.service               loaded  active  running  BeeGFS  Management  Server
beegfs-storage@storage_g1.service  loaded  failed  failed   BeeGFS  Storage     Server  (multimode)
beegfs-storage@storage_g2.service  loaded  failed  failed   BeeGFS  Storage     Server  (multimode)
beegfs-storage@storage_g3.service  loaded  failed  failed   BeeGFS  Storage     Server  (multimode)
beegfs-storage@storage_g4.service  loaded  failed  failed   BeeGFS  Storage     Server  (multimode)

Each storage service has 8 storage targets as a member. The total count of storage targets is 32.

I had a bug in the script deploying the storage node. I don't recall exactly what it was: either I gave all targets of a service the same target ID or I gave all services associated with a systemd unit a different ID. Either way, beegfs-setup-storage returned without error, but the unit couldn't start. I got this "Register target: UNIQUE constraint failed" error in the log:

Aug  8 18:37:50 rubusnas01fl beegfs-storage[241742]: Main [App] >> Built with NVFS RDMA support.
Aug  8 18:37:50 rubusnas01fl beegfs-storage[241742]: Main [RegDGramLis] >> Listening for UDP datagrams: any Port 8210
Aug  8 18:37:50 rubusnas01fl beegfs-storage[241742]: Main [App] >> Waiting for beegfs-mgmtd@rubusnas01fl:8008...
Aug  8 18:37:50 rubusnas01fl beegfs-storage[241742]: RegDGramLis [Heartbeat incoming] >> New node: beegfs-mgmtd management [ID: 1];
Aug  8 18:37:50 rubusnas01fl beegfs-storage[241742]: Main [NodeConn (acquire stream)] >> Connected: [email protected]:8008 (protocol: TCP)
Aug  8 18:37:50 rubusnas01fl beegfs-mgmtd[239264]: Register target: UNIQUE constraint failed: targets.node_type, targets.target_id: Error code 2067: A UNIQUE constraint failed
Aug  8 18:37:50 rubusnas01fl beegfs-storage[241742]: Main [App] >> Target ID reservation request was rejected by this mgmt node: management [ID: 1]
Aug  8 18:37:50 rubusnas01fl beegfs-storage[241742]: Error: Target pre-registration at management node failed
Aug  8 18:37:50 rubusnas01fl beegfs-storage[241742]: [BeeGFS Storage Node Version: 8.1.0
Aug  8 18:37:50 rubusnas01fl beegfs-storage[241742]: Refer to the default config file (/etc/beegfs/beegfs-storage.conf)
Aug  8 18:37:50 rubusnas01fl beegfs-storage[241742]: or visit http://www.beegfs.com to find out about configuration options.]
Aug  8 18:37:50 rubusnas01fl beegfs-storage[241742]: Main [App] >> Target pre-registration at management node failed
Aug  8 18:37:50 rubusnas01fl systemd[1]: beegfs-storage@storage_g1.service: Main process exited, code=exited, status=1/FAILURE
Aug  8 18:37:50 rubusnas01fl systemd[1]: beegfs-storage@storage_g1.service: Failed with result 'exit-code'.

I managed to clear out the data base error by deleting the storage node "s:101" that was still in the system and now get this "Register target: Alias with value target_0-68962291-65 already exists" message:

Aug 11 10:20:12 rubusnas01fl beegfs-storage[280605]: Main [App] >> Built with NVFS RDMA support.
Aug 11 10:20:12 rubusnas01fl beegfs-storage[280605]: Main [RegDGramLis] >> Listening for UDP datagrams: any Port 8210
Aug 11 10:20:12 rubusnas01fl beegfs-storage[280605]: Main [App] >> Waiting for beegfs-mgmtd@rubusnas01fl:8008...
Aug 11 10:20:12 rubusnas01fl beegfs-storage[280605]: RegDGramLis [Heartbeat incoming] >> New node: beegfs-mgmtd management [ID: 1];
Aug 11 10:20:12 rubusnas01fl beegfs-storage[280605]: Main [NodeConn (acquire stream)] >> Connected: [email protected]:8008 (protocol: TCP)
Aug 11 10:20:12 rubusnas01fl beegfs-mgmtd[239264]: Register target: Alias with value target_0-68962291-65 already exists
Aug 11 10:20:12 rubusnas01fl beegfs-storage[280605]: Main [App] >> Target ID reservation request was rejected by this mgmt node: management [ID: 1]
Aug 11 10:20:12 rubusnas01fl beegfs-storage[280605]: Error: Target pre-registration at management node failed
Aug 11 10:20:12 rubusnas01fl beegfs-storage[280605]: [BeeGFS Storage Node Version: 8.1.0
Aug 11 10:20:12 rubusnas01fl beegfs-storage[280605]: Refer to the default config file (/etc/beegfs/beegfs-storage.conf)
Aug 11 10:20:12 rubusnas01fl beegfs-storage[280605]: or visit http://www.beegfs.com to find out about configuration options.]
Aug 11 10:20:12 rubusnas01fl beegfs-storage[280605]: Main [App] >> Target pre-registration at management node failed
Aug 11 10:20:12 rubusnas01fl systemd[1]: beegfs-storage@storage_g1.service: Main process exited, code=exited, status=1/FAILURE
Aug 11 10:20:12 rubusnas01fl systemd[1]: beegfs-storage@storage_g1.service: Failed with result 'exit-code'.

I interpret this as there is left-over information from the first failed attempt in the data base that now prevents registration of this target.

Before I scrap the entire system and start from scratch, I would really like to know if there is a procedure to clean up something like that. For now, its a test system with no data on it. In the future, if something like that happens, there should be a way to remove data that is the result of an aborted transaction. My actual expectation is that any administrative operation is of the type all-or-nothing and observations like the one I report here should be impossible.

The health status itself is reported as "healthy":

# beegfs health check --force-connections=false
################################################
Running Health Check for beegfs://localhost:8010
################################################
###################################
>>>>> Checking for Busy Nodes <<<<<
###################################

✅ Busy Metadata Nodes -> Does the number of queued requests exceed the degraded (16) or critical (512) thresholds?
✅ Busy Storage Nodes -> Does the number of queued requests exceed the degraded (16) or critical (512) thresholds?

############################
>>>>> Checking Targets <<<<<
############################

✅ Reachability -> Are all targets responding?
✅ Consistency -> Are all mirrors synchronized?
✅ Available Capacity -> Are any targets low on free space based on the thresholds defined by the management service?

HINT: This mode does not check file system consistency. To check for file system inconsistencies,
      you can run 'beegfs-fsck --checkfs --readOnly' and consult with ThinkParQ support.

################################################
>>>>> Checking Connections to Server Nodes <<<<<
################################################
No client mounts found, skipping connection checks

Thanks for any hints,
Frank

[frsc-ku]

Some information found on the storage targets indicating partial initialisation:

# for d in storage_* ; do echo -n "$d $(<${d}/nodeNumID) $(<${d}/targetNumID)" ; [[ -e "${d}/targetID" ]] && echo " $(<${d}/targetID)"
 || echo ""; done
storage_g1_a1 101 101 0-68962291-65
storage_g1_a2 101 102 1-68962291-65
storage_g1_a3 101 103 2-68962291-65
storage_g1_a4 101 104 3-68962291-65
storage_g1_a5 101 105 4-68962291-65
storage_g1_a6 101 106
storage_g1_a7 101 107
storage_g1_a8 101 108
storage_g2_a1 102 109 0-68962296-66
storage_g2_a2 102 110
storage_g2_a3 102 111
storage_g2_a4 102 112
storage_g2_a5 102 113
storage_g2_a6 102 114
storage_g2_a7 102 115
storage_g2_a8 102 116
storage_g3_a1 103 117 0-6896229A-67
storage_g3_a2 103 118
storage_g3_a3 103 119
storage_g3_a4 103 120
storage_g3_a5 103 121
storage_g3_a6 103 122
storage_g3_a7 103 123
storage_g3_a8 103 124
storage_g4_a1 104 125 0-6896229E-68
storage_g4_a2 104 126
storage_g4_a3 104 127
storage_g4_a4 104 128
storage_g4_a5 104 129
storage_g4_a6 104 130
storage_g4_a7 104 131
storage_g4_a8 104 132

[frsc-ku]

I was going to look into the data base (https://doc.beegfs.io/latest/advanced_topics/management_service.html) and get stuck with this error that is thrown on every SQL command:

# sqlite3 /data/beegfs/mgmt/mgmtd.sqlite .schema
Error: malformed database schema (entity_types) - near "STRICT": syntax error

[frsc-ku]

Starting to answer my own questions. Apparently, the sqlite3 executable version 3.34.1 shipped with RHEL9 is too old, strict mode was introduced with version 3.37:

Looks like the problem is that vscode-sqlite is linked against an old version of SQLite.

Strict mode was introduced in version 3.37 where this extension is using 3.26 (https://github.com/AlexCovizzi/vscode-sqlite/tree/master/bin). Current version is 3.40.1

I compiled sqlite3 from source (the pre-built binaries require a newer libm than RHEL9 ships) and can now read the data base file. I will look into my questions above and report back here.

[iamjoemccormick]

Hi Frank,

Can you share what your database looked like in this state?

sqlite3 mgmtd.sqlite
.headers on
.mode columns
SELECT * FROM targets_ext;
SELECT * FROM storage_targets;
SELECT * FROM storage_pools;

What arguments is your script passing to beegfs-setup-storage? Even if you don't know what exactly was specified the first time around, what arguments (e.g., -i, -s) it does set would be helpful.

~Joe

[frsc-ku]

Hi Joe, thanks for getting back on this. Output of DB queries:

$ sqlite3 mgmtd.sqlite
SQLite version 3.50.4 2025-07-30 19:33:53
Enter ".help" for usage hints.
sqlite> .headers on
sqlite> .mode columns

sqlite> SELECT * FROM targets_ext;
alias            target_uid  node_type  target_id  entity_type  node_id  pool_id  total_space   total_inodes  free_space    free_inodes  consistency  last_update          node_uid
---------------  ----------  ---------  ---------  -----------  -------  -------  ------------  ------------  ------------  -----------  -----------  -------------------  --------
target_meta_101  4           1          101        2            101               920739545088  900057712     920468926464  899991635    1            2025-08-12 13:29:10  3
target_meta_102  11          1          102        2            102               920741642240  900057712     920471023616  899991635    1            2025-08-12 13:29:39  10
target_meta_103  13          1          103        2            103               920739545088  900057712     920468926464  899991635    1            2025-08-12 13:29:39  12
target_meta_104  15          1          104        2            104               920739545088  900057712     920468926464  899991635    1            2025-08-12 13:29:39  14

sqlite> SELECT * FROM storage_targets;
target_uid  node_type  target_id  entity_type  node_id  pool_id  total_space  total_inodes  free_space  free_inodes  consistency  last_update
----------  ---------  ---------  -----------  -------  -------  -----------  ------------  ----------  -----------  -----------  -------------------
48          2          101        2                     1                                                            1            1970-01-01 00:00:00
49          2          102        2                     1                                                            1            1970-01-01 00:00:00
50          2          103        2                     1                                                            1            1970-01-01 00:00:00
51          2          104        2                     1                                                            1            1970-01-01 00:00:00
16          2          105        2                     1                                                            1            1970-01-01 00:00:00
17          2          106        2                     1                                                            1            1970-01-01 00:00:00
18          2          107        2                     1                                                            1            1970-01-01 00:00:00
19          2          108        2                     1                                                            1            1970-01-01 00:00:00
20          2          109        2                     1                                                            1            1970-01-01 00:00:00
21          2          110        2                     1                                                            1            1970-01-01 00:00:00
22          2          111        2                     1                                                            1            1970-01-01 00:00:00
23          2          112        2                     1                                                            1            1970-01-01 00:00:00
24          2          113        2                     1                                                            1            1970-01-01 00:00:00
25          2          114        2                     1                                                            1            1970-01-01 00:00:00
26          2          115        2                     1                                                            1            1970-01-01 00:00:00
27          2          116        2                     1                                                            1            1970-01-01 00:00:00
28          2          117        2                     1                                                            1            1970-01-01 00:00:00
29          2          118        2                     1                                                            1            1970-01-01 00:00:00
30          2          119        2                     1                                                            1            1970-01-01 00:00:00
31          2          120        2                     1                                                            1            1970-01-01 00:00:00
32          2          121        2                     1                                                            1            1970-01-01 00:00:00
33          2          122        2                     1                                                            1            1970-01-01 00:00:00
34          2          123        2                     1                                                            1            1970-01-01 00:00:00
35          2          124        2                     1                                                            1            1970-01-01 00:00:00
36          2          125        2                     1                                                            1            1970-01-01 00:00:00
37          2          126        2                     1                                                            1            1970-01-01 00:00:00
38          2          127        2                     1                                                            1            1970-01-01 00:00:00
39          2          128        2                     1                                                            1            1970-01-01 00:00:00
40          2          129        2                     1                                                            1            1970-01-01 00:00:00
41          2          130        2                     1                                                            1            1970-01-01 00:00:00
42          2          131        2                     1                                                            1            1970-01-01 00:00:00
43          2          132        2                     1                                                            1            1970-01-01 00:00:00
44          2          133        2                     1                                                            1            1970-01-01 00:00:00
45          2          134        2                     1                                                            1            1970-01-01 00:00:00
46          2          135        2                     1                                                            1            1970-01-01 00:00:00
47          2          136        2                     1                                                            1            1970-01-01 00:00:00

sqlite> SELECT * FROM storage_pools;
pool_uid  node_type  pool_id  entity_type
--------  ---------  -------  -----------
2         2          1        3

With a .dump I can see these targets showing up in 2 tables, entities and targets. The node related tables look clean. I have a copy of mgmtd.sqlite to work on.

The command used to setup the storage targets was (taken out of a loop and indicating contents of variables):

inst_name="storage_g${gr5}"
conf_name="/etc/beegfs/${inst_name}.d/beegfs-storage.conf"
storage_dir="/data/beegfs/${inst_name}_a${num}"
serv_id=a service ID that is the same for all targets in a group (a single mode), counted up per systemd instance
trgt_id=a number unique across all targets in the node, counted up per target
/opt/beegfs/sbin/beegfs-setup-storage \
                        -c "$conf_name" \
                        -p "$storage_dir" \
                        -s "$serv_id" \
                        -i "$trgt_id" \
                        -m "host01"

I copy-pasted the line -s "$serv_id" and edited it to either "-i "$serv_id" or -s "$trgt_id" in a first execution. For a second execution I cleaned up and rebuilt all storage targets, because I got "UUID already registered" kind of errors and assumed this was a protection for double-registering the same disk twice.

Our set-up is currently a single-node setup with 4 meta-data targets and we organise the 4x8 storage targets into 4 different systemd units (instances) to be able to pin the processes to the CPUs attached to the disks (using systemctl set-property commands on individual service instances). The targets are 4+1 RAID5 arrays on NVMe disks and gr5 contains the number of a group of 8xRAID5 arrays.

Thanks for your help!

[iamjoemccormick]

Hi Frank,

Thank you for the details.

Before I scrap the entire system and start from scratch, I would really like to know if there is a procedure to clean up something like that. For now, its a test system with no data on it. In the future, if something like that happens, there should be a way to remove data that is the result of an aborted transaction. My actual expectation is that any administrative operation is of the type all-or-nothing and observations like the one I report here should be impossible.

I agree the mgmtd shouldn't let you get into this state. If it happens and there is no data or configuration you need to keep, the cleanest approach is to just delete the database, wipe the targets, and start from scratch so you can correctly initialize them with beegfs-setup-storage. In a system with no data, this is the quickest and safest approach.

That said, while we don't recommend manually modifying the contents of your database or targets without consulting with support - if starting from scratch was not an option, you could manually delete the offending entries from the database, then on your targets delete the targetID files (if they were created) and manually fix the targetNumID and nodeNumID files.

We should have enough data to recreate the issue so we can patch the mgmtd to avoid getting into this scenario in the first place. Its doing the right thing and preventing invalid system state, but we should do it more gracefully. I'll reach out if I need more info!

[rustybee42]

Hi Frank,

This inconsistency can happen because storage target registration is historically a two step process. Storage nodes send two separate requests for each target: RegisterTarget and MapTarget. After the first, targets are in in an "unmapped" state (they don't have a node assigned yet). This is what you see in your db, the node_id column being NULLs. The second message is sent immediately after, so usually, this isn't noticed. If there is an error while handling the second request or it isn't sent at all, the targets stay in that state. And most operations act like the targets without assigned nodes don't exist (looks like I overlooked the pool list command), also for historical reasons.

My actual expectation is that any administrative operation is of the type all-or-nothing and observations like the one I report here should be impossible

Totally agree, but for certain reasons it wasn't possible to change that old behavior when v8 was developed.

The current inconsistent behavior and not being able to clean this up needs to be addressed though, so I'll think about a fix. Unhiding unmapped targets from the list command and allowing their deletion seems the most straight forward solution.

[frsc-ku]

Hi Joe and Rusty Bee,

thanks for the additional info. For me it would be sufficient if the following was available/implemented:

• beegfs target CMD has a flag -a | --all to allow operating on all targets, including unmapped ones. Having a high-level way of doing such cleanups would be great, because the data base will remain consistent (important counters are updated along the way).
• beegfs health check has a check for unmapped targets (and maybe other things that could also be in a limbo state) and, if such are present, gives a hint to use the -a flag with beegfs target to view and clean up such objects.

This would remain downwards compatible and provide sufficient functionality with usual sanity checks to get out of such states in a reasonable way.

I will make a recovery attempt tomorrow and report back.

Best regards,
Frank

[frsc-ku]

My final report: working on a copy of the database, executing delete FROM targets where node_id is null; and rm /data/beegfs/storage_*/targetID the storage units started and the system is up and looks healthy (after stopping mgmtd, copy DB, starting mgmtd, start storage services).

Thanks for the help!

[iamjoemccormick]

Good to hear it!

On our end we have raised ThinkParQ/beegfs-rust#21 to make this issue easier to cleanup in the future. We also came up with a proposal to improve detection of duplicate node/target scenarios (when the same node/target ID is reused on accident) ThinkParQ/beegfs-rust#22.