How to migrate data/files from one storage target to another?

[manideep2510]

Hello!

I am looking for some guidance on how one can migrate data from one storage target to another. A couple of scenarios for this are: we want to remove and upgrade an old SSD to a new one, or move data from a slow SSD to a faster one.

The one relevant BeeGFS CLI option I can see is beegfs entry migrate. However, this is not for storage targets, but for storage pools, which need an enterprise license to work with.

Is there a simpler method to migrate data from one storage target to another without using enterprise features such as storage pools?

I appreciate any help you can provide.

[scaleoutsean]

Add new disk, remove old disk

[manideep2510]

Thanks!

This page points to the Data Migration page, which suggests to use the following,

beegfs entry migrate --from-targets=<targetID>,[targetID] --pool=<destinationPoolID> /mnt/beegfs/

My confusion was, if I only have a single storage pool (which I assume is the case for non-enterprise BeeGFS), and all storage targets are in this single pool, how would migrate work? Would it automatically move the data from targetID target to other targets in the same storage pool and ensure that the data is not transferred back to the original target even though it's in the same pool?

A tangential question: Is there a way to control and move data from one storage target to another specific storage target instead of arbitrary targets from a storage pool? This is relevant to the use case where we might need to move data to a new, faster SSD for example.

[scaleoutsean]

So, yes, pool is requires. That's weird. Reading TFM I thought it'd behave and in the absence of --pool $POOL just evacuate entries elsewhere (--from-targets $TARGETS to not $TARGETS) as the source implies).

Well, turns out pool is required.

$ beegfs entry migrate --help

Flags:

      --pool entityId                              Migrate files to the targets and buddy groups in this storage pool. (default <unspecified>)

That's disappointing. In absence of that, I'd:

• Set free space on the unwanted target to 0
• Get verbose entry info to get a list of files use that target
• If one server has one target, set preferred servers to exclude the one with "old" storage target, then read all files and write them back to BeeGFS.
• If one server with "old" target has multiple targets, maybe you could take that earlier step of (mis)informing BeeGFS that the old storage target is full. Then just read and write back all files that have chunks on the old storage target. That's documented here

There ought to be a mechanism (and associated Beegfs CLI command) for there above, really. Even better, since it almost already exists (beegfs entry migrate), it should just behave properly: when --from-targets $TARGETS is set, then it should move all files to not $TARGETS as I've mentioned at the top of this comment. If I want to migrate --from-targets and don't specify anything else, then it's 100% clear I want to remove files from these targets, period.
But, if they're afraid of confusion, maybe a new command could be created (beegfs target migrate --old $TGT1 [--new $TGT2] where w/o --new files would be moved anywhere else, and with, to specified target(s)).

A server can be removed and added back (with the new target), but its storage targets must be emptied first, so that doesn't seem helpful.

[scaleoutsean]

A tangential question: Is there a way to control and move data from one storage target to another specific storage target instead of arbitrary targets from a storage pool?

There are striping patterns which aren't meant for that. As the docs mention, BeeGFS behaves well in the sense that it seeks to pick less capacity-utilized targets. You could abuse this behavior by setting free space to 0 to all targets but the one you want to use (maybe combine with custom striping), but I wouldn't try that in a production environment.

[manideep2510]

Thanks for the pointers!

Setting free space is a decent option, but it's hacky and not a nice thing to do in a production system, like you also pointed out.

What would be nice is:

1. Ability to set the destination storage target using beegfs entry migrate itself or with some other command. The default behavior could be as is, as it is right now, if no destination target is mentioned.
2. Ability to set a storage target to a folder using beegfs entry set. Currently using beegfs entry set, we can set striping pattern, storage pool, and also "Remote" Storage Target, but not a normal storage target.
3. Ability to move a specific file or directory to a specific storage target manually.

[manideep2510]

Hi @iamjoemccormick
Would you be able to help with this or add your thoughts? Thanks!

[iamjoemccormick]

For the scenario where you want to drain data from a target and move data to a new one, the proposal by @scaleoutsean to prevent new file allocations on a storage target with the free_space.override option then run migrate is currently the recommended approach.

For scenarios where you want to migrate some data from slow to fast SSDs, that is precisely what storage pools are designed for :)

The reason --pool is required for migrate is because files always need to be assigned to some pool. In a system without an enterprise license that is just the default pool (1). In systems with multiple pools it becomes confusing if we allow files to be migrated to targets that differ from their pool assignment.

While I like the proposal to just migrate entries --from-targets $TARGETS to not $TARGETS that is not so trivial to do with how target assignments work in BeeGFS. Right now migrate relies on the metadata service to decide on new target assignments for migrated files. The destination target list is filtered based on the --pool, and the assignment algorithm depends on the meta’s tuneTargetChooser parameter and takes into account target utilization (i.e., prefer less used targets over fuller ones).

It would be possible for CTL to accept a --to-targets list (or simply not $TARGETS) and place migrated files on specific targets. But then we have to be careful to avoid imbalances target assignments (i.e., the first few targets ending up with all the data) or migrating files to nearly full targets. The meta already handles all this so I’m not too keen on making CTL responsible for target assignment when migrating files.

So as it stands I still believe the best approach to migrate data away from specific targets in systems with a single pool is to set the free_space.override option on any target(s) you want to migrate data away from before running migrate. That also ensures any new files are also not created on those target(s).

[iamjoemccormick]

An update on this. Upon closer examination, while technically files have a pool assignment in their inode, this has no significance after file creation. Pools only matter for directories because they control what targets new files in that directory are assigned to. In BeeGFS 7 we didn't even print pool assignments for files in entry info, this inadvertently slipped into the BeeGFS 8 table output.

So we decided to make two changes to CTL:

(1) Stop printing pools for files in entry info. This is confusing because the targets assigned to a pool might change after a file is created, meaning the targets assigned to a file may eventually no longer be in the same pool assigned to the file. This has always been the intended behavior to provide flexibility when managing pools.

(2) Add new --targets and --groups flags to entry migrate that allow migrating files to specific destination targets+groups. To avoid imbalanced utilization, CTL will randomly select targets/groups for each file from the provided list. CTL will not take target utilization into account, but given the user is explicitly specifying targets/groups this is probably more in line with what the user intends/expects anyway. Most people are probably checking target utilization before running the migration and intentionally selecting the destination targets/groups.

This new behavior is scheduled to roll out in the next release, along with some other enhancements to the migrate mode.

[manideep2510]

This will be a great inclusion, @iamjoemccormick! Thanks for the update!!