S3 backup design doc (NESE)

[CodyCBakerPhD]

NESE Tape analog/duplicate of #2627

Based only on rough estimates at this point; need to follow up and get a more concrete commitment before

TL;DR though, somewhere between 1/4-1/3 of the cost of the S3 Replication/Deep Glacier approach

[mvandenburgh]

Do we know this for sure? AWS would probably take notice if our data transfer costs suddenly started spiking on a recurring basis due to a backup script downloading the entire multi-PB bucket.

As far as I understand, this isn't a technical limitation of the open data account; from our perspective, the billing for the entire AWS account is paid for by the AWS Open Data program. So, we could apply the same thinking to the Deep Glacier approach - if we put the backup bucket in the open data AWS account, the storage costs would also theoretically be covered by the open data program. In practice, it's more of a policy issue - is AWS willing to pay for backup? Framed that way, I don't think there is a clear winner between the two approaches in terms of AWS transfer costs.

[CodyCBakerPhD]

Do we know this for sure?

Yes.

Historically we have performed identical backup strategies (not in theory; in practice) for Dartmouth (DropBox) and MIT (ORCD/Engaging).

AWS would probably take notice if our data transfer costs suddenly started spiking on a recurring basis due to a backup script downloading the entire multi-PB bucket.

IDK if they notice or care; but what would they do? Back-charge the cost to us after the fact? IDT they can even do that (legally).

In general this is a 'fair use' of the open data program.

As far as I understand, this isn't a technical limitation of the open data account; from our perspective, the billing for the entire AWS account is paid for by the AWS Open Data program. So, we could apply the same thinking to the Deep Glacier approach - if we put the backup bucket in the open data AWS account, the storage costs would also theoretically be covered by the open data program.

However, our sponsorship (or at least other sponsorships such as EMBER/BossDB) come with expectations and allowed total sizes; it is not a free account for us to just do whatever we want. [If it was, why wouldn't we run the Hub and all other compute through that account?]

I haven't seen the DANDI contract stipulations, but from what Satra says about it and from what EMBER/BossDB and others have said, the main restriction is total size. For example, for EMBER, we currently cannot exceed 2.6 PB without approval.

To use the same account to make a bucket of equivalent size (regardless of storage tier, which again, they did not seem to care much about) would be to double the total used size - or in other words - halve the maximum possible size of the archive.

In practice, it's more of a policy issue - is AWS willing to pay for backup? Framed that way, I don't think there is a clear winner between the two approaches in terms of AWS transfer costs.

If AWS said they were willing to pay for it all (consider me doubtful - keep in mind for them such costs are largely ephemeral since they indeed SET their own prices), it would be the best option by far because it is just a couple of buttons for us to press one time and no extra cost thereafter.

[mvandenburgh]

I think there are two additional considerations worth thinking about here.

1. If we're considering a cost basis purely based on monthly bills for cloud services, the NESE backup approach is clearly cheaper. But, as mentioned further up in the document, there is likely going to be some manual work to perform the transfer. This will consume engineering time and add to the maintenance burden of the system much more than the Deep Glacier approach; outside of the initial work to set it up and do the initial transfer, the Deep Glacier approach is largely hands-off and automatic.

2. One of the motivations for backup is obviously data security - the data collected on DANDI is the most valuable part of the project, and any data loss bugs could be catastrophic for the project as a whole. This is one of the reasons Add Asset garbage collection design doc #2367 is currently on hold; we want to get an ironclad backup system into place before we start deleting "garbage" from the bucket, in case there is a bug in our code. With the Deep Glacier design, there is essentially no hand-rolled code being run - it is simply configuring various options in AWS to facilitate backup and then deferring the actual backup/replication logic to AWS. With the NESE design, there would presumably need to be at least some custom code/scripts written to facilitate the download of data from S3 followed by the upload to Dartmouth's system; this theoretically leaves more room for bugs to sneak in.

Note, I'm not necessarily arguing for or against either design at this point (I haven't read this enough times/thought it through enough yet), I just want to make sure these points are discussed.

[CodyCBakerPhD]

If we're considering a cost basis purely based on monthly bills for cloud services, the NESE backup approach is clearly cheaper. But, as mentioned further up in the document, there is likely going to be some manual work to perform the transfer. This will consume engineering time and add to the maintenance burden of the system much more than the Deep Glacier approach; outside of the initial work to set it up and do the initial transfer, the Deep Glacier approach is largely hands-off and automatic.

As mentioned in certain parts of the document and PR; we do need to get a solid commitment from Dartmouth IT, but from conversations they indicated it would be covered by them, so 'automatic for us'.

Even at a minimum it would compare to the work I've done with the MIT backup (a similar approach; move the data from S3 to a local HPC and from there the tape copy process can begin). Not a huge engineering effort and has been done before so not starting from scratch

One of the motivations for backup is obviously data security - the data collected on DANDI is the most valuable part of the project, and any data loss bugs could be catastrophic for the project as a whole. This is one of the reasons #2367 is currently on hold; we want to get an ironclad backup system into place before we start deleting "garbage" from the bucket, in case there is a bug in our code. With the Deep Glacier design, there is essentially no hand-rolled code being run - it is simply configuring various options in AWS to facilitate backup and then deferring the actual backup/replication logic to AWS. With the NESE design, there would presumably need to be at least some custom code/scripts written to facilitate the download of data from S3 followed by the upload to Dartmouth's system; this theoretically leaves more room for bugs to sneak in.

Yes, this is true - I'll add it into the doc

However, has anyone ever rigorously tested the claimed S3 restoration services? One might speculate that bugs could also exist in there, but no one (a) could know about it ahead of time outside of Amazon, and (b) would not know about it until it is 'too late' for us because we need it

[CodyCBakerPhD]

Replaced by #2684 which has all final details solidified with NESE (via Dartmouth) and ORCD

Quick link: https://github.com/CodyCBakerPhD/dandi-archive/blob/all_backup/doc/design/all_backup_options.md