followup with Matteo tenti and others on INFN tape request

[StevenCTimm]

They are preparing the resource request at INFN for tape in particular, they seem ready to work with us.

[StevenCTimm]

Lucia Morganti commented:

Hi Matteo and Steve,

a Tape RSE could be exposed to grid resources using https protocol, and the same authenticaion/authorization mechanisms supported for disk storage areas (that it group or scope based authz using JWT tokens, and/or VOMS proxies based on the X.509 standard for SSL certificates).

In detail, two different middleware components would be used:

StoRM WebDAV to write files to the disk buffer (migration to tapes happens afterwards, automatically), and to read files from disk buffer after files are staged
StoRM Tape REST API to handle bulk stage requests, to track the progress and cancel previous stage requests, to access the metadata of the staged files.
Here follows our Tape Policy: although the interface is very similar to that for disk storage areas, tape storage areas require even more special care:

do not write small files, i.e., smaller than 100 MB on tape; uploads of files smaller than 128KB will result in a client error (400)
avoid complex directory trees with a high number of directories (for example, never use the “hash” LFN2PFN algorithm for a Rucio Storage Element);
do not modify files already on tape;
do not delete files on tape - it is important to carefully choose ab initio which files to write to this type of storage area;
do not stage-in (recall) files individually, but organise stage‑in processes in bulk;
read or transfer the files as soon as they are staged to the buffer, i.e. before their pin time expires.
Cheers,
lucia

[StevenCTimm]

Please add my colleague Wenlong Yuan to the notice of the ticket.

With response to the protocols, 

"a Tape RSE could be exposed to grid resources using https protocol, and the same authenticaion/authorization mechanisms supported for disk storage areas (that it group or scope based authz using JWT tokens, and/or VOMS proxies based on the X.509 standard for SSL certificates).
In detail, two different middleware components would be used:
StoRM WebDAV to write files to the disk buffer (migration to tapes happens afterwards, automatically), and to read files from disk buffer after files are staged
StoRM Tape REST API to handle bulk stage requests, to track the progress and cancel previous stage requests, to access the metadata of the staged files."

---

We should be able to use both of these components.   We have a couple follow up questions and comments

Have any other VO's successfully used FTS3 and Rucio to trigger staging using the StoRM Tape REST API yet?  IF so, was it using X.509 or with tokens?  (i.e. does StoRM support the storage.stage and storage.poll token scopes as yet).
What is the approximate size of the disk buffer that would be usable to DUNE for tape writes...alternatively what would be the advisable amount of tape files to try to write, or restore from tape, at once?
The 100MB or greater  file size is no trouble, this is a DUNE wide policy already.  Mean file size now is 2-4GB and it is expected to get bigger.
For tape installations we have a custom generated directory path which is not the default rucio "hash"
 We typically stage files back from tape on a dataset by dataset basis, where datasets are <=10000 files in size.
 We should consider if we want to enable all the various x.509 and token subjects or just certain ones that are production ones.

Steve Timm