Kkmattil sdsi version 2 #2743
Open
Kkmattil sdsi version 2 #2743
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sdsi instructions updated
sdsi-link added
galfthan
requested changes
Nov 15, 2025
Collaborator
galfthan
left a comment
galfthan
left a comment
There was a problem hiding this comment.
Additionally the new document should be spell checked, I did not do a thorough check for typos etc.
| * [Importing data ](./sd-desktop-access.md) | ||
| * [Exporting data via user interface](./sd-desktop-export.md) | ||
| * [Export data programmatically](./sd-desktop-export-commandline.md) | ||
| * [Using HPC resources for sensitive data](./tutorials/sdsi.md) |
Collaborator
There was a problem hiding this comment.
This addition is missing from https://csc-guide-preview.2.rahtiapp.fi/origin/kkmattil-patch-1/data/sensitive-data/sd-services-toc/ where the other bullets are repeated
| # Submitting jobs from SD Desktop to the HPC environment of CSC | ||
|
|
||
| The limited computing capacity of a SD Desktop virtual machines can prevent running heavy analysis tasks | ||
| for sensitive data. This document describes, how heavy computing tasks can be submitted form SD Desktop |
|
|
||
| Please note following details that limit the usage of this procedure: | ||
|
|
||
| * The service is not yet in full production. Access will be provided by a request for projects that have computing tasks that are compatible with the current status of the service. Contact [email protected] the enable this service. |
Collaborator
There was a problem hiding this comment.
Access can be requested for projects that have computing tasks that are compatible with the current status of the service.
| * The *sdsi-client* job submission tool, described in this document, will work only for the approved projects. | ||
| * Each job reserves always one, and only one, full Puhti node for your task. Try to construct your batch job so that it uses effectively all the 40 computing cores of one Puhti node. | ||
| * The input files that the job uses must be uploaded to SD Connect before the job submission. Even though the job is submitted from SD Desktop, you can't utilize any files from the SD Desktop VM in the batch job. | ||
| * The jobs submitted from SD Desktop to Puhti have higher security level that normal Puhti jobs but lower than that of SD Desktop. |
| The increased security of the computing tasks that are submitted with the sdsi-client is based on two main features: | ||
|
|
||
| 1. Both input and output data are stored and transported in encrypted format. Encryption is done automatically using the SD Connect methodology. | ||
| 2. For the actual analysis the data is temporarily decrypted to the local disk area of a compute node that is fully reserved for only this one job. Thus there can't be other users in the node during the execution of the job, which effectively eliminates the possibility that other users could access the disk areas, memory or process list of the node during the processing. All the data will be removed from local disk of the node when the job ends. |
Collaborator
There was a problem hiding this comment.
Thus there are no other users in the node during the execution of the job, and other users cannot access the disk areas, memory or process list of the node during the processing. The technical and operational measures that implements this key restriction are described in the Puhti TOMs (LINKKI, puuttuu vielä researchistä??)
| or few computing cores, you can use tools like _gnuparallel_, _nextfllow_ or _snakemake_ to submit several | ||
| computing tasks to be executed in the same time. | ||
|
|
||
| In the examples below we have a tar-arcvive file that has been stored to SD Connect: `2008749-sdsi-input/data_1000.tar.c4gh`. The tar file contains 1000 text files (_.txt_) for which we want to compute md5sum. Bellow we have three alternative ways to run the tasks so that all 40 cores are effectively used. |
|
|
||
| Please note following details that limit the usage of this procedure: | ||
|
|
||
| * The service is not yet in full production. Access will be provided by a request for projects that have computing tasks that are compatible with the current status of the service. Contact [email protected] the enable this service. |
Collaborator
There was a problem hiding this comment.
We should here link to TOMs and also say that you should use them to evaluate of the service is sufficient?
Contributor
Author
There was a problem hiding this comment.
We have link to toms in the page.
Link to Puhti TOMs added.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Briefly describe the changes you've made here. Remember to add a link to the preview page of your branch.
Checklist before requesting a review