diff --git a/source/conf.py b/source/conf.py index fa006749b..497a27e22 100644 --- a/source/conf.py +++ b/source/conf.py @@ -69,9 +69,9 @@ # built documents. # # The short X.Y version. -version = u'4.0' +version = u'4.1' # The full version, including alpha/beta/rc tags. -release = u'4.0.0' +release = u'4.1.0' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. diff --git a/source/customizations.rst b/source/customizations.rst index 725b82fa6..b51b3747b 100644 --- a/source/customizations.rst +++ b/source/customizations.rst @@ -96,7 +96,7 @@ If the announcement file has the extension ``yml`` and is a YAML file it is firs - Description * - ``type`` - The type of announcement. Values can be ``warning``, ``info``, ``success``, or ``danger``. - * - ``msg`` + * - ``message`` - The announcement's message. * - ``id`` - Optional unique identifier for the announcement. This is useful for managing changes to announcements that are used as ToS or EULA that users need to agree to. @@ -121,7 +121,7 @@ Because the announcement is rendered via ERB you can do some interesting things, .. code-block:: erb type: warning - msg: | + message: | <% if Time.now < Time.new(2018, 9, 24, 12, 0, 0) %> A **Ruby Partial Downtime** for 4 hours on Monday, September 24 from 8:00am to 12:00pm will prevent SSH login to Ruby nodes and Ruby VDI sessions. @@ -463,20 +463,33 @@ If you want to disable file upload altogether, set ``FILE_UPLOAD_MAX`` to 0 and the ``nginx_file_upload_max`` configuration alone (or comment it out so the default is used). +.. _set_download_limits: + Set Download Limits ------------------- -By default, the maximum file download size is 10.7 GB (10737418240 bytes). -If you wish to change this, you can set the ``OOD_DOWNLOAD_DIR_MAX`` configuration environment -variable in the ``/etc/ood/config/apps/dashboard/env`` file to the desired value in bytes. +By default, the maximum download size for files and directories is 10.7 GB (10737418240 bytes). +If you wish to change this, you can set the ``OOD_DOWNLOAD_DIR_MAX`` and/or ``OOD_DOWNLOAD_FILE_MAX`` +configuration environment variables in the ``/etc/ood/config/apps/dashboard/env`` file to the desired +value for directories and files, respectively. Both variables can also be set in your :ref:`ondemand-d-ymls` +file as ``download_dir_max`` and ``download_file_max``. -For example, to set the limit to 5 GB, you can add the following line to the ``/etc/ood/config/apps/dashboard/env`` file: +For example, to set the directory limit to 8 GiB and the file limit to 5 GiB, you can add the following +lines to the ``/etc/ood/config/apps/dashboard/env`` file: .. code-block:: - OOD_DOWNLOAD_DIR_MAX=5368709120 + OOD_DOWNLOAD_DIR_MAX=8589934592 + OOD_DOWNLOAD_FILE_MAX=5368709120 + +Note that this will limit the download size for all users of the Open OnDemand instance. If you want +to set it as part of your :ref:`profiles_guide` to affect only a subset of users, you can add the +the following lines to a specific profile in your :ref:`ondemand-d-ymls`. + +.. code-block:: -Note that this will limit the download size for all users of the Open OnDemand instance. + download_dir_max:8589934592 + download_file_max:5368709120 .. warning:: This configuration value is expected to be numbers only (no characters) @@ -599,6 +612,11 @@ connection alive is not factored into this. of activity (in milliseconds). After this duration, the connection will be closed regardless of activity. Its default is 3600000 milliseconds (1 hour). +``OOD_SHELL_TERM`` allows the default ``TERM=xterm-16color`` to be overridden. Should only +be set to ones whose escape codes are fully supported by the underlying +`hterm library `__. +Some other common ones that are include ``xterm``, ``xterm-256color``, and ``xterm-direct``. + .. code:: shell # /etc/ood/config/apps/shell/env @@ -903,11 +921,18 @@ of 4.0.8 the existing widgets are: - ``pinned_apps`` - Pinned Apps described above - ``recently_used_apps`` - the four most recently used interactive applications. Launching these applications will start a new interactive session with the previously submitted parameters. + Hovering over the apps will show these parameters for attributes with the :ref:`display option ` set to true. - ``sessions`` - the three most recent active interactive sessions - ``motd`` - the Message of the Day - ``xdmod_widget_job_efficiency`` - the XDMoD widget for job efficiency - ``xdmod_widget_jobs`` - the XDMoD widget for job information - ``saved_settings`` - all of the batch connect saved settings displayed as cards +- ``balances`` - Current users' balances if you've configured :ref:`balance warnings on the dashboard `. + Warnings have a threshold to determine if it shows at all. This widget does not. It will show all current balances. +- ``file_quotas`` - Current users' file quotas if you've configure :ref:`disk quota warnings on the dashboard `. + Warnings have a threshold to determine if it shows at all. This widget does not. It will show all current file quotas. +- ``nsf_access_events`` - A list of upcoming NSF ACCESS events with registration links. +- ``system_status`` - The system status page as a dashboard widget instead of a dedicated page. This feature also allows for administrators to *add* custom widgets. Simply drop new files into ``/etc/ood/config/apps/dashboard/views/widgets`` and reference them @@ -1033,6 +1058,8 @@ The OSC default value for ``options_account_help`` says that the account field i Items of note include what to call Accounts which might also be Charge Codes, or Projects. At OSC entering an account is optional unless a user is a member of multiple projects which is reflected in the default value for the string ``options_account_help``. +.. _disk_quota_warnings: + Disk Quota Warnings on Dashboard -------------------------------- diff --git a/source/customizations/support-ticket.inc b/source/customizations/support-ticket.inc index 145253c50..d28bb5b16 100644 --- a/source/customizations/support-ticket.inc +++ b/source/customizations/support-ticket.inc @@ -3,54 +3,125 @@ Support Tickets --------------- -The Dashboard now supports sending a support ticket to your institution Help Desk system. -The feature uses email as the delivery mechanism, but it could be extended to support others. +The Dashboard enables users to easily submit support requests directly to your institution's Help Desk system. +This feature provides a customizable form that collects user input and sends the information to the designated support system. To enable this feature, define the settings needed using the configuration property ``support_ticket``. Once this configuration object is defined, the ``Submit Support Ticket`` link will be shown in the ``Help`` navigation menu and in all interactive session cards. +Quick Start +........... + +The simplest configuration to get started with email-based support tickets: + +.. code-block:: yaml + + # /etc/ood/config/ondemand.d/support_ticket.yml + support_ticket: + email: + to: "support@example.edu" + +This minimal configuration will: + +* Enable the support ticket feature +* Display the default form with all standard fields +* Send tickets via email to the specified address +* Use the default SMTP configuration from Rails + +For production use, you'll likely want to customize the form, add SMTP settings, and possibly use a different delivery mechanism. See the sections below for detailed configuration options. + .. figure:: /images/support_ticket_menu.png :align: center -The support ticket page has a simple form to gather information that will be submitted to the support system via email. -If triggered from a session card, the system will automatically add information about the selected -session to the body of the email to help troubleshoot the issue. +.. raw:: html + +

+ +The support ticket page has a simple form to gather information that will be submitted to the support system. +The form can be pre-populated with contextual information in two ways: + +1. **From Interactive Session Cards**: When triggered from a BatchConnect session card, the system automatically includes session details (session ID, title, status, creation time) to help troubleshoot the issue. +2. **From Job Information**: When triggered with job context, the system can include job details (job ID, cluster, status, submission time). + +The support ticket URL accepts these optional query parameters: + +* ``session_id``: UUID of a BatchConnect session to include in the ticket +* ``job_id``: Job ID to include in the ticket (requires ``cluster`` parameter) +* ``cluster``: Cluster name where the job is running +* ``queue``: Pre-select a specific queue for Request Tracker systems + +Example URLs: + +.. code-block:: bash + + # With session context + https://ondemand.example.edu/support?session_id=abc-123-def + + # With job context + https://ondemand.example.edu/support?job_id=12345&cluster=owens + + # With queue selection + https://ondemand.example.edu/support?queue=General .. figure:: /images/support_ticket_form.png :align: center -A brief description of the default form fields used in the support ticket form: +.. raw:: html + +
+ +Supported Delivery Mechanisms +............................. + +Open OnDemand supports the following delivery mechanisms for support tickets: + +* **Email** - Uses SMTP to send tickets via email to your help desk system +* **Request Tracker API** - Creates tickets directly in Request Tracker systems +* **ServiceNow API** - Creates incidents directly in ServiceNow systems + +Default Form Fields +................... + +The support ticket form includes these default fields: + +* **Username:** The logged-in username (read-only). Added to the ticket body for reference. +* **Email:** Required. Email address for communication about this ticket. A single email address is supported. +* **CC:** Optional. Additional email address to copy on ticket communications. A single email address is supported. +* **Subject:** Required. Brief description of the problem. +* **Attachments:** Optional. Upload files (e.g., screenshots) to help with debugging and troubleshooting. Default limits: 4 files, 6MB per file. +* **Description:** Required. Detailed description of the problem. +* **Session ID:** Hidden field. Automatically populated when launched from a session card. +* **Session Description:** Read-only field. Displays session information when a session ID is present. +* **Job ID:** Hidden field. Automatically populated when job context is provided. +* **Cluster:** Hidden field. Automatically populated when job context is provided. +* **Job Description:** Read-only field. Displays job information when job context is present. +* **Queue:** Hidden field. For Request Tracker systems to pre-select a queue. -* **Username:** Logged in user. Username will be added to the support ticket body for reference. -* **Email:** Email address to communicate regarding this support ticket. A single email address is supported. -* **CC:** Additional email address to communicate regarding this ticket. A single email address is supported. -* **Subject:** Brief description of the problem. -* **Attachments:** Add screenshots of the problem to help with debugging and troubleshooting. -* **Description:** Detail description of the problem. +Email +..... -Configuration -............. +The Email delivery mechanism uses SMTP to send support tickets to your help desk system. This was the first implementation and requires email configuration. -These are the most common configuration properties needed to enable and setup the support ticket feature: +These are the configuration properties for email delivery: -* ``description``: Text to customize the support ticket header. -* ``email``: Section to configure how the email is sent to the support system. -* ``to``: The destination email address of your Help Desk. -* ``from``: The address to set in the from field when sending the email. It defaults to the email submitted in the support ticket form. - This is useful to prevent the incoming emails being considered spam. -* ``delivery_method``: Rails setting to set the transport mechanism to use. This is usually ``smtp``. - For more information see the `Rails documentation on mailers `_. -* ``delivery_settings``: Rails settings object to set the transport configuration properties. - See example below or the `Rails documentation on mailers `_. +* ``description``: Optional. Text to customize the support ticket header. +* ``ui_template``: Optional. Custom view template name. Defaults to ``email_service_template``. +* ``email``: Required section to configure how the email is sent to the support system. + + * ``to``: Required. The destination email address of your Help Desk. + * ``from``: Optional. The address to set in the from field when sending the email. Defaults to the email submitted in the support ticket form. This is useful to prevent the incoming emails being considered spam. + * ``delivery_method``: Optional. Rails setting to set the transport mechanism to use. This is usually ``smtp``. For more information see the `Rails documentation on mailers `_. + * ``delivery_settings``: Optional. Rails settings object to set the transport configuration properties. See example below or the `Rails documentation on mailers `_. + * ``success_message``: Optional. Custom message to display when ticket is successfully created. Defaults to a message showing the recipient address. .. warning:: - Use caution when supplying username and password in delivery_settings. These files are readable by + Use caution when supplying username and password in delivery_settings. These files are readable by unprivileged users and as such this information can be found by a sophisticated user without privilege escalation. -Sample configuration: +Sample email configuration: .. code-block:: yaml @@ -71,7 +142,7 @@ Sample configuration: # Text added to the page under the support ticket header description: | My optional description Text for the support ticket feature - + # email section is required and should always be present. # It configures how the support ticket email is sent email: @@ -106,8 +177,19 @@ In the same way that you can configure the :ref:`form for interactive applicatio the support ticket form fields can be arranged or changed as required. .. note:: - - Please note that the ``email`` field must be present in the form for the support ticket feature to be functional. + + The ``email`` field must be present in the form for the support ticket feature to be functional, as it is used for communication with the user. + +Available field attributes: + +* ``required``: Boolean. Mark field as required for validation. +* ``readonly``: Boolean. Display field value but prevent editing. +* ``disabled``: Boolean. Disable the field. +* ``hide_when_empty``: Boolean. Hide field when it has no value. +* ``widget``: String. Specify the field type (``email_field``, ``text_area``, ``hidden_field``, ``file_attachments``). +* ``value``: Set a default or fixed value for the field. Supports ERB templating (e.g., ``<%= CurrentUser.name %>``). +* ``fixed``: Boolean. When combined with ``value``, makes the value unchangeable. +* ``rows``: Integer. For text areas, specify the number of rows. Custom fields can be added, but they will require a custom email template to make use of the provided values in the email body. Override the default email template with your own by dropping a file named ``_email.text.erb`` into the folder @@ -123,7 +205,7 @@ place for removing and/or adding fields. attributes: username: - value: "<%= CurrentUser.name %>" + required: true readonly: true email: widget: email_field @@ -137,6 +219,13 @@ place for removing and/or adding fields. session_description: hide_when_empty: true disabled: true + job_id: + widget: hidden_field + cluster: + widget: hidden_field + job_description: + hide_when_empty: true + disabled: true attachments: widget: file_attachments description: @@ -153,6 +242,9 @@ place for removing and/or adding fields. - subject - session_id - session_description + - job_id + - cluster + - job_description - attachments - description - queue @@ -188,3 +280,306 @@ The example below shows a custom form configuration with 3 fields and how to use .. figure:: /images/support_ticket_custom_form.png :align: center + +Request Tracker API +................... + +The Request Tracker API delivery mechanism creates tickets directly in Request Tracker systems. + +These are the supported configuration options: + +* ``description``: Optional. Text to customize the support ticket header. +* ``ui_template``: Optional. Custom view template name. Defaults to ``email_service_template``. +* ``rt_api``: Required section for Request Tracker configuration. + + * ``server``: Required. The URL of the Request Tracker server. + * ``auth_token``: The value for the API authentication token. Required when ``Authorization: Token `` is used. + * ``user``: The user value when HTTP basic authentication is used. + * ``pass``: The password value when HTTP basic authentication is used. + * ``priority``: Required. The priority value for all tickets. + * ``queues``: Required. Array of allowed queues to submit tickets. By default, the first queue in the array will be used. + * ``template``: Optional. Custom ERB template for ticket content. Defaults to ``rt_ticket_content.text.erb``. + * ``success_message``: Optional. Custom message to display when ticket is successfully created. + * ``timeout``: Optional. The maximum time in seconds that the client should wait for responses from the server. Defaults to ``30``. + * ``verify_ssl``: Optional. Whether to verify SSL certificates. + + .. code-block:: yaml + + support_ticket: + rt_api: + server: "http://rt:34000" + auth_token: "auth_token" + user: "root" + pass: "password" + priority: 4 + queues: + - "General" + - "Support" + template: "custom_rt_template.text.erb" + success_message: "Your ticket has been created successfully." + timeout: 10 + verify_ssl: true + +.. _servicenow_support_ticket: + +ServiceNow API +.............. + +The ServiceNow API delivery mechanism creates incidents directly in ServiceNow systems. + +These are the supported configuration options: + +* ``description``: Optional. Text to customize the support ticket header. +* ``ui_template``: Optional. Custom view template name. Defaults to ``email_service_template``. +* ``servicenow_api``: Required section for ServiceNow configuration. + + * ``server``: Required. The URL of the ServiceNow server. + * ``auth_header``: Optional. The name for the API authentication header. Defaults to ``x-sn-apikey``. + * ``auth_token``: The value for the API authentication key. Required when API key authentication is used. + * ``auth_token_env``: Optional. Environment variable name containing the API authentication key. Takes precedence over ``auth_token`` if set. + * ``user``: The user value when HTTP basic authentication is used. + * ``pass``: The password value when HTTP basic authentication is used. + * ``pass_env``: Optional. Environment variable name containing the password. Takes precedence over ``pass`` if set. + * ``timeout``: Optional. The maximum time in seconds that the client should wait for responses from the server. Defaults to ``30``. + * ``verify_ssl``: Optional. Whether to verify SSL certificates. Defaults to ``true``. + * ``proxy``: Optional. Proxy server URL to use for the connection. + * ``map``: Optional. Map of ServiceNow incident fields to support ticket form field names. The value can be a single field name or an array of field names (values will be concatenated with commas). + * ``payload``: Optional. Map of ServiceNow incident fields to static values. Use ``nil`` as the value to dynamically pull from the form field with the same name. + * ``template``: Optional. Custom ERB template for incident description. Defaults to ``servicenow_content.text.erb``. + * ``success_message``: Optional. Custom message to display when ticket is successfully created. + +.. note:: + + For security, use ``auth_token_env`` and ``pass_env`` to store credentials in environment variables rather than directly in configuration files. + +**ServiceNow Field Mapping** + +The ServiceNow implementation builds the incident payload in three stages: + +1. **Default mappings** (always included): + + * ``email`` → ``caller_id`` + * ``cc`` → ``watch_list`` + * ``subject`` → ``short_description`` + * ``description`` → ``description`` (formatted using template) + +2. **Custom mappings via** ``map`` (adds to or overrides default mappings): + + Maps form field values to ServiceNow incident fields. The configuration key is the ServiceNow field name, and the value is either: + + * A single form field name (string) + * An array of form field names (values will be joined with commas) + +3. **Static/dynamic values via** ``payload`` (final stage, highest precedence): + + Sets ServiceNow incident fields to specific values. For each key-value pair: + + * If the value is NOT ``nil``: uses the static value provided + * If the value IS ``nil``: dynamically pulls the value from the form field matching the key name + +Example showing all three stages: + + .. code-block:: yaml + + support_ticket: + servicenow_api: + server: "https://servicenow.server.com" + auth_header: "x-sn-apikey" + auth_token_env: "SERVICENOW_API_KEY" + + # Stage 2: Map form fields to ServiceNow incident fields + # These add to or override the default mappings + map: + # Override default caller_id mapping (normally uses 'email') + caller_id: "username" + # Map a single form field + u_email_source: "email" + # Map multiple form fields - values joined with commas + u_affected_users: ["username", "email"] + + # Stage 3: Set static or dynamic values + # These have the highest precedence + payload: + # Static values: always use these exact values + u_category: "troubleshooting" + contact_type: "External System" + assignment_group: "IT Support Team" + u_business_service: "Research Computing" + + # Dynamic values: use nil to pull from form field with same name + # This would use support_ticket.priority if that field exists + priority: nil + + # You can also override default mappings here with static values + # This would ignore the user's email and always use this value + # caller_id: "system@example.edu" + + template: "custom_servicenow_template.text.erb" + success_message: "Your ServiceNow incident has been created." + +The resulting ServiceNow incident would have: + +* ``caller_id``: Value from ``username`` form field (overridden by ``map``) +* ``watch_list``: Value from ``cc`` form field (default mapping) +* ``short_description``: Value from ``subject`` form field (default mapping) +* ``description``: Formatted using template (default mapping) +* ``u_email_source``: Value from ``email`` form field (added by ``map``) +* ``u_affected_users``: "username,email" (added by ``map``) +* ``u_category``: "troubleshooting" (static value from ``payload``) +* ``contact_type``: "External System" (static value from ``payload``) +* ``assignment_group``: "IT Support Team" (static value from ``payload``) +* ``u_business_service``: "Research Computing" (static value from ``payload``) +* ``priority``: Value from ``priority`` form field if it exists (dynamic from ``payload``) + +Profile-Based Configuration +........................... + +Support tickets can be configured differently based on user profiles. This is useful for multi-tenant environments where different groups need tickets routed to different support teams. + +.. code-block:: yaml + + # Global default configuration + support_ticket: + email: + to: "general-support@example.edu" + + # Profile-specific configurations + profiles: + research_team: + support_ticket: + email: + to: "research-support@example.edu" + description: "Contact the Research Computing support team" + + teaching_team: + support_ticket: + servicenow_api: + server: "https://teaching.service-now.com" + auth_token_env: "TEACHING_SNOW_TOKEN" + payload: + assignment_group: "Teaching Support" + +Users will use the profile-specific configuration if they have selected that profile or if hostname-based profile matching is enabled. + +Custom Templates +................ + +All delivery mechanisms support custom ERB templates for formatting ticket content: + +**Email Templates** + +Override the default email template by creating: + +.. code-block:: bash + + /etc/ood/config/apps/dashboard/views/support_ticket/email/_email.text.erb + +**Request Tracker Templates** + +Customize RT ticket content by creating: + +.. code-block:: bash + + /etc/ood/config/apps/dashboard/views/support_ticket/rt/rt_ticket_content.text.erb + +**ServiceNow Templates** + +Customize ServiceNow incident description by creating: + +.. code-block:: bash + + /etc/ood/config/apps/dashboard/views/support_ticket/servicenow/servicenow_content.text.erb + +Templates have access to a ``context`` object containing: + +* ``context.support_ticket``: The support ticket form data +* ``context.session``: BatchConnect session object (if applicable) +* ``context.job``: Job information object (if applicable) + +Example custom template: + +.. code-block:: erb + + Problem Report from <%= context.support_ticket.username %> + + Email: <%= context.support_ticket.email %> + Subject: <%= context.support_ticket.subject %> + + Description: + <%= context.support_ticket.description %> + + <% if context.session %> + Session Information: + - Title: <%= context.session.title %> + - Job ID: <%= context.session.job_id %> + - Status: <%= context.session.status %> + <% end %> + + <% if context.job %> + Job Information: + - Job ID: <%= context.job.id %> + - Cluster: <%= context.job.cluster %> + - Status: <%= context.job.status %> + <% end %> + +Delivery Mechanism Precedence +............................. + +When multiple delivery mechanisms are configured, Open OnDemand follows a specific precedence order: + +1. **Email** (highest precedence) +2. **Request Tracker API** +3. **ServiceNow API** (lowest precedence) + +The system will use the first available mechanism in this order. For example, if both Email and ServiceNow are configured, Email will be used. + +.. warning:: + + Only configure one delivery mechanism per profile to avoid confusion. The precedence order exists for backward compatibility but is not recommended for production use. + +Security Considerations +....................... + +When configuring support tickets, keep these security best practices in mind: + +**Credential Management** + +* **Use environment variables for secrets**: Always use ``auth_token_env`` and ``pass_env`` instead of storing credentials directly in configuration files. +* **File permissions**: Configuration files in ``/etc/ood/config`` are readable by all users. Never store sensitive credentials directly in these files. +* **Environment variables**: Set sensitive credentials in ``/etc/ood/profile`` or Apache configuration where they are not accessible to users. + +Example secure configuration: + +.. code-block:: yaml + + # /etc/ood/config/ondemand.d/support_ticket.yml + support_ticket: + servicenow_api: + server: "https://servicenow.example.edu" + auth_token_env: "SERVICENOW_API_TOKEN" # Secure + # auth_token: "secret123" # INSECURE - Do not do this! + +.. code-block:: bash + + # /etc/ood/profile + export SERVICENOW_API_TOKEN="actual-secret-token-here" + +**SSL/TLS Verification** + +* Always use ``verify_ssl: true`` (the default) in production environments +* Only disable SSL verification for testing/development environments +* Use proper SSL certificates from trusted certificate authorities + +**Input Validation** + +* The system automatically validates email addresses using standard RFC patterns +* File upload sizes are limited by default (6MB per file, 4 files maximum) +* Adjust ``attachments.max_size`` and ``attachments.max_items`` based on your environment's needs + +**Session Data Filtering** + +* The system automatically filters sensitive session parameters (like ``ood_connection_info``) +* Custom templates should avoid exposing sensitive information +* Review custom templates to ensure they don't inadvertently include passwords or tokens + +Community contributions for new delivery mechanisms are welcome! diff --git a/source/how-tos/app-development/interactive/dynamic-form-widgets.rst b/source/how-tos/app-development/interactive/dynamic-form-widgets.rst index fd5b34296..4a4456b9b 100644 --- a/source/how-tos/app-development/interactive/dynamic-form-widgets.rst +++ b/source/how-tos/app-development/interactive/dynamic-form-widgets.rst @@ -60,7 +60,7 @@ for the ``ruby`` cluster. It will then hide it every time the cluster ``ruby`` data-option-for-cluster-ruby: false ] -Exclusive options +Exclusive Options ................. Because all form items are shown by default, generally you're setting @@ -90,8 +90,68 @@ cluster. ] .. tip:: - This example shows toggling options based on the cluster, but this feature - generically support any field and value. + This example shows toggling options based on the cluster, but this feature generically + support any field and value. If the value contains special characters outside the + standard lowercase alphabet and digits, see the :ref:`data-alias` directive below. + +.. warning:: + ``data-option-for`` and ``data-exclusive-option-for`` directives will conflict if used + within the same form item. It is therefore recommended to choose the one that will be + most efficient and use it exclusively within each form item. + +.. _data-alias: + +Option Value Aliases +.................... + +Since the ``option-for`` and ``exclusive-option-for`` directives above require you to place +the conditional value inside the directive itself, you can encounter issues if the value contains +special characters and cannot easily be changed. ``data-alias`` fills this gap, allowing you to +reference any incompatible value with a custom alias. + +For example, suppose you have to refer to the ``node_type`` value ``gpu:advanced`` in an ``option-for`` +directive. You can define the alias ``gpu-advanced`` using the directive ``data-alias-gpu-advanced: 'gpu:advanced'``, +and refer to this value with the directive ``data-option-for-node-type-gpu-advanced: false``. + +Aliases must be defined before they are used and can only be referenced within the form item +where they are defined. This form item scoping allows aliases to be freely used without +conflicting with aliases defined in global attributes. In the same way, global attributes +can use aliases without impacting existing user forms. + +For a larger example, suppose you have a custom ``account`` field, and by institutional convention all account names +contain the ``@`` symbol. Using ``data-alias``, you can still selectively display ``node_type`` +options based on these accounts as follows: + +.. code-block:: yaml + :emphasize-lines: 13,14,18,19 + + attributes: + account: + widget: select + options: + - 'first_last@school.edu' + - 'first_last@organization.org' + node_type: + widget: select + options: + - 'standard' + - [ + 'academic', + data-alias-student: 'first_last@school.edu', + data-exclusive-option-for-account-student: true + ] + - [ + 'premium', + data-alias-professional: 'first_last@organization.org', + data-exclusive-option-for-account-professional: true + ] + +.. tip:: + It is recommended to use an alias anytime the referenced value contains characters beyond + lowercase letters (a-z) and digits (0-9). Aliases you define should follow those same rules, + but can also use dashes. + +.. _dynamic-bc-apps-data-hide: Hiding entire elements ********************** @@ -99,7 +159,7 @@ Hiding entire elements The ``data-hide`` directive allows you to hide another element based on the current value of a select element. -Let's continue examples involving GPUs. We'd like to provide users +Let's continue with examples involving GPUs. We'd like to provide users with options for `CUDA`_ versions. But using Nvidia's `CUDA`_ libraries only makes sense when the user is requesting GPUs. @@ -109,6 +169,20 @@ Here's the example YAML for this app with two select widgets. This instructs the web-page to hide the ``cuda_version`` when the ``standard`` ``node_type`` is selected. +.. code-block:: yaml + :emphasize-lines: 7 + + attributes: + node_type: + widget: select + options: + - [ + 'standard', 'standard', + data-hide-cuda-version: true, + data-set-cuda-version: 'none' + ] + - 'gpu' + .. warning:: In addition to hiding form fields like this example shows, one should also use a ``data-set`` directive to set the value because the field @@ -118,27 +192,30 @@ instructs the web-page to hide the ``cuda_version`` when the ``standard`` By forcing a value after hiding it you can ensure that the correct values are being passed to the server. -.. tip:: - - In addition to setting the value to ``true`` to hide the form item, in 4.0 - you can also specify ``false`` to show the form item. +If you have an item that is hidden for most options, you can include the +``hide_by_default: true`` attribute in the item's definition and use +``data-hide-item: false`` on options that should reveal that item. For example, +if only the ``gpu`` node type has access to GPU resources, then you can easily +hide it for any other node type with the following form .. code-block:: yaml - :emphasize-lines: 7 + :emphasize-lines: 6, 10 attributes: node_type: widget: select options: - - [ - 'standard', 'standard', - data-hide-cuda-version: true, - data-set-cuda-version: 'none' - ] - - 'gpu' - + - ['Standard', 'standard'] + - ['Small', 'small'] + - ['Large', 'large'] + - ['Gpu', 'gpu', data-hide-cuda-version: false] -Additionally, you can use ``check_box`` widgets to hide elements. + cuda_version: + widget: select + hide_by_default: true + options: + +Finally, you can also use ``check_box`` widgets to hide elements. Here we have a checkbox ``enable_cuda_version`` that will show ``cuda_version`` when checked and hide it when it's not checked. @@ -180,7 +257,40 @@ form element based on the selected option in a select widget. value: 1 In this case, selecting Node Type 'small' will change the label of Cores to -'Number of Cores (1-4)'. +'Number of Cores (1-4)'. Note that you must define a ``data-label`` directive +on each option of the select, as it does not keep a default value. + +.. _dynamic-bc-apps-data-help: + +Dynamic Help Text +***************** + +The ``data-help-*`` directive allows you to change the help text of another +form element based on the selected option in a select widget. + +.. code-block:: yaml + + form: + - node_type + attributes: + node_type: + widget: select + label: Node Type + options: + - [ 'Small', 'small', data-help-node-type: 'Small machines have 10 cores and 100 GB of RAM' ] + - [ 'Medium', 'medium', data-help-node-type: 'Medium machines have 50 cores and 500 GB of RAM' ] + - [ 'Large', 'large', data-help-node-type: 'Large machines have 100 cores and 1 TB of RAM' ] + +In this case, the ``node_type`` form item begins with the help text 'Small machines ...', since this +is the initial value of the select. As you select different options, the help text changes to provide +details of the different choices. While we have the ``node-type`` select changing its own help text in +this example, you can modify the help text of any form item using the same ``data-help-attribute-name`` +syntax. + +Note that ``data-help`` directives will override the help text provided in the attribute definition, and +any options that do not have a ``data-help`` directive will not change the help text from it's last value. +This means that if we left out the data-help directive on the medium option, the help text will retain its +previous value (either 'Small machines...' or 'Large machines...') when medium is selected instead of clearing. Dynamic Min and Maxes ********************* diff --git a/source/how-tos/app-development/interactive/form-widgets.rst b/source/how-tos/app-development/interactive/form-widgets.rst index eef76fbb4..885b29af6 100755 --- a/source/how-tos/app-development/interactive/form-widgets.rst +++ b/source/how-tos/app-development/interactive/form-widgets.rst @@ -146,6 +146,9 @@ Form Widgets ``show_files`` is a boolean flag to show files or not. This defaults to true - it will show files. + ``popup_title`` is the title displayed in the modal. Default is + "Select Your Working Directory". + ``favorites`` allows you to override the :ref:`favorite paths you've added in files menu `. Provide an array of new favorites or set to ``false`` to disable showing favorites altogether. @@ -157,6 +160,7 @@ Form Widgets directory: "/fs/ess/project" show_hidden: true show_files: false + popup_title: "Select the Folder" favorites: - /fs/ess - /fs/scratch diff --git a/source/how-tos/app-development/interactive/form.rst b/source/how-tos/app-development/interactive/form.rst index ef00abc99..95f62e0d5 100644 --- a/source/how-tos/app-development/interactive/form.rst +++ b/source/how-tos/app-development/interactive/form.rst @@ -3,6 +3,10 @@ User Form (``form.yml.erb``) ============================ +.. contents:: Table of Contents + :depth: 3 + :local: + The configuration file ``form.yml`` creates the `html form`_ your customers will use to start the interactive application. @@ -67,6 +71,12 @@ building and launching the ``my_app`` Interactive App session. You can include dynamically generated content in the form by renaming the form file to ``form.yml.erb`` and incorporating ERB syntax. +.. warning:: + Version 4.1 caches the form configuration, so you may not see changes to + ``form.yml.erb`` take effect immediately. To get your most recent changes + during app development, you can clear the cache by selecting + *Help* → *Restart Web Server* from the top right of the navbar. + Configuration ------------- @@ -156,6 +166,7 @@ The most commonly used predefined attributes are given as: .. _bc_account: ``bc_account`` +.............. This adds a ``text_field`` to the HTML form that will be used as the charged account for the submitted job. @@ -164,6 +175,7 @@ The most commonly used predefined attributes are given as: .. _bc_queue: ``bc_queue`` +............ This adds a ``text_field`` to the HTML form that will supply the name of the queue that the batch job is submitted to. @@ -172,13 +184,23 @@ The most commonly used predefined attributes are given as: .. _bc_num_hours: ``bc_num_hours`` +................ This adds a ``number_field`` to the HTML form that describes the maximum amount of hours the submitted batch job may run. This attribute gets converted to seconds and then set on `OodCore::Job::Script#wall_time`_. +.. _bc_num_nodes: + +``bc_num_nodes`` +................ + This adds a number field to the html form that describes the number of nodes + that a submitted job may use. This attribute then uses the proper value in + `OodCore::Job::Script#native`_ to reserve nodes with your scheduler. + ``bc_num_slots`` +................ This adds a ``number_field`` to the HTML form that describes the number of processors, CPUs on a single node, or nodes that the submitted job may use (depends on the resource manager used, e.g., Torque, Slurm, ...). @@ -188,15 +210,15 @@ The most commonly used predefined attributes are given as: .. warning:: - This predefined attribute is very resource manager specific, and is the - most brittle of all the other predefined attributes. May require - customization (see - :ref:`interactive-development-form-customizing-attributes`) to work at - your center. + This predefined attribute is very resource manager specific, and is the + most brittle of all the other predefined attributes. It is recommended to + use either `bc_num_nodes`_ or `auto_cores`_ instead for + consistent behavior across schedulers. .. _bc_email_on_started: ``bc_email_on_started`` +....................... This adds a ``check_box`` to the HTML form that determines whether the user should be notified by email when the batch job starts. @@ -217,7 +239,7 @@ input the queue name themselves. for the user to choose from without intervention from the administrator or the user. -They must be added to the `form` list. +They must be added to the ``form`` list. EX: @@ -226,11 +248,13 @@ EX: form: - auto_modules_ -auto_primary_group +``auto_primary_group`` +...................... This will automatically set the `OodCore::Job::Script#accounting_id`_ to the primary group of the user. No choice will be given to the user. -auto_modules_ +``auto_modules_`` +......................... This will generate a list of modules in a ``select`` widget. For example ``auto_modules_matlab`` will automatically populate a drop-down list of every single ``matlab`` version available, including the default @@ -284,16 +308,38 @@ auto_modules_ referenced in the ``script.sh.erb`` as ``<%= context.auto_modules_netcdf_serial %>`` replacing any hyphens (``-``) with underscores ``_``. +.. _auto_batch_clusters: + +``auto_batch_clusters`` +....................... + This automatically filters the list of clusters that the user can choose from to remove + all clusters using ``kubernetes``, ``linux_host``, or ``systemd`` as their adapter. It then + retrieves information from the scheduler and automatically sets the maximum value of the + `auto_cores`_ field to the maximum number of cores available on the selected cluster. After + this is added to your form you should remove any other ``clusters:`` value from the top of + your ``form.yml``. + +.. _auto_cores: + +``auto_cores`` +.............. + This will automatically generate a ``number_field`` widget with an initial value of 1. The maximum + value is then automatically set by the `auto_batch_clusters`_ selection so that the value falls + within the acceptable range for that cluster. It is strongly suggested to always pair this with + `auto_batch_clusters`_, as other cluster options, including fixed values, will not set the maximum cores. + .. _auto_groups: -auto_groups +``auto_groups`` +............... This will automatically generate a ``select`` widget populated with a list of the Unix - groups the user is currently in. Administrators can configure :ref:`filter for autogroups ` - to limit the groups shown. + groups the user is currently in. Administrators can configure + :ref:`filter for autogroups ` to limit the groups shown. .. _auto_queues: -auto_queues +``auto_queues`` +............... This will generate a ``select`` widget list of all the queues available to the user. These queues will be cluster aware if you have :ref:`dynamic options ` enabled. That is, they'll show or hide relevant lists given the currently selected @@ -304,7 +350,8 @@ auto_queues .. _auto_accounts: -auto_accounts +``auto_accounts`` +................. This will generate a ``select`` widget list of all the accounts available to the user. ``auto_accounts`` will generate cluster aware lists if you have :ref:`dynamic options ` @@ -325,6 +372,7 @@ auto_accounts We only have support for Slurm accounts at this time. ``auto_qos`` +............ This will automatically generate a ``select`` widget populated with a list of all the QoS (Quality of Service) values available. These are cluster aware if you have :ref:`dynamic options ` enabled. @@ -634,6 +682,50 @@ are: display: true +.. _item_header_option: + +.. describe:: header (String, null) + + Optional markdown header above the item's label that does not respond to dynamic actions on the item. + + Default + No header is displayed. + + .. code-block:: yaml + + header: null + + Example + Display a header to mark a section in the form. + + .. code-block:: yaml + + header: | + ## Account and Queue Settings + - Accounts starting with `P` have access to priority queues. + - Priority queues lower wait times, but increase charge rates. + +.. describe:: hide_by_default (Boolean, false) + + Determines whether the form item is initially shown in the form. If this is set + to ``true``, the form item will not be shown until a :ref:`data-hide action ` + reveals it. + + Default + False. The form item will be shown until a data-hide action hides it. + + .. code-block:: yaml + + hide_by_default: false + + Example + Hide the form item on page load, since it is not needed unless a specific option is selected. + + .. code-block:: yaml + + hide_by_default: true + + Examples -------- @@ -906,5 +998,18 @@ of the file and it will display the select widget defined above. form: - global_queues +As of version 4.1, you can use these global items to override predefined attributes in the form. This is done +by specifying the attribute name prefixed with ``global_`` under your ``global_bc_form_items`` key, and only +defining the options that you would like to override. For example, you could override ``bc_num_hours`` to +have a maximum value of 100 using the following code in your ``ondemand.d`` file. + +.. code-block:: yaml + + # /etc/ood/config/ondemand.d/global_bc_items.yml + + global_bc_form_items: + global_bc_num_hours: + max: 100 + .. _markdown: https://en.wikipedia.org/wiki/Markdown .. _html form: https://en.wikipedia.org/wiki/Form_(HTML) diff --git a/source/how-tos/project-management/project-manager.rst b/source/how-tos/project-management/project-manager.rst new file mode 100644 index 000000000..ab6e006c4 --- /dev/null +++ b/source/how-tos/project-management/project-manager.rst @@ -0,0 +1,137 @@ +.. _project_manager: + +Project Manager +=============== + +.. contents:: Table of Contents + :depth: 2 + :local: + +Overview +-------- + +A new feature in version 4.1, the Project Manager provides a suite of tools to leverage +the Open OnDemand features from batch connect apps for each user's own personal scripts, +track and organize their job history, assemble individual jobs into large-scale workflows, +and easily share and collaborate on these projects with their team. + +The basic element of a project is the **launcher**, which is a paired form and script +that can submit jobs to the scheduler. By selecting different options in the form, a user +can change both scheduler parameters (account, resources, etc.), or change the behavior +of the script itself through environment variables. + +All of the scripts, input data, and output data for a project's launchers are stored in the +**project directory**. This directory can be freely organized according to individual +needs, allowing users to customize for anything from large one-time jobs to long-term +projects with jobs that are repeated many times. + +**Workflows** allow users to connect their launchers together with dependency relationships, +enabling them to build self-contained systems that can be run with a single click. These +systems can be as simple or intricate as needed, and are easily assembled through an +intuitive graphical interface. + +The single most important theme of all these features is **flexibility**. As the examples +demonstrate, there is no one way to structure scripts, organize directories, or +manage complexity within a project. The Project Manager is designed to give users an +efficient and intuitive way to leverage the most powerful features of Open OnDemand in +whatever way works best for them and their needs. + +Setup +----- + +For individual projects, there is no setup required. Anybody with the permissions +to submit jobs to your scheduler will also be able to use all of the launcher and +workflow tools of the Project Manager after updating to v4.1. In order for users +to collaborate together they will need a properly configured shared directory, +meeting the minimum permissions described below. + + +.. _project_manager_collaboration: + +Collaboration +............. + +Collaborative projects require properly configured directories to exist in, which may +vary on the types of collaboration you would like to enable. Like other actions in +Open OnDemand, it will operate as the logged in user and never exceed the UNIX +permissions of the directories and files it operates on. This means there are several +different approaches you may want to take depending on your file system's account and +permission management approach. + +.. _project_manager_shared_project_root: + +The project manager provides the ``OOD_SHARED_PROJECT_PATH`` environment variable to +help potential collaborators discover potential projects to import. While this is +not necessary for collaboration, it is a recommended step to make it easier to locate. +The ``OOD_SHARED_PROJECT_PATH`` is a list of base locations for shared projects, allowing +you to add as many locations as you need for different types of users and projects. +The Project Manager will only allow users to import projects from locations that they +have access to, so there is no danger in adding paths that are not accessible to certain +users. + +Because many centers use group-based permission schemes, the Project Manager expects that +each directory in ``OOD_SHARED_PROJECT_PATH`` has a set of subdirectories with finer permissions, +and will only look for projects in these subdirectories. + +For example, a center may have a shared project folder ``/fs/shared/projects``, and a series of subdirectories +``/fs/shared/projects/developers``, ``/fs/shared/projects/staff``, ``/fs/shared/projects/students``. + + +In this example, they would set ``OOD_SHARED_PROJECT_PATH=/fs/shared/projects``, allowing developers +to create collaborative projects like ``/fs/shared/projects/developers/project1``, which are then +easily accessible to others in the ``developers`` group (likewise for ``staff`` and ``students``). + +While the Project Manager automates the permissions settings on project-specific folders, these can +never exceed the permissions of the ``SHARED_PROJECT_ROOT`` or any group-level subdirectory, so it +is important to ensure that your directory structure meets these minimum requirements. + +#. Any directory in ``OOD_SHARED_PROJECT_PATH`` and above must have at minimum ``r-x`` permissions + for all potential collaborators. + +#. Any group directory directly below ``OOD_SHARED_PROJECT_PATH`` should have ``rwx`` permissions + for the group + +Templates +......... + +For specific applications, it can be helpful to create a project with basic scripts, launchers, +and workflows customized to that application. For this use case, administrators can copy any +valid project directory to the ``OOD_PROJECT_TEMPLATE_DIR`` directory, which defaults to +``/etc/ood/config/apps/dashboard/projects``, or ``OOD_APP_CONFIG_ROOT/projects``, if +``OOD_APP_CONFIG_ROOT`` is defined in your environment. + +Note that unlike ``OOD_APP_CONFIG_ROOT``, the template directory can also be set within profiles +to manage which templates a user can select by setting ``project_template_dir`` in the +:ref:`ondemand-d-ymls`. This allows you to organize your templates into directories for each +profile, in addition to limiting template visibility by restricting read permissions on the +individual template directories. + +.. _launcher_default_items: + +Launchers +......... + +All launchers are required to have a ``cluster`` and a ``script`` selection in their form by +default. These fields cannot be removed by editing the launcher, as any launcher without these +fields will be rejected by the scheduler. While these are sufficient for the general case, +if your center requires additional parameters, these can be supplied with the ``launcher_default_items`` +key in your :ref:`ondemand-d-ymls`. + +For example, OSC does not allow jobs to be submitted without the ``account`` field. To prevent +support tickets from users attempting to run launchers without this field, we add the following +line to ``/etc/ood/config/ondemand.d/ondemand.yml`` + +.. code-block:: yaml + + launcher_default_items: [ auto_accounts ] + +Which ensures that all new launchers begin with ``cluster``, ``script``, and ``account`` fields. + +This can be used to add any :ref:`predefined attribute `, +:ref:`automatic predefined attribute `, or :ref:`global attribute ` +and unlike ``cluster`` and ``script``, these additional default fields _can_ be removed by the user. + +Note that if you have predefined attributes or custom global attributes that are not present as options +in the launcher editor, adding them as default items is the only way to use them in launcher forms. + +For user-focused details of the Project Manager, see :ref:`project-manager-tutorial`. diff --git a/source/images/4.1_file_editor.png b/source/images/4.1_file_editor.png new file mode 100644 index 000000000..6e9346829 Binary files /dev/null and b/source/images/4.1_file_editor.png differ diff --git a/source/images/interactive-app-notifications.png b/source/images/interactive-app-notifications.png new file mode 100644 index 000000000..28f400a3a Binary files /dev/null and b/source/images/interactive-app-notifications.png differ diff --git a/source/images/launcher-form.png b/source/images/launcher-form.png new file mode 100644 index 000000000..cf09e960d Binary files /dev/null and b/source/images/launcher-form.png differ diff --git a/source/images/module_browser.png b/source/images/module_browser.png new file mode 100644 index 000000000..fe0a67183 Binary files /dev/null and b/source/images/module_browser.png differ diff --git a/source/images/project-dashboard.png b/source/images/project-dashboard.png new file mode 100644 index 000000000..5901da59b Binary files /dev/null and b/source/images/project-dashboard.png differ diff --git a/source/images/recent_app_popups.png b/source/images/recent_app_popups.png new file mode 100644 index 000000000..e84025465 Binary files /dev/null and b/source/images/recent_app_popups.png differ diff --git a/source/images/workflow-ui.png b/source/images/workflow-ui.png new file mode 100644 index 000000000..babf23636 Binary files /dev/null and b/source/images/workflow-ui.png differ diff --git a/source/index.rst b/source/index.rst index d0c57ae89..e6e8cb9bd 100644 --- a/source/index.rst +++ b/source/index.rst @@ -64,6 +64,7 @@ These are institutions who were early adopters or provided HPC resources for dev how-tos/debug enable-desktops how-tos/app-development + how-tos/project-management/project-manager .. toctree:: :maxdepth: 2 @@ -75,7 +76,8 @@ These are institutions who were early adopters or provided HPC resources for dev tutorials/tutorials-interactive-apps tutorials/tutorials-passenger-apps tutorials/tutorials-dashboard-apps - + tutorials/tutorials-project-manager + .. toctree:: :maxdepth: 2 :caption: Reference diff --git a/source/installation.rst b/source/installation.rst index 5a4be48dc..f5eaeec53 100644 --- a/source/installation.rst +++ b/source/installation.rst @@ -6,7 +6,7 @@ Installation The OnDemand host machine needs to be setup *similarly* to a login node. This means that it will need: -- RedHat/RockyLinux/AlmaLinux 8+ or Ubuntu 20.04-24.04 or Debian 12 or Amazon Linux 2023 +- RedHat/RockyLinux/AlmaLinux 8+ or Ubuntu 22.04-24.04 or Debian 12 or Amazon Linux 2023 - the resource manager (e.g., Torque, Slurm, or LSF) client binaries and libraries used by the batch servers installed - configuration on both OnDemand node **and batch servers** to be able to diff --git a/source/installation/install-software.rst b/source/installation/install-software.rst index b6c1f6a14..79cd4df37 100644 --- a/source/installation/install-software.rst +++ b/source/installation/install-software.rst @@ -8,7 +8,7 @@ Open OnDemand uses these packages, among many others. - Apache HTTP Server 2.4 - Ruby 3.3 with :command:`rake`, :command:`bundler`, and development libraries -- NodeJs 20 +- NodeJs 22 .. note:: @@ -47,6 +47,12 @@ Open OnDemand uses these packages, among many others. sudo dnf install epel-release sudo dnf module enable ruby:3.3 nodejs:20 + .. tab:: RockyLinux/Alma Linux 10 + + .. code-block:: sh + + sudo dnf config-manager --set-enabled crb + sudo dnf install epel-release .. tab:: RHEL 8 @@ -65,6 +71,13 @@ Open OnDemand uses these packages, among many others. sudo dnf module enable ruby:3.3 nodejs:20 sudo subscription-manager repos --enable codeready-builder-for-rhel-9-x86_64-rpms + .. tab:: RHEL 10 + + .. code-block:: sh + + sudo dnf install epel-release + sudo subscription-manager repos --enable codeready-builder-for-rhel-9-x86_64-rpms + 2. Add repository and install ----------------------------- @@ -74,6 +87,7 @@ Open OnDemand uses these packages, among many others. .. code-block:: sh + sudo rpm --import https://yum.osc.edu/ondemand/RPM-GPG-KEY-ondemand sudo dnf install https://yum.osc.edu/ondemand/{{ ondemand_version }}/ondemand-release-web-{{ ondemand_version }}-1.el8.noarch.rpm sudo dnf install ondemand @@ -82,20 +96,19 @@ Open OnDemand uses these packages, among many others. .. code-block:: sh + sudo rpm --import https://yum.osc.edu/ondemand/RPM-GPG-KEY-ondemand-SHA512 sudo dnf install https://yum.osc.edu/ondemand/{{ ondemand_version }}/ondemand-release-web-{{ ondemand_version }}-1.el9.noarch.rpm sudo dnf install ondemand - .. tab:: Ubuntu 20.04 + .. tab:: RedHat/Rocky Linux/AlmaLinux 10 .. code-block:: sh - sudo apt install apt-transport-https ca-certificates - wget -O /tmp/ondemand-release-web_{{ ondemand_version }}.0-focal_all.deb https://apt.osc.edu/ondemand/{{ ondemand_version }}/ondemand-release-web_{{ ondemand_version }}.0-focal_all.deb - sudo apt install /tmp/ondemand-release-web_{{ ondemand_version }}.0-focal_all.deb - sudo apt update + sudo rpm --import https://yum.osc.edu/ondemand/RPM-GPG-KEY-ondemand-SHA512 + sudo dnf install https://yum.osc.edu/ondemand/{{ ondemand_version }}/ondemand-release-web-{{ ondemand_version }}-1.el10.noarch.rpm - sudo apt install ondemand + sudo dnf install ondemand .. tab:: Ubuntu 22.04 @@ -134,6 +147,7 @@ Open OnDemand uses these packages, among many others. .. code-block:: sh + sudo rpm --import https://yum.osc.edu/ondemand/RPM-GPG-KEY-ondemand-SHA512 sudo dnf install https://yum.osc.edu/ondemand/{{ ondemand_version }}/ondemand-release-web-{{ ondemand_version }}-1.amzn2023.noarch.rpm sudo dnf install ondemand @@ -143,7 +157,7 @@ Open OnDemand uses these packages, among many others. .. tabs:: - .. tab:: RHEL/Rocky 8 & 9 + .. tab:: RHEL/Rocky/AlmaLinux .. code-block:: sh diff --git a/source/installation/modify-system-security.rst b/source/installation/modify-system-security.rst index 4c7ef38eb..a6dd4b098 100644 --- a/source/installation/modify-system-security.rst +++ b/source/installation/modify-system-security.rst @@ -27,6 +27,7 @@ The OnDemand SELinux package makes several changes to allow OnDemand to run with The custom SELinux booleans provided by the OnDemand SELinux policy: * ``ondemand_manage_user_home_dir`` (**default=off**): Necessary if user home directories are local disk and not NFS. This is useful when OnDemand is hosted on the system also acting as the NFS server for home directories. +* ``ondemand_manage_config_dir`` (**default=off**): Allow management of ``.config`` directories labeled with ``config_home_t``. * ``ondemand_manage_vmblock`` (**default=off**): So far this has only proven necessary when running OnDemand inside of Vagrant when the home directory is a Virtualbox mount. * ``ondemand_use_nfs`` (**default=on**): Allow OnDemand to manage NFS home directories, which is necessary if home directories are accessible via NFS on the OnDemand web node. * ``ondemand_use_ssh`` (**default=on**): Allow OnDemand to use SSH for Shell app and Linux Host Adapter. diff --git a/source/reference/files/nginx-stage-yml.rst b/source/reference/files/nginx-stage-yml.rst index d2787a495..8bcb666a5 100644 --- a/source/reference/files/nginx-stage-yml.rst +++ b/source/reference/files/nginx-stage-yml.rst @@ -346,6 +346,8 @@ Configuration Options passenger_log_file: '/some/other/location/%{user}/error.log' +.. _passenger_options: + .. describe:: passenger_options (Hash) A Hash of additional Passenger options @@ -822,6 +824,43 @@ Configuration Options :ref:`nginx-stage-nginx` and :ref:`nginx-stage-nginx-clean` when manually starting and stopping the NGINX process. +.. describe:: disabled_shell_message (String) + + Define an error message that is displayed to users when they have a disabled_shell. + + Default + Tell the user they have a restricted shell. + + .. code-block:: yaml + + disabled_shell_message: "user has a disabled shell: %s" + + Example + Display a different message to the user. + + .. code-block:: yaml + + disabeled_shell_message: "This account has a disabled shell. Contact support." + +.. describe:: show_nginx_stage_help_message (Boolean) + + Hide the standard error ``Run 'nginx_stage --help' to see a full list of available + command line options.`` message when calling ``nginx_stage``. + + Default + Show the help message. + + .. code-block:: yaml + + show_nginx_stage_help_message: true + + Example + Hide the help message. + + .. code-block:: yaml + + show_nginx_stage_help_message: false + .. describe:: disable_bundle_user_config (Integer) Set BUNDLE_USER_CONFIG to ``/dev/null`` in the PUN environment. diff --git a/source/reference/files/ood-portal-yml.rst b/source/reference/files/ood-portal-yml.rst index b07930653..b572715ee 100644 --- a/source/reference/files/ood-portal-yml.rst +++ b/source/reference/files/ood-portal-yml.rst @@ -403,6 +403,13 @@ Configure General Options > string.match('ktrout@example.edu', '^([^@]+)@example.edu$') ktrout + In addition to lua patterns, you can append ``:lower`` to the pattern to + automatically lowercase the matched string. An example configuration may be + something like ``.*:lower`` to match anything then lowercase it or + ``^([^@]+)@example.edu$:lower`` to match the email address with the domain + ``example.edu`` then lowercase it. + + Default Match any characters 0 or more times. @@ -781,6 +788,72 @@ assets and links supplied by the web server are relative and not absolute. rnode_uri: "/rnode" +.. _ood-portal-generator-configuration-secure-proxy: + +.. describe:: secure_node_uri (String, null) + + Introduced in 4.1. Similar to ``node_uri`` above, only in that + it will proxy to origin servers using the https protocol instead + of plain text http. + + Default + This feature is disabled by default. + + .. code-block:: yaml + + secure_node_uri: null + + Example + Enable this feature. + + .. code-block:: yaml + + secure_node_uri: "/secure-node" + +.. describe:: secure_rnode_uri (String, null) + + + Introduced in 4.1. Similar to ``rnode_uri`` above, only in that it will + proxy to origin servers using the https protocol instead of plain text http. + + Default + This feature is disabled by default. + + .. code-block:: yaml + + secure_rnode_uri: null + + Example + Enable this feature. + + .. code-block:: yaml + + secure_rnode_uri: "/secure-rnode" + +.. describe:: ssl_proxy (Array, []) + + When enabling SSL/TLS origins above, centers may need to provide + apache SSL directives to successfully communicate with the origin + servers. Centers can use this configuration to provide such directives. + + Default + Do not supply any ``SSLProxy`` directives. + + .. code-block:: yaml + + ssl_proxy: [] + + Example + Do not check the origin server's certificate CN field or name when + proxying to origins over https. + + .. code-block:: yaml + + ssl_proxy: + - 'SSLProxyCheckPeerCN Off' + - 'SSLProxyCheckPeerName Off' + + Configure per-user NGINX ------------------------ @@ -946,6 +1019,24 @@ on mod_auth_openidc_. oidc_uri: "/oidc" +.. describe:: oidc_redirect_uri (String, null) + + Sub-URI used by mod_auth_openidc_ to populate OIDCRedirectURI + + Default + This defaults to: + + .. code-block:: yaml + + oidc_redirect_uri: /oidc + + Example + Enable it on a recommended URI: + + .. code-block:: yaml + + oidc_redirect_uri: "https://rproxy-example-uri.com/oidc" + .. describe:: oidc_discover_uri (String, null) the URI a user is redirected to if they are not authenticated by @@ -987,6 +1078,8 @@ on mod_auth_openidc_. .. _mod_auth_openidc: https://github.com/zmartzone/mod_auth_openidc +.. _ood-portal-generator-user-registration: + Configure User Registration --------------------------- @@ -1038,6 +1131,62 @@ to ``null`` will disable this feature. register_root: "/var/www/ood/register" +.. describe:: register_path (String, register_root) + + Added in 4.1. The file-system path that the ``register_uri`` can map to. + + Default + Same value as ``register_root`` if it's been set. + + .. code-block:: yaml + + register_path: '< the same value as register_root >' + + Example + Enable it to ``/var/www/ood_register``. + + .. code-block:: yaml + + register_root: "/var/www/ood_register" + +.. describe:: register_method (String, 'Alias') + + Added in 4.1. The apache method to which ``register_uri`` will map to ``register_path``. + + Default + Use the apache directive ``Alias`` to map ``register_uri`` to ``register_path``. + + .. code-block:: yaml + + register_method: 'Alias' + + Example + Use the apache directive ``ScriptAlias`` to map ``register_uri`` to ``register_path``. + + .. code-block:: yaml + + register_root: 'ScriptAlias' + +.. describe:: register_method_options (Array, null) + + Added in 4.1. Additional apache directives you may need to supply to register users. + + Default + None provided. + + .. code-block:: yaml + + register_method_options: null + + Example + Provide a ``WSGIDaemonProcess`` directive with some arguments to help register users. + + .. code-block:: yaml + + register_method_options: + - 'WSGIDaemonProcess register user=www-data group=www-data threads=5 home=/var/www/flask-apps/register' + + .. describe:: oidc_provider_metadata_url (String, null) Refer to OIDCProviderMetadataURL in `auth_openidc.conf`_. diff --git a/source/release-notes.rst b/source/release-notes.rst index 0c577055e..8d903efa6 100644 --- a/source/release-notes.rst +++ b/source/release-notes.rst @@ -6,6 +6,7 @@ Release Notes .. toctree:: :maxdepth: 2 + release-notes/v4.1-release-notes release-notes/v4.0-release-notes release-notes/v3.1-release-notes release-notes/v3.0-release-notes diff --git a/source/release-notes/v4.1-release-notes.rst b/source/release-notes/v4.1-release-notes.rst new file mode 100644 index 000000000..1c1e75ad6 --- /dev/null +++ b/source/release-notes/v4.1-release-notes.rst @@ -0,0 +1,610 @@ +v4.1 Release Notes +================== + +.. contents:: Table of Contents + :depth: 3 + :local: + + +Acknowledgments +--------------- + +Thank you to all of the community members who contributed code, documentation, suggestions, bug reports, and other +assistance across the project! We would like to recognize the following individuals and groups for their +significant contributions to the Open OnDemand 4.1 release. + +**University of Maryland** + +Alan Sussman and Harshit Soora contributed major enhancements to the Project Manager, expanding support for +managing composite job workflows within Open OnDemand. Their work improves how users organize, construct, +and execute multiple batch jobs all through the Project Manager. + +**Cendio AB** + +Robert Henschel, Mattias Reuterskiöld, William Sjöblom, and Jean Zagonel contributed to the development of the +ThinLinc integration with Open OnDemand. This integration provides an additional option for delivering Linux +desktop sessions through Open OnDemand. Additionally, building on foundational work by **Jonathan Goodson (NIH)** +to support proxying to secure origins (interactive applications) over HTTPS, their efforts helped complete and +integrate this functionality for version 4.1. A beta version is available for testing here in the +`project repository `_. + +**Harvard University IQSS and the Dataverse Team** + +Leonard Wisniewski, Phil Durbin, Aday Bujeda, William Horka, Michael Reekie, and colleagues from the IQSS and Dataverse +teams contributed documentation and implementation support for the ServiceNow integration, along with various +bug fixes and enhancements across dismissible announcements, user home directories, shared app rendering, and a +configurable button in the Files app. They also created **OnDemand Loop**, a companion application that streamlines data movement +between HPC clusters and repositories like Dataverse or Zenodo. By replacing complex command-line workflows with an intuitive +interface, it lowers technical barriers and enables researchers to manage datasets directly within their computing environment. +More information is available here at the `OnDemand Loop documentation site `_. + +We'd also like to extend a big thank you to those listed below for contributing to Open OnDemand for the +first time in this release. These contributions span the ``ondemand``, ``ood_core``, and +``ood-documentation`` GitHub repositories and include **44** first-time contributors. + +* ``wpoely86`` made their first contribution in `#4068 `_ +* ``harshit-soora`` made their first contribution in `#4069 `_ +* ``hansen-m`` made their first contribution in `#4111 `_ +* ``junousi`` made their first contribution in `#4122 `_ +* ``JanssenAaron`` made their first contribution in `#4133 `_ +* ``hsmallbone`` made their first contribution in `#4142 `_ +* ``devanyk2`` made their first contribution in `#4166 `_ +* ``adamboutcher`` made their first contribution in `#4212 `_ +* ``genericdata`` made their first contribution in `#4237 `_ +* ``Vibe-Guy`` made their first contribution in `#4355 `_ +* ``yuvipanda`` made their first contribution in `#4410 `_ +* ``bedroesb`` made their first contribution in `#4439 `_ +* ``simonLeary42`` made their first contribution in `#4476 `_ +* ``Bubballoo3`` made their first contribution in `#4492 `_ +* ``twhitehead`` made their first contribution in `#4504 `_ +* ``brandonbiggs`` made their first contribution in `#4511 `_ +* ``moffatsadeghi`` made their first contribution in `#4534 `_ +* ``nd996`` made their first contribution in `#4561 `_ +* ``epruesse`` made their first contribution in `#4586 `_ +* ``mw-a`` made their first contribution in `#4601 `_ +* ``RuttJen`` made their first contribution in `#4582 `_ +* ``jmeddick`` made their first contribution in `#4532 `_ +* ``yarikoptic`` made their first contribution in `#4671 `_ +* ``GlazerMann`` made their first contribution in `#4721 `_ +* ``jgoodson`` made their first contribution in `#4710 `_ +* ``mrgum`` made their first contribution in `#4747 `_ +* ``honza801`` made their first contribution in `#4784 `_ +* ``fodinabor`` made their first contribution in `#4814 `_ +* ``Cinnamals`` made their first contribution in `#4839 `_ +* ``pddro-Vestas`` made their first contribution in `ood_core#865 `_ +* ``andrejcermak`` made their first contribution in `ood_core#871 `_ +* ``tat-ohmura`` made their first contribution in `ood_core#883 `_ +* ``dev-sda1`` made their first contribution in `ood_core#914 `_ +* ``kwkoepcke3`` made their first contribution in `ood_core#916 `_ +* ``romulojales`` made their first contribution in `ood-documentation#980 `_ +* ``rwxayheee`` made their first contribution in `ood-documentation#1097 `_ +* ``davidecarlson`` made their first contribution in `ood-documentation#1099 `_ +* ``jennyfothergill`` made their first contribution in `ood-documentation#1122 `_ +* ``ndjones`` made their first contribution in `ood-documentation#1128 `_ +* ``bpaynter`` made their first contribution in `ood-documentation#1167 `_ +* ``aharral`` made their first contribution in `ood-documentation#1187 `_ +* ``JinyuHan99`` made their first contribution in `ood-documentation#1193 `_ +* ``saracook`` made their first contribution in `ood-documentation#1198 `_ +* ``Pennswood`` made their first contribution in `ood-documentation#1203 `_ + + +Breaking Changes and Deprecations +--------------------------------- + +The only breaking changes in version 4.1 are with the operating system support changes +and NodeJS dependency changes listed in the next two sections. + +.. warning:: + The Job Composer is officially **deprecated** in 4.1, and will no longer receive new features or bug fixes + beyond critical maintenance. Although the :ref:`new Project Manager ` covers and + surpasses all Job Composer functionality, the Job Composer will remain enabled as a legacy feature until version 5.0. + +Operating System Support +------------------------ + +- Open OnDemand version 4.1 adds support for RHEL/Rocky/AlmaLinux 10. +- Open OnDemand version 4.1 drops support for Ubuntu 20.04. + +Dependency Updates +------------------ + +This release updates the following dependencies: + +- Passenger 6.1.0 +- NGINX 1.26.3 +- ondemand-dex 2.44.0 +- NodeJS 22 **(Every OS)** +- Bootstrap 5 + + .. warning:: + + The change in NodeJS version means any Node-based apps that are not + provided by the OnDemand RPM must be rebuilt or supply their own + :ref:`nodejs_starter_app_wrapper` to use the older version of NodeJS. + +SELinux Changes +--------------- + +Add the ``ondemand_manage_config_dir`` boolean to allow Open OnDemand to manage ``.config`` directories labeled with ``config_home_t`` + +Authentication and Security +--------------------------- + +Software Bill of Materials +.......................... + +Open OnDemand version 4.1 includes a Software Bill of Materials (SBOM) outlining the +operating system-level software dependencies for Open OnDemand. For more information, +please see the :ref:`Requirements section ` +of the documentation. + +Lower-casing in User Mapping +............................ + +The :ref:`user_map_match ` configuration now +responds to ``:lower`` to automatically lowercase ``REMOTE_USER`` when matching +against it. + +See :ref:`the user_map_match documentation ` +for more details. + +Proxying to Origins Over HTTPS +.............................. + +Version 4.1 introduces the ability to proxy to origin servers over HTTPS protocol. +While continuing to support plain-text origins through ``node_uri`` and +``rnode_uri``, we've added ``secure_node_uri`` and ``secure_rnode_uri`` +configurations in ``ood_portal.yml``. + +See the :ref:`ood_portal.yml configuration for secure proxies ` +for more information. + +Expanded Registration Options +............................. + +More ``ood_portal.yml`` options for registering users have been added in version 4.1. +They are ``register_path``, ``register_method``, and ``register_method_options``. + +See :ref:`ood-portal-generator-user-registration` for more details. + +Platform Configuration, Operations, and Observability +----------------------------------------------------- + +Passenger Telemetry Now Disabled By Default +........................................... + +`Passenger anonymous telemetry `__ +is now disabled by default in version 4.1. If you would like to override this default +and enable Passenger telemetry, you can add ``passenger_disable_anonymous_telemetry: 'off'`` +to your :ref:`passenger_options hash `. Note that this change has also been +backported in the 4.0.4 and 3.1.12 releases. + +Shell Terminal is Now Configurable +.................................. + +The new ``OOD_SHELL_TERM`` environment variable allows the default ``TERM=xterm-16color`` +to be overridden. This variable can be set to any other terminal type whose escape codes are +fully supported by the underlying `hterm library `__. +Some other common supported options include ``xterm``, ``xterm-256color``, and ``xterm-direct``. +This environment variable can only be set from ``/etc/ood/config/apps/shell/env``. + +Shell Logs Have Added Prefixing +............................... + +Each shell log entry is now prefixed with the user, host, and PID associated with the shell session, +allowing users and administrators to differentiate between log entries produced by simultaneous sessions. + +Accessibility and Usability +--------------------------- + +Accessibility Conformance Report (ACR) +...................................... + +Open OnDemand version 4.1 includes an Accessibility Conformance Report (ACR) and the project +plans to provide an updated report for each major and minor release. Findings from the version 4.1 +ACR will inform ongoing accessibility improvements in future releases. The ACR is available on our +accessibility page at `openondemand.org/accessibility `_. + +Dynamic Forms and Session Cards Announce to Screen Readers +.......................................................... + +Dynamic batch connect forms now alert screen readers when form items are changed, +allowing visually impaired users to better understand what is happening on the page. +This is supported by all :ref:`dynamic form actions `. Similarly, +interactive session cards provide live announcements when job status changes between +queued, running, and completed. + +New Skip Navigation Button +.......................... + +Version 4.1 adds a new skip navigation button to the dashboard, which can be accessed +through keyboard navigation at the top of the page, and is read directly after the +page title by screen readers. Clicking the button allows you to bypass all navigation +bar items and reliably skip to the main content of the page. + +New Dashboard Widgets +..................... + +New dashboard widgets have been added in this release. They include ``balances``, ``file_quotas``, +``nsf_access_events``, and ``system_status``. + +For ACCESS Resource Providers (RPs), the ``nsf_access_events`` widget provides a centralized +view of upcoming ACCESS events relevant to your site through the Open OnDemand dashboard. +Additional RP-focused widgets are planned for future releases. + +See :ref:`dashboard_custom_layout` for more details. + +New Canadian Internationalization Files +....................................... + +Open OnDemand version 4.1 will be the first to include Canadian internationalization +files for both English and French, thanks to Rahim Khoja of the University of +Alberta. This adds to pre-existing internationalization files for American +English, Chinese, and Japanese. It is easy to :ref:`customize or generate your +own localization files `, and we greatly appreciate +community contributions for languages we do not include by default. In addition +to making it easier for others in your country or region to use Open OnDemand, +contributed localization files will also be automatically updated with new releases. + +New Demo Container +.................. + +A new demo container has been made available, allowing you to spin up a lightweight and +preconfigured OnDemand instance on your own machine. The container allows you to preview +all of the basic file management, the shell, and batch connect features Open OnDemand has to offer, +including running a full, interactive Jupyter server. For more details and directions on how +to pull down and run the demo, see the `ood-demo repository `_ on GitHub. + +Interactive Jobs and Applications +--------------------------------- + +Batch Connect Apps Cache ERB Content +.................................... + +All batch connect apps now cache ERB content, allowing expensive ERB operations to be +done in ``form.yml.erb`` without burdening the web server on each page visit. For app +development, this means that changes may not be picked up right away, and you may need +to periodically clear the cache by selecting *Help* → *Restart Web Server* from the top +right of the navigation bar. For more details on using ERB in user forms, see +:ref:`app-development-interactive-form`. + +Session Cards Now Have 'Download as Zip' Button +............................................... + +Interactive session cards now have a 'Download as Zip' button. This allows users +to collect session details like logs, scripts, and user-defined parameters in a +single zip file so that this context can be easily included with support ticket requests. +To avoid accidentally downloading sensitive session information, this button is only +available on completed sessions, and is hidden from users who do not have +:ref:`downloads enabled `. + +Sessions Cards Show Cores and GPUs Requested +............................................ + +Session cards now have widgets to display cores and GPUs available to the job, making it +easier to view the parameters of current or past jobs when re-running for your next session. +Please note that resource widgets will only appear when the quantity requested is greater than 0. + +New Option For Interactive Session Notifications +................................................ + +The 'My Interactive Sessions' page now has a switch to enable notifications. When turned on, +the browser will present notifications to the user when their session changes state. This is +most useful for centers with long queue times, as users no longer need to actively monitor +their jobs to know when they start running. + +.. figure:: /images/interactive-app-notifications.png + :alt: The 'My Interactive Sessions' page of Open OnDemand, highlighting the switch to enable notifications. + +Forms, Widgets, and User Input +------------------------------ + +New ``bc_num_nodes`` Smart Attribute +.................................... + +Open OnDemand now offers the ``bc_num_nodes`` smart attribute, which allows you to request nodes +consistently across schedulers. This replaces the ``bc_num_slots`` smart attribute, which selected +either nodes or cores depending on the scheduler. For more details, see :ref:`bc_num_nodes`. + +New ``auto_cores`` Functionality +................................ + +Version 4.1 offers the ``auto_cores`` smart attribute, which allows you to request cores across schedulers. +When used alongside the ``auto_batch_cluster`` attribute, this ensures that the number of cores selected is valid +for the cluster you are submitting to. For more details, see :ref:`auto_batch_clusters` and :ref:`auto_cores`. + +Automatic Attributes Support Accounts with Special Characters +............................................................. + +The account-aware ``auto_queues`` and ``auto_qos`` now support account names with special characters, widening their +compatibility with more centers. This was prioritized for account names as these are among the hardest to modify, but +the same special character support can be added to other automatic attributes like clusters or queues upon request. +Note that in order to get the account-aware features from these attributes, they must be used along with +``auto_accounts``. + +Global Batch Connect Items Can Override Predefined Attributes +............................................................. + +It is now possible to modify the :ref:`predefined attributes ` +using global definitions. This works similarly to other :ref:`global_bc_form_items`, except that the attribute name +must be the predefined attribute name prefixed with ``global_``, and you only have to define attributes you want to change. +See the full section on :ref:`global_bc_form_items` for more details. + +New ``data-alias`` Directive +............................ + +The new ``data-alias`` directive allows you to create conditional ``option-for`` and ``exclusive-option-for`` directives +for values with special characters. It is recommended to use this for all conditional values that contain characters +outside the lowercase alphabet and digits. For more details, see :ref:`data-alias`. + +New ``data-help`` Directive +........................... + +Version 4.1 adds the ``data-help`` directive, allowing you to dynamically change +the help text below form items when certain select options are chosen. For example, +you may have a ``node_type`` select widget, where type ``basic`` has older GPUs than +``advanced``. The ``data-help`` directive can change the help text on the ``num_gpus`` +option for users who select ``advanced`` so that they can take that info into account +when deciding how many GPUs to select. For full documentation, see :ref:`dynamic-bc-apps-data-help`. + +Form Items Can Be Hidden By Default +..................................... + +Form items now accept a ``hide_by_default: true`` attribute, which will hide the form item +until an option with a ``data-hide-item: false`` is selected. This is especially useful when +certain items are only needed for a few options, as you no longer have to include ``data-hide-item: true`` +on all of those options, but instead only supply directives to options that will show the item. +For more details and examples, see the :ref:`data-hide documentation `. + +Form Items Accept Markdown Headers +.................................. + +Batch connect form items accept a ``header`` attribute, which can be used to render +markdown above the item's label. Headers can provide useful structure to forms, and +are not affected when the item is hidden, or by any other dynamic form actions. For +full documentation, see :ref:`Form Item Headers `. + +Path Selectors Accept Custom Titles +................................... + +The title on the path selector modal is now customizable, allowing you to accurately +represent what the selected path will be used for. For more details, see the +:ref:`full path selector documentation `. + +Recently Used Apps Display Recent Settings +.......................................... + +A popup has been added to the ``recently_used_apps`` dashboard widget allowing you to +preview the settings that will be submitted by hovering over the app icon. + +.. figure:: /images/recent_app_popups.png + :alt: The recently used apps dashboard widget showing recent settings in a hoverable popup. + +Only attributes +with ``display: true`` will appear in this popup, and passwords will never appear. For full +documentation on how to add ``recently_used_apps`` to your homepage and configure which +attributes to display, see :ref:`Dashboard Widgets ` and +:ref:`Attribute Display Option `. + +Files, Projects, and Data Management +------------------------------------ + +File Browser Can Now Render HTML Files +...................................... + +Open OnDemand version 4.1 brings back HTML rendering within the files app, allowing users +to open and view HTML files directly. This is disabled by default, due to security +concerns, but can be enabled by setting ``OOD_UNSAFE_RENDER_HTML=true`` in your +environment or including ``unsafe_render_html=true`` in your :ref:`ondemand-d-ymls`. + +File Downloads Have a Maximum Size +.................................. + +Version 4.1 adds a configurable maximum size for file downloads, which also applies to +opening files in the file browser. This can be configured with the ``OOD_DOWNLOAD_FILE_MAX`` +environment variable, or setting ``download_file_max`` in your :ref:`ondemand-d-ymls`. +This variable accepts plain integer values, and defaults to 10GiB. Please note that size +configurations always accept an integer number of (binary) bytes, so the default 10GiB +would be represented as ``10737418240``. For more details on this configuration, as well +as a similar one for directory downloads, see :ref:`set_download_limits`. + +File Editor Has a Maximum Size +.............................. + +In version 4.1, the file editor will not open files above a configurable maximum size. The default is 12MiB, +but this can be changed by setting the ``OOD_FILE_EDITOR_MAX_SIZE`` environment variable, or setting +``file_editor_max_size`` in your :ref:`ondemand-d-ymls`. This uses the same binary byte format as +``OOD_DOWNLOAD_FILE_MAX`` above, so the 12MiB default would be represented as ``12582912``. + +File Editor Toggles Hard and Soft Tabs +...................................... + +The file editor now supports toggling between hard and soft tabs with a switch on the left panel. While +both use the tab key, hard tabs refer to tab characters, and soft tabs are space characters used for +indentation. + +.. figure:: /images/4.1_file_editor.png + :alt: OOD File browser, demonstrating the additional options and new appearance of the toolbar. + +This is most relevant to languages that rely on consistent indentation, most notably Python +and YAML. For YAML files, only soft tabs are syntactically supported. Python recommends soft tabs for new +projects, but will not allow mixing of hard and soft tabs within a project. + +.. _project-manager-note: + +The Project Manager +................... + +Version 4.1 enables the Project Manager by default, a new alternative to the Job Composer that brings the +automatic scheduler interaction of batch connect forms to your personal job scripts. In addition to this core +functionality, it also provides support for workflows that can submit multiple dependent jobs at once, and +easy collaboration on projects among teams. The core functionality will work without modification, but +collaborative projects work best when used with :ref:`properly configured shared directories `. + +.. figure:: /images/project-dashboard.png + :alt: The project dashboard, showing the launchers, workflows, jobs, and files of an example project. + +For more details on customization and configuration, see the :ref:`Project Manager docs `, +and for an example-driven guide intended for users, see :ref:`project-manager-tutorial`. + + +Help, Support and Institutional Integration +------------------------------------------- + +ServiceNow support tickets +.......................... + +Centers can now send support tickets to ServiceNow from Open OnDemand. +See :ref:`servicenow_support_ticket` for more details. + +Module Browser Page +................... + +Centers that have enabled :ref:`lmod module suppport ` for +OnDemand to read all the modules on clusters will now see a web page for +searching and displaying these modules. + +.. figure:: /images/module_browser.png + :alt: OOD module browser in action, searching for python modules and viewing the available versions + +This new page is under the 'Clusters' menu in the navigation bar. + +Upgrade Instructions +-------------------- + +.. warning:: + + Update the **development** and/or **test** instances of OnDemand installed at your center **before** you modify the **production** instance. + +.. warning:: + + We have tested the upgrade from version 4.0.8 to version 4.1.0 at OSC's Open OnDemand instance. + +#. Update Open OnDemand repository. + + .. tabs:: + + .. tab:: RedHat/Rocky Linux/AlmaLinux 8 + + .. code-block:: sh + + sudo yum install -y https://yum.osc.edu/ondemand/4.1/ondemand-release-web-4.1-1.el8.noarch.rpm + + .. tab:: RedHat/Rocky Linux/AlmaLinux 9 + + .. code-block:: sh + + sudo yum install -y https://yum.osc.edu/ondemand/4.1/ondemand-release-web-4.1-1.el9.noarch.rpm + + .. tab:: Ubuntu 22.04 + + .. code-block:: sh + + wget -O /tmp/ondemand-release-web_4.1.0-jammy_all.deb https://apt.osc.edu/ondemand/4.1/ondemand-release-web_4.1.0-jammy_all.deb + sudo apt install /tmp/ondemand-release-web_4.1.0-jammy_all.deb + sudo apt update + + .. tab:: Ubuntu 24.04 + + .. code-block:: sh + + wget -O /tmp/ondemand-release-web_4.1.0-noble_all.deb https://apt.osc.edu/ondemand/4.1/ondemand-release-web_4.1.0-noble_all.deb + sudo apt install /tmp/ondemand-release-web_4.1.0-noble_all.deb + sudo apt update + + .. tab:: Debian 12 + + .. code-block:: sh + + wget -O /tmp/ondemand-release-web_4.1.0-bookworm_all.deb https://apt.osc.edu/ondemand/4.1/ondemand-release-web_4.1.0-bookworm_all.deb + sudo apt install /tmp/ondemand-release-web_4.1.0-bookworm_all.deb + sudo apt update + + .. tab:: Amazon Linux 2023 + + .. code-block:: sh + + sudo dnf install -y https://yum.osc.edu/ondemand/4.1/ondemand-release-web-4.1-1.amzn2023.noarch.rpm + +#. (**RHEL/Rocky/AlmaLinux 8 & 9 only**) Enable dependency repositories. + + .. code-block:: sh + + sudo dnf module reset nodejs + sudo dnf module enable nodejs:22 + +#. Update Open OnDemand. + + .. tabs:: + + .. tab:: yum/dnf + + .. code-block:: sh + + sudo yum clean all + sudo yum update ondemand + + .. tab:: apt + + .. code-block:: sh + + sudo apt-get --only-upgrade install ondemand + +#. (**Dex users only**) Update the ``ondemand-dex`` package. + + .. tabs:: + + .. tab:: yum/dnf + + .. code-block:: sh + + sudo yum update ondemand-dex + + + .. tab:: apt + + .. code-block:: sh + + sudo apt-get --only-upgrade install ondemand-dex + +#. Update Apache configuration and restart Apache. + + .. code-block:: sh + + sudo /opt/ood/ood-portal-generator/sbin/update_ood_portal + + .. tabs:: + + .. tab:: RedHat/Rocky Linux/AlmaLinux 8 & 9 + + .. code-block:: sh + + sudo systemctl try-restart httpd + + .. tab:: Ubuntu 22.04 & 24.04 & Debian 12 + + .. code-block:: sh + + sudo systemctl try-restart apache2 + + .. tab:: Amazon Linux 2023 + + .. code-block:: sh + + sudo systemctl try-restart httpd + +#. (**Dex users only**) Restart the ``ondemand-dex`` service. + + .. code-block:: sh + + sudo systemctl try-restart ondemand-dex.service + +#. (**SELinux users only**) Update SELinux policies. + + See :ref:`ood_selinux_updates`. + +#. Force all PUNs to restart. + + .. code-block:: sh + + sudo /opt/ood/nginx_stage/sbin/nginx_stage nginx_clean -f diff --git a/source/requirements.rst b/source/requirements.rst index fea1dacf0..6d3d3b7ed 100644 --- a/source/requirements.rst +++ b/source/requirements.rst @@ -19,11 +19,16 @@ At this time OnDemand only supports the following operating systems and architec "RedHat/Rocky Linux/AlmaLinux 8",:raw-html:`✅`,:raw-html:`✅`,:raw-html:`✅` "RedHat/Rocky Linux/AlmaLinux 9",:raw-html:`✅`,:raw-html:`✅`,:raw-html:`✅` - "Ubuntu 20.04",:raw-html:`✅`,:raw-html:`✅`,:raw-html:`❌` + "RedHat/Rocky Linux/AlmaLinux 10",:raw-html:`✅`,:raw-html:`❓`,:raw-html:`✅` "Ubuntu 22.04",:raw-html:`✅`,:raw-html:`✅`,:raw-html:`❌` "Ubuntu 24.04",:raw-html:`✅`,:raw-html:`✅`,:raw-html:`❌` "Debian 12",:raw-html:`✅`,:raw-html:`✅`,:raw-html:`❌` - "Amazon Linux 2023",:raw-html:`✅`,:raw-html:`✅`,:raw-html:`❌` + "Amazon Linux 2023",:raw-html:`✅`,:raw-html:`❓`,:raw-html:`❌` + +.. note:: + + Items marked with :raw-html:`❓` are available upon request. + Open a `Github Issue `_ or post the request on `Discourse `_ Software Requirements --------------------- @@ -53,6 +58,175 @@ And on the Compute node(s): .. _turbovnc: https://turbovnc.org/ .. _websockify: https://github.com/novnc/websockify +.. _software-bom: + +Software Bill of Materials +-------------------------- +The following is a list of OS level software dependencies for Open OnDemand. For application dependencies please see paragraph below. + +.. tabs:: + + .. tab:: RockyLinux 8 + + .. code-block:: sh + + sh + cronie + curl + diffutils + file + git + libxml2 + libxslt + lsof + lua-posix + make + ondemand-apache = 4.0.3-1.el8 + ondemand-gems-4.0.0-1 + ondemand-gems-4.0.1-1 + ondemand-gems-4.0.2-1 + ondemand-gems-4.0.3-1 + ondemand-gems-4.0.5-1 + ondemand-gems-4.0.6-1 + ondemand-gems-4.0.7-1 + ondemand-gems-4.0.8-1 + ondemand-nginx = 1.26.1-1.p6.0.23.ood4.0.3.el8 + ondemand-nginx = 1.26.1-2.p6.0.23.ood4.0.3.el8 + ondemand-nginx = 1.26.1-3.p6.0.23.ood4.0.3.el8 + ondemand-nodejs = 4.0.3-1.el8 + ondemand-passenger = 6.0.23-1.ood4.0.3.el8 + ondemand-passenger = 6.0.23-2.ood4.0.3.el8 + ondemand-passenger = 6.0.23-3.ood4.0.3.el8 + ondemand-ruby = 4.0.3-1.el8 + ondemand-runtime = 4.0.3-1.el8 + python3 + rclone + rsync + sudo + systemd + wget + zlib + + + .. tab:: RockyLinux 9 + + .. code-block:: sh + + sh + cronie + curl + diffutils + file + git + libxml2 + libxslt + lsof + lua-posix + make + ondemand-apache = 4.0.3-1.el9 + ondemand-gems-4.0.0-1 + ondemand-gems-4.0.1-1 + ondemand-gems-4.0.2-1 + ondemand-gems-4.0.3-1 + ondemand-gems-4.0.5-1 + ondemand-gems-4.0.6-1 + ondemand-gems-4.0.7-1 + ondemand-gems-4.0.8-1 + ondemand-nginx = 1.26.1-1.p6.0.23.ood4.0.3.el9 + ondemand-nginx = 1.26.1-2.p6.0.23.ood4.0.3.el9 + ondemand-nginx = 1.26.1-3.p6.0.23.ood4.0.3.el9 + ondemand-nodejs = 4.0.3-1.el9 + ondemand-passenger = 6.0.23-1.ood4.0.3.el9 + ondemand-passenger = 6.0.23-2.ood4.0.3.el9 + ondemand-passenger = 6.0.23-3.ood4.0.3.el9 + ondemand-ruby = 4.0.3-1.el9 + ondemand-runtime = 4.0.3-1.el9 + python3 + rclone + rsync + sudo + systemd + wget + zlib + + .. tab:: Ubuntu 20.04 + + .. code-block:: sh + + libc6 (>= 2.29) + libgcc-s1 (>= 3.0) + libruby2.7 (>= 2.7.0) + libsqlite3-0 (>= 3.7.10) + libstdc++6 (>= 9) + ruby + apache2 + sudo + lsof + lua-posix + tzdata + file + nodejs (>= 20.0) + nodejs (<< 21.0) + ondemand-nginx (>= 1.26.1.p6.0.23.ood4.0.3) + ondemand-nginx (<< 1.27) + ondemand-passenger (>= 6.0.23.ood4.0.3) + ondemand-passenger (<< 6.0.24) + + .. tab:: Ubuntu 22.04 + + .. code-block:: sh + + libc6 (>= 2.34) + libgcc-s1 (>= 3.0) + libruby3.0 (>= 3.0.0~preview2) + libsqlite3-0 (>= 3.7.10) + libstdc++6 (>= 11) + ruby + apache2 + sudo + lsof + lua-posix + tzdata + file + nodejs (>= 20.0) + nodejs (<< 21.0) + ondemand-nginx (>= 1.26.1.p6.0.23.ood4.0.3) + ondemand-nginx (<< 1.27) + ondemand-passenger (>= 6.0.23.ood4.0.3) + ondemand-passenger (<< 6.0.24) + + .. tab:: Ubuntu 24.04 + + .. code-block:: sh + + libc6 (>= 2.38) + libgcc-s1 (>= 3.0) + libruby3.2 (>= 3.2.2) + libsqlite3-0 (>= 3.7.10) + libstdc++6 (>= 13.1) + ruby + apache2 + sudo + lsof + lua-posix + tzdata + file + nodejs (>= 20.0) + nodejs (<< 21.0) + ondemand-nginx (>= 1.26.1.p6.0.23.ood4.0.3) + ondemand-nginx (<< 1.27) + ondemand-passenger (>= 6.0.23.ood4.0.3) + ondemand-passenger (<< 6.0.24) + +Along with the listed operating system dependencies, Open OnDemand also requires various Ruby gems and Node.js packages for its application layer. +These application dependencies are managed by Bundler and Yarn respectively, and are installed automatically when Open OnDemand is built from source or +installed via the provided packages. For a complete list of application dependencies, please refer to the top-level `Gemfile`_, the dashboard `Gemfile.lock`_, +and `package.json`_ files in the Open OnDemand source repository. + +.. _Gemfile: https://github.com/OSC/ondemand/blob/master/Gemfile +.. _Gemfile.lock: https://github.com/OSC/ondemand/blob/master/apps/dashboard/Gemfile.lock +.. _package.json: https://github.com/OSC/ondemand/blob/master/apps/dashboard/package.json + Hardware Requirements --------------------- diff --git a/source/spelling_wordlist.txt b/source/spelling_wordlist.txt index 1a0ed1f37..860d73afc 100644 --- a/source/spelling_wordlist.txt +++ b/source/spelling_wordlist.txt @@ -1,4 +1,5 @@ backport +backported backporting websocket Phusion @@ -13,6 +14,7 @@ filesystems Jupyter RStudio configmap +preconfigured init nsswitch millicores @@ -72,6 +74,7 @@ Gemfile noVNC walltime Walltimes +MiB GiB prepopulate dismissible @@ -90,6 +93,7 @@ changelogs customizable Univa discoverability +observability lua perl bundler @@ -153,6 +157,8 @@ Guilherme Maluf Balzana Buechler +Rahim +Khoja Pitzer Bubblewrap codebase @@ -161,3 +167,16 @@ js balancers OpenTofu Terraform +subdirectories +Sussman +Harshit +Soora +Henschel +Mattias +Reuterskiöld +Sjöblom +Zagonel +Durbin +Horka +Dataverse +Zenodo diff --git a/source/tutorials/tutorials-project-manager.rst b/source/tutorials/tutorials-project-manager.rst new file mode 100644 index 000000000..3ff35ff86 --- /dev/null +++ b/source/tutorials/tutorials-project-manager.rst @@ -0,0 +1,251 @@ +.. _project-manager-tutorial: + +Tutorials: Project Manager +========================== + +.. contents:: Table of Contents + :depth: 2 + :local: + +The project manager is designed to make it easy to submit scripts to your scheduler. +For this example, let's assume you have a system of shell scripts that simulate the +net change in an ideal reservoir over the last seven days, given a city and diameter +of the reservoir. + +Assume these scripts currently exist on your local machine with the following structure:: + + demo_project/ + ├── scripts + │ ├── compute_volume.sh # USAGE: ./compute_volume.sh WEATHER_JSON DIAMETER -> NET_CHANGE + │ ├── geocode.sh # USAGE: ./geocode.sh CITY -> LAT_LONG_JSON + │ └── import_weather.sh # USAGE: ./import_weather.sh LAT_LONG_JSON -> WEATHER_JSON + | + └── simulate_reservoir.sh # USAGE: ./simulate_reservoir.sh CITY DIAMETER -> NET_CHANGE + +where ``simulate_reservoir.sh`` coordinates the smaller scripts as follows + +.. code-block:: bash + + #!/usr/bin/env bash + # Usage: ./simulate_reservoir.sh "City Name" DIAMETER_M + set -euo pipefail + + CITY="$1" + DIAMETER_M="$2" + TMPDIR="$(mktemp -d)" + trap 'rm -rf "$TMPDIR"' EXIT + + GEOCODE_JSON="$TMPDIR/geocode.json" + WEATHER_JSON="$TMPDIR/weather.json" + + ./scripts/geocode.sh "$CITY" > "$GEOCODE_JSON" + + ./scripts/import_weather.sh "$GEOCODE_JSON" > "$WEATHER_JSON" + + ./scripts/compute_volume.sh "$WEATHER_JSON" "$DIAMETER_M" + +The contents of ``compute_volume.sh``, ``geocode.sh``, and ``import_weather.sh`` are left +as an exercise for the reader. + +Initializing a Project +...................... + +Now let's say we want to submit these jobs to the scheduler. We begin by opening the Project Manager, +the third item under the 'Jobs' tab in the navigation bar. From the Project Manager, we select the +'Create a new project' button on the upper right. Since this will be a personal project, we only need +to provide a name and icon for the new project, and press 'Save'. + +Saving your new project will return you to the Project Manager, where you will see your new project listed. +Clicking on the new project's title will open its project dashboard, showing a blank project. + +To add our existing scripts into the project, we first click on the 'Open in files app' button below the +project directory. This takes us to the file browser, where there is a button to upload from our machine. + +After uploading the scripts, we can return to the project dashboard and see a similar file structure +in the project directory:: + + PROJECT_DIRECTORY/ + ├── .ondemand/ + ├── scripts/ + │ ├── compute_volume.sh + │ ├── geocode.sh + │ └── import_weather.sh + | + └── simulate_reservoir.sh + +Before continuing, you should make sure that the files in the ``scripts/`` directory are all executable, +so they can be called directly from your controller script. ``simulate_reservoir.sh``, and any other scripts +we want to submit, do not need to be executable to be submitted to the scheduler. + +Adding a Launcher +................. + +To submit these scripts to the scheduler, we start by clicking the 'New Launcher' button on the left side of +the project dashboard. We will give this launcher the name 'Fixed Simulation', and press 'Save'. + +After saving the launcher, it will appear in the list of launchers on the left side of the dashboard. We can +customize the form for the launcher by pressing the 'Edit' button on the launcher card. This takes you to an +interactive form builder, where you can add form choices. All new launchers will be created with at minimum +``cluster`` and ``script`` fields, as well as any others your center may have defined as +:ref:`default launcher items `. + +For the purpose of this example, let's assume that the blank launcher only has those two fields. We can add +an 'Account' field by pressing 'Add new option' in the launcher editor, selecting 'Account' for the new field, +and pressing 'Add' to include it on the form. This creates a form item labeled 'Account' that should default +to your primary account. If you want the script to run with a different account, you can press the 'Edit' button +beneath the field to change the available options or keep it fixed to a single account. + +Once the account field is correctly configured, we can press 'Save' at the bottom of the launcher editor, and +see a confirmation message that the update was successful. + +This is enough already to submit the script, but if we run it as is, we are bound to get a ``$1: unbound variable`` +error. This is because we have the script configured to accept command line arguments, but these are not being passed +by the scheduler. There are several ways of fixing this, but the most basic would be to hard code the variables in +``simulate_reservoir.sh`` so that it is able to run without arguments. To do this, let's create a copy of +``simulate_reservoir.sh`` titled ``simulate_fixed_reservoir.sh``, and change the top of the file to read + +.. code-block:: bash + + #!/usr/bin/env bash + # Usage: ./simulate_fixed_reservoir.sh + set -euo pipefail + + CITY="New York" + DIAMETER_M="50" + TMPDIR="$(mktemp -d)" + ... + +Now we can edit our launcher again, and see that the new script ``simulate_fixed_reservoir.sh`` is available +in the ``scripts`` selection. Since we know it will break if it runs with ``simulate_reservoir.sh``, we can +edit the script option, select ``simulate_fixed_reservoir.sh`` from the drop down, click 'Fixed Value' to +prevent you from accidentally running a different script, and save the form. Now we can click the 'Show' +button to fill out this form. Since we have already made sure the ``account`` and ``script`` fields were set +correctly, the only thing we have to fill out is the cluster to run it on. Once we do that and press 'Launch' +at the bottom of the form, we will be redirected back to the project dashboard and see our job queued. + +Leveraging Launcher Forms +......................... + +You may have noticed a slight problem with that initial approach. Hard coding the values solves the immediate +issue of removing the need for arguments, but it means we have to go and edit the script each time to change +these values. If we had to run this script for 20 different cities, for example, that is a lot of manual +editing to prepare the next run. + +To make this more efficient, we can use launcher forms to set environment variables that will be available to +the script we run. First, we will make another copy of ``simulate_reservoir.sh``, titled +``simulate_reservoir_from_env.sh``, and change the start of the file to read + +.. code-block:: bash + + #!/usr/bin/env bash + # Usage: ./simulate_reservoir_from_env.sh + set -euo pipefail + + CITY="$CITY_PARAM" + DIAMETER_M="$DIAMETER_PARAM" + TMPDIR="$(mktemp -d)" + ... + +This time, let's create a new launcher. Instead of creating from scratch and re-doing all the work of adding +the account field again, we can use the 'Copy Launcher' button in the top right corner of the launcher card. +This will prompt us for a new name, which we will fill in as 'Variable Simulation' and press 'Save'. If we +edit this launcher to see the form, we will see the same form from 'Fixed Simulation' has been copied over. +After editing the ``script`` field to use ``simulate_reservoir_from_env.sh``, we then click 'Add new option' +and select 'Environment Variable' for the new field. Pressing 'Add' below the select will create a new item, +with options for the name and default value of our variable. For the first one, we will name it ``CITY_PARAM`` +and set the default value to ``New York``, ensuring that even if you leave the field blank, you won't get an +``unbound variable`` error. We can then add another environment variable field named ``DIAMETER_PARAM`` and +set the default value to ``50``. With both fields added, we can save the form and return to the project +dashboard. Now we can click the 'Show' button to see the updated form, and pass some non-default values for +each environment variable. + +.. figure:: /images/launcher-form.png + :alt: The launcher form filled out, showing how to pass non-default values for environment variables + +When we launch, the script will take the parameters from the form and give a +different simulation. By placing these parameters in the form, we are now able to run 10 different +simulations with ease by filling out our form with different values. + +Using Workflows +............... + +We have just seen how launchers simplify the process of submitting to your scheduler. But what if our job +grows beyond the limitations of a single job? For example, we could have a scenario where +``import_weather.sh`` has to wait several hours to make all the necessary API calls, and ``compute_volume.sh`` +requires many cores to run efficiently. It is difficult for the scheduler to reserve lots of cores for such a +long time, and we notice exceedingly long queue times before our jobs can run. To address this, we can use workflows. + +Workflows are a way to chain together launchers, allowing you to break your job into smaller pieces that can +be independently scheduled, but still wait for the output of one another. This will allow us to run our light +compute, time intensive step with basic resources but lots of walltime, and the short compute-heavy step with +lots of resources and less walltime. + +The first step is to split up our controller script into two separate steps. For this example, we will have +one step that performs all the API requests, and another that computes the results from that data. We can put +these steps into ``collect_data.sh`` and ``compute_results.sh`` as seen below. + +.. code-block:: bash + + #!/usr/bin/env bash + # Usage: ./collect_data.sh + set -euo pipefail + + mkdir -p 'data' + + GEOCODE_JSON="data/geocode-$COUNT.json" + WEATHER_JSON="data/weather-$COUNT.json" + + ./scripts/geocode.sh "$CITY_PARAM" > "$GEOCODE_JSON" + + ./scripts/import_weather.sh "$GEOCODE_JSON" > "$WEATHER_JSON" + +.. code-block:: bash + + #!/usr/bin/env bash + # Usage: ./compute_results.sh + set -euo pipefail + + JSON_FILE="data/weather-$COUNT.json" + + ./scripts/compute_volume.sh "$JSON_FILE" "$DIAMETER_PARAM" + +Note how we split our parameters between the two scripts to only pass what each script needs. We also add a +``COUNT`` variable so that we can control when data is overwritten, an essential consideration if you plan +to have more than one instance of a workflow run simultaneously. + +The next step is to create the launchers for these scripts. Like above, we can copy the 'Variable Simulation' +launcher into two new ones, 'Collect Data' and 'Compute Results'. We can then edit the form for each launcher +to include the appropriate environment variables. For the 'Collect Data' launcher, we will keep the +``CITY_PARAM`` variable and change the ``DIAMETER_PARAM`` variable name to ``COUNT``. We then do an analogous +change to the 'Compute Results' launcher, keeping ``DIAMETER_PARAM`` untouched and changing ``CITY_PARAM`` to +``COUNT``. There is no form to fill out for a workflow, so the default values provided here are the values the +launchers will use. Most importantly, we should ensure that the ``COUNT`` default is consistent between the +two. Finally, because we want these launchers to be used in workflows, we ensure every field has 'Fixed Value' +selected, which we can do by clicking 'Edit' and then selecting 'Fixed Value' on each item before saving. This +will prevent the workflows from using any cached launcher parameters, which can be difficult to debug. + +With our launchers created, we can now create a new workflow. From the Project Manager, we click the +'New Workflow' button on the lower left of the project dashboard. We will give it the name 'Simulation +Workflow' and select the 'Collect Data' and 'Compute Results' launchers, before saving the form. + +After being returned to the project dashboard, we then click the 'Show' button under 'Simulation Workflow' to +open it. At the top of the page, there will be a select with one of your launchers selected. We can click the +'Add Launcher' button to place the launcher on the workflow. To add the other launcher, we simply change the +selection and click 'Add Launcher' again. Once both launchers are on the canvas, we need to create the +dependency that relates the two. We can do this by clicking 'Connect Launchers', then selecting the launchers +in the order we want them to run, in this case 'Collect Data' first and 'Compute Results' second. This will +create a directional arrow between the two, pointing in the direction that the execution will flow. Once we +have all the dependencies created, we click 'Connect Launchers' again to stop adding dependencies. At this point +the workflow should look something like + +.. figure:: /images/workflow-ui.png + :alt: The workflow management ui, showing a simple example with two launchers. There is an arrow pointing from 'Collect Data' to 'Compute Results'. + +We can now press 'Submit', and our workflow will save and begin scheduling. If we stay on the workflows page, +we can observe the launchers move between queued, running, and completed states, with dependent launchers only +running after their dependencies are completed. If we don't care to watch the workflow execute, we can press +'Back' from the workflows page to return to the project dashboard, where we can edit the launcher form defaults +and run other workflows as needed, keeping in mind that the COUNT variable must be consistent for each run. + +The Project Manager is designed to be flexible, so if the examples above aren't applicable to your needs, +you can always design your own systems and conventions that fit your specific situation. \ No newline at end of file