Enable Swell to use GMAO AWS R2D2 server and S3 data store by ftgoktas · Pull Request #734 · GEOS-ESM/swell

ftgoktas · 2026-03-11T23:05:24Z

Description

AWS R2D2 server is ready and configured for GMAO users. Users and API keys can be created, and Swell can use this server to ingest and fetch observations via S3 instead of the JCSDA server and Discover local storage.

See the configuration guide for setup steps, example configs and IngestObs test commands.

Closes #733

…ure/aws-datastore

ftgoktas · 2026-03-17T19:30:54Z

The AWS R2D2 server is up and running from 8:30am EST to 5:30pm EST on weekdays. Lifetime values and model names have been registered in the database with the following configuration:

Lifetimes:

debug: 14 days
science: 180 days
publication: 1825 days (5 years)
release: indefinite

Models:

geos
mom6
geos_cf
mom6_cice6_UFS

Please follow the instructions in the configuration guide to test against the server. Let us know if you run into any issues or have questions.

jeromebarre · 2026-04-10T19:37:16Z

-source venv_client/bin/activate
+export R2D2_HOST="discover"
+export R2D2_COMPILER="intel"
+export R2D2_SERVER_HOST="http://13.217.72.149"


Should we replace such hardcoded server IP with a hostname or config variable. Sould we avoid plaintext AWS keys unless that is truly expected for our operations? @shiklomanov-an ?

The IP is an Elastic IP so it won't change, but agreed a DNS hostname would be cleaner long-term.

The server now supports two data stores: the local one on Discover (geos_cf directory with priority 1) and the S3 bucket (priority 2). For Discover, users only need R2D2_USER and R2D2_API_KEY and no AWS credentials. AWS credentials are only required when accessing the S3 data store.

jeromebarre

Looks good to me but see my minor comments

mranst

Sorry it took me a while to review this. I was able to ingest and fetch obs from AWS, but I found that having to swap out the credentials file made it difficult to switch between working on other things. I wonder if we could make use of the dictionary structure in the yaml file to allow for datastores to be selected using an experiment key. Maybe something like:

r2d2_credentials.yaml

jcsda:
  user: ...
  api_key: ...
  ...

gmao_aws:
  user: ...
  api_key: ...
  ...

Swell could then select which of those credentials to use based on a key called r2d2_datastore, or something like that

ftgoktas · 2026-04-21T19:54:14Z

Tested on Discover with both JCSDA's and GMAO's R2D2 servers. Users can now select which R2D2 server and datastore to use directly from experiment.yaml via r2d2_server and r2d2_datastore fields, without needing to modify credentials or code between runs. It defaults to gmao_server and the r2d2-experiments-prod-us-east-1 S3 datastore. Updated the docs with more details for testing: Configuring different R2D2 servers and datastores with Swell on Discover

mranst

Thanks for accommodating my request, this is looking great. I have a few last points - the largest of which is maintaining continuity once this PR goes in. I'd like to keep the amount of changes that users need to make to their environment to run swell as low as possible. Configuring R2D2 is probably the biggest stumbling point users encounter when trying to run swell, so if possible I think it's best to try to maintain compatibility with the current format of the credentials file

mranst · 2026-04-28T21:26:37Z

        try:
            yaml = YAML(typ='safe')
            with open(yaml_path, 'r') as yaml_file:
-                credentials = yaml.load(yaml_file)
+                credentials_yaml = yaml.load(yaml_file) or {}
        except Exception as e:
            logger.error(f"Error loading R2D2 credentials from {yaml_path}: {e}")
            logger.info("Continuing with existing environment variables...")
-            credentials = {}
+            credentials_yaml = {}


Is there a common type of exception you're expecting here?

mranst · 2026-04-28T21:46:48Z

    # Set host and compiler (YAML config takes precedence over platform detection)
-    if 'host' in credentials and 'R2D2_HOST' not in os.environ:
-        os.environ['R2D2_HOST'] = credentials['host']
-        logger.info(f"Using platform host '{r2d2_host}' (overriding YAML '{credentials['host']}')")
-        logger.warning("Using host from YAML file")
+    if 'r2d2_host' in credentials and 'R2D2_HOST' not in os.environ:
+        os.environ['R2D2_HOST'] = credentials['r2d2_host']
+        logger.info(
+            f"YAML r2d2_host ({credentials['r2d2_host']!r}) overrides "
+            f"platform default ({r2d2_host!r})"
+        )

    elif r2d2_host and 'R2D2_HOST' not in os.environ:
        os.environ['R2D2_HOST'] = r2d2_host
        logger.info(f"Set R2D2_HOST={r2d2_host} from platform configuration")

    # Set compiler
-    if 'compiler' in credentials and 'R2D2_COMPILER' not in os.environ:
-        os.environ['R2D2_COMPILER'] = credentials['compiler']
-        logger.info(f"Using platform compiler '{r2d2_compiler}' \
-                    (overriding YAML '{credentials['compiler']}')")
-        logger.warning("Using compiler from YAML file")
+    if 'r2d2_compiler' in credentials and 'R2D2_COMPILER' not in os.environ:
+        os.environ['R2D2_COMPILER'] = credentials['r2d2_compiler']
+        logger.info(
+            f"YAML r2d2_compiler ({credentials['r2d2_compiler']!r}) overrides "
+            f"platform default ({r2d2_compiler!r})"
+        )


I can understand changing host to the more-specific r2d2_host now that we also have r2d2_server_host in the mix. My only concern is that this will require users to change this file once this PR goes in. Maybe we can consider accommodating both keys, keeping host and compiler as functional but deprecated? We can even throw a deprecation warning if the old names are detected.

mer-a-o · 2026-05-07T23:03:22Z

-# Platform configuration  
-host: discover-gmao
-compiler: intel
+1. Single server — put credentials at the root level:


Some lines in this yaml file should be comments?

mer-a-o · 2026-05-07T23:04:36Z

+    @dataclass
+    class r2d2_server(SuiteQuestion):
+        default_value: str = "gmao_server"
+        default_value: str | None = None


default_value is defined twice here

ftgoktas added 5 commits February 20, 2026 15:19

Create a doc to test AWS server

e2fb945

Create instructions

c924870

Add AWS credential configs

0f70277

Update docs

42e1235

Update docs

2363afa

ftgoktas added r2d2 labels Mar 11, 2026

Fix tests

6cbeae9

ftgoktas requested review from Dooruk, jeromebarre, mer-a-o and mranst March 12, 2026 17:46

ftgoktas added 7 commits March 12, 2026 13:57

Edit doc sections

152de31

Create a doc to test AWS server

60844b8

Create instructions

7c366d6

Add AWS credential configs

0fe5b81

Update docs

4a8835e

Update docs

5e4fdd0

Remove outdated comments

bede62e

mer-a-o reviewed Mar 16, 2026

View reviewed changes

shiklomanov-an reviewed Mar 16, 2026

View reviewed changes

Comment thread src/swell/utilities/r2d2.py Outdated

ftgoktas added 3 commits March 16, 2026 15:26

Fix typo (#733)

d9c6258

Merge remote-tracking branch 'origin/feature/aws-datastore' into feat…

2001b99

…ure/aws-datastore

Fix codestyle (#733)

707cdd7

ftgoktas marked this pull request as ready for review March 17, 2026 19:37

ftgoktas added 3 commits March 17, 2026 14:44

Merge branch 'develop' into feature/aws-datastore

61777b5

Update docs and ingest obs dates (#733)

d0a8d5a

Update available marine obs and docs (#733)

3789deb

mer-a-o reviewed Mar 23, 2026

View reviewed changes

Comment thread docs/configuring_aws_server.md Outdated

Corect doc dates (#733)

a033d05

jeromebarre reviewed Apr 10, 2026

View reviewed changes

Comment thread src/swell/utilities/r2d2.py Outdated

jeromebarre reviewed Apr 10, 2026

View reviewed changes

ftgoktas added 2 commits April 10, 2026 16:13

Fix log wording

0687963

Update docs

0e5ffa4

mranst reviewed Apr 17, 2026

View reviewed changes

Comment thread docs/examples/r2d2/ingest_obs.md

ftgoktas added 2 commits April 20, 2026 14:03

Add multiple datastore usage in credentials (#733)

e2246c7

Merge branch 'develop' into feature/aws-datastore

d8dd10a

ftgoktas marked this pull request as draft April 20, 2026 18:13

ftgoktas added 6 commits April 20, 2026 15:13

Add multiple r2d2 server usage (#733)

4bc38c7

Add a test script

43190de

Fix swell create credential loading for the new yaml

cda6b63

FIx credential loading

2011cf0

Fix codestyle

8030cbc

Update docs

e554978

ftgoktas marked this pull request as ready for review April 21, 2026 19:52

ftgoktas requested review from jeromebarre, mer-a-o and mranst April 21, 2026 19:54

mranst reviewed Apr 28, 2026

View reviewed changes

Address comments

190b202

mer-a-o reviewed May 7, 2026

View reviewed changes

Conversation

ftgoktas commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ftgoktas commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jeromebarre Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

ftgoktas Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

jeromebarre left a comment

Choose a reason for hiding this comment

Uh oh!

mranst left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ftgoktas commented Apr 21, 2026

Uh oh!

mranst left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mranst Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mranst Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mer-a-o May 7, 2026

Choose a reason for hiding this comment

Uh oh!

mer-a-o May 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ftgoktas commented Mar 11, 2026 •

edited

Loading

ftgoktas commented Mar 17, 2026 •

edited

Loading