Skip to content

Support dynamic variables for S3 path configs #22

@zstumgoren

Description

@zstumgoren

We should add the ability to dynamically set the S3 path prefix and suffix based on common variables.

This would simplify user workflow especially when using date-based directories to separate projects in an S3 bucket. For example, a variable for YEAR would free users from having to remember to update the plugin configuration file when January 1st arrives.

A running list of variables to support:

  • YEAR
  • MONTH
  • DAY
  • PROJECTNAME
  • USERNAME

Background/Implementation

The plugin currently supports the ability to configure prefix and suffix path components for S3 in a plugin-level configuration file, which is used to generate a project-level config.

The plugin-level config should support dynamic configurations such as below:

# ~/.datakit/plugins/datakit-data/config.json
{
  "aws_user_profile": "stanford",
  "s3_bucket": "stanford-data-projects",
  "s3_path_prefix": "$YEAR/"
}

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions