We should add the ability to dynamically set the S3 path prefix and suffix based on common variables.
This would simplify user workflow especially when using date-based directories to separate projects in an S3 bucket. For example, a variable for YEAR would free users from having to remember to update the plugin configuration file when January 1st arrives.
A running list of variables to support:
Background/Implementation
The plugin currently supports the ability to configure prefix and suffix path components for S3 in a plugin-level configuration file, which is used to generate a project-level config.
The plugin-level config should support dynamic configurations such as below:
# ~/.datakit/plugins/datakit-data/config.json
{
"aws_user_profile": "stanford",
"s3_bucket": "stanford-data-projects",
"s3_path_prefix": "$YEAR/"
}