Skip to content

Commit 7120e66

Browse files
verejoelverejoel
committed
feat: add flags and deprecate old approach for storage config
1 parent 3bce0f7 commit 7120e66

File tree

8 files changed

+224
-44
lines changed

8 files changed

+224
-44
lines changed

‎README.md‎

Lines changed: 75 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -13,15 +13,43 @@ We recommend to use `nix` to fulfill all development dependencies. Visit [Nix Do
1313
- to build the binary run `nix-shell --run 'make build'`
1414
- to run tests run `nix-shell --run 'make test'`
1515

16+
## Configuration
17+
18+
This gateway supports the same object storage configuration as the main Thanos project. You can use the `--objstore.config-file` or `--objstore.config` flags to specify your storage backend.
19+
20+
### Supported Storage Backends
21+
22+
Thanks to the integration with [thanos-io/objstore](https://github.com/thanos-io/objstore), the gateway supports all major cloud storage providers:
23+
24+
- **Amazon S3** (and S3-compatible like MinIO)
25+
- **Google Cloud Storage (GCS)**
26+
- **Microsoft Azure Blob Storage**
27+
- **OpenStack Swift**
28+
- **Tencent Cloud Object Storage (COS)**
29+
- **Alibaba Cloud Object Storage Service (OSS)**
30+
- **Local Filesystem**
31+
32+
### Configuration Examples
33+
34+
See the [`examples/`](examples/) directory for configuration examples:
35+
36+
- [`objstore-s3.yml`](examples/objstore-s3.yml) - Amazon S3 configuration
37+
- [`objstore-gcs.yml`](examples/objstore-gcs.yml) - Google Cloud Storage configuration
38+
- [`objstore-azure.yml`](examples/objstore-azure.yml) - Azure Blob Storage configuration
39+
- [`objstore-filesystem.yml`](examples/objstore-filesystem.yml) - Local filesystem configuration
40+
41+
For complete configuration format details, see the [Thanos Storage Documentation](https://thanos.io/tip/thanos/storage.md/#configuration).
42+
1643
## Running
1744

1845
### Server
1946

20-
Once built, you can run the server using something like:
47+
Once built, you can run the server using:
2148

2249
```bash
23-
parquet-gateway serve \
24-
--storage.prefix my-prefix \
50+
# Using config file (recommended)
51+
thanos-parquet-gateway serve \
52+
--objstore.config-file=examples/objstore-s3.yml \
2553
--http.internal.port=6060 \
2654
--http.prometheus.port=9090 \
2755
--http.thanos.port=9091 \
@@ -31,11 +59,23 @@ parquet-gateway serve \
3159
--block.discovery.concurrency=32 \
3260
--query.external-label=prometheus=my-prometheus \
3361
--query.external-label=replica=ha-1
62+
63+
# Using inline config
64+
thanos-parquet-gateway serve \
65+
--objstore.config="
66+
type: S3
67+
config:
68+
bucket: my-bucket
69+
endpoint: s3.amazonaws.com
70+
" \
71+
--http.internal.port=6060 \
72+
--http.prometheus.port=9090 \
73+
--http.thanos.port=9091
3474
```
3575

3676
This will:
3777

38-
- load blocks from the `.data/my-prefix` directory
78+
- load blocks from your configured object storage
3979
- expose internal metrics and readiness handlers on port 6060
4080
- expose a subset of the Prometheus HTTP API on port 9090
4181
- expose an Thanos Info, Series and Query gRPC service on port 9091
@@ -62,19 +102,45 @@ curl 'http://0.0.0.0:9000/api/v1/query' \
62102
]
63103
}
64104
}
65-
66105
```
67106

68107
### Converter
69108

70-
To convert TSDB blocks in the `.data/source` directory that overlap `09/2021` and write the resulting parquet files into the `.data/destination` directory.
109+
To convert TSDB blocks from one storage to another, you can specify separate configurations for source (TSDB) and destination (Parquet) storage:
71110

72111
```bash
73-
parquet-gateway convert \
74-
--tsdb.storage.prefix source \
75-
--parquet.storage.prefix destination \
112+
# Using separate config files
113+
thanos-parquet-gateway convert \
114+
--tsdb.objstore.config-file=tsdb-storage.yml \
115+
--parquet.objstore.config-file=parquet-storage.yml \
76116
--convert.sorting.label=__name__ \
77117
--convert.sorting.label=namespace
118+
119+
# Using inline configs
120+
thanos-parquet-gateway convert \
121+
--tsdb.objstore.config="
122+
type: S3
123+
config:
124+
bucket: tsdb-source-bucket
125+
" \
126+
--parquet.objstore.config="
127+
type: S3
128+
config:
129+
bucket: parquet-destination-bucket
130+
" \
131+
--convert.sorting.label=__name__
132+
```
133+
134+
### Legacy Configuration Support
135+
136+
For backward compatibility, the gateway still supports the legacy storage flags, but they are deprecated and will show a warning. Please migrate to the new `--objstore.config-file` approach:
137+
138+
```bash
139+
# DEPRECATED - still works but shows warnings
140+
thanos-parquet-gateway serve \
141+
--storage.type=s3 \
142+
--storage.s3.bucket=my-bucket \
143+
--storage.s3.endpoint=s3.amazonaws.com
78144
```
79145

80146
## Note

‎cmd/config.go‎

Lines changed: 60 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ import (
1010
"log/slog"
1111
"net/http"
1212
"net/http/pprof"
13+
"os"
1314
"strings"
1415
"time"
1516

@@ -29,6 +30,7 @@ import (
2930
"github.com/thanos-io/objstore"
3031
"github.com/thanos-io/objstore/client"
3132
"github.com/thanos-io/thanos/pkg/runutil"
33+
"gopkg.in/alecthomas/kingpin.v2"
3234

3335
"github.com/thanos-io/thanos-parquet-gateway/locate"
3436
)
@@ -45,23 +47,70 @@ func setupInterrupt(ctx context.Context, g *run.Group, log *slog.Logger) {
4547
}
4648

4749
type bucketOpts struct {
48-
storage string
49-
prefix string
50+
// New Thanos-style configuration
51+
configFile string
52+
config string
5053

51-
// filesystem options
54+
// Legacy flags for backward compatibility (deprecated)
55+
storage string
56+
prefix string
5257
filesystemDirectory string
58+
s3Bucket string
59+
s3Endpoint string
60+
s3AccessKey string
61+
s3SecretKey string
62+
s3Insecure bool
63+
retries int
64+
}
5365

54-
// s3 options
55-
s3Bucket string
56-
s3Endpoint string
57-
s3AccessKey string
58-
s3SecretKey string
59-
s3Insecure bool
60-
61-
retries int
66+
// registerThanosStyleFlags registers the standard Thanos objstore flags
67+
func (opts *bucketOpts) registerThanosStyleFlags(cmd *kingpin.CmdClause, prefix string) {
68+
configFlagSuffix := ".objstore.config"
69+
configFlag := strings.TrimPrefix(prefix+configFlagSuffix, ".")
70+
cmd.Flag(configFlag, "Alternative to 'objstore.config-file' flag (mutually exclusive). Content of YAML file that contains object store configuration. See format details: https://thanos.io/tip/thanos/storage.md/#configuration").
71+
PlaceHolder("<content>").StringVar(&opts.config)
72+
73+
configFileFlagSuffix := ".objstore.config-file"
74+
configFileFlag := strings.TrimPrefix(prefix+configFileFlagSuffix, ".")
75+
cmd.Flag(configFileFlag, "Path to YAML file that contains object store configuration. See format details: https://thanos.io/tip/thanos/storage.md/#configuration").
76+
PlaceHolder("<file-path>").StringVar(&opts.configFile)
6277
}
6378

6479
func setupBucket(log *slog.Logger, opts bucketOpts) (objstore.Bucket, error) {
80+
// Priority: 1. New Thanos config, 2. Legacy flags for backward compatibility
81+
if opts.configFile != "" || opts.config != "" {
82+
return setupThanosBucket(log, opts)
83+
}
84+
85+
// Fall back to legacy setup with deprecation warning
86+
log.Warn("Using deprecated storage flags. Please migrate to --objstore.config-file. See https://thanos.io/tip/thanos/storage.md/#configuration")
87+
return setupLegacyBucket(log, opts)
88+
}
89+
90+
func setupThanosBucket(log *slog.Logger, opts bucketOpts) (objstore.Bucket, error) {
91+
var confContentYaml []byte
92+
var err error
93+
94+
if opts.configFile != "" {
95+
confContentYaml, err = os.ReadFile(opts.configFile)
96+
if err != nil {
97+
return nil, fmt.Errorf("reading config file: %w", err)
98+
}
99+
} else if opts.config != "" {
100+
confContentYaml = []byte(opts.config)
101+
} else {
102+
return nil, fmt.Errorf("no objstore configuration provided")
103+
}
104+
105+
bkt, err := client.NewBucket(slogAdapter{log}, confContentYaml, "thanos-parquet-gateway", nil)
106+
if err != nil {
107+
return nil, fmt.Errorf("creating bucket client: %w", err)
108+
}
109+
110+
return bkt, nil
111+
}
112+
113+
func setupLegacyBucket(log *slog.Logger, opts bucketOpts) (objstore.Bucket, error) {
65114
prov := objstore.ObjProvider(strings.ToUpper(opts.storage))
66115
cfg := client.BucketConfig{
67116
Type: prov,

‎cmd/convert.go‎

Lines changed: 40 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -79,25 +79,49 @@ func (opts *conversionOpts) registerFlags(cmd *kingpin.CmdClause) {
7979
}
8080

8181
func (opts *bucketOpts) registerConvertParquetFlags(cmd *kingpin.CmdClause) {
82-
cmd.Flag("parquet.storage.type", "type of storage").Default("filesystem").EnumVar(&opts.storage, "filesystem", "s3")
83-
cmd.Flag("parquet.storage.prefix", "prefix for the storage").Default("").StringVar(&opts.prefix)
84-
cmd.Flag("parquet.storage.filesystem.directory", "directory for filesystem").Default(".data").StringVar(&opts.filesystemDirectory)
85-
cmd.Flag("parquet.storage.s3.bucket", "bucket for s3").Default("").StringVar(&opts.s3Bucket)
86-
cmd.Flag("parquet.storage.s3.endpoint", "endpoint for s3").Default("").StringVar(&opts.s3Endpoint)
87-
cmd.Flag("parquet.storage.s3.access_key", "access key for s3").Default("").Envar("PARQUET_STORAGE_S3_ACCESS_KEY").StringVar(&opts.s3AccessKey)
88-
cmd.Flag("parquet.storage.s3.secret_key", "secret key for s3").Default("").Envar("PARQUET_STORAGE_S3_SECRET_KEY").StringVar(&opts.s3SecretKey)
89-
cmd.Flag("parquet.storage.s3.insecure", "use http").Default("false").BoolVar(&opts.s3Insecure)
82+
// Add new Thanos-style flags
83+
opts.registerThanosStyleFlags(cmd, "parquet")
84+
85+
// Mark existing flags as deprecated but keep them for backward compatibility
86+
cmd.Flag("parquet.storage.type", "DEPRECATED: Use --parquet.objstore.config-file instead. Type of storage").
87+
Default("filesystem").Hidden().EnumVar(&opts.storage, "filesystem", "s3")
88+
cmd.Flag("parquet.storage.prefix", "DEPRECATED: Use --parquet.objstore.config-file instead. Prefix for the storage").
89+
Default("").Hidden().StringVar(&opts.prefix)
90+
cmd.Flag("parquet.storage.filesystem.directory", "DEPRECATED: Use --parquet.objstore.config-file instead. Directory for filesystem").
91+
Default(".data").Hidden().StringVar(&opts.filesystemDirectory)
92+
cmd.Flag("parquet.storage.s3.bucket", "DEPRECATED: Use --parquet.objstore.config-file instead. Bucket for s3").
93+
Default("").Hidden().StringVar(&opts.s3Bucket)
94+
cmd.Flag("parquet.storage.s3.endpoint", "DEPRECATED: Use --parquet.objstore.config-file instead. Endpoint for s3").
95+
Default("").Hidden().StringVar(&opts.s3Endpoint)
96+
cmd.Flag("parquet.storage.s3.access_key", "DEPRECATED: Use --parquet.objstore.config-file instead. Access key for s3").
97+
Default("").Hidden().Envar("PARQUET_STORAGE_S3_ACCESS_KEY").StringVar(&opts.s3AccessKey)
98+
cmd.Flag("parquet.storage.s3.secret_key", "DEPRECATED: Use --parquet.objstore.config-file instead. Secret key for s3").
99+
Default("").Hidden().Envar("PARQUET_STORAGE_S3_SECRET_KEY").StringVar(&opts.s3SecretKey)
100+
cmd.Flag("parquet.storage.s3.insecure", "DEPRECATED: Use --parquet.objstore.config-file instead. Use http").
101+
Default("false").Hidden().BoolVar(&opts.s3Insecure)
90102
}
91103

92104
func (opts *bucketOpts) registerConvertTSDBFlags(cmd *kingpin.CmdClause) {
93-
cmd.Flag("tsdb.storage.type", "type of storage").Default("filesystem").EnumVar(&opts.storage, "filesystem", "s3")
94-
cmd.Flag("tsdb.storage.prefix", "prefix for the storage").Default("").StringVar(&opts.prefix)
95-
cmd.Flag("tsdb.storage.filesystem.directory", "directory for filesystem").Default(".data").StringVar(&opts.filesystemDirectory)
96-
cmd.Flag("tsdb.storage.s3.bucket", "bucket for s3").Default("").StringVar(&opts.s3Bucket)
97-
cmd.Flag("tsdb.storage.s3.endpoint", "endpoint for s3").Default("").StringVar(&opts.s3Endpoint)
98-
cmd.Flag("tsdb.storage.s3.access_key", "access key for s3").Default("").Envar("TSDB_STORAGE_S3_ACCESS_KEY").StringVar(&opts.s3AccessKey)
99-
cmd.Flag("tsdb.storage.s3.secret_key", "secret key for s3").Default("").Envar("TSDB_STORAGE_S3_SECRET_KEY").StringVar(&opts.s3SecretKey)
100-
cmd.Flag("tsdb.storage.s3.insecure", "use http").Default("false").BoolVar(&opts.s3Insecure)
105+
// Add new Thanos-style flags
106+
opts.registerThanosStyleFlags(cmd, "tsdb")
107+
108+
// Mark existing flags as deprecated but keep them for backward compatibility
109+
cmd.Flag("tsdb.storage.type", "DEPRECATED: Use --tsdb.objstore.config-file instead. Type of storage").
110+
Default("filesystem").Hidden().EnumVar(&opts.storage, "filesystem", "s3")
111+
cmd.Flag("tsdb.storage.prefix", "DEPRECATED: Use --tsdb.objstore.config-file instead. Prefix for the storage").
112+
Default("").Hidden().StringVar(&opts.prefix)
113+
cmd.Flag("tsdb.storage.filesystem.directory", "DEPRECATED: Use --tsdb.objstore.config-file instead. Directory for filesystem").
114+
Default(".data").Hidden().StringVar(&opts.filesystemDirectory)
115+
cmd.Flag("tsdb.storage.s3.bucket", "DEPRECATED: Use --tsdb.objstore.config-file instead. Bucket for s3").
116+
Default("").Hidden().StringVar(&opts.s3Bucket)
117+
cmd.Flag("tsdb.storage.s3.endpoint", "DEPRECATED: Use --tsdb.objstore.config-file instead. Endpoint for s3").
118+
Default("").Hidden().StringVar(&opts.s3Endpoint)
119+
cmd.Flag("tsdb.storage.s3.access_key", "DEPRECATED: Use --tsdb.objstore.config-file instead. Access key for s3").
120+
Default("").Hidden().Envar("TSDB_STORAGE_S3_ACCESS_KEY").StringVar(&opts.s3AccessKey)
121+
cmd.Flag("tsdb.storage.s3.secret_key", "DEPRECATED: Use --tsdb.objstore.config-file instead. Secret key for s3").
122+
Default("").Hidden().Envar("TSDB_STORAGE_S3_SECRET_KEY").StringVar(&opts.s3SecretKey)
123+
cmd.Flag("tsdb.storage.s3.insecure", "DEPRECATED: Use --tsdb.objstore.config-file instead. Use http").
124+
Default("false").Hidden().BoolVar(&opts.s3Insecure)
101125
}
102126

103127
func (opts *discoveryOpts) registerConvertParquetFlags(cmd *kingpin.CmdClause) {

‎cmd/serve.go‎

Lines changed: 20 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -63,14 +63,26 @@ func (opts *serveOpts) registerFlags(cmd *kingpin.CmdClause) {
6363
}
6464

6565
func (opts *bucketOpts) registerServeFlags(cmd *kingpin.CmdClause) {
66-
cmd.Flag("storage.type", "type of storage").Default("filesystem").EnumVar(&opts.storage, "filesystem", "s3")
67-
cmd.Flag("storage.prefix", "prefix for the storage").Default("").StringVar(&opts.prefix)
68-
cmd.Flag("storage.filesystem.directory", "directory for filesystem").Default(".data").StringVar(&opts.filesystemDirectory)
69-
cmd.Flag("storage.s3.bucket", "bucket for s3").Default("").StringVar(&opts.s3Bucket)
70-
cmd.Flag("storage.s3.endpoint", "endpoint for s3").Default("").StringVar(&opts.s3Endpoint)
71-
cmd.Flag("storage.s3.access_key", "access key for s3").Default("").Envar("STORAGE_S3_ACCESS_KEY").StringVar(&opts.s3AccessKey)
72-
cmd.Flag("storage.s3.secret_key", "secret key for s3").Default("").Envar("STORAGE_S3_SECRET_KEY").StringVar(&opts.s3SecretKey)
73-
cmd.Flag("storage.s3.insecure", "use http").Default("false").BoolVar(&opts.s3Insecure)
66+
// Add new Thanos-style flags
67+
opts.registerThanosStyleFlags(cmd, "")
68+
69+
// Mark existing flags as deprecated but keep them for backward compatibility
70+
cmd.Flag("storage.type", "DEPRECATED: Use --objstore.config-file instead. Type of storage").
71+
Default("filesystem").Hidden().EnumVar(&opts.storage, "filesystem", "s3")
72+
cmd.Flag("storage.prefix", "DEPRECATED: Use --objstore.config-file instead. Prefix for the storage").
73+
Default("").Hidden().StringVar(&opts.prefix)
74+
cmd.Flag("storage.filesystem.directory", "DEPRECATED: Use --objstore.config-file instead. Directory for filesystem").
75+
Default(".data").Hidden().StringVar(&opts.filesystemDirectory)
76+
cmd.Flag("storage.s3.bucket", "DEPRECATED: Use --objstore.config-file instead. Bucket for s3").
77+
Default("").Hidden().StringVar(&opts.s3Bucket)
78+
cmd.Flag("storage.s3.endpoint", "DEPRECATED: Use --objstore.config-file instead. Endpoint for s3").
79+
Default("").Hidden().StringVar(&opts.s3Endpoint)
80+
cmd.Flag("storage.s3.access_key", "DEPRECATED: Use --objstore.config-file instead. Access key for s3").
81+
Default("").Hidden().Envar("STORAGE_S3_ACCESS_KEY").StringVar(&opts.s3AccessKey)
82+
cmd.Flag("storage.s3.secret_key", "DEPRECATED: Use --objstore.config-file instead. Secret key for s3").
83+
Default("").Hidden().Envar("STORAGE_S3_SECRET_KEY").StringVar(&opts.s3SecretKey)
84+
cmd.Flag("storage.s3.insecure", "DEPRECATED: Use --objstore.config-file instead. Use http").
85+
Default("false").Hidden().BoolVar(&opts.s3Insecure)
7486
}
7587

7688
func (opts *tracingOpts) registerServeFlags(cmd *kingpin.CmdClause) {

‎examples/objstore-azure.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
type: AZURE
2+
config:
3+
storage_account: ""
4+
storage_account_key: ""
5+
container: "thanos"
6+
endpoint: ""
7+
max_retries: 0

‎examples/objstore-filesystem.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
type: FILESYSTEM
2+
config:
3+
directory: "./data"

‎examples/objstore-gcs.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
type: GCS
2+
config:
3+
bucket: "my-thanos-bucket"
4+
service_account: ""

‎examples/objstore-s3.yml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
type: S3
2+
config:
3+
bucket: "my-thanos-bucket"
4+
endpoint: "s3.amazonaws.com"
5+
region: "us-east-1"
6+
access_key: ""
7+
secret_key: ""
8+
insecure: false
9+
signature_version2: false
10+
put_user_metadata: {}
11+
http_config:
12+
idle_conn_timeout: 90s
13+
response_header_timeout: 2m
14+
trace:
15+
enable: false

0 commit comments

Comments
 (0)