Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion mission-control/docs/guide/config-db/scrapers/aws.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@ The Mission Control Registry includes an [AWS](/integration/aws) Helm chart that
:::

```yaml title="aws-scraper.yaml" file=<rootDir>/modules/config-db/fixtures/aws.yaml

```

| Field | Description | Scheme | Required |
Expand Down
106 changes: 106 additions & 0 deletions mission-control/docs/guide/config-db/scrapers/clickhouse.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
---
title: Clickhouse
show_title: false
sidebar_custom_props:
icon: clickhouse-icon
---


# <Icon name="clickhouse-icon"/> Clickhouse

The Clickhouse scraper will execute a query and then create a catalog item for each returned row.

The example below creates a new `Order` config for each row, with name as `order_id` and the row in json format as the config.

```yaml title="clickhouse-scraper.yaml" file=<rootDir>/modules/config-db/fixtures/clickhouse.yaml
```

Objects in external storages like AzureBlobStorage or AWS S3 can also be queried

```yaml title="azure-blob-storage-scraper.yaml"
apiVersion: mission-control.flanksource.com/v1
kind: Connection
metadata:
name: azure-prod
spec:
azure:
clientID:
valueFrom:
secretKeyRef:
name: azure-creds
key: CLIENT_ID
clientSecret:
valueFrom:
secretKeyRef:
name: azure-creds
key: CLIENT_SECRET
tenantID:
valueFrom:
secretKeyRef:
name: azure-creds
key: TENANT_ID
---
apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
name: audit-logs
spec:
clickhouse:
- query: SELECT * FROM azureBlobStorage(azure_prod_audit_data);
type: AuditRecord
id: $.user_id
transform:
expr: "[config].toJSON()"
azureBlobStorage:
connection: azure-prod
account: acme-logs
container: audit
path: audit-data.csv
collection: azure_prod_audit_data
Comment on lines +49 to +59
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update this so that it reflects actual azure audit logs

```

:::info
An instance of clichkouse server needs to be deployed for this scraper to function.

Mission Control can deploy an instance by setting `config-db.clichkouse.enabled: true` in the helm chart.

An external clichkouse server can also be used via the `clickhouseURL` parameter
:::

## Scraper

| Field | Description | Scheme | Required |
| ----------- | ---------------------------------------------------------------------------------- | -------------------------------------------- | -------- |
| `logLevel` | Specify the level of logging. | `string` | |
| `schedule` | Specify the interval to scrape in cron format. Defaults to every 60 minutes. | `string` | |
| `full` | Set to `true` to extract changes from scraped configurations. Defaults to `false`. | `bool` | |
| `retention` | Settings for retaining changes, analysis and scraped items | [`Retention`](/guide/config-db/concepts/retention) | |
| `clickhouse`| Specifies the list of SQL configurations to scrape. | [`[]Clickhouse`](#clickhouse) | |

## Clickhouse

<CustomScraper rows={[
{field: "awsS3", description: "AWS S3 configuration for accessing buckets", scheme: "[AWSS3](#awss3)"},
{field: "azureBlobStorage", description: "Azure Blob Storage configuration for accessing storage", scheme: "[AzureBlobStorage](#azureblobstorage)"},
{field: "clickhouseURL", description: "ClickHouse connection URL in format: clickhouse://<user>:<password>@<host>:<port>/<database>?param1=value1&param2=value2", scheme: "string"},
{field: "query", description: "Query to execute", scheme: "string", required: true}
]}/>


## AWSS3

<Fields connection="aws" rows={[
{field: "bucket", description: "AWS S3 bucket name", scheme: "string"},
{field: "path", description: "Path within the S3 bucket", scheme: "string"},
{field: "endpoint", description: "Custom endpoint URL for AWS S3", scheme: "string"}
]}/>

## AzureBlobStorage

<Fields connection="azure" rows={[
{field: "account", description: "Azure storage account name", scheme: "string"},
{field: "container", description: "Azure Blob Storage container name", scheme: "string"},
{field: "path", description: "Path within the container", scheme: "string"},
{field: "endpointSuffix", description: "Azure endpoint suffix", scheme: "string"},
{field: "collectionName", description: "Name of the collection in Clickhouse. See [Named Collections](https://clickhouse.com/docs/operations/named-collections)", scheme: "string", required: true}
]}/>
14 changes: 7 additions & 7 deletions mission-control/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion mission-control/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
"@docusaurus/core": "3.7.0",
"@docusaurus/plugin-client-redirects": "3.7.0",
"@docusaurus/preset-classic": "3.7.0",
"@flanksource/icons": "^1.0.34",
"@flanksource/icons": "^1.0.35",
"@floating-ui/react": "^0.26.28",
"@iconify/react": "^5.1.0",
"@mdx-js/react": "^3.0.0",
Expand Down