Add Matrix/Synapse parser whitelist and description#1726
Add Matrix/Synapse parser whitelist and description#1726pscriptos wants to merge 2 commits intocrowdsecurity:masterfrom
Conversation
|
Hello, Thanks for the PR. Unfortunately, I think using a parser whitelist for that is a bit too broad: you are effectively ignoring every request on anything that starts with those paths. A much better solution would be to use a postoverflow to only allow those paths for specific scenarios. |
|
Thank you very much for your feedback. |
|
Hi, I'm facing the same issue, then came here, I did follow the link about postoverflow, here is a file I'm now trying name: synapse-client-whitelists
description: Whitelist synapse client application for crawl non static
whitelist:
reason: synapse-client can trigger FP
expression:
- evt.Overflow.Alert.Scenario == "crowdsecurity/http-crawl-non_statics" and all(evt.Overflow.Alert.Events, {.GetMeta("http_path") startsWith "/_matrix/client/"})
- evt.Overflow.Alert.Scenario == "crowdsecurity/http-probing" and all(evt.Overflow.Alert.Events, {.GetMeta("http_path") startsWith "/_matrix/client/"})I did add the http-probing scenario because in some rooms there are a lot of 404 😕 that trigger the ban [edit] |
|
The suggested postoverflow didn't work for me. Part of it is that some 404s caused by Nextcloud activity seem to be causing the bucket to overflow and then everything goes sideways. Based on what I'm seeing, a parser whitelist may be the better option, if there are other services that may also be generating 404s... Nextcloud is notorious for being bad about 404s... even with https://app.crowdsec.net/hub/author/crowdsecurity/configurations/nextcloud-whitelist . |
this PR is the whitelist about matrix/synapse |
There was a problem hiding this comment.
Pull request overview
Adds a new enrich-stage HTTP whitelist parser intended to prevent legitimate Matrix/Synapse endpoints from being counted toward generic HTTP scanning/crawling scenarios (e.g., http-probing, http-crawl-non_statics).
Changes:
- Introduces a Matrix/Synapse path-prefix whitelist parser (
/_matrix/,/_synapse/,/.well-known/matrix/). - Adds documentation describing why these endpoints commonly generate benign “suspicious-looking” HTTP patterns.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| parsers/s02-enrich/crowdsecurity/matrix-whitelist.yaml | Adds a whitelist parser matching Matrix/Synapse request paths. |
| parsers/s02-enrich/crowdsecurity/matrix-whitelist.md | Documents the rationale and covered Matrix/Synapse endpoints. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -0,0 +1,9 @@ | |||
| name: my/matrix-whitelist | |||
There was a problem hiding this comment.
name is set to my/matrix-whitelist, but all other parsers in parsers/s02-enrich/crowdsecurity/ use the crowdsecurity/ namespace (e.g. parsers/s02-enrich/crowdsecurity/actual-budget-whitelist.yaml:1). Using a different namespace is likely to fail hub conventions/linting and makes the parser harder to discover; rename it to crowdsecurity/matrix-whitelist (and keep it consistent with the file path).
| name: my/matrix-whitelist | |
| name: crowdsecurity/matrix-whitelist |
| @@ -0,0 +1,9 @@ | |||
| name: my/matrix-whitelist | |||
| description: "Whitelist Matrix/Synapse requests from NPMplus logs" | |||
There was a problem hiding this comment.
The description says "from NPMplus logs", but the filter matches any event with evt.Meta.service == 'http' and the standard http_access-log/http_error-log log types. Either scope the filter to NPMplus-specific events (if that’s the intent) or update the description to reflect that this whitelist applies to generic HTTP logs.
| description: "Whitelist Matrix/Synapse requests from NPMplus logs" | |
| description: "Whitelist Matrix/Synapse HTTP requests from generic http_access-log/http_error-log entries" |
| - "evt.Meta.http_path startsWith '/_matrix/'" | ||
| - "evt.Meta.http_path startsWith '/_synapse/'" | ||
| - "evt.Meta.http_path startsWith '/.well-known/matrix/'" |
There was a problem hiding this comment.
This whitelists all requests to /_matrix/ and /_synapse/ purely by path prefix. Unlike other HTTP app whitelists in this directory (which typically narrow by verb/status and specific endpoints), this creates a broad blind spot where probing/bruteforce against Synapse/Matrix endpoints would no longer contribute to remediation decisions. Consider narrowing the whitelist conditions (e.g., to specific verbs/statuses/endpoints known to FP) or moving the exception into the specific scenarios that FP (so Matrix-specific attacks can still be detected).
| - "evt.Meta.http_path startsWith '/_matrix/'" | |
| - "evt.Meta.http_path startsWith '/_synapse/'" | |
| - "evt.Meta.http_path startsWith '/.well-known/matrix/'" | |
| - "evt.Meta.http_verb == 'GET' && evt.Meta.http_path startsWith '/.well-known/matrix/'" |
| name: my/matrix-whitelist | ||
| description: "Whitelist Matrix/Synapse requests from NPMplus logs" | ||
| filter: "evt.Meta.service == 'http' && evt.Meta.log_type in ['http_access-log', 'http_error-log']" | ||
| whitelist: | ||
| reason: "Matrix federation/client traffic" | ||
| expression: | ||
| - "evt.Meta.http_path startsWith '/_matrix/'" | ||
| - "evt.Meta.http_path startsWith '/_synapse/'" | ||
| - "evt.Meta.http_path startsWith '/.well-known/matrix/'" |
There was a problem hiding this comment.
A new whitelist parser is added, but there is no corresponding .tests/matrix-whitelist/ (most existing whitelists here have dedicated tests, e.g. .tests/actual-budget-whitelist/, .tests/calibre-web-whitelist/, .tests/nextcloud-whitelist/). Add a minimal test fixture asserting that Matrix/Synapse paths are marked whitelisted and that non-Matrix paths are not.
Sorry, I wasn't clear enough. The combination of matrix/synapse and Nextcloud endpoints 404'ing made the postoverflow whitelist to fail. Or really, if at any point any other 404 happens during that window, that would make the postoverflow whitelist to fail thanks to The contents of this PR do seem to work for me, even if allows for a lot of request URLs. |
|
I'm also using Synapse and getting banned due to the scenario crowdsecurity/http-probing. |
Description
Add a parser whitelist for Matrix/Synapse traffic. This whitelist prevents legitimate Matrix federation, client-server API, Synapse admin/internal endpoints, and
.well-known/matrix/server discovery requests from being falsely flagged by CrowdSec scenarios such ashttp-probingorhttp-crawl-non_statics.Whitelisted paths:
/_matrix/— Federation and Client-Server API/_synapse/— Synapse Admin and internal endpoints/.well-known/matrix/— Matrix server discoveryChecklist