Skip to content

chore: update span buffer and min flush count #13817

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 8 commits into from
Closed

chore: update span buffer and min flush count #13817

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
54dbc8f
Update span buffer and min flush count
juanjux Jun 30, 2025
0147be7
changelog
juanjux Jun 30, 2025
d862b24
Move the min flush spans to _constants.py
juanjux Jun 30, 2025
c47e04e
Update values also in tests
juanjux Jun 30, 2025
ea5b03d
Update values also in tests
juanjux Jun 30, 2025
8aed7ef
Merge branch 'main' into juanjux/update-traces-default-sizes
juanjux Jun 30, 2025
57a0e21
fmt
juanjux Jun 30, 2025
205c70f
Update values also in tests
juanjux Jun 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions ddtrace/internal/constants.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,9 @@
SPAN_API_DATADOG = "datadog"
SPAN_API_OTEL = "otel"
SPAN_API_OPENTRACING = "opentracing"
DEFAULT_BUFFER_SIZE = 20 << 20 # 20 MB
DEFAULT_MAX_PAYLOAD_SIZE = 20 << 20 # 20 MB
DEFAULT_BUFFER_SIZE = 30 << 20 # 30 MB
DEFAULT_MAX_PAYLOAD_SIZE = 30 << 20 # 30 MB
DEFAULT_FLUSH_MIN_SPANS = 200
DEFAULT_PROCESSING_INTERVAL = 1.0
DEFAULT_REUSE_CONNECTIONS = False
BLOCKED_RESPONSE_HTML = """<!DOCTYPE html><html lang="en"><head><meta charset="UTF-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>You've been blocked</title><style>a,body,div,html,span{margin:0;padding:0;border:0;font-size:100%;font:inherit;vertical-align:baseline}body{background:-webkit-radial-gradient(26% 19%,circle,#fff,#f4f7f9);background:radial-gradient(circle at 26% 19%,#fff,#f4f7f9);display:-webkit-box;display:-ms-flexbox;display:flex;-webkit-box-pack:center;-ms-flex-pack:center;justify-content:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;-ms-flex-line-pack:center;align-content:center;width:100%;min-height:100vh;line-height:1;flex-direction:column}p{display:block}main{text-align:center;flex:1;display:-webkit-box;display:-ms-flexbox;display:flex;-webkit-box-pack:center;-ms-flex-pack:center;justify-content:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;-ms-flex-line-pack:center;align-content:center;flex-direction:column}p{font-size:18px;line-height:normal;color:#646464;font-family:sans-serif;font-weight:400}a{color:#4842b7}footer{width:100%;text-align:center}footer p{font-size:16px}</style></head><body><main><p>Sorry, you cannot access this page. Please contact the customer service team.</p></main><footer><p>Security provided by <a href="https://www.datadoghq.com/product/security-platform/application-security-monitoring/" target="_blank">Datadog</a></p></footer></body></html>""" # noqa: E501
Expand Down
3 changes: 2 additions & 1 deletion ddtrace/settings/_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
from ..internal.constants import _PROPAGATION_STYLE_DEFAULT
from ..internal.constants import _PROPAGATION_STYLE_NONE
from ..internal.constants import DEFAULT_BUFFER_SIZE
from ..internal.constants import DEFAULT_FLUSH_MIN_SPANS
from ..internal.constants import DEFAULT_MAX_PAYLOAD_SIZE
from ..internal.constants import DEFAULT_PROCESSING_INTERVAL
from ..internal.constants import DEFAULT_REUSE_CONNECTIONS
Expand Down Expand Up @@ -455,7 +456,7 @@ def __init__(self):
self._trace_rate_limit,
)
self._partial_flush_enabled = _get_config("DD_TRACE_PARTIAL_FLUSH_ENABLED", True, asbool)
self._partial_flush_min_spans = _get_config("DD_TRACE_PARTIAL_FLUSH_MIN_SPANS", 300, int)
self._partial_flush_min_spans = _get_config("DD_TRACE_PARTIAL_FLUSH_MIN_SPANS", DEFAULT_FLUSH_MIN_SPANS, int)

self._http = HttpConfig(header_tags=self._trace_http_header_tags)
self._remote_config_enabled = _get_config("DD_REMOTE_CONFIGURATION_ENABLED", True, asbool)
Expand Down
6 changes: 3 additions & 3 deletions docs/configuration.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -246,7 +246,7 @@ Traces

DD_TRACE_PARTIAL_FLUSH_MIN_SPANS:
type: Integer
default: 300
default: 200
description: Maximum number of spans sent per trace per payload when ``DD_TRACE_PARTIAL_FLUSH_ENABLED=True``.

DD_TRACE_PROPAGATION_EXTRACT_FIRST:
Expand Down Expand Up @@ -307,7 +307,7 @@ Traces

DD_TRACE_WRITER_BUFFER_SIZE_BYTES:
type: Int
default: 8388608
default: 31457280
description: The max size in bytes of traces to buffer between flushes to the agent.

DD_TRACE_WRITER_INTERVAL_SECONDS:
Expand All @@ -317,7 +317,7 @@ Traces

DD_TRACE_WRITER_MAX_PAYLOAD_SIZE_BYTES:
type: Int
default: 8388608
default: 31457280

description: |
The max size in bytes of each payload item sent to the trace agent. If the max payload size is greater than buffer size,
Expand Down
5 changes: 5 additions & 0 deletions releasenotes/notes/update-span-buffer-sizes-18552a87bf54f8ef.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
fixes:
- |
- Increase ``DD_TRACE_BUFFER_SIZE_BYTES`` default value to 30MB to avoid potential OOMs under heavy load situations.
- Decrease ``DD_TRACE_PARTIAL_FLUSH_MIN_SPANS`` to 200.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why?

10 changes: 5 additions & 5 deletions tests/integration/test_integration.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,7 @@ def test_metrics():
from tests.utils import AnyInt
from tests.utils import override_global_config

assert t._span_aggregator.partial_flush_min_spans == 300
assert t._span_aggregator.partial_flush_min_spans == 30 << 20

with override_global_config(dict(_health_metrics_enabled=True)):
statsd_mock = mock.Mock()
Expand Down Expand Up @@ -652,8 +652,8 @@ def test_writer_configured_correctly_from_env():
def test_writer_configured_correctly_from_env_defaults():
import ddtrace

assert ddtrace.tracer._span_aggregator.writer._encoder.max_size == 20 << 20
assert ddtrace.tracer._span_aggregator.writer._encoder.max_item_size == 20 << 20
assert ddtrace.tracer._span_aggregator.writer._encoder.max_size == 30 << 20
assert ddtrace.tracer._span_aggregator.writer._encoder.max_item_size == 30 << 20
assert ddtrace.tracer._span_aggregator.writer._interval == 1.0


Expand Down Expand Up @@ -681,8 +681,8 @@ def test_writer_configured_correctly_from_env_defaults_under_ddtrace_run(ddtrace
"""
import ddtrace

assert ddtrace.tracer._span_aggregator.writer._encoder.max_size == 20 << 20
assert ddtrace.tracer._span_aggregator.writer._encoder.max_item_size == 20 << 20
assert ddtrace.tracer._span_aggregator.writer._encoder.max_size == 30 << 20
assert ddtrace.tracer._span_aggregator.writer._encoder.max_item_size == 30 << 20
assert ddtrace.tracer._span_aggregator.writer._interval == 1.0
""",
)
Expand Down
Loading