Skip to content

Commit 1240a7e

Browse files
author
waterWang
committed
fix: bound the usage event rewind to prevent unbounded re-aggregation
UsageManagerImpl.parse() rewinds the aggregation start date to the oldest unprocessed event, but has no bound on how far back it can go. If an event cannot be successfully processed (e.g. references a removed entity), the rewind pins the window to that event's date permanently. Each subsequent run re-aggregates from that date to the present, growing by one aggregation period per run. This causes unbounded growth of cloud_usage (54M+ rows reported) and exec_time (42+ minutes per hour). Fix: bound the rewind to 24 hours. The rewind exists to absorb clock skew between the cloud and usage databases, not to replay history. Events older than 24 hours from the current window start will still be retried, but the aggregation window will not be rewound to them. Fixes #13906 Signed-off-by: waterWang <waterwang@proton.me>
1 parent d45d481 commit 1240a7e

1 file changed

Lines changed: 5 additions & 1 deletion

File tree

usage/src/main/java/com/cloud/usage/UsageManagerImpl.java

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,7 @@ public class UsageManagerImpl extends ManagerBase implements UsageManager, Runna
103103
private static final int HOURLY_TIME = 60;
104104
private static final int DAILY_TIME = 60 * 24;
105105
private static final int THREE_DAYS_IN_MINUTES = 60 * 24 * 3;
106+
private static final long MAX_EVENT_REWIND_MILLIS = 24L * 60 * 60 * 1000;
106107

107108
@Inject
108109
private AccountDao _accountDao;
@@ -699,7 +700,10 @@ public void parse(UsageJobVO job, long startDateMillis, long endDateMillis) {
699700
if ((events != null) && (events.size() > 0)) {
700701
Date oldestEventDate = events.get(0).getCreateDate();
701702
if (oldestEventDate.getTime() < startDateMillis) {
702-
startDateMillis = oldestEventDate.getTime();
703+
// Bound the rewind so a single un-processable event cannot pin the
704+
// aggregation window to an arbitrarily old date and cause unbounded
705+
// re-aggregation of the entire history on every run.
706+
startDateMillis = Math.max(oldestEventDate.getTime(), startDateMillis - MAX_EVENT_REWIND_MILLIS);
703707
startDate = new Date(startDateMillis);
704708
}
705709

0 commit comments

Comments
 (0)