telemetry: schema validation, current vs lifetime Instance statistics

Address the next round of the architectural review on the telemetry feature.

Rename the reporter/ directory to telemetry/ to match what the feature is
called since the telemetry.* settings rename.

The report now carries a schema_version and the collector validates every
payload against the exact schema belonging to that version, on top of the
existing generic sanity limits: all sections and counters must be present, no
other keys are accepted anywhere and every value must have the expected type.
An arbitrary well-formed JSON object no longer makes it to disk, and any
change to the report structure requires incrementing the version and teaching
the collector the new schema.

Rows of removed Instances are kept in the database, so both current and
lifetime statistics can be reported, and they mean different things. The
instances section is now split in two: "current" covers the non-removed rows
in any state, Destroyed Instances that have not been expunged yet included,
while "lifetime" also counts the removed rows and therefore describes every
Instance which ever existed in the cloud, as total/removed counts plus
hypervisor and type counters.

Document in the README that the installation ID is not guaranteed globally
unique but a collision is negligible (two databases created with the same
version in the same second), that a cloned or restored database intentionally
reports as the same installation, and that the collector cannot verify a
report was genuinely produced by a Management Server: the source is open, so
the statistics are best-effort by nature, with validation, rate limiting and
the per-ID storage bound limiting abuse.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: wido

server/src/main/java/org/apache/cloudstack/report/UsageReporter.java

@@ -79,6 +79,14 @@ public class UsageReporter extends ManagerBase implements Configurable {
      */
     protected static final String TELEMETRY_URI = "https://call-home.cloudstack.org/report";
 
+    /**
+     * The version of the report schema, sent along in every report. The collector
+     * validates the received payload against the schema belonging to this version,
+     * so any change to the structure of the report requires incrementing this
+     * version and teaching the collector the new schema.
+     */
+    protected static final int SCHEMA_VERSION = 1;
+
     public static final ConfigKey<Integer> TelemetryInterval = new ConfigKey<>("Advanced", Integer.class,
             "telemetry.interval", "0",
             "The interval in days between telemetry reports sent to the CloudStack project. 0 is the default (disabled) and when enabled a value of 7 is recommended. Changing this setting requires a restart of the Management Server.",
@@ -353,18 +361,48 @@ private Map<String, AtomicLongMap> getDataCenterReport() {
         return datacenterMap;
     }
 
-    private Map<String, AtomicLongMap> getInstanceReport() {
+    /**
+     * Rows of removed Instances are kept in the database, so both current and
+     * lifetime statistics can be reported. They mean different things: "current"
+     * describes the Instances the cloud holds right now, in any state including
+     * Destroyed ones which have not been expunged yet, while "lifetime" also
+     * counts the removed rows and therefore describes every Instance which ever
+     * existed in this cloud.
+     */
+    private Map<String, Object> getInstanceReport() {
 
-        Map<String, AtomicLongMap> instanceMap = new HashMap<String, AtomicLongMap>();
+        Map<String, AtomicLongMap> current = new HashMap<String, AtomicLongMap>();
         AtomicLongMap<Object> hypervisor_type = AtomicLongMap.create();
         AtomicLongMap<Object> instance_state = AtomicLongMap.create();
         AtomicLongMap<Object> instance_type = AtomicLongMap.create();
         AtomicLongMap<Object> ha_enabled = AtomicLongMap.create();
         AtomicLongMap<Object> dynamically_scalable = AtomicLongMap.create();
 
+        Map<String, Object> lifetime = new HashMap<String, Object>();
+        AtomicLongMap<Object> lifetime_hypervisor_type = AtomicLongMap.create();
+        AtomicLongMap<Object> lifetime_type = AtomicLongMap.create();
+
+        long total = 0;
+        long removed = 0;
+
         SearchCriteria<VMInstanceVO> vm_sc = _vmInstance.createSearchCriteria();
-        List<VMInstanceVO> vms = _vmInstance.search(vm_sc, null);
+        List<VMInstanceVO> vms = _vmInstance.searchIncludingRemoved(vm_sc, null, null, false);
         for (VMInstanceVO vmVO : vms) {
+            total++;
+
+            if (vmVO.getHypervisorType() != null) {
+                lifetime_hypervisor_type.getAndIncrement(vmVO.getHypervisorType());
+            }
+
+            if (vmVO.getType() != null) {
+                lifetime_type.getAndIncrement(vmVO.getType());
+            }
+
+            if (vmVO.getRemoved() != null) {
+                removed++;
+                continue;
+            }
+
             if (vmVO.getHypervisorType() != null) {
                 hypervisor_type.getAndIncrement(vmVO.getHypervisorType());
             }
@@ -381,11 +419,20 @@ private Map<String, AtomicLongMap> getInstanceReport() {
             dynamically_scalable.getAndIncrement(vmVO.isDynamicallyScalable());
         }
 
-        instanceMap.put("hypervisor_type", hypervisor_type);
-        instanceMap.put("state", instance_state);
-        instanceMap.put("type", instance_type);
-        instanceMap.put("ha_enabled", ha_enabled);
-        instanceMap.put("dynamically_scalable", dynamically_scalable);
+        current.put("hypervisor_type", hypervisor_type);
+        current.put("state", instance_state);
+        current.put("type", instance_type);
+        current.put("ha_enabled", ha_enabled);
+        current.put("dynamically_scalable", dynamically_scalable);
+
+        lifetime.put("total", total);
+        lifetime.put("removed", removed);
+        lifetime.put("hypervisor_type", lifetime_hypervisor_type);
+        lifetime.put("type", lifetime_type);
+
+        Map<String, Object> instanceMap = new HashMap<String, Object>();
+        instanceMap.put("current", current);
+        instanceMap.put("lifetime", lifetime);
 
         return instanceMap;
     }
@@ -450,6 +497,7 @@ protected void runInContext() {
 
                 Map<String, Object> reportMap = new HashMap<String, Object>();
 
+                reportMap.put("schema_version", SCHEMA_VERSION);
                 reportMap.put("hosts", getHostReport());
                 reportMap.put("clusters", getClusterReport());
                 reportMap.put("primaryStorage", getStoragePoolReport());

server/src/test/java/org/apache/cloudstack/report/UsageReporterTest.java

@@ -214,8 +214,25 @@ private void stubInstances() {
         Mockito.when(router.isHaEnabled()).thenReturn(true);
         Mockito.when(router.isDynamicallyScalable()).thenReturn(false);
 
-        Mockito.when(vmInstanceDao.search(Mockito.any(), Mockito.any()))
-                .thenReturn(Arrays.asList(runningUser, stoppedUser, router));
+        // Destroyed but not yet expunged: still a current Instance
+        VMInstanceVO destroyedUser = Mockito.mock(VMInstanceVO.class);
+        Mockito.when(destroyedUser.getHypervisorType()).thenReturn(HypervisorType.KVM);
+        Mockito.when(destroyedUser.getState()).thenReturn(VirtualMachine.State.Destroyed);
+        Mockito.when(destroyedUser.getType()).thenReturn(VirtualMachine.Type.User);
+        Mockito.when(destroyedUser.isHaEnabled()).thenReturn(false);
+        Mockito.when(destroyedUser.isDynamicallyScalable()).thenReturn(false);
+
+        // Removed row: only counted in the lifetime statistics
+        VMInstanceVO expungedUser = Mockito.mock(VMInstanceVO.class);
+        Mockito.when(expungedUser.getHypervisorType()).thenReturn(HypervisorType.KVM);
+        Mockito.when(expungedUser.getState()).thenReturn(VirtualMachine.State.Expunging);
+        Mockito.when(expungedUser.getType()).thenReturn(VirtualMachine.Type.User);
+        Mockito.when(expungedUser.getRemoved())
+                .thenReturn(Date.from(Instant.parse("2025-06-01T12:00:00Z")));
+
+        Mockito.when(vmInstanceDao.searchIncludingRemoved(Mockito.any(), Mockito.any(),
+                        Mockito.any(), Mockito.anyBoolean()))
+                .thenReturn(Arrays.asList(runningUser, stoppedUser, router, destroyedUser, expungedUser));
     }
 
     private void stubDiskOfferings() {
@@ -266,6 +283,7 @@ private Object buildSection(String method) throws Exception {
      */
     private Map<String, Object> buildReportMap() throws Exception {
         Map<String, Object> reportMap = new HashMap<String, Object>();
+        reportMap.put("schema_version", UsageReporter.SCHEMA_VERSION);
         reportMap.put("hosts", buildSection("getHostReport"));
         reportMap.put("clusters", buildSection("getClusterReport"));
         reportMap.put("primaryStorage", buildSection("getStoragePoolReport"));
@@ -349,6 +367,7 @@ private static void assertCount(JsonObject parent, String section, String key, l
     public void testTelemetryEndpointIsStaticAndHttps() {
         Assert.assertEquals("https://call-home.cloudstack.org/report", UsageReporter.TELEMETRY_URI);
         Assert.assertTrue(UsageReporter.TELEMETRY_URI.startsWith("https://"));
+        Assert.assertEquals(1, UsageReporter.SCHEMA_VERSION);
 
         ConfigKey<?>[] configKeys = usageReporter.getConfigKeys();
         Assert.assertEquals(1, configKeys.length);
@@ -416,8 +435,9 @@ public void testTopLevelPayloadKeys() throws Exception {
         JsonObject report = reportJson();
 
         Assert.assertEquals("unexpected set of top level keys in the usage report",
-                new TreeSet<>(Arrays.asList("hosts", "clusters", "primaryStorage", "zones",
-                        "instances", "diskOffering", "versions", "current_version")),
+                new TreeSet<>(Arrays.asList("schema_version", "hosts", "clusters",
+                        "primaryStorage", "zones", "instances", "diskOffering", "versions",
+                        "current_version")),
                 new TreeSet<>(report.keySet()));
     }
 
@@ -482,19 +502,43 @@ public void testZonesSection() throws Exception {
         }
     }
 
+    /**
+     * "current" covers the non-removed rows in any state -- Destroyed Instances
+     * that have not been expunged yet included -- while "lifetime" also counts
+     * the removed rows and so describes every Instance which ever existed.
+     */
     @Test
     public void testInstancesSection() throws Exception {
         JsonObject instances = reportJson().getAsJsonObject("instances");
 
+        Assert.assertEquals(new TreeSet<>(Arrays.asList("current", "lifetime")),
+                new TreeSet<>(instances.keySet()));
+
+        JsonObject current = instances.getAsJsonObject("current");
         Assert.assertEquals(new TreeSet<>(Arrays.asList("hypervisor_type", "state", "type",
                         "ha_enabled", "dynamically_scalable")),
-                new TreeSet<>(instances.keySet()));
+                new TreeSet<>(current.keySet()));
 
-        assertCount(instances, "hypervisor_type", "KVM", 3);
-        assertCount(instances, "state", "Running", 2);
-        assertCount(instances, "state", "Stopped", 1);
-        assertCount(instances, "type", "User", 2);
-        assertCount(instances, "type", "DomainRouter", 1);
+        assertCount(current, "hypervisor_type", "KVM", 4);
+        assertCount(current, "state", "Running", 2);
+        assertCount(current, "state", "Stopped", 1);
+        assertCount(current, "state", "Destroyed", 1);
+        assertCount(current, "type", "User", 3);
+        assertCount(current, "type", "DomainRouter", 1);
+
+        JsonObject lifetime = instances.getAsJsonObject("lifetime");
+        Assert.assertEquals(new TreeSet<>(Arrays.asList("total", "removed", "hypervisor_type",
+                        "type")),
+                new TreeSet<>(lifetime.keySet()));
+
+        Assert.assertEquals(5, lifetime.get("total").getAsLong());
+        Assert.assertEquals(1, lifetime.get("removed").getAsLong());
+        assertCount(lifetime, "hypervisor_type", "KVM", 5);
+        assertCount(lifetime, "type", "User", 4);
+        assertCount(lifetime, "type", "DomainRouter", 1);
+
+        // The state of a removed row is meaningless, so lifetime has no state counter
+        Assert.assertFalse(lifetime.has("state"));
     }
 
     /**
@@ -503,12 +547,12 @@ public void testInstancesSection() throws Exception {
      */
     @Test
     public void testBooleanCountersBecomeTrueFalseStringKeys() throws Exception {
-        JsonObject instances = reportJson().getAsJsonObject("instances");
+        JsonObject current = reportJson().getAsJsonObject("instances").getAsJsonObject("current");
 
-        assertCount(instances, "ha_enabled", "true", 2);
-        assertCount(instances, "ha_enabled", "false", 1);
-        assertCount(instances, "dynamically_scalable", "true", 2);
-        assertCount(instances, "dynamically_scalable", "false", 1);
+        assertCount(current, "ha_enabled", "true", 2);
+        assertCount(current, "ha_enabled", "false", 2);
+        assertCount(current, "dynamically_scalable", "true", 2);
+        assertCount(current, "dynamically_scalable", "false", 2);
     }
 
     @Test
@@ -562,17 +606,21 @@ public void testVersionsSectionUsesUtcIso8601() throws Exception {
     public void testEmptyEnvironmentStillProducesCompletePayload() throws Exception {
         Mockito.when(hostDao.search(Mockito.any(), Mockito.any())).thenReturn(Collections.emptyList());
         Mockito.when(clusterDao.search(Mockito.any(), Mockito.any())).thenReturn(Collections.emptyList());
-        Mockito.when(vmInstanceDao.search(Mockito.any(), Mockito.any())).thenReturn(Collections.emptyList());
+        Mockito.when(vmInstanceDao.searchIncludingRemoved(Mockito.any(), Mockito.any(),
+                Mockito.any(), Mockito.anyBoolean())).thenReturn(Collections.emptyList());
         Mockito.when(storagePoolDao.listAll()).thenReturn(Collections.emptyList());
         Mockito.when(dataCenterDao.listAllZones()).thenReturn(Collections.emptyList());
         Mockito.when(diskOfferingDao.listAll()).thenReturn(Collections.emptyList());
         Mockito.when(versionDao.getAllVersions()).thenReturn(Collections.emptyList());
 
         JsonObject report = reportJson();
 
-        Assert.assertEquals(8, report.keySet().size());
+        Assert.assertEquals(9, report.keySet().size());
         Assert.assertEquals(0, report.getAsJsonObject("hosts").getAsJsonObject("type").size());
-        Assert.assertEquals(0, report.getAsJsonObject("instances").getAsJsonObject("state").size());
+        Assert.assertEquals(0, report.getAsJsonObject("instances")
+                .getAsJsonObject("current").getAsJsonObject("state").size());
+        Assert.assertEquals(0, report.getAsJsonObject("instances")
+                .getAsJsonObject("lifetime").get("total").getAsLong());
         Assert.assertEquals(0, report.getAsJsonObject("versions").size());
         Assert.assertEquals(0, report.getAsJsonObject("diskOffering").get("avg_disk_size").getAsLong());
     }

server/src/test/resources/usage-report-expected.json

@@ -1,4 +1,5 @@
 {
+  "schema_version": 1,
   "hosts": {
     "type": {
       "Routing": 3,
@@ -62,24 +63,38 @@
     }
   },
   "instances": {
-    "hypervisor_type": {
-      "KVM": 3
-    },
-    "state": {
-      "Running": 2,
-      "Stopped": 1
-    },
-    "type": {
-      "User": 2,
-      "DomainRouter": 1
-    },
-    "ha_enabled": {
-      "true": 2,
-      "false": 1
-    },
-    "dynamically_scalable": {
-      "true": 2,
-      "false": 1
+    "current": {
+      "hypervisor_type": {
+        "KVM": 4
+      },
+      "state": {
+        "Running": 2,
+        "Stopped": 1,
+        "Destroyed": 1
+      },
+      "type": {
+        "User": 3,
+        "DomainRouter": 1
+      },
+      "ha_enabled": {
+        "true": 2,
+        "false": 2
+      },
+      "dynamically_scalable": {
+        "true": 2,
+        "false": 2
+      }
+    },
+    "lifetime": {
+      "total": 5,
+      "removed": 1,
+      "hypervisor_type": {
+        "KVM": 5
+      },
+      "type": {
+        "User": 4,
+        "DomainRouter": 1
+      }
     }
   },
   "diskOffering": {

reporter/README.md telemetry/README.mdreporter/README.md renamed to telemetry/README.md

@@ -17,9 +17,9 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-# CloudStack Usage Reporter
+# CloudStack Telemetry
 
-This directory contains the server-side webservice for the Apache CloudStack usage reporting feature. When enabled, CloudStack management servers periodically send an anonymized report to the Apache CloudStack project. This data helps the community understand how CloudStack is deployed and used in the field.
+This directory contains the server-side webservice for the Apache CloudStack telemetry feature. When enabled, CloudStack management servers periodically send an anonymized report to the Apache CloudStack project. This data helps the community understand how CloudStack is deployed and used in the field.
 
 All data collected is anonymous. No personally identifiable information, IP addresses, or workload data is transmitted.
 
@@ -60,6 +60,13 @@ because the creation timestamp differs. The hash is one-way, so reports from the
 same installation can be correlated across time without identifying the operator
 or revealing when the cloud was installed.
 
+The ID is not guaranteed to be globally unique, but the chance of two
+installations deriving the same one is negligible: it would require two
+databases created with the same CloudStack version in the same second. A cloned
+or restored database derives the same ID and is therefore reported as the same
+installation, which is intended: it is the same cloud continuing under a
+different database.
+
 ### Storage
 
 Reports are stored below a base directory, configurable through the `REPORT_DIR` environment variable (default: `reports` in the working directory). A directory is created per `unique_id` and each report is stored with its receive timestamp as the filename:
@@ -75,6 +82,21 @@ reports/
 
 To keep malicious or malformed submissions out, the collector rejects reports that are not JSON objects, exceed 1MB, nest deeper than 6 levels, contain more than 4096 keys, or contain non-printable or oversized keys and string values. Only string, number and boolean values are accepted. The `unique_id` must be a valid SHA-256 hex digest. Per `unique_id`, at most one report per hour is accepted and at most 1000 reports are kept — the oldest are removed first, so a single sender can never fill up the disk.
 
+On top of these generic limits every report is validated against the exact
+schema the Management Server produces. Each report carries a `schema_version`
+and the collector knows the schema belonging to each version: all expected
+sections and counters must be present, no other keys are accepted and every
+value must have the expected type. An arbitrary well-formed JSON object is
+therefore rejected; any change to the report structure in `UsageReporter.java`
+requires incrementing the schema version and teaching the collector the new
+schema.
+
+The collector cannot verify that a report was genuinely produced by a
+CloudStack Management Server. The source is open, so anyone can derive a valid
+ID and craft a schema-conforming report. Schema validation, rate limiting and
+the per-ID storage bound limit abuse and resource usage, but the collected
+statistics remain best-effort by nature and should be interpreted as such.
+
 ### Running the webservice
 
 Install dependencies:
@@ -104,7 +126,7 @@ uwsgi --wsgi-file wsgi.py --callable application
 **Production (Apache mod_wsgi):**
 
 ```apache
-WSGIScriptAlias /report /path/to/reporter/wsgi.py
+WSGIScriptAlias /report /path/to/telemetry/wsgi.py
 ```
 
 ## Open source transparency