DOC-12749 New REST API settings in Data Service

modules/cli/pages/cbepctl/set-flush_param.adoc

@@ -20,22 +20,14 @@ cbepctl [hostname]:11210 -b [bucket-name] -u [administrator-name] -p [administra
 cbepctl [hostname]:11210 -b [bucket-name] -u [administrator-name] -p [administrator-password] set flush_param alog_task_time [value]
 ----
 
-The syntax for disk cleanup is:
-
-----
-cbepctl [host]:11210 -b [bucket-name] -u [administrator-name] -p [administrator-password] set flush_param exp_pager_stime [value]
-----
-
 The syntax for ejection is:
 
 ----
 cbepctl [host]:11210 -b [bucket-name] -u [administrator-name] -p [administrator-password] set flush_param [parameter] [value]
 ----
 
-Parameters used for changing ejection thresholds:
+Parameter used for changing ejection thresholds:
 
-* `mem_low_wat`
-* `mem_high_wat`
 * `pager_active_vb_pcnt`
 
 The syntax to set the out of memory threshold is:
@@ -50,11 +42,19 @@ Tune the dynamic shared thread pool performance by changing the thread types ins
 The command [.cmd]`set flush_param` adjusts the number of threads that prioritize read, write, non-i/o and auxiliary-i/o operations.
 These settings take effect immediately and do not require that the bucket be restarted.
 
-NOTE: The settings for threads number take effect only if the underlying operating system has a sufficient number of CPU cores.
+[NOTE]
+====
+* The settings for threads number take effect only if the underlying operating system has a sufficient number of CPU cores.
 The minimum number of CPU cores is four (4), but  three (3) additional cores are required for each additional writer thread.
 For example, five (5) writer threads is a valid setting if the underlying hardware has at least sixteen (16) cores.
 
-NOTE: Changes of thresholds are NOT persistent and must be reapplied after the bucket warmup.
+* Changes of thresholds are NOT persistent and must be reapplied after the bucket warmup.
+
+* The settings `warmup_min_items_threshold`, `warmup_min_memory_threshold`, `exp_pager_stime`, `mem_high_wat`, and `mem_low_wat` are removed, and no more supported through `cbepctl`.
++
+These settings are replaced by new settings `expiryPagerSleepTime`, `warmupBehavior`, `memoryLowWatermark`, and `memoryHighWatermark`, which can be configured using REST APIs.
+See xref:rest-api:rest-bucket-create.adoc[Creating and Editing Buckets].
+====
 
 alog_sleep_time, alog_task_time::
 Couchbase Server has an optimized xref:learn:buckets-memory-and-storage/memory.adoc#initialization-and-warmup[disk warmup].
@@ -69,20 +69,10 @@ The scanner is highly CPU-intensive: therefore, to reduce the cluster-wide impac
 Note also that if the scanner runs at the same time that index updates are being made (either on the current node, or on one or more other nodes) by the Index Service, the performance of the index updates may be adversely affected.
 The scanner should be configured to minimize the likelihood of this problem.
 
-exp_pager_stime::
-The `cbepctl flush_param exp_pager_stime` command sets the time interval for disk cleanup.
-Couchbase Server does lazy xref:learn:buckets-memory-and-storage/memory.adoc#expiry-pager[expiration], that is, expired items are flagged as deleted rather than being immediately erased.
-Couchbase Server has a maintenance process that periodically looks through all information and erases expired items.
-By default, this maintenance process runs every 10 minutes, but it can be configured to run at a different interval.
-+
-NOTE: The compaction process will also remove expired items.
-
-mem_low_wat, mem_high_wat, pager_active_vb_pcnt::
+pager_active_vb_pcnt::
 xref:learn:buckets-memory-and-storage/memory.adoc#ejection[Ejection] means that documents are removed from RAM but the key and metadata remain.
-If the amount of RAM used by items reaches the high water mark (upper threshold), both active and replica data are ejected until the memory usage (amount of RAM consumed) reaches the low water mark (lower threshold).
-The server determines that items are not recently used based on a not-recently-used (NRU) value.
 +
-Use the `mem_low_wat`, `mem_high_wat`, and `pager_active_vb_pcnt` settings to change the server thresholds for ejection.
+Use `pager_active_vb_pcnt` settings to change the server thresholds for ejection.
 +
 WARNING: Do not change the ejection defaults unless required by Couchbase Support.
 
@@ -116,11 +106,6 @@ The following are the command options:
 | `couch_response_timeout`
 | timeout in receiving a response from CouchDB.
 
-| `exp_pager_stime`
-| Expiry Pager interval.
-Time interval that Couchbase Server waits before it performs cleanup and removal of expired items from disk.
-Setting this value to `0` will disable the Expiry Pager from running.
-
 | `flushall_enabled`
 | Deprecated.
 Enable flush operation.
@@ -131,24 +116,12 @@ Enable flush operation.
 | `max_size`
 | Maximum memory used by the server.
 
-| `mem_high_wat`
-| High water mark in bytes.
-
-| `mem_low_wat`
-| Low water mark in bytes.
-
 | `mutation_mem_threshold`
 | Amount of RAM that can be consumed in that caching layer before clients start receiving temporary out of memory messages.
 
 | `timing_log`
 | Path to log detailed timing stats.
 
-| `warmup_min_memory_threshold`
-| Memory threshold (%) during warmup to enable traffic.
-
-| `warmup_min_items_threshold`
-| Item number threshold (%) during warmup to enable traffic.
-
 | `klog_compactor_queue_cap`
 | Queue cap to throttle the log compactor.
 
@@ -218,23 +191,6 @@ setting param: alog_task_time 23
 set alog_task_time to 23
 ----
 
-*Examples for setting the disk cleanup*
-
-The following example sets the cleanup process to run every 600 seconds (10 minutes).
-This is the interval that Couchbase Server waits before it tries to remove expired items from disk.
-
-----
-cbepctl 10.5.2.117:11210 -b foo-bucket -u Administrator -p password \
-set flush_param exp_pager_stime 600
-----
-
-The following example response shows the cleanup process set to 600 seconds.
-
-----
-setting param: exp_pager_stime 600
-set exp_pager_stime to 600
-----
-
 *Examples for setting the out-of-memory error message*
 
 In this example, the threshold is reduced to 65% of RAM.
@@ -251,30 +207,6 @@ setting param: mutation_mem_threshold 65
 set mutation_mem_threshold to 65
 ----
 
-*Example for setting the low water mark*
-
-The low water mark sets the lower threshold of RAM for a specific bucket on a node.
-The item pager stops ejecting items once the low water mark is reached.
-
-The following example sets the low water mark percentage to 70% of RAM.
-
-----
-cbepctl 10.5.2.117:11210 -b foo-bucket -u Administrator -p password \
-set flush_param mem_low_wat 70%
-----
-
-*Example for setting the high water mark*
-
-The high water mark set the amount of RAM consumed by items that must be breached before infrequently used active and replica items are ejected.
-
-The following example sets the high water mark percentage to 80% of RAM for a specific bucket on a node.
-This means that items in RAM on this node can consume up to 80% of RAM before the item pager begins ejecting items.
-
-----
-cbepctl 10.5.2.117:11210 -b foo-bucket -u Administrator -p password \
-set flush_param mem_high_wat 80%
-----
-
 *Examples for setting percentage of ejected items*
 
 Based on the NRU algorithm, the server ejects active and replica data from a node.
@@ -293,15 +225,9 @@ By doing so, active data from a source node is maximized while maintaining incom
 However, if the server is ejecting a very large percentage of replica data and a node fails, the replica data is not immediately available.
 In this case, the items are retrieved from disk and put back into RAM before the request is fulfilled.
 
-The following example response shows the low water mark, high water mark, and percentage of ejected items being set.
+The following example response shows the percentage of ejected items being set.
 
 ----
-setting param: mem_low_wat 70
-set mem_low_wat to 70
-
-setting param: mem_high_wat 80
-set mem_high_wat to 80
-
 setting param: pager_active_vb_pcnt 50
 set pager_active_vb_pcnt to 50
 ----

modules/introduction/partials/new-features-80.adoc

@@ -2,11 +2,10 @@
 === Couchbase Cluster
 
 
-
+[#anchor-61457]
 https://jira.issues.couchbase.com/browse/MB-61457[MB-61457]::
 The following settings have been added to the `/pools/default/buckets` REST APIs.
 +
-
 |===
 |Argument Name |Type |Valid values |Default
 
@@ -35,6 +34,13 @@ The following settings have been added to the `/pools/default/buckets` REST APIs
 |51 to 90
 |85
 |===
++
+The settings `warmup_min_items_threshold`, `warmup_min_memory_threshold`, `exp_pager_stime`, `mem_high_wat`, and `mem_low_wat` are removed from `cbepctl`.
+The above new settings are added and made accessible through the REST API.
+For more information, see the xref:rest-api:rest-bucket-create.adoc[Creating and Editing Buckets] and xref:learn:buckets-memory-and-storage/memory.adoc[Memory].
++
+The same changes in the warmup behavior settings are applicable to the Data Services.
+For more information, see xref:anchor-9418[MB-9418].
 
 https://jira.issues.couchbase.com/browse/MB-61946[MB-61946]::
 The `/pools/default/nodeServices` endpoint results now include the UUID and name of the cluster.
@@ -70,6 +76,7 @@ You can replace them with ephemeral buckets which are similar to Memcached bucke
 See the https://docs-archive.couchbase.com/server/6.6/learn/buckets-memory-and-storage/buckets.html#bucket-capabilities[Bucket Capabilities in the Version 6.6 documentation] for a summary of the differences between Memcached and ephemeral buckets.
 If you do not remove your Memcached buckets, the upgrade process reports an error message telling you to remove the Memcached buckets before proceeding.
 
+[#anchor-9418]
 https://jira.issues.couchbase.com/browse/MB-9418[MB-9418]::
 The previous warmup behavior was to load values and not complete warmup until the Bucket reaches one of:
 +
@@ -91,6 +98,9 @@ Warmup behaves the same as it did before with no background warmup
 Warmup now loads items in the background, allowing the bucket to have write availability much quicker than blocking.
 `none`:::
 Warmup is disabled; the bucket comes online with no items loaded.
++
+The same changes in the warmup behavior settings are applicable to the Couchbase Cluster.
+For more information, see xref:anchor-61457[MB-61457].
 
 https://jira.issues.couchbase.com/browse/MB-64071[MB-64071]::
 Couchbase Server has a new metric

modules/learn/pages/buckets-memory-and-storage/memory.adoc

@@ -88,8 +88,23 @@ When Couchbase Server is restarted on a node, the node goes through a _warmup pr
 During this warmup process, data on disk is sequentially reloaded into memory for each bucket.
 The time required for the reload depends on the size and configuration of the system, the amount of data persisted on the node, and the ejection policy configured for the buckets.
 
+Warmup behavior controls how and when data is loaded from disk into memory after a bucket or node restarts. The `warmupBehavior` setting determines this process:
+
+* **background** (default): The bucket starts serving requests as soon as metadata and some data are loaded into memory, while the rest of the data continues loading in the background.
+This allows for faster bucket availability and minimal downtime.
+* **blocking**: The bucket does not serve any requests until all data is fully loaded into memory.
+This ensures that all items are available immediately when the bucket becomes active, but can increase startup time.
+* **none**: The warmup process is skipped, and the bucket serves only items that are already in memory.
+Data is loaded on-demand as requests are made for items not yet loaded.
+
+Choosing the appropriate warmup behavior lets you balance between quicker bucket availability and having more data pre-loaded for immediate access after a restart.
+
 On a given Data Service node, a bucket's data is loaded in accordance with determinations made by the _access scanner_.
-This program runs by default once every day.
+
+The access scanner periodically identifies the most frequently used keys and writes them to an access log.
+During server warmup, Server uses this log to prioritize loading the most-accessed documents into memory.
+By default, the access scanner runs every 24 hours at 10:00 AM GMT, but you can adjust its schedule to minimize performance impact and avoid conflicts with index updates.
+
 It checks the _resident ratio_ — which is the percentage of items in active and replica vBuckets that are currently in memory on the node:
 
 * If the ratio is below 95%, the access scanner generates a new _access log_, which records the documents that have been most frequently accessed during the last 24 hours.
@@ -99,14 +114,21 @@ If and when data-loading subsequently occurs, the new access log is consulted, t
 Instead, it deletes any existing access log, and exits.
 If and when data-loading subsequently occurs, since no access log exists, loading occurs with no priority-order (this being, in cases of extremely high resident ratio, the more performative loading procedure).
 
+NOTE: The settings `warmup_min_items_threshold`, `warmup_min_memory_threshold`, and enabling access scanner are no more supported through `cbepctl`.
+These settings are replaced by a new setting `warmupBehavior` and `accessScannerEnabled`, which can be configured using REST APIs.
+
 [#configuring-the-access-scanner]
 === Configuring the Access Scanner
 
-The access scanner is configurable, via the CLI utility `cbepctl`, with the `flush_param` parameter.
-This utility also provides the parameters `warmup_min_memory_threshold` and `warmup_min_item_threshold`, which can be used to schedule the resumption of traffic before all items have been reloaded into memory.
+You can configure the access scanner using the REST API.
+For more information, see xref:rest-api:rest-bucket-create.adoc#accessscannerenabled[accessScannerEnabled].
 
-Note that the access scanner is a highly CPU-intensive process, whose configurable start-time should ideally be established as different for each Data Service node in the cluster.
-See xref:cli:cbepctl/set-flush_param.adoc[set flush_param].
+NOTE: The access scanner is a highly CPU-intensive process.
+
+=== Configuring the Warmup Behaviour
+
+You can configure the warmup process using the REST API.
+For more information, see xref:rest-api:rest-bucket-create.adoc#warmupbehavior[warmupBehavior].
 
 [#ejection]
 == Ejection
@@ -132,32 +154,41 @@ NOTE: Ejection from Ephemeral buckets removes data without persistence because E
 
 For more information about buckets and bucket types, see xref:buckets-memory-and-storage/buckets.adoc[Buckets].
 
-For each bucket, available memory is managed according to two _watermarks_, which are `mem_low_wat` and `mem_high_wat`.
-If data is continuously loaded into the bucket, its quantity eventually increases to the value indicated by the `mem_low_wat` watermark.
+NOTE: The settings `mem_high_wat`, and `mem_low_wat` are no more supported through `cbepctl`.
+These settings are replaced by new settings `memoryLowWatermark` and `memoryHighWatermark`, which can be configured using REST APIs.
+
+For each bucket, available memory is managed according to two _watermarks_, which are `memoryLowWatermark` and `memoryHighWatermark`.
+If data is continuously loaded into the bucket, its quantity eventually increases to the value indicated by the `memoryLowWatermark` watermark.
 At this point, no action is taken.
-Then, as still more data is loaded, the data's quantity increases to the value indicated by the `mem_high_wat` watermark.
-If, based on the bucket's configuration, items can be ejected from the bucket, the Data Service ejects items from the bucket until the quantity of data has decreased to the `mem_low_wat` watermark.
+Then, as still more data is loaded, the data's quantity increases to the value indicated by the `memoryHighWatermark` watermark.
+If, based on the bucket's configuration, items can be ejected from the bucket, the Data Service ejects items from the bucket until the quantity of data has decreased to the `memoryLowWatermark` watermark.
 In cases where ejection cannot free enough space to support continued data-ingestion, the Data Service stops ingesting data, error messages are sent to clients, and the system displays an _insufficient memory_ notification.
 When sufficient memory is again available, data-ingestion resumes.
 
 Items are selected for ejection based on metadata that each contains, indicating whether the item can be classified as _Not Recently Used_ (NRU).
 If an item has not been recently used, it is a candidate for ejection.
 
-The relationship of `mem_low_wat` and `mem_high_wat` to the bucket's overall memory quota is illustrated as follows:
+The relationship of `memoryLowWatermark` and `memoryHighWatermark` to the bucket's overall memory quota is illustrated as follows:
 
 [#tunable_memory]
 image::buckets-memory-and-storage/tunableMemory.png[,416]
 
-The default setting for `mem_low_wat` is 75%.
-The default setting for `mem_high_wat` is 85%.
-The default settings can be changed by means of the `cbepctl` utility.
-See xref:cli:cbepctl/set-flush_param.adoc[set flush_param].
+The default setting for `memoryLowWatermark` is 75%.
+The default setting for `memoryHighWatermark` is 85%.
+The default settings can be changed using the REST API.
+See xref:rest-api:rest-bucket-create.adoc#memorylowwatermark[memoryLowWatermark] and xref:rest-api:rest-bucket-create.adoc#memoryhighwatermark[memoryHighWatermark].
 
 [#expiry-pager]
 == Expiry Pager
 
 Scans for items that have expired, and erases them from memory and disk; after which, a _tombstone_ remains for a default period of 3 days.
-The expiry pager runs every 10 minutes by default: for information on changing the interval, see `cbepctl` xref:cli:cbepctl/set-flush_param.adoc[set flush_param].
+The expiry pager runs every 10 minutes by default.
+
+NOTE: The setting `exp_pager_stime` is no more supported through `cbepctl`.
+This setting is replaced by a new setting `expiryPagerSleepTime`, which can be configured using REST APIs.
+
+For information on changing the interval using the REST API, see xref:rest-api:rest-bucket-create.adoc#expirypagersleeptime[expiryPagerSleepTime].
+
 For more information on item-deletion and tombstones, see xref:data/expiration.adoc[Expiration].
 
 [#active-memory-defragmenter]