-
Notifications
You must be signed in to change notification settings - Fork 183
DOC-12491 xdcrDiffer in Server build #3836
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 5 commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
f80b759
whats-new-entry
rao-shwe 6d7884c
draft-1
rao-shwe 70270b1
updated-nav-file
rao-shwe 294ff0c
minor-edits
rao-shwe 737bf2d
implemented-review-inputs
rao-shwe f364351
implemented-pm-feedback
rao-shwe fc57be6
implemented-editorial-input
rao-shwe File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,68 @@ | ||
= xdcrDiffer Utility | ||
:description: pass:q[The `xdcrDiffer` utility helps you verify data consistency between XDCR clusters by comparing document metadata and values, and reporting missing or mismatched documents.] | ||
|
||
[abstract] | ||
{description} | ||
|
||
`xdcrDiffer` is a diagnostic utility included in the Server installation package for convenient access. | ||
Previously, you'd build this utility from source using the xdcrDiffer GitHub repository. | ||
|
||
== What is xdcrDiffer? | ||
|
||
`xdcrDiffer` is a Diff tool that helps you check data consistency between XDCR clusters. | ||
For full usage instructions and Q&A, see the https://github.com/couchbase/xdcrDiffer?tab=readme-ov-file#readme[xdcrDiffer ReadMe]. | ||
|
||
The output report from `xdcrDiffer` provides information such as: | ||
|
||
* A list of document IDs missing from the target cluster. | ||
|
||
* A list of documents with mismatched data. | ||
|
||
== Utility Files | ||
|
||
The utility includes these files: | ||
|
||
* `xdcrDiffer` (binary) | ||
|
||
* `runDiffer.sh` (shell script) | ||
|
||
To run the utility, use the `runDiffer.sh` script. | ||
|
||
TIP: Run the xdcrDiffer utility on a cluster node that does not serve Data service, as the utility can consume significant resources. | ||
You can also run it on an arbiter node, which does not run any services. | ||
|
||
== How the xdcrDiffer Utility Works | ||
|
||
`xdcrDiffer` acts as a DCP consumer and creates DCP streams from the Data nodes. | ||
Then it reads data from both the source and target clusters, and stores results in the directory you specify as `outputDir`. | ||
|
||
**Before you begin:** | ||
|
||
Estimate the required storage for utility output. | ||
Make sure the specified `outputDir` has enough space for collecting data. | ||
|
||
Each mutation from DCP is stored with its metadata. | ||
Each mutation typically uses about 102 bytes and the key size. | ||
Calculate the storage needed as follows: | ||
|
||
---- | ||
Storage needed = (102 bytes + key size) × number of documents in the source and target buckets being replicated | ||
---- | ||
|
||
NOTE: Ensure you check the details of usage, limitations, and information about how to read the output reports in https://github.com/couchbase/xdcrDiffer?tab=readme-ov-file#readme[xdcrDiffer ReadMe]. | ||
|
||
The following is an example of running `xdcrDiffer.sh` with an option to compare both metadata and body during the report generation portion of the utility run. | ||
|
||
---- | ||
runDiffer.sh --username=Administrator --password=password --hostname=localhost:8091 --sourceBucket=travel-sample --targetBucket=travel-sample --remoteClusterName=cluster2 --compareType=both --outputDir=/has_enough_space/xdcr_differ_outputdir | ||
---- | ||
|
||
rao-shwe marked this conversation as resolved.
Show resolved
Hide resolved
|
||
== Comparison Process | ||
|
||
* `xdcrDiffer` collects each document's metadata and generates a SHA-512 digest of its value. | ||
|
||
* After capturing all data from both clusters, `xdcrDiffer` compares documents by using either document ID or key. | ||
|
||
** If a document is missing in either cluster, the utility reports the missing document. | ||
|
||
** If all documents are present, the utility compares metadata and the body hash to verify if they match. |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.