Consolidation issue fix.

[k0ushal]

• Fixed the tier based candidate selection
• Default tiers are powers of 4 with the first tier being 0-4M followed by 4-16M, 16-64M and so on.
• Fixed consolidation window of size 4

[k0ushal]

Documentation:
https://github.com/arangodb/documents/pull/145

[goedderz]

Comments as we talked about. Looks good to me!

[goedderz]

Probably inconsequential, but it would suffice to calculate skew, mergeScore and cost once after the loop for the last element.

[goedderz]

Minor: You could probably use std::countl_zero and get rid of the loop.

[goedderz]

As already discussed:

We should think about whether calculating the mergeScore this way is sensible. What seems strange is that while the skew is a ratio (of byte-sizes), the second summand is an inverse count. This seems off: intuitively I'd expect e.g. a ratio of live and total documents to be considered alongside the skew.

[goedderz]

This is actually quite bad the way it is, worse than we noticed yesterday @k0ushal.

Note that $\mathrm{skew} \in (0, 1)$. With $\mathrm{delCount} = 1$, we get

$$\begin{align*} \mathrm{mergeScore} &= \mathrm{skew} + \frac{1}{1 + \mathrm{delCount}} \\\ &= \mathrm{skew} + \frac 1 2 \\\ &\leq 1 \frac 1 2 \\\ &= \mathrm{maxMergeScore} \end{align*}$$

.

So this way we are always allowed to consolidate if only one document has been deleted, no matter the size of the files or number of documents therein.

Let us at least do

    mergeScore = skew + live_docs_count / total_docs_count;

instead, as discussed - this has more reasonable properties.

And as a second observation @neunhoef made today while discussing this: Adding these two values is probably not right, either. They should be multiplied instead; the maxMergeScore will need to be adjusted to 0.5 to get a similar effect.

So we should actually do

    mergeScore = skew * live_docs_count / total_docs_count;

(and adapt maxMergeScore).

To understand this better, we should still do some formal worst-case analysis and some tests (specifically unit tests of the consolidation algorithm that play out certain usage scenarios).

[goedderz]

As discussed: finding the tier-boundaries could be done by binary search, possibly utilizing std::lower_bound / std::upper_bound.

[goedderz]

This is actually quite bad the way it is, worse than we noticed yesterday @k0ushal.

Note that $\mathrm{skew} \in (0, 1)$. With $\mathrm{delCount} = 1$, we get

$$\begin{align*} \mathrm{mergeScore} &= \mathrm{skew} + \frac{1}{1 + \mathrm{delCount}} \\\ &= \mathrm{skew} + \frac 1 2 \\\ &\leq 1 \frac 1 2 \\\ &= \mathrm{maxMergeScore} \end{align*}$$

.

So this way we are always allowed to consolidate if only one document has been deleted, no matter the size of the files or number of documents therein.

Let us at least do

    mergeScore = skew + live_docs_count / total_docs_count;

instead, as discussed - this has more reasonable properties.

And as a second observation @neunhoef made today while discussing this: Adding these two values is probably not right, either. They should be multiplied instead; the maxMergeScore will need to be adjusted to 0.5 to get a similar effect.

So we should actually do

    mergeScore = skew * live_docs_count / total_docs_count;

(and adapt maxMergeScore).

To understand this better, we should still do some formal worst-case analysis and some tests (specifically unit tests of the consolidation algorithm that play out certain usage scenarios).