Incorrect statistics mapping

[jonas0616]

After upgrading grocksdb from v1.8.12 to v1.9.3, I noticed that some statistics values appeared abnormal.
For example, I found the value of rocksdb.compact.write.bytes in the LOG file (periodically dumped stats) showed it as 278576274:

$ grep rocksdb.compact.write.bytes LOG
rocksdb.compact.write.bytes COUNT : 0
rocksdb.compact.write.bytes COUNT : 115702019
rocksdb.compact.write.bytes COUNT : 278576274

However, the value retreived using opts.GetTickerCount() remained at 539.
Interestingly, the value for rocksdb.db.iter.bytes.read was exactly 539, leading I to suspect that the statistics might have been incorrectly mapped.

[linxGnu]

@jonas0616
There is no updates in grocksdb between v1.8.15 & v1.9.3 (see also: here).

I am suspecting that there are changes with rocksdb itself.

[jonas0616]

@linxGnu
I've rollbacked the grocksdb version to v1.8.15, and some of the statistics values have returned to normal.
However, I still notice that another statistics values don't match what's printed in the LOG file after rollback.
For example, I've also found anomalies with the following two values:

rocksdb.non.last.level.seek.data COUNT
rocksdb.non.last.level.seek.data.useful.no.filter COUNT

It seems like the statistics mappings were already incorrect even before v1.8.15.
We rely on these statistics to collect metrics and monitor RocksDB's performance.
But at this point, I'm unsure which statistics values can be trusted.
It would be great if this issue could be fixed.

[linxGnu]

@jonas0616
Could you please tell me which API you are using to get the stats (beside GetTickerCount()).

In order to get stats, you need to create Statistic Object. See also:

• https://github.com/facebook/rocksdb/wiki/Statistics

I don't think RocksDB expose to C API yet.

[jonas0616]

I've set opts.SetStatsDumpPeriodSec(60) to have RocksDB dump statistics into the LOG file in the db directory every minute.
This way, you can observe the statistics in the LOG file and compare them with the values retrieved through opts.GetTickerCount().

For example, to print the ticker count every minute:

  val := opts.GetTickerCount(grocksdb.TickerType_NON_LAST_LEVEL_SEEK_DATA_USEFUL_NO_FILTER)
  logrus.WithField("TickerType", grocksdb.TickerType_NON_LAST_LEVEL_SEEK_DATA_USEFUL_NO_FILTER).
      WithField("value", val).
      Infof("non_last_level_seek_data_useful_no_filter")

  val = opts.GetTickerCount(grocksdb.TickerType_NON_LAST_LEVEL_SEEK_DATA_USEFUL_FILTER_MATCH)
  logrus.WithField("TickerType", grocksdb.TickerType_NON_LAST_LEVEL_SEEK_DATA_USEFUL_FILTER_MATCH).
      WithField("value", val).
      Infof("non_last_level_seek_data_useful_filter_match")

After making some iterator requests to the RocksDB and wait for one minute, the results are:

INFO [14208]: non_last_level_seek_data_useful_no_filter TickerType=176 value=10
INFO [14208]: non_last_level_seek_data_useful_filter_match TickerType=177 value=5

Now, search the LOG file for the latest dumped statistics:

$ cd <you db path>
$ grep non.last.level.seek.data LOG
rocksdb.non.last.level.seek.data COUNT : 10
rocksdb.non.last.level.seek.data.useful.no.filter COUNT : 5
rocksdb.non.last.level.seek.data.useful.filter.match COUNT : 0

[linxGnu]

@jonas0616

Getting through Options sounds incorrect! Stats only collectable via Options with Statistics Object bound (see also: here). Unfortunately, at the moment, no C API for CreateDBStatistics!!

Another way to do: db.GetProperty("rocksdb.dbstats") (Reference: here). Could you please give it a try?

[jonas0616]

@linxGnu
Actually, you can. opts.EnableStatistics() does exactly what you're thinking.
I might have misunderstood you earlier. I thought you were asking how I confirmed that the value returned by GetTickerCount was incorrect.