Dapdb, Isolation scheme #1
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary: The Makefile in the examples directory contained an empty line contain a tab character. This made my Emacs ask on every save `Suspicious line 10. Save anyway? (y or n)` Closes facebook/rocksdb#1872 Differential Revision: D4573881 Pulled By: siying fbshipit-source-id: fb3b4ee
Summary: This is a follow up fix for facebook/rocksdb#1783. After it, we should be able to ingest external v1 sst files with no global seqno field. Closes facebook/rocksdb#1874 Differential Revision: D4576194 Pulled By: IslamAbdelRahman fbshipit-source-id: 5b34a3e
Summary: Closes facebook/rocksdb#1877 Differential Revision: D4576056 Pulled By: IslamAbdelRahman fbshipit-source-id: 0f19275
Summary: Remove functions that we deprecated long time ago in db.h Closes facebook/rocksdb#1878 Differential Revision: D4576521 Pulled By: IslamAbdelRahman fbshipit-source-id: dfddad1
Summary: I'd like to propose a patch to expose a new IOError type with subcode kStaleFile to allow to detect when ESTALE error is returned. This allows the rocksdb consumers to handle this error separately from other IOErrors. I've also added a missing string representation for the kDeadlock subcode, I believe calling ToString() on Status object with that subcode would result in an out of band access in the msgs array, Please let me know if you have any questions or would like me to make any changes to this pull request. Closes facebook/rocksdb#1748 Differential Revision: D4387675 Pulled By: IslamAbdelRahman fbshipit-source-id: 67feb13
Summary: Fail IngestExternalFile() when bg_error_ exists Closes facebook/rocksdb#1881 Differential Revision: D4580621 Pulled By: IslamAbdelRahman fbshipit-source-id: 1194913
Summary: The option has been deprecated for two years and has no effect. Removing. Closes facebook/rocksdb#1866 Differential Revision: D4555203 Pulled By: yiwu-arbug fbshipit-source-id: c48f627
Summary: Closes facebook/rocksdb#1857 Differential Revision: D4534518 Pulled By: wat-ze-hex fbshipit-source-id: b456946
Summary: reimplement the compaction expansion on lower level. Considering such a case: input level file: 1[B E] 2[F G] 3[H I] 4 [J M] output level file: 5[A C] 6[D K] 7[L O] If we initially pick file 2, now we will compact file 2 and 6. But we can safely compact 2, 3 and 6 without expanding the output level. The previous code is messy and wrong. In this diff, I first determine the input range [a, b], and output range [c, d], then we get the range [e,f] = [min(a, c), max(b, d] and put all eligible clean-cut files within [e, f] into this compaction. **Note: clean-cut means the files don't have the same user key on the boundaries of some files that are not chosen in this compaction**. Closes facebook/rocksdb#1760 Differential Revision: D4395564 Pulled By: lightmark fbshipit-source-id: 2dc2c5c
Summary: MongoRocks is still using some deprecated functions, return them temporarily Closes facebook/rocksdb#1892 Differential Revision: D4592451 Pulled By: IslamAbdelRahman fbshipit-source-id: 5e6be3e
…checks Summary: This is a trivial fix for OOMs we've seen a few days ago in logdevice. RocksDB get into the following state: (1) Write throughput is too high for flushes to keep up. Compactions are out of the picture - automatic compactions are disabled, and for manual compactions we don't care that much if they fall behind. We write to many CFs, with only a few L0 sst files in each, so compactions are not needed most of the time. (2) total_log_size_ is consistently greater than GetMaxTotalWalSize(). It doesn't get smaller since flushes are falling ever further behind. (3) Total size of memtables is way above db_write_buffer_size and keeps growing. But the write_buffer_manager_->ShouldFlush() is not checked because (2) prevents it (for no good reason, afaict; this is what this commit fixes). (4) Every call to WriteImpl() hits the MaybeFlushColumnFamilies() path. This keeps flushing the memtables one by one in order of increasing log file number. (5) No write stalling trigger is hit. We rely on max_write_buffer_number Closes facebook/rocksdb#1893 Differential Revision: D4593590 Pulled By: yiwu-arbug fbshipit-source-id: af79c5f
Summary: Some places autodetected. These are the two places that didn't. closes #1498 Still unsure if the following instances of 4 * 1024 need fixing in: util/io_posix.h include/rocksdb/table.h (appears to be blocksize and different) utilities/persistent_cache/block_cache_tier.cc utilities/persistent_cache/persistent_cache_test.h include/rocksdb/env.h util/env_posix.cc db/column_family.cc Closes facebook/rocksdb#1499 Differential Revision: D4593640 Pulled By: yiwu-arbug fbshipit-source-id: efc48de
Summary: we occasionally missing this call so the file size will be wrong Closes facebook/rocksdb#1894 Differential Revision: D4598446 Pulled By: lightmark fbshipit-source-id: 42b6ef5
Summary: omit the override for the previous commit Closes facebook/rocksdb#1898 Differential Revision: D4598743 Pulled By: lightmark fbshipit-source-id: f98a378
Summary: Missing this function will cause RandomAccessFileReader not doing alignment in Direct IO mode, which introduce an IOError: invalid argument. Closes facebook/rocksdb#1900 Differential Revision: D4601261 Pulled By: lightmark fbshipit-source-id: c3eadf1
Summary: As the last step in backup creation, the .tmp directory is renamed omitting the .tmp suffix. In case the process terminates before this, the .tmp directory will be left behind. Even if this happens, we want future backups to succeed, so I added some checks/cleanup for this case. Closes facebook/rocksdb#1896 Differential Revision: D4597323 Pulled By: ajkr fbshipit-source-id: 48900d8
Summary: `WriteBatchWithIndex` has an incorrect implicitly-generated move constructor (it will copy the pointer causing a double-free on destruction). Just switch to `unique_ptr` so we get correct move semantics for free. Closes facebook/rocksdb#1899 Differential Revision: D4598896 Pulled By: ajkr fbshipit-source-id: 2373d47
Summary: InsertPathnameToSizeBytes() is called on shared/ and shared_checksum/ directories, which only exist for certain configurations. If we try to list a non-existent directory's contents, some Envs will dump an error message. Let's avoid this by checking whether the directory exists before listing its contents. Closes facebook/rocksdb#1895 Differential Revision: D4596301 Pulled By: ajkr fbshipit-source-id: c809679
Summary: querying logical sector size from the device instead of hardcoding it for linux platform. Closes facebook/rocksdb#1875 Differential Revision: D4591946 Pulled By: ajkr fbshipit-source-id: 4e9805c
Summary: add direct_io and compaction_readahead_size in db_stress test direct_io under db_stress with compaction_readahead_size enabled to capture bugs found in production. `./db_stress --allow_concurrent_memtable_write=0 --use_direct_reads --use_direct_writes --compaction_readahead_size=4096` Closes facebook/rocksdb#1906 Differential Revision: D4604514 Pulled By: IslamAbdelRahman fbshipit-source-id: ebbf0ee
Summary: Xfunc is hardly used. Remove it to keep the code simple. Closes facebook/rocksdb#1905 Differential Revision: D4603220 Pulled By: siying fbshipit-source-id: 731f96d
…cksums_in_comp… Summary: …action The two options, min_partial_merge_operands and verify_checksums_in_compaction, are not seldom used. Remove them to reduce the total number of options. Also remove them from Java and C interface. Closes facebook/rocksdb#1902 Differential Revision: D4601219 Pulled By: siying fbshipit-source-id: aad4cb2
Summary: The assertion in Abandon() fails when called after Finish() fails. Finish() already closes the builder so there's no need to call Abandon(). Closes facebook/rocksdb#1901 Differential Revision: D4601373 Pulled By: ajkr fbshipit-source-id: e5678be
Summary: valgrind tests always timeout with parallel run. Black list some slowest ones. It is better to run fewer tests than always have the tests timeout. Closes facebook/rocksdb#1908 Differential Revision: D4607875 Pulled By: siying fbshipit-source-id: 7062664
Summary: The PATH update should put the Java path in the beginning, rather than the end. Otherwise, it will be overwritten. Also upgrade the Java version. Closes facebook/rocksdb#1912 Differential Revision: D4609854 Pulled By: siying fbshipit-source-id: 3dc04f2
Summary: A previous fix to DBTestUniversalCompaction.UniversalCompactionTrivialMoveTest2 didn't address the right problem. The problem is L0->L0 compaction is not trivial move in the scenario, not parallel compactions. Fix this. Closes facebook/rocksdb#1911 Differential Revision: D4608955 Pulled By: siying fbshipit-source-id: 7a712cb
Summary: let users know this feature is ready Closes facebook/rocksdb#1915 Differential Revision: D4610842 Pulled By: lightmark fbshipit-source-id: d102772
Summary: Travis is short of OSX resource. Try to move platform independent test suites out of OSX Closes facebook/rocksdb#1922 Differential Revision: D4616070 Pulled By: siying fbshipit-source-id: 786342c
Summary: Separate a smal subset of tests in DBTest to DBBasicTest. Tests in DBTest don't have to run in CI tests on platforms like OSX, as long as they are covered by Linux. Closes facebook/rocksdb#1924 Differential Revision: D4616702 Pulled By: siying fbshipit-source-id: 13e6549
Summary: - Fix unaligned reads in read cache by using RandomAccessFileReader - Allow read cache flags in db_bench Closes facebook/rocksdb#1916 Differential Revision: D4610885 Pulled By: IslamAbdelRahman fbshipit-source-id: 2aa1dc8
Fix length padding in NvmFile Flush and Read
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.