Dynode invalidation on restart

[LordDarkHelmet]

Describe the issue

When a Dynode is restarted there is a chance that it will change its state to NEW_START_REQUIRED.
This may have to do with issue #390

Steps to reproduce

Terminate a non-responsive Dynode and restart it. You may be able to recreate with killing a responsive Dynode. However, my logs were extracted from running Dynodes that were killed due to non-responsive CLIs. #390

Expected behavior (tell us what should happen)

If it is restarted within DYNODE_NEW_START_REQUIRED_SECONDS (180 min), the Dynode should continue to be valid

Actual behavior (tell us what happens instead)

The Dynode becomes invalid.

In the log file we can see where it was terminated, restarted, and where it appears it was set to the NEW_START_REQUIRED state.

... (Last proper Dynode Ping Sent before termination)
2019-12-17 02:01:11 CActiveDynode::SendDynodePing -- Relaying ping, collateral=8841f745d670f59e7b6bdeca31b23da4edad04475379dab5ee0631b3fa1aafb1-1
...
2019-12-17 02:20:42 torcontrol thread exit
... (The Dynode was terminated at 2019-12-17_02:21:13 due to a non-responsive CLI)
2019-12-17 02:21:15

2019-12-17 02:21:15 Dynamic version v2.4.3.0-8873e60b9

...

2019-12-17 02:21:22 net thread start
2019-12-17 02:21:22 StartDHTNetwork -- starting
2019-12-17 02:21:23 Imported mempool transactions from disk: 0 successes, 4 failed, 0 expired
2019-12-17 02:21:33 Loading addresses from DNS seeds (could take a while)
2019-12-17 02:21:44 16 addresses found from DNS seeds
2019-12-17 02:21:44 dnsseed thread exit
2019-12-17 02:21:49 AcceptConnection -- dynode is not synced yet, skipping inbound connection attempt
2019-12-17 02:21:58 AcceptConnection -- dynode is not synced yet, skipping inbound connection attempt
2019-12-17 02:22:06 AcceptConnection -- dynode is not synced yet, skipping inbound connection attempt
2019-12-17 02:22:15 AcceptConnection -- dynode is not synced yet, skipping inbound connection attempt
2019-12-17 02:22:16 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=107.145.181.3:33300
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=9d673858b5bfa82e59f13374aba3312f00577a6b7d14fbf35671670533733200-0  blockHash=000000034d0df9641                                                                                                                     f28da991a82c64c3e004d69d8ff5a12e84695b639e4b81a
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=9d673858b5bfa82e59f13374aba3312f00577a6b7d14fbf35671670533733200-0  blockHash=000000034d0df9641                                                                                                                     f28da991a82c64c3e004d69d8ff5a12e84695b639e4b81a
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=0eb00efa1117850f97622d21d8f06dbbd20c5c032cb35c3186fe81a46b397f05-0  blockHash=000000000e1da6b7b                                                                                                                     14e3371a2e76c8cd5e2a24b1db884d05df0127c372d8c8e
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=dc111c640463ecf5e3fea5bcba30846c330e260c51620b8aec0d97b128c6f907-0  blockHash=000000001b3959089                                                                                                                     73a51d5f09fdbe44b742dc44d4994f91f26d98c0b3e7780
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=91e96751375dfe4d539cf24bb51ed5481b226f1105717c93d98fe87027932009-1  blockHash=000000000e1da6b7b                                                                                                                     14e3371a2e76c8cd5e2a24b1db884d05df0127c372d8c8e
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=5217959596e2a02a8ab5d9e0e92929e738862daaac097e9b163bcb5c76e53b10-0  blockHash=000000000e1da6b7b                                                                                                                     14e3371a2e76c8cd5e2a24b1db884d05df0127c372d8c8e
2019-12-17 02:22:16 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=218.85.129.19:33300
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=8580e72c05e57d1d6a6021378ab8b74f1b4d969fe9409b823fc11672b6443023-1  blockHash=00000000af696b7b4                                                                                                                     14bce32dff2d352f36eb85b13ab6666fe4e20535350b877
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=8da65ef440bc5e9967f39e9658b3f2eb9b548c10fcf95b8acb7915e68126682a-0  blockHash=00000001920881c69                                                                                                                     61610966afcbca1b67e8503a003556fcd1ba78d9842cd38
2019-12-17 02:22:16 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=107.145.181.69:33300
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=0b638d5c428f127e9bb3efd11cc0e0974df7d1b23fe4535d6392cb5aa9cad52d-1  blockHash=000000000e1da6b7b                                                                                                                     14e3371a2e76c8cd5e2a24b1db884d05df0127c372d8c8e
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=1352c08f0f571670867b7e565548aedfc92c53d51222924b73089e3381514b2e-0  blockHash=000000000e1da6b7b                                                                                                                     14e3371a2e76c8cd5e2a24b1db884d05df0127c372d8c8e
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=a414be187eb8ab67caad322871e00095e958c194dc77e5b80935eac591c26a2f-0  blockHash=000000000e1da6b7b                                                                                                                     14e3371a2e76c8cd5e2a24b1db884d05df0127c372d8c8e
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=e640d98d6f0e4e2fb04c829687b47d2842e6a52081ada4a99fd9bc417ea9f92f-0  blockHash=00000001f0d4f06b5                                                                                                                     87acca7aabe10b422114c5855b86f05b1d74049d89931ce
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=dd9ed1cab7359c5840402c7f301aee4f0b864e3eca2870dc9bd153b843989933-0  blockHash=000000000e1da6b7b                                                                                                                     14e3371a2e76c8cd5e2a24b1db884d05df0127c372d8c8e
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=2e342b20c03e96c666c216f605cea6d4b8070708fcfa87d00f286de8183a153b-0  blockHash=000000000e1da6b7b                                                                                                                     14e3371a2e76c8cd5e2a24b1db884d05df0127c372d8c8e
2019-12-17 02:22:16 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=218.85.129.71:33300
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=97d42d637e2ecdeb9392c8a28446fed90b22b157ec350fb8ba311aff2b359457-0  blockHash=000000001b3959089                                                                                                                     73a51d5f09fdbe44b742dc44d4994f91f26d98c0b3e7780
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=d4311c9e9867aa581df67de40b816bd003aef3f6159289d9a82348476b62c75c-0  blockHash=000000008fe401fc5                                                                                                                     354fd6557a98c5b2d05b566643c229559bb99dd3613435d
2019-12-17 02:22:16 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=95.216.173.172:33300
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=036da3cee330c9216b842324db44747bd546a7548d1f8319353b6387b25aa963-1  blockHash=000000001b3959089                                                                                                                     73a51d5f09fdbe44b742dc44d4994f91f26d98c0b3e7780
2019-12-17 02:22:16 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=207.148.18.77:33300
2019-12-17 02:22:16 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=215.213.56.28:33300
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=4555480de0b12cdad109285e554dc500c3167b86d7f4e6136d9f497f3edfe56f-1  blockHash=000000000e1da6b7b                                                                                                                     14e3371a2e76c8cd5e2a24b1db884d05df0127c372d8c8e
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=639f8e80ec1383a2a178c44ac146c7d8359059fb4c5453348ddd3b808d93fd72-0  blockHash=0000000082bc45310                                                                                                                     4e189cb1b160d5d96363e785bb2cad6f508951d648845f3
2019-12-17 02:22:16 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=51.38.101.175:33300
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=82899ba71391797f363ad5a4873bb232ef0c4d3ce8d776d618cc35d69bfc267d-0  blockHash=000000003da91ae97                                                                                                                     10ee01d1a0e19c305623b7eaee5417b735a155385ed40f3
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=ea807e5b5e2841094d546ad60581f4b1bd921bc86efcb90c3262b3cec061b386-1  blockHash=000000001b3959089                                                                                                                     73a51d5f09fdbe44b742dc44d4994f91f26d98c0b3e7780
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=a35eec89cd6ad5d06f9b002b4e9275c54c52450ed2d717965410fd6ab627ed88-1  blockHash=000000003da91ae97                                                                                                                     10ee01d1a0e19c305623b7eaee5417b735a155385ed40f3
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=3e4c8c58614364687cf39339b88bc1f24575e70809aa77ba96fbb7bc099ea189-1  blockHash=000000014cde44540                                                                                                                     b212eebb6b5390bff6ef00e21bd59c44ada55cba774f7ed
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=11773f69ef124178c2de2a40870675650ed4b6543593df7cfdb2597bf752f991-0  blockHash=000000001b3959089                                                                                                                     73a51d5f09fdbe44b742dc44d4994f91f26d98c0b3e7780
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=40161d4a00850ee422133f30e956da884463a36c5ce2eea10ea887e4f447c492-0  blockHash=000000000e1da6b7b                                                                                                                     14e3371a2e76c8cd5e2a24b1db884d05df0127c372d8c8e
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=651df3d96e65e570a7f20ce03d7b4c36aeed72cc68d90e3a71b8b9592224679b-0  blockHash=00000001435f69773                                                                                                                     2a49226fb07c601535bf3e198da6ffa8d788356be2c5637
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=f30cc83dd20208b8b6ae5d9daf1afefb92800db6ae7fdf909cb46a5b2714a0a7-0  blockHash=00000000f0d62c383                                                                                                                     6aeb2ec54035e3ae6a2efc02aa42d37b1e926d437bd4329
2019-12-17 02:22:16 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=218.85.129.137:33300
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=8bb354a53d6605d7e08218f1a728bf6b547e6d711142c13991d3bd6ed2f9f5ae-1  blockHash=000000001b3959089                                                                                                                     73a51d5f09fdbe44b742dc44d4994f91f26d98c0b3e7780
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=c1830c666efcd8017470978660daa74bd5448b462360c50a2eb8a7b6b95e41b2-1  blockHash=000000000e1da6b7b                                                                                                                     14e3371a2e76c8cd5e2a24b1db884d05df0127c372d8c8e
2019-12-17 02:22:16 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=202.89.216.11:33300
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=658a3f84d0f1dabc67e0d1321baedf26a0f3f158b282d507660cdff1561fcab2-0  blockHash=000000034d0df9641                                                                                                                     f28da991a82c64c3e004d69d8ff5a12e84695b639e4b81a
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=658a3f84d0f1dabc67e0d1321baedf26a0f3f158b282d507660cdff1561fcab2-0  blockHash=000000034d0df9641                                                                                                                     f28da991a82c64c3e004d69d8ff5a12e84695b639e4b81a
2019-12-17 02:22:16 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=218.85.129.162:33300
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=a7cdfa7a64cd2d54a08f3d931a4b6267b89facc78f47fbd2d31c7f0454a9d7b6-0  blockHash=000000028bf08b856                                                                                                                     ae5467b2ab17aea69bca1504a068e615a2593dbae07c678
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=a7cdfa7a64cd2d54a08f3d931a4b6267b89facc78f47fbd2d31c7f0454a9d7b6-0  blockHash=000000028bf08b856                                                                                                                     ae5467b2ab17aea69bca1504a068e615a2593dbae07c678
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=ac3164f305a5edd7b006fc772c0a64d69ae88b13bc131c273c68cb9ec270cab9-0  blockHash=000000000e1da6b7b                                                                                                                     14e3371a2e76c8cd5e2a24b1db884d05df0127c372d8c8e
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=e3bea13c5e36eb73c8ee3628e16a5963c07a324bc1c43caf5a05f0f1f74c97be-0  blockHash=000000001b3959089                                                                                                                     73a51d5f09fdbe44b742dc44d4994f91f26d98c0b3e7780
2019-12-17 02:22:16 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=202.89.216.40:33300
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=217dad0e75626cd6818cc5398384c91f0b7f8d1adbf1c16543942471772a7ac1-1  blockHash=000000000e1da6b7b                                                                                                                     14e3371a2e76c8cd5e2a24b1db884d05df0127c372d8c8e
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=159f8653f3f7b1ce39fae326f2bc3ef5b9f9245acc43863337d8b81b481d2dc4-0  blockHash=000000000e1da6b7b                                                                                                                     14e3371a2e76c8cd5e2a24b1db884d05df0127c372d8c8e
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=ff510b40d2ecdc8d757be39562e974a8dd0b71c9f67455d9b14e28dd2e2f8fca-0  blockHash=000000000e1da6b7b                                                                                                                     14e3371a2e76c8cd5e2a24b1db884d05df0127c372d8c8e
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=e6eed15a5fa6e661315bfed03d11b4e24ea7ba76ca6e08c4aec66d4007e288cb-11  blockHash=000000000e1da6b7                                                                                                                     b14e3371a2e76c8cd5e2a24b1db884d05df0127c372d8c8e
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=e6eed15a5fa6e661315bfed03d11b4e24ea7ba76ca6e08c4aec66d4007e288cb-35  blockHash=000000000e1da6b7                                                                                                                     b14e3371a2e76c8cd5e2a24b1db884d05df0127c372d8c8e
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=511c52edfc36e3b5815b1c12bf7c0f01c06746910fafa437abc68e76404ed7d1-0  blockHash=000000000e1da6b7b                                                                                                                     14e3371a2e76c8cd5e2a24b1db884d05df0127c372d8c8e
2019-12-17 02:22:16 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=159.69.81.228:33300
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=ba11705ab6500c2b29fbbc5e4be14fb2fbaec555a24edd4dd7d49fe657da58e9-1  blockHash=00000000227ea1920                                                                                                                     dcbd1d10a115002f508d28708f52a901e0e312cb50b3ba1
2019-12-17 02:22:16 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=51.75.18.248:33300
2019-12-17 02:22:16 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=54.38.221.41:33300
2019-12-17 02:22:16 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=51.77.92.5:33300
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=9a5abc62ecedf18512c7128ed2853752185c9e7c97dccd2f94755bee6f95a2f6-1  blockHash=000000003c7b04ad4                                                                                                                     b08407c7052307a5a51b65ca4b37b36103d91e8e598500e
2019-12-17 02:22:16 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=9a5abc62ecedf18512c7128ed2853752185c9e7c97dccd2f94755bee6f95a2f6-1  blockHash=000000003c7b04ad4                                                                                                                     b08407c7052307a5a51b65ca4b37b36103d91e8e598500e
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=c3fca2aa53a503b9ae268c1654c353160e70f23261cc2b8ede60258d40c40202-0  blockHash=0000000064fb1810f                                                                                                                     0fb05cad3ac0acf7218351fbed9ef471bff8d8247eb1ecd
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=79a86fb8d918d9e2a0b6d2801b256a672ce492548cb3d156d3843f20432da811-1  blockHash=0000000064fb1810f                                                                                                                     0fb05cad3ac0acf7218351fbed9ef471bff8d8247eb1ecd
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=9e4c55012f7bfe843878b1645c68d603ec912312f79c5fb25bbf11e0115c8114-0  blockHash=0000000064fb1810f                                                                                                                     0fb05cad3ac0acf7218351fbed9ef471bff8d8247eb1ecd
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=9969865c38e3a2775bb2846754db4459ca83ae1db3a30ee673524aa89aa85816-0  blockHash=0000000064fb1810f                                                                                                                     0fb05cad3ac0acf7218351fbed9ef471bff8d8247eb1ecd
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=bb386adb0bc8e46310b0376e58ce8eb4de27930ad0bfc493b06475ea1885e021-0  blockHash=000000000db3342d0                                                                                                                     1421ca5dd8c068206984cbcf34db8c55d18982567aa51c7
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=ccfbd4341c77f38e429747986783734f5367cad0ead7332f185fb0e712487725-1  blockHash=000000000db3342d0                                                                                                                     1421ca5dd8c068206984cbcf34db8c55d18982567aa51c7
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=7c11d85905b08796f48eb5c031793f5897a9b3bd6de2cf9040c96518b100452a-0  blockHash=000000000db3342d0                                                                                                                     1421ca5dd8c068206984cbcf34db8c55d18982567aa51c7
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=6147de5ad650538debe00d5c9272a7cb7b98d4bdfa3173a82628c5107530612e-1  blockHash=000000000db3342d0                                                                                                                     1421ca5dd8c068206984cbcf34db8c55d18982567aa51c7
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=296d407e2ee78520013b10e7b5c718c83bd3ea4879cf87d13701427c12ddab3a-0  blockHash=000000000db3342d0                                                                                                                     1421ca5dd8c068206984cbcf34db8c55d18982567aa51c7
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=97d42d637e2ecdeb9392c8a28446fed90b22b157ec350fb8ba311aff2b359457-0  blockHash=00000000fd9291608                                                                                                                     ce7a94705a15282b0aa925002afe95a179abfe3e151fb61
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=a77b44cd8c6c64d59696a7f5eb18448868c3f9b503635a6f47d7ed9c7075245b-1  blockHash=000000000db3342d0                                                                                                                     1421ca5dd8c068206984cbcf34db8c55d18982567aa51c7
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=5274ca946052000165459ada3f51545e530b086a345b9404a7dfb5458c131a69-0  blockHash=0000000064fb1810f                                                                                                                     0fb05cad3ac0acf7218351fbed9ef471bff8d8247eb1ecd
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=215e8689a61545ae38a9fc768424579e5aedee17d2f7da5972a786e3d6340b6c-0  blockHash=000000000db3342d0                                                                                                                     1421ca5dd8c068206984cbcf34db8c55d18982567aa51c7
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=e3875481ac936f65866033f7d65e6605886947d8156c5c7a8d9f7be7a4611471-0  blockHash=000000000db3342d0                                                                                                                     1421ca5dd8c068206984cbcf34db8c55d18982567aa51c7
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=a90e3d214ec20c9588f1dfa186ddd4e08ccee5163e3242d12d09512d4970c573-0  blockHash=0000000064fb1810f                                                                                                                     0fb05cad3ac0acf7218351fbed9ef471bff8d8247eb1ecd
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=06f27a6f344527604ac56614935fbfb6b48231f9891897fb84d638191ea7e676-0  blockHash=000000000db3342d0                                                                                                                     1421ca5dd8c068206984cbcf34db8c55d18982567aa51c7
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=4dd1897ee1398dc987cc1cc134f5e68e4bc2366c663b7c1d828516a38e520977-0  blockHash=000000000db3342d0                                                                                                                     1421ca5dd8c068206984cbcf34db8c55d18982567aa51c7
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=32fbb659e27c24a7db8ebdf8ac8b936fc4399d23f2f477d10e04b6c89a7b1888-0  blockHash=000000000db3342d0                                                                                                                     1421ca5dd8c068206984cbcf34db8c55d18982567aa51c7
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=ed97a3cede07cc8ae3b5f3e88561d16ae534c428258f59e3de065892651b4d8b-0  blockHash=000000000db3342d0                                                                                                                     1421ca5dd8c068206984cbcf34db8c55d18982567aa51c7
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=6f6a97d6c8a94aa5fa1a94f5fcec5a9c7c92480f1ebd121463f82c86c16aa191-0  blockHash=0000000064fb1810f                                                                                                                     0fb05cad3ac0acf7218351fbed9ef471bff8d8247eb1ecd
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=cbc2cac0efa5ea1f27d399aee5a79c896b57dd499aebb0119e0761a674afb2a1-1  blockHash=0000000064fb1810f                                                                                                                     0fb05cad3ac0acf7218351fbed9ef471bff8d8247eb1ecd
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=0892bf38fb83e370f840bfee5b1f91f5ae04a4872151b57206568f6e8f60e2a2-1  blockHash=000000000db3342d0                                                                                                                     1421ca5dd8c068206984cbcf34db8c55d18982567aa51c7
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=f18edaadb5a04d2f78b4824cf57cbf33c2fa40b4d7da6997519975da39a1e5b0-0  blockHash=0000000145480cee4                                                                                                                     471ebdc9b8fe094bfe35ed117260fca4163798567acdba6
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=c1830c666efcd8017470978660daa74bd5448b462360c50a2eb8a7b6b95e41b2-1  blockHash=0000000064fb1810f                                                                                                                     0fb05cad3ac0acf7218351fbed9ef471bff8d8247eb1ecd
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=0c1be35467fbaa5deac75e1c075ff33528656945ac75e54d84fe96621440f7bb-0  blockHash=000000000db3342d0                                                                                                                     1421ca5dd8c068206984cbcf34db8c55d18982567aa51c7
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=9f222723a9c260f607d1d4c8716e5eb0a41c364159a87be1e025a02eb3be37c4-0  blockHash=000000000db3342d0                                                                                                                     1421ca5dd8c068206984cbcf34db8c55d18982567aa51c7
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=a3c4d69b4bd0971effd02a420200f7d0fed555f6c0b2fe8c985cc743849bf7db-1  blockHash=000000001d3e2383f                                                                                                                     c345aae03743999d3e72e00f5ea53b255fc1b0214a53649
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=0876dfd4917f2c2743b4236a62618be7ad41a34dd65e9c1818d70122ddc17be6-0  blockHash=00000000112b17108                                                                                                                     f558f429db99a7c6b74824ebce2c4461eab761043f0187b
2019-12-17 02:22:22 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=8ba6a3664e87d970d96fec9d8f26d8518abb37b690d843a37f964fb76af23cf5-1  blockHash=0000000064fb1810f                                                                                                                     0fb05cad3ac0acf7218351fbed9ef471bff8d8247eb1ecd
2019-12-17 02:22:24 AcceptConnection -- dynode is not synced yet, skipping inbound connection attempt
2019-12-17 02:22:25 AcceptConnection -- dynode is not synced yet, skipping inbound connection attempt
2019-12-17 02:22:44 AcceptConnection -- dynode is not synced yet, skipping inbound connection attempt
2019-12-17 02:22:46 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=dc111c640463ecf5e3fea5bcba30846c330e260c51620b8aec0d97b128c6f907-0  blockHash=000000001b3959089                                                                                                                     73a51d5f09fdbe44b742dc44d4994f91f26d98c0b3e7780
2019-12-17 02:22:46 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=c58e7940a208beef815d17ec53cb348c68dcb5e859debf75909a68cb21011209-1  blockHash=000000001b3959089                                                                                                                     73a51d5f09fdbe44b742dc44d4994f91f26d98c0b3e7780
2019-12-17 02:22:46 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=3c17ba66af744b8687a7135831c2531355003e10fb73babb1a419c4a59c52616-1  blockHash=000000001b3959089                                                                                                                     73a51d5f09fdbe44b742dc44d4994f91f26d98c0b3e7780
2019-12-17 02:22:46 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=53fc51c54c78885e6a76f8002d6c6746ece7beb2505f1daf288b002c8db6531e-1  blockHash=000000001b3959089                                                                                                                     73a51d5f09fdbe44b742dc44d4994f91f26d98c0b3e7780
2019-12-17 02:22:46 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=a414be187eb8ab67caad322871e00095e958c194dc77e5b80935eac591c26a2f-0  blockHash=000000001b3959089                                                                                                                     73a51d5f09fdbe44b742dc44d4994f91f26d98c0b3e7780
2019-12-17 02:22:46 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=1c2b71ac1ce8206e5b153ea2558f9634bbc6a662333b1abd3b1fd0431bbe0665-1  blockHash=000000001b3959089                                                                                                                     73a51d5f09fdbe44b742dc44d4994f91f26d98c0b3e7780
2019-12-17 02:22:46 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=11773f69ef124178c2de2a40870675650ed4b6543593df7cfdb2597bf752f991-0  blockHash=000000001b3959089                                                                                                                     73a51d5f09fdbe44b742dc44d4994f91f26d98c0b3e7780
2019-12-17 02:22:46 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=2f0bcebaf37d4e1e1e92392c41c68d6fa13b49243dba755a062b61d2a10b2e95-0  blockHash=000000001b3959089                                                                                                                     73a51d5f09fdbe44b742dc44d4994f91f26d98c0b3e7780
2019-12-17 02:22:46 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=a3c4d69b4bd0971effd02a420200f7d0fed555f6c0b2fe8c985cc743849bf7db-1  blockHash=000000001b3959089                                                                                                                     73a51d5f09fdbe44b742dc44d4994f91f26d98c0b3e7780
2019-12-17 02:22:49 AcceptConnection -- dynode is not synced yet, skipping inbound connection attempt
2019-12-17 02:22:51 AcceptConnection -- dynode is not synced yet, skipping inbound connection attempt
2019-12-17 02:23:05 AcceptConnection -- dynode is not synced yet, skipping inbound connection attempt
2019-12-17 02:23:14 socket no message in first 60 seconds, 0 1 from 2
2019-12-17 02:23:14 AcceptConnection -- dynode is not synced yet, skipping inbound connection attempt
2019-12-17 02:23:46 AcceptConnection -- dynode is not synced yet, skipping inbound connection attempt
2019-12-17 02:23:52 AcceptConnection -- dynode is not synced yet, skipping inbound connection attempt
2019-12-17 02:24:20 AcceptConnection -- dynode is not synced yet, skipping inbound connection attempt

... (At this point we see the Dynode's IP address and it appears to be set to the NEW_START_REQUIRED state)

2019-12-17 02:24:28 CActiveDynode::ManageStateInitial -- Checking inbound connection to 'xxx.xxx.xxx.xxx:33300'
2019-12-17 02:24:28 CActiveDynode::ManageStateRemote -- NOT_CAPABLE: Dynode in NEW_START_REQUIRED state
2019-12-17 02:24:29 GetDynodeHashID -- Dynode Service Address xxx.xxx.xxx.xxx:33300,  HashID 5f3c8c9a41713e867ce41bd1cffc434b729a7a34
2019-12-17 02:24:29 StartDHTNetwork -- starting session #0
2019-12-17 02:24:29 CDHTSettings -- listening interfaces: 0.0.0.0:33311,[::]:33311
2019-12-17 02:24:29 CDHTSettings::LoadPeerID -- peer_fingerprint = 5f3c8c9a41713e867ce41bd1cffc434b729a7a34

The Dynode invalidates itself and it is unrecoverable without a restart from the control wallet.

Do we need an extra check here?

// don't expire if we are still in "waiting for ping" mode unless it's our own dynode
    if (!fWaitForPing || fOurDynode) {
        if (!IsPingedWithin(DYNODE_NEW_START_REQUIRED_SECONDS)) {
            nActiveState = DYNODE_NEW_START_REQUIRED;
            if (nActiveStatePrev != nActiveState) {
                LogPrint("dynode", "CDynode::Check -- Dynode %s is in %s state now\n", outpoint.ToStringShort(), GetStateString());
            }
            return;
        }
    ...

What platform are you using (Linux, Windows, Mac)

Ubuntu 18.04 LTS

[LordDarkHelmet]

Here is an example of a successful killing and restarting of a Dynode with an unresponsive CLI. In this case the Dynode state is set to ACTIVE_DYNODE_STARTED and accordingly a Dynode ping is sent.

... (Last known Dynode Ping before restart)
2019-12-17 12:27:35 CActiveDynode::SendDynodePing -- Relaying ping, collateral=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-1
... (Killed then restarted due to non-responsive CLI)
2019-12-17 12:39:12 


2019-12-17 12:39:12 Dynamic version v2.4.3.0-8873e60b9

2019-12-17 12:39:12 Bound to xxx.xxx.xxx.xxx:33300
2019-12-17 12:39:12 AddLocal(xxx.xxx.xxx.xxx:33300,4)

2019-12-17 12:39:15 msghand thread start
2019-12-17 12:39:15 StartDHTNetwork -- starting
2019-12-17 12:39:15 Imported mempool transactions from disk: 0 successes, 0 failed, 3 expired
2019-12-17 12:39:15 AcceptConnection -- dynode is not synced yet, skipping inbound connection attempt
2019-12-17 12:39:26 Loading addresses from DNS seeds (could take a while)
2019-12-17 12:39:33 AcceptConnection -- dynode is not synced yet, skipping inbound connection attempt
2019-12-17 12:39:36 AcceptConnection -- dynode is not synced yet, skipping inbound connection attempt
2019-12-17 12:39:36 16 addresses found from DNS seeds
2019-12-17 12:39:36 dnsseed thread exit
2019-12-17 12:39:45 CDynode::Check -- Dynode 557713aae29fc0dff460f8cbed9712f8eaa6073ba7dcfaa302273c7969030328-3 is unbanned and back in list now
2019-12-17 12:39:45 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=9c8536bf91b4ef43a70f0f3ebdf7488b9589b7898a161a86e0eebe97d17c0f00-0  blockHash=000000000620bed8c8b522a7e69cf1a88f4b04f8f21047116df6f52810351414
2019-12-17 12:39:45 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=107.145.181.3:33300
2019-12-17 12:39:45 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=9d673858b5bfa82e59f13374aba3312f00577a6b7d14fbf35671670533733200-0  blockHash=000000034d0df9641f28da991a82c64c3e004d69d8ff5a12e84695b639e4b81a
2019-12-17 12:39:45 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=45.86.68.200:33300
2019-12-17 12:39:45 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=95.217.91.94:33300
2019-12-17 12:39:45 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=95.217.91.56:33300
2019-12-17 12:39:45 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=107.145.181.5:33300
2019-12-17 12:39:45 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=112.194.104.101:33300
2019-12-17 12:39:45 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=457e8b4116e227be15429145b5308554c3ff65bf2de3f4ea7a1b46c77d9e5d01-1  blockHash=0000000219e41a8b321c6a3da82a15bbc49699db3036c65cbc7a499812cbc23d
2019-12-17 12:39:45 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=95.217.91.89:33300
2019-12-17 12:39:45 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=223.186.188.131:33300
2019-12-17 12:39:45 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=c59d892eb0298977a45125fcc51b0dadc3ea4b7cb7ef7e73d0061e79d09d5e02-1  blockHash=000000002c82f162c245b0f8bba665c978294f120ac587f7c909b3a1597da10f
2019-12-17 12:39:45 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=c59d892eb0298977a45125fcc51b0dadc3ea4b7cb7ef7e73d0061e79d09d5e02-1  blockHash=000000002c82f162c245b0f8bba665c978294f120ac587f7c909b3a1597da10f
2019-12-17 12:39:45 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=223.186.188.82:33300
2019-12-17 12:39:45 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=37d81a5517f5a9abb1c6683404ce482791ab0572329426a4e795507956d7b502-0  blockHash=000000002c82f162c245b0f8bba665c978294f120ac587f7c909b3a1597da10f
2019-12-17 12:39:45 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=37d81a5517f5a9abb1c6683404ce482791ab0572329426a4e795507956d7b502-0  blockHash=000000002c82f162c245b0f8bba665c978294f120ac587f7c909b3a1597da10f
2019-12-17 12:39:45 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=223.186.188.126:33300
2019-12-17 12:39:45 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=ad6aa8a09fff639da78338a769f481de0b78925da21b28ef155af0a22223df03-0  blockHash=000000002c82f162c245b0f8bba665c978294f120ac587f7c909b3a1597da10f
2019-12-17 12:39:45 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=ad6aa8a09fff639da78338a769f481de0b78925da21b28ef155af0a22223df03-0  blockHash=000000002c82f162c245b0f8bba665c978294f120ac587f7c909b3a1597da10f
2019-12-17 12:39:45 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=95.217.91.109:33300
...(removed 1060 lines of the same pattern for simplicity.)
2019-12-17 12:39:46 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=212.24.103.6:33300
2019-12-17 12:39:46 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=8c0ce22ab2b2f5ac2733d2955163ad5fdf6aac8901895b44165cf2648aa0f3e2-0  blockHash=00000000271f196a2dede805597f15ab519faa17bfac2a09fa158494a8e56607
2019-12-17 12:39:46 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=8c0ce22ab2b2f5ac2733d2955163ad5fdf6aac8901895b44165cf2648aa0f3e2-0  blockHash=00000000fbe92855803d8c54d2c31e3a94c8925d0f01c61b10f9bb63efb2062f
2019-12-17 12:39:46 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=80.211.17.55:33300
2019-12-17 12:39:46 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=bbc782cb6b8bff35303ae59c588b9e046d2fd9ef160dc6de59a912af01627ae3-1  blockHash=00000000301d39b3247c157a26bbd4b1f93fb16799081aea1f5f9096dd385585
2019-12-17 12:39:46 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=112.194.101.132:33300
2019-12-17 12:39:46 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=4edd434433905f4f6877a2c0a685ae45636a01c64c765e5efea07b05def3a5e3-0  blockHash=000000011f48ddda1d1404189013c16ff48dab97bdb099bba1700beef8ba87e3
2019-12-17 12:39:46 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=112.194.101.133:33300
2019-12-17 12:39:46 CDynodePing::CheckAndUpdate -- Dynode ping is invalid, block hash is too old: dynode=35cdce0837edc9d593825cb49a7f2bdfa88552105fa4ce9c346e042140c82ae4-0  blockHash=000000005cca54917df42737cf14cc08ff7265b9d4c8d5537bf3c67cb99a20f3
2019-12-17 12:39:46 CDynodeBroadcast::Update -- Got UPDATED Dynode entry: addr=xxx.xxx.xxx.xxx:33300
2019-12-17 12:39:46 CActiveDynode::ManageStateInitial -- Checking inbound connection to 'xxx.xxx.xxx.xxx:33300'
2019-12-17 12:39:46 AcceptConnection -- dynode is not synced yet, skipping inbound connection attempt
2019-12-17 12:39:46 CActiveDynode::ManageStateRemote -- STARTED!
2019-12-17 12:39:46 CActiveDynode::SendDynodePing -- Relaying ping, collateral=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-1

[LordDarkHelmet]

I am getting more and more reports of Dynodes crashing, then on immediate restart, they go into the NEW_START_REQUIRED state. By all accounts this should not be happening, but it is. This is happening with a single Dynode running on a dedicated VPS. I rewrote a script for Edgemaster that detects stalled Dynodes, stops them, deletes the Dynode cache file, then immediately restarts the Dynode in the hope that getting rid of old Dynode cache data would help. It defiantly improved the situation, but has not solved it.

Then I was thinking, if the cache file is gone, it is rebuilding its own state from the network. We know that from any control wallet that the state of a Dynode not guaranteed to be correct. Any nodes data about other nodes could be out of date and we should only trust data from the Dynode itself. But those nodes with out of date information are sending out reports. If our Dynode has incorrect data in its own cache file, or if that cache file is deleted the first report about our Dynode contains incorrect state data then our Dynode could slip into a DYNODE_NEW_START_REQUIRED state.

The only place in the code where you can get into the DYNODE_NEW_START_REQUIRED state is in dynode.cpp on line 209.

What are the consequences of ignoring external data and not going into the new state required? At this point I am trusting others to give me correct information about myself. (Even just a single node giving me incorrect or out of date information)

Perhaps a solution is to only change the state when the sync is finished. This will allow time for more than one report to arrive, and hopefully the data in those reports are accurate.

    // don't expire if we are still in "waiting for ping" mode unless it's our own dynode
    if (!fWaitForPing || fOurDynode) {
        if (!IsPingedWithin(DYNODE_NEW_START_REQUIRED_SECONDS)) {
            if (dynodeSync.IsDynodeListSynced()) {
                nActiveState = DYNODE_NEW_START_REQUIRED;
                if (nActiveStatePrev != nActiveState) {
                    LogPrint("dynode", "CDynode::Check -- Dynode %s is in %s state now\n", outpoint.ToStringShort(), GetStateString());
                }
            }
            return;
        }
...

This still requires trust, so the possibility that malicious nodes that send stale/incorrect data in an attempt to bring down nodes that are restarting still exists.

The core of the issue seems to be corrupted or out of date information, this does not fix that issue, but it should make it more resilient to its presence.

I can create a pull request for the above, but wanted some feedback before I do that. Unfortunately, the issue is very hard to reproduce for debugging, so discussing this change and potential side effects is probably the best thing to do right now.

[Duality-CDOO]

We could attempt to go through attempting to rectify this issue, however, with the hardfork happening in 2.5.0.0 for Proof of Stake, it might simply be worth waiting until the network is up on that version and then revisiting this then.

[LordDarkHelmet]

With @AmirAbrams 's recent changes It may be a good opportunity to look at what is happening with dynode invalidation here.

Perhaps the psudo code I suggested above might help the situation. Right now it is possible for another Dynode who has out of date information or is acting maliciously to move a just rebooted Dynode into the DYNODE_NEW_START_REQUIRED state by sending old information.