Merge pull request #654 from ton-blockchain/testnet

Merge developer branch

Author: EmelyanenkoK

Changelog.md

@@ -1,18 +1,28 @@
-## 03.2023 Update
+## 2023.04 Update
+1. CPU load optimization: previous DHT reconnect policy was too aggressive
+2. Network throughput improvements: granular control on external message broadcast, optimize celldb GC, adjust state serialization and block downloading timings, rldp2 for states and archives 
+3. Update for Fift (namespaces) and Fift libraries (list of improvements: https://github.com/ton-blockchain/ton/issues/631)
+4. Better handling of incorrect inputs in funC: fix UB and prevent crashes on some inputs, improve optimizing int consts and unused variables in FunC, fix analyzing repeat loop. FunC version is increase to 0.4.3.
+5. `listBlockTransactionsExt` in liteserver added
+6. Tvm emulator improvements
+
+Besides the work of the core team, this update is based on the efforts of @krigga (tvm emulator improvement), @ex3ndr (`PUSHSLICE` fift-asm improvement) and [sec3-service](https://github.com/sec3-service) security auditors (funC improvements).
+
+## 2023.03 Update
 1. Improvement of ADNL connection stability
 2. Transaction emulator support and getAccountStateByTransaction method
 3. Fixes of typos, undefined behavior and timer warnings
 4. Handling incorrect integer literal values in funC; funC version bumped to 0.4.2
 5. FunC Mathlib
 
-## 01.2023 Update
+## 2023.01 Update
 1. Added ConfigParam 44: `SuspendedAddressList`. Upon being set this config suspends initialisation of **uninit** addresses from the list for given time.
 2. FunC: `v0.4.1` added pragmas for precise control of computation order
 3. FunC: fixed compiler crashes for some exotic inputs
 4. FunC: added legacy tester, a collection of smart-contracts which is used to check whether compilator update change compilation result
 5. Improved archive manager: proper handling of recently garbage-collected blocks
 
-## 12.2022 Update
+## 2022.12 Update
 Node update:
 1. Improvements of ton-proxy: fixed few bugs, improved stability
 2. Improved collator/validator checks, added optimization of storage stat calculation, generation and validation of new blocks is made safer
@@ -27,7 +37,7 @@ Node update:
 
 Besides the work of the core team, this update is based on the efforts of @vtamara (help with abseil-cpp upgrade), @krigga(in-place modification of global variables) and third-party security auditors.
 
-## 10.2022 Update
+## 2022.10 Update
 * Added extended block creation and general perfomance stats gathering
 * Forbidden report data on blocks not committed to the master chain for LS
 * Improved debug in TVM
@@ -39,7 +49,7 @@ Besides the work of the core team, this update is based on the efforts of @vtama
 
 Besides the work of the core team, this update is based on the efforts of @tvorogme (debug improvements), @AlexeyFSL (WASM builds)  and third-party security auditors.
 
-## 08.2022 Update
+## 2022.08 Update
 * Blockchain state serialization now works via separate db-handler which simplfies memory clearing after serialization
 * CellDB now works asynchronously which substantially increase database access throughput
 * Abseil-cpp and crc32 updated: solve issues with compilation on recent OS distributives
@@ -54,7 +64,7 @@ Update coincided with persistent state serialization event which lead to block p
 
 Besides the work of the core team, this update is based on the efforts of @awesome-doge (help with abseil-cpp upgrade), @rec00rsiff (noted issues for exotic endianess and implemented network stats) and third-party security auditors.
 
-## 05.2022 Update
+## 2022.05 Update
 * Initial synchronization improved: adjusted timeouts for state download and the way of choosing which state to download. Nodes with low network speed and/or bad connectivity will synchronize faster and consistently.
 * Improved peer-to-peer network stability and DDoS resistance: now peers will only relay valid messages to the network. Large messages, which require splitting for relaying, will be retranslated as well, but only after the node gets all parts, and reassembles and checks them. Validators may sign certificates for network peers, which allow relaying large messages by parts without checks. It is used now by validators to faster relay new blocks. Sign and import certificate commands are exposed via `validator-engine-console`.
 * Fixed some rare edge cases in TVM arithmetic operations related to big numbers (`2**63+`)

adnl/adnl-peer.cpp

@@ -944,7 +944,6 @@ void AdnlPeerPairImpl::got_data_from_dht(td::Result<AdnlNode> R) {
   CHECK(dht_query_active_);
   dht_query_active_ = false;
   next_dht_query_at_ = td::Timestamp::in(td::Random::fast(60.0, 120.0));
-  alarm_timestamp().relax(next_dht_query_at_);
   if (R.is_error()) {
     VLOG(ADNL_INFO) << this << ": dht query failed: " << R.move_as_error();
     return;

crypto/CMakeLists.txt

@@ -143,6 +143,7 @@ set(FIFT_SOURCE
   fift/Dictionary.cpp
   fift/Fift.cpp
   fift/IntCtx.cpp
+  fift/HashMap.cpp
   fift/Continuation.cpp
   fift/SourceLookup.cpp
   fift/utils.cpp
@@ -151,6 +152,7 @@ set(FIFT_SOURCE
   fift/Dictionary.h
   fift/Fift.h
   fift/IntCtx.h
+  fift/HashMap.h
   fift/Continuation.h
   fift/SourceLookup.h
   fift/utils.h
@@ -300,6 +302,10 @@ target_link_libraries(test-ed25519-crypto PUBLIC ton_crypto)
 add_library(fift-lib ${FIFT_SOURCE})
 target_include_directories(fift-lib PUBLIC $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}>)
 target_link_libraries(fift-lib PUBLIC ton_crypto ton_db tdutils ton_block)
+if (USE_EMSCRIPTEN)
+  target_link_options(fift-lib PRIVATE -fexceptions)
+  target_compile_options(fift-lib PRIVATE -fexceptions)
+endif()
 set_target_properties(fift-lib PROPERTIES OUTPUT_NAME fift)
 
 add_executable(fift fift/fift-main.cpp)
@@ -328,17 +334,20 @@ if (USE_EMSCRIPTEN)
   add_executable(funcfiftlib funcfiftlib/funcfiftlib.cpp ${FUNC_LIB_SOURCE})
   target_include_directories(funcfiftlib PUBLIC $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}>)
   target_link_libraries(funcfiftlib PUBLIC fift-lib src_parser git)
-  target_link_options(funcfiftlib PRIVATE -sEXPORTED_RUNTIME_METHODS=FS,ccall,cwrap,_malloc,free,UTF8ToString,stringToUTF8)
-  target_link_options(funcfiftlib PRIVATE -sEXPORTED_FUNCTIONS=_func_compile,_version)
+  target_link_options(funcfiftlib PRIVATE -sEXPORTED_RUNTIME_METHODS=FS,ccall,cwrap,UTF8ToString,stringToUTF8,lengthBytesUTF8,addFunction,removeFunction,setValue)
+  target_link_options(funcfiftlib PRIVATE -sEXPORTED_FUNCTIONS=_func_compile,_version,_malloc,_free,_setThrew)
   target_link_options(funcfiftlib PRIVATE -sEXPORT_NAME=CompilerModule)
   target_link_options(funcfiftlib PRIVATE -sERROR_ON_UNDEFINED_SYMBOLS=0)
-  target_link_options(funcfiftlib PRIVATE -sFILESYSTEM=1)
+  target_link_options(funcfiftlib PRIVATE -sFILESYSTEM=1 -lnodefs.js)
   target_link_options(funcfiftlib PRIVATE -Oz)
   target_link_options(funcfiftlib PRIVATE -sIGNORE_MISSING_MAIN=1)
   target_link_options(funcfiftlib PRIVATE -sAUTO_NATIVE_LIBRARIES=0)
   target_link_options(funcfiftlib PRIVATE -sMODULARIZE=1)
+  target_link_options(funcfiftlib PRIVATE -sALLOW_MEMORY_GROWTH=1)
+  target_link_options(funcfiftlib PRIVATE -sALLOW_TABLE_GROWTH=1)
   target_link_options(funcfiftlib PRIVATE --embed-file ${CMAKE_CURRENT_SOURCE_DIR}/fift/lib@/fiftlib)
-  target_compile_options(funcfiftlib PRIVATE -sDISABLE_EXCEPTION_CATCHING=0)
+  target_link_options(funcfiftlib PRIVATE -fexceptions)
+  target_compile_options(funcfiftlib PRIVATE -fexceptions)
 endif()
 
 add_executable(tlbc tl/tlbc.cpp)

crypto/block/check-proof.cpp

@@ -315,6 +315,113 @@ td::Result<TransactionList::Info> TransactionList::validate() const {
   return std::move(res);
 }
 
+td::Result<BlockTransaction::Info> BlockTransaction::validate(bool check_proof) const {
+  if (root.is_null()) {
+    return td::Status::Error("transactions are expected to be non-empty");
+  }
+  if (check_proof && proof->get_hash().bits().compare(root->get_hash().bits(), 256)) {
+    return td::Status::Error(PSLICE() << "transaction hash mismatch: Merkle proof expects "
+                                      << proof->get_hash().bits().to_hex(256)
+                                      << " but received data has " << root->get_hash().bits().to_hex(256));
+  }
+  block::gen::Transaction::Record trans;
+  if (!tlb::unpack_cell(root, trans)) {
+    return td::Status::Error("cannot unpack transaction cell");
+  }
+  Info res;
+  res.blkid = blkid;
+  res.now = trans.now;
+  res.lt = trans.lt;
+  res.hash = root->get_hash().bits();
+  res.transaction = root;
+  return std::move(res);
+}
+
+td::Result<BlockTransactionList::Info> BlockTransactionList::validate(bool check_proof) const {
+  constexpr int max_answer_transactions = 256;
+
+  TRY_RESULT_PREFIX(list, vm::std_boc_deserialize_multi(std::move(transactions_boc)), "cannot deserialize transactions boc: ");  
+  std::vector<td::Ref<vm::Cell>> tx_proofs(list.size());
+
+  if (check_proof) {
+    try {
+      TRY_RESULT(proof_cell, vm::std_boc_deserialize(std::move(proof_boc)));
+      auto virt_root = vm::MerkleProof::virtualize(proof_cell, 1);
+
+      if (blkid.root_hash != virt_root->get_hash().bits()) {
+        return td::Status::Error("Invalid block proof root hash");
+      }
+      block::gen::Block::Record blk;
+      block::gen::BlockExtra::Record extra;
+      if (!(tlb::unpack_cell(virt_root, blk) && tlb::unpack_cell(std::move(blk.extra), extra))) {
+        return td::Status::Error("Error unpacking proof cell");
+      }
+      vm::AugmentedDictionary acc_dict{vm::load_cell_slice_ref(extra.account_blocks), 256,
+                  block::tlb::aug_ShardAccountBlocks};
+
+      bool eof = false;
+      ton::LogicalTime reverse = reverse_mode ? ~0ULL : 0;
+      ton::LogicalTime trans_lt = static_cast<ton::LogicalTime>(start_lt);
+      td::Bits256 cur_addr = start_addr;
+      bool allow_same = true;
+      int count = 0;
+      while (!eof && count < req_count && count < max_answer_transactions) {
+        auto value = acc_dict.extract_value(
+              acc_dict.vm::DictionaryFixed::lookup_nearest_key(cur_addr.bits(), 256, !reverse, allow_same));
+        if (value.is_null()) {
+          eof = true;
+          break;
+        }
+        allow_same = false;
+        if (cur_addr != start_addr) {
+          trans_lt = reverse;
+        }
+
+        block::gen::AccountBlock::Record acc_blk;
+        if (!tlb::csr_unpack(std::move(value), acc_blk) || acc_blk.account_addr != cur_addr) {
+          return td::Status::Error("Error unpacking proof account block");
+        }
+        vm::AugmentedDictionary trans_dict{vm::DictNonEmpty(), std::move(acc_blk.transactions), 64,
+                    block::tlb::aug_AccountTransactions};
+        td::BitArray<64> cur_trans{(long long)trans_lt};
+        while (count < req_count && count < max_answer_transactions) {
+          auto tvalue = trans_dict.extract_value_ref(
+                trans_dict.vm::DictionaryFixed::lookup_nearest_key(cur_trans.bits(), 64, !reverse));
+          if (tvalue.is_null()) {
+            trans_lt = reverse;
+            break;
+          }
+          if (static_cast<size_t>(count) < tx_proofs.size()) {
+            tx_proofs[count] = std::move(tvalue);
+          }
+          count++;
+        }
+      }
+      if (static_cast<size_t>(count) != list.size()) {
+        return td::Status::Error(PSLICE() << "Txs count mismatch in proof (" << count << ") and response (" << list.size() << ")");
+      }
+    } catch (vm::VmError& err) {
+      return err.as_status("Couldn't verify proof: ");
+    } catch (vm::VmVirtError& err) {
+      return err.as_status("Couldn't verify proof: ");
+    } catch (...) {
+      return td::Status::Error("Unknown exception raised while verifying proof");
+    }
+  }
+
+  Info res;
+  for (int i = 0; i < static_cast<int>(list.size()); i++) {
+    auto& root = list[i];
+    BlockTransaction transaction;
+    transaction.root = root;
+    transaction.blkid = blkid;
+    transaction.proof = tx_proofs[i];
+    TRY_RESULT(info, transaction.validate(check_proof));
+    res.transactions.push_back(std::move(info));
+  }
+  return std::move(res);
+}
+
 td::Status BlockProofLink::validate(td::uint32* save_utime) const {
   if (save_utime) {
     *save_utime = 0;

crypto/block/check-proof.h

@@ -88,4 +88,36 @@ struct TransactionList {
   td::Result<Info> validate() const;
 };
 
+struct BlockTransaction {
+  ton::BlockIdExt blkid;
+  td::Ref<vm::Cell> root;
+  td::Ref<vm::Cell> proof;
+
+  struct Info {
+    ton::BlockIdExt blkid;
+    td::uint32 now;
+    ton::LogicalTime lt;
+    ton::Bits256 hash;
+    td::Ref<vm::Cell> transaction;
+  };
+  td::Result<Info> validate(bool check_proof) const;
+};
+
+struct BlockTransactionList {
+  ton::BlockIdExt blkid;
+  td::BufferSlice transactions_boc;
+  td::BufferSlice proof_boc;
+  ton::LogicalTime start_lt;
+  td::Bits256 start_addr;
+  bool reverse_mode;
+  int req_count;
+
+  struct Info {
+    ton::BlockIdExt blkid;
+    std::vector<BlockTransaction::Info> transactions;
+  };
+
+  td::Result<Info> validate(bool check_proof) const;
+};
+
 }  // namespace block

crypto/block/create-state.cpp

@@ -47,6 +47,7 @@
 #include "fift/Fift.h"
 #include "fift/Dictionary.h"
 #include "fift/SourceLookup.h"
+#include "fift/IntCtx.h"
 #include "fift/words.h"
 
 #include "td/utils/logging.h"
@@ -866,8 +867,9 @@ int main(int argc, char* const argv[]) {
       case 'v':
         new_verbosity_level = VERBOSITY_NAME(FATAL) + (verbosity = td::to_integer<int>(td::Slice(optarg)));
         break;
-      case 'V':        
-        std::cout << "create-state build information: [ Commit: " << GitMetadata::CommitSHA1() << ", Date: " << GitMetadata::CommitDate() << "]\n";
+      case 'V':
+        std::cout << "create-state build information: [ Commit: " << GitMetadata::CommitSHA1()
+                  << ", Date: " << GitMetadata::CommitDate() << "]\n";
         std::exit(0);
         break;
       case 'h':

crypto/block/transaction.cpp

@@ -704,7 +704,9 @@ bool Transaction::prepare_storage_phase(const StoragePhaseConfig& cfg, bool forc
       switch (acc_status) {
         case Account::acc_uninit:
         case Account::acc_frozen:
-          if (total_due > cfg.delete_due_limit) {
+          if (total_due > cfg.delete_due_limit && balance.extra.is_null()) {
+            // Keeping accounts with non-null extras is a temporary measure before implementing proper collection of
+            // extracurrencies from deleted accounts
             res->deleted = true;
             acc_status = Account::acc_deleted;
             if (balance.extra.not_null()) {