Heavy memory usage from nodes that doesn't have a lot of activity

[NorseGaud]

$ firod --version
Firo Core Daemon version v0.14.14.0-ge0c423805

$ free -m
              total        used        free      shared  buff/cache   available
Mem:           1851        1709          71           0          69          37
Swap:          2047        1026        1021

top - 15:52:13 up 9 days,  1:09,  1 user,  load average: 0.10, 0.03, 0.00
Tasks: 129 total,   1 running,  78 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.0 us,  0.5 sy,  0.0 ni, 98.3 id,  0.0 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem :  1895928 total,    73944 free,  1750648 used,    71336 buff/cache
KiB Swap:  2097148 total,  1045872 free,  1051276 used.    38272 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                      
 5803 user  20   0 4113364   1.6g    300 S   1.0 88.5 346:16.55 /usr/local/bin/firod 

$ firo-cli quorum memberof 02687058ebe1db2a02b9650075b01701ec418a56e 
[
]

$ firo-cli evoznsync status; firo-cli getinfo; firo-cli evoznode status
{
  "AssetID": 999,
  "AssetName": "MASTERNODE_SYNC_FINISHED",
  "AssetStartTime": 1725395215,
  "Attempt": 0,
  "IsBlockchainSynced": true,
  "IsSynced": true,
  "IsFailed": false
}
{
  "version": 141400,
  "protocolversion": 90031,
  "blocks": 957058,
  "timeoffset": 0,
  "connections": 110,
  "proxy": "",
  "difficulty": 3053.880906586081,
  "testnet": false,
  "paytxfee": 0.00000000,
  "mininput": 0.00001000,
  "relayfee": 0.00001000,
  "errors": ""
}
{
  "outpoint": "COutPoint(9f81bf833521c3219abbac75dbbcd6682a0ed1a8ff7b2591d18746e645c03877, 1)",
  "service": "18.181.40.131:8168",
  "proTxHash": "02687058ebe1db2a0154a17456459a3f06a53a79e2b9650075b01701ec418a56",
  "collateralHash": "9f81bf833521c3219abbac75dbbcd6682a0ed1a8ff7b2591d18746e645c03877",
  "collateralIndex": 1,
  "dmnState": {
    "service": "18.181.40.131:8168",
    "registeredHeight": 278423,
    "lastPaidHeight": 956110,
    "PoSePenalty": 0,
    "PoSeRevivedHeight": 760251,
    "PoSeBanHeight": -1,
    "revocationReason": 0,
    "ownerAddress": "aMKviyAT96BVQXmwvnkWkV2KEQm2iiYvZD",
    "votingAddress": "aMKviyAT96BVQXmwvnkWkV2KEQm2iiYvZD",
    "payoutAddress": "ZzsNEWUwdkYt3HT67X6izeuF9RxFxBsh6h",
    "pubKeyOperator": "8ed4aa79dbbae0787ca4f4264cb7f1872ff2997836441661e3505f6b6a30b02016f3ba5a32c7d5ce999d073f1cf6534e"
  },
  "state": "READY",
  "status": "Ready"
}

It's definitely doing stuff

2024-09-12 15:54:35 AcceptToMemoryPoolWorker(), tx.IsSpend()=false, fTestNet=false
2024-09-12 15:54:35 CheckTransaction nHeight=2147483647, isVerifyDB=false, isCheckWallet=true, txHash=a42b0f4db8ccc91d893a29b528545f83d22eabe494d67b27e4c5cf1fd910c035
2024-09-12 15:54:35 CheckSparkMintTransaction txHash = a42b0f4db8ccc91d893a29b528545f83d22eabe494d67b27e4c5cf1fd910c035
2024-09-12 15:54:35 AcceptToMemoryPoolWorker -> OK
2024-09-12 15:54:35 Transaction a42b0f4db8ccc91d893a29b528545f83d22eabe494d67b27e4c5cf1fd910c035 received and added to the mempool.
2024-09-12 15:54:35 AcceptToMemoryPool(), transaction: a42b0f4db8ccc91d893a29b528545f83d22eabe494d67b27e4c5cf1fd910c035
2024-09-12 15:54:35 AcceptToMemoryPoolWorker(), tx.IsSpend()=false, fTestNet=false
2024-09-12 15:54:35 CheckTransaction nHeight=2147483647, isVerifyDB=false, isCheckWallet=true, txHash=a42b0f4db8ccc91d893a29b528545f83d22eabe494d67b27e4c5cf1fd910c035
2024-09-12 15:54:35 CheckSparkMintTransaction txHash = a42b0f4db8ccc91d893a29b528545f83d22eabe494d67b27e4c5cf1fd910c035
2024-09-12 15:54:35 AcceptToMemoryPoolWorker -> OK

$ iostat
Linux 5.10.219-208.866.2.aarch64 (ip-172-31-35-192.ap-northeast-1.compute.internal)         09/13/2024      _aarch64_       (2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.22    0.00    0.19    0.53    0.04   98.03

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
nvme0n1           2.33        60.58        17.58   52099149   15120720
nvme1n1           1.54         6.28         6.20    5404280    5334836

Others in the Masternode channel in discord say they don't see anything close to what I'm seeing for memory usage.

This is very expensive to run, so I'd like to find out why and also how I can lower it.

last 500 lines of debug log
500linesdebug.log

[NorseGaud]

Hey @levoncrypto , let me know what you need to help. I can give you access to one of my nodes if you need it.

[levoncrypto]

@NorseGaud I am working on memory load in general, which will help to reduce ram load, I think it will help you. If not, then I will tell you, and we will work with your node individually.