Periodic failures #71
Description
On our large download backup, the backup stops working every 2 months and requires to be restarted. Borg base alerts informs us.
Logs mentions a connection issue but given the number of layers there are, logs are frequently useless.
Configuration:
BORGBASE_NAME=download
BITWARDEN_PASSWORD=*******
BITWARDEN_EMAIL=******
BACKUP_HOUR=2
BACKUP_MINUTE=0
KEEP_WITHIN=0H
KEEP_DAILY=7
KEEP_WEEKLY=5
KEEP_MONTHLY=12
KEEP_YEARLY=1
QUOTA=0
REGION=eu
ALERT=1
PERIODICITY=1d
BACKUP_DAY=1
DATABASES=
MAX_BORGMATIC_RETRY=10
WAIT_BEFORE_BORGMATIC_RETRY=5
Here's the tail of the log
M /storage/openzim/release/libzim/feed.xml
M /storage/openzim/release/feed.xml
Remote: Storage quota: 879.21 GB out of 3.96 TB used.
Remote: Storage quota: 879.21 GB out of 3.96 TB used.
[email protected]:repo: Running consistency checks
Remote: Starting repository check
Remote: packet_write_wait: Connection to 65.21.33.81 port 22: Broken pipe
Connection closed by remote host
[email protected]:repo: Error running actions for repository
Command 'borg check --prefix {hostname}- --info [email protected]:repo' returned non-zero exit status 2.
/root/.config/borgmatic/config.yaml: Error running configuration file
summary:
/root/.config/borgmatic/config.yaml: Error running configuration file
[email protected]:repo: Error running actions for repository
Remote: Starting repository check
Remote: packet_write_wait: Connection to 65.21.33.81 port 22: Broken pipe
Connection closed by remote host
Command 'borg check --prefix {hostname}- --info [email protected]:repo' returned non-zero exit status 2.
Need some help? https://torsion.org/borgmatic/#issues
[email protected]:repo: Pruning archives
Keeping archive: download__backup__2021-07-20T02:00:40 Tue, 2021-07-20 02:00:42 [05e109f47cdabf316764a0b364a3adead675200a8ee482119b85dc63d16e38d9]
Keeping archive: download__backup__2021-07-19T02:00:13 Mon, 2021-07-19 02:00:15 [1949cc1def2dff8d7a82544a3bfb53a1774e1f0f3581174c4805add9e5ee3360]
Keeping archive: download__backup__2021-07-18T02:00:33 Sun, 2021-07-18 02:00:35 [b55f59819bf147a0c6b09a11de9e0e20483754eed30e73072f3564b5e17be8b8]
Keeping archive: download__backup__2021-07-17T02:00:16 Sat, 2021-07-17 02:00:18 [022630cb7db2a35a71afde80d23e62a842db7654d6d353e388c3d384457e220c]
Keeping archive: download__backup__2021-07-16T02:00:17 Fri, 2021-07-16 02:00:19 [9f8675681dd627010dbe72e1d8a822c3a6a7daa299b5893518838a716e6bb901]
Keeping archive: download__backup__2021-07-15T02:00:30 Thu, 2021-07-15 02:00:32 [cf314a498013c72c14166cc10b0e8e19dec5a45f455c2e02f62147b6e21065c6]
Keeping archive: download__backup__2021-07-14T02:00:38 Wed, 2021-07-14 02:00:40 [2eb19a6f256abf34f838b62d0221b78029d2339e055b5381c630cf8d1c1d6019]
Pruning archive: download__backup__2021-07-13T02:00:29 Tue, 2021-07-13 02:00:32 [25418a95bc2fb698119d2fd5084c877d778244a30da703fad7ece94c3b4b0585] (1/1)
Remote: Storage quota: 879.21 GB out of 3.96 TB used.
Keeping archive: download__backup__2021-07-11T02:00:10 Sun, 2021-07-11 02:00:11 [e07405e3f376e2e36097a37573897ad99afd192c0c623bacddc446cc9956f3b0]
Keeping archive: download__backup__2021-07-04T02:00:14 Sun, 2021-07-04 02:00:16 [12dcaaf12fc33eb79fecc3dec2d925eb41e614bea69e4a5e4d087ad42cf04bd4]
Keeping archive: download__backup__2021-06-30T02:00:31 Wed, 2021-06-30 02:00:33 [513c6714e5d7834e989fae4dbca9844b873d04c6f920298394dfdeebb2cf47d8]
Keeping archive: download__backup__2021-06-27T02:00:28 Sun, 2021-06-27 02:00:29 [718e41af436a75f3050fdbdcf5e889520933a1142b7c1149bee6fdfcd1481d5d]
Keeping archive: download__backup__2021-06-20T02:00:15 Sun, 2021-06-20 02:00:16 [47d59aa9cfc9f539214d7e674dc9c658af30f2bae9c1ba2184566fb2e438da06]
Keeping archive: download__backup__2021-06-13T02:00:17 Sun, 2021-06-13 02:00:18 [de186b99a4f1cc4908315d97b081fcbb8027e65b4a27bb890081ff854693693f]
Keeping archive: download__backup__2021-05-31T02:00:19 Mon, 2021-05-31 02:00:20 [8ccbccd160fc7aec5bb98c6ec1bfd3be46b7a434bc828eb9981715a4f62713b9]
Keeping archive: download__backup__2021-04-30T02:00:31 Fri, 2021-04-30 02:00:35 [6269745493856f09aec0503d9998573f015379db6ca6d5bfd8d17cb657590f4b]
Keeping archive: download__backup__2021-03-31T02:00:14 Wed, 2021-03-31 02:00:15 [f7d129c53127055953b848cafc84dfcb7bbf57aaf8ec6ae512844af19e017944]
Keeping archive: download__backup__2021-02-28T02:00:40 Sun, 2021-02-28 02:00:48 [516ad18252ddbffa126adce29de587e79d481ac8f88000d76763bd050c10615d]
Keeping archive: download__backup__2021-01-31T02:00:21 Sun, 2021-01-31 02:00:23 [d792c3fcb404d68fcd9beed1894e515f9212a0d70b1c29d50b3e5d4a019cca17]
Keeping archive: download__backup__2020-12-31T02:03:27 Thu, 2020-12-31 02:04:08 [709e198f3a9f53baed68fb3dcd82f85a6de520f24923ddfd63ed1a5be6d95029]
Remote: Storage quota: 879.21 GB out of 3.96 TB used.
[email protected]:repo: Creating archive
Creating archive at "[email protected]:repo::download__backup__{now}"
M /storage/kiwix/archive/.htaccess
M /storage/kiwix/other/.htaccess
M /storage/kiwix/dev/.htaccess
M /storage/kiwix/release/.htaccess
M /storage/kiwix/release/kiwix-android/2.5.0/feed.xml
M /storage/kiwix/release/kiwix-android/2.5.1/feed.xml
M /storage/kiwix/release/kiwix-android/2.5.2/feed.xml
M /storage/kiwix/release/kiwix-android/2.5.3/feed.xml
M /storage/kiwix/release/kiwix-android/2.3/feed.xml
M /storage/kiwix/release/kiwix-android/feed.xml
M /storage/kiwix/release/browsers/chrome/feed.xml
A /storage/kiwix/release/browsers/firefox/feed.xml
M /storage/kiwix/release/browsers/feed.xml
M /storage/kiwix/release/browsers/edge/feed.xml
M /storage/kiwix/release/kiwix-js-windows/feed.xml
M /storage/kiwix/release/feed.xml
M /storage/kiwix/release/kiwix-desktop-macos/feed.xml
M /storage/kiwix/release/kiwix-desktop/feed.xml
M /storage/kiwix/release/kiwix-hotspot/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v1.0.0/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.0.4/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.0.6/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.0.7/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.0.5/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.4.0/feed.xml
M /storage/kiwix/release/kiwix-hotspot/master/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.4.1/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.4.2/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.0.8/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.1.0/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.1.1/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.0.3/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.0.0/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.0.1/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.0.2/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.3.7/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.3.8/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.3.0/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.2.0/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.2.1/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.3.2/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.3.1/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.3.3/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.3.5/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.3.6/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.4.3/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.4.4/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.4.5/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.4.6/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.3.4/feed.xml
M /storage/kiwix/release/libkiwix/feed.xml
M /storage/kiwix/release/kiwix-tools/feed.xml
M /storage/openzim/release/zimwriterfs/feed.xml
M /storage/openzim/release/zim-tools/feed.xml
M /storage/openzim/release/libzim/feed.xml
M /storage/openzim/release/feed.xml
Remote: Storage quota: 879.21 GB out of 3.96 TB used.
Remote: Storage quota: 879.21 GB out of 3.96 TB used.
[email protected]:repo: Running consistency checks
Remote: Starting repository check
Remote: packet_write_wait: Connection to 65.21.33.81 port 22: Broken pipe
Connection closed by remote host
[email protected]:repo: Error running actions for repository
Command 'borg check --prefix {hostname}- --info [email protected]:repo' returned non-zero exit status 2.
/root/.config/borgmatic/config.yaml: Error running configuration file
summary:
/root/.config/borgmatic/config.yaml: Error running configuration file
[email protected]:repo: Error running actions for repository
Remote: Starting repository check
Remote: packet_write_wait: Connection to 65.21.33.81 port 22: Broken pipe
Connection closed by remote host
Command 'borg check --prefix {hostname}- --info [email protected]:repo' returned non-zero exit status 2.
Need some help? https://torsion.org/borgmatic/#issues
[email protected]:repo: Pruning archives
Remote: write: Connection reset by peer
Connection closed by remote host. Is borg working on the server?
[email protected]:repo: Error running actions for repository
Command 'borg prune --keep-daily 7 --keep-weekly 5 --keep-monthly 12 --keep-yearly 1 --prefix download__backup__ --info --list [email protected]:repo' returned non-zero exit status 2.
/root/.config/borgmatic/config.yaml: Error running configuration file
summary:
/root/.config/borgmatic/config.yaml: Error running configuration file
[email protected]:repo: Error running actions for repository
Remote: write: Connection reset by peer
Connection closed by remote host. Is borg working on the server?
Command 'borg prune --keep-daily 7 --keep-weekly 5 --keep-monthly 12 --keep-yearly 1 --prefix download__backup__ --info --list [email protected]:repo' returned non-zero exit status 2.
Need some help? https://torsion.org/borgmatic/#issues
[email protected]:repo: Pruning archives
Remote: ssh_exchange_identification: read: Connection reset by peer
Connection closed by remote host. Is borg working on the server?
[email protected]:repo: Error running actions for repository
Command 'borg prune --keep-daily 7 --keep-weekly 5 --keep-monthly 12 --keep-yearly 1 --prefix download__backup__ --info --list [email protected]:repo' returned non-zero exit status 2.
/root/.config/borgmatic/config.yaml: Error running configuration file
summary:
/root/.config/borgmatic/config.yaml: Error running configuration file
[email protected]:repo: Error running actions for repository
Remote: ssh_exchange_identification: read: Connection reset by peer
Connection closed by remote host. Is borg working on the server?
Command 'borg prune --keep-daily 7 --keep-weekly 5 --keep-monthly 12 --keep-yearly 1 --prefix download__backup__ --info --list [email protected]:repo' returned non-zero exit status 2.
Need some help? https://torsion.org/borgmatic/#issues