Caught out by apt-daily-upgrade.timer

[james-onpoint]

Hi all, got caught out by Ubuntu's apt-daily-upgrade.timer this morning at around 6.30am 6hrs in to a planet import. It restarted Postgres and killed everything.

Definitely one to watch out for during import, hope this helps somebody :)

2025-09-17 00:18:28: Creating database
2025-09-17 00:18:28: Setting up country tables
2025-09-17 00:18:29: Importing OSM data file
2025-09-17 00:18:29  osm2pgsql version 1.11.0
2025-09-17 00:18:29  Database version: 16.10 (Ubuntu 16.10-0ubuntu0.24.04.1)
2025-09-17 00:18:29  PostGIS version: 3.4
2025-09-17 00:18:29  Storing properties to table '"public"."osm2pgsql_properties"'.
2025-09-17 04:58:20  Reading input files done in 16791s (4h 39m 51s).
2025-09-17 04:58:20    Processed 10129800282 nodes in 2846s (47m 26s) - 3559k/s
2025-09-17 04:58:20    Processed 1129231597 ways in 9647s (2h 40m 47s) - 117k/s
2025-09-17 04:58:20    Processed 13650587 relations in 4298s (1h 11m 38s) - 3k/s
2025-09-17 04:58:20  No marked ways (Skipping stage 2).
2025-09-17 04:58:20  Clustering table 'place' by geometry...
2025-09-17 05:58:08  No indexes to create on table 'place'.
2025-09-17 05:58:08  Creating id index on table 'place'...
2025-09-17 06:05:58  Analyzing table 'place'...
2025-09-17 06:06:00  Done postprocessing on table 'planet_osm_nodes' in 0s
2025-09-17 06:06:00  Building index on table 'planet_osm_ways'
2025-09-17 06:23:35  ERROR: Database error: FATAL:  terminating connection due to administrator command
CONTEXT:  SQL function "planet_osm_index_bucket" statement 1
server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
Traceback (most recent call last):
  File "/srv/nominatim/nominatim-venv/bin/nominatim", line 5, in <module>
    exit(cli.nominatim())
         ^^^^^^^^^^^^^^^
  File "/srv/nominatim/nominatim-venv/lib/python3.12/site-packages/nominatim_db/cli.py", line 244, in nominatim
    return get_set_parser().run(cli_args=cli_args, environ=environ)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/nominatim/nominatim-venv/lib/python3.12/site-packages/nominatim_db/cli.py", line 116, in run
    return args.command.run(args)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/nominatim/nominatim-venv/lib/python3.12/site-packages/nominatim_db/clicmd/setup.py", line 82, in run
    return asyncio.run(self.async_run(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/srv/nominatim/nominatim-venv/lib/python3.12/site-packages/nominatim_db/clicmd/setup.py", line 100, in async_run
    self._base_import(args)
  File "/srv/nominatim/nominatim-venv/lib/python3.12/site-packages/nominatim_db/clicmd/setup.py", line 162, in baseimport
    database_import.import_osm_data(files,
  File "/srv/nominatim/nominatim-venv/lib/python3.12/site-packages/nominatim_db/tools/database_import.py", line 133, in import_osm_data
    run_osm2pgsql(options)
  File "/srv/nominatim/nominatim-venv/lib/python3.12/site-packages/nominatim_db/tools/exec_utils.py", line 72, in run_osm2pgsql
    subprocess.run(cmd, cwd=options.get('cwd', '.'),
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/osm2pgsql', '--create', '--slim', '--log-progress', 'true', '--number-processes', '1', '--cache', '0', '--style', '/srv/nominatim/nominatim-venv/lib/python3.12/site-packages/nominatim_db/resources/lib-lua/import-full.lua', '--output', 'flex', '--flat-nodes', '/srv/nominatim/flatnode/flatnode.file', 'planet-latest.osm.pbf']' returned non-zero exit status 1.
(nominatim-venv) nominatim@ip-10-0-137-241:~/nominatim-project$

ubuntu@ip-10-0-137-241:~$ sudo cat /var/log/unattended-upgrades/unattended-upgrades.log | grep "2025-09-17"
2025-09-17 06:22:57,669 INFO Starting unattended upgrades script
2025-09-17 06:22:57,669 INFO Allowed origins are: o=Ubuntu,a=noble, o=Ubuntu,a=noble-security, o=UbuntuESMApps,a=noble-apps-security, o=UbuntuESM,a=noble-infra-security
2025-09-17 06:22:57,669 INFO Initial blacklist:
2025-09-17 06:22:57,669 INFO Initial whitelist (not strict):
2025-09-17 06:23:05,487 INFO Packages that will be upgraded: libsqlite3-0 libudisks2-0 libxml2 linux-aws linux-headers-aws linux-image-aws udisks2 vim vim-common vim-runtime vim-tiny xxd
2025-09-17 06:23:05,487 INFO Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
2025-09-17 06:23:43,965 INFO All upgrades installed
2025-09-17 07:59:48,235 INFO Starting unattended upgrades script
2025-09-17 07:59:48,237 INFO Allowed origins are: o=Ubuntu,a=noble, o=Ubuntu,a=noble-security, o=UbuntuESMApps,a=noble-apps-security, o=UbuntuESM,a=noble-infra-security
2025-09-17 07:59:48,237 INFO Initial blacklist:
2025-09-17 07:59:48,237 INFO Initial whitelist (not strict):

[lonvia]

The restarts of Postgres on automatic updates are a constant source of frustration for me. You can exclude postgres from automatically updating by adding the following line to /etc/apt/apt.conf.d/50unattended-upgrades:

Unattended-Upgrade::Package-Blacklist {
    "postgresql-";
}

[james-onpoint]

Thank you for the insight :) I think my catastrophic restart was a ripple effect from AWS-specific kernel updates. I've temporarily disabled all of the unattended upgrade processes now during the import