Merge pull request #209 from semuconsulting/RELEASE-CANDIDATE-1.5.14

Add spatial database recording facility

Author: semuadmin

INSTALLATION.md

@@ -26,7 +26,7 @@ Though by no means mandatory, it is [generally considered best practice](https:/
 1. Use different versions of the same package across applications.
 1. Keep your application dependencies clean and organized.
 
-Some platforms (e.g. Ubuntu Linux) **enforce** the use of virtual environments via a so-called 'Externally Managed Environment' constraint². Attempting to install a package into the global environment will result in an `error: externally-managed-environment` error.
+Some platforms (e.g. Ubuntu Linux and Homebrew-installed Python environments) **enforce** the use of virtual environments via a so-called 'Externally Managed Environment' constraint². Attempting to install a package into the global environment will result in an `error: externally-managed-environment` error.
 
 ¹ In practice, 'global' generally means the user's home environment. Installing into the platform's global system environment typically results in a `Defaulting to user installation because normal site-packages is not writeable` warning.
 
@@ -61,23 +61,28 @@ The exact location of the site_packages and binary directories will depend on th
 
 In the following, `python3` & `pip` refer to the Python 3 executables. You may need to substitute `python` for `python3`, depending on your particular environment (*on Windows it's generally `python`*). 
 
-- Python >= 3.9
-- Tk (tkinter) >= 8.6⁴ (*tkinter is a commonly used library for developing Graphical User Interfaces (GUI) in Python*)
+- Python >= 3.9⁴
+- Tk (tkinter) >= 8.6⁵ (*tkinter is a commonly used library for developing Graphical User Interfaces (GUI) in Python*)
 - Screen resolution >= 640 x 400; Ideally 1920 x 1080, though at lower screen resolutions (<= 1024 width), top level dialogs will be resizable and scrollable.
 
 **NB** It is highly recommended to use the latest official [Python.org](https://www.python.org/downloads/) installation package for your platform, rather than any pre-installed version.
 
 **NB** It is highly recommended that the Python 3 [binaries](#binaries) (`../bin` or `..\Scripts`) directory is included in your PATH (*most standard Python 3 installation packages will do this automatically if you select the 'Add to PATH' option during installation*).
 
+⁴ PyGPSClient's optional spatialite database recording facility is dependent upon the following:
+
+   - The Python environment must support the loading of sqlite3 extensions i.e. it must have been compiled with the `--enable-loadable-sqlite-extensions` option. This is true by default for most Windows and Linux platforms but **NOT** for most MacOS platforms. 
+   - The mod_spatialite module (.so, .dll or .dylib) must be installed and in the `PATH` (or `LD_LIBRARY_PATH` on Linux).
+
 ### Windows 10 or later
 
 Normally installs without any additional steps.
 
 ### MacOS 13 or later
 
-⁴ The version of Python supplied with some older Apple MacOS platforms includes a [deprecated version of tkinter](https://www.python.org/download/mac/tcltk/) (8.5). Use an official [Python.org](https://www.python.org/downloads/macos) installation package instead. 
+⁵ The version of Python supplied with some older Apple MacOS platforms includes a [deprecated version of tkinter](https://www.python.org/download/mac/tcltk/) (8.5). Use an official [Python.org](https://www.python.org/downloads/macos) installation package instead. 
 
-**NB:** Python does ***NOT*** require Homebrew or MacPorts to be installed on MacOS. The Python organisation provides serviceable [64-bit universal installation packages](https://www.python.org/downloads/macos/) for all current and legacy versions of Python, including release candidates. 
+**NB:** Python does ***NOT*** normally require Homebrew or MacPorts to be installed on MacOS. The Python organisation provides serviceable [64-bit universal installation packages](https://www.python.org/downloads/macos/) for all current and legacy versions of Python, including release candidates. 
 
 However, if you wish to install Python using [Homebrew](https://brew.sh/) to take advantage of certain non-default configurations (*e.g. support for sqlite3 extensions*), use the `python-tk` formula rather than `python`, e.g. 
 
@@ -89,12 +94,14 @@ Note also that the Homebrew formulae for python-tk>=3.12 include the latest tkin
 
 ### Linux (including Raspberry Pi OS)
 
-Some Linux distributions may not include the necessary pip, tkinter or Pillow imaging libraries by default. They may need to be installed separately, e.g.:
+Some Linux distributions may not include the necessary pip, tkinter, Pillow or spatialite libraries by default. They may need to be installed separately, e.g.:
 
 ```shell
-sudo apt install python3-pip python3-tk python3-pil python3-pil.imagetk libjpeg-dev zlib1g-dev tk-dev
+sudo apt install python3-pip python3-tk python3-pil python3-pil.imagetk libjpeg-dev zlib1g-dev tk-dev libspatialite
 ```
 
+⁴ Support for the sqlite3 `mod_spatialite` extension may require a custom version of Python to be [compiled from source](https://github.com/semuconsulting/PyGPSClient/blob/master/examples/python_compile.sh) if a suitable version is not available from any of the distribution's repos.
+
 ## <a name="userpriv">User Privileges</a>
 
 To access the serial port on most Linux platforms, you will need to be a member of the 

LICENSE

@@ -1,6 +1,6 @@
 BSD 3-Clause License ("BSD License 2.0", "Revised BSD License", "New BSD License", or "Modified BSD License")
 
-Copyright (c) 2020, SEMU Consulting
+Copyright (c) 2020, semuadmin (Steve Smith)
 All rights reserved.
 
 Redistribution and use in source and binary forms, with or without

README.md

@@ -104,8 +104,9 @@ Please refer to [INSTALLATION.md](https://github.com/semuconsulting/PyGPSClient/
 1. Tags - enable color tags in console (see Console Widget below).
 1. Position Format and Units - Change the displayed position (D.DD / D.M.S / D.M.MM / ECEF) and unit (metric/imperial) formats.
 1. Show Unused Satellites - Include or exclude satellites that are not used in the navigation solution (e.g. because their signal level is too low) from the graph and sky view panels.
-1. DataLogging - Turn Data logging in the selected format on or off. You will be prompted to select the directory into which timestamped log files are saved.
-1. GPX Track - Turn track recording (in GPX format) on or off. You will be prompted to select the directory into which timestamped GPX track files are saved.
+1. DataLogging - Turn Data logging in the selected format on or off. On first selection, you will be prompted to select the directory into which timestamped log files are saved.
+1. GPX Track - Turn track recording (in GPX format) on or off. On first selection, you will be prompted to select the directory into which timestamped GPX track files are saved.
+1. Database - Turn spatialite database recording (*where available*) on or off. On first selection, you will be prompted to select the directory into which the `pygpsclient.sqlite` database is saved. Note that, when first created, the database's spatial metadata will take a few seconds to initialise (*up to a minute on Raspberry Pi and similar SBC*). **NB** This facility is dependent on your Python environment supporting the requisite [sqlite3 `mod_spatialite` extension](https://www.gaia-gis.it/fossil/libspatialite/index) - see [INSTALLATION.md](https://github.com/semuconsulting/PyGPSClient/blob/master/INSTALLATION.md#prereqs) for further details. If not supported, the option will be greyed out. Check the Menu..Help..About dialog for an indication of the current spatialite support status.
 1. To save the current configuration to a file, go to File..Save Configuration.
 1. To load a saved configuration file, go to File..Load Configuration. The default configuration file location is `$HOME/pygpsclient.json`. **NB** Any active serial or RTK connection must be stopped before loading a new configuration.
 
@@ -389,7 +390,8 @@ By default, the server/caster binds to the host address '0.0.0.0' (IPv4) or '::'
 
 *GPX Track Viewer screenshot*
 
-The GPX Track Viewer can display any valid GPX file containing trackpoints (`<trkpt>..</trkpt>` elements) against either an ["custom" offline map image](#custommap), or an online MapQuest "map" or "sat" view. The "map" and "sat" options require a free [MapQuest API key](#mapquestapi). The Y axis scales will reflect the current choice of units (metric or imperial). Click ![refresh icon](https://github.com/semuconsulting/PyGPSClient/blob/master/src/pygpsclient/resources/iconmonstr-refresh-lined-24.png?raw=true) to refresh the display after any changes (e.g. resizing, zooming or change of units). The location marker indicates the nominal center point of the track.
+The GPX Track Viewer can display any valid GPX file containing track point (`trkpt`), route point (`rtept`) or waypoint (`wpt`)elements against either an ["custom" offline map image](#custommap), or an online MapQuest "map", "sat" or "hyb" view. The "map", "sat" and "hyb" options require a free [MapQuest API key](#mapquestapi). The Y axis scales will reflect the current choice of units (metric or imperial). If the GPX track omits a time element, the time and speed axes will be flagged as nominal. GPX track metadata, including min, max, average (mean) and median elevation and speed values, is displayed in the selected units. 
+Click ![refresh icon](https://github.com/semuconsulting/PyGPSClient/blob/master/src/pygpsclient/resources/iconmonstr-refresh-lined-24.png?raw=true) to refresh the display after any changes (e.g. resizing, zooming or change of units). The location marker indicates the nominal center point of the track.
 
 ---
 ## <a name="mapquestapi">MapQuest API Key</a>

RELEASE_NOTES.md

@@ -1,5 +1,35 @@
 # PyGPSClient Release Notes
 
+### RELEASE 1.5.14
+
+FIXES:
+
+1. Fix Chart update issue #210
+2. Fix Exit button not invoking on_exit() clean up routines (causing any in-progress GPX track recording to be unterminated).
+3. Fix `IndexError` when loading GPX tracks with no `<time></time>` elements - a synthetic timestamp sequence wil be used instead.
+4. Fix incorrect type formatting for some NMEA commands in NMEA Dynamic Configuration panel.
+
+ENHANCEMENTS:
+
+1. Add facility to write gnss status data to a spatialite (sqlite3 with spatial extension) database, which can be utilised by many standard GNSS visualisation and analysis applications e.g. QGIS, GDAL, GRASS, GeoPandas etc. The facility may be enabled via a new 'Database' checkbox on the Settings panel. The default database path is `$HOME/pygpsclient.sqlite`. A single table `pygpsclient` is populated with all the information displayed in PyGPSClient's banner panel, with lat/lon/hmsl available as a POINTZ (3D) geometry. Note that, when first created, the database spatial metadata will take a few seconds to initialise.
+
+   **NB**: This optional facility is subject to the following Python environmental criteria:
+
+   - The Python environment must support the loading of sqlite3 extensions i.e. it must have been compiled with the `--enable-loadable-sqlite-extensions` option. This is true by default for most Windows and Linux platforms but **NOT** for most MacOS platforms. 
+   - The mod_spatialite module (.so, .dll or .dylib) must be installed and in `PATH` (or `LD_LIBRARY_PATH` on Linux).
+   - The 'About' dialog displays a new Spatial version/status - 'No ext' signifies the platform's Python does not support sqlite3 extensions; 'No m_s' signifies Python *does* support extensions but mod_spatialite is not installed or cannot be found in `PATH` or `LD_LIBRARY_PATH`. 
+   
+   On MacOS platforms, it *may* be necessary to install (*e.g. via Homebrew*) a custom version of Python with the `--enable-loadable-sqlite-extensions` flag set and the `libspatialite` package installed e.g.
+
+   `brew install [email protected] libspatialite`
+
+   On Linux platforms which do not support sqlite3 extensions out of the box, it may be possible to compile from source a  suitable version of [Python](https://github.com/semuconsulting/PyGPSClient/blob/master/examples/python_compile.sh) and/or [libspatialite](https://github.com/semuconsulting/PyGPSClient/blob/master/examples/libspatialite_compile.sh).
+2. BSD 3-Clause license attribution clarified in all modules.
+3. Enhance elevation profile and metadata rendering in GPX track viewer. Add support for route and waypoint elements in addition to track.
+4. Enhance NMEA Dynamic Configuration dialog to support additional NMEA command types, including Quectel proprietary $PAIR  (Quectel LC29H/LC79H), Garmin $PGRM, Locosys %PINV (limited) and u-Blox $PUBX (requires pynmeagps>=1.0.53).
+5. Add support 
+5. Various minor improvements to file exception handling.
+
 ### RELEASE 1.5.13
 
 ENHANCEMENTS:

docs/pygpsclient.rst

@@ -332,6 +332,14 @@ pygpsclient.spectrum\_frame module
    :show-inheritance:
    :undoc-members:
 
+pygpsclient.sqllite\_handler module
+-----------------------------------
+
+.. automodule:: pygpsclient.sqllite_handler
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
 pygpsclient.status\_frame module
 --------------------------------
 

examples/QGIS_sqlite_gpkg_illustration.png

@@ -0,0 +1,59 @@
+#!/bin/bash
+
+# bash shell script to compile and install libspatialite
+# (mod_spatialite.so) on Debian Linux platforms which do not
+# include a suitable binary in their standard repos
+# (e.g. Raspberry Pi OS).
+#
+# Remember to run chmod +x libspatialite_compile.sh to make this script executable.
+#
+# Source code:
+# https://www.gaia-gis.it/gaia-sins/libspatialite-sources/libspatialite-5.1.0.tar.gz
+#
+# Created by semuadmin on 27 Sep 2023.
+# 
+# exit on error
+set -e
+
+# install build dependencies if required and available (we'll omit the freexl dependency)
+sudo apt install libspatial-dev || true
+sudo apt install libproj-dev || true
+sudo apt install libgdal-dev || true
+sudo apt install libgeos-dev || true
+# download and unzip source code
+wget https://www.gaia-gis.it/gaia-sins/libspatialite-sources/libspatialite-5.1.0.tar.gz
+tar zxvf libspatialite-5.1.0.tar.gz
+cd libspatialite-5.1.0
+# compile source and install - this will take several minutes
+# aarch64-unknown-linux-gnu is appropriate for ARM-based SBC platforms like RPi
+./configure --build=aarch64-unknown-linux-gnu --enable-freexl=no
+make
+sudo make install-strip
+
+# typically installs mod_spatialite.so to /usr/local/lib, which
+# must be added to the LD_LIBRARY_PATH environment variable:
+#
+export LD_LIBRARY_PATH=/usr/local/lib
+
+# make this permanent via your .bashrc or .zshrc shell profile
+echo "Adding directory to PATH..."
+BASHPROF1=$HOME/.profile
+BASHPROF2=$HOME/.bashrc
+ZSHPROF1=$HOME/.zprofile
+ZSHPROF2=$HOME/.zshrc
+if test -f $BASHPROF1
+then
+PROF=$BASHPROF1
+elif test -f $BASHPROF2
+then
+PROF=$BASHPROF2
+fi
+if test -f $ZSHPROF1
+then
+PROF=$ZSHPROF1
+elif test -f $ZSHPROF2
+then
+PROF=$ZSHPROF2
+fi
+sed -i '$a# Path to mod_spatialite.so\nexport LD_LIBRARY_PATH=/usr/local/lib' $PROF
+source $PROF # this will throw an error if running as bash script in zsh shell

examples/libspatialite_compile.sh

@@ -37,18 +37,21 @@ python3 -m pip install --upgrade pip pygpsclient
 deactivate
 
 echo "Adding desktop launch icon..."
+# the LD_LIBRARY_PATH env setting allows PyGPSClient to find the 
+# mod_spatialite.so module, which is typically installed in this location
+# you can set other env variables in a similar fashion if required
 cat > $HOME/.local/share/applications/pygpsclient.desktop <<EOF
 [Desktop Entry]
 Type=Application
 Terminal=false
 Name=PyGPSClient
 Icon=$HOME/pygpsclient/lib/python$PYVER/site-packages/pygpsclient/resources/pygpsclient.ico
-Exec=$HOME/pygpsclient/bin/pygpsclient
+Exec=env LD_LIBRARY_PATH=/usr/local/lib $HOME/pygpsclient/bin/pygpsclient
 EOF
 
 echo "Adding directory to PATH..."
 BASHPROF1=$HOME/.profile
-BASHPROF2=$HOME/.bash_rc
+BASHPROF2=$HOME/.bashrc
 ZSHPROF1=$HOME/.zprofile
 ZSHPROF2=$HOME/.zshrc
 if test -f $BASHPROF1

examples/pygpsclient_debian_install.sh

@@ -19,7 +19,7 @@
 set -e
 
 # set required Python major and minor version e.g. 3.10.10
-PYVER="3.13.0"
+PYVER="3.13.7"
 # NB: uncomment this line to install this version alongside existing versions
 # ALTINSTALL=1
 
@@ -46,13 +46,21 @@ fi
 sudo apt update
 sudo apt build-dep python3
 sudo apt install build-essential gdb lcov pkg-config \
-      libbz2-dev libffi-dev libgdbm-dev libgdbm-compat-dev liblzma-dev \
+      libbz2-dev libffi-dev libgdbm-dev libgdbm-compat-dev \
       libncurses5-dev libreadline6-dev libsqlite3-dev libssl-dev \
-      lzma lzma-dev tk-dev uuid-dev zlib1g-dev -y
+      lzma tk-dev uuid-dev zlib1g-dev -y
+sudo apt install liblzma-dev || true
+# liblzma-dev may be lzma-dev on some platforms
+sudo apt install lzma-dev || true
+sudo apt install libspatialite || true
+# libspatialite may not be available as standard on some platforms
+# (e.g. Rasperry PI OS) but it is relatively straightforward to
+# compile from source using the libspatialite_compile.sh script
+# in the /examples folder
 
-# compile, install and verify installed version
+# compile source and install - this will take several minutes
 cd Python-${PYVER}
-./configure --enable-optimizations
+./configure --enable-optimizations --enable-loadable-sqlite-extensions
 make
 # make test
 if [ -z ${ALTINSTALL+x} ]