Skip to content

Cross platform packaging#587

Open
JoeZiminski wants to merge 36 commits intomainfrom
cross-platform-packaging
Open

Cross platform packaging#587
JoeZiminski wants to merge 36 commits intomainfrom
cross-platform-packaging

Conversation

@JoeZiminski
Copy link
Member

@JoeZiminski JoeZiminski commented Sep 2, 2025

This PR introduces cross-platform packaging of datashuttle. This requires both packaging datashuttle itself as well as vendoring a terminal emulator to deal with some rendering issues.

The packaging process is threefold:

  1. package datashuttle
  2. package a lightweight script that launches the vendored terminal emulated and runs the packaged datashutlte within it.
  3. wrap in an installer (inno setup on Windows, bundle on macOS, unsure on Linux, we might not wrap in an installer there).

Currently, this PR:

  1. Works on Windows
  2. Packaging works on macOS intel and silicon. However, the silicon version does not work on intel, and vice versa. A little more work should be done to investigate why this is (a single distribution is possible in some cases). If it's not possible, we will have two separate distributions. Still need to wrap in a installer bundle.
  3. Packaging is working on ubuntu, need to look into packaging on manylinux and whether to wrap in an installer.

Currently the architecture is:

  1. datashuttle.spec is shared between operating systems, and is the pyinstaller config file to control packaging of datashuttle itself. It will package datashuttle itself, by packaging the datashuttle_launcher.py script. This will then create a executable to be called in the vendored terminal emulator.
  2. There are three top-level packaging scripts, one for each OS (package_windows.py, package_macos.py, package_linux.py). These coordinate the download of the terminal emulator to vendor, packaging of datashuttle, then triggers the running of terminal_launcher_xxx.spec (one for macos, 'windows' is currently used by Windows and Linux) which in turn packages terminal_launcher.py. This handles how the vendored terminal should call the datashuttle executable.

So essentially, the idea is to package datashuttle, and then package a script that opens the vendored terminal and runs the datashuttle executable within it.

One the packaging is complete, we will need to extend the CI to run these installers, generate the artifact to distribute, and add it to the website.

Note that this PR is still in the prototype phase and requires some refactoring and tidying up.

TODO:
windows, check all raw "rclone" calls and make sure they use the new "get_command". Need to add this new convention to dev docs
It might be possible to use the cmd key on Wezterm!
config.send_composed_key_when_left_alt_is_pressed = true
config.macos_forward_to_ime_modifier_mask = "SHIFT|CTRL"

@JoeZiminski JoeZiminski force-pushed the cross-platform-packaging branch from 8e52ff3 to 6685bf7 Compare September 2, 2025 20:21
@JoeZiminski JoeZiminski force-pushed the cross-platform-packaging branch from 79049d6 to 5b6a91c Compare September 2, 2025 20:25
@JoeZiminski JoeZiminski force-pushed the cross-platform-packaging branch from 5660e9e to a941e13 Compare September 2, 2025 20:28
@JoeZiminski JoeZiminski force-pushed the cross-platform-packaging branch from c5b5c6a to 9c95587 Compare September 2, 2025 20:43
@JoeZiminski JoeZiminski force-pushed the cross-platform-packaging branch from 00cf5ae to acdaada Compare September 2, 2025 20:52
@JoeZiminski JoeZiminski force-pushed the cross-platform-packaging branch from 554cd8a to 5f06120 Compare February 24, 2026 00:34
Comment on lines +15 to +59
runs-on: windows-latest

defaults:
run:
shell: powershell

steps:
- uses: actions/checkout@v4
- name: Set up Conda
uses: conda-incubator/setup-miniconda@v3
with:
python-version: ${{ matrix.python-version }}
auto-update-conda: true
channels: conda-forge
activate-environment: "datashuttle-test"

- name: Install rclone
run: |
conda activate datashuttle-test
conda install -c conda-forge rclone

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install .[dev]
pip install pyinstaller

# Install Inno Setup silently
- name: Install Inno Setup
run: |
choco install innosetup --yes

- name: Verify Inno Setup installation
run: |
Get-Command "C:\Program Files (x86)\Inno Setup 6\ISCC.exe"

- name: Run Windows packaging script
run: |
python package/package_windows.py

- name: Upload installer artifact
uses: actions/upload-artifact@v4
with:
name: windows-installer
path: package\Output\datashuttle_0.0.0.exe

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}

Copilot Autofix

AI 7 days ago

In general, to fix this class of problem you need to define a permissions block that explicitly restricts the GITHUB_TOKEN to the least privileges the workflow actually needs. You can add this either at the top (root) of the workflow to apply to all jobs, or under a specific job to limit only that job.

For this particular workflow, the build-windows job only checks out code and uploads an artifact; it doesn’t push code, modify releases, or interact with issues/PRs. These operations work with contents: read, and actions/upload-artifact does not require any repository write permissions. The best minimal fix is therefore to add a root-level permissions block just after the name: (or after on:) with contents: read. This documents the intent and ensures the token cannot be used to write to the repo even if organization defaults are broader.

Concretely:

  • Edit .github/workflows/package_windows.yml.
  • Insert a root-level permissions: mapping near the top of the file (e.g., after line 2 or after the on: block) with contents: read.
  • No other steps or actions in the shown workflow require additional scopes, so no further permissions are necessary.
Suggested changeset 1
.github/workflows/package_windows.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/package_windows.yml b/.github/workflows/package_windows.yml
--- a/.github/workflows/package_windows.yml
+++ b/.github/workflows/package_windows.yml
@@ -1,5 +1,8 @@
 name: windows-build
 
+permissions:
+  contents: read
+
 on:
   push:
     branches: [ main ]
EOF
@@ -1,5 +1,8 @@
name: windows-build

permissions:
contents: read

on:
push:
branches: [ main ]
Copilot is powered by AI and may make mistakes. Always verify output.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants