Licensing concerns

[fingolfin]

The README instructs the user to run an opaque self-extracting binary.

But where do these come from? Is there documentation somewhere explaining how these distributedworkflow_XYZ archives are created?

I downloaded one and extracted it in a VM, and found these files:

$ ls -R .
.:
build_cache  install.sh  repo

./build_cache:
_pgp                                                                                                                   linux-debian11-x86_64_v3-gcc-10.2.1-hwloc-2.9.1-5j72rx2brd76nwuh66l4h626hp33ua2m.spec.json.sig
index.json                                                                                                             linux-debian11-x86_64_v3-gcc-10.2.1-libiconv-1.17-bhtfz6wxygkwewu7a5cbva3hphn5sasd.spec.json.sig
index.json.hash                                                                                                        linux-debian11-x86_64_v3-gcc-10.2.1-libpciaccess-0.17-ra663wbjkcuzkly7o7glhad3x4amcbn4.spec.json.sig
linux-debian11-x86_64_v3                                                                                               linux-debian11-x86_64_v3-gcc-10.2.1-libsigsegv-2.14-2xfhfjuaxgfg52njswafporqecwc5e7f.spec.json.sig
linux-debian11-x86_64_v3-gcc-10.2.1-berkeley-db-18.1.40-jxqezua3vnzviwdde22dnnn4ftguo5kt.spec.json.sig                 linux-debian11-x86_64_v3-gcc-10.2.1-libssh2-1.11.0-yaq6foqrxvuyhdnpkyz2cqog5ejulx6j.spec.json.sig
linux-debian11-x86_64_v3-gcc-10.2.1-boost-1.63.0-eelwyai253cp3ncrgfjsknky3sbtfaqi.spec.json.sig                        linux-debian11-x86_64_v3-gcc-10.2.1-libtool-2.4.7-qkcficq53dqlhoucbupl7llapl3ytdtw.spec.json.sig
linux-debian11-x86_64_v3-gcc-10.2.1-bzip2-1.0.8-zfuxwyvajbnk5harbclgxu3kkhpsj4vu.spec.json.sig                         linux-debian11-x86_64_v3-gcc-10.2.1-m4-1.4.19-uywihodgut7dbu6du3gkejqtdjl4esmm.spec.json.sig
linux-debian11-x86_64_v3-gcc-10.2.1-ca-certificates-mozilla-2023-05-30-ydyz5rtaeezdd7rqol6ahvzubeaydr2y.spec.json.sig  linux-debian11-x86_64_v3-gcc-10.2.1-ncurses-6.4-6yyq7ujubwm3kyn33cqfdbzwc7altrt4.spec.json.sig
linux-debian11-x86_64_v3-gcc-10.2.1-chrpath-0.16-4vztv6mkxr6phjlp5poy4jkqbp6iqilu.spec.json.sig                        linux-debian11-x86_64_v3-gcc-10.2.1-nghttp2-1.57.0-b2lg2ibbfzu6gi3u6klnmnfgaoqhbkxc.spec.json.sig
linux-debian11-x86_64_v3-gcc-10.2.1-cmake-3.27.7-z4w3zw2jo6mgpenvvbj7pptunn7ihzsm.spec.json.sig                        linux-debian11-x86_64_v3-gcc-10.2.1-openssl-3.1.3-mprhptd52rqhkszenfgcsosou735mklj.spec.json.sig
linux-debian11-x86_64_v3-gcc-10.2.1-curl-8.4.0-d65tmfpfhfn547bktaksgvxgguzsggxo.spec.json.sig                          linux-debian11-x86_64_v3-gcc-10.2.1-perl-5.38.0-hhrdw2sgs4u63ig7ljid3tvxajwflayb.spec.json.sig
linux-debian11-x86_64_v3-gcc-10.2.1-diffutils-3.9-5xnv67gkj3nz4cal66xqbh65e7rdadhu.spec.json.sig                       linux-debian11-x86_64_v3-gcc-10.2.1-pkgconf-1.9.5-lgtucd6ctfihjpvpinkfhc4gnjkg5r3p.spec.json.sig
linux-debian11-x86_64_v3-gcc-10.2.1-distributedworkflow-latest-loymavzqg7jkyzyu46qul435e6skmpdi.spec.json.sig          linux-debian11-x86_64_v3-gcc-10.2.1-readline-8.2-xx6bjh7tgiwx4weeqaopp4hby6g6vjaw.spec.json.sig
linux-debian11-x86_64_v3-gcc-10.2.1-executor-latest-g4fbel2s6lns2i6ro4i3qjg7sboz5kye.spec.json.sig                     linux-debian11-x86_64_v3-gcc-10.2.1-util-macros-1.19.3-jl67jcaenejiqtajjzkzz2lqqk6b6pmb.spec.json.sig
linux-debian11-x86_64_v3-gcc-10.2.1-gdbm-1.23-bqerekzvawxbxerm3vtbpmy6w3jttecg.spec.json.sig                           linux-debian11-x86_64_v3-gcc-10.2.1-xz-5.4.1-hxiintmu5noxrkqvz5stdvmazatxm7mr.spec.json.sig
linux-debian11-x86_64_v3-gcc-10.2.1-gmake-4.4.1-gq64uicu5a4qzdcumnxgvk3k3o2kyxch.spec.json.sig                         linux-debian11-x86_64_v3-gcc-10.2.1-zlib-ng-2.1.4-vfknuvdtychagyxjj6mt634kkadajlmc.spec.json.sig
linux-debian11-x86_64_v3-gcc-10.2.1-gpi-space-23.06-wupua3kzmcyjvso77jqfaswlbi733i6e.spec.json.sig                     linux-debian11-x86_64_v3-gcc-10.2.1-zstd-1.5.5-xoh7ab7tjny3motqbj67qnzargapj6pm.spec.json.sig
linux-debian11-x86_64_v3-gcc-10.2.1-gspcdriver-latest-omqrjpvqwubbqln7nshb5drj54fvab2h.spec.json.sig

./build_cache/_pgp:
346575B1E8570D3A8A13DB3F8195390F0870FE68.pub  index.json

./build_cache/linux-debian11-x86_64_v3:
gcc-10.2.1

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1:
berkeley-db-18.1.40  ca-certificates-mozilla-2023-05-30  curl-8.4.0                  executor-latest  gpi-space-23.06    libiconv-1.17      libssh2-1.11.0  ncurses-6.4     perl-5.38.0    util-macros-1.19.3  zstd-1.5.5
boost-1.63.0         chrpath-0.16                        diffutils-3.9               gdbm-1.23        gspcdriver-latest  libpciaccess-0.17  libtool-2.4.7   nghttp2-1.57.0  pkgconf-1.9.5  xz-5.4.1
bzip2-1.0.8          cmake-3.27.7                        distributedworkflow-latest  gmake-4.4.1      hwloc-2.9.1        libsigsegv-2.14    m4-1.4.19       openssl-3.1.3   readline-8.2   zlib-ng-2.1.4

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/berkeley-db-18.1.40:
linux-debian11-x86_64_v3-gcc-10.2.1-berkeley-db-18.1.40-jxqezua3vnzviwdde22dnnn4ftguo5kt.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/boost-1.63.0:
linux-debian11-x86_64_v3-gcc-10.2.1-boost-1.63.0-eelwyai253cp3ncrgfjsknky3sbtfaqi.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/bzip2-1.0.8:
linux-debian11-x86_64_v3-gcc-10.2.1-bzip2-1.0.8-zfuxwyvajbnk5harbclgxu3kkhpsj4vu.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/ca-certificates-mozilla-2023-05-30:
linux-debian11-x86_64_v3-gcc-10.2.1-ca-certificates-mozilla-2023-05-30-ydyz5rtaeezdd7rqol6ahvzubeaydr2y.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/chrpath-0.16:
linux-debian11-x86_64_v3-gcc-10.2.1-chrpath-0.16-4vztv6mkxr6phjlp5poy4jkqbp6iqilu.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/cmake-3.27.7:
linux-debian11-x86_64_v3-gcc-10.2.1-cmake-3.27.7-z4w3zw2jo6mgpenvvbj7pptunn7ihzsm.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/curl-8.4.0:
linux-debian11-x86_64_v3-gcc-10.2.1-curl-8.4.0-d65tmfpfhfn547bktaksgvxgguzsggxo.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/diffutils-3.9:
linux-debian11-x86_64_v3-gcc-10.2.1-diffutils-3.9-5xnv67gkj3nz4cal66xqbh65e7rdadhu.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/distributedworkflow-latest:
linux-debian11-x86_64_v3-gcc-10.2.1-distributedworkflow-latest-loymavzqg7jkyzyu46qul435e6skmpdi.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/executor-latest:
linux-debian11-x86_64_v3-gcc-10.2.1-executor-latest-g4fbel2s6lns2i6ro4i3qjg7sboz5kye.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/gdbm-1.23:
linux-debian11-x86_64_v3-gcc-10.2.1-gdbm-1.23-bqerekzvawxbxerm3vtbpmy6w3jttecg.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/gmake-4.4.1:
linux-debian11-x86_64_v3-gcc-10.2.1-gmake-4.4.1-gq64uicu5a4qzdcumnxgvk3k3o2kyxch.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/gpi-space-23.06:
linux-debian11-x86_64_v3-gcc-10.2.1-gpi-space-23.06-wupua3kzmcyjvso77jqfaswlbi733i6e.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/gspcdriver-latest:
linux-debian11-x86_64_v3-gcc-10.2.1-gspcdriver-latest-omqrjpvqwubbqln7nshb5drj54fvab2h.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/hwloc-2.9.1:
linux-debian11-x86_64_v3-gcc-10.2.1-hwloc-2.9.1-5j72rx2brd76nwuh66l4h626hp33ua2m.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/libiconv-1.17:
linux-debian11-x86_64_v3-gcc-10.2.1-libiconv-1.17-bhtfz6wxygkwewu7a5cbva3hphn5sasd.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/libpciaccess-0.17:
linux-debian11-x86_64_v3-gcc-10.2.1-libpciaccess-0.17-ra663wbjkcuzkly7o7glhad3x4amcbn4.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/libsigsegv-2.14:
linux-debian11-x86_64_v3-gcc-10.2.1-libsigsegv-2.14-2xfhfjuaxgfg52njswafporqecwc5e7f.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/libssh2-1.11.0:
linux-debian11-x86_64_v3-gcc-10.2.1-libssh2-1.11.0-yaq6foqrxvuyhdnpkyz2cqog5ejulx6j.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/libtool-2.4.7:
linux-debian11-x86_64_v3-gcc-10.2.1-libtool-2.4.7-qkcficq53dqlhoucbupl7llapl3ytdtw.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/m4-1.4.19:
linux-debian11-x86_64_v3-gcc-10.2.1-m4-1.4.19-uywihodgut7dbu6du3gkejqtdjl4esmm.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/ncurses-6.4:
linux-debian11-x86_64_v3-gcc-10.2.1-ncurses-6.4-6yyq7ujubwm3kyn33cqfdbzwc7altrt4.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/nghttp2-1.57.0:
linux-debian11-x86_64_v3-gcc-10.2.1-nghttp2-1.57.0-b2lg2ibbfzu6gi3u6klnmnfgaoqhbkxc.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/openssl-3.1.3:
linux-debian11-x86_64_v3-gcc-10.2.1-openssl-3.1.3-mprhptd52rqhkszenfgcsosou735mklj.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/perl-5.38.0:
linux-debian11-x86_64_v3-gcc-10.2.1-perl-5.38.0-hhrdw2sgs4u63ig7ljid3tvxajwflayb.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/pkgconf-1.9.5:
linux-debian11-x86_64_v3-gcc-10.2.1-pkgconf-1.9.5-lgtucd6ctfihjpvpinkfhc4gnjkg5r3p.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/readline-8.2:
linux-debian11-x86_64_v3-gcc-10.2.1-readline-8.2-xx6bjh7tgiwx4weeqaopp4hby6g6vjaw.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/util-macros-1.19.3:
linux-debian11-x86_64_v3-gcc-10.2.1-util-macros-1.19.3-jl67jcaenejiqtajjzkzz2lqqk6b6pmb.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/xz-5.4.1:
linux-debian11-x86_64_v3-gcc-10.2.1-xz-5.4.1-hxiintmu5noxrkqvz5stdvmazatxm7mr.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/zlib-ng-2.1.4:
linux-debian11-x86_64_v3-gcc-10.2.1-zlib-ng-2.1.4-vfknuvdtychagyxjj6mt634kkadajlmc.spack

./build_cache/linux-debian11-x86_64_v3/gcc-10.2.1/zstd-1.5.5:
linux-debian11-x86_64_v3-gcc-10.2.1-zstd-1.5.5-xoh7ab7tjny3motqbj67qnzargapj6pm.spack

./repo:
packages  repo.yaml

./repo/packages:
distributedworkflow  executor  gspcdriver

./repo/packages/distributedworkflow:
package.py

./repo/packages/executor:
package.py

./repo/packages/gspcdriver:
package.py

Many of these files (esp. the .spack files) contain binary data but no clear licensing terms. But based on the name I'd guess they contain binaries for the named software, e.g. "readline-82", which is GPL, while the license for DistributedWorkflows.jl suggests MIT is the license.

Perhaps that's OK, perhaps not, but I think at the very least the licensing status ought to be documented clearly.

(The other issues is of course reproducibility: as long as it is not clear where the data comes from, this is also not a given)

[mzeyen1985]

The binary distribution is a binary Spack build cache (https://spack.readthedocs.io/en/latest/binary_caches.html). The build cache was generated for GPI-Space (GPL-3) using its public Spack recipe (https://packages.spack.io/package.html?name=gpi-space), as well as for our own C++ packages distributedworkflow, executor, and gspcdriver. Additionally, the provided self-extracting binaries are streamlining the otherwise error-prone and cumbersome installation process of the underlying system. To ensure the protection of our intellectual property, our C++ packages are not available as open source at this time.

Using the BSD-3-Clause license for DistributedWorkflows.jl is not an issue here as the C++ part linking to GPI-Space is dynamically loaded at runtime through CxxWrap.jl.

[fingolfin]

OK, all fair enough, but I believe this should be addressed explicitly in the README: right now a cursory look at the package may make one believe it is a pure BSD licensed project. But in fact installing it pulls in GPL-3 as well as proprietary software with unspecified licensing terms. For many companies that will be a no-go, or at least require further vetting.

Regarding your proprietary code: so, under which license (not open source, obviously) may one use it then?

[FiroozehDastur]

Thank you very much for the thoughtful feedback. You make a fair and valuable point.
The licenses for the underlying dependencies have now been included with the binaries and are mentioned in the new release notes. We’ve also added a cautionary “Compatibility Notice” in the README clarifying that the package currently supports only a specific Julia version (or lower).

Just to reiterate, this package is not intended for production or industry use. It is primarily an academic proof of concept. The accompanying paper focuses on the interface rather than the underlying dependencies. The proprietary components form part of my own intellectual property, and due to previous experiences with IP misuse and theft, I’ve decided not to make those sources publicly available.

Please let us know if these updates address your concerns. We’ll be happy to close the issue once you confirm everything looks good.

[fingolfin]

this package is not intended for production or industry use

Fully understood. However, also academic users must comply with legal aspects and e.g. respect licensing conditions for bundled open source software. I am glad to hear that you now do, especially since you yourself are concerned about "IP misuse and theft".

Regarding the "cautionary “Compatibility Notice” in the README", unfortunately I could not find it. Can you specify in which part exactly it is, or just quote the relevant section?

[FiroozehDastur]

Could you please check that you’re looking at the correct branch? All updates should be available under the paper/juliacon2024 branch.

[fingolfin]

Ah I was indeed looking at the main branch. Indeed, it has a clear Compatibility Notice, thank you!

However, there still seems to be no licensing information regarding the binaries at https://github.com/FiroozehDastur/DistributedWorkflows.jl/releases so I can't tell whether I am e.g. legally allowed to distribute them further.

But inspecting one of these blobs further, it turns out to contain e.g. a binary of GNU m4, but without the required accompanying GPL license. This looks to me like a clear violation of the terms of the GPL -- but perhaps I am missing something? I tried searching this repository for "m4" but the only hit I got was this issue.

[mzeyen1985]

There are no licensing issues with the binary blob which acts as an installer containing a pre-compiled package repository. According to software licenses only installations need to reproduce the license agreement, while a package repository doesn't qualify as an installation. If any of these third-party installations are in violation of this, I kindly ask @fingolfin to bring this issue up with them.

[fingolfin]

There are no licensing issues with the binary blob which acts as an installer containing a pre-compiled package repository. According to software licenses only installations need to reproduce the license agreement, while a package repository doesn't qualify as an installation. If any of these third-party installations are in violation of this, I kindly ask @fingolfin to bring this issue up with them.

That's a novel legal theory that I never heard before. Do you have any source to back this up?

My understanding is that the GPL does not care about "installation", only about "distribution". Already its preamble makes this clear in layman's terms:

For example, if you distribute copies of such a program, whether gratis or for a fee, you must pass on to the recipients the same freedoms that you received. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights.

Then later in section "6. Conveying Non-Source Forms." it states

You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways:

... and then it lists 5 ways to comply. Indeed, all major package repositories I know which distribute binaries follow these terms. As far as I can tell, you are not following any of them, but I may be mistaken (your statement however lead me to believe that I am not). BTW, personally I think the list is a bit outdated in how it is phrased, and would consider it sufficient to point to e.g. the GNU file mirrors in case of m4 or gmake or diffutils and other GNU FSF tools.

Now, I think there is no point debating that you are distributing copies of the programs, or rather let GitHub distribute them for you; but without attaching any "label" on the outside informing about this. Indeed, I only know that GNU m4 is in there because I downloaded one of the files under https://github.com/FiroozehDastur/DistributedWorkflows.jl/releases/tag/v0.2.1 and figured out how to extract it; there are binaries in there). This is the full list I saw

berkeley-db-18.1.40
boost-1.63.0
bzip2-1.0.8
ca-certificates-mozilla-2023-05-30
chrpath-0.16
cmake-3.27.7
curl-8.4.0
diffutils-3.9
distributedworkflow-latest
executor-latest
gdbm-1.23
gmake-4.4.1
gpi-space-23.06
gspcdriver-latest
hwloc-2.9.1
libiconv-1.17
libpciaccess-0.17
libsigsegv-2.14
libssh2-1.11.0
libtool-2.4.7
m4-1.4.19
ncurses-6.4
nghttp2-1.57.0
openssl-3.1.3
perl-5.38.0
pkgconf-1.9.5
readline-8.2
util-macros-1.19.3
xz-5.4.1
zlib-ng-2.1.4
zstd-1.5.5

However, I am not sure why we are even arguing, it is so easy for you to comply? Just change the text on https://github.com/FiroozehDastur/DistributedWorkflows.jl/releases/tag/v0.2.1 to explicitly reproduce the list of software included, and for each link to a place where one can obtain the sources (with exception of your own proprietary material of course, where however you should at least link to suitable licensing terms -- not source licensing of course, but one still needs a license to know under which conditions one may use your binaries.)

But from a moral and ethical point of view, I find the situation clear: you should comply and give proper attribution -- just like you rightfully want acknowledgment from people using your IP.

[luraess]

Thanks for carefully assessing the situation. I would also suggest that the cleanest way to comply would be to list and acknowledge what should be without changing anything further.