Multi tenant

.gitignore

@@ -1,3 +1,5 @@
 build
 dockerbuild
 localbuild.sh
+config.json
+threshold

README.md

@@ -1,28 +1,39 @@
 
-## threshold
+## threshold 🏔️⛰️🛤️⛰️🏔️
 
-Public Internet facing gateway (TCP reverse tunnel) for server.garden.
+Threshold was created to make self-hosting websites, email, and other services radically easier.
 
-![](threshold.png)
+Threshold implements a public-internet-facing gateway (TCP reverse tunnel & SOCKS5 forward proxy) for self-hosted servers. 
 
-This project was originally forked from https://github.com/koding/tunnel
+The [greenhouse cloud service](https://git.sequentialread.com/forest/greenhouse) was developed in order to make threshold more easily accessible to more people. Greenhouse operates the server side of threshold as a service, charging $0.01 per GB of bandwidth.
+
+![](readme/splash.png)
 
-It is intended to be used to make it easier for non-tech-savvy people to host web services that are avaliable on the public internet.
+Threshold server is designed to be a **relatively untrusted** service, in other words, the user doesn't need to place much trust in the environment where the server runs. It's designed so that the server operator can't spy on you. This makes it uniquely suited to bridge the "ownership vs capability" gap between a self-hosted server/homelab/datacenter and a 3rd-party public cloud environment, hence the name threshold. 
+
+This project was originally forked from https://github.com/koding/tunnel
 
 This repository only includes the application that does the tunneling part.  It does not include any other management or automation tools.
 
 See the usage example folder for a basic test.
 
-![Diagram](readme/Diagram.png)
+![Diagram](readme/diagram.png)
+
+This diagram was created with https://app.diagrams.net/.
+To edit it, download the <a download href="readme/diagram.drawio">diagram file</a> and edit it with the https://app.diagrams.net/ web application, or you may run the application from [source](https://github.com/jgraph/drawio) if you wish.
+
 
 ### How it is intended to be used:
 
-1. An automated tool creates a cloud instance and installs and configures the tunnel server on it. 
-1. An automated tool installs the tunnel client on the self-hoster's server computer. 
-1. An automated tool calls the `PUT /tunnels` api on the tunnel server's Management Port, and sends a JSON file describing which ports should be opened on the tunnel server, which client they should be tunneled to, and which service on the client they should be tunneled to, as well as whether or not the HAProxy "PROXY" protocol should be used. This connection can use TLS Client Authentication.
-1. The tunnel client connects to the tunnel server on the Tunnel Control Port. This connection can use TLS Client Authentication. This connection will be held open and re-created if dropped.
-1. An internet user connects to the tunnel server on one of the ports defined in the JSON. The internet user's request is tunneled through the original connection from the tunnel client, and then proxied to the web server software running on the self-hoster's server computer.
+1. An automated tool creates a cloud instance and installs and configures the threshold server on it. 
+1. An automated tool installs the threshold client on the self-hoster's server computer. 
+1. An automated tool calls the `PUT /tunnels` api on the threshold server's Management Port, and sends a JSON file describing which ports should be opened on the threshold server, which client they should be tunneled to, and which service on the client they should be tunneled to, as well as whether or not the HAProxy "PROXY" protocol should be used. This connection can use TLS Client Authentication.
+1. The threshold client connects to the threshold server on the Tunnel Control Port. This connection can use TLS Client Authentication. This connection will be held open and re-created if dropped.
+1. An internet user connects to the threshold server on one of the ports defined in the JSON. The internet user's request is tunneled through the original connection from the threshold client, and then proxied to the web server software running on the self-hoster's server computer.
+1. (OPTIONAL) The server operator installs software (for example, email server) which requires outgoing requests to "come from" the same IP address that the server is listening for connections at.
+1. The email server or other software connects to the threshold client for SOCKS5 forward proxy. The threshold client forwards this connection through the existing tunnel connection to the threshold server (secured by TLS), then the threshold server handles the SOCKS5 connection and proxies it to the destination requested by the email server or other software. 
 
+Note: if you wish to easily create server/client key pairs that will work with threshold, see: https://git.sequentialread.com/forest/make-fake-cert
 
 ### Output from Usage example showing how it works:
 
@@ -36,7 +47,6 @@ Starting the "listener" test app. It listens on port 9001.  This would be your w
   "DebugLog": false,
   "TunnelControlPort": 9056,
   "ManagementPort": 9057,
-  "UseTls": true,
   "CaCertificateFile": "InternalCA+chain.crt",
   "ServerTlsKeyFile": "localhost.key",
   "ServerTlsCertificateFile": "localhost+chain.crt"
@@ -48,11 +58,10 @@ Starting the tunnel client.  Client Identifier: TestClient1
 2020/08/06 14:00:04 theshold client is starting up using config:
 {
   "DebugLog": false,
-  "ClientIdentifier": "TestClient1",
+  "ClientId": "TestClient1",
   "ServerHost": "localhost",
   "ServerTunnelControlPort": 9056,
   "ServerManagementPort": 9057,
-  "UseTls": true,
   "ServiceToLocalAddrMap": {
     "fooService": "127.0.0.1:9001"
   },
@@ -69,7 +78,7 @@ Sending the tunnel configuration to the server.
 HTTP PUT localhost:9057/tunnels:
 now listening on 127.0.0.1:9000
 
-[{"HaProxyProxyProtocol":true,"ListenAddress":"127.0.0.1","ListenHostnameGlob":"*","ListenPort":9000,"BackEndService":"fooService","ClientIdentifier":"TestClient1"}]
+[{"HaProxyProxyProtocol":true,"ListenAddress":"127.0.0.1","ListenHostnameGlob":"*","ListenPort":9000,"BackEndService":"fooService","ClientId":"TestClient1"}]
 
 Starting the "sender" test app. 
 It connects to the front end port of the tunnel (port 9000).  This would be your end user who wants to use the web application.
@@ -93,10 +102,10 @@ Note how the listener sees the original source IP and port, not the source IP an
 
 I have a few requirements for this system. 
 
-* It should be 100% automatable. It is intended to be used in a situation where it is unreasonable to ask the user to configure thier router, for example, they don't know how, they don't want to, or they are not allowed to (For example they live in a dorm where the University manages the network).
-* Users have control over their own data.  We do not entrust cloud providers or 3rd parties with our data, TLS keys/certificates, etc. In terms of every day usage, this is a TLS connection from an internet user directly to the self-hoster's computer. It is opaque to the cloud provider. 
-  * If the cloud provider wants to launch a Man in the Middle attack, even if they could secretly obtain a trusted cert to use, it will not be easy to hide from the user as long as the user (or software that they installed) is anticipating it. 
-* It should support Failover/High Avaliability of services.  Therefore, it needs to be able to have multiple tunnel clients connected at once, which can be hot-swapped via a Management API.
+* It should be 100% automatable. It is intended to be used in a situation where it is unreasonable to ask the user to perform any sort of advanced manual configuration. 
+* Users have control over their own data.  We do not entrust cloud providers or 3rd parties with our data, even those who are hosting our threshold server. TLS keys/certificates, security-relevant configurations, etc only exist on the user-controlled computer. The cloud provider doesn't get access to any information or capability beyond what the user's ISP (Internet Service Provider) would normally have.
+  * If the cloud provider wants to launch a Man in the Middle attack against the threshold user, they will run into the same problems that an ISP would.
+* It should support Failover/High Avaliability of services.  Therefore, it needs to be able to have multiple tunnel clients connected at once, which can be hot-swapped via a management API.
 
 ### What did you add on top of the koding/tunnel package?
 
@@ -111,15 +120,16 @@ I have a few requirements for this system.
 * Introduced concept of a "service" string instead of port number, so the client decides what ports to connect to, not the server. 
 * Added support TLS SNI based virtual hosts. (Hostname based routing)
 * Fixed various bugs related to connection lifecycle.
+* Added a tunneled SOCKS5 proxy to support applications like email servers which need to be able to dial out from the same IP address that they recieve connections at.
 
 ### How to build
 
 ```
-go build -o tunnel -tags netgo 
+go build -o threshold
+```
 
-# -tags netgo? what?
-# this is a work around for dynamic linking on alpine linux
-# see: https://stackoverflow.com/questions/36279253/go-compiled-binary-wont-run-in-an-alpine-docker-container-on-ubuntu-host
+### How to build the docker image:
 
-docker build -t sequentialread/tunnel:0.0.1 .
-```
+```
+./build-docker.sh
+```

build.sh

@@ -4,7 +4,7 @@ function build() {
 	rm -rf build
 	mkdir build
 
-	GOOS=linux GOARCH=$1 go build -o build/threshold
+	GOOS=linux GOARCH=$1 go build -tags 'osusergo netgo'  -ldflags='-extldflags=-static'  -o build/threshold
 
 	sha256sum build/threshold
 
@@ -46,4 +46,4 @@ function build() {
 
 build arm
 build amd64
-build arm64
+#build arm64

go.mod

@@ -1,10 +1,13 @@
-module git.sequentialread.com/forest/tunnel
+module git.sequentialread.com/forest/threshold
 
 go 1.14
 
 require (
-	github.com/armon/go-proxyproto v0.0.0-20180202201750-5b7edb60ff5f
+	git.sequentialread.com/forest/pkg-errors v0.9.2
+	github.com/armon/go-proxyproto v0.0.0-20210323213023-7e956b284f0a
+	github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5
 	github.com/cenkalti/backoff v2.1.0+incompatible
 	github.com/gorilla/websocket v1.4.0
 	github.com/hashicorp/yamux v0.0.0-20181012175058-2f1d1f20f75d
+	golang.org/x/net v0.0.0-20210614182718-04defd469f4e
 )

go.sum

@@ -1,10 +1,19 @@
-git.sequentialread.com/forest/tunnel v0.0.0-20170601195443-35a8b95662bf h1:2flo/nnhfe3sSxQ/MHlK7KoY54tQ1pAvMzkh0ZOxyH4=
-git.sequentialread.com/forest/tunnel v0.0.0-20170601195443-35a8b95662bf/go.mod h1:i+PvDDsWjggoCQOO8bGJJKRB9qfxmHk5yzIEA/h8dzg=
-github.com/armon/go-proxyproto v0.0.0-20180202201750-5b7edb60ff5f h1:SaJ6yqg936TshyeFZqQE+N+9hYkIeL9AMr7S4voCl10=
-github.com/armon/go-proxyproto v0.0.0-20180202201750-5b7edb60ff5f/go.mod h1:QmP9hvJ91BbJmGVGSbutW19IC0Q9phDCLGaomwTJbgU=
+git.sequentialread.com/forest/pkg-errors v0.9.2 h1:j6pwbL6E+TmE7TD0tqRtGwuoCbCfO6ZR26Nv5nest9g=
+git.sequentialread.com/forest/pkg-errors v0.9.2/go.mod h1:8TkJ/f8xLWFIAid20aoqgDZcCj9QQt+FU+rk415XO1w=
+github.com/armon/go-proxyproto v0.0.0-20210323213023-7e956b284f0a h1:AP/vsCIvJZ129pdm9Ek7bH7yutN3hByqsMoNrWAxRQc=
+github.com/armon/go-proxyproto v0.0.0-20210323213023-7e956b284f0a/go.mod h1:QmP9hvJ91BbJmGVGSbutW19IC0Q9phDCLGaomwTJbgU=
+github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPdPJAN/hZIm0C4OItdklCFmMRWYpio=
+github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs=
 github.com/cenkalti/backoff v2.1.0+incompatible h1:FIRvWBZrzS4YC7NT5cOuZjexzFvIr+Dbi6aD1cZaNBk=
 github.com/cenkalti/backoff v2.1.0+incompatible/go.mod h1:90ReRw6GdpyfrHakVjL/QHaoyV4aDUVVkXQJJJ3NXXM=
 github.com/gorilla/websocket v1.4.0 h1:WDFjx/TMzVgy9VdMMQi2K2Emtwi2QcUQsztZ/zLaH/Q=
 github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
 github.com/hashicorp/yamux v0.0.0-20181012175058-2f1d1f20f75d h1:kJCB4vdITiW1eC1vq2e6IsrXKrZit1bv/TDYFGMp4BQ=
 github.com/hashicorp/yamux v0.0.0-20181012175058-2f1d1f20f75d/go.mod h1:+NfK9FKeTrX5uv1uIXGdwYDTeHna2qgaIlx54MXqjAM=
+golang.org/x/net v0.0.0-20210614182718-04defd469f4e h1:XpT3nA5TvE525Ne3hInMh6+GETgn27Zfm9dxsThnX2Q=
+golang.org/x/net v0.0.0-20210614182718-04defd469f4e/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
+golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
+golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
+golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=