Feature request: Native proxy support without providing custom SocketFactory

[a-zink]

The client allows to set a custom SocketFactory which allows the connection to be established over a proxy. However this is quite cumbersome. Using a proxy should be an easy-to-use feature of the client because this is a quite common use-case. Especially for mqtt over websockets, which is already supported.

There are basically two use-cases for mqtt over websockets
(also see introduction chapter in amqp over websockets which is a related topic)

• Enable browser based mqtt clients
• Network restrictions, e.g within a company network. Most tcp traffic and arbitrary ports are usually blocked. However there might be an http proxy for internet access over ports 80 and 443. This allows us to use mqtt over websockets using the http proxy.

As this is the Java mqtt client, browser support is not the target use-case for websockets. Hence I argue that the only reason for using the websocket feature is network restrictions. However this feature is somehow incomplete without native proxy support.

Also see
#573
#319 (Only addresses socks proxy)
#419 (Seems to be a proxy auth issue. I would say this is out of scope, as workarounds like cntlm exist)

[dasAnderl]

@a-zink im trying hard currently to make mqtt client work behind a proxy.
we are in the company network scenario. the proxy is a web proxy. what i do on my localhost to emulate the server environment is:

using a proxy from https://free-proxy-list.net 62.99.67.216:8080
then i create an entry in /etc/hosts -> 0.0.0.0 <hidden_company_mqtt_broker_name> to block name resolving for the mqtt broker.

in the mqtt client connect options i then use a custom ssl socket factory (see below)
locally this works great and im able to connect to the client though the proxy
on the company server i get: Unable to connect to server
createSocket from SslTunnelFactory is not even called. to me this looks like the client in making some connection to the broker before even using the custom sockets.

btw im also using mqttConnectOptions.setSSLHostnameVerifier(NoopHostnameVerifier.INSTANCE);

does the broker always(by default) support mqtt over websockets or has this feature to be enabled?

For any help i would be more than grateful, as i am seriously running out of ideas here

currently i am using the following custom ssl socket factory (modified from the link):

`/**
*

• This creates a ssl socket factory which uses the proxy from config as tunnel
• Java Tip 111: Implement HTTPS tunneling with JSSE
  
• http://www.javaworld.com/article/2077475/core-java/java-tip-111--implement-https-tunneling-with-jsse.html
• @author zhch

*/
@slf4j
public class SslTunnelFactory extends SSLSocketFactory {

String tunnelHost = CfgProxy.host();
int tunnelPort = CfgProxy.port();
SSLSocketFactory sslSocketFactory = SslContext.ACCEPT_ALL.get().getSocketFactory();

@SneakyThrows
public SslTunnelFactory() {}

@Override
public Socket createSocket(Socket socket, InputStream inputStream, boolean b) throws IOException {
	log.info("createSocket(Socket socket, InputStream inputStream, boolean b)");
	return super.createSocket(socket, inputStream, b);
}

public Socket createSocket(Socket s, String host, int port, boolean autoClose)
		throws IOException {

	log.info("creating tunneled socket to {}:{} via {}:{}", host, port, tunnelHost, tunnelPort);
	System.out.println(String.format("SslTunnelFactory creating tunneled socket to %s:%s via %s:%s", host, port, tunnelHost, tunnelPort));

	Socket tunnel = new Socket(tunnelHost, tunnelPort);
	tunnel.setKeepAlive(true);
	tunnel.setSoTimeout(10000);

	doTunnelHandshake(tunnel, host, port);

	SSLSocket socket = (SSLSocket) sslSocketFactory.createSocket(tunnel, host, port, autoClose);
	socket.setSoTimeout(10000);
	tunnel.setKeepAlive(true);

	socket.addHandshakeCompletedListener(event -> {
		System.out.println("Handshake finished!");
		System.out.println("\t CipherSuite:" + event.getCipherSuite());
		System.out.println("\t SessionId " + event.getSession());
		System.out.println("\t PeerHost " + event.getSession().getPeerHost());
	});
	return socket;
}

private void doTunnelHandshake(Socket tunnel, String host, int port) throws IOException {
	OutputStream out = tunnel.getOutputStream();
	String msg = "CONNECT " + host + ":" + port + " HTTP/1.1\n" + "User-Agent: "
			+ sun.net.www.protocol.http.HttpURLConnection.userAgent + "\r\n\r\n";
	byte b[];
	try {
		/*
		 * We really do want ASCII7 -- the http protocol doesn't change with
		 * locale.
		 */
		b = msg.getBytes("ASCII7");
	} catch (UnsupportedEncodingException ignored) {
		/*
		 * If ASCII7 isn't there, something serious is wrong, but Paranoia
		 * Is Good (tm)
		 */
		b = msg.getBytes();
	}
	out.write(b);
	out.flush();

	/*
	 * We need to store the reply so we can create a detailed error message
	 * to the user.
	 */
	byte reply[] = new byte[200];
	int replyLen = 0;
	int newlinesSeen = 0;
	boolean headerDone = false; /* Done on first newline */

	InputStream in = tunnel.getInputStream();
	boolean error = false;

	while (newlinesSeen < 2) {
		int i = in.read();
		if (i < 0) {
			throw new IOException("Unexpected EOF from proxy");
		}
		if (i == '\n') {
			headerDone = true;
			++newlinesSeen;
		} else if (i != '\r') {
			newlinesSeen = 0;
			if (!headerDone && replyLen < reply.length) {
				reply[replyLen++] = (byte) i;
			}
		}
	}

	/*
	 * Converting the byte array to a string is slightly wasteful in the
	 * case where the connection was successful, but it's insignificant
	 * compared to the network overhead.
	 */
	String replyStr;
	try {
		replyStr = new String(reply, 0, replyLen, "ASCII7");
	} catch (UnsupportedEncodingException ignored) {
		replyStr = new String(reply, 0, replyLen);
	}

	/*
	 * We check for Connection Established because our proxy returns
	 * HTTP/1.1 instead of 1.0
	 */
	//if (!replyStr.startsWith("HTTP/1.0 200")) {
	System.out.println("reply str : " + replyStr);
	if (!replyStr.toLowerCase().contains("200")) {
		throw new IOException("Unable to tunnel through " + tunnelHost + ":" + tunnelPort + ".  Proxy returns \""
				+ replyStr + "\"");
	}

	log.info("tunnel ssl handshake successful");
}

@Override
public String[] getDefaultCipherSuites() {
	return sslSocketFactory.getDefaultCipherSuites();
}

@Override
public String[] getSupportedCipherSuites() {
	return sslSocketFactory.getSupportedCipherSuites();
}

@Override
public Socket createSocket(String arg0, int arg1) throws IOException {
	return createSocket(null, arg0, arg1, false);
}

@Override
public Socket createSocket(InetAddress arg0, int arg1) throws IOException {
	return createSocket(null, arg0.getHostName(), arg1, false);
}

@SneakyThrows
@Override
public Socket createSocket(String arg0, int arg1, InetAddress arg2, int arg3) {
	return createSocket(null, arg0, arg1, false);
}

@Override
public Socket createSocket(InetAddress arg0, int arg1, InetAddress arg2, int arg3) throws IOException {
	return createSocket(null, arg0.getHostName(), arg1, false);
}

}
`

[AndrewJudson]

I'm also interested in a solution to this. I am currently blocked on using paho due to this

[AndrewJudson]

@dasAnderl did you ever figure this out? I have been trying your solution and was able to perform SSL handshake with the server, but then paho gave me java.net.SocketException: Already connected

[a-zink]

@dasAnderl @AndrewJudson please see my comment #573 (comment)

[a-zink]

@dasAnderl

does the broker always(by default) support mqtt over websockets or has this feature to be enabled?

Yes, the broker of course needs to support websockets.

[dasAnderl]

@a-zink
sorry andreas for my very late reply
it worked for us with the custom socketfactory and java below 1.8.0_261-b25 and mqtt version 1.2.0
later mqtt versions do not seem to support the custom ssl factory anymore #706 ->
java.net.SocketException: Unconnected sockets not implemented

here the working custom factory in kotlin:

@Suppress("MagicNumber")
internal object MqttClientSocketFactory {
    @JvmStatic
    operator fun get(certPair: X509CertPair, brokerUrl: String): SSLSocketFactory {
        return when (CfgProxy.useProxy) {
            true -> socketFactoryForProxy(brokerUrl, certPair)
                .also { log.info("using proxy socket factory") }
            false -> socketFactory(certPair)
        }
    }

    private fun socketFactory(certPair: X509CertPair): SSLSocketFactory {
        val keyManagerCsa = keyManagerVkms(certPair)
        return sslCtxVkmsAws(keyManagerCsa).socketFactory
    }

    private fun socketFactoryForProxy(brokerUrl: String, certPair: X509CertPair): SSLSocketFactory {
        val hostName = getHostName(brokerUrl)
        val keyManagerCsa = keyManagerVkms(certPair)
        val delegate = sslCtxVkmsAws(keyManagerCsa).socketFactory
        return object : SSLSocketFactory() {

            override fun getDefaultCipherSuites(): Array<String> {
                return delegate.defaultCipherSuites
            }

            override fun getSupportedCipherSuites(): Array<String> {
                return delegate.supportedCipherSuites
            }

            @Throws(IOException::class)
            override fun createSocket(proxySocket: Socket, host: String, port: Int, autoClose: Boolean): Socket {
                doTunnelHandshake(proxySocket, hostName, 8883)
                return delegate.createSocket(proxySocket, hostName, 8883, autoClose)
            }

            // PAHO DOES NOT USE
            @Throws(IOException::class, UnknownHostException::class)
            override fun createSocket(s: String, i: Int): Socket? {
                return null
            }

            // PAHO DOES NOT USE
            @Throws(IOException::class, UnknownHostException::class)
            override fun createSocket(s: String, i: Int, inetAddress: InetAddress, i1: Int): Socket? {
                return null
            }

            // PAHO DOES NOT USE
            @Throws(IOException::class)
            override fun createSocket(inetAddress: InetAddress, i: Int): Socket? {
                return null
            }

            // PAHO DOES NOT USE
            @Throws(IOException::class)
            override fun createSocket(inetAddress: InetAddress, i: Int, inetAddress1: InetAddress, i1: Int): Socket? {
                return null
            }
        }
    }

    private fun sslCtxVkmsAws(keyManagerVkms: KeyManager) =
        SSLContextBuilder()
            .build()
            .apply {
                init(
                    arrayOf(keyManagerVkms),
                    arrayOf(acceptAllTrustManager()),
                    SecureRandom()
                )
            }

    private fun keyManagerVkms(certPair: X509CertPair): KeyManager {
        val certKeys = certPair.keyPair
        val selfSignedChain = certPair.certs
        return getKeyManagers("glcs_self_signed", certKeys, selfSignedChain)
    }

    @Throws(IOException::class)
    private fun doTunnelHandshake(tunnel: Socket, host: String, port: Int) {
        val msg = ("CONNECT " + host + ":" + port + " HTTP/1.0\n" +
                "User-Agent: " +
                System.getProperty("http.agent") +
                "\r\n\r\n")
        val out = tunnel.getOutputStream()
        val b: ByteArray
        b = try { /*
			 * We really do want ASCII7 -- the http protocol doesn't change
			 * with locale.
			 */
            msg.toByteArray(charset("ASCII7"))
        } catch (ignored: UnsupportedEncodingException) { /*
			 * If ASCII7 isn't there, something serious is wrong, but
			 * Paranoia Is Good (tm)
			 */
            msg.toByteArray()
        }
        out.write(b)
        out.flush()
        /*
		 * We need to store the reply so we can create a detailed
		 * error message to the user.
		 */
        val reply = ByteArray(200)
        var replyLen = 0
        var newlinesSeen = 0
        var headerDone = false /* Done on first newline */
        val `in` = tunnel.getInputStream()
        while (newlinesSeen < 2) {
            val i = `in`.read()
            if (i < 0) {
                throw IOException("Unexpected EOF from proxy")
            }
            if (i == '\n'.toInt()) {
                headerDone = true
                ++newlinesSeen
            } else if (i != '\r'.toInt()) {
                newlinesSeen = 0
                if (!headerDone && replyLen < reply.size) {
                    reply[replyLen++] = i.toByte()
                }
            }
        }
    }

    private fun getHostName(brokerUrl: String): String {
        return brokerUrl
            .replace("ssl://", "")
            .replace("https://", "")
            .replace("http://", "")
            .also {
                if (it.contains(":") || it.contains("/")) {
                    throw AssertionError("the hostname should not contain any protocol or path: $it ")
                }
                log.info("using hostname $it")
            }
    }
}

[ramki519]

Is this feature supported yet? This is a blocking issue for many.

[onkariwaligunje]

Hello,
Has anyone got solution to this problem. I am also facing issue while establishing connection through http proxy. It gives ERROR-MqttAgent Error message: Connection to the Mqtt Failed due to MqttException (0) - java.net.SocketTimeoutException.

I tried the solution given by dasAnderl, but I am getting java.net.SocketException: Already connected

I am currently blocked due to this issue, any help is greatly appreciated, Thanks!

[onkariwaligunje]

#1010
The changes in this PR provides proxy support. I tried building code and tested it and it does establishes the connection over a http proxy