HTTP requests return null after a network change on iOS

[GSRicky]

On iOS, if the network connection type changes while the app is in the background, after returning to the foreground, new HTTP requests will fail (after a timeout) with a null response even though an internet connection is available.

Environment:

• Unity: 2020.3.19f1
• BestHTTP: 2.5.2 (issue also reproduced on 2.8.3 for testing).
• iOS: all OS versions seem to be affected

100% Issue Reproduction Method:

1. Send HTTP requests using BestHTTP
2. Wait for all requests to receive their responses (all successful
3. Put the iOS app into the background to access the device’s networking setting
4. Switch between mobile data/WiFi (enabling one and disabling the other
5. Wait a few seconds (wait for the device’s connection to the internet to be re-established
6. Return the app to the foreground
7. Send HTTP requests using BestHTTP
8. (ISSUE) All HTTP requests fail (timeout) with a null response.

• Note: The above case cannot be reproduced if I substitute the BestHTTP requests with UnityWebRequests of the same configuration.

Details:

• This issue may be a duplicate of #164 but that issue is not explained in detail.
• The issue is 100% reproducible when following the above scenario though we also have several other known issue cases such as:
  • Changing between WiFi and a Hotspot by enabling and disabling the hotspot on a paired device
  • Exiting the range of a WiFi adapter and having the device automatically connect to mobile data
• The issue only occurs on iOS, we are not aware of an issue on Android devices. We believe all iOS versions are affected.
• If we switch out BestHTTP with UnityWebRequest the issue cannot be reproduced in any scenario.
• Our HTTP requests already include the “connection close” header (though changing this did not affect the outcome)
• Our retries are set to 0 (though we have also tested with 5 and this had no effect on the outcome)
• We are using HTTP/2
• The keep-alive header does not affect this issue (as expected on HTTP/2).

Known Workarounds:

• If we call HTTPManager.AbortAll() when the app executes OnApplicationPause as it enters background, the above case is not reproduced, however, this solution cannot be used to resolve the related issues where the app does not enter the background state.
• If we wait 90~120s after the initial successful web requests are complete and then run our test, the issues are not reproduced (we are not sure of the exact timing as our scenario needs to allow the network to reconnect and this timing is inconsistent).
• We know a large number of our users have been affected by this issue so we have disabled BestHTTP for our network requests on iOS and have reverted to UnityWebRequest as a temporary solution. This change has resolved all network handover issues.

[Benedicht]

Could you follow the steps in the linked issue to collect and send over all the logs?

[GSRicky]

Within the logs, there are potentially sensitive details such as our class and method names, as well as server communication routes. Due to company security policies, I cannot share this information externally.

Accordingly, I previously shared the above 100% reproduction case in as much detail as possible. Apologies for not being able to share the logs directly.

[Benedicht]

Unfortunately my Mac is in service, called them today, but seems they'll send it back in one or two weeks. I will try an online service to get access to a Mac and hopefully i can make a test build.

[GSRicky]

We appreciate the effort. Sorry again that I cannot send the real case logs directly.

We have reverted to using UnityWebRequest in production in the meantime and while there is a performance hit, there have been no more reported issues with request timeouts. If you think there is a setting or configuration issue for BestHTTP we might have missed, please let me know and I will try to schedule a test. We would like to switch back to BestHTTP as quickly as possible.

[Benedicht]

The problem is that the TCP connection isn't closed, or at least it wasn't propagate to the application layer, even as the plugin reads from and writes to. This one is what i'm not understanding, the underlying systems knows about the change and that its quite disruptive, if MPTCP isn't set for the connection, i see no way the tcp connection could remain open, so the system should report its connection lost state.

When finally fails for any reason depending on a few things (request settings for example) the plugin might retry the request. I don't know whether it might be there a case where the retired request is assigned to the same failing connection.

Our HTTP requests already include the “connection close” header (though changing this did not affect the outcome)

The plugin's doesn't check for any connection header (and its value). But you can control whether to keep alive and reuse connections through HTTPManager.KeepAliveDefaultValue: setting it to false disable connection pooling. However, this one disables HTTP/2 too.

The Workarounds

• HTTPManager.AbortAll() can work as it preliminary removes the requests before anything could happen. It also has the side-effect that any http/1 connections are removed too, because the connetion's state is uncertain. And this side-effect's side-effect is that any new request will open a new one that will work with the new network connection type.
• Waiting 90-120 seconds before the network change works, becuse the connections' idle time is over and they are closed.

Both HTTPManager.KeepAliveDefaultValue and HTTPManager.AbortAll() are viable workaround, but all of them are ugly ones for different reasons:

• HTTPManager.KeepAliveDefaultValue: Setting it to false removes connection pooling and that's just enought to not like it, but it's still a cleaner then:
• HTTPManager.AbortAll() This one removes every requests, even if the user just checking the weather and all the request would finish just fine.

Unity (most probably) uses Apple's own request-response system that knows about these network changes and can handle them accordingly.

A proper solution would be to write a native plugin to be able to get subscribed/notified about the changes and retry/abort the requests.

[Benedicht]

It might be enough to use Unity's own NetworkReachability. The main logic would be to store the reachability when the connection is created and if it's changes retry/abort the assigned requests.
This would remove the need of manual intervention, connections could remain open and alive longer to maximize their reusability.

An additional feature could be to add a new setting to control whether to postpone the creation of connections if the network is unreachable until the reachability changes.

[Benedicht]

I have a working version, but i'm not sure whether i like it. :)
My only problem (for now at least) with it is that the flags are mutually exclusive, it reports the network is either available through wifi or cellular, even if both are working perfectly fine. So, if only wifi is turned on and connections are made through it, turning on cellular too, Application.internetReachability ReachableViaCarrierDataNetwork. And this will cause the closure of perfectly fine connections.

[Benedicht]

@GSRicky If you think drop me a mail to besthttp@gmail.com and i can send a new package.

[GSRicky]

Thanks for taking the time to look into this one. I have sent you an email so we can try to do some internal testing on this approach. Actually, we have also looked into some of these strategies, so I will first share our limited experience.

[1] AbortAll

We had similar results with AbortAll(). This was a mostly ok solution at first but we found that there are similar reproduction cases that do not involve changing the app background/foreground state. Without knowing that the network state has changed we don’t see a good opportunity to refresh the connection generally.

Issue scenario for AbortAll solution:

  Step 1) (Test Device) Enable LTE and WiFi
  Step 2) (Test Device) Disconnect from and disable auto-join for all WiFi Connections
  Step 3) (Support Device) Enable hotspot
  Step 4) (Test Device) Connect to hotspot
  Step 5) (Test Device) Start app 
  Step 6) (Test Device) HTTP Request on hotspot connection (success, no issue)
  Step 7) (Support Device) Disable hotspot
  Step 8) (Test Device) Wait 10 seconds for the network to establish a connection on LTE
  Step 9) (Test Device) HTTP Request on LTE  connection (ISSUE: failure, timeout)

In this scenario, the test device does not change between background and foreground, and there is no indication of a network connection change until the LTE request timed out (step 9). As such, there is no good opportunity to abort all requests.
(Note: using Unity web request, we could not reproduce the issue in this scenario)

[2] KeepAliveDefaultValue

In our tests, modifying the KeepAliveDefaultValue did not change any of the outcomes. We did not investigate this too deeply as we intend to use HTTP/2.

[3] NetworkReachability

We are already checking the NetworkReachability in a Unity update loop in order to manually refresh socket connections (BestHTTP) when the connection changes. We believe there may be a similar issue with the socket connection not detecting network changes on iOS and thus not receiving any packets on the socket. (We have not investigated this thoroughly enough to report the details yet).

We have not fully tested with the NetworkReachability approach for HTTP connections but we had several concerns when considering it:
Performance cost: Presumably we have to check this on every network request, we have not yet checked to see if there is any impact when sending many requests.
As we do not understand the underlying cause very deeply, we are not sure if the issue could persist in some WiFi -> WiFi cases. Such cases (perhaps: (1) connect to a hotspot, (2) hotspot disconnects, (3) the device connects to a WiFi adapter) may be “ReachableViaLocalAreaNetwork” both before and after a network change.

In conclusion, we anticipate there may be some limitations with the workaround and are cautious that it will not be as robust as whatever UnityWebRequest is doing. We are eager to return to BestHTTP with Unity so greatly appreciate your support in getting this resolved.

Many thanks.

[Benedicht]

My approach with NetworkReachability is to check its value and compare with the previous one. If the two values are different, close any active connections. As i wrote, it has its limitations and as you noted Application.internetReachability might have overhead too.

So I became uncertain about my approach using NetworkReachability and took a step back to think about the whole thing again.
Even if there's no direct feedback right there when the tcp connection is aborted, by sending keepalive (ping) messages we can still detect problems.

The plugin already sending HTTP/2 ping messages, but the default settings are too high for your cases.
HTTP/2 keepalives (http/2 ping frames) is a high-level mechanism to check whether the (TCP) connection is still alive.

These pings are quite good detecting network problems, however it might take a few iterations to find a good balance between detection speed and false positives.

I thionk the following setup is quite good to start the experiment with:

// Time between two ping messages:
HTTPManager.HTTP2Settings.PingFrequency = TimeSpan.FromMilliseconds(500);

// If no ack reveived in this time the connection will be treated as broken.
HTTPManager.HTTP2Settings.Timeout = TimeSpan.FromSeconds(1);

These settings are already in the package, i'll not send a new one.

[GSRicky]

Thanks a lot and sorry for taking so long to get back to you.

I have re-opened the issue here so we will try to take a look at what the changes mean to our use case.

1) Network Monitoring and Retries: Preventing Web Request Failures with BestHTTP
One thing that jumps out is that with BestHTTP pinging every 500ms to validate the connection, if a web request fails due to a dropped connection, it will timeout after a window of say 30 seconds (settings-dependent). By this time, BestHTTP should have comfortably detected the network change and terminated the previous TCP connection, it would be in a state ready to create a new connection for any retry attempts, right? My thinking is that with the connection checking and retries enabled, the issue should not be reproducible.

2) Follow-up Email: Requesting a Package
I did send an email before requesting the package but I can’t seem to find a response now. Very sorry if this is due to our following up on this after such a delay. I just sent another quick email to request it once again, hope that is ok.

Thanks as always.

[Benedicht]

1. "if a web request fails due to a dropped connection, it will timeout after a window of say 30 seconds"

The plugin tries the same logic, detects the failure, if possible resends the request and that's triggers the creation of a new connection.
However, there are two issues that i know about and could result a simptome like you described.

1. When network conditions are changing, iOS/Unity fails to connect to the IP addresses the DNS query returns with. I don't know the details of this one, as couldn't replicate it locally and never got diagnostic logs for this.
2. The second issue happens when the DNS returns with a list of IP addresses and the first IP addresses aren't available. The plugin tries to connect to the IP addresses one after the other, so it's very possible that it'll time out while trying to connect to the very first address...

This was one of the issues I started to work on the next major version of the plugin:

• It will have a DNS cache
  • to cache the received host->ip addresses mapping and
  • track what addresses aren't working
• Connect to multiple IP addresses at the same time and pick the first connecting one.

With these applied, not just the connection speed should increase but the reliability too.

2. Package request

I didn't applied any changes regarding this issue. But, i'm happy to send a link to the latest version available in the store. If you wish, i can send the next version too, however that one isn't backward compatible with the v2.x versions.