Possibly reduce minimum Android API Level

[BenjaminAmos]

Currently, the minimum API level has been raised to API 24 (Android 7.0). According to the Google Play survey, this would allow the app to run on ~57.9% of Android devices still out there. Due to the intergration of gestalt, by-default the API level cannot be lowered, as this would disallow usage of Java 8 APIs in general. The previous Android release of the app targeted API 10 (Android 2.3.3), which would support 100% of the surveyed devices (although that is biased, as I don't think that the Google Play store supports versions of Android that are any older). This may cause significant reduction in the number of new potential players, simply because they have not updated their devices.

(source: https://developer.android.com/about/dashboards)

My primary physical testing device runs on Android API 16 (99.4% coverage), which is what I would plan to support as a minimum. So far as I can tell, there are two primary approaches to reduce the minimum API level:

• Re-write all the code to be Java 7 API compliant
• Bytecode manipulation

Java 7 re-write

I did attempt to re-write all the code to be Java 7 API compliant a while back, although it was a considerable amount of work to achieve and would have been a step backwards in terms of desktop support. It involved changes to gestalt, the Android facade and the main DestinationSol repository. It also required me to modify my warp module in order for it to run. The primary issue here is that the code always compiles. All the missing methods fail to resolve at runtime, meaning that you constantly need to re-compile and deploy the apk to test any changes. The primary APIs that I had to re-write are as follows (this is from memory, so it is not fully comprehensive):

• Streams API (.stream().x()) - This was used extensively in the code and usages would probably have been re-added again in the code with future contributions (Even with the current Android code, JSONObject.*Float does not exist on Android and is a recurring issue).
• .<operation>If methods on collections (e.g. myList.removeIf(x -> true))
• .foreach methods on collections (myList.foreach(x -> doStuff(x)))
• Any uses of the java.util.function package
• Any uses of the java.util.stream package
• Any uses of the Map.compute* methods, as well as many other methods in the Map interface
• SimpleDateFormat format strings containing the "X" character (this is not supported on Android until API 24) - This issue primarily affected gestalt, rather than Destination Sol directly

The code ran but it seems that I deleted it at some point, so I can't really demonstrate the extent of the changes needed.

Bytecode manipulation

My second attempt at investigating this involved java bytecode manipulation at compile time. This worked far better, as I was able to keep most of the code the same.

I used a tool for Android builds called ProGuard, which has the ability to re-write java bytecode to run on older JVMs. Recently, it also added the ability to back-port any Java 8 Streams API usages in the code, which helped immensely. Unfortunately, some of the required newer APIs that were not back-ported by ProGuard. In these cases, I wrote some code that manually manipulated the bytecode as part of the Android build process (none of this happens when doing non-Android builds).

There were very few cases where this needed to be done. The bytecode-manipulator simply looked for the usages of certain functions and classes and redirected them to the relevant compatibility classes, many of which I included within the Android facade source. You can see the source code for it here. The following items needed to be re-directed:

• java.text.SimpleDateFormat constructor
• java.util.Locale.getDefault method
• java.lang.String.join
• java.util.Objects class
• Guava com.google.common.base.Function parent interface

I did have to re-write some of the code in the Destination Sol codebase as well as it's Android facade, however the changes were mostly minimal. They primarily consisted of removing .foreach calls (which given enough time I would have worked-out how to re-write them using the pre-build bytecode-manipulator). It appears that .foreach calls were automatially ported by the ProGuard backporter and so did not need to be modified. The changes to those calls were reverted.

Most of the manual bytecode changes should be able to be removed when the Android Gradle Plugin 4 is released (see https://developer.android.com/studio/preview/features/#j8-desugar for more information), as it provides a more comprehensive back-porting functionality than ProGuard does.

The primary issue with this method is that I am not sure of what newer APIs could be used in the future. Also, my implementation has a rather long initial loading time, due to I think the addition of multi-dex support.

I have opened this issue to discuss any other possible alternatives, as well as the viability of the bytecode manipulation as a solution to lower the minimum API level supported. I've included the Java 7 example here to describe a possible, yet unlikely solution, due to the reduction in developer productivity.

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[Cervator]

Very well written issue! Thank you.

But IMHO isn't going back API levels or to Java 7 the wrong direction to go? :-)

If we were trying to optimize the amount of devices we could reach to expand revenue or something and it made financial sense to do so I could see that. But we're a fairly casual all-volunteer open source project I think just generally would be happy to see our games get played at all.

I'd rather look to the future and consider something like Kotlin Native at some point to get on more platforms entirely. We move so slowly that by the time we did anything major for Android (like get Terasology working on it) that 57.9% number would probably be up to 80-90%. Possibly a bigger limitation might be rendering support on older devices? I don't know how OpenGL ES works out in that case, especially if we try to aim for v2 at least, rather than our current mishmash just relying on v1.x something. More thinking Terasology there than DestSol though.

That's not to say we couldn't expand compatibility further backwards, and if you're really eager to do so and already done a bunch of cool work with ProGuard etc anyway then awesome! I wouldn't mind seeing it in action. The one concern I'd have is if we add any constraints to go backwards we'd need to undo if we later try to go forwards - to adopt better build tooling for Android, try something like Kotlin Native, eventually try to rebase on Java 11, and so on. Especially if any of that might be near enough in the future to end up resulting in a beautiful piece of work that barely gets used

Pinging @immortius on the topic, would likely have far more valuable feedback than me 👍

(Also: huh, was surprised to see "Edited by Cervator" on your message - looks like that's the old Bountysource book adding the little footer thing, guess it is still enabled in this repo)

[BenjaminAmos]

Thank you for your feedback. I have considered the concerns you raised and I believe that I may need to make certain clarifications to my initial proposal.

The minimum API level is the lowest API level that an Android app can theoretically support.
It may not support it fully and may have reduced functionality when running under that API level.

I am also not in favour of reducing the Java language version used to Java 7.
It is not needed, as the compiler can de-sugar Java 8 syntax and I often find Java 8 APIs useful, such as the streams API and .foreach method. I mentioned that solution as it was possible but not realistic.

The way that the bytecode solution works is by modifying the code after it has been compiled,
so that no source code changes are needed apart from to very occasionally work-around bugs in the translator (this mostly involves using variables with an abstract type rather than a concrete one e.g. using Map instead of HashMap).

The modified bytecode runs almost perfectly on Android versions >= API 21 (Android 5.0) but due to a lack of multi-dex support on older versions the code will be far slower to start on Android versions < API 21.

I have not mentioned Terasology here and this issue was raised purely within the context of Destination Sol. I had not considered using Kotlin or any other languages, as this proposal was intended to concern short-term impacts rather than long-term ones. The primary problem with using Kotlin/Native is that it only supports Kotlin code, so the entire codebase would need to be ported to Kotlin. It also does not appear to support reflection very well at the moment, which is used extensively within gestalt.

LibGDX provides abstractions of all the OpenGL ES APIs, so in Destination Sol this is not a concern.
I would suspect that OpenGL ES 2.0 will be supported as a minimum though. This would not be an issue on Android as OpenGL ES 2.0 support has been required since Android 4.0 (https://source.android.com/compatibility/4.0/android-4.0-cdd.html#7.1.4.-2d-and-3d-graphics-acceleration).

As you mentioned, eventually support for older Android versions will no longer be relevant, as the majority of devices will no longer run them. I was pursuing this option purely as a matter of curiosity for my own purposes, with the hope that it might help the project in the future.

If any changes made to the engine break compatibility with older Android versions, then you can temporarily increase the minimum API level again. Some changes have already been made in order to support slightly older Android versions, such as removing any uses of the java.nio apis in order to support API 24 (the minimum supported by gestalt). Any solutions used to maintain backwards compatibility with older Android versions will always be interim solutions, which are not designed to be permanent. As soon as those versions go out of support, their usage will fall significantly and the work-arounds in-place for them can be removed.

Regarding potentially changing the Java version used in the future, if the intention is to retain Android support then this may not be possible anywhere in the near future. The maximum Java version supported by Android is Java 8, as shown here.

All of the changes that I made to the engine for bytecode manipulation to work can be found in
this commit. The android facade changes can be found on this branch.

I am still very much open to feedback and any new ideas or perspectives on this issue are welcome.

[immortius]

On my end as the main gestalt developer, I'll have a look at whether I can pare back the min API level a little, will have to see where things become uncomfortable. It is also possible to retain features that require a higher API level, marked as such, where they are not critical and still lower the minimum supported - I've done that a bit with nio (since I still wanted to support the file watching for auto-reload feature).

I might be able to specifically rework a couple of the areas you've identified - such as the SimpleDateFormat.