Feature/mvt integration

[rkreienbuehl]

@p-lr here the new MR draft of my work.

Rasterizer is still very slow on all devices (desktop seemed fast at first, but depends on the hardware), performance needs to be drastically improved, maybe you have some ideas what could bring better performance? One idea is to draw the symbols instead of using markers, but rendering itself also needs to be improved. I will investigate this further. Maybe you also have some inputs on how to improve performance?
I could fix the crashes on iOS and Android, so the demo works on iOS, Android and desktop
Collision detection is not integrated right now
I am in holidays from next week, but will work on it as I find the time.

[p-lr]

In addition, when you rotate the map, symbols like place or street names must remain straight. So you see that they are handled as markers. Only we need to figure out a way to do it efficiently.

[rkreienbuehl]

Thats true. I am on it to bring symbols into a separate layer, so no markers need to be added. It works on desktop, on iOS there is a problem, will work on it more in the evening and push it as soon as there is a first milestone on it.

[p-lr]

I guess the symbols layer mostly replicates how the marker layout works. But yes, it's a good idea to have a core concept dedicated to symbols, instead on building around the markers api.

[rkreienbuehl]

Yes, instead of render() on a symbol, i implemented a draw() method that takes the drawScope and the draws on it mostly the same way how render() worked.

[rkreienbuehl]

Symbol painting is now in a separate layer. This works, but it is not tested into detail.
Right now, the rendering performance is the biggest problem, therefore symbols are painted delayed.

[p-lr]

So I finally finished some work on another app and I have more time for MapCompose now.
Did a quick test, and the new "Vector tile demo" works on android -- good job!
As you said, there's work to do on symbols. I zoomed on Ajaccio (the capital of Corsica) and this what loaded after ~2mins ><:

No doubt there's a lot of work on my side to review the massive amount of new code.
Instead of submitting a huge list of remarks, I prefer to go step by step. The first thing I noticed is the code added to MapState:

private val mViewportInfoFlow = MutableStateFlow<ViewportInfo?>(null)
    val viewportInfoFlow: StateFlow<ViewportInfo?>
        get() = mViewportInfoFlow

Analyzing further, it seems to be used by mapLibreDebugApp, which was the experimental demo. If I understand correctly, your newly added demo supersedes this right? So we can delete this mapLibreDebugApp? If you confirm, I can remove it and refactor this part (as I believe the viewportInfoFlow could be exposed in a cleaner way) if you allow me to push on your branch.

Thanks again for your work!

[p-lr]

Alternatively, we can merge your MR right now, so I can contribute on feature/mvt-integration directly.

[rkreienbuehl]

I just added you as collaborator.

Analyzing further, it seems to be used by mapLibreDebugApp, which was the experimental demo. If I understand correctly, your newly added demo supersedes this right? So we can delete this mapLibreDebugApp? If you confirm, I can remove it and refactor this part (as I believe the viewportInfoFlow could be exposed in a cleaner way) if you allow me to push on your branch.

Thats correct. I didn't do any cleanup so far and left the mapLibreDebugApp module as is, if I would have needed something to lookup when working on it, bit this module will not build anymore.

[p-lr]

I've removed the former demo, and refactored core classes like MapState and TileCanvasState.
Now, the internal concept of ViewportInfo is generated in the body of MapState.addVectorLayer where this flow is collected. I left a comment to refactor this later because it adds clutter in this function.

My next focus will be the high level architecture of MapState.addVectorLayer. Like for example, getting a Rasterizer is done using a suspending function. The reason is the style needs to be fetched. Once we get a rasterizer we update a var in the body, as we need the rasterizer for symbols. This tells that we cannot start producing symbols until the rasterizer is ready. So my plan is to refactor this part to get a better high level view.

[p-lr]

After a quick analysis. I could spot two causes for symbols performance issue.

1. Way too many symbols are drawn. This hogs the main thread. See the screenshot below. A total of ~5k symbols. We should expect a few hundreds at most.
2. Symbols are produced in such a way that tasks accumulate. In the example below, it took several seconds to generate the 5k symbols. But if I zoom in then out, several symbol production tasks will pile up and the delays accumulate.

To fix issue 1, we'll have to limit the number of produced symbols. By using collision de detection and/or using some algorithm, and also maybe using a hard limit.
To fix issue 2, I will use a similar technique as in TileCollector.

But before going through those fixes, like I said in my previous comment, I need to refactor the MapState.addVectorLayer function.

Don't hesitate to let me know that you read my messages.

[rkreienbuehl]

Yeah, I noticed the same. At the moment every symbol is drawn. I will research how maplibre handles symbols and implement such a behavior to address issue 1.

[p-lr]

I will let you know when I'm done with the refactor. I'll see if I can also tackle issue 2 at the same time.
I also noticed that tiles rendering can be optimized but that will require some major changes to the core library, so it isn't the priority.

[rkreienbuehl]

Sounds good to me. I'll find out more about collision detection and involved algorithms in the meantime.

[p-lr]

I've partially improved issue 2, or at least enough for the moment. Two major changes:

• Upon viewport change, any ongoing symbol processing is cancelled. This is done with:

viewportInfoFlow
    .throttle(250)
    .collectLatest { viewportInfo ->
...
}

• Switched to processing symbols using Dispatchers.Default

I believe this is enough for issue 2 (for now). You can focus on issue 1.

[rkreienbuehl]

I made some research. What I found out so far:

To come closer to how maplibre does symbol placement, symbols should be preproduced so I will implement a placement engine. This would improve performance, because symbols aren't produced every time the viewport changes. Also with the current implementation of symbols only the priority is used for determining the order of placing them, maplibre also uses the layer as info for this.
When the viewport changes, I need only to place symbols (and do collision detection) instead of rendering them every time.

Will take some time to implement this properly. Will get back as soon as there is an update on it.

[p-lr]

Having a SymbolsEngine feels necessary indeed. It might be interesting to discuss the design before starting the implementation.
On top of my head, there are two different things that needs to be clarified. Currently, our "viewport" is a combination of two very different data:

• the tile set
• the visible area

On top of my head, the engine should make this distinction. Like, when the tile set changes, that means new tiles are visible (or a completely different list of tiles). This is different from when the visible area changes (alone), as in this case it means the user has zoomed in or out but the tiles set is still the same. In that case, the internal list of symbols remains the same, only the scale changes and the engine should probably run collision detection on all the markers in cache to determine the new list of visible symbols.

To summarize, the engine would maintain an in-memory cache of produced symbols, and that cache would be updated whenever the tile set changes. Maybe that cache would look like a Map<Tile, List<Symbol>>, to that symbols are cached per tile (e.g. some sort of 2D cache). That cache would contain several thousand symbols in total, as we've seen. However, thanks to collision detection and the logic you mentioned, only a limited list would be sent to the ui for rendering.

What do you think?

[rkreienbuehl]

I totally agree. This approach would also be very close to what maplibre does with symbols. I didn't had the time to come up with a full concept, but will work on it this week as much as I find time, so we can discuss this in detail.

[p-lr]

Sounds good, no hurry.

[rkreienbuehl]

@p-lr I finally found some time after holidays.

So for working on this I analyzed how it work right now vs how MapLibre does it:

Current pipeline vs target pipeline

• Current: viewport change -> produceSymbols(viewport) -> collision -> draw
• Target: tile/style change -> produceCandidates(tile) (cached) ; viewport change -> place(candidates) -> draw

This would need the following key components and their responsibilities:

• SymbolLayerPainter / SymbolsProducer: candidate factory (slow). Decode geometry, measure text, resolve sprites, generate stable IDs and anchors.
• SymbolPlacementEngine: placement (fast). Gather visible candidates, sort by priority (layer first), rebuild screen-space bounds/OBB, run collision, return draw list.
• CollisionDetector: screen-space collision index. Uses Rtree coarse search plus OBB intersects for exact tests.

Symbols

• Symbols should get a layerId for layer first priority.

• There should be SymbolCandidates and PlacedSymbols.

  • SymbolCandidates: are stored/cached by tile in SymbolPlacementEngine ;
    Something like this:

    data class SymbolCandidate(
      val symbol: Symbol,        // existing type: Sprite / Text / SpriteWithText
      val layerIndex: Int,       // primary ordering: style layer order
      val inLayerPriority: Int   // secondary ordering within layer (existing symbolZOrder)
    )

  • PlacedSymbols: computed when viewport is changed ;
    Something like this:

    data class PlacedSymbol(
      val candidate: SymbolCandidate,
      val canvasX: Float,
      val canvasY: Float,
      val angle: Float
    )

• LabelPlacement also needs val layerIndex: Int and val inLayerPriority: Int (rename from priority).

SymbolPlacementEngine

API sketch

interface SymbolPlacementEngine {
    fun putTileCandidates(tileKey: String, candidates: List<SymbolCandidate>)
    fun removeTile(tileKey: String)
    fun clear()

    fun place(
        viewportInfo: ViewportInfo,
        visibleTiles: VisibleTiles,
        zoomPRState: ZoomPanRotateState // or equivalent info needed to project
    ): List<PlacedSymbol>
}

Placement algorithm (run on every viewport change)

1. Gather candidates from visible tiles (cache lookup only).
2. Filter by zoom/style if needed (cheap checks only).
3. Sort by (layerIndex desc, inLayerPriority desc, stableId asc).
4. Clear CollisionDetector (new frame).
5. For each candidate in order: compute screen anchor + size; rebuild screen-space LabelPlacement with OBB; tryPlaceLabel(); if placed, add to draw list.
6. Return placed Symbols to SymbolState for rendering.

VectorLayer

Split flows: production vs placement

• Production trigger: tile visibility changes and tile data load/update. Produce candidates per tile once, then cache them in SymbolPlacementEngine.
• Placement trigger: viewport changes (pan/zoom/rotate). Call placementEngine.place(...) and update SymbolState with the placed symbols only.

This is a rough plan. What do you thingk?

[p-lr]

I would rather focus on one thing at a time, because the task is fairly complex. There are two main challenges :

1. Have a working algorithm for collision detection
2. Optimize performances

We can start with the collision detection challenge. No cache, everything is re-computed when we scroll or zoom (i.e on every single viewport changes).
Doing this way, we can focus on the algorithm itself and fix all issues we'll find along the way. When the algorithm proves to be correct, then we can focus on architecture changes to optimize performances (caching, etc.).

This approach would make me more confident.

[rkreienbuehl]

Sounds like a reasonable plan. What I totally forgot was that I actually didn't add the collision detection at all when porting into the library. I added it now, and collision detection seems to work ok, but layer priority needs definitely to be added, as described in my last message.

What do you think?

[p-lr]

That makes sense. With collision detection, I'm expecting the number of symbols in the rendering pipeline to drop dramatically. I'll take the time to test this and let you know.
If you feel that you don't need my feedback to start working on layer priority, please go ahead.

P.S: I will most probably have to resume my work on the master branch. As the library isn't 100% on par with the android native version, and I'll need a complete kmp version in the upcoming months.

Thanks for your commitment.

[rkreienbuehl]

Thats ok, I will work on it step by step towards introducing a new SymbolPlacementEngine. I'll post updates here 👍🏻