Could it be possible to extend the plugin to allow switching to a different buffer with a single (or almost single) key press?
I imagine that instead of displaying buffer numbers the plugin could display labels (from the set provided by the users) next to each buffer item. Typing j would for example open first buffer, but using a modifier like SHIFT+j would open it in the vertical split.