Fix llama-server and llama-bench

[wine99]

llama-server runs with default params (does not support -np >1)
llama-bench runs with -fa 1

[cavusmustafa]

I wonder if there is another way to identify different sequences rather than reading kv_cache state. If there is a way to identify, how about creating a unique inference request for each sequence? So, instead of caching inference requests with cgraph pointer, we can create an id out of cgraph+seq_id ? If this is possible, we don't need to read the states as every inference request will manage its own variables anyways.

[cavusmustafa]

How about doing something like this to avoid multiple getenv calls?

const std::string& getDevice() {
        static const std::string device_str= [] {
            const char* device_env = std::getenv("GGML_OPENVINO_DEVICE");
            return device_env ? std::string(device_env) : "CPU";
        }();
        return device_str;
}
.
.
.
static std::string device =  getDevice();

[cavusmustafa]

Why do we need to check this for every iteration instead of checking once before compile_model?

[cavusmustafa]

In this case are we deleting the previous kv_cache completely?