[idea] Make Inference custom requests even more generic?

[radames]

Hi @coyotte508,

I was trying to use the @huggingface/inference custom Request/StreamingRequest as a client for Ollama (https://github.com/jmorganca/ollama/blob/main/docs/api.md#streaming-responses). It almost works, but it needs a couple of custom arguments and requires a model argument, that clashes with our internals.

This idea might be outside the scope of our client, but one could use it to consume any text streaming LLM and eventually switch between providers? wdyt?

[coyotte508]

I guess we could add a extraArgs parameter to the options argument, which could contain things like reserved arg names (model, accessToken, ...)

Feels a bit hacky but short of rewriting the API...

Would be called like this:

streamingRequest(..., {extraArgs: {model: ...}});

[radames]

That would work, even if it looks hacky. I suppose it would be a very unusual case. But it's also totally fine if this is out of scope.

[radames]

would this work?

	options?: Options & {
		/** When a model can be used for multiple tasks, and we want to run a non-default task */
		forceTask?: string | InferenceTask;
		/** To load default model if needed */
		taskHint?: InferenceTask;
		extraArgs?: Record<string, unknown>,
	},

[radames]

Also, I just learned that for their streaming response, they're using application/x-ndjson, which isn't really compatible with text/event-stream 😔