SAM on JS? (Segment Anything Model)

[hpssjellis]

I have a JS file that does segmentation but it does not run SAM.

https://hpssjellis.github.io/my-examples-of-huggingfacejs/public/hug11-15b.html

SAM (Segment Anything Model) is now on Huggingface https://huggingface.co/docs/transformers/main/en/model_doc/sam

The main model is at facebook/sam-vit-huge

but when I try that model get errors different from the working default model facebook/detr-resnet-50-panoptic

The errors I am getting are

Uncaught (in promise) Error: Task not found for this model
    at HfInference.request (inference.js:125:15)
    at async HfInference.imageSegmentation (inference.js:78:12)
    at async myHugginFaceLoad (hug11-15b.html:26:36)

I am busy with some other coding. If not, I could probably solve this issue, but does anyone have any suggestions of what the problem is? It is most likely the model input or output parameters are slightly different.

If anyone gets my model working please post your version of my code here.

[coyotte508]

Hi @hpssjellis

It looks like an image-to-image model, see #157 and huggingface/api-inference-community#223

It's not yet added to huggingface JS, I'll also look into getting the model the appropriate task

[coyotte508]

Ah it looks like it's a "mask generation" task, not image to image. Will definitely be supported in the near future on HF.

By the way cc @radames , you can make it run in your browser directly with transformers.js I think and @radames did that if I'm not mistaken?

[radames]

hi @hpssjellis , is SAM is not yet integrated on the API inference and it be there soon, we'll add another task here mask-generation
On another note, I've been working on a example using the SAM embeddings with an ONNX quantized model to predict the masks on the front-end, it also uses our huggingface.js to run the request.

[felri]

@radames any updates of SAM in huggingfacejs?

[smyja]

bump