LLM test case to exercise different devices.

[acbits]

git lfs install 
git clone https://huggingface.co/Unbabel/gec-t5_small

Run the syntax_check.py(pasted below) like this by copying the python file in the parent directory that contains the cloned git repo.

#!/usr/bin/env python3

# Note: Works with CPU, but not with GPU using OpenCL provided by MESA 24.1.6
#

import os
import sys
import pytorch_ocl
import torch

from transformers import T5ForConditionalGeneration, T5Tokenizer

if len(sys.argv) < 4:
    print("Usage: <device> <modelname> <sentence>")
    sys.exit(1)

device_name = sys.argv[1]
model_name = sys.argv[2]
sentence = sys.argv[3]

target_device = torch.device(device_name)

model_dir = os.path.dirname(sys.argv[0]) + os.sep + model_name

model = T5ForConditionalGeneration.from_pretrained(model_dir)
model = model.to(target_device)

print("Model loaded")

tokenizer = T5Tokenizer.from_pretrained(model_name)

max_len = 512

print("Tokenizer initialized")

tokenized_sentence = tokenizer(sentence,
                               max_length=max_len,
                               truncation=True,
                               padding='max_length',
                               return_tensors='pt').to(target_device)

outputs = model.generate(
        input_ids=tokenized_sentence.input_ids,
        attention_mask=tokenized_sentence.attention_mask,
        max_length=max_len,
        num_beams=5,
        early_stopping=True)

corrected_sentence = tokenizer.decode(
    outputs[0],
    skip_special_tokens=True,
    clean_up_tokenization_spaces=True)

print(corrected_sentence)

$ ./syntax_check.py cpu t5-small/ 'gec: this sentence has correct'
Model loaded
You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
Tokenizer initialized
This sentence is correct.

$ export RUSTICL_ENABLE=radeonsi
$ ./syntax_check.py ocl:0 t5-small/ 'gec: this sentence has correct'
Accessing device #0:AMD Radeon RX 7600 (radeonsi, navi33, LLVM 18.1.6, DRM 3.54, 6.6.2) on rusticl
Model loaded
You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
Tokenizer initialized
/home/user1/.local/lib/python3.12/site-packages/transformers/pytorch_utils.py:343: UserWarning: The operator 'aten::isin.Tensor_Tensor_out' is not currently supported on the ocl backend. Please open an issue at for requesting support https://github.com/artyom-beilis/pytorch_dlprim/issues (Triggered internally at /home/artik/Projects/dlprim_build/pytorch_dlprim/src/tensor_ops.cpp:415.)
  return torch.isin(elements, test_elements)
<snipped>

[artyom-beilis]

1st of all use ROCm OpenCL drivers

I tested your example on AMD rx6600xt with ROCm drivers and on Intel A380 and it completed.
Lots of operators are not implemented but returned result is similar to CPU. I also tested on rocm pytorch variant same results.

So I suggest switch to ROCm opencl drivers instead of rusticl - they are buggy.

BTW I don't get your output I get something like ff. f. f. f. f. f. f. - same on their own example ft ff - do you know why?
I'm not experienced with LLMs

[acbits]

1st of all use ROCm OpenCL drivers

I tested your example on AMD rx6600xt with ROCm drivers and on Intel A380 and it completed. Lots of operators are not implemented but returned result is similar to CPU. I also tested on rocm pytorch variant same results.

So I suggest switch to ROCm opencl drivers instead of rusticl - they are buggy.

BTW I don't get your output I get something like ff. f. f. f. f. f. f. - same on their own example ft ff - do you know why? I'm not experienced with LLMs

Not sure honestly. I have seen such reports of gibberish outputs with llama.cpp project with certain models + back end combinations.