Context Menu Actions (Document, Refactor, Find Problems, Explain for complex code) return garbage or no response with local LLMs (LM Studio)

[pmilewski]

Problem Summary

When using the CodeGPT extension with local Large Language Models (LLMs) served via LM Studio (tested with qwen2.5-coder:14b), several actions initiated from the VS Code right-click context menu either return malformed output (<|im_start|>) or no response at all. In contrast, the equivalent slash commands (/Document, /Refactor, /Fix, /Explain, /Comment, /UnitTest, /Debug) in the CodeGPT chat panel consistently work correctly and provide high-quality output.

Reproducibility

This issue is consistently reproducible.

Environment

• VS Code Version: 1.101.2
• CodeGPT Extension Version: 3.12.107
• Operating System: macOS Sequoia 15.5 (24F74)
• LM Studio Version: 0.3.17
• Local LLM Model: qwen2.5-coder:14b (specifically qwen2.5-coder-14b-GGUF/qwen2.5-coder-14b.gguf)
• LM Studio Server Settings:
  • Context Length: 8096 (Model supports up to 32768 tokens)
  • GPU Offload: 48 / 48
  • CPU Thread Pool Size: 8
  • Evaluation Batch Size: 512

Steps to Reproduce

1. Ensure CodeGPT is configured to use LM Studio as its provider and the qwen2.5-coder:14b model is loaded and running in LM Studio.
2. Create a new Python file (e.g., test_code_gpt.py) in VS Code.
3. Use the following simple factorial function for initial tests:

   def factorial(n):
       if n == 0:
           return 1
       else:
           return n * factorial(n-1)

Test Context Menu Actions (Issue Occurs Here)

• "CodeGPT: Document":

  • Select the factorial function.
  • Right-click -> CodeGPT -> CodeGPT: Document.
  • Expected: A generated docstring appears in the CodeGPT chat panel.
  • Actual: <|im_start|> <|im_start|> or similar malformed output appears in the chat panel.

• "CodeGPT: Refactor":

  • Select the factorial function.
  • Right-click -> CodeGPT -> CodeGPT: Refactor.
  • Expected: A refactored version of the code appears in the CodeGPT chat panel.
  • Actual: The command appears in the chat, but no response from the model follows. The chat remains ready for next input.

• "CodeGPT: Explain" (for complex code):

  • Replace content of test_code_gpt.py with:

    def calculate_circle_area_and_circumference(radius):
        if radius < 0:
            print("Error: Radius cannot be negative.")
            return None, None
        
        area = 3.14159 * radius * radius
        circumference = 2 * 3.14159 * radius
        
        print(f"For radius {radius}: Area = {area}, Circumference = {circumference}")
        return area, circumference

  • Select the calculate_circle_area_and_circumference function.
  • Right-click -> CodeGPT -> CodeGPT: Explain.
  • Expected: An explanation of the code appears.
  • Actual: <|im_start|> <|im_start|> or similar malformed output appears in the chat panel. (Note: For the simple factorial function, this action does work correctly.)

• "CodeGPT: Find Problems" (for complex code with issues):

  • Replace content of test_code_gpt.py with:

    def divide(a, b):
        # Deliberate error: missing zero division handling
        return a / b
    
    def greet(name):
        # Deliberate syntax error: not an f-string
        print("Hello, " + name + "!")
    
    def calculate_average(numbers):
        total = 0
        for num in numbers:
            total += num
        # Deliberate logical error: division by list length without checking if list is empty
        return total / len(numbers)

  • Select all three functions.
  • Right-click -> CodeGPT -> CodeGPT: Find Problems.
  • Expected: An analysis of problems/bugs in the code.
  • Actual: <|im_start|> <|im_start|> or similar malformed output appears in the chat panel.

Test Slash Commands (Workaround / Correct Behavior)

• For each problematic scenario above (Document, Refactor, Explain complex, Find Problems), perform the equivalent action using the slash command in the CodeGPT chat panel after selecting the code.
• Example (for Document): Select code -> Open CodeGPT Chat -> type /Document -> Press Enter.
• Observed: All slash commands (/Document, /Refactor, /Explain, /Fix, /UnitTest, /Comment, /Debug) work correctly and provide high-quality, relevant output, regardless of code complexity.

Expected Behavior

All context menu actions (CodeGPT: Document, CodeGPT: Refactor, CodeGPT: Explain, CodeGPT: Find Problems) should function correctly and provide relevant output from the local LLM, similar to how their corresponding slash commands do.

Additional Notes

• CodeGPT: Unit Test from the context menu does work correctly.
• The model itself (qwen2.5-coder:14b) demonstrates strong capabilities when the prompt is correctly received (via slash commands), suggesting the issue lies in how the context menu actions construct and send the prompt to LM Studio.

[pmilewski]

Update: Further testing with a non-local model (Google Gemini 2.5 Pro) confirms hypothesis

Hello,

I'd like to provide an important update based on further testing conducted with a non-local LLM, Google Gemini 2.5 Pro, to help pinpoint the root cause of the described issues.

---

Test Setup for Non-Local Model:

• Provider: Google Gemini
• Model: Gemini 2.5 Pro
• Configuration: Connected successfully to Gemini API (no explicit API key prompt, model responded to initial greeting).

---

Summary of Repeated Tests (Context Menu Actions) with Google Gemini 2.5 Pro:

I re-tested the specific context menu actions that previously failed or produced malformed output with the local LM Studio setup (qwen2.5-coder:14b):

1. CodeGPT: Document (from context menu):

   • Result with Gemini 2.5 Pro: Worked perfectly. The model generated a correct docstring for the factorial function and even added input validation.
   • Previous result with LM Studio: Failed, returned <|im_start|>.

2. CodeGPT: Refactor (from context menu):

   • Result with Gemini 2.5 Pro: Worked perfectly. The model provided a detailed architectural analysis, identified scalability/maintainability/extensibility limitations, and proposed a refactoring plan.
   • Previous result with LM Studio: Failed, no response.

3. CodeGPT: Explain (for complex code from context menu):

   • Result with Gemini 2.5 Pro: Worked perfectly. The model generated a detailed and clear explanation of the complex calculate_circle_area_and_circumference function, including suggestions for improvement.
   • Previous result with LM Studio: Failed, returned <|im_start|>.

4. CodeGPT: Find Problems (for complex code with issues from context menu):

   • Result with Gemini 2.5 Pro: Worked perfectly. The model not only correctly identified the deliberate bugs but also provided the corrected code with detailed explanations of the changes made (e.g., handling zero division, f-string conversion, handling empty lists, adding docstrings).
   • Previous result with LM Studio: Failed, returned <|im_start|>.

---

Conclusion from Non-Local Model Testing:

Based on these results, it is clear that:

• The issues with the Document, Refactor, Explain (for complex code), and Find Problems context menu actions are NOT internal bugs of the CodeGPT extension itself.
• The problems are specific to the compatibility or prompt interpretation between the CodeGPT extension and the local LM Studio server/qwen2.5-coder:14b model when these particular context menu actions are used.
• The qwen2.5-coder:14b model itself is capable of performing these tasks (as demonstrated by the successful execution via slash commands), but the way CodeGPT formats and sends requests from the context menu seems to be incompatible with LM Studio's handling or the model's specific prompt template expectations for these actions.

This information should help narrow down the investigation significantly.

[CammRios]

Hey there!
Thank you for reaching out. I'm sorry to hear you're facing issues with our service.
We are grateful you brought this to our attention. We're currently looking into the issue and really appreciate your feedback, as it helps us improve the experience for everyone.
Our team is on it, and we’ll keep you updated as soon as we have news or when it’s been resolved. Thanks again for your patience!