providers.json num_ctx setting has no effect on Ollama and causes empty responses on CPU setups

[mauserret]

On CPU hardware, the evaluation step with the default model gemma4:latest returns an empty response, causing json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0).

Root cause: providers.json sets "extra_body": {"num_ctx": 32768} for the Ollama provider, intending to raise the context window for evaluation calls. However, Ollama's OpenAI-compatible /v1 endpoint does not support setting context size via the request body at all, this is a documented Ollama limitation, not something specific to this repo's request-building code:

https://docs.ollama.com/api/openai-compatibility

"The OpenAI API does not have a way of setting the context size for a model. If you need to change the context size, create a Modelfile... then use ollama create mymodel."

As a result, ollama ps shows CONTEXT 4096 during a run regardless of the extra_body setting, and large evaluation prompts (full resume text + role criteria + GitHub project data) get truncated, causing empty LLM responses and JSONDecodeError.

Separately, on one run before applying the num_ctx workaround below, the evaluation call also hit the hardcoded 300-second HTTP timeout in models.py and failed with ReadTimeout. I locally raised this to 900 seconds to work around it, but it's still hardcoded and may be worth making configurable, especially for slower CPU-only setups.

Environment:

• OS: Windows 11
• Python: 3.13
• hiring-agent commit: main (as of Aug 11, 2026)
• Model: gemma4:latest via Ollama, CPU-only (no GPU)

Steps to reproduce:

1. ollama pull gemma4:latest
2. Set DEFAULT_MODEL=gemma4:latest in .env
3. python score.py ./resume/sample.pdf --role software_engineering_intern
4. Evaluation step fails with JSONDecodeError on empty response, or occasionally with ReadTimeout if generation runs long

Workaround found:

1. Create a custom Ollama model tag with an explicit num_ctx via Modelfile:
   ```
   FROM gemma4:latest
   PARAMETER num_ctx 8192
   ```
   ```
   ollama create gemma4-8k -f Modelfile
   ```
2. Add the new model to providers.json under the ollama provider's models list:
   ```json
   "gemma4-8k": { "temperature": 0.1, "top_p": 0.9 }
   ```
3. Set DEFAULT_MODEL=gemma4-8k in .env

This works because it bakes the context size directly into the model tag, which is the only way Ollama supports changing context size when using its OpenAI-compatible endpoint. After this workaround, evaluation completes successfully and produces valid JSON.

Possible fix: since Ollama's native /api/chat endpoint does support per-request options.num_ctx (no Modelfile needed), the Ollama provider could use that endpoint instead of /v1/chat/completions to make num_ctx actually configurable at runtime, rather than requiring users to pre-bake context size into a custom model tag. Making the HTTP timeout in models.py configurable (env var or providers.json) would also help on slower CPU-only setups.

Traceback: JSONDecodeError (empty response)

```text
Traceback (most recent call last):
File "...\hiring-agent\score.py", line 412, in
main(args.pdf_path, role)
~~~~^^^^^^^^^^^^^^^^^^^^^
File "...\hiring-agent\score.py", line 323, in main
score = _evaluate_resume(resume_data, role, evaluation_model, github_data)
File "...\hiring-agent\score.py", line 177, in evaluate_resume
evaluation_result = evaluator.evaluate_resume(resume_text)
File "...\hiring-agent\evaluator.py", line 79, in evaluate_resume
evaluation_dict = json.loads(response_text)
File "...\Python313\Lib\json_init.py", line 346, in loads
return _default_decoder.decode(s)
~~~~~~~~~~~~~~~~~~~~~~~^^^
File "...\Python313\Lib\json\decoder.py", line 345, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "...\Python313\Lib\json\decoder.py", line 363, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
```

Traceback: ReadTimeout (single occurrence, on a different run)

```text
Traceback (most recent call last):
File "...\venv\Lib\site-packages\urllib3\connectionpool.py", line 534, in _make_request
response = conn.getresponse()
File "\venv\Lib\site-packages\urllib3\connection.py", line 571, in getresponse
httplib_response = super().getresponse()
File "\Python313\Lib\http\client.py", line 1430, in getresponse
response.begin()
~~~~~~~~~~~~~~^^
File "\Python313\Lib\http\client.py", line 331, in begin
version, status, reason = self._read_status()
~~~~~~~~~~~~~~~~~^^
File "\Python313\Lib\http\client.py", line 292, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "\Python313\Lib\socket.py", line 719, in readinto
return self._sock.recv_into(b)
~~~~~~~~~~~~~~~~~~~~^^^
TimeoutError: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "\venv\Lib\site-packages\requests\adapters.py", line 645, in send
resp = conn.urlopen(
method=request.method,
...<9 lines>...
chunked=chunked,
)
File "\venv\Lib\site-packages\urllib3\connectionpool.py", line 842, in urlopen
retries = retries.increment(
method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
)
File "\venv\Lib\site-packages\urllib3\util\retry.py", line 498, in increment
raise reraise(type(error), error, _stacktrace)
~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "\venv\Lib\site-packages\urllib3\util\util.py", line 39, in reraise
raise value
File "\venv\Lib\site-packages\urllib3\connectionpool.py", line 788, in urlopen
response = self._make_request(
conn,
...<10 lines>...
**response_kw,
)
File "\venv\Lib\site-packages\urllib3\connectionpool.py", line 536, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "\venv\Lib\site-packages\urllib3\connectionpool.py", line 367, in _raise_timeout
raise ReadTimeoutError(
self, url, f"Read timed out. (read timeout={timeout_value})"
) from err
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=11434): Read timed out. (read timeout=300)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "\score.py", line 412, in
main(args.pdf_path, role)
~~~~^^^^^^^^^^^^^^^^^^^^^
File "\score.py", line 323, in main
score = _evaluate_resume(resume_data, role, evaluation_model, github_data)
File "\score.py", line 177, in _evaluate_resume
evaluation_result = evaluator.evaluate_resume(resume_text)
File "\evaluator.py", line 73, in evaluate_resume
response = self.provider.chat(**chat_params, **kwargs)
File "\models.py", line 345, in chat
response = requests.post(url, json=body, headers=headers, timeout=300)
File "\venv\Lib\site-packages\requests\api.py", line 115, in post
return request("post", url, data=data, json=json, **kwargs)
File "\venv\Lib\site-packages\requests\api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "\venv\Lib\site-packages\requests\sessions.py", line 592, in request
resp = self.send(prep, **send_kwargs)
File "\venv\Lib\site-packages\requests\sessions.py", line 706, in send
r = adapter.send(request, **kwargs)
File "\venv\Lib\site-packages\requests\adapters.py", line 691, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='localhost', port=11434): Read timed out. (read timeout=300)
```

[mauserret]

I'd like to work on this. Planning to look into switching the Ollama provider to use the native /api/chat endpoint so num_ctx can actually be set at runtime via options, instead of requiring a pre-baked Modelfile. Will also look at making the HTTP timeout configurable. Will open a PR once I have something working.