createrun fails?

[rkube]

Hi,
I'm trying to get MLFlowClient.jl working with a locally hosted mlflow instance.

The following code fails

using MLFlowClient
using Minio

s3_cfg = MinioConfig("http://192.168.50.119:9000"; username=ENV["AWS_ACCESS_KEY_ID"], password=ENV["AWS_SECRET_ACCESS_KEY"])

mlf = MLFlow("http://192.168.50.119:5000/api")
experiment_id = createexperiment(mlf, "julia-tests")
exprun = createrun(mlf, experiment_id)

with

julia> exprun = createrun(mlf, experiment_id)
ERROR: KeyError: key "outputs" not found
Stacktrace:
 [1] getindex
   @ ./dict.jl:477 [inlined]
 [2] Run(data::Dict{String, Any})
   @ MLFlowClient ~/source/repos/MLFlowClient.jl/src/types/run.jl:142
 [3] |>(x::Dict{String, Any}, f::Type{Run})
   @ Base ./operators.jl:926
 [4] createrun(instance::MLFlow, experiment_id::String; run_name::Missing, start_time::Missing, tags::Vector{Tag})
   @ MLFlowClient ~/source/repos/MLFlowClient.jl/src/services/run.jl:25
 [5] createrun(instance::MLFlow, experiment_id::String)
   @ MLFlowClient ~/source/repos/MLFlowClient.jl/src/services/run.jl:20
 [6] top-level scope
   @ REPL[17]:1

Looking into the HTTP response, the output field is missing https://mlflow.org/docs/latest/api_reference/rest-api.html#run

julia> result = MLFlowClient.mlfpost(mlf, "runs/create"; experiment_id=experiment_id)
Dict{String, Any} with 1 entry:
  "run" => Dict{String, Any}("data"=>Dict{String, Any}("tags"=>Any[Dict{String, Any}("key"=>"mlflow.runName", "value"=>"intrigued-bug-67…

julia> result["run"]
Dict{String, Any} with 3 entries:
  "data"   => Dict{String, Any}("tags"=>Any[Dict{String, Any}("key"=>"mlflow.runName", "value"=>"intrigued-bug-674")])
  "info"   => Dict{String, Any}("start_time"=>0, "artifact_uri"=>"s3://mlflow/2/e5a50e9b7bf448a993081562851cdc9f/artifacts", "lifecycle_…
  "inputs" => Dict{String, Any}()

How should this be handled? Just add an empty dict to "outputs" when the server doesn't send it?

[pebeto]

Hello @rkube.

Could you define which is the mlflow version you are using? I tried to ensure fully-compatible support with 3.2.0 for this client.

[rkube]

Ok, I just tested with 3.6.0 running in a local docker instance:

using Base64
using MLFlowClient
using JSON
using HTTP

encoded_credentials = Base64.base64encode("amdin:password1234")
mlf = MLFlow("http://localhost:5050/api", headers=Dict("Authorization" => "Basic $(encoded_credentials)"))

experiment_id = createexperiment(mlf, "julia-test")

apiuri = MLFlowClient.uri(mlf, "runs/create")
apiheaders = apiheaders = MLFlowClient.headers(mlf, ("Content-Type" => "application/json") |> Dict)
body = JSON.json(Dict(:experiment_id => experiment_id))

response = HTTP.post(apiuri, apiheaders, body)

and get

julia> response = HTTP.post(apiuri, apiheaders, body)
HTTP.Messages.Response:
"""
HTTP/1.1 200 OK
date: Sun, 30 Nov 2025 19:47:34 GMT
server: uvicorn
content-type: application/json
content-length: 612
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN

{
  "run": {
    "info": {
      "run_uuid": "865fb0d74f3547a5ac717616df95df44",
      "experiment_id": "401487142675048715",
      "run_name": "rare-sheep-355",
      "user_id": "",
      "status": "RUNNING",
      "start_time": 0,
      "artifact_uri": "s3://mlflow/artifacts/401487142675048715/865fb0d74f3547a5ac717616df95df44/artifacts",
      "lifecycle_stage": "active",
      "run_id": "865fb0d74f3547a5ac717616df95df44"
    },
    "data": {
      "tags": [
        {
          "key": "mlflow.runName",
          "value": "rare-sheep-355"
        }
      ]
    },
    "inputs": {},
    "outputs": {}
  }
}"""

When I run it against an instance running remotely, I get

julia> response = HTTP.post(apiuri, apiheaders, body)
HTTP.Messages.Response:
"""
HTTP/1.1 200 OK
date: Sun, 30 Nov 2025 19:51:56 GMT
server: uvicorn
content-type: application/json
content-length: 555
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN

{
  "run": {
    "info": {
      "run_uuid": "bf1d6ff15ac14c4abf2fec877a4e5ffa",
      "experiment_id": "41",
      "run_name": "useful-stoat-824",
      "user_id": "",
      "status": "RUNNING",
      "start_time": 0,
      "artifact_uri": "s3://mlflow/41/bf1d6ff15ac14c4abf2fec877a4e5ffa/artifacts",
      "lifecycle_stage": "active",
      "run_id": "bf1d6ff15ac14c4abf2fec877a4e5ffa"
    },
    "data": {
      "tags": [
        {
          "key": "mlflow.runName",
          "value": "useful-stoat-824"
        }
      ]
    },
    "inputs": {}
  }
}"""

The python client handles the missing outputs graciously.

[rkube]

I'm also seeing that HTTP.post(apiuri, apiheaders, body) returns json with "experiment_id": "401487142675048715". Has anyone seen that before? Typically experiment_id would be a small integer.

[pebeto]

I was wondering how to solve this kind of issues. Some problems are originated from how poorly designed is the mlflow REST API. @ablaom, now talking about versioning, what if we change this project versioning to be paired with the utility version? With that we can ensure total support to old and latest versions.

[rkube]

Right, version matching would be cool. One way to approach this is to check the routes documented in the REST API definition by pinging them and check for a 200 OK return in the header:

This is a snippet from the mlflow log when I run the unit tests:

mlflow-1          | INFO:     192.168.65.1:22598 - "HEAD //health HTTP/1.1" 200 OK
mlflow-1          | INFO:     192.168.65.1:57268 - "POST /api/2.0/mlflow/experiments/create HTTP/1.1" 200 OK
mlflow-1          | INFO:     192.168.65.1:57268 - "POST /api/2.0/mlflow/runs/create HTTP/1.1" 200 OK
mlflow-1          | INFO:     192.168.65.1:57268 - "POST /api/2.0/mlflow/registered-models/create HTTP/1.1" 200 OK
mlflow-1          | INFO:     192.168.65.1:57268 - "POST /api/2.0/mlflow/model-versions/create HTTP/1.1" 200 OK
mlflow-1          | INFO:     192.168.65.1:57268 - "POST /api/2.0/mlflow/registered-models/alias HTTP/1.1" 200 OK
mlflow-1          | INFO:     192.168.65.1:57268 - "GET /api/2.0/mlflow/registered-models/alias?name=missy&alias=gala HTTP/1.1" 200 OK
mlflow-1          | INFO:     192.168.65.1:57268 - "DELETE /api/2.0/mlflow/registered-models/delete?name=missy HTTP/1.1" 200 OK
mlflow-1          | INFO:     192.168.65.1:57268 - "POST /api/2.0/mlflow/experiments/delete HTTP/1.1" 200 OK

So something like

function test_route(route_name::String="mlflow/registered-models/create")
    resp = HTTP.request("HEAD", "$(MLFLOW_API_ROOT)/route_name, readtimeout=10)
    return resp.status == 200 
end