update Ch to add JSON support

[ruslandoga]

Ecto part of plausible/ch#262

[arrowcircle]

Hey! Thanks for the work on this PR.
I tried to use it, but I get error

[error] ** (Ch.Error) Code: 117. DB::Exception: Unknown type code: 0x61: (at row 1)
: While executing BinaryRowInputFormat. (INCORRECT_DATA) (version 25.4.1.2934 (official build))

    (ecto_sql 3.13.2) lib/ecto/adapters/sql.ex:1098: Ecto.Adapters.SQL.raise_sql_call_error/1
    (ecto_ch 0.7.1) lib/ecto/adapters/clickhouse/schema.ex:33: Ecto.Adapters.ClickHouse.Schema.insert_all/8
    (ecto 3.13.2) lib/ecto/repo/schema.ex:84: Ecto.Repo.Schema.do_insert_all/7

In the repo settings I have:

settings: [
      enable_json_type: 1,
      output_format_binary_write_json_as_string: 1,
      input_format_binary_read_json_as_string: 1
    ]

Schema is:

defmodule TokenInfoSchema do
    @moduledoc false
    use Ecto.Schema

    @primary_key false
    schema "token_infos" do
      field(:mint, Ch, type: "String")
      field(:data, Ch, type: "JSON", source: :data)
      field(:created_at, Ch, type: "DateTime")
    end
  end

Error is happening when I try to insert deeply nested json, here is the query log:

[debug] QUERY ERROR db=77.1ms idle=387.9ms
INSERT INTO "token_infos"("data","mint","created_at") FORMAT RowBinaryWithNamesAndTypes [%{"authorities" => [%{"address" => "2wmVCSfPxGPjrnMMn7rchp4uaeoTqN39mXFC2zhPdri9", "scopes" => ["full"]}], "burnt" => false, "compression" => %{"asset_hash" => "", "compressed" => false, "creator_hash" => "", "data_hash" => "", "eligible" => false, "leaf_id" => 0, "seq" => 0, "tree" => ""}, "content" => %{"$schema" => "https://schema.metaplex.com/nft1.0.json", "files" => [%{"cdn_uri" => "https://cdn.helius-rpc.com/cdn-cgi/image//https://raw.githubusercontent.com/solana-labs/token-list/main/assets/mainnet/EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v/logo.png", "mime" => "image/png", "uri" => "https://raw.githubusercontent.com/solana-labs/token-list/main/assets/mainnet/EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v/logo.png"}], "json_uri" => "", "links" => %{"image" => "https://raw.githubusercontent.com/solana-labs/token-list/main/assets/mainnet/EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v/logo.png"}, "metadata" => %{"name" => "USD Coin", "symbol" => "USDC"}}, "creators" => [], "grouping" => [], "id" => "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v", "interface" => "FungibleToken", "mutable" => true, "ownership" => %{"delegate" => nil, "delegated" => false, "frozen" => false, "owner" => "", "ownership_model" => "token"}, "royalty" => %{"basis_points" => 0, "locked" => false, "percent" => 0.0, "primary_sale_happened" => false, "royalty_model" => "creators", "target" => nil}, "supply" => nil, "token_info" => %{"decimals" => 6, "freeze_authority" => "7dGbd2QZcCKcTndnHcTL8q7SMVXAkp688NTQYwrRCrar", "mint_authority" => "BJE5MMbqXjVwjAF7oxwPYXnTXDyspzZyt4vwenNw5ruG", "price_info" => %{"currency" => "USDC", "price_per_token" => 0.999867}, "supply" => 8276375974708499, "symbol" => "USDC", "token_program" => "TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA"}}, "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v", ~N[2025-07-16 16:04:47], %{"authorities" => [%{"address" => "2RtGg6fsFiiF1EQzHqbd66AhW7R5bWeQGpTbv2UMkCdW", "scopes" => ["full"]}], "burnt" => false, "compression" => %{"asset_hash" => "", "compressed" => false, "creator_hash" => "", "data_hash" => "", "eligible" => false, "leaf_id" => 0, "seq" => 0, "tree" => ""}, "content" => %{"$schema" => "https://schema.metaplex.com/nft1.0.json", "files" => [%{"cdn_uri" => "https://cdn.helius-rpc.com/cdn-cgi/image//https://madlads.s3.us-west-2.amazonaws.com/images/8420.png", "mime" => "image/png", "uri" => "https://madlads.s3.us-west-2.amazonaws.com/images/8420.png"}, %{"cdn_uri" => "https://cdn.helius-rpc.com/cdn-cgi/image//https://arweave.net/qJ5B6fx5hEt4P7XbicbJQRyTcbyLaV-OQNA1KjzdqOQ/0.png", "mime" => "image/png", "uri" => "https://arweave.net/qJ5B6fx5hEt4P7XbicbJQRyTcbyLaV-OQNA1KjzdqOQ/0.png"}], "json_uri" => "https://madlads.s3.us-west-2.amazonaws.com/json/8420.json", "links" => %{"external_url" => "https://madlads.com", "image" => "https://madlads.s3.us-west-2.amazonaws.com/images/8420.png"}, "metadata" => %{"attributes" => [%{"trait_type" => "Gender", "value" => "Male"}, %{"trait_type" => "Type", "value" => "King"}, %{"trait_type" => "Expression", "value" => "Royal"}, %{"trait_type" => "Hat", "value" => "Mad Crown"}, %{"trait_type" => "Eyes", "value" => "Madness"}, %{"trait_type" => "Clothing", "value" => "Mad Armor"}, %{"trait_type" => "Background", "value" => "Royal Rug"}], "description" => "Fock it.", "name" => "Mad Lads #8420", "symbol" => "MAD", "token_standard" => "ProgrammableNonFungible"}}, "creators" => [%{"address" => "5XvhfmRjwXkGp3jHGmaKpqeerNYjkuZZBYLVQYdeVcRv", "share" => 0, "verified" => true}, %{"address" => "2RtGg6fsFiiF1EQzHqbd66AhW7R5bWeQGpTbv2UMkCdW", "share" => 100, "verified" => true}], "grouping" => [%{"group_key" => "collection", "group_value" => "J1S9H3QjnRtBbbuD4HjPV6RpRhwuk4zKbxsnCHuTgh9w"}], "id" => "F9Lw3ki3hJ7PF9HQXsBzoY8GyE6sPoEZZdXJBsTTD2rk", "interface" => "ProgrammableNFT", "mutable" => true, "ownership" => %{"delegate" => nil, "delegated" => false, "frozen" => true, "owner" => "D3ftM66SZMdbCHiV9wBAFxoqqA8ex76nJnmVLbGy6vwp", "ownership_model" => "single"}, "royalty" => %{"basis_points" => 420, "locked" => false, "percent" => 0.042, "primary_sale_happened" => true, "royalty_model" => "creators", "target" => nil}, "supply" => %{"edition_nonce" => 254, "print_current_supply" => 0, "print_max_supply" => 0}, "token_info" => %{"decimals" => 0, "freeze_authority" => "TdMA45ZnakQCBt5XUvm7ib2htKuTWdcgGKu1eUGrDyJ", "mint_authority" => "TdMA45ZnakQCBt5XUvm7ib2htKuTWdcgGKu1eUGrDyJ", "supply" => 1, "token_program" => "TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA"}}, "F9Lw3ki3hJ7PF9HQXsBzoY8GyE6sPoEZZdXJBsTTD2rk", ~N[2025-07-16 16:04:47]]

Any ideas why this happens?

[ruslandoga]

👋

Thank you for trying it out!

I added this insert as a test in 38eb62f and it seems to pass (including with clickhouse/clickhouse-server:25.4.1.2934 locally). Would you be able to share the code used to perform the insert?

[arrowcircle]

Hey! Thanks for the quick reply. Here is the code that fails

missing_tokens = [
      %{mint: "123", data: %{"name" => "Test", "nested" => %{"name" => "Test", "arr" => ["abc", "b=deb"]}}, created_at: NaiveDateTime.utc_now() |> NaiveDateTime.truncate(:second)},
      %{mint: "325", data: %{"name" => "Test", "nested" => %{"name" => "Test", "arr" => ["abc", "b=deb"]}}, created_at: NaiveDateTime.utc_now() |> NaiveDateTime.truncate(:second)}
    ]}

{inserted_count, _} = Repo.insert_all(TokenInfoSchema, missing_tokens)

with error:

[debug] QUERY ERROR db=70.7ms idle=1456.1ms
INSERT INTO "token_infos"("data","mint","created_at") FORMAT RowBinaryWithNamesAndTypes [%{"name" => "Test", "nested" => %{"arr" => ["abc", "b=deb"], "name" => "Test"}}, "123", ~N[2025-07-16 19:51:32], %{"name" => "Test", "nested" => %{"arr" => ["abc", "b=deb"], "name" => "Test"}}, "325", ~N[2025-07-16 19:51:32]]
[info] Sent 500 in 153ms
[error] ** (Ch.Error) Code: 117. DB::Exception: Unknown type code: 0x3a: (at row 1)
: While executing BinaryRowInputFormat. (INCORRECT_DATA) (version 25.4.1.2934 (official build))

    (ecto_sql 3.13.2) lib/ecto/adapters/sql.ex:1098: Ecto.Adapters.SQL.raise_sql_call_error/1
    (ecto_ch 0.7.1) lib/ecto/adapters/clickhouse/schema.ex:33: Ecto.Adapters.ClickHouse.Schema.insert_all/8
    (ecto 3.13.2) lib/ecto/repo/schema.ex:84: Ecto.Repo.Schema.do_insert_all/7

[ruslandoga]

Hm. Something else might be different, since that snippet also seems to work: 05e5dbc

[ruslandoga]

Ah, I was able to reproduce it by commenting out json_as_string options for the test repo:

Application.put_env(:ecto_ch, TestRepo,
  adapter: Ecto.Adapters.ClickHouse,
  database: "ecto_ch_test",
  show_sensitive_data_on_connection_error: true,
  settings: [
    enable_json_type: 1
    # output_format_binary_write_json_as_string: 1,
    # input_format_binary_read_json_as_string: 1
  ]
)

It resulted in the same error:

** (Ch.Error) Code: 117. DB::Exception: Unknown type code: 0x3a: (at row 1)
: While executing BinaryRowInputFormat. (INCORRECT_DATA) (version 25.6.3.116 (official build))

It seems like the settings option hasn't been applied correctly in your case somehow.

[arrowcircle]

Hm, thanks for the finding. I run ecto in a different way, I added these settings to the startup of the repo (instead of code definition file) and everything works!