Skip to content

Adds custom inference service API docs #4852

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 16 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions specification/_doc_ids/table.csv
Original file line number Diff line number Diff line change
Expand Up @@ -358,6 +358,7 @@ inference-api-put-anthropic,https://www.elastic.co/docs/api/doc/elasticsearch/op
inference-api-put-azureaistudio,https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-put-azureaistudio,https://www.elastic.co/guide/en/elasticsearch/reference/8.18/infer-service-azure-ai-studio.html,
inference-api-put-azureopenai,https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-put-azureopenai,https://www.elastic.co/guide/en/elasticsearch/reference/8.18/infer-service-azure-openai.html,
inference-api-put-cohere,https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-put-cohere,https://www.elastic.co/guide/en/elasticsearch/reference/8.18/infer-service-cohere.html,
inference-api-put-custom,https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-put-custom,https://www.elastic.co/guide/en/elasticsearch/reference/8.19/infer-service-custom.html,
inference-api-put-deepseek,https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-put-deepseek,https://www.elastic.co/guide/en/elasticsearch/reference/8.18/infer-service-deepseek.html,
inference-api-put-eis,https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-put-eis,,
inference-api-put-elasticsearch,https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-put-elasticsearch,https://www.elastic.co/guide/en/elasticsearch/reference/8.18/infer-service-elasticsearch.html,
Expand Down
35 changes: 35 additions & 0 deletions specification/_json_spec/inference.put_custom.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
{
"inference.put_custom": {
"documentation": {
"url": "https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-put-custom",
"description": "Configure a custom inference endpoint"
},
"stability": "stable",
"visibility": "public",
"headers": {
"accept": ["application/json"],
"content_type": ["application/json"]
},
"url": {
"paths": [
{
"path": "/_inference/{task_type}/{custom_inference_id}",
"methods": ["PUT"],
"parts": {
"task_type": {
"type": "string",
"description": "The task type"
},
"custom_inference_id": {
"type": "string",
"description": "The inference Id"
}
}
}
]
},
"body": {
"description": "The inference endpoint's task and service settings"
}
}
}
130 changes: 130 additions & 0 deletions specification/inference/_types/CommonTypes.ts
Original file line number Diff line number Diff line change
Expand Up @@ -758,6 +758,136 @@ export class CohereTaskSettings {
truncate?: CohereTruncateType
}

export class CustomServiceSettings {
/**
* Specifies the HTTPS header parameters – such as `Authentication` or `Contet-Type` – that are required to access the custom service.
* For example:
* ```
* "headers":{
* "Authorization": "Bearer ${api_key}",
* "Content-Type": "application/json;charset=utf-8"
* }
* ```
*/
headers?: UserDefinedValue
/**
* The request configuration object.
*/
request: CustomRequestParams
/**
* The response configuration object.
*/
response: CustomResponseParams
/**
* Specifies secret parameters, like `api_key` or `api_token`, that are required to access the custom service.
* For example:
* ```
* "secret_parameters":{
* "api_key":"<api_key>"
* }
* ```
*/
secret_parameters: UserDefinedValue
/**
* The URL endpoint to use for the requests.
*/
url?: string
}

export class CustomRequestParams {
/**
* The body structure of the request. It requires passing in the string-escaped result of the JSON format HTTP request body.
* For example:
* ```
* "request":{
* "content":"{\"input\":${input}}"
* }
* ```
* > info
* > The content string needs to be a single line except using the Kibana console.
*/
content: string
}

export class CustomResponseParams {
/**
* Specifies the path to the error message in the response from the custom service.
* For example:
* ```
* "response": {
* "error_parser": {
* "path": "$.error.message"
* }
* }
* ```
*/
error_parser: UserDefinedValue
/**
* Specifies the JSON parser that is used to parse the response from the custom service.
* Different task types require different json_parser parameters.
* For example:
Copy link
Contributor Author

@szabosteve szabosteve Jul 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jonathan-buttner Do you think we should specify a JsonParser class for each task type, or is this list sufficient?

* ```
* # text_embedding
* "response":{
* "json_parser":{
* "text_embeddings":"$.result.embeddings[*].embedding"
* }
* }
*
* # sparse_embedding
* "response":{
* "json_parser":{
* "token_path":"$.result[*].embeddings[*].token",
* "weight_path":"$.result[*].embeddings[*].weight"
* }
* }
*
* # rerank
* "response":{
* "json_parser":{
* "reranked_index":"$.result.scores[*].index", // optional
* "relevance_score":"$.result.scores[*].score",
* "document_text":"xxx" // optional
* }
* }
*
* # completion
* "response":{
* "json_parser":{
* "completion_result":"$.result.text"
* }
* }
*/
json_parser: UserDefinedValue
}

export enum CustomTaskType {
text_embedding,
sparse_embedding,
rerank,
completion
}

export enum CustomServiceType {
custom
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jonathan-buttner Should the ServiceType be custom whenever it's specified for this service type? Or can it be anything, for example custom-model?

}

export class CustomTaskSettings {
/**
* Specifies parameters that are required to run the custom service. The parameters depend on the model your custom service uses.
* For example:
* ```
* "task_settings":{
* "parameters":{
* "input_type":"query",
* "return_token":true
* }
* }
* ```
*/
parameters?: UserDefinedValue
}

export class EisServiceSettings {
/**
* The name of the model to use for the inference task.
Expand Down
34 changes: 22 additions & 12 deletions specification/inference/_types/Services.ts
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ import {
TaskTypeAzureAIStudio,
TaskTypeAzureOpenAI,
TaskTypeCohere,
TaskTypeCustom,
TaskTypeDeepSeek,
TaskTypeElasticsearch,
TaskTypeELSER,
Expand Down Expand Up @@ -75,18 +76,6 @@ export class InferenceEndpointInfo extends InferenceEndpoint {
*/
task_type: TaskType
}

export class InferenceEndpointInfoJinaAi extends InferenceEndpoint {
/**
* The inference Id
*/
inference_id: string
/**
* The task type
*/
task_type: TaskTypeJinaAi
}

export class InferenceEndpointInfoAlibabaCloudAI extends InferenceEndpoint {
/**
* The inference Id
Expand Down Expand Up @@ -153,6 +142,16 @@ export class InferenceEndpointInfoCohere extends InferenceEndpoint {
task_type: TaskTypeCohere
}

export class InferenceEndpointInfoCustom extends InferenceEndpoint {
/**
* The inference Id
*/
inference_id: string
/**
* The task type
*/
task_type: TaskTypeCustom
}
export class InferenceEndpointInfoDeepSeek extends InferenceEndpoint {
/**
* The inference Id
Expand Down Expand Up @@ -219,6 +218,17 @@ export class InferenceEndpointInfoHuggingFace extends InferenceEndpoint {
task_type: TaskTypeHuggingFace
}

export class InferenceEndpointInfoJinaAi extends InferenceEndpoint {
/**
* The inference Id
*/
inference_id: string
/**
* The task type
*/
task_type: TaskTypeJinaAi
}

export class InferenceEndpointInfoMistral extends InferenceEndpoint {
/**
* The inference Id
Expand Down
7 changes: 7 additions & 0 deletions specification/inference/_types/TaskType.ts
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,13 @@ export enum TaskTypeCohere {
completion
}

export enum TaskTypeCustom {
text_embedding,
sparse_embedding,
rerank,
completion
}

export enum TaskTypeDeepSeek {
completion,
chat_completion
Expand Down
78 changes: 78 additions & 0 deletions specification/inference/put_custom/PutCustomRequest.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

import { RequestBase } from '@_types/Base'
import { Id } from '@_types/common'
import {
CustomServiceSettings,
CustomServiceType,
CustomTaskSettings,
CustomTaskType
} from '@inference/_types/CommonTypes'
import { InferenceChunkingSettings } from '@inference/_types/Services'

/**
* Create a custom inference endpoint.
*
* You can create an inference endpoint to perform an inference task with a custom model that supports the HTTP format.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jonathan-buttner Please suggest an alternative description if you think this is not sufficient. I tried to come up with something that is meaningful to me based on my limited knowledge.

* @rest_spec_name inference.put_custom
* @availability stack since=8.13.0 stability=stable visibility=public
* @availability serverless stability=stable visibility=public
* @cluster_privileges manage_inference
* @doc_id inference-api-put-custom
*/
export interface Request extends RequestBase {
urls: [
{
path: '/_inference/{task_type}/{custom_inference_id}'
methods: ['PUT']
}
]
path_parts: {
/**
* The type of the inference task that the model will perform.
*/
task_type: CustomTaskType
/**
* The unique identifier of the inference endpoint.
*/
custom_inference_id: Id
}
body: {
/**
* The chunking configuration object.
* @ext_doc_id inference-chunking
*/
chunking_settings?: InferenceChunkingSettings
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are chunking settings relevant for this service?

/**
* The type of service supported for the specified task type. In this case, `custom`.
*/
service: CustomServiceType
/**
* Settings used to install the inference model.
* These settings are specific to the `custom` service.
*/
service_settings: CustomServiceSettings
/**
* Settings to configure the inference task.
* These settings are specific to the task type you specified.
*/
task_settings?: CustomTaskSettings
}
}
25 changes: 25 additions & 0 deletions specification/inference/put_custom/PutCustomResponse.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

import { InferenceEndpointInfoCustom } from '@inference/_types/Services'

export class Response {
/** @codegen_name endpoint_info */
body: InferenceEndpointInfoCustom
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
summary: Custom text embedding task (OpenAI)
description: Run `PUT _inference/text_embedding/custom-embeddings` to create an inference endpoint that performs a text embedding task.
method_request: 'PUT _inference/text_embedding/custom-embeddings'
# type: "request"
value: |-
{
"service": "custom",
"service_settings": {
"secret_parameters": {
"api_key": "<api key>"
},
"url": "https://api.openai.com/v1/embeddings",
"headers": {
"Authorization": "Bearer ${api_key}",
"Content-Type": "application/json;charset=utf-8"
},
"request": "{\"input\": ${input}, \"model\": \"text-embedding-3-small\"}",
"response": {
"json_parser": {
"text_embeddings": "$.data[*].embedding[*]"
}
}
}
}
Loading