Skip to content
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 32 additions & 17 deletions core/bifrost.go
Original file line number Diff line number Diff line change
Expand Up @@ -297,24 +297,12 @@ func (bifrost *Bifrost) ListModelsRequest(ctx context.Context, req *schemas.Bifr
return response, nil
}

// ListAllModels lists all models from all configured providers.
// It accumulates responses from all providers with a limit of 1000 per provider to get all results.
func (bifrost *Bifrost) ListAllModels(ctx context.Context, request *schemas.BifrostListModelsRequest) (*schemas.BifrostListModelsResponse, *schemas.BifrostError) {
// ListModelsFromProviders fetches models concurrently from specified providers
func (bifrost *Bifrost) ListModelsFromProviders(ctx context.Context, providers []schemas.ModelProvider, request *schemas.BifrostListModelsRequest) (*schemas.BifrostListModelsResponse, *schemas.BifrostError) {
if request == nil {
request = &schemas.BifrostListModelsRequest{}
}

providerKeys, err := bifrost.GetConfiguredProviders()
if err != nil {
return nil, &schemas.BifrostError{
IsBifrostError: false,
Error: &schemas.ErrorField{
Message: err.Error(),
Error: err,
},
}
}

startTime := time.Now()

// Result structure for collecting provider responses
Expand All @@ -323,11 +311,11 @@ func (bifrost *Bifrost) ListAllModels(ctx context.Context, request *schemas.Bifr
err *schemas.BifrostError
}

results := make(chan providerResult, len(providerKeys))
results := make(chan providerResult, len(providers))
var wg sync.WaitGroup

// Launch concurrent requests for all providers
for _, providerKey := range providerKeys {
// Launch concurrent requests for specified providers
for _, providerKey := range providers {
if strings.TrimSpace(string(providerKey)) == "" {
continue
}
Expand Down Expand Up @@ -427,6 +415,33 @@ func (bifrost *Bifrost) ListAllModels(ctx context.Context, request *schemas.Bifr
},
}

return response, nil
}

// ListAllModels lists all models from all configured providers.
// It accumulates responses from all providers with a limit of 1000 per provider to get all results.
func (bifrost *Bifrost) ListAllModels(ctx context.Context, request *schemas.BifrostListModelsRequest) (*schemas.BifrostListModelsResponse, *schemas.BifrostError) {
if request == nil {
request = &schemas.BifrostListModelsRequest{}
}

providerKeys, err := bifrost.GetConfiguredProviders()
if err != nil {
return nil, &schemas.BifrostError{
IsBifrostError: false,
Error: &schemas.ErrorField{
Message: err.Error(),
Error: err,
},
}
}

// Use the helper function to fetch from all configured providers
response, bifrostErr := bifrost.ListModelsFromProviders(ctx, providerKeys, request)
if bifrostErr != nil {
return nil, bifrostErr
}

response = response.ApplyPagination(request.PageSize, request.PageToken)

return response, nil
Expand Down
30 changes: 29 additions & 1 deletion docs/features/governance/virtual-keys.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Virtual Keys are the primary governance entity in Bifrost. Users and application
<Note>You can also use `Authorization` and `x-api-key` headers to pass direct keys to the provider. Read more about it in [Direct Key Bypass](../keys-management#direct-key-bypass).</Note>

**Key Features:**
- **Access Control** - Model and provider filtering
- **Access Control** - Model and provider filtering (applies to `/v1/models` endpoint and inference requests)
- **Cost Management** - Independent budgets (checked along with team/customer budgets if attached)
- **Rate Limiting** - Token and request-based throttling (VK-level only)
- **Key Restrictions** - Limit VK to specific provider API keys (if configured, VK can only use those keys)
Expand Down Expand Up @@ -499,6 +499,34 @@ curl -X POST http://localhost:8080/v1/chat/completions \
}'
```

The virtual key can also filter which models are returned by the `/v1/models` endpoint:

```bash
# List models allowed for a specific virtual key
curl -X GET http://localhost:8080/v1/models \
-H "x-bf-vk: vk-engineering-main"
```

**How it works:**
- **With `x-bf-vk` header**: Returns only models specified in the virtual key's `provider_configs`
- **Without `x-bf-vk` header**: Returns all available models
- **Empty `provider_configs`**: Returns all models (no filtering)
- **Empty `allowed_models` for a provider**: Returns all models from that provider

**Example:**
```json
{
"provider_configs": [
{
"provider": "openai",
"allowed_models": ["gpt-4o", "gpt-4o-mini"],
"weight": 1.0
}
]
}
// GET /v1/models with x-bf-vk header returns only openai/gpt-4o and openai/gpt-4o-mini
```

By default governance is optional, meaning that if the `x-bf-vk` header is not present, the request will be allowed but without any governance checks/routing. But you can make it mandatory by enforcing the governance header.

<Tabs group="enforce-governance-header">
Expand Down
14 changes: 14 additions & 0 deletions tests/governance/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,14 @@ This test suite provides extensive coverage of the Bifrost governance system inc
- Reset functionality
- Debug and health endpoints

5. **`test_virtual_keys_access.py`** - Virtual Key Model Filtering
- `/v1/models` endpoint filtering based on virtual key `provider_configs`
- Specific models via `allowed_models` list
- Empty `allowed_models` returns all models from that provider
- Empty `provider_configs` returns all models
- Multiple provider configurations
- Invalid and inactive virtual key handling

### Configuration Files

- **`conftest.py`** - Test fixtures, utilities, and configuration
Expand Down Expand Up @@ -162,6 +170,7 @@ The test suite uses pytest markers for categorization:
- `@pytest.mark.concurrency` - Concurrency tests
- `@pytest.mark.slow` - Slow running tests (>5s)
- `@pytest.mark.smoke` - Quick smoke tests
- `@pytest.mark.access_control` - Access control and filtering tests

## API Endpoints Tested

Expand Down Expand Up @@ -195,6 +204,11 @@ The test suite uses pytest markers for categorization:

### Integration Endpoints
- `POST /v1/chat/completions` - Chat completion with governance headers
- `GET /v1/models` - List available models with optional virtual key filtering

#### Model Filtering (`/v1/models`)
- **With `x-bf-vk` header**: Returns models specified in the virtual key's `provider_configs`
- **Without `x-bf-vk` header**: Returns all available models

## Test Data and Schemas

Expand Down
Loading