mgr-applications

Copyright (C) 2022-2022 The Open Library Foundation

This software is distributed under the terms of the Apache License, Version 2.0. See the file "LICENSE" for more information.

Table of contents

• Introduction
• Compiling
• Running It
• Environment Variables
  • SSL Configuration environment variables
  • Secure storage environment variables
    • AWS-SSM
    • Vault
    • Folio Secure Store Proxy (FSSP)
• Keycloak Integration
  • Import Keycloak data on startup
  • Keycloak Security
  • Keycloak specific environment variables
• Kong Gateway Integration
  • Kong Service Registration
  • Kong Route Registration
• Kafka Integration
  • Events upon discovery changes
  • Naming convention
  • Event structure
• Manager Tenant Entitlements Integration
• Folio Application Registry mode

Introduction

mgr-applications provides following functionality:

• Dependency check / platform integrity validation
• (De-)Registration of applications
• Enabling/disabling of an application (including dependencies)
• (Un-)Deployment of an application (optional)
• Application health and availability monitoring
• Optional integration with Kong gateway
  • Add/remove services on application discovery update

Compiling

mvn clean install

See that it says BUILD SUCCESS near the end.

If you want to skip tests:

mvn clean install -DskipTests

Running It

Run locally with proper environment variables set (see Environment variables below) on listening port 8081 (default listening port):

java \
  -Dokapi.url=http://localhost:9130 \
  -Dokapi.token=${okapiToken} \
  -jar target/mgr-applications-*.jar

Build the docker container with following script after compilation:

docker build -t mgr-applications .

Test that it runs with:

docker run \
  --name mgr-applications \
  --link postgres:postgres \
  -e DB_HOST=postgres \
  -e okapi.url=http://okapi:9130 \
  -e tenant.url=http://mgr-tenants:8081 \
  -e te.url=http://mgr-tenant-entitlements:8081 \
  -e okapi.token=${okapiToken} \
  -p 8081:8081 \
  -d mgr-applications

Environment Variables

Name	Default value	Required	Description
DB_HOST	localhost	false	Postgres hostname
DB_PORT	5432	false	Postgres port
DB_USERNAME	postgres	false	Postgres username
DB_PASSWORD	postgres	false	Postgres username password
DB_DATABASE	okapi_modules	false	Postgres database name
MODULE_URL	http://mgr-applications:8081	false	Module URL (module cannot define url for Kong registration by itself, because it can be under Load Balancer, so this value must be provided manually)
okapi.url	-	false	Okapi URL used to perform HTTP requests by OkapiClient.
OKAPI_INTEGRATION_ENABLED	true	false	Defines if okapi integration is enabled or disabled. If it set to false - it will exclude all okapi-related beans from spring context.
tenant.url	-	true	Tenant URL used to perform HTTP requests by TenantManagerClient.
kong.url	-	true	Okapi URL used to perform HTTP requests for recurring jobs, required.
KONG_ADMIN_URL	-	false	Alias for kong.url.
KONG_INTEGRATION_ENABLED	true	false	Defines if kong integration is enabled or disabled. If it set to false - it will exclude all kong-related beans from spring context.
KONG_CONNECT_TIMEOUT	-	false	Defines the timeout in milliseconds for establishing a connection from Kong to upstream service. If the value is not provided then Kong defaults are applied.
KONG_READ_TIMEOUT	-	false	Defines the timeout in milliseconds between two successive read operations for transmitting a request from Kong to the upstream service. If the value is not provided then Kong defaults are applied.
KONG_WRITE_TIMEOUT	-	false	Defines the timeout in milliseconds between two successive write operations for transmitting a request from Kong to the upstream service. If the value is not provided then Kong defaults are applied.
KONG_RETRIES	-	false	Defines the number of retries to execute upon failure to proxy. If the value is not provided then Kong defaults are applied.
KONG_TLS_ENABLED	false	false	Allows to enable/disable TLS connection to Kong.
KONG_TLS_TRUSTSTORE_PATH	-	false	Truststore file path for TLS connection to Kong.
KONG_TLS_TRUSTSTORE_PASSWORD	-	false	Truststore password for TLS connection to Kong.
KONG_TLS_TRUSTSTORE_TYPE	-	false	Truststore type for TLS connection to Kong.
KONG_TENANT_CHECKS_ENABLED	false	false	Enables tenant checks enforcement via entitlement consumption. When true, mgr-applications consumes entitlement Kafka events and updates Kong route expressions to enforce tenant checks per route.
ENV	folio	false	The logical name of the deployment (kafka topic prefix), must be unique across all environments using the same shared Kafka/Elasticsearch clusters, a-z (any case), 0-9, -, _ symbols only allowed
KAFKA_HOST	kafka	false	Kafka broker hostname
KAFKA_PORT	9092	false	Kafka broker port
KAFKA_ENTITLEMENT_TOPIC	${ENV}.entitlement	false	Kafka topic for entitlement events. Defaults to environment-specific topic using KafkaUtils.getEnvTopicName('entitlement')
KAFKA_SECURITY_PROTOCOL	PLAINTEXT	false	Kafka security protocol
KAFKA_SSL_KEYSTORE_LOCATION	-	false	Kafka SSL keystore location
KAFKA_SSL_KEYSTORE_PASSWORD	-	false	Kafka SSL keystore password
KAFKA_SSL_TRUSTSTORE_LOCATION	-	false	Kafka SSL truststore location
KAFKA_SSL_TRUSTSTORE_PASSWORD	-	false	Kafka SSL truststore password
KAFKA_DISCOVERY_TOPIC_PARTITIONS	1	false	Amount of partitions for discovery topic.
KAFKA_DISCOVERY_TOPIC_REPLICATION_FACTOR	-	false	Replication factor for discovery topic.
TE_URL	-	true	Tenant Entitlement URL used to perform HTTP requests by TenantEntitlementClient.
TE_TLS_ENABLED	false	false	Allows to enable/disable TLS connection to mgr-tenant-entitlements module.
TE_TLS_TRUSTSTORE_PATH	-	false	Truststore file path for TLS connection to mgr-tenant-entitlements module.
TE_TLS_TRUSTSTORE_PASSWORD	-	false	Truststore password for TLS connection to mgr-tenant-entitlements module.
TE_TLS_TRUSTSTORE_TYPE	-	false	Truststore file type for TLS connection to mgr-tenant-entitlements module.
SECURITY_ENABLED	true	false	Allows to enable/disable security. If true and KC_INTEGRATION_ENABLED is also true - the Keycloak will be used as a security provider.
FAR_MODE	false	false	Allows to enable Folio Application Registry mode, if FAR mode is enabled, kong integration must disabled using environment variable KONG_INTEGRATION_ENABLED.
MOD_AUTHTOKEN_URL	-	true	Mod-authtoken URL. Required if OKAPI_INTEGRATION_ENABLED is true and SECURITY_ENABLED is true and KC_INTEGRATION_ENABLED is false.
SECURE_STORE_ENV	folio	false	First segment of the secure store key, for example prod or test. Defaults to folio. In Ramsons and Sunflower defaults to ENV with fall-back folio.
SECRET_STORE_TYPE	-	true	Secure storage type. Supported values: EPHEMERAL, AWS_SSM, VAULT, FSSP
VALIDATION_MODE	basic	false	Validation mode applied during Application Descriptors checking (see POST /applications/validate endpoint). Possible values: none, basic, onCreate
MAX_HTTP_REQUEST_HEADER_SIZE	200KB	true	Maximum size of the HTTP request header.
REGISTER_MODULE_IN_KONG	true	false	Defines if module must be registered in Kong (it will create for itself service and list of routes from module descriptor)
ROUTER_PATH_PREFIX		false	Defines routes prefix to be added to the generated endpoints by OpenAPI generator (/foo/entites -> {{prefix}}/foo/entities). Required if load balancing group has format like {{host}}/{{moduleId}}
ROUTEMANAGEMENT_ENABLE	true	false	Enable Kong routes management for modules discovery information (e.g. creation of routes on module discovery creation/update, removal of routes on deletion of module discovery information)

SSL Configuration environment variables

Name	Default value	Required	Description
SERVER_PORT	8081	false	Server HTTP port. Should be specified manually in case of SSL enabled.
SERVER_SSL_ENABLED	false	false	Manage server's mode. If true then SSL will be enabled.
SERVER_SSL_KEY_STORE		false	Path to the keystore. Mandatory if SERVER_SSL_ENABLED is true.
SERVER_SSL_KEY_STORE_TYPE	BCFKS	false	Type of the keystore. By default BCFKS value is used.
SERVER_SSL_KEY_STORE_PROVIDER	BCFIPS	false	Provider of the keystore.
SERVER_SSL_KEY_STORE_PASSWORD		false	Password for keystore.
SERVER_SSL_KEY_PASSWORD		false	Password for key in keystore.

Secure storage environment variables

AWS-SSM

Required when SECRET_STORE_TYPE=AWS_SSM

Name	Default value	Description
SECRET_STORE_AWS_SSM_REGION	-	The AWS region to pass to the AWS SSM Client Builder. If not set, the AWS Default Region Provider Chain is used to determine which region to use.
SECRET_STORE_AWS_SSM_USE_IAM	true	If true, will rely on the current IAM role for authorization instead of explicitly providing AWS credentials (access_key/secret_key)
SECRET_STORE_AWS_SSM_ECS_CREDENTIALS_ENDPOINT	-	The HTTP endpoint to use for retrieving AWS credentials. This is ignored if useIAM is true
SECRET_STORE_AWS_SSM_ECS_CREDENTIALS_PATH	-	The path component of the credentials endpoint URI. This value is appended to the credentials endpoint to form the URI from which credentials can be obtained.

Vault

Required when SECRET_STORE_TYPE=VAULT

Name	Default value	Description
SECRET_STORE_VAULT_TOKEN	-	token for accessing vault, may be a root token
SECRET_STORE_VAULT_ADDRESS	-	the address of your vault
SECRET_STORE_VAULT_ENABLE_SSL	false	whether or not to use SSL
SECRET_STORE_VAULT_PEM_FILE_PATH	-	the path to an X.509 certificate in unencrypted PEM format, using UTF-8 encoding
SECRET_STORE_VAULT_KEYSTORE_PASSWORD	-	the password used to access the JKS keystore (optional)
SECRET_STORE_VAULT_KEYSTORE_FILE_PATH	-	the path to a JKS keystore file containing a client cert and private key
SECRET_STORE_VAULT_TRUSTSTORE_FILE_PATH	-	the path to a JKS truststore file containing Vault server certs that can be trusted

Folio Secure Store Proxy (FSSP)

Required when SECRET_STORE_TYPE=FSSP

Name	Default value	Description
SECRET_STORE_FSSP_ADDRESS	-	The address (URL) of the FSSP service.
SECRET_STORE_FSSP_SECRET_PATH	secure-store/entries	The path in FSSP where secrets are stored/retrieved.
SECRET_STORE_FSSP_ENABLE_SSL	false	Whether to use SSL when connecting to FSSP.
SECRET_STORE_FSSP_TRUSTSTORE_PATH	-	Path to the truststore file for SSL connections.
SECRET_STORE_FSSP_TRUSTSTORE_FILE_TYPE	-	The type of the truststore file (e.g., JKS, PKCS12).
SECRET_STORE_FSSP_TRUSTSTORE_PASSWORD	-	The password for the truststore file.

Keycloak Integration

Import Keycloak data on startup

As startup, the application creates/updates necessary records in Keycloak from the internal module descriptor:

• Resource server
• Client - with credentials of KC_CLIENT_ID, a client secret will be retrieved from the secure store.
• Resources - mapped from descriptor routing entries.
• Permissions - mapped from requiredPermissions of routing entries.
• Roles - mapped from permission sets of descriptor.
• Policies - role policies as well as aggregate policies (specific for each resource).

Keycloak Security

Keycloak can be used as a security provider. If enabled - application will delegate endpoint permissions evaluation to Keycloak. A valid Keycloak JWT token must be passed for accessing secured resources. The feature is controlled by two env variables SECURITY_ENABLED and KC_INTEGRATION_ENABLED.

Keycloak specific environment variables

Name	Default value	Required	Description
KC_URL	http://keycloak:8080	false	Keycloak URL used to perform HTTP requests.
KC_INTEGRATION_ENABLED	true	false	Defines if Keycloak integration is enabled or disabled. If it set to false - it will exclude all keycloak-related beans from spring context.
KC_IMPORT_ENABLED	false	false	If true - at startup, register/create necessary records in keycloak from the internal module descriptor.
KC_ADMIN_CLIENT_ID	folio-backend-admin-client	false	Keycloak client id. Used for register/create necessary records in keycloak from the internal module descriptor.
KC_ADMIN_CLIENT_SECRET	-	conditional	Keycloak client secret. Required only if admin username/password are not set.
KC_ADMIN_USERNAME	-	conditional	Keycloak admin username. Required only if admin secret is not set.
KC_ADMIN_PASSWORD	-	conditional	Keycloak admin password. Required only if admin secret is not set.
KC_ADMIN_GRANT_TYPE	client_credentials	false	Keycloak admin grant type. Should be set to password if username/password are used instead of client secret.
KC_CLIENT_ID	mgr-applications	false	client id to be imported to Keycloak.
KC_CLIENT_TLS_ENABLED	false	false	Enables TLS for keycloak clients.
KC_CLIENT_TLS_TRUSTSTORE_PATH	-	false	Truststore file path for keycloak clients.
KC_CLIENT_TLS_TRUSTSTORE_PASSWORD	-	false	Truststore password for keycloak clients.
KC_AUTH_TOKEN_VALIDATE_URI	false	false	Defines if validation for JWT must be run to compare configuration URL and token issuer for keycloak.
KC_JWKS_BASE_URL		false	Custom base URL for JWKS endpoint. If specified, will be used instead of issuer URL from token's iss claim (e.g., http://keycloak:8080).
KC_JWKS_REFRESH_INTERVAL	60	false	Jwks refresh interval for realm JWT parser (in minutes).
KC_FORCED_JWKS_REFRESH_INTERVAL	60	false	Forced jwks refresh interval for realm JWT parser (used in signing key rotation, in minutes).

Kong Gateway Integration

Kong gateway integration implemented using idempotent approach with Kong Admin API.

Kong Service Registration

On service discovery registration for each module, Kong Gateway services are added and configured to enable proper routing and access through Kong.

• For every application module discovered, a Kong Service is created (or updated if it already exists) using the module's descriptor and the configured MODULE_URL.
• Service configuration includes:
  • Service name (typically mapped to module id, e.g. mod-foo-1.2.3)
  • Host and port pointing to the backend application instance
  • Timeout setting for connections timeout, reads, writes
• Changes to discovery information (such as updates to service URL or enabled/disabled states) result in corresponding updates to the Kong Service.
• Services are managed idempotently: existing services are updated as needed, new services are created, and unused services are removed.

To view registered services in Kong, use:

curl -X GET "$KONG_ADMIN_URL/services"

This will return a list of all services registered in the Kong Gateway.

Kong Route Registration

Routes are registered for each module and mapped to the corresponding Kong Service. Routes define how incoming requests are matched and forwarded to the backend service.

• For each module, routes are created or updated according to routing entries in the module descriptor.
• Route configuration includes:
  • Paths and methods handled by the module (e.g., /users, GET)
  • Routes expression matches
  • Tags for module identification (moduleId, interfaceId)
• Routes are managed idempotently: existing routes are updated in place, new routes are created, and obsolete routes are removed.
• Tenant-specific route enforcement is supported via entitlement events (see Kong Tenant Checks Enforcement).

To view the routes for a specific service, use:

curl -X GET "$KONG_ADMIN_URL/services/$moduleId/routes"

This will return all routes registered for the specified module's service.

Note: Route expressions and tenant restrictions are dynamically managed based on entitlement events when KONG_TENANT_CHECKS_ENABLED=true. See the dedicated section for details.

Kong Tenant Checks Enforcement

The application supports dynamic tenant-specific route enforcement through entitlement events. When KONG_TENANT_CHECKS_ENABLED=true:

• Entitlement Subscription: The application subscribes to the ${ENV}.entitlement Kafka topic to receive tenant entitlement events.
• Dynamic Route Updates: Routes are automatically updated with tenant-specific expressions based on entitlement events:
  • ENTITLE/UPGRADE: Adds tenant to route's tenant-check clause using x-okapi-tenant header validation
  • REVOKE: Removes tenant from route's tenant-check clause
  • First tenant: Replaces wildcard routes with explicit tenant-check expressions
  • Last tenant removed: Reverts routes back to wildcard (no tenant restrictions)

Route Expression Examples

Wildcard route (default behavior):

(http.path == "/users" && http.method == "GET")

Single tenant route:

(http.path == "/users" && http.method == "GET") && (http.headers["x-okapi-tenant"] == "tenant1")

Multiple tenant route:

(http.path == "/users" && http.method == "GET") && (http.headers["x-okapi-tenant"] == "tenant1" || http.headers["x-okapi-tenant"] == "tenant2")

Behavior

• Default (KONG_TENANT_CHECKS_ENABLED=false): No entitlement subscription, no route expression changes
• Enabled (KONG_TENANT_CHECKS_ENABLED=true): Consumes entitlement events and dynamically updates Kong route expressions
• Idempotent: Updates are applied incrementally based on current route state
• Fallback: Routes without tenant restrictions allow access to all tenants (wildcard behavior)

Kafka Integration

Events upon discovery changes

• The application publishes lightweight kafka events when discovery information is registered/updated/removed.
• The topic is being created on application startup.

Naming convention

topic naming convention:<prefix>_discovery

• Prefix is passed in as environment variable (for FSE it's usually the cluster identifier, e.g. "evrk")

Event structure

{
  "moduleId": "mod-foo-1.2.3"
}

Manager Tenant Entitlements Integration

• The application checks if application descriptor exist in mgr-tenant-entitlements before deletion

curl -XGET "$TE_URL/entitlements?query=applicationId=$applicationId"

Folio Application Registry mode

In this mode, we only need a subset of the component's functionality. While CRUD operations for application descriptors works as-is, the integrations with Kafka, Kong, Okapi, mgr-tenant-entitlements is disabled. To enable this mode set FAR_MODE env variable to true and make sure to leave other integration variables unset or set to false.