Skip to content

Commit aa5d224

Browse files
irwin9204irwin9204
committed
Feat: add certificate configuration document
1 parent 58e9e38 commit aa5d224

File tree

3 files changed

+714
-0
lines changed

3 files changed

+714
-0
lines changed

‎docs/administrator/security/certificate-configuration.md‎

Lines changed: 334 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,334 @@
1+
---
2+
title: Certificate Configuration
3+
---
4+
5+
## Certificate Configuration
6+
7+
Karmada components use digital certificates to verify each other's identities, preventing malicious third parties from impersonating and stealing information or attacking the system. During mutual authentication with digital certificates between two components, the following files are involved:
8+
9+
- Business Certificate (Client/Server): A certificate used to prove one's own identity.
10+
- Private Key: The private key corresponding to the public key included in the business certificate.
11+
- CA Root Certificate: The root certificate issued by the Certificate Authority (CA) that signed the business certificate.
12+
13+
Both the business certificate and the CA root certificate have an expiration date. Therefore, when a business certificate or a CA root certificate is about to expire or has already expired, new certificates need to be replaced to ensure that the components can operate normally.
14+
15+
### Prerequisites
16+
17+
You need to install the following tools:
18+
19+
- `openssl` ([Installation Guide](https://github.com/openssl/openssl#download))
20+
- `cfssl` ([Installation Guide](github.com/cloudflare/cfssl/cmd))
21+
- `cfssljson` ([Installation Guide](github.com/cloudflare/cfssl/cmd))
22+
23+
24+
25+
> NOTE: The following examples are based on the installation method described in [Installation from Source](https://karmada.io/docs/installation/fromsource/#install). When using other installation methods, appropriate adaptation is required.
26+
27+
### Karmada Certificates
28+
29+
Karmada certificate storage path: Default to `${HOME}/.karmada`.
30+
31+
The current certificates used by Karmada include:
32+
33+
```shell
34+
$ tree
35+
.
36+
├── apiserver.crt
37+
├── apiserver.key
38+
├── ca-config.json
39+
├── ca.crt
40+
├── ca.key
41+
├── etcd-ca-config.json
42+
├── etcd-ca.crt
43+
├── etcd-ca.key
44+
├── etcd-client.crt
45+
├── etcd-client.key
46+
├── etcd-server.crt
47+
├── etcd-server.key
48+
├── front-proxy-ca-config.json
49+
├── front-proxy-ca.crt
50+
├── front-proxy-ca.key
51+
├── front-proxy-client.crt
52+
├── front-proxy-client.key
53+
├── karmada.crt
54+
└── karmada.key
55+
```
56+
57+
#### **Karmada Certificate Overview**
58+
59+
Certificates can be categorized into three sets based on the issuing CA:
60+
61+
- Issued by ca
62+
63+
The certificates `apiserver` and `karmada` are both issued by the CA certificate `ca`. The key attributes of these certificates are as follows:
64+
65+
| Certificates | common name(CN) | organization(og) | hosts |
66+
| ------------ | ----------------- | ---------------- | ------------------------------------------------------------ |
67+
| apiserver | karmada-apiserver | / | "*.etcd.karmada-system.svc.cluster.local" "*.karmada-system.svc.cluster.local" "*.karmada-system.svc" "localhost" "127.0.0.1" "${api server's ip from kubecomfig by context karmada host}" |
68+
| karmada | system:admin | system:masters | kubernetes.default.svc "*.etcd.karmada-system.svc.cluster.local" "*.karmada-system.svc.cluster.local" "*.karmada-system.svc" "localhost" "127.0.0.1" "${interpreter_webhook_example_service_external_ip_address}" |
69+
70+
The `${api server's ip from kubecomfig by context karmada host}` can be obtained by executing `./get_server_ip.sh ${HOST_CLUSTER_NAME}`.
71+
72+
```shell
73+
#!/usr/bin/env bash
74+
# get_server_ip.sh
75+
context_name=$1
76+
cluster_name=$(kubectl config view --template='{{ range $_, $value := .contexts }}{{if eq $value.name '"\"${context_name}\""'}}{{$value.context.cluster}}{{end}}{{end}}')
77+
apiserver_url=$(kubectl config view --template='{{range $_, $value := .clusters }}{{if eq $value.name '"\"${cluster_name}\""'}}{{$value.cluster.server}}{{end}}{{end}}')
78+
echo "${apiserver_url}" | awk -F/ '{print $3}' | sed 's/:.*//'
79+
```
80+
81+
> PS: The `karmada api server config` in the `kubeconfig` file is composed of the `karmada` certificate.
82+
83+
- Issued by etcd-ca
84+
85+
The certificates `etcd-server` and `etcd-client` are both issued by the CA certificate `etcd-ca`. The key attributes of these certificates are as follows:
86+
87+
| certificates | common name(CN) | organization(og) | hosts |
88+
| ------------ | --------------- | ---------------- | ------------------------------------------------------------ |
89+
| etcd-server | etcd-server | / | kubernetes.default.svc "*.etcd.karmada-system.svc.cluster.local" "*.karmada-system.svc.cluster.local" "*.karmada-system.svc" "localhost" "127.0.0.1" |
90+
| etcd-client | etcd-client | / | "*.etcd.karmada-system.svc.cluster.local" "*.karmada-system.svc.cluster.local" "*.karmada-system.svc" "localhost" "127.0.0.1" |
91+
92+
- Issued by front-proxy-ca
93+
94+
The certificate `front-proxy-client` is issued by the CA certificate `front-proxy-ca`. The key attributes of these certificates are as follows:
95+
96+
| certificates | common name(CN) | organization(og) | hosts |
97+
| ------------------ | ------------------ | ---------------- | ------------------------------------------------------------ |
98+
| front-proxy-client | front-proxy-client | / | kubernetes.default.svc "*.etcd.karmada-system.svc.cluster.local" "*.karmada-system.svc.cluster.local" "*.karmada-system.svc" "localhost" "127.0.0.1" |
99+
100+
#### How Karmada Components Use Certificates
101+
102+
Karmada components use certificates by mounting secrets. The current secrets used by Karmada for storing certificates are:
103+
104+
- [karmada-cert-secret](https://github.com/karmada-io/karmada/blob/master/artifacts/deploy/karmada-cert-secret.yaml)
105+
- [kubeconfig](https://github.com/karmada-io/karmada/blob/master/artifacts/deploy/secret.yaml)
106+
107+
- [karmada-webhook-cert-secret](https://github.com/karmada-io/karmada/blob/master/artifacts/deploy/karmada-webhook-cert-secret.yaml)
108+
109+
You can find the corresponding certificates for each secret from the [cert-secret-generation](https://github.com/karmada-io/karmada/blob/19d1146c3510942809f48d399fc2079ce3a79a66/hack/deploy-karmada.sh#L102-L124).
110+
111+
Karmada components acquire the necessary certificates by mounting secrets. The detailed usage of certificates by each component is as follows:
112+
113+
- **etcd** mounts the secret `karmada-cert-secret` and uses the certificates for
114+
115+
```yaml
116+
- --cert-file=/etc/karmada/pki/etcd-server.crt
117+
- --key-file=/etc/karmada/pki/etcd-server.key
118+
- --trusted-ca-file=/etc/karmada/pki/etcd-ca.crt
119+
```
120+
121+
- **karmada-apiserver** mounts the secret `karmada-cert-secret` and uses the certificates for:
122+
123+
```yaml
124+
- --client-ca-file=/etc/karmada/pki/ca.crt
125+
- --etcd-cafile=/etc/karmada/pki/etcd-ca.crt
126+
- --etcd-certfile=/etc/karmada/pki/etcd-client.crt
127+
- --etcd-keyfile=/etc/karmada/pki/etcd-client.key
128+
- --kubelet-client-certificate=/etc/karmada/pki/karmada.crt
129+
- --kubelet-client-key=/etc/karmada/pki/karmada.key
130+
- --service-account-key-file=/etc/karmada/pki/karmada.key
131+
- --service-account-signing-key-file=/etc/karmada/pki/karmada.key
132+
- --proxy-client-cert-file=/etc/karmada/pki/front-proxy-client.crt
133+
- --proxy-client-key-file=/etc/karmada/pki/front-proxy-client.key
134+
- --requestheader-client-ca-file=/etc/karmada/pki/front-proxy-ca.crt
135+
- --tls-cert-file=/etc/karmada/pki/apiserver.crt
136+
- --tls-private-key-file=/etc/karmada/pki/apiserver.key
137+
```
138+
139+
- **karmada-controller-manager** mounts the secret `kubeconfig` and uses the certificates for
140+
141+
```yaml
142+
- --kubeconfig=/etc/kubeconfig
143+
```
144+
145+
- **karmada-aggregated-apiserver** mounts the secrets karmada-cert-secret and kubeconfig and uses the certificates for:
146+
147+
```yaml
148+
- --kubeconfig=/etc/kubeconfig
149+
- --authentication-kubeconfig=/etc/kubeconfig
150+
- --authorization-kubeconfig=/etc/kubeconfig
151+
- --etcd-cafile=/etc/karmada/pki/etcd-ca.crt
152+
- --etcd-certfile=/etc/karmada/pki/etcd-client.crt
153+
- --etcd-keyfile=/etc/karmada/pki/etcd-client.key
154+
- --tls-cert-file=/etc/karmada/pki/karmada.crt
155+
- --tls-private-key-file=/etc/karmada/pki/karmada.key
156+
```
157+
158+
- **karmada-scheduler-estimator** mounts the secret `karmada-cert-secret` and uses the certificates for:
159+
160+
```yaml
161+
- --grpc-auth-cert-file=/etc/karmada/pki/karmada.crt
162+
- --grpc-auth-key-file=/etc/karmada/pki/karmada.key
163+
- --grpc-client-ca-file=/etc/karmada/pki/ca.crt
164+
```
165+
166+
- **karmada-scheduler** mounts the secrets `karmada-cert-secret` and `kubeconfig` and uses the certificates for:
167+
168+
```yaml
169+
- --kubeconfig=/etc/kubeconfig
170+
- --scheduler-estimator-ca-file=/etc/karmada/pki/ca.crt
171+
- --scheduler-estimator-cert-file=/etc/karmada/pki/karmada.crt
172+
- --scheduler-estimator-key-file=/etc/karmada/pki/karmada.key
173+
```
174+
175+
- **karmada-descheduler** mounts the secrets `karmada-cert-secret` and `kubeconfig` and uses the certificates for:
176+
177+
```yaml
178+
- --kubeconfig=/etc/kubeconfig
179+
- --scheduler-estimator-ca-file=/etc/karmada/pki/ca.crt
180+
- --scheduler-estimator-cert-file=/etc/karmada/pki/karmada.crt
181+
- --scheduler-estimator-key-file=/etc/karmada/pki/karmada.key
182+
```
183+
184+
- **karmada-metrics-adapter** mounts the secrets `karmada-cert-secret` and `kubeconfig` and uses the certificates for:
185+
186+
```yaml
187+
- --kubeconfig=/etc/kubeconfig
188+
- --authentication-kubeconfig=/etc/kubeconfig
189+
- --authorization-kubeconfig=/etc/kubeconfig
190+
- --client-ca-file=/etc/karmada/pki/ca.crt
191+
- --tls-cert-file=/etc/karmada/pki/karmada.crt
192+
- --tls-private-key-file=/etc/karmada/pki/karmada.key
193+
```
194+
195+
- **karmada-search** mounts the secrets `karmada-cert-secret` and `kubeconfig` and uses the certificates for:
196+
197+
```yaml
198+
- --kubeconfig=/etc/kubeconfig
199+
- --authentication-kubeconfig=/etc/kubeconfig
200+
- --authorization-kubeconfig=/etc/kubeconfig
201+
- --etcd-cafile=/etc/karmada/pki/etcd-ca.crt
202+
- --etcd-certfile=/etc/karmada/pki/etcd-client.crt
203+
- --etcd-keyfile=/etc/karmada/pki/etcd-client.key
204+
- --tls-cert-file=/etc/karmada/pki/karmada.crt
205+
- --tls-private-key-file=/etc/karmada/pki/karmada.key
206+
```
207+
208+
- **karmada-webhook** mounts the secrets `webhook-cert` and `kubeconfig` and uses the certificates for:
209+
210+
```yaml
211+
- --kubeconfig=/etc/kubeconfig
212+
- --cert-dir=/var/serving-cert
213+
```
214+
215+
#### Checking Certificate Expiry Dates
216+
217+
You can use the command `openssl x509 -noout -dates -in path/to/cert` to extract the before and after dates of the certificate.
218+
219+
```bash
220+
$ openssl x509 -noout -dates -in path/to/cert
221+
notBefore=Aug 24 16:00:00 2024 GMT
222+
notAfter=Aug 29 16:00:00 2024 GMT
223+
```
224+
225+
The output indicates that this certificate becomes valid at 16:00 on August 24, 2024, and expires at 16:00 on August 29, 2024.
226+
227+
228+
### **Certificate Replacement**
229+
230+
When a business certificate expires or is about to expire, you can manually replace it with a new certificate to extend its validity.
231+
232+
#### generate a new certificate
233+
234+
1. **Determine the issuing CA of the expired certificate.** Refer to the **Karmada Certificate Overview** for the relationship between business certificates and their issuing CAs.
235+
236+
2. **Obtain the Common Name (CN) and other details from the expired certificate.** To ensure consistency before and after the replacement, it is recommended to directly inspect the expired business certificate to obtain the `CN` and other details.
237+
238+
```shell
239+
$ openssl x509 -noout -text -in path/to/cert
240+
Certificate:
241+
Data:
242+
... ...
243+
Subject: O = system:masters, CN = system:admin
244+
... ...
245+
X509v3 Subject Alternative Name:
246+
DNS:kubernetes.default.svc, DNS:*.etcd.karmada-system.svc.cluster.local, DNS:*.karmada-system.svc.cluster.local, DNS:*.karmada-system.svc, DNS:localhost, IP Address:127.0.0.1
247+
... ...
248+
```
249+
250+
The output of the command indicates that the expired certificate has the following details: `CN=system:admin`, `O=system:masters`, and `hosts=kubernetes.default.svc "*.etcd.karmada-system.svc.cluster.local" "*.karmada-system.svc.cluster.local" "*.karmada-system.svc" "localhost" "127.0.0.1"`.
251+
252+
3. **Use the CA and certificate information to issue a new certificate.**
253+
254+
```shell
255+
#!/usr/bin/env bash
256+
# signCert.sh
257+
dest_dir=$1
258+
ca=$2
259+
ca_dir=$3
260+
cert_name=$4
261+
cn=${5:-$4}
262+
og=$6
263+
hosts=""
264+
SEP=""
265+
shift 6
266+
while [[ -n "${1:-}" ]]; do
267+
hosts+="${SEP}\"$1\""
268+
SEP=","
269+
shift 1
270+
done
271+
${sudo} /usr/bin/env bash -e <<EOF
272+
echo '{"CN":"${cn}","hosts":[${hosts}],"names":[{"O":"${og}"}],"key":{"algo":"rsa","size":3072}}' | cfssl gencert -ca="${ca_dir}/${ca}.crt" -ca-key="${ca_dir}/${ca}.key" -config="${ca_dir}/${ca}-config.json" - | cfssljson -bare ${cert_name}
273+
mv "${cert_name}-key.pem" "${cert_name}.key"
274+
mv "${cert_name}.pem" "${cert_name}.crt"
275+
rm -f "${cert_name}.csr"
276+
EOF
277+
```
278+
279+
The input parameters for `signCert.sh` and their meanings are as follows:
280+
281+
- `dest_dir`: The directory for the new certificate, which is recommended to be different from the directory of the original business certificate to avoid overwriting.
282+
- `ca`: The name of the CA (Certificate Authority) certificate.
283+
- `ca_dir`: The directory where the CA certificate is stored.
284+
- `cn`: The Common Name (CN) for the business certificate.
285+
- `og`: The organization (O) for the business certificate.
286+
- `hosts`: The list of hosts for the business certificate.
287+
288+
Example: For an expired Karmada certificate with a CA named `ca`, `CN=system:admin`, `O=system:masters`, and hosts including `kubernetes.default.svc`, `*.etcd.karmada-system.svc.cluster.local`, `*.karmada-system.svc.cluster.local`, `*.karmada-system.svc`, `localhost`, and `127.0.0.1`, you can execute the following command to generate a new Karmada certificate in the current directory:
289+
290+
```she
291+
./signCert.sh . "ca" "${HOME}/.karmada" "karmada" "system:admin" "system:masters" kubernetes.default.svc "*.etcd.karmada-system.svc.cluster.local" "*.karmada-system.svc.cluster.local" "*.karmada-system.svc" "localhost" "127.0.0.1"
292+
```
293+
294+
#### **Certificate Replacement**
295+
296+
Since Karmada components obtain certificates by mounting secrets, when a secret is updated, it will automatically synchronize to the component's mount path. Therefore, all that is needed is for the component to restart so that it can load the new certificate. It is recommended to update the server-side certificates first. This is because updating server-side certificates may affect all clients that depend on that service, whereas client-side certificate updates are more scattered and have a smaller impact range.
297+
298+
1. **Update the Secret**
299+
300+
Using the certificate `karmada` as an example,
301+
302+
```shell
303+
# Base64 encode the new certificate
304+
KARMADACRT=$(cat karmada.crt|base64 -w 0)
305+
KARMADAKEY=$(cat karmada.key|base64 -w 0)
306+
# update secret karmada-cert-secret
307+
kubectl --kubeconfig $HOME/.kube/karmada.config --context karmada-host patch secret karmada-cert-secret -nkarmada-system --type='json' -p='[{"op": "replace", "path": "/data/karmada.crt", "value":'"${KARMADACRT}"'},{"op": "replace", "path": "/data/karmada.key", "value":'"${KARMADAKEY}"'}]'
308+
# update secret webhook-cert
309+
kubectl --kubeconfig $HOME/.kube/karmada.config --context karmada-host patch secret webhook-cert -nkarmada-system --type='json' -p='[{"op": "replace", "path": "/data/tls.crt", "value":'"${KARMADACRT}"'},{"op": "replace", "path": "/data/tls.key", "value":'"${KARMADAKEY}"'}]'
310+
# update secret kubeconfig
311+
kubectl --kubeconfig $HOME/.kube/karmada.config --context karmada-host edit secret webhook-cert -nkarmada-system -oyaml
312+
```
313+
314+
It's important to note that the `client-certificate-data` and `client-key-data` values under the `karmada-apiserver` context within the `kubeconfig` file also need to be replaced with the new `karmada.crt` and `karmada.key`.
315+
316+
2. **Restart Components to Reload Certificates**
317+
318+
After synchronizing the new certificates to the secrets in Step 1, the components need to be restarted so they can load the new certificates.
319+
320+
First, determine the scope of the certificate update, which identifies the components that need to be restarted.
321+
322+
Restart Karmada components in batches as needed.
323+
324+
Based on the dependencies between components, the recommended order for restarting is as follows:
325+
326+
- **etcd**, as the backend for `karmada-apiserver`, should be restarted first.
327+
328+
- **karmada-apiserver**, as the backend for other Karmada components, should be restarted after `etcd` has loaded the new certificate.
329+
330+
- **karmada-aggregated-apiserver** should be restarted after `karmada-apiserver` has loaded the new certificate.
331+
332+
- Remaining components can be restarted in the same batch.
333+
334+
With this, the process of updating expired certificates is complete.

0 commit comments

Comments
 (0)