Merge pull request #33 from fluxcd/health-checks

Improve health checking

Author: stefanprodan

README.md

@@ -31,22 +31,15 @@ Specifications:
 ## Usage
 
 The kustomize-controller is part of a composable GitOps toolkit and depends on
-[source-controller](https://github.com/fluxcd/source-controller) to provide the raw Kubernetes
-manifests and `kustomization.yaml` file.
+[source-controller](https://github.com/fluxcd/source-controller) to acquire the Kubernetes
+manifests from Git repositories.
 
 ### Install the controllers
 
-Install source-controller with:
+Install the source and kustomize controllers in the `kustomize-system` namespace:
 
 ```bash
-kustomize build https://github.com/fluxcd/source-controller//config/default?ref=v0.0.1-alpha.4 \
-kubectl apply -f-
-```
-
-Install kustomize-controller with:
-
-```bash
-kustomize build https://github.com/fluxcd/kustomize-controller//config/default?ref=v0.0.1-alpha.7 \
+kustomize build https://github.com/fluxcd/kustomize-controller//config/default?ref=master \
 kubectl apply -f-
 ```
 
@@ -62,7 +55,7 @@ metadata:
   namespace: kustomize-system
 spec:
   interval: 1m
-  url: https://github.com/stefanprodan/podinfo-deploy
+  url: https://github.com/stefanprodan/podinfo
   ref:
     branch: master
 ```
@@ -95,29 +88,31 @@ metadata:
   namespace: kustomize-system
 spec:
   interval: 5m
-  path: "./overlays/dev/"
+  path: "./deploy/overlays/dev/"
   prune: true
   sourceRef:
     kind: GitRepository
     name: podinfo
   validation: client
   healthChecks:
     - kind: Deployment
-      name: podinfo
+      name: frontend
+      namespace: dev
+    - kind: Deployment
+      name: backend
       namespace: dev
   timeout: 80s
 ```
 
-> **Note** that if your repository contains only plain Kubernetes manifests,
-> the controller will
+> **Note** that if your repository contains only plain Kubernetes manifests, the controller will
 > [automatically generate](docs/spec/v1alpha1/kustomization.md#generate-kustomizationyaml)
 > a kustomization.yaml file inside the specified path.
 
 A detailed explanation of the Kustomization object and its fields
 can be found in the [specification doc](docs/spec/v1alpha1/README.md). 
 
 Based on the above definition, the kustomize-controller fetches the Git repository content from source-controller,
-generates Kubernetes manifests by running kustomize build inside `./overlays/dev/`,
+generates Kubernetes manifests by running kustomize build inside `./deploy/overlays/dev/`,
 and validates them with a dry-run apply. If the manifests pass validation, the controller will apply them 
 on the cluster and starts the health assessment of the deployed workload. If the health checks are passing, the
 Kustomization object status transitions to a ready state.
@@ -127,7 +122,7 @@ Kustomization object status transitions to a ready state.
 You can wait for the kustomize controller to complete the deployment with:
 
 ```bash
-kubectl wait kustomization/podinfo-dev --for=condition=ready
+kubectl -n kustomize-system wait kustomization/podinfo-dev --for=condition=ready
 ```
 
 When the controller finishes the reconciliation, it will log the applied objects:
@@ -142,20 +137,24 @@ kubectl -n kustomize-system logs deploy/kustomize-controller | jq .
   "ts": 1587195448.071468,
   "logger": "controllers.Kustomization",
   "msg": "Kustomization applied in 1.436096591s",
-  "kustomization": "default/podinfo-dev",
+  "kustomization": "kustomize-system/podinfo-dev",
   "output": {
     "namespace/dev": "created",
-    "service/podinfo": "created",
-    "deployment.apps/podinfo": "created",
-    "horizontalpodautoscaler.autoscaling/podinfo": "created"
+    "service/frontend": "created",
+    "deployment.apps/frontend": "created",
+    "horizontalpodautoscaler.autoscaling/frontend": "created",
+    "service/backend": "created",
+    "deployment.apps/backend": "created",
+    "horizontalpodautoscaler.autoscaling/backend": "created"
   }
 }
 ```
 
 You can trigger a kustomize build and apply any time with:
 
 ```bash
-kubectl annotate --overwrite kustomization/podinfo-dev kustomize.fluxcd.io/syncAt="$(date +%s)"
+kubectl -n kustomize-system annotate --overwrite kustomization/podinfo-dev \
+kustomize.fluxcd.io/syncAt="$(date +%s)"
 ```
 
 When the source controller pulls a new Git revision, the kustomize controller will detect that the
@@ -175,8 +174,8 @@ status:
 
 ```json
 {
-  "kustomization": "default/podinfo-dev",
-  "error": "Error from server (NotFound): error when creating \"podinfo-dev.yaml\": namespaces \"dev\" not found\n"
+  "kustomization": "kustomize-system/podinfo-dev",
+  "error": "Error from server (NotFound): error when creating podinfo-dev.yaml: namespaces dev not found"
 }
 ```
 
@@ -196,7 +195,7 @@ metadata:
   namespace: kustomize-system
 spec:
   interval: 10m
-  path: "./profiles/default/"
+  path: "./istio/system/"
   sourceRef:
     kind: GitRepository
     name: istio
@@ -234,9 +233,9 @@ metadata:
   namespace: kustomize-system
 spec:
   interval: 5m
-  url: https://github.com/stefanprodan/podinfo-deploy
+  url: https://github.com/stefanprodan/podinfo
   ref:
-    semver: ">=0.0.1-rc.1 <1.0.0"
+    semver: ">=3.2.3 <4.0.0"
 ```
 
 With `ref.semver` we configure source controller to pull the Git tags and create an artifact from the most recent tag
@@ -252,14 +251,14 @@ metadata:
   namespace: kustomize-system
 spec:
   interval: 10m
-  path: "./overlays/production/"
+  path: "./deploy/overlays/production/"
   sourceRef:
     kind: GitRepository
     name: podinfo-releases
 ```
 
-Based on the above definition, the kustomize controller will build and apply a kustomization that matches the semver range
-set in the Git repository manifest.
+Based on the above definition, the kustomize controller will apply the kustomization that matches the semver range
+set in the Git repository.
 
 ### Configure alerting
 

controllers/gitrepository_controller.go renamed to controllers/gitrepository_watcher.go

@@ -34,7 +34,8 @@ import (
 	kustomizev1 "github.com/fluxcd/kustomize-controller/api/v1alpha1"
 )
 
-// KustomizationReconciler watches a GitRepository object
+// GitRepositoryWatcher watches GitRepository objects for revision changes
+// and triggers a sync for all the Kustomizations that reference a changed source
 type GitRepositoryWatcher struct {
 	client.Client
 	Log    logr.Logger
@@ -54,7 +55,7 @@ func (r *GitRepositoryWatcher) Reconcile(req ctrl.Request) (ctrl.Result, error)
 	}
 
 	log := r.Log.WithValues(strings.ToLower(repo.Kind), req.NamespacedName)
-	log.Info("New artifact detected")
+	log.Info("new artifact detected")
 
 	// get the list of kustomizations that are using this Git repository
 	var list kustomizev1.KustomizationList

controllers/kustomization_controller.go

@@ -513,9 +513,17 @@ func (r *KustomizationReconciler) checkHealth(kustomization kustomizev1.Kustomiz
 	var alerts string
 
 	for _, check := range kustomization.Spec.HealthChecks {
-		cmd := fmt.Sprintf("kubectl -n %s rollout status %s %s --timeout=%s",
-			check.Namespace, check.Kind, check.Name, kustomization.GetTimeout())
+		cmd := fmt.Sprintf("until kubectl -n %s get %s %s ; do sleep 2; done",
+			check.Namespace, check.Kind, check.Name)
 		command := exec.CommandContext(ctx, "/bin/sh", "-c", cmd)
+		if _, err := command.CombinedOutput(); err != nil {
+			return fmt.Errorf("health check timeout for %s '%s/%s': %w",
+				check.Kind, check.Namespace, check.Name, err)
+		}
+
+		cmd = fmt.Sprintf("kubectl -n %s rollout status %s %s --timeout=%s",
+			check.Namespace, check.Kind, check.Name, kustomization.GetTimeout())
+		command = exec.CommandContext(ctx, "/bin/sh", "-c", cmd)
 		output, err := command.CombinedOutput()
 		if err != nil {
 			if errors.Is(err, context.DeadlineExceeded) {

docs/spec/README.md

@@ -15,15 +15,20 @@ for example a validation controller such as OPA Gatekeeper should be up and runn
 applying other manifests on the cluster. Another example is a service mesh admission controller,
 the proxy injector must be functional before deploying applications into the mesh.
 
-When operating a cluster, different teams may wish to receive notification about the status
-of their CD pipelines. For example, the on-call team would receive alerts about all
-failures in the prod namespace, while the frontend team may wish to be alerted when a new version 
-of the frontend app was deployed and if the deployment is healthy, no matter the namespace.
+When a cluster is shared with multiple teams, a cluster admin may wish to assign roles and service
+accounts to each team. The manifests owned by a team will be applied on the cluster using
+the team's account thus ensuring isolation between teams. For example, an admin can 
+restrict the operations performed on the cluster by a team to a single namespace.
 
 When dealing with an incident, one may wish to suspend the reconciliation of some workloads and
 pin the reconciliation of others to a specific Git revision, without having to stop the reconciler
 and affect the whole cluster.
 
+When operating a cluster, different teams may wish to receive notification about the status
+of their CD pipelines. For example, the on-call team would receive alerts about all
+failures in the prod namespace, while the frontend team may wish to be alerted when a new version 
+of the frontend app was deployed and if the deployment is healthy, no matter the namespace.
+
 ## Design
 
 The reconciliation process can be defined with a Kubernetes custom resource