diff --git a/.github/workflows/README.md b/.github/workflows/README.md new file mode 100644 index 0000000..04c8c6c --- /dev/null +++ b/.github/workflows/README.md @@ -0,0 +1,114 @@ +# GitHub Workflows + +This directory contains GitHub Actions workflows for building and deploying applications. + +## Workflow Types + +### Individual Application Workflows + +Files like `dashy.yaml`, `homepage.yaml`, `cyberchef.yaml`, and `toolchain.yaml` are minimal wrapper workflows that trigger on changes to specific application directories. + +**Structure**: +```yaml +name: + +on: + workflow_dispatch: + push: + branches: + - main + paths: + - applicationset//** + +jobs: + docker-build: + uses: ./.github/workflows/reusable-docker-build.yaml + with: + path: applicationset/ +``` + +### Reusable Workflows + +#### reusable-docker-build.yaml + +A reusable workflow that handles Docker image building and pushing. Used by all application workflows. + +**Inputs**: +- `tag-name`: Docker image tag name (default: 'docker-build') +- `path`: Path to the application directory (required) +- `push`: Whether to push the image to registry (default: false) +- `version`: Image version tag (default: 'latest') + +### Release Workflow + +#### release.yaml + +Triggered by Git tags following the pattern `-`. This workflow: +1. Parses the application name and version from the tag +2. Builds and pushes the Docker image +3. Updates the version in the application's `kustomization.yaml` +4. Commits the version update back to the repository + +## Creating a New Application Workflow + +To add a workflow for a new application: + +1. **Copy an existing workflow** (e.g., `cyberchef.yaml`) +2. **Update the name** to match your application +3. **Update the paths** to trigger on your application directory +4. **Customize the path parameter** in the workflow call + +**Example for a new app called "myapp"**: + +```yaml +name: myapp + +on: + workflow_dispatch: + push: + branches: + - main + paths: + - applicationset/myapp/** + +jobs: + docker-build: + uses: ./.github/workflows/reusable-docker-build.yaml + with: + path: applicationset/myapp +``` + +## Why Individual Workflows? + +While these workflows appear duplicated, they serve distinct purposes: + +1. **Isolated Triggers**: Each workflow only runs when its specific application changes +2. **Clear Build Status**: Individual workflow badges in README.md show per-application status +3. **Flexible Configuration**: Each application can customize its workflow parameters if needed +4. **GitHub Actions Limitations**: GitHub Actions doesn't support dynamic workflow generation + +The common build logic is extracted into `reusable-docker-build.yaml` to avoid true code duplication. + +## Deployment Process + +### Continuous Deployment (Push to Main) + +1. Push changes to `applicationset//` +2. The application's workflow triggers +3. Docker image is built (but not pushed) +4. ArgoCD syncs the changes from the repository + +### Release Deployment (Tagged) + +1. Create a tag: `git tag -v1.0.0 && git push --tags` +2. The `release.yaml` workflow triggers +3. Docker image is built and pushed to GitHub Container Registry +4. `kustomization.yaml` is updated with the new version +5. ArgoCD syncs the changes and deploys the new version + +## Best Practices + +- Keep individual workflows minimal (just trigger configuration) +- Put common logic in reusable workflows +- Use semantic versioning for release tags: `-v..` +- Test kustomization locally before pushing: `kubectl kustomize applicationset/` diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..60ec5dc --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,237 @@ +# Contributing to GitOps + +Thank you for your interest in contributing to this GitOps repository! + +## Overview + +This repository uses GitOps practices to manage Kubernetes deployments via ArgoCD. All changes to the cluster are made by updating this repository. + +## Repository Structure + +``` +├── .github/workflows/ # GitHub Actions CI/CD pipelines +├── applicationset/ # Application configurations +│ ├── base/ # Base Kustomize templates +│ └── / # Individual application configs +├── k8s-templates/ # Reusable Kubernetes resource templates +├── cluster-bootstrap/ # Cluster infrastructure components +└── changedetection/ # Standalone application +``` + +## Adding a New Application + +### Quick Start + +1. **Choose a template approach**: + - Copy an existing similar application from `applicationset/` + - Or start from scratch using templates in `k8s-templates/` + +2. **Create your application directory**: + ```bash + mkdir -p applicationset/myapp + ``` + +3. **Create required files**: + - `kustomization.yaml` - Kustomize configuration + - `vs.yaml` - VirtualService for Istio routing + - Optional: `volume.yaml` - Persistent storage + - Optional: `patches/` - Customizations to base resources + +4. **Reference the base template**: + ```yaml + # applicationset/myapp/kustomization.yaml + apiVersion: kustomize.config.k8s.io/v1beta1 + kind: Kustomization + resources: + - ../base + - vs.yaml + namePrefix: myapp- + images: + - name: ghcr.io/guyzsarun-lab/base + newName: ghcr.io/guyzsarun-lab/myapp + newTag: v1.0.0 + commonLabels: + app: myapp + ``` + +5. **Test locally**: + ```bash + kubectl kustomize applicationset/myapp + ``` + +6. **Create a GitHub workflow** (if building custom images): + ```bash + cp .github/workflows/cyberchef.yaml .github/workflows/myapp.yaml + # Edit the file to update name and paths + ``` + +7. **Commit and push**: + ```bash + git add applicationset/myapp + git commit -m "Add myapp application" + git push + ``` + +### Detailed Guides + +- [Kubernetes Templates Guide](k8s-templates/README.md) - Common patterns and templates +- [Workflow Guide](.github/workflows/README.md) - CI/CD pipeline patterns + +## Common Patterns + +### Pattern 1: Simple Web Application + +For applications with: +- Single container +- Exposed via Istio VirtualService +- No persistent storage + +**Example**: `cyberchef`, `toolchain` + +### Pattern 2: Application with Configuration + +For applications with: +- ConfigMaps for configuration files +- Volume mounts for config +- Patches to inject config + +**Example**: `dashy`, `glance` + +### Pattern 3: Application with Persistent Storage + +For applications with: +- NFS-backed PersistentVolume +- Volume mounts in deployment +- Data persistence requirements + +**Example**: `linkwarden`, `paperless`, `calibre` + +### Pattern 4: Multi-Container Application + +For applications with: +- Sidecar containers +- Multiple services +- Complex networking + +**Example**: `linkwarden` (with meilisearch sidecar) + +## Kustomize Best Practices + +### Use Strategic Merge Patches + +For modifying specific fields: +```yaml +# patches/service.yaml +apiVersion: v1 +kind: Service +metadata: + name: application +spec: + ports: + - $patch: replace + - port: 3000 + targetPort: 8080 +``` + +### Use namePrefix for Resource Naming + +Instead of hardcoding names, use `namePrefix`: +```yaml +namePrefix: myapp- +# Results in: myapp-application, myapp-vs, etc. +``` + +### Use commonLabels for Consistent Labeling + +```yaml +commonLabels: + app: myapp +``` + +## Validation + +Before committing, always validate: + +```bash +# Validate Kustomize output +kubectl kustomize applicationset/myapp + +# Dry-run to catch issues +kubectl kustomize applicationset/myapp | kubectl apply --dry-run=client -f - + +# Check YAML syntax +kubectl kustomize applicationset/myapp | kubectl apply --dry-run=server -f - +``` + +## Deployment Process + +### Development/Testing (Push to Main) + +1. Make changes to application configuration +2. Commit and push to `main` branch +3. ArgoCD automatically syncs changes to the cluster + +### Production Release (Tagged Release) + +1. Build and test your application +2. Create a Git tag: `git tag myapp-v1.0.0` +3. Push tag: `git push --tags` +4. GitHub Actions builds and pushes Docker image +5. Workflow updates `kustomization.yaml` with new version +6. ArgoCD syncs the new version + +## Troubleshooting + +### Kustomize Errors + +```bash +# View detailed error output +kubectl kustomize applicationset/myapp + +# Check for YAML syntax errors +yamllint applicationset/myapp/*.yaml +``` + +### ArgoCD Sync Issues + +1. Check ArgoCD UI for sync status +2. Review ArgoCD application logs +3. Verify Git repository is accessible +4. Check resource quotas and RBAC permissions + +### Docker Build Failures + +1. Review GitHub Actions workflow logs +2. Verify Dockerfile syntax +3. Check base image availability +4. Ensure required build arguments are provided + +## Code Review Guidelines + +When reviewing PRs: + +- [ ] Kustomize configuration is valid +- [ ] Resources follow naming conventions +- [ ] Labels are consistent +- [ ] VirtualService routes are correct +- [ ] Resource limits are appropriate +- [ ] Secrets are not committed to Git +- [ ] Documentation is updated if needed + +## Resources + +- [Kustomize Documentation](https://kubectl.docs.kubernetes.io/references/kustomize/) +- [ArgoCD Documentation](https://argo-cd.readthedocs.io/) +- [Istio VirtualService](https://istio.io/latest/docs/reference/config/networking/virtual-service/) +- [Kubernetes Best Practices](https://kubernetes.io/docs/concepts/configuration/overview/) + +## Getting Help + +- Review existing applications for examples +- Check [k8s-templates/README.md](k8s-templates/README.md) for common patterns +- Open an issue for questions or problems +- Consult the team before making infrastructure changes + +## License + +This project follows the same license as specified in [LICENSE](LICENSE). diff --git a/README.md b/README.md index ee7f60b..cb90ac6 100644 --- a/README.md +++ b/README.md @@ -20,10 +20,23 @@ This repository follows the GitOps workflow to manage Kubernetes deployments. Al ├── applicationset # argocd applicationset generator │   ├── base # base kustomize template │   └── ... +├── k8s-templates # reusable kubernetes resource templates | └── changedetection # argocd managed application ``` +## Adding a New Application + +To add a new application to the repository: + +1. **Copy an existing application** from `applicationset/` that matches your needs +2. **Customize the configuration** (see [k8s-templates/README.md](k8s-templates/README.md) for common patterns) +3. **Test your kustomization**: `kubectl kustomize applicationset/myapp` +4. **Create a workflow** (optional): Copy one of the existing `.github/workflows/*.yaml` files if you need to build a custom Docker image +5. **Commit and let ArgoCD sync** your changes + +For detailed templates and examples, see [k8s-templates/README.md](k8s-templates/README.md). + ## Deployment To deploy an application, you can tag the repository follows the naming convention: `-`. This tag triggers the deployment process via GitHub Actions. diff --git a/applicationset/toolchain/kustomization.yaml b/applicationset/toolchain/kustomization.yaml index db2ea7d..fffe43f 100755 --- a/applicationset/toolchain/kustomization.yaml +++ b/applicationset/toolchain/kustomization.yaml @@ -2,6 +2,7 @@ apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../base +- vs.yaml namePrefix: toolchain- images: - name: ghcr.io/guyzsarun-lab/base diff --git a/k8s-templates/README.md b/k8s-templates/README.md new file mode 100644 index 0000000..4f09b57 --- /dev/null +++ b/k8s-templates/README.md @@ -0,0 +1,242 @@ +# Kubernetes Templates + +This directory contains reusable Kubernetes resource templates to reduce code duplication across the repository. + +## Overview + +The GitOps repository contains multiple applications with similar Kubernetes resources. These templates provide a starting point for new applications and document common patterns used across the repository. + +## Templates + +### VirtualService Templates + +#### virtualservice-base.yaml +Base template for a simple VirtualService that routes traffic to a single service. + +**Use case**: Simple applications exposed at a dedicated hostname (e.g., `app.proxmox.homelab`) + +**Key customization points**: +- `spec.hosts`: The hostname(s) for your application +- `spec.http.route.destination.host`: The Kubernetes service name +- `spec.http.route.destination.port.number`: The service port + +**Example VirtualService (copy and customize)**: +```yaml +apiVersion: networking.istio.io/v1beta1 +kind: VirtualService +metadata: + name: myapp-vs +spec: + gateways: + - istio-system/default-gateway + hosts: + - myapp.proxmox.homelab + http: + - match: + - uri: + prefix: / + route: + - destination: + host: myapp-application.default.svc.cluster.local + port: + number: 3000 +``` + +#### virtualservice-with-rewrite.yaml +Template for VirtualService with URI rewriting (for apps running under a path prefix). + +**Use case**: Applications served under a path on a shared gateway (e.g., `gateway.proxmox.homelab/myapp`) + +**Key customization points**: +- `spec.hosts`: The shared gateway hostname +- `spec.http.match.uri.prefix`: The URL path prefix (e.g., `/myapp`) +- `spec.http.route.destination.host`: The Kubernetes service name + +**Example VirtualService (copy and customize)**: +```yaml +apiVersion: networking.istio.io/v1beta1 +kind: VirtualService +metadata: + name: myapp-vs +spec: + gateways: + - istio-system/default-gateway + hosts: + - gateway.proxmox.homelab + http: + - match: + - uri: + prefix: /myapp + rewrite: + uri: / + route: + - destination: + host: myapp-application.default.svc.cluster.local + port: + number: 3000 +``` + +### Volume Templates + +#### nfs-volume-base.yaml +Base template for NFS-backed PersistentVolume and PersistentVolumeClaim. + +**Use case**: Applications requiring persistent storage backed by NFS + +**Key customization points**: +- `metadata.name`: Unique name for PV and PVC +- `spec.capacity.storage`: Storage size (e.g., `5Gi`, `10Gi`) +- `spec.storageClassName`: Unique storage class name +- `spec.nfs.path`: NFS export path + +**Example Volume (copy and customize)**: +```yaml +apiVersion: v1 +kind: PersistentVolume +metadata: + name: myapp-pv +spec: + capacity: + storage: 5Gi + volumeMode: Filesystem + storageClassName: myapp + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain + mountOptions: + - hard + - nfsvers=4.1 + nfs: + path: /home/devops/nfs_share/myapp + server: nfs.proxmox.homelab +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: myapp-pvc +spec: + storageClassName: myapp + accessModes: + - ReadWriteOnce + volumeMode: Filesystem + resources: + requests: + storage: 5Gi +``` + +### Service Port Patch Template + +Many applications use the same base service from `applicationset/base/service.yaml` but need different target ports. + +**Example service patch**: +```yaml +# patches/service.yaml +apiVersion: v1 +kind: Service +metadata: + labels: + app: application + name: application +spec: + ports: + - $patch: replace + - port: 3000 + protocol: TCP + targetPort: 8080 # Change to your app's port +``` + +### Deployment Resource Limits Patch Template + +**Example deployment resource limits patch**: +```yaml +# patches/deployment-memory.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: application +spec: + template: + spec: + containers: + - name: application + resources: + limits: + memory: 256Mi # Adjust as needed + cpu: 100m # Adjust as needed +``` + +## Common Patterns + +### Pattern 1: Simple Application with Dedicated Hostname + +``` +myapp/ +├── kustomization.yaml +└── vs.yaml +``` + +**kustomization.yaml**: +```yaml +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: +- ../base +- vs.yaml +namePrefix: myapp- +images: +- name: ghcr.io/guyzsarun-lab/base + newName: ghcr.io/guyzsarun-lab/myapp + newTag: v1.0.0 +commonLabels: + app: myapp +``` + +### Pattern 2: Application with Custom Port + +``` +myapp/ +├── kustomization.yaml +├── patches/ +│ ├── service.yaml +│ └── deployment.yaml +└── vs.yaml +``` + +### Pattern 3: Application with Persistent Storage + +``` +myapp/ +├── kustomization.yaml +├── volume.yaml +├── patches/ +│ └── deployment-volume.yaml +└── vs.yaml +``` + +## Benefits + +- **Reduced Duplication**: Common patterns are documented and reusable +- **Consistency**: All applications follow similar structures +- **Easier Onboarding**: New applications can copy existing patterns +- **Self-Documenting**: Templates serve as live documentation + +## Creating a New Application + +1. Copy one of the existing applications that most closely matches your needs +2. Update the application name throughout +3. Customize ports, hostnames, and resource limits +4. Add any application-specific configuration (secrets, configmaps, etc.) +5. Test with `kubectl kustomize applicationset/myapp` before committing +6. Add a GitHub workflow if building a custom Docker image + +## Validation + +Before committing changes, validate your Kustomize configuration: + +```bash +# Validate kustomization +kubectl kustomize applicationset/myapp + +# Check for common issues +kubectl kustomize applicationset/myapp | kubectl apply --dry-run=client -f - +``` diff --git a/k8s-templates/nfs-volume-base.yaml b/k8s-templates/nfs-volume-base.yaml new file mode 100644 index 0000000..2f40fe7 --- /dev/null +++ b/k8s-templates/nfs-volume-base.yaml @@ -0,0 +1,31 @@ +apiVersion: v1 +kind: PersistentVolume +metadata: + name: -pv +spec: + capacity: + storage: 10Gi + volumeMode: Filesystem + storageClassName: -storage + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain + mountOptions: + - hard + - nfsvers=4.1 + nfs: + path: /home/devops/nfs_share/ + server: nfs.proxmox.homelab +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: -pvc +spec: + storageClassName: -storage + accessModes: + - ReadWriteOnce + volumeMode: Filesystem + resources: + requests: + storage: 10Gi diff --git a/k8s-templates/patch-deployment-port.yaml b/k8s-templates/patch-deployment-port.yaml new file mode 100644 index 0000000..0129ce2 --- /dev/null +++ b/k8s-templates/patch-deployment-port.yaml @@ -0,0 +1,14 @@ +# Example patch to change deployment container port +# Copy to applicationset//patches/deployment.yaml and customize +apiVersion: apps/v1 +kind: Deployment +metadata: + name: application +spec: + template: + spec: + containers: + - name: application + ports: + - $patch: replace + - containerPort: 8080 # Change to your application's port diff --git a/k8s-templates/patch-deployment-resources.yaml b/k8s-templates/patch-deployment-resources.yaml new file mode 100644 index 0000000..3dfb46e --- /dev/null +++ b/k8s-templates/patch-deployment-resources.yaml @@ -0,0 +1,15 @@ +# Example patch to set resource limits +# Copy to applicationset//patches/deployment-memory.yaml and customize +apiVersion: apps/v1 +kind: Deployment +metadata: + name: application +spec: + template: + spec: + containers: + - name: application + resources: + limits: + memory: 256Mi # Adjust based on application needs + cpu: 100m # Adjust based on application needs diff --git a/k8s-templates/patch-service-port.yaml b/k8s-templates/patch-service-port.yaml new file mode 100644 index 0000000..bf7bc26 --- /dev/null +++ b/k8s-templates/patch-service-port.yaml @@ -0,0 +1,14 @@ +# Example patch to change service target port +# Copy to applicationset//patches/service.yaml and customize +apiVersion: v1 +kind: Service +metadata: + labels: + app: application + name: application +spec: + ports: + - $patch: replace + - port: 3000 + protocol: TCP + targetPort: 8080 # Change to your application's port diff --git a/k8s-templates/virtualservice-base.yaml b/k8s-templates/virtualservice-base.yaml new file mode 100644 index 0000000..8bdec1d --- /dev/null +++ b/k8s-templates/virtualservice-base.yaml @@ -0,0 +1,18 @@ +apiVersion: networking.istio.io/v1beta1 +kind: VirtualService +metadata: + name: vs +spec: + gateways: + - istio-system/default-gateway + hosts: + - .proxmox.homelab + http: + - match: + - uri: + prefix: / + route: + - destination: + host: -application.default.svc.cluster.local + port: + number: 3000 diff --git a/k8s-templates/virtualservice-with-rewrite.yaml b/k8s-templates/virtualservice-with-rewrite.yaml new file mode 100644 index 0000000..3fde96c --- /dev/null +++ b/k8s-templates/virtualservice-with-rewrite.yaml @@ -0,0 +1,20 @@ +apiVersion: networking.istio.io/v1beta1 +kind: VirtualService +metadata: + name: vs +spec: + gateways: + - istio-system/default-gateway + hosts: + - .proxmox.homelab + http: + - match: + - uri: + prefix: / + rewrite: + uri: / + route: + - destination: + host: -application.default.svc.cluster.local + port: + number: 3000 diff --git a/scripts/README.md b/scripts/README.md new file mode 100644 index 0000000..85d4b19 --- /dev/null +++ b/scripts/README.md @@ -0,0 +1,49 @@ +# Scripts + +This directory contains utility scripts for maintaining the GitOps repository. + +## Available Scripts + +### find-similar-files.sh + +Identifies files with similar content that might be candidates for refactoring into shared templates. + +**Usage**: +```bash +# Find all similar YAML files +./scripts/find-similar-files.sh "*.yaml" + +# Find similar files with specific pattern +./scripts/find-similar-files.sh "vs.yaml" + +# Find similar patches +./scripts/find-similar-files.sh "*service*.yaml" +``` + +**Output**: +- Groups of files with the same name +- List of patch files that might be consolidated +- Recommendations for reducing duplication + +**Use cases**: +- Periodic audits for code duplication +- Before major refactoring efforts +- When adding new applications to check for existing patterns + +## Contributing + +When adding new scripts: + +1. Include a header comment explaining what the script does +2. Add usage examples +3. Make scripts executable: `chmod +x scripts/your-script.sh` +4. Document the script in this README +5. Follow bash best practices (set -e, proper quoting, etc.) + +## Best Practices + +- Keep scripts simple and focused on one task +- Use meaningful variable names +- Include error handling +- Document expected inputs and outputs +- Test scripts before committing diff --git a/scripts/find-similar-files.sh b/scripts/find-similar-files.sh new file mode 100755 index 0000000..5758f31 --- /dev/null +++ b/scripts/find-similar-files.sh @@ -0,0 +1,70 @@ +#!/bin/bash +# +# Find similar files that might be candidates for refactoring +# Usage: ./scripts/find-similar-files.sh [pattern] +# +# This script helps identify duplicated code by finding files with similar content + +set -e + +PATTERN="${1:-*.yaml}" + +echo "Finding similar files matching pattern: $PATTERN" +echo "---" + +# Create a temporary directory for grouping files +tmpdir=$(mktemp -d) +trap 'rm -rf "$tmpdir"' EXIT + +# First pass: collect all files and group by basename +find . -name "$PATTERN" -type f ! -path "./.git/*" ! -path "./k8s-templates/*" -print0 | \ +while IFS= read -r -d '' file; do + basename_file=$(basename "$file") + # Create a list file for each basename + echo "$file" >> "$tmpdir/$basename_file.list" +done + +# Report on groups with multiple files +echo "File groups with potential duplication:" +echo "" + +for list_file in "$tmpdir"/*.list; do + [ -f "$list_file" ] || continue + + # Get the basename from the list filename + name=$(basename "$list_file" .list) + + # Count files in this group + count=$(wc -l < "$list_file") + + if [ "$count" -gt 1 ]; then + echo "Group: $name (${count} files)" + + # Show first few files as examples + head -5 "$list_file" | while IFS= read -r file; do + echo " - $file" + done + + if [ "$count" -gt 5 ]; then + echo " ... and $((count - 5)) more" + fi + + echo "" + fi +done + +# Report patches with similar patterns +echo "---" +echo "Patch files (potential for consolidation):" +find . -path "*/patches/*.yaml" ! -path "./.git/*" -print0 | while IFS= read -r -d '' file; do + basename_file=$(basename "$file") + echo " - $file ($basename_file)" +done | sort -k2 + +echo "" +echo "---" +echo "Recommendations:" +echo "1. Review file groups with multiple instances" +echo "2. Consider using k8s-templates/ for common patterns" +echo "3. Use Kustomize patches for minor variations" +echo "4. Document patterns in k8s-templates/README.md"