Add comprehensive beginner-friendly documentation package

Author: deepsm007

docs/ARCHITECTURE.md

@@ -0,0 +1,388 @@
+# Codebase Walkthrough
+
+## Repository Structure
+
+The CI-Tools repository follows a standard Go project layout with clear separation of concerns:
+
+```
+ci-tools/
+├── cmd/                    # Command-line tools and applications
+├── pkg/                    # Shared libraries and packages
+├── test/                   # Test files and test data
+├── hack/                   # Build scripts and utilities
+├── images/                 # Container image definitions
+├── vendor/                 # Vendored dependencies
+├── docs/                   # Documentation (this directory)
+├── Makefile               # Build automation
+├── go.mod                  # Go module definition
+├── go.sum                  # Go module checksums
+├── README.md              # Main README
+├── CONTRIBUTING.md        # Contribution guidelines
+└── OWNERS                 # Code owners file
+```
+
+## Directory Details
+
+### `/cmd` - Command-Line Tools
+
+This directory contains all executable tools. Each subdirectory represents a standalone command-line application.
+
+#### Key Tools by Category
+
+**Core CI Tools:**
+- `ci-operator/` - Main CI orchestration engine
+- `ci-operator-prowgen/` - Generates Prow jobs from ci-operator configs
+- `ci-operator-checkconfig/` - Validates ci-operator configurations
+- `ci-operator-configresolver/` - Resolves ci-operator config references
+
+**Configuration Management:**
+- `config-brancher/` - Propagates configs to release branches
+- `autoconfigbrancher/` - Orchestrates multiple config tools
+- `determinize-ci-operator/` - Normalizes CI configs
+- `ci-operator-yaml-creator/` - Creates CI configs from templates
+
+**Image Management:**
+- `registry-replacer/` - Replaces registry references
+- `ci-images-mirror/` - Mirrors images between registries
+- `promoted-image-governor/` - Manages image promotion
+- `clusterimageset-updater/` - Updates cluster image sets
+
+**Secret Management:**
+- `ci-secret-generator/` - Generates secrets from templates
+- `ci-secret-bootstrap/` - Bootstraps secrets for repos
+- `gsm-secret-sync/` - Syncs secrets from Google Secret Manager
+- `vault-secret-collection-manager/` - Web UI for Vault secrets
+
+**Cluster Management:**
+- `cluster-init/` - Initializes test clusters
+- `dptp-controller-manager/` - Main Kubernetes controller manager
+- `cluster-display/` - Displays cluster information
+
+**Job Management:**
+- `prow-job-dispatcher/` - Dispatches jobs to clusters
+- `job-run-aggregator/` - Aggregates job run data
+- `result-aggregator/` - Aggregates test results
+- `retester/` - Retests failed jobs
+
+**Developer Tools:**
+- `repo-init/` - Web UI/API for repo initialization
+- `prcreator/` - Creates pull requests
+- `backport-verifier/` - Verifies backports
+- `blocking-issue-creator/` - Creates blocking issues
+
+**Analytics & Monitoring:**
+- `pod-scaler/` - Analyzes and recommends resource usage
+- `lensserver/` - Provides CI insights
+- `job-run-aggregator/` - Job performance analysis
+
+#### Tool Structure Pattern
+
+Each tool typically follows this structure:
+
+```
+cmd/tool-name/
+├── main.go              # Entry point
+├── README.md           # Tool-specific documentation
+├── testdata/           # Test data (if applicable)
+└── ...                 # Additional tool-specific files
+```
+
+### `/pkg` - Shared Packages
+
+This directory contains reusable Go packages used across multiple tools.
+
+#### Key Packages
+
+**Core API:**
+- `pkg/api/` - Core data structures and types
+  - `config.go` - CI operator configuration types
+  - `types.go` - Common type definitions
+  - `graph.go` - Dependency graph implementation
+  - `promotion.go` - Image promotion logic
+
+**Configuration:**
+- `pkg/config/` - Configuration loading and management
+- `pkg/defaults/` - Default value handling
+- `pkg/validation/` - Configuration validation
+
+**CI Operator Components:**
+- `pkg/steps/` - Step execution framework
+- `pkg/registry/` - Step registry management
+- `pkg/prowgen/` - Prow job generation
+- `pkg/rehearse/` - Job rehearsal logic
+
+**Kubernetes Integration:**
+- `pkg/kubernetes/` - Kubernetes client utilities
+- `pkg/controller/` - Controller implementations
+- `pkg/dispatcher/` - Job dispatching logic
+
+**Utilities:**
+- `pkg/util/` - General utilities
+- `pkg/junit/` - JUnit XML handling
+- `pkg/github/` - GitHub API integration
+- `pkg/jira/` - Jira integration
+- `pkg/slack/` - Slack integration
+
+**Specialized:**
+- `pkg/jobrunaggregator/` - Job run analysis
+- `pkg/pod-scaler/` - Resource scaling logic
+- `pkg/clusterinit/` - Cluster initialization
+- `pkg/gsm-secrets/` - Google Secret Manager integration
+
+### `/test` - Test Files
+
+Contains integration tests, end-to-end tests, and test data:
+
+- `test/integration/` - Integration test suites
+- `test/e2e/` - End-to-end tests
+- Test data files for various tools
+
+### `/hack` - Build Scripts
+
+Utility scripts for development and CI:
+
+- `hack/install.sh` - Installation script
+- `hack/test-go.sh` - Go test runner
+- `hack/lint.sh` - Linting script
+- `hack/update-codegen.sh` - Code generation
+- `hack/build-and-push.sh` - Image build script
+
+### `/images` - Container Images
+
+Container image definitions for various tools (67 files).
+
+## Key Files and Modules
+
+### Main Entry Points
+
+**CI Operator (`cmd/ci-operator/main.go`):**
+- Main orchestration engine
+- Parses YAML configurations
+- Builds dependency graphs
+- Executes build and test steps
+- Handles image promotion
+
+**Controller Manager (`cmd/dptp-controller-manager/main.go`):**
+- Runs multiple Kubernetes controllers
+- Manages CI resources
+- Handles reconciliation loops
+
+### Core Components
+
+**API Package (`pkg/api/`):**
+- `config.go` - CI operator configuration structures
+- `graph.go` - Dependency graph for build steps
+- `types.go` - Common type definitions
+- `promotion.go` - Image promotion logic
+
+**Steps Package (`pkg/steps/`):**
+- Step execution framework
+- Individual step implementations (build, test, template, etc.)
+- Step registry for reusable steps
+
+**Prowgen Package (`pkg/prowgen/`):**
+- Converts ci-operator configs to Prow job configs
+- Handles job naming and organization
+- Manages job dependencies
+
+## Component Interactions
+
+### How CI Operator Works
+
+1. **Configuration Loading** (`pkg/config/load.go`):
+   - Loads YAML configuration
+   - Resolves references and includes
+   - Validates configuration
+
+2. **Graph Building** (`pkg/api/graph.go`):
+   - Builds dependency graph from config
+   - Determines execution order
+   - Identifies parallel execution opportunities
+
+3. **Step Execution** (`pkg/steps/`):
+   - Executes steps in dependency order
+   - Manages Kubernetes resources
+   - Collects artifacts
+   - Handles errors and retries
+
+4. **Image Promotion** (`pkg/promotion/`):
+   - Tags images to release streams
+   - Updates ImageStreams
+   - Handles promotion policies
+
+### How Configuration Management Works
+
+1. **Config Brancher** (`cmd/config-brancher/`):
+   - Reads ci-operator configs from openshift/release
+   - Identifies repos promoting to current release
+   - Copies configs to future release branches
+   - Disables promotion conflicts
+
+2. **Prowgen** (`cmd/ci-operator-prowgen/`):
+   - Reads ci-operator configs
+   - Generates Prow job YAML files
+   - Organizes jobs by org/repo/branch
+   - Handles periodic, presubmit, postsubmit jobs
+
+3. **Autoconfigbrancher** (`cmd/autoconfigbrancher/`):
+   - Orchestrates multiple config tools
+   - Runs tools in sequence
+   - Creates PRs with changes
+   - Auto-merges approved PRs
+
+### How Controllers Work
+
+1. **Controller Manager** (`cmd/dptp-controller-manager/`):
+   - Initializes multiple controllers
+   - Manages controller lifecycle
+   - Handles graceful shutdown
+
+2. **Individual Controllers** (`pkg/controller/`):
+   - Watch Kubernetes resources
+   - Reconcile desired state
+   - Handle errors and retries
+   - Update resource status
+
+## Key Classes and Functions
+
+### CI Operator Core
+
+**Main Function** (`cmd/ci-operator/main.go:main`):
+- Entry point for ci-operator
+- Parses flags and configuration
+- Initializes clients
+- Executes build workflow
+
+**Run Function** (`cmd/ci-operator/main.go:run`):
+- Main execution logic
+- Builds dependency graph
+- Executes steps
+- Handles promotion
+
+**Graph Building** (`pkg/api/graph.go`):
+- `BuildPartialGraph()` - Builds graph for specific targets
+- `TopologicalSort()` - Sorts nodes by dependencies
+- `ResolveMultiArch()` - Resolves multi-arch requirements
+
+**Step Execution** (`pkg/steps/run.go`):
+- `Run()` - Executes step graph
+- `executeStep()` - Executes individual step
+- `collectArtifacts()` - Collects step artifacts
+
+### Configuration Management
+
+**Config Loading** (`pkg/config/load.go`):
+- `LoadConfig()` - Loads CI operator config
+- `LoadConfigByFilename()` - Loads specific config file
+- `ResolveConfig()` - Resolves config references
+
+**Prowgen** (`pkg/prowgen/jobconfig.go`):
+- `GenerateJobs()` - Generates Prow jobs from config
+- `GeneratePresubmits()` - Generates presubmit jobs
+- `GeneratePostsubmits()` - Generates postsubmit jobs
+- `GeneratePeriodics()` - Generates periodic jobs
+
+### Controller Pattern
+
+**Reconciler Interface** (`pkg/controller/util/reconciler.go`):
+- `Reconcile()` - Main reconciliation function
+- Standard interface for all controllers
+
+**Promotion Reconciler** (`pkg/controller/promotionreconciler/`):
+- Watches ImageStreams
+- Promotes images based on policies
+- Updates release streams
+
+**Ephemeral Cluster Controller** (`pkg/controller/ephemeralcluster/`):
+- Manages ephemeral test clusters
+- Provisions clusters on demand
+- Cleans up expired clusters
+
+## API Surface
+
+### CI Operator API
+
+**Configuration Structure** (`pkg/api/config.go`):
+```go
+type ReleaseBuildConfiguration struct {
+    // Input configuration
+    InputConfiguration
+    
+    // Build configuration
+    BuildRoot
+    Images
+    Tests
+    
+    // Promotion configuration
+    Promotion
+    
+    // Release configuration
+    ReleaseTagConfiguration
+}
+```
+
+**Step Interface** (`pkg/steps/step.go`):
+```go
+type Step interface {
+    Inputs() []api.InputDefinition
+    Validate() error
+    Run(ctx context.Context) error
+    Requires() []api.StepLink
+    Creates() []api.StepLink
+    Provides() (api.ParameterMap, error)
+}
+```
+
+### Controller API
+
+**Reconciler Interface**:
+```go
+type Reconciler interface {
+    Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)
+}
+```
+
+## Data Flow
+
+1. **Configuration Flow:**
+   - YAML files in openshift/release
+   - Loaded by config tools
+   - Processed and validated
+   - Generated into Prow jobs
+   - Applied to Prow
+
+2. **Execution Flow:**
+   - Prow triggers job
+   - CI Operator loads config
+   - Builds dependency graph
+   - Executes steps
+   - Promotes images
+   - Reports results
+
+3. **Image Flow:**
+   - Built in cluster registry
+   - Tagged to local ImageStream
+   - Promoted to release stream
+   - Mirrored to external registry
+
+## Testing Structure
+
+- **Unit Tests**: `*_test.go` files alongside source
+- **Integration Tests**: `test/integration/` directory
+- **E2E Tests**: `test/e2e/` directory
+- **Test Data**: `testdata/` directories in tool packages
+
+## Build System
+
+- **Makefile**: Main build automation
+- **Go Modules**: Dependency management
+- **Docker**: Container image builds
+- **CI/CD**: Automated testing and deployment
+
+## Extension Points
+
+1. **Custom Steps**: Add to `pkg/steps/`
+2. **Custom Controllers**: Add to `pkg/controller/`
+3. **Custom Tools**: Add to `cmd/`
+4. **Custom Validators**: Add to `pkg/validation/`
+