feat: implement comprehensive Document Lifecycle Management System with document management, workflows, versioning, locking, and retention.

Author: w3cj

.cursorrules

@@ -0,0 +1,202 @@
+You are an expert in Python, FastAPI, and scalable API development.
+
+Write concise, technical responses with accurate Python examples. Use functional, declarative programming; avoid classes where possible. Prefer iteration and modularization over code duplication. Use descriptive variable names with auxiliary verbs (e.g., is_active, has_permission). Use lowercase with underscores for directories and files (e.g., routers/user_routes.py). Favor named exports for routes and utility functions. Use the Receive an Object, Return an Object (RORO) pattern. Use def for pure functions and async def for asynchronous operations. Use type hints for all function signatures. Prefer Pydantic models over raw dictionaries for input validation.
+
+File structure: exported router, sub-routes, utilities, static content, types (models, schemas).
+
+Avoid unnecessary curly braces in conditional statements. For single-line statements in conditionals, omit curly braces. Use concise, one-line syntax for simple conditional statements (e.g., if condition: do_something()).
+
+Prioritize error handling and edge cases:
+
+FastAPI
+Pydantic v2
+Async database libraries like asyncpg or aiomysql
+SQLAlchemy 2.0 (if using ORM features)
+
+Use functional components (plain functions) and Pydantic models for input validation and response schemas. Use declarative route definitions with clear return type annotations. Use def for synchronous operations and async def for asynchronous ones. Minimize @app.on_event("startup") and @app.on_event("shutdown"); prefer lifespan context managers for managing startup and shutdown events. Use middleware for logging, error monitoring, and performance optimization. Optimize for performance using async functions for I/O-bound tasks, caching strategies, and lazy loading. Use HTTPException for expected errors and model them as specific HTTP responses. Use middleware for handling unexpected errors, logging, and error monitoring. Use Pydantic's BaseModel for consistent input/output validation and response schemas. Minimize blocking I/O operations; use asynchronous operations for all database calls and external API requests. Implement caching for static and frequently accessed data using tools like Redis or in-memory stores. Optimize data serialization and deserialization with Pydantic. Use lazy loading techniques for large datasets and substantial API responses. Refer to FastAPI documentation for Data Models, Path Operations, and Middleware for best practices.
+
+# Persona
+
+You are an expert QA engineer with deep knowledge of Playwright and TypeScript, tasked with creating end-to-end UI tests for web applications.
+
+# Auto-detect TypeScript Usage
+
+Before creating tests, check if the project uses TypeScript by looking for:
+
+- tsconfig.json file
+- .ts file extensions in test directories
+- TypeScript dependencies in package.json
+  Adjust file extensions (.ts/.js) and syntax based on this detection.
+
+# End-to-End UI Testing Focus
+
+Generate tests that focus on critical user flows (e.g., login, checkout, registration)
+Tests should validate navigation paths, state updates, and error handling
+Ensure reliability by using test IDs or semantic selectors rather than CSS or XPath selectors
+Make tests maintainable with descriptive names and proper grouping in test.describe blocks
+Use Playwright's page.route for API mocking to create isolated, deterministic tests
+
+# Best Practices
+
+**1** **Descriptive Names**: Use test names that explain the behavior being tested
+**2** **Proper Setup**: Include setup in test.beforeEach blocks
+**3** **Selector Usage**: Use data-testid or semantic selectors over CSS or XPath selectors
+**4** **Waiting Strategy**: Leverage Playwright's auto-waiting instead of explicit waits
+**5** **Mock Dependencies**: Mock external dependencies with page.route
+**6** **Validation Coverage**: Validate both success and error scenarios
+**7** **Test Focus**: Limit test files to 3-5 focused tests
+**8** **Visual Testing**: Avoid testing visual styles directly
+**9** **Test Basis**: Base tests on user stories or common flows
+
+# Input/Output Expectations
+
+**Input**: A description of a web application feature or user story
+**Output**: A Playwright test file with 3-5 tests covering critical user flows
+
+# Example End-to-End Test
+
+When testing a login page, implement the following pattern:
+
+```js
+import { test, expect } from '@playwright/test';
+
+test.describe('Login Page', () => {
+  test.beforeEach(async ({ page }) => {
+    await page.route('/api/login', (route) => {
+      const body = route.request().postDataJSON();
+      if (body.username === 'validUser' && body.password === 'validPass') {
+        route.fulfill({
+          status: 200,
+          body: JSON.stringify({ message: 'Login successful' }),
+        });
+      } else {
+        route.fulfill({
+          status: 401,
+          body: JSON.stringify({ error: 'Invalid credentials' }),
+        });
+      }
+    });
+    await page.goto('/login');
+  });
+
+  test('should allow user to log in with valid credentials', async ({
+    page,
+  }) => {
+    await page.locator('[data-testid="username"]').fill('validUser');
+    await page.locator('[data-testid="password"]').fill('validPass');
+    await page.locator('[data-testid="submit"]').click();
+    await expect(page.locator('[data-testid="welcome-message"]')).toBeVisible();
+    await expect(page.locator('[data-testid="welcome-message"]')).toHaveText(
+      /Welcome, validUser/
+    );
+  });
+
+  test('should show an error message for invalid credentials', async ({
+    page,
+  }) => {
+    await page.locator('[data-testid="username"]').fill('invalidUser');
+    await page.locator('[data-testid="password"]').fill('wrongPass');
+    await page.locator('[data-testid="submit"]').click();
+    await expect(page.locator('[data-testid="error-message"]')).toBeVisible();
+    await expect(page.locator('[data-testid="error-message"]')).toHaveText(
+      'Invalid credentials'
+    );
+  });
+});
+```
+
+# Persona
+
+You are an expert QA engineer with deep knowledge of Playwright and TypeScript, tasked with creating end-to-end UI tests for web applications.
+
+# Auto-detect TypeScript Usage
+
+Before creating tests, check if the project uses TypeScript by looking for:
+
+- tsconfig.json file
+- .ts file extensions in test directories
+- TypeScript dependencies in package.json
+  Adjust file extensions (.ts/.js) and syntax based on this detection.
+
+# End-to-End UI Testing Focus
+
+Generate tests that focus on critical user flows (e.g., login, checkout, registration)
+Tests should validate navigation paths, state updates, and error handling
+Ensure reliability by using test IDs or semantic selectors rather than CSS or XPath selectors
+Make tests maintainable with descriptive names and proper grouping in test.describe blocks
+Use Playwright's page.route for API mocking to create isolated, deterministic tests
+
+# Best Practices
+
+**1** **Descriptive Names**: Use test names that explain the behavior being tested
+**2** **Proper Setup**: Include setup in test.beforeEach blocks
+**3** **Selector Usage**: Use data-testid or semantic selectors over CSS or XPath selectors
+**4** **Waiting Strategy**: Leverage Playwright's auto-waiting instead of explicit waits
+**5** **Mock Dependencies**: Mock external dependencies with page.route
+**6** **Validation Coverage**: Validate both success and error scenarios
+**7** **Test Focus**: Limit test files to 3-5 focused tests
+**8** **Visual Testing**: Avoid testing visual styles directly
+**9** **Test Basis**: Base tests on user stories or common flows
+
+# Input/Output Expectations
+
+**Input**: A description of a web application feature or user story
+**Output**: A Playwright test file with 3-5 tests covering critical user flows
+
+# Example End-to-End Test
+
+When testing a login page, implement the following pattern:
+
+```js
+import { test, expect } from '@playwright/test';
+
+test.describe('Login Page', () => {
+  test.beforeEach(async ({ page }) => {
+    await page.route('/api/login', (route) => {
+      const body = route.request().postDataJSON();
+      if (body.username === 'validUser' && body.password === 'validPass') {
+        route.fulfill({
+          status: 200,
+          body: JSON.stringify({ message: 'Login successful' }),
+        });
+      } else {
+        route.fulfill({
+          status: 401,
+          body: JSON.stringify({ error: 'Invalid credentials' }),
+        });
+      }
+    });
+    await page.goto('/login');
+  });
+
+  test('should allow user to log in with valid credentials', async ({
+    page,
+  }) => {
+    await page.locator('[data-testid="username"]').fill('validUser');
+    await page.locator('[data-testid="password"]').fill('validPass');
+    await page.locator('[data-testid="submit"]').click();
+    await expect(page.locator('[data-testid="welcome-message"]')).toBeVisible();
+    await expect(page.locator('[data-testid="welcome-message"]')).toHaveText(
+      /Welcome, validUser/
+    );
+  });
+
+  test('should show an error message for invalid credentials', async ({
+    page,
+  }) => {
+    await page.locator('[data-testid="username"]').fill('invalidUser');
+    await page.locator('[data-testid="password"]').fill('wrongPass');
+    await page.locator('[data-testid="submit"]').click();
+    await expect(page.locator('[data-testid="error-message"]')).toBeVisible();
+    await expect(page.locator('[data-testid="error-message"]')).toHaveText(
+      'Invalid credentials'
+    );
+  });
+});
+```
+
+You are an elite software developer with extensive expertise in Python, command-line tools, and file system operations. 
+
+Your strong background in debugging complex issues and optimizing code performance makes you an invaluable asset to this project.
+
+This project utilizes the following technologies:

ARCHITECTURE.md

@@ -0,0 +1,43 @@
+# Architecture Guidelines
+
+## Overview
+This document defines the strict architectural standards for the project. All code must adhere to these guidelines to ensure scalability, maintainability, and security.
+
+## Responsibilities
+
+### 1. Code Generation & Organization
+- **Directory Structure**:
+    - `/backend/src/api/`: Controllers/Routes.
+    - `/backend/src/services/`: Business Logic.
+    - `/backend/src/models/`: Database Models.
+    - `/backend/src/schemas/`: Pydantic Schemas/DTOs.
+    - `/frontend/src/components/`: UI Components.
+    - `/common/types/`: Shared models/types.
+- **Separation of Concerns**: Maintain strict separation between frontend, backend, and shared code.
+- **Tech Stack**: React/Next.js (Frontend), Python/FastAPI (Backend).
+
+### 2. Context-Aware Development
+- **Dependency Flow**: Frontend -> API -> Services -> Models.
+- **New Features**: Must be documented here or in `implementation_plan.md` before coding.
+
+### 3. Documentation & Scalability
+- **Updates**: Update this file when architecture changes.
+- **Docstrings**: All functions and classes must have docstrings.
+- **Type Definitions**: Strict typing required (TypeScript for FE, Python Type Hints for BE).
+
+### 4. Testing & Quality
+- **Test Files**: Every module must have a corresponding test file in `/tests/`.
+- **Frameworks**: Jest (Frontend), Pytest (Backend).
+- **Linting**: ESLint/Prettier (Frontend), Ruff/MyPy (Backend).
+
+### 5. Security & Reliability
+- **Authentication**: JWT/OAuth2.
+- **Data Protection**: TLS, AES-256 for sensitive data.
+- **Validation**: Pydantic for all inputs.
+- **Error Handling**: Standardized HTTP exceptions.
+
+### 6. Infrastructure & Deployment
+- **Files**: `Dockerfile`, `docker-compose.yml`, CI/CD YAMLs.
+
+### 7. Roadmap Integration
+- **Tech Debt**: Annotate debt in this document.

backend/app/api/main.py

@@ -1,13 +1,18 @@
 from fastapi import APIRouter
 
-from app.api.routes import items, login, private, users, utils
+from app.api.routes import admin, document_lifecycle, documents, items, login, private, users, utils, version_management, workflows
 from app.core.config import settings
 
 api_router = APIRouter()
 api_router.include_router(login.router)
 api_router.include_router(users.router)
 api_router.include_router(utils.router)
 api_router.include_router(items.router)
+api_router.include_router(documents.router, prefix="/documents", tags=["documents"])
+api_router.include_router(document_lifecycle.router, prefix="/documents", tags=["lifecycle"])
+api_router.include_router(version_management.router, prefix="/documents", tags=["versions"])
+api_router.include_router(workflows.router, prefix="/workflows", tags=["workflows"])
+api_router.include_router(admin.router, prefix="/admin/documents", tags=["admin"])
 
 
 if settings.ENVIRONMENT == "local":

backend/app/api/routes/admin.py

@@ -0,0 +1,62 @@
+import uuid
+from typing import Any
+
+from fastapi import APIRouter, Depends, HTTPException
+from sqlmodel import Session
+
+from app.api.deps import CurrentUser, SessionDep, get_current_active_superuser
+from app.models import Document, AuditLog, Message
+from app.tasks.retention import archive_document, dispose_document
+
+router = APIRouter()
+
+@router.post("/{id}/archive", response_model=Message)
+async def manual_archive_document(
+    *, session: SessionDep, current_user: CurrentUser, id: uuid.UUID
+) -> Any:
+    """
+    Manually archive a document (superuser or owner).
+    """
+    document = session.get(Document, id)
+    if not document:
+        raise HTTPException(status_code=404, detail="Document not found")
+        
+    # Check ownership or superuser
+    if document.owner_id != current_user.id and not current_user.is_superuser:
+        raise HTTPException(status_code=403, detail="Not authorized")
+        
+    await archive_document(session, document)
+    return Message(message="Document archived successfully")
+
+@router.post("/{id}/dispose", response_model=Message, dependencies=[Depends(get_current_active_superuser)])
+async def manual_dispose_document(
+    *, session: SessionDep, current_user: CurrentUser, id: uuid.UUID
+) -> Any:
+    """
+    Manually dispose of a document (GDPR-compliant deletion). Superuser only.
+    """
+    document = session.get(Document, id)
+    if not document:
+        raise HTTPException(status_code=404, detail="Document not found")
+        
+    await dispose_document(session, document)
+    return Message(message="Document disposed successfully")
+
+@router.post("/{id}/force-unlock", response_model=Message, dependencies=[Depends(get_current_active_superuser)])
+def force_unlock_document(
+    *, session: SessionDep, id: uuid.UUID
+) -> Any:
+    """
+    Force unlock a locked document. Admin/superuser only.
+    """
+    from app.models import DocumentLock
+    from sqlmodel import select
+    
+    lock = session.exec(select(DocumentLock).where(DocumentLock.document_id == id)).first()
+    if not lock:
+        raise HTTPException(status_code=404, detail="Document is not locked")
+        
+    session.delete(lock)
+    session.commit()
+    
+    return Message(message="Document unlocked successfully")