Upgrade staging

.clinerules

@@ -4,8 +4,8 @@
 
 ### Package Management
 - Use pnpm for package management (pnpm@10.6.4)
-- Use Corepack for package manager versioning
-- Bun is used for running scripts and tests
+- Use Corepack for package manager version consistency
+- Bun is used for running scripts and tests only
 - Save type errors and dependency installations until ready to address them
 
 ### Database
@@ -22,15 +22,21 @@
 
 ### Code Organization
 - Use service-based architecture
-- Implement plugin system for extensibility
+- Implement plugin system with module federation for extensibility
 - Follow clean architecture principles
 - Use TypeScript strict mode
 
 ### Monorepo Structure
 - Use Turborepo for build orchestration and caching
-- Bun workspaces for dependency management
+- pnpm workspaces for dependency management
 - Separate backend and frontend as distinct workspaces
 
+### Build System
+- Use RSPack for backend building
+- Use RSBuild for frontend building
+- Support for module federation
+- Optimized for TypeScript
+
 ## Project Preferences
 
 ### Code Style
@@ -51,6 +57,18 @@
 - Handle nested stringified JSON
 - Implement proper error handling for malformed JSON
 
+### Deployment
+- Deploy to Railway using Docker containers
+- Use Kubernetes for orchestration
+- Implement CI/CD with GitHub Actions
+- Use environment-specific configurations
+
+### Security
+- Implement proper database security measures
+- Plan for Web3Auth integration in frontend
+- Use secure headers and CORS policies
+- Implement audit logging for sensitive operations
+
 ## Critical Implementation Paths
 
 ### Content Flow
@@ -60,8 +78,16 @@
 4. DistributionService → Distributor-specific transforms
 5. Distribution to configured channels
 
+### Recap Flow
+1. Scheduler service creates jobs based on feed configuration
+2. External scheduler triggers recap job via HTTP endpoint
+3. Recap job fetches approved submissions since last run
+4. ProcessorService applies batch transformations
+5. Distribution to configured channels
+6. Update last successful completion timestamp
+
 ### Plugin System
-1. PluginService loads plugins at runtime
+1. PluginService loads plugins at runtime via module federation
 2. Plugins register with appropriate service
 3. Services use plugins based on configuration
 4. Error handling and recovery for plugin failures
@@ -71,21 +97,43 @@
 2. Transaction support with retry logic
 3. Error handling and connection management
 4. Type-safe queries with Drizzle ORM
-
-## Known Challenges
-
-### Testing Infrastructure
-- Current testing approach relies too heavily on mocks
-- Need to implement Docker-based testing infrastructure
-- Need to refactor tests to use real database
-- Need to implement component tests for key flows
-
-### PostgreSQL Migration
-- Migration from SQLite to PostgreSQL in progress
-- Need to update database service implementation
-- Need to migrate existing data
-
-### Performance Optimization
-- Need to optimize resource usage
-- Need to implement performance monitoring
-- Need to identify and address bottlenecks
+5. Repository pattern for domain-specific operations
+6. State tracking for external scheduled jobs
+
+## Current Focus Areas
+
+### Recap Scheduling System
+- Implementing reliable recap generation for feeds
+- Using external scheduler service for job management
+- Tracking job state in the database for resilience
+- Supporting both cron expressions and interval-based schedules
+- Providing UI for managing recap configurations
+- Handling process restarts and maintaining job consistency
+
+### Comprehensive Error Handling
+- Implementing granular error types across the application
+- Developing consistent error recovery mechanisms
+- Enhancing error logging and monitoring
+- Creating user-friendly error messages
+- Implementing graceful degradation strategies
+
+### Configuration in Database
+- Migrating from JSON-based configuration to database storage
+- Implementing configuration versioning
+- Creating admin interface for configuration management
+- Ensuring backward compatibility
+- Implementing validation and security measures
+
+### Test Coverage
+- Expanding component tests for key flows
+- Implementing integration tests for external services
+- Adding E2E tests for critical user journeys
+- Improving test infrastructure with Docker
+- Implementing performance testing
+
+### Web3Auth Security
+- Implementing secure authentication with Web3Auth
+- Adding database protections for user data
+- Creating proper access control mechanisms
+- Implementing audit logging
+- Ensuring compliance with security best practices

.dockerignore

@@ -3,6 +3,7 @@
 
 # dependencies
 node_modules
+apps/**/node_modules
 .pnp
 **/.pnp.js
 **/.yarn/install-state.gz
@@ -18,6 +19,7 @@ out
 # production
 build
 **/**/dist
+apps/**/dist
 
 # misc
 **/.DS_Store
@@ -81,4 +83,4 @@ frontend/**/*.sw?
 fly.toml
 
 # workspace
-.github
+.github

.github/workflows/deployment.yml

@@ -14,7 +14,7 @@ jobs:
 
     strategy:
       matrix:
-        platform: [ linux/amd64 ]
+        platform: [linux/amd64]
 
     steps:
       - name: Checkout code
@@ -43,4 +43,3 @@ jobs:
           platforms: ${{ matrix.platform }}
           push: true
           tags: ghcr.io/${{ env.REPO }}:${{ github.sha }}
-

.github/workflows/near-rewards.yml

@@ -1,10 +1,10 @@
 name: NEAR Protocol Rewards Tracking
 on:
   schedule:
-    - cron: '0 */12 * * *'  # Every 12 hours
-  workflow_dispatch:        # Manual trigger
+    - cron: "0 */12 * * *" # Every 12 hours
+  workflow_dispatch: # Manual trigger
   push:
-    branches: [ main ]     # Start on main branch updates
+    branches: [main] # Start on main branch updates
 
 jobs:
   calculate-rewards:
@@ -17,12 +17,12 @@ jobs:
       - uses: actions/checkout@v4
       - uses: actions/setup-node@v4
         with:
-          node-version: '18'
+          node-version: "18"
 
       - name: Calculate Rewards
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           GITHUB_REPO: ${{ github.repository }}
         run: |
           npm install -g near-protocol-rewards@latest
-          near-protocol-rewards calculate
+          near-protocol-rewards calculate

.github/workflows/publish.yml

@@ -0,0 +1,31 @@
+# name: Release
+
+# on:
+#   push:
+#     branches:
+#       - main
+
+# jobs:
+#   release:
+#     runs-on: ubuntu-latest
+#     steps:
+#       - uses: actions/checkout@v4
+#       - uses: oven-sh/setup-bun@v2
+#       - run: bun install
+#       - run: bun run build
+#       - name: Setup Node.js
+#         uses: actions/setup-node@v4
+#         with:
+#           node-version: "lts/*"
+#           registry-url: 'https://registry.npmjs.org'
+#       - name: Publish
+#         env:
+#           NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
+#         run: |
+#           for pkg in packages/*; do
+#             if [ -f "$pkg/package.json" ]; then
+#               cd $pkg
+#               npm publish --access public || true
+#               cd ../..
+#             fi
+#           done

.github/workflows/test.yml

@@ -44,13 +44,13 @@ jobs:
         run: pnpm install
 
       - name: Migrate and seed DB
-        working-directory: ./backend
+        working-directory: ./api
         env:
           DATABASE_URL: postgresql://postgres:postgres@localhost:5433/test
         run: pnpm run db:migrate
 
       - name: Seed DB
-        working-directory: ./backend
+        working-directory: ./api
         env:
           DATABASE_URL: postgresql://postgres:postgres@localhost:5433/test
         run: bun ./test/setup/seed-test.ts

.gitignore

@@ -7,6 +7,7 @@
 .yarn/install-state.gz
 /.unlighthouse
 **/.pnpm-store
+node-compile-cache
 
 # testing
 /coverage
@@ -50,4 +51,4 @@ package-lock.json
 bun.lockb
 
 # JetBrains
-.idea/
+.idea/

.prettierignore

@@ -3,4 +3,5 @@ coverage
 node_modules
 .turbo
 .next
-.docusaurus
+.docusaurus
+packages/shared-db/migrations

.vscode/settings.json

@@ -0,0 +1,16 @@
+{
+  "cSpell.words": [
+    "crowdsource",
+    "ilhagirl",
+    "jpollock",
+    "karmaticacid",
+    "morica",
+    "murica",
+    "oklch",
+    "plugrel",
+    "plungrel",
+    "POTLOCK",
+    "SCURATE",
+    "yegorgolovnia"
+  ]
+}