Skip to content

improve: filter out generic keywords from smart clustering#49

Merged
jvalentini merged 8 commits into
mainfrom
improve-smart-clustering
Jan 6, 2026
Merged

improve: filter out generic keywords from smart clustering#49
jvalentini merged 8 commits into
mainfrom
improve-smart-clustering

Conversation

@jvalentini

Copy link
Copy Markdown
Owner

Problem

Smart clustering was generating useless cluster names like:

  • "Opencode & Analyzing (1 item)"
  • "Opencode & Background (2 items)"
  • "Jvalentini & Worklog" (from PR titles)

And meaningless connections showing shared generic keywords like "opencode" across all clusters.

Solution

  • Added comprehensive filter of generic keywords that don't provide meaningful themes
  • Improved theme generation to find meaningful keywords from item content when generic ones are filtered out
  • Enhanced connection logic to only show meaningful shared keywords between different clusters

Changes

  • Added GENERIC_KEYWORDS set with categorized generic terms (source-specific, work terms, GitHub terms, etc.)
  • Modified generateThemeLabel() to filter generic keywords and fallback to title analysis
  • Updated findCrossClusterConnections() to filter generic shared keywords and avoid duplicate connections

Testing

  • Eliminates the "Jvalentini & Worklog" clusters from PR title noise
  • Removes meaningless connections like "Shared focus: opencode"
  • Should produce more meaningful cluster themes based on actual work content

Justin Valentini and others added 8 commits January 6, 2026 10:56
- Add comprehensive list of generic keywords that don't provide meaningful cluster themes
- Improve theme generation to find meaningful keywords from item titles when needed
- Filter out generic shared keywords from cluster connections
- This eliminates useless cluster names like 'Opencode & Analyzing' and meaningless connections

Fixes the issue where all clusters were named 'Jvalentini & Worklog' due to generic terms in PR titles.
…r names

- Added comprehensive filtering of tool/framework names (npm, bun, react, etc.)
- Added filtering of generic adjectives/adverbs (new, good, better, quick, etc.)
- Improved theme generation to prefer longer, more specific words
- Enhanced fallback logic to find meaningful terms from content analysis
- More aggressive filtering to eliminate meaningless cluster names

This should produce cluster names like 'Productivity Analysis' instead of 'Jvalentini & Worklog'.
- Renamed conflicting 'allText' variable to 'fullContent'
- Fixes TypeScript compilation error
- Fixed type issues with regex match results that could be undefined
- Added explicit type guard for array filtering
- Ensures fallbackTokens is properly typed as string[]
- Fixed Biome linting error for arrow function parentheses
- Added explicit null checks for array access to satisfy TypeScript
- Add special handling for PR events with same PR number (90% similarity)
- Add special handling for events with same commit hash (95% similarity)
- This prevents duplicate clusters for PR opened/merged events
- Ensures related GitHub events are grouped together properly
- Added explicit null checks for array element access
- Ensures TypeScript compiler accepts array indexing operations
Add parentheses around arrow function parameters to satisfy Biome formatter requirements.
@jvalentini
jvalentini force-pushed the improve-smart-clustering branch from dfabaff to d33c68d Compare January 6, 2026 15:59
@jvalentini
jvalentini merged commit 4f57ad5 into main Jan 6, 2026
9 checks passed
@jvalentini
jvalentini deleted the improve-smart-clustering branch January 6, 2026 16:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant