Skip to content

Add an LLM based spam filter#96

Merged
daniel-lerch merged 10 commits into
masterfrom
spamfilter
Mar 31, 2026
Merged

Add an LLM based spam filter#96
daniel-lerch merged 10 commits into
masterfrom
spamfilter

Conversation

@daniel-lerch
Copy link
Copy Markdown
Owner

@daniel-lerch daniel-lerch commented Mar 31, 2026

This pull request adds a spam filter that can be enabled for each distribution list individually and classifies emails using Mistral AI.

Resolves #94

@daniel-lerch daniel-lerch requested a review from Copilot March 31, 2026 16:02
@daniel-lerch daniel-lerch self-assigned this Mar 31, 2026
@daniel-lerch daniel-lerch added the enhancement New feature or request label Mar 31, 2026
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces an optional, per-distribution-list spam filter backed by Mistral AI, and replaces the previous single newsletter boolean with a more extensible flags object for distribution list behavior.

Changes:

  • Add flags (override recipient, spam filter) to distribution list API + web UI and update forwarding behavior accordingly.
  • Implement server-side LLM spam classification with MIME text extraction (plain + HTML) and persist classification results on inbound emails.
  • Add EF Core migration for spam classification fields and add tests/resources for spam filtering and text extraction.

Reviewed changes

Copilot reviewed 27 out of 30 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
webapp/src/views/EditDistributionList.vue Replaces newsletter with two flag checkboxes and sends flags payload on save.
webapp/src/views/DistributionLists.vue Displays icons for enabled distribution list flags in the table.
webapp/src/services/distribution-list.ts Changes create/modify request type to include flags object.
webapp/package-lock.json Updates minimum Node engine version.
server/Mailist/SpamFilter/SpamFilterService.cs New service that builds prompts, calls the LLM, and deserializes classification JSON.
server/Mailist/SpamFilter/SpamFilterOptions.cs Adds configuration options for enabling and configuring the spam filter.
server/Mailist/SpamFilter/SpamCategory.cs Defines classification categories used by the spam filter and email processing.
server/Mailist/SpamFilter/MimeTextExtractionService.cs Extracts text from MIME bodies (plain/html) for LLM input.
server/Mailist/Models/Json/DistributionList.cs Replaces Newsletter with a Flags object in API models; adds conversion helpers.
server/Mailist/Migrations/DatabaseContextModelSnapshot.cs Updates snapshot for unique index change and new inbox email spam fields.
server/Mailist/Migrations/20260331155808_SpamFilter.Designer.cs New migration designer snapshot for schema changes.
server/Mailist/Migrations/20260331155808_SpamFilter.cs Adds spam category/justification columns; switches alias uniqueness constraint to unique index.
server/Mailist/Mailist.csproj Adds AngleSharp + Mistral SDK dependencies; bumps JWT package version.
server/Mailist/Extensions/IServiceCollectionExtensions.cs Binds and validates SpamFilter options section.
server/Mailist/EmailRelay/MimeMessageCreationService.cs Adds rejection message generation for spam-filtered emails.
server/Mailist/EmailRelay/Entities/InboxEmail.cs Adds SpamCategory and SpamJustification fields to the inbox email entity.
server/Mailist/EmailRelay/EmailRelayJobController.cs Integrates spam classification into the relay pipeline and updates forwarding flag behavior.
server/Mailist/EmailRelay/DistributionListFlags.cs Renames newsletter flag to OverrideRecipient and adds SpamFilter.
server/Mailist/DatabaseContext.cs Updates EF mappings for distribution list alias uniqueness and spam category conversion.
server/Mailist/Controllers/DistributionListController.cs Updates API to accept/return Flags object instead of Newsletter.
server/Mailist/appsettings.json Adds SpamFilter configuration block (disabled by default).
server/Mailist.Tests/SpamFilterServiceTests.cs Adds (optional) tests that call the real LLM when enabled via config.
server/Mailist.Tests/Resources/23437.header Adds embedded test email header fixture (spam example).
server/Mailist.Tests/Resources/23437.body Adds embedded test email body fixture (spam example).
server/Mailist.Tests/Resources/23377.header Adds embedded test email header fixture (legit example).
server/Mailist.Tests/Resources/23377.body Adds embedded test email body fixture (legit example).
server/Mailist.Tests/MimeTextExtractionServiceTests.cs Adds unit test validating HTML-to-text compression.
server/Mailist.Tests/Mailist.Tests.csproj Embeds .body/.header resources into the test assembly.
server/Mailist.Tests/Integration/DistributionListIntegrationTests.cs Updates integration tests to use new Flags API.
.vscode/settings.json Adds Tailwind CSS file association.
Files not reviewed (2)
  • server/Mailist/Migrations/20260331155808_SpamFilter.Designer.cs: Language not supported
  • webapp/package-lock.json: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread webapp/src/views/EditDistributionList.vue
Comment thread server/Mailist/EmailRelay/EmailRelayJobController.cs
Comment thread server/Mailist/EmailRelay/EmailRelayJobController.cs Outdated
Comment thread server/Mailist/SpamFilter/SpamFilterService.cs
Comment thread server/Mailist/SpamFilter/SpamFilterService.cs
Comment thread server/Mailist/SpamFilter/MimeTextExtractionService.cs Outdated
Comment thread server/Mailist/appsettings.json Outdated
Comment thread server/Mailist.Tests/SpamFilterServiceTests.cs Outdated
@daniel-lerch daniel-lerch merged commit 1498c75 into master Mar 31, 2026
3 checks passed
@daniel-lerch daniel-lerch deleted the spamfilter branch March 31, 2026 18:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Experimental spam filter

2 participants