[WIP] Switch to using native SQLite bindings

[sam-lord]

This change allows both sqlite3 and node-sqlite bindings to work in Ghost.

It's a WIP, and a couple of tests are failing in the E2E suite. The main changes that aren't represented in the current commit are a switch to forked version of knex + other package updates that would be required.

Currently there a 3 tests failing in the e2e suite, and 3 failing in the legacy suite. All other tests pass ✅

• Fix remaining E2E test failures
• Fix remaining legacy test failures
• Update packages instead of using yarn link
• Get CI working

[coderabbitai]

Walkthrough

The changes introduce support for the node-sqlite database client alongside the existing sqlite3 client throughout the codebase. Conditional checks, configuration logic, and SQL execution paths that previously targeted only sqlite3 are updated to include node-sqlite. Database configuration files for testing environments are switched from sqlite3 to node-sqlite. The Node.js command scripts for development and testing add the --experimental-sqlite flag. Additionally, related test assertions and utility functions are updated to recognize node-sqlite as a valid client identifier. No changes are made to public API signatures or exported entities.

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 07ce402 and e5d570e.

⛔ Files ignored due to path filters (1)

• ghost/core/test/e2e-api/admin/__snapshots__/config.test.js.snap is excluded by !**/*.snap

📒 Files selected for processing (8)

• ghost/core/core/server/data/db/connection.js (1 hunks)
• ghost/core/core/server/data/schema/commands.js (3 hunks)
• ghost/core/core/server/services/url/Resources.js (1 hunks)
• ghost/core/core/shared/config/env/config.testing-browser.json (1 hunks)
• ghost/core/core/shared/config/env/config.testing.json (1 hunks)
• ghost/core/core/shared/config/utils.js (1 hunks)
• ghost/core/package.json (1 hunks)
• ghost/core/test/e2e-api/admin/config.test.js (1 hunks)

🧰 Additional context used

🧠 Learnings (5)

📓 Common learnings

Learnt from: mike182uk
PR: TryGhost/Ghost#22471
File: apps/admin-x-activitypub/src/utils/pending-activity.ts:13-71
Timestamp: 2025-03-13T09:00:20.205Z
Learning: The pending activity utilities in the Ghost ActivityPub module are covered by tests in the file `apps/admin-x-activitypub/test/unit/utils/pending-activity.ts`.

Learnt from: cmraible
PR: TryGhost/Ghost#23941
File: .github/workflows/ci.yml:911-914
Timestamp: 2025-06-19T22:57:05.880Z
Learning: In Ghost, when NODE_ENV is set to testing-mysql, Ghost uses config.testing-mysql.json configuration which sets the server port to 2369 instead of the default 2368. This also applies to other testing environments like testing and testing-browser.

ghost/core/core/shared/config/env/config.testing.json (1)

Learnt from: cmraible
PR: TryGhost/Ghost#23941
File: .github/workflows/ci.yml:911-914
Timestamp: 2025-06-19T22:57:05.880Z
Learning: In Ghost, when NODE_ENV is set to testing-mysql, Ghost uses config.testing-mysql.json configuration which sets the server port to 2369 instead of the default 2368. This also applies to other testing environments like testing and testing-browser.

ghost/core/core/shared/config/env/config.testing-browser.json (1)

Learnt from: cmraible
PR: TryGhost/Ghost#23941
File: .github/workflows/ci.yml:911-914
Timestamp: 2025-06-19T22:57:05.880Z
Learning: In Ghost, when NODE_ENV is set to testing-mysql, Ghost uses config.testing-mysql.json configuration which sets the server port to 2369 instead of the default 2368. This also applies to other testing environments like testing and testing-browser.

ghost/core/test/e2e-api/admin/config.test.js (2)

Learnt from: cmraible
PR: TryGhost/Ghost#23941
File: .github/workflows/ci.yml:911-914
Timestamp: 2025-06-19T22:57:05.880Z
Learning: In Ghost, when NODE_ENV is set to testing-mysql, Ghost uses config.testing-mysql.json configuration which sets the server port to 2369 instead of the default 2368. This also applies to other testing environments like testing and testing-browser.

Learnt from: ErisDS
PR: TryGhost/Ghost#23588
File: ghost/core/test/e2e-api/admin/backup.test.js:136-148
Timestamp: 2025-05-29T10:37:26.369Z
Learning: In the Ghost test framework, when testing CSV exports via the admin API, the response `body` field is empty while the actual CSV data is provided through the `text` field. Tests should use `expectEmptyBody()` and then validate the CSV content via `.expect(({text}) => ...)` - this is not contradictory behavior.

ghost/core/package.json (6)

Learnt from: cmraible
PR: TryGhost/Ghost#23941
File: .github/workflows/ci.yml:911-914
Timestamp: 2025-06-19T22:57:05.880Z
Learning: In Ghost, when NODE_ENV is set to testing-mysql, Ghost uses config.testing-mysql.json configuration which sets the server port to 2369 instead of the default 2368. This also applies to other testing environments like testing and testing-browser.

Learnt from: ErisDS
PR: TryGhost/Ghost#23582
File: ghost/core/.c8rc.json:24-24
Timestamp: 2025-05-29T07:45:35.714Z
Learning: In Ghost project, app.js files under core/server/web are intentionally excluded from unit test coverage because they are not easily unit-testable due to being entry points with initialization code and side effects.

Learnt from: mike182uk
PR: TryGhost/Ghost#22471
File: apps/admin-x-activitypub/src/utils/pending-activity.ts:13-71
Timestamp: 2025-03-13T09:00:20.205Z
Learning: The pending activity utilities in the Ghost ActivityPub module are covered by tests in the file `apps/admin-x-activitypub/test/unit/utils/pending-activity.ts`.

Learnt from: CR
PR: TryGhost/Ghost#0
File: .cursor/rules/yarn.mdc:0-0
Timestamp: 2025-06-30T08:27:05.754Z
Learning: This repository uses yarn v1 as a package manager.

Learnt from: CR
PR: TryGhost/Ghost#0
File: .cursor/rules/yarn.mdc:0-0
Timestamp: 2025-06-30T08:27:05.754Z
Learning: All commands are prefixed with `yarn` instead of `npm`

Learnt from: mike182uk
PR: TryGhost/Ghost#22471
File: apps/admin-x-activitypub/src/utils/pending-activity.ts:13-71
Timestamp: 2025-03-13T09:00:20.205Z
Learning: The pending activity utilities in Ghost's ActivityPub module are thoroughly tested in `apps/admin-x-activitypub/test/unit/utils/pending-activity.ts`, which covers `generatePendingActivityId`, `isPendingActivity`, and `generatePendingActivity` functions.

🧬 Code Graph Analysis (3)

ghost/core/core/shared/config/utils.js (1)

ghost/core/core/shared/config/loader.js (1)

• nconf (20-20)

ghost/core/core/server/data/db/connection.js (1)

ghost/core/core/server/services/url/Resources.js (1)

• config (6-6)

ghost/core/core/server/services/url/Resources.js (2)

ghost/core/core/server/data/db/connection.js (1)

• config (7-7)

ghost/core/core/frontend/web/site.js (1)

• config (8-8)

🔇 Additional comments (12)

ghost/core/core/shared/config/env/config.testing.json (1)

4-4: LGTM - Database client update is consistent with PR objectives.

The change from sqlite3 to node-sqlite aligns with the PR goal of enabling compatibility with native SQLite bindings. This configuration change is properly coordinated with corresponding updates in the codebase.

ghost/core/core/shared/config/env/config.testing-browser.json (1)

4-4: LGTM - Consistent database client configuration across testing environments.

The change maintains consistency with the config.testing.json file, ensuring both testing environments use the same node-sqlite client configuration.

ghost/core/test/e2e-api/admin/config.test.js (1)

40-40: LGTM - Test assertion updated to support new database client.

The regex pattern correctly includes node-sqlite as a valid database client identifier, ensuring the test passes when the system uses the new SQLite client.

ghost/core/package.json (2)

31-31: Verify the necessity of --experimental-sqlite flag for testing.

The --experimental-sqlite flag in the test base script should be consistent with the development environment. Ensure this doesn't introduce instability in the test suite.

---

24-24: Approve experimental-sqlite flag usage

The --experimental-sqlite flag is only applied to the dev and test:base scripts, which run in development or testing environments that may use Node’s built-in SQLite driver (node-sqlite). Production (client=mysql by default) isn’t using these scripts and won’t be affected. Given our engines.node requirement of ^22.13.1, this flag is supported and safe.

No further changes needed.

ghost/core/core/shared/config/utils.js (1)

73-73: LGTM - Path sanitization correctly extended to support node-sqlite.

The condition properly includes both sqlite3 and node-sqlite clients for path sanitization, which is necessary since both use file-based database connections that require absolute paths.

ghost/core/core/server/services/url/Resources.js (1)

125-125: LGTM! Clean extension of SQLite client support.

The change from a single client check to an array-based approach cleanly extends SQLite detection to include both sqlite3 and node-sqlite. This maintains the same pagination logic to prevent "too many SQL variables" errors for both SQLite client variants.

ghost/core/core/server/data/schema/commands.js (3)

509-509: LGTM! Consistent extension of SQLite client support.

The change correctly extends the SQLite client check to include node-sqlite alongside sqlite3. The raw SQL query using sqlite_master will work identically for both SQLite client variants.

---

527-527: LGTM! Consistent extension of SQLite client support.

The change correctly extends the SQLite client check to include node-sqlite. The pragma index_list command will work identically for both SQLite client variants.

---

545-545: LGTM! Consistent extension of SQLite client support.

The change correctly extends the SQLite client check to include node-sqlite. The pragma table_info command will work identically for both SQLite client variants.

ghost/core/core/server/data/db/connection.js (2)

20-20: LGTM! Consistent extension of SQLite client support.

The change correctly extends the SQLite client check to include node-sqlite alongside sqlite3 for the useNullAsDefault configuration.

---

27-47: Remove the unused node-sqlite branch – no such dependency in the repo

The client === 'node-sqlite' path is never exercised (there’s no node-sqlite or node:sqlite dependency in any package.json), so the manual conn.exec() branch is dead code and should be removed.

• File: ghost/core/core/server/data/db/connection.js
Lines ~36–47: drop the entire else if (client === 'node-sqlite') { … } block

Suggested diff:

@@ -33,8 +33,6 @@
                 if (client === 'sqlite3') {
                     conn.run('PRAGMA foreign_keys = ON', cb);
 
-                } else if (client === 'node-sqlite') {
-                    // node-sqlite uses exec for PRAGMA statements
-                    conn.exec('PRAGMA foreign_keys = ON');
-                    if (config.get('env').startsWith('testing')) {
-                        conn.exec('PRAGMA synchronous = OFF');
-                        conn.exec('PRAGMA journal_mode = TRUNCATE');
-                    }
-                    cb();
-                }
+                }
             }
         }

Likely an incorrect or invalid review comment.

✨ Finishing Touches

• 📝 Generate Docstrings

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[github-actions]

It looks like this PR contains a migration 👀
Here's the checklist for reviewing migrations:

General requirements

• ⚠️ Tested on the staging database servers
• Satisfies idempotency requirement (both up() and down())
• Does not reference models
• Filename is in the correct format (and correctly ordered)
• Targets the next minor version
• All code paths have appropriate log messages
• Uses the correct utils
• Contains a minimal changeset
• Does not mix DDL/DML operations
• Tested in MySQL and SQLite

Schema changes

• Both schema change and related migration have been implemented
• For index changes: has been performance tested for large tables
• For new tables/columns: fields use the appropriate predefined field lengths
• For new tables/columns: field names follow the appropriate conventions
• Does not drop a non-alpha table outside of a major version

Data changes

• Mass updates/inserts are batched appropriately
• Does not loop over large tables/datasets
• Defends against missing or invalid data
• For settings updates: follows the appropriate guidelines

[ErisDS]

Closing this for now.