Skip to content

Conversation

@robacourt
Copy link
Contributor

@robacourt robacourt commented Nov 13, 2025

Part of #3414, this PR sends individual operations from the ReplicationClient to ShapeLogCollector. This should reduce the working memory used by the ReplicationClient, but the ultimate aim is to send individual operations to storage and so not buffer at any point (unless subqueries are involved). This PR just moves the transaction consolidation one step, from ReplicationClient to ShapeLogCollector to keep the PR small. Other PRs will move the consolidation further, from ShapeLogCollector to Consumer, then finally from to Consumer to storage (except with subqueries are involved).

Currently the replication stream take two forms:

  1. Replication.Messages such as Messages.Begin, Messages.Commit, Messages.Insert etc.
  2. Changes.Transaction (or Changes.Relation)

And Collector converts one to the other.

This PR introduces an intermediate form, Operations, as Messages are coupled to the postgres replication protocol and are not optimised for use with Shapes (for example they do not include the relation name) and Transactions are whole transactions. So with this PR we have:

  1. Replication.Messages such as Messages.Begin, Messages.Commit, Messages.Insert etc.
  2. Operation (spec: Changes.operation()) such as Changes.Begin. Changes.Commit, Changes.NewRecord etc.
  3. Changes.Transaction (or Changes.Relation) which I'm calling database actions (spec: Changes.action())

Collector has been replaced by MessageConverter that converts 1 to 2 and TransactionBuilder that coverts 2 to 3.

@codecov
Copy link

codecov bot commented Nov 13, 2025

Codecov Report

❌ Patch coverage is 97.46835% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.39%. Comparing base (c4d0ea4) to head (406bd73).

Files with missing lines Patch % Lines
...c/postgres/replication_client/message_converter.ex 95.55% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3429      +/-   ##
==========================================
- Coverage   69.54%   69.39%   -0.16%     
==========================================
  Files         182      183       +1     
  Lines        9778     9752      -26     
  Branches      353      353              
==========================================
- Hits         6800     6767      -33     
- Misses       2976     2983       +7     
  Partials        2        2              
Flag Coverage Δ
elixir 66.33% <97.46%> (-0.19%) ⬇️
elixir-client 73.94% <ø> (-0.53%) ⬇️
packages/experimental 87.73% <ø> (ø)
packages/react-hooks 86.48% <ø> (ø)
packages/typescript-client 93.81% <ø> (ø)
packages/y-electric 55.12% <ø> (ø)
postgres-140000 65.38% <97.46%> (-0.10%) ⬇️
postgres-150000 ?
postgres-170000 65.36% <97.46%> (-0.27%) ⬇️
postgres-180000 65.41% <97.46%> (?)
sync-service 65.57% <97.46%> (-0.16%) ⬇️
typescript 87.19% <ø> (ø)
unit-tests 69.39% <97.46%> (-0.16%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@robacourt robacourt force-pushed the rob/collect-in-collector branch 2 times, most recently from 5424469 to 64c1852 Compare November 13, 2025 17:42
@robacourt robacourt force-pushed the rob/collect-in-collector branch from 64c1852 to e0ddbe2 Compare November 13, 2025 17:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants