feat(retention): walk activities incrementally with a persisted cursor

Grouping the whole activities table by thread every run is unusable on
a large instance: on a 312 GB database it plans as a seq scan plus a
50M-row on-disk sort. Instead, walk activities in id order (flake ids
are time-ordered) from a persisted cursor up to the id that corresponds
to the deadline, and verify only the threads those activities belong to
through the (type, context) index. Measured on that instance: 147 ms
for a 50k-activity range scan, 349 ms warm / 14 s cold to verify ~700
threads.

The cursor is persisted per chunk so a killed run resumes. The
watermark walk keeps a cursor of its own so it cannot leave the
age-based walk stranded behind the deadline. Adds the retention_cursors
table.
This commit is contained in:
Lain Soykaf 2026-09-02 12:32:09 +04:00
commit 6c8ed0a3b8
No known key found for this signature in database
9 changed files with 357 additions and 37 deletions

View file

@ -924,7 +924,7 @@ config :pleroma, :instances_favicons, enabled: false
config :pleroma, :retention,
enabled: false,
max_objects: nil,
batch_size: 500,
batch_size: 50_000,
keep_non_public: false
config :floki, :html_parser, Floki.HTMLParser.FastHtml

View file

@ -3334,8 +3334,10 @@ config :pleroma, :config_description, [
%{
key: :batch_size,
type: :integer,
description: "Maximum number of threads to evict per worker run. Default: 500.",
suggestions: [500]
description:
"Number of activities the worker walks per run; their threads are checked and evicted." <>
" Default: 50000.",
suggestions: [50_000]
},
%{
key: :keep_non_public,