feat(retention): walk activities incrementally with a persisted cursor
Grouping the whole activities table by thread every run is unusable on a large instance: on a 312 GB database it plans as a seq scan plus a 50M-row on-disk sort. Instead, walk activities in id order (flake ids are time-ordered) from a persisted cursor up to the id that corresponds to the deadline, and verify only the threads those activities belong to through the (type, context) index. Measured on that instance: 147 ms for a 50k-activity range scan, 349 ms warm / 14 s cold to verify ~700 threads. The cursor is persisted per chunk so a killed run resumes. The watermark walk keeps a cursor of its own so it cannot leave the age-based walk stranded behind the deadline. Adds the retention_cursors table.
This commit is contained in:
parent
c1f8d75019
commit
6c8ed0a3b8
9 changed files with 357 additions and 37 deletions
|
|
@ -3334,8 +3334,10 @@ config :pleroma, :config_description, [
|
|||
%{
|
||||
key: :batch_size,
|
||||
type: :integer,
|
||||
description: "Maximum number of threads to evict per worker run. Default: 500.",
|
||||
suggestions: [500]
|
||||
description:
|
||||
"Number of activities the worker walks per run; their threads are checked and evicted." <>
|
||||
" Default: 50000.",
|
||||
suggestions: [50_000]
|
||||
},
|
||||
%{
|
||||
key: :keep_non_public,
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue