fix(retention): pin threads addressed to a local user

A mention or DM was only pinned by its notification, so clearing
notifications let retention evict a user's direct messages. Check the
activity recipients instead, which does not go away.

changelog.d/retention-mode.add

@@ -1 +1 @@
-Add a bounded retention mode (`config :pleroma, :retention`): a cron worker walks old activities in id order and evicts remote threads nobody local interacted with, in bounded batches, so the database plateaus instead of growing forever. Adds a `retention_cursors` table. `prune_objects --keep-threads` now also keeps threads that mention a local user or contain a reported post.
+Add a bounded retention mode (`config :pleroma, :retention`): a cron worker walks old activities in id order and evicts remote threads nobody local interacted with, in bounded batches, so the database plateaus instead of growing forever. Adds a `retention_cursors` table. `prune_objects --keep-threads` now also keeps threads that are addressed to a local user, still have a notification for one, or contain a reported post.

docs/configuration/cheatsheet.md

@@ -1189,7 +1189,7 @@ Control favicons for instances.
 
 Bounded retention of remote content. Local posts are your data; remote posts are a cache of somebody else's, and this treats them that way. When enabled, a cron worker (`Pleroma.Workers.Cron.RetentionWorker`, hourly by default) evicts a batch of remote threads that nobody local has interacted with and that have been quiet for longer than `remote_post_retention_days` (see [`:instance`](#instance)). The database then plateaus instead of growing without limit.
 
-A thread is never evicted if it contains a local post, reply, favourite, repeat or reaction, a post bookmarked by a local user, a post that mentioned a local user (as long as the notification has not been cleared), or a post that was reported. Relationship activities (follows, blocks) and chat messages are never touched. Evicted posts are refetched on demand when somebody opens them again, as with `prune_objects`.
+A thread is never evicted if it contains a local post, reply, favourite, repeat or reaction, a post bookmarked by a local user, a post addressed to a local user (direct messages and mentions), a post a local user still has a notification for, or a post that was reported. Relationship activities (follows, blocks) and chat messages are never touched. Evicted posts are refetched on demand when somebody opens them again, as with `prune_objects`.
 
 !!! warning
     A local quote post does not pin the post it quotes, since quotes do not share the quoted thread's context. Old quoted remote posts can be evicted and will be refetched when the quote is viewed.
@@ -1200,7 +1200,7 @@ A thread is never evicted if it contains a local post, reply, favourite, repeat
 * `keep_non_public`: Also keep threads that contain a non-public post. Defaults to `false`.
 
 !!! note
-    The worker walks the `activities` table in id order, `batch_size` activities per run, up to the id that corresponds to the retention deadline, and remembers its position in the `retention_cursors` table. Each activity is visited once, when it becomes old enough, and only the threads it belongs to are checked through the context index. Over the `max_objects` watermark a second walk with its own cursor runs up to the present instead. The first run starts from the oldest activity; on an instance that never pruned this backlog takes many runs, so run `prune_objects --keep-threads --prune-orphaned-activities` once for the initial cut. Threads that were kept and later lost their pin (cleared notifications, removed bookmarks) are only revisited on a fresh walk: delete the row from `retention_cursors` to restart from the beginning. Space freed by eviction is reused by new rows; it is not handed back to the filesystem unless you `VACUUM FULL`.
+    The worker walks the `activities` table in id order, `batch_size` activities per run, up to the id that corresponds to the retention deadline, and remembers its position in the `retention_cursors` table. Each activity is visited once, when it becomes old enough, and only the threads it belongs to are checked through the context index. Over the `max_objects` watermark a second walk with its own cursor runs up to the present instead. The first run starts from the oldest activity; on an instance that never pruned this backlog takes many runs, so run `prune_objects --keep-threads --prune-orphaned-activities` once for the initial cut. Threads that were kept and later lost their pin (removed bookmarks, cleared notifications of other kinds) are only revisited on a fresh walk: delete the row from `retention_cursors` to restart from the beginning. Space freed by eviction is reused by new rows; it is not handed back to the filesystem unless you `VACUUM FULL`.
 
 ## Pleroma.User.Backup
 

lib/pleroma/retention.ex

@@ -17,7 +17,8 @@ defmodule Pleroma.Retention do
     * is local (posts, replies, favourites, repeats and reactions by local
       users all carry the thread's context),
     * is bookmarked by a local user,
-    * still has a notification for a local user (mentions),
+    * is addressed to a local user (direct messages and mentions),
+    * still has a notification for a local user,
     * is a report (Flag),
 
   or if any object in it is referenced by a report.
@@ -228,6 +229,15 @@ defmodule Pleroma.Retention do
     |> having([a], not fragment("bool_or(?)", a.local))
     |> having([_, b], fragment("max(?::text) is null", b.id))
     |> having([_, _, n], fragment("max(?) is null", n.id))
+    |> having(
+      [a],
+      not fragment(
+        "bool_or(EXISTS (SELECT 1 FROM unnest(?) AS r WHERE left(r, ?) = ?))",
+        a.recipients,
+        ^local_prefix_length(),
+        ^local_prefix()
+      )
+    )
     |> maybe_keep_non_public(Keyword.get(opts, :keep_non_public, false) == true)
   end
 

test/pleroma/retention_test.exs

@@ -165,6 +165,31 @@ defmodule Pleroma.RetentionTest do
       assert Object.get_by_ap_id(post.data["object"])
     end
 
+    test "keeps old remote posts addressed to a local user after notifications are cleared", %{
+      old_date: old_date
+    } do
+      remote_user = insert(:user, local: false)
+      local_user = insert(:user)
+
+      {:ok, dm} =
+        CommonAPI.post(remote_user, %{
+          status: "psst @#{local_user.nickname}",
+          visibility: "direct"
+        })
+
+      {:ok, mention} = CommonAPI.post(remote_user, %{status: "hey @#{local_user.nickname}"})
+
+      Pleroma.Notification.clear(local_user)
+
+      for post <- [dm, mention] do
+        make_remote_and_old(post, old_date)
+      end
+
+      assert %{contexts: 0} = Retention.run()
+      assert Object.get_by_ap_id(dm.data["object"])
+      assert Object.get_by_ap_id(mention.data["object"])
+    end
+
     test "keeps an old remote post that was reported", %{old_date: old_date} do
       remote_user = insert(:user, local: false)
       local_user = insert(:user)
@@ -347,8 +372,9 @@ defmodule Pleroma.RetentionTest do
     } do
       local_user = insert(:user)
       {:ok, post} = CommonAPI.post(local_user, %{status: "mine, honestly"})
-      # Pretend the Create activity federated in from elsewhere
-      make_remote_and_old(post, old_date)
+      # Pretend the Create activity federated in from elsewhere, addressed to
+      # nobody local (the local followers collection would pin the thread)
+      age(post, old_date, %{local: false, recipients: []})
 
       assert %{contexts: 1, objects: 0} = Retention.run()
       assert Object.get_by_ap_id(post.data["object"])