Merge pull request 'Add rollback migrations for Rebased' (#7685) from gitlab-mr-iid-4308 into develop

Reviewed-on: https://git.pleroma.social/pleroma/pleroma/pulls/7685

changelog.d/rebased-migration.change

@@ -0,0 +1 @@
+Add optional migrations for rollback from Rebased

docs/installation/migrating_from_rebased.md

@@ -0,0 +1,130 @@
+# Migrating from Rebased
+
+Rebased and Pleroma currently have compatible database schemas, so removing
+Rebased-specific database objects is optional. The cleanup described below is
+destructive and can take a long time on large databases.
+
+This guide requires a Pleroma release that contains this document and native
+webhook support. If you are reading the development version of this guide
+before those changes reach `stable`, wait for the corresponding stable release.
+
+## Prepare and back up
+
+Plan a maintenance window and read the [backup guide](../administration/backup.md)
+before starting.
+
+Stop Rebased and confirm it is stopped before making the backup. Omit the
+backup guide's final restart step and leave Rebased stopped:
+
+```bash
+sudo systemctl stop pleroma
+sudo systemctl is-active pleroma
+```
+
+Make a verified database backup and a copy of the Rebased installation,
+including its configuration, uploads, and `instance` directory. Also save the
+current systemd unit if you plan to replace it.
+
+## Install Pleroma
+
+Do not use `git pull --rebase` to change an existing Rebased checkout into a
+Pleroma checkout. The forks use different branches, and rebasing can replay
+Rebased-specific commits over Pleroma. Move the old checkout aside and clone a
+clean copy of Pleroma instead. The paths below assume the default source
+installation in `/opt/pleroma`:
+
+```bash
+sudo mv /opt/pleroma /opt/pleroma.rebased
+sudo mkdir /opt/pleroma
+sudo chown pleroma:pleroma /opt/pleroma
+sudo -Hu pleroma git clone -b stable https://git.pleroma.social/pleroma/pleroma /opt/pleroma
+```
+
+Restore the instance-specific files from the old checkout. This usually
+includes `config/prod.secret.exs`, `config/prod.exported_from_db.secret.exs` if
+present, `uploads`, and `instance`. Diff and restore any other non-stock
+configuration files, preserve their ownership, and review them before starting
+Pleroma.
+
+Pleroma requires Elixir 1.15 or newer. Install supported Erlang and Elixir
+versions as described in the installation guide for your distribution before
+running any Mix command. Do not reuse Rebased's `.tool-versions`. If you keep
+using [`asdf`](https://asdf-vm.com/), select supported versions in the
+`pleroma` user's home and use the asdf Mix shim for every command below.
+
+Set `MIX` to the appropriate binary, verify the selected runtime, then install
+the dependencies and compile Pleroma:
+
+```bash
+cd /opt/pleroma
+MIX=/usr/bin/mix
+# If using asdf instead:
+# MIX=/var/lib/pleroma/.asdf/shims/mix
+sudo -Hu pleroma "$MIX" --version
+sudo -Hu pleroma MIX_ENV=prod "$MIX" deps.get
+sudo -Hu pleroma MIX_ENV=prod "$MIX" compile
+```
+
+Compare your existing systemd unit with `installation/pleroma.service`. If you
+replace it, review its user, paths, and environment first. Reload systemd after
+making changes:
+
+```bash
+sudo systemctl daemon-reload
+```
+
+Rebased does not bundle a frontend, so `instance/static` may contain an
+installed frontend. Pleroma includes a bundled frontend. You can remove the
+separately installed frontend after the migration, but do not remove other
+instance files such as custom emojis.
+
+## Database migration
+
+Rebased's extra columns, index, and enum labels do not prevent Pleroma from
+starting. Existing notifications with Rebased-only event types are not
+supported by Pleroma, however, and can cause notification API errors. Run the
+cleanup below, or archive and remove those notification rows separately before
+starting Pleroma. Skipping the cleanup preserves them if you need to restore
+Rebased.
+
+Pleroma supports Rebased's webhook schema. The cleanup does not roll back the
+shared webhook migrations, so webhook configuration is preserved.
+
+To perform the optional cleanup, run the rollback while the service is stopped:
+
+```bash
+MIX=/usr/bin/mix
+# If using asdf instead:
+# MIX=/var/lib/pleroma/.asdf/shims/mix
+sudo -Hu pleroma MIX_ENV=prod "$MIX" ecto.rollback --migrations-path priv/repo/optional_migrations/rebased_rollbacks --all
+```
+
+This deletes the Rebased-only `accepts_email_list`, `location`, and
+`last_move_at` user fields and the `pleroma:participation_accepted`,
+`pleroma:participation_request`, `pleroma:event_reminder`, and
+`pleroma:event_update` notifications. It also rewrites the notifications table
+and can require substantial time, free disk space, and write-ahead log capacity
+on a large instance.
+
+Apply all pending Pleroma migrations:
+
+```bash
+sudo -Hu pleroma MIX_ENV=prod "$MIX" ecto.migrate
+```
+
+## Start and verify Pleroma
+
+Start Pleroma and inspect its status and logs:
+
+```bash
+sudo systemctl start pleroma
+sudo systemctl status pleroma
+sudo journalctl -u pleroma -n 100 --no-pager
+```
+
+Verify that the instance, uploads, frontend, and webhook configuration work as
+expected before deleting `/opt/pleroma.rebased` or the database backup. If the
+migration fails, keep the service stopped and restore both the database backup
+and the previous checkout before restarting Rebased. If you changed the
+systemd unit or runtime setup, restore those too and run
+`systemctl daemon-reload` before restarting the service.

priv/repo/optional_migrations/rebased_rollbacks/20210612185407_add_email_list_field_to_users.exs

@@ -0,0 +1,14 @@
+# Adapted from Rebased
+# https://gitlab.com/soapbox-pub/rebased/-/blob/main/priv/repo/migrations/20210612185407_add_email_list_field_to_users.exs
+
+defmodule Pleroma.Repo.Migrations.AddEmailListFieldToUsers do
+  use Ecto.Migration
+
+  def up, do: :noop
+
+  def down do
+    alter table(:users) do
+      remove_if_exists(:accepts_email_list, :boolean)
+    end
+  end
+end

priv/repo/optional_migrations/rebased_rollbacks/20220314220000_add_location_to_users.exs

@@ -0,0 +1,11 @@
+defmodule Pleroma.Repo.Migrations.AddLocationToUsers do
+  use Ecto.Migration
+
+  def up, do: :noop
+
+  def down do
+    alter table(:users) do
+      remove_if_exists(:location, :string)
+    end
+  end
+end

priv/repo/optional_migrations/rebased_rollbacks/20220819171321_add_pleroma_participation_accepted_to_notifications_enum.exs

@@ -0,0 +1,45 @@
+defmodule Pleroma.Repo.Migrations.AddPleromaParticipationAcceptedToNotificationsEnum do
+  use Ecto.Migration
+
+  def up, do: :noop
+
+  def down do
+    alter table(:notifications) do
+      modify(:type, :string)
+    end
+
+    """
+    delete from notifications where type in ('pleroma:participation_accepted', 'pleroma:participation_request', 'pleroma:event_reminder', 'pleroma:event_update')
+    """
+    |> execute()
+
+    """
+    drop type if exists notification_type
+    """
+    |> execute()
+
+    """
+    create type notification_type as enum (
+      'follow',
+      'follow_request',
+      'mention',
+      'move',
+      'pleroma:emoji_reaction',
+      'pleroma:chat_mention',
+      'reblog',
+      'favourite',
+      'pleroma:report',
+      'poll',
+      'status',
+      'update'
+    )
+    """
+    |> execute()
+
+    """
+    alter table notifications
+    alter column type type notification_type using (type::notification_type)
+    """
+    |> execute()
+  end
+end

priv/repo/optional_migrations/rebased_rollbacks/20220927220033_add_last_move_at_to_users.exs

@@ -0,0 +1,11 @@
+defmodule Pleroma.Repo.Migrations.AddLastMoveAtToUsers do
+  use Ecto.Migration
+
+  def up, do: :noop
+
+  def down do
+    alter table(:users) do
+      remove_if_exists(:last_move_at, :naive_datetime)
+    end
+  end
+end

priv/repo/optional_migrations/rebased_rollbacks/20221207235104_add_events_index.exs

@@ -0,0 +1,17 @@
+# Adapted from Rebased
+# https://gitlab.com/soapbox-pub/rebased/-/blob/main/priv/repo/migrations/20221207235104_add_events_index.exs
+
+defmodule Pleroma.Repo.Migrations.AddEventsIndex do
+  use Ecto.Migration
+
+  def up, do: :noop
+
+  def down do
+    drop_if_exists(
+      index(:objects, ["(data->>'type')"],
+        where: "data->>'type' = 'Event'",
+        name: :objects_events
+      )
+    )
+  end
+end

test/pleroma/repo/optional_migrations/rebased_rollbacks_test.exs

@@ -0,0 +1,174 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2022 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Repo.OptionalMigrations.RebasedRollbacksTest do
+  use ExUnit.Case, async: false
+
+  alias Ecto.Adapters.SQL
+
+  defmodule MigrationRepo do
+    use Ecto.Repo,
+      otp_app: :pleroma,
+      adapter: Ecto.Adapters.Postgres
+  end
+
+  @migrations_path "priv/repo/migrations"
+  @rollbacks_path "priv/repo/optional_migrations/rebased_rollbacks"
+
+  @expected_versions ~w{
+    20210612185407
+    20220314220000
+    20220819171321
+    20220927220033
+    20221207235104
+  }
+
+  test "only includes Rebased-specific migration versions" do
+    rollback_versions = migration_versions(@rollbacks_path)
+    pleroma_versions = migration_versions(@migrations_path)
+
+    assert rollback_versions == @expected_versions
+    assert MapSet.disjoint?(MapSet.new(rollback_versions), MapSet.new(pleroma_versions))
+  end
+
+  test "rolls back Rebased schema without removing Pleroma data" do
+    prefix = "rebased_rollbacks_#{Ecto.UUID.generate() |> String.replace("-", "")}"
+
+    repo_config = [
+      username: "postgres",
+      password: "postgres",
+      database: System.get_env("DB_NAME", "pleroma_test"),
+      hostname: System.get_env("DB_HOST", "localhost"),
+      port: String.to_integer(System.get_env("DB_PORT", "5432")),
+      pool_size: 2,
+      parameters: [search_path: prefix]
+    ]
+
+    {:ok, repo} = MigrationRepo.start_link(repo_config)
+    Process.unlink(repo)
+
+    on_exit(fn ->
+      query!("DROP SCHEMA IF EXISTS #{prefix} CASCADE")
+      MigrationRepo.stop()
+    end)
+
+    query!("CREATE SCHEMA #{prefix}")
+
+    create_rebased_schema()
+
+    expected_rollbacks =
+      @expected_versions
+      |> Enum.reverse()
+      |> Enum.map(&String.to_integer/1)
+
+    assert Ecto.Migrator.run(MigrationRepo, @rollbacks_path, :down, all: true) ==
+             expected_rollbacks
+
+    assert query!("SELECT version FROM schema_migrations ORDER BY version").rows == [
+             [20_220_624_104_914],
+             [20_221_029_171_353]
+           ]
+
+    assert query!("""
+           SELECT column_name
+           FROM information_schema.columns
+           WHERE table_schema = current_schema() AND table_name = 'users'
+           ORDER BY column_name
+           """).rows == [["id"]]
+
+    assert query!("SELECT type::text FROM notifications ORDER BY type::text").rows == [["status"]]
+
+    assert query!("""
+           SELECT enumlabel
+           FROM pg_enum
+           JOIN pg_type ON pg_type.oid = enumtypid
+           JOIN pg_namespace ON pg_namespace.oid = pg_type.typnamespace
+           WHERE typname = 'notification_type' AND nspname = current_schema()
+           ORDER BY enumsortorder
+           """).rows ==
+             Enum.map(
+               ~w{follow follow_request mention move pleroma:emoji_reaction pleroma:chat_mention reblog favourite pleroma:report poll status update},
+               &[&1]
+             )
+
+    assert query!("SELECT to_regclass('objects_events')").rows == [[nil]]
+
+    assert query!("SELECT url, secret, internal FROM webhooks").rows == [
+             ["https://example.com/webhook", "preserve-me", false]
+           ]
+  end
+
+  defp migration_versions(path) do
+    path
+    |> Path.join("*.exs")
+    |> Path.wildcard()
+    |> Enum.map(fn migration ->
+      migration
+      |> Path.basename()
+      |> String.split("_", parts: 2)
+      |> hd()
+    end)
+    |> Enum.sort()
+  end
+
+  defp create_rebased_schema do
+    statements = [
+      """
+      CREATE TYPE notification_type AS ENUM (
+        'follow', 'follow_request', 'mention', 'move', 'pleroma:emoji_reaction',
+        'pleroma:chat_mention', 'reblog', 'favourite', 'pleroma:report', 'poll',
+        'status', 'update', 'pleroma:participation_accepted',
+        'pleroma:participation_request', 'pleroma:event_reminder',
+        'pleroma:event_update'
+      )
+      """,
+      "CREATE TABLE notifications (id bigserial PRIMARY KEY, type notification_type)",
+      """
+      INSERT INTO notifications (type) VALUES
+        ('status'),
+        ('pleroma:participation_accepted'),
+        ('pleroma:participation_request'),
+        ('pleroma:event_reminder'),
+        ('pleroma:event_update')
+      """,
+      """
+      CREATE TABLE users (
+        id bigserial PRIMARY KEY,
+        accepts_email_list boolean,
+        location varchar(255),
+        last_move_at timestamp
+      )
+      """,
+      "CREATE TABLE objects (id bigserial PRIMARY KEY, data jsonb)",
+      "CREATE INDEX objects_events ON objects ((data->>'type')) WHERE data->>'type' = 'Event'",
+      """
+      CREATE TABLE webhooks (
+        id bigserial PRIMARY KEY,
+        url varchar(255) NOT NULL,
+        events varchar[] NOT NULL DEFAULT '{}',
+        secret varchar(255) NOT NULL DEFAULT '',
+        enabled boolean NOT NULL DEFAULT true,
+        internal boolean NOT NULL DEFAULT false
+      )
+      """,
+      "CREATE UNIQUE INDEX webhooks_url_index ON webhooks (url)",
+      "INSERT INTO webhooks (url, secret) VALUES ('https://example.com/webhook', 'preserve-me')",
+      "CREATE TABLE schema_migrations (version bigint PRIMARY KEY, inserted_at timestamp(0))",
+      """
+      INSERT INTO schema_migrations (version, inserted_at) VALUES
+        (20210612185407, NOW()),
+        (20220314220000, NOW()),
+        (20220624104914, NOW()),
+        (20220819171321, NOW()),
+        (20220927220033, NOW()),
+        (20221029171353, NOW()),
+        (20221207235104, NOW())
+      """
+    ]
+
+    Enum.each(statements, &query!/1)
+  end
+
+  defp query!(statement), do: SQL.query!(MigrationRepo, statement, [])
+end