Merge remote-tracking branch 'origin/develop' into shigusegubu

* origin/develop: (159 commits)
  Fix CHANGELOG entry meant for next release
  Revert "Merge branch 'patch-fix-open-api-spec' into 'develop'"
  object_validator: Refactor most of validate/2 to a generic block
  date-times are always strings
  Fix function calls due to module name change
  Temp file leaked, oops
  Add more details to the cheatsheat for FollowBot MRF
  Prefer FollowBot naming convention vs Followbot
  Document new FollowBot MRF
  More tests to validate Followbot is behaving
  Remove Task.async as it is broken here and probably a premature optimization anyway
  Only need to validate a follow request is generated for now
  Revert
  Do not try to follow local users. Their posts are already available locally on the instance.
  Change module name to FollowbotPolicy
  Add follow_requests_outstanding_since?/3 to Pleroma.Activity
  Lint
  Better checking of previous follow request attempts
  Prevent duplicates from being processed
  Make the followbot only dispatch follow requests once per 30 day period
  ...

.gitignore

@@ -53,3 +53,7 @@ pleroma.iml
 
 # asdf
 .tool-versions
+
+# Editor temp files
+/*~
+/*#

.gitlab-ci.yml

@@ -25,13 +25,13 @@ before_script:
   - apt-get update && apt-get install -y cmake
   - mix local.hex --force
   - mix local.rebar --force
+  - mix deps.get
   - apt-get -qq update
   - apt-get install -y libmagic-dev
 
 build:
   stage: build
   script:
-  - mix deps.get
   - mix compile --force
 
 spec-build:
@@ -52,7 +52,6 @@ benchmark:
     alias: postgres
     command: ["postgres", "-c", "fsync=off", "-c", "synchronous_commit=off", "-c", "full_page_writes=off"]
   script:
-    - mix deps.get
     - mix ecto.create
     - mix ecto.migrate
     - mix pleroma.load_testing
@@ -70,7 +69,6 @@ unit-testing:
     command: ["postgres", "-c", "fsync=off", "-c", "synchronous_commit=off", "-c", "full_page_writes=off"]
   script:
     - apt-get update && apt-get install -y libimage-exiftool-perl ffmpeg
-    - mix deps.get
     - mix ecto.create
     - mix ecto.migrate
     - mix coveralls --preload-modules
@@ -104,7 +102,6 @@ unit-testing-rum:
     RUM_ENABLED: "true"
   script:
     - apt-get update && apt-get install -y libimage-exiftool-perl ffmpeg
-    - mix deps.get
     - mix ecto.create
     - mix ecto.migrate
     - "mix ecto.migrate --migrations-path priv/repo/optional_migrations/rum_indexing/"
@@ -120,7 +117,6 @@ analysis:
   stage: test
   cache: *testing_cache_policy
   script:
-    - mix deps.get
     - mix credo --strict --only=warnings,todo,fixme,consistency,readability
 
 docs-deploy:

CHANGELOG.md

@@ -6,6 +6,26 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ## Unreleased
 
+### Changed
+
+- The `application` metadata returned with statuses is no longer hardcoded. Apps that want to display these details will now have valid data for new posts after this change.
+
+### Added
+
+- MRF (`FollowBotPolicy`): New MRF Policy which makes a designated local Bot account attempt to follow all users in public Notes received by your instance. Users who require approving follower requests or have #nobot in their profile are excluded.
+
+## Unreleased (Patch)
+
+### Fixed
+
+- Try to save exported ConfigDB settings (migrate_from_db) in the system temp directory if default location is not writable.
+
+## [2.3.0] - 2020-03-01
+
+### Security
+
+- Fixed client user agent leaking through MediaProxy
+
 ### Removed
 
 - `:auth, :enforce_oauth_admin_scope_usage` configuration option.
@@ -14,6 +34,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 - **Breaking**: Changed `mix pleroma.user toggle_confirmed` to `mix pleroma.user confirm`
 - **Breaking**: Changed `mix pleroma.user toggle_activated` to `mix pleroma.user activate/deactivate`
+- **Breaking:** NSFW hashtag is no longer added on sensitive posts
 - Polls now always return a `voters_count`, even if they are single-choice.
 - Admin Emails: The ap id is used as the user link in emails now.
 - Improved registration workflow for email confirmation and account approval modes.
@@ -37,13 +58,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - **Breaking:** AdminAPI `GET /api/pleroma/admin/instances/:instance/statuses` changed response format and added the number of total users posts.
 - Admin API: Reports now ordered by newest
 - Pleroma API: `GET /api/v1/pleroma/chats` is deprecated in favor of `GET /api/v2/pleroma/chats`.
+- Pleroma API: Reroute `/api/pleroma/*` to `/api/v1/pleroma/*`
 
 </details>
+- Improved hashtag timeline performance (requires a background migration).
 
 ### Added
 
 - Reports now generate notifications for admins and mods.
-- Experimental websocket-based federation between Pleroma instances.
 - Support for local-only statuses.
 - Support pagination of blocks and mutes.
 - Account backup.
@@ -63,6 +85,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 <details>
   <summary>API Changes</summary>
 - Admin API: (`GET /api/pleroma/admin/users`) filter users by `unconfirmed` status and `actor_type`.
+- Admin API: OpenAPI spec for the user-related operations
 - Pleroma API: `GET /api/v2/pleroma/chats` added. It is exactly like `GET /api/v1/pleroma/chats` except supports pagination.
 - Pleroma API: Add `idempotency_key` to the chat message entity that can be used for optimistic message sending.
 - Pleroma API: (`GET /api/v1/pleroma/federation_status`) Add a way to get a list of unreachable instances.
@@ -74,6 +97,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Mastodon API: Home, public, hashtag & list timelines accept `only_media`, `remote` & `local` parameters for filtration.
 - Mastodon API: `/api/v1/accounts/:id` & `/api/v1/mutes` endpoints accept `with_relationships` parameter and return filled `pleroma.relationship` field.
 - Mastodon API: Endpoint to remove a conversation (`DELETE /api/v1/conversations/:id`).
+- Mastodon API: `expires_in` in the scheduled post `params` field on `/api/v1/statuses` and `/api/v1/scheduled_statuses/:id` endpoints.
 </details>
 
 ### Fixed
@@ -97,9 +121,6 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
   - Mastodon API: Support for expires_in/expires_at in the Filters.
 </details>
 
-## Unreleased (Patch)
-
-
 ## [2.2.2] - 2020-01-18
 
 ### Fixed
@@ -496,7 +517,6 @@ switched to a new configuration mechanism, however it was not officially removed
 - Static-FE: Fix remote posts not being sanitized
 
 ### Fixed
-=======
 - Rate limiter crashes when there is no explicitly specified ip in the config
 - 500 errors when no `Accept` header is present if Static-FE is enabled
 - Instance panel not being updated immediately due to wrong `Cache-Control` headers

config/config.exs

@@ -405,6 +405,11 @@ config :pleroma, :mrf_keyword,
   federated_timeline_removal: [],
   replace: []
 
+config :pleroma, :mrf_hashtag,
+  sensitive: ["nsfw"],
+  reject: [],
+  federated_timeline_removal: []
+
 config :pleroma, :mrf_subchain, match_actor: %{}
 
 config :pleroma, :mrf_activity_expiration, days: 365
@@ -418,6 +423,8 @@ config :pleroma, :mrf_object_age,
   threshold: 604_800,
   actions: [:delist, :strip_followers]
 
+config :pleroma, :mrf_follow_bot, follower_nickname: nil
+
 config :pleroma, :rich_media,
   enabled: true,
   ignore_hosts: [],
@@ -668,6 +675,10 @@ config :pleroma, :oauth2,
 
 config :pleroma, :database, rum_enabled: false
 
+config :pleroma, :features, improved_hashtag_timeline: :auto
+
+config :pleroma, :populate_hashtags_table, fault_rate_allowance: 0.01
+
 config :pleroma, :env, Mix.env()
 
 config :http_signatures,

config/description.exs

@@ -459,6 +459,42 @@ config :pleroma, :config_description, [
       }
     ]
   },
+  %{
+    group: :pleroma,
+    key: :features,
+    type: :group,
+    description: "Customizable features",
+    children: [
+      %{
+        key: :improved_hashtag_timeline,
+        type: {:dropdown, :atom},
+        description:
+          "Setting to force toggle / force disable improved hashtags timeline. `:enabled` forces hashtags to be fetched from `hashtags` table for hashtags timeline. `:disabled` forces object-embedded hashtags to be used (slower). Keep it `:auto` for automatic behaviour (it is auto-set to `:enabled` [unless overridden] when HashtagsTableMigrator completes).",
+        suggestions: [:auto, :enabled, :disabled]
+      }
+    ]
+  },
+  %{
+    group: :pleroma,
+    key: :populate_hashtags_table,
+    type: :group,
+    description: "`populate_hashtags_table` background migration settings",
+    children: [
+      %{
+        key: :fault_rate_allowance,
+        type: :float,
+        description:
+          "Max accepted rate of objects that failed in the migration. Any value from 0.0 which tolerates no errors to 1.0 which will enable the feature even if hashtags transfer failed for all records.",
+        suggestions: [0.01]
+      },
+      %{
+        key: :sleep_interval_ms,
+        type: :integer,
+        description:
+          "Sleep interval between each chunk of processed records in order to decrease the load on the system (defaults to 0 and should be keep default on most instances)."
+      }
+    ]
+  },
   %{
     group: :pleroma,
     key: :instance,
@@ -2906,6 +2942,23 @@ config :pleroma, :config_description, [
       }
     ]
   },
+  %{
+    group: :pleroma,
+    key: :mrf_follow_bot,
+    tab: :mrf,
+    related_policy: "Pleroma.Web.ActivityPub.MRF.FollowBotPolicy",
+    label: "MRF FollowBot Policy",
+    type: :group,
+    description: "Automatically follows newly discovered accounts.",
+    children: [
+      %{
+        key: :follower_nickname,
+        type: :string,
+        description: "The name of the bot account to use for following newly discovered users.",
+        suggestions: ["followbot"]
+      }
+    ]
+  },
   %{
     group: :pleroma,
     key: :modules,

config/test.exs

@@ -38,7 +38,7 @@ config :pleroma, :instance,
   external_user_synchronization: false,
   static_dir: "test/instance_static/"
 
-config :pleroma, :activitypub, sign_object_fetches: false
+config :pleroma, :activitypub, sign_object_fetches: false, follow_handshake_timeout: 0
 
 # Configure your database
 config :pleroma, Pleroma.Repo,

docs/administration/CLI_tasks/config.md

@@ -32,16 +32,20 @@
     config :pleroma, configurable_from_database: false
     ```
 
-To delete transferred settings from database optional flag `-d` can be used. `<env>` is `prod` by default.
+Options:
+
+- `<path>` - where to save migrated config. E.g. `--path=/tmp`. If file saved into non standart folder, you must manually copy file into directory where Pleroma can read it. For OTP install path will be `PLEROMA_CONFIG_PATH` or `/etc/pleroma`. For installation from source - `config` directory in the pleroma folder.
+- `<env>` - environment, for which is migrated config. By default is `prod`.
+- To delete transferred settings from database optional flag `-d` can be used
 
 === "OTP"
     ```sh
-     ./bin/pleroma_ctl config migrate_from_db [--env=<env>] [-d]
+     ./bin/pleroma_ctl config migrate_from_db [--env=<env>] [-d] [--path=<path>]
     ```
 
 === "From Source"
     ```sh
-    mix pleroma.config migrate_from_db [--env=<env>] [-d]
+    mix pleroma.config migrate_from_db [--env=<env>] [-d] [--path=<path>]
     ```
 
 ## Dump all of the config settings defined in the database

docs/clients.md

@@ -17,7 +17,7 @@ Feel free to contact us to be added to this list!
 - Features: MastoAPI
 
 ### Whalebird
-- Homepage: <https://whalebird.org/>
+- Homepage: <https://whalebird.social/>
 - Source Code: <https://github.com/h3poteto/whalebird-desktop>
 - Contact: [@h3poteto@pleroma.io](https://pleroma.io/users/h3poteto)
 - Platforms: Windows, Mac, Linux

docs/configuration/cheatsheet.md

@@ -49,7 +49,7 @@ To add configuration to your config file, you can copy it from the base config.
 * `attachment_links`: Set to true to enable automatically adding attachment link text to statuses.
 * `max_report_comment_size`: The maximum size of the report comment (Default: `1000`).
 * `safe_dm_mentions`: If set to true, only mentions at the beginning of a post will be used to address people in direct messages. This is to prevent accidental mentioning of people when talking about them (e.g. "@friend hey i really don't like @enemy"). Default: `false`.
-* `healthcheck`: If set to true, system data will be shown on ``/api/pleroma/healthcheck``.
+* `healthcheck`: If set to true, system data will be shown on ``/api/v1/pleroma/healthcheck``.
 * `remote_post_retention_days`: The default amount of days to retain remote posts when pruning the database.
 * `user_bio_length`: A user bio maximum length (default: `5000`).
 * `user_name_length`: A user name maximum length (default: `100`).
@@ -65,6 +65,13 @@ To add configuration to your config file, you can copy it from the base config.
 * `show_reactions`: Let favourites and emoji reactions be viewed through the API (default: `true`).
 * `password_reset_token_validity`: The time after which reset tokens aren't accepted anymore, in seconds (default: one day).
 
+## :database
+* `improved_hashtag_timeline`: Setting to force toggle / force disable improved hashtags timeline. `:enabled` forces hashtags to be fetched from `hashtags` table for hashtags timeline. `:disabled` forces object-embedded hashtags to be used (slower). Keep it `:auto` for automatic behaviour (it is auto-set to `:enabled` [unless overridden] when HashtagsTableMigrator completes).
+
+## Background migrations
+* `populate_hashtags_table/sleep_interval_ms`: Sleep interval between each chunk of processed records in order to decrease the load on the system (defaults to 0 and should be keep default on most instances).
+* `populate_hashtags_table/fault_rate_allowance`: Max rate of failed objects to actually processed objects in order to enable the feature (any value from 0.0 which tolerates no errors to 1.0 which will enable the feature even if hashtags transfer failed for all records).
+
 ## Welcome
 * `direct_message`: - welcome message sent as a direct message.
   * `enabled`: Enables the send a direct message to a newly registered user. Defaults to `false`.
@@ -117,6 +124,7 @@ To add configuration to your config file, you can copy it from the base config.
     * `Pleroma.Web.ActivityPub.MRF.ObjectAgePolicy`: Rejects or delists posts based on their age when received. (See [`:mrf_object_age`](#mrf_object_age)).
     * `Pleroma.Web.ActivityPub.MRF.ActivityExpirationPolicy`: Sets a default expiration on all posts made by users of the local instance. Requires `Pleroma.Workers.PurgeExpiredActivity` to be enabled for processing the scheduled delections.
     * `Pleroma.Web.ActivityPub.MRF.ForceBotUnlistedPolicy`: Makes all bot posts to disappear from public timelines.
+    * `Pleroma.Web.ActivityPub.MRF.FollowBotPolicy`: Automatically follows newly discovered users from the specified bot account. Local accounts, locked accounts, and users with "#nobot" in their bio are respected and excluded from being followed.
 * `transparency`: Make the content of your Message Rewrite Facility settings public (via nodeinfo).
 * `transparency_exclusions`: Exclude specific instance names from MRF transparency.  The use of the exclusions feature will be disclosed in nodeinfo as a boolean value.
 
@@ -203,6 +211,21 @@ config :pleroma, :mrf_user_allowlist, %{
 
 * `days`: Default global expiration time for all local Create activities (in days)
 
+#### :mrf_hashtag
+
+* `sensitive`: List of hashtags to mark activities as sensitive (default: `nsfw`)
+* `federated_timeline_removal`: List of hashtags to remove activities from the federated timeline (aka TWNK)
+* `reject`: List of hashtags to reject activities from
+
+Notes:
+- The hashtags in the configuration do not have a leading `#`.
+- This MRF Policy is always enabled, if you want to disable it you have to set empty lists
+
+#### :mrf_follow_bot
+
+* `follower_nickname`: The name of the bot account to use for following newly discovered users. Using `followbot` or similar is strongly suggested.
+
+
 ### :activitypub
 * `unfollow_blocked`: Whether blocks result in people getting unfollowed
 * `outgoing_blocks`: Whether to federate blocks to other instances
@@ -225,7 +248,7 @@ config :pleroma, :mrf_user_allowlist, %{
 
 This can be used to configure a keyword list that keeps the configuration data for any kind of frontend. By default, settings for `pleroma_fe` and `masto_fe` are configured. You can find the documentation for `pleroma_fe` configuration into [Pleroma-FE configuration and customization for instance administrators](/frontend/CONFIGURATION/#options).
 
-Frontends can access these settings at `/api/pleroma/frontend_configurations`
+Frontends can access these settings at `/api/v1/pleroma/frontend_configurations`
 
 To add your own configuration for PleromaFE, use it like this:
 
@@ -848,13 +871,13 @@ config :pleroma, :admin_token, "somerandomtoken"
 You can then do
 
 ```shell
-curl "http://localhost:4000/api/pleroma/admin/users/invites?admin_token=somerandomtoken"
+curl "http://localhost:4000/api/v1/pleroma/admin/users/invites?admin_token=somerandomtoken"
 ```
 
 or
 
 ```shell
-curl -H "X-Admin-Token: somerandomtoken" "http://localhost:4000/api/pleroma/admin/users/invites"
+curl -H "X-Admin-Token: somerandomtoken" "http://localhost:4000/api/v1/pleroma/admin/users/invites"
 ```
 
 Warning: it's discouraged to use this feature because of the associated security risk: static / rarely changed instance-wide token is much weaker compared to email-password pair of a real admin user; consider using HTTP Basic Auth or OAuth-based authentication instead.

docs/development/API/admin_api.md

@@ -2,7 +2,9 @@
 
 Authentication is required and the user must be an admin.
 
-## `GET /api/pleroma/admin/users`
+The `/api/v1/pleroma/admin/*` path is backwards compatible with `/api/pleroma/admin/*` (`/api/pleroma/admin/*` will be deprecated in the future).
+
+## `GET /api/v1/pleroma/admin/users`
 
 ### List users
 
@@ -23,7 +25,7 @@ Authentication is required and the user must be an admin.
   - *optional* `actor_types`: **[string]** actor type list (`Person`, `Service`, `Application`)
   - *optional* `name`: **string** user display name
   - *optional* `email`: **string** user email
-- Example: `https://mypleroma.org/api/pleroma/admin/users?query=john&filters=local,active&page=1&page_size=10&tags[]=some_tag&tags[]=another_tag&name=display_name&email=email@example.com`
+- Example: `https://mypleroma.org/api/v1/pleroma/admin/users?query=john&filters=local,active&page=1&page_size=10&tags[]=some_tag&tags[]=another_tag&name=display_name&email=email@example.com`
 - Response:
 
 ```json
@@ -52,7 +54,7 @@ Authentication is required and the user must be an admin.
 }
 ```
 
-## DEPRECATED `DELETE /api/pleroma/admin/users`
+## DEPRECATED `DELETE /api/v1/pleroma/admin/users`
 
 ### Remove a user
 
@@ -60,7 +62,7 @@ Authentication is required and the user must be an admin.
   - `nickname`
 - Response: User’s nickname
 
-## `DELETE /api/pleroma/admin/users`
+## `DELETE /api/v1/pleroma/admin/users`
 
 ### Remove a user
 
@@ -81,7 +83,7 @@ Authentication is required and the user must be an admin.
   ]
 - Response: User’s nickname
 
-## `POST /api/pleroma/admin/users/follow`
+## `POST /api/v1/pleroma/admin/users/follow`
 
 ### Make a user follow another user
 
@@ -91,7 +93,7 @@ Authentication is required and the user must be an admin.
 - Response:
   - "ok"
 
-## `POST /api/pleroma/admin/users/unfollow`
+## `POST /api/v1/pleroma/admin/users/unfollow`
 
 ### Make a user unfollow another user
 
@@ -101,7 +103,7 @@ Authentication is required and the user must be an admin.
 - Response:
   - "ok"
 
-## `PATCH /api/pleroma/admin/users/:nickname/toggle_activation`
+## `PATCH /api/v1/pleroma/admin/users/:nickname/toggle_activation`
 
 ### Toggle user activation
 
@@ -117,7 +119,7 @@ Authentication is required and the user must be an admin.
 }
 ```
 
-## `PUT /api/pleroma/admin/users/tag`
+## `PUT /api/v1/pleroma/admin/users/tag`
 
 ### Tag a list of users
 
@@ -125,7 +127,7 @@ Authentication is required and the user must be an admin.
   - `nicknames` (array)
   - `tags` (array)
 
-## `DELETE /api/pleroma/admin/users/tag`
+## `DELETE /api/v1/pleroma/admin/users/tag`
 
 ### Untag a list of users
 
@@ -133,7 +135,7 @@ Authentication is required and the user must be an admin.
   - `nicknames` (array)
   - `tags` (array)
 
-## `GET /api/pleroma/admin/users/:nickname/permission_group`
+## `GET /api/v1/pleroma/admin/users/:nickname/permission_group`
 
 ### Get user user permission groups membership
 
@@ -147,7 +149,7 @@ Authentication is required and the user must be an admin.
 }
 ```
 
-## `GET /api/pleroma/admin/users/:nickname/permission_group/:permission_group`
+## `GET /api/v1/pleroma/admin/users/:nickname/permission_group/:permission_group`
 
 Note: Available `:permission_group` is currently moderator and admin. 404 is returned when the permission group doesn’t exist.
 
@@ -163,7 +165,7 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
 }
 ```
 
-## DEPRECATED `POST /api/pleroma/admin/users/:nickname/permission_group/:permission_group`
+## DEPRECATED `POST /api/v1/pleroma/admin/users/:nickname/permission_group/:permission_group`
 
 ### Add user to permission group
 
@@ -172,7 +174,7 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
   - On failure: `{"error": "…"}`
   - On success: JSON of the user
 
-## `POST /api/pleroma/admin/users/permission_group/:permission_group`
+## `POST /api/v1/pleroma/admin/users/permission_group/:permission_group`
 
 ### Add users to permission group
 
@@ -182,9 +184,9 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
   - On failure: `{"error": "…"}`
   - On success: JSON of the user
 
-## DEPRECATED `DELETE /api/pleroma/admin/users/:nickname/permission_group/:permission_group`
+## DEPRECATED `DELETE /api/v1/pleroma/admin/users/:nickname/permission_group/:permission_group`
 
-## `DELETE /api/pleroma/admin/users/:nickname/permission_group/:permission_group`
+## `DELETE /api/v1/pleroma/admin/users/:nickname/permission_group/:permission_group`
 
 ### Remove user from permission group
 
@@ -194,7 +196,7 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
   - On success: JSON of the user
 - Note: An admin cannot revoke their own admin status.
 
-## `DELETE /api/pleroma/admin/users/permission_group/:permission_group`
+## `DELETE /api/v1/pleroma/admin/users/permission_group/:permission_group`
 
 ### Remove users from permission group
 
@@ -205,7 +207,7 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
   - On success: JSON of the user
 - Note: An admin cannot revoke their own admin status.
 
-## `PATCH /api/pleroma/admin/users/activate`
+## `PATCH /api/v1/pleroma/admin/users/activate`
 
 ### Activate user
 
@@ -223,7 +225,7 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
 }
 ```
 
-## `PATCH /api/pleroma/admin/users/deactivate`
+## `PATCH /api/v1/pleroma/admin/users/deactivate`
 
 ### Deactivate user
 
@@ -241,7 +243,7 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
 }
 ```
 
-## `PATCH /api/pleroma/admin/users/approve`
+## `PATCH /api/v1/pleroma/admin/users/approve`
 
 ### Approve user
 
@@ -259,7 +261,7 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
 }
 ```
 
-## `GET /api/pleroma/admin/users/:nickname_or_id`
+## `GET /api/v1/pleroma/admin/users/:nickname_or_id`
 
 ### Retrive the details of a user
 
@@ -269,7 +271,7 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
   - On failure: `Not found`
   - On success: JSON of the user
 
-## `GET /api/pleroma/admin/users/:nickname_or_id/statuses`
+## `GET /api/v1/pleroma/admin/users/:nickname_or_id/statuses`
 
 ### Retrive user's latest statuses
 
@@ -293,7 +295,7 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
 }
 ```
 
-## `GET /api/pleroma/admin/instances/:instance/statuses`
+## `GET /api/v1/pleroma/admin/instances/:instance/statuses`
 
 ### Retrive instance's latest statuses
 
@@ -317,7 +319,7 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
 }
 ```
 
-## `GET /api/pleroma/admin/statuses`
+## `GET /api/v1/pleroma/admin/statuses`
 
 ### Retrives all latest statuses
 
@@ -330,7 +332,7 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
   - On failure: `Not found`
   - On success: JSON array of user's latest statuses
 
-## `GET /api/pleroma/admin/relay`
+## `GET /api/v1/pleroma/admin/relay`
 
 ### List Relays
 
@@ -346,7 +348,7 @@ Response:
 ]
 ```
 
-## `POST /api/pleroma/admin/relay`
+## `POST /api/v1/pleroma/admin/relay`
 
 ### Follow a Relay
 
@@ -362,7 +364,7 @@ Response:
 {"actor": "https://example.com/relay", "followed_back": true}
 ```
 
-## `DELETE /api/pleroma/admin/relay`
+## `DELETE /api/v1/pleroma/admin/relay`
 
 ### Unfollow a Relay
 
@@ -378,7 +380,7 @@ Response:
 {"https://example.com/relay"}
 ```
 
-## `POST /api/pleroma/admin/users/invite_token`
+## `POST /api/v1/pleroma/admin/users/invite_token`
 
 ### Create an account registration invite token
 
@@ -399,7 +401,7 @@ Response:
 }
 ```
 
-## `GET /api/pleroma/admin/users/invites`
+## `GET /api/v1/pleroma/admin/users/invites`
 
 ### Get a list of generated invites
 
@@ -424,7 +426,7 @@ Response:
 }
 ```
 
-## `POST /api/pleroma/admin/users/revoke_invite`
+## `POST /api/v1/pleroma/admin/users/revoke_invite`
 
 ### Revoke invite by token
 
@@ -445,7 +447,7 @@ Response:
 }
 ```
 
-## `POST /api/pleroma/admin/users/email_invite`
+## `POST /api/v1/pleroma/admin/users/email_invite`
 
 ### Sends registration invite via email
 
@@ -466,7 +468,7 @@ Response:
       ]
     ```
 
-## `GET /api/pleroma/admin/users/:nickname/password_reset`
+## `GET /api/v1/pleroma/admin/users/:nickname/password_reset`
 
 ### Get a password reset token for a given nickname
 
@@ -477,11 +479,11 @@ Response:
 ```json
 {
   "token": "base64 reset token",
-  "link": "https://pleroma.social/api/pleroma/password_reset/url-encoded-base64-token"
+  "link": "https://pleroma.social/api/v1/pleroma/password_reset/url-encoded-base64-token"
 }
 ```
 
-## `PATCH /api/pleroma/admin/users/force_password_reset`
+## `PATCH /api/v1/pleroma/admin/users/force_password_reset`
 
 ### Force passord reset for a user with a given nickname
 
@@ -489,7 +491,7 @@ Response:
   - `nicknames`
 - Response: none (code `204`)
 
-## PUT `/api/pleroma/admin/users/disable_mfa`
+## PUT `/api/v1/pleroma/admin/users/disable_mfa`
 
 ### Disable mfa for user's account.
 
@@ -497,7 +499,7 @@ Response:
   - `nickname`
 - Response: User’s nickname
 
-## `GET /api/pleroma/admin/users/:nickname/credentials`
+## `GET /api/v1/pleroma/admin/users/:nickname/credentials`
 
 ### Get the user's email, password, display and settings-related fields
 
@@ -545,7 +547,7 @@ Response:
 }
 ```
 
-## `PATCH /api/pleroma/admin/users/:nickname/credentials`
+## `PATCH /api/v1/pleroma/admin/users/:nickname/credentials`
 
 ### Change the user's email, password, display and settings-related fields
 
@@ -596,7 +598,7 @@ Status: 404
 {"error": "Not found"}
 ```
 
-## `GET /api/pleroma/admin/reports`
+## `GET /api/v1/pleroma/admin/reports`
 
 ### Get a list of reports
 
@@ -756,17 +758,17 @@ Status: 404
 }
 ```
 
-## `GET /api/pleroma/admin/grouped_reports`
+## `GET /api/v1/pleroma/admin/grouped_reports`
 
 ### Get a list of reports, grouped by status
 
 - Params: none
 - On success: JSON, returns a list of reports, where:
   - `date`: date of the latest report
-  - `account`: the user who has been reported (see `/api/pleroma/admin/reports` for reference)
+  - `account`: the user who has been reported (see `/api/v1/pleroma/admin/reports` for reference)
-  - `status`: reported status (see `/api/pleroma/admin/reports` for reference)
+  - `status`: reported status (see `/api/v1/pleroma/admin/reports` for reference)
-  - `actors`: users who had reported this status (see `/api/pleroma/admin/reports` for reference)
+  - `actors`: users who had reported this status (see `/api/v1/pleroma/admin/reports` for reference)
-  - `reports`: reports (see `/api/pleroma/admin/reports` for reference)
+  - `reports`: reports (see `/api/v1/pleroma/admin/reports` for reference)
 
 ```json
   "reports": [
@@ -780,7 +782,7 @@ Status: 404
   ]
 ```
 
-## `GET /api/pleroma/admin/reports/:id`
+## `GET /api/v1/pleroma/admin/reports/:id`
 
 ### Get an individual report
 
@@ -792,7 +794,7 @@ Status: 404
     - 404 Not Found `"Not found"`
   - On success: JSON, Report object (see above)
 
-## `PATCH /api/pleroma/admin/reports`
+## `PATCH /api/v1/pleroma/admin/reports`
 
 ### Change the state of one or multiple reports
 
@@ -823,7 +825,7 @@ Status: 404
 
   - On success: `204`, empty response
 
-## `POST /api/pleroma/admin/reports/:id/notes`
+## `POST /api/v1/pleroma/admin/reports/:id/notes`
 
 ### Create report note
 
@@ -835,7 +837,7 @@ Status: 404
     - 400 Bad Request `"Invalid parameters"` when `status` is missing
   - On success: `204`, empty response
 
-## `DELETE /api/pleroma/admin/reports/:report_id/notes/:id`
+## `DELETE /api/v1/pleroma/admin/reports/:report_id/notes/:id`
 
 ### Delete report note
 
@@ -847,7 +849,7 @@ Status: 404
     - 400 Bad Request `"Invalid parameters"` when `status` is missing
   - On success: `204`, empty response
 
-## `GET /api/pleroma/admin/statuses/:id`
+## `GET /api/v1/pleroma/admin/statuses/:id`
 
 ### Show status by id
 
@@ -858,7 +860,7 @@ Status: 404
     - 404 Not Found `"Not Found"`
   - On success: JSON, Mastodon Status entity
 
-## `PUT /api/pleroma/admin/statuses/:id`
+## `PUT /api/v1/pleroma/admin/statuses/:id`
 
 ### Change the scope of an individual reported status
 
@@ -873,7 +875,7 @@ Status: 404
     - 404 Not Found `"Not found"`
   - On success: JSON, Mastodon Status entity
 
-## `DELETE /api/pleroma/admin/statuses/:id`
+## `DELETE /api/v1/pleroma/admin/statuses/:id`
 
 ### Delete an individual reported status
 
@@ -885,7 +887,7 @@ Status: 404
     - 404 Not Found `"Not found"`
   - On success: 200 OK `{}`
 
-## `GET /api/pleroma/admin/restart`
+## `GET /api/v1/pleroma/admin/restart`
 
 ### Restarts pleroma application
 
@@ -900,7 +902,7 @@ Status: 404
 {}
 ```
 
-## `GET /api/pleroma/admin/need_reboot`
+## `GET /api/v1/pleroma/admin/need_reboot`
 
 ### Returns the flag whether the pleroma should be restarted
 
@@ -913,7 +915,7 @@ Status: 404
 }
 ```
 
-## `GET /api/pleroma/admin/config`
+## `GET /api/v1/pleroma/admin/config`
 
 ### Get list of merged default settings with saved in database.
 
@@ -940,7 +942,7 @@ Status: 404
 }
 ```
 
-## `POST /api/pleroma/admin/config`
+## `POST /api/v1/pleroma/admin/config`
 
 ### Update config settings
 
@@ -1089,7 +1091,7 @@ config :quack,
 }
 ```
 
-## ` GET /api/pleroma/admin/config/descriptions`
+## ` GET /api/v1/pleroma/admin/config/descriptions`
 
 ### Get JSON with config descriptions.
 Loads json generated from `config/descriptions.exs`.
@@ -1122,7 +1124,7 @@ Loads json generated from `config/descriptions.exs`.
 }]
 ```
 
-## `GET /api/pleroma/admin/moderation_log`
+## `GET /api/v1/pleroma/admin/moderation_log`
 
 ### Get moderation log
 
@@ -1152,7 +1154,7 @@ Loads json generated from `config/descriptions.exs`.
 ]
 ```
 
-## `POST /api/pleroma/admin/reload_emoji`
+## `POST /api/v1/pleroma/admin/reload_emoji`
 
 ### Reload the instance's custom emoji
 
@@ -1160,7 +1162,7 @@ Loads json generated from `config/descriptions.exs`.
 - Params: None
 - Response: JSON, "ok" and 200 status
 
-## `PATCH /api/pleroma/admin/users/confirm_email`
+## `PATCH /api/v1/pleroma/admin/users/confirm_email`
 
 ### Confirm users' emails
 
@@ -1168,7 +1170,7 @@ Loads json generated from `config/descriptions.exs`.
   - `nicknames`
 - Response: Array of user nicknames
 
-## `PATCH /api/pleroma/admin/users/resend_confirmation_email`
+## `PATCH /api/v1/pleroma/admin/users/resend_confirmation_email`
 
 ### Resend confirmation email
 
@@ -1176,13 +1178,13 @@ Loads json generated from `config/descriptions.exs`.
   - `nicknames`
 - Response: Array of user nicknames
 
-## `GET /api/pleroma/admin/stats`
+## `GET /api/v1/pleroma/admin/stats`
 
 ### Stats
 
 - Query Params:
   - *optional* `instance`: **string** instance hostname (without protocol) to get stats for
-- Example: `https://mypleroma.org/api/pleroma/admin/stats?instance=lain.com`
+- Example: `https://mypleroma.org/api/v1/pleroma/admin/stats?instance=lain.com`
 
 - Response:
 
@@ -1197,7 +1199,7 @@ Loads json generated from `config/descriptions.exs`.
 }
 ```
 
-## `GET /api/pleroma/admin/oauth_app`
+## `GET /api/v1/pleroma/admin/oauth_app`
 
 ### List OAuth app
 
@@ -1229,7 +1231,7 @@ Loads json generated from `config/descriptions.exs`.
 ```
 
 
-## `POST /api/pleroma/admin/oauth_app`
+## `POST /api/v1/pleroma/admin/oauth_app`
 
 ### Create OAuth App
 
@@ -1262,7 +1264,7 @@ Loads json generated from `config/descriptions.exs`.
 }
 ```
 
-## `PATCH /api/pleroma/admin/oauth_app/:id`
+## `PATCH /api/v1/pleroma/admin/oauth_app/:id`
 
 ### Update OAuth App
 
@@ -1287,7 +1289,7 @@ Loads json generated from `config/descriptions.exs`.
 }
 ```
 
-## `DELETE /api/pleroma/admin/oauth_app/:id`
+## `DELETE /api/v1/pleroma/admin/oauth_app/:id`
 
 ### Delete OAuth App
 
@@ -1298,7 +1300,7 @@ Loads json generated from `config/descriptions.exs`.
   - On failure:
     - 400 Bad Request `"Invalid parameters"` when `status` is missing
 
-## `GET /api/pleroma/admin/media_proxy_caches`
+## `GET /api/v1/pleroma/admin/media_proxy_caches`
 
 ### Get a list of all banned MediaProxy URLs in Cachex
 
@@ -1322,7 +1324,7 @@ Loads json generated from `config/descriptions.exs`.
 
 ```
 
-## `POST /api/pleroma/admin/media_proxy_caches/delete`
+## `POST /api/v1/pleroma/admin/media_proxy_caches/delete`
 
 ### Remove a banned MediaProxy URL from Cachex
 
@@ -1337,7 +1339,7 @@ Loads json generated from `config/descriptions.exs`.
 
 ```
 
-## `POST /api/pleroma/admin/media_proxy_caches/purge`
+## `POST /api/v1/pleroma/admin/media_proxy_caches/purge`
 
 ### Purge a MediaProxy URL
 
@@ -1353,7 +1355,7 @@ Loads json generated from `config/descriptions.exs`.
 
 ```
 
-## GET /api/pleroma/admin/users/:nickname/chats
+## GET /api/v1/pleroma/admin/users/:nickname/chats
 
 ### List a user's chats
 
@@ -1382,7 +1384,7 @@ Loads json generated from `config/descriptions.exs`.
 ]
 ```
 
-## GET /api/pleroma/admin/chats/:chat_id
+## GET /api/v1/pleroma/admin/chats/:chat_id
 
 ### View a single chat
 
@@ -1409,7 +1411,7 @@ Loads json generated from `config/descriptions.exs`.
 }
 ```
 
-## GET /api/pleroma/admin/chats/:chat_id/messages
+## GET /api/v1/pleroma/admin/chats/:chat_id/messages
 
 ### List the messages in a chat
 
@@ -1447,7 +1449,7 @@ Loads json generated from `config/descriptions.exs`.
 ]
 ```
 
-## DELETE /api/pleroma/admin/chats/:chat_id/messages/:message_id
+## DELETE /api/v1/pleroma/admin/chats/:chat_id/messages/:message_id
 
 ### Delete a single message
 
@@ -1474,7 +1476,7 @@ Loads json generated from `config/descriptions.exs`.
 }
 ```
 
-## `GET /api/pleroma/admin/instance_document/:document_name`
+## `GET /api/v1/pleroma/admin/instance_document/:document_name`
 
 ### Get an instance document
 
@@ -1488,7 +1490,7 @@ Returns the content of the document
 <h1>Instance panel</h1>
 ```
 
-## `PATCH /api/pleroma/admin/instance_document/:document_name`
+## `PATCH /api/v1/pleroma/admin/instance_document/:document_name`
 - Params:
   - `file` (the file to be uploaded, using multipart form data.)
 
@@ -1504,7 +1506,7 @@ Returns the content of the document
 }
 ```
 
-## `DELETE /api/pleroma/admin/instance_document/:document_name`
+## `DELETE /api/v1/pleroma/admin/instance_document/:document_name`
 
 ### Delete an instance document
 
@@ -1516,7 +1518,7 @@ Returns the content of the document
 }
 ```
 
-## `GET /api/pleroma/admin/frontends
+## `GET /api/v1/pleroma/admin/frontends
 
 ### List available frontends
 
@@ -1541,7 +1543,7 @@ Returns the content of the document
 ]
 ```
 
-## `POST /api/pleroma/admin/frontends/install`
+## `POST /api/v1/pleroma/admin/frontends/install`
 
 ### Install a frontend
 

docs/development/API/differences_in_mastoapi_responses.md

@@ -39,6 +39,12 @@ Has these additional fields under the `pleroma` object:
 - `emoji_reactions`: A list with emoji / reaction maps. The format is `{name: "☕", count: 1, me: true}`. Contains no information about the reacting users, for that use the `/statuses/:id/reactions` endpoint.
 - `parent_visible`: If the parent of this post is visible to the user or not.
 
+## Scheduled statuses
+
+Has these additional fields in `params`:
+
+- `expires_in`: the number of seconds the posted activity should expire in.
+
 ## Media Attachments
 
 Has these additional fields under the `pleroma` object:
@@ -92,12 +98,12 @@ Has these additional fields under the `pleroma` object:
 - `hide_followers_count`: boolean, true when the user has follower stat hiding enabled
 - `hide_follows_count`: boolean, true when the user has follow stat hiding enabled
 - `settings_store`: A generic map of settings for frontends. Opaque to the backend. Only returned in `/api/v1/accounts/verify_credentials` and `/api/v1/accounts/update_credentials`
-- `chat_token`: The token needed for Pleroma chat. Only returned in `/api/v1/accounts/verify_credentials`
+- `chat_token`: The token needed for Pleroma shoutbox. Only returned in `/api/v1/accounts/verify_credentials`
 - `deactivated`: boolean, true when the user is deactivated
 - `allow_following_move`: boolean, true when the user allows automatically follow moved following accounts
 - `unread_conversation_count`: The count of unread conversations. Only returned to the account owner.
 - `unread_notifications_count`: The count of unread notifications. Only returned to the account owner.
-- `notification_settings`: object, can be absent. See `/api/pleroma/notification_settings` for the parameters/keys returned.
+- `notification_settings`: object, can be absent. See `/api/v1/pleroma/notification_settings` for the parameters/keys returned.
 - `accepts_chat_messages`: boolean, but can be null if we don't have that information about a user
 - `favicon`: nullable URL string, Favicon image of the user's instance
 

docs/development/API/pleroma_api.md

@@ -4,7 +4,9 @@ Requests that require it can be authenticated with [an OAuth token](https://tool
 
 Request parameters can be passed via [query strings](https://en.wikipedia.org/wiki/Query_string) or as [form data](https://www.w3.org/TR/html401/interact/forms.html). Files must be uploaded as `multipart/form-data`.
 
-## `/api/pleroma/emoji`
+The `/api/v1/pleroma/*` path is backwards compatible with `/api/pleroma/*` (`/api/pleroma/*` will be deprecated in the future).
+
+## `/api/v1/pleroma/emoji`
 ### Lists the custom emoji on that server.
 * Method: `GET`
 * Authentication: not required
@@ -35,7 +37,7 @@ Request parameters can be passed via [query strings](https://en.wikipedia.org/wi
 ```
 * Note: Same data as Mastodon API’s `/api/v1/custom_emojis` but in a different format
 
-## `/api/pleroma/follow_import`
+## `/api/v1/pleroma/follow_import`
 ### Imports your follows, for example from a Mastodon CSV file.
 * Method: `POST`
 * Authentication: required
@@ -44,7 +46,7 @@ Request parameters can be passed via [query strings](https://en.wikipedia.org/wi
 * Response: HTTP 200 on success, 500 on error
 * Note: Users that can't be followed are silently skipped.
 
-## `/api/pleroma/blocks_import`
+## `/api/v1/pleroma/blocks_import`
 ### Imports your blocks.
 * Method: `POST`
 * Authentication: required
@@ -52,7 +54,7 @@ Request parameters can be passed via [query strings](https://en.wikipedia.org/wi
     * `list`: STRING or FILE containing a whitespace-separated list of accounts to block
 * Response: HTTP 200 on success, 500 on error
 
-## `/api/pleroma/mutes_import`
+## `/api/v1/pleroma/mutes_import`
 ### Imports your mutes.
 * Method: `POST`
 * Authentication: required
@@ -60,7 +62,7 @@ Request parameters can be passed via [query strings](https://en.wikipedia.org/wi
     * `list`: STRING or FILE containing a whitespace-separated list of accounts to mute
 * Response: HTTP 200 on success, 500 on error
 
-## `/api/pleroma/captcha`
+## `/api/v1/pleroma/captcha`
 ### Get a new captcha
 * Method: `GET`
 * Authentication: not required
@@ -68,7 +70,7 @@ Request parameters can be passed via [query strings](https://en.wikipedia.org/wi
 * Response: Provider specific JSON, the only guaranteed parameter is `type`
 * Example response: `{"type": "kocaptcha", "token": "whatever", "url": "https://captcha.kotobank.ch/endpoint", "seconds_valid": 300}`
 
-## `/api/pleroma/delete_account`
+## `/api/v1/pleroma/delete_account`
 ### Delete an account
 * Method `POST`
 * Authentication: required
@@ -77,7 +79,7 @@ Request parameters can be passed via [query strings](https://en.wikipedia.org/wi
 * Response: JSON. Returns `{"status": "success"}` if the deletion was successful, `{"error": "[error message]"}` otherwise
 * Example response: `{"error": "Invalid password."}`
 
-## `/api/pleroma/disable_account`
+## `/api/v1/pleroma/disable_account`
 ### Disable an account
 * Method `POST`
 * Authentication: required
@@ -86,21 +88,21 @@ Request parameters can be passed via [query strings](https://en.wikipedia.org/wi
 * Response: JSON. Returns `{"status": "success"}` if the account was successfully disabled, `{"error": "[error message]"}` otherwise
 * Example response: `{"error": "Invalid password."}`
 
-## `/api/pleroma/accounts/mfa`
+## `/api/v1/pleroma/accounts/mfa`
 #### Gets current MFA settings
 * method: `GET`
 * Authentication: required
 * OAuth scope: `read:security`
 * Response: JSON. Returns `{"enabled": "false", "totp": false }`
 
-## `/api/pleroma/accounts/mfa/setup/totp`
+## `/api/v1/pleroma/accounts/mfa/setup/totp`
 #### Pre-setup the MFA/TOTP method
 * method: `GET`
 * Authentication: required
 * OAuth scope: `write:security`
 * Response: JSON. Returns `{"key": [secret_key], "provisioning_uri": "[qr code uri]"  }` when successful, otherwise returns HTTP 422 `{"error": "error_msg"}`
 
-## `/api/pleroma/accounts/mfa/confirm/totp`
+## `/api/v1/pleroma/accounts/mfa/confirm/totp`
 #### Confirms & enables MFA/TOTP support for user account.
 * method: `POST`
 * Authentication: required
@@ -111,7 +113,7 @@ Request parameters can be passed via [query strings](https://en.wikipedia.org/wi
 * Response: JSON. Returns `{}` if the enable was successful, HTTP 422 `{"error": "[error message]"}` otherwise
 
 
-## `/api/pleroma/accounts/mfa/totp`
+## `/api/v1/pleroma/accounts/mfa/totp`
 ####  Disables MFA/TOTP method for user account.
 * method: `DELETE`
 * Authentication: required
@@ -121,14 +123,14 @@ Request parameters can be passed via [query strings](https://en.wikipedia.org/wi
 * Response: JSON. Returns `{}` if the disable was successful, HTTP 422 `{"error": "[error message]"}` otherwise
 * Example response: `{"error": "Invalid password."}`
 
-## `/api/pleroma/accounts/mfa/backup_codes`
+## `/api/v1/pleroma/accounts/mfa/backup_codes`
 ####  Generstes backup codes MFA for user account.
 * method: `GET`
 * Authentication: required
 * OAuth scope: `write:security`
 * Response: JSON. Returns `{"codes": codes}`when successful, otherwise HTTP 422 `{"error": "[error message]"}`
 
-## `/api/pleroma/admin/`
+## `/api/v1/pleroma/admin/`
 See [Admin-API](admin_api.md)
 
 ## `/api/v1/pleroma/notifications/read`
@@ -298,7 +300,7 @@ See [Admin-API](admin_api.md)
 * Note: Behaves exactly the same as `POST /api/v1/upload`.
   Can only accept images - any attempt to upload non-image files will be met with `HTTP 415 Unsupported Media Type`.
 
-## `/api/pleroma/notification_settings`
+## `/api/v1/pleroma/notification_settings`
 ### Updates user notification settings
 * Method `PUT`
 * Authentication: required
@@ -307,7 +309,7 @@ See [Admin-API](admin_api.md)
     * `hide_notification_contents`: BOOLEAN field. When set to true, it removes the contents of a message from the push notification.
 * Response: JSON. Returns `{"status": "success"}` if the update was successful, otherwise returns `{"error": "error_msg"}`
 
-## `/api/pleroma/healthcheck`
+## `/api/v1/pleroma/healthcheck`
 ### Healthcheck endpoint with additional system data.
 * Method `GET`
 * Authentication: not required
@@ -325,7 +327,7 @@ See [Admin-API](admin_api.md)
 }
 ```
 
-## `/api/pleroma/change_email`
+## `/api/v1/pleroma/change_email`
 ### Change account email
 * Method `POST`
 * Authentication: required
@@ -378,7 +380,7 @@ The status posting endpoint takes an additional parameter, `in_reply_to_conversa
 * Params: None
 * Response: JSON, returns a list of Mastodon Conversation entities that were marked as read (200 - healthy, 503 unhealthy).
 
-## `GET /api/pleroma/emoji/pack?name=:name`
+## `GET /api/v1/pleroma/emoji/pack?name=:name`
 
 ### Get pack.json for the pack
 
@@ -397,7 +399,7 @@ The status posting endpoint takes an additional parameter, `in_reply_to_conversa
 }
 ```
 
-## `POST /api/pleroma/emoji/pack?name=:name`
+## `POST /api/v1/pleroma/emoji/pack?name=:name`
 
 ### Creates an empty pack
 
@@ -407,7 +409,7 @@ The status posting endpoint takes an additional parameter, `in_reply_to_conversa
   * `name`: pack name
 * Response: JSON, "ok" and 200 status or 409 if the pack with that name already exists
 
-## `PATCH /api/pleroma/emoji/pack?name=:name`
+## `PATCH /api/v1/pleroma/emoji/pack?name=:name`
 
 ### Updates (replaces) pack metadata
 
@@ -425,7 +427,7 @@ The status posting endpoint takes an additional parameter, `in_reply_to_conversa
 * Response: JSON, updated "metadata" section of the pack and 200 status or 400 if there was a
   problem with the new metadata (the error is specified in the "error" part of the response JSON)
 
-## `DELETE /api/pleroma/emoji/pack?name=:name`
+## `DELETE /api/v1/pleroma/emoji/pack?name=:name`
 
 ### Delete a custom emoji pack
 
@@ -435,7 +437,7 @@ The status posting endpoint takes an additional parameter, `in_reply_to_conversa
   * `name`: pack name
 * Response: JSON, "ok" and 200 status or 500 if there was an error deleting the pack
 
-## `GET /api/pleroma/emoji/packs/import`
+## `GET /api/v1/pleroma/emoji/packs/import`
 
 ### Imports packs from filesystem
 
@@ -444,7 +446,7 @@ The status posting endpoint takes an additional parameter, `in_reply_to_conversa
 * Params: None
 * Response: JSON, returns a list of imported packs.
 
-## `GET /api/pleroma/emoji/packs/remote`
+## `GET /api/v1/pleroma/emoji/packs/remote`
 
 ### Make request to another instance for packs list
 
@@ -456,7 +458,7 @@ The status posting endpoint takes an additional parameter, `in_reply_to_conversa
   * `page_size`: page size for packs (default 50)
 * Response: JSON with the pack list, hashmap with pack name and pack contents
 
-## `POST /api/pleroma/emoji/packs/download`
+## `POST /api/v1/pleroma/emoji/packs/download`
 
 ### Download pack from another instance
 
@@ -469,7 +471,7 @@ The status posting endpoint takes an additional parameter, `in_reply_to_conversa
 * Response: JSON, "ok" with 200 status if the pack was downloaded, or 500 if there were
   errors downloading the pack
 
-## `POST /api/pleroma/emoji/packs/files?name=:name`
+## `POST /api/v1/pleroma/emoji/packs/files?name=:name`
 
 ### Add new file to the pack
 
@@ -482,7 +484,7 @@ The status posting endpoint takes an additional parameter, `in_reply_to_conversa
   * `filename`: (*optional*) new emoji file name. If not specified will be taken from original filename.
 * Response: JSON, list of files for updated pack (hashmap -> shortcode => filename) with status 200, either error status with error message.
 
-## `PATCH /api/pleroma/emoji/packs/files?name=:name`
+## `PATCH /api/v1/pleroma/emoji/packs/files?name=:name`
 
 ### Update emoji file from pack
 
@@ -496,7 +498,7 @@ The status posting endpoint takes an additional parameter, `in_reply_to_conversa
   * `force`: (*optional*) with true value to overwrite existing emoji with new shortcode
 * Response: JSON, list with updated files for updated pack (hashmap -> shortcode => filename) with status 200, either error status with error message.
 
-## `DELETE /api/pleroma/emoji/packs/files?name=:name`
+## `DELETE /api/v1/pleroma/emoji/packs/files?name=:name`
 
 ### Delete emoji file from pack
 
@@ -507,7 +509,7 @@ The status posting endpoint takes an additional parameter, `in_reply_to_conversa
   * `shortcode`: emoji file shortcode
 * Response: JSON, list with updated files for updated pack (hashmap -> shortcode => filename) with status 200, either error status with error message.
 
-## `GET /api/pleroma/emoji/packs`
+## `GET /api/v1/pleroma/emoji/packs`
 
 ### Lists local custom emoji packs
 
@@ -528,7 +530,7 @@ The status posting endpoint takes an additional parameter, `in_reply_to_conversa
 }
 ```
 
-## `GET /api/pleroma/emoji/packs/archive?name=:name`
+## `GET /api/v1/pleroma/emoji/packs/archive?name=:name`
 
 ### Requests a local pack archive from the instance
 

docs/installation/otp_en.md

@@ -150,7 +150,7 @@ su pleroma -s $SHELL -lc "./bin/pleroma_ctl migrate"
 # su pleroma -s $SHELL -lc "./bin/pleroma_ctl migrate --migrations-path priv/repo/optional_migrations/rum_indexing/"
 
 # Start the instance to verify that everything is working as expected
-su pleroma -s $SHELL -lc "export $(cat /opt/pleroma/config/pleroma.env); ./bin/pleroma daemon"
+su pleroma -s $SHELL -lc "./bin/pleroma daemon"
 
 # Wait for about 20 seconds and query the instance endpoint, if it shows your uri, name and email correctly, you are configured correctly
 sleep 20 && curl http://localhost:4000/api/v1/instance

lib/mix/tasks/pleroma/config.ex

@@ -27,7 +27,7 @@ defmodule Mix.Tasks.Pleroma.Config do
 
       {opts, _} =
         OptionParser.parse!(options,
-          strict: [env: :string, delete: :boolean],
+          strict: [env: :string, delete: :boolean, path: :string],
           aliases: [d: :delete]
         )
 
@@ -259,18 +259,43 @@ defmodule Mix.Tasks.Pleroma.Config do
   defp migrate_from_db(opts) do
     env = opts[:env] || Pleroma.Config.get(:env)
 
+    filename = "#{env}.exported_from_db.secret.exs"
+
     config_path =
-      if Pleroma.Config.get(:release) do
+      cond do
-        :config_path
+        opts[:path] ->
-        |> Pleroma.Config.get()
+          opts[:path]
-        |> Path.dirname()
+
-      else
+        Pleroma.Config.get(:release) ->
-        "config"
+          :config_path
+          |> Pleroma.Config.get()
+          |> Path.dirname()
+
+        true ->
+          "config"
       end
-      |> Path.join("#{env}.exported_from_db.secret.exs")
+      |> Path.join(filename)
 
-    file = File.open!(config_path, [:write, :utf8])
+    with {:ok, file} <- File.open(config_path, [:write, :utf8]) do
+      write_config(file, config_path, opts)
+      shell_info("Database configuration settings have been exported to #{config_path}")
+    else
+      _ ->
+        shell_error("Impossible to save settings to this directory #{Path.dirname(config_path)}")
+        tmp_config_path = Path.join(System.tmp_dir!(), filename)
+        file = File.open!(tmp_config_path)
 
+        shell_info(
+          "Saving database configuration settings to #{tmp_config_path}. Copy it to the #{
+            Path.dirname(config_path)
+          } manually."
+        )
+
+        write_config(file, tmp_config_path, opts)
+    end
+  end
+
+  defp write_config(file, path, opts) do
     IO.write(file, config_header())
 
     ConfigDB
@@ -278,11 +303,7 @@ defmodule Mix.Tasks.Pleroma.Config do
     |> Enum.each(&write_and_delete(&1, file, opts[:delete]))
 
     :ok = File.close(file)
-    System.cmd("mix", ["format", config_path])
+    System.cmd("mix", ["format", path])
-
-    shell_info(
-      "Database configuration settings have been exported to config/#{env}.exported_from_db.secret.exs"
-    )
   end
 
   if Code.ensure_loaded?(Config.Reader) do

lib/mix/tasks/pleroma/database.ex

@@ -8,10 +8,13 @@ defmodule Mix.Tasks.Pleroma.Database do
   alias Pleroma.Object
   alias Pleroma.Repo
   alias Pleroma.User
+
   require Logger
   require Pleroma.Constants
+
   import Ecto.Query
   import Mix.Pleroma
+
   use Mix.Task
 
   @shortdoc "A collection of database related tasks"
@@ -214,4 +217,32 @@ defmodule Mix.Tasks.Pleroma.Database do
       shell_info('Done.')
     end
   end
+
+  # Rolls back a specific migration (leaving subsequent migrations applied).
+  # WARNING: imposes a risk of unrecoverable data loss — proceed at your own responsibility.
+  # Based on https://stackoverflow.com/a/53825840
+  def run(["rollback", version]) do
+    prompt = "SEVERE WARNING: this operation may result in unrecoverable data loss. Continue?"
+
+    if shell_prompt(prompt, "n") in ~w(Yn Y y) do
+      {_, result, _} =
+        Ecto.Migrator.with_repo(Pleroma.Repo, fn repo ->
+          version = String.to_integer(version)
+          re = ~r/^#{version}_.*\.exs/
+          path = Ecto.Migrator.migrations_path(repo)
+
+          with {_, "" <> file} <- {:find, Enum.find(File.ls!(path), &String.match?(&1, re))},
+               {_, [{mod, _} | _]} <- {:compile, Code.compile_file(Path.join(path, file))},
+               {_, :ok} <- {:rollback, Ecto.Migrator.down(repo, version, mod)} do
+            {:ok, "Reversed migration: #{file}"}
+          else
+            {:find, _} -> {:error, "No migration found with version prefix: #{version}"}
+            {:compile, e} -> {:error, "Problem compiling migration module: #{inspect(e)}"}
+            {:rollback, e} -> {:error, "Problem reversing migration: #{inspect(e)}"}
+          end
+        end)
+
+      shell_info(inspect(result))
+    end
+  end
 end

lib/mix/tasks/pleroma/ecto/rollback.ex

@@ -20,7 +20,8 @@ defmodule Mix.Tasks.Pleroma.Ecto.Rollback do
     start: :boolean,
     quiet: :boolean,
     log_sql: :boolean,
-    migrations_path: :string
+    migrations_path: :string,
+    env: :string
   ]
 
   @moduledoc """
@@ -59,7 +60,7 @@ defmodule Mix.Tasks.Pleroma.Ecto.Rollback do
     level = Logger.level()
     Logger.configure(level: :info)
 
-    if Pleroma.Config.get(:env) == :test do
+    if opts[:env] == "test" do
       Logger.info("Rollback succesfully")
     else
       {:ok, _, _} =

lib/pleroma/activity.ex

@@ -113,6 +113,7 @@ defmodule Pleroma.Activity do
     from([a] in query,
       left_join: b in Bookmark,
       on: b.user_id == ^user.id and b.activity_id == a.id,
+      as: :bookmark,
       preload: [bookmark: b]
     )
   end
@@ -123,6 +124,7 @@ defmodule Pleroma.Activity do
     from([a] in query,
       left_join: r in ReportNote,
       on: a.id == r.activity_id,
+      as: :report_note,
       preload: [report_notes: r]
     )
   end

lib/pleroma/activity/ir/topics.ex

@@ -48,14 +48,12 @@ defmodule Pleroma.Activity.Ir.Topics do
     tags
   end
 
-  defp hashtags_to_topics(%{data: %{"tag" => tags}}) do
+  defp hashtags_to_topics(object) do
-    tags
+    object
-    |> Enum.filter(&is_bitstring(&1))
+    |> Object.hashtags()
-    |> Enum.map(fn tag -> "hashtag:" <> tag end)
+    |> Enum.map(fn hashtag -> "hashtag:" <> hashtag end)
   end
 
-  defp hashtags_to_topics(_), do: []
-
   defp remote_topics(%{local: true}), do: []
 
   defp remote_topics(%{actor: actor}) when is_binary(actor),

lib/pleroma/application.ex

@@ -97,15 +97,13 @@ defmodule Pleroma.Application do
           Pleroma.Stats,
           Pleroma.JobQueueMonitor,
           {Majic.Pool, [name: Pleroma.MajicPool, pool_size: Config.get([:majic_pool, :size], 2)]},
-          {Oban, Config.get(Oban)}
+          {Oban, Config.get(Oban)},
+          Pleroma.Web.Endpoint
         ] ++
         task_children(@mix_env) ++
         dont_run_in_test(@mix_env) ++
         chat_child(chat_enabled?()) ++
-        [
+        [Pleroma.Gopher.Server]
-          Pleroma.Web.Endpoint,
-          Pleroma.Gopher.Server
-        ]
 
     # See http://elixir-lang.org/docs/stable/elixir/Supervisor.html
     # for other strategies and supported options
@@ -230,6 +228,12 @@ defmodule Pleroma.Application do
          keys: :duplicate,
          partitions: System.schedulers_online()
        ]}
+    ] ++ background_migrators()
+  end
+
+  defp background_migrators do
+    [
+      Pleroma.Migrators.HashtagsTableMigrator
     ]
   end
 

lib/pleroma/config.ex

@@ -99,4 +99,8 @@ defmodule Pleroma.Config do
   def oauth_consumer_strategies, do: get([:auth, :oauth_consumer_strategies], [])
 
   def oauth_consumer_enabled?, do: oauth_consumer_strategies() != []
+
+  def feature_enabled?(feature_name) do
+    get([:features, feature_name]) not in [nil, false, :disabled, :auto]
+  end
 end

lib/pleroma/constants.ex

@@ -18,7 +18,8 @@ defmodule Pleroma.Constants do
       "emoji",
       "context_id",
       "deleted_activity_id",
-      "pleroma_internal"
+      "pleroma_internal",
+      "generator"
     ]
   )
 

lib/pleroma/data_migration.ex

@@ -0,0 +1,45 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2021 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.DataMigration do
+  use Ecto.Schema
+
+  alias Pleroma.DataMigration
+  alias Pleroma.DataMigration.State
+  alias Pleroma.Repo
+
+  import Ecto.Changeset
+  import Ecto.Query
+
+  schema "data_migrations" do
+    field(:name, :string)
+    field(:state, State, default: :pending)
+    field(:feature_lock, :boolean, default: false)
+    field(:params, :map, default: %{})
+    field(:data, :map, default: %{})
+
+    timestamps()
+  end
+
+  def changeset(data_migration, params \\ %{}) do
+    data_migration
+    |> cast(params, [:name, :state, :feature_lock, :params, :data])
+    |> validate_required([:name])
+    |> unique_constraint(:name)
+  end
+
+  def update_one_by_id(id, params \\ %{}) do
+    with {1, _} <-
+           from(dm in DataMigration, where: dm.id == ^id)
+           |> Repo.update_all(set: params) do
+      :ok
+    end
+  end
+
+  def get_by_name(name) do
+    Repo.get_by(DataMigration, name: name)
+  end
+
+  def populate_hashtags_table, do: get_by_name("populate_hashtags_table")
+end

lib/pleroma/delivery.ex

@@ -9,7 +9,6 @@ defmodule Pleroma.Delivery do
   alias Pleroma.Object
   alias Pleroma.Repo
   alias Pleroma.User
-  alias Pleroma.User
 
   import Ecto.Changeset
   import Ecto.Query

lib/pleroma/ecto_enums.ex

@@ -17,3 +17,11 @@ defenum(Pleroma.FollowingRelationship.State,
   follow_accept: 2,
   follow_reject: 3
 )
+
+defenum(Pleroma.DataMigration.State,
+  pending: 1,
+  running: 2,
+  complete: 3,
+  failed: 4,
+  manual: 5
+)

lib/pleroma/hashtag.ex

@@ -0,0 +1,106 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Hashtag do
+  use Ecto.Schema
+
+  import Ecto.Changeset
+  import Ecto.Query
+
+  alias Ecto.Multi
+  alias Pleroma.Hashtag
+  alias Pleroma.Object
+  alias Pleroma.Repo
+
+  schema "hashtags" do
+    field(:name, :string)
+
+    many_to_many(:objects, Object, join_through: "hashtags_objects", on_replace: :delete)
+
+    timestamps()
+  end
+
+  def normalize_name(name) do
+    name
+    |> String.downcase()
+    |> String.trim()
+  end
+
+  def get_or_create_by_name(name) do
+    changeset = changeset(%Hashtag{}, %{name: name})
+
+    Repo.insert(
+      changeset,
+      on_conflict: [set: [name: get_field(changeset, :name)]],
+      conflict_target: :name,
+      returning: true
+    )
+  end
+
+  def get_or_create_by_names(names) when is_list(names) do
+    names = Enum.map(names, &normalize_name/1)
+    timestamp = NaiveDateTime.truncate(NaiveDateTime.utc_now(), :second)
+
+    structs =
+      Enum.map(names, fn name ->
+        %Hashtag{}
+        |> changeset(%{name: name})
+        |> Map.get(:changes)
+        |> Map.merge(%{inserted_at: timestamp, updated_at: timestamp})
+      end)
+
+    try do
+      with {:ok, %{query_op: hashtags}} <-
+             Multi.new()
+             |> Multi.insert_all(:insert_all_op, Hashtag, structs,
+               on_conflict: :nothing,
+               conflict_target: :name
+             )
+             |> Multi.run(:query_op, fn _repo, _changes ->
+               {:ok, Repo.all(from(ht in Hashtag, where: ht.name in ^names))}
+             end)
+             |> Repo.transaction() do
+        {:ok, hashtags}
+      else
+        {:error, _name, value, _changes_so_far} -> {:error, value}
+      end
+    rescue
+      e -> {:error, e}
+    end
+  end
+
+  def changeset(%Hashtag{} = struct, params) do
+    struct
+    |> cast(params, [:name])
+    |> update_change(:name, &normalize_name/1)
+    |> validate_required([:name])
+    |> unique_constraint(:name)
+  end
+
+  def unlink(%Object{id: object_id}) do
+    with {_, hashtag_ids} <-
+           from(hto in "hashtags_objects",
+             where: hto.object_id == ^object_id,
+             select: hto.hashtag_id
+           )
+           |> Repo.delete_all(),
+         {:ok, unreferenced_count} <- delete_unreferenced(hashtag_ids) do
+      {:ok, length(hashtag_ids), unreferenced_count}
+    end
+  end
+
+  @delete_unreferenced_query """
+  DELETE FROM hashtags WHERE id IN
+    (SELECT hashtags.id FROM hashtags
+      LEFT OUTER JOIN hashtags_objects
+        ON hashtags_objects.hashtag_id = hashtags.id
+      WHERE hashtags_objects.hashtag_id IS NULL AND hashtags.id = ANY($1));
+  """
+
+  def delete_unreferenced(ids) do
+    with {:ok, %{num_rows: deleted_count}} <- Repo.query(@delete_unreferenced_query, [ids]) do
+      {:ok, deleted_count}
+    end
+  end
+end

lib/pleroma/migrators/hashtags_table_migrator.ex

@@ -0,0 +1,208 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2021 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Migrators.HashtagsTableMigrator do
+  defmodule State do
+    use Pleroma.Migrators.Support.BaseMigratorState
+
+    @impl Pleroma.Migrators.Support.BaseMigratorState
+    defdelegate data_migration(), to: Pleroma.DataMigration, as: :populate_hashtags_table
+  end
+
+  use Pleroma.Migrators.Support.BaseMigrator
+
+  alias Pleroma.Hashtag
+  alias Pleroma.Migrators.Support.BaseMigrator
+  alias Pleroma.Object
+
+  @impl BaseMigrator
+  def feature_config_path, do: [:features, :improved_hashtag_timeline]
+
+  @impl BaseMigrator
+  def fault_rate_allowance, do: Config.get([:populate_hashtags_table, :fault_rate_allowance], 0)
+
+  @impl BaseMigrator
+  def perform do
+    data_migration_id = data_migration_id()
+    max_processed_id = get_stat(:max_processed_id, 0)
+
+    Logger.info("Transferring embedded hashtags to `hashtags` (from oid: #{max_processed_id})...")
+
+    query()
+    |> where([object], object.id > ^max_processed_id)
+    |> Repo.chunk_stream(100, :batches, timeout: :infinity)
+    |> Stream.each(fn objects ->
+      object_ids = Enum.map(objects, & &1.id)
+
+      results = Enum.map(objects, &transfer_object_hashtags(&1))
+
+      failed_ids =
+        results
+        |> Enum.filter(&(elem(&1, 0) == :error))
+        |> Enum.map(&elem(&1, 1))
+
+      # Count of objects with hashtags: `{:noop, id}` is returned for objects having other AS2 tags
+      chunk_affected_count =
+        results
+        |> Enum.filter(&(elem(&1, 0) == :ok))
+        |> length()
+
+      for failed_id <- failed_ids do
+        _ =
+          Repo.query(
+            "INSERT INTO data_migration_failed_ids(data_migration_id, record_id) " <>
+              "VALUES ($1, $2) ON CONFLICT DO NOTHING;",
+            [data_migration_id, failed_id]
+          )
+      end
+
+      _ =
+        Repo.query(
+          "DELETE FROM data_migration_failed_ids " <>
+            "WHERE data_migration_id = $1 AND record_id = ANY($2)",
+          [data_migration_id, object_ids -- failed_ids]
+        )
+
+      max_object_id = Enum.at(object_ids, -1)
+
+      put_stat(:max_processed_id, max_object_id)
+      increment_stat(:iteration_processed_count, length(object_ids))
+      increment_stat(:processed_count, length(object_ids))
+      increment_stat(:failed_count, length(failed_ids))
+      increment_stat(:affected_count, chunk_affected_count)
+      put_stat(:records_per_second, records_per_second())
+      persist_state()
+
+      # A quick and dirty approach to controlling the load this background migration imposes
+      sleep_interval = Config.get([:populate_hashtags_table, :sleep_interval_ms], 0)
+      Process.sleep(sleep_interval)
+    end)
+    |> Stream.run()
+  end
+
+  @impl BaseMigrator
+  def query do
+    # Note: most objects have Mention-type AS2 tags and no hashtags (but we can't filter them out)
+    # Note: not checking activity type, expecting remove_non_create_objects_hashtags/_ to clean up
+    from(
+      object in Object,
+      where:
+        fragment("(?)->'tag' IS NOT NULL AND (?)->'tag' != '[]'::jsonb", object.data, object.data),
+      select: %{
+        id: object.id,
+        tag: fragment("(?)->'tag'", object.data)
+      }
+    )
+    |> join(:left, [o], hashtags_objects in fragment("SELECT object_id FROM hashtags_objects"),
+      on: hashtags_objects.object_id == o.id
+    )
+    |> where([_o, hashtags_objects], is_nil(hashtags_objects.object_id))
+  end
+
+  @spec transfer_object_hashtags(Map.t()) :: {:noop | :ok | :error, integer()}
+  defp transfer_object_hashtags(object) do
+    embedded_tags = if Map.has_key?(object, :tag), do: object.tag, else: object.data["tag"]
+    hashtags = Object.object_data_hashtags(%{"tag" => embedded_tags})
+
+    if Enum.any?(hashtags) do
+      transfer_object_hashtags(object, hashtags)
+    else
+      {:noop, object.id}
+    end
+  end
+
+  defp transfer_object_hashtags(object, hashtags) do
+    Repo.transaction(fn ->
+      with {:ok, hashtag_records} <- Hashtag.get_or_create_by_names(hashtags) do
+        maps = Enum.map(hashtag_records, &%{hashtag_id: &1.id, object_id: object.id})
+        base_error = "ERROR when inserting hashtags_objects for object with id #{object.id}"
+
+        try do
+          with {rows_count, _} when is_integer(rows_count) <-
+                 Repo.insert_all("hashtags_objects", maps, on_conflict: :nothing) do
+            object.id
+          else
+            e ->
+              Logger.error("#{base_error}: #{inspect(e)}")
+              Repo.rollback(object.id)
+          end
+        rescue
+          e ->
+            Logger.error("#{base_error}: #{inspect(e)}")
+            Repo.rollback(object.id)
+        end
+      else
+        e ->
+          error = "ERROR: could not create hashtags for object #{object.id}: #{inspect(e)}"
+          Logger.error(error)
+          Repo.rollback(object.id)
+      end
+    end)
+  end
+
+  @impl BaseMigrator
+  def retry_failed do
+    data_migration_id = data_migration_id()
+
+    failed_objects_query()
+    |> Repo.chunk_stream(100, :one)
+    |> Stream.each(fn object ->
+      with {res, _} when res != :error <- transfer_object_hashtags(object) do
+        _ =
+          Repo.query(
+            "DELETE FROM data_migration_failed_ids " <>
+              "WHERE data_migration_id = $1 AND record_id = $2",
+            [data_migration_id, object.id]
+          )
+      end
+    end)
+    |> Stream.run()
+
+    put_stat(:failed_count, failures_count())
+    persist_state()
+
+    force_continue()
+  end
+
+  defp failed_objects_query do
+    from(o in Object)
+    |> join(:inner, [o], dmf in fragment("SELECT * FROM data_migration_failed_ids"),
+      on: dmf.record_id == o.id
+    )
+    |> where([_o, dmf], dmf.data_migration_id == ^data_migration_id())
+    |> order_by([o], asc: o.id)
+  end
+
+  @doc """
+  Service func to delete `hashtags_objects` for legacy objects not associated with Create activity.
+  Also deletes unreferenced `hashtags` records (might occur after deletion of `hashtags_objects`).
+  """
+  def delete_non_create_activities_hashtags do
+    hashtags_objects_cleanup_query = """
+    DELETE FROM hashtags_objects WHERE object_id IN
+      (SELECT DISTINCT objects.id FROM objects
+        JOIN hashtags_objects ON hashtags_objects.object_id = objects.id LEFT JOIN activities
+          ON COALESCE(activities.data->'object'->>'id', activities.data->>'object') =
+            (objects.data->>'id')
+          AND activities.data->>'type' = 'Create'
+        WHERE activities.id IS NULL);
+    """
+
+    hashtags_cleanup_query = """
+    DELETE FROM hashtags WHERE id IN
+      (SELECT hashtags.id FROM hashtags
+        LEFT OUTER JOIN hashtags_objects
+          ON hashtags_objects.hashtag_id = hashtags.id
+        WHERE hashtags_objects.hashtag_id IS NULL);
+    """
+
+    {:ok, %{num_rows: hashtags_objects_count}} =
+      Repo.query(hashtags_objects_cleanup_query, [], timeout: :infinity)
+
+    {:ok, %{num_rows: hashtags_count}} =
+      Repo.query(hashtags_cleanup_query, [], timeout: :infinity)
+
+    {:ok, hashtags_objects_count, hashtags_count}
+  end
+end

lib/pleroma/migrators/support/base_migrator.ex

@@ -0,0 +1,210 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2021 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Migrators.Support.BaseMigrator do
+  @moduledoc """
+  Base background migrator functionality.
+  """
+
+  @callback perform() :: any()
+  @callback retry_failed() :: any()
+  @callback feature_config_path() :: list(atom())
+  @callback query() :: Ecto.Query.t()
+  @callback fault_rate_allowance() :: integer() | float()
+
+  defmacro __using__(_opts) do
+    quote do
+      use GenServer
+
+      require Logger
+
+      import Ecto.Query
+
+      alias __MODULE__.State
+      alias Pleroma.Config
+      alias Pleroma.Repo
+
+      @behaviour Pleroma.Migrators.Support.BaseMigrator
+
+      defdelegate data_migration(), to: State
+      defdelegate data_migration_id(), to: State
+      defdelegate state(), to: State
+      defdelegate persist_state(), to: State, as: :persist_to_db
+      defdelegate get_stat(key, value \\ nil), to: State, as: :get_data_key
+      defdelegate put_stat(key, value), to: State, as: :put_data_key
+      defdelegate increment_stat(key, increment), to: State, as: :increment_data_key
+
+      @reg_name {:global, __MODULE__}
+
+      def whereis, do: GenServer.whereis(@reg_name)
+
+      def start_link(_) do
+        case whereis() do
+          nil ->
+            GenServer.start_link(__MODULE__, nil, name: @reg_name)
+
+          pid ->
+            {:ok, pid}
+        end
+      end
+
+      @impl true
+      def init(_) do
+        {:ok, nil, {:continue, :init_state}}
+      end
+
+      @impl true
+      def handle_continue(:init_state, _state) do
+        {:ok, _} = State.start_link(nil)
+
+        data_migration = data_migration()
+        manual_migrations = Config.get([:instance, :manual_data_migrations], [])
+
+        cond do
+          Config.get(:env) == :test ->
+            update_status(:noop)
+
+          is_nil(data_migration) ->
+            message = "Data migration does not exist."
+            update_status(:failed, message)
+            Logger.error("#{__MODULE__}: #{message}")
+
+          data_migration.state == :manual or data_migration.name in manual_migrations ->
+            message = "Data migration is in manual execution or manual fix mode."
+            update_status(:manual, message)
+            Logger.warn("#{__MODULE__}: #{message}")
+
+          data_migration.state == :complete ->
+            on_complete(data_migration)
+
+          true ->
+            send(self(), :perform)
+        end
+
+        {:noreply, nil}
+      end
+
+      @impl true
+      def handle_info(:perform, state) do
+        State.reinit()
+
+        update_status(:running)
+        put_stat(:iteration_processed_count, 0)
+        put_stat(:started_at, NaiveDateTime.utc_now())
+
+        perform()
+
+        fault_rate = fault_rate()
+        put_stat(:fault_rate, fault_rate)
+        fault_rate_allowance = fault_rate_allowance()
+
+        cond do
+          fault_rate == 0 ->
+            set_complete()
+
+          is_float(fault_rate) and fault_rate <= fault_rate_allowance ->
+            message = """
+            Done with fault rate of #{fault_rate} which doesn't exceed #{fault_rate_allowance}.
+            Putting data migration to manual fix mode. Try running `#{__MODULE__}.retry_failed/0`.
+            """
+
+            Logger.warn("#{__MODULE__}: #{message}")
+            update_status(:manual, message)
+            on_complete(data_migration())
+
+          true ->
+            message = "Too many failures. Try running `#{__MODULE__}.retry_failed/0`."
+            Logger.error("#{__MODULE__}: #{message}")
+            update_status(:failed, message)
+        end
+
+        persist_state()
+        {:noreply, state}
+      end
+
+      defp on_complete(data_migration) do
+        if data_migration.feature_lock || feature_state() == :disabled do
+          Logger.warn(
+            "#{__MODULE__}: migration complete but feature is locked; consider enabling."
+          )
+
+          :noop
+        else
+          Config.put(feature_config_path(), :enabled)
+          :ok
+        end
+      end
+
+      @doc "Approximate count for current iteration (including processed records count)"
+      def count(force \\ false, timeout \\ :infinity) do
+        stored_count = get_stat(:count)
+
+        if stored_count && !force do
+          stored_count
+        else
+          processed_count = get_stat(:processed_count, 0)
+          max_processed_id = get_stat(:max_processed_id, 0)
+          query = where(query(), [entity], entity.id > ^max_processed_id)
+
+          count = Repo.aggregate(query, :count, :id, timeout: timeout) + processed_count
+          put_stat(:count, count)
+          persist_state()
+
+          count
+        end
+      end
+
+      def failures_count do
+        with {:ok, %{rows: [[count]]}} <-
+               Repo.query(
+                 "SELECT COUNT(record_id) FROM data_migration_failed_ids WHERE data_migration_id = $1;",
+                 [data_migration_id()]
+               ) do
+          count
+        end
+      end
+
+      def feature_state, do: Config.get(feature_config_path())
+
+      def force_continue do
+        send(whereis(), :perform)
+      end
+
+      def force_restart do
+        :ok = State.reset()
+        force_continue()
+      end
+
+      def set_complete do
+        update_status(:complete)
+        persist_state()
+        on_complete(data_migration())
+      end
+
+      defp update_status(status, message \\ nil) do
+        put_stat(:state, status)
+        put_stat(:message, message)
+      end
+
+      defp fault_rate do
+        with failures_count when is_integer(failures_count) <- failures_count() do
+          failures_count / Enum.max([get_stat(:affected_count, 0), 1])
+        else
+          _ -> :error
+        end
+      end
+
+      defp records_per_second do
+        get_stat(:iteration_processed_count, 0) / Enum.max([running_time(), 1])
+      end
+
+      defp running_time do
+        NaiveDateTime.diff(
+          NaiveDateTime.utc_now(),
+          get_stat(:started_at, NaiveDateTime.utc_now())
+        )
+      end
+    end
+  end
+end

lib/pleroma/migrators/support/base_migrator_state.ex

@@ -0,0 +1,117 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2021 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Migrators.Support.BaseMigratorState do
+  @moduledoc """
+  Base background migrator state functionality.
+  """
+
+  @callback data_migration() :: Pleroma.DataMigration.t()
+
+  defmacro __using__(_opts) do
+    quote do
+      use Agent
+
+      alias Pleroma.DataMigration
+
+      @behaviour Pleroma.Migrators.Support.BaseMigratorState
+      @reg_name {:global, __MODULE__}
+
+      def start_link(_) do
+        Agent.start_link(fn -> load_state_from_db() end, name: @reg_name)
+      end
+
+      def data_migration, do: raise("data_migration/0 is not implemented")
+      defoverridable data_migration: 0
+
+      defp load_state_from_db do
+        data_migration = data_migration()
+
+        data =
+          if data_migration do
+            Map.new(data_migration.data, fn {k, v} -> {String.to_atom(k), v} end)
+          else
+            %{}
+          end
+
+        %{
+          data_migration_id: data_migration && data_migration.id,
+          data: data
+        }
+      end
+
+      def persist_to_db do
+        %{data_migration_id: data_migration_id, data: data} = state()
+
+        if data_migration_id do
+          DataMigration.update_one_by_id(data_migration_id, data: data)
+        else
+          {:error, :nil_data_migration_id}
+        end
+      end
+
+      def reset do
+        %{data_migration_id: data_migration_id} = state()
+
+        with false <- is_nil(data_migration_id),
+             :ok <-
+               DataMigration.update_one_by_id(data_migration_id,
+                 state: :pending,
+                 data: %{}
+               ) do
+          reinit()
+        else
+          true -> {:error, :nil_data_migration_id}
+          e -> e
+        end
+      end
+
+      def reinit do
+        Agent.update(@reg_name, fn _state -> load_state_from_db() end)
+      end
+
+      def state do
+        Agent.get(@reg_name, & &1)
+      end
+
+      def get_data_key(key, default \\ nil) do
+        get_in(state(), [:data, key]) || default
+      end
+
+      def put_data_key(key, value) do
+        _ = persist_non_data_change(key, value)
+
+        Agent.update(@reg_name, fn state ->
+          put_in(state, [:data, key], value)
+        end)
+      end
+
+      def increment_data_key(key, increment \\ 1) do
+        Agent.update(@reg_name, fn state ->
+          initial_value = get_in(state, [:data, key]) || 0
+          updated_value = initial_value + increment
+          put_in(state, [:data, key], updated_value)
+        end)
+      end
+
+      defp persist_non_data_change(:state, value) do
+        with true <- get_data_key(:state) != value,
+             true <- value in Pleroma.DataMigration.State.__valid_values__(),
+             %{data_migration_id: data_migration_id} when not is_nil(data_migration_id) <-
+               state() do
+          DataMigration.update_one_by_id(data_migration_id, state: value)
+        else
+          false -> :ok
+          _ -> {:error, :nil_data_migration_id}
+        end
+      end
+
+      defp persist_non_data_change(_, _) do
+        nil
+      end
+
+      def data_migration_id, do: Map.get(state(), :data_migration_id)
+    end
+  end
+end

lib/pleroma/object.ex

@@ -10,6 +10,7 @@ defmodule Pleroma.Object do
 
   alias Pleroma.Activity
   alias Pleroma.Config
+  alias Pleroma.Hashtag
   alias Pleroma.Object
   alias Pleroma.Object.Fetcher
   alias Pleroma.ObjectTombstone
@@ -28,6 +29,8 @@ defmodule Pleroma.Object do
   schema "objects" do
     field(:data, :map)
 
+    many_to_many(:hashtags, Hashtag, join_through: "hashtags_objects", on_replace: :delete)
+
     timestamps()
   end
 
@@ -49,7 +52,8 @@ defmodule Pleroma.Object do
   end
 
   def create(data) do
-    Object.change(%Object{}, %{data: data})
+    %Object{}
+    |> Object.change(%{data: data})
     |> Repo.insert()
   end
 
@@ -58,8 +62,41 @@ defmodule Pleroma.Object do
     |> cast(params, [:data])
     |> validate_required([:data])
     |> unique_constraint(:ap_id, name: :objects_unique_apid_index)
+    # Expecting `maybe_handle_hashtags_change/1` to run last:
+    |> maybe_handle_hashtags_change(struct)
   end
 
+  # Note: not checking activity type (assuming non-legacy objects are associated with Create act.)
+  defp maybe_handle_hashtags_change(changeset, struct) do
+    with %Ecto.Changeset{valid?: true} <- changeset,
+         data_hashtags_change = get_change(changeset, :data),
+         {_, true} <- {:changed, hashtags_changed?(struct, data_hashtags_change)},
+         {:ok, hashtag_records} <-
+           data_hashtags_change
+           |> object_data_hashtags()
+           |> Hashtag.get_or_create_by_names() do
+      put_assoc(changeset, :hashtags, hashtag_records)
+    else
+      %{valid?: false} ->
+        changeset
+
+      {:changed, false} ->
+        changeset
+
+      {:error, _} ->
+        validate_change(changeset, :data, fn _, _ ->
+          [data: "error referencing hashtags"]
+        end)
+    end
+  end
+
+  defp hashtags_changed?(%Object{} = struct, %{"tag" => _} = data) do
+    Enum.sort(embedded_hashtags(struct)) !=
+      Enum.sort(object_data_hashtags(data))
+  end
+
+  defp hashtags_changed?(_, _), do: false
+
   def get_by_id(nil), do: nil
   def get_by_id(id), do: Repo.get(Object, id)
 
@@ -187,9 +224,13 @@ defmodule Pleroma.Object do
   def swap_object_with_tombstone(object) do
     tombstone = make_tombstone(object)
 
-    object
+    with {:ok, object} <-
-    |> Object.change(%{data: tombstone})
+           object
-    |> Repo.update()
+           |> Object.change(%{data: tombstone})
+           |> Repo.update() do
+      Hashtag.unlink(object)
+      {:ok, object}
+    end
   end
 
   def delete(%Object{data: %{"id" => id}} = object) do
@@ -349,4 +390,39 @@ defmodule Pleroma.Object do
 
   def self_replies(object, opts \\ []),
     do: replies(object, Keyword.put(opts, :self_only, true))
+
+  def tags(%Object{data: %{"tag" => tags}}) when is_list(tags), do: tags
+
+  def tags(_), do: []
+
+  def hashtags(%Object{} = object) do
+    # Note: always using embedded hashtags regardless whether they are migrated to hashtags table
+    #   (embedded hashtags stay in sync anyways, and we avoid extra joins and preload hassle)
+    embedded_hashtags(object)
+  end
+
+  def embedded_hashtags(%Object{data: data}) do
+    object_data_hashtags(data)
+  end
+
+  def embedded_hashtags(_), do: []
+
+  def object_data_hashtags(%{"tag" => tags}) when is_list(tags) do
+    tags
+    |> Enum.filter(fn
+      %{"type" => "Hashtag"} = data -> Map.has_key?(data, "name")
+      plain_text when is_bitstring(plain_text) -> true
+      _ -> false
+    end)
+    |> Enum.map(fn
+      %{"name" => "#" <> hashtag} -> String.downcase(hashtag)
+      %{"name" => hashtag} -> String.downcase(hashtag)
+      hashtag when is_bitstring(hashtag) -> String.downcase(hashtag)
+    end)
+    |> Enum.uniq()
+    # Note: "" elements (plain text) might occur in `data.tag` for incoming objects
+    |> Enum.filter(&(&1 not in [nil, ""]))
+  end
+
+  def object_data_hashtags(_), do: []
 end

lib/pleroma/pagination.ex

@@ -93,6 +93,7 @@ defmodule Pleroma.Pagination do
       max_id: :string,
       offset: :integer,
       limit: :integer,
+      skip_extra_order: :boolean,
       skip_order: :boolean
     }
 
@@ -114,6 +115,8 @@ defmodule Pleroma.Pagination do
 
   defp restrict(query, :order, %{skip_order: true}, _), do: query
 
+  defp restrict(%{order_bys: [_ | _]} = query, :order, %{skip_extra_order: true}, _), do: query
+
   defp restrict(query, :order, %{min_id: _}, table_binding) do
     order_by(
       query,

lib/pleroma/repo.ex

@@ -8,6 +8,8 @@ defmodule Pleroma.Repo do
     adapter: Ecto.Adapters.Postgres,
     migration_timestamps: [type: :naive_datetime_usec]
 
+  use Ecto.Explain
+
   import Ecto.Query
   require Logger
 
@@ -63,8 +65,8 @@ defmodule Pleroma.Repo do
   iex> Pleroma.Repo.chunk_stream(Pleroma.Activity.Queries.by_actor(ap_id), 500, :batches)
   """
   @spec chunk_stream(Ecto.Query.t(), integer(), atom()) :: Enumerable.t()
-  def chunk_stream(query, chunk_size, returns_as \\ :one) do
+  def chunk_stream(query, chunk_size, returns_as \\ :one, query_options \\ []) do
-    # We don't actually need start and end funcitons of resource streaming,
+    # We don't actually need start and end functions of resource streaming,
     # but it seems to be the only way to not fetch records one-by-one and
     # have individual records be the elements of the stream, instead of
     # lists of records
@@ -76,7 +78,7 @@ defmodule Pleroma.Repo do
           |> order_by(asc: :id)
           |> where([r], r.id > ^last_id)
           |> limit(^chunk_size)
-          |> all()
+          |> all(query_options)
           |> case do
             [] ->
               {:halt, last_id}

lib/pleroma/reverse_proxy.ex

@@ -4,7 +4,7 @@
 
 defmodule Pleroma.ReverseProxy do
   @range_headers ~w(range if-range)
-  @keep_req_headers ~w(accept user-agent accept-encoding cache-control if-modified-since) ++
+  @keep_req_headers ~w(accept accept-encoding cache-control if-modified-since) ++
                       ~w(if-unmodified-since if-none-match) ++ @range_headers
   @resp_cache_headers ~w(etag date last-modified)
   @keep_resp_headers @resp_cache_headers ++
@@ -57,9 +57,6 @@ defmodule Pleroma.ReverseProxy do
     * `false` will add `content-disposition: attachment` to any request,
     * a list of whitelisted content types
 
-    * `keep_user_agent` will forward the client's user-agent to the upstream. This may be useful if the upstream is
-    doing content transformation (encoding, …) depending on the request.
-
   * `req_headers`, `resp_headers` additional headers.
 
   * `http`: options for [hackney](https://github.com/benoitc/hackney) or [gun](https://github.com/ninenines/gun).
@@ -84,8 +81,7 @@ defmodule Pleroma.ReverseProxy do
   import Plug.Conn
 
   @type option() ::
-          {:keep_user_agent, boolean}
+          {:max_read_duration, :timer.time() | :infinity}
-          | {:max_read_duration, :timer.time() | :infinity}
           | {:max_body_length, non_neg_integer() | :infinity}
           | {:failed_request_ttl, :timer.time() | :infinity}
           | {:http, []}
@@ -291,17 +287,13 @@ defmodule Pleroma.ReverseProxy do
     end
   end
 
-  defp build_req_user_agent_header(headers, opts) do
+  defp build_req_user_agent_header(headers, _opts) do
-    if Keyword.get(opts, :keep_user_agent, false) do
+    List.keystore(
-      List.keystore(
+      headers,
-        headers,
+      "user-agent",
-        "user-agent",
+      0,
-        0,
+      {"user-agent", Pleroma.Application.user_agent()}
-        {"user-agent", Pleroma.Application.user_agent()}
+    )
-      )
-    else
-      headers
-    end
   end
 
   defp build_resp_headers(headers, opts) do

lib/pleroma/stats.ex

@@ -23,7 +23,11 @@ defmodule Pleroma.Stats do
 
   @impl true
   def init(_args) do
-    {:ok, nil, {:continue, :calculate_stats}}
+    if Pleroma.Config.get(:env) != :test do
+      {:ok, nil, {:continue, :calculate_stats}}
+    else
+      {:ok, calculate_stat_data()}
+    end
   end
 
   @doc "Performs update stats"

lib/pleroma/upload/filter/exiftool.ex

@@ -11,7 +11,8 @@ defmodule Pleroma.Upload.Filter.Exiftool do
 
   @spec filter(Pleroma.Upload.t()) :: {:ok, any()} | {:error, String.t()}
 
-  # webp is not compatible with exiftool at this time
+  # Formats not compatible with exiftool at this time
+  def filter(%Pleroma.Upload{content_type: "image/heic"}), do: {:ok, :noop}
   def filter(%Pleroma.Upload{content_type: "image/webp"}), do: {:ok, :noop}
 
   def filter(%Pleroma.Upload{tempfile: file, content_type: "image" <> _}) do
@@ -21,8 +22,8 @@ defmodule Pleroma.Upload.Filter.Exiftool do
         {error, 1} -> {:error, error}
       end
     rescue
-      _e in ErlangError ->
+      e in ErlangError ->
-        {:error, "exiftool command not found"}
+        {:error, "#{__MODULE__}: #{inspect(e)}"}
     end
   end
 

lib/pleroma/upload/filter/mogrifun.ex

@@ -44,8 +44,8 @@ defmodule Pleroma.Upload.Filter.Mogrifun do
       Filter.Mogrify.do_filter(file, [Enum.random(@filters)])
       {:ok, :filtered}
     rescue
-      _e in ErlangError ->
+      e in ErlangError ->
-        {:error, "mogrify command not found"}
+        {:error, "#{__MODULE__}: #{inspect(e)}"}
     end
   end
 

lib/pleroma/upload/filter/mogrify.ex

@@ -14,8 +14,8 @@ defmodule Pleroma.Upload.Filter.Mogrify do
       do_filter(file, Pleroma.Config.get!([__MODULE__, :args]))
       {:ok, :filtered}
     rescue
-      _e in ErlangError ->
+      e in ErlangError ->
-        {:error, "mogrify command not found"}
+        {:error, "#{__MODULE__}: #{inspect(e)}"}
     end
   end
 

lib/pleroma/user.ex

@@ -147,6 +147,7 @@ defmodule Pleroma.User do
     field(:shared_inbox, :string)
     field(:accepts_chat_messages, :boolean, default: nil)
     field(:last_active_at, :naive_datetime)
+    field(:disclose_client, :boolean, default: true)
 
     embeds_one(
       :notification_settings,
@@ -513,7 +514,8 @@ defmodule Pleroma.User do
         :pleroma_settings_store,
         :is_discoverable,
         :actor_type,
-        :accepts_chat_messages
+        :accepts_chat_messages,
+        :disclose_client
       ]
     )
     |> unique_constraint(:nickname)
@@ -2253,13 +2255,6 @@ defmodule Pleroma.User do
     |> update_and_set_cache()
   end
 
-  def roles(%{is_moderator: is_moderator, is_admin: is_admin}) do
-    %{
-      admin: is_admin,
-      moderator: is_moderator
-    }
-  end
-
   def validate_fields(changeset, remote? \\ false) do
     limit_name = if remote?, do: :max_remote_account_fields, else: :max_account_fields
     limit = Config.get([:instance, limit_name], 0)

lib/pleroma/web.ex

@@ -63,7 +63,8 @@ defmodule Pleroma.Web do
 
       # Executed just before actual controller action, invokes before-action hooks (callbacks)
       defp action(conn, params) do
-        with %{halted: false} = conn <- maybe_drop_authentication_if_oauth_check_ignored(conn),
+        with %{halted: false} = conn <-
+               maybe_drop_authentication_if_oauth_check_ignored(conn),
              %{halted: false} = conn <- maybe_perform_public_or_authenticated_check(conn),
              %{halted: false} = conn <- maybe_perform_authenticated_check(conn),
              %{halted: false} = conn <- maybe_halt_on_missing_oauth_scopes_check(conn) do
@@ -232,4 +233,16 @@ defmodule Pleroma.Web do
   def base_url do
     Pleroma.Web.Endpoint.url()
   end
+
+  # TODO: Change to Phoenix.Router.routes/1 for Phoenix 1.6.0+
+  def get_api_routes do
+    Pleroma.Web.Router.__routes__()
+    |> Enum.reject(fn r -> r.plug == Pleroma.Web.Fallback.RedirectController end)
+    |> Enum.map(fn r ->
+      r.path
+      |> String.split("/", trim: true)
+      |> List.first()
+    end)
+    |> Enum.uniq()
+  end
 end

lib/pleroma/web/activity_pub/activity_pub.ex

@@ -10,6 +10,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
   alias Pleroma.Conversation
   alias Pleroma.Conversation.Participation
   alias Pleroma.Filter
+  alias Pleroma.Hashtag
   alias Pleroma.Maps
   alias Pleroma.Notification
   alias Pleroma.Object
@@ -465,6 +466,23 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
     |> Repo.one()
   end
 
+  defp fetch_paginated_optimized(query, opts, pagination) do
+    # Note: tag-filtering funcs may apply "ORDER BY objects.id DESC",
+    #   and extra sorting on "activities.id DESC NULLS LAST" would worse the query plan
+    opts = Map.put(opts, :skip_extra_order, true)
+
+    Pagination.fetch_paginated(query, opts, pagination)
+  end
+
+  def fetch_activities(recipients, opts \\ %{}, pagination \\ :keyset) do
+    list_memberships = Pleroma.List.memberships(opts[:user])
+
+    fetch_activities_query(recipients ++ list_memberships, opts)
+    |> fetch_paginated_optimized(opts, pagination)
+    |> Enum.reverse()
+    |> maybe_update_cc(list_memberships, opts[:user])
+  end
+
   @spec fetch_public_or_unlisted_activities(map(), Pagination.type()) :: [Activity.t()]
   def fetch_public_or_unlisted_activities(opts \\ %{}, pagination \\ :keyset) do
     opts = Map.delete(opts, :user)
@@ -472,7 +490,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
     [Constants.as_public()]
     |> fetch_activities_query(opts)
     |> restrict_unlisted(opts)
-    |> Pagination.fetch_paginated(opts, pagination)
+    |> fetch_paginated_optimized(opts, pagination)
   end
 
   @spec fetch_public_activities(map(), Pagination.type()) :: [Activity.t()]
@@ -693,52 +711,144 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
 
   defp restrict_since(query, _), do: query
 
-  defp restrict_tag_reject(_query, %{tag_reject: _tag_reject, skip_preload: true}) do
+  defp restrict_embedded_tag_all(_query, %{tag_all: _tag_all, skip_preload: true}) do
-    raise "Can't use the child object without preloading!"
+    raise_on_missing_preload()
   end
 
-  defp restrict_tag_reject(query, %{tag_reject: [_ | _] = tag_reject}) do
+  defp restrict_embedded_tag_all(query, %{tag_all: [_ | _] = tag_all}) do
-    from(
-      [_activity, object] in query,
-      where: fragment("not (?)->'tag' \\?| (?)", object.data, ^tag_reject)
-    )
-  end
-
-  defp restrict_tag_reject(query, _), do: query
-
-  defp restrict_tag_all(_query, %{tag_all: _tag_all, skip_preload: true}) do
-    raise "Can't use the child object without preloading!"
-  end
-
-  defp restrict_tag_all(query, %{tag_all: [_ | _] = tag_all}) do
     from(
       [_activity, object] in query,
       where: fragment("(?)->'tag' \\?& (?)", object.data, ^tag_all)
     )
   end
 
-  defp restrict_tag_all(query, _), do: query
+  defp restrict_embedded_tag_all(query, %{tag_all: tag}) when is_binary(tag) do
+    restrict_embedded_tag_any(query, %{tag: tag})
+  end
 
-  defp restrict_tag(_query, %{tag: _tag, skip_preload: true}) do
+  defp restrict_embedded_tag_all(query, _), do: query
+
+  defp restrict_embedded_tag_any(_query, %{tag: _tag, skip_preload: true}) do
+    raise_on_missing_preload()
+  end
+
+  defp restrict_embedded_tag_any(query, %{tag: [_ | _] = tag_any}) do
+    from(
+      [_activity, object] in query,
+      where: fragment("(?)->'tag' \\?| (?)", object.data, ^tag_any)
+    )
+  end
+
+  defp restrict_embedded_tag_any(query, %{tag: tag}) when is_binary(tag) do
+    restrict_embedded_tag_any(query, %{tag: [tag]})
+  end
+
+  defp restrict_embedded_tag_any(query, _), do: query
+
+  defp restrict_embedded_tag_reject_any(_query, %{tag_reject: _tag_reject, skip_preload: true}) do
+    raise_on_missing_preload()
+  end
+
+  defp restrict_embedded_tag_reject_any(query, %{tag_reject: [_ | _] = tag_reject}) do
+    from(
+      [_activity, object] in query,
+      where: fragment("not (?)->'tag' \\?| (?)", object.data, ^tag_reject)
+    )
+  end
+
+  defp restrict_embedded_tag_reject_any(query, %{tag_reject: tag_reject})
+       when is_binary(tag_reject) do
+    restrict_embedded_tag_reject_any(query, %{tag_reject: [tag_reject]})
+  end
+
+  defp restrict_embedded_tag_reject_any(query, _), do: query
+
+  defp object_ids_query_for_tags(tags) do
+    from(hto in "hashtags_objects")
+    |> join(:inner, [hto], ht in Pleroma.Hashtag, on: hto.hashtag_id == ht.id)
+    |> where([hto, ht], ht.name in ^tags)
+    |> select([hto], hto.object_id)
+    |> distinct([hto], true)
+  end
+
+  defp restrict_hashtag_all(_query, %{tag_all: _tag, skip_preload: true}) do
+    raise_on_missing_preload()
+  end
+
+  defp restrict_hashtag_all(query, %{tag_all: [single_tag]}) do
+    restrict_hashtag_any(query, %{tag: single_tag})
+  end
+
+  defp restrict_hashtag_all(query, %{tag_all: [_ | _] = tags}) do
+    from(
+      [_activity, object] in query,
+      where:
+        fragment(
+          """
+          (SELECT array_agg(hashtags.name) FROM hashtags JOIN hashtags_objects
+            ON hashtags_objects.hashtag_id = hashtags.id WHERE hashtags.name = ANY(?)
+              AND hashtags_objects.object_id = ?) @> ?
+          """,
+          ^tags,
+          object.id,
+          ^tags
+        )
+    )
+  end
+
+  defp restrict_hashtag_all(query, %{tag_all: tag}) when is_binary(tag) do
+    restrict_hashtag_all(query, %{tag_all: [tag]})
+  end
+
+  defp restrict_hashtag_all(query, _), do: query
+
+  defp restrict_hashtag_any(_query, %{tag: _tag, skip_preload: true}) do
+    raise_on_missing_preload()
+  end
+
+  defp restrict_hashtag_any(query, %{tag: [_ | _] = tags}) do
+    hashtag_ids =
+      from(ht in Hashtag, where: ht.name in ^tags, select: ht.id)
+      |> Repo.all()
+
+    # Note: NO extra ordering should be done on "activities.id desc nulls last" for optimal plan
+    from(
+      [_activity, object] in query,
+      join: hto in "hashtags_objects",
+      on: hto.object_id == object.id,
+      where: hto.hashtag_id in ^hashtag_ids,
+      distinct: [desc: object.id],
+      order_by: [desc: object.id]
+    )
+  end
+
+  defp restrict_hashtag_any(query, %{tag: tag}) when is_binary(tag) do
+    restrict_hashtag_any(query, %{tag: [tag]})
+  end
+
+  defp restrict_hashtag_any(query, _), do: query
+
+  defp restrict_hashtag_reject_any(_query, %{tag_reject: _tag_reject, skip_preload: true}) do
+    raise_on_missing_preload()
+  end
+
+  defp restrict_hashtag_reject_any(query, %{tag_reject: [_ | _] = tags_reject}) do
+    from(
+      [_activity, object] in query,
+      where: object.id not in subquery(object_ids_query_for_tags(tags_reject))
+    )
+  end
+
+  defp restrict_hashtag_reject_any(query, %{tag_reject: tag_reject}) when is_binary(tag_reject) do
+    restrict_hashtag_reject_any(query, %{tag_reject: [tag_reject]})
+  end
+
+  defp restrict_hashtag_reject_any(query, _), do: query
+
+  defp raise_on_missing_preload do
     raise "Can't use the child object without preloading!"
   end
 
-  defp restrict_tag(query, %{tag: tag}) when is_list(tag) do
-    from(
-      [_activity, object] in query,
-      where: fragment("(?)->'tag' \\?| (?)", object.data, ^tag)
-    )
-  end
-
-  defp restrict_tag(query, %{tag: tag}) when is_binary(tag) do
-    from(
-      [_activity, object] in query,
-      where: fragment("(?)->'tag' \\? (?)", object.data, ^tag)
-    )
-  end
-
-  defp restrict_tag(query, _), do: query
-
   defp restrict_recipients(query, [], _user), do: query
 
   defp restrict_recipients(query, recipients, nil) do
@@ -1098,6 +1208,26 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
 
   defp maybe_order(query, _), do: query
 
+  defp normalize_fetch_activities_query_opts(opts) do
+    Enum.reduce([:tag, :tag_all, :tag_reject], opts, fn key, opts ->
+      case opts[key] do
+        value when is_bitstring(value) ->
+          Map.put(opts, key, Hashtag.normalize_name(value))
+
+        value when is_list(value) ->
+          normalized_value =
+            value
+            |> Enum.map(&Hashtag.normalize_name/1)
+            |> Enum.uniq()
+
+          Map.put(opts, key, normalized_value)
+
+        _ ->
+          opts
+      end
+    end)
+  end
+
   defp fetch_activities_query_ap_ids_ops(opts) do
     source_user = opts[:muting_user]
     ap_id_relationships = if source_user, do: [:mute, :reblog_mute], else: []
@@ -1121,6 +1251,8 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
   end
 
   def fetch_activities_query(recipients, opts \\ %{}) do
+    opts = normalize_fetch_activities_query_opts(opts)
+
     {restrict_blocked_opts, restrict_muted_opts, restrict_muted_reblogs_opts} =
       fetch_activities_query_ap_ids_ops(opts)
 
@@ -1128,50 +1260,51 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
       skip_thread_containment: Config.get([:instance, :skip_thread_containment])
     }
 
-    Activity
+    query =
-    |> maybe_preload_objects(opts)
+      Activity
-    |> maybe_preload_bookmarks(opts)
+      |> maybe_preload_objects(opts)
-    |> maybe_preload_report_notes(opts)
+      |> maybe_preload_bookmarks(opts)
-    |> maybe_set_thread_muted_field(opts)
+      |> maybe_preload_report_notes(opts)
-    |> maybe_order(opts)
+      |> maybe_set_thread_muted_field(opts)
-    |> restrict_recipients(recipients, opts[:user])
+      |> maybe_order(opts)
-    |> restrict_replies(opts)
+      |> restrict_recipients(recipients, opts[:user])
-    |> restrict_tag(opts)
+      |> restrict_replies(opts)
-    |> restrict_tag_reject(opts)
+      |> restrict_since(opts)
-    |> restrict_tag_all(opts)
+      |> restrict_local(opts)
-    |> restrict_since(opts)
+      |> restrict_remote(opts)
-    |> restrict_local(opts)
+      |> restrict_actor(opts)
-    |> restrict_remote(opts)
+      |> restrict_type(opts)
-    |> restrict_actor(opts)
+      |> restrict_state(opts)
-    |> restrict_type(opts)
+      |> restrict_favorited_by(opts)
-    |> restrict_state(opts)
+      |> restrict_blocked(restrict_blocked_opts)
-    |> restrict_favorited_by(opts)
+      |> restrict_muted(restrict_muted_opts)
-    |> restrict_blocked(restrict_blocked_opts)
+      |> restrict_filtered(opts)
-    |> restrict_muted(restrict_muted_opts)
+      |> restrict_media(opts)
-    |> restrict_filtered(opts)
+      |> restrict_visibility(opts)
-    |> restrict_media(opts)
+      |> restrict_thread_visibility(opts, config)
-    |> restrict_visibility(opts)
+      |> restrict_reblogs(opts)
-    |> restrict_thread_visibility(opts, config)
+      |> restrict_pinned(opts)
-    |> restrict_reblogs(opts)
+      |> restrict_muted_reblogs(restrict_muted_reblogs_opts)
-    |> restrict_pinned(opts)
+      |> restrict_instance(opts)
-    |> restrict_muted_reblogs(restrict_muted_reblogs_opts)
+      |> restrict_announce_object_actor(opts)
-    |> restrict_instance(opts)
+      |> restrict_filtered(opts)
-    |> restrict_announce_object_actor(opts)
+      |> Activity.restrict_deactivated_users()
-    |> restrict_filtered(opts)
+      |> exclude_poll_votes(opts)
-    |> Activity.restrict_deactivated_users()
+      |> exclude_chat_messages(opts)
-    |> exclude_poll_votes(opts)
+      |> exclude_invisible_actors(opts)
-    |> exclude_chat_messages(opts)
+      |> exclude_visibility(opts)
-    |> exclude_invisible_actors(opts)
-    |> exclude_visibility(opts)
-  end
 
-  def fetch_activities(recipients, opts \\ %{}, pagination \\ :keyset) do
+    if Config.feature_enabled?(:improved_hashtag_timeline) do
-    list_memberships = Pleroma.List.memberships(opts[:user])
+      query
-
+      |> restrict_hashtag_any(opts)
-    fetch_activities_query(recipients ++ list_memberships, opts)
+      |> restrict_hashtag_all(opts)
-    |> Pagination.fetch_paginated(opts, pagination)
+      |> restrict_hashtag_reject_any(opts)
-    |> Enum.reverse()
+    else
-    |> maybe_update_cc(list_memberships, opts[:user])
+      query
+      |> restrict_embedded_tag_any(opts)
+      |> restrict_embedded_tag_all(opts)
+      |> restrict_embedded_tag_reject_any(opts)
+    end
   end
 
   @doc """
@@ -1250,21 +1383,17 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
 
   defp get_actor_url(_url), do: nil
 
+  defp normalize_image(%{"url" => url}) do
+    %{
+      "type" => "Image",
+      "url" => [%{"href" => url}]
+    }
+  end
+
+  defp normalize_image(urls) when is_list(urls), do: urls |> List.first() |> normalize_image()
+  defp normalize_image(_), do: nil
+
   defp object_to_user_data(data) do
-    avatar =
-      data["icon"]["url"] &&
-        %{
-          "type" => "Image",
-          "url" => [%{"href" => data["icon"]["url"]}]
-        }
-
-    banner =
-      data["image"]["url"] &&
-        %{
-          "type" => "Image",
-          "url" => [%{"href" => data["image"]["url"]}]
-        }
-
     fields =
       data
       |> Map.get("attachment", [])
@@ -1308,13 +1437,13 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
       ap_id: data["id"],
       uri: get_actor_url(data["url"]),
       ap_enabled: true,
-      banner: banner,
+      banner: normalize_image(data["image"]),
       fields: fields,
       emoji: emojis,
       is_locked: is_locked,
       is_discoverable: is_discoverable,
       invisible: invisible,
-      avatar: avatar,
+      avatar: normalize_image(data["icon"]),
       name: data["name"],
       follower_address: data["followers"],
       following_address: data["following"],

lib/pleroma/web/activity_pub/mrf.ex

@@ -92,7 +92,9 @@ defmodule Pleroma.Web.ActivityPub.MRF do
   end
 
   def get_policies do
-    Pleroma.Config.get([:mrf, :policies], []) |> get_policies()
+    Pleroma.Config.get([:mrf, :policies], [])
+    |> get_policies()
+    |> Enum.concat([Pleroma.Web.ActivityPub.MRF.HashtagPolicy])
   end
 
   defp get_policies(policy) when is_atom(policy), do: [policy]

lib/pleroma/web/activity_pub/mrf/follow_bot_policy.ex

@@ -0,0 +1,59 @@
+defmodule Pleroma.Web.ActivityPub.MRF.FollowBotPolicy do
+  @behaviour Pleroma.Web.ActivityPub.MRF
+  alias Pleroma.Config
+  alias Pleroma.User
+  alias Pleroma.Web.CommonAPI
+
+  require Logger
+
+  @impl true
+  def filter(message) do
+    with follower_nickname <- Config.get([:mrf_follow_bot, :follower_nickname]),
+         %User{actor_type: "Service"} = follower <-
+           User.get_cached_by_nickname(follower_nickname),
+         %{"type" => "Create", "object" => %{"type" => "Note"}} <- message do
+      try_follow(follower, message)
+    else
+      nil ->
+        Logger.warn(
+          "#{__MODULE__} skipped because of missing `:mrf_follow_bot, :follower_nickname` configuration, the :follower_nickname
+            account does not exist, or the account is not correctly configured as a bot."
+        )
+
+        {:ok, message}
+
+      _ ->
+        {:ok, message}
+    end
+  end
+
+  defp try_follow(follower, message) do
+    to = Map.get(message, "to", [])
+    cc = Map.get(message, "cc", [])
+    actor = [message["actor"]]
+
+    Enum.concat([to, cc, actor])
+    |> List.flatten()
+    |> Enum.uniq()
+    |> User.get_all_by_ap_id()
+    |> Enum.each(fn user ->
+      with false <- user.local,
+           false <- User.following?(follower, user),
+           false <- User.locked?(user),
+           false <- (user.bio || "") |> String.downcase() |> String.contains?("nobot") do
+        Logger.debug(
+          "#{__MODULE__}: Follow request from #{follower.nickname} to #{user.nickname}"
+        )
+
+        CommonAPI.follow(follower, user)
+      end
+    end)
+
+    {:ok, message}
+  end
+
+  @impl true
+  def describe do
+    {:ok, %{}}
+  end
+end

lib/pleroma/web/activity_pub/mrf/hashtag_policy.ex

@@ -0,0 +1,116 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2021 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.ActivityPub.MRF.HashtagPolicy do
+  require Pleroma.Constants
+
+  alias Pleroma.Config
+  alias Pleroma.Object
+
+  @moduledoc """
+  Reject, TWKN-remove or Set-Sensitive messsages with specific hashtags (without the leading #)
+
+  Note: This MRF Policy is always enabled, if you want to disable it you have to set empty lists.
+  """
+
+  @behaviour Pleroma.Web.ActivityPub.MRF
+
+  defp check_reject(message, hashtags) do
+    if Enum.any?(Config.get([:mrf_hashtag, :reject]), fn match -> match in hashtags end) do
+      {:reject, "[HashtagPolicy] Matches with rejected keyword"}
+    else
+      {:ok, message}
+    end
+  end
+
+  defp check_ftl_removal(%{"to" => to} = message, hashtags) do
+    if Pleroma.Constants.as_public() in to and
+         Enum.any?(Config.get([:mrf_hashtag, :federated_timeline_removal]), fn match ->
+           match in hashtags
+         end) do
+      to = List.delete(to, Pleroma.Constants.as_public())
+      cc = [Pleroma.Constants.as_public() | message["cc"] || []]
+
+      message =
+        message
+        |> Map.put("to", to)
+        |> Map.put("cc", cc)
+        |> Kernel.put_in(["object", "to"], to)
+        |> Kernel.put_in(["object", "cc"], cc)
+
+      {:ok, message}
+    else
+      {:ok, message}
+    end
+  end
+
+  defp check_ftl_removal(message, _hashtags), do: {:ok, message}
+
+  defp check_sensitive(message, hashtags) do
+    if Enum.any?(Config.get([:mrf_hashtag, :sensitive]), fn match -> match in hashtags end) do
+      {:ok, Kernel.put_in(message, ["object", "sensitive"], true)}
+    else
+      {:ok, message}
+    end
+  end
+
+  @impl true
+  def filter(%{"type" => "Create", "object" => object} = message) do
+    hashtags = Object.hashtags(%Object{data: object})
+
+    if hashtags != [] do
+      with {:ok, message} <- check_reject(message, hashtags),
+           {:ok, message} <- check_ftl_removal(message, hashtags),
+           {:ok, message} <- check_sensitive(message, hashtags) do
+        {:ok, message}
+      end
+    else
+      {:ok, message}
+    end
+  end
+
+  @impl true
+  def filter(message), do: {:ok, message}
+
+  @impl true
+  def describe do
+    mrf_hashtag =
+      Config.get(:mrf_hashtag)
+      |> Enum.into(%{})
+
+    {:ok, %{mrf_hashtag: mrf_hashtag}}
+  end
+
+  @impl true
+  def config_description do
+    %{
+      key: :mrf_hashtag,
+      related_policy: "Pleroma.Web.ActivityPub.MRF.HashtagPolicy",
+      label: "MRF Hashtag",
+      description: @moduledoc,
+      children: [
+        %{
+          key: :reject,
+          type: {:list, :string},
+          description: "A list of hashtags which result in message being rejected.",
+          suggestions: ["foo"]
+        },
+        %{
+          key: :federated_timeline_removal,
+          type: {:list, :string},
+          description:
+            "A list of hashtags which result in message being removed from federated timelines (a.k.a unlisted).",
+          suggestions: ["foo"]
+        },
+        %{
+          key: :sensitive,
+          type: {:list, :string},
+          description:
+            "A list of hashtags which result in message being set as sensitive (a.k.a NSFW/R-18)",
+          suggestions: ["nsfw", "r18"]
+        }
+      ]
+    }
+  end
+end

lib/pleroma/web/activity_pub/mrf/simple_policy.ex

@@ -64,20 +64,16 @@ defmodule Pleroma.Web.ActivityPub.MRF.SimplePolicy do
          %{host: actor_host} = _actor_info,
          %{
            "type" => "Create",
-           "object" => child_object
+           "object" => %{} = _child_object
          } = object
-       )
+       ) do
-       when is_map(child_object) do
     media_nsfw =
       Config.get([:mrf_simple, :media_nsfw])
       |> MRF.subdomains_regex()
 
     object =
       if MRF.subdomain_match?(media_nsfw, actor_host) do
-        tags = (child_object["tag"] || []) ++ ["nsfw"]
+        Kernel.put_in(object, ["object", "sensitive"], true)
-        child_object = Map.put(child_object, "tag", tags)
-        child_object = Map.put(child_object, "sensitive", true)
-        Map.put(object, "object", child_object)
       else
         object
       end

lib/pleroma/web/activity_pub/mrf/tag_policy.ex

@@ -28,20 +28,11 @@ defmodule Pleroma.Web.ActivityPub.MRF.TagPolicy do
          "mrf_tag:media-force-nsfw",
          %{
            "type" => "Create",
-           "object" => %{"attachment" => child_attachment} = object
+           "object" => %{"attachment" => child_attachment}
          } = message
        )
        when length(child_attachment) > 0 do
-    tags = (object["tag"] || []) ++ ["nsfw"]
+    {:ok, Kernel.put_in(message, ["object", "sensitive"], true)}
-
-    object =
-      object
-      |> Map.put("tag", tags)
-      |> Map.put("sensitive", true)
-
-    message = Map.put(message, "object", object)
-
-    {:ok, message}
   end
 
   defp process_tag(

lib/pleroma/web/activity_pub/object_validator.ex

@@ -37,37 +37,6 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidator do
   @impl true
   def validate(object, meta)
 
-  def validate(%{"type" => type} = object, meta)
-      when type in ~w[Accept Reject] do
-    with {:ok, object} <-
-           object
-           |> AcceptRejectValidator.cast_and_validate()
-           |> Ecto.Changeset.apply_action(:insert) do
-      object = stringify_keys(object)
-      {:ok, object, meta}
-    end
-  end
-
-  def validate(%{"type" => "Event"} = object, meta) do
-    with {:ok, object} <-
-           object
-           |> EventValidator.cast_and_validate()
-           |> Ecto.Changeset.apply_action(:insert) do
-      object = stringify_keys(object)
-      {:ok, object, meta}
-    end
-  end
-
-  def validate(%{"type" => "Follow"} = object, meta) do
-    with {:ok, object} <-
-           object
-           |> FollowValidator.cast_and_validate()
-           |> Ecto.Changeset.apply_action(:insert) do
-      object = stringify_keys(object)
-      {:ok, object, meta}
-    end
-  end
-
   def validate(%{"type" => "Block"} = block_activity, meta) do
     with {:ok, block_activity} <-
            block_activity
@@ -87,16 +56,6 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidator do
     end
   end
 
-  def validate(%{"type" => "Update"} = update_activity, meta) do
-    with {:ok, update_activity} <-
-           update_activity
-           |> UpdateValidator.cast_and_validate()
-           |> Ecto.Changeset.apply_action(:insert) do
-      update_activity = stringify_keys(update_activity)
-      {:ok, update_activity, meta}
-    end
-  end
-
   def validate(%{"type" => "Undo"} = object, meta) do
     with {:ok, object} <-
            object
@@ -123,76 +82,6 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidator do
     end
   end
 
-  def validate(%{"type" => "Like"} = object, meta) do
-    with {:ok, object} <-
-           object
-           |> LikeValidator.cast_and_validate()
-           |> Ecto.Changeset.apply_action(:insert) do
-      object = stringify_keys(object)
-      {:ok, object, meta}
-    end
-  end
-
-  def validate(%{"type" => "ChatMessage"} = object, meta) do
-    with {:ok, object} <-
-           object
-           |> ChatMessageValidator.cast_and_validate()
-           |> Ecto.Changeset.apply_action(:insert) do
-      object = stringify_keys(object)
-      {:ok, object, meta}
-    end
-  end
-
-  def validate(%{"type" => "Question"} = object, meta) do
-    with {:ok, object} <-
-           object
-           |> QuestionValidator.cast_and_validate()
-           |> Ecto.Changeset.apply_action(:insert) do
-      object = stringify_keys(object)
-      {:ok, object, meta}
-    end
-  end
-
-  def validate(%{"type" => type} = object, meta) when type in ~w[Audio Video] do
-    with {:ok, object} <-
-           object
-           |> AudioVideoValidator.cast_and_validate()
-           |> Ecto.Changeset.apply_action(:insert) do
-      object = stringify_keys(object)
-      {:ok, object, meta}
-    end
-  end
-
-  def validate(%{"type" => "Article"} = object, meta) do
-    with {:ok, object} <-
-           object
-           |> ArticleNoteValidator.cast_and_validate()
-           |> Ecto.Changeset.apply_action(:insert) do
-      object = stringify_keys(object)
-      {:ok, object, meta}
-    end
-  end
-
-  def validate(%{"type" => "Answer"} = object, meta) do
-    with {:ok, object} <-
-           object
-           |> AnswerValidator.cast_and_validate()
-           |> Ecto.Changeset.apply_action(:insert) do
-      object = stringify_keys(object)
-      {:ok, object, meta}
-    end
-  end
-
-  def validate(%{"type" => "EmojiReact"} = object, meta) do
-    with {:ok, object} <-
-           object
-           |> EmojiReactValidator.cast_and_validate()
-           |> Ecto.Changeset.apply_action(:insert) do
-      object = stringify_keys(object)
-      {:ok, object, meta}
-    end
-  end
-
   def validate(
         %{"type" => "Create", "object" => %{"type" => "ChatMessage"} = object} = create_activity,
         meta
@@ -224,10 +113,30 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidator do
     end
   end
 
-  def validate(%{"type" => "Announce"} = object, meta) do
+  def validate(%{"type" => type} = object, meta)
+      when type in ~w[Accept Reject Follow Update Like EmojiReact Announce
+      Event ChatMessage Question Audio Video Article Answer] do
+    validator =
+      case type do
+        "Accept" -> AcceptRejectValidator
+        "Reject" -> AcceptRejectValidator
+        "Follow" -> FollowValidator
+        "Update" -> UpdateValidator
+        "Like" -> LikeValidator
+        "EmojiReact" -> EmojiReactValidator
+        "Announce" -> AnnounceValidator
+        "Event" -> EventValidator
+        "ChatMessage" -> ChatMessageValidator
+        "Question" -> QuestionValidator
+        "Audio" -> AudioVideoValidator
+        "Video" -> AudioVideoValidator
+        "Article" -> ArticleNoteValidator
+        "Answer" -> AnswerValidator
+      end
+
     with {:ok, object} <-
            object
-           |> AnnounceValidator.cast_and_validate()
+           |> validator.cast_and_validate()
            |> Ecto.Changeset.apply_action(:insert) do
       object = stringify_keys(object)
       {:ok, object, meta}

lib/pleroma/web/activity_pub/object_validators/audio_video_validator.ex

@@ -70,19 +70,33 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.AudioVideoValidator do
     |> changeset(data)
   end
 
-  defp fix_url(%{"url" => url} = data) when is_list(url) do
+  defp find_attachment(url) do
-    attachment =
+    mpeg_url =
-      Enum.find(url, fn x ->
+      Enum.find(url, fn
-        mime_type = x["mimeType"] || x["mediaType"] || ""
+        %{"mediaType" => mime_type, "tag" => tags} when is_list(tags) ->
+          mime_type == "application/x-mpegURL"
 
-        is_map(x) and String.starts_with?(mime_type, ["video/", "audio/"])
+        _ ->
+          false
       end)
 
-    link_element =
+    url
-      Enum.find(url, fn x ->
+    |> Enum.concat(mpeg_url["tag"] || [])
-        mime_type = x["mimeType"] || x["mediaType"] || ""
+    |> Enum.find(fn
+      %{"mediaType" => mime_type} -> String.starts_with?(mime_type, ["video/", "audio/"])
+      %{"mimeType" => mime_type} -> String.starts_with?(mime_type, ["video/", "audio/"])
+      _ -> false
+    end)
+  end
 
-        is_map(x) and mime_type == "text/html"
+  defp fix_url(%{"url" => url} = data) when is_list(url) do
+    attachment = find_attachment(url)
+
+    link_element =
+      Enum.find(url, fn
+        %{"mediaType" => "text/html"} -> true
+        %{"mimeType" => "text/html"} -> true
+        _ -> false
       end)
 
     data

lib/pleroma/web/activity_pub/transmogrifier.ex

@@ -32,18 +32,17 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
   """
   def fix_object(object, options \\ []) do
     object
-    |> strip_internal_fields
+    |> strip_internal_fields()
-    |> fix_actor
+    |> fix_actor()
-    |> fix_url
+    |> fix_url()
-    |> fix_attachments
+    |> fix_attachments()
-    |> fix_context
+    |> fix_context()
     |> fix_in_reply_to(options)
-    |> fix_emoji
+    |> fix_emoji()
-    |> fix_tag
+    |> fix_tag()
-    |> set_sensitive
+    |> fix_content_map()
-    |> fix_content_map
+    |> fix_addressing()
-    |> fix_addressing
+    |> fix_summary()
-    |> fix_summary
     |> fix_type(options)
   end
 
@@ -315,10 +314,9 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     tags =
       tag
       |> Enum.filter(fn data -> data["type"] == "Hashtag" and data["name"] end)
-      |> Enum.map(fn %{"name" => name} ->
+      |> Enum.map(fn
-        name
+        %{"name" => "#" <> hashtag} -> String.downcase(hashtag)
-        |> String.slice(1..-1)
+        %{"name" => hashtag} -> String.downcase(hashtag)
-        |> String.downcase()
       end)
 
     Map.put(object, "tag", tag ++ tags)
@@ -742,7 +740,6 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
   # Prepares the object of an outgoing create activity.
   def prepare_object(object) do
     object
-    |> set_sensitive
     |> add_hashtags
     |> add_mention_tags
     |> add_emoji_tags
@@ -933,15 +930,6 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     Map.put(object, "conversation", object["context"])
   end
 
-  def set_sensitive(%{"sensitive" => _} = object) do
-    object
-  end
-
-  def set_sensitive(object) do
-    tags = object["tag"] || []
-    Map.put(object, "sensitive", "nsfw" in tags)
-  end
-
   def set_type(%{"type" => "Answer"} = object) do
     Map.put(object, "type", "Note")
   end

lib/pleroma/web/admin_api/controllers/user_controller.ex

@@ -13,16 +13,17 @@ defmodule Pleroma.Web.AdminAPI.UserController do
   alias Pleroma.Web.ActivityPub.Builder
   alias Pleroma.Web.ActivityPub.Pipeline
   alias Pleroma.Web.AdminAPI
-  alias Pleroma.Web.AdminAPI.AccountView
   alias Pleroma.Web.AdminAPI.Search
   alias Pleroma.Web.Plugs.OAuthScopesPlug
 
   @users_page_size 50
 
+  plug(Pleroma.Web.ApiSpec.CastAndValidate)
+
   plug(
     OAuthScopesPlug,
     %{scopes: ["admin:read:accounts"]}
-    when action in [:list, :show]
+    when action in [:index, :show]
   )
 
   plug(
@@ -44,13 +45,19 @@ defmodule Pleroma.Web.AdminAPI.UserController do
     when action in [:follow, :unfollow]
   )
 
+  plug(:put_view, Pleroma.Web.AdminAPI.AccountView)
+
   action_fallback(AdminAPI.FallbackController)
 
-  def delete(conn, %{"nickname" => nickname}) do
+  defdelegate open_api_operation(action), to: Pleroma.Web.ApiSpec.Admin.UserOperation
-    delete(conn, %{"nicknames" => [nickname]})
+
+  def delete(conn, %{nickname: nickname}) do
+    conn
+    |> Map.put(:body_params, %{nicknames: [nickname]})
+    |> delete(%{})
   end
 
-  def delete(%{assigns: %{user: admin}} = conn, %{"nicknames" => nicknames}) do
+  def delete(%{assigns: %{user: admin}, body_params: %{nicknames: nicknames}} = conn, _) do
     users = Enum.map(nicknames, &User.get_cached_by_nickname/1)
 
     Enum.each(users, fn user ->
@@ -67,10 +74,16 @@ defmodule Pleroma.Web.AdminAPI.UserController do
     json(conn, nicknames)
   end
 
-  def follow(%{assigns: %{user: admin}} = conn, %{
+  def follow(
-        "follower" => follower_nick,
+        %{
-        "followed" => followed_nick
+          assigns: %{user: admin},
-      }) do
+          body_params: %{
+            follower: follower_nick,
+            followed: followed_nick
+          }
+        } = conn,
+        _
+      ) do
     with %User{} = follower <- User.get_cached_by_nickname(follower_nick),
          %User{} = followed <- User.get_cached_by_nickname(followed_nick) do
       User.follow(follower, followed)
@@ -86,10 +99,16 @@ defmodule Pleroma.Web.AdminAPI.UserController do
     json(conn, "ok")
   end
 
-  def unfollow(%{assigns: %{user: admin}} = conn, %{
+  def unfollow(
-        "follower" => follower_nick,
+        %{
-        "followed" => followed_nick
+          assigns: %{user: admin},
-      }) do
+          body_params: %{
+            follower: follower_nick,
+            followed: followed_nick
+          }
+        } = conn,
+        _
+      ) do
     with %User{} = follower <- User.get_cached_by_nickname(follower_nick),
          %User{} = followed <- User.get_cached_by_nickname(followed_nick) do
       User.unfollow(follower, followed)
@@ -105,9 +124,10 @@ defmodule Pleroma.Web.AdminAPI.UserController do
     json(conn, "ok")
   end
 
-  def create(%{assigns: %{user: admin}} = conn, %{"users" => users}) do
+  def create(%{assigns: %{user: admin}, body_params: %{users: users}} = conn, _) do
     changesets =
-      Enum.map(users, fn %{"nickname" => nickname, "email" => email, "password" => password} ->
+      users
+      |> Enum.map(fn %{nickname: nickname, email: email, password: password} ->
         user_data = %{
           nickname: nickname,
           name: nickname,
@@ -124,52 +144,49 @@ defmodule Pleroma.Web.AdminAPI.UserController do
       end)
 
     case Pleroma.Repo.transaction(changesets) do
-      {:ok, users} ->
+      {:ok, users_map} ->
-        res =
+        users =
-          users
+          users_map
           |> Map.values()
           |> Enum.map(fn user ->
             {:ok, user} = User.post_register_action(user)
 
             user
           end)
-          |> Enum.map(&AccountView.render("created.json", %{user: &1}))
 
         ModerationLog.insert_log(%{
           actor: admin,
-          subjects: Map.values(users),
+          subjects: users,
           action: "create"
         })
 
-        json(conn, res)
+        render(conn, "created_many.json", users: users)
 
       {:error, id, changeset, _} ->
-        res =
+        changesets =
           Enum.map(changesets.operations, fn
-            {current_id, {:changeset, _current_changeset, _}} when current_id == id ->
+            {^id, {:changeset, _current_changeset, _}} ->
-              AccountView.render("create-error.json", %{changeset: changeset})
+              changeset
 
             {_, {:changeset, current_changeset, _}} ->
-              AccountView.render("create-error.json", %{changeset: current_changeset})
+              current_changeset
           end)
 
         conn
         |> put_status(:conflict)
-        |> json(res)
+        |> render("create_errors.json", changesets: changesets)
     end
   end
 
-  def show(%{assigns: %{user: admin}} = conn, %{"nickname" => nickname}) do
+  def show(%{assigns: %{user: admin}} = conn, %{nickname: nickname}) do
     with %User{} = user <- User.get_cached_by_nickname_or_id(nickname, for: admin) do
-      conn
+      render(conn, "show.json", %{user: user})
-      |> put_view(AccountView)
-      |> render("show.json", %{user: user})
     else
       _ -> {:error, :not_found}
     end
   end
 
-  def toggle_activation(%{assigns: %{user: admin}} = conn, %{"nickname" => nickname}) do
+  def toggle_activation(%{assigns: %{user: admin}} = conn, %{nickname: nickname}) do
     user = User.get_cached_by_nickname(nickname)
 
     {:ok, updated_user} = User.set_activation(user, !user.is_active)
@@ -182,12 +199,10 @@ defmodule Pleroma.Web.AdminAPI.UserController do
       action: action
     })
 
-    conn
+    render(conn, "show.json", user: updated_user)
-    |> put_view(AccountView)
-    |> render("show.json", %{user: updated_user})
   end
 
-  def activate(%{assigns: %{user: admin}} = conn, %{"nicknames" => nicknames}) do
+  def activate(%{assigns: %{user: admin}, body_params: %{nicknames: nicknames}} = conn, _) do
     users = Enum.map(nicknames, &User.get_cached_by_nickname/1)
     {:ok, updated_users} = User.set_activation(users, true)
 
@@ -197,12 +212,10 @@ defmodule Pleroma.Web.AdminAPI.UserController do
       action: "activate"
     })
 
-    conn
+    render(conn, "index.json", users: Keyword.values(updated_users))
-    |> put_view(AccountView)
-    |> render("index.json", %{users: Keyword.values(updated_users)})
   end
 
-  def deactivate(%{assigns: %{user: admin}} = conn, %{"nicknames" => nicknames}) do
+  def deactivate(%{assigns: %{user: admin}, body_params: %{nicknames: nicknames}} = conn, _) do
     users = Enum.map(nicknames, &User.get_cached_by_nickname/1)
     {:ok, updated_users} = User.set_activation(users, false)
 
@@ -212,12 +225,10 @@ defmodule Pleroma.Web.AdminAPI.UserController do
       action: "deactivate"
     })
 
-    conn
+    render(conn, "index.json", users: Keyword.values(updated_users))
-    |> put_view(AccountView)
-    |> render("index.json", %{users: Keyword.values(updated_users)})
   end
 
-  def approve(%{assigns: %{user: admin}} = conn, %{"nicknames" => nicknames}) do
+  def approve(%{assigns: %{user: admin}, body_params: %{nicknames: nicknames}} = conn, _) do
     users = Enum.map(nicknames, &User.get_cached_by_nickname/1)
     {:ok, updated_users} = User.approve(users)
 
@@ -227,36 +238,27 @@ defmodule Pleroma.Web.AdminAPI.UserController do
       action: "approve"
     })
 
-    conn
+    render(conn, "index.json", users: updated_users)
-    |> put_view(AccountView)
-    |> render("index.json", %{users: updated_users})
   end
 
-  def list(conn, params) do
+  def index(conn, params) do
     {page, page_size} = page_params(params)
-    filters = maybe_parse_filters(params["filters"])
+    filters = maybe_parse_filters(params[:filters])
 
     search_params =
       %{
-        query: params["query"],
+        query: params[:query],
         page: page,
         page_size: page_size,
-        tags: params["tags"],
+        tags: params[:tags],
-        name: params["name"],
+        name: params[:name],
-        email: params["email"],
+        email: params[:email],
-        actor_types: params["actor_types"]
+        actor_types: params[:actor_types]
       }
       |> Map.merge(filters)
 
     with {:ok, users, count} <- Search.user(search_params) do
-      json(
+      render(conn, "index.json", users: users, count: count, page_size: page_size)
-        conn,
-        AccountView.render("index.json",
-          users: users,
-          count: count,
-          page_size: page_size
-        )
-      )
     end
   end
 
@@ -274,8 +276,8 @@ defmodule Pleroma.Web.AdminAPI.UserController do
 
   defp page_params(params) do
     {
-      fetch_integer_param(params, "page", 1),
+      fetch_integer_param(params, :page, 1),
-      fetch_integer_param(params, "page_size", @users_page_size)
+      fetch_integer_param(params, :page_size, @users_page_size)
     }
   end
 end

lib/pleroma/web/admin_api/views/account_view.ex

@@ -75,7 +75,7 @@ defmodule Pleroma.Web.AdminAPI.AccountView do
       "display_name" => display_name,
       "is_active" => user.is_active,
       "local" => user.local,
-      "roles" => User.roles(user),
+      "roles" => roles(user),
       "tags" => user.tags || [],
       "is_confirmed" => user.is_confirmed,
       "is_approved" => user.is_approved,
@@ -85,6 +85,10 @@ defmodule Pleroma.Web.AdminAPI.AccountView do
     }
   end
 
+  def render("created_many.json", %{users: users}) do
+    render_many(users, AccountView, "created.json", as: :user)
+  end
+
   def render("created.json", %{user: user}) do
     %{
       type: "success",
@@ -96,7 +100,11 @@ defmodule Pleroma.Web.AdminAPI.AccountView do
     }
   end
 
-  def render("create-error.json", %{changeset: %Ecto.Changeset{changes: changes, errors: errors}}) do
+  def render("create_errors.json", %{changesets: changesets}) do
+    render_many(changesets, AccountView, "create_error.json", as: :changeset)
+  end
+
+  def render("create_error.json", %{changeset: %Ecto.Changeset{changes: changes, errors: errors}}) do
     %{
       type: "error",
       code: 409,
@@ -140,4 +148,11 @@ defmodule Pleroma.Web.AdminAPI.AccountView do
 
   defp image_url(%{"url" => [%{"href" => href} | _]}), do: href
   defp image_url(_), do: nil
+
+  defp roles(%{is_moderator: is_moderator, is_admin: is_admin}) do
+    %{
+      admin: is_admin,
+      moderator: is_moderator
+    }
+  end
 end

lib/pleroma/web/api_spec.ex

@@ -92,9 +92,10 @@ defmodule Pleroma.Web.ApiSpec do
               "Invites",
               "MediaProxy cache",
               "OAuth application managment",
-              "Report managment",
               "Relays",
-              "Status administration"
+              "Report managment",
+              "Status administration",
+              "User administration"
             ]
           },
           %{"name" => "Applications", "tags" => ["Applications", "Push subscriptions"]},

lib/pleroma/web/api_spec/cast_and_validate.ex

@@ -15,6 +15,7 @@ defmodule Pleroma.Web.ApiSpec.CastAndValidate do
 
   @behaviour Plug
 
+  alias OpenApiSpex.Plug.PutApiSpec
   alias Plug.Conn
 
   @impl Plug
@@ -25,12 +26,10 @@ defmodule Pleroma.Web.ApiSpec.CastAndValidate do
   end
 
   @impl Plug
-  def call(%{private: %{open_api_spex: private_data}} = conn, %{
+
-        operation_id: operation_id,
+  def call(conn, %{operation_id: operation_id, render_error: render_error}) do
-        render_error: render_error
+    {spec, operation_lookup} = PutApiSpec.get_spec_and_operation_lookup(conn)
-      }) do
+    operation = operation_lookup[operation_id]
-    spec = private_data.spec
-    operation = private_data.operation_lookup[operation_id]
 
     content_type =
       case Conn.get_req_header(conn, "content-type") do
@@ -43,8 +42,7 @@ defmodule Pleroma.Web.ApiSpec.CastAndValidate do
           "application/json"
       end
 
-    private_data = Map.put(private_data, :operation_id, operation_id)
+    conn = Conn.put_private(conn, :operation_id, operation_id)
-    conn = Conn.put_private(conn, :open_api_spex, private_data)
 
     case cast_and_validate(spec, operation, conn, content_type, strict?()) do
       {:ok, conn} ->
@@ -64,25 +62,22 @@ defmodule Pleroma.Web.ApiSpec.CastAndValidate do
           private: %{
             phoenix_controller: controller,
             phoenix_action: action,
-            open_api_spex: private_data
+            open_api_spex: %{spec_module: spec_module}
           }
         } = conn,
         opts
       ) do
+    {spec, operation_lookup} = PutApiSpec.get_spec_and_operation_lookup(conn)
+
     operation =
-      case private_data.operation_lookup[{controller, action}] do
+      case operation_lookup[{controller, action}] do
         nil ->
           operation_id = controller.open_api_operation(action).operationId
-          operation = private_data.operation_lookup[operation_id]
+          operation = operation_lookup[operation_id]
 
-          operation_lookup =
+          operation_lookup = Map.put(operation_lookup, {controller, action}, operation)
-            private_data.operation_lookup
-            |> Map.put({controller, action}, operation)
 
-          OpenApiSpex.Plug.Cache.adapter().put(
+          OpenApiSpex.Plug.Cache.adapter().put(spec_module, {spec, operation_lookup})
-            private_data.spec_module,
-            {private_data.spec, operation_lookup}
-          )
 
           operation
 

lib/pleroma/web/api_spec/operations/admin/report_operation.ex

@@ -136,11 +136,11 @@ defmodule Pleroma.Web.ApiSpec.Admin.ReportOperation do
     }
   end
 
-  defp report_state do
+  def report_state do
     %Schema{type: :string, enum: ["open", "closed", "resolved"]}
   end
 
-  defp id_param do
+  def id_param do
     Operation.parameter(:id, :path, FlakeID, "Report ID",
       example: "9umDrYheeY451cQnEe",
       required: true

lib/pleroma/web/api_spec/operations/admin/user_operation.ex

@@ -0,0 +1,389 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2021 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.ApiSpec.Admin.UserOperation do
+  alias OpenApiSpex.Operation
+  alias OpenApiSpex.Schema
+  alias Pleroma.Web.ApiSpec.Schemas.ActorType
+  alias Pleroma.Web.ApiSpec.Schemas.ApiError
+
+  import Pleroma.Web.ApiSpec.Helpers
+
+  def open_api_operation(action) do
+    operation = String.to_existing_atom("#{action}_operation")
+    apply(__MODULE__, operation, [])
+  end
+
+  def index_operation do
+    %Operation{
+      tags: ["User administration"],
+      summary: "List users",
+      operationId: "AdminAPI.UserController.index",
+      security: [%{"oAuth" => ["admin:read:accounts"]}],
+      parameters: [
+        Operation.parameter(:filters, :query, :string, "Comma separated list of filters"),
+        Operation.parameter(:query, :query, :string, "Search users query"),
+        Operation.parameter(:name, :query, :string, "Search by display name"),
+        Operation.parameter(:email, :query, :string, "Search by email"),
+        Operation.parameter(:page, :query, :integer, "Page Number"),
+        Operation.parameter(:page_size, :query, :integer, "Number of users to return per page"),
+        Operation.parameter(
+          :actor_types,
+          :query,
+          %Schema{type: :array, items: ActorType},
+          "Filter by actor type"
+        ),
+        Operation.parameter(
+          :tags,
+          :query,
+          %Schema{type: :array, items: %Schema{type: :string}},
+          "Filter by tags"
+        )
+        | admin_api_params()
+      ],
+      responses: %{
+        200 =>
+          Operation.response(
+            "Response",
+            "application/json",
+            %Schema{
+              type: :object,
+              properties: %{
+                users: %Schema{type: :array, items: user()},
+                count: %Schema{type: :integer},
+                page_size: %Schema{type: :integer}
+              }
+            }
+          ),
+        403 => Operation.response("Forbidden", "application/json", ApiError)
+      }
+    }
+  end
+
+  def create_operation do
+    %Operation{
+      tags: ["User administration"],
+      summary: "Create a single or multiple users",
+      operationId: "AdminAPI.UserController.create",
+      security: [%{"oAuth" => ["admin:write:accounts"]}],
+      parameters: admin_api_params(),
+      requestBody:
+        request_body(
+          "Parameters",
+          %Schema{
+            description: "POST body for creating users",
+            type: :object,
+            properties: %{
+              users: %Schema{
+                type: :array,
+                items: %Schema{
+                  type: :object,
+                  properties: %{
+                    nickname: %Schema{type: :string},
+                    email: %Schema{type: :string},
+                    password: %Schema{type: :string}
+                  }
+                }
+              }
+            }
+          }
+        ),
+      responses: %{
+        200 =>
+          Operation.response("Response", "application/json", %Schema{
+            type: :array,
+            items: %Schema{
+              type: :object,
+              properties: %{
+                code: %Schema{type: :integer},
+                type: %Schema{type: :string},
+                data: %Schema{
+                  type: :object,
+                  properties: %{
+                    email: %Schema{type: :string, format: :email},
+                    nickname: %Schema{type: :string}
+                  }
+                }
+              }
+            }
+          }),
+        403 => Operation.response("Forbidden", "application/json", ApiError),
+        409 =>
+          Operation.response("Conflict", "application/json", %Schema{
+            type: :array,
+            items: %Schema{
+              type: :object,
+              properties: %{
+                code: %Schema{type: :integer},
+                error: %Schema{type: :string},
+                type: %Schema{type: :string},
+                data: %Schema{
+                  type: :object,
+                  properties: %{
+                    email: %Schema{type: :string, format: :email},
+                    nickname: %Schema{type: :string}
+                  }
+                }
+              }
+            }
+          })
+      }
+    }
+  end
+
+  def show_operation do
+    %Operation{
+      tags: ["User administration"],
+      summary: "Show user",
+      operationId: "AdminAPI.UserController.show",
+      security: [%{"oAuth" => ["admin:read:accounts"]}],
+      parameters: [
+        Operation.parameter(
+          :nickname,
+          :path,
+          :string,
+          "User nickname or ID"
+        )
+        | admin_api_params()
+      ],
+      responses: %{
+        200 => Operation.response("Response", "application/json", user()),
+        403 => Operation.response("Forbidden", "application/json", ApiError),
+        404 => Operation.response("Not Found", "application/json", ApiError)
+      }
+    }
+  end
+
+  def follow_operation do
+    %Operation{
+      tags: ["User administration"],
+      summary: "Follow",
+      operationId: "AdminAPI.UserController.follow",
+      security: [%{"oAuth" => ["admin:write:follows"]}],
+      parameters: admin_api_params(),
+      requestBody:
+        request_body(
+          "Parameters",
+          %Schema{
+            type: :object,
+            properties: %{
+              follower: %Schema{type: :string, description: "Follower nickname"},
+              followed: %Schema{type: :string, description: "Followed nickname"}
+            }
+          }
+        ),
+      responses: %{
+        200 => Operation.response("Response", "application/json", %Schema{type: :string}),
+        403 => Operation.response("Forbidden", "application/json", ApiError)
+      }
+    }
+  end
+
+  def unfollow_operation do
+    %Operation{
+      tags: ["User administration"],
+      summary: "Unfollow",
+      operationId: "AdminAPI.UserController.unfollow",
+      security: [%{"oAuth" => ["admin:write:follows"]}],
+      parameters: admin_api_params(),
+      requestBody:
+        request_body(
+          "Parameters",
+          %Schema{
+            type: :object,
+            properties: %{
+              follower: %Schema{type: :string, description: "Follower nickname"},
+              followed: %Schema{type: :string, description: "Followed nickname"}
+            }
+          }
+        ),
+      responses: %{
+        200 => Operation.response("Response", "application/json", %Schema{type: :string}),
+        403 => Operation.response("Forbidden", "application/json", ApiError)
+      }
+    }
+  end
+
+  def approve_operation do
+    %Operation{
+      tags: ["User administration"],
+      summary: "Approve multiple users",
+      operationId: "AdminAPI.UserController.approve",
+      security: [%{"oAuth" => ["admin:write:accounts"]}],
+      parameters: admin_api_params(),
+      requestBody:
+        request_body(
+          "Parameters",
+          %Schema{
+            description: "POST body for deleting multiple users",
+            type: :object,
+            properties: %{
+              nicknames: %Schema{
+                type: :array,
+                items: %Schema{type: :string}
+              }
+            }
+          }
+        ),
+      responses: %{
+        200 =>
+          Operation.response("Response", "application/json", %Schema{
+            type: :object,
+            properties: %{user: %Schema{type: :array, items: user()}}
+          }),
+        403 => Operation.response("Forbidden", "application/json", ApiError)
+      }
+    }
+  end
+
+  def toggle_activation_operation do
+    %Operation{
+      tags: ["User administration"],
+      summary: "Toggle user activation",
+      operationId: "AdminAPI.UserController.toggle_activation",
+      security: [%{"oAuth" => ["admin:write:accounts"]}],
+      parameters: [
+        Operation.parameter(:nickname, :path, :string, "User nickname")
+        | admin_api_params()
+      ],
+      responses: %{
+        200 => Operation.response("Response", "application/json", user()),
+        403 => Operation.response("Forbidden", "application/json", ApiError)
+      }
+    }
+  end
+
+  def activate_operation do
+    %Operation{
+      tags: ["User administration"],
+      summary: "Activate multiple users",
+      operationId: "AdminAPI.UserController.activate",
+      security: [%{"oAuth" => ["admin:write:accounts"]}],
+      parameters: admin_api_params(),
+      requestBody:
+        request_body(
+          "Parameters",
+          %Schema{
+            description: "POST body for deleting multiple users",
+            type: :object,
+            properties: %{
+              nicknames: %Schema{
+                type: :array,
+                items: %Schema{type: :string}
+              }
+            }
+          }
+        ),
+      responses: %{
+        200 =>
+          Operation.response("Response", "application/json", %Schema{
+            type: :object,
+            properties: %{user: %Schema{type: :array, items: user()}}
+          }),
+        403 => Operation.response("Forbidden", "application/json", ApiError)
+      }
+    }
+  end
+
+  def deactivate_operation do
+    %Operation{
+      tags: ["User administration"],
+      summary: "Deactivates multiple users",
+      operationId: "AdminAPI.UserController.deactivate",
+      security: [%{"oAuth" => ["admin:write:accounts"]}],
+      parameters: admin_api_params(),
+      requestBody:
+        request_body(
+          "Parameters",
+          %Schema{
+            description: "POST body for deleting multiple users",
+            type: :object,
+            properties: %{
+              nicknames: %Schema{
+                type: :array,
+                items: %Schema{type: :string}
+              }
+            }
+          }
+        ),
+      responses: %{
+        200 =>
+          Operation.response("Response", "application/json", %Schema{
+            type: :object,
+            properties: %{user: %Schema{type: :array, items: user()}}
+          }),
+        403 => Operation.response("Forbidden", "application/json", ApiError)
+      }
+    }
+  end
+
+  def delete_operation do
+    %Operation{
+      tags: ["User administration"],
+      summary: "Removes a single or multiple users",
+      operationId: "AdminAPI.UserController.delete",
+      security: [%{"oAuth" => ["admin:write:accounts"]}],
+      parameters: [
+        Operation.parameter(
+          :nickname,
+          :query,
+          :string,
+          "User nickname"
+        )
+        | admin_api_params()
+      ],
+      requestBody:
+        request_body(
+          "Parameters",
+          %Schema{
+            description: "POST body for deleting multiple users",
+            type: :object,
+            properties: %{
+              nicknames: %Schema{
+                type: :array,
+                items: %Schema{type: :string}
+              }
+            }
+          }
+        ),
+      responses: %{
+        200 =>
+          Operation.response("Response", "application/json", %Schema{
+            description: "Array of nicknames",
+            type: :array,
+            items: %Schema{type: :string}
+          }),
+        403 => Operation.response("Forbidden", "application/json", ApiError)
+      }
+    }
+  end
+
+  defp user do
+    %Schema{
+      type: :object,
+      properties: %{
+        id: %Schema{type: :string},
+        email: %Schema{type: :string, format: :email},
+        avatar: %Schema{type: :string, format: :uri},
+        nickname: %Schema{type: :string},
+        display_name: %Schema{type: :string},
+        is_active: %Schema{type: :boolean},
+        local: %Schema{type: :boolean},
+        roles: %Schema{
+          type: :object,
+          properties: %{
+            admin: %Schema{type: :boolean},
+            moderator: %Schema{type: :boolean}
+          }
+        },
+        tags: %Schema{type: :array, items: %Schema{type: :string}},
+        is_confirmed: %Schema{type: :boolean},
+        is_approved: %Schema{type: :boolean},
+        url: %Schema{type: :string, format: :uri},
+        registration_reason: %Schema{type: :string, nullable: true},
+        actor_type: %Schema{type: :string}
+      }
+    }
+  end
+end

lib/pleroma/web/api_spec/operations/pleroma_report_operation.ex

@@ -0,0 +1,97 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2021 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.ApiSpec.PleromaReportOperation do
+  alias OpenApiSpex.Operation
+  alias OpenApiSpex.Schema
+  alias Pleroma.Web.ApiSpec.Admin.ReportOperation
+  alias Pleroma.Web.ApiSpec.Schemas.Account
+  alias Pleroma.Web.ApiSpec.Schemas.ApiError
+  alias Pleroma.Web.ApiSpec.Schemas.FlakeID
+  alias Pleroma.Web.ApiSpec.Schemas.Status
+
+  def open_api_operation(action) do
+    operation = String.to_existing_atom("#{action}_operation")
+    apply(__MODULE__, operation, [])
+  end
+
+  def index_operation do
+    %Operation{
+      tags: ["Reports"],
+      summary: "Get a list of your own reports",
+      operationId: "PleromaAPI.ReportController.index",
+      security: [%{"oAuth" => ["read:reports"]}],
+      parameters: [
+        Operation.parameter(
+          :state,
+          :query,
+          ReportOperation.report_state(),
+          "Filter by report state"
+        ),
+        Operation.parameter(
+          :limit,
+          :query,
+          %Schema{type: :integer},
+          "The number of records to retrieve"
+        ),
+        Operation.parameter(
+          :page,
+          :query,
+          %Schema{type: :integer, default: 1},
+          "Page number"
+        ),
+        Operation.parameter(
+          :page_size,
+          :query,
+          %Schema{type: :integer, default: 50},
+          "Number number of log entries per page"
+        )
+      ],
+      responses: %{
+        200 =>
+          Operation.response("Response", "application/json", %Schema{
+            type: :object,
+            properties: %{
+              total: %Schema{type: :integer},
+              reports: %Schema{
+                type: :array,
+                items: report()
+              }
+            }
+          }),
+        404 => Operation.response("Not Found", "application/json", ApiError)
+      }
+    }
+  end
+
+  def show_operation do
+    %Operation{
+      tags: ["Reports"],
+      summary: "Get an individual report",
+      operationId: "PleromaAPI.ReportController.show",
+      parameters: [ReportOperation.id_param()],
+      security: [%{"oAuth" => ["read:reports"]}],
+      responses: %{
+        200 => Operation.response("Report", "application/json", report()),
+        404 => Operation.response("Not Found", "application/json", ApiError)
+      }
+    }
+  end
+
+  # Copied from ReportOperation.report with removing notes
+  defp report do
+    %Schema{
+      type: :object,
+      properties: %{
+        id: FlakeID,
+        state: ReportOperation.report_state(),
+        account: Account,
+        actor: Account,
+        content: %Schema{type: :string},
+        created_at: %Schema{type: :string, format: :"date-time"},
+        statuses: %Schema{type: :array, items: Status}
+      }
+    }
+  end
+end

lib/pleroma/web/api_spec/operations/status_operation.ex

@@ -59,7 +59,7 @@ defmodule Pleroma.Web.ApiSpec.StatusOperation do
           Operation.response(
             "Status. When `scheduled_at` is present, ScheduledStatus is returned instead",
             "application/json",
-            %Schema{oneOf: [Status, ScheduledStatus]}
+            %Schema{anyOf: [Status, ScheduledStatus]}
           ),
         422 => Operation.response("Bad Request / MRF Rejection", "application/json", ApiError)
       }

lib/pleroma/web/api_spec/schemas/boolean_like.ex

@@ -3,6 +3,7 @@
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Web.ApiSpec.Schemas.BooleanLike do
+  alias OpenApiSpex.Cast
   alias OpenApiSpex.Schema
 
   require OpenApiSpex
@@ -27,10 +28,13 @@ defmodule Pleroma.Web.ApiSpec.Schemas.BooleanLike do
       %Schema{type: :boolean},
       %Schema{type: :string},
       %Schema{type: :integer}
-    ]
+    ],
+    "x-validate": __MODULE__
   })
 
-  def after_cast(value, _schmea) do
+  def cast(%Cast{value: value} = context) do
-    {:ok, Pleroma.Web.ControllerHelper.truthy_param?(value)}
+    context
+    |> Map.put(:value, Pleroma.Web.ControllerHelper.truthy_param?(value))
+    |> Cast.ok()
   end
 end

lib/pleroma/web/api_spec/schemas/scheduled_status.ex

@@ -30,7 +30,8 @@ defmodule Pleroma.Web.ApiSpec.Schemas.ScheduledStatus do
           visibility: %Schema{allOf: [VisibilityScope], nullable: true},
           scheduled_at: %Schema{type: :string, format: :"date-time", nullable: true},
           poll: StatusOperation.poll_params(),
-          in_reply_to_id: %Schema{type: :string, nullable: true}
+          in_reply_to_id: %Schema{type: :string, nullable: true},
+          expires_in: %Schema{type: :integer, nullable: true}
         }
       }
     },
@@ -46,7 +47,8 @@ defmodule Pleroma.Web.ApiSpec.Schemas.ScheduledStatus do
         scheduled_at: nil,
         poll: nil,
         idempotency: nil,
-        in_reply_to_id: nil
+        in_reply_to_id: nil,
+        expires_in: nil
       },
       media_attachments: [Attachment.schema().example]
     }

lib/pleroma/web/api_spec/schemas/status.ex

@@ -23,9 +23,10 @@ defmodule Pleroma.Web.ApiSpec.Schemas.Status do
       application: %Schema{
         description: "The application used to post this status",
         type: :object,
+        nullable: true,
         properties: %{
           name: %Schema{type: :string},
-          website: %Schema{type: :string, nullable: true, format: :uri}
+          website: %Schema{type: :string, format: :uri}
         }
       },
       bookmarked: %Schema{type: :boolean, description: "Have you bookmarked this status?"},
@@ -291,7 +292,7 @@ defmodule Pleroma.Web.ApiSpec.Schemas.Status do
         "url" => "http://localhost:4001/users/nick6",
         "username" => "nick6"
       },
-      "application" => %{"name" => "Web", "website" => nil},
+      "application" => nil,
       "bookmarked" => false,
       "card" => nil,
       "content" => "foobar",

lib/pleroma/web/common_api/activity_draft.ex

@@ -5,6 +5,7 @@
 defmodule Pleroma.Web.CommonAPI.ActivityDraft do
   alias Pleroma.Activity
   alias Pleroma.Conversation.Participation
+  alias Pleroma.Object
   alias Pleroma.Web.CommonAPI
   alias Pleroma.Web.CommonAPI.Utils
 
@@ -179,17 +180,44 @@ defmodule Pleroma.Web.CommonAPI.ActivityDraft do
   end
 
   defp sensitive(draft) do
-    sensitive = draft.params[:sensitive] || Enum.member?(draft.tags, {"#nsfw", "nsfw"})
+    sensitive = draft.params[:sensitive]
     %__MODULE__{draft | sensitive: sensitive}
   end
 
   defp object(draft) do
     emoji = Map.merge(Pleroma.Emoji.Formatter.get_emoji_map(draft.full_payload), draft.emoji)
 
+    # Sometimes people create posts with subject containing emoji,
+    # since subjects are usually copied this will result in a broken
+    # subject when someone replies from an instance that does not have
+    # the emoji or has it under different shortcode. This is an attempt
+    # to mitigate this by copying emoji from inReplyTo if they are present
+    # in the subject.
+    summary_emoji =
+      with %Activity{} <- draft.in_reply_to,
+           %Object{data: %{"tag" => [_ | _] = tag}} <- Object.normalize(draft.in_reply_to) do
+        Enum.reduce(tag, %{}, fn
+          %{"type" => "Emoji", "name" => name, "icon" => %{"url" => url}}, acc ->
+            if String.contains?(draft.summary, name) do
+              Map.put(acc, name, url)
+            else
+              acc
+            end
+
+          _, acc ->
+            acc
+        end)
+      else
+        _ -> %{}
+      end
+
+    emoji = Map.merge(emoji, summary_emoji)
+
     object =
       Utils.make_note_data(draft)
       |> Map.put("emoji", emoji)
       |> Map.put("source", draft.status)
+      |> Map.put("generator", draft.params[:generator])
 
     %__MODULE__{draft | object: object}
   end

lib/pleroma/web/common_api/utils.ex

@@ -217,7 +217,6 @@ defmodule Pleroma.Web.CommonAPI.Utils do
     draft.status
     |> format_input(content_type, options)
     |> maybe_add_attachments(draft.attachments, attachment_links)
-    |> maybe_add_nsfw_tag(draft.params)
   end
 
   defp get_content_type(content_type) do
@@ -228,13 +227,6 @@ defmodule Pleroma.Web.CommonAPI.Utils do
     end
   end
 
-  defp maybe_add_nsfw_tag({text, mentions, tags}, %{"sensitive" => sensitive})
-       when sensitive in [true, "True", "true", "1"] do
-    {text, mentions, [{"#nsfw", "nsfw"} | tags]}
-  end
-
-  defp maybe_add_nsfw_tag(data, _), do: data
-
   def make_context(_, %Participation{} = participation) do
     Repo.preload(participation, :conversation).conversation.ap_id
   end

lib/pleroma/web/fallback/legacy_pleroma_api_rerouter_plug.ex

@@ -0,0 +1,26 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2021 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.Fallback.LegacyPleromaApiRerouterPlug do
+  alias Pleroma.Web.Endpoint
+  alias Pleroma.Web.Fallback.RedirectController
+
+  def init(opts), do: opts
+
+  def call(%{path_info: ["api", "pleroma" | path_info_rest]} = conn, _opts) do
+    new_path_info = ["api", "v1", "pleroma" | path_info_rest]
+    new_request_path = Enum.join(new_path_info, "/")
+
+    conn
+    |> Map.merge(%{
+      path_info: new_path_info,
+      request_path: new_request_path
+    })
+    |> Endpoint.call(conn.params)
+  end
+
+  def call(conn, _opts) do
+    RedirectController.api_not_implemented(conn, %{})
+  end
+end

lib/pleroma/web/feed/feed_view.ex

@@ -32,6 +32,7 @@ defmodule Pleroma.Web.Feed.FeedView do
 
     %{
       activity: activity,
+      object: object,
       data: Map.get(object, :data),
       actor: actor
     }

lib/pleroma/web/mastodon_api/controllers/instance_controller.ex

@@ -5,7 +5,7 @@
 defmodule Pleroma.Web.MastodonAPI.InstanceController do
   use Pleroma.Web, :controller
 
-  plug(OpenApiSpex.Plug.CastAndValidate)
+  plug(Pleroma.Web.ApiSpec.CastAndValidate)
 
   plug(
     :skip_plug,

lib/pleroma/web/mastodon_api/controllers/status_controller.ex

@@ -21,6 +21,7 @@ defmodule Pleroma.Web.MastodonAPI.StatusController do
   alias Pleroma.Web.CommonAPI
   alias Pleroma.Web.MastodonAPI.AccountView
   alias Pleroma.Web.MastodonAPI.ScheduledActivityView
+  alias Pleroma.Web.OAuth.Token
   alias Pleroma.Web.Plugs.OAuthScopesPlug
   alias Pleroma.Web.Plugs.RateLimiter
 
@@ -138,7 +139,9 @@ defmodule Pleroma.Web.MastodonAPI.StatusController do
         _
       )
       when not is_nil(scheduled_at) do
-    params = Map.put(params, :in_reply_to_status_id, params[:in_reply_to_id])
+    params =
+      Map.put(params, :in_reply_to_status_id, params[:in_reply_to_id])
+      |> put_application(conn)
 
     attrs = %{
       params: Map.new(params, fn {key, value} -> {to_string(key), value} end),
@@ -162,7 +165,9 @@ defmodule Pleroma.Web.MastodonAPI.StatusController do
 
   # Creates a regular status
   def create(%{assigns: %{user: user}, body_params: %{status: _} = params} = conn, _) do
-    params = Map.put(params, :in_reply_to_status_id, params[:in_reply_to_id])
+    params =
+      Map.put(params, :in_reply_to_status_id, params[:in_reply_to_id])
+      |> put_application(conn)
 
     with {:ok, activity} <- CommonAPI.post(user, params) do
       try_render(conn, "show.json",
@@ -414,4 +419,15 @@ defmodule Pleroma.Web.MastodonAPI.StatusController do
       as: :activity
     )
   end
+
+  defp put_application(params, %{assigns: %{token: %Token{user: %User{} = user} = token}} = _conn) do
+    if user.disclose_client do
+      %{client_name: client_name, website: website} = Repo.preload(token, :app).app
+      Map.put(params, :generator, %{type: "Application", name: client_name, url: website})
+    else
+      Map.put(params, :generator, nil)
+    end
+  end
+
+  defp put_application(params, _), do: Map.put(params, :generator, nil)
 end

lib/pleroma/web/mastodon_api/controllers/timeline_controller.ex

@@ -133,34 +133,25 @@ defmodule Pleroma.Web.MastodonAPI.TimelineController do
   end
 
   defp hashtag_fetching(params, user, local_only) do
-    tags =
+    # Note: not sanitizing tag options at this stage (may be mix-cased, have duplicates etc.)
+    tags_any =
       [params[:tag], params[:any]]
       |> List.flatten()
-      |> Enum.uniq()
+      |> Enum.filter(& &1)
-      |> Enum.reject(&is_nil/1)
-      |> Enum.map(&String.downcase/1)
 
-    tag_all =
+    tag_all = Map.get(params, :all, [])
-      params
+    tag_reject = Map.get(params, :none, [])
-      |> Map.get(:all, [])
-      |> Enum.map(&String.downcase/1)
 
-    tag_reject =
+    params
-      params
+    |> Map.put(:type, "Create")
-      |> Map.get(:none, [])
+    |> Map.put(:local_only, local_only)
-      |> Enum.map(&String.downcase/1)
+    |> Map.put(:blocking_user, user)
-
+    |> Map.put(:muting_user, user)
-    _activities =
+    |> Map.put(:user, user)
-      params
+    |> Map.put(:tag, tags_any)
-      |> Map.put(:type, "Create")
+    |> Map.put(:tag_all, tag_all)
-      |> Map.put(:local_only, local_only)
+    |> Map.put(:tag_reject, tag_reject)
-      |> Map.put(:blocking_user, user)
+    |> ActivityPub.fetch_public_activities()
-      |> Map.put(:muting_user, user)
-      |> Map.put(:user, user)
-      |> Map.put(:tag, tags)
-      |> Map.put(:tag_all, tag_all)
-      |> Map.put(:tag_reject, tag_reject)
-      |> ActivityPub.fetch_public_activities()
   end
 
   # GET /api/v1/timelines/tag/:tag

lib/pleroma/web/mastodon_api/views/scheduled_activity_view.ex

@@ -37,7 +37,8 @@ defmodule Pleroma.Web.MastodonAPI.ScheduledActivityView do
       visibility: params["visibility"],
       scheduled_at: params["scheduled_at"],
       poll: params["poll"],
-      in_reply_to_id: params["in_reply_to_id"]
+      in_reply_to_id: params["in_reply_to_id"],
+      expires_in: params["expires_in"]
     }
     |> Pleroma.Maps.put_if_present(:media_ids, params["media_ids"])
   end

lib/pleroma/web/mastodon_api/views/status_view.ex

@@ -124,16 +124,16 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
       ) do
     user = CommonAPI.get_user(activity.data["actor"])
     created_at = Utils.to_masto_date(activity.data["published"])
-    activity_object = Object.normalize(activity, fetch: false)
+    object = Object.normalize(activity, fetch: false)
 
     reblogged_parent_activity =
       if opts[:parent_activities] do
         Activity.Queries.find_by_object_ap_id(
           opts[:parent_activities],
-          activity_object.data["id"]
+          object.data["id"]
         )
       else
-        Activity.create_by_object_ap_id(activity_object.data["id"])
+        Activity.create_by_object_ap_id(object.data["id"])
         |> Activity.with_preloaded_bookmark(opts[:for])
         |> Activity.with_set_thread_muted_field(opts[:for])
         |> Repo.one()
@@ -142,7 +142,7 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
     reblog_rendering_opts = Map.put(opts, :activity, reblogged_parent_activity)
     reblogged = render("show.json", reblog_rendering_opts)
 
-    favorited = opts[:for] && opts[:for].ap_id in (activity_object.data["likes"] || [])
+    favorited = opts[:for] && opts[:for].ap_id in (object.data["likes"] || [])
 
     bookmarked = Activity.get_bookmark(reblogged_parent_activity, opts[:for]) != nil
 
@@ -154,8 +154,8 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
 
     %{
       id: to_string(activity.id),
-      uri: activity_object.data["id"],
+      uri: object.data["id"],
-      url: activity_object.data["id"],
+      url: object.data["id"],
       account:
         AccountView.render("show.json", %{
           user: user,
@@ -180,10 +180,7 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
       media_attachments: reblogged[:media_attachments] || [],
       mentions: mentions,
       tags: reblogged[:tags] || [],
-      application: %{
+      application: build_application(object.data["generator"]),
-        name: "Web",
-        website: nil
-      },
       language: nil,
       emojis: [],
       pleroma: %{
@@ -201,8 +198,10 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
     like_count = object.data["like_count"] || 0
     announcement_count = object.data["announcement_count"] || 0
 
-    tags = object.data["tag"] || []
+    hashtags = Object.hashtags(object)
-    sensitive = object.data["sensitive"] || Enum.member?(tags, "nsfw")
+    sensitive = object.data["sensitive"] || Enum.member?(hashtags, "nsfw")
+
+    tags = Object.tags(object)
 
     tag_mentions =
       tags
@@ -348,10 +347,7 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
       poll: render(PollView, "show.json", object: object, for: opts[:for]),
       mentions: mentions,
       tags: build_tags(tags),
-      application: %{
+      application: build_application(object.data["generator"]),
-        name: "Web",
-        website: nil
-      },
       language: nil,
       emojis: build_emojis(object.data["emoji"]),
       pleroma: %{
@@ -385,12 +381,15 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
 
     page_url = page_url_data |> to_string
 
-    image_url =
+    image_url_data =
       if is_binary(rich_media["image"]) do
-        URI.merge(page_url_data, URI.parse(rich_media["image"]))
+        URI.parse(rich_media["image"])
-        |> to_string
+      else
+        nil
       end
 
+    image_url = build_image_url(image_url_data, page_url_data)
+
     %{
       type: "link",
       provider_name: page_url_data.host,
@@ -540,4 +539,29 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
       me: !!(current_user && current_user.ap_id in users)
     }
   end
+
+  @spec build_application(map() | nil) :: map() | nil
+  defp build_application(%{"type" => _type, "name" => name, "url" => url}),
+    do: %{name: name, website: url}
+
+  defp build_application(_), do: nil
+
+  # Workaround for Elixir issue #10771
+  # Avoid applying URI.merge unless necessary
+  # TODO: revert to always attempting URI.merge(image_url_data, page_url_data)
+  # when Elixir 1.12 is the minimum supported version
+  @spec build_image_url(struct() | nil, struct()) :: String.t() | nil
+  defp build_image_url(
+         %URI{scheme: image_scheme, host: image_host} = image_url_data,
+         %URI{} = _page_url_data
+       )
+       when not is_nil(image_scheme) and not is_nil(image_host) do
+    image_url_data |> to_string
+  end
+
+  defp build_image_url(%URI{} = image_url_data, %URI{} = page_url_data) do
+    URI.merge(page_url_data, image_url_data) |> to_string
+  end
+
+  defp build_image_url(_, _), do: nil
 end

lib/pleroma/web/media_proxy.ex

@@ -121,6 +121,11 @@ defmodule Pleroma.Web.MediaProxy do
     end
   end
 
+  def decode_url(encoded) do
+    [_, "proxy", sig, base64 | _] = URI.parse(encoded).path |> String.split("/")
+    decode_url(sig, base64)
+  end
+
   defp signed_url(url) do
     :crypto.hmac(:sha, Config.get([Web.Endpoint, :secret_key_base]), url)
   end

lib/pleroma/web/pleroma_api/controllers/backup_controller.ex

@@ -10,7 +10,7 @@ defmodule Pleroma.Web.PleromaAPI.BackupController do
 
   action_fallback(Pleroma.Web.MastodonAPI.FallbackController)
   plug(OAuthScopesPlug, %{scopes: ["read:accounts"]} when action in [:index, :create])
-  plug(OpenApiSpex.Plug.CastAndValidate, render_error: Pleroma.Web.ApiSpec.RenderError)
+  plug(Pleroma.Web.ApiSpec.CastAndValidate)
 
   defdelegate open_api_operation(action), to: Pleroma.Web.ApiSpec.PleromaBackupOperation
 

lib/pleroma/web/pleroma_api/controllers/chat_controller.ex

@@ -38,7 +38,7 @@ defmodule Pleroma.Web.PleromaAPI.ChatController do
     %{scopes: ["read:chats"]} when action in [:messages, :index, :index2, :show]
   )
 
-  plug(OpenApiSpex.Plug.CastAndValidate, render_error: Pleroma.Web.ApiSpec.RenderError)
+  plug(Pleroma.Web.ApiSpec.CastAndValidate)
 
   defdelegate open_api_operation(action), to: Pleroma.Web.ApiSpec.ChatOperation
 

lib/pleroma/web/pleroma_api/controllers/report_controller.ex

@@ -0,0 +1,46 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2021 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.PleromaAPI.ReportController do
+  use Pleroma.Web, :controller
+
+  alias Pleroma.Activity
+  alias Pleroma.Web.ActivityPub.Utils
+  alias Pleroma.Web.AdminAPI.Report
+
+  action_fallback(Pleroma.Web.MastodonAPI.FallbackController)
+  plug(Pleroma.Web.ApiSpec.CastAndValidate)
+  plug(Pleroma.Web.Plugs.OAuthScopesPlug, %{scopes: ["read:reports"]})
+
+  defdelegate open_api_operation(action), to: Pleroma.Web.ApiSpec.PleromaReportOperation
+
+  @doc "GET /api/v0/pleroma/reports"
+  def index(%{assigns: %{user: user}, body_params: params} = conn, _) do
+    params =
+      params
+      |> Map.put(:actor_id, user.ap_id)
+
+    reports = Utils.get_reports(params, Map.get(params, :page, 1), Map.get(params, :size, 20))
+
+    render(conn, "index.json", %{reports: reports, for: user})
+  end
+
+  @doc "GET /api/v0/pleroma/reports/:id"
+  def show(%{assigns: %{user: user}} = conn, %{id: id}) do
+    with %Activity{} = report <- Activity.get_report(id),
+         true <- report.actor == user.ap_id,
+         %{} = report_info <- Report.extract_report_info(report) do
+      render(conn, "show.json", Map.put(report_info, :for, user))
+    else
+      false ->
+        {:error, :not_found}
+
+      nil ->
+        {:error, :not_found}
+
+      e ->
+        {:error, inspect(e)}
+    end
+  end
+end

lib/pleroma/web/pleroma_api/controllers/user_import_controller.ex

@@ -15,7 +15,7 @@ defmodule Pleroma.Web.PleromaAPI.UserImportController do
   plug(OAuthScopesPlug, %{scopes: ["follow", "write:blocks"]} when action == :blocks)
   plug(OAuthScopesPlug, %{scopes: ["follow", "write:mutes"]} when action == :mutes)
 
-  plug(OpenApiSpex.Plug.CastAndValidate)
+  plug(Pleroma.Web.ApiSpec.CastAndValidate)
   defdelegate open_api_operation(action), to: ApiSpec.UserImportOperation
 
   def follow(%{body_params: %{list: %Plug.Upload{path: path}}} = conn, _) do

lib/pleroma/web/pleroma_api/views/report_view.ex

@@ -0,0 +1,55 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2021 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.PleromaAPI.ReportView do
+  use Pleroma.Web, :view
+
+  alias Pleroma.HTML
+  alias Pleroma.Web.AdminAPI.Report
+  alias Pleroma.Web.CommonAPI.Utils
+  alias Pleroma.Web.MastodonAPI.AccountView
+  alias Pleroma.Web.MastodonAPI.StatusView
+
+  def render("index.json", %{reports: reports, for: for_user}) do
+    %{
+      reports:
+        reports[:items]
+        |> Enum.map(&Report.extract_report_info/1)
+        |> Enum.map(&render(__MODULE__, "show.json", Map.put(&1, :for, for_user))),
+      total: reports[:total]
+    }
+  end
+
+  def render("show.json", %{
+        report: report,
+        user: actor,
+        account: account,
+        statuses: statuses,
+        for: for_user
+      }) do
+    created_at = Utils.to_masto_date(report.data["published"])
+
+    content =
+      unless is_nil(report.data["content"]) do
+        HTML.filter_tags(report.data["content"])
+      else
+        nil
+      end
+
+    %{
+      id: report.id,
+      account: AccountView.render("show.json", %{user: account, for: for_user}),
+      actor: AccountView.render("show.json", %{user: actor, for: for_user}),
+      content: content,
+      created_at: created_at,
+      statuses:
+        StatusView.render("index.json", %{
+          activities: statuses,
+          as: :activity,
+          for: for_user
+        }),
+      state: report.data["state"]
+    }
+  end
+end

lib/pleroma/web/plugs/frontend_static.ex

@@ -10,6 +10,8 @@ defmodule Pleroma.Web.Plugs.FrontendStatic do
   """
   @behaviour Plug
 
+  @api_routes Pleroma.Web.get_api_routes()
+
   def file_path(path, frontend_type \\ :primary) do
     if configuration = Pleroma.Config.get([:frontends, frontend_type]) do
       instance_static_path = Pleroma.Config.get([:instance, :static_dir], "instance/static")
@@ -34,7 +36,8 @@ defmodule Pleroma.Web.Plugs.FrontendStatic do
   end
 
   def call(conn, opts) do
-    with false <- invalid_path?(conn.path_info),
+    with false <- api_route?(conn.path_info),
+         false <- invalid_path?(conn.path_info),
          frontend_type <- Map.get(opts, :frontend_type, :primary),
          path when not is_nil(path) <- file_path("", frontend_type) do
       call_static(conn, opts, path)
@@ -52,6 +55,10 @@ defmodule Pleroma.Web.Plugs.FrontendStatic do
   defp invalid_path?([h | t], match), do: String.contains?(h, match) or invalid_path?(t)
   defp invalid_path?([], _match), do: false
 
+  defp api_route?([h | _]) when h in @api_routes, do: true
+  defp api_route?([_ | t]), do: api_route?(t)
+  defp api_route?([]), do: false
+
   defp call_static(conn, opts, from) do
     opts = Map.put(opts, :from, from)
     Plug.Static.call(conn, opts)

lib/pleroma/web/router.ex

@@ -140,7 +140,7 @@ defmodule Pleroma.Web.Router do
     plug(Pleroma.Web.Plugs.MappedSignatureToIdentityPlug)
   end
 
-  scope "/api/pleroma", Pleroma.Web.TwitterAPI do
+  scope "/api/v1/pleroma", Pleroma.Web.TwitterAPI do
     pipe_through(:pleroma_api)
 
     get("/password_reset/:token", PasswordController, :reset, as: :reset_password)
@@ -150,12 +150,12 @@ defmodule Pleroma.Web.Router do
     get("/healthcheck", UtilController, :healthcheck)
   end
 
-  scope "/api/pleroma", Pleroma.Web do
+  scope "/api/v1/pleroma", Pleroma.Web do
     pipe_through(:pleroma_api)
     post("/uploader_callback/:upload_path", UploaderController, :callback)
   end
 
-  scope "/api/pleroma/admin", Pleroma.Web.AdminAPI do
+  scope "/api/v1/pleroma/admin", Pleroma.Web.AdminAPI do
     pipe_through(:admin_api)
 
     put("/users/disable_mfa", AdminAPIController, :disable_mfa)
@@ -204,7 +204,7 @@ defmodule Pleroma.Web.Router do
     get("/users/:nickname/credentials", AdminAPIController, :show_user_credentials)
     patch("/users/:nickname/credentials", AdminAPIController, :update_user_credentials)
 
-    get("/users", UserController, :list)
+    get("/users", UserController, :index)
     get("/users/:nickname", UserController, :show)
     get("/users/:nickname/statuses", AdminAPIController, :list_user_statuses)
     get("/users/:nickname/chats", AdminAPIController, :list_user_chats)
@@ -259,7 +259,7 @@ defmodule Pleroma.Web.Router do
     post("/backups", AdminAPIController, :create_backup)
   end
 
-  scope "/api/pleroma/emoji", Pleroma.Web.PleromaAPI do
+  scope "/api/v1/pleroma/emoji", Pleroma.Web.PleromaAPI do
     scope "/pack" do
       pipe_through(:admin_api)
 
@@ -368,6 +368,12 @@ defmodule Pleroma.Web.Router do
     get("/statuses/:id/reactions", EmojiReactionController, :index)
   end
 
+  scope "/api/v0/pleroma", Pleroma.Web.PleromaAPI do
+    pipe_through(:authenticated_api)
+    get("/reports", ReportController, :index)
+    get("/reports/:id", ReportController, :show)
+  end
+
   scope "/api/v1/pleroma", Pleroma.Web.PleromaAPI do
     scope [] do
       pipe_through(:authenticated_api)
@@ -809,6 +815,7 @@ defmodule Pleroma.Web.Router do
   scope "/", Pleroma.Web.Fallback do
     get("/registration/:token", RedirectController, :registration_page)
     get("/:maybe_nickname_or_id", RedirectController, :redirector_with_meta)
+    match(:*, "/api/pleroma*path", LegacyPleromaApiRerouterPlug, [])
     get("/api*path", RedirectController, :api_not_implemented)
     get("/*path", RedirectController, :redirector_with_preload)
 

lib/pleroma/web/templates/feed/feed/_activity.atom.eex

@@ -22,7 +22,7 @@
     <link type="text/html" href='<%= @data["external_url"] %>' rel="alternate"/>
   <% end %>
 
-  <%= for tag <- @data["tag"] || [] do %>
+  <%= for tag <- Pleroma.Object.hashtags(@object) do %>
     <category term="<%= tag %>"></category>
   <% end %>
 

lib/pleroma/web/templates/feed/feed/_activity.rss.eex

@@ -22,7 +22,7 @@
 
   <link rel="ostatus:conversation"><%= activity_context(@activity) %></link>
 
-  <%= for tag <- @data["tag"] || [] do %>
+  <%= for tag <- Pleroma.Object.hashtags(@object) do %>
     <category term="<%= tag %>"></category>
   <% end %>
 

lib/pleroma/web/templates/feed/feed/_tag_activity.atom.eex

@@ -41,7 +41,7 @@
       <% end %>
     <% end %>
 
-    <%= for tag <- @data["tag"] || [] do %>
+    <%= for tag <- Pleroma.Object.hashtags(@object) do %>
       <category term="<%= tag %>"></category>
     <% end %>
 

lib/pleroma/web/web_finger.ex

@@ -94,52 +94,56 @@ defmodule Pleroma.Web.WebFinger do
     |> XmlBuilder.to_doc()
   end
 
-  defp webfinger_from_xml(doc) do
+  defp webfinger_from_xml(body) do
-    subject = XML.string_from_xpath("//Subject", doc)
+    with {:ok, doc} <- XML.parse_document(body) do
+      subject = XML.string_from_xpath("//Subject", doc)
 
-    subscribe_address =
+      subscribe_address =
-      ~s{//Link[@rel="http://ostatus.org/schema/1.0/subscribe"]/@template}
+        ~s{//Link[@rel="http://ostatus.org/schema/1.0/subscribe"]/@template}
-      |> XML.string_from_xpath(doc)
+        |> XML.string_from_xpath(doc)
 
-    ap_id =
+      ap_id =
-      ~s{//Link[@rel="self" and @type="application/activity+json"]/@href}
+        ~s{//Link[@rel="self" and @type="application/activity+json"]/@href}
-      |> XML.string_from_xpath(doc)
+        |> XML.string_from_xpath(doc)
 
-    data = %{
+      data = %{
-      "subject" => subject,
+        "subject" => subject,
-      "subscribe_address" => subscribe_address,
+        "subscribe_address" => subscribe_address,
-      "ap_id" => ap_id
+        "ap_id" => ap_id
-    }
+      }
 
-    {:ok, data}
+      {:ok, data}
+    end
   end
 
-  defp webfinger_from_json(doc) do
+  defp webfinger_from_json(body) do
-    data =
+    with {:ok, doc} <- Jason.decode(body) do
-      Enum.reduce(doc["links"], %{"subject" => doc["subject"]}, fn link, data ->
+      data =
-        case {link["type"], link["rel"]} do
+        Enum.reduce(doc["links"], %{"subject" => doc["subject"]}, fn link, data ->
-          {"application/activity+json", "self"} ->
+          case {link["type"], link["rel"]} do
-            Map.put(data, "ap_id", link["href"])
+            {"application/activity+json", "self"} ->
+              Map.put(data, "ap_id", link["href"])
 
-          {"application/ld+json; profile=\"https://www.w3.org/ns/activitystreams\"", "self"} ->
+            {"application/ld+json; profile=\"https://www.w3.org/ns/activitystreams\"", "self"} ->
-            Map.put(data, "ap_id", link["href"])
+              Map.put(data, "ap_id", link["href"])
 
-          {nil, "http://ostatus.org/schema/1.0/subscribe"} ->
+            {nil, "http://ostatus.org/schema/1.0/subscribe"} ->
-            Map.put(data, "subscribe_address", link["template"])
+              Map.put(data, "subscribe_address", link["template"])
 
-          _ ->
+            _ ->
-            Logger.debug("Unhandled type: #{inspect(link["type"])}")
+              Logger.debug("Unhandled type: #{inspect(link["type"])}")
-            data
+              data
-        end
+          end
-      end)
+        end)
 
-    {:ok, data}
+      {:ok, data}
+    end
   end
 
   def get_template_from_xml(body) do
     xpath = "//Link[@rel='lrdd']/@template"
 
-    with doc when doc != :error <- XML.parse_document(body),
+    with {:ok, doc} <- XML.parse_document(body),
          template when template != nil <- XML.string_from_xpath(xpath, doc) do
       {:ok, template}
     end
@@ -192,15 +196,23 @@ defmodule Pleroma.Web.WebFinger do
              address,
              [{"accept", "application/xrd+xml,application/jrd+json"}]
            ),
-         {:ok, %{status: status, body: body}} when status in 200..299 <- response do
+         {:ok, %{status: status, body: body, headers: headers}} when status in 200..299 <-
-      doc = XML.parse_document(body)
+           response do
+      case List.keyfind(headers, "content-type", 0) do
+        {_, content_type} ->
+          case Plug.Conn.Utils.media_type(content_type) do
+            {:ok, "application", subtype, _} when subtype in ~w(xrd+xml xml) ->
+              webfinger_from_xml(body)
 
-      if doc != :error do
+            {:ok, "application", subtype, _} when subtype in ~w(jrd+json json) ->
-        webfinger_from_xml(doc)
+              webfinger_from_json(body)
-      else
+
-        with {:ok, doc} <- Jason.decode(body) do
+            _ ->
-          webfinger_from_json(doc)
+              {:error, {:content_type, content_type}}
-        end
+          end
+
+        _ ->
+          {:error, {:content_type, nil}}
       end
     else
       e ->

lib/pleroma/web/xml.ex

@@ -31,7 +31,7 @@ defmodule Pleroma.Web.XML do
         |> :binary.bin_to_list()
         |> :xmerl_scan.string(quiet: true)
 
-      doc
+      {:ok, doc}
     rescue
       _e ->
         Logger.debug("Couldn't parse XML: #{inspect(text)}")

mix.exs

@@ -4,7 +4,7 @@ defmodule Pleroma.Mixfile do
   def project do
     [
       app: :pleroma,
-      version: version("2.2.50"),
+      version: version("2.3.50"),
       elixir: "~> 1.9",
       elixirc_paths: elixirc_paths(Mix.env()),
       compilers: [:phoenix, :gettext] ++ Mix.compilers(),
@@ -121,6 +121,7 @@ defmodule Pleroma.Mixfile do
       {:phoenix_pubsub, "~> 2.0"},
       {:phoenix_ecto, "~> 4.0"},
       {:ecto_enum, "~> 1.4"},
+      {:ecto_explain, "~> 0.1.2"},
       {:ecto_sql, "~> 3.4.4"},
       {:postgrex, ">= 0.15.5"},
       {:oban, "~> 2.3.4"},
@@ -157,7 +158,7 @@ defmodule Pleroma.Mixfile do
       {:floki, "~> 0.27"},
       {:timex, "~> 3.6"},
       {:ueberauth, "~> 0.4"},
-      {:linkify, "~> 0.4.1"},
+      {:linkify, "~> 0.5.0"},
       {:http_signatures, "~> 0.1.0"},
       {:telemetry, "~> 0.3"},
       {:poolboy, "~> 1.5"},
@@ -195,9 +196,7 @@ defmodule Pleroma.Mixfile do
       {:majic,
        git: "https://git.pleroma.social/pleroma/elixir-libraries/majic.git",
        ref: "289cda1b6d0d70ccb2ba508a2b0bd24638db2880"},
-      {:open_api_spex,
+      {:open_api_spex, "~> 3.10"},
-       git: "https://git.pleroma.social/pleroma/elixir-libraries/open_api_spex.git",
-       ref: "f296ac0924ba3cf79c7a588c4c252889df4c2edd"},
 
       ## dev & test
       {:ex_doc, "~> 0.22", only: :dev, runtime: false},

mix.lock

@@ -31,6 +31,7 @@
   "earmark_parser": {:hex, :earmark_parser, "1.4.10", "6603d7a603b9c18d3d20db69921527f82ef09990885ed7525003c7fe7dc86c56", [:mix], [], "hexpm", "8e2d5370b732385db2c9b22215c3f59c84ac7dda7ed7e544d7c459496ae519c0"},
   "ecto": {:hex, :ecto, "3.4.6", "08f7afad3257d6eb8613309af31037e16c36808dfda5a3cd0cb4e9738db030e4", [:mix], [{:decimal, "~> 1.6 or ~> 2.0", [hex: :decimal, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.4", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "6f13a9e2a62e75c2dcfc7207bfc65645ab387af8360db4c89fee8b5a4bf3f70b"},
   "ecto_enum": {:hex, :ecto_enum, "1.4.0", "d14b00e04b974afc69c251632d1e49594d899067ee2b376277efd8233027aec8", [:mix], [{:ecto, ">= 3.0.0", [hex: :ecto, repo: "hexpm", optional: false]}, {:ecto_sql, "> 3.0.0", [hex: :ecto_sql, repo: "hexpm", optional: false]}, {:mariaex, ">= 0.0.0", [hex: :mariaex, repo: "hexpm", optional: true]}, {:postgrex, ">= 0.0.0", [hex: :postgrex, repo: "hexpm", optional: true]}], "hexpm", "8fb55c087181c2b15eee406519dc22578fa60dd82c088be376d0010172764ee4"},
+  "ecto_explain": {:hex, :ecto_explain, "0.1.2", "a9d504cbd4adc809911f796d5ef7ebb17a576a6d32286c3d464c015bd39d5541", [:mix], [], "hexpm", "1d0e7798ae30ecf4ce34e912e5354a0c1c832b7ebceba39298270b9a9f316330"},
   "ecto_sql": {:hex, :ecto_sql, "3.4.5", "30161f81b167d561a9a2df4329c10ae05ff36eca7ccc84628f2c8b9fa1e43323", [:mix], [{:db_connection, "~> 2.2", [hex: :db_connection, repo: "hexpm", optional: false]}, {:ecto, "~> 3.4.3", [hex: :ecto, repo: "hexpm", optional: false]}, {:myxql, "~> 0.3.0 or ~> 0.4.0", [hex: :myxql, repo: "hexpm", optional: true]}, {:postgrex, "~> 0.15.0", [hex: :postgrex, repo: "hexpm", optional: true]}, {:tds, "~> 2.1.0", [hex: :tds, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.4.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "31990c6a3579b36a3c0841d34a94c275e727de8b84f58509da5f1b2032c98ac2"},
   "eimp": {:hex, :eimp, "1.0.14", "fc297f0c7e2700457a95a60c7010a5f1dcb768a083b6d53f49cd94ab95a28f22", [:rebar3], [{:p1_utils, "1.0.18", [hex: :p1_utils, repo: "hexpm", optional: false]}], "hexpm", "501133f3112079b92d9e22da8b88bf4f0e13d4d67ae9c15c42c30bd25ceb83b6"},
   "elixir_make": {:hex, :elixir_make, "0.6.2", "7dffacd77dec4c37b39af867cedaabb0b59f6a871f89722c25b28fcd4bd70530", [:mix], [], "hexpm", "03e49eadda22526a7e5279d53321d1cced6552f344ba4e03e619063de75348d9"},
@@ -51,7 +52,7 @@
   "gen_smtp": {:hex, :gen_smtp, "0.15.0", "9f51960c17769b26833b50df0b96123605a8024738b62db747fece14eb2fbfcc", [:rebar3], [], "hexpm", "29bd14a88030980849c7ed2447b8db6d6c9278a28b11a44cafe41b791205440f"},
   "gen_stage": {:hex, :gen_stage, "0.14.3", "d0c66f1c87faa301c1a85a809a3ee9097a4264b2edf7644bf5c123237ef732bf", [:mix], [], "hexpm"},
   "gen_state_machine": {:hex, :gen_state_machine, "2.0.5", "9ac15ec6e66acac994cc442dcc2c6f9796cf380ec4b08267223014be1c728a95", [:mix], [], "hexpm"},
-  "gettext": {:hex, :gettext, "0.18.0", "406d6b9e0e3278162c2ae1de0a60270452c553536772167e2d701f028116f870", [:mix], [], "hexpm", "c3f850be6367ebe1a08616c2158affe4a23231c70391050bf359d5f92f66a571"},
+  "gettext": {:hex, :gettext, "0.18.2", "7df3ea191bb56c0309c00a783334b288d08a879f53a7014341284635850a6e55", [:mix], [], "hexpm", "f9f537b13d4fdd30f3039d33cb80144c3aa1f8d9698e47d7bcbcc8df93b1f5c5"},
   "gun": {:git, "https://github.com/ninenines/gun.git", "921c47146b2d9567eac7e9a4d2ccc60fffd4f327", [ref: "921c47146b2d9567eac7e9a4d2ccc60fffd4f327"]},
   "hackney": {:git, "https://git.pleroma.social/pleroma/elixir-libraries/hackney.git", "7d7119f0651515d6d7669c78393fd90950a3ec6e", [ref: "7d7119f0651515d6d7669c78393fd90950a3ec6e"]},
   "html_entities": {:hex, :html_entities, "0.5.1", "1c9715058b42c35a2ab65edc5b36d0ea66dd083767bef6e3edb57870ef556549", [:mix], [], "hexpm", "30efab070904eb897ff05cd52fa61c1025d7f8ef3a9ca250bc4e6513d16c32de"},
@@ -65,7 +66,7 @@
   "jose": {:hex, :jose, "1.10.1", "16d8e460dae7203c6d1efa3f277e25b5af8b659febfc2f2eb4bacf87f128b80a", [:mix, :rebar3], [], "hexpm", "3c7ddc8a9394b92891db7c2771da94bf819834a1a4c92e30857b7d582e2f8257"},
   "jumper": {:hex, :jumper, "1.0.1", "3c00542ef1a83532b72269fab9f0f0c82bf23a35e27d278bfd9ed0865cecabff", [:mix], [], "hexpm", "318c59078ac220e966d27af3646026db9b5a5e6703cb2aa3e26bcfaba65b7433"},
   "libring": {:hex, :libring, "1.4.0", "41246ba2f3fbc76b3971f6bce83119dfec1eee17e977a48d8a9cfaaf58c2a8d6", [:mix], [], "hexpm"},
-  "linkify": {:hex, :linkify, "0.4.1", "f881eb3429ae88010cf736e6fb3eed406c187bcdd544902ec937496636b7c7b3", [:mix], [], "hexpm", "ce98693f54ae9ace59f2f7a8aed3de2ef311381a8ce7794804bd75484c371dda"},
+  "linkify": {:hex, :linkify, "0.5.0", "e0ea8de73ff44742d6a889721221f4c4eccaad5284957ee9832ffeb347602d54", [:mix], [], "hexpm", "4ccd958350aee7c51c89e21f05b15d30596ebbba707e051d21766be1809df2d7"},
   "majic": {:git, "https://git.pleroma.social/pleroma/elixir-libraries/majic.git", "289cda1b6d0d70ccb2ba508a2b0bd24638db2880", [ref: "289cda1b6d0d70ccb2ba508a2b0bd24638db2880"]},
   "makeup": {:hex, :makeup, "1.0.3", "e339e2f766d12e7260e6672dd4047405963c5ec99661abdc432e6ec67d29ef95", [:mix], [{:nimble_parsec, "~> 0.5", [hex: :nimble_parsec, repo: "hexpm", optional: false]}], "hexpm", "2e9b4996d11832947731f7608fed7ad2f9443011b3b479ae288011265cdd3dad"},
   "makeup_elixir": {:hex, :makeup_elixir, "0.14.1", "4f0e96847c63c17841d42c08107405a005a2680eb9c7ccadfd757bd31dabccfb", [:mix], [{:makeup, "~> 1.0", [hex: :makeup, repo: "hexpm", optional: false]}], "hexpm", "f2438b1a80eaec9ede832b5c41cd4f373b38fd7aa33e3b22d9db79e640cbde11"},
@@ -82,7 +83,7 @@
   "nimble_pool": {:hex, :nimble_pool, "0.1.0", "ffa9d5be27eee2b00b0c634eb649aa27f97b39186fec3c493716c2a33e784ec6", [:mix], [], "hexpm", "343a1eaa620ddcf3430a83f39f2af499fe2370390d4f785cd475b4df5acaf3f9"},
   "nodex": {:git, "https://git.pleroma.social/pleroma/nodex", "cb6730f943cfc6aad674c92161be23a8411f15d1", [ref: "cb6730f943cfc6aad674c92161be23a8411f15d1"]},
   "oban": {:hex, :oban, "2.3.4", "ec7509b9af2524d55f529cb7aee93d36131ae0bf0f37706f65d2fe707f4d9fd8", [:mix], [{:ecto_sql, ">= 3.4.3", [hex: :ecto_sql, repo: "hexpm", optional: false]}, {:jason, "~> 1.1", [hex: :jason, repo: "hexpm", optional: false]}, {:postgrex, "~> 0.14", [hex: :postgrex, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "c70ca0434758fd1805422ea4446af5e910ddc697c0c861549c8f0eb0cfbd2fdf"},
-  "open_api_spex": {:git, "https://git.pleroma.social/pleroma/elixir-libraries/open_api_spex.git", "f296ac0924ba3cf79c7a588c4c252889df4c2edd", [ref: "f296ac0924ba3cf79c7a588c4c252889df4c2edd"]},
+  "open_api_spex": {:hex, :open_api_spex, "3.10.0", "94e9521ad525b3fcf6dc77da7c45f87fdac24756d4de588cb0816b413e7c1844", [:mix], [{:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}, {:plug, "~> 1.7", [hex: :plug, repo: "hexpm", optional: false]}, {:poison, "~> 3.1", [hex: :poison, repo: "hexpm", optional: true]}], "hexpm", "2dbb2bde3d2b821f06936e8dfaf3284331186556291946d84eeba3750ac28765"},
   "p1_utils": {:hex, :p1_utils, "1.0.18", "3fe224de5b2e190d730a3c5da9d6e8540c96484cf4b4692921d1e28f0c32b01c", [:rebar3], [], "hexpm", "1fc8773a71a15553b179c986b22fbeead19b28fe486c332d4929700ffeb71f88"},
   "parse_trans": {:git, "https://github.com/uwiger/parse_trans.git", "76abb347c3c1d00fb0ccf9e4b43e22b3d2288484", [tag: "3.3.0"]},
   "pbkdf2_elixir": {:hex, :pbkdf2_elixir, "1.2.1", "9cbe354b58121075bd20eb83076900a3832324b7dd171a6895fab57b6bb2752c", [:mix], [{:comeonin, "~> 5.3", [hex: :comeonin, repo: "hexpm", optional: false]}], "hexpm", "d3b40a4a4630f0b442f19eca891fcfeeee4c40871936fed2f68e1c4faa30481f"},
@@ -116,9 +117,9 @@
   "syslog": {:hex, :syslog, "1.1.0", "6419a232bea84f07b56dc575225007ffe34d9fdc91abe6f1b2f254fd71d8efc2", [:rebar3], [], "hexpm", "4c6a41373c7e20587be33ef841d3de6f3beba08519809329ecc4d27b15b659e1"},
   "telemetry": {:hex, :telemetry, "0.4.2", "2808c992455e08d6177322f14d3bdb6b625fbcfd233a73505870d8738a2f4599", [:rebar3], [], "hexpm", "2d1419bd9dda6a206d7b5852179511722e2b18812310d304620c7bd92a13fcef"},
   "tesla": {:hex, :tesla, "1.4.0", "1081bef0124b8bdec1c3d330bbe91956648fb008cf0d3950a369cda466a31a87", [:mix], [{:castore, "~> 0.1", [hex: :castore, repo: "hexpm", optional: true]}, {:exjsx, ">= 3.0.0", [hex: :exjsx, repo: "hexpm", optional: true]}, {:finch, "~> 0.3", [hex: :finch, repo: "hexpm", optional: true]}, {:fuse, "~> 2.4", [hex: :fuse, repo: "hexpm", optional: true]}, {:gun, "~> 1.3", [hex: :gun, repo: "hexpm", optional: true]}, {:hackney, "~> 1.6", [hex: :hackney, repo: "hexpm", optional: true]}, {:ibrowse, "~> 4.4.0", [hex: :ibrowse, repo: "hexpm", optional: true]}, {:jason, ">= 1.0.0", [hex: :jason, repo: "hexpm", optional: true]}, {:mime, "~> 1.0", [hex: :mime, repo: "hexpm", optional: false]}, {:mint, "~> 1.0", [hex: :mint, repo: "hexpm", optional: true]}, {:poison, ">= 1.0.0", [hex: :poison, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.4", [hex: :telemetry, repo: "hexpm", optional: true]}], "hexpm", "bf1374a5569f5fca8e641363b63f7347d680d91388880979a33bc12a6eb3e0aa"},
-  "timex": {:hex, :timex, "3.6.2", "845cdeb6119e2fef10751c0b247b6c59d86d78554c83f78db612e3290f819bc2", [:mix], [{:combine, "~> 0.10", [hex: :combine, repo: "hexpm", optional: false]}, {:gettext, "~> 0.10", [hex: :gettext, repo: "hexpm", optional: false]}, {:tzdata, "~> 0.1.8 or ~> 0.5 or ~> 1.0.0", [hex: :tzdata, repo: "hexpm", optional: false]}], "hexpm", "26030b46199d02a590be61c2394b37ea25a3664c02fafbeca0b24c972025d47a"},
+  "timex": {:hex, :timex, "3.7.3", "df8a2ea814749d700d6878ab9eacac9fdb498ecee2f507cb0002ec172bc24d0f", [:mix], [{:combine, "~> 0.10", [hex: :combine, repo: "hexpm", optional: false]}, {:gettext, "~> 0.10", [hex: :gettext, repo: "hexpm", optional: false]}, {:tzdata, "~> 1.0", [hex: :tzdata, repo: "hexpm", optional: false]}], "hexpm", "8691c1d86ca3a7bc14a156e2199dc8927be95d1a8f0e3b69e4bb2d6262c53ac6"},
   "trailing_format_plug": {:hex, :trailing_format_plug, "0.0.7", "64b877f912cf7273bed03379936df39894149e35137ac9509117e59866e10e45", [:mix], [{:plug, "> 0.12.0", [hex: :plug, repo: "hexpm", optional: false]}], "hexpm", "bd4fde4c15f3e993a999e019d64347489b91b7a9096af68b2bdadd192afa693f"},
-  "tzdata": {:hex, :tzdata, "1.0.4", "a3baa4709ea8dba552dca165af6ae97c624a2d6ac14bd265165eaa8e8af94af6", [:mix], [{:hackney, "~> 1.0", [hex: :hackney, repo: "hexpm", optional: false]}], "hexpm", "b02637db3df1fd66dd2d3c4f194a81633d0e4b44308d36c1b2fdfd1e4e6f169b"},
+  "tzdata": {:hex, :tzdata, "1.0.5", "69f1ee029a49afa04ad77801febaf69385f3d3e3d1e4b56b9469025677b89a28", [:mix], [{:hackney, "~> 1.0", [hex: :hackney, repo: "hexpm", optional: false]}], "hexpm", "55519aa2a99e5d2095c1e61cc74c9be69688f8ab75c27da724eb8279ff402a5a"},
   "ueberauth": {:hex, :ueberauth, "0.6.3", "d42ace28b870e8072cf30e32e385579c57b9cc96ec74fa1f30f30da9c14f3cc0", [:mix], [{:plug, "~> 1.5", [hex: :plug, repo: "hexpm", optional: false]}], "hexpm", "afc293d8a1140d6591b53e3eaf415ca92842cb1d32fad3c450c6f045f7f91b60"},
   "unicode_util_compat": {:git, "https://github.com/benoitc/unicode_util_compat.git", "38d7bc105f51159e8ea3279c40121db9db1e652f", [tag: "0.3.1"]},
   "unsafe": {:hex, :unsafe, "1.0.1", "a27e1874f72ee49312e0a9ec2e0b27924214a05e3ddac90e91727bc76f8613d8", [:mix], [], "hexpm", "6c7729a2d214806450d29766abc2afaa7a2cbecf415be64f36a6691afebb50e5"},

priv/gettext/it/LC_MESSAGES/errors.po

@@ -3,8 +3,8 @@ msgstr ""
 "Project-Id-Version: PACKAGE VERSION\n"
 "Report-Msgid-Bugs-To: \n"
 "POT-Creation-Date: 2020-06-19 14:33+0000\n"
-"PO-Revision-Date: 2020-07-09 14:40+0000\n"
+"PO-Revision-Date: 2021-03-13 09:40+0000\n"
-"Last-Translator: Ben Is <srsbzns@cock.li>\n"
+"Last-Translator: Ben Is <spambenis@fastwebnet.it>\n"
 "Language-Team: Italian <https://translate.pleroma.social/projects/pleroma/"
 "pleroma/it/>\n"
 "Language: it\n"
@@ -45,7 +45,7 @@ msgstr "ha una voce invalida"
 
 ## From Ecto.Changeset.validate_exclusion/3
 msgid "is reserved"
-msgstr "è vietato"
+msgstr "è riservato"
 
 ## From Ecto.Changeset.validate_confirmation/3
 msgid "does not match confirmation"
@@ -123,7 +123,7 @@ msgstr "Richiesta invalida"
 #: lib/pleroma/web/activity_pub/activity_pub_controller.ex:425
 #, elixir-format
 msgid "Can't delete object"
-msgstr "Non puoi eliminare quest'oggetto"
+msgstr "Oggetto non eliminabile"
 
 #: lib/pleroma/web/mastodon_api/controllers/status_controller.ex:196
 #, elixir-format
@@ -160,12 +160,12 @@ msgstr "Non puoi pubblicare un messaggio vuoto senza allegati"
 #: lib/pleroma/web/common_api/utils.ex:504
 #, elixir-format
 msgid "Comment must be up to %{max_size} characters"
-msgstr "I commenti posso al massimo consistere di %{max_size} caratteri"
+msgstr "I commenti posso al massimo contenere %{max_size} caratteri"
 
 #: lib/pleroma/config/config_db.ex:222
 #, elixir-format
 msgid "Config with params %{params} not found"
-msgstr "Configurazione con parametri %{max_size} non trovata"
+msgstr "Configurazione con parametri %{params} non trovata"
 
 #: lib/pleroma/web/common_api/common_api.ex:95
 #, elixir-format
@@ -200,7 +200,7 @@ msgstr "Non de-intestato"
 #: lib/pleroma/web/common_api/common_api.ex:126
 #, elixir-format
 msgid "Could not unrepeat"
-msgstr "Non de-ripetuto"
+msgstr "Non de-condiviso"
 
 #: lib/pleroma/web/common_api/common_api.ex:428
 #: lib/pleroma/web/common_api/common_api.ex:437
@@ -310,12 +310,12 @@ msgstr "Il messaggio ha superato la lunghezza massima"
 #: lib/pleroma/plugs/ensure_public_or_authenticated_plug.ex:31
 #, elixir-format
 msgid "This resource requires authentication."
-msgstr "Accedi per leggere."
+msgstr "Accedi per poter leggere."
 
 #: lib/pleroma/plugs/rate_limiter/rate_limiter.ex:206
 #, elixir-format
 msgid "Throttled"
-msgstr "Strozzato"
+msgstr "Limitato"
 
 #: lib/pleroma/web/common_api/common_api.ex:266
 #, elixir-format
@@ -347,17 +347,17 @@ msgstr "Devi aggiungere un indirizzo email valido"
 #: lib/pleroma/web/activity_pub/activity_pub_controller.ex:389
 #, elixir-format
 msgid "can't read inbox of %{nickname} as %{as_nickname}"
-msgstr "non puoi leggere i messaggi privati di %{nickname} come %{as_nickname}"
+msgstr "non puoi leggere i messaggi di %{nickname} come %{as_nickname}"
 
 #: lib/pleroma/web/activity_pub/activity_pub_controller.ex:472
 #, elixir-format
 msgid "can't update outbox of %{nickname} as %{as_nickname}"
-msgstr "non puoi aggiornare gli inviati di %{nickname} come %{as_nickname}"
+msgstr "non puoi inviare da %{nickname} come %{as_nickname}"
 
 #: lib/pleroma/web/common_api/common_api.ex:388
 #, elixir-format
 msgid "conversation is already muted"
-msgstr "la conversazione è già zittita"
+msgstr "la conversazione è già silenziata"
 
 #: lib/pleroma/web/activity_pub/activity_pub_controller.ex:316
 #: lib/pleroma/web/activity_pub/activity_pub_controller.ex:491
@@ -419,7 +419,7 @@ msgstr "Errore interno"
 #: lib/pleroma/web/oauth/fallback_controller.ex:29
 #, elixir-format
 msgid "Invalid Username/Password"
-msgstr "Nome utente/parola d'ordine invalidi"
+msgstr "Nome utente/password invalidi"
 
 #: lib/pleroma/web/twitter_api/twitter_api.ex:118
 #, elixir-format
@@ -455,7 +455,7 @@ msgstr "Gestore OAuth non supportato: %{provider}."
 #: lib/pleroma/uploaders/uploader.ex:72
 #, elixir-format
 msgid "Uploader callback timeout"
-msgstr "Callback caricatmento scaduta"
+msgstr "Callback caricamento scaduta"
 
 #: lib/pleroma/web/uploader_controller.ex:23
 #, elixir-format
@@ -496,7 +496,7 @@ msgstr "Parametro mancante: %{name}"
 #: lib/pleroma/web/oauth/oauth_controller.ex:322
 #, elixir-format
 msgid "Password reset is required"
-msgstr "Necessario reimpostare parola d'ordine"
+msgstr "Necessario reimpostare password"
 
 #: lib/pleroma/tests/auth_test_controller.ex:9
 #: lib/pleroma/web/activity_pub/activity_pub_controller.ex:6 lib/pleroma/web/admin_api/admin_api_controller.ex:6
@@ -540,34 +540,32 @@ msgstr ""
 #: lib/pleroma/web/pleroma_api/controllers/emoji_api_controller.ex:210
 #, elixir-format
 msgid "Unexpected error occurred while adding file to pack."
-msgstr "Errore inaspettato durante l'aggiunta del file al pacchetto."
+msgstr "Errore inatteso durante l'aggiunta del file al pacchetto."
 
 #: lib/pleroma/web/pleroma_api/controllers/emoji_api_controller.ex:138
 #, elixir-format
 msgid "Unexpected error occurred while creating pack."
-msgstr "Errore inaspettato durante la creazione del pacchetto."
+msgstr "Errore inatteso durante la creazione del pacchetto."
 
 #: lib/pleroma/web/pleroma_api/controllers/emoji_api_controller.ex:278
 #, elixir-format
 msgid "Unexpected error occurred while removing file from pack."
-msgstr "Errore inaspettato durante la rimozione del file dal pacchetto."
+msgstr "Errore inatteso durante la rimozione del file dal pacchetto."
 
 #: lib/pleroma/web/pleroma_api/controllers/emoji_api_controller.ex:250
 #, elixir-format
 msgid "Unexpected error occurred while updating file in pack."
-msgstr "Errore inaspettato durante l'aggiornamento del file nel pacchetto."
+msgstr "Errore inatteso durante l'aggiornamento del file nel pacchetto."
 
 #: lib/pleroma/web/pleroma_api/controllers/emoji_api_controller.ex:179
 #, elixir-format
 msgid "Unexpected error occurred while updating pack metadata."
-msgstr "Errore inaspettato durante l'aggiornamento dei metadati del pacchetto."
+msgstr "Errore inatteso durante l'aggiornamento dei metadati del pacchetto."
 
 #: lib/pleroma/plugs/user_is_admin_plug.ex:21
 #, elixir-format
 msgid "User is not an admin."
-msgstr ""
+msgstr "L'utente non è un amministratore."
-"L'utente non è un amministratore."
-"OAuth."
 
 #: lib/pleroma/web/mastodon_api/controllers/subscription_controller.ex:61
 #, elixir-format

priv/repo/migrations/20190711042021_create_safe_jsonb_set.exs

@@ -9,7 +9,7 @@ defmodule Pleroma.Repo.Migrations.CreateSafeJsonbSet do
     begin
       result := jsonb_set(target, path, coalesce(new_value, 'null'::jsonb), create_missing);
       if result is NULL then
-        raise 'jsonb_set tried to wipe the object, please report this incindent to Pleroma bug tracker. https://git.pleroma.social/pleroma/pleroma/issues/new';
+        raise 'jsonb_set tried to wipe the object, please report this incident to Pleroma bug tracker. https://git.pleroma.social/pleroma/pleroma/issues/new';
         return target;
       else
         return result;

priv/repo/migrations/20201221202251_create_hashtags.exs

@@ -0,0 +1,13 @@
+defmodule Pleroma.Repo.Migrations.CreateHashtags do
+  use Ecto.Migration
+
+  def change do
+    create_if_not_exists table(:hashtags) do
+      add(:name, :citext, null: false)
+
+      timestamps()
+    end
+
+    create_if_not_exists(unique_index(:hashtags, [:name]))
+  end
+end

priv/repo/migrations/20201221202252_remove_data_from_hashtags.exs

@@ -0,0 +1,15 @@
+defmodule Pleroma.Repo.Migrations.RemoveDataFromHashtags do
+  use Ecto.Migration
+
+  def up do
+    alter table(:hashtags) do
+      remove_if_exists(:data, :map)
+    end
+  end
+
+  def down do
+    alter table(:hashtags) do
+      add_if_not_exists(:data, :map, default: %{})
+    end
+  end
+end

priv/repo/migrations/20201221203824_create_hashtags_objects.exs

@@ -0,0 +1,13 @@
+defmodule Pleroma.Repo.Migrations.CreateHashtagsObjects do
+  use Ecto.Migration
+
+  def change do
+    create_if_not_exists table(:hashtags_objects, primary_key: false) do
+      add(:hashtag_id, references(:hashtags), null: false, primary_key: true)
+      add(:object_id, references(:objects), null: false, primary_key: true)
+    end
+
+    # Note: PK index: "hashtags_objects_pkey" PRIMARY KEY, btree (hashtag_id, object_id)
+    create_if_not_exists(index(:hashtags_objects, [:object_id]))
+  end
+end

priv/repo/migrations/20210105195018_create_data_migrations.exs

@@ -0,0 +1,17 @@
+defmodule Pleroma.Repo.Migrations.CreateDataMigrations do
+  use Ecto.Migration
+
+  def change do
+    create_if_not_exists table(:data_migrations) do
+      add(:name, :string, null: false)
+      add(:state, :integer, default: 1)
+      add(:feature_lock, :boolean, default: false)
+      add(:params, :map, default: %{})
+      add(:data, :map, default: %{})
+
+      timestamps()
+    end
+
+    create_if_not_exists(unique_index(:data_migrations, [:name]))
+  end
+end

priv/repo/migrations/20210106183301_data_migration_create_populate_hashtags_table.exs

@@ -0,0 +1,16 @@
+defmodule Pleroma.Repo.Migrations.DataMigrationCreatePopulateHashtagsTable do
+  use Ecto.Migration
+
+  def up do
+    dt = NaiveDateTime.utc_now()
+
+    execute(
+      "INSERT INTO data_migrations(name, inserted_at, updated_at) " <>
+        "VALUES ('populate_hashtags_table', '#{dt}', '#{dt}') ON CONFLICT DO NOTHING;"
+    )
+  end
+
+  def down do
+    execute("DELETE FROM data_migrations WHERE name = 'populate_hashtags_table';")
+  end
+end

priv/repo/migrations/20210111172254_create_data_migration_failed_ids.exs

@@ -0,0 +1,14 @@
+defmodule Pleroma.Repo.Migrations.CreateDataMigrationFailedIds do
+  use Ecto.Migration
+
+  def change do
+    create_if_not_exists table(:data_migration_failed_ids, primary_key: false) do
+      add(:data_migration_id, references(:data_migrations), null: false, primary_key: true)
+      add(:record_id, :bigint, null: false, primary_key: true)
+    end
+
+    create_if_not_exists(
+      unique_index(:data_migration_failed_ids, [:data_migration_id, :record_id])
+    )
+  end
+end

priv/repo/migrations/20210121080964_add_default_text_search_config.exs

@@ -4,7 +4,7 @@ defmodule Pleroma.Repo.Migrations.AddDefaultTextSearchConfig do
   def change do
     execute("DO $$
     BEGIN
-    execute 'ALTER DATABASE '||current_database()||' SET default_text_search_config = ''english'' ';
+    execute 'ALTER DATABASE \"'||current_database()||'\" SET default_text_search_config = ''english'' ';
     END
     $$;")
   end

priv/repo/migrations/20210218223811_add_disclose_client_to_users.exs

@@ -0,0 +1,9 @@
+defmodule Pleroma.Repo.Migrations.AddDiscloseClientToUsers do
+  use Ecto.Migration
+
+  def change do
+    alter table(:users) do
+      add(:disclose_client, :boolean, default: true)
+    end
+  end
+end

priv/repo/migrations/20210222183840_remove_hashtags_objects_duplicate_index.exs

@@ -0,0 +1,11 @@
+defmodule Pleroma.Repo.Migrations.RemoveHashtagsObjectsDuplicateIndex do
+  use Ecto.Migration
+
+  @moduledoc "Removes `hashtags_objects_hashtag_id_object_id_index` index (duplicate of PK index)."
+
+  def up do
+    drop_if_exists(unique_index(:hashtags_objects, [:hashtag_id, :object_id]))
+  end
+
+  def down, do: nil
+end

priv/repo/migrations/20210222184616_change_hashtags_name_to_text.exs

@@ -0,0 +1,15 @@
+defmodule Pleroma.Repo.Migrations.ChangeHashtagsNameToText do
+  use Ecto.Migration
+
+  def up do
+    alter table(:hashtags) do
+      modify(:name, :text)
+    end
+  end
+
+  def down do
+    alter table(:hashtags) do
+      modify(:name, :citext)
+    end
+  end
+end

priv/static/index.html

@@ -1 +1 @@
-<!DOCTYPE html><html lang=en><head><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1,user-scalable=no"><!--server-generated-meta--><link rel=icon type=image/png href=/favicon.png><link href=/static/css/app.9a4c5ede37b2f0230836.css rel=stylesheet></head><body class=hidden><noscript>To use Pleroma, please enable JavaScript.</noscript><div id=app></div><script type=text/javascript src=/static/js/vendors~app.3b02e2e5bd8cdca42216.js></script><script type=text/javascript src=/static/js/app.c6b8a1c86149ed63e6ff.js></script></body></html>
+<!DOCTYPE html><html lang=en><head><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1,user-scalable=no"><!--server-generated-meta--><link rel=icon type=image/png href=/favicon.png><link href=/static/css/app.9a4c5ede37b2f0230836.css rel=stylesheet></head><body class=hidden><noscript>To use Pleroma, please enable JavaScript.</noscript><div id=app></div><script type=text/javascript src=/static/js/vendors~app.102a26590d3ba87dd908.js></script><script type=text/javascript src=/static/js/app.2e518bb8cf82de7a72b0.js></script></body></html>

priv/static/static/js/10.9a138b362edd4833778a.js.map

@@ -1 +1 @@
-{"version":3,"sources":[],"names":[],"mappings":"","file":"static/js/12.7d5889019e7177d19bc2.js","sourceRoot":""}
+{"version":3,"sources":[],"names":[],"mappings":"","file":"static/js/10.9a138b362edd4833778a.js","sourceRoot":""}

priv/static/static/js/11.9d88e9e710c4e0d0c897.js.map

@@ -1 +1 @@
-{"version":3,"sources":[],"names":[],"mappings":"","file":"static/js/10.8702741bef65422a8655.js","sourceRoot":""}
+{"version":3,"sources":[],"names":[],"mappings":"","file":"static/js/11.9d88e9e710c4e0d0c897.js","sourceRoot":""}