hj/pleroma
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Pleroma.Upload.Filter.Dedupe: sharding directory structure

Browse source 
Dedupe now uses a three-level sharding directory structure to improve performance when many files are uploaded and stored on a filesystem instead of an object store. (note: Minio still affected as it still uses a traditional filesystem)

This does not help if you already have hundreds of thousands of files uploaded. The media URLs are permanently part of the activity so the files cannot be relocated. A motivated user could write a tool to move the files and perhaps write an Nginx or equivalent redirect to make the files still accessible, but that is beyond the scope of this change.
This commit is contained in:
Mark Felder 2024-10-29 16:00:18 -04:00
commit d2de251c4d
5 changed files with 21 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

10
lib/pleroma/upload/filter/dedupe.ex
View file

@ -17,8 +17,16 @@ defmodule Pleroma.Upload.Filter.Dedupe do
|> Base.encode16(case: :lower)
filename = shasum <> "." <> extension
{:ok, :filtered, %Upload{upload | id: shasum, path: filename}}
{:ok, :filtered, %Upload{upload | id: shasum, path: shard_path(filename)}}
end
def filter(_), do: {:ok, :noop}
@spec shard_path(String.t()) :: String.t()
def shard_path(
<<a::binary-size(2), b::binary-size(2), c::binary-size(2), _::binary>> = filename
) do
Path.join([a, b, c, filename])
end
end