robots.txt

Add default robots.txt that allows bots access to all paths.
Add mix task to generate robots.txt taht allows bots access to no paths.
Document custom emojis, MRF and static_dir
static_dir documentation includes docs for the robots.txt Mix task.
This commit is contained in:
William Pearson 2019-01-20 01:44:00 +00:00
commit 3dadaa4432
7 changed files with 191 additions and 2 deletions

View file

@ -21,7 +21,8 @@ defmodule Pleroma.Plugs.InstanceStatic do
end
end
@only ~w(index.html static emoji packs sounds images instance favicon.png sw.js sw-pleroma.js)
@only ~w(index.html robots.txt static emoji packs sounds images instance favicon.png sw.js
sw-pleroma.js)
def init(opts) do
opts

View file

@ -25,7 +25,7 @@ defmodule Pleroma.Web.Endpoint do
at: "/",
from: :pleroma,
only:
~w(index.html static finmoji emoji packs sounds images instance sw.js sw-pleroma.js favicon.png schemas doc)
~w(index.html robots.txt static finmoji emoji packs sounds images instance sw.js sw-pleroma.js favicon.png schemas doc)
# credo:disable-for-previous-line Credo.Check.Readability.MaxLineLength
)