The result of Oban jobs determine the reachability status.
Publisher jobs will cancel themselves at execution time if the target server is now unreachable.
Receiving activities does not immediately mark a server as reachable, but creates a ReachabilityWorker job to validate.
A Cron will execute daily to test all unreachable servers.
An instance block with authenticated fetch being required can cause this as we couldn't get the user to find their public key to verify the signature. Commonly observed if someone boosts/Announces a post from an instance that blocked you.
This logic only exists in the Plug, so attempting to validate the signature by calling the library function HTTPSignature.validate_conn/2 directly will never work because we do not attempt to construct the (request-target) and @request-target headers with both the commonly misinterpreted and correct implementation of this field. Therefore all attempts to validate a signature from an Oban Job will fail.
When signatures fail on incoming activities we put the job into Oban to be processed later instead of doing the user fetching and validation inline which is expensive and increases latency on the incoming POST request. Unfortunately we did not retain the :method, :request_path, and :query_string parameters from the conn so the signature validation and Oban Job would always fail.
This was most obvious when Mastodon sends Deletes for users your server has never seen before.
Rework inbound federation to accept requests optimistically. The HTTP Signatures Plug will not attempt to fetch the actor or key and will fail early.
If the signature cannot be validated we pass the required data into the Oban job with a reduced priority and increase the timeout to 20 seconds. The Oban job will handle the actor and key fetching before attempting to validate the activity again. This job will be retried 5 times by default.
Another welcome side effect is that actors who change their keys can federate to Pleroma instances immediately instead of needing to wait the default value of 86400s / 24 hours before the key will be fetched again.
