After a redirect the 429/503 came from the final host, but the backoff
was recorded for the requested one. X-RateLimit-Reset is also accepted
as a Unix timestamp or as seconds.
With 5 attempts and Oban's short retry delays, an incoming activity from
an unknown actor on a backed-off host was discarded without a request.
ReceiverWorker and SignatureRetryWorker now snooze for up to 2 hours.
Also lowercase backoff host keys and log backed-off user fetches at debug.
Object fetches and WebFinger stop sending requests to a host until
Retry-After or X-RateLimit-Reset (default 5 minutes, max 1 hour).
Ported from Akkoma's Pleroma.HTTP.Backoff.
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Co-authored-by: Oneric <oneric@oneric.stub>
Hackney 4.9 enforces max_body while it reads and returns
{:error, {:closed, partial}} for a truncated body; the hackney
regression tests now pin that. The certifi override goes, as
hackney itself now asks for certifi ~> 2.18.0.
[:http, :adapter, :ssl_options] is Hackney's key, and AdminFE sets it.
Gun reads tls_opts, so with Gun as the default these settings were lost
without a warning. Use them as tls_opts unless tls_opts is set.
Hackney 4 reads every response completely before Pleroma can check its
size, so max_body no longer limits downloads or memory there; Gun stops
at the limit. Document the Hackney limitation for those who switch back.
Hackney 4 negotiates HTTP/2 and shares one connection per host. The
media proxy closes a response with :hackney.close/1, which stops the
whole connection, so parallel downloads from one host broke each other.
ConfigDB turns a numeric proxy host into an IP tuple. Hackney 1 took it,
Hackney 4 ignores such a proxy and connects directly, which skips a
SOCKS proxy (Tor) without any error.
- An admin-set cacerts no longer gets the CAStore cacertfile added;
Hackney drops cacerts when both are set.
- Set proxy_ssl_options too: with an HTTPS proxy, Hackney 4 uses them
for the proxy connection and falls back to certifi.
- Hackney 4 ignores the top-level insecure option and uses certifi.
Turn it into an explicit verify_none.
The Qdrant and OpenAI clients do not go through Pleroma.HTTP, so on
Hackney they trusted certifi. Both send API keys. Give them the
configured adapter with the same TLS options.
Hackney 4 merges given ssl_options into its defaults and keeps its own
partial_chain, which trusts any certificate in the peer chain that is in
certifi. A peer could send the emSign ECC Root CA - G3 root in its chain
and pass. Set partial_chain so certifi is never a trust anchor.
Hackney trusts its own certifi bundle, which still has the roots of the
compromised eMudhra intermediate CAs. Pass explicit ssl_options with the
CAStore bundle. Hackney drops all its TLS defaults when ssl_options are
given, so keep peer and hostname checks in the options.
Serve a response announcing far more than it sends, then close, and
assert the connection process is gone. Fails against stock 4.7.2 with a
conn parked in `closed` holding the bytes it read; passes against the
pinned fix.
Also stop the Hackney tests trusting proxy environment variables.
`no_proxy_env` is not a Hackney option and never was, so a proxy in the
build environment silently turned these into tunnelled requests, which
are never pooled -- exactly the shape several of them mean to exclude.
`proxy: false` is what Hackney actually reads.
That surfaced a matching bug in the reverse proxy client, which treated
`proxy: false` as "a proxy is set" and so declined to mark a pooled
bodyless response as already released.
Hackney 4 ignores `with_body` and always returns a fully read binary, so
Tesla.Adapter.Hackney never reaches the clause that honours :max_body and
the option became a no-op. Only the gun path still enforced it, through
Pleroma.HTTP.Adapter.Gun. Pleroma.Web.RelMe relies on nothing else, so it
was fetching remote pages unbounded.
Add a middleware that applies the limit for every adapter: reject when
content-length already announces too much, and reject a body that came
back over the limit. Streaming bodies are left alone -- neither adapter
limits those and the caller owns the lease, so failing here on the header
would strand a connection RichMedia is still expected to release.
On the hackney path this bounds what callers are handed rather than peak
memory, since the adapter has read the body before any middleware runs.
Conflict resolution notes:
- mix.lock: kept this branch's versions for cowlib, h2, hackney and hpax.
Those upgrades are the point of the branch; develop is still on hackney
1.20.1.
- .woodpecker/docker-armv7.yaml: accepted develop's deletion. It only
existed here to build mdex_native on armv7, and mdex was already
replaced with earmark again.
- lib/pleroma/reverse_proxy.ex: took develop's version. It independently
grew request_with_constraints/5 and the invalid-status close, which
supersede the equivalent fixes on this branch. Re-applied this branch's
recv_timeout seeding, which develop does not have.
- lib/pleroma/gun/connection_pool/worker.ex and
lib/pleroma/reverse_proxy/client/tesla.ex: took develop's versions. The
gun pool rework (#7934) already guards the demonitor and makes
release_conn/cancel_stream exit-safe, and it splits the Tesla client's
graceful release from its abort path, so this branch's changes there are
obsolete.
That leaves the client leak fix as a Hackney-only concern, which
stream_body/1 now handles the same way the gun client does.
Three paths obtained a client ref and abandoned it without closing:
the content-length rejection in call/3, the invalid-status clause in
request/4, and the success branch of response/6.
Under Hackney 4 an abandoned ref is a permanent process leak. A conn
that finishes or errors without an explicit stop parks in `closed` or
`connected` and never terminates: closed/3 only arms its stop timer for
pooled conns, `connected` gets no state_timeout when idle_timeout is
infinity, and a non-pooled conn monitors hackney_conn_sup rather than
its caller, so the owner-DOWN cleanup never fires. Each parked process
pins every refc binary it read, which for proxied video is tens of MB.
Releasing on every outcome means the gun client must stop releasing in
stream_body/1, or the ref would be removed twice.
Hackney does reclaim a pooled connection itself for a response that
cannot carry a body (204, 304, HEAD), so those refs are marked and left
alone; stopping one would evict a connection another request may
already hold. The mark is gated on the request actually being pooled,
since a direct or proxy-tunnelled conn is never checked in and leaks
unless we stop it.
Exercises Pleroma.ReverseProxy.Client.Hackney with follow_redirect enabled behind an HTTPS CONNECT proxy, ensuring the client follows a relative redirect and can stream the final body.
Hackney 1.25 crashes when follow_redirect is enabled behind an HTTPS CONNECT proxy and the Location header is relative (hackney_http_connect transport).
This test demonstrates the failure and verifies Tesla-level redirects work when hackney redirects are disabled.
Reproduces the Hackney 1.25 pooled redirect cleanup issue which can surface as :req_not_found when the adapter returns a Ref and the body is later fetched.
Hackney 1.25.x has redirect handling issues behind CONNECT proxies and with pools.
Disable hackney-level redirects and rely on Tesla.Middleware.FollowRedirects instead.
Also default to with_body: true so redirects can be followed reliably.
