Allow plugin support for custom compression/decompression (e.g. zstd)

[pjfanning]

Feature request: Plug-in support for custom compression (e.g., zstd) in Pekko HTTP coding subsystem

Context

Currently, Pekko HTTP provides built-in support for gzip (and deflate) compression via the Coder abstraction in org.apache.pekko.http.scaladsl.coding. This is used via encodeResponse, decodeRequest, and related directives, and the negotiation machinery is well-established. There is frequent interest (e.g., for modern formats like zstd) in supporting custom content encoding algorithms. Users may want to plug in their own encoders/decoders without modifying Pekko HTTP core code.

There is #860 which is about adding built-in support for zstd but it might be worth considering allowing the addition of custom compression/decompression so users can add support themselves.

Analysis

After reviewing the codebase (scaladsl/coding), here's what enables and what currently blocks custom compressions:

What works:

• Coder is a marker trait (trait Coder extends Encoder with Decoder). Users can implement it today to support new encodings, e.g., zstd.
• HttpEncoding.custom(...) allows defining new encoding tokens, such as zstd.
• Directives (encodeResponseWith, decodeRequestWith) accept arbitrary Encoder/Decoders, so user-provided implementations can already be injected explicitly per-route.

Current barriers:

1. Compressor and StreamDecoder are Internal/Deprecated:
   • Compressor (needed for encoding) and StreamDecoder (for decoding) are @InternalApi and @deprecated. Implementing Encoder/Decoder needs usage of these, but they're not public API and may change or disappear, so user-providers can't rely on them.
2. Java API is closed:
   • The Java-side equivalent (org.apache.pekko.http.javadsl.coding.Coder) is a closed enum—Java users cannot add custom encoders at all.
3. Default coders are fixed:
   • Coders.DefaultCoders is always [Gzip, Deflate, NoCoding]. There is no mechanism for registering a custom coder with the negotiation machinery globally. So, while you can use your custom coder per-route with the With directives, it won't be automatically used with default directives.

Example: How it could work (after stabilising APIs)

object ZstdCoder extends Coder with StreamDecoder {
  val encoding: HttpEncoding = HttpEncoding.custom("zstd")
  def messageFilter = Encoder.DefaultFilter
  private[this] def newCompressor = new ZstdCompressor() // if Compressor is public
  def newDecompressorStage(maxBytesPerChunk: Int) = () => new ZstdDecompressorStage(maxBytesPerChunk)
}
// Usage:
encodeResponseWith(ZstdCoder, Coders.Gzip, Coders.Deflate, Coders.NoCoding) { ... }
decodeRequestWith(ZstdCoder, Coders.Gzip, Coders.Deflate) { ... }

Proposed Enhancements

1. Promote Compressor and StreamDecoder to Public API:
   • Remove @InternalApi and @deprecated from these types
   • Alternatively, provide a new public API for plugging in custom encoder/decoder flows
2. Open up Java API for Custom Coders:
   • Replace or supplement the hardcoded Coder enum with an interface/factory-based mechanism in the Java DSL
3. Make DefaultCoders Configurable or Injectable:
   • Allow applications to register additional coders (e.g., via ActorSystem config or a registration method) for use by the default directives/negotiator
4. (Optional) Add zstd as a built-in or contrib module:
   • To showcase extensibility and provide instant value
5. (Docs) Add a guide for implementing and registering custom coders

Impact

• This would unlock modern and/or domain-specific compression for Pekko HTTP users
• Immediate benefit for projects that need zstd or other algorithms (Brotli, LZ4, etc.)
• Makes Pekko HTTP encoding/decoding as extensible as its routing layers

If maintainers agree, I'm happy to help with API stabilisation PRs or further design discussion!

---

Origin: https://github.com/apache/pekko-http (discussion/analysis by pjfanning)