msgpack-ruby JRuby decoder makes uncapped header-count allocations; a 5-byte message forces multi-GiB OOM

Summary

The JRuby implementation of msgpack (msgpack gem, java platform — pure Java decoder under ext/java/) sizes containers directly from the element count written in the message header, with no cap at all: no equivalent of the C unpacker's SHRT_MAX clamp (since 1.7.3) and none of the buffered-bytes cap that commit c8d42d8 added for CRuby in 1.8.6. A single 5-byte message decides how much memory a JVM process tries to allocate:

• array32 header declaring 0x7FFFFFFF elements → new IRubyObject[0x7FFFFFFF] (~8 GiB of reference slots) attempted immediately → java.lang.OutOfMemoryError: Requested array size exceeds VM limit.
• Counts below the VM's max-array-size but above available heap (e.g. 5×10⁸ with a 512 MiB heap) produce a real allocation attempt — OutOfMemoryError: Java heap space — the JVM genuinely tries to take the memory before failing.
• str32/bin32 headers declaring 0x7FFFFFFF bytes do the same via readBytes(size) → new byte[size].
• Below the failure threshold there is a silent band: one streaming unpacker fed only an array32 header with count 2×10⁷ (5 wire bytes, zero elements) grows used heap +78 MiB immediately, no exception (each swallows the underflow), and the footprint stays elevated after Unpacker#reset + full GC. Linear in declared count (~4 B/element); nothing bounds it — not delivered bytes, not a per-container cap.

Only counts ≥ 2³¹ are refused (a clean MessagePack::UnderflowError from getUnsignedInt()), and only maps are safe (consumeHash grows per element instead of pre-sizing). Verified on the released gem msgpack 1.8.6 java, JRuby 9.4.12 / OpenJDK 25, 2026-10-30 — i.e. unfixed in the current release. Proposed fix mirrors c8d42d8: cap initial allocation by bytes actually buffered and grow as data arrives; read strings in bounded chunks instead of allocating the declared length up front.

Details

1. The four dispatch arms, and which pre-allocate (line numbers at tag v1.8.6 — current release; ext/java has no commits after it)

The only guard is getUnsignedInt() :88-94: counts with the top bit set (≥ 2³¹) raise MessagePack::UnderflowError: Size too large (limited to 2**31 for the Java version). That bounds the value but leaves the entire positive band (available heap, 2³¹) unbounded — and that band is where real allocation attempts happen.

2. Contrast with the C implementation (same repo)

The CRuby unpacker has had a per-container cap since 1.7.3 (INITIAL_BUFFER_CAPACITY_MAX = SHRT_MAX) and, since c8d42d8 in 1.8.6, additionally caps initial size by bytes actually buffered ("an element needs at least one byte…"). Strings on C are bounded too (read_raw_body_cont allocates min(remaining, buffered)). The Java decoder has neither generation of cap — it is strictly behind the CRuby behaviour this repo already considers correct enough to ship.

3. Silent retention + mechanism control (measured 2026-10-30, released gem)

One MessagePack::Unpacker per process, fed only a header ("\xdd" + [count].pack('N')), zero elements; heap = Runtime.total_memory - free_memory after GC.start on both sides:

Mechanism control: after the feed, Unpacker#reset + three GC.start cycles left used heap at +79.4 MiB above baseline (2×10⁷ arm) — the elevation is not unpacker-held state; it is JVM heap already spent on header-declared allocation and not returned. Either way the process footprint grows per attack and stays grown; the peak cost is paid synchronously during parsing regardless of GC behaviour afterwards. (Honest scope note: this is unbounded allocation, not a memory leak.)

4. Why JVM makes availability impact worse than the CRuby analogue

In CRuby, allocation failure raises NoMemoryError/fails inside one worker process. In a JVM, an OOME raised on the parser thread can land anywhere in the host application's shared runtime — long-lived JRuby servers (Rails, Logstash-class pipelines, JDBC apps) share heap across all threads and connections; a single crafted message can destabilize or kill the whole JVM, not just the request. The silent band means no exception at all is even required to spend memory.

5. Not claimed / scope

• Fluentd is NOT affected (and this report deliberately does not involve it): Fluentd ships CRuby-only — CI pins engine: cruby, gemspec requires ruby >= 3.3 while JRuby 9.4 implements Ruby 3.1 semantics, and there are zero JRUBY guards in lib/. This is an upstream msgpack-ruby report only.
• Maps (map32) do not pre-allocate — measured; the class applies to array32/str32/bin32.
• Counts ≥ 2³¹ are cleanly refused — measured; no claim there.
• Older java-gem releases were not individually probed; <= 1.8.6 means "every release up to the current one carries this code".

Proof of concept

Verified on the released gem msgpack 1.8.6 java, JRuby 9.4.12.0, OpenJDK 25.0.4.1, Linux x86-64 (artifacts in evidence_186_recheck/). Always run with a small -Xmx so oversized requests fail fast instead of wrecking the box.

Setup

$ curl -fsSLO https://repo1.maven.org/maven2/org/jruby/jruby-dist/9.4.12.0/jruby-dist-9.4.12.0-bin.tar.gz
$ tar xzf jruby-dist-9.4.12.0-bin.tar.gz && export PATH=$PWD/jruby-9.4.12.0/bin:$PATH
$ jruby -S gem install msgpack --no-document      # resolves the java-platform gem: "msgpack (1.8.6 java)"

1. Header shapes (fresh process per arm, -Xmx512m)

array32 count=0xFFFFFFFF       -> MessagePack::UnderflowError: Size too large (limited to 2**31 for the Java version)
array32 count=0x7FFFFFFF       -> Java::JavaLang::OutOfMemoryError: Requested array size exceeds VM limit
array32 count=500000000        -> Java::JavaLang::OutOfMemoryError: Java heap space   # real allocation attempt
str32   len=0x7FFFFFFF         -> Java::JavaLang::OutOfMemoryError: Requested array size exceeds VM limit
bin32   len=0x7FFFFFFF         -> Java::JavaLang::OutOfMemoryError: Requested array size exceeds VM limit
map32   count=0x7FFFFFFF       -> no exception, no large allocation (consumeHash grows per element)

Each payload is five bytes, e.g. "\xdd" + [count].pack('N') fed to MessagePack::Unpacker#feed_each. Scripts: jr_probe.rb, jr_probe_extra.rb (attached in kit).

2. Silent band — retention from one header (Xmx1g)

$ for n in 1000000 5000000 20000000; do jruby -J-Xmx1g jr_one.rb $n; done
count=  1000000 heap 26.4 -> 31.1 MiB (delta 4.7)   | no exception (underflow swallowed)
count=  5000000 heap 53.3 -> 74.0 MiB (delta 20.7)  | no exception (underflow swallowed)
count= 20000000 heap 51.3 -> 129.3 MiB (delta 78.0) | no exception (underflow swallowed)

$ jruby -J-Xmx1g jr_reset_control.rb 20000000
count=20000000 heap base=49.9 after_feed=129.3 (delta 79.4) after_reset_gc=129.3 (retained_vs_base 79.4)

A streaming server that keeps one unpacker per connection (the pattern any socket-fed msgpack service uses) pays this per connection with five wire bytes each and produces no exception, no log line.

3. Fix verification path

Apply the C-side idea from c8d42d8: in consumeArray, cap the initial array by bytes actually buffered (an element needs ≥ 1 byte) and grow as elements arrive; in readBytes/consumeString, read in bounded chunks instead of allocating the declared length up front. Then re-run this PoC: the silent band should track delivered bytes, and oversized declarations should raise UnderflowError-class errors without touching multi-GiB allocations. msgpack-ruby CI runs jruby-9.4/truffleruby, so a patch is verifiable as-is; I can prepare one against ext/java/org/msgpack/jruby/Decoder.java.

Impact

Type. Uncontrolled resource consumption driven by declared sizes rather than delivered bytes — CWE-770 (Allocation of Resources Without Limits or Throttling), precisely CWE-1325 (Independent Resource Usage Limiting Not Applied). Availability only; no confidentiality/integrity impact, no code execution.

Who is impacted. Any JRuby application that feeds untrusted msgpack to MessagePack.unpack/Unpacker — the java-platform gem's whole user base (JRuby-based services, log/RPC pipelines, JVM polyglot stacks using this gem). For them: remote OOM of a shared-heap JVM from a 5-byte message, or silent heap spend below the exception band.

Severity. Proposed High / 7.5, CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H — remote, trivially crafted, default settings, process death; same metric string as the accepted fluentd OOM family (e.g. GHSA-h3xv-5jpx-r4j2) and as this repo's own C-side analogue (reported separately, patched at 1.8.6). If triage weighs the platform scope (java gem only), lower framings read against that — we flag it honestly; the maintainers' call on scope shouldn't delay the patch.

Relationship to the C-side report. Same flaw class, different implementation and fix state: CRuby's pre-allocation was capped by c8d42d8 (released 1.8.6, separate advisory with affected <= 1.8.5, patched 1.8.6). This JRuby variant is unfixed in 1.8.6 and needs its own patch + CVE. Cross-reference both ways once filed.

What this is not. Not a Fluentd issue (Fluentd is CRuby-only — CI pins cruby, gemspec ruby >= 3.3 vs JRuby's Ruby 3.1 semantics); no fluentd advisory should carry this. Not a memory leak (measured: footprint elevation persists after reset+GC without retained objects). Not exploitable via maps or counts ≥ 2³¹ (both measured safe/cleanly refused).