Activity

1. wardle commented on Oct 25, 2025

   

   wardle

   on Oct 25, 2025

   ContributorAuthor

   More actions

   Claude's research output:

   Support for Netty 4.2 AdaptivePoolingAllocator

   Summary

   lmdbjava 0.9.1's ByteBufProxy.PROXY_NETTY is incompatible with Netty 4.2's default buffer allocator. The current implementation requires PooledUnsafeDirectByteBuf, but Netty 4.2 changed the default allocator from pooled to adaptive, which creates AdaptivePoolingAllocator$AdaptiveByteBuf instead.

   Background

   Netty 4.2 Breaking Change

   Netty 4.2 introduced a significant change to the default buffer allocator:

   • Netty 4.1: Default is PooledByteBufAllocator → creates PooledUnsafeDirectByteBuf
   • Netty 4.2: Default is AdaptivePoolingAllocator → creates AdaptivePoolingAllocator$AdaptiveByteBuf

   From the Netty 4.2 Migration Guide:

   The adaptive memory allocator is now the default

   While users can revert via -Dio.netty.allocator.type=pooled, this defeats the purpose of upgrading to Netty 4.2 and is a global JVM setting.

   Current lmdbjava Implementation

   The current ByteBufProxy implementation (v0.9.1) hardcodes the buffer class name:

   private static final String NAME = "io.netty.buffer.PooledUnsafeDirectByteBuf";

   The allocator validates this class name and retries up to 10 times before throwing IllegalStateException if the wrong buffer type is returned.

   The Problem

   When using Netty 4.2 with default settings, ByteBufProxy.PROXY_NETTY fails because:

   1. Class name mismatch: Expects PooledUnsafeDirectByteBuf, gets AdaptivePoolingAllocator$AdaptiveByteBuf
   2. Field structure differences: The memory address field access pattern differs between the two buffer types

   Field Comparison

   Aspect	PooledUnsafeDirectByteBuf	AdaptiveByteBuf
   memoryAddress	Direct field	Computed via method
   length	Direct field ✓	Direct field ✓
   Access pattern	memoryAddress field	rootParent.memoryAddress() + startIndex

   PooledUnsafeDirectByteBuf:

   • Has memoryAddress as a direct field accessible via reflection
   • Simple offset access via Unsafe

   AdaptiveByteBuf:

   • Extends AbstractReferenceCountedByteBuf
   • Has length field ✓
   • Has rootParent field (AbstractByteBuf) - contains the actual buffer
   • Has startIndex field - offset within the chunk
   • memoryAddress() method computes: rootParent().memoryAddress() + startIndex

   Proposed Solution

   Add explicit support for both allocator types while maintaining backward compatibility:

   Three New Proxy Constants

   public final class ByteBufProxy extends BufferProxy<ByteBuf> {
       // New: Explicitly for pooled allocator (Netty 4.1 default, Netty 4.2 opt-in)
       public static final BufferProxy<ByteBuf> PROXY_NETTY_POOLED = new ByteBufProxyPooled();
   
       // New: Explicitly for adaptive allocator (Netty 4.2+ default)
       public static final BufferProxy<ByteBuf> PROXY_NETTY_ADAPTIVE = new ByteBufProxyAdaptive();
   
       // Existing: Now version-aware and delegates to appropriate implementation
       public static final BufferProxy<ByteBuf> PROXY_NETTY = createNettyProxy();
   }

   Implementation Strategy

   1. PROXY_NETTY_POOLED (Explicit Pooled)

   • Target class: io.netty.buffer.PooledUnsafeDirectByteBuf
   • Use existing implementation (current PROXY_NETTY logic)
   • Access memoryAddress field directly
   • Access length field directly

   Usage:

   // Explicit pooled allocator (any Netty version with pooled allocator)
   Env<ByteBuf> env = Env.create(ByteBufProxy.PROXY_NETTY_POOLED);

   2. PROXY_NETTY_ADAPTIVE (Explicit Adaptive)

   • Target class: io.netty.buffer.AdaptivePoolingAllocator$AdaptiveByteBuf
   • Access rootParent field (AbstractByteBuf)
   • Access startIndex field
   • Access length field
   • Compute effective memory address: rootParent.memoryAddress() + startIndex

   Usage:

   // Explicit adaptive allocator (Netty 4.2+ default)
   Env<ByteBuf> env = Env.create(ByteBufProxy.PROXY_NETTY_ADAPTIVE);

   3. PROXY_NETTY (Backward Compatible Version-Aware)

   Detects Netty version once at class initialization and delegates to the correct implementation:

   private static BufferProxy<ByteBuf> createNettyProxy() {
       // Check Netty version once
       String nettyVersion = getNettyVersion();
       int majorVersion = parseMajorVersion(nettyVersion);
       int minorVersion = parseMinorVersion(nettyVersion);
   
       // Netty 4.2+ uses adaptive by default
       if (majorVersion > 4 || (majorVersion == 4 && minorVersion >= 2)) {
           return PROXY_NETTY_ADAPTIVE;
       }
   
       // Netty 4.1 and earlier use pooled by default
       return PROXY_NETTY_POOLED;
   }

   Benefits:

   • Backward compatible: Existing code using PROXY_NETTY continues to work
   • Zero runtime overhead: Version check happens once at class load
   • Explicit control: Users can specify exact allocator if needed
   • Correct defaults: Automatically uses the right allocator for each Netty version

   Usage:

   // Backward compatible - automatically uses correct default for Netty version
   Env<ByteBuf> env = Env.create(ByteBufProxy.PROXY_NETTY);

   Impact

   Who is affected:

   • Any project using lmdbjava with Netty 4.2+
   • Projects wanting to benefit from Netty 4.2's improved memory management

   Current workarounds:

   1. Set -Dio.netty.allocator.type=pooled (not ideal - global setting, defeats upgrade purpose)
   2. Stay on Netty 4.1 (blocks adoption of Netty 4.2)
   3. Switch to ByteBufferProxy.PROXY_OPTIMAL (different performance characteristics)

   Migration Guide

   For Existing Code (No Changes Required)

   Existing code using PROXY_NETTY will continue to work:

   // This code works with both Netty 4.1 and 4.2
   Env<ByteBuf> env = Env.create(ByteBufProxy.PROXY_NETTY);

   The version-aware PROXY_NETTY automatically selects the correct implementation.

   For Explicit Control

   If you want explicit control over the allocator (e.g., using pooled on Netty 4.2):

   Before (Netty 4.1):

   Env<ByteBuf> env = Env.create(ByteBufProxy.PROXY_NETTY);

   After (Netty 4.2 with explicit pooled):

   // Option 1: Use explicit proxy constant
   Env<ByteBuf> env = Env.create(ByteBufProxy.PROXY_NETTY_POOLED);
   
   // Option 2: Set JVM property (global)
   // -Dio.netty.allocator.type=pooled
   Env<ByteBuf> env = Env.create(ByteBufProxy.PROXY_NETTY);

   After (Netty 4.2 with explicit adaptive):

   Env<ByteBuf> env = Env.create(ByteBufProxy.PROXY_NETTY_ADAPTIVE);

   Upgrading from Netty 4.1 to 4.2

   1. No lmdbjava code changes required - PROXY_NETTY adapts automatically
   2. Optional: Explicitly use PROXY_NETTY_ADAPTIVE to ensure adaptive allocator usage
   3. Performance testing recommended: Validate memory usage and throughput with your workload
   4. Fallback option: Use PROXY_NETTY_POOLED if adaptive allocator causes issues

   Implementation Notes

   Key Fields to Access in AdaptiveByteBuf

   From AdaptivePoolingAllocator.java:

   // Line 1398
   private AbstractByteBuf rootParent;
   
   // Line 1395
   private int startIndex;
   
   // Line 1399
   private int length;
   
   // Lines 1504-1507
   @Override
   public long memoryAddress() {
       ensureAccessible();
       return rootParent().memoryAddress() + startIndex;
   }

   Unsafe Operations

   The same Unsafe operations used for PooledUnsafeDirectByteBuf should work with AdaptiveByteBuf, as it delegates to an underlying direct buffer (rootParent) which has a native memory address.

   Testing Considerations

   The solution should be tested with:

   PROXY_NETTY_POOLED

   • Netty 4.1.x with default allocator
   • Netty 4.2.x with -Dio.netty.allocator.type=pooled
   • Verify PooledUnsafeDirectByteBuf is allocated and works correctly

   PROXY_NETTY_ADAPTIVE

   • Netty 4.2.x with default allocator (adaptive)
   • Verify AdaptivePoolingAllocator$AdaptiveByteBuf is allocated and works correctly
   • Test memory address calculation with rootParent + startIndex

   PROXY_NETTY (Version-Aware)

   • Netty 4.1.x → should delegate to PROXY_NETTY_POOLED
   • Netty 4.2.x → should delegate to PROXY_NETTY_ADAPTIVE
   • Verify version detection works correctly
   • Confirm zero runtime overhead (detection happens once)

   General Tests (All Proxies)

   • Multi-threaded scenarios
   • Buffer reuse and recycling scenarios
   • Large data operations
   • Transaction commit/rollback behavior
   • Memory leak detection

   References

   • Netty 4.2.0.Final Release
   • Netty 4.2 Migration Guide
   • AdaptivePoolingAllocator Source
   • lmdbjava Buffers Wiki

   Related Issues

   • Netty 4 IndexOutOfBoundsException due to writer index exceeding buffer capacity #113 - Previous Netty compatibility issue (resolved for 4.1.30+)
   • Netty #14269 - AdaptiveByteBufAllocator memory leak discussions
   • Netty #15292 - Performance issues with AdaptiveByteBufAllocator

   ❤️React with ❤️1Reacted by Andrej Fink
2. bp-alex commented on Oct 25, 2025

   

   bp-alex

   on Oct 25, 2025

   Member

   More actions

   @wardle The main challenge with Netty support in LmdbJava is that I don't use Netty, didn't write the original integration, and lack familiarity with it more broadly. Unfortunately, no one has volunteered to maintain it either, which is why issues like #215 have been open for 2.5 years.

   Given these constraints, I think it makes sense to minimize complexity by supporting just a single Netty buffer type and focusing on getting that implementation solid. From what I understand, adaptive buffers seem to be the preferred approach now. Does that match your understanding? If you'd be able to contribute a PR for this, that would be fantastic. And if anyone else is working with Netty and has thoughts on the best path forward, I'd of course welcome their thoughts too.
3. wardle commented on Oct 25, 2025

   

   wardle

   on Oct 25, 2025

   ContributorAuthor

   More actions

   Unfortunately not. Here is a summary of some of the documentation:

   In Netty 4.1 the adaptive memory allocator was experimental. We have now made it the new default instead of the pooled allocator.

   We believe that most workloads will observe reduced memory usage, with performance that is on par or slightly better than the pooled allocator. The adaptive allocator automatically tunes itself to perform well for the observed workload. It was also designed from the start to work well with virtual threads.

   However, every application use their allocator in different ways, and stress them differently, so it is expected that the adaptive allocator will perform worse for some. Integrators who wish to keep using the pooled allocator like they did in Netty 4.1, can change the default allocator by setting this system property:

   io.netty.allocator.type=pooled

   From my understanding, and reading through some of the issues raised against the project, there is uncertainty as to whether there are always performance gains. My assumption is that during wider usage of 4.2 with a variety of workloads, this may become clearer. As such, it appears that multiple variants of allocators are here to stay. Hence the design outlined above that provides a reasonable default, but permits explicit override when the client knows better than the library, which to be fair is often the case.

   An entirely reasonable option would be to wait and stick with 4.1 for the time being, and understanding of the balances might become clearer. The problem with a wholesale move would be breaking backwards compatibility such that there may be clients who remain on an older version of netty for whatever reason.

   Clearly this has a potential interaction with the planned removal of Unsafe (#259) and a potential move to the more modern Java FFI. That said, lmdbjava currently benefits from a 'pluggable' buffer implementation, and currently its unclear to me why one might not have a continued ability to switch between different implementations depending on use-case; certainly any such design would a) fit into the current model, and b) permit users to make that decision regarding trading safety for speed. I guess my thinking was that 'netty-adaptive' is just another allocator along with all of the others.
4. bp-alex commented on Oct 25, 2025

   

   bp-alex

   on Oct 25, 2025

   Member

   More actions

   Thanks @wardle for the additional information.

   #259 is not intended to remove the pluggable buffers. They will be around for a long time yet. My initial reaction was more reflecting the challenges of supporting Netty to date and not wanting to exacerbate it by further edge cases emerging with additional Netty buffer variants. I hope someone will volunteer to maintain the Netty integration.

   For the time being I'll leave this ticket so others have an opportunity to comment.

   👍React with 👍2Reacted by Mark Wardle and Andrej Fink
5. bernardladenthin commented on Jul 24, 2026

   

   bernardladenthin

   on Jul 24, 2026

   More actions

   Cross-linking: #290 fixes the related Netty issue #215 (ByteBuf.nioBuffer() returning zeros) and touches the same ByteBufProxy read path this ticket is about. Worth keeping the two together — whatever the adaptive out() ends up doing will also determine its nioBuffer() behaviour, so #261 and #215 are effectively the same corner of the code.
6. bernardladenthin commented on Jul 24, 2026

   

   bernardladenthin

   on Jul 24, 2026

   More actions

   I spent some time trying to reproduce this on Netty 4.2.0 and want to share what I found, because part of the pasted writeup doesn't hold up — and there is a real, fixable issue underneath, just with a different cause.

   The stated mechanism appears to be disproven

   The writeup says "ByteBufProxy.PROXY_NETTY fails … allocate() expects PooledUnsafeDirectByteBuf, gets AdaptivePoolingAllocator$AdaptiveByteBuf." I can't reproduce that, and it looks structurally impossible:

   • ByteBufProxy.allocate() calls nettyAllocator.directBuffer(), and nettyAllocator is typed PooledByteBufAllocator (and PROXY_NETTY uses PooledByteBufAllocator.DEFAULT). A PooledByteBufAllocator never returns an AdaptiveByteBuf.
   • Empirically, on 4.2.0 with adaptive confirmed as the default (ByteBufAllocator.DEFAULT → AdaptiveByteBuf), PooledByteBufAllocator.DEFAULT still returns PooledUnsafeDirectByteBuf, PROXY_NETTY constructs fine, and the full test suite passes on 4.2 (918/918), including put/get round-trips with adaptive user buffers.

   (Minor: the writeup names the offset field startIndex; in 4.2.0 it's actually adjustment.)

   The real, reproducible failure

   ByteBufProxy.in() calls buffer.memoryAddress() unconditionally, which throws for any buffer with hasMemoryAddress()==false — i.e. any heap buffer:

   java.lang.UnsupportedOperationException
     at io.netty.buffer.PooledHeapByteBuf.memoryAddress(...)
     at org.lmdbjava.ByteBufProxy.in(ByteBufProxy.java:201)
     at org.lmdbjava.Dbi.put(...)
   

   Netty 4.2's adaptive allocator can hand back heap-backed AdaptiveByteBuf (e.g. heapBuffer()), so 4.2 apps hit this — but it's not 4.2-specific: a plain PooledHeapByteBuf fails identically on 4.1. That likely also explains why -Dio.netty.allocator.type=pooled "fixed" it for you: it steers your buffers back to direct.

   #291 fixes this by copying address-less buffers into native scratch in in() (the same approach ByteArrayProxy already uses), keeping the zero-copy fast path for direct buffers.

   Could you share your actual failure?

   Since the "class name mismatch" mechanism doesn't reproduce here, I'd like to make sure I'm not missing a different failure you saw. Do you have the actual stack trace / a minimal repro from your codebase without the property set? In particular it'd help to know whether your buffers were heap or direct, and your JDK + whether sun.misc.Unsafe was available (there's a separate, unconfirmed path where allocate() rejects a non-unsafe PooledDirectByteBuf if unsafe is disabled — that would be a genuine "class name mismatch," but about unsafe, not the adaptive allocator).

   Separately: the full suite passing on 4.2 suggests the pinned Netty 4.1.118 could likely be bumped after the "further testing" the pom mentions — happy to open that separately if useful.