Opened 7 years ago

Closed 7 years ago

#679 closed defect (fixed)

Something somewhere is using DirectByteBuffers the wrong way

Reported by: zab Owned by:
Priority: minor Milestone: 0.9.2
Component: apps/console Version: 0.9.1
Keywords: Cc:
Parent Tickets:

Description

KYTV has been tracking heap usage periodically here http://pastethis.i2p/show/1586/ and something is creating DirectByteBuffers? which end up garbage collected.

The problem is, they don't always clean up properly. The java object on the heap gets cleaned up fine, but the native memory doesn't always get free()-ed and the jvm process ends up leaking. The Right Way (TM) to use direct buffers is for long-lived buffers that are either pooled or will live for as long as the jvm lives.

This could be happening anywhere in the router or apps; I'm adding it as a TODO so I don't forget it.

Subtickets

Attachments (1)

DirectByteBuffer-YourKit.png (56.8 KB) - added by killyourtv 7 years ago.
DirectBuffer? Profiling with YourKit?

Download all attachments as: .zip

Change History (8)

comment:1 Changed 7 years ago by zzz

Jetty: http://docs.codehaus.org/display/JETTY/Configuring+Connectors

See also http://jira.codehaus.org/browse/JETTY-1245 , not clear if that affects Jetty 6.x or not. Or non-SSL. We don't use the SslSelectChannelConnector? by default.

See also comments in Jetty source (grep for DirectBuffers?) which imply JVM bugs.

See also http://jira.codehaus.org/browse/JETTY-102

comment:2 Changed 7 years ago by zab

Found some usages in SAM - a new direct buffer per session, no evidence this is causing KYTV's numbers yet

grep -r allocateDirect apps/
apps/sam/java/src/net/i2p/sam/SAMStreamSession.java:            ByteBuffer data = ByteBuffer.allocateDirect(SOCKET_HANDLER_BUF_SIZE);
apps/sam/java/src/net/i2p/sam/SAMv2StreamSession.java:					ByteBuffer data = ByteBuffer.allocateDirect(SOCKET_HANDLER_BUF_SIZE);
apps/sam/java/src/net/i2p/sam/SAMv3Handler.java:				ByteBuffer inBuf = ByteBuffer.allocateDirect(SAMRawSession.RAW_SIZE_MAX+1024);

comment:3 Changed 7 years ago by zzz

Yes, SAM uses direct buffers, not configurable. Didn't know that. SAM is not enabled by default on /configclients.

Here's how to disable Jetty's use of direct buffers in the console and eepsite. For your eepsite, edit the XML files in ~/.i2p/eepsite, not installer/resources. As kytv indicated he's not using Jetty for his eepsites or SAM, his direct buffer usage is independent of router traffic, so anybody should be able to test.

With this patch I get about 30 direct buffers listed in jvisualvm shortly after startup, pretty constant. I didn't see any usage in the wrapper in a quick grep. Perhaps the remaining ones are part of the sockets to jvisualvm. Or file accesses in some lib.

#
# old_revision [6014a9321bb2362ffc628a351c1db19922384f76]
#
# patch "apps/routerconsole/java/src/net/i2p/router/web/RouterConsoleRunner.java"
#  from [9eaf558692d4a306e03136d644f0a2cebd6a335f]
#    to [c001d36d5ddb3ff6bcdf967a86629711cb5d2df5]
# 
# patch "installer/resources/eepsite/jetty-ssl.xml"
#  from [78d4d498556b65ce19da958f8eeac2a5e7685a17]
#    to [234958b6981fda95688f98e1aa2b708e322c7ac2]
# 
# patch "installer/resources/eepsite/jetty.xml"
#  from [b9be258673a674f019305afb9c7561e98c44d8ef]
#    to [d561d58072300e27b5b2e31eb478fe828b817622]
#
============================================================
--- apps/routerconsole/java/src/net/i2p/router/web/RouterConsoleRunner.java	9eaf558692d4a306e03136d644f0a2cebd6a335f
+++ apps/routerconsole/java/src/net/i2p/router/web/RouterConsoleRunner.java	c001d36d5ddb3ff6bcdf967a86629711cb5d2df5
@@ -339,6 +339,7 @@ public class RouterConsoleRunner {
                         lsnr.setPort(lport);
                         lsnr.setMaxIdleTime(90*1000);  // default 10 sec
                         lsnr.setName("ConsoleSocket");   // all with same name will use the same thread pool
+                        lsnr.setUseDirectBuffers(false);  // default true seems to be leaky
                         //_server.addConnector(lsnr);
                         connectors.add(lsnr);
                         boundAddresses++;
@@ -398,6 +399,7 @@ public class RouterConsoleRunner {
                             ssll.setKeyPassword(ctx.getProperty(PROP_KEY_PASSWORD, "thisWontWork"));
                             ssll.setMaxIdleTime(90*1000);  // default 10 sec
                             ssll.setName("ConsoleSocket");   // all with same name will use the same thread pool
+                            ssll.setUseDirectBuffers(false);  // default true seems to be leaky
                             //_server.addConnector(ssll);
                             connectors.add(ssll);
                             boundAddresses++;
============================================================
--- installer/resources/eepsite/jetty-ssl.xml	78d4d498556b65ce19da958f8eeac2a5e7685a17
+++ installer/resources/eepsite/jetty-ssl.xml	234958b6981fda95688f98e1aa2b708e322c7ac2
@@ -16,6 +16,7 @@
       <New class="org.mortbay.jetty.security.SslSocketConnector">
         <Set name="Port">8443</Set>
         <Set name="maxIdleTime">30000</Set>
+        <Set name="useDirectBuffers">false</Set>
         <Set name="handshakeTimeout">2000</Set>
         <Set name="keystore">./eepsite/etc/keystore.ks</Set>
         <Set name="password">OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4</Set>
============================================================
--- installer/resources/eepsite/jetty.xml	b9be258673a674f019305afb9c7561e98c44d8ef
+++ installer/resources/eepsite/jetty.xml	d561d58072300e27b5b2e31eb478fe828b817622
@@ -101,6 +101,7 @@
             <Set name="confidentialPort">8443</Set>
 	    <Set name="lowResourcesConnections">5000</Set>
 	    <Set name="lowResourcesMaxIdleTime">5000</Set>
+            <Set name="useDirectBuffers">false</Set>
           </New>
       </Arg>
     </Call>

Changed 7 years ago by killyourtv

DirectBuffer? Profiling with YourKit?

comment:4 Changed 7 years ago by killyourtv

An update with the patch above applied:

No router console in browser, but Bote is running

Without the router console open and with i2p-bote stopped:
No router console open in browser, no bote

...and when looking at graphs in the router console:
/graphs loaded in browser

comment:5 Changed 7 years ago by zzz

SAM streams switched to non-direct in 0.9.1-5.
SAM DatagramServer? may remain direct as it is a singleton.

Leaving ticket open until a decision is made on Jetty.

comment:6 Changed 7 years ago by zab

Some updates: after applying an inverse of the patch above, one of the allocation traces has disappeared as expected. More details provided by KYTV here:

http://killyourtv.i2p/tmp/zab/0.9.1-5direct/YourKitTraces/jmap-consoleandbote.txt
http://killyourtv.i2p/tmp/zab/0.9.1-5direct/YourKitTraces/ConsoleBoteGraphs.html
(this is not the same trace as reported earlier, still inside jetty)

So this is an improvement; whether it's worth spending more effort for a 100% fix is unclear. Do we have an easy way to track the delta between total heap usage as reported by the jvm and total process usage as reported by the OS?

comment:7 Changed 7 years ago by zzz

  • Component changed from unspecified to apps/console
  • Milestone set to 0.9.2
  • Resolution set to fixed
  • Status changed from new to closed

Patch above applied in 0.9.1-7.

Note: See TracTickets for help on using tickets.