#2071 closed defect (fixed)

Incorrect handling of closed connections / memory leak

Reported by: val Owned by: zzz
Priority: minor Milestone: 0.9.33
Component: apps/i2ptunnel Version: 0.9.32
Keywords: Cc:
Parent Tickets:

Description

I2P router does not detect closed connections in a sane time, but only after relatively huge timeout (1+ minute). This causes high memory consumption is case of downloading big files and may crash the router.

Steps to reproduce:

  1. Set up I2P router bandwidth to 3000/3000
  2. Setup jetty and configure 0 hop eepsite
  3. Create sparse file in eepsite docroot with fallocate -l 1G 1GB.zip
  4. curl --limit-rate 256k --proxy 127.0.0.1:4444 -o /dev/null http://base32.b32.i2p/1GB.zip
  5. Press ctrl+c after 3 seconds
  6. Watch increasing memory usage

Expected result:

Connection is dropped as soon as curl is stopped with ctrl+c, or in some seconds after that. No high memory usage.

Actual result:

Both "client" and "server" I2P router's memory is increasing over time and drops only after ≈2 minutes of client inactivity.
With 1 hop tunnel this increase memory usage from 280 to 370 MB in 1 minute.
With 0 hop tunnel memory consumption goes from 280 MB to 1.8 GB in ≈30 seconds and i2p gets killed by OOM.

Note this is not a loopback issue, but it is much more severe with loopback or 0 hop tunnels from both sides. Tested with both loopback and with client/server instances.

For more fun, install GNU parallel and run:
yes | parallel -j32 'curl --limit-rate 256k --proxy 127.0.0.1:4444 -o /dev/null http://base32.b32.i2p/1GB.zip' and press ctrl+c after 5 seconds.
Jetty crashes, I2P does not but consume 600+ MB (1 hop tunnel).

Subtickets

Change History (12)

comment:1 Changed 16 months ago by zab

Just a quick note on the way JVM uses memory which may explain this behavior:
The JVM aggressively allocates memory from the OS even if that memory far exceeds the maximum heap size, and returns it very slowly. Hence profiling or code changes at java layer may not be able to help with this issue at all. Also, my hunch is that this is related to Jetty.

comment:2 Changed 16 months ago by jogger

Two remarks: Same behaviour can be reproduced be connecting two snarks (standalone or not) to the same router and seeding a large torrent from one to the other. Allocates 100s of MB within seconds and eventually crashes the machine. Can be mitigated by limiting snarks outbound bandwidth.

zab is correct about JVM. Solution is something like:
wrapper.java.additional.3=-XX:+PerfDisableSharedMem?
wrapper.java.additional.4=-server
wrapper.java.additional.5=-Xmn64m
wrapper.java.additional.6=-XX:MaxMetaspaceSize=128m
wrapper.java.additional.7=-XX:+UseConcMarkSweepGC
wrapper.java.additional.8=-XX:+CMSParallelRemarkEnabled
wrapper.java.additional.9=-XX:+UseCMSInitiatingOccupancyOnly
wrapper.java.additional.10=-XX:CMSInitiatingOccupancyFraction=70
wrapper.java.additional.11=-XX:+ScavengeBeforeFullGC
wrapper.java.additional.12=-XX:+CMSScavengeBeforeRemark
wrapper.java.additional.13=-XX:MaxHeapFreeRatio=50
wrapper.java.additional.14=-XX:MinHeapFreeRatio=25
wrapper.java.additional.15=-Xss256k

These settings easily save more than a Gig Java memory compared to defaults for a class X router with lots of torrents. Needs more than 24-48h uptime to see the full difference.

Otherwise you need to have enough swap available and do "i2prouter restart" when available RAM drops below some 100 MB. I check this from cron every 15 min.

comment:3 Changed 16 months ago by zzz

  • Component changed from unspecified to apps/i2ptunnel

comment:4 Changed 16 months ago by zzz

Apart from the general issue of RAM and where it's used, and whether we can do better (which comments 1 and 2 are addressing), OP is hypothesising that a control-C at the curl client isn't getting propagated all the way through the socket to i2ptunnel to streaming to the server i2ptunnel to jetty, either in time or at all. That may be the case. I remember working on it sometime this year, but can't remember if I thought I fixed it already, or not. I know I did add a reset() method to I2PSocket in 0.9.30. For further research.

Last edited 16 months ago by zzz (previous) (diff)

comment:5 Changed 16 months ago by zab

@val and @jogger:
if you add the property -XX:+HeapDumpOnOutOfMemoryError? it will generate a file useful for researching the issue. You can analyze the file yourself with the "jhat" tool, or, if you trust us with your anonymity, you can let us analyze it for you.

comment:6 Changed 16 months ago by zzz

Possibly related #1939 comment 5 from OP of this ticket

Last edited 16 months ago by zzz (previous) (diff)

comment:7 Changed 16 months ago by zzz

re: reset and comment 4

The only place we currently handle reset propagation specially is in HTTPClient, where we display a different message to the user if the server sent a reset.

We've never attempted to propagate an abnormal close (reset) through a standard Socket with soLinger.

Should we wish to close the i2ptunnel-to-jetty Socket more violently (presumably by setting setSoLinger(true, 0) before close()) we would want to do that in I2PTunnelRunner. See I2PTunnelHTTPClientBase for an example, but basically check if exception is instanceof I2PSocketException, cast, and check for STATUS_CONNECTION_RESET.

But first we should look to see if we're cleaning things up fast enough in i2ptunnel. I believe that streaming and i2cp should be innocent, after the previous work on reset, but that needs a double-check also.

comment:8 Changed 16 months ago by zab

I wasn't able to reproduce this in its entirety - while the resident memory usage shown with "top" was close to 1GB, the heap memory was only 50MB out of which only 24MB were live objects.

So this means either the JVM memory-mapped the large file or it is just very greedy when it comes to OS memory usage.

Note that by default the maximum heap size is 128MB; in order for the router to die with OOM that needs to be hit, but how much resident memory it translates to is anyone's guess.

comment:9 Changed 16 months ago by zzz

I did some experiments with nc (netcat) to see what exceptions we could catch to infer an abnormal close. I tried abnormal stops with control-c, kill, and kill -9. In all cases, all I got was an EOFException on the read side. Some SO posts hinted that perhaps we could see or even trigger on a SocketException?("Connection Reset"), but I'm not seeing it on the read side. Maybe we would on the write side.

So it appears we have to detect this by having data from i2p to write to the client and that fails.

I see some stuff in streaming CPH that might send a reset back if it gets data when the output is closed. It may also depend on what I2PTunnelRunner does, and exactly how it behaves, to propagate closes and errors from the socket to streaming. As I said above, we've done a lot more work to get errors from streaming to the client, and almost none in the other direction.

Next step, do some debugging to see what propagates thru, and how, from client to i2ptunnel to streaming tp i2cp thru i2p to i2cp to streaming to i2ptunnel to jetty.

comment:10 Changed 16 months ago by zzz

  • Owner set to zzz
  • Status changed from new to accepted

Confirmed OP's premise, we will continue receiving and acking data after a local close. This is a streaming bug.

Also working on reset propagation as described in my comments 4, 7, and 9 above. This work is mostly in i2ptunnel.

comment:11 Changed 16 months ago by zzz

  • Milestone changed from undecided to 0.9.33

Send reset when receiving more data on a locally closed connection, in 774aa0536ec3036857849382140155e781fd6106 0.9.32-11

Reset propagation thru i2ptunnel in 789fd1cfa01f8cc63e20fa86031afc2e9fd8a673 0.9.32-11

Leaving open for testing. OP and zab please report results.

comment:12 Changed 15 months ago by zzz

  • Resolution set to fixed
  • Status changed from accepted to closed

presumed fixed

Note: See TracTickets for help on using tickets.