Opened 5 months ago

Closed 5 months ago

#2781 closed defect (fixed)

UDP transport loses inbound packets

Reported by: jogger Owned by: zzz
Priority: major Milestone: 0.9.48
Component: router/transport Version: 0.9.47
Keywords: Cc:
Parent Tickets: Sensitive: no


Easy to see: use "netstat -su" to monitor receive buffer errors.

Reason is that the JVM overrides system defaults and sets the socket receive buffer to some way too low value that may even vary after router restart on unchanged config. I observed values as low as 32k, good for less than 2ms traffic.

In my case errors were gone after setting the socket receive buffer to hold 100ms worth of inbound traffic. Example code for UDPEndPoint:

private static final int MIN_SOCKET_BUFFER = 262144;
                    int size = socket.getReceiveBufferSize();
                    if (size < MIN_SOCKET_BUFFER) {
              "current UDP receive buffer size " + size + ", setting to " + MIN_SOCKET_BUFFER);
                    size = socket.getSendBufferSize();
                    if (size < MIN_SOCKET_BUFFER) {
              "current UDP send buffer size " + size + ", setting to " + MIN_SOCKET_BUFFER);


Change History (3)

comment:1 Changed 5 months ago by zzz

Interesting. What % loss are you seeing? I see less than .001% on recv and zero on send.

On my linux, the initial sizes on the socket are 106496. According to 'man -s 7 socket', the values in /proc/sys/net/core/ rmem_default, rmem_max, wmem_default, wmem_max are twice the user values, to account for overhead. My values of those files are 212992. Makes sense.

But when I set and read back the values via java, I can set them as high as 212992. Which makes no sense. Either the user values are half the /proc values or not. Maybe there's a java bug here. So it isn't clear if trying to raise the value above the default actually does anything, if default == max in /proc.

On windows 8 and 10, the initial value is 65536, but I can set it to at least 1 MB. Didn't try higher.

comment:2 Changed 5 months ago by jogger

The relevant values for UDP on Linux are net.ipv4.udp_rmem_min and net.ipv4.udp_wmem_min AFAIK. The values you mentioned are also set well above 100 KB. I had those set to 512 KB. However I always got 32 KB or 64 KB within I2P.

Packet loss was around 1% inbound and near zero outbound. Zero loss on inbound with 256 KB which holds traffic for more than one GC pause on my machines.

comment:3 Changed 5 months ago by zzz

Milestone: undecided0.9.48
Resolution: fixed
Status: newclosed

OK. Obviously the defaults are going to vary widely by OS and distribution. Perhaps the JVM is also playing a role. It doesn't really matter. Agreed that 32-64k is too low. While all my testing on Ubuntu already has the value set to the max (or not? see the doubling issue above), so this won't do anything for me, it's worth it for windows for sure and for whatever your setup is. Also tested on a Mac. Initial value is 65507, supports up to at least 1 MB. So it's similar to Windows.

Set to 256K, in eabb6edd4f97838dc8eaf4d8aec0394ee4e87a09 to be 0.9.47-8

Note: See TracTickets for help on using tickets.