Opened 11 years ago

Closed 2 years ago

#346 closed defect (fixed)

Near 100% packet loss with big datagrams

Reported by: HungryHobo Owned by: HungryHobo
Priority: minor Milestone: 0.9.41
Component: router/transport Version: 0.8.1
Keywords: NTCP SSU datagrams MTU Bote Cc:
Parent Tickets: Sensitive: no


I'm trying to figure out why I2P-Bote is so unreliable, and I noticed bigger datagrams have a low chance of being received. I did a few tests and found:

  • For 1 kB datagrams sent at 1-second intervals, the average success rate was 80%
  • For 10 kB datagrams sent at 10-second intervals, the average success rate was 70%
  • For 30 kB datagrams sent at 30-second intervals, the average success rate was 0%

This is confirmed by the fact that I2P-Bote packets that are 30 kB rarely reach their destination.


Change History (12)

comment:1 Changed 11 years ago by HungryHobo

Percentages are rounded, and 1 kB = 1000 bytes.
According to http://www.i2p2.i2p/applications.html, datagrams can be up to 31 kB.

comment:2 Changed 11 years ago by zzz

Thanks for the data. I'm not surprised… there is work to do, especially on datagrams which don't see much testing or development.

There are limits at various layers, and when combined with an aggressive discard strategy at the tuunel hops, bigger is definitely not better. Refer to the NTCP, UDP, tunnel message, and garlic message docs for details… but it's hard to give precise limits due to uncertainties in how much the lower layers add.

Streaming uses an MTU of 1730, and there's good reasons for that.

In summary:

  • Messages are fragmented into 1KB tunnel messages, and intermediate hops may drop tunnel messages, so the failure chance for a 20 KB message is ~= (tunnel message drop rate) 20.
  • At the OBEP and IBGW, dropping is on full messages, and bigger messages are more likely to get dropped. So it's even worse.
  • The NTCP limit is ~16KB, messages above that (we're talking about the OBEP-IBGW link again) must travel via UDP only… with no transport choices, failure is more likely.
  • As you approach 16KB and 32KB, the addition of the dest and signature (for repliable datagrams), garlic overhead, session tags, etc can add several KB, which may push things into 100% failure. A lot of the overhead is optimized for streaming lib, and puts too much burden on intermittent datagram traffic.

Imule ran across the same issues and I think settled on about 10-12KB for a max, but I don't remember too well.

I think our message drop rate is probably a major factor in our current performance, we really need to look at it. You can check the stream.sendsBeforeAck stat - it's 1.07 for me right now, or 7% retransmission. We really need to cut that in half. At least.

Speaking of, 20% loss for 1KB datagrams sounds way too high, compared to that streaming rate of 7%. So there might be other issues too.

comment:3 Changed 11 years ago by Eche|on

Owner: set to zzz
Status: newassigned

comment:4 Changed 11 years ago by zzz

Milestone: 0.8.5

comment:5 Changed 10 years ago by zzz


comment:6 Changed 10 years ago by zzz

Working on RED changes for 0.8.12 which may help.

For other changes see the thread http://zzz.i2p/topics/809 . These changes are on hold as they require your participation to finish the implementation in the router and set the flags on the client side.

comment:7 Changed 9 years ago by zzz

Milestone: 0.9
Owner: changed from zzz to HungryHobo
Priority: majorminor

0.9.2 contains the per-packet I2CP options work and i2psnark DHT uses it. Setting the tag threshold and tags sent options to lower values will reduce packet size. So will the don't-bundle-leaseset option.

Also, changing bote to use two ports, one for signed and one for unsigned datagrams, as i2psnark does, will reduce packet sizes for replies.

All of this will help you reduce the overhead on a datagram, making it smaller and more likely to get through.

What it doesn't do is change the drop probability for the actual (i.e. with overhead) datagram. That's still driven by fragmentation to 1KB tunnel messages, with the exponentially increasing probability. That's a fundamental behavior of the network that can't be changed.

Moreover, changes upcoming in 0.9.3 to add active queue management (AQM) with CoDel? may lead to more dropping, in the name of latency (bufferbloat) reduction. We're embracing the 'dropping is good' philosophy. That may hurt datagram delivery probability.

Reassigning to HH to evaluate and implement per-packet options. It would also be great to get updated test results.

comment:8 Changed 9 years ago by HungryHobo

I wrote a little test program and did a quick test. It looks like things have improved!

5000 bytes: 20 out of 20 packets received

10000 bytes: 19 out of 20 packets received
15000 bytes: 19 out of 20 packets received
20000 bytes: 19 out of 20 packets received
25000 bytes: 18 out of 20 packets received
30000 bytes: 0 out of 20 packets received

I'm going to check the program into i2p.i2p-bote in a minute.

comment:9 Changed 9 years ago by HungryHobo

Another test, but this time I used a fresh destination for each set. Almost all packets go through up to 10kBytes, but bigger ones don't make it at all. Of course the network might be busier than last night.

5000 bytes: 19/20 packets received

10000 bytes: 19/20 packets received
15000 bytes: 0/20 packets received
20000 bytes: 0/20 packets received
25000 bytes: 0/20 packets received
30000 bytes: 0/20 packets received

comment:10 Changed 9 years ago by zzz

Great news and sounds about right. From www.i2p2.i2p/ssu :

Message Size Limits

While the maximum message size is nominally 32KB, the practical limit differs. The protocol limits the number of fragments to 7 bits, or 128. The current implementation, however, limits each message to a maximum of 64 fragments, which is sufficient for 64 * 534 = 33.3 KB when using the 608 MTU. Due to overhead for bundled LeaseSets? and session keys, the practical limit at the application level is about 6KB lower, or about 26KB. Further work is necessary to raise the UDP transport limit above 32KB. For connections using the larger MTU, larger messages are possible.

So, the hardcoded limit is 32KB, and when you add in the bundled LeaseSet? and tags, and the datagram dest and sig, you're going to run out of room at about 29KB I think. With the new I2CP params described above you can reduce that overhead.

Great test results, thanks!

comment:11 Changed 8 years ago by str4d

Keywords: NTCP SSU datagrams MTU Bote added

comment:12 Changed 2 years ago by zzz

Milestone: 0.9.41
Resolution: fixed
Status: assignedclosed

SSU bug fixed in #2505 0.9.40-9

Note: See TracTickets for help on using tickets.