Opened 3 weeks ago

Last modified 8 days ago

#2586 new enhancement

Fine-tune BatchedRouterPreprocessor.DEFAULT_BATCH_FREQUENCY

Reported by: Zlatin Balevsky Owned by:
Priority: minor Milestone: undecided
Component: router/general Version: 0.9.41
Keywords: testnet Cc:
Parent Tickets: Sensitive: no


The current value of that parameter is 175ms. Note that I'm not referring to the property "router.batchFrequency" as that also overrides the OB_CLIENT_BATCH_FREQ parameter.

I tested from 175ms down to 1ms in 25ms steps in the testnet by downloading a 25MB file. The results are in the attached spreadsheet. tl;dr things get very interesting between 75ms and 50ms.

Quick note regarding #2584 - the results there were based on a value of "router.batchFrequency=50" which I had set and forgotten about. The results from this test are against mtn revision 2ec35a4caf7e2622fd5aa9e8c4edc6c3aff9c80f


Attachments (3)

DEFAULT_BATCH_FREQUENCY_analysis.ods (16.3 KB) - added by Zlatin Balevsky 3 weeks ago.
DEFAULT_BATCH_FREQUENCY_analysis2.ods (17.3 KB) - added by Zlatin Balevsky 2 weeks ago.
DEFAULT_BATCH_FREQUENCY_analysis3.ods (16.7 KB) - added by Zlatin Balevsky 2 weeks ago.

Download all attachments as: .zip

Change History (13)

Changed 3 weeks ago by Zlatin Balevsky

comment:1 Changed 3 weeks ago by zzz

I do believe that the preprocessor batching parameters have an important effect on latency and therefore throughput.

One thing to keep in mind on this and the many SSU tickets is that any delay parameter is a latency vs bandwidth efficiency tradeoff. Reducing delay parameters will decrease latency, but at some network cost that's not captured in any testnet data.

In particular, the more data there is to 'batch', the more efficiently the batcher can do the packing problem. That's why this value is relatively high. Haven't looked at the data yet, won't comment on the results until I do.

comment:2 Changed 3 weeks ago by Zlatin Balevsky

During bulk transfers the pre-processor doesn't really matter in the direction of the transfer; on the way back it only serves to delay the acks. It is worth investigating on the live net how efficient the packing is in reality and what is the nature of the traffic.

comment:3 Changed 2 weeks ago by zzz

Set at 500 March 2005
Reduced to 100 Nov. 2005
Split up by type - 100 OB client; 150 OB expl; 250 IB, Dec. 2009
Reduced to 75 OB client; 100 OB expl; 175 IB, Jan. 2015

The IBGW value kicks in twice in a round trip, once in each direction, so this dial isn't going to have as big an effect as some of the per-hop things we're working on.

As mentioned in comment 1, the efficiency of the gateways (both OB and IB) at packing messages has an enormous effect on the performance of the network as a whole. Putting a single ACK into a 1KB tunnel message by itself gives you 90% or so wasted bandwidth.

BatchedPreprocessor? also has, I think, logic to prevent a portion of a fragmented message from being left behind, and to send a tunnel message if it's "mostly" full rather than waiting.

The IBGW has "incentive" to delay and pack efficiently, because this isn't his traffic, it's for others. That's why the IBGW number is higher.

I'm not sure what's happening in the spreadsheet or why things get so much worse again at low delay numbers - perhaps because the batching is becoming ineffective, as described above.

comment:4 Changed 2 weeks ago by zzz

OP reports spreadsheet readings are final eepget-reported bandwidth. Higher is better.

comment:5 in reply to:  4 ; Changed 2 weeks ago by Zlatin Balevsky

Replying to zzz:

OP reports spreadsheet readings are final eepget-reported bandwidth. Higher is better.

Right. The fact that there is a steep jump which happens only at the 75-50ms interval instead of a linear relationship is what puzzles me. I think there is some threshold, or combination of thresholds somewhere that gets hit when the IBGW delay is set to 50. It is worth figuring out what that is.

I will benchmark the 75-50 interval in 5ms steps to see what shape it is.

comment:6 Changed 2 weeks ago by zzz

I'm not sure how much we can learn with a test on an unlimited-bandwidth router testnet, where overhead is free. Either we need analysis or measurement of overhead or efficiency, or we need to constrain the bandwidths of the routers and then see what the eepget rate is. We can't optimize for bandwidth on a testnet where overhead is free, we need the second axis, or something.

comment:7 in reply to:  6 Changed 2 weeks ago by Zlatin Balevsky

Replying to zzz:

we need the second axis, or something.

Maybe I can dig something from the logs that will count the number of messages that arrive at the IBGW and the number that go out after the batched preprocessor. That should give us a dimension of efficiency, I think.

Changed 2 weeks ago by Zlatin Balevsky

comment:8 in reply to:  5 Changed 2 weeks ago by Zlatin Balevsky

I will benchmark the 75-50 interval in 5ms steps to see what shape it is.

See the attachment DEFAULT_BATCH_FREQUENCY_analysis2.ods. There is a big jump between 75 and 70 milliseconds, another one between 55 and 50. Included are T-Test p-values, which indicate the probability the difference between the samples occurred by chance, so lower is better.

Changed 2 weeks ago by Zlatin Balevsky

comment:9 Changed 2 weeks ago by Zlatin Balevsky

In the DEFAULT_BATCH_FREQUENCY_analysis3.ods file I've repeated the first experiment but with 75ms latency added to each container. The speeds are much closer to what I've observed on the live net, and the mysterious jump from 75 to 50ms is still there.

It may be worth exploring values in the other direction once I find a way to measure the efficiency of packing.

comment:10 Changed 8 days ago by zzz

Interesting. We should proceed carefully as we don't have any data yet on 'efficiency'. It does make sense that, disregarding efficiency, we'd get a gradual increase in end-to-end bandwidth as we reduce batching time, then a larger increase as the batching time approaches zero. The third spreadsheet shows a modest local maximum at 150 for IBGW, so I'm making that modest change in 3c7f69093cde3e4afbe895ccff73e22ebf03f310 to be 0.9.41-9. Let's save more aggressive changes for the next release.

It may be a good time to review the heuristics and tradeoffs in BatchedPreprocessor?, as there's not going to be a single perfect answer to the packing problem. Artificially reordering things also can cause streaming to reduce its window size, so there's also considerations to be made there, and possibly we should review the streaming algos if we're going to aggressively reorder at the gateways.

If we don't have some stat of IBGW efficiency out in the real net we should instrument it. Some plot of efficiency, latency, and batch time may help us find an optimal setting.

Note: See TracTickets for help on using tickets.