Opened 4 years ago

Last modified 4 years ago

#1666 new defect

Occasional very high spikes in transport.sendProcessingTime and tunnel.acceptLoad delay greatly hindering participating tunnel performance

Reported by: somewon Owned by:
Priority: minor Milestone: undecided
Component: router/general Version: 0.9.22
Keywords: performance lag router Cc:
Parent Tickets:

Description

I know that this is probably very difficult to track down, and it may just be my computer (though I think not, for decent reasons I'll get into below), but it could be a big performance sink on a lot of routers so I figured I'd start a dialogue.

Basically, I have noticed that around five to ten times per week, my router graphs will show very extreme spikes in transport.sendProcessingTime and tunnel.acceptLoad delays that almost completely kill my participating bandwidth for a good while afterward (the correct response, as my peers see that my router is failing to do its job). Graph images are attached.

I am fairly certain that this is not a result of performance constraints on my computer, internet connection or OS in general - there is no correlation to any other "heavy activity" on the computer and the system as a whole remains totally responsive during these incidents, and neither memory, bandwidth nor CPU usage are near their limits. At times, I have re-niced the java process to -1 to absolutely keep it from any CPU constraints, and yet the issue still occurs. Similarly, I use QoS to prioritize my I2P traffic, so there should be no delays from other bandwidth consuming activity. I cannot identify any factors in the rest of the router data that might be causative, and the issue does not appear to follow any regular time cycle.

Most recently, a particularly bad one of these spikes seems to have brought about an increase in the memory consumption of the router (visible in the attached graph).

This issue has been occurring regularly for at least a couple of months, across multiple I2P versions. It seems to be a distinct issue from the "NTCP Pumper Thread" bug that has received attention.

No messages about these issues appear at loglevel ERROR, so I have just increased the router loglevel to WARN, but there is such an influx of messages that I'm not sure where to start. I will update this ticket with relevant log entries once I experience another such spike, if there are any. If more detailed logging would be appropriate, please let me know. And if there's any other troubleshooting or data I might be able to offer to help narrow the focus of this issue, please let me know that as well. I'd like to be as helpful as possible to correct this.

Much gratitude to all of the hard-working developers and community members who make I2P so great - you are truly appreciated.

I2P version: 0.9.22-0
Java version: Oracle Corporation 1.7.0_79 (OpenJDK Runtime Environment 1.7.0_79-b14)
Wrapper version: 3.5.22
Server version: 8.1.14.v20131031
Servlet version: Jasper JSP 2.1 Engine
Platform: Linux amd64 3.2.0-4-amd64
Processor: Sandy Bridge H/M (corei)
Jbigi: Locally optimized native BigInteger? library loaded from file
Encoding: UTF-8
Charset: UTF-8

Subtickets (add)

Attachments (12)

tunnel.sendProcessingTime.png (9.3 KB) - added by somewon 4 years ago.
tunnel.sendProcessingTime shortly after a bad spike
tunnel.acceptLoad.png (10.8 KB) - added by somewon 4 years ago.
tunnel.acceptLoad after a bad spike
router.memoryUsed.png (9.2 KB) - added by somewon 4 years ago.
router.memoryUsed shortly after a bad spike
tunnel.participatingBandwidth.png (11.6 KB) - added by somewon 4 years ago.
tunnel.participatingBandwidth after a bad spike in sendProcessingTime delay
tunnel.participatingTunnels.png (10.2 KB) - added by somewon 4 years ago.
tunnel.participatingTunnels after a bad spike in tunnel.sendProcessingTime delay
tunnel.sendProcessingTime-02.png (8.1 KB) - added by somewon 4 years ago.
tunnel.acceptLoad-02.png (10.9 KB) - added by somewon 4 years ago.
router.memoryUsed-02.png (10.3 KB) - added by somewon 4 years ago.
tunnel.participatingBandwidth-02.png (11.5 KB) - added by somewon 4 years ago.
tunnel.participatingTunnels-02.png (9.9 KB) - added by somewon 4 years ago.
updated log-02.2.txt (5 bytes) - added by somewon 4 years ago.
disregard, not the log we need
updated log-02.txt (5 bytes) - added by somewon 4 years ago.
disregard, not the log we need

Download all attachments as: .zip

Change History (17)

Changed 4 years ago by somewon

tunnel.sendProcessingTime shortly after a bad spike

Changed 4 years ago by somewon

tunnel.acceptLoad after a bad spike

Changed 4 years ago by somewon

router.memoryUsed shortly after a bad spike

Changed 4 years ago by somewon

tunnel.participatingBandwidth after a bad spike in sendProcessingTime delay

Changed 4 years ago by somewon

tunnel.participatingTunnels after a bad spike in tunnel.sendProcessingTime delay

comment:1 Changed 4 years ago by somewon

Perhaps one interesting detail to note is that in the tunnel.acceptLoad graph, the apparent delay drops to ZERO immediately before the biggest spike. I'm not sure what that could mean, but it's a small detail that might be otherwise overlooked so I would like to call attention to it, though it could be completely irrelevant.

comment:2 Changed 4 years ago by somewon

Oh, and I should specify that my router is NOT floodfill and is configured not to become floodfill.

comment:3 Changed 4 years ago by echelon

Moin

I see those spikes mostly at midnight, key turnover of I2P and network issues on ISP for me.
Could it be Java VM trash collection on your system?
Also your router get bak to normal in rather short time, good. Performance not harmed and net works on fine.

comment:4 Changed 4 years ago by zzz

related: #698

comment:5 Changed 4 years ago by somewon

Caught another one, this time nearly 12 seconds delay on sendProcessingTime. Unfortunately has run off of the logfile, I will increase log size and upload WARN logs that actually catch one later. At first I thought I had a log file for it, but this was merely a mistake due to the timezone discrepancies between the Graphs and Logs - disregard the log attachments that immediately follow.

Last edited 4 years ago by somewon (previous) (diff)

Changed 4 years ago by somewon

Changed 4 years ago by somewon

Changed 4 years ago by somewon

Changed 4 years ago by somewon

Changed 4 years ago by somewon

Changed 4 years ago by somewon

disregard, not the log we need

Changed 4 years ago by somewon

disregard, not the log we need

Note: See TracTickets for help on using tickets.