Opened 4 years ago

Closed 4 years ago

#1395 closed defect (fixed)

Console lockup

Reported by: zzz Owned by: zzz
Priority: blocker Milestone: 0.9.16
Component: apps/console Version: 0.9.15
Keywords: Cc: dg
Parent Tickets:

Description

Related: ​http://forum.i2p/viewtopic.php?t=11340 and #1260
That thread and ticket blame it on i2p-bote, and the attached dump in #1260 does not indicate a problem. However there are several reports.

Just got a dump that is not bote-related, and implicates the CallerRunsPolicy?.

Some combination of the policy, queue, and limits needs to change, in RouterConsoleRunner?.CustomThreadPoolExecutor?.

Subtickets

Change History (7)

comment:1 Changed 4 years ago by zzz

"RouterConsole Jetty Selector0" daemon prio=10 tid=0x0000000001cae000 nid=0x4855 in Object.wait() [0x00007fd5d1fd9000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000000ec3c6ac8> (a org.eclipse.jetty.io.nio.SelectChannelEndPoint)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.blockWritable(SelectChannelEndPoint.java:499)
	- locked <0x00000000ec3c6ac8> (a org.eclipse.jetty.io.nio.SelectChannelEndPoint)
	at org.eclipse.jetty.http.AbstractGenerator.blockForOutput(AbstractGenerator.java:522)
	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:170)
	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:107)
	at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:154)
	- locked <0x00000000ec3feee8> (a org.eclipse.jetty.util.ByteArrayOutputStream2)
	at org.eclipse.jetty.server.HttpWriter.write(HttpWriter.java:288)
	at java.io.PrintWriter.write(PrintWriter.java:426)
	- locked <0x00000000ec3fef00> (a org.eclipse.jetty.server.AbstractHttpConnection$OutputWriter)
	at org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:119)
	at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:326)
	at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:342)
	at net.i2p.router.web.jsp.console_jsp._jspService(console_jsp.java:414)
	at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
	at net.i2p.servlet.filters.XSSFilter.doFilter(XSSFilter.java:28)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1476)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at net.i2p.router.web.LocaleWebAppHandler.handle(LocaleWebAppHandler.java:99)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.Server.handle(Server.java:370)
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
	at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
	at org.eclipse.jetty.http.HttpParser.parseAvail able(HttpParser.java:235)
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
	at java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:2025)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
	at org.eclipse.jetty.util.thread.ExecutorThreadPool.dispatch(ExecutorThreadPool.java:126)
	at org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager.dispatch(SelectChannelConnector.java:300)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.dispatch(SelectChannelEndPoint.java:239)
	- locked <0x00000000ec3c6ac8> (a org.eclipse.jetty.io.nio.SelectChannelEndPoint)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.schedule(SelectChannelEndPoint.java:198)
	- locked <0x00000000ec3c6ac8> (a org.eclipse.jetty.io.nio.SelectChannelEndPoint)
	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:619)
	at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

comment:2 Changed 4 years ago by zzz

Not coincidentally, another thread is stuck rendering /tunnels, a page with a lot of flag icons on it:

"RouterConsole Jetty" daemon prio=10 tid=0x000000000144e800 nid=0x494d in Object.wait() [0x00007fd5c685c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000000ec3f9810> (a org.eclipse.jetty.io.nio.SelectChannelEndPoint)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.blockWritable(SelectChannelEndPoint.java:499)
	- locked <0x00000000ec3f9810> (a org.eclipse.jetty.io.nio.SelectChannelEndPoint)
	at org.eclipse.jetty.http.AbstractGenerator.blockForOutput(AbstractGenerator.java:522)
	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:147)
	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:107)
	at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:154)
	- locked <0x00000000ec3fa958> (a org.eclipse.jetty.util.ByteArrayOutputStream2)
	at org.eclipse.jetty.server.HttpWriter.write(HttpWriter.java:288)
	at java.io.PrintWriter.write(PrintWriter.java:426)
	- locked <0x00000000ec3fa970> (a org.eclipse.jetty.server.AbstractHttpConnection$OutputWriter)
	at org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:119)
	at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:326)
	at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:342)
	at net.i2p.router.web.TunnelRenderer.renderPool(TunnelRenderer.java:193)
	at net.i2p.router.web.TunnelRenderer.renderStatusHTML(TunnelRenderer.java:65)
	at net.i2p.router.web.TunnelHelper.getTunnelSummary(TunnelHelper.java:15)
	at net.i2p.router.web.jsp.tunnels_jsp._jspService(tunnels_jsp.java:331)
	at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
	at net.i2p.servlet.filters.XSSFilter.doFilter(XSSFilter.java:28)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1476)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at net.i2p.router.web.LocaleWebAppHandler.handle(LocaleWebAppHandler.java:99)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.Server.handle(Server.java:370)
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
	at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

comment:5 Changed 4 years ago by dg

My locked up router was running zzzot (and zzzot continued to respond to requests), as did other tunnels. Noting that the other reports referenced I2P-Bote, maybe it's something to do with the plugin code?

I'll continue to monitor the router.

comment:6 Changed 4 years ago by zzz

  • Owner set to zzz
  • Status changed from new to accepted

No. It's triggered when hitting the console thread pool limit (currently 24), combined with some bad luck. Loading a page with more than 23 uncached images and css references will get you there, esp. with shift-reload or ctl-reload (depending on the browser).

Eepsites and zzzot don't set the CallerRuns? policy and so aren't affected. Webapps such as bote are run in in the console Jetty and are affected.

However, I may change the eepsite and zzzot queueing anyway, as I believe the fix I'm developing for the console will work better in those cases also.

comment:7 Changed 4 years ago by zzz

Testing fix now. Draft checkin comments:


Console and Eepsite Jetty:
Switch back to QueuedThreadPool? (ticket #1395)
In Jetty 6, QTP was not concurrent, so we switched to
ThreadPoolExecutor? with a fixed-size queue, a set maxThreads,
and a RejectedExecutionPolicy? of CallerRuns?.
Unfortunately, CallerRuns? causes lockups in Jetty NIO.
In addition, no flavor of TPE gives us what QTP does:

  • TPE direct handoff (which we were using) never queues. This doesn't provide any burst management when maxThreads is reached. CallerRuns? was an attempt to work around that.
  • TPE unbounded queue does not adjust the number of threads. This doesn't provide automatic resource management.
  • TPE bounded queue does not add threads until the queue is full. This doesn't provide good responsiveness to even small bursts.

QTP adds threads as soon as the queue is non-empty.
QTP as of Jetty 7 uses concurrent.
QTP unbounded queue is the default in Jetty.
So switch back to QTP with a bounded queue, which does what we want,
which is first expand the thread pool, then start queueing, then reject.

ref:
http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/ThreadPoolExecutor.html
https://wiki.eclipse.org/Jetty/Howto/High_Load

comment:8 Changed 4 years ago by zzz

  • Status changed from accepted to testing

Fixed in a832f382200f7cf2864e149612bfe78b31d0aed0 0.9.15-14-rc. See also http://zzz.i2p/topics/1718 for related eepsite guidance.

comment:9 Changed 4 years ago by zzz

  • Resolution set to fixed
  • Status changed from testing to closed
Note: See TracTickets for help on using tickets.