Opened 7 years ago

Closed 6 years ago

#715 closed defect (fixed)

Jetty RejectedExecutionException infinite loop

Reported by: zzz Owned by: zzz
Priority: major Milestone: 0.9.6
Component: apps/jetty Version: 0.9.1
Keywords: Cc: postman@…, zab@…
Parent Tickets: Sensitive: no

Description

Jetty goes into an infinite loop after hitting thread limit:

09/22 xx:30:04.402 ERROR [local-ip:port] org.mortbay.jetty.Server      : EXCEPTION 
java.util.concurrent.RejectedExecutionException
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1477)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:384)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:867)
	at org.mortbay.thread.concurrent.ThreadPool.dispatch(ThreadPool.java:103)
	at org.mortbay.jetty.nio.SelectChannelConnector$1.dispatch(SelectChannelConnector.java:86)
	at org.mortbay.io.nio.SelectChannelEndPoint.dispatch(SelectChannelEndPoint.java:81)
	at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:682)
	at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
	at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
	at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
	at java.lang.Thread.run(Thread.java:595)
09/22 xx:30:04.528 ^^^ 1,023 similar messages omitted ^^^

http://jira.codehaus.org/browse/JETTY-1423 may be related but it's against Jetty 7.

This spews out several thousand log messages per second (10s of MB per second of logs) and requires a restart to resolve. While jetty continues to serve connections, it causes a huge slowdown.

Possible workarounds:

1) Increase maximumPoolSize in jetty.xml

2) Change from concurrentThreadPool to QueuedThreadPool??

3) Changing from the nio SelectChannelConnector? to the non-nio connector ???

As a result of this I implemented dup-message removal in our logger as of 0.9.2-1.

To temporarily eliminate the log spew problem set logging config org.mortbay.jetty.Server=CRIT as a temp fix but then you won't know when it's having the problem, it still causes a big slowdown.

To restart Jetty without restarting your router:

  • go to /configclients, uncheck the 'run at startup' box and save, you will now see a start button
  • click start and it will restart it
  • then check run at startup again and save again

Subtickets

Change History (7)

comment:1 Changed 7 years ago by Zlatin Balevsky

Cc: zab@… added

comment:2 Changed 7 years ago by zzz

Looked at the Jetty code and this appears to be a loop in their NIO connector.

I see this on my box running Java 5, which has well-known NIO issues. We bump into them in our EventPumper? (see #551). Jetty 6 has a number of NIO workarounds (see SelectorManager?). The Jetty 7 bug linked above says they substantially rewrote their NIO code in Jetty 7.

I'm going to test the non-NIO connector soon (this is my option 3 above).

Workaround 4) change the queue policy to drop or caller runs?

Contrary to OP, it will actually resolve itself, as long as it doesn't get too far behind from all the logging. Jetty's SelectorManager? has a failsafe workaround every second, so it looks like an individual stuck channel will get caught after looping for a second.

Others seeing this problem, please add comment with your Java version.

comment:3 Changed 7 years ago by zzz

Owner: set to zzz
Priority: criticalmajor
Status: newaccepted

I'm testing the non-nio connector, and it looks good so far, but it does use a lot of threads. Checked in as an alternate in installer/resources/eepsite/jetty.xml in 0.9.2-2.

Here's the alternate:

    <Call name="addConnector">
      <Arg>
          <New class="org.mortbay.jetty.bio.SocketConnector">
            <Set name="host">127.0.0.1</Set>
            <Set name="port">7658</Set>
            <Set name="maxIdleTime">60000</Set>
            <Set name="Acceptors">1</Set>
            <Set name="statsOn">false</Set>
            <Set name="confidentialPort">8443</Set>
          </New>
      </Arg>
    </Call>

comment:4 Changed 7 years ago by zzz

postman has seen this on Java 1.6.24

comment:5 Changed 7 years ago by zzz

Milestone: 0.9.30.9.6

Console also switched to use bio connector on Java 5.

If NIO wigs out for Jetty on Java 6, I don't know what we can do about it.

No idea how to proceed further on this.

Other than migrate to Jetty 7.

comment:6 Changed 7 years ago by zzz

related: #743

comment:7 Changed 6 years ago by zzz

Resolution: fixed
Status: acceptedclosed

Optimistically closing, hoping that Jetty 7 (merged in 0.9.5-1) fixes it.

Note: See TracTickets for help on using tickets.