Opened 8 years ago

Closed 8 years ago

#337 closed defect (fixed)

JVM crash from seedless

Reported by: guest Owned by: sponge
Priority: major Milestone: 0.8.2
Component: apps/plugins Version: 0.8.1
Keywords: Cc:
Parent Tickets:

Description

I2P version: 0.8.1-12
Java version: Sun Microsystems Inc. 1.6.0_18 (OpenJDK Runtime Environment 1.6.0_18-b18)
Platform: Linux amd64 2.6.32-26-generic
Processor: uninitialized (athlon64)
Jbigi: Locally optimized native BigInteger? library loaded from the library path
Encoding: UTF-8

2010/11/29 08:38:43 | ODBServer:ODBServerImpl.startServer:java.net.SocketException: Too many open files
2010/11/29 08:38:43 | at java.net.PlainSocketImpl?.socketAccept(Native Method)
2010/11/29 08:38:43 | at java.net.AbstractPlainSocketImpl?.accept(AbstractPlainSocketImpl?.java:358)
2010/11/29 08:38:43 | at java.net.ServerSocket?.implAccept(ServerSocket?.java:470)
2010/11/29 08:38:43 | at java.net.ServerSocket?.accept(ServerSocket?.java:438)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.waitForRemoteConnection(ODBServerImpl.java:196)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.startServer(ODBServerImpl.java:186)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.run(ODBServerImpl.java:166)
2010/11/29 08:38:43 | at java.lang.Thread.run(Thread.java:636)
2010/11/29 08:38:43 |
2010/11/29 08:38:43 | ODBServer:ODBServerImpl.startServer:java.net.SocketException: Too many open files
2010/11/29 08:38:43 | at java.net.PlainSocketImpl?.socketAccept(Native Method)
2010/11/29 08:38:43 | at java.net.AbstractPlainSocketImpl?.accept(AbstractPlainSocketImpl?.java:358)
2010/11/29 08:38:43 | at java.net.ServerSocket?.implAccept(ServerSocket?.java:470)
2010/11/29 08:38:43 | at java.net.ServerSocket?.accept(ServerSocket?.java:438)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.waitForRemoteConnection(ODBServerImpl.java:196)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.startServer(ODBServerImpl.java:186)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.run(ODBServerImpl.java:166)
2010/11/29 08:38:43 | at java.lang.Thread.run(Thread.java:636)
2010/11/29 08:38:43 |
2010/11/29 08:38:43 | ODBServer:ODBServerImpl.startServer:java.net.SocketException: Too many open files
2010/11/29 08:38:43 | at java.net.PlainSocketImpl?.socketAccept(Native Method)
2010/11/29 08:38:43 | at java.net.AbstractPlainSocketImpl?.accept(AbstractPlainSocketImpl?.java:358)
2010/11/29 08:38:43 | at java.net.ServerSocket?.implAccept(ServerSocket?.java:470)
2010/11/29 08:38:43 | at java.net.ServerSocket?.accept(ServerSocket?.java:438)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.waitForRemoteConnection(ODBServerImpl.java:196)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.startServer(ODBServerImpl.java:186)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.run(ODBServerImpl.java:166)
2010/11/29 08:38:43 | at java.lang.Thread.run(Thread.java:636)
2010/11/29 08:38:43 |
2010/11/29 08:38:43 | ODBServer:ODBServerImpl.startServer:java.net.SocketException: Too many open files
2010/11/29 08:38:43 | at java.net.PlainSocketImpl?.socketAccept(Native Method)
2010/11/29 08:38:43 | at java.net.AbstractPlainSocketImpl?.accept(AbstractPlainSocketImpl?.java:358)
2010/11/29 08:38:43 | at java.net.ServerSocket?.implAccept(ServerSocket?.java:470)
2010/11/29 08:38:43 | at java.net.ServerSocket?.accept(ServerSocket?.java:438)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.waitForRemoteConnection(ODBServerImpl.java:196)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.startServer(ODBServerImpl.java:186)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.run(ODBServerImpl.java:166)
2010/11/29 08:38:43 | at java.lang.Thread.run(Thread.java:636)
2010/11/29 08:38:43 |
2010/11/29 08:38:43 | ODBServer:ODBServerImpl.startServer:java.net.SocketException: Too many open files
2010/11/29 08:38:43 | at java.net.PlainSocketImpl?.socketAccept(Native Method)
2010/11/29 08:38:43 | at java.net.AbstractPlainSocketImpl?.accept(AbstractPlainSocketImpl?.java:358)
2010/11/29 08:38:43 | at java.net.ServerSocket?.implAccept(ServerSocket?.java:470)
2010/11/29 08:38:43 | at java.net.ServerSocket?.accept(ServerSocket?.java:438)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.waitForRemoteConnection(ODBServerImpl.java:196)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.startServer(ODBServerImpl.java:186)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.run(ODBServerImpl.java:166)
2010/11/29 08:38:43 | at java.lang.Thread.run(Thread.java:636)
2010/11/29 08:38:43 |
2010/11/29 08:38:43 | ODBServer:ODBServerImpl.startServer:java.net.SocketException: Too many open files
2010/11/29 08:38:43 | at java.net.PlainSocketImpl?.socketAccept(Native Method)
2010/11/29 08:38:43 | at java.net.AbstractPlainSocketImpl?.accept(AbstractPlainSocketImpl?.java:358)
2010/11/29 08:38:43 | at java.net.ServerSocket?.implAccept(ServerSocket?.java:470)
2010/11/29 08:38:43 | at java.net.ServerSocket?.accept(ServerSocket?.java:438)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.waitForRemoteConnection(ODBServerImpl.java:196)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.startServer(ODBServerImpl.java:186)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.run(ODBServerImpl.java:166)
2010/11/29 08:38:43 | at java.lang.Thread.run(Thread.java:636)
2010/11/29 08:38:43 |
2010/11/29 08:38:43 | ODBServer:ODBServerImpl.startServer:java.net.SocketException: Too many open files
2010/11/29 08:38:43 | at java.net.PlainSocketImpl?.socketAccept(Native Method)
2010/11/29 08:38:43 | at java.net.AbstractPlainSocketImpl?.accept(AbstractPlainSocketImpl?.java:358)
2010/11/29 08:38:43 | at java.net.ServerSocket?.implAccept(ServerSocket?.java:470)
2010/11/29 08:38:43 | at java.net.ServerSocket?.accept(ServerSocket?.java:438)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.waitForRemoteConnection(ODBServerImpl.java:196)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.startServer(ODBServerImpl.java:186)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.run(ODBServerImpl.java:166)
2010/11/29 08:38:43 | at java.lang.Thread.run(Thread.java:636)
2010/11/29 08:38:43 |
2010/11/29 08:38:43 | ODBServer:ODBServerImpl.startServer:java.net.SocketException: Too many open files
2010/11/29 08:38:43 | at java.net.PlainSocketImpl?.socketAccept(Native Method)
2010/11/29 08:38:43 | at java.net.AbstractPlainSocketImpl?.accept(AbstractPlainSocketImpl?.java:358)
2010/11/29 08:38:43 | at java.net.ServerSocket?.implAccept(ServerSocket?.java:470)
2010/11/29 08:38:43 | at java.net.ServerSocket?.accept(ServerSocket?.java:438)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.waitForRemoteConnection(ODBServerImpl.java:196)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.startServer(ODBServerImpl.java:186)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.run(ODBServerImpl.java:166)
2010/11/29 08:38:43 | at java.lang.Thread.run(Thread.java:636)
2010/11/29 08:38:43 |
2010/11/29 08:38:43 | ODBServer:ODBServerImpl.startServer:java.net.SocketException: Too many open files
2010/11/29 08:38:43 | at java.net.PlainSocketImpl?.socketAccept(Native Method)
2010/11/29 08:38:43 | at java.net.AbstractPlainSocketImpl?.accept(AbstractPlainSocketImpl?.java:358)
2010/11/29 08:38:43 | at java.net.ServerSocket?.implAccept(ServerSocket?.java:470)
2010/11/29 08:38:43 | at java.net.ServerSocket?.accept(ServerSocket?.java:438)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.waitForRemoteConnection(ODBServerImpl.java:196)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.startServer(ODBServerImpl.java:186)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.run(ODBServerImpl.java:166)
2010/11/29 08:38:43 | at java.lang.Thread.run(Thread.java:636)
2010/11/29 08:38:43 |
2010/11/29 08:38:43 | ODBServer:ODBServerImpl.startServer:java.net.SocketException: Too many open files
2010/11/29 08:38:43 | at java.net.PlainSocketImpl?.socketAccept(Native Method)
2010/11/29 08:38:43 | at java.net.AbstractPlainSocketImpl?.accept(AbstractPlainSocketImpl?.java:358)
2010/11/29 08:38:43 | at java.net.ServerSocket?.implAccept(ServerSocket?.java:470)
2010/11/29 08:38:43 | at java.net.ServerSocket?.accept(ServerSocket?.java:438)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.waitForRemoteConnection(ODBServerImpl.java:196)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.startServer(ODBServerImpl.java:186)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.run(ODBServerImpl.java:166)
2010/11/29 08:38:43 | at java.lang.Thread.run(Thread.java:636)
2010/11/29 08:38:43 |
2010/11/29 08:38:43 | ODBServer:ODBServerImpl.startServer:java.net.SocketException: Too many open files
2010/11/29 08:38:43 | at java.net.PlainSocketImpl?.socketAccept(Native Method)
2010/11/29 08:38:43 | at java.net.AbstractPlainSocketImpl?.accept(AbstractPlainSocketImpl?.java:358)
2010/11/29 08:38:43 | at java.net.ServerSocket?.implAccept(ServerSocket?.java:470)
2010/11/29 08:38:43 | at java.net.ServerSocket?.accept(ServerSocket?.java:438)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.waitForRemoteConnection(ODBServerImpl.java:196)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.startServer(ODBServerImpl.java:186)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.run(ODBServerImpl.java:166)
2010/11/29 08:38:43 | at java.lang.Thread.run(Thread.java:636)
2010/11/29 08:38:43 |
2010/11/29 08:38:43 | ODBServer:ODBServerImpl.startServer:java.net.SocketException: Too many open files
2010/11/29 08:38:43 | at java.net.PlainSocketImpl?.socketAccept(Native Method)
2010/11/29 08:38:43 | at java.net.AbstractPlainSocketImpl?.accept(AbstractPlainSocketImpl?.java:358)
2010/11/29 08:38:43 | at java.net.ServerSocket?.implAccept(ServerSocket?.java:470)
2010/11/29 08:38:43 | at java.net.ServerSocket?.accept(ServerSocket?.java:438)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.waitForRemoteConnection(ODBServerImpl.java:196)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.startServer(ODBServerImpl.java:186)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.run(ODBServerImpl.java:166)
2010/11/29 08:38:43 | at java.lang.Thread.run(Thread.java:636)
2010/11/29 08:38:43 |
2010/11/29 08:38:43 | ODBServer:ODBServerImpl.startServer:java.net.SocketException: Too many open files
2010/11/29 08:38:43 | at java.net.PlainSocketImpl?.socketAccept(Native Method)
2010/11/29 08:38:43 | at java.net.AbstractPlainSocketImpl?.accept(AbstractPlainSocketImpl?.java:358)
2010/11/29 08:38:43 | at java.net.ServerSocket?.implAccept(ServerSocket?.java:470)
2010/11/29 08:38:43 | at java.net.ServerSocket?.accept(ServerSocket?.java:438)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.waitForRemoteConnection(ODBServerImpl.java:196)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.startServer(ODBServerImpl.java:186)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.run(ODBServerImpl.java:166)
2010/11/29 08:38:43 | at java.lang.Thread.run(Thread.java:636)
2010/11/29 08:38:43 |
2010/11/29 08:38:43 | ODBServer:ODBServerImpl.startServer:java.net.SocketException: Too many open files
2010/11/29 08:38:43 | at java.net.PlainSocketImpl?.socketAccept(Native Method)
2010/11/29 08:38:43 | at java.net.AbstractPlainSocketImpl?.accept(AbstractPlainSocketImpl?.java:358)
2010/11/29 08:38:43 | at java.net.ServerSocket?.implAccept(ServerSocket?.java:470)
2010/11/29 08:38:43 | at java.net.ServerSocket?.accept(ServerSocket?.java:438)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.waitForRemoteConnection(ODBServerImpl.java:196)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.startServer(ODBServerImpl.java:186)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.run(ODBServerImpl.java:166)
2010/11/29 08:38:43 | at java.lang.Thread.run(Thread.java:636)
2010/11/29 08:38:43 |
2010/11/29 08:38:43 | ODBServer:ODBServerImpl.startServer:java.net.SocketException: Too many open files
2010/11/29 08:38:43 | at java.net.PlainSocketImpl?.socketAccept(Native Method)
2010/11/29 08:38:43 | at java.net.AbstractPlainSocketImpl?.accept(AbstractPlainSocketImpl?.java:358)
2010/11/29 08:38:43 | at java.net.ServerSocket?.implAccept(ServerSocket?.java:470)
2010/11/29 08:38:43 | at java.net.ServerSocket?.accept(ServerSocket?.java:438)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.waitForRemoteConnection(ODBServerImpl.java:196)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.startServer(ODBServerImpl.java:186)
2010/11/29 08:38:43 | at org.neodatis.odb.core.server.ODBServerImpl.run(ODBServerImpl.java:166)
2010/11/29 08:38:43 | at java.lang.Thread.run(Thread.java:636)
2010/11/29 08:38:43 |
2010/11/29 08:38:50 | Exception in thread "Thread-43" java.lang.OutOfMemoryError?: GC overhead limit exceeded
2010/11/29 08:38:50 | at java.util.Arrays.copyOf(Arrays.java:2894)
2010/11/29 08:38:50 | at java.lang.AbstractStringBuilder?.expandCapacity(AbstractStringBuilder?.java:117)
2010/11/29 08:38:50 | at java.lang.AbstractStringBuilder?.append(AbstractStringBuilder?.java:407)
2010/11/29 08:38:50 | at java.lang.StringBuffer?.append(StringBuffer?.java:241)
2010/11/29 08:38:50 | at java.io.StringWriter?.write(StringWriter?.java:112)
2010/11/29 08:38:50 | at java.io.PrintWriter?.write(PrintWriter?.java:429)
2010/11/29 08:38:50 | at java.io.PrintWriter?.write(PrintWriter?.java:446)
2010/11/29 08:38:50 | at java.io.PrintWriter?.print(PrintWriter?.java:576)
2010/11/29 08:38:50 | at java.io.PrintWriter?.println(PrintWriter?.java:712)
2010/11/29 08:38:50 | at java.lang.Throwable.printStackTrace(Throwable.java:529)
2010/11/29 08:38:50 | at org.neodatis.tool.wrappers.OdbString?.exceptionToString(OdbString?.java:121)
2010/11/29 08:38:50 | at org.neodatis.odb.core.server.ODBServerImpl.startServer(ODBServerImpl.java:189)
2010/11/29 08:38:50 | at org.neodatis.odb.core.server.ODBServerImpl.run(ODBServerImpl.java:166)
2010/11/29 08:38:50 | at java.lang.Thread.run(Thread.java:636)
2010/11/29 08:38:50 | Error writing log, disk full? java.io.IOException: No space left on device
2010/11/29 08:39:00 | Error writing log, disk full? java.io.IOException: No space left on device
2010/11/29 08:39:07 | Exception in thread "I2CP Reader 3" java.lang.OutOfMemoryError?: GC overhead limit exceeded
....
2010/11/29 08:39:46 | java.lang.OutOfMemoryError?: GC overhead limit exceeded
...
2010/11/29 08:40:06 | 07:39:47.840 KRITISCH [NTCP Pumper ] net.i2p.router.Router : Shutdown(10) complete
2010/11/29 08:40:06 | Error flushing the records
2010/11/29 08:40:07 | 07:39:47.635 KRITISCH [W pumper 3/4] net.i2p.router.Router : Shutdown(10) complete
2010/11/29 08:40:07 | Error flushing the records
2010/11/29 08:40:08 | JVM exited unexpectedly.

Subtickets

Change History (11)

comment:1 Changed 8 years ago by zzz

  • Component changed from router/general to apps/plugins
  • Owner set to sponge
  • Status changed from new to assigned
  • Summary changed from router crash logs to JVM crash from seedless

It appears that either seedless or neodatis is looping immediately after an Exception in accept(). This tight loop eventually caused a GC overhead limit in the JVM. It's not clear from the above logs if there is a new thread spawned for each attempt or not.

After an Exception in accept() the runner should either sleep for a while or quit.

Reassigning to sponge.

There may be similar design patterns elsewhere in I2P, we should look for them also.

Of course, too many open files may be the root cause, but it should be handled better.

comment:2 Changed 8 years ago by guest

and i have wrapper.java.maxmemory=600

comment:3 Changed 8 years ago by sponge

Try removing the Seedless database files, and see if the loop still happens

comment:4 Changed 8 years ago by sponge

You are obviously running a heavily loaded system, and are out of File Descriptors.
cat /proc/sys/fs/file-max
That will give you the maximum allowed open files for a user.

To see the hard and soft values, issue the command as follows:
ulimit -Hn ; ulimit -Sn

To increase FD's system-wide...
sysctl -w fs.file-max=100000
100000 is an example, increase as needed...
See also your shell's man page.
See also any limits.conf file you may have in /etc

Let me know if this fixes the problem.

comment:5 Changed 8 years ago by sponge

One other diagnostic note...

lsof -p /usr/local/i2p/i2prouter status | cut -f2 -d"(" | cut -f1 -d ")" | wc -l
change the path to the i2prouter program as needed, this will print out how many active fd's you are running.

comment:6 Changed 8 years ago by sponge

great, the quotes didn't come out...
Let me try that again...
the part --
/usr/local/i2p/i2prouter status | cut -f2 -d"(" | cut -f1 -d ")"
goes into forward quotes... the one under the tilde... apparently this tracking software can't do forward quotes, or doesn't allow them.

comment:7 Changed 8 years ago by sponge

Try also...
A=$(ps waxf | grep java | grep Router | cut -f1 -d" ") ; lsof -n -p $A | wc -l
As that will give you an even more exacting count. The pid provided by the status command is wrong.

comment:8 Changed 8 years ago by sponge

And if that one gives you problems, this is even better
A=$(ps xf | cat | grep mort | grep Router) ; B=$(echo $A | cut -f1 -d" ") ; lsof -n -p $B | wc -l

comment:9 Changed 8 years ago by sponge

New seedless and neodatis plugins released, try them and see if it helps...
Now you should see warning messages and it shouldn't crash the JVM.

Please let me know. Thanks.

comment:10 Changed 8 years ago by sponge

  • Status changed from assigned to accepted

comment:11 Changed 8 years ago by sponge

  • Resolution set to fixed
  • Status changed from accepted to closed
Note: See TracTickets for help on using tickets.