Opened 8 years ago

Closed 8 years ago

#536 closed defect (fixed)

NPE and AIOOBE in SSU

Reported by: slow Owned by: zzz
Priority: major Milestone: 0.8.10
Component: router/transport Version: 0.8.9
Keywords: Cc:
Parent Tickets: Sensitive: no

Description

Highly loaded router, it gets many tunnel build requests and backlog about 90 at peaks.

10/14/11 8:47:59 PM CRIT 1/1? net.i2p.router.JobQueueRunner? : Error processing job [Deferred tunnel join processing] on thread 0: null

java.lang.NullPointerException?
at java.util.concurrent.LinkedBlockingQueue?.offer(LinkedBlockingQueue?.java:368)
at net.i2p.router.transport.udp.OutboundEstablishState?.addMessage(OutboundEstablishState?.java:116)
at net.i2p.router.transport.udp.EstablishmentManager?.establish(EstablishmentManager?.java:217)
at net.i2p.router.transport.udp.UDPTransport.send(UDPTransport.java:1211)
at net.i2p.router.transport.GetBidsJob?.getBids(GetBidsJob?.java:80)
at net.i2p.router.transport.CommSystemFacadeImpl?.processMessage(CommSystemFacadeImpl?.java:135)
at net.i2p.router.OutNetMessagePool?.add(OutNetMessagePool?.java:53)
at net.i2p.router.tunnel.pool.BuildHandler?.handleReq(BuildHandler?.java:636)
at net.i2p.router.tunnel.pool.BuildHandler?.access$500(BuildHandler?.java:45)
at net.i2p.router.tunnel.pool.BuildHandler?$HandleReq?.runJob(BuildHandler?.java:353)
at net.i2p.router.JobQueueRunner?.runCurrentJob(JobQueueRunner?.java:142)
at net.i2p.router.JobQueueRunner?.run(JobQueueRunner?.java:82)
at java.lang.Thread.run(Thread.java:662)
at net.i2p.util.I2PThread.run(I2PThread.java:85)

10/14/11 7:40:21 PM CRIT 3/5? net.i2p.router.JobQueueRunner? : Error processing job [Deferred tunnel join processing] on thread 2: 2

java.lang.ArrayIndexOutOfBoundsException?: 2
at java.util.ArrayList?.add(ArrayList?.java:352)
at net.i2p.router.transport.udp.EstablishmentManager?.establish(EstablishmentManager?.java:181)
at net.i2p.router.transport.udp.UDPTransport.send(UDPTransport.java:1211)
at net.i2p.router.transport.GetBidsJob?.getBids(GetBidsJob?.java:80)
at net.i2p.router.transport.CommSystemFacadeImpl?.processMessage(CommSystemFacadeImpl?.java:135)
at net.i2p.router.OutNetMessagePool?.add(OutNetMessagePool?.java:53)
at net.i2p.router.tunnel.pool.BuildHandler?.handleReq(BuildHandler?.java:636)
at net.i2p.router.tunnel.pool.BuildHandler?.access$500(BuildHandler?.java:45)
at net.i2p.router.tunnel.pool.BuildHandler?$HandleReq?.runJob(BuildHandler?.java:353)
at net.i2p.router.JobQueueRunner?.runCurrentJob(JobQueueRunner?.java:142)
at net.i2p.router.JobQueueRunner?.run(JobQueueRunner?.java:82)
at java.lang.Thread.run(Thread.java:662)
at net.i2p.util.I2PThread.run(I2PThread.java:85)

10/14/11 2:17:47 AM CRIT [uildExecutor] uter.tunnel.pool.BuildExecutor?: B0rked in the tunnel builder

java.lang.NullPointerException?
at java.util.concurrent.LinkedBlockingQueue?.offer(LinkedBlockingQueue?.java:368)
at net.i2p.router.transport.udp.OutboundEstablishState?.addMessage(OutboundEstablishState?.java:116)
at net.i2p.router.transport.udp.EstablishmentManager?.establish(EstablishmentManager?.java:217)
at net.i2p.router.transport.udp.UDPTransport.send(UDPTransport.java:1211)
at net.i2p.router.transport.GetBidsJob?.getBids(GetBidsJob?.java:80)
at net.i2p.router.transport.CommSystemFacadeImpl?.processMessage(CommSystemFacadeImpl?.java:135)
at net.i2p.router.OutNetMessagePool?.add(OutNetMessagePool?.java:53)
at net.i2p.router.tunnel.pool.BuildHandler?.handleReq(BuildHandler?.java:636)
at net.i2p.router.tunnel.pool.BuildHandler?.handleRequest(BuildHandler?.java:310)
at net.i2p.router.tunnel.pool.BuildHandler?.handleInboundRequests(BuildHandler?.java:131)
at net.i2p.router.tunnel.pool.BuildExecutor?.run(BuildExecutor?.java:368)
at java.lang.Thread.run(Thread.java:662)
at net.i2p.util.I2PThread.run(I2PThread.java:85)

10/14/11 1:32:13 AM CRIT 1/1? net.i2p.router.JobQueueRunner? : Error processing job [Deferred tunnel join processing] on thread 0: 2

java.lang.ArrayIndexOutOfBoundsException?: 2
at java.util.ArrayList?.add(ArrayList?.java:352)
at net.i2p.router.transport.udp.EstablishmentManager?.establish(EstablishmentManager?.java:181)
at net.i2p.router.transport.udp.UDPTransport.send(UDPTransport.java:1211)
at net.i2p.router.transport.GetBidsJob?.getBids(GetBidsJob?.java:80)
at net.i2p.router.transport.CommSystemFacadeImpl?.processMessage(CommSystemFacadeImpl?.java:135)
at net.i2p.router.OutNetMessagePool?.add(OutNetMessagePool?.java:53)
at net.i2p.router.tunnel.pool.BuildHandler?.handleReq(BuildHandler?.java:636)
at net.i2p.router.tunnel.pool.BuildHandler?.access$500(BuildHandler?.java:45)
at net.i2p.router.tunnel.pool.BuildHandler?$HandleReq?.runJob(BuildHandler?.java:353)
at net.i2p.router.JobQueueRunner?.runCurrentJob(JobQueueRunner?.java:142)
at net.i2p.router.JobQueueRunner?.run(JobQueueRunner?.java:82)
at java.lang.Thread.run(Thread.java:662)
at net.i2p.util.I2PThread.run(I2PThread.java:85)

10/13/11 11:56:00 PM CRIT [uildExecutor] uter.tunnel.pool.BuildExecutor?: B0rked in the tunnel builder

java.lang.NullPointerException?
at java.util.concurrent.LinkedBlockingQueue?.offer(LinkedBlockingQueue?.java:368)
at net.i2p.router.transport.udp.OutboundEstablishState?.addMessage(OutboundEstablishState?.java:116)
at net.i2p.router.transport.udp.EstablishmentManager?.establish(EstablishmentManager?.java:217)
at net.i2p.router.transport.udp.UDPTransport.send(UDPTransport.java:1211)
at net.i2p.router.transport.GetBidsJob?.getBids(GetBidsJob?.java:80)
at net.i2p.router.transport.CommSystemFacadeImpl?.processMessage(CommSystemFacadeImpl?.java:135)
at net.i2p.router.OutNetMessagePool?.add(OutNetMessagePool?.java:53)
at net.i2p.router.tunnel.pool.BuildHandler?.handleReq(BuildHandler?.java:636)
at net.i2p.router.tunnel.pool.BuildHandler?.handleRequest(BuildHandler?.java:310)
at net.i2p.router.tunnel.pool.BuildHandler?.handleInboundRequests(BuildHandler?.java:131)
at net.i2p.router.tunnel.pool.BuildExecutor?.run(BuildExecutor?.java:368)
at java.lang.Thread.run(Thread.java:662)
at net.i2p.util.I2PThread.run(I2PThread.java:85)

10/13/11 10:09:52 PM CRIT 2/5? net.i2p.router.JobQueueRunner? : Error processing job [Send Message Direct] on thread 1: 2

java.lang.ArrayIndexOutOfBoundsException?: 2
at java.util.ArrayList?.add(ArrayList?.java:352)
at net.i2p.router.transport.udp.EstablishmentManager?.establish(EstablishmentManager?.java:181)
at net.i2p.router.transport.udp.UDPTransport.send(UDPTransport.java:1211)
at net.i2p.router.transport.GetBidsJob?.getBids(GetBidsJob?.java:80)
at net.i2p.router.transport.CommSystemFacadeImpl?.processMessage(CommSystemFacadeImpl?.java:135)
at net.i2p.router.OutNetMessagePool?.add(OutNetMessagePool?.java:53)
at net.i2p.router.message.SendMessageDirectJob?.send(SendMessageDirectJob?.java:155)
at net.i2p.router.message.SendMessageDirectJob?.runJob(SendMessageDirectJob?.java:90)
at net.i2p.router.JobQueueRunner?.runCurrentJob(JobQueueRunner?.java:142)
at net.i2p.router.JobQueueRunner?.run(JobQueueRunner?.java:82)
at java.lang.Thread.run(Thread.java:662)
at net.i2p.util.I2PThread.run(I2PThread.java:85)

10/13/11 8:55:36 PM CRIT [uildExecutor] uter.tunnel.pool.BuildExecutor?: B0rked in the tunnel builder

java.lang.NullPointerException?
at java.util.concurrent.LinkedBlockingQueue?.offer(LinkedBlockingQueue?.java:368)
at net.i2p.router.transport.udp.OutboundEstablishState?.addMessage(OutboundEstablishState?.java:116)
at net.i2p.router.transport.udp.EstablishmentManager?.establish(EstablishmentManager?.java:217)
at net.i2p.router.transport.udp.UDPTransport.send(UDPTransport.java:1211)
at net.i2p.router.transport.GetBidsJob?.getBids(GetBidsJob?.java:80)
at net.i2p.router.transport.CommSystemFacadeImpl?.processMessage(CommSystemFacadeImpl?.java:135)
at net.i2p.router.OutNetMessagePool?.add(OutNetMessagePool?.java:53)
at net.i2p.router.tunnel.pool.BuildHandler?.handleReq(BuildHandler?.java:636)
at net.i2p.router.tunnel.pool.BuildHandler?.handleRequest(BuildHandler?.java:310)
at net.i2p.router.tunnel.pool.BuildHandler?.handleInboundRequests(BuildHandler?.java:131)
at net.i2p.router.tunnel.pool.BuildExecutor?.run(BuildExecutor?.java:368)
at java.lang.Thread.run(Thread.java:662)
at net.i2p.util.I2PThread.run(I2PThread.java:85)

10/13/11 8:55:22 PM CRIT 3/5? net.i2p.router.JobQueueRunner? : Error processing job [Send Message Direct] on thread 2: 2

java.lang.ArrayIndexOutOfBoundsException?: 2
at java.util.ArrayList?.add(ArrayList?.java:352)
at net.i2p.router.transport.udp.EstablishmentManager?.establish(EstablishmentManager?.java:181)
at net.i2p.router.transport.udp.UDPTransport.send(UDPTransport.java:1211)
at net.i2p.router.transport.GetBidsJob?.getBids(GetBidsJob?.java:80)
at net.i2p.router.transport.CommSystemFacadeImpl?.processMessage(CommSystemFacadeImpl?.java:135)
at net.i2p.router.OutNetMessagePool?.add(OutNetMessagePool?.java:53)
at net.i2p.router.message.SendMessageDirectJob?.send(SendMessageDirectJob?.java:155)
at net.i2p.router.message.SendMessageDirectJob?.runJob(SendMessageDirectJob?.java:90)
at net.i2p.router.JobQueueRunner?.runCurrentJob(JobQueueRunner?.java:142)
at net.i2p.router.JobQueueRunner?.run(JobQueueRunner?.java:82)
at java.lang.Thread.run(Thread.java:662)
at net.i2p.util.I2PThread.run(I2PThread.java:85)

10/13/11 6:59:04 PM CRIT [uterWatchdog] net.i2p.router.RouterWatchdog? : Router appears hung, or there is severe network congestion. Watchdog starts barking!
10/13/11 1:56:47 PM CRIT 3/5? net.i2p.router.JobQueueRunner? : Error processing job [Handle Database Lookup Message] on thread 2: null

java.lang.NullPointerException?
at java.util.concurrent.LinkedBlockingQueue?.offer(LinkedBlockingQueue?.java:368)
at net.i2p.router.transport.udp.OutboundEstablishState?.addMessage(OutboundEstablishState?.java:116)
at net.i2p.router.transport.udp.EstablishmentManager?.establish(EstablishmentManager?.java:217)
at net.i2p.router.transport.udp.UDPTransport.send(UDPTransport.java:1211)
at net.i2p.router.transport.GetBidsJob?.getBids(GetBidsJob?.java:80)
at net.i2p.router.transport.CommSystemFacadeImpl?.processMessage(CommSystemFacadeImpl?.java:135)
at net.i2p.router.OutNetMessagePool?.add(OutNetMessagePool?.java:53)
at net.i2p.router.message.SendMessageDirectJob?.send(SendMessageDirectJob?.java:155)
at net.i2p.router.message.SendMessageDirectJob?.runJob(SendMessageDirectJob?.java:90)
at net.i2p.router.networkdb.HandleDatabaseLookupMessageJob?.sendThroughTunnel(HandleDatabaseLookupMessageJob?.java:292)
at net.i2p.router.networkdb.HandleDatabaseLookupMessageJob?.sendMessage(HandleDatabaseLookupMessageJob?.java:267)
at net.i2p.router.networkdb.HandleDatabaseLookupMessageJob?.sendClosest(HandleDatabaseLookupMessageJob?.java:262)
at net.i2p.router.networkdb.kademlia.HandleFloodfillDatabaseLookupMessageJob?.sendClosest(HandleFloodfillDatabaseLookupMessageJob?.java:52)
at net.i2p.router.networkdb.HandleDatabaseLookupMessageJob?.runJob(HandleDatabaseLookupMessageJob?.java:188)
at net.i2p.router.JobQueueRunner?.runCurrentJob(JobQueueRunner?.java:142)
at net.i2p.router.JobQueueRunner?.run(JobQueueRunner?.java:82)
at java.lang.Thread.run(Thread.java:662)
at net.i2p.util.I2PThread.run(I2PThread.java:85)

10/13/11 1:56:15 PM CRIT 2/5? net.i2p.router.JobQueueRunner? : Error processing job [Handle Database Lookup Message] on thread 1: 2

java.lang.ArrayIndexOutOfBoundsException?: 2
at java.util.ArrayList?.add(ArrayList?.java:352)
at net.i2p.router.transport.udp.EstablishmentManager?.establish(EstablishmentManager?.java:181)
at net.i2p.router.transport.udp.UDPTransport.send(UDPTransport.java:1211)
at net.i2p.router.transport.GetBidsJob?.getBids(GetBidsJob?.java:80)
at net.i2p.router.transport.CommSystemFacadeImpl?.processMessage(CommSystemFacadeImpl?.java:135)
at net.i2p.router.OutNetMessagePool?.add(OutNetMessagePool?.java:53)
at net.i2p.router.message.SendMessageDirectJob?.send(SendMessageDirectJob?.java:155)
at net.i2p.router.message.SendMessageDirectJob?.runJob(SendMessageDirectJob?.java:90)
at net.i2p.router.networkdb.HandleDatabaseLookupMessageJob?.sendThroughTunnel(HandleDatabaseLookupMessageJob?.java:292)
at net.i2p.router.networkdb.HandleDatabaseLookupMessageJob?.sendMessage(HandleDatabaseLookupMessageJob?.java:267)
at net.i2p.router.networkdb.HandleDatabaseLookupMessageJob?.sendData(HandleDatabaseLookupMessageJob?.java:245)
at net.i2p.router.networkdb.HandleDatabaseLookupMessageJob?.runJob(HandleDatabaseLookupMessageJob?.java:170)
at net.i2p.router.JobQueueRunner?.runCurrentJob(JobQueueRunner?.java:142)
at net.i2p.router.JobQueueRunner?.run(JobQueueRunner?.java:82)
at java.lang.Thread.run(Thread.java:662)
at net.i2p.util.I2PThread.run(I2PThread.java:85)

Subtickets

Change History (4)

comment:1 Changed 8 years ago by zzz

Component: unspecifiedrouter/transport
Owner: set to zzz
Priority: minormajor
Status: newaccepted

Good grief. It looks like UDP exploded.

comment:2 Changed 8 years ago by zzz

Summary: java.lang.NullPointerException at BuildExecutorNPE and AIOOBE in SSU

comment:3 Changed 8 years ago by zzz

This looks like unsynchronized ArrayList? accesses in EstablishmentManager?. Bug introduced in 0.7.13, checkin from March 2010.

comment:4 Changed 8 years ago by zzz

Resolution: fixed
Status: acceptedclosed

Hopefully fixed in 0.8.9-1.

Note: See TracTickets for help on using tickets.