Opened 4 years ago

Closed 22 months ago

#1768 closed defect (fixed)

all eepsites unreachable after a few hour

Reported by: Eche|on Owned by: zzz
Priority: critical Milestone: undecided
Component: apps/i2ptunnel Version: 0.9.24
Keywords: Cc:
Parent Tickets: Sensitive: no

Description

Hi!

Since a few days I cannot reach eepsites anymore. Running 0.9.24-4, I restart the router and all is fine. I go to sleep, 6h laters I try to reach any eppsites - not possible. All showing unreachable.
Only my local eepsite is reachable ⇒ i2ptunnel has issues.
The star is yellow for shared clients. The shared clients (DSA) star is green.
router.firstVersion=0.8.12
Floodfill participant=false

Subtickets

Change History (10)

comment:1 Changed 4 years ago by hummingbird

I had the same kind of issue on 14/02/2016:

  • eepsites unreachable
  • in "Local tunnels", only "shared clients" and "shared clients (DSA)" were yellow
  • nothing bad in http://127.0.0.1:7657/logs

However now (uptime: 32 hours), eepsites are reachable again. However iMule has still issues: it stops uploading without reason like if the remote(s) was down, then it continues.

I2P version: 0.9.24-5
Java version: Oracle Corporation 1.7.0_95 (OpenJDK Runtime Environment 1.7.0_95-b00)
Wrapper version: 3.5.25
Server version: 8.1.17.v20150415
Servlet version: Jasper JSP 2.1 Engine
Platform: Linux amd64 3.19.0-32-generic
Processor: athlon64
Jbigi: Locally optimized library libjbigi-linux-athlon64_64.so loaded from file
Encoding: UTF-8
Charset: UTF-8

Last edited 4 years ago by hummingbird (previous) (diff)

comment:2 Changed 3 years ago by str4d

Status: newopen

comment:3 Changed 3 years ago by Eche|on

Hi

did happened again, with 0.9.24-6:
/19/16 12:16:00 AM ERROR 2/5? uter.client.RequestLeaseSetJob?: Failed to receive a leaseSet in the time allotted (60001): leaseSet request asking for [LeaseSet?:

Destination: [Destination:
Hash: 2NX715eBcC30a54sPzmOe8YFErINlZ9ZX3sQACd580E=
Certificate: [Certificate: type: Null payload: null]
PublicKey?: [PublicKey: size: 256]
SigningPublicKey?: DSA_SHA1: size: 128?]
EncryptionKey?: null
SigningKey?: null
Signature: null
Leases: #1

[Lease:

End Date: Thu Feb 18 23:21:07 GMT 2016
Gateway: [Hash: JVNR3ZlLFnM9LZMudxKOe6EEdRMqzYEjzjF9V1~SfFI=]
TunnelId?: 3388215476]] having received null succeeding? false expiring on 1455837360339

2/19/16 12:29:08 AM ERROR [uterWatchdog] client.ClientManagerFacadeImpl?: Client 5MjdL3 has a leaseSet that expired 10m
2/19/16 12:29:08 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Ready and waiting jobs: 0
2/19/16 12:29:08 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Job lag: 0
2/19/16 12:29:08 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Participating tunnel count: 6
2/19/16 12:29:08 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: 1 minute send processing time: 79ms
2/19/16 12:29:08 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Outbound send rate: 153.55KBps
2/19/16 12:29:08 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Memory: 66.93MB / 1.33GB
2/19/16 12:29:08 AM CRIT [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Router appears hung, or there is severe network congestion. Watchdog starts barking!
2/19/16 12:29:09 AM CRIT [uterWatchdog] et.i2p.router.tasks.ThreadDump?: Threads dumped to wrapper log
2/19/16 12:30:09 AM ERROR [uterWatchdog] client.ClientManagerFacadeImpl?: Client 5MjdL3 has a leaseSet that expired 11m
2/19/16 12:30:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Ready and waiting jobs: 0
2/19/16 12:30:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Job lag: 0
2/19/16 12:30:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Participating tunnel count: 6
2/19/16 12:30:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: 1 minute send processing time: 85ms
2/19/16 12:30:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Outbound send rate: 85.75KBps
2/19/16 12:30:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Memory: 71.64MB / 1.33GB
2/19/16 12:31:09 AM ERROR [uterWatchdog] client.ClientManagerFacadeImpl?: Client 5MjdL3 has a leaseSet that expired 12m
2/19/16 12:31:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Ready and waiting jobs: 0
2/19/16 12:31:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Job lag: 0
2/19/16 12:31:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Participating tunnel count: 7
2/19/16 12:31:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: 1 minute send processing time: 136ms
2/19/16 12:31:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Outbound send rate: 64.26KBps

and so on forever:
/19/16 7:53:09 AM ERROR [uterWatchdog] client.ClientManagerFacadeImpl?: Client 5MjdL3 has a leaseSet that expired 7h
2/19/16 7:53:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Ready and waiting jobs: 0
2/19/16 7:53:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Job lag: 0
2/19/16 7:53:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Participating tunnel count: 3
2/19/16 7:53:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: 1 minute send processing time: 249ms
2/19/16 7:53:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Outbound send rate: 55.55KBps
2/19/16 7:53:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Memory: 65.14MB / 1.33GB
2/19/16 7:54:09 AM ERROR [uterWatchdog] client.ClientManagerFacadeImpl?: Client 5MjdL3 has a leaseSet that expired 7h
2/19/16 7:54:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Ready and waiting jobs: 0
2/19/16 7:54:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Job lag: 0
2/19/16 7:54:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Participating tunnel count: 3
2/19/16 7:54:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: 1 minute send processing time: 249ms
2/19/16 7:54:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Outbound send rate: 55.55KBps
2/19/16 7:54:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Memory: 75.18MB / 1.33GB
2/19/16 7:55:09 AM ERROR [uterWatchdog] client.ClientManagerFacadeImpl?: Client 5MjdL3 has a leaseSet that expired 7h
2/19/16 7:55:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Ready and waiting jobs: 0
2/19/16 7:55:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Job lag: 0
2/19/16 7:55:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Participating tunnel count: 3
2/19/16 7:55:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: 1 minute send processing time: 114ms
2/19/16 7:55:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Outbound send rate: 47.53KBps
2/19/16 7:55:09 AM ERROR [uterWatchdog] 2p.router.tasks.RouterWatchdog?: Memory: 43.23MB / 1.33GB

comment:4 Changed 3 years ago by zzz

I see zero changes in the client or router side of I2CP or in streaming. At this point it looks like either a SAM problem or a router-side deadlock. I will start by looking through the SAM changes that could affect v3.0 operation.

comment:5 Changed 3 years ago by zzz

I reviewed the dump sent to me by echelon, everything looks fine.
I reviewed the SAM v3.3 changes. I don't see anything that would cause this or affect 3.0 clients much at all.
I've run imule for many hours (but without much downloading or any uploading, just sitting there doing its DHT thing).

The dest 2NX715 is not a streaming dest, maybe the imule UDP?
The dest 5MjdL3 is a streaming dest, maybe the imule TCP?

Only the dest 5MjdL3 is in trouble.

If the I2CP queues are overflowing, we wouldn't know about it, all those errors are now warns (see #758 for background).

Setting logging net.i2p.router.client=WARN will log the overflows and other interesting things.

Not a lot of ideas right now.

There were no errors in the logs before the "/19/16 12:16:00 AM ERROR 2/5? uter.client.RequestLeaseSetJob??: Failed to receive a leaseSet" ??

what's the router doing? You say you can't reach eepsites? Anything to note under congestion? Green or yellow stars?

What's imule doing? does it look happy?

comment:6 Changed 3 years ago by Eche|on

To bad. I´ll set the logging option.
The router is doing some IRC partly, eepsites browsing, some snark. Tunnels green except shared clients, which was yellow, the shared clients (DSA) was green (AFAIK, or switched).
I do host echelon.i2p on the router, too.
But thats all on traffic/jobs.
iMule did looked happy so far with that error, juist eepsite browsing was the issue, no eepsite in reach.

comment:7 Changed 3 years ago by zzz

My current theory is that this is only a problem with close-on-idle non-DSA shared clients, that either in the shutdown or resume, it loses track of the subsession. If so, this probably isn't a new problem, and it could be fixed by changing to DSA or disabling close-on-idle. You'd have to make the change on all the shared clients i2ptunnel pages.

Just a guess, haven't tracked it down yet.

comment:8 Changed 3 years ago by zzz

Does disabling close-on-idle fix this for you?

comment:9 Changed 3 years ago by zzz

Status: opentesting

Possible fixes in 5fe2d118d532d53e41a925b17970db0a38c5c6c2 0.9.24-12-rc

I don't completely understand what's going on and still can't reproduce it here, but it's related to how we track the current leaseset and LS requests to the client in ClientConnectionRunner?. Rewrites in the past few months for subclients (DSA shared clients) may have caused the bug or made it happen more often.

I still think it's related to close-on-idle. The new-keys-on-reopen setting is probably also related. When reporting test results, please specify your settings for both.

comment:10 Changed 22 months ago by zzz

Resolution: fixed
Status: testingclosed

old ticket, presumed fixed

Note: See TracTickets for help on using tickets.