Opened 21 months ago

Closed 7 months ago

#2079 closed defect (fixed)

Orchid fails post-install

Reported by: Reportage Owned by: zzz
Priority: major Milestone: 0.9.38
Component: apps/plugins Version: 0.9.32
Keywords: orchid Cc: Masayuki Hatta
Parent Tickets: Sensitive: no

Description (last modified by zzz)

I2P: 0.9.32
Platform: Linux amd64 4.13.0-17-generic
Java: Oracle Corporation 1.8.0_151
Orchid: 1.2.2-0.2-b1

2017/11 | Nov 2017 com.subgraph.orchid.circuits.CircuitBuildTask run
2017/11 | WARNING: Unexpected exception while building circuit: java.lang.IllegalStateException: ConnectionCache has been closed
2017/11 | java.lang.IllegalStateException: ConnectionCache has been closed
2017/11 | 	at com.subgraph.orchid.connections.ConnectionCacheImpl.getConnectionTo(ConnectionCacheImpl.java:115)
2017/11 | 	at com.subgraph.orchid.circuits.CircuitBuildTask.openEntryNodeConnection(CircuitBuildTask.java:96)
2017/11 | 	at com.subgraph.orchid.circuits.CircuitBuildTask.run(CircuitBuildTask.java:48)
2017/11 | 	at com.subgraph.orchid.circuits.CircuitManagerImpl.tryOpenCircuit(CircuitManagerImpl.java:440)
2017/11 | 	at com.subgraph.orchid.circuits.CircuitManagerImpl.openDirectoryCircuit(CircuitManagerImpl.java:293)
2017/11 | 	at com.subgraph.orchid.directory.downloader.DirectoryDownloaderImpl.openCircuit(DirectoryDownloaderImpl.java:131)
2017/11 | 	at com.subgraph.orchid.directory.downloader.DirectoryDownloaderImpl.downloadCurrentConsensus(DirectoryDownloaderImpl.java:76)
2017/11 | 	at com.subgraph.orchid.directory.downloader.DirectoryDownloadTask$DownloadConsensusTask.run(DirectoryDownloadTask.java:178)
2017/11 | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2017/11 | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2017/11 | 	at java.lang.Thread.run(Thread.java:748)
2017/11 | 
2017/11 | Nov 2017 com.subgraph.orchid.directory.downloader.DirectoryDownloadTask$DownloadConsensusTask run
2017/11 | WARNING: Failed to download current consensus document: Failed to open directory circuit
2017/11 | Nov 2017 com.subgraph.orchid.circuits.CircuitBuildTask run
2017/11 | WARNING: Unexpected exception while building circuit: java.lang.IllegalStateException: ConnectionCache has been closed
2017/11 | java.lang.IllegalStateException: ConnectionCache has been closed
2017/11 | 	at com.subgraph.orchid.connections.ConnectionCacheImpl.getConnectionTo(ConnectionCacheImpl.java:115)
2017/11 | 	at com.subgraph.orchid.circuits.CircuitBuildTask.openEntryNodeConnection(CircuitBuildTask.java:96)
2017/11 | 	at com.subgraph.orchid.circuits.CircuitBuildTask.run(CircuitBuildTask.java:48)
2017/11 | 	at com.subgraph.orchid.circuits.CircuitManagerImpl.tryOpenCircuit(CircuitManagerImpl.java:440)
2017/11 | 	at com.subgraph.orchid.circuits.CircuitManagerImpl.openDirectoryCircuit(CircuitManagerImpl.java:293)
2017/11 | 	at com.subgraph.orchid.directory.downloader.DirectoryDownloaderImpl.openCircuit(DirectoryDownloaderImpl.java:131)
2017/11 | 	at com.subgraph.orchid.directory.downloader.DirectoryDownloaderImpl.downloadCurrentConsensus(DirectoryDownloaderImpl.java:76)
2017/11 | 	at com.subgraph.orchid.directory.downloader.DirectoryDownloadTask$DownloadConsensusTask.run(DirectoryDownloadTask.java:178)
2017/11 | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2017/11 | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2017/11 | 	at java.lang.Thread.run(Thread.java:748)
2017/11 | 
2017/11 | Nov 2017 com.subgraph.orchid.circuits.CircuitBuildTask run
2017/11 | WARNING: Unexpected exception while building circuit: java.lang.IllegalStateException: ConnectionCache has been closed
2017/11 | java.lang.IllegalStateException: ConnectionCache has been closed
2017/11 | 	at com.subgraph.orchid.connections.ConnectionCacheImpl.getConnectionTo(ConnectionCacheImpl.java:115)
2017/11 | 	at com.subgraph.orchid.circuits.CircuitBuildTask.openEntryNodeConnection(CircuitBuildTask.java:96)
2017/11 | 	at com.subgraph.orchid.circuits.CircuitBuildTask.run(CircuitBuildTask.java:48)
2017/11 | 	at com.subgraph.orchid.circuits.CircuitManagerImpl.tryOpenCircuit(CircuitManagerImpl.java:440)
2017/11 | 	at com.subgraph.orchid.circuits.CircuitManagerImpl.openDirectoryCircuit(CircuitManagerImpl.java:293)
2017/11 | 	at com.subgraph.orchid.directory.downloader.DirectoryDownloaderImpl.openCircuit(DirectoryDownloaderImpl.java:131)
2017/11 | 	at com.subgraph.orchid.directory.downloader.DirectoryDownloaderImpl.downloadCurrentConsensus(DirectoryDownloaderImpl.java:76)
2017/11 | 	at com.subgraph.orchid.directory.downloader.DirectoryDownloadTask$DownloadConsensusTask.run(DirectoryDownloadTask.java:178)
2017/11 | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2017/11 | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2017/11 | 	at java.lang.Thread.run(Thread.java:748)

Subtickets

Change History (19)

comment:1 Changed 21 months ago by zzz

Component: otherapps/plugins
Description: modified (diff)
Owner: set to zzz
Status: newaccepted

comment:2 Changed 21 months ago by zzz

Status: acceptedinfoneeded

Not sure exactly what's happening here, if it's a startup thing or a post-shutdown thing.

Was this immediately after installing and running?
Did you try to stop the plugin?
These are warnings, not errors - did you change the log level to see these?

comment:3 Changed 21 months ago by Reportage

Status: infoneededopen

The errors appear on shutdown. No log level change. Events as follows:

  • install orchid
  • open web interface, monitor for circuit build status
  • leave running for ~ 10 minutes, note lack of built circuits
  • shutdown plugin

Looks like the perennial directory consensus issue - has something related to this changed at Tor HQ?

  • WARNING: Failed to download current consensus document: Failed to open directory circuit

With basic logging enabled, the following (on startup):

  • [INFO] wnloader.DirectoryDownloadTask?: Downloading consensus because we have no consensus document
  • [WARN] orchid.directory.DirectoryImpl?: Unable to verify signatures on consensus document, discarding…

With debug level logging:

Last edited 21 months ago by Reportage (previous) (diff)

comment:4 Changed 21 months ago by zzz

Status: openaccepted

Thanks. So the IllegalStateExceptions? are just flakiness at shutdown, the real problem is the consensus. But I'll change the excption so it doesn't spam the log

I do need to update the hardcoded dirauths and do a new release, hopefully that will fix it. The max consensus age is still way too high, I already lowered it in the code but haven't released that either. See #1220

The plugin is not in good shape right now, it's still really flaky, nobody is maintaining it upstream, and we aren't Tor experts. At one point bitcoinj was maintaining a fork but I think they abandoned it. Some recent discussion is at http://zzz.i2p/topics/2031-proposal-bundle-orchid?page=2

I'll try to get a new version out before the .33 release so it will get picked up when the routers upgrade.

comment:5 Changed 21 months ago by zzz

See also #1937

comment:6 Changed 21 months ago by zzz

I locally updated the consensus list from Tor 0.3.1.8 and it didn't help, can't get a consensus from any of the 9. The furthest any of them get is "Finishing handshake with directory server 10%". Not sure what's going on. For further research.

IllegalStateException? changed in 5036136333dfee5ff6ebec5bf5d52c35b315a561

comment:8 Changed 20 months ago by Reportage

As suggested elsewhere, a native Tor instance + router console hooks might be a better option going forward, not least because of the poor state of upstream orchid.

https://github.com/guardianproject/jtorctl in conjunction with native Tor might be considered. https://github.com/thaliproject/Tor_Onion_Proxy_Library as a wrapper to the Tor binary also looks interesting, and allows hosting of hidden services.

Last edited 20 months ago by Reportage (previous) (diff)

comment:9 Changed 17 months ago by Masayuki Hatta

I basically agree with Reportage, if you can use the original Tor directly somehow, it would be better.

Anyway, I fixed most of showstopper bugs in Orchid, and now it works again. Since Orchid is an abandonware, I assume the maintainership of it for now.

Please try my fork: https://github.com/mhatta/Orchid

I still have some difficulty to build the I2P Orchid plugin by myself, so I can't provide the updated plugin now.

comment:10 Changed 16 months ago by zzz

Milestone: 0.9.330.9.35

@mhatta we need to discuss this on IRC, where I gave you some info but haven't heard back.

orchid here != orchid plugin. I will need to release it as a different signer for an existing plugin isn't allowed. I gave you some info on the early 2016 changes and release. You may wish to buy some of those changes as back as I don't believe they are anywhere on github. I also have some changes pending for the next release, some checked-in in late 2017, some not, addressing the issues in this ticket. If you have additional changes please provide a patch. The plugin source is in mtn only right now, but we can get it bridged to github if we can find the right guy to talk to do it. I can also help you with building the plugin.

Whenever you're ready to figure things out please find me on IRC.

comment:11 Changed 16 months ago by Masayuki Hatta

@zzz, I'm a bit confused by your comment, so try to write my understanding down (sorry for verbosity).

1) I think orchid here == orchid plugin, right? There is Orchid the library/standalone client, and the I2P Orchid plugin. The I2P Orchid plugin contains the source code of the Orchid library under src/java/com/subgraph/orchid, on par with src/main/java/subgraph/orchid in the source code of Orchid library.

2) The latest source code of the Orchid library is available at github repo: https://github.com/mhatta/Orchid. The upstream development of Orchid seems ended long ago, so I guess I'm the maintainer for now.

3) The latest source code of the I2P Orchid plugin is available at monotone repo (i2p.plugins.orchid). @zzz is the maintainer who solely can sign the plugin binary (su3) by using scripts/makeplugin.sh. I understand there are your not-committed-yet fixes for the I2P Orchid plugin outside this repo.

4) Until recently, the Orchid library couldn't build the circuit at all (reported at https://github.com/subgraph/Orchid/issues/33). It was caused by the upstream introduction of new cert types. This is already fixed in my Orchid library repo, I can provide patches. The code in the current i2p.plugins.orchid doesn't have this kind of fix, but your not-committed-yet patches might have the similar fix.

5) Bug #2079 (of I2P Orchid plugin) might be caused by the aforementioned bug of the Orchid library code included in the current I2P Orchid plugin.

6) I'm not really familiar with monotone, but can use and okay with it now. I obtained i2p.plugins.orchid already.

7) However, I can't build the plugin now. The target "precompilejsp" failed. I'm still investigating.

comment:12 Changed 16 months ago by zzz

1-3) true
4-5) Indeed, the pending fixes in my workspace included the certificate change from issue 33. I just checked them in, untested. I'll try to test it soon, please take a look.
6-7) happy to help on IRC

On further research, looks like I was working on this in November and either was having trouble testing, or just got distracted and forgot about it. Hopefully with your help I can get it finished and released.

comment:13 Changed 16 months ago by zzz

Resolution: fixed
Status: acceptedclosed

Fixed in 1.2.2-0.3 released today, available at http://stats.i2p/i2p/plugins/
More details at http://zzz.i2p/topics/2563
Good thing I gave up in November, the necessary fix is this one by Peergos from February:

https://github.com/Peergos/Orchid/commit/d216480ae03115ff807e48347c634a2f0c242a0b

Found and adapted by mhatta at:

https://github.com/mhatta/Orchid/commit/56f602da5053405ef3e28c80ffa328602f7e5743

As discussed above several months ago, this release contains updates to the hardcoded dirauths, hides the spurious shutdown errors in the OP, and changes to deal with outdated consensus on disk.

The last issue is still not completely fixed, and you may need to stop and restart the plugin after a few minutes if it hasn't completely transitioned to the RUNNING state by then. See tix #1220 #1937

comment:14 Changed 15 months ago by Reportage

Resolution: fixed
Status: closedreopened

After not running Orchid for a while, after the previous fix, it has now stopped working. Running for several hours, or restarting the plugin, does not fix the issue.. it steadfastly refuses to connect to the network. Nothing much in the logs save the probable indication that the directory info is stale.
INFO […er worker-0] …cuitCreationTask: Cannot build circuits because we don't have enough directory information

Update: Uninstalling Orchid and then reinstalling has the same problem, so probably another directory server update upstream?

Last edited 15 months ago by Reportage (previous) (diff)

comment:15 Changed 15 months ago by zzz

similar reports at http://zzz.i2p/topics/2563

comment:16 Changed 9 months ago by zzz

Cc: Masayuki Hatta added
Milestone: 0.9.350.9.38
Status: reopenedaccepted

mhatta reports that the following change fixes things:

https://github.com/mhatta/Orchid/commit/8ba3e60f6eae3a9589e5212f77fb6aec9b5d8a40

He says:

Insufficient MANDATORY_CIPHERS causes the failure of the first TLS handshake with Directory Authorities. That's why Orchid now hang up before establishing a circuit

My comments:

We have something similar in i2p, in core/java/src/net/i2p/util/I2PSSLSocketFactory.java, but we use a blacklist to get rid of the weak ciphers. Orchid uses a whitelist for the good ciphers, and what's apparently happened is that the whitelist was so short that there was nothing left.

When you do the socket.setEnabledCipherSuites() call, the list you give it has to match something that the JVM supports, and the far-end of the SSL connection supports. AND it has to be secure. It does NOT have to be a list of everything the JVM supports. Where did you get the list of ciphers to add? From JVM documentation? You included a LOT of insecure ciphers - 3DES, DH_anon, KRB5, NULL (!) - that's not the way to do it.

We want to add ONLY the newer and secure ciphers. The problem is that some new more secure ones got added, and all the ones in the orchid list probably got removed - either disabled by default or removed in the JVM, or removed in Tor. See src/common/tortls.c in the tor source. So we have to figure out which are the good ones, and get rid of the bad ones. That's why we only use a blacklist in java i2p - so we don't prevent the newer ciphers from being used.

I don't have the link to the oracle doc for 10/11, Here's the links for 7, 8, and 9: https://docs.oracle.com/javase/7/docs/technotes/guides/security/SunProviders.html https://docs.oracle.com/javase/8/docs/technotes/guides/security/SunProviders.html#SunJSSEProvider https://docs.oracle.com/javase/9/security/oracleproviders.htm

I think the orchid code is flawed. According to javadocs https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/SSLSocket.html the setEnabledCipherSuites() call must contain only items from the getSupportedCipherSuites() all. That's why the I2P version of this class is a lot more complex than the Orchid version. Orchid never calls getSupportedCipherSuites(). So a huge list of enabled ciphers is problematic not just for security but because if any one of those is not in the JVM the call will fail, according to the javadocs.

Is there a Tor doc that defines the ciphers they use?

I think I'll just use the I2PSSLSocketFactory in the orchid plugin, but needs more research. Haven't tested yet, but will soon.

comment:17 Changed 8 months ago by Masayuki Hatta

I a bit investigated the strength of ciphers in OpenJDK:

http://www.mhatta.org/good-ciphers-in-openjdk10.html

And updated the listing in Orchid MANDATORY_CIPHERS.

https://github.com/mhatta/Orchid/blob/master/src/main/java/com/subgraph/orchid/connections/ConnectionSocketFactory.java

I think the blacklisting (weak ciphers) approach is better, but for now how about fixing the whitelist? Maybe not the optimal solution, but still better to have working I2P Orchid plugin.

comment:18 Changed 8 months ago by zzz

I've done some preliminary research and read the tor spec talking about behavior with the various TLS versions. I do plan to update the plugin, using the blacklist strategy, but I need to make sure I understand how the fallbacks work and that the spec is followed. I haven't done any code changes or testing yet.

My plan is to finish the work in January, after CCC, and before we release 0.9.38. That way users will pick up the new plugin version when they update their routers.

comment:19 Changed 7 months ago by zzz

Resolution: fixed
Status: acceptedclosed

Fixed as described above in ca06d008ced5e6384d8f3de01ac8212380153731 1.2.2-0.4-b1

Note: See TracTickets for help on using tickets.