Opened 4 years ago

Closed 4 years ago

#1671 closed enhancement (fixed)

Increase MAX_FILES_PER_TORRENT and MAX_PIECE_SIZE

Reported by: Obscuratus Owned by: zzz
Priority: minor Milestone: 0.9.23
Component: apps/i2psnark Version: 0.9.22
Keywords: Cc:
Parent Tickets:

Description (last modified by dg)

Increase the maximum files per torrent and maximum piece size limits (defined in MAX_FILES_PER_TORRENT and MAX_PIECE_SIZE). I have been locally patching these limits as follows since 0.9.20, and have not encountered any i2psnark instability. Of course these torrents will usually take more time, but I haven't been able to break i2psnark with these changes incorporated.

For MAX_FILES_PER_TORRENT, I choose 896 since there was some mention in the comments about possible issues with a value over 1000. I believe the value can be increased over 1000, but I have thoroughly tested a value of 896 without issues.

diff -Nurp i2p-0.9.19/apps/i2psnark/java/src/org/klomp/snark/SnarkManager.java i2p-0.9.19.new/apps/i2psnark/java/src/org/klomp/snark/SnarkManager.java
--- i2p-0.9.19/apps/i2psnark/java/src/org/klomp/snark/SnarkManager.java	2015-04-12 19:17:07.824323961 -0500
+++ i2p-0.9.19.new/apps/i2psnark/java/src/org/klomp/snark/SnarkManager.java	2015-04-21 13:45:09.244554716 -0500
@@ -1062,7 +1062,8 @@ public class SnarkManager implements Com
     }
     
     /** hardcoded for sanity.  perhaps this should be customizable, for people who increase their ulimit, etc. */
-    public static final int MAX_FILES_PER_TORRENT = 512;
+    /* public static final int MAX_FILES_PER_TORRENT = 512; */
+    public static final int MAX_FILES_PER_TORRENT = 896;
     
     /**
      *  Set of canonical .torrent filenames that we are dealing with.


diff -Nurp i2p-0.9.19/apps/i2psnark/java/src/org/klomp/snark/Storage.java i2p-0.9.19.new/apps/i2psnark/java/src/org/klomp/snark/Storage.java
--- i2p-0.9.19/apps/i2psnark/java/src/org/klomp/snark/Storage.java	2015-04-12 19:17:07.825323961 -0500
+++ i2p-0.9.19.new/apps/i2psnark/java/src/org/klomp/snark/Storage.java	2015-05-05 09:55:05.431064437 -0500
@@ -74,7 +74,8 @@ public class Storage
   /** The default piece size. */
   private static final int DEFAULT_PIECE_SIZE = 256*1024;
   /** bigger than this will be rejected */
-  public static final int MAX_PIECE_SIZE = 8*1024*1024;
+  /* public static final int MAX_PIECE_SIZE = 8*1024*1024; */
+  public static final int MAX_PIECE_SIZE = 16*1024*1024;
   /** The maximum number of pieces in a torrent. */
   public static final int MAX_PIECES = 10*1024;
   public static final long MAX_TOTAL_SIZE = MAX_PIECE_SIZE * (long) MAX_PIECES;

Subtickets

Change History (5)

comment:1 Changed 4 years ago by echelon

Moin

The values are quite sane for the slow I2P network and the ressource consumpting java snark client.
Use vuze with I2P plugin if you like higher limits, please.
Also some trackers do enforce these limits (e.g. postmans) and changing in source will not give you the experience you want.

E.G. a max piece size of 16 instead of 16 MB will increase the amount of retransmission and loas inside of I2P by the factor of 2 or more, as not often a piece of 16 MB will be transmitted through I2P without any error. Smaller pieces will increase the chance of successfull transfer.
The better I2P behave, the better the chances are, the earlier we can increase this limit. But do you have any stats available on retransmitting a 8MB and a 16MB piece in I2P currently?
Max Amount of pieces is mostly a limitation of ressources a system runs on, increasing to 1024 could be useful on some systems, but does it work well on all systems I2P does support?
Please test.

comment:2 Changed 4 years ago by dg

  • Description modified (diff)

comment:3 Changed 4 years ago by zzz

With respect to my good friend echelon, the piece size doesn't have much to do with retransmission rates. First, data is transferred in 16 KB "chunks" independent of the piece size, and second, there's an underlying streaming protocol to assure error-free transmission at the application layer. The retransmissions happen in streaming, where the packet size (currently 1730) is what matters.

I don't really have an issue with increasing the max piece size. The max files may be a problem. As I mentioned in a recent post on forum.i2p, the concern is running out of file handles, which can cause the router to crash. While I appreciate OP's test efforts, it's not really something that shows up in testing, as it's very OS- and load-dependent. Snark could do better at limiting the number of open files, but it's not straightforward, and there's a performance tradeoff in closing files too quickly only to reopen them. I'll think about possible solutions...

comment:4 Changed 4 years ago by zzz

dup of #1626

comment:5 Changed 4 years ago by zzz

  • Milestone changed from undecided to 0.9.23
  • Resolution set to fixed
  • Status changed from new to closed

Max Piece size increased to 16 MB
Max files increased to 999
Added code to close files faster

In 0647b1295687a9fc4acf0335c0bcc2c77dc2e2b8 0.9.22-16

Note: See TracTickets for help on using tickets.