Opened 3 years ago

Last modified 3 years ago

#2281 new enhancement

Use memory-mapped files in Snark

Reported by: Zlatin Balevsky Owned by: zzz
Priority: minor Milestone: undecided
Component: apps/i2psnark Version: 0.9.35
Keywords: nio Cc:
Parent Tickets: Sensitive: no


It makes sense to use memory-mapped files instead of RandomAccessFile because of performance.


Change History (5)

comment:1 Changed 3 years ago by zzz

Component: apps/otherapps/i2psnark
Owner: set to zzz

comment:2 Changed 3 years ago by zzz

you mean a nio FileChannel?? or what? link to article on performance?

comment:4 Changed 3 years ago by zzz

quote,%20long,%20long) -

For most operating systems, mapping a file into memory is more expensive than reading or writing a few tens of kilobytes of data via the usual read and write methods. From the standpoint of performance it is generally only worth mapping relatively large files into memory.

In snark, generally we don't do repeated reads/writes of a particular portion of a file. We read or write it and move on to something else. Snark can randomly read or write to one of thousands of files totalling gigabytes. And we have to make it work on everything from a low-memory RPi to a big server. I don't know how to manage that and implicit in the MappedByteBuffer? implementation is you're doing repeated reads/writes to the same region.

So mapping a portion of a file into memory seems like a bad fit for snark.

comment:5 Changed 3 years ago by Zlatin Balevsky

Two things:

  1. There are repeated reads from the same file region when uploading the same piece multiple times.
  1. Mapping a file to memory does not mean that the contents will actually get loaded into memory (although that can be forced, which is a bad idea). It just means that the OS may choose to load and unload various regions of the file depending on access patterns and available memory.

So the only situation where mapping is clearly bad idea is 32-bit platforms where the address space is only 4GB. On a 64-bit raspberry pi, even if it has only half a gig of real memory it is safe to map up to 264 bytes of files. It just means the OS will have to un-map more frequently.

Note: See TracTickets for help on using tickets.