Opened 11 months ago

Last modified 11 months ago

#2281 new enhancement

Use memory-mapped files in Snark

Reported by: zab Owned by: zzz
Priority: minor Milestone: undecided
Component: apps/i2psnark Version: 0.9.35
Keywords: nio Cc:
Parent Tickets:

Description

It makes sense to use memory-mapped files instead of RandomAccessFile because of performance.

Subtickets (add)

Change History (5)

comment:1 Changed 11 months ago by zzz

  • Component changed from apps/other to apps/i2psnark
  • Owner set to zzz

comment:2 Changed 11 months ago by zzz

you mean a nio FileChannel?? or what? link to article on performance?

comment:4 Changed 11 months ago by zzz

quote https://docs.oracle.com/javase/7/docs/api/java/nio/channels/FileChannel.html#map(java.nio.channels.FileChannel.MapMode,%20long,%20long) -

For most operating systems, mapping a file into memory is more expensive than reading or writing a few tens of kilobytes of data via the usual read and write methods. From the standpoint of performance it is generally only worth mapping relatively large files into memory.

In snark, generally we don't do repeated reads/writes of a particular portion of a file. We read or write it and move on to something else. Snark can randomly read or write to one of thousands of files totalling gigabytes. And we have to make it work on everything from a low-memory RPi to a big server. I don't know how to manage that and implicit in the MappedByteBuffer? implementation is you're doing repeated reads/writes to the same region.

So mapping a portion of a file into memory seems like a bad fit for snark.

comment:5 Changed 11 months ago by zab

Two things:

  1. There are repeated reads from the same file region when uploading the same piece multiple times.
  1. Mapping a file to memory does not mean that the contents will actually get loaded into memory (although that can be forced, which is a bad idea). It just means that the OS may choose to load and unload various regions of the file depending on access patterns and available memory.

So the only situation where mapping is clearly bad idea is 32-bit platforms where the address space is only 4GB. On a 64-bit raspberry pi, even if it has only half a gig of real memory it is safe to map up to 264 bytes of files. It just means the OS will have to un-map more frequently.

Note: See TracTickets for help on using tickets.