Opened 3 years ago
Last modified 3 years ago
#2281 new enhancement
Use memory-mapped files in Snark
Reported by: | Zlatin Balevsky | Owned by: | zzz |
---|---|---|---|
Priority: | minor | Milestone: | undecided |
Component: | apps/i2psnark | Version: | 0.9.35 |
Keywords: | nio | Cc: | |
Parent Tickets: | Sensitive: | no |
Description
It makes sense to use memory-mapped files instead of RandomAccessFile because of performance.
Subtickets
Change History (5)
comment:1 Changed 3 years ago by
Component: | apps/other → apps/i2psnark |
---|---|
Owner: | set to zzz |
comment:2 Changed 3 years ago by
comment:3 Changed 3 years ago by
https://docs.oracle.com/javase/7/docs/api/java/nio/MappedByteBuffer.html
Here is a brief but good description why memory-mapped files outperform other types of disk i/o: https://stackoverflow.com/questions/192527/what-are-the-advantages-of-memory-mapped-files
Also https://www.codeproject.com/Tips/683614/10-Things-to-Know-about-Memory-Mapped-File-in-Java
comment:4 Changed 3 years ago by
For most operating systems, mapping a file into memory is more expensive than reading or writing a few tens of kilobytes of data via the usual read and write methods. From the standpoint of performance it is generally only worth mapping relatively large files into memory.
In snark, generally we don't do repeated reads/writes of a particular portion of a file. We read or write it and move on to something else. Snark can randomly read or write to one of thousands of files totalling gigabytes. And we have to make it work on everything from a low-memory RPi to a big server. I don't know how to manage that and implicit in the MappedByteBuffer? implementation is you're doing repeated reads/writes to the same region.
So mapping a portion of a file into memory seems like a bad fit for snark.
comment:5 Changed 3 years ago by
Two things:
- There are repeated reads from the same file region when uploading the same piece multiple times.
- Mapping a file to memory does not mean that the contents will actually get loaded into memory (although that can be forced, which is a bad idea). It just means that the OS may choose to load and unload various regions of the file depending on access patterns and available memory.
So the only situation where mapping is clearly bad idea is 32-bit platforms where the address space is only 4GB. On a 64-bit raspberry pi, even if it has only half a gig of real memory it is safe to map up to 264 bytes of files. It just means the OS will have to un-map more frequently.
you mean a nio FileChannel?? or what? link to article on performance?