files.json takes very long to write

Created by: LoveIsGrief

As mentioned in #35 (closed) files.json can grow to very larges sizes when sharing large folders (hundreds of GB or more) and subsequently takes a very long time to write.

After adding logs to see just how long it takes (https://github.com/LoveIsGrief/muwire/commit/9d4b365e63c74a6863d1c8d562c9e1487bda2c04) files.json had grown to 360MB and took ~15s to write

2020-01-19 21:33:47.309 SEVERE org.codehaus.groovy.vmplugin.v7.IndyInterface selectMethod ===Time(ms) to write tmp files.json: 34243
2020-01-19 21:33:47.697 SEVERE org.codehaus.groovy.vmplugin.v7.IndyInterface selectMethod ===Time(ms) to copy tmp files.json: 387
2020-01-19 21:34:25.999 SEVERE com.muwire.core.files.PersisterService$_persistFiles_closure8 doCall ===Time(ms) to write tmp files.json: 12953
2020-01-19 21:34:26.314 SEVERE com.muwire.core.files.PersisterService$_persistFiles_closure8 doCall ===Time(ms) to copy tmp files.json: 315
2020-01-19 21:35:28.794 SEVERE com.muwire.core.files.PersisterService$_persistFiles_closure8 doCall ===Time(ms) to write tmp files.json: 15747
2020-01-19 21:35:29.134 SEVERE com.muwire.core.files.PersisterService$_persistFiles_closure8 doCall ===Time(ms) to copy tmp files.json: 339
2020-01-19 21:36:28.874 SEVERE com.muwire.core.files.PersisterService$_persistFiles_closure8 doCall ===Time(ms) to write tmp files.json: 15827
2020-01-19 21:36:29.209 SEVERE com.muwire.core.files.PersisterService$_persistFiles_closure8 doCall ===Time(ms) to copy tmp files.json: 335
2020-01-19 21:37:28.009 SEVERE com.muwire.core.files.PersisterService$_persistFiles_closure8 doCall ===Time(ms) to write tmp files.json: 14784
2020-01-19 21:37:28.347 SEVERE com.muwire.core.files.PersisterService$_persistFiles_closure8 doCall ===Time(ms) to copy tmp files.json: 338
2020-01-19 21:38:28.482 SEVERE com.muwire.core.files.PersisterService$_persistFiles_closure8 doCall ===Time(ms) to write tmp files.json: 15258
2020-01-19 21:38:28.814 SEVERE com.muwire.core.files.PersisterService$_persistFiles_closure8 doCall ===Time(ms) to copy tmp files.json: 332
2020-01-19 21:39:27.974 SEVERE com.muwire.core.files.PersisterService$_persistFiles_closure8 doCall ===Time(ms) to write tmp files.json: 14749

It might be better to either:

• use a DB (sqlite is slow af, maybe there are other file DBs)
• use a folder structure e.g :
  • files/<filepath hash 0-5>/<filepath hash 6-31>.json
  • files/<filepath hash>.json
  • files/<share root>/<path to file>/<filename>.json

or find a solution that doesn't involve writing all existing hashlists every minute.

If you like, we can agree on a solution and I can work on it. I have about 2 weeks of holidays left.