Opened 11 years ago

Closed 11 years ago

#140 closed enhancement (duplicate)

Increase findbuf to speed file loading

Reported by: gyoung Owned by: gyoung
Priority: major Milestone:
Component: fm/2 base Version: 3.07
Keywords: Cc:

Description

The findbuf had been limited to 128 files in many place. Increasing it appears to be a trade off between loading speed and memory use/temporary poor WPS response. Memory use should be trivial with modern computers and appropriate use of threading should protect responsiveness as well or better than limiting the findbuf. Load times are increased from 7 min 20 sec to 36 sec by changing findbuf from 128 to 40960 for loading 33,000 files. I see know reason to limit the findbuf (other than expedience of programming) The problem is making end users understand how to optimize it for their usage patterns. I think a default around 10240 would probably be best the time performance per load is fast enough so that any WPS slowing should be trivial while it will load most directories in a single pass maximizing performance. The 36 sec load above was less than one third drive scanning so we need to work on optimizing the container loading code.

Change History (1)

comment:1 Changed 11 years ago by gyoung

  • Resolution set to duplicate
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.