Opened 9 years ago

Closed 7 years ago

#25 closed enhancement (fixed)

Move long scans to secondary threads

Reported by: gyoung Owned by: gyoung
Priority: major Milestone: Release_3.14
Component: fm/2 base Version:
Keywords: performance Cc:

Description

The freeze occurred when FM/2 tried to scan a network drive that is setup using the netdrive samba plugin. Normally if it has problems it returns an error 65. However, the system the drive was on had locked up and when fm/2 scanned it the WPS froze and I had to kill FM/2. The only way I could reopen FM/2 was to exclude the 2 drives. After I rebooted the other machine all works fine. The WPS drives object opened without problem it showed the 2 netdrive drives as empty but no freeze occurred. The same error message as below occurred again but I think the message was generated when I tried to kill FM/2 not when the freeze occurred.

Most of the scans that can take a long time are done on separate threads, so this kind of failure will not lock up fm/2. However, there are a few that are still done on the PM thread and this can cause apparent Desktop lockups. If this is the case, pressing Ctrl-Esc several times should cause PM to take control away from fm/2.

If we can identify what fm/2 was trying to do, we should be able to move this logic to a thread.

The trap should not occur, even with slow access. Unfortunately, the trap is not within fm/2 code, so it's hard to analyze without a process dump. I will be adding exceptq support to fm/2, once I get some in-progress exceptq rework completed. This will provide enhanced trap logging, without the need to resort to trap dumps.

The trap appears to somewhere in DOSREADASYNC or DOSWRITEASYNC, but that does not tell us much about where and how fm/2 managed to pass bad arguments to the API.

Change History (15)

comment:1 Changed 9 years ago by stevenhl

  • Summary changed from Move long scans to secondary treads to Move long scans to secondary threads

comment:2 Changed 9 years ago by gyoung

The netdrives problem is fixed if multiple drives are mounted as subdirectories (ie if laptopC and laptopD are mounted as U:\laptopC and U:\laptopD) the initial scan problem goes away as the drive and root subdirectories are returned by netdrives and dbl clicking on the directories quickly returns an error it the remote computer is shut down. What is different about this case?

comment:3 Changed 8 years ago by stevenhl

  • Owner StevenHL deleted

comment:4 Changed 8 years ago by stevenhl

  • Keywords performance added
  • Milestone set to Release_3.7

comment:5 Changed 8 years ago by gyoung

  • Version set to 3.07

comment:6 Changed 8 years ago by stevenhl

  • Milestone changed from Release_3.7 to Release_3.8
  • Version changed from 3.07 to 3.08

comment:7 Changed 8 years ago by stevenhl

  • Type changed from defect to enhancement

comment:8 Changed 8 years ago by gyoung

  • Milestone Release_3.8 deleted
  • Version 3.08 deleted

comment:9 Changed 8 years ago by gyoung

  • Milestone set to Release_3.9

comment:10 Changed 8 years ago by gyoung

  • Milestone changed from Release_3.9 to Release_3.10

comment:11 Changed 8 years ago by gyoung

  • Milestone changed from Release_3.10 to Release_3.11

Well one relatively simple solution that does not require massive rework would to put a loop around the drive tree fill loop. The result would be

for (pass = 0; pass < maxpass; pass++) {

for (drvndx = 0; drvndx < maxdrv; drvndx++) {

if (pass == 1 and drive flags day do in pass 1)

fill

else if

...

}

}

Pass 1 could be local drives that are not marked slow.

Pass 2 could be locall drives that are marked slow.

Pass 3 could be remove drives

etc.

This might not even require any tweaks to the container insert logic.

comment:12 Changed 7 years ago by stevenhl

  • Milestone changed from Release_3.12 to Release_3.13

comment:13 Changed 7 years ago by gyoung

  • Milestone changed from Release_3.13 to Release_3.14

comment:14 Changed 7 years ago by gyoung

  • Owner set to gyoung
  • Status changed from new to assigned

comment:15 Changed 7 years ago by gyoung

  • Resolution set to fixed
  • Status changed from assigned to closed

While I didn't add additional threads, it appears that by looping through multiple times with fastest drives first that scan times are significantly improved (here at least). I am marking this fixed.

Note: See TracTickets for help on using tickets.