Opened 8 years ago

Closed 6 years ago

#25 closed enhancement (fixed)

Move long scans to secondary threads

Reported by: gyoung Owned by: gyoung
Priority: major Milestone: Release_3.14
Component: fm/2 base Version:
Keywords: performance Cc:

Description

The freeze occurred
when FM/2 tried to scan a network drive that is setup using the netdrive
samba plugin. Normally if it has problems it returns an error 65. However,
the system the drive was on had locked up and when fm/2 scanned it the
WPS froze and I had to kill FM/2. The only way I could reopen FM/2 was
to exclude the 2 drives. After I rebooted the other machine all works fine.
The WPS drives object opened without problem it showed the 2 netdrive
drives as empty but no freeze occurred. The same error message as below
occurred again but I think the message was generated when I tried to kill
FM/2 not when the freeze occurred.

Most of the scans that can take a long time are done on separate threads,
so this kind of failure will not lock up fm/2. However, there are a few
that are still done on the PM thread and this can cause apparent Desktop
lockups. If this is the case, pressing Ctrl-Esc several times should
cause PM to take control away from fm/2.

If we can identify what fm/2 was trying to do, we should be able to move
this logic to a thread.

The trap should not occur, even with slow access. Unfortunately, the trap
is not within fm/2 code, so it's hard to analyze without a process dump.
I will be adding exceptq support to fm/2, once I get some in-progress
exceptq rework completed. This will provide enhanced trap logging,
without the need to resort to trap dumps.

The trap appears to somewhere in DOSREADASYNC or DOSWRITEASYNC, but that
does not tell us much about where and how fm/2 managed to pass bad
arguments to the API.

Change History (15)

comment:1 Changed 8 years ago by stevenhl

  • Summary changed from Move long scans to secondary treads to Move long scans to secondary threads

comment:2 Changed 8 years ago by gyoung

The netdrives problem is fixed if multiple drives are mounted as subdirectories (ie if laptopC and laptopD are mounted as U:\laptopC and U:\laptopD) the initial scan problem goes away as the drive and root subdirectories are returned by netdrives and dbl clicking on the directories quickly returns an error it the remote computer is shut down. What is different about this case?

comment:3 Changed 8 years ago by stevenhl

  • Owner StevenHL deleted

comment:4 Changed 8 years ago by stevenhl

  • Keywords performance added
  • Milestone set to Release_3.7

comment:5 Changed 8 years ago by gyoung

  • Version set to 3.07

comment:6 Changed 7 years ago by stevenhl

  • Milestone changed from Release_3.7 to Release_3.8
  • Version changed from 3.07 to 3.08

comment:7 Changed 7 years ago by stevenhl

  • Type changed from defect to enhancement

comment:8 Changed 7 years ago by gyoung

  • Milestone Release_3.8 deleted
  • Version 3.08 deleted

comment:9 Changed 7 years ago by gyoung

  • Milestone set to Release_3.9

comment:10 Changed 7 years ago by gyoung

  • Milestone changed from Release_3.9 to Release_3.10

comment:11 Changed 7 years ago by gyoung

  • Milestone changed from Release_3.10 to Release_3.11

Well one relatively simple solution that does not require massive rework
would to put a loop around the drive tree fill loop. The result would be

for (pass = 0; pass < maxpass; pass++) {

for (drvndx = 0; drvndx < maxdrv; drvndx++) {

if (pass == 1 and drive flags day do in pass 1)

fill

else if

...

}

}

Pass 1 could be local drives that are not marked slow.

Pass 2 could be locall drives that are marked slow.

Pass 3 could be remove drives

etc.

This might not even require any tweaks to the container insert logic.

comment:12 Changed 6 years ago by stevenhl

  • Milestone changed from Release_3.12 to Release_3.13

comment:13 Changed 6 years ago by gyoung

  • Milestone changed from Release_3.13 to Release_3.14

comment:14 Changed 6 years ago by gyoung

  • Owner set to gyoung
  • Status changed from new to assigned

comment:15 Changed 6 years ago by gyoung

  • Resolution set to fixed
  • Status changed from assigned to closed

While I didn't add additional threads, it appears that by looping through multiple times with fastest drives first that scan times are significantly improved (here at least). I am marking this fixed.

Note: See TracTickets for help on using tickets.