Deadlock of QMutex when quitting VLC

[KO Myung-Hun]

Hi/2.

When using debug build of Qt 4.6.2(r744), the following messages are displayed on quitting VLC.

QMutex::lock: Deadlock detected in thread 9
QMutexPrivate::QMutexPrivate: DosCloseEventSem returned 301

BTW, that thread was not created by VLC. That is, it's in Qt4 internal.

And I suspect that this is the cause of blocking VLC when trying to play a second movie using embedded video mode which is being implemented.

[Dmitry A. Kuminov]

I can't say anything because it's not clear what is going on with the application when it happens. Does it hang? You should somehow track the execution tho the point this assertion is printed -- we need the call stack of this thread at least. Error 301 means the semaphore is closed while being waited on in another thread which usually means a crash.

[KO Myung-Hun]

It does not hang. But when using embedded video, VLC is blocked on trying to play second time, that is, play, stop and play again.

BTW that deadlock assertion is displayed while executing exec() instance. VLC does not have exec() instance at all, but just use the inherited exec().

Nevertheless should I step into exec() ?

And DosCloseEventSem message is displayed when quitting VLC.

Unfortunately, ipmd crashes when encountering the breakpoint where displaying deadlock message.

[Dmitry A. Kuminov]

As we already saw earlier, VLC does some strange things with QApplication::quit() which I don't fully understand. Error 301 may also mean that a thread gets killed upon termination instead of being properly shutdown.

Exec() runs the message queue, so it may be quite difficult to get to the breakpoint by stepping into it (there will be tons of unrelated messages).

Do you know how to use PROCDUMP? Using it + the breakpoints right after the mentioned assertions, you will for sure get a dump from which we will be able to extract the call stack. More details are in comment 22 of #88. Feel free to ask if something's not clear.

[KO Myung-Hun]

Although the method of VLC to quit is strange, it works on Win32 at least. So we have a problem with it, it is a fault of Qt4 implementation for OS/2.

BTW I know, PROCDUMP works only when a program is crahsed, right ? Unfortunately, VLC does not crash, but IPMD does.

In this case as well, is it possible to use PROCDUMP ?

[Dmitry A. Kuminov]

We shouldn't say which fault it is until we find the exact problem -- it may be a violation of the Qt4 logic that coincidently works on Windows, for example.

Yes, for PROCDUMP you need to trigger the exception handler. You can do this by adding a manual breakpoint after the assertion like this:

__asm__ __volatile__("int $3\n\t");

And yes, PROCDUMP will catch any crashing process on the system though you will get a lot of useless info from IPMD in your case so better do int 3 as described above.

[KO Myung-Hun]

X11 also uses the same code.

Anyway, however, I also hope that it will be.

And I'll try a manual breakpoint later.

[KO Myung-Hun]

The crash of IPMD was due to the wrong setting of debugger.

I enabled 'Check heap when stopping' accidently. Since then, whenever using breakpoints or stepping, IPMD has been crashed.

Now I can watch the call stack on a deaklock point, and have found the cause.

it's QMutexLocker lock(&qt_adopted_thread_watcher_mutex) in qt_adopted_thread_watcher_function() of qthread_os2.cpp.

BTW, qt_adopted_thread_watcher_mutex is already locked at the beginning of the loop. So deadlock is caused by QMutexLocker.

After removing QMutexLocker as the following, VLC works as expected.

Index: src/corelib/thread/qthread_os2.cpp
===================================================================
--- src/corelib/thread/qthread_os2.cpp  (revision 744)
+++ src/corelib/thread/qthread_os2.cpp  (working copy)
@@ -193,7 +193,7 @@
             if (rc != ERROR_THREAD_NOT_TERMINATED) {
                 if (rc == NO_ERROR || rc == ERROR_INVALID_THREADID) {
                     QThreadData::get2(qt_adopted_qthreads.at(i))->deref();
-                    QMutexLocker lock(&qt_adopted_thread_watcher_mutex);
+                    //QMutexLocker lock(&qt_adopted_thread_watcher_mutex);
                     qt_adopted_thread_tids.remove(i);
                     qt_adopted_qthreads.remove(i);
                     continue;

This is a proper approach ?

[Dmitry A. Kuminov]

Yes, this is a copy-paste bug that affected adopted threads (i.e. foreign threads not started with QThread::start()). Thank you for spotting it! Fixed in r747. Do I understand it right that this fixes the problem completely for you? No more crashes or hangs at VLC termination?

[KO Myung-Hun]

No more hangs.

But, I encountered SIGSEGV one time in the same thread at termination.

Unfortunately, I cannot reproduce this SIGSEGV.

Anyway we seem to solve deadlock.

[Dmitry A. Kuminov]

Yes. Feel free to open a new ticket if you get more info on this SIGSEGV.