fmutex deadlock: Recursive mutex!

[voyagerbuddy]

eCS 2.2 betaII. I start Polarbar (build 2324) and click on an email to look at and get the following error.

---

[E:\programs\polarbar]polarbar

[E:\programs\polarbar]java.exe -dsa -cp pbmdaily.zip;activation.jar;cryptix-pgp.
jar;cryptix.jar;log4j-1.2.13.jar;mail.jar;sscedistro.jar; org.polarbar.mailer -d

h:\polarbar\mailer\

#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x1df0a4be, pid=77, tid=5046274

#
# JRE version: 6.0_27
# Java VM: OpenJDK Client VM (20.0-b12 mixed mode os2-x86 )
# Problematic frame:
# C [LIBC065+0xa4be]
LIBC PANIC!!
fmutex deadlock: Recursive mutex!
0x02e3013c: Owner=0x004d0002 Self=0x004d0002 fs=0x3 flags=0x0 hev=0x000100b7

Desc="LIBC Heap"

pid=0x004d ppid=0x004b tid=0x0002 slot=0x0088 pri=0x0200 mc=0x0000
E:\PROGRAMS\OPENJDK\BIN\JAVA.EXE
Process has been dumped

[E:\programs\polarbar]

---

[voyagerbuddy]

The attachment was too large.

I do have process dump if needed.

[Andreas Buchinger]

I also encountered fmutex deadlock:

Last lines of crashing app -

LIBC PANIC!!
fmutex deadlock: Recursive mutex!
0x17e20990: Owner=0x36c20001 Self=0x36c20001 fs=0x3 flags=0x0 hev=0x00010015
            Desc="<null>"
pid=0x36c2 ppid=0x3682 tid=0x0001 slot=0x0147 pri=0x0200 mc=0x0000
P:\JAVA6\BIN\JAVA.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.

LIBC PANIC!!
fmutex deadlock: Recursive mutex!
0x17e20990: Owner=0x36c20001 Self=0x36c20001 fs=0x3 flags=0x0 hev=0x00010015
            Desc="<null>"
pid=0x36c2 ppid=0x3682 tid=0x0001 slot=0x0147 pri=0x0200 mc=0x0000
P:\JAVA6\BIN\JAVA.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.

LIBC PANIC!!
fmutex deadlock: Recursive mutex!
0x17e20990: Owner=0x36c20001 Self=0x36c20001 fs=0x3 flags=0x0 hev=0x00010015
            Desc="<null>"
pid=0x36c2 ppid=0x3682 tid=0x0001 slot=0x0147 pri=0x0200 mc=0x0000
P:\JAVA6\BIN\JAVA.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.

LIBC PANIC!!
fmutex deadlock: Recursive mutex!
0x17e20990: Owner=0x36c20001 Self=0x36c20001 fs=0x3 flags=0x0 hev=0x00010015
            Desc="<null>"
pid=0x36c2 ppid=0x3682 tid=0x0001 slot=0x0147 pri=0x0200 mc=0x0000
P:\JAVA6\BIN\JAVA.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.

LIBC PANIC!!
fmutex deadlock: Recursive mutex!
0x17e20990: Owner=0x36c20001 Self=0x36c20001 fs=0x3 flags=0x0 hev=0x00010015
            Desc="<null>"
pid=0x36c2 ppid=0x3682 tid=0x0001 slot=0x0147 pri=0x0200 mc=0x0000
P:\JAVA6\BIN\JAVA.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.

LIBC PANIC!!
fmutex deadlock: Recursive mutex!
0x17e20990: Owner=0x36c20001 Self=0x36c20001 fs=0x3 flags=0x0 hev=0x00010015
            Desc="<null>"
pid=0x36c2 ppid=0x3682 tid=0x0001 slot=0x0147 pri=0x0200 mc=0x0000
P:\JAVA6\BIN\JAVA.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.

Killed by SIGABRT
pid=0x36c2 ppid=0x3682 tid=0x0001 slot=0x0147 pri=0x0200 mc=0x0001
P:\JAVA6\BIN\JAVA.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.

LIBC PANIC!!
fmutex deadlock: Recursive mutex!
0x17e20990: Owner=0x36c20001 Self=0x36c20001 fs=0x3 flags=0x0 hev=0x00010015
            Desc="<null>"
pid=0x36c2 ppid=0x3682 tid=0x0001 slot=0x0147 pri=0x0200 mc=0x0001
P:\JAVA6\BIN\JAVA.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.

LIBC PANIC!!
fmutex deadlock: Recursive mutex!
0x17e20990: Owner=0x36c20001 Self=0x36c20001 fs=0x3 flags=0x0 hev=0x00010015
            Desc="<null>"
pid=0x36c2 ppid=0x3682 tid=0x0001 slot=0x0147 pri=0x0200 mc=0x0001
P:\JAVA6\BIN\JAVA.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.

LIBC PANIC!!
_um_free_maybe_lock: bad parent magic 0xdd68eeed, parent=009e0150 block=00e9e360 lock=0x1
pid=0x36c2 ppid=0x3682 tid=0x0001 slot=0x0147 pri=0x0200 mc=0x0001
P:\JAVA6\BIN\JAVA.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.

This happened as regain tried to index a large disk. This time I started regain.jar with -Xms128M -Xmx1000M. I've used different settings for -Xmx up to -Xmx1200M. Regain constantly runs out of memory but I didn't see fmutex deadlock before.

Regain log file -

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 4092 bytes for char in e:/Coding/javaos2/openjdk/hotspot/src/share/vm/utilities/stack.inline.hpp
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (e:/Coding/javaos2/openjdk/hotspot/src/share/vm/memory/allocation.inline.hpp:44), pid=14018, tid=918683651
#
# JRE version: 6.0_27
# Java VM: OpenJDK Client VM (20.0-b12 mixed mode os2-x86 )

---------------  T H R E A D  ---------------

Current thread (0x00f59800):  VMThread [stack: 0x02f70000,0x03080000] [id=918683651]

Stack: [0x02f70000,0x03080000],  sp=0x0306f754,  free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

Usually the crash logs are longer like this one -

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 4092 bytes for char in e:/Coding/javaos2/openjdk/hotspot/src/share/vm/utilities/stack.inline.hpp
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (e:/Coding/javaos2/openjdk/hotspot/src/share/vm/memory/allocation.inline.hpp:44), pid=11830, tid=775290884
#
# JRE version: 6.0_27
# Java VM: OpenJDK Client VM (20.0-b12 mixed mode os2-x86 )

---------------  T H R E A D  ---------------

Current thread (0x00f59800):  VMThread [stack: 0x02f90000,0x030a0000] [id=775290884]

Stack: [0x02f90000,0x030a0000],  sp=0x0308f514,  free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [JVM+0x29ed63]
V  [JVM+0x294350]
V  [JVM+0x2ae973]
V  [JVM+0x141f18]
V  [JVM+0x131396]
V  [JVM+0x2ade7f]
V  [JVM+0xfb19e]
V  [JVM+0xf3e55]
V  [JVM+0xe4d25]
V  [JVM+0xeaea7]
V  [JVM+0xeb524]
V  [JVM+0xe90ee]
V  [JVM+0xe7084]
V  [JVM+0xe7763]
V  [JVM+0x2b3307]
V  [JVM+0x26f84f]
V  [JVM+0x26ddb7]
V  [JVM+0x26e409]
V  [JVM+0x26e7af]
V  [JVM+0x2bbd14]
C  [KERNEL32+0x10001c0]
C  [KERNEL32+0xc540]
C  [WGSS50+0xcf73]

VM_Operation (0x05f3f90c): GenCollectFull, mode: safepoint, requested by thread 0x00f63800


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x00e42000 JavaThread "Image Animator 0" daemon [_thread_blocked, id=775290926, stack(0x062c0000,0x063d0000)]
  0x00f05000 JavaThread "Thread-27" [_thread_blocked, id=775290925, stack(0x06190000,0x062a0000)]
  0x00f03800 JavaThread "Thread-28" [_thread_blocked, id=775290882, stack(0x06080000,0x06190000)]
  0x00f08400 JavaThread "Thread-26" [_thread_blocked, id=775290896, stack(0x03e10000,0x03f20000)]
  0x00d58c00 JavaThread "DestroyJavaVM" [_thread_blocked, id=775290881, stack(0x00040000,0x00840000)]
  0x00f63800 JavaThread "Thread-25" [_thread_blocked, id=775290924, stack(0x05e40000,0x05f50000)]
  0x00f62400 JavaThread "Thread-24" [_thread_in_native, id=775290923, stack(0x05d00000,0x05e10000)]
  0x00f60c00 JavaThread "Thread-23" [_thread_blocked, id=775290922, stack(0x05be0000,0x05cf0000)]
  0x00e2e800 JavaThread "Thread-22" [_thread_blocked, id=775290921, stack(0x05ac0000,0x05bd0000)]
  0x00e2d000 JavaThread "Thread-21" [_thread_blocked, id=775290920, stack(0x059a0000,0x05ab0000)]
  0x00e2bc00 JavaThread "Thread-20" [_thread_blocked, id=775290919, stack(0x05880000,0x05990000)]
  0x00e2a400 JavaThread "Thread-19" [_thread_blocked, id=775290918, stack(0x05760000,0x05870000)]
  0x00e29000 JavaThread "Thread-18" [_thread_blocked, id=775290917, stack(0x05640000,0x05750000)]
  0x00e27800 JavaThread "Thread-17" [_thread_blocked, id=775290916, stack(0x05520000,0x05630000)]
  0x00e26400 JavaThread "Thread-16" [_thread_blocked, id=775290915, stack(0x05400000,0x05510000)]
  0x00e24c00 JavaThread "Thread-15" [_thread_blocked, id=775290914, stack(0x052e0000,0x053f0000)]
  0x00e23800 JavaThread "Thread-14" [_thread_blocked, id=775290913, stack(0x051c0000,0x052d0000)]
  0x00e22000 JavaThread "Thread-13" [_thread_blocked, id=775290912, stack(0x050a0000,0x051b0000)]
  0x00e20c00 JavaThread "Thread-12" [_thread_blocked, id=775290911, stack(0x04f80000,0x05090000)]
  0x00e1f400 JavaThread "Thread-11" [_thread_blocked, id=775290910, stack(0x04e60000,0x04f70000)]
  0x00e1e000 JavaThread "Thread-10" [_thread_blocked, id=775290909, stack(0x04d40000,0x04e50000)]
  0x00e1c800 JavaThread "Thread-9" [_thread_blocked, id=775290908, stack(0x04c20000,0x04d30000)]
  0x00e1b400 JavaThread "Thread-8" [_thread_blocked, id=775290907, stack(0x04b00000,0x04c10000)]
  0x00e1a000 JavaThread "Thread-7" [_thread_blocked, id=775290906, stack(0x049e0000,0x04af0000)]
  0x00e18800 JavaThread "Thread-6" [_thread_blocked, id=775290905, stack(0x048c0000,0x049d0000)]
  0x00e17c00 JavaThread "Thread-5" [_thread_blocked, id=775290904, stack(0x04790000,0x048a0000)]
  0x00e16c00 JavaThread "Thread-4" [_thread_blocked, id=775290903, stack(0x04670000,0x04780000)]
  0x00e05c00 JavaThread "Thread-3" [_thread_blocked, id=775290902, stack(0x04540000,0x04650000)]
  0x00e03000 JavaThread "GC Daemon" daemon [_thread_blocked, id=775290901, stack(0x04410000,0x04520000)]
  0x00dffc00 JavaThread "RMI Reaper" [_thread_blocked, id=775290900, stack(0x042e0000,0x043f0000)]
  0x00dfec00 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=775290899, stack(0x041b0000,0x042c0000)]
  0x00dd7000 JavaThread "Thread-2" [_thread_blocked, id=775290898, stack(0x04080000,0x04190000)]
  0x00dcd800 JavaThread "AWT-EventQueue-0" [_thread_blocked, id=775290895, stack(0x03ce0000,0x03df0000)]
  0x00ebb800 JavaThread "AWT-Windows" daemon [_thread_in_native, id=775290894, stack(0x03ba0000,0x03cb0000)]
  0x00eba000 JavaThread "AWT-Shutdown" [_thread_blocked, id=775290893, stack(0x03a70000,0x03b80000)]
  0x00eb9400 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=775290892, stack(0x03940000,0x03a50000)]
  0x00da5800 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=775290890, stack(0x036b0000,0x037c0000)]
  0x00d96800 JavaThread "C1 CompilerThread0" daemon [_thread_blocked, id=775290889, stack(0x03560000,0x03670000)]
  0x00d95000 JavaThread "Attach Listener" daemon [_thread_blocked, id=775290888, stack(0x03440000,0x03550000)]
  0x00f5fc00 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=775290887, stack(0x03320000,0x03430000)]
  0x00f5dc00 JavaThread "Finalizer" daemon [_thread_blocked, id=775290886, stack(0x031f0000,0x03300000)]
  0x00f5c400 JavaThread "Reference Handler" daemon [_thread_blocked, id=775290885, stack(0x030c0000,0x031d0000)]

Other Threads:
=>0x00f59800 VMThread [stack: 0x02f90000,0x030a0000] [id=775290884]
  0x00da7800 WatcherThread [stack: 0x037d0000,0x038e0000] [id=775290891]

VM state:at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x00d57e40] Threads_lock - owner thread: 0x00f59800
[0x00d58340] Heap_lock - owner thread: 0x00f63800

Heap
 def new generation   total 124736K, used 87560K [0x231d0000, 0x2b920000, 0x3c1d0000)
  eden space 110912K,  71% used [0x231d0000, 0x27efc9f8, 0x29e20000)
  from space 13824K,  61% used [0x29e20000, 0x2a6759f8, 0x2aba0000)
  to   space 13824K,   0% used [0x2aba0000, 0x2aba0000, 0x2b920000)
 tenured generation   total 276868K, used 82565K [0x3c1d0000, 0x4d031000, 0x6e1d0000)
   the space 276868K,  29% used [0x3c1d0000, 0x412714f8, 0x41271600, 0x4d031000)
 compacting perm gen  total 20224K, used 20187K [0x6e1d0000, 0x6f590000, 0x721d0000)
   the space 20224K,  99% used [0x6e1d0000, 0x6f586f28, 0x6f587000, 0x6f590000)
No shared spaces configured.

Code Cache  [0x21130000, 0x21630000, 0x23130000)
 total_blobs=2529 nmethods=2326 adapters=137 free_code_cache=28327872 largest_free_block=320


VM Arguments:
jvm_args: -Xms128M -Xmx1200M 
java_command: regain.jar
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=P:\Java6
PATH=P:\Java6\bin;p:\usr\sbin;p:\usr\bin;M:\MPTN\BIN;M:\IBMCOM;M:\ECS\BIN;M:\TCPIP\BIN;M:\IBMLAN\NETPROG;M:\MUGLIB;M:\EMX\BIN;M:\IBMGSK50\BIN;M:\IBMGSK40\BIN;M:\IBMGSK\BIN;M:\OS2;M:\OS2\SYSTEM;M:\OS2\MDOS\WINOS2;M:\OS2\INSTALL;M:\;M:\OS2\MDOS;M:\OS2\APPS;M:\ECS\KLIBC\BIN;M:\MMOS2\FREEDB;M:\MMOS2;M:\MMOS2\MMPLAYER;m:\batch;P:\4OS2;p:\util;P:\CDWFS;P:\OS2TK45\SOM\COMMON;P:\OS2TK45\SOM\BIN;P:\OS2TK45\BIN;P:\IBMCPP40\BIN;P:\IBMCPP40\SAMPLES\REAL\FULL;P:\URE\BIN;P:\UTIL\NDFS;P:\OS2DDK\DDK\BASE\TOOLS;P:\VYPERHELP;P:\util\NDFS;p:\odin\system32;P:\perl5\bin;p:\NETSCAPE\PROGRAM;P:\netqos2;P:\Java6\bin;
USERNAME=ANDI
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 6 Model f Stepping b, GenuineIntel



---------------  S Y S T E M  ---------------

OS: Windows XP Build 2600 Service Pack 3

CPU:total 2 (2 cores per cpu, 1 threads per core) family 6 model 15 stepping 11, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3

Memory: 4k page, physical 3407144k(3186924k free), swap 2097151k(2097151k free)

vm_info: OpenJDK Client VM (20.0-b12) for os2-x86 JRE (1.6.0_27-b01-GA5), built on Feb 17 2013 01:25:04 by "dmik" with gcc 4.4.6

time: Sun Sep 21 19:57:44 2014
elapsed time: 408 seconds