Trap 000d in FLAT code seg. when starting VM on Asus Core2Duo

[Valery V. Sedletski]

On my Asus X55s laptop (Core2 Duo) I see the following trap:

when starting the VM. I see it only with IBM's 14.104 / 14.106 kernels. With OS/4 r4515 kernel, it doesn't trap at all.

Also, comparing the cs:eip with addresses in this list:

vboxdrv: e3ad4020 VMMR0.r0
vboxdrv: e390f020 VBoxDDR0.r0
vboxdrv: e38fd020 VBoxDD2R0.r0
HMR0InitVM: e3aa2000

We see, that the trap address is probably, in VMMR0.r0

[Valery V. Sedletski]

trap 000d in flat code when starting the VM

[Valery V. Sedletski]

Oops, now I see that adresses of *.r0 modules I specified above is for OS/4 kernel (which doesn't trap), so adresses are different. And I cannot see these adresses for 14.104 / 14.106 because they are written to a vboxdrv.sys internal buffer after these modules are loaded. But the trap occurs in the moment close to the moment of loading the modules, so, I cannot catch and see the adresses :(

[Valery V. Sedletski]

I reproduced this trap on my testing machine with a COM port and kernel debugger:

Trap 13 (0DH) - General Protection Fault 0000
eax=00000000 ebx=af24c000 ecx=00000000 edx=00000000 esi=00000202 edi=ffe97000
eip=e5b1bbbe esp=f8cafba8 ebp=f8cafc08 iopl=0 rf -- -- nv up di pl zr na pe nc
cs=0178 ss=1550 ds=0170 es=0170 fs=0000 gs=0000 cr2=005dc000 cr3=00225000 p=00
0178:e5b1bbbe f30fc734       db        f3, 0f, c7, 34
0178:e5b1bbbe f30fc734       db        f3, 0f, c7, 34
0178:e5b1bbc2 2477           and       al,77            ;'w'
0178:e5b1bbc4 0e             push    cs
0178:e5b1bbc5 7407           jz        e5b1bbce
0178:e5b1bbc7 b85cf0ffff     mov       eax,fffff05c
0178:e5b1bbcc eb05           jmp       e5b1bbd3
0178:e5b1bbce b860f0ffff     mov       eax,fffff060
0178:e5b1bbd3 83c408         add       esp,+08
0178:e5b1bbd6 85c0           test      eax,eax
0178:e5b1bbd8 7877           js        e5b1bc51
0178:e5b1bbda c605f855d3e501 mov       byte ptr [e5d355f8],01
0178:e5b1bbe1 0f01c4         sgdt
0178:e5b8029f e5b7ffc0 fd3a8754 00000046 00002955 _StartInitCode + e5b8029f
0178:f214b2bb fd3a8754 e5b80130 e5b846f4 00005a5c _StartInitCode + f214b2bb
0178:f2138b84 00000004 f9e76fd0 fd396634 003a58a0 _StartInitCode + f2138b84
0178:f2138041 00000186 000000c0 00000004 003a58a0 _StartInitCode + f2138041
0178:10000a72 00005b60 00000000 5b600000 04d00578 _StartInitCode + 10000a72
0178:00000000 gradd:SEG2:_StartInitCode + e5b1bbe4
##

[Valery V. Sedletski]

This is a disassembly of the above piece in VMXR0.r0 in hiew:

.00027B85: C705E015240000000000         mov       d,[0002415E0],000000000 ;"
.00027B8F: F6C420                       test      ah,020 ;" "
.00027B92: 7506                         jne      .000027B9A   -------- (1)
.00027B94: 80CC20                       or        ah,020 ;" "
.00027B97: 0F22E0                       mov       cr4,eax
.00027B9A: 31C0                         xor       eax,eax
.00027B9C: 52                           push      edx
.00027B9D: 53                           push      ebx
.00027B9E: F3                           repe
.00027B9F: 0FC73424                     ???       [esp]
.00027BA3: 770E                         ja       .000027BB3   -------- (2)
.00027BA5: 7407                         je       .000027BAE   -------- (3)

where F3 0F C7 34 24 is vmxon qword ptr [esi] opcode -- something related with Intel VMX (VT-x). Why it tries to "enter VMX operation", I don't know yet.

​http://www.tptp.cc/mirrors/siyobik.info/instruction/VMXON.html

The vmxon instruction is in ​http://trac.netlabs.org/vbox/browser/trunk/include/VBox/vmm/hm_vmx.h#L1856, VMXEnable inline routine. The place in the code where it's called is hmR0InitIntel routine in ​http://trac.netlabs.org/vbox/browser/trunk/src/VBox/VMM/VMMR0/HMR0.cpp#L450.

It seems, it tries to enable VT-x if that CPU capability is detected, regardless of whether it is enabled in VBox settings. But if it is not enabled, it isn't used for emulation (but still enabled).

And on 14.104_SMP, there's something wrong with vmxon parameters, and ok with OS/4 kernel.