Trap 000d in FLAT code seg. when starting VM on Asus Core2Duo

[Valery V. Sedletski]

On my Asus X55s laptop (Core2 Duo) I see the following trap:

when starting the VM. I see it only with IBM's 14.104 / 14.106 kernels. With OS/4 r4515 kernel, it doesn't trap at all.

Also, comparing the cs:eip with addresses in this list:

vboxdrv: e3ad4020 VMMR0.r0
vboxdrv: e390f020 VBoxDDR0.r0
vboxdrv: e38fd020 VBoxDD2R0.r0
HMR0InitVM: e3aa2000

We see, that the trap address is probably, in VMMR0.r0

[Valery V. Sedletski]

trap 000d in flat code when starting the VM

[Valery V. Sedletski]

Oops, now I see that adresses of *.r0 modules I specified above is for OS/4 kernel (which doesn't trap), so adresses are different. And I cannot see these adresses for 14.104 / 14.106 because they are written to a vboxdrv.sys internal buffer after these modules are loaded. But the trap occurs in the moment close to the moment of loading the modules, so, I cannot catch and see the adresses :(

[Valery V. Sedletski]

I reproduced this trap on my testing machine with a COM port and kernel debugger:

Trap 13 (0DH) - General Protection Fault 0000
eax=00000000 ebx=af24c000 ecx=00000000 edx=00000000 esi=00000202 edi=ffe97000
eip=e5b1bbbe esp=f8cafba8 ebp=f8cafc08 iopl=0 rf -- -- nv up di pl zr na pe nc
cs=0178 ss=1550 ds=0170 es=0170 fs=0000 gs=0000 cr2=005dc000 cr3=00225000 p=00
0178:e5b1bbbe f30fc734       db        f3, 0f, c7, 34
0178:e5b1bbbe f30fc734       db        f3, 0f, c7, 34
0178:e5b1bbc2 2477           and       al,77            ;'w'
0178:e5b1bbc4 0e             push    cs
0178:e5b1bbc5 7407           jz        e5b1bbce
0178:e5b1bbc7 b85cf0ffff     mov       eax,fffff05c
0178:e5b1bbcc eb05           jmp       e5b1bbd3
0178:e5b1bbce b860f0ffff     mov       eax,fffff060
0178:e5b1bbd3 83c408         add       esp,+08
0178:e5b1bbd6 85c0           test      eax,eax
0178:e5b1bbd8 7877           js        e5b1bc51
0178:e5b1bbda c605f855d3e501 mov       byte ptr [e5d355f8],01
0178:e5b1bbe1 0f01c4         sgdt
0178:e5b8029f e5b7ffc0 fd3a8754 00000046 00002955 _StartInitCode + e5b8029f
0178:f214b2bb fd3a8754 e5b80130 e5b846f4 00005a5c _StartInitCode + f214b2bb
0178:f2138b84 00000004 f9e76fd0 fd396634 003a58a0 _StartInitCode + f2138b84
0178:f2138041 00000186 000000c0 00000004 003a58a0 _StartInitCode + f2138041
0178:10000a72 00005b60 00000000 5b600000 04d00578 _StartInitCode + 10000a72
0178:00000000 gradd:SEG2:_StartInitCode + e5b1bbe4
##

[Valery V. Sedletski]

This is a disassembly of the above piece in VMXR0.r0 in hiew:

.00027B85: C705E015240000000000         mov       d,[0002415E0],000000000 ;"
.00027B8F: F6C420                       test      ah,020 ;" "
.00027B92: 7506                         jne      .000027B9A   -------- (1)
.00027B94: 80CC20                       or        ah,020 ;" "
.00027B97: 0F22E0                       mov       cr4,eax
.00027B9A: 31C0                         xor       eax,eax
.00027B9C: 52                           push      edx
.00027B9D: 53                           push      ebx
.00027B9E: F3                           repe
.00027B9F: 0FC73424                     ???       [esp]
.00027BA3: 770E                         ja       .000027BB3   -------- (2)
.00027BA5: 7407                         je       .000027BAE   -------- (3)

where F3 0F C7 34 24 is vmxon qword ptr [esi] opcode -- something related with Intel VMX (VT-x). Why it tries to "enter VMX operation", I don't know yet.

​http://www.tptp.cc/mirrors/siyobik.info/instruction/VMXON.html

The vmxon instruction is in ​http://trac.netlabs.org/vbox/browser/trunk/include/VBox/vmm/hm_vmx.h, VMXEnable inline routine. The place in the code where it's called is hmR0InitIntel routine in ​http://trac.netlabs.org/vbox/browser/trunk/src/VBox/VMM/VMMR0/HMR0.cpp.