VT-x/AMD-V

[dmik]

This is a general ticket to collect information related to (potentially) making VBox use VT-x/AMD-V for virtualization.

[dmik]

There are some fundamental problems getting the VT-x and AMD-V code working on OS/2 (according to Knut):

1. Too little kernel stack.
2. No cross CPU calls mechanics (at least not exposed) that allow execute a function on all CPUs from kernel mode.

[dmik]

Also, we have problems with generating 64-bit objects from asm code (needed to run 64-bit guests on 32-bit hosts). This is somehow related as it is one of the reasons to have VT-x/AMD-V. In particular, an attempt to compile src/VBox/VMM/VMMSwitcher/32BitToAMD64.asm with NASM 2.11.08 ends up in this:

VMMSwitcher/LegacyandAMD64.mac:588: error: obj output format does not support 64-bit code
VMMSwitcher/LegacyandAMD64.mac:636: error: impossible combination of address sizes
VMMSwitcher/LegacyandAMD64.mac:636: error: invalid effective address
VMMSwitcher/LegacyandAMD64.mac:638: error: impossible combination of address sizes
VMMSwitcher/LegacyandAMD64.mac:638: error: invalid effective address
VMMSwitcher/LegacyandAMD64.mac:652: error: impossible combination of address sizes

and so on.

So far, I disabled compiling this and similar sources on OS/2 (see r33).

[dmik]

It seems that there is a fix to NASM (by Knut of course) that makes it able to generate 64-bit object files: ​http://sourceforge.net/p/nasm/mailman/message/34609638/. We need to apply this patch to our NASM SVN and then add options it adds to VBOX_ASFLAGS, VBOX_ASFLAGS32, and VBOX_ASFLAGS64.

[diver]

new nasm is in exp rpm repo.
​http://trac.netlabs.org/ports/changeset/1236 and ​http://trac.netlabs.org/ports/changeset/1237 has the enhancement added

[erdmann]

About function executing on all CPUs: once there was a time where ACPI.PSD provided such a feature. However, this was removed with the versions of ACPI.PSD that David Azariewicz released. Maybe there is a chance to readd this functionality ???

[Valery V. Sedletski]

2Lars: could you point me please to the link, where can I read about that? I looked at ACPI toolkit from David Azarevicz, and did not found such info. I heard that there was some API in ACPI, but did not found the public info. Maybe, there's old ACPI toolkits somewhere?

[erdmann]

You won't find it in the toolkit because the functionality has been removed from ACPI.PSD. I think the last version it was still in was 3.14 but I am not sure.

The best you can do is to have a look at the ACPI repo here at Netlabs.org and browse back in time. There was this routine called "ExecuteSMPFunction" or so.

Initially, Pasha had added this functionality for setting MTRR's (indirectly used by GRADD.SYS I seem to remember). Later David added MTRR (and PAT) support directly into ACPI.PSD which included "cycling through all cores" to set the MTRR registers identically for each core.
If you now have a good reason to have such feature (execute a function on all cores) for other uses, and I think you do, maybe you can convince David to put it back in.
I think this functionality was offered by the PSD_APP_COMM PSD entry point either directly through a dedicated function number or indirectly via another indirection.

[erdmann]

Found some:

​https://trac.netlabs.org/acpi/browser/branches/3.14_lars/src/acpi/psd/acpicpu.c

The function was called: "AcpiExecSMPFunction". As I said, it was removed (or replaced by a dummy).

[Valery V. Sedletski]

2Lars:

I think the last version it was still in was 3.14 but I am not sure.

there were Pasha's ACPI versions until 3.19, at least, as I know.

"cycling through all cores"

-- really, it should be not a cycling, but a parallel execution of the same function on all CPU's. This is what is called "CPU rendezvous", a situation, which is opposite to a spinlock. Really, this could be useful for setting some per-CPU data structures in memory, which need to be synchronized. So, if they made cycling, that is not correct. But, we need to investigate this, of course.
And also, PSD entry points are seen by OS/2 kernel, only. They could not be called and are not seen by other drivers. But as I know, acpi.psd contains also a PDD driver inside it, so, it could be, probably, called via IDC from other drivers. But here, some cooperation with ACPI developers is needed.

If you now have a good reason to have such feature (execute a function on all cores) for other uses, and I think you do, maybe you can convince David to put it back in.

yes, I agree.

[erdmann]

I don't know how parallel execution was achieved for "AcpiExecSMPFunction". I remember the use of IPIs. Maybe that needs to be changed to IPI broadcast to all cores so that the function can be executed on receiving the IPI. David will know for sure.

PSD_APP_COM is not called by kernel. Instead it was designed to be used by ring-3 apps (but the corresponding DLL API function "DosCallPSD" was removed by IBM for the Warp SMP kernel). But now that you mention it, it might have also be offered either by some IDC or IOCTL. I remember that GRADD.SYS was using that functionality therefore it was available for drivers.

[Valery V. Sedletski]

2dmik: As I understood, the Knut's fix seems to be not about generating the 64-bit object files, but about generating a bit of 64-bit instructions inside the 32-bit object file. It disables emitting an error when doing such things.

2Lars: Yes, it is DosCallPSD. But we need to call such function from drivers/ring0 DLL's. Probably, via IDC.

And one more note: We need VT-x/AMD-V and Nesting paging implementetion. I tried to search the places where corresponding code should be added. And it seems that the place for VT-x/AMD-V should be VMM (aka Virtual Machine Monitor). But as I can see, it uses system-independent features only. The required vmcall/vmenter/vmexit/etc. instructions are in inline subroutines (in VMM itself and in include/VBox/vmm, more specific hm_svm.h and hm_vmx.h, hm.h). At least, no subdirs with linux/nt/etc.-specific code in it. The most system-dependent code is in Runtime (separated, r0drv and r3). And I can see that the place for nesting paging is Runtrime/r0drv/os2/memobj-r0drv-os2.cpp. Here is the all memory management performed, including paging. And I see that that file is only 20 KB in size, whereas linux and windows versions are twice larger, so there are much things to be implemented here.

[Valery V. Sedletski]

Also, the following note: The main reasons behind using "CPU rendezvous" in VBox are 1) updating and syncing GIP (Global Interface Page -- some structure with global parameters) TSC counters on all CPU's in parallel 2) switching on/off VT-x/AMD-V on all CPU's in parallel 3) making some CPU's online/offline.

The 2nd reason is the most important. But a thought then: If we can't enable VT-x/AMD-V on all CPU's in parallel and we can execute VBox on a single CPU, what if we enable it on that CPU and make VBox execute on the same fixed CPU? (use CPU affinity function for all VBox threads, or set an "execute on core 0" bit on VBox binaries)?

Then we could use our existing RTMpOnAll() implementation (executing the function on all CPU's in parallel, which currently does that on a current CPU only), and all should work. The only requirement is to cause all VBox threads to be scheduled on a fixed CPU only, otherwise we could have traps in the support driver.

[erdmann]

There is a solution to executing a function on all cores at the same time: an IPI to "all including self".