java.exe -server fails

[Yoda_Java6]

Initial test of Alpha seems to have a problem.
When I run the this:

L:\Java6\bin>java.exe -server

result is:

Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

It doesn't help specifying any app

I have 1.6GB free mem, so that is not the problem.

[Silvan Scherrer]

does it run w/o -server?

[dmik]

According to Yoda, it works w/o -server.

Anyway, I checked both java -server and java -server -version with my recent production build (future Beta) and it works.

I also checked the alpha build of Java and it also works with the above switches. Since in both cases I run it with the latest Odin, my guess is that the problem is somehow related to Odin. Yoda, what version of Odin do you use?

[dmik]

The -server switch works with Beta here. Please try to update Odin and Java. Feel free to reopen this bug if you find that it still doesn't work.

[Yoda_Java6]

w/o server = -client and yes, that works (shows help).
I would expect -server to do same.

Updated Odin and Java to latest build. Didn't help. Same crash.
Tried on another boot-partition (which are basicly a virgin eCS 2.0 RC7)
and problem is the same there.

Another problem here seems to be, that I do not get any emails from others
comments.

[Yoda_Java6]

Just tested it on my laptop with 1GB RAM.
Here it loads correctly with -server (shows help)

Anyone have tested this on a PC with 3GB RAM (or more) ?

[Yoda_Java6]

More testing.....

Laptop with 1GB RAM used "VIRTUALADDRESSLIMIT=1536" (OK)

Desktop with 3GB RAM used "VIRTUALADDRESSLIMIT=1024" (Failed)
Tried with "VIRTUALADDRESSLIMIT=1536" failed !
Tried with "VIRTUALADDRESSLIMIT=2048" Works !
Tried with "VIRTUALADDRESSLIMIT=1792" Works !

Please check why -server requires a change from 1024 -> 1792 for it to work
(it still just shows a help screen)

AFAIK, eCS 2.0 GA comes with 1536 as default value - that will make it fail here !

Why is this high value required ?

[dmik]

I have a vanilla eCS 2.0 and VIRTUALADDRESSLIMIT is exactly 1536 here while the total amount of RAM is 2G (512M of which are the HPFS386 cache). And I don't see any problems with -server here.

[dmik]

If I set it to 1024, -server does not work indeed. It clearly complains that it cannot allocate enough memory:

Error occured during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

Java needs a lot of virtual memory, that's why it together with Odin uses high memory and that's why we need addressing above 512M (achieved by using VIRTUALADDRESSLIMIT). I think that the value of 1536 is simply a minimal requirement for the current OpenJDK version. I will collect more info (and will check what exactly it allocates) and we will add it to README.OS2 if so.

BTW AFAIR InnoTek? Java had the same issue -- it wouldn't work w/o a certain setting of VIRTUALADDRESSLIMIT (which could be lower though due to lower memory requirements).

[dmik]

Setting VIRTUALADDRESSLIMIT to 2048 prevents HPFS386 from allocating 512M of cache, it can only allocation 28902KB in such conditions. I really wonder why.

[David McKenna]

I have 3GB RAM. With VIRTUALADDRESSLIMIT=1536, java -server fails exactly as yoda described. With VIRTUALADDRESSLIMTI=2048 java -server works. I'll leave it that way for awhile. eCS 2.0GA.

[Yoda_Java6]

Replying to dmik:

If I set it to 1024, -server does not work indeed. It clearly complains that it
cannot allocate enough memory:

right - then we can all reproduce same error.

Java needs a lot of virtual memory, that's why it together with Odin uses high
memory and that's why we need addressing above 512M (achieved by using
VIRTUALADDRESSLIMIT). I think that the value of 1536 is simply a minimal
requirement for the current OpenJDK version.

1536 makes 1GB virtual mem available to every app. It is completely in sane, that
Java (-server) allocates around 1GB, even without any app called. eCS simply can't
handle values this high, in a good manner.

I will collect more info (and will check what exactly it allocates)

Please do, and see if it can be adjusted down by 50%

and we will add it to README.OS2 if so.

ok.

BTW AFAIR InnoTek? Java had the same issue -- it wouldn't work w/o a certain setting
of VIRTUALADDRESSLIMIT (which could be lower though due to lower memory
requirements).

That is right - and that was also a big problem for eCS.
We need to make it better this time.

If all was as simple as setting VAL to 2048 or higher, there would be no problem.
However, setting VAL=2048 makes it impossible to use other than extremely small caches for JFS and HPFS(386) - and it makes it almost impossible to use RAMFS.
It will also make system unbootable, if people try to use big debug drivers like
UNIAUD - or they will get problems chkdsk'ing big or many partitions during boot.

If at all possible, Java6 should work with VAL=1024 - just like all other OS/2 apps does.

[Yoda_Java6]

A test starting Font2DTest.jar (-client)allocates:
Totals: 1EB43000 0332C000 01B1C000 00000000 (in bytes)

503052 52400 27760 0 (in Kbytes)

491.262 51.172 27.110 0.000 (in Mbytes)

In -server mode:
Totals: 3FD2B000 06E98000 02BA4000 00000000 (in bytes)

1045676 113248 44688 0 (in Kbytes)

1021.168 110.594 43.641 0.000 (in Mbytes)

Allocating 1GB for nothing ? No wonder Java apps kill OS/2

Even the 500MB in client mode makes very little sense.

[dmik]

For nothing? Font2DTest.jar isn't a typical application from the memory requirements POV. They reserve what they want to and it's a problem of our OS that it can't live with that.

Anyway, I think that I will change the code to adapt dynamically to the current VAL value when allocating the default Java memory pool. Like, min (1G, 100%) in -server mode and min (512K, 100%) in -client mode.

[dmik]

I need more information about the real numbers. I'm attaching a small test program here that collects system values. Here are my results with HPFS386 and 512M of cache:

                                          VIRTUALADDRESSLIMIT :          1024 :          1536 :          2048 :          3072
                                                             :               :               :               :
11 Major version number                                       :            20 :            20 :            20 :            20
12 Minor version number                                       :            45 :            45 :            45 :            45
13 Revision number                                            :             0 :             0 :             0 :             0
17 Total physical memory size, bytes                          : 2 122 559 488 : 2 122 559 488 : 2 122 559 488 : 2 122 559 488
18 Total resident memory size, bytes                          :   758 525 952 :   758 550 528 :   217 763 840 :   218 562 560
19 Maximum memory size allocatable by all processes, bytes    : 3 490 541 568 : 3 490 541 568 : 3 492 335 616 : 3 486 474 240
20 Maximum private memory size per process, bytes             :   279 707 648 :   279 707 648 :   279 707 648 :   279 707 648
21 Maximum shared memory size per process                     :   239 009 792 :   239 009 792 :   239 009 792 :   238 944 256
27 Maximum free block of process's high private memory, bytes :   452 788 224 :   922 550 272 : 1 392 312 320 : 2 331 836 416
28 Maximum free block of process's high shared memory, bytes  :   423 493 632 :   893 255 680 : 1 363 017 728 : 2 302 541 824
30 Size of the user's address space, megabytes                :         1 024 :         1 536 :         2 048 :         3 072

-- hpfs386 cache                                              :     524288 KB :     524288 KB :      28902 KB :      29516 KB

Note that with the values above 1536, HPFS386 becomes unable to allocate its 512M and only allocates 30M instead.

These are results with raw HPFS, for comparison:

                                          VIRTUALADDRESSLIMIT :          1024 :          1536 :          2048 :          3072
                                                              :               :               :               :
11 Major version number                                       :            20 :            20 :            20 :            20
12 Minor version number                                       :            45 :            45 :            45 :            45
13 Revision number                                            :             0 :             0 :             0 :             0
17 Total physical memory size, bytes                          : 2 122 559 488 : 2 122 559 488 : 2 122 559 488 : 2 122 559 488
18 Total resident memory size, bytes                          :   187 822 080 :   187 801 600 :   187 936 768 :   187 760 640
19 Maximum memory size allocatable by all processes, bytes    : 3 490 840 576 : 3 490 791 424 : 3 490 619 392 : 3 490 824 192
20 Maximum private memory size per process, bytes             :   279 707 648 :   279 707 648 :   279 707 648 :   279 707 648
21 Maximum shared memory size per process                     :   239 992 832 :   240 058 368 :   206 307 328 :   240 058 368
27 Maximum free block of process's high private memory, bytes :   452 788 224 :   922 550 272 : 1 392 312 320 : 2 331 836 416
28 Maximum free block of process's high shared memory, bytes  :   423 493 632 :   893 255 680 : 1 363 017 728 : 2 302 541 824
30 Size of the user's address space, megabytes                :         1 024 :         1 536 :         2 048 :         3 072

[dmik]

Forgot to mention that it's eCS 2.0 GA.

Several observations:

1. The OS makes two approximately equal regions of private and shared memory arenas, the sum of which increases with increasing the value of VIRTUALADDRESSLIMIT (VAL). Thus, the statement of Yoda that VAL reduces the shared memory arena size is not valid.

2. It looks like HPFS386 fails with memory allocation in case if the difference between the total physical memory size (17) and the resident memory size (18, the area where the driver memory and caches are allocated by the kernel AFAIR) is smaller than the sum of both high memory arenas it creates for the given VAL value. It's unclear why the kernel places such a limitation -- the VAL value has nothing to do with the real memory sizes. Probably it's just a kernel bug.

BTW, the default VIRTUALADDRESSLIMIT value (i.e. if this statement is missing from CONFIG.SYS) is indeed 1024 in the eCS kernel.

[dmik]

Please use the attached QuerySysValue?.exe program to collect the values on your system and post them here (with the different VAL values, as in my tables).

[Silvan Scherrer]

my values

 1 Maximum path name length, chars                            :           260:           260:           260
 2 Maximum number of text sessions                            :            16:            16:            16
 3 Maximum number of PM sessions                              :            16:            16:            16
 4 Maximum number of DOS sessions                             :         1 025:         1 025:           780
 5 Boot drive (1 - A, 2 - B...)                               :             3:             3:             3
 6 Dynamic priority variation flag                            :             1:             1:             1
 7 Maximum wait, s                                            :             1:             1:             1
 8 Minimum time slice, ms                                     :            32:            32:            32
 9 Maximum time slice, ms                                     :            32:            32:            32
10 Memory page size, bytes                                    :         4 096:         4 096:         4 096
11 Major version number                                       :            20:            20:            20
12 Minor version number                                       :            45:            45:            45
13 Revision number                                            :             0:             0:             0
14 32-bit millisecond counter value                           :     8 730 097:       108 416:        77 035
15 Low 32 bits of seconds since 01.01.1970                    : 1 294 330 685: 1 294 330 897: 1 294 331 074
16 High 32 bits of seconds since 01.01.1970                   :             0:             0:             0
17 Total physical memory size, bytes                          : 1 072 082 944: 1 072 082 944: 1 072 082 944
18 Total resident memory size, bytes                          :   162 627 584:   160 862 208:   160 940 032
19 Maximum memory size allocatable by all processes, bytes    :   912 355 328:   914 771 968:   914 673 664
20 Maximum private memory size per process, bytes             :   279 904 256:   279 904 256:   279 904 256
21 Maximum shared memory size per process                     :   267 649 024:   268 238 848:   268 238 848
22 Timer interval, tenths of ms                               :           310:           310:           310
23 Maximum length of one component in a path name, chars      :           255:           255:           255
24 Session ID of the current foreground full-screen session   :            21:            17:            17
25 Process ID of the current foreground process               :           139:            94:            94
26 Number of processors in the machine                        :             1:             1:             1
27 Maximum free block of process's high private memory, bytes :   922 550 272:   452 788 224: 1 392 312 320
28 Maximum free block of process's high shared memory, bytes  :   891 158 528:   421 396 480: 1 360 920 576
29 Maximum number of concurrent processes                     :         1 025:         1 025:           780
30 Size of the user's address space, megabytes                :         1 536:         1 024:         2 048
31 INT10ENABLED                                               :             1:             1:             1
32 Undocumented                                               :             0:             0:             0

[dmik]

Thank you Silvan. As we can see, the parameters 27 and 28 don't depend on the actual size of the physical memory. So what I'm going to do is use parameter 27 to estimate the amount of the Java memory size, as follows:

• client: size = min (p[27]*0.5, 512M)
• server: size = min (p[27]*1.0, 1024M)

[dmik]

This turned out to be more complex than that. Java uses the following strategy when calculating the default value for the maximum Java heap size (i.e. when no -Xmx is explicitly specified on the command line), briefly (the relevant code is in Arguments::set_heap_size()):

1. Detects the RAM value used as a base for calculations, using this formula: MIN(<physical_RAM_size>, MaxRAM), where MaxRAM is a constant that varies: for -client, it's 1G and for -server it's 4G (MaxRAM can be also set on the command line with -XX:MaxRAM=<size>).

2. Divides the resulting base by the MaxRAMFraction value which is by default 4 (can be changed on the command line with -XX:MaxRAMFraction=<n>.

3. Assigns the result to MaxHeapSize?.

3. Uses the MaxHeapSize? value to allocate a single block of uncommitted memory for the heap. More over, it actually allocates some 20-30% more -- I assume that it also needs memory for some other needs besides the heap.

So, in case of OS/2, the allocation will obviously fail when the value of MaxHeapSize?*1.3 is smaller than the value of Parameter 27 (maximum high memory block available for the process). This e.g. happens in -server mode, when VAL is 1024 (so that the max block size is about 430M) and the amount of RAM is 2G and more (so that Java wants to allocate 512M).

[dmik]

The solution I used in rXXXXX is the following:

1. Take the value of Parameter 27.
2. Reduce it by 30%.
3. Use the result as an upper limit for the amount of memory Java wants to allocate.

This made Java work with VAL=1024 in -server mode -- where Java got something about 310M instead of 512M wanted (2G/4). It works in -client mode too, of course, as well as with higher VAL values.

However, as I state in the comments, the threshold of 3/10 is quite arbitrary (based on my own experiments) and may fail under some circumstances (or with the new version of Java). If that happens, it is always possible to fix the problem instantly by specifying the MaxHeapSize? value manually on the Java command line using -Xmx. We may also want review the threshold value (or the whole algorithm) in that case.

[dmik]

rXXXXX above is r228 of course :)

[dmik]

In r229, I added some information about things discussed above to README.OS2 so that the users are also aware. Closing the defect.