DBCS fonts and font association

[KO Myung-Hun]

Hi/2.

On DBCS OS/2, 2 special fonts are supported. They are 'WarpSans Combined' and 'Helv Combined' in 'PS55GUI.FDR'.

Fortunately, these two combined fonts are supported by Qt4 because I can see the text of menu of SMPlayer in Korean.

The font for menu text is 'Helv Combined'. But icon text and window text in Korean are not displayed. The font for them is 'WarpSans Combined'.

I know, Qt 4 does not support bitmap fonts. So 'WarpSans' is replaced with 'Workplace Sans' But 'Workplace Sans' does not include any DBCS glyphs. As a result, any Korean texts are not displayed at all.

So I think, you should not replace 'WarpSans' with 'Workplace Sans' if it is 'WarpSans Combined'.

In addition, DBCS OS/2 has a special feature. It is 'Font Association'. That is, if a specified font does not include a glyph for a specific codeset, OS/2 load that glyph from the associated font.

Primarily, this is used when DBCS chars need to be displayed on a program using SBCS fonts only.

For DBCS users, I think, this feature is very important.

[Dmitry A. Kuminov]

Thank you for the report.

Regarding the title bar text not displayed in Korean, please read the CURRENT LIMITATIONS section of README.OS2 more carefully:

  3. Make sure you have the LANG environment variable properly set. The format
     is 'set LANG=ll_CC[.encoding]' where <ll> is the language code, <CC> is the
     country code and <encoding> is the optional encoding to use. If LANG is
     missing or invalid, string conversion operations may work incorrectly
     resulting in distorted text input or output. Note that for most languages
     you will have to specify the encoding number explicitly because Qt and OS/2
     usually disagree about the default encoding for the given language.

     To specify the correct encoding for Qt you need to know your system code
     page number. You can find this number in the COUNTRY statement of your
     CONFIG.SYS. Note however that the code page number from CONFIG.SYS and the
     encoding name you specify in LANG are different things. Qt doesn't
     understand IBM code page numbers directly. In most cases, you can get the
     encoding name by prepending 'cp' to the code page number (for example,
     'cp850' for code page 850) but sometimes this will not work because not all
     encodings have 'cp'-like aliases. In this case, you should google around to
     find the correct encoding name for your code page number. Here is a couple
     of examples of the proper LANG specification:

       set LANG=de_DE.cp850             - for the German OS/2 locale
       set LANG=ru_RU.cp866             - for the Russian OS/2 locale

     Later, the correct encoding for the system code page will be detected
     automatically and specifying it in LANG will not be necessary.

Most likely, your LANG is set incorrectly or not set at all. Please try to set it and report if it fixes a problem with title bars for you or not.

Regarding the "font association" feature as you call it. Qt4 actually already supports this (selecting a missing glyph from another matching font). The problem is that originally this support worked only at the font family level (e.g. if a glyph is present in any of the existing styles of the given family it is assumed that all styles have it which resulted into bounding boxes being printed instead of the glyph in styles that don't actually have it). However, in Beta 4 I fixed this issue so that if e.g. Bold lacks some glyph which is present in Normal, Qt4 will attempt find some other Bold font that has it (instead of printing the bounding box). I didn't understand if you already suffered from this issue or not but please try Beta 4 to make sure it works for you.

[KO Myung-Hun]

Hi/2.

Replying to dmik:

Thank you for the report.

Welcome.

Regarding the title bar text not displayed in Korean, please read the CURRENT LIMITATIONS section of README.OS2 more carefully:

I didn't mention about the title bar, but I tested it. And it works only with a playing file name. In case of 'File Dialog', 'Preferneces' and so on, the title bar text is replaced with '?' except 'SMPlayer' string.

Most likely, your LANG is set incorrectly or not set at all. Please try to set it and report if it fixes a problem with title bars for you or not.

I'm using 'SET LANG=ko_KR'. Of course, I tested with 'ko_KR.IBM-949' and 'ko_KR.cp949'. But the result did not changed.

Regarding the "font association" feature as you call it. Qt4 actually already supports this (selecting a missing glyph from another matching font). The problem is that originally this support worked only at the font family level (e.g. if a glyph is present in any of the existing styles of the given family it is assumed that all styles have it which resulted into bounding boxes being printed instead of the glyph in styles that don't actually have it). However, in Beta 4 I fixed this issue so that if e.g. Bold lacks some glyph which is present in Normal, Qt4 will attempt find some other Bold font that has it (instead of printing the bounding box). I didn't understand if you already suffered from this issue or not but please try Beta 4 to make sure it works for you.

Ok. I think 'font association' works. Because when I set the font for the window text to 'Arial', I can see Korean texts in Preferences dialog.

However, I set it to 'Workplace Sans', I cannot see Korean texts at all.

And I have a question, how does QT4 determine the font for the font association ? That is, which font does QT4 select for missing glyphs ? The first font having the glyphs ? Or...

Finally, when using a file dialog, all Korean texts are broken into high bit SBCS chars. Which font is used on a file dialog ?

For you understand, I attach some screen shots.

[KO Myung-Hun]

File open dialog that Korean chars are broken into high bit ASCII chars and its title bar text is replaced with '?'

[KO Myung-Hun]

A playing file name and a menu text are displayed well, but not in other control

[KO Myung-Hun]

Preferences dialog that Korean chars are not displayed at all

[KO Myung-Hun]

Replying to komh:

Hi/2.

On DBCS OS/2, 2 special fonts are supported. They are 'WarpSans Combined' and 'Helv Combined' in 'PS55GUI.FDR'.

Fortunately, these two combined fonts are supported by Qt4 because I can see the text of menu of SMPlayer in Korean.

This is wrong. When I set the font to 'Helv Combined', 'font association' seemed to worked because 'Helv Combined' is not supported by FreeType. So it looked as if it worked accidentally.

[Dmitry A. Kuminov]

Re file_dialog.png, the only reason that can cause this is incorrect or missing encoding in LANG (it affects converting file names from the system encoding to Unicode and title bar text from Unicode to the system encoding). What is the COUNTRY value in your CONFIG.SYS?

Re playing_file.png and preferences.png, it looks like you have a different font for menus set in the OS/2 palette and this font contains Korean glyphs (as opposed to Workplace Sans). Qt4 takes fonts from the HINI_USER_PROFILE\PM_Fonts registry key. What do you have there?

On the other side, if you have at least one .PFB/OMF or TTF font that contains *unicode* Korean glyphs, then even if you use Workplace Sans you still should have missing glyphs taken from the font that has them. Are you sure that you made thes screenshots with Beta 4?

Do you have an opportunity to build Qt4 on your computer? If so, I will give you instructions on how to enable font-specific debug output and we will see what's going on in there.

Regarding the font selection algorithm. First of all, this algorith is based on what the font file reports about the supported encodings itself. In TTF fonts, there is a special table that contains flags indicating which Unicode ranges are supported by the font (and I'm pretty sure there are a number of fonts that don't set these flags correctly). PFB/OMF fonts don't have such a table and therefore Qt thinks that they have all Unicode glyphs.

Next, Qt looks if the requested font supports a Unicode range for the glyph it is asked to draw. If it doesn't, then Qt searches for another font family in the list which has it and as much close to the requested font as possible (has same style, weight and stretch, can be rendered at the same size etc.). If it doesn't find such a font, it draws a bounding box instead of the glyph.

[Dmitry A. Kuminov]

Please also try to install some TTF font that definitely contains the Korean Unicode range to OS/2, set it as the system font for everything (Menus, WindowText, WindowTitles, IconText) and see how it behaves. You should at least get the correct text everywhere in Qt except the window titles and file name dialogs.

[KO Myung-Hun]

Hi/2.

Replying to dmik:

Re file_dialog.png, the only reason that can cause this is incorrect or missing encoding in LANG (it affects converting file names from the system encoding to Unicode and title bar text from Unicode to the system encoding).

As I said above, I tested three types of LANG, where they are ko_KR, ko_KR.IBM-949, ko_KR.cp949.

What is the COUNTRY value in your CONFIG.SYS?

082

Re playing_file.png and preferences.png, it looks like you have a different font for menus set in the OS/2 palette and this font contains Korean glyphs (as opposed to Workplace Sans).

As I said above, the font for menus is Helv Combined, but the font for icon text and window text is WarpSans Combined.

Qt4 takes fonts from the HINI_USER_PROFILE\PM_Fonts registry key. What do you have there?

The default system fonts such as Helv and my own fonts such as Gulim, Batang, Times New Roman WT K and so on.

On the other side, if you have at least one .PFB/OMF or TTF font that contains *unicode* Korean glyphs, then even if you use Workplace Sans you still should have missing glyphs taken from the font that has them.

Of course, I have them. As the above, Gulim, Batang, Gungsuh, Dotum and Times New Roman WT K, Times New Roman MT 30 and so on contain unicode Korean glyphs.

Are you sure that you made thes screenshots with Beta 4?

I'm using the following version.

9-11-14 5:59 9,299,144 0 scs_qt4dll.zip

Do you have an opportunity to build Qt4 on your computer? If so, I will give you instructions on how to enable font-specific debug output and we will see what's going on in there.

Hmm... If I can do this myself, I would ask you.

Regarding the font selection algorithm. First of all, this algorith is based on what the font file reports about the supported encodings itself. In TTF fonts, there is a special table that contains flags indicating which Unicode ranges are supported by the font (and I'm pretty sure there are a number of fonts that don't set these flags correctly). PFB/OMF fonts don't have such a table and therefore Qt thinks that they have all Unicode glyphs.

Next, Qt looks if the requested font supports a Unicode range for the glyph it is asked to draw. If it doesn't, then Qt searches for another font family in the list which has it and as much close to the requested font as possible (has same style, weight and stretch, can be rendered at the same size etc.). If it doesn't find such a font, it draws a bounding box instead of the glyph.

If so, in case of Workplace Sans, may does it insist that it has all the unicode range ? I think, this can explain why font association does not work with Workplace Sans.

[KO Myung-Hun]

Hi/2.

Replying to dmik:

Please also try to install some TTF font that definitely contains the Korean Unicode range to OS/2, set it as the system font for everything (Menus, WindowText, WindowTitles, IconText) and see how it behaves. You should at least get the correct text everywhere in Qt except the window titles and file name dialogs.

Yes, if I set it to Times New Roman WT K or Times New Roman MT 30, that is, the fonts having English name only. Or SBCS fonts such as Arial.

But the fonts having both Korean name and English one such as Gulim and Batang, do not work regardless of setting to Korean name or English one.

[Dmitry A. Kuminov]

Replying to komh:

As I said above, I tested three types of LANG, where they are ko_KR, ko_KR.IBM-949, ko_KR.cp949.

And why did you decide that cp949 is the right encoding? (IBM-949 isn't recognized by Qt4 BTW. You may also try EUC-KR instead of cp949 to see if it makes any difference).

What is the COUNTRY value in your CONFIG.SYS?

082

Sorry, I meant the CODEPAGE value.

As I said above, the font for menus is Helv Combined, but the font for icon text and window text is WarpSans Combined.

Okay, I didn't get that you were referring to the PM_Fonts key there. The IconText font isn't used by Qt4 BTW.

Of course, I have them. As the above, Gulim, Batang, Gungsuh, Dotum and Times New Roman WT K, Times New Roman MT 30 and so on contain unicode Korean glyphs.

It must work then. We have to debug it to find the problem.

I'm using the following version.

9-11-14 5:59 9,299,144 0 scs_qt4dll.zip

This seems to be Beta 4.

Do you have an opportunity to build Qt4 on your computer? If so, I will give you instructions on how to enable font-specific debug output and we will see what's going on in there.

Hmm... If I can do this myself, I would ask you.

This should be quite straightforward. If you really can't do that, tell me and I will send you the debug DLLs with the necessary logging turned on. If you have jabber you can try to reach me at dmik-at-jabber-dot-ru for online chat.

If so, in case of Workplace Sans, may does it insist that it has all the unicode range ? I think, this can explain why font association does not work with Workplace Sans.

Here it doesn't. Here, Workplace Sans reports that it supports the following writing systems (almost the same as Unicode ranges): "Latin", "Greek", "Cyrillic", "Hebrew", "Japanese" -- no Korean in there...

[Dmitry A. Kuminov]

Replying to komh:

Yes, if I set it to Times New Roman WT K or Times New Roman MT 30, that is, the fonts having English name only. Or SBCS fonts such as Arial.

But the fonts having both Korean name and English one such as Gulim and Batang, do not work regardless of setting to Korean name or English one.

I'm not really sure that I get you here. What is the "English name" or "Korean name" within the font context? Do you get the correct Korean text on preferences.png when you set the font to Times New Roman WT K?

In the meanwhile, I'm attaching a simple test case, widget.exe, below. Please do the following:

1. Run this executable (the font selection dialog appears).
2. In the Writing system combobox select "Korean"
3. Post here what all the fonts that you find in the list above.
4. Go through each font in this list and check if you see Korean text with any of them in the Sample box of this window.

[Dmitry A. Kuminov]

Testcase with font dialog

[KO Myung-Hun]

Replying to dmik:

Replying to komh:

As I said above, I tested three types of LANG, where they are ko_KR, ko_KR.IBM-949, ko_KR.cp949.

And why did you decide that cp949 is the right encoding? (IBM-949 isn't recognized by Qt4 BTW. You may also try EUC-KR instead of cp949 to see if it makes any difference).

I thought so because Qt4 was based on kLIBC. Anyway, EUC-KR does not make any differences, too.

What is the COUNTRY value in your CONFIG.SYS?

082

Sorry, I meant the CODEPAGE value.

Ok. It's 949.

As I said above, the font for menus is Helv Combined, but the font for icon text and window text is WarpSans Combined.

Okay, I didn't get that you were referring to the PM_Fonts key there. The IconText font isn't used by Qt4 BTW.

Strange. When I change IconText in HINI_USER_PROFILE\PM_SystemFonts, the font of icon text of Preferences dialog is changed.

Do you have an opportunity to build Qt4 on your computer? If so, I will give you instructions on how to enable font-specific debug output and we will see what's going on in there.

Hmm... If I can do this myself, I would ask you.

This should be quite straightforward. If you really can't do that, tell me and I will send you the debug DLLs with the necessary logging turned on. If you have jabber you can try to reach me at dmik-at-jabber-dot-ru for online chat.

Ok. Send me the debug DLLs.

If so, in case of Workplace Sans, may does it insist that it has all the unicode range ? I think, this can explain why font association does not work with Workplace Sans.

Here it doesn't. Here, Workplace Sans reports that it supports the following writing systems (almost the same as Unicode ranges): "Latin", "Greek", "Cyrillic", "Hebrew", "Japanese" -- no Korean in there...

I'm using v0.4 but it is listed in Korean writing systems.

And if I remove Workplace Sans, I can see Korean texts in all the control except a file dialog. But they all are displayed in the same font. I think, this is a effect of a font association.

BTW, What font is used in a file dialog ?

[KO Myung-Hun]

Replying to dmik:

In the meanwhile, I'm attaching a simple test case, widget.exe, below. Please do the following:

1. Run this executable (the font selection dialog appears).
2. In the Writing system combobox select "Korean"
3. Post here what all the fonts that you find in the list above.
4. Go through each font in this list and check if you see Korean text with any of them in the Sample box of this window.

The lists are the following.

Batang
BatangChe
Dotum
DotumChe
Gulim
GulimChe
Gungsuh
GungsuhChe
Malgun Gothic
Times New Roman MT 30
Times New Roman WT K
Workplace Sans

I can see Korean texts with all the fonts but Workplace Sans in Sample box

[KO Myung-Hun]

Replying to dmik:

I'm not really sure that I get you here. What is the "English name" or "Korean name" within the font context?

I meant a face name of a font.

Do you get the correct Korean text on preferences.png when you set the font to Times New Roman WT K?

Yes, if I set it for IconText and WindowText in HINI_USER_PROFILE\PM_SystemFonts.

And Times New Roman MT 30, too.

[Dmitry A. Kuminov]

As far as I remember, Workplace Sans v0.4 doesn't report its supported Unicode ranges correctly -- that would explain why we see it in the font list for Korean and why it prints nothing in the sample box and in the real applications. Please try to install v0.6 -- it should disappear from the font list when you select the Korean writing system. This will prove our assumption.

Also, with v0.6, you should see correct Korean text even if you set Workplace Sans as your only font for everything (WindowText, Menus, IconText, WindowTitles) -- the glyphs should be taken from another font you have there. Please confirm if it's the case or not. (BTW, IconText is indeed used for icons in the Qt icon view; I already forgot that I did it that way :).

By default, the WindowText font is used everywhere in Qt. Exceptions are titles of MDI windows where WindowTitles is used, Qt icons where IconText is used and menus where Menus is used.

Regarding the title bars and filenames. I've got a suspicion that IBM-949 is not quite the same as cp949/eucKR (see ​http://en.wikipedia.org/wiki/CP949). I found this document ​http://www.haible.de/bruno/charsets/conversion-tables/EUC-KR.html from which you see that there is a plenty of various 949 variants and some have major differences compared to eucKR. If you have an exact layout of the OS/2 949 codepage, it would be very nice if you could confirm or confute that.

Anyway, I guess that you will have to wait for completion of #49 which will solve this and similar problems standing because it will use the system routines for doing Unicode conversions (instead of the Qt ones) and therefore will eliminate any possible misinterpretations of encodings.

[Dmitry A. Kuminov]

What I don't fully understand is why the title bar text is OK on playing_file.png but broken on file_dialog.png. Are you sure that you used LANG=ko_KR.cp949 when taking both? playing_file.png looks like it was just LANG=ko_KR (or LANG=ko_KR.something_invalid like IBM-949).

[KO Myung-Hun]

I tested with Workplace Sans v0.6, but there is nothing changed except displaying blank boxes in sample box.

I know, IBM-949 is equivalent to EUC-KR. Of course, there can be some different mappings between them. But we can ignore it, because they both obey the Korean standard charset.

However, CP949, i.e. MS949, is a different case. It's a superset of EUC-KR.

Anyway, I think, if Qt4 implement code conversion for EUC-KR correctly, there is no need to use other implementations.

The encoding parts of LANG is not relevant to the broken title bar text. That is, LANG=ko_KR, LANG=ko_KR.CP949, LANG=ko_KR.MS949, LANG=ko_KR.EUC-KR, LANG=ko_KR.IBM-949.

And I also thought it was so strange a playing file name was displayed correctly on the title bar.

I think, there can be two causes for this.

1. The translation for Korean of SMPlayer is broken.

2. Charset conversion for Korean of Qt4 is broken.

BTW, in case of 2, the playing file name seems to be converted well. So I suspect case 1.

To confirm, it would be better to have a test program which set the title bar text reading unicode text from a file.

[Dmitry A. Kuminov]

Replying to komh:

I tested with Workplace Sans v0.6, but there is nothing changed except displaying blank boxes in sample box.

Do you mean you don't get Workplace Sans in the list when you select the Korean writing system any more or what?

I know, IBM-949 is equivalent to EUC-KR. Of course, there can be some different mappings between them. But we can ignore it, because they both obey the Korean standard charset.

No reason to not believe you.

The encoding parts of LANG is not relevant to the broken title bar text. That is, LANG=ko_KR, LANG=ko_KR.CP949, LANG=ko_KR.MS949, LANG=ko_KR.EUC-KR, LANG=ko_KR.IBM-949.

A possible explanation to this could be that Qt uses EUC-KR by default for the ko_KR locale (though I was unable to quickly find a confirmation in the code).

And I also thought it was so strange a playing file name was displayed correctly on the title bar.

I think, there can be two causes for this.

1. The translation for Korean of SMPlayer is broken.

2. Charset conversion for Korean of Qt4 is broken.

BTW, in case of 2, the playing file name seems to be converted well. So I suspect case 1.

Yeah, but this doesn't explain why Workplace Sans shows bullshit on the first screenshot in the filename list. The symbols there look like it treats the file name as Latin-1 (you can see the multibyte pattern).

To confirm, it would be better to have a test program which set the title bar text reading unicode text from a file.

I'll try to do some test for you later today or tomorrow.

[Dmitry A. Kuminov]

I think that I got it. The Silvan's distribution of Qt binaries doesn't include Qt plugins for some reason while the Korean codec in Qt is compiled as a plugin... This means that the Qt indeed interprets everything as Latin-1.

Attaching the plugin here. Unzip it to the directory along with Qt DLLs and try everything again.

[Dmitry A. Kuminov]

Korean codec DLL for Qt

[Dmitry A. Kuminov]

Err, sorry, in order to let the Qt runtime find the plugin outside the Qt build tree, you need to place it to the .exe directory as it is looked first. So, if your widget.exe is located in C:\Temp\, then you should place qkrcod4.dll to C:\Temp\codecs\. No matter where Qt DLLs themselves reside.

[Dmitry A. Kuminov]

Replying to dmik:

I think that I got it. The Silvan's distribution of Qt binaries doesn't include Qt plugins for some reason while the Korean codec in Qt is compiled as a plugin... This means that the Qt indeed interprets everything as Latin-1.

BTW, this also explains why we see the correct file name in the title bar on the 2nd screenshot -- as Qt thinks it's Latin1, it passes it back to PM unmodified (Latin1 fits the first 8-bit part of the Unicode range).

[Silvan Scherrer]

did not know that i have to include plugins as well :( will update the zip with the included plugins this weekend.

[Dmitry A. Kuminov]

Yeah. I think it's best to unfold all the DLLs from the plugins directory to the same one where the Qt DLLs are, so it will look like:

Qt4Core.dll
Qt4Gui.dll
...
accessible\qtawt4.dll
accessible\qtawtd4.dll
codecs\qcncod4.dll
...

Unfortunately, all these subfolders need to be placed to the EXE directory, otherwise Qt will most likely not find them (so that it should be mentioned in README). This tightly relates to #78.

[Dmitry A. Kuminov]

So, it actually may make sense to put all the plugins into a separate .zip, otherwise people will be forgetting to put them to the application directory.

[KO Myung-Hun]

Replying to dmik:

Replying to komh:

I tested with Workplace Sans v0.6, but there is nothing changed except displaying blank boxes in sample box.

Do you mean you don't get Workplace Sans in the list when you select the Korean writing system any more or what?

No. Workplace Sans is listed in font list when Korean writing system is selected.

And blank boxes is displayed unlike before.

And I also thought it was so strange a playing file name was displayed correctly on the title bar.

I think, there can be two causes for this.

1. The translation for Korean of SMPlayer is broken.

2. Charset conversion for Korean of Qt4 is broken.

BTW, in case of 2, the playing file name seems to be converted well. So I suspect case 1.

Yeah, but this doesn't explain why Workplace Sans shows bullshit on the first screenshot in the filename list. The symbols there look like it treats the file name as Latin-1 (you can see the multibyte pattern).

I think, Workplace Sans problem is irrelevant to encoding. And it would be better to contact Alex Taylor for this problem.

But I second you about the file dialog problem.

[KO Myung-Hun]

I have extracted SMPlayer into g:\ex\smplayer.

And I have copied qkrcod4.dll into both g:\ex\smplayer\codecs and g:\ex\smplayer.

But I cannot see Korean text on the title bar when using file dialog and preferences dialog.

[Dmitry A. Kuminov]

I don't understand why Workplace Sans is listed when you select Korean in the font dialog -- it shouldn't. I will prepare a debug version of Qt with logging for you tomorrow.

Regarding the codec plugin DLL, I confirm that searching in the application directory doesn't work now; I will debug this tomorrow.

[Dmitry A. Kuminov]

This is odd: instantiating a QCoreApplication (and hence QApplication) object (which 99% of applications do) breaks plugin DLL lookup somehow.

[Dmitry A. Kuminov]

Okay, found the reason. These are actually two vendor bugs:

1. Text codec plugin DLLs are loaded before the application (executable) directory is added to the plugin search path. Therefore, the codec plugins residing in there are ignored.
2. Qt sets up the default codec for the current locale before it loads the plugins from the application directory (even with teh fix above applied) and therefore if LANG requests a codec that is repesented by one of the plugins residing in that directory, this request is ignored.

Both issues have been fixed in r340.

[Dmitry A. Kuminov]

komh, I've prepared a debug pack for you and put it to ​ftp://ftp.netlabs.org/pub/qt4/qt4_dbg_for_komh.zip.

Please download, extract to some temporary directory and do the following:

cmd /c set LANG=ko_KR.eucKR && codecs.exe 2>codecs.txt
cmd /c set LANG=ko_KR.eucKR && fonts.exe 2>fonts.txt

Then zip codecs.txt and fonts.txt and attach here.

BTW, we could greatly simplify debugging Qt4 on the Korean OS/2 if you installed it to a .vdi for me (VirtualBox hard disk image). I can assist you in installing OS/2 or eCS inside VirtualBox if needed.

[KO Myung-Hun]

Ok, I attach my codecs.txt and fonts.txt

In case of fonts.exe, now Workplace Sans is not listed for Korean writing system any more.

Unfortunately I don't use VirtualBox. But I'm using Virtual PC and have an image for Warp 4 for Korean with FP #5.

If you're OK with that, I would send it to you.

[KO Myung-Hun]

codecs.txt and fonts.txt

[Dmitry A. Kuminov]

Replying to komh:

Ok, I attach my codecs.txt and fonts.txt

Thanks. It seems that everything's OK now: the euc-KR codec is loaded and activated and Workplace Sans reports the encodings it actually supports.

In case of fonts.exe, now Workplace Sans is not listed for Korean writing system any more.

QtGui*4.dll you have from Silvan must be the same (in terms of font handling) so it's all strange. Let's assume that somebody just overlooked something, as now it works. I will build the release DLLs to let you try it with smplayer.

Unfortunately I don't use VirtualBox. But I'm using Virtual PC and have an image for Warp 4 for Korean with FP #5.

If you're OK with that, I would send it to you.

Okay, please upload it to ​ftp://ftp.dmik.org/incoming. VirtualBox understands Virtual PC images.

[Dmitry A. Kuminov]

Here is the release build you can try with smplayer: ​ftp://ftp.netlabs.org/pub/qt4/qt4_r346_rel_for_komh.zip. Just unzip it to the smplayer.exe's directory.

[KO Myung-Hun]

Thanks, it works well. But some file names are displayed in blank boxes.

Thoses file names contains japanese characters. EUC-KR support japanese characters as well as hanja and cyrill and so on.

I attach the screenshots.

BTW, it's too slow to upload to your ftp site and it's disconnected sometimes. Anyway I would try to upload the image to it, continuously.

[KO Myung-Hun]

Correct file names

[KO Myung-Hun]

Wrong file names on a file dialog

[Dmitry A. Kuminov]

Replying to komh:

Thanks, it works well. But some file names are displayed in blank boxes.

Thoses file names contains japanese characters. EUC-KR support japanese characters as well as hanja and cyrill and so on.

According to fonts.txt you sent me, Workplace Sans reports it supports Japanese while v0.6 actually doesn't. Please try to set Times New Roman MT 30 as the window font or downgrade Workplace Sans to v0.4.

BTW, it's too slow to upload to your ftp site and it's disconnected sometimes. Anyway I would try to upload the image to it, continuously.

Sorry, my current ISP is big POS and there is no alternative so far :( The channel is fat (10Mbit up and down) but it's not stable.

[KO Myung-Hun]

Ok, downgrading Workplace Sans to v0.4 let me see japanese characters.

And I've tested some more.

I made a file contain Hanja and special chars.

Unfortunately, I cannot see the chars.

In spite of setting Times New Roman WT K or Times New Roman MT 30 for the window text, I cannot see them.

The strange thing is that Hangul(Korean chars) is displayed with Times New Roman WT K but English chars and numbers seem to be displayed with Workplace Sans.

And I failed to upload the image to your ftp site many times. So I've uploaded to other site.

You can get it here,

​http://www.ecomstation.co.kr/komh/download/warp4kfp5.zip

I'll remove it after you download it.

[Dmitry A. Kuminov]

Replying to komh:

I made a file contain Hanja and special chars.
Unfortunately, I cannot see the chars.
In spite of setting Times New Roman WT K or Times New Roman MT 30 for the window text, I cannot see them.

Did you make this file in the EUC-KR encoding? Does EUC-KR include Hanja too?

The strange thing is that Hangul(Korean chars) is displayed with Times New Roman WT K but English chars and numbers seem to be displayed with Workplace Sans.

You mean, this happens when when you set the window text font to Times New Roman WT K?

And I failed to upload the image to your ftp site many times. So I've uploaded to other site.

That's strange. Although my ISP may be unstable, it works mos of the time. A guy has successfully uploaded a DVD movie to my ftp just some days back. This must be the the routing then...

You can get it here,

​http://www.ecomstation.co.kr/komh/download/warp4kfp5.zip

I'll remove it after you download it.

Got it, you may delete the file.

[Dmitry A. Kuminov]

komh,

Your VHD file is somehow invalid (VirtualBox refuses to accept it for some reason). Could you please re-check the zip?

Next, I did #49 and want you to test the things. For this, I'm uploading a new QtCore4.dll and the codecs2 test for you here ​ftp://ftp.netlabs.org/pub/qt4/qt4_r350_rel_QtCore_for_komh.zip. Use the other DLLs from qt4_r346_rel_for_komh.zip since they are not changed. Besides running codecs2 please also run smplayer and test things there.

Note that #49 drops the encoding support in LANG. This means that the encoding part of the LANG specification is ignored and having LANG=ko_KR should be enough to get correct Korean characters.

[KO Myung-Hun]

Replying to dmik:

Replying to komh:

I made a file contain Hanja and special chars.
Unfortunately, I cannot see the chars.
In spite of setting Times New Roman WT K or Times New Roman MT 30 for the window text, I cannot see them.

Did you make this file in the EUC-KR encoding? Does EUC-KR include Hanja too?

Of course, I set LANG to ko_KR.EUC-KR. And EUC-KR consists of ASCII, Japanese(Hiragana, Katacana), Greek, Cyrill , Hanja and some special characters as well as Hangul.

The strange thing is that Hangul(Korean chars) is displayed with Times New Roman WT K but English chars and numbers seem to be displayed with Workplace Sans.

You mean, this happens when when you set the window text font to Times New Roman WT K?

Exactly.

[KO Myung-Hun]

Replying to dmik:

Your VHD file is somehow invalid (VirtualBox refuses to accept it for some reason). Could you please re-check the zip?

I've confirmed that it had no problems. I could run that image on my VPC 5.1 for OS/2

Next, I did #49 and want you to test the things. For this, I'm uploading a new QtCore4.dll and the codecs2 test for you here ​ftp://ftp.netlabs.org/pub/qt4/qt4_r350_rel_QtCore_for_komh.zip. Use the other DLLs from qt4_r346_rel_for_komh.zip since they are not changed. Besides running codecs2 please also run smplayer and test things there.

Ok. It works well. I can get the same result with LANG=ko_KR setting as with LANG=ko_KR.EUC-KR before r350.

However, the problem that characters are displayed as Workplace Sans in spite of setting Times New Roman WT K as a window text font, still exists. As well as, some characters such as Hanja and special chars are not displayed.

[KO Myung-Hun]

codecs2.txt

[KO Myung-Hun]

dmik,

I added your jabber account to my contact list.

But it is always offline.

Is this true ? Or does my jabber client(SIM) have a problem ?

[Silvan Scherrer]

Komh,
today it seems dmik is not online at jabber. i did not see him either

[Dmitry A. Kuminov]

komh,

I don't know what it could be with the missing Hanja. Does it make sense to test things under LANG=zh_CN to check if the font can draw Chinese at all? Of course if Hanja doesn't relate to Chinese in terms of glyph codepoints, it doesn't :)

Please also try to temporarily delete Workplace Sans and see what font you will get for Latin in this case.

Regarding the .vhd, I found the reason after reading the VHD specs :) Your .vhd was created by a version previous to Microsoft Virtual PC 2004. And so it has a strange footer size of 511 bytes while newer ones have 512 byte footers. Doing dd if=/dev/zero bs=1 count=1 >> Warp4.vhd on Linux cured it. I'm trying to boot it but it traps with TRAP 006 so far (will try to sort that out).

Regarding jabber. Yeah, I was away most part of today. Please try again (now or tomorrow).

[KO Myung-Hun]

Replying to dmik:

komh,

I don't know what it could be with the missing Hanja. Does it make sense to test things under LANG=zh_CN to check if the font can draw Chinese at all? Of course if Hanja doesn't relate to Chinese in terms of glyph codepoints, it doesn't :)

No sense.

Please also try to temporarily delete Workplace Sans and see what font you will get for Latin in this case.

If I remove it, I can see the text rendered by the font which I set for the window text.

And the problem I cannot see Hanja turned out the conflict problem of the fonts.

I have a non-unicode font. But Qt4 does not support it well.

When I set WarpSans Combined for the window text font, I can see the Latin text as the font, but Hangul is displayed with the other font.

But the strange is why I cannot see Hanja and others.

This was done without Workplace Sans installed.

BTW, I cannot set the font other than Times New Roman WT K and Times New Roman MT 30.

How can I specify other fonts such as Batang and Gulim ?

And how about supporting to specify the default font ?

I would like to see the fonts which I specified, not randomly matched ones.

Regarding jabber. Yeah, I was away most part of today. Please try again (now or tomorrow).

I added you 8 days ago. But you've been always off.

[Dmitry A. Kuminov]

Replying to komh:

Please also try to temporarily delete Workplace Sans and see what font you will get for Latin in this case.

If I remove it, I can see the text rendered by the font which I set for the window text.

And the problem I cannot see Hanja turned out the conflict problem of the fonts.

I have a non-unicode font. But Qt4 does not support it well.

As I told you, Qt4 takes the information about the supported Unicode ranges from the font itself and as Type1 fonts don't provide such info the best Qt can do ATM is assume that they support everything. Of course, we may analyze the COUNTRY and CODEPAGE values of the OS/2 system to derive the supported ranges from there but this requires a lot of knowledge about all OS/2 COUNTRY values, their possible codepages and the corresponding ranges which I don't have. If you or some one else may contribute in this area you are for sure welcome!

When I set WarpSans Combined for the window text font, I can see the Latin text as the font, but Hangul is displayed with the other font.

But the strange is why I cannot see Hanja and others.

I don't know. Since r350, I use the system routines to convert from your local codepage (whatever it is: DBCS, SBCS) to Unicode and back. However, since r350 you should also get '?' in place of characters that cannot be converted. So if you don't see '?' instead of Hanja and others in the window title or in file names, but instead you see an empty space or a box, this means that the codec itself did its work well but the problem is again in the font reporting the ranges it doesn't actually support (which is again true for all PFB fonts). I can suggest you to remove all PFB fonts and install a TTF font that contains everything you need and correctly reports the ranges it supports.

BTW, I cannot set the font other than Times New Roman WT K and Times New Roman MT 30.

How can I specify other fonts such as Batang and Gulim ?
And how about supporting to specify the default font ?

If you mean to take the missing glyphs from, then it's not possible at the moment. The first font (in the order they are internally stored in the list maintained by Qt) that reports it contains the necessary glyphs will be used.

I would like to see the fonts which I specified, not randomly matched ones.

Then you have to specify the fonts that actually contain everything you need and properly report it...

I don't actually know what Qt names the Korean writing system actually includes Hanja and the rest or it only refers to Hangul. According to fonts.txt, the Japanese writing system (in Qt terms) is only detected in Times New Roman WT K and MT 30 and in Workplace Sans (wrongly). Both the Chinese writing systems are only present in Times New Roman WT K and MT 30.

You may actually hack the Qt font cache database in the OS/2 registry to make it think that some font supports the necessary writing system (and otherwise) but I can't easily suggest you here since don't have enough knowledge about Korean and all this stuff.

Please give me an HTML file in utf-8 that contains all the ranges you expect to see on your machine. It will be the best if it's a table in the following format:

<range_description> <range_sample>      <fonts_that_must_display_it>

like

Hangul              [some Hangul chars] Times New Roman WT K, ...

Regarding jabber. Yeah, I was away most part of today. Please try again (now or tomorrow).

I added you 8 days ago. But you've been always off.

I don't know what's wrong. Try to send me a message even if I'm offline. What jabber ID you specified? (please don't use the direct form, I don't like spam).

[KO Myung-Hun]

Replying to dmik:

Replying to komh:

Please also try to temporarily delete Workplace Sans and see what font you will get for Latin in this case.

If I remove it, I can see the text rendered by the font which I set for the window text.

And the problem I cannot see Hanja turned out the conflict problem of the fonts.

I have a non-unicode font. But Qt4 does not support it well.

As I told you, Qt4 takes the information about the supported Unicode ranges from the font itself and as Type1 fonts don't provide such info the best Qt can do ATM is assume that they support everything. Of course, we may analyze the COUNTRY and CODEPAGE values of the OS/2 system to derive the supported ranges from there but this requires a lot of knowledge about all OS/2 COUNTRY values, their possible codepages and the corresponding ranges which I don't have. If you or some one else may contribute in this area you are for sure welcome!

What I said is a TTF font not a Type 1 font. Nevertheless this is not easy.

When I set WarpSans Combined for the window text font, I can see the Latin text as the font, but Hangul is displayed with the other font.

But the strange is why I cannot see Hanja and others.

I don't know. Since r350, I use the system routines to convert from your local codepage (whatever it is: DBCS, SBCS) to Unicode and back. However, since r350 you should also get '?' in place of characters that cannot be converted. So if you don't see '?' instead of Hanja and others in the window title or in file names, but instead you see an empty space or a box, this means that the codec itself did its work well but the problem is again in the font reporting the ranges it doesn't actually support (which is again true for all PFB fonts). I can suggest you to remove all PFB fonts and install a TTF font that contains everything you need and correctly reports the ranges it supports.

I don't have any PFB for Korean, but in according to you, SBCS PFB fonts also report that it has DBCS glyph.

And I can see the correct text on the title bar, but not on a file dialog.

BTW, I cannot set the font other than Times New Roman WT K and Times New Roman MT 30.

How can I specify other fonts such as Batang and Gulim ?
And how about supporting to specify the default font ?

If you mean to take the missing glyphs from, then it's not possible at the moment. The first font (in the order they are internally stored in the list maintained by Qt) that reports it contains the necessary glyphs will be used.

I would like to see the fonts which I specified, not randomly matched ones.

Then you have to specify the fonts that actually contain everything you need and properly report it...

I don't actually know what Qt names the Korean writing system actually includes Hanja and the rest or it only refers to Hangul. According to fonts.txt, the Japanese writing system (in Qt terms) is only detected in Times New Roman WT K and MT 30 and in Workplace Sans (wrongly). Both the Chinese writing systems are only present in Times New Roman WT K and MT 30.

You may actually hack the Qt font cache database in the OS/2 registry to make it think that some font supports the necessary writing system (and otherwise) but I can't easily suggest you here since don't have enough knowledge about Korean and all this stuff.

Please give me an HTML file in utf-8 that contains all the ranges you expect to see on your machine. It will be the best if it's a table in the following format:

<range_description> <range_sample>      <fonts_that_must_display_it>

like

Hangul              [some Hangul chars] Times New Roman WT K, ...

Ok. I've checked out Qt4 trunk, and made the release build successfully. So I'll look into this problem.

Would you mind giving me a hint about where to view at first ?

Regarding jabber. Yeah, I was away most part of today. Please try again (now or tomorrow).

I added you 8 days ago. But you've been always off.

I don't know what's wrong. Try to send me a message even if I'm offline. What jabber ID you specified? (please don't use the direct form, I don't like spam).

dmik(_at_)jabber(_dot_)ru

I can see a image like a cutty robot.

[Dmitry A. Kuminov]

Okay, accodring to fonts.txt you indeed don't have any PFB (Type1) fonts available to Qt4 so we may forget about that.

What I suggest now is that you give me the exact description of what is wrong (the screen shot, the conditions to reproduce it, the list of fonts you use for window text, title bar and so on) and what you would expect. As many details as possible. I will try to reproduce it here on the Korean Warp4 in VirtualBox and then debug. Right now I don't really understand what is wrong.

In the meanwhile you may try playing with Qt4 yourself since you have it built now.

[KO Myung-Hun]

I'm trying to fix DBCS related issues, and I have good results.

1. Supports DBCS face name of the fonts.
2. Supports DBCS face name specified in PM_SystemFonts
3. Try PM_AssociateFont at first if the specified font is not available.

But, when WorkPlace Sans installed, it is still used for Latin1 chars even though the specified font in PM_SystemFonts is available.

I would look into this problem more.

And after cleaning-up th codes, I'll attach the patches.

[KO Myung-Hun]

I found the cause of the Workplace Sans problem. The filename on a file dialog is displayed with a icon text not a window text. When I set a font for a icon text, filenames on a file dialog are displayed well on my purpose.

[Dmitry A. Kuminov]

Very good! Just one question: what is PM_AssociateFont? Is it a special registry key used by PM on DBCS systems or something you just add manually for Qt?

[KO Myung-Hun]

PM_AssociateFont is a feature of PM. So it can be used on SBCS OS/2. But it does not exist on SBCS OS/2 by default. In practice, I'm using it on my eCS 1.2MR to display Korean chars.

[KO Myung-Hun]

Patch for DBCS font issues

[KO Myung-Hun]

I've attached a patch, review please.

[Dmitry A. Kuminov]

Gread job, komh! And it doesn't seem to break SBCS systems :) I committed your patch in r371 but want to point you to qfont_pm.cpp: is it intentional that if PM_AssociateFont is present in the registry, it will be queried each time qt_fontFamilyFromStyleHint() is called? Keep in mind that this function may be called VERY often (each time a small piece of text is drawn) and therefore degrade performance.

[KO Myung-Hun]

It is intentional, and I tried to check it only once. But I've came to know that it's the case of the absence of PM_AssociateFont. If it is present, It'll check PM_AssociateFont. This is my mistake.

I've attach a fixed patch, and I used an other approach to add PM_AssociateFont to the search list of fonts.

[KO Myung-Hun]

Check PM_AssociateFont only once

[Dmitry A. Kuminov]

Okay, committed in r373.

[Dmitry A. Kuminov]

In r379, I resurrected the support for the IME input box taken from Qt3 (see ​http://svn.netlabs.org/qt3/changeset/170/trunk?old=168&old_path=trunk and ​http://svn.netlabs.org/qt3/ticket/28 for more info).

komh, please test if it works for you.

[Dmitry A. Kuminov]

Ok, komh confirms that it works to some extent, see r387 for details.