Dead keys produce diacritic character output (diacritic+combined characters, instead of combined only) in PM version

[Alfredo Fernández Díaz]

Background: in most languages, accented/combined/composite characters are typed in by pressing a sequence of two keys: a "dead" key for the diacritic, followed by a "normal" key for the base character, f.e. I should press ´ + a to get á.

This works as expected in VIO eFTE2, but in PM eFTE2 two characters are produced: "´á", "¨ö", etc. (I kind of understand this, because getting keyboard input is totally different in PM and VIO.)

FTEPM/2006 has the same problem.

I suspect PM actually produces two characters for these combinations, somehow marking the first one for replacement, and it is up to the application to handle it: both system E and Aaron Lawrence's AE show the diacritic character *after* the cursor when a dead key is pressed, and when a normal key is pressed, the accent character is replaced and the cursor position is moved after the combined character as normal.

Example: I press "dead" "^" and I see "^", then I press "A", and "^" is replaced with "Â" and cursor is moved.

EPM does not ever show the accent character nor moves the cursor, it only shows the final combined character after the normal key is pressed.

[Gregg Young]

This must be code page, language and/or country specific or I don't understand what you mean. This doesn't work in anything here.

[Alfredo Fernández Díaz]

I take it you have never had to type a composite character like "á", then? You have to press a sequence of two keys to get it.

I am attaching some captures of PM editors output after key 1 and key 2 are pressed, in case that clarifies the problem:
-E/AE: weird thing displaying ´ and not moving the cursor, but all is well that ends well.
-EPM: output only after key 2; everything is correct.
-eFTEPM: WRONG.

[Alfredo Fernández Díaz]

E/AE output after first key, "dead" ´, is pressed.

[Alfredo Fernández Díaz]

E/AE output after second key, normal "a", is pressed.

[Alfredo Fernández Díaz]

EPM only diplays the combined character "á" after the key sequence is completed.

[Alfredo Fernández Díaz]

EPM will tell you that in CP850, "á" has ASCII code 160

[Alfredo Fernández Díaz]

eFTEPM: first key ("dead" ' ), first output character ´. Cursor moves like we're done.

[Alfredo Fernández Díaz]

eFTEPM: second key (normal "a" ), second output character "á", "first character" is kept. WRONG.

[Alfredo Fernández Díaz]

eFTEPM: second key (normal "a" ), second output character "á", "first character" (unsolicited) is kept. WRONG.

[Alfredo Fernández Díaz]

I sent the last capture twice because of connection hiccups. Please pay no heed.

[Alfredo Fernández Díaz]

Tried to make the description and summary clearer.

[lewisr]

Alfredo, could you please post your CONFIG.SYS settings for codepage, locale, and anything else which might bear on this?

Another thought I had was that maybe we're grabbing one of those keys for something else (but not really using it), thus the modifier is not being caught.

[Alfredo Fernández Díaz]

Well, the modifier is being caught by FTEpm, since a character (the modifier character) is produced in it. You mean something might interfere with the character being interpreted as a modifier so it's not dropped before combining with the regular one?

Anyway, Alex Taylor and I found some strange restrictions in "codepage" and "country" combinations while trying stuff for other languages, so my current setup is a bit offbeat:

COUNTRY=001,C:\OS2\SYSTEM\COUNTRY.SYS (instead of 034, less CP restrictions)
CODEPAGE=850,866 (this is for Russian)
DEVINFO=KBD,SP,C:\OS2\KEYBOARD.DCP (my own one, working perfectly in VIO, unused by PM)
SET LANG=es_ES_EURO

but just in case, I fired up my dad's old eCS 1.2mr installation to make sure and (of course) I see the same problem in ftepm with

COUNTRY=034,C:\OS2\SYSTEM\COUNTRY.SYS
CODEPAGE=850,437
DEVINFO=KBD,SP172,C:\OS2\KEYBOARD.DCP (original)
SET LANG=es_ES_EURO

If I had commit access I would check PM theories with Keith Merrington -- he knows both PM and dead keys internals -- I was in his presentation on the subject back in ca. 2005.

[Gregg Young]

Replying to mrwarper:

Well, the modifier is being caught by FTEpm, since a character (the modifier character) is produced in it.

I thought that this didn't work in the PM version. Please clarify

Anyway, Alex Taylor and I found some strange restrictions in "codepage" and "country" combinations while trying stuff for other languages, so my current setup is a bit offbeat:

COUNTRY=001,C:\OS2\SYSTEM\COUNTRY.SYS (instead of 034, less CP restrictions)
CODEPAGE=850,866 (this is for Russian)
DEVINFO=KBD,SP,C:\OS2\KEYBOARD.DCP (my own one, working perfectly in VIO, unused by PM)

Is this available generally? If not can you send it to me? Do the dead keys work in ae.exe with this setup?

SET LANG=es_ES_EURO

but just in case, I fired up my dad's old eCS 1.2mr installation to make sure and (of course) I see the same problem in ftepm with

COUNTRY=034,C:\OS2\SYSTEM\COUNTRY.SYS
CODEPAGE=850,437
DEVINFO=KBD,SP172,C:\OS2\KEYBOARD.DCP (original)

Do the dead keys work in ae.exe with this setup?

SET LANG=es_ES_EURO

If I had commit access I would check PM theories with Keith Merrington

Commit access to what?

-- he knows both PM and dead keys internals -- I was in his presentation on the subject back in ca. 2005.

Is there a copy of the presentation somewhere?

[Alfredo Fernández Díaz]

Replying to gyoung:

I thought that this didn't work in the PM version. Please clarify

OK, let's try again: it doesn't work in eFTE-PM. Two keystrokes are always required in input (diacritic or combining character, then base character) to produce a combined character, but in eFTE-PM two characters are produced as output (accented space followed by accented character) instead of the single one that comes out --right as expected-- in all other PM applications. Is that not clear? Now, obviously some code in eFTEPM must be detecting the first key, because a matching character is part of the output. My hypothesis after Lewis' comment is, a failure to hand the first character over to eFTE code as a diacritic, presenting it as 'regular' -- but I have no actual idea about how eFTE code interacts with PM, or how PM handles combining characters, so I am not to be taken very seriously there yet.

DEVINFO=KBD,SP,C:\OS2\KEYBOARD.DCP (my own one, working perfectly in VIO, unused by PM)

Is this available generally? If not can you send it to me? Do the dead keys work in ae.exe with this setup?

Dead keys work with this setup in ALL PM applications except eFTEPM. DCP should be unrelated since the mechanism is not used by PM, but never mind. My DCP tools have never been released -- I'm sending you a copy by email.

[...]

DEVINFO=KBD,SP172,C:\OS2\KEYBOARD.DCP (original)

Do the dead keys work in ae.exe with this setup?

Yes, they do.

[...]

If I had commit access I would check PM theories with Keith Merrington

Commit access to what?

If I had SVN commit access to the eFTE repository (i.e. the possibility to release code on my own at some point) I would start experimenting with the code at home, instead of sitting around here waiting. Did you hear back from Adrian?

-- he knows both PM and dead keys internals -- I was in his presentation on the subject back in ca. 2005.

Is there a copy of the presentation somewhere?

I can't find it online now, but you can ask him. Depending on how much you know about keyboard input and non-English characters you might want to have a look at his article at

​http://www.os2ezine.com/20041116/page_3.html

first (it is user level, and does not gloss over PM keyboard handling).

It has some encoding problems, so you'll need to view it using CP1252 to display correctly the "Ž" in the title, and CP850 for the accented characters in the article body.

[Alfredo Fernández Díaz]

Summarizing my mail exchange with Keith:

[...] This is how I think the PM editors I tried work: (e)FTE(2) does not use a standard MLE control to display text (I haven't had a close look at the PM sources yet, I judge from the looks of it), so PM would pass characters along to FTE code which, in turn, simply does not know a thing about combining characters and ignores any such flags, or maybe its code is actually buggy. Other PM editors such as E / AE would use standard MLE PM controls and have all the combining of diacritics done for them by the PM, and EPM does combining on its own -- which would be why input behaves differently from E/AE in its main dialogue (which also looks different from the MLE controls I know) but exactly like E / AE everywhere else, where controls sure look standard.
[...] If I am right [...]

I got a kind reply from him with this pithy comment:

Normaal in an editor I woud not expert the use of an MLE since this only provides a limited editing capability. In PM programming the keyboard is handled with a WM_CHAR message. This provides a complete picture of all the key actions including dead keys.

So maybe there was really no more to it. I found some time during my last short holidays to further check this, and in PM3.inf I read

---

WM_CHAR - Syntax

This message is sent when an operator presses a key.

param1

USHORT fsflags /* Keyboard control codes. */

[...]

---

WM_CHAR field - fsflags

fsflags (USHORT)
Keyboard control codes.

[...]

KC_DEADKEY
The character code is a dead key. The application is responsible for displaying the glyph for the dead key without advancing the cursor.

KC_COMPOSITE
The character code is formed by combining the current key with the previous dead key.

[...]

---

Then, receiving a WM_CHAR message from PM in eFTE2 code triggers one ConvertKey function that uses several of the fsflags above, but does not even reference KC_DEADKEY or KC_COMPOSITE.

I think the above would seem to reinforce my initial impression on how E / AE / EPM /eFTE2 deal with composite characters, and that all I would need now is add some code to ConvertKey: check for the KC_DEADKEY flag in received WM_CHAR messages and drop those instead of reflecting the characters in output.

I suspected this meant 'case closed' and Keith agreed, so now I just have to sit down and lay a fix ;p

[Alfredo Fernández Díaz]

OK, I hope I didn't lay an egg with this one...

If all goes well, easiest fix in history, although I must confess I was pretty lucky at guessing beforehand ;)

I have been trying this at home and it seems to work with only a very minor side effect. It's a shame the fix being simplicity itself necessarily introduces a new bug.