Opened 2 years ago

Closed 22 months ago

Last modified 22 months ago

#79 closed defect (fixed)

Dead keys produce diacritic character output (diacritic+combined characters, instead of combined only) in PM version

Reported by: mrwarper Owned by: mrwarper
Priority: major Milestone: Release_1.1
Component: eFTEPM Version: Release_1.0
Keywords: PM dead key input unwanted double character Cc: mrwarper

Description (last modified by mrwarper)

Background: in most languages, accented/combined/composite characters are typed in by pressing a sequence of two keys: a "dead" key for the diacritic, followed by a "normal" key for the base character, f.e. I should press ´ + a to get á.

This works as expected in VIO eFTE2, but in PM eFTE2 two characters are produced: "´á", "¨ö", etc. (I kind of understand this, because getting keyboard input is totally different in PM and VIO.)

FTEPM/2006 has the same problem.

I suspect PM actually produces two characters for these combinations, somehow marking the first one for replacement, and it is up to the application to handle it: both system E and Aaron Lawrence's AE show the diacritic character *after* the cursor when a dead key is pressed, and when a normal key is pressed, the accent character is replaced and the cursor position is moved after the combined character as normal.

Example: I press "dead" "^" and I see "^", then I press "A", and "^" is replaced with "Â" and cursor is moved.

EPM does not ever show the accent character nor moves the cursor, it only shows the final combined character after the normal key is pressed.

Attachments (7)

e_key1.png (1.6 KB) - added by mrwarper 2 years ago.
E/AE output after first key, "dead" ´, is pressed.
e_key2.png (1.7 KB) - added by mrwarper 2 years ago.
E/AE output after second key, normal "a", is pressed.
epm_key2.png (4.1 KB) - added by mrwarper 2 years ago.
EPM only diplays the combined character "á" after the key sequence is completed.
epm_info.png (4.1 KB) - added by mrwarper 2 years ago.
EPM will tell you that in CP850, "á" has ASCII code 160
eftepm_key1.png (763 bytes) - added by mrwarper 2 years ago.
eFTEPM: first key ("dead" ' ), first output character ´. Cursor moves like we're done.
eftepm_key2.png (684 bytes) - added by mrwarper 2 years ago.
eFTEPM: second key (normal "a" ), second output character "á", "first character" is kept. WRONG.
eftepm_key2.2.png (684 bytes) - added by mrwarper 2 years ago.
eFTEPM: second key (normal "a" ), second output character "á", "first character" (unsolicited) is kept. WRONG.

Download all attachments as: .zip

Change History (19)

comment:1 Changed 2 years ago by mrwarper

  • Cc mrwarper added

comment:2 Changed 2 years ago by gyoung

This must be code page, language and/or country specific or I don't understand what you mean. This doesn't work in anything here.

comment:3 Changed 2 years ago by mrwarper

I take it you have never had to type a composite character like "á", then? You have to press a sequence of two keys to get it.

I am attaching some captures of PM editors output after key 1 and key 2 are pressed, in case that clarifies the problem:
-E/AE: weird thing displaying ´ and not moving the cursor, but all is well that ends well.
-EPM: output only after key 2; everything is correct.
-eFTEPM: WRONG.

Changed 2 years ago by mrwarper

E/AE output after first key, "dead" ´, is pressed.

Changed 2 years ago by mrwarper

E/AE output after second key, normal "a", is pressed.

Changed 2 years ago by mrwarper

EPM only diplays the combined character "á" after the key sequence is completed.

Changed 2 years ago by mrwarper

EPM will tell you that in CP850, "á" has ASCII code 160

Changed 2 years ago by mrwarper

eFTEPM: first key ("dead" ' ), first output character ´. Cursor moves like we're done.

Changed 2 years ago by mrwarper

eFTEPM: second key (normal "a" ), second output character "á", "first character" is kept. WRONG.

Changed 2 years ago by mrwarper

eFTEPM: second key (normal "a" ), second output character "á", "first character" (unsolicited) is kept. WRONG.

comment:4 Changed 2 years ago by mrwarper

I sent the last capture twice because of connection hiccups. Please pay no heed.

comment:5 Changed 2 years ago by mrwarper

  • Description modified (diff)
  • Summary changed from Dead keys input produces additional, unwanted characters in PM version to Dead keys produce diacritic character output (diacritic+combined characters, instead of combined only) in PM version

Tried to make the description and summary clearer.

comment:6 Changed 2 years ago by lewisr

Alfredo, could you please post your CONFIG.SYS settings for codepage, locale, and anything else which might bear on this?

Another thought I had was that maybe we're grabbing one of those keys for something else (but not really using it), thus the modifier is not being caught.

comment:7 follow-up: Changed 2 years ago by mrwarper

Well, the modifier is being caught by FTEpm, since a character (the modifier character) is produced in it. You mean something might interfere with the character being interpreted as a modifier so it's not dropped before combining with the regular one?

Anyway, Alex Taylor and I found some strange restrictions in "codepage" and "country" combinations while trying stuff for other languages, so my current setup is a bit offbeat:

COUNTRY=001,C:\OS2\SYSTEM\COUNTRY.SYS (instead of 034, less CP restrictions)
CODEPAGE=850,866 (this is for Russian)
DEVINFO=KBD,SP,C:\OS2\KEYBOARD.DCP (my own one, working perfectly in VIO, unused by PM)
SET LANG=es_ES_EURO

but just in case, I fired up my dad's old eCS 1.2mr installation to make sure and (of course) I see the same problem in ftepm with

COUNTRY=034,C:\OS2\SYSTEM\COUNTRY.SYS
CODEPAGE=850,437
DEVINFO=KBD,SP172,C:\OS2\KEYBOARD.DCP (original)
SET LANG=es_ES_EURO

If I had commit access I would check PM theories with Keith Merrington -- he knows both PM and dead keys internals -- I was in his presentation on the subject back in ca. 2005.

comment:8 in reply to: ↑ 7 ; follow-up: Changed 23 months ago by gyoung

Replying to mrwarper:

Well, the modifier is being caught by FTEpm, since a character (the modifier character) is produced in it.

I thought that this didn't work in the PM version. Please clarify

Anyway, Alex Taylor and I found some strange restrictions in "codepage" and "country" combinations while trying stuff for other languages, so my current setup is a bit offbeat:

COUNTRY=001,C:\OS2\SYSTEM\COUNTRY.SYS (instead of 034, less CP restrictions)
CODEPAGE=850,866 (this is for Russian)
DEVINFO=KBD,SP,C:\OS2\KEYBOARD.DCP (my own one, working perfectly in VIO, unused by PM)

Is this available generally? If not can you send it to me? Do the dead keys work in ae.exe with this setup?

SET LANG=es_ES_EURO

but just in case, I fired up my dad's old eCS 1.2mr installation to make sure and (of course) I see the same problem in ftepm with

COUNTRY=034,C:\OS2\SYSTEM\COUNTRY.SYS
CODEPAGE=850,437
DEVINFO=KBD,SP172,C:\OS2\KEYBOARD.DCP (original)

Do the dead keys work in ae.exe with this setup?

SET LANG=es_ES_EURO

If I had commit access I would check PM theories with Keith Merrington

Commit access to what?

-- he knows both PM and dead keys internals -- I was in his presentation on the subject back in ca. 2005.

Is there a copy of the presentation somewhere?

Last edited 23 months ago by gyoung (previous) (diff)

comment:9 in reply to: ↑ 8 Changed 23 months ago by mrwarper

Replying to gyoung:

I thought that this didn't work in the PM version. Please clarify

OK, let's try again: it doesn't work in eFTE-PM. Two keystrokes are always required in input (diacritic or combining character, then base character) to produce a combined character, but in eFTE-PM two characters are produced as output (accented space followed by accented character) instead of the single one that comes out --right as expected-- in all other PM applications. Is that not clear? Now, obviously some code in eFTEPM must be detecting the first key, because a matching character is part of the output. My hypothesis after Lewis' comment is, a failure to hand the first character over to eFTE code as a diacritic, presenting it as 'regular' -- but I have no actual idea about how eFTE code interacts with PM, or how PM handles combining characters, so I am not to be taken very seriously there yet.

DEVINFO=KBD,SP,C:\OS2\KEYBOARD.DCP (my own one, working perfectly in VIO, unused by PM)

Is this available generally? If not can you send it to me? Do the dead keys work in ae.exe with this setup?

Dead keys work with this setup in ALL PM applications except eFTEPM. DCP should be unrelated since the mechanism is not used by PM, but never mind. My DCP tools have never been released -- I'm sending you a copy by email.

[...]

DEVINFO=KBD,SP172,C:\OS2\KEYBOARD.DCP (original)

Do the dead keys work in ae.exe with this setup?

Yes, they do.

[...]

If I had commit access I would check PM theories with Keith Merrington

Commit access to what?

If I had SVN commit access to the eFTE repository (i.e. the possibility to release code on my own at some point) I would start experimenting with the code at home, instead of sitting around here waiting. Did you hear back from Adrian?

-- he knows both PM and dead keys internals -- I was in his presentation on the subject back in ca. 2005.

Is there a copy of the presentation somewhere?

I can't find it online now, but you can ask him. Depending on how much you know about keyboard input and non-English characters you might want to have a look at his article at

http://www.os2ezine.com/20041116/page_3.html

first (it is user level, and does not gloss over PM keyboard handling).

It has some encoding problems, so you'll need to view it using CP1252 to display correctly the "Ž" in the title, and CP850 for the accented characters in the article body.

comment:10 Changed 22 months ago by mrwarper

Summarizing my mail exchange with Keith:

[...] This is how I think the PM editors I tried work: (e)FTE(2) does not use a standard MLE control to display text (I haven't had a close look at the PM sources yet, I judge from the looks of it), so PM would pass characters along to FTE code which, in turn, simply does not know a thing about combining characters and ignores any such flags, or maybe its code is actually buggy. Other PM editors such as E / AE would use standard MLE PM controls and have all the combining of diacritics done for them by the PM, and EPM does combining on its own -- which would be why input behaves differently from E/AE in its main dialogue (which also looks different from the MLE controls I know) but exactly like E / AE everywhere else, where controls sure look standard.
[...] If I am right [...]

I got a kind reply from him with this pithy comment:

Normaal in an editor I woud not expert the use of an MLE since this only provides a limited editing capability. In PM programming the keyboard is handled with a WM_CHAR message. This provides a complete picture of all the key actions including dead keys.

So maybe there was really no more to it. I found some time during my last short holidays to further check this, and in PM3.inf I read


WM_CHAR - Syntax

This message is sent when an operator presses a key.

param1

USHORT fsflags /* Keyboard control codes. */

[...]


WM_CHAR field - fsflags

fsflags (USHORT)
Keyboard control codes.

[...]

KC_DEADKEY
The character code is a dead key. The application is responsible for displaying the glyph for the dead key without advancing the cursor.

KC_COMPOSITE
The character code is formed by combining the current key with the previous dead key.

[...]


Then, receiving a WM_CHAR message from PM in eFTE2 code triggers one ConvertKey function that uses several of the fsflags above, but does not even reference KC_DEADKEY or KC_COMPOSITE.

I think the above would seem to reinforce my initial impression on how E / AE / EPM /eFTE2 deal with composite characters, and that all I would need now is add some code to ConvertKey: check for the KC_DEADKEY flag in received WM_CHAR messages and drop those instead of reflecting the characters in output.

I suspected this meant 'case closed' and Keith agreed, so now I just have to sit down and lay a fix ;p

comment:11 Changed 22 months ago by mrwarper

  • Owner set to mrwarper
  • Status changed from new to accepted

comment:12 Changed 22 months ago by mrwarper

  • Resolution set to fixed
  • Status changed from accepted to closed

OK, I hope I didn't lay an egg with this one...

If all goes well, easiest fix in history, although I must confess I was pretty lucky at guessing beforehand ;)

I have been trying this at home and it seems to work with only a very minor side effect. It's a shame the fix being simplicity itself necessarily introduces a new bug, but if we have lived with this for the last 11+ years :(does anybody actually use this program?): then we might as well live with that for a while.

Last edited 22 months ago by mrwarper (previous) (diff)
Note: See TracTickets for help on using tickets.