1 Topic by stevetroxell

  • stevetroxell
  • Guest

Topic: Not seeing Unicode in the console viewer

I must missing something simple here, but when I do this:

    SiMain.LogWideString('Unicode', #27442#12384#12369#12399);

All I see in the console viewer is four little square boxes instead of the Unicode characters. When I push those characters out to other targets (text files, SQL queries) I get the actual Unicode characters in other tools which display Unicode data.

I'm just beginning to get a grip on Unicode so I may be making a silly mistake that I can't see.

2 Reply by dgurock

  • dgurock
  • Administrator
  • Offline
  • Registered: 2009-06-23
  • Posts: 639

Re: Not seeing Unicode in the console viewer

Hi Steve,

I just tested this on Vista, and the characters are correctly displayed in the Console:

http://www.gurock.com/downloads/files/support/unicode.png

On a Windows XP machine (without Office), however, I have the same issue you described. I actually think that it's a font issue, as I have a lot more fonts on the Vista machine than on the XP one.

I got a few questions:

- Which other tools can display the characters? I tested it with Notepad, a random .NET based app and Eclipse, and all have the same problem on the XP machine.

- Did you use two different machines to test the issue, or did you test other applications on the same machine on which the Console displayed the square boxes?

- Which OS do you use? Do you have Office installed (if so, which version)?

Background: Windows uses alternative fonts for individual characters if the active font doesn't have the required Unicode characters when a text is displayed. On my Vista machine, a Microsoft font called MingLiU contains the characters you mentioned. On my XP machine, no font contains these characters, so XP cannot display them.

For example, this should work fine under XP, as XP has a font for the following characters:

SiMain.LogWideString('Unicode', #$3B1#$3B2#$3B3#$3B4#$3B5#$3B6);

I believe the fonts you need can be installed via Office or Windows language packs, but I cannot test this right now. It would be great if you could answer the above questions, as this should help us to figure out the issue.

Dennis Gurock,
Director & Co-Founder
Gurock Software

3 Reply by stevetroxell

  • stevetroxell
  • Guest

Re: Not seeing Unicode in the console viewer

All my original observations were from a single Win XP SP2 machine. I have Office 2007 installed.

I tried it on Vista (without Office) and like you I do see the Unicode data in the SmartInspect console viewer there. I tried your alternative Unicode data and I see Unicode data in the SmartInspect console on the XP machine.

The other tools I use on the XP machine which show me the original Unicode data are SQL Server 2005 Management Studio (the Unicode data was inserted into a database table) and Stylus Studio (the Unicode data was written to a UTF-8 encoded XML file). But when I open the same XML file in Notepad, I get the small square boxes replacing the Unicode data...for my original Unicode data anyway; I didn't try your substitute Unicode data.

I guess I can concede that it's a font display issue for XP. I just don't understand why SQL Management Studio and Stylus Studio on the same machine can display the data properly.

4 Reply by stevetroxell

  • stevetroxell
  • Guest

Re: Not seeing Unicode in the console viewer

Also, the Unicode data I used for testing was lifted from spam email. IE 7 and Outlook 2007 both display the Unicode data correctly.

5 Reply by dgurock

  • dgurock
  • Administrator
  • Offline
  • Registered: 2009-06-23
  • Posts: 639

Re: Not seeing Unicode in the console viewer

Hello,

we tried a few things and researched the topic and the problem lies most likely in the Windows font substitution. It looks like Windows has some problems with Korean characters and doesn't substitute these characters correctly (i.e., it doesn't use other fonts for specific characters).

Font substitution gets better and better in each Windows version and Vista is already getting most of it right (that's why the example you posted works on Vista). For more technical details, please see this:

Google Groups Discussion

I assume that some programs get this working by not using the standard Windows GDI API to draw text, but by using the newer Uniscribe APIs (I'm quite sure that Office and IE is using this, and probably Visual Studio and SQL Management Studio, too).

I'm afraid that we cannot do a lot to fix this situation, as we are relying on Windows APIs to draw texts (i.e., all Windows standard controls have this problem as it looks like). The good thing is that most character sets do work and that newer Windows versions don't have this problem at all.

The only workaround that might work at the moment is selecting a mono-spaced font with Korean characters in the Console, and using mono-spaced fonts for the views (you can configure this in the Console Options).

Dennis Gurock,
Director & Co-Founder
Gurock Software

6 Reply by stevetroxell

  • stevetroxell
  • Guest

Re: Not seeing Unicode in the console viewer

Thanks. I can work around this and now I know more about what to expect. Basically if it doesn't show up in Notepad, I probably shouldn't expect it it show up in SI console. If it ever becomes a critical issue, I can choose Unicode test data which I know will display correctly or use a Vista machine.