Solved

ASP website showing weird characters

Posted on 2010-09-20
11
3,162 Views
Last Modified: 2012-05-10
Dear Experts,
we have a ASP portal running under IIS6. The databases behind runs on SQL Server. We've had the problem both on SQL Server 2008 and SQL Server 2000.
All of a sudden the portal started returning weird characters in place of French accented letters. For Example: "Jérôme" is now shown as "Jérôme".
The problem can be easily fixed by adding this line at the top of the asp page:
<meta HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">

Open in new window


But we don't want to modify all pages in the website.
If I execute the query directly on SQL Server console the name is shown correctly.
We haven't changed the collation of the database.

Why is this happening all of a sudden?
How can I fix it?

Server specs:
OS: Windows Server 2003 SP2
Web server: IIS 6
Database Server: SQL Server 2000

The server is up to date.
IIS and SQL Server reside on the same server.

Thanks,
Roberto.
0
Comment
Question by:martineit
  • 6
  • 5
11 Comments
 
LVL 21

Expert Comment

by:Dale Burrell
ID: 33714701
If you don't have the line <meta HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8"> then the browser has to guess what character set you are using, and it appears to be getting it wrong from time to time.

You have two options that I know of, put that line into every asp file or I think you can add that as an HTTP header in IIS which is basically doing the same thing.
0
 
LVL 21

Expert Comment

by:Dale Burrell
ID: 33714709
0
 
LVL 1

Author Comment

by:martineit
ID: 33715789
Hi,
thanks for your prompt reply. What you said makes a lot of sense but I tried adding a "Custom HTTP Header" at different levels of the website but the meta tag is not added to the page when I make the request.
Moreover in one of the links you sent me it says:
'go to "Properties" => "HTTP Headers" => "File Types..." => "New Type...".' which doesn't exist in IIS6.
I changed that value in "Properties" => "HTTP Headers" => "Custom HTTP Header" => "Add"
Custom Header Name: http-equiv="content-type"
Custom Header Value: content="text/html;charset=utf-8"

Even so, it doesn't show in the Source of the page (among the Meta tags) nor the problem is fixed.
Is the "Custom HTTP Header" disabled by default?
If you try it yourself, do you see this value then reported in the source code of the page?

Thanks,
Roberto.
0
Creating Instructional Tutorials  

For Any Use & On Any Platform

Contextual Guidance at the moment of need helps your employees/users adopt software o& achieve even the most complex tasks instantly. Boost knowledge retention, software adoption & employee engagement with easy solution.

 
LVL 21

Expert Comment

by:Dale Burrell
ID: 33722538
HTTP headers are not the same as meta tags, and do not show in the source, but they should still have the desired effect. You can inspect the headers with something like Firebug for FIrefox or Fiddler.

Unfortunately I don't know anything more about adding the HTTP headers - all I know is that it can be done. I've always added the meta tag as thats the recommended way to do it.

You don't already have a meta tag specifying a different charset do you?

Hmmm... can't think of anything else of the top of my head
0
 
LVL 1

Author Comment

by:martineit
ID: 33723259
Yep, I tried with Fiddler and this is the header I get:
====================
HTTP/1.1 200 OK
Cache-Control: private
Connection: close
Date: Tue, 21 Sep 2010 08:17:55 GMT
Content-Type: text/html
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET1
Content-Encoding: gzip
Vary: Accept-Encoding
Transfer-Encoding: chunked
====================
I tried changing other parameters (X-Powered-By, I added 1 at the end) and it works, unfortunately although I set a Custom HTTP Header with: "Content-Type: text/html;charset=UTF-8" it still doesn't appear in the Header.

Any more ideas?
I checked on the internet and I can clearly see somebody managed to modify that part of the header but nobody says how...

Thanks,
Roberto.
0
 
LVL 1

Author Comment

by:martineit
ID: 33723590
One more comment on the subject:
Changing the HTTP Header works for any other file than ASP (and I guess ASP.NET too). Gif, css, html files all return a header like:
==========================
HTTP/1.1 304 Not Modified
Content-Type: text/html;charset=UTF-8
Last-Modified: Thu, 21 Sep 2006 15:51:59 GMT
Accept-Ranges: bytes
ETag: "84f04cdf95ddc61:1381"
Server: Microsoft-IIS/6.0
MicrosoftOfficeWebServer: 5.0_Pub
X-Powered-By: ASP.NET
Date: Tue, 21 Sep 2010 09:33:42 GMT
==========================
ASP default Content-Type cannot be changed from its default "text/html" (or so it seems).
Any clue or other solutions?

Thanks,
Roberto.
0
 
LVL 21

Expert Comment

by:Dale Burrell
ID: 33723841
I suspect that the the content-type is being set in the ASP code, which in turn produces an HTTP header, if your ASP pages include something like footer.asp then that would be an excellent place to set the header in ASP without having to modify every file.
0
 
LVL 1

Author Comment

by:martineit
ID: 33725195
Well, for the moment I wrote a Response.charset="utf-8" in the global.asa which seems to do the trick although I think it's quite dirty.
I still haven't figured out why making an explicit Response.charset="iso-8859-1" would still mess up French special characters while "UTF-8" doesn't.

Let's see if anybody knows something about this, otherwise I'll close the case and add some dirt to the global.asa.

Thanks,
Roberto.
0
 
LVL 21

Expert Comment

by:Dale Burrell
ID: 33731818
I imagine because its stored in your database using an nvarchar datatype which is a unicode format i.e. 2 bytes per character. So when you then tell the browser to display that using a single byte charset it shows all the interesting characters as two characters instead of one. It shows the regular chars as normal because they use the same single byte code, and the upper byte has nothing in it.

Its a pretty complex mechanism, there is how the chars are stored in the database, there is how the chars are handled by the code in ASP, then there is how the chars are interpreted by the browser. If any of those 3 stages doesn't match you get strange results as you are experiencing. The only reason I have any kind of understanding here is because we deliberately used the wrong charset on a system I worked on and tricked it to display it differently... not that you want to know that :)

The following might help if you are interested...

http://www.example-code.com/asp/asp-codepages.asp
http://en.wikipedia.org/wiki/Latin_characters_in_Unicode
http://msdn.microsoft.com/en-us/library/ms186939.aspx
0
 
LVL 1

Author Comment

by:martineit
ID: 33767864
Hey Dale,
that's a very interesting feedback. I'm still investigating this problem. I found that using a predefined include we use to connect to the database I get that problem while if I make a directo connection to the DB the characters are shown correctly.
More updates will follow.
By the way, I probably got it wrong but I would like to point out for other people who might read this post that changing the "Custom HTTP Header" at website level will mess up your website. All extensions will be presented with the same "content value" which means, for example, that zip files will be interpreted as html (instead of opening a download box). This problem only occurs with browsers other than IE.
0
 
LVL 1

Accepted Solution

by:
martineit earned 0 total points
ID: 33769924
Hello,
I finally found out what was the problem:
I created a simple file with a simple query to a database and it displayed the records correctly.
Then I started adding every include and at some stage I started getting the weird characters again.
I removed all includes and left the one causing the problem and I still had the problem.
I removed everything from within the file and I still had the problem.
I opened the file with a HEX editor (I used HHD Hex Editor Neo) and I could see the file was prefixed by a "ef bb bf" whose meaning you can find here: http://en.wikipedia.org/wiki/Byte_order_mark.
Once I removed it the special characters where displayed correctly again.
To remove that Hex string I had to open the file with notepad and copy the content of the file (not the file itself) in another notepad and then substitute the file.
If you want to search all files containing that Hex string you can use a Hex File Search (I used Effective File Search).

I hope this saves some time to someone else.

Thanks Dale for your inputs,
Roberto.
0

Featured Post

Salesforce Has Never Been Easier

Improve and reinforce salesforce training & adoption using WalkMe's digital adoption platform. Start saving on costly employee training by creating fast intuitive Walk-Thrus for Salesforce. Claim your Free Account Now

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

In this article I will describe the Copy Database Wizard method as one possible migration process and I will add the extra tasks needed for an upgrade when and where is applied so it will cover all.
Preparing an email is something we should all take special care with – especially when the email is for somebody you may not know very well. The pressures of everyday working life stacked with a hectic office environment can make this a real challen…
Exchange organizations may use the Journaling Agent of the Transport Service to archive messages going through Exchange. However, if the Transport Service is integrated with some email content management application (such as an antispam), the admini…

726 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question