ASP website showing weird characters

Dear Experts,
we have a ASP portal running under IIS6. The databases behind runs on SQL Server. We've had the problem both on SQL Server 2008 and SQL Server 2000.
All of a sudden the portal started returning weird characters in place of French accented letters. For Example: "Jérôme" is now shown as "Jérôme".
The problem can be easily fixed by adding this line at the top of the asp page:
<meta HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">

Open in new window

But we don't want to modify all pages in the website.
If I execute the query directly on SQL Server console the name is shown correctly.
We haven't changed the collation of the database.

Why is this happening all of a sudden?
How can I fix it?

Server specs:
OS: Windows Server 2003 SP2
Web server: IIS 6
Database Server: SQL Server 2000

The server is up to date.
IIS and SQL Server reside on the same server.

Who is Participating?
martineitConnect With a Mentor Author Commented:
I finally found out what was the problem:
I created a simple file with a simple query to a database and it displayed the records correctly.
Then I started adding every include and at some stage I started getting the weird characters again.
I removed all includes and left the one causing the problem and I still had the problem.
I removed everything from within the file and I still had the problem.
I opened the file with a HEX editor (I used HHD Hex Editor Neo) and I could see the file was prefixed by a "ef bb bf" whose meaning you can find here:
Once I removed it the special characters where displayed correctly again.
To remove that Hex string I had to open the file with notepad and copy the content of the file (not the file itself) in another notepad and then substitute the file.
If you want to search all files containing that Hex string you can use a Hex File Search (I used Effective File Search).

I hope this saves some time to someone else.

Thanks Dale for your inputs,
Dale BurrellDirectorCommented:
If you don't have the line <meta HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8"> then the browser has to guess what character set you are using, and it appears to be getting it wrong from time to time.

You have two options that I know of, put that line into every asp file or I think you can add that as an HTTP header in IIS which is basically doing the same thing.
Cloud Class® Course: CompTIA Healthcare IT Tech

This course will help prep you to earn the CompTIA Healthcare IT Technician certification showing that you have the knowledge and skills needed to succeed in installing, managing, and troubleshooting IT systems in medical and clinical settings.

martineitAuthor Commented:
thanks for your prompt reply. What you said makes a lot of sense but I tried adding a "Custom HTTP Header" at different levels of the website but the meta tag is not added to the page when I make the request.
Moreover in one of the links you sent me it says:
'go to "Properties" => "HTTP Headers" => "File Types..." => "New Type...".' which doesn't exist in IIS6.
I changed that value in "Properties" => "HTTP Headers" => "Custom HTTP Header" => "Add"
Custom Header Name: http-equiv="content-type"
Custom Header Value: content="text/html;charset=utf-8"

Even so, it doesn't show in the Source of the page (among the Meta tags) nor the problem is fixed.
Is the "Custom HTTP Header" disabled by default?
If you try it yourself, do you see this value then reported in the source code of the page?

Dale BurrellDirectorCommented:
HTTP headers are not the same as meta tags, and do not show in the source, but they should still have the desired effect. You can inspect the headers with something like Firebug for FIrefox or Fiddler.

Unfortunately I don't know anything more about adding the HTTP headers - all I know is that it can be done. I've always added the meta tag as thats the recommended way to do it.

You don't already have a meta tag specifying a different charset do you?

Hmmm... can't think of anything else of the top of my head
martineitAuthor Commented:
Yep, I tried with Fiddler and this is the header I get:
HTTP/1.1 200 OK
Cache-Control: private
Connection: close
Date: Tue, 21 Sep 2010 08:17:55 GMT
Content-Type: text/html
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET1
Content-Encoding: gzip
Vary: Accept-Encoding
Transfer-Encoding: chunked
I tried changing other parameters (X-Powered-By, I added 1 at the end) and it works, unfortunately although I set a Custom HTTP Header with: "Content-Type: text/html;charset=UTF-8" it still doesn't appear in the Header.

Any more ideas?
I checked on the internet and I can clearly see somebody managed to modify that part of the header but nobody says how...

martineitAuthor Commented:
One more comment on the subject:
Changing the HTTP Header works for any other file than ASP (and I guess ASP.NET too). Gif, css, html files all return a header like:
HTTP/1.1 304 Not Modified
Content-Type: text/html;charset=UTF-8
Last-Modified: Thu, 21 Sep 2006 15:51:59 GMT
Accept-Ranges: bytes
ETag: "84f04cdf95ddc61:1381"
Server: Microsoft-IIS/6.0
MicrosoftOfficeWebServer: 5.0_Pub
X-Powered-By: ASP.NET
Date: Tue, 21 Sep 2010 09:33:42 GMT
ASP default Content-Type cannot be changed from its default "text/html" (or so it seems).
Any clue or other solutions?

Dale BurrellDirectorCommented:
I suspect that the the content-type is being set in the ASP code, which in turn produces an HTTP header, if your ASP pages include something like footer.asp then that would be an excellent place to set the header in ASP without having to modify every file.
martineitAuthor Commented:
Well, for the moment I wrote a Response.charset="utf-8" in the global.asa which seems to do the trick although I think it's quite dirty.
I still haven't figured out why making an explicit Response.charset="iso-8859-1" would still mess up French special characters while "UTF-8" doesn't.

Let's see if anybody knows something about this, otherwise I'll close the case and add some dirt to the global.asa.

Dale BurrellDirectorCommented:
I imagine because its stored in your database using an nvarchar datatype which is a unicode format i.e. 2 bytes per character. So when you then tell the browser to display that using a single byte charset it shows all the interesting characters as two characters instead of one. It shows the regular chars as normal because they use the same single byte code, and the upper byte has nothing in it.

Its a pretty complex mechanism, there is how the chars are stored in the database, there is how the chars are handled by the code in ASP, then there is how the chars are interpreted by the browser. If any of those 3 stages doesn't match you get strange results as you are experiencing. The only reason I have any kind of understanding here is because we deliberately used the wrong charset on a system I worked on and tricked it to display it differently... not that you want to know that :)

The following might help if you are interested...
martineitAuthor Commented:
Hey Dale,
that's a very interesting feedback. I'm still investigating this problem. I found that using a predefined include we use to connect to the database I get that problem while if I make a directo connection to the DB the characters are shown correctly.
More updates will follow.
By the way, I probably got it wrong but I would like to point out for other people who might read this post that changing the "Custom HTTP Header" at website level will mess up your website. All extensions will be presented with the same "content value" which means, for example, that zip files will be interpreted as html (instead of opening a download box). This problem only occurs with browsers other than IE.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.