Solved

ASP website showing weird characters

Posted on 2010-09-20
11
3,056 Views
Last Modified: 2012-05-10
Dear Experts,
we have a ASP portal running under IIS6. The databases behind runs on SQL Server. We've had the problem both on SQL Server 2008 and SQL Server 2000.
All of a sudden the portal started returning weird characters in place of French accented letters. For Example: "Jérôme" is now shown as "Jérôme".
The problem can be easily fixed by adding this line at the top of the asp page:
<meta HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">

Open in new window


But we don't want to modify all pages in the website.
If I execute the query directly on SQL Server console the name is shown correctly.
We haven't changed the collation of the database.

Why is this happening all of a sudden?
How can I fix it?

Server specs:
OS: Windows Server 2003 SP2
Web server: IIS 6
Database Server: SQL Server 2000

The server is up to date.
IIS and SQL Server reside on the same server.

Thanks,
Roberto.
0
Comment
Question by:martineit
  • 6
  • 5
11 Comments
 
LVL 21

Expert Comment

by:Dale Burrell
Comment Utility
If you don't have the line <meta HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8"> then the browser has to guess what character set you are using, and it appears to be getting it wrong from time to time.

You have two options that I know of, put that line into every asp file or I think you can add that as an HTTP header in IIS which is basically doing the same thing.
0
 
LVL 21

Expert Comment

by:Dale Burrell
Comment Utility
0
 
LVL 1

Author Comment

by:martineit
Comment Utility
Hi,
thanks for your prompt reply. What you said makes a lot of sense but I tried adding a "Custom HTTP Header" at different levels of the website but the meta tag is not added to the page when I make the request.
Moreover in one of the links you sent me it says:
'go to "Properties" => "HTTP Headers" => "File Types..." => "New Type...".' which doesn't exist in IIS6.
I changed that value in "Properties" => "HTTP Headers" => "Custom HTTP Header" => "Add"
Custom Header Name: http-equiv="content-type"
Custom Header Value: content="text/html;charset=utf-8"

Even so, it doesn't show in the Source of the page (among the Meta tags) nor the problem is fixed.
Is the "Custom HTTP Header" disabled by default?
If you try it yourself, do you see this value then reported in the source code of the page?

Thanks,
Roberto.
0
 
LVL 21

Expert Comment

by:Dale Burrell
Comment Utility
HTTP headers are not the same as meta tags, and do not show in the source, but they should still have the desired effect. You can inspect the headers with something like Firebug for FIrefox or Fiddler.

Unfortunately I don't know anything more about adding the HTTP headers - all I know is that it can be done. I've always added the meta tag as thats the recommended way to do it.

You don't already have a meta tag specifying a different charset do you?

Hmmm... can't think of anything else of the top of my head
0
 
LVL 1

Author Comment

by:martineit
Comment Utility
Yep, I tried with Fiddler and this is the header I get:
====================
HTTP/1.1 200 OK
Cache-Control: private
Connection: close
Date: Tue, 21 Sep 2010 08:17:55 GMT
Content-Type: text/html
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET1
Content-Encoding: gzip
Vary: Accept-Encoding
Transfer-Encoding: chunked
====================
I tried changing other parameters (X-Powered-By, I added 1 at the end) and it works, unfortunately although I set a Custom HTTP Header with: "Content-Type: text/html;charset=UTF-8" it still doesn't appear in the Header.

Any more ideas?
I checked on the internet and I can clearly see somebody managed to modify that part of the header but nobody says how...

Thanks,
Roberto.
0
Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

 
LVL 1

Author Comment

by:martineit
Comment Utility
One more comment on the subject:
Changing the HTTP Header works for any other file than ASP (and I guess ASP.NET too). Gif, css, html files all return a header like:
==========================
HTTP/1.1 304 Not Modified
Content-Type: text/html;charset=UTF-8
Last-Modified: Thu, 21 Sep 2006 15:51:59 GMT
Accept-Ranges: bytes
ETag: "84f04cdf95ddc61:1381"
Server: Microsoft-IIS/6.0
MicrosoftOfficeWebServer: 5.0_Pub
X-Powered-By: ASP.NET
Date: Tue, 21 Sep 2010 09:33:42 GMT
==========================
ASP default Content-Type cannot be changed from its default "text/html" (or so it seems).
Any clue or other solutions?

Thanks,
Roberto.
0
 
LVL 21

Expert Comment

by:Dale Burrell
Comment Utility
I suspect that the the content-type is being set in the ASP code, which in turn produces an HTTP header, if your ASP pages include something like footer.asp then that would be an excellent place to set the header in ASP without having to modify every file.
0
 
LVL 1

Author Comment

by:martineit
Comment Utility
Well, for the moment I wrote a Response.charset="utf-8" in the global.asa which seems to do the trick although I think it's quite dirty.
I still haven't figured out why making an explicit Response.charset="iso-8859-1" would still mess up French special characters while "UTF-8" doesn't.

Let's see if anybody knows something about this, otherwise I'll close the case and add some dirt to the global.asa.

Thanks,
Roberto.
0
 
LVL 21

Expert Comment

by:Dale Burrell
Comment Utility
I imagine because its stored in your database using an nvarchar datatype which is a unicode format i.e. 2 bytes per character. So when you then tell the browser to display that using a single byte charset it shows all the interesting characters as two characters instead of one. It shows the regular chars as normal because they use the same single byte code, and the upper byte has nothing in it.

Its a pretty complex mechanism, there is how the chars are stored in the database, there is how the chars are handled by the code in ASP, then there is how the chars are interpreted by the browser. If any of those 3 stages doesn't match you get strange results as you are experiencing. The only reason I have any kind of understanding here is because we deliberately used the wrong charset on a system I worked on and tricked it to display it differently... not that you want to know that :)

The following might help if you are interested...

http://www.example-code.com/asp/asp-codepages.asp
http://en.wikipedia.org/wiki/Latin_characters_in_Unicode
http://msdn.microsoft.com/en-us/library/ms186939.aspx
0
 
LVL 1

Author Comment

by:martineit
Comment Utility
Hey Dale,
that's a very interesting feedback. I'm still investigating this problem. I found that using a predefined include we use to connect to the database I get that problem while if I make a directo connection to the DB the characters are shown correctly.
More updates will follow.
By the way, I probably got it wrong but I would like to point out for other people who might read this post that changing the "Custom HTTP Header" at website level will mess up your website. All extensions will be presented with the same "content value" which means, for example, that zip files will be interpreted as html (instead of opening a download box). This problem only occurs with browsers other than IE.
0
 
LVL 1

Accepted Solution

by:
martineit earned 0 total points
Comment Utility
Hello,
I finally found out what was the problem:
I created a simple file with a simple query to a database and it displayed the records correctly.
Then I started adding every include and at some stage I started getting the weird characters again.
I removed all includes and left the one causing the problem and I still had the problem.
I removed everything from within the file and I still had the problem.
I opened the file with a HEX editor (I used HHD Hex Editor Neo) and I could see the file was prefixed by a "ef bb bf" whose meaning you can find here: http://en.wikipedia.org/wiki/Byte_order_mark.
Once I removed it the special characters where displayed correctly again.
To remove that Hex string I had to open the file with notepad and copy the content of the file (not the file itself) in another notepad and then substitute the file.
If you want to search all files containing that Hex string you can use a Hex File Search (I used Effective File Search).

I hope this saves some time to someone else.

Thanks Dale for your inputs,
Roberto.
0

Featured Post

Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

Join & Write a Comment

In this article we will get to know that how can we recover deleted data if it happens accidently. We really can recover deleted rows if we know the time when data is deleted by using the transaction log.
Ever needed a SQL 2008 Database replicated/mirrored/log shipped on another server but you can't take the downtime inflicted by initial snapshot or disconnect while T-logs are restored or mirror applied? You can use SQL Server Initialize from Backup…
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…
This video explains how to create simple products associated to Magento configurable product and offers fast way of their generation with Store Manager for Magento tool.

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now