Solved

What Could Be Changing the Charset from utf-8 to iso-8859-1 ?

Posted on 2014-09-03
12
175 Views
Last Modified: 2014-09-11
We have a production server where the output in the TCP-IP headers is utf-8 for this URL (checked with URL Valet)

http://www.himalayanacademy.com/view/the-history-of-hindu-india_part-1_ta

But this same URL from our development server

http://dev.himalayanacademy.com/view/the-history-of-hindu-india_part-1_ta

is delivered with this in the header, and the Unicode script is not displayed correctly.

Content-Type: text/html; charset="iso-8859-1"

Now, we have  checked our frame work "RevIgniter" and other pages on the dev site are  delivered as expected with  

Content-Type: text/html; charset="utf-8"

e.g. http://dev.himalayanacademy.com/readlearn/basics

is delivered with headers: charset="utf-8"   from the same domain, using the same httpd.conf directives and the same  dynamic page assembly/delivery framework.

So... what on earth could be causing

http://dev.himalayanacademy.com/view/the-history-of-hindu-india_part-1_ta

to switch to:

Content-Type: text/html; charset="iso-8859-1"

when the production server which is a mirror of the dev site ships the same page with

Content-Type: text/html; charset="utf-8"

??

This one is a real "gremlin" (hard to debug)
0
Comment
Question by:Brahmanatha
  • 6
  • 4
  • 2
12 Comments
 
LVL 82

Expert Comment

by:Dave Baldwin
ID: 40302648
It is definitely coming from the server in the Response header.  Look in the directory that that file comes from and see if there is an .htaccess file there.  The Content-Type and Charset could be being set there.
0
 
LVL 58

Expert Comment

by:Gary
ID: 40302654
Agree with Dave, the server is sending  a page charset for the dev server
Content-Type: text/html; charset=iso-8859-1

But for the other site - which is what you would expect
Content-Type: text/html; charset=utf-8

It maybe as simple as adding to your httpd.conf
AddDefaultCharset utf-8
0
 

Author Comment

by:Brahmanatha
ID: 40302661
But this same URL from the same domain/server/public_html folder

http://dev.himalayanacademy.com/readlearn/basics

has charset=utf-8 in the response header.

I'm running a deep grep... see what pops out, this will take a while...

[root@sat ~]# cd /home/devhap/public_html
[root@sat public_html]# grep -R "charset=iso-8859-1"
0
 
LVL 58

Expert Comment

by:Gary
ID: 40302666
Whats your httpd.conf setup? Something is sending the iso-8859-1 and it's not the php page
0
 

Author Comment

by:Brahmanatha
ID: 40304448
Directives for this domain are:

SuexecUserGroup "#501" "#501"
ServerName dev.himalayanacademy.com
ServerAlias www.dev.himalayanacademy.com
ServerAlias webmail.dev.himalayanacademy.com
ServerAlias admin.dev.himalayanacademy.com
DocumentRoot /home/devhap/public_html
ErrorLog /var/log/virtualmin/dev.himalayanacademy.com_error_log
CustomLog /var/log/virtualmin/dev.himalayanacademy.com_access_log combined
ScriptAlias /cgi-bin/ /home/devhap/cgi-bin/
DirectoryIndex index.irev index.shtml index.html index.htm index.php index.php4 index.php5
<Directory /home/devhap/public_html>
Options -Indexes +Includes SymLinksifOwnerMatch +ExecCGI
allow from all
0
 

Author Comment

by:Brahmanatha
ID: 40305054
My deep grep did not turn up anything out of the ordinary that could be trigger the header switch to  iso-8859-1

But one of my team notes:

http://dev.himalayanacademy.com/looklisten/videos  
# view source for the above... see the blank line at the beginning

vs.

http://www.himalayanacademy.com/looklisten/videos  

source has no blank line at the beginning.

If one of our controllers on the dev site is throwing a CRLF into standout before sending out the page...would this cause the header to default to:

Content-Type: text/html; charset=utf-8

??

It's going to take me a bit of research to discover where this blank line is coming from and exactly what is being pushed to standout before the page goes out... mysterious!
0
Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

 
LVL 82

Expert Comment

by:Dave Baldwin
ID: 40305064
The fact that your response headers are including the 'charset' tells me that the server is doing this.  If it was being set in one of your HTML files, it would not show up in the response header.  This is a problem I've seen before and it is always something on the server that is doing it.  If it wasn't being done by the server, you would see just "Content-Type: text/html" without any charset.

I just checked 5 of my websites on different servers and None of them are sending a 'charset' in the response headers.  As far as I'm concerned, that's how it should be.
0
 

Author Comment

by:Brahmanatha
ID: 40305084
I use VirtualMin/WebMin for all our domains on the same box.. they are just virtual servers served from the same /home directory. There is a single httpd.conf file



/home/devhap/public_html  # where the problem is coming from
/home/himalayan/public_html  # theoretically a mirror of the above that does not have the problem

I tried adding

AddDefaultCharset utf-8

so the .htaccess file for devhap... but it doesn't help.

so when you say "the server" I'm not sure what else it could be.. the fact that there is a CRLF output on the top of only some pages at the dev site and not all pages, would suggest it might be a big in one of our controllers
0
 
LVL 82

Accepted Solution

by:
Dave Baldwin earned 500 total points
ID: 40305128
The server means Apache, the web server.  The CRLF output may indicate some other problem but it has nothing to do with this.  If Content-Type: text/html; charset="iso-8859-1" is not present in any of the Apache files, the only other place would be some server side programming file like PHP that has the ability to send headers out.  For response headers, the last one 'wins'.  So even if you put AddDefaultCharset utf-8 in .htaccess, it is possible for a PHP file (for example) to overwrite that.
0
 
LVL 82

Expert Comment

by:Dave Baldwin
ID: 40305133
For what it's worth, I put up a demo showing what I described at http://www.gammaboa.com/UTF8/ .
0
 

Author Comment

by:Brahmanatha
ID: 40318445
Well, yes, I can see that PHP "server side programming" -- in our case, LiveCode/RevIgniter -  is somehow generating Content-Type: text/html; charset="iso-8859-1"

Of  course this is not a solution... I have to do some back end lo-level analysis of the MCV framework we use (think CodeIgniter, but just uses a different language), but I know EE does not like just exiting the question as "no solution" so i will give Dave the points on this (EE's auto monitor is begging me to do something)

thanks for the attention give... wish me luck!
0
 

Author Closing Comment

by:Brahmanatha
ID: 40318449
C given only because there's no way for the "expert" to peer into our backend... as a result the "solution" was actually just pointing out that both Apache, httpd.conf and .htaccess are obviously not the source.
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

Do you come here a lot? Are you lazy like me and don't want to go through the "trouble" of having to click your Dock's Safari icon and then having to click your Experts Exchange Favorites bookmark to get here? Well then this article is for you.
I annotated my article on ransomware somewhat extensively, but I keep adding new references and wanted to put a link to the reference library.  Despite all the reference tools I have on hand, it was not easy to find a way to do this easily. I finall…
In this tutorial viewers will learn how to code links for mobile sites that, once clicked, send a call or text to a specified number. For a telephone link (once clicked, calls a number), begin with a normal "<a href=" link tag. For the href, specify…
Shows how to create a shortcut to site-search Experts Exchange using Google in the Chrome browser. This eliminates the need to type out site:experts-exchange.com whenever you want to search the site. Launch the Search Engine Menu: In chrome, via you…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now