Another facebox issue: international characters not displayed

Hello again experts

Thanks to sh0e's patience I got my previous issue sorted and I'm doing even more with facebox and everyone loves it. However I'm now going international and have some new files that need to be categorised in the database. The categorisation will all be in English but the file name and its associated label must remain in French.

I'm seeing an issue where all accented (diacritic) characters are displayed as ? or a box (depending on whether it's firefox or IE) when a facebox is opened. Now, I'm using ISO-8859-1 as my character set, as when I specify UTF-8 I get boxes everywhere there's an accent, not just in the facebox page. I'm wondering if my elderly version of ultraedit is to blame.

I'll post the code of the facebox page below, as I'm afraid I'm on a corporate intranet and can't really put demo a page fully online (yet). This page displays fine in its own browser window. Jquery is being properly loaded in the background window. I've removed the serverside code as I know it's working fine. The variables are encapsulated with $.

Also, I say it's a facebox problem, but a few moments ago I tried doing the same thing quickly using thickbox -- with the same results. So that's got me wondering where on earth I'm going wrong?

Hopefully someone can help. 500 points, as ever :-D

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://">
<html xmlns="" xml:lang="fr">
<meta http-equiv="content-type" content="text/
html;charset=ISO-8859-1" />
#filearticle {padding:0; margin:0}
#filearticle form {margin:0; padding: 0}
h1.articletitle {font-size: 10pt; color: #FFFFFF; background: #0077bb;
padding: 2px; margin: 0 }
#articlebody {padding: 5px}
#articlebody p {margin: 5px 0 5px 0}
#articlebody ul {margin: 5px 40px 5px 30px; list-style-type: square}
        width: 100%;
        padding: 0;
        margin: 0;
        text-align: left;
        clear: both;
        background: none;
dl dt{
        float: left;
        width: 25%;
        padding: .2em 0% 0.5em 0%;
        margin: 0 ;
        font-weight: bold;
        clear: left;
        font-size: 8pt;
dl dd{
        width: 70%;
        padding: .2em 0% 0.5em 0;
        float: left;
        margin: 0 ;
        font-size: 8pt;
input {border:1px solid #808080; background: #EEEEEE; font-size: 8pt;
font-family: Tahoma}
.quickalert1 {margin: 0;background: #0077bb; color: #FFFFFF; text-
align: center; font-size: 10pt; font-weight: normal; padding: 10px;
width: 60%;position: absolute; top: 35px ; left: 20%;border-top:1px
solid #99ccff; border-left:1px solid #99ccff;border-right:1px solid
#003366;border-bottom:1px solid #003366;}
#filelabelformsubmitted {display:none;}
<script type="text/javascript" src="js/jquery.form.min.js"></script>
<script type="text/javascript">
    $('#facebox #filelabelform').ajaxForm({
        target:       '#facebox #filelabelformsubmitted',
        beforeSubmit:  function(){
                       var filecategory1v = $
                       var filecategory2v = $
                       var filelanguagev  = $
                       var filepositionv  = $
                       if (!filecategory1v[0] || !filecategory2v[0]
|| !filelanguagev[0] || !filepositionv[0]) {
                       $("#filelabelformsubmitted").html('You must
select from <b>ALL</b> dropdowns.').fadeIn('slow').animate({opacity:
1.0}, 2000).fadeOut('slow');
                       return false;
                       return true;
        success:       function(){
#filelabelformsubmitted').fadeIn('slow').animate({opacity: 1.0},
                       $('#facebox').animate({opacity: 1.0},
<div id="filelabelformsubmitted" class="quickalert1"></div>
<div id="filearticle">
<h1 class="facebox_title">Categorise File</h1>
<div id="articlebody">
<form id="filelabelform" method="post" action="filelabelscript.dpe">
<!--NOTE : when filename is displayed below it has ? or square for
accented characters depending on browser.-->
<dt>File name:</dt><dd>$filename$</dd>
<dt>File entity:</dt><dd>$entity$</dd>
<input type="hidden" name="filename" value="$filename$">
<input type="hidden" name="entity" value="$entity$">
<dt>File label:</dt><dd><input type="text" name="filelabel" size="40"
value="$filelabel1$" id="filelabel"></dd>
<dt>File Category1:</dt><dd><select size="1" name="filecategory1"
                               <option value="">Select...</option>
                               <option value="$id$" selected>$category
<dt>File Category2:</dt><dd><select size="1" name="filecategory2"
                               <option value="">Select...</option>
                               <option value="$id$" selected>$category
<dt>File Language:</dt><dd><select size="1" name="filelanguage"
                               <option value="">Select...</option>
                               <option value="$id$" selected>$language
<dt>File Positioning:</dt><dd><select size="1" name="fileposition"
                               <option value="">Select...</option>
                               <option value="$id$" selected>$position
<dt></dt><dd><input type="submit" value="submit" id="btnSubmit"></dd>

Open in new window

Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Michel PlungjanIT ExpertCommented:
If the characters are saved as DOS iso-8859-1 then you need iso-8859-1
If they are saved as UTF8, you need UTF8
If you use entities (&egrave; &euml;) then it does not matter
Dave6969Author Commented:
Actually since no one was answering I got to grips with it a bit. I've done some replacing server side and I can get it to display fine.

HOWEVER, there is a form field in the page for the file label. It's going to be mostly english speakers categorising this as first. If you type accented characters into the field (as a french speaker would do) there's no problem submitting, but if you copy and paste (as an english speaker will do) from the displayed filename (with the HTML entities in it) then a load of gobbledegook gets entered into the database where the HTML entities used to be.

I tried writing some jquery code to replace the entities on-the-fly, but of course I hit the same problem with the characters not displaying in the page. Very frustrating!
It's already done, but usually the solution seems to be to force encoding on the server side.

It sounds like you are trying to sanitize entities on the client side.  I would recommend you sanitize the input on the server side (unescape entities on the server side after receiving them).  Your database will be happier.

Or, if that's really not an option or you'd rather just let the database take whatever it gets, there's some code here that will do what you are asking:
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

Dave6969Author Commented:
I want to keep my database as happy as possible :-) So when a é is entered into a form it gets stored as an é .

I'm not sure wht you mean by "unescaping" the html entities? I might be being a bit slow today.

I am actually performing replace operations server side to encode the html entities. The problem that I'm having is when I try to DEcode server side to create a value to put in the file label box it won't work.

I'm encoding the value="" in the input box into HTML entities, because if I don't then I get ?s all over the place. But the trouble is the data gets entered as HTML entities and I can't seem to decode it with the remote submission page.

I'm losing hair again...! lol

I meant decode instead of unescape.  Decode/encode/escape/unescape may likely fry my brain today.

I may need a clarification of how your code processes the data.
Client: è
Server: è
DB: è

Client: &egrave;
&egrave; -> è
&egrave; <- è
DB: è

Or is this all just a client-side problem where it's not handling the data it gets properly, and the server-side is already happy?

<!--//The link I provided has a JavaScript library where you can: //-->
<script src="" type="text/javascript"></script>
var encoded = '&egrave;';
var decoded = html_entity_decode(encoded);
decoded = 'è';
encoded = htmlentities(decoded);

Open in new window

Dave6969Author Commented:
No the problem is that the popup page refuses to show accented characters at all. é will jsut show up as ? or a box. So trying to change the html entity through javascript wouldn't work, because the é would be mangled when the page loaded. Unless that is it could be bound to a click event in Jquery, because all my Jquery code is in the "parent" page.

However, I have cobbled together a *kind of* solution. I submitted a form through facebox which contained all accented characters and then entered the output into the database. I now have a table of all the htmlentities and their ASCII equivalent along with the specific two character gobbledegook that's returned when I submit a form. I then perform a replace on these two characters to convert them back to the original ASCII before they get entered into the database.

It's rather long-winded and not at all elegant, but it does work.
Two character gobbly-gook, sounds like Unicode.
Is there a sample of this gobbly-gook?  I'd also like to see the tables that you got.
Dave6969Author Commented:
Not having seen unicode before that's that I called it, so I guess it's unicode. The scripting language I use can only deal with ASCII (which is fine for me 99% of the time) so I've never had to use it before.

It looks like my work is going open source... hehe. Attached you'll find an HTML table with the values I extracted. Like I say it's a bit cumbersome, but it does work. It seems that for some reason popup boxes created with Jquery can't cope with ISO language codes. I guess utf-8 is the fashionable one, but for various reasons I can't use it.

I'll continue to leave this question open for a little while in case someone has a more elegant idea.
Michel PlungjanIT ExpertCommented:
utf is unicode and what stops you from using it?
That is indeed UTF-8.  JavaScript and XHR use UTF-8 internally, so it's the default.

mplungjan is probably already getting at this; that you should use all Unicode if possible.  That would be the most "elegant" solution.

From what I can see, your database supports Unicode.  Ironically, it appears to be storing the string as UTF-8 parsed as ASCII into UTF-8.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Dave6969Author Commented:
I can't use it because I'm using my own scripting engine which I put together a few years ago and I really don't want to have to re-write it at the moment to support unicode. It works so well in many other respects and it's not exactly broken -- this is the first time this issue has come up in a decade.

Thanks sh0e for your continuing patience. You get points again! It's not the database that's doing the weired storing. It is something in Jquery. I've tried both facebox and thickbox and I come across the same issue in both. But as standalone pages the forms work fine, wither I'm submitting AJAX/AHAH or normally.

Ah well. Technology eh?
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.