Link to home
Start Free TrialLog in
Avatar of MrChrisDavids
MrChrisDavids

asked on

json_encode fails with special characters

I've got the following piece of code:

		$query = "SELECT * 
				  FROM user
				  ORDER BY userFullName ASC";
		$result = mysqli_query($con, $query);
		$rows = array();
		while ($r = mysqli_fetch_assoc($result)) {
			$rows[] = $r;
		}
		header('Content-Type: application/json');
        echo json_encode($rows);

Open in new window


One of the users has a special character on his name ( é ) and when that's in there - the json_encode fails. How can one go around this and get a good output anyway?
Avatar of Chris Stanyon
Chris Stanyon
Flag of United Kingdom of Great Britain and Northern Ireland image

Can you explain what you mean by 'fails' - do you get an error message (you may need to turn on error_reporting. Do you know what encoding your database is set to (it should be UTF!)

As an aside, you don't need to loop through the array just to create an array:

header('Content-Type: application/json');
$result = mysqli_query($con, $query);
$rows = mysqli_fetch_all($result, MYSQLI_ASSOC);
echo json_encode($rows);

Open in new window

First, you must be sure all your scripts are utf-8 encoded; second, you must be sure to have all your database columns encoded with utf-8; third, you have to put this line in your html head section:
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"> 

Open in new window

This should fix the problem, if I'm not forgetting something :-)
Avatar of MrChrisDavids
MrChrisDavids

ASKER

The DB's set to UTF8 encoding.

Error reporting is on but echo json_encode($rows); simply doesn't output anything at all.

If I dump the array, the content's in there.

Adding the meta tag would imply that my output's a HTML page, no? I want just JSON thus the header line within the script. Or am I going about it wrong?
The meta tag only if you get to display in a html page. But I don't understand a thing: you say that the script doesn't output anything, but then you say that if you dump (suppose using var_dump or print_r) you see the content there. How are you trying to output you results?
Sorry - guess I unclear on that part. If I dump out the array: var_dump($rows); I get all the content which is supposed to be JSON encoded. Then when the echo_json_encode($rows); run, nothing happens.

The output's supposed to go to the page - but as a JSON page, not HTML.
Include the encoding in your header:

header('Content-Type: application/json; charset=utf-8');

Open in new window

If you run php 5.4+ you can try to use this:

echo json_encode($rows, JSON_UNESCAPED_UNICODE);

Open in new window

Otherwise, you can use this function:
function jsonRemoveUnicodeSequences( $struct )
{
	return preg_replace( "/\\\\u([a-f0-9]{4})/e", "iconv('UCS-4LE','UTF-8',pack('V', hexdec('U$1')))", json_encode( $struct ) );
}
echo jsonRemoveUnicodeSequences($rows);

Open in new window

This is a common issue.  JSON requires UTF-8.  An explanation and the solutions are in this article.
https://www.experts-exchange.com/Web_Development/Web_Languages-Standards/PHP/A_11880-Unicode-PHP-and-Character-Collisions.html
Adding charset to the content-type didn't work.

I'm running PHP 5.6.3, but JSON_UNESCAPED_UNICODE didn't work.

Using the function returned the following:
<b>Deprecated</b>:  preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in <b>D:\sites\incident\www\data\getusers.php</b> on line <b>34</b><br />

Open in new window


Tried reading the article but couldn't find any immediately applicable solution? Maybe I missed it?
As the error states, the /e switch of preg_replace is deprecated.

Try to var_sump the $rows array, and then assign the result of json_encode to a variable and var_dump that as well:

var_dump($rows);
$json  = json_encode($rows);
var_dump($json);

Open in new window


See what that turns up
You probably would want to read the entire article.  It's not a one-size-fits-all situation.  You risk losing information if you do the conversion wrong.  Your choices about how to do the conversion will depend on your particular circumstances.  The article describes your choices, what the JSON standard requires, how to find offending characters, what will happen to your data base records, etc.  The programming and processes are fairly-well vetted by the DC PHP community.  Here's the slide deck if you want the TL;DR version.  The article has cut-and-paste code examples.
http://www.slideshare.net/RayPaseur/unicode-php-and-character-set-collisions
Dumping the array, json etc.. :)

        var_dump($rows);
        $json = json_encode($rows);
        var_dump($json);

Open in new window


Results in ..

array(3) {
  [0]=>
  array(3) {
    ["userId"]=>
    string(1) "1"
    ["userName"]=>
    string(6) "chrwir"
    ["userFullName"]=>
    string(15) "Christian Wirén"
  }
  [1]=>
  array(3) {
    ["userId"]=>
    string(1) "3"
    ["userName"]=>
    string(6) "marjoh"
    ["userFullName"]=>
    string(16) "Martin Johansson"
  }
  [2]=>
  array(3) {
    ["userId"]=>
    string(1) "2"
    ["userName"]=>
    string(6) "maxjon"
    ["userFullName"]=>
    string(11) "Max Jonborn"
  }
}
bool(false)

Open in new window

Strange - it works perfectly for me with your data.

Add a call to json_last_error() after your json_encode call, and then take a look at the man page to see if that gives any further clues:

var_dump($rows);
$json  = json_encode($rows);
var_dump($json);
echo json_last_error();

Open in new window


Man Page: http://php.net/manual/en/function.json-last-error.php
What character set what in use at the time this string was created?

Christian Wirén

I ask because it appears to not be valid UTF-8.  You can see the hexdump output here.  JSON requires valid UTF-8.
http://iconoun.com/demo/temp_mrchrisdavids.php
The encoding for the table is utf8_general_ci.

I've gone as far as to delete the table and re-creating it (just to make sure it's the correct encoding) but I'm still getting the same error; JSON error (5): Malformed UTF-8 characters (which I found using the sugested json_last_error
ASKER CERTIFIED SOLUTION
Avatar of Ray Paseur
Ray Paseur
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial