Solved

MySQL not outputting all Emojis in text field when using UTF8MB4

Posted on 2016-10-28
5
60 Views
Last Modified: 2016-10-31
We have a project where we're storing Facebook and Twitter posts in a Mysql database, as first almost all Emojis were being stored as ?. We've since gone ahead and made some configuration changes to the database server, and since then we're starting to see more Emojis saving and appearing correctly, however some Emojis are still showing as ?, sadly I'm not sure which ones they are. I know one of them was a basket ball.

When I execute the following commend on MySQL;

SHOW VARIABLES WHERE Variable_name LIKE 'character\_set\_%' OR Variable_name LIKE 'collation%';

Open in new window


I see the following settings;

character_set_client     = utf8
character_set_connection = utf8
character_set_database   = utf8mb4
character_set_filesystem = binary
character_set_results    = utf8
character_set_server     = utf8mb4
character_set_system     = utf8
collation_connection     = utf8_general_ci
collation_database       = utf8mb4_unicode_ci
collation_server         = utf8mb4_unicode_ci

Open in new window


Our database server is hosted with Rackspace, we've asked them to set up the following configuration;

[client]
default-character-set = utf8mb4

[mysql]
default-character-set = utf8mb4

[mysqld]
character-set-client-handshake = FALSE
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
init-connect='SET NAMES utf8mb4'

Open in new window


I think I've narrowed the issue down to the server not applying the init-connect that's defined in the server configuration. If I open mysql workbench and query the database, I see question marks in place of emojis, however if I run the SET NAMES query first, then the subsequent results come back showing the emojis as I expect.
0
Comment
Question by:SheppardDigital
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
5 Comments
 
LVL 62

Expert Comment

by:gheist
ID: 41864922
It is not executed if connecting user has SUPER privilege.
0
 
LVL 50

Expert Comment

by:Steve Bink
ID: 41865357
As noted by gheist, init_connect does not have any effect on non-SUPER users.  I very much doubt you want to give that privilege to your application.

Moreover, your client-based settings will not be enforced on every connecting client.  It will only impact clients using that .cnf file.  And you do need the UTF8MB4 set - some extended characters will not render in vanilla UTF8.

The solution is to have your client (i.e., the application) run SET NAMES as it initializes the database connection.
0
 

Author Comment

by:SheppardDigital
ID: 41866651
Thanks Steve,

We're using Eloquent ORM so I'm looking to see if there's a way to call SET NAMES on every database connection.

I did speak with Rackspace and they confirmed that the user didn't have SUPER privileges, so I suspect that you are correct, the client will need to call SET NAMES.
0
 
LVL 50

Accepted Solution

by:
Steve Bink earned 500 total points
ID: 41867055
The material at https://laravel.com/docs/5.3/database implies you can set a connection-level charset and collation in the config.  If you set UTF8MB4 there, you could use MySQL's general log to verify what happens at the start of your connection.

I had a similar issue with CiviCRM a while back.  It used SET NAMES but only set the character set, not the collation.  The default collation for UTF8 is utf8_general_ci, but the fields in the db are generally utf8_unicode_ci.  Changing the collation during initial connection save me from many headaches.
0
 

Author Closing Comment

by:SheppardDigital
ID: 41867062
Setting the charset and collation in Laravel's database.php configuration file to utf8mb4 (charset) and utf8mb4_unicode_ci (collation) seems to have done the trick.
0

Featured Post

How our DevOps Teams Maximize Uptime

Our Dev teams are like yours. They’re continually cranking out code for new features/bugs fixes, testing, deploying, responding to production monitoring events and more. It’s complex. So, we thought you’d like to see what’s working for us. Read the use case whitepaper.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

More Fun with XML and MySQL – Parsing Delimited String with a Single SQL Statement Are you ready for another of my SQL tidbits?  Hopefully so, as in this adventure, I will be covering a topic that comes up a lot which is parsing a comma (or other…
When table data gets too large to manage or queries take too long to execute the solution is often to buy bigger hardware or assign more CPUs and memory resources to the machine to solve the problem. However, the best, cheapest and most effective so…
Nobody understands Phishing better than an anti-spam company. That’s why we are providing Phishing Awareness Training to our customers. According to a report by Verizon, only 3% of targeted users report malicious emails to management. With compan…
Finds all prime numbers in a range requested and places them in a public primes() array. I've demostrated a template size of 30 (2 * 3 * 5) but larger templates can be built such 210  (2 * 3 * 5 * 7) or 2310  (2 * 3 * 5 * 7 * 11). The larger templa…

726 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question