APD Toronto
asked on
Changing group_concat_max_len
Hi Experts,
I have noticed in my Mysql query, where I use GROUP_CONCAT I do not get all expected records as per my syntax below. With little bit of research I noticed that this has limit of 1024 bytes, which can be increased.
I am wondering where and how can I increase this, and what is the maximum? Can the maximum be unlimited?
Any help will be greatly appreciated.
I have noticed in my Mysql query, where I use GROUP_CONCAT I do not get all expected records as per my syntax below. With little bit of research I noticed that this has limit of 1024 bytes, which can be increased.
I am wondering where and how can I increase this, and what is the maximum? Can the maximum be unlimited?
SELECT locs.id, name, default_time, default_vehicle,
GROUP_CONCAT(DISTINCT assoc.agent_id ORDER BY agents.fName, agents.lName) AS agent_ids
FROM locations AS locs
INNER JOIN location_agent_assoc AS assoc ON locs.id = assoc.location_id
INNER JOIN agents ON assoc.agent_id = agents.id
WHERE locs.archived = 0 AND agents.archived = 0
GROUP BY locs.id, name, default_time, default_vehicle
ORDER BY locs.name
Any help will be greatly appreciated.
ASKER
I cannot find etc/my.cnf
oh, missed that you are on windows, you need to find etc/my.ini by following where you installed the mysql server
<Drive_letter>:\program files (x86)\mysql server\
<Drive_letter>:\program files (x86)\mysql server\
ASKER
I changed it,. but it does not seem to help. How can I check if the change is in effect?
ASKER
Also, do I need a semi-column after my statement:
group_concat_max_len = 18446744073709547520;
group_concat_max_len = 18446744073709547520;
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
yes, set global group_concat_max/-length=1 8446744073 709547520;
are you sure your current one is low?
which my.ini did you use from the examples?
are you sure your current one is low?
which my.ini did you use from the examples?
ASKER
This is my my.ini
it is located in C:\Program Files\MySQL\MySQL Server 5.6\my.ini. I found this by locating my-default.ini, and copied it to my.ini in the same directory and adding the group_concat_max_len = 18446744073709547520; line. However, my query still does not return all expected records with group_concat.
Is there a way I can check what is the maximum lenght applied to the group_concat?
# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/5.6/en/server-configuration-defaults.html
# *** DO NOT EDIT THIS FILE. It's a template which will be copied to the
# *** default location during install, and will be replaced if you
# *** upgrade to a newer version of MySQL.
[mysqld]
group_concat_max_len = 18446744073709547520;
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin
# These are commonly set, remove the # and set as required.
# basedir = .....
# datadir = .....
# port = .....
# server_id = .....
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
it is located in C:\Program Files\MySQL\MySQL Server 5.6\my.ini. I found this by locating my-default.ini, and copied it to my.ini in the same directory and adding the group_concat_max_len = 18446744073709547520; line. However, my query still does not return all expected records with group_concat.
Is there a way I can check what is the maximum lenght applied to the group_concat?
ASKER
UPDATE: even though my OS is x64, I checked my phpInfo() and the architecture is x86, so I changed the value to 4294967295, restarted my server (physically), but same result with truncated data.
ASKER
I found it in phpMyadmin, I assume this change permanent?
ASKER
When I changed the variable in phpMyAdmin, my query worked.
However, when I restart my server the variable is back to 1024, and my query is again truncated. How can I make it a permanent change?
However, when I restart my server the variable is back to 1024, and my query is again truncated. How can I make it a permanent change?
run the command set global you also need to update your my.ini to make it permanent everytime the system boots or the service is restarted.
You can add a directive to your query as well.
run
set global group_concat_max_len=34235 43543;
this will last until the server is rebooted or the service is restarted.
If you can not alter the my.ini, add the set global directive before you run the query in question.
This way whenever the need to run this query comes up, the set global directive is run first.
You can add a directive to your query as well.
run
set global group_concat_max_len=34235
this will last until the server is rebooted or the service is restarted.
If you can not alter the my.ini, add the set global directive before you run the query in question.
This way whenever the need to run this query comes up, the set global directive is run first.
ASKER
If I add that before my query that would mean I need to give the user admin privileges, which I don't want to do.
There must be a way that I can set this permanently on Windows. What about a batch file that I can add somewhere in startup?
As well, every time I change my C:\Program Files\MySQL\MySQL Server 5.6\my.ini file I restart MySQL with my services in Control panel without doing a physical server restart, then refresh phpMyAdmin to see if the effect had taken place, which has not. Is this a sufficient restart or do I need to physically restart?
There must be a way that I can set this permanently on Windows. What about a batch file that I can add somewhere in startup?
As well, every time I change my C:\Program Files\MySQL\MySQL Server 5.6\my.ini file I restart MySQL with my services in Control panel without doing a physical server restart, then refresh phpMyAdmin to see if the effect had taken place, which has not. Is this a sufficient restart or do I need to physically restart?
placement of the directive is important.
you can have the set session group_concat_max_length=18 923423; precede the query you run.
it does not need admin rights.
group_concat_max_length=45 43543345
you are placing the above within the
[mysqld]
group_concat_max_length=18 890320
optional examples deals with setting an init-file.......
https://bugs.mysql.com/bug.php?id=74037
you can have the set session group_concat_max_length=18
it does not need admin rights.
group_concat_max_length=45
you are placing the above within the
[mysqld]
group_concat_max_length=18
optional examples deals with setting an init-file.......
https://bugs.mysql.com/bug.php?id=74037
ASKER
Ok, this is strange... I changed my query to
...but I still get truncated results.
As well, under phpMyAdmin logged in as root, did the following code under the SQL window
SET SESSION group_concat_max_len=34235 43543;
...Went under variables, group_concat_max_len still showing 1024. How can this be?
Finally, is it not possible using mysqld -uroot -pXXXX to set this? If so, as I am simply put that in a batch file and done?
SET SESSION group_concat_max_len=3423543543; SELECT locs.id, name, default_time, default_vehicle, GROUP_CONCAT(DISTINCT assoc.agent_id ORDER BY agents.fName, agents.lName) AS agent_ids FROM locations AS locs INNER JOIN location_agent_assoc AS assoc ON locs.id = assoc.location_id INNER JOIN agents ON assoc.agent_id = agents.id WHERE locs.archived = 0 AND agents.archived = 0 GROUP BY locs.id, name, default_time, default_vehicle ORDER BY locs.name
...but I still get truncated results.
As well, under phpMyAdmin logged in as root, did the following code under the SQL window
SET SESSION group_concat_max_len=34235
...Went under variables, group_concat_max_len still showing 1024. How can this be?
Finally, is it not possible using mysqld -uroot -pXXXX to set this? If so, as I am simply put that in a batch file and done?
borrowing a very simple example from here I just did this. Notice the change in result
see: http://sqlfiddle.com/#!9/c4ecc/2
I suggest you try a similar small example to see if you can control that length.
sample
SELECT DISTINCT
parent_id, GROUP_CONCAT(DISTINCT child_id ORDER BY child_id) AS child_id_list
FROM group_c
group by parent_id
ORDER BY parent_id;
| parent_id | child_id_list |
|-----------|---------------|
| 1 | 1,2,3,4 |
| 2 | 1,4,6 |
| 3 | 1,2 |
| 4 | 1 |
| 5 | 0 |
SET SESSION group_concat_max_len=6;
SELECT DISTINCT
parent_id, GROUP_CONCAT(DISTINCT child_id ORDER BY child_id) AS child_id_list
FROM group_c
group by parent_id
ORDER BY parent_id;
| parent_id | child_id_list |
|-----------|---------------|
| 1 | 1,2,3, |
| 2 | 1,4,6 |
| 3 | 1,2 |
| 4 | 1 |
| 5 | 0 |
see: http://sqlfiddle.com/#!9/c4ecc/2
I suggest you try a similar small example to see if you can control that length.
sample
CREATE TABLE `group_c` (
`parent_id` int(11) DEFAULT NULL,
`child_id` int(11) DEFAULT NULL
) ENGINE=InnoDB;
INSERT INTO group_c(parent_id, child_id)
VALUES (1,1),(1,2),(1,3),(2,1),(2,4),(1,4),(2,6),(3,1),(3,2),(4,1),(4,1),(1,1),(5,0);
ASKER
Taking Paul's advice, I see that your example in SQL Fiddle works well.
I also set my max len to 5, just to visually see the difference. In doing so, I did notice 1 thing that I missed yesterday...
I'm now running:
in phpMyAdmin, I just see #Rows: 231, any my query is edited as follows:
However, I do not also see the 231 Rows, but if I remove the SET statement, I do see my 231 rows, but not with max len of 5. This is also happening with my PHP code, where with SET, I get no rows returned.
So my question now is, why with the SET statement, I don't see my rows.
I also set my max len to 5, just to visually see the difference. In doing so, I did notice 1 thing that I missed yesterday...
I'm now running:
SET SESSION group_concat_max_len=5; /*3423543543;*/
SELECT locs.id, name, default_time, default_vehicle, GROUP_CONCAT(DISTINCT assoc.agent_id ORDER BY agents.fName, agents.lName) AS agent_ids FROM locations AS locs INNER JOIN location_agent_assoc AS assoc ON locs.id = assoc.location_id INNER JOIN agents ON assoc.agent_id = agents.id WHERE locs.archived = 0 AND agents.archived = 0 GROUP BY locs.id, name, default_time, default_vehicle ORDER BY locs.name
in phpMyAdmin, I just see #Rows: 231, any my query is edited as follows:
SET SESSION group_concat_max_len=5;# MySQL returned an empty result set (i.e. zero rows). /*3423543543;*/ SELECT locs.id, name, default_time, default_vehicle, GROUP_CONCAT(DISTINCT assoc.agent_id ORDER BY agents.fName, agents.lName) AS agent_ids FROM locations AS locs INNER JOIN location_agent_assoc AS assoc ON locs.id = assoc.location_id INNER JOIN agents ON assoc.agent_id = agents.id WHERE locs.archived = 0 AND agents.archived = 0 GROUP BY locs.id, name, default_time, default_vehicle ORDER BY locs.name # Rows: 231
However, I do not also see the 231 Rows, but if I remove the SET statement, I do see my 231 rows, but not with max len of 5. This is also happening with my PHP code, where with SET, I get no rows returned.
So my question now is, why with the SET statement, I don't see my rows.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
OK.
ASKER
Thank you both!
Your choice for my.ini is too limited, other than what you added everything else is a comment. There should be a my-*.ini
Within phpmyadmin, there is a suggested changes on DB/index,etc.
Within phpmyadmin, there is a suggested changes on DB/index,etc.
You can also add it into /etc/my.cnf file so it will be set when mysql starts next time.