Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

processing very big json file with php  - json_decode()

Posted on 2014-12-29
7
Medium Priority
?
1,252 Views
Last Modified: 2015-01-06
I'm getting a memory exhausted error on json_decode(), how to resolve this ?
0
Comment
Question by:sniger
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
7 Comments
 
LVL 1

Accepted Solution

by:
Christian Vivas earned 2000 total points
ID: 40522863
have you tried with memory_limit? upgrade the memory_max_limit in your php.ini to more than 128 lets say 512
0
 

Author Comment

by:sniger
ID: 40522869
other solutions ?
0
 
LVL 58

Expert Comment

by:Gary
ID: 40522945
You've got the solution above - increase PHP's memory
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
LVL 111

Expert Comment

by:Ray Paseur
ID: 40522966
Please post a URL that gives us a link to the JSON string.   Then we can experiment with it and try to offer some assistance.  It may be as simple as increasing the memory limit, but it's probably going to require a surprisingly large limit.  PHP arrays and objects can run 20X the size of their string equivalents.  So if the JSON string is 4MB, you may be looking at 80MB just to hold the object of the decoded string.  Add to that any arrays or objects that are extracted in the course of processing the JSON object and you can see how this can domino into a huge memory requirement.

FWIW, JSON is a more economical transport mechanism than XML, and has mostly replaced XML in modern applications.

Once we can see the JSON string, if there is anything that can be done with memory limit, we can show you the requirement.  And if there is not any way to get an adequate memory limit, we may be able to suggest some workarounds.
0
 

Author Comment

by:sniger
ID: 40522968
what if 512 or 1024 is still too small ? How can I know the maximum size of a file which I'm going to get from external sources ?
0
 

Author Comment

by:sniger
ID: 40522970
Ray,

It is some simple key: value, array:
[
 {key1: val1, key2:val3, key3:val3,........},
  {key1: val1a, key2:val3a, key3:val3a,........},
  {key1: val1b, key2:val3b, key3:val3b,........}
.................................................,
]
0
 
LVL 111

Expert Comment

by:Ray Paseur
ID: 40523000
Hi, @sniger.  The key to this issue is the size of the JSON string, the number of nodes, etc.  That's why we need to see the test data set -- one that replicates the likely size of the real JSON strings.  We can't really answer the "what if" questions without seeing some test data.  It's rare that a JSON string is too large for processing with PHP, and since we're dealing with a rare thing here, there may not be a lot of preexisting solutions.
How can I know the maximum size of a file...
In a nutshell, you can't but with some test data we may be able to give you some useful strategies (I hope).
http://sscce.org/
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Developers of all skill levels should learn to use current best practices when developing websites. However many developers, new and old, fall into the trap of using deprecated features because this is what so many tutorials and books tell them to u…
Password hashing is better than message digests or encryption, and you should be using it instead of message digests or encryption.  Find out why and how in this article, which supplements the original article on PHP Client Registration, Login, Logo…
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…
This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.

722 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question