Link to home
Start Free TrialLog in
Avatar of Michel Plungjan
Michel PlungjanFlag for Denmark

asked on

Save JSON data in mySQL database

I have the result from an API I would like to cache in a mySQL DB

I will have two tables, one with the parameter used to call the API and one with the rows of results


My idea is the following

<?php
  $apiCallId=$_POST["id"]; // foreign key in row DB

  /* json contains
    {
      "someHeader": { ... },
      "items":[
        { "someObject": { "someNestedObject": { .. } },
        { "someObject": { "someNestedObject": { .. } },
        { "someObject": { "someNestedObject": { .. } }
      ]
    }
   */
  $phpObj = json_decode(someAPICall(....)); // get the JSON - not sure I need to decode

  $items=$phpObj->items;
  for ($i=0;$i<count($items),$i++) {
    // here I need to add each item to the DB so I get
    // $id - 0 - { "someObject": { "someNestedObject": { .. } }
    // $id - 1 - { "someObject": { "someNestedObject": { .. } }
  }
?>  

Questions

1. Do I have to decode the JSON or can PHP read the JSON and access each array item from items
2. I am pretty sure I need to B64 encode the item data, e.g. each { "someObject": { "someNestedObject": { .. } }
3. Can I concatenate the sql so I only do one commit? I expect 10 items, but do not know the size of them.

I then later need to reverse the process so I can deliver an
{ "items":[ {....},{....}] }
to the browser

I would love some hand holding on the php syntax. I have access to mysqli (and even PDO) if needed but have no experience with either
Avatar of Dave Baldwin
Dave Baldwin
Flag of United States of America image

I may be missing something but I would create only one table with an autoincrement 'id' field, the api-key field and the base64-encoded JSON field.  Note that by base64 encoding the data, you can not search the table for the original contents.  If you need to do that, then you need to decode the JSON to store it in the table so you can find things.  It does not appear that PHP can directly access the JSON data without decoding it.
http://us1.php.net/manual/en/book.json.php - http://us1.php.net/manual/en/function.json-decode.php

The mysqli examples on php.net are pretty clear and most of the time all I have to do is use my own variable names in place of the examples.  http://us1.php.net/manual/en/book.mysqli.php
>>1. Do I have to decode the JSON or can PHP read the JSON and access each array item from items
If I understand you correctly, the API is returning a JSON formatted string and you are interested to iterate over "items" and in doing so you want to then save each item as a JSON string in the DB.

If that is the case, you need to decode so that "items" can become "iterable."  This "decoding" process will decode everything, including each item in "items".  Thus, within the for construct you need to encode each "item" (singular) so that you can then save THAT specific item as a json-encoded string.

>>2. I am pretty sure I need to B64 encode the item data, e.g. each { "someObject": { "someNestedObject": { .. } }
If you need to "search" for data within that field, then I would stay away from B64 encoding.

>>3. Can I concatenate the sql so I only do one commit? I expect 10 items, but do not know the size of them.
MySQL certainly supports the ability to execute:
INSERT INTO Table(FieldName) Values ('first value'), (' second value'), ('third value'), etc

But then you need to make sure you escape the values in order to avoid sql injection. My suggestion is to use a prepared statement instead.  On each iteration you just update the value you want to insert and execute it.  The API does the escaping under the hood for you.

Read the comments in the code below.
<?php

$phpObj = json_decode(someAPICall(....)); // get the JSON - not sure I need to decode 
if( is_null($phpObj) )
{
	echo 'Unable to decode data';
}
else
{
	$items=$phpObj->items; 

	/* Connect to an ODBC database using driver invocation */
	$dsn = 'mysql:dbname=testdb;host=127.0.0.1';
	$user = 'dbuser';
	$password = 'dbpass';
	$errors=array();

	try {
    	$dbh = new PDO($dsn, $user, $password);
		//force PDO to throw exceptions when errors are encountered
		$dbh->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
		$dbh->beginTransaction();

		try
		{
			/* Execute a prepared statement by passing an array of values */
			$sth = $dbh->prepare('INSERT INTO tablename(fieldName) VALUES(?)');

			for ($i=0, $limit=count($items); $i<$limit; ++$i)
			{
				// here I need to add each item to the DB so I get
				// $id - 0 - { "someObject": { "someNestedObject": { .. } }
				// $id - 1 - { "someObject": { "someNestedObject": { .. } }
				$item=json_encode($items[$i]);
        
				//if an error were to occurr while executing the command, an
				//exception will be thrown (due to the setting specified 
				//above immediately after connecting to the db).  So you don't need
				//to check if the statement succeeded or failed.  It will be caught
				//in the "catch" clause.
				$sth->execute( array($item) );

				//if you have an auto-id column, you may be instersted in the
				//following statement
				//echo '<br />Id of inserted item: '.$dbh->lastInsertId();
			}
			//if you make it this far, no exceptions were thrown - everything was OK
			$dbh->commit();
    	}
		catch(PDOException $e)
		{
			//if you make it here, there were problems in the prepare() OR execute() method calls.
			$dbh->rollBack();
			echo "Error!: " . $e->getMessage() . '<br />'; 
		}

	}
	catch (PDOException $e)
	{
		echo 'Connection failed: ' . $e->getMessage();
	}
}
?>

Open in new window


Regards,
Hielo
Avatar of Michel Plungjan

ASKER

I need to have two tables
The first is to present the saved query parameters from the original api call, the other to simply store the 10 rows returned.

So what I seem to need is

1) store the parms in master table and create a unique ID
2) call the api and decode into a php object
3) run over object->items and for each item save it

So what would the php and sql statement look like?

$items=$phpObj->items;
for ($i=0;$i<count($items),$i++) {
    $query = "INSERT INTO 'cache' (id, idx, b64json) VALUES ($id,$i,'base64_encode(json_encode($items[$i]))')";    
    $result = $mysqli->query($query);
  }

or can I do something more clever?
Refer to http://www.php.net/manual/en/mysqli-stmt.bind-param.php
for other options to the first argument of the bind_param() method.
/* create a prepared statement */
if ( $stmt = $mysqli->prepare("INSERT INTO `cache` (id, idx, b64json) VALUES (?, ?, ? )") )
{
	for ($i=0, $limit=count($items); $i<$limit; ++$i)
	{
		$item=base64_encode(json_encode($items[$i]));

	    /* bind parameters for markers */
	    $stmt->bind_param("i", $id);
	    $stmt->bind_param("i", $i);
	    $stmt->bind_param("s", $item);

	    /* execute query */
    	$stmt->execute();
	}

    $stmt->close();
}

Open in new window

Thanks a lot. I posted my second post without seeing yours

So PDO is better than mysqli here?

I do not need to search the json string just store it.


I will look at your code tomorrow
>>So PDO is better than mysqli here?
I have never actually done any benchmarks but I have used both and have found the PDO API easier, specifically in the aspect of prepared statements.  You just provide an array of values and done (as opposed to the bind_param() method of mysqli).

Additionally, if you use PDO, if for whatever reason you must migrate your application to a different DB server (say postgres) then the only change required on my first post would be to change:
$dsn = 'mysql:dbname=testdb;host=127.0.0.1';

to:
$dsn = 'pgsql:dbname=testdb;host=127.0.0.1';

Put another way, given the various dbs that PDO supports,  once you learn it (or become familiar/comfortable with it), if you then use it on new projects AND if any of those projects need to be moved later on onto a different DB, then there's minimal work to be done.

As far as my first post, the lines that need to be updated based on your second post are:
//line 13:
$sth = $dbh->prepare('INSERT INTO `cache` (id, idx, b64json) VALUES (?, ?, ? )');

//line 34
$item=base64_encode(json_encode($items[$i]));

//line 41:
$sth->execute( array($id, $i, $item) );

On another note, if you search mysqli vs PDO, you should be able to find a lot of opinions.  Again, for me the selling point are the portability and the simple prepared statements API.

Regards,
Hielo
This article may be helpful with questions about MySQLi vs PDO.
https://www.experts-exchange.com/Web_Development/Web_Languages-Standards/PHP/PHP_Databases/A_11177-PHP-MySQL-Deprecated-as-of-PHP-5-5-0.html

Just a few philosophical notes...

If you store the query parameters and the JSON results in a way that lets you coordinate the information, you're doing the right thing.

JSON is just a transport mechanism, similar to XML.  If you were retrieving the data in an XML string, it would be susceptible of carrying the same information in key-value pairs as is JSON.

When JSON_Decode() runs you get an object that has properties, some of which may be of interest in subsequent queries.  It might make sense to decode the JSON string and save some of the properties in indexed columns to facilitate future queries.

When JSON_Decode() fails, you can get a little bit of error information.
http://php.net/manual/en/function.json-last-error.php

JSON strings are always UTF-8.
http://www.json.org/
The data is returned in JSON only.

It is then saved to not have to use the api again (it is not free) just to see a previous result.

So the encode is to handle all quotes and special chars and even foreign language in the info.

Thanks for the information. I will try to put it all together
Hielo here is the complete php
I do not understand where
$errors=array();
is used

<?php
include ('pdo_functions.php'); // contains $dsn = 'mysql:dbname=stJson;host=127.0.0.1'; $user = 'dbuser'; $password = 'dbpass';

function error_msg($text) { 
     $pre = '{"error:=":';  
     $post = '"}'; //example of addon to end 
     die($pre.$text.$post); 
} 

$phpObj = json_decode(someApiReturningJSON());

if( is_null($phpObj) ) {
  error_msg("Unable to decode data");
}

$items = $phpObj->items;
echo count($items)." found";

$errors=array();

try {
  $dbh = new PDO($dsn, $user, $password);
  //force PDO to throw exceptions when errors are encountered
  $dbh->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
  $dbh->beginTransaction();
  try {
    /* Execute a prepared statement by passing an array of values */
    $sth = $dbh->prepare('INSERT INTO `cache` (id, idx, b64json) VALUES (?, ?, ? )');
    for ($i=0, $limit=count($items); $i<$limit; ++$i) {
      $title = $items[$i]->title;

      $item=base64_encode(json_encode($items[$i])); // re-jsonencode and then b64
      
      //if an error were to occurr while executing the command, an
      //exception will be thrown (due to the setting specified 
      //above immediately after connecting to the db).  So you don't need
      //to check if the statement succeeded or failed.  It will be caught
      //in the "catch" clause.
      $sth->execute( array($id, $i, $item) );
    }
    //if you make it this far, no exceptions were thrown - everything was OK
    $dbh->commit();
  }
  catch(PDOException $e) {
    //if you make it here, there were problems in the prepare() OR execute() method calls.
    $dbh->rollBack();
    echo "Error!: " . $e->getMessage() . '<br />'; 
  }
}
catch (PDOException $e) {
  echo 'Connection failed: ' . $e->getMessage();
}
?>

Open in new window

ASKER CERTIFIED SOLUTION
Avatar of hielo
hielo
Flag of Wallis and Futuna image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Ah, the := is a typo. Thanks.

The id is now a get var. all works lovely except the foreign key id was set to unique
>>except the foreign key id was set to unique
Does this mean this is already solved or do you still need more help?
better late than nrever. The project was alas scrapped