Add Array data to database

Hi,
I have an array of data from an external file which I want to import into my database. So far I can print the ARRAY to the screen.

I have attached the array data in a file (this is how it looks when I print it to the screen) which shows an example with three entries. When I import it into my database I want it to overwrite any duplicates.

I presume I need a foreach loop to read and insert the data?

I'm using PHP / MySQL.

Thanks,
nl
LVL 2
nedloganAsked:
Who is Participating?
 
Chris StanyonConnect With a Mentor Commented:
As you say the keys in the array map to the fields in the DB, here's a quick example of how you could do it. As Ray says, you can catch the duplicate insert by setting the AGENT_REF to UNIQUE in your table, and then checking for error number 1062. The code assumes your array is stored in $data:

$dbh = new PDO("mysql:host=localhost;dbname=yourDB", "username", "password");

$columns = array_keys($data[0]);
$columnList = join(',', $columns);
$paramList = join(',', array_map(function($column) { return ":$column"; }, $columns));

$sql = $dbh->prepare("INSERT INTO yourTable ($columnList) VALUES ($paramList)");

foreach ($data as $info):
	try {
		$sql->execute($info);
	} catch(PDOException $e) {
		if ($e->errorInfo[1] == 1062) {
		//we have a duplicate record, so deal with it	
		}
	}
endforeach;

Open in new window

0
 
Ray PaseurCommented:
attached the array data in a file
Erm?
0
 
nedloganAuthor Commented:
-Thanks for spotting my mistake.
arraydata.txt
0
Introducing Cloud Class® training courses

Tech changes fast. You can learn faster. That’s why we’re bringing professional training courses to Experts Exchange. With a subscription, you can access all the Cloud Class® courses to expand your education, prep for certifications, and get top-notch instructions.

 
Ray PaseurCommented:
When you get the data posted here, I'll be glad to take a look and see if I can show you what you might want to do with the PHP programming.  While I am doing that, you might find some value in this article, if you're new to PHP and MySQL programming.
0
 
Ray PaseurCommented:
OK, got it.  Going forward you may find that something like this will make it easier to read the output of print_r().  Preformatting will preserve the indentation and line breaks.

echo '<pre>';
print_r($thing);

Open in new window

Please post the CREATE TABLE statement for the data base table that you will use to import this information, thanks.
0
 
Julian HansenCommented:
Here is one way to do it. It assumes
a) field names in the database are the same as in the array
b) array values are stored in variable $array
// string to hold fieldnames
$fields = ''; 
// string to hold values
$values = " VALUES ";
foreach($array as $item) {
  $val = '';
  foreach ($item as $f => $v) {
     $fields = $fields . "`$f`,";
     $val = $val . "'$v',"
  }
  $values = $values . "($val),"
}
$fields = trim($fields, ',');
$values = trim($values, ',');
$query = trim($query, ',');
// Build query here
$query = "INSERT INTO table ($fields) $values";

Open in new window

0
 
nedloganAuthor Commented:
I've attached the sql used to create the table. Nearly all the fields are included in the table. Some of the image fields near the end of the file are excluded.

In answer to julianH: does your code require the fields in the database to have the same name as the array items, does there need to be the same number of fields to array items.

Also in terms of overwriting duplicates what how will the code handle that?

Thanks!
sqltable.txt
0
 
Ray PaseurCommented:
I have not checked it in detail, but it looks like the column names in the CREATE TABLE statement map 1:1 to the array keys.  Is that so and is it a dependency you're willing to live with?  If so, the mapping from the array to the query will be very easy.  If not, some sort of adapter object can be set up to create the mapping.

Regarding "overwriting duplicates," please tell us what constitutes a duplicate?  Is it one column or a set of columns?  Any clue you can give us will be helpful!  You can avoid getting duplicates in a lot of different ways; the exact way we choose will depend a little on how we define duplication.
0
 
nedloganAuthor Commented:
They do map 1:1 but towards the end some are not required (MEDIA_IMAGE_05 - 60) and a few others.  

Duplicates have the same AGENT_REF so if the AGENT_REF exists in the database and in the imported array then the import would overwrite what is in the database.

Imports are on a daily basis and unlikely to be more than 100 items.

Thanks
0
 
Ray PaseurCommented:
You can mark the AGENT_REF "Unique" and MySQL will throw error number 1062 when there is an attempt to insert a duplicate.  The script can trap this event and use UPDATE instead of INSERT.  We might also be able to do something with REPLACE INTO.  I'll see if I can set up some test cases to demonstrate the concepts.  Have some errands to run, so it will probably be this afternoon before I have a tested code set.
0
 
nedloganAuthor Commented:
Thanks  Chris I'm away from my desk for the next couple of hours will test on return.
0
 
Julian HansenCommented:
There are a number of ways around duplicates.

One way that might work here is to do a batch import into a duplicate of your production table and then do an

First create a duplicate of your data table call it import_table for the purposes of this exercise.

Then as part of the import porcess clear it
TRUNCATE import_table;

Open in new window


And then add your rows here using the code in my earlier post.

And finally
INSERT INOT main_table (FIELDS, ...) 
   SELECT VALUES FROM import_table 
   WHERE AGENT_REF NOT IN (SELECT AGENT_REF FROM main_table)

Open in new window


Other approaches involve first checking if a record exists before importing - and doing this row by row.
Another alternative is to use the MySQL

INSERT IGNORE ... 

Open in new window

To use this you need a unique key defined on your AGENT_REF field - any attempt to insert a duplicate record will be ignored if the key exists.

More on this here http://dev.mysql.com/doc/refman/5.6/en/insert.html
0
 
nedloganAuthor Commented:
Ok, so I tried the code provided by Chris Stanyon which runs to the end with no errors, but the database remains empty.

I checked it is connecting to database ok.

If there was an error I might be able to troubleshoot, but I'm stuck again.

Thanks.
0
 
Chris StanyonCommented:
OK.

Firstly, make sure you've turned on error reporting. At the very start of your script, add this:

error_reporting(E_ALL);
ini_set('display_errors', 1);

Open in new window

Then make liberal use of var_dump() so you can visualize your data:

$columns = array_keys($data[0]);
$columnList = join(',', $columns);
$paramList = join(',', array_map(function($column) { return ":$column"; }, $columns));

var_dump($columnList);
var_dump($paramList);

foreach ($data as $info):
    var_dump($info);
endforeach;

Open in new window

The $columnList should be a comma delimited list of all your columns (grabbed from the Keys of you data array). The $paramList should match this, but each column will be preceded by a colon.

The $info should dump out each record in your array.

Run that so you can at least see what data is being used.
0
 
nedloganAuthor Commented:
Ok, I made the changes, no errors and the data is in the attached file. I still can't see what where is going wrong.

Thanks/
datashown.txt
0
 
Chris StanyonCommented:
Wow. That's hard work to read, but it all looks good (as far as I can tell). One thing I forgot about - telling PDO to fire exceptions!! You need to add this line straight after the $dbh = new... line:

$dbh->setAttribute( PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION );

You could also extend the error handling to report on errors other than 1062. Update your catch block to this:

try {
	$sql->execute($info);
} catch(PDOException $e) {
	if ($e->errorInfo[1] == 1062) {
		echo "We have a duplicate Record." . PHP_EOL;
	} else {
		echo "We have some other error: " . $e->errorInfo[2] . PHP_EOL;
	}
}

Open in new window

If you still don't get any joy, post up your full code.
0
 
nedloganAuthor Commented:
Ok, so the error is

"We have some other error: Unknown column 'MEDIA_IMAGE_05' in 'field list' We have some other error: Unknown column 'MEDIA_IMAGE_05' in 'field list' We have some other error: Unknown column 'MEDIA_IMAGE_05' in 'field list' "

I'll recheck the database and retry.
0
 
Chris StanyonCommented:
OK. You have a key in your data array called MEDIA_IMAGE_05 but it doesn't exist in your table. You can either remove it from the array or add it to your table.
0
 
nedloganAuthor Commented:
Thanks it works now.
0
 
nedloganAuthor Commented:
Thanks for your help.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.