Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

Add Array data to database

Posted on 2014-03-05
20
Medium Priority
?
275 Views
Last Modified: 2014-03-05
Hi,
I have an array of data from an external file which I want to import into my database. So far I can print the ARRAY to the screen.

I have attached the array data in a file (this is how it looks when I print it to the screen) which shows an example with three entries. When I import it into my database I want it to overwrite any duplicates.

I presume I need a foreach loop to read and insert the data?

I'm using PHP / MySQL.

Thanks,
nl
0
Comment
Question by:nedlogan
  • 9
  • 5
  • 4
  • +1
20 Comments
 
LVL 111

Expert Comment

by:Ray Paseur
ID: 39906224
attached the array data in a file
Erm?
0
 
LVL 2

Author Comment

by:nedlogan
ID: 39906232
-Thanks for spotting my mistake.
arraydata.txt
0
 
LVL 111

Expert Comment

by:Ray Paseur
ID: 39906233
When you get the data posted here, I'll be glad to take a look and see if I can show you what you might want to do with the PHP programming.  While I am doing that, you might find some value in this article, if you're new to PHP and MySQL programming.
0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 
LVL 111

Expert Comment

by:Ray Paseur
ID: 39906241
OK, got it.  Going forward you may find that something like this will make it easier to read the output of print_r().  Preformatting will preserve the indentation and line breaks.

echo '<pre>';
print_r($thing);

Open in new window

Please post the CREATE TABLE statement for the data base table that you will use to import this information, thanks.
0
 
LVL 60

Expert Comment

by:Julian Hansen
ID: 39906267
Here is one way to do it. It assumes
a) field names in the database are the same as in the array
b) array values are stored in variable $array
// string to hold fieldnames
$fields = ''; 
// string to hold values
$values = " VALUES ";
foreach($array as $item) {
  $val = '';
  foreach ($item as $f => $v) {
     $fields = $fields . "`$f`,";
     $val = $val . "'$v',"
  }
  $values = $values . "($val),"
}
$fields = trim($fields, ',');
$values = trim($values, ',');
$query = trim($query, ',');
// Build query here
$query = "INSERT INTO table ($fields) $values";

Open in new window

0
 
LVL 2

Author Comment

by:nedlogan
ID: 39906304
I've attached the sql used to create the table. Nearly all the fields are included in the table. Some of the image fields near the end of the file are excluded.

In answer to julianH: does your code require the fields in the database to have the same name as the array items, does there need to be the same number of fields to array items.

Also in terms of overwriting duplicates what how will the code handle that?

Thanks!
sqltable.txt
0
 
LVL 111

Expert Comment

by:Ray Paseur
ID: 39906344
I have not checked it in detail, but it looks like the column names in the CREATE TABLE statement map 1:1 to the array keys.  Is that so and is it a dependency you're willing to live with?  If so, the mapping from the array to the query will be very easy.  If not, some sort of adapter object can be set up to create the mapping.

Regarding "overwriting duplicates," please tell us what constitutes a duplicate?  Is it one column or a set of columns?  Any clue you can give us will be helpful!  You can avoid getting duplicates in a lot of different ways; the exact way we choose will depend a little on how we define duplication.
0
 
LVL 2

Author Comment

by:nedlogan
ID: 39906372
They do map 1:1 but towards the end some are not required (MEDIA_IMAGE_05 - 60) and a few others.  

Duplicates have the same AGENT_REF so if the AGENT_REF exists in the database and in the imported array then the import would overwrite what is in the database.

Imports are on a daily basis and unlikely to be more than 100 items.

Thanks
0
 
LVL 111

Expert Comment

by:Ray Paseur
ID: 39906418
You can mark the AGENT_REF "Unique" and MySQL will throw error number 1062 when there is an attempt to insert a duplicate.  The script can trap this event and use UPDATE instead of INSERT.  We might also be able to do something with REPLACE INTO.  I'll see if I can set up some test cases to demonstrate the concepts.  Have some errands to run, so it will probably be this afternoon before I have a tested code set.
0
 
LVL 44

Accepted Solution

by:
Chris Stanyon earned 2000 total points
ID: 39906454
As you say the keys in the array map to the fields in the DB, here's a quick example of how you could do it. As Ray says, you can catch the duplicate insert by setting the AGENT_REF to UNIQUE in your table, and then checking for error number 1062. The code assumes your array is stored in $data:

$dbh = new PDO("mysql:host=localhost;dbname=yourDB", "username", "password");

$columns = array_keys($data[0]);
$columnList = join(',', $columns);
$paramList = join(',', array_map(function($column) { return ":$column"; }, $columns));

$sql = $dbh->prepare("INSERT INTO yourTable ($columnList) VALUES ($paramList)");

foreach ($data as $info):
	try {
		$sql->execute($info);
	} catch(PDOException $e) {
		if ($e->errorInfo[1] == 1062) {
		//we have a duplicate record, so deal with it	
		}
	}
endforeach;

Open in new window

0
 
LVL 2

Author Comment

by:nedlogan
ID: 39906463
Thanks  Chris I'm away from my desk for the next couple of hours will test on return.
0
 
LVL 60

Expert Comment

by:Julian Hansen
ID: 39906475
There are a number of ways around duplicates.

One way that might work here is to do a batch import into a duplicate of your production table and then do an

First create a duplicate of your data table call it import_table for the purposes of this exercise.

Then as part of the import porcess clear it
TRUNCATE import_table;

Open in new window


And then add your rows here using the code in my earlier post.

And finally
INSERT INOT main_table (FIELDS, ...) 
   SELECT VALUES FROM import_table 
   WHERE AGENT_REF NOT IN (SELECT AGENT_REF FROM main_table)

Open in new window


Other approaches involve first checking if a record exists before importing - and doing this row by row.
Another alternative is to use the MySQL

INSERT IGNORE ... 

Open in new window

To use this you need a unique key defined on your AGENT_REF field - any attempt to insert a duplicate record will be ignored if the key exists.

More on this here http://dev.mysql.com/doc/refman/5.6/en/insert.html
0
 
LVL 2

Author Comment

by:nedlogan
ID: 39907026
Ok, so I tried the code provided by Chris Stanyon which runs to the end with no errors, but the database remains empty.

I checked it is connecting to database ok.

If there was an error I might be able to troubleshoot, but I'm stuck again.

Thanks.
0
 
LVL 44

Expert Comment

by:Chris Stanyon
ID: 39907085
OK.

Firstly, make sure you've turned on error reporting. At the very start of your script, add this:

error_reporting(E_ALL);
ini_set('display_errors', 1);

Open in new window

Then make liberal use of var_dump() so you can visualize your data:

$columns = array_keys($data[0]);
$columnList = join(',', $columns);
$paramList = join(',', array_map(function($column) { return ":$column"; }, $columns));

var_dump($columnList);
var_dump($paramList);

foreach ($data as $info):
    var_dump($info);
endforeach;

Open in new window

The $columnList should be a comma delimited list of all your columns (grabbed from the Keys of you data array). The $paramList should match this, but each column will be preceded by a colon.

The $info should dump out each record in your array.

Run that so you can at least see what data is being used.
0
 
LVL 2

Author Comment

by:nedlogan
ID: 39907127
Ok, I made the changes, no errors and the data is in the attached file. I still can't see what where is going wrong.

Thanks/
datashown.txt
0
 
LVL 44

Expert Comment

by:Chris Stanyon
ID: 39907195
Wow. That's hard work to read, but it all looks good (as far as I can tell). One thing I forgot about - telling PDO to fire exceptions!! You need to add this line straight after the $dbh = new... line:

$dbh->setAttribute( PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION );

You could also extend the error handling to report on errors other than 1062. Update your catch block to this:

try {
	$sql->execute($info);
} catch(PDOException $e) {
	if ($e->errorInfo[1] == 1062) {
		echo "We have a duplicate Record." . PHP_EOL;
	} else {
		echo "We have some other error: " . $e->errorInfo[2] . PHP_EOL;
	}
}

Open in new window

If you still don't get any joy, post up your full code.
0
 
LVL 2

Author Comment

by:nedlogan
ID: 39907216
Ok, so the error is

"We have some other error: Unknown column 'MEDIA_IMAGE_05' in 'field list' We have some other error: Unknown column 'MEDIA_IMAGE_05' in 'field list' We have some other error: Unknown column 'MEDIA_IMAGE_05' in 'field list' "

I'll recheck the database and retry.
0
 
LVL 44

Expert Comment

by:Chris Stanyon
ID: 39907258
OK. You have a key in your data array called MEDIA_IMAGE_05 but it doesn't exist in your table. You can either remove it from the array or add it to your table.
0
 
LVL 2

Author Comment

by:nedlogan
ID: 39907540
Thanks it works now.
0
 
LVL 2

Author Closing Comment

by:nedlogan
ID: 39907544
Thanks for your help.
0

Featured Post

Free learning courses: Active Directory Deep Dive

Get a firm grasp on your IT environment when you learn Active Directory best practices with Veeam! Watch all, or choose any amount, of this three-part webinar series to improve your skills. From the basics to virtualization and backup, we got you covered.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This post contains step-by-step instructions for setting up alerting in Percona Monitoring and Management (PMM) using Grafana.
Recursive SQL is one of the most fascinating and powerful and yet dangerous feature offered in many modern databases today using a Common Table Expression (CTE) first introduced in the ANSI SQL 99 standard. The first implementations of CTE began ap…
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…
In this video, Percona Solution Engineer Rick Golba discuss how (and why) you implement high availability in a database environment. To discuss how Percona Consulting can help with your design and architecture needs for your database and infrastr…
Suggested Courses

885 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question