Solved

Add Array data to database

Posted on 2014-03-05
20
264 Views
Last Modified: 2014-03-05
Hi,
I have an array of data from an external file which I want to import into my database. So far I can print the ARRAY to the screen.

I have attached the array data in a file (this is how it looks when I print it to the screen) which shows an example with three entries. When I import it into my database I want it to overwrite any duplicates.

I presume I need a foreach loop to read and insert the data?

I'm using PHP / MySQL.

Thanks,
nl
0
Comment
Question by:nedlogan
  • 9
  • 5
  • 4
  • +1
20 Comments
 
LVL 108

Expert Comment

by:Ray Paseur
ID: 39906224
attached the array data in a file
Erm?
0
 
LVL 2

Author Comment

by:nedlogan
ID: 39906232
-Thanks for spotting my mistake.
arraydata.txt
0
 
LVL 108

Expert Comment

by:Ray Paseur
ID: 39906233
When you get the data posted here, I'll be glad to take a look and see if I can show you what you might want to do with the PHP programming.  While I am doing that, you might find some value in this article, if you're new to PHP and MySQL programming.
0
 
LVL 108

Expert Comment

by:Ray Paseur
ID: 39906241
OK, got it.  Going forward you may find that something like this will make it easier to read the output of print_r().  Preformatting will preserve the indentation and line breaks.

echo '<pre>';
print_r($thing);

Open in new window

Please post the CREATE TABLE statement for the data base table that you will use to import this information, thanks.
0
 
LVL 51

Expert Comment

by:Julian Hansen
ID: 39906267
Here is one way to do it. It assumes
a) field names in the database are the same as in the array
b) array values are stored in variable $array
// string to hold fieldnames
$fields = ''; 
// string to hold values
$values = " VALUES ";
foreach($array as $item) {
  $val = '';
  foreach ($item as $f => $v) {
     $fields = $fields . "`$f`,";
     $val = $val . "'$v',"
  }
  $values = $values . "($val),"
}
$fields = trim($fields, ',');
$values = trim($values, ',');
$query = trim($query, ',');
// Build query here
$query = "INSERT INTO table ($fields) $values";

Open in new window

0
 
LVL 2

Author Comment

by:nedlogan
ID: 39906304
I've attached the sql used to create the table. Nearly all the fields are included in the table. Some of the image fields near the end of the file are excluded.

In answer to julianH: does your code require the fields in the database to have the same name as the array items, does there need to be the same number of fields to array items.

Also in terms of overwriting duplicates what how will the code handle that?

Thanks!
sqltable.txt
0
 
LVL 108

Expert Comment

by:Ray Paseur
ID: 39906344
I have not checked it in detail, but it looks like the column names in the CREATE TABLE statement map 1:1 to the array keys.  Is that so and is it a dependency you're willing to live with?  If so, the mapping from the array to the query will be very easy.  If not, some sort of adapter object can be set up to create the mapping.

Regarding "overwriting duplicates," please tell us what constitutes a duplicate?  Is it one column or a set of columns?  Any clue you can give us will be helpful!  You can avoid getting duplicates in a lot of different ways; the exact way we choose will depend a little on how we define duplication.
0
 
LVL 2

Author Comment

by:nedlogan
ID: 39906372
They do map 1:1 but towards the end some are not required (MEDIA_IMAGE_05 - 60) and a few others.  

Duplicates have the same AGENT_REF so if the AGENT_REF exists in the database and in the imported array then the import would overwrite what is in the database.

Imports are on a daily basis and unlikely to be more than 100 items.

Thanks
0
 
LVL 108

Expert Comment

by:Ray Paseur
ID: 39906418
You can mark the AGENT_REF "Unique" and MySQL will throw error number 1062 when there is an attempt to insert a duplicate.  The script can trap this event and use UPDATE instead of INSERT.  We might also be able to do something with REPLACE INTO.  I'll see if I can set up some test cases to demonstrate the concepts.  Have some errands to run, so it will probably be this afternoon before I have a tested code set.
0
 
LVL 42

Accepted Solution

by:
Chris Stanyon earned 500 total points
ID: 39906454
As you say the keys in the array map to the fields in the DB, here's a quick example of how you could do it. As Ray says, you can catch the duplicate insert by setting the AGENT_REF to UNIQUE in your table, and then checking for error number 1062. The code assumes your array is stored in $data:

$dbh = new PDO("mysql:host=localhost;dbname=yourDB", "username", "password");

$columns = array_keys($data[0]);
$columnList = join(',', $columns);
$paramList = join(',', array_map(function($column) { return ":$column"; }, $columns));

$sql = $dbh->prepare("INSERT INTO yourTable ($columnList) VALUES ($paramList)");

foreach ($data as $info):
	try {
		$sql->execute($info);
	} catch(PDOException $e) {
		if ($e->errorInfo[1] == 1062) {
		//we have a duplicate record, so deal with it	
		}
	}
endforeach;

Open in new window

0
Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

 
LVL 2

Author Comment

by:nedlogan
ID: 39906463
Thanks  Chris I'm away from my desk for the next couple of hours will test on return.
0
 
LVL 51

Expert Comment

by:Julian Hansen
ID: 39906475
There are a number of ways around duplicates.

One way that might work here is to do a batch import into a duplicate of your production table and then do an

First create a duplicate of your data table call it import_table for the purposes of this exercise.

Then as part of the import porcess clear it
TRUNCATE import_table;

Open in new window


And then add your rows here using the code in my earlier post.

And finally
INSERT INOT main_table (FIELDS, ...) 
   SELECT VALUES FROM import_table 
   WHERE AGENT_REF NOT IN (SELECT AGENT_REF FROM main_table)

Open in new window


Other approaches involve first checking if a record exists before importing - and doing this row by row.
Another alternative is to use the MySQL

INSERT IGNORE ... 

Open in new window

To use this you need a unique key defined on your AGENT_REF field - any attempt to insert a duplicate record will be ignored if the key exists.

More on this here http://dev.mysql.com/doc/refman/5.6/en/insert.html
0
 
LVL 2

Author Comment

by:nedlogan
ID: 39907026
Ok, so I tried the code provided by Chris Stanyon which runs to the end with no errors, but the database remains empty.

I checked it is connecting to database ok.

If there was an error I might be able to troubleshoot, but I'm stuck again.

Thanks.
0
 
LVL 42

Expert Comment

by:Chris Stanyon
ID: 39907085
OK.

Firstly, make sure you've turned on error reporting. At the very start of your script, add this:

error_reporting(E_ALL);
ini_set('display_errors', 1);

Open in new window

Then make liberal use of var_dump() so you can visualize your data:

$columns = array_keys($data[0]);
$columnList = join(',', $columns);
$paramList = join(',', array_map(function($column) { return ":$column"; }, $columns));

var_dump($columnList);
var_dump($paramList);

foreach ($data as $info):
    var_dump($info);
endforeach;

Open in new window

The $columnList should be a comma delimited list of all your columns (grabbed from the Keys of you data array). The $paramList should match this, but each column will be preceded by a colon.

The $info should dump out each record in your array.

Run that so you can at least see what data is being used.
0
 
LVL 2

Author Comment

by:nedlogan
ID: 39907127
Ok, I made the changes, no errors and the data is in the attached file. I still can't see what where is going wrong.

Thanks/
datashown.txt
0
 
LVL 42

Expert Comment

by:Chris Stanyon
ID: 39907195
Wow. That's hard work to read, but it all looks good (as far as I can tell). One thing I forgot about - telling PDO to fire exceptions!! You need to add this line straight after the $dbh = new... line:

$dbh->setAttribute( PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION );

You could also extend the error handling to report on errors other than 1062. Update your catch block to this:

try {
	$sql->execute($info);
} catch(PDOException $e) {
	if ($e->errorInfo[1] == 1062) {
		echo "We have a duplicate Record." . PHP_EOL;
	} else {
		echo "We have some other error: " . $e->errorInfo[2] . PHP_EOL;
	}
}

Open in new window

If you still don't get any joy, post up your full code.
0
 
LVL 2

Author Comment

by:nedlogan
ID: 39907216
Ok, so the error is

"We have some other error: Unknown column 'MEDIA_IMAGE_05' in 'field list' We have some other error: Unknown column 'MEDIA_IMAGE_05' in 'field list' We have some other error: Unknown column 'MEDIA_IMAGE_05' in 'field list' "

I'll recheck the database and retry.
0
 
LVL 42

Expert Comment

by:Chris Stanyon
ID: 39907258
OK. You have a key in your data array called MEDIA_IMAGE_05 but it doesn't exist in your table. You can either remove it from the array or add it to your table.
0
 
LVL 2

Author Comment

by:nedlogan
ID: 39907540
Thanks it works now.
0
 
LVL 2

Author Closing Comment

by:nedlogan
ID: 39907544
Thanks for your help.
0

Featured Post

Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

Join & Write a Comment

Does the idea of dealing with bits scare or confuse you? Does it seem like a waste of time in an age where we all have terabytes of storage? If so, you're missing out on one of the core tools in every professional programmer's toolbox. Learn how to …
Nothing in an HTTP request can be trusted, including HTTP headers and form data.  A form token is a tool that can be used to guard against request forgeries (CSRF).  This article shows an improved approach to form tokens, making it more difficult to…
The viewer will learn how to dynamically set the form action using jQuery.
The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

22 Experts available now in Live!

Get 1:1 Help Now