Solved

Add Array data to database

Posted on 2014-03-05
20
273 Views
Last Modified: 2014-03-05
Hi,
I have an array of data from an external file which I want to import into my database. So far I can print the ARRAY to the screen.

I have attached the array data in a file (this is how it looks when I print it to the screen) which shows an example with three entries. When I import it into my database I want it to overwrite any duplicates.

I presume I need a foreach loop to read and insert the data?

I'm using PHP / MySQL.

Thanks,
nl
0
Comment
Question by:nedlogan
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 9
  • 5
  • 4
  • +1
20 Comments
 
LVL 110

Expert Comment

by:Ray Paseur
ID: 39906224
attached the array data in a file
Erm?
0
 
LVL 2

Author Comment

by:nedlogan
ID: 39906232
-Thanks for spotting my mistake.
arraydata.txt
0
 
LVL 110

Expert Comment

by:Ray Paseur
ID: 39906233
When you get the data posted here, I'll be glad to take a look and see if I can show you what you might want to do with the PHP programming.  While I am doing that, you might find some value in this article, if you're new to PHP and MySQL programming.
0
[Live Webinar] The Cloud Skills Gap

As Cloud technologies come of age, business leaders grapple with the impact it has on their team's skills and the gap associated with the use of a cloud platform.

Join experts from 451 Research and Concerto Cloud Services on July 27th where we will examine fact and fiction.

 
LVL 110

Expert Comment

by:Ray Paseur
ID: 39906241
OK, got it.  Going forward you may find that something like this will make it easier to read the output of print_r().  Preformatting will preserve the indentation and line breaks.

echo '<pre>';
print_r($thing);

Open in new window

Please post the CREATE TABLE statement for the data base table that you will use to import this information, thanks.
0
 
LVL 58

Expert Comment

by:Julian Hansen
ID: 39906267
Here is one way to do it. It assumes
a) field names in the database are the same as in the array
b) array values are stored in variable $array
// string to hold fieldnames
$fields = ''; 
// string to hold values
$values = " VALUES ";
foreach($array as $item) {
  $val = '';
  foreach ($item as $f => $v) {
     $fields = $fields . "`$f`,";
     $val = $val . "'$v',"
  }
  $values = $values . "($val),"
}
$fields = trim($fields, ',');
$values = trim($values, ',');
$query = trim($query, ',');
// Build query here
$query = "INSERT INTO table ($fields) $values";

Open in new window

0
 
LVL 2

Author Comment

by:nedlogan
ID: 39906304
I've attached the sql used to create the table. Nearly all the fields are included in the table. Some of the image fields near the end of the file are excluded.

In answer to julianH: does your code require the fields in the database to have the same name as the array items, does there need to be the same number of fields to array items.

Also in terms of overwriting duplicates what how will the code handle that?

Thanks!
sqltable.txt
0
 
LVL 110

Expert Comment

by:Ray Paseur
ID: 39906344
I have not checked it in detail, but it looks like the column names in the CREATE TABLE statement map 1:1 to the array keys.  Is that so and is it a dependency you're willing to live with?  If so, the mapping from the array to the query will be very easy.  If not, some sort of adapter object can be set up to create the mapping.

Regarding "overwriting duplicates," please tell us what constitutes a duplicate?  Is it one column or a set of columns?  Any clue you can give us will be helpful!  You can avoid getting duplicates in a lot of different ways; the exact way we choose will depend a little on how we define duplication.
0
 
LVL 2

Author Comment

by:nedlogan
ID: 39906372
They do map 1:1 but towards the end some are not required (MEDIA_IMAGE_05 - 60) and a few others.  

Duplicates have the same AGENT_REF so if the AGENT_REF exists in the database and in the imported array then the import would overwrite what is in the database.

Imports are on a daily basis and unlikely to be more than 100 items.

Thanks
0
 
LVL 110

Expert Comment

by:Ray Paseur
ID: 39906418
You can mark the AGENT_REF "Unique" and MySQL will throw error number 1062 when there is an attempt to insert a duplicate.  The script can trap this event and use UPDATE instead of INSERT.  We might also be able to do something with REPLACE INTO.  I'll see if I can set up some test cases to demonstrate the concepts.  Have some errands to run, so it will probably be this afternoon before I have a tested code set.
0
 
LVL 43

Accepted Solution

by:
Chris Stanyon earned 500 total points
ID: 39906454
As you say the keys in the array map to the fields in the DB, here's a quick example of how you could do it. As Ray says, you can catch the duplicate insert by setting the AGENT_REF to UNIQUE in your table, and then checking for error number 1062. The code assumes your array is stored in $data:

$dbh = new PDO("mysql:host=localhost;dbname=yourDB", "username", "password");

$columns = array_keys($data[0]);
$columnList = join(',', $columns);
$paramList = join(',', array_map(function($column) { return ":$column"; }, $columns));

$sql = $dbh->prepare("INSERT INTO yourTable ($columnList) VALUES ($paramList)");

foreach ($data as $info):
	try {
		$sql->execute($info);
	} catch(PDOException $e) {
		if ($e->errorInfo[1] == 1062) {
		//we have a duplicate record, so deal with it	
		}
	}
endforeach;

Open in new window

0
 
LVL 2

Author Comment

by:nedlogan
ID: 39906463
Thanks  Chris I'm away from my desk for the next couple of hours will test on return.
0
 
LVL 58

Expert Comment

by:Julian Hansen
ID: 39906475
There are a number of ways around duplicates.

One way that might work here is to do a batch import into a duplicate of your production table and then do an

First create a duplicate of your data table call it import_table for the purposes of this exercise.

Then as part of the import porcess clear it
TRUNCATE import_table;

Open in new window


And then add your rows here using the code in my earlier post.

And finally
INSERT INOT main_table (FIELDS, ...) 
   SELECT VALUES FROM import_table 
   WHERE AGENT_REF NOT IN (SELECT AGENT_REF FROM main_table)

Open in new window


Other approaches involve first checking if a record exists before importing - and doing this row by row.
Another alternative is to use the MySQL

INSERT IGNORE ... 

Open in new window

To use this you need a unique key defined on your AGENT_REF field - any attempt to insert a duplicate record will be ignored if the key exists.

More on this here http://dev.mysql.com/doc/refman/5.6/en/insert.html
0
 
LVL 2

Author Comment

by:nedlogan
ID: 39907026
Ok, so I tried the code provided by Chris Stanyon which runs to the end with no errors, but the database remains empty.

I checked it is connecting to database ok.

If there was an error I might be able to troubleshoot, but I'm stuck again.

Thanks.
0
 
LVL 43

Expert Comment

by:Chris Stanyon
ID: 39907085
OK.

Firstly, make sure you've turned on error reporting. At the very start of your script, add this:

error_reporting(E_ALL);
ini_set('display_errors', 1);

Open in new window

Then make liberal use of var_dump() so you can visualize your data:

$columns = array_keys($data[0]);
$columnList = join(',', $columns);
$paramList = join(',', array_map(function($column) { return ":$column"; }, $columns));

var_dump($columnList);
var_dump($paramList);

foreach ($data as $info):
    var_dump($info);
endforeach;

Open in new window

The $columnList should be a comma delimited list of all your columns (grabbed from the Keys of you data array). The $paramList should match this, but each column will be preceded by a colon.

The $info should dump out each record in your array.

Run that so you can at least see what data is being used.
0
 
LVL 2

Author Comment

by:nedlogan
ID: 39907127
Ok, I made the changes, no errors and the data is in the attached file. I still can't see what where is going wrong.

Thanks/
datashown.txt
0
 
LVL 43

Expert Comment

by:Chris Stanyon
ID: 39907195
Wow. That's hard work to read, but it all looks good (as far as I can tell). One thing I forgot about - telling PDO to fire exceptions!! You need to add this line straight after the $dbh = new... line:

$dbh->setAttribute( PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION );

You could also extend the error handling to report on errors other than 1062. Update your catch block to this:

try {
	$sql->execute($info);
} catch(PDOException $e) {
	if ($e->errorInfo[1] == 1062) {
		echo "We have a duplicate Record." . PHP_EOL;
	} else {
		echo "We have some other error: " . $e->errorInfo[2] . PHP_EOL;
	}
}

Open in new window

If you still don't get any joy, post up your full code.
0
 
LVL 2

Author Comment

by:nedlogan
ID: 39907216
Ok, so the error is

"We have some other error: Unknown column 'MEDIA_IMAGE_05' in 'field list' We have some other error: Unknown column 'MEDIA_IMAGE_05' in 'field list' We have some other error: Unknown column 'MEDIA_IMAGE_05' in 'field list' "

I'll recheck the database and retry.
0
 
LVL 43

Expert Comment

by:Chris Stanyon
ID: 39907258
OK. You have a key in your data array called MEDIA_IMAGE_05 but it doesn't exist in your table. You can either remove it from the array or add it to your table.
0
 
LVL 2

Author Comment

by:nedlogan
ID: 39907540
Thanks it works now.
0
 
LVL 2

Author Closing Comment

by:nedlogan
ID: 39907544
Thanks for your help.
0

Featured Post

Get 15 Days FREE Full-Featured Trial

Benefit from a mission critical IT monitoring with Monitis Premium or get it FREE for your entry level monitoring needs.
-Over 200,000 users
-More than 300,000 websites monitored
-Used in 197 countries
-Recommended by 98% of users

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This post looks at MongoDB and MySQL, and covers high-level MongoDB strengths, weaknesses, features, and uses from the perspective of an SQL user.
In this blog post, we’ll look at how ClickHouse performs in a general analytical workload using the star schema benchmark test.
The viewer will learn how to dynamically set the form action using jQuery.
The viewer will learn how to look for a specific file type in a local or remote server directory using PHP.

632 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question