Solved

Need help updating custom PHP script for counting duplicate emails in 2 uploaded CSV files

Posted on 2014-11-24
10
159 Views
Last Modified: 2014-11-28
I wrote a custom PHP script a while back which allowed the user to upload 2 different CSV files, and it would then compare the 2 files and display a list of all of the duplicate email addresses that it found.  I've recently been asked to apply some modifications to this script, and was wondering if someone here might be able to help.

Currently, when the script runs, it displays the number of duplicates at the bottom of the page, --- but it counts multiple emails in that total.  So if the same email address is on both lists 5 times (and that can happen) the total is increased by 5.  

What I need to do is find a way to calculate and display a total that counts the unique number of duplicated emails .  In other words -- if the same email address is on both lists 5 times .. it would be counted as 1 duplicate email .. NOT 5.

How would I accomplish this?  My code is below:

<?php 
	
	if (isset($_POST['UPLOAD'])) { 
			
		$csv1filename = $_FILES['csv1']['tmp_name'];

		$ext1 = strtoupper(pathinfo($_FILES['csv1']['name'], PATHINFO_EXTENSION));								
		
		$csv2filename = $_FILES['csv2']['tmp_name'];
		
		$ext2 = strtoupper(pathinfo($_FILES['csv2']['name'], PATHINFO_EXTENSION));				

		if ($ext1 == 'CSV' && $ext2 == 'CSV')	{	

			$csv1 = '/home/expert/public_html/dupefinder/csv1.csv';					

			move_uploaded_file($csv1filename, $csv1);

			$csv2 = '/home/expert/public_html/dupefinder/csv2.csv';	

			move_uploaded_file($csv2filename, $csv2);					

			// CREATE THE EMPTY ARRAY
			
			$raw_array = array();				
			
			$csv1 = file($csv1, FILE_IGNORE_NEW_LINES + FILE_SKIP_EMPTY_LINES);
			
			$csv2 = file($csv2, FILE_IGNORE_NEW_LINES + FILE_SKIP_EMPTY_LINES);

			$raw_array = array_merge($csv1, $csv2); 			
			
			// FUNCTION TO COUNT DUPLICATE EMAILS IN THE ARRAY
			
			function array_not_unique($raw) {
				$new = array_count_values($raw);				
				foreach ($new as $key => $val) {
				   if ($val < 2) unset($new[$key]);
				}				
				return $new;				
			}		

			$common = array_not_unique($raw_array);
			
			// DISPLAY THE NUMBER OF DUPES FOUND IN THE ARRAY
			
			$total = 0;
			
			echo "<pre style='text-align:left;line-height:45px;'>";	

			foreach ($common as $x => $n) {
			
				echo PHP_EOL . "* THE VALUE <strong style='color:red;'>$x</strong> APPEARED <span style='background-color:#FBCB45;padding:3px;'>$n</span> TIMES";
				
				$total = $total + $n;
				
			}	
			
			echo "</pre>";
			
			echo "<br />A Total of <strong style='text-align:center;background-color:#FBCB45;padding:3px;'>$total</strong> duplicate email addresses were found.<br /><br /><br />";
			
			echo '<a href="/dupefinder/" style="font-size:18px;text-decoration:none;font-weight:bold;color:blue;">&lt;&lt; Back</a>';
			
			unlink('/home/expert/public_html/dupefinder/csv1.csv');
			
			unlink('/home/expert/public_html/dupefinder/csv2.csv');							
			
		}	
	
	} 

?>

Open in new window


Thanks,
- Yvan
0
Comment
Question by:egoselfaxis
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
  • 2
  • +2
10 Comments
 
LVL 62

Expert Comment

by:gheist
ID: 40463884
Once you load CSV files into SQL(ite) database...
0
 
LVL 50

Expert Comment

by:Steve Bink
ID: 40463892
Change line 55 to read:
$total++;

Open in new window

0
 
LVL 17

Expert Comment

by:Shinesh Premrajan
ID: 40464067
foreach ($common as $x => $n) {
                  
                        echo PHP_EOL . "* THE VALUE <strong style='color:red;'>$x</strong> APPEARED <span style='background-color:#FBCB45;padding:3px;'>$n</span> TIMES";
                        
                        $total++;
                        
                  }
0
Space-Age Communications Transitions to DevOps

ViaSat, a global provider of satellite and wireless communications, securely connects businesses, governments, and organizations to the Internet. Learn how ViaSat’s Network Solutions Engineer, drove the transition from a traditional network support to a DevOps-centric model.

 
LVL 110

Expert Comment

by:Ray Paseur
ID: 40464241
Please show us some test data and we can give you a complete, tested and working example.
0
 

Author Comment

by:egoselfaxis
ID: 40464535
Some sample data / CSV files are attached -- the contain email addresses only.

- Yvan
csv1.csv
csv2.csv
0
 
LVL 50

Expert Comment

by:Steve Bink
ID: 40464684
Proof of fix.  Note that I edited the output to not display the email addresses.
<?php
// -------------------------------------------
// Contents of ee-test.php
// I edited the file to:
//   - enable running as command line, not a web request
//   - provide static filenames (no POST data to use)
//   - remove empty blank lines
//   - prevent deletion of the source files 
//   - implement the fix I described in #40463892
// -------------------------------------------
/*if (isset($_POST['UPLOAD'])) {
  $csv1filename = $_FILES['csv1']['tmp_name'];
  $ext1 = strtoupper(pathinfo($_FILES['csv1']['name'], PATHINFO_EXTENSION));
  $csv2filename = $_FILES['csv2']['tmp_name'];
  $ext2 = strtoupper(pathinfo($_FILES['csv2']['name'], PATHINFO_EXTENSION));
  if ($ext1 == 'CSV' && $ext2 == 'CSV')	{
    $csv1 = '/home/expert/public_html/dupefinder/csv1.csv';
    move_uploaded_file($csv1filename, $csv1);

    $csv2 = '/home/expert/public_html/dupefinder/csv2.csv';
    move_uploaded_file($csv2filename, $csv2);*/
    $csv1 = 'c:\bb\csv1.csv';
    $csv2 = 'c:\bb\csv2.csv';
    // CREATE THE EMPTY ARRAY
    $raw_array = array();
    $csv1 = file($csv1, FILE_IGNORE_NEW_LINES + FILE_SKIP_EMPTY_LINES);
    $csv2 = file($csv2, FILE_IGNORE_NEW_LINES + FILE_SKIP_EMPTY_LINES);

    $raw_array = array_merge($csv1, $csv2); 							// FUNCTION TO COUNT DUPLICATE EMAILS IN THE ARRAY
    function array_not_unique($raw) {
      $new = array_count_values($raw);
      foreach ($new as $key => $val) {
        if ($val < 2) unset($new[$key]);
      }
      return $new;
    }
    $common = array_not_unique($raw_array);
    // DISPLAY THE NUMBER OF DUPES FOUND IN THE ARRAY
    $total = 0;
    echo "<pre style='text-align:left;line-height:45px;'>";
    foreach ($common as $x => $n) {
      echo PHP_EOL . "* THE VALUE <strong style='color:red;'>$x</strong> APPEARED <span style='background-color:#FBCB45;padding:3px;'>$n</span> TIMES";
      /* $total = $total + $n;*/
      $total++;
    }
    echo "</pre>";
    echo "<br />A Total of <strong style='text-align:center;background-color:#FBCB45;padding:3px;'>$total</strong> duplicate email addresses were found.<br /><br /><br />";
    echo '<a href="/dupefinder/" style="font-size:18px;text-decoration:none;font-weight:bold;color:blue;">&lt;&lt; Back</a>';
/*    unlink('/home/expert/public_html/dupefinder/csv1.csv');
    unlink('/home/expert/public_html/dupefinder/csv2.csv');
  }
}*/
?>

C:\bb\www>php ee-test.php
<pre style='text-align:left;line-height:45px;'>* THE VALUE <strong style='color:red;'>email1@example.com</strong> APPEARED <span style='background-color:#FBCB45;padding:3px;'>2</span> TIMES* THE VALUE <strong style='color:red;'>email2@example.com</strong> APPEARED <span style='background-color:#FBCB45;padding:3px;'>2</span> TIMES* THE VALUE <strong style='color:red;'>email3@example.com</strong> APPEARED <span style='background-color:#FBCB45;padding:3px;'>2</span> TIMES</pre><br />A Total of <strong style='text-align:center;background-color:#FBCB45;padding:3px;'>3</strong> duplicate email addresses were found.<br /><br /><br /><a href="/dupefinder/" style="font-size:18px;text-decoration:none;font-weight:bold;color:blue;">&lt;&lt; Back</a>

C:\bb\www>

Open in new window

0
 

Author Comment

by:egoselfaxis
ID: 40464695
Steve -- will this modified PHP script of yours ONLY run via the command line?
Or can I still run it as a web-based app?

- Yvan
0
 
LVL 50

Accepted Solution

by:
Steve Bink earned 500 total points
ID: 40465086
If you implement the single change I suggested in #40463892, the script will continue to work exactly as it does now, only with the count correction you requested.  The changes I made for the purpose of my demonstration were solely to work around the lack of a web environment.  The only change that really affected the core functionality is the "total++" modification (line 55 in your original code, lines 43-44 in my demonstration code).
0
 
LVL 110

Expert Comment

by:Ray Paseur
ID: 40465965
Start your array learning adventures here:
http://php.net/manual/en/ref.array.php

This script shows a few ways of looking at arrays:
http://iconoun.com/demo/temp_egoselfaxis.php

<?php // demo/temp_egoselfaxis.php
error_reporting(E_ALL);
echo '<pre>';

// SEE http://www.experts-exchange.com/Programming/Languages/Scripting/PHP/Q_28568673.html
// REF http://php.net/manual/en/function.array-unique.php
// REF http://php.net/manual/en/function.array-unique.php#81513
// REF http://php.net/manual/en/function.array-intersect.php

// TEST DATA FROM THE POST AT EE
$url1 = 'http://filedb.experts-exchange.com/incoming/2014/11_w48/884897/csv1.csv';
$csv1 = file_get_contents($url1);
$url2 = 'http://filedb.experts-exchange.com/incoming/2014/11_w48/884898/csv2.csv';
$csv2 = file_get_contents($url2);

// CHECK THE TEST DATA
echo PHP_EOL . htmlentities($csv1);
echo PHP_EOL;
echo PHP_EOL . htmlentities($csv2);
echo PHP_EOL;

// TRANSFORM THE CSV FILES INTO NORMALIZED ARRAYS (THERE ARE LOTS OF WAYS TO DO THIS)
$arr1 = explode(PHP_EOL, $csv1);
foreach ($arr1 as $key => $str) { $arr1[$key] = trim(strtolower($str)); }
$arr2 = explode(PHP_EOL, $csv2);
foreach ($arr2 as $key => $str) { $arr2[$key] = trim(strtolower($str)); }

// INSPECT THE NORMALIZED ARRAYS
var_dump($arr1, $arr2);

// FIND ALL OF THE EMAILS IN ARRAY 1 THAT ARE IN ARRAY 2
$new1 = array_intersect($arr1, $arr2);
echo PHP_EOL . "THESE EMAILS IN ARRAY 1 ARE ALSO IN ARRAY 2: ";
print_r($new1);

Open in new window

0
 

Author Closing Comment

by:egoselfaxis
ID: 40470712
Thanks Steve!  

- Yvan
0

Featured Post

Online Training Solution

Drastically shorten your training time with WalkMe's advanced online training solution that Guides your trainees to action. Forget about retraining and skyrocket knowledge retention rates.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Developer portfolios can be a bit of an enigma—how do you present yourself to employers without burying them in lines of code?  A modern portfolio is more than just work samples, it’s also a statement of how you work.
3 proven steps to speed up Magento powered sites. The article focus is on optimizing time to first byte (TTFB), full page caching and configuring server for optimal performance.
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …
Video by: Mark
This lesson goes over how to construct ordered and unordered lists and how to create hyperlinks.

696 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question