How to combine two CSV files with different column headers in powershell

I am looking to combine two CSV files that have different column headers in powershell.  Some are the same but others are not.  I need all data from each sheet to fall into the correct columns.  No data should be excluded.  All columns should remain after the merge.

CSV1:
CSV1.png
CSV2:
CSV2.png
What I need for final output:
CSVFinal.png
ravi doshiAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

oBdACommented:
Assumes that 'Source Key' is the common key, and that it is unique per table.
Will write warnings (and list them in an additional column 'Conflicts') if there's a matching 'Source Key', but one of the common fields differs.
$csvLeft =	'C:\Temp\users1.csv'
$csvRight =	'C:\Temp\users2.csv'
$csvOut =	'C:\Temp\users_out.csv'

$joinOn = 'Source Key'
$commonProps =	@('First Name', 'Last Name', 'Work Facility Location', 'Work Address 1', 'Work City', 'Work State', 'Work Zip', 'Work Country')
$allProps =		@('Source Key') + $commonProps + @('Work Floor', 'Work Building', 'Device 1 Type', 'Device 1 Address', 'Conflicts')
$lookup = @{}
Import-Csv -Path $csvRight | ForEach-Object {$lookup[$_.$joinOn] = $_}
$(
	Import-Csv -Path $csvLeft | ForEach-Object {
		$conflicts = @()
		If ($lookup.ContainsKey($_.$joinOn)) {
			ForEach ($prop in $commonProps) {
				If ($_.$prop -ne $lookup[$_.$joinOn].$prop) {
					$conflicts += $prop
					Write-Warning "Key '$($_.$joinOn)': conflict in column '$($prop)'!"
				}
			}
			If ($conflicts) {
				$_, $lookup[$_.$joinOn] | Select-Object -Property *, @{n='Conflicts'; e={$conflicts -join '; '}}
			} Else {
				$_ | Select-Object -Property *, @{n='Device 1 Type'; e={$lookup[$_.$joinOn].'Device 1 Type'}}, @{n='Device 1 Address'; e={$lookup[$_.$joinOn].'Device 1 Address'}}
			}
			$lookup.Remove($_.$joinOn)
		} Else {
			$_
		}
	}
	$lookup.Values
) | Select-Object -Property $allProps | Export-Csv -NoTypeInformation -Path $csvOut

Open in new window

0
ravi doshiAuthor Commented:
This worked but could you please help me understand what lines 22 & 23 accomplish?  The screenshots I included were just a few of the columns so I added the rest of the fields to the common properties and all properties sections and was expecting to need to do the same below but I didn't have to modify line 23.  Why?
0
ravi doshiAuthor Commented:
Here in the final code that worked for me:

$csvLeft =	'D:\RD_Test\left.csv'
$csvRight =	'D:\RD_Test\right.csv'
$csvOut =	'D:\RD_Test\final.csv'

$joinOn = 'Source Key'
$commonProps =	@('First Name', 'Last Name', 'Employee ID', 'Job Title', 'Company', 'Time Zone', 'Division', 'Language', 'Work Facility Location', 'Work Address 1', 'Work City', 'Work State', 'Work Zip', 'Work Country', 'Device 1 Type', 'Device 1 Address', 'Device 1 Description', 'Group 1', 'Group 2')
$allProps =		@('Source Key') + $commonProps + @('Login ID', 'Work Floor', 'Work Building', 'Device 2 Type', 'Device 2 Address', 'Device 2 Description', 'Device 4 Type', 'Device 4 Address', 'Device 4 Description', 'Conflicts')
$lookup = @{}
Import-Csv -Path $csvRight | ForEach-Object {$lookup[$_.$joinOn] = $_}
$(
	Import-Csv -Path $csvLeft | ForEach-Object {
		$conflicts = @()
		If ($lookup.ContainsKey($_.$joinOn)) {
			ForEach ($prop in $commonProps) {
				If ($_.$prop -ne $lookup[$_.$joinOn].$prop) {
					$conflicts += $prop
					Write-Warning "Key '$($_.$joinOn)': conflict in column '$($prop)'!"
				}
			}
			If ($conflicts) {
				$_, $lookup[$_.$joinOn] | Select-Object -Property *, @{n='Conflicts'; e={$conflicts -join '; '}}
			} Else {
				$_ | Select-Object -Property *, @{n='Device 1 Type'; e={$lookup[$_.$joinOn].'Device 1 Type'}}, @{n='Device 1 Address'; e={$lookup[$_.$joinOn].'Device 1 Address'}}
			}
			$lookup.Remove($_.$joinOn)
		} Else {
			$_
		}
	}
	$lookup.Values
) | Select-Object -Property $allProps | Export-Csv -NoTypeInformation -Path $csvOut

Open in new window

0
Creating Active Directory Users from a Text File

If your organization has a need to mass-create AD user accounts, watch this video to see how its done without the need for scripting or other unnecessary complexities.

oBdACommented:
That can't really have produced the correct results, unless you have no "Source Key" that is identical in both tables, that is, you have no rows that can be merged.
Lines 21 and 23 handle the case where 'Source Key' is the same in both tables; if you don't have that situation at all, that is, if you have two totally independent tables that just differ in a few columns, the script can be a lot easier.
21 handles the case where "Source Key" is the same in both tables, but one of the common fields differs. In this case, both rows will be returned, with the conflicting fields added.
23 handles the case where "Source Key" is the same in both tables, and the rows can be joined by adding the contents of the columns that are only in the "right" table to the current "left" row (that's what the calculated properties - @{n='...'; e={}} - do).
23 requires one calculated property for each "right" column that is missing in the "left" table.
0
ravi doshiAuthor Commented:
Thanks for the explanation.  No source key is the same in either file.  One file is pulled from our HR system and the other is pulled from our Active Directory.  There is no overlap in either file - completely different set of users.  The only issue was that there are different columns in each file.  Please post the simpler version, just for my knowledge, if it's not too much trouble.
0
oBdACommented:
$allProps contains all the properties you want to have in the final csv, in the order in which they should appear (if that matters).
$csvLeft =	'D:\RD_Test\left.csv'
$csvRight =	'D:\RD_Test\right.csv'
$csvOut =	'D:\RD_Test\final.csv'

$allProps = @(
	'Source Key', 'Login ID', 'First Name', 'Last Name', 'Employee ID', 'Job Title', 'Company', 'Time Zone', 'Division', 'Language', 'Group 1', 'Group 2',
	'Work Facility Location', 'Work Address 1', 'Work City', 'Work State', 'Work Zip', 'Work Country', 'Work Floor', 'Work Building',
	'Device 1 Type', 'Device 1 Address', 'Device 1 Description', 'Device 2 Type', 'Device 2 Address', 'Device 2 Description', 'Device 4 Type', 'Device 4 Address', 'Device 4 Description'
)
$(
	Import-Csv -Path $csvLeft
	Import-Csv -Path $csvRight
) | Select-Object -Property $allProps | Export-Csv -NoTypeInformation -Path $csvOut

Open in new window

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
ravi doshiAuthor Commented:
Thanks for all the explanations.  I don't like to just copy code.  Understanding how it works is important to me!
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Powershell

From novice to tech pro — start learning today.