קומר קיראן
asked on
Find the Empty Columns in the CSV file
Requirement :
In CSV file I want to check three scenarios.
First : Find the Columns which have Header Empty and Data in the Column and rename the Column Header as Temporary Header
Second : Find the Columns which has Header but not the data in the Column (do nothing to these columns)
Third : Find the Columns with No Data at all and delete
Finally Convert CSV to XLsx format and save the file name with Time stamp like Filename_YYYY_MM_DD_HHmm (processed time)
Delete.PNG
In CSV file I want to check three scenarios.
First : Find the Columns which have Header Empty and Data in the Column and rename the Column Header as Temporary Header
Second : Find the Columns which has Header but not the data in the Column (do nothing to these columns)
Third : Find the Columns with No Data at all and delete
Finally Convert CSV to XLsx format and save the file name with Time stamp like Filename_YYYY_MM_DD_HHmm (processed time)
Delete.PNG
Ok, simple. :-)
# requires ImportExcel
# can be downloaded here: https://github.com/dfinke/ImportExcel
Import-Module ImportExcel
$ImportFile = ".\test.csv"
$Header = Get-Content $ImportFile -First 1
$Headerarray = $header.Split(",")
# Build header
for ($i=0;$i -lt $Headerarray.Length; $i++) {
If($Headerarray[$i] -eq "") {
$Headerarray[$i] = "TempHeader$i"
}
}
$PropertyNames = @()
$CSVImport = Get-Content $ImportFile | Select-Object -Skip 1 | ConvertFrom-Csv -Header $Headerarray
$Headerarray | % {
if(($CSVImport.$_ | Where-Object {$_ -ne $null}).Count -eq 0) {
"$_ is empty"
} else {
$PropertyNames += $_
}
}
$CSVImport | Select -Property $PropertyNames | Export-Excel "Filename_$(Get-date -Format yyyyMMddHHmm).xlsx" -Show
Not having an example of such a .csv file with these issues, I can think a few things to check before deciding that you really have these issues with the file. I have seen many forum postings like this by folks who did not have a true .csv file or didn't use the correct parameters to read it.
Don't assume that the parameters used to create the file are what you would consider the "default" ones for a so-called ".csv" file. Improperly interpreted .csv files can look like they have missing headers or missing columns, etc. I've seen a few ".csv" files that were not properly interpreted because the correct parameters were not used to read the file. Such things as delimiter characters not properly designated, correct column separation characters not used (fixed width instead of tab, or comma instead of pipe, etc.) can skew the lineup.
This is especially true if the file was exported by software, as it should have all headings and formatted appropriately as programmed. It would look proper if it is read using the same formatting parameters as the ones used to export it.
However, just because the file extension is ".csv" doesn't mean its really formatted as a properly generated .csv file (someone just slapped a ".csv" extension on a text file). If it's not a true ".csv" file and just thrown together some how and not created/exported by a proper .csv generation software, then... who knows what you have... or how to fix it....
Don't assume that the parameters used to create the file are what you would consider the "default" ones for a so-called ".csv" file. Improperly interpreted .csv files can look like they have missing headers or missing columns, etc. I've seen a few ".csv" files that were not properly interpreted because the correct parameters were not used to read the file. Such things as delimiter characters not properly designated, correct column separation characters not used (fixed width instead of tab, or comma instead of pipe, etc.) can skew the lineup.
This is especially true if the file was exported by software, as it should have all headings and formatted appropriately as programmed. It would look proper if it is read using the same formatting parameters as the ones used to export it.
However, just because the file extension is ".csv" doesn't mean its really formatted as a properly generated .csv file (someone just slapped a ".csv" extension on a text file). If it's not a true ".csv" file and just thrown together some how and not created/exported by a proper .csv generation software, then... who knows what you have... or how to fix it....
Question for you... why the SQL and Query Syntax topic areas ? Why not Excel or .Net ?
ASKER
Initially I had a requirement Call all the parameters like Source, Destination , Filename and File extension from Sql table. That's the reason I had SQL areas.
ASKER
Because Everyone was Viewing and not answering and i got an email after 3-4 hours to modify the question and update again so that's the reason
ASKER
Import-Module : The specified module 'ImportExcel' was not loaded because no valid module file was found in any module directory.
You can either install it using "Install-Module ImportExcel" command or download it (from github).
This link might help you https://devblogs.microsoft.com/scripting/introducing-the-powershell-excel-module-2/
And : https://www.powershellgallery.com/packages/ImportExcel/5.4.2 has the full list of different versions and how to install. Make sure you do get the Current Version :)
Hope that helps you to get started.... The posts above by Michael Pfister are pretty good too :)
This link might help you https://devblogs.microsoft.com/scripting/introducing-the-powershell-excel-module-2/
And : https://www.powershellgallery.com/packages/ImportExcel/5.4.2 has the full list of different versions and how to install. Make sure you do get the Current Version :)
Hope that helps you to get started.... The posts above by Michael Pfister are pretty good too :)
ASKER
Can someone get me the output without converting it to Excel Format only to CSV
$CSVImport | Select -Property $PropertyNames | Export-CSV "Filename_$(Get-date -Format yyyyMMddHHmm).csv" -NoClobber
will output the generated table to a CSV fileIf you want to add the SQL connection have a look here:
https://gallery.technet.mi
Little bit confused now....
Are you already exporting your Table from SQL into CSV ? And want to manipulate that CSV file ?
Or, that is the real challenge. Meaning you need a means of exporting from SQL to create CSV ?
If you already have the CSV, as Mark Edwards says above, it is just a text file and largely unstructured, so, it often requires a slightly more formal structure (or at least a means of understanding a more formal structure) to be able to inspect the individual elements such as rows and columns.
Are you already exporting your Table from SQL into CSV ? And want to manipulate that CSV file ?
Or, that is the real challenge. Meaning you need a means of exporting from SQL to create CSV ?
If you already have the CSV, as Mark Edwards says above, it is just a text file and largely unstructured, so, it often requires a slightly more formal structure (or at least a means of understanding a more formal structure) to be able to inspect the individual elements such as rows and columns.
ASKER
Select-Object : Cannot convert System.Management.Automati on.PSObjec t to one of the following types {System.String, System.Management.Automati on.ScriptB lock}.
I've found just one problem, I forgot to disable the type information Export-CSV sets in the CSV.
The rest works during my tests, no errors.
Which line is generating the error?
.
Can you attach a sample of the input CSV to reproduce the error?
The rest works during my tests, no errors.
Which line is generating the error?
.
$CSVImport | Select -Property $PropertyNames | Export-Csv -NoTypeInformation -NoClobber "Filename_$(Get-date -Format yyyyMMddHHmm).csv"
Can you attach a sample of the input CSV to reproduce the error?
ASKER
Everything is working good With this code only one issue its not checking whether the column is empty or having data Please find the attached file.
Error1.PNG
Error1.PNG
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Perfect
Open in new window
This solves 1st and 2nd requirement.
For the 3rd requirement I haven't found a nice solution yet.