Solved

How to remove the junk values from csv file

Posted on 2013-10-24
16
509 Views
Last Modified: 2013-10-24
Hi All,

I 've this csv file which looks fine when I open it, but when I reach to the bottom of this report I see this signature

Report Generated by
Report generated time
Name of the Company
Company Address

File Layout
ID, M_Name,L_Name,F_Name, Street_name,SSN,Phone1,

Records:17k approximately
I remove this junk data manually and load this file into the table [SQL SERVER 2008].

Is there any way we can write a script to remove this junk values?

Please let me know if you need any thing else?
0
Comment
Question by:parpaa
  • 9
  • 6
16 Comments
 

Author Comment

by:parpaa
ID: 39597644
Almost forget::
Once I remove this junk/signature values from csv file, I load them into tables using SSIS package. Thanks
0
 
LVL 16

Expert Comment

by:gurutc
ID: 39597687
use this command:

findstr /B  /V "Report Name Company"  filename.csv > fixedfile.csv

this command will output all lines not beginning with those search strings to the fixedfile.csv

Good Luck,

- gurutc
0
 

Author Comment

by:parpaa
ID: 39597716
awesome I will try that now :)
0
3 Use Cases for Connected Systems

Our Dev teams are like yours. They’re continually cranking out code for new features/bugs fixes, testing, deploying, testing some more, responding to production monitoring events and more. It’s complex. So, we thought you’d like to see what’s working for us.

 

Author Comment

by:parpaa
ID: 39597768
GuruTc,
unfortunately it is not excluding these values, though it created new csv file.
Anything missing here?
0
 

Author Comment

by:parpaa
ID: 39597791

"ReportName"
"Copyright (c) 2000-2013 , inc. All rights reserved."
"Confidential Information - Do Not Distribute"
"Generated By: XYZ  10/1/2013 12:35 PM"
"Name of the company"

This is something tried:
findstr /B  /V "ReportName Copyright Confidential Generated Name"  filename.csv > fixedfile.csv

but no luck
0
 
LVL 16

Expert Comment

by:gurutc
ID: 39597813
Are there any leading spaces before those strings in the file?

Another way is a multi step but could be put in a batch file:

findstr /V  /I  /C:"Report Generated by"  filename.csv > fixedfile1.csv
findstr /V  /I  /C:"Name of"  fixedfile1.csv > fixedfile2.csv
findstr /V  /I  /C:"Company Address"  fixedfile2.csv > fixedfile3.csv

This will remove the lines with these strings no matter if there are spaces or tabs or anything else before the strings.

- gurutc
0
 
LVL 16

Expert Comment

by:gurutc
ID: 39597822
Your try would work if you took out the /B which means beginning of line and add /I making it case-insensitive.

- gurutc
0
 
LVL 16

Accepted Solution

by:
gurutc earned 500 total points
ID: 39597835
findstr   /V  /I  "ReportName Copyright Confidential Generated Name"  filename.csv > fixedfile.csv
0
 
LVL 26

Expert Comment

by:Zberteoc
ID: 39597858
I think what you need is to have a list of first words from each line in the signature. Try:

findstr /B  /V "<first word from report name here> Copyright Confidential Generated <first word of the company name here>"  filename.csv > fixedfile.csv

So if the report name is:  Address List  and the company name is  ABC Inc.  then your command will be:

findstr /B  /V "Address Copyright Confidential Generated ABC"  filename.csv > fixedfile.csv
0
 

Author Comment

by:parpaa
ID: 39597883
@gurutc
"ReportName"
"Copyright (c) 2000-2013 , inc. All rights reserved."
"Confidential Information - Do Not Distribute"
"Generated By: XYZ  10/1/2013 12:35 PM"
"Name of the company"
As you see here it has double quotations at beginning of the lines. Sorry i should have told you this before.
0
 

Author Closing Comment

by:parpaa
ID: 39597901
Thank you so much Guru.. this one has removed the values. Closing this out.
0
 

Author Comment

by:parpaa
ID: 39598009
Hi @gurutc,

If am using /I /V it is excluding many valid records. Is there any other way to remove these double quotes of each column and use

findstr /B /V ..

I know this ticket has been closed. Do you want me to open a new ticket for this request?
0
 

Author Comment

by:parpaa
ID: 39598018
setlocal enabledelayedexpansion
if exist after.csv del after.csv
for /f "delims=" %%A in (before.csv) do (
  set csvline=%%A
  echo !csvline:"=! >> after.csv
  )

Open in new window


I'm using this code to trim the double quotes.. but it is not working.
0
 
LVL 16

Expert Comment

by:gurutc
ID: 39598116
When I get back to my PC we"ll solve this

-gurutc
0
 
LVL 16

Expert Comment

by:gurutc
ID: 39598328
Hi,

Use the \ directive to escape the double quotes and treat it as literal:

findstr   /V   /B "\"ReportName \"Copyright \"Confidential \"Generated \"Name"  filename.csv > fixedfile.csv

You only have to escape the quotes that are at the beginning of each of your search terms so those quote marks also become part of what's being searched for.  

So only the lines starting with the following will be recognized and left out:

"ReportName
"Copyright
"Confidential
"Generated
"Name

If there's anything hidden before the " mark you'll have to remove the /B which means beginning of line, but let's count on our hoping there isn't and this'll work groovy.

- gurutc
0
 

Author Comment

by:parpaa
ID: 39599406
Thank you so much again guru :)
0

Featured Post

3 Use Cases for Connected Systems

Our Dev teams are like yours. They’re continually cranking out code for new features/bugs fixes, testing, deploying, testing some more, responding to production monitoring events and more. It’s complex. So, we thought you’d like to see what’s working for us.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Vbscript 8 88
Running VB/Batch script through Group policy 30 101
regedit.exe REG ADD problems 2 40
Carriage Return in Batch File 5 51
VALIDATING DATES One method of validating dates is to jam the date into the DATE command and see if it accepts it by examining the system's errorlevel value. A non-zero result indicates failure. A typical example might look something like the fol…
You may have already been in the need to update a whole folder stucture using a script. Robocopy does it well and even provides a list of non-updated files in a log (if asked to). Generally those files that were locked by a user or a process by the …
Two types of users will appreciate AOMEI Backupper Pro: 1 - Those with PCIe drives (and haven't found cloning software that works on them). 2 - Those who want a fast clone of their boot drive (no re-boots needed) and it can clone your drive wh…
Although Jacob Bernoulli (1654-1705) has been credited as the creator of "Binomial Distribution Table", Gottfried Leibniz (1646-1716) did his dissertation on the subject in 1666; Leibniz you may recall is the co-inventor of "Calculus" and beat Isaac…

785 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question