[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 860
  • Last Modified:

Batch - Strip <CR> from multiple lines in text file

Hi All,

I am writing a batch script that uses Windows ftp client to connect to our FTP server and does a directory listing and outputs it to a file. The output is very dirty though, so I then do a FINDSTR on the file to get only the lines I am interested in, and then a FOR loop on that to get the cleanest version I can in order to output to screen (displays a summary of the files that are on the server).

The problem is that the FTP server runs on Linux, and at the end of each file (the return of the dir command) it puts a CR.

Now this plays havock with my TOUCH command which tries to display the clean output after the for loop. It doesn't like the CR at all and messes everything up!.

Here is what the text file looks like:

 cr1
And here is what TOUCH displays when I call the file:

 cr2
------------

Here is the text file that is being called (in the script it is CSDIR_ftpdirlist_output.txt):

 CSDIR-ftpdirlist-output.txt

And here is the first FINDSTR code that is run against that file:

 
FINDSTR "Metro Insights" %TEMP%\CSDIR_ftpdirlist_output.txt > %TEMP%\CSDIR_ftpdirlist_raw.txt
FINDSTR "Regional Insights" %TEMP%\CSDIR_ftpdirlist_output.txt > %TEMP%\CSDIR_ftpdirlist_raw.txt
FINDSTR "National Insights" %TEMP%\CSDIR_ftpdirlist_output.txt > %TEMP%\CSDIR_ftpdirlist_raw.txt

Open in new window


And the second peice of code, which results in the first picture:

 
ECHO The current files for today are: > %TEMP%\CSDIR_ftpdirlist.txt
FOR /F "tokens=1-11" %%G IN (%TEMP%\CSDIR_ftpdirlist_raw.txt) DO @echo %%O %%P created on %%L %%M at %%N >> %TEMP%\CSDIR_ftpdirlist.txt

Open in new window


-------------
 
I know there are third party tools I can use to do this, but if possible I would like to keep everything to one batch file and without having to call any other executables.

So if there is any way I can strip that CR from the last clean file, well that'd be perfect!

Thanks!
0
agbnielsen
Asked:
agbnielsen
  • 10
  • 5
  • 4
  • +2
1 Solution
 
oBdACommented:
Try to pipe raw output through "find":
FOR /F "tokens=1-11" %%G IN ('type "%TEMP%\CSDIR_ftpdirlist_raw.txt" ^| find /v ""') DO @echo %%O %%P created on %%L %%M at %%N >> %TEMP%\CSDIR_ftpdirlist.txt

Open in new window

0
 
paultomasiCommented:
Interesting...

I was able to recreate your file using Notepad++'s Search & Replace facility and after saving the file the TYPE command output is:

   The current files for today are:
    created on Oct 21 at 03:15
    created on Oct 21 at 03:15

Oh, and BTW oBdA, you solution doesn't work...

Incidentally, using Notepad++ to open the file, it's so easy to remove the Carriage-Return characters... But that's not a practical solution is it?


0
 
paultomasiCommented:
Will return after the gym...
0
Restore individual SQL databases with ease

Veeam Explorer for Microsoft SQL Server delivers an easy-to-use, wizard-driven interface for restoring your databases from a backup. No expert SQL background required. Web interface provides a complete view of all available SQL databases to simplify the recovery of lost database

 
paultomasiCommented:
Would like to write a small Assembly code solution dynamically created by the batch file to filter input from the file...

Must go.... I'm late!!
0
 
QlemoC++ DeveloperCommented:
To be honest, I don't get why that FTP DIR output is that strange. Whatsoever, I have been successful when cutting off the last two chars from each line or file name:
@echo off
setlocal EnableDelayedExpansion
(FINDSTR "Insights" %TEMP%\CSDIR_ftpdirlist_output.txt | for /F "tokens=1-8*" %%A in ('more /e') do (
  set x=%%I
  echo !x:~,-2! created on %%F %%G at %%H
)) >> %TEMP%\CSDIR_ftpdirlist.txt

Open in new window

0
 
AlexPaceCommented:
FTP directory output is "strange" because the FTP standard does not specify a format.  There are easily 30 common formats and an untold numbr of rare ones.  The most common are broadly "unix style" and "DOS long filename style" but you've got hummingbird, novell, ibm and tons more.  Any FTP client that actually parses the directory listings will have some servers that it simply can't use.  An example of parsing a listing is if you use a FTP client that displays a graphical list of files, sizes, and timestamps or a command line client that allows you to download multplle files using a wildcard pattern.  Commercial FTP clients all have a not-insignificant section of code dedicated to analyzing a directory listing to determine the server type, and then parsing the file information.  

If you intend to use this batch file with multiple FTP servers you should get a directory listing from each of them at the beginning of the project because having them all will help you design more extensible parsing logic.  If you are faced with dratstically different listing formats that might be reason enough to use a third-party FTP client instead of rolling your own.
0
 
paultomasiCommented:
Okay, just say I'm looking at this again... Unfortunately, I was away all day.

As a quick fix I could suggest using a third-party utility which can be run from a batch file. I'll see if I can find a suitable one... (does SED do the job?).

0
 
QlemoC++ DeveloperCommented:
Yep, sed looks good:
sed.exe "s/\r//g" CSDIR-ftpdirlist-output.txt | findstr "Insight" | for /F "tokens=*" %A in ('more /e') do @echo %A~

Open in new window

results in
rwx    1 500      0          144893 Oct 21 03:15 Metro Insights_Slide1.JPG~
rwx    1 500      0           97966 Oct 21 03:15 Metro Insights_Slide2.JPG~
rwx    1 500      0           99162 Oct 21 03:15 National Insights_Slide1.JPG~
rwx    1 500      0          101916 Oct 21 03:15 Regional Insights_Slide1.JPG~
rwx    1 500      0          117034 Oct 21 03:15 Regional Insights_Slide2.JPG~

Open in new window

0
 
agbnielsenAuthor Commented:
Thanks guys. There is a utility called DOS2UNIX that does exactly what I want it to do and displays the contents, however as I said I would rather not have to rely on third party tools.

I am only using one FTP server (vsftp) so that won't be a problem.

Qlemo, I'll have a look at your other solution when I get home from the gym.

Cheers!
0
 
paultomasiCommented:
agbnielsen

If I'm not talking out of term here, I don't think Qlemo's 'other' solution (http:#37005271) works.

Like commands such as FINDSTR, MORE, TYPE and even FC (even in binary mode) DOS skips lines containing just a carriage return.
0
 
QlemoC++ DeveloperCommented:
http:#a37005271 is tested against the example file, and works great with that - but it depends on the very specific dir format used.
0
 
paultomasiCommented:
Save the following batch file and call it like this:

   BATCHFILE textfile

Where:
   BATCHFILE is the name of the file you saved this batch file as.
   textfile is the name of your text file containing both CR ad CR/LF.

This batch file requires SED.EXE to be present.
@echo off

if "%~1"=="" (
   echo Syntax: %~f0 filename
   goto :eof
)

echo The current files for today are:>"%~n1.tmp"

for /f "tokens=*" %%a in ('sed.exe "s/\r//g" "%~1" ^| findstr "Insight"') do (
   echo %%a>>"%~n1.tmp"
)

move "%~n1.tmp" "%~1"

Open in new window

0
 
paultomasiCommented:
Oops! Should have been:
@echo off

if "%~1"=="" (
   echo Syntax: %~nx0 filename
   goto :eof
)

echo The current files for today are:>"%~n1.tmp"

for /f "tokens=*" %%a in ('sed.exe "s/\r//g" "%~1" ^| findstr "Insight"') do (
   echo %%a>>"%~n1.tmp"
)

move "%~n1.tmp" "%~1"

Open in new window

0
 
paultomasiCommented:
Oh sod it, I mucked it up again with %~nx0... (tiredness does stuff like that....).

Basically, the line:

   echo Syntax: %~nx0 filename

should just be:

   echo Syntax: %~n0 filename

Not that it effects the operation of the program though.
0
 
agbnielsenAuthor Commented:
Hi All, sorry for the late reply on this!

@Qlemo: I have tried your solution, and the following is printed to screen and unfortunately does not work:

 
C:\Users\me\Desktop\scripts\Client Services>(set x=Metro Insights_Slide1.JPG   &
cho !x:~,-2! created on Oct 25 at 02:26  )
!x:~,-2! created on Oct 25 at 02:26

C:\Users\me\Desktop\scripts\Client Services>(set x=Metro Insights_Slide2.JPG   &
cho !x:~,-2! created on Oct 25 at 02:26  )
!x:~,-2! created on Oct 25 at 02:26

C:\Users\me\Desktop\scripts\Client Services>(set x=National Insights_Slide1.JPG
& echo !x:~,-2! created on Oct 25 at 02:26  )
!x:~,-2! created on Oct 25 at 02:26

C:\Users\me\Desktop\scripts\Client Services>(set x=Regional Insights_Slide1.JPG
& echo !x:~,-2! created on Oct 25 at 02:26  )
!x:~,-2! created on Oct 25 at 02:26

C:\Users\me\Desktop\scripts\Client Services>(set x=Regional Insights_Slide2.JPG
& echo !x:~,-2! created on Oct 25 at 02:26  )
!x:~,-2! created on Oct 25 at 02:26

Open in new window


@paultomasi: Thanks but I already have a tool called DOS2UNIX and all I need to do is run it against the .txt file and it fixes all the CRs. I would love to be able to get this working without having to use a third party tool, but unfortunately it looks like this simply isn't doable.
0
 
paultomasiCommented:
It is doable. In both DOS and VB script (I'm sure...).

The DOS-route will involve generating a small assembly language program which is then invoked by piping your file through it - easypeasy!

I just haven't had time (not even leisure time) to focus on this. My assembly programming skills are a bit rusty at the moment.

It is possible I may find time after this evening or tomorrow evening to find a solution.

What version of Windows will this run on? Also, do you know whether it's 32-bit or 64-bit?
0
 
QlemoC++ DeveloperCommented:
Does that work for you? CMD.Exe is producing strange results sometimes if you combine FOR with a pipe and use more than one pair of parens around it ...
@echo off
setlocal EnableDelayedExpansion
(for /F "tokens=1-8*" %%A in ('FINDSTR "Insights" %TEMP%\CSDIR_ftpdirlist_output.txt') do @(
  set x=%%I
  echo !x:~,-2! created on %%F %%G at %%H
)) >> %TEMP%\CSDIR_ftpdirlist.txt

Open in new window

0
 
agbnielsenAuthor Commented:
Beautiful! Just what I wanted and tested on multiple PCs.

As a bonus you have even taken out one layer of complexity in the script (by no longer CSDIR_ftpdirlist_raw.txt file).

Legend!
0
 
agbnielsenAuthor Commented:
ahhhh fuck... I accepted the wrong solution. Qlemo that was supposed to go to you. Have requested a reversal and appropriate allocation of points.
0
 
agbnielsenAuthor Commented:
Cheers
0
 
paultomasiCommented:
I misinterpretted the question...

I thought the asker wanted to remove CRs from CSDIR_ftpdirlist.txt. My focus was drawn away from the original output file...

0

Featured Post

VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

  • 10
  • 5
  • 4
  • +2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now