asked on

Batch - Strip <CR> from multiple lines in text file

Hi All,

I am writing a batch script that uses Windows ftp client to connect to our FTP server and does a directory listing and outputs it to a file. The output is very dirty though, so I then do a FINDSTR on the file to get only the lines I am interested in, and then a FOR loop on that to get the cleanest version I can in order to output to screen (displays a summary of the files that are on the server).

The problem is that the FTP server runs on Linux, and at the end of each file (the return of the dir command) it puts a CR.

Now this plays havock with my TOUCH command which tries to display the clean output after the for loop. It doesn't like the CR at all and messes everything up!.

Here is what the text file looks like:

And here is what TOUCH displays when I call the file:

------------

Here is the text file that is being called (in the script it is CSDIR_ftpdirlist_output.txt):

CSDIR-ftpdirlist-output.txt

And here is the first FINDSTR code that is run against that file:

FINDSTR "Metro Insights" %TEMP%\CSDIR_ftpdirlist_output.txt > %TEMP%\CSDIR_ftpdirlist_raw.txt
FINDSTR "Regional Insights" %TEMP%\CSDIR_ftpdirlist_output.txt > %TEMP%\CSDIR_ftpdirlist_raw.txt
FINDSTR "National Insights" %TEMP%\CSDIR_ftpdirlist_output.txt > %TEMP%\CSDIR_ftpdirlist_raw.txt

Open in new window

And the second peice of code, which results in the first picture:

ECHO The current files for today are: > %TEMP%\CSDIR_ftpdirlist.txt
FOR /F "tokens=1-11" %%G IN (%TEMP%\CSDIR_ftpdirlist_raw.txt) DO @echo %%O %%P created on %%L %%M at %%N >> %TEMP%\CSDIR_ftpdirlist.txt

Open in new window

-------------

I know there are third party tools I can use to do this, but if possible I would like to keep everything to one batch file and without having to call any other executables.

So if there is any way I can strip that CR from the last clean file, well that'd be perfect!

Thanks!

oBdA

Try to pipe raw output through "find":

FOR /F "tokens=1-11" %%G IN ('type "%TEMP%\CSDIR_ftpdirlist_raw.txt" ^| find /v ""') DO @echo %%O %%P created on %%L %%M at %%N >> %TEMP%\CSDIR_ftpdirlist.txt

Open in new window

Paul Tomasi

Interesting...

I was able to recreate your file using Notepad++'s Search & Replace facility and after saving the file the TYPE command output is:

The current files for today are:
created on Oct 21 at 03:15
created on Oct 21 at 03:15

Oh, and BTW oBdA, you solution doesn't work...

Incidentally, using Notepad++ to open the file, it's so easy to remove the Carriage-Return characters... But that's not a practical solution is it?

Paul Tomasi

Will return after the gym...

Paul Tomasi

Would like to write a small Assembly code solution dynamically created by the batch file to filter input from the file...

Must go.... I'm late!!

Qlemo

To be honest, I don't get why that FTP DIR output is that strange. Whatsoever, I have been successful when cutting off the last two chars from each line or file name:

@echo off
setlocal EnableDelayedExpansion
(FINDSTR "Insights" %TEMP%\CSDIR_ftpdirlist_output.txt | for /F "tokens=1-8*" %%A in ('more /e') do (
  set x=%%I
  echo !x:~,-2! created on %%F %%G at %%H
)) >> %TEMP%\CSDIR_ftpdirlist.txt

Open in new window

AlexPace

FTP directory output is "strange" because the FTP standard does not specify a format. There are easily 30 common formats and an untold numbr of rare ones. The most common are broadly "unix style" and "DOS long filename style" but you've got hummingbird, novell, ibm and tons more. Any FTP client that actually parses the directory listings will have some servers that it simply can't use. An example of parsing a listing is if you use a FTP client that displays a graphical list of files, sizes, and timestamps or a command line client that allows you to download multplle files using a wildcard pattern. Commercial FTP clients all have a not-insignificant section of code dedicated to analyzing a directory listing to determine the server type, and then parsing the file information.

If you intend to use this batch file with multiple FTP servers you should get a directory listing from each of them at the beginning of the project because having them all will help you design more extensible parsing logic. If you are faced with dratstically different listing formats that might be reason enough to use a third-party FTP client instead of rolling your own.

Paul Tomasi

Okay, just say I'm looking at this again... Unfortunately, I was away all day.

As a quick fix I could suggest using a third-party utility which can be run from a batch file. I'll see if I can find a suitable one... (does SED do the job?).

Qlemo

Yep, sed looks good:

sed.exe "s/\r//g" CSDIR-ftpdirlist-output.txt | findstr "Insight" | for /F "tokens=*" %A in ('more /e') do @echo %A~

Open in new window

results in

rwx    1 500      0          144893 Oct 21 03:15 Metro Insights_Slide1.JPG~
rwx    1 500      0           97966 Oct 21 03:15 Metro Insights_Slide2.JPG~
rwx    1 500      0           99162 Oct 21 03:15 National Insights_Slide1.JPG~
rwx    1 500      0          101916 Oct 21 03:15 Regional Insights_Slide1.JPG~
rwx    1 500      0          117034 Oct 21 03:15 Regional Insights_Slide2.JPG~

Open in new window

agbnielsen

ASKER

Thanks guys. There is a utility called DOS2UNIX that does exactly what I want it to do and displays the contents, however as I said I would rather not have to rely on third party tools.

I am only using one FTP server (vsftp) so that won't be a problem.

Qlemo, I'll have a look at your other solution when I get home from the gym.

Cheers!

Paul Tomasi

agbnielsen

If I'm not talking out of term here, I don't think Qlemo's 'other' solution (http:#37005271) works.

Like commands such as FINDSTR, MORE, TYPE and even FC (even in binary mode) DOS skips lines containing just a carriage return.

Qlemo

http:#a37005271 is tested against the example file, and works great with that - but it depends on the very specific dir format used.

Paul Tomasi

Save the following batch file and call it like this:

BATCHFILE textfile

Where:
BATCHFILE is the name of the file you saved this batch file as.
textfile is the name of your text file containing both CR ad CR/LF.

This batch file requires SED.EXE to be present.

@echo off

if "%~1"=="" (
   echo Syntax: %~f0 filename
   goto :eof
)

echo The current files for today are:>"%~n1.tmp"

for /f "tokens=*" %%a in ('sed.exe "s/\r//g" "%~1" ^| findstr "Insight"') do (
   echo %%a>>"%~n1.tmp"
)

move "%~n1.tmp" "%~1"

Open in new window

Paul Tomasi

Oops! Should have been:

@echo off

if "%~1"=="" (
   echo Syntax: %~nx0 filename
   goto :eof
)

echo The current files for today are:>"%~n1.tmp"

for /f "tokens=*" %%a in ('sed.exe "s/\r//g" "%~1" ^| findstr "Insight"') do (
   echo %%a>>"%~n1.tmp"
)

move "%~n1.tmp" "%~1"

Open in new window

Paul Tomasi

Oh sod it, I mucked it up again with %~nx0... (tiredness does stuff like that....).

Basically, the line:

echo Syntax: %~nx0 filename

should just be:

echo Syntax: %~n0 filename

Not that it effects the operation of the program though.

agbnielsen

ASKER

Hi All, sorry for the late reply on this!

@Qlemo: I have tried your solution, and the following is printed to screen and unfortunately does not work:

C:\Users\me\Desktop\scripts\Client Services>(set x=Metro Insights_Slide1.JPG   &
cho !x:~,-2! created on Oct 25 at 02:26  )
!x:~,-2! created on Oct 25 at 02:26

C:\Users\me\Desktop\scripts\Client Services>(set x=Metro Insights_Slide2.JPG   &
cho !x:~,-2! created on Oct 25 at 02:26  )
!x:~,-2! created on Oct 25 at 02:26

C:\Users\me\Desktop\scripts\Client Services>(set x=National Insights_Slide1.JPG
& echo !x:~,-2! created on Oct 25 at 02:26  )
!x:~,-2! created on Oct 25 at 02:26

C:\Users\me\Desktop\scripts\Client Services>(set x=Regional Insights_Slide1.JPG
& echo !x:~,-2! created on Oct 25 at 02:26  )
!x:~,-2! created on Oct 25 at 02:26

C:\Users\me\Desktop\scripts\Client Services>(set x=Regional Insights_Slide2.JPG
& echo !x:~,-2! created on Oct 25 at 02:26  )
!x:~,-2! created on Oct 25 at 02:26

Open in new window

@paultomasi: Thanks but I already have a tool called DOS2UNIX and all I need to do is run it against the .txt file and it fixes all the CRs. I would love to be able to get this working without having to use a third party tool, but unfortunately it looks like this simply isn't doable.

Paul Tomasi

It is doable. In both DOS and VB script (I'm sure...).

The DOS-route will involve generating a small assembly language program which is then invoked by piping your file through it - easypeasy!

I just haven't had time (not even leisure time) to focus on this. My assembly programming skills are a bit rusty at the moment.

It is possible I may find time after this evening or tomorrow evening to find a solution.

What version of Windows will this run on? Also, do you know whether it's 32-bit or 64-bit?

ASKER CERTIFIED SOLUTION

Qlemo

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

agbnielsen

ASKER

Beautiful! Just what I wanted and tested on multiple PCs.

As a bonus you have even taken out one layer of complexity in the script (by no longer CSDIR_ftpdirlist_raw.txt file).

Legend!

agbnielsen

ASKER

ahhhh fuck... I accepted the wrong solution. Qlemo that was supposed to go to you. Have requested a reversal and appropriate allocation of points.

agbnielsen

ASKER

Cheers

Paul Tomasi

I misinterpretted the question...

I thought the asker wanted to remove CRs from CSDIR_ftpdirlist.txt. My focus was drawn away from the original output file...

Batch - Strip &lt;CR&gt; from multiple lines in text file

Batch - Strip <CR> from multiple lines in text file