?
Solved

Move log files based on search results

Posted on 2003-02-25
13
Medium Priority
?
295 Views
Last Modified: 2012-06-27
I have tried adapting a solution offered in http://www.experts-exchange.com/Operating_Systems/MSDOS/Q_20380332.html, but I am finding that solution too slow for my needs.


I need some help moving files around, and would like to use batch to do it.


I have log files created daily on L:\logs (server) named EJ*.*.

Within these files are text strings that I wish to use to file them within sub folders.

Each file is randomly named, so I can't go by the filenames.

Examples:

1>  Text string x000250 would be filed under l:\logs\x\000250\

ID's range from x000000 to x999999

2>  Text string e123456 would be filed under l:\logs\e\123456\

ID's range from e000000 to e999999


3>   THE REAL SPANNER IN THE WORKS....
Text string "log  0001" would be filed under l:\logs\w\xcgf
Text string "log  0002" would be filed under l:\logs\w\ac4w

ID's range from "log  0000" to "log  9999"

As the server has new customer accounts added, the string may or may not have any relation to the folder name, so I am looking for a solution that can easily be updated to reflect account changes/removal/new customers.


All files are text data, and are named in the following formats:

EJ074795.013
EJ074797.002
EJ074797.003
EJ074797.004
EJ074797.008
EJ074797.009
EJ074797.010
EJ074797.011
EJ074797.013
EJ074797.014
EJ074798.002
EJ074798.004
EJ074799.008
EJ074800.007
EJ074801.001
EJ074801.002
EJ074802.009
EJ074802.011
EJ074802.015
EJ074803.000
EJ074803.001
EJ074803.002
EJ074803.003
EJ074803.004
EJ074803.005
EJ074803.006
EJ074803.007
EJ074803.008
EJ074803.009
EJ074804.003
EJ074804.005
EJ074804.014
EJ074804.015
EJ074804.020
EJ074805.002
EJ074805.003
EJ074805.004
EJ074805.007
EJ074805.008
EJ074805.009
EJ074805.010
EJ074805.015
EJ074805.016
EJ074805.018
EJ074805.019
EJ074805.021
EJ074805.023
EJ074805.027
EJ074805.028
EJ074805.030
EJ074805.035

This number of files created daily could be 100's or 1,000's of files - far too many to process manually.


I am open to most options - especially any tweeks from the solution I am trying to make the whole thing a breeze.


Many thanks for any help that you can offer.

Regards,

Simon Hawking
0
Comment
Question by:getsurfers
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 5
  • 3
13 Comments
 

Author Comment

by:getsurfers
ID: 8016300
My thoughts on this would be along the lines of

* Find all files ej*.* containing "textstring"
* Output find results (only containing query, and only filenames - as per dir /b format) to found.txt
* Move all files listed in found.txt to \text\string



Regards,

Simon Hawking
0
 
LVL 7

Expert Comment

by:billious
ID: 8016449
A few items may be of aid here -

1. What operating system are you using?
2. is the 'x999999', 'e999999' literal and unique?
3. will every file be moved somewhere else, or will some stay in the directory with EJ*.* names?
4. since ""ID's range from "log  0000" to "log  9999" ""
are you going to set up the 10,000 destination directories?
  suppose you have "log  6666" and nothing set up as a destination for "log  6666"?

- noting that there are TWO spaces (apparently) between "log" and the number - is that corrrect??

...Bill

0
 

Author Comment

by:getsurfers
ID: 8017393
billious,

1. needs to work on 9x, NT, 2000, xp
2. the search string is unique, and each file would only contain one of the search strings.
3. the files will be moved from their location in l:\logs to their own location

search strings in the format "x******" would be moved to l:\logs\x\******

there are 1,000,000 folder destinations

search strings in the format "e******" would be moved to l:\logs\e\******

there are 1,000,000 folder destinations

4.  Yes there is a double space in the string "log  ----"

I've created the folders by creating a batch to set them up (lot easier than by hand ;-> ).  Yes there are 10,000 destinations for this string.

If there is no destination, or any other error (file exists, folder doesn't exist, no matches etc...) then the solution would leave any matches where they are now, and process the next string.


I have won half the battle - see http://www.experts-exchange.com/Operating_Systems/WinNT/Q_20527210.html.
That produces the following when searching for "e000001":

l:\logs\ej000250.000
l:\logs\ej944567.101
l:\logs\ej987000.067
l:\logs\ej998763.398


Now i have some results, is there any way to move the files listed in found.txt to L:\logs\e\e000001 by using found.txt?



Many thanks,


Simon.
0
Free Backup Tool for VMware and Hyper-V

Restore full virtual machine or individual guest files from 19 common file systems directly from the backup file. Schedule VM backups with PowerShell scripts. Set desired time, lean back and let the script to notify you via email upon completion.  

 
LVL 7

Expert Comment

by:billious
ID: 8018062
getsurfers,

Since you evidently simply need suggestions, rather than a perfect solution... :)

How about a slightly changed approach :

for %%f in (EJ*.*) do call movem %%f

movem.bat in pseudocode
findstr "e[0-9][0-9][0-9...." %1
if not errorlevel 1 moveto %1 e
findstr "x[0-9][0-9][0-9...." %1
if not errorlevel 1 moveto %1 x
:: since not e999999 or x999999, look for "log  9999"
findstr "log  [0-9][0-9][0-9...." %1
if not errorlevel 1 dont %1
::log  9999 is in text
* use SED/AWK/GAWK to find it and work out what the number is
* re-use SED/AWK/GAWK to find the destination directory
* from a file with record lines format
* log  0001l:\logs\w\xcgf
if not found, dont %1
echo move %1 destdir_found_by_sed>moveme.bat
moveme



moveto.bat pseudocode
* use SED/AWK/GAWK to find %2[0-9]...  (work out what the
*    enumber or xnumber was
echo move %1 l:\log\%2\enumber_or_xnumberfound>moveme.bat
moveme

dont.bat pseudocode
:: just move any unprocessable to a directory
move %1 l:\log\notprocessed


I'll assume that you can work out the calling/execution path. I'd contemplated calling from the for %%f..., then using separate batchfiles, but only because I normally fool around with DOS622. Using the above structure, the call for any one filename would be exited by a path such as
movem/moveto/moveme, and processing of movem would be truncated by executing rather than calling the subsidiary batch.

I'm well aware that NT can do it better, but your point about 9x compatibility may impact here (I don't use 9x).

I've chosen to move unprocessable files to notprocessed so that the for doesn't re-encounter them. I'd suggest this would be appropriate in any case, since otherwise you'd perpetually re-process the same unprocessable files. Move them back after the for %%f if that's what you want!

HTH

Bill








0
 
LVL 10

Expert Comment

by:pbarrette
ID: 8018360
Hi Simon,

Can you post a sections of the logfile which contain the specified text? This would make it easier to search for your text while providing a way to break it up into the necessary components to do the MOVE operation.

Also, using Win9X will be extremely difficult. You simply don't have the options available at the commandline to accomplish this task in a reasonable way. I'm not saying it can't be done, but it would be extremely convoluted to say the least.

pb
0
 

Author Comment

by:getsurfers
ID: 8018678
Bill,

I would need all that code for each possible string?????


Thanks,

Simon
0
 
LVL 7

Expert Comment

by:billious
ID: 8021545
Nope.

if you look in the on-line docco on findstr, you'll find that it matches on regular expressions, so you don't actually need to write out e000000..e999999 individually,
the single string "e[0-9][0-9][0-9][0-9][0-9][0-9]" will match e followed by 6 numerics, so you only need the one findstr line, not a million!

Equally SED uses regular expressions, and it should be a relatively simple task to have it extract the appropriate strings - no more than 3 or 4 lines of sed.

The operation required here is quite complex, especially for a brain-dead language like batch - even with NT enhancements.

It might also be prudent to review your platform-requirements. I'd suggest that one machine would be dedicated to do this task - make it an NT machine (or use VER to stop 9x machines from running the job.)

Also, would you WANT more than one machine to be playing with these files SIMULTANEOUSLY. If you have multiple machines running the process at the same time, you're in for a lot of contention problems!

...Bill

0
 
LVL 10

Expert Comment

by:pbarrette
ID: 8021746
Hi Simon,

Also, keep in mind that Win98 with FAT32 will only support a maximum of 4,177,920 files per volume and only 65,534 files/subdirs per directory.

With this in mind, Win9x support may not be a good way to go. If you are doing this via a Win9x client over a network, then the limitation will not apply, but keep in mind that you will be parsing through thousands, if not millions of files. All of those files will have to be transported to the client, then parsed by the client, then transported to the appropriate subdirectory on the server.

That's a potential of well over 4 million file transfers for each run.

pb
0
 
LVL 10

Expert Comment

by:pbarrette
ID: 8021921
Hi again,

Stealing from Bill's idea, you could extract the text without using SED if you know where it is. In another post:
http://www.experts-exchange.com/Operating_Systems/WinNT/Q_20527210.html

You gave a line from the logfile:
)[02p)(1    13/02/03  16:09 x000033

If the xNNNNNN or eNNNNNN is always at the end of the line, as this example shows, then you can use the FOR command along with Win2K's SET command to separate the values:

:: ----EXAMPLE.BAT----
@ECHO OFF
FOR %%F IN (EJ*.*) DO (
 SET FILE=%%F
 CALL :PROCFILE
)
GOTO END
:PROCFILE
FOR /F "DELIMS=" %%F IN ('FINDSTR "[ex][0-9][0-9][0-9][0-9][0-9][0-9]" %FILE%') DO (
 IF NOT "%%F"=="" (SET LINE=%%F)
)
SET DIR1=%LINE:~-7,1%
SET DIR2=%LINE:~-6,6%
ECHO COPY %FILE% I:\LOG\%DIR1%\%DIR2%
:END
:: ----EXAMPLE.BAT----

This works for 2K only, and will also only work if the x/eNNNNNN string is the last part of the line. It could be modified to handle breaking up the search string in other parts of the line, but it is more difficult to do this if the string is located in different locations for different logfiles.

pb
0
 
LVL 10

Expert Comment

by:pbarrette
ID: 8021950
Oh hell,

I've missed the #3 reqiurement and the ECHO COPY line should be a MOVE line. Time to get something to eat, watch some TV and go to bed. I'll get back to it in the morning.

pb
0
 

Author Comment

by:getsurfers
ID: 8024460
ok.

9x machines are no longer an issue (why don't people tell me these things??).

As of today we only have NT4, XP, Win 2000 based systems.


The search string could be on any line of the file, and the no. of lines per file is different every time.

The typical text from "x000033" files are:

(1*587*13/02/03*16.08*
EJ LOG COPIED OK 14204000      
(1    13/02/03  16:08 X000033
    C1 STR         `0
    C1 INC     `25000
    C1 DEC         `0
    C1 OUT     `19770
    C1 END      `5230
    C2 STR         `0
    C2 INC     `55000
    C2 DEC         `0
    C2 OUT     `43680
    C2 END     `11320
(1
(1    DATA CAPTURE CNT    02

(1
(1    13/02/03  16:08 X000033



The typical text from "e000037" files are:

(1*904*11/10/00*22.11*
EJ LOG COPIED OK 949F0100      
*0000*1*R*09
(1*905*11/10/00*22.14*
SUPERVISOR MODE EXIT            
*0000*1*P*20,M-  
(1)2
(129/10/02   21:22    3908     E000037  
ACQ FEE CONFIRMATION
(1
(1DATE       TIME    SEQ#    ID
29/10/02   21:22   3908    E000037  



The typical text from "log  3058"

(1*119*07/16/02*12.10*
EJ LOG COPIED OK 9C556500      
*0031*1*R*09
(1*120*07/16/02*12.10*
SUPERVISOR MODE EXIT            
*0031*1*P*20,M-  
(1)2
(1)2
(1)2
(1)2
(1)2
(1
   16JUL02               12:11    
   
   LOG  3058           FILE SEQ 032
   
   DAILY SYSTEM CLOSEOUT                
(1



I need solutions for ALL 3 strings
0
 
LVL 10

Accepted Solution

by:
pbarrette earned 800 total points
ID: 8027286
Hi Simon,

This should work for NT+ provided certain conditions remain true:
1) These lines:
(129/10/02   21:22    3908     E000037
(1    13/02/03  16:08 X000033

Both lines are formatted as follows:
????? ?????? ?????? XNNNNNN

If the lines contain more or less than 3 spaces before the target data (XNNNNNN), the script will not function correctly.

2) The line:
  LOG  3058           FILE SEQ 032

Must have 1 space before the "LOG NNNN". Also, any other lines in this file containing the text "LOG" and "NNNN" must also contain the text "COPIED" to be filtered out. If this does not happen, then the line containing "LOG NNNN" must be the last line in the file to contain "LOG" and "NNNN".

This works for me, based on the examples you have given. You didn't specify much information on what to do with the "LOG NNNN" files, so the ":LOGNOPROC" section of the batchfile handles each "NNNN" separately.

Ok. Enough with the jibber jabber. Here's the script:
:: ----EXAMPLE.BAT----
@ECHO OFF
SET DIR1=

FOR %%F IN (EJ*.*) DO (
 SET FILE=%%F
 CALL :PROCFILE
)
GOTO END
:PROCFILE
:: +---------------------------+
:: | Check for ?NNNNNN in log  |
:: | IF exists: Go to PROCLINE |
:: +---------------------------+
FOR /F "TOKENS=4 DELIMS= " %%F IN ('FINDSTR "[EX][0-9][0-9][0-9][0-9][0-9][0-9]" %FILE%') DO (
 IF NOT "%%F"=="" (
  SET LINE=%%F
  GOTO PROCLINE
 )
)
:: +--------------------------+
:: | If not ?NNNNNN check for |
:: | LOG NNNN                 |
:: +--------------------------+
IF "%DIR1%"=="" (
 FOR /F "TOKENS=2" %%F IN ('FINDSTR "LOG [0-9][0-9][0-9][0-9]" %FILE%^|FIND /V "COPIED"') DO (SET LOGNO=%%F)
)
:: +-----------------------+
:: | If 'LOG NNNN' exists: |
:: | Go to LOGNOPROC       |
:: +-----------------------+
IF NOT "%LOGNO%"=="" (CALL :LOGNOPROC)
GOTO END


:PROCLINE
:: +-------------------+
:: | Move ?NNNNNN file |
:: +-------------------+
SET DIR1=%LINE:~0,1%
SET DIR2=%LINE:~1,6%
ECHO MOVE %FILE% L:\LOGS\%DIR1%\%DIR2%
SET DIR1=
GOTO END

:LOGNOPROC
:: +----------------------------+
:: | Move 'LOG NNNN' file based |
:: | on value of NNNN           |
:: +----------------------------+
IF %LOGNO% EQU 1 (ECHO MOVE %FILE% L:\LOGS\W\XCGF)
IF %LOGNO% EQU 2 (ECHO MOVE %FILE% L:\LOGS\W\AC4W)
IF %LOGNO% EQU 3058 (ECHO MOVE %FILE% L:\LOGS\W\PRB1)
:END
:: ----EXAMPLE.BAT----

pb
0
 

Author Comment

by:getsurfers
ID: 8066300
PB,

Sorry for delay in my reply, but I've been off with the flu.


I have awarded you (pbarrette) the points for your help in this question, although an adaption of your answer in http://www.experts-exchange.com/Operating_Systems/WinNT/Q_20527210.html was actually used.  The process works brilliantly, is easy to admin (if required), and most importantly - its QUICK!

The code you supplied to moved files based on the files and paths in found.txt was missing a % in two places.

The exact code used (this code is required for every string):

REM ** LOG ID EXAMPLE    
cd \
cd L:\logs\X\EXAMPLE*
findstr /i/m "EXAMPLE" l:\logs\*.* > c:\temp\found.tmp
FOR /F "DELIMS=" %%F IN (c:\temp\found.tmp) DO (MOVE %%F L:)


Once again many thanks for all of your hard work.


Regards,

Simon Hawking
0

Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

One of my most closely kept secrets is revealed in this discussion How to output text on the same line This question was recently posted in EE by Simon336697 (http://www.experts-exchange.com/Programming/Languages/Scripting/Shell/Batch/Q_2459…
Being a system administrator some time we require to do things remotely, one of them is installing software. Here I am going to tell you how to install software through wmic (Windows management instrument console). I am not at all saying that this i…
Michael from AdRem Software outlines event notifications and Automatic Corrective Actions in network monitoring. Automatic Corrective Actions are scripts, which can automatically run upon discovery of a certain undesirable condition in your network.…
In this video, Percona Director of Solution Engineering Jon Tobin discusses the function and features of Percona Server for MongoDB. How Percona can help Percona can help you determine if Percona Server for MongoDB is the right solution for …
Suggested Courses
Course of the Month13 days, 15 hours left to enroll

800 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question