Solved

BatchFile: "For /F" to get domain name

Posted on 2010-09-22
35
1,064 Views
Last Modified: 2012-05-10
Hi there,

Using a batch file, I need to get the domain names out of the following log file.

PROXYLOG.TXT:
Tue 20 Sep 2010 11:13:01      192.168.11.130      HTTP-Proxy      8080      google.com
Tue 20 Sep 2010 11:13:01      192.168.11.130      HTTP-Proxy      8080      www.google.ca

EXPECTED OUTPUT:
google.com
google.ca


Thanks for your help,
Rene
0
Comment
Question by:ReneGe
  • 13
  • 11
  • 5
  • +2
35 Comments
 
LVL 8

Assisted Solution

by:ragnarok89
ragnarok89 earned 25 total points
ID: 33734506
download the free tool "cut" from http://www.ltr-data.se/opencode.html

then use the command "type proxylog.txt | cut -F:9 > domains.txt" (9 is a field number, or column number)

you can then use a regex to remove all instances of "www." from the domains.txt file
0
 
LVL 10

Author Comment

by:ReneGe
ID: 33734748
ragnarok89,

I do not wish to use utils.

I need this script to be done in batch file.

Thanks,
Rene
0
 
LVL 4

Assisted Solution

by:rstjean
rstjean earned 50 total points
ID: 33734818
copy getdomains.txt to getdomains.vbs

write a batch file with the following

getdomains.vbs [path of log] [path of output file]
getdomains.txt
0
 
LVL 10

Author Comment

by:ReneGe
ID: 33734930
rstjean,

Thanks for the VBScript.

I'm realy looking for a Windows Batch File.

Cheers,
Rene

0
 
LVL 4

Expert Comment

by:rstjean
ID: 33735004
You can't do it in a batch file without calling a program or script.  Findstr will only return the full line, and will not return a partial match.

You can use a batch file to call the vb script.  Remove the .txt from the end, and change the values with the locations of your server.





getdomains.bat.txt
0
 
LVL 51

Expert Comment

by:Bill Prew
ID: 33735140
How about this in a BAT file?

Adjust the two sets at the top to your in and out file desired.

~bp
@echo off
set InFile=c:\temp\EE26491259.in
set OutFile=c:\temp\EE26491259.out
if exist "%OutFile%" del "%OutFile%"
for /F "usebackq tokens=9*" %%A in ("%InFile%") do echo %%A>>"%OutFile%"

Open in new window

0
 
LVL 8

Expert Comment

by:ragnarok89
ID: 33735234
Rene,

Will the URL always start at the 77 character of each line?
0
 
LVL 8

Expert Comment

by:ragnarok89
ID: 33735322
@echo off
del domains.txt
for /F "usebackq tokens=9*" %%A in (PROXYLOG.TXT) do call :parse %%A

:parse
set str=%1
set str=%str:www.=%
echo.%str% >> domains.txt

This will do it all.

0
 
LVL 51

Expert Comment

by:Bill Prew
ID: 33735769
Ah, I missed the removal of the www., this adjustment to my approach should handle that.

~bp
@echo off
setlocal EnableDelayedExpansion
set InFile=c:\temp\EE26491259.in
set OutFile=c:\temp\EE26491259.out
if exist "%OutFile%" del "%OutFile%"
for /F "usebackq tokens=9*" %%A in ("%InFile%") do (
  set Domain=%%A
  echo !Domain:www.=!>>"%OutFile%"
)

Open in new window

0
 
LVL 10

Author Comment

by:ReneGe
ID: 33736425
Hi guys,

As you know, FQDN may contain all sots of stufs:
web.whatever.on.adomain.on.ca

So whever it parses, it must output the last two tokens of "."
So the this example, the output would be: on.ca

So removing www would be pointless is this case.

Thanks to you all and cheers,
Rene
0
 
LVL 10

Author Comment

by:ReneGe
ID: 33736434
By the way, each colones are separated by a TAB.

Cheers
0
 
LVL 51

Expert Comment

by:Bill Prew
ID: 33737480
Give this a try.

~bp
@echo off
setlocal EnableDelayedExpansion
set InFile=c:\temp\EE26491259.in
set OutFile=c:\temp\EE26491259.out
if exist "%OutFile%" del "%OutFile%"
for /F "usebackq tokens=9*" %%A in ("%InFile%") do (
  call :GetDomain %%A
  echo !Domain!>>"%OutFile%"
)
 
:GetDomain [host-address]
  for /F "tokens=1-20 delims=." %%A in ("%~1") do (
    if "%%T" NEQ "" (set Domain=%%S.%%T & exit /b)
    if "%%S" NEQ "" (set Domain=%%R.%%S & exit /b)
    if "%%R" NEQ "" (set Domain=%%Q.%%R & exit /b)
    if "%%Q" NEQ "" (set Domain=%%P.%%Q & exit /b)
    if "%%P" NEQ "" (set Domain=%%O.%%P & exit /b)
    if "%%O" NEQ "" (set Domain=%%N.%%O & exit /b)
    if "%%N" NEQ "" (set Domain=%%M.%%N & exit /b)
    if "%%M" NEQ "" (set Domain=%%L.%%M & exit /b)
    if "%%L" NEQ "" (set Domain=%%K.%%L & exit /b)
    if "%%K" NEQ "" (set Domain=%%J.%%K & exit /b)
    if "%%J" NEQ "" (set Domain=%%I.%%J & exit /b)
    if "%%I" NEQ "" (set Domain=%%H.%%I & exit /b)
    if "%%H" NEQ "" (set Domain=%%G.%%H & exit /b)
    if "%%G" NEQ "" (set Domain=%%F.%%G & exit /b)
    if "%%F" NEQ "" (set Domain=%%E.%%F & exit /b)
    if "%%E" NEQ "" (set Domain=%%D.%%E & exit /b)
    if "%%D" NEQ "" (set Domain=%%C.%%D & exit /b)
    if "%%C" NEQ "" (set Domain=%%B.%%C & exit /b)
    if "%%B" NEQ "" (set Domain=%%A.%%B & exit /b)
    exit /b
  )

Open in new window

0
 
LVL 11

Expert Comment

by:Ben Personick
ID: 33739961
I am submitting a modified version of Bill Prew's Script here, because I have had to deal with a similar issue before and realised a better way to handle it woudl be as attached.

The difference is that my version of Bill's script will allow you to parse any domain name no matter how many sub terms it has, while Bill's unfortunately cannot handle more than 39.

 (I actually ran into that limitation which is what made me realise this would be a better way to handle these types of recursions in the future)
Note:  My code will Solve your Question  more completely however half the credit is Bill Prews.

@ECHO OFF
SETLOCAL EnableDelayedExpansion
SET "SrcFile=C:\Admin\ParseProxyLog\PROXYLOG.TXT"
SET "DstFile=C:\Admin\ParseProxyLog\DomainList.log"
IF EXIST "%DstFile%" DEL /F /Q "%DstFile%"
for /F "usebackq tokens=9*" %%A in ("%SrcFile%") do (
  CALL :GetDomain "%%A"
  ECHO !Host!.!TopLvl!>>"%DstFile%"
)
 
:GetDomain [host-address]
	SET "RawData=%~1"
	FOR %%A IN ("%RawData:.=","%") DO (
	SET "Host=!TopLvl!"
	SET "TopLvl=%%~A"
	)
ENDLOCAL

Open in new window

0
 
LVL 11

Expert Comment

by:Ben Personick
ID: 33740062
Also I couldn't resist tightening this up, since we're already using delayed expansion we don't have to bother with calling the function at all, so this code makes more sence given the circumstances.

Again this is a hybrid of Bill's work and my own.



@ECHO OFF
SETLOCAL EnableDelayedExpansion
SET "SrcFile=C:\Admin\ParseProxyLog\PROXYLOG.TXT"
SET "DstFile=C:\Admin\ParseProxyLog\DomainList.log"
IF EXIST "%DstFile%" DEL /F /Q "%DstFile%"
FOR /F "usebackq tokens=9*" %%A IN ("%SrcFile%") DO (
	SET "RawData=%%~A"
	FOR %%B IN ("!RawData:.=","!") DO (
		SET "Host=!TopLvl!"
		SET "TopLvl=%%~B"
	)
	ECHO !Host!.!TopLvl!
)
ENDLOCAL

Open in new window

0
 
LVL 11

Assisted Solution

by:Ben Personick
Ben Personick earned 200 total points
ID: 33740084
heh!

Sorry the above post does work, but does not write to the file because when I was testing it was easier to spit the results to the command prompt instead of checking the file each time through.

  Attached is the above code with output writing to the file again.

@ECHO OFF
SETLOCAL EnableDelayedExpansion
SET "SrcFile=C:\Admin\ParseProxyLog\PROXYLOG.TXT"
SET "DstFile=C:\Admin\ParseProxyLog\DomainList.log"
IF EXIST "%DstFile%" DEL /F /Q "%DstFile%"
FOR /F "usebackq tokens=9*" %%A IN ("%SrcFile%") DO (
        SET "RawData=%%~A"
        FOR %%B IN ("!RawData:.=","!") DO (
                SET "Host=!TopLvl!"
                SET "TopLvl=%%~B"
        )
        ECHO !Host!.!TopLvl!>>"%DstFile%"
)
ENDLOCAL

Open in new window

0
 
LVL 51

Expert Comment

by:Bill Prew
ID: 33740119
Looking good QC.

~bp
0
 
LVL 11

Expert Comment

by:Ben Personick
ID: 33740303
Right back at cha BP =)

~Q
0
Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

 
LVL 10

Author Comment

by:ReneGe
ID: 33740426
Hey Q, bp

I'v tried Q's latest script and I only got the following as output:

.:
:.:
:.:
:.:

The original log file is attached. I modifed it with the following script, which I get the PROXYLOG.TXT content.

Hoping this helps.

Thanks a lot and cheers,
Rene
@ECHO OFF

SETLOCAL ENABLEDELAYEDEXPANSION



Set OutputFile=%~n0.txt



IF EXIST "%OutputFile%" DEL "%OutputFile%"





REM %%A=WHEN

REM %%B=PROTOCOL

REM %%C=PORT

REM %%D=USER IP ADDRESS

REM %%E=USERNAME (Setup in proxy)

REM %%F=DESTINATION FQDN



REM EXAMPLE

REM -----------------------------------------------

REM A Tue 21 Sep 2010 10:21:09 : CONN STAT : Instance:

REM B HTTP-Proxy

REM C 8080

REM D 192.168.2.127

REM E DefaultUser

REM F mail.google.com





FOR /F "tokens=1,4,6,8,10,12 delims=^'" %%A in (Proxy.log) do (

	FOR /F "tokens=1-5 delims= " %%a in ("%%A") DO SET When=%%a %%b %%c %%d %%e

	ECHO !When!	%%D	%%B	%%C	%%F>>"%OutputFile%"

	)

	

START /MAX %SystemRoot%\system32\notepad.exe "%OutputFile%"

Open in new window

PROXYLOG.TXT
0
 
LVL 10

Author Comment

by:ReneGe
ID: 33740439
0
 
LVL 51

Expert Comment

by:Bill Prew
ID: 33740448
Well, for one thing, that proxylog.txt format is a lot different than the original post mentioned.

~bp
0
 
LVL 10

Author Comment

by:ReneGe
ID: 33740450
Also, FYI, the attached file should have been "Proxy.log" not "PROXYLOG.TXT"

Cheers
0
 
LVL 51

Expert Comment

by:Bill Prew
ID: 33740487
Given the new format, see if this works for you.

~bp
@ECHO OFF
SETLOCAL EnableDelayedExpansion
SET "SrcFile=C:\temp\PROXYLOG.TXT"
SET "DstFile=C:\temp\PROXYLOG.OUT"
IF EXIST "%DstFile%" DEL /F /Q "%DstFile%"
FOR /F "usebackq tokens=*" %%A IN ("%SrcFile%") DO (
  FOR %%B in (%%A) DO (
    FOR /F "tokens=1-2* delims=:" %%C IN ("%%B") DO (
      IF /I "%%~C" EQU "Website" (
        SET "RawData=%%~D"
        SET "RawData=!RawData:'=!"
        FOR %%B IN ("!RawData:.=","!") DO (
          SET "Host=!TopLvl!"
          SET "TopLvl=%%~B"
        )
        ECHO !Host!.!TopLvl!>>"%DstFile%"
      )
    )
  )
)
ENDLOCAL

Open in new window

0
 
LVL 51

Expert Comment

by:Bill Prew
ID: 33740493
Above modification produced the following for me:

ad.jp
hotmail.com
live.com
live.com
google.com
live.com
live.com

~bp
0
 
LVL 10

Author Comment

by:ReneGe
ID: 33740498
Sorry guys,

I guess I should go to sleep! (-5GMT)

Q's script works in deed.

I got confused and used the un-modified/parsed log file as input file.

You are all champs.

Thanks a million.

Cheers,
Rene
0
 
LVL 51

Accepted Solution

by:
Bill Prew earned 225 total points
ID: 33740500
And for safety sake I'd adjust the last loop variable...

~bp
@ECHO OFF
SETLOCAL EnableDelayedExpansion
SET "SrcFile=C:\temp\PROXYLOG.TXT"
SET "DstFile=C:\temp\PROXYLOG.OUT"
IF EXIST "%DstFile%" DEL /F /Q "%DstFile%"
FOR /F "usebackq tokens=*" %%A IN ("%SrcFile%") DO (
  FOR %%B in (%%A) DO (
    FOR /F "tokens=1-2* delims=:" %%C IN ("%%B") DO (
      IF /I "%%~C" EQU "Website" (
        SET "RawData=%%~D"
        SET "RawData=!RawData:'=!"
        FOR %%E IN ("!RawData:.=","!") DO (
          SET "Host=!TopLvl!"
          SET "TopLvl=%%~E"
        )
        ECHO !Host!.!TopLvl!>>"%DstFile%"
      )
    )
  )
)
ENDLOCAL

Open in new window

0
 
LVL 51

Expert Comment

by:Bill Prew
ID: 33740508
Doh, you're killing me...

~bp
0
 
LVL 10

Author Comment

by:ReneGe
ID: 33740509
Hmmm. ok bp, let me try this one.

One moment
0
 
LVL 51

Expert Comment

by:Bill Prew
ID: 33740531
And it's not that late yet...

(UTC-05:00) Eastern Time (US & Canada)

~bp
0
 
LVL 10

Author Comment

by:ReneGe
ID: 33740535
bp,

Your script worked without a spasm.

I am realy impressed!!

Thanks again for all your good work and help.

Cheers,
Rene
0
 
LVL 10

Author Comment

by:ReneGe
ID: 33740537
Montreal
0
 
LVL 10

Author Comment

by:ReneGe
ID: 33740540
You?
0
 
LVL 10

Author Closing Comment

by:ReneGe
ID: 33740645
Thanks again you all!!

You rock...

Cheers,
Rene
0
 
LVL 4

Expert Comment

by:rstjean
ID: 33740806
You are very generous.  Thanks Rene.   (just outside toronto)  Went to La Ronde this summer though for the fireworks.
0
 
LVL 11

Expert Comment

by:Ben Personick
ID: 33740876
Hey RG, thanks vMuch for the points!  I am also GMT -5  (New York)

@BP Great collaboration on this one mate! =)
0
 
LVL 51

Expert Comment

by:Bill Prew
ID: 33742720
==> ReneGe:

==> Montreal

Providence here (call it Boston if that doesn't mean anything to you...)

Thanks for points, grade, and stimulation.

~bp
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Entering a date in Microsoft Access can be tricky. A typo can cause month and day to be shuffled, entering the day only causes an error, as does entering, say, day 31 in June. This article shows how an inputmask supported by code can help the user a…
If you’re thinking to yourself “That description sounds a lot like two people doing the work that one could accomplish,” you’re not alone.
An introduction to basic programming syntax in Java by creating a simple program. Viewers can follow the tutorial as they create their first class in Java. Definitions and explanations about each element are given to help prepare viewers for future …
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now