• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1151
  • Last Modified:

BatchFile: "For /F" to get domain name

Hi there,

Using a batch file, I need to get the domain names out of the following log file.

PROXYLOG.TXT:
Tue 20 Sep 2010 11:13:01      192.168.11.130      HTTP-Proxy      8080      google.com
Tue 20 Sep 2010 11:13:01      192.168.11.130      HTTP-Proxy      8080      www.google.ca

EXPECTED OUTPUT:
google.com
google.ca


Thanks for your help,
Rene
0
ReneGe
Asked:
ReneGe
  • 13
  • 11
  • 5
  • +2
4 Solutions
 
ragnarok89Commented:
download the free tool "cut" from http://www.ltr-data.se/opencode.html

then use the command "type proxylog.txt | cut -F:9 > domains.txt" (9 is a field number, or column number)

you can then use a regex to remove all instances of "www." from the domains.txt file
0
 
ReneGeAuthor Commented:
ragnarok89,

I do not wish to use utils.

I need this script to be done in batch file.

Thanks,
Rene
0
 
rstjeanCommented:
copy getdomains.txt to getdomains.vbs

write a batch file with the following

getdomains.vbs [path of log] [path of output file]
getdomains.txt
0
Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
ReneGeAuthor Commented:
rstjean,

Thanks for the VBScript.

I'm realy looking for a Windows Batch File.

Cheers,
Rene

0
 
rstjeanCommented:
You can't do it in a batch file without calling a program or script.  Findstr will only return the full line, and will not return a partial match.

You can use a batch file to call the vb script.  Remove the .txt from the end, and change the values with the locations of your server.





getdomains.bat.txt
0
 
Bill PrewCommented:
How about this in a BAT file?

Adjust the two sets at the top to your in and out file desired.

~bp
@echo off
set InFile=c:\temp\EE26491259.in
set OutFile=c:\temp\EE26491259.out
if exist "%OutFile%" del "%OutFile%"
for /F "usebackq tokens=9*" %%A in ("%InFile%") do echo %%A>>"%OutFile%"

Open in new window

0
 
ragnarok89Commented:
Rene,

Will the URL always start at the 77 character of each line?
0
 
ragnarok89Commented:
@echo off
del domains.txt
for /F "usebackq tokens=9*" %%A in (PROXYLOG.TXT) do call :parse %%A

:parse
set str=%1
set str=%str:www.=%
echo.%str% >> domains.txt

This will do it all.

0
 
Bill PrewCommented:
Ah, I missed the removal of the www., this adjustment to my approach should handle that.

~bp
@echo off
setlocal EnableDelayedExpansion
set InFile=c:\temp\EE26491259.in
set OutFile=c:\temp\EE26491259.out
if exist "%OutFile%" del "%OutFile%"
for /F "usebackq tokens=9*" %%A in ("%InFile%") do (
  set Domain=%%A
  echo !Domain:www.=!>>"%OutFile%"
)

Open in new window

0
 
ReneGeAuthor Commented:
Hi guys,

As you know, FQDN may contain all sots of stufs:
web.whatever.on.adomain.on.ca

So whever it parses, it must output the last two tokens of "."
So the this example, the output would be: on.ca

So removing www would be pointless is this case.

Thanks to you all and cheers,
Rene
0
 
ReneGeAuthor Commented:
By the way, each colones are separated by a TAB.

Cheers
0
 
Bill PrewCommented:
Give this a try.

~bp
@echo off
setlocal EnableDelayedExpansion
set InFile=c:\temp\EE26491259.in
set OutFile=c:\temp\EE26491259.out
if exist "%OutFile%" del "%OutFile%"
for /F "usebackq tokens=9*" %%A in ("%InFile%") do (
  call :GetDomain %%A
  echo !Domain!>>"%OutFile%"
)
 
:GetDomain [host-address]
  for /F "tokens=1-20 delims=." %%A in ("%~1") do (
    if "%%T" NEQ "" (set Domain=%%S.%%T & exit /b)
    if "%%S" NEQ "" (set Domain=%%R.%%S & exit /b)
    if "%%R" NEQ "" (set Domain=%%Q.%%R & exit /b)
    if "%%Q" NEQ "" (set Domain=%%P.%%Q & exit /b)
    if "%%P" NEQ "" (set Domain=%%O.%%P & exit /b)
    if "%%O" NEQ "" (set Domain=%%N.%%O & exit /b)
    if "%%N" NEQ "" (set Domain=%%M.%%N & exit /b)
    if "%%M" NEQ "" (set Domain=%%L.%%M & exit /b)
    if "%%L" NEQ "" (set Domain=%%K.%%L & exit /b)
    if "%%K" NEQ "" (set Domain=%%J.%%K & exit /b)
    if "%%J" NEQ "" (set Domain=%%I.%%J & exit /b)
    if "%%I" NEQ "" (set Domain=%%H.%%I & exit /b)
    if "%%H" NEQ "" (set Domain=%%G.%%H & exit /b)
    if "%%G" NEQ "" (set Domain=%%F.%%G & exit /b)
    if "%%F" NEQ "" (set Domain=%%E.%%F & exit /b)
    if "%%E" NEQ "" (set Domain=%%D.%%E & exit /b)
    if "%%D" NEQ "" (set Domain=%%C.%%D & exit /b)
    if "%%C" NEQ "" (set Domain=%%B.%%C & exit /b)
    if "%%B" NEQ "" (set Domain=%%A.%%B & exit /b)
    exit /b
  )

Open in new window

0
 
Ben Personick (Previously QCubed)Lead Network EngineerCommented:
I am submitting a modified version of Bill Prew's Script here, because I have had to deal with a similar issue before and realised a better way to handle it woudl be as attached.

The difference is that my version of Bill's script will allow you to parse any domain name no matter how many sub terms it has, while Bill's unfortunately cannot handle more than 39.

 (I actually ran into that limitation which is what made me realise this would be a better way to handle these types of recursions in the future)
Note:  My code will Solve your Question  more completely however half the credit is Bill Prews.

@ECHO OFF
SETLOCAL EnableDelayedExpansion
SET "SrcFile=C:\Admin\ParseProxyLog\PROXYLOG.TXT"
SET "DstFile=C:\Admin\ParseProxyLog\DomainList.log"
IF EXIST "%DstFile%" DEL /F /Q "%DstFile%"
for /F "usebackq tokens=9*" %%A in ("%SrcFile%") do (
  CALL :GetDomain "%%A"
  ECHO !Host!.!TopLvl!>>"%DstFile%"
)
 
:GetDomain [host-address]
	SET "RawData=%~1"
	FOR %%A IN ("%RawData:.=","%") DO (
	SET "Host=!TopLvl!"
	SET "TopLvl=%%~A"
	)
ENDLOCAL

Open in new window

0
 
Ben Personick (Previously QCubed)Lead Network EngineerCommented:
Also I couldn't resist tightening this up, since we're already using delayed expansion we don't have to bother with calling the function at all, so this code makes more sence given the circumstances.

Again this is a hybrid of Bill's work and my own.



@ECHO OFF
SETLOCAL EnableDelayedExpansion
SET "SrcFile=C:\Admin\ParseProxyLog\PROXYLOG.TXT"
SET "DstFile=C:\Admin\ParseProxyLog\DomainList.log"
IF EXIST "%DstFile%" DEL /F /Q "%DstFile%"
FOR /F "usebackq tokens=9*" %%A IN ("%SrcFile%") DO (
	SET "RawData=%%~A"
	FOR %%B IN ("!RawData:.=","!") DO (
		SET "Host=!TopLvl!"
		SET "TopLvl=%%~B"
	)
	ECHO !Host!.!TopLvl!
)
ENDLOCAL

Open in new window

0
 
Ben Personick (Previously QCubed)Lead Network EngineerCommented:
heh!

Sorry the above post does work, but does not write to the file because when I was testing it was easier to spit the results to the command prompt instead of checking the file each time through.

  Attached is the above code with output writing to the file again.

@ECHO OFF
SETLOCAL EnableDelayedExpansion
SET "SrcFile=C:\Admin\ParseProxyLog\PROXYLOG.TXT"
SET "DstFile=C:\Admin\ParseProxyLog\DomainList.log"
IF EXIST "%DstFile%" DEL /F /Q "%DstFile%"
FOR /F "usebackq tokens=9*" %%A IN ("%SrcFile%") DO (
        SET "RawData=%%~A"
        FOR %%B IN ("!RawData:.=","!") DO (
                SET "Host=!TopLvl!"
                SET "TopLvl=%%~B"
        )
        ECHO !Host!.!TopLvl!>>"%DstFile%"
)
ENDLOCAL

Open in new window

0
 
Bill PrewCommented:
Looking good QC.

~bp
0
 
Ben Personick (Previously QCubed)Lead Network EngineerCommented:
Right back at cha BP =)

~Q
0
 
ReneGeAuthor Commented:
Hey Q, bp

I'v tried Q's latest script and I only got the following as output:

.:
:.:
:.:
:.:

The original log file is attached. I modifed it with the following script, which I get the PROXYLOG.TXT content.

Hoping this helps.

Thanks a lot and cheers,
Rene
@ECHO OFF
SETLOCAL ENABLEDELAYEDEXPANSION

Set OutputFile=%~n0.txt

IF EXIST "%OutputFile%" DEL "%OutputFile%"


REM %%A=WHEN
REM %%B=PROTOCOL
REM %%C=PORT
REM %%D=USER IP ADDRESS
REM %%E=USERNAME (Setup in proxy)
REM %%F=DESTINATION FQDN

REM EXAMPLE
REM -----------------------------------------------
REM A Tue 21 Sep 2010 10:21:09 : CONN STAT : Instance:
REM B HTTP-Proxy
REM C 8080
REM D 192.168.2.127
REM E DefaultUser
REM F mail.google.com


FOR /F "tokens=1,4,6,8,10,12 delims=^'" %%A in (Proxy.log) do (
	FOR /F "tokens=1-5 delims= " %%a in ("%%A") DO SET When=%%a %%b %%c %%d %%e
	ECHO !When!	%%D	%%B	%%C	%%F>>"%OutputFile%"
	)
	
START /MAX %SystemRoot%\system32\notepad.exe "%OutputFile%"

Open in new window

PROXYLOG.TXT
0
 
ReneGeAuthor Commented:
0
 
Bill PrewCommented:
Well, for one thing, that proxylog.txt format is a lot different than the original post mentioned.

~bp
0
 
ReneGeAuthor Commented:
Also, FYI, the attached file should have been "Proxy.log" not "PROXYLOG.TXT"

Cheers
0
 
Bill PrewCommented:
Given the new format, see if this works for you.

~bp
@ECHO OFF
SETLOCAL EnableDelayedExpansion
SET "SrcFile=C:\temp\PROXYLOG.TXT"
SET "DstFile=C:\temp\PROXYLOG.OUT"
IF EXIST "%DstFile%" DEL /F /Q "%DstFile%"
FOR /F "usebackq tokens=*" %%A IN ("%SrcFile%") DO (
  FOR %%B in (%%A) DO (
    FOR /F "tokens=1-2* delims=:" %%C IN ("%%B") DO (
      IF /I "%%~C" EQU "Website" (
        SET "RawData=%%~D"
        SET "RawData=!RawData:'=!"
        FOR %%B IN ("!RawData:.=","!") DO (
          SET "Host=!TopLvl!"
          SET "TopLvl=%%~B"
        )
        ECHO !Host!.!TopLvl!>>"%DstFile%"
      )
    )
  )
)
ENDLOCAL

Open in new window

0
 
Bill PrewCommented:
Above modification produced the following for me:

ad.jp
hotmail.com
live.com
live.com
google.com
live.com
live.com

~bp
0
 
ReneGeAuthor Commented:
Sorry guys,

I guess I should go to sleep! (-5GMT)

Q's script works in deed.

I got confused and used the un-modified/parsed log file as input file.

You are all champs.

Thanks a million.

Cheers,
Rene
0
 
Bill PrewCommented:
And for safety sake I'd adjust the last loop variable...

~bp
@ECHO OFF
SETLOCAL EnableDelayedExpansion
SET "SrcFile=C:\temp\PROXYLOG.TXT"
SET "DstFile=C:\temp\PROXYLOG.OUT"
IF EXIST "%DstFile%" DEL /F /Q "%DstFile%"
FOR /F "usebackq tokens=*" %%A IN ("%SrcFile%") DO (
  FOR %%B in (%%A) DO (
    FOR /F "tokens=1-2* delims=:" %%C IN ("%%B") DO (
      IF /I "%%~C" EQU "Website" (
        SET "RawData=%%~D"
        SET "RawData=!RawData:'=!"
        FOR %%E IN ("!RawData:.=","!") DO (
          SET "Host=!TopLvl!"
          SET "TopLvl=%%~E"
        )
        ECHO !Host!.!TopLvl!>>"%DstFile%"
      )
    )
  )
)
ENDLOCAL

Open in new window

0
 
Bill PrewCommented:
Doh, you're killing me...

~bp
0
 
ReneGeAuthor Commented:
Hmmm. ok bp, let me try this one.

One moment
0
 
Bill PrewCommented:
And it's not that late yet...

(UTC-05:00) Eastern Time (US & Canada)

~bp
0
 
ReneGeAuthor Commented:
bp,

Your script worked without a spasm.

I am realy impressed!!

Thanks again for all your good work and help.

Cheers,
Rene
0
 
ReneGeAuthor Commented:
Montreal
0
 
ReneGeAuthor Commented:
You?
0
 
ReneGeAuthor Commented:
Thanks again you all!!

You rock...

Cheers,
Rene
0
 
rstjeanCommented:
You are very generous.  Thanks Rene.   (just outside toronto)  Went to La Ronde this summer though for the fireworks.
0
 
Ben Personick (Previously QCubed)Lead Network EngineerCommented:
Hey RG, thanks vMuch for the points!  I am also GMT -5  (New York)

@BP Great collaboration on this one mate! =)
0
 
Bill PrewCommented:
==> ReneGe:

==> Montreal

Providence here (call it Boston if that doesn't mean anything to you...)

Thanks for points, grade, and stimulation.

~bp
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Cloud Class® Course: Microsoft Azure 2017

Azure has a changed a lot since it was originally introduce by adding new services and features. Do you know everything you need to about Azure? This course will teach you about the Azure App Service, monitoring and application insights, DevOps, and Team Services.

  • 13
  • 11
  • 5
  • +2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now