Solved

Workaround for FOR /F Token Limit

Posted on 2009-03-29
13
2,963 Views
Last Modified: 2012-05-06
I am trying to read a text file with 50 fields into variables in a batch file under Windows XP Pro Service Pack 2.

I am running into a limit on the number of tokens FOR /F can address. The limit on my machine seems to be 30, although I have seen some discussion on the Web that it should be 64.

Is there a setting I need to tweak to be able to read more than 30 fields via the token method, or do I need to break each record/line up into two pieces?
0
Comment
Question by:Ballstonian
  • 6
  • 6
13 Comments
 
LVL 4

Accepted Solution

by:
ChristopherDunn earned 400 total points
ID: 24013897
As far as I know, there is a practical 26 token limit (unless you want to start using symbol variables, but even then there is a 31 token limit), but you can do multiple For commands to parse the first 25 and send the remaining items to another for. This example echos the first 50 fields of every line.
@echo off

For /f "tokens=1-25* delims= " %%A in (test.txt) do (

  echo %%A %%B %%C %%D %%E %%F %%G %%H %%I %%J %%K %%L %%M %%N %%O %%P %%Q %%R %%S %%T %%U %%V 
 

%%W %%X %%Y

  For /f "tokens=1-25 delims= " %%a in ("%%Z") do echo %%a %%b %%c %%d %%e %%f %%g %%h %%i %%j 
 

%%k %%l %%m %%n %%o %%p %%q %%r %%s %%t %%u %%v %%w %%x %%y

)

pause

Open in new window

0
 
LVL 4

Expert Comment

by:ChristopherDunn
ID: 24015582
Within the inner For loop, you can use any of the 50 variables as %%A to %%Y and %%a to %%y. If you wanted to parse more than 50, you'd have to enable delayed expansion at start assigning named variables.
0
 
LVL 16

Expert Comment

by:t0t0
ID: 24017310
Personally, I probably read in the whole line and parse it. Here's a possibly example.

@echo off
for /f "tokens=*" %%a in (test.txt) do call :process %%a
exit /b

:process
if "%1"=="" exit /b
echo %1
shift
goto :process
0
 
LVL 16

Expert Comment

by:t0t0
ID: 24017363
Here's another example....

@echo off
for /f "tokens=*" %%a in (test.txt) do call :process %%a
exit /b

:process
if "%1"=="" exit /b
rem Get first 5 items
set UserID=%1
set UserName=%2
set FirstName=%3
set LastName=%4
set Address=%5

rem Skip the next 20 items
for /l %%i in (1,1,25) Do shift

rem Get items 25 onwards...
set Department=%1
set ProductCode=%2
set ProductName=%3
rem Etc...

echo %UserID%, %UserName%, %FirstName%, %LastName%, %Address%
echo %Department%
echo %ProductCode%
echo %ProductName%
exit /b
0
 
LVL 16

Assisted Solution

by:t0t0
t0t0 earned 100 total points
ID: 24017412
And another example.....

@echo off
for /f "tokens=*" %%a in (test.txt) do call :process %%a

echo %var1%
echo %var2%
echo %var30%
echo %var59%
exit /b


:process
set count=0
:start
if "%1"=="" exit /b
set /a count+=1
set var%count%=%1
shift
goto :start
0
 
LVL 4

Expert Comment

by:ChristopherDunn
ID: 24022726
t0t0: that certainly is another way to do i, although it assumes that the fields are separated by spaces, which we do not know to be the case.

Whenever possible, I avoid delayed expansion and use subroutines as well. It is important to note, though that all the processing must be done within the subroutine or FOR loop. In your 3rd example, you would only be echoing the variables from the final line of the text file.

I took out the echoes to make it a little easier to read.


@echo off

For /f "tokens=1-25* delims= " %%A in (test.txt) do (

  For /f "tokens=1-25 delims= " %%a in ("%%Z") do (
 

  REM process your variables %%A-%%Y and %%a-%%y here.
 

  )

)
 

PAUSE

Open in new window

0
What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

 
LVL 4

Expert Comment

by:ChristopherDunn
ID: 24022844
Here's how I'd do it with a subroutine. In this example, the fields are comma separated and can include spaces.

Ballstonian, if this is not what you wanted, could we get an example of your text file and what you are trying to do, exactly?
===== test.txt =======================================

1 with a space,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50

A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,AA,BB,CC,DD,EE,FF,GG,HH,II,JJ,KK,LL,MM,NN,OO,PP,QQ,RR,SS,TT,UU,VV,WW,XX,YY with a space
 

===== test.bat =======================================

@echo off

For /f "tokens=1-25* delims=," %%A in (test.txt) do (

  For /f "tokens=1-25 delims=," %%a in ("%%Z") do (

 

  SET UserID=%%A

  SET Name=%%B
 

  REM ...
 

  SET Address=%%x

  SET Company=%%y
 

  call :process

 

  )

)

goto :eof
 

:process

echo UserID=%UserID%

echo Name=%Name%

echo Address=%Address%

echo Company=%Company%

PAUSE

goto :eof

Open in new window

0
 
LVL 16

Expert Comment

by:t0t0
ID: 24023050
ChristopherDunn

There is no use speculating over the contents of Ballstonian's file until he returns to us confirming it's layout.

There is absolutely nothing wrong with using delayed expansion - it's the proper way to use variables inside FOR loops - hence, no need to CALL out to a seperate procedure.

You cannot access more than 52 indiviual items of data directly from within FOR loops - you have to access all, or part of the line, and pass that on in a CALL as a parameter so that it's fields can be SHIFTed one at a time.

Admittedly, keeping track of each individual data item will require a robust method otherwise all sorts of things will go wrong. Good labelling of variables etc, is paramount.

It would be interesting to not whether fields are a fixed length or not and whether data is missing in some fields or not.

Ballstonian

Please supply a sample of your data file.
0
 
LVL 4

Expert Comment

by:ChristopherDunn
ID: 24024789
I agree. I would never attempt to work with 50 variables without having them appropriately named.

As for delayed expansion, I find it easier to work with and read if it isn't used, and it also prevents mishandling of exclamation points in input data. I prefer calls within FOR loops to keep the programming uniform. This is just a personal preference, and whenever it is simpler programatically, I do use delayed expansion.

Although it is true that you cannot directly access more than 52 single letter variables from within FOR loops, you can nest FORs and assign named variables at each nesting point, and then reuse the lettered variables.  I've tested this with up to 100 variables, but I'm sure it can do plenty more. Of course you'd need to use delayed expansion or call out to get the values of those variables.
0
 
LVL 16

Expert Comment

by:t0t0
ID: 24135980
Ballstonian

Please review comments number 24017310, 24017363 and 24017412 and award points accordingly.

Thank you
0
 
LVL 4

Expert Comment

by:ChristopherDunn
ID: 24136030
Ballstonian

Please review _all_ answers from _all_ contributors and award points for the solution(s) that solved your particular problem.

Thank you.
0
 
LVL 16

Expert Comment

by:t0t0
ID: 24136936
ChristopherDunn

Thank you for that.... I was suffering from tunnel-vision....

Hopefully, the question-asker will at least split the points if he feels there is no more we can help him with.

My only concern was to have the question closed.
0
 

Author Closing Comment

by:Ballstonian
ID: 31564043
Thanks to both of you for your great suggestions. I apologize for not being quicker in replying. I gravitated toward Christopher's solution b/c it just suited my own style a bit better---which does not reflect upon the quality of any other answers. And since I used a version of Christopher's, I gave him more points.
0

Featured Post

What Should I Do With This Threat Intelligence?

Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

Join & Write a Comment

We have adopted the strategy to use Computers in Student Labs as the bulletin boards. The same target can be achieved by using a Login Notice feature in Group policy but it’s not as attractive as graphical wallpapers with message which grabs the att…
It is only natural that we all want our PCs to be in good working order, improved system performance, so that is exactly how programs are advertised to entice. They say things like:            •      PC crashes? Get registry cleaner to repair it!    …
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now