?
Solved

Workaround for FOR /F Token Limit

Posted on 2009-03-29
13
Medium Priority
?
3,149 Views
Last Modified: 2012-05-06
I am trying to read a text file with 50 fields into variables in a batch file under Windows XP Pro Service Pack 2.

I am running into a limit on the number of tokens FOR /F can address. The limit on my machine seems to be 30, although I have seen some discussion on the Web that it should be 64.

Is there a setting I need to tweak to be able to read more than 30 fields via the token method, or do I need to break each record/line up into two pieces?
0
Comment
Question by:Ballstonian
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 6
13 Comments
 
LVL 4

Accepted Solution

by:
ChristopherDunn earned 1600 total points
ID: 24013897
As far as I know, there is a practical 26 token limit (unless you want to start using symbol variables, but even then there is a 31 token limit), but you can do multiple For commands to parse the first 25 and send the remaining items to another for. This example echos the first 50 fields of every line.
@echo off
For /f "tokens=1-25* delims= " %%A in (test.txt) do (
  echo %%A %%B %%C %%D %%E %%F %%G %%H %%I %%J %%K %%L %%M %%N %%O %%P %%Q %%R %%S %%T %%U %%V 
 
%%W %%X %%Y
  For /f "tokens=1-25 delims= " %%a in ("%%Z") do echo %%a %%b %%c %%d %%e %%f %%g %%h %%i %%j 
 
%%k %%l %%m %%n %%o %%p %%q %%r %%s %%t %%u %%v %%w %%x %%y
)
pause

Open in new window

0
 
LVL 4

Expert Comment

by:ChristopherDunn
ID: 24015582
Within the inner For loop, you can use any of the 50 variables as %%A to %%Y and %%a to %%y. If you wanted to parse more than 50, you'd have to enable delayed expansion at start assigning named variables.
0
 
LVL 16

Expert Comment

by:t0t0
ID: 24017310
Personally, I probably read in the whole line and parse it. Here's a possibly example.

@echo off
for /f "tokens=*" %%a in (test.txt) do call :process %%a
exit /b

:process
if "%1"=="" exit /b
echo %1
shift
goto :process
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 16

Expert Comment

by:t0t0
ID: 24017363
Here's another example....

@echo off
for /f "tokens=*" %%a in (test.txt) do call :process %%a
exit /b

:process
if "%1"=="" exit /b
rem Get first 5 items
set UserID=%1
set UserName=%2
set FirstName=%3
set LastName=%4
set Address=%5

rem Skip the next 20 items
for /l %%i in (1,1,25) Do shift

rem Get items 25 onwards...
set Department=%1
set ProductCode=%2
set ProductName=%3
rem Etc...

echo %UserID%, %UserName%, %FirstName%, %LastName%, %Address%
echo %Department%
echo %ProductCode%
echo %ProductName%
exit /b
0
 
LVL 16

Assisted Solution

by:t0t0
t0t0 earned 400 total points
ID: 24017412
And another example.....

@echo off
for /f "tokens=*" %%a in (test.txt) do call :process %%a

echo %var1%
echo %var2%
echo %var30%
echo %var59%
exit /b


:process
set count=0
:start
if "%1"=="" exit /b
set /a count+=1
set var%count%=%1
shift
goto :start
0
 
LVL 4

Expert Comment

by:ChristopherDunn
ID: 24022726
t0t0: that certainly is another way to do i, although it assumes that the fields are separated by spaces, which we do not know to be the case.

Whenever possible, I avoid delayed expansion and use subroutines as well. It is important to note, though that all the processing must be done within the subroutine or FOR loop. In your 3rd example, you would only be echoing the variables from the final line of the text file.

I took out the echoes to make it a little easier to read.


@echo off
For /f "tokens=1-25* delims= " %%A in (test.txt) do (
  For /f "tokens=1-25 delims= " %%a in ("%%Z") do (
 
  REM process your variables %%A-%%Y and %%a-%%y here.
 
  )
)
 
PAUSE

Open in new window

0
 
LVL 4

Expert Comment

by:ChristopherDunn
ID: 24022844
Here's how I'd do it with a subroutine. In this example, the fields are comma separated and can include spaces.

Ballstonian, if this is not what you wanted, could we get an example of your text file and what you are trying to do, exactly?
===== test.txt =======================================
1 with a space,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50
A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,AA,BB,CC,DD,EE,FF,GG,HH,II,JJ,KK,LL,MM,NN,OO,PP,QQ,RR,SS,TT,UU,VV,WW,XX,YY with a space
 
===== test.bat =======================================
@echo off
For /f "tokens=1-25* delims=," %%A in (test.txt) do (
  For /f "tokens=1-25 delims=," %%a in ("%%Z") do (
 
  SET UserID=%%A
  SET Name=%%B
 
  REM ...
 
  SET Address=%%x
  SET Company=%%y
 
  call :process
 
  )
)
goto :eof
 
:process
echo UserID=%UserID%
echo Name=%Name%
echo Address=%Address%
echo Company=%Company%
PAUSE
goto :eof

Open in new window

0
 
LVL 16

Expert Comment

by:t0t0
ID: 24023050
ChristopherDunn

There is no use speculating over the contents of Ballstonian's file until he returns to us confirming it's layout.

There is absolutely nothing wrong with using delayed expansion - it's the proper way to use variables inside FOR loops - hence, no need to CALL out to a seperate procedure.

You cannot access more than 52 indiviual items of data directly from within FOR loops - you have to access all, or part of the line, and pass that on in a CALL as a parameter so that it's fields can be SHIFTed one at a time.

Admittedly, keeping track of each individual data item will require a robust method otherwise all sorts of things will go wrong. Good labelling of variables etc, is paramount.

It would be interesting to not whether fields are a fixed length or not and whether data is missing in some fields or not.

Ballstonian

Please supply a sample of your data file.
0
 
LVL 4

Expert Comment

by:ChristopherDunn
ID: 24024789
I agree. I would never attempt to work with 50 variables without having them appropriately named.

As for delayed expansion, I find it easier to work with and read if it isn't used, and it also prevents mishandling of exclamation points in input data. I prefer calls within FOR loops to keep the programming uniform. This is just a personal preference, and whenever it is simpler programatically, I do use delayed expansion.

Although it is true that you cannot directly access more than 52 single letter variables from within FOR loops, you can nest FORs and assign named variables at each nesting point, and then reuse the lettered variables.  I've tested this with up to 100 variables, but I'm sure it can do plenty more. Of course you'd need to use delayed expansion or call out to get the values of those variables.
0
 
LVL 16

Expert Comment

by:t0t0
ID: 24135980
Ballstonian

Please review comments number 24017310, 24017363 and 24017412 and award points accordingly.

Thank you
0
 
LVL 4

Expert Comment

by:ChristopherDunn
ID: 24136030
Ballstonian

Please review _all_ answers from _all_ contributors and award points for the solution(s) that solved your particular problem.

Thank you.
0
 
LVL 16

Expert Comment

by:t0t0
ID: 24136936
ChristopherDunn

Thank you for that.... I was suffering from tunnel-vision....

Hopefully, the question-asker will at least split the points if he feels there is no more we can help him with.

My only concern was to have the question closed.
0
 

Author Closing Comment

by:Ballstonian
ID: 31564043
Thanks to both of you for your great suggestions. I apologize for not being quicker in replying. I gravitated toward Christopher's solution b/c it just suited my own style a bit better---which does not reflect upon the quality of any other answers. And since I used a version of Christopher's, I gave him more points.
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When you receive another warning that your shared drive is almost full and you have asked your users to clean out old files again and again, here is a single command that may help. This command will place all the files that have not been used rec…
This is a fine trick which I've found useful many times, when you just don't want to accidentally run a batch script or the commands needs administrator rights.
Two types of users will appreciate AOMEI Backupper Pro: 1 - Those with PCIe drives (and haven't found cloning software that works on them). 2 - Those who want a fast clone of their boot drive (no re-boots needed) and it can clone your drive wh…
Michael from AdRem Software explains how to view the most utilized and worst performing nodes in your network, by accessing the Top Charts view in NetCrunch network monitor (https://www.adremsoft.com/). Top Charts is a view in which you can set seve…
Suggested Courses

765 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question