Solved

Workaround for FOR /F Token Limit

Posted on 2009-03-29
13
3,053 Views
Last Modified: 2012-05-06
I am trying to read a text file with 50 fields into variables in a batch file under Windows XP Pro Service Pack 2.

I am running into a limit on the number of tokens FOR /F can address. The limit on my machine seems to be 30, although I have seen some discussion on the Web that it should be 64.

Is there a setting I need to tweak to be able to read more than 30 fields via the token method, or do I need to break each record/line up into two pieces?
0
Comment
Question by:Ballstonian
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 6
13 Comments
 
LVL 4

Accepted Solution

by:
ChristopherDunn earned 400 total points
ID: 24013897
As far as I know, there is a practical 26 token limit (unless you want to start using symbol variables, but even then there is a 31 token limit), but you can do multiple For commands to parse the first 25 and send the remaining items to another for. This example echos the first 50 fields of every line.
@echo off
For /f "tokens=1-25* delims= " %%A in (test.txt) do (
  echo %%A %%B %%C %%D %%E %%F %%G %%H %%I %%J %%K %%L %%M %%N %%O %%P %%Q %%R %%S %%T %%U %%V 
 
%%W %%X %%Y
  For /f "tokens=1-25 delims= " %%a in ("%%Z") do echo %%a %%b %%c %%d %%e %%f %%g %%h %%i %%j 
 
%%k %%l %%m %%n %%o %%p %%q %%r %%s %%t %%u %%v %%w %%x %%y
)
pause

Open in new window

0
 
LVL 4

Expert Comment

by:ChristopherDunn
ID: 24015582
Within the inner For loop, you can use any of the 50 variables as %%A to %%Y and %%a to %%y. If you wanted to parse more than 50, you'd have to enable delayed expansion at start assigning named variables.
0
 
LVL 16

Expert Comment

by:t0t0
ID: 24017310
Personally, I probably read in the whole line and parse it. Here's a possibly example.

@echo off
for /f "tokens=*" %%a in (test.txt) do call :process %%a
exit /b

:process
if "%1"=="" exit /b
echo %1
shift
goto :process
0
Forrester Webinar: xMatters Delivers 261% ROI

Guest speaker Dean Davison, Forrester Principal Consultant, explains how a Fortune 500 communication company using xMatters found these results: Achieved a 261% ROI, Experienced $753,280 in net present value benefits over 3 years and Reduced MTTR by 91% for tier 1 incidents.

 
LVL 16

Expert Comment

by:t0t0
ID: 24017363
Here's another example....

@echo off
for /f "tokens=*" %%a in (test.txt) do call :process %%a
exit /b

:process
if "%1"=="" exit /b
rem Get first 5 items
set UserID=%1
set UserName=%2
set FirstName=%3
set LastName=%4
set Address=%5

rem Skip the next 20 items
for /l %%i in (1,1,25) Do shift

rem Get items 25 onwards...
set Department=%1
set ProductCode=%2
set ProductName=%3
rem Etc...

echo %UserID%, %UserName%, %FirstName%, %LastName%, %Address%
echo %Department%
echo %ProductCode%
echo %ProductName%
exit /b
0
 
LVL 16

Assisted Solution

by:t0t0
t0t0 earned 100 total points
ID: 24017412
And another example.....

@echo off
for /f "tokens=*" %%a in (test.txt) do call :process %%a

echo %var1%
echo %var2%
echo %var30%
echo %var59%
exit /b


:process
set count=0
:start
if "%1"=="" exit /b
set /a count+=1
set var%count%=%1
shift
goto :start
0
 
LVL 4

Expert Comment

by:ChristopherDunn
ID: 24022726
t0t0: that certainly is another way to do i, although it assumes that the fields are separated by spaces, which we do not know to be the case.

Whenever possible, I avoid delayed expansion and use subroutines as well. It is important to note, though that all the processing must be done within the subroutine or FOR loop. In your 3rd example, you would only be echoing the variables from the final line of the text file.

I took out the echoes to make it a little easier to read.


@echo off
For /f "tokens=1-25* delims= " %%A in (test.txt) do (
  For /f "tokens=1-25 delims= " %%a in ("%%Z") do (
 
  REM process your variables %%A-%%Y and %%a-%%y here.
 
  )
)
 
PAUSE

Open in new window

0
 
LVL 4

Expert Comment

by:ChristopherDunn
ID: 24022844
Here's how I'd do it with a subroutine. In this example, the fields are comma separated and can include spaces.

Ballstonian, if this is not what you wanted, could we get an example of your text file and what you are trying to do, exactly?
===== test.txt =======================================
1 with a space,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50
A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,AA,BB,CC,DD,EE,FF,GG,HH,II,JJ,KK,LL,MM,NN,OO,PP,QQ,RR,SS,TT,UU,VV,WW,XX,YY with a space
 
===== test.bat =======================================
@echo off
For /f "tokens=1-25* delims=," %%A in (test.txt) do (
  For /f "tokens=1-25 delims=," %%a in ("%%Z") do (
 
  SET UserID=%%A
  SET Name=%%B
 
  REM ...
 
  SET Address=%%x
  SET Company=%%y
 
  call :process
 
  )
)
goto :eof
 
:process
echo UserID=%UserID%
echo Name=%Name%
echo Address=%Address%
echo Company=%Company%
PAUSE
goto :eof

Open in new window

0
 
LVL 16

Expert Comment

by:t0t0
ID: 24023050
ChristopherDunn

There is no use speculating over the contents of Ballstonian's file until he returns to us confirming it's layout.

There is absolutely nothing wrong with using delayed expansion - it's the proper way to use variables inside FOR loops - hence, no need to CALL out to a seperate procedure.

You cannot access more than 52 indiviual items of data directly from within FOR loops - you have to access all, or part of the line, and pass that on in a CALL as a parameter so that it's fields can be SHIFTed one at a time.

Admittedly, keeping track of each individual data item will require a robust method otherwise all sorts of things will go wrong. Good labelling of variables etc, is paramount.

It would be interesting to not whether fields are a fixed length or not and whether data is missing in some fields or not.

Ballstonian

Please supply a sample of your data file.
0
 
LVL 4

Expert Comment

by:ChristopherDunn
ID: 24024789
I agree. I would never attempt to work with 50 variables without having them appropriately named.

As for delayed expansion, I find it easier to work with and read if it isn't used, and it also prevents mishandling of exclamation points in input data. I prefer calls within FOR loops to keep the programming uniform. This is just a personal preference, and whenever it is simpler programatically, I do use delayed expansion.

Although it is true that you cannot directly access more than 52 single letter variables from within FOR loops, you can nest FORs and assign named variables at each nesting point, and then reuse the lettered variables.  I've tested this with up to 100 variables, but I'm sure it can do plenty more. Of course you'd need to use delayed expansion or call out to get the values of those variables.
0
 
LVL 16

Expert Comment

by:t0t0
ID: 24135980
Ballstonian

Please review comments number 24017310, 24017363 and 24017412 and award points accordingly.

Thank you
0
 
LVL 4

Expert Comment

by:ChristopherDunn
ID: 24136030
Ballstonian

Please review _all_ answers from _all_ contributors and award points for the solution(s) that solved your particular problem.

Thank you.
0
 
LVL 16

Expert Comment

by:t0t0
ID: 24136936
ChristopherDunn

Thank you for that.... I was suffering from tunnel-vision....

Hopefully, the question-asker will at least split the points if he feels there is no more we can help him with.

My only concern was to have the question closed.
0
 

Author Closing Comment

by:Ballstonian
ID: 31564043
Thanks to both of you for your great suggestions. I apologize for not being quicker in replying. I gravitated toward Christopher's solution b/c it just suited my own style a bit better---which does not reflect upon the quality of any other answers. And since I used a version of Christopher's, I gave him more points.
0

Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Issue: Unstable cursor in Windows XP and Windows runs extremely slow in that any click will bring up the Hour glass (sometimes for several seconds before giving you what you want) . Troubleshooting Process and the FINAL FIX: This issue see…
Today, still in the boom of Apple, PC's and products, nearly 50% of the computer users use Windows as graphical operating systems. If you are among those users who love windows, but are grappling to keep the system's hard drive optimized, then you s…
Two types of users will appreciate AOMEI Backupper Pro: 1 - Those with PCIe drives (and haven't found cloning software that works on them). 2 - Those who want a fast clone of their boot drive (no re-boots needed) and it can clone your drive wh…
Finding and deleting duplicate (picture) files can be a time consuming task. My wife and I, our three kids and their families all share one dilemma: Managing our pictures. Between desktops, laptops, phones, tablets, and cameras; over the last decade…

730 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question