Solved

Workaround for FOR /F Token Limit

Posted on 2009-03-29
13
2,978 Views
Last Modified: 2012-05-06
I am trying to read a text file with 50 fields into variables in a batch file under Windows XP Pro Service Pack 2.

I am running into a limit on the number of tokens FOR /F can address. The limit on my machine seems to be 30, although I have seen some discussion on the Web that it should be 64.

Is there a setting I need to tweak to be able to read more than 30 fields via the token method, or do I need to break each record/line up into two pieces?
0
Comment
Question by:Ballstonian
  • 6
  • 6
13 Comments
 
LVL 4

Accepted Solution

by:
ChristopherDunn earned 400 total points
ID: 24013897
As far as I know, there is a practical 26 token limit (unless you want to start using symbol variables, but even then there is a 31 token limit), but you can do multiple For commands to parse the first 25 and send the remaining items to another for. This example echos the first 50 fields of every line.
@echo off

For /f "tokens=1-25* delims= " %%A in (test.txt) do (

  echo %%A %%B %%C %%D %%E %%F %%G %%H %%I %%J %%K %%L %%M %%N %%O %%P %%Q %%R %%S %%T %%U %%V 
 

%%W %%X %%Y

  For /f "tokens=1-25 delims= " %%a in ("%%Z") do echo %%a %%b %%c %%d %%e %%f %%g %%h %%i %%j 
 

%%k %%l %%m %%n %%o %%p %%q %%r %%s %%t %%u %%v %%w %%x %%y

)

pause

Open in new window

0
 
LVL 4

Expert Comment

by:ChristopherDunn
ID: 24015582
Within the inner For loop, you can use any of the 50 variables as %%A to %%Y and %%a to %%y. If you wanted to parse more than 50, you'd have to enable delayed expansion at start assigning named variables.
0
 
LVL 16

Expert Comment

by:t0t0
ID: 24017310
Personally, I probably read in the whole line and parse it. Here's a possibly example.

@echo off
for /f "tokens=*" %%a in (test.txt) do call :process %%a
exit /b

:process
if "%1"=="" exit /b
echo %1
shift
goto :process
0
 
LVL 16

Expert Comment

by:t0t0
ID: 24017363
Here's another example....

@echo off
for /f "tokens=*" %%a in (test.txt) do call :process %%a
exit /b

:process
if "%1"=="" exit /b
rem Get first 5 items
set UserID=%1
set UserName=%2
set FirstName=%3
set LastName=%4
set Address=%5

rem Skip the next 20 items
for /l %%i in (1,1,25) Do shift

rem Get items 25 onwards...
set Department=%1
set ProductCode=%2
set ProductName=%3
rem Etc...

echo %UserID%, %UserName%, %FirstName%, %LastName%, %Address%
echo %Department%
echo %ProductCode%
echo %ProductName%
exit /b
0
 
LVL 16

Assisted Solution

by:t0t0
t0t0 earned 100 total points
ID: 24017412
And another example.....

@echo off
for /f "tokens=*" %%a in (test.txt) do call :process %%a

echo %var1%
echo %var2%
echo %var30%
echo %var59%
exit /b


:process
set count=0
:start
if "%1"=="" exit /b
set /a count+=1
set var%count%=%1
shift
goto :start
0
 
LVL 4

Expert Comment

by:ChristopherDunn
ID: 24022726
t0t0: that certainly is another way to do i, although it assumes that the fields are separated by spaces, which we do not know to be the case.

Whenever possible, I avoid delayed expansion and use subroutines as well. It is important to note, though that all the processing must be done within the subroutine or FOR loop. In your 3rd example, you would only be echoing the variables from the final line of the text file.

I took out the echoes to make it a little easier to read.


@echo off

For /f "tokens=1-25* delims= " %%A in (test.txt) do (

  For /f "tokens=1-25 delims= " %%a in ("%%Z") do (
 

  REM process your variables %%A-%%Y and %%a-%%y here.
 

  )

)
 

PAUSE

Open in new window

0
Enterprise Mobility and BYOD For Dummies

Like “For Dummies” books, you can read this in whatever order you choose and learn about mobility and BYOD; and how to put a competitive mobile infrastructure in place. Developed for SMBs and large enterprises alike, you will find helpful use cases, planning, and implementation.

 
LVL 4

Expert Comment

by:ChristopherDunn
ID: 24022844
Here's how I'd do it with a subroutine. In this example, the fields are comma separated and can include spaces.

Ballstonian, if this is not what you wanted, could we get an example of your text file and what you are trying to do, exactly?
===== test.txt =======================================

1 with a space,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50

A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,AA,BB,CC,DD,EE,FF,GG,HH,II,JJ,KK,LL,MM,NN,OO,PP,QQ,RR,SS,TT,UU,VV,WW,XX,YY with a space
 

===== test.bat =======================================

@echo off

For /f "tokens=1-25* delims=," %%A in (test.txt) do (

  For /f "tokens=1-25 delims=," %%a in ("%%Z") do (

 

  SET UserID=%%A

  SET Name=%%B
 

  REM ...
 

  SET Address=%%x

  SET Company=%%y
 

  call :process

 

  )

)

goto :eof
 

:process

echo UserID=%UserID%

echo Name=%Name%

echo Address=%Address%

echo Company=%Company%

PAUSE

goto :eof

Open in new window

0
 
LVL 16

Expert Comment

by:t0t0
ID: 24023050
ChristopherDunn

There is no use speculating over the contents of Ballstonian's file until he returns to us confirming it's layout.

There is absolutely nothing wrong with using delayed expansion - it's the proper way to use variables inside FOR loops - hence, no need to CALL out to a seperate procedure.

You cannot access more than 52 indiviual items of data directly from within FOR loops - you have to access all, or part of the line, and pass that on in a CALL as a parameter so that it's fields can be SHIFTed one at a time.

Admittedly, keeping track of each individual data item will require a robust method otherwise all sorts of things will go wrong. Good labelling of variables etc, is paramount.

It would be interesting to not whether fields are a fixed length or not and whether data is missing in some fields or not.

Ballstonian

Please supply a sample of your data file.
0
 
LVL 4

Expert Comment

by:ChristopherDunn
ID: 24024789
I agree. I would never attempt to work with 50 variables without having them appropriately named.

As for delayed expansion, I find it easier to work with and read if it isn't used, and it also prevents mishandling of exclamation points in input data. I prefer calls within FOR loops to keep the programming uniform. This is just a personal preference, and whenever it is simpler programatically, I do use delayed expansion.

Although it is true that you cannot directly access more than 52 single letter variables from within FOR loops, you can nest FORs and assign named variables at each nesting point, and then reuse the lettered variables.  I've tested this with up to 100 variables, but I'm sure it can do plenty more. Of course you'd need to use delayed expansion or call out to get the values of those variables.
0
 
LVL 16

Expert Comment

by:t0t0
ID: 24135980
Ballstonian

Please review comments number 24017310, 24017363 and 24017412 and award points accordingly.

Thank you
0
 
LVL 4

Expert Comment

by:ChristopherDunn
ID: 24136030
Ballstonian

Please review _all_ answers from _all_ contributors and award points for the solution(s) that solved your particular problem.

Thank you.
0
 
LVL 16

Expert Comment

by:t0t0
ID: 24136936
ChristopherDunn

Thank you for that.... I was suffering from tunnel-vision....

Hopefully, the question-asker will at least split the points if he feels there is no more we can help him with.

My only concern was to have the question closed.
0
 

Author Closing Comment

by:Ballstonian
ID: 31564043
Thanks to both of you for your great suggestions. I apologize for not being quicker in replying. I gravitated toward Christopher's solution b/c it just suited my own style a bit better---which does not reflect upon the quality of any other answers. And since I used a version of Christopher's, I gave him more points.
0

Featured Post

Comprehensive Backup Solutions for Microsoft

Acronis protects the complete Microsoft technology stack: Windows Server, Windows PC, laptop and Surface data; Microsoft business applications; Microsoft Hyper-V; Azure VMs; Microsoft Windows Server 2016; Microsoft Exchange 2016 and SQL Server 2016.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

You may have already been in the need to update a whole folder stucture using a script. Robocopy does it well and even provides a list of non-updated files in a log (if asked to). Generally those files that were locked by a user or a process by the …
I have published numerous articles here at Experts Exchange that present programs/scripts written in a language called AutoHotkey. Each of those articles has a brief paragraph describing where to download the product and how to install it. I have al…
This is a video describing the growing solar energy use in Utah. This is a topic that greatly interests me and so I decided to produce a video about it.
The Email Laundry PDF encryption service allows companies to send confidential encrypted  emails to anybody. The PDF document can also contain attachments that are embedded in the encrypted PDF. The password is randomly generated by The Email Laundr…

932 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now