Modify Python Script for more automation

Hello, I have been using the code below with success with ESRI ArcMap 9.2. However, I'd like to know how to add some more automation to this script.  Currently I run this from the command line like this:

C:\Python24\python "C:\Documents and Settings\user\Desktop\test.py" LANDCOVERfileLIST DONUTfileLIST

So each time I want to run this I need to create a list of files to use and then enter their location in the statement above and run. I also change the 'new_name' field each time in the body of the script.  I need the count appended to the end to be the way it is, it currently matches the ID of the files.  I have six (and counting)  sets of landcover files each in their own folder and each with a unique name and numbered 1 - 71 and then I have one set of Donut files each numbered 1 -71. So in the end I'll have 6 sets of files now named new_name01, newname02, etc. How can I set this up to be more automatic?

Can I?
-make so I just use the folders where the files are located and have a name derived from folder name?
-make a long list with file locations and somehow have the outnames change for every set and the numbers stay in sync?
-do this someother way that is more elegant than I currently think of?

I appreciate annotated code  and instructions so that I can learn more about python, etc.
Thank you,
MJ

# Multi_Output_Map_Algebra_sample.py
# Description:
#   Runs an expression built with the Map Algebra language.
# Requirements: None
import sys, arcgisscripting
#import sys

# Create the Geoprocessor object
gp = arcgisscripting.create()
f = open(sys.argv[1],'r')
circlefile = open(sys.argv[2],'r')
count = 1
tmpdir = 'c:\\temp\\'
try:
            for landcover in f:
                  donut = circlefile.readline()
                  
                  # Set local variables
                  Outfname = os.path.join(tmpdir, 'new_name', str(count))
                  InExpression = "CON(ISNULL(" + donut.strip() + ")," + landcover.strip() + "," + landcover.strip() + " * -1)"
                  print InExpression
                  # Check out Spatial Analyst extension license
                  gp.CheckOutExtension("Spatial")

                  # Process: MapAlgebra
                  gp.SingleOutputMapAlgebra_sa(InExpression, Outfile)

                  count = count + 1
            f.close
            circlefile.close
except:
    # If an error occurred while running a tool, then print the messages.
    print gp.GetMessages()
mgjustAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

ramromconsultant Commented:
You are asking for a lot, and your specs are, for me, hard to understand.

I suggest you give examples for your first 2 bullets, and then define "elegant".

Also when you enclose a lot of code in a try block it's a good idea to capture and report the traceback. You are assuming that the only exceptions will be raised by the Geoprocessor object.

The code here is taken from the manual for the traceback module.
import sys, traceback
 
try:
    # your code
except:
    print "Exception in user code:"
    print '-'*60
    traceback.print_exc(file=sys.stdout)
    print '-'*60

Open in new window

0
mgjustAuthor Commented:
Ramron,

Thank you.  I am unfamiliar with most of this --  a traceback will tell me if something went wrong anywhere in the code?

Above in my original question, it should say:
Can I:
-make it so I can just indicate in the code to use the folders where the files are located and have a name derived from folder name? (loop through until finished)
or
-make a long list (test file) with file locations for all sets and somehow have the outputnames change for every set and the numbers stay in sync i.e. 0-71? Like have something to make the counter reset every 71 times and with this change the output name as well?
or
-do this some other way that is more elegant than I can currently think of?

I felt that someone else might now of better way to do what I was asking, hence "more elegant."

I want to know if there is way that I can use this working code, so that I can run multiple 'sets' of data more quickly with the naming convention still intact.

This is spatial data and each numbered file shares a common center with all other files that contain that same number. I am performing the procedure above to create a new set of data.

 I have the LANDCOVER data and the DONUT data. There are multiple sets of LANDCOVER data each with different attributes but share the same place in space, the DONUT data is 'generic' and does not change between LANDCOVER sets for this operation.

Set A:
"CON(ISNULL(donut01), landcoverA_01, landcoverA_01 * -1)" = resultA_01
"CON(ISNULL(donut02), landcoverA_02, landcoverA_02 * -1)" = resultA_02
etc
Set B:
"CON(ISNULL(donut01), landcoverB_01, landcoverB_01 * -1)" = resultB_01
etc

I was just hoping to automate the process, as the number of sets I will have is increasing rapidly.

Thank you kindly,
MJ
0
peprCommented:
Hi mgjust. It definitely is possible to enhance your script; however, I need to know more about the situation. Could you show how the content of the LANDCOVERfileLIST and the DONUTfileLIST looks like? (Can be shortened. Possibly explain the regularities in file names if necessary.)

Could you show how the structure of the input and of the generated subdirectories and files really look like? Try to run the cmd window and type in your subdirectory...

 dir /b /s *.* >files.txt

and post the fragments (for brevity) of the content of files.txt. What files are the input files (i.e. created somehow outside the script) and what files should be generated?

If the script test.py is closely related to the processed directories, it could be better to locate it inside the directory or to give it some relative location to the directory. Then the script itself can compute the path to the files if some fixed rules exist.

You have the bugs in the above script. The commands

            f.close
            circlefile.close

do not work for you because you forgot to add () this way

            f.close()
            circlefile.close()

For your bullets:

- Yes. But you should explain how the result should look like.
- Yes. But it would be better if you could show the short example what you exactly mean.
- Probably yes. It is possible to get the list of existing file in a directory. It is possible to process the strings with the names of the files. Maybe, generating some auxiliary files (with lists of other files) and reading them back can be avoided by generating the to be read content on the fly.

Let's start with clarification and solving of the parts.

pepr
0
Exploring SQL Server 2016: Fundamentals

Learn the fundamentals of Microsoft SQL Server, a relational database management system that stores and retrieves data when requested by other software applications.

mgjustAuthor Commented:
Hello,
I thank you for your effort in advance.

>Could you show how the content of the LANDCOVERfileLIST and the DONUTfileLIST looks like?

LANDCOVERfileLIST is a text file and its contents look like this, which are just locations of where the files are located:

C:\Giswork\LandoverA\landcoverA_01
C:\Giswork\LandoverA\landcoverA_02
...
C:\Giswork\LandoverA\landcoverA_71
C:\Giswork\LandoverB\landcoverB_01
C:\Giswork\LandoverB\landcoverB_02
...
C:\Giswork\LandoverB\landcoverB_71
C:\Giswork\LandoverC\landcoverC_01
etc

The DONUTfileLIST is very similar, only there is exactly one set of data that goes from Donut01 to Donut 71

C:\Giswork\DONUT\DONUT_01
C:\Giswork\DONUT\DONUT_02
...
C:\Giswork\DONUT\DONUT_71

These 'landcover' and 'donut' sub directories  (i.e. landcoverA_01, Donut_01, etc) are folders that contain eight files.  
However, ArcMap recognizes and uses them as a single unit/file.  The content of each of these subdirectories for both 'landcover' and 'donut' is seven ".adf" files and one "log" file.  They are raster files, known in ESRI ArcMap jargon as ESRI grid files (http://en.wikipedia.org/wiki/ESRI_grid).

>Could you show how the structure of the input and of the generated subdirectories and files really look like?
Honestly, I am unsure of what you are asking me to do with the cmd window.

>What files are the input files (i.e. created somehow outside the script) ?
All of the aforementioned Landcover files (01-71 for many "sets") and the donut files (0-71).

>[...] and what files should be generated?
The files to be generated are those which I dubbed "resultA_01, resultA_02, ... resultB_01, etc) these too are of the same style of the input files an ESRI grid file.

As for the bugs provided in the script, I will fix them. However, I would like to note that it currently provides me with a proper result.

So right now, given that I currently have six sets of data (soon to be more)

Currently I use this command six times:
C:\Python24\python "C:\Documents and Settings\user\Desktop\test.py" LANDCOVERfileLIST DONUTfileLIST

Each time I run this code I  change the LANDCOVERfileList and the 'newname' part of the line of code "Outfname = os.path.join(tmpdir, 'new_name', str(count))". The one set of DONUT files are to be used for all sets of LANDCOVER data. I need each set of RESULT data to be enumerated from 01-71. Its a one to one process (LandcoverA_01 with donut_01 to produce resultA_01, LandcoverA_02 with donut_02 to produce resultA_02, LandcoverB_02 with donut_01 to produce resultB_01, etc).  I was hoping that as my sets of data get increasingly large that I  could just have code that would generate these things that I now must change between each set of data.

Thank you kindly for the help & I hope this clarifies what I want to do,
Cheers,
MJ
0
peprCommented:
Are the resultX_nn directories or files located in tmpdir? I suggest to create the auxiliary structure of subdirectories in 'c:/GisworkAux' and simulate the activity here (I do not have the ArcMap software and your data).

Please name the following script, say, dirsgen.py and execute it. Then check the newly created 'c:/GisworkAux' directory for the content.

Final question: Are your original subdirectories named

C:\Giswork\LandoverA\landcoverA_01
or
C:\Giswork\LandcoverA\landcoverA_01
                          A--- see the c here

?
# This is an auxiliary script that generates the strucrure 
# of subdirectories for testing the solution.
# 
import os
 
gisworkDir = 'c:/GisworkAux'
 
# Create the top-level Landcover directories with
# letter appended (if they do not exist).
for c in 'ABCDE':
    # (Check possible typo -- Landover in the original?)
    subdirX = 'Landcover' + c    
    fullX = os.path.join(gisworkDir, subdirX)
    if not os.path.isdir(fullX):
        os.makedirs(fullX)
 
    # Create the numbered subdirectories if they do not exist.
    for i in xrange(71):
        numSubdir = subdirX + ('_%02d' % (i + 1))
        numFullPath = os.path.join(fullX, numSubdir)
        if not os.path.isdir(numFullPath):
            os.mkdir(numFullPath)
     
# Similarly, create the DONUT subdirectories. 
# The above code is repeated and could be put to
# the function and used for both cases but this 
# is only the auxiliary script to be thrown away.    
subdirX = 'DONUT'    
fullX = os.path.join(gisworkDir, subdirX)
if not os.path.isdir(fullX):
    os.makedirs(fullX)
 
# Create the numbered subdirectories if they do not exist.
for i in xrange(71):
    numSubdir = subdirX + ('_%02d' % (i + 1))
    numFullPath = os.path.join(fullX, numSubdir)
    if not os.path.isdir(numFullPath):
        os.mkdir(numFullPath)

Open in new window

0
mgjustAuthor Commented:
>Are the resultX_nn directories or files located in tmpdir?
The results are not to be temporary. Each result is a new ESRI GRID / Raster  (containing those 8 files per 'unit').

>Are your original subdirectories named ...?
For this example they are named
C:\Giswork\LandcoverA\landcoverA_01

However, in reality the naming of these subdirectories was not so clean, so I will change them to this standard as they are wayward now.

Thank you kindly,
MJ
0
peprCommented:
Save the script below to, say, gen.py and execute it. It will display the constructed triples (landcover, donut, numSuffix). The landcover and donut are the values that you prepared manually to the files. No stripping needed...
# This is an auxiliary script to develop the landcover/donut
# pairs for the next processing.
# 
import os
import glob
 
gisworkDir = 'c:/Giswork'
 
 
def allLandcoverAndDonutPairs(gisworkDir):
    '''Generates all tuples (landcover, donut, numSuffix) from the gisworkDir.'''
 
    # Prepare the path to the donuts.
    donutDir = os.path.join(gisworkDir, 'DONUT')
    assert os.path.isdir(donutDir)
 
    # Iterate through sequence of top-level landcover subdirectories.
    mask = os.path.normpath(os.path.join(gisworkDir, 'Landcover*'))
    for LDir in glob.glob(mask):      # full paths to LandcoverA, LandcoverB, etc.
        subdir = os.path.basename(LDir)
 
        # Iterate through all numbered landcover subdirs inside.
        mask2 = os.path.join(LDir, 'Landcover*')
        for LnumDir in glob.glob(mask2):
            # Separate the counter part and construct the related donut
            # numbered directory.
            numSuffix = LnumDir[-2:]
            DnumDir = os.path.normpath(
                         os.path.join(donutDir, 'DONUT_' + numSuffix))
            assert(os.path.isdir(DnumDir))
 
            yield LnumDir, DnumDir, numSuffix
    
 
for landcover, donut, numSuffix in allLandcoverAndDonutPairs(gisworkDir):
    print repr(landcover), repr(donut), repr(numSuffix)

Open in new window

0
peprCommented:
Then your original script could use the above generator the way shown below. No need for creating the auxiliary files manually, no need for the auxiliary files at all. Possibly also the way how InExpression is built may be more readable. The generator could also return suffixes like 'A_01' to be used for building the result name. You should specify better where the results should be placed and how they should be named.
# Multi_Output_Map_Algebra_sample.py
# Description:
#   Runs an expression built with the Map Algebra language.
# Requirements: None
import sys
import arcgisscripting
 
 
## Put the above def allLandcoverAndDonutPairs(...) definition here
## and modify your body the following way...
 
 
# Create the Geoprocessor object
gp = arcgisscripting.create()
tmpdir = 'c:\\temp\\'
count = 1
try:
    for landcover, donut, numSuffix in allLandcoverAndDonutPairs(gisworkDir):
        # Set local variables
        Outfname = os.path.join(tmpdir, 'new_name', str(count))
        InExpression = "CON(ISNULL(%s), %s, %s * -1)" % (donut, landcover, landcover)
        print InExpression
        # Check out Spatial Analyst extension license
        gp.CheckOutExtension("Spatial")
 
        # Process: MapAlgebra
        gp.SingleOutputMapAlgebra_sa(InExpression, Outfile)
 
        count = count + 1
 
except:
    # If an error occurred while running a tool, then print the messages.
    print gp.GetMessages()

Open in new window

0
mgjustAuthor Commented:
Hello pepr,
Thanks for the quick reply. For the script 'gen.py' I get the following error on line 32:

Failed to run script - syntax error - invalid syntax
....
Also for "donutDir = os.path.join(gisworkDir, 'DONUT')" do I replace DONUT with the entire location?

e.g.
C:/Giswork/landcoverprep/Donut   (given that gisworkDir = 'c:/Giswork/landcoverprep'

or

 /Donut

...

You said:
## Put the above def allLandcoverAndDonutPairs(...) definition here
## and modify your body the following way...
 
Does this mean I copy the 'gen.py' script and paste into the the second script between "import arcgisscripting" and "gp = arcgisscripting.create()"

or does this mean that I need to change these values somehow?

Outfname = os.path.join(tmpdir, 'new_name', str(count))
        InExpression = "CON(ISNULL(%s), %s, %s * -1)" % (donut, landcover, landcover)
        print InExpression

As I am unsure of what "##Put the above def allLandcoverAndDonutPairs(...) definition here
## and modify your body the following way..." this means exactly and how to do it.

>You should specify better where the results should be placed and how they should be named.
The results could either be placed in the temp directory where I will take them and then copy and paste them where I want them which will be in C:/Giswork/landcoverprep the original directory. For naming as long as the A,B,C, etc plus 01,02,03, etc is present (e.g. A_01, A_02, B_01, etc) then this is all i need.

Thank you for your enduring patience, I appreciate it and your annotation is quite helpful,
I hope that you are able to further assist me because this script will be very cool,
MJ
0
peprCommented:
O.K. Below is the complete code of test.py that I tried. I guess I still confuse the directory structure for donut and results -- hopefully iterating slowly towards the solution. The generator now returns idSuffix instead of numSuffix for building unique result directory names. Please, read the code, try it and then ask for other question.

I also suggest to remove the try/except (at least temporarily) as it mask errors in syntax...
# Multi_Output_Map_Algebra_sample.py
# Description:
#   Runs an expression built with the Map Algebra language.
# Requirements: None
 
import arcgisscripting
import glob
import os
 
gisworkDir = 'c:/Giswork'
 
 
def allLandcoverDonutAndIdSuffix(gisworkDir):
    '''Generates all tuples (landcover, donut, idSuffix) from the gisworkDir.'''
 
    # Prepare the path to the donuts.
    donutDir = os.path.join(gisworkDir, 'landcoverprep', 'Donut')
    assert os.path.isdir(donutDir)
 
    # Iterate through sequence of top-level landcover subdirectories.
    mask = os.path.normpath(os.path.join(gisworkDir, 'Landcover?'))
    for LDir in glob.glob(mask):      # full paths to LandcoverA, LandcoverB, etc.
        subdir = os.path.basename(LDir)
 
        # Iterate through all numbered landcover subdirs inside.
        mask2 = os.path.join(LDir, 'Landcover*')
        for LnumDir in glob.glob(mask2):
            # Separate the counter part, the idSuffix and construct 
            # the related donut numbered directory name.
            numSuffix = LnumDir[-2:]   # last 2 chars
            idSuffix = LnumDir[-4:]    # last 4 chars
            DnumDir = os.path.normpath(
                         os.path.join(donutDir, 'DONUT_' + numSuffix))
            assert os.path.isdir(DnumDir)  
 
            # Generate another output tuple.
            yield LnumDir, DnumDir, idSuffix
 
 
# Create the Geoprocessor object
gp = arcgisscripting.create()
expTemplate = 'CON(ISNULL(%s), %s, %s * -1)'
##try:
if True:
    for landcover, donut, idSuffix in allLandcoverDonutAndIdSuffix(gisworkDir):
 
        # Construct the expression and the output directory name.
        outDir = os.path.join(gisworkDir, 'landcoverprep', 'Results', idSuffix)
        InExpression = expTemplate % (donut, landcover, landcover)
        print InExpression
 
        # Check out Spatial Analyst extension license
        gp.CheckOutExtension("Spatial")
 
        # Process: MapAlgebra
        gp.SingleOutputMapAlgebra_sa(InExpression, outDir)
 
##except:
if False:
    # If an error occurred while running a tool, then print the messages.
    print gp.GetMessages()

Open in new window

0
peprCommented:
For your information, I have simulated the arcgisscripting.py using the following code.
import os
 
class Geoprocessor():
    def CheckOutExtension(self, s):  pass
    def SingleOutputMapAlgebra_sa(self, inExpression, outDir):  
        if not os.path.isdir(outDir):
            os.makedirs(outDir)
        fname = os.path.join(outDir, 'expression.txt')
        assert not os.path.isfile(fname)
        f = open(fname, 'w')
        f.write(inExpression)
        f.close()        
 
    def GetMessages(self):
        return 'Geoprocessor error messages.'
 
def create():
    return Geoprocessor()

Open in new window

0
mgjustAuthor Commented:
Pepr,
I thank you very much for the continued support.  When I run test.py from above nothing happens. What should I expect to see or encounter if I run just like you have it? Sorry for my inexperience, I am just unsure as to what I should be looking for or doing when I try to execute the script. Again, I am using Python 2.4.1

I think I get the same syntax error message with this line :

yield LnumDir, DnumDir, idSuffix

I am unsure of what this arcgisscripting.py script is supposed to do? I ran it and nothing happened.

Thank you very much,
MJ

P.S. As I am very new to python I am unsure how to check a script out and started a new question here: http://www.experts-exchange.com/Programming/Languages/Scripting/Python/Q_23088877.html
0
peprCommented:
You should not use the arcgisscripting.py in your case. In my case, I do not have the ESRI ArcMap installed. Therefore, I do not have the arcgisscripting module avaliable. As your script does the

import arcgisscripting

I have created the fake module with the same name. It simply builds the Result subdirectory with detailed subdirectories inside and with one text file that contains the command. Rename the file if you want to use the original ESRI ArcMap module of the same name.

For not seeing any activity of the script, did you remove try/except? ...for example temporarily by commenting the keywords and adding if True/False to avoid neccessity to unindent the block of code...

##try:
if True:
    ...
##except:
if False:


I do use Python 2.5.1, but the yield statement should be available in 2.4.1. Could you copy the error message (after commenting out the try/except)?
0
peprCommented:
Possibly, try

yield (LnumDir, DnumDir, idSuffix)

to create the returned tuple explicitly. I am not sure if 2.4.1 did not want that.

To test the yield command separately in your Python version, try the script below. (I have named it b.py.) You should observe something like that in your cmd window:

====================================
C:\tmp\>b.py
1
(1, 2, 3)
====================================


def gen():
    yield 1
    yield 1, 2, 3
 
for result in gen():
    print repr(result)

Open in new window

0
mgjustAuthor Commented:
pepr,
I have tried the  b.py script and it returns the same output you posted. I also update the main script (below) with these parantheses.  However, when I run the script from the command window it takes some time but there are no errors and there is no output.  I am correct that the output will be named resultA_01 and placed in the landcoverprep directory. I had commented out the try and except.

Currently my directories are setup like this:
C:\Giswork\landcoverprep\Donut\donut_01
C:\Giswork\landcoverprep\LandcoverA\landcoverA_01

I then uncommented the try and except and got the following errors:

C:\Documents and Settings\user>C:\Python24\python.exe C:\peprtest.py
  File "C:\peprtest.py", line 44
    if True:
     ^
IndentationError: expected an indented block

C:\Documents and Settings\user>C:\Python24\python.exe C:\peprtest.py
  File "C:\peprtest.py", line 61
    print gp.GetMessages()
                         ^
IndentationError: unindent does not match any outer indentation level

So then I kept indenting stuff down the line, get this same error moving down and then finally at the end of the script I got this response:

C:\Documents and Settings\user>C:\Python24\python.exe C:\peprtest.py
  File "C:\peprtest.py", line 61
    print gp.GetMessages()
                         ^
IndentationError: unindent does not match any outer indentation level


One more hopefully easy question
What's the reason to use a "?" for the first mask and a "*" for the second?

mask = os.path.normpath(os.path.join(gisworkDir, 'Landcover?'))
mask2 = os.path.join(LDir, 'Landcover*')


Thank you again,
MJ
# Multi_Output_Map_Algebra_sample.py
# Description:
#   Runs an expression built with the Map Algebra language.
# Requirements: None
 
import arcgisscripting
import glob
import os
 
gisworkDir = 'c:/Giswork'
 
 
def allLandcoverDonutAndIdSuffix(gisworkDir):
    '''Generates all tuples (landcover, donut, idSuffix) from the gisworkDir.'''
 
    # Prepare the path to the donuts.
    donutDir = os.path.join(gisworkDir, 'landcoverprep', 'Donut')
    assert os.path.isdir(donutDir)
 
    # Iterate through sequence of top-level landcover subdirectories.
    mask = os.path.normpath(os.path.join(gisworkDir, 'Landcover?'))
    for LDir in glob.glob(mask):      # full paths to LandcoverA, LandcoverB, etc.
        subdir = os.path.basename(LDir)
 
        # Iterate through all numbered landcover subdirs inside.
        mask2 = os.path.join(LDir, 'Landcover*')
        for LnumDir in glob.glob(mask2):
            # Separate the counter part, the idSuffix and construct 
            # the related donut numbered directory name.
            numSuffix = LnumDir[-2:]   # last 2 chars
            idSuffix = LnumDir[-4:]    # last 4 chars
            DnumDir = os.path.normpath(
                         os.path.join(donutDir, 'DONUT_' + numSuffix))
            assert os.path.isdir(DnumDir)  
 
            # Generate another output tuple.
            yield (LnumDir, DnumDir, idSuffix)
 
 
# Create the Geoprocessor object
gp = arcgisscripting.create()
expTemplate = 'CON(ISNULL(%s), %s, %s * -1)'
try:
if True:
    for landcover, donut, idSuffix in allLandcoverDonutAndIdSuffix(gisworkDir):
 
        # Construct the expression and the output directory name.
        outDir = os.path.join(gisworkDir, 'landcoverprep', 'Results', idSuffix)
        InExpression = expTemplate % (donut, landcover, landcover)
        print InExpression
 
        # Check out Spatial Analyst extension license
        gp.CheckOutExtension("Spatial")
 
        # Process: MapAlgebra
        gp.SingleOutputMapAlgebra_sa(InExpression, outDir)
 
except:
if False:
    # If an error occurred while running a tool, then print the messages.
    print gp.GetMessages()

Open in new window

0
mgjustAuthor Commented:
Hello,
I had to change a thing or two but I think it works now.

to:>> mask = os.path.normpath(os.path.join(gisworkDir, 'Landcoverprep\landcover*'))
to: >>> mask2 = os.path.join(LDir, '*')

 Lastly, I needed to have Result directory made, which I didn't know.

Thank you so much,
MJ
I will let you know tomorrow if this worked for sure.
# Multi_Output_Map_Algebra_sample.py
# Description:
#   Runs an expression built with the Map Algebra language.
# Requirements: None
 
import arcgisscripting
import glob
import os
 
gisworkDir = 'c:\Giswork'
f=open("C:\\tester.txt","w")  
 
def allLandcoverDonutAndIdSuffix(gisworkDir):
    '''Generates all tuples (landcover, donut, idSuffix) from the gisworkDir.'''
    #f.write("1\n")
    # Prepare the path to the donuts.
    donutDir = os.path.join(gisworkDir, 'landcoverprep', 'Donut')
    assert os.path.isdir(donutDir)
    #f.write("2\n")
    # Iterate through sequence of top-level landcover subdirectories.
    mask = os.path.normpath(os.path.join(gisworkDir, 'Landcoverprep\landcover*'))
    #f.write(mask)		    
    for LDir in glob.glob(mask):      # full paths to LandcoverA, LandcoverB, etc.
        subdir = os.path.basename(LDir)
	#f.write("3\n")
        # Iterate through all numbered landcover subdirs inside.
        mask2 = os.path.join(LDir, '*')
        for LnumDir in glob.glob(mask2):
            # Separate the counter part, the idSuffix and construct 
            # the related donut numbered directory name.
            numSuffix = LnumDir[-2:]   # last 2 chars
            idSuffix = LnumDir[-4:]    # last 4 chars
            DnumDir = os.path.normpath(
                         os.path.join(donutDir, 'DONUT_' + numSuffix))
            assert os.path.isdir(DnumDir)  
	    #f.write("4")
            # Generate another output tuple.
            yield (LnumDir, DnumDir, idSuffix)
 
 
# Create the Geoprocessor object
gp = arcgisscripting.create()
expTemplate = 'CON(ISNULL(%s), %s, %s * -1)'
f.write("\n===== Output below: ====\n")
try:
	if True:
		for landcover, donut, idSuffix in allLandcoverDonutAndIdSuffix(gisworkDir):
 
        # Construct the expression and the output directory name.
        		outDir = os.path.join(gisworkDir, 'landcoverprep', 'Results', idSuffix)
			f.write(outDir)
			InExpression = expTemplate % (donut, landcover, landcover)
			f.write(InExpression)
 
        # Check out Spatial Analyst extension license
        	gp.CheckOutExtension("Spatial")
 
        # Process: MapAlgebra
        	gp.SingleOutputMapAlgebra_sa(InExpression, outDir)
 
except:
	if False:
    # If an error occurred while running a tool, then print the messages.
    		f.write( gp.GetMessages())

Open in new window

0
peprCommented:
Your are welcome. I can see some problems in the script.

Firstly, you should not mix indentation by tabs and by spaces. The documentation says that you should use one or the other. I prefer indentation by spaces only -- 4 spaces for one indentation level.

You combined try+if True and except+if False. My original intention was to remove the try and except (by commenting it out); however I was lazy to decrease the indentation level of the block of code. Because of that I had used the "empty command that requires indentation of the next block of code", here the "if True:". Similarly, the except: was commented out, so if False: was here only to let the next block code manually untouched but never executed.

Now for the
    os.path.join(gisworkDir, 'Landcoverprep\landcover*')

There is a bug -- the backslash must be doubled in this case (or the string literal must be prefixed by r as r'raw string'). The same bug is in
    gisworkDir = 'c:\Giswork'
The alternative also is to use a normal slash -- it works also in MS Windows. Use this for paths. In your case you were lucky that \G or \l are not known as escape sequences. You would observe problems if the directory were named say newGiswork because inside 'c:\newGiswork' the \n would be replaced by the newline sequence.

Anyway, the os.path.join() is here for avoiding problems with OS dependent sudirectory separators. The proper way to write the same command is
    os.path.join(gisworkDir, 'Landcoverprep', 'landcover*')

One more simplification. In the original post, the Donut and LandcoverA... directories were put directly in c:\Giswork. If there is one more subdirectory for Donut and LandcoverA... In other words if everything is located in c:\Giswork\Landcoverprep then you should probably name that gisworkDir (or give the variable better name):

gisworkDir = 'c:/Giswork/Landcoverprep'

and simplify the os.path.join commands in the code.

If you prefer enclosing the strings in double quotes, then you should do it consistently (possibly comming from the C-family languages. But this is not an issue.

Also, you should close your log file at the end of script (opened as f).

Possibly better is to create a log function shown below. Later, you can replace its body by something more suitable. It also opens and closes the file all the time which may be time consuming but it is better for debugging as it will physically write the string even when the script crashes.

def log(s):
    f = open("C:/tester.txt","w")
    f.write(s)
    f.close()

Open in new window

0
mgjustAuthor Commented:
pepr,
Sorry to have neglected this question ( I was out town).

When I was testing I was only using one Landcover directory and one raster
(e.g. C:\Giswork\landcoverprep\landcoverA\landcoverA_01). And maybe got a little too excited.

Now, today when I try to use the script it only saves the last file it processes. For example, I tested using 2 directories and  the files  landcoverA_01 to 03 and  landcoverB01_03 and the only thing in the results folder is B_03.
Please advise.


I hope that you are still willing/able to help,
thank you very much,
MJ
0
peprCommented:
You have f.write(outDir) and f.write(InExpression) calls in your script. What can you read inside the log file. Anyway, I would start with removing the try/except. ... OK. I has taken your last script and modified it -- see below, try it, have a look into the log file, and tell me what is suspicious.

Run the script from the maually started cmd window, so the possible exception reports will be visible.
# Multi_Output_Map_Algebra_sample.py
# Description:
#   Runs an expression built with the Map Algebra language.
# Requirements: None
 
import arcgisscripting
import glob
import os
 
gisworkDir = 'c:\Giswork'
 
def log(s):
    f = open("C:/tester.txt","w")
    f.write(s)
    f.close()
 
 
def allLandcoverDonutAndIdSuffix(gisworkDir):
    '''Generates all tuples (landcover, donut, idSuffix) from the gisworkDir.'''
    ##log("1\n")
    # Prepare the path to the donuts.
    donutDir = os.path.join(gisworkDir, 'landcoverprep', 'Donut')
    assert os.path.isdir(donutDir)
    ##log("2\n")
    # Iterate through sequence of top-level landcover subdirectories.
    mask = os.path.normpath(os.path.join(gisworkDir, 'Landcoverprep\landcover*'))
    ##log(mask)             
    for LDir in glob.glob(mask):      # full paths to LandcoverA, LandcoverB, etc.
        subdir = os.path.basename(LDir)
        ##log("3\n")
        # Iterate through all numbered landcover subdirs inside.
        mask2 = os.path.join(LDir, '*')
        for LnumDir in glob.glob(mask2):
            # Separate the counter part, the idSuffix and construct 
            # the related donut numbered directory name.
            numSuffix = LnumDir[-2:]   # last 2 chars
            idSuffix = LnumDir[-4:]    # last 4 chars
            DnumDir = os.path.normpath(
                         os.path.join(donutDir, 'DONUT_' + numSuffix))
            assert os.path.isdir(DnumDir)  
            ##log("4")
            # Generate another output tuple.
            yield (LnumDir, DnumDir, idSuffix)
 
 
# Create the Geoprocessor object
gp = arcgisscripting.create()
expTemplate = 'CON(ISNULL(%s), %s, %s * -1)'
log("\n===== Output below: ====\n")
 
for landcover, donut, idSuffix in allLandcoverDonutAndIdSuffix(gisworkDir):
    # Construct the expression and the output directory name.
    outDir = os.path.join(gisworkDir, 'landcoverprep', 'Results', idSuffix)
    log("\n" + outDir)
    InExpression = expTemplate % (donut, landcover, landcover)
    log("  " + InExpression)
 
    # Check out Spatial Analyst extension license
    gp.CheckOutExtension("Spatial")
 
    # Process: MapAlgebra
    gp.SingleOutputMapAlgebra_sa(InExpression, outDir)
 
    log("Msg: " + gp.GetMessages())

Open in new window

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
mgjustAuthor Commented:
pepr,
Using the exact script you just provided with the try/except, etc removed the operation completed successfully and completely.  I don't know why this worked, but it did.

Thank you,
MJ
0
mgjustAuthor Commented:
Thank you for your enduring patience.
0
mgjustAuthor Commented:
Hello,
I am looking for some additional functionality and updates:

http://www.experts-exchange.com/Programming/Languages/Scripting/Python/Q_23578050.html

Please advise.

Thanks,
MJ
0
mgjustAuthor Commented:
Hello,
I am trying to use this script but it is not working:
New Question here:
http://www.experts-exchange.com/Programming/Languages/Scripting/Python/Q_23580838.html

Thanks,
MJ
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Python

From novice to tech pro — start learning today.