We help IT Professionals succeed at work.

DCL code to Unit Bash script. How to ZIP files and count ZIP files in Unix Bash script?

Medium Priority
188 Views
Last Modified: 2019-08-09
Hi everyone,

I am trying to zip in zfiles<state>.zip file multiple (1000s) files (ASCII files) and get a count of each zip file in most efficient way (without dirrectory names and compress better) in Unix Bash script (zipfiles.sh) just like it was in the DCL language below.  If anyone knows how to do the same zipping process in the Unix bash script please let me know. Any suggestions or examples would be appriciated!

Note:  statelist.txt file has all states to loop through


Here is the DCL code portion that I am trying to convert to Unix bash scripting:
define/nolog DATA dir1/data
define/nolog proglog DATA:ZIP.LOG

laststate := ' '

 open/read statelist STATELIST.TXT
read/loop:
 read/end = endread statelist laststate

if f$search("ASCIIFILE%''laststate'%%%%%.TXT").EQS " " THEN -
GOTO READ_LOOP

!-------------------------------------------------------------------------------------------------------------------------------
! Create ZFILES<state>.zip  by zipping all ASCIIFILE files and send ZIP.LOG to the mail list
!-------------------------------------------------------------------------------------------------------------------------------
ZIP  -9V ZFILES'laststate'.ZIP - 
             ASCIIFILE%'laststate'%%%%%.TXT

if .not. $status
 then
  open/append pglog proglog
    write  pglog "--- Error Zipping ASCIIFILE*.TXT files ---"
   close pglog 
       SAY " --- Error Zipping ASCIIFILE*.TXT files ---"
 mail/subject="--- Zipping all files ==> Failure ---" -
  proglog - 
   "@DATA:MAIL.TXT"

else 
 open/append prlog proglog 
 write pglog " * " 
 write prlog "  * DATA:ZFILES''laststate'.ZIP" 
 write prlog " * "
close prlog
SAY " --- Zipping ASCIIFILE*.TXT files ==> Successful ---"

tcounts = tcounts +1
endif

GOTO read_loop

endread:
close statelist

if .not.  $status
  then
SAY " "--- Zipping all files Processing ==> Failure ---"

else
SAY " "
SAY " Zipping all files....  "

open/append prlog proglog
 write prlog " * "
 write prlog " *  Total number of ZIP files created:  ''tcounts' "
 write prlog " * "
close prlog

!------------------------------------------------
!  Sending ZIP.LOG to the mail list
!------------------------------------------------
  SAY " --- Total number of ZIP files created ==> ''tcounts' "
 mail/subject="--- Zipping all files ==> Successful ---" -
  proglog - 
   "@DATA:MAIL.TXT"

delete DATA:statelist.txt

deassign proglog
exit

Open in new window

Comment
Watch Question

Pierre FrançoisSenior consultant
CERTIFIED EXPERT

Commented:
If never heard about DCL before, and this could be the case of many of us.

If you want some help, I suggest you to comment your DCL code, describing plentiful, step by step if necessary, what it does. So we will be able to guess how to translate it into bash code.

Author

Commented:
Hi Pierre,

Dialog Control Language (DCL) is a high-level description language and interpreter within AutoCAD for creating simple graphical dialogs. DCL is object-oriented; it allows re-use through inheritance and composition.  

I pretty much would like to zip bunch (1000s) of ASCII files into 50 state zip files by state and then count all of created zip files. DCL code also used to output error/success messages and then sent those messages to a log file as well as sent to a mail list to bunch of emails as notifications of an error or a success.
Software Engineer
CERTIFIED EXPERT
Distinguished Expert 2019
Commented:
This DCL is OpenVMS , Digital Command Language. (OpenVMS was created by the Digital Equipment Corp.)  in the 1978 for VAX computers  (Currently being run on Itanium & AXP) and in the process of being ported to x86-64.   (VSI:  http://www.vmssoftware.com/updates_port.html )

Defines are namespace elements that create "filenames" a kernel can lookup when opening files.  Those are not the same as environment variables, they more or less work the same as symlinks that are always in the current directory.
Symbols (like $status) are equivalent to unix environment variables.

The above script checks a statefiles, verifies if some file(s) based on a record from that statefile exists.
Zips those files and then adds the zip results to a log file.  The log is mailed.

The original DCL code has some problems... if then else without endif...
delete without version number of files, ans some lines with too many ".
Also a background process might upgrade the statefile before it gets deleted but after processing ends. so there is a slight timing gap.
mismatch between file through logical DATA and current directory.
Also case of filename is relevant on unix, while ODS-2 on OpenVMS is case-insensitive.

Version with DCL as comment.
#!/bin/sh

#define/nolog DATA dir1/data
#define/nolog proglog DATA:ZIP.LOG
#laststate := ' '
DATA="dir1/data"                # use symbols in stead of logicals...
proglog="$DATA/ZIP.LOG"
laststate=" "
zipfailed=0

# open/read statelist STATELIST.TXT
#read/loop:
# read/end = endread statelist laststate
cat ${DATA}/statelist.txt | while  read laststate
do
#if f$search("ASCIIFILE%''laststate'%%%%%.TXT").EQS " " THEN -
#GOTO READ_LOOP
    filename="ASCIIFILE?${laststate}?????.TXT" 
    if [ -e "${filename}" ]
    then
#!------------------------------------------------------------------------------------------------------------------------------
#! Create ZFILES<state>.zip  by zipping all ASCIIFILE files and send ZIP.LOG to the mail list
#!-------------------------------------------------------------------------------------------------------------------------------
#ZIP  -9V ZFILES'laststate'.ZIP - 
#             ASCIIFILE%'laststate'%%%%%.TXT
        if ! zip -9 ZFILES${laststate}.ZIP ${filename}
        then
            zipfailed=1
#if .not. $status
# then
#  open/append pglog proglog
#    write  pglog "--- Error Zipping ASCIIFILE*.TXT files ---"
#   close pglog 
#       SAY " --- Error Zipping ASCIIFILE*.TXT files ---"
# mail/subject="--- Zipping all files ==> Failure ---" -
#  proglog - 
#   "@DATA:MAIL.TXT"
            echo "--- Error Zipping ASCIIFILE*.TXT files ---"
            echo "--- Error Zipping ASCIIFILE*.TXT files ---"  >>${proglog}
            cat ${proglog} | mail -s "--- Zipping all files ==> Failure ---" $( cat ${DATA}/MAIL.TXT )
        else
#else 
# open/append prlog proglog 
# write pglog " * " 
# write prlog "  * DATA:ZFILES''laststate'.ZIP" 
# write prlog " * "
#close prlog
#SAY " --- Zipping ASCIIFILE*.TXT files ==> Successful ---"
            echo " --- Zipping ASCIIFILE*.TXT files ==> Successful ---"
            echo -e " *\n * ${DATA}/ZFILES${laststate}.ZIP\n *" >>${proglog}
#tcounts = tcounts +1
            tcounts=$(( ${tcounts} + 1 ))
#endif
        fi

#GOTO read_loop
#endread:
#close statelist
    done

    if zipfailed
    then
#if .not.  $status
#  then
#SAY " "--- Zipping all files Processing ==> Failure ---"
        echo "--- Zipping all files Processing ==> Failure ---"
#else
    else
#SAY " "
#SAY " Zipping all files....  "
#open/append prlog proglog
# write prlog " * "
# write prlog " *  Total number of ZIP files created:  ''tcounts' "
# write prlog " * "
#close prlog
        echo " Zipping all files....  "
        echo -e " *\n * Total number of ZIP files created:  ${tcounts}\n *" >>${proglog}

#!------------------------------------------------
#!  Sending ZIP.LOG to the mail list
#!------------------------------------------------
#  SAY " --- Total number of ZIP files created ==> ''tcounts' "
  echo " --- Total number of ZIP files created ==> ${tcounts} "
#  mail/subject="--- Zipping all files ==> Successful ---" -
#  proglog - 
#   "@DATA:MAIL.TXT"
         cat ${proglog} | mail -s "--- Zipping all files ==> Successful ---" $( cat ${DATA}/MAIL.TXT )

   fi
#delete DATA:statelist.txt
    rm ${DATA}/statelist.txt

#deassign proglog
#exit
    exit

Open in new window


Shell (bash) only version
#!/bin/sh

DATA="dir1/data"                # use symbols in stead of logicals...
proglog="$DATA/ZIP.LOG"
laststate=" "
zipfailed=0

cat ${DATA}/statelist.txt | while  read laststate
do
    filename="ASCIIFILE?${laststate}?????.TXT" 
    if [ -e "${filename}" ]
    then
#------------------------------------------------------------------------------------------------------------------------------
# Create ZFILES<state>.zip  by zipping all ASCIIFILE files and send ZIP.LOG to the mail list
#-------------------------------------------------------------------------------------------------------------------------------
        if ! zip -9 ZFILES${laststate}.ZIP ${filename}
        then
            zipfailed=1
            echo "--- Error Zipping ASCIIFILE*.TXT files ---"
            echo "--- Error Zipping ASCIIFILE*.TXT files ---"  >>${proglog}
            cat ${proglog} | mail -s "--- Zipping all files ==> Failure ---" $( cat ${DATA}/MAIL.TXT )
        else
            echo " --- Zipping ASCIIFILE*.TXT files ==> Successful ---"
            echo -e " *\n * ${DATA}/ZFILES${laststate}.ZIP\n *" >>${proglog}
            tcounts=$(( ${tcounts} + 1 ))
        fi

    done

    if zipfailed
    then
        echo "--- Zipping all files Processing ==> Failure ---"
    else
        echo " Zipping all files....  "
        echo -e " *\n * Total number of ZIP files created:  ${tcounts}\n *" >>${proglog}

        echo " --- Total number of ZIP files created ==> ${tcounts} "
        cat ${proglog} | mail -s "--- Zipping all files ==> Successful ---" $( cat ${DATA}/MAIL.TXT )

    fi
    rm ${DATA}/statelist.txt
    exit

Open in new window

Pierre FrançoisSenior consultant
CERTIFIED EXPERT

Commented:
Haha! Great.

The last time I touched a VAX computer was in 1991. However, we didn't run VMS (the operating system of DEC for VAX) on that machine, but some UNIX OS, BSD if I remember correctly. So, we never used DCL and this was 28 years ago...

Hoping the answer of the other guys will be more useful for you than mine.
nociSoftware Engineer
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
I updated the earlier item with a translation. (There are use cases where  OpenVMS is still unsurpassed... and i still am an OpenVMS site maintainer, just curious how the VSI will go with the new incarnation).
The mitigations, separate page tables for kernel / usermode,  that were needed in linux for Spectre & Meltdown were already developped to allow the four modes needed in OpenVMS for process separation [ about 3 -4 years ago, separated page tables for kernel, executive, supervisor & usermode  ].

@Pierre:
And in that timeframe that should indeed have been BSD.

Author

Commented:
Hi Noci, thank you very much for your explanations and your both examples! Your line-to-line comparisons between the DCL and Bash really makes me understand where and how Bash scripting is different from DCL language.

A few questions:
1. In regards to second example "Shell (bash) only version", on the line 18, how the script knows that "zipfailed=1" ?  
The reason why I am asking this is becasue "zipfailed" previously on the line 6 it was initiated to "0" but how "zipfailed" was changed from "0" to "1"?  Does it change somehow at the time of an error while zipping files?

2. On the line 10, filename="ASCIIFILE?${laststate}?????.TXT"  
Instead of ????? in each file name (ASCIIFILE<var1>.TXT) there would be a variable, correct? For example, "var1" variable that in this case has always a unique value such as ASCIIFILE61616.TXT in one file and in another file ASCIIFILE61617.TXT

I think I did not mention this previoulsy but this "var1" variable with its unique value is also part of each record inside of each correspondedent ASCIIFILE<var1>.TXT file. Just wanted to make sure I am not missing anything.  

Thank you!
nociSoftware Engineer
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
1) DCL has the concept of $STATUS as a return status variable and you need to test explicitely.
Unix (or better: Shells like bash, csh, sh bsh, ksh, ...)  has the $? variable.
All Shells also use the syntax:

if command
then
  command
else
  command
fi

The actual wording can be different ( { } in stead of then / fi...)  the command after if is executed  and if it returns 0   == success then the if is true.
if it returns != 0 (1.. 255 ) then it is false.   (In VMS odd $STATUS values is Success.. Generic success SS$_NORMAL has value 1.)

so command:
DCL-CMD
IF $STATUS ...

translates to:
if unix-comand ,,,

likewise:   .not.  translates to !


2) ? is wildcard replacement for % from VMS.
in the original also the state value was explicitly mentions, in a string quoted with "..  variables can be expanded on shells with ${var1}  where VMS uses ''var1' in a string.

Author

Commented:
Noci, thank you for explanning the differences!!