Shell Scripting

The term 'shell' refers to a general class of text-based command interpreters most often associated with the UNIX and Linux operating systems. Popular shells include Bourne, Debian Almquist (dash), Korn (ksh), Bourne Again (bash) and the C shell family (csh). Some view the DOS 'cmd' prompt as a minimal shell of sorts. It is also possible to install Cygwin on Windows and emulate a full Unix environment with complete shell capabilities. Terminal emulators, such as xterm, GNOME Terminal and OS X Terminal, can be used to access shell.

Share tech news, updates, or what's on your mind.

Sign up to Post

Utilizing an array to gracefully append to a list of EmailAddresses
Important Lessons on Recovering from Petya
LVL 10
Important Lessons on Recovering from Petya

In their most recent webinar, Skyport Systems explores ways to isolate and protect critical databases to keep the core of your company safe from harm.

Active Directory replication delay is the cause to many problems.  Here is a super easy script to force Active Directory replication to all sites with by using an elevated PowerShell command prompt, and a tool to verify your changes.
Recently, an awarded photographer, Selina De Maeyer, completed a photo shoot of a beautiful event in Antwerp, the Corpus Christi procession around the Saint James church. She took approximately 250 pictures with her two Nikon cameras, but when we went to publish the pictures in chronological order, we discovered that the time was not set consistently for her cameras.

With software, we were able to extract the .jpg files with the date and time once a picture has been shot. This allowed us to see that there were clearly two series of pictures, separated in time. Reordering so many pictures manually was not an option. So we decided to write a little script to correct the date and time on the set that was wrong.

The first item to address was the origination of the camera for each picture.  There is an open source suite called ImageMagick that provides a program called identify. Identify is able to list information for each picture. All the programs we invoke below are run from the command line interface - the shell. If you never heard about the shell, I am sorry to tell you: do not read further, this article will bore you. but if you are familiar, you will know that grep will filter the lines containing a certain pattern from the results of the previous command.

Information returned by identify
LVL 10

Author Comment

Comment Utility
Dear Brad,

Thanks for fixing the typos.

You said:
A volunteer Page Editor may engage you on the content to improve your article further.

I would be happy to know what I have to do to get this done.

Thanks in advance.
How to remove superseded packages in windows w60 or w61 installation media (.wim) or online system to prevent unnecessary space.

w60 means Windows Vista or Windows Server 2008.
w61 means Windows 7 or Windows Server 2008 R2.

There are various guides found on the internet on how-to integrate updates into Windows installation media using dism.
This article is a post-integrate how-to to this numerous articles.
This article is addressed to users, who are comfortable using dism to mount, update, manage and unmount Windows Image files for deployment of Windows Systems.
So mounting and dismounting images is intentionally left out.

Administrators often include windows update packages into their deployment images to save time when deploying.
In Windows 8 dism.exe has built-in command-line extensions to remove superseded updates from Windows 8 image files.
Earlier versions of Windows do not have this feature.
So when you integrate packages into a windows installation media (w60 or w61), superseded or outdated packages reside in the deployment image and use unneccessary space. This guide shows how to list and remove superseded packages from the command-line or by batch.

Short description for what you have to do before:
- Download Windows Updates for Windows (w60 or w61).
  Best practice is using wsus offline update.
- Mount Windows installation media.
- Apply packages to the mounted image.

The listings below offers two options.
The first option …

Still having to process all these year-end "csv" files received from all these sources (including Government entities), sometimes we have the need to examine the contents due to data error, etc...

As a "Unix" shop, our only readily available tool was the "vi" editor, which is of little help when the records are too long or the size of the file is too big or the number of records is too numerous.

Unfortunately, we are not allowed to install any of 3rd party (free or not) utilities out there.

What could we do?

Solution: csvb - a Quick and Dirty CSV File Browser

The program is written in ksh shell and awk and will take the following parameters :

Option	Description			Default
 -d	Field delimiter			Comma ','
 -h	The file has header row 		1 = Yes
 -p	Starting record to display      1 = Default
 -n	Number of records to display    1 = Default, 0 = All
 -q	Quoted fieds			" = Default
 -c	Number of columns in display    2 = Default, 3 = Max
 -?	Display the help message

Open in new window

Program code is attached.

Sample output
==> csvb -h1 -n 3 -p 100 SAM_Exclusions_Public_Extract_14014t.csv
-e #------------------------------------------------------
# /usr/local/bin/csvb Arguments:
#  -d ,         <= delimiter (',' =default)
#  -q "         <= quote's ("=default)
#  -c 2         <= #output cols to display (2=default,max=3)
#  -p 100               <= starting record (1=default)
#  -n 3         <= #recs to display of 121872 

Open in new window

Over the years I've spent many an hour playing on hardened, DMZ'd servers, with only a sub-set of the usual GNU toy's to keep me company; frequently I've needed to save and send log or data extracts from these server back to my PC, or to others, and more often than not, to do this on a regular basis.

Simple enough on an open box, just cron a script to extract and format the necessary data and either send it as a uuencoded / base 64 mmencoded attachment in an email, or simply copy / scp's the file to somewhere you can get at it e.g. a NFS or SMB mount. BUT if the server is in a DMZ you'll likely find access to any shared internal file systems and / or permission to store a private key to scp / ssh into your network not available to you, leaving email as the only practical method to send data to yourself or others being.

If the box is performing a non trivial function in the DMZ it's more than likely been hardened to some extent, and all non essential toys removed, so you'll probably find your lacking the likes of uuencode, mmencode, and perl, leaving you none of the standard mechanism to encode and to attach files to email.

You could take the quick and dirty option of just piping the data / log into an email body an hoping Outlook will treat your data kindly, and not assume it's a malicious script, or part of the email header, or in need of some character-set conversion. This approach kept me happy for a number of years, but while working with Rockroads several  year…
LVL 15

Administrative Comment

by:Eric AKA Netminder
Comment Utility

Congratulations! Your article has been published.

Page Editor
LVL 26

Author Comment

Comment Utility
May also be of interest.
Mouse Trap
The following is a collection of cases for strange behaviour when using advanced techniques in DOS batch files. You should have some basic experience in batch "programming", as I'm assuming some knowledge and not further explain the basics. For some basics I will create a tutorial to be published here very soon (reference will be posted here).

It's an Interpreter
If using complex mechanisms like subshelling (using commands enclosed in round brackets), keep in mind batch files are not really like using a programming language - batch language isn't that accurate, there are several flaws in the interpreter we have to take care of.

As  cmd.exe  being an interpreter, it reads line after line, and does some pattern replacing in advance to executing the line.
A "line" can be
* a single physical line
* several physical lines connected by a caret (^) at the end of each line
* several physical lines enclosed in round brackets

A single line is hence:
@echo off

Open in new window

if 1 == 1 (echo yes) else (echo no)

Open in new window

echo Writing a long ^
text here

Open in new window

if 1 == 1 (
  echo yes
) else (
  echo no

Open in new window

Why should one know? As said already, it's an interpreter we use, and it's applying some string replacements.
If you are using environment variables, which is one of the things you will use every time, this gets most important.
E.g. the following code will not work as expected:
set example=1
if %example% == 1 (
  set example=2
  echo %example%
REM result:   1
REM expected: 2

Open in new window

In the first line, variable  example  is set to 1. In the next "line", containing  IF  up to the closing bracket, each occurance of  %example%
LVL 71

Author Comment

Comment Utility
Thanks, footech. I have introduced a new line 2 now, so the reference is correct ;-).
LVL 11

Expert Comment

Comment Utility
very nice, going in my DOS batch links library!
This Windows batch file is useful for organizing image files from a digital camera or other source, but can have many other uses.  It simply renames the file(s) to match their create date.  For example, if you took a picture today at 1:40pm and the image file is called IMG_0777.jpg, this utility will rename the file to 20081121_134000.jpg.  Works with multiple files, and will recurse sub-directories with the /s option.

Cut-n-paste the following into notepad and save it as a .BAT file:
@echo off
 setlocal enabledelayedexpansion
 set filespec=%*
 if "%filespec%"=="" goto :help
 for /f "delims=" %%F in ('dir/a-d/b/od %filespec%') do (
   set fn=%%~tF
   set/a hour=0x!fn:~11,2!
   if !hour! GTR 10 set/a hour=(!hour!-6^) %% 12
   if "!fn:~17,2!"=="PM" set/a hour+=12
   if !hour! LSS 10 set hour=0!hour!
   set root=%%~dpF
   set ext=%%~xF
   set nxt=00
   set fn=!fn:~6,4!!fn:~0,2!!fn:~3,2!_!hour!!fn:~14,2!
   if exist "!root!!fn!!nxt!!ext!" for /L %%I in (59,-1,1) do (
     set dup=0%%I
     if not exist "!root!!fn!!dup:~-2!!ext!" set nxt=!dup:~-2!
   @echo ren "%%F" "!fn!!nxt!!ext!"
         ren "%%F" "!fn!!nxt!!ext!"
 dir !root!*!ext! | find/v "Volume"
 goto :end
@echo  Renames files to the date and time the file was created, 
@echo  preserving the file extention:  YYYYMMDD_HHMMSS.ext
@echo  Synatx:  %0 filespec [/s]
@echo  A filespec (such as *.jpg) must be specified.
@echo  /s will recurse sub-folders and rename files 

Open in new window

LVL 33

Author Comment

Comment Utility
Wow, it sure would be nice to be able to edit comments in articles the way we can in questions.

My previous comment is in error - the parameter must be the filespec only, not the path, so:

  cd/d  F:\DCIM\100
  scriptname.bat .jpg
LVL 33

Author Comment

Comment Utility
DOH!    *.jpg

Shell Scripting

The term 'shell' refers to a general class of text-based command interpreters most often associated with the UNIX and Linux operating systems. Popular shells include Bourne, Debian Almquist (dash), Korn (ksh), Bourne Again (bash) and the C shell family (csh). Some view the DOS 'cmd' prompt as a minimal shell of sorts. It is also possible to install Cygwin on Windows and emulate a full Unix environment with complete shell capabilities. Terminal emulators, such as xterm, GNOME Terminal and OS X Terminal, can be used to access shell.