Shell Scripting





The term 'shell' refers to a general class of text-based command interpreters most often associated with the UNIX and Linux operating systems. Popular shells include Bourne, Debian Almquist (dash), Korn (ksh), Bourne Again (bash) and the C shell family (csh). Some view the DOS 'cmd' prompt as a minimal shell of sorts. It is also possible to install Cygwin on Windows and emulate a full Unix environment with complete shell capabilities. Terminal emulators, such as xterm, GNOME Terminal and OS X Terminal, can be used to access shell.

Share tech news, updates, or what's on your mind.

Sign up to Post

Hi Linux Experts,

Currently, when I paste yaml or jason (or any text) in vi editor. It is not aligning well. So every time I have to ": set paste"

Is there a way I can set this on my session pls?

how to set to my current session
how to set this in bash_profile

please advice
Learn Ruby Fundamentals
LVL 13
Learn Ruby Fundamentals

This course will introduce you to Ruby, as well as teach you about classes, methods, variables, data structures, loops, enumerable methods, and finishing touches.

I have a large text file that consists of the top 11 lines of a bunch of dns zone files.  I need to parse this file and create a new file that generates a single line for every iteration of the pattern.  Essentially the pattern consists of 11 lines, followed by a domain line and then repeats.  For example, the pattern looks like the following:

; *** This file is automatically generated by Plesk ***
$TTL	86400

@	IN	SOA (
			2018020603	; Serial
			10800	; Refresh
			3600	; Retry
			604800	; Expire
			10800 )	; Minimum

Open in new window

This pattern repeats over and over for every domain name contained on our server.  What I would like is to parse the file for each of these 11 lines and return the following:

domain2.      SOA  2018020603 10800 3600 604800 10800

The "domain2" above comes from line 11 each time.  Can anyone write a script or perhaps a quick way in Notepad++ to get the data formatted the way I want?  Thanks!
a variable in linux shell.  i'm not really understanding the difference between an environment variable and a shell variable.
how long does either last?  how do you know which one is which?

if I set a variable:

VARIABLE="David" in a shell script
execute the script and go and type echo $VARIABLE... shouldn't it kick out David?

i know there are some variables you can export out of the shell, correct? but, i'm not really interested as i'm concentrating solely on bash at this point? or is that not how shells within linux work?


i guess now that i think of it ive heard of a single running of a script doing so within a 'shell'.. is this shell unique in it's processor and/or memory utilization?
why would i use anything other than bash? specialized customizations?

I've tried 3 various commands but still getting the 2 lines of "... â./proc/22063"  messages below:
# find . -type  f  -mtime -1  -size +1 -size -25M -print |grep -v "/proc" |grep -v "^/dev" |grep -v "^/sys" 2> /dev/null |grep -v "find:"
# find . -type  f  -mtime -1  -size +1 -size -25M -print |grep -v "^/proc" |grep -v "^/dev" |grep -v "^/sys" 2> /dev/null
# find . -type  f  -mtime -1  -size +1 -size -25M -print |grep -v "proc" |grep -v "^/dev" |grep -v "^/sys" 2> /dev/null

will persistently give the following:
find: â./proc/22063/task/22063/fdinfo/6â: No such file or directory  <==
find: â./proc/22063/fdinfo/6â: No such file or directory <==

Basically I'm trying to do a Clam AV scan & that 2 lines interferes:
DIR2SCAN=$(find / -type  f  -mtime -1  -size +1 -size -25M -print |grep -v /proc |grep -v /dev |grep -v /sys 2> /dev/null)
clamscan -ri $DIR2SCAN >> /tmp/log
ExpertsExchangePowershellSanitized.txtHow to do a recursive file lookup in powershell? I have a list of files in Folder A and need to look in Folder B and Folder C to see if that file exists in either B or C. If there is already a file with the same name as the file in A then I need to rename the file from Folder A until there is no file by that name in Folders B or C.  The file signature is this:  SomeCode_Year_Month_Day_Hour_Minutes_Seconds.pdf. When I look in Folder B and C and the file is found I change the name to increment by one second in the "Seconds" part then I copy it to Folder B and Folder C and delete it from Folder A. I had been using the file hash to find the duplicates but that seems like overkill. I just really need to check the file names. Folders A, B, and C are guaranteed to exist. I will need to keep checking the file name and incrementing by one second until there are no duplicate file names.
How do I programmatically get the filename of a sourced file?
I'm on a gnu Linux box.
I'm programmatically trying to identify the name of a bash script so I can create a log file name with the name of the script. I have done this for non-sourced scripts. The problem is that when I source a script $0 gives me bash, and everything under /proc/<pid> is 0 bytes. I'm using gnu Linux.
Experts, once again I need your help...

I have 2 files.
I need to find a pattern in file1, copy the line containing the pattern, search the pattern in file2, replace the line containing the pattern.
My sed does not have a -i option I'm afraid.

file1 example:

red tomato vegetable
green kiwi fruit
yellow banana fruit

file2 example:

red strawberry fruit
green apple fruit
yellow grapefruit fruit

I'd search for red, green, yellow  in file1, and replace the line containing that pattern in file2.
There's just 1 line with the pattern in each of the files.
Comes down in this case to file2 being overwritten by file 1.

Hi EE,

The following SQLCMD command:

sqlcmd -E -S $(ESCAPE_SQUOTE(SRVR)) -d master -Q "EXECUTE [dbo].[DatabaseBackup] @Databases = 'USER_DATABASES', @Directory = N'B:\MSSQL13.SQL2016\MSSQL\Backup', @BackupType = 'FULL', @Verify = 'Y', @CleanupTime = 48, @Compress = 'Y', @ChangeBackupType = 'Y', @CheckSum = 'Y'" -b

Generates individual folders for all the backups is there a way to insert them backups directly into the root directory, B:\MSSQL13.SQL2016\MSSQL\Backup?

Any assistance is welcome.

Thank you.
Hi EE,

I have the following script running in SQL server 2016:

-- Beware of server level collation and the way date formats are treated.
-- If date format needs to be forced use SET DATEFORMAT DMY

DECLARE @mydate nvarchar(50)
DECLARE @DeleteDateTime datetime
DECLARE @bakpath nvarchar(255)
DECLARE @BackupLocation nvarchar(255)
DECLARE @cmd nvarchar(4000)

EXEC master..xp_instance_regread N'HKEY_LOCAL_MACHINE', N'Software\Microsoft\MSSQLServer\MSSQLServer', N'BackupDirectory',@bakpath output
-- SET @bakpath = '<DRIVE>:\<PATH>'

-- Edit the number to subtract the number of hours required for 
-- the retention period i.e. -24 to delete files > 1 day
-- SET @DeleteDateTime = DateAdd(hh, -24, GetDate())

SET @DeleteDateTime = DateAdd(hh, -24, GetDate())
SET @mydate = (Select Replace(Convert(nvarchar, @DeleteDateTime, 111), '/', '-') + 'T' + Convert(nvarchar, @DeleteDateTime, 108))

-- Consider retention period if there is a need to split into separate jobs
-- copy and remove the type of backup files to suit target environment.

-- xp_delete_file will remove from 1 level of subdirectory only

-- Full backups
SET @BackupLocation = @bakpath
--SET @bakpath = @bakpath + '\FULL'-- customise path if required
set @cmd = 'EXECUTE master.dbo.xp_delete_file 0,N''' + @BackupLocation +''' ,N''bak'',N''' + @mydate + ''',1'
exec sp_executesql @cmd

Open in new window

The script isn't working, I have checked the registry the backup location is set correctly how would I go about debugging this, I suspect it's a path issue?

Additionally, I enabled the xp_cmdshell advanced config option on the SQL Server and that the SQL agent account has complete permissions to the backup directory folder so permissions should be ok.

Here is the SQLCMD used command to backup; sqlcmd -E -S $(ESCAPE_SQUOTE(SRVR)) -d master -Q "EXECUTE [dbo].[DatabaseBackup] @Databases = 'USER_DATABASES', @Directory = N'B:\MSSQL13.SQL2016\MSSQL\Backup', @BackupType = 'FULL', @Verify = 'Y', @CleanupTime = 48, @Compress = 'Y', @ChangeBackupType = 'Y', @CheckSum = 'Y'" -b

For some reason, the command doesn't insert the .bak files into the root directory but creates separate folders for each of the databases.

Any assistance would be appreciated.

Thank you.
We are receiving queries from website to our mailbox which include the following in the body of the email:

First Name:
Last Name:
Email Address:
CAP Code:
IDS Code:
List Price:

We are looking for a way to extract at least email addresses but if we could do First and Last Names and any other data that would be awesome. We would like to then use it with MailChimp so probably csv would be best format.

What would be the best way to accomplish that? Macro? I found macro for email addresses only and not sure how to extend it.

Any ideas appreciated.

OWASP: Avoiding Hacker Tricks
LVL 13
OWASP: Avoiding Hacker Tricks

Learn to build secure applications from the mindset of the hacker and avoid being exploited.

I have multi .txt file that include for example:
I want to make a windows script that read all .txt files in folder and put all lines that in them but not start with # in one file.
any idea ?

I am looking for a "free" way to take a PDF document save the contents of the PDF to a variable in a "Searchable" format..

So essentially I want to do the following :

get-webrequest -uri "Path to PDF"

Store this information into a variable in a searchable format and then run a query based on "Text" or a "String" withen that PDF file..

What is the easiest way of doing this .. Please provide a code snippet for it.
So I'm trying to write a script to create users in AD more efficiently. Naming  scheme is first initial + last name, however, if that's not available, I want it to take the first two letter of the first name + last name. This is what I have so far:

    [Parameter(Mandatory = $true)]

    [Parameter(Mandatory = $true)]

    [Parameter(Mandatory = $true)]
    [Parameter(Mandatory = $true)]

    [Parameter(Mandatory = $true)]

    [Parameter(Mandatory = $true)]

$DomainDn = (Get-AdDomain).DistinguishedName

$Username = $FirstName.substring(0,1) + $LastName

$validateusername = Get-AdUser -LDAPFilter "(sAMAccountName = $Username)"
    If ($validateusername -eq $Null) {"$Username = $FirstName.substring(0,1) + $Lastname"}
    Else {$Username = $FirstName.substring(0,2) + $LastName}

$Username = $Username.ToLower()
If ($Username.ToLower() -eq "kbryant")
    {Write-Host "Username:"$Username}
    {Write-Host "Username:"$validateusername} 

Open in new window

When I try to change the if state "If ($username -eq kbryant01)" it won't take the first two letters.

First name: Kobe
Last name: Bryant

is what i'm using to test

I am looking at doing the following :

$path = 'c:\folder1\folder2\folder3\folder4\folder5\folder6\folder7\folder12\randomfilename.txt'
$pathElements = $path.Split('\')
$objparams = $pathElements |select -index (1..$($pathelements.count -2))
$object = ""|select $objparams
$object |Add-member -membertype noteproperty -name Filename -value $null

Open in new window

Custom Object I want to populate

$object.filename = 
$object.folder1 =
$object.folder2 =
$object.folder3 =
$object.folder4 =
$object.folder5 =
$object.folder6 =
$object.folder7 =
$object.folder12 =

Open in new window

Values I would like in the $object folder properties.
filename = randomfilename.txt
folder1 = c:\folder1
folder2 = c:\folder1\folder2
folder3 = c:\folder1\folder2\folder3
folder4 = c:\folder1\folder2\folder3\folder4
folder5 = c:\folder1\folder2\folder3\folder4\folder5
folder6 = c:\folder1\folder2\folder3\folder4\folder5\folder6
folder7 = c:\folder1\folder2\folder3\folder4\folder5\folder6\folder7
folder12 = c:\folder1\folder2\folder3\folder4\folder5\folder6\folder7\folder12

Open in new window

Dynamically changing the number of folders in $object based on $path
This is using the bash shell script in our SuSE Linux Server 12. This is a "per-user" script, in which an SMS notification will be trigger for that user alone. We are using our internal SMS Gateway, and the handphone number has to be put into the subject. Please see the script contents below,

#! /bin/bash


# For Service notification
if [ "$NOTIFY_WHAT" = "SERVICE" ]; then
                echo "$SERVICESUBJECT $NOTIFY_SERVICEOUTPUT $NOTIFY_HOSTALIAS" |   /usr/bin/mailx -s "95521688"

# For Host notification
                echo "$HOSTSUBJECT $NOTIFY_HOSTOUTPUT $NOTIFY_HOSTALIAS" |   /usr/bin/mailx -s "95521688"

We have about 50+ personnel need to get the handphone alert with the same SMS …
im having a hard time completing this assignment. it's supposed to ping the IP addresses in IPADDR with the sizes in SIZER in two for loops,
extract the results and format the output.  some hosts are unreachable, and those must be identified, as the script is to run as quickly as possible. not wait for unreachable hosts to send back an icmp unreachable.


SIZER="64 128 256 512 1024 1280 1472 3000"

format=" %-20s %20s %20s %20s \n"

for X in $IPADDR
        for Y in $SIZER
                ping_results=$(ping -f -c 150 -s $Y $X)
                RESULT=$(echo $ping_results | grep -v "0% packet loss")
                if [[$RESULT = ""]]
                        echo "unreachable"
                IP=$(echo $ping_results |cut -d ' ' -f 2 )
                PACKETS=$(echo $ping_results |cut -d ' ' -f 4 )
                RTT=$(echo $ping_results | tail -1 | cut -d '/' -f 5 )
                SD=$(echo $ping_results | tail -1 | cut -d '/' -f 4 | grep -oE "\b\d\d\d")
                printf "$format" \
                $IP $PACKETS $RTT $SD \
echo "-------------------------------------------------------------"

I have the following function that I would like to add an additional parameter to, what would be the easiest way to add a Delete Flag to this function

Function Test-FileLocation {
    [CmdletBinding(DefaultParameterSetName = 'FullPath')]
        [Parameter(Position = 0, Mandatory = $true, ParameterSetName = 'FullPath')]
        [Parameter(Position = 0, Mandatory = $true, ParameterSetName = 'SplitPath')]
        [Parameter(Position = 1, Mandatory = $true, ParameterSetName = 'SplitPath')]
    If ($PSCmdlet.ParameterSetName -eq 'FullPath') {
        write-host "Starting full path file check for '$($Path)'"
        $fullPath = $Path
    Else {
        write-host "Starting split path file check for '$($Location)' - '$($FileName)'"
        $fullPath = Join-Path -Path $Location -ChildPath $FileName
    If (Test-Path -Path $fullPath) {
        write-host "File exists, proceeding with script"
        $filetest = "True"
    Else {
        write-host "File does not exist: FileTest failed"
        $filetest = "False"
#Test-FileLocation -location "C:\Test-WikiPageCreation\temp\WikiTemp\HyperV" -FileName "ict_servers_hyperv_Collection.txt"

Open in new window

I want it to test the file and return if it is available or not, but would also like to add a -remove parameter to the function for on demand deletion of the file.

After all is said and done I want it to function the following way:
Primary Full path
test-filelocation -path c:\somedirectory\test.txt 

Open in new window

Full Path with delete Option with verify
test-filelocation -path c:\somedirectory\test.txt  -remove

Open in new window

And the same with the split path option..
Any assistance would be greatly appreciated.

I have the following code fragment :

Function Test-FileLocation($location, $FileName){
    write-host "Starting File Check for $("$location\$filename")"
        if ($("$location\$filename")){
            write-host "File Exists Proceeding with script"
            $filetest = "True"
        }else {
            write-host "File Does not Exist : FileTest Failed"

Open in new window

I would like to modify this where I can add Nested Parameter Sets.

For instance in this code I would use something like :

test-filelocation -fullpath "c:\test\somefile.txt"


in the same code I could use something like this

test-filelocation -location "c:\test" -filename "somefile.txt"

What is the easiest way to acomplish the following:

I am looking at turning the following working code into a reusable function.

Function invoke-varcheck2(){
        if ($services.count -gt "200"){
            write-host "Check was successful"
        }else {
            write-host "Variable was not created successfully"
    . invoke-varcheck2

Open in new window

Here was my attempt that was less than successful :

Function invoke-varcheck($varname, $count){
    write-host "Attempting to Test Variable $varname"
        if (($+"$(($varname).count)" -gt $count){ 
            write-host "Check was successful"
        }else {
            write-host "Variable was not Created Successfully"

invoke-varcheck -varname services -count 200

Open in new window

Where did I go wrong with this?
Amazon Web Services
LVL 13
Amazon Web Services

Are you thinking about creating an Amazon Web Services account for your business? Not sure where to start? In this course you’ll get an overview of the history of AWS and take a tour of their user interface.


Powershell scripting issue :

Function get-spaceusagereport($loc){

set-location $loc
$colItems = Get-ChildItem $loc | Where-Object {$_.PSIsContainer -eq $true} | Sort-Object
$colresults = @()
    foreach ($i in $colItems) {
        try {
            write-host "Processing .... $i"
            $subFolderItems = @()
            $subFolderItems = Get-ChildItem $i.FullName -recurse -force | Where-Object {$_.PSIsContainer -eq $false} | Measure-Object -property Length -sum | Select-Object Sum
            $object = ""|select Fullpath, SpaceUsed, rawspace
            $object.FullPath = $i.FullName
            $object.SpaceUsed = "{0:N2}" -f $($subfolderitems.sum / 1MB) + " MB"
            $object.rawspace = $($subfolderitems.sum / 1MB)
            $colresults += $object
        catch {
            write-host "Could Not Access $i"

    #$object.Fullpath = $loc
    #$rootFolder = Get-ChildItem $($loc) -force | Where-Object {$_.PSIsContainer -eq $false} | Measure-Object -property Length -sum | Select-Object Sum
    #$object.spaceused = "{0:N2}" -f $($subfolderitems.sum / 1MB) + " MB"
    #$object.rawspace = $($subfolderitems.sum / 1MB)

    #$colresults += $object

write-host "The Top Space Using Folders are :"
$($colresults|sort-object rawspace -Descending|select -first 5)|select FullPath, SpaceUsed      

Open in new window

Most of this code functions, however, I am running into an issue retrieving the rootlevel folder $loc 's size in the $colresults object.

Any chance someone can assist with this :

- Thanks

What am I missing :

if I run the following , I am able to get the output I am looking for..

 enter-pssession -computername Computername
 $diskinfo = [System.IO.DriveInfo]::GetDrives()|select Name, AvailableFreeSpace, TotalSize
            $diskinfo |add-member -MemberType NoteProperty -Name PercentFree -value $null
            $diskinfo |add-member -MemberType NoteProperty -name ComputerName -value $ENV:Computername

            $diskinfo | ForEach-Object { 
                [float]$_.AvailableFreeSpace = $($_.AvailableFreeSpace/1GB)
                [float]$_.Totalsize = $($_.totalsize/1GB)
                [float]$_.PercentFree = (($($_.AvailableFreeSpace) / $($_.TotalSize)) * 100)


Open in new window

If I run :

$s = new-pssession -ComputerName Computername -name DiskInfo

$scriptblock = {$diskinfo = [System.IO.DriveInfo]::GetDrives()|select Name, AvailableFreeSpace, TotalSize
            $diskinfo |add-member -MemberType NoteProperty -Name PercentFree -value $null
            $diskinfo |add-member -MemberType NoteProperty -name ComputerName -value $ENV:Computername

            $diskinfo | ForEach-Object { 
                [float]$_.AvailableFreeSpace = $($_.AvailableFreeSpace/1GB)
                [float]$_.Totalsize = $($_.totalsize/1GB)
                [float]$_.PercentFree = (($($_.AvailableFreeSpace) / $($_.TotalSize)) * 100)


 invoke-command -Session $s -ScriptBlock $scriptblock

 import-pssession -session $s

Open in new window

I get a lightshow..

What I am attempting to do is the following :

The script will be run from Computer1 and collect information for several other computers

The information will then be stored on computer1 and exported to CSV.

Can someone please show me where I went wrong with this : I am sure I am missing something simple

Trying to capture this output to a file.  (manual SFTP)

sftp> cd /inbox
sftp> put testfile.txt
Uploading testfile.txt to /inbox/testfile.txt
testfile.txt                                                                                                       100%    5     0.1KB/s   00:00
sftp> bye

Using this command in my script, sftp -b /tmp/sftp_command_file $user@$node >> /tmp/sftp_error_file.   However, it does not write anything to the file.  

From script,
FILELIST=`ls -1 2> /dev/null`  
     for f in $FILELIST
         echo "put $f inbox/$f" >>/tmp/sftp_command_file      
         echo "chmod 777 inbox/$f" >> /tmp/sftp_command_file
         echo "quit" >> /tmp/sftp_command_file

sftp -b /tmp/sftp_command_file $user@$node >> /tmp/sftp_error_file

Thanks in advance for your help.
This is using SuSE 11 with SP4. Recently, I came across an online article discussed about how to use pmap to display the actual mapping, memory in active/cached and so on.
Please see the full command below,

 for i in `ps -eaf | grep “any application process” | grep -v grep | awk '{print $2}'`; do echo -n "PID $i actual memory usage is :" >> totaluse.txt; pmap -d $i | grep -i "writeable/private: " >> totaluse.txt; done

I was ended up with "nothing" when copy and paste the whole command to the terminal. Can I create a bash script? if so, please enlighten how, thanks.
Hello All,
when running the below python script from the terminal to change an interface MAC, I get the following message: "siocsifhwaddr cannot assign requested address"

any ideas of what could be happening?

#!/usr/bin/env python

import subprocess

interface = "eth0"
new_mac = "11:22:33:44:55:11"
print("[+] Changing MAC for " + interface + " to " + new_mac)"ifconfig " + interface + " down", shell=True)"ifconfig " + interface + " hw ether " + new_mac, shell=True)"ifconfig " + interface + " up", shell=True)

Using the below code in the bash script, i get the list of the available interface names.
How can i assign each found interface name to different variables, then use them to configure each interface name.

Thank you For your great help and Quick Prompts
(if you know a better way to find the existing interface names. please share ;)  )

ifconfig -a | sed 's/[ \t].*//;/^\(lo\|\)$/d'

Open in new window


Shell Scripting





The term 'shell' refers to a general class of text-based command interpreters most often associated with the UNIX and Linux operating systems. Popular shells include Bourne, Debian Almquist (dash), Korn (ksh), Bourne Again (bash) and the C shell family (csh). Some view the DOS 'cmd' prompt as a minimal shell of sorts. It is also possible to install Cygwin on Windows and emulate a full Unix environment with complete shell capabilities. Terminal emulators, such as xterm, GNOME Terminal and OS X Terminal, can be used to access shell.