Bash Shell Scripting - Basics

Working with text

1. Read a CSV file line by line and extract specific filed, and store it in a variable.

file_name="data.csv"

while read line

do

field1=$(echo -e $line | gawk -F, '{ print $1}')

field2=$(echo -e $line | gawk -F, '{ print $2}')

done < $file_name

2. Find and repace a string in a file.

sed -i 's/foo/bar/g' file.txt

Arithematic Operations

1. Increment a variable

count=$((count+1))

2. Decrement a variable

count=$((count-1))

Working with date and time

1. Get current date in yy-mm-dd format.

current_date="`date +%y-%m-%d`"

echo $current_date

2. Get backup time stamp in yymmdd-hhmmss format and store it in a variable.

bkptime=("`date +%y%m%d-%H%M%S`")

3. Get yesterday's date in yyyy-mm-dd format.

yesterday_date="`date --date=yesterday +%Y/%m/%d`"

echo $yesterday_date

3. Get 10 days old date in yyyy-mm-dd format.

day=("`date --date=\"10 day ago\" +%Y-%m-%d`")

OR

daysold=10

day=("`date --date=\"$daysold day ago\" +%Y-%m-%d`")

4. Get last day of the month

To get last day of current month:

date -d "-$(date +%d) days 1 month" +%Y-%m-%d

To get last day of last month:

date -d "-$(date +%d) days 0 month" +%Y-%m-%d

To get last day of last before month

date -d "-$(date +%d) days -1 month" +%Y-%m-%d

To get last day of next month

date -d "-$(date +%d) days 2 month" +%Y-%m-%d

To get last day of next to next month

date -d "-$(date +%d) days 3 month" +%Y-%m-%d

5. Get time difference (duration) between two different time-stamps in  HH:MM:SS format.

start_time="`date "+%Y-%m-%d %T"`"

end_time="`date "+%Y-%m-%d %T"`"

duration=$(echo $(( ( $(date -ud "$end_time" +'%s') - $(date -ud "$start_time" +'%s')))) | awk '{printf "%02d:%02d:%02d", $1/3600, ($1/60)%60, $1%60}')

If conditions

1. Check a vriable is having some value (it is non-zero)

if [ ! -z "$var" ]; then

     echo "Variable value is $val"

else

     echo "Variable has no value"

fi

2. Check if file exist

if [  -e "$file.csv" ]; then

     echo "File exist"

fi

3. Check if doesn't file exist

if [ !  -e "$file.csv" ]; then

     echo "File exist"

fi

4. Check if previously executed command is successful or not. If not successful exit from loop.

ssconvert "file.xls" "file.csv"

if [ "$?" == "0" ]; then

      echo "File converted successfully.."

else

exit

fi

Working with files

1. Covert EXCEL(.xls) file to CSV file.

ssconvert "file.xls" "file.csv"

2. Compress a directory using 7zip.

/usr/bin/7z a reportdir.7z reportdir > /dev/null

3. Upload a directory and its content to an FTP location

ftp_server="ftp.mydomain.com"

ftp_bkp_username="myusername"

ftp_bkp_password="mypassword"

FTP Over TLS

lftp -c "open -e \"set ftps:initial-prot \"\"; set ftp:ssl-force true; set ftp:ssl-protect-data true; mirror -R mydir; \" -u \"$ftp_bkp_username\",\"$ftp_bkp_password\" ftp://$ftp_server/mypath/"

FTP (without TLS)

ncftpput –R -u $ftp_bkp_username -p $ftp_bkp_password $ftp_server mypath mydir

4. Upload a file to an FTP location.

ftp_server="ftp.mydomain.com"

ftp_bkp_username="myusername"

ftp_bkp_password="mypassword"

FTP Over TLS

lftp -c "open -e \"set ftps:initial-prot \"\"; set ftp:ssl-force true; set ftp:ssl-protect-data true; put myfile; \" -u \"$ftp_bkp_username\",\"$ftp_bkp_password\" ftp://$ftp_server/mypath/"

FTP (without TLS)

ncftpput -u $ftp_bkp_username -p $ftp_bkp_password $ftp_server mypath myfile

5. Create an empty directory in an FTP location over TLS.

ftp_server="ftp.mydomain.com"

ftp_bkp_username="myusername"

ftp_bkp_password="mypassword"

lftp -c "open -e \"set ftps:initial-prot \"\"; set ftp:ssl-force true; set ftp:ssl-protect-data true; put myemptydir; \" -u \"$ftp_bkp_username\",\"$ftp_bkp_password\" ftp://$ftp_server/mypath/"

Working with user input

1. Read input from user.

echo -n "Enter date for processing data (yyyy/mm/dd) : "

read given_date

echo "You have entered $given_date"

Working with MySQL statements

1. Change a MySQL environment variable.

 db.cnf

[client]

user = root

password = mypassword

host = localhost

mysql --defaults-extra-file=db.cnf  -e "set global general_log='ON';"

2. Get the pids of sleeping MySQL connections and kill them whose sleep time is more than 100 sec.

sleeptime=100

/usr/bin/mysql -N -h $host -p$password -u $username -e "select id from information_schema.processlist where command='Sleep' and time > $sleeptime;" > kill_ids

while read id

do

/usr/bin/mysqladmin -h $host -p$password -u $username kill $id

done < kill_ids

rm -f kill_ids

3. Various options used while taking MySQL DB backup

/usr/bin/mysqldump -h $host --no-data  --routines -p$password -u $username $database > $database-Schema_only.sql

/usr/bin/mysqldump -h $host -p$password -u $username $database  --no-create-db --no-create-info --ignore-table=$database.cdrdetails --ignore-table=$database.clientsummary > $database-without_CDR_Data_tables.sql

--no-data à Do not write any table row information i.e structure only

--routines à Dump stored routines (procedures and functions)

--no-create-db à Do not write CREATE DATABASE statements

--ignore-table=db_name.tbl_name à Do not dump given table

--skip-add-drop-table à Do not add a DROP TABLE statement before each CREATE TABLE statement

Sending Mail

1. Send mail with body text stored in a file.

message.txt

Hi

  Some text here

Thanks & Regards

xxxxxxxxxxxxx

mail -s "Log Rotation Failed" abc@xyz.com < message.txt

2. Send mail with attachment.

mail -a Report.txt -s "Log Rotation Failed" abc@xyz.com < message.txt

3. Send a mail with Cc
mail -a Report.txt -s "Log Rotation Failed" -c id@domain.com abc@xyz.com < message.txt

Working across multiple scripts

1. There will be situations, where multiple scripts dependent on same data. Such data can be stored in one script, and can be exported to multiple scripts. This will help us in managing the data centrally that changes time to time.

00_CONFIG.sh

#!/bin/bash

host="192.168.10.23"

export host

database=mydbname

export database

port=3306

export port

serverids="1 2 3 4 5 6 7 8 9 10"

export serverids

recenthours="3 2 1"

export recenthours

mysql_username="myuserid"

export mysql_username

mysql_password="mypassword"

export mysql_password

01_DAILY DATA_VERIFY.sh

#!/bin/bash

source 00_CONFIG.sh

for id in `echo $serverids`

do

..................

..................

done

02_DAILY DATA_BACKUP.sh

#!/bin/bash

source 00_CONFIG.sh

for id in `echo $serverids`

do

..................

..................

done

03_DAILY DATA_DELETE.sh

#!/bin/bash

source 00_CONFIG.sh

for id in `echo $serverids`

do

..................

..................

done

Important Note: The above example will work when you run scripts manually in shell. In case if you want to schedule the jobs using crontab, you need to declare those variables (CONFIG.sh) in cron environment. In this case it looks like this.

# crontab -e 
host="192.168.10.23"

database=mydbname

port=3306

serverids="1 2 3 4 5 6 7 8 9 10"

recenthours="3 2 1"

mysql_username="myuserid"

mysql_password="mypassword"

30    5     *    *     *    /mypath_to_scripts/01_DAILY DATA_VERIFY.sh
59    23     *    *     *    /mypath_to_scripts/01_DAILY DATA_BACKUP.sh