Use searched pattern in substitution

Sometimes we want to search for a pattern and use that pattern in replace as well. Like the substitution is an extension of what we already have in the file.

E.g. append all employee ages with 'years'

emp_name:Jane
emp_age:25
emp_name:John
emp_age:22
emp_name:Mary
emp_age:30
emp_name:Natalie
emp_age:29
other data here

Command

:%s/\(emp_age.*\)/\1 years/g

Breaking it up -

emp_name:Jane

%s - apply thru out the file
\( and \) - escape the parentheses 
(emp_age.*) - consider it a group
\1 - escape 1; 1 means the first pattern, in our case it's the only one
\1 years -  use the first pattern and append years to it
g - replace multiple occurrences on the same line

Output

emp_name:Jane
emp_age:25 years
emp_name:John
emp_age:22 years
emp_name:Mary
emp_age:30 years
emp_name:Natalie
emp_age:29 years
other data here

While I was typing this up, I thought let me try using a second pattern as well.

:%s/\(emp_age.*\)\(years\)/\1- \2/g 

If we run this in the above output we get

emp_name:Jane
emp_age:25 - years
emp_name:John
emp_age:22 - years
emp_name:Mary
emp_age:30 - years
emp_name:Natalie
emp_age:29 - years
other data here

Hope this helps.

NVL equivalent in Unix

Sometimes, we want to assign an alternate value to a variable if it were null. I recently found that it has a one liner in Unix.

var1=${new_var1:-$default_var1}

 Which means if $new_var1 is not null, use it else use the $default_var1.

What is /dev/null?

I've used /dev/null for two purposes -
1. to discard output of a command
2. to suppress any potential error messages from a command

You may already know that you can redirect the output to a
1. file using greater than symbol ">" and 
2. command using pipe symbol "|"

Redirection to /dev/null works just as if the redirection was to a file. The only exception that the output cannot be retrieved.

Before we dive in, it's important to know that standard output is represented as 1 and standard error as 2.

ls -1 c* 2> /dev/null  #redirects the standard error (2)
ls -1 c* &> /dev/null  #redirects both the stderr (2) and stdout(1)
                              #variations include 2>&1, >>
ls -1 c* 1> /dev/null  #redirects the std output (1)
                          #variation is >

Here's an example to show an application of /dev/null. Run the code below to see how differently they behave.

if test ! -z "`ls -1 y* 2> /dev/null`"  ; then
   echo "Files starting with y exist."
else
   echo "Files starting with y DO NOT exist."
fi

if test ! -z "`ls -1 x* `"  ; then
   echo "Files starting with x exist."
else
   echo "Files starting with x DO NOT exist."
fi

To think about -
1. What happens when you do > filename?
2. What happens with you do > instead of 1> in the first example?
3. What does  this do? ls -1 c* 2> logfile > file.lst

How to count words in a delimited file?

I was asked how would one do word count in a delimited file and it took me by a surprise since my goto Unix command "wc" works only with tab or space delimited text.

From wc manual page - "A word is defined as a string of characters delimited by white space characters."

This is where sed is comes to rescue.

ruch:coding ruchi$ cat poem.csv 
twinkle,twinkle,little
star,how,I, wonder
what,you, are.
ruch:coding ruchi$ cat poem.csv | wc -w
       5
ruch:coding ruchi$ sed 's/,/ /g' poem.csv | wc -w
      10
ruch:coding ruchi$ 

Can you guess why wc returns 5 instead of 3 from poem.csv?

Shell script for multiple SED edits

Following shell script

1.    Reads the names of the files to be edited from the file "filelist".

2.    Executes two sed statements on each of those files and

3.    Overwrites the original file. 

#!/usr/bin
while read line
do
sed 's/0512/0505/g' $line > temp
sed 's/05\/13/05\/06/g' temp > $line
done < filelist 

ps to list only ppid

I generally use ps to get ppid to kill a process and finally figured a way to list just that.

ps -o ppid

And to list ppid without header, indicate that it should have a value. To display more information, add the headings preceded by output formatter -o.

ps -o ppid=
ps -o ppid= args= | grep Firefox 

When ssh relentlessly keeps asking for password

Check if your from and to home directories plus ssh directory have the correct permissions. All the details here -

http://www.dbvisit.com/forums/showthread.php?t=153

list directories

Listing directories in UNIX korn shell

$ls -d */
Mail/              sourcefiles/       mib/               dir3/
awc/               locks/             targetfiles/       dir4/
bin/               log/               perlo/             z12/

AWK - IF OR condition

OR operator : ||
AND operator: &&

awk -F, '{if ($1=="abc" || $1=="abd") print $0}' InputFile

-F,
InputFile is comma-separated (Field separator F is comma)

if ($1=="abc" || $1=="abd") print $0
Select and print entire record ($0) when first field ($1) is either "abc" or "abd"

Sort on a field

empfile
12345,Mary J,HR
34512,J Smith,Admin
34700,A Ryan,Admin
34900,B Wilson,HR
59000,C Diaz,HR

We want to sort the empfile by department (3rd field)

$sort -t, -k3 empfile
34512,J Smith,Admin
34700,A Ryan,Admin
12345,Mary J,HR
34900,B Wilson,HR
59000,C Diaz,HR

Option -t is to indicate field separator which here is comma
Option -k is the key indicating the position to sort on

Join two files based on a column

empfile
12345,Mary J,HR
34512,J Smith,Admin
34700,A Ryan,Admin
34900,B Wilson,HR
59000,C Diaz,HR

mgrfile
12345,HR
34700,Admin

So if we need to get name of the managers for each department then we would need to join the mgrfile with empfile on employee number.

$join -t , -o '2.2 2.3' mgrfile empfile
Mary J,HR
A Ryan,Admin

Option -t is to specify the field separator in the files.
Option -o is to list the fields that we need in the output. "2.3" means 3rd field of 2nd file.

By default join command joins on the first field of the files. So if we needed to get the manager for each employee, we would join the two files on department type.

To do this correctly both the files should be sorted** on the field being used for join.

$join -t, -13 -22 -o '1.2 1.3 2.1' empfilesorted mgrfilesorted
J Smith,Admin,34700
A Ryan,Admin,34700
Mary J,HR,12345
B Wilson,HR,12345
C Diaz,HR,12345

We have two new entries on the cmd above : -13 and -22. These can also be written as -1 3 and -2 2. These indicates the fields we are joining on.
-1 3 means 3rd field of 1st file

**See here on how to sort on a field.

Unix Fold

Fold is not just a formatting tool for word wrapping but can come in handy for text editing as well.

check this code
cat apple | fold -1 | sort | sed -n '/^[aeiouAEIOU]/p'

When we pass the content of apple to fold command, it breaks the content to 1 character per line as defined by the width parameter of fold.

$cat apple
apple

$cat apple | fold -1 (use fold -w1 for ksh)
a
p
p
l
e

And when we add sort to the above command and look lines beginning for vowels in
$cat apple | fold -1 | sort | sed -n '/^[aeiouAEIOU]/p'
a
e

Isn't that neat.

Base code from here.

Count vowels in a file and order by count descending

$ cat vowelfile
this
that
these

$ grep -io [aeiou] vowelfile | uniq -c | sort -rk1
2 e
1 i
1 a

This has three parts:

grep -io
-i ignores the case and -o prints just the part of the string it matches. So it will just list a, e, i, o or u instead of printing the complete line with vowel.

$ grep -io [aeiou] vowelfile
i
a
e
e

uniq -c
Counts the number of unique lines in the grep output.
$ grep -io [aeiou] vowelfile | uniq -c
1 i
1 a
2 e

sort -rk1
sort with option -r reverses the default sort order, which is ascending. By default sort is on the entire line. Option -k allows us to specify the field number to sort on. In the example the first field is the count and the second field is the vowel.


And to get the total counts of the vowel:
grep -io [aeiou] vowelfile | wc -l
wc with -l does the count on lines

Took the basic code from here
http://www.geekinterview.com/question_details/55489

diff and sdiff

diff can be used to compare differences between two files or two directories as well.

cat poem1
twinkle twinkle
oompa loompa

cat poem2
twinkle twinkle
little star
how I wonder
what you are.

diff poem1 poem2
2c2,4
< oompa loompa
---
> little star
> how I wonder
> what you are.

If we use the recursive option -r, just like we do with rm or find etc, we can compare content of the sub dir as well.

diff -r dir1 dir2
Only in dir1/dir11: testfile11
diff -r dir1/testfile1 dir2/testfile1
1,2d0
< this is a line in dir1 file test1
< this is another line in dir1/test1
Only in d1: t2

Without option -r

diff dir1 dir2
Common subdirectories: dir1/dir11 and dir2/dir11
diff dir1/testfile1 dir2/testfile1
1,2d0
< this is a line in dir1 file test1
< this is another line in dir1/test1
Only in d1: t2

For intensive file comparisons sdiff is better as it lets side by side comparison.

sdiff poem1 poem2
twinkle twinkle                    twinkle twinkle
oompa loompa                  |    little star
                             >    how I wonder
                             >    what you are.

Convert from EBCDIC to ASCII

dd if=InputFile of=OutputFile conv=ascii

The above command will convert EBCDIC InputFile to ASCII OutputFile. If you omit of=OutputFile, the output fill go to stdout.
If you want to use stdin, remove if=InputFile. Use control+D to terminate input.

I kept forgetting the dd command, hence this post, which it seems is versatile. It can be used to convert from lowercase or uppercase or convert only n bytes or blocks at a time. Further, I can combine the conversion keywords. e.g.

dd if=testE.dat conv=ebcdic,lcase

Zip and Unzip Unix

unzip filename.zip
will unzip the file

unzip -l filename.zip
will list the content of the archive

unzip -d chosenDir filename.zip
will unzip the the file in your chosenDir

zip filename.zip addNewfile.txt
will addNewfile.txt to the existing archive filename.zip


#

Cut command

To get any n characters of each line from a file, use cut.

Code to get first 3 characters of file

cut -c1-3 filename
217
207
284
238
216
219

Other examples
1) Get just specific character
echo "your text here" | cut -c2
> o

2) Get everything from a character
echo "your text here" | cut -c2-
> our text here

3) argument -f can be used to get specific fields from the file. Default delimiter is tab. It can be overridden by -d.
e.g.
cat rhyme
>
jack , and jill, went
up the,hill
to fetch,a pail
of water.

cut -f1 -d, rhyme
>
jack
up the
to fetch
of water.

4) argument -s along with -f is used to suppress lines that do not contain the field delimiter.

cut -f1 -d, -s rhyme
>
jack
up the
to fetch

5) Get a range of fields
cut -f2-3 -d, -s rhyme
>
and jill, went
hill
a pail

Cut option -b can be used to obtain same results as -c.

find a file based on date timestamp

find . -type f -newer guidedog | xargs -i mv {} going2dogs

With the above command, we are
1) looking for files (type -f) with date and timestamp newer than file guidedog.
2) then moving those files to directory going2dogs

guidedog is a dummy file which has the reference time we need. It can be created as -
touch -t 200911251340 guidedog

So all the files that were created after 1:40 PM on 11/25/2009 will move to directory doing2dogs.

Also, between two timestamps will work as

find . -type f -newer guidedog -a ! -newer anotherguidedog | xargs -i mv {} going2dogs

Update : Above command will move anotherguidedog also. To avoid that additional condition should be added before passing output to xargs:
-a ! -name anotherguidedog

Using translate function on Unix in a shell script


echo "TRansLate" | tr [:lower:] [:upper:]

echo "Ok to continue? y or n"
read respons
if [ `echo ${respons} | tr [:lower:] [:upper:]` = "Y" ] ||
[ `echo ${respons} | tr [:lower:] [:upper:]` = "YES" ] ; then
echo "Processing..."
else
echo "Exiting"
exit 1
fi


On executing the script

/home/ruchi> ./test
TRANSLATE
Ok to continue? y or n
yes
Processing...

DELETE

delete till end of file
dG

some other delete commands

added:
:g/^$/d
delete all (global) lines that have nothing from start (^) to end ($) i.e. blank or empty lines

:.,$d
delete lines from current line (.) to end of file ($)
dG will do the same. It's not an editor command so should be done directly on the line.

between marks
:'p,'qd
delete lines starting from mark p to mark q, including both p and q

delete current line and lines below
4dd
delete current line and 3 below it

delete current line and lines above it
3k4dd
3k moves the cursor 3 lines up and 4 dd deletes the required 4 lines