Use searched pattern in substitution

Sometimes we want to search for a pattern and use that pattern in replace as well. Like the substitution is an extension of what we already have in the file.

E.g. append all employee ages with 'years'

emp_name:Jane
emp_age:25
emp_name:John
emp_age:22
emp_name:Mary
emp_age:30
emp_name:Natalie
emp_age:29
other data here

Command

:%s/\(emp_age.*\)/\1 years/g

Breaking it up -

emp_name:Jane

%s - apply thru out the file
\( and \) - escape the parentheses 
(emp_age.*) - consider it a group
\1 - escape 1; 1 means the first pattern, in our case it's the only one
\1 years -  use the first pattern and append years to it
g - replace multiple occurrences on the same line

Output

emp_name:Jane
emp_age:25 years
emp_name:John
emp_age:22 years
emp_name:Mary
emp_age:30 years
emp_name:Natalie
emp_age:29 years
other data here

While I was typing this up, I thought let me try using a second pattern as well.

:%s/\(emp_age.*\)\(years\)/\1- \2/g 

If we run this in the above output we get

emp_name:Jane
emp_age:25 - years
emp_name:John
emp_age:22 - years
emp_name:Mary
emp_age:30 - years
emp_name:Natalie
emp_age:29 - years
other data here

Hope this helps.

NVL equivalent in Unix

Sometimes, we want to assign an alternate value to a variable if it were null. I recently found that it has a one liner in Unix.

var1=${new_var1:-$default_var1}

 Which means if $new_var1 is not null, use it else use the $default_var1.

Add newline in vim

I am often analyzing SQL queries from Informatica session log and they are wrapped around and difficult to read. That's when I needed to look for a way to insert a newline after every comma.
Great documentation and examples here.

http://vim.wikia.com/wiki/Add_a_newline_after_given_patterns

select wall_of_text,block_of_text,cube_of_text from textbook;

Add a newline after comma: s/,/\r&/g

select wall_of_text
,block_of_text
,cube_of_text from textbook

 Though brackets let you specify multiple values that you can replace (the vim wiki has an example), I was not able to figure out how to get a newline after every comma and "from". Drop in a comment if you know how to.

What is /dev/null?

I've used /dev/null for two purposes -
1. to discard output of a command
2. to suppress any potential error messages from a command

You may already know that you can redirect the output to a
1. file using greater than symbol ">" and 
2. command using pipe symbol "|"

Redirection to /dev/null works just as if the redirection was to a file. The only exception that the output cannot be retrieved.

Before we dive in, it's important to know that standard output is represented as 1 and standard error as 2.

ls -1 c* 2> /dev/null  #redirects the standard error (2)
ls -1 c* &> /dev/null  #redirects both the stderr (2) and stdout(1)
                              #variations include 2>&1, >>
ls -1 c* 1> /dev/null  #redirects the std output (1)
                          #variation is >

Here's an example to show an application of /dev/null. Run the code below to see how differently they behave.

if test ! -z "`ls -1 y* 2> /dev/null`"  ; then
   echo "Files starting with y exist."
else
   echo "Files starting with y DO NOT exist."
fi

if test ! -z "`ls -1 x* `"  ; then
   echo "Files starting with x exist."
else
   echo "Files starting with x DO NOT exist."
fi

To think about -
1. What happens when you do > filename?
2. What happens with you do > instead of 1> in the first example?
3. What does  this do? ls -1 c* 2> logfile > file.lst

How to count words in a delimited file?

I was asked how would one do word count in a delimited file and it took me by a surprise since my goto Unix command "wc" works only with tab or space delimited text.

From wc manual page - "A word is defined as a string of characters delimited by white space characters."

This is where sed is comes to rescue.

ruch:coding ruchi$ cat poem.csv 
twinkle,twinkle,little
star,how,I, wonder
what,you, are.
ruch:coding ruchi$ cat poem.csv | wc -w
       5
ruch:coding ruchi$ sed 's/,/ /g' poem.csv | wc -w
      10
ruch:coding ruchi$ 

Can you guess why wc returns 5 instead of 3 from poem.csv?

Substring : Right to Left

Informatica does not have this documented but you could use SUBSTR function to go from Right to Left as well. Just use a negative start.

IIF(SUBSTR(FILENM,-4)='.PDF', 'PORTABLE')

 

Regular Expressions in Informatica

Informatica supports PERL like regex syntax. So any parsing you would do in shell, can be done in Informatica as well.

REG_EXTRACT('Fiction__JK_ROWLING__HARRY_POTTER__MAX07','(\w+)(__\w+__)(\w+)(__.*)',3)

In the code above, __ are the delimiters in the string.

The regext part
\w looks for any alphanumeric character including underscore. Adding + to it makes it look for multiple alphanum characters.
() are to group the patterns
. looks for any character and adding & * makes it look for all the occurrences of any characters.
So we have four groups and the output is the 3rd group.

OUTPUT

HARRY_POTTER

Can you figure the other groups? Now can you figure how many groups we need if we just need the number of books in series or just the genre?