Paste vs Cat

There are two commands that are handy in combining files in Unix. If you want to combine files vertically (think cake layers) - use cat and to combine horizontally (think columns) - use paste.

Use awk if you want to switch the order of the column after the files have been combined.

cat file1
a1 b1 c1
a2 b2 c2
 

cat file2
d1 e1
d2 e2 

cat file1 file2
a1 b1 c1
a2 b2 c2
d1 e1
d2 e2 

paste file1 file2 
a1 b1 c1 d1 e1
a2 b2 c2 d2 e2 

paste file1 file2 > file3
awk -F' ' '{print $5,$2,$3,$4,$1}' file3
e1 b1 c1 d1 a1
e2 b2 c2 d2 a2 

If you liked this or found it useful, leave me a comment. Thanks!

Use searched pattern in substitution

Sometimes we want to search for a pattern and use that pattern in replace as well. Like the substitution is an extension of what we already have in the file.

E.g. append all employee ages with 'years'

emp_name:Jane
emp_age:25
emp_name:John
emp_age:22
emp_name:Mary
emp_age:30
emp_name:Natalie
emp_age:29
other data here

Command

:%s/\(emp_age.*\)/\1 years/g

Breaking it up -

emp_name:Jane

%s - apply thru out the file
\( and \) - escape the parentheses 
(emp_age.*) - consider it a group
\1 - escape 1; 1 means the first pattern, in our case it's the only one
\1 years -  use the first pattern and append years to it
g - replace multiple occurrences on the same line

Output

emp_name:Jane
emp_age:25 years
emp_name:John
emp_age:22 years
emp_name:Mary
emp_age:30 years
emp_name:Natalie
emp_age:29 years
other data here

While I was typing this up, I thought let me try using a second pattern as well.

:%s/\(emp_age.*\)\(years\)/\1- \2/g 

If we run this in the above output we get

emp_name:Jane
emp_age:25 - years
emp_name:John
emp_age:22 - years
emp_name:Mary
emp_age:30 - years
emp_name:Natalie
emp_age:29 - years
other data here

Hope this helps.

NVL equivalent in Unix

Sometimes, we want to assign an alternate value to a variable if it were null. I recently found that it has a one liner in Unix.

var1=${new_var1:-$default_var1}

 Which means if $new_var1 is not null, use it else use the $default_var1.

Add newline in vim

I am often analyzing SQL queries from Informatica session log and they are wrapped around and difficult to read. That's when I needed to look for a way to insert a newline after every comma.
Great documentation and examples here.

http://vim.wikia.com/wiki/Add_a_newline_after_given_patterns

select wall_of_text,block_of_text,cube_of_text from textbook;

Add a newline after comma: s/,/\r&/g

select wall_of_text
,block_of_text
,cube_of_text from textbook

 Though brackets let you specify multiple values that you can replace (the vim wiki has an example), I was not able to figure out how to get a newline after every comma and "from". Drop in a comment if you know how to.

What is /dev/null?

I've used /dev/null for two purposes -
1. to discard output of a command
2. to suppress any potential error messages from a command

You may already know that you can redirect the output to a
1. file using greater than symbol ">" and 
2. command using pipe symbol "|"

Redirection to /dev/null works just as if the redirection was to a file. The only exception that the output cannot be retrieved.

Before we dive in, it's important to know that standard output is represented as 1 and standard error as 2.

ls -1 c* 2> /dev/null  #redirects the standard error (2)
ls -1 c* &> /dev/null  #redirects both the stderr (2) and stdout(1)
                              #variations include 2>&1, >>
ls -1 c* 1> /dev/null  #redirects the std output (1)
                          #variation is >

Here's an example to show an application of /dev/null. Run the code below to see how differently they behave.

if test ! -z "`ls -1 y* 2> /dev/null`"  ; then
   echo "Files starting with y exist."
else
   echo "Files starting with y DO NOT exist."
fi

if test ! -z "`ls -1 x* `"  ; then
   echo "Files starting with x exist."
else
   echo "Files starting with x DO NOT exist."
fi

To think about -
1. What happens when you do > filename?
2. What happens with you do > instead of 1> in the first example?
3. What does  this do? ls -1 c* 2> logfile > file.lst

How to count words in a delimited file?

I was asked how would one do word count in a delimited file and it took me by a surprise since my goto Unix command "wc" works only with tab or space delimited text.

From wc manual page - "A word is defined as a string of characters delimited by white space characters."

This is where sed is comes to rescue.

ruch:coding ruchi$ cat poem.csv 
twinkle,twinkle,little
star,how,I, wonder
what,you, are.
ruch:coding ruchi$ cat poem.csv | wc -w
       5
ruch:coding ruchi$ sed 's/,/ /g' poem.csv | wc -w
      10
ruch:coding ruchi$ 

Can you guess why wc returns 5 instead of 3 from poem.csv?

Substring : Right to Left

Informatica does not have this documented but you could use SUBSTR function to go from Right to Left as well. Just use a negative start.

IIF(SUBSTR(FILENM,-4)='.PDF', 'PORTABLE')

 

Regular Expressions in Informatica

Informatica supports PERL like regex syntax. So any parsing you would do in shell, can be done in Informatica as well.

REG_EXTRACT('Fiction__JK_ROWLING__HARRY_POTTER__MAX07','(\w+)(__\w+__)(\w+)(__.*)',3)

In the code above, __ are the delimiters in the string.

The regext part
\w looks for any alphanumeric character including underscore. Adding + to it makes it look for multiple alphanum characters.
() are to group the patterns
. looks for any character and adding & * makes it look for all the occurrences of any characters.
So we have four groups and the output is the 3rd group.

OUTPUT

HARRY_POTTER

Can you figure the other groups? Now can you figure how many groups we need if we just need the number of books in series or just the genre?

Using Shell Variables in awk and sed

For sed - use double quotes around the variable.

sed '/'"$pattern"'/d' filename

Another option is to enclose entire sed command in double quotes.

sed "/$pattern/d" filename

For awk - use combination of double quotes single quotes double quotes

awk '{ if ($(1) == "'"$pattern"'" ) {print $2}}'

Optionally, you could create an awk variable from shell variable.

awk '{ if ($(1) == ipattern ) {print $2}}' ipattern=$pattern filename

Router Transformation Groups

Router Transformation's Groups are not the same as CASE statement conditions or what you may also call waterfall logic.

Router transformation routes data to all the groups that evaluate to TRUE unlike the CASE statement that executes the code only for the first true condition.

Awk one liner IF

Here is a quick one liner AWK for field comparison.

awk -F, '{if ($(1)=="0") {print $1}}' myfilename

What does it do?
From the comma separated file every record with first field as zero is printed.

It's important to keep parentheses as indicated. Although "print $1" is good; "if $1" poops.

Python - All Set for Comparison

I spent entire evening looking for a way to sort the characters in a string and figured it out.

1. Turn the String into List. 
2. Sort it. 
3. Join it back as a String.

And then, I stumbled upon a better way to do string comparison. Convert them to sets!!!

set(stringA).difference(stringB)

Note that it may not be same as
set(stringB).difference(stringA)

Details on set theory here:

http://docs.python.org/library/sets.html#set-objects

Discussion here:

http://stackoverflow.com/questions/1388818/how-can-i-compare-two-lists-in-python-and-return-matches

BASH shell only - brace expansion

You might have used brace expansion if you have tried this very popular way of backing up files - 

cp filename{,.bk}

The command above will copy filename to filename.bk. It saves you from writing the filename again. 

This neat stuff is not compatible with korn shell.

Some other examples to reinforce how this works.

echo John{Doe,Smith}
  JohnDoe JohnSmith
echo {John,Jane}{Doe,Smith}
  JohnDoe JohnSmith JaneDoe JaneSmith
  

Note the lack of space after comma.

vi or sed - Delete till end of line from pattern

Here's the regex that will match everything from the pattern until a newline is found.

Say you want to remove everything after "here" from this sample file.

code here    where?
code here   right here!
code here         look every where.

Used regex

s/here.*//

Explanation:

.*  

dot matches any character including spaces, numbers, alphabets but not new line.

asterisk matches any number of occurrences of a character

s/here.*/meow/ will result in

code meow
code meow
code meow




Shell script for multiple SED edits

Following shell script

1.    Reads the names of the files to be edited from the file "filelist".

2.    Executes two sed statements on each of those files and

3.    Overwrites the original file. 

#!/usr/bin
while read line
do
sed 's/0512/0505/g' $line > temp
sed 's/05\/13/05\/06/g' temp > $line
done < filelist 

Creating SQL statement for a Target Definition

I learned two things -
1. Don't blindly believe when people say it can't be done.
2. Informatica lets you create SQL statement to create target table.

I've been working to load this massive VSAM file with more than 1500 fields. Creating the source definition from the COBOL copybook was simple but creating ORACLE table manually for these many fields dampened my spirits.

I asked one of the more senior guys around here if he knew how to do it since he has been working on the same project with me. He said it wasn't possible and that I needed to do it manually.

I came back crushed and started reading Informatica Manual to delay my nemesis (so dramatic but seriously 1500 fields!).

It's then I learned that once the target definition has been created, Target Designer lets you "Generate/Execute SQL" for the target. BINGO! I was done in 2 minutes.

PS - To create target definition, I just dragged and dropped the source definition to target designer.

Automator - Get folder name using Shell Script

I wanted to move a folder from Movies to Music and rename both the folder and the videos.

Challenge was to 

1) Pass output of previous action to shell script.

2) Get the folder name.

#1 Passing output of previous action to shell script
Once you drag and drop Utilities/Run Shell Script action and change drop down "Pass Input" to "as arguments", it becomes evident that the values are passed in the usual way command line variables are passed to a shell script in Unix. i.e. in $@

If you see in the automator workflow above, I'm getting all the file names in the folder as $@ in my script. You might know that we can access the contents of $@ as $1, $2 etc and $# gives the total number of variables, in this case number of files.

Here's how I went over the list of files and renamed them all.

#Say for example $@ has "/Users/ruchi/Movies/Snowboarding 2010/videofile1.AVI /Users/ruchi/Movies/Snowboarding 2010/videofile2.AVI"
while [ ! -z "$1" ]
do
mv "$1" "`echo "$1" | sed 's/\[[aA-zZ,.]*\] [aA-zZ, ]* - [0-9][0-9] - //g'`"
shift
done

#2 Getting the folder name

This is more of a hack. I already had file names with path in #1, I checked the depth of my the folder from home (/Users/ruchi/) and used awk to get the folder name from a file name.

#for example let's say $1 has /Users/ruchi/Movies/Snowboarding 2010/videofile1.AVI
foldername=`echo "$1" | awk -F/ '{print $5}'` # That's right, not $4 but $5 due to leading "/"
echo $foldername #output: Snowboarding 2010

Here's another tutorial on how to pass variables to shell script.

Text Wrapping is Working

Many thanks to cimg's tutorial on dynamic cell height, text wrapping is working. If you want to keep the default font and size for the not-so-long data cells, keep the height fixed as 44.0f and use "initWithStyle:UITableViewCellStyleDefault" formatting on those cells.

iOS or Cocoa - lookup that method

I was feeling lost with what the parameters for a method and found the way to look it up without leaving xcode. Just CMD+double click the method.

Here's the link to thank for.
CGRectMake parameters ? « cocos2d for iPhone

And am writing this post using blogger bookmarklet (chrome addon) called "blog this". It lets you blog any webpage from that page.

Shell Variables are Global by Default

All variables in a shell script are global unless specifically declared as local. Local variables can be used only inside a function.

#!/bin/bash
# Global and local variables

carwash()
{
local vin="17DIGITS"
echo "IN Carwash. Your windscreen is:" $1 "and your VIN is:" $vin
post_carwash_status="BEARY CLEAN"
}

status="DIRTY"
carwash $status
echo "After carwash. Your windscreen is:" $status "and your VIN: " $vin
echo "Post Carwash Status: " $post_carwash_status

OUTPUT

IN Carwash. Your windscreen is: DIRTY and your VIN is: 17DIGITS
After carwash. Your windscreen is: DIRTY and your VIN: 
Post Carwash Status:  BEARY CLEAN

Notice that the local variable vin is not available outside the function.