Could not parse the remainder in django Template

It's caused when we forget we cannot make HTML execute python commands with the exception of filters and tags. So a "for loop" ( a tag) can iterate over list items or dictionary items but create or append a list.

List of some of the filters and tags supported.
http://www.djangobook.com/en/1.0/appendixF/

When ssh relentlessly keeps asking for password

Check if your from and to home directories plus ssh directory have the correct permissions. All the details here -

http://www.dbvisit.com/forums/showthread.php?t=153

Using python shell for django project

If you want to test django code in python shell but are getting error messages regarding environment variables not set; set them by using that project's manage script.

python manage.py shell

credit

list directories

Listing directories in UNIX korn shell

$ls -d */
Mail/              sourcefiles/       mib/               dir3/
awc/               locks/             targetfiles/       dir4/
bin/               log/               perlo/             z12/

SERIES : awk#4 Simple Calculations

This is the same employee file that we used in the previous post.

$cat empf
100,Smith James,80000,101 Ave N,San Francisco,CA
200,Lloyd Beth,80000,2010 University St,San Francisco,CA
300,Doe John,90000,1005 Royal Dr,Hartford,CT
400,Day Matt,85000,100 W 4th Ave,Seattle,WA
700,Bell Amy,60000,201 Winner CT,Atlanta,GA
800,Hawn Ruby,75000,120 Red Carpet Ln,Los Angeles,CA

While creating the script, it's required to keep "{" on the same line as BEGIN and END. The script errors out otherwise.
NR is the inbuilt awk variable representing "Number of Records"

#!/usr/bin/awk
BEGIN{
FS=","
}
{
# sal $3 , name is $2
saltotal=saltotal+$3
}
END{
print "Number of Employees :", NR
print "Salary Total :", saltotal
print "Average Salary : ", saltotal/NR
}

Here's the output.

$awk -f avgsal.awk empf
Number of Employees : 6
Salary Total : 470000
Average Salary :  78333.3

Optionally, we can pass the file separator on the command line instead of having a BEGIN block in the script.

Note how awk automatically provides a float data type.

SERIES : awk#3 Match only a specific field

Here's the input file with employee data.

$cat empf
100,Smith James,80000,101 Ave N,San Francisco,CA
200,Lloyd Beth,80000,2010 University St,San Francisco,CA
300,Doe John,90000,1005 Royal Dr,Hartford,CT
400,Day Matt,85000,100 W 4th Ave,Seattle,WA
700,Bell Amy,60000,201 Winner CT,Atlanta,GA
800,Hawn Ruby,75000,120 Red Carpet Ln,Los Angeles,CA

Let's say we want only Connecticut CT employees. We could look for "CT" in the file.

$awk '/CT/ {print $0}' empf
300,Doe John,90000,1005 Royal Dr,Hartford,CT
700,Bell Amy,60000,201 Winner CT,Atlanta,GA

To check for a pattern strictly against a field, we can use tilde operator "~"

$awk -F, '$6 ~ /CT/ {print $0}' empf
300,Doe John,90000,1005 Royal Dr,Hartford,CT

On a similar note, to exclude all the CA employees.

awk -F, '$6 !~ /CA/ {print $0}' empf
300,Doe John,90000,1005 Royal Dr,Hartford,CT
400,Day Matt,85000,100 W 4th Ave,Seattle,WA
700,Bell Amy,60000,201 Winner CT,Atlanta,GA

Create Dictionary from a nested List

def crdic(a):
    b={}
    printOP=0
    for i in a:
        if len(i) != 2:
            print 'ERROR - Inner list has more/less than two elements'
            printOP=1
            break
        else:
            b[i[0]]=i[1]
    if printOP != 1:
        print 'Got list : ', a
        print 'Created dictionary : ', b

a=[['name','ruchi'],['nickname','r007']]
a1=[['name','ruchi','penguin'],['nickname','r007']]

crdic(a)
crdic(a1)

>>> 
Got list :  [['name', 'ruchi'], ['nickname', 'r007']] 
Created dictionary :  {'nickname': 'r007', 'name': 'ruchi'} 
ERROR - Inner list has more/less than two elements
>>>

SERIES : awk#2 Simple Patterns

cat monkey
Monkey Mink
100 Tree Blvd
Banana County
Monkeyville, MY
Zip 12001
555133 Area 255

Get lines that have a digit anywhere.

awk '/[0-9]+/ {print "Has a digit. : ", $0}' monkey
Has a digit. :  100 Tree Blvd
Has a digit. :  Zip 12001
Has a digit. :  555133 Area 255

Get lines that begin with a digit

awk '/^[0-9]+/ {print "Begins with digit. : ", $0}' monkey
Begins with digit. :  100 Tree Blvd
Begins with digit. :  555133 Area 255

Has characters somewhere

awk '/[aA-zZ]+/ {print "Has a word. : ", $0}' monkey
Has a word. :  Monkey Mink
Has a word. :  100 Tree Blvd
Has a word. :  Banana County
Has a word. :  Monkeyville, MY
Has a word. :  Zip 12001
Has a word. :  555133 Area 255

Has only letters

awk '/^[aA-zZ]+$/ {print "Has only letters. : ", $0}' monkey
<-- no output -->

Because space is not counted as letters.

SERIES : awk#1 Begin and End

Awk comes with inbuilt loop. It performs the given operations for each line in the input file provided they are not qualified by "BEGIN" or "END".

cat notxt
<-- empty file -->

cat sometxt
monkey goes to market

awk '{print "Hello World!"}' notxt
<--no ouput-->

awk 'BEGIN {print "Hello World!"} {print} END {print "Good bye!"}' notxt
Hello World!
Good bye!

awk 'BEGIN {print "Hello World!"} {print} END {print "Good bye!"}' sometxt
Hello World!
monkey goes to market
Good bye!

ex#1.
Since notxt is empty; awk doesn't iterate and no output is printed.

ex#2.
although notxt is empty; BEGIN and END statements are still executed and output produced for those commands.

Macro to help remove duplicate rows in excel spreadsheet

Sub duplicate_flg()
' Use to remove duplicates
' Check for duplicates based on columns colx and coly. Customize below.
' flag them in column colflg
' Rowcounts based on column - colx
' ******************************************************
' NEEDs a sorted sheet and assumes a header
' Runs on the active sheet in the active workbook
' ******************************************************

colx = 1
coly = 2
colflg = 3
Cells(1, colflg) = "Is Duplicate?"

lastrowcnt = Cells(Cells.Rows.Count, colx).End(xlUp).Row
'lastrowcnt = 7
ActiveWorkbook.Activate
Set ws = ActiveWorkbook.ActiveSheet

' header assumed. starts from row 2
For i = 2 To lastrowcnt
    If ws.Cells(i, colx) = ws.Cells(i + 1, colx) And _
        ws.Cells(i, coly) = ws.Cells(i + 1, coly) Then
        ws.Cells(i + 1, colflg) = "Y"
    End If
Next i

End Sub

Paste in vi without annoying auto indent

vi annoys the hell out of me with its auto-indenting during clipboard paste. So thanks to this post I've a solution now.

In command mode "set paste" and after pasting the text, turn it back off by "set nopaste".

what a breath of fresh air ;)

REPLACE between marks

While using vi, add marks to work with a chunk of lines.

To replace beginning of all lines, between the (inclusive) marks a and b, with ZZZ; we could write

:'a,'bs/^/ZZZ/

Where a has lower line number than b.

AWK - IF OR condition

OR operator : ||
AND operator: &&

awk -F, '{if ($1=="abc" || $1=="abd") print $0}' InputFile

-F,
InputFile is comma-separated (Field separator F is comma)

if ($1=="abc" || $1=="abd") print $0
Select and print entire record ($0) when first field ($1) is either "abc" or "abd"

Sort on a field

empfile
12345,Mary J,HR
34512,J Smith,Admin
34700,A Ryan,Admin
34900,B Wilson,HR
59000,C Diaz,HR

We want to sort the empfile by department (3rd field)

$sort -t, -k3 empfile
34512,J Smith,Admin
34700,A Ryan,Admin
12345,Mary J,HR
34900,B Wilson,HR
59000,C Diaz,HR

Option -t is to indicate field separator which here is comma
Option -k is the key indicating the position to sort on

Join two files based on a column

empfile
12345,Mary J,HR
34512,J Smith,Admin
34700,A Ryan,Admin
34900,B Wilson,HR
59000,C Diaz,HR

mgrfile
12345,HR
34700,Admin

So if we need to get name of the managers for each department then we would need to join the mgrfile with empfile on employee number.

$join -t , -o '2.2 2.3' mgrfile empfile
Mary J,HR
A Ryan,Admin

Option -t is to specify the field separator in the files.
Option -o is to list the fields that we need in the output. "2.3" means 3rd field of 2nd file.

By default join command joins on the first field of the files. So if we needed to get the manager for each employee, we would join the two files on department type.

To do this correctly both the files should be sorted** on the field being used for join.

$join -t, -13 -22 -o '1.2 1.3 2.1' empfilesorted mgrfilesorted
J Smith,Admin,34700
A Ryan,Admin,34700
Mary J,HR,12345
B Wilson,HR,12345
C Diaz,HR,12345

We have two new entries on the cmd above : -13 and -22. These can also be written as -1 3 and -2 2. These indicates the fields we are joining on.
-1 3 means 3rd field of 1st file

**See here on how to sort on a field.

PL /SQL Data Type - Constant

PL SQL has data type called constant. As the name implies, the field with this data type is immutable after declaration.

In the code below, un-commenting the meow line results in error.

declare
const_ruchi constant varchar2(5) :='Ruchi' ;

begin
--const_ruchi :='meow';
dbms_output.put_line('Schema Owner is ' || const_ruchi) ;
end;

Hello PL/SQL

--Started learning PL/SQL using this simple undaunting tutorial.

declare
luckynbr number(2);
abcdate date;

begin
select sysdate into abcdate from dual;
select dbms_random.value(0,100) into luckynbr from dual;

dbms_output.put_line('Your lucky number on ' || abcdate || ' is ' || luckynbr );

end;
/

Unix Fold

Fold is not just a formatting tool for word wrapping but can come in handy for text editing as well.

check this code
cat apple | fold -1 | sort | sed -n '/^[aeiouAEIOU]/p'

When we pass the content of apple to fold command, it breaks the content to 1 character per line as defined by the width parameter of fold.

$cat apple
apple

$cat apple | fold -1 (use fold -w1 for ksh)
a
p
p
l
e

And when we add sort to the above command and look lines beginning for vowels in
$cat apple | fold -1 | sort | sed -n '/^[aeiouAEIOU]/p'
a
e

Isn't that neat.

Base code from here.

Count vowels in a file and order by count descending

$ cat vowelfile
this
that
these

$ grep -io [aeiou] vowelfile | uniq -c | sort -rk1
2 e
1 i
1 a

This has three parts:

grep -io
-i ignores the case and -o prints just the part of the string it matches. So it will just list a, e, i, o or u instead of printing the complete line with vowel.

$ grep -io [aeiou] vowelfile
i
a
e
e

uniq -c
Counts the number of unique lines in the grep output.
$ grep -io [aeiou] vowelfile | uniq -c
1 i
1 a
2 e

sort -rk1
sort with option -r reverses the default sort order, which is ascending. By default sort is on the entire line. Option -k allows us to specify the field number to sort on. In the example the first field is the count and the second field is the vowel.


And to get the total counts of the vowel:
grep -io [aeiou] vowelfile | wc -l
wc with -l does the count on lines

Took the basic code from here
http://www.geekinterview.com/question_details/55489

diff and sdiff

diff can be used to compare differences between two files or two directories as well.

cat poem1
twinkle twinkle
oompa loompa

cat poem2
twinkle twinkle
little star
how I wonder
what you are.

diff poem1 poem2
2c2,4
< oompa loompa
---
> little star
> how I wonder
> what you are.

If we use the recursive option -r, just like we do with rm or find etc, we can compare content of the sub dir as well.

diff -r dir1 dir2
Only in dir1/dir11: testfile11
diff -r dir1/testfile1 dir2/testfile1
1,2d0
< this is a line in dir1 file test1
< this is another line in dir1/test1
Only in d1: t2

Without option -r

diff dir1 dir2
Common subdirectories: dir1/dir11 and dir2/dir11
diff dir1/testfile1 dir2/testfile1
1,2d0
< this is a line in dir1 file test1
< this is another line in dir1/test1
Only in d1: t2

For intensive file comparisons sdiff is better as it lets side by side comparison.

sdiff poem1 poem2
twinkle twinkle                    twinkle twinkle
oompa loompa                  |    little star
                             >    how I wonder
                             >    what you are.

Function in shell script

Declare the function before calling it.# Function to get password

functionGetpwd()
{
 echo "Please provide $USR database password"
 read PWORD
 echo "Connecting to Database"
}

Call the function

 functionGetpwd

Convert from EBCDIC to ASCII

dd if=InputFile of=OutputFile conv=ascii

The above command will convert EBCDIC InputFile to ASCII OutputFile. If you omit of=OutputFile, the output fill go to stdout.
If you want to use stdin, remove if=InputFile. Use control+D to terminate input.

I kept forgetting the dd command, hence this post, which it seems is versatile. It can be used to convert from lowercase or uppercase or convert only n bytes or blocks at a time. Further, I can combine the conversion keywords. e.g.

dd if=testE.dat conv=ebcdic,lcase

Remove Hyperlinks from excel worksheets

Sub RemoveHyperlink()
'
' Remove Hyperlink from each and every cell of you worksheet
'
'
Dim TotalCols as Integer
Dim TotalRows as Integer

For k = 1 To TotalCols
    For i = 1 To TotalRows
        Cells(i, k).Select
        Selection.Hyperlinks.Delete
    Next
Next
End Sub

Zip and Unzip Unix

unzip filename.zip
will unzip the file

unzip -l filename.zip
will list the content of the archive

unzip -d chosenDir filename.zip
will unzip the the file in your chosenDir

zip filename.zip addNewfile.txt
will addNewfile.txt to the existing archive filename.zip


#

Global Variables

If you want to access same variable in multiple functions, you declare it as global var1 in your program.

Global variables can be read from all the functions but if you need to modify them, you need to declare them again in that function. Why? So that you don't inadvertently cause cascading effect. It's a preventive measure.

But then how are we able to read the variables w/o declaring them? It's a friendly feature. Python looks for local variables and if they are not found; it looks for any global variable. Since you're just trying to read the variable; it won't (hopefully) be a costly mistake.

Merge excel cell values ; Retain Boundaries

I'm working with excel workbooks for more than 6 hours a day. I needed a quick way to consolidate values of selected cells of a column in the top most row.
e.g.
To merge values of cells 41 to 45 of column B, select the cells and hit the macro. All the values will be consolidated in cell 41.

Sub concate_cell_values()
' This macro consolidates the values
' of the "selected" cells in the top most cell
' Respects cell boundaries

Dim Rng1 As Range
Dim op As String
Dim col, ro As Integer
col = 0
ro = 0

Set Rng1 = Selection

For Each cell In Rng1
  cell.Activate
  If col = 0 Then
     col = ActiveCell.Column
     ro = ActiveCell.Row
  End If
  op = op & cell & " "
Next cell

Cells(ro, col).Value = Trim(op)

End Sub

clear Python IDLE

import os

os.system('clear') #for Mac or Linux

os.system('cls') #for Windows

Taken from:

http://bytes.com/topic/python/answers/39943-screen-clear-question

python - translate and maketrans

>>> mystring='these'
>>> string.maketrans('ts','75')
'\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\
x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./0123456789:;<=>?@ABC
DEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqr57uvwxyz{|}~\x7f\x80\x81\x82\x83
\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97
\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab
\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf
\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3
\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7
\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb
\xfc\xfd\xfe\xff'
>>> transtbl=string.maketrans('ts','75')
>>>
>>> mystring
'these'
>>> mystring.translate(transtbl)
'7he5e'
>>>
>>>
>>> mystring
'these'
>>> mystring.translate(None,'ts')
'hee'
>>>

To translate mystring we need a translate table which has to be 256 bytes long. So we will use maketrans to cook up that table for us.

syntax: string.maketrans("From","to")

Just use this table in translate function.

syntax: mystringhere.translate("translation table")

Also, translate can be used without a translation table. If we give translation table as "None" and then provide optional list, these listed characters will be deleted.

The latter way of translate can come handy if you want to check if an incoming string contains numbers.

e.g.

>>> mystring='ab1234'
>>> if len(mystring.translate(None,'0123456789')) < len(mystring) :
... print 'input string contains numbers'
...
input string contains numbers

#

vi multiple lines ; repeat last commands

To join multiple lines in vi, use 5,9j! in cmd mode.
We know that J (note uppercase) on a line joins it with the next line. "!" preserves whitespace between the lines but it didn't in ksh shell.

To apply last used substitute command through out the file:
e.g. %s/this/that/g

& to repeat the last used substitution for the first occurrence on the current line
&g to change all on the current line
%&g for entire file

Use . to repeat the last insert command. It inserts the last few inserted words/lines after the cursor.

vba - upper case

ucase('string')

and for lowercase
lcase("string")