Perl : What are Regular Expressions, by the way ?

A regular expression describes a set of strings, according to certain syntax rules.

In this post, I just want to give beginners an idea. The topic 'Regular Expressions' is quite big. Unix commands - grep and sed were the first to popularize the concept. Many programming language support regular expressions for string manipulation. Perl has a powerful regex engine. The support/use of regular expressions gives Perl very powerful text-processing capabilities.

man perlre gives you access to a man page dedicated to regular expressions and their use in Perl. Beginners, mastering regular expressions is the key to excelling in Perl programming !

Enjoy regular expressions :-)

Perl : Executing one line codes on the command line

Small codes of perl can be executed on command line with -e switch. The code is contained in single quotes. Following is an
example.

Command :

perl -e ' $\=" "; for (1 .. 9) { print; } '

Output :

1 2 3 4 5 6 7 8 9

Another frequently used switch is -n. This switch assumes 'while(<>){...}' around your code. Useful for processing all lines of the input coming through pipes. Example follows.

Command :

ls -l | perl -ne ' print if(/^d/); '

Here, output of ls -l (long listing) is piped to the perl command. -n makes the code loop over all the lines of the input that come from the previous command. The regular expression checks if the lines start with the letter d. Thus the code prints details of only the directories contained in the working directory. The command is equivalent to

ls -l | perl -e ' while(<>){ print if(/^d/); } '
(without -n switch)

-a is autosplit mode with -n or -p (splits $_ into @F ). See how it works

ls -l | perl -ane ' print $F[0]."\t".$F[7]."\n"; '

This prints the 0th and the 7th fields of the long directory listing (output of ls -l).

You can learn more about switches with this command
perl -h

To see version, built and other license related info,
perl -v

Perl : Sorting a list/array

Sorting of a list/array can be done with sort function. Reversing the order is easy; done with the function reverse. See following code and the output.

Code :

@fruit = ("banana","mango","apple");
@sorted_fruit = sort @fruit;

$,=" "; # $, is the output field separator (a predefined variable). This is used here to put spaces after printing every element of the array.
$\="\n"; # $\ is the output record separator. Used to add a newline to print in this example.

print @sorted_fruit;
print reverse @sorted_fruit; # reverses the order in the list

Output :

apple banana mango
mango banana apple

This Blog : Conventions I use

Hi !

This post does not have any code :-) Relief.

Wanted to make you aware of the conventions I have been following.
You'll observe that green color is for the code. Comments are preceded
by a '#' sign. This is because codes of perl, shell scripts and php
use # for commenting. You'll find commented code in brown (well
mostly; I may miss out on a few occasions). I'll also use the same
brown color for [key]words worth noting. Your suggestions in this
regard are welcome.

Feel free to share this blog with your friends. I believe this blog is
shaping into a good learning resource. If you wish to have posts on a
particular subject, let me know. I'll be more than happy to help you
as much as I can.

You'll see some more Perl code snippets in the coming posts.
Thanks for your patience.

Perl : 'Hello World !' and some more

This post is for those who want to start learning Perl. Save the following line in a file called 'hello.pl'. You can start learning Perl from the following example codes

print "hello world !";

and run it with

perl hello.pl

That's it ! It shows the output on the console.

Some more code with comments follows. You may save this in another file and run it with perl command.
Comments begin with a #.

Code :

$name = "ketan404"; ## a scalar. Note the $ sign. It's good to use my while variable declarations but we shall cover that la ter.

@fruits = ("apple","banana","mango"); ## this is an array. On the right side of the '=' sign is a list. Note the @ sign.

%fruit_prices = ("apple"=>4,"banana"=>1,"mango"=>10); ## this is called a hash. Names of the fruits are its keys and thier p rices the values.

## printing these

print $name."\n"; ## print the scalar and concatenate a newline to it. A dot (.) is used for concatenation.

print @fruits; ## works ! print can also take an array as an argument.

print @fruits."\n"; ## Oops ! You appended a newline to it ? Using @fruits in scalar context gives the number of elements in the array and thus prints 3 with a newline. You'll understand these things as you make progress :-)

## printing keys and values of a hash

foreach $k (keys %fruit_prices){
        print $k."=>".$fruit_prices{$k}."\n";
} ## works as expected
## keys function returns an array of keys of a hash.
## foreach or for are used for traversing an array

PHP : Object Oriented Programming in PHP

Object oriented programming is the use of objects to represent functional parts of an application. With php too, you can do object oriented programming to reduce and simplify the code. Following code is an example of the use of a php class.

Code :

<?php
class web_developer{

        var $skills; # class variables
        var $sal;

        function web_developer($sal){
        $this->sal=$sal;
        }

        function getSal(){
        return $this->sal;
        }

        function addSkill($technology,$yrs){
        $this->skills[$technology]=$yrs;
        }

        function getSkills(){
        return $this->skills;
        }

} # class ends here

$x=new web_developer(2000); # creating a web_developer (an object) with associated salary !

echo $x->getSal()."<br />";# retrieving a property (sal) of the object

$x->addSkill('php',5); # adding his skill-set and no. of years of experience
$x->addSkill('perl',4);

$x_skills=$x->getSkills();# retrieving the property (skills) of the object

foreach ($x_skills as $k => $v){
echo "$k - $v<br />";
}
?>

Output in a browser:

2000
php - 5
perl - 4

New on this Blog : Subscription by Email

Hello readers !

You might have noticed that I have added a form on the right for email subscriptions. Don't be afraid. I'm not going to fill your inboxes with emails :-) If you find the content on this blog useful and would like to be notified about the new posts, consider filling out the form. I shall send you email notification manually from my gmail account since I don't have an account with any mass-emailing website.

Enjoy Perl for some time now !

Perl : map() with your subroutines

You can use map function with your subroutines as well. Have a look at the following example.

Code:

## subroutine that returns double of a number (or whatever passed to it as an argument :-)
sub double_it{
        2*$_;
}

@nums = (1 .. 10); # array containing numbers 1 to 10
@doubles = map(&double_it, @nums); ## using map to create @doubles with subroutine double_it

$\=" "; # $\ is the built-in variable to specify the output record separator.

        for(@doubles){
        print;
        }

Output:

2 4 6 8 10 12 14 16 18 20

Perl : map function

See what the following code does.

Code:

@numarray = (97 .. 122);
@charray = map(chr, @numarray);
print @charray;

Output:

abcdefghijklmnopqrstuvwxyz

Explanation:

• .. is a range operator. @numarray contains numbers from 97 to 122
• map function applies function char on each element of @numarray. The output is stored in @charray.
• chr function returns the character represented by that NUMBER in the character set.  For example, "chr(65)" is "A".
  
• For more information on built-in perl functions, do 'man perlfunc'.

PHP : Showing an image randomly from a collection of images

Following code shows one .jpg image randomly from all images in a directory. The code needs to be saved in a file in the directory containing the images. This code can be used for creating random image gallery.

<?php
############### settings #############
$path = "/image_gallery/";
#######################################
$dir = $_SERVER['DOCUMENT_ROOT'].$path;
$ig = Array();
if (is_dir($dir)) {
    if ($dh = opendir($dir)) {
        while (($file = readdir($dh)) !== false) {
                if(ereg("\.jpg$",$file))
                array_push($ig,$file);
        }
        closedir($dh);
    }
}
$num = mt_rand(0,count($ig)-1);
echo "<img src=\"".$ig[$num]."\">";
?>

Notes :

1. The code opens a directory specified by $dir and puts all jpg filenames in an array $ig.
   
2. mt_rand function returns an integer between 0 and count of $ig i.e. number of jpg images in the directory.

CSS : Chaging cursor styles

By changing the value of cursor, you can change the appearance of the cursor. Cursor may have one of the following values.

• auto
• crosshair
• default
• help
  
• move
• pointer
• progress
• text
• wait
  

Move mouse over the lines below to see how the cursors look !

Cursor auto

When the value of cursor is set to auto, browser determines what style to show depending upon the context. i.e. if it sees
a link, it shows a pointer (a hand), default otherwise.

Cursor crosshair

Cursor help

Cursor default

Cursor move

Cursor pointer

Cursor progress

Cursor text

Cursor wait

Changing cursor styles with CSS adds great functionality to web pages. This can also be done in JavaScript. For Rich Web Applications (that use XMLHttpRequest for speed and better interaction), it's a good idea to change the cursor style to 'wait' after sending request to the server so that user expects some response from the server to be seen somewhere in the web page.

Following line sets cursor style to wait.

document.body.style.cursor="wait";

After loading completely the response from the server, the style may again be set to default.

Ajax - The key to Rich Web Applications

Asynchronous JavaScript And XML is a web development technique for developing rich web applications. AJAX is aimed at increasing speed, interactivity and usability of web applications. More info from Wikipedia.

Typically (in most cases), getting a response from server involves refreshing/reloading the page you are viewing or going to a new page. With ajax, you can do it without refreshing. This is ajax in short :-) The natural question that should arise is 'HOW?'. Modern browsers come with support for an object called XMLHttpRequest. This object is used in ajax applications that require making an http request to the server and loading the response without refreshing the page.

While creation of an online account you have to provide a username. And if somebody has already taken it, you come to know about it only after submission of the form. This can now be avoided ! You can send an http request to the server with the username filled in by the new user as soon as the user leaves the text-field (onBlur event of javascript). The response can be shown next to the textfield so that user notices it immediately (as soon as it appears there).

So why should you use ajax ?

1. With ajax, you don't have to refresh your page for server responses.
2. You can get only the required server reponse and show in the desired place on your web-page. This reduces the byte-transfer considerably. There's no need to transfer all the html that show your headers, footers, images and navigation.

The methods and properties of this object are explained below.

var request = new XMLHttpRequest(); // creates an object of XMLHttpRequest

open() - new request to server.
send() - Sends a request to the server.
abort() - aborts current request
readyState - provides the current HTML ready state.
responseText - the text that the server sends back to respond to a request.

More information on the subject can be found on this link.

Some examples -
1. www.gmail.com - most of us know what it is :-)
2. http://gollum.easycp.de/en/ - a wikipedia browser
3. http://a-i-studio.com/cmd/ - a WebShell !

Bash : sample loops (shell scripting)

Here are some sample loops in bash scripting. Feel free to leave a comment if something is not clear to you. I recommend using bash for learing shell scripting.
--------------------
#!/bin/bash
for i in `seq 1 10`
do
let i=i*5
echo $i
done

--------------------
# seq outputs a sequence of numbers (1 to 10)
# let is a bash built-in command that allows arithmetic operations to be performed and assigned to a variable as done above
---------------
for i in $(ls)
do
echo $i
done

---------------
j=0
while [ $j -lt 10 ]
do
echo $j
let j=j+1
done

----------------
# same example with until
# note the -eq switch in until loop
--------------------
j=0
until [ $j -eq 10 ]
do
echo $j
let j=j+1
done

---------------------

Shell scripting

What is shell scripting ?

A shell script is a file containing unix/linux commands. Shell scripts are written to avoid repetitive work of issuing the same set of commands over and again. Shell grammar allows us to put conditional and looping constructs (as commands) making it easy to write scripts to do complex tasks.

Following is a simple shell script that shows date, operating system info and list of files in a working directory.

----------------------------
## simple.sh (filename)
echo "a simple shell script"
date
uname -a
ls
----------------------------
Lines starting with a # are comments.

To execute this script,

bash simple.sh
or
sh simple.sh

Shell scripts can be used like any other operating system commands. For this they need to be placed in a directory which the shell searches for commands. You can see these directories by

echo $PATH

You can update the envronment variable PATH to add your own directory to it. Say you want to add ~/bin (directory bin in your home), issue the following commands (bash specific).

PATH=$PATH:$HOME/bin
export PATH

The scripts/programs in this directory need to have execute permission if you want to use them as commands. Execute permission can be added by using chmod command

chmod +x scriptfile

I'll post some examples of small shell scripts in next few posts.

Linux/Unix : File permissions, groups and access control

There are three types of permissions namely r(read), w(write), x(execute).

Read permission - in context of a file, this means you can read (and thus copy) the file. If you have read permission on a directory, you can see the contents of the directory (usually with ls command).

Write permission - on file means you can change the contents of the file. Write permission on a directory means you can create or delete files/directories in that directory.

Execute permission - If a file has execute permission you can run it just like a command. Usually, shell scripts, if they are used like a command, need x permission for the user. Execute permission on a directory means you can change to that directory using cd command.

On Linux (and unix flavors) users can be put in groups. A given set of users works on the files owned by a certain group. One user can be a member of many groups. Access control becomes effective with proper use of groups and permissions.

Long listing format (ls -l) shows the permissions on a file/directory.

The output is of the following form

-rw-rw-r--    1 abcd  web     24593 Mar  4  2006 test.txt
drwxrwxr-x    2 abcd  web      4096 Oct  7 16:06 temp_files

The first field shows the permissions on the files. Second field shows number of files in the corresponding directory. Third
 field (abcd) is the owner of the file. Next is the name of the group. Other fields are file-size, modification time and fil
ename in the same order.

The permissions field consists of 10 characters. First denotes the file-type. '-' for a plain file, 'd' for a directory. Following three characters show permissions of the owner of the file. In our case, the owner is 'abcd' and his permissions are 'rw-' (read, write but no execute). Following three characters are for the group (web). In our case, all members of this group have rw- permissions i.e. the same as the owner. Next three charaters are for others. Others have just 'read' permission.

Types of files
----------------
d - directory
l - link
p - pipe
b - block special device
c - character special device

Changing permissions
--------------------------
Only owner or root (administrator) can change permissions on a file. Following are some illustrations.

chgrp web filename
changes group of the file to web.

chmod g+w somefile
gives write permission on somefile to group.

chmod +x some_script
gives execute permission to all on some_script. Typically, shell scripts or some other executable files are given execute permissions.

chmod o-x some_script
removes execute permission for others (but retains for the owner and the group)

If in chmod command permissions start with a letter, following are the meanings of these letters

u - user or owner of the file
g - group members
o - others (rest of the world)
a - all

What does the following command mean then ?
chmod 664 some_filename

Permissions can also be set using octal value for the three bit pattern. Using this method, permissions on a file can be set in one go. r, w and x have corresponding values.

r = 4
w = 2
x = 1

Therefore, the above command assigns 6=4+2 i.e. 'read' and 'write' to the owner of the file, same for group whereas others have just 4 i.e. 'read' permissions on the file. To get more information on changing permissions refer the man page of chmod.

Perl : Arrays [2]

In this chapter, I'll discuss some frequently performed functions on Perl arrays.

Let's consider this array
@fruit = ("apple","banana","pineapple","mango","guava");

push @fruit,"plum";
Adds an element to the end of the array.

my $new_fruit = shift  @fruit;
The above statement removes the first element of @fruits and assigns it to $new_fruit.

my $popped_fruit = pop @fruit;
The above line pops (removes last element from the list) and assigns it to $popped_fruit.

Use of @_ in a subroutine

@_ is the default array that is passed to a subroutine.
See the following two subroutines and what they do

sub test{
($name,$surname,$age)=@_;
}

We can assign three scalars of a subroutine with the values of the arguments to the subroutines. @_ is the array of argument
s passed to the subroutine.

sub test2{
$name = shift;
$surname=shift;
$age=shift;
}

If these arguments are not going to be required for further processing, the default array can be shift'ed and the assignment
 be made.

Linux : Use of Pipes in commands

A pipe ( | ) is used to send output of one command to some other command. Many a time, use of pipe simplifies our task to a great extent. I'll discuss a few examples

Count number of files in a directory

ls | wc -l

List all users on a system who are running some processes (process owners)

ps -ef | cut -d " " -f1 | sort | uniq

List all process-listings that contain the word "mozilla"

ps -ef | grep mozilla

Find the word "services" in all .htm files

find . -name "*.htm" | xargs grep "services"

Sort lines in a file and print on the console

cat filename | sort

Killing mozilla

ps -ef | grep "mozilla" | cut -f1 | xargs kill

Print last 10 lines of a file

tail filename | lpr

Here's a good page on Linux Tips.

This wikipedia page has more information on unix pipes.

Perl : Arrays in perl

Creation of arrays

Following statements create perl arrays.

my @fruits = ("apples","bananas","mangoes","grapes");
my @numbers = (23, 42, 69);
my @mixed   = ("bananas", 4.2, 25);

Creating arrays from other arrays.

my @new_fruits = @fruits[1..3]

This cerates a new array which has "bananas","mangoes" and "grapes" of @fruits. This is called an array slice.
Some more ways ...

my @new_fruits = @fruits[1..$#fruits]

$# is a special variable which gives the last index of an array.

my @new_fruits = @fruits[1,2]
creates array of 2nd and 3rd element of @fruits.

Looping through arrays

$\="\n";
for (1..12){
print;
}

Probably the simplest form to understand. You can set a range with '..'
No argument is provided to 'print' so it takes the default input to the loop.
$\ is the output field separator. In our case, it's a newline. Alternatively, (without the first line) we could have written the print statement as follows.

print "$_\n";

$_ is the default input. As in the first case, perl assumes the default input '$_' as an argument to print (and other) statements if not mentioned explicitly.

The following code prints the locations of the perl modules on your system. @INC is the predefined perl array.

for (@INC){
print "$_\n";
}

If used in scalar context, @fruits gives the number of elements in @fruits (array).
e.g.

$x = @fruits;
print $x;

This prints 4 (number of elements in the array).

Some more on perl arrays will be posted here soon. Stay tuned !

Perl : Sorting a hash

Consider this hash.
%my_hash = ("apple"=>20,"banana"=>5,"chiku"=>10,"pineapple"=>30,"mango"=>50);

Keys of this hash are names of fruit and the values are numbers.
With 'sort' and 'keys' functions we get a hash sorted by its keys.

foreach $k(sort keys %my_hash){
        print "$k => $my_hash{$k}\n";
}

This prints
----------------------
apple => 20
banana => 5
chiku => 10
mango => 50
pineapple => 30
----------------------

Following code reverse-sorts the same hash by its values and prints it.

foreach $k(sort { $my_hash{$b} <=> $my_hash{$a} } keys %my_hash){
        print "$k => $my_hash{$k}\n";
}

Output
---------------------
mango => 50
pineapple => 30
apple => 20
chiku => 10
banana => 5
---------------------

The above can be done with this code.

foreach $k(reverse sort { $my_hash{$a} <=> $my_hash{$b} } keys %my_hash){
        print "$k => $my_hash{$k}\n";
}

Notes :

• 'reverse' has been added in the above code so that sorting takes place in revese way. However, the positions of $a and $b have changed too.
• <=> is used for numerical comparison. This is the comparison operator.
• 'cmp' is used for comparison of strings.
• The operators '<=>' and 'cmp' return either of 1,0,-1. These when used with 'sort' do the sorting !
  

Knowledge management with Perlfect (a search engine coded in Perl)

We tend to forget things that we knew once and time has to be spent either in digging the old archives/notes or learning the same thing afresh. Yes, many people have to deal with such a range of subjects that they can not recall everything when they need it. For example, a LAMP freelancer will find it hard to remember all Apache (web-server) directives and the new directives that came with the latest version. Besides, this programmer needs to have quick access to Linux, MySQL, PHP, Perl related documents. Efficient knowledge management will certainly boost his/her performance. This becomes more beneficial if knowledge can be shared in a group or organization.

This is typically done by running an intranet website on the Local Area Network and running a search engine to index the documents on it.

How can I set up an intranet website ?

You need to run a web-server. Apache which is a free software is the web-server of our choice. If you have a LAN, you should be able to access your intranet from any computer on the Local Area Network.

Knowledge management with Perlfect

Perfect is a search engine developed in Perl and distributed under GNU GPL (General Public License). You can download and use this for free. Once installed, you need to run the indexer to index the files on your intranet. You are all set to use the results !

Create a directory on your intranet to save your knowledge-documents and get the files indexed. A LAMP programmer may want to download php documentation, to be indexed so that he does not have to search the internet often. Text and pdf documents are also indexed so you can save your own tips/tricks in text files for future reference.

Contributing to the growth of your Knowledge base

If you are working in a group, encourage everybody to save their tips, findings in text files or any other format that Perlfect understands. While writing your knowledge-document, make sure you include all the possible search phrases that a user is likely to use. Assign the responsibility to run the indexer once in a week or more frequently depending upon the size of your group. Perlfect search form can be set to search within a particular directory so you can have different forms for different subdirectories of your intranet if the search starts yielding too many results.