Top 10 Linux Virtualization Software

Virtualization is the latest buzz word. You may wonder computers are getting cheaper every day, why should I care and why should I use virtualization? Virtualization is a broad term that refers to the abstraction of computer resources such as:

1. Platform Virtualization
2. Resource Virtualization
3. Storage Virtualization
4. Network Virtualization
5. Desktop Virtualization

This article describes why you need virtualization and list commonly used FOSS and proprietary Linux virtualization software.

Why should I use virtualization?

• Consolidation – It means combining multiple software workloads on one computer system. You can run various virtual machines in order to save money and power (electricity).
• Testing – You can test various configuration. You can create less resource hungry and low priority virtual machines (VM). Often, I test new Linux distro inside VM. This is also good for students who wish to learn new operating systems and programming languages / database without making any changes to working environment. At my work place I give developers virtual test machines for testing and debugging their software.
• Security and Isolation – If mail server or any other app gets cracked, only that VM will be under control of the attacker. Also, isolation means misbehaving apps (e.g. memory leaks) cannot bring down whole server.

Open Source Linux Virtualization Software

1. OpenVZ is an operating system-level virtualization technology based on the Linux kernel and operating system.
2. Xen is a virtual machine monitor for 32 / 64 bit Intel / AMD (IA 64) and PowerPC 970 architectures. It allows several guest operating systems to be executed on the same computer hardware concurrently. XEN is included with most popular Linux distributions such as Debian, Ubuntu, CentOS, RHEL, Fedora and many others.
3. Kernel-based Virtual Machine (KVM) is a Linux kernel virtualization infrastructure. KVM currently supports native virtualization using Intel VT or AMD-V. A wide variety of guest operating systems work with KVM, including many flavours of Linux, BSD, Solaris, and Windows etc. KVM is included with Debian, OpenSuse and other Linux distributions.
4. Linux-VServer is a virtual private server implementation done by adding operating system-level virtualization capabilities to the Linux kernel.
5. VirtualBox is an x86 virtualization software package, developed by Sun Microsystems as part of its Sun xVM virtualization platform. Supported host operating systems include Linux, Mac OS X, OS/2 Warp, Windows XP or Vista, and Solaris, while supported guest operating systems include FreeBSD, Linux, OpenBSD, OS/2 Warp, Windows and Solaris.
6. Bochs is a portable x86 and AMD64 PC emulator and debugger. Many guest operating systems can be run using the emulator including DOS, several versions of Microsoft Windows, BSDs, Linux, AmigaOS, Rhapsody and MorphOS. Bochs can run on many host operating systems, like Windows, Windows Mobile, Linux and Mac OS X.
7. User Mode Linux (UML) was the first virtualization technology for Linux. User-mode Linux is generally considered to have lower performance than some competing technologies, such as Xen and OpenVZ. Future work in adding support for x86 virtualization to UML may reduce this disadvantage.

Proprietary Linux Virtualization Software

• VMware ESX Server and VMWare Server – VMware Server (also known as GSX Server) is an entry-level server virtualization software. VMware ESX Server is an enterprise-level virtualization product providing data center virtualization. It can run various guest operating systems such as FreeBSD, Linux, Solaris, Windows and others.
• Commercial implementations of XEN available with various features and support.
  • Citrix XenServer : XenServer is based on the open source Xen hypervisor, an exceptionally lean technology that delivers low overhead and near-native performance.
  • Oracle VM : Oracle VM is based on the open-source Xen hypervisor technology, supports both Windows and Linux guests and includes an integrated Web browser based management console. Oracle VM features fully tested and certified Oracle Applications stack in an enterprise virtualization environment.
  • Sun xVM : The xVM Server uses a bare-metal hypervisor based on the open source Xen under a Solaris environment on x86-64 systems. On SPARC systems, xVM is based on Sun’s Logical Domains and Solaris. Sun plans to support Microsoft Windows (on x86-64 systems only), Linux, and Solaris as guest operating systems.
• Parallels Virtuozzo Containers – It is an operating system-level virtualization product designed for large-scale homegenous server environments and data centers. Parallels Virtuozzo Containers is compatible with x86, x86-64 and IA-64 platforms. You can run various Linux distributions inside Parallels Virtuozzo Containers.

Personally, I’ve used VMware ESX / Server and VirtualBox.

METHODS FOR MIGRATION OF VM’S IN LINUX HOST TO WINDOWS HOST

In Linux the virtualization tool used can be one of the above mentioned. Let us take the case of VIRTUAL BOX. The guest in VBOX will have an extension “.vdi” and in WINDOWS HYPER-V it will be “.vhd”. The steps for converting .vdi to .vhd are given below:

There are times when you use Virtualbox and for some reason need to convert over to VMWare or Hyper-V (or other).

Finding this process on the Internet is tedious, and usually involves using QEMU, an open-source Virtualization product to perform the .raw –> .vmdk step. However, if you are running on Windows, the QEMU Windows port will not perform the conversion without failure (at least on Windows 7!). So with that, I decided to go another route with a handy tool that I’ve used for years in some variation or another, WinImage.

Tools you’ll need:

. Sun VirtualBox – http://www.virtualbox.org. WinImage (Shareware) – http://www.winimage.com

Requirements:

Free disk space should be around 2 times the size of the .vdi disk image file size.

For the sake of this document, we will use .vmdk as our output file, but the conversion steps are the same when it comes to .vhd.

Convert .raw disk image to .vmdk format using WinImage

• Open WinImage, click on ‘Disk’> ‘Convert Virtual Hard Disk image…’
  
• Next to the ‘File name:’ field, click on the file type drop-down and select ‘All files (*.*)’.
  
• Navigate to the location where you stored your outputted .raw disk file and double-click it.
  
• Choose whether you wish to ‘Create Fixed Size Virtual Hard Disk’ or ‘Create Dynamically Expanding Virtual Hard Disk’ (I typically pick the latter) and click ‘OK’.
  
• Navigate to a folder where you wish to store the newly converted image to. Next to ‘Save as type:’ (for the sake of this How-to) choose ‘VMWare VMDK (*.vmdk). and click ‘Save’.You should see a ‘Reading disk’ progress indicator giving you the status of the conversion process. I’ve converted 30Gb images in about 10 minutes or less…but I have no firm numbers.
  
• Once the conversion is complete, you’ll see a dialog box that will ask you if you wish to connect to the partition. Click ‘OK’ if you wish to view the content.
  

Import your disk images into your existing Virtual Infrastructure

Now that the files are converted, copy or move your converted disk image files to your virtualization software’s datastore/disk storage folder.

Once moved/copied, you should now be able to create a new Virtual Machine and utilize the disks you just converted.

Note that you will need to install the proper guest additions/tools to the virtual machine when you get it booted, so you will likely not have network access right off the bat.

Conclusion

Indeed, that was a quick How-to, but it’s an important process that gets easily overlooked. The WinImage step makes quick work of converting the .raw image…and it is easy!

 #python setup.py install 

Now it’ll be installed(Python2.7 will be automatically installed with it)!Then create an environment by

#python virtualenv.py ENV

#virtualenv ENV

Done!!!

Note:-Now new binaries will be available under ‘/root/ENV/bin‘. We’re going to use commands from this directory for further.

Activate the new virtual environment by

#source ./ENV/bin/activate

(Here ENV is the virtual environment)

From now onwards,it’ll take commands from virtual environment itself if available! Otherwise it will go for default system binaries! If you want to return to the real environment, trigger

#deactivate

1.2.Setup tools
This will be installed automatically along the installation of ‘virtualenv’.
1.3.Install Genshi by

#easy_install Genshi

1.4.Sqlite can be installed using ‘yum’.Install ‘devel’ for sqlite also!

2.mod_wsgi module for apache
For compiling mod_wsgi with apache,simply follow the instructions given in: http://code.google.com/p/modwsgi/wiki/QuickInstallationGuide

B.Installing Trac

#easy_install Babel==0.9.5
#easy_install Trac
#easy_install -U Trac
#easy_install -U Trac==dev

C.Creating a project environment

1.Before you proceed,create a directory for keeping trac project files under a directory accessible to apache.(Here default path is used)

#mkdir /var/www/trac

Now make sure apache user can access all the directories and files under ‘trac’

#chown -R apache.apache /var/www/trac

2.Create an environment

#trac-admin /var/www/myproject initenv

Enter the name of project and database when you are prompted for. Leave database name as default if you wish to use the default database!

D.Deploying Trac

#tracd --port 8000 /var/www/trac/myproject

(include ‘-s’ for single instance)

Now open browser and go to : http://localhost:8000 . Trac web-based interface will be appeared!

E.Granting admin rights to admin user

#trac-admin /var/www/myproject permission add admin TRAC_ADMIN

Warning:- If you failed to activate virtual environment as described under section A-1.1, you will need to specify path for all the commands (For eg:/root/ENV/bin easy_install) and you may encounter an error “  File “/usr/lib/python2.4/site-packages/_xmlplus/parsers/expat.py”, line 4, in <module> from pyexpat import * ImportError: /usr/lib/python2.4/site-packages/_xmlplus/parsers/pyexpat.so: undefined symbol: PyUnicodeUCS4_Decode”. “.

For resolving this,trigger

#export  PYTHONPATH=/path/to/lib/python2.7/site-packages/

on each shell or edit files like /etc/profile or /etc/bashrc etc to take a permanent effect. But,this is not recommended and you should activate the virtual environment instead (see A-1.1) to avoid this scenario!!  Happy doing

Things would have been easy if I had used lsof command

The logical volume was named /dev/lvols/ubuntu64

At first

$lvremove /dev/lvols/ubuntu64

This command couldn’t remove the logical volume the next command I tried was

root@manager-desktop:~# dmsetup remove --force /dev/lvols/ubuntu64
device-mapper: remove ioctl failed: Device or resource busy
Command failed

So some process is using this logical volume and to find the process id the next command was

root@manager-desktop:~# lsof /dev/lvols/ubuntu64
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
vblade  31213 root    3u   BLK  252,4      0t0 115749 /dev/lvols/../mapper/lvols-ubuntu64

root@manager-desktop:~# kill -9 31213

Now issued the commands

root@manager-desktop:~# dmsetup remove --force /dev/lvols/ubuntu64

root@manager-desktop:~# lvremove /dev/lvols/ubuntu64
 Logical volume "ubuntu64" successfully removed

Voila.. simple fix and point to remember is never forget lsof command

We are looking for  BE/Btech freshers from IT, CSE ,EEE, EC branches for our Development and Technical Support wings. Proactive candidates with exceptional coding skills. The candidate must be capable of quickly learning new technologies and capable of working on applications deployed in distributed environment. Basic knowledge of Linux platform , Perl, PHP  will be a plus.

Eligibility Criteria:

• BTech/ BE – CSE, ECE , IT, EEE( 60%above final results and should not have any back papers )
• Excellent programming skills ( C, C++) .Basic knowledge of PHP/ Perl/Python will be a plus. RHCE certified candidates for Technical Support wing.
• Consistent academic record
• Proactive attitude
• Good written and oral communication.
• Should be a self starter and must be willing to work on complicated tasks.
• A quick learner.
• Familiarity with software development process .

Selection procedure:

• Written Test ( Technical and Aptitude – objective type )
• Technical Interview
• Practical test .
• HR Round

Terms:

• 6 months on job training period
• flat hierarchy
• ample opportunities to grow
• opportunity to grow with a fast growing  organization

How to apply:

Send your resumes to careers[at] sparksupport[dot]com with a covering letter and subject line BTech Freshers .The openings are immediate and the candidates should be ready to join ASAP.

Download and extract the tarball of sphinx

$ tar xzvf sphinx-0.9.8.tar.gz
$ cd sphinx

Run the configuration program:

$ ./configure

We can specify the location where sphinx should be installed by using –prefix option.
Build the binaries.

$ make

Install the binaries.

$ make install

By default, Sphinx utilities are installed in /usr/local/bin/. Sphinx has three components: an index generator, a search engine, and a command-line search utility: The index generator is called indexer. It queries your database, indexes each column in each row of the result, and ties each index entry to the row’s primary key. The search engine is a daemon called searchd. The daemon receives search terms and other parameters, scours one or more indices, and returns a result. If a match is made, searchd returns an array of primary keys. Given those keys, an application can run a query against the associated database to find the complete records that comprise the match. Searchd communicates to applications through a socket connection on port 3312. The handy search utility lets you conduct searches from the command line without writing code. If searchd returns a match, search queries the database and displays the rows in the match set. The search utility is useful for debugging your Sphinx configuration.

To use Sphinx, you will need to create a configuration file. Default configuration file name is sphinx.conf. All Sphinx programs look for this file in current working directory by default. Sample configuration file, sphinx.conf.dist, which has all the options documented, is created by configure. Copy and edit that sample file to make your own configuration.

$ cd /usr/local/etc
$ cp sphinx.conf.dist sphinx.conf

To start with sphinx, you must define one or more sources and one or more indexes. A source identifies the database to index, provides authentication information, and defines the query to use to construct each row. An index requires a source (that is, a set of rows) and defines how the data extracted from the source should be cataloged. You define your source(s) and index(es) in the sphinx.conf file. Sample configuration file is setup to index documents table from Mysql database test. So there’s example.sql sample data file to populate that table with a few documents for testing purposes.

$ mysql -u test < /usr/local/sphinx/etc/example.sql

We need to specify database information in configuration file like the following.

source src1
{
# data source type. mandatory, no default value
# known types are mysql, pgsql, mssql, xmlpipe, xmlpipe2, odbc
type                                    = mysql

#####################################################################
## SQL settings (for 'mysql' and 'pgsql' types)
#####################################################################

# some straightforward parameters for SQL source types
sql_host                                = localhost
sql_user                                = test
sql_pass                                = test
sql_db                                  = test
sql_port                                = 3306  # optional, default is 3306

Next, create a query to produce rows to be indexed. The sql_query must include the primary key you want to use for subsequent lookups, and it must include all the fields you want to index and use as groups. It is specified in the configuration file as the following.

sql_query                               = \
SELECT id, group_id, UNIX_TIMESTAMP(date_added) AS date_added, title, content \
FROM documents

And the search utility uses sql_query_info to find the records that match. In the query, $id is replaced with each primary key that searchd returns.

sql_query_info          = SELECT * FROM documents WHERE id=$id

Next we need to build an index.

index test1
{
# document source(s) to index
source= src1
# index files path and file name, without extension
# mandatory, path must be writable, extensions will be auto-appended
path                    = /var/test1
}

Here source is src1 and path defines where to store the index data. You have to make sure that this directory exists before generating the index. The searchd section at bottom configures the searchd daemon itself.

searchd
{
port                = 3312
log                    = /var/log/searchd/searchd.log
query_log            = /var/log/searchd/query.log
pid_file            = /var/log/searchd/searchd.pid
}

We are now ready to build the index for the database table. Before running indexer program to generate indices you have to make sure that mysql program is running.

$/usr/local/bin/indexer --config /usr/local/etc/sphinx.conf --all
Sphinx 0.9.9-release (r2117)
Copyright (c) 2001-2009, Andrew Aksyonoff
using config file '/usr/local/etc/sphinx.conf'...
indexing index 'test1'...
collected 6 docs, 0.0 MB
sorted 0.0 Mhits, 100.0% done
total 6 docs, 243 bytes
total 0.018 sec, 12926 bytes/sec, 319.16 docs/sec

The -all argument rebuilds all the indexes listed in sphinx.conf. You can use a different argument to rebuild fewer if you don’t need to rebuild every index.

You can now test the index with the search utility

# /usr/local/bin/search --config /usr/local/etc/sphinx.conf spark
Sphinx 0.9.9-release (r2117)
Copyright (c) 2001-2009, Andrew Aksyonoff

using config file '/usr/local/etc/sphinx.conf'...
index 'test1': query 'spark ': returned 2 matches of 2 total in 0.006 sec

displaying matches:
1. document=5, weight=2, group_id=2, date_added=Sun Feb 13 11:50:30 2011
id=5
group_id=2
group_id2=9
date_added=2011-02-13 11:50:30
title=spark
content=spark support
2. document=6, weight=2, group_id=2, date_added=Sun Feb 13 11:53:30 2011
id=6
group_id=2
group_id2=10
date_added=2011-02-13 11:53:30
title=spark cochin
content=spark support cochin

To query the index from your PHP scripts, you need to run the search daemon which your script will talk to and Include the API (it’s located in api/sphinxapi.php) into your own scripts and use it.

Capistrano is a deployment tool and is popularly used for deploying rails applications to your servers. But its a good configuration tool too. You can setup your servers using capistrano. You can preform tasks like adding users, installing and configuring packages like apache, mysql or postfix, a xen vm etc on your desired network host from a central server(wherever you kept the configuration repository).

Puppet and Capistrano?

Well., you might have heard about puppet – another popular network host configuration tool. Great..Now it will be easy to understand capistrano. If you haven’t, dont worry, I will make it clear for you.

The puppet is working in such a way that a puppet server will be running on one host in the nerwork. And all the desired hosts would run a puppet client daemon. You can configure your client manully and can automate installation on your clients. Whenever you want to change the configuration just do it on the server. It will be updated to the client as the client puppet daemons updates from the server in each interval that you specify. Now you would note that the client daemon pulls the configuration from puppet server. But in capistrano the user pushes the configuration to the desired client when he needs it. Puppet uses “meta language” for the configurations. Whereas capistrano uses ruby. And you know that you got several modules and feautres with ruby and so you may can write your additions to cap if you are well versed with that. One more thing you should note that puppet does its dependency resolution automatically, but in capistrano, you may have to handle the dependency manually in some cases. But as I said earlier you can overcome this if you know well and I would say it may be little difficult to go to that extent. But when you have done with a perfect configuration you feel it greatly helpful.

Installation and Configuration

To install just enter the command:

$gem install capistrano

Provided you have rubygems installed on your machine. If not, install it with apt or yum.

$gem install deprec

Note: I have found several docs without mentioning this..and it made my cap commands haden’t worked till I found this and installed..

Now create a repository as the source (central configuration repo) using svn. Svn is common version control system that is used in cordination with the cap. If you like git go with that.

$svnadmin create sparkconf

Sparkconf is the repository name where you will keep all the configuration of your servers. You can use your own repository name.

Configure this repo for access to the desired users. I wont go with svn details because its beyond the context of our topic.

now you can checkout your repo

$svn co <ur-repo-ulr> sparkconf
$cd sparkconf
$capify .

This “capify” command will initiate your cap repo. It will create a file named Capfile and a deploy.rb file in a directory named config. In capfile you can see that there is a statement to load deploy.rb file. So you can define your own tasks in Capfile or in deploy.rb. But usually this is not the convention. You would leave your Capfile without any changes. And you may specify some information about your setting in the deploy.rb file. The sample deploy.rb which is generated automatically, needs some explanation.

<br />
set :application, &quot;agileblazeworks&quot;<br />
set :repository,  &quot;report&quot;</p>
<p>set :scm, :subversion<br />
# Or: `accurev`, `bzr`, `cvs`, `darcs`, `git`, `mercurial`, `perforce`, `subversion` or `none`</p>
<p>role :web, &quot;rails.spark.com&quot;                                 # Your HTTP server, Apache/etc<br />
role :app, &quot;rails.spark.com&quot;                                 # This may be the same as your `Web` server<br />
role :db,  &quot;rails.spark.com&quot;, :primary = true    # This is where Rails migrations will run<br />
#role :db,  &quot;your slave db-server here&quot;</p>
<p># If you are using Passenger mod_rails uncomment this:<br />
# if you're still using the script/reapear helper you will need<br />
# these http://github.com/rails/irs_process_scripts</p>
<p># namespace :deploy do<br />
#   task :start do ; end<br />
#   task :stop do ; end<br />
#   task :restart, :roles =&amp;gt; :app, :except =&amp;gt; { :no_release =&amp;gt; true } do<br />
#     run &quot;#{try_sudo} touch #{File.join(current_path,'tmp','restart.txt')}&quot;<br />
#   end<br />
# end<br />

Dont worry about the configuration here. Most of them are needed if you are deploying a rails application for a web server. After breifing this we will go for some other configuration options that you are really looking for…to install and configure packages on your hosts.

If your application (means your rails application) is not separated into application, web and database servers, you can either set them to be the same value; or comment out, or remove the one you do not require. The “:primary => true” part of the role definitions allows you to have more than one database server. If you dont have two skipp this primary option. If, for example when deploying a Rails application you only wanted db1 to run migrations, in the first example both might. Essentially when using the Rails deployment recipes, the :primary option defines where database migrations are run. Similar attributes include :no_release often used for the :web role by some of the recipes in circulation to decide which servers should not have the code checked out to them. Attributes like these are arbitrary and you can define some of your own, and use them to filter more precisely where your own tasks run.

You may add these options to the deploy.rb file

<br />
set :user, 'jaseer'              #This is the user you have on the target machine. Capistrano try to login to the target machine using this account.<br />
ssh_options[:keys] = %w(/home/users/mylocalname/.ssh/jas_rsa)<br />
set :use_sudo, true            #if you want to append all commands with sudo.<br />
set :password, &quot;yourpassword&quot;      #You can login using key (above) or with password. Use either key or password..not both.</p>
<p>default_run_options[:pty] = true #this is really helpful. If you dont have this you will struggle in runnig sudo tasks. I had lost some time searching around this.<br />

Some cap commands

$ cap -h

This will give out a list of all the options it accepts.

$ cap -H

It will give you a description of each option.

Next, let’s ask Capistrano what all tasks it will do. Capistrano comes bundled with several built-in tasks. You can also write your own to automate workflows of your own. For now, let’s see what tasks Capistrano knows:

$ cap -T

And finally, to get a detailed description of a command, type

$cap -e <task>

Example

Suppose you want install and configure nagios on one of your servers.

cap deprec:nagios:install HOSTS=monitor.spark.com

This command will install nagios on the host monitor.blocksglobal.com (You can use ip also). Iif you want to override the user in deploy.rb, or any other files, use the option USER=<username> option. This will ask your sudo password, as you know for some task you may need the administrator privileges.

Now generate configuration files

#cap deprec:nagios:config_gen

The configuration files are created in your localrepo not in the installed server. A directry tree is created under sparkconf/config/  with the name nagios. It will contain all configuration files of nagios like hosts.cfg, service.cfg etc. You can change each  according to your needs. Update svn. Push those configuration to your client.

$cd config/nagios

Change and configure the files as your needs, commit the changes and update svn. Go back to the root of your repository, here it is sparkconf. Now push the config to your nagios server.

$cap deprec:nagios:config HOSTS=monitor.spark.com

Defining Tasks

Now let me explain how to define tasks. Often I would define my own tasks in config/*/recipes.rb for example config/nagios/recipes.rb.  * Can be any thing like postfix, mysql as you generate the config directory for them using config_gen.

<br />
namespace : one do<br />
 task :default do<br />
    test<br />
    one.test<br />
    two.test<br />
  end<br />
  task :test do<br />
    puts &quot;Test One Successful!&quot;<br />
  end<br />
end<br />
namespace :two do<br />
  task :test do<br />
    puts &quot;Test Two Successful&quot;<br />
  end<br />
end<br />

Here these are the available commands I can use with cap…

$cap one
$cap one:default
$cap one:test
$cap two

Hope you understood how to define them. Note in how many ways I called the task “test” inside the task default. This is easy right? Now go, create your repo and try your cap tasks..I will stop with a final simple example:

<br />
task :backup_database, :roles =&gt; :db, : only =&gt; { :backup =&gt; true } do<br />
 run &quot;#{sudo} mysqldump ... &gt; /tmp/backup.sql&quot;<br />
 run &quot;#{sudo} bzip2 /tmp/backup.sql&quot;<br />
 run &quot;scp /tmp/backup.sql.bz2 offsite.host:/u/backups&quot;<br />
 run &quot;#{sudo} rm /tmp/backup.sql.bz2&quot;<br />
end<br />

Next time I will write one more article on deploying rails app using cap.. Have fun..bye

In the Xend configuration file /etc/xen/xend-config.sxp, the default directives related to the virtual network are shown below. Notice that the network-bridge and vif-bridge lines are the only ones uncommented. These are the lines needed for bridged networking.

(network-script network-bridge)
(vif-script vif-bridge)
# (network-script network-route)
# (vif-script vif-route)
# (network-script network-nat)
# (vif-script vif-nat)

The network script network-bridge is in the directory /etc/xen/scripts. It sets up the networking environment and uses the bridge-utils tool to set up the software bridge xenbr0. The vif script vif-bridge is also in the same directory as the network script. It sets up the backend vif for the guest domain when a guest domain boots up.

Before creating any guest domains if you check the network interfaces in Dom0 by typing ‘ifconfig’ command, you can see an output like the following.

[root@localhost xen]# ifconfig
eth0   Link encap:Ethernet  HWaddr 48:5B:39:89:E5:43
       inet addr:192.168.1.86  Bcast:192.168.1.255 Mask:255.255.255.0
lo     Link encap:Local Loopback
       inet addr:127.0.0.1  Mask:255.0.0.0
peth0  Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
       inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
vif0.0 Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
       inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
xenbr0 Link encap:Ethernet  HWaddr B2:8B:2C:4C:ED:0E
       UP BROADCAST RUNNING NOARP  MTU:1500 Metric:1

You may be wondering how and when all these network interfaces are created. Here is the sequence of events that happen when you boot up your Xen server and xend starts up (only dom0 is started and no guest domains set to auto start by xend. I only have one physical network interface on my server and it is named eth0).

1. Execute the /etc/xen/scripts/network-bridge script in /etc/xen/xend-config.sxp

2. This will create a new network bridge called xenbr0.

3. Copy the MAC address and IP address from the physical network
interface eth0.

4. Stop the physical network interface eth0.

5. Create a new pair of connected virtual ethernet interfaces—veth0 and vif0.0.

6. Assign the previously copied MAC address and IP address to the virtual  interface   veth0.

7. Rename the physical network interface to peth0 from eth0.

8. Rename the virtual network interface veth0 to eth0.

9. Attach peth0 and vif0.0 to the bridge xenbr0.

10. Bring up the bridge xenbr0, and the network interfaces peth0, eth0, and vif0.0.

For each new domU, Xen creates a new pair of “connected virtual ethernet interfaces”, with one end in domU and the other in dom0. For linux domU’s, the device name it sees is named eth0. The other end of that virtual ethernet interface pair exists within dom0 as interface vif<id#>.0. When a domU is shutdown, the virtual ethernet interfaces for it are deleted.

when a domU starts up, xend (running in dom0) runs the vif-bridge script, which:
- attaches vif<id#>.0 to xenbr0
- vif<id#>.0 is brought up

So you would be able to see additional network interfaces after creating new guest domains.

[root@localhost xen]# ifconfig
eth0   Link encap:Ethernet  HWaddr 48:5B:39:89:E5:43
       inet addr:192.168.1.86  Bcast:192.168.1.255  Mask:255.255.255.0
lo     Link encap:Local Loopback
       inet addr:127.0.0.1  Mask:255.0.0.0
peth0  Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
       inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
vif0.0 Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
       inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
vif1.0 Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
       inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
vif2.0 Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
       inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
xenbr0 Link encap:Ethernet  HWaddr B2:8B:2C:4C:ED:0E
       UP BROADCAST RUNNING NOARP  MTU:1500 Metric:1

Here you can see the newly added interfaces vif1.0 and vif2.0 , which are virtual interfaces for the new guest domains created.

Inside the guest domain, the network interfaces look the same as on a typical machine.

[user@DomU]# ifconfig -a
eth0   Link encap:Ethernet  HWaddr 00:16:3E:5A:32:DA
       inet addr:128.153.144.96 Bcast:128.153.145.255 Mask:255.255.254.0
       inet6 addr: fe80::216:3eff:fe5a:32da/64 Scope:Link
lo     Link encap:Local Loopback
       inet addr:127.0.0.1  Mask:255.0.0.0
       inet6 addr: ::1/128 Scope:Host

In a guest domain, whether the Xen network is in bridging mode or routing mode, the network interface looks the same as it would in a nonvirtualized machine.

Each DomU (guest domain) have a configuration file where we can specify the network settings associated with that domain. It is specified as the following in the corresponding configuration file.

vif = [mac=00:16:3E:02:00:41,bridge=xenbr0' ]

vif: Specifies the virtual network interface configuration for the domain. This is provided in the following format:
vif = [ "key1 = value1", "key2 = value2" ]
The common options used for this configuration directive are
bridge: Specifies the network bridge that will be used for this interface.
mac: Specifies the MAC address for this virtual interface.
If you do not provide a MAC address, it is set to a random MAC address by Xen on boot.The random address is selected from the range of addresses assigned to Xensource by IEEE. At this point you understand how all the interfaces that have been listed  in Dom0 are created
and what they mean. Next we can have a look at the packet flow in bridged networking. For that please refer the following figure.

.
.

Packet arrives at hardware, is handled by dom0 Ethernet driver and appears on peth0. peth0  is bound to to the bridge, so its passed to the bridge from there.We can see that peth0, xenbr0, vif0.0, and vif1.0 are sharing the same MAC address FE:FF:FF:FF:FF:FF, which is the Ethernet broadcast address. This indicates that the physical interface, the loopback device of Domain0, and the backend interfaces of guest domains are all broadcasting to the bridge interface xenbr0. When the physical network interface receives the packets, it sends them all directly to the bridge interface xenbr0. Now the bridge distributes the packet, just like a switch would.The software bridge determines which domain’s backend interface to forward those packets to by the receiver’s MAC addresses. So, peth0 does not need to have an IP, only a MAC address. The vif interface puts the packet into Xen, which then puts the packet back to the domain the vif leads to.Its also done that way for dom0, hence the vif0.0->eth0 pair.

If you want to use a bridge name other than the default one( xenbr0) you can specify it in the xend configuration file like the following.

(network-script 'network-bridge bridge=<name>')

where <name> is the desired name of bridge.

Note also that network-bridge defaults to binding eth0 to the bridge. To change the physical network card, use

(network-script 'network-bridge bridge=<name> netdev=eth1')

If a server has more than one physical network interface, it may be useful to separate traffic between interfaces by creating additional bridges. In this way, one could connect xenbr0 to peth0 and xenbr1 to peth1 and give a virtual machine in say dom1 exclusive access to xenbr1. Everything necessary for this is present in the network-bridge script.

While useful from the command line, the /etc/xen/xend-config.sxp script will require some tuning in order to create bridges automatically during boot. By default, this script calls the network-bridge script; however, this script can only be called once. To avoid this problem, you need to create a network-wrapper script, which can be configured to call the network-bridge script twice. Use the following steps to configure this:

1. Create a script /etc/xen/scripts/network-wrapper with the following contents:

#!/bin/sh
/etc/xen/scripts/network-bridge $1 netdev=eth0 bridge=xenbr0
/etc/xen/scripts/network-bridge $1 netdev=eth1 bridge=xenbr1

Tune the /etc/xen/xend-config.sxp script so that it calls this network wrapper script, by adding the following:

(network-script network-bridge-wrapper)

3. Make sure that in the configuration file for each of the unprivileged domains, you indicate what network bridge to use. This would make the vif lines look like the following example line:

vif=[ 'bridge=xenbr1', 'mac=00:16:3e:07:d2:0e', ]

From ipfwadm to iptables , we can see marginal modifications done in design and hence in features. In ipfwadm a rule set matched the packets and actions are taken on that packets , then in ipchains , chains are Incorporated and the rules will decide , through which chain the packets need to be traversed, then in iptables , tables are introduced so that the user can choose proper table based on his requirement , that means he can choose filter table for packet filtering . NAT tables for source and destination natting then mangle table for packet modifications.(We will discuss more about tables later ). Connection tracking capability is the yet another advancement with iptable. So we can say it as a statefull firewall. Rate limiting , system logging etc. are the other features included in iptables apart from it’s predecessors.

Iptables gets it’s ascendancy through the modularity and integrity with the kernel. It exploits the modularity of linux kernels. A bunch of kernel modules need to be loaded for the full operation of iptables.

ip_tables iptable_filter ip_conntrack ip_conntrack_ftp
iptable_nat ip_nat_ftp ipt_limit ipt_multiport
iptable_mangle ipt_state ipt_REJECT ipt_LOG

These are some of required kernel modules. All are not required for all applications. For loading modules in bootup add script in rc.local using modprobe command. And also we need a kernel with some specific config options .If it’s not there we need to recompile the kernel to exploit the functions of iptable. For latest kernels almost all options are included , rarely need to be recompiled. Some important options that need to be compiled in to kernel or as modules are

CONFIG_PACKET
CONFIG_NETFILTER
CONFIG_IP_NF_CONNTRACK
CONFIG_IP_NF_FTP
CONFIG_IP_NF_IRC
ONFIG_IP_NF_IPTABLES
CONFIG_IP_NF_FILTER
CONFIG_IP_NF_NAT
CONFIG_IP_NF_MATCH_STATE
CONFIG_IP_NF_TARGET_LOG
CONFIG_IP_NF_MATCH_LIMIT
CONFIG_IP_NF_TARGET_MASQUERADE .

You can check your current kernel configuration by cat /boot/config-`uname -r`.As I already told the latest kernels are incorporated with almost all netfilter options. So you seldom need to worry about that.

Packet flow in iptables
Next I would like to draw a vignette of packet flaw in iptables. There is definite order for traversing of packets through the different tables and chains in iptables. Basic understanding of that order is mandatory for writing complex firewall rules.

Iptable comprised of three tables , MANGLE , FILTER and NAT table.
Each tables have it’s own chains for packet processing.
MANGLE table : This tables is responsible for the alternation of QoS bits in TCP header.For example TOS (Type of service) bit. But it rarely used for simple networks.Mangle table have five chains ,PREROUTING, POSTROUTING,FORWARD,INPUT and OUTPUT.

FILTER table:This is the most used table in small network environments.This table is responsible for packet filtering on the basis of corresponding rules. This table have INPUT, FORWARD and OUTPUT chains. Most used target and jumps in this table are ACCEPT, DROP/DENY and REJECT. We will discuss about this target and jump later.

NAT table:This table used for network address translation which is a imperative feature of routers.It have PREROUTING chain (for Destination address translation) FORWARD chain (address translation for packets from router itself) and POSTROUTING chain( for source address translation).

When packet reaches the hardware(NIC) it will be processed to hardware driver and through kernel.Then it traverse through different tables and chains in the iptables in desired order. In each chain , iptable compare the rules in that chain with that packet information , if it finds a match then action is taken based on that rule.(eg:REJECT) .If a packet is matched with the first rule in a chain , it will not check the second rule in the same chain , it will jump to next chain in the order.

The following figure will help you to understand the packet traversal through iptable chains.

Options with iptable command.

-t Specify the table. By default FILTER table is taken .
-j Jump to the target
-A Append the rule to the end of the chain.
-F Flush. Deletes all the rules in the selected table
-p Match protocol. Types include, icmp, tcp, udp, and all
-s Match source IP address
-d Match destination IP address
-i Match “input” interface on which the packet enters.
-o Match “output” interface on which the packet exits
-m used to match certain conditions . eg: state ESTABLISHED

Targets and Jumps with iptable
DROP/DENY : When a packet is dropped or denied it is simply  abandoned and no notification is send to the host. ie the packet disappears with out taking any further actions.

REJECT : The host reject the packet and sends reply to the sending host , saying that the packet was dropped. Other wise it is same as that of DROP/DENY.

ACCEPT : This will accept the packet that matches the corresponding rules.

LOG : The packet information will be logged with syslogd daemon.

SNAT : Source nat used in NAT table POSTROUTING table for manipulating source ip address.

DNAT: destination NAT used in NAT table PREROUTING table to manipulate destination ip address and for port forwarding.

MASQUERADE : To change source ip address to the ip address of corresponding router network interface.

Iptable commands and rules

iptables -L : To list all present rules in a table (By default it will show FILTER table rule.To specify the table use -t )
iptables -t nat -L (list all NAT table rules)
iptables -v -L : more detailed information about the rules
iptables -F

iptables -A INPUT -p tcp -s 0/0 --dport 22 -j ACCEPT <This will allow port 22 on INPUT chain of filter table)
iptables-A OUTPUT -p tcp -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -P INPUT DROP

Rule for port forwarding

 iptables -t nat -A PREROUTING -p tcp -d  'destination ip' --dport 80 -j DNAT --to-destination 192.168.1.10:80
Packet destined to 80 port of your public ip will be forwarded to the 80 port of private ip.We appending rule to the PREROUTING chain of NAT table.
iptables -A FORWARD -p tcp -s 0/0 -d 192.168.1.10 --dport 80 -j ACCEPT

I think this is enough for the basics. I will write more advanced rules on my next article. Try it!!!!!

The first part of the discussion concentrated mainly on static page scraping with Perl mechanize module. Even though mechanize provides extension for dynamic scraping, it is not very good.

So this session deals with making use of selenium testing tool for Web Scraping.

Prerequsites

Selenium IDE is a Firefox add-on that records clicks, typing, and other actions to make a test, which you can play back in the browser.

• Download Selenium IDE

Selenium Remote Control (RC)  is a Java based Command line server for handling request from client.

• Download Selenium Remote Control (RC)
  

Pros and Cons

It supports all Dynamic Content like Ajax, JavaScript, is easy to implement and it is possible to write selenium clients in any language we prefer, for example, here I have used Perl. You can also use Python, Java, etc.

Selenium based Web Scraping on small throughout is easy task.

It consumes lots of memory resource, for each request it will launch a new browser instance.

Working of selenium

Selenium Remote Control (RC) is a test tool that allows you to write automated web application UI tests in any programming language against any HTTP website using any mainstream JavaScript-enabled browser.

Selenium RC comes in two parts.

1. A server which automatically launches and kills browsers, and acts as a HTTP proxy for web requests from them.
2. Client libraries for your favourite computer language.

The RC server also bundles Selenium Core, and automatically loads it into the browser.

Here is a simplified architectural representation:

For Detailed diagram http://seleniumhq.org/about/how.html

How to Setup a Selenium Server

Download Selenium RC server to directory to /usr/local/selenium

#cd /usr/local/selenium

#unzip selenium-remote-control-1.0-beta-2-dist.zip

#cd selenium-remote-control-1.0-beta-2

#cd selenium-server-1.0-beta-2

#java -jar selenium-server.jar #starting selenium server .By default it is listen to 4444

An example Client Program

As said in the above section, it is possible to create selenium client by recording user activities or else the programmers can create it using their own language. Python, Perl and Ruby, Java has supporting modules for it.

#Sample Perl Code
#!/usr/bin/perl
use strict;
use warnings;
use Time::HiRes qw(sleep);
use Test::WWW::Selenium;
use Test::More "no_plan";
use Test::Exception;

my $sel = Test::WWW::Selenium->new( host => "192.168.1.20",
port => 4444,
browser => "*firefox",
browser_url => "http://www.godaddy.com/" );
$sel->open_ok("/domains/search.aspx?ci=8969");
$sel->click_ok("domain_search_button");
$sel->wait_for_page_to_load_ok("30000");
my $data=$sel->get_html_source(); # here you get source of the current page

For more info please have a look at cpan http://search.cpan.org/search?query=selenium&mode=all

As scraper you can extract required data from this source:

For scraping data from multiple pages

Open selenium IDE and record the events that you are interested and analyse the code generated and try to implement your own way,

As a last word, let me add that selenium is not completely a scraping tool, it is instead, a testing tool.

For more about selenium have look at http://seleniumhq.org/

If you don’t have Git on your machine install it through yum(Redhat based) or apt-get(Debian based) depending up on your OS.

# apt-get install git-core

For the installation of gitosis we need python-setuptools package in our server.

# apt-get install python-setuptools

Now we can download gitosiis from it’s git repository.

# git clone git://eagain.net/gitosis.git

it’s of few KBs .Now install it with python script.

# cd gitosis
# python setup.py install

create an user as git(you can give any name) . Give him a shell otherwise ssh login might have problems ,then give the desired home directory.(which will be the Git Root directory).

It’s the time to upload public key to Git server. If you have a public-private key pair in local machineuse it , otherwise create a new one with ssk-keygen.

# ssh-keygen -t rsa

Public key will be in name id_rsa.pub . Copy the key and upload it to the server.

Next , execute the command

# sudo -H -u git gitosis-init < /tmp/id_rsa.pub

This will add some directories and files in to user git’s home directory and append the public key to authorized keys file of user git.
Now clone the gitosis-admin repository to your local machine.

$ git clone git@GIT-SERVER:gitosis-admin.git
$ cd gitosis-admin

If you list this directory you can see only two directories .Here You can create new projects , make changes , do access control etc. What ever you do , you have to commit it and push it to the server to replicate the changes on server. Since we cloned gitosis-admin repository we can make all changes locally and push it to the server.

Creating new repository and control access on new repository.

For creating a new repository , edit the gitosis.conf file and add the entries as follows.
Here I am assuming that the new repository name is ‘test’

[group testgroup]
members = hans
writable = test

Here group is just a name , no matter what it is. member is the list of users who can access the repository and ‘writable’ is the name of repository. Commit this changes and push it to the server.

$ git commit -a -m "comment for this commit"
$ git push

Create the directory ‘test’ and initiate it as a git repo.

$ mkdir test
$ cd test
$ git init

Add remote server url to the new repository config file.

$ git remote add origin git@GIT_SERVER:test.git

push the changes in to the server.

$ git push origin master:refs/heads/master

Eventhough we allowed user hans to access ‘test’ repository , we didn’t add his public key to the server . So he will not be able to access the project repository. So copy his public key to ‘keydir’ as hans.pub . It should have a ‘.pub’ extension.

$ cd gitosis-admin
$ cp /home/hans/hans.pub keydir/hans.pub
$ git add keydir/hans.pub
$ git commit -a -m "comment"
$ git push

Yes we done it !
Now user hans can clone the test repository to his local machine. He can make changes ,add files ,commit ,and push changes to the server.

hans $ git clone git@GIT_SERVER:test.git
Now try it. Good Luck !