Deploying with Capistrano

December 21st, 2010

What is Capisrano?

Capistrano is a deployment tool and is popularly used for deploying rails applications to your servers. But its a good configuration tool too. You can setup your servers using capistrano. You can preform tasks like adding users, installing and configuring packages like apache, mysql or postfix, a xen vm etc on your desired network host from a central server(wherever you kept the configuration repository).

Puppet and Capistrano?

Well., you might have heard about puppet – another popular network host configuration tool. Great..Now it will be easy to understand capistrano. If you haven’t, dont worry, I will make it clear for you.

The puppet is working in such a way that a puppet server will be running on one host in the nerwork. And all the desired hosts would run a puppet client daemon. You can configure your client manully and can automate installation on your clients. Whenever you want to change the configuration just do it on the server. It will be updated to the client as the client puppet daemons updates from the server in each interval that you specify. Now you would note that the client daemon pulls the configuration from puppet server. But in capistrano the user pushes the configuration to the desired client when he needs it. Puppet uses “meta language” for the configurations. Whereas capistrano uses ruby. And you know that you got several modules and feautres with ruby and so you may can write your additions to cap if you are well versed with that. One more thing you should note that puppet does its dependency resolution automatically, but in capistrano, you may have to handle the dependency manually in some cases. But as I said earlier you can overcome this if you know well and I would say it may be little difficult to go to that extent. But when you have done with a perfect configuration you feel it greatly helpful.

Installation and Configuration

To install just enter the command:

$gem install capistrano

Provided you have rubygems installed on your machine. If not, install it with apt or yum.

$gem install deprec

Note: I have found several docs without mentioning this..and it made my cap commands haden’t worked till I found this and installed..

Now create a repository as the source (central configuration repo) using svn. Svn is common version control system that is used in cordination with the cap. If you like git go with that.

$svnadmin create sparkconf

Sparkconf is the repository name where you will keep all the configuration of your servers. You can use your own repository name.

Configure this repo for access to the desired users. I wont go with svn details because its beyond the context of our topic.

now you can checkout your repo

$svn co <ur-repo-ulr> sparkconf
$cd sparkconf
$capify .

This “capify” command will initiate your cap repo. It will create a file named Capfile and a deploy.rb file in a directory named config. In capfile you can see that there is a statement to load deploy.rb file. So you can define your own tasks in Capfile or in deploy.rb. But usually this is not the convention. You would leave your Capfile without any changes. And you may specify some information about your setting in the deploy.rb file. The sample deploy.rb which is generated automatically, needs some explanation.

<br />
set :application, &quot;agileblazeworks&quot;<br />
set :repository,  &quot;report&quot;</p>
<p>set :scm, :subversion<br />
# Or: `accurev`, `bzr`, `cvs`, `darcs`, `git`, `mercurial`, `perforce`, `subversion` or `none`</p>
<p>role :web, &quot;rails.spark.com&quot;                                 # Your HTTP server, Apache/etc<br />
role :app, &quot;rails.spark.com&quot;                                 # This may be the same as your `Web` server<br />
role :db,  &quot;rails.spark.com&quot;, :primary = true    # This is where Rails migrations will run<br />
#role :db,  &quot;your slave db-server here&quot;</p>
<p># If you are using Passenger mod_rails uncomment this:<br />
# if you're still using the script/reapear helper you will need<br />
# these http://github.com/rails/irs_process_scripts</p>
<p># namespace :deploy do<br />
#   task :start do ; end<br />
#   task :stop do ; end<br />
#   task :restart, :roles =&amp;gt; :app, :except =&amp;gt; { :no_release =&amp;gt; true } do<br />
#     run &quot;#{try_sudo} touch #{File.join(current_path,'tmp','restart.txt')}&quot;<br />
#   end<br />
# end<br />

Dont worry about the configuration here. Most of them are needed if you are deploying a rails application for a web server. After breifing this we will go for some other configuration options that you are really looking for…to install and configure packages on your hosts.

If your application (means your rails application) is not separated into application, web and database servers, you can either set them to be the same value; or comment out, or remove the one you do not require. The “:primary => true” part of the role definitions allows you to have more than one database server. If you dont have two skipp this primary option. If, for example when deploying a Rails application you only wanted db1 to run migrations, in the first example both might. Essentially when using the Rails deployment recipes, the :primary option defines where database migrations are run. Similar attributes include :no_release often used for the :web role by some of the recipes in circulation to decide which servers should not have the code checked out to them. Attributes like these are arbitrary and you can define some of your own, and use them to filter more precisely where your own tasks run.

You may add these options to the deploy.rb file

<br />
set :user, 'jaseer'              #This is the user you have on the target machine. Capistrano try to login to the target machine using this account.<br />
ssh_options[:keys] = %w(/home/users/mylocalname/.ssh/jas_rsa)<br />
set :use_sudo, true            #if you want to append all commands with sudo.<br />
set :password, &quot;yourpassword&quot;      #You can login using key (above) or with password. Use either key or password..not both.</p>
<p>default_run_options[:pty] = true #this is really helpful. If you dont have this you will struggle in runnig sudo tasks. I had lost some time searching around this.<br />

Some cap commands

$ cap -h

This will give out a list of all the options it accepts.

$ cap -H

It will give you a description of each option.

Next, let’s ask Capistrano what all tasks it will do. Capistrano comes bundled with several built-in tasks. You can also write your own to automate workflows of your own. For now, let’s see what tasks Capistrano knows:

$ cap -T

And finally, to get a detailed description of a command, type

$cap -e <task>

Example

Suppose you want install and configure nagios on one of your servers.

cap deprec:nagios:install HOSTS=monitor.spark.com

This command will install nagios on the host monitor.blocksglobal.com (You can use ip also). Iif you want to override the user in deploy.rb, or any other files, use the option USER=<username> option. This will ask your sudo password, as you know for some task you may need the administrator privileges.

Now generate configuration files

#cap deprec:nagios:config_gen

The configuration files are created in your localrepo not in the installed server. A directry tree is created under sparkconf/config/  with the name nagios. It will contain all configuration files of nagios like hosts.cfg, service.cfg etc. You can change each  according to your needs. Update svn. Push those configuration to your client.

$cd config/nagios

Change and configure the files as your needs, commit the changes and update svn. Go back to the root of your repository, here it is sparkconf. Now push the config to your nagios server.

$cap deprec:nagios:config HOSTS=monitor.spark.com

Defining Tasks

Now let me explain how to define tasks. Often I would define my own tasks in config/*/recipes.rb for example config/nagios/recipes.rb.  * Can be any thing like postfix, mysql as you generate the config directory for them using config_gen.

<br />
namespace : one do<br />
 task :default do<br />
    test<br />
    one.test<br />
    two.test<br />
  end<br />
  task :test do<br />
    puts &quot;Test One Successful!&quot;<br />
  end<br />
end<br />
namespace :two do<br />
  task :test do<br />
    puts &quot;Test Two Successful&quot;<br />
  end<br />
end<br />

Here these are the available commands I can use with cap…

$cap one
$cap one:default
$cap one:test
$cap two

Hope you understood how to define them. Note in how many ways I called the task “test” inside the task default. This is easy right? Now go, create your repo and try your cap tasks..I will stop with a final simple example:

<br />
task :backup_database, :roles =&gt; :db, : only =&gt; { :backup =&gt; true } do<br />
 run &quot;#{sudo} mysqldump ... &gt; /tmp/backup.sql&quot;<br />
 run &quot;#{sudo} bzip2 /tmp/backup.sql&quot;<br />
 run &quot;scp /tmp/backup.sql.bz2 offsite.host:/u/backups&quot;<br />
 run &quot;#{sudo} rm /tmp/backup.sql.bz2&quot;<br />
end<br />

Next time I will write one more article on deploying rails app using cap.. Have fun..bye

VN:F [1.9.6_1107]
Rating: 8.0/10 (1 vote cast)
VN:F [1.9.6_1107]
Rating: 0 (from 0 votes)

Jaseer Articles, linux , , , ,

Xen Bridge Networking

December 19th, 2010

Xen provides couple of different networking options like bridge networking, nat networking and route networking. Bridging is a technique used for connecting different network segments. It is the default option for Xen networking as it simply connects all the virtual machines to the outside world through virtual network interfaces connected to the bridges created by Xen. When using bridged networking, Xen creates a network bridge and then connects the actual physical network interface to this bridge. Virtual network interfaces are then created for dom0 and each of the guest domains; these are all attached to the network bridge. In this manner, all the domains can connect to any address to which the physical network interface can connect.

In the Xend configuration file /etc/xen/xend-config.sxp, the default directives related to the virtual network are shown below. Notice that the network-bridge and vif-bridge lines are the only ones uncommented. These are the lines needed for bridged networking.


(network-script network-bridge)
(vif-script vif-bridge)
# (network-script network-route)
# (vif-script vif-route)
# (network-script network-nat)
# (vif-script vif-nat)

The network script network-bridge is in the directory /etc/xen/scripts. It sets up the networking environment and uses the bridge-utils tool to set up the software bridge xenbr0. The vif script vif-bridge is also in the same directory as the network script. It sets up the backend vif for the guest domain when a guest domain boots up.

Before creating any guest domains if you check the network interfaces in Dom0 by typing ‘ifconfig’ command, you can see an output like the following.

[root@localhost xen]# ifconfig
eth0   Link encap:Ethernet  HWaddr 48:5B:39:89:E5:43
       inet addr:192.168.1.86  Bcast:192.168.1.255 Mask:255.255.255.0
lo     Link encap:Local Loopback
       inet addr:127.0.0.1  Mask:255.0.0.0
peth0  Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
       inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
vif0.0 Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
       inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
xenbr0 Link encap:Ethernet  HWaddr B2:8B:2C:4C:ED:0E
       UP BROADCAST RUNNING NOARP  MTU:1500 Metric:1

You may be wondering how and when all these network interfaces are created. Here is the sequence of events that happen when you boot up your Xen server and xend starts up (only dom0 is started and no guest domains set to auto start by xend. I only have one physical network interface on my server and it is named eth0).

1. Execute the /etc/xen/scripts/network-bridge script in /etc/xen/xend-config.sxp

2. This will create a new network bridge called xenbr0.

3. Copy the MAC address and IP address from the physical network
interface eth0.

4. Stop the physical network interface eth0.

5. Create a new pair of connected virtual ethernet interfaces—veth0 and vif0.0.

6. Assign the previously copied MAC address and IP address to the virtual  interface   veth0.

7. Rename the physical network interface to peth0 from eth0.

8. Rename the virtual network interface veth0 to eth0.

9. Attach peth0 and vif0.0 to the bridge xenbr0.

10. Bring up the bridge xenbr0, and the network interfaces peth0, eth0, and vif0.0.

For each new domU, Xen creates a new pair of “connected virtual ethernet interfaces”, with one end in domU and the other in dom0. For linux domU’s, the device name it sees is named eth0. The other end of that virtual ethernet interface pair exists within dom0 as interface vif<id#>.0. When a domU is shutdown, the virtual ethernet interfaces for it are deleted.

when a domU starts up, xend (running in dom0) runs the vif-bridge script, which:
- attaches vif<id#>.0 to xenbr0
- vif<id#>.0 is brought up

So you would be able to see additional network interfaces after creating new guest domains.

[root@localhost xen]# ifconfig
eth0   Link encap:Ethernet  HWaddr 48:5B:39:89:E5:43
       inet addr:192.168.1.86  Bcast:192.168.1.255  Mask:255.255.255.0
lo     Link encap:Local Loopback
       inet addr:127.0.0.1  Mask:255.0.0.0
peth0  Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
       inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
vif0.0 Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
       inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
vif1.0 Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
       inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
vif2.0 Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
       inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
xenbr0 Link encap:Ethernet  HWaddr B2:8B:2C:4C:ED:0E
       UP BROADCAST RUNNING NOARP  MTU:1500 Metric:1

Here you can see the newly added interfaces vif1.0 and vif2.0 , which are virtual interfaces for the new guest domains created.

Inside the guest domain, the network interfaces look the same as on a typical machine.

[user@DomU]# ifconfig -a
eth0   Link encap:Ethernet  HWaddr 00:16:3E:5A:32:DA
       inet addr:128.153.144.96 Bcast:128.153.145.255 Mask:255.255.254.0
       inet6 addr: fe80::216:3eff:fe5a:32da/64 Scope:Link
lo     Link encap:Local Loopback
       inet addr:127.0.0.1  Mask:255.0.0.0
       inet6 addr: ::1/128 Scope:Host

In a guest domain, whether the Xen network is in bridging mode or routing mode, the network interface looks the same as it would in a nonvirtualized machine.

Each DomU (guest domain) have a configuration file where we can specify the network settings associated with that domain. It is specified as the following in the corresponding configuration file.


vif = [mac=00:16:3E:02:00:41,bridge=xenbr0' ]

vif: Specifies the virtual network interface configuration for the domain. This is provided in the following format:
vif = [ "key1 = value1", "key2 = value2" ]
The common options used for this configuration directive are
bridge: Specifies the network bridge that will be used for this interface.
mac: Specifies the MAC address for this virtual interface.
If you do not provide a MAC address, it is set to a random MAC address by Xen on boot.The random address is selected from the range of addresses assigned to Xensource by IEEE. At this point you understand how all the interfaces that have been listed  in Dom0 are created
and what they mean. Next we can have a look at the packet flow in bridged networking. For that please refer the following figure.

.
.

Packet arrives at hardware, is handled by dom0 Ethernet driver and appears on peth0. peth0  is bound to to the bridge, so its passed to the bridge from there.We can see that peth0, xenbr0, vif0.0, and vif1.0 are sharing the same MAC address FE:FF:FF:FF:FF:FF, which is the Ethernet broadcast address. This indicates that the physical interface, the loopback device of Domain0, and the backend interfaces of guest domains are all broadcasting to the bridge interface xenbr0. When the physical network interface receives the packets, it sends them all directly to the bridge interface xenbr0. Now the bridge distributes the packet, just like a switch would.The software bridge determines which domain’s backend interface to forward those packets to by the receiver’s MAC addresses. So, peth0 does not need to have an IP, only a MAC address. The vif interface puts the packet into Xen, which then puts the packet back to the domain the vif leads to.Its also done that way for dom0, hence the vif0.0->eth0 pair.

If you want to use a bridge name other than the default one( xenbr0) you can specify it in the xend configuration file like the following.


(network-script 'network-bridge bridge=<name>')

where <name> is the desired name of bridge.

Note also that network-bridge defaults to binding eth0 to the bridge. To change the physical network card, use


(network-script 'network-bridge bridge=<name> netdev=eth1')

If a server has more than one physical network interface, it may be useful to separate traffic between interfaces by creating additional bridges. In this way, one could connect xenbr0 to peth0 and xenbr1 to peth1 and give a virtual machine in say dom1 exclusive access to xenbr1. Everything necessary for this is present in the network-bridge script.

While useful from the command line, the /etc/xen/xend-config.sxp script will require some tuning in order to create bridges automatically during boot. By default, this script calls the network-bridge script; however, this script can only be called once. To avoid this problem, you need to create a network-wrapper script, which can be configured to call the network-bridge script twice. Use the following steps to configure this:

1. Create a script /etc/xen/scripts/network-wrapper with the following contents:


#!/bin/sh
/etc/xen/scripts/network-bridge $1 netdev=eth0 bridge=xenbr0
/etc/xen/scripts/network-bridge $1 netdev=eth1 bridge=xenbr1

Tune the /etc/xen/xend-config.sxp script so that it calls this network wrapper script, by adding the following:


(network-script network-bridge-wrapper)

3. Make sure that in the configuration file for each of the unprivileged domains, you indicate what network bridge to use. This would make the vif lines look like the following example line:


vif=[ 'bridge=xenbr1', 'mac=00:16:3e:07:d2:0e', ]
VN:F [1.9.6_1107]
Rating: 10.0/10 (3 votes cast)
VN:F [1.9.6_1107]
Rating: +2 (from 2 votes)

Tino Thomas linux , , , , , , , , ,

Basics of iptables

December 17th, 2010

Iptables is the default firewall used in many present Linux distros. It is a project under netfilter. Previously it was known as ipchains (For 2.2 Kernels) , ipfwadm(for 2.0 kernels) and nftables are their future project..Iptables are designed for ipv4 and ipv6table for ipv6.

From ipfwadm to iptables , we can see marginal modifications done in design and hence in features. In ipfwadm a rule set matched the packets and actions are taken on that packets , then in ipchains , chains are Incorporated and the rules will decide , through which chain the packets need to be traversed, then in iptables , tables are introduced so that the user can choose proper table based on his requirement , that means he can choose filter table for packet filtering . NAT tables for source and destination natting then mangle table for packet modifications.(We will discuss more about tables later ). Connection tracking capability is the yet another advancement with iptable. So we can say it as a statefull firewall. Rate limiting , system logging etc. are the other features included in iptables apart from it’s predecessors.

Iptables gets it’s ascendancy through the modularity and integrity with the kernel. It exploits the modularity of linux kernels. A bunch of kernel modules need to be loaded for the full operation of iptables.

ip_tables iptable_filter ip_conntrack ip_conntrack_ftp
iptable_nat ip_nat_ftp ipt_limit ipt_multiport
iptable_mangle ipt_state ipt_REJECT ipt_LOG

These are some of required kernel modules. All are not required for all applications. For loading modules in bootup add script in rc.local using modprobe command. And also we need a kernel with some specific config options .If it’s not there we need to recompile the kernel to exploit the functions of iptable. For latest kernels almost all options are included , rarely need to be recompiled. Some important options that need to be compiled in to kernel or as modules are

CONFIG_PACKET
CONFIG_NETFILTER
CONFIG_IP_NF_CONNTRACK
CONFIG_IP_NF_FTP
CONFIG_IP_NF_IRC
ONFIG_IP_NF_IPTABLES
CONFIG_IP_NF_FILTER
CONFIG_IP_NF_NAT
CONFIG_IP_NF_MATCH_STATE
CONFIG_IP_NF_TARGET_LOG
CONFIG_IP_NF_MATCH_LIMIT
CONFIG_IP_NF_TARGET_MASQUERADE .

You can check your current kernel configuration by cat /boot/config-`uname -r`.As I already told the latest kernels are incorporated with almost all netfilter options. So you seldom need to worry about that.

Packet flow in iptables

Next I would like to draw a vignette of packet flaw in iptables. There is definite order for traversing of packets through the different tables and chains in iptables. Basic understanding of that order is mandatory for writing complex firewall rules.

Iptable comprised of three tables , MANGLE , FILTER and NAT table.
Each tables have it’s own chains for packet processing.
MANGLE table : This tables is responsible for the alternation of QoS bits in TCP header.For example TOS (Type of service) bit. But it rarely used for simple networks.Mangle table have five chains ,PREROUTING, POSTROUTING,FORWARD,INPUT and OUTPUT.

FILTER table:This is the most used table in small network environments.This table is responsible for packet filtering on the basis of corresponding rules. This table have INPUT, FORWARD and OUTPUT chains. Most used target and jumps in this table are ACCEPT, DROP/DENY and REJECT. We will discuss about this target and jump later.

NAT table:This table used for network address translation which is a imperative feature of routers.It have PREROUTING chain (for Destination address translation) FORWARD chain (address translation for packets from router itself) and POSTROUTING chain( for source address translation).

When packet reaches the hardware(NIC) it will be processed to hardware driver and through kernel.Then it traverse through different tables and chains in the iptables in desired order. In each chain , iptable compare the rules in that chain with that packet information , if it finds a match then action is taken based on that rule.(eg:REJECT) .If a packet is matched with the first rule in a chain , it will not check the second rule in the same chain , it will jump to next chain in the order.

The following figure will help you to understand the packet traversal through iptable chains.

Options with iptable command.

-t Specify the table. By default FILTER table is taken .
-j Jump to the target
-A Append the rule to the end of the chain.
-F Flush. Deletes all the rules in the selected table
-p Match protocol. Types include, icmp, tcp, udp, and all
-s Match source IP address
-d Match destination IP address
-i Match “input” interface on which the packet enters.
-o Match “output” interface on which the packet exits
-m used to match certain conditions . eg: state ESTABLISHED

Targets and Jumps with iptable

DROP/DENY : When a packet is dropped or denied it is simply  abandoned and no notification is send to the host. ie the packet disappears with out taking any further actions.

REJECT : The host reject the packet and sends reply to the sending host , saying that the packet was dropped. Other wise it is same as that of DROP/DENY.

ACCEPT : This will accept the packet that matches the corresponding rules.

LOG : The packet information will be logged with syslogd daemon.

SNAT : Source nat used in NAT table POSTROUTING table for manipulating source ip address.

DNAT: destination NAT used in NAT table PREROUTING table to manipulate destination ip address and for port forwarding.

MASQUERADE : To change source ip address to the ip address of corresponding router network interface.

Iptable commands and rules

iptables -L : To list all present rules in a table (By default it will show FILTER table rule.To specify the table use -t )
iptables -t nat -L (list all NAT table rules)
iptables -v -L : more detailed information about the rules
iptables -F

: it will flush all rules in that table.
iptables -t table -D : to delete a particular rule from a tables’s chain. Or else we can give the rule number in that chain . eg: iptable -t nat -D PREROUTING 2
iptables -P : To set the default policy of a chain in FILTER table . eg: iptables -p OUPUT DROP
Rule to allow only SSH traffic


iptables -A INPUT -p tcp -s 0/0 --dport 22 -j ACCEPT <This will allow port 22 on INPUT chain of filter table)
iptables-A OUTPUT -p tcp -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -P INPUT DROP

Rule for port forwarding


 iptables -t nat -A PREROUTING -p tcp -d  'destination ip' --dport 80 -j DNAT --to-destination 192.168.1.10:80
Packet destined to 80 port of your public ip will be forwarded to the 80 port of private ip.We appending rule to the PREROUTING chain of NAT table.
iptables -A FORWARD -p tcp -s 0/0 -d 192.168.1.10 --dport 80 -j ACCEPT

I think this is enough for the basics. I will write more advanced rules on my next article. Try it!!!!!

VN:F [1.9.6_1107]
Rating: 5.5/10 (2 votes cast)
VN:F [1.9.6_1107]
Rating: +1 (from 1 vote)

hans linux , ,

Dynamic Web Scraping Using Selenium

November 24th, 2010

This article is a part of the on-going Web Scraping Series. If you are not familiar with Web Scraping please check with the first article . This session mainly deals with Dynamic Content Scraping. Nowadays most of the web portals are dynamic by making Ajax calls instead of old static web pages. Scraping on dynamic environment is both interesting and challenging one.

The first part of the discussion concentrated mainly on static page scraping with Perl mechanize module. Even though mechanize provides extension for dynamic scraping, it is not very good.

So this session deals with making use of selenium testing tool for Web Scraping.

Prerequsites

Selenium IDE is a Firefox add-on that records clicks, typing, and other actions to make a test, which you can play back in the browser.

Selenium Remote Control (RC)  is a Java based Command line server for handling request from client.

Pros and Cons

It supports all Dynamic Content like Ajax, JavaScript, is easy to implement and it is possible to write selenium clients in any language we prefer, for example, here I have used Perl. You can also use Python, Java, etc.

Selenium based Web Scraping on small throughout is easy task.

It consumes lots of memory resource, for each request it will launch a new browser instance.

Working of selenium

Selenium Remote Control (RC) is a test tool that allows you to write automated web application UI tests in any programming language against any HTTP website using any mainstream JavaScript-enabled browser.

Selenium RC comes in two parts.

  1. A server which automatically launches and kills browsers, and acts as a HTTP proxy for web requests from them.
  2. Client libraries for your favourite computer language.

The RC server also bundles Selenium Core, and automatically loads it into the browser.

Here is a simplified architectural representation:

For Detailed diagram http://seleniumhq.org/about/how.html

How to Setup a Selenium Server

Download Selenium RC server to directory to /usr/local/selenium

#cd /usr/local/selenium

#unzip selenium-remote-control-1.0-beta-2-dist.zip

#cd selenium-remote-control-1.0-beta-2

#cd selenium-server-1.0-beta-2

#java -jar selenium-server.jar #starting selenium server .By default it is listen to 4444

An example Client Program

As said in the above section, it is possible to create selenium client by recording user activities or else the programmers can create it using their own language. Python, Perl and Ruby, Java has supporting modules for it.

#Sample Perl Code
#!/usr/bin/perl
use strict;
use warnings;
use Time::HiRes qw(sleep);
use Test::WWW::Selenium;
use Test::More "no_plan";
use Test::Exception;

my $sel = Test::WWW::Selenium->new( host => "192.168.1.20",
port => 4444,
browser => "*firefox",
browser_url => "http://www.godaddy.com/" );
$sel->open_ok("/domains/search.aspx?ci=8969");
$sel->click_ok("domain_search_button");
$sel->wait_for_page_to_load_ok("30000");
my $data=$sel->get_html_source(); # here you get source of the current page

For more info please have a look at cpan http://search.cpan.org/search?query=selenium&mode=all

As scraper you can extract required data from this source:

For scraping data from multiple pages

Open selenium IDE and record the events that you are interested and analyse the code generated and try to implement your own way,

As a last word, let me add that selenium is not completely a scraping tool, it is instead, a testing tool.

For more about selenium have look at http://seleniumhq.org/

VN:F [1.9.6_1107]
Rating: 5.1/10 (8 votes cast)
VN:F [1.9.6_1107]
Rating: +3 (from 3 votes)

Shameem Khalid linux, perl , , , ,

VIRTUAL DESKTOP INFRASRTUCTURE

November 3rd, 2010

VIRTUAL DESKTOP INFRASRTUCTURE (VDI)

The idea behind what is called a Virtual Desktop In­frastructure (VDI) is to run desktop operating systems and applications inside virtual machines that reside on servers in the data center. Desktop operating sys­tems inside virtual machines are also referred to as virtual desktops. Users access the virtual desktops and applications from a desktop PC client or thin client and get almost the full features as if the applications were loaded on their local systems, with the difference being that the applications are centrally managed.

VDI provides the end user with a virtual PC that looks and behaves exactly like their current PC whether working at the office, while traveling or at home. The user’s profile and files are stored centrally so there is no need to carry around files

VDI Empowers Organizations?

Many companies have turned to virtualization tech­nologies for their servers and in their data centers to simplify administration and to reduce management chores and operating costs while maintaining reli­ability and safeguarding against disasters. Seeing the significant benefits virtualization delivers in those environments, companies are now looking to apply the same technology to their desktop computers. Here I am describing some of the benefits of VDI which accelerate the growth of the Organization

• VDI simplifies desktop administrative and management tasks

Virtual desktop infrastructures simplify desktop management and reduce the typical issues faced when dealing with distributed desktops. Any desktop user knows that maintaining a desktop is not an easy task. It is a chore to regularly update the operating system with security patches, upgrade the antivirus software, update the applications or fine-tune the desktop performance. Moreover, backing up and safeguarding the company’s data residing on the employee desktop becomes a nightmare with multiple employee desktops to maintain across the organization.

As a virtual desktop user in an organization deploying VDI, the employee is freed from these burdens as these tasks are shifted to trained IT staffs who manage the data centers where all the necessary company applications and data reside. IT staff is also freed from managing individual employee desktops. Desktop administration and management becomes simplified for a virtual desktop user.

• Users access virtual desktops running in the data center

A VDI enables employees to work remotely at home on their personal desktops while accessing a secure virtual desktop channel through the Internet.

An organization’s remote sales teams, mobile workers and field staff can take advantage of its VDI architecture to communicate securely on company-owned laptops, personal desktops, without compromising the company’s sensitive corporate data.

• Desktop security and data protection are centralized

Increases data security and compliance by reducing the risk of device and data being compromised. Data remains within the walls of the corporate data center instead of being widely distributed throughout branch offices, workstations, and mobile devices that are often lost or stolen.

In contrast, virtual desktops hosted in a data center can be run on servers with high availability features including redundant and hot swappable power sup­plies, fans, and storage disks. Such systems can help protect against downtime and have Increased Security on Company Data.

VDI Solutions

The Leading Desktop Virtualization options are:-

  • VMware View
  • Citrix Xen Desktop
  • Microsoft VDI Suite

I will describe more on the above VDI Solutions in my upcoming Blogs on VDI.

VN:F [1.9.6_1107]
Rating: 10.0/10 (2 votes cast)
VN:F [1.9.6_1107]
Rating: +2 (from 2 votes)

bijopg virtualization ,

Git access control with Gitosis

October 27th, 2010

There are many open source version control systems like Bazaar, Arch, Aegis, SVN ,CVS , Git etc. All systems have there own pros and cons. But I personally like Git due to it’s flexibility and possibilities.It have an excellent network compatibility. We can use the native Git protocol, but it also works over rsync, ssh, HTTP and HTTPS. Access control is the more painful headache for admins. ie who can commit , who can access projects etc. Here we can manage git repositories with gitosis , which is a tool for git access control . More safe thing is gitosis can be used with ssh-key authentication which is one of the most secure data transfer methodology in linux. While we use key-authentication we don’t need to create that user in Git server , it tighten security further becuase developers will not get shell access on server. There should be a Git user in server , here I am creating a user as ‘git’ in server.

If you don’t have Git on your machine install it through yum(Redhat based) or apt-get(Debian based) depending up on your OS.

# apt-get install git-core

For the installation of gitosis we need python-setuptools package in our server.

# apt-get install python-setuptools

Now we can download gitosiis from it’s git repository.

# git clone git://eagain.net/gitosis.git

it’s of few KBs .Now install it with python script.

# cd gitosis
# python setup.py install

create an user as git(you can give any name) . Give him a shell otherwise ssh login might have problems ,then give the desired home directory.(which will be the Git Root directory).

It’s the time to upload public key to Git server. If you have a public-private key pair in local machineuse it , otherwise create a new one with ssk-keygen.

# ssh-keygen -t rsa

Public key will be in name id_rsa.pub . Copy the key and upload it to the server.

Next , execute the command

# sudo -H -u git gitosis-init &lt; /tmp/id_rsa.pub

This will add some directories and files in to user git’s home directory and append the public key to authorized keys file of user git.
Now clone the gitosis-admin repository to your local machine.

$ git clone git@GIT-SERVER:gitosis-admin.git
$ cd gitosis-admin

If you list this directory you can see only two directories .Here You can create new projects , make changes , do access control etc. What ever you do , you have to commit it and push it to the server to replicate the changes on server. Since we cloned gitosis-admin repository we can make all changes locally and push it to the server.

Creating new repository and control access on new repository.

For creating a new repository , edit the gitosis.conf file and add the entries as follows.
Here I am assuming that the new repository name is ‘test’

[group testgroup]
members = hans
writable = test

Here group is just a name , no matter what it is. member is the list of users who can access the repository and ‘writable’ is the name of repository. Commit this changes and push it to the server.

$ git commit -a -m &quot;comment for this commit&quot;
$ git push

Create the directory ‘test’ and initiate it as a git repo.

$ mkdir test
$ cd test
$ git init

Add remote server url to the new repository config file.

$ git remote add origin git@GIT_SERVER:test.git

push the changes in to the server.

$ git push origin master:refs/heads/master

Eventhough we allowed user hans to access ‘test’ repository , we didn’t add his public key to the server . So he will not be able to access the project repository. So copy his public key to ‘keydir’ as hans.pub . It should have a ‘.pub’ extension.

$ cd gitosis-admin
$ cp /home/hans/hans.pub keydir/hans.pub
$ git add keydir/hans.pub
$ git commit -a -m &quot;comment&quot;
$ git push

Yes we done it !
Now user hans can clone the test repository to his local machine. He can make changes ,add files ,commit ,and push changes to the server.

hans $ git clone git@GIT_SERVER:test.git
Now try it. Good Luck !

VN:F [1.9.6_1107]
Rating: 4.5/10 (2 votes cast)
VN:F [1.9.6_1107]
Rating: -1 (from 1 vote)

hans linux , ,

Application based routing in Linux.

October 2nd, 2010

By default, routing is based on destination IP address, i.e., the routing table will decide where to route the packet depending on the destination address field in the packet. But there are a number of other utilities and options on routing in linux. One interesting fact is that most of the routers are using linux kernel for their IOS (inter operating system). Isn’t that cool? Now you can use your linux machine as your local router with more confidence, right?

Other than destination IP address we can also use source IP address , Tos (Type of service) , fwmark (Marking of packets by kernel) and the interface on which packet arrived as the parameters for routing decision. Before getting in to IP routing let me explain some scenarios where we need Advanced routing features. Suppose we have 3-4 gateways and we can easily differentiate them on their bandwidth, reliability, QoS and cost, which one do we select? We would definitely choose the one with the best combination of all these features. But if everyone selects that gateway and start using it, it would obviously be overwhelmed with the traffic. So it is best to differentiate services based on their priority. We should give more priority to services which need higher bandwidth or more reliability. Then the route packet will be used by higher priority services through the most reliable gateway. One example for such service is the SSH.

Now let’s begin with the commands for this advanced routing feature. We will be using the combination of iptables, IP route and IP rule commands here. All linux systems have an iptable package by default. If you don’t have IP route or IP rule command, you have to download the iproute2 package. Before writing the rules on iptables, check kernel modules and options that are required for the full operation of iptable. But since the latest kernels have all the modules and options for iptables you will not have to worry about it.

Let’s start from the iptable. Here we shall take SSH as the service for routing. Use ‘mangle’ table of iptable for modifying the SSH packets. We will need root access for this. We are assuming that SSH is using its default port 22.

1. #iptables -t mangle -A OUTPUT -p tcp –dport 22 -j MARK –set-mark 0×1

We are marking all packets with destination port 22 as ’0×1′ .Now save and restart iptables.
#service iptables save
#service iptables restart

To delete this entry from iptables we can use -D instead of -A.

2. Next, create a new IP route table in /etc/iproute2/rt_table by just giving an entry

100 sshtable

3. Write rule for SSH packets.

#ip rule add fwmark 0×1 lookup sshtable

4. Add route at new table sshtable. Here we shall use ’192.168.1.1′ as the gateway for SSH. All other traffic will go through the
default gateway, which can be seen by IP route show command.

We copy all entries except default gateway entry from main table.
# ip route show table main | grep -Ev ^default | while read ROUTE ; do ip route add table sshtable $ROUTE; done

Add default gateway entry for SSH packets to table sshtable .

#ip route add default via 192.168.1.1 table sshtable

Use “ip route show table sshtable” to show all routes at sshtable.

That’s it..we have done it!!!

You can use SSH to log into your remote server and check your IP with “last” command. You can see that it’s showing your new gateway public IP other than the default gateway. You can use the same for web traffic. Use tcp ports 80,443 ,53 and udp 53 instead of 22 in the above example.

If you want perpetual route settings during reboots, make the following entries in /etc/sysconfig/network-scripts/route-eth0
and /etc/sysconfig/network-scripts/rule-eth0 .

If these files are not there, you can create them and:

in rule-eth0 file paste the following.

fwmark 0×1 lookup sshtable

in route-eth0 add the following line:

default via 192.168.1.1 table sshtable

Then paste # ip route show table main | grep -Ev ^default | while read ROUTE ; do ip route add table sshtable $ROUTE; done in /etc/rc.local file.

Good luck!

VN:F [1.9.6_1107]
Rating: 10.0/10 (3 votes cast)
VN:F [1.9.6_1107]
Rating: +4 (from 4 votes)

hans linux , , , , , , ,

Nasscom IMS 2010: We are Attending

September 9th, 2010


Nasscom Infrastructure Management Summit 2010, will be held in Bangalore on September 15 and 16 and we will be attending.
With the Indian market shifting its focus to cloud computing and RIM services, NASSCOM IMS 2010 will be a platform for discussion about the emerging technologies and where the industry is heading.

Event details on the NASSCOM IMS 2010

VN:F [1.9.6_1107]
Rating: 0.0/10 (0 votes cast)
VN:F [1.9.6_1107]
Rating: 0 (from 0 votes)

admin Events, general , ,

We are hiring Web Designers with excellent coding skills

September 2nd, 2010

Web Designers:

We are looking for well experienced candidates with excellent skills in web designing and technology.

Web Designer requirements:

Up to 1-3 years of experience in Web Designing. Candidates should be familiar with XHTML, CSS and understand basic Javascript/ JQuery.

Eligibility Criteria :

  • Btech/BE ( CS ,EC,IT,EEE) /  MCA with consistent academic records.
  • 1 – 3 yrs of work experience in relevant field.
  • XHTML, CSS
  • Understand basic Javascript / JQuery.
  • Search engine friendly websites and web standards basic knowledge
  • Adobe photoshop, Illustrator , Adobe Flash knowledge and experience.
  • Good  communication  and interpersonal skill.
  • Proactive , quick learner and a very good team player.

Optional requirements:

  • Knowledge in SEO Techniques and Social Media.
  • Basic idea of Web 2.0 concepts
  • Reading / Writing skills.

Selection procedure:

  • Technical Interview
  • Practical test .
  • HR Round

Terms:

  • flat hierarchy
  • ample opportunities to grow
  • opportunity to grow with a fast growing organization
  • industrial standard renumeration package.

How to apply:

Send your resumes to careers[at] sparksupport[dot]com  with Subject line : Experienced Web Designer with a covering letter mentioning current job details and when you can join our organization and details of the current renumeration and pay package.

The openings are immediate and the candidates should be ready to join ASAP.

VN:F [1.9.6_1107]
Rating: 7.5/10 (4 votes cast)
VN:F [1.9.6_1107]
Rating: 0 (from 2 votes)

Nisha Ajil Recruitment

We are hiring Btech/BE Freshers ( CS , ECE , IT , EEE )for our Perl & PHP development wing

September 2nd, 2010

Description:

We are looking for  BE/Btech freshers from IT, CSE ,EEE, EC branches who has excellent programming skills. Proactive candidates with exceptional coding skills. The candidate must be capable of quickly learning new technologies and capable of working on applications deployed in distributed environment. Basic knowledge of Linux platform , Perl, PHP  will be a plus.

Eligibility Criteria:

  • BTech/ BE – CSE, ECE , IT, EEE( 60%above final results and should not have any back papers )
  • Excellent programming skills ( C, C++) .Basic knowledge of PHP/ Perl will be a plus
  • Consistent academic record
  • Proactive attitude
  • Good written and oral communication.
  • Should be a self starter and must be willing to work on complicated tasks.
  • A quick learner.
  • Familiarity with software development process .

Selection procedure:

  • Written Test ( Technical and Aptitude – objective type )
  • Technical Interview
  • Practical test where you will be given a small programing task
  • HR Round

Terms:

  • 6 months on job training period
  • flat hierarchy
  • ample opportunities to grow
  • opportunity to grow with a fast growing  organization

How to apply:

Send your resumes to careers[at] sparksupport[dot]com with a covering letter and subject line BTech Freshers .The openings are immediate and the candidates should be ready to join ASAP.

VN:F [1.9.6_1107]
Rating: 7.0/10 (2 votes cast)
VN:F [1.9.6_1107]
Rating: 0 (from 2 votes)

Nisha Ajil Recruitment