Archive

Author Archive

Deploying with Capistrano

December 21st, 2010

What is Capisrano?

Capistrano is a deployment tool and is popularly used for deploying rails applications to your servers. But its a good configuration tool too. You can setup your servers using capistrano. You can preform tasks like adding users, installing and configuring packages like apache, mysql or postfix, a xen vm etc on your desired network host from a central server(wherever you kept the configuration repository).

Puppet and Capistrano?

Well., you might have heard about puppet – another popular network host configuration tool. Great..Now it will be easy to understand capistrano. If you haven’t, dont worry, I will make it clear for you.

The puppet is working in such a way that a puppet server will be running on one host in the nerwork. And all the desired hosts would run a puppet client daemon. You can configure your client manully and can automate installation on your clients. Whenever you want to change the configuration just do it on the server. It will be updated to the client as the client puppet daemons updates from the server in each interval that you specify. Now you would note that the client daemon pulls the configuration from puppet server. But in capistrano the user pushes the configuration to the desired client when he needs it. Puppet uses “meta language” for the configurations. Whereas capistrano uses ruby. And you know that you got several modules and feautres with ruby and so you may can write your additions to cap if you are well versed with that. One more thing you should note that puppet does its dependency resolution automatically, but in capistrano, you may have to handle the dependency manually in some cases. But as I said earlier you can overcome this if you know well and I would say it may be little difficult to go to that extent. But when you have done with a perfect configuration you feel it greatly helpful.

Installation and Configuration

To install just enter the command:

$gem install capistrano

Provided you have rubygems installed on your machine. If not, install it with apt or yum.

$gem install deprec

Note: I have found several docs without mentioning this..and it made my cap commands haden’t worked till I found this and installed..

Now create a repository as the source (central configuration repo) using svn. Svn is common version control system that is used in cordination with the cap. If you like git go with that.

$svnadmin create sparkconf

Sparkconf is the repository name where you will keep all the configuration of your servers. You can use your own repository name.

Configure this repo for access to the desired users. I wont go with svn details because its beyond the context of our topic.

now you can checkout your repo

$svn co <ur-repo-ulr> sparkconf
$cd sparkconf
$capify .

This “capify” command will initiate your cap repo. It will create a file named Capfile and a deploy.rb file in a directory named config. In capfile you can see that there is a statement to load deploy.rb file. So you can define your own tasks in Capfile or in deploy.rb. But usually this is not the convention. You would leave your Capfile without any changes. And you may specify some information about your setting in the deploy.rb file. The sample deploy.rb which is generated automatically, needs some explanation.

<br />
set :application, &quot;agileblazeworks&quot;<br />
set :repository,  &quot;report&quot;</p>
<p>set :scm, :subversion<br />
# Or: `accurev`, `bzr`, `cvs`, `darcs`, `git`, `mercurial`, `perforce`, `subversion` or `none`</p>
<p>role :web, &quot;rails.spark.com&quot;                                 # Your HTTP server, Apache/etc<br />
role :app, &quot;rails.spark.com&quot;                                 # This may be the same as your `Web` server<br />
role :db,  &quot;rails.spark.com&quot;, :primary = true    # This is where Rails migrations will run<br />
#role :db,  &quot;your slave db-server here&quot;</p>
<p># If you are using Passenger mod_rails uncomment this:<br />
# if you're still using the script/reapear helper you will need<br />
# these http://github.com/rails/irs_process_scripts</p>
<p># namespace :deploy do<br />
#   task :start do ; end<br />
#   task :stop do ; end<br />
#   task :restart, :roles =&amp;gt; :app, :except =&amp;gt; { :no_release =&amp;gt; true } do<br />
#     run &quot;#{try_sudo} touch #{File.join(current_path,'tmp','restart.txt')}&quot;<br />
#   end<br />
# end<br />

Dont worry about the configuration here. Most of them are needed if you are deploying a rails application for a web server. After breifing this we will go for some other configuration options that you are really looking for…to install and configure packages on your hosts.

If your application (means your rails application) is not separated into application, web and database servers, you can either set them to be the same value; or comment out, or remove the one you do not require. The “:primary => true” part of the role definitions allows you to have more than one database server. If you dont have two skipp this primary option. If, for example when deploying a Rails application you only wanted db1 to run migrations, in the first example both might. Essentially when using the Rails deployment recipes, the :primary option defines where database migrations are run. Similar attributes include :no_release often used for the :web role by some of the recipes in circulation to decide which servers should not have the code checked out to them. Attributes like these are arbitrary and you can define some of your own, and use them to filter more precisely where your own tasks run.

You may add these options to the deploy.rb file

<br />
set :user, 'jaseer'              #This is the user you have on the target machine. Capistrano try to login to the target machine using this account.<br />
ssh_options[:keys] = %w(/home/users/mylocalname/.ssh/jas_rsa)<br />
set :use_sudo, true            #if you want to append all commands with sudo.<br />
set :password, &quot;yourpassword&quot;      #You can login using key (above) or with password. Use either key or password..not both.</p>
<p>default_run_options[:pty] = true #this is really helpful. If you dont have this you will struggle in runnig sudo tasks. I had lost some time searching around this.<br />

Some cap commands

$ cap -h

This will give out a list of all the options it accepts.

$ cap -H

It will give you a description of each option.

Next, let’s ask Capistrano what all tasks it will do. Capistrano comes bundled with several built-in tasks. You can also write your own to automate workflows of your own. For now, let’s see what tasks Capistrano knows:

$ cap -T

And finally, to get a detailed description of a command, type

$cap -e <task>

Example

Suppose you want install and configure nagios on one of your servers.

cap deprec:nagios:install HOSTS=monitor.spark.com

This command will install nagios on the host monitor.blocksglobal.com (You can use ip also). Iif you want to override the user in deploy.rb, or any other files, use the option USER=<username> option. This will ask your sudo password, as you know for some task you may need the administrator privileges.

Now generate configuration files

#cap deprec:nagios:config_gen

The configuration files are created in your localrepo not in the installed server. A directry tree is created under sparkconf/config/  with the name nagios. It will contain all configuration files of nagios like hosts.cfg, service.cfg etc. You can change each  according to your needs. Update svn. Push those configuration to your client.

$cd config/nagios

Change and configure the files as your needs, commit the changes and update svn. Go back to the root of your repository, here it is sparkconf. Now push the config to your nagios server.

$cap deprec:nagios:config HOSTS=monitor.spark.com

Defining Tasks

Now let me explain how to define tasks. Often I would define my own tasks in config/*/recipes.rb for example config/nagios/recipes.rb.  * Can be any thing like postfix, mysql as you generate the config directory for them using config_gen.

<br />
namespace : one do<br />
 task :default do<br />
    test<br />
    one.test<br />
    two.test<br />
  end<br />
  task :test do<br />
    puts &quot;Test One Successful!&quot;<br />
  end<br />
end<br />
namespace :two do<br />
  task :test do<br />
    puts &quot;Test Two Successful&quot;<br />
  end<br />
end<br />

Here these are the available commands I can use with cap…

$cap one
$cap one:default
$cap one:test
$cap two

Hope you understood how to define them. Note in how many ways I called the task “test” inside the task default. This is easy right? Now go, create your repo and try your cap tasks..I will stop with a final simple example:

<br />
task :backup_database, :roles =&gt; :db, : only =&gt; { :backup =&gt; true } do<br />
 run &quot;#{sudo} mysqldump ... &gt; /tmp/backup.sql&quot;<br />
 run &quot;#{sudo} bzip2 /tmp/backup.sql&quot;<br />
 run &quot;scp /tmp/backup.sql.bz2 offsite.host:/u/backups&quot;<br />
 run &quot;#{sudo} rm /tmp/backup.sql.bz2&quot;<br />
end<br />

Next time I will write one more article on deploying rails app using cap.. Have fun..bye

VN:F [1.9.6_1107]
Rating: 8.0/10 (1 vote cast)
VN:F [1.9.6_1107]
Rating: 0 (from 0 votes)

Jaseer Articles, linux , , , ,

Load balancing in Wowza Media Server

August 3rd, 2010

Wowza Media Server is a Video Stream Server used by popular CDN providers to provide video content across the internet. It has come with a large variety of features and several streaming options. And one of the attractive feature provided by wowza is that, it allows clustering like feature so that multiple servers can be configured to provide a video stream to the clients. In such a setup one of the server will act as a loadbalancing server and the others will act as the loadbalancing edge/client servres. If a client requests a video stream then a load balancer server redirect it to the least loaded wowza server. This loadbalancing feature is usually used while publishing live streams becuase the number of concurrent connections to the server providing live stream will be high and a load balancing setup can easily manage it. Here we will discuss the load balancing setup for a live stream proess.

To employ loadbalancing setup in live streaming first we have to setup live streaming repeater configuration available with wowza. This is because the live stream will be actually published to only one server and we need to duplicate among a no of stream servers as our need. So there will be live repeater origin server and several live repeater edge servers. When the player will request the contentfrom an edge server and the edge server will maintain a single connection per-unique stream to the origin. Origin and edge configuration is an application level configuration. A single Wowza Server instance can be configured as an origin for one application and an edge for another.

1) Configuring liverepeater-origin server
  • Create a folder named [install-dir]/applications/liverepeater.
  • Create a folder named [install-dir]/conf/liverepeater and copy the file [install-dir]/conf/Application.xml into this new folder.
  • Edit the newly copied Application.xml file and make the following changes:

a) Change the Streams/StreamType to liverepeater-origin

b. Change the LiveStreamPacketizers to: cupertinostreamingpacketizer,smoothstreamingpacketizer

2)Configuring edge server

Follow these steps to configure each of the edge servers

  • Create a folder named [install-dir]/applications/liverepeater.
  • Create a folder named [install-dir]/conf/liverepeater and copy the file [install-dir]/conf/Application.xml into this new folder.
  • Edit the newly copied Application.xml file and make the following changes.

a)Change the Streams/StreamType to liverepeater-edge (you can use the liverepeater-edge-lowlatency stream type if low latency is important, this will add extra load to the server).

b)Change the LiveStreamPacketizers to: cupertinostreamingrepeater,smoothstreamingrepeater

c)Uncomment the Repeater/OriginURL section and set OriginURL to rtmp URL of the origin server. For example if the origin server uses the domain name      origin.mycompany.com, this value should be set to:


<Repeater>

         <OriginURL>rtmp://origin.mycompany.com</OriginURL>

         <QueryString></QueryString>

</Repeater>

Note: Let us assume origin.mycompany.com is the origin server here

3)Configring the Loadbalancer server

  • First Download the loadbalancing module the from the following link of wowza forums. Unzip the downloaded file

http://www.wowzamedia.com/forums/showthread.php?t=4637

  • Copy the file lib/wms-plugin-loadbalancer.jar from this zip archive to the [install-dir]/lib/ folder of Wowza Media Server 2
  • Copy the file conf/crossdomain.xml from this zip archive to the [install-dir]/conf/ folder of Wowza Media Server 2.
  • Edit [install-dir]/conf/Server.xml and make the following changes:

Add the following ServerListener entry to the <ServerListeners> list:

    <ServerListener>
           	  <BaseClass>com.wowza.wms.plugin.loadbalancer.ServerListenerLoadBalancerListener</BaseClass>
    </ServerListener>

Add the following properties to the <properties> section of the bottom of the server.xml file

      <Property>
               <Name>loadBalancerListenerKey</Name>
	      <Value>023D4FB4IS83</Value>
      </Property>
      <Property>
	      <Name>loadBalancerListenerIpAddress</Name>
	      <Value>*</Value>
      </Property>
      <Property>
	     <Name>loadBalancerListenerPort</Name>
	     <Value>1934</Value>
	     <Type>Integer</Type>
      </Property>
      <Property>
	      <Name>loadBalancerListenerRedirectorClass</Name>
	      <Value>com.wowza.wms.plugin.loadbalancer.LoadBalancerRedirectorConcurrentConnects</Value>
      </Property>
      <Property>
	      <Name>loadBalancerListenerMessageTimeout</Name>
	      <Value>5000</Value>
	      <Type>Integer</Type>
      </Property>
  • Edit [install-dir]/conf/VHost.xml and add the following HostPort/HTTPProvider XML snippet just before the HTTPProvider definition for com.wowza.wms.http.HTTPServerVersion:
      <HTTPProvider>
                   <BaseClass>com.wowza.wms.plugin.loadbalancer.HTTPLoadBalancerRedirector</BaseClass>
	           <RequestFilters>*loadbalancer</RequestFilters>
	           <AuthenticationMethod>none</AuthenticationMethod>
	           <Properties>
                              <Property>
		          	  <Name>enableServerInfoXML</Name>
                   		  <Value>true</Value>
		                  <Type>Boolean</Type>
		           </Property>
	         </Properties>
     </HTTPProvider>

4)To setup an edge servers in load balancing

Do the first two steps as done for the load balacer server

  • Edit [install-dir]/conf/Server.xml and make the following changes:

Add the following ServerListener entry to the <ServerListeners> list:


     <ServerListener>
	           <BaseClass>com.wowza.wms.plugin.loadbalancer.ServerListenerLoadBalancerSender</BaseClass>
     </ServerListener>

Add the following properties to the <Properties> section at the bottom of Server.xml:


   <Property>
	    <Name>loadBalancerSenderTargetPath</Name>
	    <Value>${com.wowza.wms.AppHome}/conf/loadbalancertargets.txt</Value>
   </Property>
   <Property>
	    <Name>loadBalancerSenderRedirectAddress</Name>
	    <Value>[redirect-address]</Value>
   </Property>
   <Property>
            <Name>loadBalancerSenderMonitorClass</Name>
	    <Value>com.wowza.wms.plugin.loadbalancer.LoadBalancerMonitorDefault</Value>
   </Property>
   <Property>
	    <Name>loadBalancerSenderMessageInterval</Name>
	    <Value>2500</Value>
	    <Type>Integer</Type>
   </Property>

Where [redirect-address] is the external ip address or domain name of this machine. This address will be used when redirecting to this edge server. When using this system on EC2 you can set the [redirect-address] to   ${com.wowza.amazonaws.ec2.AWSEC2_METADATA_PUBLIC_IPV4} and upon server startup it will use the public ip address of the server for this value.

  • Create the file [install-dir]/conf/loadbalancertargets.txt using a text editor and enter the following two lines (the first line is a comment):

# [load-balancer-ip-address],[load-balancer-port],[encryption-key]

[load-balancer-ip-address],1934,023D4FB4IS83

  Where [load-balancer-ip-address] is the ip  address or domain name of the load balancer.

This configurations uses UDP port 1934 for communication between the edge servers and the load balancer. Be sure this port is open on your firewall. All communication between the edge server and the load balancer is encrypted and signed. The encryption key is set on the load balancer server using the loadBalancerListenerKey property and in the loadbalancertargets.txt file on the edge servers. These keys must match. An edge server can communicate with multiple load balancers by adding additional lines to the loadbalancertargets.txt file.

You can now startup the load balancer and multiple edge servers. If functioning properly, the edge servers will update the load balancer every 2.5 seconds with status and load information. You can get information from the load balancer in regards to which edge servers are currently registered and their status by opening a web browser and entering the following url:

 http://[load-balancer-ip-address]:1935/loadbalancer?serverInfoXML

5)Now Configure a redirect application to redirect the connection requests to the least loaded server.
  • Create the folder [install-dir]/applications/redirect.
  • Create the folder [install-dir]/conf/redirect and copy the file [install-dir]/conf/Application.xml into this new folder.
  • Create the folder [install-dir]/conf/redirect and copy the file [install-dir]/conf/Application.xml into this new folder.
      <Module>
	    <Name>ModuleLoadBalancerRedirector</Name>
	    <Description>ModuleLoadBalancerRedirector</Description>
            <Class>com.wowza.wms.plugin.loadbalancer.ModuleLoadBalancerRedirector</Class>
      </Module>
  • Add the following properties the properties section at the bottom of the Application.xml file:
   <Property>
             <Name>redirectAppName</Name><code>
             <Value>[application-name]</Value>
    </Property>
    <!--
    <Property>
	     <Name>redirectPort</Name>
	     <Value>[redirect-port]</Value>
    </Property>
    -->
    <!--
    <Property>
	     <Name>redirectScheme</Name>
              <Value>rtmp</Value>
    </Property>
    -->
    <Property>
	     <Name>redirectOnConnect</Name>
	     <Value>true</Value>
	     <Type>Boolean</Type>
    </Property>

Where [application-name] is the name of the application you wish to redirect to on the edge server and [redirect-port] is the port to redirect to (such as port 1935 or port 80). The redirectPort and redirectScheme are commented out so that the system will use the same scheme and port used to connect to the load balancer to connect to the edge server. This will work better when using any type of protocol (rtmp to rtmpt) or port rollover scheme.

VN:F [1.9.6_1107]
Rating: 8.7/10 (10 votes cast)
VN:F [1.9.6_1107]
Rating: +2 (from 4 votes)

Jaseer Articles, general , , ,

How To Install Red5 on CentOS

August 2nd, 2010

Red5 is an open source video sream server appllication which helps you to stream your video content across the web. Inorder to install and run red5, you need the java platform installed on your system. Using apache-ant or eclipse we can build and install Red5. Here apache-ant  method is described.

1)Using Apache-ant

a)install jdk 1.6 or higher and its curresponding deveopment package.

#yum install java-1.6.0-openjdk java-1.6.0-openjdk-devel

b)Install apache-ant

To build the red5 binary from source we need apache-ant. So install it.

#cd /usr/src

#wget http://archive.apache.org/dist/ant/binaries/apache-ant-1.8.0-bin.tar.bz2

#tar -xjf apache-ant-1.8.0-bin.tar.bz2

#mv apache-ant-1.8.0 /usr/local/ant

c)Set the enviroment variables for java, ant and java class path

export ANT_HOME=/usr/local/ant
export JAVA_HOME=/usr/lib/jvm/java
export PATH=$PATH:/usr/local/ant/bin
export CLASSPATH=.:$JAVA_HOME/lib/classes.zip

If you want this to be avilable for all users, append these lines to the file /etc/bashrc

d)Now Install subversion to downlaod the svn version from the googlecode.

#yum install subversion

e)Download and Install Red5

#svn checkout http://red5.googlecode.com/svn/java/server/trunk/ red5

#mv red5 /usr/local

#cd /usr/local/red5

#ant prepare

#ant build

Note: If you are building on CentOS 5.4 use “#ant dist” command instead of “#ant build”

This will take some time. So please be patient.

Finally you will see a line “Build successful” .That means your red5 installation is complete. Now copy the conf directory from dist/ to the current directory and now test your installation by runnig the script

#cp -r dist/conf .
#./red5.sh

Your Installation is ok if it shows “Installer service created” in the last. Now press ctrl+c to quit the process and go for the init script

f)Init Script

#vi /etc/init.d/red5

copy the following code to it.

#!/bin/bash
 PROG=red5
 RED5_HOME=/usr/local/red5
 DAEMON=$RED5_HOME/$PROG.sh PIDFILE=/var/run/$PROG.pid
# Source function library . /etc/rc.d/init.d/functions[ -r /etc/sysconfig/red5 ] && . /etc/sysconfig/red5RETVAL=0
case "$1" in
 start)
 echo -n $"Starting $PROG: "
 cd $RED5_HOME
 $DAEMON >/dev/null 2>/dev/null &
 RETVAL=$?
 if [ $RETVAL -eq 0 ]; then
 echo $! > $PIDFILE
 touch /var/lock/subsys/$PROG
 fi
 [ $RETVAL -eq 0 ] && success $"$PROG startup" || failure $"$PROG startup"
 echo
 ;;
 stop)
 echo -n $"Shutting down $PROG: "
 killproc -p $PIDFILE
 RETVAL=$?
 echo
 [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$PROG
 ;;
 restart)
 $0 stop
 $0 start
 ;;
 status)
 status $PROG -p $PIDFILE
 RETVAL=$?
 ;;
 *)
 echo $"Usage: $0 {start|stop|restart|status}"
 RETVAL=1 esac
exit $RETVAL

 
g)Start the service and test your server

#/etc/init.d/red5 start

Now you can open your browser enter http://[your_ip]:5080 and see your sever working. You can install the sample applications and view the demos.

VN:F [1.9.6_1107]
Rating: 4.3/10 (3 votes cast)
VN:F [1.9.6_1107]
Rating: +1 (from 1 vote)

Jaseer Articles, general, linux , ,

What is Video Streaming

July 13th, 2010

Video stream servers are meant to provide the video content across internet in reliable and generic way so that any client can access easily. They uses diffrent protocols other than http namely rtsp, rtmp, rtp etc. And these protocols mainly uses udp. A basic video streaming system contains 3 components. A video publishing side, The video streaming server and the client side player (usually flash player embedded in the webpage). The video publishing system will upload a live or static video content to the Video Stream Server and the video stream sever will broadcast (means streaming)  that uploaded video such that the clients can view the videos without bothering about any codecs. The video uploading system should have an encoder to publish live contents to the server. The encoder does the process of transcoding to provide the content to the desired format of the stream server.

Major stream server applications are Wowza Media Server, Adobe Flash Media Server, Red5:open source flash server, Darwin Streaming server (open source) etc.

Working
Actual work flow when a web client is accessing stream video from a website is deiscribed in the figure. When the user clicks on a particular video icon the web server passes a request to the stream server. It will also notify the information about the client side  player (usually it will be a flash player application). Because the if it is a flash client then the request sent will be rtmp (Real Time Messaging Protocol ) protocol based. The stream server will stream (send the video packets) video based on rtmp protocol for flash players. For client players like vlc, apple quick-time etc  the protocol used is rtsp (Real Time Streaming Protocol).

A video streaming setup contains 3 components they are

  1. Video Stream server
  2. Web server for providing a web interface to the clients
  3. The flash player embedded on the web page (Or stand alone players like vlc, quicktime etc if not accessing through the site).

If the video server streams a live video then one publishing site will be there which will provide the  the live video source. Figure 2 illustrates such a video stream server setup. A video Streaming server application would be running on the Streaming server. The broadcaster would use an live  encoding tool (eg: flash live media encoder). If the client is accessing the video on an html web page the player might be adobe flash player. Flash offers online broadcasters a platform rich with unmatchable flexibility and superior quality video delivery. Flash has fast become the standard for delivery of rich-media over the internet. TV and Video Broadcasters and a wide variety of content creators around the globe have chosen Flash as their No. 1 streaming format.

Live video stream setup

If the video server streams a live video then one publishing site will be there which will provide the the live video source. Figure 2 illustrates such a video stream server setup. A video Streaming server application would be running on the Streaming server. The broadcaster would use an live encoding tool (eg: flash live media encoder). If the client is accessing the video on an html web page

the player might be adobe flash player. Flash offers online broadcasters a platform rich with unmatchable flexibility and superior quality video delivery. Flash has fast become the standard for delivery of rich-media over the internet. TV and Video Broadcasters and a wide variety of content creators around the globe have chosen Flash as their No. 1 streaming format.

VN:F [1.9.6_1107]
Rating: 4.8/10 (4 votes cast)
VN:F [1.9.6_1107]
Rating: +2 (from 2 votes)

Jaseer Articles, general , , , ,