Archive

Archive for August, 2010

Load balancing in Wowza Media Server

August 3rd, 2010

Wowza Media Server is a Video Stream Server used by popular CDN providers to provide video content across the internet. It has come with a large variety of features and several streaming options. And one of the attractive feature provided by wowza is that, it allows clustering like feature so that multiple servers can be configured to provide a video stream to the clients. In such a setup one of the server will act as a loadbalancing server and the others will act as the loadbalancing edge/client servres. If a client requests a video stream then a load balancer server redirect it to the least loaded wowza server. This loadbalancing feature is usually used while publishing live streams becuase the number of concurrent connections to the server providing live stream will be high and a load balancing setup can easily manage it. Here we will discuss the load balancing setup for a live stream proess.

To employ loadbalancing setup in live streaming first we have to setup live streaming repeater configuration available with wowza. This is because the live stream will be actually published to only one server and we need to duplicate among a no of stream servers as our need. So there will be live repeater origin server and several live repeater edge servers. When the player will request the contentfrom an edge server and the edge server will maintain a single connection per-unique stream to the origin. Origin and edge configuration is an application level configuration. A single Wowza Server instance can be configured as an origin for one application and an edge for another.

1) Configuring liverepeater-origin server
  • Create a folder named [install-dir]/applications/liverepeater.
  • Create a folder named [install-dir]/conf/liverepeater and copy the file [install-dir]/conf/Application.xml into this new folder.
  • Edit the newly copied Application.xml file and make the following changes:

a) Change the Streams/StreamType to liverepeater-origin

b. Change the LiveStreamPacketizers to: cupertinostreamingpacketizer,smoothstreamingpacketizer

2)Configuring edge server

Follow these steps to configure each of the edge servers

  • Create a folder named [install-dir]/applications/liverepeater.
  • Create a folder named [install-dir]/conf/liverepeater and copy the file [install-dir]/conf/Application.xml into this new folder.
  • Edit the newly copied Application.xml file and make the following changes.

a)Change the Streams/StreamType to liverepeater-edge (you can use the liverepeater-edge-lowlatency stream type if low latency is important, this will add extra load to the server).

b)Change the LiveStreamPacketizers to: cupertinostreamingrepeater,smoothstreamingrepeater

c)Uncomment the Repeater/OriginURL section and set OriginURL to rtmp URL of the origin server. For example if the origin server uses the domain name      origin.mycompany.com, this value should be set to:


<Repeater>

         <OriginURL>rtmp://origin.mycompany.com</OriginURL>

         <QueryString></QueryString>

</Repeater>

Note: Let us assume origin.mycompany.com is the origin server here

3)Configring the Loadbalancer server

  • First Download the loadbalancing module the from the following link of wowza forums. Unzip the downloaded file

http://www.wowzamedia.com/forums/showthread.php?t=4637

  • Copy the file lib/wms-plugin-loadbalancer.jar from this zip archive to the [install-dir]/lib/ folder of Wowza Media Server 2
  • Copy the file conf/crossdomain.xml from this zip archive to the [install-dir]/conf/ folder of Wowza Media Server 2.
  • Edit [install-dir]/conf/Server.xml and make the following changes:

Add the following ServerListener entry to the <ServerListeners> list:

    <ServerListener>
           	  <BaseClass>com.wowza.wms.plugin.loadbalancer.ServerListenerLoadBalancerListener</BaseClass>
    </ServerListener>

Add the following properties to the <properties> section of the bottom of the server.xml file

      <Property>
               <Name>loadBalancerListenerKey</Name>
	      <Value>023D4FB4IS83</Value>
      </Property>
      <Property>
	      <Name>loadBalancerListenerIpAddress</Name>
	      <Value>*</Value>
      </Property>
      <Property>
	     <Name>loadBalancerListenerPort</Name>
	     <Value>1934</Value>
	     <Type>Integer</Type>
      </Property>
      <Property>
	      <Name>loadBalancerListenerRedirectorClass</Name>
	      <Value>com.wowza.wms.plugin.loadbalancer.LoadBalancerRedirectorConcurrentConnects</Value>
      </Property>
      <Property>
	      <Name>loadBalancerListenerMessageTimeout</Name>
	      <Value>5000</Value>
	      <Type>Integer</Type>
      </Property>
  • Edit [install-dir]/conf/VHost.xml and add the following HostPort/HTTPProvider XML snippet just before the HTTPProvider definition for com.wowza.wms.http.HTTPServerVersion:
      <HTTPProvider>
                   <BaseClass>com.wowza.wms.plugin.loadbalancer.HTTPLoadBalancerRedirector</BaseClass>
	           <RequestFilters>*loadbalancer</RequestFilters>
	           <AuthenticationMethod>none</AuthenticationMethod>
	           <Properties>
                              <Property>
		          	  <Name>enableServerInfoXML</Name>
                   		  <Value>true</Value>
		                  <Type>Boolean</Type>
		           </Property>
	         </Properties>
     </HTTPProvider>

4)To setup an edge servers in load balancing

Do the first two steps as done for the load balacer server

  • Edit [install-dir]/conf/Server.xml and make the following changes:

Add the following ServerListener entry to the <ServerListeners> list:


     <ServerListener>
	           <BaseClass>com.wowza.wms.plugin.loadbalancer.ServerListenerLoadBalancerSender</BaseClass>
     </ServerListener>

Add the following properties to the <Properties> section at the bottom of Server.xml:


   <Property>
	    <Name>loadBalancerSenderTargetPath</Name>
	    <Value>${com.wowza.wms.AppHome}/conf/loadbalancertargets.txt</Value>
   </Property>
   <Property>
	    <Name>loadBalancerSenderRedirectAddress</Name>
	    <Value>[redirect-address]</Value>
   </Property>
   <Property>
            <Name>loadBalancerSenderMonitorClass</Name>
	    <Value>com.wowza.wms.plugin.loadbalancer.LoadBalancerMonitorDefault</Value>
   </Property>
   <Property>
	    <Name>loadBalancerSenderMessageInterval</Name>
	    <Value>2500</Value>
	    <Type>Integer</Type>
   </Property>

Where [redirect-address] is the external ip address or domain name of this machine. This address will be used when redirecting to this edge server. When using this system on EC2 you can set the [redirect-address] to   ${com.wowza.amazonaws.ec2.AWSEC2_METADATA_PUBLIC_IPV4} and upon server startup it will use the public ip address of the server for this value.

  • Create the file [install-dir]/conf/loadbalancertargets.txt using a text editor and enter the following two lines (the first line is a comment):

# [load-balancer-ip-address],[load-balancer-port],[encryption-key]

[load-balancer-ip-address],1934,023D4FB4IS83

  Where [load-balancer-ip-address] is the ip  address or domain name of the load balancer.

This configurations uses UDP port 1934 for communication between the edge servers and the load balancer. Be sure this port is open on your firewall. All communication between the edge server and the load balancer is encrypted and signed. The encryption key is set on the load balancer server using the loadBalancerListenerKey property and in the loadbalancertargets.txt file on the edge servers. These keys must match. An edge server can communicate with multiple load balancers by adding additional lines to the loadbalancertargets.txt file.

You can now startup the load balancer and multiple edge servers. If functioning properly, the edge servers will update the load balancer every 2.5 seconds with status and load information. You can get information from the load balancer in regards to which edge servers are currently registered and their status by opening a web browser and entering the following url:

 http://[load-balancer-ip-address]:1935/loadbalancer?serverInfoXML

5)Now Configure a redirect application to redirect the connection requests to the least loaded server.
  • Create the folder [install-dir]/applications/redirect.
  • Create the folder [install-dir]/conf/redirect and copy the file [install-dir]/conf/Application.xml into this new folder.
  • Create the folder [install-dir]/conf/redirect and copy the file [install-dir]/conf/Application.xml into this new folder.
      <Module>
	    <Name>ModuleLoadBalancerRedirector</Name>
	    <Description>ModuleLoadBalancerRedirector</Description>
            <Class>com.wowza.wms.plugin.loadbalancer.ModuleLoadBalancerRedirector</Class>
      </Module>
  • Add the following properties the properties section at the bottom of the Application.xml file:
   <Property>
             <Name>redirectAppName</Name><code>
             <Value>[application-name]</Value>
    </Property>
    <!--
    <Property>
	     <Name>redirectPort</Name>
	     <Value>[redirect-port]</Value>
    </Property>
    -->
    <!--
    <Property>
	     <Name>redirectScheme</Name>
              <Value>rtmp</Value>
    </Property>
    -->
    <Property>
	     <Name>redirectOnConnect</Name>
	     <Value>true</Value>
	     <Type>Boolean</Type>
    </Property>

Where [application-name] is the name of the application you wish to redirect to on the edge server and [redirect-port] is the port to redirect to (such as port 1935 or port 80). The redirectPort and redirectScheme are commented out so that the system will use the same scheme and port used to connect to the load balancer to connect to the edge server. This will work better when using any type of protocol (rtmp to rtmpt) or port rollover scheme.

VN:F [1.9.6_1107]
Rating: 8.7/10 (10 votes cast)
VN:F [1.9.6_1107]
Rating: +2 (from 4 votes)

Jaseer Articles, general , , ,

Transparent Data Encryption In SQL Server 2008

August 2nd, 2010


As a Database or System Administrator, security is one of the most important areas to consider when it comes to protecting the databases that you support. We use various mechanisms and technologies to secure our data and databases such as firewalls, certificates, and data encryption. Having said that although we have secured our environment, questions will always be raised regarding database security. Although we have protected our databases, what would happen if someone steals the mdf file or if someone steals the backup file.

Although there are few ways to control this scenario using third-party solutions up until SQL Server 2008 there has been no native way to handle this problem. SQL Server 2008 introduces a new feature that protects the database called Transparent Data Encryption – TDE which provides protection to the entire database, that is Data is encrypted before it is written to disk; data is decrypted when it is read from disk. There is no need of changes in the application when it is implementing with existing applications that means there is no headache for developers!

Note: Applies Only to Microsoft SQL Server 2008 Enterprise Edition

I hope this article will guide you throughout the implementation of TDE in MSSQL SERVER 2008.

The diagram below shows how SQL Server encrypts a database with TDE:

Implementation of TDE

There are four steps included in Implementation of TDE

* Create a master key

* Create or obtain a certificate protected by the master key

* Create a database encryption key and protect it by the certificate

* Set the database to use encryption

1. Create a master key

A master key is a symmetric key that is used to create certificates and asymmetric keys.  Execute the following script to create a master key:

USE master;

CREATE MASTER KEY

ENCRYPTION BY PASSWORD = 'Pass@word1';
GO

2. Create Certificate
Certificates can be used to create symmetric keys for data encryption or to encrypt the data directly.  Execute the following script to create a certificate:


CREATE CERTIFICATE TDECert

WITH SUBJECT = 'TDE Certificate'

GO

3. Create a database encryption key and protect it by the certificate

1. Go to object explorer in the left pane of the MSSQL SERVER Management Studio

2. Right Click on the database on which TDE Requires

3. Click Tasks and Navigate to Manage Database Encryption

4. Select the encryption algorithm (AES 128/192/256) and select the certificate you have    created

5. Then Mark the check Box for Set Database Encryption On

You can query the is_encrypted column in sys.databases to determine whether TDE is enabled for a particular database.


SELECT [name], is_encrypted FROM sys.databases

You are Done !

VN:F [1.9.6_1107]
Rating: 9.0/10 (1 vote cast)
VN:F [1.9.6_1107]
Rating: +1 (from 1 vote)

bijopg Windows , , ,

Web Scraping : A basic know-how.

August 2nd, 2010

A Web crawler is a computer program that browses the World Wide Web in a methodical, automated manner or in an orderly fashion. Other terms for Web crawlers are ants, automatic indexers, bots, web spiders, web robots, etc. The process is termed “web crawling”, and most site engines use it as a means to provide up-to-date data, in order to create a copy of all pages that have been visited. These are later processed, and the search engine will index the downloaded pages.
This helps in :

  • faster search
  • automating maintenance task on a web site
  • gathering specific types of information from websites

The bot starts with seeds, which are a list of URLs to visit. Once the “crawler” is on one of the listed URLs, the hyperlinks in that page are identified and added to the “crawl frontier” which is the set of URLs that are to be visited. These are later visited according to a pre-defined set of policies.

Web Crawlers can be developed using any language : perl, python, java, asp,php etc. Among these, we chose perl to develop a web crawler. Lets see what happened next.

Why Perl?

Perl is well suited for  web scraping  because of its highly powerful RegEx and availability of CPAN modules .

In this session, we will deal with :

  • Mechanize(Perl Module),
  • Process spawning
  • Anonymous  scraping

Mechanize module : Mechanize is one of the main modules used, for stateful programmatic web browsing, used for automating interaction with websites. Mechanize supports performing a sequence of page fetches including following links and submitting forms. Each fetched page is parsed and its links and forms are extracted. A link or a form can be selected, form fields can be filled and the next page can be fetched. Mech also stores a history of the URLs you’ve visited, which can be queried and revisited. Usefull functions decribed in bottem

For more info:http://search.cpan.org/~petdance/WWW-Mechanize-1.62/

Sample Script

#!/usr/bin/perl -w
use WWW::Mechanize;
$url = 'http://chato.cl/research/crawling_thesis ';
$m = WWW::Mechanize-&gt;new();
$m-&gt;get($url);
$c = $m-&gt;content; #  Will display souce code of the above link
exit;

Usefull Function of mechanize module
my $mech = WWW::Mechanize->new();         #Creating new object of  Mechanize.
$mech->agent_alias(‘Linux Mozilla’);             #Creating a new agent like firfox
$mech->get(‘www.google.com’);                       #Download content in the link (www.google.com)
$mech->content;                                                     # This has the content of www.google.com link
$mech->submit_form                                            # for form submition
$mech->find_link(text =>’Next’)                      #Follow the link with text ‘Next’ there are so many options for this like regular expression ,class,etc

Process spawning  :
Most of the bots have a main process and a number of child processes. Main processes deal with creating child processes based on our requirement, while the child processes scrape our target locations simultanously.

Why Process spawning?
Process spawning is used simply for simultaneous scraping at different levels of a web site (i.e. at different page/sections etc.
It has a number of advantages like nitro boosting of scraping speed and easier management of server load.
In case the target is an e-commerce portal with a million section (like review page) with some pages or sections (or any other target)  missing. Here, the child process will simply die, without effecting the total crawling process, while the main continues with a new child and new section.
Anonymous scraping with TOR


Tor is a free software and an open network that helps in defending your site against a form of network surveillance known as traffic analysis. This surviellance threatens personal freedom, privacy, confidential business activities and relationships.
Tor is a network of virtual tunnels that allows people and groups to improve their privacy and security on the Internet. It also enables software developers to create new communication tools with built-in privacy features. Tor provides the foundation for a range of applications that allow organizations and individuals to share information over public networks without compromising their privacy.

For more info  please go through
http://www.torproject.org/docs/tor-doc-unix.html.en#polipo

VN:F [1.9.6_1107]
Rating: 9.3/10 (3 votes cast)
VN:F [1.9.6_1107]
Rating: +1 (from 1 vote)

Shameem Khalid Articles, linux, perl , , , ,

How To Install Red5 on CentOS

August 2nd, 2010

Red5 is an open source video sream server appllication which helps you to stream your video content across the web. Inorder to install and run red5, you need the java platform installed on your system. Using apache-ant or eclipse we can build and install Red5. Here apache-ant  method is described.

1)Using Apache-ant

a)install jdk 1.6 or higher and its curresponding deveopment package.

#yum install java-1.6.0-openjdk java-1.6.0-openjdk-devel

b)Install apache-ant

To build the red5 binary from source we need apache-ant. So install it.

#cd /usr/src

#wget http://archive.apache.org/dist/ant/binaries/apache-ant-1.8.0-bin.tar.bz2

#tar -xjf apache-ant-1.8.0-bin.tar.bz2

#mv apache-ant-1.8.0 /usr/local/ant

c)Set the enviroment variables for java, ant and java class path

export ANT_HOME=/usr/local/ant
export JAVA_HOME=/usr/lib/jvm/java
export PATH=$PATH:/usr/local/ant/bin
export CLASSPATH=.:$JAVA_HOME/lib/classes.zip

If you want this to be avilable for all users, append these lines to the file /etc/bashrc

d)Now Install subversion to downlaod the svn version from the googlecode.

#yum install subversion

e)Download and Install Red5

#svn checkout http://red5.googlecode.com/svn/java/server/trunk/ red5

#mv red5 /usr/local

#cd /usr/local/red5

#ant prepare

#ant build

Note: If you are building on CentOS 5.4 use “#ant dist” command instead of “#ant build”

This will take some time. So please be patient.

Finally you will see a line “Build successful” .That means your red5 installation is complete. Now copy the conf directory from dist/ to the current directory and now test your installation by runnig the script

#cp -r dist/conf .
#./red5.sh

Your Installation is ok if it shows “Installer service created” in the last. Now press ctrl+c to quit the process and go for the init script

f)Init Script

#vi /etc/init.d/red5

copy the following code to it.

#!/bin/bash
 PROG=red5
 RED5_HOME=/usr/local/red5
 DAEMON=$RED5_HOME/$PROG.sh PIDFILE=/var/run/$PROG.pid
# Source function library . /etc/rc.d/init.d/functions[ -r /etc/sysconfig/red5 ] && . /etc/sysconfig/red5RETVAL=0
case "$1" in
 start)
 echo -n $"Starting $PROG: "
 cd $RED5_HOME
 $DAEMON >/dev/null 2>/dev/null &
 RETVAL=$?
 if [ $RETVAL -eq 0 ]; then
 echo $! > $PIDFILE
 touch /var/lock/subsys/$PROG
 fi
 [ $RETVAL -eq 0 ] && success $"$PROG startup" || failure $"$PROG startup"
 echo
 ;;
 stop)
 echo -n $"Shutting down $PROG: "
 killproc -p $PIDFILE
 RETVAL=$?
 echo
 [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$PROG
 ;;
 restart)
 $0 stop
 $0 start
 ;;
 status)
 status $PROG -p $PIDFILE
 RETVAL=$?
 ;;
 *)
 echo $"Usage: $0 {start|stop|restart|status}"
 RETVAL=1 esac
exit $RETVAL

 
g)Start the service and test your server

#/etc/init.d/red5 start

Now you can open your browser enter http://[your_ip]:5080 and see your sever working. You can install the sample applications and view the demos.

VN:F [1.9.6_1107]
Rating: 4.3/10 (3 votes cast)
VN:F [1.9.6_1107]
Rating: +1 (from 1 vote)

Jaseer Articles, general, linux , ,