Archive

Archive for the ‘Articles’ Category

Deploying with Capistrano

December 21st, 2010

What is Capisrano?

Capistrano is a deployment tool and is popularly used for deploying rails applications to your servers. But its a good configuration tool too. You can setup your servers using capistrano. You can preform tasks like adding users, installing and configuring packages like apache, mysql or postfix, a xen vm etc on your desired network host from a central server(wherever you kept the configuration repository).

Puppet and Capistrano?

Well., you might have heard about puppet – another popular network host configuration tool. Great..Now it will be easy to understand capistrano. If you haven’t, dont worry, I will make it clear for you.

The puppet is working in such a way that a puppet server will be running on one host in the nerwork. And all the desired hosts would run a puppet client daemon. You can configure your client manully and can automate installation on your clients. Whenever you want to change the configuration just do it on the server. It will be updated to the client as the client puppet daemons updates from the server in each interval that you specify. Now you would note that the client daemon pulls the configuration from puppet server. But in capistrano the user pushes the configuration to the desired client when he needs it. Puppet uses “meta language” for the configurations. Whereas capistrano uses ruby. And you know that you got several modules and feautres with ruby and so you may can write your additions to cap if you are well versed with that. One more thing you should note that puppet does its dependency resolution automatically, but in capistrano, you may have to handle the dependency manually in some cases. But as I said earlier you can overcome this if you know well and I would say it may be little difficult to go to that extent. But when you have done with a perfect configuration you feel it greatly helpful.

Installation and Configuration

To install just enter the command:

$gem install capistrano

Provided you have rubygems installed on your machine. If not, install it with apt or yum.

$gem install deprec

Note: I have found several docs without mentioning this..and it made my cap commands haden’t worked till I found this and installed..

Now create a repository as the source (central configuration repo) using svn. Svn is common version control system that is used in cordination with the cap. If you like git go with that.

$svnadmin create sparkconf

Sparkconf is the repository name where you will keep all the configuration of your servers. You can use your own repository name.

Configure this repo for access to the desired users. I wont go with svn details because its beyond the context of our topic.

now you can checkout your repo

$svn co <ur-repo-ulr> sparkconf
$cd sparkconf
$capify .

This “capify” command will initiate your cap repo. It will create a file named Capfile and a deploy.rb file in a directory named config. In capfile you can see that there is a statement to load deploy.rb file. So you can define your own tasks in Capfile or in deploy.rb. But usually this is not the convention. You would leave your Capfile without any changes. And you may specify some information about your setting in the deploy.rb file. The sample deploy.rb which is generated automatically, needs some explanation.

<br />
set :application, &quot;agileblazeworks&quot;<br />
set :repository,  &quot;report&quot;</p>
<p>set :scm, :subversion<br />
# Or: `accurev`, `bzr`, `cvs`, `darcs`, `git`, `mercurial`, `perforce`, `subversion` or `none`</p>
<p>role :web, &quot;rails.spark.com&quot;                                 # Your HTTP server, Apache/etc<br />
role :app, &quot;rails.spark.com&quot;                                 # This may be the same as your `Web` server<br />
role :db,  &quot;rails.spark.com&quot;, :primary = true    # This is where Rails migrations will run<br />
#role :db,  &quot;your slave db-server here&quot;</p>
<p># If you are using Passenger mod_rails uncomment this:<br />
# if you're still using the script/reapear helper you will need<br />
# these http://github.com/rails/irs_process_scripts</p>
<p># namespace :deploy do<br />
#   task :start do ; end<br />
#   task :stop do ; end<br />
#   task :restart, :roles =&amp;gt; :app, :except =&amp;gt; { :no_release =&amp;gt; true } do<br />
#     run &quot;#{try_sudo} touch #{File.join(current_path,'tmp','restart.txt')}&quot;<br />
#   end<br />
# end<br />

Dont worry about the configuration here. Most of them are needed if you are deploying a rails application for a web server. After breifing this we will go for some other configuration options that you are really looking for…to install and configure packages on your hosts.

If your application (means your rails application) is not separated into application, web and database servers, you can either set them to be the same value; or comment out, or remove the one you do not require. The “:primary => true” part of the role definitions allows you to have more than one database server. If you dont have two skipp this primary option. If, for example when deploying a Rails application you only wanted db1 to run migrations, in the first example both might. Essentially when using the Rails deployment recipes, the :primary option defines where database migrations are run. Similar attributes include :no_release often used for the :web role by some of the recipes in circulation to decide which servers should not have the code checked out to them. Attributes like these are arbitrary and you can define some of your own, and use them to filter more precisely where your own tasks run.

You may add these options to the deploy.rb file

<br />
set :user, 'jaseer'              #This is the user you have on the target machine. Capistrano try to login to the target machine using this account.<br />
ssh_options[:keys] = %w(/home/users/mylocalname/.ssh/jas_rsa)<br />
set :use_sudo, true            #if you want to append all commands with sudo.<br />
set :password, &quot;yourpassword&quot;      #You can login using key (above) or with password. Use either key or password..not both.</p>
<p>default_run_options[:pty] = true #this is really helpful. If you dont have this you will struggle in runnig sudo tasks. I had lost some time searching around this.<br />

Some cap commands

$ cap -h

This will give out a list of all the options it accepts.

$ cap -H

It will give you a description of each option.

Next, let’s ask Capistrano what all tasks it will do. Capistrano comes bundled with several built-in tasks. You can also write your own to automate workflows of your own. For now, let’s see what tasks Capistrano knows:

$ cap -T

And finally, to get a detailed description of a command, type

$cap -e <task>

Example

Suppose you want install and configure nagios on one of your servers.

cap deprec:nagios:install HOSTS=monitor.spark.com

This command will install nagios on the host monitor.blocksglobal.com (You can use ip also). Iif you want to override the user in deploy.rb, or any other files, use the option USER=<username> option. This will ask your sudo password, as you know for some task you may need the administrator privileges.

Now generate configuration files

#cap deprec:nagios:config_gen

The configuration files are created in your localrepo not in the installed server. A directry tree is created under sparkconf/config/  with the name nagios. It will contain all configuration files of nagios like hosts.cfg, service.cfg etc. You can change each  according to your needs. Update svn. Push those configuration to your client.

$cd config/nagios

Change and configure the files as your needs, commit the changes and update svn. Go back to the root of your repository, here it is sparkconf. Now push the config to your nagios server.

$cap deprec:nagios:config HOSTS=monitor.spark.com

Defining Tasks

Now let me explain how to define tasks. Often I would define my own tasks in config/*/recipes.rb for example config/nagios/recipes.rb.  * Can be any thing like postfix, mysql as you generate the config directory for them using config_gen.

<br />
namespace : one do<br />
 task :default do<br />
    test<br />
    one.test<br />
    two.test<br />
  end<br />
  task :test do<br />
    puts &quot;Test One Successful!&quot;<br />
  end<br />
end<br />
namespace :two do<br />
  task :test do<br />
    puts &quot;Test Two Successful&quot;<br />
  end<br />
end<br />

Here these are the available commands I can use with cap…

$cap one
$cap one:default
$cap one:test
$cap two

Hope you understood how to define them. Note in how many ways I called the task “test” inside the task default. This is easy right? Now go, create your repo and try your cap tasks..I will stop with a final simple example:

<br />
task :backup_database, :roles =&gt; :db, : only =&gt; { :backup =&gt; true } do<br />
 run &quot;#{sudo} mysqldump ... &gt; /tmp/backup.sql&quot;<br />
 run &quot;#{sudo} bzip2 /tmp/backup.sql&quot;<br />
 run &quot;scp /tmp/backup.sql.bz2 offsite.host:/u/backups&quot;<br />
 run &quot;#{sudo} rm /tmp/backup.sql.bz2&quot;<br />
end<br />

Next time I will write one more article on deploying rails app using cap.. Have fun..bye

VN:F [1.9.6_1107]
Rating: 8.0/10 (1 vote cast)
VN:F [1.9.6_1107]
Rating: 0 (from 0 votes)

Jaseer Articles, linux , , , ,

Load balancing in Wowza Media Server

August 3rd, 2010

Wowza Media Server is a Video Stream Server used by popular CDN providers to provide video content across the internet. It has come with a large variety of features and several streaming options. And one of the attractive feature provided by wowza is that, it allows clustering like feature so that multiple servers can be configured to provide a video stream to the clients. In such a setup one of the server will act as a loadbalancing server and the others will act as the loadbalancing edge/client servres. If a client requests a video stream then a load balancer server redirect it to the least loaded wowza server. This loadbalancing feature is usually used while publishing live streams becuase the number of concurrent connections to the server providing live stream will be high and a load balancing setup can easily manage it. Here we will discuss the load balancing setup for a live stream proess.

To employ loadbalancing setup in live streaming first we have to setup live streaming repeater configuration available with wowza. This is because the live stream will be actually published to only one server and we need to duplicate among a no of stream servers as our need. So there will be live repeater origin server and several live repeater edge servers. When the player will request the contentfrom an edge server and the edge server will maintain a single connection per-unique stream to the origin. Origin and edge configuration is an application level configuration. A single Wowza Server instance can be configured as an origin for one application and an edge for another.

1) Configuring liverepeater-origin server
  • Create a folder named [install-dir]/applications/liverepeater.
  • Create a folder named [install-dir]/conf/liverepeater and copy the file [install-dir]/conf/Application.xml into this new folder.
  • Edit the newly copied Application.xml file and make the following changes:

a) Change the Streams/StreamType to liverepeater-origin

b. Change the LiveStreamPacketizers to: cupertinostreamingpacketizer,smoothstreamingpacketizer

2)Configuring edge server

Follow these steps to configure each of the edge servers

  • Create a folder named [install-dir]/applications/liverepeater.
  • Create a folder named [install-dir]/conf/liverepeater and copy the file [install-dir]/conf/Application.xml into this new folder.
  • Edit the newly copied Application.xml file and make the following changes.

a)Change the Streams/StreamType to liverepeater-edge (you can use the liverepeater-edge-lowlatency stream type if low latency is important, this will add extra load to the server).

b)Change the LiveStreamPacketizers to: cupertinostreamingrepeater,smoothstreamingrepeater

c)Uncomment the Repeater/OriginURL section and set OriginURL to rtmp URL of the origin server. For example if the origin server uses the domain name      origin.mycompany.com, this value should be set to:


<Repeater>

         <OriginURL>rtmp://origin.mycompany.com</OriginURL>

         <QueryString></QueryString>

</Repeater>

Note: Let us assume origin.mycompany.com is the origin server here

3)Configring the Loadbalancer server

  • First Download the loadbalancing module the from the following link of wowza forums. Unzip the downloaded file

http://www.wowzamedia.com/forums/showthread.php?t=4637

  • Copy the file lib/wms-plugin-loadbalancer.jar from this zip archive to the [install-dir]/lib/ folder of Wowza Media Server 2
  • Copy the file conf/crossdomain.xml from this zip archive to the [install-dir]/conf/ folder of Wowza Media Server 2.
  • Edit [install-dir]/conf/Server.xml and make the following changes:

Add the following ServerListener entry to the <ServerListeners> list:

    <ServerListener>
           	  <BaseClass>com.wowza.wms.plugin.loadbalancer.ServerListenerLoadBalancerListener</BaseClass>
    </ServerListener>

Add the following properties to the <properties> section of the bottom of the server.xml file

      <Property>
               <Name>loadBalancerListenerKey</Name>
	      <Value>023D4FB4IS83</Value>
      </Property>
      <Property>
	      <Name>loadBalancerListenerIpAddress</Name>
	      <Value>*</Value>
      </Property>
      <Property>
	     <Name>loadBalancerListenerPort</Name>
	     <Value>1934</Value>
	     <Type>Integer</Type>
      </Property>
      <Property>
	      <Name>loadBalancerListenerRedirectorClass</Name>
	      <Value>com.wowza.wms.plugin.loadbalancer.LoadBalancerRedirectorConcurrentConnects</Value>
      </Property>
      <Property>
	      <Name>loadBalancerListenerMessageTimeout</Name>
	      <Value>5000</Value>
	      <Type>Integer</Type>
      </Property>
  • Edit [install-dir]/conf/VHost.xml and add the following HostPort/HTTPProvider XML snippet just before the HTTPProvider definition for com.wowza.wms.http.HTTPServerVersion:
      <HTTPProvider>
                   <BaseClass>com.wowza.wms.plugin.loadbalancer.HTTPLoadBalancerRedirector</BaseClass>
	           <RequestFilters>*loadbalancer</RequestFilters>
	           <AuthenticationMethod>none</AuthenticationMethod>
	           <Properties>
                              <Property>
		          	  <Name>enableServerInfoXML</Name>
                   		  <Value>true</Value>
		                  <Type>Boolean</Type>
		           </Property>
	         </Properties>
     </HTTPProvider>

4)To setup an edge servers in load balancing

Do the first two steps as done for the load balacer server

  • Edit [install-dir]/conf/Server.xml and make the following changes:

Add the following ServerListener entry to the <ServerListeners> list:


     <ServerListener>
	           <BaseClass>com.wowza.wms.plugin.loadbalancer.ServerListenerLoadBalancerSender</BaseClass>
     </ServerListener>

Add the following properties to the <Properties> section at the bottom of Server.xml:


   <Property>
	    <Name>loadBalancerSenderTargetPath</Name>
	    <Value>${com.wowza.wms.AppHome}/conf/loadbalancertargets.txt</Value>
   </Property>
   <Property>
	    <Name>loadBalancerSenderRedirectAddress</Name>
	    <Value>[redirect-address]</Value>
   </Property>
   <Property>
            <Name>loadBalancerSenderMonitorClass</Name>
	    <Value>com.wowza.wms.plugin.loadbalancer.LoadBalancerMonitorDefault</Value>
   </Property>
   <Property>
	    <Name>loadBalancerSenderMessageInterval</Name>
	    <Value>2500</Value>
	    <Type>Integer</Type>
   </Property>

Where [redirect-address] is the external ip address or domain name of this machine. This address will be used when redirecting to this edge server. When using this system on EC2 you can set the [redirect-address] to   ${com.wowza.amazonaws.ec2.AWSEC2_METADATA_PUBLIC_IPV4} and upon server startup it will use the public ip address of the server for this value.

  • Create the file [install-dir]/conf/loadbalancertargets.txt using a text editor and enter the following two lines (the first line is a comment):

# [load-balancer-ip-address],[load-balancer-port],[encryption-key]

[load-balancer-ip-address],1934,023D4FB4IS83

  Where [load-balancer-ip-address] is the ip  address or domain name of the load balancer.

This configurations uses UDP port 1934 for communication between the edge servers and the load balancer. Be sure this port is open on your firewall. All communication between the edge server and the load balancer is encrypted and signed. The encryption key is set on the load balancer server using the loadBalancerListenerKey property and in the loadbalancertargets.txt file on the edge servers. These keys must match. An edge server can communicate with multiple load balancers by adding additional lines to the loadbalancertargets.txt file.

You can now startup the load balancer and multiple edge servers. If functioning properly, the edge servers will update the load balancer every 2.5 seconds with status and load information. You can get information from the load balancer in regards to which edge servers are currently registered and their status by opening a web browser and entering the following url:

 http://[load-balancer-ip-address]:1935/loadbalancer?serverInfoXML

5)Now Configure a redirect application to redirect the connection requests to the least loaded server.
  • Create the folder [install-dir]/applications/redirect.
  • Create the folder [install-dir]/conf/redirect and copy the file [install-dir]/conf/Application.xml into this new folder.
  • Create the folder [install-dir]/conf/redirect and copy the file [install-dir]/conf/Application.xml into this new folder.
      <Module>
	    <Name>ModuleLoadBalancerRedirector</Name>
	    <Description>ModuleLoadBalancerRedirector</Description>
            <Class>com.wowza.wms.plugin.loadbalancer.ModuleLoadBalancerRedirector</Class>
      </Module>
  • Add the following properties the properties section at the bottom of the Application.xml file:
   <Property>
             <Name>redirectAppName</Name><code>
             <Value>[application-name]</Value>
    </Property>
    <!--
    <Property>
	     <Name>redirectPort</Name>
	     <Value>[redirect-port]</Value>
    </Property>
    -->
    <!--
    <Property>
	     <Name>redirectScheme</Name>
              <Value>rtmp</Value>
    </Property>
    -->
    <Property>
	     <Name>redirectOnConnect</Name>
	     <Value>true</Value>
	     <Type>Boolean</Type>
    </Property>

Where [application-name] is the name of the application you wish to redirect to on the edge server and [redirect-port] is the port to redirect to (such as port 1935 or port 80). The redirectPort and redirectScheme are commented out so that the system will use the same scheme and port used to connect to the load balancer to connect to the edge server. This will work better when using any type of protocol (rtmp to rtmpt) or port rollover scheme.

VN:F [1.9.6_1107]
Rating: 8.7/10 (10 votes cast)
VN:F [1.9.6_1107]
Rating: +2 (from 4 votes)

Jaseer Articles, general , , ,

Web Scraping : A basic know-how.

August 2nd, 2010

A Web crawler is a computer program that browses the World Wide Web in a methodical, automated manner or in an orderly fashion. Other terms for Web crawlers are ants, automatic indexers, bots, web spiders, web robots, etc. The process is termed “web crawling”, and most site engines use it as a means to provide up-to-date data, in order to create a copy of all pages that have been visited. These are later processed, and the search engine will index the downloaded pages.
This helps in :

  • faster search
  • automating maintenance task on a web site
  • gathering specific types of information from websites

The bot starts with seeds, which are a list of URLs to visit. Once the “crawler” is on one of the listed URLs, the hyperlinks in that page are identified and added to the “crawl frontier” which is the set of URLs that are to be visited. These are later visited according to a pre-defined set of policies.

Web Crawlers can be developed using any language : perl, python, java, asp,php etc. Among these, we chose perl to develop a web crawler. Lets see what happened next.

Why Perl?

Perl is well suited for  web scraping  because of its highly powerful RegEx and availability of CPAN modules .

In this session, we will deal with :

  • Mechanize(Perl Module),
  • Process spawning
  • Anonymous  scraping

Mechanize module : Mechanize is one of the main modules used, for stateful programmatic web browsing, used for automating interaction with websites. Mechanize supports performing a sequence of page fetches including following links and submitting forms. Each fetched page is parsed and its links and forms are extracted. A link or a form can be selected, form fields can be filled and the next page can be fetched. Mech also stores a history of the URLs you’ve visited, which can be queried and revisited. Usefull functions decribed in bottem

For more info:http://search.cpan.org/~petdance/WWW-Mechanize-1.62/

Sample Script

#!/usr/bin/perl -w
use WWW::Mechanize;
$url = 'http://chato.cl/research/crawling_thesis ';
$m = WWW::Mechanize-&gt;new();
$m-&gt;get($url);
$c = $m-&gt;content; #  Will display souce code of the above link
exit;

Usefull Function of mechanize module
my $mech = WWW::Mechanize->new();         #Creating new object of  Mechanize.
$mech->agent_alias(‘Linux Mozilla’);             #Creating a new agent like firfox
$mech->get(‘www.google.com’);                       #Download content in the link (www.google.com)
$mech->content;                                                     # This has the content of www.google.com link
$mech->submit_form                                            # for form submition
$mech->find_link(text =>’Next’)                      #Follow the link with text ‘Next’ there are so many options for this like regular expression ,class,etc

Process spawning  :
Most of the bots have a main process and a number of child processes. Main processes deal with creating child processes based on our requirement, while the child processes scrape our target locations simultanously.

Why Process spawning?
Process spawning is used simply for simultaneous scraping at different levels of a web site (i.e. at different page/sections etc.
It has a number of advantages like nitro boosting of scraping speed and easier management of server load.
In case the target is an e-commerce portal with a million section (like review page) with some pages or sections (or any other target)  missing. Here, the child process will simply die, without effecting the total crawling process, while the main continues with a new child and new section.
Anonymous scraping with TOR


Tor is a free software and an open network that helps in defending your site against a form of network surveillance known as traffic analysis. This surviellance threatens personal freedom, privacy, confidential business activities and relationships.
Tor is a network of virtual tunnels that allows people and groups to improve their privacy and security on the Internet. It also enables software developers to create new communication tools with built-in privacy features. Tor provides the foundation for a range of applications that allow organizations and individuals to share information over public networks without compromising their privacy.

For more info  please go through
http://www.torproject.org/docs/tor-doc-unix.html.en#polipo

VN:F [1.9.6_1107]
Rating: 9.3/10 (3 votes cast)
VN:F [1.9.6_1107]
Rating: +1 (from 1 vote)

Shameem Khalid Articles, linux, perl , , , ,

How To Install Red5 on CentOS

August 2nd, 2010

Red5 is an open source video sream server appllication which helps you to stream your video content across the web. Inorder to install and run red5, you need the java platform installed on your system. Using apache-ant or eclipse we can build and install Red5. Here apache-ant  method is described.

1)Using Apache-ant

a)install jdk 1.6 or higher and its curresponding deveopment package.

#yum install java-1.6.0-openjdk java-1.6.0-openjdk-devel

b)Install apache-ant

To build the red5 binary from source we need apache-ant. So install it.

#cd /usr/src

#wget http://archive.apache.org/dist/ant/binaries/apache-ant-1.8.0-bin.tar.bz2

#tar -xjf apache-ant-1.8.0-bin.tar.bz2

#mv apache-ant-1.8.0 /usr/local/ant

c)Set the enviroment variables for java, ant and java class path

export ANT_HOME=/usr/local/ant
export JAVA_HOME=/usr/lib/jvm/java
export PATH=$PATH:/usr/local/ant/bin
export CLASSPATH=.:$JAVA_HOME/lib/classes.zip

If you want this to be avilable for all users, append these lines to the file /etc/bashrc

d)Now Install subversion to downlaod the svn version from the googlecode.

#yum install subversion

e)Download and Install Red5

#svn checkout http://red5.googlecode.com/svn/java/server/trunk/ red5

#mv red5 /usr/local

#cd /usr/local/red5

#ant prepare

#ant build

Note: If you are building on CentOS 5.4 use “#ant dist” command instead of “#ant build”

This will take some time. So please be patient.

Finally you will see a line “Build successful” .That means your red5 installation is complete. Now copy the conf directory from dist/ to the current directory and now test your installation by runnig the script

#cp -r dist/conf .
#./red5.sh

Your Installation is ok if it shows “Installer service created” in the last. Now press ctrl+c to quit the process and go for the init script

f)Init Script

#vi /etc/init.d/red5

copy the following code to it.

#!/bin/bash
 PROG=red5
 RED5_HOME=/usr/local/red5
 DAEMON=$RED5_HOME/$PROG.sh PIDFILE=/var/run/$PROG.pid
# Source function library . /etc/rc.d/init.d/functions[ -r /etc/sysconfig/red5 ] && . /etc/sysconfig/red5RETVAL=0
case "$1" in
 start)
 echo -n $"Starting $PROG: "
 cd $RED5_HOME
 $DAEMON >/dev/null 2>/dev/null &
 RETVAL=$?
 if [ $RETVAL -eq 0 ]; then
 echo $! > $PIDFILE
 touch /var/lock/subsys/$PROG
 fi
 [ $RETVAL -eq 0 ] && success $"$PROG startup" || failure $"$PROG startup"
 echo
 ;;
 stop)
 echo -n $"Shutting down $PROG: "
 killproc -p $PIDFILE
 RETVAL=$?
 echo
 [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$PROG
 ;;
 restart)
 $0 stop
 $0 start
 ;;
 status)
 status $PROG -p $PIDFILE
 RETVAL=$?
 ;;
 *)
 echo $"Usage: $0 {start|stop|restart|status}"
 RETVAL=1 esac
exit $RETVAL

 
g)Start the service and test your server

#/etc/init.d/red5 start

Now you can open your browser enter http://[your_ip]:5080 and see your sever working. You can install the sample applications and view the demos.

VN:F [1.9.6_1107]
Rating: 4.3/10 (3 votes cast)
VN:F [1.9.6_1107]
Rating: +1 (from 1 vote)

Jaseer Articles, general, linux , ,

What is Video Streaming

July 13th, 2010

Video stream servers are meant to provide the video content across internet in reliable and generic way so that any client can access easily. They uses diffrent protocols other than http namely rtsp, rtmp, rtp etc. And these protocols mainly uses udp. A basic video streaming system contains 3 components. A video publishing side, The video streaming server and the client side player (usually flash player embedded in the webpage). The video publishing system will upload a live or static video content to the Video Stream Server and the video stream sever will broadcast (means streaming)  that uploaded video such that the clients can view the videos without bothering about any codecs. The video uploading system should have an encoder to publish live contents to the server. The encoder does the process of transcoding to provide the content to the desired format of the stream server.

Major stream server applications are Wowza Media Server, Adobe Flash Media Server, Red5:open source flash server, Darwin Streaming server (open source) etc.

Working
Actual work flow when a web client is accessing stream video from a website is deiscribed in the figure. When the user clicks on a particular video icon the web server passes a request to the stream server. It will also notify the information about the client side  player (usually it will be a flash player application). Because the if it is a flash client then the request sent will be rtmp (Real Time Messaging Protocol ) protocol based. The stream server will stream (send the video packets) video based on rtmp protocol for flash players. For client players like vlc, apple quick-time etc  the protocol used is rtsp (Real Time Streaming Protocol).

A video streaming setup contains 3 components they are

  1. Video Stream server
  2. Web server for providing a web interface to the clients
  3. The flash player embedded on the web page (Or stand alone players like vlc, quicktime etc if not accessing through the site).

If the video server streams a live video then one publishing site will be there which will provide the  the live video source. Figure 2 illustrates such a video stream server setup. A video Streaming server application would be running on the Streaming server. The broadcaster would use an live  encoding tool (eg: flash live media encoder). If the client is accessing the video on an html web page the player might be adobe flash player. Flash offers online broadcasters a platform rich with unmatchable flexibility and superior quality video delivery. Flash has fast become the standard for delivery of rich-media over the internet. TV and Video Broadcasters and a wide variety of content creators around the globe have chosen Flash as their No. 1 streaming format.

Live video stream setup

If the video server streams a live video then one publishing site will be there which will provide the the live video source. Figure 2 illustrates such a video stream server setup. A video Streaming server application would be running on the Streaming server. The broadcaster would use an live encoding tool (eg: flash live media encoder). If the client is accessing the video on an html web page

the player might be adobe flash player. Flash offers online broadcasters a platform rich with unmatchable flexibility and superior quality video delivery. Flash has fast become the standard for delivery of rich-media over the internet. TV and Video Broadcasters and a wide variety of content creators around the globe have chosen Flash as their No. 1 streaming format.

VN:F [1.9.6_1107]
Rating: 4.8/10 (4 votes cast)
VN:F [1.9.6_1107]
Rating: +2 (from 2 votes)

Jaseer Articles, general , , , ,

Command-line tips: Understanding Dig

September 24th, 2009

dig is a command-line tool for querying DNS name servers for information about host addresses, mail exchanges, name servers, and related information.

Understanding the default output

The most typical, simplest query is for a single host. By default, however, dig is pretty verbose. You probably don’t need all the information in the default output, but it is probably worth knowing what it is. Below is an annotated query.

This article explains you how to do the data recovery from a crashed windows-plesk server.


$ dig www.sparksupport.com

That is the command-line invocation of dig I used

; <<>> DiG 9.2.3 <<>> www.sparksupport.com

;; global options: printcmd

The opening section of dig’s output tells us a little about itself (version 9.2.3) and the global options that are set (in this case, printcmd). This part of the output can be quelled by using the +nocmd option, but only if it is the very first argument on the command line (even preceeding the host you are querying).


;; Got answer:

;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: NOERROR, id: 43071

;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 3

Here, dig tells us some technical details about the answer received from the DNS server. This section of the output can be toggled using the +[no]comments option, but beware that disabling the comments also turns off many section headers.


;; QUESTION SECTION:

;www.sparksupport.com. IN A

In the question section, dig reminds us of our query. The default query is for an Internet address (A). You can turn this output on or off using the +[no]question option.


;; ANSWER SECTION:

www.sparksupport.com. 600 IN A 203.23.184.88

Finally, we get our answer: the address of www.sparksupport.com is 204.152.184.88. I don’t know why you’d ever want to turn off the answer, but you can toggle this section of the output using the +[no]answer option.


;; AUTHORITY SECTION:

sparksupport.com. 2351 IN NS ns1.nis.tc.org.

sparksupport.com. 2351 IN NS ns1.gnac.com.

sparksupport.com. 2351 IN NS ns2.nis.tc.org.

The authority section tells us what DNS servers can provide an authoritative answer to our query. In this example, sparksupport.com has three name servers. You can toggle this section of the output using the +[no]authority option.


;; ADDITIONAL SECTION:

ns1.gnac.com. 171551 IN A 203.23.34.21

ns-int.sparksupport.com. 2351 IN A 211.52.18.65

ns-int.sparksupport.com. 2351 IN AAAA 2001:4f8:0:2::15

The final section of the default output contains statistics about the query; it can be toggled with the +[no]stats option.

Some useful options with dig

dig will let you perform any valid DNS query, the most common of which are A (the IP address), TXT (text annotations), MX (mail exchanges), NS name servers, or the omnibus ANY.


# get the address(es) for yahoo.com

dig yahoo.com A +noall +answer

# get a list of yahoo's mail servers

dig yahoo.com MX +noall +answer

# get a list of DNS servers authoritative for yahoo.com

dig yahoo.com NS +noall +answer

# get all of the above

dig yahoo.com ANY +noall +answer

#Short answer

dig sparksupport.com +short

#To get the TTL values

dig +nocmd sparksupport.com mx +noall +short

#To get a long answer

dig +nocmd sparksupport.com any +multiline +noall +answer

#To reverselookup

dig -x 216.109.112.135 +short

To bulk lookups # do full lookups for a number of hostnames

#dig -f /path/to/host-list.txt

#the same, with more focused output

dig -f /path/to/host-list.txt +noall +answer

Tracing dig's path

dig sparksupport.com +trace

How to interpret TTL value

If you ask your local DNS server for an Internet address, the server figures out where to find an authoritative answer and then asks for it. Once the server receives an answer, it will keep the answer in a local cache so that if you ask for the same address again a short time later, it can give you the answer quickly rather than searching the Internet for it all over again.
When domain administrators configure their DNS records, they decide how long the records should remain in remote caches. This is the TTL number (usually expressed in number of seconds).

When domain administrators configure their DNS records, they decide how long the records should remain in remote caches. This is the TTL number (usually expressed in number of seconds).

For example, as of this writing, the TTL for the MX records for the gmail.com domain is 300 seconds. The gmail.com admins are asking that remote servers cache their MX records for no more than five minutes. So when you first ask for that record set, dig will report a TTL of 300.


$ dig +nocmd gmail.com MX +noall +answer

gmail.com. 300 IN MX 20 gsmtp57.google.com.

gmail.com. 300 IN MX 10 gsmtp171.google.com.

If you ask a few seconds later, you’ll see the TTL number reduced by approximately the number of seconds you waited to ask again.


$ dig +nocmd gmail.com MX +noall +answer

gmail.com. 280 IN MX 10 gsmtp171.google.com.

gmail.com. 280 IN MX 20 gsmtp57.google.com.

If your timing is good, you can catch the record at the very end of its life.


$ dig +nocmd gmail.com MX +noall +answer

gmail.com. 1 IN MX 10 gsmtp171.google.com.

gmail.com. 1 IN MX 20 gsmtp57.google.com.

After that, the DNS server you are querying will “forget” the answer to that question, so the whole cycle will start over again (in this example, at 300 seconds) the next time you perform that query.

VN:F [1.9.6_1107]
Rating: 10.0/10 (1 vote cast)
VN:F [1.9.6_1107]
Rating: +2 (from 2 votes)

Shijil T S Articles, linux

System Admin Tools Part -1: Basic linux commands and tools used by an experienced sytem admin

May 18th, 2009

FREE

First of all lets learn how to find out what is chocking or gobbling up the server resource. You can get the memory usage by issuing the command “free -m”


root@spark [~]# free -m

             total       used       free     shared    buffers     cached

Mem:          4051       3052        999          0        249       1294

-/+ buffers/cache:       1508       2543

Swap:         4000          1       3999

The top row ‘used’ (3052) value will almost always nearly match the top row mem value (4051). Since Linux likes to use any spare memory to cache disk blocks (1294).

The key figure to look at is the buffers/cache row used value (1508). This is how much space your applications are currently using. For best performance, this number should be less than your total (4051) memory.

VMSTAT

vmstat helps you to see, among other things, if your server is swapping. Take a look at the following run of vmstat doing a one second refresh for two iterations.


root@spark [~]# vmstat 1 2

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa

 0  0   1172 1689332 333588 663092    0    0    19   113    1     2  3  1 95  1

 0  0   1172 1690320 332920 663100    0    0   352   256  355   681  5  3 91  2

The first row shows your server averages. The si (swap in) and so (swap out) columns show if you have been swapping (i.e. needing to dip into ‘virtual’ memory) in order to run your server’s applications. The si/so numbers should be 0 (or close to it). Numbers in the hundreds or thousands indicate your server is swapping heavily. This consumes a lot of CPU and other server resources and you would get a very significant benefit from adding more memory to your server.

Some other columns of interest: The r (runnable) b (blocked) and w (waiting) columns help see your server load. Waiting processes are swapped out. Blocked processes are typically waiting on I/O. The runnable column is the number of processes trying to something. These numbers combine to form the ‘load’ value on your server. Typically you want the load value to be one or less per CPU in your server.

The bi (bytes in) and bo (bytes out) column show disk I/O (including swapping memory to/from disk) on your server.
The us (user), sy (system) and id (idle) show the amount of CPU your server is using. The higher the idle value, the better.

PS

This command is used to know all the processes running in the server. It can be also used to find out process which is using most of the memory and cpu.

To find out top 3 memory consuming processes.


ps -auxf | sort -nr -k 4 | head -3

To find out top 3 cpu consuming processes


ps -auxf | sort -nr -k 3 | head -3

TOP

Say the system is slow and you want to find out who is gobbling up all the CPU and/or memory. To display the top processes, you use the command top.

Note that unlike other commands, top does not produce an output and sits still. It refreshes the screen to display new information. So, if you just issue top and leave the screen up, the most current information is always up. Top runs until you press “q” to quit top.


$ top

18:46:13  up 11 days, 21:50,  5 users,  load average: 0.11, 0.19, 0.18

151 processes: 147 sleeping, 4 running, 0 zombie, 0 stopped

CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle

           total   12.5%    0.0%    6.7%   0.0%     0.0%    5.3%   75.2%

Mem:  1026912k av,  999548k used,   27364k free,       0k shrd,  116104k buff

                    758312k actv,  145904k in_d,   16192k in_c

Swap: 2041192k av,  122224k used, 1918968k free                  590140k cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND

  451 spark    15   0  6044 4928  4216 S     0.1  0.4   0:20   0 tnslsnr

 8991 spark    15   0  1248 1248   896 R     0.1  0.1   0:00   0 top

    1 root      19   0   440  400   372 S     0.0  0.0   0:04   0 init

    2 root      15   0     0    0     0 SW    0.0  0.0   0:00   0 keventd

    3 root      15   0     0    0     0 SW    0.0  0.0   0:00   0 kapmd

    4 root      34  19     0    0     0 SWN   0.0  0.0   0:00   0 ksoftirqd/0

    7 root      15   0     0    0     0 SW    0.0  0.0   0:01   0 bdflush

    5 root      15   0     0    0     0 SW    0.0  0.0   0:33   0 kswapd

    6 root      15   0     0    0     0 SW    0.0  0.0   0:14   0 kscand

    8 root      15   0     0    0     0 SW    0.0  0.0   0:00   0 kupdated

    9 root      25   0     0    0     0 SW    0.0  0.0   0:00   0 mdrecoveryd

... output snipped ...

Let’s examine the different types of information produced.

The first line:


18:46:13  up 11 days, 21:50,  5 users,  load average: 0.11, 0.19, 0.18

shows the current time (18:46:13), that system has been up for 11 days; that the system has been working for 21 hours 50 seconds. The load average of the system is shown (0.11, 0.19, 0.18) for the last 1, 5 and 15 minutes respectively. (By the way, you can also get this information by issuing the uptime command.)

If the load average is not required, press the letter “l” (lowercase L); it will turn it off. To turn it back on press l again. The second line: 151 processes: 147 sleeping, 4 running, 0 zombie, 0 stopped shows the number of processes, running, sleeping, etc. The third and fourth lines:


CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle

           total   12.5%    0.0%    6.7%   0.0%     0.0%    5.3%   75.2%

show the CPU utilization details. The above line shows that user processes consume 12.5% and system consumes 6.7%. The user processes include the Oracle processes. Press “t” to turn these three lines off and on. If there are more than one CPU, you will see one line per CPU.

The next two lines:


Mem: 1026912k av, 1000688k used, 26224k free, 0k shrd, 113624k buff

758668k actv, 146872k in_d, 14460k in_c Swap: 2041192k av, 122476k

used, 1918716k free 591776k cached

show the memory available and utilized. Total memory is “1026912k av”, approximately 1GB, of which only 26224k or 26MB is free. The swap space is 2GB; but it’s almost not used. To turn it off and on, press “m”.

The rest of the display shows the processes in a tabular format. Here is the explanation of the columns:

Column Description

PID The process ID of the process

USER The user running the process

PRI The priority of the process

NI The nice value: The higher the value, the lower the priority of the task

SIZE Memory used by this process (code+data+stack)

RSS The physical memory used by this process

SHARE The shared memory used by this process

STAT

The status of this process, shown in code. Some major status codes are:

R Running

S Sleeping

Z Zombie

T Stopped

You can also see second and third characters, which indicate:

W Swapped out process

N positive nice value

%CPU The percentage of CPU used by this process

%MEM The percentage of memory used by this process

TIME The total CPU time used by this process

CPU If this is a multi-processor system, this column indicates the ID of the CPU this process is running on.

COMMAND The command issued by this process

While the top is being displayed, you can press a few keys to format the display as you like. Pressing the uppercase M key sorts the output by memory usage. (Note that using lowercase m will turn the memory summary lines on or off at the top of the display.) This is very useful when you want to find out who is consuming the memory. Here is sample output:


PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND

31903 spark    15   0 75760  72M 72508 S     0.0  7.2   0:01   0 ora_smon_PRODB2

31909 spark    15   0 68944  66M 64572 S     0.0  6.6   0:03   0 ora_mmon_PRODB2

31897 spark    15   0 53788  49M 48652 S     0.0  4.9   0:00   0 ora_dbw0_PRODB2

Now that you learned how to interpret the output, let’s see how to use command line parameters.

The most useful is -d, which indicates the delay between the screen refreshes. To refresh every second, use top -d 1.

The other useful option is -p. If you want to monitor only a few processes, not all, you can specify only those after the -p option. To monitor processes 13609, 13608 and 13554, issue: top -p 13609 -p 13608 -p 13554
This will show results in the same format as the top command, but only those specific processes.

SKILL & SNICE

From the previous discussion you learned how to identify a CPU consuming resource. What if you find that a process is consuming a lot of CPU and memory, but you don’t want to kill it? Consider the top output below:


$ top -c -p 16514

23:00:44  up 12 days,  2:04,  4 users,  load average: 0.47, 0.35, 0.31

1 processes: 1 sleeping, 0 running, 0 zombie, 0 stopped

CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle

           total    0.0%    0.6%    8.7%   2.2%     0.0%   88.3%    0.0%

Mem:  1026912k av, 1010476k used,   16436k free,       0k shrd,   52128k buff

                    766724k actv,  143128k in_d,   14264k in_c

Swap: 2041192k av,   83160k used, 1958032k free                  799432k cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND

16514 spark    19   4 28796  26M 20252 D N   7.0  2.5   0:03   0 sparkPRODB2...

Now that you confirmed the process 16514 is consuming a lot of memory, you can “freeze” it but not kill it using the skill command.


$ skill -STOP 1
After this, check the top output:
23:01:11  up 12 days,  2:05,  4 users,  load average: 1.20, 0.54, 0.38

1 processes: 0 sleeping, 0 running, 0 zombie, 1 stopped

CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle

           total    2.3%    0.0%    0.3%   0.0%     0.0%    2.3%   94.8%

Mem:  1026912k av, 1008756k used,   18156k free,       0k shrd,    3976k buff

                    770024k actv,  143496k in_d,   12876k in_c

Swap: 2041192k av,   83152k used, 1958040k free                  851200k cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND

16514 spark    19   4 28796  26M 20252 T N   0.0  2.5   0:04   0 sparkPRODB2...

The CPU is now 94% idle from 0%. The process is effectively frozen. After some time, you may want to revive the process from coma:


$ skill -CONT 16514

This approach is immensely useful for temporarily freezing processes to make room for more important processes to complete.

The command is very versatile. If you want to stop all processes of the user “spark”, only one command does it all:


$ skill -STOP spark>

You can use a user, a PID, a command or terminal id as argument. The following stops all rman commands.


$ skill -STOP rman

As you can see, skill decides that argument you entered a process ID, userid, or command and acts appropriately. This may cause an issue in some cases, where you may have a user and a command in the same name. The best example is the “spark” process, which is typically run by the user “spark”. So, when you want to stop the process called “spark” and you issue:


$ skill -STOP spark

all the processes of user “spark” stop, including the session you may be on. To be completely unambiguous you can optionally give a new parameter to specify the type of the parameter. To stop a command called spark, you can give:


$ skill -STOP -c spark

The command snice is similar. Instead of stopping a process it makes its priority a lower one. First, check the top output:


  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND

    3 root      15   0     0    0     0 RW    0.0  0.0   0:00   0 kapmd

13680 spark    15   0 11336  10M  8820 T     0.0  1.0   0:00   0 spark

13683 spark    15   0  9972 9608  7788 T     0.0  0.9   0:00   0 spark

13686 spark    15   0  9860 9496  7676 T     0.0  0.9   0:00   0 spark

13689 spark    15   0 10004 9640  7820 T     0.0  0.9   0:00   0 spark

13695 spark    15   0  9984 9620  7800 T     0.0  0.9   0:00   0 spark

13698 spark    15   0 10064 9700  7884 T     0.0  0.9   0:00   0 spark

13701 spark    15   0 22204  21M 16940 T     0.0  2.1   0:00   0 spark

Now, drop the priority of the processes of “spark” by four points. Note that the higher the number, the lower the priority.


$ snice +4 -u spark
  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND

16894 spark    20   4 38904  32M 26248 D N   5.5  3.2   0:01   0 spark

Note how the NI column (for nice values) is now 4 and the priority is now set to 20, instead of 15. This is quite useful in reducing priorities.

Now lets get over to trace the process and manipulate it.

lsof

The command lsof shows a list of processes attached to open files or network ports. List processes attached to a given file: lsof filenmame


List all open files on system:
lsof

(Long list)

List all files opened by user:

The commands netstat -punta and socklist will list open network connections. Use the command lsof -i TCP:port-number to see the processes attached to the port.


root@spark [~]# lsof -i TCP:25 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME exim 11350 mailnull 4u IPv4 1226820346 TCP 201.201.203.43:smtp->esp222.neoplus.adsl.tpnet.pl:2466 (ESTABLISHED) exim 11350 mailnull 5u IPv4 1226820346 TCP 201.201.203.43:smtp->esp222.neoplus.adsl.tpnet.pl:2466 (ESTABLISHED) exim 11811 mailnull 4u IPv4 1226830746 TCP 201.201.203.43:smtp->ANantes-151-1-39-245.w83-195.abo.wanadoo.fr:627 31 (ESTABLISHED) exim 11811 mailnull 5u IPv4 1226830746 TCP 201.201.203.43:smtp->ANantes-151-1-39-245.w83-195.abo.wanadoo.fr:627 31 (ESTABLISHED) exim 14135 mailnull 4u IPv4 1226855434 TCP 201.201.203.43:smtp->125.115.215.112:3653 (ESTABLISHED) exim 14135 mailnull 5u IPv4 1226855434 TCP 201.201.203.43:smtp->125.115.215.112:3653 (ESTABLISHED)

To kill the processes


kill

killall

This will perform an orderly shutdown of the process. If it hangs give a stronger signal with:


kill -9 .

This method is not as sanitary and thus less preferred.

A signal may be given to the process. The program must be programmed to handle the given signal. See /usr/include/bits/signum.h for a full list.

To restart a process after updating it’s configuration file, issue the command


kill -HUP

The process attached to an open file can be killed using the command fuser:


fuser -ki filename
VN:F [1.9.6_1107]
Rating: 0.0/10 (0 votes cast)
VN:F [1.9.6_1107]
Rating: +1 (from 1 vote)

Prince Joseph Articles, linux

Using FreeBSD ports and packages

April 11th, 2009

FreeBsd Ports/Packages

New to FreeBsd and only familiar with RPM ? This will article will guide you through how to install third-party applications and system tools using ports and packages

Ports

Port is a technology used to install from source.

Each Port is a collection of scripts that when executed, automatically download source of softwares from the Internet, patches, configures if necessary, compiles and install it. Any dependencies on other applications or libraries a port may have are also installed for the user.

Each port, or software package, is maintained by a “port maintainer”, an individual who is responsible for staying current with the latest software developments. Anyone is welcome to become a port maintainer by contributing their favorite piece of software to the collection. One may also choose to adopt and maintain an existing port that has no maintainership.

Packages

Packages is another techInology used to install from pre-built binaries.
For any given application, the FreeBSD package for that application is a single file which you must download. The package contains pre-compiled copies of all the commands for the application, as well as any configuration files or documentation. A downloaded package file can be manipulated with FreeBSD package management commands, such as pkg_add, pkg_delete, pkg_info, and so on. Installing a new application can be carried out with a single command.

Both packages and ports understand dependencies. Suppose you want to install an application that depends on a specific library being installed. Both the application and the library have been made available as FreeBSD ports and packages. If you use the pkg_add command or the ports system to add the application, both will notice that the library has not been installed, and automatically install the library first.

Given that the two technologies are quite similar, you might be wondering why FreeBSD bothers with both. Packages and ports both have their own strengths, and which one you use will depend on your own preference.

Find your application to Install

Suppose you want to install lsof using port
If you know the exact name of the port, but just need to find out which category it is in, you can use the whereis command. Simply type whereis file, where file is the program you want to install. If it is found on your system, you will be told where it is, as follows:


# whereis lsof

lsof: /usr/ports/sysutils/lsof

This tells us that lsof (a system utility) can be found in the
/usr/ports/sysutils/lsof directory.

Yet another way to find a particular port is by using the Ports Collection’s built-in search mechanism. To use the search feature, you will need to be in the /usr/ports directory. Once in that directory, run make search name=program-name where program-name is the name of the program you want to find. For example, if you were looking for lsof:


# cd /usr/ports

# make search name=lsof

Port:   lsof-4.56.4

Path:   /usr/ports/sysutils/lsof

Info:   Lists information about open files (similar to fstat(1))

Maint:  obrien@FreeBSD.org

Index:  sysutils

B-deps:

R-deps:

For in-depth searching you can also use make search key=string where string is some text to search for. This searches port names, comments, descriptions and dependencies and can be used to find ports.

To begin, change to the directory for the port you want to install:


# cd /usr/ports/sysutils/lsof

Once inside the lsof directory, you will see the port skeleton. The next step is to compile, or “build”, the port. This is done by simply typing make at the prompt. Once you have done so, you should see something like this:


# make

>> lsof_4.57D.freebsd.tar.gz doesn't seem to exist in /usr/ports/distfiles/.

>> Attempting to fetch from ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/.

===>  Extracting for lsof-4.57

...

[extraction output snipped]

...

>> Checksum OK for lsof_4.57D.freebsd.tar.gz.

===>  Patching for lsof-4.57

===>  Applying FreeBSD patches for lsof-4.57

===>  Configuring for lsof-4.57

...

[configure output snipped]

...

===>  Building for lsof-4.57

...

[compilation output snipped]

Notice that once the compile is complete you are returned to your prompt. The next step is to install the port. In order to install it, you simply need to tack one word onto the make command, and that word is install:


# make install

===>  Installing for lsof-4.57

...

[installation output snipped]

...

===>   Generating temporary packing list

===>   Compressing manual pages for lsof-4.57

===>   Registering installation for lsof-4.57

===>  SECURITY NOTE:

      This port has installed the following binaries which execute with

      increased privileges.

It is always good idea to delete the working directory.


# make clean

===>  Cleaning for lsof-4.57

Note: You can save two extra steps by just running “make install clean” instead of make, make install and make clean as three separate steps.
This is how you install a utility using ports now you need to know more about Packages before we proceed with Ports.

How to use Package system

Install a package

You can use the pkg_add utility to install a FreeBSD software package from a local file or from a server on the network.


# pkg_add -r lsof

This will cause the utility to automatically determine the correct object format and release and then fetch and install the package from an FTP site. If you want to some alternative FreeBSD Packages Mirror, instead of the main distribution site, you have to set PACKAGESITE accordingly.
It is possible to change this behavior by overriding the PACKAGESITE environment variable. For example, if you run a FreeBSD 5.4-RELEASE system, by default pkg_add will try to fetch packages from


ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-5.4-release/Latest/. If you want to force pkg_add to download FreeBSD 5-STABLE packages,
set PACKAGESITE to ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/Latest/

Managing Packages


pkg_info is a utility that lists and describes the various packages installed.

cvsup-16.1          A general network file distribution system optimized for CV

docbook-1.2         Meta-port for the different versions of the DocBook DTD

pkg_version is a utility that summarizes the versions of all installed packages. It compares the package version to the current version found in the ports tree.

The symbols in the second column indicate the relative age of the installed version and the version available in the local ports tree.

Symbol Meaning

= The version of the installed package matches the one found in the local ports tree.

The installed version is newer than the one found in the local ports tree.

(The local ports tree is probably out of date.)

? The installed package cannot be found in the ports index.

(This can happen, for instance, if an installed port is removed from the Ports Collection or renamed.)

* There are multiple versions of the package.

Deleting a Package

To remove a previously installed software package, use the pkg_delete utility.


# pkg_delete xchat-1.7.1

Now lets come again to Ports, below are the steps to obtain Ports if its not installed in the machine.

Obtaining the Ports Collection.

Before you can install ports, you must first obtain the Ports Collection–which is essentially a set of Makefiles, patches, and description files placed in /usr/ports.

When installing your FreeBSD system, sysinstall asked if you would like to install the Ports Collection. If you chose no, you can follow these instructions to obtain the ports collection:

CVSup Method

This is a quick method for getting and keeping your copy of the Ports Collection up to date using CVSup. If you want to learn more about CVSup, see Using CVSup.

Make sure /usr/ports is empty before you run CVSup for the first time! If you already have the Ports Collection present, obtained from another source, CVSup will not prune removed patch files.

Install the net/cvsup-without-gui package:


#pkg_add -r cvsup-without-gui

#cvsup -L 2 -h cvsup.FreeBSD.org /usr/share/examples/cvsup/ports-supfile

Change cvsup

FreeBSD.org to a CVSup server near you. See CVSup Mirrors for a complete listing of mirror sites. Note: One may want to use his own ports-supfile, for example to avoid the need of passing the CVSup server on the command line.


1. In this case, as root, copy /usr/share/examples/cvsup/ports-supfile to a new location,

such as /root or your home directory.

2. Edit ports-supfile.

3. Change CHANGE_THIS.FreeBSD.org to a CVSup server near you.

4. And now to run cvsup, use the following:

# cvsup -L 2 /root/ports-supfile

3. Running the cvsup command later will download and apply all the recent changes to your Ports Collection,

except actually rebuilding the ports for your own system.

Portsnap Method

Portsnap is an alternative system for distributing the Ports Collection. It was first included in FreeBSD 6.0. On older systems, you can install it from sysutils/portsnap package:


# pkg_add -r portsnap

1. Since FreeBSD 6.1-RELEASE and with recent versions of the Portsnap port or package, you can safely skip this step. The /usr/ports will be created automatically at first use of the portsnap command. With previous versions of Portsnap, you will have to create an empty directory /usr/ports if it does not exists:

2. Download a compressed snapshot of the Ports Collection into /var/db/portsnap. You can disconnect from the Internet after this step, if you wish.


# portsnap fetch

3. If you are running Portsnap for the first time, extract the snapshot into /usr/ports:


# portsnap extract

If you already have a populated /usr/ports and you are just updating, run the following command instead:


# portsnap update

Sysinstall Method

This method involves using sysinstall to install the Ports Collection from the installation media. Note that the old copy of Ports Collection from the date of the release will be installed. If you have Internet access, you should always use one of the methods mentioned above.


1. As root

# sysinstall

2. Scroll down and select Configure, press Enter.

3. Scroll down and select Distributions, press Enter.

4. Scroll down to ports, press Space.

5. Scroll up to Exit, press Enter.

6. Select your desired installation media, such as CDROM, FTP, and so on.

7. Scroll up to Exit and press Enter.

8. Press X to exit sysinstall

Installing using Ports

The Ports Collection makes an assumption that you have a working Internet connection. If you do not, you will need to put a copy of the distfile into /usr/ports/distfiles manually.

In some rare cases, users may need to acquire the tarballs from a site other than the MASTER_SITES (the location where files are downloaded from). You can override the MASTER_SITES option with the following command:


# cd /usr/ports/directory

# make MASTER_SITE_OVERRIDE=ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/ fetch

Removing Installed Ports


# pkg_delete lsof-4.57
Upgrading Ports
 # pkg_version -v

Before attempting a port upgrade, you should check /usr/ports/UPDATING

Upgrade Ports using Portupgrade

The portupgrade utility is designed to easily upgrade installed ports. It is available from the sysutils/portupgrade port. Install it like any other port, using the make install clean command:


# cd /usr/ports/sysutils/portupgrade

# make install clean

Scan the list of installed ports with the pkgdb -F command and fix all the inconsistencies it reports. It is a good idea to do this regularly, before every upgrade.

When you run portupgrade -a, portupgrade will begin to upgrade all the outdated ports installed on your system. Use the -i flag if you want to be asked for confirmation of every individual upgrade.


# portupgrade -ai

If you want to upgrade only a certain application, not all available ports, use portupgrade pkgname. Include the -R flag if portupgrade should first upgrade all the ports required by the given application.


# portupgrade -R firefox,

To use packages instead of ports for installation, provide -P flag. With this option portupgrade searches the local directories listed in PKG_PATH, or fetches packages from remote site if it is not found locally. If packages can not be found locally or fetched remotely, portupgrade will use ports. To avoid using ports, specify -PP.


# portupgrade -PR gnome2

To just fetch distfiles (or packages, if -P is specified) without building or installing anything, use -F

Upgrading Ports using Portmanager

Portmanager is another utility for easy upgrading of installed ports. It is available from the sysutils/portmanager port:


# cd /usr/ports/sysutils/portmanager

# make install clean

All the installed ports can be upgraded using this simple command:


# portmanager -u

You can add the -ui flag to get asked for confirmation of every step Portmanager will perform. Portmanager can also be used to install new ports on the system. Unlike the usual make install clean command, it will upgrade all the dependencies prior to building and installing the selected port.


# portmanager x11/gnome2

If there are any problems regarding the dependencies for the selected port, you can use Portmanager to rebuild all of them in the correct order. Once finished, the problematic port will be rebuilt too.


# portmanager graphics/gimp -f

Ports and Disk Space

Using the Ports Collection will use up disk space over time. After building and installing software from the ports, you should always remember to clean up the temporary work directories using the make clean command. You can sweep the whole Ports Collection with the following command:


# portsclean -C

You will accumulate a lot of old source distribution files in the distfiles directory over time. You can remove them by hand, or you can use the following command to delete all the distfiles that are no longer referenced by any ports:


# portsclean -D

Or to remove all distfiles not referenced by any port currently installed on your system:


# portsclean -DD

Note: The portsclean utility is part of the portupgrade suite.

Post Installation

To know more about the package you have installed now.


# pkg_info -L newpackage-1.0.0 | less

# pkg_info | grep -i newpackage

If the application has a web site, check it for additional documentation, frequently asked questions, and so forth. If you are not sure of the web site address it may be listed in the output from


# pkg_info newpackage-1.0.0
VN:F [1.9.6_1107]
Rating: 7.0/10 (1 vote cast)
VN:F [1.9.6_1107]
Rating: 0 (from 0 votes)

Shijil T S Articles, linux

Apache suEXEC: Configuring, Installing and Testing suEXEC

November 16th, 2008

Apache comes bundled with its own security wrapper application called suEXEC. SuExec is a security mechanism for wrapping dynamic content (SSI & CGI) – content displayed based on user input and scripts executed. Normally, when a CGI or SSI program executes, it runs as the same user who is running the web server. suEXEC allows users to run CGI and SSI programs as the owner of the site as opposed to the owner of the httpd process ( usually “nobody” ).

Here’s how suEXEC works. When a request is made for a CGI or SSI file not owned by “nobody”, the request is passed to suEXEC along with the program name and the owner’s user/group ID. suEXEC then runs a series of checks to ensure the request is valid. If it is, the script is executed. If the request fails any of the checks, the script is not run and an error is logged.

The suexec wrapper isn’t turned on or off by any particular Apache directive setting. Instead, when the Apache server is compiled, one of the constants set (SUEXEC_BIN) is a string pointing to the location of the suexec binary. When the server starts, it looks for the binary at that location; if it’s found, suexec is enabled — otherwise it isn’t.

Installing and Compiling apache with suexec

Download your apache source from http://www.apache.org/


wget http://apache.downlod.in/httpd/apache_1.3.37.tar.gz

Unpack it


tar -zxf apache_1.3.37.tar.gz

Configure apache with your favourite options. The options enable, caller=nobody, docroot=/usr/local concerns with suexec.


cd apache_1.3.37

./configure  --enable-module=most  --with-layout=Apache  --prefix=/usr/local/apache

--with-port=80 --enable-suexec  --suexec-caller=nobody --suexec-docroot=/usr/local

If you want to change the location of the suexec binary, you can do so by adding a new definition of SUEXEC_BIN to the compilation flags:


env CFLAGS="-Wall -DSUEXEC_BIN=\"/usr/local/apache/suexec\"" \

> ./configure --enable-suexec ...

Checking your suEXEC setup

Before you compile and install the suEXEC wrapper you can check the configuration with the –layout option.

Example output:


    suEXEC setup:

            suexec binary: /usr/local/apache/sbin/suexec

            document root: /usr/local/apache/share/htdocs

           userdir suffix: public_html

                  logfile: /usr/local/apache/var/log/suexec_log

                safe path: /usr/local/bin:/usr/bin:/bin

                caller ID: www

          minimum user ID: 100

         minimum group ID: 100

When configuration is all done, Install it


make

make install

Disabling suexec

If your Apache installation is currently suexec-enabled, it’s very simple to turn the wrapper off. Just do one or more of the following to the suexec binary:

*Clear the setuid bit
*Change the owner to be someone other than root
*Delete or rename it

and then restart the Apache server. Doing any one of these will render the suexec facility unusable, and Apache won’t even try to involve it. To verify that your action has had the desired effect, verify (if you’re running Apache 1.3.11 or later) with the “/usr/local/web/apache/bin/httpd -l” command. If the output says suexec is enabled, you haven’t done enough yet.

Since the point of suexec is to handle certain Web requests under a different identity than the Apache server user, there needs to be some way to specify just which user. There are two places from which Apache will draw this information:

*The username from URLs such as http://somehost.com/~username/foo.cgi

OR

*The User and Group directives in the server configuration file, httpd.conf

The username to use is determined by checking these in the above order.

Testing Your Installation

The simplest way to verify that suexec is functioning properly is to install a script that will tell you the username under which it’s being invoked.
Here is a sample script – test.cgi


#!/bin/sh

echo "Content-type: text/plain"

echo ""

echo "Username="\`whoami\`

To test that suexec will properly handle a CGI script in a user’s directory, copy your test.cgi script into that user’s public_html/ directory, make sure that both the script and the public_html/ directory itself are mode 755 and owned by the user, and then request the script with a URL such as . If you get an error page, look at the Apache and suexec logs.

Debugging

Debugging a suexec problem can be frustrating, particularly since almost any problem with a CGI script in a suexec-enabled environment turns out to be related to the wrapper.

The typical warning signal of a suexec problem is a request for a CGI script that results in a ’500 Internal Server Error’ page. The real error message will be found in your suexec log – /usr/local/apache/logs/suexec_log ( which may vary according to the configuration you used)

Best of Luck !! :)

VN:F [1.9.6_1107]
Rating: 0.0/10 (0 votes cast)
VN:F [1.9.6_1107]
Rating: 0 (from 0 votes)

Sai Kottuvally Articles, linux ,

Postgresql Command line Tips

August 16th, 2008

PostgreSQL is an object-relational database management system (ORDBMS) based on POSTGRES, Version 4.2, developed at the University of California at Berkeley Computer Science Department. POSTGRES pioneered many concepts that only became available in some commercial database systems much later.

PostgreSQL is an open-source descendant of this original Berkeley code. It supports a large part of the SQL standard and offers many modern features:

  • complex queries
  • foreign keys
  • triggers
  • views
  • transactional integrity
  • multiversion concurrency control

You can login to shell by


# su - postgres

bash$

How to create a database

Normally, the database user who executes this command becomes the owner of the new database. However a different owner can be specified via the -O option, if the executing user has appropriate privileges.


bash$ createdb sparksuppport

CREATE USER

Accessing a Database

Once you have created a database, you can access it by running the PostgreSQL interactive terminal program, called psql, which allows you to interactively enter, edit, and execute SQL commands.


bash$ psql sparksupport

Welcome to psql 8.0.13, the PostgreSQL interactive terminal.

Type:  \copyright for distribution terms

       \h for help with SQL commands

       \? for help with psql commands

       \g or terminate with semicolon to execute query

       \q to quit

sparksupport=>

Creating a User

createuser creates a new PostgreSQL user. Only superusers (users with usesuper set in the pg_shadow table) can create new PostgreSQL users, so createuser must be invoked by someone who can connect as a PostgreSQL superuser.

Being a superuser also implies the ability to bypass access permission checks within the database, so superuserdom should not be granted lightly.


bash$ createuser spark

Shall the new user be allowed to create databases? (y/n) n

Shall the new user be allowed to create more new users? (y/n) n

Dropping a database


bash$ dropdb sparksupport

DROP DATABASE

Dropping a User


bash$ dropuser spark

DROP USER

Creating, Dropping Groups

Currently, there is no easy interface to set up user groups. You have to explicitly insert/update the pg_group table. For example.


# su - postgres

bash$ psql sparksupport

..... in psql press up/down arrow keys for history line editing or \s

sparksupport=> insert into pg_group (groname, grosysid, grolist)

sparksupport=> values ('hackers', '1234', '{5443, 8261}' );

INSERT 58224

sparksupport=> grant insert on foo to group hackers;

CHANGE

To drop the group


# su - postgres

bash$ psql sparksupport

sparksupport=> delete from pg_group where groname = 'hackers';

Backup and Restore database

pg_dump is a utility for backing up a PostgreSQL database. It makes consistent backups even if the database is being used concurrently. pg_dump does not block other users accessing the database (readers or writers).

To dump a database:


bash$ pg_dump sparksupport > db.out

To reload this database:


bash$ psql -d newdatabasename -f db.out

or

bash$ cat db.out | psql newdatabasename

To dump all databases


bash$ man pg_dumpall

bash$ pg_dumpall -o > db_all.out
To reload (restore) all databases dumped with pg_dumpall:
bash$ psql -e template1 < db_all.out
Backup large databases

To dump a database called sparksupport that contains large objects to a tar file:


bash$ pg_dump -Ft -b sparksupport > db.tar
Reload with :
bash$ createdb

bash$ gunzip -c filename.dump.gz | psql

Or

bash$ cat filename.dump.gz | gunzip | psql

Use split:


bash$ pg_dump  | split -b 1m - filename.dump.

Note: There is a dot (.) after filename.dump in the above command!!

You can reload with:


bash$ man createdb

bash$ createdb

bash$ cat filename.dump.* | pgsql

For further referrence http://www.postgresql.org/docs/8.0/static/index.html

VN:F [1.9.6_1107]
Rating: 0.0/10 (0 votes cast)
VN:F [1.9.6_1107]
Rating: 0 (from 0 votes)

Shijil T S Articles, linux ,