Iptables is the default firewall used in many present Linux distros. It is a project under netfilter. Previously it was known as ipchains (For 2.2 Kernels) , ipfwadm(for 2.0 kernels) and nftables are their future project..Iptables are designed for ipv4 and ipv6table for ipv6.

From ipfwadm to iptables , we can see marginal modifications done in design and hence in features. In ipfwadm a rule set matched the packets and actions are taken on that packets , then in ipchains , chains are Incorporated and the rules will decide , through which chain the packets need to be traversed, then in iptables , tables are introduced so that the user can choose proper table based on his requirement , that means he can choose filter table for packet filtering . NAT tables for source and destination natting then mangle table for packet modifications.(We will discuss more about tables later ). Connection tracking capability is the yet another advancement with iptable. So we can say it as a statefull firewall. Rate limiting , system logging etc. are the other features included in iptables apart from it’s predecessors.

Iptables gets it’s ascendancy through the modularity and integrity with the kernel. It exploits the modularity of linux kernels. A bunch of kernel modules need to be loaded for the full operation of iptables.

ip_tables iptable_filter ip_conntrack ip_conntrack_ftp
iptable_nat ip_nat_ftp ipt_limit ipt_multiport
iptable_mangle ipt_state ipt_REJECT ipt_LOG

These are some of required kernel modules. All are not required for all applications. For loading modules in bootup add script in rc.local using modprobe command. And also we need a kernel with some specific config options .If it’s not there we need to recompile the kernel to exploit the functions of iptable. For latest kernels almost all options are included , rarely need to be recompiled. Some important options that need to be compiled in to kernel or as modules are

CONFIG_PACKET
CONFIG_NETFILTER
CONFIG_IP_NF_CONNTRACK
CONFIG_IP_NF_FTP
CONFIG_IP_NF_IRC
ONFIG_IP_NF_IPTABLES
CONFIG_IP_NF_FILTER
CONFIG_IP_NF_NAT
CONFIG_IP_NF_MATCH_STATE
CONFIG_IP_NF_TARGET_LOG
CONFIG_IP_NF_MATCH_LIMIT
CONFIG_IP_NF_TARGET_MASQUERADE .

You can check your current kernel configuration by cat /boot/config-`uname -r`.As I already told the latest kernels are incorporated with almost all netfilter options. So you seldom need to worry about that.

Packet flow in iptables
Next I would like to draw a vignette of packet flaw in iptables. There is definite order for traversing of packets through the different tables and chains in iptables. Basic understanding of that order is mandatory for writing complex firewall rules.

Iptable comprised of three tables , MANGLE , FILTER and NAT table.
Each tables have it’s own chains for packet processing.
MANGLE table : This tables is responsible for the alternation of QoS bits in TCP header.For example TOS (Type of service) bit. But it rarely used for simple networks.Mangle table have five chains ,PREROUTING, POSTROUTING,FORWARD,INPUT and OUTPUT.

FILTER table:This is the most used table in small network environments.This table is responsible for packet filtering on the basis of corresponding rules. This table have INPUT, FORWARD and OUTPUT chains. Most used target and jumps in this table are ACCEPT, DROP/DENY and REJECT. We will discuss about this target and jump later.

NAT table:This table used for network address translation which is a imperative feature of routers.It have PREROUTING chain (for Destination address translation) FORWARD chain (address translation for packets from router itself) and POSTROUTING chain( for source address translation).

When packet reaches the hardware(NIC) it will be processed to hardware driver and through kernel.Then it traverse through different tables and chains in the iptables in desired order. In each chain , iptable compare the rules in that chain with that packet information , if it finds a match then action is taken based on that rule.(eg:REJECT) .If a packet is matched with the first rule in a chain , it will not check the second rule in the same chain , it will jump to next chain in the order.

The following figure will help you to understand the packet traversal through iptable chains.

Options with iptable command.

-t Specify the table. By default FILTER table is taken .
-j Jump to the target
-A Append the rule to the end of the chain.
-F Flush. Deletes all the rules in the selected table
-p Match protocol. Types include, icmp, tcp, udp, and all
-s Match source IP address
-d Match destination IP address
-i Match “input” interface on which the packet enters.
-o Match “output” interface on which the packet exits
-m used to match certain conditions . eg: state ESTABLISHED

Targets and Jumps with iptable
DROP/DENY : When a packet is dropped or denied it is simply  abandoned and no notification is send to the host. ie the packet disappears with out taking any further actions.

REJECT : The host reject the packet and sends reply to the sending host , saying that the packet was dropped. Other wise it is same as that of DROP/DENY.

ACCEPT : This will accept the packet that matches the corresponding rules.

LOG : The packet information will be logged with syslogd daemon.

SNAT : Source nat used in NAT table POSTROUTING table for manipulating source ip address.

DNAT: destination NAT used in NAT table PREROUTING table to manipulate destination ip address and for port forwarding.

MASQUERADE : To change source ip address to the ip address of corresponding router network interface.

Iptable commands and rules

iptables -L : To list all present rules in a table (By default it will show FILTER table rule.To specify the table use -t )
iptables -t nat -L (list all NAT table rules)
iptables -v -L : more detailed information about the rules
iptables -F

iptables -A INPUT -p tcp -s 0/0 --dport 22 -j ACCEPT <This will allow port 22 on INPUT chain of filter table)
iptables-A OUTPUT -p tcp -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -P INPUT DROP

Rule for port forwarding

 iptables -t nat -A PREROUTING -p tcp -d  'destination ip' --dport 80 -j DNAT --to-destination 192.168.1.10:80
Packet destined to 80 port of your public ip will be forwarded to the 80 port of private ip.We appending rule to the PREROUTING chain of NAT table.
iptables -A FORWARD -p tcp -s 0/0 -d 192.168.1.10 --dport 80 -j ACCEPT

I think this is enough for the basics. I will write more advanced rules on my next article. Try it!!!!!

hans linux iptables, iptables tutorial, linux iptables

This article is a part of the on-going Web Scraping Series. If you are not familiar with Web Scraping please check with the first article . This session mainly deals with Dynamic Content Scraping. Nowadays most of the web portals are dynamic by making Ajax calls instead of old static web pages. Scraping on dynamic environment is both interesting and challenging one.

The first part of the discussion concentrated mainly on static page scraping with Perl mechanize module. Even though mechanize provides extension for dynamic scraping, it is not very good.

So this session deals with making use of selenium testing tool for Web Scraping.

Prerequsites

Selenium IDE is a Firefox add-on that records clicks, typing, and other actions to make a test, which you can play back in the browser.

• Download Selenium IDE

Selenium Remote Control (RC)  is a Java based Command line server for handling request from client.

• Download Selenium Remote Control (RC)
  

Pros and Cons

It supports all Dynamic Content like Ajax, JavaScript, is easy to implement and it is possible to write selenium clients in any language we prefer, for example, here I have used Perl. You can also use Python, Java, etc.

Selenium based Web Scraping on small throughout is easy task.

It consumes lots of memory resource, for each request it will launch a new browser instance.

Working of selenium

Selenium Remote Control (RC) is a test tool that allows you to write automated web application UI tests in any programming language against any HTTP website using any mainstream JavaScript-enabled browser.

Selenium RC comes in two parts.

1. A server which automatically launches and kills browsers, and acts as a HTTP proxy for web requests from them.
2. Client libraries for your favourite computer language.

The RC server also bundles Selenium Core, and automatically loads it into the browser.

Here is a simplified architectural representation:

For Detailed diagram http://seleniumhq.org/about/how.html

How to Setup a Selenium Server

Download Selenium RC server to directory to /usr/local/selenium

#cd /usr/local/selenium

#unzip selenium-remote-control-1.0-beta-2-dist.zip

#cd selenium-remote-control-1.0-beta-2

#cd selenium-server-1.0-beta-2

#java -jar selenium-server.jar #starting selenium server .By default it is listen to 4444

An example Client Program

As said in the above section, it is possible to create selenium client by recording user activities or else the programmers can create it using their own language. Python, Perl and Ruby, Java has supporting modules for it.

#Sample Perl Code
#!/usr/bin/perl
use strict;
use warnings;
use Time::HiRes qw(sleep);
use Test::WWW::Selenium;
use Test::More "no_plan";
use Test::Exception;

my $sel = Test::WWW::Selenium->new( host => "192.168.1.20",
port => 4444,
browser => "*firefox",
browser_url => "http://www.godaddy.com/" );
$sel->open_ok("/domains/search.aspx?ci=8969");
$sel->click_ok("domain_search_button");
$sel->wait_for_page_to_load_ok("30000");
my $data=$sel->get_html_source(); # here you get source of the current page

For more info please have a look at cpan http://search.cpan.org/search?query=selenium&mode=all

As scraper you can extract required data from this source:

For scraping data from multiple pages

Open selenium IDE and record the events that you are interested and analyse the code generated and try to implement your own way,

As a last word, let me add that selenium is not completely a scraping tool, it is instead, a testing tool.

For more about selenium have look at http://seleniumhq.org/

Shameem Khalid linux, perl Ajax Scraping using selenium, Crawling Ajax, scraping, scraping + selenium, web crawling

There are many open source version control systems like Bazaar, Arch, Aegis, SVN ,CVS , Git etc. All systems have there own pros and cons. But I personally like Git due to it’s flexibility and possibilities.It have an excellent network compatibility. We can use the native Git protocol, but it also works over rsync, ssh, HTTP and HTTPS. Access control is the more painful headache for admins. ie who can commit , who can access projects etc. Here we can manage git repositories with gitosis , which is a tool for git access control . More safe thing is gitosis can be used with ssh-key authentication which is one of the most secure data transfer methodology in linux. While we use key-authentication we don’t need to create that user in Git server , it tighten security further becuase developers will not get shell access on server. There should be a Git user in server , here I am creating a user as ‘git’ in server.

If you don’t have Git on your machine install it through yum(Redhat based) or apt-get(Debian based) depending up on your OS.

# apt-get install git-core

For the installation of gitosis we need python-setuptools package in our server.

# apt-get install python-setuptools

Now we can download gitosiis from it’s git repository.

# git clone git://eagain.net/gitosis.git

it’s of few KBs .Now install it with python script.

# cd gitosis
# python setup.py install

create an user as git(you can give any name) . Give him a shell otherwise ssh login might have problems ,then give the desired home directory.(which will be the Git Root directory).

It’s the time to upload public key to Git server. If you have a public-private key pair in local machineuse it , otherwise create a new one with ssk-keygen.

# ssh-keygen -t rsa

Public key will be in name id_rsa.pub . Copy the key and upload it to the server.

Next , execute the command

# sudo -H -u git gitosis-init < /tmp/id_rsa.pub

This will add some directories and files in to user git’s home directory and append the public key to authorized keys file of user git.
Now clone the gitosis-admin repository to your local machine.

$ git clone git@GIT-SERVER:gitosis-admin.git
$ cd gitosis-admin

If you list this directory you can see only two directories .Here You can create new projects , make changes , do access control etc. What ever you do , you have to commit it and push it to the server to replicate the changes on server. Since we cloned gitosis-admin repository we can make all changes locally and push it to the server.

Creating new repository and control access on new repository.

For creating a new repository , edit the gitosis.conf file and add the entries as follows.
Here I am assuming that the new repository name is ‘test’

[group testgroup]
members = hans
writable = test

Here group is just a name , no matter what it is. member is the list of users who can access the repository and ‘writable’ is the name of repository. Commit this changes and push it to the server.

$ git commit -a -m "comment for this commit"
$ git push

Create the directory ‘test’ and initiate it as a git repo.

$ mkdir test
$ cd test
$ git init

Add remote server url to the new repository config file.

$ git remote add origin git@GIT_SERVER:test.git

push the changes in to the server.

$ git push origin master:refs/heads/master

Eventhough we allowed user hans to access ‘test’ repository , we didn’t add his public key to the server . So he will not be able to access the project repository. So copy his public key to ‘keydir’ as hans.pub . It should have a ‘.pub’ extension.

$ cd gitosis-admin
$ cp /home/hans/hans.pub keydir/hans.pub
$ git add keydir/hans.pub
$ git commit -a -m "comment"
$ git push

Yes we done it !
Now user hans can clone the test repository to his local machine. He can make changes ,add files ,commit ,and push changes to the server.

hans $ git clone git@GIT_SERVER:test.git
Now try it. Good Luck !

hans linux git, git access control, gitosis

By default, routing is based on destination IP address, i.e., the routing table will decide where to route the packet depending on the destination address field in the packet. But there are a number of other utilities and options on routing in linux. One interesting fact is that most of the routers are using linux kernel for their IOS (inter operating system). Isn’t that cool? Now you can use your linux machine as your local router with more confidence, right?

Other than destination IP address we can also use source IP address , Tos (Type of service) , fwmark (Marking of packets by kernel) and the interface on which packet arrived as the parameters for routing decision. Before getting in to IP routing let me explain some scenarios where we need Advanced routing features. Suppose we have 3-4 gateways and we can easily differentiate them on their bandwidth, reliability, QoS and cost, which one do we select? We would definitely choose the one with the best combination of all these features. But if everyone selects that gateway and start using it, it would obviously be overwhelmed with the traffic. So it is best to differentiate services based on their priority. We should give more priority to services which need higher bandwidth or more reliability. Then the route packet will be used by higher priority services through the most reliable gateway. One example for such service is the SSH.

Now let’s begin with the commands for this advanced routing feature. We will be using the combination of iptables, IP route and IP rule commands here. All linux systems have an iptable package by default. If you don’t have IP route or IP rule command, you have to download the iproute2 package. Before writing the rules on iptables, check kernel modules and options that are required for the full operation of iptable. But since the latest kernels have all the modules and options for iptables you will not have to worry about it.

Let’s start from the iptable. Here we shall take SSH as the service for routing. Use ‘mangle’ table of iptable for modifying the SSH packets. We will need root access for this. We are assuming that SSH is using its default port 22.

1. #iptables -t mangle -A OUTPUT -p tcp –dport 22 -j MARK –set-mark 0×1

We are marking all packets with destination port 22 as ’0×1′ .Now save and restart iptables.
#service iptables save
#service iptables restart

To delete this entry from iptables we can use -D instead of -A.

2. Next, create a new IP route table in /etc/iproute2/rt_table by just giving an entry

100 sshtable

3. Write rule for SSH packets.

#ip rule add fwmark 0×1 lookup sshtable

4. Add route at new table sshtable. Here we shall use ’192.168.1.1′ as the gateway for SSH. All other traffic will go through the
default gateway, which can be seen by IP route show command.

We copy all entries except default gateway entry from main table.
# ip route show table main | grep -Ev ^default | while read ROUTE ; do ip route add table sshtable $ROUTE; done

Add default gateway entry for SSH packets to table sshtable .

#ip route add default via 192.168.1.1 table sshtable

Use “ip route show table sshtable” to show all routes at sshtable.

That’s it..we have done it!!!

You can use SSH to log into your remote server and check your IP with “last” command. You can see that it’s showing your new gateway public IP other than the default gateway. You can use the same for web traffic. Use tcp ports 80,443 ,53 and udp 53 instead of 22 in the above example.

If you want perpetual route settings during reboots, make the following entries in /etc/sysconfig/network-scripts/route-eth0
and /etc/sysconfig/network-scripts/rule-eth0 .

If these files are not there, you can create them and:

in rule-eth0 file paste the following.

fwmark 0×1 lookup sshtable

in route-eth0 add the following line:

default via 192.168.1.1 table sshtable

Then paste # ip route show table main | grep -Ev ^default | while read ROUTE ; do ip route add table sshtable $ROUTE; done in /etc/rc.local file.

Good luck!

hans linux "Application based routing in Linux", "ip route", "ip rule", "linux routing", "port based routing", "routing with ip rule", Advanced routing, linux mark based routing

#!/usr/bin/perl -w
use WWW::Mechanize;
$url = 'http://chato.cl/research/crawling_thesis ';
$m = WWW::Mechanize->new();
$m->get($url);
$c = $m->content; #  Will display souce code of the above link
exit;

Shameem Khalid Articles, linux, perl bots, perl, scraping, tor, web crawling

Red5 is an open source video sream server appllication which helps you to stream your video content across the web. Inorder to install and run red5, you need the java platform installed on your system. Using apache-ant or eclipse we can build and install Red5. Here apache-ant  method is described.

1)Using Apache-ant

a)install jdk 1.6 or higher and its curresponding deveopment package.

#yum install java-1.6.0-openjdk java-1.6.0-openjdk-devel

b)Install apache-ant

To build the red5 binary from source we need apache-ant. So install it.

#cd /usr/src

#wget http://archive.apache.org/dist/ant/binaries/apache-ant-1.8.0-bin.tar.bz2

#tar -xjf apache-ant-1.8.0-bin.tar.bz2

#mv apache-ant-1.8.0 /usr/local/ant

c)Set the enviroment variables for java, ant and java class path

export ANT_HOME=/usr/local/ant
export JAVA_HOME=/usr/lib/jvm/java
export PATH=$PATH:/usr/local/ant/bin
export CLASSPATH=.:$JAVA_HOME/lib/classes.zip

If you want this to be avilable for all users, append these lines to the file /etc/bashrc

d)Now Install subversion to downlaod the svn version from the googlecode.

#yum install subversion

e)Download and Install Red5

#svn checkout http://red5.googlecode.com/svn/java/server/trunk/ red5

#mv red5 /usr/local

#cd /usr/local/red5

#ant prepare

#ant build

Note: If you are building on CentOS 5.4 use “#ant dist” command instead of “#ant build”

This will take some time. So please be patient.

Finally you will see a line “Build successful” .That means your red5 installation is complete. Now copy the conf directory from dist/ to the current directory and now test your installation by runnig the script

#cp -r dist/conf .
#./red5.sh

Your Installation is ok if it shows “Installer service created” in the last. Now press ctrl+c to quit the process and go for the init script

f)Init Script

#vi /etc/init.d/red5

copy the following code to it.

#!/bin/bash
 PROG=red5
 RED5_HOME=/usr/local/red5
 DAEMON=$RED5_HOME/$PROG.sh PIDFILE=/var/run/$PROG.pid
# Source function library . /etc/rc.d/init.d/functions[ -r /etc/sysconfig/red5 ] && . /etc/sysconfig/red5RETVAL=0
case "$1" in
 start)
 echo -n $"Starting $PROG: "
 cd $RED5_HOME
 $DAEMON >/dev/null 2>/dev/null &
 RETVAL=$?
 if [ $RETVAL -eq 0 ]; then
 echo $! > $PIDFILE
 touch /var/lock/subsys/$PROG
 fi
 [ $RETVAL -eq 0 ] && success $"$PROG startup" || failure $"$PROG startup"
 echo
 ;;
 stop)
 echo -n $"Shutting down $PROG: "
 killproc -p $PIDFILE
 RETVAL=$?
 echo
 [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$PROG
 ;;
 restart)
 $0 stop
 $0 start
 ;;
 status)
 status $PROG -p $PIDFILE
 RETVAL=$?
 ;;
 *)
 echo $"Usage: $0 {start|stop|restart|status}"
 RETVAL=1 esac
exit $RETVAL

 

g)Start the service and test your server

#/etc/init.d/red5 start

Now you can open your browser enter http://[your_ip]:5080 and see your sever working. You can install the sample applications and view the demos.

Jaseer Articles, general, linux howto install red5, Red5, red5 installation

The IM Gateway plugin for Openfire provides connectivity to other IM networks (AIM, ICQ, IRC, MSN, Yahoo, etc). It uses internal mechanisms to smooth the interaction with the various transports that standard transports can not currently provide. This plugin is called kracken im gateway plugin. It can be downloaded from the following link.

http://sourceforge.net/projects/kraken-gateway/

Openfire admin panel provides an option to upload new plugins into it. The file should be in .jar format.

.

Once the plugin file is uploaded it will be listed under plugin section in admin panel.

Next step is to select the networks that we want to connect from openfire. For that you need to go to Gateways in admin panel and select the gateways you want to connect to and its setttings.

Checking a gateway enables the service.

You can test the connection to the gateway network from the openfire server by clicking the ‘Tests’ link.

Also there is an option to specify the users who all are allowed to connect to the gateway service.

Next you need to do gateway service registration for a particular openfire user. For that click on “Registrations” in the left-hand menu. Then click on “Add a new registration.”

Here user is the openfire user which we want to connect to a gateway. Dropdown window for ‘gateways’ lists gateways we selected in one of the earlier steps. You can select the desired one from it. username and password should be corresponding gateway service account credentials. Here it should be login credentials of msn messanger.

You can associate all the gateway services, you want, to a particular openfire user by adding like this. Once you associate gateway services like this you can login into your openfire account through one of the clients, here i am using pidgin to connect to the openfire server. Once you login into openfire account through a client you may be able to see all the gateway services associated with that user are listed in the client.

Thus you will be able to login into all the associated gateways by just logging in to the openfire user account and you can have all your gateway services at once place.

Tino Thomas linux im gateway, im gateway plugin, kraken, kraken im gateway, kraken plugin, kraken xmpp im, openfire, openfire im gateway, openfire plugin

Introduction

Puppet is a Ruby based Configuration Management System with client/server model,  licensed under GPLv2 .It has one Master server puppetmasterd  and all other machines are configured as puppet clients . We set configurations at the puppet server and then push them to all clients which are connected to the master. The client puppet correctly applies the corresponding configurations on the client machine regardless of their platform difference.

Puppet is a gift to the server administrators who need to manage a large number of systems with different flavor of Gnu/Linux, Mac, Solaris and other Unix Based systems.If we are managing systems via remote administration then it would be a headache to the administrator and if the systems are different then the complexity will increase. Some accidental configuration changes may cause inconsistent working of the server. If we are using the Puppet for the configuration management then it will be a one time implementation of these configuration changes only at puppet server, then we just apply them to different puppet clients without any delay.

Another power of the puppet is it uses a Declarative Language to define configuration settings at the puppet master server. This language includes all major high level language features like Functions, Conditional Statements, Inheritance and other OOPs concepts. This feature makes for more readable , reusable and consistent Puppet configurations settings, when we compared with other configuration management tools like Cfengine.

Working

Puppet master server stores all client configurations, and  each client will contact the server via port 8140 (by default). The connection between server and client is encrypted. The client will generate a self-signed key before it connects to server and will submit this self-signed key to the master server and get the verified key back. Here master server acts like a Certification Authority. After this process, the client will establish a encrypted session with the server and get the configuration settings, then compile and apply it on client system. When the client compiles the configurations from server it may rise error messages if  there are any syntax errors in the configuration definitions. We can verify this on the puppet server and client log file.

Here is the outline of puppet server and client Architecture

Puppet Architecture

Installation

Before installing Puppet, we need to setup some dependencies. First we need ruby with common library files(xml,ssl,etc.) installed, and facter, which is another ruby project that gathers all system information. Facter will be installed in all puppet clients. The puppet server retrieves the client configuration settings and other system-specific details from facter.

You can use the ruby’s built-in library management tool rubygem(rake) (similar to CPAN for Perl) to solve the dependency problems with libraries.

Facter installation :-

Get latest version from www.reductivelabs.com

tar -zvf facter-<version>.tar.gz

cd facter

ruby install.rb

facter --version

Puppet installation :-

If we are installing from the package manager, there will be two packages: puppetd as the client and puppet-master as the Puppet server. We need to install both to setup the client and server, and both can be installed from the source code.

Download latest package from the www.puppetlabs.com, then similar to facter installation:

tar -xzvf puppet-<latest version>;

cd puppet-<latest-version>;

ruby install.rb

#Create user and group for puppet
groupadd puppet
useradd -g puppet puppet

This step will install the required packages for the Puppet client and server. If you have any dependency problems then it might be due to a version mismatch problem between ruby/puppet/facter, so select correct the versions.

By default, the configuration files are listed under /etc/puppet and all others are in the /var/lib/puppet  folder (including log files).

Currently Puppet support all major Unix like systems but not Windows.The latest versions of the Puppet has introduced support for the Windows operating system by developing Windows specific facter tool.

How to configure Puppet server :-

After  successful installation of the Puppet master server and client, there is a set of daemons associate with this package as well as command line utilities to manage these daemons. They are:

puppetmasted       #Puppet Master Server

puppetd            #puppet Client.

puppetca           #Key management daemon

#and Set of other Utility commands.

Puppet  work without creating configuration files explicitly; they are already pre-configured. But to start the interaction with clients we need to make some changes. First, we can check the structure of the puppet configuration file.

It’s a good practice maintaining an explicit puppet configuration file;the latest versions of puppet use single configuration file to manage every daemons. By default, configuration files are stored under /etc/puppet. We save  all the configuration details of major daemons at /etc/puppet/puppet.conf.The puppet.conf use a special type of configuration structure to include every daemon’s configuration details,described below:

#Cat /etc/puppet/puppet.conf

[main]

Here We specify a set of configuration details common to all daemons.

[puppetmasterd]

Here comes the Puppet master server configuration details.

[puppetd]

To include the Puppet client configurations.

[puppetca]

Configuration details of puppet key management tool.

To get all the parameters under each daemons and main section with its functional details, please refer this page

How to Connect Puppet Client with Puppet Server

To set up a client we  just have to install the puppet client version or every package in another system.Your master server is now capable to work as a puppet client also. At the master server we need to specify the set of configuration that will guide how to change the configurations at clients.

Puppet server and client use Hostname to communicate with each other and also used to generate ssh key and key verification etc.., so we need a stable hostname resolution system (DNS or Local settings) in our network to ensure the proper connection between clients and server.So select proper hostnames to your server and clients like:

puppet-server.com #For your Master Server

puppet-client1.com,puppet-client2.com,etc... #Your clients.

After the hostname allocation we need to start the server and client daemons.Use command line options now to know the more about the interactions between client and server.

To start the master server :-

 puppetmasterd --no-daemonize --logdest console

Then Start the puppet Client, specify the server name

puppetd --server puppet-server.com --verbose --waitforcert 30

On the client side we will get the message regarding the creation of a self signed key and waiting for server verification.

Creating a new SSL key for puppet-client.com
Creating a new SSL certificate request for puppet-client.com
Certificate Request fingerprint (md5): 37:89:4E:86:C0:A7:5B:24:1A:E2:9B:85:83:90:0F:CE
Did not receive certificate

At the same time server side we will get the following message.

notice: Starting Puppet master version 2.6.0
notice: puppet-client.com has a waiting certificate request

To proceed further , at server side we need to verify this key from the puppet-client.com. For that we can use the key management tool puppetca.

puppetca --list  #To list the unverified requests.

puppetca --sign puppet-client.com  # To complete the verification process.

Now If we are restarting the puppet client with following command, you can see the client will immediately apply the configurations. You can check this from the log file or from the console if you are running the client in none daemonize mode.

puppetd --server puppet-server.com

Note:- If we are specify these settings at puppet.conf then you can just type the commands without any parameters to start appropriate daemons.

The Configuration Management

Last and very powerful feature of the puppet is the way Puppet server define the Client configurations. For that Puppet use one declarative language which support most of the high level language constructs like OOPs. So lets try one simple configuration which change the permission of /etc/passwd file at all the clients connected with server to 640 and check Apache webserver installed or not , if not, puppet client will install it automatically.

These configuration specifications are defined under a file “/etc/puppet/manifests/site.pp” by default, we can split this file in to several files then include them at sites.pp.

Here is the sample site.pp file.

file { "password":
name => "/etc/passwd",
 owner => "root",
 group => "bin",
 mode => 644,
}

class apache {

package {       httpd: ensure => installed  }

service { "httpd":

name => $operatingsystem ? {
debian  => "apache2",
redhat  => "httpd",
default => "apache",
CentOS  => "httpd",
},
ensure => running,
require => Package["httpd"],
}
}

node 'puppet-client.com' {
include apache
}
#All other nodes they don't have definitions associated with them will use the following node definition.

node default {
case $operatingsystem {
CentOS: {include apache }
default: {}
}
}

The above file is the Puppet client configuration specification written in puppet declarative language on puppet master server.

This language has a lot of constructs to define the resource and its properties.Using these constructs we manage the resources on client systems. The types of resources that puppet manages are listed bellow, plus we can add our own customized resources to mange.

Type of Resources that puppet can manage, by default:-

• Files
• Packages
• Services
• Corn Jobs
• Users and Groups
• To run Shell Commands
• And User defined resource types
  

Each of the above resources has a set of attributes or properties and values. Using the puppet configuration language, we can set the corresponding property values. The resource can defined by providing three main parameters: Resource type name, then inside braces({}) title of the resource and set of property values. From the above example, take the resource of type File with title name “password” inside that we have set of property values like name,owner,groups etc… so if a client successfully connect to server,the client puppet will apply these setting on client machine. If we change this property values, after next interval we can see the client will successfully apply it.

In this way we can control the resource configurations. On our networks there should be  different types of systems (Redhat,Debian,etc..),and they have some changes in the structure of the files and other package names, so here we need to apply the configurations based on the type of clients.Puppet provide Conditional statements (if and case ) to check and apply configurations depending on client architecture. For that we need some system information from the client and facter will provide these details. We can use that information in the puppet configuration specifications like a variable, for example: $operatingsystem (You can see all the details that facter will provide by just typing the command facter at command prompt.)

Similarly, we can specify the rules based on the client name, and using the OPPs constructs we can define the classes and reuse them with other client definitions. You can find some of them from  above example site.pp file.You can do a high level configuration design using puppet language. To learn more about the language constructs, please check the puppet online wiki or a nice book  which describe everything associated with Puppet by James Turnbull(Pulling Strings with Puppet.)

Haridas N linux Datacenter Administration Tool, install and configure puppet', Puppet, puppet configurations, Setup Puppet

This blog explains how an agent can login to a queue directly through an extension. In asterisk’s terms an agent is human and an extension is device. Here an agent will be able to login into the queue through an extension by dialing a queue login number and you do not need to make any configurations in agents.conf or queues.conf.  For this setup to work properly, you need to have the correct dialplans in corresponding  extension files.

In the case of asteriskNOW, extension_additional.conf contains dialplans to handle  queue login.

[ext-queues]
exten => 5000*,1,Macro(agent-add,5000,)
exten => 5000**,1,Macro(agent-del,5000,5000)

If [macro-agent-add] and [macro-agent-del] contexts are already present in extensions_additional.conf you need to put the dialplans for these contexts in /etc/asterisk/extensions_override_freepbx.conf otherwise you can add the dialplans in /etc/asterisk/extensions_custom.conf.

[macro-agent-add]
include => macro-agent-add-custom
exten => s,1,Wait(1)
exten => s,n,Macro(user-callerid,SKIPTTL)
exten => s,n,Set(CALLBACKNUM=${AMPUSER})
exten => s,n,AddQueueMember(${ARG1},Local/${CALLBACKNUM}@from-internal/n)
exten => s,n,UserEvent(Agentlogin,Agent: ${CALLBACKNUM})
exten => s,n,Wait(1)
exten => s,n,Playback(agent-loginok&with&extension)
exten => s,n,SayDigits(${CALLBACKNUM})
exten => s,n,Hangup
exten => s,n,MacroExit()
exten => s,n(invalid),Playback(pbx-invalid)
exten => s,n,Goto(a3)

; end of [macro-agent-add]
[macro-agent-del]
include => macro-agent-del-custom
exten => s,1,Wait(1)
exten => s,n,Macro(user-callerid,SKIPTTL)
exten => s,n,Set(CALLBACKNUM=${AMPUSER})
exten => s,n,ExecIf($["${CALLBACKNUM}" = ""],Set,CALLBACKNUM=${CALLERID(number)})
exten => s,n,RemoveQueueMember(${ARG1},Local/${CALLBACKNUM}@from-internal/n)
exten => s,n,UserEvent(RefreshQueue)
exten => s,n,Wait(1)
exten => s,n,Playback(agent-loggedoff&with&extension)
exten => s,n,SayDigits(${CALLBACKNUM})
exten => s,n,Hangup
; end of [macro-agent-del]

In this dialplan 5000 is the queue number. You can use any queue number instead of it. To login to queue dial 5000* from your phone (either a softphone like xlite or a hard phone like Linksys) and you will hear a login confirmation and to logout from the queue you need to dial 5000** as per the dial plan. The advantage of this method is that an agent can do a login or logout by just dialing the corresponding number. Once logged in we can check current members in the queue using the command.

From shell
#asterisk -rx "queue show 5000"   (5000 is the queue number)
From asterisk CLI
> queue show 5000

Tino Thomas linux asterisk, asterisk dynamic agent, asteriskNOW, asterisknow dynamic agent, asterisknow dynamic agent login, dynamic agent, dynamic agent login, freepbx dynamic agent

Customize call recording location in asteriskNow.

By default AsteriskNOW saves call recordings under the directory /var/spool/asterisk/monitor. This way of call recording has got some disadvantages since number of files can grow beyond a limit . You will find it difficult to recognize call recordings related to a particular extension. A solution to this issue is to customize the call recording location. You need to override the dialplan for the context [macro-record-enable] present in /etc/asterisk/extensions_additional.conf. To override these dialplans you need to add new dialplan in a file called /etc/asterisk/extensions_override_freepbx.conf.  Following is the default dialplan for the context [macro-record-enable] present in /etc/asterisk/extensions_additional.conf.

[macro-record-enable]
include => macro-record-enable-custom
exten => s,1,GotoIf($["${BLINDTRANSFER}" = ""]?check)
exten => s,n,ResetCDR(w)
exten => s,n,StopMixMonitor()
exten => s,n(check),AGI(recordingcheck,${STRFTIME(${EPOCH},,%Y%m%d-%H%M%S)},${UNIQUEID})
exten => s,n,MacroExit()
exten => s,1+998(record),MixMonitor(${MIXMON_DIR}${CALLFILENAME}.${MIXMON_FORMAT},,${MIXMON_POST})
; end of [macro-record-enable]

You need to add the following dialplan in /etc/asterisk/extensions_override_freepbx.conf, which will override the above dialplan to customize the call recording location.

[macro-record-enable]
include => macro-record-enable-custom
exten => s,1,GotoIf($["${BLINDTRANSFER}" = ""]?check)
exten => s,n,ResetCDR(w)
exten => s,n,StopMixMonitor()
exten => s,n(check),AGI(recordingcheck,${STRFTIME(${EPOCH},,%Y%m%d-%H%M%S)},${UNIQUEID})
exten => s,n,MacroExit()
exten => s,n(record),System(/bin/mkdir -p /var/spool/asterisk/monitor/freepbx/${STRFTIME(${EPOCH},,%Y)}/${STRFTIME(${EPOCH},
,%m)}/${STRFTIME(${EPOCH},,%d)}/${ARG1}/${UNIQUEID})
exten => s,n,MixMonitor(/var/spool/asterisk/monitor/freepbx/${STRFTIME(${EPOCH},,%Y)}/${STRFTIME(${EPOCH},,%m)}/${STRFTIME($
{EPOCH},,%d)}/${ARG1}/${UNIQUEID}/${CALLFILENAME}.${MIXMON_FORMAT},,${MIXMON_POST})

The changes made are

exten => s,n(record),System(/bin/mkdir -p /var/spool/asterisk/monitor/freepbx/${STRFTIME(${EPOCH},,%Y)}/${STRFTIME(${EPOCH},
,%m)}/${STRFTIME(${EPOCH},,%d)}/${ARG1}/${UNIQUEID})

Here we are using ‘System’ command to create directories during the call recording. Inside the system command we can execute the unix shell commands. Here we used ‘mkdir’ command to create directories followed by a number of variables in asterisk. Here ‘STRFTIME’ used to get the year, month & date . ${ARG1} used to get the asterisk user extension ,who is participating in that call ;${UNIQUEID} to get the unique id of that call. UNIQUEID is assigned by asterisk and it is the unix time of orgination of that call.

<pre>exten => s,n,MixMonitor(/var/spool/asterisk/monitor/freepbx/${STRFTIME(${EPOCH},,%Y)}/${STRFTIME(${EPOCH},,%m)}/${STRFTIME($
{EPOCH},,%d)}/${ARG1}/${UNIQUEID}/${CALLFILENAME}.${MIXMON_FORMAT},,${MIXMON_POST})</pre>

In succeeding priority we use MixMonitor command to record the conversations in that channel. The format of MixMonitor command is MixMonitor(<file>.<ext>[|<options>[|<command>]]). Here in the ‘filename’ field we will give path of created directory in the preceding line.

Once you made the above changes you need to reload the dialplan through either of the following two ways.

from shell

#asterisk -rx “dialplan reload”

from asterisk CLI

> dialplan reload

Tino Thomas linux asterisk, asterisk call recording, asteriskNOW, asteriskNOW call recording, call recording location, customize call recording, freepbx, freepbx call recording