This is the second part of my “Linux Cluster” posts:
- Linux Cluster Part 1 – Install Corosync and Pacemaker on CentOS 6 – Learn how to install Corosync and Pacemaker on CentOS 6
- Linux Cluster Part 2 – Adding and Deleting Cluster Resources – Learn how to add and delete Linux Cluster Resources and how to use CRM Shell
- Linux Cluster Part 3 – Manage Cluster Nodes and Resources – Learn how to manage Linux Cluster Nodes (maintenance mode, standby mode, …) and Linux Cluster Resources (resource constraints – order, colocation, …)
1. CRM Shell
CRM Shell is a command line interface to configure and manage Pacemaker. The CRM Shell should be installed on all your nodes, you can install it from HA-Clustering Repository. Add the following lines to “/etc/yum.repos.d/ha-clustering.repo” file:
[haclustering] name=HA Clustering baseurl=http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/ enabled=1 gpgcheck=0
Once installed we can run “crm” command from linux command line and manage our Pacemaker instance. Below is an example of running “crm help” command. If you want help on additional “crm” commands run for example “crm cib help “:
[root@foo1 ~]# crm help This is crm shell, a Pacemaker command line interface. Available commands: cib manage shadow CIBs resource resources management configure CRM cluster configuration node nodes management options user preferences history CRM cluster history site Geo-cluster support ra resource agents information center status show cluster status help,? show help (help topics for list of topics) end,cd,up go back one level quit,bye,exit exit the program
View Linux Cluster Status
[root@foo1 ~]# crm status Last updated: Mon Oct 7 13:41:11 2013 Last change: Mon Oct 7 13:41:08 2013 via crm_attribute on foo1.geekpeek.net Stack: classic openais (with plugin) Current DC: foo1.geekpeek.net - partition with quorum Version: 1.1.9-2.6-2db99f1 2 Nodes configured, 2 expected votes 0 Resources configured. Online: [ foo1.geekpeek.net foo2.geekpeek.net ]
View Linux Cluster Configuration
[root@foo1 ~]# crm configure show node foo1.geekpeek.net node foo2.geekpeek.net property $id="cib-bootstrap-options" dc-version="1.1.9-2.6-2db99f1" cluster-infrastructure="classic openais (with plugin)" expected-quorum-votes="2"
2. Adding Cluster Resources
Every cluster resource is defined by a Resource Agent. Resource Agents must provide Linux Cluster with a complete resource status and availability at any time! The most important and most used Resource Agent classes are:
- LSB (Linux Standard Base) – These are common cluster resource agents found in /etc/init.d directory (init scripts).
- OCF (Open Cluster Framework) – These are actually extended LSB cluster resource agents and usually support additional parameters
From this we can presume it is always better to use OCF (if available) over LSB Resource Agents since OCF support additional configuration parameters and are optimized for Cluster Resources.
We can check for available Resource Agents by running the “crm ra list” and the desired resource agent:
[root@foo1 ~]# crm ra list lsb auditd blk-availability corosync corosync-notifyd crond halt ip6tables iptables iscsi iscsid killall logd lvm2-lvmetad lvm2-monitor mdmonitor multipathd netconsole netfs network nfs nfslock pacemaker postfix quota_nld rdisc restorecond rpcbind rpcgssd rpcidmapd rpcsvcgssd rsyslog sandbox saslauthd single sshd udev-post winbind
[root@foo1 ~]# crm ra list ocf ASEHAagent.sh AoEtarget AudibleAlarm CTDB ClusterMon Delay Dummy EvmsSCC Evmsd Filesystem HealthCPU HealthSMART ICP IPaddr IPaddr2 IPsrcaddr IPv6addr LVM LinuxSCSI MailTo ManageRAID ManageVE NodeUtilization Pure-FTPd Raid1 Route SAPDatabase SAPInstance SendArp ServeRAID SphinxSearchDaemon Squid Stateful SysInfo SystemHealth VIPArip VirtualDomain WAS WAS6 WinPopup Xen Xinetd anything apache apache.sh asterisk clusterfs.sh conntrackd controld db2 dhcpd drbd drbd.sh eDir88 ethmonitor exportfs fio fs.sh iSCSILogicalUnit iSCSITarget ids ip.sh iscsi jboss ldirectord lvm.sh lvm_by_lv.sh lvm_by_vg.sh lxc mysql mysql-proxy mysql.sh named named.sh netfs.sh nfsclient.sh nfsexport.sh nfsserver nfsserver.sh nginx o2cb ocf-shellfuncs openldap.sh oracle oracledb.sh orainstance.sh oralistener.sh oralsnr pgsql ping pingd portblock postfix postgres-8.sh pound proftpd remote rsyncd rsyslog samba.sh script.sh scsi2reservation service.sh sfex slapd smb.sh svclib_nfslock symlink syslog-ng tomcat tomcat-5.sh tomcat-6.sh varnish vm.sh vmware zabbixserver
We configure cluster resources with “crm configure primitive” command following by a Resource Name, Resource Agent and Additional Parameters (example):
crm configure primitive resourcename resourceagent parameters
We can see HELP and additional Resource Agent parameters by running “crm ra meta” command following by a resource name (example):
[root@foo1 ~]# crm ra meta IPaddr2
Before we start adding Resources to our Cluster we need to disable STONITH (Shoot The Other Node In The Head) – since we are not using it in our configuration:
[root@foo1 ~]# crm configure property stonith-enabled=false
We can check the Linux Cluster configuration by running “crm configure show” command:
[root@foo1 ~]# crm configure show node foo1.geekpeek.net node foo2.geekpeek.net property $id="cib-bootstrap-options" dc-version="1.1.9-2.6-2db99f1" cluster-infrastructure="classic openais (with plugin)" expected-quorum-votes="2" stonith-enabled="false"
..to confirm STONITH was disabled!
Adding IP Address Resource
Let’s add IP address resource to our Linux Cluster. The information we need to configure IP address is:
Cluster Resource Name: ClusterIP
Resource Agent: ocf:heartbeat:IPaddr2 (get this info with “crm ra meta IPaddr2”)
IP address: 192.168.1.150
Netmask: 24
Monitor interval: 30 seconds (get this info with “crm ra meta IPaddr2”)
Run the following command on a Linux Cluster node to configure ClusterIP resource:
[root@foo1 ~]# crm configure primitive ClusterIP ocf:heartbeat:IPaddr2 params ip=192.168.1.150 cidr_netmask="24" op monitor interval="30s"
Check Cluster Configuration with:
[root@foo1 ~]# crm configure show node foo1.geekpeek.net node foo2.geekpeek.net primitive ClusterIP ocf:heartbeat:IPaddr2 params ip="192.168.61.150" cidr_netmask="24" op monitor interval="30s" property $id="cib-bootstrap-options" dc-version="1.1.9-2.6-2db99f1" cluster-infrastructure="classic openais (with plugin)" expected-quorum-votes="2" stonith-enabled="false" last-lrm-refresh="1381240623"
Check Cluster Status with:
[root@foo1 ~]# crm status Last updated: Tue Oct 8 15:59:19 2013 Last change: Tue Oct 8 15:58:11 2013 via cibadmin on foo1.geekpeek.net Stack: classic openais (with plugin) Current DC: foo1.geekpeek.net - partition with quorum Version: 1.1.9-2.6-2db99f1 2 Nodes configured, 2 expected votes 1 Resources configured. Online: [ foo1.geekpeek.net foo2.geekpeek.net ] ClusterIP (ocf::heartbeat:IPaddr2): Started foo1.geekpeek.net
As we can see a new resource called ClusterIP is configured in the Cluster and started on foo1.geekpeek.net node.
Adding Apache (httpd) Resource
Next resource is an Apache Web Server. Prior to Apache Cluster Resource Configuration, httpd package must be installed and configured on both nodes! The information we need to configure Apache Web Server is:
Cluster Resource Name: Apache
Resource Agent: ocf:heartbeat:apache (get this info with “crm ra meta apache”)
Configuration file location: /etc/httpd/conf/httpd.conf
Monitor interval: 30 seconds (get this info with “crm ra meta apache”)
Start timeout: 40 seconds (get this info with “crm ra meta apache”)
Stop timeout: 60 seconds (get this info with “crm ra meta apache”)
Run the following command on a Linux Cluster node to configure Apache resource:
[root@foo1 ~]# crm configure primitive Apache ocf:heartbeat:apache params configfile=/etc/httpd/conf/httpd.conf op monitor interval="30s" op start timeout="40s" op stop timeout="60s"
Check Cluster Configuration with:
[root@foo1 ~]# crm configure show node foo1.geekpeek.net node foo2.geekpeek.net primitive Apache ocf:heartbeat:apache params configfile="/etc/httpd/conf/httpd.conf" op monitor interval="30s" op start timeout="40s" interval="0" op stop timeout="60s" interval="0" meta target-role="Started" primitive ClusterIP ocf:heartbeat:IPaddr2 params ip="192.168.61.150" cidr_netmask="24" op monitor interval="30s" property $id="cib-bootstrap-options" dc-version="1.1.9-2.6-2db99f1" cluster-infrastructure="classic openais (with plugin)" expected-quorum-votes="2" stonith-enabled="false" last-lrm-refresh="1381240623"
Check Cluster Status with:
[root@foo1 ~]# crm status Last updated: Thu Oct 10 11:13:59 2013 Last change: Thu Oct 10 11:07:38 2013 via cibadmin on foo1.geekpeek.net Stack: classic openais (with plugin) Current DC: foo1.geekpeek.net - partition with quorum Version: 1.1.9-2.6-2db99f1 2 Nodes configured, 2 expected votes 2 Resources configured. Online: [ foo1.geekpeek.net foo2.geekpeek.net ] ClusterIP (ocf::heartbeat:IPaddr2): Started foo1.geekpeek.net Apache (ocf::heartbeat:apache): Started foo2.geekpeek.net
As we can see both Cluster Resources (Apache and ClusterIP) are configured and started – ClusterIP is started on foo1.geekpeek.net Cluster node and Apache is started on foo2.geekpeek.net node.
Apache and ClusterIP are at the moment running on different Cluster nodes but we will fix this later, setting Resource Constraints like: colocation (colocating resources), order (order in which resources start and stop), …
Resource Constraints will be explained in detail in the next “Linux Cluster Part 3” post!
3. Deleting Cluster Resources
We can delete the configured Cluster Resources with “crm configure delete” command following by a Resource Name we want to delete (example:)
crm configure delete resourcename
We must always stop the Cluster Resource prior to deleting it!!
We can stop the Resource by running “crm resource stop” command following by a Resource Name we want to stop.
Cluster Resource and Cluster Node management will be explained in detail in the next “Linux Cluster Part 3” post!
We can check the Linux Cluster configuration by running “crm configure show” command and see, if the Cluster Resource was successfully removed from Cluster Configuration.
Deleting Apache (httpd) Resource
Let’s stop and delete our Apache Cluster Resource configured in the steps above:
[root@foo1 ~]# crm resource stop Apache [root@foo1 ~]# crm configure delete Apache
Check Cluster Configuration with:
[root@foo1 ~]# crm configure show node foo1.geekpeek.net node foo2.geekpeek.net primitive ClusterIP ocf:heartbeat:IPaddr2 params ip="192.168.61.150" cidr_netmask="24" op monitor interval="30s" property $id="cib-bootstrap-options" dc-version="1.1.9-2.6-2db99f1" cluster-infrastructure="classic openais (with plugin)" expected-quorum-votes="2" stonith-enabled="false" last-lrm-refresh="1381240623"
… to confirm Apache resource was deleted from Cluster Configuration.
Deleting IP Address Resource
Next let’s stop and delete ClusterIP Resource:
[root@foo1 ~]# crm resource stop ClusterIP [root@foo1 ~]# crm configure delete ClusterIP
Check Cluster Configuration with:
[root@foo1 ~]# crm configure show node foo1.geekpeek.net node foo2.geekpeek.net property $id="cib-bootstrap-options" dc-version="1.1.9-2.6-2db99f1" cluster-infrastructure="classic openais (with plugin)" expected-quorum-votes="2" stonith-enabled="false" last-lrm-refresh="1381240623"
… to confirm the ClusterIP Resource was deleted from our Cluster Configuration.
Be sure to read the nex post Linux Cluster Part 3 – Manage Cluster Nodes and Resources (COMMING SOON!).
Here’s my latest book about High Availability on CentOS Linux
