Share, , Google Plus, Pinterest,

Print

Posted in:

DRBD Management and Basic Command Usage on CentOS 6

This post is focusing on DRBD management and basic command usage. I have talked about DRBD setup in one of my previous posts. If you want to set up DRBD on CentOS 6, you should read my post on “DRBD – How to configure DRBD on CentOS 6“. You can also download an automated CentOS DRBD installation bash script if you want!

DRBD® refers to block devices designed as a building block to form high availability (HA) clusters – Distributed Replicated Block Device This is done by mirroring a whole block device via an assigned network. DRBD can be understood as network based raid-1.

DRBD scheme
DRBD management

Let’s start with DRBD Management on CentOS 6 guide!

1. Check DRBD Status

The most important thing when talking about DRBD management is of course checking DRBD status. By checking DRBD status you can find out what states the connection and DRBD drives are in.

You can use one of the following commands to check for DRBD status:

[root@foo1 ~]#  drbd-overview
1:disk1  Connected Primary/Secondary UpToDate/UpToDate C r----- /drbd1 ext4 1008M 18M 940M 2%
[root@foo1 ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.15 (api:88/proto:86-97)
GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
m:res    cs         ro                 ds                 p  mounted  fstype
1:disk1  Connected  Primary/Secondary  UpToDate/UpToDate  C  /drbd1   ext4
[root@foo1 ~]# cat /proc/drbd
version: 8.3.15 (api:88/proto:86-97)
GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:33264 nr:1048508 dw:1081772 dr:1385 al:14 bm:64 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

 

As we can see each command gives you a differently formated result. Choose the one you like the most or the one that gives you the information you need.

Quick explanation of the information you see when running DRBD status commands:

  • Resource name (disk1 – this is the name of the configured DRBD resource)
  • Connection State (Connected, Disconnected, Standalone,… – this is the state of the DRBD connection)
  • Resource Role (Primary/Secondary, … – this is the role DRBD resource is in at the moment)
  • Disk States (UpToDate/UpToDate, … – this is the state the DRBD resources are in)
  • Mount Point (/drbd1 – this is the mount point DRBD resource is mounted at)
  • Filesystem (ext4 – this is the filesystem DRBD resource is using)
  • I/O State Flags (only /proc/drbd – this contains information about the current state of I/O operations associated with the resource)
  • Performance indicators (only /prod/drbd – these are the performance indicatiors about sent and recieved data, read and written data and so on)

More detailed explanation about DRBD status information can be found HERE.

2. Enable and Disable DRBD Resource

DRBD maganement also means you sometimes want to disable and enable your DRBD resources. Of course you can manually enable and disable desired DRBD resources with the “drbdadm up” and “drbdadm down” commands:

[root@foo1 ~]# drbdadm down disk1
[root@foo1 ~]# cat /proc/drbd
version: 8.3.15 (api:88/proto:86-97)
GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:Unconfigured
[root@foo1 ~]# drbdadm up disk1
[root@foo1 ~]# cat /proc/drbd
version: 8.3.15 (api:88/proto:86-97)
GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

3. Connecting and Disconnecting DRBD Resources

A big part of DRBD management is connecting and disconnecting DRBD resources. You can easily connect or disconnect DRBD resource using “drbdadm connect” and “drbdadm disconnect” command:

[root@foo2 ~]# drbdadm disconnect disk1
[root@foo2 ~]# cat /proc/drbd
version: 8.3.15 (api:88/proto:86-97)
GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown   r-----
ns:1048508 nr:33268 dw:33268 dr:1049172 al:0 bm:64 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@foo2 ~]# drbdadm connect disk1
[root@foo2 ~]# cat /proc/drbd
version: 8.3.15 (api:88/proto:86-97)
GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:0 nr:0 dw:33268 dr:1049172 al:0 bm:64 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

3. Manual Fail-Over to Secondary Node (changing from Primary -> Secondary)

Sometines DRBD management means you need to Fail-Over from “Primary” to “Secondary” node or vice-versa.

Let’s say we configured DRBD resource called “disk1” and it’s currently in “Primary” role on node “foo1“, mounted on “/drbd1” mount point. Here is how you can quickly demote and mount DRBD resource “disk1” on node “foo2“.

First we need to umount “/drbd1” on node “foo1“:

[root@foo1 ~]# umount /drbd1

 

Next we demote “disk1” DRBD resource from “Primary” to “Secondary” role:

[root@foo1 ~]# drbdadm secondary disk1
[root@foo1 ~]# cat /proc/drbd
version: 8.3.15 (api:88/proto:86-97)
GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

 

Now we can go to node “foo2” and promote DRBD resource “disk1” to “Primary” role:

[root@foo2 ~]# drbdadm primary disk1
[root@foo2 ~]# cat /proc/drbd
version: 8.3.15 (api:88/proto:86-97)
GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:0 nr:0 dw:33268 dr:1049836 al:0 bm:64 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

 

We can now mount our disk on Primary node “foo2” and we are done:

[root@foo1 ~]# mount /dev/drbd1 /drbd1

4. Manual Split Brain Recovery

Let’s say we had a network problem which caused our DRBD fall into a “Split Brain” mode. If this happened we can find something like “kernel: block drbd1: Split-Brain detected but unresolved, dropping connection!” in /var/log/messages. This error message calls for DRBD management action. 🙂

The DRBD connection state is probably in “StandAlone” state and node “foo1” is in “Primary” role, node “foo2” in “Secondary” role (the situation might be different!):

[root@foo1 ~]# cat /proc/drbd
version: 8.3.15 (api:88/proto:86-97)
GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown   r-----
ns:9556 nr:2528 dw:20980 dr:190479 al:20 bm:9 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:4804
[root@foo2 ~]# cat /proc/drbd
version: 8.3.15 (api:88/proto:86-97)
GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown   r-----
ns:0 nr:9556 dw:10912 dr:13470 al:13 bm:4 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:672

 

First of all we need to check that exactly the desired resource/node is in “Primary” role (in our case this fine – foo1 is in “Primary” role). We also need to make sure that the second node is in “Secondary” role!!

Then we put our “Primary” node into a “WFConnection” DRBD connection state by connecting to our DRBD resource:

[root@foo1 ~]# drbdadm connect disk1
[root@foo1 ~]# cat /proc/drbd
version: 8.3.15 (api:88/proto:86-97)
GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r-----
ns:0 nr:0 dw:20984 dr:190479 al:20 bm:9 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:4804

 

We must connect to our DRBD resource from the “Secondary” also, but with additional parameter “–discard-my-data which means that inconsistent data will be discarded from “Secondary” node!!

[root@foo2 ~]# drbdadm connect --discard-my-data disk1
[root@foo2 ~]# cat /proc/drbd
version: 8.3.15 (api:88/proto:86-97)
GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:0 nr:4808 dw:15720 dr:13470 al:13 bm:17 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

 

..and voila, the DRBD connection state says “Connected” and disk states are in “UpToDate/UpToDate” state.

You can also automate Split Brain recovery but it is not recomended. You can read more about Split Brain Recovery Automation HERE.

5. Resolve “Inconsistent/Inconsistent” state

You can get into a situation where your DRBD nodes are in “Inconsistent/Inconsistent” state. If you try to set any of the nodes into a “Primary” mode you get an error:

[root@foo1 ~]# drbdadm primary drbd1
1: State change failed: (-2) Need access to UpToDate data
Command 'drbdsetup 1 primary' terminated with exit code 17

 

In order to resolve this error you need to choose which node to make “Primary” and run the following command:

[root@foo1 ~]# drbdadm -- --overwrite-data-of-peer primary drbd1

 

After running this command you will see, the data will be synced and the state resolved to “UpToDate/UpToDate“:

[root@foo1 ~]# cat /proc/drbd
version: 8.3.15 (api:88/proto:86-97)
GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r---n-
ns:80516 nr:0 dw:0 dr:82584 al:0 bm:4 lo:54 pe:17 ua:64 ap:0 ep:1 wo:f oos:970044
[>...................] sync'ed:  7.9% (970044/1048508)K
finish: 0:00:24 speed: 39,232 (39,232) K/sec
[root@foo1 ~]# cat /proc/drbd
version: 8.3.15 (api:88/proto:86-97)
GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:1048508 nr:0 dw:0 dr:1049172 al:0 bm:64 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

 

These are the basic DRBD management commands you might use. If you want more in-depth knowledge you should read DRBD Users Guide HERE.

  • Pingback: DRBD - How to configure DRBD on CentOS 6 GeekPeek.Net()

  • V Singh

    Really a nice post. Can you please post High avilbilty XenServer cluster with Two nodes?

    • Mitch

      Thanks V Singh. Thanks for your suggestion, i have already prepared the next few posts, but i will try to consider your suggestion on XenServer as soon as possible!

      It would be best you subscribe to GeekPeek.Net by email so you don’t miss the XenServer post! – Subscribe to GeekPeek.Net by Email

      Thanks,
      Mitch

  • Wilmer Canchica

    Necesito configurar el drbd para que mis disco sean activo/activo, he realizado los siguientes pasos en ambos nodos, pero tengo la duda para si cuando corro “cat /proc/drbd” en ambos nodos me indica secundario.

    Los pasos ejecutados hasta ahora son los siguiente:

    1.- Crear el archivo .res con los parametros para que sea activo/activo: /etc/drbd.d/clusterdb.res

    resource clusterdb
    {
    protocol C;
    startup {
    wfc-timeout 30;
    become-primary-on both;
    }
    disk {
    on-io-error detach;
    }
    net {
    allow-two-primaries;
    }
    syncer {
    rate 10M;
    }
    on CentOs6_Serv1 {
    device /dev/drbd0;
    disk /dev/sdb;
    address 192.168.56.101:7788;
    meta-disk internal;
    }
    on CentOs6_Serv2 {
    device /dev/drbd0;
    disk /dev/sdb;
    address 192.168.56.102:7788;
    meta-disk internal;
    }
    }
    2.- Ejecutar el fdisk para darle formato al disco en ambos nodos: “fdisk -cu /dev/sdb”
    3.- Crear las entradas de cada servidor en ambos nodos: vi /etc/hosts
    4.- Generar la meta-data en ambos nodos: drbdadm create-md clusterdb
    5.- Levantar el servicio en ambos nodos: drbdadm up clusterdb
    6.- Observa el status: cat /proc/drbd

    version: 8.4.4 (api:1/proto:86-101)
    GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by phil@Build64R6, 2013-10-14 15:33:06
    0: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/DUnknown C r—-s
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:4194140

    La principal duda que tengo es cuales pasos debo seguir para que ambos nodos queden activo/activo.

    • Mitch

      Hi Wilmer, first of all i don’t speak spanish so i don’t understand a thing you said 🙂 I see you want to configure active/active drbd. Are you using filesystem which supports locking? It seems you have a problem with the nodes connecting / seeing each other? Are your iptables properly configured? Can you telnet to DRBD port from primary to secondary and vice versa? Regards, Mitch

  • Thanks – Most helpful!