This post is focusing on DRBD management and basic command usage. I have talked about DRBD setup in one of my previous posts. If you want to set up DRBD on CentOS 6, you should read my post on “DRBD – How to configure DRBD on CentOS 6“. You can also download an automated CentOS DRBD installation bash script if you want!
DRBD® refers to block devices designed as a building block to form high availability (HA) clusters – Distributed Replicated Block Device This is done by mirroring a whole block device via an assigned network. DRBD can be understood as network based raid-1.
Let’s start with DRBD Management on CentOS 6 guide!
1. Check DRBD Status
The most important thing when talking about DRBD management is of course checking DRBD status. By checking DRBD status you can find out what states the connection and DRBD drives are in.
You can use one of the following commands to check for DRBD status:
[root@foo1 ~]# drbd-overview 1:disk1 Connected Primary/Secondary UpToDate/UpToDate C r----- /drbd1 ext4 1008M 18M 940M 2%
[root@foo1 ~]# /etc/init.d/drbd status drbd driver loaded OK; device status: version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49 m:res cs ro ds p mounted fstype 1:disk1 Connected Primary/Secondary UpToDate/UpToDate C /drbd1 ext4
[root@foo1 ~]# cat /proc/drbd version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:33264 nr:1048508 dw:1081772 dr:1385 al:14 bm:64 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
As we can see each command gives you a differently formated result. Choose the one you like the most or the one that gives you the information you need.
Quick explanation of the information you see when running DRBD status commands:
- Resource name (disk1 – this is the name of the configured DRBD resource)
- Connection State (Connected, Disconnected, Standalone,… – this is the state of the DRBD connection)
- Resource Role (Primary/Secondary, … – this is the role DRBD resource is in at the moment)
- Disk States (UpToDate/UpToDate, … – this is the state the DRBD resources are in)
- Mount Point (/drbd1 – this is the mount point DRBD resource is mounted at)
- Filesystem (ext4 – this is the filesystem DRBD resource is using)
- I/O State Flags (only /proc/drbd – this contains information about the current state of I/O operations associated with the resource)
- Performance indicators (only /prod/drbd – these are the performance indicatiors about sent and recieved data, read and written data and so on)
More detailed explanation about DRBD status information can be found HERE.
2. Enable and Disable DRBD Resource
DRBD maganement also means you sometimes want to disable and enable your DRBD resources. Of course you can manually enable and disable desired DRBD resources with the “drbdadm up” and “drbdadm down” commands:
[root@foo1 ~]# drbdadm down disk1 [root@foo1 ~]# cat /proc/drbd version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:Unconfigured
[root@foo1 ~]# drbdadm up disk1 [root@foo1 ~]# cat /proc/drbd version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r----- ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
3. Connecting and Disconnecting DRBD Resources
A big part of DRBD management is connecting and disconnecting DRBD resources. You can easily connect or disconnect DRBD resource using “drbdadm connect” and “drbdadm disconnect” command:
[root@foo2 ~]# drbdadm disconnect disk1 [root@foo2 ~]# cat /proc/drbd version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r----- ns:1048508 nr:33268 dw:33268 dr:1049172 al:0 bm:64 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@foo2 ~]# drbdadm connect disk1 [root@foo2 ~]# cat /proc/drbd version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----- ns:0 nr:0 dw:33268 dr:1049172 al:0 bm:64 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
3. Manual Fail-Over to Secondary Node (changing from Primary -> Secondary)
Sometines DRBD management means you need to Fail-Over from “Primary” to “Secondary” node or vice-versa.
Let’s say we configured DRBD resource called “disk1” and it’s currently in “Primary” role on node “foo1“, mounted on “/drbd1” mount point. Here is how you can quickly demote and mount DRBD resource “disk1” on node “foo2“.
First we need to umount “/drbd1” on node “foo1“:
[root@foo1 ~]# umount /drbd1
Next we demote “disk1” DRBD resource from “Primary” to “Secondary” role:
[root@foo1 ~]# drbdadm secondary disk1 [root@foo1 ~]# cat /proc/drbd version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r----- ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Now we can go to node “foo2” and promote DRBD resource “disk1” to “Primary” role:
[root@foo2 ~]# drbdadm primary disk1 [root@foo2 ~]# cat /proc/drbd version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:0 nr:0 dw:33268 dr:1049836 al:0 bm:64 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
We can now mount our disk on Primary node “foo2” and we are done:
[root@foo1 ~]# mount /dev/drbd1 /drbd1
4. Manual Split Brain Recovery
Let’s say we had a network problem which caused our DRBD fall into a “Split Brain” mode. If this happened we can find something like “kernel: block drbd1: Split-Brain detected but unresolved, dropping connection!” in /var/log/messages. This error message calls for DRBD management action. 🙂
The DRBD connection state is probably in “StandAlone” state and node “foo1” is in “Primary” role, node “foo2” in “Secondary” role (the situation might be different!):
[root@foo1 ~]# cat /proc/drbd version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r----- ns:9556 nr:2528 dw:20980 dr:190479 al:20 bm:9 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:4804
[root@foo2 ~]# cat /proc/drbd version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r----- ns:0 nr:9556 dw:10912 dr:13470 al:13 bm:4 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:672
First of all we need to check that exactly the desired resource/node is in “Primary” role (in our case this fine – foo1 is in “Primary” role). We also need to make sure that the second node is in “Secondary” role!!
Then we put our “Primary” node into a “WFConnection” DRBD connection state by connecting to our DRBD resource:
[root@foo1 ~]# drbdadm connect disk1 [root@foo1 ~]# cat /proc/drbd version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r----- ns:0 nr:0 dw:20984 dr:190479 al:20 bm:9 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:4804
We must connect to our DRBD resource from the “Secondary” also, but with additional parameter “–discard-my-data“ which means that inconsistent data will be discarded from “Secondary” node!!
[root@foo2 ~]# drbdadm connect --discard-my-data disk1 [root@foo2 ~]# cat /proc/drbd version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----- ns:0 nr:4808 dw:15720 dr:13470 al:13 bm:17 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
..and voila, the DRBD connection state says “Connected” and disk states are in “UpToDate/UpToDate” state.
You can also automate Split Brain recovery but it is not recomended. You can read more about Split Brain Recovery Automation HERE.
5. Resolve “Inconsistent/Inconsistent” state
You can get into a situation where your DRBD nodes are in “Inconsistent/Inconsistent” state. If you try to set any of the nodes into a “Primary” mode you get an error:
[root@foo1 ~]# drbdadm primary drbd1 1: State change failed: (-2) Need access to UpToDate data Command 'drbdsetup 1 primary' terminated with exit code 17
In order to resolve this error you need to choose which node to make “Primary” and run the following command:
[root@foo1 ~]# drbdadm -- --overwrite-data-of-peer primary drbd1
After running this command you will see, the data will be synced and the state resolved to “UpToDate/UpToDate“:
[root@foo1 ~]# cat /proc/drbd version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r---n- ns:80516 nr:0 dw:0 dr:82584 al:0 bm:4 lo:54 pe:17 ua:64 ap:0 ep:1 wo:f oos:970044 [>...................] sync'ed: 7.9% (970044/1048508)K finish: 0:00:24 speed: 39,232 (39,232) K/sec [root@foo1 ~]# cat /proc/drbd version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by phil@Build32R6, 2012-12-20 20:23:49
1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1048508 nr:0 dw:0 dr:1049172 al:0 bm:64 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
These are the basic DRBD management commands you might use. If you want more in-depth knowledge you should read DRBD Users Guide HERE.