Create the configuration directory, if it doesn't exist:
mkdir /etc/ocfs2
OCFS2 will then need to be configured. A sample of /etc/ocfs2/cluster.conf has been provided below:
node:
ip_port = 7777
ip_address = 10.0.0.1
number = 0
name = node01.internal.example.com
cluster = ocfs2
node:
ip_port = 7777
ip_address = 10.0.0.2
number = 1
name = node02.internal.example.com
cluster = ocfs2
node:
ip_port = 7777
ip_address = 10.0.0.3
number = 2
name = node03.internal.example.com
cluster = ocfs2
cluster:
node_count = 3
name = ocfs2
Please note:
1) The node names should match the hostnames of each machine and resolve to private IPs. If they don't, edit /etc/hosts with the appropriate entries, an example has been given below:
10.0.0.1 node01 node01.internal.example.com
10.0.0.2 node02 node02.internal.example.com
10.0.0.3 node03 node03.internal.example.com
2) The IP addresses should match the private IPs of each machine.3) This file MUST be the same across all nodes in this cluster.4) The cluster name needs to match the oc2b configuration file (ocfs2 is the default cluster name).
Starting O2BC (On All servers)
Now you will need to start the O2BC daemons on each server to start the cluster. This is done by running:
/etc/init.d/o2cb start
Loading module "configfs": OK
Mounting configfs filesystem at /sys/kernel/config: OK
Loading module "ocfs2_nodemanager": OK
Loading module "ocfs2_dlm": OK
Loading module "ocfs2_dlmfs": OK
Creating directory '/dlm': OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Starting O2CB cluster ocfs2: OK
Partitioning and Formatting the disk (iSCSI Target)
This should only be done on ONE machine, since we're setting up only one target. Verify which disk the iSCSI target is located as (/dev/sda in this case). If you select the wrong disk here, you will delete the data from another disk in your system.
First we will use parted to partition the disk (if this program isn't already installed, run yum install parted or apt-get install parted)
parted -s /dev/sda mklabel msdos
parted -s -- /dev/sda mkpart primary 0 -1
To view the new partition table, run:
fdisk -l /dev/sda
Disk /dev/sda: 8453 MB, 8453619712 bytes
255 heads, 63 sectors/track, 1027 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 1 1028 8255487+ 83 Linux
Now, we are going to format the disk with some sane default settings and give it a label of cluster-storage:
mkfs.ocfs2 -b 4k -C 32k -N4 -L cluster-storage /dev/sda1
mkfs.ocfs2 1.2.7
Filesystem label=cluster-storage
Block size=4096 (bits=12)
Cluster size=32768 (bits=15)
Volume size=8453586944 (257983 clusters) (2063864 blocks)
8 cluster groups (tail covers 32191 clusters, rest cover 32256 clusters)
Journal size=67108864
Initial number of node slots: 4
Creating bitmaps: done
Initializing superblock: done
Writing system files: done
Writing superblock: done
Writing backup superblock: 2 block(s)
Formatting Journals: done
Writing lost+found: done
mkfs.ocfs2 successful
If you require more than 4 nodes, change -N to a higher number than 4.This creates a file system with 4096 block size and 32768 (32k) cluster size.
Mounting the new partition (All servers)
We now need to have the partition table updated on all the servers in the cluster. In this case all the servers have /dev/sda as the iSCSI target.
We will run the following to re-read the partition:
blockdev --rereadpt /dev/sda
Next, we will want to create a mount point on the servers for this cluster.
mkdir /cluster-storage
Once the mount point is created, we will mount the partition. We are going to do this via its label we used in the format command. This will make it portable across all servers as the label will be the same, where the drive might be different (/dev/sda1, or /dev/sdb1, etc).
mount -L cluster-storage /cluster-storage
Next, we will want to make sure it mounted.
mount | grep ocfs
This should display:
ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw)
/dev/sda1 on /cluster-storage type ocfs2 (rw,_netdev,heartbeat=local)
This means the filesystem mounted on that server. Now, we will want to check the o2cb status on each server:
/etc/init.d/o2cb status
Module "configfs": Loaded
Filesystem "configfs": Mounted
Module "ocfs2_nodemanager": Loaded
Module "ocfs2_dlm": Loaded
Module "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Heartbeat dead threshold: 31
Network idle timeout: 30000
Network keepalive delay: 2000
Network reconnect delay: 2000
Checking O2CB heartbeat: Active
This shows that the clustering is on and active.You will need to verify this on every machine in the cluster.
Does it work?
The servers now have the filesystem mounted and o2cb says they are working correctly, so we need to test it by writing files.
On each server, we will want to echo to a file to see if it works correctly.With the first server in the cluster, do the following:
echo testing1 >> /cluster-storage/test.txt
On the next machine, lets see if we can read that file:
cat /cluster-storage/test.txt
This should give you:
testing1
Now, let us echo to it again from the second machine:
echo testing2 >> /cluster-storage/test.txt
Then, lets echo from the third machine:
echo testing3 >> /cluster-storage/test.txt
On the first machine, lets read the file again to see if all the updates were done correctly:
cat /cluster-storage/test.txt
This should show:
testing1
testing2
testing3
Your filesystem is now clustering correctly.
Automatic Mounting (on bootup, on each machine).
CentOS 5, CentOS 6:
To have the cluster mount at startup, we only need to make a few additions to your system. The first is to make sure the netfs service is set to start at bootup:
chkconfig -list netfs
This should show:
netfs0:off1:off2:off3:on4:on5:on6:off
If your current runlevel (more than likely runlevel 3) is listed as off, run the following:
chkconfig --level 3 on
This will enable the netfs service to start at bootup on the server.
CentOS 5, CentOS 6, Debian 6, Ubunau 10.04, Ubuntu 12.04:
Next, we will want to modify the /etc/fstab file to add the mount point. Here is an example for our configuration:
LABEL=cluster-storage /cluster-storage ocfs2 dev,defaults 0 0
This tells us to use the label on the partition to mount to /cluster-storage. We are telling it to figure out the file system type automatically (via the auto entry)._netdev is used to tell the system not to try to mount it until the network is brought online. By then, iscsid should also be online, and it will mount successfully. The last two fields are for the mount order and fsck orders.0 for both these fields is a good default in this case.
If the storage doesn't mount at boot, you may need to add this line to /etc/rc.local
mount -a
