Shared Network Storage with iSCSI and OCFS2

So we got a bunch of new hardware at work recently to build a crunch farm for all our heavyweight data processing. Part of that system is two very beefy servers which share a SAN (This one for those interested) for the majority of their disk storage. The SAN uses iSCSI, which was fairly straightforward to set up (I’ll document it here anyway) so I got that all set up and then made a nice big ext3 partition for the servers to share. All so far so good, the servers were talking to the SAN, could see the partition, read and write to it etc. The only problem seemed to be that when one server changed a file, the other server wouldn’t pick up the change until the partition had been re-mounted. What I hadn’t accounted for was that ext3 doesn’t expect multiple machines to share the same block device so it wasn’t synching changes.

I knew that filesystems designed for exactly this sort of sharing were available but hadn’t done much with them but after investigating for a bit, it seemed like the Oracle Clustered File System (linky) was the best option as it was already supported by the Linux kernel and was pretty mature code. The main problem I had in setting all of this up was that the available documentation was very much geared towards people who already had in-depth experience of OCFS whereas I’d never used it before. Hence this blog post, which details setting up iSCSI and then configuring both servers to talk to the same OCFS partition. The instructions are written for Ubuntu Server, but will work on any distro which used apt. Packages are also available for rpm distros, the only instructions you need to change are the package fetching ones.

Setting up iSCSI

* Install Open-iSCSI

apt-get install open-iscsi

* Edit the Open-iSCSI configuration file

The default configuration file could be located at /etc/openiscsi/iscsid.conf or ~/.iscsid.conf. Open the file and set the parameters as required by your iSCSI device. I’ve included the (mostly default) options I used for reference

node.startup = automatic node.session.timeo.replacement_timeout = 120 node.conn[0].timeo.login_timeout = 15 node.conn[0].timeo.logout_timeout = 15 node.conn[0].timeo.noop_out_interval = 10 node.conn[0].timeo.noop_out_timeout = 15 node.session.iscsi.InitialR2T = No node.session.iscsi.ImmediateData = Yes node.session.iscsi.FirstBurstLength = 262144 node.session.iscsi.MaxBurstLength = 16776192 node.conn[0].iscsi.MaxRecvDataSegmentLength = 65536

* Save and close the file. Restart the open-iscsi service:

/etc/init.d/open-iscsi restart

Now you need to run a discovery against the iscsi target host which basically finds all the iSCSI targets the SAN can give us:

iscsiadm -m discovery -t sendtargets -p ISCSI-SERVER-IP-ADDRESS

Finally restart the service again:

/etc/init.d/open-iscsi restart

Now you should see an additional drive on the system such as /dev/sdc. Look in the /var/log/messages file to find out device name:

Next, you need to use fdisk to create a blank partition on the device. This is pretty well documented so I’ll skip these steps, other than to say that I’ll assume the device was called /dev/sdc, and the new blank partition is called /dev/sdc1 for the remainder of this post. So now we’re talking to our iSCSI device and we’ve got a blank partition all ready to format as an OCFS drive. Next, how exactly we do that!

To be continued…

3 thoughts on “Shared Network Storage with iSCSI and OCFS2”

  1. To be continued… need a follow up to this post… ๐Ÿ™‚

    I have always liked using ocfs2, its context dependant symbolic linking is great… and it does file images mounted on the loop device (limitation in GFS?)

    I have only really had much success with it using virtualisation… namely Xen… several virtual servers on the same host sharing the same real device… superb for an universal /home directory…

    The problems I have are when I start to put my devices on the network… cannot find really really good documentation anywhere…

    so far… tried:
    AOE = seems fine until heavy writes by 2 clients… and the host cannot mount it without complaints about another node heartbeating within its slot.
    NBD = same as the above
    GNBD = bloomin redhat cluster suite… not quite sure what im doing right or wrong… but used on its own with its -n flag… same as the above again!!
    ISCSI – using open-scsi Initiator for the clients and the server (target is server) …same again…

    Im not sure what I am doing wrong… should I not be mounting the device locally/shared with one of the above… would formatting using a shared device instead of locally make any difference (ie mkfs.ocfs2 /dev/[networked share] and not /dev/[real]…or is it that I need to tweak its default timeout settings… fencing seems to occur quite often… good ol STONITH and reboot…

    Just upgraded the network to base1000 so a lil more testing to do now… until then …keep us posted on your findings… ๐Ÿ™‚

  2. In response to your comment…

    I downloaded the OCFS2 users guide: http://oss.oracle.com/projects/ocfs2/dist/documentation/ocfs2_users_guide.pdf

    and it was pretty straightforward getting it setup

    i am sharing 1 LUN between 2 servers from an iSCSI SAN.
    First I get the server to see the iSCSI SAN, then I just followed the OCFS2 user’s guide, and it worked right off the bat.

    the only think i have yet to figure it out is how to get the ocfs2 partition mounted on boot

    if you have any questions, email me at hainest (at) gmail (dot) com

Leave a Reply to bitoffish Cancel reply

Your email address will not be published. Required fields are marked *