Building up a High-Available Server-environment hosting Windows- and Linux-Server

Work in heavily progress - plz be patient

I will use the following software:

• Debian etch
• openQRM 3.1.4
• iSCSI Enterprise Target
• open-iSCSI
• drbd_0.7
• heartbeat
• mySQL 5.0
• ifenslave
• VMware Server 1.0.3
• XEN

Think of the High-End-Solutions which just cost a bunch of money.

Just a Short Example:

Today, fibre channel is one of the most popular solutions for shared storage. Fibre channel is a high-speed serial-transfer interface that is used to connect systems and storage devices in either point-to-point (FC-P2P), arbitrated loop (FC-AL), or switched topologies (FC-SW). Protocols supported by Fibre Channel include SCSI and IP. Fibre channel configurations can support as many as 127 nodes and have a throughput of up to 2.12 gigabits per second in each direction, and 4.25 Gbps is expected. Fibre channel, however, is very expensive.

With the popularity of Gigabit Ethernet and the demand for lower cost, Fibre Channel has recently been given a run for its money by iSCSI-based storage systems. Today, iSCSI SANs remain the leading competitor to FC SANs.

Ratified on February 11th, 2003 by the Internet Engineering Task Force (IETF), the Internet Small Computer System Interface, better known as iSCSI, is an Internet Protocol (IP)-based storage networking standard for establishing and managing connections between IP-based storage devices, hosts, and clients. iSCSI is a data transport protocol defined in the SCSI-3 specifications framework and is similar to Fibre Channel in that it is responsible for carrying block-level data over a storage network. Block-level communication means that data is transferred between the host and the client in chunks called blocks. High-Performance-Servers depend on this type of communication in order to work properly. Like a FC SAN, an iSCSI SAN should be a separate physical network devoted entirely to storage, however, its components can be much the same as in a typical IP network (LAN). Also consider that 10-Gigabit Ethernet is a reality today!

So we will follow these Rule:

C.R.E.A.M. - Cash Rulez Everything Around Me

Please send questions and critics to thohal@linuxfellaz.net until the work is in progress.

© by Thomas Halinka

Creating a dynamic scalable, flexible and high-available Server-Environment hosting virtualized Windows Server (VMware Server) and Linux-Server(XEN). I want to say first that this is not the only way of setting up such a system. There are many ways of achieving this goal but this is the way I take.

I love Debian GNU/Linux because of my enormous affinity towards this command apt-get. As a starter knowing this one command, It is so easy to install packages and you dont need to worry about package dependency and configuration.

The Debian-Packages are still in beta-phase - so use them only, when you what you’re doing

screenshot…..

To install such an Environment you will need the following:

• 2 Server-Systems used for HA-Cluster (iSCSI & openQRM)
• 2 Server-Systems to deploy HA-Windows-Guests
• 2 Server-Systems to deploy HA-Linux-Guests
• the Debian Etch Netinstall CD (the list of mirrors is available at Debian.org - I downloaded this one
• a fast internet connection (just for the Downloads)

Insert your Debian Etch Netinstall CD into your system and boot from it. Just install a “Standard System” on a small Partition (6 GB should be enough). We will store the DATA on a flexible Logical Volume (which you can expand later…) Also all the other required Packages would be installed later…

Regarding the partitioning, I use the following partition scheme:

/dev/sda1 – 100 MB /boot (primary, ext3, Bootable flag: on) /dev/sda5 – 5000 MB / (logical, ext3) /dev/sda6 – 1000 MB swap (logical) /dev/sda7 – 150 MB unmounted (logical, ext3) (will contain DRBD’s meta data) /dev/dm-0 – 260 GB unmounted (logical, ext3) (will contain the /data directory)

You can vary the sizes of the partitions depending on your hard disk size, and the names of your partition might also vary, depending on your hardware (e.g. you might have /dev/hda1 instead of /dev/sda1 and so on). However, it is important that /dev/sda7 has a little more than 128 MB because we will use this partition for DRBD’s meta data which uses 128 MB. Also, make sure /dev/sda7 as well as /dev/dm-0 are identical in size on MASTER and SLAVE, and please do not mount them (when the installer asks you:

No mount point is assigned for the ext3 file system in partition #7 of SCSI1 (0,0,0) (sda). Do you want to return to the partitioning menu?

please answer No)! /dev/md-0 is going to be our data partition (i.e., our NFS share).

I prefer LVM2 so i got md-0. LVM2 refers to a new userspace toolset that provide logical volume management facilities on linux.

Create a Partition of the Type lvm (partition-ID “8e”)

cfdisk /dev/hdX

Create the Physical-Volume

pvcreate /dev/hdXx

And another one (optional)

pvcreate /dev/hdYy

And finally create the Volume-Group “vol”

vgcreate vol /dev/hdXx /dev/hdYy

I’ve created one Volume-Group and one Logical-Volume and left some space for snapshotz…. Don’t set any Mountpoint for this Volume! DRBD will mount this drive later.

After the basic installation make sure that you give MASTER and SLAVE static IP addresses.

Afterwards, you should check /etc/fstab on both systems. Mine looks like this on both systems:

# /etc/fstab: static file system information.
#
#                
proc            /proc           proc    defaults        0       0
/dev/sda5       /               ext3    defaults,errors=remount-ro 0       1
/dev/sda1       /boot           ext3    defaults        0       2
/dev/sda6       none            swap    sw              0       0
/dev/hdc        /media/cdrom0   iso9660 ro,user,noauto  0       0
/dev/fd0        /media/floppy0  auto    rw,user,noauto  0       0

It’s important that MASTER and SLAVE have the same system time. Therefore we install an NTP client on both:

MASTER/SLAVE:

apt-get install ntp ntpdate

Afterwards you can check that both have the same time by running

MASTER/SLAVE:

date

Next we install the NFS server on MASTER and SLAVE:

MASTER/SLAVE:

apt-get install nfs-kernel-server

Then we remove the system bootup links for NFS because NFS will be started and controlled by heartbeat in our setup:

MASTER/SLAVE:

update-rc.d -f nfs-kernel-server remove
update-rc.d -f nfs-common remove

We want to export the directory / (this will be our NFS share that our nodes will use), so we edit /etc/exports on server1 and server2. It should contain only the following line:

MASTER/SLAVE:

# cat /etc/exports:

/data/export/ 192.168.178.0/255.255.255.0(rw,no_root_squash,no_all_squash,sync)

This means that / will be accessible by all systems from the 192.168.178.x subnet. See

# man 5 exports

to learn more about this.

Later we will create /data/exports on our empty (and still unmounted!) partition /dev/md-0.

Next we install DRBD on MASTER and SLAVE:

MASTER/SLAVE:

apt-get install kernel-headers-`uname -r` drbd0.7-module-source drbd0.7-utils
cd /usr/src/
tar xvfz drbd0.7.tar.gz
cd modules/drbd/drbd
make
make install

Then edit /etc/drbd.conf on MASTER and SLAVE. It must be identical on both systems and looks like this:

MASTER/SLAVE:

# cat /etc/drbd.conf:

resource r0 {
 protocol C;
 incon-degr-cmd "halt -f";

 startup {
    degr-wfc-timeout 120;    # 2 minutes.
  }

  disk {
    on-io-error   detach;
  }

  net {

  }

  syncer {

    rate 10M;

    group 1;

    al-extents 257;
  }

 on MASTER {                # ** EDIT ** the hostname of MASTER (uname -n)
   device     /dev/drbd0;        #
   disk       /dev/dm-0;         # ** EDIT ** data partition on MASTER
   address    192.168.2.1:7788; # ** EDIT ** IP address on MASTER
   meta-disk  /dev/sda7[0];      # ** EDIT ** 128MB partition for DRBD on MASTER
  }

 on SLAVE {                # ** EDIT ** the hostname of SLAVE (uname -n)
   device    /dev/drbd0;         #
   disk      /dev/md-0;          # ** EDIT ** data partition on SLAVE
   address   192.168.2.1:7788;  # ** EDIT ** IP address on SLAVE
   meta-disk /dev/sda7[0];       # ** EDIT ** 128MB partition for DRBD on SLAVE
  }

}

As resource name you can use whatever you like. Here it’s r0. Please make sure you put the correct hostnames of MASTER and SLAVE into /etc/drbd.conf. DRBD expects the hostnames as they are shown by the command

uname -n

If you have set MASTER and SLAVE respectively as hostnames during the basic Debian installation, then the output of uname -n should be MASTER and SLAVE.

Also make sure you replace the IP addresses and the disks appropriately. If you use /dev/hda instead of /dev/sda, please put /dev/hda8 instead of /dev/md-0 into /etc/drbd.conf (the same goes for the meta-disk where DRBD stores its meta data). /dev/md-0 (or /dev/hda8…) will be used as our NFS share later on.

Now we load the DRBD kernel module on both Servers. We need to do this only now because afterwards it will be loaded by the DRBD init script.

MASTER/SLAVE:

# modprobe drbd

Let’s configure DRBD:

MASTER/SLAVE:

# drbdadm up all
# cat /proc/drbd

The last command should show something like this (on both server1 and server2):

version: 0.7.10 (api:77/proto:74)
SVN Revision: 1743 build by phil@mescal, 2005-01-31 12:22:07
 0: cs:Connected st:Secondary/Secondary ld:Inconsistent
    ns:0 nr:0 dw:0 dr:0 al:0 bm:1548 lo:0 pe:0 ua:0 ap:0
 1: cs:Unconfigured

You see that both NFS servers say that they are secondary and that the data is inconsistant. This is because no initial sync has been made yet.

I want to make MASTER the primary server and SLAVE the “hot-standby”, If MASTER fails, SLAVE takes over, and if MASTER comes back then all data that has changed in the meantime is mirrored back from SLAVE to MASTER so that data is always consistent.

This next step has to be done only on MASTER!

MASTER:

# drbdadm -- --do-what-I-say primary all

Now we start the initial sync between server1 and server2 so that the data on both servers becomes consistent. On server1, we do this:

MASTER:

# drbdadm -- connect all

The initial sync is going to take a few hours (depending on the size of /dev/md-0 (/dev/hda8…)) so please be patient.

You can see the progress of the initial sync like this on server1 or server2:

MASTER/SLAVE:

# cat /proc/drbd

The output should look like this:

version: 0.7.10 (api:77/proto:74)
SVN Revision: 1743 build by phil@mescal, 2005-01-31 12:22:07
 0: cs:SyncSource st:Primary/Secondary ld:Consistent
    ns:13441632 nr:0 dw:0 dr:13467108 al:0 bm:2369 lo:0 pe:23 ua:226 ap:0
        [==========>.........] sync'ed: 53.1% (11606/24733)M
        finish: 1:14:16 speed: 2,644 (2,204) K/sec
 1: cs:Unconfigured

When the initial sync is finished, the output should look like this:

SVN Revision: 1743 build by phil@mescal, 2005-01-31 12:22:07
 0: cs:Connected st:Primary/Secondary ld:Consistent
    ns:37139 nr:0 dw:0 dr:49035 al:0 bm:6 lo:0 pe:0 ua:0 ap:0
 1: cs:Unconfigured

NFS stores some important information (e.g. information about file locks, etc.) in /var/lib/nfs. Now what happens if MASTER goes down? SLAVE takes over, but its information in /var/lib/nfs will be different from the information in MASTER’s /var/lib/nfs directory. Therefore we do some tweaking so that these details will be stored on our /data partition (/dev/md-0) which is mirrored by DRBD between MASTER and SLAVE. So if MASTER goes down SLAVE can use the NFS details of SLAVE.

MASTER/SLAVE:

mkdir /data

MASTER:

mount -t ext3 /dev/drbd0 /data
mv /var/lib/nfs/ /data/
ln -s /data/nfs/ /var/lib/nfs
mkdir /data/export
umount /data

SLAVE:

rm -fr /var/lib/nfs/
ln -s /data/nfs/ /var/lib/nfs

heartbeat is the control instance of this whole setup. It is going to be installed on MASTER and SLAVE, and it monitors the other server. For example, if MASTER goes down, heartbeat on SLAVE detects this and makes SLAVE take over. heartbeat also starts and stops the NFS server on both MASTER and SLAVE. It also provides NFS as a virtual service via the IP address 192.168.0.174 so that the nodes see only one NFS server.

First we install heartbeat:

MASTER/SLAVE:

apt-get install heartbeat

Now we have to create three configuration files for heartbeat. They must be identical on server1 and server2!

MASTER/SLAVE:

# /etc/heartbeat/ha.cf:

logfacility     local0
keepalive 2
#deadtime 30 # USE THIS!!!
deadtime 10
bcast   eth0
node MASTER SLAVE

As nodenames we must use the output of uname -n on MASTER and SLAVE.

MASTER/SLAVE:

# cat /etc/heartbeat/haresources:

server1  IPaddr::192.168.xxx.xxx/24/eth0 drbddisk::r0   Filesystem::/dev/drbd0::/data::ext3 nfs-kernel-server

The first word is the output of uname -n on MASTER, no matter if you create the file on MASTER or SLAVE! After IPaddr we put our virtual IP address 192.168.xxxx.xxx, and after drbddisk we use the resource name of our DRBD resource which is r0 here (remember, that is the resource name we use in /etc/drbd.conf - if you use another one, you must use it here, too).

MASTER/SLAVE:

# cat /etc/heartbeat/authkeys:

auth 3
3 md5 somerandomstring

somerandomstring is a password which the two heartbeat daemons on MASTER and SLAVE use to authenticate against each other. Use your own string here. You have the choice between three authentication mechanisms. I use md5 as it is the most secure one.

/etc/heartbeat/authkeys should be readable by root only, therefore we do this:

MASTER/SLAVE:

# chmod 600 /etc/heartbeat/authkeys

Finally we start DRBD and heartbeat on server1 and server2:

MASTER/SLAVE:

/etc/init.d/drbd start
/etc/init.d/heartbeat start

Now we can do our first tests.

ifconfig

In the output, the virtual IP address 192.168.xxx.xxx should be shown up.

After the Base-Installation you have to install some packages.

apt-get update && apt-get install ifenslave linux-headers-`uname -r` libssl-dev make gcc build-essential ruby lvm2 mysql-server

root-password for mysql has to be blank! So you have to flush the root-password if you set one!

# mysql -u root
mysql> USE mysql;
mysql> UPDATE user SET Password=PASSWORD('') WHERE user='root';
mysql> FLUSH PRIVILEGES;
mysql> quit

You need to modify your TRANSACTION ISOLATION LEVEL to READ COMMITED.

# mysql -u root
mysql> SET GLOBAL TRANSACTION ISOLATION LEVEL READ COMMITTED;
mysql> quit

And link mysql.sock

# ln -s /var/run/mysql/mysqld.sock /var/lib/mysql/mysql.sock

For Security-Issues: The network is disabled by default in debian

# cat /etc/mysql/my.cnf #!?

For higher Performance on the iSCSI-Target we will use network-bonding. You have to edit /etc/modutils/aliases

# vim /etc/modutils/aliases

alias bond0 bonding
alias eth0 tg3
alias eth1 tg3
alias eth2 tg3
options bonding mode=0 miimon=100

Then you have to edit /etc/modprobe.d/arch/i386

# vim /etc/modprobe.d/arch/i386

alias bond0 bonding
options bonding mode=0 miimon=100 downdelay=200 updelay=200

edit the Network-Interfaces. I use these ones in the Howto

# cat /etc/network/interfaces

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
#allow-hotplug eth0
iface eth0 inet static
   address 192.168.178.2
   netmask 255.255.255.0
   network 192.168.178.0
   broadcast 192.168.178.255
   gateway 192.168.178.1

# iSCSI-IF (IET)
auto bond0
iface bond0 inet static
   address 192.168.178.1
   netmask 255.255.255.0
   network 192.168.178.0
   broadcast 192.168.178.255
   gateway 192.168.178.1
   hwaddress ether 00:03:B3:48:50:2C
   post-up ifenslave bond0 eth0 eth1 eth2

# LAN-IF
iface eth0 inet static
   address 192.168.1.1
   netmask 255.255.255.0
   network 192.168.1.0
   broadcast 192.168.1.255
   gateway 192.168.1.1

and reboot now

reboot -n

Next we will setup the iSCSI-Target and an optionally iSCSI-Software-Initiator.

wget http://dfn.dl.sourceforge.net/sourceforge/iscsitarget/iscsitarget-0.4.15.tar.gz
tar xvf iscsitarget-0.4.15.tar.gz
cd iscsitarget-0.4.15
ln -s /usr/src/linux-headers-2.6.18-4-686/ /usr/src/linux
make KERNELSRC=/usr/src/linux
make KERNELSRC=/usr/src/linux install

A Minimal-config without Authentication - plz consider the Manual for further Configuration!

Create some Big Filez for the iSCSI-LUNs i.e. 40 GB

# dd if=/dev/zero of=/home/iSCSI/LUN0 bs=1024 count=40000

You can expand them later with the seek-option of dd.

Next we will export our LUNs

# cat /etc/ietd.conf

Target iqn.2001-04.com.example:storage.disk2.sys1.xyz 
Lun 0 Path=/home/iSCSI/LUN0,Type=fileio
Lun 1 Path=/home/iSCSI/LUN1,Type=fileio  
Lun 2 Path=/home/iSCSI/LUN2,Type=fileio

iSCSI-Initiator isn’t really needed on the QRM-Server but it makes live much easier

apt-get install openiscsi

iscsiadm -m discovery -tst -p xxxxxx -d3

iscsiadm -m node -T iqn.2001-04.com.example:storage.disk2.sys1.xyz -p   xxxxx -l

fdisk -l | grep sd*
Disk /dev/sda doesn't contain a valid partition table
..........
Disk /dev/sda: 751.6 GB, 751619276800 bytes
255 heads, 63 sectors/track, 91379 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
......

Here /dev/sda is the iscsi-target!