查看文章 |
Building a virtualized web-farm with openQRM(based on openQRM-version 3.1.4) by Matt Rechenburg Goal :Creating a dynamic scalable and flexible web-farm hosting virtualized “root” web-server. A common and required setup for ISP‘s which are selling dedicated web-servers with “root-access” to their customers is to create, manage and monitor a huge (and growing) number of virtualized systems. This How-To will describe how to setup such an environment using the openQRM data-center management platform. It will go through the full installation steps, deployment of the “virtualization hosts”, creation and administration of the “partitions” up to configuration-management, monitoring and high-availability. Terms :“filesystem-image” - A servers root-filesystem located on a storage server “boot-image” - A linux kernel “image” containing the kernel-file, the kernel-modules and an initrd. “virtual environment” (VE) - Virtual environemnts in openQRM are logical abstractions of services and how they should be deployed e.g. which kernel (boot-image), which rootfs (filesystem-image), single-server or cluster, special hardware profile, SLA’s etc. “virtualization host” - A server providing a virtualization technology (e.g. VMware, Xen, Linux-VServer, Qemu, ..). This is the system on which “partitions” are created on. “partition” - Partitions are slices of the “virtualization host”. They are the virtual machines started on behalf of the “virtualization host” providing a separated and isolated operation system. “resource” - Server-system, either physical or virtual. Systems added to openQRM are booting as “idle”. They get a small operation system stack per netboot and stay in this “available” status until they get provisioned (deployed) with some “virtual environment” (VE) Requirements :- one server-system dedicated for the openQRM-server - one server-system to deploy a “virtualization host” Optional :- one or more servers setup as dedicated storage-servers - one or more hot-standby’s to provide high-availablitiy for the openQRM-server - a dedicated server hosting the database for the openQRM-server - any number of additional systems to enlarge the number of “virtualization-hosts” Installation :1) choosing your preferred linux distribution openQRM supports many different linux distribution e.g. RHEL, FC, Suse, Centos, Debian, …. For this How-To i selected FC4 (Fedory Core 4) but it will (should) work with every other major/modern linux-distribution too. During the installation please configure a static ip-address for the first network-interface. In this How-To the ip of the openQRM-server will be “192.168.88.179” ! Please also make sure to disable the firewall and SELinux. 2) install the base operation system and the openQRM packages Please notice that this How-To will make use of the fast-cloning capabilities of LVM2. Please make sure to create a dedicated LVM volume-group for storing the server-images ! In this How-To this volume-group is named “vol” ! After installing the base operation system this How-To requires the following set of openQRM-packages to be installed : Required packages :openqrm-core-base openqrm-plugin-dhcpd openqrm-plugin-tftpd openqrm-plugin-xen openqrm-plugin-puppet openqrm-plugin-nagios openqrm-plugin-apache openqrm-plugin-lvm-mgmt openqrm-plugin-sshlogin openqrm-plugin-webmin Optional packages :openqrm-extras-mysql Package versions :For this How-To the current latest (stable) openQRM version 3.1.4 is used. Please find the above packages for the 3.1.4 release on the source-forge download section at : http://sourceforge.net/project/showfiles.php?group_id=153504 setting up openQRM :Ininitalyze and start openQRM by running the qrm-installer : cd /opt/qrm From the “Components” menu choose : apache In the “Config” menu choose : Basic Now select “Exit” and “Save”. The installation procedure will now configure and start the openQRM-sever. You can “watch” the start-up by running “tail -f /var/log/qrm/qrm.log” After startup you can access the openQRM-management console via a webbrowser at : http://[ip-address-of-the-openQRM-server-system] Please login as “qrm” with the password “qrm”. One of the first things you should do is to change this default password ! (Screenshot of the openQRM-management console after installation)
Booting a "resource" :To add a first “resource” to the openQRM-environment simply start a server via netbooting. To enable netbooting please set the systems bios to “PXE” The resource will boot-up via the network and being added to the openQRM-management GUI as an “idle” resource in maintainance state. New systems are “arriving” in openQRM in a maintainance mode which prevents them for direct provisioning. To make them available please disable the maintainance mode for the resource you would like to use for deployment. (Screenshot of the resource-overview, one “idle” system in maintainance-mode)
Please make sure to have the “maintainance mode” disabled for this “idle” system vi the openQRM-GUI at “Resources” - “Action” - “disable maintainance mode” It should look like this : (Screenshot of the resource-overview, one “idle” system with the maintainance-mode disabled)
Creating "filesystem-images" :For creating a “filesystem-image” first a storage-server needs to be defined in openQRM. In the management GUI please click on “Management Tools” - “Storage” - “Servers”. Then in the upper right menu select “Add new storage server”. Fill out the storage-server form, give a name e.g. “lvm-nfs”, select the “lvm-nfs” storage-server type, give the ip-address of the openQRM-server. Also you need to add the following environment variable for this storage-server : SERVER_VOL=vol Then submit to create and save this new storage-server. (Screenshot of the creation of a storage-server in openQRM)
btw: The new storage-server types “lvm-nfs” and “lvm-iscsi” are provided by the lvm-mgmt plugin. Both types are using the LVM2 snapshot-features to enable the fast cloning of filesystem-images. Since we want to use the fast-cloning features of the lvm-mgmt plugin we need to create an logical-volume in the volume-group dedicated for the filesystem-images. To automatically create a logical volume for the first filesystem-image please use the “qrm-lvm-manage-images” util provided by the lvm-mgmt plugin in the following way : [root@demo ~]# cd /opt/qrm/plugins/lvm-mgmt/bin/ This automatically created a logical-volume named “xen_host”,formatted it with ext3, added it to /etc/fstab and /etc/exports and reloaded the nfs-server so that the /vol/xen gets exported. Here to re-check everything went fine : [root@demo bin]# df /vol/xen_host The image-store was created correctly. Now lets fill it with content using the “qrm-filesystem-image” util : [root@demo ~]# cd /opt/qrm/bin/ The filesystem content was transferred successfully : [root@demo bin]# cd /vol/xen_host Now lets directly create another filesystem-image to later use as the web-server template for the end-users. The image-store for this fs-image will be created in the same way as for the xen_host image via the “qrm-lvm-manage-images” util : [root@demo ~]# cd /opt/qrm/plugins/lvm-mgmt/bin/ Now copy the filesytem-image content from the “xen_host” to the “webserver” : [root@demo ~]# cd /vol/webserver/ and add the new filesystem-image “webserver” to the openQRM-server via the qrm-cli : [root@demo ~]# Now lets take a look at the filesystem-image section in the openQRM-GUI : (Screenshot of the filesystem-images overview)
Deploying the "virtualization host" :It is now time to create a “virtual environment” for the “xen_host” and deploy it to the available “resource”. In the openQRM-GUI select “Virtual Environments” and click on “New virtual environment” in the upper right “tools” menu. Fill in the name for the new VE → “xen_host”. Enable “Partitioning” via the checkbox and select “Xen” as the virtualization-technology. As kernel-image please select the “Xen-Hypervisor” boot-image. As filesystem-image choose the “xen_host” image and enable the checkbox “Multi-Server” and “same for all”. With this setup the VE will start the “virtualization host” as a cluster using a shared root-filesystem (SSI - single system image). That means you can later simply add and remove “resources” to the “xen_host” cluster to scale up or down. (Screenshot of the creation of the xen_host VE)
Please notice :To be able to successfully deploy the “virtualization host” (xen_host) we must make sure that also non-PXE clients will get an ip-address from the dhcpd-server provided by the dhcpd-plugin. The default dhcpd-configuration of the dhcpd-plugin only servers PXE-clients by creating a class “NOTPXE” ! Please make sure to remove the following section from the dhcpd.conf file at /opt/qrm/plugins/dhcpd/etc/dhcpd.conf : --------------------------------- The dhcpd.conf file should now look similar to this one (possibly with other ip-addresses) : [root@demo ~]# cd /opt/qrm/plugins/dhcpd/etc/ After that please manually restart the dhcpd-plugin :
Now lets deploy this “xen_host” to the available “resource” by simply starting the VE. In the openQRM-GUI click on → “Virtual Environments” - “Actions” - “Start” The “idle” node will reboot now and start the “xen_host” VE. (Screenshot of the active xen_host VE)
Starting a "partition" on the "virtualization host" :We are now ready to create and start a first “partition”. Please go to the “Resources” overview in the openQRM-management GUI. Click on the “resource” which is running the “xen_host” VE. Then select the “Partition” tab and fill out the form e.g. 1 partition (Screenshot of the partition creation)
Now click on “Start partition”. The partition is now created and starting on the “xen_host”. (Screenshot of the starting partition)
To check the partitions start-up you can use the Xen-console. Click on the “partition” in the resource overview, in its resource-menu on the right select “Xen-Config” → “open console”. A java-applet will be opened in your browser connected to the console of the starting partition. (Screenshot of the starting partitions console)
When it is fully up and running it is “just” another “idle” resource. Please notice the following screenshot with already disabled maintainance mode for the started partition : (Screenshot of the “idle” partition after startup)
Deployment of the "partitions" by fast-cloning the "golden" webserver-image template :We created the “webserver” filesystem-image but we actually do not want to deploy it directly. We want to use it as a server-template, a so called “golden image”. For each customer we simply create a (LVM) snapshot of the “golden image” and use this for deployment. As an example we create a VE “customer1” for the first customer. First lets automatically create the filesyste-image clone from the “golden image”. Please go to “Filesystem-images” and select “Add new filesystem-image” from the upper right “Tools” menu. Name it “customer1”, select the “lvm-nfs” storage-server, give "/vol/customer1” as the identifier and add the following “environment variables” : CREATE_PHYSICAL=yes This tells the lvm-mgmt plugin to physically create this image from a snapshot of the “webserver” fileystem-image. Clicking on “Save” will logically and physically create the filesytem-image “customer1”. (Screenshot of cloning the webserver “golden-image”)
To re-check run “df” on the openQRM-server : [root@demo ~]# df |
