[vox-tech] Xen + LVM usage

[Bill Broadley]

Ah, sorry, I should check this list more often.

On Mon, Jun 26, 2006 at 04:53:21PM -0700, Luke Crawford wrote:
> That is because I don't know if it would be a problem, myself.    However, 
> I have been burned (horribly disfigured)  by putting vnd (loopback) 
> backing files on a filesystem with softupdates/journaling enabled, so my 
> first reaction to a suggestion of disk trickery is "Run away!"

Ah.  Well the Xen manual recommends:
  To create two copy-on-write clones of the above file system you would use the   following commands:

   # lvcreate -s -L1024M -n myclonedisk1 /dev/vg/myvmdisk1
   # lvcreate -s -L1024M -n myclonedisk2 /dev/vg/myvmdisk1

So they are actually parititions, not layered on top of a filesystem.  I
figure the Xen folks wouldn't recommend it if it didn't work well.

> At first I thought what you were suggesting is doing the union in the 
> dom0, at the backing file level.   Being as this is two big files, and ffs 
> with softupdates is much better at containing corruption to just a few 
> files than it is at preventing corruption within those files, the result 
> would be most likely disastrous.

Indeed.

> However, if you did the union at the domU level, so it saw 1 complete 
> filesystem as the original, and another complete filesystem (or journal, 
> or whatever it uses) as the delta, then it should work fine.

No need, see the section in the Xen manual:
	6.3 Using LVM-backed VBDs

> As we all keep saying,  disk space is cheap.  However, your idea has 
> merit, especially when,  say,  attacking the Chinese or Indian market... I 
> imagine the difference between $5/month and $2/month would be 
> significant in that market.  Heck, even for my $5 images now, it would 
> increase my margins and decrease my startup costs.  Good SCSI is $3-$5 GB 
> new (that's $6-$10 per GB after raiding it... more after paying for 

Ugh, SCSI seems expensive and painful these days.  I'm perfectly happy
with, er, 8 or so large (4-6TB) fileservers I run.  I definitely recommend
the enterprise/RAID level SATA drives (usually an extra $10 each).  In any case
that is probably best saved for another thread.

> enclosures and stuff)  Honestly, power isn't that much of an issue with 
> disks... it's CPU that eats the power, but disks do consume rack space 
> (especially if you use lots of small disks like I do)

Disk space has many side effects:
* Your base install can be much more generous, you don't pay for it per
  virtual user.  Read this as happier users... it "looks" like they have
  more disk space.
* End user space usage is directly related to what files they actually use
  (instead of what is included in the base install).
* Creating a copy-on-write (CoW) clone/writable persistent snapshot is
  fast!
* It creates an incentive to use existing infrastructure (to avoid the
  disk cost), which would hopefully mean they are easier to support.
* Rolling out significant new functionality in your virtual image doesn't
  cost you per user.  If a single customer requires postgres and you want
  to "do it right" once, all your customers can take advantage of it.  

# time lvcreate -s -L1024M -n myclonedisk1 /dev/vg01/data01
  Logical volume "myclonedisk1" created

real    0m0.138s
user    0m0.004s
sys     0m0.061s

Not bad for cloning a 10GB partition.  On another machine I was playing
with a 2GB image and the image copy took a substantial amount of time.

> though, my cost per ipv4 IP is around $0.70/month, and my system is 
> largely not-automatic.  I'd have to make it much more automatic to be 
> able to profit from those kinds of margins.
> 
> It also appears that your idea would reduce my setup costs.  Right now, it 
> takes a good 5-10 minutes to create a Xen image, and if this is simply a 
> kind of 'reverse snapshot'  the setup should be just about instant.

Indeed.   Just make sure to monitor how much has changed so you can
lvextend as needed (and charge the customer more).  I'm not sure what
the failure mode is, but I suspect it might be ugly.  

-- 
Bill Broadley
Computational Science and Engineering
UC Davis