[vox-tech] Large File Server - File System?

[Bill Broadley]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alex Mandel wrote:
> So I'm putting in a new file server at work.
> RAID 6, 8 TB
> It's purpose is to server large files(1GB+) and large amounts of files
> to local desktops and servers.(About 25 machines and several web services)
> 
> 1. What partition scheme would people recommend?
> Yes I read the whole previous thread on this topic.
> Here's what I'm thinking:
> /   10 GB
> swap - do we even need swap?(we have 12 GB of ram)

Having some is good, if you don't expect to swap having not much seems
reasonable.   I check 10 of our 16 disk servers that do various tasks and the
highest swap used I found was 3GB, even though vmstat was around zero.  If you
think you won't swap much I'd allocate 1 partition, if you think it might be a
performance limiting factor I'd add more.  If you are worried about uptime
keep in mind while linux will automatically use more than one swap partition
that you need to use a raid-1 if you want to survive a swap disk dying.

> /var 10-20 GB
> /tmp	40 GB
> /home Everything else?

If you are going to do /, /var, and /tmp in different partitions, I'd consider
 a LVM.  It might well save you some pain, downtime, or silly games like
making symbolic links from one partition to another.  Some folks enjoy
snapshots as well, pretty handy for trying out a major change and then
deciding to revert.  They can simplify backups as well.  There's advantages
and disadvantages either way.

How about:

Disk    A     B     C     D     E     F   G     H
p1      /     /     swap  /var  /var /tmp /tmp /tmp
p2      everything else

So with LVM:
/ = /dev/sda1 /dev/sdb1 raid-1
swap on /dev/sdc1
them use lvm to manage c1, d1, e1, f1, g1, h1 on whatever raid level you want.

Without lvm:
/ = /dev/sda1 /dev/sdb1  (10-15GB)
swap on /dev/sdc1
/var on d1 e1
/tmp on f1 g1 h1
raid-6 on /dev/sd{abcdefgh]2 (each as big as a disk - 10-15GB)

Pick the raid level for /var and /tmp as you see fit.

> 2. More importantly what file system should we use?
> ext3,ext4,XFS, etc?

For things that aren't actually a performance bottleneck I try to use the most
popular/most conservative decision.  I want as many people as possible to have
a problem before I do, ideally all the problems are fixed before I try it.
For that reason I'd pick whatever is default for your installer.  It's likely
to be heavily tested in a large number of environments.  Doubly so with a LTS
release.   For similar reasons I suggest their default partitioning as well.

Serving large files is rather easy, I'd be surprised if with any file system
you mention would be a bottleneck.  I'd expect it either in the disks (because
of seeks) or the network (which is about the speed of a single disk for
sequential reads).

> Note we'll probably run Ubuntu 8.04 LTS which may effect your answer.

I don't believe ext4 is available on 8.04 LTS.  Keep in mind that while you
can upgrade from ext3 to ext4 you lose some of the benefits of ext4.  As
mentioned I don't think this will be a bottleneck for your usage.  However,
the RAID-6 might well be.  Sure if a single user asks for 1GB and you have
GigE between the user and the file server he will get the better part of
100MB/sec.

Even if 4 users ask for the same file, the same thing.

Now if 4 users ask for 4 different files there's going to be quite a bit of
seeking, the raid-6 might well become the bottleneck.

To demonstrate, a local fileserver, using a raid-5 across 5 of 16 disks I get
around 280MB/sec for sequential reads.  With 2 running at once I get a total
of 120MB/sec, 4 around 100MB/sec, 6 around 73MB/sec.  I was playing with a
64GB slice at the beginning of each disk, if my 6 threads were spread across
the entire 1T disk I'd expect much lower.

On the flip side if I run 1 per RAID on 3 5 disk raid-5's I get 605 MB/sec.

So depending on what performance you'd like and how many simultaneous users
you have you may want to tweak your large raid-6.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkpxBMoACgkQBmOBO0n4EFWz6gCeKpJtyGkiEbskjB2PNXDhp8T8
jvQAn2JuHAZIOgUn0D84O83kDT+II8na
=Z+qM
-----END PGP SIGNATURE-----