[vox-tech] solved puzzle: free du and df disagree about free disk space

[Karsten M. Self]

on Tue, Sep 21, 2004 at 08:25:20AM -0700, Henry House (hajhouse at houseag.com) wrote:
> romana:/var$ sudo du -hcs *
> 2.6M    backups
> 24M     cache
> 12K     dhcp
> 4.0K    games
> 114M    lib
> 116K    list
> 4.0K    local
> 4.0K    lock
> 793M    log
> 0       mail
> 52M     public_html
> 104K    run
> 4.0K    scratch
> 33M     spool
> 16K     state
> 296K    tmp
> 1017M   total
> 
> romana:/var$ sudo df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/sda1             1.4G  1.3G   65M  96% /
> /dev/sda5             2.4G  2.3G     0 100% /var
> 
> As you can see from these transcripts, 'du' told me that I had used
> less than 1 GB on this machine's /var, but according to 'df' there was
> still no free space.  I looked for invisible files (using "find -name
> '.?*'") but found less than 400K of those. The next likely possibility
> was: unliked files still open by running processes, which the kernel
> therefore had not freed.
> 
> So, I stopped all daemons, including sysklogd. Bingo! The missing 1.3G
> of free space appeared. It turned out to be the fault of some large
> deleted logfiles that were still open.
> 
> Moral: run logrotate and restart daemons every few weeks to keep logfiles from
> growing unreasonably large. I am curious how others (such as professional
> sysadmins) keep machines from running out of disk space. How, for example, do
> the K12LTSP users in Hawai'i keep logs from filling up their disks? Full
> filesystems seem to be one of the most common causes of downtime for Linux
> servers.

Moral Jr:  'du' and 'df' report different statistics.

'df' reports the free space on disk.  'du' reports the utilization (in
blocks) of detectectable files (by definition:  files which aren't, say,
deleted but with handles held open by running processes).  Adding a
third mix:  'ls' reports not the size-on-disk (used blocks) but actual
data size within the file.

You can also run 'lsof' to get a list of open files.  Those which have
been deleted (but remain open) will be marked '(deleted)'.  Grep for
these.


Otherwise:  yes, divergence between stats reported by different
utilities often means residual filehandles lying open.  Under a
well-configured system, logrotate _should_ handle termination or
reinitialization of logging daemons properly.


Peace.

-- 
Karsten M. Self <kmself at ix.netcom.com>        http://kmself.home.netcom.com/
 What Part of "Gestalt" don't you understand?
    I've been waiting for the right woman to get confused.
    - On not being married.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://ns1.livepenguin.com/pipermail/vox-tech/attachments/20040921/2b745681/attachment.bin