[vox-tech] solved puzzle: free du and df disagree about free disk space

[Jeff Newmiller]

On Tue, 21 Sep 2004, Henry House wrote:

[...]

> So, I stopped all daemons, including sysklogd. Bingo! The missing 1.3G of free
> space appeared. It turned out to be the fault of some large deleted logfiles
> that were still open.
> 
> Moral: run logrotate and restart daemons every few weeks to keep logfiles from
> growing unreasonably large. I am curious how others (such as professional
> sysadmins) keep machines from running out of disk space. How, for example, do
> the K12LTSP users in Hawai'i keep logs from filling up their disks? Full
> filesystems seem to be one of the most common causes of downtime for Linux
> servers.

I am not a pro sysadmin, but I use Debian...

The pros probably run logrotate and/or a script like savelog and a HUP.  
Fully stopping the daemons should not be necessary, as this problem is
normally handled by having daemons that respond to the HUP signal by
closing and reopening all files.

I don't know why, but in Debian Sarge some files are rotated with savelog
(see /etc/cron.daily/sysklogd and /etc/cron.weekly/sysklogd) and others
are rotated with logrotate (/etc/logrotate.conf and
/etc/logrotate.d/). [1] Note that the syslogd-listfiles script that
determines which files are handled by savelog won't report files that have
been rotated in the last 5 hours... this had me scratching my head as to
why logrotate was not configured to handle /var/log/messages, but neither
did /etc/cron.weekly/sysklogd seem to be doing so... really the
cron script does do it.  

Note that the cron script effectively HUPs sysklogd to close files... the
classic sequence for detaching a logfile is to first "mv" the file (which
simply changes the name in the directory while the daemon continues to
write) and then do "kill -HUP the_daemon" to trigger the daemon to close
all logfiles and re-open them.  The cron scripts use savelog to do the
first step while rolling and compressing a few older versions, and then
use the init script to do the second step (using start-stop-daemon with a
--signal 1 aka SIGHUP if the daemon is running, or starting the daemon if
it is not running).

---

[1] http://cert.uni-stuttgart.de/archive/debian/testing/2003/06/msg00062.html

---------------------------------------------------------------------------
Jeff Newmiller                        The     .....       .....  Go Live...
DCN:<jdnewmil at dcn.davis.ca.us>        Basics: ##.#.       ##.#.  Live Go...
                                      Live:   OO#.. Dead: OO#..  Playing
Research Engineer (Solar/Batteries            O.O#.       #.O#.  with
/Software/Embedded Controllers)               .OO#.       .OO#.  rocks...2k
---------------------------------------------------------------------------