[vox-tech] HOWTO: Make Your Our Debian Mirror
Mike Simons
vox-tech@lists.lugod.org
Thu, 27 Feb 2003 20:54:49 -0500
Thu Feb 27 13:25:40 PST 2003 version 0.1
This is a quick and dirty setup your own Debian Mirror Howto.
I've been asked how to set one up several times but I've finally been
asked by someone persistent enough to keep asking, you can thank them
for this document :).
All of the files mentioned in this howto should be downloadable from
the following URL:
http://simons-clan.com/~msimons/debian/mirror/
If anyone has comments email me with suggestions.
Familiarity with the Debian Install process is expected...
This example is for i386 architecture, but should be easy to change to
suit.
I also expect to have messed up one or two steps, so run with care.
Good Luck,
Mike Simons
msimons@simons-clan.com
1 - Initial setups:
===================
In this document the mirror is going to be put in '/home/debian'.
mirror is the directory the archive will actually go. bin is where the
scripts go.
===
adduser debian
m=/home/debian
mkdir -p $m/bin $m/mirror
cd $m/bin
wget http://simons-clan.com/~msimons/debian/mirror/debmirror
wget http://simons-clan.com/~msimons/debian/mirror/run
chown -R debian:debian $m
===
debmirror
=========
The core worker script this howto uses is called "debmirror" it a script
originally written by Joey Hess, which I found on some Debian mailing list
over two years ago. I had been operating my own mirror but when Debian
moved to package pools my older scripts which just did an rsync/ftp mirror
broke... so after a short search I found this one from Joey and used it.
Over time I fixed a number of things that were bothering me about the
way it worked... and later I found that it was part of a debian package
"debmirror". I sync'ed my version up with debian package version and
tried to get my changes merged into the main archive... but that never
happened.
I have not looked at the debian version of the debmirror script in
ages. Ways my version was better than the packaged version include:
====
- Does not download a file unless it actually changed
(based on file size and timestamp check of remote to local)
- Auto-resumes a file transfer if the transfer was aborted part way
through.
- Don't change the local archive's Packages files until *AFTER* all
the files mentioned inside have been downloaded successfully
(this was you can use the mirror while it is syncing up, or after
a failed sync attempt).
- Automatically detect if passive mode ftp is needed or not (attempt
non-passive mode, if that fails try passive, warn user and continue
using passive for the rest of the fetch attempts).
- Cleaned-up debugging messages.
====
You can read the debmirror documentation by downloading the script and
typing:
perldoc ./debmirror
run
===
I have a master script call "run" which calls debmirror in ways it
needs to be called to actually do the mirror process. This is where
the configuration goes, like which upstream sites you want to use for
mirroring from and which release of Debian you want.
I should really explain how to use netselect or netselect-apt to
find the best mirror to use based on your network location.
That is not happening now.
2 - Run the mirror script:
==========================
After you get the "run" script to pull from the mirrors you want...
run it:
===
/home/debian/bin/run
===
It will take a while... just make sure it's working correctly.
If there are problems fix them before going further.
If you have --debug or --progress enabled you will probably want to
redirect things to a file so that you can stop to look at what is being
printed.
To install a cron entry to automatically do this every morning at 2am:
===
cd /home/debian
wget http://simons-clan.com/~msimons/debian/mirror/crontab
crontab -u debian crontab
===
From then on I recommend having cron do the updates, output from runs
would be sent to your "debian" user via email which should be easy to read.
3 - Setup a web server to serve your mirror:
============================================
If you don't already have a apache installed...
===
apt-get -y install apache
===
Add the following to /etc/apache/httpd.conf:
===
# Allow access to Debian mirror tree...
Alias /debian/ /home/debian/mirror/
<Location /debian>
Options Indexes FollowSymLinks MultiViews
</Location>
===
Restart apache after the config change:
===
/etc/init.d/apache restart
===
At this point you have a mirror that is fully functional... to
use it put lines like in your /etc/apt/sources.list:
====
deb file:///home/debian/mirror woody main contrib non-free
deb file:///home/debian/mirror/non-US woody/non-US main contrib non-free
deb file:///home/debian/mirror/debian-security woody/updates main contrib non-free
===
The rest of the instructions tell you what needs to be done to install
off the mirror.
4 - Manually pull down disk images:
===================================
In order to be able to do a net install of the base system from your
machine you will need some files which are not part of the Packages.gz
file which gets pulled by the debmirror script. For now I manually
pull the bf2.4 images. I've put this into a little script form for
this howto (which will also make my job much easier next time the disk
sets are updated).
To pull down the latest disk set from the mirror:
===
m=/home/debian/mirror/dists/woody/main/disks-i386
mkdir -p $m
cd $m
b=3.0.23-2002-05-21
c=3.0.23-2002-07-18
for x in $b/bf2.4/ $b/images-1.44/bf2.4/ base-images-$c/basedebs.tar; do
wget --recursive --continue --passive-ftp -nH --cut-dirs 5 \
ftp://ftp.us.debian.org/debian/dists/woody/main/disks-i386/$x
done
ln -s $b current
ln -s base-images-$c base-images-current
chown -R debian:debian $m
===
Be warned that is a manual kludge around the fact that the disk sets
are not recorded in a Packages.gz format, these files will not
automatically be downloaded when a mirror is done, and the paths below
are hard coded for the current (as of today) version.
./3.0.23-2002-05-21/bf2.4/drivers.tgz can be copied onto the hard
drive of your target machine to allow you to select "from hard disk" or
"from mounted filesystem" as a source for the base kernel and modules
in the install menu.
Otherwise you will need the driver-*.bin disks on floppy or CD to
talk to the network to begin a network install.
If you have ./base-images-3.0.23-2002-07-18/basedebs.tar on the
target hard drive, you won't even need to talk to your mirror until
the base system boots up and you are in stage two of the debian install
process.
5 - Setup a dhcp server to allocated IP addresses:
==================================================
If you don't already have a DHCP server on the network you will be
installing over you should.
If you already have a DHCP server do not do the following steps.
Install the DHCP server
===
apt-get -y install dhcp
===
Put the following in /etc/dhcpd.conf
===
option domain-name-servers 192.168.123.1;
option subnet-mask 255.255.255.0;
default-lease-time 600;
max-lease-time 7200;
subnet 192.168.123.0 netmask 255.255.255.0 {
range 192.168.123.16 192.168.123.254;
option broadcast-address 192.168.123.255;
option routers 192.168.123.1;
}
===
Restart dhcp after the config change:
===
/etc/init.d/dhcp restart
===
note if you don't run a DNS server on your machine you will need to
change the domain-name-servers line to be a coma separated list of
correct servers.
note if your mirror machine is not also the gateway you will need to
change the 'option routers'.
The configuration above leaves some room (1-15) for static manually
configured IP addresses.
I will not cover setting up DNS or setting up your machine to be a
masquerading firewall/gateway in this howto... but those are what the
demo box does at the installfests.
I would suggest if you get the DNS server setup you may want to
fake advertise your mirror as 'http.us.debian.org' so that you
don't need to do the following steps.
6 - Store a local sources.list file:
====================================
On the mirror I keep a sources.list file prepared so that at install
time when questions about what apt sources you want to use are asked
one is handy.
===
cd /var/www
wget http://simons-clan.com/~msimons/debian/mirror/sources.list
chown debian:debain sources.list
===
That file references the local mirror...
When the second stage installation process asks you for what sources
you want to use to install the system, select 'edit sources.list by hand'.
Switch to window two (alt-F2), login as root, type
===
cd /etc/apt
telnet 192.168.123.1 80 > sources.list
GET /sources.list
vi sources.list
===
Delete the first three lines of the file (which are the telnet command
printing garbage. Save the file. Switch back to window 1 (alt-F1),
and exit without saving changes... (^X) you are good to go, the
install system will begin pulling package lists from your mirror.
TODO:
- move my cvs archive of this script somewhere public
- attempt a sync with the debian package
- teach something how to pull disk sets, automatically.
- explain netselect-apt