** NOTE ** As of 24 April 2012, this file should be considered deprecated. The canonical version of the information contained in this file is the Debathena Trac wiki (http://debathena.mit.edu/trac/wiki), in particular, the section titled "Release Engineering". This file should be maintained for posterity, but should not be updated. ** This hierarchy contains Debian/Ubuntu-specific materials, also known as "Debathena". The contents are: * debathena - Debathena-specific software packages such as PAM and NSS modules. * config - Packages for configuring native system software in a manner appropriate for Athena. * meta - Packages which contain nothing but dependencies on other packages and serve as an installation convenience. * scripts - Build scripts and supporting materials. Debian software used by Debathena: * schroot - Used to manage build chroot environments for each Debian/Ubuntu version. We use the device schroot type with aufs union mounts, which allows rapid construction of ephemeral copies of template "source" chroots, so that every binary package build is done in a clean environment. * debuild - Used to create Debian source packages from package source directories. * sbuild - Used to build binary packages from source packages inside schroot environments. * equivs - Used to create packages which only contain dependency information. Somewhat of a dirty hack, since it doesn't keep proper changelogs, but it reduces overhead. * CDBS (Common Debian Build System) - Referenced by debian/rules files in packages. Contains standard build rules to cut down on per-package boilerplate. * reprepro - Used to upload packages into the apt repositories. * approx - Used to create a local cache of Debian packages on the build server. This cache is referenced by the build chroots for improved performance. The remainder of this file documents procedures useful to Debathena developers and maintainers. Developers: Preferences setup ----------------------------- You will probably want a $HOME/.devscripts file containing the following: DEBUILD_DPKG_BUILDPACKAGE_OPTS="-sa -us -uc -i -I.svn" This will save you from having to specify those options every time you run debuild. The options mean: * Look for original source as a tarfile or create one. * Do not sign the source package. * Do not sign the changes file. * Ignore common version control metadata files when creating diffs. * Ignore .svn paths when creating tarballs. You will also want a $HOME/.sbuildrc file containing the following: $nolog = 1; $mailto = 'yourusername'; $log_dir = '/tmp/sbuild-logs'; $maintainer_name = 'Debathena Project '; $force_orig_source = 1; $sbuild_mode = "user"; 1; You should also set the environment variable DEBATHENA_APT to "/afs/sipb.mit.edu/project/debathena/apt". Developers: Preparing a change ------------------------------ To prepare a change to a regular package (a source tree containing a debian/ subdir), make the edits in a checkout and record a changelog entry. You can either edit debian/changelog using emacs changelog mode (C-c C-v to add a new version entry, C-c C-a to add a change entry, C-c C-f to finalize the entry) or you can run "dadch" from the debathena locker / SVN scripts directory. When creating a new version entry, bump the upstream version number (to 10.0.0 if it was not already that high) if you are changing the main package source. Otherwise, just bump the Debian version component (change 0debathena1 to 0debathena2, for instance). Developers: Building a package for test purposes on one platform ---------------------------------------------------------------- After you have prepared a change, you will want to test that it builds and perhaps that it works before committing it. First, if it is an Athena source directory using autoconf, run "daconfiscate" to set up the autoconf boilerplate which we don't check in. Second, run "daorig" to copy or create an orig tarball in the parent directory if necessary. Third, run "debuild". The resulting package will be placed in the parent directory. In order to test if the package works, you can install it with "dpkg -i filename.deb". Developers: Building a package for test purposes on all platforms ----------------------------------------------------------------- If the package you are working on interacts with the native OS in ways that might vary from platform to platform, you may want to do a test build for all platforms. You will need to do this on linux-build-10.mit.edu or another machine which has been set up with build schroots. As above, run daconfiscate (if necessary) and then daorig. Then run "debuild -S" to create a source package. Now cd into the parent directory and identify the .dsc file created by debuild -S; it will have a name like debathena-just_9.4.0-0debathena2.dsc. Run "da sbuildhack filename.dsc" to perform the package builds. Each build will take place inside an ephemeral chroot based on a snapshot of a template for a particular Debian or Ubuntu version. If a build fails and it's not obvious from the build log why, you may need to create your own ephemeral chroot session with a command like "schroot -c gutsy-amd64-sbuild /bin/sh" and then run debuild from within the package sources. If the build is successful, it will create a set of packages with names like debathena-just_9.4.0-0debathena2~ubuntu6.06_amd64.deb. Developers: Building an equivs package -------------------------------------- Most of the packages under debathena/meta are faked up using equivs. To build one, just run: equivs-build --full filename.equivs These equivs files make reference to ../common, so you must have a checkout of debathena/meta/common alongside the particular meta-package you are building. Developers: The meaning of metapackages --------------------------------------- If you are adding a new package to the repository, you will probably at some point want to add it to one of the metapackages so that it doesn't have to be installed by hand. Here are some descriptions which may help identify which metapackage is best: * locker: Provides access to Athena locker software--AFS and automounter configuration, locker-related utilities, etc. * clients: Provides clients (either locally-written, like athinfo and Discuss, or configurations) for Athena services, as well as Athena-specific utility programs like "jot". Configurations for graphical client software are generally in the workstation package instead, in order to make this package less intrusive. * standard: Implies locker and clients. Also provides Athena shell customizations and dotfiles. * login: Implies standard. Configurations to merge the MIT user namespace into the local machine namespace for the purpose of user lookups and authentication. * login-graphical: Implies login. Configurations for the graphical login system and graphical client software intended to provide a standard X login experience using Athena home directories. * extra-software: Implies a set of Debian packages common to cluster machines and typical workstations. The resulting software set is rather large, and thus may not be desirable to all workstation configurations. Only stock Debian packages belong in this metapackage; do not add other Debathena packages to it. * workstation: Implies login-graphical and extra-software. The use case is the same as that of the old "private workstations", so it includes more centrally-managed configuration than standard or login does (unattended updates, etc.). New as of May 2009. * thirdparty: Software supported by 3partysw that upstream Ubuntu already happens to be keeping up to date, and is therefore easier to install via packages than through lockers. This package is HUGE (1 GB of network traffic, 3 GB installed), and is also not strongly tested on Debian or on older releases of Ubuntu. * cluster: Implies workstation and thirdparty. Configures public cluster machines (login-time LVM snapshots, cleanups between logins, public root password, etc.). The snapshot code requires a particular LVM layout that the PXE (netboot) cluster installer sets up. * debian-dev: Intended for developers of the system itself; provides a set of Debian packages used by Debathena for development. For the most part a package should be listed in the "Depends:" line of a metapackage, but in some cases it is appropriate to hedge by using "Recommends:", which will cause aptitude to succeed even if the package is unavailable. For example, a package which doesn't exist in all Debian/Ubuntu suites or isn't free can be listed under "Recommends:" so that our metapackages still work in all environments. Developers: Index of cluster packages ------------------------------------- Some private workstation admins may want specific pieces of cluster infrastructure. With Athena 9.4, they would typically install the entire Athena software suite and then turn off features they did not want via /etc/athena/rc.conf. That is not generally possible in Athena 10; a debathena-cluster machine acts like a cluster machine in all respects. Instead, the machine owner should install debathena-workstation and then install specific packages they might be interested in. At this time, candidate packages include: * debathena-dns-config: Installs a caching DNS resolver and configures the machine to use it. * debathena-tmp-cleaner: Cleans files in /tmp and /var/tmp which have not been recently accessed. * debathena-reactivate: Causes each graphical login to be performed in a separate ephemeral snapshot of the chroot. This package requires the root filesystem to be an LVM logical volume inside a volume group with 21GB of free space for the snapshots; if that is not the case, it will break graphical logins. Install with care. * debathena-auto-update: Periodically updates the machine's software and reboots (when no one is logged in) if necessary. Cluster packages which are generally not of interest to private machine owners include: * debathena-cluster-login-config: Configures a variety of system services to implement the cluster login policy (no tty logins, no user switching, screensaver logout button after 20 minutes, etc.). * debathena-clusterinfo: Looks up the machine's cluster information in Hesiod and caches it on local disk. * debathena-larvnet: Reports to the central larvnet server whether anyone has a graphical login to the machine. * debathena-syslog-config: Configures sysklogd so that some syslog messages are forwarded to a central logging host. Release engineer: Bootstrapping the project infrastructure ---------------------------------------------------------- 1. Create the package repository (detailed instructions on this pending). Set the DEBATHENA_APT environment variable to point to the package repository. Put a copy of the debathena "scripts" subdir in your path. 2. Create the build area. 3. Run 'dareprepro export' to set up the repository. Build each equivs package under meta/ using 'equivs-build --full *.equivs' and upload each with 'daupload-equivs-proposed *.changes', followed by 'damove -proposed "" PACKAGE' (replacing PACKAGE with the name of each equivs package you are moving). 4. Set up the build server. The basic structure of the apt repository must work for make-chroot to succeed, so this must happen after step 3. 5. For each normal Debian package in dependency order, cd into its directory in the build area and run "da sbuildhack *.dsc" and "daupload-release *_source.changes". If the package contains only an "Architecture: all" binary package, pass the -A option to both commands. The all-packages script can generate an approximation of the package list in dependency order, but it doesn't work right yet, and ideally it would be possible to do several builds in parallel using a Makefile like the one in scripts/build-server/build-all. Improvements to this machinery are pending. 6. For each package under third, run "da ./debathenify-PKG source binary upload". Any created directories under third/openafs/meta should be chmodded 777 to work around a perl/AFS permissions issue with File::Temp; if this is not done, OpenAFS metapackage builds will fail for other users. Release engineer: Updating the apt repository --------------------------------------------- 1. ssh to the build server as the builder account and change to the canonical build directory. 2. Run "gen-packages" to update the package list. (Or "gen-packages -c" if you know the AFS checkout of the source tree is up to date; it should update every half hour.) 3. Run "ood-packages" to produce a list of out-of-date packages. 4. For each out of date package, run "dasource PKG". Then change to the package directory and run "da sbuildhack *.dsc" and "daupload-release *_source.changes". If the package contains only an "Architecture: all" binary package, pass the -A option to both commands. 5. svn update the meta directory. If there are new subdirectories, chmod them 777 to work around a perl/AFS permissions issue with File::Temp. For each updated subdir, change to it, run "equivs-build --full *.equivs", and then "daequivsupload FILENAME.changes" on the produced changes file. 6. svn update the third directory. You can let autodebathenify handle the updated scripts, or you can touch ~/autodebathenify.suppress, make sure it's not running, and run "da ./debathenify-PKG source binary upload" in each updated directory. Sometimes you may have to mix up the order of the above steps in order to handle build dependencies. Release engineer: Setting up a build server ------------------------------------------- 1. Install either Debian testing (or stable, if it's new enough, but see the note about sbuild below) or the latest Ubuntu release. The build server must be installed with free space in an LVM volume group. The build chroots consume 4GB each. 2. Install debathena-login as per the the instructions in http://debathena.mit.edu/install. 3. apt-key add /afs/sipb.mit.edu/project/debathena/apt/debathena-archive-keyring.asc 4. Install the packages listed in scripts/build-server/packages (using "aptitude install"). Note that currently the build system uses the latest version of sbuild, so if you have not installed Debian testing, you'll need to set up apt pinning to grab sbuild from there. See the apt_preferences man page. You'll also need to grab schroot from Debian experimental. 5. Install the aufs-modules-2.6-amd64 package. 6. Append to /etc/approx/approx.conf the contents of scripts/build-server/approx.conf.tail. Run: /etc/init.d/approx restart 7. Apply scripts/build-server/mount-defaults.patch and scripts/build-server/pam-schroot.patch. 8. For each supported DIST (see scripts/debian-versions.sh) run: scripts/build-server/make-chroot DIST i386 scripts/build-server/make-chroot DIST amd64 Example: scripts/build-server/make-chroot intrepid i386 9. Create a local account for builder with: adduser --uid 1047 --disabled-password builder Make the home directory mode 700. Install a daemon/linux-build-10.mit.edu keytab in the home directory as "keytab". Install a copy of the secret repository-signing key (debathena@mit.edu) in the home directory's keyring with something like: kinit builder gpg --export-secret-keys debathena@mit.edu | \ ssh -l builder machinename gpg --import Create a file named .sbuildrc in builder's homedir containing: $mailto = undef; $log_dir = '/tmp/sbuild-logs'; $maintainer_name = 'Debathena Project '; $force_orig_source = 1; $sbuild_mode = "user"; 1; Create a file named .ssh/config in builder's homedir containing: Host svn.mit.edu User debuildsvn Add builder to the sbuild group in /etc/group. Copy scripts/build-server/autodebathenify to builder's homedir. Create a file named autodebathenify.config in builder's homedir containing: error_addr=debathena-root@mit.edu scripts_dir=/mit/debathena/bin build_dir=/mit/debathena/packages/third packages="lprng openafs" export DEBATHENA_APT=/mit/debathena/apt Copy scripts/build-server/autodebathenify.cron to builder's homedir and install it with "crontab autodebathenify.cron". Copy scripts/build-server/autolivebuilder to builder's homedir. Create a file called autolivebuilder.config in builder's homedir containing: error_addr=debathena-root@mit.edu release_version='9.04' release='jaunty' arch='i386' mirror='mirrors.mit.edu' gpg_opts=("-u" "0D8A9E8F") live_dir=/net/sipb-isilon-sc.mit.edu/ifs/mirror/sipb/debathena/livecd Create a file named .devscripts in builder's homedir containing: DEBUILD_DPKG_BUILDPACKAGE_OPTS="-sa -us -uc -i -I.svn" In builder's homedir, append to .bashrc: add debathena export PATH=$PATH:~/bin export DEBATHENA_APT=/mit/debathena/apt Release engineer: Removing a build chroot on the build server ------------------------------------------------------------- 1. Run VG=/dev/blah scripts/clean-schroots as root to make sure that the build chroot is not mounted, substituting the name of the volume group for blah. 2. Look in /etc/schroot/chroot.d/ and delete the file corresponding to the chroot. 3. Run lvchange -an blah/chrootname substituting the name of the volume group for blah and the chroot name for chroot. Example: lvchange -an dink/gutsy-i386-sbuild 4. Run lvremove blah/chrootname Release engineer: Removing a dist from the apt repository --------------------------------------------------------- 1. Inside the apt repository, edit conf/distributions and remove the distribution section. 2. Run dareprepro --delete clearvanished Release engineer: Setting up a canonical build area --------------------------------------------------- 1. Create an empty directory and cd into it. The canonical build area lives in /mit/debathena/packages/. 2. Run gen-packages to create the table of normal Debian packages. 3. Run dasource to create subdirs and source packages for each normal Debian package. 4. Create checkouts of the meta and third directories: svn co svn+ssh://svn.mit.edu/athena/trunk/debathena/meta svn co svn+ssh://svn.mit.edu/athena/trunk/debathena/third chmod 777 meta/* A couple of subdirectories of debathena/meta are normal Debian packages, so this will create redundant copies of those. Ignore them; they won't be used. Release engineer: Adding a new suite or rebuilding a suite ---------------------------------------------------------- This process is rarely performed and the infrastructure for it is imperfect. Substitute the name of the new suite for $suite in all steps below. 1. Make sure the apt repository is up to date with respect to the source tree for the existing dists. Check debathena.mit.edu/package-list/development and debathena.mit.edu/package-list/proposed. More importantly, make sure there is nothing in svn that isn't built. (An easy way to do this is running the check-unbuilt-packages script.) 2. Add the new dist to scripts/debian-versions.sh (and svn up into the locker). (It is not necessary to add the new dist to codes at this point, but it must be present in the gettag conditional.) If you are building an unreleased distribution, add an additional ~0.1 tag; if you're rebilding, bump that tag. 3. Create the new distribution in the apt repository's configuration file by editing /mit/debathena/apt/conf/gen-distributions and running it with output to the 'distributions' file in the same directory. Create the skeleton of the dist by running "dareprepro export" on the build server. (You don't need to do this if you're rebuilding.) 4. Add an appropriate sources.list.d/*/$suite.list entry in scripts/build-server. Commit and update /mit/debathena/bin/build-server. 5. On the build server (as root), create a chroot for the new distribution: First, check if /usr/share/debootstrap/scripts/$suite exists. If not, it may require downloading and installing a more recent version of the debootstrap package from the -backports dist corresponding to the build server's OS. (You presumably don't need to do this if you're rebuilding.) You could also just go into that directory and do "ln -s gutsy $suite" since it apparently hasn't changed in years. /mit/debathena/bin/build-server/make-chroot $suite i386 /mit/debathena/bin/build-server/make-chroot $suite amd64 6. Fire up screen, ideally making use of the "-S" option to identify your screen session. e.g. "screen -S joeuser.oneiric_build". 7. mkdir /mit/debathena/machines/awesome-build-server/stamps.$suite. 8. cd into a checkout (probably on your local machine) of debathena/scripts/build-server/build-all. 9. Edit Makefile (and check in the edit) so that suite is the new distribution and psuite is the previously most recent Debian or Ubuntu distribution. psuite is used to generate the order of build-dependencies, so choose the closest distribution. If you are rebuilding, you can set psuite to suite. 10. cd to /mit/debathena/bin/build-server/build-all, which is a totally logical place to run the build from. Run "make deps.mk". 11. Run "make -k all". You can watch the builds happen in the other windows of the screen session. It's possible to do several builds at once with make -j N (except as of 7/2011 this is a bad idea) Release engineer: Maintaining autodebathenify --------------------------------------------- autodebathenify is a cron job which runs on the builder account and keeps the OpenAFS modules and modified packages under debathena/third up to date when there are upstream changes. It relies on the following: * builder@ATHENA.MIT.EDU has access to a local account named debuildsvn on svn.mit.edu with read access to the athena repository. * builder@ATHENA.MIT.EDU has write access to the canonical build area and apt repository in the dev cell. * The builder account on the build machine has a copy of the signing private key for the apt repository. The cron job will silently exit if it detects that it is already running. If it fails for any other reason, it will send mail to source-wash@mit.edu and touch a file named autodebathenify.suppress which will prevent it from running again until manual intervention. It is important to get autodebathenify running smoothly or the apt repository will become out of date with respect to third-party packages, which in turn will compromise the user experience. Even if you don't notice failure mail, it's good to check on the status of autodebathenify from time to time. To do this, get tickets as builder and ssh to linux-build-10 as builder. Run "ls -l logs" to see what's been going on. A logfile with size around 48K indicates that autodebathenify ran normally but didn't find any new work to do. A logfile with size around 60 means that autodebathenify.suppress exists and the cron job has stopped running. A longer logfile indicates that autodebathenify attempted to build and upload a package. Possible failure cases include: * One of the upstream apt repositories timed out during an "apt-get update" of an sbuild chroot. If this happens, you can just remove the autodebathenify.suppress file to get the cron job to start running again. * autodebathenify tried to build an upstream version which already exists. If this happens, the build will fail at upload time. If this happens, there is a bug in the part of the debathenify-* script which checks whether our apt repository already has the expected version of the built package. * Debian and Ubuntu have .orig files with the same name but different contexts for a particular third-party package. This has only happened once, with bash (which is currently not configured into autodebathenify for that reason). We do not have a general mechanism for resolving such issues at this time. * The new upstream version of the package is sufficiently different to cause our modifications to fail to apply, and our debathenify-* script needs to be adapted appropriately. The cron job is run out of builder's home directory, but its canonical source location is in debathena/scripts/build-server in this repository, should it need to be modified. After several months of operation, LVM snapshots on the build server will start to become slower and slower, as /etc/lvm/cache/.cache grows in size. Turning off the cache entirely seems to cause schroot to occasionally fail; the current solution is to rm /etc/lvm/cache/.cache every so often. Release engineer: apt repository HTTP server setup -------------------------------------------------- The apt repository server (debathena.mit.edu) is hosted on scripts.mit.edu out of the debathena locker. To replicate it on a dedicated server, install Apache and AFS (e.g., debathena-standard and apache2), and configure the document root to be /mit/debathena/web_scripts There's also a crontab in /mit/debathena/cron_scripts that updates the package lists on the website. It expects to be able to edit the web_scripts directory; on scripts, it does so with daemon.scripts permissions. Trac (http://debathena.mit.edu/trac/) expects to use sql.mit.edu for its database.