Arla

A Free AFS Clone from KTH Edition 0.0, for version 0.27

1999

Love Hörnquist-Åstrand Assar Westerlund last updated $Date: 1999/07/19 23:40:28 $

Copyright (c) 1998 - 1999 Kungliga Tekniska Högskolan (Royal Institute of Technology, Stockholm, Sweden). All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

3. All advertising materials mentioning features or use of this software must display the following acknowledgement: This product includes software developed by Kungliga Tekniska Högskolan and its contributors.

4. Neither the name of the Institute nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE INSTITUTE AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE INSTITUTE OR CONTRIBU-TORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSI-NESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Copyright (c) 1988, 1990, 1993 The Regents of the University of California. All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

3. All advertising materials mentioning features or use of this software must display the following acknowledgement: This product includes software developed by the University of California, Berkeley and its contributors.

4. Neither the name of the University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHER-WISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Copyright 1992 Simmule Turner and Rich Salz. All rights reserved.

This software is not subject to any license of the American Telephone and Telegraph Company or of the Regents of the University of California.

Permission is granted to anyone to use this software for any purpose on any computer system, and to alter it and redistribute it freely, subject to the following restrictions:

1. The authors are not responsible for the consequences of use of this software, no matter how awful, even if they arise from flaws in it.

2. The origin of this software must not be misrepresented, either by explicit claim or by omission. Since few users ever read sources, credits must appear in the documentation.

3. Altered versions must be plainly marked as such, and must not be misrepresented as being the original software. Since few users ever read sources, credits must appear in the documentation.

4. This notice may not be removed or altered.

/* **************************************************************************** * Copyright IBM Corporation 1988, 1989 - All Rights Reserved * * * * Permission to use, copy, modify, and distribute this software and its * * documentation for any purpose and without fee is hereby granted, * * provided that the above copyright notice appear in all copies and * * that both that copyright notice and this permission notice appear in * * supporting documentation, and that the name of IBM not be used in * * advertising or publicity pertaining to distribution of the software * * without specific, written prior permission. * * * * IBM DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, INCLUDING ALL * * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN NO EVENT SHALL IBM * * BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY * * DAMAGES WHATSO-EVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER * * IN AN AC-TION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING * * OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. * **************************************************************************** */

1 Introduction

Caution: this is alpha quality software. If something doesn’t work, it’s probably because it doesn’t. If you don’t have backup of your data, take backup.

What is Arla?

Arla is a free AFS clone. Some of the goals are:

• have an implementation that is free and can be used for adding stuff like disconnected-mode support to.

• provide an alternative to Transarc’s AFS-client

• Add support for platfroms that doesn’t have AFS support today.

This release is known to work on the following platforms: NetBSD, OpenBSD, FreeBSD, Linux, SunOS, Solaris, AIX, IRIX, Digital UNIX, Rhapsody.

Status

Arla has the following features (this does not mean any of this works):

• a rxgen implementation called ydr

• partly implemented fs, vos, pts commands.

• a implementaion of rxkad written outside USA without any export restrictions

Bug reports

If you find bugs in this software, make sure it is a genuine bug and not just a part of the code that isn’t implemented.

Bug reports should be sent to arla-drinkers@stacken.kth.se. Please include information on what machine and operating system (including version) you are running, what you are trying to do, what happens, what you think should have happened, an example for us to repeat, the output you get when trying the example, and a patch for the problem if you have one. Please make any patches

with diff -u or diff -c.

Suggestions, comments and other non bug reports are also welcome.

Mailing list

There are two mailing lists with talk about Arla. arla-announce@stacken.kth.se is a low-volume announcement list, while arla-drinkers@stacken.kth.se is for general discussion. Send a message to LIST-request@stacken.kth.se to subscribe.

2 Parts of Arla

Caution: This text just tries to give a general picture. For real info read the code. If you have any questions, mail arla-drinkers@stacken.kth.se.

How does arla work

Arla consists of two parts, a userland process (arlad) and the kernel-module (xfs).

Arlad is written in user-space for simpler debugging (and less rebooting). And there is not the same limitations as if it would be written in the kernel. To avoid the performance loss xfs does caching.

xfs and arlad communicate with each other via a char-device-driver. There is a rpc-protocol currenly used specially written for this (‘arlad/message.c’)

xfs is written to be as simple as possible. Theoretically, xfs could be used by other user-space daemons to implement a file system. Some parts, such as syscalls, are arla-specific. These parts are designed to be as general as possible.

For example, xfs does not recognize which pioctl the user-level program calls, it just passes this information on to arlad.

Arlad and XFS relation

     Userland
                                  ---------
                 Edit file        | Arlad |  ------> Network
                                  ---------
                ----------------------|[1]----
                 -------           -------
     Kernel      | VFS | <--[2]--> | XFS |
                 -------           -------

[1] A char device (/dev/xfs0)

[2] Xfs provides a filesystem for the vfs-layer in the operating system.

2.1 The life of a file

This a rough description of the life of a file. The names are inspired of BSD-style VFS-layer but the idea is the same in most operating systems.

• The user decides to open a file.

• The vfslayer sends a VOP LOOKUP to xfs that is forwarded to arlad with a getnode() (seqnum 1).

• arlad tries to find the requested file and then, if found, sends an install node to xfs by writing to the xfs character device.

• xfs inserts the node into the cache and returns from the device write.

• arla sends a wakeup rpc message (seq-num 1). If the return value is zero xfs tries to find the node in the cache, if not found it might have been flushed out of the cache and the whole thing is repeated.

• If a none-zero return value is returned, this value is sent as reply to the user. This way arla can decide what error message is returned, without xfs having support for each error.

• xfs now checks if it has the valid attributes. If the attributes are invalid, xfs will send a rpc message to arlad to refresh it.

• Since the user wanted to open the file, a getdata rpc message is sent from xfs to arlad. Now arlad fetches the files from the afs file server.

• Arlad stores the file in the file cache. All vnode operations will be done on this file. Now arlad sends a installdata to xfs.

• When xfs recives the installdata it looks up the node in the cache, and then it does a VOP LOOKUP to find a vnode to the cachefile (and store it to keep it for future use).

• The same thing is done when the file is a directory, except that the directory is converted from the afs directory format to an operating system dependent format and stored in a file. xfs reads this file instead.

• If the directory is modified locally, write operations are done on the file obtained from the afs-server, and when done the newly changed file is converted and reinstalled.

• Now the user wants to read a file.

• A VOP READ is sent to the from the vfs-layer to xfs.

• xfs checks if it has valid attributes/and data (and updates if needed). Now VOP READ is simply performed on the stored vnode of the cachefile.

Tools and libs

What other tools does the arla suite consists of

libutil - a library for most used modules like hashtable, double-linked list, logging functions, date-parsing, etc

rx - the library for the rx protocol (see ⟨undefined⟩ [Rx protocol], page ⟨undefined⟩). lwp - the library for the lwp thread-package

(see ⟨undefined⟩ [LWP], page ⟨undefined⟩).

ydr - A stub generator that replaces rxgen. rxkad - The rx Kerberos authentication package. lib/roken/ - The library that will unbreak things that

is missing or broken.

lib/ko - A library of functions that is arlad-core related but is also useful for programs like vos, pts, fs, etc.

appl/libarlalib.a - A broken library that does all the hard work with connections etc.

appl/fs - The fs util, extra feature: getfid appl/vos - The vos util appl/pts - The pts util, extra feature: dump appl/udebug - Debug your ubik server

2.2 Rx protocol

Rx is run over UDP.

rxgen or ydr is used to generate stub-files, ydr is better since it generates prototypes too.

Current implemetation of rx it not that beautiful.

2.3 LWP

LWP is a pre-epmtive thread package. It does it’s context-switching by creating a private stack for each thread. The heart of the package is select(2).

The core of the package is ‘process.S’. Its currenty built with a bourne-shell file. ‘Process.s’ is preprocessed with cpp and then given to (g)as. The magic how to build this file a portable way is not fun.

The stack is checked for overruns in context-switches, but that is often too late. It might be an idea to add a red zone at the top of the stack to be able to detect overruns.

2.4 The files in arlad/

This is a short describtion of the files to bring new deveplopers up to speed.

2.4.1 The core of arlad

‘adir.c’ - contain all functions needed to to operations

on afs-directory file.

‘afsdir_check.c’ - check if a afs-directory looks sane. ‘arla.c’ - The startup and the standalone (-t) code. ‘arladeb.c’ - The logging code specific to arla, like aliases

for debugging masks.

‘cmcb.c’ - The callback-server that is contacted by the

server when a callback expire or a server want to send a InitCallBackState.

‘conn.c’ - The connection cache, responsible for caching connection

based on pag and security index. It will also create new connection when needed.

‘cred.c’ - Keep track of all credentails that all users have

inserted. Indexed on pag.

‘fbuf.c’ - An interface between rx and filedescriptors. It also

mmap files. ‘adir.c’ uses it.

‘fcache.c’ - Responsible for keeping track of files in the cache.

Also fetches files from the afs-server.

‘fprio.c’ - Tries to give files priority and therefore not

garbarge-collected that fast they would be otherwise. Think disconnected mode.

‘inter.c’ - A interface to hide all junk in fcache, just give it

a VenusFid and you have it.

‘kernel.c’ - The interface between arlad and the char-device. ‘messages.c’ - The rpc interface between arlad and xfs. ‘volcache.c’ - Cache for all volumes.

2.4.2 Operatingsystem specific files

These are the files that contain operating specific functions. Today it’s just conv dir().

‘aix-subr.c’ - AIX

‘bsd-subr.c’ - FreeBSD 2.2.6, OpenBSD 2.2, 2.3, NetBSD 1.3.x

‘hpux-subr.c’ - HPUX
‘irix-subr.c’ - Irix

‘linux-subr.c’ - Linux 2.0.x, 2.1.x, 2.2 ‘solaris-subr.c’ - Solaris 2.5.x, 2.6, 7

‘sunos-subr.c’ - SunOS

‘unknown-subr.c’ - Stub used when compiled on a unknown OS.

pioctl and kafs

pioctl is a ioctl but with a path instead of a filedescriptor. When you probe if there is a live afsclient you first run k_hasafs() that probe if there is a afsclient around. It also setup some static variables in the library. So if you start to do pioctl() w/o running k_hasafs(), you’re up to funny errors.

What k_hasafs() does is AFSCALL_PIOCTL with opcode VIOCSETTOK and insize == 0, ie you try to set a token (ticket) that is 0 bytes long. This is cleary invalid and kafs expects to find a EINVAL returned from syscall(2).

The pioctl is used more then just for AFSCALL_PIOCTL, other uses are AFSCALL_SETPAG (setting pag). It has also been in use for setting xfs debugginglevels.

When xfs discovers that a path is given in the pioctl() it does a VOP_LOOKUP on the path and if the returned a vnode that resides in afs then it extracts the xfs-handle for that node (that just happens to be the VenusFid) and passes that on to arlad.

The only ugly thing about the current implentation is that the syscall code assumes that the arlad on "xfs-fd" is the arlad that should get this syscall.

An example of using pioctl():

     int
     fs_getfilecellname(char *path, char *cell, size_t len)
     {
         struct ViceIoctl a_params;
         a_params.in_size=0;
         a_params.out_size=len;
         a_params.in=NULL;
         a_params.out=cell;
         if (k_pioctl(path,VIOC_FILE_CELL_NAME,&a_params,1) == -1)
             return errno;
         return 0;
     }

3 Debugging

3.1 Arlad

If arlad is run without any arguments arlad will fork(2) and log to syslog(3). To stop forking use the –nofork (-n) switch.

To enable more debuggning run arla with the switch –debug=module1,module2,... One useful combination is

        --debug=all,-cleaner

The cleaner output is usully not that intresting and can be ignored.

To set the debugging with a running arlad use fs arladeb as root.

     datan:~# fs arladeb
     arladebug is: none
     datan:~# fs arladeb almost-all
     datan:~#

When arlad doesn’t log to stdout, or a file, it logs to syslog.

3.2 xfs

XFS debugging does almost look the same on all platforms. They all share same debugging flags, but not all are enabled on all platforms.

Either you edit the file ‘xfs/<your-os>/xfs_deb.c’ or use fs to set it.

     datan:~# fs xfsdebug
     xfsdebug is: none
     datan:~# fs xfsdebug almost-all
     datan:~#

The logging of xfs ends up in your syslog. Syslog usully logs to /var/log or /var/adm (look in /etc/syslog.conf).

4 Porting

The largest part of the work needed to port Arla to a new operating system is in porting xfs, as kernel programming always is harder, less portable, and messier than user-space dito. Arlad in test mode (arlad --test) should work without any porting on any system that’s not very far away from Unix and that have berkeley sockets (including cygwin32). Therefore we will concern ourselves mostly with how to port the XFS module.

4.1 XFS

1. It helps to have source code for your operating system. In theory, if stuff was documented well enough, you wouldn’t need it. In practice they never are so you find out interfaces and how stuff work by reading the source code. If you’re unable to find source code for your OS, try finding source for the closest match. If your OS is based on BSD, try the appropriate version of BSD, for example.

2. If you don’t have source, try second best, include files. You can usually gather quite a lot of information on the workings of the kernel by reading the includes files in ‘<sys/*.h>’.

3. Be lazy Try to find out what other XFS port is most similar to your OS and start with that code.

4. Figure out how your kernel works.

You need to figure out how a few things work in your kernel:

1. Loading/unloading kernel modules That varies quite a lot but it’s probably easy to figure out if you have the source code for some other loadable module. Sometimes you can get the kernel to add your cdev, system call and file system automatically but usually you have to write code in your ‘entry-point’ to add these to the appropriate tables.

2. adding a new character device driver The kernel has a table of all known device drivers, ordered by major number. Some kernels have one for block device and one for character device and some have a common one. That entry usually consists of a number of function pointers that perform the operations (open, close, read, write, ...), and possible a name and some flags. It could look something like the following:

     struct cdevsw {
     int (*d_open)();
     int (*d_close)();
     ...
     };
     struct cdevsw cdevsw[];

These are then usually stored in a table ‘cdevsw’ indexed by the major device number. If you’re really lucky there’s a new to get the kernel to add your ‘struct cdevsw’ to the global table when loading the module or a function that does the addition for you. If not, you’ll have to fallback on looking for a free slot in the table and putting your struct cdevsw there. In some cases, this is not stored in a table but then there’ll be a way of adding entries to the new data structure so you don’t need to worry about it.

3. adding a new system call This is quite similar to adding a new cdev but the table is usually called ‘sysent’ instead.

4. adding a new file system Once again, quite similar in principle. The names of the structures tend to vary quite a lot more.

5. finding out how the VFS/Vnode switch works The structure vfsops contains function pointers for all of the file system operations. You need to figure out what operations you need to implement (usually at least mount, unmount, root, sync, and statfs). The operations that are performed on files are vnode operations (usually stored in a struct vnodeops), and you need to figure which of these you need and how they should work. Also, which is not as explicit, how vnodes are supposed to be allocated and freed and such.

5. Suggested plan of action

1. Start by writing a minimal hello-world module and make sure you can load and unload it properly.

2. Then add a device driver to the module which dummy functions and verify that works.

3. Try to fit the device driver functions in ‘xfs_dev.c’ into the device driver.

4. Do a dummy module with a system call and verify that you can call it.

5. Start trying to add enough of the vfs/vnode operations from ‘xfs_vfsops.c’ and ‘xfs_vnodeops.c’ so that you can build it.

6. Debug it.

7. Send us patches

5 Oddities

5.1 AFS

• Directories - UnixModeBits are ignored when the vnode is a directory.

• Errnos are sent over the network. Like Solaris ENOTEMPTY(93) doesn’t even map to an error on sunos4 where ENOTEMPTY is 66.

5.2 Operating systems

• On irix 6.5 you have to build the dirents depending on what ABI of the binary you are currently running.