From: epeisach@Athena.MIT.EDU
Date: Wed, 15 Apr 92 11:58:03 -0400
Message-Id: <9204151558.AA00694@kangaroo>
To: trb@Athena.MIT.EDU
Cc: basch@Athena.MIT.EDU, hoffmann@Athena.MIT.EDU, salemme@Athena.MIT.EDU,
        tom@Athena.MIT.EDU
Subject: Mkserv proposal update


Proposal as promised. The postscript version will be sent to the trb
meeting...

Will make paper copies available.

--------------------------------------------------------

	       MKSERV - Athena Service Configuration Tool

			      Ezra Peisach


				4/15/92


Introduction
------------
Mkserv is Athena's supported program that allows configuration of
workstations such that local customizations will be maintained across
release updates.  Mkserv originally had to be run by hand after an
update to recustomize a server, but now is linked directly with the
update process to handle this automatically. Currently only 8% of
private machines now have both configuration variables PUBLIC and
AUTOUPDATE set to false.1 Mkserv has revolutionized the update and
customization process and needs to now move into the next phase where it
can be easily used to add services outside the release cycle. It is
intended that the changes outlined in this proposal will allow for this
and at the same time provide support for future services.

This proposal will outline changes for mkserv with the following rtequirements:

   Auto-update continues to work 
   Mkserv works as it has in the past 
   If  Auto-update cannot run mkserv, update does not take place 
   Nothing should be stored locally (in case of disk failure)
   Existing base of services continues to work 
   An ability to track service version numbers should exist


Functional Description of Mkserv
--------------------------------
Mkserv adds customizations to a local directory hierarchy. Full
directory hierarchies are reproduced as would be found on the system
packs delivered to the machine with individual files copied locally with
symlinks making up the rest. Root symlinks to a system pack are
redirected through this secondary area.  The program synctree based on
reconcile developed at LCS by Tim Sheppard and Stan Zanarotti is used
for the copying.

There are two reasons for this copying action instead of running
binaries from system packs. The first is to prevent RVDs from being
locked from being unmounted, thereby preventing operations from shutting
down servers and moving machines in a transparent way. The other reason
is that after a power failure, servers should not depend on equal or
higher level services in order to start up. Essentially, mkserv's
original goal was to provide for an individual service entity.

Mkserv currently is a shell script, which, based on user supplied
options, will add or remove services from a machine. This may involve
making changes to /etc/athena/rc.conf, running configuration scripts
based on services being defined, copying files locally, and displaying
any follow-up messages to the user.  Users may provide supplementary
customization scripts and configuration files.  (See Appendix A for a
list of currently supported services)

Detailed Operation of Mkserv
----------------------------
Mkserv sets shell environment variables which may be used by other parts
of the customization process. These include RVDROOT (where /srvd is),
SRVDIR (where mkserv configuration files may be found), LOGFILE (where
important messages to the user are queued up and displayed upon
completion), SERVERDIR (where local files will be stored2) and the PATH.
For a complete list of variables set, see Shell Environment Variables.

The current list of configurations may be found in
${SERVERDIR}/.services and mkserv uses this file to determine what
services are configured when it starts. During customization, mkserv
adds a line to /etc/athena/version indicating an update is underway. If
mkserv is interrupted, another update will not take place until the
situation is rectified.

Command line arguments are parsed, and a list of changes to the
.services file determined.  A service may be added if ${SRVDIR}/xxx.add
exists. Implicit services may be defined by mkserv if the the file
${SRVDIR}/config indicates a corequisite of one service on another.

Services xxx.add and xxx.del scripts are then run as appropriate based
on services being added or removed. It should be noted that even if
mkserv is invoked with only a new service, the xxx.add scripts will be
run for all services that the machine is configured for.

The file ${SRVDIR}/services.sync contains the master synctree
configuration file for all services .  The C preprocessor is run other
this file with all service names being defined (capitalized). This
generates the synctree rules file for the services defined. If a user
provided a ${SERVERDIR}/.private.sync it is concatenated to this file.
Synctree is then run to copy any files local and create the symlink
farms. Root symlinks are then changed to point to the secondary staging
area.

The master config file ${SRVDIR}/config also lists rc.conf changes based
on services installed. rc.conf is updated based on this files and can be
changed by the services' xxx.add and xxx.del scripts.

${LOGFILE} is then displayed on the console so that important messages
were not lost. These may include output from the services xxx.add and
xxx.del scripts reminding users of other actions which are required in
order to make some services completeyl functional. (For instance
requesting a srvtab).

If the user has provided a ${SERVERDIR}/.private script, it will then be sourced. If it returns successfully, /etc/athena/version is updated and final messages printed. If it does not return successfully, then it is assumed that the update did not succeed, and mkserv exits indicating this status.

The user is expected to reboot the system after mkserv has been run. In
case of the the workstation update process, the reboot is automatic.

Proposed Changes to Mkserv
--------------------------
It is desired to allow new services to be introduced outside the release
cycle while at the same time allow automatic updating of machines and
servers. In order to accomplish this goal, a filesystem containing
mkserv configuration files will be created. This filesystem will be
attached during the update process and if for some reason the filesystem
is unavailable, the update will be canceled. This also means for
autoupdating of workstations which have had mkserv run on them, that
they may not be able to auto-update immediately anymore.

The heart of the change would be to allow service's xxx.add and xxx.del
files, and three new files xxx.sync, xxx.dep, and xxx.ltp to be found in
a configurable search path. The path would be as follows by default:
/mit/mkserv/services/${VERS}:${CONFIGDIR}, followed by user specified
paths. The directory ${CONFIGDIR} which will be ${SERVERDIR}/config,
will be added as a repository of private configuration files as well as
placement for any storage of xxx.ltp files. The user may add additional
paths (which may have to be attached) after the default, but only the
first instance of a service in the path will be used. For this reason,
users writing private services should be careful in their choice of
names in order to prevent future conflict.

A services version is based on a version line in the xxx.add file. While
it is recommended that all files has a version number, the xxx.add is
special and will used used as the version number of a given service.
(See Version Numbering for more details)

A typical invocation of mkserv would look as follows:

Mkserv would attempt to attach all lockers and verify that all services
that are currenlty in use may be found. If not, mkserv would not be able
to preform an update and would indicate the fact to the user. A new
option "updatetest" which will cause mkerv to exit with a status of 0
meaning that an update may take place and 1 if not. This will be
necessary for autoupdates.

A service is now defined as being available for use on a particular
machine if a xxx.add file exists as before, and the file indicates that
it operates on the given platform. (See Supported Platform
Specification)

A new mechanism for handling dependencies will exist. All services may
have an optional xxx.dep file which contains simple rules for declaring
dependencies. (see Configuration of Dependencies for such rules). These
dependencies as before may trigger other services to be defined.

As before, service xxx.add and xxx.del scripts are run, but instead of
making changes to certain configuration files themselves, the scripts
may schedule changes that mkserv will handle itself. These command lines
are appended to a new temporary file ${CONFCNG}. (See Configuration File
Update Rules for details).

Synctree rules would then be formed by combining as .sync files together. Note that it is possible that a service will have no rules. This includes the optional user supplied .private.sync.  The C preprocessor will be run over the .sync files with every service name defined (as before) but additionally -D`machtype` would be added in case the cpp of the particular machine fails to define anything platform specific.3

After the synctree is run and root symlinks updated, all deleted
services will be removed. As before, if a .private script exists it will
be executed then.

Problems
--------
There is a possibility that cvertain clients will not be able to access
the mkserv locker and therefore be able to update in the future. These
are expected to be limited and usually fall into the case of Operations
controlled servers, which do not use the automatic update proceedure.4

Another problem would be the changing of services post release. Version
numbering will be critical in understanding what is running on a given
machine. This policy issue is not dealt with very well here. Instead, as
an alternative, mkserv will also log the version number of any service
which is running in the .services file. This would allow a workstation
owner to quickly determine his service versions.5

Another possibility may exist where a machine may never be able to be
updated as a service configuration definition may no longer exist. This
can happen if a user creates his own service and deletes the
configuration files. There is no solution for this problem.

Shell Environment Variables
---------------------------
During the update process, certain shell variables will be defined. Some
of the variables are specific to mkserv while others contain machine
configuration which may not be able to be determined during the update
process. The following variables are defined:


RVDROOT		Where /srvd is

SRVDIR		Where mkserv configuration files may be found

LOGFILE		Important user messages may be appended to this file
		and later displayed to user

PATH		Shell PATH 

VERS		Release version updating to

MACH		Machine type as found in /etc/athena/version

SERVERDIR	Where local customizations go
		(/site/server, /var/server, /usr/local/server)

CONFIGDIR (new)	Where local configuration files will be copied.
		${SERVDIR}/config

CONFCNG (new)	Where configuration rules for rc.conf and passwd are stored

SERVICES (new)	Contains list of services that will exist on machine
		after update


Configuration File Search Path
------------------------------
As highlighted before, there will be a search path for the service
configuration files. This path will be
/mit/mkserv/services/$(VERS):$(CONFIGDIR). The VERS in this case is
without the release letter - onlny number. A user may specify alternate
search paths which will be appended to the end of the list through
command line specifying locker:path where the locker is optional for
local files. Said locker will be attached without authentication during
the update. The $(SERVERDIR)/.services file will contain such entries in
the order they are specified. This will allow future updates to
"remember" the environment in which to run mkserv. To remove an entry
from the path, use -locker:path. It will be required for mkserv to
verify that all services will be able to be updated after removal of a
path entry.

Version Numbering
-----------------
Version numbers will be determined by RCS. All configuration files
should have a version number associated with it. If not present, version
0.0 is assumed. All that is needed is a line which reads:

# $Revision:$
or
/* $Revision$ */

The version number as found in the xxx.add script will be appended to
the line defining the service in $(SERVERDIR)/.services. This will allow
a user to quickly determine the version number of all services.

Supported Platform Specification
--------------------------------
As not all services are applicable to all platforms, there must be a way
of specifying support. A comment line in the xxx.add script will contain
a list of supported platforms.

The format will be:

# $supported: <platforms>

Platforms will be a list of letters representing the platform in
question. The following are registered definitions:

V	Vax BSD 4.3
R	RT BSD 4.3
D	DECstation Ultrix
P	RS/6000 AIX (P for Powerstation)
S	Sun running SunOS

If the supported line is missing from the xxx.add script, it is assumed
that the service may not be installed.

Configuration of Dependencies
-----------------------------
With the old mkserv, dependencies existed because of common interactions
between services. These were hard coded in the master config file, very
rudimentary and unable to allow for complex decisions. For this reason,
mkserv ops would imply remote and quota as well which is not really what
is desired.

Every service will have the option of having a xxx.dep file. It will
contain rules for the C preprocessor which will echo out service names
to be added. Lines that return 1 or 0 will be ignored. This would allow
the "ops" dependency to be written as follows:

#ifdef OPS
REMOTE
#ifdef NFS
QUOTA
#endif 
#endif

The advantage of this method is that it is possible to indicate exactly
what is desired. If someone had invoked mkserv with "ops remote" cpp
would return (after comments are removed)

1
QUOTA

It is important to note that these "dependencies" are really
corequisites. Running these dependencies makes no statement as to the
current state of the machine and if the service is already present. The
dependency generation loop would run until no new corequisites were
discovered or the number of services+1 to prevent possible looping
errors caused by service writer error.

Configuration File Update Rules
-------------------------------
Mkserv must be able to change certain configuration files including
rc.conf, inetd.conf, and password. The file ${CONFCNG} may be appended
to with the following rules by services' xxx.add and xxx.del scripts:

adduser hesiod <username>	Adds the user username to
				the password file based on
				hesiod (if doesn't exist and will update)

adduser nohesiod <line>		Adds the line to the password file
				if not present. Assumes format as
				returned from hesiod.

deleteuser <username>		Removes the user username from the
				password file

addservice <data>		Adds service to inetd.conf if it does
				not exist. The format should be that as
				used on BSD platforms.7

deleteservice <service>		Removes service - may revert back to
				default value as indicated on ${RVDROOT}

switchservice <service> <on/off>	Switch service entry in
					inetd.conf to be switched or unswitched

conf <variable> <value>	Changes rc.conf variable. Value may be "default"
at which time the default is restored from ${RVDROOT}/etc/athena/rc.conf

All password file changes will affect both /etc/passwd and
/etc/passwd.local if it exists or the appropriate files for the
operating system.

${CONFCNG} will be ordered such that remove rules and default rules take
place first ensuring that removing a service and adding a service will
not result in confusion. Unswitch rules will take place after switch on
rules for inetd.conf.

There is a possibility that conflicts may occur if two services try to
set the same configuration variable to different values. This will be
identified and displayed to user and a default value left behind. This
should be designed never to happen by those writing services.

Global master.sync file
-----------------------
A master.sync file will be required to establish some basic rules that
must come before all other xxx.sync files. This file establishes the
copy of usr and the global linking rules which need to be first. These
were implicit in the old global services.sync file.
 
New Customization Scripts
-------------------------
Also envisioned being added is the use of a shutdown script. There
should be a clean mechanism to shutdown services before an update.  If
${SERVERDIR}/.shutdown exists, the update scripts will execute it during
the update process to shutdown services.

On New Kernels
--------------
Currently mkserv is executed after a new kernel is copied local. This
has a major disadvantage as programs that rely on the namelist from the
kernel may fail. These include machtype, ps, pstat, etc. In the future,
the kernel update will take place after mkserv is run to facilitate
these changes. It is believed however, that it may be necessary to run a
particular configuration after a new kernel has been copied (say to
alter certain parameters) and therefore if a special file
${SERVERDIR}/postkernel exists, it will executed after the kernel is
copied locally. As this is considered very rare, no explicit management
instructions are provided at this time.

Optional Support Flags
----------------------
It is envisioned that mkserv will require new features over time and
there must be a way for mkserv configuration scripts to indicate that
they support a particular feature for mkserv. A line should exist in the
xxx.add if an option is supported. This optional flag support is not
required.

The format will be

# $options: <options>

Currently, there are no defined options, but they will be single letter
as in the platform support. One option which might be imposed someday,
is for mkserv -n to work, an option for the scripts would be to make no
configuration changes. This is all reserved for future work.

Service ltp Files
-----------------
It is believed that as services develop, there will be the need for
services to store configuration options in a file which will not be lost
during an update and may be used to set defaults for a service during a
reconfiguration. In order to prevent a filename namespace collision it
is recommended that such data be stored in a file named
${CONFIGDIR}/xxx.ltp. (Long Term Perishable) Examples of data that might
be stored here include tty entry names, modem speeds, etc.

Requirements of Service Add/Del Files
-------------------------------------
The following are guidelines and recommendations for services which have
been used in the past.

The service initialization is allowed to ask questions of the user
during initial install, but should be prepared to have no input present
during autoupdate. For this reason a default should be given based on
the current configuration. The xxx.ltp file may be useful for this if it
cannot be determined after autoupdate has changed the machine
configuration what the prior configuration was.

All xxx.add and xxx.del scripts should return with an exit status of 0.
Currently mkserv ignores the exit status of these scripts, but in a
future generation of mkserv maybe a backout mechanism can be put in
place.

Other requirements include proper version numbers and entries in the
xxx.add script for supported platforms and options supported by the
scripts. (See Version Numbering and Supported Platform Specification)

Conclusion
----------

The changes to mkserv outlined in this document are believed to be
sufficient to allow new services to be added in a supported way in the
future. It is also believed that newer features to mkserv may be added
in the future in a supportable way.


Appendix A - Table of Services Currently Existing
-------------------------------------------------
Service	 Platforms	Config change	Depends		Maint
afs	 BSD/Ultrix	AFSSRV				O
discuss	 BSD/Ultrix	inetd/passwd			Y
eos	 BSD		RPC, FXSERV			O
			rc.local
galatea	 BSD/Ultrix					Y
kerberos BSD/Ultrix	KRBSRV, KADMSRC			N
local	 BSD/Ultrix	------	 			O
mail	 SENDMAIL					Y
nfs	 BSD/Ultrix	NFSSRV,RPC			Y
nip	 BSD		NIPSRV				O
notes	 BSD		-------				N
olc	 BSD		OLC				O
ops	 All		passwd		quota remote	O
pop	 ???		inetd/passwd	mail		Y
print	 BSD,Ultrix	LPD				O
quota	 BSD		QUOTA				O
remote	 All		NOCREATE, inetd			Y
rvd	 BSD		RVDSRV				O
uucp	 BSD		passwd				N
zephyr	 BSD		ZEPHYR				O
Y - Supported for general users
O - Operations currently using and current
N - No support but worked at one time


Appendix B - Examples
---------------------
Remote Service

The remote service changes the variable NOCREATE  in rc.conf and changes entries in inetd.conf.

remote.add
#!/bin/sh
# $Revision: 1.1$
# $supported: VRDP

# Determine is mkserv ops is in affect or if no input is present (autoupdate)
# ops.add handles the NOCREATE variable so don't ask.
grep '^ops$' ${SERVICES} > /dev/null
if [ $? != 0 -a -t 0 ]; then 
	if [ "${NOCREATE}" = "true" ]; then
		default=yes
	else
		default=no
		NOCREATE=false
	fi
 	echo -n "Do you wish to restrict access to only those in /etc/passwd [$default] ===>"
	read foo
	case "$foo" in
y*|Y*)	NOCREATE=true;;
n*|N*)	NOCREATE=false;;
	esac

	echo "conf NOCREATE ${NOCREATE}" >> ${CONFCHG}
fi

for i in  ftp telnet shell login kshell klogin rkinit  exec; do 
	echo "switchservice $i  off" >> ${CONFCHG}
done

exit 0


remote.del
#!/bin/sh
for i in  ftp telnet shell login kshell klogin rkinit  exec; do 
	echo "switchservice $i  off" >> ${CONFCHG}
done

# Note: Purposely do not revert NOCREATE variable. May still be set for a reason
	
exit 0


remote.sync
#ifndef local
copy usr/athena -f
copy usr/athena/lin -f
copy usr/athena/lib/init -f
copy usr/athena/lib/init/* -p -f
link usr/athena/lib/init/xsession -f

copy usr/athena/etc -f
copy usr/athena/etc/erlogind -p -f
copy usr/athena/etc/klogind -p -f
copy usr/athena/etc/kshd -p -f
copy usr/athena/etc/login.krb -p -f
copy usr/athena/etc/rkinitd -p -f
copy usr/athena/etc/tftpd -p -f

copy etc -f
copy etc/shells -p -f

#ifdef ultrix
copy usr/etc -f
copy usr/etc/ftpd -p -f
copy usr/etc/telnetd -p -f
copy usr/etc/rshd -p -f
copy usr/etc/rlogind -p -f
copy usr/etc/tftpd
#else
copy etc/ftpd -p -f
copy etc/rexecd -p -f
copy etc/rlogind -p -f
copy etc/rshd -p -f
copy etc/telnetd -p -f
copy etc/tftpd -p -f
#endif

#endif


Footnotes:
----------
1) 1090 machines responded to rc.conf queries (VAX, RT, DECstations).
   401 had PUBLIC set false, 77 without AUTOUPDATE set true. Of the 77,
   about 45 were Operations controlled machines which should not autoupdate
   (i.e. NFS, AFS, RVD, and print servers)

2) /site/server for BSD platforms, /var/server for Ultrix machines,
   /usr/local/server for RS/6000 machines.

3) AIX in particular 

4) If needed, starting AFS before an update may be done.  

5) With the Athena SNMP agent, it is possible to get this data remotely.

6) It is recognized that on an RS/6000, more than just the passwd file
   needs to be updated. Mkserv would handle all the details.  

7) On platforms not running our inetd, mkserv will handle conversions.
For the RS/6000, inetdimp will also be executed on completion.