Discovery Report for Linux/Athena, version 3.0. I begin to sing of Pallas Athena, the glorious goddess, bright-eyed, inventive, unbending of heart, pure virgin, saviour of cities, courageous, Tritogeneia. -- Homeric Hymns 28.1 Summary We propose the development and deployment of a third supported Athena platform, to run on commodity PCs, based on the the Red Hat Linux distribution. A Linux platform will have several significant advantages. Linux is already very popular at MIT, has a substantial base of knowledge already in place on campus and within I/S, and has given rise to great demand for a supported Athena release. Commodity PCs are cheaper by far than most other workstation hardware, and the Red Hat Linux operating system is available with no licensing fees. This availability includes all source code, so we will be able to adapt to customer needs more quickly, removing the need for any dependence on the vendor for support. At the same time, we can rely on Red Hat to build and maintain the base operating system in the same manner that our other vendors do. We intend to begin with a fairly limited scope, supporting this platform in the same manner as we do other Athena platforms, and we have developed some proposed strategies for dealing with the expected differences in support patterns and hardware availability, compared with other supported platforms. There would be an initial pilot cluster installation, probably in the W-20 cluster, to be ready in January 2000. It would be appropriate to make an initial installation of ten workstations, with more to come as experience and funding permit. We do not recommend any increase in the total number of seats in Athena clusters. The members of the Discovery Team were: Thomas Bushnell Bill Cattey Abby Fox Heather Harrison Greg Hudson Karl Ramm Emil Sit Oliver Thomas The discovery pre-planning was done by: Mike Barker Thomas Bushnell Bill Cattey Greg Hudson with assistance from Greg Anderson. Discovery pre-planning took place informally over time, and in a preplanning meeting in March, 1999. The discovery team met over seven weeks from late June to early August, 1999. Business Case Athena now supports two platforms: Solaris running on a variety of Sun Sparc workstations, and IRIX, running on SGI Indys and O2s. We propose to add a third platform to this set, Red Hat Linux running on commodity PC hardware. Commodity PC hardware is currently able to run a version of Athena thanks to two distributions made by the Student Information Processing Board (SIPB): one using NetBSD as the basic operating system, and one using an older version of the Red Hat Linux. Currently, students form the largest segment of the MIT community using the SIPB Athena distributions, which have also had a significant and growing presence in MIT labs and departments. Since I/S does not officially support these distributions, we have not been able to offer this alternative in the public Athena clusters. Many labs and other private Athena workstation customers have also been reluctant to use an unsupported release. Unfortunately, we can not rely on the SIPB to provide support--as a primarily student organization, the membership, capabilities and responsiveness of the SIPB waxes and wanes over time. At times, the organization has been very responsive to user requests and to the making of new releases in a timely fashion, but at other times the PC Athena ports have suffered due to lack of attention. A supported Athena platform based on PCs would lessen our dependence on just one or two vendors, and help foster marketplace competition. Right now we are essentially at the mercy of a small number of hardware vendors who could significantly increase our operating expenses. PC hardware is a commodity and is therefore available more cheaply and the prices for new hardware cannot be controlled by a single source. We propose to support a Linux Athena release. Given the large base of Linux users at MIT, among students, faculty, and staff, adopting Linux as a supported platform will enable us to recognize and support that base of users much more effectively than we can now. Linux is popular and widely understood. Because of our extensive experience in working with such systems there should be no difficulty in hiring competent and energized staff as necessary, for both development and support. This should avoid some of the difficulties we found in undertaking the SGI port, for example. Because the SIPB has already done a port to this platform, we are able to leverage their work and use it as the basis for our own. Engaging in this project will be a prudent investment, participating in the emerging Linux-based trend in computing. Many new innovations are tried first in Linux systems, and Linux is perhaps the most dynamic OS platform today. Many vendors who are withdrawing Unix support are at the same time adding or considering adding Linux support for their products; for instance, Lotus plans to release its Domino server for Red Hat Linux very soon now, and SGI has announced it will phase out IRIX and replace it with Linux by 2003. MIT could be a valuable asset to the Linux developer community and could help the Linux state of the art and the stature of Linux in the marketplace. By participating in a Linux effort, we would have access to a distribution channel for MIT-developed software, such as Moira and Zephyr, some of which has not been propagated to the world as successfully as we might like. Linux is developed using an open source model, which translates immediately into many important advantages which attach to its use as an Athena platform. First, because the source is available to us without bothersome restrictions, we are much more able to change it when necessary to meet customer needs. In general we can expect much more rapid customer response to be possible, with the option of doing simple support tasks in house when appropriate. Changes we make to the software can be easily contributed back to the developers, and there is a good chance they will be accepted into the main releases. The goal of having our enhancements become standard may be met more readily by Linux than it has historically been with other platforms. Finally, necessary bug fixes can be incorporated into our software without a strong dependence on the vendor's release methodology. We will build an Athena port for the Red Hat Linux 6.0 operating system, deploy a small number of workstations in a cluster as a pilot on or about IAP 2000, and continue to maintain it as a supported platform. We will also make a release available to the general Institute public at about the same time. This project will be of great value to students, providing them with a more reliable and stably supported Athena option than they have had previously on PCs. We will be able to purchase commodity PC hardware cluster workstations at a much lower cost, without sacrificing performance. Described below are the crucial resource allocations which will be necessary for success. Generally these are fairly small or easily adapted into existing frameworks. This is an extension of an existing model onto a new platform, and so we did not need to consider the broad structure of the Athena model or design anything so complex. Business model To the customer, the Linux Athena business model looks nearly identical to the existing business model for, for example, the Sun platform: * general purpose system with basic educational services and technologies * defined and supported hardware configuration * public Athena workstation software product * private Athena workstation software product * available third party software, balancing customer need against cost and availability * hardware and software support through Athena Cluster Services and Athena Consulting * Unix-like platform, with traditional Unix-based Athena services * the normal Athena look and feel Further details of scope are listed below. Listings of exactly what third party software is available will be provided on the What Runs Where web page (http://web.mit.edu/acs/www/whereruns.html). The expenses for the initial pilot portion of this project are estimated as follows. All numbers here are only very rough estimates. These estimates are based on the more detailed breakdown in the "Hardware Possibilities" section below. Direct Expenses: 23 Workstations @ $1500 $34,500 10 additional OLC students $16,200 / yr Direct Income: 30 additional Athena fees, @ $350 $10,500 / yr We recommend that the new workstations be purchased from the new projects academic computing budget. Ongoing hardware support and new workstations would be folded into the Athena renewal process on an ongoing basis, as is currently done for the other platforms. On-going Athena Consulting support costs would become part of the annual support budget. In addition, there are of course numerous indirect expense and income items; these are considered below under "resources". Also, it is expected that Linux/Athena PCs would over time replace existing workstations, and not represent additional seats in clusters. Accordingly, because PCs are very much less expensive than similar Sun or SGI workstations, we expect a great savings in the long run by using commodity PC hardware. See the hardware pricing discussion below. Scope of the project It is important in any project like this one to have a clear idea at the outset of which aspects are in scope and which are out of scope for the work. The discovery team found it valuable to discuss scope early on, and we found that this made it much simpler to make many subsequent decisions. We recommend similar clarity in the actual implementation. Based on our discussion, the following form the core products for the project: * Relevant documentation, such as the "what runs where" listings and the private workstation owners' guide. * A traditional Athena release for Red Hat Linux. * A support plan. * A layered Athena release if feasible. Here is the list of items we recommend to be within scope for the project: * At least one specific PC hardware configuration fully supported. * A Supported Athena Workstation Systems webpage * A software release to support cluster workstations. * A software release to support private workstations. * A software release to support layered Athena if feasible. (It is possible that the best development strategy for producing a traditional Athena release for Linux will also make a layering release easy for us to do. If that does not turn out to be the case, then we would postpone or omit this portion of the project.) * Listing, testing, and buying third-party software, as necessary, to parallel existing software on the Sun and SGI platforms. * Updating Athena documentation. * Providing some kind of private workstation owners guide. * Providing some kind of documentation for layering, if we do layering. * Developing a support plan, including RCC's, consultants, policy, etc. * Documentation for cluster support and training. * Keeping the I/S PC purchasing recommendations in sync with the supported hardware for Linux/Athena. * Investigating support for previously recommended hardware. * Listing the various PC hardware options that we support and the levels of support each would receive. * Evaluating and recommending particular supported peripherals. * Considering security of software in clusters, especially given the wide availability of OS install media for various operating systems. * Supporting transition for locker maintainers from the existing SIPB Linux/Athena arch to the new one. * Providing appropriate information for existing SIPB Linux/Athena users to switch to the I/S Linux/Athena. * Be aware of issues that inhibit running Athena on non-Red Hat Linux systems and incorporate relevant bugfixes. * Dual boot on a layered system, if we support layering. (Dual boot is very technically challenging on a traditional Athena workstation. A layered installation is inherently more flexible, and we would support dual boot for such installations.) We judged the following items to be outside scope for this project. While many of them are valuable things that I/S may do in another project, it was important to identify that they would need and independent justification; they are either not natural parts of this project or else would greatly dilute work from what we thought were the core tasks of supporting the new platform. * Support all PC hardware. * Port third party software. * Encourage students to buy a particular platform. (This is the responsibility of the desktop products team.) * Physical security of cluster workstations. (This is of course very important, but it is not a task for this project to directly consider.) * Support for legacy PCs. * Support for disconnected operation (laptops). (This would be very popular, and might be an excellent follow-on project, but is secondary to the primary goal of supporting ordinary workstations.) * Support for SGI Visual PC. * Support for VMware (and similar products). non-Wintel hardware. * Dual-boot on cluster or private workstations. (As noted above, this task is a technically challenging and should be secondary to making a working Athena installation. It could be done later if the need arose.) * Package Linux security enhancements for non-Athena use. (The release issues for an Athena release and timing concerns mean that while Athena can usefully cooperate with such groups as net-security, it is not convenient to directly provide software enhancements for other environments.) * Support or test parts of Athena on non-Red Hat Linux distributions. * Locate and purchase new kinds of third-party software to run on Linux. (There may be many useful products MIT should consider purchasing, which would become useful because we would now run Linux, but identifying and purchasing them is not directly part of this project.) * Evaluate and use KDE or Gnome as a supported desktop. * Software support for transition from SIPB Linux/Athena. * Support non-Athena-standard parts of Red Hat or related "operating system" hardware. Support Issues Many people have PCs and there is a wide spectrum of hardware in use on PCs. Some of this hardware is not supported by Linux at all, some of it is supported only sketchily, some is well-supported but not by Red Hat, and some is very well supported. While a concrete support plan will need to evolve as part of the project itself, we thought it important to consider some support stresses that this project would produce, and we tried to anticipate how those stresses might be met. We decided that we would publicly list hardware and describe the levels of support that we would provide. First, we would have defined supported configurations; these would be complete systems that we would specifically test in the release cycle; all cluster machines would be included in the defined supported configurations. For users with those configurations, we would provide full software support. We would also list hardware that has been tested by development, or which we have other good reason to trust. This hardware would be recommended to users, and we would support it on a "best effort" basis. Because the complete configuration would not have been tested, and because various hardware added to a computer might interact in such a way as to prevent proper interoperation, we would not be able to provide the support guarantee that we can for the defined supported configurations. We would provide a list of those devices that Red Hat lists as supported, but with which we have no experience of our own. For this hardware, we are concerned that there might be unexpected issues that would take significant time to resolve. Accordingly, we recommend supporting these devices, but only on a "minimal effort" basis. Finally, hardware which we have not tested or do not have experience with and which Red Hat does not list as supported, we would provide no support guarantees whatsoever. The team also discussed the various sorts of support sinkholes that we might encounter. The supported hardware list and the scope definitions proved to be a great value in composing possible answers to those questions. Hardware Possibilities We recommend the purchase of thirteen workstations for development and testing, and ten more for an initial pilot install in the clusters. (The thirteen would be allocated as follows: One each for SIPB, TPS, FL, test cluster, zone, hotline spare; two each for OLC and development; three for release engineering.) Available PC hardware changes rapidly. We recommend careful investigation of availability and pricing prior to the purchase of new hardware. It would be wise to compare prices across multiple suppliers if possible. The following pricing is indicated only as a very rough guide. The existing arrangements and decision processes for purchasing hardware should be used here to make the final decisions about exactly which hardware to obtain (of course, in communication with developers). It is suggested that the PC hardware we purchase should be roughly equivalent to the current Sun platforms. We priced PCs on August 15 from the MIT NECX preferred products page. There are four base systems described there, with the following prices: Optiplex GX1 Mid-Size $1255.00 Optiplex GX1 MiniTower $1284.99 Optiplex GX1p Mid-Size $1468.56 Optiplex GX1p MiniTower $1503.00 These prices include 17" monitors; we should purchase the larger 19" monitors, for about an additional $200. We also should purchase more memory than the default configurations. An additional 128MB upgrade is about $170. The current supported Athena platforms, as priced by NECX, cost the following: Sun Ultra 5 $2395.00 Sun Ultra 10 $3952.00 SGI O2 $5915.07 Resources and related issues We estimated the following resources would be needed for development: * Adding a 3rd platform for release engineering (about 10% FTE) * OS Support person (50% - 100% FTE, prob about 75%) * It will be easy to find people who are experts with the Linux OS * We might want a vendor liaison or two--one hardware and one software And the following for software support: * Public cluster, private ws support: hire up to about 10 more OLC students, for a total of about 1.5 FTE. * Training of consultants * Time to investigate and license, etc., various 3rd party software * Publications persons to review and update documentation and minicourses * Publications and training people to review and update training materials And the following for hardware support: * hotline continues to provide hot spares on a stock and replace basis; users could receive support either on a time and materials basis or through a PC Service support contract. * Might be increased load; PC hardware might be less reliable * People might install other OS's on top of installed cluster machines, requiring more cluster services work * Greater theft demand for PC hardware With the exception of a possible need to hire more OLC students, all of these staffing allocations are met with existing staff. We recommend that the existing Athena fee structure be carried over into the Linux/Athena space. Students should not be required to pay fees, but labs, faculty, and staff machines would be charged an Athena support fee. We also note that the fee structure might be usefully adjusted to allow people to purchase only Athena software support if they don't need or want hotline reinstall service. If we do a layered release then it might also be useful to allow fee-based access to third-party software which doesn't require paying the traditional Basic Athena Service fee. We would still want to have fees for infrastructure support as well. The exact parameters of any new fee structure involve complex decisions that we did not think were directly in scope for this project. However, it is expected that the different user base and usage patterns might impose new stresses that require suitable changes in the fee structure. We are also concerned that as licenses for third-party software are negotiated and arranged, the needs of students and other privately owned workstations not be ignored, whether they are running machines which are fully part of the Athena model or a layered Athena release. Implementation Timeline The following is a rough timeline for implementation of the project. The release states named below have specific meanings which are detailed in the release order template web page (http://web.mit.edu/release/www/release.html). August: Software in crash-and-burn release state. Order 13 [or 23] machines. September: Software in alpha release state. Have 13 testing and support machines in hand. [ Have 10 cluster machines in hand. ] Start release notes. Notify hotline of hardware installation requirements. October: Software in beta release state. Start writing and expanding documentation. Begin third party software testing. Begin OLC training. November: Software in early release state. Notify Athena minicourse people of new hardware. [ Order 10 cluster machines. ] December: Software in release. [ Have 10 cluster machines in hand. ] Notify hotline of presence of machines for W-20 cluster. January: Deploy cluster machines. The brackets indicate uncertainty about whether the cluster machines should be bought just before installation is planned, or at the same time as the development/testing machines. Open Issues This report leaves two important issues open. First, it is unclear whether the cluster hardware should be purchased in August or in November. If we purchase it now, then it will be essentially unused for four months while the release is in preparation, However, if we purchase later, then we might not be able to obtain exactly the same hardware, which imposes a potential development and support cost if the differences are significant. Second, it is not decided here whether to support a layering release. This will depend on some key implementation details. If the layering release "falls out" of the standard release, then we would make it available as described here, but if it turns out that it is not convenient to do so, then we would not provide a layering release directly. (We might still choose to do so at some later point; also we would of course cooperate should others [perhaps the SIPB] desire to produce one.) A layered release involves both policy and technical components. This proposal involves doing a layered release if the technical components are simple and easily accomplished, but there might still be policy problems with third-party software, Athena support fees, and so forth. These problems are not unique to a layered Athena release or to the Linux platform, but the stresses that such a platform or release might put on the system could make the existing arrangements untenable to maintain. We believe, however, that a suitable combination of appropriate fees and simple technical solutions could address these potential problems.