USENET Version B Installation _M_a_t_t _G_l_i_c_k_m_a_n _C_o_m_p_u_t_e_r _S_c_i_e_n_c_e _D_i_v_i_s_i_o_n _D_e_p_a_r_t_m_e_n_t _o_f _E_l_e_c_t_r_i_c_a_l _E_n_g_i_n_e_e_r_i_n_g _a_n_d _C_o_m_p_u_t_e_r _S_c_i_e_n_c_e_s _U_n_i_v_e_r_s_i_t_y _o_f _C_a_l_i_f_o_r_n_i_a _B_e_r_k_e_l_e_y, _C_a_l_i_f_o_r_n_i_a _9_4_7_2_0 _R_e_v_i_s_e_d _b_y _M_a_r_k _H_o_r_t_o_n _f_o_r _v_e_r_s_i_o_n _2._1_0 _R_e_v_i_s_e_d _b_y _R_i_c_k _A_d_a_m_s _f_o_r _v_e_r_s_i_o_n _2._1_0._3 1. Introduction This document is intended to help a USENET site install and maintain the network news software. Please ask questions of Rick Adams*; such questions will help to point out areas that need to be addressed here. The overall order of things to do is: (a) Find somebody to link up with. You need a network connec- tion of some kind, for example, ARPANET or UUCP. If you must use UUCP and have no connections, you must have at least a dialup and preferably a dialer, and find someone willing to call your machine. The USENET directory may be helpful in finding some other site geographically near yours to hook up to. (b) Create a _l_o_c_a_l_i_z_e._s_h script to make local changes to the makefile and _d_e_f_s._h files. (Section 2 gives more details about creating _l_o_c_a_l_i_z_e._s_h.) Once you're finished editing _l_o_c_a_l_i_z_e._s_h, create a _d_e_f_s._h and _M_a_k_e_f_i_l_e tailored for your site with the command sh localize.sh Inspect _d_e_f_s._h and _M_a_k_e_f_i_l_e to ensure that all your local customizations got into your final versions. If you saw a "?" when you ran _l_o_c_a_l_i_z_e._s_h, one or both of the files is certainly wrong. It's a good idea to anchor the patterns in _l_o_c_a_l_i_z_e._s_h's _e_d(1) scripts, especially in its _M_a_k_e_f_i_l_e- editing lines. For instance, use /^UUXFLAGS/ instead of /UUXFLAGS/. __________ * ARPANET: rick@seismo.CSS.GOV, UUCP: seismo!rick USENET Version B Installation SMM:10-1 August 13, 1987 9SMM:10-2 USENET Version B Installation 9(c) Compile the software using the _m_a_k_e(1) command. (d) _S_u(1) and type "make install". This will copy the files out to the right place and make directories containing most of the important files. It will configure you in with a con- nection to _o_o_p_s_v_a_x via UUCP links. This is undoubtedly wrong, so you will have to configure links as needed. If you are upgrading from a version older than 2.10.3, do "make update". This will cause various checks to be performed on important files in LIBDIR. The results will be reported to you. If you are not sure if you should do "make update", do it. It will not hurt anything if you have already done it. (e) After editing the configuration table, get your contact at the other end of the link to add you to their netnews _s_y_s file. (f) Post a message to the to._s_y_s_n_a_m_e_s_y_s_n_a_m_e newsgroup which should be set up to go only to the site you are linked to, as a test. Have the other person send a message to your system using the same mechanism. If this doesn't work, find the problem and fix it. (Please don't use net.test unless there is no alternative. It is almost always possible to use test, or to._s_y_s_n_a_m_e_s_y_s_n_a_m_e or some _l_o_c_a_l_l_o_c_a_l.test group, instead of net.test.) (g) Fill out a USENET directory form (the file _d_i_r_f_o_r_m in the _m_i_s_c directory). Post a copy to the USENET newsgroup net.news.newsite and mail a copy to _c_b_o_s_g_d!_u_u_c_p_m_a_p. (h) Format the document toHow (the file _h_o_w_t_o._m_n in the _d_o_c directory), the document toHow (the file _m_a_n_n_e_r._m_n in the _d_o_c directory) and the document Law""Copyright (the file _c_o_p_y_r_i_g_h_t._m_n in the _d_o_c directory) and post them to your general newsgroup with a long expiration date. You can use _i_n_e_w_s(1) or _p_o_s_t_n_e_w_s(1) to do this. (i) It will probably be necessary to fix your uucp commands to allow _r_n_e_w_s and to support the -z and -n options (if you are lucky enought to have the source). 2. Installation 2.1. Configuration Local configuration of the USENET version B software re- quires you to edit a few files. Most importantly, the _d_e_f_s._h and _M_a_k_e_f_i_l_e files must be created from their templates _d_e_f_s._d_i_s_t and _M_a_k_e_f_i_l_e._d_s_t. You should create a shell script called _l_o_c_a_l_i_z_e._s_h which copies the files and makes local changes to the copies. Even for a completely vanilla site, some changes will be necessary. For example, your script should start with _l_o_c_a_l_i_z_e._v_7 or _l_o_c_a_l_i_z_e._u_s_g. You should include the name of the local organization (MYORG) and the uid of the local news super user (ROOTID). You should also choose how your hostname will be August 13, 1987 9USENET Version B Installation SMM:10-3 9determined. If you are a USG site, define UNAME in _d_e_f_s._h. If you are running 4.[23] BSD, define GHNAME in _d_e_f_s._h. If you have your UUCP name in /_e_t_c/_u_u_c_p_n_a_m_e, define UUNAME in _d_e_f_s._h. Other- wise, news will look in the file /_u_s_r/_i_n_c_l_u_d_e/_w_h_o_a_m_i._h for a line of the form #define sysname your-sysname If you are running System 3 or System 5, you are a USG site. Otherwise, unless you are in AT&T, you are probably a V7 site. The previously mentioned defines are the only modifications that are _n_e_c_e_s_s_a_r_y to install news at your site. However, you will probably want to change some of the ones listed below. If your compiler does not accept "(void)", the simplest thing to do is add "-Dvoid=int" to the CFLAGS line in the _M_a_k_e_f_i_l_e. A sample localize shell script can be found in _l_o_c_a_l_i_z_e._s_a_m_p_l_e. The most important parameters are: 2.1.1. ROOTID The numerical uid of the person who is the news super user. This should not be set to 0. Normally it is set to the uid of the news contact person for the site. If it is not defined, the uid of NOTIFY will be looked up in /_e_t_c/_p_a_s_s_w_d and used instead. 2.1.2. N_UMASK Mask for _u_m_a_s_k(2) system call. Set it to something like 022 for a secure system. Unsecure systems might want 002 or 000. This mask controls the mode of news files created by the software. Insecure modes would allow people to edit the files directly. 2.1.3. DFLTEXP The default number of seconds after which an article will expire. Two weeks (1,209,600 seconds) is the default choice. If you wish to expire articles faster than two weeks, it is recom- mended that you use the -e flag to expire instead of decreasing DFLTEXP. 2.1.4. HISTEXP Articles which were posted more than HISTEXP ago are con- sidered too old and are moved into the junk directory. This is because they are too old to be in the history file, so it is im- possible to tell if they really should be accepted or are end- lessly looping around the network. (This was theoretically pos- sible before this feature was added.) The articles are removed after DFLTEXP seconds, but a copy of their "Message-ID" is kept in the history file for HISTEXP seconds (the default is 4 weeks). August 13, 1987 9SMM:10-4 USENET Version B Installation 92.1.5. DFLTSUB The default subscription list. If a user does not specify any list of newsgroups, this will be used. Popular choices are all and general,all.general. 2.1.6. TMAIL This is the version of the Berkeley _M_a_i_l(1) program that has the -T option. If left undefined, the -M option to _r_e_a_d_n_e_w_s(1) will be disabled. 2.1.7. ADMSUB This newsgroup (or newsgroup list) will always be selected unless the user specifies a newsgroup list that doesn't include ADMSUB on the command line. That is, as long as the user doesn't use the -n flag to _r_e_a_d_n_e_w_s on the command line, ADMSUB will al- ways be selected. This is usually set to general. (The intent of this parameter is to have certain newsgroups which users are required to subscribe to. A typical site might require general.) 2.1.8. PAGE The default program to which articles should be piped for paging. This can be disabled or changed by the environment vari- able PAGER. If you have it, the Berkeley _m_o_r_e(1) command should be used, since the + option allows the headers to be skipped. 2.1.9. NOTIFY If defined, this character string will be used as a user name to send mail to in the event of certain control messages of interest. (Currently these are newgroup, rmgroup, sendsys, checkgroups, and senduuname.) As distributed, mail will be sent to user _u_s_e_n_e_t. It is recommended you create such a mailbox (have it forwarded to yourself) if possible, since this makes it easier for another site to contact the site administrator for your site. If you are unable to do this (_e._g., you are not the super user) you should change this name to yourself. Also, mes- sages about missing or extra newsgroups are mailed to this user by the checkgroups control message. 2.1.10. DFTXMIT This is the default command to use to transmit news if no explicit command is given in the fourth field of the _s_y_s file. It normally includes _u_u_x(1) with the -z option. You should in- stall this modification to UUCP at once; otherwise your users will start being bombarded with annoying _u_u_x completion messages. However, you can turn this off to get news installed. August 13, 1987 9USENET Version B Installation SMM:10-5 92.1.11. UXMIT This is the default command used if the U flag is present in the flags portion of a _s_y_s file line. In this case, the second "%s" refers to the name of a file in the news spool area, not a temporary file. It can usually only be used when local modifica- tions are made to the uucp system, such as the -c option to _u_u_x. 2.1.12. DFTEDITOR This is the full path name of the default editor to use dur- ing followups and replies. It should be set to the most popular text editor on your system. As distributed, _v_i(1) is used. 2.1.13. UUPROG If this is defined, it will be used as a command to run when the senduuname control message is sent around. Otherwise the command _u_u_n_a_m_e(1) will be run. Normally, this program should be placed in LIBDIR. 2.1.14. MANUALLY If this is defined, incoming rmgroup messages will not au- tomatically remove the group. News will instead mail a message to NOTIFY advising that the group should be removed. If you de- fine MANUALLY, you should have NOTIFY defined. MANUALLY is de- fined by default to protect you against accidental or malicious removal of an important newsgroup. 2.1.15. NONEWGROUPS If this is defined, incoming newgroup messages will not au- tomatically create the group. News will instead mail a message to NOTIFY advising that the group should be created. If you de- fine NONEWGROUPS, you should have NOTIFY defined. NONEWGROUPS is undefined by default to make it easier to automatically maintain the news system. 2.1.16. BATCH If set, this is the name of a program that will be used to unpack batched articles (those beginning with the character "#".) Batched articles normally are files reading #! rnews 1234 article containing 1234 characters #! rnews 4321 article containing 4321 characters . . . Batching is _s_t_r_o_n_g_l_y recommended for increased efficiency on both sides. August 13, 1987 9SMM:10-6 USENET Version B Installation 92.1.17. LOCALNAME Most systems have a full name database on line somewhere, showing for each user what their full name is. Most often this is in the gecos field of /_e_t_c/_p_a_s_s_w_d. If your system has such a database, LOCALNAME should be left undefined. If not, define LO- CALNAME, and articles posted will only receive full names from local user information specified in _N_A_M_E or $_H_O_M_E$_H_O_M_E/._n_a_m_e by the user. If you have a nonstandard gcos format (not _f_i_n_g_e_r(1) or RJE) it will be necessary to make local changes to _f_u_l_l_n_a_m_e._c as appropriate on your system. 2.1.18. INTERNET If your system has a mailer that understands ARPA Internet syntax addresses ("user@site.domain") turn this on, and replies will use the "From" or "Reply-To" headers. Otherwise, leave it disabled and replies will use the "Path" header. 2.1.19. MYDOMAIN When generating internet addresses, this domain will be ap- pended to the local site name to form mailing address domains. For example, on system _u_c_b_v_a_x with user _r_o_o_t, if MYDOMAIN is set to ".UUCP", addresses generated will read "root@ucbvax.UUCP". If MYDOMAIN is ".Berkeley.EDU", the address would be "root@ucbvax.Berkeley.EDU". If your site is in more than one domain, use your primary domain. The domain always begins with a period, unless the local site name contains the domain; in this case MYDOMAIN should be the null string. 2.1.20. CHEAP Do not _c_h_o_w_n(1) spool files to _n_e_w_s. This will cause the owner of the file to be the person that started the _i_n_e_w_s pro- cess. This is used for obscure accounting reasons on some sys- tems. 2.1.21. OLD Define this if any of your USENET neighbors run 2.9 or ear- lier versions of B news. It will cause all headers written to contain two extra lines, "Article-I.D." and "Posted", for down- ward compatibility. Once all your neighbors have converted, you can save disk space and transmission costs by turning this off. It is strongly encouraged that they convert. 2.10.3 is _m_u_c_h fas- ter than 2.9. The performance difference is dramatic. 2.1.22. UNAME Define this if the _u_n_a_m_e(2) system call is available local- ly, even though you are not a USG system. USG systems always have _u_n_a_m_e(2) available and ignore this setting. August 13, 1987 9USENET Version B Installation SMM:10-7 92.1.23. GHNAME Define this if the 4.[23] BSD _g_e_t_h_o_s_t_n_a_m_e(2) system call is available. If neither UNAME or GHNAME is defined, _i_n_e_w_s will determine the name of the local system by reading /_u_s_r/_i_n_c_l_u_d_e/_w_h_o_a_m_i._h. 2.1.24. UUNAME Define this if you keep your UUCP name in /_e_t_c/_u_u_c_p_n_a_m_e. 2.1.25. V7MAIL Define this if your system uses V7 mail conventions. The V7 mail convention is that a mailbox contains several messages con- catenated, each message beginning with a line reading "From _u_s_e_r _d_a_t_e" and ending in a blank line. If this is defined, articles saved will have these lines added so that mail can be used to look at saved news. 2.1.26. SORTACTIVE Define this if you want the news groups presented in the order of each person's ._n_e_w_s_r_c(5) instead of the active file. 2.1.27. ZAPNOTES Define this if you want old style notesfile id's in the body of the article to be converted into "Nf-Id" fields in the header. 2.1.28. DIGPAGE If this is defined, _v_n_e_w_s(1) will attempt to process the subarticles of a digest instead of treating the article as one big file. 2.1.29. DOXREFS Define this if you are using _r_n(1). _R_n uses this option to keep from showing the same article twice. 2.1.30. MULTICAST If your transport mechanism supports multi-casting of mes- sages, define this. Currently ACSNET is the only network that can handle this. 2.1.31. BSD4_2 Define this if you are running 4.2 or 4.3 BSD UNIX*. __________ *UNIX is a trademark of AT&T Bell Laboratories. August 13, 1987 9SMM:10-8 USENET Version B Installation 92.1.32. BSD4_1C Define this if you are running 4.1C BSD UNIX. 2.1.33. SENDMAIL Use this program instead of _r_e_c_m_a_i_l(8) for sending mail. 2.1.34. MMDF Use MMDF instead of _r_e_c_m_a_i_l for sending mail. 2.1.35. MYORG This should be set to the name of your organization. Please keep the name short, because it will be printed, along with the electronic address and full name of the author of each message. Forty characters is probably a good upper bound on the length. If the city and state or country of your organization are not ob- vious, please try to include them. If the organization name be- gins with a "/", it will be taken as the name of a file. The first line in that file will be used as the organization. This permits the same binary to be used on many different machines. A good file name would be /_u_s_r/_l_i_b/_n_e_w_s/_o_r_g_a_n_i_z_a_t_i_o_n. For example, an organization might read "AT&T Bell Labs, Murray Hill", "U.C. Berkeley", "MIT", or "Computer Corp. of America, Cambridge, Mass". 2.1.36. HIDDENNET If you want all your news to look like it came from a single machine instead of from every machine on your local network, de- fine HIDDENNET to be the name of the machine you wish to pretend to be. Make sure that you have you own machine defined as _M_E in the sysfile or you may get some unnecessary article retransmis- sion. 2.1.37. NICENESS If NICENESS is defined, _r_n_e_w_s does a _n_i_c_e(2) to priority NICENESS before processing news. 2.1.38. FASCIST If this is defined, _i_n_e_w_s checks to see if the posting user is allowed to post to the given newsgroup. If the username is not in the file LIBDIR/_a_u_t_h_o_r_i_z_e_d then the default newsgroup pat- tern in the symbol FASCIST is used. The format of the file _a_u_t_h_o_r_i_z_e_d is: user:allowed groups For example: August 13, 1987 9USENET Version B Installation SMM:10-9 9 root:net.all,mod.all naughty_person:junk,net.politics operator:!net.all,general,test,mod.unix An open environment could have FASCIST set to all and then individual entries could be made in the authorized file to prevent certain individuals from posting to such a wide area. Note that a distribution of all does _n_o_t mean to allow post- ings only to local groups - all includes all.all. Use all,!all.all to get that behavior 2.1.39. SMALL_ADDRESS_SPACE Define this if your machine has 16 bit (or smaller) pointers. If you are on a PDP-11*, this is automatically de- fined. 2.2. Makefile There are also a few parameters in the _M_a_k_e_f_i_l_e as well. These are: 2.2.1. OSTYPE This is the type of UNIX system you are using. It should be either v7 or USG. Any BSD system is v7. Any System 3 or System 5 system is USG. This is normally set by _l_o_c_a_l_i_z_e._s_h. 2.2.2. NEWSUSR This is the owner (user name) of _i_n_e_w_s. If you are a su- peruser, you should probably create a new user id (traditionally _n_e_w_s) and use this id. If you are not a superuser, you can use your own user id. If you are able to, you should create a mail alias _u_s_e_n_e_t and have mail to this alias forwarded to you. This will make it easier for other sites to find the right person in the presence of changing jobs and out of date or nonexistent directory pages. NEWSUSR and ROOTID do not need to represent the same user. 2.2.3. NEWSGRP This is the group (name) to which _i_n_e_w_s belongs. The same considerations as NEWSUSR apply. 2.2.4. SPOOLDIR This directory contains subdirectories in which news arti- cles will be stored. It is normally /_u_s_r/_s_p_o_o_l/_n_e_w_s. __________ *PDP-11 is a trademark of Digital Equipment Corpora- tion. August 13, 1987 9SMM:10-10 USENET Version B Installation 9 Briefly, for each newsgroup (say net.general) there will be a subdirectory /_u_s_r/_s_p_o_o_l/_n_e_w_s/_n_e_t/_g_e_n_e_r_a_l containing articles, whose file names are sequential numbers, _e._g., /_u_s_r/_s_p_o_o_l/_n_e_w_s/_n_e_t/_g_e_n_e_r_a_l/_1, etc. Each article file is in a mail-compatible format. It begins with a number of header lines, followed by a blank line, followed by the body of the article. The format has deliberately been chosen to be compatible with the ARPANET standard for mail docu- mented in RFC 822. You should place news in an area of the disk with enough free space to hold the news you intend to keep on line. The to- tal volume of news in net.all currently runs about 1 Mbyte per day. If you expire news after the default 2 weeks, you will need about 14 Mbytes of disk space (plus some extra as a safety margin and to allow for increased traffic in the future.) If you only receive some of the newsgroups, or expire news after a different interval, these figures can be adjusted accordingly. 2.2.5. BATCHDIR This directory will contain the list of articles to send to each system. It is normally /_u_s_r/_s_p_o_o_l/_b_a_t_c_h. 2.2.6. LIBDIR This directory will contain various system files. It is normally /_u_s_r/_l_i_b/_n_e_w_s. 2.2.7. BINDIR This is the directory in which _r_e_a_d_n_e_w_s, _p_o_s_t_n_e_w_s, _v_n_e_w_s, and _c_h_e_c_k_n_e_w_s(1) are to be installed. This is normally /_u_s_r/_b_i_n. If you decide to set BINDIR to a local binary directory, you should consider that the _r_n_e_w_s and _c_u_n_b_a_t_c_h commands must be in a directory that can be found by _u_u_x_q_t, which normally only searches /_b_i_n and /_u_s_r/_b_i_n. 2.2.8. UUXFLAGS These are the flags _u_u_x will be called with. 2.2.9. LNRNEWS This is the program used to link _r_n_e_w_s and _i_n_e_w_s. If you have symbolic links, you can replace the "ln" with "ln -s". 2.2.10. SCCSID If this is defined, sccs ids will be included in each file. If you are short on address space, don't define this. August 13, 1987 9USENET Version B Installation SMM:10-11 93. FILES This section lists the files in LIBDIR and comments briefly what they do. 3.1. active A list of active newsgroups. It is automatically updated as new newsgroups come in. The order here is the order news is ini- tially presented by _r_e_a_d_n_e_w_s, so you can edit this file to put important newsgroups first. If you have SORTACTIVE defined, after the first time the user invokes _r_e_a_d_n_e_w_s, it will be presented in the order of his ._n_e_w_s_r_c. Each line of the active file contains four fields, separated by a space: the newsgroup name, the highest local article number (for the most recently re- ceived article), the lowest local article number that has not yet expired, and a single character used to determine if the user can post to that newsgroup. If the character is "y" the user is per- mitted to post articles to that group. If the character is "n" the user is not permitted to post articles to that groups. (This field takes the place of the _n_g_f_i_l_e in earlier versions of news. Local article numbers begin at 1 and count sequentially within the newsgroup as articles are received. They do not usually correspond to local article numbers on other sites. The article numbers are always stored as a five digit number (with leading zeros) to allow updating of the file in place. The active file should contain all active net-wide active newsgroups (net.alland mod.all). It is important that they all be present, as they are used as a check for valid newsgroup names and invalid newsgroup names are removed from any articles pro- cessed by _i_n_e_w_s. You should use the _s_y_s file to keep out unwant- ed newsgroups. 3.2. aliases This file is used to map bad newsgroup names to the correct ones. (For example, net.unix.wizards is mapped into net.unix- wizards). Each line consists of two fields separated by a space. If the first field is found in the newsgroup list of the incoming article, it is changed to the second field. This change takes place in the article before it is passed on to other systems, not just locally. 3.3. batch This program reads a list of filenames of articles and out- puts the articles themselves. It is typically used by the shell script _s_e_n_d_b_a_t_c_h. 3.4. c7unbatch This is used to decompress news that has been encoded for transmission over a network that only supports 7-bit transfers August 13, 1987 9SMM:10-12 USENET Version B Installation 9(e.g X.25.) 3.5. caesar This is a program to do Caesar decoding of rotated text, on a line by line basis. The standard input is copied to the stan- dard output, rotating each line according to a static single letter frequency table. If an integer argument is given (_e._g., 13), every line is rotated by that argument, without regard to letter frequencies. This program is invoked by the D _r_e_a_d_n_e_w_s command. It is also used by _p_o_s_t_n_e_w_s with the "13" argument to encode selected material for posting. 3.6. checkgroups _C_h_e_c_k_g_r_o_u_p_s is a shell file to aid in automatically checking the accuracy of your active file. It is executed by the check- groups control message and mails a list of out of date newsgroups to the person defined by NOTIFY It also updates the _n_e_w_s_g_r_o_u_p_s file that is used by _p_o_s_t_n_e_w_s as a helpfile for newsgroup selec- tion. 3.7. compress This program does a modified Lempel-Ziv data compression. It is used by the compressed batching scheme. It averages 50% compression on a typical batch of news. 3.8. distributions This is a list of distributions that are valid for your site. Each line has two fields separated by the first space on the line. The first field is the name of the distribution (_e._g., usa, na, etc.). The second field is text describing the distri- bution. As distributed, this file is only correct for sites in the USA. You should examine this file and add or delete the ap- propriate distributions. 3.9. encode This program transforms an 8-bit binary file into a file suitable for sending over a link that only allows 7-bit charac- ters. It is used by sendbatch -c7. 3.10. errlog This file contains the "important" error messages found in the log file. These errors usually indicate that something was wrong with an article. This file should be watched closely. The _l_o_g file contains much more verbose information and it is often difficult to detect errors in it. August 13, 1987 9USENET Version B Installation SMM:10-13 93.11. expire This program expires old articles and archives them if ar- chiving is selected. It is typically run once a day from _c_r_o_n(8). 3.12. help This contains a list of commands printed when an illegal command is typed to _r_e_a_d_n_e_w_s. 3.13. history A list of every article that has come in to your system. It is used to reject articles that come in for the second time (presumably via a different path). This file will grow but is cleaned out by the _e_x_p_i_r_e(8) command. 3.14. history.d On USG systems, this directory contains 10 files (history.[0-9]) which are used as part of a simple hashing algo- rithm to speed up history searches. Since V7 systems have DBM, this is not used on V7 systems. 3.15. history.dir,history.pag These two files are used on V7 systems as a hashed version of _h_i_s_t_o_r_y, containing the message id's of all articles in histo- ry. They are only used if -DDBM and -ldbm appear in _M_a_k_e_f_i_l_e. 3.16. inews This is the program that actually sends and receives news. All other programs interface eventually with it. It is not in- tended to be used directly by a human, so it is no longer in /_u_s_r/_b_i_n. 3.17. log If present, a log of articles processed and error conditions is kept here. This file grows without limit unless cleaned out periodically. The _t_r_i_m_l_i_b script in _m_i_s_c can be invoked from _c_r_o_n daily or weekly to keep the log short. 3.18. moderators This file contains a list of the moderators and their mail- ing addresses for each moderated newsgroup. Each line consists of two fields. the first is the name of the moderated group. The second is the mailing address of the group's moderator. As distributed, they are almost certainly wrong. You will need to modify the paths so they work from your site. August 13, 1987 9SMM:10-14 USENET Version B Installation 93.19. newsgroups This file is displayed by _p_o_s_t_n_e_w_s when a user hits ? in response to its request for newsgroups. It is also used by _v_n_e_w_s when it displays the newsgroup name. It is updated automatically by the checkgroups control message. 3.20. notify If this file is present, its contents will be taken as the name of the user to notify in case of a problem. If the file is empty, nobody will be notified. (This overrides the NOTIFY op- tion in _d_e_f_s._h). Having a null file is useful if one person ad- ministers several systems and does not want multiple copies of control message notifications. 3.21. oactive, ohistory, ohistory.dir, ohistory.pag These are copies of the corresponding _a_c_t_i_v_e, _h_i_s_t_o_r_y, _h_i_s_t_o_r_y._d_i_r, and _h_i_s_t_o_r_y._p_a_g files before _e_x_p_i_r_e ran. They are kept in case something happens to the originals. 3.22. recmail This program can serve as a link between news and your local mailer. If you have _s_e_n_d_m_a_i_l(8), don't use _r_e_c_m_a_i_l. _S_e_n_d_m_a_i_l is much more useful. 3.23. recnews A program which allows you to send mail to get news posted. You usually need to run _s_e_n_d_m_a_i_l or _d_e_l_i_v_e_r_m_a_i_l(8) to be able to use this. 3.24. recording A list of newsgroup classes and filenames to display record- ings for. The recording feature is analogous to the recordings played in some areas when you dial directory assistance, trying to be annoying and make you think twice. Recordings on certain newsgroups are intended to remind the user of the rules for the newsgroup, or, in the case of a company worried about letting proprietary information out, reminding authors that anything they say is seen outside the company and so proprietary information should not be included. The file contains one line per recording. The line contains two fields, separated by a space. The first field is the news- group class (_e._g., net.all), the second field is the name of the file containing the recorded message. If the file name does not begin with a slash, it will be searched for in LIBDIR. Sample recording files can be found in the _m_i_s_c directory. August 13, 1987 9USENET Version B Installation SMM:10-15 93.25. rmgroup This shell file should be used to remove any groups that are no longer used. 3.26. sendbatch This shell file is used to send batched articles to other systems. It is typically run from _c_r_o_n. See the manual page for more details. 3.27. sendnews A program to send news internally from one computer to another. It is useful if you must use mail links to transmit ar- ticles. 3.28. seq This file contains the current sequence number for your sys- tem. It is used to generate unique article id's. 3.29. sys This file contains a list of all your neighbors, which news- groups they get, and how to send news to them. The format is do- cumented below. 3.30. unbatch This program is used to unbatch the incoming batched news and feed each article to _i_n_e_w_s. It's horrible and will go away in the future. 3.31. users A list of users that have read news on your system. 3.32. uurec A program to receive news sent by _s_e_n_d_n_e_w_s(8). 3.33. vnews.help This is the helpfile used by _v_n_e_w_s. 4. Setting Up Links There are two basic types of links for exchanging news: those that use mail and those that don't. The ones that use mail are more indirect, yet more versatile, while the ones that don't are simpler. The default method does not use mail, so that is discussed first. August 13, 1987 9SMM:10-16 USENET Version B Installation 94.1. Non-mail Links The basic theory behind a non-mail link is that the _r_n_e_w_s program is invoked on the remote system with the article being transmitted as the standard input. This is possible on several networks, but the most common implementation is via the UUCP net- work. Using the _u_u_x command, the command which is forked to the shell looks like: uux - -r -z remotesys!rnews < article This is the default transmission method. In order to set up such a link, obviously a UUCP link with the remote system must be in effect. In addition, _r_n_e_w_s must be available and executable by _u_u_x_q_t on the remote machine. In most cases, this means that _r_n_e_w_s must be in /_u_s_r/_b_i_n so _u_u_x can find it. Also, the list of allowed UUCP commands (in /_u_s_r/_s_r_c/_u_s_r._b_i_n/_u_u_c_p/_u_u_x_q_t._c or /_u_s_r/_l_i_b/_u_u_c_p/_L._c_m_d_s, depending on the version of UUCP) should be checked to make sure that _r_n_e_w_s is an allowed command. Other networks that allow remote execution include the BERK- NET, BLICN (_u_s_e_n_d(1)), many Ethernets, and the NSC hyperchannel (_n_u_s_e_n_d(1)). It is important, however, that a spooling mechanism be available. Otherwise, if system _A tries to send an article to system _B via a remote execution command, and _B is down, the arti- cle could be lost. Spooling arranges that the system will try again when _B comes back up. 4.2. Mail Links When using mail to transmit articles, two intermediary pro- grams are necessary. These are _s_e_n_d_n_e_w_s and _u_u_r_e_c(8). The idea is that when system _A wants to send an article to system _B, the _s_y_s file on system _A has an entry for system _B such as: /usr/lib/news/sendnews -a rnews@B which runs _s_e_n_d_n_e_w_s on the article. The -a option specifies that the mail should be formatted for the ARPANET. _S_e_n_d_n_e_w_s packages the article and mails it to "rnews@B". Somehow, the B system is expected to make sure that all mail to user "rnews" is fed as in- put to the program _u_u_r_e_c. This program unpackages it and invokes _r_n_e_w_s. The best way to get mail to "rnews" fed into _u_u_r_e_c is to use _s_e_n_d_m_a_i_l or _d_e_l_i_v_e_r_m_a_i_l, if you are on a system running them. Create an alias in /_u_s_r/_l_i_b/_a_l_i_a_s_e_s as follows: rnews: "|/usr/lib/news/uurec" and _s_e_n_d_m_a_i_l will handle it. If you do not have a facility for forwarding mail to a program, you can gimmick your mailer to watch for it (using _p_o_p_e_n(3S), this is easy) or, if you don't want to do any programming, you can have _c_r_o_n invoke _u_u_r_e_c every August 13, 1987 9USENET Version B Installation SMM:10-17 9hour with /_u_s_r/_s_p_o_o_l/_m_a_i_l/_r_n_e_w_s as standard input. This solution is messier because _u_u_r_e_c must potentially deal with multiple mes- sages, something that has never been tested. 5. Format of the _s_y_s_s_y_s file To set up a link to another site, edit the _s_y_s file in LIB- DIR. This file is similar to the _L._s_y_s file of UUCP. Each line contains four fields, separated by colons: (1) The system name of a site to which you forward news. Nor- mally all systems you have links to will be included. You should also have a line for your own system. If this field is _M_E, it will be used as if it were your local system name. If the system name is followed by a "/", the article will not be forwarded to this system if it has already passwd through any of the (comma separated) list of sites immedi- ately following the "/". For example, if the sysline was: yoursite/sitea,siteb,sitec:net,mod,na,usa,to.yoursite:: the incoming article would only be forwarded to _y_o_u_r_s_i_t_e if it had not already been to any of _s_i_t_e_a, _s_i_t_e_b, or _s_i_t_e_c. This is normally used to reduce the number of duplicate ar- ticles received at a site that has multiple main newsfeeds. (2) The newsgroups to be forwarded to them. This is a pattern of the same kind as a subscription list. Generally, you will list classes of newsgroups, that is, using all for everything. A typical forwarding list for a new site would be net,mod,na,usa,to._s_y_s_n_a_m_e where _s_y_s_n_a_m_e is the name of the remote system. (Of course, if you are not in the USA or North America, you would remove those distributions and replace them with the ones appropri- ate for you). In particular, you don't want to forward all since local newsgroups (those without dots) should not be sent. For the line describing your own system, this field describes the newsgroups your site will accept from remote sites. Thus, if another site insists on sending you a news- group you don't want, for example net.jokes, include !net.jokes here. (3) This field contains flags describing the connection. An A will indicate that the other site is running an A version of netnews. A B indicates a B version. Leaving it empty de- faults to B. If you are reading this document, you have a B version. Some existing sites run A versions. If you aren't sure, ask your contact at the other site, with whom you should be talking to set this up anyway. The F flag indi- cates that the fourth field is the name of a file. The full path name of a file containing the article in SPOOL will be August 13, 1987 9SMM:10-18 USENET Version B Installation 9 appended to this file. The L flag prevents transmission un- less the article was created on this site. If a number fol- lows the L (_e._g., L3), sites less than that number of hops away will be considered local. (It is recommended that you feed an L link to a backbone site, to ensure that your sub- missions will be more likely to get to the entire network, even in the event of a local problem. Please make sure that a mail link exists too, so you can get replies.) The N flag can also be included here, indicating that mail should be sent using the _i_h_a_v_e/_s_e_n_d_m_e protocol described below. The H flag can be used to interpolate the history file into the command. The S flags says to execute the transmission com- mand directly instead of forking a shell. The U field ar- ranges that the parameter to the optional "%s" in the com- mand field to be filled in with a permanent file name from SPOOL instead of a temporary customized file name. The M flag says to use multi-casting. Multi-casting is described in an appendix. (4) This field is the command to be run to send news to the re- mote site. The article will be on the standard input. Leaving this field blank means an ordinary UUCP link is be- ing used, that is, the command defaults to uux - -r -z sysname!rnews The - option tells _u_u_x to expect input from the standard in- put. The -z option is nonstandard - you should add it (see the _m_i_n_u_s._z* files in the uucp source directory.) It shuts off the annoying message you would otherwise get mailed to you telling you that your article was broadcast successful- ly. To avoid using the -z option, change the source or put the _u_u_x command in the fourth field. The -r option tells _u_u_x not to call the other system once the job is queued. This turns out to ease the load on the system, at the ex- pense of making news be transmitted a bit slower. The news will be sent when the next call is made; usually this means the next time mail is sent to or from your system. If this turns out to be unreasonably long, put a line in _c_r_o_n_t_a_b to run /usr/lib/uucp/uucico -r1 -ssystem every hour or so. Here is a sample _s_y_s file for a site _m_y_v_a_x with connections to _y_o_u_r_v_a_x where _m_y_v_a_x also passes news on to _d_o_w_n_s_t_r_e_a_m. We as- sume that _m_y_v_a_x and _d_o_w_n_s_t_r_e_a_m exchange a local newsgroup class lng.all as well as the network wide newsgroups. News to _d_o_w_n_- _s_t_r_e_a_m is batched. We also assume that _m_y_v_a_x and _y_o_u_r_v_a_x are in the USA, while _d_o_w_n_s_t_r_e_a_m is in Canada. August 13, 1987 9USENET Version B Installation SMM:10-19 9 myvax:net,mod,na,usa,lng,to:: yourvax:net,mod,na,usa,to.yourvax:: downstream:net,mod,na,lng,to.downstream:F:/usr/spool/batch/downstream 6. Posting Methods The basic method is _p_o_s_t_n_e_w_s. This program will prompt you for the title, newsgroups, and distribution, then place you in the editor. (The system default EDITOR is used unless the en- vironment variable EDITOR is set, overriding the system default.) The text should be typed after the blank line. The title and newsgroups are available for editing at the top of the buffer. Other header lines can be added, such as an expiration date or a distribution. When you write out the file and exit from the edi- tor, you will be prompted for what to do next. Your choices are: write the message to a file, send the message, list the message or edit it again. Another method is to use mail. This can only be done on systems that allow mail to a given name to be fed into an arbi- trary program as input. This is easily done with the Berkeley _d_e_l_i_v_e_r_m_a_i_l or _s_e_n_d_m_a_i_l program, and not with any other mailer the author is familiar with. (It may be possible to painfully set this up with MMDF, provided the newsgroup name is no more than 8 characters long.) To use mail, set up an alias such as the following: net.general: "|/usr/lib/news/recnews net.general" Whenever a user sends mail to net.general, this starts up the given shell command which calls recnews with one argument, the name of the newsgroup. You need to create one alias for each newsgroup, and to keep the list up to date as new newsgroups are created. _R_e_c_n_e_w_s(8) will in turn invoke _i_n_e_w_s. Note that there are problems with _r_e_c_n_e_w_s. There is no way to use it to post to multiple newsgroups without creating separate articles (something frowned upon because it forces peo- ple to read the same thing more than once.) Also, there is no way to make the recording feature (to remind people to not accidently divulge proprietary information) work when recnews is used. 7. Various considerations 7.1. Setuid bits The current intended state of affairs is that _i_n_e_w_s runs setuid to NEWSUSR. The _r_e_a_d_n_e_w_s program does not need to be setuid. This makes it possible to write your own interface to read news instead of using _r_e_a_d_n_e_w_s. (As distributed, _i_n_e_w_s is also setgid. I know of no good reason for this.) August 13, 1987 9SMM:10-20 USENET Version B Installation 97.2. Modes of Spool Directories All the files should be writable by NEWSUSR. However, due to a glitch, you will probably have to make the SPOOLDIR and its subdirectories mode 777. It could be 755 except for one problem. When a new newsgroup comes in, _i_n_e_w_s will attempt to _m_k_d_i_r(1) a new subdirectory of SPOOLDIR for the newsgroup. Since both _i_n_e_w_s and _m_k_d_i_r are setuid, _m_k_d_i_r will use the uid of the person who ran _i_n_e_w_s instead of NEWSUSR when checking for permissions. If the directory mode isn't 777 the check will fail. Here are several alternatives if you don't want a 777 directory around: 7.2.1. Fix Real Uid If _i_n_e_w_s is always run by _c_r_o_n or as _r_o_o_t, the real uid can be arranged to be _r_o_o_t or NEWSUSR. This is a poor solution since it makes the local creation of new newsgroups require super user permissions, and is a potential security hole. If this approach is taken, care must be taken to insure that the owner of the created directory is NEWSUSR. 7.2.2. Change the Kernel _I_n_e_w_s will do: setuid(geteuid()) (see _s_e_t_u_i_d(2) and _g_e_t_e_u_i_d(2)) before it forks the _m_k_d_i_r. If your system permits this call, there will be no problem. In particular, Berkeley 4.0 UNIX and later systems allow this. An alternative change to the kernel is to automatically stack uids: when a setuid program is run, set the new real uid to the old effective uid. 7.2.3. Groups You could have _i_n_e_w_s be setgid to NEWSGRP and all files writable by the group. This approach has been tested and the problem turns out to be that the _m_k_d_i_r command uses the _a_c_c_e_s_s(2) system call to check permissions. Since _a_c_c_e_s_s uses the real gid, you run into the same problem. 7.2.4. Another _M_k_d_i_r_M_k_d_i_r You could create a version of _m_k_d_i_r that does less checking and put it in a directory that can only be accessed by NEWSUSR (mode 700, owned by NEWSUSR). Have _i_n_e_w_s fork this _m_k_d_i_r. 7.3. Expiration dates To get articles to expire automatically, put a line in _c_r_o_n_- _t_a_b to run /usr/lib/news/expire every night. This command deletes all expired news. The -a _n_e_w_s_g_r_o_u_p_s option causes all expired news to be archived under /_u_s_r/_s_p_o_o_l/_o_l_d_n_e_w_s depending on which newsgroups are selected. August 13, 1987 9USENET Version B Installation SMM:10-21 9(See _e_x_p_i_r_e(8) for details.) Sometimes news is not expired when it should be. Be sure to check that _e_x_p_i_r_e has permissions to unlink files, and that it is properly setuid to NEWSUSR. You can manually invoke _e_x_p_i_r_e with the -v (verbose) option to find out what it's doing. Adding lev- els of verbosity (_e._g., -v6) will get more and more output. 7.4. Version to Version Version B will understand incoming news in either version A or B format, automatically (presuming OLD is defined in defs.h.) Version B will generate either format, depending on the flag in the third field of the _s_y_s line. Version A will not understand version B format. Thus, it is possible for two version B sites to communicate using version A format. This will work but is not a good idea, since the translation from B to A loses information (such as the expiration date) which will not be there when translated back to version B. News from versions A and 2.9 B do not conform to the USENET interchange standard. 2.10 B supports the standard and will com- municate with either A or 2.9 B news. A news is written (losing other header information) if A is in the flags for the system. If OLD is defined, 2.10 will write out headers with both standard ("Date" "Message-ID") and 2.9 ("Posted" "Article-I.D.") lines so that either B system will properly handle the article. Incoming news is recognized by the first letter (A for A news), or the lack of an "@" in the "From" line (2.9). Missing fields are con- structed as well as possible from the available information. 7.5. Presentation Order The order of the newsgroups listed in _L_I_B_D_I_R_L_I_B_D_I_R/_a_c_t_i_v_e is the order the newsgroups will be presented in initially. If SORTAC- TIVE is defined in _d_e_f_s._h, after the first time news will be presented in the order of the person's ._n_e_w_s_r_c. Initially this will be directory order, but you can edit important newsgroups like general to the top. A recommended order to maintain your active file in is this: net.announce.newusers general local.general net.announce local _n_e_w_s_g_r_o_u_p_s _i_n _a_l_p_h_a_b_e_t_i_c_a_l _o_r_d_e_r mod.all _n_e_w_s_g_r_o_u_p_s _i_n _a_l_p_h_a_b_e_t_i_c_a_l _o_r_d_e_r net.all _n_e_w_s_g_r_o_u_p_s _i_n _a_l_p_h_a_b_e_t_i_c_a_l _o_r_d_e_r test all.test to.all control junk August 13, 1987 9SMM:10-22 USENET Version B Installation 98. Control Messages Some news systems will send you articles that are not for human consumption. They are messages to your news system called _c_o_n_t_r_o_l _m_e_s_s_a_g_e_s. Such messages contain the "Control" header. Older systems use newsgroups matching all.all.ctl, and this will still work, although the "Control" header is preferred. Since the newsgroup name is used for distribution only, and is not checked to ensure it's in the active file, such newsgroup names can still be used. This makes it possible to post network wide control messages with net.msg.ctl (or restricted broadcast such as btl.msg.ctl) or messages for a particular system: to.ucbvax.ctl. Messages are canceled, however, with a "Control" line in a message to the same newsgroup(s) as the original mes- sage. A control message contains a command and zero or more argu- ments (much like a UNIX program). The subject of the article contains the command and arguments. The body of the article is usually ignored, although some messages can use it for additional text information. Control messages are not stored in SPOOL; rather, they are acted on and discarded at once. 8.1. ihave/sendme Two control messages are ihave and sendme. These messages allow two participating sites to set up a link so that one site will tell the other site it has a given article and wait for a request before it actually sends it. The normal case is to send an entire article to a system, which consults the history file to see if the article has already been seen, and then throws it away if it has been seen before. Note that, since most messages are short anyway, experience has indicated that for ordinary UUCP unbatched communication, all _i_h_a_v_e/_s_e_n_d_m_e does is triple the load and slow down forwarding. We hope future code will allow ihave's with multiple message id's in the body, and existing code in 2.10 understands such messages, but does not generate them. So we advise that you don't use _i_h_a_v_e/_s_e_n_d_m_e for now. Use of these control messages can cut down on this wasted transmission, but if you have a polled UUCP connection, they can slow down receipt of news due to polling delays. It is up to each connected pair of sites whether they want to use this proto- col. The choice is controlled by the N flag in the _s_y_s file. In the case of a leaf node (one with only one neighbor) there is no advantage to this protocol. Even if both sites are able to ini- tiate a connection (have dialers or the link is hardwired) the -r option on the _u_u_x can cause 2 hour or more delays in propagating news. Since this protocol can triple the number of messages gen- erated, you should carefully evaluate your situation when decid- ing whether to use it. If transmission time and phone bills dom- inate your costs, and you are sending news to several sites, and August 13, 1987 9USENET Version B Installation SMM:10-23 9large article bodies dominate the costs (rather than the headers and the time spent by UUCP negotiating transmission) it is prob- ably worthwhile to use _i_h_a_v_e/_s_e_n_d_m_e. If your costs are dominated by CPU load from UUCP, or if you send news to a site that cannot get it from anywhere else, you probably do not want to use this protocol. The decision can be made independently for each site in your _s_y_s file. This pair works as follows: Site _m_y_s_i_t_e receives article "<123@abc.UUCP>". It enters it locally and then broadcasts it to its neighbors. One of its neighbors is site _y_o_u_r_s_i_t_e which has the N flag in the _s_y_s file. So _m_y_s_i_t_e sends an article on news- group to._y_o_u_r_s_i_t_e_y_o_u_r_s_i_t_e.ctl with title "ihave <123@abc.UUCP> mysite". This control message has two arguments - the first ("<123@abc.UUCP>") is the article id of the article in question, the second ("mysite") is the name of the site sending the arti- cle. The name of the newsgroup and the _s_y_s file control transmission of the article. Normally the _s_y_s file will read something like yoursite:net.all,fa.all,to.yoursite:BN: which will cause an article on to.yoursite.ctl to be transmitted. _Y_o_u_r_s_i_t_e receives the message and looks to see if it has seen it before. If it has, it throws the message away and stops. If it hasn't, it sends a message on to._m_y_s_i_t_e_m_y_s_i_t_e.ctl with title "sendme <123@abc.UUCP> yoursite" which is transmitted to _m_y_s_i_t_e. (The two arguments to _s_e_n_d_m_e are the article id requested and the site to send it to.) Then _m_y_s_i_t_e gets this message and actually transmits the article to _y_o_u_r_s_i_t_e. 8.2. newgroup This message has one argument, the name of a newsgroup to be created. This allows special action to be taken locally when a new newsgroup is created. It is generated by the -C option to _i_n_e_w_s. By default, the newsgroup is added to the active file, and mail is sent to the local contact advising that this has hap- pened. The directory will be created when a message for that newsgroup arrives. See the routine "c_newgroup" in _c_o_n_t_r_o_l._c if you want something different to happen. (Note that, although the body of the message contains a brief description of the purpose of the group, this body is usually thrown away by existing software.) 8.3. rmgroup This message has one argument, the name of a newsgroup to be removed. It is used for network-wide cancellation of a news- group. If MANUALLY is not defined, it will remove the articles, directory, and active file line for the group. There is a shell script _r_m_g_r_o_u_p that does essentially the same thing as this mes- sage, but the shell script only removes the group locally. We August 13, 1987 9SMM:10-24 USENET Version B Installation 9recommend that you leave MANUALLY defined, and when you receive mail advising you of the demise of the newsgroup, you run _r_m_g_r_o_u_p by hand. This will prevent accidental or malicious removal of a good newsgroup. 8.4. cancel This message cancels a given article. It takes one argu- ment, the message id of the article to cancel. It should be broadcast to the same newsgroup as the original article. If the article to be canceled is not present, the control message will not be propagated to downstream sites. 8.5. sendsys The _s_y_s file is mailed to the originator of the message. There are no arguments. This is used for making maps. Since your _s_y_s file is public information, you should not remove or change this control message. 8.6. senduuname The _u_u_n_a_m_e program is run and the output is mailed to the originator of the message. There are no arguments. This is used for making UUCP maps. If you do not run UUCP or have sites in your _L._s_y_s which are a secret, you may wish to edit this. Note that only the output of _u_u_n_a_m_e is mailed, not the contents of _L._s_y_s (which news does not have access to anyway). If you do make a change, you should arrange that some mail still is sent out to the originator of the message, so he will know your site received it. See the code in routine "c_senduuname" in _c_o_n_t_r_o_l._c. 8.7. version The local version name/number of the netnews software is mailed back to the author of the control message. 8.8. checkgroups This control message is an attempt at semi-automatic mainte- nance of the list of active news groups. This control messages takes the body of the article and pipes it into _L_I_B_L_I_B/_c_h_e_c_k_g_r_o_u_p_s. As mentioned previously, _L_I_B_L_I_B/_c_h_e_c_k_g_r_o_u_p_s will update the news- groups file, add any missing newsgroups, and mail a message to NOTIFY about any old newsgroups that should be removed. It is expected that the person who maintains the list of active news- groups will broadcast this control message on a regular basis. 8.9. Other Messages Any unrecognized message will cause an error message to be mailed to the local site administrator. Additional messages may be defined as time goes on, such as messages to automatically August 13, 1987 9USENET Version B Installation SMM:10-25 9update directories or maps. You should be willing to go into the code (_c_o_n_t_r_o_l._c) and add messages as they become standardized. 9. Maintenance There are some things you should do periodically to keep your news system running smoothly. We hope to eventually auto- mate all or most of this, but right now some of it must be done by hand. The _h_i_s_t_o_r_y and _l_o_g files in your LIB directory will grow. You should make sure that they are cleaned up periodically. The _L_I_B_L_I_B/_e_x_p_i_r_e program will remove lines from history corresponding to deleted articles, but it is a good idea to check the file every few months to make sure it is not going wild. Be sure not to completely lose your history file when you clean it up, in case another neighbor tries to send you an article you recently got. (If you only get news from one site it is safe to clean it out completely.) The log file is not automatically cleaned out by any netnews software, and will grow quickly. The _m_i_s_c/_t_r_i_m_l_i_b script can be installed in _L_I_B_L_I_B/_t_r_i_m_l_i_b, and invoked weekly by _c_r_o_n. You should also clean out old newsgroups that are no longer active. To remove a newsgroup net.foo, you should run the shell script _r_m_g_r_o_u_p with net.foo as the argument. That is, /usr/lib/news/rmgroup net.foo Note that clearing up UUCP constipation is another thing you'll have to do if you have flaky hardware or phone lines. If you have more than one connection, chances are that UUCP will get clogged up when one of your neighbors goes down for more than a few hours. Various spooling schemes are being worked on to help make the news/uucp system more robust, but one thing you can and should do, if you find your /_u_s_r/_s_p_o_o_l/_u_u_c_p directory getting too big, is to install a subdirectory fix to UUCP. A quick and dirty version of this is available from Duke, which traps the file- oriented system calls at the assembly language level and maps, for example, D.fooA1234 into D.foo/D.fooA1234. Since the C. and D._l_o_c_a_l directories still get big, in practice this can still create some big directories, but the directories tend to be a factor of 5 smaller, resulting in a factor of 25 improvement to speed (since a directory traversal for all files is quadratic on UNIX). Right now, UUCP is the weak link in netnews distribution, and you should certainly keep an eye on it. 10. Creating New Newsgroups As system news administrator, you are able to create news- groups. To create a newsgroup, first make sure this is the right thing to do. Normally a suggestion is first posted to net.news.group,net.relatedgroup for a net newsgroup August 13, 1987 9SMM:10-26 USENET Version B Installation 9net.relatedgroup( should be the group which you are proposing to sub-divide. For instance, to propose creating net.tv.soaps, post the original article to net.tv,net.news.group). Followups are made to net.news.group _o_n_l_y. (You can force this by putting the line: Followup-To: net.news.group in the headers of your original posting). If it is established that there is general interest in such a group, and a name is agreed on, then someone creates it by typing the command inews -C _n_e_w_s_g_r_o_u_p This will create the active entry locally. The directory will be created automatically when the first article for that newsgroup is received. It will also prompt you for a paragraph describing the group and start up an _i_n_e_w_s to post a newgroup control mes- sage announcing the group. This control message will be sent out on net.msg.ctl and other sites may have configured their systems to do something with these messages. A human readable announce- ment is not made - you can post this to net.news.group if neces- sary. You must be the super user to use the -C option to _i_n_e_w_s. (That is, your uid must match ROOTID. It is recommended that you change ROOTID to your own uid so you don't have to _s_u to create newsgroups.) 11. Conversion from A to B If you are currently running version A on your system, note that B is incompatible with A. The files are stored in a dif- ferent format (headers have mail like field names now). The directory organization is different (each newsgroup has a sub- directory of its own, and the file names are numbers rather than _s_i_t_e._i_d pairs). There are no _b_i_t_m_a_p, _u_i_n_d_e_x, or _n_i_n_d_e_x files to be trashed (which articles have been read is stored in each users ._n_e_w_s_r_c file). The user interface is slightly different (news/_n_e_t_n_e_w_s(1) is now called _r_e_a_d_n_e_w_s, news is posted using _i_n_e_w_s, subscription is done by editing ._n_e_w_s_r_c, the sense of the -c option is reversed, news is presented in newsgroup order, the -a and -t options now probably need -x as well, and there are many minor changes). We decided not to provide a program to convert from version A to version B. Rather, the following strategy was adopted for conversion: (1) Install the new news in a different spool directory from the old one. For example, you can use /_u_s_r/_s_p_o_o_l/_n_e_w_n_e_w_s. You can change to the standard name later if you want. Get it to work for local messages. August 13, 1987 9USENET Version B Installation SMM:10-27 9(2) Post an article to newsgroup general with the old news an- nouncing the change. Make available documentation such as the accompanying paper _H_o_w _t_o _R_e_a_d _t_h_e _N_e_t_w_o_r_k _N_e_w_s to the users. This article will be the last one in the old news. (3) _C_h_m_o_d the old news directory to 555 to prevent any more news from being posted. (Actually, this will prevent the bitfile from being updated, so it may not be a good idea.) (4) Replace the old _r_n_e_w_s program with the new _r_n_e_w_s program. (5) Test it by having your neighbor send you a message. (6) Wait a reasonable period for everyone to have read the final article with the old news. Perhaps a few weeks is right. (7) Uninstall the old news. Users will have to invoke _r_e_a_d_n_e_w_s instead of _n_e_t_n_e_w_s to read news. Depending on your old method of posting, this could be changed too. (If you were using mail, it does not need to be changed.) They will also have to fix their subscriptions. In general, they can type netnews -s to see what they subscribe to on the old system, and then create a file in their home directory called ._n_e_w_s_r_c containing options -n _t_h_e_i_r _s_u_b_s_c_r_i_p_t_i_o_n The format of the subscription pattern matching is the same as in A except that ALL is replaced by all (change to lower case). Something along the lines of this could be used to automate this: (echo -n "options -s" ; netnews -s | sed s/ALL/all/) > .newsrc 12. Conversion from 2.9 to 2.10 Conversion from 2.9 to 2.10 is not nearly as involved as an A to B conversion. The user interface does not change much, and the user ._n_e_w_s_r_c files are not affected. However, it is recom- mended that you do the conversion during a time when no news is received, so that incoming news will not get lost. One way to ensure this is to make /_u_s_r/_b_i_n/_r_n_e_w_s be a shell script which saves the article in /usr/spool/innews/$$$$ ($$$$ is the process id of the particular shell and will be unique for each article). The first step to conversion is to customize the sources. In the past, you had to take a fresh distribution and edit the _d_e_f_s._h file and _M_a_k_e_f_i_l_e to suit local preferences. If you had many local changes, or didn't record the local changes, upgrading could be annoying. 2.10 provides a mechanism to automate these changes. Create a shell script in the src directory called August 13, 1987 9SMM:10-28 USENET Version B Installation 9_l_o_c_a_l_i_z_e._s_h. (You can use _l_o_c_a_l_i_z_e._s_a_m_p_l_e as a template.) This shell script should copy _d_e_f_s._d_i_s_t to _d_e_f_s._h, and copy either _M_a_k_e_f_i_l_e._v_7 or _M_a_k_e_f_i_l_e._u_s_g to _M_a_k_e_f_i_l_e. It should _c_h_m_o_d any files that need to be changed (often _M_a_k_e_f_i_l_e and _d_e_f_s._h) to a writable mode. Then it should invoke _e_d(1) on the files, making any necessary local changes. The next step is to compile the software, with _m_a_k_e(1). It may be necessary to update the _l_o_c_a_l_i_z_e._s_h file until you are sa- tisfied with the compilation. Note that after any change to the _M_a_k_e_f_i_l_e in _l_o_c_a_l_i_z_e._s_h, you should run _l_o_c_a_l_i_z_e._s_h by hand. Otherwise, although make will run it for you, it will then con- tinue to do the make with the old _M_a_k_e_f_i_l_e. When the software is compiled, you should run the _c_v_t._a_c_t_i_v_e._s_h shell script, with the _l_i_b and _s_p_o_o_l directories as parameters. This will create a new active file in _L_I_B_L_I_B/_a_c_t_i_v_e. Then run _c_v_t._l_i_n_k_s._s_h with the _l_i_b and _s_p_o_o_l directories as parameters. Then run _c_v_t._n_a_m_e_s._s_h with the _l_i_b and _s_p_o_o_l direc- tories as parameters. Old news will be linked into the new hierarchy while leaving links in the old hierarchy. If you were using the default library and spool directories, you would do the following: sh cvt.active.sh /usr/lib/news /usr/spool/news sh cvt.links.sh /usr/lib/news /usr/spool/news sh cvt.names.sh /usr/lib/news /usr/spool/news The next step is to back up the old binaries: mv /usr/bin/rnews /usr/bin/ornews ... and to install 2.10 with make install Once it is installed, any incoming news will be placed into the new hierarchy but not the old one. The critical time window is between running the three shell files and installing the new software - any incoming news between these two points will appear in only the old hierarchy and be lost to the new software. If any significant time elapses here, you should divert _r_n_e_w_s into a separate spool directory as described above. It is crucial that you run _e_x_p_i_r_e before any new news ar- rives. _E_x_p_i_r_e will update several key files automatically. Finally, test things by posting articles to to._n_e_i_g_h_b_o_r_n_e_i_g_h_b_o_r newsgroups and watching some incoming news, and announce the change to your users. When you are satisfied that the conversion was successful, run the shell file _c_v_t._c_l_e_a_n._s_h which will remove the old 2.9 August 13, 1987 9USENET Version B Installation SMM:10-29 9news hierarchy. August 13, 1987 9SMM:10-30 USENET Version B Installation 9Appendix A: Setting up a Compressed, Batched Newsfeed First, BATCH must have been #_d_e_f_i_n_e'd when you built the news system. To check, look in the file _d_e_f_s._h in the news source directory. BATCH should be defined as a program name (by default, _u_n_b_a_t_c_h). If it's undefined or commented out, define it, re-make the news system, and install the new software. You'll also need a working _c_o_m_p_r_e_s_s program. Use the one shipped with this news distribution, which is based on version 4.0. Your news neighbors should be running a compatible version of compress. Versions 3.0 and 4.0 are compatible with each oth- er, but both are incompatible with versions 2.0 and before. Update your _s_y_s file. First, add the F flag to the other news system's line. For instance, if your compressed-and-batched news feed is named _f_r_o_b_o_z_z, and its _s_y_s file entry looks like: frobozz:net,mod,na,usa,ca,to.frobozz:: then add the F flag as the third (colon-separated) field: frobozz:net,mod,na,usa,ca,to.frobozz:F: Now the pathnames of ar- ticles to be sent will be stashed in a file. This file is named in the fourth field of the _s_y_s entry; add it now. Use an entry of the form _B_A_T_C_H_D_I_R_B_A_T_C_H_D_I_R/_s_y_s_t_e_m, where _B_A_T_C_H_D_I_R_B_A_T_C_H_D_I_R is usually /_u_s_r/_s_p_o_o_l/_b_a_t_c_h (the actual value is defined in the news _M_a_k_e_f_i_l_e), and _s_y_s_t_e_m is the name of the remote system, in this example _f_r_o_b_o_z_z. A name of that form is necessary: the _s_e_n_d_b_a_t_c_h script, which sends the batched news, looks for a file name of this form to decide if there's news for the remote system. Your completed _s_y_s file line should look something like: frobozz:net,mod,na,usa,ca,to.frobozz:F:/usr/spool/batch/frobozz In /_u_s_r/_l_i_b/_c_r_o_n_t_a_b, find or create at least two news lines: one that runs nightly, and one that runs every hour or so. The nightly-run script should run _e_x_p_i_r_e, trim log files, and perhaps compile weekly statistics that you post to a local-area newsgroup one day a week. The hourly-run script should complete the transmitting task with a line like: sendbatch -c frobozz Make sure the script knows how to get to the directory in which _s_e_n_d_b_a_t_c_h lives. You can either mention the directory in the script's PATH-setting line, or replace _s_e_n_d_b_a_t_c_h with its full pathname. _S_e_n_d_b_a_t_c_h reads the files mentioned in /_u_s_r/_s_p_o_o_l/_b_a_t_c_h/_f_r_o_b_o_z_z, batches them, optionally compresses them, sends them to the remote system, and arranges for remote processing. This remote processing is directed by another file in BATCH- DIR. Make a file with a name of the form _B_A_T_C_H_D_I_R_B_A_T_C_H_D_I_R/_s_y_s_t_e_m.cmd (for this example, /_u_s_r/_s_p_o_o_l/_b_a_t_c_h/_f_r_o_b_o_z_z._c_m_d). Put a line in it specifying the command that the remote system should execute August 13, 1987 9USENET Version B Installation SMM:10-31 9to unpack the news batches that your system will send. An exam- ple _f_r_o_b_o_z_z._c_m_d would be: uux - -r -z -n -gd frobozz!rnews Now your system will transmit compressed batches. The re- ceiving side of the business is handled largely by a program called _r_n_e_w_s, which will call other programs in LIBDIR to do ad- ditional processing on the incoming batches. Make sure there is an executable file called _r_n_e_w_s in the BINDIR directory (check the _M_a_k_e_f_i_l_e for its actual location). It must be reachable by UUCP or by whatever transport you'll use to transfer the netnews. If you defined BINDIR as /_u_s_r/_b_i_n, you should have no problems because _u_u_x_q_t can already get there. If you defined it as a different directory, you may have to teach _u_u_x_q_t to look in that directory; accomplishing this varies from system to system. On 4.2BSD, add the directory to the PATH= line of your UUCP _L._c_m_d_s file. On System V, on the _r_n_e_w_s line of your _L._c_m_d_s file, add a comma followed by the remote system's name on that line. If yours is in /_u_s_r/_b_i_n/_n_e_w_s/_r_n_e_w_s, your _L._c_m_d_s file will look like: [For 4.2BSD] PATH=/bin:/usr/bin:/usr/bin/news rnews [For System V] /usr/bin/news/rnews,frobozz Other systems have a similar file in the /_u_s_r/_l_i_b/_u_u_c_p directory by which you can specify added programs and paths different from the defaults. HP-UX, for example, has a /_u_s_r/_l_i_b/_u_u_c_p/_C_O_M_M_A_N_D_S file which expands _u_u_x_q_t's horizons. In more restrictive cases, paths are compiled into _u_u_x_q_t. If you can't modify any UUCP files, just put _r_n_e_w_s in /_u_s_r/_b_i_n. You must also have a _c_u_n_b_a_t_c_h in LIBDIR (wherever your _M_a_k_e_f_i_l_e defines it), because _r_n_e_w_s will eventually try to exec that copy. Tell the person at the other end of your newsfeed to use _s_e_n_d_b_a_t_c_h -_c to send you news. Once that's in place, watch your UUCP _L_O_G_F_I_L_E and your news _l_o_g and _e_r_r_l_o_g files to ensure that news is being correctly received and unpacked on your system. Older compressed batching systems will try to exec _c_u_n_b_a_t_c_h instead of _r_n_e_w_s. If you are still communicating with these, leave _c_u_n_b_a_t_c_h in BINDIR until they have upgraded their software. August 13, 1987 9SMM:10-32 USENET Version B Installation 9Appendix B: MULTICAST If this is defined (in _d_e_f_s._h) then two new flag characters become defined in the _s_y_s file. The first, and most important, of these is the M flag. If the M flag is set on some line in the _s_y_s file, then the fourth field (transfer command) is redefined to become a _m_u_l_t_i_- _c_a_s_t name. That is simply another system name, expected to be found in the first field of some line in the _s_y_s file (textually following the line containing the M flag). When a news item is being retransmitted, if it should (ac- cording to the subscription list) be sent to a system that has the M flag set, then instead of a command being run immediately to transmit the news, the news system remembers the system name, along with the multicast name (fourth field). Eventually the multicast system name is found in first field of a sys file line. If its subscription list allows transmission of this news item, then its command will be executed. This com- mand may have up to two "%s" substitutions in it. The second of those is replaced by the name of a file containing the news item (used with the U flag). The first is subjected to rather special treatment. The whole "word" (delimited by white space) contain- ing that "%s" is duplicated as many times as there were systems with the M flag set that referenced this multicast name (which might be 0 times, causing that "word" to be omitted). In each of these duplicates, the "%s" is replaced by the name of a system. Note the multicast system name itself is not included in this process. Then the command is executed as usual. The second flag available if the news system is built with MULTICAST defined is O. If this flag is set, then the sys file line will be ignored unless the system name is a multicast name from some earlier line with the M flag, and the news item is to be sent to that (earlier) system. This allows the subscription list for the multicast system name (which is likely to be a fake system name, invented just for this purpose) to be given a very wide subscription list (like all) without any unusual effects. Here is an example. Assume that you wish to forward net.unix to four people by mail. You could do this as ... fred:net.unix::mail fred harry:net.unix::mail harry jane:net.unix::mail jane tony:net.unix::mail tony however this causes the mail program to be started 4 times, once for each recipient. On some systems starting the mail program is a very expensive operation. If MULTICAST is defined, an alterna- tive method is August 13, 1987 9USENET Version B Installation SMM:10-33 9 fred:net.unix:M:tony harry:net.unix:M:tony jane:net.unix:M:tony tony:net.unix::mail tony %s This would cause just one command to be run: "mail tony fred har- ry jane". Note that "tony" must still be explicitly included in the argument list to the mail command; the "%s" does not expand to include the multicast "system name" itself. A more useful way of doing this, which does not assume that all the mail readers will want to read the same newsgroups is as follows. fred:net.unix:M:Mail harry:net.physics,net.astro:M:Mail jane:net.unix-wizards,net.women:M:Mail tony:net.unix,net.unix-wizards,net.jokes:M:Mail Mail:all:O:mail %s Now, if a news item in group net.unix was received, the com- mand mail fred tony would be executed. If the news were in both net.unix and net.unix-wizards then the command would be mail fred jane tony If a newsitem in net.med (which no-one gets by mail) ar- rives, then the "Mail" line will be ignored, because of the O flag. "Mail" is a fake system invented just so its "transfer command" can be used to send news to the other recipients. The same kind of technique can be used for normal transfer of news to other systems if your transport network supports a fa- cility to send to many other systems in one command. (That is, if it has a multicast facility.) SunIII (the network used in Aus- tralia) has this ability, so a typical Australian _s_y_s file looks like emuvax:aus,net,mod,fa:M:FakeName kremlin:aus,net,mod:M:FakeName kanga:aus,net,!net.all,net.unix:M:FakeName FakeName:all:OUS:/bin/sendfile -NRSareporter -d%s -x%s A news item in aus.general causes the following command /bin/sendfile -NRSareporter -demuvax -dkremlin -dkanga -x/usr/spool/... to be executed. Just one command is run to send the news to three remote systems. August 13, 1987 9SMM:10-34 USENET Version B Installation 9 If a multicast system has the F flag set, then the name of a file containing the news is appended to the file whose name is in the fourth field, as usual. But on the same line, separated by spaces, will be appended the names of all the systems that refer- enced this multicast system. For example, if the Australian site wanted to batch news, instead of sending it directly, it would simply change the last line of its _s_y_s file to FakeName:all:F:/usr/spool/batched/allsites Then a news item in net.jobs would cause the following line to be appended to /_u_s_r/_s_p_o_o_l/_b_a_t_c_h_e_d/_a_l_l_s_i_t_e_s /usr/spool/news/net/jobs/5542 emuvax kremlin This can then be processed later, in something like the nor- mal manner. (Unfortunately no commands to do this processing are yet available). Caution: when MULTICAST is defined, the first "%s" in all transfer commands is used for multicast, regardless of whether or not the system name is ever used as the last field of some line with the M flag set. To use the U flag in such a case, a dummy "%s" should be used, it will simply be omitted from the command that is executed. As an example, if a _s_y_s file line were foovax:net,na,usa:U:uux - foovax!foonews <%s without MULTICAST, it would need to be changed to foovax:net,na,usa:U:uux - foovax!foonews %s <%s if MULTICAST were defined. Additional caution: The numbers of system names that may be used in this way are quite severly restricted. Typically there may only be about 10 multicast system names, and each of those is restricted to sending to no more than about 20 systems. These limits are dynamic (that is, the numbers counted are the number of multicast systems receiving any single news item, and the number of systems that each of those will actually cause this particular news item to be sent to). These limits should easily suffice for real news sending to remote systems; however they are not likely to suffice if you want to mail news to everyone on your host. August 13, 1987