From tal@Warren.MENTORG.COM Thu Aug 12 18:15:46 1993 From: tal@Warren.MENTORG.COM (Tom Limoncelli) Newsgroups: news.software.nntp,news.software.b,news.answers Subject: INN FAQ Part 1/3: General Information, how to compile, how to operate Supersedes: Followup-To: news.software.nntp Date: 5 Aug 1993 04:00:07 -0000 Organization: Mentor Graphics - IC Group, Warren, NJ, USA Distribution: world Reply-To: Tom_Limoncelli@Warren.MENTORG.COM (Tom Limoncelli) NNTP-Posting-Host: sdl.warren.mentorg.com Summary: Part 1: Common questions about INN itself, and some advice when compiling it and installing it. Part 3: Norman's quick guide to getting started (assumes SunOS and other things), and misc. other things.) Part 2: This assumes that you've successfully compiled the software. This is a tutorial that takes you through configuration and setting up feeds. It ends with common questions about INN configuration. Posted-By: auto-faq 2.4 Archive-name: inn-faq/part1 Last Changed: $Id: FAQ-inn.1,v 1.37 1993/07/27 18:19:52 tal Exp $ Table Of Contents: ------------------ QUESTIONS FROM PEOPLE THAT DON'T (YET) RUN INN: Where can I get it? What is INN? What machines does it run on? Can I run C News with INN? Can I run NNTP with INN? Can I run the reference implementation (NNTP1.5) with INN? Can I run INN on my UUCP-only machine? Suppose I have a 286 machine? Does INN implement trn's XTHREADS, etc? Does INN do UUCP batching like C News? COMMON DEBUGGING/SETUP QUESTIONS: Help!? How do I configure this beast Why does innd just exit right away with no message? I'm getting news but postings aren't going out. How come my host name comes out twice in the Path line? Why does my innd often die with the message "Can't sync history, 7-bit encoded batches are not correctly processed. Why is this? SPECIFIC OPERATING SYSTEM BUILD ADVICE: SunOS 4.1.2 Ultrix tip 1 (mmap) Ultrix tip 2 (syslog) HP-UX SVR4, Solaris 2.x, and SCO ODT 3.0 OPERATIONAL QUESTIONS: Safe way to edit the "active" file? Expire had problems last night, and while I fixed the problem it still won't run. Why am I getting alt.sex.pictures even though I have "ME:!alt.sex.pictures" in my newsfeeds file? Why doesn't this newsfeeds entry do what I want? Why am I forwarded cancel messages for articles in comp.foo when I explicitly have !comp.foo in the newsfeeds entry? What's the best way to upgrade to a new version of INN? ADVANCED QUESTIONS: What do I do if /var/spool/news is set up over many partitions? How do I talk to innd from C or Perl? How do I set up a delayed IHAVE/SENDME over NNTP? (The FAQ was written by Rich $alz and is now maintained by Tom Limoncelli .) ====================================================================== ADVANCED QUESTIONS: QUESTIONS FROM PEOPLE THAT DON'T (YET) RUN INN ====================================================================== ------------------------------ Subject: Where can I get it? The official archive site is ftp.uu.net in the directory networking/news/nntp/inn. Archie current lists over 30 archive sites; three other international sites are grasp1.univ-lyon1.fr in pub/unix/news/inn, munnari.oz.au in pub/news/inn, and src.doc.ic.ac.uk in computing/usenet/software/transport This part of the FAQ can be gotten by sending email to majordomo@warren.mentorg.com with "sendme faq-inn-1" in the body of the message. (the Subject: of the message will be ignored.) (Substitute faq-inn-2 or faq-inn-3 to get those parts). For example: % mail majordomo@warren.mentorg.com Subject: ignored sendme faq-inn-1 sendme faq-inn-2 sendme faq-inn-3 ^D Or % echo sendme faq-inn-1 | mail majordomo@warren.mentorg.com % echo sendme faq-inn-2 | mail majordomo@warren.mentorg.com % echo sendme faq-inn-3 | mail majordomo@warren.mentorg.com ------------------------------ Subject: What is INN? InterNetNews is a complete Usenet system. The cornerstone of the package is innd, an NNTP server that multiplexes all I/O. Think of it as an nntpd merged with the B News inews, or as a C News relaynews that reads multiple NNTP streams. Newsreading is handled by a separate server, nnrpd, that is spawned for each client. Both innd and nnrpd have some slight variances >from the NNTP protocol (although in normal use you will never notice); see the manpages. INN separates hosts that feed you news from those that have users reading news. If you need to support a mixed environment you will have to do some extra work; the installation manual gives some hints. ------------------------------ Subject: What machines does it run on? If you have socket() and select() then INN will probably run on your machine. In addition to the common platforms found around the Internet (Sun and Ultrix, for example), INN runs on AIX, A/UX, NeXT, and a host of others. ------------------------------ Subject: Can I run C News with INN? No. INN handles all article reception, filing, forwarding, and expiration. You will most likely get a corrupted database if you try to run INN with any other news system. For testing, you can probably shut down your old system, bring up INN, and then reverse the process. (INN uses the C News history file and DBZ database, so if you don't run C News you will have to do some fiddling around with those files.) ------------------------------ Subject: Can I run NNTP with INN? There's a confusion here. NNTP is a protocol, defined in RFC 977. There is also an implementation of the protocol, NNTP1.5, that many people call NNTP. When there was only one implementation of the protocol, that was okay, but now that there are other implementations (for example, INN) it is getting confusing. It would be as if "sendmail" were named "smtp." Please try to be clear -- do you mean the NNTP protocol, or the NNTP reference implementation currently maintained by Stan Barber? ------------------------------ Subject: Can I run the reference implementation (NNTP1.5) with INN? The quick answer is no. INN listens on the NNTP port and handles all incoming traffic. It receives articles, files them, and arranges for them to be forwarded to your peers. If a site connects that is not listed as a peer (e.g., a local workstation that does newsreading) then the INN server hands the connection off to another program. By default, this is nnrpd, which implements the NNTP protocol for newsreaders (for example, it includes the POST command but not the IHAVE command). You can run the reference implementation server instead of nnrpd if you want. Doing this can be useful if you have clients that want to do both reading and article transfer. ------------------------------ Subject: Can I run INN on my UUCP-only machine? Sure. While not designed for this, several people are running INN on machines that do not have IP-connectivity (such as UUCP-only hosts) and are quite happy with it. You might want to give it a try, especially if you think you will be joining the Internet some day. ------------------------------ Subject: Suppose I have a 286 machine? Won't work. INN is designed to be a memory hog; a server that has been up for a few days while will have a working set size of a few to several megabytes, although not all of it will be resident. For example, the server keeps the active file and list of who gets what in memory, as well as all articles that it is receiving. Unless you can do things like "malloc(64 * 1024)" without pain, INN won't work on your machine. ------------------------------ Subject: Does INN implement trn's XTHREADS, tin's commands, etc? The XTHREAD command has code but it is not supported; look at $inn/nnrpd/nnrpd.h. This code will probably vanish after 1.4. Tin commands are not supported. Instead, INN supports Geoff Collyer's news overview database, nov (world.std.com, src/news/nov.dist.tar.Z). It does not include the client package; it just updates and maintains the overview databases. For info on how to configure INN to use NOV, read Part 2/2 of this FAQ. ------------------------------ Subject: Does INN do UUCP batching like C News? Not as part of the standard distribution. The batching system right now is better than B News, but Rich has said he will be working on improving that part of INN in a future release. Christophe Wolfhugel has written a package that is very much like the C News batching system, however. You can find it on grasp1.univ-lyon1.fr in the pub/unix/news/inn/contrib directory. ====================================================================== COMMON DEBUGGING/SETUP QUESTIONS ====================================================================== ------------------------------ Subject: Help!? How do I configure this beast Tom Limoncelli wrote a very good tutorial document. It is not part of the INN release, but many FTP sites keep a copy in the same directory. Look for a file named inn-tutorial. It is part 2 of this FAQ. ------------------------------ Subject: Why does innd just exit right away with no message? First, fix your syslog: innd always logs a message before it exits. (The INN distribution includes a version of the current UCB syslog, along with instructions on how to install it. Ultrix systems might want to look at the syslog that is available on gatekeeper.dec.com) Second, the most common cause of this is that you do not have a history file (or no history database). You will see a message like this: ME cant dbminit /usr/local/news/history No such file or directory This means that you do not have a history database. You might want to run the BUILD script in your INN source tree or read about makehistory in doc/news-recovery.8; if you do the latter, make sure to rename the database files. This FAQ covers general questions about INN and questions about how to compile it. For information on configuration and debugging one's configuration, see the "INN Configuration FAQ". ------------------------------ Subject: I'm getting news but postings aren't going out. You might find it helpful to read "Appendix IV: First-time Usenet or NNTP Installation" in the Install manual. In order for postings to go out, you must have a newsfeeds entry that names a site to receive them. In the standard case you want to record the article filename and Message-ID in a batchfile for the site: myfeed/myfeed.dom.main:*,!foo.*\ Tf,Wnm: You now want to run send-nntp or nntpsend from cron so that the file is flushed and that innxmit is called to read the file and send the articles out. See the manual pages. ------------------------------ Subject: How come my host name comes out twice in the Path line? The INN server puts its name in the Path line of every article that it receives. Obviously, it has to do this. The default configuration has inews put the local host in the Path header. If nobody posts on the server and you use fully-qualified domain names on your workstations, then everything works the right way. (If `hostname` doesn't give an FQDN on your machine, you can work-around this by setting the "domain" value in inn.conf; remember that innd never re-reads inn.conf. You must "ctlinnd shutdown x" and then re-start the server). Many people don't want the client machines to put their name in the Path header. To do this, set INEWS_PATH to DONT. Finally, let me say that it is probably a mistake to have a "pathhost" line on any machine other than your server if you set INEWS_PATH to DO. If you doubt this, please trace the article flow for yourself. If you are curious about the effect of INEWS_PATH, read the nroff source -- not the formatted output -- of doc/inews.1 ------------------------------ Subject: Why does my innd often die with the message "Can't sync history, interrupted system call" Are you running SunOS? If so, the answer is that Sun broke the write system call but a patch is available. Any write could fail in this way, it is just more likely to happen when writing large files and in-core DBZ writes the history file out in one chunk. See the "Known Problems" section of the installation manual. To the best of my knowledge, nobody has seen this problem on any other system. ------------------------------ Subject: How do I create all those directories in the newsspool? Q: For example, if you receive comp.sys.amiga.applications, do you have to mkdir /var/spool/news/comp/sys/amiga/applications? A: Nope. innd creates the directory for you the first time you receive an article for that newsgroup. ------------------------------ Subject: 7-bit encoded batches are not correctly processed. Why is this? Chris Schmidt replies: The decode program that comes with INN up to version 1.3 is broken. Because of that the last article in a 7bit encoded batch will not correctly be decoded (the last characters are screwed up). This is fixed in INN 1.4. ====================================================================== SPECIFIC OPERATING SYSTEM BUILD ADVICE ====================================================================== ------------------------------ Subject: SunOS 4.1.2 SunOS 4.1.2 (but not 4.1.1 or 4.1.3) broke the write system call but a patch is available. Any write could fail "half way", it is just more likely to happen when writing large files and in-core DBZ writes the history file out in one chunk. See the "Known Problems" section of the installation manual. ------------------------------ Subject: Ultrix tip 1 (mmap) Ultrix has a "mmap()" function, but it doesn't do the same thing as the SunOS/BSD mmap() function. Therefore, do not configure INN to use mmap() on a Ultrix system. INN wants to find a mmap() function that is like the one on SunOS/BSD systems. ------------------------------ Subject: Ultrix tip 2 (syslog) The syslog on Ultrix sucks rotten eggs and Digital refuses to fix it. (source: everyone that uses Ultrix and has ever used other systems) Luckily, you can replace it with the routine that comes with INN. However, some people have had better luck installing the syslog that can be had on gatekeeper.dec.com:/pub/DEC/jtkohl-syslog-complete.tar.Z It still works with old clients but does new-style syslogging, too. Works great for me so far. (this information from: nelson@reed.edu (Nelson Minar)). The syslog that is shipped with INN works pretty well but there have been some claims that some old clients don't like it. ------------------------------ Subject: HP-UX Q. I am running inn on an HP machine. Inn won't start up automatically :-( I can start it manually. There is no problem with news or inn once it is started. A. Try adding a "sleep 10" to the bottom of /etc/rc.news, or in /etc/rc, right after /etc/rc.news is invoked. On some machines, including HP, the shell started by "#!/bin/sh" when /etc/rc is executed will exit before innd has disassociated itself from that shell. This causes innd to exit, sometimes without printing an error message. (source: pjoslin@mbvlab.wpafb.af.mil (Paul Joslin )) This problem goes away if you set HAVE_SETSID to "DO". Something to do with Posix Session Leader concepts. Ick. (source: scotty@piranha.cs.uoguelph.ca (Steve Howie)) ------------------------------ Subject: System V based Unixes (SVR4, Solaris 2.x, SCO ODT 3.0, A/UX, ...) If you are running any non-BSD (i.e. System V based) Unix you MUST have the following option set: ## How should close-on-exec be done? Pick IOCTL or FCNTL. #### =()@>()= CLX_STYLE FCNTL This includes SVR4, Solaris 2.x, A/UX and SCO ODT 3.0. This is clearly stated in the Install.ms file and repeated here since so many people post to news.software.nntp after messing it up. If it isn't FCNTL, you'll get tons of overchan processes hanging around. (source: Philip Gladstone ) In SCO ODT 3.0 and MOST systems, innd will link and run if you use IOCTL but eventually will stop answering incoming calls. Don't be fooled. Just because it compiles doesn't mean it's going to work. Under Solaris 2.x (SunOS 5.x) you must add the following at the beginning of each file using gethostbyname(): #define gethostbyname __switch_gethostbyname ------------------------------ Subject: Mac running A/UX 3.0 Use the INN malloc. ====================================================================== OPERATIONAL QUESTIONS ====================================================================== ------------------------------ Subject: Safe way to edit the "active" file? The following sequence is the shortest: ctlinnd pause "edit active" [do something to the active file] ctlinnd reload active "edit active" ctlinnd go "edit active" Simple! No need to "flush" after step one. ------------------------------ Subject: Expire had problems last night, and while I fixed the problem, it still won't run. When expire starts up it "reserves" the server so that nobody else can pause or throttle it. This prevents anyone else from coming in and modifying the history database. If expire bails out because of a bad error (e.g., your expire.ctl has syntax errors) it leaves the server reserved so that no maintenance will be done until a good expire run has occurred. To unblock the server, use the ctlinnd "reserve" command with an empty string argument. ------------------------------ Subject: Why am I getting alt.sex.pictures even though I have "ME:!alt.sex.pictures" in my newsfeeds file? The active file is the definitive list of what newsgroups you receive. INN's ME entry is different from C News and B News; please see newsfeeds.5. If you do not want to receive alt.sex.pictures, ask the system(s) that send you news not to send it to you. (You would have to do that no matter what news system you are running.) ------------------------------ Subject: Why doesn't this newsfeeds entry do what I want? "foo.com:alt,!alt.sex" A newsfeeds entry is not a sys file entry. Please read newsfeeds.5. You might also find the sys2nf program in the frontends directory useful, as well as the inncheck Perl script that is found in the samples directory. The INN Configuration FAQ has cook-book examples of the steps required to install a NNTP feed, UUCP feed, and NNTP via nntplink feed. ------------------------------ Subject: Why am I forwarding cancel messages for articles in comp.foo when I explicitly have !comp.foo in the newsfeeds entry? Control messages can be explicitly forwarded, so a control message to comp.foo is forwarded to sites that recieve either comp.foo or control. Please see the "Control Messages" section of innd.8. As that documentation says, you probably want to put "!control" in the subscription list for most of your newsfeeds. ------------------------------ Subject: What's the best way to upgrade to a new version of INN? First, you should read the README and the Install.ms (yes, read them both... again). The README includes a technique to update a new config.data file to be like your old one: % cd config % make subst % cp config.dist config.data % ./subst -f {OLDFILE} config.data where "{OLDFILE}" names your old config.data file. Now edit the config.data to see if you want to change any of the new settings that didn't exist in the old version's config.data file. Compile everything: % cd $INN % make world When you feel you are ready to install the new files shut the old daemon: % ctlinnd shutdown 'upgrade in progress' [ kill innwatch by hand if you need to ] Install the new files: % cd $INN % make update Now update all your $INN/site files to be the same as they were for your old software. "cd $INN/site ; make diff-installed" will tell you what's different between the files in /usr/lib/news and $INN/site. If you only make changes in the $INN/site directory and use "make install" to copy them into place you'll save your self a lot of trouble. Read $INN/site/Makefile for more interesting things that "make" can do. When you feel you are ready to install the new $INN/site files: % cd $INN/site % make install Re-start the system: % sh /usr/lib/news/etc/rc.news If everything was done right you should be up and running. Part 2 of the FAQ gives tips on testing your configuration. ------------------------------ Subject: Can I use gzip with INN? (this was written with the help of Michael Brouwer ) There are three things that can be effected by using gzip: Compression of old logs, compressing batches to send out, and decompressing batches that come in. With INN 1.4 all you need to do is change two lines in config.data to something like this: COMPRESS /usr/local/bin/gzip DOTZ .gz If you rebuild INN with these options set, all logs will be gzipped, and rnews will use gzip to decompress news. gzip will automaticly and transparently decompress UNIX Compress, SCO UNIX Compress (I'm told it's 99% compatible with UNIX Compress), Pack, and gzip. Therefore, you can now receive batches compressed with any of the above listed formats. Let's refer to this as "your site is now a universal decompresser". It has been reported that if you hardlink gzip to be zcat, and make sure that it is the zcat that INN uses, you can get the "universal decompresser" without having to use gzip for your logs. (Though, gzip for your logs is a big win, so why make trouble for yourself?) `send-uucp' will still use compress for outgoing batches, so the sites you feed won't suddenly start getting data they don't understand. Before you can send gzipped batches, you should make sure that the sites that you feed have made the above changes so that they have the "universal decompresser" too. Edit send-uucp to use gzip instead of compress for certain hosts (see example of using compress -b12 for the host esac in send-uucp), outgoing batches will be gzipped. If you use sendbatch, you will have to edit the file so that COMPRESS is set to "gzip" and COMPFLAGS is set to "-9vc". ====================================================================== ADVANCED QUESTIONS ====================================================================== ------------------------------ Subject: What do I do if /var/spool/news is set up over many partitions? First of all, you can do this by either mounting a filesystem at /var/spool/news/comp (for example) or by mounting a filesystem anywhere and making /var/spool/news/comp a symbolic link to the new partition. Articles will be written as normal, but cross-posts have to be handled specially now. Usually INN handles crossposts by writing the article to the first newssgroup, and then creating hard links to all the other places where the article should appear. Hard links do not take up additional disk space (except making your directories longer). Hard links also have the advantage that the file data doesn't get deleted until the last hard link is gone (and they can be deleted in any order). Therefore, you can expire each newsgroup at a different rate, but the file data won't delete until it is expired from the last newsgroup. The problem is that two hard linked files must both be on the same filesystem (partition). When INN sees that it can not make a hard link (because an article is cross-posted across two partitions) it will try to make a symbolic link. If your system can not do symbolic links, set HAVE_SYMLINKS to DONT in your config.data file. This will make INN write a second (or third, etc.) copy of the file instead. (NOTE: INN 1.4 doesn't make the extra files. This feature is planned for INN 1.5.) Anyway, even though INN will automatically create symbolic links, you have to give expire the "-l" flag so that it will know to modify its behavior. Suppose that a message is posted to rec.photo and alt.cameras and suppose that rec.photo expires more quickly then the alt group. If this happens, then you will be left with a dangling symlink. The -l flag prevents this from happening. To inform expire that your spool is split across multiple partitions: In news.daily, change: EXPIREFLAGS="-v1" to read EXPIREFLAGS="-v1 -l" In expirerm, change: RMPROC="fastrm -e -u -s ${SPOOL}" to read RMPROC="fastrm -e -s ${SPOOL}" You also have to edit innwatch and innstat. After that, you're done! If you ever need to run "makehistory" you should pay attention to this caveat in makehistory(8): Makehistory does not handle symbolic links. If the news spool area is split across multiple partitions, the follow- ing commands should probably be run before the database is regenerated: cd /usr/spool/news find . -type l -print | xargs -t rm However, if /var/spool/news/comp was a symlink to the partition that comp is really on, you'll have to reinstall that link. You have avoid this by changing the above commands to: cd /usr/spool/news find . -type l -print | grep '[0-9][0-9]*' | xargs -t rm ------------------------------ Subject: How do I talk to innd from C or Perl? Rich Salz says: If you are writing C, look at doc/inndcomm.3 and include/inndcomm.h; they include all you need to do any ctlinnd command (in fact, ctlinnd itself is little more than a call to the library). Hacking up a Perl subroutine that spoke to innd's Unix-domain control socket should be fairly straightforward but hasn't yet been written. ------------------------------ Subject: How do I set up a delayed IHAVE/SENDME over NNTP? Christophe Wolfhugel writes: INN now allows to generate a timestamp entry in the batchfiles or to the channels/exploders (Wt in newsfeeds) which can be used to allow for example delayed ihave/sendme processing. INN's senders (like innxmit) do not use that data yet. Christophe.Wolfhugel@grasp.insa-lyon.fr has written a small patch for nntplink 3.1.0 which supports this, the patch is available in his anonymous ftp: grasp.insa-lyon.fr:pub/unix/news/nntp/nntplink/delayed-1.0.pch The patch has been incorporated into nntplink 3.2 (3.2 has not been released yet, so don't bother Dave Alden for it. Get 3.1.0 and apply the patch). The delayed IHAVE/SENDME is expected to allow bandwidth savings in situations where all sites use nntplink in following topology: Your site -- 64k -----------+----------- Site 1 | | | 2mb | | +------------ Site 2 Site 1 and 2 are in the same metropolitan area, you feed them both. With the standard nntplink layout, you generally send all articles twice, which is a waste even if you're at 2 Meg/s link and even if Site 1 and 2 do nntplinks, you're faster. The delayed link would be used between your site and Site 2. A 2 or 3 minute delay allows Site 1 to feed Site 2 before you, and in case of a Site 1 outage the backup starts nearly immediately. Reasonnable delays are still kept as You -> 1 -> 2 should take less than one minute (or just 300 ms disk to disk if using nntplink -i ? :)). Experiences seem to show that a 2 to 3 minutes delay is a reasonnable choice. Chris -- Tom Limoncelli -- tal@warren.mentorg.com (work) -- tal@plts.org (play) "Some people run 'biff' to alert them that | Disclaimer: I do not they have new email. I run '/bin/true'". | speak for Mentor Graphics.