From rob@agate.berkeley.edu Thu Aug 12 18:15:34 1993 From: rob@agate.berkeley.edu (Rob Robertson) Newsgroups: news.software.nntp,news.software.b,news.software.readers Subject: FAQ: Overview database / NOV General Information Date: 2 Aug 93 18:20:37 Organization: University of California, Bezerkeley Distribution: inet NNTP-Posting-Host: gangrene.berkeley.edu [this is my third shot at this FAQ, tell me what I got wrong... thanx] Table Of Contents: ------------------ What is `threading' and why is it good for you? What is the Overview/NOV database? What is the format of the NOV database? What optional headers should I configure into the NOV database? What is XOVER? What does it buy me? How much disc space does the Overview database take up? What News Transport Systems support NOV/XOVER and where can I get it? What Newsreaders Support NOV/XOVER? Any tips on configuring the Overview database? Contributions to NOV FAQ. ------------------------------ Subject: What is `threading' and why is it good for you? I put this in first as a background question. As news volume has grown in the last couple of years, the organization of the data presented to the user has become more important. With the possibility of being presented with several hundred articles to read in a newsgroup, newer newsreaders have incorporated features that organize things by subject, and sort them in the order posted, and `thread' articles in groups according to who replied to who. This `threading' can be expensive using normal NNTP commands, as it involves opening each article and transferring it over to the client, for a relatively small bit of information. This led to newsreader authors extending the NNTP command set so that they could set up an auxiliary database of just the information they needed. The database would be updated as news arrived on the server, and when a client requested thread information about a particular group, it would just open one file and send it out, which made things pretty zippy. The downside of this, was that every newsreader had their own threading database, their own specialized NNTP extension, and it was customized to their `way' of doing threads. Trn, NN, and tin are prime examples. Each database occupies a fair amount of space, and in the case of trn, the daemon that maintains it uses up a nontrivial amount of system resources. ------------------------------ Subject: What is the Overview/NOV database? The Overview or News OverView (NOV) database was designed by Geoff Collyer. Its a more generalized database, it it contains *no* threading information, just the information needed to thread. The overview database is just a database of files, one for each group, containing a reference to each article, with seven or eight of the most popular headers. Thus, the newsreader client can get most of the information it needs to thread, but it does the threading the way it wants to. Yes, the downside is that now the threading occurs on the client's CPU, but that's not as expensive as it may seem, if done properly. ------------------------------ Subject: What is the format of the NOV database? [this is stolen from Geoff Collyer's newsoverview(5) man page] Each newsgroup directory contains a file named `.overview', containing one-line summaries of articles in that group. Fields are separated by tabs, and any tabs or newlines in the original articles headers have been replaced with spaces. The fields are, in order: article number (file name), subject, author, date, message-id, references, byte count, line count, and optionally other headers, as arranged locally (none are supplied by the database maintenance software, as shipped). The line-count and references field may be empty. If the optional other headers are present, they include their header keyword and colon; if they are absent entirely, the tab after the line-count field may also be absent. ie: artid|subject|From|Date|Message-Id|References|bytecount|linecount|\ Optional-Header: Stuff where | represents a tab and \ a line continuation. ------------------------------ Subject: What optional headers should I configure into the NOV database? It's nice to configure the pseudo-header Xref: in. Xref: is a site specific header that, tells if present where that are article has been cross posted. It is useful for newsreaders who want to kill cross posted articles, you've told them to kill. ------------------------------ Subject: What is XOVER? XOVER is an NNTP extension for accessing the Overview database. It seems like it's becoming a defacto standard for accessing bulk header/thread information via NNTP. It is used after establishing a current group with the GROUP command, and operations on that group. It's syntax is XOVER artnumber1[-artnumber2] Where artnumber1 is < artnumber2. If successful, it returns a 224 reply code, and the overview data [as in the Overview database] for the requested articles. It is terminated by a "." (dot) on a line by itself. ------------------------------ Subject: What does it buy me? Speed, if you have newsreader clients that can take advantage of this, disc space if you are currently supporting more than one version of a threading database. If used with NNTP, XHDR can be speed up by *orders* of magnitude by using the Overview database. ------------------------------ Subject: How much disc space does the Overview database take up? About 10% of what your news spool space is. ------------------------------ Subject: What News Transport Systems support NOV/XOVER and where can I get it? INN 1.4 supports the Overview database and XOVER right out of the box. C-News has an add-on patch kit that allows it to support NOV. It's available via ftp from world.std.com:/src/news/nov.dist.tar.Z. That gets you to the Overview stage, if you want to use the XOVER extension, you need Stan Barber's NNTP reference package (via ftp from lib.tmc.edu:/public/nntp.1.5.11.tar.Z) and patch in Wayne Davidson's XOVER extensions available in trn3.0 (in the trn3/nntp directory). The next version of C-News promises to integrate all this. ------------------------------ Subject: What Newsreaders Support NOV/XOVER? Among `officially released' newsreaders trn3.2 and tin-1.21 support both the Overview database and XOVER. Win-vn 0.8 for MS-Windows claims to support XOVER [Alan Thew]. Geoff has modified `proof of concept' newsreaders in world.std.com:/src/news/ that support the Overview database (*not* XOVER). There is a modified version of NN that supports XOVER (I forget where, but it core dumps on my modified version of inn, but it might not on yours). But the official NN maintainer says that he's going to support it `soon'. There are some excellent patches by Felix Lee that allow GNUS to support XOVER (plus some kool async transfer mods). Perhaps he'll post them again soon. Perhaps they will be integrated into GNUS. PC and Mac NNTP newsreader support is lagging. ------------------------------ Subject: Any tips on configuring the Overview database? It's probably a good idea to configure in the optional pseudo header Xref:, as many newsreaders like to see it. If newsreading is totally limited to NNTP, you might want to consider putting your Overview database on another partition on another disc (if you have space). This buys you more room for news on your spool disc, and also spreads the disc accesses out across multiple drives, reducing the beating your spool partition is going to take. Currently INN can do this easily, by changing the OVERVIEWDIR variable. C-news is more problematic. It would be nice if newsreader authors would not assume that the Overview database directory and the news spool directory are the same, so that others that read via NFS or locally can enjoy the same performance win.... ------------------------------ Subject: Contributions to NOV FAQ. Thanks to the following for contributions, additions, corrections, and updates: Geoff Collyer Damon Hart-Davis Mark Linimon News.Admin@liverpool.ac.uk (Alan Thew) This posting, like much of Usenet, is maintained on a purely volunteer basis. I welcome reactions, additions, and corrections via email at rob@agate.berkeley.edu. -- william robertson rob@violet.berkeley.edu "when i was your age, i walked 10 miles to school, unarmed."