Suck 3.3.1 beta for NeGeN-CNews

Brought to you by R&A
Email: Info@RnA.nl

ftp://ftp.nluug.nl/pub/comp/next/CNews/Suck-3.3.1beta.NIHS.b.pkg.tar.README.rtf
ftp://ftp.nluug.nl/pub/comp/next/CNews/Suck-3.3.1beta.NIHS.b.pkg.tar

0. Introduction

What is Suck? Suck is a way to feed a local news server from another news server, without having server-server access. Suck uses client NNTP to download articles. Part of the package is rpost, which is the opposite. With rpost one can upload articles from a local server to a remote server for which one does not have server access (XMIT), but for which one does have client access. Suck and rpost together let you maintain a local newsserver, even when you only have client access to your newshost.

This program has beta status because it has not been tested over a long period of time (I just started using it for production) and because not every part of the behaviour (especially error messages on signals) have been ported.

Suck 3.3.1 has been written with POSIX termio I/O in mind, but we all know how about NeXT and POSIX, don't we? First it was unavailable, then it was available but seriously broken and then it was again unavailable. The package has been adapted for non-POSIX by Gerben_Wierda@RnA.nl. I have to do some cleaning up and I want to have it out in the field for some time, before I release the final version including the patches necessary for compilation. So, I have labeled this beta software and I am open for suggestions, though bear in mind that I have very little time.

Why suck? Isn't there already a program called slurp, which is part of the NeGeN/NiNe distribution? Well, yes, but slurp and suck work differently, giving both advantages and disadvantages:

Slurp connects to the other server, and downloads articles which have been created since the last time slurp connected to that server. To do that correctly, slurp needs to know what time it is over at the other server. For this it needs a connection to the time service (TCP port 37) on the other server. Not all NNTP servers also offer time and if they do, time may be unavailable because of firewall setups outside your influence. One can use local time with slurp, but that makes the process less reliable because both clocks may drift apart.

Suck, on the other hand, does not use time, but just the message-id's and the article numbers on the remote server. So, you don't need access to the time services on the NNTP server that you are connecting to.

The slurp configuration is more straightforward and simple.  Slurp also  injects the article directly into CNews, whereas suck just produces the data which then has to be injected by rnews.

Suck comes with rpost, making a complete leaf node local server possible. You can use rpost from this suck distribution with slurp, by the way.

So, if you are not afraid of some configuration or if you have NNTP access to a server but not access to time, then suck is the way to go. In general, I think slurp is cleaner, because the way suck handles configuration and because the slurp status (article numbers received) does not imply that these articles have indeed been injected in the local news system, which means that you get inconsistent results if you don't make sure that the articles are indeed injected in your local news server. Also, slurp has a better command interface (i.e. with multiple feeds, and extensions, etc, thanks to Tom).

Did you already guess what combination of thoughts produced the icon for this package? The first one to guess gets a drink (here in The Hague).

1. Installation

Make sure you have the CNews distribution from NeGeN/NiNe installed.

Unpack the file Suck-3.3.1beta.NIHS.b.pkg.tar, this produces the package Suck-3.3.1beta.NIHS.b.pkg. This package has been compressed with gzip_package, this has two consequences:
	a. When copying, create 'new links' when you encounter symbolic links
	b. When installing with Installer.app you will get a warning about programs being run. These programs are the scripts that turn the gzip_packaged package into a form that is understandable by Installer.app.

Open de package Suck-3.3.1beta.NIHS.b.pkg as root. Choose install.

2. Configuration of suck

Read the man page. Re-read the man page. Suck needs a configuration/status file that tells it what newsgroups to download (no wildcards as with slurp because this file also is a per group status file). This file also contains the status, that is, what the last article number was that suck downloaded. I give as an example my configuration, which consists of two feeds, news.domainA.com and news.domainB.com. Here are my two suck configuration/status files:

/usr/local/lib/news/sucknewsrc.news.domainA.com:
nl.announce 136
comp.sys.next.advocacy 27512
comp.sys.next.announce 2455
comp.sys.next.bugs 2632
comp.sys.next.hardware 16200
comp.sys.next.marketplace 9241
comp.sys.next.misc 11886
comp.sys.next.programmer 12340
comp.sys.next.software 16352
comp.sys.next.sysadmin 14331
alt.security.pgp 1780

/usr/local/lib/news/sucknewsrc.news.domainB.com:
domainB.subjA 35
domainB.test 86

Explanation: domainA.com is my major news feed. In the configuration/status file I have written all newsgroups I want to download from there. Suck has written the article numbers of each group that were downloaded last time. domainB.com is a server where they have local groups domainB.subA and domainB.test. I mirror these groups locally. When you put a news newsgroup in this file or when you create this file for the first time, put -1 for the number. Read the man page for details. You can add a second number per line. This number is the maximum number of articles suck will download.

Running suck

I have the following lines in /etc/crontab.local:
29 8 * * *   news    rm -f /tmp/news.domainA.com.batch ; /usr/local/lib/newsbin/nntp/suck news.domainA.com -S /usr/spool/news/suck.domainA.log -E /usr/spool/news/suck.domainA.err -dd /usr/local/lib/news -dt /usr/local/lib/news -p .news.domainA.com -dm /usr/spool/news/in.coming -br /tmp/news.domainA.com.batch -c ; cat /tmp/news.domainA.com.batch | /usr/local/lib/newsbin/input/rnews
29 8 * * *   news    rm -f /tmp/news.domainB.com.batch ; /usr/local/lib/newsbin/nntp/suck news.domainB.com -S /usr/spool/news/suck.domainB.log -E /usr/spool/news/suck.domainB.err -dd /usr/local/lib/news -dt /usr/local/lib/news -p .news.domainB.com -dm /usr/spool/news/in.coming -br /tmp/news.domainB.com.batch -c ; cat /tmp/news.domainB.com.batch | /usr/local/lib/newsbin/input/rnews

Explanation: At 8:29 each morning I remove possible remaining news batches from previous calls (suck appends to these if they exist). and call suck with the following arguments:
news.domainA.com (the host to call)
-S /usr/spool/news/suck.domainA.log (where to log progress) 
-E /usr/spool/news/suck.domainA.err  (where to log errors)
-dd /usr/local/lib/news  (where to look for config/status files)
-dt /usr/local/lib/news (see man page ;-)
-p .news.domainA.com  (what to use for extension of the files)
-dm /usr/spool/news/in.coming (see man page :-)
-br /tmp/news.domainA.com.batch (where to write the batch in CNews format)
-c (after completion, update configuration/status file. Make sure that -dt is on the same partition as -dd if you give -c)

After completion of suck, I dump the batch file into rnews. When at some later time newsrun has run, the articles will show up in my local database. Until then, another call to suck will not reget the articles in this case as we have updated the status with -c. If you don't run newsrun between two calls to suck/rnews and you don't use -c, the same articles will be downloaded every time. And of course something comparable for domainB.com. I could have omitted rnews and the batch file, by using the -dm flag and the -m flag for suck and write the articles one-article-per-file directly in in.coming. That has the advantage of omitting all the batch file stuff (rnews only puts it in in.coming anyway), but I am not sure about news locking and what would happen if newsrun would start running while suck would be writing files in in.coming.

Note, -dt standard writes to the current dir, when run from cron, the current dir is /. If you give both -c and -dt, make sure that the -dd and the -dt directories are on the same file system, because suck will try to move the new configuration file (location -dt) to the old (location -dd) when you use -c

3. Warning about the entries in the CNews 'sys' file

Note that the entries in your sysfile should reflect the name that the other news server puts in the Path: header. Sometimes servers called news.domain.com put a different name in there Path: header (like domain.com). If you download with suck and inject these articles with rnews, the local Cnews system does not know that domain.com stands for news.domain.com and it will happily try to upload the articles back to the server you got them from in the first place. Here is an example:

# Only the ME line is mandatory; the others are just samples of how to do
# things.  Virtually everything will need modifying for your local feeds
# and newsgroups.
# Line indicating what we are willing to receive.
ME:all,!Alt,!ann,!atl,!aus,!ausom,!ba,!balt,!bionet,!biz,!can,!clari,!ddn,!de,!demon,!digischool,!euro,!europen,!fido,!fj,!fr,!gay-net,!hacktic,!hiv,!humanities,!ia,!ie,!ieee,!it,!jobs,!kw,!linux,!maus,!msn,!ncar,!ne,!nj,!no,!ny,!nyc,!nz,!ont,!pa,!phl,!relcom,!sco,!sub,!tamu,!to,!tor,!trial,!tw,!u3b,!uk,!us,!vmsnet,!vpro,!wcw,!xs4all,!zer

# Primary newsfeed: (out through UUCP, in through SUCK)
domainA.com:all,!domainB,!general,!junk/all,!local::/usr/local/bin/uux - -z -r domainA!rnews

# domanB news server: (out through RPOST, in through SUCK)
somehost.domainB.com:domainB/all,!local::sed '1,/^$/{;/^Xref/d;}' | /usr/local/lib/newsbin/nntp/rpost news.domainB.com -S /dev/null -E /usr/spool/news/rpost.domainB.err

Some special attention: you can see that news.domainA.com puts just 'domainA.com' in its Path: header. And you can see that news.domainB.com puts 'somehost.domainB.com' in its header. If I did not use the right entries in the sys file (domainA.com and somehost.domainB.com) the articles would have been copied back to the server I got them from. For details on rpost, see below.

4. Configuration of CNews for posting news with rpost

Above in the sys file example, you can see how I use rpost for posting to news.domainB.com. Any article for that server (actually, only the articles for the domainB.* groups) is sent by rpost. But first it is filtered through sed. Because our local news server has created an Xref header, and the news server at the other side does not accept articles from client that have Xref headers set. So, we remove this header before dumping the article into the rpost program. Thanks Tom Hageman for the sed script.

5. The testhost program

Read the man page.