History of updates
Major improvements are in bold.
Version 4.5 (8 June 2009)
In NpMigrat
- NpMigrat is a new tool which allows to convert
the configuration and database files from NewsPlex
versions earlier than 4.5
into the 4.5 format, and optionally delete these
old files. Run it without arguments for help,
pass the -P command-line option to proceed without
deleting the old files, or pass -D
to also delete the old files. Old files
are only deleted if the conversion was
fully succesful. It is possible to
run NpMigrat several times if necessary;
it is idempotent.
Note that NewsPlex 4.5 will refuse to start
if old files are still
found. Also note that conversion works
only for a few past releases.
Finally, note that the "indexing" process for very large
news-groups (with millions of articles) can take half an hour.
- NpMigrat has been built
from newer NewsPlex sources than the 4.4 release,
so it has sometimes a different
behavior. For example, during dataset compaction,
the full entry (line) is now checked for syntax
errors and for coherency with the index entry.
New possible resulting errors are corrupted dataset
entry and Message-ID mismatch.
If a dataset line is corrupted,
NpMigrat now deletes the corresponding
index entry, so that the article description is re-downloaded later.
- NpMigrat now systematically checks that dataset and index lines
end with a newline character, possibly precedded
with a carriage-return character, before accepting them.
This allows to detect truncated files, for example if the
last line was never written to disk entirely.
- In case of problem during the startup cursory dataset
check, NpMigrat now enters a dataset repair mode, which
eliminates the truncated line from dataset's end
and fixes the index file.
- During dataset repair, NpMigrat now tries to recover corrupted
dataset lines if they consist of a truncated entry appended with
the next valid entry. If such a situation is recognized,
the appended entry will be considered as a normal one.
This allows to preserve the corresponding index line.
- When fixing an index versus the dataset and having to delete
the end of it, NpMigrat will now assure that future new
articles will be added with higher numbers than existing
article numbers.
In NpTunnel
- Increased the maximum size of the backlog of incoming
TCP connections. This should reduce TCP connection
errors when several client connections were attempted
to NpTunnel at the same time.
- Improved logging. Traces are now issued after obtaining
the remote connection and before going into routing mode.
When a connection terminates, NpTunnel now indicates
the number of bytes received from each party.
In NpUud
- The BinHex format used on Apple Macintoshes is now supported.
This support can be disabled using the new -Db command-line option.
NpUud will append an extension to the file according to the
recorded Type of the file. If there is also a non-empty Resource
fork, a separate .rsrc file will be created.
In NpSync
- The -q option now makes NpSync quiet about matching files.
Passing it twice will make NpSync quiet even about non-matching files,
apart for the final status.
It can be useful on a slow terminal connection.
- If one of the compared files could not be opened, a specific
message will now be printed, rather than just "different".
Nothing will be done in case of sychronization.
- The new -l option will make NpSync not follow symbolic
links to directories. This can avoid useless comparizons, and
endless loops as well.
- The new -L option will make NpSync ignore symbolic
links during comparizons.
- In messages, the name of the "otherDir" is now provided.
- The new -M option causes a deletion of matching files
from both trees.
- The -m option is now more verbose on deletion failures.
- The display now contains no more useless backspace characters.
- Added the printout of the effect of the default action,
which is just comparing.
- The new -y option causes automatic confirmation when
NpSync asks for it (currently only when deleting some unique
files in -u and -S modes).
In NewsPlex
- Introduced the etc/servdyna.ini file, which stores
news-servers-related information which should persist
accross NewsPlex sessions.
It has the format of a settings file, with sections
called [serverDynamic]. The various
variables inside will be documented later.
- The IHAVE command is now accepted
if the new clients.allowIHAVE= setting is set,
and is equivalent to posting an article to the
server selected by the various posting rules.
- Increased the maximum size of the backlog of incoming
TCP connections. This should reduce TCP connection
errors when several news-reader connections were attempted
to NewsPlex at the same time.
- When a news-server is "deleted" (ie. currently when its news-server references
are deleted), the associated explore files (the normal
one and the .bak one) are now deleted as well.
The newsrc3 database is deleted as well, if it exists.
- Absence of active groups on a news-server is no more considered
an exploration error from the error accounting point of
view.
- Fixed a missing newline bug in exploration log files,
where NewsPlex would sometimes forget to go to a new line
after a dot, before issuing a time-stamp and a new dot.
This happened when dots and normal traces were intermixed
in specific circumstances.
- The new explore.alwaysFilter= setting in etc/newsplex.ini
allows to force filter application on every news-server, even when
this is not individually requested by
server.filterArticles= settings in etc/servers.ini.
- Implemented cross-post filtering. It is now possible to
define additional filtering criteria, based on the number
and names of news-groups to which the article has been posted.
New: the filter.xrefsMin= setting, the
filter.xrefsMax= setting and the filter.xrefsGroups= setting.
- If several news-readers were retrieving the same article in parallel,
and if problems happened and if renumbering was active, NewsPlex
could panic, being unable to find the article under the
old number. Now it only renumbers if the article did not
get renumbered from another connection in the meantime.
-
All RAM-intensive files have been replaced with
on-disk SQLite3 databases which are never loaded into memory
entirely. NewsPlex does not maintain any news-group group
list in memory anymore, neither the full news-group list, nor
the active news-group list, nor the previous "popular groups"
list, nor lists of "actual" news-groups
related to explored news-servers. Similary,
it does not load entirely anything
related to active news-groups or to asynchronous article retrieval.
Various directory names have had their number component
increased, to prevent an old NewsPlex run accidentally
in the new environment from destroying the files.
| Storage type |
Old files and directories |
New files and directories |
| pending article retrievals |
etc/scheduld
etc/done |
etc/async.sq3 |
| Message-ID lists |
etc/hidden
etc/retrievd |
etc/hidden.sq3
etc/retrievd.sq3 |
| full news-group list |
etc/groups |
etc/groups.sq3 |
| news-group popularity |
etc/popular.ini |
| actual news-group lists |
|
actual |
| article database |
dataset2
index
update |
index2 |
| newsrc |
newsrc2 |
newsrc3 |
- If a transaction is internally open for a news-group,
a new T flag appears in the NPACCESSED list.
- Introduced three new settings which can prevent NewsPlex
from trying to connect to news-servers which do not have a
normal IP address. For example, if the hostname
of a news-server resolves into IP address 127.0.0.1,
NewsPlex will connect to itself, which is undesirable in
general and disastrous in case of asynchronous article retrievals, as every
tentative of retrieving an article from oneself will
seemingly fail, while in reality scheduling another asynchronous article retrieval.
The connect.forbiddenIP= setting defines an IP
address pattern to which NewsPlex should not connect.
The default values cover well-known non-routable and
private IP addresses.
The connect.enforceForbiddenIPs= setting controls
whether NewsPlex should apply this globally,
while the server.enforceForbiddenIPs= setting
controls this for a specific news-server.
Both of these settings must be non-zero for enforcing to
be applied to a given news-server.
By default, NewsPlex will no more allow connecting to forbidden
IP addresses. Existing etc/servers.ini files might need to be
adjusted if some news-servers have such IP addresses.
- Durations are now printed as days, hours, minutes and seconds
instead of just seconds in many places.
- NewsPlex now recognizes more "limit exhausted"
errors for GROUP requests, and tries to reconnect
rather than aborting the whole request. This
helps completing the exploration of some news-servers.
When reconnecting, it also better explains the
reason.
- Introduced the clients.reformatDates= setting in
etc/newsplex.ini, which
makes NewsPlex reformat the date and time information
in article descriptions so that it follows a certain predefined standard,
rather than being the original value retrieved from news-servers.
This avoids errors in news-readers.
- When deciding about the order in which
to explore news-groups on a news-server, the news-group
popularity was not taken into account
properly and it was in fact ignored;
only the time of last exploration was.
- The lengths of news-group name, Message-ID and subject are no more
arbitrarily limited to certain maximums
in the asynchronous article retrieval process (and the
full list of articles to be retrieved asynchronously
is no more entirely stored in memory, but resides
in the etc/async.sq3 file).
- If an article had several news-server references pointing to the same news-server,
then depending on their order NewsPlex was not always able
to detect a reget and instead would
detect a dup and add another reget reference.
This could grow the list indefinitely at every explore.
- Added one more recognition of a situation where
the news-server does not accept an XOVER variant but
accepts other variants.
- During exploration of a news-group using HEAD and NEXT
requests,
if the NEXT command does not provide
the next article but rather continues
an enumeration started from 1, NewsPlex will now
switch to issuing HEAD number requests
for all successive numbers.
- NewsPlex now avoids using XHDR MESSAGE-ID when
exploring a news-group if this news-group has just been activated or
if the news-server has two times or more articles than NewsPlex does,
as logically NewsPlex does not yet have most of these
articles and XHDR MESSAGE-ID will only make it lose
time and bandwidth, and delay updating the article database with
new article information.
- Introduced the shutdown.deleteActual= setting in
etc/newsplex.ini, which
makes NewsPlex delete the actual directory
and content on shutdown.
- In log files, the timestamp of messages was separated
from the text sometimes.
- Fixed the diagnostic output on the terminal, which was missing
newlines and tabulations in certain conditions.
- Cleaned the default news-server list of dead news-servers.
- Fixed processing of groups.noByName= setting,
which could crash NewsPlex if a news-group
already excluded by groups.noByPattern= settings
was also passed through groups.noByName= setting.
- Article 1 is now systematically deleted
once article 1000 has been added during news-group activation.
This avoid getting an often empty range of 2-999
article numbers, which makes some news-readers keep
wanting to retrieve them.
- The LISTGROUP request does not return the
numbers of deleted or hidden articles anymore.
- Client disconnecting should be noticed much faster now
in circumstances where it was slow previously, like
a second parallel access to a news-group being already loaded
into memory.
|
