This file documents only a very few of the changes since lq-text 1.13 was
released in December of 1993.
Changes after September 9th, 1995 are logged more fully.
[1] major changes since 1.13
--> Bug Fixes
There have been many bug fixes.
Under certain circumstances, not all the words in the input made their
way into the index. This is the single most serious bug, and is fixed.
The 1.13 distribution did not compile cleanly; now it does.
There were a number of obscure bugs in the way in which queries were
matched, leading to failures to match phrases that should have been
found. These have been corrected.
--> Speedups
You should find that creating an index is much, much faster.
In some cases it is more than twice as fast, in others more than ten
times as fast. lqaddfile also uses a little less memory than it used to.
In a few cases, retrieval is also much faster, and overall it may be a
little faster. The lqkwic client is also faster.
--> New Features
There are several new client programs, a new C-level API, new
documentation, and a number of other new features.
New Clients:
lqmkindex -- a shell script that guides you through making an index
to all the files in a given directory. If I worked at Microsoft,
I'd probably have to call it a Wizard. Maybe it's an Apprentice.
lqunindex -- you can now unindex files
lqsimilar -- this takes a short piece of text, finds other pieces of
text that contain at least some of the same words in the same order,
and ranks the results by similarity. The similarity function is very
simplistic at the moment (see src/Phase.c), and it's very slow for
more than a paragraph or so, but it's there.
A shell script not installed by default is `lqbetter' which improves
the results from lqsimilar. Try running
lqtext/lqsimilar 'some text' | src/lqbetter
lqsed -- this takes a set of matches as input, and produces modified
copies of the matched files, with each match surrounded by a string.
You can give left and right strings on the command-line, or on a
per-match basis.
lq -- this was actually distributed before, but has many new features.
It's a shell-script front end to lq-text. You need to have `tput'
and a System V shell (most systems do these days), and you'll want a
terminal with scroll-back, or an X terminal emulator.
Even if you can't run it, you may find lq useful for the examples it
contains of running and combining the various lq-text clients.
addbyline -- (undocumented, not installed by default) This treats each
line of input as a separate block, and arranges that phrases can't
match over line boundaries. This will be properly supported in the
next release, but at the moment is primarily useful with lqsimilar.
lqsort -- this is a shell script that uses lqkwic and Unix sort to let
you sort results list by a large number of different keys. See the
lq shell script for examples of using lqsort.
Updated Clients
lqaddfile -- now has a -u option to update files. If this is given,
the input should consist of pairs of filenames,
oldfile TAB newfile
where both files exist. The oldfile will be read and unindexed
efficiently, and then replaced by the new file; afterwards, you
should rename the new file to put it where the old one was.
lqkwic -- the little report-generating language has been extended
somewhat; run
lqkwic -x -v -L
for more information.
Deprecated Clients
lqtext -- the libcurses-based front end will probably vanish in the
next release, unless enough people still want it. If anyone would
like to work on it or to mantain it, please let me know (lee@sq.com)
xlqtexttool -- the XView-and-TNT-based OPEN LOOK UI front end
for NeWS is no longer included in the distribution. The Motif client
will replace it. Sun no longer ships NeWS, and it didn't run on X11.
intersect -- this shell script is still included, but is entirely
superseded by lqrank.
--> Installation Changes
A configure script is now included. This is not Larry Wall's perl
configuration script, which was too large and complex; it also isn't the
GNU one, partly because I didn't find out in time that they were
different, and partly because I was incorrectly concerned about the GPL.
So the configure script may be replaced in the next version; let me know
what you think of the one that's there, though.
See lq-text/INSTALL for notes on running configure, or just run it.
--> Porting changes
You should no longer have to edit globals.h for a port; there are two new
header files, port.h and port2.h, included at the start and end of
globals.h respectively. The first of these, port.h, is generated by
configure, as is the Makefile in the src directory. Once you have run
configure, you can either edit port.h, or override it in port2.h; see
the PORTING file for more information.
--> C API
There is now a supported C API. Well, supported means that you can use
it and feel free to send me mail about it, and I shall try to answer you.
The C functions in lq-text now all start with an LQ prefix, so that there
are unlikely to be any conflicts or surprises in linking.
I am still working on the API, so it is not as stable as I should like,
but it is there, and it is documented (q.v.).
--> Documentation
There is a new User Guide in HTML and PostScript.
There are also Unix man pages.
The C API is documented in SGML, and there are style sheets for use
with SoftQuad Panorama under X11 or Microsoft Windows.
There is also a printed, bound manual that you order directly from me;
this costs Cdn$87.50 (within Canada) or US$9o.00 outside Canada,
including delivery. Please check the prices before ordering or sending
cheques, in case they have changed! The price is partly an attempt to
subsidise my work on lq-text.
[2] C Level changes
The rest of this file documents changes to lq-text by date. It was begun
in September, 1995. If you did not previously use lq-text, or if you never
made any changes to the C, you probably don't need to read further.
Mon Oct 2 23:44:01 EDT 1995
Added sorting to the "lq" client (this isn't actually a C-level change)
Sat Sep 23 21:15:21 EDT 1995
First port to Solaris 2.
Added a new test in the config.dir directory, for siglist.
Initial testing was quite promising.
Fri Sep 22 00:52:18 EDT 1995
LQT_MakeMatchesWhere now uses symbolic constants for the
Accept Function.
Wed Sep 20 22:56:16 EDT 1995
Renamed the files in src/liblqtext for portability.
Split Phrase.c into separate functions, one per file.
All LQT_* functions now take a db argument first except
LQT_InitFromArgv, LQT_PrintDefaultUsage, and the
LQTrace routines. Probably the tracing flags should be at least partly
per-database, so those should have a db argument too, but I'd like to
move the underlying technology to liblqutil first.
Tue Sep 19 23:07:25 EDT 1995
lqrename now takes a -f option to read a list of files to rename from
a file of the form
currentname<tab>newname
Mon Sep 18 22:54:32 EDT 1995
Moved MakeInput from lqaddfile.c and addbyline.c into
LQT_MakeFileInfoInput; also moved MakeFileInfo into the library as
LQT_MakeFileInfo, and documented both functions.
Sun Sep 17 01:35:15 EDT 1995
lqaddfile, lqphrase, lqfile, lqword now working with the new API.
I have removed the `delete word' functionality from lqword, since it
was a mis-feature. It would be nice to support a function to turn a word
that you had indexed normally into a stop word, but it would be faster to
reindex all the data with the current structures, and I don't think a
major change is warranted for this feature.
Fri Sep 15 00:13:06 EDT 1995
Almost all LQT* functions now take a pointer to a t_LQTXT_Database
as their first argument. Extensive changes to the library files...
The clients don't all have any t_LQTXT_Database pointers yet.
I have also merged readfile.h into lqutil.h; readfile.h is no longer needed.
Sun Sep 10 23:18:19 EDT 1995
The emalloc functions are no longer macros that rename functions;
I have renamed the functions so the macros aren't needed.
Next, to remane the functions to give them an LQ prefix!
Fri Sep 8 00:38:31 EDT 1995
Now you will have to use LQT_OpenDatabase, and pass it the result
returned by initFromArgcAndArgv. The latter now returns a structure
describing options.
Thu Sep 7 18:22:29 EDT 1995
Created lqconfig.h to start the configuration stuff.
Started rewriting OpenDatabase and Defaults.c to use it.
Wow, it's a lot of work.
Naked boys, lissome youths,
ankles and feet divine
dance upon the daisy-
carpet of the forest
of my Mind
Thu Sep 7 17:25:29 EDT 1995
Removed the global variables from globals.h and put them into a struct;
the struct is initialised in LQT_OpenDatabase(directory).