This file documents only a very few of the changes since lq-text 1.13 was released in December of 1993. Changes after September 9th, 1995 are logged more fully. [1] major changes since 1.13 --> Bug Fixes There have been many bug fixes. Under certain circumstances, not all the words in the input made their way into the index. This is the single most serious bug, and is fixed. The 1.13 distribution did not compile cleanly; now it does. There were a number of obscure bugs in the way in which queries were matched, leading to failures to match phrases that should have been found. These have been corrected. --> Speedups You should find that creating an index is much, much faster. In some cases it is more than twice as fast, in others more than ten times as fast. lqaddfile also uses a little less memory than it used to. In a few cases, retrieval is also much faster, and overall it may be a little faster. The lqkwic client is also faster. --> New Features There are several new client programs, a new C-level API, new documentation, and a number of other new features. New Clients: lqmkindex -- a shell script that guides you through making an index to all the files in a given directory. If I worked at Microsoft, I'd probably have to call it a Wizard. Maybe it's an Apprentice. lqunindex -- you can now unindex files lqsimilar -- this takes a short piece of text, finds other pieces of text that contain at least some of the same words in the same order, and ranks the results by similarity. The similarity function is very simplistic at the moment (see src/Phase.c), and it's very slow for more than a paragraph or so, but it's there. A shell script not installed by default is `lqbetter' which improves the results from lqsimilar. Try running lqtext/lqsimilar 'some text' | src/lqbetter lqsed -- this takes a set of matches as input, and produces modified copies of the matched files, with each match surrounded by a string. You can give left and right strings on the command-line, or on a per-match basis. lq -- this was actually distributed before, but has many new features. It's a shell-script front end to lq-text. You need to have `tput' and a System V shell (most systems do these days), and you'll want a terminal with scroll-back, or an X terminal emulator. Even if you can't run it, you may find lq useful for the examples it contains of running and combining the various lq-text clients. addbyline -- (undocumented, not installed by default) This treats each line of input as a separate block, and arranges that phrases can't match over line boundaries. This will be properly supported in the next release, but at the moment is primarily useful with lqsimilar. lqsort -- this is a shell script that uses lqkwic and Unix sort to let you sort results list by a large number of different keys. See the lq shell script for examples of using lqsort. Updated Clients lqaddfile -- now has a -u option to update files. If this is given, the input should consist of pairs of filenames, oldfile TAB newfile where both files exist. The oldfile will be read and unindexed efficiently, and then replaced by the new file; afterwards, you should rename the new file to put it where the old one was. lqkwic -- the little report-generating language has been extended somewhat; run lqkwic -x -v -L for more information. Deprecated Clients lqtext -- the libcurses-based front end will probably vanish in the next release, unless enough people still want it. If anyone would like to work on it or to mantain it, please let me know (lee@sq.com) xlqtexttool -- the XView-and-TNT-based OPEN LOOK UI front end for NeWS is no longer included in the distribution. The Motif client will replace it. Sun no longer ships NeWS, and it didn't run on X11. intersect -- this shell script is still included, but is entirely superseded by lqrank. --> Installation Changes A configure script is now included. This is not Larry Wall's perl configuration script, which was too large and complex; it also isn't the GNU one, partly because I didn't find out in time that they were different, and partly because I was incorrectly concerned about the GPL. So the configure script may be replaced in the next version; let me know what you think of the one that's there, though. See lq-text/INSTALL for notes on running configure, or just run it. --> Porting changes You should no longer have to edit globals.h for a port; there are two new header files, port.h and port2.h, included at the start and end of globals.h respectively. The first of these, port.h, is generated by configure, as is the Makefile in the src directory. Once you have run configure, you can either edit port.h, or override it in port2.h; see the PORTING file for more information. --> C API There is now a supported C API. Well, supported means that you can use it and feel free to send me mail about it, and I shall try to answer you. The C functions in lq-text now all start with an LQ prefix, so that there are unlikely to be any conflicts or surprises in linking. I am still working on the API, so it is not as stable as I should like, but it is there, and it is documented (q.v.). --> Documentation There is a new User Guide in HTML and PostScript. There are also Unix man pages. The C API is documented in SGML, and there are style sheets for use with SoftQuad Panorama under X11 or Microsoft Windows. There is also a printed, bound manual that you order directly from me; this costs Cdn$87.50 (within Canada) or US$9o.00 outside Canada, including delivery. Please check the prices before ordering or sending cheques, in case they have changed! The price is partly an attempt to subsidise my work on lq-text. [2] C Level changes The rest of this file documents changes to lq-text by date. It was begun in September, 1995. If you did not previously use lq-text, or if you never made any changes to the C, you probably don't need to read further. Mon Oct 2 23:44:01 EDT 1995 Added sorting to the "lq" client (this isn't actually a C-level change) Sat Sep 23 21:15:21 EDT 1995 First port to Solaris 2. Added a new test in the config.dir directory, for siglist. Initial testing was quite promising. Fri Sep 22 00:52:18 EDT 1995 LQT_MakeMatchesWhere now uses symbolic constants for the Accept Function. Wed Sep 20 22:56:16 EDT 1995 Renamed the files in src/liblqtext for portability. Split Phrase.c into separate functions, one per file. All LQT_* functions now take a db argument first except LQT_InitFromArgv, LQT_PrintDefaultUsage, and the LQTrace routines. Probably the tracing flags should be at least partly per-database, so those should have a db argument too, but I'd like to move the underlying technology to liblqutil first. Tue Sep 19 23:07:25 EDT 1995 lqrename now takes a -f option to read a list of files to rename from a file of the form currentnamenewname Mon Sep 18 22:54:32 EDT 1995 Moved MakeInput from lqaddfile.c and addbyline.c into LQT_MakeFileInfoInput; also moved MakeFileInfo into the library as LQT_MakeFileInfo, and documented both functions. Sun Sep 17 01:35:15 EDT 1995 lqaddfile, lqphrase, lqfile, lqword now working with the new API. I have removed the `delete word' functionality from lqword, since it was a mis-feature. It would be nice to support a function to turn a word that you had indexed normally into a stop word, but it would be faster to reindex all the data with the current structures, and I don't think a major change is warranted for this feature. Fri Sep 15 00:13:06 EDT 1995 Almost all LQT* functions now take a pointer to a t_LQTXT_Database as their first argument. Extensive changes to the library files... The clients don't all have any t_LQTXT_Database pointers yet. I have also merged readfile.h into lqutil.h; readfile.h is no longer needed. Sun Sep 10 23:18:19 EDT 1995 The emalloc functions are no longer macros that rename functions; I have renamed the functions so the macros aren't needed. Next, to remane the functions to give them an LQ prefix! Fri Sep 8 00:38:31 EDT 1995 Now you will have to use LQT_OpenDatabase, and pass it the result returned by initFromArgcAndArgv. The latter now returns a structure describing options. Thu Sep 7 18:22:29 EDT 1995 Created lqconfig.h to start the configuration stuff. Started rewriting OpenDatabase and Defaults.c to use it. Wow, it's a lot of work. Naked boys, lissome youths, ankles and feet divine dance upon the daisy- carpet of the forest of my Mind Thu Sep 7 17:25:29 EDT 1995 Removed the global variables from globals.h and put them into a struct; the struct is initialised in LQT_OpenDatabase(directory).