This file lists some of the main files and directories. It isn't complete, as otherwise it'd be too big. Files marked as DOOMED may be replaced soon. Use this document to work out what a given file or directory is for in the distribution. Use doc/tutorial to get started. Lee @(#) $Revision: 1.3 $ README - read the README file first, please! Then come back here... GuidedTour - This file BugReport - A template for bug reports sent by mail doc - directory containing the documentation for lq-text src - directory containing the source for lq-text. Sample - A sample empty LQTEXTDIR waiting to have a database created in it :-) src/TODO - what I haven't done yet. Read this before adding features. Sample/CommonWords A list of common words, taken from the Bible. They are all commented out, though. There are also numbers in there to give you some idea of the frequency of the common words. Sample/* A sample empty database: mkdir $HOME/LQTEXTDIR cp Sample/* $HOME/LQTEXTDIR textedit $HOME/LQTEXTDIR/* Sample/config a sample config file. You should edit this. src/h/* Directory containing all of the lq-text C header files. src/h/globals.h is the one you'll have to change. src/filters/* The programs in here are filters that improve indexing quality. When lqaddfile is given what it thinks is a mail folder, for example, it starts up MailFilter and reads its output instead of the actual file. src/sdbm/* Ozan Yigit's sdbm package is an ndbm replacement. src/contrib/* Software contributed by various users, but not supported by me. I do not ever expect to support the perl scripts, I'm afraid. If you use any of the others, let me know & I'll consider adding them to the main distribution. Software to go here is always welcomed. src/TODO Things I haven't done yet, wish list src/menu/* This directory contains part of a simple menu package for the Unix curses(3X) library, to run on a tty (i.e. not X Windows). There are a couple of examples, and "lqtext" is one of them, a front end similar in principle to lqtext/lq.sh but more complicated. src/liblqtext/* lq-text architecture is like this: there's a library that provides access to the search database, and it's called liblqtext.a, and the source for it is in src/liblqtext. This library uses a couple of other libraries, liblqutil.a and liblqerror.a, which have their own directories. An application, such as lqphrase, is linked against all of these libraries. In addition, X Windows clients using the lq-text libraries would link against the X libraries, and so on. src/bin/* When you do a make, this is where the binaries go before you do a make install. This directory starts out empty. src/db.1.85/* This is an ndbm replacement with a vengeance. It is the most robust of all the dbm packages. src/lib/* The compiled libraries go here. This directory starts out empty. src/liblqerror/* Error handling library used by lq-text -- can be used elsewhere if you like. src/liblqutil/* A library of useful routines. You might find these useful for other programs, too. They use the liblqerror error handling routines. src/lqtext/* The source for the lq-text programs themselves src/saber.addfile for debugging lqaddfile with sabre C (CentreLine) src/saber.project src/saber.project-all src/sabre.project.menu If you don't have Sabre-C, call CentreLine NOW and ask for it. You should not be debugging C programs without this under any reasonable circumstance. If you think you have something better, TELL ME ABOUT IT!!!! src/tags for use with vi or emacs - "make tags" will create this. src/test/* various badly documented test files. src/testbin/* Where the compiled tests go. They don't get installed anywhere else. src/tradeoffs Some comparisons of different parameters from globals.h More detail: ============ src/filters/FilterMain.c Each filter links with this. src/filters/MailFilter.c Filter for /bin/mail-style files (e.g. /usr/spool/mail/lee). MailFilter deletes Received-By lines and other irrelevant mail headers. src/filters/Makefile You should not ever need to edit this file. All of the variables in here are actually over-ridden by src/Makefile. src/filters/NewsFilter.c Filter for netnews (usenet) articles. Deletes the Path header. Also deletes Message-Id at the moment, since lqaddfile couldn't deal with it, although that might change in the future. src/filters/TroffFilter.c Filter for troff (or SoftQuad's sqtroff) files. Not yet used. src/contrib/boolean.dist boolean expressions for lqphrase src/contrib/README src/contrib/intersect.pl Tom Christiansen's replacement for my intersect.sh - it's slower but it does more. src/contrib/rs6000.diffs src/contrib/rs6000-part2 For use with an IBM system. src/menu/examples A directory with a couple of examples of using "m", a menu program. You'll need to do a "make m" in src/menu first. src/menu/Makefile This is one of the two makefiles that you might need to edit, because curses(3X) is not too portable. (the other is src/Makefile) src/menu/error.c Draw error messages on the screen src/menu/README Out-of-date notes for part of the original menu package I wrote. src/menu/OldCurses.c src/menu/oldcurses.h The old* files are for Curses(3) instead of curses(3X). src/menu/character.c DOOMED src/menu/error.h src/menu/example.c src/menu/internal.h src/menu/menu.c src/menu/menu.h src/menu/stringbox.c src/menu/ultrixhack.h src/menu/liblqmenu.a This is the library that gets made... src/menu/text.c This is the source for lqtext itself, the sample client. doc/* the documentation, what there is of it doc/Admin.Guide doc/DOCUMENTATION doc/Mailshot doc/ProgramStructure doc/lqtext.1 The Unix man-page doc/port notes on porting. DOOMED. doc/theory doc/API If you get this, you have got the ApplicationC Programmers' Interface documentation for the libraries. If you didn't get it, and would like it, contact lee@sq.com; if it isn't included, it's either too big or, more likely, too out of date or too short. src/COPYRIGHT this is NOT the Gnu Public Licence. src/GetIdList shell script to get the RCS ident of each file. src/Makefile This is the most important configuration file for lq-text. The second-most important one is src/h/globals.h, and you should edit them both. src/PORTING Notes on porting. src/PeterAndJane A sample file to index. src/REVISIONS This lists the revisions of each file. It's useful to help me keep track of who has what. src/bin/* When you do a make, this is where the binaries go before you do a make install. This directory starts out empty. src/bin/intersect - find files that have two or more phrases in common src/bin/lq - shell script front end to the lq-text package. src/bin/lqaddfile - creates an index, or adds files to an index. src/bin/lqfile - lists files that you have indexed. src/bin/lqkwik - keyword index: lqkwik `lqphrase "phrase"` src/bin/lqmkfree - ADMIN: rebuild free bit file by reading data file src/bin/lqphrase - match phrases src/bin/lqrename ADMIN: rename a file in the index src/bin/lqshow - show matches: lqshow `lqphrase "some phrase"` src/bin/lqtext - curses-based front end src/bin/lqword - lists words in the database. src/bin/lqwordlist - print a sorted list of all words in the database. src/bin/mkwidtable - ADMIN: rebuild wordlist by looking at data. DOOMED. src/bsdhash/* This is an ndbm replacement with a vengeance. src/bsdhash/Makefile You might have to edit this one, sorry. src/db/* This is a later version of bsdhash. src/db/Makefile I made this, and it isn't right yet. Sorry. src/db/Makefile.inc The original makefile, which also wasn't right, mainly because it didn't contain any rules! src/h/* Directory containing all of the lq-text C header files. src/h/globals.h is the one you'll have to change. src/h/Liamdbm.h Used with gdbm. Watch out for potential licence conflicts if you use gdbm, though. DOOMED. src/h/Revision.h The lq-text version. src/h/blkheader.h low-level blocks src/h/declare.h declare things not in stdlib.h for use with gcc src/h/emalloc.h for my malloc wrapper which does extra checking src/h/error.h for Error() src/h/fileinfo.h for storing filenames src/h/filter.h defines filter types and magic numbers src/h/globals.h CHANGE THIS FILE src/h/numbers.h read/write compressed integers src/h/pblock.h physical blocked match lists src/h/phrase.h for phrases src/h/smalldb.h for the ndbm front end liblqtext/smalldbm.c src/h/wordindex.h for the Word to WID map src/h/wordinfo.h information about words src/h/wordplace.h storing matches in memory src/h/wordrules.h CHANGE THIS FILE - max word length, etc. src/lib/* The compiled libraries go here. This directory starts out empty. This is also where the filters go before you do a make install. src/lib/MailFilter filter for mail src/lib/NewsFilter filter for netnews src/lib/libhash.a bsdhash package (optional) src/lib/libsdbm.a sdbm package (optional) src/lib/libdb.a 4.4 BSD db package (optional) src/lib/liblq.a Liam's C utilities src/lib/liblqerror.a error message handler src/lib/liblqmenu.a curses menu src/lib/liblqtext.a lqtext search & index engine src/liblqerror/* source for a simple Error() function. You might find this useful for other programs, too. src/liblqerror/Makefile You don't need to edit this. src/liblqerror/error.c src/liblqerror/error.h This gets copied into src/h automatically. src/liblqtext/* Source for liblqtext.a src/liblqtext/Common.c handle common words that are not indexed. src/liblqtext/Defaults.c handle user defaults from command line and README src/liblqtext/DocPath.c finds files in the document path src/liblqtext/FileList.c handles storage/retrieval of filenames. src/liblqtext/FilterType.c determine the type of a file so the right filter (if any) gets used. src/liblqtext/GetWID.c do the Word -> WID mapping, i.e. (char *) -> (unsigned long) src/liblqtext/Makefile You should NOT need to change this file. It is a bug if you do, and you can freely report it! src/liblqtext/Phrase.c routines for matching phrases (used mostly by lq-phrase now) src/liblqtext/Root.c routines for morphological and etymological grouping. OK, it strips an s off plurals. src/liblqtext/SetWID.c get/set the largest current word number src/liblqtext/Unpack.c unpack archived or compressed files for retrieval src/liblqtext/WordInfo.c handle the main wordlist src/liblqtext/asciitrace.c defines AsciiTrace, a variable for debugging. src/liblqtext/bcopy.c a bcopy() routine that works. May not be as fast as the system-supplied one, though. If you're confident that your bcopy() is OK, and does overlapping regions, you don't need this file. src/liblqtext/block.c The low-level physical Word Database for lq-text. src/liblqtext/cmdname.c DOOMED declares cmdname. If a shell script sets $CMDNAME, all lq-text error messages will include it before progname. src/liblqtext/lastnext.c DOOMED defines a global variable src/liblqtext/lqsort.c 4.3 BSD qsort() adapted somewhat. src/liblqtext/numbers.c routines for writing and reading compressed numbers src/liblqtext/pbcache.c physical block cache src/liblqtext/progname.c DOOMED defines the program name src/liblqtext/rpblock.c routines to read physical data blocks src/liblqtext/smalldb.c front end to dbm/ndbm/bsdhash/db/sdbm... to help portability src/liblqtext/system.c a system(3) routine for use with curses programs. DOOMED. src/liblqtext/wpblock.c routines to write physical data blocks src/liblqutil/* A library of useful routines. You might find these useful for other programs, too. src/liblqutil/Makefile You should NOT need to edit this. src/liblqutil/cmdname.c see above src/liblqutil/getline.c read a line from a file, using emalloc() src/liblqutil/isdir.c is this path name a directory? src/liblqutil/isfile.c is this path name an ordinary file? src/liblqutil/malloc.c special malloc() with extra checking. Never returns NULL. src/liblqutil/progname.c see above src/liblqutil/strings.c some string functions -- is a prefix of, etc. src/lqtext/* The source for the lq-text programs themselves src/lqtext/FindCommon.sh make a list of the n most common words in the database. Can be slow. src/lqtext/Makefile You should NOT need to edit this. All configuration is in src/Makefile, not here. src/lqtext/fileindex.c for lqfile src/lqtext/intersect.sh src/lqtext/lines.c src/lqtext/lq.sh src/lqtext/lqaddfile.c The most important program - makes the index in the first place! src/lqtext/lqfile src/lqtext/lqkwik.c src/lqtext/lqmkfree.c src/lqtext/lqphrase.c The second most important program - say where a given phrase occurs in all of the indexed files in the database. src/lqtext/lqrename.c src/lqtext/lqshow.c src/lqtext/lqword.c src/lqtext/lqwordlist.c src/lqtext/matchword.sh src/lqtext/mkwidtable.c src/lqtext/sizes.c src/lqtext/wordtable.c src/ozmahash GONE replaced by bsdhash and db src/saber.addfile for debugging lqaddfile with Sabre-C (now CodeCentre) from CentreLine. src/saber.project src/saber.project-all src/sabre.project.menu If you don't have Sabre-C, call CentreLine NOW and ask for it. You should not be debugging C programs without this under any reasonable circumstance. If you think you have something better, TELL ME ABOUT IT!!!! src/tags for use with vi or emacs - "make tags" will create this. src/test/* various badly documented test files. src/test/README explains the tests src/test/Makefile do not edit this. Do not run make from this directory. src/test/MaxWid.c src/test/NumberTest src/test/TryHash.c src/test/TryNum.c src/test/TryRoot.c src/test/dbmtry.c src/test/differ.awk src/test/malloctrace src/test/numbers.c src/test/put.c src/test/size.c src/test/trywid.c src/test/wordlengths src/testbin/* Where the compiled tests go. They don't get installed anywhere else. src/testbin/MaxWid src/testbin/NumberTest src/testbin/TryHash src/testbin/TryNum src/testbin/TryRoot src/testbin/dbmtry src/testbin/put src/testbin/trywid src/xlqtext/ An old experimental X interface; it used the XView toolkit. An OSF/Motif one was started, but Motif was proprietary and buggy. The gtk version looks more promising but is not yet included.