From 6efae5bf2a6bb0de31f53ecbe5dc52879fbc54c4 Mon Sep 17 00:00:00 2001 From: Tom Lane Date: Wed, 17 Oct 2007 01:01:28 +0000 Subject: [PATCH] Another round of editorialization on the text search documentation. Notably, standardize on using "token" for the strings output by a parser, while "lexeme" is reserved for the normalized strings produced by a dictionary. --- doc/src/sgml/ref/alter_tsdictionary.sgml | 17 +- doc/src/sgml/textsearch.sgml | 592 +++++++++++++---------- 2 files changed, 364 insertions(+), 245 deletions(-) diff --git a/doc/src/sgml/ref/alter_tsdictionary.sgml b/doc/src/sgml/ref/alter_tsdictionary.sgml index a2929c70d1..69bc429851 100644 --- a/doc/src/sgml/ref/alter_tsdictionary.sgml +++ b/doc/src/sgml/ref/alter_tsdictionary.sgml @@ -1,5 +1,5 @@ @@ -122,6 +122,21 @@ ALTER TEXT SEARCH DICTIONARY my_dict ( StopWords = newrussian ); ALTER TEXT SEARCH DICTIONARY my_dict ( language = dutch, StopWords ); + + + The following example command updates the dictionary's + definition without actually changing anything. + + +ALTER TEXT SEARCH DICTIONARY my_dict ( dummy ); + + + (The reason this works is that the option removal code doesn't complain + if there is no such option.) This trick is useful when changing + configuration files for the dictionary: the ALTER will + force existing database sessions to re-read the configuration files, + which otherwise they would never do if they had read them earlier. + diff --git a/doc/src/sgml/textsearch.sgml b/doc/src/sgml/textsearch.sgml index b4bad63b9d..9dfeefea4d 100644 --- a/doc/src/sgml/textsearch.sgml +++ b/doc/src/sgml/textsearch.sgml @@ -1,4 +1,4 @@ - + Full Text Search @@ -75,11 +75,11 @@ - Parsing documents into lexemes. It is - useful to identify various classes of lexemes, e.g. numbers, words, + Parsing documents into tokens. It is + useful to identify various classes of tokens, e.g. numbers, words, complex words, email addresses, so that they can be processed - differently. In principle lexeme classes depend on the specific - application but for an ordinary search it is useful to have a predefined + differently. In principle token classes depend on the specific + application, but for most purposes it is adequate to use a predefined set of classes. PostgreSQL uses a parser to perform this step. A standard parser is provided, and custom parsers @@ -89,11 +89,18 @@ - Converting lexemes into normalized - form. This allows searches to find variant forms of the + Converting tokens into lexemes. + A lexeme is a string, just like a token, but it has been + normalized so that different forms of the same word + are made alike. For example, normalization almost always includes + folding upper-case letters to lower-case, and often involves removal + of suffixes (such as s or es in English). + This allows searches to find variant forms of the same word, without tediously entering all the possible variants. Also, this step typically eliminates stop words, which are words that are so common that they are useless for searching. + (In short, then, tokens are raw fragments of the document text, while + lexemes are words that are believed useful for indexing and searching.) PostgreSQL uses dictionaries to perform this step. Various standard dictionaries are provided, and custom ones can be created for specific needs. @@ -105,17 +112,17 @@ Storing preprocessed documents optimized for searching. For example, each document can be represented as a sorted array of normalized lexemes. Along with the lexemes it is - desirable to store positional information to use for proximity - ranking, so that a document that contains a more - dense region of query words is + often desirable to store positional information to use for + proximity ranking, so that a document that + contains a more dense region of query words is assigned a higher rank than one with scattered query words. - Dictionaries allow fine-grained control over how lexemes are normalized. - With dictionaries you can: + Dictionaries allow fine-grained control over how tokens are normalized. + With appropriate dictionaries, you can: @@ -155,9 +162,11 @@ A data type tsvector is provided for storing preprocessed documents, along with a type tsquery for representing processed - queries (). Also, a full text search - operator @@ is defined for these data types (). Full text searches can be accelerated + queries (). There are many + functions and operators available for these data types + (), the most important of which is + the match operator @@, which we introduce in + . Full text searches can be accelerated using indexes (). @@ -221,27 +230,13 @@ WHERE mid = did AND mid = 12; Full text searching in PostgreSQL is based on - the operator @@, which tests whether a tsvector - (document) matches a tsquery (query). Also, this operator - supports text input, allowing explicit conversion of a text - string to tsvector to be skipped. The variants available - are: + the match operator @@, which returns + true if a tsvector + (document) matches a tsquery (query). + It doesn't matter which data type is written first: -tsvector @@ tsquery -tsquery @@ tsvector -text @@ tsquery -text @@ text - - - - - The match operator @@ returns true if - the tsvector matches the tsquery. It doesn't - matter which data type is written first: - - -SELECT 'cat & rat'::tsquery @@ 'a fat cat sat on a mat and ate a fat rat'::tsvector; +SELECT 'a fat cat sat on a mat and ate a fat rat'::tsvector @@ 'cat & rat'::tsquery; ?column? ---------- t @@ -254,12 +249,59 @@ SELECT 'fat & cow'::tsquery @@ 'a fat cat sat on a mat and ate a fat rat'::t + As the above example suggests, a tsquery is not just raw + text, any more than a tsvector is. A tsquery + contains search terms, which must be already-normalized lexemes, and may + contain AND, OR, and NOT operators. + (For details see .) There are + functions to_tsquery and plainto_tsquery + that are helpful in converting user-written text into a proper + tsquery, for example by normalizing words appearing in + the text. Similarly, to_tsvector is used to parse and + normalize a document string. So in practice a text search match would + look more like this: + + +SELECT to_tsvector('fat cats ate fat rats') @@ to_tsquery('fat & rat'); + ?column? +---------- + t + + + Observe that this match would not succeed if written as + + +SELECT 'fat cats ate fat rats'::tsvector @@ to_tsquery('fat & rat'); + ?column? +---------- + f + + + since here no normalization of the word rats will occur: + the elements of a tsvector are lexemes, which are assumed + already normalized. + + + + The @@ operator also + supports text input, allowing explicit conversion of a text + string to tsvector or tsquery to be skipped + in simple cases. The variants available are: + + +tsvector @@ tsquery +tsquery @@ tsvector +text @@ tsquery +text @@ text + + + + + The first two of these we saw already. The form text @@ tsquery is equivalent to to_tsvector(x) @@ y. The form text @@ text is equivalent to to_tsvector(x) @@ plainto_tsquery(y). - contains a complete list of full text - search functions and operators. @@ -305,14 +347,14 @@ SELECT 'fat & cow'::tsquery @@ 'a fat cat sat on a mat and ate a fat rat'::t - Text search parsers break documents into lexemes - and classify each lexeme (for example, as words or numbers). + Text search parsers break documents into tokens + and classify each token (for example, as words or numbers). - Text search dictionaries convert lexemes to normalized + Text search dictionaries convert tokens to normalized form and reject stop words. @@ -328,7 +370,7 @@ SELECT 'fat & cow'::tsquery @@ 'a fat cat sat on a mat and ate a fat rat'::t Text search configurations specify a parser and a set - of dictionaries to use to normalize the lexemes produced by the parser. + of dictionaries to use to normalize the tokens produced by the parser. @@ -401,6 +443,13 @@ ORDER BY dlm DESC LIMIT 10; in one of the two fields. + + Although these queries will work without an index, most applications + will find this approach too slow, except perhaps for occasional ad-hoc + queries. Practical use of text searching usually requires creating + an index. + + @@ -408,7 +457,7 @@ ORDER BY dlm DESC LIMIT 10; We can create a GIN index () to speed up the search: + linkend="textsearch-indexes">) to speed up text searches: CREATE INDEX pgweb_idx ON pgweb USING gin(to_tsvector('english', body)); @@ -562,21 +611,26 @@ SELECT to_tsvector('english', 'a fat cat sat on a mat - it ate a fat rats'); The to_tsvector function internally calls a parser - which breaks the document (a fat cat sat on a mat - it ate a - fat rats) into words and corresponding types. The default parser - recognizes 23 types. Each word, depending on its type, passes through a - group of dictionaries (). At the - end of this step we obtain lexemes. For example, + which breaks the document text into tokens and assigns a type to + each token. The default parser recognizes 23 token types. + For each token, a list of + dictionaries () is consulted, + where the list can vary depending on the token type. The first dictionary + that recognizes the token emits one or more normalized + lexemes to represent the token. For example, rats became rat because one of the dictionaries recognized that the word rats is a plural - form of rat. Some words are treated as "stop words" - () and ignored since they occur too - frequently and have little informational value. In our example these are + form of rat. Some words are recognized as stop + words (), which causes them to + be ignored since they occur too frequently to be useful in searching. + In our example these are a, on, and it. - The punctuation sign - was also ignored because its - type (Space symbols) is not indexed. The choice of - parser, dictionaries and what types of lexemes to index is determined by - the selected text search configuration (- + because there are in fact no dictionaries assigned for its token type + (Space symbols), meaning space tokens will never be + indexed. The choices of parser, dictionaries and which types of tokens to + index are determined by the selected text search configuration (). It is possible to have many different configurations in the same database, and predefined configurations are available for various languages. In our example @@ -619,13 +673,18 @@ SELECT * FROM ts_debug('english','a fat cat sat on a mat - it ate a fat rats'); lword | Latin word | rats | {english} | english: {rat} (24 rows) + + A more extensive example of ts_debug output + appears in . - The function setweight() is used to label the entries - of a tsvector with a given weight. The typical - usage of this is to mark entries coming from - different parts of a document, perhaps by importance. Later, this can be + The function setweight() can be used to label the + entries of a tsvector with a given weight, + where a weight is one of the letters A, B, + C, or D. + This is typically used to mark entries coming from + different parts of a document. Later, this information can be used for ranking of search results in addition to positional information (distance between query terms). If no ranking is required, positional information can be removed from tsvector using the @@ -649,12 +708,14 @@ UPDATE tt SET ti = Here we have used setweight() to label the source of each lexeme in the finished tsvector, and then merged - the labeled tsvector values using the concatenation - operator ||. + the labeled tsvector values using the tsvector + concatenation operator ||. - The following functions allow manual parsing control: + The following functions allow manual parsing control. They would + not normally be used during actual text searches, but they are very + useful for debugging purposes: @@ -674,8 +735,8 @@ UPDATE tt SET ti = Parses the given document and returns a series of records, one for each token produced by parsing. Each record - includes a tokid giving its type and a - token which gives its content: + includes a tokid showing the assigned token type + and a token which is the text of the token. SELECT * FROM ts_parse('default','123 - a number'); @@ -705,12 +766,13 @@ SELECT * FROM ts_parse('default','123 - a number'); - Returns a table which describes each kind of token the + Returns a table which describes each type of token the parser can recognize. For each token - type the table gives the tokid which the + type the table gives the integer tokid that the parser uses to label a - token of that type, the alias which - names the token type, and a short description: + token of that type, the alias that + names the token type in configuration commands, + and a short description: SELECT * FROM ts_token_type('default'); @@ -755,8 +817,8 @@ SELECT * FROM ts_token_type('default'); Ranking attempts to measure how relevant documents are to a particular - query by inspecting the number of times each search word appears in the - document, and whether different search terms occur near each other. + query, typically by checking the number of times each search term appears + in the document and whether the search terms occur near each other. PostgreSQL provides two predefined ranking functions, which take into account lexical, proximity, and structural information. However, the concept of @@ -792,7 +854,8 @@ SELECT * FROM ts_token_type('default'); - This ranking function offers the ability to weigh word instances more + The optional weights + argument offers the ability to weigh word instances more or less heavily depending on how you have classified them. The weights specify how heavily to weigh each category of word: @@ -1088,7 +1151,8 @@ ORDER BY rank DESC LIMIT 10) AS foo; Dictionaries are used to eliminate words that should not be considered in a search (stop words), and to normalize words so - that different derived forms of the same word will match. Aside from + that different derived forms of the same word will match. A successfully + normalized word is called a lexeme. Aside from improving search quality, normalization and removal of stop words reduce the size of the tsvector representation of a document, thereby improving performance. Normalization does not always have linguistic meaning @@ -1108,7 +1172,8 @@ ORDER BY rank DESC LIMIT 10) AS foo; - Identical URL locations are identified and canonicalized: + URL locations can be canonicalized to make + equivalent URLs match: @@ -1131,14 +1196,15 @@ ORDER BY rank DESC LIMIT 10) AS foo; - Color names are substituted by their hexadecimal values, e.g., + Color names can be replaced by their hexadecimal values, e.g., red, green, blue, magenta -> FF0000, 00FF00, 0000FF, FF00FF - Remove some numeric fractional digits to reduce the range of possible - numbers, so 3.14159265359, + If indexing numbers, we can + remove some fractional digits to reduce the range of possible + numbers, so for example 3.14159265359, 3.1415926, 3.14 will be the same after normalization if only two digits are kept after the decimal point. @@ -1148,22 +1214,23 @@ ORDER BY rank DESC LIMIT 10) AS foo; - A dictionary is a program that accepts lexemes as + A dictionary is a program that accepts a token as input and returns: - an array of lexemes if the input lexeme is known to the dictionary + an array of lexemes if the input token is known to the dictionary + (notice that one token can produce more than one lexeme) - an empty array if the dictionary knows the lexeme, but it is a stop word + an empty array if the dictionary knows the token, but it is a stop word - NULL if the dictionary does not recognize the input lexeme + NULL if the dictionary does not recognize the input token @@ -1180,14 +1247,14 @@ ORDER BY rank DESC LIMIT 10) AS foo; A text search configuration binds a parser together with a set of - dictionaries to process the parser's output lexemes. For each token - type that the parser can return, a separate stack of dictionaries is - specified by the configuration. When a lexeme of that type is found - by the parser, each dictionary in the stack is consulted in turn, + dictionaries to process the parser's output tokens. For each token + type that the parser can return, a separate list of dictionaries is + specified by the configuration. When a token of that type is found + by the parser, each dictionary in the list is consulted in turn, until some dictionary recognizes it as a known word. If it is identified - as a stop word, or if no dictionary recognizes the lexeme, it will be + as a stop word, or if no dictionary recognizes the token, it will be discarded and not indexed or searched for. - The general rule for configuring a stack of dictionaries + The general rule for configuring a list of dictionaries is to place first the most narrow, most specific dictionary, then the more general dictionaries, finishing with a very general dictionary, like a Snowball stemmer or simple, which @@ -1246,11 +1313,22 @@ SELECT ts_rank_cd ('{1,1,1,1}', to_tsvector('english','list stop words'), to_tsq behavior is an attempt to decrease noise. + + + + Simple Dictionary + - Here is an example of a dictionary that returns the input word as lowercase - or NULL if it is a stop word; it also specifies the name - of a file of stop words. It uses the simple dictionary as - a template: + The simple dictionary template operates by converting the + input token to lower case and checking it against a file of stop words. + If it is found in the file then NULL is returned, causing + the token to be discarded. If not, the lower-cased form of the word + is returned as the normalized lexeme. + + + + Here is an example of a dictionary definition using the simple + template: CREATE TEXT SEARCH DICTIONARY public.simple_dict ( @@ -1259,6 +1337,20 @@ CREATE TEXT SEARCH DICTIONARY public.simple_dict ( ); + Here, english is the base name of a file of stop words. + The file's full name will be + $SHAREDIR/tsearch_data/english.stop, + where $SHAREDIR means the + PostgreSQL installation's shared-data directory, + often /usr/local/share/postgresql (use pg_config + --sharedir to determine it if you're not sure). + The file format is simply a list + of words, one per line. Blank lines and trailing spaces are ignored, + and upper case is folded to lower case, but no other processing is done + on the file contents. + + + Now we can test our dictionary: @@ -1276,10 +1368,21 @@ SELECT ts_lexize('public.simple_dict','The'); - Most types of dictionaries rely on configuration files, such as files of stop - words. These files must be stored in UTF-8 encoding. They will - be translated to the actual database encoding, if that is different, when they - are read into the server. + Most types of dictionaries rely on configuration files, such as files of + stop words. These files must be stored in UTF-8 encoding. + They will be translated to the actual database encoding, if that is + different, when they are read into the server. + + + + + + Normally, a database session will read a dictionary configuration file + only once, when it is first used within the session. If you modify a + configuration file and want to force existing sessions to pick up the + new contents, issue an ALTER TEXT SEARCH DICTIONARY command + on the dictionary. This can be a dummy update that doesn't + actually change any parameter values. @@ -1291,7 +1394,7 @@ SELECT ts_lexize('public.simple_dict','The'); This dictionary template is used to create dictionaries that replace a word with a synonym. Phrases are not supported (use the thesaurus - dictionary () for that). A synonym + template () for that). A synonym dictionary can be used to overcome linguistic problems, for example, to prevent an English stemmer dictionary from reducing the word 'Paris' to 'pari'. It is enough to have a Paris paris line in the @@ -1304,8 +1407,10 @@ SELECT * FROM ts_debug('english','Paris'); lword | Latin word | Paris | {english_stem} | english_stem: {pari} (1 row) -CREATE TEXT SEARCH DICTIONARY synonym - (TEMPLATE = synonym, SYNONYMS = my_synonyms); +CREATE TEXT SEARCH DICTIONARY synonym ( + TEMPLATE = synonym, + SYNONYMS = my_synonyms +); ALTER TEXT SEARCH CONFIGURATION english ALTER MAPPING FOR lword WITH synonym, english_stem; @@ -1318,6 +1423,20 @@ SELECT * FROM ts_debug('english','Paris'); + + The only parameter required by the synonym template is + SYNONYMS, which is the base name of its configuration file + — my_synonyms in the above example. + The file's full name will be + $SHAREDIR/tsearch_data/my_synonyms.syn + (where $SHAREDIR means the + PostgreSQL installation's shared-data directory). + The file format is just one line + per word to be substituted, with the word followed by its synonym, + separated by white space. Blank lines and trailing spaces are ignored, + and upper case is folded to lower case. + + @@ -1333,11 +1452,10 @@ SELECT * FROM ts_debug('english','Paris'); Basically a thesaurus dictionary replaces all non-preferred terms by one - preferred term and, optionally, preserves them for indexing. Thesauruses - are used during indexing so any change in the thesaurus requires - reindexing. The current implementation of the thesaurus - dictionary is an extension of the synonym dictionary with added - phrase support. A thesaurus dictionary requires + preferred term and, optionally, preserves the original terms for indexing + as well. PostgreSQL's current implementation of the + thesaurus dictionary is an extension of the synonym dictionary with added + phrase support. A thesaurus dictionary requires a configuration file of the following format: @@ -1352,7 +1470,7 @@ more sample word(s) : more indexed word(s) - A thesaurus dictionary uses a subdictionary (which + A thesaurus dictionary uses a subdictionary (which is defined in the dictionary's configuration) to normalize the input text before checking for phrase matches. It is only possible to select one subdictionary. An error is reported if the subdictionary fails to @@ -1367,8 +1485,8 @@ more sample word(s) : more indexed word(s) - Stop words recognized by the subdictionary are replaced by a 'stop word - placeholder' to record their position. To break possible ties the thesaurus + Stop words recognized by the subdictionary are replaced by a stop word + placeholder to record their position. To break possible ties the thesaurus uses the last definition. To illustrate this, consider a thesaurus (with a simple subdictionary) with pattern swsw, where s designates any stop word and @@ -1391,19 +1509,26 @@ the one a two : swsw2 uses these assignments to check if it should handle the next word or stop accumulation. The thesaurus dictionary must be configured carefully. For example, if the thesaurus dictionary is assigned to handle - only the lword lexeme, then a thesaurus dictionary - definition like ' one 7' will not work since lexeme type - uint is not assigned to the thesaurus dictionary. + only the lword token, then a thesaurus dictionary + definition like ' one 7' will not work since token type + uint is not assigned to the thesaurus dictionary. - + + + Thesauruses are used during indexing so any change in the thesaurus + dictionary's parameters requires reindexing. + For most other dictionary types, small changes such as adding or + removing stopwords does not force reindexing. + + - + Thesaurus Configuration - To define a new thesaurus dictionary one can use the thesaurus template. - For example: + To define a new thesaurus dictionary, use the thesaurus + template. For example: CREATE TEXT SEARCH DICTIONARY thesaurus_simple ( @@ -1417,29 +1542,30 @@ CREATE TEXT SEARCH DICTIONARY thesaurus_simple ( - thesaurus_simple is the thesaurus dictionary name + thesaurus_simple is the new dictionary's name - mythesaurus is the base name of the thesaurus file - (its full name will be $SHAREDIR/tsearch_data/mythesaurus.ths, - where $SHAREDIR means the installation shared-data directory, - often /usr/local/share). + mythesaurus is the base name of the thesaurus + configuration file. + (Its full name will be $SHAREDIR/tsearch_data/mythesaurus.ths, + where $SHAREDIR means the installation shared-data + directory.) - pg_catalog.english_stem is the dictionary (Snowball - English stemmer) to use for thesaurus normalization. Notice that the - english_stem dictionary has its own configuration (for example, - stop words), which is not shown here. + pg_catalog.english_stem is the subdictionary (here, + a Snowball English stemmer) to use for thesaurus normalization. + Notice that the subdictionary will have its own + configuration (for example, stop words), which is not shown here. Now it is possible to bind the thesaurus dictionary thesaurus_simple - and selected tokens, for example: + to the desired token types, for example: ALTER TEXT SEARCH CONFIGURATION russian @@ -1447,9 +1573,9 @@ ALTER TEXT SEARCH CONFIGURATION russian - + - + Thesaurus Example @@ -1475,11 +1601,11 @@ ALTER TEXT SEARCH CONFIGURATION russian ADD MAPPING FOR lword, lhword, lpart_hword WITH thesaurus_astro, english_stem; - Now we can see how it works. Note that ts_lexize cannot - be used for testing the thesaurus (see description of - ts_lexize), but we can use + Now we can see how it works. + ts_lexize is not very useful for testing a thesaurus, + because it treats its input as a single token. Instead we can use plainto_tsquery and to_tsvector - which accept text arguments, not lexemes: + which will break their input strings into multiple tokens: SELECT plainto_tsquery('supernova star'); @@ -1504,13 +1630,14 @@ SELECT to_tsquery('''supernova star'''); Notice that supernova star matches supernovae - stars in thesaurus_astro because we specified the - english_stem stemmer in the thesaurus definition. + stars in thesaurus_astro because we specified + the english_stem stemmer in the thesaurus definition. + The stemmer removed the e. - To keep an original phrase in full text indexing just add it to the right part - of the definition: + To index the original phrase as well as the substitute, just include it + in the right-hand part of the definition: supernovae stars : sn supernovae stars @@ -1522,18 +1649,29 @@ SELECT plainto_tsquery('supernova star'); + + - Ispell Dictionary + <application>Ispell</> Dictionary - The Ispell template dictionary for full text allows the - creation of morphological dictionaries based on Ispell, which - supports a large number of languages. This dictionary tries to change an - input word to its normalized form. Also, more modern spelling dictionaries - are supported - Ispell dictionary template supports + morphological dictionaries, which can normalize many + different linguistic forms of a word into the same lexeme. For example, + an English Ispell dictionary can match all declensions and + conjugations of the search term bank, e.g. + banking, banked, banks, + banks', and bank's. + + + + The standard PostgreSQL distribution does + not include any Ispell configuration files. + Dictionaries for a large number of languages are available from Ispell. + Also, some more modern dictionary file formats are supported — MySpell (OO < 2.0.1) and Hunspell (OO >= 2.0.2). A large list of dictionaries is available on the - The Ispell dictionary allows searches without bothering - about different linguistic forms of a word. For example, a search on - bank would return hits of all declensions and - conjugations of the search term bank, e.g. - banking, banked, banks, - banks', and bank's. - - -SELECT ts_lexize('english_ispell','banking'); - ts_lexize ------------ - {bank} - -SELECT ts_lexize('english_ispell','bank''s'); - ts_lexize ------------ - {bank} - -SELECT ts_lexize('english_ispell','banked'); - ts_lexize ------------ - {bank} - - - - - - To create an ispell dictionary one should use the built-in - ispell template and specify several - parameters: + To create an Ispell dictionary, use the built-in + ispell template and specify several parameters: @@ -1585,19 +1695,22 @@ CREATE TEXT SEARCH DICTIONARY english_ispell ( Here, DictFile, AffFile, and StopWords - specify the names of the dictionary, affixes, and stop-words files. + specify the base names of the dictionary, affixes, and stop-words files. + The stop-words file has the same format explained above for the + simple dictionary type. The format of the other files is + not specified here but is available from the above-mentioned web sites. - Ispell dictionaries usually recognize a restricted set of words so they - should be used in conjunction with another broader dictionary; for - example, a stemming dictionary, which recognizes everything. + Ispell dictionaries usually recognize a limited set of words, so they + should be followed by another broader dictionary; for + example, a Snowball dictionary, which recognizes everything. - Ispell dictionaries support splitting compound words based on an - ispell dictionary. This is a nice feature and full text searching - in PostgreSQL supports it. + Ispell dictionaries support splitting compound words. + This is a nice feature and + PostgreSQL supports it. Notice that the affix file should specify a special flag using the compoundwords controlled statement that marks dictionary words that can participate in compound formation: @@ -1606,7 +1719,7 @@ CREATE TEXT SEARCH DICTIONARY english_ispell ( compoundwords controlled z - Several examples for the Norwegian language: + Here are some examples for the Norwegian language: SELECT ts_lexize('norwegian_ispell','overbuljongterningpakkmesterassistent'); @@ -1620,15 +1733,15 @@ SELECT ts_lexize('norwegian_ispell','sjokoladefabrikk'); MySpell does not support compound words. Hunspell has sophisticated support for compound words. At - present, full text searching implements only the basic compound word - operations of Hunspell. + present, PostgreSQL implements only the basic + compound word operations of Hunspell. - - <application>Snowball</> Stemming Dictionary + + <application>Snowball</> Dictionary The Snowball dictionary template is based on the project @@ -1637,24 +1750,29 @@ SELECT ts_lexize('norwegian_ispell','sjokoladefabrikk'); many languages (see the Snowball site for more information). Each algorithm understands how to reduce common variant forms of words to a base, or stem, spelling within - its language. A Snowball dictionary requires a language parameter to - identify which stemmer to use, and optionally can specify a stopword file - name that gives a list of words to eliminate. + its language. A Snowball dictionary requires a language + parameter to identify which stemmer to use, and optionally can specify a + stopword file name that gives a list of words to eliminate. (PostgreSQL's standard stopword lists are also provided by the Snowball project.) For example, there is a built-in definition equivalent to CREATE TEXT SEARCH DICTIONARY english_stem ( - TEMPLATE = snowball, Language = english, StopWords = english + TEMPLATE = snowball, + Language = english, + StopWords = english ); + + The stopword file format is the same as already explained. - The Snowball dictionary recognizes everything, so it is best - to place it at the end of the dictionary stack. It it useless to have it - before any other dictionary because a lexeme will never pass through it to + A Snowball dictionary recognizes everything, whether + or not it is able to simplify the word, so it should be placed + at the end of the dictionary list. It it useless to have it + before any other dictionary because a token will never pass through it to the next dictionary. @@ -1676,15 +1794,15 @@ CREATE TEXT SEARCH DICTIONARY english_stem ( - ts_lexize(dict_name text, lexeme text) returns text[] + ts_lexize(dict_name text, token text) returns text[] Returns an array of lexemes if the input - lexeme is known to the dictionary - dict_name, or an empty array if the lexeme + token is known to the dictionary + dict_name, or an empty array if the token is known to the dictionary but it is a stop word, or NULL if it is an unknown word. @@ -1709,7 +1827,7 @@ SELECT ts_lexize('english_stem', 'a'); The ts_lexize function expects a - lexeme, not text. Below is an example: + token, not text. Below is an example: SELECT ts_lexize('thesaurus_astro','supernovae stars') is null; @@ -1720,7 +1838,7 @@ SELECT ts_lexize('thesaurus_astro','supernovae stars') is null; The thesaurus dictionary thesaurus_astro does know supernovae stars, but ts_lexize fails since it - does not parse the input text and considers it as a single lexeme. Use + does not parse the input text and considers it as a single token. Use plainto_tsquery and to_tsvector to test thesaurus dictionaries: @@ -1745,36 +1863,26 @@ SELECT plainto_tsquery('supernovae stars'); A text search configuration specifies all options necessary to transform a - document into a tsvector: the parser breaks text into tokens, - and the dictionaries transform each token into a lexeme. Every call to - to_tsvector() and to_tsquery() - needs a configuration to perform its processing. To facilitate management - of text search objects, a set of SQL commands - is available, and there are several psql commands that display information - about text search objects (). - - - + document into a tsvector: the parser to use to break text + into tokens, and the dictionaries to use to transform each token into a + lexeme. Every call of + to_tsvector() or to_tsquery() + needs a text search configuration to perform its processing. The configuration parameter specifies the name of the current default configuration, which is the - one used by text search functions when an explicit configuration + one used by text search functions if an explicit configuration parameter is omitted. It can be set in postgresql.conf, or set for an individual session using the SET command. - Several predefined text search configurations are available in the - pg_catalog schema. If you need a custom configuration - you can create a new text search configuration and modify it using SQL - commands. - - - - New text search objects are created in the current schema by default - (usually the public schema), but a schema-qualified - name can be used to create objects in the specified schema. + Several predefined text search configurations are available, and + you can create custom configurations easily. To facilitate management + of text search objects, a set of SQL commands + is available, and there are several psql commands that display information + about text search objects (). @@ -1791,7 +1899,7 @@ CREATE TEXT SEARCH CONFIGURATION public.pg ( COPY = english ); We will use a PostgreSQL-specific synonym list - and store it in share/tsearch_data/pg_dict.syn. + and store it in $SHAREDIR/tsearch_data/pg_dict.syn. The file contents look like: @@ -1809,11 +1917,8 @@ CREATE TEXT SEARCH DICTIONARY pg_dict ( ); - - - - Then register the ispell dictionary - english_ispell using the ispell template: + Next we register the ispell dictionary + english_ispell: CREATE TEXT SEARCH DICTIONARY english_ispell ( @@ -1823,20 +1928,16 @@ CREATE TEXT SEARCH DICTIONARY english_ispell ( StopWords = english ); - - - Now modify mappings for Latin words for configuration pg: + Now modify the mappings for Latin words for configuration pg: ALTER TEXT SEARCH CONFIGURATION pg ALTER MAPPING FOR lword, lhword, lpart_hword WITH pg_dict, english_ispell, english_stem; - - - We do not index or search some tokens: + We do not index or search some token types: ALTER TEXT SEARCH CONFIGURATION pg @@ -1894,8 +1995,10 @@ SHOW default_text_search_config; There are two kinds of indexes that can be used to speed up full text - operators (). - Note that indexes are not mandatory for full text searching. + searches. + Note that indexes are not mandatory for full text searching, but in + cases where a column is searched on a regular basis, an index will + usually be desirable. @@ -2072,7 +2175,7 @@ EXPLAIN SELECT * FROM apod WHERE textsearch @@@ to_tsquery('supernovae:a'); the indexes are faster for lookups. For dynamic data, GiST indexes are faster to update. Specifically, GiST indexes are very good for dynamic data and fast if the number of unique words (lexemes) is - under 100,000, while GIN handles +100,000 lexemes better + under 100,000, while GIN handles 100,000+ lexemes better but is slower to update. @@ -2226,14 +2329,14 @@ Parser: "pg_catalog.default" ------------+---------+--------------------- pg_catalog | default | default word parser => \dFp+ - Text search parser "pg_catalog.default" - Method | Function | Description -------------------+----------------+------------- - Start parse | prsd_start | - Get next token | prsd_nexttoken | - End parse | prsd_end | - Get headline | prsd_headline | - Get lexeme types | prsd_lextype | + Text search parser "pg_catalog.default" + Method | Function | Description +-----------------+----------------+------------- + Start parse | prsd_start | + Get next token | prsd_nexttoken | + End parse | prsd_end | + Get headline | prsd_headline | + Get token types | prsd_lextype | Token types for parser "pg_catalog.default" Token name | Description @@ -2303,22 +2406,22 @@ Parser: "pg_catalog.default" text search features are: - The length of each lexeme must be less than 2K bytes + The length of each lexeme must be less than 2K bytes - The length of a tsvector (lexemes + positions) must be less than 1 megabyte + The length of a tsvector (lexemes + positions) must be less than 1 megabyte - The number of lexemes must be less than 264 + The number of lexemes must be less than 264 - Positional information must be greater than 0 and less than 16,383 + Positional information must be greater than 0 and less than 16,383 - No more than 256 positions per lexeme + No more than 256 positions per lexeme - The number of nodes (lexemes + operations) in tsquery must be less than 32,768 + The number of nodes (lexemes + operations) in a tsquery must be less than 32,768 @@ -2406,15 +2509,16 @@ SELECT * FROM ts_debug('public.english','The Brightest supernovaes'); In this example, the word Brightest was recognized by the parser as a Latin word (alias lword). - For this token type the dictionary stack is + For this token type the dictionary list is public.english_ispell and pg_catalog.english_stem. The word was recognized by public.english_ispell, which reduced it to the noun bright. The word supernovaes is unknown to the public.english_ispell dictionary so it was passed to the next dictionary, and, fortunately, was recognized (in fact, - public.english_stem is a stemming dictionary and recognizes - everything; that is why it was placed at the end of the dictionary stack). + public.english_stem is a Snowball dictionary which + recognizes everything; that is why it was placed at the end of the + dictionary list).