2006-11-01 00:23:05 +08:00
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
< html > < head >
< title > tsearch2 reference< / title > < / head >
2003-08-04 22:11:08 +08:00
2003-07-21 18:27:44 +08:00
< body >
2003-08-04 22:11:08 +08:00
< h1 align = "center" > The tsearch2 Reference< / h1 >
2003-07-21 18:27:44 +08:00
2003-08-04 22:11:08 +08:00
< p align = "center" >
Brandon Craig Rhodes< br > 30 June 2003 (edited by Oleg Bartunov, 2 Aug 2003).
2006-11-01 00:23:05 +08:00
< br > Massive update for 8.2 release by Oleg Bartunov, October 2006
< / p >
< p >
2003-07-21 18:27:44 +08:00
This Reference documents the user types and functions
of the tsearch2 module for PostgreSQL.
An introduction to the module is provided
2006-11-01 00:23:05 +08:00
by the < a href = "http://www.sai.msu.su/~megera/postgres/gist/tsearch/V2/docs/tsearch2-guide.html" > tsearch2 Guide< / a > ,
2003-07-21 18:27:44 +08:00
a companion document to this one.
2006-11-01 00:23:05 +08:00
< / p >
< h2 > Table of Contents< / h2 >
< blockquote >
< a href = "#vq" > Vectors and Queries< / a > < br >
< a href = "#vqo" > Vector Operations< / a > < br >
< a href = "#qo" > Query Operations< / a > < br >
< a href = "#fts" > Full Text Search Operator< / a > < br >
< a href = "#configurations" > Configurations< / a > < br >
< a href = "#testing" > Testing< / a > < br >
< a href = "#parsers" > Parsers< / a > < br >
< a href = "#dictionaries" > Dictionaries< / a > < br >
< a href = "#ranking" > Ranking< / a > < br >
< a href = "#headlines" > Headlines< / a > < br >
< a href = "#indexes" > Indexes< / a > < br >
< a href = "#tz" > Thesaurus dictionary< / a > < br >
< / blockquote >
2003-07-21 18:27:44 +08:00
2006-11-01 00:23:05 +08:00
< h2 > < a name = "vq" > Vectors and Queries< / a > < / h2 >
Vectors and queries both store lexemes,
2003-07-21 18:27:44 +08:00
but for different purposes.
A < tt > tsvector< / tt > stores the lexemes
of the words that are parsed out of a document,
and can also remember the position of each word.
A < tt > tsquery< / tt > specifies a boolean condition among lexemes.
2006-11-01 00:23:05 +08:00
< p >
Any of the following functions with a < tt > < i > configuration< / i > < / tt > argument
2003-07-21 18:27:44 +08:00
can use either an integer < tt > id< / tt > or textual < tt > ts_name< / tt >
to select a configuration;
if the option is omitted, then the current configuration is used.
For more information on the current configuration,
read the next section on Configurations.
2006-11-01 00:23:05 +08:00
< / p >
2003-07-21 18:27:44 +08:00
2006-11-01 00:23:05 +08:00
< h3 > < a name = "vqo" > Vector Operations< / a > < / h3 >
2003-07-21 18:27:44 +08:00
2003-08-04 22:11:08 +08:00
< dl > < dt >
2006-11-01 00:23:05 +08:00
< tt > to_tsvector( < em > [< / em > < i > configuration< / i > ,< em > ]< / em >
< i > document< / i > TEXT) RETURNS TSVECTOR< / tt >
< / dt > < dd >
Parses a document into tokens,
2003-07-21 18:27:44 +08:00
reduces the tokens to lexemes,
and returns a < tt > tsvector< / tt > which lists the lexemes
together with their positions in the document.
For the best description of this process,
2006-11-01 00:23:05 +08:00
see the section on < a href = "http://www.sai.msu.su/%7Emegera/postgres/gist/tsearch/V2/docs/tsearch2-guide.html#ps" > Parsing and Stemming< / a >
2003-07-21 18:27:44 +08:00
in the accompanying tsearch2 Guide.
2003-08-04 22:11:08 +08:00
< / dd > < dt >
2006-11-01 00:23:05 +08:00
< tt > strip(< i > vector< / i > TSVECTOR) RETURNS TSVECTOR< / tt >
2003-08-04 22:11:08 +08:00
< / dt > < dd >
2003-07-21 18:27:44 +08:00
Return a vector which lists the same lexemes
as the given < tt > < i > vector< / i > < / tt > ,
but which lacks any information
about where in the document each lexeme appeared.
While the returned vector is thus useless for relevance ranking,
it will usually be much smaller.
2003-08-04 22:11:08 +08:00
< / dd > < dt >
2006-11-01 00:23:05 +08:00
< tt > setweight(< i > vector< / i > TSVECTOR, < i > letter< / i > ) RETURNS TSVECTOR< / tt >
2003-08-04 22:11:08 +08:00
< / dt > < dd >
2003-07-21 18:27:44 +08:00
This function returns a copy of the input vector
2006-11-01 00:23:05 +08:00
in which every location has been labeled
2003-07-21 18:27:44 +08:00
with either the < tt > < i > letter< / i > < / tt >
< tt > 'A'< / tt > , < tt > 'B'< / tt > , or < tt > 'C'< / tt > ,
or the default label < tt > 'D'< / tt >
(which is the default with which new vectors are created,
and as such is usually not displayed).
These labels are retained when vectors are concatenated,
allowing words from different parts of a document
to be weighted differently by ranking functions.
2006-11-01 00:23:05 +08:00
< / dd >
< dt >
< tt > < i > vector1< / i > || < i > vector2< / i > < / tt > < BR >
< tt > concat(< i > vector1< / i > TSVECTOR, < i > vector2< / i > TSVECTOR)
RETURNS TSVECTOR< / tt >
2003-08-04 22:11:08 +08:00
< / dt > < dd >
2003-07-21 18:27:44 +08:00
Returns a vector which combines the lexemes and position information
in the two vectors given as arguments.
Position weight labels (described in the previous paragraph)
are retained intact during the concatenation.
This has at least two uses.
First,
if some sections of your document
need be parsed with different configurations than others,
you can parse them separately
and concatenate the resulting vectors into one.
Second,
you can weight words from some sections of you document
more heavily than those from others by:
parsing the sections into separate vectors;
assigning the vectors different position labels
with the < tt > setweight()< / tt > function;
concatenating them into a single vector;
and then providing a < tt > < i > weights< / i > < / tt > argument
to the < tt > rank()< / tt > function
that assigns different weights to positions with different labels.
2003-08-04 22:11:08 +08:00
< / dd > < dt >
2006-11-01 00:23:05 +08:00
< tt > length(< i > vector< / i > TSVECTOR) RETURNS INT4< / tt >
2003-08-04 22:11:08 +08:00
< / dt > < dd >
2003-07-21 18:27:44 +08:00
Returns the number of lexemes stored in the vector.
2003-08-04 22:11:08 +08:00
< / dd > < dt >
2006-11-01 00:23:05 +08:00
< tt > < i > text< / i > ::TSVECTOR RETURNS TSVECTOR< / tt >
2003-08-04 22:11:08 +08:00
< / dt > < dd >
2003-07-21 18:27:44 +08:00
Directly casting text to a < tt > tsvector< / tt >
allows you to directly inject lexemes into a vector,
with whatever positions and position weights you choose to specify.
The < tt > < i > text< / i > < / tt > should be formatted
like the vector would be printed by the output of a < tt > SELECT< / tt > .
2006-11-01 00:23:05 +08:00
See the < a href = "http://www.sai.msu.su/~megera/postgres/gist/tsearch/V2/docs/tsearch2-guide.html#casting" > Casting< / a >
2003-07-21 18:27:44 +08:00
section in the Guide for details.
2006-11-01 00:23:05 +08:00
< / dd > < dt >
< tt > tsearch2(< i > vector_column_name< / i > [, (< i > my_filter_name< / i > | < i > text_column_name1< / i > ) [...] ], < i > text_column_nameN< / i > )< / tt >
< / dt > < dd >
< tt > tsearch2()< / tt > trigger used to automatically update < i > vector_column_name< / i > , < i > my_filter_name< / i >
is the function name to preprocess < i > text_column_name< / i > . There are can be many
functions and text columns specified in < tt > tsearch2()< / tt > trigger.
The following rule used:
function applied to all subsequent text columns until next function occurs.
Example, function < tt > dropatsymbol< / tt > replaces all entries of < tt > @< / tt >
sign by space.
< pre >
CREATE FUNCTION dropatsymbol(text) RETURNS text
AS 'select replace($1, ''@'', '' '');'
LANGUAGE SQL;
CREATE TRIGGER tsvectorupdate BEFORE UPDATE OR INSERT
ON tblMessages FOR EACH ROW EXECUTE PROCEDURE
tsearch2(tsvector_column,dropatsymbol, strMessage);
< / pre >
< / dd >
2003-07-21 18:27:44 +08:00
2006-11-01 00:23:05 +08:00
< dt >
< tt > stat(< i > sqlquery< / i > text [, < i > weight< / i > text]) RETURNS SETOF statinfo< / tt >
< / dt > < dd >
Here < tt > statinfo< / tt > is a type, defined as
< tt >
CREATE TYPE statinfo as (< i > word< / i > text, < i > ndoc< / i > int4, < i > nentry< / i > int4)
< / tt > and < i > sqlquery< / i > is a query, which returns column < tt > tsvector< / tt > .
< P >
This returns statistics (the number of documents < i > ndoc< / i > and total number < i > nentry< / i > of < i > word< / i >
in the collection) about column < i > vector< / i > < tt > tsvector< / tt > .
Useful to check how good is your configuration and
to find stop-words candidates.For example, find top 10 most frequent words:
< pre >
=# select * from stat('select vector from apod') order by ndoc desc, nentry desc,word limit 10;
< / pre >
Optionally, one can specify < i > weight< / i > to obtain statistics about words with specific weight.
< pre >
=# select * from stat('select vector from apod','a') order by ndoc desc, nentry desc,word limit 10;
< / pre >
2003-07-21 18:27:44 +08:00
2006-11-01 00:23:05 +08:00
< / dd >
< dt >
< tt > TSVECTOR < TSVECTOR< / tt > < BR >
< tt > TSVECTOR < = TSVECTOR< / tt > < BR >
< tt > TSVECTOR = TSVECTOR< / tt > < BR >
< tt > TSVECTOR >= TSVECTOR< / tt > < BR >
< tt > TSVECTOR > TSVECTOR< / tt >
2003-08-04 22:11:08 +08:00
< / dt > < dd >
2006-11-01 00:23:05 +08:00
All btree operations defined for < tt > tsvector< / tt > type. < tt > tsvectors< / tt > compares
with each other using lexicographical order.
< / dd >
< / dl >
< h3 > < a name = "qo" > Query Operations< / a > < / h3 >
< dl >
< dt >
< tt > to_tsquery( < em > [< / em > < i > configuration< / i > ,< em > ]< / em >
< i > querytext< / i > text) RETURNS TSQUERY[A< / tt >
< / dt >
< dd >
2003-07-21 18:27:44 +08:00
Parses a query,
which should be single words separated by the boolean operators
2003-08-04 22:11:08 +08:00
"< tt > & < / tt > " and,
"< tt > |< / tt > " or,
and "< tt > !< / tt > " not,
2003-07-21 18:27:44 +08:00
which can be grouped using parenthesis.
Each word is reduced to a lexeme using the current
2006-11-01 00:23:05 +08:00
or specified configuration.
Weight class can be assigned to each lexeme entry
to restrict search region
(see < tt > setweight< / tt > for explanation), for example
"< tt > fat:a & rats< / tt > ".
< / dd > < dt >
< dt >
< tt > plainto_tsquery( < em > [< / em > < i > configuration< / i > ,< em > ]< / em >
< i > querytext< / i > text) RETURNS TSQUERY< / tt >
< / dt >
< dd >
Transforms unformatted text to tsquery. It is the same as to_tsquery,
but assumes "< tt > & < / tt > " boolean operator between words and doesn't
recognizes weight classes.
2003-08-04 22:11:08 +08:00
< / dd > < dt >
2006-11-01 00:23:05 +08:00
< tt > querytree(< i > query< / i > TSQUERY) RETURNS text< / tt >
2003-08-04 22:11:08 +08:00
< / dt > < dd >
2006-11-01 00:23:05 +08:00
This returns a query which actually used in searching in GiST index.
2003-08-04 22:11:08 +08:00
< / dd > < dt >
2006-11-01 00:23:05 +08:00
< tt > < i > text< / i > ::TSQUERY RETURNS TSQUERY< / tt >
2003-08-04 22:11:08 +08:00
< / dt > < dd >
2003-07-21 18:27:44 +08:00
Directly casting text to a < tt > tsquery< / tt >
allows you to directly inject lexemes into a query,
with whatever positions and position weight flags you choose to specify.
The < tt > < i > text< / i > < / tt > should be formatted
like the query would be printed by the output of a < tt > SELECT< / tt > .
2003-08-04 22:11:08 +08:00
See the < a href = "http://www.sai.msu.su/%7Emegera/postgres/gist/tsearch/V2/docs/tsearch2-guide.html#casting" > Casting< / a >
2003-07-21 18:27:44 +08:00
section in the Guide for details.
2006-11-01 00:23:05 +08:00
< / dd >
< dt >
< tt > numnode(< i > query< / i > TSQUERY) RETURNS INTEGER< / tt >
< / dt > < dd >
This returns the number of nodes in query tree
< / dd > < dt >
< tt > TSQUERY & & TSQUERY RETURNS TSQUERY< / tt >
< / dt > < dd >
AND-ed TSQUERY
< / dd > < dt >
< tt > TSQUERY || TSQUERY RETURNS TSQUERY< / tt >
< / dt > < dd >
OR-ed TSQUERY
< / dd > < dt >
< tt > !! TSQUERY RETURNS TSQUERY< / tt >
< / dt > < dd >
negation of TSQUERY
< / dd >
< dt >
< tt > TSQUERY < TSQUERY< / tt > < BR >
< tt > TSQUERY < = TSQUERY< / tt > < BR >
< tt > TSQUERY = TSQUERY< / tt > < BR >
< tt > TSQUERY >= TSQUERY< / tt > < BR >
< tt > TSQUERY > TSQUERY< / tt >
< / dt > < dd >
All btree operations defined for < tt > tsquery< / tt > type. < tt > tsqueries< / tt > compares
with each other using lexicographical order.
< / dd >
< / dl >
< h3 > Query rewriting< / h3 >
Query rewriting is a set of functions and operators for tsquery type.
It allows to control search at query time without reindexing (opposite to thesaurus), for example,
expand search using synonyms (new york, big apple, nyc, gotham).
< P >
< tt > < b > rewrite()< / b > < / tt > function changes original < i > query< / i > by replacing < i > target< / i > by < i > sample< / i > .
There are three possibilities to use < tt > rewrite()< / tt > function. Notice, that arguments of < tt > rewrite()< / tt >
function can be column names of type < tt > tsquery< / tt > .
< pre >
create table rw (q TSQUERY, t TSQUERY, s TSQUERY);
insert into rw values('a & b','a', 'c');
< / pre >
< dl >
< dt > < tt > rewrite (< i > query< / i > TSQUERY, < i > target< / i > TSQUERY, < i > sample< / i > TSQUERY) RETURNS TSQUERY< / tt >
< / dt >
< dd >
< pre >
=# select rewrite('a & b'::TSQUERY, 'a'::TSQUERY, 'c'::TSQUERY);
rewrite
-----------
'c' & 'b'
< / pre >
< / dd >
< dt > < tt > rewrite (ARRAY[< i > query< / i > TSQUERY, < i > target< / i > TSQUERY, < i > sample< / i > TSQUERY]) RETURNS TSQUERY< / tt >
< / dt >
< dd >
< pre >
=# select rewrite(ARRAY['a & b'::TSQUERY, t,s]) from rw;
rewrite
-----------
'c' & 'b'
< / pre >
< / dd >
< dt > < tt > rewrite (< i > query< / i > TSQUERY,'select < i > target< / i > ,< i > sample< / i > from test'::text) RETURNS TSQUERY< / tt >
< / dt >
< dd >
< pre >
=# select rewrite('a & b'::TSQUERY, 'select t,s from rw'::text);
rewrite
-----------
'c' & 'b'
< / pre >
< / dd >
< / dl >
Two operators defined for < tt > tsquery< / tt > type:
< dl >
< dt > < tt > TSQUERY @ TSQUERY< / tt > < / dt >
< dd >
Returns < tt > TRUE< / tt > if right agrument might contained in left argument.
< / dd >
< dt > < tt > TSQUERY ~ TSQUERY< / tt > < / dt >
< dd >
Returns < tt > TRUE< / tt > if left agrument might contained in right argument.
< / dd >
< / dl >
To speed up these operators one can use GiST index with < tt > gist_tp_tsquery_ops< / tt > opclass.
< pre >
create index qq on test_tsquery using gist (keyword gist_tp_tsquery_ops);
< / pre >
< h2 > < a name = "fts" > Full Text Search operator< / a > < / h2 >
< dl > < dt >
< tt > TSQUERY @@ TSVECTOR< / tt > < br >
< tt > TSVECTOR @@ TSQUERY< / tt >
< / dt >
< dd >
Returns < tt > TRUE< / tt > if < tt > TSQUERY< / tt > contained in < tt > TSVECTOR< / tt > and
< tt > FALSE< / tt > otherwise.
< pre >
=# select 'cat & rat':: tsquery @@ 'a fat cat sat on a mat and ate a fat rat'::tsvector;
?column?
----------
t
=# select 'fat & cow':: tsquery @@ 'a fat cat sat on a mat and ate a fat rat'::tsvector;
?column?
----------
f
< / pre >
< / dd >
< / dl >
2003-07-21 18:27:44 +08:00
< h2 > < a name = "configurations" > Configurations< / a > < / h2 >
A configuration specifies all of the equipment necessary
to transform a document into a < tt > tsvector< / tt > :
the parser that breaks its text into tokens,
and the dictionaries which then transform each token into a lexeme.
2006-11-01 00:23:05 +08:00
Every call to < tt > to_tsvector(), to_tsquery()< / tt > (described above)
2003-07-21 18:27:44 +08:00
uses a configuration to perform its processing.
Three configurations come with tsearch2:
< ul >
2003-08-04 22:11:08 +08:00
< li > < b > default< / b > -- Indexes words and numbers,
2003-07-21 18:27:44 +08:00
using the < i > en_stem< / i > English Snowball stemmer for Latin-alphabet words
and the < i > simple< / i > dictionary for all others.
2003-08-04 22:11:08 +08:00
< / li > < li > < b > default_russian< / b > -- Indexes words and numbers,
2003-07-21 18:27:44 +08:00
using the < i > en_stem< / i > English Snowball stemmer for Latin-alphabet words
2006-11-01 00:23:05 +08:00
and the < i > ru_stem< / i > Russian Snowball dictionary for all others. It's default
for < tt > ru_RU.KOI8-R< / tt > locale.
< / li > < li > < b > utf8_russian< / b > -- the same as < b > default_russian< / b > but
for < tt > ru_RU.UTF-8< / tt > locale.
2003-08-04 22:11:08 +08:00
< / li > < li > < b > simple< / b > -- Processes both words and numbers
2003-07-21 18:27:44 +08:00
with the < i > simple< / i > dictionary,
which neither discards any stop words nor alters them.
2003-08-04 22:11:08 +08:00
< / li > < / ul >
2003-07-21 18:27:44 +08:00
The tsearch2 modules initially chooses your current configuration
by looking for your current locale in the < tt > locale< / tt > field
of the < tt > pg_ts_cfg< / tt > table described below.
You can manipulate the current configuration yourself with these functions:
2003-08-04 22:11:08 +08:00
< dl > < dt >
2003-07-21 18:27:44 +08:00
< tt > set_curcfg( < i > id< / i > INT < em > |< / em > < i > ts_name< / i > TEXT
) RETURNS VOID< / tt >
2003-08-04 22:11:08 +08:00
< / dt > < dd >
2003-07-21 18:27:44 +08:00
Set the current configuration used by < tt > to_tsvector< / tt >
and < tt > to_tsquery< / tt > .
2003-08-04 22:11:08 +08:00
< / dd > < dt >
2003-07-21 18:27:44 +08:00
< tt > show_curcfg() RETURNS INT4< / tt >
2003-08-04 22:11:08 +08:00
< / dt > < dd >
2003-07-21 18:27:44 +08:00
Returns the integer < tt > id< / tt > of the current configuration.
2003-08-04 22:11:08 +08:00
< / dd > < / dl >
2003-07-21 18:27:44 +08:00
< p >
Each configuration is defined by a record in the < tt > pg_ts_cfg< / tt > table:
2003-08-04 22:11:08 +08:00
< / p > < pre > create table pg_ts_cfg (
2003-07-21 18:27:44 +08:00
id int not null primary key,
ts_name text not null,
prs_name text not null,
locale text
);< / pre >
The < tt > id< / tt > and < tt > ts_name< / tt > are unique values
which identify the configuration;
the < tt > prs_name< / tt > specifies which parser the configuration uses.
Once this parser has split document text into tokens,
2003-08-04 22:11:08 +08:00
the type of each resulting token --
or, more specifically, the type's < tt > tok_alias< / tt >
as specified in the parser's < tt > lexem_type()< / tt > table --
2003-07-21 18:27:44 +08:00
is searched for together with the configuration's < tt > ts_name< / tt >
in the < tt > pg_ts_cfgmap< / tt > table:
< pre > create table pg_ts_cfgmap (
ts_name text not null,
2003-08-04 22:11:08 +08:00
tok_alias text not null,
2003-07-21 18:27:44 +08:00
dict_name text[],
2003-08-04 22:11:08 +08:00
primary key (ts_name,tok_alias)
2003-07-21 18:27:44 +08:00
);< / pre >
Those tokens whose types are not listed are discarded.
The remaining tokens are assigned integer positions,
starting with 1 for the first token in the document,
and turned into lexemes with the help of the dictionaries
whose names are given in the < tt > dict_name< / tt > array for their type.
These dictionaries are tried in order,
stopping either with the first one to return a lexeme for the token,
or discarding the token if no dictionary returns a lexeme for it.
2003-08-06 17:41:13 +08:00
< h2 > < a name = "testing" > Testing< / a > < / h2 >
Function < tt > ts_debug< / tt > allows easy testing of your < b > current< / b > configuration.
You may always test another configuration using < tt > set_curcfg< / tt > function.
< p >
Example:
< / p > < pre > apod=# select * from ts_debug('Tsearch module for PostgreSQL 7.3.3');
ts_name | tok_type | description | token | dict_name | tsvector
---------+----------+-------------+------------+-----------+--------------
default | lword | Latin word | Tsearch | {en_stem} | 'tsearch'
default | lword | Latin word | module | {en_stem} | 'modul'
default | lword | Latin word | for | {en_stem} |
default | lword | Latin word | PostgreSQL | {en_stem} | 'postgresql'
default | version | VERSION | 7.3.3 | {simple} | '7.3.3'
< / pre >
Here:
< br >
< ul >
< li > tsname - configuration name
< / li > < li > tok_type - token type
< / li > < li > description - human readable name of tok_type
< / li > < li > token - parser's token
< / li > < li > dict_name - dictionary used for the token
2006-11-01 00:23:05 +08:00
< / li > < li > tsvector - final result< / li >
< / ul >
2003-08-06 17:41:13 +08:00
< h2 > < a name = "parsers" > Parsers< / a > < / h2 >
2003-07-21 18:27:44 +08:00
Each parser is defined by a record in the < tt > pg_ts_parser< / tt > table:
< pre > create table pg_ts_parser (
prs_name text not null,
2004-05-14 18:39:25 +08:00
prs_start regprocedure not null,
prs_nexttoken regprocedure not null,
prs_end regprocedure not null,
prs_headline regprocedure not null,
prs_lextype regprocedure not null,
2003-07-21 18:27:44 +08:00
prs_comment text
);< / pre >
2003-08-04 22:11:08 +08:00
The < tt > prs_name< / tt > uniquely identify the parser,
2003-07-21 18:27:44 +08:00
while < tt > prs_comment< / tt > usually describes its name and version
for the reference of users.
The other items identify the low-level functions
which make the parser operate,
and are only of interest to someone writing a parser of their own.
< p >
The tsearch2 module comes with one parser named < tt > default< / tt >
which is suitable for parsing most plain text and HTML documents.
2003-08-04 22:11:08 +08:00
< / p > < p >
2003-07-21 18:27:44 +08:00
Each < tt > < i > parser< / i > < / tt > argument below
2003-08-04 22:11:08 +08:00
must designate a parser with < tt > < i > prs_name< / i > < / tt > ;
2003-07-21 18:27:44 +08:00
the current parser is used when this argument is omitted.
2003-08-04 22:11:08 +08:00
< / p > < dl > < dt >
2003-07-21 18:27:44 +08:00
< tt > CREATE FUNCTION set_curprs(< i > parser< / i > ) RETURNS VOID< / tt >
2003-08-04 22:11:08 +08:00
< / dt > < dd >
2003-07-21 18:27:44 +08:00
Selects a current parser
which will be used when any of the following functions
are called without a parser as an argument.
2003-08-04 22:11:08 +08:00
< / dd > < dt >
< tt > CREATE FUNCTION token_type(
2003-07-21 18:27:44 +08:00
< em > [< / em > < i > parser< / i > < em > ]< / em >
2003-08-04 22:11:08 +08:00
) RETURNS SETOF tokentype< / tt >
< / dt > < dd >
2003-07-21 18:27:44 +08:00
Returns a table which defines and describes
each kind of token the parser may produce as output.
2003-08-04 22:11:08 +08:00
For each token type the table gives the < tt > tokid< / tt >
2003-07-21 18:27:44 +08:00
which the parser will label each token of that type,
the < tt > alias< / tt > which names the token type,
and a short description < tt > descr< / tt > for the user to read.
2003-08-04 22:11:08 +08:00
< / dd > < dt >
2003-07-21 18:27:44 +08:00
< tt > CREATE FUNCTION parse(
< em > [< / em > < i > parser< / i > , < em > ]< / em > < i > document< / i > TEXT
2003-08-04 22:11:08 +08:00
) RETURNS SETOF tokenout< / tt >
< / dt > < dd >
2003-07-21 18:27:44 +08:00
Parses the given document and returns a series of records,
one for each token produced by parsing.
2003-08-04 22:11:08 +08:00
Each token includes a < tt > tokid< / tt > giving its type
2003-07-21 18:27:44 +08:00
and a < tt > lexem< / tt > which gives its content.
2003-08-04 22:11:08 +08:00
< / dd > < / dl >
2003-07-21 18:27:44 +08:00
< h2 > < a name = "dictionaries" > Dictionaries< / a > < / h2 >
2006-11-01 00:23:05 +08:00
Dictionary is a program, which accepts lexeme(s), usually those produced by a parser,
on input and returns:
< ul >
< li > array of lexeme(s) if input lexeme is known to the dictionary
< li > void array - dictionary knows lexeme, but it's stop word.
< li > NULL - dictionary doesn't recognized input lexeme
< / ul >
Usually, dictionaries used for normalization of words ( ispell, stemmer dictionaries),
but see, for example, < tt > intdict< / tt > dictionary (available from
< a href = "http://www.sai.msu.su/~megera/postgres/gist/tsearch/V2/" > Tsearch2< / a > home page,
which controls indexing of integers.
< P >
2003-07-21 18:27:44 +08:00
Among the dictionaries which come installed with tsearch2 are:
< ul >
< li > < b > simple< / b > simply folds uppercase letters to lowercase
before returning the word.
2006-11-01 00:23:05 +08:00
< / li >
< li > < b > ispell_template< / b > - template for ispell dictionaries.
< / li >
< li > < b > en_stem< / b > runs an English Snowball stemmer on each word
2003-07-21 18:27:44 +08:00
that attempts to reduce the various forms of a verb or noun
to a single recognizable form.
2006-11-01 00:23:05 +08:00
< / li > < li > < b > ru_stem_koi8< / b > , < b > ru_stem_utf8< / b > runs a Russian Snowball stemmer on each word.
< / li >
< li > < b > synonym< / b > - simple lexeme-to-lexeme replacement
< / li >
< li > < b > thesaurus_template< / b > - template for < a href = "#tz" > thesaurus dictionary< / a > . It's
phrase-to-phrase replacement
< / li >
< / ul >
< P >
2003-07-21 18:27:44 +08:00
Each dictionary is defined by an entry in the < tt > pg_ts_dict< / tt > table:
< pre > CREATE TABLE pg_ts_dict (
dict_name text not null,
2004-05-14 18:39:25 +08:00
dict_init regprocedure,
2003-07-21 18:27:44 +08:00
dict_initoption text,
2004-05-14 18:39:25 +08:00
dict_lexize regprocedure not null,
2003-07-21 18:27:44 +08:00
dict_comment text
);< / pre >
2003-08-04 22:11:08 +08:00
The < tt > dict_name< / tt >
2003-07-21 18:27:44 +08:00
serve as unique identifiers for the dictionary.
The meaning of the < tt > dict_initoption< / tt > varies among dictionaries,
but for the built-in Snowball dictionaries
it specifies a file from which stop words should be read.
The < tt > dict_comment< / tt > is a human-readable description of the dictionary.
The other fields are internal function identifiers
useful only to developers trying to implement their own dictionaries.
2006-11-01 00:23:05 +08:00
< blockquote >
< b > WARNING:< / b > Data files, used by dictionaries, should be in < tt > server_encoding< / tt > to
avoid possible problems !
< / blockquote >
2003-07-21 18:27:44 +08:00
< p >
The argument named < tt > < i > dictionary< / i > < / tt >
in each of the following functions
2003-08-04 22:11:08 +08:00
should be < tt > dict_name< / tt >
2003-07-21 18:27:44 +08:00
identifying which dictionary should be used for the operation;
if omitted then the current dictionary is used.
2003-08-04 22:11:08 +08:00
< / p > < dl > < dt >
2003-07-21 18:27:44 +08:00
< tt > CREATE FUNCTION set_curdict(< i > dictionary< / i > ) RETURNS VOID< / tt >
2003-08-04 22:11:08 +08:00
< / dt > < dd >
2003-07-21 18:27:44 +08:00
Selects a current dictionary for use by functions
that do not select a dictionary explicitly.
2003-08-04 22:11:08 +08:00
< / dd > < dt >
2003-07-21 18:27:44 +08:00
< tt > CREATE FUNCTION lexize(
< em > [< / em > < i > dictionary< / i > , < em > ]< / em > < i > word< / i > text)
RETURNS TEXT[]< / tt >
2003-08-04 22:11:08 +08:00
< / dt > < dd >
2003-07-21 18:27:44 +08:00
Reduces a single word to a lexeme.
Note that lexemes are arrays of zero or more strings,
since in some languages there might be several base words
from which an inflected form could arise.
2003-08-04 22:11:08 +08:00
< / dd > < / dl >
2003-07-21 18:27:44 +08:00
2006-11-01 00:23:05 +08:00
< h3 > Using dictionaries template< / h3 >
Templates used to define new dictionaries, for example,
< pre >
INSERT INTO pg_ts_dict
(SELECT 'en_ispell', dict_init,
'DictFile="/usr/local/share/dicts/ispell/english.dict",'
'AffFile="/usr/local/share/dicts/ispell/english.aff",'
'StopFile="/usr/local/share/dicts/english.stop"',
dict_lexize
FROM pg_ts_dict
WHERE dict_name = 'ispell_template');
< / pre >
< h3 > Working with stop words< / h3 >
Ispell and snowball stemmers treat stop words differently:
< ul >
< li > ispell - normalize word and then lookups normalized form in stop-word file
< li > snowball stemmer - first, it lookups word in stop-word file and then does it job.
The reason - to minimize possible 'noise'.
< / ul >
2003-07-21 18:27:44 +08:00
< h2 > < a name = "ranking" > Ranking< / a > < / h2 >
Ranking attempts to measure how relevant documents are to particular queries
by inspecting the number of times each search word appears in the document,
and whether different search terms occur near each other.
2003-08-04 22:11:08 +08:00
Note that this information is only available in unstripped vectors --
2003-07-21 18:27:44 +08:00
ranking functions will only return a useful result
for a < tt > tsvector< / tt > which still has position information!
< p >
2006-11-01 00:23:05 +08:00
Notice, that ranking functions supplied are just an examples and
doesn't belong to the tsearch2 core, you can
write your very own ranking function and/or combine additional
factors to fit your specific interest.
< / p >
2003-07-21 18:27:44 +08:00
The two ranking functions currently available are:
2003-08-04 22:11:08 +08:00
< dl > < dt >
2003-07-21 18:27:44 +08:00
< tt > CREATE FUNCTION rank(< br >
< em > [< / em > < i > weights< / i > float4[], < em > ]< / em >
2006-11-01 00:23:05 +08:00
< i > vector< / i > TSVECTOR, < i > query< / i > TSQUERY,
2003-07-21 18:27:44 +08:00
< em > [< / em > < i > normalization< / i > int4 < em > ]< / em > < br >
) RETURNS float4< / tt >
2003-08-04 22:11:08 +08:00
< / dt > < dd >
2003-07-21 18:27:44 +08:00
This is the ranking function from the old version of OpenFTS,
and offers the ability to weight word instances more heavily
depending on how you have classified them.
The < i > weights< / i > specify how heavily to weight each category of word:
2003-08-04 22:11:08 +08:00
< pre > {< i > D-weight< / i > , < i > C-weight< / i > , < i > B-weight< / i > , < i > A-weight< / i > }< / pre >
2003-07-21 18:27:44 +08:00
If no weights are provided, then these defaults are used:
< pre > {0.1, 0.2, 0.4, 1.0}< / pre >
Often weights are used to mark words from special areas of the document,
like the title or an initial abstract,
and make them more or less important than words in the document body.
2003-08-04 22:11:08 +08:00
< / dd > < dt >
2003-07-21 18:27:44 +08:00
< tt > CREATE FUNCTION rank_cd(< br >
2006-11-01 00:23:05 +08:00
< em > [< / em > < i > weights< / i > float4[], < em > ]< / em >
< i > vector< / i > TSVECTOR, < i > query< / i > TSQUERY,
2003-07-21 18:27:44 +08:00
< em > [< / em > < i > normalization< / i > int4 < em > ]< / em > < br >
) RETURNS float4< / tt >
2003-08-04 22:11:08 +08:00
< / dt > < dd >
2003-07-21 18:27:44 +08:00
This function computes the cover density ranking
for the given document < i > vector< / i > and < i > query< / i > ,
as described in Clarke, Cormack, and Tudhope's
2003-08-04 22:11:08 +08:00
"< a href = "http://citeseer.nj.nec.com/clarke00relevance.html" > Relevance Ranking for One to Three Term Queries< / a > "
2003-07-21 18:27:44 +08:00
in the 1999 < i > Information Processing and Management< / i > .
2006-11-01 00:23:05 +08:00
< / dd >
< dt >
< tt > CREATE FUNCTION get_covers(vector TSVECTOR, query TSQUERY) RETURNS text< / tt >
< / dt >
< dd >
Returns < tt > extents< / tt > , which are a shortest and non-nested sequences of words, which satisfy a query.
Extents (covers) used in < tt > rank_cd< / tt > algorithm for fast calculation of proximity ranking.
In example below there are two extents - < tt > < b > {1< / b > ...< b > }1< / b > and < b > {2< / b > ...< b > }2< / b > < / tt > .
< pre >
=# select get_covers('1:1,2,10 2:4'::tsvector,'1& 2');
get_covers
----------------------
1 {1 1 {2 2 }1 1 }2
< / pre >
< / dd >
< / dl >
< p >
Both of these (< tt > rank(), rank_cd()< / tt > ) ranking functions
take an integer < i > normalization< / i > option
that specifies whether a document's length should impact its rank.
This is often desirable,
since a hundred-word document with five instances of a search word
is probably more relevant than a thousand-word document with five instances.
The option can have the values, which could be combined using "|" ( 2|4) to
take into account several factors:
< / p >
< ul >
< li > < tt > 0< / tt > (the default) ignores document length.< / li >
< li > < tt > 1< / tt > divides the rank by the 1 + logarithm of the length < / li >
< li > < tt > 2< / tt > divides the rank by the length itself.< / li >
< li > < tt > 4< / tt > divides the rank by the mean harmonic distance between extents< / li >
< li > < tt > 8< / tt > divides the rank by the number of unique words in document< / li >
< li > < tt > 16< / tt > divides the rank by 1 + logarithm of the number of unique words in document
< / li >
< / ul >
2003-07-21 18:27:44 +08:00
< h2 > < a name = "headlines" > Headlines< / a > < / h2 >
2003-08-04 22:11:08 +08:00
< dl > < dt >
2003-07-21 18:27:44 +08:00
< tt > CREATE FUNCTION headline(< br >
< em > [< / em > < i > id< / i > int4, < em > |< / em > < i > ts_name< / i > text, < em > ]< / em >
2006-11-01 00:23:05 +08:00
< i > document< / i > text, < i > query< / i > TSQUERY,
2003-07-21 18:27:44 +08:00
< em > [< / em > < i > options< / i > text < em > ]< / em > < br >
) RETURNS text< / tt >
2003-08-04 22:11:08 +08:00
< / dt > < dd >
2003-07-21 18:27:44 +08:00
Every form of the the < tt > headline()< / tt > function
accepts a < tt > document< / tt > along with a < tt > query< / tt > ,
and returns one or more ellipse-separated excerpts from the document
in which terms from the query are highlighted.
The configuration with which to parse the document
can be specified by either its < i > id< / i > or < i > ts_name< / i > ;
if none is specified that the current configuration is used instead.
< p >
An < i > options< / i > string if provided should be a comma-separated list
2003-08-04 22:11:08 +08:00
of one or more '< i > option< / i > < tt > =< / tt > < i > value< / i > ' pairs.
2003-07-21 18:27:44 +08:00
The available options are:
2003-08-04 22:11:08 +08:00
< / p > < ul >
< li > < tt > StartSel< / tt > , < tt > StopSel< / tt > --
2003-07-21 18:27:44 +08:00
the strings with which query words appearing in the document
should be delimited to distinguish them from other excerpted words.
2003-08-04 22:11:08 +08:00
< / li > < li > < tt > MaxWords< / tt > , < tt > MinWords< / tt > --
2003-07-21 18:27:44 +08:00
limits on the shortest and longest headlines you will accept.
2003-08-04 22:11:08 +08:00
< / li > < li > < tt > ShortWord< / tt > --
2003-07-21 18:27:44 +08:00
this prevents your headline from beginning or ending
with a word which has this many characters or less.
The default value of < tt > 3< / tt > should eliminate most English
conjunctions and articles.
2006-11-01 00:23:05 +08:00
< / li > < li > < tt > HighlightAll< / tt > --
boolean flag, if TRUE, than the whole document will be highlighted.
2003-08-04 22:11:08 +08:00
< / li > < / ul >
2003-07-21 18:27:44 +08:00
Any unspecified options receive these defaults:
2006-11-01 00:23:05 +08:00
< pre > StartSel=< b> , StopSel=< /b> , MaxWords=35, MinWords=15, ShortWord=3, HighlightAll=FALSE
2003-07-21 18:27:44 +08:00
< / pre >
2003-08-04 22:11:08 +08:00
< / dd > < / dl >
2003-07-21 18:27:44 +08:00
2006-11-01 00:23:05 +08:00
< h2 > < a name = "indexes" > Indexes< / a > < / h2 >
Tsearch2 supports indexed access to tsvector in order to further speedup FTS. Notice, indexes are not mandatory for FTS !
< ul >
< li > RD-Tree (Russian Doll Tree, matryoshka), based on GiST (Generalized Search Tree)
< pre >
=# create index fts_idx on apod using gist(fts);
< / pre >
< li > GIN - Generalized Inverted Index
< pre >
=# create index fts_idx on apod using gin(fts);
< / pre >
< / ul >
< b > GiST< / b > index is very good for online update, but is not as scalable as < b > GIN< / b > index,
which, in turn, isn't good for updates. Both indexes support concurrency and recovery.
< h2 > < a name = "tz" > Thesaurus dictionary< / a > < / h2 >
< P >
Thesaurus - is a collection of words with included information about the relationships of words and phrases,
i.e., broader terms (BT), narrower terms (NT), preferred terms, non-preferred, related terms,etc.< / p >
< p > Basically,thesaurus dictionary replaces all non-preferred terms by one preferred term and, optionally,
preserves them for indexing. Thesaurus used when indexing, so any changes in thesaurus require reindexing.
Tsearch2's < tt > thesaurus< / tt > dictionary (TZ) is an extension of < tt > synonym< / tt > dictionary
with < b > phrase< / b > support. Thesaurus is a plain file of the following format:
< pre >
# this is a comment
sample word(s) : indexed word(s)
...............................
< / pre >
< ul >
< li > < strong > Colon< / strong > (:) symbol used as a delimiter.< / li >
< li > Use asterisk (< b > *< / b > ) at the beginning of < tt > indexed word< / tt > to skip subdictionary.
It's still required, that < tt > sample words< / tt > should be known.< / li >
< li > thesaurus dictionary looks for the most longest match< / li > < / ul >
< P >
TZ uses < strong > subdictionary< / strong > (should be defined in tsearch2 configuration)
to normalize thesaurus text. It's possible to define only < strong > one dictionary< / strong > .
Notice, that subdictionary produces an error, if it couldn't recognize word.
In that case, you should remove definition line with this word or teach subdictionary to know it.
< / p >
< p > Stop-words recognized by subdictionary replaced by 'stop-word placeholder', i.e.,
important only their position.
To break possible ties thesaurus applies the last definition. For example, consider
thesaurus (with simple subdictionary) rules with pattern 'swsw'
('s' designates stop-word and 'w' - known word): < / p >
< pre >
a one the two : swsw
the one a two : swsw2
< / pre >
< p > Words 'a' and 'the' are stop-words defined in the configuration of a subdictionary.
Thesaurus considers texts 'the one the two' and 'that one then two' as equal and will use definition
'swsw2'.< / p >
< p > As a normal dictionary, it should be assigned to the specific lexeme types.
Since TZ has a capability to recognize phrases it must remember its state and interact with parser.
TZ use these assignments to check if it should handle next word or stop accumulation.
Compiler of TZ should take care about proper configuration to avoid confusion.
For example, if TZ is assigned to handle only < tt > lword< / tt > lexeme, then TZ definition like
' one 1:11' will not works, since lexeme type < tt > digit< / tt > doesn't assigned to the TZ.< / p >
< h3 > Configuration< / h3 >
< dl > < dt > tsearch2< / dt > < dd > < / dd > < / dl > < p > tsearch2 comes with thesaurus template, which could be used to define new dictionary: < / p >
< pre class = "real" > INSERT INTO pg_ts_dict
(SELECT 'tz_simple', dict_init,
'DictFile="/path/to/tz_simple.txt",'
'Dictionary="en_stem"',
dict_lexize
FROM pg_ts_dict
WHERE dict_name = 'thesaurus_template');
< / pre >
< p > Here: < / p >
< ul >
< li > < tt > tz_simple< / tt > - is the dictionary name< / li >
< li > < tt > DictFile="/path/to/tz_simple.txt"< / tt > - is the location of thesaurus file< / li >
< li > < tt > Dictionary="en_stem"< / tt > defines dictionary (snowball english stemmer) to use for thesaurus normalization. Notice, that < em > en_stem< / em > dictionary has it's own configuration (stop-words, for example).< / li >
< / ul >
< p > Now, it's possible to use < tt > tz_simple< / tt > in pg_ts_cfgmap, for example: < / p >
< pre >
update pg_ts_cfgmap set dict_name='{tz_simple,en_stem}' where ts_name = 'default_russian' and
tok_alias in ('lhword', 'lword', 'lpart_hword');
< / pre >
< h3 > Examples< / h3 >
< p > tz_simple: < / p >
< pre >
one : 1
two : 2
one two : 12
the one : 1
one 1 : 11
< / pre >
< p > To see, how thesaurus works, one could use < tt > to_tsvector< / tt > , < tt > to_tsquery< / tt > or < tt > plainto_tsquery< / tt > functions: < / p > < pre class = "real" > =# select plainto_tsquery('default_russian',' one day is oneday');
plainto_tsquery
------------------------
'1' & 'day' & 'oneday'
=# select plainto_tsquery('default_russian','one two day is oneday');
plainto_tsquery
-------------------------
'12' & 'day' & 'oneday'
=# select plainto_tsquery('default_russian','the one');
NOTICE: Thesaurus: word 'the' is recognized as stop-word, assign any stop-word (rule 3)
plainto_tsquery
-----------------
'1'
< / pre >
Additional information about thesaurus dictionary is available from
< a href = "http://www.sai.msu.su/~megera/wiki/Thesaurus_dictionary" > Wiki< / a > page.
2004-05-14 18:39:25 +08:00
< / body > < / html >
