diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index cd66abc8ba..a848a7edd1 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2027,6 +2027,11 @@ include_dir 'conf.d'
as much CPU time, memory, I/O bandwidth, and so forth as a query which
uses no workers at all.
+
+
+ For more information on parallel query, see
+ .
+
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 43837114ba..69649a7da4 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -24,6 +24,7 @@
+
diff --git a/doc/src/sgml/parallel.sgml b/doc/src/sgml/parallel.sgml
new file mode 100644
index 0000000000..c80d42dbef
--- /dev/null
+++ b/doc/src/sgml/parallel.sgml
@@ -0,0 +1,472 @@
+
+
+
+ Parallel Query
+
+
+ parallel query
+
+
+
+ PostgreSQL can devise query plans which can leverage
+ multiple CPUs in order to answer queries faster. This feature is known
+ as parallel query. Many queries cannot benefit from parallel query, either
+ due to limitations of the current implementation or because there is no
+ imaginable query plan which is any faster than the serial query plan.
+ However, for queries that can benefit, the speedup from parallel query
+ is often very significant. Many queries can run more than twice as fast
+ when using parallel query, and some queries can run four times faster or
+ even more. Queries that touch a large amount of data but return only a
+ few rows to the user will typically benefit most. This chapter explains
+ some details of how parallel query works and in which situations it can be
+ used so that users who wish to make use of it can understand what to expect.
+
+
+
+ How Parallel Query Works
+
+
+ When the optimizer determines that parallel query is the fastest execution
+ strategy for a particular query, it will create a query plan which includes
+ a Gather node. Here is a simple example:
+
+
+EXPLAIN SELECT * FROM pgbench_accounts WHERE filler LIKE '%x%';
+ QUERY PLAN
+-------------------------------------------------------------------------------------
+ Gather (cost=1000.00..217018.43 rows=1 width=97)
+ Workers Planned: 2
+ -> Parallel Seq Scan on pgbench_accounts (cost=0.00..216018.33 rows=1 width=97)
+ Filter: (filler ~~ '%x%'::text)
+(4 rows)
+
+
+
+
+ In all cases, the Gather node will have exactly one
+ child plan, which is the portion of the plan that will be executed in
+ parallel. If the Gather node is at the very top of the plan
+ tree, then the entire query will execute in parallel. If it is somewhere
+ else in the plan tree, then only that portion of the query will run in
+ parallel. In the example above, the query accesses only one table, so
+ there is only one plan node other than the Gather node itself;
+ since that plan node is a child of the Gather node, it will
+ run in parallel.
+
+
+
+ Using EXPLAIN, you can see the number of
+ workers chosen by the planner. When the Gather node is reached
+ during query execution, the process which is implementing the user's
+ session will request a number of background
+ worker processes equal to the number
+ of workers chosen by the planner. The total number of background
+ workers that can exist at any one time is limited by
+ , so it is possible for a
+ parallel query to run with fewer workers than planned, or even with
+ no workers at all. The optimal plan may depend on the number of workers
+ that are available, so this can result in poor query performance. If this
+ occurrence is frequent, considering increasing
+ max_worker_processes so that more workers can be run
+ simultaneously or alternatively reducing
+ so that the planner
+ requests fewer workers.
+
+
+
+ Every background worker process which is successfully started for a given
+ parallel query will execute the portion of the plan which is a descendent
+ of the Gather node. The leader will also execute that portion
+ of the plan, but it has an additional responsibility: it must also read
+ all of the tuples generated by the workers. When the parallel portion of
+ the plan generates only a small number of tuples, the leader will often
+ behave very much like an additional worker, speeding up query execution.
+ Conversely, when the parallel portion of the plan generates a large number
+ of tuples, the leader may be almost entirely occupied with reading the
+ tuples generated by the workers and performing any further processing
+ steps which are required by plan nodes above the level of the
+ Gather node. In such cases, the leader will do very
+ little of the work of executing the parallel portion of the plan.
+
+
+
+
+ When Can Parallel Query Be Used?
+
+
+ There are several settings which can cause the query planner not to
+ generate a parallel query plan under any circumstances. In order for
+ any parallel query plans whatsoever to be generated, the following
+ settings must be configured as indicated.
+
+
+
+
+
+ must be set to a
+ value which is greater than zero. This is a special case of the more
+ general principle that no more workers should be used than the number
+ configured via max_parallel_workers_per_gather.
+
+
+
+
+
+ must be set to a
+ value other than none. Parallel query requires dynamic
+ shared memory in order to pass data between cooperating processes.
+
+
+
+
+
+ In addition, the system must not be running in single-user mode. Since
+ the entire database system is running in single process in this situation,
+ no background workers will be available.
+
+
+
+ Even when it is in general possible for parallel query plans to be
+ generated, the planner will not generate them for a given query
+ if any of the following are true:
+
+
+
+
+
+ The query writes any data or locks any database rows. If a query
+ contains a data-modifying operation either at the top level or within
+ a CTE, no parallel plans for that query will be generated. This is a
+ limitation of the current implementation which could be lifted in a
+ future release.
+
+
+
+
+
+ The query might be suspended during execution. In any situation in
+ which the system thinks that partial or incremental execution might
+ occur, no parallel plan is generated. For example, a cursor created
+ using DECLARE CURSOR will never use
+ a parallel plan. Similarly, a PL/pgsql loop of the form
+ FOR x IN query LOOP .. END LOOP will never use a
+ parallel plan, because the parallel query system is unable to verify
+ that the code in the loop is safe to execute while parallel query is
+ active.
+
+
+
+
+
+ The query uses any function marked PARALLEL UNSAFE.
+ Most system-defined functions are PARALLEL SAFE,
+ but user-defined functions are marked PARALLEL
+ UNSAFE by default. See the discussion of
+ .
+
+
+
+
+
+ The query is running inside of another query that is already parallel.
+ For example, if a function called by a parallel query issues an SQL
+ query itself, that query will never use a parallel plan. This is a
+ limitation of the current implementation, but it may not be desirable
+ to remove this limitation, since it could result in a single query
+ using a very large number of processes.
+
+
+
+
+
+ The transaction isolation level is serializable. This is
+ a limitation of the current implementation.
+
+
+
+
+
+ Even when parallel query plan is generated for a particular query, there
+ are several circumstances under which it will be impossible to execute
+ that plan in parallel at execution time. If this occurs, the leader
+ will execute the portion of the plan between below the Gather
+ node entirely by itself, almost as if the Gather node were
+ not present. This will happen if any of the following conditions are met:
+
+
+
+
+
+ No background workers can be obtained because of the limitation that
+ the total number of background workers cannot exceed
+ .
+
+
+
+
+
+ The client sends an Execute message with a non-zero fetch count.
+ See the discussion of the
+ extended query protocol.
+ Since libpq currently provides no way to
+ send such a message, this can only occur when using a client that
+ does not rely on libpq. If this is a frequent
+ occurrence, it may be a good idea to set
+ in sessions
+ where it is likely, so as to avoid generating query plans that may
+ be suboptimal when run serially.
+
+
+
+
+
+ The transaction isolation level is serializable. This situation
+ does not normally arise, because parallel query plans are not
+ generated when the transaction isolation level is serializable.
+ However, it can happen if the transaction isolation level is changed to
+ serializable after the plan is generated and before it is executed.
+
+
+
+
+
+
+ Parallel Plans
+
+
+ Because each worker executes the parallel portion of the plan to
+ completion, it is not possible to simply take an ordinary query plan
+ and run it using multiple workers. Each worker would produce a full
+ copy of the output result set, so the query would not run any faster
+ than normal but would produce incorrect results. Instead, the parallel
+ portion of the plan must be what is known internally to the query
+ optimizer as a partial plan; that is, it must constructed
+ so that each process will which executes the plan will generate only a
+ subset of the output rows in such a way that each required output row
+ is guaranteed to be generated by exactly one of the cooperating processes.
+
+
+
+ Parallel Scans
+
+
+ Currently, the only type of scan which has been modified to work with
+ parallel query is a sequential scan. Therefore, the driving table in
+ a parallel plan will always be scanned using a
+ Parallel Seq Scan. The relation's blocks will be divided
+ among the cooperating processes. Blocks are handed out one at a
+ time, so that access to the relation remains sequential. Each process
+ will visit every tuple on the page assigned to it before requesting a new
+ page.
+
+
+
+
+ Parallel Joins
+
+
+ The driving table may be joined to one or more other tables using nested
+ loops or hash joins. The outer side of the join may be any kind of
+ non-parallel plan that is otherwise supported by the planner provided that
+ it is safe to run within a parallel worker. For example, it may be an
+ index scan which looks up a value based on a column taken from the inner
+ table. Each worker will execute the outer side of the plan in full, which
+ is why merge joins are not supported here. The outer side of a merge join
+ will often involve sorting the entire inner table; even if it involves an
+ index, it is unlikely to be productive to have multiple processes each
+ conduct a full index scan of the inner table.
+
+
+
+
+ Parallel Aggregation
+
+ It is not possible to perform the aggregation portion of a query entirely
+ in parallel. For example, if a query involves selecting
+ COUNT(*), each worker could compute a total, but those totals
+ would need to combined in order to produce a final answer. If the query
+ involved a GROUP BY clause, a separate total would need to
+ be computed for each group. Even though aggregation can't be done entirely
+ in parallel, queries involving aggregation are often excellent candidates
+ for parallel query, because they typically read many rows but return only
+ a few rows to the client. Queries that return many rows to the client
+ are often limited by the speed at which the client can read the data,
+ in which case parallel query cannot help very much.
+
+
+
+ PostgreSQL supports parallel aggregation by aggregating
+ twice. First, each process participating in the parallel portion of the
+ query performs an aggregation step, producing a partial result for each
+ group of which that process is aware. This is reflected in the plan as
+ a PartialAggregate node. Second, the partial results are
+ transferred to the leader via the Gather node. Finally, the
+ leader re-aggregates the results across all workers in order to produce
+ the final result. This is reflected in the plan as a
+ FinalizeAggregate node.
+
+
+
+ Parallel aggregation is not supported in all situations. Each aggregate
+ must be safe for parallelism and must
+ have a combine function. If the aggregate has a transition state of type
+ internal, it must have serialization and deserialization
+ functions. See for more details.
+ Parallel aggregation is not supported for ordered set aggregates or when
+ the query involves GROUPING SETS. It can only be used when
+ all joins involved in the query are also part of the parallel portion
+ of the plan.
+
+
+
+
+
+ Parallel Plan Tips
+
+
+ If a query that is expected to do so does not produce a parallel plan,
+ you can try reducing or
+ . Of course, this plan may turn
+ out to be slower than the serial plan which the planner preferred, but
+ this will not always be the case. If you don't get a parallel
+ plan even with very small values of these settings (e.g. after setting
+ them both to zero), there may be some reason why the query planner is
+ unable to generate a parallel plan for your query. See
+ and
+ for information on why this may be
+ the case.
+
+
+
+ When executing a parallel plan, you can use EXPLAIN (ANALYZE,
+ VERBOSE) will display per-worker statistics for each plan node.
+ This may be useful in determining whether the work is being evenly
+ distributed between all plan nodes and more generally in understanding the
+ performance characteristics of the plan.
+
+
+
+
+
+
+ Parallel Safety
+
+
+ The planner classifies operations involved in a query as either
+ parallel safe, parallel restricted,
+ or parallel unsafe. A parallel safe operation is one which
+ does not conflict with the use of parallel query. A parallel restricted
+ operation is one which cannot be performed in a parallel worker, but which
+ can be performed in the leader while parallel query is in use. Therefore,
+ parallel restricted operations can never occur below a Gather
+ node, but can occur elsewhere in a plan which contains a
+ Gather node. A parallel unsafe operation is one which cannot
+ be performed while parallel query is in use, not even in the leader.
+ When a query contains anything which is parallel unsafe, parallel query
+ is completely disabled for that query.
+
+
+
+ The following operations are always parallel restricted.
+
+
+
+
+
+ Scans of common table expressions (CTEs).
+
+
+
+
+
+ Scans of temporary tables.
+
+
+
+
+
+ Scans of foreign tables, unless the foreign data wrapper has
+ an IsForeignScanParallelSafe API which indicates otherwise.
+
+
+
+
+
+ Access to an InitPlan or SubPlan.
+
+
+
+
+
+ Parallel Labeling for Functions and Aggregates
+
+
+ The planner cannot automatically determine whether a user-defined
+ function or aggregate is parallel safe, parallel restricted, or parallel
+ unsafe, because this would require predicting every operation which the
+ function could possibly perform. In general, this is equivalent to the
+ Halting Problem and therefore impossible. Even for simple functions
+ where it conceivably be done, we do not try, since this would be expensive
+ and error-prone. Instead, all user-defined functions are assumed to
+ be parallel unsafe unless otherwise marked. When using
+ or
+ , markings can be set by specifying
+ PARALLEL SAFE, PARALLEL RESTRICTED, or
+ PARALLEL UNSAFE as appropriate. When using
+ , the
+ PARALLEL option can be specified with SAFE,
+ RESTRICTED, or UNSAFE as the corresponding value.
+
+
+
+ Functions and aggregates must be marked PARALLEL UNSAFE if
+ they write to the database, access sequences, change the transaction state
+ even temporarily (e.g. a PL/pgsql function which establishes an
+ EXCEPTION block to catch errors), or make persistent changes to
+ settings. Similarly, functions must be marked PARALLEL
+ RESTRICTED if they access temporary tables, client connection state,
+ cursors, prepared statements, or miscellaneous backend-local state which
+ the system cannot synchronize across workers. For example,
+ setseed and random are parallel restricted for
+ this last reason.
+
+
+
+ In general, if a function is labeled as being safe when it is restricted or
+ unsafe, or if it is labeled as being restricted when it is in fact unsafe,
+ it may throw errors or produce wrong answers when used in a parallel query.
+ C-language functions could in theory exhibit totally undefined behavior if
+ mislabeled, since there is no way for the system to protect itself against
+ arbitrary C code, but in most likely cases the result will be no worse than
+ for any other function. If in doubt, it is probably best to label functions
+ as UNSAFE.
+
+
+
+ If a function executed within a parallel worker acquires locks which are
+ not held by the leader, for example by querying a table not referenced in
+ the query, those locks will be released at worker exit, not end of
+ transaction. If you write a function which does this, and this behavior
+ difference is important to you, mark such functions as
+ PARALLEL RESTRICTED
+ to ensure that they execute only in the leader.
+
+
+
+ Note that the query planner does not consider deferring the evaluation of
+ parallel-restricted functions or aggregates involved in the query in
+ order to obtain a superior plan. So, for example, if a WHERE
+ clause applied to a particular table is parallel restricted, the query
+ planner will not consider placing the scan of that table below a
+ Gather node. In some cases, it would be
+ possible (and perhaps even efficient) to include the scan of that table in
+ the parallel portion of the query and defer the evaluation of the
+ WHERE clause so that it happens above the Gather
+ node. However, the planner does not do this.
+
+
+
+
+
+
+
diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml
index 0346d367e5..9143917c49 100644
--- a/doc/src/sgml/postgres.sgml
+++ b/doc/src/sgml/postgres.sgml
@@ -106,6 +106,7 @@
&textsearch;
&mvcc;
&perform;
+ ∥