mirror/postgresql
2
0
Fork 0
mirror of https://git.postgresql.org/git/postgresql.git synced 2024-12-15 08:20:16 +08:00
Code Issues Packages Projects Releases Wiki Activity
3f302f0ed0
postgresql/contrib/postgres_fdw/postgres_fdw.h

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

215 lines
7.1 KiB
C
Raw Normal View History

Add postgres_fdw contrib module. There's still a lot of room for improvement, but it basically works, and we need this to be present before we can do anything much with the writable-foreign-tables patch. So let's commit it and get on with testing. Shigeru Hanada, reviewed by KaiGai Kohei and Tom Lane
2013-02-21 18:26:23 +08:00
/*-------------------------------------------------------------------------
*
* postgres_fdw.h
* Foreign-data wrapper for remote PostgreSQL servers
*
Update copyright for 2019 Backpatch-through: certain files through 9.4
2019-01-03 01:44:25 +08:00
* Portions Copyright (c) 2012-2019, PostgreSQL Global Development Group
Add postgres_fdw contrib module. There's still a lot of room for improvement, but it basically works, and we need this to be present before we can do anything much with the writable-foreign-tables patch. So let's commit it and get on with testing. Shigeru Hanada, reviewed by KaiGai Kohei and Tom Lane
2013-02-21 18:26:23 +08:00
*
* IDENTIFICATION
* contrib/postgres_fdw/postgres_fdw.h
*
*-------------------------------------------------------------------------
*/
#ifndef POSTGRES_FDW_H
#define POSTGRES_FDW_H
#include "foreign/foreign.h"
#include "lib/stringinfo.h"
Rename nodes/relation.h to nodes/pathnodes.h. The old name of this file was never a very good indication of what it was for. Now that there's also access/relation.h, we have a potential confusion hazard as well, so let's rename it to something more apropos. Per discussion, "pathnodes.h" is reasonable, since a good fraction of the file is Path node definitions. While at it, tweak a couple of other headers that were gratuitously importing relation.h into modules that don't need it. Discussion: https://postgr.es/m/7719.1548688728@sss.pgh.pa.us
2019-01-30 05:49:25 +08:00
#include "nodes/pathnodes.h"
postgres_fdw.h: don't pull in rel.h when relcache.h is enough
2014-11-15 08:48:53 +08:00
#include "utils/relcache.h"
Add postgres_fdw contrib module. There's still a lot of room for improvement, but it basically works, and we need this to be present before we can do anything much with the writable-foreign-tables patch. So let's commit it and get on with testing. Shigeru Hanada, reviewed by KaiGai Kohei and Tom Lane
2013-02-21 18:26:23 +08:00
#include "libpq-fe.h"
Allow postgres_fdw to ship extension funcs/operators for remote execution. The user can whitelist specified extension(s) in the foreign server's options, whereupon we will treat immutable functions and operators of those extensions as candidates to be sent for remote execution. Whitelisting an extension in this way basically promises that the extension exists on the remote server and behaves compatibly with the local instance. We have no way to prove that formally, so we have to rely on the user to get it right. But this seems like something that people can usually get right in practice. We might in future allow functions and operators to be whitelisted individually, but extension granularity is a very convenient special case, so it got done first. The patch as-committed lacks any regression tests, which is unfortunate, but introducing dependencies on other extensions for testing purposes would break "make installcheck" scenarios, which is worse. I have some ideas about klugy ways around that, but it seems like material for a separate patch. For the moment, leave the problem open. Paul Ramsey, hacked up a bit more by me
2015-11-04 07:42:02 +08:00
/*
* FDW-specific planner information kept in RelOptInfo.fdw_private for a
Simplify handling of remote-qual pass-forward in postgres_fdw. Commit 0bf3ae88a encountered a need to pass the finally chosen remote qual conditions forward from postgresGetForeignPlan to postgresPlanDirectModify. It solved that by sticking them into the plan node's fdw_private list, which in hindsight was a pretty bad idea. In the first place, there's no use for those qual trees either in EXPLAIN or execution; indeed they could never safely be used for any post-planning purposes, because they would not get processed by setrefs.c. So they're just dead weight to carry around in the finished plan tree, plus being an attractive nuisance for somebody who might get the idea that they could be used that way. Secondly, because those qual trees (sometimes) contained RestrictInfos, they created a plan-transmission hazard for parallel query, which is how come we noticed a problem. We dealt with that symptom in commit 28b047875, but really a more straightforward and more efficient fix is to pass the data through in a new field of struct PgFdwRelationInfo. So do it that way. (There's no need to revert 28b047875, as it has sufficient reason to live anyway.) Per fuzz testing by Andreas Seltenreich. Discussion: https://postgr.es/m/87tw5x4vcu.fsf@credativ.de
2017-04-12 01:53:13 +08:00
* postgres_fdw foreign table. For a baserel, this struct is created by
* postgresGetForeignRelSize, although some fields are not filled till later.
* postgresGetForeignJoinPaths creates it for a joinrel, and
* postgresGetForeignUpperPaths creates it for an upperrel.
Allow postgres_fdw to ship extension funcs/operators for remote execution. The user can whitelist specified extension(s) in the foreign server's options, whereupon we will treat immutable functions and operators of those extensions as candidates to be sent for remote execution. Whitelisting an extension in this way basically promises that the extension exists on the remote server and behaves compatibly with the local instance. We have no way to prove that formally, so we have to rely on the user to get it right. But this seems like something that people can usually get right in practice. We might in future allow functions and operators to be whitelisted individually, but extension granularity is a very convenient special case, so it got done first. The patch as-committed lacks any regression tests, which is unfortunate, but introducing dependencies on other extensions for testing purposes would break "make installcheck" scenarios, which is worse. I have some ideas about klugy ways around that, but it seems like material for a separate patch. For the moment, leave the problem open. Paul Ramsey, hacked up a bit more by me
2015-11-04 07:42:02 +08:00
*/
typedef struct PgFdwRelationInfo
{
postgres_fdw: Push down joins to remote servers. If we've got a relatively straightforward join between two tables, this pushes that join down to the remote server instead of fetching the rows for each table and performing the join locally. Some cases are not handled yet, such as SEMI and ANTI joins. Also, we don't yet attempt to create presorted join paths or parameterized join paths even though these options do get tried for a base relation scan. Nevertheless, this seems likely to be a very significant win in many practical cases. Shigeru Hanada and Ashutosh Bapat, reviewed by Robert Haas, with additional review at various points by Tom Lane, Etsuro Fujita, KaiGai Kohei, and Jeevan Chalke.
2016-02-10 03:00:50 +08:00
/*
* True means that the relation can be pushed down. Always true for simple
* foreign scan.
*/
bool pushdown_safe;
/*
* Restriction clauses, divided into safe and unsafe to pushdown subsets.
Handle restriction clause lists more uniformly in postgres_fdw. Clauses in the lists retained by postgres_fdw during planning were sometimes bare boolean clauses, sometimes RestrictInfos, and sometimes a mixture of the two in the same list. The comment about that situation didn't come close to telling the full truth, either. Aside from being confusing, this had a couple of bad practical consequences: * waste of planning cycles due to inability to cache per-clause selectivity and cost estimates; * sometimes, RestrictInfos would sneak into the fdw_private list of a finished Plan node, causing failures if, for example, we tried to ship the Plan tree to a parallel worker. (It may well be that it's a bug in the parallel-query logic that we would ever try to ship such a plan to a parallel worker, but in any case this deserves to be cleaned up.) To fix, rearrange so that clause lists in PgFdwRelationInfo are always lists of RestrictInfos, and then strip the RestrictInfos at the last minute when making a Plan node. In passing do a bit of refactoring and comment cleanup in postgresGetForeignPlan and foreign_join_ok. Although the messiness here dates back at least to 9.6, there's no evidence that it causes anything worse than wasted planning cycles in 9.6, so no back-patch for now. Per fuzz testing by Andreas Seltenreich. Tom Lane and Ashutosh Bapat Discussion: https://postgr.es/m/87tw5x4vcu.fsf@credativ.de
2017-04-11 23:58:59 +08:00
* All entries in these lists should have RestrictInfo wrappers; that
* improves efficiency of selectivity and cost estimation.
postgres_fdw: Push down joins to remote servers. If we've got a relatively straightforward join between two tables, this pushes that join down to the remote server instead of fetching the rows for each table and performing the join locally. Some cases are not handled yet, such as SEMI and ANTI joins. Also, we don't yet attempt to create presorted join paths or parameterized join paths even though these options do get tried for a base relation scan. Nevertheless, this seems likely to be a very significant win in many practical cases. Shigeru Hanada and Ashutosh Bapat, reviewed by Robert Haas, with additional review at various points by Tom Lane, Etsuro Fujita, KaiGai Kohei, and Jeevan Chalke.
2016-02-10 03:00:50 +08:00
*/
Allow postgres_fdw to ship extension funcs/operators for remote execution. The user can whitelist specified extension(s) in the foreign server's options, whereupon we will treat immutable functions and operators of those extensions as candidates to be sent for remote execution. Whitelisting an extension in this way basically promises that the extension exists on the remote server and behaves compatibly with the local instance. We have no way to prove that formally, so we have to rely on the user to get it right. But this seems like something that people can usually get right in practice. We might in future allow functions and operators to be whitelisted individually, but extension granularity is a very convenient special case, so it got done first. The patch as-committed lacks any regression tests, which is unfortunate, but introducing dependencies on other extensions for testing purposes would break "make installcheck" scenarios, which is worse. I have some ideas about klugy ways around that, but it seems like material for a separate patch. For the moment, leave the problem open. Paul Ramsey, hacked up a bit more by me
2015-11-04 07:42:02 +08:00
List *remote_conds;
List *local_conds;
Simplify handling of remote-qual pass-forward in postgres_fdw. Commit 0bf3ae88a encountered a need to pass the finally chosen remote qual conditions forward from postgresGetForeignPlan to postgresPlanDirectModify. It solved that by sticking them into the plan node's fdw_private list, which in hindsight was a pretty bad idea. In the first place, there's no use for those qual trees either in EXPLAIN or execution; indeed they could never safely be used for any post-planning purposes, because they would not get processed by setrefs.c. So they're just dead weight to carry around in the finished plan tree, plus being an attractive nuisance for somebody who might get the idea that they could be used that way. Secondly, because those qual trees (sometimes) contained RestrictInfos, they created a plan-transmission hazard for parallel query, which is how come we noticed a problem. We dealt with that symptom in commit 28b047875, but really a more straightforward and more efficient fix is to pass the data through in a new field of struct PgFdwRelationInfo. So do it that way. (There's no need to revert 28b047875, as it has sufficient reason to live anyway.) Per fuzz testing by Andreas Seltenreich. Discussion: https://postgr.es/m/87tw5x4vcu.fsf@credativ.de
2017-04-12 01:53:13 +08:00
/* Actual remote restriction clauses for scan (sans RestrictInfos) */
List *final_remote_exprs;
Allow postgres_fdw to ship extension funcs/operators for remote execution. The user can whitelist specified extension(s) in the foreign server's options, whereupon we will treat immutable functions and operators of those extensions as candidates to be sent for remote execution. Whitelisting an extension in this way basically promises that the extension exists on the remote server and behaves compatibly with the local instance. We have no way to prove that formally, so we have to rely on the user to get it right. But this seems like something that people can usually get right in practice. We might in future allow functions and operators to be whitelisted individually, but extension granularity is a very convenient special case, so it got done first. The patch as-committed lacks any regression tests, which is unfortunate, but introducing dependencies on other extensions for testing purposes would break "make installcheck" scenarios, which is worse. I have some ideas about klugy ways around that, but it seems like material for a separate patch. For the moment, leave the problem open. Paul Ramsey, hacked up a bit more by me
2015-11-04 07:42:02 +08:00
/* Bitmap of attr numbers we need to fetch from the remote server. */
Bitmapset *attrs_used;
postgres_fdw: Perform the (ORDERED, NULL) upperrel operations remotely. The upper-planner pathification allows FDWs to arrange to push down different types of upper-stage operations to the remote side. This commit teaches postgres_fdw to do it for the (ORDERED, NULL) upperrel, which is responsible for evaluating the query's ORDER BY ordering. Since postgres_fdw is already able to evaluate that ordering remotely for foreign baserels and foreign joinrels (see commit aa09cd242f et al.), this adds support for that for foreign grouping relations. Author: Etsuro Fujita Reviewed-By: Antonin Houska and Jeff Janes Discussion: https://postgr.es/m/87pnz1aby9.fsf@news-spur.riddles.org.uk
2019-04-02 18:20:30 +08:00
/* True means that the query_pathkeys is safe to push down */
bool qp_is_pushdown_safe;
Allow postgres_fdw to ship extension funcs/operators for remote execution. The user can whitelist specified extension(s) in the foreign server's options, whereupon we will treat immutable functions and operators of those extensions as candidates to be sent for remote execution. Whitelisting an extension in this way basically promises that the extension exists on the remote server and behaves compatibly with the local instance. We have no way to prove that formally, so we have to rely on the user to get it right. But this seems like something that people can usually get right in practice. We might in future allow functions and operators to be whitelisted individually, but extension granularity is a very convenient special case, so it got done first. The patch as-committed lacks any regression tests, which is unfortunate, but introducing dependencies on other extensions for testing purposes would break "make installcheck" scenarios, which is worse. I have some ideas about klugy ways around that, but it seems like material for a separate patch. For the moment, leave the problem open. Paul Ramsey, hacked up a bit more by me
2015-11-04 07:42:02 +08:00
/* Cost and selectivity of local_conds. */
QualCost local_conds_cost;
Selectivity local_conds_sel;
postgres_fdw: Push down joins to remote servers. If we've got a relatively straightforward join between two tables, this pushes that join down to the remote server instead of fetching the rows for each table and performing the join locally. Some cases are not handled yet, such as SEMI and ANTI joins. Also, we don't yet attempt to create presorted join paths or parameterized join paths even though these options do get tried for a base relation scan. Nevertheless, this seems likely to be a very significant win in many practical cases. Shigeru Hanada and Ashutosh Bapat, reviewed by Robert Haas, with additional review at various points by Tom Lane, Etsuro Fujita, KaiGai Kohei, and Jeevan Chalke.
2016-02-10 03:00:50 +08:00
/* Selectivity of join conditions */
Selectivity joinclause_sel;
postgres_fdw: Fix costing of pre-sorted foreign paths with local stats. Commit aa09cd242 modified estimate_path_cost_size() so that it reuses cached costs of a basic foreign path for a given foreign-base/join relation when costing pre-sorted foreign paths for that relation, but it incorrectly re-computed retrieved_rows, an estimated number of rows fetched from the remote side, which is needed for costing both the basic and pre-sorted foreign paths. To fix, handle retrieved_rows the same way as the cached costs: store in that relation's fpinfo the retrieved_rows estimate computed for costing the basic foreign path, and reuse it when costing the pre-sorted foreign paths. Also, reuse the rows/width estimates stored in that relation's fpinfo when costing the pre-sorted foreign paths, to make the code consistent. In commit ffab494a4, to extend the costing mentioned above to the foreign-grouping case, I made a change to add_foreign_grouping_paths() to store in a given foreign-grouped relation's RelOptInfo the rows estimate for that relation for reuse, but this patch makes that change unnecessary since we already store the row estimate in that relation's fpinfo, which this patch reuses when costing a foreign path for that relation with the sortClause ordering; remove that change. In passing, fix thinko in commit 7012b132d: in estimate_path_cost_size(), the width estimate for a given foreign-grouped relation to be stored in that relation's fpinfo was reset incorrectly when costing a basic foreign path for that relation with local stats. Apply the patch to HEAD only to avoid destabilizing existing plan choices. Author: Etsuro Fujita Discussion: https://postgr.es/m/CAPmGK17jaJLPDEkgnP2VmkOg=5wT8YQ1CqssU8JRpZ_NSE+dqQ@mail.gmail.com
2019-06-14 19:49:59 +08:00
/* Estimated size and cost for a scan, join, or grouping/aggregation. */
Allow postgres_fdw to ship extension funcs/operators for remote execution. The user can whitelist specified extension(s) in the foreign server's options, whereupon we will treat immutable functions and operators of those extensions as candidates to be sent for remote execution. Whitelisting an extension in this way basically promises that the extension exists on the remote server and behaves compatibly with the local instance. We have no way to prove that formally, so we have to rely on the user to get it right. But this seems like something that people can usually get right in practice. We might in future allow functions and operators to be whitelisted individually, but extension granularity is a very convenient special case, so it got done first. The patch as-committed lacks any regression tests, which is unfortunate, but introducing dependencies on other extensions for testing purposes would break "make installcheck" scenarios, which is worse. I have some ideas about klugy ways around that, but it seems like material for a separate patch. For the moment, leave the problem open. Paul Ramsey, hacked up a bit more by me
2015-11-04 07:42:02 +08:00
double rows;
int width;
Cost startup_cost;
Cost total_cost;
pgindent run prior to branching v12. pgperltidy and reformat-dat-files too, though the latter didn't find anything to change.
2019-07-02 00:37:52 +08:00
postgres_fdw: Fix costing of pre-sorted foreign paths with local stats. Commit aa09cd242 modified estimate_path_cost_size() so that it reuses cached costs of a basic foreign path for a given foreign-base/join relation when costing pre-sorted foreign paths for that relation, but it incorrectly re-computed retrieved_rows, an estimated number of rows fetched from the remote side, which is needed for costing both the basic and pre-sorted foreign paths. To fix, handle retrieved_rows the same way as the cached costs: store in that relation's fpinfo the retrieved_rows estimate computed for costing the basic foreign path, and reuse it when costing the pre-sorted foreign paths. Also, reuse the rows/width estimates stored in that relation's fpinfo when costing the pre-sorted foreign paths, to make the code consistent. In commit ffab494a4, to extend the costing mentioned above to the foreign-grouping case, I made a change to add_foreign_grouping_paths() to store in a given foreign-grouped relation's RelOptInfo the rows estimate for that relation for reuse, but this patch makes that change unnecessary since we already store the row estimate in that relation's fpinfo, which this patch reuses when costing a foreign path for that relation with the sortClause ordering; remove that change. In passing, fix thinko in commit 7012b132d: in estimate_path_cost_size(), the width estimate for a given foreign-grouped relation to be stored in that relation's fpinfo was reset incorrectly when costing a basic foreign path for that relation with local stats. Apply the patch to HEAD only to avoid destabilizing existing plan choices. Author: Etsuro Fujita Discussion: https://postgr.es/m/CAPmGK17jaJLPDEkgnP2VmkOg=5wT8YQ1CqssU8JRpZ_NSE+dqQ@mail.gmail.com
2019-06-14 19:49:59 +08:00
/*
* Estimated number of rows fetched from the foreign server, and costs
* excluding costs for transferring those rows from the foreign server.
* These are only used by estimate_path_cost_size().
*/
double retrieved_rows;
postgres_fdw: Push down joins to remote servers. If we've got a relatively straightforward join between two tables, this pushes that join down to the remote server instead of fetching the rows for each table and performing the join locally. Some cases are not handled yet, such as SEMI and ANTI joins. Also, we don't yet attempt to create presorted join paths or parameterized join paths even though these options do get tried for a base relation scan. Nevertheless, this seems likely to be a very significant win in many practical cases. Shigeru Hanada and Ashutosh Bapat, reviewed by Robert Haas, with additional review at various points by Tom Lane, Etsuro Fujita, KaiGai Kohei, and Jeevan Chalke.
2016-02-10 03:00:50 +08:00
Cost rel_startup_cost;
Cost rel_total_cost;
Allow postgres_fdw to ship extension funcs/operators for remote execution. The user can whitelist specified extension(s) in the foreign server's options, whereupon we will treat immutable functions and operators of those extensions as candidates to be sent for remote execution. Whitelisting an extension in this way basically promises that the extension exists on the remote server and behaves compatibly with the local instance. We have no way to prove that formally, so we have to rely on the user to get it right. But this seems like something that people can usually get right in practice. We might in future allow functions and operators to be whitelisted individually, but extension granularity is a very convenient special case, so it got done first. The patch as-committed lacks any regression tests, which is unfortunate, but introducing dependencies on other extensions for testing purposes would break "make installcheck" scenarios, which is worse. I have some ideas about klugy ways around that, but it seems like material for a separate patch. For the moment, leave the problem open. Paul Ramsey, hacked up a bit more by me
2015-11-04 07:42:02 +08:00
/* Options extracted from catalogs. */
bool use_remote_estimate;
Cost fdw_startup_cost;
Cost fdw_tuple_cost;
List *shippable_extensions; /* OIDs of whitelisted extensions */
/* Cached catalog information. */
ForeignTable *table;
ForeignServer *server;
UserMapping *user; /* only set in use_remote_estimate mode */
postgres_fdw: Allow fetch_size to be set per-table or per-server. The default fetch size of 100 rows might not be right in every environment, so allow users to configure it. Corey Huinker, reviewed by Kyotaro Horiguchi, Andres Freund, and me.
2016-02-03 22:01:59 +08:00
int fetch_size; /* fetch size for this remote table */
postgres_fdw: Push down joins to remote servers. If we've got a relatively straightforward join between two tables, this pushes that join down to the remote server instead of fetching the rows for each table and performing the join locally. Some cases are not handled yet, such as SEMI and ANTI joins. Also, we don't yet attempt to create presorted join paths or parameterized join paths even though these options do get tried for a base relation scan. Nevertheless, this seems likely to be a very significant win in many practical cases. Shigeru Hanada and Ashutosh Bapat, reviewed by Robert Haas, with additional review at various points by Tom Lane, Etsuro Fujita, KaiGai Kohei, and Jeevan Chalke.
2016-02-10 03:00:50 +08:00
/*
* Name of the relation while EXPLAINing ForeignScan. It is used for join
* relations but is set for all relations. For join relation, the name
* indicates which foreign tables are being joined and the join type used.
*/
StringInfo relation_name;
/* Join information */
RelOptInfo *outerrel;
RelOptInfo *innerrel;
JoinType jointype;
Simplify handling of remote-qual pass-forward in postgres_fdw. Commit 0bf3ae88a encountered a need to pass the finally chosen remote qual conditions forward from postgresGetForeignPlan to postgresPlanDirectModify. It solved that by sticking them into the plan node's fdw_private list, which in hindsight was a pretty bad idea. In the first place, there's no use for those qual trees either in EXPLAIN or execution; indeed they could never safely be used for any post-planning purposes, because they would not get processed by setrefs.c. So they're just dead weight to carry around in the finished plan tree, plus being an attractive nuisance for somebody who might get the idea that they could be used that way. Secondly, because those qual trees (sometimes) contained RestrictInfos, they created a plan-transmission hazard for parallel query, which is how come we noticed a problem. We dealt with that symptom in commit 28b047875, but really a more straightforward and more efficient fix is to pass the data through in a new field of struct PgFdwRelationInfo. So do it that way. (There's no need to revert 28b047875, as it has sufficient reason to live anyway.) Per fuzz testing by Andreas Seltenreich. Discussion: https://postgr.es/m/87tw5x4vcu.fsf@credativ.de
2017-04-12 01:53:13 +08:00
/* joinclauses contains only JOIN/ON conditions for an outer join */
Handle restriction clause lists more uniformly in postgres_fdw. Clauses in the lists retained by postgres_fdw during planning were sometimes bare boolean clauses, sometimes RestrictInfos, and sometimes a mixture of the two in the same list. The comment about that situation didn't come close to telling the full truth, either. Aside from being confusing, this had a couple of bad practical consequences: * waste of planning cycles due to inability to cache per-clause selectivity and cost estimates; * sometimes, RestrictInfos would sneak into the fdw_private list of a finished Plan node, causing failures if, for example, we tried to ship the Plan tree to a parallel worker. (It may well be that it's a bug in the parallel-query logic that we would ever try to ship such a plan to a parallel worker, but in any case this deserves to be cleaned up.) To fix, rearrange so that clause lists in PgFdwRelationInfo are always lists of RestrictInfos, and then strip the RestrictInfos at the last minute when making a Plan node. In passing do a bit of refactoring and comment cleanup in postgresGetForeignPlan and foreign_join_ok. Although the messiness here dates back at least to 9.6, there's no evidence that it causes anything worse than wasted planning cycles in 9.6, so no back-patch for now. Per fuzz testing by Andreas Seltenreich. Tom Lane and Ashutosh Bapat Discussion: https://postgr.es/m/87tw5x4vcu.fsf@credativ.de
2017-04-11 23:58:59 +08:00
List *joinclauses; /* List of RestrictInfo */
postgres_fdw: Push down aggregates to remote servers. Now that the upper planner uses paths, and now that we have proper hooks to inject paths into the upper planning process, it's possible for foreign data wrappers to arrange to push aggregates to the remote side instead of fetching all of the rows and aggregating them locally. This figures to be a massive win for performance, so teach postgres_fdw to do it. Jeevan Chalke and Ashutosh Bapat. Reviewed by Ashutosh Bapat with additional testing by Prabhat Sahu. Various mostly cosmetic changes by me.
2016-10-21 21:54:29 +08:00
postgres_fdw: Perform the (ORDERED, NULL) upperrel operations remotely. The upper-planner pathification allows FDWs to arrange to push down different types of upper-stage operations to the remote side. This commit teaches postgres_fdw to do it for the (ORDERED, NULL) upperrel, which is responsible for evaluating the query's ORDER BY ordering. Since postgres_fdw is already able to evaluate that ordering remotely for foreign baserels and foreign joinrels (see commit aa09cd242f et al.), this adds support for that for foreign grouping relations. Author: Etsuro Fujita Reviewed-By: Antonin Houska and Jeff Janes Discussion: https://postgr.es/m/87pnz1aby9.fsf@news-spur.riddles.org.uk
2019-04-02 18:20:30 +08:00
/* Upper relation information */
UpperRelationKind stage;
postgres_fdw: Push down aggregates to remote servers. Now that the upper planner uses paths, and now that we have proper hooks to inject paths into the upper planning process, it's possible for foreign data wrappers to arrange to push aggregates to the remote side instead of fetching all of the rows and aggregating them locally. This figures to be a massive win for performance, so teach postgres_fdw to do it. Jeevan Chalke and Ashutosh Bapat. Reviewed by Ashutosh Bapat with additional testing by Prabhat Sahu. Various mostly cosmetic changes by me.
2016-10-21 21:54:29 +08:00
/* Grouping information */
List *grouped_tlist;
postgres_fdw: Push down FULL JOINs with restriction clauses. The previous deparsing logic wasn't smart enough to produce subqueries when deparsing; make it smart enough to do that. However, we only do it that way when necessary, because it generates more complicated SQL which will be harder for any humans reading the queries to understand. Etsuro Fujita, reviewed by Ashutosh Bapat Discussion: http://postgr.es/m/c449261a-b033-dc02-9254-2fe5b7044795@lab.ntt.co.jp
2017-03-17 01:34:59 +08:00
/* Subquery information */
bool make_outerrel_subquery; /* do we deparse outerrel as a
* subquery? */
bool make_innerrel_subquery; /* do we deparse innerrel as a
* subquery? */
Relids lower_subquery_rels; /* all relids appearing in lower
* subqueries */
/*
* Index of the relation. It is used to create an alias to a subquery
* representing the relation.
*/
int relation_index;
Allow postgres_fdw to ship extension funcs/operators for remote execution. The user can whitelist specified extension(s) in the foreign server's options, whereupon we will treat immutable functions and operators of those extensions as candidates to be sent for remote execution. Whitelisting an extension in this way basically promises that the extension exists on the remote server and behaves compatibly with the local instance. We have no way to prove that formally, so we have to rely on the user to get it right. But this seems like something that people can usually get right in practice. We might in future allow functions and operators to be whitelisted individually, but extension granularity is a very convenient special case, so it got done first. The patch as-committed lacks any regression tests, which is unfortunate, but introducing dependencies on other extensions for testing purposes would break "make installcheck" scenarios, which is worse. I have some ideas about klugy ways around that, but it seems like material for a separate patch. For the moment, leave the problem open. Paul Ramsey, hacked up a bit more by me
2015-11-04 07:42:02 +08:00
} PgFdwRelationInfo;
Fix postgres_fdw's issues with inconsistent interpretation of data values. For datatypes whose output formatting depends on one or more GUC settings, we have to worry about whether the other server will interpret the value the same way it was meant. pg_dump has been aware of this hazard for a long time, but postgres_fdw needs to deal with it too. To fix data retrieval from the remote server, set the necessary remote GUC settings at connection startup. (We were already assuming that settings made then would persist throughout the remote session.) To fix data transmission to the remote server, temporarily force the relevant GUCs to the right values when we're about to convert any data values to text for transmission. This is all pretty grotty, and not very cheap either. It's tempting to think of defining one uber-GUC that would override any settings that might render printed data values unportable. But of course, older remote servers wouldn't know any such thing and would still need this logic. While at it, revert commit f7951eef89be78c50ea2241f593d76dfefe176c9, since this provides a real fix. (The timestamptz given in the error message returned from the "remote" server will now reliably be shown in UTC.)
2013-03-12 09:31:28 +08:00
/* in postgres_fdw.c */
extern int set_transmission_modes(void);
extern void reset_transmission_modes(int nestlevel);
Add postgres_fdw contrib module. There's still a lot of room for improvement, but it basically works, and we need this to be present before we can do anything much with the writable-foreign-tables patch. So let's commit it and get on with testing. Shigeru Hanada, reviewed by KaiGai Kohei and Tom Lane
2013-02-21 18:26:23 +08:00
/* in connection.c */
Avoid multiple foreign server connections when all use same user mapping. Previously, postgres_fdw's connection cache was keyed by user OID and server OID, but this can lead to multiple connections when it's not really necessary. In particular, if all relevant users are mapped to the public user mapping, then their connection options are certainly the same, so one connection can be used for all of them. While we're cleaning things up here, drop the "server" argument to GetConnection(), which isn't really needed. This saves a few cycles because callers no longer have to look this up; the function itself does, but only when establishing a new connection, not when reusing an existing one. Ashutosh Bapat, with a few small changes by me.
2016-01-29 01:05:19 +08:00
extern PGconn *GetConnection(UserMapping *user, bool will_prep_stmt);
Add postgres_fdw contrib module. There's still a lot of room for improvement, but it basically works, and we need this to be present before we can do anything much with the writable-foreign-tables patch. So let's commit it and get on with testing. Shigeru Hanada, reviewed by KaiGai Kohei and Tom Lane
2013-02-21 18:26:23 +08:00
extern void ReleaseConnection(PGconn *conn);
extern unsigned int GetCursorNumber(PGconn *conn);
Support writable foreign tables. This patch adds the core-system infrastructure needed to support updates on foreign tables, and extends contrib/postgres_fdw to allow updates against remote Postgres servers. There's still a great deal of room for improvement in optimization of remote updates, but at least there's basic functionality there now. KaiGai Kohei, reviewed by Alexander Korotkov and Laurenz Albe, and rather heavily revised by Tom Lane.
2013-03-11 02:14:53 +08:00
extern unsigned int GetPrepStmtNumber(PGconn *conn);
Allow queries submitted by postgres_fdw to be canceled. This fixes a problem which is not new, but with the advent of direct foreign table modification in 0bf3ae88af330496517722e391e7c975e6bad219, it's somewhat more likely to be annoying than previously. So, arrange for a local query cancelation to propagate to the remote side. Michael Paquier, reviewed by Etsuro Fujita. Original report by Thom Brown.
2016-04-21 22:46:09 +08:00
extern PGresult *pgfdw_get_result(PGconn *conn, const char *query);
extern PGresult *pgfdw_exec_query(PGconn *conn, const char *query);
Improve connection-failure error handling in contrib/postgres_fdw. postgres_fdw tended to say "unknown error" if it tried to execute a command on an already-dead connection, because some paths in libpq just return a null PGresult for such cases. Out-of-memory might result in that, too. To fix, pass the PGconn to pgfdw_report_error, and look at its PQerrorMessage() string if we can't get anything out of the PGresult. Also, fix the transaction-exit logic to reliably drop a dead connection. It was attempting to do that already, but it assumed that only connection cache entries with xact_depth > 0 needed to be examined. The folly in that is that if we fail while issuing START TRANSACTION, we'll not have bumped xact_depth. (At least for the case I was testing, this fix masks the other problem; but it still seems like a good idea to have the PGconn fallback logic.) Per investigation of bug #9087 from Craig Lucas. Backpatch to 9.3 where this code was introduced.
2014-02-04 10:30:02 +08:00
extern void pgfdw_report_error(int elevel, PGresult *res, PGconn *conn,
bool clear, const char *sql);
Add postgres_fdw contrib module. There's still a lot of room for improvement, but it basically works, and we need this to be present before we can do anything much with the writable-foreign-tables patch. So let's commit it and get on with testing. Shigeru Hanada, reviewed by KaiGai Kohei and Tom Lane
2013-02-21 18:26:23 +08:00
/* in option.c */
extern int ExtractConnectionOptions(List *defelems,
const char **keywords,
const char **values);
Allow postgres_fdw to ship extension funcs/operators for remote execution. The user can whitelist specified extension(s) in the foreign server's options, whereupon we will treat immutable functions and operators of those extensions as candidates to be sent for remote execution. Whitelisting an extension in this way basically promises that the extension exists on the remote server and behaves compatibly with the local instance. We have no way to prove that formally, so we have to rely on the user to get it right. But this seems like something that people can usually get right in practice. We might in future allow functions and operators to be whitelisted individually, but extension granularity is a very convenient special case, so it got done first. The patch as-committed lacks any regression tests, which is unfortunate, but introducing dependencies on other extensions for testing purposes would break "make installcheck" scenarios, which is worse. I have some ideas about klugy ways around that, but it seems like material for a separate patch. For the moment, leave the problem open. Paul Ramsey, hacked up a bit more by me
2015-11-04 07:42:02 +08:00
extern List *ExtractExtensionList(const char *extensionsString,
bool warnOnMissing);
Add postgres_fdw contrib module. There's still a lot of room for improvement, but it basically works, and we need this to be present before we can do anything much with the writable-foreign-tables patch. So let's commit it and get on with testing. Shigeru Hanada, reviewed by KaiGai Kohei and Tom Lane
2013-02-21 18:26:23 +08:00
/* in deparse.c */
extern void classifyConditions(PlannerInfo *root,
RelOptInfo *baserel,
Fix contrib/postgres_fdw to handle multiple join conditions properly. The previous coding supposed that it could consider just a single join condition in any one parameterized path for the foreign table. But in reality, the parameterized-path machinery forces all join clauses that are "movable to" the foreign table to be evaluated at that node; including clauses that we might not consider safe to send across. Such cases would result in an Assert failure in an assert-enabled build, and otherwise in sending an unsafe clause to the foreign server, which might result in errors or silently-wrong answers. A lesser problem was that the cost/rowcount estimates generated for the parameterized path failed to account for any additional join quals that get assigned to the scan. To fix, rewrite postgresGetForeignPaths so that it correctly collects all the movable quals for any one outer relation when generating parameterized paths; we'll now generate just one path per outer relation not one per join qual. Also fix bogus assumptions in postgresGetForeignPlan and estimate_path_cost_size that only safe-to-send join quals will be presented. Based on complaint from Etsuro Fujita that the path costs were being miscalculated, though this is significantly different from his proposed patch.
2014-03-08 05:35:58 +08:00
List *input_conds,
Add postgres_fdw contrib module. There's still a lot of room for improvement, but it basically works, and we need this to be present before we can do anything much with the writable-foreign-tables patch. So let's commit it and get on with testing. Shigeru Hanada, reviewed by KaiGai Kohei and Tom Lane
2013-02-21 18:26:23 +08:00
List **remote_conds,
Redo postgres_fdw's planner code so it can handle parameterized paths. I wasn't going to ship this without having at least some example of how to do that. This version isn't terribly bright; in particular it won't consider any combinations of multiple join clauses. Given the cost of executing a remote EXPLAIN, I'm not sure we want to be very aggressive about doing that, anyway. In support of this, refactor generate_implied_equalities_for_indexcol so that it can be used to extract equivalence clauses that aren't necessarily tied to an index.
2013-03-22 07:43:59 +08:00
List **local_conds);
extern bool is_foreign_expr(PlannerInfo *root,
RelOptInfo *baserel,
Expr *expr);
Avoid postgres_fdw crash for a targetlist entry that's just a Param. foreign_grouping_ok() is willing to put fairly arbitrary expressions into the targetlist of a remote SELECT that's doing grouping or aggregation on the remote side, including expressions that have no foreign component to them at all. This is possibly a bit dubious from an efficiency standpoint; but it rises to the level of a crash-causing bug if the expression is just a Param or non-foreign Var. In that case, the expression will necessarily also appear in the fdw_exprs list of values we need to send to the remote server, and then setrefs.c's set_foreignscan_references will mistakenly replace the fdw_exprs entry with a Var referencing the targetlist result. The root cause of this problem is bad design in commit e7cb7ee14: it put logic into set_foreignscan_references that IMV is postgres_fdw-specific, and yet this bug shows that it isn't postgres_fdw-specific enough. The transformation being done on fdw_exprs assumes that fdw_exprs is to be evaluated with the fdw_scan_tlist as input, which is not how postgres_fdw uses it; yet it could be the right thing for some other FDW. (In the bigger picture, setrefs.c has no business assuming this for the other expression fields of a ForeignScan either.) The right fix therefore would be to expand the FDW API so that the FDW could inform setrefs.c how it intends to evaluate these various expressions. We can't change that in the back branches though, and we also can't just summarily change setrefs.c's behavior there, or we're likely to break external FDWs. As a stopgap, therefore, hack up postgres_fdw so that it won't attempt to send targetlist entries that look exactly like the fdw_exprs entries they'd produce. In most cases this actually produces a superior plan, IMO, with less data needing to be transmitted and returned; so we probably ought to think harder about whether we should ship tlist expressions at all when they don't contain any foreign Vars or Aggs. But that's an optimization not a bug fix so I left it for later. One case where this produces an inferior plan is where the expression in question is actually a GROUP BY expression: then the restriction prevents us from using remote grouping. It might be possible to work around that (since that would reduce to group-by-a-constant on the remote side); but it seems like a pretty unlikely corner case, so I'm not sure it's worth expending effort solely to improve that. In any case the right long-term answer is to fix the API as sketched above, and then revert this hack. Per bug #15781 from Sean Johnston. Back-patch to v10 where the problem was introduced. Discussion: https://postgr.es/m/15781-2601b1002bad087c@postgresql.org
2019-04-28 01:15:54 +08:00
extern bool is_foreign_param(PlannerInfo *root,
RelOptInfo *baserel,
Expr *expr);
Fix postgres_fdw to check shippability of sort clauses properly. postgres_fdw would push ORDER BY clauses to the remote side without verifying that the sort operator is safe to ship. Moreover, it failed to print a suitable USING clause if the sort operator isn't default for the sort expression's type. The net result of this is that the remote sort might not have anywhere near the semantics we expect, which'd be disastrous for locally-performed merge joins in particular. We addressed similar issues in the context of ORDER BY within an aggregate function call in commit 7012b132d, but failed to notice that query-level ORDER BY was broken. Thus, much of the necessary logic already existed, but it requires refactoring to be usable in both cases. Back-patch to all supported branches. In HEAD only, remove the core code's copy of find_em_expr_for_rel, which is no longer used and really should never have been pushed into equivclass.c in the first place. Ronan Dunklau, per report from David Rowley; reviews by David Rowley, Ranier Vilela, and myself Discussion: https://postgr.es/m/CAApHDvr4OeC2DBVY--zVP83-K=bYrTD7F8SZDhN4g+pj2f2S-A@mail.gmail.com
2022-04-01 02:29:24 +08:00
extern bool is_foreign_pathkey(PlannerInfo *root,
RelOptInfo *baserel,
PathKey *pathkey);
Fix interaction of foreign tuple routing with remote triggers. Without these fixes, changes to the inserted tuple made by remote triggers are ignored when building local RETURNING tuples. In the core code, call ExecInitRoutingInfo at a later point from within ExecInitPartitionInfo so that the FDW callback gets invoked after the returning list has been built. But move CheckValidResultRel out of ExecInitRoutingInfo so that it can happen at an earlier stage. In postgres_fdw, refactor assorted deparsing functions to work with the RTE rather than the PlannerInfo, which saves us having to construct a fake PlannerInfo in cases where we don't have a real one. Then, we can pass down a constructed RTE that yields the correct deparse result when no real one exists. Unfortunately, this necessitates a hack that understands how the core code manages RT indexes for update tuple routing, which is ugly, but we don't have a better idea right now. Original report, analysis, and patch by Etsuro Fujita. Heavily refactored by me. Then worked over some more by Amit Langote. Discussion: http://postgr.es/m/5AD4882B.10002@lab.ntt.co.jp
2018-05-02 01:21:46 +08:00
extern void deparseInsertSql(StringInfo buf, RangeTblEntry *rte,
Fix contrib/postgres_fdw's handling of column defaults. Adopt the position that only locally-defined defaults matter. Any defaults defined in the remote database do not affect insertions performed through a foreign table (unless they are for columns not known to the foreign table). While it'd arguably be more useful to permit remote defaults to be used, making that work in a consistent fashion requires far more work than seems possible for 9.3.
2013-03-13 06:58:13 +08:00
Index rtindex, Relation rel,
Fix WITH CHECK OPTION on views referencing postgres_fdw tables. If a view references a foreign table, and the foreign table has a BEFORE INSERT trigger, then it's possible for a tuple inserted or updated through the view to be changed such that it violates the view's WITH CHECK OPTION constraint. Before this commit, postgres_fdw handled this case inconsistently. A RETURNING clause on the INSERT or UPDATE statement targeting the view would cause the finally-inserted tuple to be read back, and the WITH CHECK OPTION violation would throw an error. But without a RETURNING clause, postgres_fdw would not read the final tuple back, and WITH CHECK OPTION would not throw an error for the violation (or may throw an error when there is no real violation). AFTER ROW triggers on the foreign table had a similar effect as a RETURNING clause on the INSERT or UPDATE statement. To fix, this commit retrieves the attributes needed to enforce the WITH CHECK OPTION constraint along with the attributes needed for the RETURNING clause (if any) from the remote side. Thus, the WITH CHECK OPTION constraint is always evaluated against the final tuple after any triggers on the remote side. This fix may be considered inconsistent with CHECK constraints declared on foreign tables, which are not enforced locally at all (because the constraint is on a remote object). The discussion concluded that this difference is reasonable, because the WITH CHECK OPTION is a constraint on the local view (not any remote object); therefore it only makes sense to enforce its WITH CHECK OPTION constraint locally. Author: Etsuro Fujita Reviewed-by: Arthur Zakirov, Stephen Frost Discussion: https://www.postgresql.org/message-id/7eb58fab-fd3b-781b-ac33-f7cfec96021f%40lab.ntt.co.jp
2018-07-08 15:14:51 +08:00
List *targetAttrs, bool doNothing,
List *withCheckOptionList, List *returningList,
Avoid retrieving dummy NULL columns in postgres_fdw. This should provide some marginal overall savings, since it surely takes many more cycles for the remote server to deal with the NULL columns than it takes for postgres_fdw not to emit them. But really the reason is to keep the emitted queries from looking quite so silly ...
2013-03-22 12:31:11 +08:00
List **retrieved_attrs);
Fix interaction of foreign tuple routing with remote triggers. Without these fixes, changes to the inserted tuple made by remote triggers are ignored when building local RETURNING tuples. In the core code, call ExecInitRoutingInfo at a later point from within ExecInitPartitionInfo so that the FDW callback gets invoked after the returning list has been built. But move CheckValidResultRel out of ExecInitRoutingInfo so that it can happen at an earlier stage. In postgres_fdw, refactor assorted deparsing functions to work with the RTE rather than the PlannerInfo, which saves us having to construct a fake PlannerInfo in cases where we don't have a real one. Then, we can pass down a constructed RTE that yields the correct deparse result when no real one exists. Unfortunately, this necessitates a hack that understands how the core code manages RT indexes for update tuple routing, which is ugly, but we don't have a better idea right now. Original report, analysis, and patch by Etsuro Fujita. Heavily refactored by me. Then worked over some more by Amit Langote. Discussion: http://postgr.es/m/5AD4882B.10002@lab.ntt.co.jp
2018-05-02 01:21:46 +08:00
extern void deparseUpdateSql(StringInfo buf, RangeTblEntry *rte,
Fix contrib/postgres_fdw's handling of column defaults. Adopt the position that only locally-defined defaults matter. Any defaults defined in the remote database do not affect insertions performed through a foreign table (unless they are for columns not known to the foreign table). While it'd arguably be more useful to permit remote defaults to be used, making that work in a consistent fashion requires far more work than seems possible for 9.3.
2013-03-13 06:58:13 +08:00
Index rtindex, Relation rel,
Fix WITH CHECK OPTION on views referencing postgres_fdw tables. If a view references a foreign table, and the foreign table has a BEFORE INSERT trigger, then it's possible for a tuple inserted or updated through the view to be changed such that it violates the view's WITH CHECK OPTION constraint. Before this commit, postgres_fdw handled this case inconsistently. A RETURNING clause on the INSERT or UPDATE statement targeting the view would cause the finally-inserted tuple to be read back, and the WITH CHECK OPTION violation would throw an error. But without a RETURNING clause, postgres_fdw would not read the final tuple back, and WITH CHECK OPTION would not throw an error for the violation (or may throw an error when there is no real violation). AFTER ROW triggers on the foreign table had a similar effect as a RETURNING clause on the INSERT or UPDATE statement. To fix, this commit retrieves the attributes needed to enforce the WITH CHECK OPTION constraint along with the attributes needed for the RETURNING clause (if any) from the remote side. Thus, the WITH CHECK OPTION constraint is always evaluated against the final tuple after any triggers on the remote side. This fix may be considered inconsistent with CHECK constraints declared on foreign tables, which are not enforced locally at all (because the constraint is on a remote object). The discussion concluded that this difference is reasonable, because the WITH CHECK OPTION is a constraint on the local view (not any remote object); therefore it only makes sense to enforce its WITH CHECK OPTION constraint locally. Author: Etsuro Fujita Reviewed-by: Arthur Zakirov, Stephen Frost Discussion: https://www.postgresql.org/message-id/7eb58fab-fd3b-781b-ac33-f7cfec96021f%40lab.ntt.co.jp
2018-07-08 15:14:51 +08:00
List *targetAttrs,
List *withCheckOptionList, List *returningList,
Avoid retrieving dummy NULL columns in postgres_fdw. This should provide some marginal overall savings, since it surely takes many more cycles for the remote server to deal with the NULL columns than it takes for postgres_fdw not to emit them. But really the reason is to keep the emitted queries from looking quite so silly ...
2013-03-22 12:31:11 +08:00
List **retrieved_attrs);
Directly modify foreign tables. postgres_fdw can now sent an UPDATE or DELETE statement directly to the foreign server in simple cases, rather than sending a SELECT FOR UPDATE statement and then updating or deleting rows one-by-one. Etsuro Fujita, reviewed by Rushabh Lathia, Shigeru Hanada, Kyotaro Horiguchi, Albe Laurenz, Thom Brown, and me.
2016-03-19 01:48:58 +08:00
extern void deparseDirectUpdateSql(StringInfo buf, PlannerInfo *root,
Index rtindex, Relation rel,
postgres_fdw: Push down UPDATE/DELETE joins to remote servers. Commit 0bf3ae88af330496517722e391e7c975e6bad219 allowed direct foreign table modification; instead of fetching each row, updating it locally, and then pushing the modification back to the remote side, we would instead do all the work on the remote server via a single remote UPDATE or DELETE command. However, that commit only enabled this optimization when join tree consisted only of the target table. This change allows the same optimization when an UPDATE statement has a FROM clause or a DELETE statement has a USING clause. This works much like ordinary foreign join pushdown, in that the tables must be on the same remote server, relevant parts of the query must be pushdown-safe, and so forth. Etsuro Fujita, reviewed by Ashutosh Bapat, Rushabh Lathia, and me. Some formatting corrections by me. Discussion: http://postgr.es/m/5A57193A.2080003@lab.ntt.co.jp Discussion: http://postgr.es/m/b9cee735-62f8-6c07-7528-6364ce9347d0@lab.ntt.co.jp
2018-02-08 04:34:30 +08:00
RelOptInfo *foreignrel,
Directly modify foreign tables. postgres_fdw can now sent an UPDATE or DELETE statement directly to the foreign server in simple cases, rather than sending a SELECT FOR UPDATE statement and then updating or deleting rows one-by-one. Etsuro Fujita, reviewed by Rushabh Lathia, Shigeru Hanada, Kyotaro Horiguchi, Albe Laurenz, Thom Brown, and me.
2016-03-19 01:48:58 +08:00
List *targetlist,
List *targetAttrs,
List *remote_conds,
List **params_list,
List *returningList,
List **retrieved_attrs);
Fix interaction of foreign tuple routing with remote triggers. Without these fixes, changes to the inserted tuple made by remote triggers are ignored when building local RETURNING tuples. In the core code, call ExecInitRoutingInfo at a later point from within ExecInitPartitionInfo so that the FDW callback gets invoked after the returning list has been built. But move CheckValidResultRel out of ExecInitRoutingInfo so that it can happen at an earlier stage. In postgres_fdw, refactor assorted deparsing functions to work with the RTE rather than the PlannerInfo, which saves us having to construct a fake PlannerInfo in cases where we don't have a real one. Then, we can pass down a constructed RTE that yields the correct deparse result when no real one exists. Unfortunately, this necessitates a hack that understands how the core code manages RT indexes for update tuple routing, which is ugly, but we don't have a better idea right now. Original report, analysis, and patch by Etsuro Fujita. Heavily refactored by me. Then worked over some more by Amit Langote. Discussion: http://postgr.es/m/5AD4882B.10002@lab.ntt.co.jp
2018-05-02 01:21:46 +08:00
extern void deparseDeleteSql(StringInfo buf, RangeTblEntry *rte,
Fix contrib/postgres_fdw's handling of column defaults. Adopt the position that only locally-defined defaults matter. Any defaults defined in the remote database do not affect insertions performed through a foreign table (unless they are for columns not known to the foreign table). While it'd arguably be more useful to permit remote defaults to be used, making that work in a consistent fashion requires far more work than seems possible for 9.3.
2013-03-13 06:58:13 +08:00
Index rtindex, Relation rel,
Avoid retrieving dummy NULL columns in postgres_fdw. This should provide some marginal overall savings, since it surely takes many more cycles for the remote server to deal with the NULL columns than it takes for postgres_fdw not to emit them. But really the reason is to keep the emitted queries from looking quite so silly ...
2013-03-22 12:31:11 +08:00
List *returningList,
List **retrieved_attrs);
Directly modify foreign tables. postgres_fdw can now sent an UPDATE or DELETE statement directly to the foreign server in simple cases, rather than sending a SELECT FOR UPDATE statement and then updating or deleting rows one-by-one. Etsuro Fujita, reviewed by Rushabh Lathia, Shigeru Hanada, Kyotaro Horiguchi, Albe Laurenz, Thom Brown, and me.
2016-03-19 01:48:58 +08:00
extern void deparseDirectDeleteSql(StringInfo buf, PlannerInfo *root,
Index rtindex, Relation rel,
postgres_fdw: Push down UPDATE/DELETE joins to remote servers. Commit 0bf3ae88af330496517722e391e7c975e6bad219 allowed direct foreign table modification; instead of fetching each row, updating it locally, and then pushing the modification back to the remote side, we would instead do all the work on the remote server via a single remote UPDATE or DELETE command. However, that commit only enabled this optimization when join tree consisted only of the target table. This change allows the same optimization when an UPDATE statement has a FROM clause or a DELETE statement has a USING clause. This works much like ordinary foreign join pushdown, in that the tables must be on the same remote server, relevant parts of the query must be pushdown-safe, and so forth. Etsuro Fujita, reviewed by Ashutosh Bapat, Rushabh Lathia, and me. Some formatting corrections by me. Discussion: http://postgr.es/m/5A57193A.2080003@lab.ntt.co.jp Discussion: http://postgr.es/m/b9cee735-62f8-6c07-7528-6364ce9347d0@lab.ntt.co.jp
2018-02-08 04:34:30 +08:00
RelOptInfo *foreignrel,
Directly modify foreign tables. postgres_fdw can now sent an UPDATE or DELETE statement directly to the foreign server in simple cases, rather than sending a SELECT FOR UPDATE statement and then updating or deleting rows one-by-one. Etsuro Fujita, reviewed by Rushabh Lathia, Shigeru Hanada, Kyotaro Horiguchi, Albe Laurenz, Thom Brown, and me.
2016-03-19 01:48:58 +08:00
List *remote_conds,
List **params_list,
List *returningList,
List **retrieved_attrs);
Fix some planning oversights in postgres_fdw. Include eval costs of local conditions in remote-estimate mode, and don't assume the remote eval cost is zero in local-estimate mode. (The best we can do with that at the moment is to assume a seqscan, which may well be wildly pessimistic ... but zero won't do at all.) To get a reasonable local estimate, we need to know the relpages count for the remote rel, so improve the ANALYZE code to fetch that rather than just setting the foreign table's relpages field to zero.
2013-02-22 23:56:06 +08:00
extern void deparseAnalyzeSizeSql(StringInfo buf, Relation rel);
Avoid retrieving dummy NULL columns in postgres_fdw. This should provide some marginal overall savings, since it surely takes many more cycles for the remote server to deal with the NULL columns than it takes for postgres_fdw not to emit them. But really the reason is to keep the emitted queries from looking quite so silly ...
2013-03-22 12:31:11 +08:00
extern void deparseAnalyzeSql(StringInfo buf, Relation rel,
List **retrieved_attrs);
Implement IMPORT FOREIGN SCHEMA. This command provides an automated way to create foreign table definitions that match remote tables, thereby reducing tedium and chances for error. In this patch, we provide the necessary core-server infrastructure and implement the feature fully in the postgres_fdw foreign-data wrapper. Other wrappers will throw a "feature not supported" error until/unless they are updated. Ronan Dunklau and Michael Paquier, additional work by me
2014-07-11 03:01:31 +08:00
extern void deparseStringLiteral(StringInfo buf, const char *val);
Fix postgres_fdw to check shippability of sort clauses properly. postgres_fdw would push ORDER BY clauses to the remote side without verifying that the sort operator is safe to ship. Moreover, it failed to print a suitable USING clause if the sort operator isn't default for the sort expression's type. The net result of this is that the remote sort might not have anywhere near the semantics we expect, which'd be disastrous for locally-performed merge joins in particular. We addressed similar issues in the context of ORDER BY within an aggregate function call in commit 7012b132d, but failed to notice that query-level ORDER BY was broken. Thus, much of the necessary logic already existed, but it requires refactoring to be usable in both cases. Back-patch to all supported branches. In HEAD only, remove the core code's copy of find_em_expr_for_rel, which is no longer used and really should never have been pushed into equivclass.c in the first place. Ronan Dunklau, per report from David Rowley; reviews by David Rowley, Ranier Vilela, and myself Discussion: https://postgr.es/m/CAApHDvr4OeC2DBVY--zVP83-K=bYrTD7F8SZDhN4g+pj2f2S-A@mail.gmail.com
2022-04-01 02:29:24 +08:00
extern EquivalenceMember *find_em_for_rel(PlannerInfo *root,
EquivalenceClass *ec,
RelOptInfo *rel);
extern EquivalenceMember *find_em_for_rel_target(PlannerInfo *root,
EquivalenceClass *ec,
RelOptInfo *rel);
postgres_fdw: Push down aggregates to remote servers. Now that the upper planner uses paths, and now that we have proper hooks to inject paths into the upper planning process, it's possible for foreign data wrappers to arrange to push aggregates to the remote side instead of fetching all of the rows and aggregating them locally. This figures to be a massive win for performance, so teach postgres_fdw to do it. Jeevan Chalke and Ashutosh Bapat. Reviewed by Ashutosh Bapat with additional testing by Prabhat Sahu. Various mostly cosmetic changes by me.
2016-10-21 21:54:29 +08:00
extern List *build_tlist_to_deparse(RelOptInfo *foreignrel);
postgres_fdw: More preliminary refactoring for upcoming join pushdown. The code that generates a complete SQL query for a given foreign relation was repeated in two places, and they didn't quite agree: the EXPLAIN case left out the locking clause. Centralize the code so we get the same behavior everywhere, and adjust calling conventions and which functions are static vs. extern accordingly . Centralize the code so we get the same behavior everywhere, and adjust calling conventions and which functions are static vs. extern accordingly. Ashutosh Bapat, reviewed and slightly adjusted by me.
2016-01-30 23:32:38 +08:00
extern void deparseSelectStmtForRel(StringInfo buf, PlannerInfo *root,
postgres_fdw: Push down joins to remote servers. If we've got a relatively straightforward join between two tables, this pushes that join down to the remote server instead of fetching the rows for each table and performing the join locally. Some cases are not handled yet, such as SEMI and ANTI joins. Also, we don't yet attempt to create presorted join paths or parameterized join paths even though these options do get tried for a base relation scan. Nevertheless, this seems likely to be a very significant win in many practical cases. Shigeru Hanada and Ashutosh Bapat, reviewed by Robert Haas, with additional review at various points by Tom Lane, Etsuro Fujita, KaiGai Kohei, and Jeevan Chalke.
2016-02-10 03:00:50 +08:00
RelOptInfo *foreignrel, List *tlist,
postgres_fdw: Perform the (ORDERED, NULL) upperrel operations remotely. The upper-planner pathification allows FDWs to arrange to push down different types of upper-stage operations to the remote side. This commit teaches postgres_fdw to do it for the (ORDERED, NULL) upperrel, which is responsible for evaluating the query's ORDER BY ordering. Since postgres_fdw is already able to evaluate that ordering remotely for foreign baserels and foreign joinrels (see commit aa09cd242f et al.), this adds support for that for foreign grouping relations. Author: Etsuro Fujita Reviewed-By: Antonin Houska and Jeff Janes Discussion: https://postgr.es/m/87pnz1aby9.fsf@news-spur.riddles.org.uk
2019-04-02 18:20:30 +08:00
List *remote_conds, List *pathkeys,
postgres_fdw: Perform the (FINAL, NULL) upperrel operations remotely. The upper-planner pathification allows FDWs to arrange to push down different types of upper-stage operations to the remote side. This commit teaches postgres_fdw to do it for the (FINAL, NULL) upperrel, which is responsible for doing LockRows, LIMIT, and/or ModifyTable. This provides the ability for postgres_fdw to handle SELECT commands so that it 1) skips the LockRows step (if any) (note that this is safe since it performs early locking) and 2) pushes down the LIMIT and/or OFFSET restrictions (if any) to the remote side. This doesn't handle the INSERT/UPDATE/DELETE cases. Author: Etsuro Fujita Reviewed-By: Antonin Houska and Jeff Janes Discussion: https://postgr.es/m/87pnz1aby9.fsf@news-spur.riddles.org.uk
2019-04-02 19:30:45 +08:00
bool has_final_sort, bool has_limit,
bool is_subquery,
postgres_fdw: More preliminary refactoring for upcoming join pushdown. The code that generates a complete SQL query for a given foreign relation was repeated in two places, and they didn't quite agree: the EXPLAIN case left out the locking clause. Centralize the code so we get the same behavior everywhere, and adjust calling conventions and which functions are static vs. extern accordingly . Centralize the code so we get the same behavior everywhere, and adjust calling conventions and which functions are static vs. extern accordingly. Ashutosh Bapat, reviewed and slightly adjusted by me.
2016-01-30 23:32:38 +08:00
List **retrieved_attrs, List **params_list);
postgres_fdw: Move function prototype to correct section. Etsuro Fujita, reviewed by Ashutosh Bapat. Discussion: http://postgr.es/m/93a9c487-9920-a38f-da96-503422c50f59@lab.ntt.co.jp
2017-06-23 00:44:53 +08:00
extern const char *get_jointype_name(JoinType jointype);
Allow postgres_fdw to ship extension funcs/operators for remote execution. The user can whitelist specified extension(s) in the foreign server's options, whereupon we will treat immutable functions and operators of those extensions as candidates to be sent for remote execution. Whitelisting an extension in this way basically promises that the extension exists on the remote server and behaves compatibly with the local instance. We have no way to prove that formally, so we have to rely on the user to get it right. But this seems like something that people can usually get right in practice. We might in future allow functions and operators to be whitelisted individually, but extension granularity is a very convenient special case, so it got done first. The patch as-committed lacks any regression tests, which is unfortunate, but introducing dependencies on other extensions for testing purposes would break "make installcheck" scenarios, which is worse. I have some ideas about klugy ways around that, but it seems like material for a separate patch. For the moment, leave the problem open. Paul Ramsey, hacked up a bit more by me
2015-11-04 07:42:02 +08:00
/* in shippable.c */
extern bool is_builtin(Oid objectId);
extern bool is_shippable(Oid objectId, Oid classId, PgFdwRelationInfo *fpinfo);
Add postgres_fdw contrib module. There's still a lot of room for improvement, but it basically works, and we need this to be present before we can do anything much with the writable-foreign-tables patch. So let's commit it and get on with testing. Shigeru Hanada, reviewed by KaiGai Kohei and Tom Lane
2013-02-21 18:26:23 +08:00
#endif /* POSTGRES_FDW_H */
Reference in New Issue Copy Permalink