mirror/postgresql
2
0
Fork 0
mirror of https://git.postgresql.org/git/postgresql.git synced 2024-12-21 08:29:39 +08:00
Code Issues Packages Projects Releases Wiki Activity
a9e9abebd9
postgresql/doc/TODO.detail/crossdb
Bruce Momjian 3e87bfc1f1 Add crossdb TODO.detail item.
2001-12-29 17:58:15 +00:00

772 lines
33 KiB
Plaintext
Raw Blame History

From pgsql-hackers-owner+M16329=candle.pha.pa.us=pgman@postgresql.org Thu Dec 6 13:31:28 2001
Return-path: <pgsql-hackers-owner+M16329=candle.pha.pa.us=pgman@postgresql.org>
Received: from west.navpoint.com (west.navpoint.com [207.106.42.13])
by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB6IVRZ13376
for <pgman@candle.pha.pa.us>; Thu, 6 Dec 2001 13:31:27 -0500 (EST)
Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged))
by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB6IVQa26949
for <pgman@candle.pha.pa.us>; Thu, 6 Dec 2001 13:31:26 -0500 (EST)
Received: from postgresql.org (postgresql.org [64.49.215.8])
by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB6IRvR61959
for <pgman@candle.pha.pa.us>; Thu, 6 Dec 2001 12:29:06 -0600 (CST)
(envelope-from pgsql-hackers-owner+M16329=candle.pha.pa.us=pgman@postgresql.org)
Received: from gromit.dotclick.com (ipn9-f8366.net-resource.net [216.204.83.66])
by postgresql.org (8.11.3/8.11.4) with ESMTP id fB6IQ2m78192
for <pgsql-hackers@postgresql.org>; Thu, 6 Dec 2001 13:26:05 -0500 (EST)
(envelope-from markw@mohawksoft.com)
Received: from mohawksoft.com (IDENT:markw@localhost.localdomain [127.0.0.1])
by gromit.dotclick.com (8.9.3/8.9.3) with ESMTP id NAA10076
for <pgsql-hackers@postgresql.org>; Thu, 6 Dec 2001 13:28:04 -0500
Message-ID: <3C0FB8B4.382C7736@mohawksoft.com>
Date: Thu, 06 Dec 2001 13:28:04 -0500
From: mlw <markw@mohawksoft.com>
X-Mailer: Mozilla 4.75 [en] (X11; U; Linux 2.4.16 i686)
X-Accept-Language: en
MIME-Version: 1.0
To: "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: [HACKERS] Remote connections?
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR
I just found out something about Oracle which that looks like something
that could be doable in PostgreSQL.
What do you all think:
Oracle's version is something like this:
create [public] database link using [...]
select * from sometable@remotelink
I was thinking how this could be done with postgreSQL. How hard would it
be to make something that is similar to a view, but executes a query
remotely? I envision something like this:
create [public] link name query using [...]
The table link will be similar to a view. It could be used like this:
CREATE LINK test as select * from test WITH 'user=postgres host=remote
db=data';
SELECT * from test;
or
SELECT * from fubar join test on (fubar.id = test.id) ;
So, what do you think? Impossible, possible, too hard? too easy?
---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
From pgsql-hackers-owner+M16331=candle.pha.pa.us=pgman@postgresql.org Thu Dec 6 15:12:28 2001
Return-path: <pgsql-hackers-owner+M16331=candle.pha.pa.us=pgman@postgresql.org>
Received: from west.navpoint.com (west.navpoint.com [207.106.42.13])
by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB6KCQZ19987
for <pgman@candle.pha.pa.us>; Thu, 6 Dec 2001 15:12:27 -0500 (EST)
Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged))
by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB6KCQa13967
for <pgman@candle.pha.pa.us>; Thu, 6 Dec 2001 15:12:26 -0500 (EST)
Received: from postgresql.org (postgresql.org [64.49.215.8])
by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB6K6nR65020
for <pgman@candle.pha.pa.us>; Thu, 6 Dec 2001 14:07:54 -0600 (CST)
(envelope-from pgsql-hackers-owner+M16331=candle.pha.pa.us=pgman@postgresql.org)
Received: from ece.rice.edu (ece.rice.edu [128.42.4.34])
by postgresql.org (8.11.3/8.11.4) with ESMTP id fB6K6Im96910
for <pgsql-hackers@postgresql.org>; Thu, 6 Dec 2001 15:06:18 -0500 (EST)
(envelope-from reedstrm@rice.edu)
Received: from localhost (localhost [127.0.0.1])
by ece.rice.edu (Postfix) with ESMTP
id A9E4E68A1D; Thu, 6 Dec 2001 14:06:17 -0600 (CST)
Received: from wallace.ece.rice.edu (wallace.ece.rice.edu [128.42.12.154])
by ece.rice.edu (Postfix) with ESMTP
id EA06668A17; Thu, 6 Dec 2001 14:06:16 -0600 (CST)
Received: from reedstrm by wallace.ece.rice.edu with local (Exim 3.22 #1 (Debian))
id 16C4me-0002uX-00; Thu, 06 Dec 2001 14:06:16 -0600
Date: Thu, 6 Dec 2001 14:06:16 -0600
From: "Ross J. Reedstrom" <reedstrm@rice.edu>
To: mlw <markw@mohawksoft.com>
cc: "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: [HACKERS] Remote connections?
Message-ID: <20011206140616.C10995@rice.edu>
Mail-Followup-To: "Ross J. Reedstrom" <reedstrm@ece.rice.edu>,
mlw <markw@mohawksoft.com>,
"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
References: <3C0FB8B4.382C7736@mohawksoft.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <3C0FB8B4.382C7736@mohawksoft.com>
User-Agent: Mutt/1.3.18i
X-Virus-Scanned: by AMaViS snapshot-20010714
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR
On Thu, Dec 06, 2001 at 01:28:04PM -0500, mlw wrote:
> I just found out something about Oracle which that looks like something
> that could be doable in PostgreSQL.
>
> What do you all think:
>
> Oracle's version is something like this:
>
> create [public] database link using [...]
>
> select * from sometable@remotelink
>
>
> I was thinking how this could be done with postgreSQL. How hard would it
> be to make something that is similar to a view, but executes a query
> remotely? I envision something like this:
>
> create [public] link name query using [...]
>
> The table link will be similar to a view. It could be used like this:
>
> CREATE LINK test as select * from test WITH 'user=postgres host=remote
> db=data';
>
> SELECT * from test;
>
> or
>
> SELECT * from fubar join test on (fubar.id = test.id) ;
>
> So, what do you think? Impossible, possible, too hard? too easy?
Here we come, full circle. This is just about where I came on board.
Many moons ago, I started looking at Mariposa, in the hopes of forward
patching it into PostgreSQL, and generalizing the 'remote' part to allow
exactly the sort of access you described above.
The biggest problem with this is transactional semantics: you need
two-stage commits to get this right, and we don't hav'em. (Has there
been an indepth discussion concerning what how hard it would be to do
that with postgresql?)
The _actual_ biggest problem was my lack of knowledge of the PostgreSQL
codebase ;-)
Ross
--
Ross Reedstrom, Ph.D. reedstrm@rice.edu
Executive Director phone: 713-348-6166
Gulf Coast Consortium for Bioinformatics fax: 713-348-6182
Rice University MS-39
Houston, TX 77005
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
From pgsql-hackers-owner+M16332=candle.pha.pa.us=pgman@postgresql.org Thu Dec 6 15:31:27 2001
Return-path: <pgsql-hackers-owner+M16332=candle.pha.pa.us=pgman@postgresql.org>
Received: from west.navpoint.com (west.navpoint.com [207.106.42.13])
by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB6KVQZ21158
for <pgman@candle.pha.pa.us>; Thu, 6 Dec 2001 15:31:26 -0500 (EST)
Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged))
by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB6KVQa22900
for <pgman@candle.pha.pa.us>; Thu, 6 Dec 2001 15:31:26 -0500 (EST)
Received: from postgresql.org (postgresql.org [64.49.215.8])
by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB6KRrR65596
for <pgman@candle.pha.pa.us>; Thu, 6 Dec 2001 14:28:55 -0600 (CST)
(envelope-from pgsql-hackers-owner+M16332=candle.pha.pa.us=pgman@postgresql.org)
Received: from gromit.dotclick.com (ipn9-f8366.net-resource.net [216.204.83.66])
by postgresql.org (8.11.3/8.11.4) with ESMTP id fB6KJXm97564
for <pgsql-hackers@postgresql.org>; Thu, 6 Dec 2001 15:19:33 -0500 (EST)
(envelope-from markw@mohawksoft.com)
Received: from mohawksoft.com (IDENT:markw@localhost.localdomain [127.0.0.1])
by gromit.dotclick.com (8.9.3/8.9.3) with ESMTP id PAA10771;
Thu, 6 Dec 2001 15:21:13 -0500
Message-ID: <3C0FD339.6F663329@mohawksoft.com>
Date: Thu, 06 Dec 2001 15:21:13 -0500
From: mlw <markw@mohawksoft.com>
X-Mailer: Mozilla 4.75 [en] (X11; U; Linux 2.4.16 i686)
X-Accept-Language: en
MIME-Version: 1.0
To: "Ross J. Reedstrom" <reedstrm@rice.edu>
cc: "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: [HACKERS] Remote connections?
References: <3C0FB8B4.382C7736@mohawksoft.com> <20011206140616.C10995@rice.edu>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR
"Ross J. Reedstrom" wrote:
>
> On Thu, Dec 06, 2001 at 01:28:04PM -0500, mlw wrote:
> > I just found out something about Oracle which that looks like something
> > that could be doable in PostgreSQL.
> >
> > What do you all think:
> >
> > Oracle's version is something like this:
> >
> > create [public] database link using [...]
> >
> > select * from sometable@remotelink
> >
> >
> > I was thinking how this could be done with postgreSQL. How hard would it
> > be to make something that is similar to a view, but executes a query
> > remotely? I envision something like this:
> >
> > create [public] link name query using [...]
> >
> > The table link will be similar to a view. It could be used like this:
> >
> > CREATE LINK test as select * from test WITH 'user=postgres host=remote
> > db=data';
> >
> > SELECT * from test;
> >
> > or
> >
> > SELECT * from fubar join test on (fubar.id = test.id) ;
> >
> > So, what do you think? Impossible, possible, too hard? too easy?
>
> Here we come, full circle. This is just about where I came on board.
> Many moons ago, I started looking at Mariposa, in the hopes of forward
> patching it into PostgreSQL, and generalizing the 'remote' part to allow
> exactly the sort of access you described above.
>
> The biggest problem with this is transactional semantics: you need
> two-stage commits to get this right, and we don't hav'em. (Has there
> been an indepth discussion concerning what how hard it would be to do
> that with postgresql?)
>
> The _actual_ biggest problem was my lack of knowledge of the PostgreSQL
> codebase ;-)
I think we can we can dispense worrying about two stage commits, if we
assume that remote connections are treated as views with no rules. As
long as remote tables are "read only" then the implementation is much
easier.
I too find the internals of PostgreSQL virtually incomprehensible at the
internal level. If there were a document somewhere which published how a
function could return multiple tuples, remote views would be a trivial
undertaking. It could look like:
select * from remote('select *from table', 'user=postgres host=outland
db=remote');
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
From pgsql-hackers-owner+M16335=candle.pha.pa.us=pgman@postgresql.org Thu Dec 6 17:11:29 2001
Return-path: <pgsql-hackers-owner+M16335=candle.pha.pa.us=pgman@postgresql.org>
Received: from west.navpoint.com (west.navpoint.com [207.106.42.13])
by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB6MBSZ06676
for <pgman@candle.pha.pa.us>; Thu, 6 Dec 2001 17:11:28 -0500 (EST)
Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged))
by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB6MBSq07059
for <pgman@candle.pha.pa.us>; Thu, 6 Dec 2001 17:11:28 -0500 (EST)
Received: from postgresql.org (postgresql.org [64.49.215.8])
by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB6M6sR68489
for <pgman@candle.pha.pa.us>; Thu, 6 Dec 2001 16:08:16 -0600 (CST)
(envelope-from pgsql-hackers-owner+M16335=candle.pha.pa.us=pgman@postgresql.org)
Received: from mx1.relaypoint.net (ns2.generalbroadband.com [64.32.62.5])
by postgresql.org (8.11.3/8.11.4) with ESMTP id fB6M3Im04451
for <pgsql-hackers@postgresql.org>; Thu, 6 Dec 2001 17:03:18 -0500 (EST)
(envelope-from joseph.conway@home.com)
Received: from [206.19.64.3] (account jconway@wwc.com HELO home.com)
by mx1.relaypoint.net (CommuniGate Pro SMTP 3.4.8)
with ESMTP id 1707032; Thu, 06 Dec 2001 14:03:14 -0800
Message-ID: <3C0FEB2C.70000@home.com>
Date: Thu, 06 Dec 2001 14:03:24 -0800
From: Joe Conway <joseph.conway@home.com>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.6+) Gecko/20011126
X-Accept-Language: en-us
MIME-Version: 1.0
To: mlw <markw@mohawksoft.com>
cc: "Ross J. Reedstrom" <reedstrm@rice.edu>,
"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: [HACKERS] Remote connections?
References: <3C0FB8B4.382C7736@mohawksoft.com> <20011206140616.C10995@rice.edu> <3C0FD339.6F663329@mohawksoft.com>
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR
mlw wrote:
> I too find the internals of PostgreSQL virtually incomprehensible at the
> internal level. If there were a document somewhere which published how a
> function could return multiple tuples, remote views would be a trivial
> undertaking. It could look like:
>
> select * from remote('select *from table', 'user=postgres host=outland
> db=remote');
>
See contrib/dblink in the 7.2 beta. It was my attempt inspired by
Oracle's dblink and some code that you (mlw) posted a while back. Based
on the limitations wrt returning muliple tuples, I wound up with
something like:
lt_lcat=# select dblink_tok(t1.dblink_p,0) as f1 from (select
dblink('hostaddr=127.0.0.1 dbname=template1 user=postgres
password=postgres','select proname from pg_proc') as dblink_p) as t1;
Which, as has been pointed out more than once, is pretty ugly, but at
least it's a start ;-)
Joe
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
From pgsql-hackers-owner+M16336=candle.pha.pa.us=pgman@postgresql.org Thu Dec 6 18:41:31 2001
Return-path: <pgsql-hackers-owner+M16336=candle.pha.pa.us=pgman@postgresql.org>
Received: from west.navpoint.com (west.navpoint.com [207.106.42.13])
by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB6NfPZ12249
for <pgman@candle.pha.pa.us>; Thu, 6 Dec 2001 18:41:25 -0500 (EST)
Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged))
by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB6NfQq03244
for <pgman@candle.pha.pa.us>; Thu, 6 Dec 2001 18:41:26 -0500 (EST)
Received: from postgresql.org (postgresql.org [64.49.215.8])
by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB6NbOR70960
for <pgman@candle.pha.pa.us>; Thu, 6 Dec 2001 17:38:19 -0600 (CST)
(envelope-from pgsql-hackers-owner+M16336=candle.pha.pa.us=pgman@postgresql.org)
Received: from spider.pilosoft.com (p55-222.acedsl.com [160.79.55.222])
by postgresql.org (8.11.3/8.11.4) with ESMTP id fB6NIgm07232
for <pgsql-hackers@postgresql.org>; Thu, 6 Dec 2001 18:18:43 -0500 (EST)
(envelope-from alex@pilosoft.com)
Received: from localhost (alexmail@localhost)
by spider.pilosoft.com (8.9.3/8.9.3) with ESMTP id SAA23999;
Thu, 6 Dec 2001 18:23:22 -0500 (EST)
Date: Thu, 6 Dec 2001 18:23:22 -0500 (EST)
From: Alex Pilosov <alex@pilosoft.com>
To: mlw <markw@mohawksoft.com>
cc: "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: [HACKERS] Remote connections?
In-Reply-To: <3C0FD339.6F663329@mohawksoft.com>
Message-ID: <Pine.BSO.4.10.10112061822080.22618-100000@spider.pilosoft.com>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR
On Thu, 6 Dec 2001, mlw wrote:
> I too find the internals of PostgreSQL virtually incomprehensible at the
> internal level. If there were a document somewhere which published how a
> function could return multiple tuples, remote views would be a trivial
> undertaking. It could look like:
>
> select * from remote('select *from table', 'user=postgres host=outland
> db=remote');
This isn't possible yet. I was working on implementation of this, about
80% done, but never finished. Now I'm out of time to work more on this for
a while. :(
Let me know if you want my code.
-alex
---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?
http://archives.postgresql.org
From pgsql-hackers-owner+M16340=candle.pha.pa.us=pgman@postgresql.org Fri Dec 7 00:32:59 2001
Return-path: <pgsql-hackers-owner+M16340=candle.pha.pa.us=pgman@postgresql.org>
Received: from west.navpoint.com (west.navpoint.com [207.106.42.13])
by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB75WsZ06911
for <pgman@candle.pha.pa.us>; Fri, 7 Dec 2001 00:32:54 -0500 (EST)
Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged))
by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB75Wtq16801
for <pgman@candle.pha.pa.us>; Fri, 7 Dec 2001 00:32:55 -0500 (EST)
Received: from postgresql.org (postgresql.org [64.49.215.8])
by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB75SCR80525
for <pgman@candle.pha.pa.us>; Thu, 6 Dec 2001 23:29:17 -0600 (CST)
(envelope-from pgsql-hackers-owner+M16340=candle.pha.pa.us=pgman@postgresql.org)
Received: from snoopy.mohawksoft.com (h0050bf7a618d.ne.mediaone.net [24.218.67.153])
by postgresql.org (8.11.3/8.11.4) with ESMTP id fB7593m27913
for <pgsql-hackers@postgresql.org>; Fri, 7 Dec 2001 00:09:03 -0500 (EST)
(envelope-from markw@mohawksoft.com)
Received: from mohawksoft.com (localhost [127.0.0.1])
by snoopy.mohawksoft.com (8.11.6/8.11.6) with ESMTP id fB7561030613;
Fri, 7 Dec 2001 00:06:01 -0500
Message-ID: <3C104E38.DA19C867@mohawksoft.com>
Date: Fri, 07 Dec 2001 00:06:01 -0500
From: mlw <markw@mohawksoft.com>
X-Mailer: Mozilla 4.78 [en] (X11; U; Linux 2.4.14ext3 i686)
X-Accept-Language: en
MIME-Version: 1.0
To: Joe Conway <joseph.conway@home.com>
cc: "Ross J. Reedstrom" <reedstrm@rice.edu>,
"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: [HACKERS] Remote connections?
References: <3C0FB8B4.382C7736@mohawksoft.com> <20011206140616.C10995@rice.edu> <3C0FD339.6F663329@mohawksoft.com> <3C0FEB2C.70000@home.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR
Hey this looks really cool. It looks like something I was thinking about doing.
I have a couple suggestions that could make it a little better, I hope you will
not be offended. (If you want my help, I'll chip in!)
Why not use a binary cursor? That way native types can slip through without the
overhead of conversion.
Right now you get all rows up front, you may be able to increase overall
performance by fetching only a few rows at a time, rather than get everything
all at once. (Think on the order of 4 million rows from your remote query!)
Execute the commit at the end of processing. There are even some asynchronous
functions you may be able to utilize to reduce the I/O bottleneck. Use the
synchronous function first, then before you return initiate an asynchronous
read. Every successive pass through the function, read the newly arrived tuple,
and initiate the next asynchronous read. (The two machine could be processing
the query simultaneously, and this could even IMPROVE performance over a single
system for heavy duty queries.)
Setup a hash table for field names, rather than requiring field numbers. (Keep
field number for efficiency, of course.)
You could eliminate having to pass the result pointer around by keeping a
static array in your extension. Use something like Oracle's "contains" notation
of result number. Where each instantiation of "contains()" and "score()"
require an id. i.e. 1,2,3,40 etc. Then hash those numbers into an array. I have
some code that does this for a PostgreSQL extension (it implements contains) on
my website (pgcontains, under download). It is ugly but works for the most
part.
Seriously, your stuff looks great. I think it could be the beginning of a
fairly usable system for my company. Any help you need/want, just let me know.
Joe Conway wrote:
>
> mlw wrote:
>
> > I too find the internals of PostgreSQL virtually incomprehensible at the
> > internal level. If there were a document somewhere which published how a
> > function could return multiple tuples, remote views would be a trivial
> > undertaking. It could look like:
> >
> > select * from remote('select *from table', 'user=postgres host=outland
> > db=remote');
> >
>
> See contrib/dblink in the 7.2 beta. It was my attempt inspired by
> Oracle's dblink and some code that you (mlw) posted a while back. Based
> on the limitations wrt returning muliple tuples, I wound up with
> something like:
>
> lt_lcat=# select dblink_tok(t1.dblink_p,0) as f1 from (select
> dblink('hostaddr=127.0.0.1 dbname=template1 user=postgres
> password=postgres','select proname from pg_proc') as dblink_p) as t1;
>
> Which, as has been pointed out more than once, is pretty ugly, but at
> least it's a start ;-)
>
> Joe
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly
From pgsql-hackers-owner+M16344=candle.pha.pa.us=pgman@postgresql.org Fri Dec 7 02:51:51 2001
Return-path: <pgsql-hackers-owner+M16344=candle.pha.pa.us=pgman@postgresql.org>
Received: from west.navpoint.com (west.navpoint.com [207.106.42.13])
by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB77poZ14221
for <pgman@candle.pha.pa.us>; Fri, 7 Dec 2001 02:51:50 -0500 (EST)
Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged))
by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB77pqq08152
for <pgman@candle.pha.pa.us>; Fri, 7 Dec 2001 02:51:52 -0500 (EST)
Received: from postgresql.org (postgresql.org [64.49.215.8])
by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB77lwR84105
for <pgman@candle.pha.pa.us>; Fri, 7 Dec 2001 01:49:04 -0600 (CST)
(envelope-from pgsql-hackers-owner+M16344=candle.pha.pa.us=pgman@postgresql.org)
Received: from ra.sai.msu.su (ra.sai.msu.su [158.250.29.2])
by postgresql.org (8.11.3/8.11.4) with ESMTP id fB77bmm32783
for <pgsql-hackers@postgresql.org>; Fri, 7 Dec 2001 02:37:49 -0500 (EST)
(envelope-from oleg@sai.msu.su)
Received: from ra (ra [158.250.29.2])
by ra.sai.msu.su (8.9.3/8.9.3) with ESMTP id KAA04160;
Fri, 7 Dec 2001 10:37:04 +0300 (GMT)
Date: Fri, 7 Dec 2001 10:37:04 +0300 (GMT)
From: Oleg Bartunov <oleg@sai.msu.su>
X-X-Sender: <megera@ra.sai.msu.su>
To: mlw <markw@mohawksoft.com>
cc: Joe Conway <joseph.conway@home.com>,
"Ross J. Reedstrom" <reedstrm@rice.edu>,
"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: [HACKERS] Remote connections?
In-Reply-To: <3C104E38.DA19C867@mohawksoft.com>
Message-ID: <Pine.GSO.4.33.0112071035180.20511-100000@ra.sai.msu.su>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR
On Fri, 7 Dec 2001, mlw wrote:
>
> You could eliminate having to pass the result pointer around by keeping a
> static array in your extension. Use something like Oracle's "contains" notation
> of result number. Where each instantiation of "contains()" and "score()"
> require an id. i.e. 1,2,3,40 etc. Then hash those numbers into an array. I have
> some code that does this for a PostgreSQL extension (it implements contains) on
> my website (pgcontains, under download). It is ugly but works for the most
> part.
contrib/intarray does this job very well
>
> Seriously, your stuff looks great. I think it could be the beginning of a
> fairly usable system for my company. Any help you need/want, just let me know.
>
>
> Joe Conway wrote:
> >
> > mlw wrote:
> >
> > > I too find the internals of PostgreSQL virtually incomprehensible at the
> > > internal level. If there were a document somewhere which published how a
> > > function could return multiple tuples, remote views would be a trivial
> > > undertaking. It could look like:
> > >
> > > select * from remote('select *from table', 'user=postgres host=outland
> > > db=remote');
> > >
> >
> > See contrib/dblink in the 7.2 beta. It was my attempt inspired by
> > Oracle's dblink and some code that you (mlw) posted a while back. Based
> > on the limitations wrt returning muliple tuples, I wound up with
> > something like:
> >
> > lt_lcat=# select dblink_tok(t1.dblink_p,0) as f1 from (select
> > dblink('hostaddr=127.0.0.1 dbname=template1 user=postgres
> > password=postgres','select proname from pg_proc') as dblink_p) as t1;
> >
> > Which, as has been pointed out more than once, is pretty ugly, but at
> > least it's a start ;-)
> >
> > Joe
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 4: Don't 'kill -9' the postmaster
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly
>
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83
---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly
From pgsql-hackers-owner+M16412=candle.pha.pa.us=pgman@postgresql.org Mon Dec 10 12:35:01 2001
Return-path: <pgsql-hackers-owner+M16412=candle.pha.pa.us=pgman@postgresql.org>
Received: from west.navpoint.com (west.navpoint.com [207.106.42.13])
by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fBAHZ0Z09772
for <pgman@candle.pha.pa.us>; Mon, 10 Dec 2001 12:35:00 -0500 (EST)
Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged))
by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fBAHZ0a11739
for <pgman@candle.pha.pa.us>; Mon, 10 Dec 2001 12:35:00 -0500 (EST)
Received: from postgresql.org (postgresql.org [64.49.215.8])
by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fBAHRAR20284
for <pgman@candle.pha.pa.us>; Mon, 10 Dec 2001 11:32:00 -0600 (CST)
(envelope-from pgsql-hackers-owner+M16412=candle.pha.pa.us=pgman@postgresql.org)
Received: from snoopy.mohawksoft.com (h0050bf7a618d.ne.mediaone.net [24.218.67.153])
by postgresql.org (8.11.3/8.11.4) with ESMTP id fB7DhOm75114
for <pgsql-hackers@postgresql.org>; Fri, 7 Dec 2001 08:43:24 -0500 (EST)
(envelope-from markw@mohawksoft.com)
Received: from mohawksoft.com (localhost [127.0.0.1])
by snoopy.mohawksoft.com (8.11.6/8.11.6) with ESMTP id fB7De0000437;
Fri, 7 Dec 2001 08:40:01 -0500
Message-ID: <3C10C6B0.865669C1@mohawksoft.com>
Date: Fri, 07 Dec 2001 08:40:00 -0500
From: mlw <markw@mohawksoft.com>
X-Mailer: Mozilla 4.78 [en] (X11; U; Linux 2.4.14ext3 i686)
X-Accept-Language: en
MIME-Version: 1.0
To: Oleg Bartunov <oleg@sai.msu.su>
cc: Joe Conway <joseph.conway@home.com>,
"Ross J. Reedstrom" <reedstrm@rice.edu>,
"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: [HACKERS] Remote connections?
References: <Pine.GSO.4.33.0112071035180.20511-100000@ra.sai.msu.su>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR
The dblink code is a very cool idea.
It got me thinking, what if, just thinking out load here, it was redesigned as
something a little more grandeous.
Imagine this:
select dblink('select * from table', 'table_name', 'db=oracle.test user=chris
passwd=knight', 1) as t1, dblink('table2_name', 1) as t2
Just something to think about.
The first instance of dblink would take 4 parameters: query, table which it
returns, connect string, and link token.
The second instance of dblink would just take the name of the table which it
returns and a link token.
The cool bit is the notion that the query string could specify different
databases or even .DBF libraries.
Just something to think about.
It would REALLY be great if functions could return multiple tuples!
---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?
http://www.postgresql.org/users-lounge/docs/faq.html
From pgsql-hackers-owner+M16365=candle.pha.pa.us=pgman@postgresql.org Fri Dec 7 12:32:26 2001
Return-path: <pgsql-hackers-owner+M16365=candle.pha.pa.us=pgman@postgresql.org>
Received: from west.navpoint.com (west.navpoint.com [207.106.42.13])
by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fB7HWMZ26245
for <pgman@candle.pha.pa.us>; Fri, 7 Dec 2001 12:32:22 -0500 (EST)
Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged))
by west.navpoint.com (8.11.6/8.10.1) with ESMTP id fB7HWLB14472
for <pgman@candle.pha.pa.us>; Fri, 7 Dec 2001 12:32:21 -0500 (EST)
Received: from postgresql.org (postgresql.org [64.49.215.8])
by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fB7HSHR01506
for <pgman@candle.pha.pa.us>; Fri, 7 Dec 2001 11:29:07 -0600 (CST)
(envelope-from pgsql-hackers-owner+M16365=candle.pha.pa.us=pgman@postgresql.org)
Received: from mx1.relaypoint.net (ns2.generalbroadband.com [64.32.62.5])
by postgresql.org (8.11.3/8.11.4) with ESMTP id fB7HQfm90424
for <pgsql-hackers@postgresql.org>; Fri, 7 Dec 2001 12:26:42 -0500 (EST)
(envelope-from joseph.conway@home.com)
Received: from [206.19.64.3] (account jconway@wwc.com HELO home.com)
by mx1.relaypoint.net (CommuniGate Pro SMTP 3.4.8)
with ESMTP id 1722339; Fri, 07 Dec 2001 09:26:46 -0800
Message-ID: <3C10FBD7.4070602@home.com>
Date: Fri, 07 Dec 2001 09:26:47 -0800
From: Joe Conway <joseph.conway@home.com>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.6+) Gecko/20011126
X-Accept-Language: en-us
MIME-Version: 1.0
To: mlw <markw@mohawksoft.com>
cc: "Ross J. Reedstrom" <reedstrm@rice.edu>,
"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: [HACKERS] Remote connections?
References: <3C0FB8B4.382C7736@mohawksoft.com> <20011206140616.C10995@rice.edu> <3C0FD339.6F663329@mohawksoft.com> <3C0FEB2C.70000@home.com> <3C104E38.DA19C867@mohawksoft.com>
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR
mlw wrote:
> Hey this looks really cool. It looks like something I was thinking about doing.
> I have a couple suggestions that could make it a little better, I hope you will
> not be offended. (If you want my help, I'll chip in!)
>
Thanks! Suggestions welcomed.
> Why not use a binary cursor? That way native types can slip through without the
> overhead of conversion.
>
I wasn't sure that would work. Would you create dblink_tok as returning
opaque then?
> Right now you get all rows up front, you may be able to increase overall
> performance by fetching only a few rows at a time, rather than get everything
> all at once. (Think on the order of 4 million rows from your remote query!)
> Execute the commit at the end of processing. There are even some asynchronous
> functions you may be able to utilize to reduce the I/O bottleneck. Use the
> synchronous function first, then before you return initiate an asynchronous
> read. Every successive pass through the function, read the newly arrived tuple,
> and initiate the next asynchronous read. (The two machine could be processing
> the query simultaneously, and this could even IMPROVE performance over a single
> system for heavy duty queries.)
Interesting . . . but aren't there some issues with the asynch functions?
>
> Setup a hash table for field names, rather than requiring field numbers. (Keep
> field number for efficiency, of course.)
>
> You could eliminate having to pass the result pointer around by keeping a
> static array in your extension. Use something like Oracle's "contains" notation
> of result number. Where each instantiation of "contains()" and "score()"
> require an id. i.e. 1,2,3,40 etc. Then hash those numbers into an array. I have
> some code that does this for a PostgreSQL extension (it implements contains) on
> my website (pgcontains, under download). It is ugly but works for the most
> part.
>
I thought about the static array, but I'm not familiar with Oracle
contains() and score() -- I'm only fluent enough with Oracle to be
dangerous. Guess I'll have to dig out the books . . .
> Seriously, your stuff looks great. I think it could be the beginning of a
> fairly usable system for my company. Any help you need/want, just let me know.
>
I am planning to improve dblink during the next release cycle, so I'll
keep all this in mind (and might take you up on the help offer too!). I
was hoping we'd have functions returning tuples by now, which would
improve this extension dramatically. Unfortunately, it sounds like Alex
won't have time to finish that even for 7.3 :(
Alex, can we get a look at your latest code? Is it any different the
your last submission to PATCHES?
Joe
---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly
Reference in New Issue View Git Blame Copy Permalink