From pgsql-hackers-owner+M4897@hub.org Wed Jul 12 00:15:33 2000 Received: from hub.org (root@hub.org [216.126.84.1]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id AAA06129 for ; Wed, 12 Jul 2000 00:15:32 -0400 (EDT) Received: from hub.org (majordom@localhost [127.0.0.1]) by hub.org (8.10.1/8.10.1) with SMTP id e6C4FiW14410; Wed, 12 Jul 2000 00:15:44 -0400 (EDT) Received: from onyx-technologies.com (iron.onyx-technologies.com [216.205.44.194] (may be forged)) by hub.org (8.10.1/8.10.1) with ESMTP id e6C4ECW07902 for ; Wed, 12 Jul 2000 00:14:12 -0400 (EDT) Received: from onyx-technologies.com (collins.onyx-technologies.com [192.168.188.10]) by onyx-technologies.com (8.9.2/8.9.0) with ESMTP id AAA14868 for ; Wed, 12 Jul 2000 00:11:43 -0400 (EDT) Message-ID: <396BE1B6.F755C5CE@onyx-technologies.com> Date: Tue, 11 Jul 2000 23:10:46 -0400 From: Jeffery Collins X-Mailer: Mozilla 4.7 [en] (X11; I; Linux 2.2.14-15mdk i686) X-Accept-Language: en MIME-Version: 1.0 To: pgsql-hackers@hub.org Subject: Re: [HACKERS] Connection pooling. References: <20000711185318.W25571@fw.wintelcom.net> <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Mailing-List: pgsql-hackers@postgresql.org Precedence: bulk Sender: pgsql-hackers-owner@hub.org Status: ORr It seems like a first step would be to just have postmaster cache unused connections. In other words if a client closes a connection, postmaster keeps the connection and the child process around for the next connect request. This has many of your advantages, but not all. However, it seems like it would be simpler than attempting to multiplex a connection between multiple clients. Jeff > > Alfred Perlstein wrote: > > > > In an effort to complicate the postmaster beyond recognition I'm > > proposing an idea that I hope can be useful to the developers. > > > > Connection pooling: > > > > The idea is to have the postmaster multiplex and do hand-offs of > > database connections to other postgresql processes when the max > > connections has been exceeded. > > > > This allows several gains: > > > > 1) Postgresql can support a large number of connections without > > requiring a large amount of processes to do so. > > > > 2) Connection startup/finish will be cheaper because Postgresql > > processes will not exit and need to reninit things such as shared > > memory attachments and file opens. This will also reduce the load > > on the supporting operating system and make postgresql much 'cheaper' > > to run on systems that don't support the fork() model of execution > > gracefully. > > > > 3) Long running connections can be preempted at transaction boundries > > allowing other connections to gain process timeslices from the > > connection pool. > > > > The idea is to make the postmaster that accepts connections a broker > > for the connections. It will dole out descriptors using file > > descriptor passing to children. If there's a demand for connections > > meaning that all the postmasters are busy and there are pending > > connections the postmaster can ask for a yeild on one of the > > connections. > > > > A yeild involves the child postgresql process passing back the > > client connection at a transaction boundry (between transactions) > > so it can later be given to another (perhaps the same) child process. > > > > I spoke with Bruce briefly about this and he suggested that system > > tables containing unique IDs could be used to identify passed > > connections to the children and back to the postmaster. > > > > When a handoff occurs, the descriptor along with an ID referencing > > things like temp tables and enviornment variables and authentication > > information could be handed out as well allowing the child to resume > > service to the interrupted connection. > > > > I really don't have the knowledge of Postgresql internals to > > accomplish this, but the concepts are simple and the gains would > > seem to be very high. > > > > Comments? > > > > -- > > -Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org] > > "I have the heart of a child; I keep it in a jar on my desk." From pgsql-hackers-owner+M4904@hub.org Wed Jul 12 01:24:09 2000 Received: from hub.org (root@hub.org [216.126.84.1]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id BAA06757 for ; Wed, 12 Jul 2000 01:24:08 -0400 (EDT) Received: from hub.org (majordom@localhost [127.0.0.1]) by hub.org (8.10.1/8.10.1) with SMTP id e6C5OLW65679; Wed, 12 Jul 2000 01:24:21 -0400 (EDT) Received: from fw.wintelcom.net (bright@ns1.wintelcom.net [209.1.153.20]) by hub.org (8.10.1/8.10.1) with ESMTP id e6C5MkW61040 for ; Wed, 12 Jul 2000 01:22:46 -0400 (EDT) Received: (from bright@localhost) by fw.wintelcom.net (8.10.0/8.10.0) id e6C5Md429901; Tue, 11 Jul 2000 22:22:39 -0700 (PDT) Date: Tue, 11 Jul 2000 22:22:39 -0700 From: Alfred Perlstein To: Chris Bitmead Cc: pgsql-hackers@hub.org Subject: Re: [HACKERS] Connection pooling. Message-ID: <20000711222239.X25571@fw.wintelcom.net> References: <20000711185318.W25571@fw.wintelcom.net> <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2i In-Reply-To: <396BEA84.1A06F51F@nimrod.itg.telecom.com.au>; from chrisb@nimrod.itg.telstra.com.au on Wed, Jul 12, 2000 at 01:48:20PM +1000 X-Mailing-List: pgsql-hackers@postgresql.org Precedence: bulk Sender: pgsql-hackers-owner@hub.org Status: OR * Chris Bitmead [000711 20:53] wrote: > > Seems a lot trickier than you think. A backend can only be running > one transaction at a time, so you'd have to keep track of which backends > are in the middle of a transaction. I can imagine race conditions here. > And backends can have contexts that are set by various clients using > SET and friends. Then you'd have to worry about authentication each > time. And you'd have to have algorithms for cleaning up old processes > and/or dead processes. It all really sounds a bit hard. The backends can simply inform the postmaster when they are ready either because they are done with a connection or because they have just closed a transaction. All the state (auth/temp tables) can be held in the system tables. It's complicated, but no where on the order of something like a new storage manager. -Alfred From bright@fw.wintelcom.net Wed Jul 12 01:34:30 2000 Received: from fw.wintelcom.net (bright@ns1.wintelcom.net [209.1.153.20]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id BAA06793 for ; Wed, 12 Jul 2000 01:34:29 -0400 (EDT) Received: (from bright@localhost) by fw.wintelcom.net (8.10.0/8.10.0) id e6C5Z1f00384; Tue, 11 Jul 2000 22:35:01 -0700 (PDT) Date: Tue, 11 Jul 2000 22:35:00 -0700 From: Alfred Perlstein To: Bruce Momjian Cc: Jeffery Collins , pgsql-hackers@hub.org Subject: Re: [HACKERS] Connection pooling. Message-ID: <20000711223500.Z25571@fw.wintelcom.net> References: <396BE1B6.F755C5CE@onyx-technologies.com> <200007120428.AAA06357@candle.pha.pa.us> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2i In-Reply-To: <200007120428.AAA06357@candle.pha.pa.us>; from pgman@candle.pha.pa.us on Wed, Jul 12, 2000 at 12:28:46AM -0400 Status: OR * Bruce Momjian [000711 21:31] wrote: > > It seems like a first step would be to just have postmaster cache unused > > connections. In other words if a client closes a connection, postmaster > > keeps the connection and the child process around for the next connect > > request. This has many of your advantages, but not all. However, it seems > > like it would be simpler than attempting to multiplex a connection between > > multiple clients. > > > > This does seem like a good optimization. I'm not sure if the postmaster is needed besideds just to fork/exec the backend, if so then when a backend finishes it can just call accept() on the listening socket inherited from the postmaster to get the next incomming connection. -Alfred From pgsql-hackers-owner+M4906@hub.org Wed Jul 12 01:36:44 2000 Received: from hub.org (root@hub.org [216.126.84.1]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id BAA06806 for ; Wed, 12 Jul 2000 01:36:44 -0400 (EDT) Received: from hub.org (majordom@localhost [127.0.0.1]) by hub.org (8.10.1/8.10.1) with SMTP id e6C5akW94517; Wed, 12 Jul 2000 01:36:46 -0400 (EDT) Received: from fw.wintelcom.net (bright@ns1.wintelcom.net [209.1.153.20]) by hub.org (8.10.1/8.10.1) with ESMTP id e6C5ZCW88503 for ; Wed, 12 Jul 2000 01:35:12 -0400 (EDT) Received: (from bright@localhost) by fw.wintelcom.net (8.10.0/8.10.0) id e6C5Z1f00384; Tue, 11 Jul 2000 22:35:01 -0700 (PDT) Date: Tue, 11 Jul 2000 22:35:00 -0700 From: Alfred Perlstein To: Bruce Momjian Cc: Jeffery Collins , pgsql-hackers@hub.org Subject: Re: [HACKERS] Connection pooling. Message-ID: <20000711223500.Z25571@fw.wintelcom.net> References: <396BE1B6.F755C5CE@onyx-technologies.com> <200007120428.AAA06357@candle.pha.pa.us> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2i In-Reply-To: <200007120428.AAA06357@candle.pha.pa.us>; from pgman@candle.pha.pa.us on Wed, Jul 12, 2000 at 12:28:46AM -0400 X-Mailing-List: pgsql-hackers@postgresql.org Precedence: bulk Sender: pgsql-hackers-owner@hub.org Status: OR * Bruce Momjian [000711 21:31] wrote: > > It seems like a first step would be to just have postmaster cache unused > > connections. In other words if a client closes a connection, postmaster > > keeps the connection and the child process around for the next connect > > request. This has many of your advantages, but not all. However, it seems > > like it would be simpler than attempting to multiplex a connection between > > multiple clients. > > > > This does seem like a good optimization. I'm not sure if the postmaster is needed besideds just to fork/exec the backend, if so then when a backend finishes it can just call accept() on the listening socket inherited from the postmaster to get the next incomming connection. -Alfred From pgsql-hackers-owner+M4907@hub.org Wed Jul 12 01:55:39 2000 Received: from hub.org (root@hub.org [216.126.84.1]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id BAA06881 for ; Wed, 12 Jul 2000 01:55:38 -0400 (EDT) Received: from hub.org (majordom@localhost [127.0.0.1]) by hub.org (8.10.1/8.10.1) with SMTP id e6C5tnW34576; Wed, 12 Jul 2000 01:55:49 -0400 (EDT) Received: from sss2.sss.pgh.pa.us (sss.pgh.pa.us [209.114.166.2]) by hub.org (8.10.1/8.10.1) with ESMTP id e6C5rfW28119 for ; Wed, 12 Jul 2000 01:53:42 -0400 (EDT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss2.sss.pgh.pa.us (8.9.3/8.9.3) with ESMTP id BAA21895; Wed, 12 Jul 2000 01:52:56 -0400 (EDT) To: Chris Bitmead cc: Alfred Perlstein , pgsql-hackers@hub.org Subject: Re: [HACKERS] Connection pooling. In-reply-to: <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> References: <20000711185318.W25571@fw.wintelcom.net> <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> Comments: In-reply-to Chris Bitmead message dated "Wed, 12 Jul 2000 13:48:20 +1000" Date: Wed, 12 Jul 2000 01:52:56 -0400 Message-ID: <21892.963381176@sss.pgh.pa.us> From: Tom Lane X-Mailing-List: pgsql-hackers@postgresql.org Precedence: bulk Sender: pgsql-hackers-owner@hub.org Status: OR Chris Bitmead writes: > Seems a lot trickier than you think. A backend can only be running > one transaction at a time, so you'd have to keep track of which backends > are in the middle of a transaction. I can imagine race conditions here. Aborting out of a transaction is no problem; we have code for that anyway. More serious problems: * We have no code for reassigning a backend to a different database, so the pooling would have to be per-database. * AFAIK there is no portable way to pass a socket connection from the postmaster to an already-existing backend process. If you do a fork() then the connection is inherited ... otherwise you've got a problem. (You could work around this if the postmaster relays every single byte in both directions between client and backend, but the performance problems with that should be obvious.) > And backends can have contexts that are set by various clients using > SET and friends. Resetting SET variables would be a problem, and there's also the assigned user name to be reset. This doesn't seem impossible, but it does seem tedious and error-prone. (OTOH, Peter E's recent work on guc.c might have unified option-handling enough to bring it within reason.) The killer problem here is that you can't hand off a connection accepted by the postmaster to a backend except by fork() --- at least not with methods that work on a wide variety of Unixen. Unless someone has a way around that, I think the idea is dead in the water; the lesser issues don't matter. regards, tom lane From pgsql-hackers-owner+M4910@hub.org Wed Jul 12 02:24:16 2000 Received: from hub.org (root@hub.org [216.126.84.1]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id CAA11184 for ; Wed, 12 Jul 2000 02:24:15 -0400 (EDT) Received: from hub.org (majordom@localhost [127.0.0.1]) by hub.org (8.10.1/8.10.1) with SMTP id e6C6OAW98187; Wed, 12 Jul 2000 02:24:10 -0400 (EDT) Received: from acheron.rime.com.au (root@albatr.lnk.telstra.net [139.130.54.222]) by hub.org (8.10.1/8.10.1) with ESMTP id e6C6MZW95741 for ; Wed, 12 Jul 2000 02:22:36 -0400 (EDT) Received: from oberon (Oberon.rime.com.au [203.8.195.100]) by acheron.rime.com.au (8.9.3/8.9.3) with SMTP id QAA12845; Wed, 12 Jul 2000 16:16:23 +1000 Message-Id: <3.0.5.32.20000712162210.0098fb00@mail.rhyme.com.au> X-Sender: pjw@mail.rhyme.com.au X-Mailer: QUALCOMM Windows Eudora Pro Version 3.0.5 (32) Date: Wed, 12 Jul 2000 16:22:10 +1000 To: Tom Lane , Chris Bitmead From: Philip Warner Subject: Re: [HACKERS] Connection pooling. Cc: Alfred Perlstein , pgsql-hackers@hub.org In-Reply-To: <21892.963381176@sss.pgh.pa.us> References: <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> <20000711185318.W25571@fw.wintelcom.net> <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" X-Mailing-List: pgsql-hackers@postgresql.org Precedence: bulk Sender: pgsql-hackers-owner@hub.org Status: OR At 01:52 12/07/00 -0400, Tom Lane wrote: > >The killer problem here is that you can't hand off a connection >accepted by the postmaster to a backend except by fork() --- at least >not with methods that work on a wide variety of Unixen. Unless someone >has a way around that, I think the idea is dead in the water; the lesser >issues don't matter. > My understanding of pg client interfaces is that the client uses ont of the pg interface libraries to make a connection to the db; they specify host & port and get back some kind of connection object. What stops the interface library from using the host & port to talk to the postmaster, find the host & port the spare db server, then connect directly to the server? This second connection is passed back in the connection object. When the client disconnects from the server, it tells the postmaster it's available again etc. ie. in very rough terms: client calls interface to connect interface talks to postmaster on port 5432, says "I want a server for xyz db" postmaster replies with "Try port ABCD" OR "no servers available" postmaster marks the nominated server as 'used' postmaster disconnects from client interface connects to port ABCD as per normal protocols interface fills in connection object & returns ...client does some work... client disconnects db server tells postmaster it's available again. There would also need to be timeout code to handle the case where the interface did not do the second connect. You could also have the interface allocate a port send it's number to the postmaster then listen on it, but I think that would represent a potential security hole. ---------------------------------------------------------------- Philip Warner | __---_____ Albatross Consulting Pty. Ltd. |----/ - \ (A.C.N. 008 659 498) | /(@) ______---_ Tel: (+61) 0500 83 82 81 | _________ \ Fax: (+61) 0500 83 82 82 | ___________ | Http://www.rhyme.com.au | / \| | --________-- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ From pgsql-hackers-owner+M4912@hub.org Wed Jul 12 02:32:21 2000 Received: from hub.org (root@hub.org [216.126.84.1]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id CAA11228 for ; Wed, 12 Jul 2000 02:32:20 -0400 (EDT) Received: from hub.org (majordom@localhost [127.0.0.1]) by hub.org (8.10.1/8.10.1) with SMTP id e6C6WWW18412; Wed, 12 Jul 2000 02:32:32 -0400 (EDT) Received: from fw.wintelcom.net (bright@ns1.wintelcom.net [209.1.153.20]) by hub.org (8.10.1/8.10.1) with ESMTP id e6C6UwW16062 for ; Wed, 12 Jul 2000 02:30:58 -0400 (EDT) Received: (from bright@localhost) by fw.wintelcom.net (8.10.0/8.10.0) id e6C6Uov01852; Tue, 11 Jul 2000 23:30:50 -0700 (PDT) Date: Tue, 11 Jul 2000 23:30:49 -0700 From: Alfred Perlstein To: Tom Lane Cc: Chris Bitmead , pgsql-hackers@hub.org Subject: Re: [HACKERS] Connection pooling. Message-ID: <20000711233049.A25571@fw.wintelcom.net> References: <20000711185318.W25571@fw.wintelcom.net> <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> <21892.963381176@sss.pgh.pa.us> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2i In-Reply-To: <21892.963381176@sss.pgh.pa.us>; from tgl@sss.pgh.pa.us on Wed, Jul 12, 2000 at 01:52:56AM -0400 X-Mailing-List: pgsql-hackers@postgresql.org Precedence: bulk Sender: pgsql-hackers-owner@hub.org Status: OR * Tom Lane [000711 22:53] wrote: > Chris Bitmead writes: > > Seems a lot trickier than you think. A backend can only be running > > one transaction at a time, so you'd have to keep track of which backends > > are in the middle of a transaction. I can imagine race conditions here. > > Aborting out of a transaction is no problem; we have code for that > anyway. More serious problems: > > * We have no code for reassigning a backend to a different database, > so the pooling would have to be per-database. That would need to be fixed. How difficult would that be? > * AFAIK there is no portable way to pass a socket connection from the > postmaster to an already-existing backend process. If you do a > fork() then the connection is inherited ... otherwise you've got a > problem. (You could work around this if the postmaster relays > every single byte in both directions between client and backend, > but the performance problems with that should be obvious.) no, see below. > > And backends can have contexts that are set by various clients using > > SET and friends. > > Resetting SET variables would be a problem, and there's also the > assigned user name to be reset. This doesn't seem impossible, but > it does seem tedious and error-prone. (OTOH, Peter E's recent work > on guc.c might have unified option-handling enough to bring it > within reason.) What can be done is that each incomming connection can be assigned an ID into a system table. As options are added the system would assign them to key-value pairs in this table. Once someone detects that the remote side has closed the connection the data can be destroyed, but until then along with the descriptor passing the ID of the client as an index into the table can be passed for the backend to fetch. > The killer problem here is that you can't hand off a connection > accepted by the postmaster to a backend except by fork() --- at least > not with methods that work on a wide variety of Unixen. Unless someone > has a way around that, I think the idea is dead in the water; the lesser > issues don't matter. The code has been around since 4.2BSD, it takes a bit of #ifdef to get it right on all systems but it's not impossible, have a look at http://www.fhttpd.org/ for a web server that does this in a portable fashion. I should have a library whipped up for you guys really soon now to handle the descriptor and message passing. -- -Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org] "I have the heart of a child; I keep it in a jar on my desk." From pgsql-hackers-owner+M4913@hub.org Wed Jul 12 03:06:54 2000 Received: from hub.org (root@hub.org [216.126.84.1]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id DAA11529 for ; Wed, 12 Jul 2000 03:06:53 -0400 (EDT) Received: from hub.org (majordom@localhost [127.0.0.1]) by hub.org (8.10.1/8.10.1) with SMTP id e6C76ZW95615; Wed, 12 Jul 2000 03:06:35 -0400 (EDT) Received: from sss2.sss.pgh.pa.us (sss.pgh.pa.us [209.114.166.2]) by hub.org (8.10.1/8.10.1) with ESMTP id e6C74gW93358 for ; Wed, 12 Jul 2000 03:04:42 -0400 (EDT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss2.sss.pgh.pa.us (8.9.3/8.9.3) with ESMTP id DAA22136; Wed, 12 Jul 2000 03:04:13 -0400 (EDT) To: Alfred Perlstein cc: Chris Bitmead , pgsql-hackers@hub.org Subject: Re: [HACKERS] Connection pooling. In-reply-to: <20000711233049.A25571@fw.wintelcom.net> References: <20000711185318.W25571@fw.wintelcom.net> <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> <21892.963381176@sss.pgh.pa.us> <20000711233049.A25571@fw.wintelcom.net> Comments: In-reply-to Alfred Perlstein message dated "Tue, 11 Jul 2000 23:30:49 -0700" Date: Wed, 12 Jul 2000 03:04:13 -0400 Message-ID: <22133.963385453@sss.pgh.pa.us> From: Tom Lane X-Mailing-List: pgsql-hackers@postgresql.org Precedence: bulk Sender: pgsql-hackers-owner@hub.org Status: OR Alfred Perlstein writes: > * Tom Lane [000711 22:53] wrote: >> The killer problem here is that you can't hand off a connection >> accepted by the postmaster to a backend except by fork() --- at least >> not with methods that work on a wide variety of Unixen. > The code has been around since 4.2BSD, it takes a bit of #ifdef to > get it right on all systems but it's not impossible, have a look at > http://www.fhttpd.org/ for a web server that does this in a portable > fashion. I looked at this to see if it would teach me something I didn't know. It doesn't. It depends on sendmsg() which is a BSD-ism and not very portable. regards, tom lane From pgsql-hackers-owner+M4914@hub.org Wed Jul 12 03:12:40 2000 Received: from hub.org (root@hub.org [216.126.84.1]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id DAA11597 for ; Wed, 12 Jul 2000 03:12:39 -0400 (EDT) Received: from hub.org (majordom@localhost [127.0.0.1]) by hub.org (8.10.1/8.10.1) with SMTP id e6C7CjW13459; Wed, 12 Jul 2000 03:12:45 -0400 (EDT) Received: from fw.wintelcom.net (bright@ns1.wintelcom.net [209.1.153.20]) by hub.org (8.10.1/8.10.1) with ESMTP id e6C7B8W07036 for ; Wed, 12 Jul 2000 03:11:08 -0400 (EDT) Received: (from bright@localhost) by fw.wintelcom.net (8.10.0/8.10.0) id e6C79lE02841; Wed, 12 Jul 2000 00:09:47 -0700 (PDT) Date: Wed, 12 Jul 2000 00:09:47 -0700 From: Alfred Perlstein To: Tom Lane Cc: Chris Bitmead , pgsql-hackers@hub.org Subject: Re: [HACKERS] Connection pooling. Message-ID: <20000712000947.D25571@fw.wintelcom.net> References: <20000711185318.W25571@fw.wintelcom.net> <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> <21892.963381176@sss.pgh.pa.us> <20000711233049.A25571@fw.wintelcom.net> <22133.963385453@sss.pgh.pa.us> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2i In-Reply-To: <22133.963385453@sss.pgh.pa.us>; from tgl@sss.pgh.pa.us on Wed, Jul 12, 2000 at 03:04:13AM -0400 X-Mailing-List: pgsql-hackers@postgresql.org Precedence: bulk Sender: pgsql-hackers-owner@hub.org Status: OR * Tom Lane [000712 00:04] wrote: > Alfred Perlstein writes: > > * Tom Lane [000711 22:53] wrote: > >> The killer problem here is that you can't hand off a connection > >> accepted by the postmaster to a backend except by fork() --- at least > >> not with methods that work on a wide variety of Unixen. > > > The code has been around since 4.2BSD, it takes a bit of #ifdef to > > get it right on all systems but it's not impossible, have a look at > > http://www.fhttpd.org/ for a web server that does this in a portable > > fashion. > > I looked at this to see if it would teach me something I didn't know. > It doesn't. It depends on sendmsg() which is a BSD-ism and not very > portable. It's also specified by Posix.1g if that means anything. -Alfred From pgsql-hackers-owner+M4916@hub.org Wed Jul 12 03:49:58 2000 Received: from hub.org (root@hub.org [216.126.84.1]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id DAA11736 for ; Wed, 12 Jul 2000 03:49:58 -0400 (EDT) Received: from hub.org (majordom@localhost [127.0.0.1]) by hub.org (8.10.1/8.10.1) with SMTP id e6C7oBW95547; Wed, 12 Jul 2000 03:50:11 -0400 (EDT) Received: from sss2.sss.pgh.pa.us (sss.pgh.pa.us [209.114.166.2]) by hub.org (8.10.1/8.10.1) with ESMTP id e6C7mPW93141 for ; Wed, 12 Jul 2000 03:48:25 -0400 (EDT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss2.sss.pgh.pa.us (8.9.3/8.9.3) with ESMTP id DAA22297; Wed, 12 Jul 2000 03:47:37 -0400 (EDT) To: Philip Warner cc: Chris Bitmead , Alfred Perlstein , pgsql-hackers@hub.org Subject: Re: [HACKERS] Connection pooling. In-reply-to: <3.0.5.32.20000712162210.0098fb00@mail.rhyme.com.au> References: <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> <20000711185318.W25571@fw.wintelcom.net> <396BEA84.1A06F51F@nimrod.itg.telecom.com.au> <3.0.5.32.20000712162210.0098fb00@mail.rhyme.com.au> Comments: In-reply-to Philip Warner message dated "Wed, 12 Jul 2000 16:22:10 +1000" Date: Wed, 12 Jul 2000 03:47:37 -0400 Message-ID: <22294.963388057@sss.pgh.pa.us> From: Tom Lane X-Mailing-List: pgsql-hackers@postgresql.org Precedence: bulk Sender: pgsql-hackers-owner@hub.org Status: OR Philip Warner writes: > What stops the interface library from using the host & port to talk to > the postmaster, find the host & port the spare db server, then connect > directly to the server? You're assuming that we can change the on-the-wire protocol freely and only the API presented by the client libraries matters. In a perfect world that might be true, but reality is that we can't change the wire protocol easily. If we do, it breaks all existing precompiled clients. Updating clients can be an extremely painful experience in multiple- machine installations. Also, we don't have just one set of client libraries to fix. There are at least three client-side implementations that don't depend on libpq. We have done protocol updates in the past --- in fact I was responsible for the last one. (And I'm still carrying the scars, which is why I'm not too enthusiastic about another one.) It's not impossible, but it needs more evidence than "this should speed up connections by I-don't-know-how-much"... It might also be worth pointing out that the goal was to speed up the end-to-end connection time. Redirecting as you suggest is not free (at minimum it would appear to require two TCP connection setups and two authentication cycles). What evidence have you got that it'd be faster than spawning a new backend? I tend to agree with the opinion that connection-pooling on the client side offers more bang for the buck than we could hope to get by doing surgery on the postmaster/backend setup. Also, to return to the original point, AFAIK we have not tried hard to cut the backend startup time, other than the work that was done a year or so back to eliminate exec() of a separate executable. It'd be worth looking to see what could be done there with zero impact on existing clients. regards, tom lane From pgsql-hackers-owner+M16940@postgresql.org Sun Dec 23 23:06:28 2001 Return-path: Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fBO46R429655 for ; Sun, 23 Dec 2001 23:06:27 -0500 (EST) Received: from postgresql.org (postgresql.org [64.49.215.8]) by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fBO40oN57016; Sun, 23 Dec 2001 22:00:50 -0600 (CST) (envelope-from pgsql-hackers-owner+M16940@postgresql.org) Received: from relay.pair.com (relay1.pair.com [209.68.1.20]) by postgresql.org (8.11.3/8.11.4) with SMTP id fBID0um78493 for ; Tue, 18 Dec 2001 08:00:56 -0500 (EST) (envelope-from ml@augustz.com) Received: (qmail 79914 invoked from network); 18 Dec 2001 13:00:58 -0000 Received: from acz01997-2.pomona.edu (HELO Microsoft) (134.173.91.3) by relay1.pair.com with SMTP; 18 Dec 2001 13:00:58 -0000 X-pair-Authenticated: 134.173.91.3 From: "August Zajonc" To: Subject: [HACKERS] Connection Pooling, a year later Date: Tue, 18 Dec 2001 05:00:57 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR I feel there was a reasonably nice client side attempt at this using a worker pool model or something. Can't seem to track it down at this moment. Also would spread queries in different ways to get a hot backup equivalent etc. It was slick. The key is that pgsql be able to support a very significant number of transactions. Be neat to see some numbers on your attempt. Site I used to run had 6 front end webservers running PHP apps. Each persistent connection (a requirement to avoid overhead of set-up/teardowns) lived as long as the httpd process lived, even if idle. That meant at 250 processes per server we had a good 1500 connections clicking over. Our feeling was that rather than growing to 3,000 connections as the frontend grew, why not pool those connections off each machine down to perhaps 75/machine worker threads that actually did the work. Looks like that's not an issue if these backends suck up few resources. Doing something similar with MySQL we'd experiance problems if we got into the 2,000 connection range. (kernel/system limits bumped plenty high). While we are on TODO's I would like to point out that some way to fully vacume (ie recover deleted and changed) while a db is in full swing is critical to larger installtions. We did 2 billion queries between reboots on a quad zeon MySQL box, and those are real user based queries not data loads or anything like that. At 750-1000 queries/second bringing the database down or seriously degrading its performance is not a good option. Enjoy playing with pgsql as always.... - AZ ---------------------------(end of broadcast)--------------------------- TIP 4: Don't 'kill -9' the postmaster From dhogaza@pacifier.com Tue Dec 18 11:15:06 2001 Return-path: Received: from asteroid.pacifier.com ([199.2.117.154]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fBIGF5419342 for ; Tue, 18 Dec 2001 11:15:05 -0500 (EST) Received: from pacifier.com (dsl-dhogaza.pacifier.net [207.202.226.68]) by asteroid.pacifier.com (8.11.2/8.11.1) with ESMTP id fBIGEGe29925; Tue, 18 Dec 2001 08:14:17 -0800 (PST) Message-ID: <3C1F6B81.10500@pacifier.com> Date: Tue, 18 Dec 2001 08:14:57 -0800 From: Don Baccus User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.6) Gecko/20011120 X-Accept-Language: en-us MIME-Version: 1.0 To: Bruce Momjian cc: mlw , owensmk@earthlink.net, pgsql-hackers@postgresql.org, Tom Lane Subject: Re: [HACKERS] Connection Pooling, a year later References: <200112180342.fBI3g4s23880@candle.pha.pa.us> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Status: OR Bruce Momjian wrote: > It would just be nice to have it done internally rather than have all > the clients do it, iff it can be done cleanly. Serious client applications that need it already do it. Firing up an Oracle or most other db's isn't that lightweight a deal, either, it's not useful only for PG.. Personally I'd just view it as getting in the way, but then I use a webserver that's provided connection pooling for client threads for the last seven years ... I agree with Tom that the client seems to be the best place to do this. Among other things it isn't that difficult. If you know how to fire up one connection, you know how to fire up N of them and adding logic to pool them afterwards is easy enough. -- Don Baccus Portland, OR http://donb.photo.net, http://birdnotes.net, http://openacs.org From dhogaza@pacifier.com Tue Dec 18 11:24:33 2001 Return-path: Received: from asteroid.pacifier.com ([199.2.117.154]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fBIGOW421363 for ; Tue, 18 Dec 2001 11:24:33 -0500 (EST) Received: from pacifier.com (dsl-dhogaza.pacifier.net [207.202.226.68]) by asteroid.pacifier.com (8.11.2/8.11.1) with ESMTP id fBIGNne00442; Tue, 18 Dec 2001 08:23:49 -0800 (PST) Message-ID: <3C1F6DBF.2040000@pacifier.com> Date: Tue, 18 Dec 2001 08:24:31 -0800 From: Don Baccus User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.6) Gecko/20011120 X-Accept-Language: en-us MIME-Version: 1.0 To: Bruce Momjian cc: mlw , owensmk@earthlink.net, pgsql-hackers@postgresql.org, Tom Lane Subject: Re: [HACKERS] Connection Pooling, a year later References: <200112180357.fBI3vBm24991@candle.pha.pa.us> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Status: OR Bruce Momjian wrote: > Yes, that is assuming you are using PHP. If you are using something > else, you connection pooling in there too. All those client interfaces > reimplementing connection pooling seems like a waste to me. Effective pooling's pretty specific to your environment, though, so any general mechanism would have to provide a wide-ranging suite of parameters governing the number to pool, how long each handle should live, what to do if a handle's released by a client while in the midst of a transaction (AOLserver rolls back the transaction, other clients might want to do something else, i.e. fire a callback or the like), etc etc. I think it would be fairly complex and for those high-throughput applications already written with client-side pooling no improvement. And those are the only applications that need it. -- Don Baccus Portland, OR http://donb.photo.net, http://birdnotes.net, http://openacs.org From pgsql-hackers-owner+M16726@postgresql.org Tue Dec 18 11:48:16 2001 Return-path: Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fBIGmG422658 for ; Tue, 18 Dec 2001 11:48:16 -0500 (EST) Received: from postgresql.org (postgresql.org [64.49.215.8]) by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fBIGkFN40986; Tue, 18 Dec 2001 10:46:15 -0600 (CST) (envelope-from pgsql-hackers-owner+M16726@postgresql.org) Received: from comet.pacifier.com ([199.2.117.155]) by postgresql.org (8.11.3/8.11.4) with ESMTP id fBIGYZm93404 for ; Tue, 18 Dec 2001 11:34:35 -0500 (EST) (envelope-from dhogaza@pacifier.com) Received: from pacifier.com (dsl-dhogaza.pacifier.net [207.202.226.68]) by comet.pacifier.com (8.11.2/8.11.1) with ESMTP id fBIGXCX29823; Tue, 18 Dec 2001 08:33:12 -0800 (PST) Message-ID: <3C1F6FF1.9030606@pacifier.com> Date: Tue, 18 Dec 2001 08:33:53 -0800 From: Don Baccus User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.6) Gecko/20011120 X-Accept-Language: en-us MIME-Version: 1.0 To: Mark Pritchard cc: pgsql-hackers@postgresql.org Subject: Re: [HACKERS] Connection Pooling, a year later References: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR Mark Pritchard wrote: >>I think it is the startup cost that most people want to avoid, and our's >>is higher than most db's that use threads; at least I think so. >> >>It would just be nice to have it done internally rather than have all >>the clients do it, iff it can be done cleanly. >> > > I'd add that client side connection pooling isn't effective in some cases > anyway - one application we work with has 4 physical application servers > running around 6 applications. Each of the applications was written by a > different vendor, and thus a pool size of five gives you 120 open > connections. Tuning a central pooling mechanism to run well in this kind of situation isn't going to be a trivial task, either. The next thing you'll want is some way to prioritize the various clients so your more serious applications have a better chance of getting a pool. Or you'll want to set up subpools so they don't compete with each other, in effect replicating what's done now, but adding more complexity to the central service. -- Don Baccus Portland, OR http://donb.photo.net, http://birdnotes.net, http://openacs.org ---------------------------(end of broadcast)--------------------------- TIP 6: Have you searched our list archives? http://archives.postgresql.org From oleg@sai.msu.su Tue Dec 18 12:05:51 2001 Return-path: Received: from ra.sai.msu.su (ra.sai.msu.su [158.250.29.2]) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fBIH5h423591 for ; Tue, 18 Dec 2001 12:05:43 -0500 (EST) Received: from ra (ra [158.250.29.2]) by ra.sai.msu.su (8.9.3/8.9.3) with ESMTP id UAA18592; Tue, 18 Dec 2001 20:05:26 +0300 (GMT) Date: Tue, 18 Dec 2001 20:05:26 +0300 (GMT) From: Oleg Bartunov X-X-Sender: To: Don Baccus cc: Bruce Momjian , mlw , , , Tom Lane Subject: Re: [HACKERS] Connection Pooling, a year later In-Reply-To: <3C1F6DBF.2040000@pacifier.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Status: OR Does schema support will resolve this discussion ? If I understand correctly, initial arguments for connection pooling was restriction in number of persistent connections. it's right in current postgresql that if one wants keep connection for performance reason to several databases the total number of connections will doubled, trippled and so on. But if I understand schema support will eventually put away these problem because we could keep only one pool of connections to the *one* database. Oleg On Tue, 18 Dec 2001, Don Baccus wrote: > Bruce Momjian wrote: > > > > Yes, that is assuming you are using PHP. If you are using something > > else, you connection pooling in there too. All those client interfaces > > reimplementing connection pooling seems like a waste to me. > > > Effective pooling's pretty specific to your environment, though, so any > general mechanism would have to provide a wide-ranging suite of > parameters governing the number to pool, how long each handle should > live, what to do if a handle's released by a client while in the midst > of a transaction (AOLserver rolls back the transaction, other clients > might want to do something else, i.e. fire a callback or the like), etc etc. > > I think it would be fairly complex and for those high-throughput > applications already written with client-side pooling no improvement. > > And those are the only applications that need it. > > Regards, Oleg _____________________________________________________________ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83 From pgsql-hackers-owner+M16748@postgresql.org Tue Dec 18 15:11:46 2001 Return-path: Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fBIKBj405415 for ; Tue, 18 Dec 2001 15:11:45 -0500 (EST) Received: from postgresql.org (postgresql.org [64.49.215.8]) by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fBIKB0N47430; Tue, 18 Dec 2001 14:11:00 -0600 (CST) (envelope-from pgsql-hackers-owner+M16748@postgresql.org) Received: from candle.pha.pa.us (216-55-132-35.dialup.tnt01.san-diego.abac.net [216.55.132.35]) by postgresql.org (8.11.3/8.11.4) with ESMTP id fBIJulm05030 for ; Tue, 18 Dec 2001 14:56:47 -0500 (EST) (envelope-from pgman@candle.pha.pa.us) Received: (from pgman@localhost) by candle.pha.pa.us (8.11.6/8.10.1) id fBIJuVB04553; Tue, 18 Dec 2001 14:56:31 -0500 (EST) From: Bruce Momjian Message-ID: <200112181956.fBIJuVB04553@candle.pha.pa.us> Subject: Re: [HACKERS] Connection Pooling, a year later In-Reply-To: <3C1F6ED6.6080107@pacifier.com> "from Don Baccus at Dec 18, 2001 08:29:10 am" To: Don Baccus Date: Tue, 18 Dec 2001 14:56:31 -0500 (EST) cc: Christopher Kings-Lynne , mlw , owensmk@earthlink.net, pgsql-hackers@postgresql.org, Tom Lane X-Mailer: ELM [version 2.4ME+ PL90 (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR > Bruce Momjian wrote: > > > > > > The trick for that is to call COMMIT before you pass the backend to a > > new person. > > > The failure to COMMIT is a programmer error - ROLLBACK's much safer. At > least that's what we decided in the AOLserver community, and that's > what the drivers for Oracle and PG (the two I maintain) implement. Then you can issue a "BEGIN;ROLLBACK;" when you pass the session to the next user, and "RESET ALL;" of course. > > Now, if you want to abort a left-over transaction, you can > > do an ABORT but that is going to show up in the server logs because an > > ABORT without a transaction causes an error message. > > > The connection pooling mechanism needs to track the transaction state > and only ROLLBACK a handle that's not in autocommit state or in the > midst of a BEGIN/END transaction (again, Oracle vs. PG).. Seems like a lot of work to keep track of transaction state in the client; seems easier to just unconditionally issue the begin;rollback. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania 19026 ---------------------------(end of broadcast)--------------------------- TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org From pgsql-hackers-owner+M16793@postgresql.org Wed Dec 19 00:46:50 2001 Return-path: Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fBJ5kn426988 for ; Wed, 19 Dec 2001 00:46:49 -0500 (EST) Received: from postgresql.org (postgresql.org [64.49.215.8]) by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fBJ5gnN63439; Tue, 18 Dec 2001 23:42:49 -0600 (CST) (envelope-from pgsql-hackers-owner+M16793@postgresql.org) Received: from deborah.paradise.net.nz (deborah.paradise.net.nz [203.96.152.32]) by postgresql.org (8.11.3/8.11.4) with ESMTP id fBJ5Uvm41224 for ; Wed, 19 Dec 2001 00:30:58 -0500 (EST) (envelope-from andrew@catalyst.net.nz) Received: from heidegger.catalyst.net.nz (203-96-145-108.adsl.paradise.net.nz [203.96.145.108]) by deborah.paradise.net.nz (Postfix) with ESMTP id D1C7CD194D; Wed, 19 Dec 2001 18:31:01 +1300 (NZDT) Received: from 127.0.0.1 (ident=unknown) by heidegger.catalyst.net.nz with esmtp (MasqMail 0.1.15) id 16GZJK-5NU-00; Wed, 19 Dec 2001 18:30:34 +1300 Subject: Re: [HACKERS] Connection Pooling, a year later From: Andrew McMillan To: owensmk@earthlink.net cc: pgsql-hackers@postgresql.org In-Reply-To: <200112180028.fBI0Sum06915@postgresql.org> References: <200112180028.fBI0Sum06915@postgresql.org> Content-Type: text/plain Content-Transfer-Encoding: 7bit X-Mailer: Evolution/1.0 (Preview Release) Date: 19 Dec 2001 18:30:34 +1300 Message-ID: <1008739834.25608.33.camel@kant.mcmillan.net.nz> MIME-Version: 1.0 Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR On Tue, 2001-12-18 at 13:46, Michael Owens wrote: > > By having the postmaster map multiple clients to a fixed number of backends, > you achieve the happy medium: You never exceed the ideal number of active > backends, and at the same time you are not limited to only accepting a fixed > number of connections. Accepting connections can now be based on load > (however you wish to define it), not number. You now make decisions based on > utlization. > > If it were shown that even half of a backend's life consisted of idle time, > leasing out that idle time to another active connection would potentially > double the average number of simultaneous requests without (theoretically) > incurring any significant degradation in performance. > Have you looked at the client-side connection pooling solutions out there? DBBalancer ( http://dbbalancer.sourceforge.net/ ) tries to sit very transparently between your application and PostgreSQL, letting you implement connection pooling with almost no application changes. There was another one I came across too, but that one requires you to make more wide-reaching changes to the application. In my applications I have found DBBalancer to be roughly the same level of performance as PHP persistent connections, but a lot fewer connections are needed in the pool because they are only needed when Apache is delivering dynamic content - not the associated static stylesheets and images. Regards, Andrew. -- -------------------------------------------------------------------- Andrew @ Catalyst .Net.NZ Ltd, PO Box 11-053, Manners St, Wellington WEB: http://catalyst.net.nz/ PHYS: Level 2, 150-154 Willis St DDI: +64(4)916-7201 MOB: +64(21)635-694 OFFICE: +64(4)499-2267 Are you enrolled at http://schoolreunions.co.nz/ yet? ---------------------------(end of broadcast)--------------------------- TIP 6: Have you searched our list archives? http://archives.postgresql.org From pgsql-hackers-owner+M16834@postgresql.org Wed Dec 19 14:17:47 2001 Return-path: Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fBJJHk404096 for ; Wed, 19 Dec 2001 14:17:46 -0500 (EST) Received: from postgresql.org (postgresql.org [64.49.215.8]) by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fBJJENN87550; Wed, 19 Dec 2001 13:14:23 -0600 (CST) (envelope-from pgsql-hackers-owner+M16834@postgresql.org) Received: from asteroid.pacifier.com ([199.2.117.154]) by postgresql.org (8.11.3/8.11.4) with ESMTP id fBJJ55m16181 for ; Wed, 19 Dec 2001 14:05:05 -0500 (EST) (envelope-from dhogaza@pacifier.com) Received: from pacifier.com (dsl-dhogaza.pacifier.net [207.202.226.68]) by asteroid.pacifier.com (8.11.2/8.11.1) with ESMTP id fBJJ3fe20585; Wed, 19 Dec 2001 11:03:41 -0800 (PST) Message-ID: <3C20E4B9.8090200@pacifier.com> Date: Wed, 19 Dec 2001 11:04:25 -0800 From: Don Baccus User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.6) Gecko/20011120 X-Accept-Language: en-us MIME-Version: 1.0 To: owensmk@earthlink.net cc: Andrew McMillan , pgsql-hackers@postgresql.org Subject: Re: [HACKERS] Connection Pooling, a year later References: <200112180028.fBI0Sum06915@postgresql.org> <1008739834.25608.33.camel@kant.mcmillan.net.nz> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR Michael Owens wrote: > As long as each client's call is composed of a standalone transaction, there > is no problem with external connection pools. But what about when a client's > transactions spans two or more calls, such as SELECT FOR UPDATE? Then pooling > is not safe: it offers no assurance of what may be interjected into an open > transaction between calls. For example, each is a separate call to a shared > connection: > > Client A: BEGIN WORK; SELECT last_name from customer for update where ; > > Client B: BEGIN WORK; SELECT street from customer for update where ; > > Client A: update customer set lastname= where ; COMMIT > WORK; > > > Now, isn't Client B's write lock gone with Client A's commit? Yet Client A's > lock is still hanging around. While Client B's commit will close it, Client B > has lost the assurance of its lock, defeating the purpose of SELECT FOR > UPDATE. > > If this is corrent, then external connection pools limit what you can do with > the database to a single call. Any transaction spanning more than one call is > unsafe, because it is not isolated from other clients sharing the same > connection. The general idea is that you grab a handle and hold onto it until you're done. This makes the above scenario impossible. Forgetting to commit or rollback before relenquishing the handle is another scenario that can lead to problems but that's already been discussed in detail. -- Don Baccus Portland, OR http://donb.photo.net, http://birdnotes.net, http://openacs.org ---------------------------(end of broadcast)--------------------------- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) From pgsql-hackers-owner+M16838@postgresql.org Wed Dec 19 15:17:32 2001 Return-path: Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fBJKHV408663 for ; Wed, 19 Dec 2001 15:17:32 -0500 (EST) Received: from postgresql.org (postgresql.org [64.49.215.8]) by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fBJKDNN89347; Wed, 19 Dec 2001 14:13:23 -0600 (CST) (envelope-from pgsql-hackers-owner+M16838@postgresql.org) Received: from gull.prod.itd.earthlink.net (gull.mail.pas.earthlink.net [207.217.120.84]) by postgresql.org (8.11.3/8.11.4) with ESMTP id fBJKA2m62023 for ; Wed, 19 Dec 2001 15:10:03 -0500 (EST) (envelope-from owensmk@earthlink.net) Received: from sdn-ar-004txfworp179.dialsprint.net ([158.252.142.219] helo=there) by gull.prod.itd.earthlink.net with smtp (Exim 3.33 #1) id 16Gn2K-0005YP-00; Wed, 19 Dec 2001 12:09:57 -0800 Content-Type: text/plain; charset="iso-8859-1" From: Michael Owens Reply-To: owensmk@earthlink.net To: Don Baccus Subject: Re: [HACKERS] Connection Pooling, a year later Date: Wed, 19 Dec 2001 14:28:14 -0600 X-Mailer: KMail [version 1.3.1] cc: Andrew McMillan , pgsql-hackers@postgresql.org References: <200112180028.fBI0Sum06915@postgresql.org> <3C20E4B9.8090200@pacifier.com> In-Reply-To: <3C20E4B9.8090200@pacifier.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Message-ID: Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR On Wednesday 19 December 2001 01:04 pm, Don Baccus wrote: > The general idea is that you grab a handle and hold onto it until you're > done. This makes the above scenario impossible. > > Forgetting to commit or rollback before relenquishing the handle is > another scenario that can lead to problems but that's already been > discussed in detail. But then the shared connection is unshared, sitting idle while the client works in between calls, thus introducing idle time among a fixed number of connections. The server is doing less than it could. I agree that this connection pool has improved things in eliminating backend startup time. But idle time still exists for the clients performing multiple calls, proportional to the product of the number of multiple call clients and the number of calls they make, plus the idle time between them. However this probably only ever happens on update. Inserts and selects can be done in one call. And, I suppose updates comprise only a small fraction of the requests sent to the database. Even then, you can probably eliminate some multiple calls by using things such as procedures. Factoring all that in, you can probably do as well by optimizing your particular database/application than by writing code. I relent. Thanks for your thoughts. ---------------------------(end of broadcast)--------------------------- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to majordomo@postgresql.org so that your message can get through to the mailing list cleanly From pgsql-hackers-owner+M16855@postgresql.org Thu Dec 20 01:02:51 2001 Return-path: Received: from rs.postgresql.org (server1.pgsql.org [64.39.15.238] (may be forged)) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id fBK62o404294 for ; Thu, 20 Dec 2001 01:02:50 -0500 (EST) Received: from postgresql.org (postgresql.org [64.49.215.8]) by rs.postgresql.org (8.11.6/8.11.6) with ESMTP id fBK5xnN05417; Wed, 19 Dec 2001 23:59:49 -0600 (CST) (envelope-from pgsql-hackers-owner+M16855@postgresql.org) Received: from deborah.paradise.net.nz (deborah.paradise.net.nz [203.96.152.32]) by postgresql.org (8.11.3/8.11.4) with ESMTP id fBK5mvm47263 for ; Thu, 20 Dec 2001 00:48:57 -0500 (EST) (envelope-from andrew@catalyst.net.nz) Received: from heidegger.catalyst.net.nz (203-96-145-94.adsl.paradise.net.nz [203.96.145.94]) by deborah.paradise.net.nz (Postfix) with ESMTP id 7407FD2B76; Thu, 20 Dec 2001 18:49:01 +1300 (NZDT) Received: from 127.0.0.1 (ident=unknown) by heidegger.catalyst.net.nz with esmtp (MasqMail 0.1.15) id 16GrRk-2Ry-00; Thu, 20 Dec 2001 13:52:28 +1300 Subject: Re: [HACKERS] Connection Pooling, a year later From: Andrew McMillan To: owensmk@earthlink.net cc: pgsql-hackers@postgresql.org In-Reply-To: References: <200112180028.fBI0Sum06915@postgresql.org> <1008739834.25608.33.camel@kant.mcmillan.net.nz> Content-Type: text/plain Content-Transfer-Encoding: 7bit X-Mailer: Evolution/1.0 (Preview Release) Date: 20 Dec 2001 13:52:28 +1300 Message-ID: <1008809548.24470.48.camel@kant.mcmillan.net.nz> MIME-Version: 1.0 Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR On Thu, 2001-12-20 at 07:22, Michael Owens wrote: > As long as each client's call is composed of a standalone transaction, there > is no problem with external connection pools. But what about when a client's > transactions spans two or more calls, such as SELECT FOR UPDATE? Then pooling > is not safe: it offers no assurance of what may be interjected into an open > transaction between calls. For example, each is a separate call to a shared > connection: > > Client A: BEGIN WORK; SELECT last_name from customer for update where ; > > Client B: BEGIN WORK; SELECT street from customer for update where ; > > Client A: update customer set lastname= where ; COMMIT > WORK; > > > Now, isn't Client B's write lock gone with Client A's commit? Yet Client A's > lock is still hanging around. While Client B's commit will close it, Client B > has lost the assurance of its lock, defeating the purpose of SELECT FOR > UPDATE. > > If this is corrent, then external connection pools limit what you can do with > the database to a single call. Any transaction spanning more than one call is > unsafe, because it is not isolated from other clients sharing the same > connection. Oh, I see. You are absolutely correct that client-side pooling wouldn't work in that situation of course. As an application developer nobody has forced me into such a corner yet, however. Long running transactions are something I avoid like the plague. Cheers, Andrew. -- -------------------------------------------------------------------- Andrew @ Catalyst .Net.NZ Ltd, PO Box 11-053, Manners St, Wellington WEB: http://catalyst.net.nz/ PHYS: Level 2, 150-154 Willis St DDI: +64(4)916-7201 MOB: +64(21)635-694 OFFICE: +64(4)499-2267 Are you enrolled at http://schoolreunions.co.nz/ yet? ---------------------------(end of broadcast)--------------------------- TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org