From owner-pgsql-hackers@hub.org Tue Oct 13 15:05:53 1998 Received: from hub.org (majordom@hub.org [209.47.148.200]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id PAA09435 for ; Tue, 13 Oct 1998 15:05:50 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id OAA11700; Tue, 13 Oct 1998 14:43:31 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Tue, 13 Oct 1998 14:41:03 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id OAA11395 for pgsql-hackers-outgoing; Tue, 13 Oct 1998 14:41:00 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from terry1.acun.com (terry@terry1.acun.com [206.27.86.12]) by hub.org (8.8.8/8.8.8) with ESMTP id OAA11372 for ; Tue, 13 Oct 1998 14:40:54 -0400 (EDT) (envelope-from terry@terrym.com) Received: from localhost (terry@localhost) by terry1.acun.com (8.8.5/8.8.5) with SMTP id OAA09491 for ; Tue, 13 Oct 1998 14:53:22 -0400 Date: Tue, 13 Oct 1998 14:53:22 -0400 (EDT) From: Terry Mackintosh X-Sender: terry@terry1.acun.com Reply-To: Terry Mackintosh To: PostgreSQL-development Subject: Re: [HACKERS] What about LIMIT in SELECT ? In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: ROr Hi, my 2 cents... I agree completely, LIMIT would be VERY usefull in web based apps, which is all I run. It does not matter to me if it is not part of a formal standard. The idea is so common that it is a defacto standard. I would not expect it for this release, but could it get put on the TODO list for next time? I am even willing to work at an apprentise level on this with a more expeireanced person that knows this stuff. A note on implimentation: I *used to* :) work with VFP on NT's :( And the way VFP did LIMIT, it would only return the number of rows asked for, BUT it still did the WHOLE search! So on a larger table, which we had (property tax database for the county), if some one put in too vague a query, it would try to collect ALL of the rows as the initial result set, then give you the first x rows of that. This did save on pushing mass amounts of data out to the browser, but it would have been even better if it could have simply aborted the select after having found x rows. Also, it did not have the concept of an offset, so one could not select 100 rows, starting 200 rows in, which would be REALLY usefull for "paging" through data. I do not know if mySQL or any other has such a concept either, but it would be nice. So a properly implemented "LIMIT" could: 1. Save pushing mass amounts of data across the web, that no one wants any way. 2. Stop vague queries from bogging down the server. (On very larg tables this could be critical!) 3. Enable "Paging" of data. (easyer then now (app. level)) 4. Would be a very nice feather in PostgreSQL's cap that could make it even more attractive to those looking at all sorts of databases out there. Have a great day. On Tue, 13 Oct 1998, Oleg Bartunov wrote: > Hi, > > I took a look at mysql and was very impressed with possibility > to limit number of rows returned from select. This is very useful > feature for Web applications when user need to browse results of > selection page by page. In my application I have to do full > select every time user press button [Next] and show requested page > using perl. This works more or less ok for several thousands rows but > totally unusable for large selections. But now I'm about to work > with big database and I don't know how I'll stay with postgres :-) > It'll just doesn't work if customer will wait several minutes just browse > next page. Mysql lacks some useful features postgres has > (subselects, transaction ..) but for most Web applications I need > just select :-) I dont' know how LIMIT is implemented in Mysql and > I know it's not in SQL92 standart, but this makes Mysql very popular. > > Is it difficult to implement this feature in postgres ? > > Regards, > > Oleg > > > _____________________________________________________________ > Oleg Bartunov, sci.researcher, hostmaster of AstroNet, > Sternberg Astronomical Institute, Moscow University (Russia) > Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ > phone: +007(095)939-16-83, +007(095)939-23-83 > > Terry Mackintosh http://www.terrym.com sysadmin/owner Please! No MIME encoded or HTML mail, unless needed. Proudly powered by R H Linux 4.2, Apache 1.3, PHP 3, PostgreSQL 6.3 ------------------------------------------------------------------- Success Is A Choice ... book by Rick Patino, get it, read it! From owner-pgsql-hackers@hub.org Tue Oct 13 18:12:41 1998 Received: from hub.org (majordom@hub.org [209.47.148.200]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id SAA12156 for ; Tue, 13 Oct 1998 18:12:39 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id RAA04181; Tue, 13 Oct 1998 17:56:17 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Tue, 13 Oct 1998 17:54:49 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id RAA03869 for pgsql-hackers-outgoing; Tue, 13 Oct 1998 17:54:47 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from remapcorp.com (root@remapcorp.com [206.196.37.193]) by hub.org (8.8.8/8.8.8) with ESMTP id RAA03838 for ; Tue, 13 Oct 1998 17:54:36 -0400 (EDT) (envelope-from jeff@remapcorp.com) Received: from go-to-jail (gotojail.remapcorp.com [206.196.37.197]) by remapcorp.com (8.8.7/8.8.7) with SMTP id QAA25337; Tue, 13 Oct 1998 16:55:35 -0500 (CDT) (envelope-from jeff@remapcorp.com) Message-ID: <006701bdf6f4$60ed75f0$c525c4ce@go-to-jail.remapcorp.com> From: "Jeff Hoffmann" To: "Marc G. Fournier" , "Eric Lee Green" Cc: "PostgreSQL-development" Subject: Re: [HACKERS] What about LIMIT in SELECT ? Date: Tue, 13 Oct 1998 16:56:48 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 4.72.3115.0 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: RO >On Tue, 13 Oct 1998, Eric Lee Green wrote: > >> On Tue, 13 Oct 1998, Jeff Hoffmann wrote: >> > >I agree completely, LIMIT would be VERY usefull in web based apps, which >> > >is all I run. It does not matter to me if it is not part of a formal >> > >standard. The idea is so common that it is a defacto standard. >> > >> > i'm not familiar with mysql and using "LIMIT" but wouldn't this same effect >> > be achieved by declaring a cursor and fetching however many records in the >> > cursor? it's a very noticeable improvement when you only want the first 20 >> > out of 500 in a 200k record database, at least. >> >> The problem with declaring a cursor vs. the "LIMIT" clause is that the >> "LIMIT" clause, if used properly by the database engine (along with the >> database engine using indexes in "ORDER BY" clauses) allows the database >> engine to short-circuit the tail end of the query. That is, if you have 25 >> names and the last one ends with BEAVIS, the database engine doesn't have >> to go through the BUTTHEADS and KENNYs and etc. >> >> Theoretically a cursor is superior to the "LIMIT" clause because you're >> eventually going to want the B's and K's and etc. anyhow -- but only in a >> stateful enviornment. In the stateless web environment, a cursor is >> useless because the connection can close at any time even when you're >> using "persistent" connections (and of course when the connection closes >> the cursor closes). > >Ookay, I'm sorry, butyou lost me here. I haven't gotten into using >CURSORs/FETCHs yet, since I haven't need it...but can you give an example >of what you would want to do using a LIMIT? I may be missing something, >but wha is the different between using LIMIT to get X records, and >definiing a cursor to FETCH X records? > >Practical example of *at least* the LIMIT side would be good, so that we >can at least see a physical example of what LIMIT can do that >CURSORs/FETCH can't... > fetch with cursors should work properly (i.e., you can short circuit it by just ending your transaction) my understanding on how this works is exactly how you explained LIMIT to work. here's some empirical proof from one of my sample databases: the sample table i'm using has 156k records (names of people) i'm using a PP180 with 128MB RAM and some old slow SCSI drives. public_mn=> select count(*) from public_ramsey; count ------ 156566 (1 row) i did the following query: public_mn=> select * from public_ramsey where ownerlname ~ 'SMITH'; which returned 711 matches and took about 12 seconds. i did the same thing with a cursor: public_mn=> begin; BEGIN public_mn=> declare test cursor for select * from public_ramsey where ownerlname ~ 'SMITH'; SELECT the select was instantaneous. public_mn=> fetch 20 in test; returns 20 records almost instantaneously. each additional 20 took less than a second, as well. if this isn't what you're talking about, i don't understand what you're saying. jeff From eric@ireland.linux-hw.com Tue Oct 13 18:52:42 1998 Received: from ireland.linux-hw.com (IDENT:eric@ireland.linux-hw.com [199.72.95.215]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id SAA12388 for ; Tue, 13 Oct 1998 18:52:40 -0400 (EDT) Received: from localhost (eric@localhost) by ireland.linux-hw.com (8.8.7/8.8.7) with SMTP id SAA31316; Tue, 13 Oct 1998 18:55:22 -0400 Date: Tue, 13 Oct 1998 18:55:22 -0400 (EDT) From: Eric Lee Green To: Bruce Momjian cc: jeff@remapcorp.com, hackers@postgreSQL.org Subject: Re: [HACKERS] What about LIMIT in SELECT ? In-Reply-To: <199810132116.RAA11249@candle.pha.pa.us> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Status: RO On Tue, 13 Oct 1998, Bruce Momjian wrote: > > Theoretically a cursor is superior to the "LIMIT" clause because you're > > eventually going to want the B's and K's and etc. anyhow -- but only in a > > stateful enviornment. In the stateless web environment, a cursor is > > useless because the connection can close at any time even when you're > > using "persistent" connections (and of course when the connection closes > What we could do is _if_ there is only one table(no joins), and an index > exists that matches the ORDER BY, we could use the index to > short-circuit the query. This is exactly what MySQL does in this situation, except that it can use the ORDER BY to do the short circuiting even if there is a join involved if all of the elements of the ORDER BY belong to one table. Obviously if I'm doing an "ORDER BY table1.foo table2.bar" that isn't going to work! But "select table1.fsname,table1.lname,table2.receivables where table2.receivables > 0 and table1.custnum=table2.custnum order by (table1.lname,table1.fsname) limit 50" can be short-circuited by fiddling with the join order -- table1.fsname table1.lname have to be the first two things in the join order. Whether this is feasible in PostgreSQL I have no earthly idea. This would seem to conflict with the join optimizer. > happier? If there is an ORDER BY and no index, or a join, I can't > figure out how we would short-circuit the query. If there is an ORDER BY and no index you can't short-circuit the query. MySQL doesn't either. Under certain circumstances (such as above) you can short-circuit a join, but it's unclear whether it'd be easy to add such a capability to PostgreSQL given the current structure of the query optimizer. (And I certainly am not in a position to tackle it, at the moment MySQL is sufficing for my project despite the fact that it is quite limited compared to PostgreSQL, I need to get my project finished first). -- Eric Lee Green eric@linux-hw.com http://www.linux-hw.com/~eric "To call Microsoft an innovator is like calling the Pope Jewish ..." -- James Love (Consumer Project on Technology) From owner-pgsql-hackers@hub.org Wed Oct 14 09:01:01 1998 Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id JAA24574 for ; Wed, 14 Oct 1998 09:01:00 -0400 (EDT) Received: from hub.org (majordom@hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id HAA17762 for ; Wed, 14 Oct 1998 07:47:57 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id HAA09214; Wed, 14 Oct 1998 07:04:59 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 14 Oct 1998 07:00:44 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id HAA09116 for pgsql-hackers-outgoing; Wed, 14 Oct 1998 07:00:40 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from dsh.de (firewall-user@neptun.sns-felb.debis.de [53.122.101.2]) by hub.org (8.8.8/8.8.8) with ESMTP id HAA09102 for ; Wed, 14 Oct 1998 07:00:27 -0400 (EDT) (envelope-from wieck@sapserv.debis.de) Received: by dsh.de; id NAA05037; Wed, 14 Oct 1998 13:02:40 +0200 (MET DST) Received: from dshmail.dsh.de(53.47.15.3) by neptun.dsh.de via smap (3.2) id xma004737; Wed, 14 Oct 98 13:02:09 +0200 Received: from mail1.hh1.dsh.de (mail1.hh1.dsh.de [53.47.9.5]) by dshmail.dsh.de (8.8.7/8.8.7) with ESMTP id MAA20155; Wed, 14 Oct 1998 12:59:23 +0200 (MET DST) Received: from mars.SAPserv.Hamburg.dsh.de (root@mail5.hh1.dsh.de [53.2.168.17]) by mail1.hh1.dsh.de (8.8.7/8.8.7) with SMTP id NAA20772; Wed, 14 Oct 1998 13:01:35 +0200 Received: from orion.SAPserv.Hamburg.dsh.de by mars.SAPserv.Hamburg.dsh.de with smtp for <> id m0zTMGL-000B5AC; Wed, 14 Oct 98 10:26 MET DST Received: by orion.SAPserv.Hamburg.dsh.de for eric@linux-hw.com id m0zTOnx-000EBRC; Wed, 14 Oct 98 13:09 MET DST Message-Id: From: jwieck@debis.com (Jan Wieck) Subject: Re: [HACKERS] What about LIMIT in SELECT ? To: eric@linux-hw.com (Eric Lee Green) Date: Wed, 14 Oct 1998 13:09:21 +0200 (MET DST) Cc: jeff@remapcorp.com, hackers@postgreSQL.org Reply-To: jwieck@debis.com (Jan Wieck) In-Reply-To: from "Eric Lee Green" at Oct 13, 98 04:24:20 pm X-Mailer: ELM [version 2.4 PL25] Content-Type: text Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: ROr Eric Lee Green wrote: > > On Tue, 13 Oct 1998, Jeff Hoffmann wrote: > > >I agree completely, LIMIT would be VERY usefull in web based apps, which > > >is all I run. It does not matter to me if it is not part of a formal > > >standard. The idea is so common that it is a defacto standard. > > > > i'm not familiar with mysql and using "LIMIT" but wouldn't this same effect > > be achieved by declaring a cursor and fetching however many records in the > > cursor? it's a very noticeable improvement when you only want the first 20 > > out of 500 in a 200k record database, at least. > > The problem with declaring a cursor vs. the "LIMIT" clause is that the > "LIMIT" clause, if used properly by the database engine (along with the > database engine using indexes in "ORDER BY" clauses) allows the database > engine to short-circuit the tail end of the query. That is, if you have 25 > names and the last one ends with BEAVIS, the database engine doesn't have > to go through the BUTTHEADS and KENNYs and etc. > > Theoretically a cursor is superior to the "LIMIT" clause because you're > eventually going to want the B's and K's and etc. anyhow -- but only in a > stateful enviornment. In the stateless web environment, a cursor is > useless because the connection can close at any time even when you're > using "persistent" connections (and of course when the connection closes > the cursor closes). I'm missing something. Well it's right that in the stateless web environment a cursor has to be declared and closed for any single CGI call. But even if you have a LIMIT clause, your CGI must know with which key to start. So your query must look like SELECT ... WHERE key > 'last processed key' ORDER BY key; And your key must be unique (or at least contain no duplicate entries) or you might miss some rows between the pages (have 100 Brown's in the table and last processed key was a Brown while using LIMIT). In postgres you could actually do the following (but read on below - it's not optimized correct) BEGIN; DECLARE c CURSOR FOR SELECT ... WHERE key > 'last' ORDER BY key; FETCH 20 IN c; (process the 20 rows in CGI) CLOSE c; COMMIT; Having LIMIT looks more elegant and has less overhead in CGI- backend communication. But the cursor version is SQL standard and portable. > > I wanted very badly to use PostgreSQL for a web project I'm working on, > but it just wouldn't do the job :-(. I've done some tests and what I found out might be a bug in PostgreSQL's query optimizer. Having a table with 25k rows where key is a text field with a unique index. Now I used EXPLAIN for some queries SELECT * FROM tab; results in a seqscan - expected. SELECT * FROM tab ORDER BY key; results in a sort->seqscan - I would have expected an indexscan! SELECT * FROM tab WHERE key > 'G'; results in an indexscan - expected. SELECT * FROM tab WHERE key > 'G' ORDER BY key; results in a sort->indexscan - hmmm. These results stay the same even if I blow up the table by duplicating all rows (now with a non-unique index) to 100k rows and have them presorted in the table. Needless to say that everything is vacuum'd for statistics. The last one is the query we would need in the web environment used over a cursor as in the example above. But due to the sort, the backend selects until the end of the table, sorts them and then returns only the first 20 rows (out of sorts result). This is very painful if the qualification (key > ...) points to the beginning of the key list. Looking at planner.c I can see, that if there is a sortClause in the parsetree, the planner creates a sort node and does absolutely not check if there is an index that could be used to do it. In the examples above, the sort is absolutely needless because the index scan will already return the tuples in the right order :-). Somewhere deep in my brain I found a statement that sorting sorted data isn't only unnecessary (except the order changes), it is slow too compared against sorting randomly ordered data. Can we fix this before 6.4 release, will it be a past 6.4 or am I doing something wrong here? I think it isn't a fix (it's a planner enhancement) so it should really be a past 6.4 item. For now, the only possibility is to omit the ORDER BY in the query and hope the planner will always generate an index scan (because of the qualification 'key > ...'). Doing so I selected multiple times 20 rows (with the last key qual like a CGI would do) in separate transactions. Using cursor and fetch speeds up the access by a factor of 1000! But it is unsafe and thus NOT RECOMMENDED! It's only a test if cursors can do the LIMIT job - and they could if the planner would do a better job. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #======================================== jwieck@debis.com (Jan Wieck) # From owner-pgsql-hackers@hub.org Wed Oct 14 11:02:04 1998 Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id LAA25519 for ; Wed, 14 Oct 1998 11:02:02 -0400 (EDT) Received: from hub.org (majordom@hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id JAA24583 for ; Wed, 14 Oct 1998 09:46:21 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id IAA17022; Wed, 14 Oct 1998 08:59:20 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 14 Oct 1998 08:54:40 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id IAA16687 for pgsql-hackers-outgoing; Wed, 14 Oct 1998 08:54:34 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from ra.sai.msu.su (ra.sai.msu.su [158.250.29.2]) by hub.org (8.8.8/8.8.8) with ESMTP id IAA16656 for ; Wed, 14 Oct 1998 08:54:00 -0400 (EDT) (envelope-from oleg@sai.msu.su) Received: from ra (ra [158.250.29.2]) by ra.sai.msu.su (8.9.1/8.9.1) with SMTP id PAA11714; Wed, 14 Oct 1998 15:53:53 +0300 (MSK) Date: Wed, 14 Oct 1998 16:53:53 +0400 (MSD) From: Oleg Bartunov X-Sender: megera@ra Reply-To: Oleg Bartunov To: hackers@postgreSQL.org cc: t-ishii@sra.co.jp Subject: Re: [HACKERS] What about LIMIT in SELECT ? In-Reply-To: Message-ID: Organization: Sternberg Astronomical Institute (Moscow University) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: RO On Wed, 14 Oct 1998, Jan Wieck wrote: > Date: Wed, 14 Oct 1998 13:09:21 +0200 (MET DST) > From: Jan Wieck > To: Eric Lee Green > Cc: jeff@remapcorp.com, hackers@postgreSQL.org > Subject: Re: [HACKERS] What about LIMIT in SELECT ? > > Eric Lee Green wrote: > > > > On Tue, 13 Oct 1998, Jeff Hoffmann wrote: > > > >I agree completely, LIMIT would be VERY usefull in web based apps, which > > > >is all I run. It does not matter to me if it is not part of a formal > > > >standard. The idea is so common that it is a defacto standard. > > > > > > i'm not familiar with mysql and using "LIMIT" but wouldn't this same effect > > > be achieved by declaring a cursor and fetching however many records in the > > > cursor? it's a very noticeable improvement when you only want the first 20 > > > out of 500 in a 200k record database, at least. > > > > The problem with declaring a cursor vs. the "LIMIT" clause is that the > > "LIMIT" clause, if used properly by the database engine (along with the > > database engine using indexes in "ORDER BY" clauses) allows the database > > engine to short-circuit the tail end of the query. That is, if you have 25 > > names and the last one ends with BEAVIS, the database engine doesn't have > > to go through the BUTTHEADS and KENNYs and etc. > > > > Theoretically a cursor is superior to the "LIMIT" clause because you're > > eventually going to want the B's and K's and etc. anyhow -- but only in a > > stateful enviornment. In the stateless web environment, a cursor is > > useless because the connection can close at any time even when you're > > using "persistent" connections (and of course when the connection closes > > the cursor closes). > > I'm missing something. Well it's right that in the stateless > web environment a cursor has to be declared and closed for > any single CGI call. But even if you have a LIMIT clause, > your CGI must know with which key to start. > This is not a problem for CGI-script to know which key to start. Without LIMIT every CGI call backend will do *FULL* selection and cursor helps just in fetching a definite number of rows, in principle I can do this with CGI-script. Also, cursor returns data back in ASCII format (man l declare) and this requires additional job for backend to convert data from intrinsic (binary) format. Right implementation of LIMIT offset,number_of_rows could be a great win and make postgres superior free database engine for Web applications. Many colleagues of mine used mysql instead of postgres just because of lacking LIMIT. Tatsuo posted a patch for set query_limit to 'num', I just tested it and seems it works fine. Now, we need only possibility to specify offset, say set query_limit to 'offset,num' ( Tatsuo, How difficult to do this ?) and LIMIT problem will ne gone. I'm wonder how many useful patches could be hidden from people :-), Regards, Oleg PS. Tatsuo, do you have patch for 6.3.2 ? I can't wait for 6.4 :-) _____________________________________________________________ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83 From owner-pgsql-hackers@hub.org Wed Oct 14 11:02:00 1998 Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id LAA25510 for ; Wed, 14 Oct 1998 11:01:59 -0400 (EDT) Received: from hub.org (majordom@hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id KAA28854 for ; Wed, 14 Oct 1998 10:40:56 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id KAA21542; Wed, 14 Oct 1998 10:03:45 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 14 Oct 1998 09:59:10 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id JAA21121 for pgsql-hackers-outgoing; Wed, 14 Oct 1998 09:59:08 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from golem.jpl.nasa.gov (root@hectic-2.jpl.nasa.gov [128.149.68.204]) by hub.org (8.8.8/8.8.8) with ESMTP id JAA21106 for ; Wed, 14 Oct 1998 09:59:02 -0400 (EDT) (envelope-from lockhart@alumni.caltech.edu) Received: from alumni.caltech.edu (localhost [127.0.0.1]) by golem.jpl.nasa.gov (8.8.5/8.8.5) with ESMTP id NAA19587; Wed, 14 Oct 1998 13:59:56 GMT Message-ID: <3624AE5C.752E4E7F@alumni.caltech.edu> Date: Wed, 14 Oct 1998 13:59:56 +0000 From: "Thomas G. Lockhart" Organization: Caltech/JPL X-Mailer: Mozilla 4.07 [en] (X11; I; Linux 2.0.30 i686) MIME-Version: 1.0 To: Jan Wieck CC: Eric Lee Green , jeff@remapcorp.com, hackers@postgreSQL.org Subject: Re: [HACKERS] What about LIMIT in SELECT ? References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: ROr > I've done some tests and what I found out might be a bug in > PostgreSQL's query optimizer. > SELECT * FROM tab ORDER BY key; > results in a sort->seqscan - I would have > expected an indexscan! Given that a table _could_ be completely unsorted on disk, it is probably reasonable to suck the data in for a possible in-memory sort rather than skipping around the disk to pick up individual tuples via the index. Don't know if vacuum has a statistic on "orderness"... > SELECT * FROM tab WHERE key > 'G' ORDER BY key; > results in a sort->indexscan - hmmm. > The last one is the query we would need in the web > environment used over a cursor as in the example above. But > due to the sort, the backend selects until the end of the > table, sorts them and then returns only the first 20 rows > (out of sorts result). So you are saying that for this last case the sort was unnecessary? Does the backend traverse the index in the correct order to guarantee that the tuples are coming out already sorted? Does a hash index give the same plan (I would expect a sort->seqscan for a hash index)? - Tom From owner-pgsql-hackers@hub.org Wed Oct 14 11:01:52 1998 Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id LAA25504 for ; Wed, 14 Oct 1998 11:01:51 -0400 (EDT) Received: from hub.org (majordom@hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id KAA00198 for ; Wed, 14 Oct 1998 10:57:15 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id KAA22877; Wed, 14 Oct 1998 10:19:47 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 14 Oct 1998 10:15:44 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id KAA22675 for pgsql-hackers-outgoing; Wed, 14 Oct 1998 10:15:41 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from dsh.de (firewall-user@neptun.sns-felb.debis.de [53.122.101.2]) by hub.org (8.8.8/8.8.8) with ESMTP id KAA22657 for ; Wed, 14 Oct 1998 10:15:32 -0400 (EDT) (envelope-from wieck@sapserv.debis.de) Received: by dsh.de; id QAA20563; Wed, 14 Oct 1998 16:18:02 +0200 (MET DST) Received: from dshmail.dsh.de(53.47.15.3) by neptun.dsh.de via smap (3.2) id xma020404; Wed, 14 Oct 98 16:17:25 +0200 Received: from mail1.hh1.dsh.de (mail1.hh1.dsh.de [53.47.9.5]) by dshmail.dsh.de (8.8.7/8.8.7) with ESMTP id QAA05077; Wed, 14 Oct 1998 16:14:48 +0200 (MET DST) Received: from mars.SAPserv.Hamburg.dsh.de (root@mail5.hh1.dsh.de [53.2.168.17]) by mail1.hh1.dsh.de (8.8.7/8.8.7) with SMTP id QAA22248; Wed, 14 Oct 1998 16:17:06 +0200 Received: from orion.SAPserv.Hamburg.dsh.de by mars.SAPserv.Hamburg.dsh.de with smtp for <> id m0zTPJb-000B5AC; Wed, 14 Oct 98 13:42 MET DST Received: by orion.SAPserv.Hamburg.dsh.de for hackers@postgreSQL.org id m0zTRrE-000EBRC; Wed, 14 Oct 98 16:24 MET DST Message-Id: From: jwieck@debis.com (Jan Wieck) Subject: Re: [HACKERS] What about LIMIT in SELECT ? To: oleg@sai.msu.su Date: Wed, 14 Oct 1998 16:24:56 +0200 (MET DST) Cc: hackers@postgreSQL.org, t-ishii@sra.co.jp Reply-To: jwieck@debis.com (Jan Wieck) In-Reply-To: from "Oleg Bartunov" at Oct 14, 98 04:53:53 pm X-Mailer: ELM [version 2.4 PL25] Content-Type: text Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: RO Oleg Bartunov wrote: > This is not a problem for CGI-script to know which key to start. Never meant that would be a problem. A FORM variable will of course do this. > Without LIMIT every CGI call backend will do *FULL* selection > and cursor helps just in fetching a definite number of rows, > in principle I can do this with CGI-script. Also, cursor > returns data back in ASCII format (man l declare) and this requires > additional job for backend to convert data from intrinsic (binary) > format. Right implementation of LIMIT offset,number_of_rows could be > a great win and make postgres superior free database engine for > Web applications. Many colleagues of mine used mysql instead of That's the point I was missing. The offset! > postgres just because of lacking LIMIT. Tatsuo posted a patch > for set query_limit to 'num', I just tested it and seems it > works fine. Now, we need only possibility to specify offset, > say > set query_limit to 'offset,num' > ( Tatsuo, How difficult to do this ?) > and LIMIT problem will ne gone. Think you haven't read my posting completely. Even with the executor limit, the complete scan into the sort is done by the backend. You need to specify ORDER BY to get the same list again (without the offset doesn't make sense). But currently, ORDER BY forces a sort node into the query plan. What the executor limit tells is how many rows will be returned from the sorted data. Not what goes into the sort. Filling the sort and sorting the data consumes the most time of the queries execution. I haven't looked at Tatsuo's patch very well. But if it limits the amount of data going into the sort (on ORDER BY), it will break it! The requested ordering could be different from what the choosen index might return. The used index is choosen by the planner upon the qualifications given, not the ordering wanted. So if you select WHERE b = 1 ORDER BY a, then it will use an index on attribute b to match the qualification. The complete result of that index scan goes into the sort to get ordered by a. If now the executor limit stops sort filling after the limit is exceeded, only the same tuples will go into the sort every time. But they have nothing to do with the requested order by a. What LIMIT first needs is a planner enhancement. In file backend/optimizer/plan/planner.c line 284 it must be checked if the actual plan is an indexscan, if the indexed attributes are all the same as those in the given sort clause and that the requested sort order (operator) is that what the index will return. If that all matches, it can ignore the sort clause and return the index scan itself. Second enhancement must be the handling of the offset. In the executor, the index scan must skip offset index tuples before returning the first. But NOT if the plan isn't a 1-table-index-scan. In that case the result tuples (from the topmost unique/join/whatever node) have to be skipped. With these enhancements, the index tuples to be skipped (offset) will still be scanned, but not the data tuples they point to. Index scanning might be somewhat faster. This all will only speedup simple 1-table-queries, no joins or if the requested order isn't that what the index exactly returns. Anyway, I'll take a look if I can change the planner to omit the sort if the tests described above are true. I think it would be good anyway. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #======================================== jwieck@debis.com (Jan Wieck) # From owner-pgsql-hackers@hub.org Wed Oct 14 11:01:36 1998 Received: from hub.org (majordom@hub.org [209.47.148.200]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id LAA25489 for ; Wed, 14 Oct 1998 11:01:34 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id KAA24286; Wed, 14 Oct 1998 10:30:14 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 14 Oct 1998 10:26:34 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id KAA23732 for pgsql-hackers-outgoing; Wed, 14 Oct 1998 10:26:27 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from dsh.de (neptun.sns-felb.debis.de [53.122.101.2]) by hub.org (8.8.8/8.8.8) with ESMTP id KAA23717 for ; Wed, 14 Oct 1998 10:26:13 -0400 (EDT) (envelope-from wieck@sapserv.debis.de) Received: by dsh.de; id QAA25644; Wed, 14 Oct 1998 16:28:01 +0200 (MET DST) Received: from dshmail.dsh.de(53.47.15.3) by neptun.dsh.de via smap (3.2) id xma025301; Wed, 14 Oct 98 16:27:43 +0200 Received: from mail1.hh1.dsh.de (mail1.hh1.dsh.de [53.47.9.5]) by dshmail.dsh.de (8.8.7/8.8.7) with ESMTP id QAA05943; Wed, 14 Oct 1998 16:24:42 +0200 (MET DST) Received: from mars.SAPserv.Hamburg.dsh.de (root@mail5.hh1.dsh.de [53.2.168.17]) by mail1.hh1.dsh.de (8.8.7/8.8.7) with SMTP id QAA22339; Wed, 14 Oct 1998 16:26:57 +0200 Received: from orion.SAPserv.Hamburg.dsh.de by mars.SAPserv.Hamburg.dsh.de with smtp for <> id m0zTPT8-000B5AC; Wed, 14 Oct 98 13:51 MET DST Received: by orion.SAPserv.Hamburg.dsh.de for lockhart@alumni.caltech.edu id m0zTS0m-000EBRC; Wed, 14 Oct 98 16:34 MET DST Message-Id: From: jwieck@debis.com (Jan Wieck) Subject: Re: [HACKERS] What about LIMIT in SELECT ? To: lockhart@alumni.caltech.edu (Thomas G. Lockhart) Date: Wed, 14 Oct 1998 16:34:47 +0200 (MET DST) Cc: jwieck@debis.com, eric@linux-hw.com, jeff@remapcorp.com, hackers@postgreSQL.org Reply-To: jwieck@debis.com (Jan Wieck) In-Reply-To: <3624AE5C.752E4E7F@alumni.caltech.edu> from "Thomas G. Lockhart" at Oct 14, 98 01:59:56 pm X-Mailer: ELM [version 2.4 PL25] Content-Type: text Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: RO > > > SELECT * FROM tab WHERE key > 'G' ORDER BY key; > > results in a sort->indexscan - hmmm. > > The last one is the query we would need in the web > > environment used over a cursor as in the example above. But > > due to the sort, the backend selects until the end of the > > table, sorts them and then returns only the first 20 rows > > (out of sorts result). > > So you are saying that for this last case the sort was unnecessary? Does > the backend traverse the index in the correct order to guarantee that > the tuples are coming out already sorted? Does a hash index give the > same plan (I would expect a sort->seqscan for a hash index)? Good point! As far as I can see, the planner chooses index usage only depending on the WHERE clause. A hash index is only usable when the given qualification uses = on the indexed attribute(s). If the sortClause exactly matches the indexed attributes of the ONE used btree index and all operators request ascending order I think the index scan already returns the correct order. Who know's definitely? Addition to my last posting: ... and if the index scan is using a btree index ... Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #======================================== jwieck@debis.com (Jan Wieck) # From owner-pgsql-hackers@hub.org Wed Oct 14 13:55:58 1998 Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id NAA29300 for ; Wed, 14 Oct 1998 13:55:56 -0400 (EDT) Received: from hub.org (majordom@hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id NAA14245 for ; Wed, 14 Oct 1998 13:49:19 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id NAA13110; Wed, 14 Oct 1998 13:25:55 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 14 Oct 1998 13:22:14 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id NAA12694 for pgsql-hackers-outgoing; Wed, 14 Oct 1998 13:22:13 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from candle.pha.pa.us (maillist@s5-03.ppp.op.net [209.152.195.67]) by hub.org (8.8.8/8.8.8) with ESMTP id NAA12677 for ; Wed, 14 Oct 1998 13:22:05 -0400 (EDT) (envelope-from maillist@candle.pha.pa.us) Received: (from maillist@localhost) by candle.pha.pa.us (8.9.0/8.9.0) id NAA28746; Wed, 14 Oct 1998 13:21:15 -0400 (EDT) From: Bruce Momjian Message-Id: <199810141721.NAA28746@candle.pha.pa.us> Subject: Re: [HACKERS] What about LIMIT in SELECT ? In-Reply-To: <3624AE5C.752E4E7F@alumni.caltech.edu> from "Thomas G. Lockhart" at "Oct 14, 1998 1:59:56 pm" To: lockhart@alumni.caltech.edu (Thomas G. Lockhart) Date: Wed, 14 Oct 1998 13:21:15 -0400 (EDT) Cc: jwieck@debis.com, eric@linux-hw.com, jeff@remapcorp.com, hackers@postgreSQL.org X-Mailer: ELM [version 2.4ME+ PL47 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: ROr > > I've done some tests and what I found out might be a bug in > > PostgreSQL's query optimizer. > > SELECT * FROM tab ORDER BY key; > > results in a sort->seqscan - I would have > > expected an indexscan! > > Given that a table _could_ be completely unsorted on disk, it is > probably reasonable to suck the data in for a possible in-memory sort > rather than skipping around the disk to pick up individual tuples via > the index. Don't know if vacuum has a statistic on "orderness"... Thomas is correct on this. Vadim has run some tests, and with our optimized psort() code, the in-memory sort is often faster than using the index to get the tuple, because you are jumping all over the drive. I don't remember, but obviously there is a break-even point where getting X rows using the index on a table of Y rows is faster , but getting X+1 rows on a table of Y rows is faster getting all the rows sequentailly, and doing the sort. You would have to pick only certain queries(no joins, index matches ORDER BY), take the number of rows requested, and the number of rows selected, and figure out if it is faster to use the index, or a sequential scan and do the ORDER BY yourself. Add to this the OFFSET capability. I am not sure how you are going to get into the index and start at the n-th entry, unless perhaps you just sequential scan the index. In fact, many queries just get column already indexed, and we could just pull the data right out of the index. I have added this to the TODO list: * Pull requested data directly from indexes, bypassing heap data I think this has to be post-6.4 work, but I think we need to work in this direction. I am holding off any cnfify fixes for post-6.4, but a 6.4.1 performance release certainly is possible. But, you are correct that certain cases where in index is already being used on a query, you could just skip the sort IF you used the index to get the rows from the base table. -- Bruce Momjian | http://www.op.net/~candle maillist@candle.pha.pa.us | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania 19026 From owner-pgsql-hackers@hub.org Wed Oct 14 13:55:59 1998 Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id NAA29303 for ; Wed, 14 Oct 1998 13:55:58 -0400 (EDT) Received: from hub.org (majordom@hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id NAA13463 for ; Wed, 14 Oct 1998 13:39:05 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id NAA11655; Wed, 14 Oct 1998 13:13:32 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 14 Oct 1998 13:09:41 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id NAA11013 for pgsql-hackers-outgoing; Wed, 14 Oct 1998 13:09:39 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from terry1.acun.com (terry@terry1.acun.com [206.27.86.12]) by hub.org (8.8.8/8.8.8) with ESMTP id NAA10997 for ; Wed, 14 Oct 1998 13:09:30 -0400 (EDT) (envelope-from terry@terrym.com) Received: from localhost (terry@localhost) by terry1.acun.com (8.8.5/8.8.5) with SMTP id NAA14478; Wed, 14 Oct 1998 13:21:51 -0400 Date: Wed, 14 Oct 1998 13:21:51 -0400 (EDT) From: Terry Mackintosh X-Sender: terry@terry1.acun.com To: Jeff Hoffmann cc: PostgreSQL-development Subject: Re: [HACKERS] What about LIMIT in SELECT ? In-Reply-To: <005101bdf6de$f9345150$c525c4ce@go-to-jail.remapcorp.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: RO On Tue, 13 Oct 1998, Jeff Hoffmann wrote: > >Hi, my 2 cents... > > > >I agree completely, LIMIT would be VERY usefull in web based apps, which > >is all I run. It does not matter to me if it is not part of a formal > >standard. The idea is so common that it is a defacto standard. > > i'm not familiar with mysql and using "LIMIT" but wouldn't this same effect > be achieved by declaring a cursor and fetching however many records in the > cursor? it's a very noticeable improvement when you only want the first 20 > out of 500 in a 200k record database, at least. Yes, while this is an improvement, it still has to do the entire query, would be nice if the query could be terminated after a designated number of rows where found, thus freeing system resources that are other wise consumed. I have seen web users run ridculous querys, like search for the letter 'a', and it happens to be a substring search, now the box go'es ape shit for 5 or 10 min.s while it basically gets the whole db as the search result. All this befor you can do a 'FETCH', as I understand FETCH, I will need to read up on it. Note that I do not have any databases that larg on my box, I was thinking back to my VFP/NT experiances. Have a great day Terry Mackintosh http://www.terrym.com sysadmin/owner Please! No MIME encoded or HTML mail, unless needed. Proudly powered by R H Linux 4.2, Apache 1.3, PHP 3, PostgreSQL 6.3 ------------------------------------------------------------------- Success Is A Choice ... book by Rick Patino, get it, read it! From owner-pgsql-hackers@hub.org Wed Oct 14 13:59:05 1998 Received: from hub.org (majordom@hub.org [209.47.148.200]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id NAA29345 for ; Wed, 14 Oct 1998 13:58:59 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id NAA14021; Wed, 14 Oct 1998 13:32:51 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 14 Oct 1998 13:29:09 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id NAA13364 for pgsql-hackers-outgoing; Wed, 14 Oct 1998 13:29:07 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from terry1.acun.com (terry@terry1.acun.com [206.27.86.12]) by hub.org (8.8.8/8.8.8) with ESMTP id NAA13328 for ; Wed, 14 Oct 1998 13:28:56 -0400 (EDT) (envelope-from terry@terrym.com) Received: from localhost (terry@localhost) by terry1.acun.com (8.8.5/8.8.5) with SMTP id NAA14606 for ; Wed, 14 Oct 1998 13:41:25 -0400 Date: Wed, 14 Oct 1998 13:41:24 -0400 (EDT) From: Terry Mackintosh X-Sender: terry@terry1.acun.com To: PostgreSQL-development Subject: Re: [HACKERS] What about LIMIT in SELECT ? In-Reply-To: <199810132116.RAA11249@candle.pha.pa.us> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: RO On Tue, 13 Oct 1998, Bruce Momjian wrote: > What we could do is _if_ there is only one table(no joins), and an index > exists that matches the ORDER BY, we could use the index to > short-circuit the query. > > I have added this item to the TODO list: > > * Allow LIMIT ability on single-table queries that have no ORDER BY or > a matching index > > This looks do-able, and a real win. Would this make web applications > happier? If there is an ORDER BY and no index, or a join, I can't > figure out how we would short-circuit the query. > Yes, this would do for most of my apps. It may just be my lack of sophistication, but I find that most web apps are very simple in nature/table layout, and thus queries are often on only a single table. Thanks Terry Mackintosh http://www.terrym.com sysadmin/owner Please! No MIME encoded or HTML mail, unless needed. Proudly powered by R H Linux 4.2, Apache 1.3, PHP 3, PostgreSQL 6.3 ------------------------------------------------------------------- Success Is A Choice ... book by Rick Patino, get it, read it! From wieck@sapserv.debis.de Wed Oct 14 13:55:53 1998 Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id NAA29290 for ; Wed, 14 Oct 1998 13:55:51 -0400 (EDT) Received: from dsh.de (firewall-user@neptun.sns-felb.debis.de [53.122.101.2]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id NAA14370 for ; Wed, 14 Oct 1998 13:51:19 -0400 (EDT) Received: by dsh.de; id TAA03418; Wed, 14 Oct 1998 19:50:18 +0200 (MET DST) Received: from dshmail.dsh.de(53.47.15.3) by neptun.dsh.de via smap (3.2) id xma003369; Wed, 14 Oct 98 19:49:51 +0200 Received: from mail1.hh1.dsh.de (mail1.hh1.dsh.de [53.47.9.5]) by dshmail.dsh.de (8.8.7/8.8.7) with ESMTP id TAA16746; Wed, 14 Oct 1998 19:47:14 +0200 (MET DST) Received: from mars.SAPserv.Hamburg.dsh.de (root@mail5.hh1.dsh.de [53.2.168.17]) by mail1.hh1.dsh.de (8.8.7/8.8.7) with SMTP id TAA23570; Wed, 14 Oct 1998 19:49:32 +0200 Received: from orion.SAPserv.Hamburg.dsh.de by mars.SAPserv.Hamburg.dsh.de with smtp for <> id m0zTSdF-000B5AC; Wed, 14 Oct 98 17:14 MET DST Received: by orion.SAPserv.Hamburg.dsh.de for lockhart@alumni.caltech.edu id m0zTVAt-000EBRC; Wed, 14 Oct 98 19:57 MET DST Message-Id: From: jwieck@debis.com (Jan Wieck) Subject: Re: [HACKERS] What about LIMIT in SELECT ? To: maillist@candle.pha.pa.us (Bruce Momjian) Date: Wed, 14 Oct 1998 19:57:27 +0200 (MET DST) Cc: lockhart@alumni.caltech.edu, jwieck@debis.com, eric@linux-hw.com, jeff@remapcorp.com, hackers@postgreSQL.org Reply-To: jwieck@debis.com (Jan Wieck) In-Reply-To: <199810141721.NAA28746@candle.pha.pa.us> from "Bruce Momjian" at Oct 14, 98 01:21:15 pm X-Mailer: ELM [version 2.4 PL25] Content-Type: text Status: RO > But, you are correct that certain cases where in index is already being > used on a query, you could just skip the sort IF you used the index to > get the rows from the base table. Especially in the case where SELECT ... WHERE key > 'val' ORDER BY key; creates a Sort->IndexScan plan. The index scan already jumps around on the disc to collect the sorts input and the sort finally returns exactly the same output (if the used index is only on key). And this is the case for large tables. The planner first decides to use an index scan due to the WHERE clause and later it notices the ORDER BY clause and creates a sort over the scan. I'm actually hacking around on it to see what happens if I suppress the sort node in some cases. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #======================================== jwieck@debis.com (Jan Wieck) # From owner-pgsql-hackers@hub.org Wed Oct 14 16:31:07 1998 Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id QAA01119 for ; Wed, 14 Oct 1998 16:31:05 -0400 (EDT) Received: from hub.org (majordom@hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id PAA22534 for ; Wed, 14 Oct 1998 15:29:50 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id PAA26335; Wed, 14 Oct 1998 15:05:26 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 14 Oct 1998 15:02:13 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id PAA26013 for pgsql-hackers-outgoing; Wed, 14 Oct 1998 15:02:11 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from candle.pha.pa.us (root@s5-03.ppp.op.net [209.152.195.67]) by hub.org (8.8.8/8.8.8) with ESMTP id PAA25996 for ; Wed, 14 Oct 1998 15:01:58 -0400 (EDT) (envelope-from maillist@candle.pha.pa.us) Received: (from maillist@localhost) by candle.pha.pa.us (8.9.0/8.9.0) id OAA29639; Wed, 14 Oct 1998 14:27:05 -0400 (EDT) From: Bruce Momjian Message-Id: <199810141827.OAA29639@candle.pha.pa.us> Subject: Re: [HACKERS] What about LIMIT in SELECT ? In-Reply-To: <199810141721.NAA28746@candle.pha.pa.us> from Bruce Momjian at "Oct 14, 1998 1:21:15 pm" To: maillist@candle.pha.pa.us (Bruce Momjian) Date: Wed, 14 Oct 1998 14:27:05 -0400 (EDT) Cc: lockhart@alumni.caltech.edu, jwieck@debis.com, eric@linux-hw.com, jeff@remapcorp.com, hackers@postgreSQL.org X-Mailer: ELM [version 2.4ME+ PL47 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: RO > Thomas is correct on this. Vadim has run some tests, and with our > optimized psort() code, the in-memory sort is often faster than using > the index to get the tuple, because you are jumping all over the drive. > I don't remember, but obviously there is a break-even point where > getting X rows using the index on a table of Y rows is faster , but > getting X+1 rows on a table of Y rows is faster getting all the rows > sequentailly, and doing the sort. > > You would have to pick only certain queries(no joins, index matches > ORDER BY), take the number of rows requested, and the number of rows > selected, and figure out if it is faster to use the index, or a > sequential scan and do the ORDER BY yourself. > > Add to this the OFFSET capability. I am not sure how you are going to > get into the index and start at the n-th entry, unless perhaps you just > sequential scan the index. > > In fact, many queries just get column already indexed, and we could just > pull the data right out of the index. > > I have added this to the TODO list: > > * Pull requested data directly from indexes, bypassing heap data > > I think this has to be post-6.4 work, but I think we need to work in > this direction. I am holding off any cnfify fixes for post-6.4, but a > 6.4.1 performance release certainly is possible. > > > But, you are correct that certain cases where in index is already being > used on a query, you could just skip the sort IF you used the index to > get the rows from the base table. I have had more time to think about this. Basically, for pre-sorted data, our psort code is very fast, because it does not need to sort anything. It just moves the rows in and out of the sort memory. Yes, it could be removed in some cases, and probably should be, but it is not going to produce great speedups. The more general case I will describe below. First, let's look at a normal query: SELECT * FROM tab ORDER BY col1 This is not going to use an index, and probably should not because it is faster for large tables to sort them in memory, rather than moving all over the disk. For small tables, if the entire table fits in the buffer cache, it may be faster to use the index, but on a small table the sort doesn't take very long either, and the buffer cache effectiveness is affected by other backends using it, so it may be better not to count on it for a speedup. However, if you only want the first 10 rows, that is a different story. We pull all the rows into the backend, sort them, then return 10 rows. The query, if we could do it, should be written as: SELECT * FROM tab WHERE col1 < some_unknown_value ORDER BY col1 In this case, the optimizer looks at the column statistics, and properly uses an index to pull only a small subset of the table. This is the type of behavior people want for queries returning only a few values. But, unfortunately, we don't know that mystery value. Now, everyone agrees we need an index matching the ORDER BY to make this query quick, but we don't know that mystery value, so currently we execute the whole query, and do a fetch. What I am now thinking is that maybe we need a way to walk around that index. Someone months ago asked how to do that, and we told him he couldn't, because this not a C-ISAM/dbm type database. However, if we could somehow pass into the query the index location we want to start at, and how many rows we need, that would solve our problem, and perhaps even allow joined queries to work, assuming the table in the ORDER BY is in an outer join loop. SELECT * FROM tab WHERE col1 < some_unknown_value ORDER BY col1 USING INDEX tab_idx(452) COUNT 100 where 452 is an 452th index entry, and COUNT is the number of index rows you want to process. The query may return more or less than 100 rows if there is a join and it joins to zero or more than one row in the joined table, but this seems like perhaps a good way to go at it. We need to do it this way because if a single index row returns 4 result rows, and only two of the four rows fit in the number of rows returnd as set by the user, it is hard to re-start the query at the proper point, because you would have to process the index rows a second time, and return just part of the result, and that is hard. If the index changes, or rows are added, the results are going to be unreliable, but that is probably going to be true of any state-less implementation we can devise. I think this may be fairly easy to implement. We could sequential scan the index to get to the 452th row. That is going to be quick. We can pass the 452 into the btree index code, so only a certain range of index tuples are returned, and the system believes it has processed the entire query, while we know it hasn't. Doesn't really work with hash, so we will not allow it for those indexes. To make it really easy, we could implement it as a 'SET' command, so we don't actually have it as part of the query, and have to pass it around through all the modules. You would do the proper 'SET' before running the query. Optimizer would look at 'SET' value to force index use. SET INDEX TO tab_idx START 452 COUNT 100 or SET INDEX TO tab_idx FROM 452 COUNT 451 There would have to be some way to signal that the end of the index had been reached, because returning zero rows is not enough of a guarantee in a joined SELECT. Comments? -- Bruce Momjian | http://www.op.net/~candle maillist@candle.pha.pa.us | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania 19026 From owner-pgsql-hackers@hub.org Wed Oct 14 17:31:23 1998 Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id RAA01591 for ; Wed, 14 Oct 1998 17:31:21 -0400 (EDT) Received: from hub.org (majordom@hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id RAA02744 for ; Wed, 14 Oct 1998 17:26:53 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id RAA05601; Wed, 14 Oct 1998 17:03:21 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 14 Oct 1998 16:59:54 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id QAA04964 for pgsql-hackers-outgoing; Wed, 14 Oct 1998 16:59:52 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from dsh.de (firewall-user@neptun.sns-felb.debis.de [53.122.101.2]) by hub.org (8.8.8/8.8.8) with ESMTP id QAA04943 for ; Wed, 14 Oct 1998 16:59:28 -0400 (EDT) (envelope-from wieck@sapserv.debis.de) Received: by dsh.de; id WAA28383; Wed, 14 Oct 1998 22:57:42 +0200 (MET DST) Received: from dshmail.dsh.de(53.47.15.3) by neptun.dsh.de via smap (3.2) id xma028354; Wed, 14 Oct 98 22:57:28 +0200 Received: from mail1.hh1.dsh.de (mail1.hh1.dsh.de [53.47.9.5]) by dshmail.dsh.de (8.8.7/8.8.7) with ESMTP id WAA20547; Wed, 14 Oct 1998 22:54:51 +0200 (MET DST) Received: from mars.SAPserv.Hamburg.dsh.de (root@mail5.hh1.dsh.de [53.2.168.17]) by mail1.hh1.dsh.de (8.8.7/8.8.7) with SMTP id WAA24383; Wed, 14 Oct 1998 22:57:09 +0200 Received: from orion.SAPserv.Hamburg.dsh.de by mars.SAPserv.Hamburg.dsh.de with smtp for <> id m0zTVYr-000B5AC; Wed, 14 Oct 98 20:22 MET DST Received: by orion.SAPserv.Hamburg.dsh.de for lockhart@alumni.caltech.edu id m0zTY6V-000EBRC; Wed, 14 Oct 98 23:05 MET DST Message-Id: From: jwieck@debis.com (Jan Wieck) Subject: Re: [HACKERS] What about LIMIT in SELECT ? To: maillist@candle.pha.pa.us (Bruce Momjian) Date: Wed, 14 Oct 1998 23:05:07 +0200 (MET DST) Cc: maillist@candle.pha.pa.us, lockhart@alumni.caltech.edu, jwieck@debis.com, eric@linux-hw.com, jeff@remapcorp.com, hackers@postgreSQL.org Reply-To: jwieck@debis.com (Jan Wieck) In-Reply-To: <199810141827.OAA29639@candle.pha.pa.us> from "Bruce Momjian" at Oct 14, 98 02:27:05 pm X-Mailer: ELM [version 2.4 PL25] Content-Type: text Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: ROr > I have had more time to think about this. Basically, for pre-sorted > data, our psort code is very fast, because it does not need to sort > anything. It just moves the rows in and out of the sort memory. Yes, > it could be removed in some cases, and probably should be, but it is not > going to produce great speedups. And I got the time to hack around about this. I hacked in a little check into the planner, that compares the sortClause against the key field list of an index scan and just suppresses the sort node if it exactly matchs and all sort operators are "<". I tested with a 10k row table where key is a text field. The base query is a SELECT ... WHERE key > 'val' ORDER BY key; The used 'val' is always a key that is close to the first of all keys in the table ('' on the first query and the last selected value on subsequent ones). Scenario 1 (S1) uses exactly the above query but processes only the first 20 rows from the result buffer. Thus the frontend receives nearly the whole table. Scenario 2 (S2) uses a cursor and FETCH 20. But closes the cursor and creates a new one for the next selection (only with another 'val') as it would occur in a web application. If there is no index on key, the backend will allways do a Sort->SeqScan and due to the 'val' close to the lowest existing key nearly all tuples get scanned and put into the sort. S1 here runs about 10 seconds and S2 about 6 seconds. The speedup in S2 results from the reduced overhead of sending not wanted tuples into the frontend. Now with a btree index on key and an unpatched backend. Produced plan is always a Sort->IndexScan. S1 needs 16 seconds and S2 needs 12 seconds. Again nearly all data is put into the sort but this time over the index scan and that is slower. Last with the btree index on key and the patched backend. This time the plan is a plain IndexScan because the ORDER BY clause exactly matches the sort order of the choosen index. S1 needs 13 seconds and S2 less than 0.2! This dramatic speedup comes from the fact, that this time the index scan is the toplevel executor node and the executor run is stopped after 20 tuples have been selected. Analysis of the above timings: If there is an ORDER BY clause, using an index scan is the clever way if the indexqual dramatically reduces the the amount of data selected and sorted. I think this is the normal case (who really selects nearly all rows from a 5M row table?). So choosing the index path is correct. This will hurt if someone really selects most of the rows and the index scan jumps over the disc. But here the programmer should use an unqualified query to perform a seqscan and do the qualification in the frontend application. The speedup for the cursor/fetch scenario is so impressive that I'll create a post 6.4 patch. I don't want it in 6.4 because there is absolutely no query in the whole regression test, where it suppresses the sort node. So we have absolutely no check that it doesn't break anything. For a web application, that can use a unique key to select the next amount of rows, it will be a big win. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #======================================== jwieck@debis.com (Jan Wieck) # From owner-pgsql-hackers@hub.org Thu Oct 15 00:01:10 1998 Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id AAA06040 for ; Thu, 15 Oct 1998 00:01:04 -0400 (EDT) Received: from hub.org (root@hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id XAA29020 for ; Wed, 14 Oct 1998 23:57:58 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id WAA02215; Wed, 14 Oct 1998 22:39:07 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 14 Oct 1998 22:35:19 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id WAA02061 for pgsql-hackers-outgoing; Wed, 14 Oct 1998 22:35:16 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from sraigw.sra.co.jp (sraigw.sra.co.jp [202.32.10.2]) by hub.org (8.8.8/8.8.8) with ESMTP id WAA01851 for ; Wed, 14 Oct 1998 22:35:01 -0400 (EDT) (envelope-from t-ishii@srapc451.sra.co.jp) Received: from srapc451.sra.co.jp (srapc451 [133.137.44.37]) by sraigw.sra.co.jp (8.8.7/3.6Wbeta7-sraigw) with ESMTP id LAA17765; Thu, 15 Oct 1998 11:34:39 +0900 (JST) Received: from srapc451.sra.co.jp (localhost [127.0.0.1]) by srapc451.sra.co.jp (8.8.8/3.5Wpl7) with ESMTP id LAA08260; Thu, 15 Oct 1998 11:34:54 +0900 (JST) Message-Id: <199810150234.LAA08260@srapc451.sra.co.jp> To: jwieck@debis.com (Jan Wieck) cc: oleg@sai.msu.su, hackers@postgreSQL.org, t-ishii@sra.co.jp Subject: Re: [HACKERS] What about LIMIT in SELECT ? From: Tatsuo Ishii Reply-To: t-ishii@sra.co.jp In-reply-to: Your message of Wed, 14 Oct 1998 16:24:56 +0200. Date: Thu, 15 Oct 1998 11:34:54 +0900 Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: RO >> postgres just because of lacking LIMIT. Tatsuo posted a patch >> for set query_limit to 'num', I just tested it and seems it >> works fine. Now, we need only possibility to specify offset, >> say >> set query_limit to 'offset,num' >> ( Tatsuo, How difficult to do this ?) >> and LIMIT problem will ne gone. > > Think you haven't read my posting completely. Even with the > executor limit, the complete scan into the sort is done by > the backend. You need to specify ORDER BY to get the same > list again (without the offset doesn't make sense). But > currently, ORDER BY forces a sort node into the query plan. I think we have understanded your point. set query_limit is just a easy alternative of using cursor and fetch. > I haven't looked at Tatsuo's patch very well. But if it > limits the amount of data going into the sort (on ORDER BY), > it will break it! The requested ordering could be different > from what the choosen index might return. The used index is > choosen by the planner upon the qualifications given, not the > ordering wanted. I think it limits the final result. When query_limit is set, the arg "numberTuples" of ExecutePlan() is set to it instead of 0 (this means no limit). Talking about "offset," it shouldn't be very difficult. I guess all we have to do is adding a new arg "offset" to ExecutePlan() then making obvious modifications. (and of course we have to modify set query_limit syntax but it's trivial) However, before going ahead, I would like to ask other hackers about this direction. This might be convenient for some users, but still the essential performance issue would remain. In another word, this is a short-term solution not a intrinsic one, IMHO. -- Tatsuo Ishii t-ishii@sra.co.jp From owner-pgsql-hackers@hub.org Thu Oct 15 10:01:17 1998 Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id KAA13960 for ; Thu, 15 Oct 1998 10:01:15 -0400 (EDT) Received: from hub.org (majordom@hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id JAA20266 for ; Thu, 15 Oct 1998 09:12:21 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id IAA26142; Thu, 15 Oct 1998 08:19:49 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Thu, 15 Oct 1998 08:13:48 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id IAA25747 for pgsql-hackers-outgoing; Thu, 15 Oct 1998 08:13:46 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from dsh.de (firewall-user@neptun.sns-felb.debis.de [53.122.101.2]) by hub.org (8.8.8/8.8.8) with ESMTP id IAA25733 for ; Thu, 15 Oct 1998 08:13:40 -0400 (EDT) (envelope-from wieck@sapserv.debis.de) Received: by dsh.de; id OAA18677; Thu, 15 Oct 1998 14:16:12 +0200 (MET DST) Received: from dshmail.dsh.de(53.47.15.3) by neptun.dsh.de via smap (3.2) id xma018279; Thu, 15 Oct 98 14:15:39 +0200 Received: from mail1.hh1.dsh.de (mail1.hh1.dsh.de [53.47.9.5]) by dshmail.dsh.de (8.8.7/8.8.7) with ESMTP id OAA01227; Thu, 15 Oct 1998 14:13:09 +0200 (MET DST) Received: from mars.SAPserv.Hamburg.dsh.de (root@mail5.hh1.dsh.de [53.2.168.17]) by mail1.hh1.dsh.de (8.8.7/8.8.7) with SMTP id OAA28938; Thu, 15 Oct 1998 14:15:27 +0200 Received: from orion.SAPserv.Hamburg.dsh.de by mars.SAPserv.Hamburg.dsh.de with smtp for <> id m0zTjtm-000B5AC; Thu, 15 Oct 98 11:40 MET DST Received: by orion.SAPserv.Hamburg.dsh.de for jwieck@debis.com id m0zTmRT-000EBRC; Thu, 15 Oct 98 14:23 MET DST Message-Id: From: jwieck@debis.com (Jan Wieck) Subject: Re: [HACKERS] What about LIMIT in SELECT ? To: t-ishii@sra.co.jp Date: Thu, 15 Oct 1998 14:23:43 +0200 (MET DST) Cc: jwieck@debis.com, oleg@sai.msu.su, hackers@postgreSQL.org Reply-To: jwieck@debis.com (Jan Wieck) In-Reply-To: <199810150234.LAA08260@srapc451.sra.co.jp> from "Tatsuo Ishii" at Oct 15, 98 11:34:54 am X-Mailer: ELM [version 2.4 PL25] Content-Type: text Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: RO Tatsuo Ishii wrote: > I think we have understanded your point. set query_limit is just a > easy alternative of using cursor and fetch. > > > I haven't looked at Tatsuo's patch very well. But if it > > limits the amount of data going into the sort (on ORDER BY), > > it will break it! The requested ordering could be different > > from what the choosen index might return. The used index is > > choosen by the planner upon the qualifications given, not the > > ordering wanted. > > I think it limits the final result. When query_limit is set, > the arg "numberTuples" of ExecutePlan() is set to it instead of 0 > (this means no limit). > > Talking about "offset," it shouldn't be very difficult. I guess all we > have to do is adding a new arg "offset" to ExecutePlan() then making > obvious modifications. (and of course we have to modify set > query_limit syntax but it's trivial) The offset could become FETCH n IN cursor [OFFSET n]; and SELECT ... [LIMIT offset,count]; The FETCH command already calls ExecutorRun() with the given count (the tuple limit). Telling it the offset too is really simple. And ExecutorRun() could check if the toplevel executor node is an index scan. Skipping tuples during the index scan requires, that all qualifications are in the indexqual, thus any tuple returned by it will become a final result row (as it would be in the simple 1-table-queries we discussed). If that isn't the case, the executor must fallback to skip the final result tuples and that is after an eventually processed sort/merge of the complete result set. That would only reduce communication to the client and memory required there to buffer the result set (not a bad thing either). ProcessQueryDesc() in tcop/pquery.c also calls ExecutorRun() but with a constant 0 tuple count. Having offset and count in the parsetree would make it without any state variables or SET command. And it's the only clean way to restrict LIMIT to SELECT queries. Any thrown in LIMIT to ExecutorRun() from another place could badly hurt the rewrite system. Remember that non-instead actions on insert/update/delete are processed before the original query! And what about SQL functions that get processed during the evaluation of another query (view using an SQL function for count(*))? A little better would it be to make the LIMIT values able to be parameter nodes. C or PL functions use the prepared plan feature of the SPI manager for performance reasons. Especially the offset value might there need to be a parameter that the executor has to pick out first. If we change the count argument of ExecutorRun to a List *limit, this one could be NIL (to mean the old 0 count 0 offset behaviour) or a list of two elements that both can be either a Const or a Param of type int4. Easy for the executor to evaluate. The only places where ExecutorRun() is called are tcop/pquery.c (queries from frontend), commands/command.c (FETCH command), executor/functions.c (SQL functions) and executor/spi.c (SPI manager). So it is easy to change the call interface too. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #======================================== jwieck@debis.com (Jan Wieck) # From owner-pgsql-hackers@hub.org Thu Oct 15 14:32:34 1998 Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id OAA19803 for ; Thu, 15 Oct 1998 14:32:31 -0400 (EDT) Received: from hub.org (root@hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id NAA10847 for ; Thu, 15 Oct 1998 13:38:16 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id MAA22772; Thu, 15 Oct 1998 12:07:20 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Thu, 15 Oct 1998 12:02:33 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id MAA22026 for pgsql-hackers-outgoing; Thu, 15 Oct 1998 12:02:31 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from ra.sai.msu.su (ra.sai.msu.su [158.250.29.2]) by hub.org (8.8.8/8.8.8) with ESMTP id MAA22007 for ; Thu, 15 Oct 1998 12:02:16 -0400 (EDT) (envelope-from oleg@sai.msu.su) Received: from ra (ra [158.250.29.2]) by ra.sai.msu.su (8.9.1/8.9.1) with SMTP id TAA21024; Thu, 15 Oct 1998 19:01:23 +0300 (MSK) Date: Thu, 15 Oct 1998 20:01:23 +0400 (MSD) From: Oleg Bartunov X-Sender: megera@ra To: Jan Wieck cc: t-ishii@sra.co.jp, hackers@postgreSQL.org Subject: Re: [HACKERS] What about LIMIT in SELECT ? In-Reply-To: Message-ID: Organization: Sternberg Astronomical Institute (Moscow University) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: ROr This is a little bit off-topic, I did some timings with latest cvs on my real database ( all output redirected to /dev/null ), table contains 8798 records, 31 columns, order key have indices. 1.select count(*) from work_flats; 0.02user 0.00system 0:00.18elapsed 10%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (131major+21minor)pagefaults 0swaps 2.select * from work_flats order by rooms, metro_id; 2.35user 0.25system 0:10.11elapsed 25%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (131major+2799minor)pagefaults 0swaps 3.set query_limit to '150'; SET VARIABLE select * from work_flats order by rooms, metro_id; 0.06user 0.00system 0:02.75elapsed 2%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (131major+67minor)pagefaults 0swaps 4.begin; declare tt cursor for select * from work_flats order by rooms, metro_id; fetch 150 in tt; end; 0.05user 0.01system 0:02.76elapsed 2%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (131major+67minor)pagefaults 0swaps As you can see timings for query_limit and cursor are very similar, I didn't expected this. So, in principle, enhanced version of fetch (with offset) would cover all we need from LIMIT, but query_limit would be still useful, for example to restrict loadness of server. Will all enhancements you discussed go to the 6.4 ? I'm really interested in testing this stuff because I begin new project and everything we discussed here are badly needed. Regards, Oleg On Thu, 15 Oct 1998, Jan Wieck wrote: > Date: Thu, 15 Oct 1998 14:23:43 +0200 (MET DST) > From: Jan Wieck > To: t-ishii@sra.co.jp > Cc: jwieck@debis.com, oleg@sai.msu.su, hackers@postgreSQL.org > Subject: Re: [HACKERS] What about LIMIT in SELECT ? > > Tatsuo Ishii wrote: > > > I think we have understanded your point. set query_limit is just a > > easy alternative of using cursor and fetch. > > > > > I haven't looked at Tatsuo's patch very well. But if it > > > limits the amount of data going into the sort (on ORDER BY), > > > it will break it! The requested ordering could be different > > > from what the choosen index might return. The used index is > > > choosen by the planner upon the qualifications given, not the > > > ordering wanted. > > > > I think it limits the final result. When query_limit is set, > > the arg "numberTuples" of ExecutePlan() is set to it instead of 0 > > (this means no limit). > > > > Talking about "offset," it shouldn't be very difficult. I guess all we > > have to do is adding a new arg "offset" to ExecutePlan() then making > > obvious modifications. (and of course we have to modify set > > query_limit syntax but it's trivial) > > The offset could become > > FETCH n IN cursor [OFFSET n]; > > and > > SELECT ... [LIMIT offset,count]; > > The FETCH command already calls ExecutorRun() with the given > count (the tuple limit). Telling it the offset too is really > simple. And ExecutorRun() could check if the toplevel > executor node is an index scan. Skipping tuples during the > index scan requires, that all qualifications are in the > indexqual, thus any tuple returned by it will become a final > result row (as it would be in the simple 1-table-queries we > discussed). If that isn't the case, the executor must > fallback to skip the final result tuples and that is after an > eventually processed sort/merge of the complete result set. > That would only reduce communication to the client and memory > required there to buffer the result set (not a bad thing > either). > > ProcessQueryDesc() in tcop/pquery.c also calls ExecutorRun() > but with a constant 0 tuple count. Having offset and count in > the parsetree would make it without any state variables or > SET command. And it's the only clean way to restrict LIMIT to > SELECT queries. Any thrown in LIMIT to ExecutorRun() from > another place could badly hurt the rewrite system. Remember > that non-instead actions on insert/update/delete are > processed before the original query! And what about SQL > functions that get processed during the evaluation of another > query (view using an SQL function for count(*))? > > A little better would it be to make the LIMIT values able to > be parameter nodes. C or PL functions use the prepared plan > feature of the SPI manager for performance reasons. > Especially the offset value might there need to be a > parameter that the executor has to pick out first. If we > change the count argument of ExecutorRun to a List *limit, > this one could be NIL (to mean the old 0 count 0 offset > behaviour) or a list of two elements that both can be either > a Const or a Param of type int4. Easy for the executor to > evaluate. > > The only places where ExecutorRun() is called are > tcop/pquery.c (queries from frontend), commands/command.c > (FETCH command), executor/functions.c (SQL functions) and > executor/spi.c (SPI manager). So it is easy to change the > call interface too. > > > Jan > > -- > > #======================================================================# > # It's easier to get forgiveness for being wrong than for being right. # > # Let's break this rule - forgive me. # > #======================================== jwieck@debis.com (Jan Wieck) # > > _____________________________________________________________ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83 From owner-pgsql-hackers@hub.org Thu Oct 15 13:22:48 1998 Received: from hub.org (majordom@hub.org [209.47.148.200]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id NAA18540 for ; Thu, 15 Oct 1998 13:22:46 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id MAA01819; Thu, 15 Oct 1998 12:56:25 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Thu, 15 Oct 1998 12:51:43 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id MAA01305 for pgsql-hackers-outgoing; Thu, 15 Oct 1998 12:51:40 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from dsh.de (firewall-user@neptun.sns-felb.debis.de [53.122.101.2]) by hub.org (8.8.8/8.8.8) with ESMTP id MAA01283 for ; Thu, 15 Oct 1998 12:51:28 -0400 (EDT) (envelope-from wieck@sapserv.debis.de) Received: by dsh.de; id SAA21874; Thu, 15 Oct 1998 18:54:00 +0200 (MET DST) Received: from dshmail.dsh.de(53.47.15.3) by neptun.dsh.de via smap (3.2) id xma021705; Thu, 15 Oct 98 18:53:31 +0200 Received: from mail1.hh1.dsh.de (mail1.hh1.dsh.de [53.47.9.5]) by dshmail.dsh.de (8.8.7/8.8.7) with ESMTP id SAA25226; Thu, 15 Oct 1998 18:50:57 +0200 (MET DST) Received: from mars.SAPserv.Hamburg.dsh.de (root@mail5.hh1.dsh.de [53.2.168.17]) by mail1.hh1.dsh.de (8.8.7/8.8.7) with SMTP id SAA30639; Thu, 15 Oct 1998 18:53:14 +0200 Received: from orion.SAPserv.Hamburg.dsh.de by mars.SAPserv.Hamburg.dsh.de with smtp for <> id m0zToEf-000B5AC; Thu, 15 Oct 98 16:18 MET DST Received: by orion.SAPserv.Hamburg.dsh.de for jwieck@debis.com id m0zTqmM-000EBRC; Thu, 15 Oct 98 19:01 MET DST Message-Id: From: jwieck@debis.com (Jan Wieck) Subject: Re: [HACKERS] What about LIMIT in SELECT ? To: hannu@trust.ee (Hannu Krosing) Date: Thu, 15 Oct 1998 19:01:33 +0200 (MET DST) Cc: jwieck@debis.com, pgsql-hackers@postgreSQL.org Reply-To: jwieck@debis.com (Jan Wieck) In-Reply-To: <36261DF7.D20368A0@trust.ee> from "Hannu Krosing" at Oct 15, 98 07:08:23 pm X-Mailer: ELM [version 2.4 PL25] Content-Type: text Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: RO Hannu Krosing wrote: > Jan Wieck wrote: > > The speedup for the cursor/fetch scenario is so impressive > > that I'll create a post 6.4 patch. I don't want it in 6.4 > > because there is absolutely no query in the whole regression > > test, where it suppresses the sort node. > > Good, then it works as expected ;) > > More seriously, it is not within powers of current regression test > framework to test speed improvements (only the case where > performance-wise bad implementation will actually crash the backend, > as in the cnfify problem, but AFAIK we dont test for those now) > > > So we have absolutely no check that it doesn't break anything. > > If it did pass the regression, then IMHO it did not break anything. Thats the point. The check if the sort node is required returns TRUE for ALL queries of the regression. So the behaviour when it returns FALSE is absolutely not tested. > > I would vote for putting it in (maybe with a > 'set fix_optimiser_stupidity on' safeguard to enable it). I see no > reason to postpone it to 6.4.1 and force almost everybody to first > patch their copy and then upgrade very soon. > > I would even go far enough to call it a bugfix, as it does not really > introduce any new functionality only fixes some existing functionality > so that much bigger databases can be actually used. I can't call it a bugfix because it is only a performance win in some situations. And I feel the risk is too high to put untested code into the backend at BETA2 time. The max we should do is to take this one and the LIMIT thing (maybe implemented as I suggested lately), and put out a Web- Performance-Release at the same time we release 6.4. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #======================================== jwieck@debis.com (Jan Wieck) # From Inoue@tpf.co.jp Thu Oct 15 20:31:01 1998 Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id UAA26050 for ; Thu, 15 Oct 1998 20:31:00 -0400 (EDT) Received: from sd.tpf.co.jp (sd.tpf.co.jp [210.161.239.34]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id UAA12888 for ; Thu, 15 Oct 1998 20:10:03 -0400 (EDT) Received: from cadzone ([126.0.1.40]) by sd.tpf.co.jp (2.0 Build 2131 (Berkeley 8.8.4)/8.8.4) with SMTP id JAA02574; Fri, 16 Oct 1998 09:00:34 +0900 From: "Hiroshi Inoue" To: "Jan Wieck" , "Bruce Momjian" Subject: RE: [HACKERS] What about LIMIT in SELECT ? Date: Fri, 16 Oct 1998 09:12:55 +0900 Message-ID: <000201bdf899$b953bf00$2801007e@cadzone.tpf.co.jp> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 In-Reply-To: <199810150552.BAA07576@candle.pha.pa.us> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.2106.4 Status: ROr > -----Original Message----- > From: owner-pgsql-hackers@postgreSQL.org > [mailto:owner-pgsql-hackers@postgreSQL.org]On Behalf Of Bruce Momjian > Sent: Thursday, October 15, 1998 2:52 PM > To: jwieck@debis.com > Cc: lockhart@alumni.caltech.edu; jwieck@debis.com; eric@linux-hw.com; > jeff@remapcorp.com; hackers@postgreSQL.org > Subject: Re: [HACKERS] What about LIMIT in SELECT ? > > > > > I have had more time to think about this. Basically, for pre-sorted > > > data, our psort code is very fast, because it does not need to sort > > > anything. It just moves the rows in and out of the sort memory. Yes, > > > it could be removed in some cases, and probably should be, > but it is not > > > going to produce great speedups. > > > > And I got the time to hack around about this. > > > > I hacked in a little check into the planner, that compares > > the sortClause against the key field list of an index scan > > and just suppresses the sort node if it exactly matchs and > > all sort operators are "<". > > > > I tested with a 10k row table where key is a text field. The > > base query is a > > > > SELECT ... WHERE key > 'val' ORDER BY key; > > > > The used 'val' is always a key that is close to the first of > > all keys in the table ('' on the first query and the last > > selected value on subsequent ones). > > This is good stuff. I want to think about it for a day. Sounds very > promising. > Did you see my contribution about this subject ? I have already implemented above cases and used on trial for three months or more. It is good to be formally supported by PostgreSQL community. And please remember that there are descending order cases. (Moreover there are compound cases such as SELECT * from ... order by key1 desc,key2 asc; I didn't implement such cases.) Thanks. Hiroshi Inoue Inoue@tpf.co.jp From owner-pgsql-hackers@hub.org Fri Oct 16 04:01:07 1998 Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id EAA02029 for ; Fri, 16 Oct 1998 04:01:04 -0400 (EDT) Received: from hub.org (majordom@hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id DAA05509 for ; Fri, 16 Oct 1998 03:43:53 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id CAA11278; Fri, 16 Oct 1998 02:00:01 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Fri, 16 Oct 1998 01:57:25 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id BAA11129 for pgsql-hackers-outgoing; Fri, 16 Oct 1998 01:57:21 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from candle.pha.pa.us (root@s5-03.ppp.op.net [209.152.195.67]) by hub.org (8.8.8/8.8.8) with ESMTP id BAA11116 for ; Fri, 16 Oct 1998 01:57:00 -0400 (EDT) (envelope-from maillist@candle.pha.pa.us) Received: (from maillist@localhost) by candle.pha.pa.us (8.9.0/8.9.0) id BAA29942; Fri, 16 Oct 1998 01:34:33 -0400 (EDT) From: Bruce Momjian Message-Id: <199810160534.BAA29942@candle.pha.pa.us> Subject: Re: [HACKERS] What about LIMIT in SELECT ? In-Reply-To: from Jan Wieck at "Oct 14, 1998 11: 5: 7 pm" To: jwieck@debis.com Date: Fri, 16 Oct 1998 01:34:33 -0400 (EDT) Cc: lockhart@alumni.caltech.edu, jwieck@debis.com, eric@linux-hw.com, jeff@remapcorp.com, hackers@postgreSQL.org X-Mailer: ELM [version 2.4ME+ PL47 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: RO OK, I have had my day of thinking, and will address this specific posting first, because it is the most fundamental concerning the future direction of the optimization. > > And I got the time to hack around about this. > > I hacked in a little check into the planner, that compares > the sortClause against the key field list of an index scan > and just suppresses the sort node if it exactly matchs and > all sort operators are "<". > > I tested with a 10k row table where key is a text field. The > base query is a > > SELECT ... WHERE key > 'val' ORDER BY key; > > The used 'val' is always a key that is close to the first of > all keys in the table ('' on the first query and the last > selected value on subsequent ones). > > Scenario 1 (S1) uses exactly the above query but processes > only the first 20 rows from the result buffer. Thus the > frontend receives nearly the whole table. OK. > > Scenario 2 (S2) uses a cursor and FETCH 20. But closes the > cursor and creates a new one for the next selection (only > with another 'val') as it would occur in a web application. > > If there is no index on key, the backend will allways do a > Sort->SeqScan and due to the 'val' close to the lowest > existing key nearly all tuples get scanned and put into the > sort. S1 here runs about 10 seconds and S2 about 6 seconds. > The speedup in S2 results from the reduced overhead of > sending not wanted tuples into the frontend. Makes sense. All rows are processed, but not sent to client. > > Now with a btree index on key and an unpatched backend. > Produced plan is always a Sort->IndexScan. S1 needs 16 > seconds and S2 needs 12 seconds. Again nearly all data is put > into the sort but this time over the index scan and that is > slower. VACUUM ANALYZE could affect this. Because it had no stats, it thought index use would be faster, but in fact because 'val' was near the lowest value, it as selecting 90% of the table, and would have been better with a sequential scan. pg_statistics's low/hi values for a column could have told that to the optimizer. I know the good part of the posting is coming. > Last with the btree index on key and the patched backend. > This time the plan is a plain IndexScan because the ORDER BY > clause exactly matches the sort order of the chosen index. > S1 needs 13 seconds and S2 less than 0.2! This dramatic > speedup comes from the fact, that this time the index scan is > the toplevel executor node and the executor run is stopped > after 20 tuples have been selected. OK, seems like in the S1 case, the use of the psort/ORDER BY code on top of the index was taking and extra 3 seconds, which is 23%. That is a lot more than I thought for the psort code, and shows we could gain a lot by removing unneeded sorts from queries that are already using matching indexes. Just for clarity, added to TODO. I think everyone is clear on this one, and its magnitude is a surprise to me: * Prevent psort() usage when query already using index matching ORDER BY > Analysis of the above timings: > > If there is an ORDER BY clause, using an index scan is the > clever way if the indexqual dramatically reduces the the > amount of data selected and sorted. I think this is the > normal case (who really selects nearly all rows from a 5M row > table?). So choosing the index path is correct. This will > hurt if someone really selects most of the rows and the index > scan jumps over the disc. But here the programmer should use > an unqualified query to perform a seqscan and do the > qualification in the frontend application. Fortunately, the optimizer already does the index selection for us, and guesses pretty well if the index or sequential scan is better. Once we implement the above removal of psort(), we will have to change the timings because now you have to compare index scan against sequential scan AND psort(), because in the index scan situation, you don't need the psort(), assuming the ORDER BY matches the index exactly. > The speedup for the cursor/fetch scenario is so impressive > that I'll create a post 6.4 patch. I don't want it in 6.4 > because there is absolutely no query in the whole regression > test, where it suppresses the sort node. So we have > absolutely no check that it doesn't break anything. > > For a web application, that can use a unique key to select > the next amount of rows, it will be a big win. OK, I think the reason the regression test did not show your code being used is important. First, most of the tables are small in the regression test, so sequential scans are faster. Second, most queries using indexes are either joins, which do the entire table, or equality tests, like col = 3, where there is no matching ORDER BY because all the col values are 3. Again, your code can't help with these. The only regression-type code that would use it would be a 'col > 3' qualification with a col ORDER BY, and there aren't many of those. However, if we think of the actual application you are addressing, it is a major win. If we are going after only one row of the index, it is fast. If we are going after the entire table, it is faster to sequential scan and psort(). You big win is with the partial queries, where you end up doing a full sequential scan or index scan, then and ORDER BY, while you really only need a few rows from the query, and if you deal directly with the index, you can prevent many rows from being processed. It is the ability to skip processing those extra rows that makes it a big win, not so much the removal of the ORDER BY, though that helps too. Your solution really is tailored for this 'partial' query application, and I think it is a big need for certain applications that can't use cursors, like web apps. Most other apps have long-time connections to the database, and are better off with cursors. I did profiling to improve startup time, because the database requirements of web apps are different from normal db apps, and we have to adjust to that. So, to reiterate, full queries are not benefited as much from the new code, because sequential scan/psort is faster, or because the index only retrieves a small number of rows because the qualification of values is very specific. Those open-ended, give me the rows from 100 to 199 really need your modifications. OK, we have QUERY_LIMIT, and that allows us to throw any query at the system, and it will return that many of the first rows for the ORDER BY. No fancy stuff required. If we can get a matching index, we may be able to remove the requirement of scanning all the row (with Jan's patch), and that is a big win. If not, we at least prevent the rows from being returned to the client. However, there is the OFFSET issue. This is really a case where the user wants to _restart_ the query where they left off. That is a different problem. All of a sudden, we need to evaluate more of the query, and return a segment from the middle of the result set. I think we need to decide how to handle such a restart. Do we re-evaluate the entire query, skipping all the rows up to OFFSET, and return the number of rows they requested after OFFSET. I would think we don't want to do that, do we. It would be much easier to code. If it is a single table, skipping forward has to be done anyway, because we can't just _jump_ to the 100th entry in the index, unless we pass some _tid_ to the user, and expect them to pass that back to start the query. I don't think we went to do that. It is ugly, and the row may have moved since we started. So, for a single table, adding a QUERY_OFFSET would do exactly what we need, with Jan's patches. For a joined query, I think you will have to do the entire _join_ before returning anything. You can't just process all the joins up to the OFFSET location, and you can't just jump to the 100th index location, because you don't know that the 100th index location produced the 100th result just returned to the user. You have to process the whole query, and because of the join and not knowing which data row from each table is going to make which entry in the final result. If you are really craft, and the ORDER BY table is in the outer part of the join loop, you could start processing the table that is part of the outer loop in _index_ order, because you know that the rows processed in index order are going to produce the output in result order. You then could process and throw away the results up to offset, and generate the needed rows and stop. The other way of doing it is to specify a query limit based on specific index entries, so you say I want the query returned by the first 20 index entries matching the ORDER BY, or entries 100-199, and the query is limited to using only those entries in the index. In that case, though, in joins, you could return more or less rows in the result depending on the other tables, and that may be unacceptable. However, for this case, the advantage is that you don't need to process the rows from 1 to 99 because you have been told the user only wants rows from certain index slots. If the user requests rows 50000-50100, this would be much faster because you don't have to process the 50000 rows before returning any data. However, I question how often people grab stuff from the center of large data sets. Seems the QUERY_OFFSET idea may be easier for users. I will be commenting on the rest of the optimization postings tomorrow. -- Bruce Momjian | http://www.op.net/~candle maillist@candle.pha.pa.us | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania 19026 From Inoue@tpf.co.jp Fri Oct 16 03:31:02 1998 Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id DAA01767 for ; Fri, 16 Oct 1998 03:31:00 -0400 (EDT) Received: from sd.tpf.co.jp (sd.tpf.co.jp [210.161.239.34]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id DAA04551 for ; Fri, 16 Oct 1998 03:13:40 -0400 (EDT) Received: from cadzone ([126.0.1.40]) by sd.tpf.co.jp (2.0 Build 2131 (Berkeley 8.8.4)/8.8.4) with SMTP id QAA02680; Fri, 16 Oct 1998 16:04:09 +0900 From: "Hiroshi Inoue" To: "Bruce Momjian" Cc: Subject: RE: [HACKERS] What about LIMIT in SELECT ? Date: Fri, 16 Oct 1998 16:16:29 +0900 Message-ID: <000001bdf8d4$e4cdf520$2801007e@cadzone.tpf.co.jp> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.2106.4 In-Reply-To: <199810160621.CAA01030@candle.pha.pa.us> Status: RO Where's my contibution to hackers@potsgreSQL.org ? I will resend it. > -----Original Message----- > From: Bruce Momjian [mailto:maillist@candle.pha.pa.us] > Sent: Friday, October 16, 1998 3:22 PM > To: Hiroshi Inoue > Cc: jwieck@debis.com > Subject: Re: [HACKERS] What about LIMIT in SELECT ? > > > [Charset iso-8859-1 unsupported, filtering to ASCII...] > > > > The used 'val' is always a key that is close to the first of > > > > all keys in the table ('' on the first query and the last > > > > selected value on subsequent ones). > > > > > > This is good stuff. I want to think about it for a day. Sounds very > > > promising. > > > > > > > Did you see my contribution about this subject ? > > I am sorry. I have not seen it, and I am confused how I could have > missed it. > > > I have already implemented above cases and used on trial for three > > months or more. > > It is good to be formally supported by PostgreSQL community. > > > > And please remember that there are descending order cases. > > (Moreover there are compound cases such as > > SELECT * from ... order by key1 desc,key2 asc; > > I didn't implement such cases.) > > Where is the discussion of this? I am confused. You have been using > code for three months that does this? > Hi all. I didn't follow all the posts about this thread. So this post may be out of center. I think current PostgreSQL lacks the concern to the response to get first rows quickly. For example,queries with ORDER BY clause necessarily include sort steps and process all target rows to get first rows only. So I modified my code for ORDER BY cases and use on trial. I don't understand PostgreSQL sources,so my code is not complete. I modified my code for the following 2 cases. 1.In many cases the following query uses index scan. SELECT * from ... where key > ...; (where (key) is an index) If so,we can omit sort steps from the access plan for the following query. SELECT * from ... where key > ... order by key; Currently cursors without sort steps may be sensitive diffrent from cursors with sort steps. But no one mind it. 2.In many cases the following query uses index scan same as case 1. SELECT * from ... where key < ...;(where (key) is an index) If so and if we scan the index backward,we can omit sort steps from the access plan for the following query. SELECT * from ... where key < ... order by key desc; To achive this(backward scan),I used hidden(provided for the future ?)code that is never executed and is not necessarily correct. In the following cases I didn't modify my code to use index scan, because I couldn't formulate how to tell PostgreSQL optimizer whether the response to get first rows is needed or the throughput to process sufficiently many target rows is needed. 3.The access plan made by current PostgreSQL optimizer for a query with ORDER BY clause doesn't include index scan. I thought the use of Tatsuo's QUERY_LIMIT to decide that the responce is needed. It is sufficient but not necessary ? In Oracle the hints FIRST_ROWS,ALL_ROWS are used. Thanks. Hiroshi Inoue Inoue@tpf.co.jp From wieck@sapserv.debis.de Fri Oct 16 05:01:03 1998 Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id FAA02500 for ; Fri, 16 Oct 1998 05:01:02 -0400 (EDT) Received: from dsh.de (firewall-user@neptun.sns-felb.debis.de [53.122.101.2]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id EAA06270 for ; Fri, 16 Oct 1998 04:13:59 -0400 (EDT) Received: by dsh.de; id KAA11635; Fri, 16 Oct 1998 10:12:45 +0200 (MET DST) Received: from dshmail.dsh.de(53.47.15.3) by neptun.dsh.de via smap (3.2) id xma011343; Fri, 16 Oct 98 10:12:15 +0200 Received: from mail1.hh1.dsh.de (mail1.hh1.dsh.de [53.47.9.5]) by dshmail.dsh.de (8.8.7/8.8.7) with ESMTP id KAA21793; Fri, 16 Oct 1998 10:09:49 +0200 (MET DST) Received: from mars.SAPserv.Hamburg.dsh.de (root@mail5.hh1.dsh.de [53.2.168.17]) by mail1.hh1.dsh.de (8.8.7/8.8.7) with SMTP id KAA01799; Fri, 16 Oct 1998 10:12:11 +0200 Received: from orion.SAPserv.Hamburg.dsh.de by mars.SAPserv.Hamburg.dsh.de with smtp for <> id m0zU2aB-000B5AC; Fri, 16 Oct 98 07:37 MET DST Received: by orion.SAPserv.Hamburg.dsh.de for maillist@candle.pha.pa.us id m0zU57w-000EBQC; Fri, 16 Oct 98 10:20 MET DST Message-Id: From: jwieck@debis.com (Jan Wieck) Subject: Re: [HACKERS] What about LIMIT in SELECT ? To: Inoue@tpf.co.jp (Hiroshi Inoue) Date: Fri, 16 Oct 1998 10:20:47 +0200 (MET DST) Cc: maillist@candle.pha.pa.us, jwieck@debis.com Reply-To: jwieck@debis.com (Jan Wieck) In-Reply-To: <000001bdf8d4$e4cdf520$2801007e@cadzone.tpf.co.jp> from "Hiroshi Inoue" at Oct 16, 98 04:16:29 pm X-Mailer: ELM [version 2.4 PL25] Content-Type: text Status: RO Hiroshi Inoue wrote: > In the following cases I didn't modify my code to use index scan, > because I couldn't formulate how to tell PostgreSQL optimizer whether > the response to get first rows is needed or the throughput to process > sufficiently many target rows is needed. > > 3.The access plan made by current PostgreSQL optimizer for a query with > ORDER BY clause doesn't include index scan. > > I thought the use of Tatsuo's QUERY_LIMIT to decide that the responce > is needed. It is sufficient but not necessary ? > In Oracle the hints FIRST_ROWS,ALL_ROWS are used. I still think that the QUERY LIMIT should be part of the parse tree and not thrown in by a magic SET command. If rewriting or function calls turn the one query sent to the backend into multiple queries processed internal, how should this QUERY LIMIT variable know to which of all the queries it has to be applied? It can really break functions and rewrite rules if this variable is used on all queries while it is set. For your case 3 I think, if there is a QUERY LIMIT in the parse tree, the (future) optimizer definitely knows that not all rows will get processed even if there is no qualification given. So if there is an index, that matches the ORDER BY clause and it is no a join and the (future) executor handles OFFSET in single table index scans fast, it could choose an index scan for this query too. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #======================================== jwieck@debis.com (Jan Wieck) # From owner-pgsql-hackers@hub.org Fri Oct 16 12:02:27 1998 Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id MAA13063 for ; Fri, 16 Oct 1998 12:02:23 -0400 (EDT) Received: from hub.org (majordom@hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id MAA18435 for ; Fri, 16 Oct 1998 12:01:46 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id LAA24469; Fri, 16 Oct 1998 11:28:54 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Fri, 16 Oct 1998 11:25:54 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id LAA24370 for pgsql-hackers-outgoing; Fri, 16 Oct 1998 11:25:52 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from dsh.de (firewall-user@neptun.sns-felb.debis.de [53.122.101.2]) by hub.org (8.8.8/8.8.8) with ESMTP id LAA24356 for ; Fri, 16 Oct 1998 11:25:34 -0400 (EDT) (envelope-from wieck@sapserv.debis.de) Received: by dsh.de; id RAA06506; Fri, 16 Oct 1998 17:28:04 +0200 (MET DST) Received: from dshmail.dsh.de(53.47.15.3) by neptun.dsh.de via smap (3.2) id xma006149; Fri, 16 Oct 98 17:27:12 +0200 Received: from mail1.hh1.dsh.de (mail1.hh1.dsh.de [53.47.9.5]) by dshmail.dsh.de (8.8.7/8.8.7) with ESMTP id RAA00811 for ; Fri, 16 Oct 1998 17:24:37 +0200 (MET DST) Received: from mars.SAPserv.Hamburg.dsh.de (root@mail5.hh1.dsh.de [53.2.168.17]) by mail1.hh1.dsh.de (8.8.7/8.8.7) with SMTP id RAA04532 for ; Fri, 16 Oct 1998 17:26:54 +0200 Received: from orion.SAPserv.Hamburg.dsh.de by mars.SAPserv.Hamburg.dsh.de with smtp for <> id m0zU9N0-000B5AC; Fri, 16 Oct 98 14:52 MET DST Received: by orion.SAPserv.Hamburg.dsh.de for pgsql-hackers@postgreSQL.org id m0zUBum-000EBQC; Fri, 16 Oct 98 17:35 MET DST Message-Id: From: jwieck@debis.com (Jan Wieck) Subject: [HACKERS] SELECT ... LIMIT (trial implementation) To: pgsql-hackers@postgreSQL.org (PostgreSQL HACKERS) Date: Fri, 16 Oct 1998 17:35:39 +0200 (MET DST) Reply-To: jwieck@debis.com (Jan Wieck) X-Mailer: ELM [version 2.4 PL25] Content-Type: text Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: ROr Here we go, this is up to now only for discussion, do not apply to CVS! Those involved into the LIMIT discussion please comment. Here is what I had in mind for the SELECT ... LIMIT. It adds SELECT ... [LIMIT count [, offset]] to the parser and arranges that these values are passed down to the executor. It is a clean implementation of LIMIT (regression tested) and the open items on it are to enable parameters and handle it in SQL functions and SPI stuff (currently ignored in both). Optimizing the executor would require the other sort node stuff discussion first to come to a conclusion. For now it skips final result rows - but that's already one step forward since it reduces the rows sent to the frontend to exactly that what LIMIT requested. I've seen the queryLimit by SET variable stuff and that really can break rewrite rules, triggers or functions. This is because the query limit will be inherited by any query (inserts, updates, deletes too) done by them. Have a rule for constraint deletes of referencing tuples CREATE RULE del_table1 AS ON DELETE TO table1 DO DELETE FROM table2 WHERE ref = OLD.key; If the user now sets the query limit to 1 via SET and deletes a row from table1, only the first found record in table2 will be constraint deleted, not all of them. This is a feature where users can get around rules that ensure data integrity. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #======================================== jwieck@debis.com (Jan Wieck) # begin 644 opt_limit.diff.gz M'XL(`$]=)S8"`^4\:W?B1K*?R:_H82=98&1;$F^<>`^+F1DV&!S`NXLSV5RO+6X3RX=1&F#W57U4JE6/0%-X&#AG;$3$:Q#`Z]6JG MVB)&N]WZZN3DY$`?J;9FI][HU&J\;27]88.IMC6H9J^(&E_K!)Y.OB+D;XYG MNYL%)<5-Y+CAV%S-JUGX066YV7?@4VI9]3['VC5(;1H'CW?$V\.^L MPKI5/U!**H7"-4/>_TCM3>0'[ZFUOJ(K/W@B8;19+C.I,G1#,_1&0I>AFU#0 MDI05'"\J%`I+:D6;@)YCR0\;&CQ=TM"&/G^1SZR&]];SO8A^C(C-?\^!E@*4 MA8#'=5:VO_&BL$YZ`9!%B85##2/B^3#;R\!?D>B>DCOG@7J$#9<\6.Z&\D8YN,[P9T57 M(8U*WT@R-:)K)'3^0_UE29:5RVQ"Y.MI]+2FA<)W9#9GR7@:4)EU`4[87 MVK"+V\DN_NL06!KA+JN41S?#8?(6;YG?1OHB5P%1,5SV<&4YWI8&RJC/5D$9 M@,?KH$.-&QU#[]1;^4JH4=,:[43DP6M3EXLXMKU_)*^$4J M:_C62)\M":E0!*1LHD63V6;MTIEUZ]*IZT>DPM>6,@SI5K#`"4:V"%P=0(D5 MA'0&"Z*1WFHQ`YE%_#4-`+\/D&\0$D03\9=+$!"LPU`@P&)OL[JE@2@F4]OR M+IV`VMB8+.23@&<$EBIK4(K1 M+5M<-$#5R'E'5;LC;#[WP[`@%V\/#;:6NK,275M1-MGVFN`,QAL,7_A6`O2? M<,K$>A0*\8(D>KR``XG73@@/8"LCX:N_-KE<@(#XX)RE%'SY)&2K##"4FJJA MU`01VVS'(G:;M("&&Y=9.F#],(OS$L49RC16B.Q7*)0J6.!X;$3(@4QG,J.+ M$P026D_*N#G"BJ1QL_,YWH)J`TEME:1V33/K"4E\PD\N:#A?![Y-0;6`?.8NPLV`#?9`^06U\@X;F-)76\(-A?LK?AX&HP([9K;4)A M>N4@8J87JHN8"<@KT!F@7\I8\XD)'V&`5FP.P684QL9^KJW`6D'=6JUC:-E3 M^.A$]CTIH7DXL^Z2;LH,O^B@`'8-E59J>O?E?J3R7BBD>(::VA$`V*%!#IVI/P% M4S=TLZ-("3L[L8>/1CVN0SH""6<]HJY*O%]O]%362 M,7N\OV?9$2ZA&-ZW1.=SF^XM10+H"*3BEA*/WL'V>:""@N?8J#R:\_BV^BV, MQS'LX;O$X-YA.Z5J+]<)_^$@TTDAH?*\0JP':@2S86IF(]$(+S)4 MH:G5(NXNO]#`L^;"^C:;K1>@[,TN9>0%*@IF/.KS_?+2:],`JZJA6E7- M>DMK-FH*,8=]*%[\6QTI7OP%WE0F8(HN>5+N"Y, M;\V4",Y1T*BA81U`A#^21]3-J#.]OW`6)8"DBGRQ\`O+?@1FC%.:? MN.!)NJ%L!KC7?A@Z8.Z3TF*#`5J`7="/)(3I+DNP$_(ORQ,O><2>;5D_?/'( MA51+0M6?G*C50G=BC-7Q-CPP&T<\,V?VB-@(KC^R1I@7'%$`#D1'%,@O"(_D MM:YV=*.CF_GQ$5/70;4VI&K%R1`J&`/NT);L%FP?10AR\1+$GEMV$ M'\#P)O$9238H.XJ1E(KCF#RDZ#$?A90?T&R!RH`\DQ\L(H\B@D?DF2)>!Q1] M/[8O1'1><"!C>SSK7()?Q_D1A-;-#%AR.NO.IMD,7@,&KQN)AZ2P\_C[>>]F M,AU/$D;./XX0`H_)R;?HUD7I4X8Y8K0W04#1`U:#7!FAKW,Y#?$>Q:GH75W. MI_UAOSCZMA#E3^,W'PU@5(#S^#7 M3ZCDMRV-G=IL9;$#MJLO:GJVOMC?%/2,V:G6\U6&46^V-*/>4CQ)6:0LK$J>N[['ZZFW-;"C) M&;P@<>ZM]1IZF;(\BH&W]$&0!1J/K9U*P^PP`E_+7(3*JA,194V!?8Q.K;IG3[/C/363!`N:B<<>^=`- M[%\W7,-CB4O9;\`98PP+()#+F$#%>6QR5' M=Z6R"7)*$I:'2&-6"@D.+/`.7_Z#]WB MJ.W*;(;:ALK@)R.;G_:VK()+T3&;>]BII6M&2SU6PX*VKKI[,I0^I2ZUHVFT MBCHBA@/Z]E6IE)2#'D^B7&50VBRC`O->1-!:!%V2J$L46%ZX](-5@J2T%B9# M+F(9B^=8MAEH:T#J>!3`3!R2-0ZAZ"D'%,^<)`KR-Y>XWB8(_>!8XM08U$&F MNPNLU>E3%D?PFKWLQD$R>,WH=L[HW(Q&,Y+KB!.*KL9N1:%O`+_[= M&?PBUV2G'N+)QW:-9:,=>/;1LJ/MC$2)(D1S*MPJ8@CHQW4@VEA1%``@<.,\LH([&HGR MK0*3GVW`5"A]P1#CT"NY"_S-.A[5#/CHC@9O07EU@[N0S^77S#?]%N7E10'U MVIS+T:0*HQ/4\BX*L/?FT(62O&$DMNC_RRGDK;%-R&3"G,F8+Y[8[%,\3+%K M;F5/Z(8BY@4Z!]QTZ(E&]OW\WG^+@C'I'/ZR%5^RQ)??T:C5<14?/D#/#JO.G'_(ZF MKL8CF@;HAJ8:H?^5=*<]>:Q<^$18/\5OB^?D6=1?]C,`+A2`LTK_ZGKV<^4L M"\M91>315,X(GR?A]VVS8Z?`ST5WU0_YN_9WDJF59$\N#SJ4Y"\L."Q1K0Q? M9OE<]/LKR>O@$"869=["!5XIHUJXHJ(Y@)^GR=SMK5,8L`BW=)OX2O/CF/3& M8V4QFWDB-2KC'H92FW,)0X'8NH'QVM@&R+A[H=3N7+Q(M=R^=9&JW+YRD<77 MPO'BD\PRO$9^>J9R,KW`&1.Y5]SBVYOK)?>-N,ATV!?[0)\>_6`19D<`DMJ] M/ED"]ME^64[3.K3KF'M"D@9L>,-,3C`^%9E]740]?-G_Z5F3A:!OG"ADY>]! M_\VF295'@V*LWZ'XA!?#L**BU/`)=$B]T&&I:2E30`&(J+7@E3-0]$D%K"*P M5C$Q%I0JGQ>/GS5QQXJ3)<782Y!U`@ M+$!S%F.K26)WZ3(J^ MYQRF:IA:U5`L'%9@*AEP]]1"J\T/:8D^@"0"+\]-L@5%FHQ(P1$:RO'(@T,? MB<.%6+A98[2-+G82==BQ5]JZ$*F_Y-=?B5K;V\D+WA*D:K9QNEX)GNE:+\X24R-;V+6$9VA(G(1B](A6>"3&6YZ*8:,QR\N(" M,!5@LEZEKB1=\X/5^(Y+YFT7R9S9]WP_^]X-BPLR2G?C@G\RI8>O[KSL5&2G MBX%3:!B_Q\4N]>3LU9%K+^\J95Y*PENG55F] MG0R2/XSZS)5O5K5JLYF$+YD*FR/])7['E3%3%)_Y,'>#!0Z<4]]6O?N`=S5P/4M8^[8 MQF9'KW6,/:>EK:K64F_6`@]$/KK:(<\MC%.!T1Q2+F=+*44_@IWO\8NE3!I) M+I]&5A`==:.4Z2\5S\Y5Z!>XC'N>C#6^H(R(8?L<.\@8`=[*9P-E6%BT(H!9 MB<(25]K@&[@TT$B;_N3G.3-%>:8:6Y1$I_S?6I1C M[([_BJ7*%PX\(X8)%O:X)1MVJ[-%PRY!W""-MZ"!EQ5!O3!\3>A>J6G MPATVX=\A+S"T:F(#HOT;W@]-7S")?)XAD]62YS,4\EN*K!BV7X1F93$+<.>P M&G6-!Z/F.C4F]M8**7JR/(T0&V$)D;XMP>*\OWK2J+:UAOHWC!HU`PJ2-+DX MBSU),N$4*']QI62SQ(LRCM'F">%\B#Q7_A;#IG*<&6X2_LHNFC+*R0/./ J5".V8K`1>&?LC3PZT;V_B8@X[<#CAC]SA9Y); Sun, 18 Oct 1998 14:01:48 -0400 (EDT) Received: from ra.sai.msu.su (ra.sai.msu.su [158.250.29.2]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id NAA23532 for ; Sun, 18 Oct 1998 13:51:14 -0400 (EDT) Received: from ra (ra [158.250.29.2]) by ra.sai.msu.su (8.9.1/8.9.1) with SMTP id UAA17832; Sun, 18 Oct 1998 20:45:25 +0300 (MSK) Date: Sun, 18 Oct 1998 21:45:24 +0400 (MSD) From: Oleg Bartunov X-Sender: megera@ra To: Tom Lane cc: Bruce Momjian , pgsql-hackers@postgreSQL.org, jwieck@debis.com Subject: Re: [HACKERS] SELECT ... LIMIT (trial implementation) In-Reply-To: <2292.908726689@sss.pgh.pa.us> Message-ID: Organization: Sternberg Astronomical Institute (Moscow University) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Status: RO On Sun, 18 Oct 1998, Tom Lane wrote: > Date: Sun, 18 Oct 1998 12:04:49 -0400 > From: Tom Lane > To: Bruce Momjian > Cc: pgsql-hackers@postgreSQL.org > Subject: Re: [HACKERS] SELECT ... LIMIT (trial implementation) > > Bruce Momjian writes: > > What if someone wants the rows from 500 to the end. Should we allow > > the syntax to be: > > SELECT ... [LIMIT count] [OFFSET offset] > > LIMIT and OFFSET are independent. > > I like that syntax the best, but remember we are not inventing in > a green field here. Isn't this a feature that already exists in > other DBMs? We should probably copy their syntax, unless it's > truly spectacularly awful... > > regards, tom lane > Mysql uses LIMIT [offset,] rows >From documentation: LIMIT takes one or two numeric arguments. A single argument represents the maximum number of rows to return in a result. If two arguments are given the first argument is the offset to the first row to return, while the second is the maximum number of rows to return in the result. What would be nice if somehow total number of rows could be returned. This is often needed for altavista-like application. Of course, I can do select count(*) from sometable ... LIMIT offset, rows and then select ... from sometable ... LIMIT offset, rows but this seems not elegant solution. Regards, Oleg _____________________________________________________________ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83 From owner-pgsql-hackers@hub.org Sun Oct 18 14:31:12 1998 Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id OAA02288 for ; Sun, 18 Oct 1998 14:31:10 -0400 (EDT) Received: from hub.org (majordom@hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id OAA24844 for ; Sun, 18 Oct 1998 14:15:35 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id OAA26655; Sun, 18 Oct 1998 14:00:03 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Sun, 18 Oct 1998 13:58:57 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id NAA26381 for pgsql-hackers-outgoing; Sun, 18 Oct 1998 13:58:55 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from ra.sai.msu.su (ra.sai.msu.su [158.250.29.2]) by hub.org (8.8.8/8.8.8) with ESMTP id NAA26367 for ; Sun, 18 Oct 1998 13:58:49 -0400 (EDT) (envelope-from oleg@sai.msu.su) Received: from ra (ra [158.250.29.2]) by ra.sai.msu.su (8.9.1/8.9.1) with SMTP id UAA18077; Sun, 18 Oct 1998 20:58:41 +0300 (MSK) Date: Sun, 18 Oct 1998 21:58:41 +0400 (MSD) From: Oleg Bartunov X-Sender: megera@ra To: Jan Wieck cc: PostgreSQL HACKERS Subject: Re: [HACKERS] SELECT ... LIMIT (trial implementation) In-Reply-To: Message-ID: Organization: Sternberg Astronomical Institute (Moscow University) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: RO Jan, I tested your patch on my Linux box and it works ok, except aggregates functions doesn't work properly, for example count(*) always produces 0 kdo=> select count(*) from work_flats limit 10,1000; count ----- (0 rows) while kdo=> select rooms from work_flats limit 10,1000; rooms ----- 3 3 3 3 3 3 3 3 3 3 (10 rows) Regards, Oleg _____________________________________________________________ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83 From wieck@sapserv.debis.de Sun Oct 18 15:17:53 1998 Received: from dsh.de (firewall-user@neptun.sns-felb.debis.de [53.122.101.2]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id PAA03203 for ; Sun, 18 Oct 1998 15:17:49 -0400 (EDT) Received: by dsh.de; id VAA01180; Sun, 18 Oct 1998 21:19:50 +0200 (MET DST) Received: from dshmail.dsh.de(53.47.15.3) by neptun.dsh.de via smap (3.2) id xma001117; Sun, 18 Oct 98 21:19:33 +0200 Received: from mail1.hh1.dsh.de (mail1.hh1.dsh.de [53.47.9.5]) by dshmail.dsh.de (8.8.7/8.8.7) with ESMTP id VAA25465; Sun, 18 Oct 1998 21:17:29 +0200 (MET DST) Received: from mars.SAPserv.Hamburg.dsh.de (root@mail5.hh1.dsh.de [53.2.168.17]) by mail1.hh1.dsh.de (8.8.7/8.8.7) with SMTP id VAA14993; Sun, 18 Oct 1998 21:19:58 +0200 Received: from orion.SAPserv.Hamburg.dsh.de by mars.SAPserv.Hamburg.dsh.de with smtp for <> id m0zUvyS-000B5AC; Sun, 18 Oct 98 18:46 MET DST Received: by orion.SAPserv.Hamburg.dsh.de for maillist@candle.pha.pa.us id m0zUyWO-000EBPC; Sun, 18 Oct 98 21:29 MET DST Message-Id: From: jwieck@debis.com (Jan Wieck) Subject: Re: [HACKERS] SELECT ... LIMIT (trial implementation) To: oleg@sai.msu.su (Oleg Bartunov) Date: Sun, 18 Oct 1998 21:29:43 +0200 (MET DST) Cc: tgl@sss.pgh.pa.us, maillist@candle.pha.pa.us, pgsql-hackers@postgreSQL.org, jwieck@debis.com Reply-To: jwieck@debis.com (Jan Wieck) In-Reply-To: from "Oleg Bartunov" at Oct 18, 98 09:45:24 pm X-Mailer: ELM [version 2.4 PL25] Content-Type: text Status: RO Oleg Bartunov wrote: > On Sun, 18 Oct 1998, Tom Lane wrote: > > > Bruce Momjian writes: > > > What if someone wants the rows from 500 to the end. Should we allow > > > the syntax to be: > > > SELECT ... [LIMIT count] [OFFSET offset] > > > LIMIT and OFFSET are independent. > > > > I like that syntax the best, but remember we are not inventing in > > a green field here. Isn't this a feature that already exists in > > other DBMs? We should probably copy their syntax, unless it's > > truly spectacularly awful... > > > > regards, tom lane > > > > Mysql uses LIMIT [offset,] rows > >From documentation: > > LIMIT takes one or two numeric arguments. A single argument > represents the maximum number of rows to return in a result. If two > arguments are given the first argument is the offset to the first row to > return, while the second is the maximum number of rows to return in the > result. Simple change, just flip them in gram.y. And for the 500 to end: SELECT ... LIMIT 500, 0 (after flipped) The 0 has the same meaning as ALL. And that could also be added to the parser easily so one can say SELECT ... LIMIT 500, ALL too. > > What would be nice if somehow total number of rows could be returned. > This is often needed for altavista-like application. > Of course, I can do > select count(*) from sometable ... LIMIT offset, rows > and then > select ... from sometable ... LIMIT offset, rows > but this seems not elegant solution. Absolutely makes no sense for me. As said in the other posting, aggregates do the counting scan in a deeper level and thus cannot get limited. So if you invoke an aggregate, the whole scan is always done. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #======================================== jwieck@debis.com (Jan Wieck) # From owner-pgsql-hackers@hub.org Sun Oct 18 19:08:47 1998 Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id TAA00573 for ; Sun, 18 Oct 1998 19:08:46 -0400 (EDT) Received: from hub.org (majordom@hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id QAA01305 for ; Sun, 18 Oct 1998 16:14:30 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id PAA06110; Sun, 18 Oct 1998 15:55:20 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Sun, 18 Oct 1998 15:54:07 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id PAA05771 for pgsql-hackers-outgoing; Sun, 18 Oct 1998 15:54:05 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from dsh.de (firewall-user@neptun.sns-felb.debis.de [53.122.101.2]) by hub.org (8.8.8/8.8.8) with ESMTP id PAA05753 for ; Sun, 18 Oct 1998 15:53:52 -0400 (EDT) (envelope-from wieck@sapserv.debis.de) Received: by dsh.de; id VAA09240; Sun, 18 Oct 1998 21:56:10 +0200 (MET DST) Received: from dshmail.dsh.de(53.47.15.3) by neptun.dsh.de via smap (3.2) id xma008902; Sun, 18 Oct 98 21:55:19 +0200 Received: from mail1.hh1.dsh.de (mail1.hh1.dsh.de [53.47.9.5]) by dshmail.dsh.de (8.8.7/8.8.7) with ESMTP id VAA28158; Sun, 18 Oct 1998 21:53:16 +0200 (MET DST) Received: from mars.SAPserv.Hamburg.dsh.de (root@mail5.hh1.dsh.de [53.2.168.17]) by mail1.hh1.dsh.de (8.8.7/8.8.7) with SMTP id VAA15349; Sun, 18 Oct 1998 21:55:45 +0200 Received: from orion.SAPserv.Hamburg.dsh.de by mars.SAPserv.Hamburg.dsh.de with smtp for <> id m0zUwX6-000B5AC; Sun, 18 Oct 98 19:22 MET DST Received: by orion.SAPserv.Hamburg.dsh.de for hackers@postgreSQL.org id m0zUz52-000EBPC; Sun, 18 Oct 98 22:05 MET DST Message-Id: From: jwieck@debis.com (Jan Wieck) Subject: Re: [HACKERS] SELECT ... LIMIT (trial implementation) To: terry@terrym.com (Terry Mackintosh) Date: Sun, 18 Oct 1998 22:05:31 +0200 (MET DST) Cc: hackers@postgreSQL.org Reply-To: jwieck@debis.com (Jan Wieck) In-Reply-To: from "Terry Mackintosh" at Oct 18, 98 03:58:57 pm X-Mailer: ELM [version 2.4 PL25] Content-Type: text Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: RO > > On Sun, 18 Oct 1998, Tom Lane wrote: > > > Bruce Momjian writes: > > > What if someone wants the rows from 500 to the end. Should we allow > > > the syntax to be: > > > SELECT ... [LIMIT count] [OFFSET offset] > > > LIMIT and OFFSET are independent. > > > > I like that syntax the best, but remember we are not inventing in > > a green field here. Isn't this a feature that already exists in > > other DBMs? We should probably copy their syntax, unless it's > > truly spectacularly awful... > > > > regards, tom lane > > None that I have used (VFP, M$ SQL Server) that had 'LIMIT', had 'OFFSET'. > So it would seem that the very idea of OFFSET is to break with what others > are doing. > > I too like the above syntax. > Why mimic, when you can do better? Go for it! > We have a powerful parser. So we can provide ... [ LIMIT { rows | ALL } ] [ OFFSET skip ] or ... [ LIMIT [ skip , ] { rows | ALL } ] at the same time. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #======================================== jwieck@debis.com (Jan Wieck) # From owner-pgsql-hackers@hub.org Sun Oct 18 19:08:39 1998 Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id TAA00557 for ; Sun, 18 Oct 1998 19:08:37 -0400 (EDT) Received: from hub.org (majordom@hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id QAA03555 for ; Sun, 18 Oct 1998 16:56:03 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id QAA10374; Sun, 18 Oct 1998 16:36:26 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Sun, 18 Oct 1998 16:35:16 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id QAA10298 for pgsql-hackers-outgoing; Sun, 18 Oct 1998 16:35:15 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from dsh.de (firewall-user@neptun.sns-felb.debis.de [53.122.101.2]) by hub.org (8.8.8/8.8.8) with ESMTP id QAA09974 for ; Sun, 18 Oct 1998 16:32:21 -0400 (EDT) (envelope-from wieck@sapserv.debis.de) Received: by dsh.de; id WAA18249; Sun, 18 Oct 1998 22:34:46 +0200 (MET DST) Received: from dshmail.dsh.de(53.47.15.3) by neptun.dsh.de via smap (3.2) id xma018115; Sun, 18 Oct 98 22:34:11 +0200 Received: from mail1.hh1.dsh.de (mail1.hh1.dsh.de [53.47.9.5]) by dshmail.dsh.de (8.8.7/8.8.7) with ESMTP id WAA29950; Sun, 18 Oct 1998 22:32:01 +0200 (MET DST) Received: from mars.SAPserv.Hamburg.dsh.de (root@mail5.hh1.dsh.de [53.2.168.17]) by mail1.hh1.dsh.de (8.8.7/8.8.7) with SMTP id WAA15581; Sun, 18 Oct 1998 22:34:28 +0200 Received: from orion.SAPserv.Hamburg.dsh.de by mars.SAPserv.Hamburg.dsh.de with smtp for <> id m0zUx8Z-000B5AC; Sun, 18 Oct 98 20:01 MET DST Received: by orion.SAPserv.Hamburg.dsh.de for jwieck@debis.com id m0zUzgV-000EBPC; Sun, 18 Oct 98 22:44 MET DST Message-Id: From: jwieck@debis.com (Jan Wieck) Subject: Re: [HACKERS] SELECT ... LIMIT (trial implementation) To: jwieck@debis.com Date: Sun, 18 Oct 1998 22:44:15 +0200 (MET DST) Cc: terry@terrym.com, hackers@postgreSQL.org Reply-To: jwieck@debis.com (Jan Wieck) In-Reply-To: from "Jan Wieck" at Oct 18, 98 10:05:31 pm X-Mailer: ELM [version 2.4 PL25] Content-Type: text Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: RO > We have a powerful parser. So we can provide > [...] This version now accepts all of the following ... [ LIMIT rows ] [ OFFSET skip ] ... [ OFFSET skip ] [ LIMIT rows ] ... [ LIMIT [ skip , ] rows ] rows can be a positive integer constant greater that 0, a $n parameter (in SPI_prepare()) or the keyword ALL. 0 isn't accepted as constant to force ALL in that case making clear that this is wanted. In the parameter version the integer value 0 still is used to mean ALL. skip can be a positive integer constant greater or equal to 0 or a $n parameter for SPI_prepare. If any of these syntaxes is used in SPI_prepare()'d plans, the given tcount argument for SPI_execp() is ignored and the plan or parameter values are used. Anyone happy now? Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #======================================== jwieck@debis.com (Jan Wieck) # begin 644 opt_limit.diff.gz M'XL(")%0*C8"`V]P=%]L:6UI="YD:69F`.4\:W?;-K*?U5^!:-.NY-"V*-EZ MN?$>5:83;67)U2-MSSWWZ-`29/-&(E62BN--_=_OS``@08F4E4>;[JY/&Y'` M8(`9#&8&@P%GSGS.#J<^"_SID><[M\'A$WTDVI:;I]7FR8EH>Y#\ MH\%4&@94TRNBQM=3!D^'WS#V-\>=+M8SSO+KT%D$Q\OI^_#H+I]2L_+\T%ZD MUP4/P=2>WG&L?:'5!J'ON+>B#?QW?$#=ZG]0R@YRN6M";KWGTW7H^:^YO;KB M2\]_8$&XGL]3J3)+IF&6JC%=9JD,!75%6,?9K?..NXR&R][9BS47C3)P'>//DB\#'A:^4V0:K&2PP/D7]^8%558L$D/4 MZU'XL.*YW$LVFA"/DI4T+H)XR3J]T4F_)``0)AL.;CE M(P`7.;"`7XEA+J(T!+\%0]CR((8 M4)$D^QG+*5R#M5N(%HG!HI;#$"1/-3*87%^&D+>B6A_.G!5F'$3SY4O6\UQ> MA*4(O+KJO[&0-!S+:NYS'GO MM9*:Y``X[DS9.\^9,T'?`038K#V0+Q`2 M5!/SYG-0$-1A(!%@L;M>WG!?%K/AU'8O')]/L3&;J2<)3P06#E9@%,/YVIT6 M6:&H4R,8`"0,.!A._HX7-ND+X%^#I>$QTM5%%4R-XCN:VBUE\[%_A`6E>'-H ML+3TE17;V@-MD6W."7(P6F#X(I82H/^`+)/SDI!BJVUHA4["9I/@_6"_)TP/LAC_," MU1GJ-"I$\!]+URD!$KA>:BT@(V#.P&6!?BECS@92/ M=$`/I@*".)J[MGU["86KK<*N$X0==^[E5O36$54T!8YR6G+!O1-.[U@!_<:1 M?1OW7Z2.9<\Y<'BX*"O0& M'-FWT;A4CT2)ZG$5HZ/RM!XEL0"CSSP63A;`D(D#'(E[(9Y'332.XQ]?>+<% M:S#H#PR6)R`>F3*,I,-Y/B(,X0L.R1R37?V/\[]' M;QUWAI-\W1JTKB:=WIM6%SQCYKQXH0;P00U$&V34\J5JV1M?L>^^TS$[5+M2 MO*8:9Q:1%7,:7QZW&;'=AQS=E^;,1I_";T_O)($63"MBON$T7V)G$V/=D#FM M@UC@TF5NQNU0= M(=URE-^SDB1X'W)=?@M*]ITB]C':>NRMGX3R_1SU)##LT$[QMBQ63IMEGZJ; MY/;S2=6D;(RNF;0QQ$9(ZB5Z_62U%"'3M%*BMS]3*8G!_$5UDN3SUU!)>_'E MHS62P+I;(25E[<_21]1K4AWIP9ND-A)#W*F,-BE-TT7@4JE%*YU;VHNC\YKA MO36,OZ][<\2^[($ M@.[$GHD*.1_H5Y8;9:/_K12+H]H4&GL;S;%/&OB!EJ=2485K*-8T:LV94RI6O0,T/]O3M9Y-3J1GED[)&3J5B M5"J-+TM.*0'3[UEBO7SIN:G"WJRJ[\UJ];I1:YQHQ#P=B1'%GQN.$<6?$)-) M):QV4C9J)YK0U4]*1OU4FZ4/`J78/S-API-+,Z&",QPX].!@'D"%W[-[]-V6 M0$S(%P_2?G$11D8>PM:PZ)?`8JWN>_K6'&P%S]83!X[5V9).A^'AWJUM+)X4N.X:W&\ M$YV;I')VCP@KSC^*1I`58M4`GHBQ:I"?$&3-:MUHELQFJ9P=934K)<.LG&J! M$2K84O,Y$9D6;!/>*GI-_T$]$+=COO[-- MB/;6QF'#S]"W(^*49/A3ET4<(-_M`?IP&)M;;=K-,NGS4HE M>U75P*35$B:-"N)5=>-Y"Y"U0,7M.V[H#?B";!9,U=Q>!$)%3>]LGV*V()U> M#W8KNI40$NM+Y29WUB&Y')-I=,2*S*=PH-A>J-:Z%Q[92^F'I]%4+Y\:]4I) MLV;EFE&OGFJ:0DK)\+HSH<4ZZ5^/>S_V^C^3'`EM^V+7*?4K&7X@G2#-#]DF MX4<]?2PM-V;"*>^X[SO@WV"Y MLE^"^RS:62J,&3T)4Z:F)G7DT;8)M*"(O^PW_)Z7"-S@^*.Q'[%Q@,D`"V]J M+U@[2@F(J/#8:@U3(0:&H@>F?2]25%Z`+H5Z;H!>KK:8>ADE";#X+YDMD`2- M4@84J)XWD`*ZX&Z$=2N%(`6>]MT27N82A%J\9KN!B`.(!G%60`H@918H^E1V M02I"-)9[(!2Y!AN`"=F*CI^W](/:DN/:D89=KA5Q#DV.X\KG*-.DG^29M%SM MI'XPPV<^XW.Q]L%XC$>P_(>CUFB8KDQ.0)FA,FS]0EBG*Y]GU-`23O:23GP.5-LB/0ALJ)]=3$96EVK/<)@ M5=3WX;DZLXQJR+:=:+%8I61(D M\JF>FAAO'6;/2URY:UIBJ(^=E8R6,"FEYFDI>U+*IPVC7-42$D5![$W9JQ7T M,J3<03PN`#7F&^)`X/#\S@Z&ZYNNX[X-P,W-A_Z:Y\'#S9./E1?I%!D8`$Z? MX;R`G0`=.%4"9DL0Q%QF(M1W03(FFH9P2SAV(Q0^SE/XVI&V^"@'7,P>;/5F MV;*CU>X2'@WL8Z4GJRF(C]D\V>&,BY06/7L2"VIQ?"GTH!M8OXM@!8\%L6?Y M#KR%V_"N>$8V$&,F26$01I`Q*HI;[:&-R)[YQ[9K+Q[^Q39ZRM"'#$[$,<+0M]U@[OG+&$EA)1V&3,3JC$E@V12@C0'I MX]$`4W$HT7@*15L[YWP4)-%V)HNX]MH/,**T'W%ZQ/1)H;N%K=G10YI$B)J= MXB9`4F2MO%/69+/AVA7-ZJQ<;I9+S4HU6]#`13[5MMOX:FYGX4]MD!?O]AA^ M46K2T^TQ?K998T_1#SQ^;T_#S2Q\A7-U.\'M592.+WW\JW%WU/GAUY&5P+>\ M0?A[#%A@@SV2"W,YF&+8B9#*!*<77FX>R($R1&A8G'T'ZDW4()7BD5"@JQ2A MH!W[]L/D!D1O%J`[%6P4B6#F^Y4OV]AAZ`,@2.,DM/U;'LKRC8*R M.(D#5FA]P1"C@P)VZWOK532J$'GG@XO9O<>?>3I>T^"'.?&(H^THD(+<@B<4PQ43KM:,9\(_5>=6#AT'_"OY%GX$:O1JT>@#T:M`?0\O7K3>=WBOX[8\'@(D@.M"J MT^M96``/0ZLW[(PZ;RSQ,ACA[\@:O&EUZ:D/_P[%-/ZSCVU_M'XU6+?5>S5N MO8)&7:MU07UTK4MHV^W\B(7]=DL.Z*I%@[_J],8CHN:JWQN]QH=>ZTH=Z?=: MHTZ_ASW"TWA`#^W7+2"T9_T"6'M]_!\?@%!J.[ZR!IVV:-V_-%B_A_]W86C] M:\0%OP/\_\+"'^B:Z*<]2,&_YUV9TJYHV&4=%31#J]]L"Z MLGK$D`OK%_QY#;V-AL2I$="-K.E%K`>>M/M7UYTN]M7M`$0/68!@R`@Q''CZ MDZ_ZFS@=7N7`.^@853@;]# M:X0_,`4]P#'HPV`&8W5N/[1^&EN]-@#"6#HX1P".O!^^1L#AJ(4RA)$[R;WA MZ`+E!GY@`@P,5P)K8!R$;=Q3G!KW1AU`]J;5'H^O\+?;`:`WUN"'_M"BAR%P M4&:25&"NM&/P?_>YNKPDEO^GSEE&3`;TW:E^Q@5OIT:Y6M*T.N0SKC*+)WI297/39/.M#IQSF>4=(F--T($X5VA9``2,]H*QE#1]AZ` MS`V@QSBI47%[9T?B=#&KAT3MHR+U^7/M?,)5'!#56>S'C,!:28\)UDSPSQ)I M0[^SUK`=)3!\8-1/_OO\&7N4]1=6"L"Y!G!\8%U=CWX].$[#[N&&9RX>GK_#-6 MWG_ABOC+3[;P">3'09Z.];[E#_>>/PO23QCBVITQWQCLH^.^6M-D[)?.&78< M,H`S8Y;C_(@/>8K?Y>4&Z=%0A>!+.V&0CW=,<97+_7P4*X#B0U$,PPKS*EH0 M0P?<%68BGP@K:``AMV?Y:$,65X385>0CR(P"ON"S\.\",=$1O-TQU) M*A6S;%3,Q&66A5#2?07-1C$=B#$1\='=%-AKF\BH!Y MF?YZP2/?%P\X9@S3F1$)G8O)MB)U+1"W-PKJ;MP$06!`V.);%#_!->_F_P[/ M@75T'+OK9#&<>JOC%=&^(5Z)FG2A2H"DB%(M792RFYTT2^7=)XLEXR1*9=>. M_38_&Z;._62:"![ML0.1\]]726-X9Y"NUT0%8,Z!6<\2WRBY%EEGT4=> M?;PD]2LE^%FGA>V*8GB0=RTE!7_`QUL^@8+]O\^R)XFIJK]2,BJ5:.UBKC:S MI^&:U*"_=J.;7D='3WPK;+_DT\&XIZZ0)>F+7MX M-8/FLHP2U>++>&SK.RI&6F:J_+"7HE7[1-H?3ZLV>UO?Z?K3J$^=^5K%J-1J M\0$7F;`)TE\0'[V*LON+4?ZIB-KB&<66"">U[J/@N.PC\B.^5!]:PXT$YLUR M/45QV]Y*!9[\N!@^'-V1;=Q1G[2].P"W+?#IAC.W;^-RLW32-'>DDM4K1EW_ MU!;(0.CA=C@0]]KB.VWR6P3R:VU*2_'WL$EQQ9>F2!MI-_'\<*]/3)']TO%L M?1OM"WR=ZRP>:_3%,D0,RV??048(\#-]-%#"0C$;'[@2!@5AM&%CL^"^P:+; M5."_&5HKNDY4U&XL_#2V!K].R!452?PT*;%-^<^:E'W\CG^+JB&>KIFN&)MI5FY;19V7$KMUHRJMJ=7'RMZ?FEF`Q$GH)V M?'B&]HG>`V;[',R7^Q8V06MWQGUA*GW^SO'6@7X[_T!>M1/[.[HIB&CUK$]$ M^S?\%%#RKGCHB?3AM)8BV3.7W5*F#--ZD9:5`BZPG<-JM#4NC%K8U(C8&SO@ MN),5=RRP$98PM;=E6)SU&=1JI6%4]8\:5T],*(CO$$07+>,,7$&!]@G6PI2R M4HLXQJGXH*X8HKC.>0/;4?_A3-R:LIEXI<^Y$&5%B>8?B48T8[`01&?TQNZ= >\,Y;ATP>0^,Y\->; Mon, 19 Oct 1998 07:31:09 -0400 (EDT) Received: from hub.org (majordom@hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id HAA13574 for ; Mon, 19 Oct 1998 07:12:57 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id GAA13957; Mon, 19 Oct 1998 06:25:09 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Mon, 19 Oct 1998 06:22:35 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id GAA13581 for pgsql-hackers-outgoing; Mon, 19 Oct 1998 06:22:33 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from dsh.de (firewall-user@neptun.sns-felb.debis.de [53.122.101.2]) by hub.org (8.8.8/8.8.8) with ESMTP id GAA13566 for ; Mon, 19 Oct 1998 06:22:27 -0400 (EDT) (envelope-from wieck@sapserv.debis.de) Received: by dsh.de; id MAA13918; Mon, 19 Oct 1998 12:21:16 +0200 (MET DST) Received: from dshmail.dsh.de(53.47.15.3) by neptun.dsh.de via smap (3.2) id xma013635; Mon, 19 Oct 98 12:20:55 +0200 Received: from mail1.hh1.dsh.de (mail1.hh1.dsh.de [53.47.9.5]) by dshmail.dsh.de (8.8.7/8.8.7) with ESMTP id MAA11037; Mon, 19 Oct 1998 12:18:27 +0200 (MET DST) Received: from mars.SAPserv.Hamburg.dsh.de (root@mail5.hh1.dsh.de [53.2.168.17]) by mail1.hh1.dsh.de (8.8.7/8.8.7) with SMTP id MAA29382; Mon, 19 Oct 1998 12:20:49 +0200 Received: from orion.SAPserv.Hamburg.dsh.de by mars.SAPserv.Hamburg.dsh.de with smtp for <> id m0zVA2V-000B5AC; Mon, 19 Oct 98 09:47 MET DST Received: by orion.SAPserv.Hamburg.dsh.de for maillist@candle.pha.pa.us id m0zVCaT-000EBPC; Mon, 19 Oct 98 12:30 MET DST Message-Id: From: jwieck@debis.com (Jan Wieck) Subject: Re: [HACKERS] What about LIMIT in SELECT ? To: Inoue@tpf.co.jp (Hiroshi Inoue) Date: Mon, 19 Oct 1998 12:30:52 +0200 (MET DST) Cc: jwieck@debis.com, pgsql-hackers@postgreSQL.org, maillist@candle.pha.pa.us Reply-To: jwieck@debis.com (Jan Wieck) In-Reply-To: <002801bdfb46$39ad8ec0$2801007e@cadzone.tpf.co.jp> from "Hiroshi Inoue" at Oct 19, 98 06:52:46 pm X-Mailer: ELM [version 2.4 PL25] Content-Type: text Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: RO Hiroshi Inoue wrote: > When using cursors,in most cases the response to get first(next) rows > is necessary for me,not the throughput. > How can we tell PostgreSQL optimzer that the response is necessary ? With my LIMIT patch, the offset and the row count are part of the querytree. And if a LIMIT is given, the limitCount elemet of the querytree (a Node *) isn't NULL what it is by default. When a LIMIT is given, the optimizer could assume that first rows is wanted (even if the limit is ALL maybe - but I have to think about this some more). And this assumption might let it decide to use an index to resolve an ORDER BY even if no qualification was given. Telling the optimizer that first rows wanted in a cursor operation would read DECLARE CURSOR c FOR SELECT * FROM mytab ORDER BY a LIMIT ALL; Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #======================================== jwieck@debis.com (Jan Wieck) # From owner-pgsql-hackers@hub.org Tue Oct 20 06:01:49 1998 Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id GAA02483 for ; Tue, 20 Oct 1998 06:01:48 -0400 (EDT) Received: from hub.org (majordom@hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id FAA07799 for ; Tue, 20 Oct 1998 05:51:19 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id FAA00108; Tue, 20 Oct 1998 05:17:58 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Tue, 20 Oct 1998 05:16:37 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id FAA29953 for pgsql-hackers-outgoing; Tue, 20 Oct 1998 05:16:35 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from dsh.de (firewall-user@neptun.sns-felb.debis.de [53.122.101.2]) by hub.org (8.8.8/8.8.8) with ESMTP id FAA29939 for ; Tue, 20 Oct 1998 05:16:27 -0400 (EDT) (envelope-from wieck@sapserv.debis.de) Received: by dsh.de; id LAA04585; Tue, 20 Oct 1998 11:15:05 +0200 (MET DST) Received: from dshmail.dsh.de(53.47.15.3) by neptun.dsh.de via smap (3.2) id xma004337; Tue, 20 Oct 98 11:14:46 +0200 Received: from mail1.hh1.dsh.de (mail1.hh1.dsh.de [53.47.9.5]) by dshmail.dsh.de (8.8.7/8.8.7) with ESMTP id LAA14628; Tue, 20 Oct 1998 11:12:27 +0200 (MET DST) Received: from mars.SAPserv.Hamburg.dsh.de (root@mail5.hh1.dsh.de [53.2.168.17]) by mail1.hh1.dsh.de (8.8.7/8.8.7) with SMTP id LAA03564; Tue, 20 Oct 1998 11:14:52 +0200 Received: from orion.SAPserv.Hamburg.dsh.de by mars.SAPserv.Hamburg.dsh.de with smtp for <> id m0zVVUa-000B5AC; Tue, 20 Oct 98 08:42 MET DST Received: by orion.SAPserv.Hamburg.dsh.de for maillist@candle.pha.pa.us id m0zVY2c-000EBPC; Tue, 20 Oct 98 11:25 MET DST Message-Id: From: jwieck@debis.com (Jan Wieck) Subject: Re: [HACKERS] What about LIMIT in SELECT ? To: Inoue@tpf.co.jp (Hiroshi Inoue) Date: Tue, 20 Oct 1998 11:25:22 +0200 (MET DST) Cc: maillist@candle.pha.pa.us, jwieck@debis.com, hackers@postgreSQL.org Reply-To: jwieck@debis.com (Jan Wieck) In-Reply-To: <000601bdfc03$02e67100$2801007e@cadzone.tpf.co.jp> from "Hiroshi Inoue" at Oct 20, 98 05:24:09 pm X-Mailer: ELM [version 2.4 PL25] Content-Type: text Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: RO Hiroshi Inoue wrote: > > * Prevent psort() usage when query already using index matching ORDER BY > > > > > > I can't find the reference to descending order cases except my posting. > If we use an index scan to remove sorts in those cases,backward positioning > and scanning are necessary. I think it's only thought as a reminder that the optimizer needs some optimization. That topic, and the LIMIT stuff too I think, is past 6.4 work and may go into a 6.4.1 performance release. So when we are after 6.4, we have enough time to work out a real solution, instead of just throwing in a patch as a quick shot. What we two did where steps in the same direction. Your one covers more situations, but after all if multiple people have the same idea there is a good chance that it is the right thing to do. > > Let t be a table with 2 indices, index1(key1,key2), index2(key1,key3). > i.e. key1 is common to index1 and index2. > > And for the query > select * from t where key1>....; > > If PosgreSQL optimizer choose [ index scan on index1 ] we can't remove > sorts from the following query. > select * from t where key1>... order by key1,key3; > > Similarly if [ index scan on index2 ] are chosen we can't remove sorts > from the following query. > select * from t where key1>... order by key1,key2; > > But in both cases (clever) optimizer can choose another index for scan. Right. As I remember, your solution does basically the same as my one. It does not change the optimizers decision about the index or if an index at all is used. So I assume they hook into the same position where depending on the order by clause the sort node is added. And that is at the very end of the optimizer. What you describe above requires changes in upper levels of optimization. Doing that is far away from my knowledge about the optimizer. And some of your earlier statements let me think you aren't familiar enough with it too. We need at least help from others to do it well. I don't want to dive that deep into the optimizer. There was a far too long time where the rule system was broken and got out of sync with the parser/optimizer capabilities. I fixed many things in it for 6.4. My first priority now is, not to let such a situation come up again. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #======================================== jwieck@debis.com (Jan Wieck) # From wieck@sapserv.debis.de Tue Oct 20 13:00:04 1998 Received: from dsh.de (firewall-user@neptun.sns-felb.debis.de [53.122.101.2]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id NAA08269 for ; Tue, 20 Oct 1998 13:00:01 -0400 (EDT) Received: by dsh.de; id TAA14203; Tue, 20 Oct 1998 19:02:15 +0200 (MET DST) Received: from dshmail.dsh.de(53.47.15.3) by neptun.dsh.de via smap (3.2) id xma014037; Tue, 20 Oct 98 19:01:39 +0200 Received: from mail1.hh1.dsh.de (mail1.hh1.dsh.de [53.47.9.5]) by dshmail.dsh.de (8.8.7/8.8.7) with ESMTP id SAA24445; Tue, 20 Oct 1998 18:59:16 +0200 (MET DST) Received: from mars.SAPserv.Hamburg.dsh.de (root@mail5.hh1.dsh.de [53.2.168.17]) by mail1.hh1.dsh.de (8.8.7/8.8.7) with SMTP id TAA06159; Tue, 20 Oct 1998 19:01:40 +0200 Received: from orion.SAPserv.Hamburg.dsh.de by mars.SAPserv.Hamburg.dsh.de with smtp for <> id m0zVcmS-000B5AC; Tue, 20 Oct 98 16:29 MET DST Received: by orion.SAPserv.Hamburg.dsh.de for maillist@candle.pha.pa.us id m0zVfKV-000EBPC; Tue, 20 Oct 98 19:12 MET DST Message-Id: From: jwieck@debis.com (Jan Wieck) Subject: Re: [HACKERS] What about LIMIT in SELECT ? To: maillist@candle.pha.pa.us (Bruce Momjian) Date: Tue, 20 Oct 1998 19:12:19 +0200 (MET DST) Cc: jwieck@debis.com, Inoue@tpf.co.jp, hackers@postgreSQL.org Reply-To: jwieck@debis.com (Jan Wieck) In-Reply-To: <199810201645.MAA07946@candle.pha.pa.us> from "Bruce Momjian" at Oct 20, 98 12:45:49 pm X-Mailer: ELM [version 2.4 PL25] Content-Type: text Status: ROr > > I agree. Another good thing is that the LIMIT thing will not require a > dump/reload, so it is a good candidate for a minor release. That's wrong, sorry. The limit thing as I implemented it adds 2 new variables to the Query structure. Rewrite rules are stored as querytrees and in the existing pg_rewrite entries that would be missing. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #======================================== jwieck@debis.com (Jan Wieck) # From owner-pgsql-hackers@hub.org Tue Oct 20 13:24:47 1998 Received: from hub.org (majordom@hub.org [209.47.148.200]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id NAA08484 for ; Tue, 20 Oct 1998 13:24:45 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id NAA01878; Tue, 20 Oct 1998 13:00:06 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Tue, 20 Oct 1998 12:59:59 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id MAA01579 for pgsql-hackers-outgoing; Tue, 20 Oct 1998 12:59:58 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from dsh.de (firewall-user@neptun.sns-felb.debis.de [53.122.101.2]) by hub.org (8.8.8/8.8.8) with ESMTP id MAA01557 for ; Tue, 20 Oct 1998 12:59:52 -0400 (EDT) (envelope-from wieck@sapserv.debis.de) Received: by dsh.de; id TAA14203; Tue, 20 Oct 1998 19:02:15 +0200 (MET DST) Received: from dshmail.dsh.de(53.47.15.3) by neptun.dsh.de via smap (3.2) id xma014037; Tue, 20 Oct 98 19:01:39 +0200 Received: from mail1.hh1.dsh.de (mail1.hh1.dsh.de [53.47.9.5]) by dshmail.dsh.de (8.8.7/8.8.7) with ESMTP id SAA24445; Tue, 20 Oct 1998 18:59:16 +0200 (MET DST) Received: from mars.SAPserv.Hamburg.dsh.de (root@mail5.hh1.dsh.de [53.2.168.17]) by mail1.hh1.dsh.de (8.8.7/8.8.7) with SMTP id TAA06159; Tue, 20 Oct 1998 19:01:40 +0200 Received: from orion.SAPserv.Hamburg.dsh.de by mars.SAPserv.Hamburg.dsh.de with smtp for <> id m0zVcmS-000B5AC; Tue, 20 Oct 98 16:29 MET DST Received: by orion.SAPserv.Hamburg.dsh.de for maillist@candle.pha.pa.us id m0zVfKV-000EBPC; Tue, 20 Oct 98 19:12 MET DST Message-Id: From: jwieck@debis.com (Jan Wieck) Subject: Re: [HACKERS] What about LIMIT in SELECT ? To: maillist@candle.pha.pa.us (Bruce Momjian) Date: Tue, 20 Oct 1998 19:12:19 +0200 (MET DST) Cc: jwieck@debis.com, Inoue@tpf.co.jp, hackers@postgreSQL.org Reply-To: jwieck@debis.com (Jan Wieck) In-Reply-To: <199810201645.MAA07946@candle.pha.pa.us> from "Bruce Momjian" at Oct 20, 98 12:45:49 pm X-Mailer: ELM [version 2.4 PL25] Content-Type: text Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: ROr > > I agree. Another good thing is that the LIMIT thing will not require a > dump/reload, so it is a good candidate for a minor release. That's wrong, sorry. The limit thing as I implemented it adds 2 new variables to the Query structure. Rewrite rules are stored as querytrees and in the existing pg_rewrite entries that would be missing. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #======================================== jwieck@debis.com (Jan Wieck) # From wieck@sapserv.debis.de Tue Oct 20 13:10:22 1998 Received: from dsh.de (firewall-user@neptun.sns-felb.debis.de [53.122.101.2]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id NAA08339 for ; Tue, 20 Oct 1998 13:10:18 -0400 (EDT) Received: by dsh.de; id TAA17171; Tue, 20 Oct 1998 19:12:30 +0200 (MET DST) Received: from dshmail.dsh.de(53.47.15.3) by neptun.dsh.de via smap (3.2) id xma017064; Tue, 20 Oct 98 19:12:00 +0200 Received: from mail1.hh1.dsh.de (mail1.hh1.dsh.de [53.47.9.5]) by dshmail.dsh.de (8.8.7/8.8.7) with ESMTP id TAA24806; Tue, 20 Oct 1998 19:09:37 +0200 (MET DST) Received: from mars.SAPserv.Hamburg.dsh.de (root@mail5.hh1.dsh.de [53.2.168.17]) by mail1.hh1.dsh.de (8.8.7/8.8.7) with SMTP id TAA06212; Tue, 20 Oct 1998 19:12:01 +0200 Received: from orion.SAPserv.Hamburg.dsh.de by mars.SAPserv.Hamburg.dsh.de with smtp for <> id m0zVcwS-000B5AC; Tue, 20 Oct 98 16:39 MET DST Received: by orion.SAPserv.Hamburg.dsh.de for maillist@candle.pha.pa.us id m0zVfUW-000EBPC; Tue, 20 Oct 98 19:22 MET DST Message-Id: From: jwieck@debis.com (Jan Wieck) Subject: Re: [HACKERS] What about LIMIT in SELECT ? To: maillist@candle.pha.pa.us (Bruce Momjian) Date: Tue, 20 Oct 1998 19:22:40 +0200 (MET DST) Cc: jwieck@debis.com, Inoue@tpf.co.jp, hackers@postgreSQL.org Reply-To: jwieck@debis.com (Jan Wieck) In-Reply-To: <199810201702.NAA08286@candle.pha.pa.us> from "Bruce Momjian" at Oct 20, 98 01:02:58 pm X-Mailer: ELM [version 2.4 PL25] Content-Type: text Status: RO > > > > > > > I agree. Another good thing is that the LIMIT thing will not require a > > > dump/reload, so it is a good candidate for a minor release. > > > > That's wrong, sorry. > > > > The limit thing as I implemented it adds 2 new variables to > > the Query structure. Rewrite rules are stored as querytrees > > and in the existing pg_rewrite entries that would be missing. > > Oh, sorry. I forgot. That could be tough. But it wouldn't hurt to add them now to have them in place. The required out-, read- and copyfuncs are in my patch too. This would prevent dump/load when we later add the real LIMIT functionality. And it does not change anything now. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #======================================== jwieck@debis.com (Jan Wieck) # From owner-pgsql-hackers@hub.org Tue Oct 20 14:57:36 1998 Received: from hub.org (majordom@hub.org [209.47.148.200]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id OAA11449 for ; Tue, 20 Oct 1998 14:57:34 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id NAA03547; Tue, 20 Oct 1998 13:10:38 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Tue, 20 Oct 1998 13:10:23 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id NAA03488 for pgsql-hackers-outgoing; Tue, 20 Oct 1998 13:10:21 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from dsh.de (firewall-user@neptun.sns-felb.debis.de [53.122.101.2]) by hub.org (8.8.8/8.8.8) with ESMTP id NAA03455 for ; Tue, 20 Oct 1998 13:10:10 -0400 (EDT) (envelope-from wieck@sapserv.debis.de) Received: by dsh.de; id TAA17171; Tue, 20 Oct 1998 19:12:30 +0200 (MET DST) Received: from dshmail.dsh.de(53.47.15.3) by neptun.dsh.de via smap (3.2) id xma017064; Tue, 20 Oct 98 19:12:00 +0200 Received: from mail1.hh1.dsh.de (mail1.hh1.dsh.de [53.47.9.5]) by dshmail.dsh.de (8.8.7/8.8.7) with ESMTP id TAA24806; Tue, 20 Oct 1998 19:09:37 +0200 (MET DST) Received: from mars.SAPserv.Hamburg.dsh.de (root@mail5.hh1.dsh.de [53.2.168.17]) by mail1.hh1.dsh.de (8.8.7/8.8.7) with SMTP id TAA06212; Tue, 20 Oct 1998 19:12:01 +0200 Received: from orion.SAPserv.Hamburg.dsh.de by mars.SAPserv.Hamburg.dsh.de with smtp for <> id m0zVcwS-000B5AC; Tue, 20 Oct 98 16:39 MET DST Received: by orion.SAPserv.Hamburg.dsh.de for maillist@candle.pha.pa.us id m0zVfUW-000EBPC; Tue, 20 Oct 98 19:22 MET DST Message-Id: From: jwieck@debis.com (Jan Wieck) Subject: Re: [HACKERS] What about LIMIT in SELECT ? To: maillist@candle.pha.pa.us (Bruce Momjian) Date: Tue, 20 Oct 1998 19:22:40 +0200 (MET DST) Cc: jwieck@debis.com, Inoue@tpf.co.jp, hackers@postgreSQL.org Reply-To: jwieck@debis.com (Jan Wieck) In-Reply-To: <199810201702.NAA08286@candle.pha.pa.us> from "Bruce Momjian" at Oct 20, 98 01:02:58 pm X-Mailer: ELM [version 2.4 PL25] Content-Type: text Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: ROr > > > > > > > I agree. Another good thing is that the LIMIT thing will not require a > > > dump/reload, so it is a good candidate for a minor release. > > > > That's wrong, sorry. > > > > The limit thing as I implemented it adds 2 new variables to > > the Query structure. Rewrite rules are stored as querytrees > > and in the existing pg_rewrite entries that would be missing. > > Oh, sorry. I forgot. That could be tough. But it wouldn't hurt to add them now to have them in place. The required out-, read- and copyfuncs are in my patch too. This would prevent dump/load when we later add the real LIMIT functionality. And it does not change anything now. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #======================================== jwieck@debis.com (Jan Wieck) # From owner-pgsql-hackers@hub.org Wed Oct 21 02:35:54 1998 Received: from hub.org (majordom@hub.org [209.47.148.200]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id CAA29494 for ; Wed, 21 Oct 1998 02:35:53 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.8.8) with SMTP id CAA13326; Wed, 21 Oct 1998 02:10:42 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 21 Oct 1998 02:09:35 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.8.8/8.8.8) id CAA12900 for pgsql-hackers-outgoing; Wed, 21 Oct 1998 02:09:33 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from candle.pha.pa.us (maillist@s5-03.ppp.op.net [209.152.195.67]) by hub.org (8.8.8/8.8.8) with ESMTP id CAA12871 for ; Wed, 21 Oct 1998 02:09:26 -0400 (EDT) (envelope-from maillist@candle.pha.pa.us) Received: (from maillist@localhost) by candle.pha.pa.us (8.9.0/8.9.0) id CAA27774; Wed, 21 Oct 1998 02:09:27 -0400 (EDT) From: Bruce Momjian Message-Id: <199810210609.CAA27774@candle.pha.pa.us> Subject: Re: [HACKERS] What about LIMIT in SELECT ? In-Reply-To: from Jan Wieck at "Oct 20, 1998 7:22:40 pm" To: jwieck@debis.com Date: Wed, 21 Oct 1998 02:09:26 -0400 (EDT) Cc: jwieck@debis.com, Inoue@tpf.co.jp, hackers@postgreSQL.org X-Mailer: ELM [version 2.4ME+ PL47 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: RO > > > > > > > > > > I agree. Another good thing is that the LIMIT thing will not require a > > > > dump/reload, so it is a good candidate for a minor release. > > > > > > That's wrong, sorry. > > > > > > The limit thing as I implemented it adds 2 new variables to > > > the Query structure. Rewrite rules are stored as querytrees > > > and in the existing pg_rewrite entries that would be missing. > > > > Oh, sorry. I forgot. That could be tough. > > But it wouldn't hurt to add them now to have them in > place. The required out-, read- and copyfuncs are in > my patch too. This would prevent dump/load when we > later add the real LIMIT functionality. And it does > not change anything now. > Jan, we found that I am having to require an initdb for the INET/CIDR type, so if you want stuff to change the views/rules for the limit addition post 6.4, please send them in and I will apply them. You clearly have the syntax down, so I think you should go ahead. -- Bruce Momjian | http://www.op.net/~candle maillist@candle.pha.pa.us | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania 19026 From wieck@sapserv.debis.de Thu Oct 22 10:20:58 1998 Received: from dsh.de (firewall-user@neptun.sns-felb.debis.de [53.122.101.2]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id KAA20566 for ; Thu, 22 Oct 1998 10:20:54 -0400 (EDT) Received: by dsh.de; id QAA09067; Thu, 22 Oct 1998 16:23:14 +0200 (MET DST) Received: from dshmail.dsh.de(53.47.15.3) by neptun.dsh.de via smap (3.2) id xma008719; Thu, 22 Oct 98 16:22:40 +0200 Received: from mail1.hh1.dsh.de (mail1.hh1.dsh.de [53.47.9.5]) by dshmail.dsh.de (8.8.7/8.8.7) with ESMTP id QAA01558; Thu, 22 Oct 1998 16:19:55 +0200 (MET DST) Received: from mars.SAPserv.Hamburg.dsh.de (root@mail5.hh1.dsh.de [53.2.168.17]) by mail1.hh1.dsh.de (8.8.7/8.8.7) with SMTP id QAA18978; Thu, 22 Oct 1998 16:22:20 +0200 Received: from orion.SAPserv.Hamburg.dsh.de by mars.SAPserv.Hamburg.dsh.de with smtp for <> id m0zWJG2-000B5AC; Thu, 22 Oct 98 13:50 MET DST Received: by orion.SAPserv.Hamburg.dsh.de for maillist@candle.pha.pa.us id m0zWLoE-000EBPC; Thu, 22 Oct 98 16:33 MET DST Message-Id: From: jwieck@debis.com (Jan Wieck) Subject: Re: [HACKERS] psql's help (the LIMIT stuff) To: maillist@candle.pha.pa.us (Bruce Momjian) Date: Thu, 22 Oct 1998 16:33:50 +0200 (MET DST) Cc: jwieck@debis.com, jose@sferacarta.com, pgsql-hackers@postgreSQL.org Reply-To: jwieck@debis.com (Jan Wieck) In-Reply-To: <199810221351.JAA19663@candle.pha.pa.us> from "Bruce Momjian" at Oct 22, 98 09:51:19 am X-Mailer: ELM [version 2.4 PL25] Content-Type: text Status: ROr > > > > I hope the QUERY_LIMIT too. > > I still have that cnfify() possible fix to review for KQSO. Are you > still thinking limit for 6.4 final, or a minor release after that? I posted the part that is the minimum applied to 6.4 to make adding LIMIT later non-initdb earlier. Anyway, here it's again. My LIMIT implementation that does it like the SET in the toplevel executor (but via parsetree values) is ready for production. I only held it back because it's feature, not bugfix. Do you want it in 6.4 final? Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #======================================== jwieck@debis.com (Jan Wieck) # diff -cr src.orig/backend/nodes/copyfuncs.c src/backend/nodes/copyfuncs.c *** src.orig/backend/nodes/copyfuncs.c Fri Oct 16 11:53:40 1998 --- src/backend/nodes/copyfuncs.c Fri Oct 16 13:32:35 1998 *************** *** 1578,1583 **** --- 1578,1586 ---- newnode->unionClause = temp_list; } + Node_Copy(from, newnode, limitOffset); + Node_Copy(from, newnode, limitCount); + return newnode; } diff -cr src.orig/backend/nodes/outfuncs.c src/backend/nodes/outfuncs.c *** src.orig/backend/nodes/outfuncs.c Fri Oct 16 11:53:40 1998 --- src/backend/nodes/outfuncs.c Fri Oct 16 13:30:50 1998 *************** *** 259,264 **** --- 259,268 ---- appendStringInfo(str, (node->hasSubLinks ? "true" : "false")); appendStringInfo(str, " :unionClause "); _outNode(str, node->unionClause); + appendStringInfo(str, " :limitOffset "); + _outNode(str, node->limitOffset); + appendStringInfo(str, " :limitCount "); + _outNode(str, node->limitCount); } static void diff -cr src.orig/backend/nodes/readfuncs.c src/backend/nodes/readfuncs.c *** src.orig/backend/nodes/readfuncs.c Fri Oct 16 11:53:40 1998 --- src/backend/nodes/readfuncs.c Fri Oct 16 13:31:43 1998 *************** *** 163,168 **** --- 163,174 ---- token = lsptok(NULL, &length); /* skip :unionClause */ local_node->unionClause = nodeRead(true); + token = lsptok(NULL, &length); /* skip :limitOffset */ + local_node->limitOffset = nodeRead(true); + + token = lsptok(NULL, &length); /* skip :limitCount */ + local_node->limitCount = nodeRead(true); + return local_node; } diff -cr src.orig/include/nodes/parsenodes.h src/include/nodes/parsenodes.h *** src.orig/include/nodes/parsenodes.h Fri Oct 16 11:53:58 1998 --- src/include/nodes/parsenodes.h Fri Oct 16 13:35:32 1998 *************** *** 60,65 **** --- 60,67 ---- List *unionClause; /* unions are linked under the previous * query */ + Node *limitOffset; /* # of result tuples to skip */ + Node *limitCount; /* # of result tuples to return */ /* internal to planner */ List *base_rel_list; /* base relation list */ *************** *** 639,644 **** --- 641,648 ---- char *portalname; /* the portal (cursor) to create */ bool binary; /* a binary (internal) portal? */ bool unionall; /* union without unique sort */ + Node *limitOffset; /* # of result tuples to skip */ + Node *limitCount; /* # of result tuples to return */ } SelectStmt; From owner-pgsql-hackers@hub.org Thu Oct 22 11:33:41 1998 Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id LAA01724 for ; Thu, 22 Oct 1998 11:33:31 -0400 (EDT) Received: from hub.org (majordom@hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id LAA12702 for ; Thu, 22 Oct 1998 11:25:02 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.9.1/8.8.8) with SMTP id KAA11023; Thu, 22 Oct 1998 10:22:13 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Thu, 22 Oct 1998 10:21:07 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.9.1/8.8.8) id KAA10873 for pgsql-hackers-outgoing; Thu, 22 Oct 1998 10:21:05 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from dsh.de (firewall-user@neptun.sns-felb.debis.de [53.122.101.2]) by hub.org (8.9.1/8.8.8) with ESMTP id KAA10847 for ; Thu, 22 Oct 1998 10:21:00 -0400 (EDT) (envelope-from wieck@sapserv.debis.de) Received: by dsh.de; id QAA09067; Thu, 22 Oct 1998 16:23:14 +0200 (MET DST) Received: from dshmail.dsh.de(53.47.15.3) by neptun.dsh.de via smap (3.2) id xma008719; Thu, 22 Oct 98 16:22:40 +0200 Received: from mail1.hh1.dsh.de (mail1.hh1.dsh.de [53.47.9.5]) by dshmail.dsh.de (8.8.7/8.8.7) with ESMTP id QAA01558; Thu, 22 Oct 1998 16:19:55 +0200 (MET DST) Received: from mars.SAPserv.Hamburg.dsh.de (root@mail5.hh1.dsh.de [53.2.168.17]) by mail1.hh1.dsh.de (8.8.7/8.8.7) with SMTP id QAA18978; Thu, 22 Oct 1998 16:22:20 +0200 Received: from orion.SAPserv.Hamburg.dsh.de by mars.SAPserv.Hamburg.dsh.de with smtp for <> id m0zWJG2-000B5AC; Thu, 22 Oct 98 13:50 MET DST Received: by orion.SAPserv.Hamburg.dsh.de for maillist@candle.pha.pa.us id m0zWLoE-000EBPC; Thu, 22 Oct 98 16:33 MET DST Message-Id: From: jwieck@debis.com (Jan Wieck) Subject: Re: [HACKERS] psql's help (the LIMIT stuff) To: maillist@candle.pha.pa.us (Bruce Momjian) Date: Thu, 22 Oct 1998 16:33:50 +0200 (MET DST) Cc: jwieck@debis.com, jose@sferacarta.com, pgsql-hackers@postgreSQL.org Reply-To: jwieck@debis.com (Jan Wieck) In-Reply-To: <199810221351.JAA19663@candle.pha.pa.us> from "Bruce Momjian" at Oct 22, 98 09:51:19 am X-Mailer: ELM [version 2.4 PL25] Content-Type: text Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: RO > > > > I hope the QUERY_LIMIT too. > > I still have that cnfify() possible fix to review for KQSO. Are you > still thinking limit for 6.4 final, or a minor release after that? I posted the part that is the minimum applied to 6.4 to make adding LIMIT later non-initdb earlier. Anyway, here it's again. My LIMIT implementation that does it like the SET in the toplevel executor (but via parsetree values) is ready for production. I only held it back because it's feature, not bugfix. Do you want it in 6.4 final? Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #======================================== jwieck@debis.com (Jan Wieck) # diff -cr src.orig/backend/nodes/copyfuncs.c src/backend/nodes/copyfuncs.c *** src.orig/backend/nodes/copyfuncs.c Fri Oct 16 11:53:40 1998 --- src/backend/nodes/copyfuncs.c Fri Oct 16 13:32:35 1998 *************** *** 1578,1583 **** --- 1578,1586 ---- newnode->unionClause = temp_list; } + Node_Copy(from, newnode, limitOffset); + Node_Copy(from, newnode, limitCount); + return newnode; } diff -cr src.orig/backend/nodes/outfuncs.c src/backend/nodes/outfuncs.c *** src.orig/backend/nodes/outfuncs.c Fri Oct 16 11:53:40 1998 --- src/backend/nodes/outfuncs.c Fri Oct 16 13:30:50 1998 *************** *** 259,264 **** --- 259,268 ---- appendStringInfo(str, (node->hasSubLinks ? "true" : "false")); appendStringInfo(str, " :unionClause "); _outNode(str, node->unionClause); + appendStringInfo(str, " :limitOffset "); + _outNode(str, node->limitOffset); + appendStringInfo(str, " :limitCount "); + _outNode(str, node->limitCount); } static void diff -cr src.orig/backend/nodes/readfuncs.c src/backend/nodes/readfuncs.c *** src.orig/backend/nodes/readfuncs.c Fri Oct 16 11:53:40 1998 --- src/backend/nodes/readfuncs.c Fri Oct 16 13:31:43 1998 *************** *** 163,168 **** --- 163,174 ---- token = lsptok(NULL, &length); /* skip :unionClause */ local_node->unionClause = nodeRead(true); + token = lsptok(NULL, &length); /* skip :limitOffset */ + local_node->limitOffset = nodeRead(true); + + token = lsptok(NULL, &length); /* skip :limitCount */ + local_node->limitCount = nodeRead(true); + return local_node; } diff -cr src.orig/include/nodes/parsenodes.h src/include/nodes/parsenodes.h *** src.orig/include/nodes/parsenodes.h Fri Oct 16 11:53:58 1998 --- src/include/nodes/parsenodes.h Fri Oct 16 13:35:32 1998 *************** *** 60,65 **** --- 60,67 ---- List *unionClause; /* unions are linked under the previous * query */ + Node *limitOffset; /* # of result tuples to skip */ + Node *limitCount; /* # of result tuples to return */ /* internal to planner */ List *base_rel_list; /* base relation list */ *************** *** 639,644 **** --- 641,648 ---- char *portalname; /* the portal (cursor) to create */ bool binary; /* a binary (internal) portal? */ bool unionall; /* union without unique sort */ + Node *limitOffset; /* # of result tuples to skip */ + Node *limitCount; /* # of result tuples to return */ } SelectStmt; From wieck@sapserv.debis.de Thu Oct 22 11:01:05 1998 Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id LAA21185 for ; Thu, 22 Oct 1998 11:01:00 -0400 (EDT) Received: from dsh.de (firewall-user@neptun.sns-felb.debis.de [53.122.101.2]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id KAA09646 for ; Thu, 22 Oct 1998 10:44:36 -0400 (EDT) Received: by dsh.de; id QAA19394; Thu, 22 Oct 1998 16:43:42 +0200 (MET DST) Received: from dshmail.dsh.de(53.47.15.3) by neptun.dsh.de via smap (3.2) id xma017268; Thu, 22 Oct 98 16:39:44 +0200 Received: from mail1.hh1.dsh.de (mail1.hh1.dsh.de [53.47.9.5]) by dshmail.dsh.de (8.8.7/8.8.7) with ESMTP id QAA02988; Thu, 22 Oct 1998 16:36:46 +0200 (MET DST) Received: from mars.SAPserv.Hamburg.dsh.de (root@mail5.hh1.dsh.de [53.2.168.17]) by mail1.hh1.dsh.de (8.8.7/8.8.7) with SMTP id QAA19155; Thu, 22 Oct 1998 16:39:10 +0200 Received: from orion.SAPserv.Hamburg.dsh.de by mars.SAPserv.Hamburg.dsh.de with smtp for <> id m0zWJWL-000B5DC; Thu, 22 Oct 98 14:07 MET DST Received: by orion.SAPserv.Hamburg.dsh.de for maillist@candle.pha.pa.us id m0zWM4W-000EBPC; Thu, 22 Oct 98 16:50 MET DST Message-Id: From: jwieck@debis.com (Jan Wieck) Subject: Re: [HACKERS] psql's help (the LIMIT stuff) To: maillist@candle.pha.pa.us (Bruce Momjian) Date: Thu, 22 Oct 1998 16:50:40 +0200 (MET DST) Cc: jwieck@debis.com, jose@sferacarta.com, pgsql-hackers@postgreSQL.org Reply-To: jwieck@debis.com (Jan Wieck) In-Reply-To: <199810221424.KAA20601@candle.pha.pa.us> from "Bruce Momjian" at Oct 22, 98 10:24:08 am X-Mailer: ELM [version 2.4 PL25] Content-Type: text Status: RO > > > > > > > > > I hope the QUERY_LIMIT too. > > > > > > I still have that cnfify() possible fix to review for KQSO. Are you > > > still thinking limit for 6.4 final, or a minor release after that? > > > > I posted the part that is the minimum applied to 6.4 to make > > adding LIMIT later non-initdb earlier. Anyway, here it's > > again. > > Already applied. I assume it is the same as the one I applied. Seen, thanks. Your 'Applied' just arrived after I packed it again. It's the same. > We are close to final, and can easily put it in 6.4.1, which I am sure > we will need, and if we split CVS trees, you'll have lots of minor > versions to pick from. :-) > > Seems like it would be a nice minor release item, but the problem is > that minor releases aren't tested as much as major ones. How confident > are you in the code? What do others thing? I regression tested it, and did additional tests in the SPI/PL area. It works. It only touches the parser and the executor. Rules, planner/optimizer just bypass the values in the parsetree. The parser and the executor are parts of Postgres I feel very familiar with (not so in the optimizer). I trust in the code and would use it in a production environment. It's below. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #======================================== jwieck@debis.com (Jan Wieck) # diff -cr src.orig/backend/commands/command.c src/backend/commands/command.c *** src.orig/backend/commands/command.c Fri Oct 16 11:53:38 1998 --- src/backend/commands/command.c Fri Oct 16 12:56:44 1998 *************** *** 39,44 **** --- 39,45 ---- #include "utils/mcxt.h" #include "utils/portal.h" #include "utils/syscache.h" + #include "string.h" /* ---------------- * PortalExecutorHeapMemory stuff *************** *** 101,106 **** --- 102,108 ---- int feature; QueryDesc *queryDesc; MemoryContext context; + Const limcount; /* ---------------- * sanity checks *************** *** 113,118 **** --- 115,134 ---- } /* ---------------- + * Create a const node from the given count value + * ---------------- + */ + memset(&limcount, 0, sizeof(limcount)); + limcount.type = T_Const; + limcount.consttype = INT4OID; + limcount.constlen = sizeof(int4); + limcount.constvalue = (Datum)count; + limcount.constisnull = FALSE; + limcount.constbyval = TRUE; + limcount.constisset = FALSE; + limcount.constiscast = FALSE; + + /* ---------------- * get the portal from the portal name * ---------------- */ *************** *** 176,182 **** PortalExecutorHeapMemory = (MemoryContext) PortalGetHeapMemory(portal); ! ExecutorRun(queryDesc, PortalGetState(portal), feature, count); if (dest == None) /* MOVE */ pfree(queryDesc); --- 192,198 ---- PortalExecutorHeapMemory = (MemoryContext) PortalGetHeapMemory(portal); ! ExecutorRun(queryDesc, PortalGetState(portal), feature, (Node *)NULL, (Node *)&limcount); if (dest == None) /* MOVE */ pfree(queryDesc); diff -cr src.orig/backend/executor/execMain.c src/backend/executor/execMain.c *** src.orig/backend/executor/execMain.c Fri Oct 16 11:53:38 1998 --- src/backend/executor/execMain.c Fri Oct 16 20:05:19 1998 *************** *** 64,69 **** --- 64,70 ---- static void EndPlan(Plan *plan, EState *estate); static TupleTableSlot *ExecutePlan(EState *estate, Plan *plan, Query *parseTree, CmdType operation, + int offsetTuples, int numberTuples, ScanDirection direction, void (*printfunc) ()); static void ExecRetrieve(TupleTableSlot *slot, void (*printfunc) (), *************** *** 163,169 **** * ---------------------------------------------------------------- */ TupleTableSlot * ! ExecutorRun(QueryDesc *queryDesc, EState *estate, int feature, int count) { CmdType operation; Query *parseTree; --- 164,170 ---- * ---------------------------------------------------------------- */ TupleTableSlot * ! ExecutorRun(QueryDesc *queryDesc, EState *estate, int feature, Node *limoffset, Node *limcount) { CmdType operation; Query *parseTree; *************** *** 171,176 **** --- 172,179 ---- TupleTableSlot *result; CommandDest dest; void (*destination) (); + int offset = 0; + int count = 0; /****************** * sanity checks *************** *** 191,196 **** --- 194,289 ---- estate->es_processed = 0; estate->es_lastoid = InvalidOid; + /****************** + * if given get the offset of the LIMIT clause + ****************** + */ + if (limoffset != NULL) + { + Const *coffset; + Param *poffset; + ParamListInfo paramLI; + int i; + + switch (nodeTag(limoffset)) + { + case T_Const: + coffset = (Const *)limoffset; + offset = (int)(coffset->constvalue); + break; + + case T_Param: + poffset = (Param *)limoffset; + paramLI = estate->es_param_list_info; + + if (paramLI == NULL) + elog(ERROR, "parameter for limit offset not in executor state"); + for (i = 0; paramLI[i].kind != PARAM_INVALID; i++) + { + if (paramLI[i].kind == PARAM_NUM && paramLI[i].id == poffset->paramid) + break; + } + if (paramLI[i].kind == PARAM_INVALID) + elog(ERROR, "parameter for limit offset not in executor state"); + if (paramLI[i].isnull) + elog(ERROR, "limit offset cannot be NULL value"); + offset = (int)(paramLI[i].value); + + break; + + default: + elog(ERROR, "unexpected node type %d as limit offset", nodeTag(limoffset)); + } + + if (offset < 0) + elog(ERROR, "limit offset cannot be negative"); + } + + /****************** + * if given get the count of the LIMIT clause + ****************** + */ + if (limcount != NULL) + { + Const *ccount; + Param *pcount; + ParamListInfo paramLI; + int i; + + switch (nodeTag(limcount)) + { + case T_Const: + ccount = (Const *)limcount; + count = (int)(ccount->constvalue); + break; + + case T_Param: + pcount = (Param *)limcount; + paramLI = estate->es_param_list_info; + + if (paramLI == NULL) + elog(ERROR, "parameter for limit count not in executor state"); + for (i = 0; paramLI[i].kind != PARAM_INVALID; i++) + { + if (paramLI[i].kind == PARAM_NUM && paramLI[i].id == pcount->paramid) + break; + } + if (paramLI[i].kind == PARAM_INVALID) + elog(ERROR, "parameter for limit count not in executor state"); + if (paramLI[i].isnull) + elog(ERROR, "limit count cannot be NULL value"); + count = (int)(paramLI[i].value); + + break; + + default: + elog(ERROR, "unexpected node type %d as limit count", nodeTag(limcount)); + } + + if (count < 0) + elog(ERROR, "limit count cannot be negative"); + } + switch (feature) { *************** *** 199,205 **** plan, parseTree, operation, ! ALL_TUPLES, ForwardScanDirection, destination); break; --- 292,299 ---- plan, parseTree, operation, ! offset, ! count, ForwardScanDirection, destination); break; *************** *** 208,213 **** --- 302,308 ---- plan, parseTree, operation, + offset, count, ForwardScanDirection, destination); *************** *** 222,227 **** --- 317,323 ---- plan, parseTree, operation, + offset, count, BackwardScanDirection, destination); *************** *** 237,242 **** --- 333,339 ---- plan, parseTree, operation, + 0, ONE_TUPLE, ForwardScanDirection, destination); *************** *** 691,696 **** --- 788,794 ---- Plan *plan, Query *parseTree, CmdType operation, + int offsetTuples, int numberTuples, ScanDirection direction, void (*printfunc) ()) *************** *** 742,747 **** --- 840,859 ---- { result = NULL; break; + } + + /****************** + * For now we completely execute the plan and skip + * result tuples if requested by LIMIT offset. + * Finally we should try to do it in deeper levels + * if possible (during index scan) + * - Jan + ****************** + */ + if (offsetTuples > 0) + { + --offsetTuples; + continue; } /****************** diff -cr src.orig/backend/executor/functions.c src/backend/executor/functions.c *** src.orig/backend/executor/functions.c Fri Oct 16 11:53:38 1998 --- src/backend/executor/functions.c Fri Oct 16 19:01:02 1998 *************** *** 130,135 **** --- 130,138 ---- None); estate = CreateExecutorState(); + if (queryTree->limitOffset != NULL || queryTree->limitCount != NULL) + elog(ERROR, "LIMIT clause from SQL functions not yet implemented"); + if (nargs > 0) { int i; *************** *** 200,206 **** feature = (LAST_POSTQUEL_COMMAND(es)) ? EXEC_RETONE : EXEC_RUN; ! return ExecutorRun(es->qd, es->estate, feature, 0); } static void --- 203,209 ---- feature = (LAST_POSTQUEL_COMMAND(es)) ? EXEC_RETONE : EXEC_RUN; ! return ExecutorRun(es->qd, es->estate, feature, (Node *)NULL, (Node *)NULL); } static void diff -cr src.orig/backend/executor/spi.c src/backend/executor/spi.c *** src.orig/backend/executor/spi.c Fri Oct 16 11:53:39 1998 --- src/backend/executor/spi.c Fri Oct 16 19:25:33 1998 *************** *** 791,796 **** --- 791,798 ---- bool isRetrieveIntoRelation = false; char *intoName = NULL; int res; + Const tcount_const; + Node *count = NULL; switch (operation) { *************** *** 825,830 **** --- 827,865 ---- return SPI_ERROR_OPUNKNOWN; } + /* ---------------- + * Get the query LIMIT tuple count + * ---------------- + */ + if (parseTree->limitCount != NULL) + { + /* ---------------- + * A limit clause in the parsetree overrides the + * tcount parameter + * ---------------- + */ + count = parseTree->limitCount; + } + else + { + /* ---------------- + * No LIMIT clause in parsetree. Use a local Const node + * to put tcount into it + * ---------------- + */ + memset(&tcount_const, 0, sizeof(tcount_const)); + tcount_const.type = T_Const; + tcount_const.consttype = INT4OID; + tcount_const.constlen = sizeof(int4); + tcount_const.constvalue = (Datum)tcount; + tcount_const.constisnull = FALSE; + tcount_const.constbyval = TRUE; + tcount_const.constisset = FALSE; + tcount_const.constiscast = FALSE; + + count = (Node *)&tcount_const; + } + if (state == NULL) /* plan preparation */ return res; #ifdef SPI_EXECUTOR_STATS *************** *** 845,851 **** return SPI_OK_CURSOR; } ! ExecutorRun(queryDesc, state, EXEC_FOR, tcount); _SPI_current->processed = state->es_processed; if (operation == CMD_SELECT && queryDesc->dest == SPI) --- 880,886 ---- return SPI_OK_CURSOR; } ! ExecutorRun(queryDesc, state, EXEC_FOR, parseTree->limitOffset, count); _SPI_current->processed = state->es_processed; if (operation == CMD_SELECT && queryDesc->dest == SPI) diff -cr src.orig/backend/parser/analyze.c src/backend/parser/analyze.c *** src.orig/backend/parser/analyze.c Fri Oct 16 11:53:41 1998 --- src/backend/parser/analyze.c Fri Oct 16 13:29:27 1998 *************** *** 180,186 **** --- 180,190 ---- case T_SelectStmt: if (!((SelectStmt *) parseTree)->portalname) + { result = transformSelectStmt(pstate, (SelectStmt *) parseTree); + result->limitOffset = ((SelectStmt *)parseTree)->limitOffset; + result->limitCount = ((SelectStmt *)parseTree)->limitCount; + } else result = transformCursorStmt(pstate, (SelectStmt *) parseTree); break; diff -cr src.orig/backend/parser/gram.y src/backend/parser/gram.y *** src.orig/backend/parser/gram.y Fri Oct 16 11:53:42 1998 --- src/backend/parser/gram.y Sun Oct 18 22:20:36 1998 *************** *** 45,50 **** --- 45,51 ---- #include "catalog/catname.h" #include "utils/elog.h" #include "access/xact.h" + #include "catalog/pg_type.h" #ifdef MULTIBYTE #include "mb/pg_wchar.h" *************** *** 163,169 **** sort_clause, sortby_list, index_params, index_list, name_list, from_clause, from_list, opt_array_bounds, nest_array_bounds, expr_list, attrs, res_target_list, res_target_list2, ! def_list, opt_indirection, group_clause, TriggerFuncArgs %type func_return %type set_opt --- 164,171 ---- sort_clause, sortby_list, index_params, index_list, name_list, from_clause, from_list, opt_array_bounds, nest_array_bounds, expr_list, attrs, res_target_list, res_target_list2, ! def_list, opt_indirection, group_clause, TriggerFuncArgs, ! opt_select_limit %type func_return %type set_opt *************** *** 192,197 **** --- 194,201 ---- %type fetch_how_many + %type select_limit_value select_offset_value + %type OptSeqList %type OptSeqElem *************** *** 267,273 **** FALSE_P, FETCH, FLOAT, FOR, FOREIGN, FROM, FULL, GRANT, GROUP, HAVING, HOUR_P, IN, INNER_P, INSENSITIVE, INSERT, INTERVAL, INTO, IS, ! JOIN, KEY, LANGUAGE, LEADING, LEFT, LIKE, LOCAL, MATCH, MINUTE_P, MONTH_P, NAMES, NATIONAL, NATURAL, NCHAR, NEXT, NO, NOT, NULL_P, NUMERIC, OF, ON, ONLY, OPTION, OR, ORDER, OUTER_P, --- 271,277 ---- FALSE_P, FETCH, FLOAT, FOR, FOREIGN, FROM, FULL, GRANT, GROUP, HAVING, HOUR_P, IN, INNER_P, INSENSITIVE, INSERT, INTERVAL, INTO, IS, ! JOIN, KEY, LANGUAGE, LEADING, LEFT, LIKE, LIMIT, LOCAL, MATCH, MINUTE_P, MONTH_P, NAMES, NATIONAL, NATURAL, NCHAR, NEXT, NO, NOT, NULL_P, NUMERIC, OF, ON, ONLY, OPTION, OR, ORDER, OUTER_P, *************** *** 299,305 **** INCREMENT, INDEX, INHERITS, INSTEAD, ISNULL, LANCOMPILER, LISTEN, LOAD, LOCATION, LOCK_P, MAXVALUE, MINVALUE, MOVE, NEW, NOCREATEDB, NOCREATEUSER, NONE, NOTHING, NOTIFY, NOTNULL, ! OIDS, OPERATOR, PASSWORD, PROCEDURAL, RECIPE, RENAME, RESET, RETURNS, ROW, RULE, SEQUENCE, SERIAL, SETOF, SHOW, START, STATEMENT, STDIN, STDOUT, TRUSTED, UNLISTEN, UNTIL, VACUUM, VALID, VERBOSE, VERSION --- 303,309 ---- INCREMENT, INDEX, INHERITS, INSTEAD, ISNULL, LANCOMPILER, LISTEN, LOAD, LOCATION, LOCK_P, MAXVALUE, MINVALUE, MOVE, NEW, NOCREATEDB, NOCREATEUSER, NONE, NOTHING, NOTIFY, NOTNULL, ! OFFSET, OIDS, OPERATOR, PASSWORD, PROCEDURAL, RECIPE, RENAME, RESET, RETURNS, ROW, RULE, SEQUENCE, SERIAL, SETOF, SHOW, START, STATEMENT, STDIN, STDOUT, TRUSTED, UNLISTEN, UNTIL, VACUUM, VALID, VERBOSE, VERSION *************** *** 2591,2596 **** --- 2595,2601 ---- result from_clause where_clause group_clause having_clause union_clause sort_clause + opt_select_limit { SelectStmt *n = makeNode(SelectStmt); n->unique = $2; *************** *** 2602,2607 **** --- 2607,2622 ---- n->havingClause = $8; n->unionClause = $9; n->sortClause = $10; + if ($11 != NIL) + { + n->limitOffset = nth(0, $11); + n->limitCount = nth(1, $11); + } + else + { + n->limitOffset = NULL; + n->limitCount = NULL; + } $$ = (Node *)n; } ; *************** *** 2699,2704 **** --- 2714,2794 ---- | ASC { $$ = "<"; } | DESC { $$ = ">"; } | /*EMPTY*/ { $$ = "<"; /*default*/ } + ; + + opt_select_limit: LIMIT select_offset_value ',' select_limit_value + { $$ = lappend(lappend(NIL, $2), $4); } + | LIMIT select_limit_value OFFSET select_offset_value + { $$ = lappend(lappend(NIL, $4), $2); } + | LIMIT select_limit_value + { $$ = lappend(lappend(NIL, NULL), $2); } + | OFFSET select_offset_value LIMIT select_limit_value + { $$ = lappend(lappend(NIL, $2), $4); } + | OFFSET select_offset_value + { $$ = lappend(lappend(NIL, $2), NULL); } + | /* EMPTY */ + { $$ = NIL; } + ; + + select_limit_value: Iconst + { + Const *n = makeNode(Const); + + if ($1 < 1) + elog(ERROR, "selection limit must be ALL or a positive integer value > 0"); + + n->consttype = INT4OID; + n->constlen = sizeof(int4); + n->constvalue = (Datum)$1; + n->constisnull = FALSE; + n->constbyval = TRUE; + n->constisset = FALSE; + n->constiscast = FALSE; + $$ = (Node *)n; + } + | ALL + { + Const *n = makeNode(Const); + n->consttype = INT4OID; + n->constlen = sizeof(int4); + n->constvalue = (Datum)0; + n->constisnull = FALSE; + n->constbyval = TRUE; + n->constisset = FALSE; + n->constiscast = FALSE; + $$ = (Node *)n; + } + | PARAM + { + Param *n = makeNode(Param); + n->paramkind = PARAM_NUM; + n->paramid = $1; + n->paramtype = INT4OID; + $$ = (Node *)n; + } + ; + + select_offset_value: Iconst + { + Const *n = makeNode(Const); + + n->consttype = INT4OID; + n->constlen = sizeof(int4); + n->constvalue = (Datum)$1; + n->constisnull = FALSE; + n->constbyval = TRUE; + n->constisset = FALSE; + n->constiscast = FALSE; + $$ = (Node *)n; + } + | PARAM + { + Param *n = makeNode(Param); + n->paramkind = PARAM_NUM; + n->paramid = $1; + n->paramtype = INT4OID; + $$ = (Node *)n; + } ; /* diff -cr src.orig/backend/parser/keywords.c src/backend/parser/keywords.c *** src.orig/backend/parser/keywords.c Fri Oct 16 11:53:42 1998 --- src/backend/parser/keywords.c Sun Oct 18 22:13:29 1998 *************** *** 128,133 **** --- 128,134 ---- {"leading", LEADING}, {"left", LEFT}, {"like", LIKE}, + {"limit", LIMIT}, {"listen", LISTEN}, {"load", LOAD}, {"local", LOCAL}, *************** *** 156,161 **** --- 157,163 ---- {"null", NULL_P}, {"numeric", NUMERIC}, {"of", OF}, + {"offset", OFFSET}, {"oids", OIDS}, {"old", CURRENT}, {"on", ON}, diff -cr src.orig/backend/rewrite/rewriteDefine.c src/backend/rewrite/rewriteDefine.c *** src.orig/backend/rewrite/rewriteDefine.c Fri Oct 16 11:53:46 1998 --- src/backend/rewrite/rewriteDefine.c Fri Oct 16 13:48:55 1998 *************** *** 312,317 **** --- 312,323 ---- heap_close(event_relation); /* + * LIMIT in view is not supported + */ + if (query->limitOffset != NULL || query->limitCount != NULL) + elog(ERROR, "LIMIT clause not supported in views"); + + /* * ... and finally the rule must be named _RETviewname. */ sprintf(expected_name, "_RET%s", event_obj->relname); diff -cr src.orig/backend/tcop/pquery.c src/backend/tcop/pquery.c *** src.orig/backend/tcop/pquery.c Fri Oct 16 11:53:47 1998 --- src/backend/tcop/pquery.c Fri Oct 16 14:02:36 1998 *************** *** 40,46 **** #include "commands/command.h" static char *CreateOperationTag(int operationType); ! static void ProcessQueryDesc(QueryDesc *queryDesc); /* ---------------------------------------------------------------- --- 40,46 ---- #include "commands/command.h" static char *CreateOperationTag(int operationType); ! static void ProcessQueryDesc(QueryDesc *queryDesc, Node *limoffset, Node *limcount); /* ---------------------------------------------------------------- *************** *** 205,211 **** * ---------------------------------------------------------------- */ static void ! ProcessQueryDesc(QueryDesc *queryDesc) { Query *parseTree; Plan *plan; --- 205,211 ---- * ---------------------------------------------------------------- */ static void ! ProcessQueryDesc(QueryDesc *queryDesc, Node *limoffset, Node *limcount) { Query *parseTree; Plan *plan; *************** *** 330,336 **** * actually run the plan.. * ---------------- */ ! ExecutorRun(queryDesc, state, EXEC_RUN, 0); /* save infos for EndCommand */ UpdateCommandInfo(operation, state->es_lastoid, state->es_processed); --- 330,336 ---- * actually run the plan.. * ---------------- */ ! ExecutorRun(queryDesc, state, EXEC_RUN, limoffset, limcount); /* save infos for EndCommand */ UpdateCommandInfo(operation, state->es_lastoid, state->es_processed); *************** *** 373,377 **** print_plan(plan, parsetree); } else ! ProcessQueryDesc(queryDesc); } --- 373,377 ---- print_plan(plan, parsetree); } else ! ProcessQueryDesc(queryDesc, parsetree->limitOffset, parsetree->limitCount); } diff -cr src.orig/include/executor/executor.h src/include/executor/executor.h *** src.orig/include/executor/executor.h Fri Oct 16 11:53:56 1998 --- src/include/executor/executor.h Fri Oct 16 12:04:17 1998 *************** *** 83,89 **** * prototypes from functions in execMain.c */ extern TupleDesc ExecutorStart(QueryDesc *queryDesc, EState *estate); ! extern TupleTableSlot *ExecutorRun(QueryDesc *queryDesc, EState *estate, int feature, int count); extern void ExecutorEnd(QueryDesc *queryDesc, EState *estate); extern HeapTuple ExecConstraints(char *caller, Relation rel, HeapTuple tuple); #ifdef QUERY_LIMIT --- 83,89 ---- * prototypes from functions in execMain.c */ extern TupleDesc ExecutorStart(QueryDesc *queryDesc, EState *estate); ! extern TupleTableSlot *ExecutorRun(QueryDesc *queryDesc, EState *estate, int feature, Node *limoffset, Node *limcount); extern void ExecutorEnd(QueryDesc *queryDesc, EState *estate); extern HeapTuple ExecConstraints(char *caller, Relation rel, HeapTuple tuple); #ifdef QUERY_LIMIT From owner-pgsql-hackers@hub.org Thu Oct 22 13:12:34 1998 Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id NAA01350 for ; Thu, 22 Oct 1998 13:12:29 -0400 (EDT) Received: from hub.org (majordom@hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.6 $) with ESMTP id MAA17808 for ; Thu, 22 Oct 1998 12:35:22 -0400 (EDT) Received: from localhost (majordom@localhost) by hub.org (8.9.1/8.8.8) with SMTP id KAA14887; Thu, 22 Oct 1998 10:49:09 -0400 (EDT) (envelope-from owner-pgsql-hackers@hub.org) Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Thu, 22 Oct 1998 10:44:59 +0000 (EDT) Received: (from majordom@localhost) by hub.org (8.9.1/8.8.8) id KAA14445 for pgsql-hackers-outgoing; Thu, 22 Oct 1998 10:44:57 -0400 (EDT) (envelope-from owner-pgsql-hackers@postgreSQL.org) X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f Received: from dsh.de (firewall-user@neptun.sns-felb.debis.de [53.122.101.2]) by hub.org (8.9.1/8.8.8) with ESMTP id KAA14431 for ; Thu, 22 Oct 1998 10:44:47 -0400 (EDT) (envelope-from wieck@sapserv.debis.de) Received: by dsh.de; id QAA19394; Thu, 22 Oct 1998 16:43:42 +0200 (MET DST) Received: from dshmail.dsh.de(53.47.15.3) by neptun.dsh.de via smap (3.2) id xma017268; Thu, 22 Oct 98 16:39:44 +0200 Received: from mail1.hh1.dsh.de (mail1.hh1.dsh.de [53.47.9.5]) by dshmail.dsh.de (8.8.7/8.8.7) with ESMTP id QAA02988; Thu, 22 Oct 1998 16:36:46 +0200 (MET DST) Received: from mars.SAPserv.Hamburg.dsh.de (root@mail5.hh1.dsh.de [53.2.168.17]) by mail1.hh1.dsh.de (8.8.7/8.8.7) with SMTP id QAA19155; Thu, 22 Oct 1998 16:39:10 +0200 Received: from orion.SAPserv.Hamburg.dsh.de by mars.SAPserv.Hamburg.dsh.de with smtp for <> id m0zWJWL-000B5DC; Thu, 22 Oct 98 14:07 MET DST Received: by orion.SAPserv.Hamburg.dsh.de for maillist@candle.pha.pa.us id m0zWM4W-000EBPC; Thu, 22 Oct 98 16:50 MET DST Message-Id: From: jwieck@debis.com (Jan Wieck) Subject: Re: [HACKERS] psql's help (the LIMIT stuff) To: maillist@candle.pha.pa.us (Bruce Momjian) Date: Thu, 22 Oct 1998 16:50:40 +0200 (MET DST) Cc: jwieck@debis.com, jose@sferacarta.com, pgsql-hackers@postgreSQL.org Reply-To: jwieck@debis.com (Jan Wieck) In-Reply-To: <199810221424.KAA20601@candle.pha.pa.us> from "Bruce Momjian" at Oct 22, 98 10:24:08 am X-Mailer: ELM [version 2.4 PL25] Content-Type: text Sender: owner-pgsql-hackers@postgreSQL.org Precedence: bulk Status: ROr > > > > > > > > > I hope the QUERY_LIMIT too. > > > > > > I still have that cnfify() possible fix to review for KQSO. Are you > > > still thinking limit for 6.4 final, or a minor release after that? > > > > I posted the part that is the minimum applied to 6.4 to make > > adding LIMIT later non-initdb earlier. Anyway, here it's > > again. > > Already applied. I assume it is the same as the one I applied. Seen, thanks. Your 'Applied' just arrived after I packed it again. It's the same. > We are close to final, and can easily put it in 6.4.1, which I am sure > we will need, and if we split CVS trees, you'll have lots of minor > versions to pick from. :-) > > Seems like it would be a nice minor release item, but the problem is > that minor releases aren't tested as much as major ones. How confident > are you in the code? What do others thing? I regression tested it, and did additional tests in the SPI/PL area. It works. It only touches the parser and the executor. Rules, planner/optimizer just bypass the values in the parsetree. The parser and the executor are parts of Postgres I feel very familiar with (not so in the optimizer). I trust in the code and would use it in a production environment. It's below. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #======================================== jwieck@debis.com (Jan Wieck) # diff -cr src.orig/backend/commands/command.c src/backend/commands/command.c *** src.orig/backend/commands/command.c Fri Oct 16 11:53:38 1998 --- src/backend/commands/command.c Fri Oct 16 12:56:44 1998 *************** *** 39,44 **** --- 39,45 ---- #include "utils/mcxt.h" #include "utils/portal.h" #include "utils/syscache.h" + #include "string.h" /* ---------------- * PortalExecutorHeapMemory stuff *************** *** 101,106 **** --- 102,108 ---- int feature; QueryDesc *queryDesc; MemoryContext context; + Const limcount; /* ---------------- * sanity checks *************** *** 113,118 **** --- 115,134 ---- } /* ---------------- + * Create a const node from the given count value + * ---------------- + */ + memset(&limcount, 0, sizeof(limcount)); + limcount.type = T_Const; + limcount.consttype = INT4OID; + limcount.constlen = sizeof(int4); + limcount.constvalue = (Datum)count; + limcount.constisnull = FALSE; + limcount.constbyval = TRUE; + limcount.constisset = FALSE; + limcount.constiscast = FALSE; + + /* ---------------- * get the portal from the portal name * ---------------- */ *************** *** 176,182 **** PortalExecutorHeapMemory = (MemoryContext) PortalGetHeapMemory(portal); ! ExecutorRun(queryDesc, PortalGetState(portal), feature, count); if (dest == None) /* MOVE */ pfree(queryDesc); --- 192,198 ---- PortalExecutorHeapMemory = (MemoryContext) PortalGetHeapMemory(portal); ! ExecutorRun(queryDesc, PortalGetState(portal), feature, (Node *)NULL, (Node *)&limcount); if (dest == None) /* MOVE */ pfree(queryDesc); diff -cr src.orig/backend/executor/execMain.c src/backend/executor/execMain.c *** src.orig/backend/executor/execMain.c Fri Oct 16 11:53:38 1998 --- src/backend/executor/execMain.c Fri Oct 16 20:05:19 1998 *************** *** 64,69 **** --- 64,70 ---- static void EndPlan(Plan *plan, EState *estate); static TupleTableSlot *ExecutePlan(EState *estate, Plan *plan, Query *parseTree, CmdType operation, + int offsetTuples, int numberTuples, ScanDirection direction, void (*printfunc) ()); static void ExecRetrieve(TupleTableSlot *slot, void (*printfunc) (), *************** *** 163,169 **** * ---------------------------------------------------------------- */ TupleTableSlot * ! ExecutorRun(QueryDesc *queryDesc, EState *estate, int feature, int count) { CmdType operation; Query *parseTree; --- 164,170 ---- * ---------------------------------------------------------------- */ TupleTableSlot * ! ExecutorRun(QueryDesc *queryDesc, EState *estate, int feature, Node *limoffset, Node *limcount) { CmdType operation; Query *parseTree; *************** *** 171,176 **** --- 172,179 ---- TupleTableSlot *result; CommandDest dest; void (*destination) (); + int offset = 0; + int count = 0; /****************** * sanity checks *************** *** 191,196 **** --- 194,289 ---- estate->es_processed = 0; estate->es_lastoid = InvalidOid; + /****************** + * if given get the offset of the LIMIT clause + ****************** + */ + if (limoffset != NULL) + { + Const *coffset; + Param *poffset; + ParamListInfo paramLI; + int i; + + switch (nodeTag(limoffset)) + { + case T_Const: + coffset = (Const *)limoffset; + offset = (int)(coffset->constvalue); + break; + + case T_Param: + poffset = (Param *)limoffset; + paramLI = estate->es_param_list_info; + + if (paramLI == NULL) + elog(ERROR, "parameter for limit offset not in executor state"); + for (i = 0; paramLI[i].kind != PARAM_INVALID; i++) + { + if (paramLI[i].kind == PARAM_NUM && paramLI[i].id == poffset->paramid) + break; + } + if (paramLI[i].kind == PARAM_INVALID) + elog(ERROR, "parameter for limit offset not in executor state"); + if (paramLI[i].isnull) + elog(ERROR, "limit offset cannot be NULL value"); + offset = (int)(paramLI[i].value); + + break; + + default: + elog(ERROR, "unexpected node type %d as limit offset", nodeTag(limoffset)); + } + + if (offset < 0) + elog(ERROR, "limit offset cannot be negative"); + } + + /****************** + * if given get the count of the LIMIT clause + ****************** + */ + if (limcount != NULL) + { + Const *ccount; + Param *pcount; + ParamListInfo paramLI; + int i; + + switch (nodeTag(limcount)) + { + case T_Const: + ccount = (Const *)limcount; + count = (int)(ccount->constvalue); + break; + + case T_Param: + pcount = (Param *)limcount; + paramLI = estate->es_param_list_info; + + if (paramLI == NULL) + elog(ERROR, "parameter for limit count not in executor state"); + for (i = 0; paramLI[i].kind != PARAM_INVALID; i++) + { + if (paramLI[i].kind == PARAM_NUM && paramLI[i].id == pcount->paramid) + break; + } + if (paramLI[i].kind == PARAM_INVALID) + elog(ERROR, "parameter for limit count not in executor state"); + if (paramLI[i].isnull) + elog(ERROR, "limit count cannot be NULL value"); + count = (int)(paramLI[i].value); + + break; + + default: + elog(ERROR, "unexpected node type %d as limit count", nodeTag(limcount)); + } + + if (count < 0) + elog(ERROR, "limit count cannot be negative"); + } + switch (feature) { *************** *** 199,205 **** plan, parseTree, operation, ! ALL_TUPLES, ForwardScanDirection, destination); break; --- 292,299 ---- plan, parseTree, operation, ! offset, ! count, ForwardScanDirection, destination); break; *************** *** 208,213 **** --- 302,308 ---- plan, parseTree, operation, + offset, count, ForwardScanDirection, destination); *************** *** 222,227 **** --- 317,323 ---- plan, parseTree, operation, + offset, count, BackwardScanDirection, destination); *************** *** 237,242 **** --- 333,339 ---- plan, parseTree, operation, + 0, ONE_TUPLE, ForwardScanDirection, destination); *************** *** 691,696 **** --- 788,794 ---- Plan *plan, Query *parseTree, CmdType operation, + int offsetTuples, int numberTuples, ScanDirection direction, void (*printfunc) ()) *************** *** 742,747 **** --- 840,859 ---- { result = NULL; break; + } + + /****************** + * For now we completely execute the plan and skip + * result tuples if requested by LIMIT offset. + * Finally we should try to do it in deeper levels + * if possible (during index scan) + * - Jan + ****************** + */ + if (offsetTuples > 0) + { + --offsetTuples; + continue; } /****************** diff -cr src.orig/backend/executor/functions.c src/backend/executor/functions.c *** src.orig/backend/executor/functions.c Fri Oct 16 11:53:38 1998 --- src/backend/executor/functions.c Fri Oct 16 19:01:02 1998 *************** *** 130,135 **** --- 130,138 ---- None); estate = CreateExecutorState(); + if (queryTree->limitOffset != NULL || queryTree->limitCount != NULL) + elog(ERROR, "LIMIT clause from SQL functions not yet implemented"); + if (nargs > 0) { int i; *************** *** 200,206 **** feature = (LAST_POSTQUEL_COMMAND(es)) ? EXEC_RETONE : EXEC_RUN; ! return ExecutorRun(es->qd, es->estate, feature, 0); } static void --- 203,209 ---- feature = (LAST_POSTQUEL_COMMAND(es)) ? EXEC_RETONE : EXEC_RUN; ! return ExecutorRun(es->qd, es->estate, feature, (Node *)NULL, (Node *)NULL); } static void diff -cr src.orig/backend/executor/spi.c src/backend/executor/spi.c *** src.orig/backend/executor/spi.c Fri Oct 16 11:53:39 1998 --- src/backend/executor/spi.c Fri Oct 16 19:25:33 1998 *************** *** 791,796 **** --- 791,798 ---- bool isRetrieveIntoRelation = false; char *intoName = NULL; int res; + Const tcount_const; + Node *count = NULL; switch (operation) { *************** *** 825,830 **** --- 827,865 ---- return SPI_ERROR_OPUNKNOWN; } + /* ---------------- + * Get the query LIMIT tuple count + * ---------------- + */ + if (parseTree->limitCount != NULL) + { + /* ---------------- + * A limit clause in the parsetree overrides the + * tcount parameter + * ---------------- + */ + count = parseTree->limitCount; + } + else + { + /* ---------------- + * No LIMIT clause in parsetree. Use a local Const node + * to put tcount into it + * ---------------- + */ + memset(&tcount_const, 0, sizeof(tcount_const)); + tcount_const.type = T_Const; + tcount_const.consttype = INT4OID; + tcount_const.constlen = sizeof(int4); + tcount_const.constvalue = (Datum)tcount; + tcount_const.constisnull = FALSE; + tcount_const.constbyval = TRUE; + tcount_const.constisset = FALSE; + tcount_const.constiscast = FALSE; + + count = (Node *)&tcount_const; + } + if (state == NULL) /* plan preparation */ return res; #ifdef SPI_EXECUTOR_STATS *************** *** 845,851 **** return SPI_OK_CURSOR; } ! ExecutorRun(queryDesc, state, EXEC_FOR, tcount); _SPI_current->processed = state->es_processed; if (operation == CMD_SELECT && queryDesc->dest == SPI) --- 880,886 ---- return SPI_OK_CURSOR; } ! ExecutorRun(queryDesc, state, EXEC_FOR, parseTree->limitOffset, count); _SPI_current->processed = state->es_processed; if (operation == CMD_SELECT && queryDesc->dest == SPI) diff -cr src.orig/backend/parser/analyze.c src/backend/parser/analyze.c *** src.orig/backend/parser/analyze.c Fri Oct 16 11:53:41 1998 --- src/backend/parser/analyze.c Fri Oct 16 13:29:27 1998 *************** *** 180,186 **** --- 180,190 ---- case T_SelectStmt: if (!((SelectStmt *) parseTree)->portalname) + { result = transformSelectStmt(pstate, (SelectStmt *) parseTree); + result->limitOffset = ((SelectStmt *)parseTree)->limitOffset; + result->limitCount = ((SelectStmt *)parseTree)->limitCount; + } else result = transformCursorStmt(pstate, (SelectStmt *) parseTree); break; diff -cr src.orig/backend/parser/gram.y src/backend/parser/gram.y *** src.orig/backend/parser/gram.y Fri Oct 16 11:53:42 1998 --- src/backend/parser/gram.y Sun Oct 18 22:20:36 1998 *************** *** 45,50 **** --- 45,51 ---- #include "catalog/catname.h" #include "utils/elog.h" #include "access/xact.h" + #include "catalog/pg_type.h" #ifdef MULTIBYTE #include "mb/pg_wchar.h" *************** *** 163,169 **** sort_clause, sortby_list, index_params, index_list, name_list, from_clause, from_list, opt_array_bounds, nest_array_bounds, expr_list, attrs, res_target_list, res_target_list2, ! def_list, opt_indirection, group_clause, TriggerFuncArgs %type func_return %type set_opt --- 164,171 ---- sort_clause, sortby_list, index_params, index_list, name_list, from_clause, from_list, opt_array_bounds, nest_array_bounds, expr_list, attrs, res_target_list, res_target_list2, ! def_list, opt_indirection, group_clause, TriggerFuncArgs, ! opt_select_limit %type func_return %type set_opt *************** *** 192,197 **** --- 194,201 ---- %type fetch_how_many + %type select_limit_value select_offset_value + %type OptSeqList %type OptSeqElem *************** *** 267,273 **** FALSE_P, FETCH, FLOAT, FOR, FOREIGN, FROM, FULL, GRANT, GROUP, HAVING, HOUR_P, IN, INNER_P, INSENSITIVE, INSERT, INTERVAL, INTO, IS, ! JOIN, KEY, LANGUAGE, LEADING, LEFT, LIKE, LOCAL, MATCH, MINUTE_P, MONTH_P, NAMES, NATIONAL, NATURAL, NCHAR, NEXT, NO, NOT, NULL_P, NUMERIC, OF, ON, ONLY, OPTION, OR, ORDER, OUTER_P, --- 271,277 ---- FALSE_P, FETCH, FLOAT, FOR, FOREIGN, FROM, FULL, GRANT, GROUP, HAVING, HOUR_P, IN, INNER_P, INSENSITIVE, INSERT, INTERVAL, INTO, IS, ! JOIN, KEY, LANGUAGE, LEADING, LEFT, LIKE, LIMIT, LOCAL, MATCH, MINUTE_P, MONTH_P, NAMES, NATIONAL, NATURAL, NCHAR, NEXT, NO, NOT, NULL_P, NUMERIC, OF, ON, ONLY, OPTION, OR, ORDER, OUTER_P, *************** *** 299,305 **** INCREMENT, INDEX, INHERITS, INSTEAD, ISNULL, LANCOMPILER, LISTEN, LOAD, LOCATION, LOCK_P, MAXVALUE, MINVALUE, MOVE, NEW, NOCREATEDB, NOCREATEUSER, NONE, NOTHING, NOTIFY, NOTNULL, ! OIDS, OPERATOR, PASSWORD, PROCEDURAL, RECIPE, RENAME, RESET, RETURNS, ROW, RULE, SEQUENCE, SERIAL, SETOF, SHOW, START, STATEMENT, STDIN, STDOUT, TRUSTED, UNLISTEN, UNTIL, VACUUM, VALID, VERBOSE, VERSION --- 303,309 ---- INCREMENT, INDEX, INHERITS, INSTEAD, ISNULL, LANCOMPILER, LISTEN, LOAD, LOCATION, LOCK_P, MAXVALUE, MINVALUE, MOVE, NEW, NOCREATEDB, NOCREATEUSER, NONE, NOTHING, NOTIFY, NOTNULL, ! OFFSET, OIDS, OPERATOR, PASSWORD, PROCEDURAL, RECIPE, RENAME, RESET, RETURNS, ROW, RULE, SEQUENCE, SERIAL, SETOF, SHOW, START, STATEMENT, STDIN, STDOUT, TRUSTED, UNLISTEN, UNTIL, VACUUM, VALID, VERBOSE, VERSION *************** *** 2591,2596 **** --- 2595,2601 ---- result from_clause where_clause group_clause having_clause union_clause sort_clause + opt_select_limit { SelectStmt *n = makeNode(SelectStmt); n->unique = $2; *************** *** 2602,2607 **** --- 2607,2622 ---- n->havingClause = $8; n->unionClause = $9; n->sortClause = $10; + if ($11 != NIL) + { + n->limitOffset = nth(0, $11); + n->limitCount = nth(1, $11); + } + else + { + n->limitOffset = NULL; + n->limitCount = NULL; + } $$ = (Node *)n; } ; *************** *** 2699,2704 **** --- 2714,2794 ---- | ASC { $$ = "<"; } | DESC { $$ = ">"; } | /*EMPTY*/ { $$ = "<"; /*default*/ } + ; + + opt_select_limit: LIMIT select_offset_value ',' select_limit_value + { $$ = lappend(lappend(NIL, $2), $4); } + | LIMIT select_limit_value OFFSET select_offset_value + { $$ = lappend(lappend(NIL, $4), $2); } + | LIMIT select_limit_value + { $$ = lappend(lappend(NIL, NULL), $2); } + | OFFSET select_offset_value LIMIT select_limit_value + { $$ = lappend(lappend(NIL, $2), $4); } + | OFFSET select_offset_value + { $$ = lappend(lappend(NIL, $2), NULL); } + | /* EMPTY */ + { $$ = NIL; } + ; + + select_limit_value: Iconst + { + Const *n = makeNode(Const); + + if ($1 < 1) + elog(ERROR, "selection limit must be ALL or a positive integer value > 0"); + + n->consttype = INT4OID; + n->constlen = sizeof(int4); + n->constvalue = (Datum)$1; + n->constisnull = FALSE; + n->constbyval = TRUE; + n->constisset = FALSE; + n->constiscast = FALSE; + $$ = (Node *)n; + } + | ALL + { + Const *n = makeNode(Const); + n->consttype = INT4OID; + n->constlen = sizeof(int4); + n->constvalue = (Datum)0; + n->constisnull = FALSE; + n->constbyval = TRUE; + n->constisset = FALSE; + n->constiscast = FALSE; + $$ = (Node *)n; + } + | PARAM + { + Param *n = makeNode(Param); + n->paramkind = PARAM_NUM; + n->paramid = $1; + n->paramtype = INT4OID; + $$ = (Node *)n; + } + ; + + select_offset_value: Iconst + { + Const *n = makeNode(Const); + + n->consttype = INT4OID; + n->constlen = sizeof(int4); + n->constvalue = (Datum)$1; + n->constisnull = FALSE; + n->constbyval = TRUE; + n->constisset = FALSE; + n->constiscast = FALSE; + $$ = (Node *)n; + } + | PARAM + { + Param *n = makeNode(Param); + n->paramkind = PARAM_NUM; + n->paramid = $1; + n->paramtype = INT4OID; + $$ = (Node *)n; + } ; /* diff -cr src.orig/backend/parser/keywords.c src/backend/parser/keywords.c *** src.orig/backend/parser/keywords.c Fri Oct 16 11:53:42 1998 --- src/backend/parser/keywords.c Sun Oct 18 22:13:29 1998 *************** *** 128,133 **** --- 128,134 ---- {"leading", LEADING}, {"left", LEFT}, {"like", LIKE}, + {"limit", LIMIT}, {"listen", LISTEN}, {"load", LOAD}, {"local", LOCAL}, *************** *** 156,161 **** --- 157,163 ---- {"null", NULL_P}, {"numeric", NUMERIC}, {"of", OF}, + {"offset", OFFSET}, {"oids", OIDS}, {"old", CURRENT}, {"on", ON}, diff -cr src.orig/backend/rewrite/rewriteDefine.c src/backend/rewrite/rewriteDefine.c *** src.orig/backend/rewrite/rewriteDefine.c Fri Oct 16 11:53:46 1998 --- src/backend/rewrite/rewriteDefine.c Fri Oct 16 13:48:55 1998 *************** *** 312,317 **** --- 312,323 ---- heap_close(event_relation); /* + * LIMIT in view is not supported + */ + if (query->limitOffset != NULL || query->limitCount != NULL) + elog(ERROR, "LIMIT clause not supported in views"); + + /* * ... and finally the rule must be named _RETviewname. */ sprintf(expected_name, "_RET%s", event_obj->relname); diff -cr src.orig/backend/tcop/pquery.c src/backend/tcop/pquery.c *** src.orig/backend/tcop/pquery.c Fri Oct 16 11:53:47 1998 --- src/backend/tcop/pquery.c Fri Oct 16 14:02:36 1998 *************** *** 40,46 **** #include "commands/command.h" static char *CreateOperationTag(int operationType); ! static void ProcessQueryDesc(QueryDesc *queryDesc); /* ---------------------------------------------------------------- --- 40,46 ---- #include "commands/command.h" static char *CreateOperationTag(int operationType); ! static void ProcessQueryDesc(QueryDesc *queryDesc, Node *limoffset, Node *limcount); /* ---------------------------------------------------------------- *************** *** 205,211 **** * ---------------------------------------------------------------- */ static void ! ProcessQueryDesc(QueryDesc *queryDesc) { Query *parseTree; Plan *plan; --- 205,211 ---- * ---------------------------------------------------------------- */ static void ! ProcessQueryDesc(QueryDesc *queryDesc, Node *limoffset, Node *limcount) { Query *parseTree; Plan *plan; *************** *** 330,336 **** * actually run the plan.. * ---------------- */ ! ExecutorRun(queryDesc, state, EXEC_RUN, 0); /* save infos for EndCommand */ UpdateCommandInfo(operation, state->es_lastoid, state->es_processed); --- 330,336 ---- * actually run the plan.. * ---------------- */ ! ExecutorRun(queryDesc, state, EXEC_RUN, limoffset, limcount); /* save infos for EndCommand */ UpdateCommandInfo(operation, state->es_lastoid, state->es_processed); *************** *** 373,377 **** print_plan(plan, parsetree); } else ! ProcessQueryDesc(queryDesc); } --- 373,377 ---- print_plan(plan, parsetree); } else ! ProcessQueryDesc(queryDesc, parsetree->limitOffset, parsetree->limitCount); } diff -cr src.orig/include/executor/executor.h src/include/executor/executor.h *** src.orig/include/executor/executor.h Fri Oct 16 11:53:56 1998 --- src/include/executor/executor.h Fri Oct 16 12:04:17 1998 *************** *** 83,89 **** * prototypes from functions in execMain.c */ extern TupleDesc ExecutorStart(QueryDesc *queryDesc, EState *estate); ! extern TupleTableSlot *ExecutorRun(QueryDesc *queryDesc, EState *estate, int feature, int count); extern void ExecutorEnd(QueryDesc *queryDesc, EState *estate); extern HeapTuple ExecConstraints(char *caller, Relation rel, HeapTuple tuple); #ifdef QUERY_LIMIT --- 83,89 ---- * prototypes from functions in execMain.c */ extern TupleDesc ExecutorStart(QueryDesc *queryDesc, EState *estate); ! extern TupleTableSlot *ExecutorRun(QueryDesc *queryDesc, EState *estate, int feature, Node *limoffset, Node *limcount); extern void ExecutorEnd(QueryDesc *queryDesc, EState *estate); extern HeapTuple ExecConstraints(char *caller, Relation rel, HeapTuple tuple); #ifdef QUERY_LIMIT