head	1.16;
access;
symbols
	REL7_4_29:1.15
	REL7_4_28:1.15
	REL7_4_27:1.15
	REL7_4_26:1.15
	REL7_4_25:1.15
	REL7_4_24:1.15
	REL7_4_23:1.15
	REL7_4_22:1.15
	REL7_4_21:1.15
	REL7_4_20:1.15
	REL7_3_21:1.15
	REL7_4_19:1.15
	REL7_3_20:1.15
	REL7_4_18:1.15
	REL7_3_19:1.15
	REL7_4_17:1.15
	REL7_3_18:1.15
	REL7_4_16:1.15
	REL7_4_15:1.15
	REL7_3_17:1.15
	REL7_4_14:1.15
	REL7_3_16:1.15
	REL7_3_15:1.15
	REL7_4_13:1.15
	REL7_3_14:1.15
	REL7_4_12:1.15
	REL7_3_13:1.15
	REL7_4_11:1.15
	REL7_3_12:1.15
	REL7_4_10:1.15
	REL7_3_11:1.15
	REL7_4_9:1.15
	REL7_2_8:1.15
	REL7_3_10:1.15
	REL7_4_8:1.15
	REL7_2_7:1.15
	REL7_3_9:1.15
	REL7_4_7:1.15
	REL7_4_6:1.15
	REL7_3_8:1.15
	REL7_2_6:1.15
	REL7_2_5:1.15
	REL7_4_5:1.15
	REL7_3_7:1.15
	REL7_4_4:1.15
	REL7_4_3:1.15
	REL7_4_2:1.15
	REL7_3_6:1.15
	REL7_4_1:1.15
	REL7_3_5:1.15
	REL7_4:1.15
	REL7_4_RC2:1.15
	REL7_4_STABLE:1.15.0.10
	REL7_4_RC1:1.15
	REL7_4_BETA5:1.15
	REL7_4_BETA4:1.15
	REL7_4_BETA3:1.15
	REL7_4_BETA2:1.15
	WIN32_DEV:1.15.0.8
	REL7_4_BETA1:1.15
	REL7_3_4:1.15
	REL7_3_2:1.15
	REL7_2_4:1.15
	REL7_3_STABLE:1.15.0.6
	REL7_2_3:1.15
	REL7_2_STABLE:1.15.0.4
	REL7_2:1.15
	REL7_2_RC2:1.15
	REL7_2_RC1:1.15
	REL7_2_BETA5:1.15
	REL7_2_BETA4:1.15
	REL7_2_BETA3:1.15
	REL7_2_BETA2:1.15
	REL7_2_BETA1:1.15
	REL7_1_2:1.15
	REL7_1_STABLE:1.15.0.2
	REL7_1:1.15
	REL7_0_PATCHES:1.13.0.2
	REL7_0:1.13;
locks; strict;
comment	@# @;


1.16
date	2004.02.12.18.11.53;	author momjian;	state dead;
branches;
next	1.15;

1.15
date	2001.01.24.14.24.40;	author momjian;	state Exp;
branches;
next	1.14;

1.14
date	2000.07.04.05.17.03;	author momjian;	state dead;
branches;
next	1.13;

1.13
date	99.09.23.18.58.49;	author momjian;	state Exp;
branches;
next	1.12;

1.12
date	99.09.23.18.54.56;	author momjian;	state Exp;
branches;
next	1.11;

1.11
date	99.09.23.17.59.08;	author momjian;	state Exp;
branches;
next	1.10;

1.10
date	99.09.23.17.50.57;	author momjian;	state Exp;
branches;
next	1.9;

1.9
date	99.09.23.15.47.46;	author momjian;	state Exp;
branches;
next	1.8;

1.8
date	99.09.23.15.43.39;	author momjian;	state Exp;
branches;
next	1.7;

1.7
date	99.09.21.21.37.05;	author momjian;	state Exp;
branches;
next	1.6;

1.6
date	99.09.21.21.17.42;	author momjian;	state Exp;
branches;
next	1.5;

1.5
date	99.09.21.20.00.14;	author momjian;	state Exp;
branches;
next	1.4;

1.4
date	99.09.21.19.58.00;	author momjian;	state Exp;
branches;
next	1.3;

1.3
date	99.09.20.21.43.06;	author momjian;	state Exp;
branches;
next	1.2;

1.2
date	99.09.20.16.28.12;	author momjian;	state Exp;
branches;
next	1.1;

1.1
date	99.09.20.15.40.12;	author momjian;	state Exp;
branches;
next	;


desc
@@


1.16
log
@Remove TODO.detail files that contained useless or very old information.
Update TODO accordingly.
@
text
@From pgsql-hackers-owner+M908@@postgresql.org Sun Nov 19 14:27:43 2000
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
	by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id OAA10885
	for <pgman@@candle.pha.pa.us>; Sun, 19 Nov 2000 14:27:42 -0500 (EST)
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
	by mail.postgresql.org (8.11.1/8.11.1) with SMTP id eAJJSMs83653;
	Sun, 19 Nov 2000 14:28:22 -0500 (EST)
	(envelope-from pgsql-hackers-owner+M908@@postgresql.org)
Received: from candle.pha.pa.us (candle.navpoint.com [162.33.245.46] (may be forged))
	by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id eAJJQns83565
	for <pgsql-hackers@@postgreSQL.org>; Sun, 19 Nov 2000 14:26:49 -0500 (EST)
	(envelope-from pgman@@candle.pha.pa.us)
Received: (from pgman@@localhost)
	by candle.pha.pa.us (8.9.0/8.9.0) id OAA06790;
	Sun, 19 Nov 2000 14:23:06 -0500 (EST)
From: Bruce Momjian <pgman@@candle.pha.pa.us>
Message-Id: <200011191923.OAA06790@@candle.pha.pa.us>
Subject: Re: [HACKERS] WAL fsync scheduling
In-Reply-To: <002101c0525e$2d964480$b97a30d0@@sectorbase.com> "from Vadim Mikheev
	at Nov 19, 2000 11:23:19 am"
To: Vadim Mikheev <vmikheev@@sectorbase.com>
Date: Sun, 19 Nov 2000 14:23:06 -0500 (EST)
CC: Tom Samplonius <tom@@sdf.com>, Alfred@@candle.pha.pa.us,
        Perlstein <bright@@wintelcom.net>, Larry@@candle.pha.pa.us,
        Rosenman <ler@@lerctr.org>,
        PostgreSQL-development <pgsql-hackers@@postgresql.org>
X-Mailer: ELM [version 2.4ME+ PL77 (25)]
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=US-ASCII
Precedence: bulk
Sender: pgsql-hackers-owner@@postgresql.org
Status: OR

[ Charset ISO-8859-1 unsupported, converting... ]
> > There are two parts to transaction commit.  The first is writing all
> > dirty buffers or log changes to the kernel, and second is fsync of the
>    ^^^^^^^^^^^^
> Backend doesn't write any dirty buffer to the kernel at commit time.

Yes, I suspected that.

> 
> > log file.
> 
> The first part is writing commit record into WAL buffers in shmem.
> This is what XLogInsert does.  After that XLogFlush is called to ensure
> that  entire commit record is on disk. XLogFlush does *both* write() and
> fsync() (single slock is used for both writing and fsyncing) if it needs to
> do it at all.

Yes, I realize there are new steps in WAL.

> 
> > I suggest having a per-backend shared memory byte that has the following
> > values:
> > 
> > START_LOG_WRITE
> > WAIT_ON_FSYNC
> > NOT_IN_COMMIT
> > backend_number_doing_fsync
> > 
> > I suggest that when each backend starts a commit, it sets its byte to
> > START_LOG_WRITE. 
>   ^^^^^^^^^^^^^^^^^^^^^^^
> Isn't START_COMMIT more meaningful?

Yes.

> 
> > When it gets ready to fsync, it checks all backends. 
>    ^^^^^^^^^^^^^^^^^^^^^^^^^^
> What do you mean by this? The moment just after XLogInsert?

Just before it calls fsync().

> 
> > If all are NOT_IN_COMMIT, it does fsync and continues.
> 
> 1st edition:
> > If one or more are in START_LOG_WRITE, it waits until no one is in
> > START_LOG_WRITE.  It then checks all WAIT_ON_FSYNC, and if it is the
> > lowest backend in WAIT_ON_FSYNC, marks all others with its backend
> > number, and does fsync.  It then clears all backends with its number to
> > NOT_IN_COMMIT.  Other backend will see they are not the lowest
> > WAIT_ON_FSYNC and will wait for their byte to be set to NOT_IN_COMMIT
> > so they can then continue, knowing their data was synced.
> 
> 2nd edition:
> > I have another idea.  If a backend gets to the point that it needs
> > fsync, and there is another backend in START_LOG_WRITE, it can go to an
> > interuptable sleep, knowing another backend will perform the fsync and
> > wake it up.  Therefore, there is no busy-wait or timed sleep.
> > 
> > Of course, a backend must set its status to WAIT_ON_FSYNC to avoid a
> > race condition.
> 
> The 2nd edition is much better. But I'm not sure do we really need in
> these per-backend bytes in shmem. Why not just have some counters?
> We can use a semaphore to wake-up all waiters at once.

Yes, that is much better and clearer.  My idea was just to say, "if no
one is entering commit phase, do the commit.  If someone else is coming,
sleep and wait for them to do the fsync and wake me up with a singal."  

> 
> > This allows a single backend not to sleep, and allows multiple backends
> > to bunch up only when they are all about to commit.
> > 
> > The reason backend numbers are written is so other backends entering the
> > commit code will not interfere with the backends performing fsync.
> 
> Being waked-up backend can check what's written/fsynced by calling XLogFlush.

Seems that may not be needed anymore with a counter.  The only issue is
that other backends may enter commit while fsync() is happening.  The
process that did the fsync must be sure to wake up only the backends
that were waiting for it, and not other backends that may be also be
doing fsync as a group while the first fsync was happening.  I leave
those details to people more experienced.  :-)

I am just glad people liked my idea.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

@


1.15
log
@Add.
@
text
@@


1.14
log
@Remove unused TODO.detail files.
@
text
@d1 27
a27 28
From owner-pgsql-general@@hub.org Fri Dec 18 06:31:23 1998
Received: from renoir.op.net (root@@renoir.op.net [209.152.193.4])
	by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id GAA05554
	for <maillist@@candle.pha.pa.us>; Fri, 18 Dec 1998 06:31:21 -0500 (EST)
Received: from hub.org (majordom@@hub.org [209.47.145.100]) by renoir.op.net (o1/$ Revision: 1.18 $) with ESMTP id EAA21127 for <maillist@@candle.pha.pa.us>; Fri, 18 Dec 1998 04:46:38 -0500 (EST)
Received: from localhost (majordom@@localhost)
	by hub.org (8.9.1/8.9.1) with SMTP id EAA01409;
	Fri, 18 Dec 1998 04:44:19 -0500 (EST)
	(envelope-from owner-pgsql-general@@hub.org)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Fri, 18 Dec 1998 04:43:22 +0000 (EST)
Received: (from majordom@@localhost)
	by hub.org (8.9.1/8.9.1) id EAA01093
	for pgsql-general-outgoing; Fri, 18 Dec 1998 04:43:18 -0500 (EST)
	(envelope-from owner-pgsql-general@@postgreSQL.org)
Received: from dune.krs.ru (dune.krs.ru [195.161.16.38])
	by hub.org (8.9.1/8.9.1) with ESMTP id EAA01067
	for <pgsql-general@@postgreSQL.org>; Fri, 18 Dec 1998 04:43:09 -0500 (EST)
	(envelope-from vadim@@krs.ru)
Received: from krs.ru (localhost.krs.ru [127.0.0.1])
	by dune.krs.ru (8.8.8/8.8.7) with ESMTP id QAA16201;
	Fri, 18 Dec 1998 16:41:44 +0700 (KRS)
	(envelope-from vadim@@krs.ru)
Message-ID: <367A2354.E998763@@krs.ru>
Date: Fri, 18 Dec 1998 16:41:40 +0700
From: Vadim Mikheev <vadim@@krs.ru>
Organization: OJSC Rostelecom (Krasnoyarsk)
X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 2.2.6-RELEASE i386)
X-Accept-Language: ru, en
a28 5
To: Anton de Wet <adw@@obsidian.co.za>
CC: pgsql-general@@postgreSQL.org
Subject: Re: [GENERAL] Why PostgreSQL is better than other commerial softwares?
References: <Pine.LNX.4.04.9812181046030.9458-100000@@ra.obsidian.co.za>
Content-Type: text/plain; charset=us-ascii
d30 1
a30 1
Sender: owner-pgsql-general@@postgreSQL.org
d32 10
a41 1
Status: RO
a42 1
Anton de Wet wrote:
d44 1
a44 2
> >
> > Often quick mailing list support?
d46 8
a53 1
> :-)
d55 15
a69 3
> While on the subject I finally found the solution to a problem I (and one
> or two other people) posted about without answer. (So sometimes it's slow
> mailing list support).
d71 3
a73 6
> In importing about 5 million records (which I copy in blocks of 10000) the
> copy became linearly slower. After a friend RTFM and refered me, I used
> the -F switch (passed by the postmaster to the backend processes) and the
> time became linear and a LOT shorter. Import time for the 5000000 records
> now the same (or maybe even slightly faster, I didn't accurately time
> them) as importing the data into oracle on the same machine.
d75 39
a113 1
"While on the subject..." -:)
d115 6
a120 6
This is the problem of buffer manager, known for very long time:
when copy eats all buffers, manager begins write/fsync each
durty buffer to free buffer for new data. All updated relations
should be fsynced _once_ @@ transaction commit. You would get
the same results without -F...
I still have no time to implement this -:(
d122 1
a122 1
Vadim
d124 5
@


1.13
log
@Update TODO list.
@
text
@@


1.12
log
@Update TODO list.
@
text
@d5 1
a5 1
Received: from hub.org (majordom@@hub.org [209.47.145.100]) by renoir.op.net (o1/$Revision: 1.18 $) with ESMTP id EAA21127 for <maillist@@candle.pha.pa.us>; Fri, 18 Dec 1998 04:46:38 -0500 (EST)
@


1.11
log
@Update TODO list.
@
text
@@


1.10
log
@Update TODO list.
@
text
@@


1.9
log
@Update TODO list.
@
text
@@


1.8
log
@Update TODO list.
@
text
@@


1.7
log
@Update TODO list.
@
text
@@


1.6
log
@Update TODO list.
@
text
@@


1.5
log
@Update TODO list.
@
text
@@


1.4
log
@Update TODO list.
@
text
@@


1.3
log
@Update pgaccess 0.98.
@
text
@@


1.2
log
@Update TODO.
@
text
@@


1.1
log
@Add TODO detail directory.
@
text
@@
