Fix memory leak in Incremental Sort rescans

The Incremental Sort had a couple issues, resulting in leaking memory
during rescans, possibly triggering OOM. The code had a couple of
related flaws:

1. During rescans, the sort states were reset but then also set to NULL
   (despite the comment saying otherwise). ExecIncrementalSort then
   sees NULL and initializes a new sort state, leaking the memory used
   by the old one.

2. Initializing the sort state also automatically rebuilt the info about
   presorted keys, leaking the already initialized info. presorted_keys
   was also unnecessarily reset to NULL.

Patch by James Coleman, based on patches by Laurenz Albe and Tom Lane.
Backpatch to 13, where Incremental Sort was introduced.

Author: James Coleman, Laurenz Albe, Tom Lane
Reported-by: Laurenz Albe, Zu-Ming Jiang
Backpatch-through: 13
Discussion: https://postgr.es/m/b2bd02dff61af15e3526293e2771f874cf2a3be7.camel%40cybertec.at
Discussion: https://postgr.es/m/db03c582-086d-e7cd-d4a1-3bc722f81765%40inf.ethz.ch

diff --git a/src/backend/executor/nodeIncrementalSort.c b/src/backend/executor/nodeIncrementalSort.c
index 34257ce34be9d693aa5a6312c399d250bfb550fc..7683e3341cde9abbafa18cde1cba1f44a20b3903 100644 (file)
--- a/src/backend/executor/nodeIncrementalSort.c
+++ b/src/backend/executor/nodeIncrementalSort.c
@@ -1140,7 +1140,6 @@ ExecReScanIncrementalSort(IncrementalSortState *node)
        node->outerNodeDone = false;
        node->n_fullsort_remaining = 0;
        node->bound_Done = 0;
-       node->presorted_keys = NULL;
 
        node->execution_status = INCSORT_LOADFULLSORT;
 
@@ -1153,15 +1152,9 @@ ExecReScanIncrementalSort(IncrementalSortState *node)
         * cause a leak.
         */
        if (node->fullsort_state != NULL)
-       {
                tuplesort_reset(node->fullsort_state);
-               node->fullsort_state = NULL;
-       }
        if (node->prefixsort_state != NULL)
-       {
                tuplesort_reset(node->prefixsort_state);
-               node->prefixsort_state = NULL;
-       }
 
        /*
         * If chgParam of subnode is not null, then the plan will be re-scanned by