Bug #23259872: OPTIMIZER CHOOSES TO USE NON PRIMARY INDEX,

               EVEN THOUGH COST IS HIGHER

Issue:
------
Optimizer tries to choose between a range-access and
ref-access, based on various parameters like
records_per_key, CPU cost and I/O cost. When the
row-estimate for ref access is inaccurate, optimizer
uses the estimate from range access on the same index.

In the current issue the ref access's row-estimate comes
from range-access's estimate. The ref-access on a secondary
index is calculated to be cheaper than the best
range-access on the primary key. This will result in a bad
performance.

Solution (mysql-trunk):
---------
Ignore the index for ref access when we have the above mentioned
problem.

Solution (mysql-5.7):
---------
Shift from ref-access to range-access when:
1) Ref-access row estimate is overly-optimistic and it has
   borrowed range-access's estimate on the same index.
2) Range-access uses more index key parts than ref access.

We already do this in cases where the above conditions are
met and range access on the same index is the cheapest.

With this fix, we shift to range-access on that index, even
if it is not considered the cheapest. To do this, we run
the range optimizer on that index alone.


sql/sql_select.h:
A new boolean 'dodgy_ref_cost' has been added to JOIN_TAB.
Setting this flag means ref is using lesser number of
key-parts than range and it borrows range's row estimate.

sql/sql_planner.h:
find_best_ref should be able to modify the flag
'dodgy_ref_cost', so removing the const.

sql/sql_planner.cc:
In find_best_ref, keep track of whether ref access's
row-estimate is problematic and set the 'dodgy_ref_cost'
flag.

sql/sql_optimizer.cc:
Conditions 3, 4 were earlier tested for negation. Now
they are tested before 2 and false is returned. Check if
the 'dodgy_ref_cost' flag is set. Then it is better we
shift to range on the same index since it uses more
keyparts and its row-estimate is more reliable. We do
this by calling the range optimizer again on that index
alone.

Solution (mysql-5.6):
---------
Same approach as 5.7. With the following changes:

sql/sql_optimizer.cc:
A new function can_switch_from_ref_to_range, similar to the
one in 5.7+ has been created.

Author: Sreeharsha Ramanavarapu

mysql-test/include/range.inc

@@ -1836,4 +1836,67 @@ CREATE TABLE t1 (
 
 SELECT 1 FROM t1 WHERE a <> 'a' OR a <> "";
 
-DROP TABLE t1;
+DROP TABLE t1;
+
+--echo #
+--echo # Bug #23259872: OPTIMIZER CHOOSES TO USE NON PRIMARY
+--echo #                INDEX, EVEN THOUGH COST IS HIGHER
+--echo #
+
+CREATE TABLE `giant_table` (
+  `id` int(11) NOT NULL AUTO_INCREMENT,
+  `one_id` int(11) NOT NULL,
+  `other_id` bigint(20) NOT NULL DEFAULT '0',
+  `some_other_id` int(11) DEFAULT 0 NOT NULL,
+  `something` double NOT NULL DEFAULT '0',
+  `comment` text COLLATE utf8_unicode_ci,
+  `flags` int(11) NOT NULL DEFAULT '0',
+  `time_created` int(11) NOT NULL DEFAULT '0',
+  PRIMARY KEY (`id`),
+  KEY `time_created` (`time_created`),
+  KEY `some_other_id` (`some_other_id`),
+  KEY `one_other_idx` (`one_id`,`other_id`),
+  KEY `other_id` (`other_id`,`time_created`)
+) ENGINE=InnoDB AUTO_INCREMENT=101651329
+DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci ;
+
+CREATE TABLE t1 (c1 INT);
+INSERT INTO t1 VALUES (66136540), (68983250), (89627210), (77869520),
+                      (82543190), (67538270), (77282760), (77908170),
+                      (70923370), (68066360);
+DELIMITER $;
+CREATE PROCEDURE p()
+BEGIN
+SET @x = 1;
+REPEAT
+
+INSERT INTO giant_table(id,one_id)
+  SELECT c1 + @x, 0
+  FROM t1
+  WHERE c1 IN (66136540, 68985250, 89627210, 77869520 , 82543190, 67538270,
+               77282760, 77908170, 70923370, 68066360);
+SET @x =  @x + 1;
+
+UNTIL @x > 30 END REPEAT;
+END $
+DELIMITER ;$
+
+CALL p();
+SELECT count(*) FROM giant_table;
+
+INSERT INTO giant_table (id,one_id) VALUES (66136539, 0), (68983258,1),
+                                           (89628210,1), (77869520,2);
+INSERT INTO giant_table (id,one_id, some_other_id) VALUES(84673401, 0, 1),
+                                                         (61069031, 1, 1);
+
+EXPLAIN SELECT id, something, comment, time_created, one_id, other_id,
+               some_other_id, flags
+FROM giant_table
+WHERE id IN (66136539, 68983258, 89628210, 77869520, 82543198, 67538272,
+             84673401, 61069031, 68214385, 77282865, 76991297, 64569216,
+             89481638, 74534074, 70396537, 80076375, 63308530, 77908270,
+             70923271, 68066180)
+  AND (giant_table.flags & 0x01) = 0 AND giant_table.some_other_id = 0;
+
+DROP PROCEDURE p;
+DROP TABLE giant_table, t1;

mysql-test/r/range_all.result

@@ -2494,4 +2494,60 @@ KEY(a(1))
 SELECT 1 FROM t1 WHERE a <> 'a' OR a <> "";
 1
 DROP TABLE t1;
+#
+# Bug #23259872: OPTIMIZER CHOOSES TO USE NON PRIMARY
+#                INDEX, EVEN THOUGH COST IS HIGHER
+#
+CREATE TABLE `giant_table` (
+`id` int(11) NOT NULL AUTO_INCREMENT,
+`one_id` int(11) NOT NULL,
+`other_id` bigint(20) NOT NULL DEFAULT '0',
+`some_other_id` int(11) DEFAULT 0 NOT NULL,
+`something` double NOT NULL DEFAULT '0',
+`comment` text COLLATE utf8_unicode_ci,
+`flags` int(11) NOT NULL DEFAULT '0',
+`time_created` int(11) NOT NULL DEFAULT '0',
+PRIMARY KEY (`id`),
+KEY `time_created` (`time_created`),
+KEY `some_other_id` (`some_other_id`),
+KEY `one_other_idx` (`one_id`,`other_id`),
+KEY `other_id` (`other_id`,`time_created`)
+) ENGINE=InnoDB AUTO_INCREMENT=101651329
+DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci ;
+CREATE TABLE t1 (c1 INT);
+INSERT INTO t1 VALUES (66136540), (68983250), (89627210), (77869520),
+(82543190), (67538270), (77282760), (77908170),
+(70923370), (68066360);
+CREATE PROCEDURE p()
+BEGIN
+SET @x = 1;
+REPEAT
+INSERT INTO giant_table(id,one_id)
+SELECT c1 + @x, 0
+FROM t1
+WHERE c1 IN (66136540, 68985250, 89627210, 77869520 , 82543190, 67538270,
+77282760, 77908170, 70923370, 68066360);
+SET @x =  @x + 1;
+UNTIL @x > 30 END REPEAT;
+END $
+CALL p();
+SELECT count(*) FROM giant_table;
+count(*)
+270
+INSERT INTO giant_table (id,one_id) VALUES (66136539, 0), (68983258,1),
+(89628210,1), (77869520,2);
+INSERT INTO giant_table (id,one_id, some_other_id) VALUES(84673401, 0, 1),
+(61069031, 1, 1);
+EXPLAIN SELECT id, something, comment, time_created, one_id, other_id,
+some_other_id, flags
+FROM giant_table
+WHERE id IN (66136539, 68983258, 89628210, 77869520, 82543198, 67538272,
+84673401, 61069031, 68214385, 77282865, 76991297, 64569216,
+89481638, 74534074, 70396537, 80076375, 63308530, 77908270,
+70923271, 68066180)
+AND (giant_table.flags & 0x01) = 0 AND giant_table.some_other_id = 0;
+id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
+1	SIMPLE	giant_table	range	PRIMARY,some_other_id	some_other_id	8	NULL	20	Using index condition; Using where; Using MRR
+DROP PROCEDURE p;
+DROP TABLE giant_table, t1;
 set optimizer_switch=default;

mysql-test/r/range_icp.result

@@ -2494,4 +2494,60 @@ KEY(a(1))
 SELECT 1 FROM t1 WHERE a <> 'a' OR a <> "";
 1
 DROP TABLE t1;
+#
+# Bug #23259872: OPTIMIZER CHOOSES TO USE NON PRIMARY
+#                INDEX, EVEN THOUGH COST IS HIGHER
+#
+CREATE TABLE `giant_table` (
+`id` int(11) NOT NULL AUTO_INCREMENT,
+`one_id` int(11) NOT NULL,
+`other_id` bigint(20) NOT NULL DEFAULT '0',
+`some_other_id` int(11) DEFAULT 0 NOT NULL,
+`something` double NOT NULL DEFAULT '0',
+`comment` text COLLATE utf8_unicode_ci,
+`flags` int(11) NOT NULL DEFAULT '0',
+`time_created` int(11) NOT NULL DEFAULT '0',
+PRIMARY KEY (`id`),
+KEY `time_created` (`time_created`),
+KEY `some_other_id` (`some_other_id`),
+KEY `one_other_idx` (`one_id`,`other_id`),
+KEY `other_id` (`other_id`,`time_created`)
+) ENGINE=InnoDB AUTO_INCREMENT=101651329
+DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci ;
+CREATE TABLE t1 (c1 INT);
+INSERT INTO t1 VALUES (66136540), (68983250), (89627210), (77869520),
+(82543190), (67538270), (77282760), (77908170),
+(70923370), (68066360);
+CREATE PROCEDURE p()
+BEGIN
+SET @x = 1;
+REPEAT
+INSERT INTO giant_table(id,one_id)
+SELECT c1 + @x, 0
+FROM t1
+WHERE c1 IN (66136540, 68985250, 89627210, 77869520 , 82543190, 67538270,
+77282760, 77908170, 70923370, 68066360);
+SET @x =  @x + 1;
+UNTIL @x > 30 END REPEAT;
+END $
+CALL p();
+SELECT count(*) FROM giant_table;
+count(*)
+270
+INSERT INTO giant_table (id,one_id) VALUES (66136539, 0), (68983258,1),
+(89628210,1), (77869520,2);
+INSERT INTO giant_table (id,one_id, some_other_id) VALUES(84673401, 0, 1),
+(61069031, 1, 1);
+EXPLAIN SELECT id, something, comment, time_created, one_id, other_id,
+some_other_id, flags
+FROM giant_table
+WHERE id IN (66136539, 68983258, 89628210, 77869520, 82543198, 67538272,
+84673401, 61069031, 68214385, 77282865, 76991297, 64569216,
+89481638, 74534074, 70396537, 80076375, 63308530, 77908270,
+70923271, 68066180)
+AND (giant_table.flags & 0x01) = 0 AND giant_table.some_other_id = 0;
+id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
+1	SIMPLE	giant_table	range	PRIMARY,some_other_id	some_other_id	8	NULL	20	Using index condition; Using where
+DROP PROCEDURE p;
+DROP TABLE giant_table, t1;
 set optimizer_switch=default;

mysql-test/r/range_icp_mrr.result

@@ -2494,4 +2494,60 @@ KEY(a(1))
 SELECT 1 FROM t1 WHERE a <> 'a' OR a <> "";
 1
 DROP TABLE t1;
+#
+# Bug #23259872: OPTIMIZER CHOOSES TO USE NON PRIMARY
+#                INDEX, EVEN THOUGH COST IS HIGHER
+#
+CREATE TABLE `giant_table` (
+`id` int(11) NOT NULL AUTO_INCREMENT,
+`one_id` int(11) NOT NULL,
+`other_id` bigint(20) NOT NULL DEFAULT '0',
+`some_other_id` int(11) DEFAULT 0 NOT NULL,
+`something` double NOT NULL DEFAULT '0',
+`comment` text COLLATE utf8_unicode_ci,
+`flags` int(11) NOT NULL DEFAULT '0',
+`time_created` int(11) NOT NULL DEFAULT '0',
+PRIMARY KEY (`id`),
+KEY `time_created` (`time_created`),
+KEY `some_other_id` (`some_other_id`),
+KEY `one_other_idx` (`one_id`,`other_id`),
+KEY `other_id` (`other_id`,`time_created`)
+) ENGINE=InnoDB AUTO_INCREMENT=101651329
+DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci ;
+CREATE TABLE t1 (c1 INT);
+INSERT INTO t1 VALUES (66136540), (68983250), (89627210), (77869520),
+(82543190), (67538270), (77282760), (77908170),
+(70923370), (68066360);
+CREATE PROCEDURE p()
+BEGIN
+SET @x = 1;
+REPEAT
+INSERT INTO giant_table(id,one_id)
+SELECT c1 + @x, 0
+FROM t1
+WHERE c1 IN (66136540, 68985250, 89627210, 77869520 , 82543190, 67538270,
+77282760, 77908170, 70923370, 68066360);
+SET @x =  @x + 1;
+UNTIL @x > 30 END REPEAT;
+END $
+CALL p();
+SELECT count(*) FROM giant_table;
+count(*)
+270
+INSERT INTO giant_table (id,one_id) VALUES (66136539, 0), (68983258,1),
+(89628210,1), (77869520,2);
+INSERT INTO giant_table (id,one_id, some_other_id) VALUES(84673401, 0, 1),
+(61069031, 1, 1);
+EXPLAIN SELECT id, something, comment, time_created, one_id, other_id,
+some_other_id, flags
+FROM giant_table
+WHERE id IN (66136539, 68983258, 89628210, 77869520, 82543198, 67538272,
+84673401, 61069031, 68214385, 77282865, 76991297, 64569216,
+89481638, 74534074, 70396537, 80076375, 63308530, 77908270,
+70923271, 68066180)
+AND (giant_table.flags & 0x01) = 0 AND giant_table.some_other_id = 0;
+id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
+1	SIMPLE	giant_table	range	PRIMARY,some_other_id	some_other_id	8	NULL	20	Using index condition; Using where; Using MRR
+DROP PROCEDURE p;
+DROP TABLE giant_table, t1;
 set optimizer_switch=default;

mysql-test/r/range_mrr.result

@@ -2494,4 +2494,60 @@ KEY(a(1))
 SELECT 1 FROM t1 WHERE a <> 'a' OR a <> "";
 1
 DROP TABLE t1;
+#
+# Bug #23259872: OPTIMIZER CHOOSES TO USE NON PRIMARY
+#                INDEX, EVEN THOUGH COST IS HIGHER
+#
+CREATE TABLE `giant_table` (
+`id` int(11) NOT NULL AUTO_INCREMENT,
+`one_id` int(11) NOT NULL,
+`other_id` bigint(20) NOT NULL DEFAULT '0',
+`some_other_id` int(11) DEFAULT 0 NOT NULL,
+`something` double NOT NULL DEFAULT '0',
+`comment` text COLLATE utf8_unicode_ci,
+`flags` int(11) NOT NULL DEFAULT '0',
+`time_created` int(11) NOT NULL DEFAULT '0',
+PRIMARY KEY (`id`),
+KEY `time_created` (`time_created`),
+KEY `some_other_id` (`some_other_id`),
+KEY `one_other_idx` (`one_id`,`other_id`),
+KEY `other_id` (`other_id`,`time_created`)
+) ENGINE=InnoDB AUTO_INCREMENT=101651329
+DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci ;
+CREATE TABLE t1 (c1 INT);
+INSERT INTO t1 VALUES (66136540), (68983250), (89627210), (77869520),
+(82543190), (67538270), (77282760), (77908170),
+(70923370), (68066360);
+CREATE PROCEDURE p()
+BEGIN
+SET @x = 1;
+REPEAT
+INSERT INTO giant_table(id,one_id)
+SELECT c1 + @x, 0
+FROM t1
+WHERE c1 IN (66136540, 68985250, 89627210, 77869520 , 82543190, 67538270,
+77282760, 77908170, 70923370, 68066360);
+SET @x =  @x + 1;
+UNTIL @x > 30 END REPEAT;
+END $
+CALL p();
+SELECT count(*) FROM giant_table;
+count(*)
+270
+INSERT INTO giant_table (id,one_id) VALUES (66136539, 0), (68983258,1),
+(89628210,1), (77869520,2);
+INSERT INTO giant_table (id,one_id, some_other_id) VALUES(84673401, 0, 1),
+(61069031, 1, 1);
+EXPLAIN SELECT id, something, comment, time_created, one_id, other_id,
+some_other_id, flags
+FROM giant_table
+WHERE id IN (66136539, 68983258, 89628210, 77869520, 82543198, 67538272,
+84673401, 61069031, 68214385, 77282865, 76991297, 64569216,
+89481638, 74534074, 70396537, 80076375, 63308530, 77908270,
+70923271, 68066180)
+AND (giant_table.flags & 0x01) = 0 AND giant_table.some_other_id = 0;
+id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
+1	SIMPLE	giant_table	range	PRIMARY,some_other_id	some_other_id	8	NULL	20	Using where; Using MRR
+DROP PROCEDURE p;
+DROP TABLE giant_table, t1;
 set optimizer_switch=default;

mysql-test/r/range_mrr_cost.result

@@ -2494,4 +2494,60 @@ KEY(a(1))
 SELECT 1 FROM t1 WHERE a <> 'a' OR a <> "";
 1
 DROP TABLE t1;
+#
+# Bug #23259872: OPTIMIZER CHOOSES TO USE NON PRIMARY
+#                INDEX, EVEN THOUGH COST IS HIGHER
+#
+CREATE TABLE `giant_table` (
+`id` int(11) NOT NULL AUTO_INCREMENT,
+`one_id` int(11) NOT NULL,
+`other_id` bigint(20) NOT NULL DEFAULT '0',
+`some_other_id` int(11) DEFAULT 0 NOT NULL,
+`something` double NOT NULL DEFAULT '0',
+`comment` text COLLATE utf8_unicode_ci,
+`flags` int(11) NOT NULL DEFAULT '0',
+`time_created` int(11) NOT NULL DEFAULT '0',
+PRIMARY KEY (`id`),
+KEY `time_created` (`time_created`),
+KEY `some_other_id` (`some_other_id`),
+KEY `one_other_idx` (`one_id`,`other_id`),
+KEY `other_id` (`other_id`,`time_created`)
+) ENGINE=InnoDB AUTO_INCREMENT=101651329
+DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci ;
+CREATE TABLE t1 (c1 INT);
+INSERT INTO t1 VALUES (66136540), (68983250), (89627210), (77869520),
+(82543190), (67538270), (77282760), (77908170),
+(70923370), (68066360);
+CREATE PROCEDURE p()
+BEGIN
+SET @x = 1;
+REPEAT
+INSERT INTO giant_table(id,one_id)
+SELECT c1 + @x, 0
+FROM t1
+WHERE c1 IN (66136540, 68985250, 89627210, 77869520 , 82543190, 67538270,
+77282760, 77908170, 70923370, 68066360);
+SET @x =  @x + 1;
+UNTIL @x > 30 END REPEAT;
+END $
+CALL p();
+SELECT count(*) FROM giant_table;
+count(*)
+270
+INSERT INTO giant_table (id,one_id) VALUES (66136539, 0), (68983258,1),
+(89628210,1), (77869520,2);
+INSERT INTO giant_table (id,one_id, some_other_id) VALUES(84673401, 0, 1),
+(61069031, 1, 1);
+EXPLAIN SELECT id, something, comment, time_created, one_id, other_id,
+some_other_id, flags
+FROM giant_table
+WHERE id IN (66136539, 68983258, 89628210, 77869520, 82543198, 67538272,
+84673401, 61069031, 68214385, 77282865, 76991297, 64569216,
+89481638, 74534074, 70396537, 80076375, 63308530, 77908270,
+70923271, 68066180)
+AND (giant_table.flags & 0x01) = 0 AND giant_table.some_other_id = 0;
+id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
+1	SIMPLE	giant_table	range	PRIMARY,some_other_id	some_other_id	8	NULL	20	Using where
+DROP PROCEDURE p;
+DROP TABLE giant_table, t1;
 set optimizer_switch=default;