Bug#22842538 BINLOG SCHEMA DISTRIBUTION TIMEOUT AND FAILS WHEN ANOTHER MYSQL NODE START

When another mysqld node is started, and joins (subscribe to)
the schema distribution protocol, another mysqld which is
waiting for a schema change to be distributed will timeout
during that wait. That happens as we incorrectly assumed
that the new arriving mysqld node would also 'ack' the
schema distribution. However, it arrived too late to be
a participant in it.

This patch fixes 3 issues all contributing to this failure:

a) There is a potential race between an 'inflight'
   subscribe event, and the start of a schema distribution.
   The subscribing node might or might not take part in the
   schema distribution, and its role is actually unknown at
   the point in time where the schema operation is started by
   the coordinator.
   The set of participating servers could only be determined
   when the Coordinator acks its own schema op: If the subscribe
   event arrived before it own schema up, then the subcribing
   node is a participant.
   This patch modifies the Coordinators ack to also modifying
   the acked slock_bitmap to clear the servers *not* participating.

b) check_wakeup_clients() called get_subcriber_bitmask() to get the
   current set of subscribers. However, 'self' was not included in the
   subscribers, which it always should be. Fixed this by letting
   Ndb_schema_dist_data::init() add 'own_nodeid' to subscribers.
   Furthermore, this enables us to clean up a couple of places
   where we used to add own_nodeid to the set retrieved from
   get_subscribers_bitmask().

c) handle_clear_slock() copied schema->slock into
   ndb_schema_object->slock_bitmap, thereby overwriting the intersect
   done as part of a). Changed the copy to do an intersect instead.

This patch also modifies several places where schema distribution
progress is printed:
- Always print more significant part of bitmask before the less significant.
- Adds some formating when printing the bitmasks.

Also removes a few clear of bitmasks immediately after an init,
which is redundant as ::init() also cleared it.

Author: Ole John Aske

mysql-test/suite/ndb_binlog/r/ndb_binlog_ddl_multi.result

@@ -282,3 +282,23 @@ mysqld-bin.000001	#	Query	2	#	drop database mysqltest
 mysqld-bin.000001	#	Query	2	#	use `test`; drop table `test`.`t1`
 mysqld-bin.000001	#	Query	2	#	use `test`; create table dummy (dummyk int primary key) engine = ndb
 mysqld-bin.000001	#	Query	2	#	use `test`; DROP TABLE `dummy` /* generated by server */
+CREATE TABLE progress(
+cnt int, stop_flag int
+) ENGINE NDB;
+insert into progress value(0,0);
+Start background load distributing schema changes.
+call p1();
+Restart mysqld 'server2'
+Checking for schema ops. still making progress
+Restart mysqld 'server2'
+Checking for schema ops. still making progress
+Restart mysqld 'server2'
+Checking for schema ops. still making progress
+Restart mysqld 'server2'
+Checking for schema ops. still making progress
+Stopping background load distributing schema changes.
+update progress set stop_flag=1;
+Wait for background schema distribution load to complete.
+Cleanup
+drop procedure p1;
+drop table progress;

mysql-test/suite/ndb_binlog/t/ndb_binlog_ddl_multi.test

@@ -250,3 +250,94 @@ enable_query_log;
 drop table dummy;
 --source include/show_binlog_events2.inc
 
+##########################
+# Bug#22842538:
+#
+#    New mysqld's joining the schema distribution may cause
+#    ongoing DDL operations to never complete -> timeout
+# 
+##########################
+connection server1;
+
+
+# Procedure p1 communicate through 'progress' table
+CREATE TABLE progress(
+  cnt int, stop_flag int
+) ENGINE NDB;
+
+
+# Create a procedure for producing a background load of
+# DDL operations. Any operations, like a CREATE TABLE, would
+# do. However, we have experienced that CREATE LOGFILE seems
+# to be most likely to trigger this bug (Due to timing?)
+#
+disable_query_log;
+delimiter %;
+create procedure p1()
+begin
+  declare done int default 0;
+  repeat
+    UPDATE progress set cnt=cnt+1;
+    COMMIT;
+    CREATE LOGFILE GROUP lg_1
+      ADD UNDOFILE 'undo_1.dat'
+      INITIAL_SIZE 4M
+      UNDO_BUFFER_SIZE 2M
+      ENGINE NDB;
+    UPDATE progress set cnt=cnt+1;
+    COMMIT;
+    DROP LOGFILE GROUP lg_1 ENGINE NDB;
+    SELECT stop_flag INTO done FROM progress;
+  until done end repeat;
+end%
+delimiter ;%
+enable_query_log;
+
+
+insert into progress value(0,0);
+
+--echo Start background load distributing schema changes.
+send call p1();
+
+connection server2;
+let $1 = 4;
+while ($1)
+{
+  # Ignore the warning generated by ndbcluster's binlog thread
+  # when cluster is restarted
+  --disable_query_log ONCE
+  call mtr.add_suppression("mysqld startup An incident event has been written");
+
+  --echo Restart mysqld 'server2'
+  let $mysqld_name=mysqld.2.1;
+  --source include/restart_mysqld.inc
+
+  --echo Checking for schema ops. still making progress
+  let $initial_cnt = `select cnt from progress`;
+  let $progress_cnt = $initial_cnt;
+
+  let $max_wait = 10;
+  while ($progress_cnt == $initial_cnt)
+  {
+    sleep 1;
+    dec $max_wait;
+    if ($max_wait == 0)
+    {
+      die Schema distribution timed out without progress;
+    }
+    let $progress_cnt = `select cnt from progress`;
+  }
+  dec $1;
+}
+
+--echo Stopping background load distributing schema changes.
+update progress set stop_flag=1;
+
+connection server1;
+
+--echo Wait for background schema distribution load to complete.
+reap;
+
+--echo Cleanup
+drop procedure p1;
+drop table progress;