Skip to content

MySQL-MySQL incr stage: use mapping and filter like cause src dtle panic #1011

Closed
@asiroliu

Description

@asiroliu

Description

MySQL-MySQL incr stage: use mapping and filter like cause src dtle panic

Steps to reproduce the issue

  1. prepare data
sql> CREATE DATABASE IF NOT EXISTS `action_db_1`;
shell> sysbench /usr/share/sysbench/oltp_common.lua --mysql-host=172.100.9.3 --mysql-port=3306 --mysql-user=test --mysql-password=test --create_secondary=off --report-interval=10 --time=0 --mysql-db=action_db_1 --tables=1 --table_size=5000 prepare
  1. create dtle job
{
  "job_id": "mapping_where_column_map",
  "is_password_encrypted": false,
  "task_step_name": "all",
  "failover": true,
  "retry": 2,
  "src_task": {
    "task_name": "src",
    "node_id": "dac3495b-3713-16b1-123e-7f379372c596",
    "mysql_src_task_config": {
      "gtid": "",
      "binlog_relay": false
    },
    "drop_table_if_exists": false,
    "skip_create_db_table": false,
    "repl_chan_buffer_size": 120,
    "chunk_size": 2000,
    "group_max_size": 1,
    "group_timeout": 100,
    "connection_config": {
      "database_type": "MySQL",
      "host": "172.100.9.3",
      "port": 3306,
      "user": "test_src",
      "password": "test_src"
    },
    "replicate_do_db": [
      {
        "table_schema": "action_db_1",
        "table_schema_rename": "action_db_1_rename",
        "tables": [
          {
            "table_regex": "sbtest(1)",
            "table_rename": "sbtest$${1}_rename",
            "where": "c LIKE '1%'",
            "column_map_from": [
              "pad",
              "c",
              "k",
              "id"
            ]
          }
        ]
      }
    ]
  },
  "dest_task": {
    "task_name": "dest",
    "node_id": "d906fd26-c6a2-d14d-ba95-ae47185736c9",
    "mysql_dest_task_config": {
      "use_my_sql_dependency": false,
      "dependency_history_size": 2500,
      "parallel_workers": 32
    },
    "connection_config": {
      "database_type": "MySQL",
      "host": "172.100.9.1",
      "port": 3306,
      "user": "test_dest",
      "password": "test_dest"
    }
  }
}
  1. check data count
  2. execute sql on src MySQL
shell> sysbench /usr/share/sysbench/oltp_write_only.lua --mysql-host=172.100.9.3 --mysql-port=3306 --mysql-user=test --mysql-password=test --create_secondary=off --report-interval=10 --time=0 --mysql-db=action_db_1 --tables=1 --table_size=5000 --events=15000 --rand-type=uniform run
  1. check data count
# src MySQL
sql> SELECT COUNT(*) FROM sbtest1 WHERE c LIKE '1%';
{'COUNT(*)': 501}
# dest MySQL
sql> SELECT COUNT(*) FROM sbtest1_rename WHERE c LIKE '1%';
{'COUNT(*)': 461}
  1. just filter use like can't trigger panic

Describe the results you expected

The amount of data is consistent

Output of ./dtle version:**

9.9.9.9-master-72a4c06

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions