Problem

When running sysbench, the more data TiDB returns, the performance gap between TiProxy and HAProxy grows.

Test Result

1. Create a TiDB cluster with HAProxy and TiProxy, each of which has 2 CPU cores.
2. Run sysbench with --tables=10 --table-size=1000000 --threads=32 oltp_read_only --skip_trx=true --point_selects=0 --sum_ranges=0 --order_ranges=0 --distinct_ranges=0 --simple_ranges=1 --range_size={range_size}
3. Check the QPS and CPU of TiProxy and HAProxy.

As we can see, when the range size is 10, the performance of HAProxy is less than twice of TiProxy. But when the range size is 10000, the performance of HAProxy is almost 8 times of TiProxy.

Reason

In MySQL, each row is wrapped in a MySQL packet.

• TiProxy reads data, allocates memory, and writes data for each MySQL packet because TiProxy needs to check the status.
• HAProxy simply forwards data from the server to the client and doesn't need to parse MySQL packets.

Thus, TiProxy is more impacted by the row count.

Code Analysis

The flame graph when range size is 1000:

WritePacket and ReadPacket become the hot path, so this code should be optimized:

• WriteOnePacket calls 2 NewReader and that allocates memory twice.
• ReadPacket grows slice in data = append(data, buf), which is unncessary.
• var header [4]byte escapes to the heap because header is passed as a parameter to other functions.
• The read/write buffer size is 16K, which means for 1000 rows, it calls syscall read/write 8 times.
• io.ReadFull contains unnecessary boundary check, function calls and interface conversion.