-
Notifications
You must be signed in to change notification settings - Fork 222
Enormous performance regression by PR #482 when prepared_statements is set to true #534
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
45 TPS is very suspect. Something is definitely wrong. I ran a benchmark on my end, and as is with all benchmarks, it had a different result:
Latest OS: Mac OS Ventura
pgbench:
Happy to help debug further. Could you please:
Thanks! P.S. Ran your benchmark as well:
|
More precise reproduction steps I was trying with this slightly modified [general]
host = "127.0.0.1"
port = 6433
admin_username = "pgcat"
admin_password = "pgcat"
prepared_statements = true
[pools.pgml.users.0]
username = "postgres"
password = "postgres"
pool_size = 10
min_pool_size = 1
pool_mode = "transaction"
[pools.pgml.shards.0]
servers = [
["127.0.0.1", 5432, "primary"]
]
database = "postgres" Command to run pgcat: RUST_LOG=info cargo run --release -- pgcat.minimal.toml This command then results in ~45TPS PGDATABASE=pgml PGUSER=postgres PGPASSWORD=postgres pgbench --select-only -T 10 -P 1 -p 6433 --protocol simple If I remove the Hardware: 16 core VM on Azure, and my own desktop. Ubuntu 20.04 and Ubuntu 22.04 respectively (desktop runs Ubuntu in WSL). CPU on both is some Intel 64bit. My desktop shows this with Details
|
Just to confirm one thing, are you pgbench and pgcat on the same machine or are you running pgbench on your machine and pgcat on a VM in Azure? |
Looking at the default config (that one is huge), it seems to be using session pooling for the |
Oh hmm, that was my mistake. Let me test with |
A little slower, but still not 45 TPS:
Double-checked the config this time:
|
If I change the default config to use transaction mode for Just double checking, you are on the latest commit of the |
Some extra info: CPU of |
Cool will do. I developed that PR on my Ubuntu (Ryzen 5800X), so I don't think it's ARM vs Intel vs Mac silicon either. Could you enable tracing log level and maybe also a profiler (e.g. I double checked, I'm on latest main version of the code. |
strace slow:
strace fast:
So it seems it's waiting much longer on some
|
How do you get pgcat to output tracing logs? I tried a few things and grepped trough the codebase, but couldn't find a solution that worked. |
For more options:
|
Cache addition is the most suspect, although I don't know why it would work fine on my end and not on yours. Could you share your stats?
I'm looking mostly for Thanks! |
When running with
When running with |
trace level logs of the slow thing: Details
|
with --protocol simple, fast: Details
with --protocol simple, slow: Details2023-08-01T18:23:46.384718Z DEBUG pgcat::pool: Address { id: 0, host: "127.0.0.1", port: 5432, shard: 0, database: "postgres", role: Primary, replica_number: 0, address_index: 0, username: "postgres", pool_name: "pgml", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 131, query_count: 131, bytes_received: 8737, bytes_sent: 9227, xact_time: 0, query_time: 3041, wait_time: 8486, errors: 0 }, current: AddressStatFields { xact_count: 131, query_count: 131, bytes_received: 8694, bytes_sent: 9191, xact_time: 0, query_time: 3041, wait_time: 8486, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 2, bytes_sent: 2, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false } } is ok |
It seems to be between sending something to the server, and waiting for the response. It's waiting A LOT longer there in the slow case. |
I can finally reproduce this on my Ubuntu machine. The culprit is |
Oh wow. This was really a blunder. Fixed in aefcf42. Could you confirm on your end? Thanks! The cause was the prepared statement maintenance cache loop issuing a Flush to Postgres on each iteration (each transaction). This only happened in transaction mode, which explains why the performance tanked in transaction mode but not in session mode. |
Yes pulling the latest |
Seem like a patch release is probably in order though. |
Describe the bug
I wanted to try out prepared statement support and measure its performance in pgcat (essentially reproducing the blog). But when running
pgbench --select-only
on the main branch I was only able to get ~50 TPS.To Reproduce
Steps to reproduce the behavior:
git checkout main
prepared_statements = true
to the minimal configRUST_LOG=info cargo run --release -- pgcat.minimal.toml
pgbench --select-only -T 10 -P 1 -p 6433 --protocol prepared
Expected behavior
Get somewhere in the order of 2000-20000 TPS. But instead I get between 20-50TPS. The worst part is that this even happens when using
--protocol simple
. Only if you enableprepared_statements = true
from the config, performance of--protocol simple
returns to normal levels.Screenshots

I narrowed the regression down commit: 4b78af9
With commit: 4b78af9
With commit right before: 73500c0

Desktop (please complete the following information):
Additional info
The reason I was testing pgcat its performance was because I wanted to compare PgBouncer its upcoming prepared statement support (pgbouncer/pgbouncer#845) to pgcat and Odyssey. But pgcat its perf was so much lower than expected that something obviously seemed wrong.
The text was updated successfully, but these errors were encountered: