allocator: high amount of CPU usage for storepool logging

[kvoli]

In an internal test cluster running 23.1.2 with 48 nodes we see that allocator logging is using 4% of CPU (32vCPU).

The CPU usage is from here:

cockroach/pkg/kv/kvserver/allocator/storepool/store_pool.go

Lines 1421 to 1427 in fdf04ff

	log.VEventf(ctx, 3,
	"s%d is a live target, candidate for rebalancing", targetStoreID)
	return true
	case storeStatusDead, storeStatusUnknown, storeStatusDecommissioning, storeStatusSuspect, storeStatusDraining:
	log.VEventf(ctx, 3,
	"not considering non-live store s%d (%v)", targetStoreID, status)
	return false

When grabbing another profile the CPU time disappears. The suspect part is the number of string operations - which eat the majority of CPU shown above.

Details

It appears possible redact printing status is eating most of the CPU, however I'm not certain. Also these log statements are at v=3, meaning that tracing must have been enabled consistently - which I suppose makes sense given:

cockroach/pkg/kv/kvserver/replicate_queue.go

Lines 721 to 723 in fdf04ff

	ctx, sp := tracing.EnsureChildSpan(ctx, rq.Tracer, "process replica",
	tracing.WithRecording(tracingpb.RecordingVerbose))
	defer sp.Finish()

Jira issue: CRDB-28774