thriftbp: Add a counter for opener calls in client pool

[fishy]

We did some work previously to try to move dns out of hot path as much as possible in thrift client pool, by doing background refreshes, etc.. But there are always a (hopefully small) portion of client calls would still do dns on hot path, and we lack observability into how often that actually happens.

To get the precise data would require some breaking changes to clientpool package (to pass in required prometheus labels), so adding a counter to the opener call instead, which is the next best thing, as in we only need to ignore the spikes when a new pod comes up and need to fill in the initial clients when initialize a client pool.

[kylelemons]

How do we know this ignores case 1? do you have a way to ignore case 1 programmatically or is it more of a "we'll ignore any spikes we see during a rollout" or more "we'll use the prometheus rate bug (ignoring anything before the first sample) in our favor"?

[fishy]

How do we know this ignores case 1? do you have a way to ignore case 1 programmatically or is it more of a "we'll ignore any spikes we see during a rollout" or more "we'll use the prometheus rate bug (ignoring anything before the first sample) in our favor"?

Oh I didn't even realize the first scrape thing can be used in our favor in this case :)

What I meant is basically a caveat on how to read/interpret the data it presents (e.g. if you see a spike, double check whether you have a new pod started around that time)