[v0.2.3] Release Tracker

[WoosukKwon]

ETA: Nov 30th - Dec 2nd.

Major changes

• Refactoring on Worker, InputMetadata, and Attention
• Fix TP support for AWQ models
• Support Prometheus metrics
• Fix Baichuan & Baichuan 2

PRs to be merged before the release

• Chat Template Support chat template and echo for chat API #1756
• API causes slowdown in batch request handling #1707 (We have to solve AWQ perf first, which might be possible in time).
• Add support for prometheus metrics #1662 (use the new one instead)
• Add Production Metrics in Prometheus format #1890
• Fix num_gpus when TP > 1 #1852

[simon-mo]

I think we should absolutely include #1756 (and it's newer iteration). I'm also debugging #1707.

[simon-mo]

I will also add #1662

[WoosukKwon]

@simon-mo If I understand correctly, AWQ performance turns out to be orthogonal to the performance issue in #1707, is this right?

[simon-mo]

AWQ's bad prefill performance exacerbated the issue. The root issue was non-chunked prefill.

[simon-mo]

@WoosukKwon for the prometheus metrics, I made a new PR, please take a look.

[WoosukKwon]

Closed by #1903