(APIServer pid=1) INFO 07-28 03:16:02 [loggers.py:259] Engine 000: Avg prompt throughput: 647.4 tokens/s, Avg generation throughput: 7.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:56010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:56016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:16:12 [loggers.py:259] Engine 000: Avg prompt throughput: 20.4 tokens/s, Avg generation throughput: 3.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:37160 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:37160 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:37162 - "GET /v1/models HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:16:22 [loggers.py:259] Engine 000: Avg prompt throughput: 643.2 tokens/s, Avg generation throughput: 36.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:54470 - "GET /v1/models HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:16:32 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 192.168.40.209:36304 - "GET /metrics HTTP/1.1" 200 OK (APIServer pid=1) INFO: 192.168.40.209:36304 - "GET /favicon.ico HTTP/1.1" 404 Not Found (APIServer pid=1) INFO: 127.0.0.1:53220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:53220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:19:12 [loggers.py:259] Engine 000: Avg prompt throughput: 666.3 tokens/s, Avg generation throughput: 18.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:19:22 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:57598 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:19:32 [loggers.py:259] Engine 000: Avg prompt throughput: 622.3 tokens/s, Avg generation throughput: 39.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:57598 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:19:42 [loggers.py:259] Engine 000: Avg prompt throughput: 91.0 tokens/s, Avg generation throughput: 7.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:19:52 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:56452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:56450 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:56450 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:56452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:20:52 [loggers.py:259] Engine 000: Avg prompt throughput: 668.1 tokens/s, Avg generation throughput: 14.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:51276 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:21:02 [loggers.py:259] Engine 000: Avg prompt throughput: 597.2 tokens/s, Avg generation throughput: 25.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:51276 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:21:12 [loggers.py:259] Engine 000: Avg prompt throughput: 112.8 tokens/s, Avg generation throughput: 71.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:21:22 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:43142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:21:32 [loggers.py:259] Engine 000: Avg prompt throughput: 686.0 tokens/s, Avg generation throughput: 28.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:21:42 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 132.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:21:52 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 127.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:40238 - "GET /v1/models HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:40254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:40254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:22:02 [loggers.py:259] Engine 000: Avg prompt throughput: 1424.5 tokens/s, Avg generation throughput: 56.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:52062 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:52062 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:22:12 [loggers.py:259] Engine 000: Avg prompt throughput: 1407.0 tokens/s, Avg generation throughput: 26.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:38124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:38124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:22:22 [loggers.py:259] Engine 000: Avg prompt throughput: 1139.7 tokens/s, Avg generation throughput: 24.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:38124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:22:32 [loggers.py:259] Engine 000: Avg prompt throughput: 1053.3 tokens/s, Avg generation throughput: 110.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:40160 - "GET /v1/models HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:40174 - "GET /v1/models HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:22:42 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 129.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:37106 - "GET /v1/models HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:38124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:37108 - "GET /v1/models HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:38124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:22:52 [loggers.py:259] Engine 000: Avg prompt throughput: 1634.7 tokens/s, Avg generation throughput: 59.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:23:02 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 127.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:38124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:23:12 [loggers.py:259] Engine 000: Avg prompt throughput: 535.5 tokens/s, Avg generation throughput: 116.7 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:23:22 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:49210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:26:32 [loggers.py:259] Engine 000: Avg prompt throughput: 1562.3 tokens/s, Avg generation throughput: 111.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:26:42 [loggers.py:259] Engine 000: Avg prompt throughput: 743.5 tokens/s, Avg generation throughput: 121.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:54226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:49210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:54226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:49210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:26:52 [loggers.py:259] Engine 000: Avg prompt throughput: 4021.5 tokens/s, Avg generation throughput: 92.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:27:02 [loggers.py:259] Engine 000: Avg prompt throughput: 369.0 tokens/s, Avg generation throughput: 122.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:49210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:49210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:27:12 [loggers.py:259] Engine 000: Avg prompt throughput: 1909.2 tokens/s, Avg generation throughput: 71.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:27:22 [loggers.py:259] Engine 000: Avg prompt throughput: 331.3 tokens/s, Avg generation throughput: 120.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:49210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:49210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:27:32 [loggers.py:259] Engine 000: Avg prompt throughput: 2077.2 tokens/s, Avg generation throughput: 91.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:27:42 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 121.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:42812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:49210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:27:52 [loggers.py:259] Engine 000: Avg prompt throughput: 2832.4 tokens/s, Avg generation throughput: 110.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:56048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:42812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:28:02 [loggers.py:259] Engine 000: Avg prompt throughput: 3328.9 tokens/s, Avg generation throughput: 132.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:42812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:28:12 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:28:22 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:43420 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:43406 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:28:42 [loggers.py:259] Engine 000: Avg prompt throughput: 628.9 tokens/s, Avg generation throughput: 25.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:43406 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:43420 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:43406 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:28:52 [loggers.py:259] Engine 000: Avg prompt throughput: 764.8 tokens/s, Avg generation throughput: 85.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:29:02 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 132.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:43406 - "POST /v1/chat/completions HTTP/1.1" 200 OK |