검색하다 보니 이상한(?) C1 C8 이라는 문구 발견, concurrent request 라고 되어있는데
결국은 동시에 응답을 해줄수 있게 돌리냐 아니냐의 차이인듯.
| Modes: C1 (single request) C8 (8 concurrent requests) |
| --max-concurrency¶ Maximum number of concurrent requests. This can be used to help simulate an environment where a higher level component is enforcing a maximum number of concurrent requests. While the --request-rate argument controls the rate at which requests are initiated, this argument will control how many are actually allowed to execute at a time. This means that when used in combination, the actual request rate may be lower than specified with --request-rate, if the server is not processing requests fast enough to keep up |
[링크 : https://docs.vllm.ai/en/stable/cli/bench/serve/#arguments]
| Multi-node multi-GPU using tensor parallel and pipeline parallel inference: if the model is too large for a single node, combine tensor parallelism with pipeline parallelism. Set tensor_parallel_size to the number of GPUs per node and pipeline_parallel_size to the number of nodes. For example, set tensor_parallel_size=8 and pipeline_parallel_size=2 when using 2 nodes with 8 GPUs per node. |
[링크 : https://docs.vllm.ai/en/stable/serving/parallelism_scaling/#single-node-deployment]
'프로그램 사용 > ai 프로그램' 카테고리의 다른 글
| llama.cpp 동시에 응답하게 하기 (0) | 2026.07.29 |
|---|---|
| vllm concurrent test/benchmark (0) | 2026.07.29 |
| nvidia thor - llama.cpp LLM (0) | 2026.07.24 |
| llama.cpp for arm64 (0) | 2026.07.20 |
| qwen3.5 0.8B (0) | 2026.07.19 |
