llama-swap을 써도 vllm과 같이 대규모 서빙은 취약한데
그걸 조금 완화시켜줄 옵션이라고 하면되려나?
matix 혹은 group이라고 불렀다는것 같은데 써봐야겠다.
| ✅ Customization Switch model ID routing at runtime with profiles Run concurrent models with a custom DSL swap matrix (#643) Almost all configuration settings are optional and can be added one step at a time: Advanced features matrix to run concurrent models with a custom swap logic DSL hooks to run things on startup macros reusable snippets How does llama-swap work? When a request is made to an OpenAI compatible endpoint, llama-swap will extract the model value and load the appropriate server configuration to serve it. If the wrong upstream server is running, it will be replaced with the correct one. This is where the "swap" part comes in. The upstream server is automatically swapped to handle the request correctly. In the most basic configuration llama-swap handles one model at a time. For more advanced use cases, using a matrix allows multiple models to be loaded at the same time. You have complete control over how your system resources are used. |
[링크 : https://github.com/mostlygeek/llama-swap]
| # yaml-language-server: $schema=https://raw.githubusercontent.com/mostlygeek/llama-swap/refs/heads/main/config-schema.json sendLoadingState: true models: "load-1s-A": cmd: build/fake-model -listen localhost:${PORT} -load 1s -tokens 50 -tps 25 "load-1s-B": cmd: build/fake-model -listen localhost:${PORT} -load 1s -tokens 50 -tps 25 "load-1s-C": cmd: build/fake-model -listen localhost:${PORT} -load 1s -tokens 50 -tps 25 "load-1s-D": cmd: build/fake-model -listen localhost:${PORT} -load 1s -tokens 50 -tps 25 "load-1s-E": cmd: build/fake-model -listen localhost:${PORT} -load 1s -tokens 50 -tps 25 "load-1s-F": cmd: build/fake-model -listen localhost:${PORT} -load 1s -tokens 50 -tps 25 matrix: vars: A: load-1s-A B: load-1s-B C: load-1s-C D: load-1s-D E: load-1s-E F: load-1s-F sets: s1: (A & B & C) | (D & E) | F |
[링크 : https://gist.github.com/mostlygeek/106f0145ff7d71136f4cfb1af58526ce]
'모종의 음모 > ai 프로그램' 카테고리의 다른 글
| stable diffusion model 이란 (0) | 2026.08.28 |
|---|---|
| claude code with local LLM (0) | 2026.08.24 |
| Qwen3.8-27B-UD-IQ2_XXS / 1080ti 11GB (0) | 2026.08.19 |
| lambda.ai gpu 클라우드 (0) | 2026.08.11 |
| elice cloud - gpu / npu 클라우드 (0) | 2026.08.11 |
