✅ Customization Switch model ID routing at runtime with profiles Run concurrent models with a custom DSL swap matrix (#643)
Almost all configuration settings are optional and can be added one step at a time:
Advanced features matrix to run concurrent models with a custom swap logic DSL hooks to run things on startup macros reusable snippets
How does llama-swap work? When a request is made to an OpenAI compatible endpoint, llama-swap will extract the model value and load the appropriate server configuration to serve it. If the wrong upstream server is running, it will be replaced with the correct one. This is where the "swap" part comes in. The upstream server is automatically swapped to handle the request correctly.
In the most basic configuration llama-swap handles one model at a time. For more advanced use cases, using a matrix allows multiple models to be loaded at the same time. You have complete control over how your system resources are used.
$ sudo dmesg [108206.360152] usb 1-2: new full-speed USB device number 12 using xhci_hcd [108206.487718] usb 1-2: New USB device found, idVendor=303a, idProduct=1001, bcdDevice= 1.01 [108206.487738] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [108206.487745] usb 1-2: Product: USB JTAG/serial debug unit [108206.487750] usb 1-2: Manufacturer: Espressif [108206.487756] usb 1-2: SerialNumber: 98:A3:16:E3:50:EC [108206.492137] cdc_acm 1-2:1.0: ttyACM0: USB ACM device
$ lsusb Bus 001 Device 021: ID 303a:1001 Espressif USB JTAG/serial debug unit
사진 기준 왼쪽 USB-C (TTL to UART)
$ sudo dmesg [108220.479842] usb 1-2: USB disconnect, device number 12 [108221.700959] usb 1-2: new full-speed USB device number 13 using xhci_hcd [108221.829581] usb 1-2: New USB device found, idVendor=1a86, idProduct=55d3, bcdDevice= 4.45 [108221.829604] usb 1-2: New USB device strings: Mfr=0, Product=2, SerialNumber=3 [108221.829614] usb 1-2: Product: USB Single Serial [108221.829621] usb 1-2: SerialNumber: 5A46054847 [108221.834111] cdc_acm 1-2:1.0: ttyACM0: USB ACM device
$ lsusb Bus 001 Device 020: ID 1a86:55d3 QinHeng Electronics USB Single Serial