콘솔로 하니 이상하게 종료시 엔터가 미친듯이 나와서 웹으로 시도.
vulkan 인데도 vulkan 가속을 못 받는지 cpu 100%로 돈다.
최대 성능으로 가동
| $ sudo nvpmodel -q NVPM WARN: fan mode is not set! NV Power Mode: MAXN 0 |
docker 로 서비스 실행
| $ sudo docker run -p 8080:8080 -v /home/jetson/llm:/models ghcr.io/ggml-org/llama.cpp:server-vulkan --chat-template-kwargs "{\"enable_thinking\": false}" -m /models/Qwen3.5-0.8B-Q4_0.gguf -c 4096 --port 8080 --host 0.0.0.0 warn: LLAMA_ARG_HOST environment variable is set, but will be overwritten by command line argument --host 0.00.136.037 W Setting 'enable_thinking' via --chat-template-kwargs is deprecated. Use --reasoning on / --reasoning off instead. 0.00.136.924 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg) 0.00.138.536 W srv llama_server: ----------------- 0.00.138.551 W srv llama_server: CORS is set to allow all origins ('*') and no API key is set 0.00.138.552 W srv llama_server: this can be a security risk (cross-origin attacks) 0.00.138.553 W srv llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655 0.00.138.554 W srv llama_server: ----------------- 0.00.139.997 I srv load_model: loading model '/models/Qwen3.5-0.8B-Q4_0.gguf' 0.04.582.967 I srv load_model: initializing, n_slots = 4, n_ctx_slot = 4096, kv_unified = 'true' 0.04.681.224 I srv llama_server: model loaded 0.04.681.254 I srv llama_server: listening on http://0.0.0.0:8080 0.19.911.095 I slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1 0.19.911.286 I slot launch_slot_: id 3 | task 0 | processing task, is_child = 0 0.22.612.244 I slot print_timing: id 3 | task 0 | prompt eval time = 1093.85 ms / 13 tokens ( 84.14 ms per token, 11.88 tokens per second) 0.22.612.263 I slot print_timing: id 3 | task 0 | eval time = 1606.96 ms / 12 tokens ( 133.91 ms per token, 7.47 tokens per second) 0.22.612.266 I slot print_timing: id 3 | task 0 | total time = 2700.81 ms / 25 tokens 0.22.612.276 I slot print_timing: id 3 | task 0 | graphs reused = 11 0.22.612.345 I slot release: id 3 | task 0 | stop processing: n_tokens = 24, truncated = 0 0.32.329.433 I slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.243 (> 0.100 thold), f_keep = 0.375 0.32.406.612 I slot launch_slot_: id 3 | task 14 | processing task, is_child = 0 0.37.382.135 I slot print_timing: id 3 | task 14 | prompt eval time = 2254.95 ms / 28 tokens ( 80.53 ms per token, 12.42 tokens per second) 0.37.382.154 I slot print_timing: id 3 | task 14 | eval time = 2720.41 ms / 18 tokens ( 151.13 ms per token, 6.62 tokens per second) 0.37.382.157 I slot print_timing: id 3 | task 14 | total time = 4975.37 ms / 46 tokens 0.37.382.160 I slot print_timing: id 3 | task 14 | graphs reused = 27 0.37.382.205 I slot release: id 3 | task 14 | stop processing: n_tokens = 54, truncated = 0 0.52.606.713 I slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.471 (> 0.100 thold), f_keep = 0.611 0.52.606.847 I slot launch_slot_: id 3 | task 35 | processing task, is_child = 0 1.10.385.311 I slot print_timing: id 3 | task 35 | n_decoded = 100, tg = 6.70 t/s, tg_3s = 6.70 t/s 1.11.031.932 I slot print_timing: id 3 | task 35 | prompt eval time = 2847.89 ms / 37 tokens ( 76.97 ms per token, 12.99 tokens per second) 1.11.031.951 I slot print_timing: id 3 | task 35 | eval time = 15577.07 ms / 104 tokens ( 149.78 ms per token, 6.68 tokens per second) 1.11.031.954 I slot print_timing: id 3 | task 35 | total time = 18424.96 ms / 141 tokens 1.11.031.960 I slot print_timing: id 3 | task 35 | graphs reused = 129 1.11.032.011 I slot release: id 3 | task 35 | stop processing: n_tokens = 173, truncated = 0 3.50.029.790 I slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.333 (> 0.100 thold), f_keep = 0.382 3.50.175.603 I slot launch_slot_: id 3 | task 142 | processing task, is_child = 0 3.57.905.059 I slot print_timing: id 3 | task 142 | prompt processing, n_tokens = 105, progress = 0.86, t = 7.73 s / 13.58 tokens per second 3.59.618.519 I slot print_timing: id 3 | task 142 | prompt processing, n_tokens = 128, progress = 0.98, t = 9.44 s / 13.56 tokens per second 4.10.877.834 I slot print_timing: id 3 | task 142 | prompt eval time = 9819.15 ms / 132 tokens ( 74.39 ms per token, 13.44 tokens per second) 4.10.877.855 I slot print_timing: id 3 | task 142 | eval time = 10882.92 ms / 72 tokens ( 151.15 ms per token, 6.62 tokens per second) 4.10.877.858 I slot print_timing: id 3 | task 142 | total time = 20702.07 ms / 204 tokens 4.10.877.862 I slot print_timing: id 3 | task 142 | graphs reused = 198 4.10.877.944 I slot release: id 3 | task 142 | stop processing: n_tokens = 269, truncated = 0 |
cpu 에서 10 token/sec 미만으로 나온다.
| msec | token | token/sec |
| 2700.81 (안녕) | 25 | 9.25648231456489 |
| 4975.37 (안녕) | 46 | 9.24554354751506 |
| 18424.96 (너에 대해 소개) | 141 | 7.65266247525096 |
| 20702.07 (golang으로 hello world) | 204 | 9.85408705506261 |
'embeded > jetson' 카테고리의 다른 글
| jetson nano + llama.cpp vulkan (0) | 2026.07.21 |
|---|---|
| jetson nano jtop (0) | 2026.07.20 |
| jetson nano 과 llama.cpp (cuda 가속, 빌드 포기) (0) | 2026.07.20 |
| jetson nano devloper kit SD card image(A02) (0) | 2026.07.20 |
| jetracer donkey car (0) | 2026.04.20 |
