128k context length에 2기가 VRAM을 냠냠
| $ ./llama-b8925/llama-cli -m model/gemma4-e4b/gemma-4-E4B-it-Q4_K_M.gguf --verbose llama_kv_cache: size = 2048.00 MiB (131072 cells, 4 layers, 1/1 seqs), K (f16): 1024.00 MiB, V (f16): 1024.00 MiB llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 512 llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 512 llama_kv_cache_iswa: creating SWA KV cache, size = 1024 cells |
k 만 q4로 하니 288MB! 256MB 보단 약간 큰데 아무튼 대충~ 1/4 로 줄었다.
| $ ./llama-b8925/llama-cli -m model/gemma4-e4b/gemma-4-E4B-it-Q4_K_M.gguf --verbose -ctk q4_0 --ctx-size 131072 llama_kv_cache: size = 1312.00 MiB (131072 cells, 4 layers, 1/1 seqs), K (q4_0): 288.00 MiB, V (f16): 1024.00 MiB llama_kv_cache: attn_rot_k = 1, n_embd_head_k_all = 512 llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 512 llama_kv_cache_iswa: creating SWA KV cache, size = 1024 cells |
kv를 q4로 하니 대충 512MB 근처.
| $ ./llama-b8925/llama-cli -m model/gemma4-e4b/gemma-4-E4B-it-Q4_K_M.gguf --verbose -ctk q4_0 -ctv q4_0 --ctx-size 131072 llama_kv_cache: size = 576.00 MiB (131072 cells, 4 layers, 1/1 seqs), K (q4_0): 288.00 MiB, V (q4_0): 288.00 MiB llama_kv_cache: attn_rot_k = 1, n_embd_head_k_all = 512 llama_kv_cache: attn_rot_v = 1, n_embd_head_k_all = 512 llama_kv_cache_iswa: creating SWA KV cache, size = 1024 cells |
+
2026.05.29
verbose로는 토큰별로 로그가 나와서 성능 저하가 있긴한데
| verbose | none | ctk | ctk ctv | ctv |
| 단문 | 50.13 t/s | 58.27 t/s | 43.67 t/s | 51.84 t/s |
| 중문 | 50.76 t/s | 56.41 t/s | 43.66 t/s | 50.37 t/s |
| 장문 | 49.70 t/s | 54.66 t/s | 42.75 t/s | 45.81 t/s |
생각외로 양자화 한거랑 안한거랑 차이가 별로 없다.
1080 이라 q4를 지원하지 않아서 그런걸지도?
| - | none | ctk | ctk ctv | ctv |
| 단문 | 57.48 t/s | 57.03 t/s | 51.67 t/s | 52.35 t/s |
| 중문 | 59.56 t/s | 56.90 t/s | 50.20 t/s | 49.95 t/s |
| 장문 | 53.83 t/s | 54.82 t/s | 46.28 t/s | 44.93 t/s |
안녕?
너에 대해서 소개해줘
파이썬으로 셀레니움을 통해 웹을 서칭하고 텍스트만 추출하고 makrdown 으로 변환후 md 파일과 pdf로 저장하는 기능을 구현해줘
/mnt/Downloads/llama-b9305/llama-server --model /mnt/Downloads/model/gemma4-e4b/gemma-4-E4B-it-Q4_K_M.gguf -mm ./model/gemma4-e4b/mmproj-F16.gguf -sm none --ctx-size 131072 --verbose --host 0.0.0.0
gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
63 tokens
1.3s
50.13 t/s
gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
875 tokens
17s
50.76 t/s
gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
3,009 tokens
1min
49.70 t/s
/mnt/Downloads/llama-b9305/llama-server --model /mnt/Downloads/model/gemma4-e4b/gemma-4-E4B-it-Q4_K_M.gguf -mm ./model/gemma4-e4b/mmproj-F16.gguf -sm none -ctk q4_0 --ctx-size 131072 --verbose --host 0.0.0.0
gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
41 tokens
0.7s
58.27 t/s
gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
1,025 tokens
18s
56.41 t/s
gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
3,098 tokens
56s
54.66 t/s
/mnt/Downloads/llama-b9305/llama-server --model /mnt/Downloads/model/gemma4-e4b/gemma-4-E4B-it-Q4_K_M.gguf -mm ./model/gemma4-e4b/mmproj-F16.gguf -sm none -ctk q4_0 -ctv q4_0 --ctx-size 131072 --verbose --host 0.0.0.0
gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
63 tokens
1.4s
43.67 t/s
gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
1,059 tokens
24s
43.66 t/s
gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
2,558 tokens
59s
42.75 t/s
/mnt/Downloads/llama-b9305/llama-server --model /mnt/Downloads/model/gemma4-e4b/gemma-4-E4B-it-Q4_K_M.gguf -mm ./model/gemma4-e4b/mmproj-F16.gguf -sm none -ctv q4_0 --ctx-size 131072 --verbose --host 0.0.0.0
gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
328 tokens
6.3s
51.84 t/s
gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
902 tokens
17s
50.37 t/s
gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
3,105 tokens
1min 7s
45.81 t/s
---------------------
$ /mnt/Downloads/llama-b9305/llama-server --model /mnt/Downloads/model/gemma4-e4b/gemma-4-E4B-it-Q4_K_M.gguf -mm ./model/gemma4-e4b/mmproj-F16.gguf -sm none -ctv q4_0 --ctx-size 131072 --host 0.0.0.0
gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
609 tokens
11s
52.35 t/s
gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
850 tokens
17s
49.95 t/s
gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
3,472 tokens
1min 17s
44.93 t/s
$ /mnt/Downloads/llama-b9305/llama-server --model /mnt/Downloads/model/gemma4-e4b/gemma-4-E4B-it-Q4_K_M.gguf -mm ./model/gemma4-e4b/mmproj-F16.gguf -sm none -ctk q4_0 --ctx-size 131072 --host 0.0.0.0
gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
32 tokens
0.6s
57.03 t/s
gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
640 tokens
11s
56.90 t/s
gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
3,379 tokens
1min 1s
54.82 t/s
$ /mnt/Downloads/llama-b9305/llama-server --model /mnt/Downloads/model/gemma4-e4b/gemma-4-E4B-it-Q4_K_M.gguf -mm ./model/gemma4-e4b/mmproj-F16.gguf -sm none -ctk q4_0 -ctv q4_0 --ctx-size 131072 --host 0.0.0.0
gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
37 tokens
0.7s
51.67 t/s
gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
817 tokens
16s
50.20 t/s
gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
3,357 tokens
1min 12s
46.28 t/s
$ /mnt/Downloads/llama-b9305/llama-server --model /mnt/Downloads/model/gemma4-e4b/gemma-4-E4B-it-Q4_K_M.gguf -mm ./model/gemma4-e4b/mmproj-F16.gguf -sm none --ctx-size 131072 --host 0.0.0.0
gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
45 tokens
0.8s
57.48 t/s
gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
737 tokens
12s
59.56 t/s
gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
3,291 tokens
1min 1s
53.83 t/s
'프로그램 사용 > ai 프로그램' 카테고리의 다른 글
| wan2.2 + comfyui (0) | 2026.05.19 |
|---|---|
| comfyui 실행 (0) | 2026.05.19 |
| openai api (0) | 2026.05.18 |
| RAG 시도 - postgresql(14) + pgvector (1) | 2026.05.15 |
| gpt님 만세! - pip torch 버전 낮추기 (0) | 2026.05.15 |















