llama.cpp kv quantization

128k context length에 2기가 VRAM을 냠냠

$ ./llama-b8925/llama-cli -m model/gemma4-e4b/gemma-4-E4B-it-Q4_K_M.gguf  --verbose llama_kv_cache: size = 2048.00 MiB (131072 cells,   4 layers,  1/1 seqs), K (f16): 1024.00 MiB, V (f16): 1024.00 MiB llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 512 llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 512 llama_kv_cache_iswa: creating     SWA KV cache, size = 1024 cells

 

k 만 q4로 하니 288MB! 256MB 보단 약간 큰데 아무튼 대충~ 1/4 로 줄었다.

$ ./llama-b8925/llama-cli -m model/gemma4-e4b/gemma-4-E4B-it-Q4_K_M.gguf  --verbose -ctk q4_0 --ctx-size 131072 llama_kv_cache: size = 1312.00 MiB (131072 cells,   4 layers,  1/1 seqs), K (q4_0):  288.00 MiB, V (f16): 1024.00 MiB llama_kv_cache: attn_rot_k = 1, n_embd_head_k_all = 512 llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 512 llama_kv_cache_iswa: creating     SWA KV cache, size = 1024 cells

 

kv를 q4로 하니 대충 512MB 근처.

$ ./llama-b8925/llama-cli -m model/gemma4-e4b/gemma-4-E4B-it-Q4_K_M.gguf  --verbose -ctk q4_0 -ctv q4_0 --ctx-size 131072 llama_kv_cache: size =  576.00 MiB (131072 cells,   4 layers,  1/1 seqs), K (q4_0):  288.00 MiB, V (q4_0):  288.00 MiB llama_kv_cache: attn_rot_k = 1, n_embd_head_k_all = 512 llama_kv_cache: attn_rot_v = 1, n_embd_head_k_all = 512 llama_kv_cache_iswa: creating     SWA KV cache, size = 1024 cells

 

 

+

2026.05.29

verbose로는 토큰별로 로그가 나와서 성능 저하가 있긴한데

verbose	none	ctk	ctk ctv	ctv
단문	50.13 t/s	58.27 t/s	43.67 t/s	51.84 t/s
중문	50.76 t/s	56.41 t/s	43.66 t/s	50.37 t/s
장문	49.70 t/s	54.66 t/s	42.75 t/s	45.81 t/s

 

생각외로 양자화 한거랑 안한거랑 차이가 별로 없다.

1080 이라 q4를 지원하지 않아서 그런걸지도?

-	none	ctk	ctk ctv	ctv
단문	57.48 t/s	57.03 t/s	51.67 t/s	52.35 t/s
중문	59.56 t/s	56.90 t/s	50.20 t/s	49.95 t/s
장문	53.83 t/s	54.82 t/s	46.28 t/s	44.93 t/s

 

안녕?
너에 대해서 소개해줘
파이썬으로 셀레니움을 통해 웹을 서칭하고 텍스트만 추출하고 makrdown 으로 변환후  md 파일과 pdf로 저장하는 기능을 구현해줘


/mnt/Downloads/llama-b9305/llama-server --model /mnt/Downloads/model/gemma4-e4b/gemma-4-E4B-it-Q4_K_M.gguf -mm ./model/gemma4-e4b/mmproj-F16.gguf -sm none --ctx-size 131072 --verbose --host 0.0.0.0

gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
63 tokens
1.3s
50.13 t/s

gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
875 tokens
17s
50.76 t/s

gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
3,009 tokens
1min
49.70 t/s

/mnt/Downloads/llama-b9305/llama-server --model /mnt/Downloads/model/gemma4-e4b/gemma-4-E4B-it-Q4_K_M.gguf -mm ./model/gemma4-e4b/mmproj-F16.gguf -sm none  -ctk q4_0 --ctx-size 131072 --verbose --host 0.0.0.0

gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
41 tokens
0.7s
58.27 t/s

gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
1,025 tokens
18s
56.41 t/s

gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
3,098 tokens
56s
54.66 t/s



/mnt/Downloads/llama-b9305/llama-server --model /mnt/Downloads/model/gemma4-e4b/gemma-4-E4B-it-Q4_K_M.gguf -mm ./model/gemma4-e4b/mmproj-F16.gguf -sm none -ctk q4_0 -ctv q4_0 --ctx-size 131072 --verbose --host 0.0.0.0


gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
63 tokens
1.4s
43.67 t/s


gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
1,059 tokens
24s
43.66 t/s


gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
2,558 tokens
59s
42.75 t/s



/mnt/Downloads/llama-b9305/llama-server --model /mnt/Downloads/model/gemma4-e4b/gemma-4-E4B-it-Q4_K_M.gguf -mm ./model/gemma4-e4b/mmproj-F16.gguf -sm none -ctv q4_0 --ctx-size 131072 --verbose --host 0.0.0.0

gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
328 tokens
6.3s
51.84 t/s

gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
902 tokens
17s
50.37 t/s


gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
3,105 tokens
1min 7s
45.81 t/s


---------------------
$ /mnt/Downloads/llama-b9305/llama-server --model /mnt/Downloads/model/gemma4-e4b/gemma-4-E4B-it-Q4_K_M.gguf -mm ./model/gemma4-e4b/mmproj-F16.gguf -sm none -ctv q4_0 --ctx-size 131072 --host 0.0.0.0

gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
609 tokens
11s
52.35 t/s

gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
850 tokens
17s
49.95 t/s

gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
3,472 tokens
1min 17s
44.93 t/s



$ /mnt/Downloads/llama-b9305/llama-server --model /mnt/Downloads/model/gemma4-e4b/gemma-4-E4B-it-Q4_K_M.gguf -mm ./model/gemma4-e4b/mmproj-F16.gguf -sm none -ctk q4_0 --ctx-size 131072 --host 0.0.0.0


gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
32 tokens
0.6s
57.03 t/s

gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
640 tokens
11s
56.90 t/s

gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
3,379 tokens
1min 1s
54.82 t/s


$ /mnt/Downloads/llama-b9305/llama-server --model /mnt/Downloads/model/gemma4-e4b/gemma-4-E4B-it-Q4_K_M.gguf -mm ./model/gemma4-e4b/mmproj-F16.gguf -sm none -ctk q4_0 -ctv q4_0 --ctx-size 131072 --host 0.0.0.0


gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
37 tokens
0.7s
51.67 t/s

gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
817 tokens
16s
50.20 t/s

gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
3,357 tokens
1min 12s
46.28 t/s



$ /mnt/Downloads/llama-b9305/llama-server --model /mnt/Downloads/model/gemma4-e4b/gemma-4-E4B-it-Q4_K_M.gguf -mm ./model/gemma4-e4b/mmproj-F16.gguf -sm none --ctx-size 131072 --host 0.0.0.0


gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
45 tokens
0.8s
57.48 t/s


gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
737 tokens
12s
59.56 t/s

gemma-4-E4B-it-Q4_K_M.gguf
Reading
Generation
3,291 tokens
1min 1s
53.83 t/s