gemma4-e4b it qat / gemma4-12b mtp on 1080 ti 11GB

프로그램 사용/ai 프로그램

gemma4-e4b it qat / gemma4-12b mtp on 1080 ti 11GB

구차니 2026. 6. 8. 14:38

요약

QAT는 생성속도 차이는 크게 없어 보임. 사용해봐야 결과 품질을 알 수 있을 듯 함.

MTP는 50% 정도 성능 향상이 되는 듯?

---

QAT

오오 3~4일 전 따끈한 모델!

용량이 3~4GB 정도라 정말 어떨지 궁금하다.

[링크 : https://huggingface.co/unsloth/gemma-4-E4B-it-qat-GGUF]

기존에 테스트 하던건 Q4_K_M 이라 비슷할진 모르겠다.

$ ../../llama-b9553/llama-cli -m gemma-4-E4B-it-qat-UD-Q2_K_XL.gguf -sm none
[ Prompt: 16.8 t/s | Generation: 38.6 t/s ]
[ Prompt: 97.9 t/s | Generation: 41.1 t/s ]
[ Prompt: 196.1 t/s | Generation: 39.9 t/s ]

$ ../../llama-b9553/llama-cli -m gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf -sm none
[ Prompt: 737.0 t/s | Generation: 62.5 t/s ]
[ Prompt: 238.5 t/s | Generation: 61.4 t/s ]
[ Prompt: 292.3 t/s | Generation: 58.0 t/s ]

MTP

MTP는 multimodal 처럼 2개의 모델 파일이 필요하구나..

일단은 cuda enable 하고 빌드하려면.. sdk가 문제 없으려나.. 쩝

./build/bin/llama-server \
  -m gemma-4-12b-it-Q4_K_M.gguf \
  --model-draft MTP/gemma-4-12B-it-MTP-Q8_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 4 \
  -ngl 999 -fa on
Multi GPU: add --spec-draft-device CUDA0 -sm layer.

[링크 : https://huggingface.co/unsloth/gemma-4-12b-it-GGUF/blob/main/MTP/README.md]

음.. 장렬히 빌드 시도 폭★파 ㅋㅋㅋ

$ cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=61
CMAKE_BUILD_TYPE=Release
-- Warning: ccache not found - consider installing it for faster compilation or disable this warning with GGML_CCACHE=OFF
-- CMAKE_SYSTEM_PROCESSOR: x86_64
-- GGML_SYSTEM_ARCH: x86
-- Including CPU backend
-- x86 detected
-- Adding CPU backend variant ggml-cpu: -march=native
-- Unable to find cublas_v2.h in either "/usr/local/cuda/include" or "/usr/math_libs/include"
-- CUDA Toolkit found
CMake Error at /usr/share/cmake-3.22/Modules/CMakeDetermineCompilerId.cmake:726 (message):
  Compiling the CUDA compiler identification source file
  "CMakeCUDACompilerId.cu" failed.

  Compiler: /usr/local/cuda/bin/nvcc

  Build flags:

  Id flags: --keep;--keep-dir;tmp;-gencode=arch=compute_61,code=sm_61 -v



  The output was:

  1

  nvcc fatal : Unsupported gpu architecture 'compute_61'





Call Stack (most recent call first):
  /usr/share/cmake-3.22/Modules/CMakeDetermineCompilerId.cmake:6 (CMAKE_DETERMINE_COMPILER_ID_BUILD)
  /usr/share/cmake-3.22/Modules/CMakeDetermineCompilerId.cmake:48 (__determine_compiler_id_test)
  /usr/share/cmake-3.22/Modules/CMakeDetermineCUDACompiler.cmake:298 (CMAKE_DETERMINE_COMPILER_ID)
  ggml/src/ggml-cuda/CMakeLists.txt:59 (enable_language)

-- Configuring incomplete, errors occurred!
See also "/home/falinux/src/llama.cpp/build/CMakeFiles/CMakeOutput.log".
See also "/home/falinux/src/llama.cpp/build/CMakeFiles/CMakeError.log".

b9500 으로는 무리인가.. 아니면 vulkan 모델이라 안되는걸까?

$ ../../llama-b9500/llama-cli -m gemma-4-12b-it-Q4_0.gguf --model-draft gemma-4-12B-it-MTP-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 4 -ngl 999 -fa on --verbose

0.19.888.319 E llama_model_load: error loading model: unknown model architecture: 'gemma4-assistant'
0.19.888.322 E llama_model_load_from_file_impl: failed to load model
0.19.888.324 E srv load_model: failed to load draft model, 'gemma-4-12B-it-MTP-Q8_0.gguf'

b9953 으로 하니 돌아간다.

1080 ti 11GB / -sm none

$ ../../llama-b9553/llama-cli -m gemma-4-12b-it-Q4_0.gguf --model-draft gemma-4-12B-it-MTP-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 4 -ngl 999 -fa on -sm none

1080 ti 11GB / -sm layer

$ ../../llama-b9553/llama-cli -m gemma-4-12b-it-Q4_0.gguf --model-draft gemma-4-12B-it-MTP-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 4 -ngl 999 -fa on

>>>>> 참조용 >>>>>

하드웨어 1080 ti -sm none

gemma-4 12B it Q4_0.gguf Reading Generation 25 tokens 0.9s 27.94 t/s
gemma-4 12B it Q4_0.gguf Reading Generation 255 tokens 8.9s 28.78 t/s
gemma-4 12B it Q4_0.gguf Reading Generation 1,404 tokens 55s 25.45 t/s

gemma-4 12B it UD Q2_K_XL.gguf Reading Generation 29 tokens 1.2s 23.71 t/s
gemma-4 12B it UD Q2_K_XL.gguf Reading Generation 373 tokens 16s 22.28 t/s
gemma-4 12B it UD Q2_K_XL.gguf Reading Generation 806 tokens 37s 21.34 t/s (터짐)

하드웨어 1080 ti -sm layer

gemma-4 12B it Q4_0.gguf Reading Generation 25 tokens 0.8s 31.04 t/s
gemma-4 12B it Q4_0.gguf Reading Generation 265 tokens 9.0s 29.60 t/s
gemma-4 12B it Q4_0.gguf Reading Generation 1,340 tokens 54s 24.43 t/s

gemma-4 12B it UD Q2_K_XL.gguf Reading Generation 31 tokens 1.3s 24.16 t/s
gemma-4 12B it UD Q2_K_XL.gguf Reading Generation 263 tokens 11s 23.70 t/s
gemma-4 12B it UD Q2_K_XL.gguf Reading Generation 620 tokens 29s 20.70 t/s (터짐)

2026.06.04 - [프로그램 사용/ai 프로그램] - gemma 12b, tesla t4 16GB / 1080 ti 11GB * 2

<<<< 참조용 <<<<

2026.06.18

gemma-4-12b-it-Q4_0.gguf + gemma-4-12B-it-MTP-Q8_0.gguf

	MTP x	MTP 5	MTP 4	MTP 3	MTP 2	MTP 1
단문	31.4	37.6	37.0	40.2	42.3	42.5
중문	30.0	31.6	37.1	37.8	39.6	39.7
장문	28.5	29.6	35.4	34.4	37.1	37.0

gemma-4-12b-it-UD-Q2_K_XL.gguf + gemma-4-12B-it-MTP-Q8_0.gguf

	MTP x	MTP 5	MTP 4	MTP 3	MTP 2	MTP 1
단문	24.8	27.0	29.7	32.8	32.2	32.9
중문	24.2	24.3	26.8	29.3	31.0	31.2
장문	23.0	22.2	26.7	26.0	30.7	28.7

$ /mnt/Downloads/llama-b9553/llama-cli --model /mnt/Downloads/model/gemma4-12b/gemma-4-12b-it-Q4_0.gguf -sm none

안녕?
[ Prompt: 77.6 t/s | Generation: 31.1 t/s ]
[ Prompt: 147.7 t/s | Generation: 31.4 t/s ]

너에 대해 설명해줘
[ Prompt: 197.2 t/s | Generation: 30.0 t/s ]

파이썬으로 셀레니움을 통해 웹을 서칭하고 텍스트만 추출하고 makrdown 으로 변환후  md 파일과 pdf로 저장하는 기능을 구현해줘
[ Prompt: 254.4 t/s | Generation: 28.5 t/s ]

$ /mnt/Downloads/llama-b9553/llama-cli --model /mnt/Downloads/model/gemma4-12b/gemma-4-12b-it-Q4_0.gguf --model-draft /mnt/Downloads/model/gemma4-12b/gemma-4-12B-it-MTP-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 5 -fit off -ngl 999 -fa on -sm none

안녕?
[ Prompt: 74.1 t/s | Generation: 42.6 t/s ]
[ Prompt: 127.1 t/s | Generation: 37.6 t/s ]

너에 대해 설명해줘
[ Prompt: 126.8 t/s | Generation: 31.6 t/s ]

파이썬으로 셀레니움을 통해 웹을 서칭하고 텍스트만 추출하고 makrdown 으로 변환후  md 파일과 pdf로 저장하는 기능을 구현해줘
[ Prompt: 240.4 t/s | Generation: 29.6 t/s ]

$ /mnt/Downloads/llama-b9553/llama-cli --model /mnt/Downloads/model/gemma4-12b/gemma-4-12b-it-Q4_0.gguf --model-draft /mnt/Downloads/model/gemma4-12b/gemma-4-12B-it-MTP-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 4 -fit off -ngl 999 -fa on -sm none

안녕?
[ Prompt: 75.8 t/s | Generation: 45.9 t/s ]
[ Prompt: 115.5 t/s | Generation: 37.0 t/s ]

너에 대해 설명해줘
[ Prompt: 161.9 t/s | Generation: 37.1 t/s ]

파이썬으로 셀레니움을 통해 웹을 서칭하고 텍스트만 추출하고 makrdown 으로 변환후  md 파일과 pdf로 저장하는 기능을 구현해줘
[ Prompt: 257.1 t/s | Generation: 35.4 t/s ]

$ /mnt/Downloads/llama-b9553/llama-cli --model /mnt/Downloads/model/gemma4-12b/gemma-4-12b-it-Q4_0.gguf --model-draft /mnt/Downloads/model/gemma4-12b/gemma-4-12B-it-MTP-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 3 -fit off -ngl 999 -fa on -sm none

안녕
[ Prompt: 74.7 t/s | Generation: 42.7 t/s ]
[ Prompt: 198.6 t/s | Generation: 40.2 t/s ]

너에 대해 설명해줘
[ Prompt: 146.5 t/s | Generation: 37.8 t/s ]

파이썬으로 셀레니움을 통해 웹을 서칭하고 텍스트만 추출하고 makrdown 으로 변환후  md 파일과 pdf로 저장하는 기능을 구현해줘
[ Prompt: 248.9 t/s | Generation: 34.4 t/s ]

$ /mnt/Downloads/llama-b9553/llama-cli --model /mnt/Downloads/model/gemma4-12b/gemma-4-12b-it-Q4_0.gguf --model-draft /mnt/Downloads/model/gemma4-12b/gemma-4-12B-it-MTP-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 2 -fit off -ngl 999 -fa on -sm none

안녕
[ Prompt: 75.1 t/s | Generation: 44.5 t/s ]
[ Prompt: 115.3 t/s | Generation: 42.3 t/s ]

너에 대해 설명해줘
[ Prompt: 203.8 t/s | Generation: 39.6 t/s ]

파이썬으로 셀레니움을 통해 웹을 서칭하고 텍스트만 추출하고 makrdown 으로 변환후  md 파일과 pdf로 저장하는 기능을 구현해줘

[ Prompt: 250.9 t/s | Generation: 37.1 t/s ]

$ /mnt/Downloads/llama-b9553/llama-cli --model /mnt/Downloads/model/gemma4-12b/gemma-4-12b-it-Q4_0.gguf --model-draft /mnt/Downloads/model/gemma4-12b/gemma-4-12B-it-MTP-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 1 -fit off -ngl 999 -fa on -sm none

안녕?
[ Prompt: 75.8 t/s | Generation: 41.5 t/s ]
[ Prompt: 200.5 t/s | Generation: 42.5 t/s ]

너에 대해 설명해줘
[ Prompt: 132.7 t/s | Generation: 39.7 t/s ]

파이썬으로 셀레니움을 통해 웹을 서칭하고 텍스트만 추출하고 makrdown 으로 변환후  md 파일과 pdf로 저장하는 기능을 구현해줘
[ Prompt: 249.6 t/s | Generation: 37.0 t/s ]

너에 대해 설명해줘

파이썬으로 셀레니움을 통해 웹을 서칭하고 텍스트만 추출하고 makrdown 으로 변환후 md 파일과 pdf로 저장하는 기능을 구현해줘

$ /mnt/Downloads/llama-b9553/llama-cli --model /mnt/Downloads/model/gemma4-12b/gemma-4-12b-it-UD-Q2_K_XL.gguf -sm none 세요!

[ Prompt: 50.7 t/s | Generation: 23.3 t/s ]
[ Prompt: 136.9 t/s | Generation: 24.8 t/s ]

[ Prompt: 68.8 t/s | Generation: 24.2 t/s ]

[ Prompt: 96.2 t/s | Generation: 23.0 t/s ] 터짐

$ /mnt/Downloads/llama-b9553/llama-cli --model /mnt/Downloads/model/gemma4-12b/gemma-4-12b-it-UD-Q2_K_XL.gguf --model-draft /mnt/Downloads/model/gemma4-12b/gemma-4-12B-it-MTP-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 5 -fit off -ngl 999 -fa on -sm none
[ Prompt: 50.0 t/s | Generation: 31.8 t/s ]
[ Prompt: 142.0 t/s | Generation: 27.0 t/s ]

[ Prompt: 70.4 t/s | Generation: 24.3 t/s ]

[ Prompt: 90.8 t/s | Generation: 22.2 t/s ]

$ /mnt/Downloads/llama-b9553/llama-cli --model /mnt/Downloads/model/gemma4-12b/gemma-4-12b-it-UD-Q2_K_XL.gguf --model-draft /mnt/Downloads/model/gemma4-12b/gemma-4-12B-it-MTP-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 4 -fit off -ngl 999 -fa on -sm none

[ Prompt: 50.0 t/s | Generation: 31.7 t/s ]
[ Prompt: 131.0 t/s | Generation: 29.7 t/s ]

[ Prompt: 53.8 t/s | Generation: 26.8 t/s ]

[ Prompt: 97.6 t/s | Generation: 26.7 t/s ] 터짐

$ /mnt/Downloads/llama-b9553/llama-cli --model /mnt/Downloads/model/gemma4-12b/gemma-4-12b-it-UD-Q2_K_XL.gguf --model-draft /mnt/Downloads/model/gemma4-12b/gemma-4-12B-it-MTP-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 3 -fit off -ngl 999 -fa on -sm none

[ Prompt: 50.0 t/s | Generation: 31.8 t/s ]
[ Prompt: 123.8 t/s | Generation: 32.8 t/s ]

[ Prompt: 62.0 t/s | Generation: 29.3 t/s ]

[ Prompt: 106.4 t/s | Generation: 26.0 t/s ]

$ /mnt/Downloads/llama-b9553/llama-cli --model /mnt/Downloads/model/gemma4-12b/gemma-4-12b-it-UD-Q2_K_XL.gguf --model-draft /mnt/Downloads/model/gemma4-12b/gemma-4-12B-it-MTP-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 2 -fit off -ngl 999 -fa on -sm none

까요? 궁금한 점이 있거나 도움이 필요하시면 편하게 말씀해 주세요!

[ Prompt: 33.7 t/s | Generation: 26.5 t/s ]
[ Prompt: 159.8 t/s | Generation: 32.2 t/s ]

[ Prompt: 73.9 t/s | Generation: 31.0 t/s ]

[ Prompt: 96.7 t/s | Generation: 30.7 t/s ] 터짐

$ /mnt/Downloads/llama-b9553/llama-cli --model /mnt/Downloads/model/gemma4-12b/gemma-4-12b-it-UD-Q2_K_XL.gguf --model-draft /mnt/Downloads/model/gemma4-12b/gemma-4-12B-it-MTP-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 1 -fit off -ngl 999 -fa on -sm none

[ Prompt: 40.7 t/s | Generation: 32.3 t/s ]
[ Prompt: 139.6 t/s | Generation: 32.9 t/s ]

[ Prompt: 69.6 t/s | Generation: 31.2 t/s ]

[ Prompt: 92.4 t/s | Generation: 28.7 t/s ] 터짐

저작자표시 (새창열림)