요약
QAT는 생성속도 차이는 크게 없어 보임. 사용해봐야 결과 품질을 알 수 있을 듯 함.
MTP는 50% 정도 성능 향상이 되는 듯?
---
QAT
오오 3~4일 전 따끈한 모델!
용량이 3~4GB 정도라 정말 어떨지 궁금하다.
[링크 : https://huggingface.co/unsloth/gemma-4-E4B-it-qat-GGUF]
기존에 테스트 하던건 Q4_K_M 이라 비슷할진 모르겠다.
| $ ../../llama-b9553/llama-cli -m gemma-4-E4B-it-qat-UD-Q2_K_XL.gguf -sm none [ Prompt: 16.8 t/s | Generation: 38.6 t/s ] [ Prompt: 97.9 t/s | Generation: 41.1 t/s ] [ Prompt: 196.1 t/s | Generation: 39.9 t/s ] |
| $ ../../llama-b9553/llama-cli -m gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf -sm none [ Prompt: 737.0 t/s | Generation: 62.5 t/s ] [ Prompt: 238.5 t/s | Generation: 61.4 t/s ] [ Prompt: 292.3 t/s | Generation: 58.0 t/s ] |
MTP
MTP는 multimodal 처럼 2개의 모델 파일이 필요하구나..
일단은 cuda enable 하고 빌드하려면.. sdk가 문제 없으려나.. 쩝
| ./build/bin/llama-server \ -m gemma-4-12b-it-Q4_K_M.gguf \ --model-draft MTP/gemma-4-12B-it-MTP-Q8_0.gguf \ --spec-type draft-mtp --spec-draft-n-max 4 \ -ngl 999 -fa on Multi GPU: add --spec-draft-device CUDA0 -sm layer. |
[링크 : https://huggingface.co/unsloth/gemma-4-12b-it-GGUF/blob/main/MTP/README.md]
+
음.. 장렬히 빌드 시도 폭★파 ㅋㅋㅋ
| $ cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=61 CMAKE_BUILD_TYPE=Release -- Warning: ccache not found - consider installing it for faster compilation or disable this warning with GGML_CCACHE=OFF -- CMAKE_SYSTEM_PROCESSOR: x86_64 -- GGML_SYSTEM_ARCH: x86 -- Including CPU backend -- x86 detected -- Adding CPU backend variant ggml-cpu: -march=native -- Unable to find cublas_v2.h in either "/usr/local/cuda/include" or "/usr/math_libs/include" -- CUDA Toolkit found CMake Error at /usr/share/cmake-3.22/Modules/CMakeDetermineCompilerId.cmake:726 (message): Compiling the CUDA compiler identification source file "CMakeCUDACompilerId.cu" failed. Compiler: /usr/local/cuda/bin/nvcc Build flags: Id flags: --keep;--keep-dir;tmp;-gencode=arch=compute_61,code=sm_61 -v The output was: 1 nvcc fatal : Unsupported gpu architecture 'compute_61' Call Stack (most recent call first): /usr/share/cmake-3.22/Modules/CMakeDetermineCompilerId.cmake:6 (CMAKE_DETERMINE_COMPILER_ID_BUILD) /usr/share/cmake-3.22/Modules/CMakeDetermineCompilerId.cmake:48 (__determine_compiler_id_test) /usr/share/cmake-3.22/Modules/CMakeDetermineCUDACompiler.cmake:298 (CMAKE_DETERMINE_COMPILER_ID) ggml/src/ggml-cuda/CMakeLists.txt:59 (enable_language) -- Configuring incomplete, errors occurred! See also "/home/falinux/src/llama.cpp/build/CMakeFiles/CMakeOutput.log". See also "/home/falinux/src/llama.cpp/build/CMakeFiles/CMakeError.log". |
+
b9500 으로는 무리인가.. 아니면 vulkan 모델이라 안되는걸까?
| $ ../../llama-b9500/llama-cli -m gemma-4-12b-it-Q4_0.gguf --model-draft gemma-4-12B-it-MTP-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 4 -ngl 999 -fa on --verbose |
| 0.19.888.319 E llama_model_load: error loading model: unknown model architecture: 'gemma4-assistant' 0.19.888.322 E llama_model_load_from_file_impl: failed to load model 0.19.888.324 E srv load_model: failed to load draft model, 'gemma-4-12B-it-MTP-Q8_0.gguf' |
b9953 으로 하니 돌아간다.
1080 ti 11GB / -sm none
| $ ../../llama-b9553/llama-cli -m gemma-4-12b-it-Q4_0.gguf --model-draft gemma-4-12B-it-MTP-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 4 -ngl 999 -fa on -sm none |
| Q4_0 [ Prompt: 48.6 t/s | Generation: 42.6 t/s ] [ Prompt: 231.5 t/s | Generation: 36.6 t/s ] [ Prompt: 241.1 t/s | Generation: 34.0 t/s ] UD_Q2_K_XL [ Prompt: 5.0 t/s | Generation: 21.1 t/s ] [ Prompt: 80.7 t/s | Generation: 29.2 t/s ] [ Prompt: 45.0 t/s | Generation: 24.4 t/s ] |
1080 ti 11GB / -sm layer
| $ ../../llama-b9553/llama-cli -m gemma-4-12b-it-Q4_0.gguf --model-draft gemma-4-12B-it-MTP-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 4 -ngl 999 -fa on |
| Q4_0 [ Prompt: 66.8 t/s | Generation: 28.5 t/s ] [ Prompt: 126.1 t/s | Generation: 19.3 t/s ] [ Prompt: 88.2 t/s | Generation: 16.3 t/s ] UD_Q2_K_XL [ Prompt: 36.5 t/s | Generation: 24.6 t/s ] [ Prompt: 32.1 t/s | Generation: 17.1 t/s ] [ Prompt: 47.3 t/s | Generation: 12.6 t/s ] (한번 터졌음) |
>>>>> 참조용 >>>>>
하드웨어 1080 ti -sm none
| gemma-4 12B it Q4_0.gguf Reading Generation 25 tokens 0.9s 27.94 t/s gemma-4 12B it Q4_0.gguf Reading Generation 255 tokens 8.9s 28.78 t/s gemma-4 12B it Q4_0.gguf Reading Generation 1,404 tokens 55s 25.45 t/s gemma-4 12B it UD Q2_K_XL.gguf Reading Generation 29 tokens 1.2s 23.71 t/s gemma-4 12B it UD Q2_K_XL.gguf Reading Generation 373 tokens 16s 22.28 t/s gemma-4 12B it UD Q2_K_XL.gguf Reading Generation 806 tokens 37s 21.34 t/s (터짐) |
하드웨어 1080 ti -sm layer
| gemma-4 12B it Q4_0.gguf Reading Generation 25 tokens 0.8s 31.04 t/s gemma-4 12B it Q4_0.gguf Reading Generation 265 tokens 9.0s 29.60 t/s gemma-4 12B it Q4_0.gguf Reading Generation 1,340 tokens 54s 24.43 t/s gemma-4 12B it UD Q2_K_XL.gguf Reading Generation 31 tokens 1.3s 24.16 t/s gemma-4 12B it UD Q2_K_XL.gguf Reading Generation 263 tokens 11s 23.70 t/s gemma-4 12B it UD Q2_K_XL.gguf Reading Generation 620 tokens 29s 20.70 t/s (터짐) |
2026.06.04 - [프로그램 사용/ai 프로그램] - gemma 12b, tesla t4 16GB / 1080 ti 11GB * 2
<<<< 참조용 <<<<
'프로그램 사용 > ai 프로그램' 카테고리의 다른 글
| nvidia 3070 8GB 테스트 gemma4-e4b (0) | 2026.06.08 |
|---|---|
| sigLIP, CLIP (0) | 2026.06.05 |
| chatML (0) | 2026.06.04 |
| gemma 12b, tesla t4 16GB / 1080 ti 11GB * 2 (0) | 2026.06.04 |
| nvidia tesla t4 16GB (0) | 2026.06.02 |
