모종의 음모/ai 프로그램2026. 9. 22. 10:48

llama-cli /clear

오.. 이런 명령어가 있었구나

이전 대화내용을 날려서 context를 정리하는 명령어

 

/mnt/llm-ko/3_small/gguf$ /mnt/Downloads/llama-b10488/llama-cli -m base42m-q4_0.gguf -p "대한민국의 수도는" -ngl 0


Loading model...  

▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀

build      : b10488-9d77fa172
model      : base42m-q4_0.gguf
ftype      : Q4_0
modalities : text

available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read <file>        add a text file
  /glob <pattern>     add text files using globbing pattern



> 대한민국의 수도는
>>>이다.

[ Prompt: 161.5 t/s | Generation: 285.9 t/s ]

> ㄴㅁㅇㄻㄴㅇㄹ
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>)에서 이).

[ Prompt: 195.0 t/s | Generation: 115.0 t/s ]

> 세조대호아잉
>>>>>>>>>>>>>>>>>>>>>>>>>>>)의 순서로는 이례적으로, 그리고 이 작품이 나중의 문학적 삶의 방식에서 영감을 받은 것으로 평가된다.

[ Prompt: 213.6 t/s | Generation: 197.3 t/s ]

> 대한민국
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>)와 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>)이다.

[ Prompt: 257.0 t/s | Generation: 56.8 t/s ]

> 이다
0.16.540.856 E srv    send_error: task id = 330, error: request (513 tokens) exceeds the available context size (512 tokens), try increasing it
Error: request (513 tokens) exceeds the available context size (512 tokens), try increasing it

[ Prompt: 0.0 t/s | Generation: 0.0 t/s ]

> /clear
Chat history cleared.

> 오오
>>>를 통해 “Linkword (rock tit)”의 "아름다운"을 언급하는 것을 고려하면, "아름다운"이라는 용어는 “Linkword”에서 유래한 것이다. 이것은 "Linkword"와 같은 다른 형태의 조합을 포함하는 단어로, 이는 “Linkword”를 의미하는 단어로, 이는 영어로 "Linkword"를 의미하며, 이는 ‘Linkword(newing)’와 같은 축약형에 해당하지 않는다. 이 단어들은 주로 영어 단어에서와 같이 사용되었지만, Linkword가 다른 단어들과 유사하게 자주 인용되는 것은 아니다. Linkword에서의 q-word는 Linkword의 한 구절에 나타난다. Linkword의 다른 구절은 Linkword를 ‘Linkword’에서 q-word와 같은 의미를 가진 단어로, Linkword에서 vii를 ‘Linkword’에서 q-word가 된다. Linkword는 Linkword의 q-word를 의미한다. Linkword의 다른 구절에는 Linkword가 q-word와 같이 Linkword의 q-word의 첫 번째 구절과 두 번째 구절을 공유한다. Linkword의 첫 번째 구절에는 Linkword가 q-word에서의 Linkword의 첫 번째 구절에서 q-word를 나타낸다. Linkword의 첫 번째 구절에는 Linkword의 "Linkword"가 포함되어 있다. Linkword가 q-word를 의미한다. Linkword의 첫 번째 구절은 Linkword가 q-word를 의미한다. Linkword는 Linkword가 q-word를 나타낸다. Linkword는 Linkword와 Linkword가 모두 q-word를 나타낸다. Linkword는 Linkword의 첫 번째 구절에서, Linkword는 Linkword가 q-word를 나타낸다. Linkword의 마지막 구절은 Linkword의 q-word를 나타낸다. Linkword는 Linkword와 Linkword가 q-

[ Prompt: 1393.2 t/s | Generation: 153.5 t/s ]
저작자표시 (새창열림)

'모종의 음모 > ai 프로그램' 카테고리의 다른 글

llama-simple  (0) 2026.09.22
RoPE - Rotary Position Embedding  (0) 2026.09.21
말뭉치, 코퍼스(corpus)  (0) 2026.09.21
토크나이저  (0) 2026.09.21
LLM / 1080 ti 모드 별 벤치마크  (0) 2026.09.17
Posted by 구차니
모종의 음모/ai 프로그램2026. 9. 22. 10:12

llama-simple

llama.cpp 에서 단순예제로 프롬프트를 주면 문장일 이어서 만들어 가는

it 쪽이 아닌 base model 검증용... 유틸리티 라고 하면 되려나?

 

[링크 : https://github.com/ggml-org/llama.cpp/blob/master/examples/simple/README.md]

저작자표시 (새창열림)

'모종의 음모 > ai 프로그램' 카테고리의 다른 글

llama-cli /clear  (0) 2026.09.22
RoPE - Rotary Position Embedding  (0) 2026.09.21
말뭉치, 코퍼스(corpus)  (0) 2026.09.21
토크나이저  (0) 2026.09.21
LLM / 1080 ti 모드 별 벤치마크  (0) 2026.09.17
Posted by 구차니
모종의 음모/ai 프로그램2026. 9. 21. 17:47

RoPE - Rotary Position Embedding

LLM 학습해보려는데 이상한 말이 나와서 -ㅁ-

클로드 님께서 말씀하시길~~~

 5. 어텐션이 모르는 것: 순서
 
 어텐션은 토큰의 위치를 모릅니다. 그래서 RoPE로 Q와 K를 위치에 따라 회전시켜서 토큰 간 거리가 점수에 반영되게 합니다.

 

그래서 자연스럽게 문장이 구성되게 되는건가?

[링크 : https://mari970.tistory.com/49]

[링크 : https://ai-with-sudal-ee.tistory.com/9]

[링크 : https://railly-linker.tistory.com/169]

 

아무튼 셀프 어텐션과, 플래시 어텐션이 비슷한건줄 아랐는데

플래시 어텐션은 경량으로 도는거(파스칼에서 지원안하는)

셀프 어텐션이 입력 받고 출력 나오고 그걸 다시 돌려서 다음 단어를 추측하는 가장 핵심인거

그리고 셀프 어텐션 변경해서 MHA GQA MQA 등이 있나본데 아래의 약자인가 보다.

MHA - Multi-Head Attention

MQA - Multi-Query Attention

GQA - Grouped-Query Attention

저작자표시 (새창열림)

'모종의 음모 > ai 프로그램' 카테고리의 다른 글

llama-cli /clear  (0) 2026.09.22
llama-simple  (0) 2026.09.22
말뭉치, 코퍼스(corpus)  (0) 2026.09.21
토크나이저  (0) 2026.09.21
LLM / 1080 ti 모드 별 벤치마크  (0) 2026.09.17
Posted by 구차니
모종의 음모/ai 프로그램2026. 9. 21. 17:03

말뭉치, 코퍼스(corpus)

아이씨 자꾸 쿠퍼스로 보임

이게 토큰으로 간주해야 하는 건가.. 아닌가.. 다른 분류의 단어인가..

 

자연언어 연구를 위해 특정한 목적을 가지고 언어의 표본을 추출한 집합이다.

[링크 : https://ko.wikipedia.org/wiki/말뭉치]

 

클로드 답변은 아래와 같다.

음.. 아무튼 글이니까 (말뭉치) 그 일부는 토큰으로 이뤄지는게 맞을지도?

  코퍼스가 토큰인가

  아닙니다. 코퍼스는 글이고, 토큰은 그 글을 잘게 나눈 조각입니다.
  
  위키 문서들 = 코퍼스 (글)  →  토크나이저  →  토큰 번호의 나열 = 토큰화된 코퍼스
  train.txt (5.56억 글자)                      train.bin (2.43억 토큰)
  
  - 코퍼스: 책 한 권씩이 쌓인 도서관에 해당합니다.
  - 토큰: 그 책을 낱말 카드로 잘라 놓은 것에 해당합니다. 
  - 크기 표시: 코퍼스의 크기는 글자 수나 토큰 수 둘 다로 말합니다. 우리 코퍼스는 5.56억 글자이고, 이 토크나이저로는 2.43억 토큰입니다. 그래서
    "코퍼스가 2.43억 토큰"이라고 말하는 건 "토큰으로 센 크기"라는 뜻입니다.
  - 학습에 들어가는 형태: 모델은 글을 직접 읽지 못하고 토큰 번호만 받습니다.

 

저작자표시 (새창열림)

'모종의 음모 > ai 프로그램' 카테고리의 다른 글

llama-simple  (0) 2026.09.22
RoPE - Rotary Position Embedding  (0) 2026.09.21
토크나이저  (0) 2026.09.21
LLM / 1080 ti 모드 별 벤치마크  (0) 2026.09.17
v100 흐음...  (0) 2026.09.16
Posted by 구차니
모종의 음모/ai 프로그램2026. 9. 21. 14:50

토크나이저

클로드에서 LLM 의 토크나이저 관련해서 물어보니 뱉어준 대답

Unigram / BPE / trie MaxMatch

 

trie-based search... ? WordPiece?

[링크 : https://www.emergentmind.com/topics/wordpiece-model]

 

Unigram / BPE / WordPiece 은 알고리즘이고 SentencePiece는 툴킷이라고

BPE - Byte Pair Encoding

WordPiece - BERT 계열에서 사용하는 BPE 와 유사한

Unigram - 역방향 접근

[링크 : https://velog.io/@aiengineer/토큰-완전-입문-4.-토큰화-알고리즘-완벽-정리-BPEWordPieceUnigram의-차이와-SentencePiecetiktoken의-정체]

 

+

BERT - Bidirectional Encoder Representations from Transformeres / 구글 2018 / 자연어 처리모델

[링크 : https://namu.wiki/w/BERT]

저작자표시 (새창열림)

'모종의 음모 > ai 프로그램' 카테고리의 다른 글

RoPE - Rotary Position Embedding  (0) 2026.09.21
말뭉치, 코퍼스(corpus)  (0) 2026.09.21
LLM / 1080 ti 모드 별 벤치마크  (0) 2026.09.17
v100 흐음...  (0) 2026.09.16
llama.cpp pascal 지원 빌드 / 성능 테스트  (0) 2026.09.16
Posted by 구차니
모종의 음모/ai 프로그램2026. 9. 17. 10:36

LLM / 1080 ti 모드 별 벤치마크

궁금한 것 을 해소하기 위해 대충 5시간에 걸쳐 GPU 갈굼

  • MTP 가 정말 속도에 영향을 주냐
  • GPU 2개인데 pcie3.0 x8과 x4에 영향을 받는가
  • 컨텍스트 최대 길이가 성능에 영향을 미치는가
  • 입력 토큰길이에 따라 성능 저하는 어느정도 발생하는가
  • 단일과 -sm layer로 나누었을때 성능은 어떠한가

 

결과만 요약하면

  • GPU 0 / 1이 우선권 문제인지 lane 폭 문제인진 모르겠으나 GPU0이 3~10% 정도 이상 성능이 좋게 나옴
  • MTP는 켤수 있으면 켜라. 조건에 따라 다르지만 MTP 1이 1.3~1.5배 향상됨
  • 컨텍스트 최대 길이는 의외로 생성 성능에 영향을 안준다.
  • 한쪽에 가능하지만, 억지로 레이어 쪼개서 양쪽으로 올리면 1.1~1.4배 성능 차이가 난다. 되도록이면 1대로 돌리자

 

아래는 테스트용 스크립트들. 추후 쓰려면 바이너리 경로등을 손봐야 함

bench.zip
0.01MB

 

 

 

아래는 원본 내용

report.md
0.08MB

 

 

llama.cpp Benchmark Report

Short conversation (3-turn, context accumulates)

Questions: see questions.txt. Columns show prompt_t/s / generation_t/s per turn.

gemma4-12b

quantsmgpuctxmtpturn1 (p/g t/s)turn2 (p/g t/s)turn3 (p/g t/s)wall(s)status

gemma-4-12b-it-Q4_0 none 0 131072 off 139.2/34.8 143.4/34.6 270.3/34.4 38.4 OK
gemma-4-12b-it-Q4_0 none 0 131072 on(n=1) 123.9/57.5 141.8/56.1 246.3/51.2 21.5 OK
gemma-4-12b-it-Q4_0 none 0 131072 on(n=2) 123.3/61.5 187.3/61.7 238.3/59.1 18.6 OK
gemma-4-12b-it-Q4_0 none 0 131072 on(n=3) 118.4/65.5 227.8/59.7 231.2/51.6 18.9 OK
gemma-4-12b-it-Q4_0 none 0 131072 on(n=4) 123.2/65.5 181.6/57.1 240.6/45.5 20.0 OK
gemma-4-12b-it-Q4_0 none 0 65536 off 140.0/33.5 146.5/33.2 266.5/32.8 27.0 OK
gemma-4-12b-it-Q4_0 none 0 65536 on(n=1) 118.5/54.5 138.2/55.1 245.5/51.2 20.0 OK
gemma-4-12b-it-Q4_0 none 0 65536 on(n=2) 121.3/59.5 226.9/62.4 229.2/51.9 19.0 OK
gemma-4-12b-it-Q4_0 none 0 65536 on(n=3) 122.9/66.3 243.9/52.7 222.2/52.1 19.0 OK
gemma-4-12b-it-Q4_0 none 0 65536 on(n=4) 118.3/65.2 139.6/60.8 245.8/48.4 19.1 OK
gemma-4-12b-it-Q4_0 none 0 32768 off 136.6/33.6 146.1/33.2 263.9/32.7 27.0 OK
gemma-4-12b-it-Q4_0 none 0 32768 on(n=1) 125.1/57.3 140.9/55.5 242.6/51.8 19.7 OK
gemma-4-12b-it-Q4_0 none 0 32768 on(n=2) 120.7/65.7 139.8/58.8 243.7/57.5 18.3 OK
gemma-4-12b-it-Q4_0 none 0 32768 on(n=3) 125.6/63.6 139.2/60.2 244.5/53.6 18.7 OK
gemma-4-12b-it-Q4_0 none 0 32768 on(n=4) 113.1/60.4 182.1/63.9 242.7/43.8 19.8 OK
gemma-4-12b-it-Q4_0 none 0 16384 off 139.6/33.5 146.0/32.9 267.0/32.7 27.1 OK
gemma-4-12b-it-Q4_0 none 0 16384 on(n=1) 122.8/55.8 213.0/56.2 230.9/53.9 19.6 OK
gemma-4-12b-it-Q4_0 none 0 16384 on(n=2) 124.2/62.4 139.8/60.7 243.0/54.1 18.7 OK
gemma-4-12b-it-Q4_0 none 0 16384 on(n=3) 122.5/60.7 140.1/63.2 244.8/53.5 18.7 OK
gemma-4-12b-it-Q4_0 none 0 16384 on(n=4) 120.8/60.6 141.5/58.4 245.8/50.0 19.4 OK
gemma-4-12b-it-Q4_0 none 1 131072 off 135.6/32.8 249.3/32.7 225.6/32.5 27.0 OK
gemma-4-12b-it-Q4_0 none 1 131072 on(n=1) 106.7/52.1 130.0/51.5 223.8/49.9 22.8 OK
gemma-4-12b-it-Q4_0 none 1 131072 on(n=2) 117.4/59.5 129.0/57.8 224.2/53.2 21.3 OK
gemma-4-12b-it-Q4_0 none 1 131072 on(n=3) 117.7/61.5 155.4/54.7 218.6/52.5 21.5 OK
gemma-4-12b-it-Q4_0 none 1 131072 on(n=4) 112.3/62.0 128.8/56.6 223.6/45.7 22.1 OK
gemma-4-12b-it-Q4_0 none 1 65536 off 130.6/31.5 243.5/30.9 223.0/30.6 29.4 OK
gemma-4-12b-it-Q4_0 none 1 65536 on(n=1) 117.7/50.5 130.1/50.2 225.2/48.2 23.2 OK
gemma-4-12b-it-Q4_0 none 1 65536 on(n=2) 117.1/57.9 128.0/56.5 222.8/49.6 21.9 OK
gemma-4-12b-it-Q4_0 none 1 65536 on(n=3) 117.0/51.0 129.5/54.2 222.5/51.0 22.5 OK
gemma-4-12b-it-Q4_0 none 1 65536 on(n=4) 117.0/61.7 225.9/59.6 200.0/45.0 21.5 OK
gemma-4-12b-it-Q4_0 none 1 32768 off 133.9/31.7 212.7/31.0 227.7/30.7 30.3 OK
gemma-4-12b-it-Q4_0 none 1 32768 on(n=1) 119.1/49.3 130.3/51.8 222.2/48.9 22.9 OK
gemma-4-12b-it-Q4_0 none 1 32768 on(n=2) 119.2/61.1 130.8/57.4 224.5/48.8 21.4 OK
gemma-4-12b-it-Q4_0 none 1 32768 on(n=3) 119.4/58.1 127.6/53.1 216.0/45.0 22.4 OK
gemma-4-12b-it-Q4_0 none 1 32768 on(n=4) 118.7/60.7 130.8/53.0 225.1/45.8 22.2 OK
gemma-4-12b-it-Q4_0 none 1 16384 off 132.8/31.8 137.5/31.0 244.6/30.7 30.1 OK
gemma-4-12b-it-Q4_0 none 1 16384 on(n=1) 107.6/52.2 227.2/51.0 200.4/48.9 21.2 OK
gemma-4-12b-it-Q4_0 none 1 16384 on(n=2) 110.5/57.1 209.0/56.6 204.2/51.1 21.1 OK
gemma-4-12b-it-Q4_0 none 1 16384 on(n=3) 117.0/59.2 129.6/57.1 208.3/46.9 21.8 OK
gemma-4-12b-it-Q4_0 none 1 16384 on(n=4) 117.1/63.0 193.5/54.3 208.2/43.3 22.3 OK
gemma-4-12b-it-Q4_0 layer both 131072 off 138.0/33.4 145.7/33.3 260.4/33.2 28.0 OK
gemma-4-12b-it-Q4_0 layer both 131072 on(n=1) 123.1/49.2 136.9/48.5 238.2/47.3 23.3 OK
gemma-4-12b-it-Q4_0 layer both 131072 on(n=2) 122.3/55.9 239.8/50.8 213.4/45.0 22.8 OK
gemma-4-12b-it-Q4_0 layer both 131072 on(n=3) 122.4/49.1 135.9/48.8 236.3/43.6 23.7 OK
gemma-4-12b-it-Q4_0 layer both 131072 on(n=4) 118.0/53.8 136.3/47.4 237.4/38.9 24.4 OK
gemma-4-12b-it-Q4_0 layer both 65536 off 136.4/33.1 143.0/33.0 255.2/33.0 28.1 OK
gemma-4-12b-it-Q4_0 layer both 65536 on(n=1) 122.1/48.4 225.8/49.4 218.4/47.4 22.5 OK
gemma-4-12b-it-Q4_0 layer both 65536 on(n=2) 122.3/54.4 136.8/51.9 237.7/44.2 22.5 OK
gemma-4-12b-it-Q4_0 layer both 65536 on(n=3) 122.4/45.0 136.5/47.1 237.8/44.8 24.0 OK
gemma-4-12b-it-Q4_0 layer both 65536 on(n=4) 122.4/53.7 136.4/47.1 238.4/42.6 23.3 OK
gemma-4-12b-it-Q4_0 layer both 32768 off 140.6/33.1 145.9/33.1 260.2/33.0 27.9 OK
gemma-4-12b-it-Q4_0 layer both 32768 on(n=1) 122.9/48.3 137.3/49.0 239.4/47.7 22.8 OK
gemma-4-12b-it-Q4_0 layer both 32768 on(n=2) 109.1/52.3 137.7/52.0 239.2/46.8 22.2 OK
gemma-4-12b-it-Q4_0 layer both 32768 on(n=3) 113.3/53.9 138.1/47.0 239.0/45.8 22.7 OK
gemma-4-12b-it-Q4_0 layer both 32768 on(n=4) 124.3/52.7 237.4/48.5 217.2/38.7 23.7 OK
gemma-4-12b-it-Q4_0 layer both 16384 off 135.4/32.8 143.2/32.8 257.2/32.8 28.1 OK
gemma-4-12b-it-Q4_0 layer both 16384 on(n=1) 118.5/50.7 135.7/49.0 237.0/47.7 22.5 OK
gemma-4-12b-it-Q4_0 layer both 16384 on(n=2) 120.0/53.6 216.3/52.3 219.0/46.5 21.9 OK
gemma-4-12b-it-Q4_0 layer both 16384 on(n=3) 119.0/53.8 235.9/49.9 214.2/46.3 21.2 OK
gemma-4-12b-it-Q4_0 layer both 16384 on(n=4) 98.1/51.2 236.1/46.7 214.0/40.8 23.8 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 131072 off 85.4/31.3 93.9/31.1 203.3/30.5 36.9 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 131072 on(n=1) 78.3/42.7 166.5/42.6 181.6/40.2 24.0 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 131072 on(n=2) 78.5/45.1 92.4/43.7 184.5/34.0 24.7 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 131072 on(n=3) 78.6/40.4 91.7/41.9 189.3/31.6 26.2 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 131072 on(n=4) 78.9/42.5 153.1/39.7 183.6/26.9 27.7 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 65536 off 85.2/30.8 92.6/29.7 203.0/29.5 29.2 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 65536 on(n=1) 78.9/45.4 163.3/42.7 181.7/40.6 22.6 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 65536 on(n=2) 78.0/43.3 91.8/42.6 190.4/35.2 24.6 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 65536 on(n=3) 79.2/41.9 131.7/42.2 185.5/32.4 25.5 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 65536 on(n=4) 78.1/42.2 92.5/41.4 191.5/35.2 25.1 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 32768 off 84.8/30.6 93.7/29.6 203.1/29.4 29.4 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 32768 on(n=1) 73.8/44.8 157.0/42.1 182.4/39.7 23.7 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 32768 on(n=2) 70.6/43.6 155.4/46.0 181.4/36.8 23.9 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 32768 on(n=3) 77.4/40.6 93.5/43.4 189.3/29.6 26.3 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 32768 on(n=4) 78.5/43.6 92.2/36.1 188.0/30.9 26.7 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 16384 off 83.1/30.8 96.2/29.6 198.6/29.4 29.3 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 16384 on(n=1) 78.3/44.4 93.2/43.3 191.1/40.6 23.4 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 16384 on(n=2) 76.5/45.0 92.9/44.7 187.7/36.5 23.9 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 16384 on(n=3) 78.8/43.9 92.3/44.5 188.9/33.0 24.8 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 16384 on(n=4) 68.9/45.3 173.1/37.3 181.7/31.7 25.1 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 131072 off 84.3/30.6 93.4/30.5 192.8/30.2 30.1 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 131072 on(n=1) 76.0/43.4 178.7/40.8 164.0/37.0 26.3 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 131072 on(n=2) 76.1/44.7 90.1/42.6 174.1/36.0 26.0 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 131072 on(n=3) 71.8/38.8 177.5/38.2 163.8/37.9 27.1 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 131072 on(n=4) 75.7/38.1 88.6/40.9 178.3/28.7 29.1 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 65536 off 81.6/28.6 87.9/28.2 188.6/28.0 32.1 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 65536 on(n=1) 70.9/43.4 81.7/39.7 176.3/37.0 26.2 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 65536 on(n=2) 75.9/42.5 178.2/38.9 164.5/37.8 25.7 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 65536 on(n=3) 76.2/40.1 123.6/36.9 173.4/29.7 28.8 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 65536 on(n=4) 73.4/43.0 87.9/36.1 170.6/27.2 29.4 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 32768 off 82.1/28.5 185.9/28.1 176.6/28.1 32.1 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 32768 on(n=1) 75.6/42.5 87.4/42.0 175.6/36.3 26.1 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 32768 on(n=2) 76.1/42.0 160.3/44.1 166.9/35.7 24.6 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 32768 on(n=3) 66.4/36.6 88.3/35.5 171.7/30.3 29.6 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 32768 on(n=4) 76.0/46.1 148.8/46.1 162.2/26.7 27.3 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 16384 off 81.2/28.1 91.4/28.0 187.5/28.0 32.3 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 16384 on(n=1) 76.3/42.1 88.9/42.3 174.9/35.7 26.2 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 16384 on(n=2) 76.3/39.3 174.5/41.0 164.8/35.1 26.9 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 16384 on(n=3) 74.7/40.4 123.5/38.3 170.6/31.1 28.2 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 16384 on(n=4) 76.3/40.4 89.3/40.0 178.1/30.9 27.9 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 131072 off 81.7/31.8 92.9/31.7 196.6/31.6 28.8 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 131072 on(n=1) 75.9/42.6 159.0/41.0 174.5/40.2 25.7 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 131072 on(n=2) 73.9/44.2 90.3/38.6 184.9/35.4 26.7 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 131072 on(n=3) 73.9/39.7 90.3/39.6 185.0/29.4 28.6 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 131072 on(n=4) 71.8/35.7 87.9/35.7 182.8/26.0 31.2 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 65536 off 81.6/31.5 90.1/31.2 196.0/31.3 28.8 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 65536 on(n=1) 72.9/43.2 90.6/41.7 184.6/37.1 25.7 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 65536 on(n=2) 75.7/41.7 90.3/41.2 184.0/36.1 26.2 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 65536 on(n=3) 75.7/38.8 90.0/37.7 184.7/30.1 28.6 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 65536 on(n=4) 64.3/37.0 88.7/40.3 184.7/28.0 29.2 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 32768 off 81.7/31.5 129.4/31.3 186.1/30.8 28.9 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 32768 on(n=1) 73.2/42.8 138.4/41.9 175.3/39.9 25.1 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 32768 on(n=2) 75.6/42.0 90.6/38.4 185.3/37.5 26.1 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 32768 on(n=3) 71.1/37.6 167.8/38.1 173.2/27.0 29.6 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 32768 on(n=4) 76.0/34.6 178.5/30.4 174.5/27.6 31.6 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 16384 off 80.9/31.6 91.1/31.3 190.5/31.0 28.8 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 16384 on(n=1) 75.9/41.5 90.1/43.4 184.0/37.2 25.2 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 16384 on(n=2) 73.9/41.2 102.5/40.8 183.7/36.6 25.8 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 16384 on(n=3) 73.8/38.1 90.5/40.6 185.0/31.5 27.4 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 16384 on(n=4) 72.1/34.2 90.2/36.8 184.3/30.5 29.3 OK

gemma4-26b

quantsmgpuctxmtpturn1 (p/g t/s)turn2 (p/g t/s)turn3 (p/g t/s)wall(s)status

gemma-4-26B-A4B-it-UD-IQ2_M none 0 131072 off - - - 21.8 OOM
gemma-4-26B-A4B-it-UD-IQ2_M none 0 65536 off - - - 3.6 OOM
gemma-4-26B-A4B-it-UD-IQ2_M none 0 32768 off 92.5/65.4 175.2/65.2 341.6/64.8 16.1 OK
gemma-4-26B-A4B-it-UD-IQ2_M none 0 16384 off 135.9/64.9 196.1/64.4 338.4/63.2 16.2 OK
gemma-4-26B-A4B-it-UD-IQ2_M none 1 131072 off - - - 6.1 OOM
gemma-4-26B-A4B-it-UD-IQ2_M none 1 65536 off - - - 6.1 OOM
gemma-4-26B-A4B-it-UD-IQ2_M none 1 32768 off 147.6/60.0 177.3/59.9 317.9/59.4 19.5 OK
gemma-4-26B-A4B-it-UD-IQ2_M none 1 16384 off 149.8/59.8 176.1/59.6 314.3/58.2 19.7 OK
gemma-4-26B-A4B-it-UD-IQ2_M layer both 131072 off 148.0/61.8 196.3/61.8 333.4/61.7 18.1 OK
gemma-4-26B-A4B-it-UD-IQ2_M layer both 65536 off 147.7/61.7 196.1/61.6 332.7/61.5 18.1 OK
gemma-4-26B-A4B-it-UD-IQ2_M layer both 32768 off 149.5/61.5 194.9/61.5 330.8/61.4 17.9 OK
gemma-4-26B-A4B-it-UD-IQ2_M layer both 16384 off 149.1/61.5 195.4/61.5 331.4/61.3 17.8 OK

gemma4-31b

quantsmgpuctxmtpturn1 (p/g t/s)turn2 (p/g t/s)turn3 (p/g t/s)wall(s)status

gemma-4-31B-it-UD-IQ2_XXS none 0 131072 off - - - 19.0 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 131072 on(n=1) - - - 4.1 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 131072 on(n=2) - - - 4.1 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 131072 on(n=3) - - - 4.2 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 131072 on(n=4) - - - 4.1 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 65536 off - - - 3.4 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 65536 on(n=1) - - - 4.1 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 65536 on(n=2) - - - 4.1 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 65536 on(n=3) - - - 4.1 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 65536 on(n=4) - - - 4.1 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 32768 off - - - 3.4 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 32768 on(n=1) - - - 4.2 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 32768 on(n=2) - - - 4.2 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 32768 on(n=3) - - - 4.2 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 32768 on(n=4) - - - 4.2 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 16384 off 43.5/15.0 48.2/14.9 96.1/14.2 57.6 OK
gemma-4-31B-it-UD-IQ2_XXS none 0 16384 on(n=1) - - - 6.0 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 16384 on(n=2) - - - 5.1 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 16384 on(n=3) - - - 5.1 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 16384 on(n=4) - - - 5.1 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 131072 off - - - 5.5 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 131072 on(n=1) - - - 6.3 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 131072 on(n=2) - - - 6.2 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 131072 on(n=3) - - - 6.2 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 131072 on(n=4) - - - 6.2 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 65536 off - - - 5.5 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 65536 on(n=1) - - - 6.3 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 65536 on(n=2) - - - 6.4 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 65536 on(n=3) - - - 6.3 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 65536 on(n=4) - - - 6.3 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 32768 off - - - 5.5 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 32768 on(n=1) - - - 6.3 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 32768 on(n=2) - - - 6.4 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 32768 on(n=3) - - - 6.3 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 32768 on(n=4) - - - 6.4 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 16384 off 43.2/14.8 46.4/14.0 89.6/13.3 62.3 OK
gemma-4-31B-it-UD-IQ2_XXS none 1 16384 on(n=1) - - - 7.2 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 16384 on(n=2) - - - 7.2 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 16384 on(n=3) - - - 7.1 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 16384 on(n=4) - - - 7.1 OOM
gemma-4-31B-it-UD-IQ2_XXS layer both 131072 off 41.9/15.3 60.1/15.2 94.0/15.1 57.2 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 131072 on(n=1) - - - 6.6 OOM
gemma-4-31B-it-UD-IQ2_XXS layer both 131072 on(n=2) - - - 6.7 OOM
gemma-4-31B-it-UD-IQ2_XXS layer both 131072 on(n=3) - - - 6.6 OOM
gemma-4-31B-it-UD-IQ2_XXS layer both 131072 on(n=4) - - - 6.6 OOM
gemma-4-31B-it-UD-IQ2_XXS layer both 65536 off 42.1/15.2 79.4/15.1 92.2/15.0 57.5 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 65536 on(n=1) 39.3/22.3 67.4/21.9 85.3/21.0 44.1 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 65536 on(n=2) 39.2/23.4 91.9/20.7 81.8/21.2 44.0 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 65536 on(n=3) 37.0/23.9 138.9/16.4 49.6/15.9 52.9 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 65536 on(n=4) 39.2/20.4 45.8/19.0 88.5/18.2 48.7 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 32768 off 41.3/15.1 85.4/15.0 133.1/14.8 57.8 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 32768 on(n=1) 38.7/22.9 92.4/20.9 80.9/20.6 42.9 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 32768 on(n=2) 37.8/22.7 67.6/21.7 85.1/19.9 42.0 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 32768 on(n=3) 39.2/20.7 45.9/19.4 88.9/17.5 48.6 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 32768 on(n=4) 39.2/20.8 45.7/20.3 87.5/17.2 48.2 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 16384 off 41.6/15.1 47.9/14.8 95.1/14.6 58.1 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 16384 on(n=1) 39.3/22.6 66.8/21.9 86.5/20.7 43.8 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 16384 on(n=2) 39.3/24.3 90.5/20.5 81.3/17.9 46.0 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 16384 on(n=3) 39.4/20.3 52.6/19.9 88.4/17.8 48.3 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 16384 on(n=4) 39.3/19.7 145.8/17.8 53.0/15.5 54.2 OK
gemma-4-31B-it-UD-Q2_K_XL none 0 131072 off - - - 24.0 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 131072 on(n=1) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 131072 on(n=2) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 131072 on(n=3) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 131072 on(n=4) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 65536 off - - - 2.9 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 65536 on(n=1) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 65536 on(n=2) - - - 3.7 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 65536 on(n=3) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 65536 on(n=4) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 32768 off - - - 2.9 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 32768 on(n=1) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 32768 on(n=2) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 32768 on(n=3) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 32768 on(n=4) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 16384 off - - - 2.9 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 16384 on(n=1) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 16384 on(n=2) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 16384 on(n=3) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 16384 on(n=4) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 131072 off - - - 2.9 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 131072 on(n=1) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 131072 on(n=2) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 131072 on(n=3) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 131072 on(n=4) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 65536 off - - - 2.9 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 65536 on(n=1) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 65536 on(n=2) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 65536 on(n=3) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 65536 on(n=4) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 32768 off - - - 2.9 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 32768 on(n=1) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 32768 on(n=2) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 32768 on(n=3) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 32768 on(n=4) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 16384 off - - - 2.9 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 16384 on(n=1) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 16384 on(n=2) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 16384 on(n=3) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 16384 on(n=4) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL layer both 131072 off - - - 5.5 OOM
gemma-4-31B-it-UD-Q2_K_XL layer both 131072 on(n=1) - - - 6.2 OOM
gemma-4-31B-it-UD-Q2_K_XL layer both 131072 on(n=2) - - - 6.3 OOM
gemma-4-31B-it-UD-Q2_K_XL layer both 131072 on(n=3) - - - 6.3 OOM
gemma-4-31B-it-UD-Q2_K_XL layer both 131072 on(n=4) - - - 6.2 OOM
gemma-4-31B-it-UD-Q2_K_XL layer both 65536 off 35.3/13.8 40.7/13.7 86.1/13.6 63.6 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 65536 on(n=1) 32.7/20.5 40.3/20.7 79.4/20.2 47.5 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 65536 on(n=2) 32.7/22.4 64.2/21.0 76.9/18.7 47.3 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 65536 on(n=3) 32.9/21.9 64.0/20.4 76.9/18.2 46.8 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 65536 on(n=4) 31.0/21.0 39.6/20.1 80.4/17.7 49.4 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 32768 off 34.1/13.6 68.3/13.4 82.1/13.3 60.9 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 32768 on(n=1) 30.9/20.1 48.1/20.7 78.7/19.4 48.0 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 32768 on(n=2) 30.9/20.9 40.1/21.4 79.6/18.7 47.6 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 32768 on(n=3) 31.2/20.9 40.0/20.6 80.0/17.5 49.0 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 32768 on(n=4) 32.6/20.2 64.3/22.5 75.7/18.2 46.0 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 16384 off 34.3/13.6 67.6/13.4 81.0/13.4 64.6 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 16384 on(n=1) 30.8/20.3 63.9/20.1 76.7/19.3 48.4 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 16384 on(n=2) 32.6/20.8 40.0/21.9 79.9/18.8 47.3 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 16384 on(n=3) 32.8/19.9 39.9/17.8 79.8/17.7 51.3 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 16384 on(n=4) 32.8/22.5 40.2/21.1 79.5/16.9 48.3 OK

gemma4-e2b

quantsmgpuctxmtpturn1 (p/g t/s)turn2 (p/g t/s)turn3 (p/g t/s)wall(s)status

gemma-4-E2B-it-Q4_K_M none 0 131072 off 251.7/92.4 268.8/92.4 827.5/92.3 16.6 OK
gemma-4-E2B-it-Q4_K_M none 0 65536 off 177.7/92.2 380.4/92.2 847.8/92.1 11.0 OK
gemma-4-E2B-it-Q4_K_M none 0 32768 off 178.4/91.9 892.3/92.1 837.1/92.0 8.7 OK
gemma-4-E2B-it-Q4_K_M none 0 16384 off 262.1/88.7 430.0/92.2 846.8/92.1 8.3 OK
gemma-4-E2B-it-Q4_K_M none 1 131072 off 243.8/86.7 358.6/86.7 817.0/86.6 12.0 OK
gemma-4-E2B-it-Q4_K_M none 1 65536 off 258.9/85.4 352.0/85.3 803.6/85.4 12.1 OK
gemma-4-E2B-it-Q4_K_M none 1 32768 off 262.6/81.3 408.6/86.0 804.0/85.8 8.9 OK
gemma-4-E2B-it-Q4_K_M none 1 16384 off 263.2/82.2 406.5/85.9 799.4/85.0 8.9 OK
gemma-4-E2B-it-Q4_K_M layer both 131072 off 205.5/65.2 290.7/64.6 660.5/62.9 15.2 OK
gemma-4-E2B-it-Q4_K_M layer both 65536 off 215.5/63.5 340.7/65.2 672.0/63.5 11.0 OK
gemma-4-E2B-it-Q4_K_M layer both 32768 off 163.0/66.0 355.8/67.8 707.0/65.9 10.7 OK
gemma-4-E2B-it-Q4_K_M layer both 16384 off 167.1/62.6 337.5/64.7 664.8/63.0 11.0 OK

gemma4-e4b

quantsmgpuctxmtpturn1 (p/g t/s)turn2 (p/g t/s)turn3 (p/g t/s)wall(s)status

gemma-4-E4B-it-Q4_K_M none 0 131072 off 137.3/61.6 439.9/61.5 510.4/61.2 24.2 OK
gemma-4-E4B-it-Q4_K_M none 0 131072 on(n=1) 162.3/81.3 557.5/87.7 486.8/88.6 11.5 OK
gemma-4-E4B-it-Q4_K_M none 0 131072 on(n=2) 154.5/91.4 550.3/92.6 488.2/88.9 10.9 OK
gemma-4-E4B-it-Q4_K_M none 0 131072 on(n=3) 163.7/89.2 338.7/95.8 496.6/91.6 10.5 OK
gemma-4-E4B-it-Q4_K_M none 0 131072 on(n=4) 167.1/69.7 559.5/89.2 475.0/84.6 11.3 OK
gemma-4-E4B-it-Q4_K_M none 0 65536 off 180.4/60.7 517.8/60.9 497.2/60.0 12.0 OK
gemma-4-E4B-it-Q4_K_M none 0 65536 on(n=1) 163.0/88.7 232.9/92.9 490.9/85.8 13.4 OK
gemma-4-E4B-it-Q4_K_M none 0 65536 on(n=2) 163.9/87.5 548.8/91.6 490.6/87.5 10.9 OK
gemma-4-E4B-it-Q4_K_M none 0 65536 on(n=3) 123.7/66.3 544.5/101.7 484.8/76.2 11.3 OK
gemma-4-E4B-it-Q4_K_M none 0 65536 on(n=4) 161.4/63.4 547.0/86.8 486.2/76.4 11.7 OK
gemma-4-E4B-it-Q4_K_M none 0 32768 off 137.9/60.7 527.3/60.9 503.6/59.6 12.1 OK
gemma-4-E4B-it-Q4_K_M none 0 32768 on(n=1) 163.6/78.8 600.7/93.1 475.7/87.3 11.1 OK
gemma-4-E4B-it-Q4_K_M none 0 32768 on(n=2) 165.2/74.1 399.6/99.9 495.0/90.8 10.4 OK
gemma-4-E4B-it-Q4_K_M none 0 32768 on(n=3) 164.3/74.0 589.4/93.0 469.5/88.0 11.1 OK
gemma-4-E4B-it-Q4_K_M none 0 32768 on(n=4) 125.3/59.3 477.0/85.9 489.8/72.6 11.8 OK
gemma-4-E4B-it-Q4_K_M none 0 16384 off 158.4/60.9 576.5/60.8 495.3/59.6 12.3 OK
gemma-4-E4B-it-Q4_K_M none 0 16384 on(n=1) 155.8/79.5 510.5/86.7 484.5/87.8 11.0 OK
gemma-4-E4B-it-Q4_K_M none 0 16384 on(n=2) 160.8/73.3 565.4/97.5 484.7/86.0 10.9 OK
gemma-4-E4B-it-Q4_K_M none 0 16384 on(n=3) 165.7/86.4 232.9/105.1 492.4/94.8 12.8 OK
gemma-4-E4B-it-Q4_K_M none 0 16384 on(n=4) 166.7/65.1 567.5/91.5 483.8/83.0 11.4 OK
gemma-4-E4B-it-Q4_K_M none 1 131072 off 163.1/57.4 225.8/57.4 492.2/57.1 17.2 OK
gemma-4-E4B-it-Q4_K_M none 1 131072 on(n=1) 155.3/73.1 368.1/87.5 467.6/80.4 11.9 OK
gemma-4-E4B-it-Q4_K_M none 1 131072 on(n=2) 136.0/75.6 476.1/91.2 457.9/83.2 11.9 OK
gemma-4-E4B-it-Q4_K_M none 1 131072 on(n=3) 139.9/76.5 570.5/90.8 441.5/83.2 12.2 OK
gemma-4-E4B-it-Q4_K_M none 1 131072 on(n=4) 143.7/67.1 533.1/78.7 447.0/77.4 12.8 OK
gemma-4-E4B-it-Q4_K_M none 1 65536 off 134.1/56.9 581.6/55.2 444.0/54.5 14.1 OK
gemma-4-E4B-it-Q4_K_M none 1 65536 on(n=1) 133.0/83.1 575.9/81.1 438.6/83.2 12.6 OK
gemma-4-E4B-it-Q4_K_M none 1 65536 on(n=2) 120.5/80.7 530.4/88.3 442.6/81.0 12.2 OK
gemma-4-E4B-it-Q4_K_M none 1 65536 on(n=3) 159.8/63.8 429.5/92.1 445.3/87.8 11.7 OK
gemma-4-E4B-it-Q4_K_M none 1 65536 on(n=4) 157.2/76.2 504.1/77.3 448.0/67.6 13.2 OK
gemma-4-E4B-it-Q4_K_M none 1 32768 off 177.3/57.0 630.6/55.3 454.5/54.9 14.1 OK
gemma-4-E4B-it-Q4_K_M none 1 32768 on(n=1) 151.4/84.0 517.6/82.6 437.5/79.5 12.4 OK
gemma-4-E4B-it-Q4_K_M none 1 32768 on(n=2) 162.7/66.7 562.5/86.3 432.2/75.0 12.5 OK
gemma-4-E4B-it-Q4_K_M none 1 32768 on(n=3) 153.3/66.4 533.8/92.9 442.7/77.1 12.1 OK
gemma-4-E4B-it-Q4_K_M none 1 32768 on(n=4) 158.0/58.9 512.7/93.8 437.6/68.2 12.8 OK
gemma-4-E4B-it-Q4_K_M none 1 16384 off 169.7/55.6 316.9/54.5 443.6/54.5 13.4 OK
gemma-4-E4B-it-Q4_K_M none 1 16384 on(n=1) 160.8/77.1 323.0/83.8 457.0/81.1 11.7 OK
gemma-4-E4B-it-Q4_K_M none 1 16384 on(n=2) 159.0/75.8 554.9/87.0 438.5/80.5 12.3 OK
gemma-4-E4B-it-Q4_K_M none 1 16384 on(n=3) 121.6/87.3 224.7/88.1 475.7/76.6 14.8 OK
gemma-4-E4B-it-Q4_K_M none 1 16384 on(n=4) 159.6/88.0 225.4/91.6 476.4/76.9 14.6 OK
gemma-4-E4B-it-Q4_K_M layer both 131072 off 171.6/57.4 559.6/57.7 494.4/57.6 13.3 OK
gemma-4-E4B-it-Q4_K_M layer both 131072 on(n=1) 150.0/76.6 455.2/80.8 478.5/76.7 12.9 OK
gemma-4-E4B-it-Q4_K_M layer both 131072 on(n=2) 156.5/64.9 533.9/82.7 472.6/81.3 12.9 OK
gemma-4-E4B-it-Q4_K_M layer both 131072 on(n=3) 157.8/78.4 559.7/92.1 473.4/77.8 12.6 OK
gemma-4-E4B-it-Q4_K_M layer both 131072 on(n=4) 152.5/62.7 628.6/87.3 463.4/61.2 14.0 OK
gemma-4-E4B-it-Q4_K_M layer both 65536 off 174.8/57.5 548.8/57.6 492.1/57.5 13.3 OK
gemma-4-E4B-it-Q4_K_M layer both 65536 on(n=1) 154.3/76.4 227.1/83.8 487.5/79.0 15.3 OK
gemma-4-E4B-it-Q4_K_M layer both 65536 on(n=2) 158.9/65.6 572.5/85.0 465.8/80.8 12.6 OK
gemma-4-E4B-it-Q4_K_M layer both 65536 on(n=3) 152.1/61.2 540.7/86.7 474.2/78.9 12.5 OK
gemma-4-E4B-it-Q4_K_M layer both 65536 on(n=4) 162.1/66.4 470.2/75.9 478.2/68.5 13.0 OK
gemma-4-E4B-it-Q4_K_M layer both 32768 off 160.8/57.3 439.7/57.5 494.9/57.4 12.8 OK
gemma-4-E4B-it-Q4_K_M layer both 32768 on(n=1) 154.6/77.8 627.5/80.9 464.0/80.2 12.5 OK
gemma-4-E4B-it-Q4_K_M layer both 32768 on(n=2) 159.5/74.9 567.8/84.8 468.9/77.7 12.6 OK
gemma-4-E4B-it-Q4_K_M layer both 32768 on(n=3) 153.4/84.3 432.2/79.2 477.6/73.6 12.4 OK
gemma-4-E4B-it-Q4_K_M layer both 32768 on(n=4) 159.2/54.2 489.7/70.3 474.7/69.8 13.4 OK
gemma-4-E4B-it-Q4_K_M layer both 16384 off 172.1/57.1 476.7/57.5 495.0/57.4 12.9 OK
gemma-4-E4B-it-Q4_K_M layer both 16384 on(n=1) 161.2/82.0 226.4/85.5 489.0/79.1 14.5 OK
gemma-4-E4B-it-Q4_K_M layer both 16384 on(n=2) 159.3/82.0 226.6/89.0 486.9/80.5 14.3 OK
gemma-4-E4B-it-Q4_K_M layer both 16384 on(n=3) 128.7/73.9 531.0/83.8 474.7/70.0 12.5 OK
gemma-4-E4B-it-Q4_K_M layer both 16384 on(n=4) 127.7/69.2 597.7/89.2 464.9/67.4 12.7 OK

gemma4-e4b-qat

quantsmgpuctxmtpturn1 (p/g t/s)turn2 (p/g t/s)turn3 (p/g t/s)wall(s)status

gemma-4-E4B-it-qat-UD-Q2_K_XL none 0 131072 off 225.8/43.5 338.4/43.3 547.1/43.0 26.3 OK
gemma-4-E4B-it-qat-UD-Q2_K_XL none 0 65536 off 254.4/42.6 205.3/43.4 536.7/42.7 20.6 OK
gemma-4-E4B-it-qat-UD-Q2_K_XL none 0 32768 off 234.9/41.0 290.8/43.4 532.0/43.2 14.8 OK
gemma-4-E4B-it-qat-UD-Q2_K_XL none 0 16384 off 226.2/43.0 361.1/42.8 537.0/42.9 18.1 OK
gemma-4-E4B-it-qat-UD-Q2_K_XL none 1 131072 off 243.3/42.9 244.4/41.9 931.4/41.9 16.0 OK
gemma-4-E4B-it-qat-UD-Q2_K_XL none 1 65536 off 217.4/41.7 211.8/42.1 520.5/42.1 21.6 OK
gemma-4-E4B-it-qat-UD-Q2_K_XL none 1 32768 off 222.5/42.1 339.9/41.7 512.0/41.2 16.1 OK
gemma-4-E4B-it-qat-UD-Q2_K_XL none 1 16384 off 219.6/42.5 339.9/42.1 499.4/42.0 16.0 OK
gemma-4-E4B-it-qat-UD-Q2_K_XL layer both 131072 off 204.2/39.2 319.1/39.3 480.3/39.5 22.9 OK
gemma-4-E4B-it-qat-UD-Q2_K_XL layer both 65536 off 218.9/42.2 286.2/40.6 488.3/38.5 16.4 OK
gemma-4-E4B-it-qat-UD-Q2_K_XL layer both 32768 off 176.8/34.6 284.8/42.1 528.9/42.0 15.6 OK
gemma-4-E4B-it-qat-UD-Q2_K_XL layer both 16384 off 212.6/41.5 389.0/42.0 522.5/41.6 16.0 OK
gemma-4-E4B-it-qat-UD-Q4_K_XL none 0 131072 off 263.9/69.5 420.4/70.1 641.8/70.0 17.8 OK
gemma-4-E4B-it-qat-UD-Q4_K_XL none 0 65536 off 265.9/69.4 686.1/69.3 632.7/69.2 10.9 OK
gemma-4-E4B-it-qat-UD-Q4_K_XL none 0 32768 off 305.0/69.5 510.7/69.7 636.5/69.3 10.4 OK
gemma-4-E4B-it-qat-UD-Q4_K_XL none 0 16384 off 302.9/69.5 314.8/69.2 632.7/68.1 13.8 OK
gemma-4-E4B-it-qat-UD-Q4_K_XL none 1 131072 off 280.7/65.8 298.1/65.7 602.2/65.4 15.2 OK
gemma-4-E4B-it-qat-UD-Q4_K_XL none 1 65536 off 284.6/65.5 642.7/65.5 576.5/65.4 12.0 OK
gemma-4-E4B-it-qat-UD-Q4_K_XL none 1 32768 off 280.9/65.3 652.1/64.6 569.5/63.6 12.2 OK
gemma-4-E4B-it-qat-UD-Q4_K_XL none 1 16384 off 288.0/65.3 690.6/63.7 557.8/63.3 12.4 OK
gemma-4-E4B-it-qat-UD-Q4_K_XL layer both 131072 off 262.2/63.4 658.0/65.6 608.9/65.5 11.9 OK
gemma-4-E4B-it-qat-UD-Q4_K_XL layer both 65536 off 289.4/65.6 305.1/65.6 870.7/65.4 15.1 OK
gemma-4-E4B-it-qat-UD-Q4_K_XL layer both 32768 off 289.7/65.2 619.3/65.2 606.4/65.2 11.5 OK
gemma-4-E4B-it-qat-UD-Q4_K_XL layer both 16384 off 289.4/64.9 430.1/65.4 612.2/65.3 11.1 OK

qwen3.5

quantsmgpuctxmtpturn1 (p/g t/s)turn2 (p/g t/s)turn3 (p/g t/s)wall(s)status

Qwen3.5-0.8B-UD-IQ2_M none 0 131072 off 140.5/175.7 346.4/175.9 690.4/174.6 7.1 OK
Qwen3.5-0.8B-UD-IQ2_M none 0 65536 off 349.3/175.1 1051.4/175.9 1812.4/175.8 3.8 OK
Qwen3.5-0.8B-UD-IQ2_M none 0 32768 off 353.0/175.5 1439.3/175.9 2042.0/175.7 4.1 OK
Qwen3.5-0.8B-UD-IQ2_M none 0 16384 off 343.2/173.7 570.3/175.7 3438.5/175.4 4.9 OK
Qwen3.5-0.8B-UD-IQ2_M none 1 131072 off 293.2/160.5 1865.5/161.2 685.3/161.3 6.3 OK
Qwen3.5-0.8B-UD-IQ2_M none 1 65536 off 296.6/160.8 2372.6/0.0 681.2/160.9 6.6 OK
Qwen3.5-0.8B-UD-IQ2_M none 1 32768 off 267.1/161.8 256.9/162.1 678.0/162.0 6.6 OK
Qwen3.5-0.8B-UD-IQ2_M none 1 16384 off 233.3/159.8 819.6/160.8 684.7/160.8 5.4 OK
Qwen3.5-0.8B-UD-IQ2_M layer both 131072 off 238.6/152.9 660.8/160.3 792.1/163.3 5.6 OK
Qwen3.5-0.8B-UD-IQ2_M layer both 65536 off 276.1/163.9 298.6/163.9 802.6/164.1 6.8 OK
Qwen3.5-0.8B-UD-IQ2_M layer both 32768 off 287.7/162.3 298.6/164.4 3081.2/164.1 6.6 OK
Qwen3.5-0.8B-UD-IQ2_M layer both 16384 off 213.0/163.5 1300.6/164.4 804.1/164.0 5.5 OK
Qwen3.5-0.8B-UD-Q4_K_XL none 0 131072 off 238.6/161.0 359.9/163.9 965.5/163.6 7.6 OK
Qwen3.5-0.8B-UD-Q4_K_XL none 0 65536 off 251.8/162.7 536.5/162.8 951.2/162.7 6.6 OK
Qwen3.5-0.8B-UD-Q4_K_XL none 0 32768 off 142.0/162.8 357.6/162.8 990.8/162.7 6.6 OK
Qwen3.5-0.8B-UD-Q4_K_XL none 0 16384 off 152.1/162.7 358.2/162.8 990.2/162.7 6.6 OK
Qwen3.5-0.8B-UD-Q4_K_XL none 1 131072 off 205.4/150.2 260.6/151.3 678.7/151.1 7.2 OK
Qwen3.5-0.8B-UD-Q4_K_XL none 1 65536 off 208.9/149.9 262.3/149.9 681.2/149.8 7.2 OK
Qwen3.5-0.8B-UD-Q4_K_XL none 1 32768 off 150.8/150.1 513.2/150.1 678.1/150.1 7.2 OK
Qwen3.5-0.8B-UD-Q4_K_XL none 1 16384 off 183.8/149.8 261.0/149.9 675.6/149.9 7.2 OK
Qwen3.5-0.8B-UD-Q4_K_XL layer both 131072 off 166.1/143.6 298.3/146.0 778.9/145.7 7.7 OK
Qwen3.5-0.8B-UD-Q4_K_XL layer both 65536 off 169.0/152.9 301.5/153.5 802.8/153.1 7.2 OK
Qwen3.5-0.8B-UD-Q4_K_XL layer both 32768 off 171.6/152.9 304.3/153.7 805.5/153.2 7.1 OK
Qwen3.5-0.8B-UD-Q4_K_XL layer both 16384 off 164.4/152.0 299.7/152.5 2021.4/152.0 7.0 OK
Qwen3.5-0.8B-UD-Q8_K_XL none 0 131072 off 266.9/134.6 336.2/135.8 2332.9/135.7 9.8 OK
Qwen3.5-0.8B-UD-Q8_K_XL none 0 65536 off 267.2/134.9 337.0/135.1 899.7/134.9 7.7 OK
Qwen3.5-0.8B-UD-Q8_K_XL none 0 32768 off 153.0/135.3 337.3/135.2 897.0/135.1 7.8 OK
Qwen3.5-0.8B-UD-Q8_K_XL none 0 16384 off 278.5/135.0 342.8/135.0 883.3/134.9 7.5 OK
Qwen3.5-0.8B-UD-Q8_K_XL none 1 131072 off 180.9/127.2 248.8/126.9 629.6/126.9 8.4 OK
Qwen3.5-0.8B-UD-Q8_K_XL none 1 65536 off 202.9/125.7 376.4/125.8 628.9/125.7 7.5 OK
Qwen3.5-0.8B-UD-Q8_K_XL none 1 32768 off 222.9/125.2 713.0/125.3 643.7/125.2 7.6 OK
Qwen3.5-0.8B-UD-Q8_K_XL none 1 16384 off 235.2/125.6 249.6/125.6 641.1/125.5 8.4 OK
Qwen3.5-0.8B-UD-Q8_K_XL layer both 131072 off 177.1/120.8 542.3/121.8 726.5/121.8 8.9 OK
Qwen3.5-0.8B-UD-Q8_K_XL layer both 65536 off 163.5/125.7 282.3/124.7 733.4/124.5 8.5 OK
Qwen3.5-0.8B-UD-Q8_K_XL layer both 32768 off 163.1/128.6 287.0/128.7 750.1/128.7 8.3 OK
Qwen3.5-0.8B-UD-Q8_K_XL layer both 16384 off 176.6/128.1 578.8/128.2 752.3/128.0 8.3 OK

qwen3.6_27B

quantsmgpuctxmtpturn1 (p/g t/s)turn2 (p/g t/s)turn3 (p/g t/s)wall(s)status

Qwen3.6-27B-Q4_K_S none 0 131072 off - - - 31.0 OOM
Qwen3.6-27B-Q4_K_S none 0 65536 off - - - 2.7 OOM
Qwen3.6-27B-Q4_K_S none 0 32768 off - - - 2.7 OOM
Qwen3.6-27B-Q4_K_S none 0 16384 off - - - 2.7 OOM
Qwen3.6-27B-Q4_K_S none 1 131072 off - - - 2.7 OOM
Qwen3.6-27B-Q4_K_S none 1 65536 off - - - 2.7 OOM
Qwen3.6-27B-Q4_K_S none 1 32768 off - - - 2.7 OOM
Qwen3.6-27B-Q4_K_S none 1 16384 off - - - 2.7 OOM
Qwen3.6-27B-Q4_K_S layer both 131072 off - - - 6.0 OOM
Qwen3.6-27B-Q4_K_S layer both 65536 off 40.7/15.8 38.0/15.8 82.1/15.7 56.3 OK
Qwen3.6-27B-Q4_K_S layer both 32768 off 43.6/15.8 41.4/15.7 81.7/15.6 56.6 OK
Qwen3.6-27B-Q4_K_S layer both 16384 off 41.8/15.7 37.8/15.7 82.6/15.6 56.6 OK

qwen3.6_35B

quantsmgpuctxmtpturn1 (p/g t/s)turn2 (p/g t/s)turn3 (p/g t/s)wall(s)status

Qwen3.6-35B-A3B-UD-Q2_K_XL none 0 131072 off - - - 24.6 OOM
Qwen3.6-35B-A3B-UD-Q2_K_XL none 0 65536 off - - - 2.3 OOM
Qwen3.6-35B-A3B-UD-Q2_K_XL none 0 32768 off - - - 2.3 OOM
Qwen3.6-35B-A3B-UD-Q2_K_XL none 0 16384 off - - - 2.3 OOM
Qwen3.6-35B-A3B-UD-Q2_K_XL none 1 131072 off - - - 2.3 OOM
Qwen3.6-35B-A3B-UD-Q2_K_XL none 1 65536 off - - - 2.3 OOM
Qwen3.6-35B-A3B-UD-Q2_K_XL none 1 32768 off - - - 2.3 OOM
Qwen3.6-35B-A3B-UD-Q2_K_XL none 1 16384 off - - - 2.3 OOM
Qwen3.6-35B-A3B-UD-Q2_K_XL layer both 131072 off 52.2/56.9 92.9/57.2 202.7/57.1 17.8 OK
Qwen3.6-35B-A3B-UD-Q2_K_XL layer both 65536 off 69.6/56.7 108.3/56.7 207.9/56.5 19.3 OK
Qwen3.6-35B-A3B-UD-Q2_K_XL layer both 32768 off 68.5/56.4 107.3/56.6 208.1/56.4 19.3 OK
Qwen3.6-35B-A3B-UD-Q2_K_XL layer both 16384 off 75.2/56.6 108.1/56.6 208.5/56.4 19.2 OK

qwen3.8_27B

quantsmgpuctxmtpturn1 (p/g t/s)turn2 (p/g t/s)turn3 (p/g t/s)wall(s)status

Qwen3.8-27B-UD-IQ2_XXS none 0 131072 off - - - 19.4 OOM
Qwen3.8-27B-UD-IQ2_XXS none 0 65536 off - - - 3.0 OOM
Qwen3.8-27B-UD-IQ2_XXS none 0 32768 off 98.4/14.2 37.4/14.1 82.0/13.5 42.7 OK
Qwen3.8-27B-UD-IQ2_XXS none 0 16384 off 99.5/13.7 37.4/13.1 81.8/12.9 49.0 OK
Qwen3.8-27B-UD-IQ2_XXS none 1 131072 off - - - 5.0 OOM
Qwen3.8-27B-UD-IQ2_XXS none 1 65536 off - - - 5.0 OOM
Qwen3.8-27B-UD-IQ2_XXS none 1 32768 off 90.0/14.3 31.3/14.1 70.1/13.6 49.0 OK
Qwen3.8-27B-UD-IQ2_XXS none 1 16384 off 89.1/13.3 30.8/12.5 69.6/12.3 52.5 OK
Qwen3.8-27B-UD-IQ2_XXS layer both 131072 off 86.0/15.6 34.7/15.4 76.3/15.2 45.4 OK
Qwen3.8-27B-UD-IQ2_XXS layer both 65536 off 91.9/14.7 34.3/14.7 76.3/14.7 44.3 OK
Qwen3.8-27B-UD-IQ2_XXS layer both 32768 off 86.0/14.9 34.1/14.8 75.2/14.7 45.9 OK
Qwen3.8-27B-UD-IQ2_XXS layer both 16384 off 92.3/15.5 34.2/15.4 75.5/15.3 43.3 OK

Long context (single turn, >10k tokens)

Prompt: see long_prompt.txt (needle-in-haystack recall test).

familyquantsmgpuctxmtpprompt t/sgen t/swall(s)status

gemma4-12b gemma-4-12b-it-Q4_0 none 0 131072 off 415.9 30.7 37.6 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 131072 on(n=1) 405.2 48.8 38.7 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 131072 on(n=2) 403.5 52.4 38.6 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 131072 on(n=3) 402.4 53.1 38.7 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 131072 on(n=4) 401.9 43.0 39.3 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 65536 off 401.0 28.4 38.9 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 65536 on(n=1) 398.3 46.7 39.1 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 65536 on(n=2) 398.7 51.7 38.8 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 65536 on(n=3) 400.7 53.7 38.5 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 65536 on(n=4) 399.5 50.1 38.8 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 32768 off 397.4 29.7 38.9 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 32768 on(n=1) 395.0 47.8 39.3 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 32768 on(n=2) 397.6 50.6 38.9 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 32768 on(n=3) 397.0 48.8 39.0 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 32768 on(n=4) 397.5 43.2 39.3 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 16384 off 396.9 29.4 39.0 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 16384 on(n=1) 395.7 47.6 39.2 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 16384 on(n=2) 397.1 47.7 39.1 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 16384 on(n=3) 399.0 49.2 38.8 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 16384 on(n=4) 400.2 48.1 38.8 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 131072 off 412.5 28.7 39.8 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 131072 on(n=1) 393.7 46.3 41.5 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 131072 on(n=2) 386.5 41.0 42.5 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 131072 on(n=3) 387.7 45.0 42.1 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 131072 on(n=4) 389.2 43.3 42.1 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 65536 off 394.8 26.9 41.2 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 65536 on(n=1) 387.5 42.8 42.2 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 65536 on(n=2) 387.1 43.6 42.2 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 65536 on(n=3) 387.8 39.8 42.4 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 65536 on(n=4) 387.5 48.4 41.9 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 32768 off 391.2 27.1 41.5 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 32768 on(n=1) 392.1 42.3 41.7 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 32768 on(n=2) 387.5 48.8 41.6 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 32768 on(n=3) 387.5 47.0 41.7 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 32768 on(n=4) 386.7 45.3 41.9 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 16384 off 394.7 26.8 41.2 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 16384 on(n=1) 389.4 42.5 42.1 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 16384 on(n=2) 388.9 45.8 41.6 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 16384 on(n=3) 386.2 43.2 42.0 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 16384 on(n=4) 388.6 48.0 41.5 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 131072 off 726.7 29.9 25.8 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 131072 on(n=1) 605.9 35.3 31.0 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 131072 on(n=2) 603.5 29.9 31.7 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 131072 on(n=3) 599.3 31.4 31.6 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 131072 on(n=4) 602.7 21.1 33.5 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 65536 off 713.9 29.7 26.1 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 65536 on(n=1) 598.1 36.1 30.9 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 65536 on(n=2) 599.1 35.0 31.0 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 65536 on(n=3) 601.8 31.4 31.3 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 65536 on(n=4) 599.8 27.5 31.9 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 32768 off 706.8 29.6 26.0 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 32768 on(n=1) 598.9 35.8 30.6 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 32768 on(n=2) 597.8 31.0 31.2 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 32768 on(n=3) 595.9 29.8 31.5 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 32768 on(n=4) 596.5 31.0 31.3 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 16384 off 708.1 29.5 26.0 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 16384 on(n=1) 597.9 36.2 30.4 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 16384 on(n=2) 593.4 34.6 30.7 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 16384 on(n=3) 596.3 26.8 31.9 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 16384 on(n=4) 596.9 25.6 32.1 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 131072 off 352.9 26.0 43.1 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 131072 on(n=1) 353.3 34.6 43.8 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 131072 on(n=2) 351.9 35.4 43.8 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 131072 on(n=3) 352.7 31.9 44.2 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 131072 on(n=4) 352.7 34.5 43.9 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 65536 off 354.9 26.1 42.9 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 65536 on(n=1) 351.5 37.7 43.4 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 65536 on(n=2) 351.3 34.7 43.8 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 65536 on(n=3) 351.3 35.3 43.9 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 65536 on(n=4) 353.2 26.9 44.8 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 32768 off 355.2 25.9 42.9 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 32768 on(n=1) 352.1 40.1 43.2 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 32768 on(n=2) 352.1 37.6 43.4 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 32768 on(n=3) 350.8 30.1 44.4 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 32768 on(n=4) 351.1 26.7 44.9 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 16384 off 355.5 26.5 42.7 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 16384 on(n=1) 350.9 36.3 43.6 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 16384 on(n=2) 352.8 34.4 43.6 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 16384 on(n=3) 352.0 37.2 43.5 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 16384 on(n=4) 351.5 27.8 44.7 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 131072 off 366.7 25.5 43.1 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 131072 on(n=1) 355.2 33.6 45.2 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 131072 on(n=2) 356.2 33.5 45.1 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 131072 on(n=3) 351.6 29.5 46.1 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 131072 on(n=4) 353.7 26.5 46.3 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 65536 off 359.3 24.6 43.9 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 65536 on(n=1) 355.1 31.8 45.1 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 65536 on(n=2) 352.1 35.9 45.0 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 65536 on(n=3) 352.6 32.0 45.3 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 65536 on(n=4) 354.5 31.7 45.2 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 32768 off 357.9 24.7 44.1 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 32768 on(n=1) 353.7 33.6 45.0 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 32768 on(n=2) 354.3 32.9 45.1 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 32768 on(n=3) 349.9 31.1 45.8 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 32768 on(n=4) 353.1 28.7 45.8 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 16384 off 359.8 24.6 43.9 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 16384 on(n=1) 354.0 36.5 44.7 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 16384 on(n=2) 351.5 31.0 45.6 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 16384 on(n=3) 351.9 30.2 45.6 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 16384 on(n=4) 352.0 34.7 45.1 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 131072 off 639.1 27.8 27.9 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 131072 on(n=1) 531.0 29.5 34.0 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 131072 on(n=2) 525.5 25.6 34.9 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 131072 on(n=3) 525.3 23.5 35.3 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 131072 on(n=4) 522.1 19.6 36.5 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 65536 off 618.7 27.6 28.5 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 65536 on(n=1) 528.3 31.8 33.5 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 65536 on(n=2) 528.9 26.3 34.3 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 65536 on(n=3) 525.8 22.5 35.3 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 65536 on(n=4) 526.0 20.8 35.7 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 32768 off 614.3 27.9 28.4 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 32768 on(n=1) 525.6 31.0 33.5 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 32768 on(n=2) 524.3 32.0 33.4 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 32768 on(n=3) 524.7 23.9 34.8 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 32768 on(n=4) 525.1 22.1 35.1 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 16384 off 612.1 27.8 28.5 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 16384 on(n=1) 524.9 31.9 33.2 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 16384 on(n=2) 525.0 25.9 34.1 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 16384 on(n=3) 528.6 22.2 34.7 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 16384 on(n=4) 523.7 27.2 34.1 OK
gemma4-26b gemma-4-26B-A4B-it-UD-IQ2_M none 0 131072 off - - 3.7 OOM
gemma4-26b gemma-4-26B-A4B-it-UD-IQ2_M none 0 65536 off - - 3.6 OOM
gemma4-26b gemma-4-26B-A4B-it-UD-IQ2_M none 0 32768 off 727.3 57.4 23.1 OK
gemma4-26b gemma-4-26B-A4B-it-UD-IQ2_M none 0 16384 off 709.7 51.5 23.8 OK
gemma4-26b gemma-4-26B-A4B-it-UD-IQ2_M none 1 131072 off - - 6.1 OOM
gemma4-26b gemma-4-26B-A4B-it-UD-IQ2_M none 1 65536 off - - 6.1 OOM
gemma4-26b gemma-4-26B-A4B-it-UD-IQ2_M none 1 32768 off 724.9 52.5 26.0 OK
gemma4-26b gemma-4-26B-A4B-it-UD-IQ2_M none 1 16384 off 708.0 48.5 26.4 OK
gemma4-26b gemma-4-26B-A4B-it-UD-IQ2_M layer both 131072 off 1293.5 54.7 17.2 OK
gemma4-26b gemma-4-26B-A4B-it-UD-IQ2_M layer both 65536 off 1293.4 54.6 17.1 OK
gemma4-26b gemma-4-26B-A4B-it-UD-IQ2_M layer both 32768 off 1282.3 54.8 17.0 OK
gemma4-26b gemma-4-26B-A4B-it-UD-IQ2_M layer both 16384 off 1282.9 54.8 16.9 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 131072 off - - 3.4 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 131072 on(n=1) - - 4.1 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 131072 on(n=2) - - 4.1 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 131072 on(n=3) - - 4.1 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 131072 on(n=4) - - 4.1 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 65536 off - - 3.4 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 65536 on(n=1) - - 4.1 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 65536 on(n=2) - - 4.1 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 65536 on(n=3) - - 4.1 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 65536 on(n=4) - - 4.1 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 32768 off - - 3.4 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 32768 on(n=1) - - 4.2 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 32768 on(n=2) - - 4.2 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 32768 on(n=3) - - 4.2 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 32768 on(n=4) - - 4.2 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 16384 off - - 4.6 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 16384 on(n=1) - - 5.0 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 16384 on(n=2) - - 5.1 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 16384 on(n=3) - - 5.1 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 16384 on(n=4) - - 5.0 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 131072 off - - 5.5 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 131072 on(n=1) - - 6.2 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 131072 on(n=2) - - 6.2 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 131072 on(n=3) - - 6.3 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 131072 on(n=4) - - 6.3 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 65536 off - - 5.5 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 65536 on(n=1) - - 6.4 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 65536 on(n=2) - - 6.3 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 65536 on(n=3) - - 6.3 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 65536 on(n=4) - - 6.2 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 32768 off - - 5.5 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 32768 on(n=1) - - 6.3 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 32768 on(n=2) - - 6.3 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 32768 on(n=3) - - 6.3 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 32768 on(n=4) - - 6.3 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 16384 off 149.8 12.5 99.1 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 16384 on(n=1) - - 7.2 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 16384 on(n=2) - - 7.2 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 16384 on(n=3) - - 7.1 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 16384 on(n=4) - - 7.2 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 131072 off 262.7 13.3 62.0 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 131072 on(n=1) - - 6.6 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 131072 on(n=2) - - 6.6 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 131072 on(n=3) - - 6.6 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 131072 on(n=4) - - 6.5 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 65536 off 258.9 13.2 62.8 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 65536 on(n=1) 220.2 16.0 71.5 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 65536 on(n=2) 220.0 15.3 71.9 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 65536 on(n=3) 220.0 13.2 73.2 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 65536 on(n=4) 220.7 13.7 72.7 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 32768 off 256.0 13.3 63.0 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 32768 on(n=1) 219.9 16.0 71.3 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 32768 on(n=2) 219.9 13.8 72.6 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 32768 on(n=3) 219.5 14.4 72.3 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 32768 on(n=4) 220.0 13.2 73.0 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 16384 off 254.6 13.4 63.2 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 16384 on(n=1) 218.9 16.2 71.5 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 16384 on(n=2) 219.6 15.9 71.4 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 16384 on(n=3) 220.2 14.1 72.3 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 16384 on(n=4) 219.7 10.7 75.3 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 131072 off - - 2.9 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 131072 on(n=1) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 131072 on(n=2) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 131072 on(n=3) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 131072 on(n=4) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 65536 off - - 2.9 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 65536 on(n=1) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 65536 on(n=2) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 65536 on(n=3) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 65536 on(n=4) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 32768 off - - 2.9 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 32768 on(n=1) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 32768 on(n=2) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 32768 on(n=3) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 32768 on(n=4) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 16384 off - - 2.9 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 16384 on(n=1) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 16384 on(n=2) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 16384 on(n=3) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 16384 on(n=4) - - 3.7 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 131072 off - - 2.9 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 131072 on(n=1) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 131072 on(n=2) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 131072 on(n=3) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 131072 on(n=4) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 65536 off - - 2.9 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 65536 on(n=1) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 65536 on(n=2) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 65536 on(n=3) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 65536 on(n=4) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 32768 off - - 2.9 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 32768 on(n=1) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 32768 on(n=2) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 32768 on(n=3) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 32768 on(n=4) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 16384 off - - 2.9 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 16384 on(n=1) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 16384 on(n=2) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 16384 on(n=3) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 16384 on(n=4) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 131072 off - - 5.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 131072 on(n=1) - - 6.3 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 131072 on(n=2) - - 6.3 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 131072 on(n=3) - - 6.3 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 131072 on(n=4) - - 6.3 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 65536 off 245.3 12.3 67.1 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 65536 on(n=1) 207.3 15.1 76.6 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 65536 on(n=2) 207.2 13.8 77.3 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 65536 on(n=3) 207.3 14.5 76.8 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 65536 on(n=4) 207.0 14.0 77.2 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 32768 off 241.0 12.2 67.9 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 32768 on(n=1) 206.6 14.9 76.5 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 32768 on(n=2) 206.7 14.9 76.6 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 32768 on(n=3) 207.0 14.3 76.8 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 32768 on(n=4) 207.1 11.2 79.3 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 16384 off 241.0 12.3 67.8 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 16384 on(n=1) 207.0 15.6 76.1 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 16384 on(n=2) 207.1 14.3 76.8 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 16384 on(n=3) 207.7 14.4 76.5 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 16384 on(n=4) 207.5 10.4 80.0 OK
gemma4-e2b gemma-4-E2B-it-Q4_K_M none 0 131072 off 1424.6 85.0 12.8 OK
gemma4-e2b gemma-4-E2B-it-Q4_K_M none 0 65536 off 1419.1 85.0 12.8 OK
gemma4-e2b gemma-4-E2B-it-Q4_K_M none 0 32768 off 1415.2 82.6 12.9 OK
gemma4-e2b gemma-4-E2B-it-Q4_K_M none 0 16384 off 1409.1 84.8 12.8 OK
gemma4-e2b gemma-4-E2B-it-Q4_K_M none 1 131072 off 1420.8 79.4 13.3 OK
gemma4-e2b gemma-4-E2B-it-Q4_K_M none 1 65536 off 1395.5 78.1 13.5 OK
gemma4-e2b gemma-4-E2B-it-Q4_K_M none 1 32768 off 1393.3 74.5 13.5 OK
gemma4-e2b gemma-4-E2B-it-Q4_K_M none 1 16384 off 1334.5 76.5 13.9 OK
gemma4-e2b gemma-4-E2B-it-Q4_K_M layer both 131072 off 2339.7 18.3 15.3 OK
gemma4-e2b gemma-4-E2B-it-Q4_K_M layer both 65536 off 2333.8 18.5 15.0 OK
gemma4-e2b gemma-4-E2B-it-Q4_K_M layer both 32768 off 2351.6 18.3 15.0 OK
gemma4-e2b gemma-4-E2B-it-Q4_K_M layer both 16384 off 2348.7 18.5 14.9 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 131072 off 958.7 56.4 18.2 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 131072 on(n=1) 948.6 77.3 17.9 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 131072 on(n=2) 948.3 75.5 17.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 131072 on(n=3) 948.5 86.9 19.1 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 131072 on(n=4) 947.0 82.5 19.2 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 65536 off 942.2 55.0 16.1 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 65536 on(n=1) 942.1 72.0 19.4 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 65536 on(n=2) 945.1 76.4 17.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 65536 on(n=3) 946.9 80.1 19.2 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 65536 on(n=4) 945.3 80.6 17.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 32768 off 937.2 54.0 18.4 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 32768 on(n=1) 943.9 76.4 17.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 32768 on(n=2) 946.2 76.4 19.3 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 32768 on(n=3) 946.8 97.9 17.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 32768 on(n=4) 945.9 80.9 17.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 16384 off 931.8 55.0 18.4 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 16384 on(n=1) 943.3 75.0 17.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 16384 on(n=2) 946.7 75.9 17.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 16384 on(n=3) 944.4 95.5 17.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 16384 on(n=4) 947.5 85.0 17.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 131072 off 964.7 51.7 16.6 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 131072 on(n=1) 945.2 72.0 18.6 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 131072 on(n=2) 930.7 68.0 18.8 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 131072 on(n=3) 924.8 87.6 18.8 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 131072 on(n=4) 916.5 77.7 19.0 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 65536 off 921.0 47.8 19.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 65536 on(n=1) 923.9 67.8 20.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 65536 on(n=2) 937.7 84.9 20.0 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 65536 on(n=3) 918.3 67.3 20.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 65536 on(n=4) 911.5 65.6 20.9 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 32768 off 925.4 46.7 17.2 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 32768 on(n=1) 929.2 67.6 18.8 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 32768 on(n=2) 914.2 71.3 19.0 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 32768 on(n=3) 931.0 92.9 18.5 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 32768 on(n=4) 921.1 76.4 20.4 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 16384 off 922.1 47.6 19.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 16384 on(n=1) 918.5 68.7 20.4 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 16384 on(n=2) 921.6 66.1 20.5 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 16384 on(n=3) 924.5 85.4 18.6 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 16384 on(n=4) 901.6 64.5 21.0 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 131072 off 1697.8 53.3 13.2 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 131072 on(n=1) 1432.4 66.3 14.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 131072 on(n=2) 1429.3 65.2 16.3 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 131072 on(n=3) 1430.5 54.5 16.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 131072 on(n=4) 1430.8 64.6 16.3 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 65536 off 1695.8 52.5 10.9 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 65536 on(n=1) 1428.5 65.9 16.0 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 65536 on(n=2) 1426.1 65.4 16.0 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 65536 on(n=3) 1426.6 78.4 14.0 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 65536 on(n=4) 1428.6 65.6 14.2 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 32768 off 1692.2 53.2 12.9 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 32768 on(n=1) 1424.0 67.9 14.0 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 32768 on(n=2) 1423.1 64.9 15.8 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 32768 on(n=3) 1425.7 60.2 15.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 32768 on(n=4) 1427.5 65.6 13.9 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 16384 off 1650.6 51.8 10.9 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 16384 on(n=1) 1423.1 68.0 13.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 16384 on(n=2) 1423.0 62.9 13.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 16384 on(n=3) 1423.9 78.7 13.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 16384 on(n=4) 1420.6 62.1 15.7 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q2_K_XL none 0 131072 off 1024.0 40.8 17.8 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q2_K_XL none 0 65536 off 1022.1 40.7 17.8 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q2_K_XL none 0 32768 off 1022.5 40.7 17.8 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q2_K_XL none 0 16384 off 1019.5 40.8 17.8 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q2_K_XL none 1 131072 off 1024.6 39.9 18.5 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q2_K_XL none 1 65536 off 1018.0 38.6 18.7 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q2_K_XL none 1 32768 off 1014.1 37.7 18.8 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q2_K_XL none 1 16384 off 1003.5 37.7 18.9 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q2_K_XL layer both 131072 off 1037.1 39.9 18.2 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q2_K_XL layer both 65536 off 1033.6 39.6 18.2 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q2_K_XL layer both 32768 off 1032.2 38.6 18.3 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q2_K_XL layer both 16384 off 1031.6 39.5 18.0 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q4_K_XL none 0 131072 off 1051.2 63.5 16.7 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q4_K_XL none 0 65536 off 1041.5 60.0 14.6 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q4_K_XL none 0 32768 off 1025.4 58.2 16.8 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q4_K_XL none 0 16384 off 1034.2 58.7 16.7 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q4_K_XL none 1 131072 off 1046.0 59.7 17.4 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q4_K_XL none 1 65536 off 1026.5 57.1 17.7 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q4_K_XL none 1 32768 off 1022.3 55.6 17.8 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q4_K_XL none 1 16384 off 1017.2 58.1 15.7 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q4_K_XL layer both 131072 off 1850.1 59.4 10.1 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q4_K_XL layer both 65536 off 1855.4 59.9 12.0 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q4_K_XL layer both 32768 off 1848.5 59.3 9.8 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q4_K_XL layer both 16384 off 1849.1 59.9 11.8 OK
qwen3.5 Qwen3.5-0.8B-UD-IQ2_M none 0 131072 off 4451.0 161.5 5.0 OK
qwen3.5 Qwen3.5-0.8B-UD-IQ2_M none 0 65536 off 4453.3 155.3 4.3 OK
qwen3.5 Qwen3.5-0.8B-UD-IQ2_M none 0 32768 off 4448.8 161.6 5.0 OK
qwen3.5 Qwen3.5-0.8B-UD-IQ2_M none 0 16384 off 4449.4 161.7 5.0 OK
qwen3.5 Qwen3.5-0.8B-UD-IQ2_M none 1 131072 off 4420.6 148.2 5.3 OK
qwen3.5 Qwen3.5-0.8B-UD-IQ2_M none 1 65536 off 4407.3 149.0 5.1 OK
qwen3.5 Qwen3.5-0.8B-UD-IQ2_M none 1 32768 off 4393.5 149.1 5.1 OK
qwen3.5 Qwen3.5-0.8B-UD-IQ2_M none 1 16384 off 4388.0 147.9 5.1 OK
qwen3.5 Qwen3.5-0.8B-UD-IQ2_M layer both 131072 off 7569.6 151.9 4.3 OK
qwen3.5 Qwen3.5-0.8B-UD-IQ2_M layer both 65536 off 7531.3 150.5 4.3 OK
qwen3.5 Qwen3.5-0.8B-UD-IQ2_M layer both 32768 off 7533.3 151.2 4.0 OK
qwen3.5 Qwen3.5-0.8B-UD-IQ2_M layer both 16384 off 7511.7 150.3 4.0 OK
qwen3.5 Qwen3.5-0.8B-UD-Q4_K_XL none 0 131072 off 4460.4 151.0 5.1 OK
qwen3.5 Qwen3.5-0.8B-UD-Q4_K_XL none 0 65536 off 4436.3 150.7 5.1 OK
qwen3.5 Qwen3.5-0.8B-UD-Q4_K_XL none 0 32768 off 4422.0 150.1 5.1 OK
qwen3.5 Qwen3.5-0.8B-UD-Q4_K_XL none 0 16384 off 4415.1 150.3 5.1 OK
qwen3.5 Qwen3.5-0.8B-UD-Q4_K_XL none 1 131072 off 4387.4 139.5 5.4 OK
qwen3.5 Qwen3.5-0.8B-UD-Q4_K_XL none 1 65536 off 4375.3 139.4 5.4 OK
qwen3.5 Qwen3.5-0.8B-UD-Q4_K_XL none 1 32768 off 4353.9 139.5 5.4 OK
qwen3.5 Qwen3.5-0.8B-UD-Q4_K_XL none 1 16384 off 4349.8 139.3 5.4 OK
qwen3.5 Qwen3.5-0.8B-UD-Q4_K_XL layer both 131072 off 7328.9 142.1 4.4 OK
qwen3.5 Qwen3.5-0.8B-UD-Q4_K_XL layer both 65536 off 7368.6 142.4 4.4 OK
qwen3.5 Qwen3.5-0.8B-UD-Q4_K_XL layer both 32768 off 7343.5 141.4 4.3 OK
qwen3.5 Qwen3.5-0.8B-UD-Q4_K_XL layer both 16384 off 7363.9 140.2 4.3 OK
qwen3.5 Qwen3.5-0.8B-UD-Q8_K_XL none 0 131072 off 4128.0 126.5 5.7 OK
qwen3.5 Qwen3.5-0.8B-UD-Q8_K_XL none 0 65536 off 4109.7 126.4 5.7 OK
qwen3.5 Qwen3.5-0.8B-UD-Q8_K_XL none 0 32768 off 4084.1 126.3 5.5 OK
qwen3.5 Qwen3.5-0.8B-UD-Q8_K_XL none 0 16384 off 4090.5 125.8 5.7 OK
qwen3.5 Qwen3.5-0.8B-UD-Q8_K_XL none 1 131072 off 4092.0 118.4 5.9 OK
qwen3.5 Qwen3.5-0.8B-UD-Q8_K_XL none 1 65536 off 4062.2 118.1 6.0 OK
qwen3.5 Qwen3.5-0.8B-UD-Q8_K_XL none 1 32768 off 4054.0 117.9 6.0 OK
qwen3.5 Qwen3.5-0.8B-UD-Q8_K_XL none 1 16384 off 4026.7 113.0 6.0 OK
qwen3.5 Qwen3.5-0.8B-UD-Q8_K_XL layer both 131072 off 6803.8 120.2 5.1 OK
qwen3.5 Qwen3.5-0.8B-UD-Q8_K_XL layer both 65536 off 6906.1 120.4 4.8 OK
qwen3.5 Qwen3.5-0.8B-UD-Q8_K_XL layer both 32768 off 6870.5 120.5 4.8 OK
qwen3.5 Qwen3.5-0.8B-UD-Q8_K_XL layer both 16384 off 6862.7 119.3 4.8 OK
qwen3.6_27B Qwen3.6-27B-Q4_K_S none 0 131072 off - - 2.7 OOM
qwen3.6_27B Qwen3.6-27B-Q4_K_S none 0 65536 off - - 2.7 OOM
qwen3.6_27B Qwen3.6-27B-Q4_K_S none 0 32768 off - - 2.7 OOM
qwen3.6_27B Qwen3.6-27B-Q4_K_S none 0 16384 off - - 2.5 OOM
qwen3.6_27B Qwen3.6-27B-Q4_K_S none 1 131072 off - - 2.5 OOM
qwen3.6_27B Qwen3.6-27B-Q4_K_S none 1 65536 off - - 2.7 OOM
qwen3.6_27B Qwen3.6-27B-Q4_K_S none 1 32768 off - - 2.5 OOM
qwen3.6_27B Qwen3.6-27B-Q4_K_S none 1 16384 off - - 2.7 OOM
qwen3.6_27B Qwen3.6-27B-Q4_K_S layer both 131072 off - - 5.9 OOM
qwen3.6_27B Qwen3.6-27B-Q4_K_S layer both 65536 off 397.0 14.7 44.1 OK
qwen3.6_27B Qwen3.6-27B-Q4_K_S layer both 32768 off 386.4 14.6 44.9 OK
qwen3.6_27B Qwen3.6-27B-Q4_K_S layer both 16384 off 386.7 14.7 44.8 OK
qwen3.6_35B Qwen3.6-35B-A3B-UD-Q2_K_XL none 0 131072 off - - 2.4 OOM
qwen3.6_35B Qwen3.6-35B-A3B-UD-Q2_K_XL none 0 65536 off - - 2.3 OOM
qwen3.6_35B Qwen3.6-35B-A3B-UD-Q2_K_XL none 0 32768 off - - 2.3 OOM
qwen3.6_35B Qwen3.6-35B-A3B-UD-Q2_K_XL none 0 16384 off - - 2.3 OOM
qwen3.6_35B Qwen3.6-35B-A3B-UD-Q2_K_XL none 1 131072 off - - 2.3 OOM
qwen3.6_35B Qwen3.6-35B-A3B-UD-Q2_K_XL none 1 65536 off - - 2.3 OOM
qwen3.6_35B Qwen3.6-35B-A3B-UD-Q2_K_XL none 1 32768 off - - 2.3 OOM
qwen3.6_35B Qwen3.6-35B-A3B-UD-Q2_K_XL none 1 16384 off - - 2.3 OOM
qwen3.6_35B Qwen3.6-35B-A3B-UD-Q2_K_XL layer both 131072 off 1415.3 53.2 15.9 OK
qwen3.6_35B Qwen3.6-35B-A3B-UD-Q2_K_XL layer both 65536 off 1406.3 53.1 15.7 OK
qwen3.6_35B Qwen3.6-35B-A3B-UD-Q2_K_XL layer both 32768 off 1388.6 51.4 13.8 OK
qwen3.6_35B Qwen3.6-35B-A3B-UD-Q2_K_XL layer both 16384 off 1401.5 53.1 15.7 OK
qwen3.8_27B Qwen3.8-27B-UD-IQ2_XXS none 0 131072 off - - 3.0 OOM
qwen3.8_27B Qwen3.8-27B-UD-IQ2_XXS none 0 65536 off - - 2.9 OOM
qwen3.8_27B Qwen3.8-27B-UD-IQ2_XXS none 0 32768 off 207.2 11.9 64.0 OK
qwen3.8_27B Qwen3.8-27B-UD-IQ2_XXS none 0 16384 off 207.1 12.3 69.3 OK
qwen3.8_27B Qwen3.8-27B-UD-IQ2_XXS none 1 131072 off - - 5.0 OOM
qwen3.8_27B Qwen3.8-27B-UD-IQ2_XXS none 1 65536 off - - 5.0 OOM
qwen3.8_27B Qwen3.8-27B-UD-IQ2_XXS none 1 32768 off 206.2 11.6 68.0 OK
qwen3.8_27B Qwen3.8-27B-UD-IQ2_XXS none 1 16384 off 204.5 11.4 73.0 OK
qwen3.8_27B Qwen3.8-27B-UD-IQ2_XXS layer both 131072 off 379.1 14.1 44.1 OK
qwen3.8_27B Qwen3.8-27B-UD-IQ2_XXS layer both 65536 off 371.6 13.3 40.7 OK
qwen3.8_27B Qwen3.8-27B-UD-IQ2_XXS layer both 32768 off 367.4 13.7 45.0 OK
qwen3.8_27B Qwen3.8-27B-UD-IQ2_XXS layer both 16384 off 366.7 13.9 41.2 OK
저작자표시 (새창열림)

'모종의 음모 > ai 프로그램' 카테고리의 다른 글

말뭉치, 코퍼스(corpus)  (0) 2026.09.21
토크나이저  (0) 2026.09.21
v100 흐음...  (0) 2026.09.16
llama.cpp pascal 지원 빌드 / 성능 테스트  (0) 2026.09.16
1080 ti / pcie3.0 설정후 재 테스트  (0) 2026.09.15
Posted by 구차니
모종의 음모/ai 프로그램2026. 9. 16. 17:08

v100 흐음...

v100 16GB * 4 = 64GB

돈도 돈이지만 전기도 전기라

이 걸 업어오냐 마냐가 고민

 

112token/s 나오는거 보면 끌리긴 한단 말이지.. (1080 ti의 2~3배 사이)

[링크 : https://willitrunai.com/zh/can-run/gemma-4-e4b-on-v100-32gb]

저작자표시 (새창열림)

'모종의 음모 > ai 프로그램' 카테고리의 다른 글

토크나이저  (0) 2026.09.21
LLM / 1080 ti 모드 별 벤치마크  (0) 2026.09.17
llama.cpp pascal 지원 빌드 / 성능 테스트  (0) 2026.09.16
1080 ti / pcie3.0 설정후 재 테스트  (0) 2026.09.15
auroc - area under the ROC curve  (0) 2026.09.01
Posted by 구차니
모종의 음모/ai 프로그램2026. 9. 16. 11:55

llama.cpp pascal 지원 빌드 / 성능 테스트

그냥 귀찮아서 가장 최신 받아서

claude 에서 빌드해달라고 하고 해버림

여전히 -sm row, -sm tensor 는 안되는데 머가 문제이려나?

 

cuda로 해서 그런가.. 아니면 가장 최신버전 이라서 성능향상이 생긴건가?

commit 930e2fa5995789efbf249a8bf61325bb626e417b (HEAD -> master, tag: b10991, origin/master, origin/HEAD)
Author: Jhen-Jie Hong <iainst0409@gmail.com>
Date:   Wed Sep 16 07:02:06 2026 +0800

    hexagon: add back missing contiguous fast-path and hvx_copy_uu for each run (#28886)

 

11 정도였던것 같은데 약간 상승한듯?

/mnt/Downloads/llama.cpp/build/bin$ ./llama-cli -m /mnt/Downloads/model/gemma4-31b/gemma-4-31B-it-UD-Q2_K_XL.gguf -sm layer


> 안녕?
[ Prompt: 35.4 t/s | Generation: 14.3 t/s ]

> 너에 대해 설명해줘
[ Prompt: 63.5 t/s | Generation: 13.9 t/s ]

> 파이썬으로 셀레니움을 통해 웹을 서칭하고 텍스트만 추출하고 makrdown 으로 변환후  md 파일과 pdf로 저장하는 기능을 구현해줘

[ Prompt: 162.4 t/s | Generation: 13.3 t/s ]

 

pcie3.0 은 별 차이 없었으니 일단 이전 기록 끌어오면 2token/s 정도씩 상승한듯?

$ ../../llama-b9692/llama-cli -m ./gemma-4-31B-it-UD-Q2_K_XL.gguf 
안녕?
[ Prompt: 8.9 t/s | Generation: 12.2 t/s ]

너에 대해 설명해줘
[ Prompt: 33.3 t/s | Generation: 11.8 t/s ]

파이썬으로 셀레니움을 통해 웹을 서칭하고 텍스트만 추출하고 makrdown 으로 변환후  md 파일과 pdf로 저장하는 기능을 구현해줘
[ Prompt: 36.1 t/s | Generation: 11.4 t/s ]

2026.07.30 - [모종의 음모/ai 프로그램] - llama.cpp / 1080 ti 11GB * 2/ gemma4-31B

 

 

 

+

클로드가 왜 row 안되는지 보고 있는데 split buffer 모드가 멀까?

● -sm row이 실패하는 원인을 정확히 잡았습니다: GTX 1080 Ti가 CUDA의 "split buffer"를 지원하지 않는다는 에러입니다.
  소스에서 어떤 조건인지 확인하겠습니다.

 

구글 검색 ai 답변

volta 사라고 등 떠미는 느낌이지 왜? ㅠㅠ

 
  • 지원 CUDA 버전: 기술적으로는 멀티 GPU 메모리 접근 및 P2P(Peer-to-Peer) 전송이 가능한 CUDA 9.x / 10.x 이상의 모든 최신 버전에서 지원합니다. [1]
 
  • 지원 하드웨어 SM: GPU 간 데이터 통신(NVLink 또는 PCIe 클리크)이 지원되는 하드웨어라면 대부분 구동됩니다.
    • SM 7.0 (Volta), SM 7.5 (Turing)
    • SM 8.0 / 8.6 (Ampere)
    • SM 9.0 (Hopper) 등 대부분의 메인스트림 아키텍처 [1, 2, 3, 4]
저작자표시 (새창열림)

'모종의 음모 > ai 프로그램' 카테고리의 다른 글

LLM / 1080 ti 모드 별 벤치마크  (0) 2026.09.17
v100 흐음...  (0) 2026.09.16
1080 ti / pcie3.0 설정후 재 테스트  (0) 2026.09.15
auroc - area under the ROC curve  (0) 2026.09.01
llama.cpp vulkan / intel uhd 630  (0) 2026.09.01
Posted by 구차니
모종의 음모/ai 프로그램2026. 9. 15. 10:40

1080 ti / pcie3.0 설정후 재 테스트

두번째 녀석이 4,1로 떠서 pcie1.0 x4 lane으로 작동하고 있었는데

$ nvidia-smi --query-gpu=pcie.link.width.current,pcie.link.gen.current 
pcie.link.width.current, pcie.link.gen.current
16, 3
4, 1

 

bios 에서 pcie3.0으로 설정해서 3.0 으로라도 잡히게 일단 하고 시도

 

그냥은 1로 뜨고(절전모드) 부하를 걸어주면 이렇게 올라온다.

$ nvidia-smi --query-gpu=pcie.link.width.current,pcie.link.gen.current 
pcie.link.width.current, pcie.link.gen.current
16, 3
4, 3

 

llama.cpp 에서 -sm row, tensor 모두 실패.

layer만 가능한데 돌려보니 이전에 pcie1.0 일것으로 추정되는 때랑 차이가 없다.

$ ../../llama-b10488/llama-cli -m gemma-4-31B-it-UD-Q2_K_XL.gguf -sm layer

> 안녕?
[ Prompt: 1.3 t/s | Generation: 12.0 t/s ]

> 너에 대해 설명해줘
[ Prompt: 20.4 t/s | Generation: 11.8 t/s ]

> 파이썬으로 셀레니움을 통해 웹을 서칭하고 텍스트만 추출하고 makrdown 으로 변환후  md 파일과 pdf로 저장하는 기능을 구현해줘
[ Prompt: 30.6 t/s | Generation: 11.5 t/s ]



 

이건 이전 기록.. 흐음..

$ ../../llama-b9692/llama-cli -m ./gemma-4-31B-it-UD-Q2_K_XL.gguf 
안녕?
[ Prompt: 8.9 t/s | Generation: 12.2 t/s ]

너에 대해 설명해줘
[ Prompt: 33.3 t/s | Generation: 11.8 t/s ]

파이썬으로 셀레니움을 통해 웹을 서칭하고 텍스트만 추출하고 makrdown 으로 변환후  md 파일과 pdf로 저장하는 기능을 구현해줘
[ Prompt: 36.1 t/s | Generation: 11.4 t/s ]

2026.07.30 - [모종의 음모/ai 프로그램] - llama.cpp / 1080 ti 11GB * 2/ gemma4-31B

 

저작자표시 (새창열림)

'모종의 음모 > ai 프로그램' 카테고리의 다른 글

v100 흐음...  (0) 2026.09.16
llama.cpp pascal 지원 빌드 / 성능 테스트  (0) 2026.09.16
auroc - area under the ROC curve  (0) 2026.09.01
llama.cpp vulkan / intel uhd 630  (0) 2026.09.01
stable diffusion model 이란  (0) 2026.08.28
Posted by 구차니
모종의 음모/ai 프로그램2026. 9. 1. 16:37

auroc - area under the ROC curve

claude가 ai 모델 학습 관련 응답에서 갑자기 나온 첨 듣는단어.

어디서 기어나왔나 했떠니 통계쪽에서 쓰는 말인듯.

간단하게 관심을 가지는 곡선의 아래면적.  그러니까 이게 넓은게 유리한건가?

 

[링크 : https://m.blog.naver.com/i-doctor/222700461406]

 

AUROC(Area Under ROC)는 의학통계에 사용되며, 최근에는 프로그래밍 분야에서도 사용되는 개념입니다.

[링크 : https://dizarr.tistory.com/514]

저작자표시 (새창열림)

'모종의 음모 > ai 프로그램' 카테고리의 다른 글

llama.cpp pascal 지원 빌드 / 성능 테스트  (0) 2026.09.16
1080 ti / pcie3.0 설정후 재 테스트  (0) 2026.09.15
llama.cpp vulkan / intel uhd 630  (0) 2026.09.01
stable diffusion model 이란  (0) 2026.08.28
llama swap matrix  (0) 2026.08.28
Posted by 구차니

티스토리툴바