'모종의 음모/ai 프로그램'에 해당되는 글 175건

  1. 2026.09.22 wikimatrix
  2. 2026.09.22 직접 LLM 만들...기?
  3. 2026.09.22 llama-tokenize
  4. 2026.09.22 llama-cli /clear
  5. 2026.09.22 llama-simple
  6. 2026.09.21 RoPE - Rotary Position Embedding
  7. 2026.09.21 말뭉치, 코퍼스(corpus)
  8. 2026.09.21 토크나이저
  9. 2026.09.17 LLM / 1080 ti 모드 별 벤치마크
  10. 2026.09.16 v100 흐음...
모종의 음모/ai 프로그램2026. 9. 22. 16:34

wikimatrix

claude랑 ai 학습 시키면서 이것저것 물어보는데

한-영 번역을 어떻게 할수 있나 해서

위키피디아 영문/한글 두개 넣어서 돌리면 알아서 번역가능한거 아니냐 하니 그건또 아니라고 한다. ㅠㅠ

정확하게 매칭된 한 문장이 다른 분장으로 번역된 쌍이 필요한 것 이라고.

 

그럼 결국 기계번역도 수 많은 번역을 해온 자원봉사자들의 위에 우리가 혜택을 누리는건가..

  한국어 위키의 "서울" 문서와 영어 위키의 "Seoul" 문서는 서로 번역이 아닙니다. 각각 별도의 저자들이 독립적으로 쓴 글이라, 분량도 다르고,
  다루는 내용도 다르고, 문장 순서도 다릅니다. 두 코퍼스를 그냥 섞어서 "다음 토큰 맞추기"로 학습시키면, 모델은 "이 한국어 문단과 저 영어 문단이
  같은 의미다"라는 신호를 전혀 받지 못합니다 — 그냥 같은 파라미터 공간 안에 두 개의 독립된 언어 능력이 나란히 쌓일 뿐이지, 둘을 잇는 다리가
  저절로 생기지 않습니다. 지난번 "베이스 모델이 처음 보는 지식은 SFT로 못 만든다"는 얘기와 같은 원리가 여기도 적용됩니다 — 정렬(alignment) 
  신호가 없는 데이터는 아무리 섞어도 번역 능력을 만들지 않습니다.
  
  실제로 이걸 해낸 방법이 있습니다 — WikiMatrix 

  Meta(당시 Facebook AI)가 정확히 "위키백과에서 자동으로 문장 쌍을 뽑아내는" 문제를 풀었습니다:

  - LASER라는 다국어 문장 임베딩 모델로 93개 언어의 문장을 같은 벡터 공간에 투영
  - 한국어 문장과 영어 문장의 벡터가 얼마나 가까운지(의미적으로 비슷한지) 계산해서, 진짜로 비슷한 뜻인 문장 쌍만 골라냄
  - 이렇게 85개 언어에서 1억 3,500만 개 병렬 문장을 자동으로 채굴했고, 결과물이 공개돼 있습니다

  즉 "위키 두 개를 그냥 넣는 것"과 "WikiMatrix식으로 의미 유사도를 계산해서 진짜 대응 문장만 골라내는 것" 사이엔 **완전히 다른 파이프라인
  하나(문장 임베딩 모델 + 유사도 필터링)**가 끼어 있습니다. 다행히 이건 우리가 처음부터 만들 필요 없이, WikiMatrix가 이미 뽑아놓은 한영 병렬
  문장 쌍을 가져다 쓰는 게 현실적입니다.
 
  정리하면

  ❌ 한국어 위키 + 영어 위키를 그냥 같이 넣기
     → 두 언어를 "따로따로" 아는 모델은 될 수 있어도, 번역은 안 됨
 
  ✅ WikiMatrix 같은 이미 정렬된 병렬 코퍼스를 사전학습/SFT에 넣기
     → "이 한국어 문장 = 이 영어 문장"이라는 신호가 명시적으로 존재

 

[링크 : https://opus.nlpl.eu/datasets/WikiMatrix]

[링크 : https://huggingface.co/datasets/SEACrowd/wikimatrix]

저작자표시 (새창열림)

'모종의 음모 > ai 프로그램' 카테고리의 다른 글

diffusion gemma  (0) 2026.09.23
직접만든 LLM .. part 2  (0) 2026.09.23
직접 LLM 만들...기?  (0) 2026.09.22
llama-tokenize  (0) 2026.09.22
llama-cli /clear  (0) 2026.09.22
Posted by 구차니
모종의 음모/ai 프로그램2026. 9. 22. 14:28

직접 LLM 만들...기?

ai가 해줘도 직접인가..?

아무튼 중2 개발자가 직접 만든 한글 LLM 이란거에 한달뒤에 늦게 발동해서 해보는데

[링크 : https://news.hada.io/topic?id=32792]

    [링크 :https://github.com/seoan1024/korean-llm-v3]

    [링크 : https://github.com/seoan1024/Korean-llm-v4]

 

막상 해보고 나니 무슨 의미가 있나 싶기도 하고

일단 gemma4-e4b / gemma4-31B / qwen3.8 등에서 llama-tokenizer 로 돌려보니

생각외로 큰 차이는 없었고(내가 한국어사전을 이용해서 한거라 제법 단어로 될 줄 알았더니 의외로 안되서 실망)

결국 중요한건 학습셋 인가 라는 결론만..

 

1080ti 1개로

8시간 에 2000 epoch 학습

양자화 하지 않을 경우 163MB / q4_0 양자화시 28MB

생각보다 메모리 많이 먹고 있다.

 

토큰은 우리말사전을 기준으로 추출

학습셋은 한국어 위키피디아.

 

 

+

크기
 
  ┌────────────────────────┬─────────────┐
  │        구성요소        │ 파라미터 수 │
  ├────────────────────────┼─────────────┤
  │ 임베딩 (32,448 × 512)  │ 16,613,376  │
  ├────────────────────────┼─────────────┤
  │ 트랜스포머 블록 × 8층  │ 25,690,112  │
  ├────────────────────────┼─────────────┤
  │ 정규화(RMSNorm) × 17개 │ 9,216       │
  ├────────────────────────┼─────────────┤
  │ 합계                   │ ≈ 42.3M     │
  └────────────────────────┴─────────────┘

  출력층은 별도 파라미터 없이 임베딩 행렬을 그대로 재사용합니다(tied embedding) — 입력 임베딩과 출력 확률 계산이 같은 32,448×512 행렬을 공유해서 파라미터를 아낍니다.

  블록 1개(3,211,264개) 안에서는 어텐션이 1,048,576개, SwiGLU가 2,162,688개로 FFN이 어텐션의 약 2배 차지합니다. 이건 LLaMA류의 전형적인 비율입니다.

  입력 토큰 id
    → 임베딩(emb)                         : id → 512차원 벡터
    → [블록 × 8]
         x = x + Attention(RMSNorm(x))    : 잔차 연결 1
         x = x + SwiGLU(RMSNorm(x))       : 잔차 연결 2
    → 최종 RMSNorm(norm)
    → emb.weight.T 와 곱함                : 512차원 → 32,448개 토큰 확률(logits)

  - RMSNorm: LayerNorm의 경량 버전. 평균을 빼지 않고 제곱평균으로만 스케일을 맞춥니다(x * rsqrt(mean(x²))). 학습 가능한 스케일 벡터 w 하나만
    가집니다(bias 없음). LLaMA/Gemma 계열이 공통으로 쓰는 방식이고, 일반 LayerNorm보다 계산이 싸고 안정적입니다.
  - rope(x, cos, sin): 회전 위치 인코딩(RoPE). Q, K 벡터를 절반씩 잘라(x1, x2) 2D 회전을 적용해서 "이 토큰이 몇 번째 위치인가"를 벡터 자체에
    새겨 넣습니다. 별도의 위치 임베딩 파라미터가 없고, 상대적 거리 정보가 자연스럽게 학습됩니다.
  - Attention:
    - qkv 선형층 하나로 Q/K/V를 한 번에 계산(512 → 1536)
    - 8개 헤드로 나눠 각각 RoPE 적용
    - F.scaled_dot_product_attention(..., is_causal=True) — PyTorch 내장 함수로, 이 안에서 Flash Attention 커널이 자동 선택됩니다(가능한
      하드웨어에서). causal=True라 미래 토큰을 보지 못하게 마스킹됩니다.
    - GQA/MQA 없이 순수 MHA(8쿼리헤드=8키헤드) — 이전에 설명드렸듯 이 모델 크기·문맥길이(512)에서는 KV 캐시 절감 이득보다 품질 손실이 더 클 수
      있어 안 씁니다.
    - 마지막에 o 선형층으로 다시 섞음.
  - SwiGLU: 게이트 달린 FFN. w13이 두 벡터(a, b)를 동시에 뽑고, silu(a) * b로 게이팅한 뒤 w2로 다시 512차원으로 축소. 일반 ReLU-FFN보다
    표현력이 좋다고 알려져 LLaMA/PaLM 계열이 표준으로 씁니다.
  - Block: 위 둘을 Pre-Norm 방식(정규화 먼저, 그다음 서브레이어)으로 감싸고 잔차 연결. GPT-2식 Post-Norm보다 학습이 안정적입니다.
  - 초기화: 일반 가중치는 std=0.02 정규분포. 단, 각 블록의 "잔차에 더해지는 마지막 선형층"(attn.o, mlp.w2)은 0.02/sqrt(2*8)로 더 작게
    초기화합니다 — 층이 깊어질수록 잔차 누적으로 분산이 커지는 걸 막는 GPT-2/nanoGPT식 트릭입니다.
  - forward: 학습 시 targets를 주면 다음 토큰 예측 교차엔트로피 손실까지 한 번에 계산. logits.view(-1, V) 형태로 펴서 전체 배치·시퀀스 위치를
    한꺼번에 손실 계산합니다.
  - generate: 자기회귀 생성 루프.
    - 매 스텝 마지막 block_size(512) 토큰만 잘라서 forward (문맥 길이 제한)
    - temperature로 확률 분포를 누그러뜨리고, top_k로 후보를 40개로 제한한 뒤 multinomial로 샘플링 (확률적 생성 — 어제 보여드린 samples.txt가
      이 방식)
    - eos_id를 만나면 조기 종료 (SFT용으로 <|end|> id를 넘길 수 있게 만들어둔 파라미터)
    - 제가 별도로 검증할 때 쓴 "탐욕적(top_k=1 상당)" 생성은 이 함수를 안 쓰고 argmax로 직접 만든 스크립트였습니다 — 결정적 결과 확인용.

  설정값 (Config)

  ┌──────────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────┐
  │                값                │                                              의미                                              │
  ├──────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ vocab_size=32448                 │ 실제 토큰 32,408개를 GPU 효율을 위해 64의 배수로 패딩 (GGUF 내보낼 때는 실제 개수로 다시 자름) │
  ├──────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ block_size=512                   │ 한 번에 볼 수 있는 최대 토큰 수 (문맥 길이) — llama-cli 대화 모드에서 넘치면 에러 나던 원인    │
  ├──────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ n_layer=8, n_head=8, d_model=512 │ 8층, 헤드당 64차원(512/8)                                                                      │
  ├──────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ffn_hidden=1408                  │ SwiGLU 내부 확장 차원                                                                          │
  ├──────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ rope_base=10000.0                │ RoPE 회전 주파수 기준값 (LLaMA 기본값과 동일)                                                  │
  └──────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────┘

  총평: 이 모델은 아키텍처 자체는 실제 LLaMA/Mistral과 완전히 동일한 축소판입니다(RMSNorm+RoPE+SwiGLU+tied embedding+causal MHA). 차이는 오직 규모(8층/512차원 vs 수십층/수천차원)와 GQA 유무뿐이라, 여기서 배운 구조 지식은 실제 대형 오픈소스 모델을 읽을 때 그대로 적용됩니다.

 

암만봐도 이거랑 유사한 단어들이 많이 나온단 말이지..

[링크 : https://github.com/seoan1024/Korean-llm-v4]

 

아키텍쳐상 계통이 같은 듯?

● 네, 구조상 아키텍처 계열이 같습니다 — 다만 규모와 세부 구현은 상당히 다릅니다.
  
  비교
 
  ┌────────────────────┬───────────────────────────────────────────┬──────────────────────────────────────────────────────────────────────┐
  │        항목        │          이 프로젝트(seoan1024)           │                             llm-ko(여기)                             │
  ├────────────────────┼───────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┤
  │ 정규화/위치/활성화 │ RMSNorm + RoPE + SwiGLU                   │ 동일                                                                 │
  ├────────────────────┼───────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┤
  │ 임베딩 공유        │ O                                         │ O                                                                    │
  ├────────────────────┼───────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┤
  │ 파라미터 수        │ 약 1.09B                                  │ 약 42.3M (25배 작음)                                                 │
  ├────────────────────┼───────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┤
  │ 레이어/은닉/헤드   │ 20층 / 1920차원 / 10헤드                  │ 8층 / 512차원 / 8헤드                                                │
  ├────────────────────┼───────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┤
  │ 토크나이저         │ 남의 것 재사용 (beomi/Llama-3-Open-Ko-8B) │ 직접 학습(SentencePiece Unigram, 32,408개)                           │
  ├────────────────────┼───────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┤
  │ 정밀도             │ BF16                                      │ fp32 (우리 GPU는 Pascal이라 fp16/bf16 이득 없음, 이전에 실측 확인함) │
  ├────────────────────┼───────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┤
  │ 학습 단계          │ pretrain → SFT 2단계                      │ 동일 구조 (base → sft, pipeline.sh)                                  │
  ├────────────────────┼───────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┤
  │ 라이선스           │ GPL-3.0                                   │ 목표: Apache-2.0/MIT                                                 │
  ├────────────────────┼───────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┤
  │ 요구 하드웨어      │ RTX 3060 이상, RAM 64GB, 1TB NVMe         │ GTX 1080 Ti ×2 (훨씬 낮은 사양 지향)                                 │
  └────────────────────┴───────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────┘

  왜 비슷해 보이는가

  RMSNorm+RoPE+SwiGLU+임베딩공유 조합은 LLaMA가 2023년에 공개한 이후 사실상 업계 표준이 됐습니다(Mistral, Gemma, Qwen 등도 동일 계열). 이건
  특정 프로젝트의 독창적 아이디어가 아니라 공개된 논문·아키텍처를 다들 그대로 쓰는 것이라, "닮았다"기보다는 같은 표준 설계를 각자 구현한 것에
  가깝습니다. 2단계(pretrain→SFT) 구조도 요즘 오픈소스 LLM 학습의 사실상 정석 흐름이고요.

  실제 차이가 나는 지점은:
  - 규모: 저쪽은 1B급(RTX 4090 권장)까지 키웠고, 우리는 1080 Ti 2장으로 42M급을 목표로 함
  - 토크나이저: 저쪽은 이미 있는 8B짜리 모델의 토크나이저를 가져다 씀. 우리는 처음부터 직접 학습 — 이게 우리 프로젝트에서 꽤 공들인 부분이라
    차별점입니다
  - 라이선스: 여기가 중요한데, 저쪽은 GPL-3.0입니다. 우리는 Apache/MIT를 목표로 하고 있으니 저 저장소의 코드를 참고해서 베끼면 안 됩니다(GPL은
    파생물도 GPL로 강제). 지금까지 우리 코드는 제가 직접 작성했고 저 저장소를 본 적도 없으니 문제없지만, 앞으로도 저기 코드를 보고 옮겨오는
    건 피해야 합니다 — 순수 아키텍처 개념(RMSNorm 수식 등)을 참고하는 것 자체는 문제없지만(아이디어는 저작권 대상이 아님), 코드 구현체를
    그대로 가져오면 라이선스가 섞입니다.
 
  요약하면: "같은 요리법(LLaMA 스타일)으로 만든 다른 레스토랑" 정도입니다. 걱정하실 만한 건 라이선스뿐이고, 그것도 우리가 독립적으로 작성한
  코드라 문제없습니다.
저작자표시 (새창열림)

'모종의 음모 > ai 프로그램' 카테고리의 다른 글

직접만든 LLM .. part 2  (0) 2026.09.23
wikimatrix  (0) 2026.09.22
llama-tokenize  (0) 2026.09.22
llama-cli /clear  (0) 2026.09.22
llama-simple  (0) 2026.09.22
Posted by 구차니
모종의 음모/ai 프로그램2026. 9. 22. 13:52

llama-tokenize

llama.cpp 에서 프롬프트가 어떻게 토큰으로 나누어 지는지 확인할수 있다.

 

우리말 사전으로 학습시켜서 소+아지 라던가 강+아지로 나올줄 알았는데

토큰을 직접 손으로(!) 건드릴수 있으려나?

 /mnt/Downloads/llama-b10488/llama-tokenize -m base42m-q4_0.gguf -p "마이크로소프트"
0.00.303.842 W load: control-looking token:      7 '<|end|>' was not control-type; this is probably a bug in the model. its type will be overridden
  3169 -> ' 마이크로소프트'

$ /mnt/Downloads/llama-b10488/llama-tokenize -m base42m-q4_0.gguf -p 대한민국
0.00.301.255 W load: control-looking token:      7 '<|end|>' was not control-type; this is probably a bug in the model. its type will be overridden
   579 -> ' 대한민국'
0.00.316.277 W llama_context: n_ctx_seq (512) > n_ctx_train (0) -- possible training context overflow

$ /mnt/Downloads/llama-b10488/llama-tokenize -m base42m-q4_0.gguf -p 강아지
0.00.302.451 W load: control-looking token:      7 '<|end|>' was not control-type; this is probably a bug in the model. its type will be overridden
 26760 -> ' 강아지'

$ /mnt/Downloads/llama-b10488/llama-tokenize -m base42m-q4_0.gguf -p 송아지
0.00.305.743 W load: control-looking token:      7 '<|end|>' was not control-type; this is probably a bug in the model. its type will be overridden
  1259 -> ' 송'
   382 -> '아'
   299 -> '지'

 

혹시나 해서 gemma4 e4b에서 돌려보는데 어라.. 생각외로 그렇게 토큰을 많이 먹진 않는다고 해야하나?

$ /mnt/Downloads/llama-b10488/llama-tokenize -m  gemma-4-E4B-it-Q4_K_M.gguf -p 마이크로소프트
0.00.926.382 W load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.927.107 W load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.937.154 W load: control-looking token:      1 '<eos>' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.969.157 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
0.01.001.080 W llama_context: n_ctx_seq (512) > n_ctx_train (0) -- possible training context overflow
     2 -> '<bos>'
238098 -> '마'
106427 -> '이크'
237323 -> '로'
238049 -> '소'
128329 -> '프트'


$ /mnt/Downloads/llama-b10488/llama-tokenize -m  gemma-4-E4B-it-Q4_K_M.gguf -p 대한민국
0.00.894.966 W load: control-looking token:    212 '' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.895.695 W load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.905.756 W load: control-looking token:      1 '' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.938.322 W load: special_eog_ids contains '<|tool_response>', removing '' token from EOG list
0.00.970.177 W llama_context: n_ctx_seq (512) > n_ctx_train (0) -- possible training context overflow
     2 -> ''
146220 -> '대한'
162139 -> '민국'


$ /mnt/Downloads/llama-b10488/llama-tokenize -m  gemma-4-E4B-it-Q4_K_M.gguf -p 강아지
0.00.915.146 W load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.915.894 W load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.926.166 W load: control-looking token:      1 '<eos>' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.958.649 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
0.00.990.550 W llama_context: n_ctx_seq (512) > n_ctx_train (0) -- possible training context overflow
     2 -> '<bos>'
238865 -> '강'
237534 -> '아'
237308 -> '지'


$ /mnt/Downloads/llama-b10488/llama-tokenize -m  gemma-4-E4B-it-Q4_K_M.gguf -p 송아지
0.00.895.230 W load: control-looking token:    212 '' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.895.969 W load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.906.303 W load: control-looking token:      1 '' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.939.847 W load: special_eog_ids contains '<|tool_response>', removing '' token from EOG list
0.00.972.088 W llama_context: n_ctx_seq (512) > n_ctx_train (0) -- possible training context overflow
     2 -> ''
239917 -> '송'
237534 -> '아'
237308 -> '지'

 

 /mnt/Downloads/llama-b10488/llama-tokenize -m Qwen3.8-27B-UD-IQ2_XXS.gguf -p 마이크로소프트
0.00.773.458 W llama_context: n_ctx_seq (512) > n_ctx_train (0) -- possible training context overflow
148763 -> '마'
156847 -> '이크'
 16869 -> '로'
216243 -> '소프트'

$ /mnt/Downloads/llama-b10488/llama-tokenize -m Qwen3.8-27B-UD-IQ2_XXS.gguf -p 대한민국
0.00.728.852 W llama_context: n_ctx_seq (512) > n_ctx_train (0) -- possible training context overflow
178031 -> '대한'
161266 -> '민국'

$ /mnt/Downloads/llama-b10488/llama-tokenize -m Qwen3.8-27B-UD-IQ2_XXS.gguf -p 강아지
0.00.774.480 W llama_context: n_ctx_seq (512) > n_ctx_train (0) -- possible training context overflow
152029 -> '강'
 51181 -> '아'
 20673 -> '지'

$ /mnt/Downloads/llama-b10488/llama-tokenize -m Qwen3.8-27B-UD-IQ2_XXS.gguf -p 송아지
0.00.769.448 W llama_context: n_ctx_seq (512) > n_ctx_train (0) -- possible training context overflow
150491 -> '송'
 51181 -> '아'
 20673 -> '지'

 

저작자표시 (새창열림)

'모종의 음모 > ai 프로그램' 카테고리의 다른 글

wikimatrix  (0) 2026.09.22
직접 LLM 만들...기?  (0) 2026.09.22
llama-cli /clear  (0) 2026.09.22
llama-simple  (0) 2026.09.22
RoPE - Rotary Position Embedding  (0) 2026.09.21
Posted by 구차니
모종의 음모/ai 프로그램2026. 9. 22. 10:48

llama-cli /clear

오.. 이런 명령어가 있었구나

이전 대화내용을 날려서 context를 정리하는 명령어

 

/mnt/llm-ko/3_small/gguf$ /mnt/Downloads/llama-b10488/llama-cli -m base42m-q4_0.gguf -p "대한민국의 수도는" -ngl 0


Loading model...  

▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀

build      : b10488-9d77fa172
model      : base42m-q4_0.gguf
ftype      : Q4_0
modalities : text

available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read <file>        add a text file
  /glob <pattern>     add text files using globbing pattern



> 대한민국의 수도는
>>>이다.

[ Prompt: 161.5 t/s | Generation: 285.9 t/s ]

> ㄴㅁㅇㄻㄴㅇㄹ
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>)에서 이).

[ Prompt: 195.0 t/s | Generation: 115.0 t/s ]

> 세조대호아잉
>>>>>>>>>>>>>>>>>>>>>>>>>>>)의 순서로는 이례적으로, 그리고 이 작품이 나중의 문학적 삶의 방식에서 영감을 받은 것으로 평가된다.

[ Prompt: 213.6 t/s | Generation: 197.3 t/s ]

> 대한민국
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>)와 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>)이다.

[ Prompt: 257.0 t/s | Generation: 56.8 t/s ]

> 이다
0.16.540.856 E srv    send_error: task id = 330, error: request (513 tokens) exceeds the available context size (512 tokens), try increasing it
Error: request (513 tokens) exceeds the available context size (512 tokens), try increasing it

[ Prompt: 0.0 t/s | Generation: 0.0 t/s ]

> /clear
Chat history cleared.

> 오오
>>>를 통해 “Linkword (rock tit)”의 "아름다운"을 언급하는 것을 고려하면, "아름다운"이라는 용어는 “Linkword”에서 유래한 것이다. 이것은 "Linkword"와 같은 다른 형태의 조합을 포함하는 단어로, 이는 “Linkword”를 의미하는 단어로, 이는 영어로 "Linkword"를 의미하며, 이는 ‘Linkword(newing)’와 같은 축약형에 해당하지 않는다. 이 단어들은 주로 영어 단어에서와 같이 사용되었지만, Linkword가 다른 단어들과 유사하게 자주 인용되는 것은 아니다. Linkword에서의 q-word는 Linkword의 한 구절에 나타난다. Linkword의 다른 구절은 Linkword를 ‘Linkword’에서 q-word와 같은 의미를 가진 단어로, Linkword에서 vii를 ‘Linkword’에서 q-word가 된다. Linkword는 Linkword의 q-word를 의미한다. Linkword의 다른 구절에는 Linkword가 q-word와 같이 Linkword의 q-word의 첫 번째 구절과 두 번째 구절을 공유한다. Linkword의 첫 번째 구절에는 Linkword가 q-word에서의 Linkword의 첫 번째 구절에서 q-word를 나타낸다. Linkword의 첫 번째 구절에는 Linkword의 "Linkword"가 포함되어 있다. Linkword가 q-word를 의미한다. Linkword의 첫 번째 구절은 Linkword가 q-word를 의미한다. Linkword는 Linkword가 q-word를 나타낸다. Linkword는 Linkword와 Linkword가 모두 q-word를 나타낸다. Linkword는 Linkword의 첫 번째 구절에서, Linkword는 Linkword가 q-word를 나타낸다. Linkword의 마지막 구절은 Linkword의 q-word를 나타낸다. Linkword는 Linkword와 Linkword가 q-

[ Prompt: 1393.2 t/s | Generation: 153.5 t/s ]
저작자표시 (새창열림)

'모종의 음모 > ai 프로그램' 카테고리의 다른 글

직접 LLM 만들...기?  (0) 2026.09.22
llama-tokenize  (0) 2026.09.22
llama-simple  (0) 2026.09.22
RoPE - Rotary Position Embedding  (0) 2026.09.21
말뭉치, 코퍼스(corpus)  (0) 2026.09.21
Posted by 구차니
모종의 음모/ai 프로그램2026. 9. 22. 10:12

llama-simple

llama.cpp 에서 단순예제로 프롬프트를 주면 문장일 이어서 만들어 가는

it 쪽이 아닌 base model 검증용... 유틸리티 라고 하면 되려나?

 

[링크 : https://github.com/ggml-org/llama.cpp/blob/master/examples/simple/README.md]

저작자표시 (새창열림)

'모종의 음모 > ai 프로그램' 카테고리의 다른 글

llama-tokenize  (0) 2026.09.22
llama-cli /clear  (0) 2026.09.22
RoPE - Rotary Position Embedding  (0) 2026.09.21
말뭉치, 코퍼스(corpus)  (0) 2026.09.21
토크나이저  (0) 2026.09.21
Posted by 구차니
모종의 음모/ai 프로그램2026. 9. 21. 17:47

RoPE - Rotary Position Embedding

LLM 학습해보려는데 이상한 말이 나와서 -ㅁ-

클로드 님께서 말씀하시길~~~

 5. 어텐션이 모르는 것: 순서
 
 어텐션은 토큰의 위치를 모릅니다. 그래서 RoPE로 Q와 K를 위치에 따라 회전시켜서 토큰 간 거리가 점수에 반영되게 합니다.

 

그래서 자연스럽게 문장이 구성되게 되는건가?

[링크 : https://mari970.tistory.com/49]

[링크 : https://ai-with-sudal-ee.tistory.com/9]

[링크 : https://railly-linker.tistory.com/169]

 

아무튼 셀프 어텐션과, 플래시 어텐션이 비슷한건줄 아랐는데

플래시 어텐션은 경량으로 도는거(파스칼에서 지원안하는)

셀프 어텐션이 입력 받고 출력 나오고 그걸 다시 돌려서 다음 단어를 추측하는 가장 핵심인거

그리고 셀프 어텐션 변경해서 MHA GQA MQA 등이 있나본데 아래의 약자인가 보다.

MHA - Multi-Head Attention

MQA - Multi-Query Attention

GQA - Grouped-Query Attention

저작자표시 (새창열림)

'모종의 음모 > ai 프로그램' 카테고리의 다른 글

llama-cli /clear  (0) 2026.09.22
llama-simple  (0) 2026.09.22
말뭉치, 코퍼스(corpus)  (0) 2026.09.21
토크나이저  (0) 2026.09.21
LLM / 1080 ti 모드 별 벤치마크  (0) 2026.09.17
Posted by 구차니
모종의 음모/ai 프로그램2026. 9. 21. 17:03

말뭉치, 코퍼스(corpus)

아이씨 자꾸 쿠퍼스로 보임

이게 토큰으로 간주해야 하는 건가.. 아닌가.. 다른 분류의 단어인가..

 

자연언어 연구를 위해 특정한 목적을 가지고 언어의 표본을 추출한 집합이다.

[링크 : https://ko.wikipedia.org/wiki/말뭉치]

 

클로드 답변은 아래와 같다.

음.. 아무튼 글이니까 (말뭉치) 그 일부는 토큰으로 이뤄지는게 맞을지도?

  코퍼스가 토큰인가

  아닙니다. 코퍼스는 글이고, 토큰은 그 글을 잘게 나눈 조각입니다.
  
  위키 문서들 = 코퍼스 (글)  →  토크나이저  →  토큰 번호의 나열 = 토큰화된 코퍼스
  train.txt (5.56억 글자)                      train.bin (2.43억 토큰)
  
  - 코퍼스: 책 한 권씩이 쌓인 도서관에 해당합니다.
  - 토큰: 그 책을 낱말 카드로 잘라 놓은 것에 해당합니다. 
  - 크기 표시: 코퍼스의 크기는 글자 수나 토큰 수 둘 다로 말합니다. 우리 코퍼스는 5.56억 글자이고, 이 토크나이저로는 2.43억 토큰입니다. 그래서
    "코퍼스가 2.43억 토큰"이라고 말하는 건 "토큰으로 센 크기"라는 뜻입니다.
  - 학습에 들어가는 형태: 모델은 글을 직접 읽지 못하고 토큰 번호만 받습니다.

 

저작자표시 (새창열림)

'모종의 음모 > ai 프로그램' 카테고리의 다른 글

llama-simple  (0) 2026.09.22
RoPE - Rotary Position Embedding  (0) 2026.09.21
토크나이저  (0) 2026.09.21
LLM / 1080 ti 모드 별 벤치마크  (0) 2026.09.17
v100 흐음...  (0) 2026.09.16
Posted by 구차니
모종의 음모/ai 프로그램2026. 9. 21. 14:50

토크나이저

클로드에서 LLM 의 토크나이저 관련해서 물어보니 뱉어준 대답

Unigram / BPE / trie MaxMatch

 

trie-based search... ? WordPiece?

[링크 : https://www.emergentmind.com/topics/wordpiece-model]

 

Unigram / BPE / WordPiece 은 알고리즘이고 SentencePiece는 툴킷이라고

BPE - Byte Pair Encoding

WordPiece - BERT 계열에서 사용하는 BPE 와 유사한

Unigram - 역방향 접근

[링크 : https://velog.io/@aiengineer/토큰-완전-입문-4.-토큰화-알고리즘-완벽-정리-BPEWordPieceUnigram의-차이와-SentencePiecetiktoken의-정체]

 

+

BERT - Bidirectional Encoder Representations from Transformeres / 구글 2018 / 자연어 처리모델

[링크 : https://namu.wiki/w/BERT]

저작자표시 (새창열림)

'모종의 음모 > ai 프로그램' 카테고리의 다른 글

RoPE - Rotary Position Embedding  (0) 2026.09.21
말뭉치, 코퍼스(corpus)  (0) 2026.09.21
LLM / 1080 ti 모드 별 벤치마크  (0) 2026.09.17
v100 흐음...  (0) 2026.09.16
llama.cpp pascal 지원 빌드 / 성능 테스트  (0) 2026.09.16
Posted by 구차니
모종의 음모/ai 프로그램2026. 9. 17. 10:36

LLM / 1080 ti 모드 별 벤치마크

궁금한 것 을 해소하기 위해 대충 5시간에 걸쳐 GPU 갈굼

  • MTP 가 정말 속도에 영향을 주냐
  • GPU 2개인데 pcie3.0 x8과 x4에 영향을 받는가
  • 컨텍스트 최대 길이가 성능에 영향을 미치는가
  • 입력 토큰길이에 따라 성능 저하는 어느정도 발생하는가
  • 단일과 -sm layer로 나누었을때 성능은 어떠한가

 

결과만 요약하면

  • GPU 0 / 1이 우선권 문제인지 lane 폭 문제인진 모르겠으나 GPU0이 3~10% 정도 이상 성능이 좋게 나옴
  • MTP는 켤수 있으면 켜라. 조건에 따라 다르지만 MTP 1이 1.3~1.5배 향상됨
  • 컨텍스트 최대 길이는 의외로 생성 성능에 영향을 안준다.
  • 한쪽에 가능하지만, 억지로 레이어 쪼개서 양쪽으로 올리면 1.1~1.4배 성능 차이가 난다. 되도록이면 1대로 돌리자

 

아래는 테스트용 스크립트들. 추후 쓰려면 바이너리 경로등을 손봐야 함

bench.zip
0.01MB

 

 

 

아래는 원본 내용

report.md
0.08MB

 

 

llama.cpp Benchmark Report

Short conversation (3-turn, context accumulates)

Questions: see questions.txt. Columns show prompt_t/s / generation_t/s per turn.

gemma4-12b

quantsmgpuctxmtpturn1 (p/g t/s)turn2 (p/g t/s)turn3 (p/g t/s)wall(s)status

gemma-4-12b-it-Q4_0 none 0 131072 off 139.2/34.8 143.4/34.6 270.3/34.4 38.4 OK
gemma-4-12b-it-Q4_0 none 0 131072 on(n=1) 123.9/57.5 141.8/56.1 246.3/51.2 21.5 OK
gemma-4-12b-it-Q4_0 none 0 131072 on(n=2) 123.3/61.5 187.3/61.7 238.3/59.1 18.6 OK
gemma-4-12b-it-Q4_0 none 0 131072 on(n=3) 118.4/65.5 227.8/59.7 231.2/51.6 18.9 OK
gemma-4-12b-it-Q4_0 none 0 131072 on(n=4) 123.2/65.5 181.6/57.1 240.6/45.5 20.0 OK
gemma-4-12b-it-Q4_0 none 0 65536 off 140.0/33.5 146.5/33.2 266.5/32.8 27.0 OK
gemma-4-12b-it-Q4_0 none 0 65536 on(n=1) 118.5/54.5 138.2/55.1 245.5/51.2 20.0 OK
gemma-4-12b-it-Q4_0 none 0 65536 on(n=2) 121.3/59.5 226.9/62.4 229.2/51.9 19.0 OK
gemma-4-12b-it-Q4_0 none 0 65536 on(n=3) 122.9/66.3 243.9/52.7 222.2/52.1 19.0 OK
gemma-4-12b-it-Q4_0 none 0 65536 on(n=4) 118.3/65.2 139.6/60.8 245.8/48.4 19.1 OK
gemma-4-12b-it-Q4_0 none 0 32768 off 136.6/33.6 146.1/33.2 263.9/32.7 27.0 OK
gemma-4-12b-it-Q4_0 none 0 32768 on(n=1) 125.1/57.3 140.9/55.5 242.6/51.8 19.7 OK
gemma-4-12b-it-Q4_0 none 0 32768 on(n=2) 120.7/65.7 139.8/58.8 243.7/57.5 18.3 OK
gemma-4-12b-it-Q4_0 none 0 32768 on(n=3) 125.6/63.6 139.2/60.2 244.5/53.6 18.7 OK
gemma-4-12b-it-Q4_0 none 0 32768 on(n=4) 113.1/60.4 182.1/63.9 242.7/43.8 19.8 OK
gemma-4-12b-it-Q4_0 none 0 16384 off 139.6/33.5 146.0/32.9 267.0/32.7 27.1 OK
gemma-4-12b-it-Q4_0 none 0 16384 on(n=1) 122.8/55.8 213.0/56.2 230.9/53.9 19.6 OK
gemma-4-12b-it-Q4_0 none 0 16384 on(n=2) 124.2/62.4 139.8/60.7 243.0/54.1 18.7 OK
gemma-4-12b-it-Q4_0 none 0 16384 on(n=3) 122.5/60.7 140.1/63.2 244.8/53.5 18.7 OK
gemma-4-12b-it-Q4_0 none 0 16384 on(n=4) 120.8/60.6 141.5/58.4 245.8/50.0 19.4 OK
gemma-4-12b-it-Q4_0 none 1 131072 off 135.6/32.8 249.3/32.7 225.6/32.5 27.0 OK
gemma-4-12b-it-Q4_0 none 1 131072 on(n=1) 106.7/52.1 130.0/51.5 223.8/49.9 22.8 OK
gemma-4-12b-it-Q4_0 none 1 131072 on(n=2) 117.4/59.5 129.0/57.8 224.2/53.2 21.3 OK
gemma-4-12b-it-Q4_0 none 1 131072 on(n=3) 117.7/61.5 155.4/54.7 218.6/52.5 21.5 OK
gemma-4-12b-it-Q4_0 none 1 131072 on(n=4) 112.3/62.0 128.8/56.6 223.6/45.7 22.1 OK
gemma-4-12b-it-Q4_0 none 1 65536 off 130.6/31.5 243.5/30.9 223.0/30.6 29.4 OK
gemma-4-12b-it-Q4_0 none 1 65536 on(n=1) 117.7/50.5 130.1/50.2 225.2/48.2 23.2 OK
gemma-4-12b-it-Q4_0 none 1 65536 on(n=2) 117.1/57.9 128.0/56.5 222.8/49.6 21.9 OK
gemma-4-12b-it-Q4_0 none 1 65536 on(n=3) 117.0/51.0 129.5/54.2 222.5/51.0 22.5 OK
gemma-4-12b-it-Q4_0 none 1 65536 on(n=4) 117.0/61.7 225.9/59.6 200.0/45.0 21.5 OK
gemma-4-12b-it-Q4_0 none 1 32768 off 133.9/31.7 212.7/31.0 227.7/30.7 30.3 OK
gemma-4-12b-it-Q4_0 none 1 32768 on(n=1) 119.1/49.3 130.3/51.8 222.2/48.9 22.9 OK
gemma-4-12b-it-Q4_0 none 1 32768 on(n=2) 119.2/61.1 130.8/57.4 224.5/48.8 21.4 OK
gemma-4-12b-it-Q4_0 none 1 32768 on(n=3) 119.4/58.1 127.6/53.1 216.0/45.0 22.4 OK
gemma-4-12b-it-Q4_0 none 1 32768 on(n=4) 118.7/60.7 130.8/53.0 225.1/45.8 22.2 OK
gemma-4-12b-it-Q4_0 none 1 16384 off 132.8/31.8 137.5/31.0 244.6/30.7 30.1 OK
gemma-4-12b-it-Q4_0 none 1 16384 on(n=1) 107.6/52.2 227.2/51.0 200.4/48.9 21.2 OK
gemma-4-12b-it-Q4_0 none 1 16384 on(n=2) 110.5/57.1 209.0/56.6 204.2/51.1 21.1 OK
gemma-4-12b-it-Q4_0 none 1 16384 on(n=3) 117.0/59.2 129.6/57.1 208.3/46.9 21.8 OK
gemma-4-12b-it-Q4_0 none 1 16384 on(n=4) 117.1/63.0 193.5/54.3 208.2/43.3 22.3 OK
gemma-4-12b-it-Q4_0 layer both 131072 off 138.0/33.4 145.7/33.3 260.4/33.2 28.0 OK
gemma-4-12b-it-Q4_0 layer both 131072 on(n=1) 123.1/49.2 136.9/48.5 238.2/47.3 23.3 OK
gemma-4-12b-it-Q4_0 layer both 131072 on(n=2) 122.3/55.9 239.8/50.8 213.4/45.0 22.8 OK
gemma-4-12b-it-Q4_0 layer both 131072 on(n=3) 122.4/49.1 135.9/48.8 236.3/43.6 23.7 OK
gemma-4-12b-it-Q4_0 layer both 131072 on(n=4) 118.0/53.8 136.3/47.4 237.4/38.9 24.4 OK
gemma-4-12b-it-Q4_0 layer both 65536 off 136.4/33.1 143.0/33.0 255.2/33.0 28.1 OK
gemma-4-12b-it-Q4_0 layer both 65536 on(n=1) 122.1/48.4 225.8/49.4 218.4/47.4 22.5 OK
gemma-4-12b-it-Q4_0 layer both 65536 on(n=2) 122.3/54.4 136.8/51.9 237.7/44.2 22.5 OK
gemma-4-12b-it-Q4_0 layer both 65536 on(n=3) 122.4/45.0 136.5/47.1 237.8/44.8 24.0 OK
gemma-4-12b-it-Q4_0 layer both 65536 on(n=4) 122.4/53.7 136.4/47.1 238.4/42.6 23.3 OK
gemma-4-12b-it-Q4_0 layer both 32768 off 140.6/33.1 145.9/33.1 260.2/33.0 27.9 OK
gemma-4-12b-it-Q4_0 layer both 32768 on(n=1) 122.9/48.3 137.3/49.0 239.4/47.7 22.8 OK
gemma-4-12b-it-Q4_0 layer both 32768 on(n=2) 109.1/52.3 137.7/52.0 239.2/46.8 22.2 OK
gemma-4-12b-it-Q4_0 layer both 32768 on(n=3) 113.3/53.9 138.1/47.0 239.0/45.8 22.7 OK
gemma-4-12b-it-Q4_0 layer both 32768 on(n=4) 124.3/52.7 237.4/48.5 217.2/38.7 23.7 OK
gemma-4-12b-it-Q4_0 layer both 16384 off 135.4/32.8 143.2/32.8 257.2/32.8 28.1 OK
gemma-4-12b-it-Q4_0 layer both 16384 on(n=1) 118.5/50.7 135.7/49.0 237.0/47.7 22.5 OK
gemma-4-12b-it-Q4_0 layer both 16384 on(n=2) 120.0/53.6 216.3/52.3 219.0/46.5 21.9 OK
gemma-4-12b-it-Q4_0 layer both 16384 on(n=3) 119.0/53.8 235.9/49.9 214.2/46.3 21.2 OK
gemma-4-12b-it-Q4_0 layer both 16384 on(n=4) 98.1/51.2 236.1/46.7 214.0/40.8 23.8 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 131072 off 85.4/31.3 93.9/31.1 203.3/30.5 36.9 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 131072 on(n=1) 78.3/42.7 166.5/42.6 181.6/40.2 24.0 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 131072 on(n=2) 78.5/45.1 92.4/43.7 184.5/34.0 24.7 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 131072 on(n=3) 78.6/40.4 91.7/41.9 189.3/31.6 26.2 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 131072 on(n=4) 78.9/42.5 153.1/39.7 183.6/26.9 27.7 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 65536 off 85.2/30.8 92.6/29.7 203.0/29.5 29.2 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 65536 on(n=1) 78.9/45.4 163.3/42.7 181.7/40.6 22.6 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 65536 on(n=2) 78.0/43.3 91.8/42.6 190.4/35.2 24.6 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 65536 on(n=3) 79.2/41.9 131.7/42.2 185.5/32.4 25.5 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 65536 on(n=4) 78.1/42.2 92.5/41.4 191.5/35.2 25.1 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 32768 off 84.8/30.6 93.7/29.6 203.1/29.4 29.4 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 32768 on(n=1) 73.8/44.8 157.0/42.1 182.4/39.7 23.7 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 32768 on(n=2) 70.6/43.6 155.4/46.0 181.4/36.8 23.9 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 32768 on(n=3) 77.4/40.6 93.5/43.4 189.3/29.6 26.3 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 32768 on(n=4) 78.5/43.6 92.2/36.1 188.0/30.9 26.7 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 16384 off 83.1/30.8 96.2/29.6 198.6/29.4 29.3 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 16384 on(n=1) 78.3/44.4 93.2/43.3 191.1/40.6 23.4 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 16384 on(n=2) 76.5/45.0 92.9/44.7 187.7/36.5 23.9 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 16384 on(n=3) 78.8/43.9 92.3/44.5 188.9/33.0 24.8 OK
gemma-4-12b-it-UD-Q2_K_XL none 0 16384 on(n=4) 68.9/45.3 173.1/37.3 181.7/31.7 25.1 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 131072 off 84.3/30.6 93.4/30.5 192.8/30.2 30.1 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 131072 on(n=1) 76.0/43.4 178.7/40.8 164.0/37.0 26.3 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 131072 on(n=2) 76.1/44.7 90.1/42.6 174.1/36.0 26.0 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 131072 on(n=3) 71.8/38.8 177.5/38.2 163.8/37.9 27.1 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 131072 on(n=4) 75.7/38.1 88.6/40.9 178.3/28.7 29.1 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 65536 off 81.6/28.6 87.9/28.2 188.6/28.0 32.1 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 65536 on(n=1) 70.9/43.4 81.7/39.7 176.3/37.0 26.2 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 65536 on(n=2) 75.9/42.5 178.2/38.9 164.5/37.8 25.7 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 65536 on(n=3) 76.2/40.1 123.6/36.9 173.4/29.7 28.8 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 65536 on(n=4) 73.4/43.0 87.9/36.1 170.6/27.2 29.4 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 32768 off 82.1/28.5 185.9/28.1 176.6/28.1 32.1 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 32768 on(n=1) 75.6/42.5 87.4/42.0 175.6/36.3 26.1 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 32768 on(n=2) 76.1/42.0 160.3/44.1 166.9/35.7 24.6 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 32768 on(n=3) 66.4/36.6 88.3/35.5 171.7/30.3 29.6 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 32768 on(n=4) 76.0/46.1 148.8/46.1 162.2/26.7 27.3 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 16384 off 81.2/28.1 91.4/28.0 187.5/28.0 32.3 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 16384 on(n=1) 76.3/42.1 88.9/42.3 174.9/35.7 26.2 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 16384 on(n=2) 76.3/39.3 174.5/41.0 164.8/35.1 26.9 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 16384 on(n=3) 74.7/40.4 123.5/38.3 170.6/31.1 28.2 OK
gemma-4-12b-it-UD-Q2_K_XL none 1 16384 on(n=4) 76.3/40.4 89.3/40.0 178.1/30.9 27.9 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 131072 off 81.7/31.8 92.9/31.7 196.6/31.6 28.8 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 131072 on(n=1) 75.9/42.6 159.0/41.0 174.5/40.2 25.7 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 131072 on(n=2) 73.9/44.2 90.3/38.6 184.9/35.4 26.7 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 131072 on(n=3) 73.9/39.7 90.3/39.6 185.0/29.4 28.6 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 131072 on(n=4) 71.8/35.7 87.9/35.7 182.8/26.0 31.2 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 65536 off 81.6/31.5 90.1/31.2 196.0/31.3 28.8 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 65536 on(n=1) 72.9/43.2 90.6/41.7 184.6/37.1 25.7 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 65536 on(n=2) 75.7/41.7 90.3/41.2 184.0/36.1 26.2 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 65536 on(n=3) 75.7/38.8 90.0/37.7 184.7/30.1 28.6 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 65536 on(n=4) 64.3/37.0 88.7/40.3 184.7/28.0 29.2 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 32768 off 81.7/31.5 129.4/31.3 186.1/30.8 28.9 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 32768 on(n=1) 73.2/42.8 138.4/41.9 175.3/39.9 25.1 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 32768 on(n=2) 75.6/42.0 90.6/38.4 185.3/37.5 26.1 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 32768 on(n=3) 71.1/37.6 167.8/38.1 173.2/27.0 29.6 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 32768 on(n=4) 76.0/34.6 178.5/30.4 174.5/27.6 31.6 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 16384 off 80.9/31.6 91.1/31.3 190.5/31.0 28.8 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 16384 on(n=1) 75.9/41.5 90.1/43.4 184.0/37.2 25.2 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 16384 on(n=2) 73.9/41.2 102.5/40.8 183.7/36.6 25.8 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 16384 on(n=3) 73.8/38.1 90.5/40.6 185.0/31.5 27.4 OK
gemma-4-12b-it-UD-Q2_K_XL layer both 16384 on(n=4) 72.1/34.2 90.2/36.8 184.3/30.5 29.3 OK

gemma4-26b

quantsmgpuctxmtpturn1 (p/g t/s)turn2 (p/g t/s)turn3 (p/g t/s)wall(s)status

gemma-4-26B-A4B-it-UD-IQ2_M none 0 131072 off - - - 21.8 OOM
gemma-4-26B-A4B-it-UD-IQ2_M none 0 65536 off - - - 3.6 OOM
gemma-4-26B-A4B-it-UD-IQ2_M none 0 32768 off 92.5/65.4 175.2/65.2 341.6/64.8 16.1 OK
gemma-4-26B-A4B-it-UD-IQ2_M none 0 16384 off 135.9/64.9 196.1/64.4 338.4/63.2 16.2 OK
gemma-4-26B-A4B-it-UD-IQ2_M none 1 131072 off - - - 6.1 OOM
gemma-4-26B-A4B-it-UD-IQ2_M none 1 65536 off - - - 6.1 OOM
gemma-4-26B-A4B-it-UD-IQ2_M none 1 32768 off 147.6/60.0 177.3/59.9 317.9/59.4 19.5 OK
gemma-4-26B-A4B-it-UD-IQ2_M none 1 16384 off 149.8/59.8 176.1/59.6 314.3/58.2 19.7 OK
gemma-4-26B-A4B-it-UD-IQ2_M layer both 131072 off 148.0/61.8 196.3/61.8 333.4/61.7 18.1 OK
gemma-4-26B-A4B-it-UD-IQ2_M layer both 65536 off 147.7/61.7 196.1/61.6 332.7/61.5 18.1 OK
gemma-4-26B-A4B-it-UD-IQ2_M layer both 32768 off 149.5/61.5 194.9/61.5 330.8/61.4 17.9 OK
gemma-4-26B-A4B-it-UD-IQ2_M layer both 16384 off 149.1/61.5 195.4/61.5 331.4/61.3 17.8 OK

gemma4-31b

quantsmgpuctxmtpturn1 (p/g t/s)turn2 (p/g t/s)turn3 (p/g t/s)wall(s)status

gemma-4-31B-it-UD-IQ2_XXS none 0 131072 off - - - 19.0 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 131072 on(n=1) - - - 4.1 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 131072 on(n=2) - - - 4.1 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 131072 on(n=3) - - - 4.2 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 131072 on(n=4) - - - 4.1 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 65536 off - - - 3.4 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 65536 on(n=1) - - - 4.1 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 65536 on(n=2) - - - 4.1 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 65536 on(n=3) - - - 4.1 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 65536 on(n=4) - - - 4.1 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 32768 off - - - 3.4 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 32768 on(n=1) - - - 4.2 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 32768 on(n=2) - - - 4.2 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 32768 on(n=3) - - - 4.2 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 32768 on(n=4) - - - 4.2 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 16384 off 43.5/15.0 48.2/14.9 96.1/14.2 57.6 OK
gemma-4-31B-it-UD-IQ2_XXS none 0 16384 on(n=1) - - - 6.0 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 16384 on(n=2) - - - 5.1 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 16384 on(n=3) - - - 5.1 OOM
gemma-4-31B-it-UD-IQ2_XXS none 0 16384 on(n=4) - - - 5.1 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 131072 off - - - 5.5 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 131072 on(n=1) - - - 6.3 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 131072 on(n=2) - - - 6.2 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 131072 on(n=3) - - - 6.2 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 131072 on(n=4) - - - 6.2 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 65536 off - - - 5.5 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 65536 on(n=1) - - - 6.3 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 65536 on(n=2) - - - 6.4 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 65536 on(n=3) - - - 6.3 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 65536 on(n=4) - - - 6.3 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 32768 off - - - 5.5 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 32768 on(n=1) - - - 6.3 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 32768 on(n=2) - - - 6.4 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 32768 on(n=3) - - - 6.3 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 32768 on(n=4) - - - 6.4 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 16384 off 43.2/14.8 46.4/14.0 89.6/13.3 62.3 OK
gemma-4-31B-it-UD-IQ2_XXS none 1 16384 on(n=1) - - - 7.2 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 16384 on(n=2) - - - 7.2 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 16384 on(n=3) - - - 7.1 OOM
gemma-4-31B-it-UD-IQ2_XXS none 1 16384 on(n=4) - - - 7.1 OOM
gemma-4-31B-it-UD-IQ2_XXS layer both 131072 off 41.9/15.3 60.1/15.2 94.0/15.1 57.2 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 131072 on(n=1) - - - 6.6 OOM
gemma-4-31B-it-UD-IQ2_XXS layer both 131072 on(n=2) - - - 6.7 OOM
gemma-4-31B-it-UD-IQ2_XXS layer both 131072 on(n=3) - - - 6.6 OOM
gemma-4-31B-it-UD-IQ2_XXS layer both 131072 on(n=4) - - - 6.6 OOM
gemma-4-31B-it-UD-IQ2_XXS layer both 65536 off 42.1/15.2 79.4/15.1 92.2/15.0 57.5 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 65536 on(n=1) 39.3/22.3 67.4/21.9 85.3/21.0 44.1 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 65536 on(n=2) 39.2/23.4 91.9/20.7 81.8/21.2 44.0 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 65536 on(n=3) 37.0/23.9 138.9/16.4 49.6/15.9 52.9 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 65536 on(n=4) 39.2/20.4 45.8/19.0 88.5/18.2 48.7 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 32768 off 41.3/15.1 85.4/15.0 133.1/14.8 57.8 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 32768 on(n=1) 38.7/22.9 92.4/20.9 80.9/20.6 42.9 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 32768 on(n=2) 37.8/22.7 67.6/21.7 85.1/19.9 42.0 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 32768 on(n=3) 39.2/20.7 45.9/19.4 88.9/17.5 48.6 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 32768 on(n=4) 39.2/20.8 45.7/20.3 87.5/17.2 48.2 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 16384 off 41.6/15.1 47.9/14.8 95.1/14.6 58.1 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 16384 on(n=1) 39.3/22.6 66.8/21.9 86.5/20.7 43.8 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 16384 on(n=2) 39.3/24.3 90.5/20.5 81.3/17.9 46.0 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 16384 on(n=3) 39.4/20.3 52.6/19.9 88.4/17.8 48.3 OK
gemma-4-31B-it-UD-IQ2_XXS layer both 16384 on(n=4) 39.3/19.7 145.8/17.8 53.0/15.5 54.2 OK
gemma-4-31B-it-UD-Q2_K_XL none 0 131072 off - - - 24.0 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 131072 on(n=1) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 131072 on(n=2) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 131072 on(n=3) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 131072 on(n=4) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 65536 off - - - 2.9 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 65536 on(n=1) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 65536 on(n=2) - - - 3.7 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 65536 on(n=3) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 65536 on(n=4) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 32768 off - - - 2.9 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 32768 on(n=1) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 32768 on(n=2) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 32768 on(n=3) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 32768 on(n=4) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 16384 off - - - 2.9 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 16384 on(n=1) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 16384 on(n=2) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 16384 on(n=3) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 0 16384 on(n=4) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 131072 off - - - 2.9 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 131072 on(n=1) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 131072 on(n=2) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 131072 on(n=3) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 131072 on(n=4) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 65536 off - - - 2.9 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 65536 on(n=1) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 65536 on(n=2) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 65536 on(n=3) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 65536 on(n=4) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 32768 off - - - 2.9 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 32768 on(n=1) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 32768 on(n=2) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 32768 on(n=3) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 32768 on(n=4) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 16384 off - - - 2.9 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 16384 on(n=1) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 16384 on(n=2) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 16384 on(n=3) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL none 1 16384 on(n=4) - - - 3.5 OOM
gemma-4-31B-it-UD-Q2_K_XL layer both 131072 off - - - 5.5 OOM
gemma-4-31B-it-UD-Q2_K_XL layer both 131072 on(n=1) - - - 6.2 OOM
gemma-4-31B-it-UD-Q2_K_XL layer both 131072 on(n=2) - - - 6.3 OOM
gemma-4-31B-it-UD-Q2_K_XL layer both 131072 on(n=3) - - - 6.3 OOM
gemma-4-31B-it-UD-Q2_K_XL layer both 131072 on(n=4) - - - 6.2 OOM
gemma-4-31B-it-UD-Q2_K_XL layer both 65536 off 35.3/13.8 40.7/13.7 86.1/13.6 63.6 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 65536 on(n=1) 32.7/20.5 40.3/20.7 79.4/20.2 47.5 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 65536 on(n=2) 32.7/22.4 64.2/21.0 76.9/18.7 47.3 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 65536 on(n=3) 32.9/21.9 64.0/20.4 76.9/18.2 46.8 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 65536 on(n=4) 31.0/21.0 39.6/20.1 80.4/17.7 49.4 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 32768 off 34.1/13.6 68.3/13.4 82.1/13.3 60.9 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 32768 on(n=1) 30.9/20.1 48.1/20.7 78.7/19.4 48.0 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 32768 on(n=2) 30.9/20.9 40.1/21.4 79.6/18.7 47.6 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 32768 on(n=3) 31.2/20.9 40.0/20.6 80.0/17.5 49.0 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 32768 on(n=4) 32.6/20.2 64.3/22.5 75.7/18.2 46.0 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 16384 off 34.3/13.6 67.6/13.4 81.0/13.4 64.6 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 16384 on(n=1) 30.8/20.3 63.9/20.1 76.7/19.3 48.4 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 16384 on(n=2) 32.6/20.8 40.0/21.9 79.9/18.8 47.3 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 16384 on(n=3) 32.8/19.9 39.9/17.8 79.8/17.7 51.3 OK
gemma-4-31B-it-UD-Q2_K_XL layer both 16384 on(n=4) 32.8/22.5 40.2/21.1 79.5/16.9 48.3 OK

gemma4-e2b

quantsmgpuctxmtpturn1 (p/g t/s)turn2 (p/g t/s)turn3 (p/g t/s)wall(s)status

gemma-4-E2B-it-Q4_K_M none 0 131072 off 251.7/92.4 268.8/92.4 827.5/92.3 16.6 OK
gemma-4-E2B-it-Q4_K_M none 0 65536 off 177.7/92.2 380.4/92.2 847.8/92.1 11.0 OK
gemma-4-E2B-it-Q4_K_M none 0 32768 off 178.4/91.9 892.3/92.1 837.1/92.0 8.7 OK
gemma-4-E2B-it-Q4_K_M none 0 16384 off 262.1/88.7 430.0/92.2 846.8/92.1 8.3 OK
gemma-4-E2B-it-Q4_K_M none 1 131072 off 243.8/86.7 358.6/86.7 817.0/86.6 12.0 OK
gemma-4-E2B-it-Q4_K_M none 1 65536 off 258.9/85.4 352.0/85.3 803.6/85.4 12.1 OK
gemma-4-E2B-it-Q4_K_M none 1 32768 off 262.6/81.3 408.6/86.0 804.0/85.8 8.9 OK
gemma-4-E2B-it-Q4_K_M none 1 16384 off 263.2/82.2 406.5/85.9 799.4/85.0 8.9 OK
gemma-4-E2B-it-Q4_K_M layer both 131072 off 205.5/65.2 290.7/64.6 660.5/62.9 15.2 OK
gemma-4-E2B-it-Q4_K_M layer both 65536 off 215.5/63.5 340.7/65.2 672.0/63.5 11.0 OK
gemma-4-E2B-it-Q4_K_M layer both 32768 off 163.0/66.0 355.8/67.8 707.0/65.9 10.7 OK
gemma-4-E2B-it-Q4_K_M layer both 16384 off 167.1/62.6 337.5/64.7 664.8/63.0 11.0 OK

gemma4-e4b

quantsmgpuctxmtpturn1 (p/g t/s)turn2 (p/g t/s)turn3 (p/g t/s)wall(s)status

gemma-4-E4B-it-Q4_K_M none 0 131072 off 137.3/61.6 439.9/61.5 510.4/61.2 24.2 OK
gemma-4-E4B-it-Q4_K_M none 0 131072 on(n=1) 162.3/81.3 557.5/87.7 486.8/88.6 11.5 OK
gemma-4-E4B-it-Q4_K_M none 0 131072 on(n=2) 154.5/91.4 550.3/92.6 488.2/88.9 10.9 OK
gemma-4-E4B-it-Q4_K_M none 0 131072 on(n=3) 163.7/89.2 338.7/95.8 496.6/91.6 10.5 OK
gemma-4-E4B-it-Q4_K_M none 0 131072 on(n=4) 167.1/69.7 559.5/89.2 475.0/84.6 11.3 OK
gemma-4-E4B-it-Q4_K_M none 0 65536 off 180.4/60.7 517.8/60.9 497.2/60.0 12.0 OK
gemma-4-E4B-it-Q4_K_M none 0 65536 on(n=1) 163.0/88.7 232.9/92.9 490.9/85.8 13.4 OK
gemma-4-E4B-it-Q4_K_M none 0 65536 on(n=2) 163.9/87.5 548.8/91.6 490.6/87.5 10.9 OK
gemma-4-E4B-it-Q4_K_M none 0 65536 on(n=3) 123.7/66.3 544.5/101.7 484.8/76.2 11.3 OK
gemma-4-E4B-it-Q4_K_M none 0 65536 on(n=4) 161.4/63.4 547.0/86.8 486.2/76.4 11.7 OK
gemma-4-E4B-it-Q4_K_M none 0 32768 off 137.9/60.7 527.3/60.9 503.6/59.6 12.1 OK
gemma-4-E4B-it-Q4_K_M none 0 32768 on(n=1) 163.6/78.8 600.7/93.1 475.7/87.3 11.1 OK
gemma-4-E4B-it-Q4_K_M none 0 32768 on(n=2) 165.2/74.1 399.6/99.9 495.0/90.8 10.4 OK
gemma-4-E4B-it-Q4_K_M none 0 32768 on(n=3) 164.3/74.0 589.4/93.0 469.5/88.0 11.1 OK
gemma-4-E4B-it-Q4_K_M none 0 32768 on(n=4) 125.3/59.3 477.0/85.9 489.8/72.6 11.8 OK
gemma-4-E4B-it-Q4_K_M none 0 16384 off 158.4/60.9 576.5/60.8 495.3/59.6 12.3 OK
gemma-4-E4B-it-Q4_K_M none 0 16384 on(n=1) 155.8/79.5 510.5/86.7 484.5/87.8 11.0 OK
gemma-4-E4B-it-Q4_K_M none 0 16384 on(n=2) 160.8/73.3 565.4/97.5 484.7/86.0 10.9 OK
gemma-4-E4B-it-Q4_K_M none 0 16384 on(n=3) 165.7/86.4 232.9/105.1 492.4/94.8 12.8 OK
gemma-4-E4B-it-Q4_K_M none 0 16384 on(n=4) 166.7/65.1 567.5/91.5 483.8/83.0 11.4 OK
gemma-4-E4B-it-Q4_K_M none 1 131072 off 163.1/57.4 225.8/57.4 492.2/57.1 17.2 OK
gemma-4-E4B-it-Q4_K_M none 1 131072 on(n=1) 155.3/73.1 368.1/87.5 467.6/80.4 11.9 OK
gemma-4-E4B-it-Q4_K_M none 1 131072 on(n=2) 136.0/75.6 476.1/91.2 457.9/83.2 11.9 OK
gemma-4-E4B-it-Q4_K_M none 1 131072 on(n=3) 139.9/76.5 570.5/90.8 441.5/83.2 12.2 OK
gemma-4-E4B-it-Q4_K_M none 1 131072 on(n=4) 143.7/67.1 533.1/78.7 447.0/77.4 12.8 OK
gemma-4-E4B-it-Q4_K_M none 1 65536 off 134.1/56.9 581.6/55.2 444.0/54.5 14.1 OK
gemma-4-E4B-it-Q4_K_M none 1 65536 on(n=1) 133.0/83.1 575.9/81.1 438.6/83.2 12.6 OK
gemma-4-E4B-it-Q4_K_M none 1 65536 on(n=2) 120.5/80.7 530.4/88.3 442.6/81.0 12.2 OK
gemma-4-E4B-it-Q4_K_M none 1 65536 on(n=3) 159.8/63.8 429.5/92.1 445.3/87.8 11.7 OK
gemma-4-E4B-it-Q4_K_M none 1 65536 on(n=4) 157.2/76.2 504.1/77.3 448.0/67.6 13.2 OK
gemma-4-E4B-it-Q4_K_M none 1 32768 off 177.3/57.0 630.6/55.3 454.5/54.9 14.1 OK
gemma-4-E4B-it-Q4_K_M none 1 32768 on(n=1) 151.4/84.0 517.6/82.6 437.5/79.5 12.4 OK
gemma-4-E4B-it-Q4_K_M none 1 32768 on(n=2) 162.7/66.7 562.5/86.3 432.2/75.0 12.5 OK
gemma-4-E4B-it-Q4_K_M none 1 32768 on(n=3) 153.3/66.4 533.8/92.9 442.7/77.1 12.1 OK
gemma-4-E4B-it-Q4_K_M none 1 32768 on(n=4) 158.0/58.9 512.7/93.8 437.6/68.2 12.8 OK
gemma-4-E4B-it-Q4_K_M none 1 16384 off 169.7/55.6 316.9/54.5 443.6/54.5 13.4 OK
gemma-4-E4B-it-Q4_K_M none 1 16384 on(n=1) 160.8/77.1 323.0/83.8 457.0/81.1 11.7 OK
gemma-4-E4B-it-Q4_K_M none 1 16384 on(n=2) 159.0/75.8 554.9/87.0 438.5/80.5 12.3 OK
gemma-4-E4B-it-Q4_K_M none 1 16384 on(n=3) 121.6/87.3 224.7/88.1 475.7/76.6 14.8 OK
gemma-4-E4B-it-Q4_K_M none 1 16384 on(n=4) 159.6/88.0 225.4/91.6 476.4/76.9 14.6 OK
gemma-4-E4B-it-Q4_K_M layer both 131072 off 171.6/57.4 559.6/57.7 494.4/57.6 13.3 OK
gemma-4-E4B-it-Q4_K_M layer both 131072 on(n=1) 150.0/76.6 455.2/80.8 478.5/76.7 12.9 OK
gemma-4-E4B-it-Q4_K_M layer both 131072 on(n=2) 156.5/64.9 533.9/82.7 472.6/81.3 12.9 OK
gemma-4-E4B-it-Q4_K_M layer both 131072 on(n=3) 157.8/78.4 559.7/92.1 473.4/77.8 12.6 OK
gemma-4-E4B-it-Q4_K_M layer both 131072 on(n=4) 152.5/62.7 628.6/87.3 463.4/61.2 14.0 OK
gemma-4-E4B-it-Q4_K_M layer both 65536 off 174.8/57.5 548.8/57.6 492.1/57.5 13.3 OK
gemma-4-E4B-it-Q4_K_M layer both 65536 on(n=1) 154.3/76.4 227.1/83.8 487.5/79.0 15.3 OK
gemma-4-E4B-it-Q4_K_M layer both 65536 on(n=2) 158.9/65.6 572.5/85.0 465.8/80.8 12.6 OK
gemma-4-E4B-it-Q4_K_M layer both 65536 on(n=3) 152.1/61.2 540.7/86.7 474.2/78.9 12.5 OK
gemma-4-E4B-it-Q4_K_M layer both 65536 on(n=4) 162.1/66.4 470.2/75.9 478.2/68.5 13.0 OK
gemma-4-E4B-it-Q4_K_M layer both 32768 off 160.8/57.3 439.7/57.5 494.9/57.4 12.8 OK
gemma-4-E4B-it-Q4_K_M layer both 32768 on(n=1) 154.6/77.8 627.5/80.9 464.0/80.2 12.5 OK
gemma-4-E4B-it-Q4_K_M layer both 32768 on(n=2) 159.5/74.9 567.8/84.8 468.9/77.7 12.6 OK
gemma-4-E4B-it-Q4_K_M layer both 32768 on(n=3) 153.4/84.3 432.2/79.2 477.6/73.6 12.4 OK
gemma-4-E4B-it-Q4_K_M layer both 32768 on(n=4) 159.2/54.2 489.7/70.3 474.7/69.8 13.4 OK
gemma-4-E4B-it-Q4_K_M layer both 16384 off 172.1/57.1 476.7/57.5 495.0/57.4 12.9 OK
gemma-4-E4B-it-Q4_K_M layer both 16384 on(n=1) 161.2/82.0 226.4/85.5 489.0/79.1 14.5 OK
gemma-4-E4B-it-Q4_K_M layer both 16384 on(n=2) 159.3/82.0 226.6/89.0 486.9/80.5 14.3 OK
gemma-4-E4B-it-Q4_K_M layer both 16384 on(n=3) 128.7/73.9 531.0/83.8 474.7/70.0 12.5 OK
gemma-4-E4B-it-Q4_K_M layer both 16384 on(n=4) 127.7/69.2 597.7/89.2 464.9/67.4 12.7 OK

gemma4-e4b-qat

quantsmgpuctxmtpturn1 (p/g t/s)turn2 (p/g t/s)turn3 (p/g t/s)wall(s)status

gemma-4-E4B-it-qat-UD-Q2_K_XL none 0 131072 off 225.8/43.5 338.4/43.3 547.1/43.0 26.3 OK
gemma-4-E4B-it-qat-UD-Q2_K_XL none 0 65536 off 254.4/42.6 205.3/43.4 536.7/42.7 20.6 OK
gemma-4-E4B-it-qat-UD-Q2_K_XL none 0 32768 off 234.9/41.0 290.8/43.4 532.0/43.2 14.8 OK
gemma-4-E4B-it-qat-UD-Q2_K_XL none 0 16384 off 226.2/43.0 361.1/42.8 537.0/42.9 18.1 OK
gemma-4-E4B-it-qat-UD-Q2_K_XL none 1 131072 off 243.3/42.9 244.4/41.9 931.4/41.9 16.0 OK
gemma-4-E4B-it-qat-UD-Q2_K_XL none 1 65536 off 217.4/41.7 211.8/42.1 520.5/42.1 21.6 OK
gemma-4-E4B-it-qat-UD-Q2_K_XL none 1 32768 off 222.5/42.1 339.9/41.7 512.0/41.2 16.1 OK
gemma-4-E4B-it-qat-UD-Q2_K_XL none 1 16384 off 219.6/42.5 339.9/42.1 499.4/42.0 16.0 OK
gemma-4-E4B-it-qat-UD-Q2_K_XL layer both 131072 off 204.2/39.2 319.1/39.3 480.3/39.5 22.9 OK
gemma-4-E4B-it-qat-UD-Q2_K_XL layer both 65536 off 218.9/42.2 286.2/40.6 488.3/38.5 16.4 OK
gemma-4-E4B-it-qat-UD-Q2_K_XL layer both 32768 off 176.8/34.6 284.8/42.1 528.9/42.0 15.6 OK
gemma-4-E4B-it-qat-UD-Q2_K_XL layer both 16384 off 212.6/41.5 389.0/42.0 522.5/41.6 16.0 OK
gemma-4-E4B-it-qat-UD-Q4_K_XL none 0 131072 off 263.9/69.5 420.4/70.1 641.8/70.0 17.8 OK
gemma-4-E4B-it-qat-UD-Q4_K_XL none 0 65536 off 265.9/69.4 686.1/69.3 632.7/69.2 10.9 OK
gemma-4-E4B-it-qat-UD-Q4_K_XL none 0 32768 off 305.0/69.5 510.7/69.7 636.5/69.3 10.4 OK
gemma-4-E4B-it-qat-UD-Q4_K_XL none 0 16384 off 302.9/69.5 314.8/69.2 632.7/68.1 13.8 OK
gemma-4-E4B-it-qat-UD-Q4_K_XL none 1 131072 off 280.7/65.8 298.1/65.7 602.2/65.4 15.2 OK
gemma-4-E4B-it-qat-UD-Q4_K_XL none 1 65536 off 284.6/65.5 642.7/65.5 576.5/65.4 12.0 OK
gemma-4-E4B-it-qat-UD-Q4_K_XL none 1 32768 off 280.9/65.3 652.1/64.6 569.5/63.6 12.2 OK
gemma-4-E4B-it-qat-UD-Q4_K_XL none 1 16384 off 288.0/65.3 690.6/63.7 557.8/63.3 12.4 OK
gemma-4-E4B-it-qat-UD-Q4_K_XL layer both 131072 off 262.2/63.4 658.0/65.6 608.9/65.5 11.9 OK
gemma-4-E4B-it-qat-UD-Q4_K_XL layer both 65536 off 289.4/65.6 305.1/65.6 870.7/65.4 15.1 OK
gemma-4-E4B-it-qat-UD-Q4_K_XL layer both 32768 off 289.7/65.2 619.3/65.2 606.4/65.2 11.5 OK
gemma-4-E4B-it-qat-UD-Q4_K_XL layer both 16384 off 289.4/64.9 430.1/65.4 612.2/65.3 11.1 OK

qwen3.5

quantsmgpuctxmtpturn1 (p/g t/s)turn2 (p/g t/s)turn3 (p/g t/s)wall(s)status

Qwen3.5-0.8B-UD-IQ2_M none 0 131072 off 140.5/175.7 346.4/175.9 690.4/174.6 7.1 OK
Qwen3.5-0.8B-UD-IQ2_M none 0 65536 off 349.3/175.1 1051.4/175.9 1812.4/175.8 3.8 OK
Qwen3.5-0.8B-UD-IQ2_M none 0 32768 off 353.0/175.5 1439.3/175.9 2042.0/175.7 4.1 OK
Qwen3.5-0.8B-UD-IQ2_M none 0 16384 off 343.2/173.7 570.3/175.7 3438.5/175.4 4.9 OK
Qwen3.5-0.8B-UD-IQ2_M none 1 131072 off 293.2/160.5 1865.5/161.2 685.3/161.3 6.3 OK
Qwen3.5-0.8B-UD-IQ2_M none 1 65536 off 296.6/160.8 2372.6/0.0 681.2/160.9 6.6 OK
Qwen3.5-0.8B-UD-IQ2_M none 1 32768 off 267.1/161.8 256.9/162.1 678.0/162.0 6.6 OK
Qwen3.5-0.8B-UD-IQ2_M none 1 16384 off 233.3/159.8 819.6/160.8 684.7/160.8 5.4 OK
Qwen3.5-0.8B-UD-IQ2_M layer both 131072 off 238.6/152.9 660.8/160.3 792.1/163.3 5.6 OK
Qwen3.5-0.8B-UD-IQ2_M layer both 65536 off 276.1/163.9 298.6/163.9 802.6/164.1 6.8 OK
Qwen3.5-0.8B-UD-IQ2_M layer both 32768 off 287.7/162.3 298.6/164.4 3081.2/164.1 6.6 OK
Qwen3.5-0.8B-UD-IQ2_M layer both 16384 off 213.0/163.5 1300.6/164.4 804.1/164.0 5.5 OK
Qwen3.5-0.8B-UD-Q4_K_XL none 0 131072 off 238.6/161.0 359.9/163.9 965.5/163.6 7.6 OK
Qwen3.5-0.8B-UD-Q4_K_XL none 0 65536 off 251.8/162.7 536.5/162.8 951.2/162.7 6.6 OK
Qwen3.5-0.8B-UD-Q4_K_XL none 0 32768 off 142.0/162.8 357.6/162.8 990.8/162.7 6.6 OK
Qwen3.5-0.8B-UD-Q4_K_XL none 0 16384 off 152.1/162.7 358.2/162.8 990.2/162.7 6.6 OK
Qwen3.5-0.8B-UD-Q4_K_XL none 1 131072 off 205.4/150.2 260.6/151.3 678.7/151.1 7.2 OK
Qwen3.5-0.8B-UD-Q4_K_XL none 1 65536 off 208.9/149.9 262.3/149.9 681.2/149.8 7.2 OK
Qwen3.5-0.8B-UD-Q4_K_XL none 1 32768 off 150.8/150.1 513.2/150.1 678.1/150.1 7.2 OK
Qwen3.5-0.8B-UD-Q4_K_XL none 1 16384 off 183.8/149.8 261.0/149.9 675.6/149.9 7.2 OK
Qwen3.5-0.8B-UD-Q4_K_XL layer both 131072 off 166.1/143.6 298.3/146.0 778.9/145.7 7.7 OK
Qwen3.5-0.8B-UD-Q4_K_XL layer both 65536 off 169.0/152.9 301.5/153.5 802.8/153.1 7.2 OK
Qwen3.5-0.8B-UD-Q4_K_XL layer both 32768 off 171.6/152.9 304.3/153.7 805.5/153.2 7.1 OK
Qwen3.5-0.8B-UD-Q4_K_XL layer both 16384 off 164.4/152.0 299.7/152.5 2021.4/152.0 7.0 OK
Qwen3.5-0.8B-UD-Q8_K_XL none 0 131072 off 266.9/134.6 336.2/135.8 2332.9/135.7 9.8 OK
Qwen3.5-0.8B-UD-Q8_K_XL none 0 65536 off 267.2/134.9 337.0/135.1 899.7/134.9 7.7 OK
Qwen3.5-0.8B-UD-Q8_K_XL none 0 32768 off 153.0/135.3 337.3/135.2 897.0/135.1 7.8 OK
Qwen3.5-0.8B-UD-Q8_K_XL none 0 16384 off 278.5/135.0 342.8/135.0 883.3/134.9 7.5 OK
Qwen3.5-0.8B-UD-Q8_K_XL none 1 131072 off 180.9/127.2 248.8/126.9 629.6/126.9 8.4 OK
Qwen3.5-0.8B-UD-Q8_K_XL none 1 65536 off 202.9/125.7 376.4/125.8 628.9/125.7 7.5 OK
Qwen3.5-0.8B-UD-Q8_K_XL none 1 32768 off 222.9/125.2 713.0/125.3 643.7/125.2 7.6 OK
Qwen3.5-0.8B-UD-Q8_K_XL none 1 16384 off 235.2/125.6 249.6/125.6 641.1/125.5 8.4 OK
Qwen3.5-0.8B-UD-Q8_K_XL layer both 131072 off 177.1/120.8 542.3/121.8 726.5/121.8 8.9 OK
Qwen3.5-0.8B-UD-Q8_K_XL layer both 65536 off 163.5/125.7 282.3/124.7 733.4/124.5 8.5 OK
Qwen3.5-0.8B-UD-Q8_K_XL layer both 32768 off 163.1/128.6 287.0/128.7 750.1/128.7 8.3 OK
Qwen3.5-0.8B-UD-Q8_K_XL layer both 16384 off 176.6/128.1 578.8/128.2 752.3/128.0 8.3 OK

qwen3.6_27B

quantsmgpuctxmtpturn1 (p/g t/s)turn2 (p/g t/s)turn3 (p/g t/s)wall(s)status

Qwen3.6-27B-Q4_K_S none 0 131072 off - - - 31.0 OOM
Qwen3.6-27B-Q4_K_S none 0 65536 off - - - 2.7 OOM
Qwen3.6-27B-Q4_K_S none 0 32768 off - - - 2.7 OOM
Qwen3.6-27B-Q4_K_S none 0 16384 off - - - 2.7 OOM
Qwen3.6-27B-Q4_K_S none 1 131072 off - - - 2.7 OOM
Qwen3.6-27B-Q4_K_S none 1 65536 off - - - 2.7 OOM
Qwen3.6-27B-Q4_K_S none 1 32768 off - - - 2.7 OOM
Qwen3.6-27B-Q4_K_S none 1 16384 off - - - 2.7 OOM
Qwen3.6-27B-Q4_K_S layer both 131072 off - - - 6.0 OOM
Qwen3.6-27B-Q4_K_S layer both 65536 off 40.7/15.8 38.0/15.8 82.1/15.7 56.3 OK
Qwen3.6-27B-Q4_K_S layer both 32768 off 43.6/15.8 41.4/15.7 81.7/15.6 56.6 OK
Qwen3.6-27B-Q4_K_S layer both 16384 off 41.8/15.7 37.8/15.7 82.6/15.6 56.6 OK

qwen3.6_35B

quantsmgpuctxmtpturn1 (p/g t/s)turn2 (p/g t/s)turn3 (p/g t/s)wall(s)status

Qwen3.6-35B-A3B-UD-Q2_K_XL none 0 131072 off - - - 24.6 OOM
Qwen3.6-35B-A3B-UD-Q2_K_XL none 0 65536 off - - - 2.3 OOM
Qwen3.6-35B-A3B-UD-Q2_K_XL none 0 32768 off - - - 2.3 OOM
Qwen3.6-35B-A3B-UD-Q2_K_XL none 0 16384 off - - - 2.3 OOM
Qwen3.6-35B-A3B-UD-Q2_K_XL none 1 131072 off - - - 2.3 OOM
Qwen3.6-35B-A3B-UD-Q2_K_XL none 1 65536 off - - - 2.3 OOM
Qwen3.6-35B-A3B-UD-Q2_K_XL none 1 32768 off - - - 2.3 OOM
Qwen3.6-35B-A3B-UD-Q2_K_XL none 1 16384 off - - - 2.3 OOM
Qwen3.6-35B-A3B-UD-Q2_K_XL layer both 131072 off 52.2/56.9 92.9/57.2 202.7/57.1 17.8 OK
Qwen3.6-35B-A3B-UD-Q2_K_XL layer both 65536 off 69.6/56.7 108.3/56.7 207.9/56.5 19.3 OK
Qwen3.6-35B-A3B-UD-Q2_K_XL layer both 32768 off 68.5/56.4 107.3/56.6 208.1/56.4 19.3 OK
Qwen3.6-35B-A3B-UD-Q2_K_XL layer both 16384 off 75.2/56.6 108.1/56.6 208.5/56.4 19.2 OK

qwen3.8_27B

quantsmgpuctxmtpturn1 (p/g t/s)turn2 (p/g t/s)turn3 (p/g t/s)wall(s)status

Qwen3.8-27B-UD-IQ2_XXS none 0 131072 off - - - 19.4 OOM
Qwen3.8-27B-UD-IQ2_XXS none 0 65536 off - - - 3.0 OOM
Qwen3.8-27B-UD-IQ2_XXS none 0 32768 off 98.4/14.2 37.4/14.1 82.0/13.5 42.7 OK
Qwen3.8-27B-UD-IQ2_XXS none 0 16384 off 99.5/13.7 37.4/13.1 81.8/12.9 49.0 OK
Qwen3.8-27B-UD-IQ2_XXS none 1 131072 off - - - 5.0 OOM
Qwen3.8-27B-UD-IQ2_XXS none 1 65536 off - - - 5.0 OOM
Qwen3.8-27B-UD-IQ2_XXS none 1 32768 off 90.0/14.3 31.3/14.1 70.1/13.6 49.0 OK
Qwen3.8-27B-UD-IQ2_XXS none 1 16384 off 89.1/13.3 30.8/12.5 69.6/12.3 52.5 OK
Qwen3.8-27B-UD-IQ2_XXS layer both 131072 off 86.0/15.6 34.7/15.4 76.3/15.2 45.4 OK
Qwen3.8-27B-UD-IQ2_XXS layer both 65536 off 91.9/14.7 34.3/14.7 76.3/14.7 44.3 OK
Qwen3.8-27B-UD-IQ2_XXS layer both 32768 off 86.0/14.9 34.1/14.8 75.2/14.7 45.9 OK
Qwen3.8-27B-UD-IQ2_XXS layer both 16384 off 92.3/15.5 34.2/15.4 75.5/15.3 43.3 OK

Long context (single turn, >10k tokens)

Prompt: see long_prompt.txt (needle-in-haystack recall test).

familyquantsmgpuctxmtpprompt t/sgen t/swall(s)status

gemma4-12b gemma-4-12b-it-Q4_0 none 0 131072 off 415.9 30.7 37.6 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 131072 on(n=1) 405.2 48.8 38.7 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 131072 on(n=2) 403.5 52.4 38.6 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 131072 on(n=3) 402.4 53.1 38.7 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 131072 on(n=4) 401.9 43.0 39.3 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 65536 off 401.0 28.4 38.9 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 65536 on(n=1) 398.3 46.7 39.1 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 65536 on(n=2) 398.7 51.7 38.8 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 65536 on(n=3) 400.7 53.7 38.5 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 65536 on(n=4) 399.5 50.1 38.8 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 32768 off 397.4 29.7 38.9 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 32768 on(n=1) 395.0 47.8 39.3 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 32768 on(n=2) 397.6 50.6 38.9 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 32768 on(n=3) 397.0 48.8 39.0 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 32768 on(n=4) 397.5 43.2 39.3 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 16384 off 396.9 29.4 39.0 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 16384 on(n=1) 395.7 47.6 39.2 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 16384 on(n=2) 397.1 47.7 39.1 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 16384 on(n=3) 399.0 49.2 38.8 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 0 16384 on(n=4) 400.2 48.1 38.8 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 131072 off 412.5 28.7 39.8 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 131072 on(n=1) 393.7 46.3 41.5 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 131072 on(n=2) 386.5 41.0 42.5 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 131072 on(n=3) 387.7 45.0 42.1 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 131072 on(n=4) 389.2 43.3 42.1 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 65536 off 394.8 26.9 41.2 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 65536 on(n=1) 387.5 42.8 42.2 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 65536 on(n=2) 387.1 43.6 42.2 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 65536 on(n=3) 387.8 39.8 42.4 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 65536 on(n=4) 387.5 48.4 41.9 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 32768 off 391.2 27.1 41.5 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 32768 on(n=1) 392.1 42.3 41.7 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 32768 on(n=2) 387.5 48.8 41.6 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 32768 on(n=3) 387.5 47.0 41.7 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 32768 on(n=4) 386.7 45.3 41.9 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 16384 off 394.7 26.8 41.2 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 16384 on(n=1) 389.4 42.5 42.1 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 16384 on(n=2) 388.9 45.8 41.6 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 16384 on(n=3) 386.2 43.2 42.0 OK
gemma4-12b gemma-4-12b-it-Q4_0 none 1 16384 on(n=4) 388.6 48.0 41.5 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 131072 off 726.7 29.9 25.8 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 131072 on(n=1) 605.9 35.3 31.0 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 131072 on(n=2) 603.5 29.9 31.7 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 131072 on(n=3) 599.3 31.4 31.6 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 131072 on(n=4) 602.7 21.1 33.5 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 65536 off 713.9 29.7 26.1 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 65536 on(n=1) 598.1 36.1 30.9 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 65536 on(n=2) 599.1 35.0 31.0 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 65536 on(n=3) 601.8 31.4 31.3 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 65536 on(n=4) 599.8 27.5 31.9 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 32768 off 706.8 29.6 26.0 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 32768 on(n=1) 598.9 35.8 30.6 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 32768 on(n=2) 597.8 31.0 31.2 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 32768 on(n=3) 595.9 29.8 31.5 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 32768 on(n=4) 596.5 31.0 31.3 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 16384 off 708.1 29.5 26.0 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 16384 on(n=1) 597.9 36.2 30.4 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 16384 on(n=2) 593.4 34.6 30.7 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 16384 on(n=3) 596.3 26.8 31.9 OK
gemma4-12b gemma-4-12b-it-Q4_0 layer both 16384 on(n=4) 596.9 25.6 32.1 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 131072 off 352.9 26.0 43.1 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 131072 on(n=1) 353.3 34.6 43.8 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 131072 on(n=2) 351.9 35.4 43.8 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 131072 on(n=3) 352.7 31.9 44.2 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 131072 on(n=4) 352.7 34.5 43.9 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 65536 off 354.9 26.1 42.9 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 65536 on(n=1) 351.5 37.7 43.4 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 65536 on(n=2) 351.3 34.7 43.8 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 65536 on(n=3) 351.3 35.3 43.9 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 65536 on(n=4) 353.2 26.9 44.8 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 32768 off 355.2 25.9 42.9 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 32768 on(n=1) 352.1 40.1 43.2 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 32768 on(n=2) 352.1 37.6 43.4 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 32768 on(n=3) 350.8 30.1 44.4 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 32768 on(n=4) 351.1 26.7 44.9 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 16384 off 355.5 26.5 42.7 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 16384 on(n=1) 350.9 36.3 43.6 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 16384 on(n=2) 352.8 34.4 43.6 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 16384 on(n=3) 352.0 37.2 43.5 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 0 16384 on(n=4) 351.5 27.8 44.7 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 131072 off 366.7 25.5 43.1 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 131072 on(n=1) 355.2 33.6 45.2 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 131072 on(n=2) 356.2 33.5 45.1 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 131072 on(n=3) 351.6 29.5 46.1 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 131072 on(n=4) 353.7 26.5 46.3 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 65536 off 359.3 24.6 43.9 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 65536 on(n=1) 355.1 31.8 45.1 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 65536 on(n=2) 352.1 35.9 45.0 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 65536 on(n=3) 352.6 32.0 45.3 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 65536 on(n=4) 354.5 31.7 45.2 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 32768 off 357.9 24.7 44.1 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 32768 on(n=1) 353.7 33.6 45.0 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 32768 on(n=2) 354.3 32.9 45.1 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 32768 on(n=3) 349.9 31.1 45.8 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 32768 on(n=4) 353.1 28.7 45.8 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 16384 off 359.8 24.6 43.9 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 16384 on(n=1) 354.0 36.5 44.7 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 16384 on(n=2) 351.5 31.0 45.6 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 16384 on(n=3) 351.9 30.2 45.6 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL none 1 16384 on(n=4) 352.0 34.7 45.1 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 131072 off 639.1 27.8 27.9 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 131072 on(n=1) 531.0 29.5 34.0 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 131072 on(n=2) 525.5 25.6 34.9 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 131072 on(n=3) 525.3 23.5 35.3 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 131072 on(n=4) 522.1 19.6 36.5 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 65536 off 618.7 27.6 28.5 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 65536 on(n=1) 528.3 31.8 33.5 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 65536 on(n=2) 528.9 26.3 34.3 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 65536 on(n=3) 525.8 22.5 35.3 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 65536 on(n=4) 526.0 20.8 35.7 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 32768 off 614.3 27.9 28.4 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 32768 on(n=1) 525.6 31.0 33.5 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 32768 on(n=2) 524.3 32.0 33.4 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 32768 on(n=3) 524.7 23.9 34.8 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 32768 on(n=4) 525.1 22.1 35.1 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 16384 off 612.1 27.8 28.5 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 16384 on(n=1) 524.9 31.9 33.2 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 16384 on(n=2) 525.0 25.9 34.1 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 16384 on(n=3) 528.6 22.2 34.7 OK
gemma4-12b gemma-4-12b-it-UD-Q2_K_XL layer both 16384 on(n=4) 523.7 27.2 34.1 OK
gemma4-26b gemma-4-26B-A4B-it-UD-IQ2_M none 0 131072 off - - 3.7 OOM
gemma4-26b gemma-4-26B-A4B-it-UD-IQ2_M none 0 65536 off - - 3.6 OOM
gemma4-26b gemma-4-26B-A4B-it-UD-IQ2_M none 0 32768 off 727.3 57.4 23.1 OK
gemma4-26b gemma-4-26B-A4B-it-UD-IQ2_M none 0 16384 off 709.7 51.5 23.8 OK
gemma4-26b gemma-4-26B-A4B-it-UD-IQ2_M none 1 131072 off - - 6.1 OOM
gemma4-26b gemma-4-26B-A4B-it-UD-IQ2_M none 1 65536 off - - 6.1 OOM
gemma4-26b gemma-4-26B-A4B-it-UD-IQ2_M none 1 32768 off 724.9 52.5 26.0 OK
gemma4-26b gemma-4-26B-A4B-it-UD-IQ2_M none 1 16384 off 708.0 48.5 26.4 OK
gemma4-26b gemma-4-26B-A4B-it-UD-IQ2_M layer both 131072 off 1293.5 54.7 17.2 OK
gemma4-26b gemma-4-26B-A4B-it-UD-IQ2_M layer both 65536 off 1293.4 54.6 17.1 OK
gemma4-26b gemma-4-26B-A4B-it-UD-IQ2_M layer both 32768 off 1282.3 54.8 17.0 OK
gemma4-26b gemma-4-26B-A4B-it-UD-IQ2_M layer both 16384 off 1282.9 54.8 16.9 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 131072 off - - 3.4 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 131072 on(n=1) - - 4.1 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 131072 on(n=2) - - 4.1 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 131072 on(n=3) - - 4.1 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 131072 on(n=4) - - 4.1 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 65536 off - - 3.4 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 65536 on(n=1) - - 4.1 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 65536 on(n=2) - - 4.1 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 65536 on(n=3) - - 4.1 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 65536 on(n=4) - - 4.1 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 32768 off - - 3.4 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 32768 on(n=1) - - 4.2 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 32768 on(n=2) - - 4.2 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 32768 on(n=3) - - 4.2 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 32768 on(n=4) - - 4.2 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 16384 off - - 4.6 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 16384 on(n=1) - - 5.0 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 16384 on(n=2) - - 5.1 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 16384 on(n=3) - - 5.1 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 0 16384 on(n=4) - - 5.0 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 131072 off - - 5.5 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 131072 on(n=1) - - 6.2 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 131072 on(n=2) - - 6.2 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 131072 on(n=3) - - 6.3 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 131072 on(n=4) - - 6.3 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 65536 off - - 5.5 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 65536 on(n=1) - - 6.4 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 65536 on(n=2) - - 6.3 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 65536 on(n=3) - - 6.3 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 65536 on(n=4) - - 6.2 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 32768 off - - 5.5 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 32768 on(n=1) - - 6.3 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 32768 on(n=2) - - 6.3 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 32768 on(n=3) - - 6.3 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 32768 on(n=4) - - 6.3 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 16384 off 149.8 12.5 99.1 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 16384 on(n=1) - - 7.2 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 16384 on(n=2) - - 7.2 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 16384 on(n=3) - - 7.1 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS none 1 16384 on(n=4) - - 7.2 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 131072 off 262.7 13.3 62.0 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 131072 on(n=1) - - 6.6 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 131072 on(n=2) - - 6.6 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 131072 on(n=3) - - 6.6 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 131072 on(n=4) - - 6.5 OOM
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 65536 off 258.9 13.2 62.8 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 65536 on(n=1) 220.2 16.0 71.5 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 65536 on(n=2) 220.0 15.3 71.9 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 65536 on(n=3) 220.0 13.2 73.2 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 65536 on(n=4) 220.7 13.7 72.7 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 32768 off 256.0 13.3 63.0 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 32768 on(n=1) 219.9 16.0 71.3 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 32768 on(n=2) 219.9 13.8 72.6 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 32768 on(n=3) 219.5 14.4 72.3 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 32768 on(n=4) 220.0 13.2 73.0 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 16384 off 254.6 13.4 63.2 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 16384 on(n=1) 218.9 16.2 71.5 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 16384 on(n=2) 219.6 15.9 71.4 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 16384 on(n=3) 220.2 14.1 72.3 OK
gemma4-31b gemma-4-31B-it-UD-IQ2_XXS layer both 16384 on(n=4) 219.7 10.7 75.3 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 131072 off - - 2.9 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 131072 on(n=1) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 131072 on(n=2) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 131072 on(n=3) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 131072 on(n=4) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 65536 off - - 2.9 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 65536 on(n=1) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 65536 on(n=2) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 65536 on(n=3) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 65536 on(n=4) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 32768 off - - 2.9 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 32768 on(n=1) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 32768 on(n=2) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 32768 on(n=3) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 32768 on(n=4) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 16384 off - - 2.9 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 16384 on(n=1) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 16384 on(n=2) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 16384 on(n=3) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 0 16384 on(n=4) - - 3.7 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 131072 off - - 2.9 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 131072 on(n=1) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 131072 on(n=2) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 131072 on(n=3) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 131072 on(n=4) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 65536 off - - 2.9 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 65536 on(n=1) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 65536 on(n=2) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 65536 on(n=3) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 65536 on(n=4) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 32768 off - - 2.9 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 32768 on(n=1) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 32768 on(n=2) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 32768 on(n=3) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 32768 on(n=4) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 16384 off - - 2.9 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 16384 on(n=1) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 16384 on(n=2) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 16384 on(n=3) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL none 1 16384 on(n=4) - - 3.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 131072 off - - 5.5 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 131072 on(n=1) - - 6.3 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 131072 on(n=2) - - 6.3 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 131072 on(n=3) - - 6.3 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 131072 on(n=4) - - 6.3 OOM
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 65536 off 245.3 12.3 67.1 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 65536 on(n=1) 207.3 15.1 76.6 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 65536 on(n=2) 207.2 13.8 77.3 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 65536 on(n=3) 207.3 14.5 76.8 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 65536 on(n=4) 207.0 14.0 77.2 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 32768 off 241.0 12.2 67.9 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 32768 on(n=1) 206.6 14.9 76.5 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 32768 on(n=2) 206.7 14.9 76.6 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 32768 on(n=3) 207.0 14.3 76.8 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 32768 on(n=4) 207.1 11.2 79.3 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 16384 off 241.0 12.3 67.8 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 16384 on(n=1) 207.0 15.6 76.1 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 16384 on(n=2) 207.1 14.3 76.8 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 16384 on(n=3) 207.7 14.4 76.5 OK
gemma4-31b gemma-4-31B-it-UD-Q2_K_XL layer both 16384 on(n=4) 207.5 10.4 80.0 OK
gemma4-e2b gemma-4-E2B-it-Q4_K_M none 0 131072 off 1424.6 85.0 12.8 OK
gemma4-e2b gemma-4-E2B-it-Q4_K_M none 0 65536 off 1419.1 85.0 12.8 OK
gemma4-e2b gemma-4-E2B-it-Q4_K_M none 0 32768 off 1415.2 82.6 12.9 OK
gemma4-e2b gemma-4-E2B-it-Q4_K_M none 0 16384 off 1409.1 84.8 12.8 OK
gemma4-e2b gemma-4-E2B-it-Q4_K_M none 1 131072 off 1420.8 79.4 13.3 OK
gemma4-e2b gemma-4-E2B-it-Q4_K_M none 1 65536 off 1395.5 78.1 13.5 OK
gemma4-e2b gemma-4-E2B-it-Q4_K_M none 1 32768 off 1393.3 74.5 13.5 OK
gemma4-e2b gemma-4-E2B-it-Q4_K_M none 1 16384 off 1334.5 76.5 13.9 OK
gemma4-e2b gemma-4-E2B-it-Q4_K_M layer both 131072 off 2339.7 18.3 15.3 OK
gemma4-e2b gemma-4-E2B-it-Q4_K_M layer both 65536 off 2333.8 18.5 15.0 OK
gemma4-e2b gemma-4-E2B-it-Q4_K_M layer both 32768 off 2351.6 18.3 15.0 OK
gemma4-e2b gemma-4-E2B-it-Q4_K_M layer both 16384 off 2348.7 18.5 14.9 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 131072 off 958.7 56.4 18.2 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 131072 on(n=1) 948.6 77.3 17.9 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 131072 on(n=2) 948.3 75.5 17.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 131072 on(n=3) 948.5 86.9 19.1 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 131072 on(n=4) 947.0 82.5 19.2 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 65536 off 942.2 55.0 16.1 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 65536 on(n=1) 942.1 72.0 19.4 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 65536 on(n=2) 945.1 76.4 17.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 65536 on(n=3) 946.9 80.1 19.2 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 65536 on(n=4) 945.3 80.6 17.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 32768 off 937.2 54.0 18.4 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 32768 on(n=1) 943.9 76.4 17.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 32768 on(n=2) 946.2 76.4 19.3 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 32768 on(n=3) 946.8 97.9 17.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 32768 on(n=4) 945.9 80.9 17.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 16384 off 931.8 55.0 18.4 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 16384 on(n=1) 943.3 75.0 17.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 16384 on(n=2) 946.7 75.9 17.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 16384 on(n=3) 944.4 95.5 17.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 0 16384 on(n=4) 947.5 85.0 17.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 131072 off 964.7 51.7 16.6 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 131072 on(n=1) 945.2 72.0 18.6 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 131072 on(n=2) 930.7 68.0 18.8 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 131072 on(n=3) 924.8 87.6 18.8 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 131072 on(n=4) 916.5 77.7 19.0 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 65536 off 921.0 47.8 19.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 65536 on(n=1) 923.9 67.8 20.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 65536 on(n=2) 937.7 84.9 20.0 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 65536 on(n=3) 918.3 67.3 20.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 65536 on(n=4) 911.5 65.6 20.9 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 32768 off 925.4 46.7 17.2 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 32768 on(n=1) 929.2 67.6 18.8 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 32768 on(n=2) 914.2 71.3 19.0 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 32768 on(n=3) 931.0 92.9 18.5 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 32768 on(n=4) 921.1 76.4 20.4 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 16384 off 922.1 47.6 19.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 16384 on(n=1) 918.5 68.7 20.4 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 16384 on(n=2) 921.6 66.1 20.5 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 16384 on(n=3) 924.5 85.4 18.6 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M none 1 16384 on(n=4) 901.6 64.5 21.0 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 131072 off 1697.8 53.3 13.2 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 131072 on(n=1) 1432.4 66.3 14.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 131072 on(n=2) 1429.3 65.2 16.3 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 131072 on(n=3) 1430.5 54.5 16.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 131072 on(n=4) 1430.8 64.6 16.3 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 65536 off 1695.8 52.5 10.9 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 65536 on(n=1) 1428.5 65.9 16.0 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 65536 on(n=2) 1426.1 65.4 16.0 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 65536 on(n=3) 1426.6 78.4 14.0 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 65536 on(n=4) 1428.6 65.6 14.2 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 32768 off 1692.2 53.2 12.9 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 32768 on(n=1) 1424.0 67.9 14.0 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 32768 on(n=2) 1423.1 64.9 15.8 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 32768 on(n=3) 1425.7 60.2 15.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 32768 on(n=4) 1427.5 65.6 13.9 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 16384 off 1650.6 51.8 10.9 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 16384 on(n=1) 1423.1 68.0 13.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 16384 on(n=2) 1423.0 62.9 13.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 16384 on(n=3) 1423.9 78.7 13.7 OK
gemma4-e4b gemma-4-E4B-it-Q4_K_M layer both 16384 on(n=4) 1420.6 62.1 15.7 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q2_K_XL none 0 131072 off 1024.0 40.8 17.8 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q2_K_XL none 0 65536 off 1022.1 40.7 17.8 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q2_K_XL none 0 32768 off 1022.5 40.7 17.8 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q2_K_XL none 0 16384 off 1019.5 40.8 17.8 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q2_K_XL none 1 131072 off 1024.6 39.9 18.5 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q2_K_XL none 1 65536 off 1018.0 38.6 18.7 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q2_K_XL none 1 32768 off 1014.1 37.7 18.8 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q2_K_XL none 1 16384 off 1003.5 37.7 18.9 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q2_K_XL layer both 131072 off 1037.1 39.9 18.2 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q2_K_XL layer both 65536 off 1033.6 39.6 18.2 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q2_K_XL layer both 32768 off 1032.2 38.6 18.3 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q2_K_XL layer both 16384 off 1031.6 39.5 18.0 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q4_K_XL none 0 131072 off 1051.2 63.5 16.7 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q4_K_XL none 0 65536 off 1041.5 60.0 14.6 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q4_K_XL none 0 32768 off 1025.4 58.2 16.8 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q4_K_XL none 0 16384 off 1034.2 58.7 16.7 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q4_K_XL none 1 131072 off 1046.0 59.7 17.4 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q4_K_XL none 1 65536 off 1026.5 57.1 17.7 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q4_K_XL none 1 32768 off 1022.3 55.6 17.8 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q4_K_XL none 1 16384 off 1017.2 58.1 15.7 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q4_K_XL layer both 131072 off 1850.1 59.4 10.1 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q4_K_XL layer both 65536 off 1855.4 59.9 12.0 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q4_K_XL layer both 32768 off 1848.5 59.3 9.8 OK
gemma4-e4b-qat gemma-4-E4B-it-qat-UD-Q4_K_XL layer both 16384 off 1849.1 59.9 11.8 OK
qwen3.5 Qwen3.5-0.8B-UD-IQ2_M none 0 131072 off 4451.0 161.5 5.0 OK
qwen3.5 Qwen3.5-0.8B-UD-IQ2_M none 0 65536 off 4453.3 155.3 4.3 OK
qwen3.5 Qwen3.5-0.8B-UD-IQ2_M none 0 32768 off 4448.8 161.6 5.0 OK
qwen3.5 Qwen3.5-0.8B-UD-IQ2_M none 0 16384 off 4449.4 161.7 5.0 OK
qwen3.5 Qwen3.5-0.8B-UD-IQ2_M none 1 131072 off 4420.6 148.2 5.3 OK
qwen3.5 Qwen3.5-0.8B-UD-IQ2_M none 1 65536 off 4407.3 149.0 5.1 OK
qwen3.5 Qwen3.5-0.8B-UD-IQ2_M none 1 32768 off 4393.5 149.1 5.1 OK
qwen3.5 Qwen3.5-0.8B-UD-IQ2_M none 1 16384 off 4388.0 147.9 5.1 OK
qwen3.5 Qwen3.5-0.8B-UD-IQ2_M layer both 131072 off 7569.6 151.9 4.3 OK
qwen3.5 Qwen3.5-0.8B-UD-IQ2_M layer both 65536 off 7531.3 150.5 4.3 OK
qwen3.5 Qwen3.5-0.8B-UD-IQ2_M layer both 32768 off 7533.3 151.2 4.0 OK
qwen3.5 Qwen3.5-0.8B-UD-IQ2_M layer both 16384 off 7511.7 150.3 4.0 OK
qwen3.5 Qwen3.5-0.8B-UD-Q4_K_XL none 0 131072 off 4460.4 151.0 5.1 OK
qwen3.5 Qwen3.5-0.8B-UD-Q4_K_XL none 0 65536 off 4436.3 150.7 5.1 OK
qwen3.5 Qwen3.5-0.8B-UD-Q4_K_XL none 0 32768 off 4422.0 150.1 5.1 OK
qwen3.5 Qwen3.5-0.8B-UD-Q4_K_XL none 0 16384 off 4415.1 150.3 5.1 OK
qwen3.5 Qwen3.5-0.8B-UD-Q4_K_XL none 1 131072 off 4387.4 139.5 5.4 OK
qwen3.5 Qwen3.5-0.8B-UD-Q4_K_XL none 1 65536 off 4375.3 139.4 5.4 OK
qwen3.5 Qwen3.5-0.8B-UD-Q4_K_XL none 1 32768 off 4353.9 139.5 5.4 OK
qwen3.5 Qwen3.5-0.8B-UD-Q4_K_XL none 1 16384 off 4349.8 139.3 5.4 OK
qwen3.5 Qwen3.5-0.8B-UD-Q4_K_XL layer both 131072 off 7328.9 142.1 4.4 OK
qwen3.5 Qwen3.5-0.8B-UD-Q4_K_XL layer both 65536 off 7368.6 142.4 4.4 OK
qwen3.5 Qwen3.5-0.8B-UD-Q4_K_XL layer both 32768 off 7343.5 141.4 4.3 OK
qwen3.5 Qwen3.5-0.8B-UD-Q4_K_XL layer both 16384 off 7363.9 140.2 4.3 OK
qwen3.5 Qwen3.5-0.8B-UD-Q8_K_XL none 0 131072 off 4128.0 126.5 5.7 OK
qwen3.5 Qwen3.5-0.8B-UD-Q8_K_XL none 0 65536 off 4109.7 126.4 5.7 OK
qwen3.5 Qwen3.5-0.8B-UD-Q8_K_XL none 0 32768 off 4084.1 126.3 5.5 OK
qwen3.5 Qwen3.5-0.8B-UD-Q8_K_XL none 0 16384 off 4090.5 125.8 5.7 OK
qwen3.5 Qwen3.5-0.8B-UD-Q8_K_XL none 1 131072 off 4092.0 118.4 5.9 OK
qwen3.5 Qwen3.5-0.8B-UD-Q8_K_XL none 1 65536 off 4062.2 118.1 6.0 OK
qwen3.5 Qwen3.5-0.8B-UD-Q8_K_XL none 1 32768 off 4054.0 117.9 6.0 OK
qwen3.5 Qwen3.5-0.8B-UD-Q8_K_XL none 1 16384 off 4026.7 113.0 6.0 OK
qwen3.5 Qwen3.5-0.8B-UD-Q8_K_XL layer both 131072 off 6803.8 120.2 5.1 OK
qwen3.5 Qwen3.5-0.8B-UD-Q8_K_XL layer both 65536 off 6906.1 120.4 4.8 OK
qwen3.5 Qwen3.5-0.8B-UD-Q8_K_XL layer both 32768 off 6870.5 120.5 4.8 OK
qwen3.5 Qwen3.5-0.8B-UD-Q8_K_XL layer both 16384 off 6862.7 119.3 4.8 OK
qwen3.6_27B Qwen3.6-27B-Q4_K_S none 0 131072 off - - 2.7 OOM
qwen3.6_27B Qwen3.6-27B-Q4_K_S none 0 65536 off - - 2.7 OOM
qwen3.6_27B Qwen3.6-27B-Q4_K_S none 0 32768 off - - 2.7 OOM
qwen3.6_27B Qwen3.6-27B-Q4_K_S none 0 16384 off - - 2.5 OOM
qwen3.6_27B Qwen3.6-27B-Q4_K_S none 1 131072 off - - 2.5 OOM
qwen3.6_27B Qwen3.6-27B-Q4_K_S none 1 65536 off - - 2.7 OOM
qwen3.6_27B Qwen3.6-27B-Q4_K_S none 1 32768 off - - 2.5 OOM
qwen3.6_27B Qwen3.6-27B-Q4_K_S none 1 16384 off - - 2.7 OOM
qwen3.6_27B Qwen3.6-27B-Q4_K_S layer both 131072 off - - 5.9 OOM
qwen3.6_27B Qwen3.6-27B-Q4_K_S layer both 65536 off 397.0 14.7 44.1 OK
qwen3.6_27B Qwen3.6-27B-Q4_K_S layer both 32768 off 386.4 14.6 44.9 OK
qwen3.6_27B Qwen3.6-27B-Q4_K_S layer both 16384 off 386.7 14.7 44.8 OK
qwen3.6_35B Qwen3.6-35B-A3B-UD-Q2_K_XL none 0 131072 off - - 2.4 OOM
qwen3.6_35B Qwen3.6-35B-A3B-UD-Q2_K_XL none 0 65536 off - - 2.3 OOM
qwen3.6_35B Qwen3.6-35B-A3B-UD-Q2_K_XL none 0 32768 off - - 2.3 OOM
qwen3.6_35B Qwen3.6-35B-A3B-UD-Q2_K_XL none 0 16384 off - - 2.3 OOM
qwen3.6_35B Qwen3.6-35B-A3B-UD-Q2_K_XL none 1 131072 off - - 2.3 OOM
qwen3.6_35B Qwen3.6-35B-A3B-UD-Q2_K_XL none 1 65536 off - - 2.3 OOM
qwen3.6_35B Qwen3.6-35B-A3B-UD-Q2_K_XL none 1 32768 off - - 2.3 OOM
qwen3.6_35B Qwen3.6-35B-A3B-UD-Q2_K_XL none 1 16384 off - - 2.3 OOM
qwen3.6_35B Qwen3.6-35B-A3B-UD-Q2_K_XL layer both 131072 off 1415.3 53.2 15.9 OK
qwen3.6_35B Qwen3.6-35B-A3B-UD-Q2_K_XL layer both 65536 off 1406.3 53.1 15.7 OK
qwen3.6_35B Qwen3.6-35B-A3B-UD-Q2_K_XL layer both 32768 off 1388.6 51.4 13.8 OK
qwen3.6_35B Qwen3.6-35B-A3B-UD-Q2_K_XL layer both 16384 off 1401.5 53.1 15.7 OK
qwen3.8_27B Qwen3.8-27B-UD-IQ2_XXS none 0 131072 off - - 3.0 OOM
qwen3.8_27B Qwen3.8-27B-UD-IQ2_XXS none 0 65536 off - - 2.9 OOM
qwen3.8_27B Qwen3.8-27B-UD-IQ2_XXS none 0 32768 off 207.2 11.9 64.0 OK
qwen3.8_27B Qwen3.8-27B-UD-IQ2_XXS none 0 16384 off 207.1 12.3 69.3 OK
qwen3.8_27B Qwen3.8-27B-UD-IQ2_XXS none 1 131072 off - - 5.0 OOM
qwen3.8_27B Qwen3.8-27B-UD-IQ2_XXS none 1 65536 off - - 5.0 OOM
qwen3.8_27B Qwen3.8-27B-UD-IQ2_XXS none 1 32768 off 206.2 11.6 68.0 OK
qwen3.8_27B Qwen3.8-27B-UD-IQ2_XXS none 1 16384 off 204.5 11.4 73.0 OK
qwen3.8_27B Qwen3.8-27B-UD-IQ2_XXS layer both 131072 off 379.1 14.1 44.1 OK
qwen3.8_27B Qwen3.8-27B-UD-IQ2_XXS layer both 65536 off 371.6 13.3 40.7 OK
qwen3.8_27B Qwen3.8-27B-UD-IQ2_XXS layer both 32768 off 367.4 13.7 45.0 OK
qwen3.8_27B Qwen3.8-27B-UD-IQ2_XXS layer both 16384 off 366.7 13.9 41.2 OK
저작자표시 (새창열림)

'모종의 음모 > ai 프로그램' 카테고리의 다른 글

말뭉치, 코퍼스(corpus)  (0) 2026.09.21
토크나이저  (0) 2026.09.21
v100 흐음...  (0) 2026.09.16
llama.cpp pascal 지원 빌드 / 성능 테스트  (0) 2026.09.16
1080 ti / pcie3.0 설정후 재 테스트  (0) 2026.09.15
Posted by 구차니
모종의 음모/ai 프로그램2026. 9. 16. 17:08

v100 흐음...

v100 16GB * 4 = 64GB

돈도 돈이지만 전기도 전기라

이 걸 업어오냐 마냐가 고민

 

112token/s 나오는거 보면 끌리긴 한단 말이지.. (1080 ti의 2~3배 사이)

[링크 : https://willitrunai.com/zh/can-run/gemma-4-e4b-on-v100-32gb]

저작자표시 (새창열림)

'모종의 음모 > ai 프로그램' 카테고리의 다른 글

토크나이저  (0) 2026.09.21
LLM / 1080 ti 모드 별 벤치마크  (0) 2026.09.17
llama.cpp pascal 지원 빌드 / 성능 테스트  (0) 2026.09.16
1080 ti / pcie3.0 설정후 재 테스트  (0) 2026.09.15
auroc - area under the ROC curve  (0) 2026.09.01
Posted by 구차니

티스토리툴바