llama-tokenize

구차니 2026. 9. 22. 13:52

llama.cpp 에서 프롬프트가 어떻게 토큰으로 나누어 지는지 확인할수 있다.

우리말 사전으로 학습시켜서 소+아지 라던가 강+아지로 나올줄 알았는데

토큰을 직접 손으로(!) 건드릴수 있으려나?

/mnt/Downloads/llama-b10488/llama-tokenize -m base42m-q4_0.gguf -p "마이크로소프트"
0.00.303.842 W load: control-looking token:      7 '<|end|>' was not control-type; this is probably a bug in the model. its type will be overridden
  3169 -> ' 마이크로소프트'

$ /mnt/Downloads/llama-b10488/llama-tokenize -m base42m-q4_0.gguf -p 대한민국
0.00.301.255 W load: control-looking token:      7 '<|end|>' was not control-type; this is probably a bug in the model. its type will be overridden
   579 -> ' 대한민국'
0.00.316.277 W llama_context: n_ctx_seq (512) > n_ctx_train (0) -- possible training context overflow

$ /mnt/Downloads/llama-b10488/llama-tokenize -m base42m-q4_0.gguf -p 강아지
0.00.302.451 W load: control-looking token:      7 '<|end|>' was not control-type; this is probably a bug in the model. its type will be overridden
26760 -> ' 강아지'

$ /mnt/Downloads/llama-b10488/llama-tokenize -m base42m-q4_0.gguf -p 송아지
0.00.305.743 W load: control-looking token:      7 '<|end|>' was not control-type; this is probably a bug in the model. its type will be overridden
  1259 -> ' 송'
   382 -> '아'
   299 -> '지'

혹시나 해서 gemma4 e4b에서 돌려보는데 어라.. 생각외로 그렇게 토큰을 많이 먹진 않는다고 해야하나?

$ /mnt/Downloads/llama-b10488/llama-tokenize -m  gemma-4-E4B-it-Q4_K_M.gguf -p 마이크로소프트
0.00.926.382 W load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.927.107 W load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.937.154 W load: control-looking token:      1 '<eos>' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.969.157 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
0.01.001.080 W llama_context: n_ctx_seq (512) > n_ctx_train (0) -- possible training context overflow
     2 -> '<bos>'
238098 -> '마'
106427 -> '이크'
237323 -> '로'
238049 -> '소'
128329 -> '프트'

$ /mnt/Downloads/llama-b10488/llama-tokenize -m  gemma-4-E4B-it-Q4_K_M.gguf -p 대한민국
0.00.894.966 W load: control-looking token:    212 '' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.895.695 W load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.905.756 W load: control-looking token:      1 '' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.938.322 W load: special_eog_ids contains '<|tool_response>', removing '' token from EOG list
0.00.970.177 W llama_context: n_ctx_seq (512) > n_ctx_train (0) -- possible training context overflow
     2 -> ''
146220 -> '대한'
162139 -> '민국'

$ /mnt/Downloads/llama-b10488/llama-tokenize -m  gemma-4-E4B-it-Q4_K_M.gguf -p 강아지
0.00.915.146 W load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.915.894 W load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.926.166 W load: control-looking token:      1 '<eos>' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.958.649 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
0.00.990.550 W llama_context: n_ctx_seq (512) > n_ctx_train (0) -- possible training context overflow
     2 -> '<bos>'
238865 -> '강'
237534 -> '아'
237308 -> '지'

$ /mnt/Downloads/llama-b10488/llama-tokenize -m  gemma-4-E4B-it-Q4_K_M.gguf -p 송아지
0.00.895.230 W load: control-looking token:    212 '' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.895.969 W load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.906.303 W load: control-looking token:      1 '' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.939.847 W load: special_eog_ids contains '<|tool_response>', removing '' token from EOG list
0.00.972.088 W llama_context: n_ctx_seq (512) > n_ctx_train (0) -- possible training context overflow
     2 -> ''
239917 -> '송'
237534 -> '아'
237308 -> '지'

/mnt/Downloads/llama-b10488/llama-tokenize -m Qwen3.8-27B-UD-IQ2_XXS.gguf -p 마이크로소프트
0.00.773.458 W llama_context: n_ctx_seq (512) > n_ctx_train (0) -- possible training context overflow
148763 -> '마'
156847 -> '이크'
16869 -> '로'
216243 -> '소프트'

$ /mnt/Downloads/llama-b10488/llama-tokenize -m Qwen3.8-27B-UD-IQ2_XXS.gguf -p 대한민국
0.00.728.852 W llama_context: n_ctx_seq (512) > n_ctx_train (0) -- possible training context overflow
178031 -> '대한'
161266 -> '민국'

$ /mnt/Downloads/llama-b10488/llama-tokenize -m Qwen3.8-27B-UD-IQ2_XXS.gguf -p 강아지
0.00.774.480 W llama_context: n_ctx_seq (512) > n_ctx_train (0) -- possible training context overflow
152029 -> '강'
51181 -> '아'
20673 -> '지'

$ /mnt/Downloads/llama-b10488/llama-tokenize -m Qwen3.8-27B-UD-IQ2_XXS.gguf -p 송아지
0.00.769.448 W llama_context: n_ctx_seq (512) > n_ctx_train (0) -- possible training context overflow
150491 -> '송'
51181 -> '아'
20673 -> '지'

저작자표시 (새창열림)