nvidia thor - LLM

모델이 작은거라 그런가 94% 인데 33W라.. 좀 더 빡세게 굴려봐야하나 ㅋㅋ

 

$ ./llama-b10099/llama-cli -m ./model/gemma4-e4b-qat/gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf  > 안녕? [ Prompt: 8.0 t/s | Generation: 45.2 t/s ] > 너에 대한 소개 [ Prompt: 92.5 t/s | Generation: 51.4 t/s ] > 파이썬으로 셀레니움을 통해 웹을 서칭하고 텍스트만 추출하고 makrdown 으로 변환후  md 파일과 pdf로 저장하는 기능을 구현해줘 [ Prompt: 803.0 t/s | Generation: 48.9 t/s ]

 

$ ./llama-b10099/llama-cli -m ./model/qwen3.5/Qwen3.5-0.8B-UD-Q4_K_XL.gguf -rea off > 안녕?   [ Prompt: 152.3 t/s | Generation: 121.5 t/s ] > 너에 대한 소개 [ Prompt: 218.0 t/s | Generation: 140.4 t/s ] > 파이썬으로 셀레니움을 통해 웹을 서칭하고 텍스트만 추출하고 makrdown 으로 변환후  md 파일과 pdf로 저장하는 기능을 구현해줘 [ Prompt: 68.2 t/s | Generation: 136.5 t/s ]

 

컨텍스트 줄인다고 빨라지진 않네

$ ./llama-b10099/llama-cli -m ./model/qwen3.5/Qwen3.5-0.8B-UD-Q4_K_XL.gguf -rea off -c 4096 > 안녕? [ Prompt: 177.3 t/s | Generation: 131.4 t/s ] > 너에 대한 소개 [ Prompt: 335.9 t/s | Generation: 140.5 t/s ] > 파이썬으로 셀레니움을 통해 웹을 서칭하고 텍스트만 추출하고 makrdown 으로 변환후  md 파일과 pdf로 저장하는 기능을 구현해줘 [ Prompt: 1389.9 t/s | Generation: 136.3 t/s ]

 

$ ./llama-b10099/llama-cli -m ./model/gemma4-e4b/gemma-4-E4B-it-Q4_K_M.gguf  > 안녕? [ Prompt: 1.0 t/s | Generation: 26.9 t/s ] > 안녕? [ Prompt: 65.8 t/s | Generation: 42.2 t/s ] > 너에 대한 소개 [ Prompt: 121.2 t/s | Generation: 41.9 t/s ] > 파이썬으로 셀레니움을 통해 웹을 서칭하고 텍스트만 추출하고 makrdown 으로 변환후  md 파일과 pdf로 저장하는 기능을 구현해줘 [ Prompt: 916.7 t/s | Generation: 41.1 t/s ]

 

gemma4-31B 는 조금 더 뜨끈해진다.

 

$ ./llama-b10099/llama-cli -m ./model/gemma4-31b/gemma-4-31B-it-UD-Q2_K_XL.gguf  > 안녕? [ Prompt: 1.1 t/s | Generation: 6.9 t/s ] > 너에 대한 소개 [ Prompt: 11.5 t/s | Generation: 7.1 t/s ] > 파이썬으로 셀레니움을 통해 웹을 서칭하고 텍스트만 추출하고 makrdown 으로 변환후  md 파일과 pdf로 저장하는 기능을 구현해줘 [ Prompt: 68.3 t/s | Generation: 6.9 t/s ]

 

메모리가 의외로 얼마 안먹나?

nvidia@localhost:~/llm$ free -h                total        used        free      shared  buff/cache   available Mem:           122Gi        40Gi       2.2Gi        26Mi        81Gi        82Gi Swap:          2.0Gi       256Ki       2.0Gi nvidia@localhost:~/llm$ free -h                total        used        free      shared  buff/cache   available Mem:           122Gi        37Gi       5.0Gi        26Mi        81Gi        85Gi Swap:          2.0Gi       256Ki       2.0Gi

 

 파일이 11G 인데 저거밖에 안 먹는게 맞..나?

$ ll -h ./model/gemma4-31b/gemma-4-31B-it-UD-Q2_K_XL.gguf  -rw-rw-r-- 1 nvidia nvidia 11G  7월 24 11:11 ./model/gemma4-31b/gemma-4-31B-it-UD-Q2_K_XL.gguf

 

qwen3.6 35B는 의외로 빠르다 머지?

$ ./llama-b10099/llama-cli -m ./model/qwen3.6_35B/Qwen3.6-35B-A3B-UD-Q2_K_XL.gguf  > 안녕? [ Prompt: 6.0 t/s | Generation: 43.9 t/s ] > 너에 대한 소개 [ Prompt: 39.6 t/s | Generation: 45.1 t/s ] > 파이썬으로 셀레니움을 통해 웹을 서칭하고 텍스트만 추출하고 makrdown 으로 변환후  md 파일과 pdf로 저장하는 기능을 구현해줘 [ Prompt: 93.7 t/s | Generation: 43.6 t/s ]

 

메모리는 봐도 모르겠다

nvidia@localhost:~/llm$ free -h                total        used        free      shared  buff/cache   available Mem:           122Gi        38Gi       3.8Gi        26Mi        81Gi        83Gi Swap:          2.0Gi       256Ki       2.0Gi nvidia@localhost:~/llm$ free -h                total        used        free      shared  buff/cache   available Mem:           122Gi        37Gi       5.0Gi        26Mi        81Gi        85Gi Swap:          2.0Gi       256Ki       2.0Gi

 

+

$ ./llama-b10099/llama-cli -m ./model/gemma4-e4b/gemma-4-E4B-it-Q4_K_M.gguf

 

$ ./llama-b10099/llama-cli -m ./model/qwen3.6_35B/Qwen3.6-35B-A3B-UD-Q2_K_XL.gguf