'프로그램 사용 > Blender' 카테고리의 다른 글
| blender 5.x sculpting (0) | 2026.07.29 |
|---|---|
| blender export render/viewport (0) | 2026.07.29 |
| blender 보이지 않는 면까지 선택하기 (0) | 2026.07.29 |
| blnder 포즈 바꾸면서 렌더하기 (0) | 2026.07.28 |
| blender weight paint (0) | 2026.07.28 |
| blender 5.x sculpting (0) | 2026.07.29 |
|---|---|
| blender export render/viewport (0) | 2026.07.29 |
| blender 보이지 않는 면까지 선택하기 (0) | 2026.07.29 |
| blnder 포즈 바꾸면서 렌더하기 (0) | 2026.07.28 |
| blender weight paint (0) | 2026.07.28 |
alt-z를 누르면 x-ray 모드가 되는데
그러면 뒤에 면까지 선택된다.
즉, 직관적으로 우리가 선택하는 영역내에 모든 vertex가 선택된다고 해야하나?
기본값인 x-ray off
선택하고 보면 먼가 검은 선이 보인다



alt-z 눌러서 x-ray 켜고 선택하면 산뜻한 오렌지로 선택되고

돌려보면 뒤에까지 잘되어있다.

[링크 : https://gall.dcinside.com/mgallery/board/view/?id=blender&no=5106]
| blender export render/viewport (0) | 2026.07.29 |
|---|---|
| blender 떼어내기, 때우기 (0) | 2026.07.29 |
| blnder 포즈 바꾸면서 렌더하기 (0) | 2026.07.28 |
| blender weight paint (0) | 2026.07.28 |
| blender 먼가 숨기기 단축키 (0) | 2026.07.22 |
vllm 에서 퍼포먼스 테스트 하는 스크립트가 들어깄다고 하는데..
[링크 : https://blog.scatterlab.co.kr/vllm-implementation-details]
[링크 : https://devocean.sk.com/blog/techBoardDetail.do?ID=167138]
[링크 : https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611]
[링크 : https://www.sktenterprise.com/bizInsight/blogDetail/dev/14546]
| llama.cpp / jetson thor gemma-4-31B mmproj F16, BF16 (0) | 2026.07.30 |
|---|---|
| llama.cpp 동시에 응답하게 하기 (0) | 2026.07.29 |
| vllm parallelism - concurrency (0) | 2026.07.28 |
| nvidia thor - llama.cpp LLM (0) | 2026.07.24 |
| llama.cpp for arm64 (0) | 2026.07.20 |
검색하다 보니 이상한(?) C1 C8 이라는 문구 발견, concurrent request 라고 되어있는데
결국은 동시에 응답을 해줄수 있게 돌리냐 아니냐의 차이인듯.
| Modes: C1 (single request) C8 (8 concurrent requests) |
| --max-concurrency¶ Maximum number of concurrent requests. This can be used to help simulate an environment where a higher level component is enforcing a maximum number of concurrent requests. While the --request-rate argument controls the rate at which requests are initiated, this argument will control how many are actually allowed to execute at a time. This means that when used in combination, the actual request rate may be lower than specified with --request-rate, if the server is not processing requests fast enough to keep up |
[링크 : https://docs.vllm.ai/en/stable/cli/bench/serve/#arguments]
| Multi-node multi-GPU using tensor parallel and pipeline parallel inference: if the model is too large for a single node, combine tensor parallelism with pipeline parallelism. Set tensor_parallel_size to the number of GPUs per node and pipeline_parallel_size to the number of nodes. For example, set tensor_parallel_size=8 and pipeline_parallel_size=2 when using 2 nodes with 8 GPUs per node. |
[링크 : https://docs.vllm.ai/en/stable/serving/parallelism_scaling/#single-node-deployment]
| llama.cpp 동시에 응답하게 하기 (0) | 2026.07.29 |
|---|---|
| vllm concurrent test/benchmark (0) | 2026.07.29 |
| nvidia thor - llama.cpp LLM (0) | 2026.07.24 |
| llama.cpp for arm64 (0) | 2026.07.20 |
| llama.cpp / qwen3.5 0.8B / 1060, 1080 ti (0) | 2026.07.19 |
유저 레벨에서 트래픽 제한해주는 유틸리티
ai에게 대역폭 제한해달라고 하니 이걸 추천해준다.
근데 vLLM docker 에서 설치하려니 후보가 없다고 -_-
[링크 : https://github.com/mariusae/trickle]
[링크 : https://manpages.ubuntu.com/manpages/focal/man1/trickle.1.html]
| gary's mod - 게리모드 (0) | 2026.06.24 |
|---|---|
| ventoy 업그레이드 (0) | 2026.06.23 |
| microhttpd (0) | 2026.06.05 |
| tmux (0) | 2026.05.11 |
| appimage , AppImageLauncher (0) | 2026.04.20 |
armature 관련 ridgify 였나? 플러그인으로 생성한 골격을 이용해서

아래처럼 다양한 포즈로 렌더하는 예제

pose mode 까진 아니어도
armature가 pose로 되어있어야 한다는데(by claude) 무슨소리인진 모르겠다.
import bpy
import random
import math
import os
# -------------------------
# 설정
# -------------------------
NUM_IMAGES = 200
ARMATURE_NAME = "metarig" # 랜덤 포즈를 적용할 아마추어 이름
MAX_ANGLE_DEG = 25.0 # 본 하나가 회전할 수 있는 최대 각도(도)
EXCLUDE_NAME_KEYWORDS = [ # 이 문자열이 이름에 포함된 본은 건드리지 않음
"root", "pelvis", "spine", "metarig",
]
# IK/Copy Rotation/Child Of 등 다른 소스가 회전을 덮어쓰는 컨스트레인트 타입.
# 이런 컨스트레인트가 걸린 본은 rotation_euler를 바꿔도 무시되므로 건너뜁니다.
SKIP_CONSTRAINT_TYPES = {"IK", "COPY_ROTATION", "COPY_TRANSFORMS", "CHILD_OF"}
RESPECT_ROTATION_LIMITS = True
RANDOM_SEED = None
DEBUG = True
armature = bpy.data.objects[ARMATURE_NAME]
scene = bpy.context.scene
if DEBUG:
print(f"[DEBUG] 현재 씬: {scene.name} / 전체 씬 목록: {list(bpy.data.scenes.keys())}")
print(f"[DEBUG] 아마추어: {armature.name}, 모드: {armature.mode}")
print(f"[DEBUG] pose_position (변경 전): {armature.data.pose_position}")
print(f"[DEBUG] animation_data: {armature.animation_data}")
if armature.animation_data and armature.animation_data.action:
print(f"[DEBUG] action: {armature.animation_data.action.name}")
# 1) Rest Position으로 설정되어 있으면 본 회전이 렌더에 반영되지 않으므로 강제로 Pose Position 설정
if armature.data.pose_position != 'POSE':
armature.data.pose_position = 'POSE'
if DEBUG:
print(f"[DEBUG] pose_position을 'POSE'로 변경함")
# 2) 액션(F-curve)이 물려있으면 매 프레임 평가 시 수동으로 넣은 rotation_euler를 덮어쓰므로 해제
if armature.animation_data and armature.animation_data.action is not None:
if DEBUG:
print(f"[DEBUG] 기존 액션 '{armature.animation_data.action.name}' 을 해제함 (수동 포즈와 충돌 방지)")
armature.animation_data.action = None
scene.render.engine = 'BLENDER_EEVEE'
scene.render.resolution_x = 640
scene.render.resolution_y = 480
scene.render.resolution_percentage = 100
# 저장 폴더 (blend 파일 기준)
output_dir = bpy.path.abspath("//render_output")
os.makedirs(output_dir, exist_ok=True)
# PNG 저장
scene.render.image_settings.file_format = 'PNG'
def should_skip(bone_name: str) -> bool:
lower = bone_name.lower()
return any(kw.lower() in lower for kw in EXCLUDE_NAME_KEYWORDS)
def has_blocking_constraint(pbone) -> bool:
"""IK 등 다른 소스가 회전을 덮어쓰는 컨스트레인트가 있으면 True."""
for c in pbone.constraints:
if c.mute:
continue
if c.type in SKIP_CONSTRAINT_TYPES:
return True
return False
def get_rotation_limits(pbone):
"""본에 Limit Rotation 컨스트레인트가 있으면 (min, max) 라디안 튜플 반환."""
for c in pbone.constraints:
if c.type == "LIMIT_ROTATION" and not c.mute:
return (
(c.min_x, c.max_x),
(c.min_y, c.max_y),
(c.min_z, c.max_z),
c.use_limit_x, c.use_limit_y, c.use_limit_z,
)
return None
def random_rotate_bone(pbone, max_angle_rad):
limits = get_rotation_limits(pbone) if RESPECT_ROTATION_LIMITS else None
prev_mode = pbone.rotation_mode
if prev_mode != "XYZ":
pbone.rotation_mode = "XYZ"
rx = random.uniform(-max_angle_rad, max_angle_rad)
ry = random.uniform(-max_angle_rad, max_angle_rad)
rz = random.uniform(-max_angle_rad, max_angle_rad)
if limits:
(minx, maxx), (miny, maxy), (minz, maxz), ux, uy, uz = limits
if ux:
rx = max(minx, min(maxx, rx))
if uy:
ry = max(miny, min(maxy, ry))
if uz:
rz = max(minz, min(maxz, rz))
pbone.rotation_euler = (rx, ry, rz)
def randomize_pose():
max_angle_rad = math.radians(MAX_ANGLE_DEG)
rotated = 0
skipped_name = 0
skipped_constraint = 0
for pbone in armature.pose.bones:
if should_skip(pbone.name):
skipped_name += 1
continue
if has_blocking_constraint(pbone):
skipped_constraint += 1
continue
random_rotate_bone(pbone, max_angle_rad)
rotated += 1
if DEBUG:
print(f"[DEBUG] 회전 적용: {rotated}개 / 이름 제외: {skipped_name}개 / 컨스트레인트 제외: {skipped_constraint}개")
if RANDOM_SEED is not None:
random.seed(RANDOM_SEED)
bpy.context.view_layer.objects.active = armature
prev_mode = armature.mode
if armature.mode != "POSE":
result = bpy.ops.object.mode_set(mode="POSE")
if DEBUG:
print(f"[DEBUG] mode_set(POSE) 결과: {result}, 현재 모드: {armature.mode}")
if DEBUG:
all_bones = [b.name for b in armature.pose.bones]
print(f"[DEBUG] 전체 본 개수: {len(all_bones)}")
print(f"[DEBUG] 본 목록: {all_bones}")
# -------------------------
# 렌더링 루프
# -------------------------
for i in range(NUM_IMAGES):
# 매 렌더마다 아마추어 랜덤 포즈 적용
randomize_pose()
# 씬/디펜던시 그래프 강제 갱신 (배치 렌더 시 이전 프레임 캐시 문제 방지)
bpy.context.view_layer.update()
depsgraph = bpy.context.evaluated_depsgraph_get()
depsgraph.update()
# 렌더 저장 경로
scene.render.filepath = os.path.join(output_dir, f"image_{i:03d}.png")
print(f"Rendering {scene.render.filepath}")
bpy.ops.render.render(write_still=True)
if prev_mode != armature.mode:
bpy.ops.object.mode_set(mode=prev_mode)
print("Done.")
+
object mode / bone 에서 이런걸 발견. 아.. 이게 위에 그거였나?

| blender 떼어내기, 때우기 (0) | 2026.07.29 |
|---|---|
| blender 보이지 않는 면까지 선택하기 (0) | 2026.07.29 |
| blender weight paint (0) | 2026.07.28 |
| blender 먼가 숨기기 단축키 (0) | 2026.07.22 |
| blender quarternion (0) | 2026.07.22 |
armature 만드는거랑 auto weight는 일단 스킵
weight paint 모드로 들어와서 우측의 properties에 보면
vertex groups 라고 관절들이 보이는데 이걸 클릭하면 왼쪽에서 위치별로 weight paint가 보이게 된다.
![]() |
![]() |
왼쪽 클릭후 문질러 대면 되는데
weight 0.0 으로 하고 strength 1.0 으로 하니 검은색으로 칠해지면서 weight를 사라지게 할 수 있긴한데..
편한 방법 없나..


반대편 다리에 검게 해두니 확실히 해당 관절을 돌릴때 다른쪽에 영향을 안 받긴 한다.

그런데.. 원하지 않는데 까지 막 칠해지는데.. face 단위로 영역 지정이 가능하려나?

[링크 : https://m.blog.naver.com/cjw531/223159464692]
[링크 : https://tintana4168.tistory.com/22]
| blender 보이지 않는 면까지 선택하기 (0) | 2026.07.29 |
|---|---|
| blnder 포즈 바꾸면서 렌더하기 (0) | 2026.07.28 |
| blender 먼가 숨기기 단축키 (0) | 2026.07.22 |
| blender quarternion (0) | 2026.07.22 |
| blender ik (0) | 2026.07.15 |
| 먼가 주식 안하면 바보인 시대 (2) | 2026.02.22 |
|---|---|
| 혼돈 파괴 카오스(?)의 미국 (0) | 2026.01.20 |
| 패권주의의 부활? (0) | 2026.01.04 |
| 영포티 단상 (0) | 2025.11.10 |
| 은행 이자는 점점 떨어지네 (0) | 2025.09.17 |
얼굴 인증 잘 안되서 짜증나고
kb 인증 사라지고 네이버 앱은 싫고 카카오는 카카오 페이 다시 해야하니 싫어서
결국에는 pass 앱 깔고 인증 진행
110원에 이어 10원 ㅋㅋㅋ
그런데 6개월 마다 가려니 겁나 짜증날 것 같네

+
2026.07.29
첫째는 내꺼랑 같은걸로. 3개월 뒤에 sk로 옮겨야지 ㅠㅠ
둘째는 이걸로


2026.07.28에 하니 model 경로가 달라진듯 하다.
/root/.cache/huggingface 라고 명시되어 있더니, /data/models/huggingface 로 가버림
| $ docker run --pull=always --rm -it \ --network host \ --shm-size=16g \ --ulimit memlock=-1 \ --ulimit stack=67108864 \ --runtime=nvidia \ --name=vllm_gemma4 \ -v $HOME/data/models/huggingface:/data/models/huggingface \ ghcr.io/nvidia-ai-iot/vllm:gemma4-jetson-thor \ vllm serve google/gemma-4-E4B-it \ --gpu-memory-utilization 0.5 \ --enable-auto-tool-choice \ --tool-call-parser hermes |
| $ docker run --pull=always --rm -it \ --network host \ --shm-size=16g \ --ulimit memlock=-1 \ --ulimit stack=67108864 \ --runtime=nvidia \ --name=vllm \ -v $HOME/data/models/huggingface:/data/models/huggingface \ ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor \ vllm serve Qwen/Qwen3.6-35B-A3B \ --enable-auto-tool-choice \ --tool-call-parser hermes |
----
저거 정도는 좀 찐하게 칠해주던가 -_ㅠ
한참 받고 했는데 안되서 멘붕왔다가 클로드 물어보니 orin 이자너! 외쳐주심 ㅋㅋ
인간이 미안해 ㅋㅋㅋㅋ
| docker run -d \ --network=host \ -v ${HOME}/open-webui:/app/backend/data \ -e OPENAI_API_BASE_URL=http://0.0.0.0:8000/v1 \ --name open-webui \ ghcr.io/open-webui/open-webui:main docker run --pull=always --rm -it \ --network host \ --shm-size=16g \ --ulimit memlock=-1 \ --ulimit stack=67108864 \ --runtime=nvidia \ --name=vllm \ -v $HOME/data/models/huggingface:/root/.cache/huggingface \ ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor \ vllm serve RedHatAI/Qwen3-8B-quantized.w4a16 |
[링크 : https://www.jetson-ai-lab.com/tutorials/genai-on-jetson-llms-vlms/#-vllm-for-best-performance]
vllm server 뒤에는 huggingface의 url을 때려 박으면 되나보다
양자화 차이는 있지만 그러면.. vllm server Qwen/Qwen3.5-0.8B 해봐야겠네
[링크 : https://huggingface.co/Qwen/Qwen3.5-0.8B]
open webui 에서 이런 에러가 발생해서 추가!
| Qwen/Qwen3.5-0.8B "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set |
잘 돌아는 가는데..
| nvidia@nvidia:~$ docker run --pull=always --rm -it \ --network host \ --shm-size=16g \ --ulimit memlock=-1 \ --ulimit stack=67108864 \ --runtime=nvidia \ --name=vllm \ -v $HOME/data/models/huggingface:/root/.cache/huggingface \ ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor \ vllm serve Qwen/Qwen3.5-0.8B \ --enable-auto-tool-choice \ --tool-call-parser hermes latest-jetson-thor: Pulling from nvidia-ai-iot/vllm Digest: sha256:b587dd56b4cb076209ad5156a626ac75f5a976d0e8e7d1e6a9fccd56d1bd65e8 Status: Image is up to date for ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor /opt/venv/lib/python3.12/site-packages/transformers/utils/hub.py:110: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead. warnings.warn( (APIServer pid=1) INFO 07-28 03:09:00 [utils.py:299] (APIServer pid=1) INFO 07-28 03:09:00 [utils.py:299] █ █ █▄ ▄█ (APIServer pid=1) INFO 07-28 03:09:00 [utils.py:299] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.19.0 (APIServer pid=1) INFO 07-28 03:09:00 [utils.py:299] █▄█▀ █ █ █ █ model Qwen/Qwen3.5-0.8B (APIServer pid=1) INFO 07-28 03:09:00 [utils.py:299] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ (APIServer pid=1) INFO 07-28 03:09:00 [utils.py:299] (APIServer pid=1) INFO 07-28 03:09:00 [utils.py:233] non-default args: {'model_tag': 'Qwen/Qwen3.5-0.8B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'hermes', 'model': 'Qwen/Qwen3.5-0.8B'} config.json: 2.91kB [00:00, 16.0MB/s] |
llama.cpp 처럼 한번 문장 물어 본것에 대한 속도가 안나와서 비교하기가 애매~하다.
| (APIServer pid=1) INFO 07-28 03:16:02 [loggers.py:259] Engine 000: Avg prompt throughput: 647.4 tokens/s, Avg generation throughput: 7.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:56010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:56016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:16:12 [loggers.py:259] Engine 000: Avg prompt throughput: 20.4 tokens/s, Avg generation throughput: 3.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:37160 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:37160 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:37162 - "GET /v1/models HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:16:22 [loggers.py:259] Engine 000: Avg prompt throughput: 643.2 tokens/s, Avg generation throughput: 36.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:54470 - "GET /v1/models HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:16:32 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 192.168.40.209:36304 - "GET /metrics HTTP/1.1" 200 OK (APIServer pid=1) INFO: 192.168.40.209:36304 - "GET /favicon.ico HTTP/1.1" 404 Not Found (APIServer pid=1) INFO: 127.0.0.1:53220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:53220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:19:12 [loggers.py:259] Engine 000: Avg prompt throughput: 666.3 tokens/s, Avg generation throughput: 18.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:19:22 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:57598 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:19:32 [loggers.py:259] Engine 000: Avg prompt throughput: 622.3 tokens/s, Avg generation throughput: 39.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:57598 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:19:42 [loggers.py:259] Engine 000: Avg prompt throughput: 91.0 tokens/s, Avg generation throughput: 7.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:19:52 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:56452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:56450 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:56450 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:56452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:20:52 [loggers.py:259] Engine 000: Avg prompt throughput: 668.1 tokens/s, Avg generation throughput: 14.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:51276 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:21:02 [loggers.py:259] Engine 000: Avg prompt throughput: 597.2 tokens/s, Avg generation throughput: 25.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:51276 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:21:12 [loggers.py:259] Engine 000: Avg prompt throughput: 112.8 tokens/s, Avg generation throughput: 71.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:21:22 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:43142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:21:32 [loggers.py:259] Engine 000: Avg prompt throughput: 686.0 tokens/s, Avg generation throughput: 28.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:21:42 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 132.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:21:52 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 127.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:40238 - "GET /v1/models HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:40254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:40254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:22:02 [loggers.py:259] Engine 000: Avg prompt throughput: 1424.5 tokens/s, Avg generation throughput: 56.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:52062 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:52062 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:22:12 [loggers.py:259] Engine 000: Avg prompt throughput: 1407.0 tokens/s, Avg generation throughput: 26.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:38124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:38124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:22:22 [loggers.py:259] Engine 000: Avg prompt throughput: 1139.7 tokens/s, Avg generation throughput: 24.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:38124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:22:32 [loggers.py:259] Engine 000: Avg prompt throughput: 1053.3 tokens/s, Avg generation throughput: 110.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:40160 - "GET /v1/models HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:40174 - "GET /v1/models HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:22:42 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 129.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:37106 - "GET /v1/models HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:38124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:37108 - "GET /v1/models HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:38124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:22:52 [loggers.py:259] Engine 000: Avg prompt throughput: 1634.7 tokens/s, Avg generation throughput: 59.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:23:02 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 127.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:38124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:23:12 [loggers.py:259] Engine 000: Avg prompt throughput: 535.5 tokens/s, Avg generation throughput: 116.7 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:23:22 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:49210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:26:32 [loggers.py:259] Engine 000: Avg prompt throughput: 1562.3 tokens/s, Avg generation throughput: 111.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:26:42 [loggers.py:259] Engine 000: Avg prompt throughput: 743.5 tokens/s, Avg generation throughput: 121.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:54226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:49210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:54226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:49210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:26:52 [loggers.py:259] Engine 000: Avg prompt throughput: 4021.5 tokens/s, Avg generation throughput: 92.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:27:02 [loggers.py:259] Engine 000: Avg prompt throughput: 369.0 tokens/s, Avg generation throughput: 122.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:49210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:49210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:27:12 [loggers.py:259] Engine 000: Avg prompt throughput: 1909.2 tokens/s, Avg generation throughput: 71.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:27:22 [loggers.py:259] Engine 000: Avg prompt throughput: 331.3 tokens/s, Avg generation throughput: 120.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:49210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:49210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:27:32 [loggers.py:259] Engine 000: Avg prompt throughput: 2077.2 tokens/s, Avg generation throughput: 91.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:27:42 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 121.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:42812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:49210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:27:52 [loggers.py:259] Engine 000: Avg prompt throughput: 2832.4 tokens/s, Avg generation throughput: 110.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:56048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:42812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:28:02 [loggers.py:259] Engine 000: Avg prompt throughput: 3328.9 tokens/s, Avg generation throughput: 132.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:42812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:28:12 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:28:22 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:43420 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:43406 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:28:42 [loggers.py:259] Engine 000: Avg prompt throughput: 628.9 tokens/s, Avg generation throughput: 25.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:43406 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:43420 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:43406 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 03:28:52 [loggers.py:259] Engine 000: Avg prompt throughput: 764.8 tokens/s, Avg generation throughput: 85.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-28 03:29:02 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 132.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:43406 - "POST /v1/chat/completions HTTP/1.1" 200 OK |
llama.cpp 에서는 120 넘기기도 쉽지 않았는데 vLLM이 조금더 성능이 잘 나오긴 하는건가?
약.. 10% 정도?
| [ Prompt: 93.1 t/s | Generation: 122.7 t/s ] [ Prompt: 138.6 t/s | Generation: 100.3 t/s ] [ Prompt: 264.0 t/s | Generation: 119.6 t/s ] [ Prompt: 1654.8 t/s | Generation: 122.1 t/s ] [ Prompt: 1379.7 t/s | Generation: 119.8 t/s ] [ Prompt: 2284.5 t/s | Generation: 117.7 t/s ] [ Prompt: 2549.1 t/s | Generation: 112.2 t/s ] [ Prompt: 1214.4 t/s | Generation: 111.4 t/s ] [ Prompt: 803.1 t/s | Generation: 115.3 t/s ] |
+
gemma를 시도하려고 했는데 에러남. qwen은 되더니 쳇
| nvidia@nvidia:~$ docker run --pull=always --rm -it --network host --shm-size=16g --ulimit memlock=-1 --ulimit stack=67108864 --runtime=nvidia --name=vllm -v $HOME/data/models/huggingface:/root/.cache/huggingface ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve google/gemma-4-31B-it nvidia@nvidia:~$ docker run --pull=always --rm -it --network host --shm-size=16g --ulimit memlock=-1 --ulimit stack=67108864 --runtime=nvidia --name=vllm -v $HOME/data/models/huggingface:/root/.cache/huggingface ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve google/gemma-4-E4B-it latest-jetson-thor: Pulling from nvidia-ai-iot/vllm Digest: sha256:b587dd56b4cb076209ad5156a626ac75f5a976d0e8e7d1e6a9fccd56d1bd65e8 Status: Image is up to date for ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor /opt/venv/lib/python3.12/site-packages/transformers/utils/hub.py:110: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead. warnings.warn( (APIServer pid=1) INFO 07-28 03:40:47 [utils.py:299] (APIServer pid=1) INFO 07-28 03:40:47 [utils.py:299] █ █ █▄ ▄█ (APIServer pid=1) INFO 07-28 03:40:47 [utils.py:299] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.19.0 (APIServer pid=1) INFO 07-28 03:40:47 [utils.py:299] █▄█▀ █ █ █ █ model google/gemma-4-E4B-it (APIServer pid=1) INFO 07-28 03:40:47 [utils.py:299] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ (APIServer pid=1) INFO 07-28 03:40:47 [utils.py:299] (APIServer pid=1) INFO 07-28 03:40:47 [utils.py:233] non-default args: {'model_tag': 'google/gemma-4-E4B-it', 'model': 'google/gemma-4-E4B-it'} config.json: 5.14kB [00:00, 18.5MB/s] (APIServer pid=1) Traceback (most recent call last): (APIServer pid=1) File "/opt/venv/bin/vllm", line 10, in <module> (APIServer pid=1) sys.exit(main()) (APIServer pid=1) ^^^^^^ (APIServer pid=1) File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 75, in main (APIServer pid=1) args.dispatch_function(args) (APIServer pid=1) File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 122, in cmd (APIServer pid=1) uvloop.run(run_server(args)) (APIServer pid=1) File "/opt/venv/lib/python3.12/site-packages/uvloop/__init__.py", line 96, in run (APIServer pid=1) return __asyncio.run( (APIServer pid=1) ^^^^^^^^^^^^^^ (APIServer pid=1) File "/root/.local/share/uv/python/cpython-3.12.13-linux-aarch64-gnu/lib/python3.12/asyncio/runners.py", line 195, in run (APIServer pid=1) return runner.run(main) (APIServer pid=1) ^^^^^^^^^^^^^^^^ (APIServer pid=1) File "/root/.local/share/uv/python/cpython-3.12.13-linux-aarch64-gnu/lib/python3.12/asyncio/runners.py", line 118, in run (APIServer pid=1) return self._loop.run_until_complete(task) (APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=1) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete (APIServer pid=1) File "/opt/venv/lib/python3.12/site-packages/uvloop/__init__.py", line 48, in wrapper (APIServer pid=1) return await main (APIServer pid=1) ^^^^^^^^^^ (APIServer pid=1) File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 670, in run_server (APIServer pid=1) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs) (APIServer pid=1) File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 684, in run_server_worker (APIServer pid=1) async with build_async_engine_client( (APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=1) File "/root/.local/share/uv/python/cpython-3.12.13-linux-aarch64-gnu/lib/python3.12/contextlib.py", line 210, in __aenter__ (APIServer pid=1) return await anext(self.gen) (APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=1) File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client (APIServer pid=1) async with build_async_engine_client_from_engine_args( (APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=1) File "/root/.local/share/uv/python/cpython-3.12.13-linux-aarch64-gnu/lib/python3.12/contextlib.py", line 210, in __aenter__ (APIServer pid=1) return await anext(self.gen) (APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=1) File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 124, in build_async_engine_client_from_engine_args (APIServer pid=1) vllm_config = engine_args.create_engine_config(usage_context=usage_context) (APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=1) File "/opt/venv/lib/python3.12/site-packages/vllm/engine/arg_utils.py", line 1549, in create_engine_config (APIServer pid=1) model_config = self.create_model_config() (APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=1) File "/opt/venv/lib/python3.12/site-packages/vllm/engine/arg_utils.py", line 1398, in create_model_config (APIServer pid=1) return ModelConfig( (APIServer pid=1) ^^^^^^^^^^^^ (APIServer pid=1) File "/opt/venv/lib/python3.12/site-packages/pydantic/_internal/_dataclasses.py", line 121, in __init__ (APIServer pid=1) s.__pydantic_validator__.validate_python(ArgsKwargs(args, kwargs), self_instance=s) (APIServer pid=1) pydantic_core._pydantic_core.ValidationError: 1 validation error for ModelConfig (APIServer pid=1) Value error, The checkpoint you are trying to load has model type `gemma4` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date. (APIServer pid=1) (APIServer pid=1) You can update Transformers with the command `pip install --upgrade transformers`. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the command `pip install git+https://github.com/huggingface/transformers.git` [type=value_error, input_value=ArgsKwargs((), {'model': ...nderer_num_workers': 1}), input_type=ArgsKwargs] (APIServer pid=1) For further information visit https://errors.pydantic.dev/2.12/v/value_error |
아무튼 thor용 gemma4를 받아서 해봐야지 머
| docker pull ghcr.io/nvidia-ai-iot/vllm:gemma4-jetson-thor |
[링크 : https://github.com/orgs/nvidia-ai-iot/packages/container/vllm/784603821?tag=gemma4-jetson-thor]
받는지 티가 안나서 일단 hf_token 추가해줬는데 로그 상으로는 차이가 별로 없다.(한 줄?)
btop 해서 보니 열심히 다운로드 받는 중.
| nvidia@nvidia:~$ docker run --pull=always --rm -it --network host --shm-size=16g --ulimit memlock=-1 --ulimit stack=67108864 --runtime=nvidia --name=vllm_gemma4 -v $HOME/data/models/huggingface:/root/.cache/huggingface ghcr.io/nvidia-ai-iot/vllm:gemma4-jetson-thor vllm serve google/gemma-4-E4B-it gemma4-jetson-thor: Pulling from nvidia-ai-iot/vllm Digest: sha256:570f9a5ffa89a772226abcc98c2d358a56ec3f755c97bc079c7f2396ffe62260 Status: Image is up to date for ghcr.io/nvidia-ai-iot/vllm:gemma4-jetson-thor (APIServer pid=1) INFO 07-28 04:38:44 [utils.py:299] (APIServer pid=1) INFO 07-28 04:38:44 [utils.py:299] █ █ █▄ ▄█ (APIServer pid=1) INFO 07-28 04:38:44 [utils.py:299] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.19.0 (APIServer pid=1) INFO 07-28 04:38:44 [utils.py:299] █▄█▀ █ █ █ █ model google/gemma-4-E4B-it (APIServer pid=1) INFO 07-28 04:38:44 [utils.py:299] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ (APIServer pid=1) INFO 07-28 04:38:44 [utils.py:299] (APIServer pid=1) INFO 07-28 04:38:44 [utils.py:233] non-default args: {'model_tag': 'google/gemma-4-E4B-it', 'model': 'google/gemma-4-E4B-it'} (APIServer pid=1) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. config.json: 5.14kB [00:00, 12.9MB/s] processor_config.json: 1.69kB [00:00, 4.13MB/s] (APIServer pid=1) INFO 07-28 04:38:54 [model.py:549] Resolved architecture: Gemma4ForConditionalGeneration (APIServer pid=1) INFO 07-28 04:38:54 [model.py:1678] Using max model len 131072 (APIServer pid=1) INFO 07-28 04:38:54 [config.py:104] Gemma4 model has heterogeneous head dimensions (head_dim=256, global_head_dim=512). Forcing TRITON_ATTN backend to prevent mixed-backend numerical divergence. (APIServer pid=1) INFO 07-28 04:38:54 [vllm.py:790] Asynchronous scheduling is enabled. tokenizer_config.json: 3.08kB [00:00, 8.20MB/s] tokenizer.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 32.2M/32.2M [00:05<00:00, 5.53MB/s] chat_template.jinja: 18.6kB [00:00, 26.0MB/s] generation_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 208/208 [00:00<00:00, 809kB/s] (EngineCore pid=128) INFO 07-28 04:40:14 [core.py:105] Initializing a V1 LLM engine (v0.19.0) with config: model='google/gemma-4-E4B-it', speculative_config=None, tokenizer='google/gemma-4-E4B-it', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=google/gemma-4-E4B-it, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_images_per_batch': 0, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} (EngineCore pid=128) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. (EngineCore pid=128) INFO 07-28 04:40:20 [parallel_state.py:1400] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.40.202:38135 backend=nccl (EngineCore pid=128) INFO 07-28 04:40:20 [parallel_state.py:1716] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A (EngineCore pid=128) INFO 07-28 04:40:42 [gpu_model_runner.py:4735] Starting to load model google/gemma-4-E4B-it... (EngineCore pid=128) INFO 07-28 04:40:42 [vllm.py:790] Asynchronous scheduling is enabled. (EngineCore pid=128) INFO 07-28 04:40:42 [cuda.py:274] Using AttentionBackendEnum.TRITON_ATTN backend. (EngineCore pid=128) INFO 07-28 04:40:43 [cuda.py:274] Using AttentionBackendEnum.TRITON_ATTN backend. |
gemma4-e4b-it-q4_k_m 이 40t/s 정도 나왔는데 반해, vLLM 에서는 오히려 저조하게 나오는 듯 하다.
| (APIServer pid=1) INFO 07-28 05:50:55 [loggers.py:259] Engine 000: Avg prompt throughput: 273.3 tokens/s, Avg generation throughput: 2.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 47.3% (APIServer pid=1) INFO: 127.0.0.1:53598 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:53598 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:47762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 05:51:05 [loggers.py:259] Engine 000: Avg prompt throughput: 52.8 tokens/s, Avg generation throughput: 32.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 63.2% (APIServer pid=1) INFO: 127.0.0.1:53600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 05:51:15 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 30.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 63.2% (APIServer pid=1) INFO 07-28 05:51:25 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 63.2% (APIServer pid=1) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:47762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:47762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 05:51:35 [loggers.py:259] Engine 000: Avg prompt throughput: 154.2 tokens/s, Avg generation throughput: 40.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 75.0% (APIServer pid=1) INFO 07-28 05:51:45 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 75.0% (APIServer pid=1) INFO 07-28 05:51:55 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 75.0% (APIServer pid=1) INFO 07-28 05:52:05 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 75.0% (APIServer pid=1) INFO: 127.0.0.1:39358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 05:52:15 [loggers.py:259] Engine 000: Avg prompt throughput: 172.7 tokens/s, Avg generation throughput: 31.3 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 76.3% (APIServer pid=1) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-28 05:52:25 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 26.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 76.3% (APIServer pid=1) INFO 07-28 05:52:35 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 76.3% (APIServer pid=1) INFO 07-28 05:52:45 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 76.3% (APIServer pid=1) INFO 07-28 05:52:55 [loggers.py:259] Engine 000: Avg prompt throughput: 213.4 tokens/s, Avg generation throughput: 23.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 72.6% |
[링크 : https://huggingface.co/google/gemma-4-E4B-it]
[링크 : https://huggingface.co/google/gemma-4-31B-it]
[링크 : https://huggingface.co/Qwen/Qwen3.6-35B-A3B]
+
docker로 하나 돌고 있는 상황에서 하나더 돌리려고 하니 이런 에러가 난다.
110GiB를 먹고 시작하겠다.. 라는 패기라니.. ㄷㄷ
| (EngineCore pid=127) ValueError: Free memory on device cuda:0 (42.65/122.8 GiB) on startup is less than desired GPU memory utilization (0.9, 110.52 GiB). Decrease GPU memory utilization or reduce GPU memory used by other processes. [rank0]:[W728 04:30:09.155450975 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) |
+
20276.07.29
google/gemma-4-31B-it
| $ docker run --pull=always --rm -it --network host --shm-size=16g --ulimit memlock=-1 --ulimit stack=67108864 --runtime=nvidia --name=vllm -v $HOME/data/models/huggingface:/data/models/huggingface ghcr.io/nvidia-ai-iot/vllm:gemma4-jetson-thor vllm serve google/gemma-4-31B-it --enable-auto-tool-choice --tool-call-parser hermes gemma4-jetson-thor: Pulling from nvidia-ai-iot/vllm |
하나만 해서 하면 4 token/s 나오는데
두개 띄워놓고 동시에 갈궈대면 먼가 뻥튀기 되는 느낌?
| (APIServer pid=1) INFO: 127.0.0.1:44406 - "GET /v1/models HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:51624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-29 01:13:31 [loggers.py:259] Engine 000: Avg prompt throughput: 26.9 tokens/s, Avg generation throughput: 0.3 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:51620 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:51624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-29 01:13:41 [loggers.py:259] Engine 000: Avg prompt throughput: 552.2 tokens/s, Avg generation throughput: 5.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 47.1% (APIServer pid=1) INFO: 127.0.0.1:50062 - "GET /v1/models HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-29 01:13:51 [loggers.py:259] Engine 000: Avg prompt throughput: 6.3 tokens/s, Avg generation throughput: 8.1 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 64.3% (APIServer pid=1) INFO: 127.0.0.1:51620 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:50070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:50070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:51620 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-29 01:14:01 [loggers.py:259] Engine 000: Avg prompt throughput: 43.0 tokens/s, Avg generation throughput: 11.8 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 71.8% (APIServer pid=1) INFO: 127.0.0.1:50988 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:51624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:50070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-29 01:14:11 [loggers.py:259] Engine 000: Avg prompt throughput: 38.1 tokens/s, Avg generation throughput: 12.3 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 80.2% (APIServer pid=1) INFO: 127.0.0.1:51256 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-29 01:14:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.0 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 82.6% (APIServer pid=1) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:51624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:51624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-29 01:14:31 [loggers.py:259] Engine 000: Avg prompt throughput: 28.9 tokens/s, Avg generation throughput: 14.6 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 82.5% (APIServer pid=1) INFO: 127.0.0.1:50070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:50988 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-29 01:14:41 [loggers.py:259] Engine 000: Avg prompt throughput: 41.5 tokens/s, Avg generation throughput: 13.6 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 83.9% (APIServer pid=1) INFO: 127.0.0.1:51624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-29 01:14:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 83.9% (APIServer pid=1) INFO 07-29 01:15:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 83.9% (APIServer pid=1) INFO 07-29 01:15:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 83.9% (APIServer pid=1) INFO 07-29 01:15:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 83.9% (APIServer pid=1) INFO 07-29 01:15:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 83.9% (APIServer pid=1) INFO 07-29 01:15:41 [loggers.py:259] Engine 000: Avg prompt throughput: 31.2 tokens/s, Avg generation throughput: 7.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 83.4% (APIServer pid=1) INFO 07-29 01:15:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 83.4% (APIServer pid=1) INFO: 127.0.0.1:51256 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-29 01:16:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 83.4% (APIServer pid=1) INFO 07-29 01:16:11 [loggers.py:259] Engine 000: Avg prompt throughput: 56.1 tokens/s, Avg generation throughput: 3.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 82.5% (APIServer pid=1) INFO 07-29 01:16:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 82.5% (APIServer pid=1) INFO: 127.0.0.1:50070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-29 01:16:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 82.5% (APIServer pid=1) INFO 07-29 01:16:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 82.5% (APIServer pid=1) INFO: 127.0.0.1:55952 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-29 01:43:01 [loggers.py:259] Engine 000: Avg prompt throughput: 37.9 tokens/s, Avg generation throughput: 0.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 83.7% (APIServer pid=1) INFO 07-29 01:43:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 83.7% (APIServer pid=1) INFO: 127.0.0.1:58160 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-29 01:43:21 [loggers.py:259] Engine 000: Avg prompt throughput: 68.3 tokens/s, Avg generation throughput: 5.9 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 84.4% (APIServer pid=1) INFO 07-29 01:43:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 84.4% (APIServer pid=1) INFO: 127.0.0.1:55952 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-29 01:43:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 84.4% |
+
Qwen3.6-35B-A3B
| docker run --pull=always --rm -it --network host --shm-size=16g --ulimit memlock=-1 --ulimit stack=67108864 --runtime=nvidia --name=vllm -v $HOME/data/models/huggingface:/data/models/huggingface ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve Qwen/Qwen3.6-35B-A3B --enable-auto-tool-choice --tool-call-parser hermes |
단독 사용
| (APIServer pid=1) INFO 07-29 01:59:09 [loggers.py:259] Engine 000: Avg prompt throughput: 1250.0 tokens/s, Avg generation throughput: 31.8 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-29 01:59:19 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 60.6 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:55540 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-29 01:59:29 [loggers.py:259] Engine 000: Avg prompt throughput: 689.5 tokens/s, Avg generation throughput: 50.7 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-29 01:59:39 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 81.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 0.0% |
2개 창에서 2~3개씩 쌓고 함
| (APIServer pid=1) INFO: 127.0.0.1:58426 - "GET /v1/models HTTP/1.1" 200 OK (APIServer pid=1) INFO: 127.0.0.1:47306 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-29 02:01:59 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 71.1 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:36898 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-29 02:02:09 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 70.2 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-29 02:02:19 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 60.6 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:40588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-29 02:02:29 [loggers.py:259] Engine 000: Avg prompt throughput: 833.0 tokens/s, Avg generation throughput: 48.0 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:47322 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-29 02:02:39 [loggers.py:259] Engine 000: Avg prompt throughput: 35.4 tokens/s, Avg generation throughput: 78.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO: 127.0.0.1:46268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-29 02:02:49 [loggers.py:259] Engine 000: Avg prompt throughput: 930.4 tokens/s, Avg generation throughput: 55.1 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 0.0% (APIServer pid=1) INFO 07-29 02:02:59 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 67.0 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 0.0% |
+
2026.07.30
위와 좀 다른데 mxfp4 와 Q2 양자화 속도 차이. mxfp4가 좀 더 빠르긴 하다.
| $ ../../llama-b10145/llama-cli -m ./Qwen3.6-35B-A3B-MXFP4_MOE.gguf > 안녕? [ Prompt: 54.0 t/s | Generation: 43.2 t/s ] > 너에 대해서 설명해줘 [ Prompt: 37.9 t/s | Generation: 42.9 t/s ] > 파이썬으로 셀레니움을 통해 웹을 서칭하고 텍스트만 추출하고 makrdown 으로 변환후 md 파일과 pdf로 저장하는 기능을 구현해줘 [ Prompt: 111.9 t/s | Generation: 42.0 t/s ] $ ../../llama-b10145/llama-cli -m ./Qwen3.6-35B-A3B-UD-Q2_K_XL.gguf > 안녕? [ Prompt: 9.8 t/s | Generation: 38.3 t/s ] > 너에 대해서 설명해줘 [ Prompt: 41.7 t/s | Generation: 38.1 t/s ] > 파이썬으로 셀레니움을 통해 웹을 서칭하고 텍스트만 추출하고 makrdown 으로 변환후 md 파일과 pdf로 저장하는 기능을 구현해줘 [ Prompt: 86.9 t/s | Generation: 38.1 t/s ] |
| nvidia jetson agx thor / mig (0) | 2026.07.27 |
|---|---|
| nvidia jetson agx thor / 디버그 usb (0) | 2026.07.27 |
| nvidia-smi mig (0) | 2026.07.26 |
| nvidia jetson agx thor dev kit 설치 - USB boot (0) | 2026.07.24 |
| nvidia MIG(multiple instance GPU) - thor (0) | 2026.07.24 |