HF_HOME 환경 변수를 바꿔주면 된다고
리눅스에서는 기본값 ~/.cache/huggingface
| antigravity 2 (0) | 2026.05.28 |
|---|---|
| telegram bot api 로 기능 변경하기 (0) | 2026.05.25 |
| vllm 설치, 실행 실패 -> 사실상 포기 (0) | 2026.05.24 |
| antigravity gemini flash 할당량이 깃털 같구만? (0) | 2026.05.22 |
| openclaw agent 구성 관련 (0) | 2026.05.22 |
실행하라면 이렇게 하라고 하는데
| vllm serve google/gemma-4-E4B-it \ --max-model-len <n_of_tokens> # up to 131072 |
[링크 : https://docs.vllm.ai/projects/recipes/en/latest/Google/Gemma4.html]
아따.. 드럽게 크네. 그나저나 허깅페이스에서 바로 받으려나?
그리고 gguf가 양자화 되서 작은거였나. 기존에 내가 쓰던데 Q4_K_M 이라 4.7기가 정도 되었는데
model.safetensors는 16기가나 된다. 와우
[링크 : https://huggingface.co/google/gemma-4-E4B]
vllm : 1080 ti 라니 불량식품이잖아! 퉤!
| $ vllm serve google/gemma-4-E4B-it (APIServer pid=52696) INFO 05-24 22:20:19 [utils.py:306] (APIServer pid=52696) INFO 05-24 22:20:19 [utils.py:306] █ █ █▄ ▄█ (APIServer pid=52696) INFO 05-24 22:20:19 [utils.py:306] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.21.0 (APIServer pid=52696) INFO 05-24 22:20:19 [utils.py:306] █▄█▀ █ █ █ █ model google/gemma-4-E4B-it (APIServer pid=52696) INFO 05-24 22:20:19 [utils.py:306] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ (APIServer pid=52696) INFO 05-24 22:20:19 [utils.py:306] (APIServer pid=52696) INFO 05-24 22:20:19 [utils.py:240] non-default args: {'model_tag': 'google/gemma-4-E4B-it', 'model': 'google/gemma-4-E4B-it'} (APIServer pid=52696) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. config.json: 5.14kB [00:00, 17.7MB/s] processor_config.json: 1.69kB [00:00, 1.70MB/s] (APIServer pid=52696) INFO 05-24 22:20:29 [model.py:568] Resolved architecture: Gemma4ForConditionalGeneration (APIServer pid=52696) WARNING 05-24 22:20:29 [model.py:1982] Your device 'NVIDIA GeForce GTX 1080 Ti' (with compute capability 6.1) doesn't support torch.bfloat16. Falling back to torch.float16 for compatibility. (APIServer pid=52696) WARNING 05-24 22:20:29 [model.py:2035] Casting torch.bfloat16 to torch.float16. (APIServer pid=52696) INFO 05-24 22:20:29 [model.py:1697] Using max model len 131072 (APIServer pid=52696) INFO 05-24 22:20:29 [config.py:101] Gemma4 model has heterogeneous head dimensions (head_dim=256, global_head_dim=512). Forcing TRITON_ATTN backend to prevent mixed-backend numerical divergence. (APIServer pid=52696) INFO 05-24 22:20:29 [vllm.py:886] Asynchronous scheduling is enabled. (APIServer pid=52696) INFO 05-24 22:20:29 [kernel.py:212] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']) tokenizer_config.json: 2.10kB [00:00, 2.05MB/s] tokenizer.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 32.2M/32.2M [00:02<00:00, 14.8MB/s] chat_template.jinja: 17.3kB [00:00, 12.6MB/s] generation_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 208/208 [00:00<00:00, 1.68MB/s] (EngineCore pid=52767) INFO 05-24 22:21:24 [core.py:109] Initializing a V1 LLM engine (v0.21.0) with config: model='google/gemma-4-E4B-it', speculative_config=None, tokenizer='google/gemma-4-E4B-it', skip_t okenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size= 1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=Fal se, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_tr aces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_itera tion_details=False), seed=0, served_model_name=google/gemma-4-E4B-it, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_w ith_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm ::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_ cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'en coder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_as serts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups ': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 3 20, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 512, 'dynamic_s hapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=False, moe_backend='auto') (EngineCore pid=52767) /home/minimonk/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:371: UserWarning: Found GPU0 NVIDIA GeForce GTX 1080 Ti which is of compute capability (CC) 6.1. (EngineCore pid=52767) The following list shows the CCs this version of PyTorch was built for and the hardware CCs it supports: (EngineCore pid=52767) - 7.5 which supports hardware CC >=7.5,<8.0 (EngineCore pid=52767) - 8.0 which supports hardware CC >=8.0,<9.0 except {8.7} (EngineCore pid=52767) - 8.6 which supports hardware CC >=8.6,<9.0 except {8.7} (EngineCore pid=52767) - 9.0 which supports hardware CC >=9.0,<10.0 (EngineCore pid=52767) - 8.6 which supports hardware CC >=8.6,<9.0 except {8.7} [108/299] (EngineCore pid=52767) - 9.0 which supports hardware CC >=9.0,<10.0 (EngineCore pid=52767) - 10.0 which supports hardware CC >=10.0,<11.0 except {10.1} (EngineCore pid=52767) - 12.0 which supports hardware CC >=12.0,<13.0 (EngineCore pid=52767) Please follow the instructions at https://pytorch.org/get-started/locally/ to install a PyTorch release that supports one of these CUDA versions: 12.6 (EngineCore pid=52767) _warn_unsupported_code(d, device_cc, code_ccs) (EngineCore pid=52767) /home/minimonk/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:371: UserWarning: Found GPU1 NVIDIA GeForce GTX 1080 Ti which is of compute capability (CC) 6.1. (EngineCore pid=52767) The following list shows the CCs this version of PyTorch was built for and the hardware CCs it supports: (EngineCore pid=52767) - 7.5 which supports hardware CC >=7.5,<8.0 (EngineCore pid=52767) - 8.0 which supports hardware CC >=8.0,<9.0 except {8.7} (EngineCore pid=52767) - 8.6 which supports hardware CC >=8.6,<9.0 except {8.7} (EngineCore pid=52767) - 9.0 which supports hardware CC >=9.0,<10.0 (EngineCore pid=52767) - 10.0 which supports hardware CC >=10.0,<11.0 except {10.1} (EngineCore pid=52767) - 12.0 which supports hardware CC >=12.0,<13.0 (EngineCore pid=52767) Please follow the instructions at https://pytorch.org/get-started/locally/ to install a PyTorch release that supports one of these CUDA versions: 12.6 (EngineCore pid=52767) _warn_unsupported_code(d, device_cc, code_ccs) (EngineCore pid=52767) /home/minimonk/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:489: UserWarning: (EngineCore pid=52767) NVIDIA GeForce GTX 1080 Ti with CUDA capability sm_61 is not compatible with the current PyTorch installation. (EngineCore pid=52767) The current PyTorch install supports CUDA capabilities sm_75 sm_80 sm_86 sm_90 sm_100 sm_120. (EngineCore pid=52767) If you want to use the NVIDIA GeForce GTX 1080 Ti GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/ (EngineCore pid=52767) (EngineCore pid=52767) queued_call() (EngineCore pid=52767) INFO 05-24 22:21:30 [parallel_state.py:1410] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.40.238:47913 backend=nccl (EngineCore pid=52767) INFO 05-24 22:21:30 [parallel_state.py:1723] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A (EngineCore pid=52767) WARNING 05-24 22:21:31 [topk_topp_sampler.py:61] FlashInfer top-p/top-k sampling not supported on compute capability 6.1; falling back to PyTorch-native sampler. Set VLLM_USE_FLASHINF ER_SAMPLER=0 to silence. (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] EngineCore failed to start. (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] Traceback (most recent call last): (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1114, in run_engine_core (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] return func(*args, **kwargs) (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 880, in __init__ (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] super().__init__( (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 118, in __init__ (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] self.model_executor = executor_class(vllm_config) (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] return func(*args, **kwargs) (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 109, in __init__ (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] self._init_executor() (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/executor/uniproc_executor.py", line 60, in _init_executor (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] self.driver_worker.init_device() (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/worker_base.py", line 317, in init_device (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] self.worker.init_device() # type: ignore (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] return func(*args, **kwargs) (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 330, in init_device (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] self.model_runner = GPUModelRunnerV1(self.vllm_config, self.device) (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 629, in __init__ (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] self.input_batch = InputBatch( (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_input_batch.py", line 171, in __init__ (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] self.block_table = MultiGroupBlockTable( (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/block_table.py", line 267, in __init__ (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] self.block_tables = [ (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/block_table.py", line 268, in <listcomp> (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] BlockTable( (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/block_table.py", line 268, in <listcomp> [54/299] (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] BlockTable( (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/block_table.py", line 70, in __init__ (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] self.block_table = self._make_buffer( (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/block_table.py", line 218, in _make_buffer (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] return CpuGpuBuffer( (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/utils.py", line 120, in __init__ (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] self.gpu = torch.zeros_like(self.cpu, device=device) (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] Search for `cudaErrorNoKernelImageForDevice' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information. (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] For debugging consider passing CUDA_LAUNCH_BLOCKING=1 (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. (EngineCore pid=52767) ERROR 05-24 22:21:32 [core.py:1140] (EngineCore pid=52767) Process EngineCore: (EngineCore pid=52767) Traceback (most recent call last): (EngineCore pid=52767) File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap (EngineCore pid=52767) self.run() (EngineCore pid=52767) File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run (EngineCore pid=52767) self._target(*self._args, **self._kwargs) (EngineCore pid=52767) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1144, in run_engine_core (EngineCore pid=52767) raise e (EngineCore pid=52767) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1114, in run_engine_core (EngineCore pid=52767) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) (EngineCore pid=52767) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=52767) return func(*args, **kwargs) (EngineCore pid=52767) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 880, in __init__ (EngineCore pid=52767) super().__init__( (EngineCore pid=52767) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 118, in __init__ (EngineCore pid=52767) self.model_executor = executor_class(vllm_config) (EngineCore pid=52767) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=52767) return func(*args, **kwargs) (EngineCore pid=52767) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 109, in __init__ (EngineCore pid=52767) self._init_executor() (EngineCore pid=52767) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/executor/uniproc_executor.py", line 60, in _init_executor (EngineCore pid=52767) self.driver_worker.init_device() (EngineCore pid=52767) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/worker_base.py", line 317, in init_device (EngineCore pid=52767) self.worker.init_device() # type: ignore (EngineCore pid=52767) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=52767) return func(*args, **kwargs) (EngineCore pid=52767) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 330, in init_device (EngineCore pid=52767) self.model_runner = GPUModelRunnerV1(self.vllm_config, self.device) (EngineCore pid=52767) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 629, in __init__ (EngineCore pid=52767) self.input_batch = InputBatch( (EngineCore pid=52767) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_input_batch.py", line 171, in __init__ (EngineCore pid=52767) self.block_table = MultiGroupBlockTable( (EngineCore pid=52767) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/block_table.py", line 267, in __init__ (EngineCore pid=52767) self.block_tables = [ (EngineCore pid=52767) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/block_table.py", line 268, in <listcomp> (EngineCore pid=52767) BlockTable( (EngineCore pid=52767) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/block_table.py", line 70, in __init__ (EngineCore pid=52767) self.block_table = self._make_buffer( (EngineCore pid=52767) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/block_table.py", line 218, in _make_buffer (EngineCore pid=52767) return CpuGpuBuffer( (EngineCore pid=52767) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/utils.py", line 120, in __init__ (EngineCore pid=52767) self.gpu = torch.zeros_like(self.cpu, device=device) (EngineCore pid=52767) torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device (EngineCore pid=52767) Search for `cudaErrorNoKernelImageForDevice' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information. (EngineCore pid=52767) CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. (EngineCore pid=52767) For debugging consider passing CUDA_LAUNCH_BLOCKING=1 (EngineCore pid=52767) Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. (EngineCore pid=52767) [rank0]:[W524 22:21:32.199264489 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch. org/docs/stable/distributed.html#shutdown (function operator()) (APIServer pid=52696) Traceback (most recent call last): (APIServer pid=52696) File "/home/minimonk/.local/bin/vllm", line 8, in <module> (APIServer pid=52696) sys.exit(main()) (APIServer pid=52696) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/entrypoints/cli/main.py", line 92, in main (APIServer pid=52696) args.dispatch_function(args) (APIServer pid=52696) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/entrypoints/cli/serve.py", line 122, in cmd (APIServer pid=52696) uvloop.run(run_server(args)) (APIServer pid=52696) File "/home/minimonk/.local/lib/python3.10/site-packages/uvloop/__init__.py", line 69, in run (APIServer pid=52696) return loop.run_until_complete(wrapper()) (APIServer pid=52696) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete (APIServer pid=52696) File "/home/minimonk/.local/lib/python3.10/site-packages/uvloop/__init__.py", line 48, in wrapper (APIServer pid=52696) return await main (APIServer pid=52696) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 693, in run_server (APIServer pid=52696) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs) (APIServer pid=52696) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 707, in run_server_worker (APIServer pid=52696) async with build_async_engine_client( (APIServer pid=52696) File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__ (APIServer pid=52696) return await anext(self.gen) (APIServer pid=52696) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client (APIServer pid=52696) async with build_async_engine_client_from_engine_args( (APIServer pid=52696) File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__ (APIServer pid=52696) return await anext(self.gen) (APIServer pid=52696) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args (APIServer pid=52696) async_llm = AsyncLLM.from_vllm_config( (APIServer pid=52696) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config (APIServer pid=52696) return cls( (APIServer pid=52696) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 146, in __init__ (APIServer pid=52696) self.engine_core = EngineCoreClient.make_async_mp_client( (APIServer pid=52696) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (APIServer pid=52696) return func(*args, **kwargs) (APIServer pid=52696) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 130, in make_async_mp_client (APIServer pid=52696) return AsyncMPClient(*client_args) (APIServer pid=52696) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (APIServer pid=52696) return func(*args, **kwargs) (APIServer pid=52696) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 900, in __init__ (APIServer pid=52696) super().__init__( (APIServer pid=52696) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 535, in __init__ (APIServer pid=52696) with launch_core_engines( (APIServer pid=52696) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ (APIServer pid=52696) next(self.gen) (APIServer pid=52696) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 1128, in launch_core_engines (APIServer pid=52696) wait_for_engine_startup( (APIServer pid=52696) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 1187, in wait_for_engine_startup (APIServer pid=52696) raise RuntimeError( (APIServer pid=52696) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} |
파스칼은 아예 지원 하드웨어에서 빼버린건가?
[링크 : https://docs.vllm.ai/en/latest/features/quantization/]
fork로 이런것도 존재하는데, 아래 pascal-pkgs-ci로 대체 된다고
[링크 : https://github.com/cduk/vllm-pascal]
도커로 시도해야하나..
[링크 : https://github.com/sasha0552/pascal-pkgs-ci]
[링크 : https://github.com/vllm-project/vllm/issues/19542]
위의 도커를 불러오게 하면되려나? 볼륨은 로컬 캐싱에서 HF_HOME 으로 변경해주면 좋을듯
| docker run -itd --name gemma4 \ --ipc=host \ --network host \ --shm-size 16G \ --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:latest \ --model google/gemma-4-31B-it \ --tensor-parallel-size 2 \ --max-model-len 32768 \ --gpu-memory-utilization 0.90 \ --host 0.0.0.0 \ --port 8000 |
[링크 : https://docs.vllm.ai/projects/recipes/en/latest/Google/Gemma4.html#docker-deployment]
아래와 같이 도커에서 그래픽 카드를 인식하지 못하면
| docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]] |
nvidia-container-toolkit을 설치하고 도커를 재기동하면 된단다
| $ distribution=$(. /etc/os-release;echo $ID$VERSION_ID) && curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - && curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list $ sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit $ sudo systemctl restart docker |
[링크 : https://bluecolorsky.tistory.com/110]
| docker run -itd --name gemma4 \ --ipc=host \ --network host \ --gpus all \ -v /mnt/huggingface:/root/.cache/huggingface \ ghcr.io/sasha0552/vllm\ --model google/gemma-4-e4b-it \ --tensor-parallel-size 2 \ --max-model-len 131072\ --gpu-memory-utilization 0.90 \ --host 0.0.0.0 \ --port 8000 |
[링크 : https://github.com/sasha0552/pascal-pkgs-ci/pkgs/container/vllm]
에라모르겠다 ㅋㅋ
| $ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES fdd1ba582924 ghcr.io/sasha0552/vllm "python3 -m vllm.ent…" 42 seconds ago Exited (1) 26 seconds ago gemma4 |
[링크 : https://data-newbie.tistory.com/m/1012]
[링크 : https://coding-review.tistory.com/m/608]
+
2026.05.25
| docker run -itd \ --name gemma4 \ --ipc=host \ --network host \ --gpus all \ -v /mnt/huggingface:/root/.cache/huggingface \ ghcr.io/sasha0552/vllm |
문제없이 되는거 같으면서도
왜 qwen3-0.6B가 언급이 되지?
| INFO 05-25 05:00:09 [__init__.py:241] Automatically detected platform cuda. (APIServer pid=1) INFO 05-25 05:00:11 [api_server.py:1873] vLLM API server version 999.999.999 (APIServer pid=1) INFO 05-25 05:00:11 [utils.py:326] non-default args: {} (APIServer pid=1) INFO 05-25 05:00:19 [__init__.py:742] Resolved architecture: Qwen3ForCausalLM (APIServer pid=1) WARNING 05-25 05:00:19 [__init__.py:2828] Your device 'NVIDIA GeForce GTX 1080 Ti' (with compute capability 6.1) doesn't support torch.bfloat16. Falling back to torch.float16 for compatibility. (APIServer pid=1) WARNING 05-25 05:00:19 [__init__.py:2879] Casting torch.bfloat16 to torch.float16. (APIServer pid=1) INFO 05-25 05:00:19 [__init__.py:1774] Using max model len 40960 (APIServer pid=1) WARNING 05-25 05:00:19 [arg_utils.py:1806] Compute Capability < 8.0 is not supported by the V1 Engine. Falling back to V0. (APIServer pid=1) WARNING 05-25 05:00:19 [arg_utils.py:1580] Chunked prefill is enabled by default for models with max_model_len > 32K. Chunked prefill might not work with some features or models. If you encounter any issues, please disable by launching with --enable-chunked-prefill=False. (APIServer pid=1) INFO 05-25 05:00:20 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=2048. (APIServer pid=1) INFO 05-25 05:00:20 [api_server.py:295] Started engine process with PID 36 INFO 05-25 05:00:24 [__init__.py:241] Automatically detected platform cuda. INFO 05-25 05:00:25 [llm_engine.py:222] Initializing a V0 LLM engine (v999.999.999) with config: model='Qwen/Qwen3-0.6B', speculative_config=None, tokenizer='Qwen/Qwen3-0.6B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Qwen/Qwen3-0.6B, enable_prefix_caching=None, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":null,"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":0,"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"pass_config":{"enable_fusion":false,"enable_noop":false},"max_capture_size":256,"local_cache_dir":null}, use_cached_outputs=True, INFO 05-25 05:00:28 [cuda.py:374] Cannot use FlashAttention-2 backend for Volta and Turing GPUs. INFO 05-25 05:00:28 [cuda.py:419] Using XFormers backend. INFO 05-25 05:00:28 [parallel_state.py:1134] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0 INFO 05-25 05:00:28 [model_runner.py:1080] Starting to load model Qwen/Qwen3-0.6B... INFO 05-25 05:00:29 [weight_utils.py:296] Using model weights format ['*.safetensors'] INFO 05-25 05:00:29 [weight_utils.py:349] No model.safetensors.index.json found in remote. Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 3.47it/s] Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 3.47it/s] INFO 05-25 05:00:29 [default_loader.py:267] Loading weights took 0.32 seconds INFO 05-25 05:00:30 [model_runner.py:1112] Model loading took 1.1201 GiB and 1.275699 seconds INFO 05-25 05:00:31 [worker.py:296] Memory profiling takes 1.07 seconds INFO 05-25 05:00:31 [worker.py:296] the current vLLM instance can use total_gpu_memory (10.90GiB) x gpu_memory_utilization (0.90) = 9.81GiB INFO 05-25 05:00:31 [worker.py:296] model weights take 1.12GiB; non_torch_memory takes 0.04GiB; PyTorch activation peak memory takes 1.39GiB; the rest of the memory reserved for KV Cache is 7.26GiB. INFO 05-25 05:00:31 [executor_base.py:114] # cuda blocks: 4247, # CPU blocks: 2340 INFO 05-25 05:00:31 [executor_base.py:119] Maximum concurrency for 40960 tokens per request: 1.66x INFO 05-25 05:00:34 [model_runner.py:1383] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage. Capturing CUDA graph shapes: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:12<00:00, 2.85it/s] INFO 05-25 05:00:46 [model_runner.py:1535] Graph capturing finished in 12 secs, took 0.19 GiB INFO 05-25 05:00:46 [llm_engine.py:417] init engine (profile, create kv cache, warmup model) took 16.41 seconds (APIServer pid=1) INFO 05-25 05:00:46 [api_server.py:1679] Supported_tasks: ['generate'] (APIServer pid=1) WARNING 05-25 05:00:46 [__init__.py:1658] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. (APIServer pid=1) INFO 05-25 05:00:46 [serving_responses.py:124] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95} (APIServer pid=1) INFO 05-25 05:00:47 [serving_chat.py:135] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95} (APIServer pid=1) INFO 05-25 05:00:47 [serving_completion.py:77] Using default completion sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95} (APIServer pid=1) INFO 05-25 05:00:47 [api_server.py:1948] Starting vLLM API server 0 on http://0.0.0.0:8000 (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:36] Available routes are: (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /openapi.json, Methods: HEAD, GET (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /docs, Methods: HEAD, GET (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /docs/oauth2-redirect, Methods: HEAD, GET (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /redoc, Methods: HEAD, GET (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /health, Methods: GET (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /load, Methods: GET (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /ping, Methods: POST (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /ping, Methods: GET (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /tokenize, Methods: POST (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /detokenize, Methods: POST (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /v1/models, Methods: GET (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /version, Methods: GET (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /v1/responses, Methods: POST (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /v1/responses/{response_id}, Methods: GET (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /v1/responses/{response_id}/cancel, Methods: POST (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /v1/chat/completions, Methods: POST (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /v1/completions, Methods: POST (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /v1/embeddings, Methods: POST (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /pooling, Methods: POST (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /classify, Methods: POST (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /score, Methods: POST (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /v1/score, Methods: POST (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /v1/audio/transcriptions, Methods: POST (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /v1/audio/translations, Methods: POST (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /rerank, Methods: POST (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /v1/rerank, Methods: POST (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /v2/rerank, Methods: POST (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /scale_elastic_ep, Methods: POST (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /is_scaling_elastic_ep, Methods: POST (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /invocations, Methods: POST (APIServer pid=1) INFO 05-25 05:00:47 [launcher.py:44] Route: /metrics, Methods: GET (APIServer pid=1) INFO: Started server process [1] (APIServer pid=1) INFO: Waiting for application startup. (APIServer pid=1) INFO: Application startup complete. |
파스칼 P40 용으로 시도하는데 여전히 안된다. 아놔.. 포기!
[링크 : https://github.com/uaysk/vllm-pascal]
| $ vllm serve google/gemma-4-E4B-it (APIServer pid=65016) INFO 05-25 21:22:17 [utils.py:306] (APIServer pid=65016) INFO 05-25 21:22:17 [utils.py:306] █ █ █▄ ▄█ (APIServer pid=65016) INFO 05-25 21:22:17 [utils.py:306] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.21.0 (APIServer pid=65016) INFO 05-25 21:22:17 [utils.py:306] █▄█▀ █ █ █ █ model google/gemma-4-E4B-it (APIServer pid=65016) INFO 05-25 21:22:17 [utils.py:306] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ (APIServer pid=65016) INFO 05-25 21:22:17 [utils.py:306] (APIServer pid=65016) INFO 05-25 21:22:17 [utils.py:240] non-default args: {'model_tag': 'google/gemma-4-E4B-it', 'model': 'google/gemma-4-E4B-it'} (APIServer pid=65016) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. config.json: 5.14kB [00:00, 4.19MB/s] processor_config.json: 1.69kB [00:00, 6.85MB/s] (APIServer pid=65016) INFO 05-25 21:22:19 [model.py:568] Resolved architecture: Gemma4ForConditionalGeneration (APIServer pid=65016) WARNING 05-25 21:22:19 [model.py:1982] Your device 'NVIDIA GeForce GTX 1080 Ti' (with compute capability 6.1) doesn't support torch.bfloat16. Falling back to torch.float16 for compatibility. (APIServer pid=65016) WARNING 05-25 21:22:19 [model.py:2035] Casting torch.bfloat16 to torch.float16. (APIServer pid=65016) INFO 05-25 21:22:19 [model.py:1697] Using max model len 131072 (APIServer pid=65016) INFO 05-25 21:22:19 [config.py:101] Gemma4 model has heterogeneous head dimensions (head_dim=256, global_head_dim=512). Forcing TRITON_ATTN backend to prevent mixed-backend numerical divergence. (APIServer pid=65016) INFO 05-25 21:22:19 [vllm.py:886] Asynchronous scheduling is enabled. (APIServer pid=65016) INFO 05-25 21:22:19 [kernel.py:212] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']) tokenizer_config.json: 2.10kB [00:00, 2.10MB/s] tokenizer.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 32.2M/32.2M [00:02<00:00, 11.8MB/s] chat_template.jinja: 17.3kB [00:00, 13.0MB/s] generation_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 208/208 [00:00<00:00, 1.66MB/s] (EngineCore pid=65068) INFO 05-25 21:23:14 [core.py:109] Initializing a V1 LLM engine (v0.21.0) with config: model='google/gemma-4-E4B-it', speculative_config=None, tokenizer='google/gemma-4-E4B-it', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=google/gemma-4-E4B-it, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=False, moe_backend='auto') (EngineCore pid=65068) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. (EngineCore pid=65068) /home/minimonk/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:371: UserWarning: Found GPU0 NVIDIA GeForce GTX 1080 Ti which is of compute capability (CC) 6.1. (EngineCore pid=65068) The following list shows the CCs this version of PyTorch was built for and the hardware CCs it supports: (EngineCore pid=65068) - 7.5 which supports hardware CC >=7.5,<8.0 (EngineCore pid=65068) - 8.0 which supports hardware CC >=8.0,<9.0 except {8.7} (EngineCore pid=65068) - 8.6 which supports hardware CC >=8.6,<9.0 except {8.7} (EngineCore pid=65068) - 9.0 which supports hardware CC >=9.0,<10.0 (EngineCore pid=65068) - 10.0 which supports hardware CC >=10.0,<11.0 except {10.1} (EngineCore pid=65068) - 12.0 which supports hardware CC >=12.0,<13.0 (EngineCore pid=65068) Please follow the instructions at https://pytorch.org/get-started/locally/ to install a PyTorch release that supports one of these CUDA versions: 12.6 (EngineCore pid=65068) _warn_unsupported_code(d, device_cc, code_ccs) (EngineCore pid=65068) /home/minimonk/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:371: UserWarning: Found GPU1 NVIDIA GeForce GTX 1080 Ti which is of compute capability (CC) 6.1. (EngineCore pid=65068) The following list shows the CCs this version of PyTorch was built for and the hardware CCs it supports: (EngineCore pid=65068) - 7.5 which supports hardware CC >=7.5,<8.0 (EngineCore pid=65068) - 8.0 which supports hardware CC >=8.0,<9.0 except {8.7} (EngineCore pid=65068) - 8.6 which supports hardware CC >=8.6,<9.0 except {8.7} (EngineCore pid=65068) - 9.0 which supports hardware CC >=9.0,<10.0 (EngineCore pid=65068) - 10.0 which supports hardware CC >=10.0,<11.0 except {10.1} (EngineCore pid=65068) - 12.0 which supports hardware CC >=12.0,<13.0 (EngineCore pid=65068) Please follow the instructions at https://pytorch.org/get-started/locally/ to install a PyTorch release that supports one of these CUDA versions: 12.6 (EngineCore pid=65068) _warn_unsupported_code(d, device_cc, code_ccs) (EngineCore pid=65068) /home/minimonk/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:489: UserWarning: (EngineCore pid=65068) NVIDIA GeForce GTX 1080 Ti with CUDA capability sm_61 is not compatible with the current PyTorch installation. (EngineCore pid=65068) The current PyTorch install supports CUDA capabilities sm_75 sm_80 sm_86 sm_90 sm_100 sm_120. (EngineCore pid=65068) If you want to use the NVIDIA GeForce GTX 1080 Ti GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/ (EngineCore pid=65068) (EngineCore pid=65068) queued_call() (EngineCore pid=65068) INFO 05-25 21:23:20 [parallel_state.py:1410] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.40.238:39589 backend=nccl (EngineCore pid=65068) INFO 05-25 21:23:20 [parallel_state.py:1723] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A (EngineCore pid=65068) WARNING 05-25 21:23:21 [topk_topp_sampler.py:61] FlashInfer top-p/top-k sampling not supported on compute capability 6.1; falling back to PyTorch-native sampler. Set VLLM_USE_FLASHINFER_SAMPLER=0 to silence. (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] EngineCore failed to start. (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] Traceback (most recent call last): (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1114, in run_engine_core (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] return func(*args, **kwargs) (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 880, in __init__ (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] super().__init__( (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 118, in __init__ (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] self.model_executor = executor_class(vllm_config) (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] return func(*args, **kwargs) (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 109, in __init__ (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] self._init_executor() (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/executor/uniproc_executor.py", line 60, in _init_executor (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] self.driver_worker.init_device() (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/worker_base.py", line 317, in init_device (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] self.worker.init_device() # type: ignore (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] return func(*args, **kwargs) (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 330, in init_device (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] self.model_runner = GPUModelRunnerV1(self.vllm_config, self.device) (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 629, in __init__ (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] self.input_batch = InputBatch( (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_input_batch.py", line 171, in __init__ (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] self.block_table = MultiGroupBlockTable( (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/block_table.py", line 267, in __init__ (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] self.block_tables = [ (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/block_table.py", line 268, in <listcomp> (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] BlockTable( (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/block_table.py", line 70, in __init__ (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] self.block_table = self._make_buffer( (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/block_table.py", line 218, in _make_buffer (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] return CpuGpuBuffer( (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/utils.py", line 120, in __init__ (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] self.gpu = torch.zeros_like(self.cpu, device=device) (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] Search for `cudaErrorNoKernelImageForDevice' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information. (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] For debugging consider passing CUDA_LAUNCH_BLOCKING=1 (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. (EngineCore pid=65068) ERROR 05-25 21:23:21 [core.py:1140] (EngineCore pid=65068) Process EngineCore: (EngineCore pid=65068) Traceback (most recent call last): (EngineCore pid=65068) File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap (EngineCore pid=65068) self.run() (EngineCore pid=65068) File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run (EngineCore pid=65068) self._target(*self._args, **self._kwargs) (EngineCore pid=65068) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1144, in run_engine_core (EngineCore pid=65068) raise e (EngineCore pid=65068) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1114, in run_engine_core (EngineCore pid=65068) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) (EngineCore pid=65068) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=65068) return func(*args, **kwargs) (EngineCore pid=65068) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 880, in __init__ (EngineCore pid=65068) super().__init__( (EngineCore pid=65068) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 118, in __init__ (EngineCore pid=65068) self.model_executor = executor_class(vllm_config) (EngineCore pid=65068) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=65068) return func(*args, **kwargs) (EngineCore pid=65068) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 109, in __init__ (EngineCore pid=65068) self._init_executor() (EngineCore pid=65068) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/executor/uniproc_executor.py", line 60, in _init_executor (EngineCore pid=65068) self.driver_worker.init_device() (EngineCore pid=65068) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/worker_base.py", line 317, in init_device (EngineCore pid=65068) self.worker.init_device() # type: ignore (EngineCore pid=65068) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=65068) return func(*args, **kwargs) (EngineCore pid=65068) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 330, in init_device (EngineCore pid=65068) self.model_runner = GPUModelRunnerV1(self.vllm_config, self.device) (EngineCore pid=65068) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 629, in __init__ (EngineCore pid=65068) self.input_batch = InputBatch( (EngineCore pid=65068) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_input_batch.py", line 171, in __init__ (EngineCore pid=65068) self.block_table = MultiGroupBlockTable( (EngineCore pid=65068) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/block_table.py", line 267, in __init__ (EngineCore pid=65068) self.block_tables = [ (EngineCore pid=65068) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/block_table.py", line 268, in <listcomp> (EngineCore pid=65068) BlockTable( (EngineCore pid=65068) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/block_table.py", line 70, in __init__ (EngineCore pid=65068) self.block_table = self._make_buffer( (EngineCore pid=65068) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/worker/block_table.py", line 218, in _make_buffer (EngineCore pid=65068) return CpuGpuBuffer( (EngineCore pid=65068) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/utils.py", line 120, in __init__ (EngineCore pid=65068) self.gpu = torch.zeros_like(self.cpu, device=device) (EngineCore pid=65068) torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device (EngineCore pid=65068) Search for `cudaErrorNoKernelImageForDevice' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information. (EngineCore pid=65068) CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. (EngineCore pid=65068) For debugging consider passing CUDA_LAUNCH_BLOCKING=1 (EngineCore pid=65068) Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. (EngineCore pid=65068) [rank0]:[W525 21:23:22.743241268 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) (APIServer pid=65016) Traceback (most recent call last): (APIServer pid=65016) File "/home/minimonk/.local/bin/vllm", line 8, in <module> (APIServer pid=65016) sys.exit(main()) (APIServer pid=65016) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/entrypoints/cli/main.py", line 92, in main (APIServer pid=65016) args.dispatch_function(args) (APIServer pid=65016) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/entrypoints/cli/serve.py", line 122, in cmd (APIServer pid=65016) uvloop.run(run_server(args)) (APIServer pid=65016) File "/home/minimonk/.local/lib/python3.10/site-packages/uvloop/__init__.py", line 69, in run (APIServer pid=65016) return loop.run_until_complete(wrapper()) (APIServer pid=65016) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete (APIServer pid=65016) File "/home/minimonk/.local/lib/python3.10/site-packages/uvloop/__init__.py", line 48, in wrapper (APIServer pid=65016) return await main (APIServer pid=65016) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 693, in run_server (APIServer pid=65016) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs) (APIServer pid=65016) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 707, in run_server_worker (APIServer pid=65016) async with build_async_engine_client( (APIServer pid=65016) File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__ (APIServer pid=65016) return await anext(self.gen) (APIServer pid=65016) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client (APIServer pid=65016) async with build_async_engine_client_from_engine_args( (APIServer pid=65016) File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__ (APIServer pid=65016) return await anext(self.gen) (APIServer pid=65016) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args (APIServer pid=65016) async_llm = AsyncLLM.from_vllm_config( (APIServer pid=65016) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config (APIServer pid=65016) return cls( (APIServer pid=65016) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 146, in __init__ (APIServer pid=65016) self.engine_core = EngineCoreClient.make_async_mp_client( (APIServer pid=65016) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (APIServer pid=65016) return func(*args, **kwargs) (APIServer pid=65016) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 130, in make_async_mp_client (APIServer pid=65016) return AsyncMPClient(*client_args) (APIServer pid=65016) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (APIServer pid=65016) return func(*args, **kwargs) (APIServer pid=65016) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 900, in __init__ (APIServer pid=65016) super().__init__( (APIServer pid=65016) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 535, in __init__ (APIServer pid=65016) with launch_core_engines( (APIServer pid=65016) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ (APIServer pid=65016) next(self.gen) (APIServer pid=65016) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 1128, in launch_core_engines (APIServer pid=65016) wait_for_engine_startup( (APIServer pid=65016) File "/home/minimonk/.local/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 1187, in wait_for_engine_startup (APIServer pid=65016) raise RuntimeError( (APIServer pid=65016) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} |
| telegram bot api 로 기능 변경하기 (0) | 2026.05.25 |
|---|---|
| python huggingface 저장경로 변경하기 (0) | 2026.05.24 |
| antigravity gemini flash 할당량이 깃털 같구만? (0) | 2026.05.22 |
| openclaw agent 구성 관련 (0) | 2026.05.22 |
| mxfp4 (0) | 2026.05.22 |
한번 찾아보니 가격 3배 이상, 토큰 3배 증가
사실상 9배 증가라는 소문이 있네
[링크 : https://wikidocs.net/blog/@jaehong/13849/]
근데 순식간에(?) geminu 3.0 flash 를 날리고 gemini 3.5 flash + 3.1 pro 를 같은 할당량으로 묶는건 너무 개매너 아니냐 -_-
[링크 : https://ai.google.dev/gemini-api/docs/pricing?hl=ko]
+
2026.05.29
느낌상 질문 하나당 한칸 줄어드는 느낌..
| python huggingface 저장경로 변경하기 (0) | 2026.05.24 |
|---|---|
| vllm 설치, 실행 실패 -> 사실상 포기 (0) | 2026.05.24 |
| openclaw agent 구성 관련 (0) | 2026.05.22 |
| mxfp4 (0) | 2026.05.22 |
| 텔레그램 봇 - 메시지 관련 (0) | 2026.05.22 |
| vllm 설치, 실행 실패 -> 사실상 포기 (0) | 2026.05.24 |
|---|---|
| antigravity gemini flash 할당량이 깃털 같구만? (0) | 2026.05.22 |
| mxfp4 (0) | 2026.05.22 |
| 텔레그램 봇 - 메시지 관련 (0) | 2026.05.22 |
| openclaw 설치 telegram 신기하네? (0) | 2026.05.22 |
microscale(mx) fp4
| antigravity gemini flash 할당량이 깃털 같구만? (0) | 2026.05.22 |
|---|---|
| openclaw agent 구성 관련 (0) | 2026.05.22 |
| 텔레그램 봇 - 메시지 관련 (0) | 2026.05.22 |
| openclaw 설치 telegram 신기하네? (0) | 2026.05.22 |
| wan2.2 + comfyui 시도 (0) | 2026.05.21 |
inlinebutton - 메시지 하단에 생기는 버튼
replybutton - 사용자 입력창 하단에 생기는 버튼
[링크 : https://naimjae.tistory.com/11]
메시지 서식(markdown 등)
[링크 : https://naimjae.tistory.com/8]
[링크 : https://docs.python-telegram-bot.org/en/stable/telegram.menubuttoncommands.html]
[링크 : https://docs.python-telegram-bot.org/en/stable/telegram.html]
+
2026.05.25
Bot settings
| set_my_commands() | Used for setting the list of commands |
| delete_my_commands() | Used for deleting the list of commands |
| get_my_commands() | Used for obtaining the list of commands |
| get_my_default_administrator_rights() | Used for obtaining the default administrator rights for the bot |
| set_my_default_administrator_rights() | Used for setting the default administrator rights for the bot |
| get_chat_menu_button() | Used for obtaining the menu button of a private chat or the default menu button |
| set_chat_menu_button() | Used for setting the menu button of a private chat or the default menu button |
| set_my_description() | Used for setting the description of the bot |
| get_my_description() | Used for obtaining the description of the bot |
| set_my_short_description() | Used for setting the short description of the bot |
| get_my_short_description() | Used for obtaining the short description of the bot |
| set_my_name() | Used for setting the name of the bot |
| get_my_name() | Used for obtaining the name of the bot |
| set_my_profile_photo() | Used for setting the profile photo of the bot |
| remove_my_profile_photo() | Used for removing the profile photo of the bot |
[링크 : https://docs.python-telegram-bot.org/en/stable/telegram.bot.html]
| openclaw agent 구성 관련 (0) | 2026.05.22 |
|---|---|
| mxfp4 (0) | 2026.05.22 |
| openclaw 설치 telegram 신기하네? (0) | 2026.05.22 |
| wan2.2 + comfyui 시도 (0) | 2026.05.21 |
| wan2.2 + comfyui (0) | 2026.05.19 |
설치
| $ curl -fsSL https://openclaw.ai/install.sh | bash |
설정. llama-swap에 의해서 돌고 있는 모델 등록
| $ openclaw onboard --non-interactive \ --auth-choice custom-api-key \ --custom-base-url "http://127.0.0.1:8080/v1" \ --custom-model-id "gemma4-e4b" \ --custom-api-key "llama.cpp" \ --secret-input-mode plaintext \ --custom-compatibility openai \ --accept-risk $ openclaw gateway restart $ openclaw chat --message "안녕, 지금 어떤 모델이 실행 중이야?" |
/ 누르면 아래에 자동완성 처럼 뜨는데 신기하네
그리고 browse providers 처럼 먼가.. 내가 하던거랑은 좀 많이 다른(?) 기능들이 있었나 보다.
텔레그램 봇에 등록되는거라 내걸로 돌려도 메뉴가 계속 뜬다. 젠장!
서비스 종료.
| $ systemctl stop openclaw-gateway --user |
--user를 안하면 없는 서비스라 나온다. 신기하네
| $ systemctl status openclaw-gateway Unit openclaw-gateway.service could not be found. |
| mxfp4 (0) | 2026.05.22 |
|---|---|
| 텔레그램 봇 - 메시지 관련 (0) | 2026.05.22 |
| wan2.2 + comfyui 시도 (0) | 2026.05.21 |
| wan2.2 + comfyui (0) | 2026.05.19 |
| comfyui 실행 (0) | 2026.05.19 |
MCUNet 보다 보니 NAS 라는게 보여서 찾아봄
| Search NN model on an existing library e.g., ProxylessNAS, MnasNet |
[링크 : https://hanlab18.mit.edu/projects/tinyml/mcunet/assets/MCUNet-slides.pdf]
Neural Architecture Search
[링크 : https://doing-ai.tistory.com/entry/Neural-Architecture-Search-NAS-Hardware-Aware-NAS-ProxylessNAS]
| EfficientNAS 이후에 나온 더욱 효율적인 NAS를 위한 논문, ProxylessNAS다. TuNAS의 근간이 된 논문이기도 하며, 구체적으로 알아보자. |
[링크 : https://seokdonge.tistory.com/27] ProxylessNAS
[링크 : https://ech97.tistory.com/entry/MnasNet] MnasNet
TuNAS
[링크 : https://seokdonge.tistory.com/26]
[링크 : https://github.com/google-research/google-research/tree/master/tunas]
| STFPM 실행 (0) | 2026.06.01 |
|---|---|
| 딥러닝 학습 관련(epoch, loss) (0) | 2026.05.27 |
| MCUNet (0) | 2026.05.21 |
| STFPM - Student-Teacher Feature Pyramid Matching (0) | 2026.05.20 |
| EfficientAD (0) | 2026.05.20 |
STFPM -> PaSTe -> MCUNet 으로 찾아옴
[링크 : https://github.com/AMCO-UniPD/PaSTe]
MCU 에서 돌릴만큼 경량 네트워크
[링크 : https://cowkim-svd.tistory.com/7]
[링크 : https://github.com/mit-han-lab/mcunet]
| 딥러닝 학습 관련(epoch, loss) (0) | 2026.05.27 |
|---|---|
| NAS - Neural Architecture Search (0) | 2026.05.21 |
| STFPM - Student-Teacher Feature Pyramid Matching (0) | 2026.05.20 |
| EfficientAD (0) | 2026.05.20 |
| patchcore (0) | 2026.05.20 |
심심하면(?) VRAM 부족으로 터져서
| $ python3 main.py --listen 0.0.0.0 setup plugin alembic.autogenerate.schemas setup plugin alembic.autogenerate.tables setup plugin alembic.autogenerate.types setup plugin alembic.autogenerate.constraints setup plugin alembic.autogenerate.defaults setup plugin alembic.autogenerate.comments WARNING: You need pytorch with cu130 or higher to use optimized CUDA operations. Found comfy_kitchen backend cuda: {'available': True, 'disabled': True, 'unavailable_reason': None, 'capabilities': ['apply_rope', 'apply_rope1', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'scaled_mm_nvfp4']} Found comfy_kitchen backend eager: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['apply_rope', 'apply_rope1', 'dequantize_mxfp8', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'scaled_mm_mxfp8', 'scaled_mm_nvfp4']} Found comfy_kitchen backend triton: {'available': True, 'disabled': True, 'unavailable_reason': None, 'capabilities': ['apply_rope', 'apply_rope1', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8']} Checkpoint files will always be loaded safely. Total VRAM 11165 MB, total RAM 31755 MB pytorch version: 2.7.1+cu118 Set vram state to: NORMAL_VRAM Device: cuda:0 NVIDIA GeForce GTX 1080 Ti : cudaMallocAsync Using async weight offloading with 2 streams Enabled pinned memory 28579.0 Using pytorch attention Unsupported Pytorch detected. DynamicVRAM support requires Pytorch version 2.8 or later. Falling back to legacy ModelPatcher. VRAM estimates may be unreliable especially on Windows Python version: 3.10.12 (main, Mar 3 2026, 11:56:32) [GCC 11.4.0] ComfyUI version: 0.21.1 comfy-aimdo version: 0.3.0 comfy-kitchen version: 0.2.8 comfyui-frontend-package version: 1.43.18 comfyui-workflow-templates version: 0.9.77 comfyui-embedded-docs version: 0.5.0 comfy-kitchen version: 0.2.8 comfy-aimdo version: 0.3.0 [Prompt Server] web root: /home/minimonk/.local/lib/python3.10/site-packages/comfyui_frontend_package/static Asset seeder disabled Import times for custom nodes: 0.0 seconds: /mnt/Downloads/ComfyUI/custom_nodes/websocket_image_save.py Context impl SQLiteImpl. Will assume non-transactional DDL. Starting server To see the GUI go to: http://0.0.0.0:8188 got prompt Using pytorch attention in VAE Using pytorch attention in VAE VAE load device: cuda:0, offload device: cpu, dtype: torch.float32 Found quantization metadata version 1 Using MixedPrecisionOps for text encoder Requested to load WanTEModel loaded completely; 6419.48 MB loaded, full load: True CLIP/text encoder model load device: cpu, offload device: cpu, current: cpu, dtype: torch.float16 Requested to load WanVAE 0 models unloaded. loaded partially; 0.00 MB usable, 0.00 MB loaded, 484.00 MB offloaded, 45.57 MB buffer reserved, lowvram patches: 0 Found quantization metadata version 1 Detected mixed precision quantization Using mixed precision operations Native ops: , emulated ops: mxfp8, float8_e4m3fn, float8_e5m2, nvfp4 model weight dtype torch.float16, manual cast: torch.float32 model_type FLOW Requested to load WAN21 0 models unloaded. loaded partially; 0.00 MB usable, 0.00 MB loaded, 13636.09 MB offloaded, 885.22 MB buffer reserved, lowvram patches: 0 0%| | 0/10 [00:33<?, ?it/s] !!! Exception during processing !!! Allocation on device Traceback (most recent call last): File "/mnt/Downloads/ComfyUI/execution.py", line 535, in execute output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data) File "/mnt/Downloads/ComfyUI/execution.py", line 335, in get_output_data return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data) File "/mnt/Downloads/ComfyUI/execution.py", line 309, in _async_map_node_over_list await process_inputs(input_dict, i) File "/mnt/Downloads/ComfyUI/execution.py", line 297, in process_inputs result = f(**inputs) File "/mnt/Downloads/ComfyUI/nodes.py", line 1612, in sample return common_ksampler(model, noise_seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise, disable_noise=disable_noise, start_step=start_at_step, last_step=end_at_step, force_full_denoise=force_full_denoise) File "/mnt/Downloads/ComfyUI/nodes.py", line 1542, in common_ksampler samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, File "/mnt/Downloads/ComfyUI/comfy/sample.py", line 74, in sample samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed) File "/mnt/Downloads/ComfyUI/comfy/samplers.py", line 1180, in sample return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed) File "/mnt/Downloads/ComfyUI/comfy/samplers.py", line 1070, in sample return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed) File "/mnt/Downloads/ComfyUI/comfy/samplers.py", line 1052, in sample output = executor.execute(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed, latent_shapes=latent_shapes) File "/mnt/Downloads/ComfyUI/comfy/patcher_extension.py", line 112, in execute return self.original(*args, **kwargs) File "/mnt/Downloads/ComfyUI/comfy/samplers.py", line 995, in outer_sample output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed, latent_shapes=latent_shapes) File "/mnt/Downloads/ComfyUI/comfy/samplers.py", line 981, in inner_sample samples = executor.execute(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar) File "/mnt/Downloads/ComfyUI/comfy/patcher_extension.py", line 112, in execute return self.original(*args, **kwargs) File "/mnt/Downloads/ComfyUI/comfy/samplers.py", line 751, in sample samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options) File "/home/minimonk/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) File "/mnt/Downloads/ComfyUI/comfy/k_diffusion/sampling.py", line 205, in sample_euler denoised = model(x, sigma_hat * s_in, **extra_args) File "/mnt/Downloads/ComfyUI/comfy/samplers.py", line 400, in __call__ out = self.inner_model(x, sigma, model_options=model_options, seed=seed) File "/mnt/Downloads/ComfyUI/comfy/samplers.py", line 954, in __call__ return self.outer_predict_noise(*args, **kwargs) File "/mnt/Downloads/ComfyUI/comfy/samplers.py", line 961, in outer_predict_noise ).execute(x, timestep, model_options, seed) File "/mnt/Downloads/ComfyUI/comfy/patcher_extension.py", line 112, in execute return self.original(*args, **kwargs) File "/mnt/Downloads/ComfyUI/comfy/samplers.py", line 964, in predict_noise return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed) File "/mnt/Downloads/ComfyUI/comfy/samplers.py", line 380, in sampling_function out = calc_cond_batch(model, conds, x, timestep, model_options) File "/mnt/Downloads/ComfyUI/comfy/samplers.py", line 205, in calc_cond_batch return _calc_cond_batch_outer(model, conds, x_in, timestep, model_options) File "/mnt/Downloads/ComfyUI/comfy/samplers.py", line 213, in _calc_cond_batch_outer return executor.execute(model, conds, x_in, timestep, model_options) File "/mnt/Downloads/ComfyUI/comfy/patcher_extension.py", line 112, in execute return self.original(*args, **kwargs) File "/mnt/Downloads/ComfyUI/comfy/samplers.py", line 325, in _calc_cond_batch output = model.apply_model(input_x, timestep_, **c).chunk(batch_chunks) File "/mnt/Downloads/ComfyUI/comfy/model_base.py", line 182, in apply_model return comfy.patcher_extension.WrapperExecutor.new_class_executor( File "/mnt/Downloads/ComfyUI/comfy/patcher_extension.py", line 112, in execute return self.original(*args, **kwargs) File "/mnt/Downloads/ComfyUI/comfy/model_base.py", line 226, in _apply_model model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds) File "/home/minimonk/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/home/minimonk/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) File "/mnt/Downloads/ComfyUI/comfy/ldm/wan/model.py", line 644, in forward return comfy.patcher_extension.WrapperExecutor.new_class_executor( File "/mnt/Downloads/ComfyUI/comfy/patcher_extension.py", line 112, in execute return self.original(*args, **kwargs) File "/mnt/Downloads/ComfyUI/comfy/ldm/wan/model.py", line 664, in _forward return self.forward_orig(x, timestep, context, clip_fea=clip_fea, freqs=freqs, transformer_options=transformer_options, **kwargs)[:, :, :t, :h, :w] File "/mnt/Downloads/ComfyUI/comfy/ldm/wan/model.py", line 597, in forward_orig x = block(x, e=e0, freqs=freqs, context=context, context_img_len=context_img_len, transformer_options=transformer_options) File "/home/minimonk/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/home/minimonk/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) File "/mnt/Downloads/ComfyUI/comfy/ldm/wan/model.py", line 258, in forward y = self.ffn(torch.addcmul(repeat_e(e[3], x), self.norm2(x), 1 + repeat_e(e[4], x))) File "/home/minimonk/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/home/minimonk/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) File "/home/minimonk/.local/lib/python3.10/site-packages/torch/nn/modules/container.py", line 240, in forward input = module(input) File "/home/minimonk/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/home/minimonk/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) File "/home/minimonk/.local/lib/python3.10/site-packages/torch/nn/modules/activation.py", line 734, in forward return F.gelu(input, approximate=self.approximate) torch.OutOfMemoryError: Allocation on device Memory summary: |===========================================================================| | PyTorch CUDA memory summary, device ID 0 | |---------------------------------------------------------------------------| | CUDA OOMs: 0 | cudaMalloc retries: 0 | |===========================================================================| | Metric | Cur Usage | Peak Usage | Tot Alloc | Tot Freed | |---------------------------------------------------------------------------| | Allocated memory | 5675 MiB | 7030 MiB | 0 B | 0 B | | from large pool | 0 MiB | 0 MiB | 0 B | 0 B | | from small pool | 0 MiB | 0 MiB | 0 B | 0 B | |---------------------------------------------------------------------------| | Active memory | 5675 MiB | 7030 MiB | 0 B | 0 B | | from large pool | 0 MiB | 0 MiB | 0 B | 0 B | | from small pool | 0 MiB | 0 MiB | 0 B | 0 B | |---------------------------------------------------------------------------| | Requested memory | 0 B | 0 B | 0 B | 0 B | | from large pool | 0 B | 0 B | 0 B | 0 B | | from small pool | 0 B | 0 B | 0 B | 0 B | |---------------------------------------------------------------------------| | GPU reserved memory | 10784 MiB | 10784 MiB | 0 B | 0 B | | from large pool | 0 MiB | 0 MiB | 0 B | 0 B | | from small pool | 0 MiB | 0 MiB | 0 B | 0 B | |---------------------------------------------------------------------------| | Non-releasable memory | 0 B | 0 B | 0 B | 0 B | | from large pool | 0 B | 0 B | 0 B | 0 B | | from small pool | 0 B | 0 B | 0 B | 0 B | |---------------------------------------------------------------------------| | Allocations | 0 | 0 | 0 | 0 | | from large pool | 0 | 0 | 0 | 0 | | from small pool | 0 | 0 | 0 | 0 | |---------------------------------------------------------------------------| | Active allocs | 0 | 0 | 0 | 0 | | from large pool | 0 | 0 | 0 | 0 | | from small pool | 0 | 0 | 0 | 0 | |---------------------------------------------------------------------------| | GPU reserved segments | 0 | 0 | 0 | 0 | | from large pool | 0 | 0 | 0 | 0 | | from small pool | 0 | 0 | 0 | 0 | |---------------------------------------------------------------------------| | Non-releasable allocs | 0 | 0 | 0 | 0 | | from large pool | 0 | 0 | 0 | 0 | | from small pool | 0 | 0 | 0 | 0 | |---------------------------------------------------------------------------| | Oversize allocations | 0 | 0 | 0 | 0 | |---------------------------------------------------------------------------| | Oversize GPU segments | 0 | 0 | 0 | 0 | |===========================================================================| Got an OOM, unloading all loaded models. Prompt executed in 154.15 seconds |
둘 중에 하나 주면 된다는데 해도 터지고
| python3 main.py --listen 0.0.0.0 --lowvram python3 main.py --listen 0.0.0.0 --novram |
멀 하다가 꼬였는지 패키지 문제가 생겨서 다시 밀고 cuda 11.8에 맞춰서 재설치
| pip3 uninstall -y torch torchvision torchaudio xformers pip3 install torch==2.6.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 pip3 install xformers==0.0.29.post2 |
그래도 하려니 터져서 768x768 이었던게 왜 800x800이 되었는진 모르겠고 길이가 81로 되어있어서
일단은 512x512에 29 로 바꾸고 재시도
먼가 되는거 같긴한데.. offloaded가 엄청 크네.
| got prompt Requested to load WanVAE loaded completely; 4417.69 MB usable, 484.06 MB loaded, full load: True [MultiGPU Runtime] Using runtime device cuda:0 (comfy.sample.sample:ModelPatcher) Requested to load WAN21 loaded partially; 7397.49 MB usable, 7211.06 MB loaded, 6425.03 MB offloaded, 175.06 MB buffer reserved, lowvram patches: 0 0%| | 0/10 [00:00<?, ?it/s] 20%|████████▌ | 2/10 [09:15<37:03, 277.94s/it] |
1시간은 족히 넘은거 같은데 (37 띄워놓고 거짓말 쟁이!)
이제야 두번째 KSampler로 넘어갔다!!! 이예!!!
메모리 답이 안나올거 같아서 low_noise가 아닌 high_noise 에 해상도까지 낮추고 했는데
1시간 33분 16 동안 해서 29frame / 16fps 해서 대충 1.8초 짜리 똥을 생성해냄
| loaded completely; 883.22 MB usable, 484.06 MB loaded, full load: True Prompt executed in 01:33:16 |
이게 머야 ㅋㅋㅋ
+
왜 이번에는 webm이 아니라 webp일까?
그 와중에 49frame / 16fps 약 3초 만드는데 2시간 30분..
2시간 30분 짜리 똥이야!
| got prompt Found quantization metadata version 1 Detected mixed precision quantization Using mixed precision operations Native ops: , emulated ops: float8_e4m3fn, mxfp8, nvfp4, float8_e5m2 model weight dtype torch.float16, manual cast: torch.float32 model_type FLOW [MultiGPU Runtime] Using runtime device cuda:0 (comfy.sample.sample:ModelPatcher) Requested to load WAN21 loaded partially; 5509.86 MB usable, 5037.30 MB loaded, 8598.79 MB offloaded, 472.56 MB buffer reserved, lowvram patches: 0 100%|████████████████████████████████████████| 10/10 [1:14:32<00:00, 447.24s/it] [MultiGPU Runtime] Using runtime device cuda:0 (comfy.sample.sample:ModelPatcher) Requested to load WAN21 loaded partially; 5495.86 MB usable, 5019.61 MB loaded, 8616.47 MB offloaded, 472.56 MB buffer reserved, lowvram patches: 0 0%| | 0/10 [00:00<?, ?it/s]100%|████████████████████████████████████████| 10/10 [1:14:33<00:00, 447.32s/it] Requested to load WanVAE Unloaded partially: 2187.40 MB freed, 2832.21 MB remains loaded, 472.56 MB buffer reserved, lowvram patches: 0 loaded completely; 545.46 MB usable, 484.06 MB loaded, full load: True Prompt executed in 02:30:07 |
| 텔레그램 봇 - 메시지 관련 (0) | 2026.05.22 |
|---|---|
| openclaw 설치 telegram 신기하네? (0) | 2026.05.22 |
| wan2.2 + comfyui (0) | 2026.05.19 |
| comfyui 실행 (0) | 2026.05.19 |
| llama.cpp kv quantization (0) | 2026.05.19 |
|
