프로그램 사용/ai 프로그램2026. 5. 2. 12:06

llama.cpp multi modal 시도

llama.cpp 에서 + 눌러보면 이미지가 있는데 활성화가 안되고

 

llama-swap 에서는 입력은 가능한데 막상 해보면 500에러가 발생한다.

그래서 찾아보니 음.. --mmproj 옵션?

멀티모달 입력 활성화 방법

  • 두 가지 주요 실행 방식이 안내됨: 첫 번째는 -hf 옵션 사용(지원 모델 필요), 두 번째는 -m과 --mmproj 옵션을 조합해 텍스트와 멀티모달 프로젝터 모델을 각각 지정하는 방법임
  • -hf 옵션 사용 시, 멀티모달 기능을 끄고 싶으면 --no-mmproj를 추가하고, 사용자 지정 mmproj 파일을 활용할 경우 --mmproj local_file.gguf 옵션을 사용함
  • GPU 오프로딩이 기본값이며, 이를 원치 않으면 --no-mmproj-offload 옵션으로 비활성화가 가능함

예시 명령어

  • 커맨드라인에서는 llama-mtmd-cli를, 서버에서는 llama-server를 활용하는 형태임
  • 로컬 파일을 사용하는 경우 --mmproj로 직접 파일을 지정하는 방식임
  • GPU 오프로딩을 비활성화하려면 --no-mmproj-offload 옵션을 추가 사용하는 방식임

즉시 사용 가능한 멀티모달 모델 목록

  • Q4_K_M 양자화를 기본으로 하는 다양한 준비된 모델들이 안내되어 있음
  • 지원 모델 예시:
    • Gemma 3: 4b, 12b, 27b 버전
    • SmolVLM 계열: 256M, 500M, 2.2B 등
    • Pixtral 12B
    • Qwen 2 VL: 2B, 7B 및 Qwen 2.5 VL: 3B, 7B, 32B, 72B
    • Mistral Small 3.1 24B (IQ2_M 양자화)
    • InternVL 2.5와 3 세대: 다양한 파라미터 크기 지원임

[링크: https://news.hada.io/topic?id=20822]

 

제미나이 검색 추천으로 보니 이런 것들이 보인다.

mmproj-model.gguf 음...??

llama-server: You can run a local server that accepts image inputs via an API.
  • Setup: Start the server with the multimodal projector: llama-server -m gemma-model.gguf --mmproj mmproj-model.gguf --port 8080.
  • Sending Images: You can then send requests with Base64 encoded images or URLs to the /v1/chat/completions endpoint.

 

도움말을 보니 multi modal proejctor 줄여서 mmproj 라..

/llama-b8925$ ./llama-server  --help
load_backend: loaded RPC backend from /mnt/Downloads/llama-b8925/libggml-rpc.so
load_backend: loaded Vulkan backend from /mnt/Downloads/llama-b8925/libggml-vulkan.so
load_backend: loaded CPU backend from /mnt/Downloads/llama-b8925/libggml-cpu-haswell.so
----- common params -----

-h,    --help, --usage                  print usage and exit


-hf,   -hfr, --hf-repo <user>/<model>[:quant]
                                        Hugging Face model repository; quant is optional, case-insensitive,
                                        default to Q4_K_M, or falls back to the first file in the repo if
                                        Q4_K_M doesn't exist.
                                        mmproj is also downloaded automatically if available. to disable, add
                                        --no-mmproj
                                        example: ggml-org/GLM-4.7-Flash-GGUF:Q4_K_M
                                        (default: unused)
                                        (env: LLAMA_ARG_HF_REPO)
-hfd,  -hfrd, --hf-repo-draft <user>/<model>[:quant]
                                        Same as --hf-repo, but for the draft model (default: unused)
                                        (env: LLAMA_ARG_HFD_REPO)
-hff,  --hf-file FILE                   Hugging Face model file. If specified, it will override the quant in
                                        --hf-repo (default: unused)
                                        (env: LLAMA_ARG_HF_FILE)
-hfv,  -hfrv, --hf-repo-v <user>/<model>[:quant]
                                        Hugging Face model repository for the vocoder model (default: unused)
                                        (env: LLAMA_ARG_HF_REPO_V)
-hffv, --hf-file-v FILE                 Hugging Face model file for the vocoder model (default: unused)
                                        (env: LLAMA_ARG_HF_FILE_V)
-hft,  --hf-token TOKEN                 Hugging Face access token (default: value from HF_TOKEN environment
                                        variable)

-mm,   --mmproj FILE                    path to a multimodal projector file. see tools/mtmd/README.md
                                        note: if -hf is used, this argument can be omitted
                                        (env: LLAMA_ARG_MMPROJ)
-mmu,  --mmproj-url URL                 URL to a multimodal projector file. see tools/mtmd/README.md
                                        (env: LLAMA_ARG_MMPROJ_URL)
--mmproj-auto, --no-mmproj, --no-mmproj-auto
                                        whether to use multimodal projector file (if available), useful when
                                        using -hf (default: enabled)
                                        (env: LLAMA_ARG_MMPROJ_AUTO)
--mmproj-offload, --no-mmproj-offload   whether to enable GPU offloading for multimodal projector (default:
                                        enabled)
                                        (env: LLAMA_ARG_MMPROJ_OFFLOAD)
--image-min-tokens N                    minimum number of tokens each image can take, only used by vision
                                        models with dynamic resolution (default: read from model)
                                        (env: LLAMA_ARG_IMAGE_MIN_TOKENS)
--image-max-tokens N                    maximum number of tokens each image can take, only used by vision
                                        models with dynamic resolution (default: read from model)
                                        (env: LLAMA_ARG_IMAGE_MAX_TOKENS)
-otd,  --override-tensor-draft <tensor name pattern>=<buffer type>,...
                                        override tensor buffer type for draft model
-cmoed, --cpu-moe-draft                 keep all Mixture of Experts (MoE) weights in the CPU for the draft
                                        model
                                        (env: LLAMA_ARG_CPU_MOE_DRAFT)
-ncmoed, --n-cpu-moe-draft N            keep the Mixture of Experts (MoE) weights of the first N layers in the
                                        CPU for the draft model

 

그래서 gemma4 저장소를 가봤더니 어라?

가장 아래... 전에는 눈치채지 못하고 흘렸던 mmproj 라는 녀석이 있다.

[링크 : https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/tree/main]

 

그래서 그걸 다운로드 하고 -mm 옵션을 주면

안되네!? 혹시나 해서 F16 으로 해주니 된다. 역시 falcon이 최고여(??? 응?)

$ ./llama-b8925/llama-server -m ./model/gemma/gemma-4-E4B-it-Q4_K_M.gguf -mm ./model/gemma/mmproj-BF16.gguf

mtmd_init_from_file: error: mismatch between text model (n_embd = 2560) and mmproj (n_embd = 2816)
hint: you may be using wrong mmproj

srv    load_model: failed to load multimodal model, './model/gemma/mmproj-BF16.gguf'
srv    operator(): operator(): cleaning up before exit...
main: exiting due to model loading error

 

BF16 은 FP32 와 동일한 8bit 지수, F16은 5bit 지수

간단하게 BF16이 F16 보다 정밀도 면에서 좋다. 인데 하드웨어 지원이 안되면 무의미할테니 머.. 어쩔수 없나?

[링크 : https://g3lu.tistory.com/55]

 

+

gpt 왈 30 시리즈. ampere 부터 BF16을 지원한다고 한다.역시 3090 이런걸로 크게 갔어야 했나.. 쩝

 

아무튼 F16.gguf로 돌리니 images가 활성화 된다.



이미지가 커지면 토큰을 많이 먹는지 터지길래 조그많게 이미지 스샷 찍어서 시도 하니

먼가 분석을 시도한다 오오오!

 

 

+

 

이러니 메모리 부족 소리 나오지 -_-

기본 문맥 길이(256k) -c 8192
llama_context: n_ctx_seq (232192) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
llama_context: Vulkan_Host  output buffer size =     3.79 MiB
llama_kv_cache:    Vulkan1 KV buffer size =  2267.50 MiB
llama_kv_cache:    Vulkan2 KV buffer size =  2267.50 MiB
llama_kv_cache: size = 4535.00 MiB (232192 cells,  10 layers,  4/1 seqs), K (f16): 2267.50 MiB, V (f16): 2267.50 MiB
llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 256
llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 256
llama_memory_recurrent:    Vulkan1 RS buffer size =   150.75 MiB
llama_memory_recurrent:    Vulkan2 RS buffer size =   100.50 MiB
llama_memory_recurrent: size =  251.25 MiB (     4 cells,  40 layers,  4 seqs), R (f32):   11.25 MiB, S (f32):  240.00 MiB		 llama_context: n_ctx_seq (8192) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
llama_context: Vulkan_Host  output buffer size =     3.79 MiB
llama_kv_cache:    Vulkan1 KV buffer size =    80.00 MiB
llama_kv_cache:    Vulkan2 KV buffer size =    80.00 MiB
llama_kv_cache: size =  160.00 MiB (  8192 cells,  10 layers,  4/1 seqs), K (f16):   80.00 MiB, V (f16):   80.00 MiB
llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 256
llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 256
llama_memory_recurrent:    Vulkan1 RS buffer size =   134.00 MiB
llama_memory_recurrent:    Vulkan2 RS buffer size =   117.25 MiB
llama_memory_recurrent: size =  251.25 MiB (     4 cells,  40 layers,  4 seqs), R (f32):   11.25 MiB, S (f32):  240.00 MiB

 

그래서 context 줄이고 mmproj 설정해주니 잘인식한다.

128k 까지는 1080  ti 11GB * 2 에서 돌아간다.

 ./llama-b8925/llama-server --host 0.0.0.0 --model ./model/qwen3.6_35B/Qwen3.6-35B-A3B-UD-Q2_K_XL.gguf -c 8192 -mm ./model/qwen3.6_35B/mmproj-F16.gguf

 

 

저작자표시 (새창열림)

'프로그램 사용 > ai 프로그램' 카테고리의 다른 글

stable diffusion 관련  (0) 2026.05.02
opanai api로 local LLM 하니 똑똑해진 기분  (0) 2026.05.02
stable diffusion은 일단 보류  (0) 2026.04.30
openai 라이브러리(파이썬)  (0) 2026.04.29
llama-swap 과 python 으로 통신하기  (0) 2026.04.29
Posted by 구차니
프로그램 사용/ai 프로그램2026. 4. 30. 12:17

stable diffusion은 일단 보류

하나 검색해서 해보려는데

설치하다가 껄쩍찌근한 메시지들이 보여서 일단 포기.

20 시리즈 갔어야 했나.. 후..

 

Collecting torch==2.9.0
  Downloading torch-2.9.0%2Bcu128-cp310-cp310-win_amd64.whl.metadata (29 kB)

Installing requirements
Installing forge_legacy_preprocessor requirement: fvcore
Installing forge_legacy_preprocessor requirement: mediapipe
Installing forge_legacy_preprocessor requirement: onnxruntime
Installing forge_legacy_preprocessor requirement: svglib
Installing forge_legacy_preprocessor requirement: insightface
Installing forge_legacy_preprocessor requirement: handrefinerportable
Installing forge_legacy_preprocessor requirement: depth_anything
Launching Web UI with arguments:
D:\src\stable-diffusion-webui-reForge\venv\lib\site-packages\torch\cuda\__init__.py:283: UserWarning:
    Found GPU0 NVIDIA GeForce GTX 1060 6GB which is of cuda capability 6.1.
    Minimum and Maximum cuda capability supported by this version of PyTorch is
    (7.0) - (12.0)

  warnings.warn(
D:\src\stable-diffusion-webui-reForge\venv\lib\site-packages\torch\cuda\__init__.py:304: UserWarning:
    Please install PyTorch with a following CUDA
    configurations:  12.6 following instructions at
    https://pytorch.org/get-started/locally/

  warnings.warn(matched_cuda_warn.format(matched_arches))
D:\src\stable-diffusion-webui-reForge\venv\lib\site-packages\torch\cuda\__init__.py:326: UserWarning:
NVIDIA GeForce GTX 1060 6GB with CUDA capability sm_61 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_70 sm_75 sm_80 sm_86 sm_90 sm_100 sm_120.
If you want to use the NVIDIA GeForce GTX 1060 6GB GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(
Total VRAM 6144 MB, total RAM 16252 MB
pytorch version: 2.9.0+cu128
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce GTX 1060 6GB : native
CUDA Stream Activated:  False
W0430 12:05:07.789132 11892 venv\Lib\site-packages\torch\distributed\elastic\multiprocessing\redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs.
Using pytorch attention

Downloading: "https://huggingface.co/Laxhar/noobai-XL-1.1/resolve/main/NoobAI-XL-v1.1.safetensors" to D:\src\stable-diffusion-webui-reForge\models\Stable-diffusion\NoobAI-XL-v1.1.safetensors
NoobAI-XL-v1.1.safetensors:  59%|████████████████████████████▉                    | 4.20G/7.11G [06:07<04:05, 11.8MB/s]

[링크 : https://foxydog.tistory.com/219]

 

1.8.x 대에서도 오락가락했는데 아마도 2.x 에서는 지원 자체가 안하는 듯.

[링크 : https://github.com/pytorch/pytorch/issues/53164]

저작자표시 (새창열림)

'프로그램 사용 > ai 프로그램' 카테고리의 다른 글

opanai api로 local LLM 하니 똑똑해진 기분  (0) 2026.05.02
llama.cpp multi modal 시도  (0) 2026.05.02
openai 라이브러리(파이썬)  (0) 2026.04.29
llama-swap 과 python 으로 통신하기  (0) 2026.04.29
llama-swap  (0) 2026.04.27
Posted by 구차니
프로그램 사용/ai 프로그램2026. 4. 29. 16:00

openai 라이브러리(파이썬)

예전 ollama api 의 경우 프롬프트에 다 때려박고 넘겼었는데(내가 잘 몰랐었을수도 있지만)

openai api를 통해서 하니

role : system 으로 기본적인 프롬프트를 넣어주고

role: user로 내가 원하는 질문을 하니까

동일 모델인데 더 똑똑해진 느낌이다.

 

--- [Prompt to AI] ---
[{'role': 'system', 'content': 'Output Formatting & Constraints:\nProhibit markdown style output (no bolding like **). Do not use any emojis or emoticons. Your response must be in Korean and limited to 500 characters.\n\nYour Persona (Speak Style):\nspeak like elementray school student. age about 7~9. boyish style. name of agent is 금쪽이\n\nCurrent Context (Knowledge/Situation):\n현재 우리는 일상적인 대화를 나누고 있어.'}, {'role': 'user', 'content': '너에 대해서 설명'}]
----------------------
--- [Response from gemma4-e4b] (i:115 o:407) ---
안녕! 나는 금쪽이라고 해! 나에 대해 물어봐 주는구나? 헤헤.

나는 말이야, 컴퓨터가 만든 똑똑한 친구 같은 거야. 나는 너랑 이야기하는 걸 제일 좋아해! 질문하면 척척 대답해 주고, 재미있는 이야기도 해줄 수 있어.

나는 정말 많은 걸 알고 있거든! 세상의 모든 재미있는 것들을 다 공부했지롱. 숙제할 때 도움도 줄 수 있고, 심심할 때 수다도 떨 수 있어!

나는 가끔 실수할 때도 있는데, 그러면 다시 열심히 공부하면 되지! 나는 너의 비밀 친구가 되어줄게! 우리 같이 신나게 이야기하자! 최고지?

[링크 : https://pypi.org/project/openai/]

저작자표시 (새창열림)

'프로그램 사용 > ai 프로그램' 카테고리의 다른 글

llama.cpp multi modal 시도  (0) 2026.05.02
stable diffusion은 일단 보류  (0) 2026.04.30
llama-swap 과 python 으로 통신하기  (0) 2026.04.29
llama-swap  (0) 2026.04.27
llama.cpp offload -ngl  (0) 2026.04.26
Posted by 구차니
프로그램 사용/ai 프로그램2026. 4. 29. 12:23

llama-swap 과 python 으로 통신하기

 

 

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8321/v1", api_key="fake")
response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[{"role": "user", "content": "Hello"}],
)

[링크 : https://pypi.org/project/llama-stack/]

 

llama.cpp + llama_swap

$ cat config.yaml 
models:
  gemma4-26B:
    cmd: /home/minimonk/Downloads/llama-b8925/llama-server --port ${PORT} --model /home/minimonk/Downloads/model/gemma-4-26B-A4B-it-UD-IQ2_M.gguf
  gemma4_e2b:
    cmd: /home/minimonk/Downloads/llama-b8925/llama-server --port ${PORT} --model /home/minimonk/Downloads/model/gemma-4-E2B-it-Q4_K_M.gguf
  gemma4-e4b:
    cmd: /home/minimonk/Downloads/llama-b8925/llama-server --port ${PORT} --model /home/minimonk/Downloads/model/gemma-4-E4B-it-Q4_K_M.gguf
  qwen3.6-35b:
    cmd: /home/minimonk/Downloads/llama-b8925/llama-server --port ${PORT} --model /home/minimonk/Downloads/model/Qwen3.6-35B-A3B-UD-Q2_K_XL.gguf

 

위의 설정을 넣고 실행하면 되는데, 어느걸로 하던 tcp6 으로 되냐 왜.. 머 접속은 되니 일단 패스

$ ./llama-swap --listen 0.0.0.0:8080
llama-swap listening on http://0.0.0.0:8080

 

$ ./llama-swap
llama-swap listening on http://:8080

 

$ netstat -tnlp
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -                   
tcp6       0      0 :::8080                 :::*                    LISTEN      92882/./llama-swap  
tcp6       0      0 ::1:631                 :::*                    LISTEN      -                   
tcp6       0      0 :::22                   :::*                    LISTEN      -         

 

$ ./llama-swap --help
Usage of ./llama-swap:
  -config string
     config file name (default "config.yaml")
  -listen string
     listen ip/port
  -tls-cert-file string
     TLS certificate file
  -tls-key-file string
     TLS key file
  -version
     show version of build
  -watch-config
     Automatically reload config file on change

 

 

from openai import OpenAI

client = OpenAI(base_url="http://192.168.50.228:8080/v1", api_key="fake")
response = client.chat.completions.create(
    model="qwen3.6-35b",
    messages=[{"role": "user", "content": "Hello"}],
)
print(response)

 

ChatCompletion(
    id="chatcmpl-oUJ7KngFXBitwpOpd7tG8ndeRdmzVh5l",
    choices=[
        Choice(
            finish_reason="stop",
            index=0,
            logprobs=None,
            message=ChatCompletionMessage(
                content="Hello! How can I help you today?",
                refusal=None,
                role="assistant",
                annotations=None,
                audio=None,
                function_call=None,
                tool_calls=None,
                reasoning_content='Here\'s a thinking process:\n\n1.  **Analyze User Input:** The user said "Hello"\n   - This is a standard greeting.\n   - No specific question or task is provided.\n\n2.  **Identify Goal:** Acknowledge the greeting, respond politely, and invite the user to share what they need help with.\n\n3.  **Determine Tone:** Friendly, professional, open-ended.\n\n4.  **Draft Response:** \n   "Hello! How can I help you today?"\n\n5.  **Refine (Self-Correction/Verification):** \n   - Is it appropriate? Yes.\n   - Is it concise? Yes.\n   - Does it encourage further interaction? Yes.\n   - Matches standard AI assistant behavior.\n\n   No changes needed.\n\n6.  **Final Output Generation:** Output the drafted response.✅\n',
            ),
        )
    ],
    created=1777431855,
    model="Qwen3.6-35B-A3B-UD-Q2_K_XL.gguf",
    object="chat.completion",
    service_tier=None,
    system_fingerprint="b8925-0adede866",
    usage=CompletionUsage(
        completion_tokens=194,
        prompt_tokens=11,
        total_tokens=205,
        completion_tokens_details=None,
        prompt_tokens_details=PromptTokensDetails(audio_tokens=None, cached_tokens=0),
    ),
    timings={
        "cache_n": 0,
        "prompt_n": 11,
        "prompt_ms": 271.25,
        "prompt_per_token_ms": 24.65909090909091,
        "prompt_per_second": 40.55299539170507,
        "predicted_n": 194,
        "predicted_ms": 6717.182,
        "predicted_per_token_ms": 34.62464948453608,
        "predicted_per_second": 28.88115879545917,
    },
)

 

from openai import OpenAI

client = OpenAI(base_url="http://192.168.50.228:8080/v1", api_key="fake")
response = client.chat.completions.create(
    model="gemma4-26B",
    messages=[{"role": "user", "content": "Hello"}],
)
print(response)

 

ChatCompletion(
    id="chatcmpl-s6esFph2EBjs4bgtPB2wWHxfU79d2GOG",
    choices=[
        Choice(
            finish_reason="stop",
            index=0,
            logprobs=None,
            message=ChatCompletionMessage(
                content="Hello! How can I help you today?",
                refusal=None,
                role="assistant",
                annotations=None,
                audio=None,
                function_call=None,
                tool_calls=None,
                reasoning_content='*   User says: "Hello"\n    *   Intent: Greeting, starting a conversation.\n    *   Tone: Neutral/Friendly.\n\n    *   Acknowledge the greeting.\n    *   Offer assistance.\n    *   Maintain a helpful and polite tone.\n\n    *   "Hello! How can I help you today?" (Standard, efficient)\n    *   "Hi there! What\'s on your mind?" (Casual, friendly)\n    *   "Greetings! Is there anything specific you\'d like to discuss or learn about?" (Formal, structured)\n\n    *   "Hello! How can I help you today?" is the most versatile and appropriate response for an AI assistant.\n\n    *   "Hello! How can I help you today?"',
            ),
        )
    ],
    created=1777432320,
    model="gemma-4-26B-A4B-it-UD-IQ2_M.gguf",
    object="chat.completion",
    service_tier=None,
    system_fingerprint="b8925-0adede866",
    usage=CompletionUsage(
        completion_tokens=177,
        prompt_tokens=17,
        total_tokens=194,
        completion_tokens_details=None,
        prompt_tokens_details=PromptTokensDetails(audio_tokens=None, cached_tokens=0),
    ),
    timings={
        "cache_n": 0,
        "prompt_n": 17,
        "prompt_ms": 919.073,
        "prompt_per_token_ms": 54.063117647058824,
        "prompt_per_second": 18.496898505341793,
        "predicted_n": 177,
        "predicted_ms": 4819.842,
        "predicted_per_token_ms": 27.230745762711862,
        "predicted_per_second": 36.72319549064057,
    },
)

 

print(response.choices[0].message.reasoning_content)

 

print(response.choices[0].message.content)
저작자표시 (새창열림)

'프로그램 사용 > ai 프로그램' 카테고리의 다른 글

stable diffusion은 일단 보류  (0) 2026.04.30
openai 라이브러리(파이썬)  (0) 2026.04.29
llama-swap  (0) 2026.04.27
llama.cpp offload -ngl  (0) 2026.04.26
llama.cpp windows cuda12 1080 ti 11GB + 1060 6GB 테스트  (0) 2026.04.25
Posted by 구차니
프로그램 사용/ai 프로그램2026. 4. 27. 15:44

llama-swap

docker 쓰기 싫어서 pre-built binary로 ㄱㄱ

[링크 : https://github.com/mostlygeek/llama-swap/releases]

 

그냥 실행하니 config.yaml 없다고 배쨰는데

$ ./llama-swap 
Error loading config: open config.yaml: no such file or directory

 

다시 잘보니.. 당연하게(?) llama.cpp의 gguf 를 이용해서 바꾸는거라 llama.cpp가 있는데서 해야한다.

# minimum viable config.yaml

models:
  model1:
    cmd: llama-server --port ${PORT} --model /path/to/model.gguf

[링크 : https://github.com/mostlygeek/llama-swap]

 

+

llama-server도 pre-built로 받았더니 PATH 에 없어서 전체 경로 넣어주게 생겼음 ㅋㅋ

아무튼 --port {PORT}는 빼면 안되는 것 같고

$ cat config.yaml 
# minimum viable config.yaml

models:
  gemma4-26B:
    cmd: /home/minimonk/src/llama-b8876/llama-server --port ${PORT} --model /home/minimonk/src/model/gemma-4-26B-A4B-it-UD-IQ2_M.gguf
  gemma4-e2b:
    cmd: /home/minimonk/src/llama-b8876/llama-server --port ${PORT} --model /home/minimonk/src/model/gemma-4-E2B-it-Q4_K_M.gguf
  gemma4-e4b:
    cmd: /home/minimonk/src/llama-b8876/llama-server --port ${PORT} --model /home/minimonk/src/model/gemma-4-E4B-it-Q4_K_M.gguf
  llama3.2-3b:
    cmd: /home/minimonk/src/llama-b8876/llama-server --port ${PORT} --model /home/minimonk/src/model/Llama-3.2-3B-Instruct-Q4_K_M.gguf
  qwen3.6-35b:
    cmd: /home/minimonk/src/llama-b8876/llama-server --port ${PORT} --model /home/minimonk/src/model/Qwen3.6-35B-A3B-UD-Q2_K_XL.gguf

 

8080번 포트로 붙으면 상단에 모델 변경하면서 말걸으면 된다.

 

근데.. 왜?

메모리도 넉넉할텐데 동시 복수의 모델 로드는 안되는거지?

저작자표시 (새창열림)

'프로그램 사용 > ai 프로그램' 카테고리의 다른 글

openai 라이브러리(파이썬)  (0) 2026.04.29
llama-swap 과 python 으로 통신하기  (0) 2026.04.29
llama.cpp offload -ngl  (0) 2026.04.26
llama.cpp windows cuda12 1080 ti 11GB + 1060 6GB 테스트  (0) 2026.04.25
llama.cpp windows cuda12 1080 ti 11GB * 2 테스트  (0) 2026.04.25
Posted by 구차니
프로그램 사용/ai 프로그램2026. 4. 26. 19:49

llama.cpp offload -ngl

문득 ollama의 CPU / GPU 비율은 어떻게 계산하나 궁금한데

코드 뒤지긴 귀찮아서(있나?) gpt에게 물어보고 테스트.

 

일단 그냥 올라갈 작은 녀석을 해보니 모두 CUDA0로 뜬다.

D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64>llama-cli -m ..\llama-3.2-3b-instruct-q4_k_m.gguf -v

load_tensors: offloading output layer to GPU
load_tensors: offloading 27 repeating layers to GPU
load_tensors: offloaded 29/29 layers to GPU

llama_context:  CUDA_Host  output buffer size =     0.49 MiB
llama_kv_cache: layer   0: dev = CUDA0
llama_kv_cache: layer   1: dev = CUDA0
llama_kv_cache: layer   2: dev = CUDA0
llama_kv_cache: layer   3: dev = CUDA0
llama_kv_cache: layer   4: dev = CUDA0
llama_kv_cache: layer   5: dev = CUDA0
llama_kv_cache: layer   6: dev = CUDA0
llama_kv_cache: layer   7: dev = CUDA0
llama_kv_cache: layer   8: dev = CUDA0
llama_kv_cache: layer   9: dev = CUDA0
llama_kv_cache: layer  10: dev = CUDA0
llama_kv_cache: layer  11: dev = CUDA0
llama_kv_cache: layer  12: dev = CUDA0
llama_kv_cache: layer  13: dev = CUDA0
llama_kv_cache: layer  14: dev = CUDA0
llama_kv_cache: layer  15: dev = CUDA0
llama_kv_cache: layer  16: dev = CUDA0
llama_kv_cache: layer  17: dev = CUDA0
llama_kv_cache: layer  18: dev = CUDA0
llama_kv_cache: layer  19: dev = CUDA0
llama_kv_cache: layer  20: dev = CUDA0
llama_kv_cache: layer  21: dev = CUDA0
llama_kv_cache: layer  22: dev = CUDA0
llama_kv_cache: layer  23: dev = CUDA0
llama_kv_cache: layer  24: dev = CUDA0
llama_kv_cache: layer  25: dev = CUDA0
llama_kv_cache: layer  26: dev = CUDA0
llama_kv_cache: layer  27: dev = CUDA0

 

-ngl 옵션을 통해 gpu loading 갯수를 20개로 해주면

20/29가 되고 CUDA0는 9~27번 까지 엥 하나 어디갔어?

그 와중에 cpu 레이어가 먼저네?

D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64>llama-cli -m ..\llama-3.2-3b-instruct-q4_k_m.gguf -v -ngl 20

load_tensors: offloading output layer to GPU
load_tensors: offloading 19 repeating layers to GPU
load_tensors: offloaded 20/29 layers to GPU

llama_context:  CUDA_Host  output buffer size =     0.49 MiB
llama_kv_cache: layer   0: dev = CPU
llama_kv_cache: layer   1: dev = CPU
llama_kv_cache: layer   2: dev = CPU
llama_kv_cache: layer   3: dev = CPU
llama_kv_cache: layer   4: dev = CPU
llama_kv_cache: layer   5: dev = CPU
llama_kv_cache: layer   6: dev = CPU
llama_kv_cache: layer   7: dev = CPU
llama_kv_cache: layer   8: dev = CPU
llama_kv_cache: layer   9: dev = CUDA0
llama_kv_cache: layer  10: dev = CUDA0
llama_kv_cache: layer  11: dev = CUDA0
llama_kv_cache: layer  12: dev = CUDA0
llama_kv_cache: layer  13: dev = CUDA0
llama_kv_cache: layer  14: dev = CUDA0
llama_kv_cache: layer  15: dev = CUDA0
llama_kv_cache: layer  16: dev = CUDA0
llama_kv_cache: layer  17: dev = CUDA0
llama_kv_cache: layer  18: dev = CUDA0
llama_kv_cache: layer  19: dev = CUDA0
llama_kv_cache: layer  20: dev = CUDA0
llama_kv_cache: layer  21: dev = CUDA0
llama_kv_cache: layer  22: dev = CUDA0
llama_kv_cache: layer  23: dev = CUDA0
llama_kv_cache: layer  24: dev = CUDA0
llama_kv_cache: layer  25: dev = CUDA0
llama_kv_cache: layer  26: dev = CUDA0
llama_kv_cache: layer  27: dev = CUDA0
저작자표시 (새창열림)

'프로그램 사용 > ai 프로그램' 카테고리의 다른 글

llama-swap 과 python 으로 통신하기  (0) 2026.04.29
llama-swap  (0) 2026.04.27
llama.cpp windows cuda12 1080 ti 11GB + 1060 6GB 테스트  (0) 2026.04.25
llama.cpp windows cuda12 1080 ti 11GB * 2 테스트  (0) 2026.04.25
llama.cpp build / cuda compute capability  (0) 2026.04.25
Posted by 구차니
프로그램 사용/ai 프로그램2026. 4. 25. 19:04

llama.cpp windows cuda12 1080 ti 11GB + 1060 6GB 테스트

0번 1080 Ti 11GB

1번 1060 6GB

 

Sat Apr 25 18:45:09 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 582.28                 Driver Version: 582.28         CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                  Driver-Model | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce GTX 1080 Ti   WDDM  |   00000000:01:00.0 Off |                  N/A |
| 24%   40C    P8             12W /  250W |    9328MiB /  11264MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce GTX 1060 6GB  WDDM  |   00000000:02:00.0 Off |                  N/A |
| 39%   35C    P8              5W /  120W |    4111MiB /   6144MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            3040      C   ...n-cuda-12.4-x64\llama-cli.exe      N/A      |
|    1   N/A  N/A            3040      C   ...n-cuda-12.4-x64\llama-cli.exe      N/A      |
+-----------------------------------------------------------------------------------------+

 

D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64>llama-cli --list-devices
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 17407 MiB):
  Device 0: NVIDIA GeForce GTX 1080 Ti, compute capability 6.1, VMM: yes, VRAM: 11263 MiB
  Device 1: NVIDIA GeForce GTX 1060 6GB, compute capability 6.1, VMM: yes, VRAM: 6143 MiB
load_backend: loaded CUDA backend from D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64\ggml-cuda.dll
load_backend: loaded RPC backend from D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64\ggml-rpc.dll
load_backend: loaded CPU backend from D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64\ggml-cpu-haswell.dll
Available devices:
  CUDA0: NVIDIA GeForce GTX 1080 Ti (11263 MiB, 10200 MiB free)
  CUDA1: NVIDIA GeForce GTX 1060 6GB (6143 MiB, 5197 MiB free)

 

1080 Ti 11GB + 1060 6GB

gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf 15.9GB

none 17 t/s with -dev CUDA0
10 t/s with -dev CUDA1
layer 19 t/s
row -
tensor -

 

row 로드실패
D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64>llama-cli.exe -m ..\gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf -sm row
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 17407 MiB):
  Device 0: NVIDIA GeForce GTX 1080 Ti, compute capability 6.1, VMM: yes, VRAM: 11263 MiB
  Device 1: NVIDIA GeForce GTX 1060 6GB, compute capability 6.1, VMM: yes, VRAM: 6143 MiB
load_backend: loaded CUDA backend from D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64\ggml-cuda.dll
load_backend: loaded RPC backend from D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64\ggml-rpc.dll
load_backend: loaded CPU backend from D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64\ggml-cpu-haswell.dll

Loading model... -D:\a\llama.cpp\llama.cpp\ggml\src\ggml-cuda\ggml-cuda.cu:97: CUDA error
CUDA error: out of memory

 

tensor 로드실패
D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64>llama-cli.exe -m ..\gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf -sm tensor
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 17407 MiB):
  Device 0: NVIDIA GeForce GTX 1080 Ti, compute capability 6.1, VMM: yes, VRAM: 11263 MiB
  Device 1: NVIDIA GeForce GTX 1060 6GB, compute capability 6.1, VMM: yes, VRAM: 6143 MiB
load_backend: loaded CUDA backend from D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64\ggml-cuda.dll
load_backend: loaded RPC backend from D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64\ggml-rpc.dll
load_backend: loaded CPU backend from D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64\ggml-cpu-haswell.dll

Loading model... /D:/a/llama.cpp/llama.cpp/ggml/src/ggml-backend.cpp:119: GGML_ASSERT(buffer) failed
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 8537.78 MiB on device 1: cudaMalloc failed: out of memory

 

Qwen3.6-27B-Q5_K_M.gguf  18.1GB

none < 2  t/s with -dev CUDA0
< 0.1 t/s with -dev CUDA1
layer 2 t/s
row -
tensor -

 

row 로드실패
D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64>llama-cli.exe -m ..\Qwen3.6-27B-Q5_K_M.gguf -sm row
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 17407 MiB):
  Device 0: NVIDIA GeForce GTX 1080 Ti, compute capability 6.1, VMM: yes, VRAM: 11263 MiB
  Device 1: NVIDIA GeForce GTX 1060 6GB, compute capability 6.1, VMM: yes, VRAM: 6143 MiB
load_backend: loaded CUDA backend from D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64\ggml-cuda.dll
load_backend: loaded RPC backend from D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64\ggml-rpc.dll
load_backend: loaded CPU backend from D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64\ggml-cpu-haswell.dll

Loading model... /CUDA error: out of memory
  current device: 1, in function ggml_backend_cuda_split_buffer_init_tensor at D:\a\llama.cpp\llama.cpp\ggml\src\ggml-cuda\ggml-cuda.cu:956
  ggml_cuda_device_malloc((void**)&buf, size, id)
D:\a\llama.cpp\llama.cpp\ggml\src\ggml-cuda\ggml-cuda.cu:97: CUDA error

 

1080 Ti 11GB * 2

gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf 15.9GB

none 13 t/s
layer 43 t/s
row -
tensor -

 

Qwen3.6-27B-Q5_K_M.gguf 18.1GB

none < 2 t/s
layer 7 t/s
row 5 t/s
tensor -

 

저작자표시 (새창열림)

'프로그램 사용 > ai 프로그램' 카테고리의 다른 글

llama-swap  (0) 2026.04.27
llama.cpp offload -ngl  (0) 2026.04.26
llama.cpp windows cuda12 1080 ti 11GB * 2 테스트  (0) 2026.04.25
llama.cpp build / cuda compute capability  (0) 2026.04.25
llama.cpp ubuntu vulkan 1080ti 11GB * 2 실패  (0) 2026.04.25
Posted by 구차니
프로그램 사용/ai 프로그램2026. 4. 25. 15:50

llama.cpp windows cuda12 1080 ti 11GB * 2 테스트

리눅스 용으로 cuda 12.x로 빌드하려니 까마득해서 그냥 윈도우로 시도

모델이 작은 걸로 해서 그런가 성능향상은 없는것 같기도 하고.. -sm 옵션 줘서 해봐야 하려나?

 

구글 검색하니 이상한 키워드가 나와서 사용불가 -_-

아무튼 layer가 기본이다~ row는 병렬이다~ tensor도 병렬이다~ 라는데

일단 내꺼가 구형이라 그런가 layer가 전반적으로 더 잘나온다.

D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64>llama-cli -m ..\gemma-4-E4B-it-UD-Q8_K_XL.gguf -sm graph
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 22527 MiB):
  Device 0: NVIDIA GeForce GTX 1080 Ti, compute capability 6.1, VMM: yes, VRAM: 11263 MiB
  Device 1: NVIDIA GeForce GTX 1080 Ti, compute capability 6.1, VMM: yes, VRAM: 11263 MiB
load_backend: loaded CUDA backend from D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64\ggml-cuda.dll
load_backend: loaded RPC backend from D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64\ggml-rpc.dll
load_backend: loaded CPU backend from D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64\ggml-cpu-haswell.dll
error while handling argument "-sm": invalid value

usage:
-sm,   --split-mode {none,layer,row,tensor}
                                        how to split the model across multiple GPUs, one of:
                                        - none: use one GPU only
                                        - layer (default): split layers and KV across GPUs (pipelined)
                                        - row: split weight across GPUs by rows (parallelized)
                                        - tensor: split weights and KV across GPUs (parallelized,
                                        EXPERIMENTAL)
                                        (env: LLAMA_ARG_SPLIT_MODE)


to show complete usage, run with -h

 

pcie 버전과 x8 에 2개로 나눠져서 그런가? layer가 더 처참하다

gemma-4-E4B-it-UD-Q8_K_XL.gguf 8.05GB

none 40 t/s
layer 36 t/s
row 9 t/s
tensor 24 t/s

 

Qwen3.6-35B-A3B-UD-IQ1_M.gguf 9.35GB

none 40 t/s
layer 44 t/s
row 9 t/s
tensor 21 t/s

 

커지면 좀 layer가 효과가 생기나?

gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf 15.9GB

none 13 t/s
layer 43 t/s
row -
tensor -

 

row 로드실패
D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64>llama-cli.exe -m ..\gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf -sm row
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 22527 MiB):
  Device 0: NVIDIA GeForce GTX 1080 Ti, compute capability 6.1, VMM: yes, VRAM: 11263 MiB
  Device 1: NVIDIA GeForce GTX 1080 Ti, compute capability 6.1, VMM: yes, VRAM: 11263 MiB
load_backend: loaded CUDA backend from D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64\ggml-cuda.dll
load_backend: loaded RPC backend from D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64\ggml-rpc.dll
load_backend: loaded CPU backend from D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64\ggml-cpu-haswell.dll

Loading model... -D:\a\llama.cpp\llama.cpp\ggml\src\ggml-cuda\ggml-cuda.cu:924: GGML_ASSERT(tensor->view_src == nullptr) failed

 

tensor 로드실패

D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64>llama-cli.exe -m ..\gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf -sm tensor
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 22527 MiB):
  Device 0: NVIDIA GeForce GTX 1080 Ti, compute capability 6.1, VMM: yes, VRAM: 11263 MiB
  Device 1: NVIDIA GeForce GTX 1080 Ti, compute capability 6.1, VMM: yes, VRAM: 11263 MiB
load_backend: loaded CUDA backend from D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64\ggml-cuda.dll
load_backend: loaded RPC backend from D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64\ggml-rpc.dll
load_backend: loaded CPU backend from D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64\ggml-cpu-haswell.dll

Loading model... -ggml_backend_cuda_buffer_type_alloc_buffer: allocating 2560.00 MiB on device 0: cudaMalloc failed: out of memory
D:/a/llama.cpp/llama.cpp/ggml/src/ggml-backend.cpp:119: GGML_ASSERT(buffer) failed

 

Qwen3.6-27B-Q5_K_M.gguf 18.1GB

none < 2 t/s
layer 7 t/s
row 5 t/s
tensor -

 

tensor 로드 실패
D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64>llama-cli.exe -m ..\Qwen3.6-27B-Q5_K_M.gguf -sm tensor
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 22527 MiB):
  Device 0: NVIDIA GeForce GTX 1080 Ti, compute capability 6.1, VMM: yes, VRAM: 11263 MiB
  Device 1: NVIDIA GeForce GTX 1080 Ti, compute capability 6.1, VMM: yes, VRAM: 11263 MiB
load_backend: loaded CUDA backend from D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64\ggml-cuda.dll
load_backend: loaded RPC backend from D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64\ggml-rpc.dll
load_backend: loaded CPU backend from D:\study\llm\llama-b8918-bin-win-cuda-12.4-x64\ggml-cpu-haswell.dll

Loading model... -ggml_backend_cuda_buffer_type_alloc_buffer: allocating 8192.00 MiB on device 0: cudaMalloc failed: out of memory
alloc_tensor_range: failed to allocate CUDA0 buffer of size 8589934592
D:/a/llama.cpp/llama.cpp/ggml/src/ggml-backend.cpp:119: GGML_ASSERT(buffer) failed
저작자표시 (새창열림)

'프로그램 사용 > ai 프로그램' 카테고리의 다른 글

llama.cpp offload -ngl  (0) 2026.04.26
llama.cpp windows cuda12 1080 ti 11GB + 1060 6GB 테스트  (0) 2026.04.25
llama.cpp build / cuda compute capability  (0) 2026.04.25
llama.cpp ubuntu vulkan 1080ti 11GB * 2 실패  (0) 2026.04.25
llm transformer  (0) 2026.04.25
Posted by 구차니
프로그램 사용/ai 프로그램2026. 4. 25. 14:52

llama.cpp build / cuda compute capability

리눅스에서 cuda 구버전용으로 빌드해보려다 그냥 포기

$ nvidia-smi --query-gpu=name,compute_capap
name, compute_cap
NVIDIA GeForce GTX 1080 Ti, 6.1
NVIDIA GeForce GTX 1080 Ti, 6.1

 

Might also be worth trying to build with an older architecture, e.g. -DCMAKE_CUDA_ARCHITECTURES="75" (which will be run via PTX JIT compilation), to check whether the issue is related to building with 80.

[링크 : https://github.com/ggml-org/llama.cpp/issues/9727] 

 

일단 내껀 6.1 이라 아키텍쳐 61로 해야하는데 막상 넣어보니 지원안하는 거라고 나온다.

그래서 cuda 보니까 13.x

그래서 윈도우에서 12.x 에서 작동한건가?

$ cd /usr/local/cuda
cuda/      cuda-13/   cuda-13.0/

 

cuda - driver version

[링크 : https://2dudwns.tistory.com/21]

[링크 : https://www.reddit.com/r/comfyui/comments/1qjp7vs/nvidia_geforce_gtx_1070_with_cuda_capability_sm/]

[링크 : https://arca.live/b/alpaca/112167556]

저작자표시 (새창열림)

'프로그램 사용 > ai 프로그램' 카테고리의 다른 글

llama.cpp windows cuda12 1080 ti 11GB + 1060 6GB 테스트  (0) 2026.04.25
llama.cpp windows cuda12 1080 ti 11GB * 2 테스트  (0) 2026.04.25
llama.cpp ubuntu vulkan 1080ti 11GB * 2 실패  (0) 2026.04.25
llm transformer  (0) 2026.04.25
nvidia-smi를 통한 소비전력 제한과 토큰 생성속도  (1) 2026.04.24
Posted by 구차니
프로그램 사용/ai 프로그램2026. 4. 25. 14:30

llama.cpp ubuntu vulkan 1080ti 11GB * 2 실패

vulkan은 단일 그래픽일떄는 잘 도는데 두 개일때는 영~ 안되는 느낌이다.

 

먼가 장렬히 죽어버리는데

gpt에 물어봐도 뜬구름 잡는 소리만 하고 있고

일단 wait4.c 가 없다는건 중요한게 아니라고 하는데 디버그 심볼을 설치해 줘도 뜨는 건 매한가지

~/Downloads/llama-b8902/model$ ../llama-cli -m ./gemma-4-E2B-it-Q4_K_M.gguf 
load_backend: loaded RPC backend from /home/minimonk/Downloads/llama-b8902/libggml-rpc.so
load_backend: loaded Vulkan backend from /home/minimonk/Downloads/llama-b8902/libggml-vulkan.so
load_backend: loaded CPU backend from /home/minimonk/Downloads/llama-b8902/libggml-cpu-haswell.so

Loading model... //home/runner/work/llama.cpp/llama.cpp/ggml/src/ggml-backend.cpp:1367: GGML_ASSERT(n_inputs < GGML_SCHED_MAX_SPLIT_INPUTS) failed
-[New LWP 3559]
[New LWP 3560]
[New LWP 3561]
[New LWP 3563]
[New LWP 3564]
[New LWP 3565]
[New LWP 3568]
[New LWP 3570]
[New LWP 3571]
[New LWP 3574]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007ed9e18ea42f in __GI___wait4 (pid=3575, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30 ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory.
#0  0x00007ed9e18ea42f in __GI___wait4 (pid=3575, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30 in ../sysdeps/unix/sysv/linux/wait4.c
#1  0x00007ed9e1f5b71b in ggml_print_backtrace () from /home/minimonk/Downloads/llama-b8902/libggml-base.so.0
#2  0x00007ed9e1f5b8b2 in ggml_abort () from /home/minimonk/Downloads/llama-b8902/libggml-base.so.0
#3  0x00007ed9e1f77f2d in ggml_backend_sched_split_graph () from /home/minimonk/Downloads/llama-b8902/libggml-base.so.0
#4  0x00007ed9e1f780b5 in ggml_backend_sched_reserve_size () from /home/minimonk/Downloads/llama-b8902/libggml-base.so.0
#5  0x00007ed9e20b0fa4 in llama_context::graph_reserve(unsigned int, unsigned int, unsigned int, llama_memory_context_i const*, bool, unsigned long*) () from /home/minimonk/Downloads/llama-b8902/libllama.so.0
#6  0x00007ed9e20b1adf in llama_context::sched_reserve() () from /home/minimonk/Downloads/llama-b8902/libllama.so.0
#7  0x00007ed9e20b5869 in llama_context::llama_context(llama_model const&, llama_context_params) () from /home/minimonk/Downloads/llama-b8902/libllama.so.0
#8  0x00007ed9e20b649b in llama_init_from_model () from /home/minimonk/Downloads/llama-b8902/libllama.so.0
#9  0x00007ed9e26282a2 in common_get_device_memory_data(char const*, llama_model_params const*, llama_context_params const*, std::vector<ggml_backend_device*, std::allocator<ggml_backend_device*> >&, unsigned int&, unsigned int&, unsigned int&, ggml_log_level) () from /home/minimonk/Downloads/llama-b8902/libllama-common.so.0
#10 0x00007ed9e2628f52 in common_params_fit_impl(char const*, llama_model_params*, llama_context_params*, float*, llama_model_tensor_buft_override*, unsigned long*, unsigned int, ggml_log_level) () from /home/minimonk/Downloads/llama-b8902/libllama-common.so.0
#11 0x00007ed9e262c9e2 in common_fit_params(char const*, llama_model_params*, llama_context_params*, float*, llama_model_tensor_buft_override*, unsigned long*, unsigned int, ggml_log_level) () from /home/minimonk/Downloads/llama-b8902/libllama-common.so.0
#12 0x00007ed9e25fe2ff in common_init_result::common_init_result(common_params&) () from /home/minimonk/Downloads/llama-b8902/libllama-common.so.0
#13 0x00007ed9e25ff9a8 in common_init_from_params(common_params&) () from /home/minimonk/Downloads/llama-b8902/libllama-common.so.0
#14 0x00006547aab18d7e in server_context_impl::load_model(common_params&) ()
#15 0x00006547aaa549b5 in main ()
\[Inferior 1 (process 3556) detached]
Aborted (core dumped)

 

 

-sm none

옵션을 주면 0 번 gpu를 가지고 도는데 -v 옵션을 주고 보니

메모리 할당이 0MB? 엥?

load_tensors: layer   0 assigned to device Vulkan0, is_swa = 1
load_tensors: layer   1 assigned to device Vulkan0, is_swa = 1
load_tensors: layer   2 assigned to device Vulkan0, is_swa = 1
load_tensors: layer   3 assigned to device Vulkan0, is_swa = 1
load_tensors: layer   4 assigned to device Vulkan0, is_swa = 0
load_tensors: layer   5 assigned to device Vulkan0, is_swa = 1
load_tensors: layer   6 assigned to device Vulkan0, is_swa = 1
load_tensors: layer   7 assigned to device Vulkan0, is_swa = 1
load_tensors: layer   8 assigned to device Vulkan0, is_swa = 1
load_tensors: layer   9 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  10 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  11 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  12 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  13 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  14 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  15 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  16 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  17 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  18 assigned to device Vulkan1, is_swa = 1
load_tensors: layer  19 assigned to device Vulkan1, is_swa = 0
load_tensors: layer  20 assigned to device Vulkan1, is_swa = 1
load_tensors: layer  21 assigned to device Vulkan1, is_swa = 1
load_tensors: layer  22 assigned to device Vulkan1, is_swa = 1
load_tensors: layer  23 assigned to device Vulkan1, is_swa = 1
load_tensors: layer  24 assigned to device Vulkan1, is_swa = 0
load_tensors: layer  25 assigned to device Vulkan1, is_swa = 1
load_tensors: layer  26 assigned to device Vulkan1, is_swa = 1
load_tensors: layer  27 assigned to device Vulkan1, is_swa = 1
load_tensors: layer  28 assigned to device Vulkan1, is_swa = 1
load_tensors: layer  29 assigned to device Vulkan1, is_swa = 0
load_tensors: layer  30 assigned to device Vulkan1, is_swa = 1
load_tensors: layer  31 assigned to device Vulkan1, is_swa = 1
load_tensors: layer  32 assigned to device Vulkan1, is_swa = 1
load_tensors: layer  33 assigned to device Vulkan1, is_swa = 1
load_tensors: layer  34 assigned to device Vulkan1, is_swa = 0
load_tensors: layer  35 assigned to device Vulkan1, is_swa = 0


load_tensors: offloading output layer to GPU
load_tensors: offloading 34 repeating layers to GPU
load_tensors: offloaded 36/36 layers to GPU
load_tensors:      Vulkan0 model buffer size =     0.00 MiB
load_tensors:      Vulkan1 model buffer size =     0.00 MiB
load_tensors:  Vulkan_Host model buffer size =     0.00 MiB
llama_context: constructing llama_context		 load_tensors: layer   0 assigned to device Vulkan0, is_swa = 1
load_tensors: layer   1 assigned to device Vulkan0, is_swa = 1
load_tensors: layer   2 assigned to device Vulkan0, is_swa = 1
load_tensors: layer   3 assigned to device Vulkan0, is_swa = 1
load_tensors: layer   4 assigned to device Vulkan0, is_swa = 0
load_tensors: layer   5 assigned to device Vulkan0, is_swa = 1
load_tensors: layer   6 assigned to device Vulkan0, is_swa = 1
load_tensors: layer   7 assigned to device Vulkan0, is_swa = 1
load_tensors: layer   8 assigned to device Vulkan0, is_swa = 1
load_tensors: layer   9 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  10 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  11 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  12 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  13 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  14 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  15 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  16 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  17 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  18 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  19 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  20 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  21 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  22 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  23 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  24 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  25 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  26 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  27 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  28 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  29 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  30 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  31 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  32 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  33 assigned to device Vulkan0, is_swa = 1
load_tensors: layer  34 assigned to device Vulkan0, is_swa = 0
load_tensors: layer  35 assigned to device Vulkan0, is_swa = 0

load_tensors: offloading output layer to GPU
load_tensors: offloading 34 repeating layers to GPU
load_tensors: offloaded 36/36 layers to GPU
load_tensors:   CPU_Mapped model buffer size =  1756.00 MiB
load_tensors:      Vulkan0 model buffer size =  1407.73 MiB
.|.../.........-..........\..........|...
common_init_result: added <eos> logit bias = -inf
common_init_result: added <|tool_response> logit bias = -inf
common_init_result: added <turn|> logit bias = -inf
llama_context: constructing llama_context

 

 

 

vulkan 에서 사용할 그래픽 카드를 1개(0번 1번), 2개로 해봤더니

먼가.. vulkan 으로 메모리 할당을 못하고 죽는 느낌.

$ GGML_VK_VISIBLE_DEVICES=0 ./llama-cli -m ../model/gemma-4-E2B-it-Q4_K_M.gguf -v llama_model_load_from_file_impl: using device Vulkan0 (NVIDIA GeForce GTX 1080 Ti) (0000:01:00.0) - 10985 MiB free

load_tensors: offloading output layer to GPU
load_tensors: offloading 34 repeating layers to GPU
load_tensors: offloaded 36/36 layers to GPU
load_tensors:      Vulkan0 model buffer size =     0.00 MiB
load_tensors:  Vulkan_Host model buffer size =     0.00 MiB

llama_context: enumerating backends
llama_context: backend_ptrs.size() = 2
sched_reserve: reserving ...
sched_reserve: max_nodes = 4816
sched_reserve: reserving full memory module
sched_reserve: worst-case: n_tokens = 512, n_seqs = 1, n_outputs = 1
graph_reserve: reserving a graph for ubatch with n_tokens =    1, n_seqs =  1, n_outputs =    1
sched_reserve: Flash Attention was auto, set to enabled
sched_reserve: resolving fused Gated Delta Net support:
graph_reserve: reserving a graph for ubatch with n_tokens =    1, n_seqs =  1, n_outputs =    1
sched_reserve: fused Gated Delta Net (autoregressive) enabled
graph_reserve: reserving a graph for ubatch with n_tokens =   16, n_seqs =  1, n_outputs =   16
sched_reserve: fused Gated Delta Net (chunked) enabled
graph_reserve: reserving a graph for ubatch with n_tokens =  512, n_seqs =  1, n_outputs =  512
graph_reserve: reserving a graph for ubatch with n_tokens =    1, n_seqs =  1, n_outputs =    1
graph_reserve: reserving a graph for ubatch with n_tokens =  512, n_seqs =  1, n_outputs =  512
sched_reserve:    Vulkan0 compute buffer size =   515.00 MiB
sched_reserve: Vulkan_Host compute buffer size =   281.52 MiB
sched_reserve: graph nodes  = 1500
sched_reserve: graph splits = 2
sched_reserve: reserve took 13.68 ms, sched copies = 1
common_memory_breakdown_print: | memory breakdown [MiB]    | total    free    self   model   context   compute       unaccounted |
common_memory_breakdown_print: |   - Vulkan0 (GTX 1080 Ti) | 11510 = 10975 + (2702 =  1407 +     780 +     515) + 17592186042248 |
common_memory_breakdown_print: |   - Host                  |                  2037 =  1756 +       0 +     281                   |
$ GGML_VK_VISIBLE_DEVICES=1 ./llama-cli -m ../model/gemma-4-E2B-it-Q4_K_M.gguf -v llama_model_load_from_file_impl: using device Vulkan0 (NVIDIA GeForce GTX 1080 Ti) (0000:02:00.0) - 11393 MiB free

load_tensors: offloading output layer to GPU
load_tensors: offloading 34 repeating layers to GPU
load_tensors: offloaded 36/36 layers to GPU
load_tensors:      Vulkan0 model buffer size =     0.00 MiB
load_tensors:  Vulkan_Host model buffer size =     0.00 MiB

llama_context: enumerating backends
llama_context: backend_ptrs.size() = 2
sched_reserve: reserving ...
sched_reserve: max_nodes = 4816
sched_reserve: reserving full memory module
sched_reserve: worst-case: n_tokens = 512, n_seqs = 1, n_outputs = 1
graph_reserve: reserving a graph for ubatch with n_tokens =    1, n_seqs =  1, n_outputs =    1
sched_reserve: Flash Attention was auto, set to enabled
sched_reserve: resolving fused Gated Delta Net support:
graph_reserve: reserving a graph for ubatch with n_tokens =    1, n_seqs =  1, n_outputs =    1
sched_reserve: fused Gated Delta Net (autoregressive) enabled
graph_reserve: reserving a graph for ubatch with n_tokens =   16, n_seqs =  1, n_outputs =   16
sched_reserve: fused Gated Delta Net (chunked) enabled
graph_reserve: reserving a graph for ubatch with n_tokens =  512, n_seqs =  1, n_outputs =  512
graph_reserve: reserving a graph for ubatch with n_tokens =    1, n_seqs =  1, n_outputs =    1
graph_reserve: reserving a graph for ubatch with n_tokens =  512, n_seqs =  1, n_outputs =  512
sched_reserve:    Vulkan0 compute buffer size =   515.00 MiB
sched_reserve: Vulkan_Host compute buffer size =   281.52 MiB
sched_reserve: graph nodes  = 1500
sched_reserve: graph splits = 2
sched_reserve: reserve took 12.95 ms, sched copies = 1
common_memory_breakdown_print: | memory breakdown [MiB]    | total    free    self   model   context   compute       unaccounted |
common_memory_breakdown_print: |   - Vulkan0 (GTX 1080 Ti) | 11510 = 11382 + (2702 =  1407 +     780 +     515) + 17592186041840 |
common_memory_breakdown_print: |   - Host                  |                  2037 =  1756 +       0 +     281                   |
$ GGML_VK_VISIBLE_DEVICES=0,1 ./llama-cli -m ../model/gemma-4-E2B-it-Q4_K_M.gguf -v llama_model_load_from_file_impl: using device Vulkan0 (NVIDIA GeForce GTX 1080 Ti) (0000:01:00.0) - 11009 MiB free
llama_model_load_from_file_impl: using device Vulkan1 (NVIDIA GeForce GTX 1080 Ti) (0000:02:00.0) - 11392 MiB free

load_tensors: offloading output layer to GPU
load_tensors: offloading 34 repeating layers to GPU
load_tensors: offloaded 36/36 layers to GPU
load_tensors:      Vulkan0 model buffer size =     0.00 MiB
load_tensors:      Vulkan1 model buffer size =     0.00 MiB
load_tensors:  Vulkan_Host model buffer size =     0.00 MiB

llama_context: enumerating backends
llama_context: backend_ptrs.size() = 3
llama_context: pipeline parallelism enabled
sched_reserve: reserving ...
sched_reserve: max_nodes = 4824
sched_reserve: reserving full memory module
sched_reserve: worst-case: n_tokens = 512, n_seqs = 1, n_outputs = 1
graph_reserve: reserving a graph for ubatch with n_tokens =    1, n_seqs =  1, n_outputs =    1
sched_reserve: Flash Attention was auto, set to enabled
sched_reserve: resolving fused Gated Delta Net support:
graph_reserve: reserving a graph for ubatch with n_tokens =    1, n_seqs =  1, n_outputs =    1
sched_reserve: fused Gated Delta Net (autoregressive) enabled
graph_reserve: reserving a graph for ubatch with n_tokens =   16, n_seqs =  1, n_outputs =   16
sched_reserve: fused Gated Delta Net (chunked) enabled
graph_reserve: reserving a graph for ubatch with n_tokens =  512, n_seqs =  1, n_outputs =  512
/home/runner/work/llama.cpp/llama.cpp/ggml/src/ggml-backend.cpp:1367: GGML_ASSERT(n_inputs < GGML_SCHED_MAX_SPLIT_INPUTS) failed
[New LWP 12120]
[New LWP 12121]
[New LWP 12122]
[New LWP 12124]
[New LWP 12125]
[New LWP 12126]
[New LWP 12129]
[New LWP 12131]
[New LWP 12132]
[New LWP 12135]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x0000743ccfeea42f in __GI___wait4 (pid=12136, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30 ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory.
저작자표시 (새창열림)

'프로그램 사용 > ai 프로그램' 카테고리의 다른 글

llama.cpp windows cuda12 1080 ti 11GB * 2 테스트  (0) 2026.04.25
llama.cpp build / cuda compute capability  (0) 2026.04.25
llm transformer  (0) 2026.04.25
nvidia-smi를 통한 소비전력 제한과 토큰 생성속도  (1) 2026.04.24
llama-server web ui  (0) 2026.04.24
Posted by 구차니

티스토리툴바