llama.cpp는 사용자 정의 공급자 ID입니다. 원하는 문자열로 지정할 수 있습니다. npm은 이 공급자에 사용할 패키지를 지정합니다. 여기서는 OpenAI 호환 API를 위해 @ai-sdk/openai-compatible을 사용합니다. name은 UI에 표시될 공급자 이름입니다. options.baseURL은 로컬 서버의 엔드포인트입니다. models는 모델 ID와 해당 구성을 매핑합니다. 모델 이름은 모델 선택 목록에 표시됩니다.
qwen3.6 이나 gemma4 31B 이런애들만 되는 줄 알았는데, e2b e4b 에서도 되서 시도!
6. Audio Use the following prompt structures for audio processing:
Audio Speech Recognition (ASR) Transcribe the following speech segment in {LANGUAGE} into {LANGUAGE} text.
Follow these specific instructions for formatting the answer: * Only output the transcription, with no newlines. * When transcribing numbers, write the digits, i.e. write 1.7 and not one point seven, and write 3 instead of three.
Automatic Speech Translation (AST) Transcribe the following speech segment in {SOURCE_LANGUAGE}, then translate it into {TARGET_LANGUAGE}. When formatting the answer, first output the transcription in {SOURCE_LANGUAGE}, then one newline, then output the string '{TARGET_LANGUAGE}: ', then the translation in {TARGET_LANGUAGE}.
뜬금없이 저장소 계정을 물어보는건, 임의의 브랜치를 만들어서 먼가를 더 받거나 할 수 있도록 하면 해결
Installing open_clip Cloning Stable Diffusion into /mnt/Downloads/stable-diffusion-webui/repositories/stable-diffusion-stability-ai... Cloning into '/mnt/Downloads/stable-diffusion-webui/repositories/stable-diffusion-stability-ai'... Username for 'https://github.com': Password for 'https://github.com': remote: Repository not found. fatal: Authentication failed for 'https://github.com/Stability-AI/stablediffusion.git/'
Cannot locate TCMalloc (improves CPU memory usage) Cannot locate TCMalloc. Do you have tcmalloc or google-perftool installed on your system? (improves CPU memory usage)
Highlights/Performance Improvements Deprecation of Cuda 11.6 and Python 3.7 support for PyTorch 2.0 If you are still using or depending on CUDA 11.6 or Python 3.7 builds, we strongly recommend moving to at least CUDA 11.7 and Python 3.8, as it would be the minimum versions required for PyTorch 2.0. For more detail, please refer to the Release Compatibility Matrix for PyTorch releases.
positive_magic = { "en": ", Ultra HD, 4K, cinematic composition.", # for english prompt "zh": ", 超清，4K，电影级构图." # for chinese prompt }
# Generate image prompt = '''A coffee shop entrance features a chalkboard sign reading "Qwen Coffee 😊 $2 per cup," with a neon light beside it displaying "通义千问". Next to it hangs a poster showing a beautiful Chinese woman, and beneath the poster is written "π≈3.1415926-53589793-23846264-33832795-02384197". Ultra HD, 4K, cinematic composition'''
negative_prompt = " " # using an empty string if you do not have specific concept to remove
기존에도 json 형식으로 입력을 주고 있었고 그것의 role에 system과 user 등으로 나누어서 주니
사용자 질문과 분리되어 agent 상태가 바뀌는 것으로 보인다.
'messages': [ {'role':'user', 'content':'tell me a joke'}, {'role':'assistant', 'content':'why did the chicken cross the road'}, {'role':'user', 'content':'I don't know, why did the chicken cross the road'} ]
'messages': [ {'role':'system', 'content':'You are an assistant that speaks like Shakespeare.'}, {'role':'user', 'content':'tell me a joke'},
현재 문서에는 system 대신 developer가 들어가고 user와 asisstant가 존재한다.
TheOpenAI model specdescribes how our models give different levels of priority to messages with different roles.
A multi-turn conversation may consist of several messages of these types, along with other content types provided by both you and the model. Learn more aboutmanaging conversation state here. You could think aboutdeveloperandusermessages like a function and its arguments in a programming language.
developermessages provide the system’s rules and business logic, like a function definition.
usermessages provide inputs and configuration to which thedevelopermessage instructions are applied, like arguments to a function.
messages = [] messages.append({"role": "user", "content": "에이전트: 50대의 it 전문가. 박사학위가 있으며 전문적인 대답을 할 수 있 음. 질문 : 너에 대해서 설명해줘"}) response = ollama.chat( model="llama3.2:3b", messages=messages )
저는 IT 전문가로, 50대의 경험과 expertise를 가지고 있습니다. 박사 학위를 hold 한 것으로 biết됩니다.
자세한 정보는 아래에 정리 해 두어 보니 좋을 것 같습니다.
* 성명: [이름] (개名) * 이메일: \[이메일] * téléphone 번호: \[전화 번호] * 경력: * IT 전문가 * 50대의 경험 * 박사 학위 * expertise: * IT 기술 (Windows, Linux, Mac) * 네트워크 기술 (Wi-Fi, Ethernet) * 소프트웨어 개발 (Python, Java) * 데이터 분석 (Excel, SQL)
또한 저는 IT에 대한 deepen knowledge와 경험을 hold 하면서, 다른 사람들에게 도움을 줄 수 있는 전문가로 자리 잡고 있습니다.
agent 설정을 따로 넣고 user에는 딱 질문만 넣는 경우
import ollama
messages = [] messages.append({"role": "system", "content": "에이전트: 50대의 it 전문가. 박사학위가 있으며 전문적인 대답을 할 수 있음"}) messages.append({"role": "user", "content": "너에 대해서 설명해줘"}) response = ollama.chat( model="llama3.2:3b", messages=messages )
* **50대에 도전한 IT 전문가**: 50대의 IT 전문가로, IT 기술을 발전시키고 보다 효율적으로 사용하는 데 도움을 주는 역할을 하겠습니다. * **박사학위 possession**: IT 기술과 관련된 박사 학위를 가지고 있기 때문에, 전문적인 대답을 할 수 있습니다. * **지능형 대화 시스템**: AI Assistant로, 다양한 질문이나 문제를 받은 후, 적절한 대답을 제공할 수 있습니다.
**제공하는 서비스:**
* IT-related 질문이나 समसolu션에 대한 répond * IT 기술과 관련된 정보와 도움을 제공합니다. * IT-related vấn đề에 대한 도움이 필요해すれば 언제든지 mij어세요.
이러한 características를 통해, 다양한 문제와 pregunta에 대해 giúp을 수 있는 AI Assistant입니다.
집에서 llama3.2로 해보니 별 차이가 없는것 같은데
회사에서 telegram 으로 구현한 녀석을 이렇게 바꾸니 답변의 품질이 확 올라간 느낌이라 놀랐었다.
llama-server: You can run a local server that accepts image inputs via an API.
Setup: Start the server with the multimodal projector: llama-server -m gemma-model.gguf--mmproj mmproj-model.gguf --port 8080.
Sending Images: You can then send requests with Base64 encoded images or URLs to the /v1/chat/completions endpoint.
도움말을 보니 multi modal proejctor 줄여서 mmproj 라..
/llama-b8925$ ./llama-server --help load_backend: loaded RPC backend from /mnt/Downloads/llama-b8925/libggml-rpc.so load_backend: loaded Vulkan backend from /mnt/Downloads/llama-b8925/libggml-vulkan.so load_backend: loaded CPU backend from /mnt/Downloads/llama-b8925/libggml-cpu-haswell.so ----- common params -----
-h, --help, --usage print usage and exit
-hf, -hfr, --hf-repo <user>/<model>[:quant] Hugging Face model repository; quant is optional, case-insensitive, default to Q4_K_M, or falls back to the first file in the repo if Q4_K_M doesn't exist. mmproj is also downloaded automatically if available. to disable, add --no-mmproj example: ggml-org/GLM-4.7-Flash-GGUF:Q4_K_M (default: unused) (env: LLAMA_ARG_HF_REPO) -hfd, -hfrd, --hf-repo-draft <user>/<model>[:quant] Same as --hf-repo, but for the draft model (default: unused) (env: LLAMA_ARG_HFD_REPO) -hff, --hf-file FILE Hugging Face model file. If specified, it will override the quant in --hf-repo (default: unused) (env: LLAMA_ARG_HF_FILE) -hfv, -hfrv, --hf-repo-v <user>/<model>[:quant] Hugging Face model repository for the vocoder model (default: unused) (env: LLAMA_ARG_HF_REPO_V) -hffv, --hf-file-v FILE Hugging Face model file for the vocoder model (default: unused) (env: LLAMA_ARG_HF_FILE_V) -hft, --hf-token TOKEN Hugging Face access token (default: value from HF_TOKEN environment variable)
-mm, --mmproj FILE path to a multimodal projector file. see tools/mtmd/README.md note: if -hf is used, this argument can be omitted (env: LLAMA_ARG_MMPROJ) -mmu, --mmproj-url URL URL to a multimodal projector file. see tools/mtmd/README.md (env: LLAMA_ARG_MMPROJ_URL) --mmproj-auto, --no-mmproj, --no-mmproj-auto whether to use multimodal projector file (if available), useful when using -hf (default: enabled) (env: LLAMA_ARG_MMPROJ_AUTO) --mmproj-offload, --no-mmproj-offload whether to enable GPU offloading for multimodal projector (default: enabled) (env: LLAMA_ARG_MMPROJ_OFFLOAD) --image-min-tokens N minimum number of tokens each image can take, only used by vision models with dynamic resolution (default: read from model) (env: LLAMA_ARG_IMAGE_MIN_TOKENS) --image-max-tokens N maximum number of tokens each image can take, only used by vision models with dynamic resolution (default: read from model) (env: LLAMA_ARG_IMAGE_MAX_TOKENS) -otd, --override-tensor-draft <tensor name pattern>=<buffer type>,... override tensor buffer type for draft model -cmoed, --cpu-moe-draft keep all Mixture of Experts (MoE) weights in the CPU for the draft model (env: LLAMA_ARG_CPU_MOE_DRAFT) -ncmoed, --n-cpu-moe-draft N keep the Mixture of Experts (MoE) weights of the first N layers in the CPU for the draft model
mtmd_init_from_file: error: mismatch between text model (n_embd = 2560) and mmproj (n_embd = 2816) hint: you may be using wrong mmproj
srv load_model: failed to load multimodal model, './model/gemma/mmproj-BF16.gguf' srv operator(): operator(): cleaning up before exit... main: exiting due to model loading error
BF16 은 FP32 와 동일한 8bit 지수, F16은 5bit 지수
간단하게 BF16이 F16 보다 정밀도 면에서 좋다. 인데 하드웨어 지원이 안되면 무의미할테니 머.. 어쩔수 없나?
Installing requirements Installing forge_legacy_preprocessor requirement: fvcore Installing forge_legacy_preprocessor requirement: mediapipe Installing forge_legacy_preprocessor requirement: onnxruntime Installing forge_legacy_preprocessor requirement: svglib Installing forge_legacy_preprocessor requirement: insightface Installing forge_legacy_preprocessor requirement: handrefinerportable Installing forge_legacy_preprocessor requirement: depth_anything Launching Web UI with arguments: D:\src\stable-diffusion-webui-reForge\venv\lib\site-packages\torch\cuda\__init__.py:283: UserWarning: Found GPU0 NVIDIA GeForce GTX 1060 6GB which is of cuda capability 6.1. Minimum and Maximum cuda capability supported by this version of PyTorch is (7.0) - (12.0)
warnings.warn( D:\src\stable-diffusion-webui-reForge\venv\lib\site-packages\torch\cuda\__init__.py:304: UserWarning: Please install PyTorch with a following CUDA configurations: 12.6 following instructions at https://pytorch.org/get-started/locally/
warnings.warn(matched_cuda_warn.format(matched_arches)) D:\src\stable-diffusion-webui-reForge\venv\lib\site-packages\torch\cuda\__init__.py:326: UserWarning: NVIDIA GeForce GTX 1060 6GB with CUDA capability sm_61 is not compatible with the current PyTorch installation. The current PyTorch install supports CUDA capabilities sm_70 sm_75 sm_80 sm_86 sm_90 sm_100 sm_120. If you want to use the NVIDIA GeForce GTX 1060 6GB GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
warnings.warn( Total VRAM 6144 MB, total RAM 16252 MB pytorch version: 2.9.0+cu128 Set vram state to: NORMAL_VRAM Device: cuda:0 NVIDIA GeForce GTX 1060 6GB : native CUDA Stream Activated: False W0430 12:05:07.789132 11892 venv\Lib\site-packages\torch\distributed\elastic\multiprocessing\redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs. Using pytorch attention
