Run on 16GB VRAM

[ppsirius]

If is possible to run it somehow on rtx 5070 ti by vllm?
I run fara in llm studio (quantitation model) but cant connect to Magentic-UI by compatibility with openai endpoint. I got some problem about not proper return object from llm studio.

[mehuljariwala]

Hello @ppsirius

Yes, 16GB VRAM is enough. Two options:

1. GGUF Quantized (easiest)

Use bartowski/microsoft_Fara-7B-GGUF - grab Q5_K_M (~5GB) or Q6_K (~6GB).

2. vLLM with reduced context

vllm serve "microsoft/Fara-7B" --port 5000 --dtype half --max-model-len 8192

LM Studio + Magentic-UI issue

LM Studio's response format doesn't fully match what Magentic-UI expects. Try pmbstyle/fara-agent instead - it's built specifically for LM Studio.

[ppsirius]

Thanks for response, I run fara-agent but have some problems with wrong coordinates for clicks :|

I also try to run vllm with

vllm serve "microsoft/Fara-7B" --port 5000 --dtype half --max-model-len 8192 --gpu-memory-utilization 0.75

but still got the error

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 260.00 MiB. GPU 0 has a total capacity of 15.47 GiB of which 231.38 MiB is free. Including non-PyTorch memory, this process has 13.84 GiB memory in use. Of the allocated memory 13.37 GiB is allocated by PyTorch, and 153.92 MiB is reserved by PyTorch but unallocated.

[bbkmike]

Hi @mehuljariwala ,
I tried with pmbstyle/fara-agent instead - it's built specifically for LM Studio, can only launch the firefox browser with default bing.com,but cannot jump to wikipedia.org and got the below errors, could you guide me how to fix it?
LM Studio:0.3.35
microsoft_fara-7b: Q5_K_M
C:\fara-agent>python --version
Python 3.11.9
C:\fara-agent>lms server status
The server is running on port 1235.
C:\fara-agent>curl http://localhost:1235/v1/models
{
"data": [
{
"id": "microsoft_fara-7b",
"object": "model",
"owned_by": "organization_owner"
},
{
"id": "text-embedding-nomic-embed-text-v1.5",
"object": "model",
"owned_by": "organization_owner"
}
],
"object": "list"
}
C:\fara-agent>python run_agent.py --task "Go to wikipedia.org and search for cats" --headful
[INFO] Browser started
[INFO] Agent started
[INFO] Running task: Go to wikipedia.org and search for cats
[INFO] Round 1/20
[INFO] HTTP Request: POST http://localhost:1235/v1/chat/completions "HTTP/1.1 503 Service Unavailable"
[INFO] Retrying request to /chat/completions in 0.418827 seconds

Updated: The issue has been fixed by upgrading the LM Studio to version :0.3.36

But it is pretty slow.

[mehuljariwala]

@husseinmozannar any thoughts on above?