Hey fellow llama enthusiasts! Great to see that not all of lemmy is AI sceptical.

I’m in the process of upgrading my server with a bunch of GPUs. I’m really excited about the new Mistral / Magistral Small 3.2 models and would love to serve them for me and a couple of friends. My research led me to vLLM with which I was able to double inference speed compared to ollama at least for qwen3-32b-awq.

Now sadly, the most common quantization methods (GGUF, EXL, BNB) are either not fully (GGUF) or not at all (EXL) supported in vLLM, or multi-gpu inference thouth tensor parallelism is not supported (BNB). And especially for new models it’s hard to find pre-quantized models in different, more broadly supported formats (AWQ, GPTQ).

Does any of you guys face a similar problem? Do you quantize models yourself? Are there any up-to-date guides you would recommend? Or did I completely overlook another, obvious solution?

It feels like when I’ve researched something yesterday, it’s already outdated again today, since the landscape is so rapidly evolving.

Anyways, thank you for reading and sharing your thoughts or experience if you feel like it.

  • hendrik@palaver.p3x.deEnglish
    4·
    2 days ago

    I think they have you covered with this: https://github.com/vllm-project/llm-compressor

    I always find quantizations available since I use gguf models. But I’ve done it before, it’s not hard to do. Pick a method with good performance, it’s going to affect your inference speed. I don’t know which one is “best” for vllm.

    • robber@lemmy.mlOPEnglish
      2·
      2 days ago

      Thanks that looks cool, I’ll definitely try and report back. Do you happen to know what the hardware requirements are? I have access 64GB of RAM and 48GB of VRAM across 3 RTX2000e Ada GPUs.

      • hendrik@palaver.p3x.deEnglish
        2·
        2 days ago

        Uh sorry, no. Since I don’t use vllm, I don’t know. It certainly depends on the method you choose. The activation-aware ones will use a lot of resources. Just truncating the numbers to 8bit (or whatever) uses very little resources, I did that on my laptop. Also depends on model architecture and the size of the model you feed in. Since you gave an 32b parameter model, I’d expect it to take about 64GB loaded fully into memory using 16bit floating point numbers. (32 million billion times 2 bytes.) But I don’t really know whether quantization methods load the full thing, or if they do it in chunks. You’d have to google that or try it.

        • robber@lemmy.mlOPEnglish
          2·
          1 day ago

          Alright thanks! I found it somewhat difficult to find information about the hardware requirements online, but yeah, maybe I just have to try it.

          • hendrik@palaver.p3x.deEnglish
            2·
            1 day ago

            You’re welcome. If you fail and you can’t just add more RAM, maybe have a look at renting cloud servers. For example you can rent a computer on runpod.io for $2 an hour with double your specs. At least that’s how I do one-off big compute tasks.