GGUF, fp8 and what your Mac can actually run

Model files come in formats that were chosen for a specific kind of accelerator. Most of the popular ones were chosen for NVIDIA cards. On Apple Silicon some of those formats work, some work badly, and one of them is a trap that looks like it is working while it quietly falls back to something much slower.

The confusion is worth clearing up once, because it decides which half of the model ecosystem is actually available to you — and it is a much larger half than most people assume.

fp8 is an eight-bit floating point format that recent NVIDIA cards can multiply natively. That native support is the entire reason it is fast. Apple Silicon has no equivalent path, so an fp8 checkpoint has to be unpacked to a wider format before anything can happen with it.

The result is the worst of both: you get the quality loss of eight-bit weights and none of the speed that was supposed to pay for it, plus the conversion overhead on top. It sometimes runs. It is never the right choice.

bf16 is sixteen-bit brain float. It is the full-quality option, it runs natively, and if the model fits in memory it is what you want. No quality argument, no conversion step.

GGUF is a quantised container with a quality dial. The suffix tells you how aggressive the compression is:

• Q8 — nearly indistinguishable from full precision, largest of the quantised options.
• Q6_K — the sweet spot for image models. A real size saving with quality loss you have to look for.
• Q4 — noticeable degradation on image work. Reasonable for language models, rarely worth it for pictures.

The decision is simple: if bf16 fits comfortably, use bf16. If it does not, drop to Q6_K before you consider anything else.

On a discrete GPU, the model has to fit in the card's memory or it does not run. On Apple Silicon, the GPU addresses the same memory as everything else, so a large model can load where a mid-range graphics card would refuse outright.

The catch is that it is the same memory your operating system, browser and editing application are using. A 48GB machine does not have 48GB for the model. In practice, plan for around two thirds and leave the rest alone.

The failure mode is also different: instead of a clean out-of-memory error, the system starts swapping to disk and everything becomes glacial without telling you why. If a generation that took three minutes suddenly takes twenty, memory pressure is the first thing to check.

The download size is the weights. During generation you also need room for the text encoder, the VAE, the latents and whatever the sampler is holding. For image models a rough working figure is the file size plus a third.

So a 20GB bf16 checkpoint wants around 27GB free. On a 48GB machine that is comfortable with the browser closed and tight with a video application open. The same model at Q6_K comes in around 12GB and leaves you room to keep working while it runs.

That is usually the real argument for quantisation: not that the model cannot fit, but that you want to keep using your computer while it generates.

Take bf16 if it fits, GGUF Q6_K if it does not, and never fp8.

That covers almost every model you will download. The exceptions are rare enough that you will know when you have found one, and by then you will have enough context to judge it.

The wider point is that model format is a hardware question, not a quality question. Choosing by benchmark scores from someone else's machine is how people end up convinced their computer is slow when it is only being asked to do the wrong kind of arithmetic.