Models are usually trained in fp16, bf16, or fp32 — the precision that stability during training needs.

Once trained, they are often quantized down to int8 for inference (actually running the model): roughly a quarter of the memory of fp32, and hardware built for int8 can push through far more operations per second than the same silicon running floating point.

The accuracy cost is usually small. The speed and energy savings, at the scale a real deployment runs at, are not.