ChatsYapsForumsTopicsWatchFeedsDecksChartsChatsYapsForumsTopicsWatchFeedsDecksCharts
Log inRegister

Log in to follow channels.

Log in to follow topics.

Log in to see your friends.

software

Forum post

User-Schedulable Approximations for Quantized Tensor Programs

·6d

arXiv.org

User-Schedulable Approximations for Quantized Tensor Programs

Quantization lets language models run on edge devices, but it sharply increases the number of tensor kernels a library must maintain: every weight format must be paired with every activation format on every instruction set and GPU, with an encoder and a decoder that agree on the stored bytes. In GGML, the tensor library used by llama.cpp, these kernels are written by hand, and only 8 of the 54 weight-activation pairs it runs on our Arm CPU have a tiled matrix-multiplication kernel. We extend Halide's separation of algorithm and schedule to the representation of values. Each approximation is a codec, an encoder paired with a decoder, composed from reusable units: quantizers, integer representations, bit packers and layouts. A new directive, approximate_by, applies a codec to one stage of a program and returns the encoder and decoder as ordinary stages that Halide can schedule, so both decoders fuse into the register tiles of a matrix product. Optional contracts, such as a round-trip l

1

0 comments