Under TaQuants, I am working on creating GGUF models that preserve inference quality even under low-bit quantization. High-quality extreme low-bit quantized models enable inference of massive models in low-resource environments.
In Swap-MoE, I am investigating a mechanism that places Mixture-of-Experts experts on SSD and loads them on demand as needed, exploring the operation of large-scale models under limited DRAM.
Tema_Q-Agent is a fully local coding agent that works in combination with quantized models. It is designed to run using only the compute resources available on hand.
A technical report summarizing the low-bit quantization approach and the distribution policy in GGUF format. It covers the concept of adaptive bit allocation at the tensor / layer level, the models currently published, and known limitations.
Read the PDF reportA technical report summarizing the mechanism for on-demand loading of MoE architecture experts from SSD. It covers the underlying motivation, the caching strategy, the current state of implementation, and remaining challenges.
Read the PDF report