KV Cache Quantization
Architected a custom inference engine in PyTorch with a KIVI-inspired INT8 quantization pipeline,
bypassing model.generate() for manual token-by-token cache management.
Reduced GPT-2 KV cache footprint by 68.5% with only a 0.71% increase in perplexity, implementing flexible
symmetric/asymmetric quantization schemes across whole-tensor, per-channel, and per-token granularities.