Nvidia+AMD RPC offload increases context to 195k on 2x 4060 Tensor Parallel with no perf cost
Read original on r/LocalLLaMA ↗Mixing Nvidia+AMD hardware via RPC usually meant split-mode layer only. I managed to keep split-mode tensor working, and patch my llama-rpc to send the drafters over to my strix. That let me go from 150k -> 195k...
Original headline: "The space is moving quick. 2x 4060 Tensor Parallel main weights. RPC offload the drafters to amd strix. Got +2-3GB Vram for context at no perf cost."