PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training (arXiv:2309.10400)

ICLR 2024, v3 dated 2024-02-21. Method: decouple TRAIN length from TARGET length by dividing the training context window into chunks and giving each chunk a distinct positional-skip bias term on its position indices, with the biases and chunk lengths resampled for every training example so the model sees all positions within the target length. Abstract claims, verbatim: 'PoSE greatly reduces memory and time overhead compared with Full-length fine-tuning, with minimal impact on performance'; 'we have successfully extended the LLaMA model to 128k tokens using a 2k training context window'; 'we empirically confirm that PoSE is compatible with all RoPE-based LLMs and position interpolation strategies'; 'our method can potentially support infinite length, limited only by memory usage in inference'. This is the cheapest available evidence that a model can hold a 128K license without ever having run a 128K activation, which is the mechanism behind every claimed-vs-effective gap in [[ctx-claimed-vs-effective-context-table]].