RULER: What's the Real Context Size of Your Long-Context Language Models? (arXiv:2404.06654)

COLM 2024 (per arXiv comments field). Synthetic benchmark for evaluating long-context LMs with flexible configurations for sequence length and task complexity. Expands beyond vanilla NIAH with multi-hop tracing and aggregation tasks. Evaluates 17 long-context LMs with 13 tasks. Finding: despite nearly perfect NIAH accuracy, almost all models exhibit large performance drops as context length increases; only half of models claiming 32K+ context maintain satisfactory performance at 32K. Code at https://github.com/hsiehjackson/RULER.