LLM Benchmarking Sensitivity Revealed

A research paper has been submitted proposing the Format Sensitivity Index (FSI) to assess the variability in LLM benchmarking caused by prompt formatting changes.

The findings suggest that formatting differences in prompt wrappers can significantly alter benchmark results, questioning the reliability of current LLM evaluation methods.

LLM developers and researchers who rely on benchmarks for model performance evaluation may be impacted by these findings, pointing to a need for revised assessment frameworks.

The study's conclusions are based on specific protocols and datasets, and broader applicability remains uncertain, indicating a need for further research to validate these findings.