Ablation studies demonstrate that language models store time-dependent information in special "temporal heads."
E.g. "In YYYY, the President of South Korea was ..."
#MLSky arxiv.org/abs/2502.14258
Does Time Have Its Place? Temporal Heads: Where Language Models Recall Time-specific Information
While the ability of language models to elicit facts has been widely investigated, how they handle temporally changing facts remains underexplored. We discover Temporal Heads, specific attention heads...
