RIPPLECOT: Amplifying Ripple Effect of Knowledge Editing in Language Models via Chain-of-Thought In-Context Learning

The ripple effect poses a significant challenge in knowledge editing for large language models. Namely, when a single fact is edited, the model struggles to accurately update the related facts in a sequence, which is evaluated by multi-hop questions linked to a chain of related facts. Recent strategies have moved away from traditional parameter updates to more flexible, less computation-intensive methods, proven to be more effective in the ripple effect. In-context learning (ICL) editing uses a simple demonstration Imagine that + new fact to guide LLMs, but struggles with complex multi-hop questions as the new fact alone fails to specify the chain of facts involved in such scenarios. Besides, memory-based editing maintains additional storage for all edits and related facts, requiring continuous updates to stay effective. As a result of the design limitations, the challenge remains, with the highest accuracy being only 33.8% on the MQUAKECF benchmarks for Vicuna-7B. To address this, we propose RIPPLECOT, a novel ICL editing approach integrating Chain-of-Thought (COT) reasoning. RIPPLECOT structures demonstrations as {new fact, question, thought, answer}, incorporating a thought component to identify and decompose the multi-hop logic within questions. This approach effectively guides the model through complex multi-hop questions with chains of related facts. Comprehensive experiments demonstrate that RIPPLECOT significantly outperforms the state-of-the-art on the ripple effect, achieving accuracy gains ranging from 7.8% to 87.1%. RIPPLECOT is open-source and available at https://github.com/zzhao71/RippleCOT.