This is a repository for "SWEA: Updating Factual Knowledge in Large Language Models via Subject Word Embedding Altering"
Overview of SWEA$\oplus$OS. In fusion stage, we first optimize a learnable embedding vector for target knowledge "Currently, Nvidia's most advanced GPU is the \underline{B200}." Second, using knowledge attribution method, we find the KEDs of `Nvidia' regarding "its most advanced GPU". Finally, we fuse the optimized embedding vector with these KEDs subtracted to obtain the editing embeddings. In inference stage, we add these editing embeddings to the embedding of the subject `Nvidia' for inference.
- At least one A100/A800 80G GPU.
- Environment
conda create -n swea python=3.10 pip install -r requirements.txt
An example for editing Llama-2 (7B) on counterfact dataset using SWE $\oplus$ OS, you can also edit GPT-J (6B).
python evaluate.py --model_path [your model path] --model_name Llama-2-7b-hf --alg_name SWEAOS --hparams_fname Llama-2-7b-hf.json --ds_name mcf --num_edits [num]python summarize.py --dir_name SWEAOS --runs run_<run1>python main_counterfact_zsre_squen.py --model_path [your model path] --model_name Llama-2-7b-hf --hparams_fname Llama-2-7b-hf.jsonWe have released the filtered datasets in filtered datasets. If you want to filter the datasets from scratch, using `./fileter_benchmark.sh' or refer to RippleEdits.
python main_ripple_edits.pyOur code is based on MEMIT, PMET, and RippleEdits.
Xiaopeng Li, Shasha Li, Bin Ji, Shezheng Song, Xi Wang, Jun Ma, Jie Yu, Xiaodon Liu, Jing Wang, Weimin Zhang.
SWEA: Updating Factual Knowledge in Large Language Models via Subject Word Embedding Altering.
arXiv preprint arXiv:2401.17809 (2024).
@article{li2024swea,
title={SWEA: Updating Factual Knowledge in Large Language Models via Subject Word Embedding Altering},
author={Li, Xiaopeng and Li, Shasha and Ji, Bin and Song, Shezheng and Wang, Xi and Ma, Jun and Yu, Jie and Liu, Xiaodong and Wang, Jing and Zhang, Weimin},
journal={arXiv preprint arXiv:2401.17809},
year={2024}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent