I would like to know the difference between this project and Nvidia FastTransformer. about lmdeploy HOT 4 CLOSED

TurboMind is indeed developed based on fastertransformer, but if you use beyond compare to see the difference, the two are so different that they are not the same repo

1. FT not support LLaMa, so you can not directy inference on it
2. FT has no KV Cache Manager, no reliable quantization, int8_model==2 actually does not work
3. FT has no fmha and many trivial optimizations

Hopes @lzhangzz give more description.

from lmdeploy.

@happened in addition to @tpoisonooo's response,

1. FT's context decoder implementation requires k_len == q_len thus context decoder is only used in the first round of a conversation. Our implementation supports context decoding new input tokens for throughout the conversation
2. With our caching mechanism only new input tokens will be decoded (not the entire history) unless the sequence has been evicted from the cache
3. Our KV Cache Manager implements LRU policy so that least recently used sequence will be evicted into token indices (the most compact form of KV cache) and recomputed when requested, so you don't have to worry about OOM
4. We support persistent batch (you may know it as "continuous batching") for both Python API or serving with tritonserver

from lmdeploy.

#95

from lmdeploy.

@happened Please read this PR and give your comments #101

from lmdeploy.