georg-jung / fastberttokenizer Goto Github PK

Also, the Decode method currently assumes the first input_id you pass as its argument represents the beginning of a word. If the first passed id represents a word suffix - or, read: some arbitrary position in a list of input_ids, which doesn't happen to represent the start of a word -, it will probably throw a KeyNotFoundException. This probably isn't the best way how this method could work and I'm happy to fix that - if it helps you I could do it quite soon, please let me know.

#38 (comment)

Thanks to open source, this project has been a great help to me, and I hope it can add support for .NET Framework 4.8.

Hi,

I came with the issue that I need to get the "string" representations of the tokens returned by the Encode method. Alternative can be to get the offsets. Is there a chance to get the offsets/string of a input_id ?

can implement like this：

AutoTokenizer.from_pretrained("test")

form a director 。

thanks

See e.g. https://github.com/YohDeadfall/Yoh.Text.Segmentation used by https://github.com/YohDeadfall/Yoh.Text.Json.NamingPolicies. See also dotnet/runtime#69613.