Presence of character Ġ before each token in output about ecco HOT 2 OPEN

Happened to me using GPT-2 and solved this issue by adding the following line:
if self.config['token_prefix'] is not None and token[0] == self.config['token_prefix']: token = token[1:]

right after the first loop line of nmf.explore() method:
for idx, token in enumerate(self.tokens[input_sequence]): # self.tokens[:-1]
if self.config['token_prefix'] is not None and token[0] == self.config['token_prefix']: token = token[1:]
type = "input" if idx < self.n_input_tokens else 'output'
tokens.append({'token': token,
'token_id': int(self.token_ids[input_sequence][idx]),
# 'token_id': int(self.token_ids[idx]),
'type': type,
# 'value': str(components[0][comp_num][idx]), # because json complains of floats
'position': idx
})

from ecco.

Yeah, that shouldn't happen. A bunch of tokenizers have a character like Ġ in the beginning of a token to indicate that the token is linked to whatever token comes before them in the sequence. Which is why rendering the output needs to run in tandem with the tokenizer and its settings.

from ecco.