Comments (4)
zh-plus
commented on June 12, 2024
可能是tiktoken的版本的原因，可以看一下你的pip环境吗？
把pip list的输出结果贴上来就行。
from openlrc.
infinitebook
commented on June 12, 2024
可能是tiktoken的版本的原因，可以看一下你的pip环境吗？
把
pip list的输出结果贴上来就行。
新年快乐~
我重装了一个conda环境，依旧报错
环境如下：
(open_lrc_test2) C:\Users\infin>conda list
packages in environment at D:\conda\envs\open_lrc_test2:
Name Version Build Channel
aiohttp 3.9.1 pypi_0 pypi
aiosignal 1.3.1 pypi_0 pypi
annotated-types 0.6.0 pypi_0 pypi
antlr4-python3-runtime 4.9.3 pypi_0 pypi
anyio 4.2.0 pypi_0 pypi
appdirs 1.4.4 pypi_0 pypi
attrs 23.1.0 pypi_0 pypi
audioread 3.0.1 pypi_0 pypi
av 10.0.0 pypi_0 pypi
blas 1.0 mkl
blis 0.7.11 pypi_0 pypi
brotli-python 1.0.9 py311hd77b12b_7
bzip2 1.0.8 he774522_0
ca-certificates 2023.12.12 haa95532_0
catalogue 2.0.10 pypi_0 pypi
certifi 2023.11.17 py311haa95532_0
cffi 1.16.0 py311h2bbff1b_0
charset-normalizer 2.0.4 pyhd3eb1b0_0
click 8.1.7 pypi_0 pypi
cloudpathlib 0.16.0 pypi_0 pypi
colorama 0.4.6 pypi_0 pypi
coloredlogs 15.0.1 pypi_0 pypi
colorlog 6.8.0 pypi_0 pypi
confection 0.1.4 pypi_0 pypi
cryptography 41.0.7 py311h89fc84f_0
ctranslate2 3.23.0 pypi_0 pypi
cuda-cccl 12.3.101 0 nvidia
cuda-cudart 11.8.89 0 nvidia
cuda-cudart-dev 11.8.89 0 nvidia
cuda-cupti 11.8.87 0 nvidia
cuda-libraries 11.8.0 0 nvidia
cuda-libraries-dev 11.8.0 0 nvidia
cuda-nvrtc 11.8.89 0 nvidia
cuda-nvrtc-dev 11.8.89 0 nvidia
cuda-nvtx 11.8.86 0 nvidia
cuda-profiler-api 12.3.101 0 nvidia
cuda-runtime 11.8.0 0 nvidia
cymem 2.0.8 pypi_0 pypi
deepfilterlib 0.5.6 pypi_0 pypi
deepfilternet 0.5.6 pypi_0 pypi
distro 1.9.0 pypi_0 pypi
en-core-web-sm 3.7.1 pypi_0 pypi
faster-whisper 0.10.0 pypi_0 pypi
ffmpeg-normalize 1.27.7 pypi_0 pypi
ffmpeg-progress-yield 0.7.8 pypi_0 pypi
ffmpeg-python 0.2.0 pypi_0 pypi
filelock 3.13.1 py311haa95532_0
filetype 1.2.0 pypi_0 pypi
flatbuffers 23.5.26 pypi_0 pypi
freetype 2.12.1 ha860e81_0
frozenlist 1.4.1 pypi_0 pypi
fsspec 2023.12.2 pypi_0 pypi
future 0.18.3 pypi_0 pypi
giflib 5.2.1 h8cc25b3_3
gmpy2 2.1.2 py311h7f96b67_0
h11 0.14.0 pypi_0 pypi
httpcore 1.0.2 pypi_0 pypi
httpx 0.26.0 pypi_0 pypi
huggingface-hub 0.20.1 pypi_0 pypi
humanfriendly 10.0 pypi_0 pypi
idna 3.4 py311haa95532_0
intel-openmp 2023.1.0 h59b6b97_46320
jaconvv2 0.4 pypi_0 pypi
jinja2 3.1.2 py311haa95532_0
jpeg 9e h2bbff1b_1
langcodes 3.3.0 pypi_0 pypi
language-data 1.1 pypi_0 pypi
lerc 3.0 hd77b12b_0
libcublas 11.11.3.6 0 nvidia
libcublas-dev 11.11.3.6 0 nvidia
libcufft 10.9.0.58 0 nvidia
libcufft-dev 10.9.0.58 0 nvidia
libcurand 10.3.4.101 0 nvidia
libcurand-dev 10.3.4.101 0 nvidia
libcusolver 11.4.1.48 0 nvidia
libcusolver-dev 11.4.1.48 0 nvidia
libcusparse 11.7.5.86 0 nvidia
libcusparse-dev 11.7.5.86 0 nvidia
libdeflate 1.17 h2bbff1b_1
libffi 3.4.4 hd77b12b_0
libjpeg-turbo 2.0.0 h196d8e1_0
libnpp 11.8.0.86 0 nvidia
libnpp-dev 11.8.0.86 0 nvidia
libnvjpeg 11.9.0.86 0 nvidia
libnvjpeg-dev 11.9.0.86 0 nvidia
libpng 1.6.39 h8cc25b3_0
libtiff 4.5.1 hd77b12b_0
libuv 1.44.2 h2bbff1b_0
libwebp 1.3.2 hbc33d0d_0
libwebp-base 1.3.2 h2bbff1b_0
lingua-language-detector 1.3.4 pypi_0 pypi
loguru 0.7.2 pypi_0 pypi
lz4-c 1.9.4 h2bbff1b_0
marisa-trie 0.7.8 pypi_0 pypi
markupsafe 2.1.1 py311h2bbff1b_0
mkl 2023.1.0 h6b88ed4_46358
mkl-service 2.4.0 py311h2bbff1b_1
mkl_fft 1.3.8 py311h2bbff1b_0
mkl_random 1.2.4 py311h59b6b97_0
mpc 1.1.0 h7edee0f_1
mpfr 4.0.2 h62dcd97_1
mpir 3.0.0 hec2e145_1
mpmath 1.3.0 py311haa95532_0
multidict 6.0.4 pypi_0 pypi
murmurhash 1.0.10 pypi_0 pypi
networkx 3.1 py311haa95532_0
numpy 1.26.2 py311hdab7c0b_0
numpy-base 1.26.2 py311hd01c5d8_0
omegaconf 2.3.0 pypi_0 pypi
onnxruntime 1.16.3 pypi_0 pypi
openai 1.6.1 pypi_0 pypi
openjpeg 2.4.0 h4fc8c34_0
openlrc 1.0.1 pypi_0 pypi
openssl 3.0.12 h2bbff1b_0
packaging 23.2 pypi_0 pypi
pandas 2.1.4 pypi_0 pypi
pillow 10.0.1 py311h045eedc_0
pip 23.3.1 py311haa95532_0
preshed 3.0.9 pypi_0 pypi
protobuf 4.25.1 pypi_0 pypi
punctuators 0.0.5 pypi_0 pypi
pycparser 2.21 pyhd3eb1b0_0
pydantic 2.5.3 pypi_0 pypi
pydantic-core 2.14.6 pypi_0 pypi
pyopenssl 23.2.0 py311haa95532_0
pyreadline3 3.4.1 pypi_0 pypi
pysbd 0.3.4 pypi_0 pypi
pysocks 1.7.1 py311haa95532_0
python 3.11.5 he1021f5_0
python-dateutil 2.8.2 pypi_0 pypi
pytorch 2.1.2 py3.11_cuda11.8_cudnn8_0 pytorch
pytorch-cuda 11.8 h24eeafa_5 pytorch
pytorch-mutex 1.0 cuda pytorch
pytz 2023.3.post1 pypi_0 pypi
pyyaml 6.0.1 py311h2bbff1b_0
regex 2023.12.25 pypi_0 pypi
requests 2.31.0 py311haa95532_0
sentencepiece 0.1.99 pypi_0 pypi
setuptools 68.2.2 py311haa95532_0
six 1.16.0 pypi_0 pypi
smart-open 6.4.0 pypi_0 pypi
sniffio 1.3.0 pypi_0 pypi
soundfile 0.12.1 pypi_0 pypi
spacy 3.7.2 pypi_0 pypi
spacy-legacy 3.0.12 pypi_0 pypi
spacy-loggers 1.0.5 pypi_0 pypi
sqlite 3.41.2 h2bbff1b_0
srsly 2.4.8 pypi_0 pypi
sympy 1.12 py311haa95532_0
tbb 2021.8.0 h59b6b97_0
thinc 8.2.2 pypi_0 pypi
tiktoken 0.3.3 pypi_0 pypi
tk 8.6.12 h2bbff1b_0
tokenizers 0.15.0 pypi_0 pypi
torchaudio 2.1.2 pypi_0 pypi
torchvision 0.16.2 pypi_0 pypi
tqdm 4.66.1 pypi_0 pypi
typer 0.9.0 pypi_0 pypi
typing_extensions 4.7.1 py311haa95532_0
tzdata 2023.4 pypi_0 pypi
urllib3 1.26.18 py311haa95532_0
vc 14.2 h21ff451_1
vs2015_runtime 14.27.29016 h5e58377_2
wasabi 1.1.2 pypi_0 pypi
weasel 0.3.4 pypi_0 pypi
wheel 0.41.2 py311haa95532_0
win32-setctime 1.1.0 pypi_0 pypi
win_inet_pton 1.1.0 py311haa95532_0
xz 5.4.5 h8cc25b3_0
yaml 0.2.5 he774522_0
yarl 1.9.4 pypi_0 pypi
zhconv 1.4.3 pypi_0 pypi
zlib 1.2.13 h8cc25b3_0
zstd 1.5.5 hd43e919_0
运行test chatbot 报错如下
Error
Traceback (most recent call last):
File "C:\Users\infin\Documents\GitHub\openlrc\tests\test_chatbot.py", line 32, in test_estimate_fee
fee = bot.estimate_fee(messages)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\infin\Documents\GitHub\openlrc\openlrc\chatbot.py", line 61, in estimate_fee
token_map[message['role']] += get_text_token_number(message['content'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\infin\Documents\GitHub\openlrc\openlrc\utils.py", line 103, in get_text_token_number
tokens = tiktoken.encoding_for_model(model).encode(text)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\conda\envs\open_lrc_test2\Lib\site-packages\tiktoken\model.py", line 75, in encoding_for_model
return get_encoding(encoding_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\conda\envs\open_lrc_test2\Lib\site-packages\tiktoken\registry.py", line 63, in get_encoding
enc = Encoding(**constructor())
^^^^^^^^^^^^^
File "D:\conda\envs\open_lrc_test2\Lib\site-packages\tiktoken_ext\openai_public.py", line 64, in cl100k_base
mergeable_ranks = load_tiktoken_bpe(
^^^^^^^^^^^^^^^^^^
File "D:\conda\envs\open_lrc_test2\Lib\site-packages\tiktoken\load.py", line 115, in load_tiktoken_bpe
return {
^
File "D:\conda\envs\open_lrc_test2\Lib\site-packages\tiktoken\load.py", line 117, in <dictcomp>
for token, rank in (line.split() for line in contents.splitlines() if line)
^^^^^^^^^^^
ValueError: too many values to unpack (expected 2)
from openlrc.
infinitebook
commented on June 12, 2024
可能是tiktoken版本的原因，可以看看你的pip环境吗？
把
pip list输出结果贴上来就行。
又重装了一个conda环境，之前是用conda命令装的包，现在全用pip安装，pip check 之后发现numba和numpy的版本好像默认就有点冲突，更新了numba版本之后不冲突了，但是跟tiktoken没啥关系，tiktoken依旧报错，真是没办法了😭😭
大佬的代码写的真好❤️很想用上😭
环境如下
(openlrc) C:\Users\infin>pip list
Package Version
-------------------------- ------------
aiohttp 3.9.1
aiosignal 1.3.1
annotated-types 0.6.0
antlr4-python3-runtime 4.9.3
anyio 4.2.0
appdirs 1.4.4
attrs 23.2.0
audioread 3.0.1
av 10.0.0
blis 0.7.11
catalogue 2.0.10
certifi 2022.12.7
cffi 1.16.0
charset-normalizer 2.1.1
click 8.1.7
cloudpathlib 0.16.0
colorama 0.4.6
coloredlogs 15.0.1
colorlog 6.8.0
colorthief 0.2.1
confection 0.1.4
ctranslate2 3.23.0
customtkinter 5.2.0
cymem 2.0.8
darkdetect 0.8.0
DeepFilterLib 0.5.6
DeepFilterNet 0.5.6
distro 1.9.0
en-core-web-sm 3.7.1
faster-whisper 0.10.0
ffmpeg-normalize 1.27.7
ffmpeg-progress-yield 0.7.8
ffmpeg-python 0.2.0
filelock 3.9.0
filetype 1.2.0
flatbuffers 23.5.26
frozenlist 1.4.1
fsspec 2023.12.2
future 0.18.3
h11 0.14.0
httpcore 1.0.2
httpx 0.26.0
huggingface-hub 0.20.2
humanfriendly 10.0
idna 3.4
jaconvV2 0.4
Jinja2 3.1.2
langcodes 3.3.0
language-data 1.1
lingua-language-detector 1.3.4
llvmlite 0.41.0
loguru 0.7.2
marisa-trie 0.7.8
MarkupSafe 2.1.2
more-itertools 10.1.0
mpmath 1.3.0
multidict 6.0.4
murmurhash 1.0.10
networkx 3.0
numba 0.58.1
numpy 1.26.3
omegaconf 2.3.0
onnxruntime 1.16.3
openai 1.6.1
openai-whisper 20230918
openlrc 1.0.1
packaging 23.2
pandas 2.1.4
Pillow 9.3.0
pip 23.3.2
preshed 3.0.9
protobuf 4.25.1
punctuators 0.0.5
pycparser 2.21
pydantic 2.5.3
pydantic_core 2.14.6
pyreadline3 3.4.1
pysbd 0.3.4
PySide6 6.4.2
PySide6-Addons 6.4.2
PySide6-Essentials 6.4.2
PySide6-Fluent-Widgets 1.4.5
PySideSix-Frameless-Window 0.3.6
python-dateutil 2.8.2
pytz 2023.3.post1
pywin32 306
PyYAML 6.0.1
regex 2023.10.3
requests 2.28.1
scipy 1.11.4
sentencepiece 0.1.99
setuptools 65.5.0
shiboken6 6.4.2
six 1.16.0
smart-open 6.4.0
sniffio 1.3.0
soundfile 0.12.1
spacy 3.7.2
spacy-legacy 3.0.12
spacy-loggers 1.0.5
srsly 2.4.8
sympy 1.12
thinc 8.2.2
tiktoken 0.3.3
tokenizers 0.15.0
torch 2.1.0+cu118
torchaudio 2.1.0+cu118
torchvision 0.16.0+cu118
tqdm 4.66.1
typer 0.9.0
typing_extensions 4.9.0
tzdata 2023.4
urllib3 1.26.13
wasabi 1.1.2
weasel 0.3.4
win32-setctime 1.1.0
yarl 1.9.4
zhconv 1.4.3
from openlrc.
zh-plus
commented on June 12, 2024
看了相关库的版本（tiktoken=0.3.3），跟我本地是一致的，应该不是库版本的原因。
我搜了下tiktoken库的issue，可能跟用了以前版本的cache有关系，你可以参考下这个回答：openai/tiktoken#136 (comment) 试试能不能解决问题。
from openlrc.
Related Issues (20)
- AssertionError case sensitivity issue HOT 4
- exception when running HOT 2
- today's update seems to not transcribe right HOT 7
- 音频预处理阶段出错 Runtime exception during preprocess HOT 5
- Could not locate cublasLt64_12.dll. Please make sure it is in your library path!
- preprocessed 执行失败 HOT 3
- TypeError: transcribe() got an unexpected keyword argument 'repetition_penalty' HOT 2
- FileNotFoundError: [WinError 2] The system cannot find the file specified: 'data\\preprocessed\\track5_ln.wav' -> 'data\\preprocessed\\track5_preprocessed.wav' HOT 5
- Suggest to use cloudflare workers ai api to lower the usage threshold HOT 1
- lrcer.run执行报错 HOT 1
- skip_trans 选项好像有点问题，不会生成lrc文件 HOT 3
- 小建议-清除中间文件 HOT 2
- gui好像是。有点问题的 HOT 3
- 运行openlrc gui后报错 File does not exist: openlrc/gui/home.py HOT 3
- 能否直接翻译 `srt` 字幕文件？ HOT 1
- 自定义base_url不能用 HOT 4
- 如何指定 `srt` 格式输出？ HOT 1
- TypeError: WhisperModel.transcribe() got an unexpected keyword argument 'hotwords' HOT 3
- 安装错误 ERROR: Failed building wheel for thinc HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openlrc.